[DNM][WiP][PoC] userspace LL scheduling: LLEXT & multicore#10945
Draft
lyakh wants to merge 125 commits into
Draft
[DNM][WiP][PoC] userspace LL scheduling: LLEXT & multicore#10945lyakh wants to merge 125 commits into
lyakh wants to merge 125 commits into
Conversation
Add a built option HOST_DMA_IPC_POSITION_UPDATES to control whether functionality to send IPC stream position updates is enabled or not. Most platforms provide more efficient means for host to monitor DMA state, so this code is in most cases unncessary. The current IPC sending code (from audio context) also assume kernel context, so making this functionality user-space compatible will require extra work. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Drop the IRQ disable/enable in ipc4_search_for_drv(). The driver list is only modified at FW boot and when a new driver is registered at runtime via SOF_IPC4_GLB_LOAD_LIBRARY IPC. ipc4_search_for_drv() is only used when processing IPC messages. As IPC processing is serialized, it is not possible for the driver list to be modified concurrently with a call to ipc4_search_for_drv(). Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
The component driver list is only modified at FW boot and at runtime when a library is loaded. At boot, module init runs serially on the primary core (Zephyr SYS_INIT at APPLICATION level, before secondary cores are started; .initcall walked on a single core for XTOS). At runtime, registration happens from the IPC thread, which is serialized with only one command processed at a time. These two phases never overlap, as IPC message processing only begins after boot completes, so the list can never be modified concurrently. The lock was also already incoherent: comp_set_adapter_ops() iterate the list without holding the lock, so it provided no real mutual exclusion. Drop the spinlock from comp_register() and comp_unregister(), and from the UUID search in the IPC3 get_drv() reader. Remove the now-unused lock field from struct comp_driver_list and its initialization. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Add support for registering user-space LL tasks, and ability to use the task scheduling functions from user-space. The implementation splits scheduler list into kernel and user portions if SOF is built with CONFIG_SOF_USERSPACE_LL. A scheduler type can be either maintained in kernel or user, never both. With this patch, the SOF_SCHEDULE_LL_TIMER is moved to user managed if CONFIG_SOF_USERSPACE_LL is used. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Ensure the scheduler objects and lists of schedulers are allocated such that they can be used with both kernel and user-space LL scheduler implementations. The SOF_MEM_FLAG_KERNEL flag is removed. This flag has been a no-op for a while, and given scheduler list is not always in kernel anymore, it would be highly confusing to keep it. When CONFIG_SOF_USERSPACE_LL is set, the context of all schedulers is managed in the LL user-space domain. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
The real fix is to remove the locking around dai_get_properties() altogether, but this depends on fixes in Zephyr DAI drivers. To unblock user-space work, remove the calls to spinlocks for now. This opens up possibility to hit issues with concurrent playback and capture cases on multiple cores, so this commit remains a WIP until fixes in Zephyr drivers land. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Modify code to allocate DAI properties object on stack and use dai_get_properties_copy(). This is required when DAI code is run in user-space and a syscall is needed to talk to the DAI driver. It's not possible to return a pointer to kernel memory, so instead data needs to be copied to caller stack. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Turn the pdata->sem into a dynamic object in userspace LL builds, implemented with Zephyr k_sem. Add POSIX no-op stubs for sys_sem to maintain testbench build compatibility. Keep statically allocated semaphore for kernel LL builds. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Add function scheduler_get_data_for_core() to look up scheduler data for a particular type of scheduler. This variant allows to pass the core number as an argument, so it can be called from unprivileged code. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Add function user_ll_grant_access() to allow other threads to access the scheduler mutex. This is needed if work is submitted from other threads to the scheduler. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
When LL scheduler is run in user-space, use a different Zephyr thread name. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
The COHERENT_CHECK_NONSHARED_CORES debug macros call cpu_get_id() which invokes arch_proc_id() - a privileged hardware register read that faults in user-space context. Disable the entire debug block at compile time when CONFIG_SOF_USERSPACE_LL is enabled. This also fixes the same latent issue in CORE_CHECK_STRUCT and CORE_CHECK_STRUCT_INIT. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Add new functions to lock/unlock the LL scheduler for a given core. This is intended for audio application code that needs to modify the audio pipelines and needs an interface to get exclusive access to the pipelines on a particular core. This interface is specific to SOF builds with CONFIG_SOF_USERSPACE_LL. If LL scheduler is running in kernel space, there is option to disable interrupts for similar effect. For now these code paths are kept separate. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
In user-space LL builds (CONFIG_SOF_USERSPACE_LL), the IPC user thread cannot block interrupts while making modifications to the audio graph. To workaround this limitation, one could either protect each pipeline object with locks, or keep the LL level lock held while executing LL tasks. This patch implements support for the latter approach. If building SOF for user LL, do not release the lock when running a task. This reduces number of syscalls during a LL iteration, and allows to safely implement IPC handlers that need to modify the audio graph. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Modify the locking approach for CONFIG_SOF_USERSPACE_LL builds. Kernel LL implementation heavily relies on ability to disable interrupts when IPC handler is modifying the graph. This ensures a new LL tick and execution of a new graph cycle does not start before the graph modifications done by IPC handler are complete. In user-space, this approach is not available as user-space thread cannot disable interrupts. In commit 1e59ce2 ("pipeline: protect component connections with a mutex"), a sys_mutex based locking was implemented to protect the component list and modifications to it. This approach does not scale in the end as this would require taking the mutex for each component of each pipeline, and take the locks on every LL cycle tick. This results in significant system call overhead. Additionally Zephyr sys_mutex does not work correctly if the lock object is put into dynamically allocated user memory. In this commit, locking the LL graph is moved to a higher level. A single lock is used to protect the whole LL graph, and the lock is taken at start of LL tick. The same lock is taken by the IPC handlers when modifications to the graph are taken. The mutex interface supports priority inversion, so this usage is safe if LL timer tick happens while IPC processing is still in progress. The patch only changes behaviour for userspace LL SOF builds. If LL scheduling is kept in kernel, locking is done as before. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Modify the checks in zephyr_ll_assert_core() to make them safe to call from user-space LL threads. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Needs more review, but makes the tests pass again. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Build fails when building with CONFIG_THREAD_NAME disabled. Fix the issue by conditional compilation of code using CONFIG_THREAD_MAX_NAME_LEN. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Add support to run pipeline_schedule_triggered() in user-space. Use the user_ll_lock/unlock_sched() interface if building with CONFIG_SOF_USERSPACE_LL. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
In user-space LL builds the low-latency scheduler runs its work in a dedicated privileged domain thread, created together with its timer and access grants by scheduler_init_context() (zephyr_ll_init_context() -> domain_thread_init()). This context is per-core and must exist on every core that runs LL tasks. So far it was only established for the primary core, so LL tasks could not be scheduled on secondary cores when CONFIG_SOF_USERSPACE_LL is enabled. Allocate an LL task in secondary_core_init() and run scheduler_init_context() on it, giving each secondary core its own LL domain thread. A dedicated sec_core_init UUID is registered for the task. The whole block is compiled in only for CONFIG_SOF_USERSPACE_LL. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Copier set_chmap() blocks IRQs to atomically update the converters. This code is not safe to be moved to user-space, so replace the locks with calls to block LL scheduler execution. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Currently building with userspace LL and the debug overlay generates non-functional images. This is due to two incompatibilities: (1) cpu_get_id() called in userspace in zephyr_ll_task_init() and (2) the heap_alloc boot-test currently causing an exception. Fix the former and disable the latter to re-enabld debug builds with the userspace enabled. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
set_processing_mode() and get_processing_mode() methods of struct module_interface aren't unused, they are used by IADK. Fix respective comments. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
The global driver list should never be accessed from the audio context. Add an assertion for that. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
scheduler_get_data_for_core() returns void pointer, no need to type-cast it to other pointer types. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
drivers in comp_unregister() is unused, remove it. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
syscall-related code can be used when CONFIG_USERSPACE is undefined too. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Define the "core" variable only when it is used. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Define the "ipc" variable only for cases, when it's used. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
ipc4_init_module_instance() should be called when CONFIG_SOF_USERSPACE_LL isn't selected but also when initializing a DP module. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
If LL runs in userspace, it needs access to loaded LLEXT modules, running in DP more too. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
scheduler_is_user() should return true for DP as well. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
schedule_free() is only called from testbench and stand-alone ztest. Remove it and all scheduler .scheduler_free() methods for all other builds. Also fix memory leaks in the Zephyr LL version. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
When running the LL scheduler in userspace, it can happen that no kernel-mode schedulers get registered on a running secondary core. To recognise such cases add a check for userspace schedulers to check_restore(). Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
In kernel context cpu_is_me() should return a correct result for correct multicore support. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Make ll_sch_is_current() available for builds with CONFIG_COLD_STORE_EXECUTE_DEBUG unselected. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Add two syscall functions to allocate and map, and to unmap vregion for userspace modules. For now only used for DP modules. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
vregion_get() and vregion_put() should also be callable from the userspace. Make them syscalls. Also remove redundant symbol exporting since the vregion API shouldn't be used directly by LLEXT modules. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
scheduler_dp_ll_tick() is currently registered as a notifier callback, but it's always triggered deterministically, always with the same-core-only flag, which leads to it being called immediately. So the notifier only adds a layer of indirection and reduces clarity. Replace it with a direct function call. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
scheduler_dp_ll_tick(() has to recalculate DP deadlines and reschedule DP threads. Make it a syscall to be able to call it from the userspace LL scheduler. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Remove a repeated line in arch_schedulers_get() and update arch_user_schedulers_get() to look more like the former. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Make scheduler_dp_internal_free() a syscall in the "application" DP implementation. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
In case of userspace LL scheduling the (also userspace) IPC thread needs access rights to DP assets like the thread itself and its stack and synchronisation primitives. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
zephyr_ll_domain() and scheduler_get_task_info_ll() are needed in non-userspace builds too, make them universally available. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
The LL userspace thread has to interact with the DP one. Grant required rights. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
When LL runs in userspace, multiple DP functions are called in userspace mode too. They cannot use privileged instructions then. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
DP scheduler operations, instance data and DP module memory have to be accessible to the userspace LL scheduler. Allocate dynamic data on the userspace heap and place static data in the userspace accessible ELF section. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Remove duplicate ll_schedule_domain.h inclusion in ipc-common.c. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Switching to the userspace mode in DP and LL cases differs. Add a comment to explain that. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This includes #10558 and my patches on top to enable LLEXT and multicore. Current status: passes simple tests with nocodec with both core 0 and core 1 streaming. 2 streams simultaneously run into a problem when the first of them terminates. WiP.