* [PATCH] drm/amdkfd: Flush the process wq before creating a kfd_process
@ 2024-04-26 18:55 Lancelot SIX
2024-04-26 21:58 ` Felix Kuehling
0 siblings, 1 reply; 2+ messages in thread
From: Lancelot SIX @ 2024-04-26 18:55 UTC (permalink / raw
To: amd-gfx; +Cc: Lancelot SIX
There is a race condition when re-creating a kfd_process for a process.
This has been observed when a process under the debugger executes
exec(3). In this scenario:
- The process executes exec.
- This will eventually release the process's mm, which will cause the
kfd_process object associated with the process to be freed
(kfd_process_free_notifier decrements the reference count to the
kfd_process to 0). This causes kfd_process_ref_release to enqueue
kfd_process_wq_release to the kfd_process_wq.
- The debugger receives the PTRACE_EVENT_EXEC notification, and tries to
re-enable AMDGPU traps (KFD_IOC_DBG_TRAP_ENABLE).
- When handling this request, KFD tries to re-create a kfd_process.
This eventually calls kfd_create_process and kobject_init_and_add.
At this point the call to kobject_init_and_add can fail because the
old kfd_process.kobj has not been freed yet by kfd_process_wq_release.
This patch proposes to avoid this race by making sure to drain
kfd_process_wq before creating a new kfd_process object. This way, we
know that any cleanup task is done executing when we reach
kobject_init_and_add.
Signed-off-by: Lancelot SIX <lancelot.six@amd.com>
---
drivers/gpu/drm/amd/amdkfd/kfd_process.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
index 58c1fe542193..451bb058cc62 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
@@ -829,6 +829,14 @@ struct kfd_process *kfd_create_process(struct task_struct *thread)
if (process) {
pr_debug("Process already found\n");
} else {
+ /* If the process just called exec(3), it is possible that the
+ * cleanup of the kfd_process (following the release of the mm
+ * of the old process image) is still in the cleanup work queue.
+ * Make sure to drain any job before trying to recreate any
+ * resource for this process.
+ */
+ flush_workqueue(kfd_process_wq);
+
process = create_process(thread);
if (IS_ERR(process))
goto out;
base-commit: cf743996352e327f483dc7d66606c90276f57380
--
2.34.1
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [PATCH] drm/amdkfd: Flush the process wq before creating a kfd_process
2024-04-26 18:55 [PATCH] drm/amdkfd: Flush the process wq before creating a kfd_process Lancelot SIX
@ 2024-04-26 21:58 ` Felix Kuehling
0 siblings, 0 replies; 2+ messages in thread
From: Felix Kuehling @ 2024-04-26 21:58 UTC (permalink / raw
To: Lancelot SIX, amd-gfx
On 2024-04-26 14:55, Lancelot SIX wrote:
> There is a race condition when re-creating a kfd_process for a process.
> This has been observed when a process under the debugger executes
> exec(3). In this scenario:
> - The process executes exec.
> - This will eventually release the process's mm, which will cause the
> kfd_process object associated with the process to be freed
> (kfd_process_free_notifier decrements the reference count to the
> kfd_process to 0). This causes kfd_process_ref_release to enqueue
> kfd_process_wq_release to the kfd_process_wq.
> - The debugger receives the PTRACE_EVENT_EXEC notification, and tries to
> re-enable AMDGPU traps (KFD_IOC_DBG_TRAP_ENABLE).
> - When handling this request, KFD tries to re-create a kfd_process.
> This eventually calls kfd_create_process and kobject_init_and_add.
>
> At this point the call to kobject_init_and_add can fail because the
> old kfd_process.kobj has not been freed yet by kfd_process_wq_release.
>
> This patch proposes to avoid this race by making sure to drain
> kfd_process_wq before creating a new kfd_process object. This way, we
> know that any cleanup task is done executing when we reach
> kobject_init_and_add.
>
> Signed-off-by: Lancelot SIX <lancelot.six@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
> ---
> drivers/gpu/drm/amd/amdkfd/kfd_process.c | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
> index 58c1fe542193..451bb058cc62 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
> @@ -829,6 +829,14 @@ struct kfd_process *kfd_create_process(struct task_struct *thread)
> if (process) {
> pr_debug("Process already found\n");
> } else {
> + /* If the process just called exec(3), it is possible that the
> + * cleanup of the kfd_process (following the release of the mm
> + * of the old process image) is still in the cleanup work queue.
> + * Make sure to drain any job before trying to recreate any
> + * resource for this process.
> + */
> + flush_workqueue(kfd_process_wq);
> +
> process = create_process(thread);
> if (IS_ERR(process))
> goto out;
>
> base-commit: cf743996352e327f483dc7d66606c90276f57380
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2024-04-26 21:58 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-04-26 18:55 [PATCH] drm/amdkfd: Flush the process wq before creating a kfd_process Lancelot SIX
2024-04-26 21:58 ` Felix Kuehling
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.