From: "Christian König" <christian.koenig@amd.com>
To: "Zeng, Oak" <oak.zeng@intel.com>,
Felix Kuehling <felix.kuehling@amd.com>,
Danilo Krummrich <dakr@redhat.com>,
Dave Airlie <airlied@redhat.com>, Daniel Vetter <daniel@ffwll.ch>,
"Shah, Ankur N" <ankur.n.shah@intel.com>,
"Winiarski, Michal" <michal.winiarski@intel.com>
Cc: "Brost, Matthew" <matthew.brost@intel.com>,
"Thomas.Hellstrom@linux.intel.com"
<Thomas.Hellstrom@linux.intel.com>,
"Welty, Brian" <brian.welty@intel.com>,
"dri-devel@lists.freedesktop.org"
<dri-devel@lists.freedesktop.org>,
"Ghimiray, Himal Prasad" <himal.prasad.ghimiray@intel.com>,
"Gupta, saurabhg" <saurabhg.gupta@intel.com>,
"Bommu, Krishnaiah" <krishnaiah.bommu@intel.com>,
"Vishwanathapura,
Niranjana" <niranjana.vishwanathapura@intel.com>,
"intel-xe@lists.freedesktop.org" <intel-xe@lists.freedesktop.org>
Subject: Re: Making drm_gpuvm work across gpu devices
Date: Fri, 26 Jan 2024 14:23:46 +0100 [thread overview]
Message-ID: <5a0a26b2-3116-4395-b0fc-84b2f248ac15@amd.com> (raw)
In-Reply-To: <SA1PR11MB69913BCFDA5F28199D5E002F927A2@SA1PR11MB6991.namprd11.prod.outlook.com>
[-- Attachment #1: Type: text/plain, Size: 11127 bytes --]
Am 25.01.24 um 19:37 schrieb Zeng, Oak:
>> -----Original Message-----
>> From: Felix Kuehling<felix.kuehling@amd.com>
>> Sent: Thursday, January 25, 2024 12:16 PM
>> To: Zeng, Oak<oak.zeng@intel.com>; Christian König
>> <christian.koenig@amd.com>; Danilo Krummrich<dakr@redhat.com>; Dave
>> Airlie<airlied@redhat.com>; Daniel Vetter<daniel@ffwll.ch>; Shah, Ankur N
>> <ankur.n.shah@intel.com>; Winiarski, Michal<michal.winiarski@intel.com>
>> Cc: Welty, Brian<brian.welty@intel.com>;dri-devel@lists.freedesktop.org; intel-
>> xe@lists.freedesktop.org; Bommu, Krishnaiah<krishnaiah.bommu@intel.com>;
>> Ghimiray, Himal Prasad<himal.prasad.ghimiray@intel.com>;
>> Thomas.Hellstrom@linux.intel.com; Vishwanathapura, Niranjana
>> <niranjana.vishwanathapura@intel.com>; Brost, Matthew
>> <matthew.brost@intel.com>; Gupta, saurabhg<saurabhg.gupta@intel.com>
>> Subject: Re: Making drm_gpuvm work across gpu devices
>>
>>
>> On 2024-01-24 20:17, Zeng, Oak wrote:
>>> Hi Christian,
>>>
>>> Even though I mentioned KFD design, I didn’t mean to copy the KFD
>>> design. I also had hard time to understand the difficulty of KFD under
>>> virtualization environment.
>>>
>> The problem with virtualization is related to virtualization design
>> choices. There is a single process that proxies requests from multiple
>> processes in one (or more?) VMs to the GPU driver. That means, we need a
>> single process with multiple contexts (and address spaces). One proxy
>> process on the host must support multiple guest address spaces.
> My first response is, why processes on the virtual machine can't open /dev/kfd device itself?
Because it's not using SRIOV, we are using native context and so the KFD
driver is on the host and not the guest.
> Also try to picture why base amdgpu driver (which is per hardware device based) doesn't have this problem... creating multiple contexts under single amdgpu device, each context servicing one guest process?
Yes, exactly that.
>> I don't know much more than these very high level requirements, and I
>> only found out about those a few weeks ago. Due to my own bias I can't
>> comment whether there are bad design choices in the proxy architecture
>> or in KFD or both. The way we are considering fixing this, is to enable
>> creating multiple KFD contexts in the same process. Each of those
>> contexts will still represent a shared virtual address space across
>> devices (but not the CPU). Because the device address space is not
>> shared with the CPU, we cannot support our SVM API in this situation.
>>
> One kfd process, multiple contexts, each context has a shared address space across devices.... I do see some complications 😊
Yeah, if the process which talks to the driver is not the process which
initiated the work then the concept to tie anything to device state
obviously doesn't work.
Regards,
Christian.
>
>> I still believe that it makes sense to have the kernel mode driver aware
>> of a shared virtual address space at some level. A per-GPU API and an
>> API that doesn't require matching CPU and GPU virtual addresses would
>> enable more flexibility at the cost duplicate information tracking for
>> multiple devices and duplicate overhead for things like MMU notifiers
>> and interval tree data structures. Having to coordinate multiple devices
>> with potentially different address spaces would probably make it more
>> awkward to implement memory migration. The added flexibility would go
>> mostly unused, except in some very niche applications.
>>
>> Regards,
>> Felix
>>
>>
>>> For us, Xekmd doesn't need to know it is running under bare metal or
>>> virtualized environment. Xekmd is always a guest driver. All the
>>> virtual address used in xekmd is guest virtual address. For SVM, we
>>> require all the VF devices share one single shared address space with
>>> guest CPU program. So all the design works in bare metal environment
>>> can automatically work under virtualized environment. +@Shah, Ankur N
>>> <mailto:ankur.n.shah@intel.com> +@Winiarski, Michal
>>> <mailto:michal.winiarski@intel.com> to backup me if I am wrong.
>>>
>>> Again, shared virtual address space b/t cpu and all gpu devices is a
>>> hard requirement for our system allocator design (which means
>>> malloc’ed memory, cpu stack variables, globals can be directly used in
>>> gpu program. Same requirement as kfd SVM design). This was aligned
>>> with our user space software stack.
>>>
>>> For anyone who want to implement system allocator, or SVM, this is a
>>> hard requirement. I started this thread hoping I can leverage the
>>> drm_gpuvm design to manage the shared virtual address space (as the
>>> address range split/merge function was scary to me and I didn’t want
>>> re-invent). I guess my takeaway from this you and Danilo is this
>>> approach is a NAK. Thomas also mentioned to me drm_gpuvm is a overkill
>>> for our svm address range split/merge. So I will make things work
>>> first by manage address range xekmd internally. I can re-look
>>> drm-gpuvm approach in the future.
>>>
>>> Maybe a pseudo user program can illustrate our programming model:
>>>
>>> Fd0 = open(card0)
>>>
>>> Fd1 = open(card1)
>>>
>>> Vm0 =xe_vm_create(fd0) //driver create process xe_svm on the process's
>>> first vm_create
>>>
>>> Vm1 = xe_vm_create(fd1) //driver re-use xe_svm created above if called
>>> from same process
>>>
>>> Queue0 = xe_exec_queue_create(fd0, vm0)
>>>
>>> Queue1 = xe_exec_queue_create(fd1, vm1)
>>>
>>> //check p2p capability calling L0 API….
>>>
>>> ptr = malloc()//this replace bo_create, vm_bind, dma-import/export
>>>
>>> Xe_exec(queue0, ptr)//submit gpu job which use ptr, on card0
>>>
>>> Xe_exec(queue1, ptr)//submit gpu job which use ptr, on card1
>>>
>>> //Gpu page fault handles memory allocation/migration/mapping to gpu
>>>
>>> As you can see, from above model, our design is a little bit different
>>> than the KFD design. user need to explicitly create gpuvm (vm0 and vm1
>>> above) for each gpu device. Driver internally have a xe_svm represent
>>> the shared address space b/t cpu and multiple gpu devices. But end
>>> user doesn’t see and no need to create xe_svm. The shared virtual
>>> address space is really managed by linux core mm (through the vma
>>> struct, mm_struct etc). From each gpu device’s perspective, it just
>>> operate under its own gpuvm, not aware of the existence of other
>>> gpuvm, even though in reality all those gpuvm shares a same virtual
>>> address space.
>>>
>>> See one more comment inline
>>>
>>> *From:*Christian König<christian.koenig@amd.com>
>>> *Sent:* Wednesday, January 24, 2024 3:33 AM
>>> *To:* Zeng, Oak<oak.zeng@intel.com>; Danilo Krummrich
>>> <dakr@redhat.com>; Dave Airlie<airlied@redhat.com>; Daniel Vetter
>>> <daniel@ffwll.ch>; Felix Kuehling<felix.kuehling@amd.com>
>>> *Cc:* Welty, Brian<brian.welty@intel.com>;
>>> dri-devel@lists.freedesktop.org;intel-xe@lists.freedesktop.org;
>>> Bommu, Krishnaiah<krishnaiah.bommu@intel.com>; Ghimiray, Himal Prasad
>>> <himal.prasad.ghimiray@intel.com>;Thomas.Hellstrom@linux.intel.com;
>>> Vishwanathapura, Niranjana<niranjana.vishwanathapura@intel.com>;
>>> Brost, Matthew<matthew.brost@intel.com>; Gupta, saurabhg
>>> <saurabhg.gupta@intel.com>
>>> *Subject:* Re: Making drm_gpuvm work across gpu devices
>>>
>>> Am 23.01.24 um 20:37 schrieb Zeng, Oak:
>>>
>>> [SNIP]
>>>
>>> Yes most API are per device based.
>>>
>>> One exception I know is actually the kfd SVM API. If you look at the svm_ioctl
>> function, it is per-process based. Each kfd_process represent a process across N
>> gpu devices.
>>>
>>> Yeah and that was a big mistake in my opinion. We should really not do
>>> that ever again.
>>>
>>>
>>> Need to say, kfd SVM represent a shared virtual address space across CPU
>> and all GPU devices on the system. This is by the definition of SVM (shared virtual
>> memory). This is very different from our legacy gpu *device* driver which works
>> for only one device (i.e., if you want one device to access another device's
>> memory, you will have to use dma-buf export/import etc).
>>>
>>> Exactly that thinking is what we have currently found as blocker for a
>>> virtualization projects. Having SVM as device independent feature
>>> which somehow ties to the process address space turned out to be an
>>> extremely bad idea.
>>>
>>> The background is that this only works for some use cases but not all
>>> of them.
>>>
>>> What's working much better is to just have a mirror functionality
>>> which says that a range A..B of the process address space is mapped
>>> into a range C..D of the GPU address space.
>>>
>>> Those ranges can then be used to implement the SVM feature required
>>> for higher level APIs and not something you need at the UAPI or even
>>> inside the low level kernel memory management.
>>>
>>> When you talk about migrating memory to a device you also do this on a
>>> per device basis and *not* tied to the process address space. If you
>>> then get crappy performance because userspace gave contradicting
>>> information where to migrate memory then that's a bug in userspace and
>>> not something the kernel should try to prevent somehow.
>>>
>>> [SNIP]
>>>
>>> I think if you start using the same drm_gpuvm for multiple devices you
>>>
>>> will sooner or later start to run into the same mess we have seen with
>>>
>>> KFD, where we moved more and more functionality from the KFD to the
>> DRM
>>> render node because we found that a lot of the stuff simply doesn't work
>>>
>>> correctly with a single object to maintain the state.
>>>
>>> As I understand it, KFD is designed to work across devices. A single pseudo
>> /dev/kfd device represent all hardware gpu devices. That is why during kfd open,
>> many pdd (process device data) is created, each for one hardware device for this
>> process.
>>>
>>> Yes, I'm perfectly aware of that. And I can only repeat myself that I
>>> see this design as a rather extreme failure. And I think it's one of
>>> the reasons why NVidia is so dominant with Cuda.
>>>
>>> This whole approach KFD takes was designed with the idea of extending
>>> the CPU process into the GPUs, but this idea only works for a few use
>>> cases and is not something we should apply to drivers in general.
>>>
>>> A very good example are virtualization use cases where you end up with
>>> CPU address != GPU address because the VAs are actually coming from
>>> the guest VM and not the host process.
>>>
>>> I don’t get the problem here. For us, under virtualization, both the
>>> cpu address and gpu virtual address operated in xekmd is guest virtual
>>> address. They can still share the same virtual address space (as SVM
>>> required)
>>>
>>> Oak
>>>
>>>
>>>
>>> SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should
>>> not have any influence on the design of the kernel UAPI.
>>>
>>> If you want to do something similar as KFD for Xe I think you need to
>>> get explicit permission to do this from Dave and Daniel and maybe even
>>> Linus.
>>>
>>> Regards,
>>> Christian.
>>>
[-- Attachment #2: Type: text/html, Size: 15425 bytes --]
next prev parent reply other threads:[~2024-01-26 13:24 UTC|newest]
Thread overview: 123+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-17 22:12 [PATCH 00/23] XeKmd basic SVM support Oak Zeng
2024-01-17 22:12 ` [PATCH 01/23] drm/xe/svm: Add SVM document Oak Zeng
2024-01-17 22:12 ` [PATCH 02/23] drm/xe/svm: Add svm key data structures Oak Zeng
2024-01-17 22:12 ` [PATCH 03/23] drm/xe/svm: create xe svm during vm creation Oak Zeng
2024-01-17 22:12 ` [PATCH 04/23] drm/xe/svm: Trace svm creation Oak Zeng
2024-01-17 22:12 ` [PATCH 05/23] drm/xe/svm: add helper to retrieve svm range from address Oak Zeng
2024-01-17 22:12 ` [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range Oak Zeng
2024-04-05 0:39 ` Jason Gunthorpe
2024-04-05 3:33 ` Zeng, Oak
2024-04-05 12:37 ` Jason Gunthorpe
2024-04-05 16:42 ` Zeng, Oak
2024-04-05 18:02 ` Jason Gunthorpe
2024-04-09 16:45 ` Zeng, Oak
2024-04-09 17:24 ` Jason Gunthorpe
2024-04-23 21:17 ` Zeng, Oak
2024-04-24 2:31 ` Matthew Brost
2024-04-24 13:57 ` Jason Gunthorpe
2024-04-24 16:35 ` Matthew Brost
2024-04-24 16:44 ` Jason Gunthorpe
2024-04-24 16:56 ` Matthew Brost
2024-04-24 17:48 ` Jason Gunthorpe
2024-04-24 13:48 ` Jason Gunthorpe
2024-04-24 23:59 ` Zeng, Oak
2024-04-25 1:05 ` Jason Gunthorpe
2024-04-26 9:55 ` Thomas Hellström
2024-04-26 12:00 ` Jason Gunthorpe
2024-04-26 14:49 ` Thomas Hellström
2024-04-26 16:35 ` Jason Gunthorpe
2024-04-29 8:25 ` Thomas Hellström
2024-04-30 17:30 ` Jason Gunthorpe
2024-04-30 18:57 ` Daniel Vetter
2024-05-01 0:09 ` Jason Gunthorpe
2024-05-02 8:04 ` Daniel Vetter
2024-05-02 9:11 ` Thomas Hellström
2024-05-02 12:46 ` Jason Gunthorpe
2024-05-02 15:01 ` Thomas Hellström
2024-05-02 19:25 ` Zeng, Oak
2024-05-03 13:37 ` Jason Gunthorpe
2024-05-03 14:43 ` Zeng, Oak
2024-05-03 16:28 ` Jason Gunthorpe
2024-05-03 20:29 ` Zeng, Oak
2024-05-04 1:03 ` Dave Airlie
2024-05-06 13:04 ` Daniel Vetter
2024-05-06 23:50 ` Matthew Brost
2024-05-07 11:56 ` Jason Gunthorpe
2024-05-06 13:33 ` Jason Gunthorpe
2024-04-09 17:33 ` Matthew Brost
2024-01-17 22:12 ` [PATCH 07/23] drm/xe/svm: Add helper for binding hmm range to gpu Oak Zeng
2024-01-17 22:12 ` [PATCH 08/23] drm/xe/svm: Add helper to invalidate svm range from GPU Oak Zeng
2024-01-17 22:12 ` [PATCH 09/23] drm/xe/svm: Remap and provide memmap backing for GPU vram Oak Zeng
2024-01-17 22:12 ` [PATCH 10/23] drm/xe/svm: Introduce svm migration function Oak Zeng
2024-01-17 22:12 ` [PATCH 11/23] drm/xe/svm: implement functions to allocate and free device memory Oak Zeng
2024-01-17 22:12 ` [PATCH 12/23] drm/xe/svm: Trace buddy block allocation and free Oak Zeng
2024-01-17 22:12 ` [PATCH 13/23] drm/xe/svm: Handle CPU page fault Oak Zeng
2024-01-17 22:12 ` [PATCH 14/23] drm/xe/svm: trace svm range migration Oak Zeng
2024-01-17 22:12 ` [PATCH 15/23] drm/xe/svm: Implement functions to register and unregister mmu notifier Oak Zeng
2024-01-17 22:12 ` [PATCH 16/23] drm/xe/svm: Implement the mmu notifier range invalidate callback Oak Zeng
2024-01-17 22:12 ` [PATCH 17/23] drm/xe/svm: clean up svm range during process exit Oak Zeng
2024-01-17 22:12 ` [PATCH 18/23] drm/xe/svm: Move a few structures to xe_gt.h Oak Zeng
2024-01-17 22:12 ` [PATCH 19/23] drm/xe/svm: migrate svm range to vram Oak Zeng
2024-01-17 22:12 ` [PATCH 20/23] drm/xe/svm: Populate svm range Oak Zeng
2024-01-17 22:12 ` [PATCH 21/23] drm/xe/svm: GPU page fault support Oak Zeng
2024-01-23 2:06 ` Welty, Brian
2024-01-23 3:09 ` Zeng, Oak
2024-01-23 3:21 ` Making drm_gpuvm work across gpu devices Zeng, Oak
2024-01-23 11:13 ` Christian König
2024-01-23 19:37 ` Zeng, Oak
2024-01-23 20:17 ` Felix Kuehling
2024-01-25 1:39 ` Zeng, Oak
2024-01-23 23:56 ` Danilo Krummrich
2024-01-24 3:57 ` Zeng, Oak
2024-01-24 4:14 ` Zeng, Oak
2024-01-24 6:48 ` Christian König
2024-01-25 22:13 ` Danilo Krummrich
2024-01-24 8:33 ` Christian König
2024-01-25 1:17 ` Zeng, Oak
2024-01-25 1:25 ` David Airlie
2024-01-25 5:25 ` Zeng, Oak
2024-01-26 10:09 ` Christian König
2024-01-26 20:13 ` Zeng, Oak
2024-01-29 10:10 ` Christian König
2024-01-29 20:09 ` Zeng, Oak
2024-01-25 11:00 ` 回复：Making " 周春明(日月)
2024-01-25 17:00 ` Zeng, Oak
2024-01-25 17:15 ` Making " Felix Kuehling
2024-01-25 18:37 ` Zeng, Oak
2024-01-26 13:23 ` Christian König [this message]
2024-01-25 16:42 ` Zeng, Oak
2024-01-25 18:32 ` Daniel Vetter
2024-01-25 21:02 ` Zeng, Oak
2024-01-26 8:21 ` Thomas Hellström
2024-01-26 12:52 ` Christian König
2024-01-27 2:21 ` Zeng, Oak
2024-01-29 10:19 ` Christian König
2024-01-30 0:21 ` Zeng, Oak
2024-01-30 8:39 ` Christian König
2024-01-30 22:29 ` Zeng, Oak
2024-01-30 23:12 ` David Airlie
2024-01-31 9:15 ` Daniel Vetter
2024-01-31 20:17 ` Zeng, Oak
2024-01-31 20:59 ` Zeng, Oak
2024-02-01 8:52 ` Christian König
2024-02-29 18:22 ` Zeng, Oak
2024-03-08 4:43 ` Zeng, Oak
2024-03-08 10:07 ` Christian König
2024-01-30 8:43 ` Thomas Hellström
2024-01-29 15:03 ` Felix Kuehling
2024-01-29 15:33 ` Christian König
2024-01-29 16:24 ` Felix Kuehling
2024-01-29 16:28 ` Christian König
2024-01-29 17:52 ` Felix Kuehling
2024-01-29 19:03 ` Christian König
2024-01-29 20:24 ` Felix Kuehling
2024-02-23 20:12 ` Zeng, Oak
2024-02-27 6:54 ` Christian König
2024-02-27 15:58 ` Zeng, Oak
2024-02-28 19:51 ` Zeng, Oak
2024-02-29 9:41 ` Christian König
2024-02-29 16:05 ` Zeng, Oak
2024-02-29 17:12 ` Thomas Hellström
2024-03-01 7:01 ` Christian König
2024-01-17 22:12 ` [PATCH 22/23] drm/xe/svm: Add DRM_XE_SVM kernel config entry Oak Zeng
2024-01-17 22:12 ` [PATCH 23/23] drm/xe/svm: Add svm memory hints interface Oak Zeng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5a0a26b2-3116-4395-b0fc-84b2f248ac15@amd.com \
--to=christian.koenig@amd.com \
--cc=Thomas.Hellstrom@linux.intel.com \
--cc=airlied@redhat.com \
--cc=ankur.n.shah@intel.com \
--cc=brian.welty@intel.com \
--cc=dakr@redhat.com \
--cc=daniel@ffwll.ch \
--cc=dri-devel@lists.freedesktop.org \
--cc=felix.kuehling@amd.com \
--cc=himal.prasad.ghimiray@intel.com \
--cc=intel-xe@lists.freedesktop.org \
--cc=krishnaiah.bommu@intel.com \
--cc=matthew.brost@intel.com \
--cc=michal.winiarski@intel.com \
--cc=niranjana.vishwanathapura@intel.com \
--cc=oak.zeng@intel.com \
--cc=saurabhg.gupta@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).