From: "Tian, Kevin" <kevin.tian@intel.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: Jason Gunthorpe <jgg@nvidia.com>,
"will@kernel.org" <will@kernel.org>,
"robin.murphy@arm.com" <robin.murphy@arm.com>,
"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
"joro@8bytes.org" <joro@8bytes.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
"linux-arm-kernel@lists.infradead.org"
<linux-arm-kernel@lists.infradead.org>,
"linux-tegra@vger.kernel.org" <linux-tegra@vger.kernel.org>,
"Liu, Yi L" <yi.l.liu@intel.com>,
"eric.auger@redhat.com" <eric.auger@redhat.com>,
"vasant.hegde@amd.com" <vasant.hegde@amd.com>,
"jon.grimm@amd.com" <jon.grimm@amd.com>,
"santosh.shukla@amd.com" <santosh.shukla@amd.com>,
"Dhaval.Giani@amd.com" <Dhaval.Giani@amd.com>,
"shameerali.kolothum.thodi@huawei.com"
<shameerali.kolothum.thodi@huawei.com>
Subject: RE: [PATCH RFCv1 08/14] iommufd: Add IOMMU_VIOMMU_SET_DEV_ID ioctl
Date: Thu, 30 May 2024 03:05:05 +0000 [thread overview]
Message-ID: <BN9PR11MB5276DC3F8F085FE0E699D9448CF32@BN9PR11MB5276.namprd11.prod.outlook.com> (raw)
In-Reply-To: <ZlfPPwsnY6P4SoGF@Asurada-Nvidia>
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Thursday, May 30, 2024 8:59 AM
>
> On Thu, May 30, 2024 at 12:28:43AM +0000, Tian, Kevin wrote:
> > > From: Nicolin Chen <nicolinc@nvidia.com>
> > > Sent: Wednesday, May 29, 2024 11:21 AM
> > > On Wed, May 29, 2024 at 02:58:11AM +0000, Tian, Kevin wrote:
> > > > My question is why that option is chosen instead of going with 1:1
> > > > mapping between vSMMU and viommu i.e. letting the kernel to
> > > > figure out which pSMMU should be sent an invalidation cmd to, as
> > > > how VT-d is virtualized.
> > > >
> > > > I want to know whether doing so is simply to be compatible with
> > > > what VCMDQ requires, or due to another untold reason.
> > >
> > > Because we use viommu as a VMID holder for SMMU. So a pSMMU must
> > > have its own viommu to store its VMID for a shared s2_hwpt:
> > > |-- viommu0 (VMIDx) --|-- pSMMU0 --|
> > > vSMMU--|-- viommu1 (VMIDy) --|-- pSMMU1 --|--s2_hwpt
> > > |-- viommu2 (VMIDz) --|-- pSMMU2 --|
> > >
> >
> > there are other options, e.g. you can have one viommu holding multiple
> > VMIDs each associating to a pSMMU.
>
> Well, possibly. But everything previously in a viommu would have
> to be a list (for number of VMIDs) in the kernel level: not only
> a VMID list, but also a 2D virtual ID lists, something like:
>
> struct xarray vdev_ids[num_of_vmid]; // per-IOMMU vID to pID lookup
ah does it mean that dev ID space on ARM is local to SMMU?
It's not the case on x86 platforms where the RID is platform-wise.
>
> And a driver in this case would be difficult to get a complete
> concept of a viommu object since it's core object and shared by
> all kernel-level IOMMU instances. If a driver wants to extend a
> viommu object for some additional feature, e.g. VINTF in this
> series, it would likely have to create another per-driver object
> and again another list of this kind of objects in struct viommu.
>
> At the end of day, we have to duplicate it one way or another for
> the amount of physical IOMMUs. And it seems to cleaner by doing
> it with multiple viommu objects.
>
> > so it's really an implementation choice that you want a same viommu
> > topology scheme w/ or w/o VCMDQ.
> >
> > we all know there are some constraints of copying the physical topology,
> > e.g. hotplugging a device or migration. for VCMDQ it's clearly an
> > acceptable tradeoff for performance. w/o VCMDQ I'm not sure whether
> > you want to keep more flexibility for the user. 😊
>
> Oh. With regular nested SMMU, there is only one virtual SMMU in
> the guest VM. No need of copying physical topology. Just the VMM
> needs to allocate three viommus to add them to a list of its own.
>
Okay I missed this point. Then the number of viommus is really about
the contract between the vmm and the kernel, which is invisible to
the guest or the admin who launches the Qemu.
but wait, isn't there another problem - does the VMM have the
permission to enumerate the topology of physical SMMUs? Or probably
the VMM only needs to know the number of relevant SMMUs for
assigned devices and such info can be indirectly composed by extending GET_HW_INFO...
next prev parent reply other threads:[~2024-05-30 3:05 UTC|newest]
Thread overview: 118+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-13 3:46 [PATCH RFCv1 00/14] Add Tegra241 (Grace) CMDQV Support (part 2/2) Nicolin Chen
2024-04-13 3:46 ` [PATCH RFCv1 01/14] iommufd: Move iommufd_object to public iommufd header Nicolin Chen
2024-05-12 13:21 ` Jason Gunthorpe
2024-05-12 22:40 ` Nicolin Chen
2024-05-13 22:30 ` Jason Gunthorpe
2024-04-13 3:46 ` [PATCH RFCv1 02/14] iommufd: Swap _iommufd_object_alloc and __iommufd_object_alloc Nicolin Chen
2024-05-12 13:26 ` Jason Gunthorpe
2024-05-13 2:29 ` Nicolin Chen
2024-05-13 3:44 ` Nicolin Chen
2024-05-13 22:30 ` Jason Gunthorpe
2024-04-13 3:47 ` [PATCH RFCv1 03/14] iommufd: Prepare for viommu structures and functions Nicolin Chen
2024-05-12 13:42 ` Jason Gunthorpe
2024-05-13 2:35 ` Nicolin Chen
2024-04-13 3:47 ` [PATCH RFCv1 04/14] iommufd: Add struct iommufd_viommu and iommufd_viommu_ops Nicolin Chen
2024-05-12 14:03 ` Jason Gunthorpe
2024-05-13 3:34 ` Nicolin Chen
2024-05-14 15:55 ` Jason Gunthorpe
2024-05-22 8:58 ` Tian, Kevin
2024-05-22 9:57 ` Baolu Lu
2024-05-22 13:39 ` Jason Gunthorpe
2024-05-23 1:43 ` Tian, Kevin
2024-05-23 4:01 ` Nicolin Chen
2024-05-23 5:40 ` Tian, Kevin
2024-05-23 12:58 ` Jason Gunthorpe
2024-05-24 2:16 ` Tian, Kevin
2024-05-24 13:03 ` Jason Gunthorpe
2024-05-24 2:36 ` Tian, Kevin
2024-04-13 3:47 ` [PATCH RFCv1 05/14] iommufd: Add IOMMUFD_OBJ_VIOMMU and IOMMUFD_CMD_VIOMMU_ALLOC Nicolin Chen
2024-05-12 14:27 ` Jason Gunthorpe
2024-05-13 4:33 ` Nicolin Chen
2024-05-14 15:38 ` Jason Gunthorpe
2024-05-15 1:20 ` Nicolin Chen
2024-05-21 18:05 ` Jason Gunthorpe
2024-05-22 0:13 ` Nicolin Chen
2024-05-22 16:46 ` Jason Gunthorpe
2024-04-13 3:47 ` [PATCH RFCv1 06/14] iommufd/selftest: Add IOMMU_VIOMMU_ALLOC test coverage Nicolin Chen
2024-04-13 3:47 ` [PATCH RFCv1 07/14] iommufd: Add viommu set/unset_dev_id ops Nicolin Chen
2024-05-12 14:46 ` Jason Gunthorpe
2024-05-13 4:39 ` Nicolin Chen
2024-05-14 15:53 ` Jason Gunthorpe
2024-05-15 1:59 ` Nicolin Chen
2024-05-21 18:24 ` Jason Gunthorpe
2024-05-21 22:27 ` Nicolin Chen
2024-05-22 13:59 ` Jason Gunthorpe
2024-05-23 6:19 ` Tian, Kevin
2024-05-23 15:01 ` Jason Gunthorpe
2024-05-24 2:21 ` Tian, Kevin
2024-05-24 3:26 ` Nicolin Chen
2024-05-24 5:24 ` Tian, Kevin
2024-05-24 5:57 ` Nicolin Chen
2024-05-24 7:21 ` Tian, Kevin
2024-05-24 13:12 ` Jason Gunthorpe
2024-05-24 13:05 ` Jason Gunthorpe
2024-05-23 5:44 ` Tian, Kevin
2024-05-23 6:09 ` Nicolin Chen
2024-05-23 6:22 ` Tian, Kevin
2024-05-23 13:33 ` Jason Gunthorpe
2024-05-12 14:51 ` Jason Gunthorpe
2024-04-13 3:47 ` [PATCH RFCv1 08/14] iommufd: Add IOMMU_VIOMMU_SET_DEV_ID ioctl Nicolin Chen
2024-05-12 14:58 ` Jason Gunthorpe
2024-05-13 5:24 ` Nicolin Chen
2024-05-17 5:14 ` Nicolin Chen
2024-05-21 18:30 ` Jason Gunthorpe
2024-05-22 2:15 ` Nicolin Chen
2024-05-23 6:42 ` Tian, Kevin
2024-05-24 5:40 ` Nicolin Chen
2024-05-24 7:13 ` Tian, Kevin
2024-05-24 13:19 ` Jason Gunthorpe
2024-05-27 1:08 ` Tian, Kevin
2024-05-28 20:22 ` Nicolin Chen
2024-05-28 20:33 ` Nicolin Chen
2024-05-29 2:58 ` Tian, Kevin
2024-05-29 3:20 ` Nicolin Chen
2024-05-30 0:28 ` Tian, Kevin
2024-05-30 0:58 ` Nicolin Chen
2024-05-30 3:05 ` Tian, Kevin [this message]
2024-05-30 4:26 ` Nicolin Chen
2024-06-01 21:45 ` Jason Gunthorpe
2024-06-03 3:25 ` Nicolin Chen
2024-06-06 18:24 ` Jason Gunthorpe
2024-06-06 18:44 ` Nicolin Chen
2024-06-07 0:27 ` Jason Gunthorpe
2024-06-07 0:36 ` Tian, Kevin
2024-06-07 14:49 ` Jason Gunthorpe
2024-06-07 21:19 ` Nicolin Chen
2024-06-10 12:04 ` Jason Gunthorpe
2024-06-10 20:01 ` Nicolin Chen
2024-06-10 22:01 ` Jason Gunthorpe
2024-06-10 23:04 ` Nicolin Chen
2024-06-11 0:28 ` Jason Gunthorpe
2024-06-11 0:44 ` Nicolin Chen
2024-06-11 12:17 ` Jason Gunthorpe
2024-06-11 19:11 ` Nicolin Chen
2024-05-28 20:30 ` Nicolin Chen
2024-05-24 13:20 ` Jason Gunthorpe
2024-04-13 3:47 ` [PATCH RFCv1 09/14] iommufd/selftest: Add IOMMU_VIOMMU_SET_DEV_ID test coverage Nicolin Chen
2024-04-13 3:47 ` [PATCH RFCv1 10/14] iommufd/selftest: Add IOMMU_TEST_OP_MV_CHECK_DEV_ID Nicolin Chen
2024-04-13 3:47 ` [PATCH RFCv1 11/14] iommufd: Add struct iommufd_vqueue and its related viommu ops Nicolin Chen
2024-04-13 3:47 ` [PATCH RFCv1 12/14] iommufd: Add IOMMUFD_OBJ_VQUEUE and IOMMUFD_CMD_VQUEUE_ALLOC Nicolin Chen
2024-05-12 15:02 ` Jason Gunthorpe
2024-05-13 4:41 ` Nicolin Chen
2024-05-13 22:36 ` Jason Gunthorpe
2024-05-23 6:57 ` Tian, Kevin
2024-05-24 4:42 ` Nicolin Chen
2024-05-24 5:26 ` Tian, Kevin
2024-05-24 6:03 ` Nicolin Chen
2024-05-23 7:05 ` Tian, Kevin
2024-04-13 3:47 ` [PATCH RFCv1 13/14] iommufd: Add mmap infrastructure Nicolin Chen
2024-05-12 15:19 ` Jason Gunthorpe
2024-05-13 4:43 ` Nicolin Chen
2024-04-13 3:47 ` [PATCH RFCv1 14/14] iommu/tegra241-cmdqv: Add user-space use support Nicolin Chen
2024-05-22 8:40 ` [PATCH RFCv1 00/14] Add Tegra241 (Grace) CMDQV Support (part 2/2) Tian, Kevin
2024-05-22 16:48 ` Jason Gunthorpe
2024-05-22 19:47 ` Nicolin Chen
2024-05-22 23:28 ` Jason Gunthorpe
2024-05-22 23:43 ` Tian, Kevin
2024-05-23 3:09 ` Nicolin Chen
2024-05-23 12:48 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=BN9PR11MB5276DC3F8F085FE0E699D9448CF32@BN9PR11MB5276.namprd11.prod.outlook.com \
--to=kevin.tian@intel.com \
--cc=Dhaval.Giani@amd.com \
--cc=eric.auger@redhat.com \
--cc=iommu@lists.linux.dev \
--cc=jgg@nvidia.com \
--cc=jon.grimm@amd.com \
--cc=joro@8bytes.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-tegra@vger.kernel.org \
--cc=nicolinc@nvidia.com \
--cc=robin.murphy@arm.com \
--cc=santosh.shukla@amd.com \
--cc=shameerali.kolothum.thodi@huawei.com \
--cc=suravee.suthikulpanit@amd.com \
--cc=vasant.hegde@amd.com \
--cc=will@kernel.org \
--cc=yi.l.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).