From: Nicolin Chen <nicolinc@nvidia.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: <will@kernel.org>, <robin.murphy@arm.com>, <kevin.tian@intel.com>,
<suravee.suthikulpanit@amd.com>, <joro@8bytes.org>,
<linux-kernel@vger.kernel.org>, <iommu@lists.linux.dev>,
<linux-arm-kernel@lists.infradead.org>,
<linux-tegra@vger.kernel.org>, <yi.l.liu@intel.com>,
<eric.auger@redhat.com>, <vasant.hegde@amd.com>,
<jon.grimm@amd.com>, <santosh.shukla@amd.com>,
<Dhaval.Giani@amd.com>, <shameerali.kolothum.thodi@huawei.com>
Subject: Re: [PATCH RFCv1 07/14] iommufd: Add viommu set/unset_dev_id ops
Date: Tue, 21 May 2024 15:27:02 -0700 [thread overview]
Message-ID: <Zk0ftlf3f4gBaNgy@nvidia.com> (raw)
In-Reply-To: <20240521182448.GN20229@nvidia.com>
On Tue, May 21, 2024 at 03:24:48PM -0300, Jason Gunthorpe wrote:
> On Tue, May 14, 2024 at 06:59:07PM -0700, Nicolin Chen wrote:
> > So, you want a proxy S1 domain for a device to attach, in case
> > of a stage-2 only setup, because an S2 domain will no longer has
> > a VMID, since it's shared among viommus. In the SMMU driver case,
> > an arm_smmu_domain won't have an smmu pointer, so a device can't
> > attach to an S2 domain but always an nested S1 domain, right?
>
> That seems like a simple solution to the VMID lifetime, but it means
> the kernel has to decode more types of vSTE.
Yea. For vSTE=abort, likely we need a nested block domain too?
> > > Functionally we could use that global nesting domain
> > > to deliver the DEV_INVALIDATE too.
> >
> > If my narrative above is correct, the device is actually still
> > attached to S2 domain via a proxy nested S1 domain. What cache
> > do we need to invalidate except S2 mappings in this case?
>
> qemu needs a reliable place to send the invalidation commands to (ie
> what ID to provide).
>
> If we add IOMMU_VIOMMU_INVALIDATE then the ID is the viommu id.
>
> If we enhance IOMMU_HWPT_INVALIDATE then the ID is the identity
> nesting domain.
>
> Either case leads to the viommu object in the kernel.
Ooohh! I didn't connect the dots this far. Yes. This turns the
IOMMU_HWPT_INVALIDATE back to the very first version supporting
device cache flush. Though using IOMMU_HWPT_INVALIDATE feels a
bit rule breaking now since it assumes the nested HWPT keeps a
vdev_id lookup table somewhere in its associates...
> I don't know if there is merit one way or the other. A more specific
> API surface is nice, but the two APIs are completely duplicating.
>
> So maybe:
>
> #define IOMMU_VIOMMU_INVALIDATE IOMMU_HWPT_INVALIDATE
>
> As documentation and have the kernel just detect based on the type of
> the passed ID?
Yea, the only difference is viommu_id v.s. hwpt_id that we can
document.
Then in this case, we have two mostly identical uAPIs for the
SMMU driver to use. Should we implement both?
> > > > So again, yes, it makes sense to me that we move viommu and the
> > > > set_dev_id to the nested series, and then drop DEV_INVALIDATE.
> > >
> > > I would like to do this bit by bit. viommu is a big series on its own.
> > >
> > > DEV_INVALIDATE is fine, it just can't do ATS invalidation.
> >
> > I am not very sure about AMD.
>
> AMD will need the same vRID -> pRID stuff and we want that to run on
> the VIOMMU
>
> > Same question: any other case can we use the DEV_INVALIDATE for?
>
> DEV_INVALIDATE was interesting before the viommu idea because
> userspace could process each invalidation command and when it reaches
> ATS it would invoke the correct DEV_INVALIDATE.
Agreed. That helped a lot in VMM dispatching the invalidation
requests.
> With viommu we expect ATS supporting drivers to support viommu and
> then to do vRID -> pRID in the other invalidation paths. In this case
> I don't see a reason to do DEV_INVALIDATE right now.
Yea. I guessed so.
> > > We can add ATS invalidation after either as an enhancement as part of
> > > adding the VIOMMU either as DEV_INVALIDATE or VIOMMU_INVALIDATE (or
> > > both)
> >
> > Yea, maybe step by step like this:
> >
> > Part-1 VIOMMU_ALLOC and VIOMMU_ATTACH
> > Part-2 VIOMMU_SET/UNSET_VDEV_ID
> > Part-3 VIOMMU_INVALIDATE
> > Part-4 VQUEUE_ALLOC
> > ...
>
> So we have this stuff still open:
> - Identity STE with PASID (part 2b)
> - IOMMU_GET_HW_INFO (part 3)
> - IOMMU_HWPT_ALLOC_NEST_PARENT (part 3)
> - IOMMU_HWPT_DATA_ARM_SMMUV3 (part 3)
> - IOMMU_HWPT_INVALIDATE_DATA_ARM_SMMUV3
> - VIOMMU_ALLOC, VIOMMU_ATTACH
> - VIOMMU_INVALIDATE
> - VIOMMU_SET/UNSET_VDEV_ID
> - VQUEUE_ALLOC / vCMDQ
>
> I feel like IOMMU_HWPT_INVALIDATE_DATA_ARM_SMMUV3 is a reasonable fit
> to part 3. Then part 4 would be VIOMMU_ALLOC -> VIOMMU_SET/UNSET_VDEV_ID
> which brings ATS support the API.
There is some conflict at passing in viommu_id/viommu v.s. parent
hwpt_id/domain for a nested domain allocation. Do you think that
should be addressed later in VIOMMU series v.s. part3?
More specifically, I have two drafts in my viommu series:
87a659e65229 WAR: iommufd: Allow pt_it to carry viommu_id
7c5fd8f50bc9 WAR pass in viommu pointer to domain_alloc_user op
I know that these two only make sense with VIOMMU_ALOC. Yet, will
there be a problem, if we establish nested domain allocation with
parent domain/hwpt by part3, in the uAPI, and then change later?
Will we end up with supporting two for backward compatibility?
> vCMDQ hypervisor support would sit on top of that with just VQUEUE?
Yea.
Thanks
Nicolin
next prev parent reply other threads:[~2024-05-21 22:27 UTC|newest]
Thread overview: 118+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-13 3:46 [PATCH RFCv1 00/14] Add Tegra241 (Grace) CMDQV Support (part 2/2) Nicolin Chen
2024-04-13 3:46 ` [PATCH RFCv1 01/14] iommufd: Move iommufd_object to public iommufd header Nicolin Chen
2024-05-12 13:21 ` Jason Gunthorpe
2024-05-12 22:40 ` Nicolin Chen
2024-05-13 22:30 ` Jason Gunthorpe
2024-04-13 3:46 ` [PATCH RFCv1 02/14] iommufd: Swap _iommufd_object_alloc and __iommufd_object_alloc Nicolin Chen
2024-05-12 13:26 ` Jason Gunthorpe
2024-05-13 2:29 ` Nicolin Chen
2024-05-13 3:44 ` Nicolin Chen
2024-05-13 22:30 ` Jason Gunthorpe
2024-04-13 3:47 ` [PATCH RFCv1 03/14] iommufd: Prepare for viommu structures and functions Nicolin Chen
2024-05-12 13:42 ` Jason Gunthorpe
2024-05-13 2:35 ` Nicolin Chen
2024-04-13 3:47 ` [PATCH RFCv1 04/14] iommufd: Add struct iommufd_viommu and iommufd_viommu_ops Nicolin Chen
2024-05-12 14:03 ` Jason Gunthorpe
2024-05-13 3:34 ` Nicolin Chen
2024-05-14 15:55 ` Jason Gunthorpe
2024-05-22 8:58 ` Tian, Kevin
2024-05-22 9:57 ` Baolu Lu
2024-05-22 13:39 ` Jason Gunthorpe
2024-05-23 1:43 ` Tian, Kevin
2024-05-23 4:01 ` Nicolin Chen
2024-05-23 5:40 ` Tian, Kevin
2024-05-23 12:58 ` Jason Gunthorpe
2024-05-24 2:16 ` Tian, Kevin
2024-05-24 13:03 ` Jason Gunthorpe
2024-05-24 2:36 ` Tian, Kevin
2024-04-13 3:47 ` [PATCH RFCv1 05/14] iommufd: Add IOMMUFD_OBJ_VIOMMU and IOMMUFD_CMD_VIOMMU_ALLOC Nicolin Chen
2024-05-12 14:27 ` Jason Gunthorpe
2024-05-13 4:33 ` Nicolin Chen
2024-05-14 15:38 ` Jason Gunthorpe
2024-05-15 1:20 ` Nicolin Chen
2024-05-21 18:05 ` Jason Gunthorpe
2024-05-22 0:13 ` Nicolin Chen
2024-05-22 16:46 ` Jason Gunthorpe
2024-04-13 3:47 ` [PATCH RFCv1 06/14] iommufd/selftest: Add IOMMU_VIOMMU_ALLOC test coverage Nicolin Chen
2024-04-13 3:47 ` [PATCH RFCv1 07/14] iommufd: Add viommu set/unset_dev_id ops Nicolin Chen
2024-05-12 14:46 ` Jason Gunthorpe
2024-05-13 4:39 ` Nicolin Chen
2024-05-14 15:53 ` Jason Gunthorpe
2024-05-15 1:59 ` Nicolin Chen
2024-05-21 18:24 ` Jason Gunthorpe
2024-05-21 22:27 ` Nicolin Chen [this message]
2024-05-22 13:59 ` Jason Gunthorpe
2024-05-23 6:19 ` Tian, Kevin
2024-05-23 15:01 ` Jason Gunthorpe
2024-05-24 2:21 ` Tian, Kevin
2024-05-24 3:26 ` Nicolin Chen
2024-05-24 5:24 ` Tian, Kevin
2024-05-24 5:57 ` Nicolin Chen
2024-05-24 7:21 ` Tian, Kevin
2024-05-24 13:12 ` Jason Gunthorpe
2024-05-24 13:05 ` Jason Gunthorpe
2024-05-23 5:44 ` Tian, Kevin
2024-05-23 6:09 ` Nicolin Chen
2024-05-23 6:22 ` Tian, Kevin
2024-05-23 13:33 ` Jason Gunthorpe
2024-05-12 14:51 ` Jason Gunthorpe
2024-04-13 3:47 ` [PATCH RFCv1 08/14] iommufd: Add IOMMU_VIOMMU_SET_DEV_ID ioctl Nicolin Chen
2024-05-12 14:58 ` Jason Gunthorpe
2024-05-13 5:24 ` Nicolin Chen
2024-05-17 5:14 ` Nicolin Chen
2024-05-21 18:30 ` Jason Gunthorpe
2024-05-22 2:15 ` Nicolin Chen
2024-05-23 6:42 ` Tian, Kevin
2024-05-24 5:40 ` Nicolin Chen
2024-05-24 7:13 ` Tian, Kevin
2024-05-24 13:19 ` Jason Gunthorpe
2024-05-27 1:08 ` Tian, Kevin
2024-05-28 20:22 ` Nicolin Chen
2024-05-28 20:33 ` Nicolin Chen
2024-05-29 2:58 ` Tian, Kevin
2024-05-29 3:20 ` Nicolin Chen
2024-05-30 0:28 ` Tian, Kevin
2024-05-30 0:58 ` Nicolin Chen
2024-05-30 3:05 ` Tian, Kevin
2024-05-30 4:26 ` Nicolin Chen
2024-06-01 21:45 ` Jason Gunthorpe
2024-06-03 3:25 ` Nicolin Chen
2024-06-06 18:24 ` Jason Gunthorpe
2024-06-06 18:44 ` Nicolin Chen
2024-06-07 0:27 ` Jason Gunthorpe
2024-06-07 0:36 ` Tian, Kevin
2024-06-07 14:49 ` Jason Gunthorpe
2024-06-07 21:19 ` Nicolin Chen
2024-06-10 12:04 ` Jason Gunthorpe
2024-06-10 20:01 ` Nicolin Chen
2024-06-10 22:01 ` Jason Gunthorpe
2024-06-10 23:04 ` Nicolin Chen
2024-06-11 0:28 ` Jason Gunthorpe
2024-06-11 0:44 ` Nicolin Chen
2024-06-11 12:17 ` Jason Gunthorpe
2024-06-11 19:11 ` Nicolin Chen
2024-05-28 20:30 ` Nicolin Chen
2024-05-24 13:20 ` Jason Gunthorpe
2024-04-13 3:47 ` [PATCH RFCv1 09/14] iommufd/selftest: Add IOMMU_VIOMMU_SET_DEV_ID test coverage Nicolin Chen
2024-04-13 3:47 ` [PATCH RFCv1 10/14] iommufd/selftest: Add IOMMU_TEST_OP_MV_CHECK_DEV_ID Nicolin Chen
2024-04-13 3:47 ` [PATCH RFCv1 11/14] iommufd: Add struct iommufd_vqueue and its related viommu ops Nicolin Chen
2024-04-13 3:47 ` [PATCH RFCv1 12/14] iommufd: Add IOMMUFD_OBJ_VQUEUE and IOMMUFD_CMD_VQUEUE_ALLOC Nicolin Chen
2024-05-12 15:02 ` Jason Gunthorpe
2024-05-13 4:41 ` Nicolin Chen
2024-05-13 22:36 ` Jason Gunthorpe
2024-05-23 6:57 ` Tian, Kevin
2024-05-24 4:42 ` Nicolin Chen
2024-05-24 5:26 ` Tian, Kevin
2024-05-24 6:03 ` Nicolin Chen
2024-05-23 7:05 ` Tian, Kevin
2024-04-13 3:47 ` [PATCH RFCv1 13/14] iommufd: Add mmap infrastructure Nicolin Chen
2024-05-12 15:19 ` Jason Gunthorpe
2024-05-13 4:43 ` Nicolin Chen
2024-04-13 3:47 ` [PATCH RFCv1 14/14] iommu/tegra241-cmdqv: Add user-space use support Nicolin Chen
2024-05-22 8:40 ` [PATCH RFCv1 00/14] Add Tegra241 (Grace) CMDQV Support (part 2/2) Tian, Kevin
2024-05-22 16:48 ` Jason Gunthorpe
2024-05-22 19:47 ` Nicolin Chen
2024-05-22 23:28 ` Jason Gunthorpe
2024-05-22 23:43 ` Tian, Kevin
2024-05-23 3:09 ` Nicolin Chen
2024-05-23 12:48 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Zk0ftlf3f4gBaNgy@nvidia.com \
--to=nicolinc@nvidia.com \
--cc=Dhaval.Giani@amd.com \
--cc=eric.auger@redhat.com \
--cc=iommu@lists.linux.dev \
--cc=jgg@nvidia.com \
--cc=jon.grimm@amd.com \
--cc=joro@8bytes.org \
--cc=kevin.tian@intel.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-tegra@vger.kernel.org \
--cc=robin.murphy@arm.com \
--cc=santosh.shukla@amd.com \
--cc=shameerali.kolothum.thodi@huawei.com \
--cc=suravee.suthikulpanit@amd.com \
--cc=vasant.hegde@amd.com \
--cc=will@kernel.org \
--cc=yi.l.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).