From: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: Jason Gunthorpe <jgg@nvidia.com>,
"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
Linuxarm <linuxarm@huawei.com>,
Zhangfei Gao <zhangfei.gao@linaro.org>,
Michael Shavit <mshavit@google.com>,
Eric Auger <eric.auger@redhat.com>,
Moritz Fischer <mdf@kernel.org>,
"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>
Subject: RE: Query on ARM SMMUv3 nested support
Date: Tue, 2 Apr 2024 07:25:56 +0000 [thread overview]
Message-ID: <02f3fbc5145d4449b3313eb802ecfa2c@huawei.com> (raw)
In-Reply-To: <ZgZqMymQY4pL77iw@Asurada-Nvidia>
> -----Original Message-----
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Friday, March 29, 2024 7:14 AM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Jason Gunthorpe <jgg@nvidia.com>; iommu@lists.linux.dev; Linuxarm
> <linuxarm@huawei.com>; Zhangfei Gao <zhangfei.gao@linaro.org>; Michael
> Shavit <mshavit@google.com>; Eric Auger <eric.auger@redhat.com>; Moritz
> Fischer <mdf@kernel.org>; baolu.lu@linux.intel.com
> Subject: Re: Query on ARM SMMUv3 nested support
>
> Hi Shameer,
>
> Sorry for the late reply. Having been occupied by multiple tasks
> lately.
>
> On Fri, Mar 22, 2024 at 03:04:42PM +0000, Shameerali Kolothum Thodi
> wrote:
>
> > Thanks for the reference to above CMDQ-V. Looks interesting.
> > I think we can have a go with retry logic first to allocate a new hwpt
> > if the attach fails. For now, I have a temp fix where I allocate a new
> > hwpt every time.
> >
> > I also noticed another problem with the commit below,
> > 6691a2f("hw/arm/smmu-common: Use sysmem for get_address _space
> until !!s2_hwpti")
> >
> > This actually breaks a virtio-pci dev assignment when that is the only
> > assigned device to the Guest without any pass-through dev in nested
> > scenario.
> >
> > I have a temp fix for that as well here,
> > (I know it is ugly!). I will revisit that again. Please let me know if you have
> > any ideas.
>
> I am not quite getting the case here. Is that virtio-pci device
> behind a nested-SMMU while there is no S2 hwpt? In that case, why
> does this virtio-pci device require a "nested-smmuv3"?
Hi Nicolin,
Let me try to explain with a use case,
Consider a guest boot with below options,
./qemu-system-aarch64 virt,gic-version=3,iommu=nested-smmuv3,iommufd=iommufd0 \
-enable-kvm -cpu host -m 4G -smp cpus=8,maxcpus=8 \
-object iommufd,id=iommufd0 \
-bios QEMU_EFI.fd \
-kernel Image \
-device virtio-blk-device,drive=fs \
-drive if=none,file=ubuntu.img,id=fs \
-device ioh3420,id=rp1 \
-device virtio-9p-pci,fsdev=p9fs,mount_tag=p9 \
-fsdev local,id=p9fs,path=p9root,security_model=mapped \
--append "rdinit=init console=ttyAMA0 root=/dev/vda rw earlycon=pl011,0x9000000" \
-net none \
-nographic
Here we have specified nested-smmuv3 and have a PCIe root port(ioh3420) and
a virtio-pci dev(for virtfs). Once this VM is up and running, both the ioh3420
and virtio-9p-pci will not work as expected as we return sysmem address space
for them.
And then later when you try to hotplug a vfio dev to the PCIe root port,
device_add vfio-pci,host=0000:7d:02.1,bus=rp1,id=net1,iommufd=iommufd0
it won't work either because the address space for ioh3420 is wrong.
Taking another look at this, I think we can replace my earlier attempt to fix
this by detecting if the dev is vfio or not and add that into the check below,
--- a/hw/arm/smmu-common.c
+++ b/hw/arm/smmu-common.c
@@ -619,8 +619,15 @@ static AddressSpace *smmu_find_add_as(PCIBus
*bus, void *opaque, int devfn)
SMMUState *s = opaque;
SMMUPciBus *sbus = smmu_get_sbus(s, bus);
SMMUDevice *sdev = smmu_get_sdev(s, sbus, bus, devfn);
+ bool is_vfio = false;
+ PCIDevice *pdev;
- if (s->nested && !s->s2_hwpt) {
+ pdev = pci_find_device(bus, pci_bus_num(bus), devfn);
+ if (object_dynamic_cast(OBJECT(pdev), "vfio-pci")) {
+ is_vfio = true;
+ }
+
+ if (is_vfio && s->nested && !s->s2_hwpt) {
return &sdev->as_sysmem;
} else {
return &sdev->as;
Please let me know your thoughts on this.
Thanks,
Shameer
next prev parent reply other threads:[~2024-04-02 7:26 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-13 10:13 Query on ARM SMMUv3 nested support Shameerali Kolothum Thodi
2024-03-13 23:50 ` Nicolin Chen
2024-03-18 16:22 ` Jason Gunthorpe
2024-03-22 15:04 ` Shameerali Kolothum Thodi
2024-03-29 7:13 ` Nicolin Chen
2024-04-02 7:25 ` Shameerali Kolothum Thodi [this message]
2024-04-02 11:28 ` Jason Gunthorpe
2024-04-09 6:12 ` Nicolin Chen
2024-04-09 19:47 ` Nicolin Chen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=02f3fbc5145d4449b3313eb802ecfa2c@huawei.com \
--to=shameerali.kolothum.thodi@huawei.com \
--cc=baolu.lu@linux.intel.com \
--cc=eric.auger@redhat.com \
--cc=iommu@lists.linux.dev \
--cc=jgg@nvidia.com \
--cc=linuxarm@huawei.com \
--cc=mdf@kernel.org \
--cc=mshavit@google.com \
--cc=nicolinc@nvidia.com \
--cc=zhangfei.gao@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.