From: Nicolin Chen <nicolinc@nvidia.com>
To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
Cc: Jason Gunthorpe <jgg@nvidia.com>,
"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
Linuxarm <linuxarm@huawei.com>,
Zhangfei Gao <zhangfei.gao@linaro.org>,
Michael Shavit <mshavit@google.com>,
Eric Auger <eric.auger@redhat.com>,
Moritz Fischer <mdf@kernel.org>
Subject: Re: Query on ARM SMMUv3 nested support
Date: Wed, 13 Mar 2024 16:50:58 -0700 [thread overview]
Message-ID: <ZfI74miZ1ss9+Kz0@Asurada-Nvidia> (raw)
In-Reply-To: <b8c1b2fb281246da94b35ceb5e27b538@huawei.com>
On Wed, Mar 13, 2024 at 10:13:58AM +0000, Shameerali Kolothum Thodi wrote:
> Hi Nicolin,
>
> Thanks for the latest repos with basic SMMUv3 nested support enabled[1].
> I did some basic sanity runs on a HiSilicon platform and they seems to work as
> expected. The only problem being we can't assign two devices to the VM if
> they are on different physical SMMUs.
>
> qemu-system-aarch64: -device
> vfio-pci,host=0000:75:00.1,iommufd=iommufd0: [iommufd=29] error attach
> 0000:75:00.1 (36) to id=4: Invalid argument
> qemu-system-aarch64: -device
> vfio-pci,host=0000:75:00.1,iommufd=iommufd0: Unable to attach dev to
> stage-2 HW pagetable: -1
> Segmentation fault (core dumped)
> ...
>
> I see that on the Qemu side we now allocate a single s2 hwpt and attach that into
> all the devices . But this will only work currently if all the assigned devices
> are under the same physical SMMUv3 as we have a check in kernel
> whether domains are having the same SMMUv3. I remember Jason mentioning
> that he is planning to relax that. So are the Qemu side changes based on that
> assumption? And any idea how we are planning to relax that restriction? We
> do a check for compatibility of the phys SMMUv3s and then allow/restrict in kernel?
> Do Qemu can then try allocating a separate s2 hwpt for those and attach again?
> Sorry if this was already discussed elsewhere and I missed that.
You are very right about this. I haven't thought about supporting
that case yet. Likely I need to spare some time to refine the QEMU
part in the coming weeks or so (have been waiting for Zhenzhong's
full nesting patches).
Also, NVIDIA has an interest to support CMDQ-V accelerator, where
we need more vSMMU instances, than just one. I haven't decided how
to handle this for both single-vSMMU and multi-vSMMU versions.
Yet, some of the ongoing work might help:
https://github.com/nicolinc/iommufd/commit/b7520901184fd9fa127abb88c1f0be16b9967cff
https://github.com/nicolinc/iommufd/commit/d969ffeac899491d3a6e54557bf6a46d52b95865
So, each device should poll hw_info to get its SMMU ID, if it is
not in the S2 list of the vSMMU's, add it and allocate a new S2.
Jason is on vacation. I believe we'll have a more solid discussion
later.
Thanks
Nicolin
next prev parent reply other threads:[~2024-03-13 23:51 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-13 10:13 Query on ARM SMMUv3 nested support Shameerali Kolothum Thodi
2024-03-13 23:50 ` Nicolin Chen [this message]
2024-03-18 16:22 ` Jason Gunthorpe
2024-03-22 15:04 ` Shameerali Kolothum Thodi
2024-03-29 7:13 ` Nicolin Chen
2024-04-02 7:25 ` Shameerali Kolothum Thodi
2024-04-02 11:28 ` Jason Gunthorpe
2024-04-09 6:12 ` Nicolin Chen
2024-04-09 19:47 ` Nicolin Chen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZfI74miZ1ss9+Kz0@Asurada-Nvidia \
--to=nicolinc@nvidia.com \
--cc=eric.auger@redhat.com \
--cc=iommu@lists.linux.dev \
--cc=jgg@nvidia.com \
--cc=linuxarm@huawei.com \
--cc=mdf@kernel.org \
--cc=mshavit@google.com \
--cc=shameerali.kolothum.thodi@huawei.com \
--cc=zhangfei.gao@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.