From: Jason Gunthorpe <jgg@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: "Alex Williamson" <alex.williamson@redhat.com>,
"Liu, Yi L" <yi.l.liu@intel.com>,
"joro@8bytes.org" <joro@8bytes.org>,
"robin.murphy@arm.com" <robin.murphy@arm.com>,
"eric.auger@redhat.com" <eric.auger@redhat.com>,
"nicolinc@nvidia.com" <nicolinc@nvidia.com>,
"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
"chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
"Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
"Pan, Jacob jun" <jacob.jun.pan@intel.com>,
"Cédric Le Goater" <clg@redhat.com>
Subject: Re: [PATCH v2 0/4] vfio-pci support pasid attach/detach
Date: Mon, 29 Apr 2024 14:15:15 -0300 [thread overview]
Message-ID: <20240429171515.GI941030@nvidia.com> (raw)
In-Reply-To: <BL1PR11MB527133859BC129E2F65A61718C142@BL1PR11MB5271.namprd11.prod.outlook.com>
On Sun, Apr 28, 2024 at 06:19:29AM +0000, Tian, Kevin wrote:
> > From: Alex Williamson <alex.williamson@redhat.com>
> > Sent: Saturday, April 27, 2024 4:14 AM
> >
> > On Fri, 26 Apr 2024 11:11:17 -0300
> > Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > > On Wed, Apr 24, 2024 at 02:13:49PM -0600, Alex Williamson wrote:
> > >
> > > > This is kind of an absurd example to portray as a ubiquitous problem.
> > > > Typically the config space layout is a reflection of hardware whether
> > > > the device supports migration or not.
> > >
> > > Er, all our HW has FW constructed config space. It changes with FW
> > > upgrades. We change it during the life of the product. This has to be
> > > considered..
> >
> > So as I understand it, the concern is that you have firmware that
> > supports migration, but it also openly hostile to the fundamental
> > aspects of exposing a stable device ABI in support of migration.
> >
> > > > If a driver were to insert a
> > > > virtual capability, then yes it would want to be consistent about it if
> > > > it also cares about migration. If the driver needs to change the
> > > > location of a virtual capability, problems will arise, but that's also
> > > > not something that every driver needs to do.
> > >
> > > Well, mlx5 has to cope with this. It supports so many devices with so
> > > many config space layouts :( I don't know if we can just hard wire an
> > > offset to stick in a PASID cap and expect that to work...
>
> Are those config space layout differences usually also coming with
> mmio-side interface change?
Not really
> If yes there are more to handle for
> running V1 instance on V2 device and it'd make sense to manage
> everything about compatibility in one place.
It's complicated :|
The config space layout can't change once the device is discovered by
the OS and I have a feeling it can't differ between VFs in a SRIOV.
So even if we did say that a device HW/FW will change it's personality
on the fly, I'm not sure we actually can and still conform to the
PCI specs?
> If we pursue the direction deciding the vconfig layout in VMM, does
> it imply that anything related to mmio layout would also be put in
> VMM too?
I'd guess no, MMIO doesn't have the OS and standards entanglements. If
the FW can reprogram the MMIO layout then when it loads the migration
state, or provisions the device, it should put the MMIO to the right
configuration.
> Do we envision things like above in the variant driver or in VMM?
In the device itself. I think it would be an extreme case for someone
to make a device that had a different MMIO layout that could be
backwards compatible to an old layout and NOT provide a way for the
device HW to go back to the old layout directly. If so that device
would have to mediate MMIO in the VMM or kernel. Depends on the
complexity I suppose where to do it.
> > > Consider standards based migration between wildly different
> > > devices. The devices will not standardize their physical config space,
> > > but an operator could generate a consistent vPCI config space that
> > > works with all the devices in their fleet.
>
> It's hard to believe that 'wildly different' devices only have difference
> in the layout of vPCI config space.
Well, I mean a device from vendor A and another device from vendor
B. They don't have anything in common except implementing the
standard.
The standard perscribes the MMIO layout. Both devices have a way to
accept a migration stream defined by a standard. The MMIO will be
adjusted as the standard requires after loading the migration stream.
Config space primiarily remains a problem in this imagined world.
My best view is that it needs to be solved by config space synthesis
in the hypervisor and not via HW/FW on the physical device.
> and looks this community lacks of a clear criteria on what burden
> should be put in the kernel vs. in the VMM.
Right, we've always discussed this. I've fairly consistently wanted to
push things to userspace, principally for security.
> e.g. in earlier nvgrace-gpu discussion a major open was whether
> the PCI bar emulation should be done by the variant driver or
> by the VMM (with variant driver providing a device feature).
Sort of, this was actually also about config space synthesis. If we
had this kind of scheme then perhaps the argument would be different.
> It ends up to be in the variant driver with one major argument
> that doing so avoids the burden in various VMMs.
Yes, that has always been the argument that has won out.
> btw while this discussion may continue some time, I wonder whether
> this vPASID reporting open can be handled separately from the
> pasid attach/detach series so we can move the ball and merge
> something already in agreement. anyway it's just a read-only cap so
> won't affect how VFIO/IOMMUFD handles the pasid related requests.
Yes, this makes sense to me. Ultimately we all agree the config space
will have to be synthesized.
Jason
next prev parent reply other threads:[~2024-04-29 17:15 UTC|newest]
Thread overview: 72+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-12 8:21 [PATCH v2 0/4] vfio-pci support pasid attach/detach Yi Liu
2024-04-12 8:21 ` [PATCH v2 1/4] ida: Add ida_get_lowest() Yi Liu
2024-04-16 16:03 ` Alex Williamson
2024-04-18 7:02 ` Yi Liu
2024-04-18 16:23 ` Alex Williamson
2024-04-18 17:12 ` Jason Gunthorpe
2024-04-19 13:43 ` Yi Liu
2024-04-19 13:55 ` Alex Williamson
2024-04-19 14:00 ` Jason Gunthorpe
2024-04-23 7:19 ` Yi Liu
2024-04-19 13:40 ` Yi Liu
2024-04-12 8:21 ` [PATCH v2 2/4] vfio-iommufd: Support pasid [at|de]tach for physical VFIO devices Yi Liu
2024-04-16 9:01 ` Tian, Kevin
2024-04-16 9:24 ` Yi Liu
2024-04-16 9:47 ` Tian, Kevin
2024-04-18 7:04 ` Yi Liu
2024-04-23 12:43 ` Jason Gunthorpe
2024-04-24 0:33 ` Tian, Kevin
2024-04-24 4:48 ` Yi Liu
2024-04-12 8:21 ` [PATCH v2 3/4] vfio: Add VFIO_DEVICE_PASID_[AT|DE]TACH_IOMMUFD_PT Yi Liu
2024-04-16 9:13 ` Tian, Kevin
2024-04-16 9:36 ` Yi Liu
2024-04-23 12:45 ` Jason Gunthorpe
2024-04-12 8:21 ` [PATCH v2 4/4] vfio: Report PASID capability via VFIO_DEVICE_FEATURE ioctl Yi Liu
2024-04-16 9:40 ` Tian, Kevin
2024-04-16 17:57 ` Alex Williamson
2024-04-17 7:09 ` Tian, Kevin
2024-04-17 20:25 ` Alex Williamson
2024-04-18 0:21 ` Tian, Kevin
2024-04-18 8:23 ` Yi Liu
2024-04-18 16:34 ` Alex Williamson
2024-04-23 12:39 ` Jason Gunthorpe
2024-04-24 0:24 ` Tian, Kevin
2024-04-24 13:59 ` Jason Gunthorpe
2024-04-16 8:38 ` [PATCH v2 0/4] vfio-pci support pasid attach/detach Tian, Kevin
2024-04-16 17:50 ` Jason Gunthorpe
2024-04-17 7:16 ` Tian, Kevin
2024-04-17 12:20 ` Jason Gunthorpe
2024-04-17 23:02 ` Alex Williamson
2024-04-18 0:06 ` Tian, Kevin
2024-04-18 9:03 ` Yi Liu
2024-04-18 20:37 ` Alex Williamson
2024-04-19 5:52 ` Tian, Kevin
2024-04-19 16:35 ` Alex Williamson
2024-04-23 7:43 ` Tian, Kevin
2024-04-23 12:01 ` Jason Gunthorpe
2024-04-23 23:47 ` Tian, Kevin
2024-04-24 0:12 ` Jason Gunthorpe
2024-04-24 2:57 ` Tian, Kevin
2024-04-24 12:29 ` Baolu Lu
2024-04-24 14:04 ` Jason Gunthorpe
2024-04-24 5:19 ` Tian, Kevin
2024-04-24 14:15 ` Jason Gunthorpe
2024-04-24 18:38 ` Alex Williamson
2024-04-24 18:45 ` Jason Gunthorpe
2024-04-24 18:24 ` Alex Williamson
2024-04-24 18:36 ` Jason Gunthorpe
2024-04-24 20:13 ` Alex Williamson
2024-04-26 14:11 ` Jason Gunthorpe
2024-04-26 20:13 ` Alex Williamson
2024-04-28 6:19 ` Tian, Kevin
2024-04-29 7:43 ` Yi Liu
2024-04-29 17:15 ` Jason Gunthorpe [this message]
2024-04-29 17:44 ` Jason Gunthorpe
2024-04-27 5:05 ` Christoph Hellwig
2024-04-25 9:26 ` Yi Liu
2024-04-25 12:58 ` Alex Williamson
2024-04-26 9:01 ` Yi Liu
2024-04-19 13:59 ` Jason Gunthorpe
2024-04-23 7:58 ` Yi Liu
2024-04-23 12:05 ` Jason Gunthorpe
2024-04-19 13:34 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240429171515.GI941030@nvidia.com \
--to=jgg@nvidia.com \
--cc=alex.williamson@redhat.com \
--cc=baolu.lu@linux.intel.com \
--cc=chao.p.peng@linux.intel.com \
--cc=clg@redhat.com \
--cc=eric.auger@redhat.com \
--cc=iommu@lists.linux.dev \
--cc=jacob.jun.pan@intel.com \
--cc=joro@8bytes.org \
--cc=kevin.tian@intel.com \
--cc=kvm@vger.kernel.org \
--cc=nicolinc@nvidia.com \
--cc=robin.murphy@arm.com \
--cc=yi.l.liu@intel.com \
--cc=zhenzhong.duan@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.