From: "Tian, Kevin" <kevin.tian@intel.com>
To: Alex Williamson <alex.williamson@redhat.com>,
Jason Gunthorpe <jgg@nvidia.com>
Cc: "Vetter, Daniel" <daniel.vetter@intel.com>,
"Zhao, Yan Y" <yan.y.zhao@intel.com>,
"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"x86@kernel.org" <x86@kernel.org>,
"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
"pbonzini@redhat.com" <pbonzini@redhat.com>,
"seanjc@google.com" <seanjc@google.com>,
"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
"luto@kernel.org" <luto@kernel.org>,
"peterz@infradead.org" <peterz@infradead.org>,
"tglx@linutronix.de" <tglx@linutronix.de>,
"mingo@redhat.com" <mingo@redhat.com>,
"bp@alien8.de" <bp@alien8.de>, "hpa@zytor.com" <hpa@zytor.com>,
"corbet@lwn.net" <corbet@lwn.net>,
"joro@8bytes.org" <joro@8bytes.org>,
"will@kernel.org" <will@kernel.org>,
"robin.murphy@arm.com" <robin.murphy@arm.com>,
"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
"Liu, Yi L" <yi.l.liu@intel.com>
Subject: RE: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in non-coherent domains
Date: Fri, 24 May 2024 00:30:20 +0000 [thread overview]
Message-ID: <BN9PR11MB52768B71B41ADD916ED51D348CF52@BN9PR11MB5276.namprd11.prod.outlook.com> (raw)
In-Reply-To: <20240523164753.32e714d5.alex.williamson@redhat.com>
> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Friday, May 24, 2024 6:48 AM
>
> On Thu, 23 May 2024 11:58:48 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> > On Wed, May 22, 2024 at 11:40:58PM +0000, Tian, Kevin wrote:
> > > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > > Sent: Thursday, May 23, 2024 7:32 AM
> > > >
> > > > On Wed, May 22, 2024 at 11:26:21PM +0000, Tian, Kevin wrote:
> > > > > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > > > > Sent: Wednesday, May 22, 2024 8:30 PM
> > > > > >
> > > > > > On Wed, May 22, 2024 at 06:24:14AM +0000, Tian, Kevin wrote:
> > > > > > > I'm fine to do a special check in the attach path to enable the flush
> > > > > > > only for Intel GPU.
> > > > > >
> > > > > > We already effectively do this already by checking the domain
> > > > > > capabilities. Only the Intel GPU will have a non-coherent domain.
> > > > > >
> > > > >
> > > > > I'm confused. In earlier discussions you wanted to find a way to not
> > > > > publish others due to the check of non-coherent domain, e.g. some
> > > > > ARM SMMU cannot force snoop.
> > > > >
> > > > > Then you and Alex discussed the possibility of reducing pessimistic
> > > > > flushes by virtualizing the PCI NOSNOOP bit.
> > > > >
> > > > > With that in mind I was thinking whether we explicitly enable this
> > > > > flush only for Intel GPU instead of checking non-coherent domain
> > > > > in the attach path, since it's the only device with such requirement.
> > > >
> > > > I am suggesting to do both checks:
> > > > - If the iommu domain indicates it has force coherency then leave PCI
> > > > no-snoop alone and no flush
> > > > - If the PCI NOSNOOP bit is or can be 0 then no flush
> > > > - Otherwise flush
> > >
> > > How to judge whether PCI NOSNOOP can be 0? If following PCI spec
> > > it can always be set to 0 but then we break the requirement for Intel
> > > GPU. If we explicitly exempt Intel GPU in 2nd check then what'd be
> > > the value of doing that generic check?
> >
> > Non-PCI environments still have this problem, and the first check does
> > help them since we don't have PCI config space there.
> >
> > PCI can supply more information (no snoop impossible) and variant
> > drivers can add in too (want no snoop)
>
> I'm not sure I follow either. Since i915 doesn't set or test no-snoop
> enable, I think we need to assume drivers expect the reset value, so a
> device that supports no-snoop expects to use it, ie. we can't trap on
> no-snoop enable being set, the device is more likely to just operate
> with reduced performance if we surreptitiously clear the bit.
>
> The current proposal is to enable flushing based only on the domain
> enforcement of coherency. I think the augmentation is therefore that
> if the device is PCI and the no-snoop enable bit is zero after reset
> (indicating hardwired to zero), we also don't need to flush.
>
> I'm not sure the polarity of the variant drive statement above is
> correct. If the no-snoop enable bit is set after reset, we'd assume
> no-snoop is possible, so the variant driver would only need a way to
> indicate the device doesn't actually use no-snoop. For that it might
> just virtualize the no-snoop enable setting to vfio-pci-core. Thanks,
>
Yeah. I re-checked the use of PCI_EXP_DEVCTL_NOSNOOP_EN and
actually all references are about clearing the bit, echo'ing the point
that if a driver wants to use nosnoop it expects the reset value w/o
doing an explicit set and the virtualization of the no-snoop enable
bit is more reasonable to catch the intention of 'clear'.
next prev parent reply other threads:[~2024-05-24 0:30 UTC|newest]
Thread overview: 78+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-07 6:18 [PATCH 0/5] Enforce CPU cache flush for non-coherent device assignment Yan Zhao
2024-05-07 6:19 ` [PATCH 1/5] x86/pat: Let pat_pfn_immune_to_uc_mtrr() check MTRR for untracked PAT range Yan Zhao
2024-05-07 8:26 ` Tian, Kevin
2024-05-07 9:12 ` Yan Zhao
2024-05-08 22:14 ` Alex Williamson
2024-05-09 3:36 ` Yan Zhao
2024-05-16 7:42 ` Tian, Kevin
2024-05-16 14:07 ` Sean Christopherson
2024-05-20 2:36 ` Tian, Kevin
2024-05-07 6:20 ` [PATCH 2/5] KVM: x86/mmu: Fine-grained check of whether a invalid & RAM PFN is MMIO Yan Zhao
2024-05-07 8:39 ` Tian, Kevin
2024-05-07 9:19 ` Yan Zhao
2024-05-07 6:20 ` [PATCH 3/5] x86/mm: Introduce and export interface arch_clean_nonsnoop_dma() Yan Zhao
2024-05-07 8:51 ` Tian, Kevin
2024-05-07 9:40 ` Yan Zhao
2024-05-20 14:07 ` Christoph Hellwig
2024-05-21 15:49 ` Jason Gunthorpe
2024-05-21 16:00 ` Jason Gunthorpe
2024-05-22 3:41 ` Yan Zhao
2024-05-28 6:37 ` Christoph Hellwig
2024-06-01 19:46 ` Jason Gunthorpe
2024-06-06 2:48 ` Yan Zhao
2024-06-06 11:55 ` Jason Gunthorpe
2024-06-07 9:39 ` Yan Zhao
2024-05-07 6:21 ` [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in non-coherent domains Yan Zhao
2024-05-09 18:10 ` Alex Williamson
2024-05-10 10:31 ` Yan Zhao
2024-05-10 16:57 ` Alex Williamson
2024-05-13 7:11 ` Yan Zhao
2024-05-16 7:53 ` Tian, Kevin
2024-05-16 8:34 ` Tian, Kevin
2024-05-16 20:31 ` Alex Williamson
2024-05-17 17:11 ` Jason Gunthorpe
2024-05-20 2:52 ` Tian, Kevin
2024-05-21 16:07 ` Jason Gunthorpe
2024-05-21 16:21 ` Alex Williamson
2024-05-21 16:34 ` Jason Gunthorpe
2024-05-21 18:19 ` Alex Williamson
2024-05-21 18:37 ` Jason Gunthorpe
2024-05-22 6:24 ` Tian, Kevin
2024-05-22 12:29 ` Jason Gunthorpe
2024-05-22 14:43 ` Alex Williamson
2024-05-22 16:52 ` Jason Gunthorpe
2024-05-22 18:22 ` Alex Williamson
2024-05-22 23:26 ` Tian, Kevin
2024-05-22 23:32 ` Jason Gunthorpe
2024-05-22 23:40 ` Tian, Kevin
2024-05-23 14:58 ` Jason Gunthorpe
2024-05-23 22:47 ` Alex Williamson
2024-05-24 0:30 ` Tian, Kevin [this message]
2024-05-24 13:50 ` Jason Gunthorpe
2024-05-22 3:33 ` Yan Zhao
2024-05-22 3:24 ` Yan Zhao
2024-05-22 12:26 ` Jason Gunthorpe
2024-05-24 3:07 ` Yan Zhao
2024-05-16 20:50 ` Alex Williamson
2024-05-17 3:11 ` Yan Zhao
2024-05-17 4:44 ` Alex Williamson
2024-05-17 5:00 ` Yan Zhao
2024-05-07 6:22 ` [PATCH 5/5] iommufd: " Yan Zhao
2024-05-09 14:13 ` Jason Gunthorpe
2024-05-10 8:03 ` Yan Zhao
2024-05-10 13:29 ` Jason Gunthorpe
2024-05-13 7:43 ` Yan Zhao
2024-05-14 15:11 ` Jason Gunthorpe
2024-05-15 7:06 ` Yan Zhao
2024-05-15 20:43 ` Jason Gunthorpe
2024-05-16 2:32 ` Yan Zhao
2024-05-16 8:38 ` Tian, Kevin
2024-05-16 9:48 ` Yan Zhao
2024-05-17 17:04 ` Jason Gunthorpe
2024-05-20 2:45 ` Yan Zhao
2024-05-21 16:04 ` Jason Gunthorpe
2024-05-22 3:17 ` Yan Zhao
2024-05-22 6:29 ` Yan Zhao
2024-05-22 17:01 ` Jason Gunthorpe
2024-05-27 7:15 ` Yan Zhao
2024-06-01 16:48 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=BN9PR11MB52768B71B41ADD916ED51D348CF52@BN9PR11MB5276.namprd11.prod.outlook.com \
--to=kevin.tian@intel.com \
--cc=alex.williamson@redhat.com \
--cc=baolu.lu@linux.intel.com \
--cc=bp@alien8.de \
--cc=corbet@lwn.net \
--cc=daniel.vetter@intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=hpa@zytor.com \
--cc=iommu@lists.linux.dev \
--cc=jgg@nvidia.com \
--cc=joro@8bytes.org \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=luto@kernel.org \
--cc=mingo@redhat.com \
--cc=pbonzini@redhat.com \
--cc=peterz@infradead.org \
--cc=robin.murphy@arm.com \
--cc=seanjc@google.com \
--cc=tglx@linutronix.de \
--cc=will@kernel.org \
--cc=x86@kernel.org \
--cc=yan.y.zhao@intel.com \
--cc=yi.l.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).