From: Tomasz Jeznach <tjeznach@rivosinc.com>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>,
Robin Murphy <robin.murphy@arm.com>,
Paul Walmsley <paul.walmsley@sifive.com>,
Palmer Dabbelt <palmer@dabbelt.com>,
Albert Ou <aou@eecs.berkeley.edu>,
Anup Patel <apatel@ventanamicro.com>,
Sunil V L <sunilvl@ventanamicro.com>,
Nick Kossifidis <mick@ics.forth.gr>,
Sebastien Boeuf <seb@rivosinc.com>,
Rob Herring <robh+dt@kernel.org>,
Krzysztof Kozlowski <krzk+dt@kernel.org>,
Conor Dooley <conor+dt@kernel.org>,
devicetree@vger.kernel.org, iommu@lists.linux.dev,
linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org,
linux@rivosinc.com
Subject: Re: [PATCH v3 7/7] iommu/riscv: Paging domain support
Date: Fri, 3 May 2024 10:44:14 -0700 [thread overview]
Message-ID: <CAH2o1u63GjMnYrfa8W-c1hdp+TAA0R-FyxXM4dEiFF+KEGWwbA@mail.gmail.com> (raw)
In-Reply-To: <20240501145621.GD1723318@ziepe.ca>
On Wed, May 1, 2024 at 7:56 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Tue, Apr 30, 2024 at 01:01:57PM -0700, Tomasz Jeznach wrote:
>
> > +#define iommu_domain_to_riscv(iommu_domain) \
> > + container_of(iommu_domain, struct riscv_iommu_domain, domain)
> > +
> > +#define dev_to_domain(dev) \
> > + iommu_domain_to_riscv(dev_iommu_priv_get(dev))
>
> Please use the priv properly and put a struct around it, you'll
> certainly need this eventually to do the rest of the advanced
> features.
>
Done. Yes, indeed, I was going to introduce proper struct in follow up
patches anyway. Pulled this change sooner.
> > +static void riscv_iommu_bond_unlink(struct riscv_iommu_domain *domain, struct device *dev)
> > +{
> > + struct riscv_iommu_bond *bond, *found = NULL;
> > + unsigned long flags;
> > +
> > + if (!domain)
> > + return;
> > +
> > + spin_lock_irqsave(&domain->lock, flags);
>
> This is never locked from an irq, you don't need to use the irqsave
> variations.
>
Good point. done in v4.
> > + list_for_each_entry_rcu(bond, &domain->bonds, list) {
> > + if (bond->dev == dev) {
> > + list_del_rcu(&bond->list);
> > + found = bond;
> > + }
> > + }
> > + spin_unlock_irqrestore(&domain->lock, flags);
> > +
> > + /* Release and wait for all read-rcu critical sections have completed. */
> > + kfree_rcu(found, rcu);
> > + synchronize_rcu();
>
> Please no, synchronize_rcu() on a path like this is not so
> reasonable.. Also you don't need kfree_rcu() if you write it like this.
>
> This still looks better to do what I said before, put the iommu not
> the dev in the bond struct.
>
>
I was trying not to duplicate data in bond struct and use whatever is
available to be referenced from dev pointer (eg iommu / ids / private
iommu dev data). If I'm reading core iommu code correctly, device
pointer and iommu pointers should be valid between _probe_device and
_release_device calls. I've moved synchronize_rcu out of the domain
attach path to _release_device, LMK if that would be ok for now.
I'll have a second another to rework other patches to avoid storing
dev pointers at all.
> > +static struct iommu_domain *riscv_iommu_alloc_paging_domain(struct device *dev)
> > +{
> > + struct riscv_iommu_domain *domain;
> > + struct riscv_iommu_device *iommu;
> > +
> > + iommu = dev ? dev_to_iommu(dev) : NULL;
> > + domain = kzalloc(sizeof(*domain), GFP_KERNEL);
> > + if (!domain)
> > + return ERR_PTR(-ENOMEM);
> > +
> > + INIT_LIST_HEAD_RCU(&domain->bonds);
> > + spin_lock_init(&domain->lock);
> > + domain->numa_node = NUMA_NO_NODE;
> > +
> > + /*
> > + * Follow system address translation mode.
> > + * RISC-V IOMMU ATP mode values match RISC-V CPU SATP mode values.
> > + */
> > + domain->pgd_mode = satp_mode >> SATP_MODE_SHIFT;
>
> This seems really strange, the iommu paging domains should be
> unrelated to what the CPU is doing. There is no connection between
> these two concepts.
>
> Just pick a size that the iommu supports.
>
> The number of radix levels is a tunable alot of iommus have that we
> haven't really exposed to anything else yet.
>
Makes sense. I've left an option to pick mode from MMU for cases where
dev/iommu is not known at allocation time (with iommu_domain_alloc()).
I'd guess it's reasonable to assume IOMMU supported page modes will
match MMU.
> > + /*
> > + * Note: RISC-V Privilege spec mandates that virtual addresses
> > + * need to be sign-extended, so if (VA_BITS - 1) is set, all
> > + * bits >= VA_BITS need to also be set or else we'll get a
> > + * page fault. However the code that creates the mappings
> > + * above us (e.g. iommu_dma_alloc_iova()) won't do that for us
> > + * for now, so we'll end up with invalid virtual addresses
> > + * to map. As a workaround until we get this sorted out
> > + * limit the available virtual addresses to VA_BITS - 1.
> > + */
> > + domain->domain.geometry.aperture_start = 0;
> > + domain->domain.geometry.aperture_end = DMA_BIT_MASK(VA_BITS - 1);
> > + domain->domain.geometry.force_aperture = true;
>
> Yikes.. This is probably the best solution long term anyhow, unless
> you need to use the last page in VFIO for some reason.
>
> > static int riscv_iommu_device_domain_type(struct device *dev)
> > {
> > - return IOMMU_DOMAIN_IDENTITY;
> > + struct riscv_iommu_device *iommu = dev_to_iommu(dev);
> > +
> > + if (iommu->ddt_mode == RISCV_IOMMU_DDTP_MODE_BARE)
> > + return IOMMU_DOMAIN_IDENTITY;
> > +
>
> Is there even a point of binding an iommu driver if the HW can't
> support a DDT table? Just return -ENODEV from probe_device?
>
> Logically a iommu block that can't decode the RID has no association
> at all with a Linux struct device :)
>
Done. Good point ;)
Thanks for review,
- Tomasz
> Jason
next prev parent reply other threads:[~2024-05-03 17:44 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-30 20:01 [PATCH v3 0/7] Linux RISC-V IOMMU Support Tomasz Jeznach
2024-04-30 20:01 ` [PATCH v3 1/7] dt-bindings: iommu: riscv: Add bindings for RISC-V IOMMU Tomasz Jeznach
2024-05-01 9:30 ` Conor Dooley
2024-05-01 13:15 ` Rob Herring
2024-05-02 2:47 ` Tomasz Jeznach
2024-05-02 15:15 ` Conor Dooley
2024-04-30 20:01 ` [PATCH v3 2/7] iommu/riscv: Add RISC-V IOMMU platform device driver Tomasz Jeznach
2024-05-01 10:26 ` Baolu Lu
2024-05-01 14:20 ` Jason Gunthorpe
2024-05-02 2:23 ` Baolu Lu
2024-05-02 2:44 ` Tomasz Jeznach
2024-04-30 20:01 ` [PATCH v3 3/7] iommu/riscv: Add RISC-V IOMMU PCIe " Tomasz Jeznach
2024-05-01 10:01 ` Baolu Lu
2024-04-30 20:01 ` [PATCH v3 4/7] iommu/riscv: Enable IOMMU registration and device probe Tomasz Jeznach
2024-05-01 9:53 ` Baolu Lu
2024-04-30 20:01 ` [PATCH v3 5/7] iommu/riscv: Device directory management Tomasz Jeznach
2024-05-01 14:57 ` Jason Gunthorpe
2024-05-02 1:38 ` Baolu Lu
2024-05-02 1:57 ` Baolu Lu
2024-05-02 2:06 ` Baolu Lu
2024-04-30 20:01 ` [PATCH v3 6/7] iommu/riscv: Command and fault queue support Tomasz Jeznach
2024-05-02 3:51 ` Baolu Lu
2024-04-30 20:01 ` [PATCH v3 7/7] iommu/riscv: Paging domain support Tomasz Jeznach
2024-05-01 14:56 ` Jason Gunthorpe
2024-05-03 17:44 ` Tomasz Jeznach [this message]
2024-05-03 18:10 ` Jason Gunthorpe
2024-05-03 19:44 ` Tomasz Jeznach
2024-05-05 15:46 ` Jason Gunthorpe
2024-05-07 2:22 ` Tomasz Jeznach
2024-05-07 16:51 ` Jason Gunthorpe
2024-05-08 16:23 ` Tomasz Jeznach
2024-05-02 3:50 ` Baolu Lu
2024-05-02 4:39 ` Tomasz Jeznach
2024-05-01 16:07 ` [PATCH v3 0/7] Linux RISC-V IOMMU Support Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAH2o1u63GjMnYrfa8W-c1hdp+TAA0R-FyxXM4dEiFF+KEGWwbA@mail.gmail.com \
--to=tjeznach@rivosinc.com \
--cc=aou@eecs.berkeley.edu \
--cc=apatel@ventanamicro.com \
--cc=conor+dt@kernel.org \
--cc=devicetree@vger.kernel.org \
--cc=iommu@lists.linux.dev \
--cc=jgg@ziepe.ca \
--cc=joro@8bytes.org \
--cc=krzk+dt@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-riscv@lists.infradead.org \
--cc=linux@rivosinc.com \
--cc=mick@ics.forth.gr \
--cc=palmer@dabbelt.com \
--cc=paul.walmsley@sifive.com \
--cc=robh+dt@kernel.org \
--cc=robin.murphy@arm.com \
--cc=seb@rivosinc.com \
--cc=sunilvl@ventanamicro.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).