From: Jason Gunthorpe <jgg@ziepe.ca> To: Tomasz Jeznach <tjeznach@rivosinc.com> Cc: Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>, Paul Walmsley <paul.walmsley@sifive.com>, Palmer Dabbelt <palmer@dabbelt.com>, Albert Ou <aou@eecs.berkeley.edu>, Anup Patel <apatel@ventanamicro.com>, Sunil V L <sunilvl@ventanamicro.com>, Nick Kossifidis <mick@ics.forth.gr>, Sebastien Boeuf <seb@rivosinc.com>, Rob Herring <robh+dt@kernel.org>, Krzysztof Kozlowski <krzk+dt@kernel.org>, Conor Dooley <conor+dt@kernel.org>, devicetree@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux@rivosinc.com Subject: Re: [PATCH v2 5/7] iommu/riscv: Device directory management. Date: Fri, 19 Apr 2024 09:40:17 -0300 [thread overview] Message-ID: <20240419124017.GC223006@ziepe.ca> (raw) In-Reply-To: <232b2824d5dfd9b8dcb3553bfd506444273c3305.1713456598.git.tjeznach@rivosinc.com> On Thu, Apr 18, 2024 at 09:32:23AM -0700, Tomasz Jeznach wrote: > @@ -31,13 +32,350 @@ MODULE_LICENSE("GPL"); > /* Timeouts in [us] */ > #define RISCV_IOMMU_DDTP_TIMEOUT 50000 > > -static int riscv_iommu_attach_identity_domain(struct iommu_domain *domain, > - struct device *dev) > +/* RISC-V IOMMU PPN <> PHYS address conversions, PHYS <=> PPN[53:10] */ > +#define phys_to_ppn(va) (((va) >> 2) & (((1ULL << 44) - 1) << 10)) > +#define ppn_to_phys(pn) (((pn) << 2) & (((1ULL << 44) - 1) << 12)) > + > +#define dev_to_iommu(dev) \ > + container_of((dev)->iommu->iommu_dev, struct riscv_iommu_device, iommu) We have iommu_get_iommu_dev() now > +static unsigned long riscv_iommu_get_pages(struct riscv_iommu_device *iommu, unsigned int order) > +{ > + struct riscv_iommu_devres *devres; > + struct page *pages; > + > + pages = alloc_pages_node(dev_to_node(iommu->dev), > + GFP_KERNEL_ACCOUNT | __GFP_ZERO, order); > + if (unlikely(!pages)) { > + dev_err(iommu->dev, "Page allocation failed, order %u\n", order); > + return 0; > + } This needs adjusting for the recently merged allocation accounting > +static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, > + struct device *dev, > + struct iommu_domain *iommu_domain) > +{ > + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); > + struct riscv_iommu_dc *dc; > + u64 fsc, ta, tc; > + int i; > + > + if (!iommu_domain) { > + ta = 0; > + tc = 0; > + fsc = 0; > + } else if (iommu_domain->type == IOMMU_DOMAIN_IDENTITY) { > + ta = 0; > + tc = RISCV_IOMMU_DC_TC_V; > + fsc = FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, RISCV_IOMMU_DC_FSC_MODE_BARE); > + } else { > + /* This should never happen. */ > + return -ENODEV; > + } Please don't write it like this. This function is already being called by functions that are already under specific ops, don't check domain->type here. Instead have the caller compute and pass in the ta/tc/fsc values. Maybe in a tidy struct.. > + /* Update existing or allocate new entries in device directory */ > + for (i = 0; i < fwspec->num_ids; i++) { > + dc = riscv_iommu_get_dc(iommu, fwspec->ids[i], !iommu_domain); > + if (!dc && !iommu_domain) > + continue; > + if (!dc) > + return -ENODEV; But if this fails some of the fwspecs were left in a weird state ? Drivers should try hard to have attach functions that fail and make no change at all or fully succeed. Meaning ideally preallocate any required memory before doing any change to the HW visable structures. > + > + /* Swap device context, update TC valid bit as the last operation */ > + xchg64(&dc->fsc, fsc); > + xchg64(&dc->ta, ta); > + xchg64(&dc->tc, tc); This doesn't loook right? When you get to adding PAGING suport fsc has the page table pfn and ta has the cache tag, so this will end up tearing the data for sure, eg when asked to replace a PAGING domain with another PAGING domain? That will create a functional/security problem, right? I would encourage you to re-use the ARM sequencing code, ideally moved to some generic helper library. Every iommu driver dealing with multi-quanta descriptors seems to have this same fundamental sequencing problem. > +static void riscv_iommu_release_device(struct device *dev) > +{ > + struct riscv_iommu_device *iommu = dev_to_iommu(dev); > + > + riscv_iommu_attach_domain(iommu, dev, NULL); > +} The release_domain has landed too now. Please don't invent weird NULL domain types that have special meaning. I assume clearing the V bit is a blocking behavior? So please implement a proper blocking domain and set release_domain = &riscv_iommu_blocking and just omit this release function. > @@ -133,12 +480,14 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) > rc = riscv_iommu_init_check(iommu); > if (rc) > return dev_err_probe(iommu->dev, rc, "unexpected device state\n"); > - /* > - * Placeholder for a complete IOMMU device initialization. > - * For now, only bare minimum: enable global identity mapping mode and register sysfs. > - */ > - riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP, > - FIELD_PREP(RISCV_IOMMU_DDTP_MODE, RISCV_IOMMU_DDTP_MODE_BARE)); > + > + rc = riscv_iommu_ddt_alloc(iommu); > + if (WARN(rc, "cannot allocate device directory\n")) > + goto err_init; memory allocation failure already makes noisy prints, more prints are not needed.. > + rc = riscv_iommu_set_ddtp_mode(iommu, RISCV_IOMMU_DDTP_MODE_MAX); > + if (WARN(rc, "cannot enable iommu device\n")) > + goto err_init; This is not a proper use of WARN, it should only be used for things that cannot happen not undesired error paths. Jason
WARNING: multiple messages have this Message-ID (diff)
From: Jason Gunthorpe <jgg@ziepe.ca> To: Tomasz Jeznach <tjeznach@rivosinc.com> Cc: Anup Patel <apatel@ventanamicro.com>, devicetree@vger.kernel.org, Conor Dooley <conor+dt@kernel.org>, Albert Ou <aou@eecs.berkeley.edu>, linux@rivosinc.com, Will Deacon <will@kernel.org>, Joerg Roedel <joro@8bytes.org>, linux-kernel@vger.kernel.org, Rob Herring <robh+dt@kernel.org>, Sebastien Boeuf <seb@rivosinc.com>, iommu@lists.linux.dev, Palmer Dabbelt <palmer@dabbelt.com>, Paul Walmsley <paul.walmsley@sifive.com>, Nick Kossifidis <mick@ics.forth.gr>, Krzysztof Kozlowski <krzk+dt@kernel.org>, Robin Murphy <robin.murphy@arm.com>, linux-riscv@lists.infradead.org Subject: Re: [PATCH v2 5/7] iommu/riscv: Device directory management. Date: Fri, 19 Apr 2024 09:40:17 -0300 [thread overview] Message-ID: <20240419124017.GC223006@ziepe.ca> (raw) In-Reply-To: <232b2824d5dfd9b8dcb3553bfd506444273c3305.1713456598.git.tjeznach@rivosinc.com> On Thu, Apr 18, 2024 at 09:32:23AM -0700, Tomasz Jeznach wrote: > @@ -31,13 +32,350 @@ MODULE_LICENSE("GPL"); > /* Timeouts in [us] */ > #define RISCV_IOMMU_DDTP_TIMEOUT 50000 > > -static int riscv_iommu_attach_identity_domain(struct iommu_domain *domain, > - struct device *dev) > +/* RISC-V IOMMU PPN <> PHYS address conversions, PHYS <=> PPN[53:10] */ > +#define phys_to_ppn(va) (((va) >> 2) & (((1ULL << 44) - 1) << 10)) > +#define ppn_to_phys(pn) (((pn) << 2) & (((1ULL << 44) - 1) << 12)) > + > +#define dev_to_iommu(dev) \ > + container_of((dev)->iommu->iommu_dev, struct riscv_iommu_device, iommu) We have iommu_get_iommu_dev() now > +static unsigned long riscv_iommu_get_pages(struct riscv_iommu_device *iommu, unsigned int order) > +{ > + struct riscv_iommu_devres *devres; > + struct page *pages; > + > + pages = alloc_pages_node(dev_to_node(iommu->dev), > + GFP_KERNEL_ACCOUNT | __GFP_ZERO, order); > + if (unlikely(!pages)) { > + dev_err(iommu->dev, "Page allocation failed, order %u\n", order); > + return 0; > + } This needs adjusting for the recently merged allocation accounting > +static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, > + struct device *dev, > + struct iommu_domain *iommu_domain) > +{ > + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); > + struct riscv_iommu_dc *dc; > + u64 fsc, ta, tc; > + int i; > + > + if (!iommu_domain) { > + ta = 0; > + tc = 0; > + fsc = 0; > + } else if (iommu_domain->type == IOMMU_DOMAIN_IDENTITY) { > + ta = 0; > + tc = RISCV_IOMMU_DC_TC_V; > + fsc = FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, RISCV_IOMMU_DC_FSC_MODE_BARE); > + } else { > + /* This should never happen. */ > + return -ENODEV; > + } Please don't write it like this. This function is already being called by functions that are already under specific ops, don't check domain->type here. Instead have the caller compute and pass in the ta/tc/fsc values. Maybe in a tidy struct.. > + /* Update existing or allocate new entries in device directory */ > + for (i = 0; i < fwspec->num_ids; i++) { > + dc = riscv_iommu_get_dc(iommu, fwspec->ids[i], !iommu_domain); > + if (!dc && !iommu_domain) > + continue; > + if (!dc) > + return -ENODEV; But if this fails some of the fwspecs were left in a weird state ? Drivers should try hard to have attach functions that fail and make no change at all or fully succeed. Meaning ideally preallocate any required memory before doing any change to the HW visable structures. > + > + /* Swap device context, update TC valid bit as the last operation */ > + xchg64(&dc->fsc, fsc); > + xchg64(&dc->ta, ta); > + xchg64(&dc->tc, tc); This doesn't loook right? When you get to adding PAGING suport fsc has the page table pfn and ta has the cache tag, so this will end up tearing the data for sure, eg when asked to replace a PAGING domain with another PAGING domain? That will create a functional/security problem, right? I would encourage you to re-use the ARM sequencing code, ideally moved to some generic helper library. Every iommu driver dealing with multi-quanta descriptors seems to have this same fundamental sequencing problem. > +static void riscv_iommu_release_device(struct device *dev) > +{ > + struct riscv_iommu_device *iommu = dev_to_iommu(dev); > + > + riscv_iommu_attach_domain(iommu, dev, NULL); > +} The release_domain has landed too now. Please don't invent weird NULL domain types that have special meaning. I assume clearing the V bit is a blocking behavior? So please implement a proper blocking domain and set release_domain = &riscv_iommu_blocking and just omit this release function. > @@ -133,12 +480,14 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) > rc = riscv_iommu_init_check(iommu); > if (rc) > return dev_err_probe(iommu->dev, rc, "unexpected device state\n"); > - /* > - * Placeholder for a complete IOMMU device initialization. > - * For now, only bare minimum: enable global identity mapping mode and register sysfs. > - */ > - riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP, > - FIELD_PREP(RISCV_IOMMU_DDTP_MODE, RISCV_IOMMU_DDTP_MODE_BARE)); > + > + rc = riscv_iommu_ddt_alloc(iommu); > + if (WARN(rc, "cannot allocate device directory\n")) > + goto err_init; memory allocation failure already makes noisy prints, more prints are not needed.. > + rc = riscv_iommu_set_ddtp_mode(iommu, RISCV_IOMMU_DDTP_MODE_MAX); > + if (WARN(rc, "cannot enable iommu device\n")) > + goto err_init; This is not a proper use of WARN, it should only be used for things that cannot happen not undesired error paths. Jason _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv
next prev parent reply other threads:[~2024-04-19 12:40 UTC|newest] Thread overview: 60+ messages / expand[flat|nested] mbox.gz Atom feed top 2024-04-18 16:32 [PATCH v2 0/7] Linux RISC-V IOMMU Support Tomasz Jeznach 2024-04-18 16:32 ` Tomasz Jeznach 2024-04-18 16:32 ` [PATCH v2 1/7] dt-bindings: iommu: riscv: Add bindings for RISC-V IOMMU Tomasz Jeznach 2024-04-18 16:32 ` Tomasz Jeznach 2024-04-18 17:04 ` Conor Dooley 2024-04-18 17:04 ` Conor Dooley 2024-04-24 22:37 ` Tomasz Jeznach 2024-04-24 22:37 ` Tomasz Jeznach 2024-04-25 17:11 ` Conor Dooley 2024-04-25 17:11 ` Conor Dooley 2024-04-22 14:04 ` Rob Herring 2024-04-22 14:04 ` Rob Herring 2024-04-18 16:32 ` [PATCH v2 2/7] iommu/riscv: Add RISC-V IOMMU platform device driver Tomasz Jeznach 2024-04-18 16:32 ` Tomasz Jeznach 2024-04-18 21:22 ` Robin Murphy 2024-04-18 21:22 ` Robin Murphy 2024-04-24 21:59 ` Tomasz Jeznach 2024-04-24 21:59 ` Tomasz Jeznach 2024-04-25 11:23 ` Robin Murphy 2024-04-25 11:23 ` Robin Murphy 2024-04-18 16:32 ` [PATCH v2 3/7] iommu/riscv: Add RISC-V IOMMU PCIe " Tomasz Jeznach 2024-04-18 16:32 ` Tomasz Jeznach 2024-04-18 22:07 ` Robin Murphy 2024-04-18 22:07 ` Robin Murphy 2024-04-18 16:32 ` [PATCH v2 4/7] iommu/riscv: Enable IOMMU registration and device probe Tomasz Jeznach 2024-04-18 16:32 ` Tomasz Jeznach 2024-04-18 16:32 ` [PATCH v2 5/7] iommu/riscv: Device directory management Tomasz Jeznach 2024-04-18 16:32 ` Tomasz Jeznach 2024-04-19 12:40 ` Jason Gunthorpe [this message] 2024-04-19 12:40 ` Jason Gunthorpe 2024-04-24 23:01 ` Tomasz Jeznach 2024-04-24 23:01 ` Tomasz Jeznach 2024-04-24 23:07 ` Jason Gunthorpe 2024-04-24 23:07 ` Jason Gunthorpe 2024-04-22 5:11 ` Baolu Lu 2024-04-22 5:11 ` Baolu Lu 2024-04-24 23:07 ` Tomasz Jeznach 2024-04-24 23:07 ` Tomasz Jeznach 2024-04-18 16:32 ` [PATCH v2 6/7] iommu/riscv: Command and fault queue support Tomasz Jeznach 2024-04-18 16:32 ` Tomasz Jeznach 2024-04-18 16:32 ` [PATCH v2 7/7] iommu/riscv: Paging domain support Tomasz Jeznach 2024-04-18 16:32 ` Tomasz Jeznach 2024-04-19 12:56 ` Jason Gunthorpe 2024-04-19 12:56 ` Jason Gunthorpe 2024-04-22 7:40 ` Baolu Lu 2024-04-22 7:40 ` Baolu Lu 2024-04-24 23:30 ` Tomasz Jeznach 2024-04-24 23:30 ` Tomasz Jeznach 2024-04-24 23:39 ` Jason Gunthorpe 2024-04-24 23:39 ` Jason Gunthorpe 2024-04-24 23:54 ` Tomasz Jeznach 2024-04-24 23:54 ` Tomasz Jeznach 2024-04-25 0:48 ` Jason Gunthorpe 2024-04-25 0:48 ` Jason Gunthorpe 2024-04-22 5:21 ` Baolu Lu 2024-04-22 5:21 ` Baolu Lu 2024-04-22 19:30 ` Jason Gunthorpe 2024-04-22 19:30 ` Jason Gunthorpe 2024-04-23 17:00 ` Andrew Jones 2024-04-23 17:00 ` Andrew Jones
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20240419124017.GC223006@ziepe.ca \ --to=jgg@ziepe.ca \ --cc=aou@eecs.berkeley.edu \ --cc=apatel@ventanamicro.com \ --cc=conor+dt@kernel.org \ --cc=devicetree@vger.kernel.org \ --cc=iommu@lists.linux.dev \ --cc=joro@8bytes.org \ --cc=krzk+dt@kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-riscv@lists.infradead.org \ --cc=linux@rivosinc.com \ --cc=mick@ics.forth.gr \ --cc=palmer@dabbelt.com \ --cc=paul.walmsley@sifive.com \ --cc=robh+dt@kernel.org \ --cc=robin.murphy@arm.com \ --cc=seb@rivosinc.com \ --cc=sunilvl@ventanamicro.com \ --cc=tjeznach@rivosinc.com \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.