From: Vidya Sagar <vidyas@nvidia.com>
To: <corbet@lwn.net>, <bhelgaas@google.com>, <galshalom@nvidia.com>,
<leonro@nvidia.com>, <jgg@nvidia.com>, <treding@nvidia.com>,
<jonathanh@nvidia.com>
Cc: <mmoshrefjava@nvidia.com>, <shahafs@nvidia.com>,
<vsethi@nvidia.com>, <sdonthineni@nvidia.com>, <jan@nvidia.com>,
<tdave@nvidia.com>, <linux-doc@vger.kernel.org>,
<linux-pci@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
<kthota@nvidia.com>, <mmaddireddy@nvidia.com>,
<vidyas@nvidia.com>, <sagar.tv@gmail.com>
Subject: [PATCH V1] PCI: Extend ACS configurability
Date: Wed, 8 May 2024 16:56:58 +0530 [thread overview]
Message-ID: <20240508112658.3555882-1-vidyas@nvidia.com> (raw)
For iommu_groups to form correctly, the ACS settings in the PCIe fabric
need to be setup early in the boot process, either via the BIOS or via
the kernel disable_acs_redir parameter.
disable_acs_redir allows clearing the RR|CR|EC ACS flags, but the PCIe
spec Rev3.0 already defines 7 different ACS related flags with many more
useful combinations depending on the fabric design.
For backward compatibility, leave the 'disable_acs_redir' as is and add
a new parameter 'config_acs'so that the user can directly specify the ACS
flags to set on a per-device basis. Use a similar syntax to the existing
'resource_alignment' parameter by using the @ character and have the user
specify the ACS flags using a bit encoding. If both 'disable_acs_redir' and
'config_acs' are specified for a particular device, configuration specified
through 'config_acs' takes precedence over the other.
Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
---
.../admin-guide/kernel-parameters.txt | 22 ++++
drivers/pci/pci.c | 119 ++++++++++++++----
2 files changed, 119 insertions(+), 22 deletions(-)
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 41644336e..b4a8207eb 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -4456,6 +4456,28 @@
bridges without forcing it upstream. Note:
this removes isolation between devices and
may put more devices in an IOMMU group.
+ config_acs=
+ Format:
+ =<ACS flags>@<pci_dev>[; ...]
+ Specify one or more PCI devices (in the format
+ specified above) optionally prepended with flags
+ and separated by semicolons. The respective
+ capabilities will be enabled, disabled or unchanged
+ based on what is specified in flags.
+ ACS Flags is defined as follows
+ bit-0 : ACS Source Validation
+ bit-1 : ACS Translation Blocking
+ bit-2 : ACS P2P Request Redirect
+ bit-3 : ACS P2P Completion Redirect
+ bit-4 : ACS Upstream Forwarding
+ bit-5 : ACS P2P Egress Control
+ bit-6 : ACS Direct Translated P2P
+ Each bit can be marked as
+ ‘0‘ – force disabled
+ ‘1’ – force enabled
+ ‘x’ – unchanged.
+ Note: this may remove isolation between devices
+ and may put more devices in an IOMMU group.
force_floating [S390] Force usage of floating interrupts.
nomio [S390] Do not use MIO instructions.
norid [S390] ignore the RID field and force use of
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index a607f277c..0ad48ade9 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -887,30 +887,59 @@ void pci_request_acs(void)
}
static const char *disable_acs_redir_param;
+static const char *config_acs_param;
-/**
- * pci_disable_acs_redir - disable ACS redirect capabilities
- * @dev: the PCI device
- *
- * For only devices specified in the disable_acs_redir parameter.
- */
-static void pci_disable_acs_redir(struct pci_dev *dev)
+static void __pci_config_acs(struct pci_dev *dev, const char *p,
+ u16 mask, u16 flags)
{
+ char *delimit;
int ret = 0;
- const char *p;
- int pos;
- u16 ctrl;
+ u16 ctrl, pos;
- if (!disable_acs_redir_param)
- return;
-
- p = disable_acs_redir_param;
while (*p) {
+ if (!mask) {
+ /* Check for ACS flags */
+ delimit = strstr(p, "@");
+ if (delimit) {
+ int end;
+ u32 shift = 0;
+
+ end = delimit - p - 1;
+
+ while (end > -1) {
+ if (*(p + end) == '0') {
+ mask |= 1 << shift;
+ shift++;
+ end--;
+ } else if (*(p + end) == '1') {
+ mask |= 1 << shift;
+ flags |= 1 << shift;
+ shift++;
+ end--;
+ } else if ((*(p + end) == 'x') || (*(p + end) == 'X')) {
+ shift++;
+ end--;
+ } else {
+ pci_err(dev, "Invalid ACS flags... Ignoring\n");
+ return;
+ }
+ }
+ p = delimit + 1;
+ } else {
+ pci_err(dev, "ACS Flags missing\n");
+ return;
+ }
+ }
+
+ if (mask & ~(PCI_ACS_SV | PCI_ACS_TB | PCI_ACS_RR | PCI_ACS_CR |
+ PCI_ACS_UF | PCI_ACS_EC | PCI_ACS_DT)) {
+ pci_err(dev, "Invalid ACS flags specified\n");
+ return;
+ }
+
ret = pci_dev_str_match(dev, p, &p);
if (ret < 0) {
- pr_info_once("PCI: Can't parse disable_acs_redir parameter: %s\n",
- disable_acs_redir_param);
-
+ pr_info_once("PCI: Can't parse acs command line parameter\n");
break;
} else if (ret == 1) {
/* Found a match */
@@ -932,18 +961,60 @@ static void pci_disable_acs_redir(struct pci_dev *dev)
pos = dev->acs_cap;
if (!pos) {
- pci_warn(dev, "cannot disable ACS redirect for this hardware as it does not have ACS capabilities\n");
+ pci_warn(dev, "cannot configure ACS for this hardware as it does not have ACS capabilities\n");
return;
}
+ pci_dbg(dev, "ACS mask = 0x%X\n", mask);
+ pci_dbg(dev, "ACS flags = 0x%X\n", flags);
+
pci_read_config_word(dev, pos + PCI_ACS_CTRL, &ctrl);
+ ctrl &= ~mask;
+ ctrl |= flags;
+ pci_write_config_word(dev, pos + PCI_ACS_CTRL, ctrl);
- /* P2P Request & Completion Redirect */
- ctrl &= ~(PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_EC);
+ pci_info(dev, "Configured ACS\n");
+}
- pci_write_config_word(dev, pos + PCI_ACS_CTRL, ctrl);
+/**
+ * pci_disable_acs_redir - disable ACS redirect capabilities
+ * @dev: the PCI device
+ *
+ * For only devices specified in the disable_acs_redir parameter.
+ */
+static void pci_disable_acs_redir(struct pci_dev *dev)
+{
+ const char *p;
+ u16 mask = 0, flags = 0;
+
+ if (!disable_acs_redir_param)
+ return;
+
+ p = disable_acs_redir_param;
+
+ mask = PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_EC;
+ flags = ~(PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_EC);
+
+ __pci_config_acs(dev, p, mask, flags);
+}
+
+/**
+ * pci_config_acs - configure ACS capabilities
+ * @dev: the PCI device
+ *
+ * For only devices specified in the config_acs parameter.
+ */
+static void pci_config_acs(struct pci_dev *dev)
+{
+ const char *p;
+ u16 mask = 0, flags = 0;
+
+ if (!config_acs_param)
+ return;
+
+ p = config_acs_param;
- pci_info(dev, "disabled ACS redirect\n");
+ __pci_config_acs(dev, p, mask, flags);
}
/**
@@ -1005,6 +1076,7 @@ static void pci_enable_acs(struct pci_dev *dev)
* preferences.
*/
pci_disable_acs_redir(dev);
+ pci_config_acs(dev);
}
/**
@@ -7023,6 +7095,8 @@ static int __init pci_setup(char *str)
pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
} else if (!strncmp(str, "disable_acs_redir=", 18)) {
disable_acs_redir_param = str + 18;
+ } else if (!strncmp(str, "config_acs=", 11)) {
+ config_acs_param = str + 11;
} else {
pr_err("PCI: Unknown option `%s'\n", str);
}
@@ -7047,6 +7121,7 @@ static int __init pci_realloc_setup_params(void)
resource_alignment_param = kstrdup(resource_alignment_param,
GFP_KERNEL);
disable_acs_redir_param = kstrdup(disable_acs_redir_param, GFP_KERNEL);
+ config_acs_param = kstrdup(config_acs_param, GFP_KERNEL);
return 0;
}
--
2.25.1
next reply other threads:[~2024-05-08 11:27 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-08 11:26 Vidya Sagar [this message]
2024-05-08 13:14 ` [PATCH V1] PCI: Extend ACS configurability Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240508112658.3555882-1-vidyas@nvidia.com \
--to=vidyas@nvidia.com \
--cc=bhelgaas@google.com \
--cc=corbet@lwn.net \
--cc=galshalom@nvidia.com \
--cc=jan@nvidia.com \
--cc=jgg@nvidia.com \
--cc=jonathanh@nvidia.com \
--cc=kthota@nvidia.com \
--cc=leonro@nvidia.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=mmaddireddy@nvidia.com \
--cc=mmoshrefjava@nvidia.com \
--cc=sagar.tv@gmail.com \
--cc=sdonthineni@nvidia.com \
--cc=shahafs@nvidia.com \
--cc=tdave@nvidia.com \
--cc=treding@nvidia.com \
--cc=vsethi@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).