From: Lior Weintraub <liorw@pliops.com>
To: "linux-embedded@vger.kernel.org" <linux-embedded@vger.kernel.org>
Subject: [kernel 6.10.10][aarch64] PCIe Bridge - NVMe SSD - No SMMU (or IOMMU)
Date: Mon, 16 Sep 2024 11:22:33 +0000 [thread overview]
Message-ID: <PR3P195MB055551DBC4A87ED666237576C3602@PR3P195MB0555.EURP195.PROD.OUTLOOK.COM> (raw)
Dear friends,
I am running Linux kernel 6.10.10 on a Cortex A53 single CPU running @ 1.3GHz.
The CPU is part of a SoC with a PCIe bridge (root port) from Synopsis (using compatible = "snps,dw-pcie").
A Gen4 SSD is connected to the PCIe RP so when I run lspci I see both the PCIe bridge and the SSD:
# lspci
00:00.0 PCI bridge: Device 1e7e:abcd (rev 01)
01:00.0 Non-Volatile memory controller: Sandisk Corp Device 5040 (rev 03)
The problem I am facing is that the NVMe driver fails to load:
[ 0.862737][ T10] nvme nvme0: 1/0/0 default/read/poll queues
[ 0.874457][ C0] could not locate request for tag 0xfff
[ 0.879977][ C0] nvme nvme0: invalid id 65535 completed on queue 1
[ 31.820058][ T8] nvme nvme0: I/O tag 128 (0080) opcode 0x2 (I/O Cmd) QID 1 timeout, aborting req_op:READ(0) size:4096
[ 31.831882][ C0] nvme nvme0: Abort status: 0x0
[ 62.540052][ T8] nvme nvme0: I/O tag 128 (0080) opcode 0x2 (I/O Cmd) QID 1 timeout, reset controller
[ 62.596074][ T20] nvme nvme0: 1/0/0 default/read/poll queues
[ 62.602059][ C0] could not locate request for tag 0xfff
[ 62.607567][ C0] nvme nvme0: invalid id 65535 completed on queue 1
[ 93.260066][ T8] nvme nvme0: I/O tag 129 (0081) QID 1 timeout, disable controller
[ 93.274391][ T8] I/O error, dev nvme0n1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 93.283627][ T8] Buffer I/O error on dev nvme0n1, logical block 0, async page read
[ 93.291676][ T20] nvme nvme0: failed to mark controller live state
[ 93.298111][ T20] nvme nvme0: Disabling device after reset failure: -19
[ 93.305094][ T20] Buffer I/O error on dev nvme0n1, logical block 0, async page read
[ 93.313101][ T27] nvme0n1: unable to read partition table
I found on drivers/nvme/host/pci.c that the address written to the device is the dma address which is different from the physical address.
Adding some prints to nvme_pci_configure_admin_queue:
[ 0.825383][ T10] nvme nvme0: ----> sq_dma_addr 0xa36000 cq_dma_addr 0xa35000
[ 0.832767][ T10] nvme nvme0: ----> phys sq = 0x8afd000 phys cq = 0x8af5000
After reading the DMA api howto page (https://docs.kernel.org/core-api/dma-api-howto.html) it is clear and understandable the use of Bus Address Space and IOMMU (or SMMU).
My SoC doesn't have SMMU and the mapping between the device and the physical address is 1:1 mapping.
So the question is:
How to disable the MMU and cause the kernel allocating dma addresses that match physical addresses.
Tried the following (each one separately) and none worked:
1. pass boot argument "iommu.passthrough=1"
2. Remove "IOMMU Hardware Support" from kernel configuration.
3. Enable "IOMMU Hardware Support" but set the CONFIG_IOMMU_DEFAULT_PASSTHROUGH=y
Thanks,
Lior.
reply other threads:[~2024-09-16 11:22 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=PR3P195MB055551DBC4A87ED666237576C3602@PR3P195MB0555.EURP195.PROD.OUTLOOK.COM \
--to=liorw@pliops.com \
--cc=linux-embedded@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).