From: Nilay Shroff <nilay@linux.ibm.com>
To: "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>
Cc: Keith Busch <kbusch@kernel.org>, Daniel Wagner <dwagner@suse.de>,
Christoph Hellwig <hch@lst.de>, Sagi Grimberg <sagi@grimberg.me>,
Gregory Joyce <gjoyce@ibm.com>,
Srimannarayana Murthy Maram <msmurthy@imap.linux.ibm.com>,
"axboe@fb.com" <axboe@fb.com>
Subject: nvme-cli : attaching a namespace to an undiscovered nvme controller on multi-controller nvme disk
Date: Mon, 6 May 2024 19:45:20 +0530 [thread overview]
Message-ID: <f1a7c1e2-3203-4b7a-a922-82fa812455bd@linux.ibm.com> (raw)
Hi,
I have got an NVMe disk with multi-controller support. This disk has two controllers
however kernel could only discover one of the controllers. On this disk if we create
a namespace and attach it to a controller (which is not discovered by the kernel) then
that namespace can't used for IO. In fact, there's no path to reach such a namespace
from kernel.
Please find the details below:
# lspci
0018:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM173X
# nvme list -v
Subsystem Subsystem-NQN Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys0 nqn.1994-11.com.samsung:nvme:PM1735:2.5-inch:S6EUNA0R500358 nvme0
Device SN MN FR TxPort Asdress Slot Subsystem Namespaces
-------- -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ----------------
nvme0 S6EUNA0R500358 1.6TB NVMe Gen4 U.2 SSD REV.SN49 pcie 0018:01:00.0 nvme-subsys0
Device Generic NSID Usage Format Controllers
------------ ------------ ---------- -------------------------- ---------------- ----------------
# nvme id-ctrl /dev/nvme0 -H
NVME Identify Controller:
vid : 0x144d
ssvid : 0x1014
sn : S6EUNA0R500358
mn : 1.6TB NVMe Gen4 U.2 SSD
fr : REV.SN49
rab : 8
ieee : 002538
cmic : 0x3
[3:3] : 0 ANA not supported
[2:2] : 0 PCI
[1:1] : 0x1 Multi Controller <== this is multi-controller disk
[0:0] : 0x1 Multi Port
<snip>
<snip>
# nvme list-ctrl /dev/nvme0
num of ctrls present: 2
[ 0]:0x41
[ 1]:0x42
# cat /sys/class/nvme/nvme0/cntlid
65
So it's apparent from the above output that kernel has discovered one controller (nvme0)
with cntlid 65(0x41). The other controller with cntlid 0x42 remain undiscovered.
Please note even though kernel couldn't discover both disk controllers, it's possible to
create and attach namespace to a controller which is not discovered by the kernel.
# nvme create-ns /dev/nvme0 --nsze=0x1749A956 --ncap=0x1749A956 --block-size=4096
create-ns: Success, created nsid:1
Now, attach namespace to a controller which is not discovered/known by the kernel.
# nvme attach-ns /dev/nvme0 -c 0x42 -n 1
attach-ns: Success, nsid:1
# nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
The nvme list doesn't show any namespace and the reason being the namespace is attached
to a controller which is not discovered by the kernel. The kernel doesn't have any path to
access this namespace. So in essence now this namespace can't be used for any IO.
Looking at the above output (namespace list is empty) if we try creating a new namespace
then we see the below error.
# nvme create-ns /dev/nvme0 --nsze=0x1749A956 --ncap=0x1749A956 --block-size=4096
NVMe status: Namespace Insufficient Capacity: Creating the namespace requires more free space than is currently available(0x2115)
# nvme id-ctrl /dev/nvme0 -H
NVME Identify Controller:
vid : 0x144d
ssvid : 0x1014
sn : S6EUNA0R500358
mn : 1.6TB NVMe Gen4 U.2 SSD
<snip>
<snip>
tnvmcap : 1600321314816
[127:0] : 1600321314816
Total NVM Capacity (TNVMCAP)
unvmcap : 0
[127:0] : 0
Unallocated NVM Capacity (UNVMCAP)
<snip>
<snip>
We find from the above output that all available capacity of disk has been used but
nvme list output is empty.
Please note that "kernel is unable to discover both NVMe controllers" is not a kernel bug.
Also I think, this is not a bug with NVMe disk or its firmware. This multi controller NVMe
disk has a "DualPortEn#" pin and by default, the disk is configured to operate a single x4 port.
Asserting the "DualPortEn#" pin enables the Dual x2 port mode. When disk operates in Dual x2 mode,
kernel could discover both controllers. However when "DualportEn#" pin is not asserted
disk would only enable one of the controllers and hence kernel could discover one controller.
There could be multiple ways to address this issue. However my proposal would be to address
it in nvme-cli. As nvme-cli builds the nvme topology it shall be easy for nvme-cli to detect
this anomaly when a namespace is being attached to undiscovered nvme controller and shows some
WARNING message. The WARNING could be shown for ~10 seconds so that it gives enough time to a
user to cancel this operation. If the operation is not cancelled then nvme-cli would proceed
as usual.
Please help review and as usual, comments/feedback/suggestions are most welcome!
Thanks,
--Nilay
next reply other threads:[~2024-05-06 14:15 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-06 14:15 Nilay Shroff [this message]
2024-05-06 15:05 ` nvme-cli : attaching a namespace to an undiscovered nvme controller on multi-controller nvme disk Daniel Wagner
2024-05-07 11:44 ` Nilay Shroff
2024-05-16 6:25 ` Nilay Shroff
2024-05-27 11:20 ` Nilay Shroff
2024-05-27 13:36 ` Daniel Wagner
2024-05-27 14:02 ` Nilay Shroff
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f1a7c1e2-3203-4b7a-a922-82fa812455bd@linux.ibm.com \
--to=nilay@linux.ibm.com \
--cc=axboe@fb.com \
--cc=dwagner@suse.de \
--cc=gjoyce@ibm.com \
--cc=hch@lst.de \
--cc=kbusch@kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=msmurthy@imap.linux.ibm.com \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).