History

Merge tag 'nvme-7.3-2026-08-13' of git://git.infradead.org/nvme into for-7.3/block

Pull NVMe updates from Keith: "- Enable context analysis for the nvme host driver, annotating the subsystem's locks, along with the LIST_HEAD_GUARDED support it needs (Nilay, Marco) - Harden the tcp host and target against malformed PDUs and out of range SGL lengths (Yehyeong, Ibrahim, Greg) - Fix unserialized page_frag_cache use in nvme-tcp request setup (Dmitry) - Bound identify, FDP and passthrough descriptor parsing to the allocated buffers (Hari, Guixin) - Zoned namespace fixes for host and the target (Xixin, Guixin, Yao) - Apple controller fixes: page aligned admin queue buffers, NVMMU TCB setup, DMA direction and admin queue teardown (Sven, Gui-Dong) - Add a namespace level debugfs directory exposing reservation state, and ABI documentation for the host sysfs and target configfs interfaces (Guixin) - Fix cdev and namespace lifetimes (John) - Parallelize nvme-rdma I/O queue allocation and startup (Surabhi) - Fix nvmet-rdma response resource leak on queue teardown (Shin'ichiro) - Authentication fixes: AUTH_RECEIVE buffer and an out of bounds read in negotiate (Xixin, Bryam, Guixin, Eric) - Fix pci-epf use-after-free and CQ reference leak (Shin'ichiro, Yifei) - Reject passthrough of driver managed Set Features (Chao) - Various error path and teardown fixes across the host and target addressing issues with use-after-free and leaking resources (Guixin, Maurizio, Ewan, Zhengrong, Jiang HongHui, Myeonghun, Yang, Geliang, Yehyeong) - Various cleanups and typo fixes (Nilay, Guixin, Pan Chuang)" * tag 'nvme-7.3-2026-08-13' of git://git.infradead.org/nvme: (81 commits) nvmet: fix max_qid race between configfs and controller allocation nvme: nvme-fc: Fix nvme_fc_create_hw_io_queues() queue deletion in error path nvme: ratelimit the completion-path messages driven by device data nvme-tcp: fix host memory disclosure on R2T for a read command nvme-tcp: do not accept C2HData based on blk_rq_payload_bytes() alone nvme-tcp: reject a read that transferred too few bytes nvmet: zns: reject full zone report when buffer is too small nvme-tcp: fix usage of page_frag_cache nvme: reject passthrough of driver-managed Set Features nvmet: fix NULL pointer dereference in nvmet_execute_identify_ns_zns() nvmet: pci-epf: fix use-after-free in nvmet_pci_epf_exec_iod_work() nvmet: pci-epf: put CQ ref on create_cq mapping failure nvme-apple: Drop the PRP null check chicken bit nvme-apple: Require page aligned buffers on the admin queue nvme: Add a quirk for page aligned admin queue buffers nvme-apple: Never set the opcode in the NVMMU TCB nvme-apple: Don't set a DMA direction for commands without a data transfer nvme-apple: Destroy the admin queue on removal nvmet: fix heap out-of-bounds read in nvmet_auth_negotiate() nvme: raise FDP placement handle cap to U8_MAX and warn on overflow ...

Jens Axboe committed 3 months ago

Commit 642ec082 parents 30df3ab f1a884636 files · +1561 −289Browse files

Unified · Split

36 files
  1. Documentation/ABI/stable/configfs-nvmet +352 −0
  2. Documentation/ABI/stable/sysfs-nvme +453 −0
  3. Documentation/ABI/testing/sysfs-nvme +0 −13
  4. MAINTAINERS +4 −0
  5. drivers/nvme/common/auth.c +6 −7
  6. drivers/nvme/host/Makefile +1 −0
  7. drivers/nvme/host/apple.c +39 −27
  8. drivers/nvme/host/auth.c +2 −1
  9. drivers/nvme/host/core.c +45 −7
  10. drivers/nvme/host/fabrics.c +2 −2
  11. drivers/nvme/host/fc.c +7 −1
  12. drivers/nvme/host/ioctl.c +79 −42
  13. drivers/nvme/host/multipath.c +34 −30
  14. drivers/nvme/host/nvme.h +50 −17
  15. drivers/nvme/host/pci.c +24 −7
  16. drivers/nvme/host/rdma.c +97 −63
  17. drivers/nvme/host/sysfs.c +3 −1
  18. drivers/nvme/host/tcp.c +64 −7
  19. drivers/nvme/host/zns.c +8 −3
  20. drivers/nvme/target/admin-cmd.c +8 −6
  21. drivers/nvme/target/configfs.c +6 −4
  22. drivers/nvme/target/core.c +31 −23
  23. drivers/nvme/target/debugfs.c +103 −0
  24. drivers/nvme/target/debugfs.h +5 −0
  25. drivers/nvme/target/fabrics-cmd-auth.c +13 −4
  26. drivers/nvme/target/fabrics-cmd.c +1 −1
  27. drivers/nvme/target/fc.c +1 −1
  28. drivers/nvme/target/nvmet.h +9 −0
  29. drivers/nvme/target/passthru.c +9 −0
  30. drivers/nvme/target/pci-epf.c +8 −4
  31. drivers/nvme/target/pr.c +30 −7
  32. drivers/nvme/target/rdma.c +28 −3
  33. drivers/nvme/target/tcp.c +17 −2
  34. drivers/nvme/target/zns.c +11 −4
  35. include/linux/list.h +9 −0
  36. include/linux/nvme.h +2 −2
Documentation/ABI/stable/configfs-nvmet+352−0
@@ -0,0 +1,352 @@
1+What: /config/nvmet/ports/N/addr_adrfam
2+What: /config/nvmet/ports/N/addr_portid
3+What: /config/nvmet/ports/N/addr_traddr
4+What: /config/nvmet/ports/N/addr_trsvcid
5+What: /config/nvmet/ports/N/addr_trtype
6+What: /config/nvmet/ports/N/addr_treq
7+Date: June 2016
8+KernelVersion: 4.8
9+Contact: Christoph Hellwig <hch@lst.de>
10+Description:
11+ Address attributes for an NVMe-oF target port.
12+
13+ addr_adrfam: Shows or sets the address family. Accepted
14+ values: "pcie", "ipv4", "ipv6", "ib", "fc", "pci", "loop".
15+
16+ addr_portid: Shows or sets the port identifier (u16).
17+
18+ addr_traddr: Shows or sets the transport address string.
19+
20+ addr_trsvcid: Shows or sets the transport service identifier.
21+
22+ addr_trtype: Shows or sets the transport type. Accepted
23+ values: "rdma", "fc", "tcp", "pci", "loop". Also
24+ initializes default TSAS values.
25+
26+ addr_treq: Shows or sets the transport security requirements.
27+ Accepted values: "not specified", "required",
28+ "not required". For TCP with TLS1.3, "not specified" is
29+ rejected.
30+
31+ All attributes require the port to be disabled before
32+ modification.
33+
34+What: /config/nvmet/ports/N/referrals/NAME/addr_adrfam
35+What: /config/nvmet/ports/N/referrals/NAME/addr_portid
36+What: /config/nvmet/ports/N/referrals/NAME/addr_traddr
37+What: /config/nvmet/ports/N/referrals/NAME/addr_trsvcid
38+What: /config/nvmet/ports/N/referrals/NAME/addr_trtype
39+What: /config/nvmet/ports/N/referrals/NAME/addr_treq
40+What: /config/nvmet/ports/N/referrals/NAME/enable
41+Date: June 2016
42+KernelVersion: 4.8
43+Contact: Christoph Hellwig <hch@lst.de>
44+Description:
45+ Address attributes and enable control for a referral entry
46+ under a port. The addr_* attributes have the same semantics
47+ as the corresponding port-level attributes. The enable
48+ attribute shows or sets whether this referral is enabled
49+ (boolean).
50+
51+What: /config/nvmet/ports/N/param_inline_data_size
52+Date: June 2018
53+KernelVersion: 4.19
54+Contact: Steve Wise <swise@opengridcomputing.com>
55+Description:
56+ Shows or sets the inline data size for this port. Default
57+ is -1 which lets the transport choose. The port must be
58+ disabled before modification.
59+
60+What: /config/nvmet/ports/N/ana_groups/ID/ana_state
61+Date: June 2018
62+KernelVersion: 4.19
63+Contact: Christoph Hellwig <hch@lst.de>
64+Description:
65+ Shows or sets the ANA (Asymmetric Namespace Access) state
66+ for this group on this port. Accepted values: "optimized",
67+ "non-optimized", "inaccessible", "persistent-loss",
68+ "change". Changes trigger an ANA change event.
69+
70+What: /config/nvmet/ports/N/param_pi_enable
71+Date: May 2020
72+KernelVersion: 5.8
73+Contact: Israel Rukshin <israelr@mellanox.com>
74+Description:
75+ Shows or sets whether protection information (PI) is
76+ enabled/supported for this port. Accepts boolean value.
77+ Only available when CONFIG_BLK_DEV_INTEGRITY is enabled.
78+ The port must be disabled before modification.
79+
80+What: /config/nvmet/ports/N/addr_tsas
81+Date: August 2023
82+KernelVersion: 6.7
83+Contact: Hannes Reinecke <hare@suse.de>
84+Description:
85+ Shows or sets the transport-specific address subtype. For
86+ TCP transport, accepted values: "none", "tls1.3" (requires
87+ CONFIG_NVME_TARGET_TCP_TLS). For RDMA transport, shows the
88+ QP type: "connected" or "datagram". The port must be
89+ disabled before modification.
90+
91+What: /config/nvmet/ports/N/param_max_queue_size
92+Date: January 2024
93+KernelVersion: 6.9
94+Contact: Max Gurtovoy <mgurtovoy@nvidia.com>
95+Description:
96+ Shows or sets the maximum queue size for this port. Default
97+ is -1 which lets the transport choose. The port must be
98+ disabled before modification.
99+
100+What: /config/nvmet/ports/N/param_mdts
101+Date: April 2026
102+KernelVersion: 7.1
103+Contact: Aurelien Aptel <aaptel@nvidia.com>
104+Description:
105+ Shows or sets the maximum data transfer size for this port.
106+ Default is -1 which lets the transport choose. The port
107+ must be disabled before modification.
108+
109+What: /config/nvmet/subsystems/NAME/attr_allow_any_host
110+Date: June 2016
111+KernelVersion: 4.8
112+Contact: Christoph Hellwig <hch@lst.de>
113+Description:
114+ Shows or sets whether any host is allowed to connect.
115+ Accepts boolean value. Cannot be set to 1 if explicit
116+ hosts are linked in the allowed_hosts/ directory.
117+
118+What: /config/nvmet/subsystems/NAME/namespaces/NSID/device_path
119+What: /config/nvmet/subsystems/NAME/namespaces/NSID/device_nguid
120+What: /config/nvmet/subsystems/NAME/namespaces/NSID/enable
121+Date: June 2016
122+KernelVersion: 4.8
123+Contact: Christoph Hellwig <hch@lst.de>
124+Description:
125+ Namespace attributes added with the initial NVMe target.
126+
127+ device_path: Shows or sets the backend block device path.
128+ The namespace must be disabled before modification.
129+
130+ device_nguid: Shows or sets the NGUID (128-bit identifier).
131+ Accepts 32 hex digits with optional "-" or ":" separators.
132+ The namespace must be disabled before modification.
133+
134+ enable: Shows or sets whether this namespace is enabled
135+ (boolean).
136+
137+What: /config/nvmet/subsystems/NAME/namespaces/NSID/device_uuid
138+Date: June 2017
139+KernelVersion: 4.13
140+Contact: Johannes Thumshirn <jthumshirn@suse.de>
141+Description:
142+ Shows or sets the UUID for this namespace. The namespace
143+ must be disabled before modification.
144+
145+What: /config/nvmet/subsystems/NAME/attr_version
146+What: /config/nvmet/subsystems/NAME/attr_serial
147+Date: July 2017
148+KernelVersion: 4.13
149+Contact: Johannes Thumshirn <jthumshirn@suse.de>
150+Description:
151+ attr_version: Shows or sets the NVMe version reported by
152+ this subsystem. Format: "major.minor" or
153+ "major.minor.tertiary". Cannot be changed after the
154+ subsystem has been discovered.
155+
156+ attr_serial: Shows or sets the serial number. Must be a
157+ 1-20 byte ASCII string (characters 0x20-0x7e). Cannot be
158+ changed after the subsystem has been discovered.
159+
160+What: /config/nvmet/subsystems/NAME/namespaces/NSID/ana_grpid
161+Date: June 2018
162+KernelVersion: 4.19
163+Contact: Christoph Hellwig <hch@lst.de>
164+Description:
165+ Shows or sets the ANA (Asymmetric Namespace Access) Group
166+ ID for this namespace. Must be between 1 and 128. Changing
167+ triggers an ANA event notification.
168+
169+What: /config/nvmet/subsystems/NAME/namespaces/NSID/buffered_io
170+Date: June 2018
171+KernelVersion: 4.19
172+Contact: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
173+Description:
174+ Shows or sets whether buffered I/O is used for this
175+ namespace. Accepts boolean value. The namespace must be
176+ disabled before modification.
177+
178+What: /config/nvmet/subsystems/NAME/namespaces/NSID/p2pmem
179+Date: October 2018
180+KernelVersion: 4.20
181+Contact: Logan Gunthorpe <logang@deltatee.com>
182+Description:
183+ Shows or sets the P2P DMA memory device for this namespace.
184+ Accepts a PCI device BDF, "auto", or "none". The namespace
185+ must be disabled before modification. Only available when
186+ CONFIG_PCI_P2PDMA is enabled.
187+
188+What: /config/nvmet/subsystems/NAME/attr_cntlid_min
189+What: /config/nvmet/subsystems/NAME/attr_cntlid_max
190+Date: January 2020
191+KernelVersion: 5.7
192+Contact: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
193+Description:
194+ attr_cntlid_min: Shows or sets the minimum controller ID
195+ (u16). Must be nonzero and not greater than attr_cntlid_max.
196+
197+ attr_cntlid_max: Shows or sets the maximum controller ID
198+ (u16). Must be nonzero and not less than attr_cntlid_min.
199+
200+What: /config/nvmet/subsystems/NAME/attr_model
201+Date: January 2020
202+KernelVersion: 5.7
203+Contact: Mark Ruijter <MRuijter@onestopsystems.com>
204+Description:
205+ Shows or sets the model number for this subsystem. Must
206+ be a 1-40 byte ASCII string (characters 0x20-0x7e).
207+ Cannot be changed after the subsystem has been discovered.
208+
209+What: /config/nvmet/subsystems/NAME/attr_pi_enable
210+Date: May 2020
211+KernelVersion: 5.8
212+Contact: Israel Rukshin <israelr@mellanox.com>
213+Description:
214+ Shows or sets whether protection information (PI) is
215+ enabled/supported for this subsystem. Accepts boolean
216+ value. Only available when CONFIG_BLK_DEV_INTEGRITY is
217+ enabled.
218+
219+What: /config/nvmet/subsystems/NAME/namespaces/NSID/revalidate_size
220+Date: May 2020
221+KernelVersion: 5.8
222+Contact: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
223+Description:
224+ Write-only. Writing 1 triggers namespace size revalidation.
225+ If the size has changed, a namespace changed AEN is sent.
226+ The namespace must be enabled.
227+
228+What: /config/nvmet/subsystems/NAME/passthru/device_path
229+What: /config/nvmet/subsystems/NAME/passthru/enable
230+Date: July 2020
231+KernelVersion: 5.9
232+Contact: Logan Gunthorpe <logang@deltatee.com>
233+Description:
234+ Passthrough mode attributes.
235+
236+ device_path: Shows or sets the NVMe controller character
237+ device path (e.g., /dev/nvme0). Cannot be changed while
238+ the passthrough controller is active.
239+
240+ enable: Shows or sets whether passthrough mode is enabled
241+ (boolean).
242+
243+ Only available when CONFIG_NVME_TARGET_PASSTHRU is enabled.
244+
245+What: /config/nvmet/subsystems/NAME/passthru/admin_timeout
246+What: /config/nvmet/subsystems/NAME/passthru/io_timeout
247+Date: November 2020
248+KernelVersion: 5.11
249+Contact: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
250+Description:
251+ admin_timeout: Shows or sets the admin command timeout for
252+ passthrough mode, in jiffies.
253+
254+ io_timeout: Shows or sets the I/O command timeout for
255+ passthrough mode, in jiffies.
256+
257+ Only available when CONFIG_NVME_TARGET_PASSTHRU is enabled.
258+
259+What: /config/nvmet/subsystems/NAME/passthru/clear_ids
260+Date: June 2022
261+KernelVersion: 5.19
262+Contact: Alan Adamson <alan.adamson@oracle.com>
263+Description:
264+ Shows or sets whether to clear identify data IDs in
265+ passthrough mode. Only available when
266+ CONFIG_NVME_TARGET_PASSTHRU is enabled.
267+
268+What: /config/nvmet/subsystems/NAME/attr_qid_max
269+Date: August 2022
270+KernelVersion: 6.1
271+Contact: Daniel Wagner <dwagner@suse.de>
272+Description:
273+ Shows or sets the maximum queue ID (number of I/O queues,
274+ u16). Must be between 1 and 128. Changing this value
275+ forces reconnection of all connected controllers.
276+
277+What: /config/nvmet/subsystems/NAME/attr_ieee_oui
278+Date: November 2022
279+KernelVersion: 6.2
280+Contact: Aleksandr Miloserdov <a.miloserdov@yadro.com>
281+Description:
282+ Shows or sets the IEEE OUI for this subsystem. Displayed
283+ in "0x%06x" format. Must be a 24-bit value. Cannot be
284+ changed after the subsystem has been discovered.
285+
286+What: /config/nvmet/subsystems/NAME/attr_firmware
287+Date: November 2022
288+KernelVersion: 6.2
289+Contact: Aleksandr Miloserdov <a.miloserdov@yadro.com>
290+Description:
291+ Shows or sets the firmware revision string for this
292+ subsystem. Must be a 1-8 byte ASCII string (characters
293+ 0x20-0x7e). Cannot be changed after the subsystem has
294+ been discovered.
295+
296+What: /config/nvmet/subsystems/NAME/namespaces/NSID/resv_enable
297+Date: November 2024
298+KernelVersion: 6.13
299+Contact: Guixin Liu <kanie@linux.alibaba.com>
300+Description:
301+ Shows or sets whether persistent reservation support is
302+ enabled for this namespace. Accepts boolean value. The
303+ namespace must be disabled before modification.
304+
305+What: /config/nvmet/subsystems/NAME/attr_vendor_id
306+What: /config/nvmet/subsystems/NAME/attr_subsys_vendor_id
307+Date: January 2025
308+KernelVersion: 6.14
309+Contact: Damien Le Moal <dlemoal@kernel.org>
310+Description:
311+ attr_vendor_id: Shows or sets the PCI vendor ID reported
312+ by this subsystem. Displayed in "0x%x" format.
313+
314+ attr_subsys_vendor_id: Shows or sets the PCI subsystem
315+ vendor ID. Displayed in "0x%x" format.
316+
317+What: /config/nvmet/hosts/HOSTNQN/dhchap_key
318+What: /config/nvmet/hosts/HOSTNQN/dhchap_ctrl_key
319+What: /config/nvmet/hosts/HOSTNQN/dhchap_hash
320+What: /config/nvmet/hosts/HOSTNQN/dhchap_dhgroup
321+Date: June 2022
322+KernelVersion: 6.0
323+Contact: Hannes Reinecke <hare@suse.de>
324+Description:
325+ DH-HMAC-CHAP authentication attributes.
326+
327+ dhchap_key: Shows or sets the host secret key. Accepts a
328+ key string in "DHHC-1:" format.
329+
330+ dhchap_ctrl_key: Shows or sets the controller secret key
331+ for bidirectional authentication. Same format as dhchap_key.
332+
333+ dhchap_hash: Shows or sets the HMAC hash algorithm.
334+ Accepted values: "hmac(sha256)", "hmac(sha384)",
335+ "hmac(sha512)".
336+
337+ dhchap_dhgroup: Shows or sets the Diffie-Hellman group for
338+ DH-HMAC-CHAP key exchange. Accepted values: "null",
339+ "ffdhe2048", "ffdhe3072", "ffdhe4096", "ffdhe6144".
340+ Non-null groups require the corresponding KPP crypto
341+ algorithm to be available.
342+
343+ Only available when CONFIG_NVME_TARGET_AUTH is enabled.
344+
345+What: /config/nvmet/discovery_nqn
346+Date: April 2024
347+KernelVersion: 6.9
348+Contact: Hannes Reinecke <hare@kernel.org>
349+Description:
350+ Shows or sets the NQN of the discovery subsystem. The
351+ value must be unique and not duplicate any existing
352+ subsystem name.
Documentation/ABI/stable/sysfs-nvme+453−0
@@ -0,0 +1,453 @@
1+What: /sys/class/nvme/nvmeX/model
2+What: /sys/class/nvme/nvmeX/serial
3+What: /sys/class/nvme/nvmeX/firmware_rev
4+Date: January 2016
5+KernelVersion: 4.5
6+Contact: Keith Busch <kbusch@kernel.org>
7+Description:
8+ Shows the model, serial number, or firmware revision string
9+ of the NVMe controller, as reported in the Identify
10+ Controller data structure.
11+
12+What: /sys/class/nvme/nvmeX/cntlid
13+Date: February 2016
14+KernelVersion: 4.6
15+Contact: Ming Lin <ming.l@ssi.samsung.com>
16+Description:
17+ Shows the controller identifier assigned by the NVMe
18+ subsystem.
19+
20+What: /sys/class/nvme/nvmeX/cntrltype
21+What: /sys/class/nvme/nvmeX/dctype
22+Date: February 2022
23+KernelVersion: 5.18
24+Contact: Martin Belanger <martin.belanger@dell.com>
25+Description:
26+ cntrltype: Shows the controller type. Possible values: "io",
27+ "discovery", "admin", "reserved".
28+
29+ dctype: Shows the discovery controller type. Possible values:
30+ "none", "ddc", "cdc", "reserved".
31+
32+What: /sys/class/nvme/nvmeX/reset_controller
33+Date: November 2015
34+KernelVersion: 4.5
35+Contact: Christoph Hellwig <hch@lst.de>
36+Description:
37+ Write-only. Writing any value triggers a synchronous
38+ controller reset.
39+
40+What: /sys/class/nvme/nvmeX/rescan_controller
41+Date: April 2016
42+KernelVersion: 4.7
43+Contact: Keith Busch <kbusch@kernel.org>
44+Description:
45+ Write-only. Writing any value triggers a namespace rescan
46+ on this controller.
47+
48+What: /sys/class/nvme/nvmeX/transport
49+What: /sys/class/nvme/nvmeX/subsysnqn
50+What: /sys/class/nvme/nvmeX/address
51+What: /sys/class/nvme/nvmeX/delete_controller
52+What: /sys/class/nvme/nvmeX/reconnect_delay
53+What: /sys/class/nvme/nvmeX/ctrl_loss_tmo
54+Date: June 2016
55+KernelVersion: 4.8
56+Contact: Ming Lin <ming.l@ssi.samsung.com>
57+Description:
58+ Fabrics controller attributes added with NVMe-oF support.
59+
60+ transport: Shows the transport type string. Possible values:
61+ "pcie", "tcp", "rdma", "fc", "loop".
62+
63+ subsysnqn: Shows the NVMe Qualified Name (NQN) of the
64+ subsystem this controller belongs to.
65+
66+ address: Shows the transport-specific address string. Only
67+ available for fabrics controllers.
68+
69+ delete_controller: Write-only. Triggers deletion of this
70+ fabrics controller.
71+
72+ reconnect_delay: Shows or sets the reconnect delay in
73+ seconds. Reading returns the delay value, or "off" if
74+ disabled.
75+
76+ ctrl_loss_tmo: Shows or sets the controller loss timeout in
77+ seconds. Reading returns the timeout value, or "off" if
78+ infinite reconnects are allowed. Writing a negative value
79+ disables the timeout.
80+
81+What: /sys/class/nvme/nvmeX/hostnqn
82+What: /sys/class/nvme/nvmeX/hostid
83+Date: February 2020
84+KernelVersion: 5.7
85+Contact: Sagi Grimberg <sagi@grimberg.me>
86+Description:
87+ hostnqn: Shows the host NQN used by this fabrics controller.
88+
89+ hostid: Shows the host identifier (UUID format) used by this
90+ fabrics controller.
91+
92+ Only available for fabrics controllers.
93+
94+What: /sys/class/nvme/nvmeX/fast_io_fail_tmo
95+Date: November 2020
96+KernelVersion: 5.11
97+Contact: Victor Gladkov <Victor.Gladkov@kioxia.com>
98+Description:
99+ Shows or sets the fast I/O fail timeout in seconds. Reading
100+ returns the timeout value, or "off" if disabled. Writing a
101+ negative value disables the fast I/O fail. Only available
102+ for fabrics controllers.
103+
104+What: /sys/class/nvme/nvmeX/kato
105+Date: April 2021
106+KernelVersion: 5.13
107+Contact: Hannes Reinecke <hare@suse.de>
108+Description:
109+ Shows the Keep Alive Timeout value in milliseconds for
110+ this controller.
111+
112+What: /sys/class/nvme/nvmeX/cmb
113+Date: October 2016
114+KernelVersion: 4.9
115+Contact: Stephen Bates <sbates@raithlin.com>
116+Description:
117+ Shows the Controller Memory Buffer (CMB) register values
118+ in format "cmbloc : 0x%08x\ncmbsz : 0x%08x\n". Only
119+ visible when the controller has a CMB (cmbsz != 0).
120+ PCI transport only.
121+
122+What: /sys/class/nvme/nvmeX/cmbloc
123+What: /sys/class/nvme/nvmeX/cmbsz
124+What: /sys/class/nvme/nvmeX/hmb
125+Date: July 2021
126+KernelVersion: 5.15
127+Contact: Keith Busch <kbusch@kernel.org>
128+Description:
129+ cmbloc: Shows the CMBLOC register value.
130+
131+ cmbsz: Shows the CMBSZ register value.
132+
133+ cmbloc and cmbsz are only visible when the controller has
134+ a CMB. PCI transport only.
135+
136+ hmb: Shows or sets whether the Host Memory Buffer (HMB) is
137+ enabled. Reading returns 1 (enabled) or 0 (disabled).
138+ Writing 1 enables HMB; writing 0 disables it. Only
139+ visible when the controller supports HMB (hmpre != 0).
140+ PCI transport only.
141+
142+What: /sys/class/nvme/nvmeX/state
143+Date: November 2016
144+KernelVersion: 4.11
145+Contact: Sagi Grimberg <sagi@grimberg.me>
146+Description:
147+ Shows the current state of the controller. Possible values:
148+ "new", "live", "resetting", "connecting", "deleting",
149+ "deleting (no IO)", "dead".
150+
151+What: /sys/class/nvme/nvmeX/numa_node
152+Date: November 2018
153+KernelVersion: 5.0
154+Contact: Hannes Reinecke <hare@suse.de>
155+Description:
156+ Shows the NUMA node the controller is attached to.
157+
158+What: /sys/class/nvme/nvmeX/queue_count
159+What: /sys/class/nvme/nvmeX/sqsize
160+Date: September 2019
161+KernelVersion: 5.4
162+Contact: James Smart <jsmart2021@gmail.com>
163+Description:
164+ queue_count: Shows the total number of queues (admin + I/O)
165+ for this controller.
166+
167+ sqsize: Shows the submission queue size for this controller.
168+
169+What: /sys/class/nvme/nvmeX/dhchap_secret
170+What: /sys/class/nvme/nvmeX/dhchap_ctrl_secret
171+Date: June 2022
172+KernelVersion: 6.0
173+Contact: Hannes Reinecke <hare@suse.de>
174+Description:
175+ dhchap_secret: Shows or sets the host DH-HMAC-CHAP secret
176+ for this controller. Reading returns "none" if not set.
177+ Writing must use the "DHHC-1:" key format and triggers
178+ re-authentication.
179+
180+ dhchap_ctrl_secret: Shows or sets the controller
181+ DH-HMAC-CHAP secret for bidirectional authentication.
182+ Same format as dhchap_secret.
183+
184+ Only available when CONFIG_NVME_HOST_AUTH is enabled and
185+ for fabrics controllers.
186+
187+What: /sys/class/nvme/nvmeX/tls_key
188+Date: August 2023
189+KernelVersion: 6.7
190+Contact: Hannes Reinecke <hare@suse.de>
191+Description:
192+ Shows the serial of the currently active TLS PSK as hex.
193+ Returns empty if no TLS key is active. Only available for
194+ TCP controllers with TLS or secure concatenation enabled
195+ (CONFIG_NVME_TCP_TLS).
196+
197+What: /sys/class/nvme/nvmeX/tls_configured_key
198+Date: July 2024
199+KernelVersion: 6.12
200+Contact: Hannes Reinecke <hare@kernel.org>
201+Description:
202+ Shows the serial of the configured TLS key. Writing 0
203+ triggers a PSK reauthentication (REPLACETLSPSK) with
204+ the target. After reauthentication the returned serial
205+ will be the new key. Only available for TCP controllers
206+ with secure concatenation enabled (CONFIG_NVME_TCP_TLS).
207+
208+What: /sys/class/nvme/nvmeX/tls_keyring
209+Date: July 2024
210+KernelVersion: 6.12
211+Contact: Hannes Reinecke <hare@kernel.org>
212+Description:
213+ Shows the TLS keyring description. Only available for TCP
214+ controllers with a keyring configured (CONFIG_NVME_TCP_TLS).
215+
216+What: /sys/class/nvme/nvmeX/tls_mode
217+Date: April 2026
218+KernelVersion: 7.1
219+Contact: Daniel Wagner <wagi@kernel.org>
220+Description:
221+ Shows the TLS mode: "tls" for direct TLS or "concat" for
222+ secure concatenation. Only available for TCP controllers
223+ with TLS or secure concatenation enabled
224+ (CONFIG_NVME_TCP_TLS).
225+
226+What: /sys/class/nvme/nvmeX/passthru_err_log_enabled
227+Date: January 2024
228+KernelVersion: 6.8
229+Contact: Alan Adamson <alan.adamson@oracle.com>
230+Description:
231+ Shows or sets whether admin passthrough error logging is
232+ enabled for this controller. Reading returns "on" or "off".
233+ Writing accepts a boolean value.
234+
235+What: /sys/class/nvme/nvmeX/quirks
236+Date: November 2025
237+KernelVersion: 7.0
238+Contact: Maurizio Lombardi <mlombard@redhat.com>
239+Description:
240+ Shows the active quirk names for this controller, one per
241+ line. Shows "none" if no quirks are active.
242+
243+What: /sys/class/nvme/nvmeX/admin_timeout
244+What: /sys/class/nvme/nvmeX/io_timeout
245+Date: May 2026
246+KernelVersion: 7.2
247+Contact: Maurizio Lombardi <mlombard@redhat.com>
248+Description:
249+ admin_timeout: Shows or sets the admin command timeout in
250+ milliseconds.
251+
252+ io_timeout: Shows or sets the I/O command timeout in
253+ milliseconds. Changes are propagated to all namespace
254+ request queues.
255+
256+ The value must be nonzero. Only writable after the
257+ controller has been started at least once.
258+
259+What: /sys/block/nvmeXnY/uuid
260+What: /sys/block/nvmeXnY/eui
261+What: /sys/block/nvmeXnY/nsid
262+Date: December 2015
263+KernelVersion: 4.5
264+Contact: Keith Busch <kbusch@kernel.org>
265+Description:
266+ Namespace identification attributes.
267+
268+ uuid: Shows the UUID for this namespace. Falls back to
269+ showing the NGUID for backward compatibility. Hidden if
270+ both are all zeros.
271+
272+ eui: Shows the IEEE Extended Unique Identifier (EUI-64).
273+ Hidden if all zeros.
274+
275+ nsid: Shows the namespace identifier (NSID).
276+
277+What: /sys/block/nvmeXnY/wwid
278+Date: February 2016
279+KernelVersion: 4.6
280+Contact: Keith Busch <kbusch@kernel.org>
281+Description:
282+ Shows the World Wide Identifier for this namespace. The
283+ format depends on available identifiers (in priority
284+ order): "uuid.{UUID}", "eui.{NGUID}", "eui.{EUI64}", or
285+ "nvme.{VID}-{SERIAL}-{MODEL}-{NSID}".
286+
287+What: /sys/block/nvmeXnY/nguid
288+Date: June 2017
289+KernelVersion: 4.13
290+Contact: Johannes Thumshirn <jthumshirn@suse.de>
291+Description:
292+ Shows the Namespace Globally Unique Identifier (NGUID).
293+ Hidden if the NGUID is all zeros.
294+
295+What: /sys/block/nvmeXcYnZ/ana_grpid
296+What: /sys/block/nvmeXcYnZ/ana_state
297+Date: May 2018
298+KernelVersion: 4.19
299+Contact: Christoph Hellwig <hch@lst.de>
300+Description:
301+ ana_grpid: Shows the ANA Group ID for this namespace
302+ path device.
303+
304+ ana_state: Shows the ANA state. Possible values:
305+ "optimized", "non-optimized", "inaccessible",
306+ "persistent-loss", "change".
307+
308+ Only visible when the controller supports ANA.
309+ Requires CONFIG_NVME_MULTIPATH.
310+
311+What: /sys/block/nvmeXcYnZ/queue_depth
312+Date: June 2024
313+KernelVersion: 6.11
314+Contact: Thomas Song <tsong@purestorage.com>
315+Description:
316+ Shows the current active I/O count on this path's
317+ controller. Returns empty if iopolicy is not "queue-depth".
318+ Requires CONFIG_NVME_MULTIPATH.
319+
320+What: /sys/block/nvmeXcYnZ/numa_nodes
321+Date: January 2025
322+KernelVersion: 6.15
323+Contact: Nilay Shroff <nilay@linux.ibm.com>
324+Description:
325+ Shows the NUMA node mask for which this path is the
326+ currently selected path. Returns empty if iopolicy is not
327+ "numa". Requires CONFIG_NVME_MULTIPATH.
328+
329+What: /sys/block/nvmeXnY/delayed_removal_secs
330+Date: May 2025
331+KernelVersion: 6.16
332+Contact: Nilay Shroff <nilay@linux.ibm.com>
333+Description:
334+ Shows or sets the delayed removal timeout in seconds for
335+ the multipath head device. When nonzero, I/O is queued
336+ instead of failed when all paths are gone, and head removal
337+ is deferred. Only visible on multipath head devices.
338+ Requires CONFIG_NVME_MULTIPATH.
339+
340+What: /sys/block/nvmeXnY/csi
341+What: /sys/block/nvmeXnY/metadata_bytes
342+What: /sys/block/nvmeXnY/nuse
343+Date: December 2023
344+KernelVersion: 6.8
345+Contact: Daniel Wagner <dwagner@suse.de>
346+Description:
347+ csi: Shows the Command Set Identifier for this namespace.
348+
349+ metadata_bytes: Shows the metadata size in bytes.
350+
351+ nuse: Shows the Namespace Utilization (NUSE) value. Reading
352+ triggers an Identify Namespace command to refresh the
353+ value (rate-limited to avoid excessive commands).
354+
355+What: /sys/block/nvmeXnY/passthru_err_log_enabled
356+Date: January 2024
357+KernelVersion: 6.8
358+Contact: Alan Adamson <alan.adamson@oracle.com>
359+Description:
360+ Shows or sets whether I/O passthrough error logging is
361+ enabled for this namespace. Reading returns "on" or "off".
362+ Writing accepts a boolean value.
363+
364+What: /sys/class/nvme/nvmeX/diag/command_error_count
365+What: /sys/class/nvme/nvmeX/diag/reset_count
366+What: /sys/class/nvme/nvmeX/diag/reconnect_count
367+Date: May 2026
368+KernelVersion: 7.2
369+Contact: Nilay Shroff <nilay@linux.ibm.com>
370+Description:
371+ Controller diagnostic counters.
372+
373+ command_error_count: Admin command error counter.
374+
375+ reset_count: Controller reset counter.
376+
377+ reconnect_count: Accumulated reconnect counter. Only
378+ available for fabrics controllers.
379+
380+ All counters can be reset by writing a value.
381+
382+What: /sys/block/nvmeXnY/diag/command_retries_count
383+What: /sys/block/nvmeXnY/diag/command_error_count
384+Date: May 2026
385+KernelVersion: 7.2
386+Contact: Nilay Shroff <nilay@linux.ibm.com>
387+Description:
388+ Namespace diagnostic counters for non-multipath
389+ configurations (when CONFIG_NVME_MULTIPATH is not
390+ configured).
391+
392+ command_retries_count: I/O command retry counter.
393+
394+ command_error_count: I/O command error counter.
395+
396+ All counters can be reset by writing any value.
397+
398+What: /sys/block/nvmeXcYnZ/diag/command_retries_count
399+What: /sys/block/nvmeXcYnZ/diag/command_error_count
400+What: /sys/block/nvmeXcYnZ/diag/multipath_failover_count
401+What: /sys/block/nvmeXnY/diag/io_requeue_no_usable_path_count
402+What: /sys/block/nvmeXnY/diag/io_fail_no_available_path_count
403+Date: May 2026
404+KernelVersion: 7.2
405+Contact: Nilay Shroff <nilay@linux.ibm.com>
406+Description:
407+ Namespace diagnostic counters for multipath
408+ configurations (when CONFIG_NVME_MULTIPATH is
409+ configured).
410+
411+ command_retries_count: I/O command retry counter.
412+
413+ command_error_count: I/O command error counter.
414+
415+ multipath_failover_count: Multipath failover counter.
416+
417+ io_requeue_no_usable_path_count: Counter of I/Os
418+ requeued because no usable path was available.
419+
420+ io_fail_no_available_path_count: Counter of I/Os
421+ failed because no available path existed.
422+
423+ All counters can be reset by writing any value.
424+
425+What: /sys/class/nvme-subsystem/nvme-subsysX/model
426+What: /sys/class/nvme-subsystem/nvme-subsysX/serial
427+What: /sys/class/nvme-subsystem/nvme-subsysX/firmware_rev
428+What: /sys/class/nvme-subsystem/nvme-subsysX/subsysnqn
429+Date: November 2017
430+KernelVersion: 4.15
431+Contact: Hannes Reinecke <hare@suse.de>
432+Description:
433+ Shows the model, serial number, firmware revision, or NQN
434+ of the NVMe subsystem.
435+
436+What: /sys/class/nvme-subsystem/nvme-subsysX/iopolicy
437+Date: February 2019
438+KernelVersion: 5.1
439+Contact: Hannes Reinecke <hare@suse.de>
440+Description:
441+ Shows or sets the multipath I/O path selection policy for
442+ this subsystem. Accepted values: "numa", "round-robin",
443+ "queue-depth". Changing the policy clears all current path
444+ selections. Only available when CONFIG_NVME_MULTIPATH is
445+ enabled.
446+
447+What: /sys/class/nvme-subsystem/nvme-subsysX/subsystype
448+Date: September 2021
449+KernelVersion: 5.16
450+Contact: Hannes Reinecke <hare@suse.de>
451+Description:
452+ Shows the subsystem type. Possible values: "discovery",
453+ "nvm", "reserved".
Documentation/ABI/testing/sysfs-nvme+0−13
@@ -1,13 +0,0 @@
1−What: /sys/devices/virtual/nvme-fabrics/ctl/.../tls_configured_key
2−Date: November 2025
3−KernelVersion: 6.19
4−Contact: Linux NVMe mailing list <linux-nvme@lists.infradead.org>
5−Description:
6− The file is avaliable when using a secure concatanation
7− connection to a NVMe target. Reading the file will return
8− the serial of the currently negotiated key.
9−
10− Writing 0 to the file will trigger a PSK reauthentication
11− (REPLACETLSPSK) with the target. After a reauthentication
12− the value returned by tls_configured_key will be the new
13− serial.
MAINTAINERS+4−0
@@ -19373,6 +19373,9 @@ L: linux-nvme@lists.infradead.org · 19372 unmodified lines
1937319373 S: Supported
1937419374 W: http://git.infradead.org/nvme.git
1937519375 T: git git://git.infradead.org/nvme.git
19376+F: Documentation/ABI/stable/sysfs-nvme
19377+F: Documentation/admin-guide/nvme-multipath.rst
19378+F: Documentation/fault-injection/nvme-fault-injection.rst
1937619379 F: Documentation/nvme/
1937719380 F: drivers/nvme/common/
1937819381 F: drivers/nvme/host/
@@ -19415,6 +19418,7 @@ L: linux-nvme@lists.infradead.org · 36 unmodified lines
1941519418 S: Supported
1941619419 W: http://git.infradead.org/nvme.git
1941719420 T: git git://git.infradead.org/nvme.git
19421+F: Documentation/ABI/stable/configfs-nvmet
1941819422 F: drivers/nvme/target/
1941919423
1942019424 NVMEM FRAMEWORK
drivers/nvme/common/auth.c+6−7
@@ -692,8 +692,7 @@ int nvme_auth_derive_tls_psk(int hmac_id, const u8 *psk, size_t psk_len, · 691 unmodified lines
692692 const char *psk_digest, u8 **ret_psk)
693693 {
694694 static const u8 default_salt[NVME_AUTH_MAX_DIGEST_SIZE];
695− static const char label[] = "tls13 nvme-tls-psk";
696− const size_t label_len = sizeof(label) - 1;
695+ static const char label[18] = "tls13 nvme-tls-psk";
697696 u8 prk[NVME_AUTH_MAX_DIGEST_SIZE];
698697 size_t hash_len, ctx_len;
699698 u8 *hmac_data = NULL, *tls_key;
@@ -729,7 +728,7 @@ int nvme_auth_derive_tls_psk(int hmac_id, const u8 *psk, size_t psk_len, · 29 unmodified lines
729728 */
730729
731730 hmac_data = kmalloc(/* output length */ 2 +
732− /* label */ 1 + label_len +
731+ /* label */ 1 + sizeof(label) +
733732 /* context (max) */ 1 + 3 + 1 + strlen(psk_digest) +
734733 /* counter */ 1,
735734 GFP_KERNEL);
@@ -743,10 +742,10 @@ int nvme_auth_derive_tls_psk(int hmac_id, const u8 *psk, size_t psk_len, · 7 unmodified lines
743742 hmac_data[i++] = hash_len;
744743
745744 /* label */
746− static_assert(label_len <= 255);
747− hmac_data[i] = label_len;
748− memcpy(&hmac_data[i + 1], label, label_len);
749− i += 1 + label_len;
745+ static_assert(sizeof(label) <= 255);
746+ hmac_data[i] = sizeof(label);
747+ memcpy(&hmac_data[i + 1], label, sizeof(label));
748+ i += 1 + sizeof(label);
750749
751750 /* context */
752751 ctx_len = sprintf(&hmac_data[i + 1], "%02d %s", hmac_id, psk_digest);
drivers/nvme/host/Makefile+1−0
@@ -1,5 +1,6 @@
11 # SPDX-License-Identifier: GPL-2.0
22
3+CONTEXT_ANALYSIS := y
34 ccflags-y += -I$(src)
45
56 obj-$(CONFIG_NVME_CORE) += nvme-core.o
drivers/nvme/host/apple.c+39−27
@@ -47,9 +47,6 @@ · 46 unmodified lines
4747 #define APPLE_ANS_BOOT_STATUS 0x1300
4848 #define APPLE_ANS_BOOT_STATUS_OK 0xde71ce55
4949
50−#define APPLE_ANS_UNKNOWN_CTRL 0x24008
51−#define APPLE_ANS_PRP_NULL_CHECK BIT(11)
52−
5350 #define APPLE_ANS_LINEAR_SQ_CTRL 0x24908
5451 #define APPLE_ANS_LINEAR_SQ_EN BIT(0)
5552
@@ -151,6 +148,23 @@ struct apple_nvme_queue { · 95 unmodified lines
151148 bool enabled;
152149 };
153150
151+static inline bool apple_nvme_queue_enabled(struct apple_nvme_queue *q)
152+{
153+ /* Pair with apple_nvme_enable_queue(). */
154+ return smp_load_acquire(&q->enabled);
155+}
156+
157+static inline void apple_nvme_enable_queue(struct apple_nvme_queue *q)
158+{
159+ /* Publish queue initialization before setting q->enabled. */
160+ smp_store_release(&q->enabled, true);
161+}
162+
163+static inline void apple_nvme_disable_queue(struct apple_nvme_queue *q)
164+{
165+ WRITE_ONCE(q->enabled, false);
166+}
167+
154168 /*
155169 * The apple_nvme_iod describes the data in an I/O.
156170 *
@@ -318,13 +332,15 @@ static void apple_nvme_submit_cmd_t8103(struct apple_nvme_queue *q, · 161 unmodified lines
318332 u32 tag = nvme_tag_from_cid(cmd->common.command_id);
319333 struct apple_nvmmu_tcb *tcb = &q->tcbs[tag];
320334
321− tcb->opcode = cmd->common.opcode;
335+ tcb->opcode = 0;
322336 tcb->prp1 = cmd->common.dptr.prp1;
323337 tcb->prp2 = cmd->common.dptr.prp2;
324338 tcb->length = cmd->rw.length;
325339 tcb->command_id = tag;
326340
327− if (nvme_is_write(cmd))
341+ if (!cmd->common.dptr.prp1)
342+ tcb->dma_flags = 0;
343+ else if (nvme_is_write(cmd))
328344 tcb->dma_flags = APPLE_ANS_TCB_DMA_TO_DEVICE;
329345 else
330346 tcb->dma_flags = APPLE_ANS_TCB_DMA_FROM_DEVICE;
@@ -677,7 +693,7 @@ static bool apple_nvme_handle_cq(struct apple_nvme_queue *q, bool force) · 346 unmodified lines
677693 bool found;
678694 DEFINE_IO_COMP_BATCH(iob);
679695
680− if (!READ_ONCE(q->enabled) && !force)
696+ if (!apple_nvme_queue_enabled(q) && !force)
681697 return false;
682698
683699 found = apple_nvme_poll_cq(q, &iob);
@@ -780,7 +796,7 @@ static blk_status_t apple_nvme_queue_rq(struct blk_mq_hw_ctx *hctx, · 96 unmodified lines
780796 * We should not need to do this, but we're still using this to
781797 * ensure we can drain requests on a dying queue.
782798 */
783− if (unlikely(!READ_ONCE(q->enabled)))
799+ if (unlikely(!apple_nvme_queue_enabled(q)))
784800 return BLK_STS_IOERR;
785801
786802 if (!nvme_check_ready(&anv->ctrl, req, true))
@@ -863,7 +879,7 @@ static void apple_nvme_disable(struct apple_nvme *anv, bool shutdown) · 76 unmodified lines
863879 nvme_quiesce_io_queues(&anv->ctrl);
864880
865881 if (!dead) {
866− if (READ_ONCE(anv->ioq.enabled)) {
882+ if (apple_nvme_queue_enabled(&anv->ioq)) {
867883 apple_nvme_remove_sq(anv);
868884 apple_nvme_remove_cq(anv);
869885 }
@@ -887,8 +903,8 @@ static void apple_nvme_disable(struct apple_nvme *anv, bool shutdown) · 17 unmodified lines
887903 nvme_disable_ctrl(&anv->ctrl, false);
888904 }
889905
890− WRITE_ONCE(anv->ioq.enabled, false);
891− WRITE_ONCE(anv->adminq.enabled, false);
906+ apple_nvme_disable_queue(&anv->ioq);
907+ apple_nvme_disable_queue(&anv->adminq);
892908 mb(); /* ensure that nvme_queue_rq() sees that enabled is cleared */
893909 nvme_quiesce_admin_queue(&anv->ctrl);
894910
@@ -1016,8 +1032,7 @@ static void apple_nvme_init_queue(struct apple_nvme_queue *q) · 121 unmodified lines
10161032 memset(q->tcbs, 0, anv->hw->max_queue_depth
10171033 * sizeof(struct apple_nvmmu_tcb));
10181034 memset(q->cqes, 0, depth * sizeof(struct nvme_completion));
1019− WRITE_ONCE(q->enabled, true);
1020− wmb(); /* ensure the first interrupt sees the initialization */
1035+ apple_nvme_enable_queue(q);
10211036 }
10221037
10231038 static void apple_nvme_reset_work(struct work_struct *work)
@@ -1125,17 +1140,6 @@ static void apple_nvme_reset_work(struct work_struct *work) · 101 unmodified lines
11251140 /* Setup the NVMMU for the maximum admin and IO queue depth */
11261141 writel(anv->hw->max_queue_depth - 1,
11271142 anv->mmio_nvme + APPLE_NVMMU_NUM_TCBS);
1128−
1129− /*
1130− * This is probably a chicken bit: without it all commands
1131− * where any PRP is set to zero (including those that don't use
1132− * that field) fail and the co-processor complains about
1133− * "completed with err BAD_CMD-" or a "NULL_PRP_PTR_ERR" in the
1134− * syslog
1135− */
1136− writel(readl(anv->mmio_nvme + APPLE_ANS_UNKNOWN_CTRL) &
1137− ~APPLE_ANS_PRP_NULL_CHECK,
1138− anv->mmio_nvme + APPLE_ANS_UNKNOWN_CTRL);
11391143 }
11401144
11411145 /* Setup the admin queue */
@@ -1567,10 +1571,8 @@ static struct apple_nvme *apple_nvme_alloc(struct platform_device *pdev) · 425 unmodified lines
15671571
15681572 ret = devm_request_irq(anv->dev, anv->irq, apple_nvme_irq, 0,
15691573 "nvme-apple", anv);
1570− if (ret) {
1571− dev_err_probe(dev, ret, "Failed to request IRQ");
1574+ if (ret)
15721575 goto put_dev;
1573− }
15741576
15751577 anv->rtk =
15761578 devm_apple_rtkit_init(dev, anv, NULL, 0, &apple_nvme_rtkit_ops);
@@ -1581,7 +1583,8 @@ static struct apple_nvme *apple_nvme_alloc(struct platform_device *pdev) · 4 unmodified lines
15811583 }
15821584
15831585 ret = nvme_init_ctrl(&anv->ctrl, anv->dev, &nvme_ctrl_ops,
1584− NVME_QUIRK_SKIP_CID_GEN | NVME_QUIRK_IDENTIFY_CNS);
1586+ NVME_QUIRK_SKIP_CID_GEN | NVME_QUIRK_IDENTIFY_CNS |
1587+ NVME_QUIRK_ADMIN_PAGE_ALIGN);
15851588 if (ret) {
15861589 dev_err_probe(dev, ret, "Failed to initialize nvme_ctrl");
15871590 goto put_dev;
@@ -1636,6 +1639,15 @@ static void apple_nvme_remove(struct platform_device *pdev) · 48 unmodified lines
16361639 nvme_stop_ctrl(&anv->ctrl);
16371640 nvme_remove_namespaces(&anv->ctrl);
16381641 apple_nvme_disable(anv, true);
1642+ if (anv->ctrl.admin_q && !blk_queue_dying(anv->ctrl.admin_q)) {
1643+ /*
1644+ * If the controller was reset during removal, it's possible
1645+ * user requests may be waiting on a stopped queue. Start the
1646+ * queue to flush these to completion.
1647+ */
1648+ nvme_unquiesce_admin_queue(&anv->ctrl);
1649+ blk_mq_destroy_queue(anv->ctrl.admin_q);
1650+ }
16391651 nvme_uninit_ctrl(&anv->ctrl);
16401652
16411653 if (apple_rtkit_is_running(anv->rtk)) {
drivers/nvme/host/auth.c+2−1
@@ -8,6 +8,7 @@ · 7 unmodified lines
88 #include <linux/prandom.h>
99 #include <linux/unaligned.h>
1010 #include <crypto/dh.h>
11+#include <crypto/utils.h>
1112 #include "nvme.h"
1213 #include "fabrics.h"
1314 #include <linux/nvme-auth.h>
@@ -361,7 +362,7 @@ static int nvme_auth_process_dhchap_success1(struct nvme_ctrl *ctrl, · 347 unmodified lines
361362 return 0;
362363
363364 /* Validate controller response */
364− if (memcmp(chap->response, data->rval, data->hl)) {
365+ if (crypto_memneq(chap->response, data->rval, data->hl)) {
365366 dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
366367 __func__, chap->qid, (int)chap->hash_len, data->rval);
367368 dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
drivers/nvme/host/core.c+45−7
@@ -33,6 +33,13 @@ · 32 unmodified lines
3333
3434 #define NVME_MINORS (1U << MINORBITS)
3535
36+/*
37+ * Write hints (bio->bi_write_stream) are u8, so FDP placement handles beyond
38+ * U8_MAX can never be selected. Cap the handle count to bound both the RUH
39+ * status buffer and the per-head plids array.
40+ */
41+#define NVME_MAX_PLIDS U8_MAX
42+
3643 struct nvme_ns_info {
3744 struct nvme_ns_ids ids;
3845 u32 nsid;
@@ -126,8 +133,8 @@ EXPORT_SYMBOL_GPL(nvme_reset_wq); · 87 unmodified lines
126133 struct workqueue_struct *nvme_delete_wq;
127134 EXPORT_SYMBOL_GPL(nvme_delete_wq);
128135
129−static LIST_HEAD(nvme_subsystems);
130136 DEFINE_MUTEX(nvme_subsystems_lock);
137+static LIST_HEAD_GUARDED(nvme_subsystems, nvme_subsystems_lock);
131138
132139 static DEFINE_IDA(nvme_instance_ida);
133140 static dev_t nvme_ctrl_base_chr_devt;
@@ -693,6 +700,11 @@ static void nvme_free_ns_head(struct kref *ref) · 559 unmodified lines
693700 kfree(head);
694701 }
695702
703+void nvme_get_ns_head(struct nvme_ns_head *head)
704+{
705+ kref_get(&head->ref);
706+}
707+
696708 bool nvme_tryget_ns_head(struct nvme_ns_head *head)
697709 {
698710 return kref_get_unless_zero(&head->ref);
@@ -1268,7 +1280,7 @@ u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u8 opcode) · 569 unmodified lines
12681280 }
12691281 EXPORT_SYMBOL_NS_GPL(nvme_passthru_start, "NVME_TARGET_PASSTHRU");
12701282
1271−void nvme_passthru_end(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u32 effects,
1283+u32 nvme_passthru_end(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u32 effects,
12721284 struct nvme_command *cmd, int status)
12731285 {
12741286 if (effects & NVME_CMD_EFFECTS_CSE_MASK) {
@@ -1289,7 +1301,7 @@ void nvme_passthru_end(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u32 effects, · 14 unmodified lines
12891301 flush_work(&ctrl->scan_work);
12901302 }
12911303 if (ns)
1292− return;
1304+ return effects;
12931305
12941306 switch (cmd->common.opcode) {
12951307 case nvme_admin_set_features:
@@ -1310,6 +1322,8 @@ void nvme_passthru_end(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u32 effects, · 14 unmodified lines
13101322 default:
13111323 break;
13121324 }
1325+
1326+ return effects;
13131327 }
13141328 EXPORT_SYMBOL_NS_GPL(nvme_passthru_end, "NVME_TARGET_PASSTHRU");
13151329
@@ -1583,8 +1597,12 @@ static int nvme_identify_ns_descs(struct nvme_ctrl *ctrl, · 267 unmodified lines
15831597 for (pos = 0; pos < NVME_IDENTIFY_DATA_SIZE; pos += len) {
15841598 struct nvme_ns_id_desc *cur = data + pos;
15851599
1600+ if (pos + sizeof(*cur) > NVME_IDENTIFY_DATA_SIZE)
1601+ break;
15861602 if (cur->nidl == 0)
15871603 break;
1604+ if (pos + sizeof(*cur) + cur->nidl > NVME_IDENTIFY_DATA_SIZE)
1605+ break;
15881606
15891607 len = nvme_process_ns_desc(ctrl, &info->ids, cur, &csi_seen);
15901608 if (len < 0)
@@ -2071,7 +2089,10 @@ static void nvme_set_ctrl_limits(struct nvme_ctrl *ctrl, · 480 unmodified lines
20712089 lim->max_integrity_segments = ctrl->max_integrity_segments;
20722090 lim->virt_boundary_mask = ctrl->ops->get_virt_boundary(ctrl, is_admin);
20732091 lim->max_segment_size = UINT_MAX;
2074− lim->dma_alignment = 3;
2092+ if (is_admin && (ctrl->quirks & NVME_QUIRK_ADMIN_PAGE_ALIGN))
2093+ lim->dma_alignment = NVME_CTRL_PAGE_SIZE - 1;
2094+ else
2095+ lim->dma_alignment = 3;
20752096 }
20762097
20772098 static bool nvme_update_disk_info(struct nvme_ns *ns, struct nvme_id_ns *id,
@@ -2342,7 +2363,7 @@ static int nvme_query_fdp_info(struct nvme_ns *ns, struct nvme_ns_info *info) · 264 unmodified lines
23422363 if (!info->runs)
23432364 return ret;
23442365
2345− size = struct_size(ruhs, ruhsd, S8_MAX - 1);
2366+ size = struct_size(ruhs, ruhsd, NVME_MAX_PLIDS);
23462367 ruhs = kzalloc(size, GFP_KERNEL);
23472368 if (!ruhs)
23482369 return -ENOMEM;
@@ -2357,7 +2378,7 @@ static int nvme_query_fdp_info(struct nvme_ns *ns, struct nvme_ns_info *info) · 8 unmodified lines
23572378 goto free;
23582379 }
23592380
2360− head->nr_plids = le16_to_cpu(ruhs->nruhsd);
2381+ head->nr_plids = min(le16_to_cpu(ruhs->nruhsd), NVME_MAX_PLIDS);
23612382 if (!head->nr_plids)
23622383 goto free;
23632384
@@ -2592,11 +2613,15 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_ns_info *info) · 228 unmodified lines
25922613 lim.max_write_streams = ns_lim->max_write_streams;
25932614 lim.write_stream_granularity = ns_lim->write_stream_granularity;
25942615 ret = queue_limits_commit_update(ns->head->disk->queue, &lim);
2616+ if (ret)
2617+ goto unfreeze_head_queue;
25952618
25962619 set_capacity_and_notify(ns->head->disk, get_capacity(ns->disk));
25972620 set_disk_ro(ns->head->disk, nvme_ns_is_readonly(ns, info));
25982621 nvme_mpath_revalidate_paths(ns->head);
2622+ ret = nvme_mpath_revalidate_zones(ns->head);
25992623
2624+unfreeze_head_queue:
26002625 blk_mq_unfreeze_queue(ns->head->disk->queue, memflags);
26012626 }
26022627
@@ -3198,6 +3223,7 @@ static void nvme_put_subsystem(struct nvme_subsystem *subsys) · 595 unmodified lines
31983223 }
31993224
32003225 static struct nvme_subsystem *__nvme_find_get_subsystem(const char *subsysnqn)
3226+ __must_hold(&nvme_subsystems_lock)
32013227 {
32023228 struct nvme_subsystem *subsys;
32033229
@@ -3242,6 +3268,7 @@ static inline bool nvme_is_io_ctrl(struct nvme_ctrl *ctrl) · 38 unmodified lines
32423268
32433269 static bool nvme_validate_cntlid(struct nvme_subsystem *subsys,
32443270 struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
3271+ __must_hold(&nvme_subsystems_lock)
32453272 {
32463273 struct nvme_ctrl *tmp;
32473274
@@ -3272,6 +3299,7 @@ static bool nvme_validate_cntlid(struct nvme_subsystem *subsys, · 24 unmodified lines
32723299 }
32733300
32743301 static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
3302+ __context_unsafe(/* initialize unpublished/lock-guarded variables */)
32753303 {
32763304 struct nvme_subsystem *subsys, *found;
32773305 int ret;
@@ -3843,6 +3871,7 @@ static const struct file_operations nvme_dev_fops = { · 565 unmodified lines
38433871
38443872 static struct nvme_ns_head *nvme_find_ns_head(struct nvme_ctrl *ctrl,
38453873 unsigned nsid)
3874+ __must_hold(&ctrl->subsys->lock)
38463875 {
38473876 struct nvme_ns_head *h;
38483877
@@ -3865,6 +3894,7 @@ static struct nvme_ns_head *nvme_find_ns_head(struct nvme_ctrl *ctrl, · 16 unmodified lines
38653894
38663895 static int nvme_subsys_check_duplicate_ids(struct nvme_subsystem *subsys,
38673896 struct nvme_ns_ids *ids)
3897+ __must_hold(&subsys->lock)
38683898 {
38693899 bool has_uuid = !uuid_is_null(&ids->uuid);
38703900 bool has_nguid = memchr_inv(ids->nguid, 0, sizeof(ids->nguid));
@@ -3890,6 +3920,11 @@ static int nvme_subsys_check_duplicate_ids(struct nvme_subsystem *subsys, · 19 unmodified lines
38903920 static void nvme_cdev_rel(struct device *dev)
38913921 {
38923922 ida_free(&nvme_ns_chr_minor_ida, MINOR(dev->devt));
3923+ if (dev->parent->class == &nvme_class)
3924+ nvme_put_ns(container_of(dev, struct nvme_ns, cdev_device));
3925+ else
3926+ nvme_put_ns_head(container_of(dev, struct nvme_ns_head,
3927+ cdev_device));
38933928 }
38943929
38953930 void nvme_cdev_del(struct cdev *cdev, struct device *cdev_device)
@@ -3955,10 +3990,12 @@ static void nvme_add_ns_cdev(struct nvme_ns *ns) · 59 unmodified lines
39553990 snprintf(name, sizeof(name), "ng%dn%d", ns->ctrl->instance,
39563991 ns->head->instance);
39573992
3993+ nvme_get_ns(ns); /* Undone in nvme_cdev_rel() */
39583994 if (nvme_cdev_add(name, &ns->cdev, &ns->cdev_device,
39593995 &nvme_ns_chr_fops, ns->ctrl->ops->module)) {
39603996 dev_err(ns->ctrl->device, "Unable to create the %s device\n",
39613997 name);
3998+ nvme_put_ns(ns);
39623999 return;
39634000 }
39644001 set_bit(NVME_NS_CDEV_LIVE, &ns->flags);
@@ -3966,6 +4003,7 @@ static void nvme_add_ns_cdev(struct nvme_ns *ns) · 1 unmodified line
39664003
39674004 static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
39684005 struct nvme_ns_info *info)
4006+ __must_hold(&ctrl->subsys->lock)
39694007 {
39704008 struct nvme_ns_head *head;
39714009 size_t size = sizeof(*head);
@@ -5195,7 +5233,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev, · 1223 unmodified lines
51955233
51965234 BUILD_BUG_ON(NVME_DSM_MAX_RANGES * sizeof(struct nvme_dsm_range) >
51975235 PAGE_SIZE);
5198− ctrl->discard_page = alloc_page(GFP_KERNEL);
5236+ ctrl->discard_page = alloc_page(GFP_KERNEL | __GFP_ZERO);
51995237 if (!ctrl->discard_page) {
52005238 ret = -ENOMEM;
52015239 goto out;
drivers/nvme/host/fabrics.c+2−2
@@ -14,11 +14,11 @@ · 13 unmodified lines
1414 #include "fabrics.h"
1515 #include <linux/nvme-keyring.h>
1616
17−static LIST_HEAD(nvmf_transports);
1817 static DECLARE_RWSEM(nvmf_transports_rwsem);
18+static LIST_HEAD_GUARDED(nvmf_transports, nvmf_transports_rwsem);
1919
20−static LIST_HEAD(nvmf_hosts);
2120 static DEFINE_MUTEX(nvmf_hosts_mutex);
21+static LIST_HEAD_GUARDED(nvmf_hosts, nvmf_hosts_mutex);
2222
2323 static struct nvmf_host *nvmf_default_host;
2424
drivers/nvme/host/fc.c+7−1
@@ -2100,9 +2100,15 @@ __nvme_fc_init_request(struct nvme_fc_ctrl *ctrl, · 2099 unmodified lines
21002100 dev_err(ctrl->dev,
21012101 "FCP Op failed - rspiu dma mapping failed.\n");
21022102 ret = -EFAULT;
2103+ goto out_unmap;
21032104 }
21042105
21052106 atomic_set(&op->state, FCPOP_STATE_IDLE);
2107+ return 0;
2108+
2109+out_unmap:
2110+ fc_dma_unmap_single(ctrl->lport->dev, op->fcp_req.cmddma,
2111+ sizeof(op->cmd_iu), DMA_TO_DEVICE);
21062112 out_on_error:
21072113 return ret;
21082114 }
@@ -2318,7 +2324,7 @@ nvme_fc_create_hw_io_queues(struct nvme_fc_ctrl *ctrl, u16 qsize) · 209 unmodified lines
23182324 return 0;
23192325
23202326 delete_queues:
2321− for (; i > 0; i--)
2327+ for (--i; i > 0; i--)
23222328 __nvme_fc_delete_hw_queue(ctrl, &ctrl->queues[i], i);
23232329 return ret;
23242330 }
drivers/nvme/host/ioctl.c+79−42
@@ -14,45 +14,54 @@ enum { · 13 unmodified lines
1414 NVME_IOCTL_PARTITION = (1 << 1),
1515 };
1616
17−static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,
18− unsigned int flags, bool open_for_write)
17+static bool nvme_admin_cmd_allowed(struct nvme_ctrl *ctrl,
18+ struct nvme_command *c)
1919 {
20− u32 effects;
21−
22− /*
23− * Do not allow unprivileged passthrough on partitions, as that allows an
24− * escape from the containment of the partition.
25− */
26− if (flags & NVME_IOCTL_PARTITION)
27− goto admin;
28−
29− /*
30− * Do not allow unprivileged processes to send vendor specific or fabrics
31− * commands as we can't be sure about their effects.
32− */
33− if (c->common.opcode >= nvme_cmd_vendor_start ||
34− c->common.opcode == nvme_fabrics_command)
35− goto admin;
36−
3720 /*
3821 * Do not allow unprivileged passthrough of admin commands except
3922 * for a subset of identify commands that contain information required
4023 * to form proper I/O commands in userspace and do not expose any
4124 * potentially sensitive information.
4225 */
43− if (!ns) {
44− if (c->common.opcode == nvme_admin_identify) {
45− switch (c->identify.cns) {
46− case NVME_ID_CNS_NS:
47− case NVME_ID_CNS_CS_NS:
48− case NVME_ID_CNS_NS_CS_INDEP:
49− case NVME_ID_CNS_CS_CTRL:
50− case NVME_ID_CNS_CTRL:
51− return true;
52− }
26+ switch (c->common.opcode) {
27+ case nvme_admin_identify:
28+ switch (c->identify.cns) {
29+ case NVME_ID_CNS_NS:
30+ case NVME_ID_CNS_CS_NS:
31+ case NVME_ID_CNS_NS_CS_INDEP:
32+ case NVME_ID_CNS_CS_CTRL:
33+ case NVME_ID_CNS_CTRL:
34+ return true;
35+ }
36+ break;
37+ case nvme_admin_set_features:
38+ /*
39+ * Reject Set Features that change controller state the driver
40+ * manages itself; setting them behind the driver's back from
41+ * userspace leaves it unable to react correctly. Keep Alive is
42+ * only armed for fabrics - on other transports it has no
43+ * reserved tag and harms idle power states.
44+ */
45+ switch (le32_to_cpu(c->features.fid) & 0xff) {
46+ case NVME_FEAT_KATO:
47+ if (ctrl->ops->flags & NVME_F_FABRICS)
48+ break;
49+ fallthrough;
50+ case NVME_FEAT_HOST_BEHAVIOR:
51+ case NVME_FEAT_HOST_MEM_BUF:
52+ case NVME_FEAT_NUM_QUEUES:
53+ case NVME_FEAT_AUTO_PST:
54+ return false;
5355 }
54− goto admin;
56+ break;
5557 }
58+ return capable(CAP_SYS_ADMIN);
59+}
60+
61+static bool nvme_ns_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,
62+ bool open_for_write)
63+{
64+ u32 effects;
5665
5766 /*
5867 * Check if the controller provides a Commands Supported and Effects log
@@ -61,7 +70,7 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, · 2 unmodified lines
6170 */
6271 effects = nvme_command_effects(ns->ctrl, ns, c->common.opcode);
6372 if (!(effects & NVME_CMD_EFFECTS_CSUPP))
64− goto admin;
73+ return capable(CAP_SYS_ADMIN);
6574
6675 /*
6776 * Don't allow passthrough for command that have intrusive (or unknown)
@@ -70,7 +79,7 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, · 2 unmodified lines
7079 if (effects & ~(NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
7180 NVME_CMD_EFFECTS_UUID_SEL |
7281 NVME_CMD_EFFECTS_SCOPE_MASK))
73− goto admin;
82+ return capable(CAP_SYS_ADMIN);
7483
7584 /*
7685 * Only allow I/O commands that transfer data to the controller or that
@@ -79,11 +88,34 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, · 2 unmodified lines
7988 */
8089 if ((nvme_is_write(c) || (effects & NVME_CMD_EFFECTS_LBCC)) &&
8190 !open_for_write)
82− goto admin;
91+ return capable(CAP_SYS_ADMIN);
8392
8493 return true;
85−admin:
86− return capable(CAP_SYS_ADMIN);
94+}
95+
96+static bool nvme_cmd_allowed(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
97+ struct nvme_command *c, unsigned int flags,
98+ bool open_for_write)
99+{
100+ /*
101+ * Do not allow unprivileged passthrough on partitions, as that
102+ * allows an escape from the containment of the partition.
103+ */
104+ if (flags & NVME_IOCTL_PARTITION)
105+ return capable(CAP_SYS_ADMIN);
106+
107+ /*
108+ * Do not allow unprivileged processes to send vendor specific or
109+ * fabrics commands as we can't be sure about their effects.
110+ */
111+ if (c->common.opcode >= nvme_cmd_vendor_start ||
112+ c->common.opcode == nvme_fabrics_command)
113+ return capable(CAP_SYS_ADMIN);
114+
115+ if (!ns)
116+ return nvme_admin_cmd_allowed(ctrl, c);
117+
118+ return nvme_ns_cmd_allowed(ns, c, open_for_write);
87119 }
88120
89121 /*
@@ -202,7 +234,8 @@ out_free_req: · 112 unmodified lines
202234 return ret;
203235 }
204236
205−static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
237+static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio,
238+ unsigned int flags, bool open_for_write)
206239 {
207240 struct nvme_user_io io;
208241 struct nvme_command c;
@@ -260,6 +293,9 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio) · 51 unmodified lines
260293 c.rw.lbat = cpu_to_le16(io.apptag);
261294 c.rw.lbatm = cpu_to_le16(io.appmask);
262295
296+ if (!nvme_cmd_allowed(ns->ctrl, ns, &c, flags, open_for_write))
297+ return -EACCES;
298+
263299 return nvme_submit_user_cmd(ns->queue, &c, io.addr, length, metadata,
264300 meta_len, NULL, 0, 0);
265301 }
@@ -307,7 +343,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, · 41 unmodified lines
307343 c.common.cdw14 = cpu_to_le32(cmd.cdw14);
308344 c.common.cdw15 = cpu_to_le32(cmd.cdw15);
309345
310− if (!nvme_cmd_allowed(ns, &c, 0, open_for_write))
346+ if (!nvme_cmd_allowed(ctrl, ns, &c, 0, open_for_write))
311347 return -EACCES;
312348
313349 if (cmd.timeout_ms)
@@ -354,7 +390,7 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns, · 40 unmodified lines
354390 c.common.cdw14 = cpu_to_le32(cmd.cdw14);
355391 c.common.cdw15 = cpu_to_le32(cmd.cdw15);
356392
357− if (!nvme_cmd_allowed(ns, &c, flags, open_for_write))
393+ if (!nvme_cmd_allowed(ctrl, ns, &c, flags, open_for_write))
358394 return -EACCES;
359395
360396 if (cmd.timeout_ms)
@@ -449,6 +485,7 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, · 88 unmodified lines
449485 const struct nvme_uring_cmd *cmd = io_uring_sqe128_cmd(ioucmd->sqe,
450486 struct nvme_uring_cmd);
451487 struct request_queue *q = ns ? ns->queue : ctrl->admin_q;
488+ bool open_for_write = ioucmd->file->f_mode & FMODE_WRITE;
452489 struct nvme_uring_data d;
453490 struct nvme_command c;
454491 struct iov_iter iter;
@@ -479,7 +516,7 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, · 24 unmodified lines
479516 c.common.cdw14 = cpu_to_le32(READ_ONCE(cmd->cdw14));
480517 c.common.cdw15 = cpu_to_le32(READ_ONCE(cmd->cdw15));
481518
482− if (!nvme_cmd_allowed(ns, &c, 0, ioucmd->file->f_mode & FMODE_WRITE))
519+ if (!nvme_cmd_allowed(ctrl, ns, &c, 0, open_for_write))
483520 return -EACCES;
484521
485522 d.metadata = READ_ONCE(cmd->metadata);
@@ -595,7 +632,7 @@ static int nvme_ns_ioctl(struct nvme_ns *ns, unsigned int cmd, · 109 unmodified lines
595632 case NVME_IOCTL_SUBMIT_IO32:
596633 #endif
597634 case NVME_IOCTL_SUBMIT_IO:
598− return nvme_submit_io(ns, argp);
635+ return nvme_submit_io(ns, argp, flags, open_for_write);
599636 case NVME_IOCTL_IO64_CMD_VEC:
600637 flags |= NVME_IOCTL_VEC;
601638 fallthrough;
@@ -692,14 +729,14 @@ int nvme_ns_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd, · 90 unmodified lines
692729 static int nvme_ns_head_ctrl_ioctl(struct nvme_ns *ns, unsigned int cmd,
693730 void __user *argp, struct nvme_ns_head *head, int srcu_idx,
694731 bool open_for_write)
695− __releases(&head->srcu)
732+ __releases_shared(&head->srcu)
696733 {
697734 struct nvme_ctrl *ctrl = ns->ctrl;
698735 int ret;
699736
700737 nvme_get_ctrl(ns->ctrl);
701738 srcu_read_unlock(&head->srcu, srcu_idx);
702− ret = nvme_ctrl_ioctl(ns->ctrl, cmd, argp, open_for_write);
739+ ret = nvme_ctrl_ioctl(ctrl, cmd, argp, open_for_write);
703740
704741 nvme_put_ctrl(ctrl);
705742 return ret;
drivers/nvme/host/multipath.c+34−30
@@ -288,6 +288,25 @@ void nvme_mpath_revalidate_paths(struct nvme_ns_head *head) · 287 unmodified lines
288288 kblockd_schedule_work(&head->requeue_work);
289289 }
290290
291+#ifdef CONFIG_BLK_DEV_ZONED
292+int nvme_mpath_revalidate_zones(struct nvme_ns_head *head)
293+{
294+ struct gendisk *disk = head->disk;
295+ int ret;
296+
297+ if (!disk || !blk_queue_is_zoned(disk->queue) ||
298+ !test_bit(NVME_NSHEAD_DISK_LIVE, &head->flags))
299+ return 0;
300+
301+ ret = blk_revalidate_disk_zones(disk);
302+ if (ret)
303+ dev_warn_ratelimited(disk_to_dev(disk),
304+ "failed to revalidate zoned namespace head: %d\n",
305+ ret);
306+ return ret;
307+}
308+#endif /* CONFIG_BLK_DEV_ZONED */
309+
291310 static bool nvme_path_is_disabled(struct nvme_ns *ns)
292311 {
293312 enum nvme_ctrl_state state = nvme_ctrl_state(ns->ctrl);
@@ -306,6 +325,7 @@ static bool nvme_path_is_disabled(struct nvme_ns *ns) · 12 unmodified lines
306325 }
307326
308327 static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node)
328+ __must_hold_shared(&head->srcu)
309329 {
310330 int found_distance = INT_MAX, fallback_distance = INT_MAX, distance;
311331 struct nvme_ns *found = NULL, *fallback = NULL, *ns;
@@ -348,6 +368,7 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node) · 36 unmodified lines
348368
349369 static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head,
350370 struct nvme_ns *ns)
371+ __must_hold_shared(&head->srcu)
351372 {
352373 ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns,
353374 siblings);
@@ -357,6 +378,7 @@ static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head, · 3 unmodified lines
357378 }
358379
359380 static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head)
381+ __must_hold_shared(&head->srcu)
360382 {
361383 struct nvme_ns *ns, *found = NULL;
362384 int node = numa_node_id();
@@ -405,6 +427,7 @@ out: · 42 unmodified lines
405427 }
406428
407429 static struct nvme_ns *nvme_queue_depth_path(struct nvme_ns_head *head)
430+ __must_hold_shared(&head->srcu)
408431 {
409432 struct nvme_ns *best_opt = NULL, *best_nonopt = NULL, *ns;
410433 unsigned int min_depth_opt = UINT_MAX, min_depth_nonopt = UINT_MAX;
@@ -448,6 +471,7 @@ static inline bool nvme_path_is_optimized(struct nvme_ns *ns) · 37 unmodified lines
448471 }
449472
450473 static struct nvme_ns *nvme_numa_path(struct nvme_ns_head *head)
474+ __must_hold_shared(&head->srcu)
451475 {
452476 int node = numa_node_id();
453477 struct nvme_ns *ns;
@@ -473,6 +497,7 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head) · 19 unmodified lines
473497 }
474498
475499 static bool nvme_available_path(struct nvme_ns_head *head)
500+ __must_hold_shared(&head->srcu)
476501 {
477502 struct nvme_ns *ns;
478503
@@ -611,28 +636,8 @@ const struct block_device_operations nvme_ns_head_ops = { · 132 unmodified lines
611636 .pr_ops = &nvme_pr_ops,
612637 };
613638
614−static inline struct nvme_ns_head *cdev_to_ns_head(struct cdev *cdev)
615−{
616− return container_of(cdev, struct nvme_ns_head, cdev);
617−}
618−
619−static int nvme_ns_head_chr_open(struct inode *inode, struct file *file)
620−{
621− if (!nvme_tryget_ns_head(cdev_to_ns_head(inode->i_cdev)))
622− return -ENXIO;
623− return 0;
624−}
625−
626−static int nvme_ns_head_chr_release(struct inode *inode, struct file *file)
627−{
628− nvme_put_ns_head(cdev_to_ns_head(inode->i_cdev));
629− return 0;
630−}
631−
632639 static const struct file_operations nvme_ns_head_chr_fops = {
633640 .owner = THIS_MODULE,
634− .open = nvme_ns_head_chr_open,
635− .release = nvme_ns_head_chr_release,
636641 .unlocked_ioctl = nvme_ns_head_chr_ioctl,
637642 .compat_ioctl = compat_ptr_ioctl,
638643 .uring_cmd = nvme_ns_head_chr_uring_cmd,
@@ -647,10 +652,12 @@ static void nvme_add_ns_head_cdev(struct nvme_ns_head *head) · 8 unmodified lines
647652 snprintf(name, sizeof(name), "ng%dn%d", head->subsys->instance,
648653 head->instance);
649654
655+ nvme_get_ns_head(head); /* Undone in nvme_cdev_rel() */
650656 if (nvme_cdev_add(name, &head->cdev, &head->cdev_device,
651657 &nvme_ns_head_chr_fops, THIS_MODULE)) {
652658 dev_err(disk_to_dev(head->disk),
653659 "Unable to create the %s device\n", name);
660+ nvme_put_ns_head(head);
654661 return;
655662 }
656663 set_bit(NVME_NSHEAD_CDEV_LIVE, &head->flags);
@@ -692,14 +699,15 @@ static void nvme_remove_head(struct nvme_ns_head *head) · 35 unmodified lines
692699 {
693700 if (test_and_clear_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
694701 /*
695− * requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared
696− * to allow multipath to fail all I/O.
702+ * Requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared
703+ * to allow multipath to fail all I/O. First synchronize to
704+ * add any bios to the requeue list.
697705 */
706+ synchronize_srcu(&head->srcu);
698707 kblockd_schedule_work(&head->requeue_work);
699708
700709 if (test_and_clear_bit(NVME_NSHEAD_CDEV_LIVE, &head->flags))
701710 nvme_cdev_del(&head->cdev, &head->cdev_device);
702− synchronize_srcu(&head->srcu);
703711 del_gendisk(head->disk);
704712 }
705713 nvme_put_ns_head(head);
@@ -728,12 +736,10 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head) · 22 unmodified lines
728736 struct queue_limits lim;
729737
730738 mutex_init(&head->lock);
731− bio_list_init(&head->requeue_list);
732739 spin_lock_init(&head->requeue_lock);
733740 INIT_WORK(&head->requeue_work, nvme_requeue_work);
734741 INIT_WORK(&head->partition_scan_work, nvme_partition_scan_work);
735742 INIT_DELAYED_WORK(&head->remove_work, nvme_remove_head_work);
736− head->delayed_removal_secs = 0;
737743
738744 /*
739745 * If "multipath_always_on" is enabled, a multipath node is added
@@ -777,7 +783,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head) · 37 unmodified lines
777783 set_bit(GD_SUPPRESS_PART_SCAN, &head->disk->state);
778784 sprintf(head->disk->disk_name, "nvme%dn%d",
779785 ctrl->subsys->instance, head->instance);
780− nvme_tryget_ns_head(head);
786+ nvme_get_ns_head(head);
781787 return 0;
782788 }
783789
@@ -819,12 +825,14 @@ static void nvme_mpath_set_live(struct nvme_ns *ns) · 35 unmodified lines
819825 mutex_unlock(&head->lock);
820826
821827 synchronize_srcu(&head->srcu);
828+ nvme_mpath_revalidate_zones(head);
822829 kblockd_schedule_work(&head->requeue_work);
823830 }
824831
825832 static int nvme_parse_ana_log(struct nvme_ctrl *ctrl, void *data,
826833 int (*cb)(struct nvme_ctrl *ctrl, struct nvme_ana_group_desc *,
827834 void *))
835+ __must_hold(&ctrl->ana_lock)
828836 {
829837 void *base = ctrl->ana_log_buf;
830838 size_t offset = sizeof(struct nvme_ana_rsp_hdr);
@@ -1375,10 +1383,6 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid) · 544 unmodified lines
13751383 nvme_mpath_set_live(ns);
13761384 }
13771385
1378−#ifdef CONFIG_BLK_DEV_ZONED
1379− if (blk_queue_is_zoned(ns->queue) && ns->head->disk)
1380− ns->head->disk->nr_zones = ns->disk->nr_zones;
1381−#endif
13821386 }
13831387
13841388 void nvme_mpath_remove_disk(struct nvme_ns_head *head)
drivers/nvme/host/nvme.h+50−17
@@ -178,6 +178,11 @@ enum nvme_quirks { · 177 unmodified lines
178178 * Align dma pool segment size to 512 bytes
179179 */
180180 NVME_QUIRK_DMAPOOL_ALIGN_512 = (1 << 22),
181+
182+ /*
183+ * Admin queue DMA buffers must be page aligned
184+ */
185+ NVME_QUIRK_ADMIN_PAGE_ALIGN = (1 << 23),
181186 };
182187
183188 static inline char *nvme_quirk_name(enum nvme_quirks q)
@@ -229,6 +234,8 @@ static inline char *nvme_quirk_name(enum nvme_quirks q) · 45 unmodified lines
229234 return "broken_msi";
230235 case NVME_QUIRK_DMAPOOL_ALIGN_512:
231236 return "dmapool_align_512";
237+ case NVME_QUIRK_ADMIN_PAGE_ALIGN:
238+ return "admin_page_align";
232239 }
233240
234241 return "unknown";
@@ -361,7 +368,8 @@ struct nvme_ctrl { · 126 unmodified lines
361368 wait_queue_head_t state_wq;
362369
363370 struct nvme_subsystem *subsys;
364− struct list_head subsys_entry;
371+ struct list_head subsys_entry
372+ __guarded_by(&nvme_subsystems_lock);
365373
366374 struct opal_dev *opal_dev;
367375
@@ -493,10 +501,13 @@ struct nvme_subsystem { · 125 unmodified lines
493501 * a separate refcount.
494502 */
495503 struct kref ref;
496− struct list_head entry;
504+ struct list_head entry
505+ __guarded_by(&nvme_subsystems_lock);
497506 struct mutex lock;
498− struct list_head ctrls;
499− struct list_head nsheads;
507+ struct list_head ctrls
508+ __guarded_by(&nvme_subsystems_lock);
509+ struct list_head nsheads
510+ __guarded_by(&lock);
500511 char subnqn[NVMF_NQN_SIZE];
501512 char serial[20];
502513 char model[40];
@@ -561,20 +572,22 @@ struct nvme_ns_head { · 58 unmodified lines
561572 u16 nr_plids;
562573 u16 *plids;
563574 #ifdef CONFIG_NVME_MULTIPATH
564− struct bio_list requeue_list;
575+ struct bio_list requeue_list
576+ __guarded_by(&requeue_lock);
565577 spinlock_t requeue_lock;
566578 struct work_struct requeue_work;
567579 struct work_struct partition_scan_work;
568580 struct mutex lock;
569581 unsigned long flags;
570582 struct delayed_work remove_work;
571− unsigned int delayed_removal_secs;
583+ unsigned int delayed_removal_secs
584+ __guarded_by(&subsys->lock);
572585 atomic_long_t io_requeue_no_usable_path_count;
573586 atomic_long_t io_fail_no_available_path_count;
574587 #define NVME_NSHEAD_DISK_LIVE 0
575588 #define NVME_NSHEAD_QUEUE_IF_NO_PATH 1
576589 #define NVME_NSHEAD_CDEV_LIVE 2
577− struct nvme_ns __rcu *current_path[];
590+ struct nvme_ns __rcu_guarded *current_path[];
578591 #endif
579592 };
580593
@@ -679,12 +692,12 @@ static inline struct request *nvme_find_rq(struct blk_mq_tags *tags, · 98 unmodified lines
679692
680693 rq = blk_mq_tag_to_rq(tags, tag);
681694 if (unlikely(!rq)) {
682− pr_err("could not locate request for tag %#x\n",
683− tag);
695+ pr_err_ratelimited("could not locate request for tag %#x\n",
696+ tag);
684697 return NULL;
685698 }
686699 if (unlikely(nvme_genctr_mask(nvme_req(rq)->genctr) != genctr)) {
687− dev_err(nvme_req(rq)->ctrl->device,
700+ dev_err_ratelimited(nvme_req(rq)->ctrl->device,
688701 "request %#x genctr mismatch (got %#x expected %#x)\n",
689702 tag, genctr, nvme_genctr_mask(nvme_req(rq)->genctr));
690703 return NULL;
@@ -995,6 +1008,7 @@ int nvme_delete_ctrl(struct nvme_ctrl *ctrl); · 304 unmodified lines
9951008 void nvme_queue_scan(struct nvme_ctrl *ctrl);
9961009 int nvme_get_log(struct nvme_ctrl *ctrl, u32 nsid, u8 log_page, u8 lsp, u8 csi,
9971010 void *log, size_t size, u64 offset);
1011+void nvme_get_ns_head(struct nvme_ns_head *head);
9981012 bool nvme_tryget_ns_head(struct nvme_ns_head *head);
9991013 void nvme_put_ns_head(struct nvme_ns_head *head);
10001014 int nvme_cdev_add(const char *name, struct cdev *cdev,
@@ -1032,16 +1046,20 @@ extern const struct attribute_group *nvme_dev_attr_groups[]; · 31 unmodified lines
10321046 extern const struct block_device_operations nvme_bdev_ops;
10331047
10341048 void nvme_delete_ctrl_sync(struct nvme_ctrl *ctrl);
1035−struct nvme_ns *nvme_find_path(struct nvme_ns_head *head);
1049+struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
1050+ __must_hold_shared(&head->srcu);
10361051 #ifdef CONFIG_NVME_MULTIPATH
10371052 static inline bool nvme_ctrl_use_ana(struct nvme_ctrl *ctrl)
10381053 {
10391054 return ctrl->ana_log_buf != NULL;
10401055 }
10411056
1042−void nvme_mpath_unfreeze(struct nvme_subsystem *subsys);
1043−void nvme_mpath_wait_freeze(struct nvme_subsystem *subsys);
1044−void nvme_mpath_start_freeze(struct nvme_subsystem *subsys);
1057+void nvme_mpath_unfreeze(struct nvme_subsystem *subsys)
1058+ __must_hold(&subsys->lock);
1059+void nvme_mpath_wait_freeze(struct nvme_subsystem *subsys)
1060+ __must_hold(&subsys->lock);
1061+void nvme_mpath_start_freeze(struct nvme_subsystem *subsys)
1062+ __must_hold(&subsys->lock);
10451063 void nvme_mpath_default_iopolicy(struct nvme_subsystem *subsys);
10461064 void nvme_failover_req(struct request *req);
10471065 void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl);
@@ -1184,6 +1202,15 @@ static inline bool nvme_mpath_queue_if_no_path(struct nvme_ns_head *head) · 136 unmodified lines
11841202 }
11851203 #endif /* CONFIG_NVME_MULTIPATH */
11861204
1205+#if defined(CONFIG_NVME_MULTIPATH) && defined(CONFIG_BLK_DEV_ZONED)
1206+int nvme_mpath_revalidate_zones(struct nvme_ns_head *head);
1207+#else
1208+static inline int nvme_mpath_revalidate_zones(struct nvme_ns_head *head)
1209+{
1210+ return 0;
1211+}
1212+#endif
1213+
11871214 int nvme_ns_get_unique_id(struct nvme_ns *ns, u8 id[16],
11881215 enum blk_unique_id type);
11891216
@@ -1290,10 +1317,16 @@ static inline void nvme_auth_revoke_tls_key(struct nvme_ctrl *ctrl) {}; · 100 unmodified lines
12901317
12911318 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
12921319 u8 opcode);
1293−u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u8 opcode);
1320+u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u8 opcode)
1321+ __cond_acquires(nonzero, &ctrl->subsys->lock)
1322+ __cond_acquires(nonzero, &ctrl->scan_lock);
1323+
12941324 int nvme_execute_rq(struct request *rq, bool at_head);
1295−void nvme_passthru_end(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u32 effects,
1296− struct nvme_command *cmd, int status);
1325+u32 nvme_passthru_end(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u32 effects,
1326+ struct nvme_command *cmd, int status)
1327+ __cond_releases(nonzero, &ctrl->scan_lock)
1328+ __cond_releases(nonzero, &ctrl->subsys->lock);
1329+
12971330 struct nvme_ctrl *nvme_ctrl_from_file(struct file *file);
12981331 struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid);
12991332 bool nvme_get_ns(struct nvme_ns *ns);
drivers/nvme/host/pci.c+24−7
@@ -213,6 +213,7 @@ static int quirks_param_set(const char *value, const struct kernel_param *kp) · 212 unmodified lines
213213 if (nvme_parse_quirk_entry(field, &qlist[i])) {
214214 pr_err("nvme: failed to parse quirk string %s\n",
215215 value);
216+ err = -EINVAL;
216217 goto out_free_qlist;
217218 }
218219
@@ -366,7 +367,8 @@ struct nvme_queue { · 147 unmodified lines
366367 struct nvme_dev *dev;
367368 struct nvme_descriptor_pools descriptor_pools;
368369 spinlock_t sq_lock;
369− void *sq_cmds;
370+ void *sq_cmds
371+ __guarded_by(&sq_lock);
370372 /* only used for poll queues: */
371373 spinlock_t cq_poll_lock ____cacheline_aligned_in_smp;
372374 struct nvme_completion *cqes;
@@ -375,9 +377,11 @@ struct nvme_queue { · 2 unmodified lines
375377 u32 __iomem *q_db;
376378 u32 q_depth;
377379 u16 cq_vector;
378− u16 sq_tail;
379− u16 last_sq_tail;
380380 u16 cq_head;
381+ u16 sq_tail
382+ __guarded_by(&sq_lock);
383+ u16 last_sq_tail
384+ __guarded_by(&sq_lock);
381385 u16 qid;
382386 u8 cq_phase;
383387 u8 sqes;
@@ -716,6 +720,7 @@ static void nvme_pci_map_queues(struct blk_mq_tag_set *set) · 332 unmodified lines
716720 * Write sq tail if we are asked to, or if the next command would wrap.
717721 */
718722 static inline void nvme_write_sq_db(struct nvme_queue *nvmeq, bool write_sq)
723+ __must_hold(&nvmeq->sq_lock)
719724 {
720725 if (!write_sq) {
721726 u16 next_tail = nvmeq->sq_tail + 1;
@@ -734,6 +739,7 @@ static inline void nvme_write_sq_db(struct nvme_queue *nvmeq, bool write_sq) · 12 unmodified lines
734739
735740 static inline void nvme_sq_copy_cmd(struct nvme_queue *nvmeq,
736741 struct nvme_command *cmd)
742+ __must_hold(&nvmeq->sq_lock)
737743 {
738744 memcpy(nvmeq->sq_cmds + (nvmeq->sq_tail << nvmeq->sqes),
739745 absolute_pointer(cmd), sizeof(*cmd));
@@ -1580,13 +1586,18 @@ static inline void nvme_handle_cqe(struct nvme_queue *nvmeq, · 840 unmodified lines
15801586
15811587 req = nvme_find_rq(nvme_queue_tagset(nvmeq), command_id);
15821588 if (unlikely(!req)) {
1583− dev_warn(nvmeq->dev->ctrl.device,
1584− "invalid id %d completed on queue %d\n",
1585− command_id, le16_to_cpu(cqe->sq_id));
1589+ dev_warn_ratelimited(nvmeq->dev->ctrl.device,
1590+ "invalid id %d completed on queue %d\n",
1591+ command_id, le16_to_cpu(cqe->sq_id));
15861592 return;
15871593 }
15881594
1589− trace_nvme_sq(req, cqe->sq_head, nvmeq->sq_tail);
1595+ /*
1596+ * Tracing only; annotate a lockless snapshot of nvmeq->sq_tail using
1597+ * data_race(). This would also help suppress context analysis warning
1598+ * while accessing nvmeq->sq_tail without acquiring ->sq_lock.
1599+ */
1600+ trace_nvme_sq(req, cqe->sq_head, data_race(nvmeq->sq_tail));
15901601 if (!nvme_try_complete_req(req, cqe->status, cqe->result) &&
15911602 !blk_mq_add_to_batch(req, iob,
15921603 nvme_req(req)->status != NVME_SC_SUCCESS,
@@ -2013,6 +2024,7 @@ disable: · 420 unmodified lines
20132024 }
20142025
20152026 static void nvme_free_queue(struct nvme_queue *nvmeq)
2027+ __context_unsafe(/* frees queue which is no longer in use */)
20162028 {
20172029 dma_free_coherent(nvmeq->dev->dev, CQ_SIZE(nvmeq),
20182030 (void *)nvmeq->cqes, nvmeq->cq_dma_addr);
@@ -2107,6 +2119,7 @@ static int nvme_cmb_qdepth(struct nvme_dev *dev, int nr_io_queues, · 88 unmodified lines
21072119
21082120 static int nvme_alloc_sq_cmds(struct nvme_dev *dev, struct nvme_queue *nvmeq,
21092121 int qid)
2122+ __context_unsafe(/* safe to allocate sq_cmds without any protection */)
21102123 {
21112124 struct pci_dev *pdev = to_pci_dev(dev->dev);
21122125
@@ -2181,6 +2194,7 @@ static int queue_request_irq(struct nvme_queue *nvmeq) · 68 unmodified lines
21812194 }
21822195
21832196 static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
2197+ __context_unsafe(/* initialize unpublished/lock-guarded variables */)
21842198 {
21852199 struct nvme_dev *dev = nvmeq->dev;
21862200
@@ -2199,6 +2213,7 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid) · 12 unmodified lines
21992213 * Try getting shutdown_lock while setting up IO queues.
22002214 */
22012215 static int nvme_setup_io_queues_trylock(struct nvme_dev *dev)
2216+ __cond_acquires(0, &dev->shutdown_lock)
22022217 {
22032218 /*
22042219 * Give up if the lock is being held by nvme_dev_disable.
@@ -2400,6 +2415,7 @@ static int nvme_pci_configure_admin_queue(struct nvme_dev *dev) · 195 unmodified lines
24002415 result = queue_request_irq(nvmeq);
24012416 if (result) {
24022417 dev->online_queues--;
2418+ nvme_disable_ctrl(&dev->ctrl, false);
24032419 return result;
24042420 }
24052421
@@ -3838,6 +3854,7 @@ out_disable: · 1432 unmodified lines
38383854 nvme_dev_remove_admin(dev);
38393855 nvme_dbbuf_dma_free(dev);
38403856 nvme_free_queues(dev, 0);
3857+ nvme_release_descriptor_pools(dev);
38413858 out_release_iod_mempool:
38423859 mempool_destroy(dev->dmavec_mempool);
38433860 out_dev_unmap:
drivers/nvme/host/rdma.c+97−63
@@ -16,6 +16,7 @@ · 15 unmodified lines
1616 #include <linux/types.h>
1717 #include <linux/list.h>
1818 #include <linux/mutex.h>
19+#include <linux/async.h>
1920 #include <linux/scatterlist.h>
2021 #include <linux/nvme.h>
2122 #include <linux/unaligned.h>
@@ -39,11 +40,18 @@ · 17 unmodified lines
3940 #define NVME_RDMA_METADATA_SGL_SIZE \
4041 (sizeof(struct scatterlist) * NVME_INLINE_METADATA_SG_CNT)
4142
43+static DEFINE_MUTEX(device_list_mutex);
44+static LIST_HEAD_GUARDED(device_list, device_list_mutex);
45+
46+static DEFINE_MUTEX(nvme_rdma_ctrl_mutex);
47+static LIST_HEAD_GUARDED(nvme_rdma_ctrl_list, nvme_rdma_ctrl_mutex);
48+
4249 struct nvme_rdma_device {
4350 struct ib_device *dev;
4451 struct ib_pd *pd;
4552 struct kref ref;
46− struct list_head entry;
53+ struct list_head entry
54+ __guarded_by(&device_list_mutex);
4755 unsigned int num_inline_segments;
4856 };
4957
@@ -100,6 +108,11 @@ struct nvme_rdma_queue { · 50 unmodified lines
100108 struct mutex queue_lock;
101109 };
102110
111+struct nvme_rdma_setup_ctx {
112+ struct nvme_rdma_queue *queue;
113+ int *err;
114+};
115+
103116 struct nvme_rdma_ctrl {
104117 /* read only in the hot path */
105118 struct nvme_rdma_queue *queues;
@@ -112,7 +125,8 @@ struct nvme_rdma_ctrl { · 6 unmodified lines
112125
113126 struct delayed_work reconnect_work;
114127
115− struct list_head list;
128+ struct list_head list
129+ __guarded_by(&nvme_rdma_ctrl_mutex);
116130
117131 struct blk_mq_tag_set admin_tag_set;
118132 struct nvme_rdma_device *device;
@@ -132,12 +146,6 @@ static inline struct nvme_rdma_ctrl *to_rdma_ctrl(struct nvme_ctrl *ctrl) · 13 unmodified lines
132146 return container_of(ctrl, struct nvme_rdma_ctrl, ctrl);
133147 }
134148
135−static LIST_HEAD(device_list);
136−static DEFINE_MUTEX(device_list_mutex);
137−
138−static LIST_HEAD(nvme_rdma_ctrl_list);
139−static DEFINE_MUTEX(nvme_rdma_ctrl_mutex);
140−
141149 /*
142150 * Disabling this option makes small I/O goes faster, but is fundamentally
143151 * unsafe. With it turned off we will have to register a global rkey that
@@ -566,16 +574,14 @@ out_put_dev: · 422 unmodified lines
566574 return ret;
567575 }
568576
569−static int nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl,
570− int idx, size_t queue_size)
577+static int nvme_rdma_alloc_queue(struct nvme_rdma_queue *queue)
571578 {
572− struct nvme_rdma_queue *queue;
579+ struct nvme_rdma_ctrl *ctrl = queue->ctrl;
580+ int idx = nvme_rdma_queue_idx(queue);
573581 struct sockaddr *src_addr = NULL;
574582 int ret;
575583
576− queue = &ctrl->queues[idx];
577584 mutex_init(&queue->queue_lock);
578− queue->ctrl = ctrl;
579585 if (idx && ctrl->ctrl.max_integrity_segments)
580586 queue->pi_support = true;
581587 else
@@ -587,8 +593,6 @@ static int nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl, · 5 unmodified lines
587593 else
588594 queue->cmnd_capsule_len = sizeof(struct nvme_command);
589595
590− queue->queue_size = queue_size;
591−
592596 queue->cm_id = rdma_create_id(&init_net, nvme_rdma_cm_handler, queue,
593597 RDMA_PS_TCP, IB_QPT_RC);
594598 if (IS_ERR(queue->cm_id)) {
@@ -694,59 +698,68 @@ static int nvme_rdma_start_queue(struct nvme_rdma_ctrl *ctrl, int idx) · 99 unmodified lines
694698 return ret;
695699 }
696700
697−static int nvme_rdma_start_io_queues(struct nvme_rdma_ctrl *ctrl,
698− int first, int last)
701+static void nvme_rdma_setup_queue_async(void *data, async_cookie_t cookie)
699702 {
700− int i, ret = 0;
703+ struct nvme_rdma_setup_ctx *ctx = data;
704+ struct nvme_rdma_queue *queue;
705+ int ret;
701706
702− for (i = first; i < last; i++) {
703− ret = nvme_rdma_start_queue(ctrl, i);
704− if (ret)
705− goto out_stop_queues;
706− }
707+ queue = ctx->queue;
708+ ret = nvme_rdma_alloc_queue(queue);
709+ if (ret)
710+ goto out_err;
707711
708− return 0;
712+ ret = nvme_rdma_start_queue(queue->ctrl, nvme_rdma_queue_idx(queue));
713+ if (ret)
714+ goto out_err;
709715
710−out_stop_queues:
711− for (i--; i >= first; i--)
712− nvme_rdma_stop_queue(&ctrl->queues[i]);
713− return ret;
716+ return;
717+out_err:
718+ WRITE_ONCE(*ctx->err, ret);
714719 }
715720
716−static int nvme_rdma_alloc_io_queues(struct nvme_rdma_ctrl *ctrl)
721+static int nvme_rdma_setup_io_queues(struct nvme_rdma_ctrl *ctrl,
722+ unsigned int first, unsigned int last, size_t queue_size)
717723 {
718− struct nvmf_ctrl_options *opts = ctrl->ctrl.opts;
719− unsigned int nr_io_queues;
720− int i, ret;
721−
722− nr_io_queues = nvmf_nr_io_queues(opts);
723− ret = nvme_set_queue_count(&ctrl->ctrl, &nr_io_queues);
724− if (ret)
725− return ret;
724+ ASYNC_DOMAIN_EXCLUSIVE(queue_domain);
725+ struct nvme_rdma_setup_ctx *ctxs;
726+ int nr_queues = last - first;
727+ int err = 0, i, ret;
726728
727− if (nr_io_queues == 0) {
728− dev_err(ctrl->ctrl.device,
729− "unable to set any I/O queues\n");
729+ ctxs = kmalloc_objs(*ctxs, nr_queues);
730+ if (!ctxs)
730731 return -ENOMEM;
731− }
732732
733− ctrl->ctrl.queue_count = nr_io_queues + 1;
734− dev_info(ctrl->ctrl.device,
735− "creating %d I/O queues.\n", nr_io_queues);
733+ for (i = 0; i < nr_queues; i++) {
734+ struct nvme_rdma_queue *queue = &ctrl->queues[first + i];
736735
737− nvmf_set_io_queues(opts, nr_io_queues, ctrl->io_queues);
738− for (i = 1; i < ctrl->ctrl.queue_count; i++) {
739− ret = nvme_rdma_alloc_queue(ctrl, i,
740− ctrl->ctrl.sqsize + 1);
741− if (ret)
742− goto out_free_queues;
736+ queue->ctrl = ctrl;
737+ queue->queue_size = queue_size;
738+
739+ ctxs[i].queue = queue;
740+ ctxs[i].err = &err;
741+ async_schedule_domain(nvme_rdma_setup_queue_async, &ctxs[i],
742+ &queue_domain);
743743 }
744744
745− return 0;
745+ async_synchronize_full_domain(&queue_domain);
746+ kfree(ctxs);
746747
748+ ret = READ_ONCE(err);
749+ if (ret)
750+ goto out_free_queues;
751+
752+ return 0;
747753 out_free_queues:
748− for (i--; i >= 1; i--)
749− nvme_rdma_free_queue(&ctrl->queues[i]);
754+ for (i = 0; i < nr_queues; i++) {
755+ struct nvme_rdma_queue *queue =
756+ &ctrl->queues[first + i];
757+
758+ if (test_bit(NVME_RDMA_Q_LIVE, &queue->flags))
759+ nvme_rdma_stop_queue(queue);
760+ if (test_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
761+ nvme_rdma_free_queue(queue);
762+ }
750763
751764 return ret;
752765 }
@@ -783,7 +796,9 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, · 30 unmodified lines
783796 bool pi_capable = false;
784797 int error;
785798
786− error = nvme_rdma_alloc_queue(ctrl, 0, NVME_AQ_DEPTH);
799+ ctrl->queues[0].ctrl = ctrl;
800+ ctrl->queues[0].queue_size = NVME_AQ_DEPTH;
801+ error = nvme_rdma_alloc_queue(&ctrl->queues[0]);
787802 if (error)
788803 return error;
789804
@@ -863,12 +878,23 @@ out_free_queue: · 73 unmodified lines
863878
864879 static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
865880 {
881+ unsigned int nr_io_queues;
866882 int ret, nr_queues;
867883
868− ret = nvme_rdma_alloc_io_queues(ctrl);
884+ nr_io_queues = nvmf_nr_io_queues(ctrl->ctrl.opts);
885+ ret = nvme_set_queue_count(&ctrl->ctrl, &nr_io_queues);
869886 if (ret)
870887 return ret;
871888
889+ if (nr_io_queues == 0) {
890+ dev_err(ctrl->ctrl.device, "unable to set any I/O queues\n");
891+ return -ENOMEM;
892+ }
893+
894+ ctrl->ctrl.queue_count = nr_io_queues + 1;
895+ dev_info(ctrl->ctrl.device, "creating %d I/O queues.\n", nr_io_queues);
896+ nvmf_set_io_queues(ctrl->ctrl.opts, nr_io_queues, ctrl->io_queues);
897+
872898 if (new) {
873899 ret = nvme_rdma_alloc_tag_set(&ctrl->ctrl);
874900 if (ret)
@@ -881,7 +907,9 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) · 6 unmodified lines
881907 * queue number might have changed.
882908 */
883909 nr_queues = min(ctrl->tag_set.nr_hw_queues + 1, ctrl->ctrl.queue_count);
884− ret = nvme_rdma_start_io_queues(ctrl, 1, nr_queues);
910+ ret = nvme_rdma_setup_io_queues(ctrl, 1, nr_queues,
911+ ctrl->ctrl.sqsize + 1);
912+
885913 if (ret)
886914 goto out_cleanup_tagset;
887915
@@ -905,12 +933,15 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) · 17 unmodified lines
905933
906934 /*
907935 * If the number of queues has increased (reconnect case)
908− * start all new queues now.
936+ * setup all new queues now.
909937 */
910− ret = nvme_rdma_start_io_queues(ctrl, nr_queues,
911− ctrl->tag_set.nr_hw_queues + 1);
912− if (ret)
913− goto out_wait_freeze_timed_out;
938+ if (ctrl->tag_set.nr_hw_queues + 1 > nr_queues) {
939+ ret = nvme_rdma_setup_io_queues(ctrl, nr_queues,
940+ ctrl->tag_set.nr_hw_queues + 1,
941+ ctrl->ctrl.sqsize + 1);
942+ if (ret)
943+ goto out_wait_freeze_timed_out;
944+ }
914945
915946 return 0;
916947
@@ -969,7 +1000,7 @@ static void nvme_rdma_free_ctrl(struct nvme_ctrl *nctrl) · 52 unmodified lines
9691000 {
9701001 struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl);
9711002
972− if (list_empty(&ctrl->list))
1003+ if (list_empty_careful(&ctrl->list))
9731004 goto free_ctrl;
9741005
9751006 mutex_lock(&nvme_rdma_ctrl_mutex);
@@ -2254,7 +2285,10 @@ static struct nvme_rdma_ctrl *nvme_rdma_alloc_ctrl(struct device *dev, · 1278 unmodified lines
22542285 if (!ctrl)
22552286 return ERR_PTR(-ENOMEM);
22562287 ctrl->ctrl.opts = opts;
2257− INIT_LIST_HEAD(&ctrl->list);
2288+ /*
2289+ * Safe to init list while allocating ctrl object.
2290+ */
2291+ context_unsafe(INIT_LIST_HEAD(&ctrl->list));
22582292
22592293 if (!(opts->mask & NVMF_OPT_TRSVCID)) {
22602294 opts->trsvcid =
drivers/nvme/host/sysfs.c+3−1
@@ -240,8 +240,10 @@ static ssize_t nuse_show(struct device *dev, struct device_attribute *attr, · 239 unmodified lines
240240 ret = ns_head_update_nuse(head);
241241 else
242242 ret = ns_update_nuse(disk->private_data);
243− if (ret)
243+ if (ret < 0)
244244 return ret;
245+ else if (ret > 0)
246+ return -EIO;
245247
246248 return sysfs_emit(buf, "%llu\n", head->nuse);
247249 }
drivers/nvme/host/tcp.c+64−7
@@ -80,6 +80,7 @@ struct nvme_tcp_request { · 79 unmodified lines
8080
8181 struct bio *curr_bio;
8282 struct iov_iter iter;
83+ u32 data_recvd;
8384
8485 /* send state */
8586 size_t offset;
@@ -108,6 +109,7 @@ struct nvme_tcp_queue { · 22 unmodified lines
108109
109110 struct mutex queue_lock;
110111 struct mutex send_mutex;
112+ struct mutex pf_cache_lock;
111113 struct llist_head req_list;
112114 struct list_head send_list;
113115
@@ -149,13 +151,17 @@ struct nvme_tcp_queue { · 35 unmodified lines
149151 #endif
150152 };
151153
154+static DEFINE_MUTEX(nvme_tcp_ctrl_mutex);
155+static LIST_HEAD_GUARDED(nvme_tcp_ctrl_list, nvme_tcp_ctrl_mutex);
156+
152157 struct nvme_tcp_ctrl {
153158 /* read only in the hot path */
154159 struct nvme_tcp_queue *queues;
155160 struct blk_mq_tag_set tag_set;
156161
157162 /* other member variables */
158− struct list_head list;
163+ struct list_head list
164+ __guarded_by(&nvme_tcp_ctrl_mutex);
159165 struct blk_mq_tag_set admin_tag_set;
160166 struct sockaddr_storage addr;
161167 struct sockaddr_storage src_addr;
@@ -167,8 +173,6 @@ struct nvme_tcp_ctrl { · 5 unmodified lines
167173 u32 io_queues[HCTX_MAX_TYPES];
168174 };
169175
170−static LIST_HEAD(nvme_tcp_ctrl_list);
171−static DEFINE_MUTEX(nvme_tcp_ctrl_mutex);
172176 static struct workqueue_struct *nvme_tcp_wq;
173177 static const struct blk_mq_ops nvme_tcp_mq_ops;
174178 static const struct blk_mq_ops nvme_tcp_admin_mq_ops;
@@ -550,9 +554,11 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set, · 375 unmodified lines
550554 struct nvme_tcp_queue *queue = &ctrl->queues[queue_idx];
551555 u8 hdgst = nvme_tcp_hdgst_len(queue);
552556
557+ mutex_lock(&queue->pf_cache_lock);
553558 req->pdu = page_frag_alloc(&queue->pf_cache,
554559 sizeof(struct nvme_tcp_cmd_pdu) + hdgst,
555560 GFP_KERNEL | __GFP_ZERO);
561+ mutex_unlock(&queue->pf_cache_lock);
556562 if (!req->pdu)
557563 return -ENOMEM;
558564
@@ -612,6 +618,29 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl) · 53 unmodified lines
612618 queue_work(nvme_reset_wq, &to_tcp_ctrl(ctrl)->err_work);
613619 }
614620
621+/*
622+ * NVMe has no short read: a read that completes successfully must
623+ * have transferred everything it asked for.
624+ */
625+static bool nvme_tcp_data_in_short(struct nvme_tcp_queue *queue,
626+ struct request *rq)
627+{
628+ struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
629+
630+ if (le16_to_cpu(req->status) >> 1)
631+ return false;
632+ if (req_op(rq) != REQ_OP_READ || !req->data_len)
633+ return false;
634+ if (likely(req->data_recvd == req->data_len))
635+ return false;
636+
637+ dev_err(queue->ctrl->ctrl.device,
638+ "queue %d tag %#x short data-in: got %u of %u\n",
639+ nvme_tcp_queue_id(queue), rq->tag,
640+ req->data_recvd, req->data_len);
641+ return true;
642+}
643+
615644 static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
616645 struct nvme_completion *cqe)
617646 {
@@ -631,6 +660,9 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, · 13 unmodified lines
631660 if (req->status == cpu_to_le16(NVME_SC_SUCCESS))
632661 req->status = cqe->status;
633662
663+ if (unlikely(nvme_tcp_data_in_short(queue, rq)))
664+ return -EPROTO;
665+
634666 if (!nvme_try_complete_req(rq, req->status, cqe->result))
635667 nvme_complete_rq(rq);
636668 queue->nr_cqe++;
@@ -641,6 +673,7 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, · 4 unmodified lines
641673 static int nvme_tcp_handle_c2h_data(struct nvme_tcp_queue *queue,
642674 struct nvme_tcp_data_pdu *pdu)
643675 {
676+ struct nvme_tcp_request *req;
644677 struct request *rq;
645678
646679 rq = nvme_find_rq(nvme_tcp_tagset(queue), pdu->command_id);
@@ -651,7 +684,8 @@ static int nvme_tcp_handle_c2h_data(struct nvme_tcp_queue *queue, · 4 unmodified lines
651684 return -ENOENT;
652685 }
653686
654− if (!blk_rq_payload_bytes(rq)) {
687+ req = blk_mq_rq_to_pdu(rq);
688+ if (!blk_rq_payload_bytes(rq) || !req->curr_bio || !req->data_len) {
655689 dev_err(queue->ctrl->ctrl.device,
656690 "queue %d tag %#x unexpected data\n",
657691 nvme_tcp_queue_id(queue), rq->tag);
@@ -745,6 +779,13 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue, · 87 unmodified lines
745779 }
746780 req = blk_mq_rq_to_pdu(rq);
747781
782+ if (unlikely(rq_data_dir(rq) != WRITE)) {
783+ dev_err(queue->ctrl->ctrl.device,
784+ "req %d unexpected r2t for a non-write command\n",
785+ rq->tag);
786+ return -EPROTO;
787+ }
788+
748789 if (unlikely(!r2t_length)) {
749790 dev_err(queue->ctrl->ctrl.device,
750791 "req %d r2t len is %u, probably a bug...\n",
@@ -953,6 +994,7 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · 202 unmodified lines
953994 *len -= recv_len;
954995 *offset += recv_len;
955996 queue->data_remaining -= recv_len;
997+ req->data_recvd += recv_len;
956998 }
957999
9581000 if (!queue->data_remaining) {
@@ -961,6 +1003,8 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb, · 2 unmodified lines
9611003 queue->ddgst_remaining = NVME_TCP_DIGEST_LENGTH;
9621004 } else {
9631005 if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) {
1006+ if (unlikely(nvme_tcp_data_in_short(queue, rq)))
1007+ return -EPROTO;
9641008 nvme_tcp_end_request(rq,
9651009 le16_to_cpu(req->status));
9661010 queue->nr_cqe++;
@@ -1009,6 +1053,9 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue, · 42 unmodified lines
10091053 pdu->command_id);
10101054 struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
10111055
1056+ if (unlikely(nvme_tcp_data_in_short(queue, rq)))
1057+ return -EPROTO;
1058+
10121059 nvme_tcp_end_request(rq, le16_to_cpu(req->status));
10131060 queue->nr_cqe++;
10141061 }
@@ -1417,9 +1464,11 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl) · 402 unmodified lines
14171464 struct nvme_tcp_request *async = &ctrl->async_req;
14181465 u8 hdgst = nvme_tcp_hdgst_len(queue);
14191466
1467+ mutex_lock(&queue->pf_cache_lock);
14201468 async->pdu = page_frag_alloc(&queue->pf_cache,
14211469 sizeof(struct nvme_tcp_cmd_pdu) + hdgst,
14221470 GFP_KERNEL | __GFP_ZERO);
1471+ mutex_unlock(&queue->pf_cache_lock);
14231472 if (!async->pdu)
14241473 return -ENOMEM;
14251474
@@ -1461,6 +1510,7 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) · 35 unmodified lines
14611510 kfree(queue->pdu);
14621511 mutex_destroy(&queue->send_mutex);
14631512 mutex_destroy(&queue->queue_lock);
1513+ mutex_destroy(&queue->pf_cache_lock);
14641514
14651515 #ifdef CONFIG_DEBUG_LOCK_ALLOC
14661516 lockdep_unregister_key(&queue->nvme_tcp_sk_key);
@@ -1788,6 +1838,7 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid, · 321 unmodified lines
17881838 INIT_LIST_HEAD(&queue->send_list);
17891839 mutex_init(&queue->send_mutex);
17901840 INIT_WORK(&queue->io_work, nvme_tcp_io_work);
1841+ mutex_init(&queue->pf_cache_lock);
17911842
17921843 if (qid > 0)
17931844 queue->cmnd_capsule_len = nctrl->ioccsz * 16;
@@ -1928,6 +1979,7 @@ err_sock: · 134 unmodified lines
19281979 err_destroy_mutex:
19291980 mutex_destroy(&queue->send_mutex);
19301981 mutex_destroy(&queue->queue_lock);
1982+ mutex_destroy(&queue->pf_cache_lock);
19311983 return ret;
19321984 }
19331985
@@ -2577,7 +2629,7 @@ static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl) · 643 unmodified lines
25772629 {
25782630 struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
25792631
2580− if (list_empty(&ctrl->list))
2632+ if (list_empty_careful(&ctrl->list))
25812633 goto free_ctrl;
25822634
25832635 mutex_lock(&nvme_tcp_ctrl_mutex);
@@ -2736,6 +2788,7 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns, · 152 unmodified lines
27362788 req->status = cpu_to_le16(NVME_SC_SUCCESS);
27372789 req->offset = 0;
27382790 req->data_sent = 0;
2791+ req->data_recvd = 0;
27392792 req->pdu_len = 0;
27402793 req->pdu_sent = 0;
27412794 req->h2cdata_left = 0;
@@ -2919,7 +2972,10 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev, · 177 unmodified lines
29192972 if (!ctrl)
29202973 return ERR_PTR(-ENOMEM);
29212974
2922− INIT_LIST_HEAD(&ctrl->list);
2975+ /*
2976+ * Safe to init list while allocating ctrl object.
2977+ */
2978+ context_unsafe(INIT_LIST_HEAD(&ctrl->list));
29232979 ctrl->ctrl.opts = opts;
29242980 ctrl->ctrl.queue_count = opts->nr_io_queues + opts->nr_write_queues +
29252981 opts->nr_poll_queues + 1;
@@ -2960,7 +3016,8 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev, · 34 unmodified lines
29603016 }
29613017
29623018 if (opts->mask & NVMF_OPT_HOST_IFACE) {
2963− if (!__dev_get_by_name(&init_net, opts->host_iface)) {
3019+ if (!__dev_get_by_name(current->nsproxy->net_ns,
3020+ opts->host_iface)) {
29643021 pr_err("invalid interface passed: %s\n",
29653022 opts->host_iface);
29663023 ret = -ENODEV;
drivers/nvme/host/zns.c+8−3
@@ -155,7 +155,8 @@ static int nvme_zone_parse_entry(struct nvme_ns *ns, · 154 unmodified lines
155155 struct blk_zone zone = { };
156156
157157 if ((entry->zt & 0xf) != NVME_ZONE_TYPE_SEQWRITE_REQ) {
158− dev_err(ns->ctrl->device, "invalid zone type %#x\n", entry->zt);
158+ dev_err(ns->ctrl->device, "invalid zone type %#x at zone %u\n",
159+ entry->zt, idx);
159160 return -EINVAL;
160161 }
161162
@@ -178,7 +179,7 @@ int nvme_ns_report_zones(struct nvme_ns *ns, sector_t sector, · 16 unmodified lines
178179 struct nvme_zone_report *report;
179180 struct nvme_command c = { };
180181 int ret, zone_idx = 0;
181− unsigned int nz, i;
182+ unsigned int max_in_buf, nz, i;
182183 size_t buflen;
183184
184185 if (ns->head->ids.csi != NVME_CSI_ZNS)
@@ -188,6 +189,9 @@ int nvme_ns_report_zones(struct nvme_ns *ns, sector_t sector, · 3 unmodified lines
188189 if (!report)
189190 return -ENOMEM;
190191
192+ max_in_buf = (buflen - sizeof(struct nvme_zone_report)) /
193+ sizeof(struct nvme_zone_descriptor);
194+
191195 c.zmr.opcode = nvme_cmd_zone_mgmt_recv;
192196 c.zmr.nsid = cpu_to_le32(ns->head->ns_id);
193197 c.zmr.numd = cpu_to_le32(nvme_bytes_to_numd(buflen));
@@ -207,7 +211,8 @@ int nvme_ns_report_zones(struct nvme_ns *ns, sector_t sector, · 13 unmodified lines
207211 goto out_free;
208212 }
209213
210− nz = min((unsigned int)le64_to_cpu(report->nr_zones), nr_zones);
214+ nz = min3((unsigned int)le64_to_cpu(report->nr_zones),
215+ nr_zones - zone_idx, max_in_buf);
211216 if (!nz)
212217 break;
213218
drivers/nvme/target/admin-cmd.c+8−6
@@ -309,8 +309,10 @@ static void nvmet_execute_get_log_page_rmi(struct nvmet_req *req) · 308 unmodified lines
309309 }
310310
311311 log = kzalloc_obj(*log);
312− if (!log)
312+ if (!log) {
313+ status = NVME_SC_INTERNAL;
313314 goto out;
315+ }
314316
315317 log->endgid = req->cmd->get_log_page.lsi;
316318 disk = req->ns->bdev->bd_disk;
@@ -958,7 +960,7 @@ static void nvmet_execute_identify_nslist(struct nvmet_req *req, bool match_css) · 641 unmodified lines
958960 nvmet_for_each_enabled_ns(&ctrl->subsys->namespaces, idx, ns) {
959961 if (ns->nsid <= min_nsid)
960962 continue;
961− if (match_css && req->ns->csi != req->cmd->identify.csi)
963+ if (match_css && ns->csi != req->cmd->identify.csi)
962964 continue;
963965 list[i++] = cpu_to_le32(ns->nsid);
964966 if (i == buf_size / sizeof(__le32))
@@ -1335,7 +1337,7 @@ static u16 nvmet_set_feat_arbitration(struct nvmet_req *req) · 370 unmodified lines
13351337
13361338 void nvmet_execute_set_features(struct nvmet_req *req)
13371339 {
1338− struct nvmet_subsys *subsys = nvmet_req_subsys(req);
1340+ struct nvmet_ctrl *ctrl = nvmet_req_ctrl(req);
13391341 u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
13401342 u32 cdw11 = le32_to_cpu(req->cmd->common.cdw11);
13411343 u16 status = 0;
@@ -1357,7 +1359,7 @@ void nvmet_execute_set_features(struct nvmet_req *req) · 15 unmodified lines
13571359 break;
13581360 }
13591361 nvmet_set_result(req,
1360− (subsys->max_qid - 1) | ((subsys->max_qid - 1) << 16));
1362+ (ctrl->max_qid - 1) | ((ctrl->max_qid - 1) << 16));
13611363 break;
13621364 case NVME_FEAT_IRQ_COALESCE:
13631365 status = nvmet_set_feat_irq_coalesce(req);
@@ -1494,7 +1496,7 @@ void nvmet_get_feat_async_event(struct nvmet_req *req) · 130 unmodified lines
14941496
14951497 void nvmet_execute_get_features(struct nvmet_req *req)
14961498 {
1497− struct nvmet_subsys *subsys = nvmet_req_subsys(req);
1499+ struct nvmet_ctrl *ctrl = nvmet_req_ctrl(req);
14981500 u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
14991501 u16 status = 0;
15001502
@@ -1534,7 +1536,7 @@ void nvmet_execute_get_features(struct nvmet_req *req) · 33 unmodified lines
15341536 break;
15351537 case NVME_FEAT_NUM_QUEUES:
15361538 nvmet_set_result(req,
1537− (subsys->max_qid-1) | ((subsys->max_qid-1) << 16));
1539+ (ctrl->max_qid-1) | ((ctrl->max_qid-1) << 16));
15381540 break;
15391541 case NVME_FEAT_KATO:
15401542 nvmet_get_feat_kato(req);
drivers/nvme/target/configfs.c+6−4
@@ -312,15 +312,17 @@ static ssize_t nvmet_param_mdts_store(struct config_item *item, · 311 unmodified lines
312312 const char *page, size_t count)
313313 {
314314 struct nvmet_port *port = to_nvmet_port(item);
315− int ret;
315+ int ret, mdts;
316316
317317 if (nvmet_is_port_enabled(port, __func__))
318318 return -EACCES;
319− ret = kstrtoint(page, 0, &port->mdts);
320− if (ret) {
321− pr_err("Invalid value '%s' for mdts\n", page);
319+ ret = kstrtoint(page, 0, &mdts);
320+ if (ret || mdts < 0 || mdts > NVMET_MAX_MDTS) {
321+ pr_err("Invalid value '%s' for mdts, should be 0-%d\n",
322+ page, NVMET_MAX_MDTS);
322323 return -EINVAL;
323324 }
325+ port->mdts = mdts;
324326 return count;
325327 }
326328
drivers/nvme/target/core.c+31−23
@@ -610,12 +610,14 @@ int nvmet_ns_enable(struct nvmet_ns *ns) · 609 unmodified lines
610610 goto out_dev_put;
611611 }
612612
613− if (percpu_ref_init(&ns->ref, nvmet_destroy_namespace, 0, GFP_KERNEL))
613+ ret = percpu_ref_init(&ns->ref, nvmet_destroy_namespace, 0, GFP_KERNEL);
614+ if (ret)
614615 goto out_pr_exit;
615616
616617 nvmet_ns_changed(subsys, ns->nsid);
617618 ns->enabled = true;
618619 xa_set_mark(&subsys->namespaces, ns->nsid, NVMET_NS_ENABLED);
620+ nvmet_debugfs_ns_setup(ns);
619621 ret = 0;
620622 out_unlock:
621623 mutex_unlock(&subsys->lock);
@@ -642,6 +644,7 @@ void nvmet_ns_disable(struct nvmet_ns *ns) · 20 unmodified lines
642644
643645 ns->enabled = false;
644646 xa_clear_mark(&subsys->namespaces, ns->nsid, NVMET_NS_ENABLED);
647+ nvmet_debugfs_ns_free(ns);
645648
646649 list_for_each_entry(ctrl, &subsys->ctrls, subsys_entry)
647650 pci_dev_put(radix_tree_delete(&ctrl->p2p_ns_map, ns->nsid));
@@ -875,7 +878,7 @@ u16 nvmet_check_cqid(struct nvmet_ctrl *ctrl, u16 cqid, bool create) · 227 unmodified lines
875878 if (!ctrl->cqs)
876879 return NVME_SC_INTERNAL | NVME_STATUS_DNR;
877880
878− if (cqid > ctrl->subsys->max_qid)
881+ if (cqid > ctrl->max_qid)
879882 return NVME_SC_QID_INVALID | NVME_STATUS_DNR;
880883
881884 if ((create && ctrl->cqs[cqid]) || (!create && !ctrl->cqs[cqid]))
@@ -923,7 +926,7 @@ u16 nvmet_check_sqid(struct nvmet_ctrl *ctrl, u16 sqid, · 41 unmodified lines
923926 if (!ctrl->sqs)
924927 return NVME_SC_INTERNAL | NVME_STATUS_DNR;
925928
926− if (sqid > ctrl->subsys->max_qid)
929+ if (sqid > ctrl->max_qid)
927930 return NVME_SC_QID_INVALID | NVME_STATUS_DNR;
928931
929932 if ((create && ctrl->sqs[sqid]) ||
@@ -1652,11 +1655,29 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) · 722 unmodified lines
16521655 if (!ctrl->changed_ns_list)
16531656 goto out_free_ctrl;
16541657
1655− ctrl->sqs = kzalloc_objs(struct nvmet_sq *, subsys->max_qid + 1);
1658+ /*
1659+ * Discovery controllers may use some arbitrary high value
1660+ * in order to cleanup stale discovery sessions
1661+ */
1662+ if (nvmet_is_disc_subsys(ctrl->subsys) && !kato)
1663+ kato = NVMET_DISC_KATO_MS;
1664+
1665+ /* keep-alive timeout in seconds */
1666+ ctrl->kato = DIV_ROUND_UP(kato, 1000);
1667+
1668+ ctrl->err_counter = 0;
1669+ spin_lock_init(&ctrl->error_lock);
1670+
1671+ down_read(&nvmet_config_sem);
1672+ mutex_lock(&subsys->lock);
1673+
1674+ ctrl->max_qid = subsys->max_qid;
1675+
1676+ ctrl->sqs = kzalloc_objs(struct nvmet_sq *, ctrl->max_qid + 1);
16561677 if (!ctrl->sqs)
16571678 goto out_free_changed_ns_list;
16581679
1659− ctrl->cqs = kzalloc_objs(struct nvmet_cq *, subsys->max_qid + 1);
1680+ ctrl->cqs = kzalloc_objs(struct nvmet_cq *, ctrl->max_qid + 1);
16601681 if (!ctrl->cqs)
16611682 goto out_free_sqs;
16621683
@@ -1669,22 +1690,6 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) · 6 unmodified lines
16691690 }
16701691 ctrl->cntlid = ret;
16711692
1672− /*
1673− * Discovery controllers may use some arbitrary high value
1674− * in order to cleanup stale discovery sessions
1675− */
1676− if (nvmet_is_disc_subsys(ctrl->subsys) && !kato)
1677− kato = NVMET_DISC_KATO_MS;
1678−
1679− /* keep-alive timeout in seconds */
1680− ctrl->kato = DIV_ROUND_UP(kato, 1000);
1681−
1682− ctrl->err_counter = 0;
1683− spin_lock_init(&ctrl->error_lock);
1684−
1685− nvmet_start_keep_alive_timer(ctrl);
1686−
1687− mutex_lock(&subsys->lock);
16881693 ret = nvmet_ctrl_init_pr(ctrl);
16891694 if (ret)
16901695 goto init_pr_fail;
@@ -1692,6 +1697,9 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) · 1 unmodified line
16921697 nvmet_setup_p2p_ns_map(ctrl, args->p2p_client);
16931698 nvmet_debugfs_ctrl_setup(ctrl);
16941699 mutex_unlock(&subsys->lock);
1700+ up_read(&nvmet_config_sem);
1701+
1702+ nvmet_start_keep_alive_timer(ctrl);
16951703
16961704 if (args->hostid)
16971705 uuid_copy(&ctrl->hostid, args->hostid);
@@ -1721,14 +1729,14 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) · 23 unmodified lines
17211729 return ctrl;
17221730
17231731 init_pr_fail:
1724− mutex_unlock(&subsys->lock);
1725− nvmet_stop_keep_alive_timer(ctrl);
17261732 ida_free(&cntlid_ida, ctrl->cntlid);
17271733 out_free_cqs:
17281734 kfree(ctrl->cqs);
17291735 out_free_sqs:
17301736 kfree(ctrl->sqs);
17311737 out_free_changed_ns_list:
1738+ mutex_unlock(&subsys->lock);
1739+ up_read(&nvmet_config_sem);
17321740 kfree(ctrl->changed_ns_list);
17331741 out_free_ctrl:
17341742 kfree(ctrl);
drivers/nvme/target/debugfs.c+103−0
@@ -153,6 +153,109 @@ static int nvmet_ctrl_tls_concat_show(struct seq_file *m, void *p) · 152 unmodified lines
153153 NVMET_DEBUGFS_ATTR(nvmet_ctrl_tls_concat);
154154 #endif
155155
156+static const char *const nvmet_pr_type_names[] = {
157+ [NVME_PR_WRITE_EXCLUSIVE] = "write_exclusive",
158+ [NVME_PR_EXCLUSIVE_ACCESS] = "exclusive_access",
159+ [NVME_PR_WRITE_EXCLUSIVE_REG_ONLY] = "write_exclusive_reg_only",
160+ [NVME_PR_EXCLUSIVE_ACCESS_REG_ONLY] = "exclusive_access_reg_only",
161+ [NVME_PR_WRITE_EXCLUSIVE_ALL_REGS] = "write_exclusive_all_regs",
162+ [NVME_PR_EXCLUSIVE_ACCESS_ALL_REGS] = "exclusive_access_all_regs",
163+};
164+
165+static const char *nvmet_pr_type_to_str(enum nvme_pr_type type)
166+{
167+ if (type < ARRAY_SIZE(nvmet_pr_type_names) &&
168+ nvmet_pr_type_names[type])
169+ return nvmet_pr_type_names[type];
170+ return "unknown";
171+}
172+
173+static const char *const nvmet_pr_notify_names[] = {
174+ [NVME_PR_NOTIFY_BIT_REG_PREEMPTED] = "reg_preempted",
175+ [NVME_PR_NOTIFY_BIT_RESV_RELEASED] = "resv_released",
176+ [NVME_PR_NOTIFY_BIT_RESV_PREEMPTED] = "resv_preempted",
177+};
178+
179+static void nvmet_pr_notify_mask_to_str(struct seq_file *m, unsigned long mask)
180+{
181+ bool sep = false;
182+ int i;
183+
184+ if (!mask) {
185+ seq_puts(m, "none");
186+ return;
187+ }
188+
189+ for (i = 0; i < ARRAY_SIZE(nvmet_pr_notify_names); i++) {
190+ if (!test_bit(i, &mask) || !nvmet_pr_notify_names[i])
191+ continue;
192+ if (sep)
193+ seq_putc(m, ',');
194+ seq_puts(m, nvmet_pr_notify_names[i]);
195+ sep = true;
196+ }
197+}
198+
199+static int nvmet_ns_pr_show(struct seq_file *m, void *p)
200+{
201+ struct nvmet_ns *ns = m->private;
202+ struct nvmet_pr *pr = &ns->pr;
203+ struct nvmet_pr_registrant *holder, *reg;
204+
205+ seq_printf(m, "enable=%d\n", pr->enable);
206+ if (!pr->enable)
207+ return 0;
208+
209+ seq_printf(m, "generation=%u\n", atomic_read(&pr->generation));
210+ seq_puts(m, "notify_mask=");
211+ nvmet_pr_notify_mask_to_str(m, pr->notify_mask);
212+ seq_putc(m, '\n');
213+
214+ rcu_read_lock();
215+ holder = rcu_dereference(pr->holder);
216+ if (holder) {
217+ seq_printf(m, "rtype=%s\n",
218+ nvmet_pr_type_to_str(holder->rtype));
219+ seq_printf(m, "holder=%pUb,0x%llx\n",
220+ &holder->hostid, holder->rkey);
221+ } else {
222+ seq_puts(m, "rtype=none\n");
223+ seq_puts(m, "holder=none\n");
224+ }
225+
226+ list_for_each_entry_rcu(reg, &pr->registrant_list, entry) {
227+ seq_printf(m, "reg=%pUb,0x%llx\n",
228+ &reg->hostid, reg->rkey);
229+ }
230+ rcu_read_unlock();
231+
232+ return 0;
233+}
234+NVMET_DEBUGFS_ATTR(nvmet_ns_pr);
235+
236+void nvmet_debugfs_ns_setup(struct nvmet_ns *ns)
237+{
238+ char name[16];
239+ struct dentry *parent = ns->subsys->debugfs_dir;
240+
241+ if (!parent)
242+ return;
243+ snprintf(name, sizeof(name), "ns%u", ns->nsid);
244+ ns->debugfs_dir = debugfs_create_dir(name, parent);
245+ if (IS_ERR(ns->debugfs_dir)) {
246+ ns->debugfs_dir = NULL;
247+ return;
248+ }
249+ debugfs_create_file("reservation", 0400, ns->debugfs_dir, ns,
250+ &nvmet_ns_pr_fops);
251+}
252+
253+void nvmet_debugfs_ns_free(struct nvmet_ns *ns)
254+{
255+ debugfs_remove_recursive(ns->debugfs_dir);
256+ ns->debugfs_dir = NULL;
257+}
258+
156259 int nvmet_debugfs_ctrl_setup(struct nvmet_ctrl *ctrl)
157260 {
158261 char name[32];
drivers/nvme/target/debugfs.h+5−0
@@ -14,6 +14,8 @@ int nvmet_debugfs_subsys_setup(struct nvmet_subsys *subsys); · 13 unmodified lines
1414 void nvmet_debugfs_subsys_free(struct nvmet_subsys *subsys);
1515 int nvmet_debugfs_ctrl_setup(struct nvmet_ctrl *ctrl);
1616 void nvmet_debugfs_ctrl_free(struct nvmet_ctrl *ctrl);
17+void nvmet_debugfs_ns_setup(struct nvmet_ns *ns);
18+void nvmet_debugfs_ns_free(struct nvmet_ns *ns);
1719
1820 int __init nvmet_init_debugfs(void);
1921 void nvmet_exit_debugfs(void);
@@ -30,6 +32,9 @@ static inline int nvmet_debugfs_ctrl_setup(struct nvmet_ctrl *ctrl) · 10 unmodified lines
3032 }
3133 static inline void nvmet_debugfs_ctrl_free(struct nvmet_ctrl *ctrl) {}
3234
35+static inline void nvmet_debugfs_ns_setup(struct nvmet_ns *ns) {}
36+static inline void nvmet_debugfs_ns_free(struct nvmet_ns *ns) {}
37+
3338 static inline int __init nvmet_init_debugfs(void)
3439 {
3540 return 0;
drivers/nvme/target/fabrics-cmd-auth.c+13−4
@@ -9,6 +9,7 @@ · 8 unmodified lines
99 #include <linux/random.h>
1010 #include <linux/nvme-auth.h>
1111 #include <crypto/kpp.h>
12+#include <crypto/utils.h>
1213 #include "nvmet.h"
1314
1415 static void nvmet_auth_expired_work(struct work_struct *work)
@@ -30,12 +31,16 @@ void nvmet_auth_sq_init(struct nvmet_sq *sq) · 15 unmodified lines
3031 sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
3132 }
3233
33−static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d)
34+static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d, u32 tl)
3435 {
3536 struct nvmet_ctrl *ctrl = req->sq->ctrl;
3637 struct nvmf_auth_dhchap_negotiate_data *data = d;
3738 int i, hash_id = 0, fallback_hash_id = 0, dhgid, fallback_dhgid;
3839
40+ if (tl < sizeof(*data) +
41+ sizeof(struct nvmf_auth_dhchap_protocol_descriptor))
42+ return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
43+
3944 pr_debug("%s: ctrl %d qid %d: data sc_d %d napd %d authid %d halen %d dhlen %d\n",
4045 __func__, ctrl->cntlid, req->sq->qid,
4146 data->sc_c, data->napd, data->auth_protocol[0].dhchap.authid,
@@ -71,6 +76,10 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d) · 29 unmodified lines
7176 NVME_AUTH_DHCHAP_AUTH_ID)
7277 return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
7378
79+ if (data->auth_protocol[0].dhchap.dhlen > NVME_AUTH_DHCHAP_MAX_DH_IDS ||
80+ data->auth_protocol[0].dhchap.halen > NVME_AUTH_DHCHAP_MAX_HASH_IDS)
81+ return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
82+
7483 for (i = 0; i < data->auth_protocol[0].dhchap.halen; i++) {
7584 u8 host_hmac_id = data->auth_protocol[0].dhchap.idlist[i];
7685
@@ -177,7 +186,7 @@ static u8 nvmet_auth_reply(struct nvmet_req *req, void *d, u32 tl) · 100 unmodified lines
177186 return NVME_AUTH_DHCHAP_FAILURE_FAILED;
178187 }
179188
180− if (memcmp(data->rval, response, data->hl)) {
189+ if (crypto_memneq(data->rval, response, data->hl)) {
181190 pr_info("ctrl %d qid %d host response mismatch\n",
182191 ctrl->cntlid, req->sq->qid);
183192 pr_debug("ctrl %d qid %d rval %*ph\n",
@@ -316,7 +325,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req) · 132 unmodified lines
316325 } else if (data->auth_id != req->sq->dhchap_step)
317326 goto done_failure1;
318327 /* Validate negotiation parameters */
319− dhchap_status = nvmet_auth_negotiate(req, d);
328+ dhchap_status = nvmet_auth_negotiate(req, d, tl);
320329 if (dhchap_status == 0)
321330 req->sq->dhchap_step =
322331 NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE;
@@ -557,7 +566,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req) · 234 unmodified lines
557566 return;
558567 }
559568
560− d = kmalloc(al, GFP_KERNEL);
569+ d = kzalloc(al, GFP_KERNEL);
561570 if (!d) {
562571 status = NVME_SC_INTERNAL;
563572 goto done;
drivers/nvme/target/fabrics-cmd.c+1−1
@@ -370,7 +370,7 @@ static void nvmet_execute_io_connect(struct nvmet_req *req) · 369 unmodified lines
370370 goto out;
371371 }
372372
373− if (unlikely(qid > ctrl->subsys->max_qid)) {
373+ if (unlikely(qid > ctrl->max_qid)) {
374374 pr_warn("invalid queue id (%d)\n", qid);
375375 status = NVME_SC_CONNECT_INVALID_PARAM | NVME_STATUS_DNR;
376376 req->cqe->result.u32 = IPO_IATTR_CONNECT_SQE(qid);
drivers/nvme/target/fc.c+1−1
@@ -566,7 +566,7 @@ out_fail: · 565 unmodified lines
566566 list_del(&iod->ls_rcv_list);
567567 }
568568
569− kfree(iod);
569+ kfree(tgtport->iod);
570570
571571 return -EFAULT;
572572 }
drivers/nvme/target/nvmet.h+9−0
@@ -128,6 +128,9 @@ struct nvmet_ns { · 127 unmodified lines
128128 u8 csi;
129129 struct nvmet_pr pr;
130130 struct xarray pr_per_ctrl_refs;
131+#ifdef CONFIG_NVME_TARGET_DEBUGFS
132+ struct dentry *debugfs_dir;
133+#endif
131134 };
132135
133136 static inline struct nvmet_ns *to_nvmet_ns(struct config_item *item)
@@ -265,6 +268,7 @@ struct nvmet_ctrl { · 131 unmodified lines
265268
266269 uuid_t hostid;
267270 u16 cntlid;
271+ u16 max_qid;
268272 u32 kato;
269273
270274 struct nvmet_port *port;
@@ -753,6 +757,11 @@ static inline struct nvmet_subsys *nvmet_req_subsys(struct nvmet_req *req) · 482 unmodified lines
753757 return req->sq->ctrl->subsys;
754758 }
755759
760+static inline struct nvmet_ctrl *nvmet_req_ctrl(struct nvmet_req *req)
761+{
762+ return req->sq->ctrl;
763+}
764+
756765 static inline bool nvmet_is_disc_subsys(struct nvmet_subsys *subsys)
757766 {
758767 return subsys->type != NVME_NQN_NVME;
drivers/nvme/target/passthru.c+9−0
@@ -53,13 +53,22 @@ static u16 nvmet_passthru_override_id_descs(struct nvmet_req *req) · 52 unmodified lines
5353 for (pos = 0; pos < NVME_IDENTIFY_DATA_SIZE; pos += len) {
5454 struct nvme_ns_id_desc *cur = data + pos;
5555
56+ if (pos + sizeof(*cur) > NVME_IDENTIFY_DATA_SIZE)
57+ break;
58+
5659 if (cur->nidl == 0)
5760 break;
61+
5862 if (cur->nidt == NVME_NIDT_CSI) {
63+ if (pos + sizeof(*cur) + NVME_NIDT_CSI_LEN >
64+ NVME_IDENTIFY_DATA_SIZE)
65+ break;
66+
5967 memcpy(&csi, cur + 1, NVME_NIDT_CSI_LEN);
6068 csi_seen = true;
6169 break;
6270 }
71+
6372 len = sizeof(struct nvme_ns_id_desc) + cur->nidl;
6473 }
6574
drivers/nvme/target/pci-epf.c+8−4
@@ -1339,6 +1339,7 @@ err_unmap_queue: · 1338 unmodified lines
13391339 nvmet_pci_epf_mem_unmap(ctrl->nvme_epf, &cq->pci_map);
13401340 err_internal:
13411341 status = NVME_SC_INTERNAL | NVME_STATUS_DNR;
1342+ nvmet_cq_put(&cq->nvme_cq);
13421343 err:
13431344 if (test_and_clear_bit(NVMET_PCI_EPF_Q_IRQ_ENABLED, &cq->flags))
13441345 nvmet_pci_epf_remove_irq_vector(ctrl, cq->vector);
@@ -1594,6 +1595,7 @@ static void nvmet_pci_epf_exec_iod_work(struct work_struct *work) · 249 unmodified lines
15941595 struct nvmet_pci_epf_iod *iod =
15951596 container_of(work, struct nvmet_pci_epf_iod, work);
15961597 struct nvmet_req *req = &iod->req;
1598+ bool no_wait;
15971599 int ret;
15981600
15991601 if (!iod->ctrl->link_up) {
@@ -1638,14 +1640,16 @@ static void nvmet_pci_epf_exec_iod_work(struct work_struct *work) · 38 unmodified lines
16381640 }
16391641 }
16401642
1641− req->execute(req);
1642−
16431643 /*
16441644 * If we do not have data to transfer after the command execution
16451645 * finishes, nvmet_pci_epf_queue_response() will complete the command
16461646 * directly. No need to wait for the completion in this case.
16471647 */
1648− if (!iod->data_len || iod->dma_dir != DMA_TO_DEVICE)
1648+ no_wait = !iod->data_len || iod->dma_dir != DMA_TO_DEVICE;
1649+
1650+ req->execute(req);
1651+
1652+ if (no_wait)
16491653 return;
16501654
16511655 wait_for_completion(&iod->done);
@@ -2077,7 +2081,7 @@ static int nvmet_pci_epf_create_ctrl(struct nvmet_pci_epf *nvme_epf, · 425 unmodified lines
20772081 }
20782082
20792083 /* Allocate our queues, up to the maximum number. */
2080− ctrl->nr_queues = min(ctrl->tctrl->subsys->max_qid + 1, max_nr_queues);
2084+ ctrl->nr_queues = min(ctrl->tctrl->max_qid + 1, max_nr_queues);
20812085 ret = nvmet_pci_epf_alloc_queues(ctrl);
20822086 if (ret)
20832087 goto out_put_ctrl;
drivers/nvme/target/pr.c+30−7
@@ -8,7 +8,7 @@ · 7 unmodified lines
88 #include <linux/unaligned.h>
99 #include "nvmet.h"
1010
11−#define NVMET_PR_NOTIFI_MASK_ALL \
11+#define NVMET_PR_NOTIFY_MASK_ALL \
1212 (1 << NVME_PR_NOTIFY_BIT_REG_PREEMPTED | \
1313 1 << NVME_PR_NOTIFY_BIT_RESV_RELEASED | \
1414 1 << NVME_PR_NOTIFY_BIT_RESV_PREEMPTED)
@@ -44,7 +44,7 @@ u16 nvmet_set_feat_resv_notif_mask(struct nvmet_req *req, u32 mask) · 29 unmodified lines
4444 unsigned long idx;
4545 u16 status;
4646
47− if (mask & ~(NVMET_PR_NOTIFI_MASK_ALL)) {
47+ if (mask & ~(NVMET_PR_NOTIFY_MASK_ALL)) {
4848 req->error_loc = offsetof(struct nvme_common_command, cdw11);
4949 return NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
5050 }
@@ -169,7 +169,7 @@ static void nvmet_pr_resv_released(struct nvmet_pr *pr, uuid_t *hostid) · 118 unmodified lines
169169 nvmet_pr_add_resv_log(ctrl,
170170 NVME_PR_LOG_RESERVATION_RELEASED, ns->nsid);
171171 nvmet_add_async_event(ctrl, NVME_AER_CSS,
172− NVME_AEN_RESV_LOG_PAGE_AVALIABLE,
172+ NVME_AEN_RESV_LOG_PAGE_AVAILABLE,
173173 NVME_LOG_RESERVATION);
174174 }
175175 }
@@ -188,7 +188,7 @@ static void nvmet_pr_send_event_to_host(struct nvmet_pr *pr, uuid_t *hostid, · 12 unmodified lines
188188 if (uuid_equal(hostid, &ctrl->hostid)) {
189189 nvmet_pr_add_resv_log(ctrl, log_type, ns->nsid);
190190 nvmet_add_async_event(ctrl, NVME_AER_CSS,
191− NVME_AEN_RESV_LOG_PAGE_AVALIABLE,
191+ NVME_AEN_RESV_LOG_PAGE_AVAILABLE,
192192 NVME_LOG_RESERVATION);
193193 }
194194 }
@@ -201,7 +201,7 @@ static void nvmet_pr_resv_preempted(struct nvmet_pr *pr, uuid_t *hostid) · 6 unmodified lines
201201 return;
202202
203203 nvmet_pr_send_event_to_host(pr, hostid,
204− NVME_PR_LOG_RESERVATOIN_PREEMPTED);
204+ NVME_PR_LOG_RESERVATION_PREEMPTED);
205205 }
206206
207207 static void nvmet_pr_registration_preempted(struct nvmet_pr *pr,
@@ -355,9 +355,15 @@ static u16 nvmet_pr_replace(struct nvmet_req *req, · 147 unmodified lines
355355 u16 status = NVME_SC_RESERVATION_CONFLICT | NVME_STATUS_DNR;
356356 struct nvmet_ctrl *ctrl = req->sq->ctrl;
357357 struct nvmet_pr *pr = &req->ns->pr;
358− struct nvmet_pr_registrant *reg;
358+ struct nvmet_pr_registrant *reg, *new = NULL;
359359 u64 nrkey = le64_to_cpu(d->nrkey);
360360
361+ if (ignore_key && nrkey) {
362+ new = kzalloc_obj(*new);
363+ if (!new)
364+ return NVME_SC_INTERNAL;
365+ }
366+
361367 down(&pr->pr_sem);
362368 list_for_each_entry_rcu(reg, &pr->registrant_list, entry) {
363369 if (uuid_equal(&reg->hostid, &ctrl->hostid)) {
@@ -365,9 +371,26 @@ static u16 nvmet_pr_replace(struct nvmet_req *req, · 1 unmodified line
365371 status = nvmet_pr_update_reg_attr(pr, reg,
366372 nvmet_pr_update_reg_rkey,
367373 &nrkey);
368− break;
374+ goto free_data;
375+ }
376+ }
377+
378+ if (ignore_key) {
379+ if (!nrkey) {
380+ status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
381+ goto free_data;
369382 }
383+ INIT_LIST_HEAD(&new->entry);
384+ new->rkey = nrkey;
385+ uuid_copy(&new->hostid, &ctrl->hostid);
386+ list_add_tail_rcu(&new->entry, &pr->registrant_list);
387+ status = NVME_SC_SUCCESS;
388+ goto out;
370389 }
390+
391+free_data:
392+ kfree(new);
393+out:
371394 up(&pr->pr_sem);
372395 return status;
373396 }
drivers/nvme/target/rdma.c+28−3
@@ -657,18 +657,25 @@ static void nvmet_rdma_rw_ctx_destroy(struct nvmet_rdma_rsp *rsp) · 656 unmodified lines
657657 req->sg, req->sg_cnt, nvmet_data_dir(req));
658658 }
659659
660−static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
660+static void nvmet_rdma_free_rsp_resources(struct nvmet_rdma_rsp *rsp)
661661 {
662662 struct nvmet_rdma_queue *queue = rsp->queue;
663663
664− atomic_add(1 + rsp->n_rdma, &queue->sq_wr_avail);
665−
666664 if (rsp->n_rdma)
667665 nvmet_rdma_rw_ctx_destroy(rsp);
668666
669667 if (rsp->req.sg < rsp->cmd->inline_sg ||
670668 rsp->req.sg >= rsp->cmd->inline_sg + queue->dev->inline_page_count)
671669 nvmet_req_free_sgls(&rsp->req);
670+}
671+
672+static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
673+{
674+ struct nvmet_rdma_queue *queue = rsp->queue;
675+
676+ atomic_add(1 + rsp->n_rdma, &queue->sq_wr_avail);
677+
678+ nvmet_rdma_free_rsp_resources(rsp);
672679
673680 if (unlikely(!list_empty_careful(&queue->rsp_wr_wait_list)))
674681 nvmet_rdma_process_wr_wait_list(queue);
@@ -1338,9 +1345,27 @@ err_destroy_cq: · 663 unmodified lines
13381345 goto out;
13391346 }
13401347
1348+static bool nvmet_rdma_reclaim_rsp(struct sbitmap *sb, unsigned int bitnr,
1349+ void *data)
1350+{
1351+ struct nvmet_rdma_queue *queue = data;
1352+
1353+ nvmet_rdma_free_rsp_resources(&queue->rsps[bitnr]);
1354+
1355+ return true;
1356+}
1357+
13411358 static void nvmet_rdma_destroy_queue_ib(struct nvmet_rdma_queue *queue)
13421359 {
13431360 ib_drain_qp(queue->qp);
1361+
1362+ /*
1363+ * Reclaim resources of a response that is still in-flight when the
1364+ * queue is being torn down. This happens when the connection was
1365+ * forcefully disconnected while an I/O is in flight.
1366+ */
1367+ sbitmap_for_each_set(&queue->rsp_tags, nvmet_rdma_reclaim_rsp, queue);
1368+
13441369 if (queue->cm_id)
13451370 rdma_destroy_id(queue->cm_id);
13461371 ib_destroy_qp(queue->qp);
drivers/nvme/target/tcp.c+17−2
@@ -422,6 +422,19 @@ static int nvmet_tcp_map_data(struct nvmet_tcp_cmd *cmd) · 421 unmodified lines
422422 if (!len)
423423 return 0;
424424
425+ /*
426+ * inline_data_size only bounds the in-capsule (type 0x01) SGL
427+ * descriptor below. A non-inline transport SGL data-block
428+ * descriptor skips that check entirely and would otherwise reach
429+ * sgl_alloc() with an attacker-controlled len of up to 4 GiB,
430+ * pinning that much kernel memory for a command that may never
431+ * complete. Bound every descriptor type here, before allocating
432+ * anything, using the same ceiling this file already applies to
433+ * per-PDU H2C data.
434+ */
435+ if (len > NVMET_TCP_MAXH2CDATA)
436+ return NVME_SC_SGL_INVALID_DATA | NVME_STATUS_DNR;
437+
425438 if (sgl->type == ((NVME_SGL_FMT_DATA_DESC << 4) |
426439 NVME_SGL_FMT_OFFSET)) {
427440 if (!nvme_is_write(cmd->req.cmd))
@@ -433,13 +446,15 @@ static int nvmet_tcp_map_data(struct nvmet_tcp_cmd *cmd) · 5 unmodified lines
433446 }
434447 cmd->req.transfer_len += len;
435448
436− cmd->req.sg = sgl_alloc(len, GFP_KERNEL, &cmd->req.sg_cnt);
449+ cmd->req.sg = sgl_alloc(len, GFP_KERNEL | __GFP_NOWARN,
450+ &cmd->req.sg_cnt);
437451 if (!cmd->req.sg)
438452 return NVME_SC_INTERNAL;
439453 cmd->cur_sg = cmd->req.sg;
440454
441455 if (nvmet_tcp_has_data_in(cmd)) {
442− cmd->iov = kmalloc_objs(*cmd->iov, cmd->req.sg_cnt);
456+ cmd->iov = kmalloc_objs(*cmd->iov, cmd->req.sg_cnt,
457+ GFP_KERNEL | __GFP_NOWARN);
443458 if (!cmd->iov)
444459 goto err;
445460 }
drivers/nvme/target/zns.c+11−4
@@ -116,7 +116,7 @@ void nvmet_execute_identify_ns_zns(struct nvmet_req *req) · 115 unmodified lines
116116 mutex_unlock(&req->ns->subsys->lock);
117117 }
118118
119− if (!bdev_is_zoned(req->ns->bdev)) {
119+ if (!req->ns->bdev || !bdev_is_zoned(req->ns->bdev)) {
120120 status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
121121 req->error_loc = offsetof(struct nvme_identify, nsid);
122122 goto out;
@@ -295,11 +295,18 @@ static void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w) · 172 unmodified lines
295295 }
296296
297297 /*
298− * When partial bit is set nr_zones must indicate the number of zone
299− * descriptors actually transferred.
298+ * Partial report (PR bit set): the host accepts an incomplete listing,
299+ * so cap Number of Zones to the descriptors that fit in the buffer.
300+ * Full report (PR bit clear): Number of Zones is the match count; fail
301+ * if the buffer cannot hold every matching zone descriptor.
300302 */
301− if (req->cmd->zmr.pr)
303+ if (req->cmd->zmr.pr) {
302304 rz_data.nr_zones = min(rz_data.nr_zones, rz_data.out_nr_zones);
305+ } else if (rz_data.nr_zones > rz_data.out_nr_zones) {
306+ req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, numd);
307+ status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
308+ goto out;
309+ }
303310
304311 nr_zones = cpu_to_le64(rz_data.nr_zones);
305312 status = nvmet_copy_to_sgl(req, 0, &nr_zones, sizeof(nr_zones));
include/linux/list.h+9−0
@@ -33,6 +33,14 @@ · 32 unmodified lines
3333 #define LIST_HEAD(name) \
3434 struct list_head name = LIST_HEAD_INIT(name)
3535
36+/**
37+ * LIST_HEAD_GUARDED - define a &struct list_head annotated with __guarded_by()
38+ * @name: name of the list_head
39+ * @lock: lock protecting the list
40+ */
41+#define LIST_HEAD_GUARDED(name, lock) \
42+ __guarded_by(&(lock)) LIST_HEAD(name)
43+
3644 /**
3745 * INIT_LIST_HEAD - Initialize a list_head structure
3846 * @list: list_head structure to be initialized.
@@ -436,6 +444,7 @@ static inline void list_del_init_careful(struct list_head *entry) · 397 unmodified lines
436444 * if another CPU could re-list_add() it.
437445 */
438446 static inline int list_empty_careful(const struct list_head *head)
447+ __context_unsafe(/* intentional lockless access to @head */)
439448 {
440449 struct list_head *next = smp_load_acquire(&head->next);
441450 return list_is_head(next, head) && (next == READ_ONCE(head->prev));
include/linux/nvme.h+2−2
@@ -2272,14 +2272,14 @@ struct nvme_completion { · 2271 unmodified lines
22722272 #define NVME_TERTIARY(ver) ((ver) & 0xff)
22732273
22742274 enum {
2275− NVME_AEN_RESV_LOG_PAGE_AVALIABLE = 0x00,
2275+ NVME_AEN_RESV_LOG_PAGE_AVAILABLE = 0x00,
22762276 };
22772277
22782278 enum {
22792279 NVME_PR_LOG_EMPTY_LOG_PAGE = 0x00,
22802280 NVME_PR_LOG_REGISTRATION_PREEMPTED = 0x01,
22812281 NVME_PR_LOG_RESERVATION_RELEASED = 0x02,
2282− NVME_PR_LOG_RESERVATOIN_PREEMPTED = 0x03,
2282+ NVME_PR_LOG_RESERVATION_PREEMPTED = 0x03,
22832283 };
22842284
22852285 enum {

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t