History

Merge tag 'net-next-6.4' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next

Pull networking updates from Paolo Abeni: "Core: - Introduce a config option to tweak MAX_SKB_FRAGS. Increasing the default value allows for better BIG TCP performances - Reduce compound page head access for zero-copy data transfers - RPS/RFS improvements, avoiding unneeded NET_RX_SOFTIRQ when possible - Threaded NAPI improvements, adding defer skb free support and unneeded softirq avoidance - Address dst_entry reference count scalability issues, via false sharing avoidance and optimize refcount tracking - Add lockless accesses annotation to sk_err[_soft] - Optimize again the skb struct layout - Extends the skb drop reasons to make it usable by multiple subsystems - Better const qualifier awareness for socket casts BPF: - Add skb and XDP typed dynptrs which allow BPF programs for more ergonomic and less brittle iteration through data and variable-sized accesses - Add a new BPF netfilter program type and minimal support to hook BPF programs to netfilter hooks such as prerouting or forward - Add more precise memory usage reporting for all BPF map types - Adds support for using {FOU,GUE} encap with an ipip device operating in collect_md mode and add a set of BPF kfuncs for controlling encap params - Allow BPF programs to detect at load time whether a particular kfunc exists or not, and also add support for this in light skeleton - Bigger batch of BPF verifier improvements to prepare for upcoming BPF open-coded iterators allowing for less restrictive looping capabilities - Rework RCU enforcement in the verifier, add kptr_rcu and enforce BPF programs to NULL-check before passing such pointers into kfunc - Add support for kptrs in percpu hashmaps, percpu LRU hashmaps and in local storage maps - Enable RCU semantics for task BPF kptrs and allow referenced kptr tasks to be stored in BPF maps - Add support for refcounted local kptrs to the verifier for allowing shared ownership, useful for adding a node to both the BPF list and rbtree - Add BPF verifier support for ST instructions in convert_ctx_access() which will help new -mcpu=v4 clang flag to start emitting them - Add ARM32 USDT support to libbpf - Improve bpftool's visual program dump which produces the control flow graph in a DOT format by adding C source inline annotations Protocols: - IPv4: Allow adding to IPv4 address a 'protocol' tag. Such value indicates the provenance of the IP address - IPv6: optimize route lookup, dropping unneeded R/W lock acquisition - Add the handshake upcall mechanism, allowing the user-space to implement generic TLS handshake on kernel's behalf - Bridge: support per-{Port, VLAN} neighbor suppression, increasing resilience to nodes failures - SCTP: add support for Fair Capacity and Weighted Fair Queueing schedulers - MPTCP: delay first subflow allocation up to its first usage. This will allow for later better LSM interaction - xfrm: Remove inner/outer modes from input/output path. These are not needed anymore - WiFi: - reduced neighbor report (RNR) handling for AP mode - HW timestamping support - support for randomized auth/deauth TA for PASN privacy - per-link debugfs for multi-link - TC offload support for mac80211 drivers - mac80211 mesh fast-xmit and fast-rx support - enable Wi-Fi 7 (EHT) mesh support Netfilter: - Add nf_tables 'brouting' support, to force a packet to be routed instead of being bridged - Update bridge netfilter and ovs conntrack helpers to handle IPv6 Jumbo packets properly, i.e. fetch the packet length from hop-by-hop extension header. This is needed for BIT TCP support - The iptables 32bit compat interface isn't compiled in by default anymore - Move ip(6)tables builtin icmp matches to the udptcp one. This has the advantage that icmp/icmpv6 match doesn't load the iptables/ip6tables modules anymore when iptables-nft is used - Extended netlink error report for netdevice in flowtables and netdev/chains. Allow for incrementally add/delete devices to netdev basechain. Allow to create netdev chain without device Driver API: - Remove redundant Device Control Error Reporting Enable, as PCI core has already error reporting enabled at enumeration time - Move Multicast DB netlink handlers to core, allowing devices other then bridge to use them - Allow the page_pool to directly recycle the pages from safely localized NAPI - Implement lockless TX queue stop/wake combo macros, allowing for further code de-duplication and sanitization - Add YNL support for user headers and struct attrs - Add partial YNL specification for devlink - Add partial YNL specification for ethtool - Add tc-mqprio and tc-taprio support for preemptible traffic classes - Add tx push buf len param to ethtool, specifies the maximum number of bytes of a transmitted packet a driver can push directly to the underlying device - Add basic LED support for switch/phy - Add NAPI documentation, stop relaying on external links - Convert dsa_master_ioctl() to netdev notifier. This is a preparatory work to make the hardware timestamping layer selectable by user space - Add transceiver support and improve the error messages for CAN-FD controllers New hardware / drivers: - Ethernet: - AMD/Pensando core device support - MediaTek MT7981 SoC - MediaTek MT7988 SoC - Broadcom BCM53134 embedded switch - Texas Instruments CPSW9G ethernet switch - Qualcomm EMAC3 DWMAC ethernet - StarFive JH7110 SoC - NXP CBTX ethernet PHY - WiFi: - Apple M1 Pro/Max devices - RealTek rtl8710bu/rtl8188gu - RealTek rtl8822bs, rtl8822cs and rtl8821cs SDIO chipset - Bluetooth: - Realtek RTL8821CS, RTL8851B, RTL8852BS - Mediatek MT7663, MT7922 - NXP w8997 - Actions Semi ATS2851 - QTI WCN6855 - Marvell 88W8997 - Can: - STMicroelectronics bxcan stm32f429 Drivers: - Ethernet NICs: - Intel (1G, icg): - add tracking and reporting of QBV config errors - add support for configuring max SDU for each Tx queue - Intel (100G, ice): - refactor mailbox overflow detection to support Scalable IOV - GNSS interface optimization - Intel (i40e): - support XDP multi-buffer - nVidia/Mellanox: - add the support for linux bridge multicast offload - enable TC offload for egress and engress MACVLAN over bond - add support for VxLAN GBP encap/decap flows offload - extend packet offload to fully support libreswan - support tunnel mode in mlx5 IPsec packet offload - extend XDP multi-buffer support - support MACsec VLAN offload - add support for dynamic msix vectors allocation - drop RX page_cache and fully use page_pool - implement thermal zone to report NIC temperature - Netronome/Corigine: - add support for multi-zone conntrack offload - Solarflare/Xilinx: - support offloading TC VLAN push/pop actions to the MAE - support TC decap rules - support unicast PTP - Other NICs: - Broadcom (bnxt): enforce software based freq adjustments only on shared PHC NIC - RealTek (r8169): refactor to addess ASPM issues during NAPI poll - Micrel (lan8841): add support for PTP_PF_PEROUT - Cadence (macb): enable PTP unicast - Engleder (tsnep): add XDP socket zero-copy support - virtio-net: implement exact header length guest feature - veth: add page_pool support for page recycling - vxlan: add MDB data path support - gve: add XDP support for GQI-QPL format - geneve: accept every ethertype - macvlan: allow some packets to bypass broadcast queue - mana: add support for jumbo frame - Ethernet high-speed switches: - Microchip (sparx5): Add support for TC flower templates - Ethernet embedded switches: - Broadcom (b54): - configure 6318 and 63268 RGMII ports - Marvell (mv88e6xxx): - faster C45 bus scan - Microchip: - lan966x: - add support for IS1 VCAP - better TX/RX from/to CPU performances - ksz9477: add ETS Qdisc support - ksz8: enhance static MAC table operations and error handling - sama7g5: add PTP capability - NXP (ocelot): - add support for external ports - add support for preemptible traffic classes - Texas Instruments: - add CPSWxG SGMII support for J7200 and J721E - Intel WiFi (iwlwifi): - preparation for Wi-Fi 7 EHT and multi-link support - EHT (Wi-Fi 7) sniffer support - hardware timestamping support for some devices/firwmares - TX beacon protection on newer hardware - Qualcomm 802.11ax WiFi (ath11k): - MU-MIMO parameters support - ack signal support for management packets - RealTek WiFi (rtw88): - SDIO bus support - better support for some SDIO devices (e.g. MAC address from efuse) - RealTek WiFi (rtw89): - HW scan support for 8852b - better support for 6 GHz scanning - support for various newer firmware APIs - framework firmware backwards compatibility - MediaTek WiFi (mt76): - P2P support - mesh A-MSDU support - EHT (Wi-Fi 7) support - coredump support" * tag 'net-next-6.4' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next: (2078 commits) net: phy: hide the PHYLIB_LEDS knob net: phy: marvell-88x2222: remove unnecessary (void*) conversions tcp/udp: Fix memleaks of sk and zerocopy skbs with TX timestamp. net: amd: Fix link leak when verifying config failed net: phy: marvell: Fix inconsistent indenting in led_blink_set lan966x: Don't use xdp_frame when action is XDP_TX tsnep: Add XDP socket zero-copy TX support tsnep: Add XDP socket zero-copy RX support tsnep: Move skb receive action to separate function tsnep: Add functions for queue enable/disable tsnep: Rework TX/RX queue initialization tsnep: Replace modulo operation with mask net: phy: dp83867: Add led_brightness_set support net: phy: Fix reading LED reg property drivers: nfc: nfcsim: remove return value check of `dev_dir` net: phy: dp83867: Remove unnecessary (void*) conversions net: ethtool: coalesce: try to make user settings stick twice net: mana: Check if netdev/napi_alloc_frag returns single page net: mana: Rename mana_refill_rxoob and remove some empty lines net: veth: add page_pool stats ...

Linus Torvalds committed 3 years ago

Commit 6e98b092 parents b68ee1c 9b78d911925 files · +138910 −47345Browse files

Showing the changes to lib (4 of 1925 files). Show all

Unified · Split

4 files
  1. lib/Makefile +1 −1
  2. lib/cpu_rmap.c +46 −11
  3. lib/packing.c +0 −1
  4. lib/rcuref.c +281 −0
lib/Makefile+1−1
@@ -47,7 +47,7 @@ obj-y += bcd.o sort.o parser.o debug_locks.o random32.o \ · 46 unmodified lines
4747 list_sort.o uuid.o iov_iter.o clz_ctz.o \
4848 bsearch.o find_bit.o llist.o memweight.o kfifo.o \
4949 percpu-refcount.o rhashtable.o base64.o \
50− once.o refcount.o usercopy.o errseq.o bucket_locks.o \
50+ once.o refcount.o rcuref.o usercopy.o errseq.o bucket_locks.o \
5151 generic-radix-tree.o
5252 obj-$(CONFIG_STRING_SELFTEST) += test_string.o
5353 obj-y += string_helpers.o
lib/cpu_rmap.c+46−11
@@ -128,19 +128,31 @@ debug_print_rmap(const struct cpu_rmap *rmap, const char *prefix) · 127 unmodified lines
128128 }
129129 #endif
130130
131+static int get_free_index(struct cpu_rmap *rmap)
132+{
133+ int i;
134+
135+ for (i = 0; i < rmap->size; i++)
136+ if (!rmap->obj[i])
137+ return i;
138+
139+ return -ENOSPC;
140+}
141+
131142 /**
132143 * cpu_rmap_add - add object to a rmap
133144 * @rmap: CPU rmap allocated with alloc_cpu_rmap()
134145 * @obj: Object to add to rmap
135146 *
136− * Return index of object.
147+ * Return index of object or -ENOSPC if no free entry was found
137148 */
138149 int cpu_rmap_add(struct cpu_rmap *rmap, void *obj)
139150 {
140− u16 index;
151+ int index = get_free_index(rmap);
152+
153+ if (index < 0)
154+ return index;
141155
142− BUG_ON(rmap->used >= rmap->size);
143− index = rmap->used++;
144156 rmap->obj[index] = obj;
145157 return index;
146158 }
@@ -230,9 +242,10 @@ void free_irq_cpu_rmap(struct cpu_rmap *rmap) · 83 unmodified lines
230242 if (!rmap)
231243 return;
232244
233− for (index = 0; index < rmap->used; index++) {
245+ for (index = 0; index < rmap->size; index++) {
234246 glue = rmap->obj[index];
235− irq_set_affinity_notifier(glue->notify.irq, NULL);
247+ if (glue)
248+ irq_set_affinity_notifier(glue->notify.irq, NULL);
236249 }
237250
238251 cpu_rmap_put(rmap);
@@ -268,9 +281,21 @@ static void irq_cpu_rmap_release(struct kref *ref) · 29 unmodified lines
268281 container_of(ref, struct irq_glue, notify.kref);
269282
270283 cpu_rmap_put(glue->rmap);
284+ glue->rmap->obj[glue->index] = NULL;
271285 kfree(glue);
272286 }
273287
288+/**
289+ * irq_cpu_rmap_remove - remove an IRQ from a CPU affinity reverse-map
290+ * @rmap: The reverse-map
291+ * @irq: The IRQ number
292+ */
293+int irq_cpu_rmap_remove(struct cpu_rmap *rmap, int irq)
294+{
295+ return irq_set_affinity_notifier(irq, NULL);
296+}
297+EXPORT_SYMBOL(irq_cpu_rmap_remove);
298+
274299 /**
275300 * irq_cpu_rmap_add - add an IRQ to a CPU affinity reverse-map
276301 * @rmap: The reverse-map
@@ -293,12 +318,22 @@ int irq_cpu_rmap_add(struct cpu_rmap *rmap, int irq) · 16 unmodified lines
293318 glue->notify.release = irq_cpu_rmap_release;
294319 glue->rmap = rmap;
295320 cpu_rmap_get(rmap);
296− glue->index = cpu_rmap_add(rmap, glue);
321+ rc = cpu_rmap_add(rmap, glue);
322+ if (rc < 0)
323+ goto err_add;
324+
325+ glue->index = rc;
297326 rc = irq_set_affinity_notifier(irq, &glue->notify);
298− if (rc) {
299− cpu_rmap_put(glue->rmap);
300− kfree(glue);
301− }
327+ if (rc)
328+ goto err_set;
329+
330+ return rc;
331+
332+err_set:
333+ rmap->obj[glue->index] = NULL;
334+err_add:
335+ cpu_rmap_put(glue->rmap);
336+ kfree(glue);
302337 return rc;
303338 }
304339 EXPORT_SYMBOL(irq_cpu_rmap_add);
lib/packing.c+0−1
@@ -198,5 +198,4 @@ int packing(void *pbuf, u64 *uval, int startbit, int endbit, size_t pbuflen, · 197 unmodified lines
198198 }
199199 EXPORT_SYMBOL(packing);
200200
201−MODULE_LICENSE("GPL v2");
202201 MODULE_DESCRIPTION("Generic bitfield packing and unpacking");
lib/rcuref.c+281−0
@@ -0,0 +1,281 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+
3+/*
4+ * rcuref - A scalable reference count implementation for RCU managed objects
5+ *
6+ * rcuref is provided to replace open coded reference count implementations
7+ * based on atomic_t. It protects explicitely RCU managed objects which can
8+ * be visible even after the last reference has been dropped and the object
9+ * is heading towards destruction.
10+ *
11+ * A common usage pattern is:
12+ *
13+ * get()
14+ * rcu_read_lock();
15+ * p = get_ptr();
16+ * if (p && !atomic_inc_not_zero(&p->refcnt))
17+ * p = NULL;
18+ * rcu_read_unlock();
19+ * return p;
20+ *
21+ * put()
22+ * if (!atomic_dec_return(&->refcnt)) {
23+ * remove_ptr(p);
24+ * kfree_rcu((p, rcu);
25+ * }
26+ *
27+ * atomic_inc_not_zero() is implemented with a try_cmpxchg() loop which has
28+ * O(N^2) behaviour under contention with N concurrent operations.
29+ *
30+ * rcuref uses atomic_add_negative_relaxed() for the fast path, which scales
31+ * better under contention.
32+ *
33+ * Why not refcount?
34+ * =================
35+ *
36+ * In principle it should be possible to make refcount use the rcuref
37+ * scheme, but the destruction race described below cannot be prevented
38+ * unless the protected object is RCU managed.
39+ *
40+ * Theory of operation
41+ * ===================
42+ *
43+ * rcuref uses an unsigned integer reference counter. As long as the
44+ * counter value is greater than or equal to RCUREF_ONEREF and not larger
45+ * than RCUREF_MAXREF the reference is alive:
46+ *
47+ * ONEREF MAXREF SATURATED RELEASED DEAD NOREF
48+ * 0 0x7FFFFFFF 0x8000000 0xA0000000 0xBFFFFFFF 0xC0000000 0xE0000000 0xFFFFFFFF
49+ * <---valid --------> <-------saturation zone-------> <-----dead zone----->
50+ *
51+ * The get() and put() operations do unconditional increments and
52+ * decrements. The result is checked after the operation. This optimizes
53+ * for the fast path.
54+ *
55+ * If the reference count is saturated or dead, then the increments and
56+ * decrements are not harmful as the reference count still stays in the
57+ * respective zones and is always set back to STATURATED resp. DEAD. The
58+ * zones have room for 2^28 racing operations in each direction, which
59+ * makes it practically impossible to escape the zones.
60+ *
61+ * Once the last reference is dropped the reference count becomes
62+ * RCUREF_NOREF which forces rcuref_put() into the slowpath operation. The
63+ * slowpath then tries to set the reference count from RCUREF_NOREF to
64+ * RCUREF_DEAD via a cmpxchg(). This opens a small window where a
65+ * concurrent rcuref_get() can acquire the reference count and bring it
66+ * back to RCUREF_ONEREF or even drop the reference again and mark it DEAD.
67+ *
68+ * If the cmpxchg() succeeds then a concurrent rcuref_get() will result in
69+ * DEAD + 1, which is inside the dead zone. If that happens the reference
70+ * count is put back to DEAD.
71+ *
72+ * The actual race is possible due to the unconditional increment and
73+ * decrements in rcuref_get() and rcuref_put():
74+ *
75+ * T1 T2
76+ * get() put()
77+ * if (atomic_add_negative(-1, &ref->refcnt))
78+ * succeeds-> atomic_cmpxchg(&ref->refcnt, NOREF, DEAD);
79+ *
80+ * atomic_add_negative(1, &ref->refcnt); <- Elevates refcount to DEAD + 1
81+ *
82+ * As the result of T1's add is negative, the get() goes into the slow path
83+ * and observes refcnt being in the dead zone which makes the operation fail.
84+ *
85+ * Possible critical states:
86+ *
87+ * Context Counter References Operation
88+ * T1 0 1 init()
89+ * T2 1 2 get()
90+ * T1 0 1 put()
91+ * T2 -1 0 put() tries to mark dead
92+ * T1 0 1 get()
93+ * T2 0 1 put() mark dead fails
94+ * T1 -1 0 put() tries to mark dead
95+ * T1 DEAD 0 put() mark dead succeeds
96+ * T2 DEAD+1 0 get() fails and puts it back to DEAD
97+ *
98+ * Of course there are more complex scenarios, but the above illustrates
99+ * the working principle. The rest is left to the imagination of the
100+ * reader.
101+ *
102+ * Deconstruction race
103+ * ===================
104+ *
105+ * The release operation must be protected by prohibiting a grace period in
106+ * order to prevent a possible use after free:
107+ *
108+ * T1 T2
109+ * put() get()
110+ * // ref->refcnt = ONEREF
111+ * if (!atomic_add_negative(-1, &ref->refcnt))
112+ * return false; <- Not taken
113+ *
114+ * // ref->refcnt == NOREF
115+ * --> preemption
116+ * // Elevates ref->refcnt to ONEREF
117+ * if (!atomic_add_negative(1, &ref->refcnt))
118+ * return true; <- taken
119+ *
120+ * if (put(&p->ref)) { <-- Succeeds
121+ * remove_pointer(p);
122+ * kfree_rcu(p, rcu);
123+ * }
124+ *
125+ * RCU grace period ends, object is freed
126+ *
127+ * atomic_cmpxchg(&ref->refcnt, NOREF, DEAD); <- UAF
128+ *
129+ * This is prevented by disabling preemption around the put() operation as
130+ * that's in most kernel configurations cheaper than a rcu_read_lock() /
131+ * rcu_read_unlock() pair and in many cases even a NOOP. In any case it
132+ * prevents the grace period which keeps the object alive until all put()
133+ * operations complete.
134+ *
135+ * Saturation protection
136+ * =====================
137+ *
138+ * The reference count has a saturation limit RCUREF_MAXREF (INT_MAX).
139+ * Once this is exceedded the reference count becomes stale by setting it
140+ * to RCUREF_SATURATED, which will cause a memory leak, but it prevents
141+ * wrap arounds which obviously cause worse problems than a memory
142+ * leak. When saturation is reached a warning is emitted.
143+ *
144+ * Race conditions
145+ * ===============
146+ *
147+ * All reference count increment/decrement operations are unconditional and
148+ * only verified after the fact. This optimizes for the good case and takes
149+ * the occasional race vs. a dead or already saturated refcount into
150+ * account. The saturation and dead zones are large enough to accomodate
151+ * for that.
152+ *
153+ * Memory ordering
154+ * ===============
155+ *
156+ * Memory ordering rules are slightly relaxed wrt regular atomic_t functions
157+ * and provide only what is strictly required for refcounts.
158+ *
159+ * The increments are fully relaxed; these will not provide ordering. The
160+ * rationale is that whatever is used to obtain the object to increase the
161+ * reference count on will provide the ordering. For locked data
162+ * structures, its the lock acquire, for RCU/lockless data structures its
163+ * the dependent load.
164+ *
165+ * rcuref_get() provides a control dependency ordering future stores which
166+ * ensures that the object is not modified when acquiring a reference
167+ * fails.
168+ *
169+ * rcuref_put() provides release order, i.e. all prior loads and stores
170+ * will be issued before. It also provides a control dependency ordering
171+ * against the subsequent destruction of the object.
172+ *
173+ * If rcuref_put() successfully dropped the last reference and marked the
174+ * object DEAD it also provides acquire ordering.
175+ */
176+
177+#include <linux/export.h>
178+#include <linux/rcuref.h>
179+
180+/**
181+ * rcuref_get_slowpath - Slowpath of rcuref_get()
182+ * @ref: Pointer to the reference count
183+ *
184+ * Invoked when the reference count is outside of the valid zone.
185+ *
186+ * Return:
187+ * False if the reference count was already marked dead
188+ *
189+ * True if the reference count is saturated, which prevents the
190+ * object from being deconstructed ever.
191+ */
192+bool rcuref_get_slowpath(rcuref_t *ref)
193+{
194+ unsigned int cnt = atomic_read(&ref->refcnt);
195+
196+ /*
197+ * If the reference count was already marked dead, undo the
198+ * increment so it stays in the middle of the dead zone and return
199+ * fail.
200+ */
201+ if (cnt >= RCUREF_RELEASED) {
202+ atomic_set(&ref->refcnt, RCUREF_DEAD);
203+ return false;
204+ }
205+
206+ /*
207+ * If it was saturated, warn and mark it so. In case the increment
208+ * was already on a saturated value restore the saturation
209+ * marker. This keeps it in the middle of the saturation zone and
210+ * prevents the reference count from overflowing. This leaks the
211+ * object memory, but prevents the obvious reference count overflow
212+ * damage.
213+ */
214+ if (WARN_ONCE(cnt > RCUREF_MAXREF, "rcuref saturated - leaking memory"))
215+ atomic_set(&ref->refcnt, RCUREF_SATURATED);
216+ return true;
217+}
218+EXPORT_SYMBOL_GPL(rcuref_get_slowpath);
219+
220+/**
221+ * rcuref_put_slowpath - Slowpath of __rcuref_put()
222+ * @ref: Pointer to the reference count
223+ *
224+ * Invoked when the reference count is outside of the valid zone.
225+ *
226+ * Return:
227+ * True if this was the last reference with no future references
228+ * possible. This signals the caller that it can safely schedule the
229+ * object, which is protected by the reference counter, for
230+ * deconstruction.
231+ *
232+ * False if there are still active references or the put() raced
233+ * with a concurrent get()/put() pair. Caller is not allowed to
234+ * deconstruct the protected object.
235+ */
236+bool rcuref_put_slowpath(rcuref_t *ref)
237+{
238+ unsigned int cnt = atomic_read(&ref->refcnt);
239+
240+ /* Did this drop the last reference? */
241+ if (likely(cnt == RCUREF_NOREF)) {
242+ /*
243+ * Carefully try to set the reference count to RCUREF_DEAD.
244+ *
245+ * This can fail if a concurrent get() operation has
246+ * elevated it again or the corresponding put() even marked
247+ * it dead already. Both are valid situations and do not
248+ * require a retry. If this fails the caller is not
249+ * allowed to deconstruct the object.
250+ */
251+ if (atomic_cmpxchg_release(&ref->refcnt, RCUREF_NOREF, RCUREF_DEAD) != RCUREF_NOREF)
252+ return false;
253+
254+ /*
255+ * The caller can safely schedule the object for
256+ * deconstruction. Provide acquire ordering.
257+ */
258+ smp_acquire__after_ctrl_dep();
259+ return true;
260+ }
261+
262+ /*
263+ * If the reference count was already in the dead zone, then this
264+ * put() operation is imbalanced. Warn, put the reference count back to
265+ * DEAD and tell the caller to not deconstruct the object.
266+ */
267+ if (WARN_ONCE(cnt >= RCUREF_RELEASED, "rcuref - imbalanced put()")) {
268+ atomic_set(&ref->refcnt, RCUREF_DEAD);
269+ return false;
270+ }
271+
272+ /*
273+ * This is a put() operation on a saturated refcount. Restore the
274+ * mean saturation value and tell the caller to not deconstruct the
275+ * object.
276+ */
277+ if (cnt > RCUREF_MAXREF)
278+ atomic_set(&ref->refcnt, RCUREF_SATURATED);
279+ return false;
280+}
281+EXPORT_SYMBOL_GPL(rcuref_put_slowpath);