Pull networking updates from Paolo Abeni: "Core: - Introduce a config option to tweak MAX_SKB_FRAGS. Increasing the default value allows for better BIG TCP performances - Reduce compound page head access for zero-copy data transfers - RPS/RFS improvements, avoiding unneeded NET_RX_SOFTIRQ when possible - Threaded NAPI improvements, adding defer skb free support and unneeded softirq avoidance - Address dst_entry reference count scalability issues, via false sharing avoidance and optimize refcount tracking - Add lockless accesses annotation to sk_err[_soft] - Optimize again the skb struct layout - Extends the skb drop reasons to make it usable by multiple subsystems - Better const qualifier awareness for socket casts BPF: - Add skb and XDP typed dynptrs which allow BPF programs for more ergonomic and less brittle iteration through data and variable-sized accesses - Add a new BPF netfilter program type and minimal support to hook BPF programs to netfilter hooks such as prerouting or forward - Add more precise memory usage reporting for all BPF map types - Adds support for using {FOU,GUE} encap with an ipip device operating in collect_md mode and add a set of BPF kfuncs for controlling encap params - Allow BPF programs to detect at load time whether a particular kfunc exists or not, and also add support for this in light skeleton - Bigger batch of BPF verifier improvements to prepare for upcoming BPF open-coded iterators allowing for less restrictive looping capabilities - Rework RCU enforcement in the verifier, add kptr_rcu and enforce BPF programs to NULL-check before passing such pointers into kfunc - Add support for kptrs in percpu hashmaps, percpu LRU hashmaps and in local storage maps - Enable RCU semantics for task BPF kptrs and allow referenced kptr tasks to be stored in BPF maps - Add support for refcounted local kptrs to the verifier for allowing shared ownership, useful for adding a node to both the BPF list and rbtree - Add BPF verifier support for ST instructions in convert_ctx_access() which will help new -mcpu=v4 clang flag to start emitting them - Add ARM32 USDT support to libbpf - Improve bpftool's visual program dump which produces the control flow graph in a DOT format by adding C source inline annotations Protocols: - IPv4: Allow adding to IPv4 address a 'protocol' tag. Such value indicates the provenance of the IP address - IPv6: optimize route lookup, dropping unneeded R/W lock acquisition - Add the handshake upcall mechanism, allowing the user-space to implement generic TLS handshake on kernel's behalf - Bridge: support per-{Port, VLAN} neighbor suppression, increasing resilience to nodes failures - SCTP: add support for Fair Capacity and Weighted Fair Queueing schedulers - MPTCP: delay first subflow allocation up to its first usage. This will allow for later better LSM interaction - xfrm: Remove inner/outer modes from input/output path. These are not needed anymore - WiFi: - reduced neighbor report (RNR) handling for AP mode - HW timestamping support - support for randomized auth/deauth TA for PASN privacy - per-link debugfs for multi-link - TC offload support for mac80211 drivers - mac80211 mesh fast-xmit and fast-rx support - enable Wi-Fi 7 (EHT) mesh support Netfilter: - Add nf_tables 'brouting' support, to force a packet to be routed instead of being bridged - Update bridge netfilter and ovs conntrack helpers to handle IPv6 Jumbo packets properly, i.e. fetch the packet length from hop-by-hop extension header. This is needed for BIT TCP support - The iptables 32bit compat interface isn't compiled in by default anymore - Move ip(6)tables builtin icmp matches to the udptcp one. This has the advantage that icmp/icmpv6 match doesn't load the iptables/ip6tables modules anymore when iptables-nft is used - Extended netlink error report for netdevice in flowtables and netdev/chains. Allow for incrementally add/delete devices to netdev basechain. Allow to create netdev chain without device Driver API: - Remove redundant Device Control Error Reporting Enable, as PCI core has already error reporting enabled at enumeration time - Move Multicast DB netlink handlers to core, allowing devices other then bridge to use them - Allow the page_pool to directly recycle the pages from safely localized NAPI - Implement lockless TX queue stop/wake combo macros, allowing for further code de-duplication and sanitization - Add YNL support for user headers and struct attrs - Add partial YNL specification for devlink - Add partial YNL specification for ethtool - Add tc-mqprio and tc-taprio support for preemptible traffic classes - Add tx push buf len param to ethtool, specifies the maximum number of bytes of a transmitted packet a driver can push directly to the underlying device - Add basic LED support for switch/phy - Add NAPI documentation, stop relaying on external links - Convert dsa_master_ioctl() to netdev notifier. This is a preparatory work to make the hardware timestamping layer selectable by user space - Add transceiver support and improve the error messages for CAN-FD controllers New hardware / drivers: - Ethernet: - AMD/Pensando core device support - MediaTek MT7981 SoC - MediaTek MT7988 SoC - Broadcom BCM53134 embedded switch - Texas Instruments CPSW9G ethernet switch - Qualcomm EMAC3 DWMAC ethernet - StarFive JH7110 SoC - NXP CBTX ethernet PHY - WiFi: - Apple M1 Pro/Max devices - RealTek rtl8710bu/rtl8188gu - RealTek rtl8822bs, rtl8822cs and rtl8821cs SDIO chipset - Bluetooth: - Realtek RTL8821CS, RTL8851B, RTL8852BS - Mediatek MT7663, MT7922 - NXP w8997 - Actions Semi ATS2851 - QTI WCN6855 - Marvell 88W8997 - Can: - STMicroelectronics bxcan stm32f429 Drivers: - Ethernet NICs: - Intel (1G, icg): - add tracking and reporting of QBV config errors - add support for configuring max SDU for each Tx queue - Intel (100G, ice): - refactor mailbox overflow detection to support Scalable IOV - GNSS interface optimization - Intel (i40e): - support XDP multi-buffer - nVidia/Mellanox: - add the support for linux bridge multicast offload - enable TC offload for egress and engress MACVLAN over bond - add support for VxLAN GBP encap/decap flows offload - extend packet offload to fully support libreswan - support tunnel mode in mlx5 IPsec packet offload - extend XDP multi-buffer support - support MACsec VLAN offload - add support for dynamic msix vectors allocation - drop RX page_cache and fully use page_pool - implement thermal zone to report NIC temperature - Netronome/Corigine: - add support for multi-zone conntrack offload - Solarflare/Xilinx: - support offloading TC VLAN push/pop actions to the MAE - support TC decap rules - support unicast PTP - Other NICs: - Broadcom (bnxt): enforce software based freq adjustments only on shared PHC NIC - RealTek (r8169): refactor to addess ASPM issues during NAPI poll - Micrel (lan8841): add support for PTP_PF_PEROUT - Cadence (macb): enable PTP unicast - Engleder (tsnep): add XDP socket zero-copy support - virtio-net: implement exact header length guest feature - veth: add page_pool support for page recycling - vxlan: add MDB data path support - gve: add XDP support for GQI-QPL format - geneve: accept every ethertype - macvlan: allow some packets to bypass broadcast queue - mana: add support for jumbo frame - Ethernet high-speed switches: - Microchip (sparx5): Add support for TC flower templates - Ethernet embedded switches: - Broadcom (b54): - configure 6318 and 63268 RGMII ports - Marvell (mv88e6xxx): - faster C45 bus scan - Microchip: - lan966x: - add support for IS1 VCAP - better TX/RX from/to CPU performances - ksz9477: add ETS Qdisc support - ksz8: enhance static MAC table operations and error handling - sama7g5: add PTP capability - NXP (ocelot): - add support for external ports - add support for preemptible traffic classes - Texas Instruments: - add CPSWxG SGMII support for J7200 and J721E - Intel WiFi (iwlwifi): - preparation for Wi-Fi 7 EHT and multi-link support - EHT (Wi-Fi 7) sniffer support - hardware timestamping support for some devices/firwmares - TX beacon protection on newer hardware - Qualcomm 802.11ax WiFi (ath11k): - MU-MIMO parameters support - ack signal support for management packets - RealTek WiFi (rtw88): - SDIO bus support - better support for some SDIO devices (e.g. MAC address from efuse) - RealTek WiFi (rtw89): - HW scan support for 8852b - better support for 6 GHz scanning - support for various newer firmware APIs - framework firmware backwards compatibility - MediaTek WiFi (mt76): - P2P support - mesh A-MSDU support - EHT (Wi-Fi 7) support - coredump support" * tag 'net-next-6.4' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next: (2078 commits) net: phy: hide the PHYLIB_LEDS knob net: phy: marvell-88x2222: remove unnecessary (void*) conversions tcp/udp: Fix memleaks of sk and zerocopy skbs with TX timestamp. net: amd: Fix link leak when verifying config failed net: phy: marvell: Fix inconsistent indenting in led_blink_set lan966x: Don't use xdp_frame when action is XDP_TX tsnep: Add XDP socket zero-copy TX support tsnep: Add XDP socket zero-copy RX support tsnep: Move skb receive action to separate function tsnep: Add functions for queue enable/disable tsnep: Rework TX/RX queue initialization tsnep: Replace modulo operation with mask net: phy: dp83867: Add led_brightness_set support net: phy: Fix reading LED reg property drivers: nfc: nfcsim: remove return value check of `dev_dir` net: phy: dp83867: Remove unnecessary (void*) conversions net: ethtool: coalesce: try to make user settings stick twice net: mana: Check if netdev/napi_alloc_frag returns single page net: mana: Rename mana_refill_rxoob and remove some empty lines net: veth: add page_pool stats ...
Showing the changes to lib (4 of 1925 files). Show all
Unified · Split
| @@ -47,7 +47,7 @@ obj-y += bcd.o sort.o parser.o debug_locks.o random32.o \ · 46 unmodified lines | |||
| 47 | list_sort.o uuid.o iov_iter.o clz_ctz.o \ | 47 | list_sort.o uuid.o iov_iter.o clz_ctz.o \ |
| 48 | bsearch.o find_bit.o llist.o memweight.o kfifo.o \ | 48 | bsearch.o find_bit.o llist.o memweight.o kfifo.o \ |
| 49 | percpu-refcount.o rhashtable.o base64.o \ | 49 | percpu-refcount.o rhashtable.o base64.o \ |
| 50 | once.o refcount.o usercopy.o errseq.o bucket_locks.o \ | 50 | once.o refcount.o rcuref.o usercopy.o errseq.o bucket_locks.o \ |
| 51 | generic-radix-tree.o | 51 | generic-radix-tree.o |
| 52 | obj-$(CONFIG_STRING_SELFTEST) += test_string.o | 52 | obj-$(CONFIG_STRING_SELFTEST) += test_string.o |
| 53 | obj-y += string_helpers.o | 53 | obj-y += string_helpers.o |
| @@ -128,19 +128,31 @@ debug_print_rmap(const struct cpu_rmap *rmap, const char *prefix) · 127 unmodified lines | |||
| 128 | } | 128 | } |
| 129 | #endif | 129 | #endif |
| 130 | 130 | ||
| 131 | static int get_free_index(struct cpu_rmap *rmap) | ||
| 132 | { | ||
| 133 | int i; | ||
| 134 | |||
| 135 | for (i = 0; i < rmap->size; i++) | ||
| 136 | if (!rmap->obj[i]) | ||
| 137 | return i; | ||
| 138 | |||
| 139 | return -ENOSPC; | ||
| 140 | } | ||
| 141 | |||
| 131 | /** | 142 | /** |
| 132 | * cpu_rmap_add - add object to a rmap | 143 | * cpu_rmap_add - add object to a rmap |
| 133 | * @rmap: CPU rmap allocated with alloc_cpu_rmap() | 144 | * @rmap: CPU rmap allocated with alloc_cpu_rmap() |
| 134 | * @obj: Object to add to rmap | 145 | * @obj: Object to add to rmap |
| 135 | * | 146 | * |
| 136 | * Return index of object. | 147 | * Return index of object or -ENOSPC if no free entry was found |
| 137 | */ | 148 | */ |
| 138 | int cpu_rmap_add(struct cpu_rmap *rmap, void *obj) | 149 | int cpu_rmap_add(struct cpu_rmap *rmap, void *obj) |
| 139 | { | 150 | { |
| 140 | u16 index; | 151 | int index = get_free_index(rmap); |
| 152 | |||
| 153 | if (index < 0) | ||
| 154 | return index; | ||
| 141 | 155 | ||
| 142 | BUG_ON(rmap->used >= rmap->size); | ||
| 143 | index = rmap->used++; | ||
| 144 | rmap->obj[index] = obj; | 156 | rmap->obj[index] = obj; |
| 145 | return index; | 157 | return index; |
| 146 | } | 158 | } |
| @@ -230,9 +242,10 @@ void free_irq_cpu_rmap(struct cpu_rmap *rmap) · 83 unmodified lines | |||
| 230 | if (!rmap) | 242 | if (!rmap) |
| 231 | return; | 243 | return; |
| 232 | 244 | ||
| 233 | for (index = 0; index < rmap->used; index++) { | 245 | for (index = 0; index < rmap->size; index++) { |
| 234 | glue = rmap->obj[index]; | 246 | glue = rmap->obj[index]; |
| 235 | irq_set_affinity_notifier(glue->notify.irq, NULL); | 247 | if (glue) |
| 248 | irq_set_affinity_notifier(glue->notify.irq, NULL); | ||
| 236 | } | 249 | } |
| 237 | 250 | ||
| 238 | cpu_rmap_put(rmap); | 251 | cpu_rmap_put(rmap); |
| @@ -268,9 +281,21 @@ static void irq_cpu_rmap_release(struct kref *ref) · 29 unmodified lines | |||
| 268 | container_of(ref, struct irq_glue, notify.kref); | 281 | container_of(ref, struct irq_glue, notify.kref); |
| 269 | 282 | ||
| 270 | cpu_rmap_put(glue->rmap); | 283 | cpu_rmap_put(glue->rmap); |
| 284 | glue->rmap->obj[glue->index] = NULL; | ||
| 271 | kfree(glue); | 285 | kfree(glue); |
| 272 | } | 286 | } |
| 273 | 287 | ||
| 288 | /** | ||
| 289 | * irq_cpu_rmap_remove - remove an IRQ from a CPU affinity reverse-map | ||
| 290 | * @rmap: The reverse-map | ||
| 291 | * @irq: The IRQ number | ||
| 292 | */ | ||
| 293 | int irq_cpu_rmap_remove(struct cpu_rmap *rmap, int irq) | ||
| 294 | { | ||
| 295 | return irq_set_affinity_notifier(irq, NULL); | ||
| 296 | } | ||
| 297 | EXPORT_SYMBOL(irq_cpu_rmap_remove); | ||
| 298 | |||
| 274 | /** | 299 | /** |
| 275 | * irq_cpu_rmap_add - add an IRQ to a CPU affinity reverse-map | 300 | * irq_cpu_rmap_add - add an IRQ to a CPU affinity reverse-map |
| 276 | * @rmap: The reverse-map | 301 | * @rmap: The reverse-map |
| @@ -293,12 +318,22 @@ int irq_cpu_rmap_add(struct cpu_rmap *rmap, int irq) · 16 unmodified lines | |||
| 293 | glue->notify.release = irq_cpu_rmap_release; | 318 | glue->notify.release = irq_cpu_rmap_release; |
| 294 | glue->rmap = rmap; | 319 | glue->rmap = rmap; |
| 295 | cpu_rmap_get(rmap); | 320 | cpu_rmap_get(rmap); |
| 296 | glue->index = cpu_rmap_add(rmap, glue); | 321 | rc = cpu_rmap_add(rmap, glue); |
| 322 | if (rc < 0) | ||
| 323 | goto err_add; | ||
| 324 | |||
| 325 | glue->index = rc; | ||
| 297 | rc = irq_set_affinity_notifier(irq, &glue->notify); | 326 | rc = irq_set_affinity_notifier(irq, &glue->notify); |
| 298 | if (rc) { | 327 | if (rc) |
| 299 | cpu_rmap_put(glue->rmap); | 328 | goto err_set; |
| 300 | kfree(glue); | 329 | |
| 301 | } | 330 | return rc; |
| 331 | |||
| 332 | err_set: | ||
| 333 | rmap->obj[glue->index] = NULL; | ||
| 334 | err_add: | ||
| 335 | cpu_rmap_put(glue->rmap); | ||
| 336 | kfree(glue); | ||
| 302 | return rc; | 337 | return rc; |
| 303 | } | 338 | } |
| 304 | EXPORT_SYMBOL(irq_cpu_rmap_add); | 339 | EXPORT_SYMBOL(irq_cpu_rmap_add); |
| @@ -198,5 +198,4 @@ int packing(void *pbuf, u64 *uval, int startbit, int endbit, size_t pbuflen, · 197 unmodified lines | |||
| 198 | } | 198 | } |
| 199 | EXPORT_SYMBOL(packing); | 199 | EXPORT_SYMBOL(packing); |
| 200 | 200 | ||
| 201 | MODULE_LICENSE("GPL v2"); | ||
| 202 | MODULE_DESCRIPTION("Generic bitfield packing and unpacking"); | 201 | MODULE_DESCRIPTION("Generic bitfield packing and unpacking"); |
| @@ -0,0 +1,281 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only | ||
| 2 | |||
| 3 | /* | ||
| 4 | * rcuref - A scalable reference count implementation for RCU managed objects | ||
| 5 | * | ||
| 6 | * rcuref is provided to replace open coded reference count implementations | ||
| 7 | * based on atomic_t. It protects explicitely RCU managed objects which can | ||
| 8 | * be visible even after the last reference has been dropped and the object | ||
| 9 | * is heading towards destruction. | ||
| 10 | * | ||
| 11 | * A common usage pattern is: | ||
| 12 | * | ||
| 13 | * get() | ||
| 14 | * rcu_read_lock(); | ||
| 15 | * p = get_ptr(); | ||
| 16 | * if (p && !atomic_inc_not_zero(&p->refcnt)) | ||
| 17 | * p = NULL; | ||
| 18 | * rcu_read_unlock(); | ||
| 19 | * return p; | ||
| 20 | * | ||
| 21 | * put() | ||
| 22 | * if (!atomic_dec_return(&->refcnt)) { | ||
| 23 | * remove_ptr(p); | ||
| 24 | * kfree_rcu((p, rcu); | ||
| 25 | * } | ||
| 26 | * | ||
| 27 | * atomic_inc_not_zero() is implemented with a try_cmpxchg() loop which has | ||
| 28 | * O(N^2) behaviour under contention with N concurrent operations. | ||
| 29 | * | ||
| 30 | * rcuref uses atomic_add_negative_relaxed() for the fast path, which scales | ||
| 31 | * better under contention. | ||
| 32 | * | ||
| 33 | * Why not refcount? | ||
| 34 | * ================= | ||
| 35 | * | ||
| 36 | * In principle it should be possible to make refcount use the rcuref | ||
| 37 | * scheme, but the destruction race described below cannot be prevented | ||
| 38 | * unless the protected object is RCU managed. | ||
| 39 | * | ||
| 40 | * Theory of operation | ||
| 41 | * =================== | ||
| 42 | * | ||
| 43 | * rcuref uses an unsigned integer reference counter. As long as the | ||
| 44 | * counter value is greater than or equal to RCUREF_ONEREF and not larger | ||
| 45 | * than RCUREF_MAXREF the reference is alive: | ||
| 46 | * | ||
| 47 | * ONEREF MAXREF SATURATED RELEASED DEAD NOREF | ||
| 48 | * 0 0x7FFFFFFF 0x8000000 0xA0000000 0xBFFFFFFF 0xC0000000 0xE0000000 0xFFFFFFFF | ||
| 49 | * <---valid --------> <-------saturation zone-------> <-----dead zone-----> | ||
| 50 | * | ||
| 51 | * The get() and put() operations do unconditional increments and | ||
| 52 | * decrements. The result is checked after the operation. This optimizes | ||
| 53 | * for the fast path. | ||
| 54 | * | ||
| 55 | * If the reference count is saturated or dead, then the increments and | ||
| 56 | * decrements are not harmful as the reference count still stays in the | ||
| 57 | * respective zones and is always set back to STATURATED resp. DEAD. The | ||
| 58 | * zones have room for 2^28 racing operations in each direction, which | ||
| 59 | * makes it practically impossible to escape the zones. | ||
| 60 | * | ||
| 61 | * Once the last reference is dropped the reference count becomes | ||
| 62 | * RCUREF_NOREF which forces rcuref_put() into the slowpath operation. The | ||
| 63 | * slowpath then tries to set the reference count from RCUREF_NOREF to | ||
| 64 | * RCUREF_DEAD via a cmpxchg(). This opens a small window where a | ||
| 65 | * concurrent rcuref_get() can acquire the reference count and bring it | ||
| 66 | * back to RCUREF_ONEREF or even drop the reference again and mark it DEAD. | ||
| 67 | * | ||
| 68 | * If the cmpxchg() succeeds then a concurrent rcuref_get() will result in | ||
| 69 | * DEAD + 1, which is inside the dead zone. If that happens the reference | ||
| 70 | * count is put back to DEAD. | ||
| 71 | * | ||
| 72 | * The actual race is possible due to the unconditional increment and | ||
| 73 | * decrements in rcuref_get() and rcuref_put(): | ||
| 74 | * | ||
| 75 | * T1 T2 | ||
| 76 | * get() put() | ||
| 77 | * if (atomic_add_negative(-1, &ref->refcnt)) | ||
| 78 | * succeeds-> atomic_cmpxchg(&ref->refcnt, NOREF, DEAD); | ||
| 79 | * | ||
| 80 | * atomic_add_negative(1, &ref->refcnt); <- Elevates refcount to DEAD + 1 | ||
| 81 | * | ||
| 82 | * As the result of T1's add is negative, the get() goes into the slow path | ||
| 83 | * and observes refcnt being in the dead zone which makes the operation fail. | ||
| 84 | * | ||
| 85 | * Possible critical states: | ||
| 86 | * | ||
| 87 | * Context Counter References Operation | ||
| 88 | * T1 0 1 init() | ||
| 89 | * T2 1 2 get() | ||
| 90 | * T1 0 1 put() | ||
| 91 | * T2 -1 0 put() tries to mark dead | ||
| 92 | * T1 0 1 get() | ||
| 93 | * T2 0 1 put() mark dead fails | ||
| 94 | * T1 -1 0 put() tries to mark dead | ||
| 95 | * T1 DEAD 0 put() mark dead succeeds | ||
| 96 | * T2 DEAD+1 0 get() fails and puts it back to DEAD | ||
| 97 | * | ||
| 98 | * Of course there are more complex scenarios, but the above illustrates | ||
| 99 | * the working principle. The rest is left to the imagination of the | ||
| 100 | * reader. | ||
| 101 | * | ||
| 102 | * Deconstruction race | ||
| 103 | * =================== | ||
| 104 | * | ||
| 105 | * The release operation must be protected by prohibiting a grace period in | ||
| 106 | * order to prevent a possible use after free: | ||
| 107 | * | ||
| 108 | * T1 T2 | ||
| 109 | * put() get() | ||
| 110 | * // ref->refcnt = ONEREF | ||
| 111 | * if (!atomic_add_negative(-1, &ref->refcnt)) | ||
| 112 | * return false; <- Not taken | ||
| 113 | * | ||
| 114 | * // ref->refcnt == NOREF | ||
| 115 | * --> preemption | ||
| 116 | * // Elevates ref->refcnt to ONEREF | ||
| 117 | * if (!atomic_add_negative(1, &ref->refcnt)) | ||
| 118 | * return true; <- taken | ||
| 119 | * | ||
| 120 | * if (put(&p->ref)) { <-- Succeeds | ||
| 121 | * remove_pointer(p); | ||
| 122 | * kfree_rcu(p, rcu); | ||
| 123 | * } | ||
| 124 | * | ||
| 125 | * RCU grace period ends, object is freed | ||
| 126 | * | ||
| 127 | * atomic_cmpxchg(&ref->refcnt, NOREF, DEAD); <- UAF | ||
| 128 | * | ||
| 129 | * This is prevented by disabling preemption around the put() operation as | ||
| 130 | * that's in most kernel configurations cheaper than a rcu_read_lock() / | ||
| 131 | * rcu_read_unlock() pair and in many cases even a NOOP. In any case it | ||
| 132 | * prevents the grace period which keeps the object alive until all put() | ||
| 133 | * operations complete. | ||
| 134 | * | ||
| 135 | * Saturation protection | ||
| 136 | * ===================== | ||
| 137 | * | ||
| 138 | * The reference count has a saturation limit RCUREF_MAXREF (INT_MAX). | ||
| 139 | * Once this is exceedded the reference count becomes stale by setting it | ||
| 140 | * to RCUREF_SATURATED, which will cause a memory leak, but it prevents | ||
| 141 | * wrap arounds which obviously cause worse problems than a memory | ||
| 142 | * leak. When saturation is reached a warning is emitted. | ||
| 143 | * | ||
| 144 | * Race conditions | ||
| 145 | * =============== | ||
| 146 | * | ||
| 147 | * All reference count increment/decrement operations are unconditional and | ||
| 148 | * only verified after the fact. This optimizes for the good case and takes | ||
| 149 | * the occasional race vs. a dead or already saturated refcount into | ||
| 150 | * account. The saturation and dead zones are large enough to accomodate | ||
| 151 | * for that. | ||
| 152 | * | ||
| 153 | * Memory ordering | ||
| 154 | * =============== | ||
| 155 | * | ||
| 156 | * Memory ordering rules are slightly relaxed wrt regular atomic_t functions | ||
| 157 | * and provide only what is strictly required for refcounts. | ||
| 158 | * | ||
| 159 | * The increments are fully relaxed; these will not provide ordering. The | ||
| 160 | * rationale is that whatever is used to obtain the object to increase the | ||
| 161 | * reference count on will provide the ordering. For locked data | ||
| 162 | * structures, its the lock acquire, for RCU/lockless data structures its | ||
| 163 | * the dependent load. | ||
| 164 | * | ||
| 165 | * rcuref_get() provides a control dependency ordering future stores which | ||
| 166 | * ensures that the object is not modified when acquiring a reference | ||
| 167 | * fails. | ||
| 168 | * | ||
| 169 | * rcuref_put() provides release order, i.e. all prior loads and stores | ||
| 170 | * will be issued before. It also provides a control dependency ordering | ||
| 171 | * against the subsequent destruction of the object. | ||
| 172 | * | ||
| 173 | * If rcuref_put() successfully dropped the last reference and marked the | ||
| 174 | * object DEAD it also provides acquire ordering. | ||
| 175 | */ | ||
| 176 | |||
| 177 | #include <linux/export.h> | ||
| 178 | #include <linux/rcuref.h> | ||
| 179 | |||
| 180 | /** | ||
| 181 | * rcuref_get_slowpath - Slowpath of rcuref_get() | ||
| 182 | * @ref: Pointer to the reference count | ||
| 183 | * | ||
| 184 | * Invoked when the reference count is outside of the valid zone. | ||
| 185 | * | ||
| 186 | * Return: | ||
| 187 | * False if the reference count was already marked dead | ||
| 188 | * | ||
| 189 | * True if the reference count is saturated, which prevents the | ||
| 190 | * object from being deconstructed ever. | ||
| 191 | */ | ||
| 192 | bool rcuref_get_slowpath(rcuref_t *ref) | ||
| 193 | { | ||
| 194 | unsigned int cnt = atomic_read(&ref->refcnt); | ||
| 195 | |||
| 196 | /* | ||
| 197 | * If the reference count was already marked dead, undo the | ||
| 198 | * increment so it stays in the middle of the dead zone and return | ||
| 199 | * fail. | ||
| 200 | */ | ||
| 201 | if (cnt >= RCUREF_RELEASED) { | ||
| 202 | atomic_set(&ref->refcnt, RCUREF_DEAD); | ||
| 203 | return false; | ||
| 204 | } | ||
| 205 | |||
| 206 | /* | ||
| 207 | * If it was saturated, warn and mark it so. In case the increment | ||
| 208 | * was already on a saturated value restore the saturation | ||
| 209 | * marker. This keeps it in the middle of the saturation zone and | ||
| 210 | * prevents the reference count from overflowing. This leaks the | ||
| 211 | * object memory, but prevents the obvious reference count overflow | ||
| 212 | * damage. | ||
| 213 | */ | ||
| 214 | if (WARN_ONCE(cnt > RCUREF_MAXREF, "rcuref saturated - leaking memory")) | ||
| 215 | atomic_set(&ref->refcnt, RCUREF_SATURATED); | ||
| 216 | return true; | ||
| 217 | } | ||
| 218 | EXPORT_SYMBOL_GPL(rcuref_get_slowpath); | ||
| 219 | |||
| 220 | /** | ||
| 221 | * rcuref_put_slowpath - Slowpath of __rcuref_put() | ||
| 222 | * @ref: Pointer to the reference count | ||
| 223 | * | ||
| 224 | * Invoked when the reference count is outside of the valid zone. | ||
| 225 | * | ||
| 226 | * Return: | ||
| 227 | * True if this was the last reference with no future references | ||
| 228 | * possible. This signals the caller that it can safely schedule the | ||
| 229 | * object, which is protected by the reference counter, for | ||
| 230 | * deconstruction. | ||
| 231 | * | ||
| 232 | * False if there are still active references or the put() raced | ||
| 233 | * with a concurrent get()/put() pair. Caller is not allowed to | ||
| 234 | * deconstruct the protected object. | ||
| 235 | */ | ||
| 236 | bool rcuref_put_slowpath(rcuref_t *ref) | ||
| 237 | { | ||
| 238 | unsigned int cnt = atomic_read(&ref->refcnt); | ||
| 239 | |||
| 240 | /* Did this drop the last reference? */ | ||
| 241 | if (likely(cnt == RCUREF_NOREF)) { | ||
| 242 | /* | ||
| 243 | * Carefully try to set the reference count to RCUREF_DEAD. | ||
| 244 | * | ||
| 245 | * This can fail if a concurrent get() operation has | ||
| 246 | * elevated it again or the corresponding put() even marked | ||
| 247 | * it dead already. Both are valid situations and do not | ||
| 248 | * require a retry. If this fails the caller is not | ||
| 249 | * allowed to deconstruct the object. | ||
| 250 | */ | ||
| 251 | if (atomic_cmpxchg_release(&ref->refcnt, RCUREF_NOREF, RCUREF_DEAD) != RCUREF_NOREF) | ||
| 252 | return false; | ||
| 253 | |||
| 254 | /* | ||
| 255 | * The caller can safely schedule the object for | ||
| 256 | * deconstruction. Provide acquire ordering. | ||
| 257 | */ | ||
| 258 | smp_acquire__after_ctrl_dep(); | ||
| 259 | return true; | ||
| 260 | } | ||
| 261 | |||
| 262 | /* | ||
| 263 | * If the reference count was already in the dead zone, then this | ||
| 264 | * put() operation is imbalanced. Warn, put the reference count back to | ||
| 265 | * DEAD and tell the caller to not deconstruct the object. | ||
| 266 | */ | ||
| 267 | if (WARN_ONCE(cnt >= RCUREF_RELEASED, "rcuref - imbalanced put()")) { | ||
| 268 | atomic_set(&ref->refcnt, RCUREF_DEAD); | ||
| 269 | return false; | ||
| 270 | } | ||
| 271 | |||
| 272 | /* | ||
| 273 | * This is a put() operation on a saturated refcount. Restore the | ||
| 274 | * mean saturation value and tell the caller to not deconstruct the | ||
| 275 | * object. | ||
| 276 | */ | ||
| 277 | if (cnt > RCUREF_MAXREF) | ||
| 278 | atomic_set(&ref->refcnt, RCUREF_SATURATED); | ||
| 279 | return false; | ||
| 280 | } | ||
| 281 | EXPORT_SYMBOL_GPL(rcuref_put_slowpath); | ||