History

Merge branch 'remove-task-and-cgroup-local-storage-percpu-counters'

Amery Hung says: ==================== Remove task and cgroup local storage percpu counters * Motivation * The goal of this patchset is to make bpf syscalls and helpers updating task and cgroup local storage more robust by removing percpu counters in them. Task local storage and cgroup storage each employs a percpu counter to prevent deadlock caused by recursion. Since the underlying bpf local storage takes spinlocks in various operations, bpf programs running recursively may try to take a spinlock which is already taken. For example, when a tracing bpf program called recursively during bpf_task_storage_get(..., F_CREATE) tries to call bpf_task_storage_get(..., F_CREATE) again, it will cause AA deadlock if the percpu variable is not in place. However, sometimes, the percpu counter may cause bpf syscalls or helpers to return errors spuriously, as soon as another threads is also updating the local storage or the local storage map. Ideally, the two threads could have taken turn to take the locks and perform their jobs respectively. However, due to the percpu counter, the syscalls and helpers can return -EBUSY even if one of them does not run recursively in another one. All it takes for this to happen is if the two threads run on the same CPU. This happened when BPF-CI ran the selftest of task local data. Since CI runs the test on VM with 2 CPUs, bpf_task_storage_get(..., F_CREATE) can easily fail. The failure mode is not good for users as they need to add retry logic in user space or bpf programs to avoid it. Even with retry, there is no guaranteed upper bound of the loop for a success call. Therefore, this patchset seeks to remove the percpu counter and makes the related bpf syscalls and helpers more reliable, while still make sure recursion deadlock will not happen, with the help of resilient queued spinlock (rqspinlock). * Implementation * To remove the percpu counter without introducing deadlock, bpf_local_storage is refactored by changing the locks from raw_spin_lock to rqspinlock, which prevents deadlock with deadlock detection and a timeout mechanism. The refactor basically repalces the locks with rqspinlock and propagates errors returned by the locking function to BPF helpers or syscalls. bpf_selem_unlink_nofail() is introduced to handle rqspinlock errors in two lock acquiring functions that cannot fail, bpf_local_storage_destroy() and bpf_local_storage_map_free() (i.e., local storage is being freed by the subsystem or the map is being freed). The high level idea is to bitfiel and atomic operation to track who is referencing an selem when any locks cannot be acquired. Additional care is needed to make sure special fields are freed and owner memory are uncharged safely and correctly. If not familiar with local storage, the last section briefly describe the locks and structure of local storage. It also shows the abbreviation used in the rest of the letter. * Test * Task and cgroup local storage selftests have already covered deadlock caused by recursion. Patch 14 updates the expected result of task local storage selftests as task local storage bpf helpers can now run on the same CPU as they don't cause deadlock. * Benchmark * ./bench -p 1 local-storage-create --storage-type <socket,task> \ --batch-size <16,32,64> The benchmark is a microbenchmark stress-testing how fast local storage can be created. After swicthing to rqspinlock and bpf_unlink_selem_nofail(), socket local storage creation speed has a ~5% gain. For task local storage, the number remains the same. Socket local storage batch creation speed creation speed diff --------------- ---- ------------------ ---- Before 16 134.371 ± 0.884k/s 3.12 kmallocs/create 32 133.032 ± 3.405k/s 3.12 kmallocs/create 64 133.494 ± 0.862k/s 3.12 kmallocs/create After 16 140.778 ± 1.306k/s 3.12 kmallocs/create +4.8% 32 140.550 ± 2.058k/s 3.11 kmallocs/create +5.7% 64 139.311 ± 0.911k/s 3.13 kmallocs/create +4.4% Task local storage batch creation speed creation speed diff --------------- ---- ------------------ ---- Before 16 25.301 ± 0.089k/s 2.43 kmallocs/create 32 23.797 ± 0.106k/s 2.51 kmallocs/create 64 23.251 ± 0.187k/s 2.51 kmallocs/create After 16 25.307 ± 0.080k/s 2.45 kmallocs/create +0.0% 32 23.889 ± 0.089k/s 2.46 kmallocs/create +0.0% 64 23.230 ± 0.113k/s 2.63 kmallocs/create -0.1% * Patchset organization * Patch 1-4 convert local storage internal helpers to failable. Patch 5 changes the locks to rqspinlock and propagate the error returned from raw_res_spin_lock_irqsave() to BPF heleprs and syscalls. Patch 6-8 remove percpu counters in task and cgroup local storage. Patch 9-11 address the unlikely rqspinlock errors by switching to bpf_selem_unlink_nofail() in map_free() and destroy(). Patch 12-17 update selftests. * Appendix: local storage internal * There are two locks in bpf_local_storage due to the ownership model as illustrated in the figure below. A map value, which consists of a pointer to the map and the data, is a bpf_local_storage_map_data (sdata) stored in a bpf_local_storage_elem (selem). A selem belongs to a bpf_local_storage and bpf_local_storage_map at the same time. bpf_local_storage::lock (lock_storage->lock in short) protects the list in a bpf_local_storage and bpf_local_storage_map_bucket::lock (b->lock) protects the hash bucket in a bpf_local_storage_map. task_struct ┌ task1 ───────┐ bpf_local_storage │ *bpf_storage │---->┌─────────┐ └──────────────┘<----│ *owner │ bpf_local_storage_elem │ *cache[16] (selem) selem │ *smap │ ┌──────────┐ ┌──────────┐ │ list │------->│ snode │<------->│ snode │ │ lock │ ┌---->│ map_node │<--┐ ┌-->│ map_node │ └─────────┘ │ │ sdata = │ │ │ │ sdata = │ task_struct │ │ {&mapA,} │ │ │ │ {&mapB,} │ ┌ task2 ───────┐ bpf_local_storage └──────────┘ │ │ └──────────┘ │ *bpf_storage │---->┌─────────┐ │ │ │ └──────────────┘<----│ *owner │ │ │ │ │ *cache[16] │ selem │ │ selem │ *smap │ │ ┌──────────┐ │ │ ┌──────────┐ │ list │--│---->│ snode │<--│-│-->│ snode │ │ lock │ │ ┌-->│ map_node │ └-│-->│ map_node │ └─────────┘ │ │ │ sdata = │ │ │ sdata = │ bpf_local_storage_map │ │ │ {&mapB,} │ │ │ {&mapA,} │ (smap) │ │ └──────────┘ │ └──────────┘ ┌ mapA ───────┐ │ │ │ │ bpf_map map │ bpf_local_storage_map_bucket │ │ *buckets │---->┌ b[0] ┐ │ │ │ └─────────────┘ │ list │------┘ │ │ │ lock │ │ │ └──────┘ │ │ smap ... │ │ ┌ mapB ───────┐ │ │ │ bpf_map map │ bpf_local_storage_map_bucket │ │ *buckets │---->┌ b[0] ┐ │ │ └─────────────┘ │ list │--------┘ │ │ lock │ │ └──────┘ │ ┌ b[1] ┐ │ │ list │-----------------------------┘ │ lock │ └──────┘ ... * Changelog * v6 -> v7 - Minor comment and commit msg tweaks - Patch 9: Remove unused "owner" (kernel test robot) - Patch 13: Update comments in task_ls_recursion.c (AI) Link: https://lore.kernel.org/bpf/20260205070208.186382-1-ameryhung@gmail.com/ v5 -> v6 - Redo benchmark - Patch 9: Remove storage->smap as it is not used any more - Patch 17: Remove storage->smap check in selftests - Patch 10, 11: Pass reuse_now = true to bpf_selem_free() and bpf_local_storage_free() to allow faster memory reclaim (Martin) - Patch 10: Use bitfield instead of refcount to track selem state to be more precise, which removes the possibility map_free missing an selem (Martin) - Patch 10: Allow map_free() to free local_storage and drop the change in bpf_local_storage_map_update() (Martin) - Patch 11: Simplify destroy() by not deferring work as an owner is unlikely to have too many maps that stalls RCU (Martin) Link: https://lore.kernel.org/bpf/20260201175050.468601-1-ameryhung@gmail.com/ v4 -> v5 - Patch 1: Fix incorrect bucket calculation (AI) - Patch 3: Fix memory leak in bpf_sk_storage_clone() (AI) - Patch 5: Fix memory leak in bpf_local_storage_update() (AI) - Fix typo/comment/commit msg (AI) - Patch 10: Replace smp_rmb() with smp_mb(). smp_rmb does not imply acquire semantics Link: https://lore.kernel.org/bpf/20260131050920.2574084-1-ameryhung@gmail.com/ v3 -> v4 - Add performance numbers - Avoid stale element when calling bpf_local_storage_map_free() by allowing it to unlink selem from local_storage->list and uncharge memory. Block destroy() from returning when pending map_free() are uncharging - Fix an -EAGAIN bug in bpf_local_storage_update() as map_free() now does not free local storage - Fix possible double-free of selem by ensuring an selem is only processed once for each caller (Kumar) - Fix possible inifinite loop in bpf_selem_unlink_nofail() when iterating b->list by replacing while loop with hlist_for_each_entry_rcu - Fix unsafe iteration in destroy() by iterating local_storage->list using hlist_for_each_entry_rcu - Fix UAF due to clearing storage_owner after destroy(). Flip the order to fix it - Misc clean-up suggested by Martin Link: https://lore.kernel.org/bpf/20251218175628.1460321-1-ameryhung@gmail.com/ v2 -> v3 - Rebase to bpf-next where BPF memory allocator is replaced with kmalloc_nolock() - Revert to selecting bucket based on selem - Introduce bpf_selem_unlink_lockless() to allow unlinking and freeing selem without taking locks Link: https://lore.kernel.org/bpf/20251002225356.1505480-1-ameryhung@gmail.com/ v1 -> v2 - Rebase to bpf-next - Select bucket based on local_storage instead of selem (Martin) - Simplify bpf_selem_unlink (Martin) - Change handling of rqspinlock errors in bpf_local_storage_destroy() and bpf_local_storage_map_free(). Retry instead of WARN_ON. Link: https://lore.kernel.org/bpf/20250729182550.185356-1-ameryhung@gmail.com/ ====================

Martin KaFai Lau committed 9 months ago

Commit db975de2 parents b8c89f5 97b859b15 files · +354 −561Browse files

Unified · Split

15 files
  1. include/linux/bpf_local_storage.h +19 −10
  2. kernel/bpf/bpf_cgrp_storage.c +9 −53
  3. kernel/bpf/bpf_inode_storage.c +2 −4
  4. kernel/bpf/bpf_local_storage.c +275 −133
  5. kernel/bpf/bpf_task_storage.c +19 −135
  6. kernel/bpf/helpers.c +0 −4
  7. net/core/bpf_sk_storage.c +14 −6
  8. tools/testing/selftests/bpf/map_tests/task_storage_map.c +0 −128
  9. tools/testing/selftests/bpf/prog_tests/btf_dump.c +2 −2
  10. tools/testing/selftests/bpf/prog_tests/task_local_storage.c +5 −5
  11. tools/testing/selftests/bpf/progs/local_storage.c +2 −17
  12. tools/testing/selftests/bpf/progs/read_bpf_task_storage_busy.c +0 −38
  13. tools/testing/selftests/bpf/progs/sk_storage_omem_uncharge.c +3 −9
  14. tools/testing/selftests/bpf/progs/task_ls_recursion.c +2 −12
  15. tools/testing/selftests/bpf/progs/task_storage_nodeadlock.c +2 −5
include/linux/bpf_local_storage.h+19−10
@@ -15,12 +15,13 @@ · 14 unmodified lines
15#include <linux/types.h>15#include <linux/types.h>
16#include <linux/bpf_mem_alloc.h>16#include <linux/bpf_mem_alloc.h>
17#include <uapi/linux/btf.h>17#include <uapi/linux/btf.h>
18#include <asm/rqspinlock.h>
18 19
19#define BPF_LOCAL_STORAGE_CACHE_SIZE 1620#define BPF_LOCAL_STORAGE_CACHE_SIZE 16
20 21
21struct bpf_local_storage_map_bucket {22struct bpf_local_storage_map_bucket {
22 struct hlist_head list;23 struct hlist_head list;
23 raw_spinlock_t lock;24 rqspinlock_t lock;
24};25};
25 26
26/* Thp map is not the primary owner of a bpf_local_storage_elem.27/* Thp map is not the primary owner of a bpf_local_storage_elem.
@@ -67,6 +68,11 @@ struct bpf_local_storage_data { · 40 unmodified lines
67 u8 data[] __aligned(8);68 u8 data[] __aligned(8);
68};69};
69 70
71#define SELEM_MAP_UNLINKED (1 << 0)
72#define SELEM_STORAGE_UNLINKED (1 << 1)
73#define SELEM_UNLINKED (SELEM_MAP_UNLINKED | SELEM_STORAGE_UNLINKED)
74#define SELEM_TOFREE (1 << 2)
75
70/* Linked to bpf_local_storage and bpf_local_storage_map */76/* Linked to bpf_local_storage and bpf_local_storage_map */
71struct bpf_local_storage_elem {77struct bpf_local_storage_elem {
72 struct hlist_node map_node; /* Linked to bpf_local_storage_map */78 struct hlist_node map_node; /* Linked to bpf_local_storage_map */
@@ -79,7 +85,9 @@ struct bpf_local_storage_elem { · 6 unmodified lines
79 * after raw_spin_unlock85 * after raw_spin_unlock
80 */86 */
81 };87 };
82 /* 8 bytes hole */88 atomic_t state;
89 bool use_kmalloc_nolock;
90 /* 3 bytes hole */
83 /* The data is stored in another cacheline to minimize91 /* The data is stored in another cacheline to minimize
84 * the number of cachelines access during a cache hit.92 * the number of cachelines access during a cache hit.
85 */93 */
@@ -88,13 +96,14 @@ struct bpf_local_storage_elem { · 2 unmodified lines
88 96
89struct bpf_local_storage {97struct bpf_local_storage {
90 struct bpf_local_storage_data __rcu *cache[BPF_LOCAL_STORAGE_CACHE_SIZE];98 struct bpf_local_storage_data __rcu *cache[BPF_LOCAL_STORAGE_CACHE_SIZE];
91 struct bpf_local_storage_map __rcu *smap;
92 struct hlist_head list; /* List of bpf_local_storage_elem */99 struct hlist_head list; /* List of bpf_local_storage_elem */
93 void *owner; /* The object that owns the above "list" of100 void *owner; /* The object that owns the above "list" of
94 * bpf_local_storage_elem.101 * bpf_local_storage_elem.
95 */102 */
96 struct rcu_head rcu;103 struct rcu_head rcu;
97 raw_spinlock_t lock; /* Protect adding/removing from the "list" */104 rqspinlock_t lock; /* Protect adding/removing from the "list" */
105 u64 mem_charge; /* Copy of mem charged to owner. Protected by "lock" */
106 refcount_t owner_refcnt;/* Used to pin owner when map_free is uncharging */
98 bool use_kmalloc_nolock;107 bool use_kmalloc_nolock;
99};108};
100 109
@@ -162,11 +171,10 @@ bpf_local_storage_lookup(struct bpf_local_storage *local_storage, · 61 unmodified lines
162 return SDATA(selem);171 return SDATA(selem);
163}172}
164 173
165void bpf_local_storage_destroy(struct bpf_local_storage *local_storage);174u32 bpf_local_storage_destroy(struct bpf_local_storage *local_storage);
166 175
167void bpf_local_storage_map_free(struct bpf_map *map,176void bpf_local_storage_map_free(struct bpf_map *map,
168 struct bpf_local_storage_cache *cache,177 struct bpf_local_storage_cache *cache);
169 int __percpu *busy_counter);
170 178
171int bpf_local_storage_map_check_btf(const struct bpf_map *map,179int bpf_local_storage_map_check_btf(const struct bpf_map *map,
172 const struct btf *btf,180 const struct btf *btf,
@@ -176,10 +184,11 @@ int bpf_local_storage_map_check_btf(const struct bpf_map *map, · 3 unmodified lines
176void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,184void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,
177 struct bpf_local_storage_elem *selem);185 struct bpf_local_storage_elem *selem);
178 186
179void bpf_selem_unlink(struct bpf_local_storage_elem *selem, bool reuse_now);187int bpf_selem_unlink(struct bpf_local_storage_elem *selem);
180 188
181void bpf_selem_link_map(struct bpf_local_storage_map *smap,189int bpf_selem_link_map(struct bpf_local_storage_map *smap,
182 struct bpf_local_storage_elem *selem);190 struct bpf_local_storage *local_storage,
191 struct bpf_local_storage_elem *selem);
183 192
184struct bpf_local_storage_elem *193struct bpf_local_storage_elem *
185bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, void *value,194bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, void *value,
kernel/bpf/bpf_cgrp_storage.c+9−53
@@ -11,29 +11,6 @@ · 10 unmodified lines
11 11
12DEFINE_BPF_STORAGE_CACHE(cgroup_cache);12DEFINE_BPF_STORAGE_CACHE(cgroup_cache);
13 13
14static DEFINE_PER_CPU(int, bpf_cgrp_storage_busy);
15
16static void bpf_cgrp_storage_lock(void)
17{
18 cant_migrate();
19 this_cpu_inc(bpf_cgrp_storage_busy);
20}
21
22static void bpf_cgrp_storage_unlock(void)
23{
24 this_cpu_dec(bpf_cgrp_storage_busy);
25}
26
27static bool bpf_cgrp_storage_trylock(void)
28{
29 cant_migrate();
30 if (unlikely(this_cpu_inc_return(bpf_cgrp_storage_busy) != 1)) {
31 this_cpu_dec(bpf_cgrp_storage_busy);
32 return false;
33 }
34 return true;
35}
36
37static struct bpf_local_storage __rcu **cgroup_storage_ptr(void *owner)14static struct bpf_local_storage __rcu **cgroup_storage_ptr(void *owner)
38{15{
39 struct cgroup *cg = owner;16 struct cgroup *cg = owner;
@@ -45,16 +22,14 @@ void bpf_cgrp_storage_free(struct cgroup *cgroup) · 5 unmodified lines
45{22{
46 struct bpf_local_storage *local_storage;23 struct bpf_local_storage *local_storage;
47 24
48 rcu_read_lock_dont_migrate();25 rcu_read_lock();
49 local_storage = rcu_dereference(cgroup->bpf_cgrp_storage);26 local_storage = rcu_dereference(cgroup->bpf_cgrp_storage);
50 if (!local_storage)27 if (!local_storage)
51 goto out;28 goto out;
52 29
53 bpf_cgrp_storage_lock();
54 bpf_local_storage_destroy(local_storage);30 bpf_local_storage_destroy(local_storage);
55 bpf_cgrp_storage_unlock();
56out:31out:
57 rcu_read_unlock_migrate();32 rcu_read_unlock();
58}33}
59 34
60static struct bpf_local_storage_data *35static struct bpf_local_storage_data *
@@ -83,9 +58,7 @@ static void *bpf_cgrp_storage_lookup_elem(struct bpf_map *map, void *key) · 22 unmodified lines
83 if (IS_ERR(cgroup))58 if (IS_ERR(cgroup))
84 return ERR_CAST(cgroup);59 return ERR_CAST(cgroup);
85 60
86 bpf_cgrp_storage_lock();
87 sdata = cgroup_storage_lookup(cgroup, map, true);61 sdata = cgroup_storage_lookup(cgroup, map, true);
88 bpf_cgrp_storage_unlock();
89 cgroup_put(cgroup);62 cgroup_put(cgroup);
90 return sdata ? sdata->data : NULL;63 return sdata ? sdata->data : NULL;
91}64}
@@ -102,10 +75,8 @@ static long bpf_cgrp_storage_update_elem(struct bpf_map *map, void *key, · 10 unmodified lines
102 if (IS_ERR(cgroup))75 if (IS_ERR(cgroup))
103 return PTR_ERR(cgroup);76 return PTR_ERR(cgroup);
104 77
105 bpf_cgrp_storage_lock();
106 sdata = bpf_local_storage_update(cgroup, (struct bpf_local_storage_map *)map,78 sdata = bpf_local_storage_update(cgroup, (struct bpf_local_storage_map *)map,
107 value, map_flags, false, GFP_ATOMIC);79 value, map_flags, false, GFP_ATOMIC);
108 bpf_cgrp_storage_unlock();
109 cgroup_put(cgroup);80 cgroup_put(cgroup);
110 return PTR_ERR_OR_ZERO(sdata);81 return PTR_ERR_OR_ZERO(sdata);
111}82}
@@ -118,8 +89,7 @@ static int cgroup_storage_delete(struct cgroup *cgroup, struct bpf_map *map) · 6 unmodified lines
118 if (!sdata)89 if (!sdata)
119 return -ENOENT;90 return -ENOENT;
120 91
121 bpf_selem_unlink(SELEM(sdata), false);92 return bpf_selem_unlink(SELEM(sdata));
122 return 0;
123}93}
124 94
125static long bpf_cgrp_storage_delete_elem(struct bpf_map *map, void *key)95static long bpf_cgrp_storage_delete_elem(struct bpf_map *map, void *key)
@@ -132,9 +102,7 @@ static long bpf_cgrp_storage_delete_elem(struct bpf_map *map, void *key) · 6 unmodified lines
132 if (IS_ERR(cgroup))102 if (IS_ERR(cgroup))
133 return PTR_ERR(cgroup);103 return PTR_ERR(cgroup);
134 104
135 bpf_cgrp_storage_lock();
136 err = cgroup_storage_delete(cgroup, map);105 err = cgroup_storage_delete(cgroup, map);
137 bpf_cgrp_storage_unlock();
138 cgroup_put(cgroup);106 cgroup_put(cgroup);
139 return err;107 return err;
140}108}
@@ -151,7 +119,7 @@ static struct bpf_map *cgroup_storage_map_alloc(union bpf_attr *attr) · 10 unmodified lines
151 119
152static void cgroup_storage_map_free(struct bpf_map *map)120static void cgroup_storage_map_free(struct bpf_map *map)
153{121{
154 bpf_local_storage_map_free(map, &cgroup_cache, &bpf_cgrp_storage_busy);122 bpf_local_storage_map_free(map, &cgroup_cache);
155}123}
156 124
157/* *gfp_flags* is a hidden argument provided by the verifier */125/* *gfp_flags* is a hidden argument provided by the verifier */
@@ -159,7 +127,6 @@ BPF_CALL_5(bpf_cgrp_storage_get, struct bpf_map *, map, struct cgroup *, cgroup, · 1 unmodified line
159 void *, value, u64, flags, gfp_t, gfp_flags)127 void *, value, u64, flags, gfp_t, gfp_flags)
160{128{
161 struct bpf_local_storage_data *sdata;129 struct bpf_local_storage_data *sdata;
162 bool nobusy;
163 130
164 WARN_ON_ONCE(!bpf_rcu_lock_held());131 WARN_ON_ONCE(!bpf_rcu_lock_held());
165 if (flags & ~(BPF_LOCAL_STORAGE_GET_F_CREATE))132 if (flags & ~(BPF_LOCAL_STORAGE_GET_F_CREATE))
@@ -168,38 +135,27 @@ BPF_CALL_5(bpf_cgrp_storage_get, struct bpf_map *, map, struct cgroup *, cgroup, · 2 unmodified lines
168 if (!cgroup)135 if (!cgroup)
169 return (unsigned long)NULL;136 return (unsigned long)NULL;
170 137
171 nobusy = bpf_cgrp_storage_trylock();138 sdata = cgroup_storage_lookup(cgroup, map, true);
172
173 sdata = cgroup_storage_lookup(cgroup, map, nobusy);
174 if (sdata)139 if (sdata)
175 goto unlock;140 goto out;
176 141
177 /* only allocate new storage, when the cgroup is refcounted */142 /* only allocate new storage, when the cgroup is refcounted */
178 if (!percpu_ref_is_dying(&cgroup->self.refcnt) &&143 if (!percpu_ref_is_dying(&cgroup->self.refcnt) &&
179 (flags & BPF_LOCAL_STORAGE_GET_F_CREATE) && nobusy)144 (flags & BPF_LOCAL_STORAGE_GET_F_CREATE))
180 sdata = bpf_local_storage_update(cgroup, (struct bpf_local_storage_map *)map,145 sdata = bpf_local_storage_update(cgroup, (struct bpf_local_storage_map *)map,
181 value, BPF_NOEXIST, false, gfp_flags);146 value, BPF_NOEXIST, false, gfp_flags);
182 147
183unlock:148out:
184 if (nobusy)
185 bpf_cgrp_storage_unlock();
186 return IS_ERR_OR_NULL(sdata) ? (unsigned long)NULL : (unsigned long)sdata->data;149 return IS_ERR_OR_NULL(sdata) ? (unsigned long)NULL : (unsigned long)sdata->data;
187}150}
188 151
189BPF_CALL_2(bpf_cgrp_storage_delete, struct bpf_map *, map, struct cgroup *, cgroup)152BPF_CALL_2(bpf_cgrp_storage_delete, struct bpf_map *, map, struct cgroup *, cgroup)
190{153{
191 int ret;
192
193 WARN_ON_ONCE(!bpf_rcu_lock_held());154 WARN_ON_ONCE(!bpf_rcu_lock_held());
194 if (!cgroup)155 if (!cgroup)
195 return -EINVAL;156 return -EINVAL;
196 157
197 if (!bpf_cgrp_storage_trylock())158 return cgroup_storage_delete(cgroup, map);
198 return -EBUSY;
199
200 ret = cgroup_storage_delete(cgroup, map);
201 bpf_cgrp_storage_unlock();
202 return ret;
203}159}
204 160
205const struct bpf_map_ops cgrp_storage_map_ops = {161const struct bpf_map_ops cgrp_storage_map_ops = {
kernel/bpf/bpf_inode_storage.c+2−4
@@ -110,9 +110,7 @@ static int inode_storage_delete(struct inode *inode, struct bpf_map *map) · 109 unmodified lines
110 if (!sdata)110 if (!sdata)
111 return -ENOENT;111 return -ENOENT;
112 112
113 bpf_selem_unlink(SELEM(sdata), false);113 return bpf_selem_unlink(SELEM(sdata));
114
115 return 0;
116}114}
117 115
118static long bpf_fd_inode_storage_delete_elem(struct bpf_map *map, void *key)116static long bpf_fd_inode_storage_delete_elem(struct bpf_map *map, void *key)
@@ -186,7 +184,7 @@ static struct bpf_map *inode_storage_map_alloc(union bpf_attr *attr) · 67 unmodified lines
186 184
187static void inode_storage_map_free(struct bpf_map *map)185static void inode_storage_map_free(struct bpf_map *map)
188{186{
189 bpf_local_storage_map_free(map, &inode_cache, NULL);187 bpf_local_storage_map_free(map, &inode_cache);
190}188}
191 189
192const struct bpf_map_ops inode_storage_map_ops = {190const struct bpf_map_ops inode_storage_map_ops = {
kernel/bpf/bpf_local_storage.c+275−133
@@ -19,9 +19,9 @@ · 18 unmodified lines
19 19
20static struct bpf_local_storage_map_bucket *20static struct bpf_local_storage_map_bucket *
21select_bucket(struct bpf_local_storage_map *smap,21select_bucket(struct bpf_local_storage_map *smap,
22 struct bpf_local_storage_elem *selem)22 struct bpf_local_storage *local_storage)
23{23{
24 return &smap->buckets[hash_ptr(selem, smap->bucket_log)];24 return &smap->buckets[hash_ptr(local_storage, smap->bucket_log)];
25}25}
26 26
27static int mem_charge(struct bpf_local_storage_map *smap, void *owner, u32 size)27static int mem_charge(struct bpf_local_storage_map *smap, void *owner, u32 size)
@@ -61,11 +61,6 @@ static bool selem_linked_to_storage(const struct bpf_local_storage_elem *selem) · 33 unmodified lines
61 return !hlist_unhashed(&selem->snode);61 return !hlist_unhashed(&selem->snode);
62}62}
63 63
64static bool selem_linked_to_map_lockless(const struct bpf_local_storage_elem *selem)
65{
66 return !hlist_unhashed_lockless(&selem->map_node);
67}
68
69static bool selem_linked_to_map(const struct bpf_local_storage_elem *selem)64static bool selem_linked_to_map(const struct bpf_local_storage_elem *selem)
70{65{
71 return !hlist_unhashed(&selem->map_node);66 return !hlist_unhashed(&selem->map_node);
@@ -90,6 +85,8 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, · 18 unmodified lines
90 85
91 if (selem) {86 if (selem) {
92 RCU_INIT_POINTER(SDATA(selem)->smap, smap);87 RCU_INIT_POINTER(SDATA(selem)->smap, smap);
88 atomic_set(&selem->state, 0);
89 selem->use_kmalloc_nolock = smap->use_kmalloc_nolock;
93 90
94 if (value) {91 if (value) {
95 /* No need to call check_and_init_map_value as memory is zero init */92 /* No need to call check_and_init_map_value as memory is zero init */
@@ -198,9 +195,11 @@ static void bpf_selem_free_rcu(struct rcu_head *rcu) · 102 unmodified lines
198 /* The bpf_local_storage_map_free will wait for rcu_barrier */195 /* The bpf_local_storage_map_free will wait for rcu_barrier */
199 smap = rcu_dereference_check(SDATA(selem)->smap, 1);196 smap = rcu_dereference_check(SDATA(selem)->smap, 1);
200 197
201 migrate_disable();198 if (smap) {
202 bpf_obj_free_fields(smap->map.record, SDATA(selem)->data);199 migrate_disable();
203 migrate_enable();200 bpf_obj_free_fields(smap->map.record, SDATA(selem)->data);
201 migrate_enable();
202 }
204 kfree_nolock(selem);203 kfree_nolock(selem);
205}204}
206 205
@@ -219,13 +218,14 @@ void bpf_selem_free(struct bpf_local_storage_elem *selem, · 12 unmodified lines
219 218
220 smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held());219 smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held());
221 220
222 if (!smap->use_kmalloc_nolock) {221 if (!selem->use_kmalloc_nolock) {
223 /*222 /*
224 * No uptr will be unpin even when reuse_now == false since uptr223 * No uptr will be unpin even when reuse_now == false since uptr
225 * is only supported in task local storage, where224 * is only supported in task local storage, where
226 * smap->use_kmalloc_nolock == true.225 * smap->use_kmalloc_nolock == true.
227 */226 */
228 bpf_obj_free_fields(smap->map.record, SDATA(selem)->data);227 if (smap)
228 bpf_obj_free_fields(smap->map.record, SDATA(selem)->data);
229 __bpf_selem_free(selem, reuse_now);229 __bpf_selem_free(selem, reuse_now);
230 return;230 return;
231 }231 }
@@ -256,6 +256,36 @@ static void bpf_selem_free_list(struct hlist_head *list, bool reuse_now) · 24 unmodified lines
256 bpf_selem_free(selem, reuse_now);256 bpf_selem_free(selem, reuse_now);
257}257}
258 258
259static void bpf_selem_unlink_storage_nolock_misc(struct bpf_local_storage_elem *selem,
260 struct bpf_local_storage_map *smap,
261 struct bpf_local_storage *local_storage,
262 bool free_local_storage, bool pin_owner)
263{
264 void *owner = local_storage->owner;
265 u32 uncharge = smap->elem_size;
266
267 if (rcu_access_pointer(local_storage->cache[smap->cache_idx]) ==
268 SDATA(selem))
269 RCU_INIT_POINTER(local_storage->cache[smap->cache_idx], NULL);
270
271 if (pin_owner && !refcount_inc_not_zero(&local_storage->owner_refcnt))
272 return;
273
274 uncharge += free_local_storage ? sizeof(*local_storage) : 0;
275 mem_uncharge(smap, local_storage->owner, uncharge);
276 local_storage->mem_charge -= uncharge;
277
278 if (free_local_storage) {
279 local_storage->owner = NULL;
280
281 /* After this RCU_INIT, owner may be freed and cannot be used */
282 RCU_INIT_POINTER(*owner_storage(smap, owner), NULL);
283 }
284
285 if (pin_owner)
286 refcount_dec(&local_storage->owner_refcnt);
287}
288
259/* local_storage->lock must be held and selem->local_storage == local_storage.289/* local_storage->lock must be held and selem->local_storage == local_storage.
260 * The caller must ensure selem->smap is still valid to be290 * The caller must ensure selem->smap is still valid to be
261 * dereferenced for its smap->elem_size and smap->cache_idx.291 * dereferenced for its smap->elem_size and smap->cache_idx.
@@ -266,124 +296,219 @@ static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_stor · 4 unmodified lines
266{296{
267 struct bpf_local_storage_map *smap;297 struct bpf_local_storage_map *smap;
268 bool free_local_storage;298 bool free_local_storage;
269 void *owner;
270 299
271 smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held());300 smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held());
272 owner = local_storage->owner;
273
274 /* All uncharging on the owner must be done first.
275 * The owner may be freed once the last selem is unlinked
276 * from local_storage.
277 */
278 mem_uncharge(smap, owner, smap->elem_size);
279 301
280 free_local_storage = hlist_is_singular_node(&selem->snode,302 free_local_storage = hlist_is_singular_node(&selem->snode,
281 &local_storage->list);303 &local_storage->list);
282 if (free_local_storage) {
283 mem_uncharge(smap, owner, sizeof(struct bpf_local_storage));
284 local_storage->owner = NULL;
285 304
286 /* After this RCU_INIT, owner may be freed and cannot be used */305 bpf_selem_unlink_storage_nolock_misc(selem, smap, local_storage,
287 RCU_INIT_POINTER(*owner_storage(smap, owner), NULL);306 free_local_storage, false);
288 307
289 /* local_storage is not freed now. local_storage->lock is
290 * still held and raw_spin_unlock_bh(&local_storage->lock)
291 * will be done by the caller.
292 *
293 * Although the unlock will be done under
294 * rcu_read_lock(), it is more intuitive to
295 * read if the freeing of the storage is done
296 * after the raw_spin_unlock_bh(&local_storage->lock).
297 *
298 * Hence, a "bool free_local_storage" is returned
299 * to the caller which then calls then frees the storage after
300 * all the RCU grace periods have expired.
301 */
302 }
303 hlist_del_init_rcu(&selem->snode);308 hlist_del_init_rcu(&selem->snode);
304 if (rcu_access_pointer(local_storage->cache[smap->cache_idx]) ==
305 SDATA(selem))
306 RCU_INIT_POINTER(local_storage->cache[smap->cache_idx], NULL);
307 309
308 hlist_add_head(&selem->free_node, free_selem_list);310 hlist_add_head(&selem->free_node, free_selem_list);
309 311
310 if (rcu_access_pointer(local_storage->smap) == smap)
311 RCU_INIT_POINTER(local_storage->smap, NULL);
312
313 return free_local_storage;312 return free_local_storage;
314}313}
315 314
316static void bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem,315void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,
317 bool reuse_now)316 struct bpf_local_storage_elem *selem)
317{
318 struct bpf_local_storage_map *smap;
319
320 smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held());
321 local_storage->mem_charge += smap->elem_size;
322
323 RCU_INIT_POINTER(selem->local_storage, local_storage);
324 hlist_add_head_rcu(&selem->snode, &local_storage->list);
325}
326
327static int bpf_selem_unlink_map(struct bpf_local_storage_elem *selem)
328{
329 struct bpf_local_storage *local_storage;
330 struct bpf_local_storage_map *smap;
331 struct bpf_local_storage_map_bucket *b;
332 unsigned long flags;
333 int err;
334
335 local_storage = rcu_dereference_check(selem->local_storage,
336 bpf_rcu_lock_held());
337 smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held());
338 b = select_bucket(smap, local_storage);
339 err = raw_res_spin_lock_irqsave(&b->lock, flags);
340 if (err)
341 return err;
342
343 hlist_del_init_rcu(&selem->map_node);
344 raw_res_spin_unlock_irqrestore(&b->lock, flags);
345
346 return 0;
347}
348
349static void bpf_selem_unlink_map_nolock(struct bpf_local_storage_elem *selem)
350{
351 hlist_del_init_rcu(&selem->map_node);
352}
353
354int bpf_selem_link_map(struct bpf_local_storage_map *smap,
355 struct bpf_local_storage *local_storage,
356 struct bpf_local_storage_elem *selem)
357{
358 struct bpf_local_storage_map_bucket *b;
359 unsigned long flags;
360 int err;
361
362 b = select_bucket(smap, local_storage);
363
364 err = raw_res_spin_lock_irqsave(&b->lock, flags);
365 if (err)
366 return err;
367
368 hlist_add_head_rcu(&selem->map_node, &b->list);
369 raw_res_spin_unlock_irqrestore(&b->lock, flags);
370
371 return 0;
372}
373
374static void bpf_selem_link_map_nolock(struct bpf_local_storage_map_bucket *b,
375 struct bpf_local_storage_elem *selem)
376{
377 hlist_add_head_rcu(&selem->map_node, &b->list);
378}
379
380/*
381 * Unlink an selem from map and local storage with lock held.
382 * This is the common path used by local storages to delete an selem.
383 */
384int bpf_selem_unlink(struct bpf_local_storage_elem *selem)
318{385{
319 struct bpf_local_storage *local_storage;386 struct bpf_local_storage *local_storage;
320 bool free_local_storage = false;387 bool free_local_storage = false;
321 HLIST_HEAD(selem_free_list);388 HLIST_HEAD(selem_free_list);
322 unsigned long flags;389 unsigned long flags;
390 int err;
323 391
324 if (unlikely(!selem_linked_to_storage_lockless(selem)))392 if (unlikely(!selem_linked_to_storage_lockless(selem)))
325 /* selem has already been unlinked from sk */393 /* selem has already been unlinked from sk */
326 return;394 return 0;
327 395
328 local_storage = rcu_dereference_check(selem->local_storage,396 local_storage = rcu_dereference_check(selem->local_storage,
329 bpf_rcu_lock_held());397 bpf_rcu_lock_held());
330 398
331 raw_spin_lock_irqsave(&local_storage->lock, flags);399 err = raw_res_spin_lock_irqsave(&local_storage->lock, flags);
332 if (likely(selem_linked_to_storage(selem)))400 if (err)
401 return err;
402
403 if (likely(selem_linked_to_storage(selem))) {
404 /* Always unlink from map before unlinking from local_storage
405 * because selem will be freed after successfully unlinked from
406 * the local_storage.
407 */
408 err = bpf_selem_unlink_map(selem);
409 if (err)
410 goto out;
411
333 free_local_storage = bpf_selem_unlink_storage_nolock(412 free_local_storage = bpf_selem_unlink_storage_nolock(
334 local_storage, selem, &selem_free_list);413 local_storage, selem, &selem_free_list);
335 raw_spin_unlock_irqrestore(&local_storage->lock, flags);414 }
415out:
416 raw_res_spin_unlock_irqrestore(&local_storage->lock, flags);
336 417
337 bpf_selem_free_list(&selem_free_list, reuse_now);418 bpf_selem_free_list(&selem_free_list, false);
338 419
339 if (free_local_storage)420 if (free_local_storage)
340 bpf_local_storage_free(local_storage, reuse_now);421 bpf_local_storage_free(local_storage, false);
341}
342 422
343void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,423 return err;
344 struct bpf_local_storage_elem *selem)
345{
346 RCU_INIT_POINTER(selem->local_storage, local_storage);
347 hlist_add_head_rcu(&selem->snode, &local_storage->list);
348}424}
349 425
350static void bpf_selem_unlink_map(struct bpf_local_storage_elem *selem)426/*
427 * Unlink an selem from map and local storage with lockless fallback if callers
428 * are racing or rqspinlock returns error. It should only be called by
429 * bpf_local_storage_destroy() or bpf_local_storage_map_free().
430 */
431static void bpf_selem_unlink_nofail(struct bpf_local_storage_elem *selem,
432 struct bpf_local_storage_map_bucket *b)
351{433{
434 bool in_map_free = !!b, free_storage = false;
435 struct bpf_local_storage *local_storage;
352 struct bpf_local_storage_map *smap;436 struct bpf_local_storage_map *smap;
353 struct bpf_local_storage_map_bucket *b;
354 unsigned long flags;437 unsigned long flags;
438 int err, unlink = 0;
355 439
356 if (unlikely(!selem_linked_to_map_lockless(selem)))440 local_storage = rcu_dereference_check(selem->local_storage, bpf_rcu_lock_held());
357 /* selem has already be unlinked from smap */
358 return;
359
360 smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held());441 smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held());
361 b = select_bucket(smap, selem);
362 raw_spin_lock_irqsave(&b->lock, flags);
363 if (likely(selem_linked_to_map(selem)))
364 hlist_del_init_rcu(&selem->map_node);
365 raw_spin_unlock_irqrestore(&b->lock, flags);
366}
367 442
368void bpf_selem_link_map(struct bpf_local_storage_map *smap,443 if (smap) {
369 struct bpf_local_storage_elem *selem)444 b = b ? : select_bucket(smap, local_storage);
370{445 err = raw_res_spin_lock_irqsave(&b->lock, flags);
371 struct bpf_local_storage_map_bucket *b = select_bucket(smap, selem);446 if (!err) {
372 unsigned long flags;447 /*
448 * Call bpf_obj_free_fields() under b->lock to make sure it is done
449 * exactly once for an selem. Safe to free special fields immediately
450 * as no BPF program should be referencing the selem.
451 */
452 if (likely(selem_linked_to_map(selem))) {
453 hlist_del_init_rcu(&selem->map_node);
454 bpf_obj_free_fields(smap->map.record, SDATA(selem)->data);
455 unlink++;
456 }
457 raw_res_spin_unlock_irqrestore(&b->lock, flags);
458 }
459 /*
460 * Highly unlikely scenario: resource leak
461 *
462 * When map_free(selem1), destroy(selem1) and destroy(selem2) are racing
463 * and both selem belong to the same bucket, if destroy(selem2) acquired
464 * b->lock and block for too long, neither map_free(selem1) and
465 * destroy(selem1) will be able to free the special field associated
466 * with selem1 as raw_res_spin_lock_irqsave() returns -ETIMEDOUT.
467 */
468 WARN_ON_ONCE(err && in_map_free);
469 if (!err || in_map_free)
470 RCU_INIT_POINTER(SDATA(selem)->smap, NULL);
471 }
373 472
374 raw_spin_lock_irqsave(&b->lock, flags);473 if (local_storage) {
375 hlist_add_head_rcu(&selem->map_node, &b->list);474 err = raw_res_spin_lock_irqsave(&local_storage->lock, flags);
376 raw_spin_unlock_irqrestore(&b->lock, flags);475 if (!err) {
377}476 if (likely(selem_linked_to_storage(selem))) {
477 free_storage = hlist_is_singular_node(&selem->snode,
478 &local_storage->list);
479 /*
480 * Okay to skip clearing owner_storage and storage->owner in
481 * destroy() since the owner is going away. No user or bpf
482 * programs should be able to reference it.
483 */
484 if (smap && in_map_free)
485 bpf_selem_unlink_storage_nolock_misc(
486 selem, smap, local_storage,
487 free_storage, true);
488 hlist_del_init_rcu(&selem->snode);
489 unlink++;
490 }
491 raw_res_spin_unlock_irqrestore(&local_storage->lock, flags);
492 }
493 if (!err || !in_map_free)
494 RCU_INIT_POINTER(selem->local_storage, NULL);
495 }
378 496
379void bpf_selem_unlink(struct bpf_local_storage_elem *selem, bool reuse_now)497 if (unlink != 2)
380{498 atomic_or(in_map_free ? SELEM_MAP_UNLINKED : SELEM_STORAGE_UNLINKED, &selem->state);
381 /* Always unlink from map before unlinking from local_storage499
382 * because selem will be freed after successfully unlinked from500 /*
383 * the local_storage.501 * Normally, an selem can be unlinked under local_storage->lock and b->lock, and
502 * then freed after an RCU grace period. However, if destroy() and map_free() are
503 * racing or rqspinlock returns errors in unlikely situations (unlink != 2), free
504 * the selem only after both map_free() and destroy() see the selem.
384 */505 */
385 bpf_selem_unlink_map(selem);506 if (unlink == 2 ||
386 bpf_selem_unlink_storage(selem, reuse_now);507 atomic_cmpxchg(&selem->state, SELEM_UNLINKED, SELEM_TOFREE) == SELEM_UNLINKED)
508 bpf_selem_free(selem, true);
509
510 if (free_storage)
511 bpf_local_storage_free(local_storage, true);
387}512}
388 513
389void __bpf_local_storage_insert_cache(struct bpf_local_storage *local_storage,514void __bpf_local_storage_insert_cache(struct bpf_local_storage *local_storage,
@@ -391,16 +516,20 @@ void __bpf_local_storage_insert_cache(struct bpf_local_storage *local_storage, · 1 unmodified line
391 struct bpf_local_storage_elem *selem)516 struct bpf_local_storage_elem *selem)
392{517{
393 unsigned long flags;518 unsigned long flags;
519 int err;
394 520
395 /* spinlock is needed to avoid racing with the521 /* spinlock is needed to avoid racing with the
396 * parallel delete. Otherwise, publishing an already522 * parallel delete. Otherwise, publishing an already
397 * deleted sdata to the cache will become a use-after-free523 * deleted sdata to the cache will become a use-after-free
398 * problem in the next bpf_local_storage_lookup().524 * problem in the next bpf_local_storage_lookup().
399 */525 */
400 raw_spin_lock_irqsave(&local_storage->lock, flags);526 err = raw_res_spin_lock_irqsave(&local_storage->lock, flags);
527 if (err)
528 return;
529
401 if (selem_linked_to_storage(selem))530 if (selem_linked_to_storage(selem))
402 rcu_assign_pointer(local_storage->cache[smap->cache_idx], SDATA(selem));531 rcu_assign_pointer(local_storage->cache[smap->cache_idx], SDATA(selem));
403 raw_spin_unlock_irqrestore(&local_storage->lock, flags);532 raw_res_spin_unlock_irqrestore(&local_storage->lock, flags);
404}533}
405 534
406static int check_flags(const struct bpf_local_storage_data *old_sdata,535static int check_flags(const struct bpf_local_storage_data *old_sdata,
@@ -424,6 +553,8 @@ int bpf_local_storage_alloc(void *owner, · 17 unmodified lines
424{553{
425 struct bpf_local_storage *prev_storage, *storage;554 struct bpf_local_storage *prev_storage, *storage;
426 struct bpf_local_storage **owner_storage_ptr;555 struct bpf_local_storage **owner_storage_ptr;
556 struct bpf_local_storage_map_bucket *b;
557 unsigned long flags;
427 int err;558 int err;
428 559
429 err = mem_charge(smap, owner, sizeof(*storage));560 err = mem_charge(smap, owner, sizeof(*storage));
@@ -441,14 +572,21 @@ int bpf_local_storage_alloc(void *owner, · 11 unmodified lines
441 goto uncharge;572 goto uncharge;
442 }573 }
443 574
444 RCU_INIT_POINTER(storage->smap, smap);
445 INIT_HLIST_HEAD(&storage->list);575 INIT_HLIST_HEAD(&storage->list);
446 raw_spin_lock_init(&storage->lock);576 raw_res_spin_lock_init(&storage->lock);
447 storage->owner = owner;577 storage->owner = owner;
578 storage->mem_charge = sizeof(*storage);
448 storage->use_kmalloc_nolock = smap->use_kmalloc_nolock;579 storage->use_kmalloc_nolock = smap->use_kmalloc_nolock;
580 refcount_set(&storage->owner_refcnt, 1);
449 581
450 bpf_selem_link_storage_nolock(storage, first_selem);582 bpf_selem_link_storage_nolock(storage, first_selem);
451 bpf_selem_link_map(smap, first_selem);583
584 b = select_bucket(smap, storage);
585 err = raw_res_spin_lock_irqsave(&b->lock, flags);
586 if (err)
587 goto uncharge;
588
589 bpf_selem_link_map_nolock(b, first_selem);
452 590
453 owner_storage_ptr =591 owner_storage_ptr =
454 (struct bpf_local_storage **)owner_storage(smap, owner);592 (struct bpf_local_storage **)owner_storage(smap, owner);
@@ -464,10 +602,12 @@ int bpf_local_storage_alloc(void *owner, · 9 unmodified lines
464 */602 */
465 prev_storage = cmpxchg(owner_storage_ptr, NULL, storage);603 prev_storage = cmpxchg(owner_storage_ptr, NULL, storage);
466 if (unlikely(prev_storage)) {604 if (unlikely(prev_storage)) {
467 bpf_selem_unlink_map(first_selem);605 bpf_selem_unlink_map_nolock(first_selem);
606 raw_res_spin_unlock_irqrestore(&b->lock, flags);
468 err = -EAGAIN;607 err = -EAGAIN;
469 goto uncharge;608 goto uncharge;
470 }609 }
610 raw_res_spin_unlock_irqrestore(&b->lock, flags);
471 611
472 return 0;612 return 0;
473 613
@@ -489,8 +629,9 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, · 15 unmodified lines
489 struct bpf_local_storage_data *old_sdata = NULL;629 struct bpf_local_storage_data *old_sdata = NULL;
490 struct bpf_local_storage_elem *alloc_selem, *selem = NULL;630 struct bpf_local_storage_elem *alloc_selem, *selem = NULL;
491 struct bpf_local_storage *local_storage;631 struct bpf_local_storage *local_storage;
632 struct bpf_local_storage_map_bucket *b;
492 HLIST_HEAD(old_selem_free_list);633 HLIST_HEAD(old_selem_free_list);
493 unsigned long flags;634 unsigned long flags, b_flags;
494 int err;635 int err;
495 636
496 /* BPF_EXIST and BPF_NOEXIST cannot be both set */637 /* BPF_EXIST and BPF_NOEXIST cannot be both set */
@@ -549,7 +690,9 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, · 52 unmodified lines
549 if (!alloc_selem)690 if (!alloc_selem)
550 return ERR_PTR(-ENOMEM);691 return ERR_PTR(-ENOMEM);
551 692
552 raw_spin_lock_irqsave(&local_storage->lock, flags);693 err = raw_res_spin_lock_irqsave(&local_storage->lock, flags);
694 if (err)
695 goto free_selem;
553 696
554 /* Recheck local_storage->list under local_storage->lock */697 /* Recheck local_storage->list under local_storage->lock */
555 if (unlikely(hlist_empty(&local_storage->list))) {698 if (unlikely(hlist_empty(&local_storage->list))) {
@@ -574,22 +717,30 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, · 18 unmodified lines
574 goto unlock;717 goto unlock;
575 }718 }
576 719
720 b = select_bucket(smap, local_storage);
721
722 err = raw_res_spin_lock_irqsave(&b->lock, b_flags);
723 if (err)
724 goto unlock;
725
577 alloc_selem = NULL;726 alloc_selem = NULL;
578 /* First, link the new selem to the map */727 /* First, link the new selem to the map */
579 bpf_selem_link_map(smap, selem);728 bpf_selem_link_map_nolock(b, selem);
580 729
581 /* Second, link (and publish) the new selem to local_storage */730 /* Second, link (and publish) the new selem to local_storage */
582 bpf_selem_link_storage_nolock(local_storage, selem);731 bpf_selem_link_storage_nolock(local_storage, selem);
583 732
584 /* Third, remove old selem, SELEM(old_sdata) */733 /* Third, remove old selem, SELEM(old_sdata) */
585 if (old_sdata) {734 if (old_sdata) {
586 bpf_selem_unlink_map(SELEM(old_sdata));735 bpf_selem_unlink_map_nolock(SELEM(old_sdata));
587 bpf_selem_unlink_storage_nolock(local_storage, SELEM(old_sdata),736 bpf_selem_unlink_storage_nolock(local_storage, SELEM(old_sdata),
588 &old_selem_free_list);737 &old_selem_free_list);
589 }738 }
590 739
740 raw_res_spin_unlock_irqrestore(&b->lock, b_flags);
591unlock:741unlock:
592 raw_spin_unlock_irqrestore(&local_storage->lock, flags);742 raw_res_spin_unlock_irqrestore(&local_storage->lock, flags);
743free_selem:
593 bpf_selem_free_list(&old_selem_free_list, false);744 bpf_selem_free_list(&old_selem_free_list, false);
594 if (alloc_selem) {745 if (alloc_selem) {
595 mem_uncharge(smap, owner, smap->elem_size);746 mem_uncharge(smap, owner, smap->elem_size);
@@ -657,13 +808,13 @@ int bpf_local_storage_map_check_btf(const struct bpf_map *map, · 61 unmodified lines
657 return 0;808 return 0;
658}809}
659 810
660void bpf_local_storage_destroy(struct bpf_local_storage *local_storage)811/*
812 * Destroy local storage when the owner is going away. Caller must uncharge memory
813 * if memory charging is used.
814 */
815u32 bpf_local_storage_destroy(struct bpf_local_storage *local_storage)
661{816{
662 struct bpf_local_storage_elem *selem;817 struct bpf_local_storage_elem *selem;
663 bool free_storage = false;
664 HLIST_HEAD(free_selem_list);
665 struct hlist_node *n;
666 unsigned long flags;
667 818
668 /* Neither the bpf_prog nor the bpf_map's syscall819 /* Neither the bpf_prog nor the bpf_map's syscall
669 * could be modifying the local_storage->list now.820 * could be modifying the local_storage->list now.
@@ -674,27 +825,20 @@ void bpf_local_storage_destroy(struct bpf_local_storage *local_storage) · 4 unmodified lines
674 * when unlinking elem from the local_storage->list and825 * when unlinking elem from the local_storage->list and
675 * the map's bucket->list.826 * the map's bucket->list.
676 */827 */
677 raw_spin_lock_irqsave(&local_storage->lock, flags);828 hlist_for_each_entry_rcu(selem, &local_storage->list, snode)
678 hlist_for_each_entry_safe(selem, n, &local_storage->list, snode) {829 bpf_selem_unlink_nofail(selem, NULL);
679 /* Always unlink from map before unlinking from830
680 * local_storage.831 if (!refcount_dec_and_test(&local_storage->owner_refcnt)) {
681 */832 while (refcount_read(&local_storage->owner_refcnt))
682 bpf_selem_unlink_map(selem);833 cpu_relax();
683 /* If local_storage list has only one element, the834 /*
684 * bpf_selem_unlink_storage_nolock() will return true.835 * Paired with refcount_dec() in bpf_selem_unlink_nofail()
685 * Otherwise, it will return false. The current loop iteration836 * to make sure destroy() sees the correct local_storage->mem_charge.
686 * intends to remove all local storage. So the last iteration
687 * of the loop will set the free_cgroup_storage to true.
688 */837 */
689 free_storage = bpf_selem_unlink_storage_nolock(838 smp_mb();
690 local_storage, selem, &free_selem_list);
691 }839 }
692 raw_spin_unlock_irqrestore(&local_storage->lock, flags);
693
694 bpf_selem_free_list(&free_selem_list, true);
695 840
696 if (free_storage)841 return local_storage->mem_charge;
697 bpf_local_storage_free(local_storage, true);
698}842}
699 843
700u64 bpf_local_storage_map_mem_usage(const struct bpf_map *map)844u64 bpf_local_storage_map_mem_usage(const struct bpf_map *map)
@@ -736,7 +880,7 @@ bpf_local_storage_map_alloc(union bpf_attr *attr, · 35 unmodified lines
736 880
737 for (i = 0; i < nbuckets; i++) {881 for (i = 0; i < nbuckets; i++) {
738 INIT_HLIST_HEAD(&smap->buckets[i].list);882 INIT_HLIST_HEAD(&smap->buckets[i].list);
739 raw_spin_lock_init(&smap->buckets[i].lock);883 raw_res_spin_lock_init(&smap->buckets[i].lock);
740 }884 }
741 885
742 smap->elem_size = offsetof(struct bpf_local_storage_elem,886 smap->elem_size = offsetof(struct bpf_local_storage_elem,
@@ -758,8 +902,7 @@ free_smap: · 15 unmodified lines
758}902}
759 903
760void bpf_local_storage_map_free(struct bpf_map *map,904void bpf_local_storage_map_free(struct bpf_map *map,
761 struct bpf_local_storage_cache *cache,905 struct bpf_local_storage_cache *cache)
762 int __percpu *busy_counter)
763{906{
764 struct bpf_local_storage_map_bucket *b;907 struct bpf_local_storage_map_bucket *b;
765 struct bpf_local_storage_elem *selem;908 struct bpf_local_storage_elem *selem;
@@ -789,15 +932,14 @@ void bpf_local_storage_map_free(struct bpf_map *map, · 23 unmodified lines
789 932
790 rcu_read_lock();933 rcu_read_lock();
791 /* No one is adding to b->list now */934 /* No one is adding to b->list now */
792 while ((selem = hlist_entry_safe(935restart:
793 rcu_dereference_raw(hlist_first_rcu(&b->list)),936 hlist_for_each_entry_rcu(selem, &b->list, map_node) {
794 struct bpf_local_storage_elem, map_node))) {937 bpf_selem_unlink_nofail(selem, b);
795 if (busy_counter)938
796 this_cpu_inc(*busy_counter);939 if (need_resched()) {
797 bpf_selem_unlink(selem, true);940 cond_resched_rcu();
798 if (busy_counter)941 goto restart;
799 this_cpu_dec(*busy_counter);942 }
800 cond_resched_rcu();
801 }943 }
802 rcu_read_unlock();944 rcu_read_unlock();
803 }945 }
kernel/bpf/bpf_task_storage.c+19−135
@@ -20,29 +20,6 @@ · 19 unmodified lines
20 20
21DEFINE_BPF_STORAGE_CACHE(task_cache);21DEFINE_BPF_STORAGE_CACHE(task_cache);
22 22
23static DEFINE_PER_CPU(int, bpf_task_storage_busy);
24
25static void bpf_task_storage_lock(void)
26{
27 cant_migrate();
28 this_cpu_inc(bpf_task_storage_busy);
29}
30
31static void bpf_task_storage_unlock(void)
32{
33 this_cpu_dec(bpf_task_storage_busy);
34}
35
36static bool bpf_task_storage_trylock(void)
37{
38 cant_migrate();
39 if (unlikely(this_cpu_inc_return(bpf_task_storage_busy) != 1)) {
40 this_cpu_dec(bpf_task_storage_busy);
41 return false;
42 }
43 return true;
44}
45
46static struct bpf_local_storage __rcu **task_storage_ptr(void *owner)23static struct bpf_local_storage __rcu **task_storage_ptr(void *owner)
47{24{
48 struct task_struct *task = owner;25 struct task_struct *task = owner;
@@ -70,17 +47,15 @@ void bpf_task_storage_free(struct task_struct *task) · 21 unmodified lines
70{47{
71 struct bpf_local_storage *local_storage;48 struct bpf_local_storage *local_storage;
72 49
73 rcu_read_lock_dont_migrate();50 rcu_read_lock();
74 51
75 local_storage = rcu_dereference(task->bpf_storage);52 local_storage = rcu_dereference(task->bpf_storage);
76 if (!local_storage)53 if (!local_storage)
77 goto out;54 goto out;
78 55
79 bpf_task_storage_lock();
80 bpf_local_storage_destroy(local_storage);56 bpf_local_storage_destroy(local_storage);
81 bpf_task_storage_unlock();
82out:57out:
83 rcu_read_unlock_migrate();58 rcu_read_unlock();
84}59}
85 60
86static void *bpf_pid_task_storage_lookup_elem(struct bpf_map *map, void *key)61static void *bpf_pid_task_storage_lookup_elem(struct bpf_map *map, void *key)
@@ -106,9 +81,7 @@ static void *bpf_pid_task_storage_lookup_elem(struct bpf_map *map, void *key) · 19 unmodified lines
106 goto out;81 goto out;
107 }82 }
108 83
109 bpf_task_storage_lock();
110 sdata = task_storage_lookup(task, map, true);84 sdata = task_storage_lookup(task, map, true);
111 bpf_task_storage_unlock();
112 put_pid(pid);85 put_pid(pid);
113 return sdata ? sdata->data : NULL;86 return sdata ? sdata->data : NULL;
114out:87out:
@@ -143,11 +116,9 @@ static long bpf_pid_task_storage_update_elem(struct bpf_map *map, void *key, · 28 unmodified lines
143 goto out;116 goto out;
144 }117 }
145 118
146 bpf_task_storage_lock();
147 sdata = bpf_local_storage_update(119 sdata = bpf_local_storage_update(
148 task, (struct bpf_local_storage_map *)map, value, map_flags,120 task, (struct bpf_local_storage_map *)map, value, map_flags,
149 true, GFP_ATOMIC);121 true, GFP_ATOMIC);
150 bpf_task_storage_unlock();
151 122
152 err = PTR_ERR_OR_ZERO(sdata);123 err = PTR_ERR_OR_ZERO(sdata);
153out:124out:
@@ -155,8 +126,7 @@ out: · 1 unmodified line
155 return err;126 return err;
156}127}
157 128
158static int task_storage_delete(struct task_struct *task, struct bpf_map *map,129static int task_storage_delete(struct task_struct *task, struct bpf_map *map)
159 bool nobusy)
160{130{
161 struct bpf_local_storage_data *sdata;131 struct bpf_local_storage_data *sdata;
162 132
@@ -164,12 +134,7 @@ static int task_storage_delete(struct task_struct *task, struct bpf_map *map, · 1 unmodified line
164 if (!sdata)134 if (!sdata)
165 return -ENOENT;135 return -ENOENT;
166 136
167 if (!nobusy)137 return bpf_selem_unlink(SELEM(sdata));
168 return -EBUSY;
169
170 bpf_selem_unlink(SELEM(sdata), false);
171
172 return 0;
173}138}
174 139
175static long bpf_pid_task_storage_delete_elem(struct bpf_map *map, void *key)140static long bpf_pid_task_storage_delete_elem(struct bpf_map *map, void *key)
@@ -194,111 +159,50 @@ static long bpf_pid_task_storage_delete_elem(struct bpf_map *map, void *key) · 18 unmodified lines
194 goto out;159 goto out;
195 }160 }
196 161
197 bpf_task_storage_lock();162 err = task_storage_delete(task, map);
198 err = task_storage_delete(task, map, true);
199 bpf_task_storage_unlock();
200out:163out:
201 put_pid(pid);164 put_pid(pid);
202 return err;165 return err;
203}166}
204 167
205/* Called by bpf_task_storage_get*() helpers */168/* *gfp_flags* is a hidden argument provided by the verifier */
206static void *__bpf_task_storage_get(struct bpf_map *map,169BPF_CALL_5(bpf_task_storage_get, struct bpf_map *, map, struct task_struct *,
207 struct task_struct *task, void *value,170 task, void *, value, u64, flags, gfp_t, gfp_flags)
208 u64 flags, gfp_t gfp_flags, bool nobusy)
209{171{
210 struct bpf_local_storage_data *sdata;172 struct bpf_local_storage_data *sdata;
211 173
212 sdata = task_storage_lookup(task, map, nobusy);174 WARN_ON_ONCE(!bpf_rcu_lock_held());
175 if (flags & ~BPF_LOCAL_STORAGE_GET_F_CREATE || !task)
176 return (unsigned long)NULL;
177
178 sdata = task_storage_lookup(task, map, true);
213 if (sdata)179 if (sdata)
214 return sdata->data;180 return (unsigned long)sdata->data;
215 181
216 /* only allocate new storage, when the task is refcounted */182 /* only allocate new storage, when the task is refcounted */
217 if (refcount_read(&task->usage) &&183 if (refcount_read(&task->usage) &&
218 (flags & BPF_LOCAL_STORAGE_GET_F_CREATE) && nobusy) {184 (flags & BPF_LOCAL_STORAGE_GET_F_CREATE)) {
219 sdata = bpf_local_storage_update(185 sdata = bpf_local_storage_update(
220 task, (struct bpf_local_storage_map *)map, value,186 task, (struct bpf_local_storage_map *)map, value,
221 BPF_NOEXIST, false, gfp_flags);187 BPF_NOEXIST, false, gfp_flags);
222 return IS_ERR(sdata) ? NULL : sdata->data;188 return IS_ERR(sdata) ? (unsigned long)NULL : (unsigned long)sdata->data;
223 }189 }
224 190
225 return NULL;191 return (unsigned long)NULL;
226}
227
228/* *gfp_flags* is a hidden argument provided by the verifier */
229BPF_CALL_5(bpf_task_storage_get_recur, struct bpf_map *, map, struct task_struct *,
230 task, void *, value, u64, flags, gfp_t, gfp_flags)
231{
232 bool nobusy;
233 void *data;
234
235 WARN_ON_ONCE(!bpf_rcu_lock_held());
236 if (flags & ~BPF_LOCAL_STORAGE_GET_F_CREATE || !task)
237 return (unsigned long)NULL;
238
239 nobusy = bpf_task_storage_trylock();
240 data = __bpf_task_storage_get(map, task, value, flags,
241 gfp_flags, nobusy);
242 if (nobusy)
243 bpf_task_storage_unlock();
244 return (unsigned long)data;
245}
246
247/* *gfp_flags* is a hidden argument provided by the verifier */
248BPF_CALL_5(bpf_task_storage_get, struct bpf_map *, map, struct task_struct *,
249 task, void *, value, u64, flags, gfp_t, gfp_flags)
250{
251 void *data;
252
253 WARN_ON_ONCE(!bpf_rcu_lock_held());
254 if (flags & ~BPF_LOCAL_STORAGE_GET_F_CREATE || !task)
255 return (unsigned long)NULL;
256
257 bpf_task_storage_lock();
258 data = __bpf_task_storage_get(map, task, value, flags,
259 gfp_flags, true);
260 bpf_task_storage_unlock();
261 return (unsigned long)data;
262}
263
264BPF_CALL_2(bpf_task_storage_delete_recur, struct bpf_map *, map, struct task_struct *,
265 task)
266{
267 bool nobusy;
268 int ret;
269
270 WARN_ON_ONCE(!bpf_rcu_lock_held());
271 if (!task)
272 return -EINVAL;
273
274 nobusy = bpf_task_storage_trylock();
275 /* This helper must only be called from places where the lifetime of the task
276 * is guaranteed. Either by being refcounted or by being protected
277 * by an RCU read-side critical section.
278 */
279 ret = task_storage_delete(task, map, nobusy);
280 if (nobusy)
281 bpf_task_storage_unlock();
282 return ret;
283}192}
284 193
285BPF_CALL_2(bpf_task_storage_delete, struct bpf_map *, map, struct task_struct *,194BPF_CALL_2(bpf_task_storage_delete, struct bpf_map *, map, struct task_struct *,
286 task)195 task)
287{196{
288 int ret;
289
290 WARN_ON_ONCE(!bpf_rcu_lock_held());197 WARN_ON_ONCE(!bpf_rcu_lock_held());
291 if (!task)198 if (!task)
292 return -EINVAL;199 return -EINVAL;
293 200
294 bpf_task_storage_lock();
295 /* This helper must only be called from places where the lifetime of the task201 /* This helper must only be called from places where the lifetime of the task
296 * is guaranteed. Either by being refcounted or by being protected202 * is guaranteed. Either by being refcounted or by being protected
297 * by an RCU read-side critical section.203 * by an RCU read-side critical section.
298 */204 */
299 ret = task_storage_delete(task, map, true);205 return task_storage_delete(task, map);
300 bpf_task_storage_unlock();
301 return ret;
302}206}
303 207
304static int notsupp_get_next_key(struct bpf_map *map, void *key, void *next_key)208static int notsupp_get_next_key(struct bpf_map *map, void *key, void *next_key)
@@ -313,7 +217,7 @@ static struct bpf_map *task_storage_map_alloc(union bpf_attr *attr) · 8 unmodified lines
313 217
314static void task_storage_map_free(struct bpf_map *map)218static void task_storage_map_free(struct bpf_map *map)
315{219{
316 bpf_local_storage_map_free(map, &task_cache, &bpf_task_storage_busy);220 bpf_local_storage_map_free(map, &task_cache);
317}221}
318 222
319BTF_ID_LIST_GLOBAL_SINGLE(bpf_local_storage_map_btf_id, struct, bpf_local_storage_map)223BTF_ID_LIST_GLOBAL_SINGLE(bpf_local_storage_map_btf_id, struct, bpf_local_storage_map)
@@ -332,17 +236,6 @@ const struct bpf_map_ops task_storage_map_ops = { · 12 unmodified lines
332 .map_owner_storage_ptr = task_storage_ptr,236 .map_owner_storage_ptr = task_storage_ptr,
333};237};
334 238
335const struct bpf_func_proto bpf_task_storage_get_recur_proto = {
336 .func = bpf_task_storage_get_recur,
337 .gpl_only = false,
338 .ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
339 .arg1_type = ARG_CONST_MAP_PTR,
340 .arg2_type = ARG_PTR_TO_BTF_ID_OR_NULL,
341 .arg2_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
342 .arg3_type = ARG_PTR_TO_MAP_VALUE_OR_NULL,
343 .arg4_type = ARG_ANYTHING,
344};
345
346const struct bpf_func_proto bpf_task_storage_get_proto = {239const struct bpf_func_proto bpf_task_storage_get_proto = {
347 .func = bpf_task_storage_get,240 .func = bpf_task_storage_get,
348 .gpl_only = false,241 .gpl_only = false,
@@ -354,15 +247,6 @@ const struct bpf_func_proto bpf_task_storage_get_proto = { · 5 unmodified lines
354 .arg4_type = ARG_ANYTHING,247 .arg4_type = ARG_ANYTHING,
355};248};
356 249
357const struct bpf_func_proto bpf_task_storage_delete_recur_proto = {
358 .func = bpf_task_storage_delete_recur,
359 .gpl_only = false,
360 .ret_type = RET_INTEGER,
361 .arg1_type = ARG_CONST_MAP_PTR,
362 .arg2_type = ARG_PTR_TO_BTF_ID_OR_NULL,
363 .arg2_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
364};
365
366const struct bpf_func_proto bpf_task_storage_delete_proto = {250const struct bpf_func_proto bpf_task_storage_delete_proto = {
367 .func = bpf_task_storage_delete,251 .func = bpf_task_storage_delete,
368 .gpl_only = false,252 .gpl_only = false,
kernel/bpf/helpers.c+0−4
@@ -2167,12 +2167,8 @@ bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 2166 unmodified lines
2167 return &bpf_get_cgroup_classid_curr_proto;2167 return &bpf_get_cgroup_classid_curr_proto;
2168#endif2168#endif
2169 case BPF_FUNC_task_storage_get:2169 case BPF_FUNC_task_storage_get:
2170 if (bpf_prog_check_recur(prog))
2171 return &bpf_task_storage_get_recur_proto;
2172 return &bpf_task_storage_get_proto;2170 return &bpf_task_storage_get_proto;
2173 case BPF_FUNC_task_storage_delete:2171 case BPF_FUNC_task_storage_delete:
2174 if (bpf_prog_check_recur(prog))
2175 return &bpf_task_storage_delete_recur_proto;
2176 return &bpf_task_storage_delete_proto;2172 return &bpf_task_storage_delete_proto;
2177 default:2173 default:
2178 break;2174 break;
net/core/bpf_sk_storage.c+14−6
@@ -40,29 +40,30 @@ static int bpf_sk_storage_del(struct sock *sk, struct bpf_map *map) · 39 unmodified lines
40 if (!sdata)40 if (!sdata)
41 return -ENOENT;41 return -ENOENT;
42 42
43 bpf_selem_unlink(SELEM(sdata), false);43 return bpf_selem_unlink(SELEM(sdata));
44
45 return 0;
46}44}
47 45
48/* Called by __sk_destruct() & bpf_sk_storage_clone() */46/* Called by __sk_destruct() & bpf_sk_storage_clone() */
49void bpf_sk_storage_free(struct sock *sk)47void bpf_sk_storage_free(struct sock *sk)
50{48{
51 struct bpf_local_storage *sk_storage;49 struct bpf_local_storage *sk_storage;
50 u32 uncharge;
52 51
53 rcu_read_lock_dont_migrate();52 rcu_read_lock_dont_migrate();
54 sk_storage = rcu_dereference(sk->sk_bpf_storage);53 sk_storage = rcu_dereference(sk->sk_bpf_storage);
55 if (!sk_storage)54 if (!sk_storage)
56 goto out;55 goto out;
57 56
58 bpf_local_storage_destroy(sk_storage);57 uncharge = bpf_local_storage_destroy(sk_storage);
58 if (uncharge)
59 atomic_sub(uncharge, &sk->sk_omem_alloc);
59out:60out:
60 rcu_read_unlock_migrate();61 rcu_read_unlock_migrate();
61}62}
62 63
63static void bpf_sk_storage_map_free(struct bpf_map *map)64static void bpf_sk_storage_map_free(struct bpf_map *map)
64{65{
65 bpf_local_storage_map_free(map, &sk_cache, NULL);66 bpf_local_storage_map_free(map, &sk_cache);
66}67}
67 68
68static struct bpf_map *bpf_sk_storage_map_alloc(union bpf_attr *attr)69static struct bpf_map *bpf_sk_storage_map_alloc(union bpf_attr *attr)
@@ -191,7 +192,14 @@ int bpf_sk_storage_clone(const struct sock *sk, struct sock *newsk) · 122 unmodified lines
191 }192 }
192 193
193 if (new_sk_storage) {194 if (new_sk_storage) {
194 bpf_selem_link_map(smap, copy_selem);195 ret = bpf_selem_link_map(smap, new_sk_storage, copy_selem);
196 if (ret) {
197 bpf_selem_free(copy_selem, true);
198 atomic_sub(smap->elem_size,
199 &newsk->sk_omem_alloc);
200 bpf_map_put(map);
201 goto out;
202 }
195 bpf_selem_link_storage_nolock(new_sk_storage, copy_selem);203 bpf_selem_link_storage_nolock(new_sk_storage, copy_selem);
196 } else {204 } else {
197 ret = bpf_local_storage_alloc(newsk, smap, copy_selem, GFP_ATOMIC);205 ret = bpf_local_storage_alloc(newsk, smap, copy_selem, GFP_ATOMIC);
tools/testing/selftests/bpf/map_tests/task_storage_map.c+0−128
@@ -1,128 +0,0 @@
1// SPDX-License-Identifier: GPL-2.0
2/* Copyright (C) 2022. Huawei Technologies Co., Ltd */
3#define _GNU_SOURCE
4#include <sched.h>
5#include <unistd.h>
6#include <stdlib.h>
7#include <stdbool.h>
8#include <errno.h>
9#include <string.h>
10#include <pthread.h>
11
12#include <bpf/bpf.h>
13#include <bpf/libbpf.h>
14
15#include "bpf_util.h"
16#include "test_maps.h"
17#include "task_local_storage_helpers.h"
18#include "read_bpf_task_storage_busy.skel.h"
19
20struct lookup_ctx {
21 bool start;
22 bool stop;
23 int pid_fd;
24 int map_fd;
25 int loop;
26};
27
28static void *lookup_fn(void *arg)
29{
30 struct lookup_ctx *ctx = arg;
31 long value;
32 int i = 0;
33
34 while (!ctx->start)
35 usleep(1);
36
37 while (!ctx->stop && i++ < ctx->loop)
38 bpf_map_lookup_elem(ctx->map_fd, &ctx->pid_fd, &value);
39 return NULL;
40}
41
42static void abort_lookup(struct lookup_ctx *ctx, pthread_t *tids, unsigned int nr)
43{
44 unsigned int i;
45
46 ctx->stop = true;
47 ctx->start = true;
48 for (i = 0; i < nr; i++)
49 pthread_join(tids[i], NULL);
50}
51
52void test_task_storage_map_stress_lookup(void)
53{
54#define MAX_NR_THREAD 4096
55 unsigned int i, nr = 256, loop = 8192, cpu = 0;
56 struct read_bpf_task_storage_busy *skel;
57 pthread_t tids[MAX_NR_THREAD];
58 struct lookup_ctx ctx;
59 cpu_set_t old, new;
60 const char *cfg;
61 int err;
62
63 cfg = getenv("TASK_STORAGE_MAP_NR_THREAD");
64 if (cfg) {
65 nr = atoi(cfg);
66 if (nr > MAX_NR_THREAD)
67 nr = MAX_NR_THREAD;
68 }
69 cfg = getenv("TASK_STORAGE_MAP_NR_LOOP");
70 if (cfg)
71 loop = atoi(cfg);
72 cfg = getenv("TASK_STORAGE_MAP_PIN_CPU");
73 if (cfg)
74 cpu = atoi(cfg);
75
76 skel = read_bpf_task_storage_busy__open_and_load();
77 err = libbpf_get_error(skel);
78 CHECK(err, "open_and_load", "error %d\n", err);
79
80 /* Only for a fully preemptible kernel */
81 if (!skel->kconfig->CONFIG_PREEMPTION) {
82 printf("%s SKIP (no CONFIG_PREEMPTION)\n", __func__);
83 read_bpf_task_storage_busy__destroy(skel);
84 skips++;
85 return;
86 }
87
88 /* Save the old affinity setting */
89 sched_getaffinity(getpid(), sizeof(old), &old);
90
91 /* Pinned on a specific CPU */
92 CPU_ZERO(&new);
93 CPU_SET(cpu, &new);
94 sched_setaffinity(getpid(), sizeof(new), &new);
95
96 ctx.start = false;
97 ctx.stop = false;
98 ctx.pid_fd = sys_pidfd_open(getpid(), 0);
99 ctx.map_fd = bpf_map__fd(skel->maps.task);
100 ctx.loop = loop;
101 for (i = 0; i < nr; i++) {
102 err = pthread_create(&tids[i], NULL, lookup_fn, &ctx);
103 if (err) {
104 abort_lookup(&ctx, tids, i);
105 CHECK(err, "pthread_create", "error %d\n", err);
106 goto out;
107 }
108 }
109
110 ctx.start = true;
111 for (i = 0; i < nr; i++)
112 pthread_join(tids[i], NULL);
113
114 skel->bss->pid = getpid();
115 err = read_bpf_task_storage_busy__attach(skel);
116 CHECK(err, "attach", "error %d\n", err);
117
118 /* Trigger program */
119 sys_gettid();
120 skel->bss->pid = 0;
121
122 CHECK(skel->bss->busy != 0, "bad bpf_task_storage_busy", "got %d\n", skel->bss->busy);
123out:
124 read_bpf_task_storage_busy__destroy(skel);
125 /* Restore affinity setting */
126 sched_setaffinity(getpid(), sizeof(old), &old);
127 printf("%s:PASS\n", __func__);
128}
tools/testing/selftests/bpf/prog_tests/btf_dump.c+2−2
@@ -875,8 +875,8 @@ static void test_btf_dump_var_data(struct btf *btf, struct btf_dump *d, · 874 unmodified lines
875 TEST_BTF_DUMP_VAR(btf, d, NULL, str, "cpu_number", int, BTF_F_COMPACT,875 TEST_BTF_DUMP_VAR(btf, d, NULL, str, "cpu_number", int, BTF_F_COMPACT,
876 "int cpu_number = (int)100", 100);876 "int cpu_number = (int)100", 100);
877#endif877#endif
878 TEST_BTF_DUMP_VAR(btf, d, NULL, str, "bpf_cgrp_storage_busy", int, BTF_F_COMPACT,878 TEST_BTF_DUMP_VAR(btf, d, NULL, str, "bpf_bprintf_nest_level", int, BTF_F_COMPACT,
879 "static int bpf_cgrp_storage_busy = (int)2", 2);879 "static int bpf_bprintf_nest_level = (int)2", 2);
880}880}
881 881
882struct btf_dump_string_ctx {882struct btf_dump_string_ctx {
tools/testing/selftests/bpf/prog_tests/task_local_storage.c+5−5
@@ -112,24 +112,24 @@ static void test_recursion(void) · 111 unmodified lines
112 task_ls_recursion__detach(skel);112 task_ls_recursion__detach(skel);
113 113
114 /* Refer to the comment in BPF_PROG(on_update) for114 /* Refer to the comment in BPF_PROG(on_update) for
115 * the explanation on the value 201 and 100.115 * the explanation on the value 200 and 1.
116 */116 */
117 map_fd = bpf_map__fd(skel->maps.map_a);117 map_fd = bpf_map__fd(skel->maps.map_a);
118 err = bpf_map_lookup_elem(map_fd, &task_fd, &value);118 err = bpf_map_lookup_elem(map_fd, &task_fd, &value);
119 ASSERT_OK(err, "lookup map_a");119 ASSERT_OK(err, "lookup map_a");
120 ASSERT_EQ(value, 201, "map_a value");120 ASSERT_EQ(value, 200, "map_a value");
121 ASSERT_EQ(skel->bss->nr_del_errs, 1, "bpf_task_storage_delete busy");121 ASSERT_EQ(skel->bss->nr_del_errs, 0, "bpf_task_storage_delete busy");
122 122
123 map_fd = bpf_map__fd(skel->maps.map_b);123 map_fd = bpf_map__fd(skel->maps.map_b);
124 err = bpf_map_lookup_elem(map_fd, &task_fd, &value);124 err = bpf_map_lookup_elem(map_fd, &task_fd, &value);
125 ASSERT_OK(err, "lookup map_b");125 ASSERT_OK(err, "lookup map_b");
126 ASSERT_EQ(value, 100, "map_b value");126 ASSERT_EQ(value, 1, "map_b value");
127 127
128 prog_fd = bpf_program__fd(skel->progs.on_update);128 prog_fd = bpf_program__fd(skel->progs.on_update);
129 memset(&info, 0, sizeof(info));129 memset(&info, 0, sizeof(info));
130 err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len);130 err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len);
131 ASSERT_OK(err, "get prog info");131 ASSERT_OK(err, "get prog info");
132 ASSERT_EQ(info.recursion_misses, 0, "on_update prog recursion");132 ASSERT_EQ(info.recursion_misses, 2, "on_update prog recursion");
133 133
134 prog_fd = bpf_program__fd(skel->progs.on_enter);134 prog_fd = bpf_program__fd(skel->progs.on_enter);
135 memset(&info, 0, sizeof(info));135 memset(&info, 0, sizeof(info));
tools/testing/selftests/bpf/progs/local_storage.c+2−17
@@ -62,7 +62,6 @@ SEC("lsm/inode_unlink") · 61 unmodified lines
62int BPF_PROG(unlink_hook, struct inode *dir, struct dentry *victim)62int BPF_PROG(unlink_hook, struct inode *dir, struct dentry *victim)
63{63{
64 __u32 pid = bpf_get_current_pid_tgid() >> 32;64 __u32 pid = bpf_get_current_pid_tgid() >> 32;
65 struct bpf_local_storage *local_storage;
66 struct local_storage *storage;65 struct local_storage *storage;
67 struct task_struct *task;66 struct task_struct *task;
68 bool is_self_unlink;67 bool is_self_unlink;
@@ -88,15 +87,10 @@ int BPF_PROG(unlink_hook, struct inode *dir, struct dentry *victim) · 19 unmodified lines
88 if (!storage || storage->value)87 if (!storage || storage->value)
89 return 0;88 return 0;
90 89
91 if (bpf_task_storage_delete(&task_storage_map, task))90 if (bpf_task_storage_delete(&task_storage_map2, task))
92 return 0;91 return 0;
93 92
94 /* Ensure that the task_storage_map is disconnected from the storage.93 if (bpf_task_storage_delete(&task_storage_map, task))
95 * The storage memory should not be freed back to the
96 * bpf_mem_alloc.
97 */
98 local_storage = task->bpf_storage;
99 if (!local_storage || local_storage->smap)
100 return 0;94 return 0;
101 95
102 task_storage_result = 0;96 task_storage_result = 0;
@@ -164,18 +158,9 @@ int BPF_PROG(socket_bind, struct socket *sock, struct sockaddr *address, · 61 unmodified lines
164 if (bpf_sk_storage_delete(&sk_storage_map2, sk))158 if (bpf_sk_storage_delete(&sk_storage_map2, sk))
165 return 0;159 return 0;
166 160
167 storage = bpf_sk_storage_get(&sk_storage_map2, sk, 0,
168 BPF_LOCAL_STORAGE_GET_F_CREATE);
169 if (!storage)
170 return 0;
171
172 if (bpf_sk_storage_delete(&sk_storage_map, sk))161 if (bpf_sk_storage_delete(&sk_storage_map, sk))
173 return 0;162 return 0;
174 163
175 /* Ensure that the sk_storage_map is disconnected from the storage. */
176 if (!sk->sk_bpf_storage || sk->sk_bpf_storage->smap)
177 return 0;
178
179 sk_storage_result = 0;164 sk_storage_result = 0;
180 return 0;165 return 0;
181}166}
tools/testing/selftests/bpf/progs/read_bpf_task_storage_busy.c+0−38
@@ -1,38 +0,0 @@
1// SPDX-License-Identifier: GPL-2.0
2/* Copyright (C) 2022. Huawei Technologies Co., Ltd */
3#include "vmlinux.h"
4#include <bpf/bpf_helpers.h>
5#include <bpf/bpf_tracing.h>
6
7extern bool CONFIG_PREEMPTION __kconfig __weak;
8extern const int bpf_task_storage_busy __ksym;
9
10char _license[] SEC("license") = "GPL";
11
12int pid = 0;
13int busy = 0;
14
15struct {
16 __uint(type, BPF_MAP_TYPE_TASK_STORAGE);
17 __uint(map_flags, BPF_F_NO_PREALLOC);
18 __type(key, int);
19 __type(value, long);
20} task SEC(".maps");
21
22SEC("raw_tp/sys_enter")
23int BPF_PROG(read_bpf_task_storage_busy)
24{
25 int *value;
26
27 if (!CONFIG_PREEMPTION)
28 return 0;
29
30 if (bpf_get_current_pid_tgid() >> 32 != pid)
31 return 0;
32
33 value = bpf_this_cpu_ptr(&bpf_task_storage_busy);
34 if (value)
35 busy = *value;
36
37 return 0;
38}
tools/testing/selftests/bpf/progs/sk_storage_omem_uncharge.c+3−9
@@ -6,7 +6,6 @@ · 5 unmodified lines
6#include <bpf/bpf_tracing.h>6#include <bpf/bpf_tracing.h>
7#include <bpf/bpf_core_read.h>7#include <bpf/bpf_core_read.h>
8 8
9void *local_storage_ptr = NULL;
10void *sk_ptr = NULL;9void *sk_ptr = NULL;
11int cookie_found = 0;10int cookie_found = 0;
12__u64 cookie = 0;11__u64 cookie = 0;
@@ -19,21 +18,17 @@ struct { · 6 unmodified lines
19 __type(value, int);18 __type(value, int);
20} sk_storage SEC(".maps");19} sk_storage SEC(".maps");
21 20
22SEC("fexit/bpf_local_storage_destroy")21SEC("fexit/bpf_sk_storage_free")
23int BPF_PROG(bpf_local_storage_destroy, struct bpf_local_storage *local_storage)22int BPF_PROG(bpf_sk_storage_free, struct sock *sk)
24{23{
25 struct sock *sk;24 if (sk_ptr != sk)
26
27 if (local_storage_ptr != local_storage)
28 return 0;25 return 0;
29 26
30 sk = bpf_core_cast(sk_ptr, struct sock);
31 if (sk->sk_cookie.counter != cookie)27 if (sk->sk_cookie.counter != cookie)
32 return 0;28 return 0;
33 29
34 cookie_found++;30 cookie_found++;
35 omem = sk->sk_omem_alloc.counter;31 omem = sk->sk_omem_alloc.counter;
36 local_storage_ptr = NULL;
37 32
38 return 0;33 return 0;
39}34}
@@ -50,7 +45,6 @@ int BPF_PROG(inet6_sock_destruct, struct sock *sk) · 10 unmodified lines
50 if (value && *value == 0xdeadbeef) {45 if (value && *value == 0xdeadbeef) {
51 cookie_found++;46 cookie_found++;
52 sk_ptr = sk;47 sk_ptr = sk;
53 local_storage_ptr = sk->sk_bpf_storage;
54 }48 }
55 49
56 return 0;50 return 0;
tools/testing/selftests/bpf/progs/task_ls_recursion.c+2−12
@@ -36,14 +36,9 @@ int BPF_PROG(on_update) · 35 unmodified lines
36 if (!test_pid || task->pid != test_pid)36 if (!test_pid || task->pid != test_pid)
37 return 0;37 return 0;
38 38
39 /* This will succeed as there is no real deadlock */
39 ptr = bpf_task_storage_get(&map_a, task, 0,40 ptr = bpf_task_storage_get(&map_a, task, 0,
40 BPF_LOCAL_STORAGE_GET_F_CREATE);41 BPF_LOCAL_STORAGE_GET_F_CREATE);
41 /* ptr will not be NULL when it is called from
42 * the bpf_task_storage_get(&map_b,...F_CREATE) in
43 * the BPF_PROG(on_enter) below. It is because
44 * the value can be found in map_a and the kernel
45 * does not need to acquire any spin_lock.
46 */
47 if (ptr) {42 if (ptr) {
48 int err;43 int err;
49 44
@@ -53,12 +48,7 @@ int BPF_PROG(on_update) · 3 unmodified lines
53 nr_del_errs++;48 nr_del_errs++;
54 }49 }
55 50
56 /* This will still fail because map_b is empty and51 /* This will succeed as there is no real deadlock */
57 * this BPF_PROG(on_update) has failed to acquire
58 * the percpu busy lock => meaning potential
59 * deadlock is detected and it will fail to create
60 * new storage.
61 */
62 ptr = bpf_task_storage_get(&map_b, task, 0,52 ptr = bpf_task_storage_get(&map_b, task, 0,
63 BPF_LOCAL_STORAGE_GET_F_CREATE);53 BPF_LOCAL_STORAGE_GET_F_CREATE);
64 if (ptr)54 if (ptr)
tools/testing/selftests/bpf/progs/task_storage_nodeadlock.c+2−5
@@ -1,15 +1,12 @@
1// SPDX-License-Identifier: GPL-2.01// SPDX-License-Identifier: GPL-2.0
2 2
3#include "vmlinux.h"3#include "vmlinux.h"
4#include <errno.h>
4#include <bpf/bpf_helpers.h>5#include <bpf/bpf_helpers.h>
5#include <bpf/bpf_tracing.h>6#include <bpf/bpf_tracing.h>
6 7
7char _license[] SEC("license") = "GPL";8char _license[] SEC("license") = "GPL";
8 9
9#ifndef EBUSY
10#define EBUSY 16
11#endif
12
13extern bool CONFIG_PREEMPTION __kconfig __weak;10extern bool CONFIG_PREEMPTION __kconfig __weak;
14int nr_get_errs = 0;11int nr_get_errs = 0;
15int nr_del_errs = 0;12int nr_del_errs = 0;
@@ -40,7 +37,7 @@ int BPF_PROG(socket_post_create, struct socket *sock, int family, int type, · 24 unmodified lines
40 37
41 ret = bpf_task_storage_delete(&task_storage,38 ret = bpf_task_storage_delete(&task_storage,
42 bpf_get_current_task_btf());39 bpf_get_current_task_btf());
43 if (ret == -EBUSY)40 if (ret == -EDEADLK || ret == -ETIMEDOUT)
44 __sync_fetch_and_add(&nr_del_errs, 1);41 __sync_fetch_and_add(&nr_del_errs, 1);
45 42
46 return 0;43 return 0;