History

Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next

Pull networking updates from David Miller: 1) Allow setting bluetooth L2CAP modes via socket option, from Luiz Augusto von Dentz. 2) Add GSO partial support to igc, from Sasha Neftin. 3) Several cleanups and improvements to r8169 from Heiner Kallweit. 4) Add IF_OPER_TESTING link state and use it when ethtool triggers a device self-test. From Andrew Lunn. 5) Start moving away from custom driver versions, use the globally defined kernel version instead, from Leon Romanovsky. 6) Support GRO vis gro_cells in DSA layer, from Alexander Lobakin. 7) Allow hard IRQ deferral during NAPI, from Eric Dumazet. 8) Add sriov and vf support to hinic, from Luo bin. 9) Support Media Redundancy Protocol (MRP) in the bridging code, from Horatiu Vultur. 10) Support netmap in the nft_nat code, from Pablo Neira Ayuso. 11) Allow UDPv6 encapsulation of ESP in the ipsec code, from Sabrina Dubroca. Also add ipv6 support for espintcp. 12) Lots of ReST conversions of the networking documentation, from Mauro Carvalho Chehab. 13) Support configuration of ethtool rxnfc flows in bcmgenet driver, from Doug Berger. 14) Allow to dump cgroup id and filter by it in inet_diag code, from Dmitry Yakunin. 15) Add infrastructure to export netlink attribute policies to userspace, from Johannes Berg. 16) Several optimizations to sch_fq scheduler, from Eric Dumazet. 17) Fallback to the default qdisc if qdisc init fails because otherwise a packet scheduler init failure will make a device inoperative. From Jesper Dangaard Brouer. 18) Several RISCV bpf jit optimizations, from Luke Nelson. 19) Correct the return type of the ->ndo_start_xmit() method in several drivers, it's netdev_tx_t but many drivers were using 'int'. From Yunjian Wang. 20) Add an ethtool interface for PHY master/slave config, from Oleksij Rempel. 21) Add BPF iterators, from Yonghang Song. 22) Add cable test infrastructure, including ethool interfaces, from Andrew Lunn. Marvell PHY driver is the first to support this facility. 23) Remove zero-length arrays all over, from Gustavo A. R. Silva. 24) Calculate and maintain an explicit frame size in XDP, from Jesper Dangaard Brouer. 25) Add CAP_BPF, from Alexei Starovoitov. 26) Support terse dumps in the packet scheduler, from Vlad Buslov. 27) Support XDP_TX bulking in dpaa2 driver, from Ioana Ciornei. 28) Add devm_register_netdev(), from Bartosz Golaszewski. 29) Minimize qdisc resets, from Cong Wang. 30) Get rid of kernel_getsockopt and kernel_setsockopt in order to eliminate set_fs/get_fs calls. From Christoph Hellwig. * git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next: (2517 commits) selftests: net: ip_defrag: ignore EPERM net_failover: fixed rollback in net_failover_open() Revert "tipc: Fix potential tipc_aead refcnt leak in tipc_crypto_rcv" Revert "tipc: Fix potential tipc_node refcnt leak in tipc_rcv" vmxnet3: allow rx flow hash ops only when rss is enabled hinic: add set_channels ethtool_ops support selftests/bpf: Add a default $(CXX) value tools/bpf: Don't use $(COMPILE.c) bpf, selftests: Use bpf_probe_read_kernel s390/bpf: Use bcr 0,%0 as tail call nop filler s390/bpf: Maintain 8-byte stack alignment selftests/bpf: Fix verifier test selftests/bpf: Fix sample_cnt shared between two threads bpf, selftests: Adapt cls_redirect to call csum_level helper bpf: Add csum_level helper for fixing up csum levels bpf: Fix up bpf_skb_adjust_room helper's skb csum setting sfc: add missing annotation for efx_ef10_try_update_nic_stats_vf() crypto/chtls: IPv6 support for inline TLS Crypto/chcr: Fixes a coccinile check error Crypto/chcr: Fixes compilations warnings ...

Linus Torvalds committed 6 years ago

Commit cb8e59c2 parents 2e63f6c 065fcfd2097 files · +151766 −46213Browse files

Showing the changes to kernel (45 of 2097 files). Show all

Unified · Split

45 files
  1. kernel/bpf/Makefile +3 −5
  2. kernel/bpf/arraymap.c +5 −5
  3. kernel/bpf/bpf_iter.c +539 −0
  4. kernel/bpf/bpf_lsm.c +1 −1
  5. kernel/bpf/bpf_struct_ops.c +1 −1
  6. kernel/bpf/btf.c +48 −1
  7. kernel/bpf/cgroup.c +90 −56
  8. kernel/bpf/core.c +8 −2
  9. kernel/bpf/cpumap.c +5 −20
  10. kernel/bpf/devmap.c +113 −20
  11. kernel/bpf/hashtab.c +2 −2
  12. kernel/bpf/helpers.c +124 −1
  13. kernel/bpf/inode.c +4 −1
  14. kernel/bpf/lpm_trie.c +1 −1
  15. kernel/bpf/map_in_map.c +1 −1
  16. kernel/bpf/map_iter.c +102 −0
  17. kernel/bpf/net_namespace.c +373 −0
  18. kernel/bpf/queue_stack_maps.c +2 −2
  19. kernel/bpf/reuseport_array.c +1 −1
  20. kernel/bpf/ringbuf.c +501 −0
  21. kernel/bpf/stackmap.c +1 −1
  22. kernel/bpf/syscall.c +503 −94
  23. kernel/bpf/task_iter.c +353 −0
  24. kernel/bpf/verifier.c +299 −88
  25. kernel/bpf/xskmap.c +0 −265
  26. kernel/cgroup/cgroup.c +0 −27
  27. kernel/events/callchain.c +1 −1
  28. kernel/events/core.c +2 −4
  29. kernel/kprobes.c +1 −1
  30. kernel/latencytop.c +2 −2
  31. kernel/module.c +3 −0
  32. kernel/pid_namespace.c +1 −1
  33. kernel/printk/printk.c +1 −1
  34. kernel/sched/core.c +4 −5
  35. kernel/sched/fair.c +1 −2
  36. kernel/sched/rt.c +4 −6
  37. kernel/sched/topology.c +1 −1
  38. kernel/seccomp.c +1 −1
  39. kernel/sysctl.c +1891 −1990
  40. kernel/time/timer.c +1 −2
  41. kernel/trace/bpf_trace.c +235 −6
  42. kernel/trace/trace.c +1 −1
  43. kernel/umh.c +1 −1
  44. kernel/utsname_sysctl.c +1 −1
  45. kernel/watchdog.c +6 −6
kernel/bpf/Makefile+3−5
@@ -2,9 +2,9 @@ · 1 unmodified line
2obj-y := core.o2obj-y := core.o
3CFLAGS_core.o += $(call cc-disable-warning, override-init)3CFLAGS_core.o += $(call cc-disable-warning, override-init)
4 4
5obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o5obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o map_iter.o task_iter.o
6obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o6obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o
7obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o7obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
8obj-$(CONFIG_BPF_SYSCALL) += disasm.o8obj-$(CONFIG_BPF_SYSCALL) += disasm.o
9obj-$(CONFIG_BPF_JIT) += trampoline.o9obj-$(CONFIG_BPF_JIT) += trampoline.o
10obj-$(CONFIG_BPF_SYSCALL) += btf.o10obj-$(CONFIG_BPF_SYSCALL) += btf.o
@@ -12,10 +12,8 @@ obj-$(CONFIG_BPF_JIT) += dispatcher.o · 1 unmodified line
12ifeq ($(CONFIG_NET),y)12ifeq ($(CONFIG_NET),y)
13obj-$(CONFIG_BPF_SYSCALL) += devmap.o13obj-$(CONFIG_BPF_SYSCALL) += devmap.o
14obj-$(CONFIG_BPF_SYSCALL) += cpumap.o14obj-$(CONFIG_BPF_SYSCALL) += cpumap.o
15ifeq ($(CONFIG_XDP_SOCKETS),y)
16obj-$(CONFIG_BPF_SYSCALL) += xskmap.o
17endif
18obj-$(CONFIG_BPF_SYSCALL) += offload.o15obj-$(CONFIG_BPF_SYSCALL) += offload.o
16obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o
19endif17endif
20ifeq ($(CONFIG_PERF_EVENTS),y)18ifeq ($(CONFIG_PERF_EVENTS),y)
21obj-$(CONFIG_BPF_SYSCALL) += stackmap.o19obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
kernel/bpf/arraymap.c+5−5
@@ -77,7 +77,7 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) · 76 unmodified lines
77 bool percpu = attr->map_type == BPF_MAP_TYPE_PERCPU_ARRAY;77 bool percpu = attr->map_type == BPF_MAP_TYPE_PERCPU_ARRAY;
78 int ret, numa_node = bpf_map_attr_numa_node(attr);78 int ret, numa_node = bpf_map_attr_numa_node(attr);
79 u32 elem_size, index_mask, max_entries;79 u32 elem_size, index_mask, max_entries;
80 bool unpriv = !capable(CAP_SYS_ADMIN);80 bool bypass_spec_v1 = bpf_bypass_spec_v1();
81 u64 cost, array_size, mask64;81 u64 cost, array_size, mask64;
82 struct bpf_map_memory mem;82 struct bpf_map_memory mem;
83 struct bpf_array *array;83 struct bpf_array *array;
@@ -95,7 +95,7 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) · 11 unmodified lines
95 mask64 -= 1;95 mask64 -= 1;
96 96
97 index_mask = mask64;97 index_mask = mask64;
98 if (unpriv) {98 if (!bypass_spec_v1) {
99 /* round up array size to nearest power of 2,99 /* round up array size to nearest power of 2,
100 * since cpu will speculate within index_mask limits100 * since cpu will speculate within index_mask limits
101 */101 */
@@ -149,7 +149,7 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) · 47 unmodified lines
149 return ERR_PTR(-ENOMEM);149 return ERR_PTR(-ENOMEM);
150 }150 }
151 array->index_mask = index_mask;151 array->index_mask = index_mask;
152 array->map.unpriv_array = unpriv;152 array->map.bypass_spec_v1 = bypass_spec_v1;
153 153
154 /* copy mandatory map attributes */154 /* copy mandatory map attributes */
155 bpf_map_init_from_attr(&array->map, attr);155 bpf_map_init_from_attr(&array->map, attr);
@@ -219,7 +219,7 @@ static u32 array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) · 63 unmodified lines
219 219
220 *insn++ = BPF_ALU64_IMM(BPF_ADD, map_ptr, offsetof(struct bpf_array, value));220 *insn++ = BPF_ALU64_IMM(BPF_ADD, map_ptr, offsetof(struct bpf_array, value));
221 *insn++ = BPF_LDX_MEM(BPF_W, ret, index, 0);221 *insn++ = BPF_LDX_MEM(BPF_W, ret, index, 0);
222 if (map->unpriv_array) {222 if (!map->bypass_spec_v1) {
223 *insn++ = BPF_JMP_IMM(BPF_JGE, ret, map->max_entries, 4);223 *insn++ = BPF_JMP_IMM(BPF_JGE, ret, map->max_entries, 4);
224 *insn++ = BPF_ALU32_IMM(BPF_AND, ret, array->index_mask);224 *insn++ = BPF_ALU32_IMM(BPF_AND, ret, array->index_mask);
225 } else {225 } else {
@@ -1058,7 +1058,7 @@ static u32 array_of_map_gen_lookup(struct bpf_map *map, · 832 unmodified lines
1058 1058
1059 *insn++ = BPF_ALU64_IMM(BPF_ADD, map_ptr, offsetof(struct bpf_array, value));1059 *insn++ = BPF_ALU64_IMM(BPF_ADD, map_ptr, offsetof(struct bpf_array, value));
1060 *insn++ = BPF_LDX_MEM(BPF_W, ret, index, 0);1060 *insn++ = BPF_LDX_MEM(BPF_W, ret, index, 0);
1061 if (map->unpriv_array) {1061 if (!map->bypass_spec_v1) {
1062 *insn++ = BPF_JMP_IMM(BPF_JGE, ret, map->max_entries, 6);1062 *insn++ = BPF_JMP_IMM(BPF_JGE, ret, map->max_entries, 6);
1063 *insn++ = BPF_ALU32_IMM(BPF_AND, ret, array->index_mask);1063 *insn++ = BPF_ALU32_IMM(BPF_AND, ret, array->index_mask);
1064 } else {1064 } else {
kernel/bpf/bpf_iter.c+539−0
@@ -0,0 +1,539 @@
1// SPDX-License-Identifier: GPL-2.0-only
2/* Copyright (c) 2020 Facebook */
3
4#include <linux/fs.h>
5#include <linux/anon_inodes.h>
6#include <linux/filter.h>
7#include <linux/bpf.h>
8
9struct bpf_iter_target_info {
10 struct list_head list;
11 const struct bpf_iter_reg *reg_info;
12 u32 btf_id; /* cached value */
13};
14
15struct bpf_iter_link {
16 struct bpf_link link;
17 struct bpf_iter_target_info *tinfo;
18};
19
20struct bpf_iter_priv_data {
21 struct bpf_iter_target_info *tinfo;
22 struct bpf_prog *prog;
23 u64 session_id;
24 u64 seq_num;
25 bool done_stop;
26 u8 target_private[] __aligned(8);
27};
28
29static struct list_head targets = LIST_HEAD_INIT(targets);
30static DEFINE_MUTEX(targets_mutex);
31
32/* protect bpf_iter_link changes */
33static DEFINE_MUTEX(link_mutex);
34
35/* incremented on every opened seq_file */
36static atomic64_t session_id;
37
38static int prepare_seq_file(struct file *file, struct bpf_iter_link *link);
39
40static void bpf_iter_inc_seq_num(struct seq_file *seq)
41{
42 struct bpf_iter_priv_data *iter_priv;
43
44 iter_priv = container_of(seq->private, struct bpf_iter_priv_data,
45 target_private);
46 iter_priv->seq_num++;
47}
48
49static void bpf_iter_dec_seq_num(struct seq_file *seq)
50{
51 struct bpf_iter_priv_data *iter_priv;
52
53 iter_priv = container_of(seq->private, struct bpf_iter_priv_data,
54 target_private);
55 iter_priv->seq_num--;
56}
57
58static void bpf_iter_done_stop(struct seq_file *seq)
59{
60 struct bpf_iter_priv_data *iter_priv;
61
62 iter_priv = container_of(seq->private, struct bpf_iter_priv_data,
63 target_private);
64 iter_priv->done_stop = true;
65}
66
67/* bpf_seq_read, a customized and simpler version for bpf iterator.
68 * no_llseek is assumed for this file.
69 * The following are differences from seq_read():
70 * . fixed buffer size (PAGE_SIZE)
71 * . assuming no_llseek
72 * . stop() may call bpf program, handling potential overflow there
73 */
74static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size,
75 loff_t *ppos)
76{
77 struct seq_file *seq = file->private_data;
78 size_t n, offs, copied = 0;
79 int err = 0;
80 void *p;
81
82 mutex_lock(&seq->lock);
83
84 if (!seq->buf) {
85 seq->size = PAGE_SIZE;
86 seq->buf = kmalloc(seq->size, GFP_KERNEL);
87 if (!seq->buf) {
88 err = -ENOMEM;
89 goto done;
90 }
91 }
92
93 if (seq->count) {
94 n = min(seq->count, size);
95 err = copy_to_user(buf, seq->buf + seq->from, n);
96 if (err) {
97 err = -EFAULT;
98 goto done;
99 }
100 seq->count -= n;
101 seq->from += n;
102 copied = n;
103 goto done;
104 }
105
106 seq->from = 0;
107 p = seq->op->start(seq, &seq->index);
108 if (!p)
109 goto stop;
110 if (IS_ERR(p)) {
111 err = PTR_ERR(p);
112 seq->op->stop(seq, p);
113 seq->count = 0;
114 goto done;
115 }
116
117 err = seq->op->show(seq, p);
118 if (err > 0) {
119 /* object is skipped, decrease seq_num, so next
120 * valid object can reuse the same seq_num.
121 */
122 bpf_iter_dec_seq_num(seq);
123 seq->count = 0;
124 } else if (err < 0 || seq_has_overflowed(seq)) {
125 if (!err)
126 err = -E2BIG;
127 seq->op->stop(seq, p);
128 seq->count = 0;
129 goto done;
130 }
131
132 while (1) {
133 loff_t pos = seq->index;
134
135 offs = seq->count;
136 p = seq->op->next(seq, p, &seq->index);
137 if (pos == seq->index) {
138 pr_info_ratelimited("buggy seq_file .next function %ps "
139 "did not updated position index\n",
140 seq->op->next);
141 seq->index++;
142 }
143
144 if (IS_ERR_OR_NULL(p))
145 break;
146
147 /* got a valid next object, increase seq_num */
148 bpf_iter_inc_seq_num(seq);
149
150 if (seq->count >= size)
151 break;
152
153 err = seq->op->show(seq, p);
154 if (err > 0) {
155 bpf_iter_dec_seq_num(seq);
156 seq->count = offs;
157 } else if (err < 0 || seq_has_overflowed(seq)) {
158 seq->count = offs;
159 if (offs == 0) {
160 if (!err)
161 err = -E2BIG;
162 seq->op->stop(seq, p);
163 goto done;
164 }
165 break;
166 }
167 }
168stop:
169 offs = seq->count;
170 /* bpf program called if !p */
171 seq->op->stop(seq, p);
172 if (!p) {
173 if (!seq_has_overflowed(seq)) {
174 bpf_iter_done_stop(seq);
175 } else {
176 seq->count = offs;
177 if (offs == 0) {
178 err = -E2BIG;
179 goto done;
180 }
181 }
182 }
183
184 n = min(seq->count, size);
185 err = copy_to_user(buf, seq->buf, n);
186 if (err) {
187 err = -EFAULT;
188 goto done;
189 }
190 copied = n;
191 seq->count -= n;
192 seq->from = n;
193done:
194 if (!copied)
195 copied = err;
196 else
197 *ppos += copied;
198 mutex_unlock(&seq->lock);
199 return copied;
200}
201
202static int iter_open(struct inode *inode, struct file *file)
203{
204 struct bpf_iter_link *link = inode->i_private;
205
206 return prepare_seq_file(file, link);
207}
208
209static int iter_release(struct inode *inode, struct file *file)
210{
211 struct bpf_iter_priv_data *iter_priv;
212 struct seq_file *seq;
213
214 seq = file->private_data;
215 if (!seq)
216 return 0;
217
218 iter_priv = container_of(seq->private, struct bpf_iter_priv_data,
219 target_private);
220
221 if (iter_priv->tinfo->reg_info->fini_seq_private)
222 iter_priv->tinfo->reg_info->fini_seq_private(seq->private);
223
224 bpf_prog_put(iter_priv->prog);
225 seq->private = iter_priv;
226
227 return seq_release_private(inode, file);
228}
229
230const struct file_operations bpf_iter_fops = {
231 .open = iter_open,
232 .llseek = no_llseek,
233 .read = bpf_seq_read,
234 .release = iter_release,
235};
236
237/* The argument reg_info will be cached in bpf_iter_target_info.
238 * The common practice is to declare target reg_info as
239 * a const static variable and passed as an argument to
240 * bpf_iter_reg_target().
241 */
242int bpf_iter_reg_target(const struct bpf_iter_reg *reg_info)
243{
244 struct bpf_iter_target_info *tinfo;
245
246 tinfo = kmalloc(sizeof(*tinfo), GFP_KERNEL);
247 if (!tinfo)
248 return -ENOMEM;
249
250 tinfo->reg_info = reg_info;
251 INIT_LIST_HEAD(&tinfo->list);
252
253 mutex_lock(&targets_mutex);
254 list_add(&tinfo->list, &targets);
255 mutex_unlock(&targets_mutex);
256
257 return 0;
258}
259
260void bpf_iter_unreg_target(const struct bpf_iter_reg *reg_info)
261{
262 struct bpf_iter_target_info *tinfo;
263 bool found = false;
264
265 mutex_lock(&targets_mutex);
266 list_for_each_entry(tinfo, &targets, list) {
267 if (reg_info == tinfo->reg_info) {
268 list_del(&tinfo->list);
269 kfree(tinfo);
270 found = true;
271 break;
272 }
273 }
274 mutex_unlock(&targets_mutex);
275
276 WARN_ON(found == false);
277}
278
279static void cache_btf_id(struct bpf_iter_target_info *tinfo,
280 struct bpf_prog *prog)
281{
282 tinfo->btf_id = prog->aux->attach_btf_id;
283}
284
285bool bpf_iter_prog_supported(struct bpf_prog *prog)
286{
287 const char *attach_fname = prog->aux->attach_func_name;
288 u32 prog_btf_id = prog->aux->attach_btf_id;
289 const char *prefix = BPF_ITER_FUNC_PREFIX;
290 struct bpf_iter_target_info *tinfo;
291 int prefix_len = strlen(prefix);
292 bool supported = false;
293
294 if (strncmp(attach_fname, prefix, prefix_len))
295 return false;
296
297 mutex_lock(&targets_mutex);
298 list_for_each_entry(tinfo, &targets, list) {
299 if (tinfo->btf_id && tinfo->btf_id == prog_btf_id) {
300 supported = true;
301 break;
302 }
303 if (!strcmp(attach_fname + prefix_len, tinfo->reg_info->target)) {
304 cache_btf_id(tinfo, prog);
305 supported = true;
306 break;
307 }
308 }
309 mutex_unlock(&targets_mutex);
310
311 if (supported) {
312 prog->aux->ctx_arg_info_size = tinfo->reg_info->ctx_arg_info_size;
313 prog->aux->ctx_arg_info = tinfo->reg_info->ctx_arg_info;
314 }
315
316 return supported;
317}
318
319static void bpf_iter_link_release(struct bpf_link *link)
320{
321}
322
323static void bpf_iter_link_dealloc(struct bpf_link *link)
324{
325 struct bpf_iter_link *iter_link =
326 container_of(link, struct bpf_iter_link, link);
327
328 kfree(iter_link);
329}
330
331static int bpf_iter_link_replace(struct bpf_link *link,
332 struct bpf_prog *new_prog,
333 struct bpf_prog *old_prog)
334{
335 int ret = 0;
336
337 mutex_lock(&link_mutex);
338 if (old_prog && link->prog != old_prog) {
339 ret = -EPERM;
340 goto out_unlock;
341 }
342
343 if (link->prog->type != new_prog->type ||
344 link->prog->expected_attach_type != new_prog->expected_attach_type ||
345 link->prog->aux->attach_btf_id != new_prog->aux->attach_btf_id) {
346 ret = -EINVAL;
347 goto out_unlock;
348 }
349
350 old_prog = xchg(&link->prog, new_prog);
351 bpf_prog_put(old_prog);
352
353out_unlock:
354 mutex_unlock(&link_mutex);
355 return ret;
356}
357
358static const struct bpf_link_ops bpf_iter_link_lops = {
359 .release = bpf_iter_link_release,
360 .dealloc = bpf_iter_link_dealloc,
361 .update_prog = bpf_iter_link_replace,
362};
363
364bool bpf_link_is_iter(struct bpf_link *link)
365{
366 return link->ops == &bpf_iter_link_lops;
367}
368
369int bpf_iter_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
370{
371 struct bpf_link_primer link_primer;
372 struct bpf_iter_target_info *tinfo;
373 struct bpf_iter_link *link;
374 bool existed = false;
375 u32 prog_btf_id;
376 int err;
377
378 if (attr->link_create.target_fd || attr->link_create.flags)
379 return -EINVAL;
380
381 prog_btf_id = prog->aux->attach_btf_id;
382 mutex_lock(&targets_mutex);
383 list_for_each_entry(tinfo, &targets, list) {
384 if (tinfo->btf_id == prog_btf_id) {
385 existed = true;
386 break;
387 }
388 }
389 mutex_unlock(&targets_mutex);
390 if (!existed)
391 return -ENOENT;
392
393 link = kzalloc(sizeof(*link), GFP_USER | __GFP_NOWARN);
394 if (!link)
395 return -ENOMEM;
396
397 bpf_link_init(&link->link, BPF_LINK_TYPE_ITER, &bpf_iter_link_lops, prog);
398 link->tinfo = tinfo;
399
400 err = bpf_link_prime(&link->link, &link_primer);
401 if (err) {
402 kfree(link);
403 return err;
404 }
405
406 return bpf_link_settle(&link_primer);
407}
408
409static void init_seq_meta(struct bpf_iter_priv_data *priv_data,
410 struct bpf_iter_target_info *tinfo,
411 struct bpf_prog *prog)
412{
413 priv_data->tinfo = tinfo;
414 priv_data->prog = prog;
415 priv_data->session_id = atomic64_inc_return(&session_id);
416 priv_data->seq_num = 0;
417 priv_data->done_stop = false;
418}
419
420static int prepare_seq_file(struct file *file, struct bpf_iter_link *link)
421{
422 struct bpf_iter_priv_data *priv_data;
423 struct bpf_iter_target_info *tinfo;
424 struct bpf_prog *prog;
425 u32 total_priv_dsize;
426 struct seq_file *seq;
427 int err = 0;
428
429 mutex_lock(&link_mutex);
430 prog = link->link.prog;
431 bpf_prog_inc(prog);
432 mutex_unlock(&link_mutex);
433
434 tinfo = link->tinfo;
435 total_priv_dsize = offsetof(struct bpf_iter_priv_data, target_private) +
436 tinfo->reg_info->seq_priv_size;
437 priv_data = __seq_open_private(file, tinfo->reg_info->seq_ops,
438 total_priv_dsize);
439 if (!priv_data) {
440 err = -ENOMEM;
441 goto release_prog;
442 }
443
444 if (tinfo->reg_info->init_seq_private) {
445 err = tinfo->reg_info->init_seq_private(priv_data->target_private);
446 if (err)
447 goto release_seq_file;
448 }
449
450 init_seq_meta(priv_data, tinfo, prog);
451 seq = file->private_data;
452 seq->private = priv_data->target_private;
453
454 return 0;
455
456release_seq_file:
457 seq_release_private(file->f_inode, file);
458 file->private_data = NULL;
459release_prog:
460 bpf_prog_put(prog);
461 return err;
462}
463
464int bpf_iter_new_fd(struct bpf_link *link)
465{
466 struct file *file;
467 unsigned int flags;
468 int err, fd;
469
470 if (link->ops != &bpf_iter_link_lops)
471 return -EINVAL;
472
473 flags = O_RDONLY | O_CLOEXEC;
474 fd = get_unused_fd_flags(flags);
475 if (fd < 0)
476 return fd;
477
478 file = anon_inode_getfile("bpf_iter", &bpf_iter_fops, NULL, flags);
479 if (IS_ERR(file)) {
480 err = PTR_ERR(file);
481 goto free_fd;
482 }
483
484 err = prepare_seq_file(file,
485 container_of(link, struct bpf_iter_link, link));
486 if (err)
487 goto free_file;
488
489 fd_install(fd, file);
490 return fd;
491
492free_file:
493 fput(file);
494free_fd:
495 put_unused_fd(fd);
496 return err;
497}
498
499struct bpf_prog *bpf_iter_get_info(struct bpf_iter_meta *meta, bool in_stop)
500{
501 struct bpf_iter_priv_data *iter_priv;
502 struct seq_file *seq;
503 void *seq_priv;
504
505 seq = meta->seq;
506 if (seq->file->f_op != &bpf_iter_fops)
507 return NULL;
508
509 seq_priv = seq->private;
510 iter_priv = container_of(seq_priv, struct bpf_iter_priv_data,
511 target_private);
512
513 if (in_stop && iter_priv->done_stop)
514 return NULL;
515
516 meta->session_id = iter_priv->session_id;
517 meta->seq_num = iter_priv->seq_num;
518
519 return iter_priv->prog;
520}
521
522int bpf_iter_run_prog(struct bpf_prog *prog, void *ctx)
523{
524 int ret;
525
526 rcu_read_lock();
527 migrate_disable();
528 ret = BPF_PROG_RUN(prog, ctx);
529 migrate_enable();
530 rcu_read_unlock();
531
532 /* bpf program can only return 0 or 1:
533 * 0 : okay
534 * 1 : retry the same object
535 * The bpf_iter_run_prog() return value
536 * will be seq_ops->show() return value.
537 */
538 return ret == 0 ? 0 : -EAGAIN;
539}
kernel/bpf/bpf_lsm.c+1−1
@@ -49,6 +49,6 @@ const struct bpf_prog_ops lsm_prog_ops = { · 48 unmodified lines
49};49};
50 50
51const struct bpf_verifier_ops lsm_verifier_ops = {51const struct bpf_verifier_ops lsm_verifier_ops = {
52 .get_func_proto = bpf_tracing_func_proto,52 .get_func_proto = tracing_prog_func_proto,
53 .is_valid_access = btf_ctx_access,53 .is_valid_access = btf_ctx_access,
54};54};
kernel/bpf/bpf_struct_ops.c+1−1
@@ -557,7 +557,7 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) · 556 unmodified lines
557 struct bpf_map *map;557 struct bpf_map *map;
558 int err;558 int err;
559 559
560 if (!capable(CAP_SYS_ADMIN))560 if (!bpf_capable())
561 return ERR_PTR(-EPERM);561 return ERR_PTR(-EPERM);
562 562
563 st_ops = bpf_struct_ops_find_value(attr->btf_vmlinux_value_type_id);563 st_ops = bpf_struct_ops_find_value(attr->btf_vmlinux_value_type_id);
kernel/bpf/btf.c+48−1
@@ -3482,6 +3482,7 @@ extern char __weak __stop_BTF[]; · 3481 unmodified lines
3482extern struct btf *btf_vmlinux;3482extern struct btf *btf_vmlinux;
3483 3483
3484#define BPF_MAP_TYPE(_id, _ops)3484#define BPF_MAP_TYPE(_id, _ops)
3485#define BPF_LINK_TYPE(_id, _name)
3485static union {3486static union {
3486 struct bpf_ctx_convert {3487 struct bpf_ctx_convert {
3487#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \3488#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
@@ -3508,6 +3509,7 @@ static u8 bpf_ctx_convert_map[] = { · 20 unmodified lines
3508 0, /* avoid empty array */3509 0, /* avoid empty array */
3509};3510};
3510#undef BPF_MAP_TYPE3511#undef BPF_MAP_TYPE
3512#undef BPF_LINK_TYPE
3511 3513
3512static const struct btf_member *3514static const struct btf_member *
3513btf_get_prog_ctx_type(struct bpf_verifier_log *log, struct btf *btf,3515btf_get_prog_ctx_type(struct bpf_verifier_log *log, struct btf *btf,
@@ -3692,7 +3694,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 178 unmodified lines
3692 struct bpf_verifier_log *log = info->log;3694 struct bpf_verifier_log *log = info->log;
3693 const struct btf_param *args;3695 const struct btf_param *args;
3694 u32 nr_args, arg;3696 u32 nr_args, arg;
3695 int ret;3697 int i, ret;
3696 3698
3697 if (off % 8) {3699 if (off % 8) {
3698 bpf_log(log, "func '%s' offset %d is not multiple of 8\n",3700 bpf_log(log, "func '%s' offset %d is not multiple of 8\n",
@@ -3789,6 +3791,14 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 90 unmodified lines
3789 3791
3790 /* this is a pointer to another type */3792 /* this is a pointer to another type */
3791 info->reg_type = PTR_TO_BTF_ID;3793 info->reg_type = PTR_TO_BTF_ID;
3794 for (i = 0; i < prog->aux->ctx_arg_info_size; i++) {
3795 const struct bpf_ctx_arg_aux *ctx_arg_info = &prog->aux->ctx_arg_info[i];
3796
3797 if (ctx_arg_info->offset == off) {
3798 info->reg_type = ctx_arg_info->reg_type;
3799 break;
3800 }
3801 }
3792 3802
3793 if (tgt_prog) {3803 if (tgt_prog) {
3794 ret = btf_translate_to_vmlinux(log, btf, t, tgt_prog->type, arg);3804 ret = btf_translate_to_vmlinux(log, btf, t, tgt_prog->type, arg);
@@ -3828,6 +3838,7 @@ int btf_struct_access(struct bpf_verifier_log *log, · 33 unmodified lines
3828 const struct btf_type *mtype, *elem_type = NULL;3838 const struct btf_type *mtype, *elem_type = NULL;
3829 const struct btf_member *member;3839 const struct btf_member *member;
3830 const char *tname, *mname;3840 const char *tname, *mname;
3841 u32 vlen;
3831 3842
3832again:3843again:
3833 tname = __btf_name_by_offset(btf_vmlinux, t->name_off);3844 tname = __btf_name_by_offset(btf_vmlinux, t->name_off);
@@ -3836,7 +3847,43 @@ again: · 2 unmodified lines
3836 return -EINVAL;3847 return -EINVAL;
3837 }3848 }
3838 3849
3850 vlen = btf_type_vlen(t);
3839 if (off + size > t->size) {3851 if (off + size > t->size) {
3852 /* If the last element is a variable size array, we may
3853 * need to relax the rule.
3854 */
3855 struct btf_array *array_elem;
3856
3857 if (vlen == 0)
3858 goto error;
3859
3860 member = btf_type_member(t) + vlen - 1;
3861 mtype = btf_type_skip_modifiers(btf_vmlinux, member->type,
3862 NULL);
3863 if (!btf_type_is_array(mtype))
3864 goto error;
3865
3866 array_elem = (struct btf_array *)(mtype + 1);
3867 if (array_elem->nelems != 0)
3868 goto error;
3869
3870 moff = btf_member_bit_offset(t, member) / 8;
3871 if (off < moff)
3872 goto error;
3873
3874 /* Only allow structure for now, can be relaxed for
3875 * other types later.
3876 */
3877 elem_type = btf_type_skip_modifiers(btf_vmlinux,
3878 array_elem->type, NULL);
3879 if (!btf_type_is_struct(elem_type))
3880 goto error;
3881
3882 off = (off - moff) % elem_type->size;
3883 return btf_struct_access(log, elem_type, off, size, atype,
3884 next_btf_id);
3885
3886error:
3840 bpf_log(log, "access beyond struct %s at off %u size %u\n",3887 bpf_log(log, "access beyond struct %s at off %u size %u\n",
3841 tname, off, size);3888 tname, off, size);
3842 return -EACCES;3889 return -EACCES;
kernel/bpf/cgroup.c+90−56
@@ -557,8 +557,9 @@ found: · 556 unmodified lines
557 *557 *
558 * Must be called with cgroup_mutex held.558 * Must be called with cgroup_mutex held.
559 */559 */
560int __cgroup_bpf_replace(struct cgroup *cgrp, struct bpf_cgroup_link *link,560static int __cgroup_bpf_replace(struct cgroup *cgrp,
561 struct bpf_prog *new_prog)561 struct bpf_cgroup_link *link,
562 struct bpf_prog *new_prog)
562{563{
563 struct list_head *progs = &cgrp->bpf.progs[link->type];564 struct list_head *progs = &cgrp->bpf.progs[link->type];
564 struct bpf_prog *old_prog;565 struct bpf_prog *old_prog;
@@ -583,6 +584,30 @@ int __cgroup_bpf_replace(struct cgroup *cgrp, struct bpf_cgroup_link *link, · 18 unmodified lines
583 return 0;584 return 0;
584}585}
585 586
587static int cgroup_bpf_replace(struct bpf_link *link, struct bpf_prog *new_prog,
588 struct bpf_prog *old_prog)
589{
590 struct bpf_cgroup_link *cg_link;
591 int ret;
592
593 cg_link = container_of(link, struct bpf_cgroup_link, link);
594
595 mutex_lock(&cgroup_mutex);
596 /* link might have been auto-released by dying cgroup, so fail */
597 if (!cg_link->cgroup) {
598 ret = -ENOLINK;
599 goto out_unlock;
600 }
601 if (old_prog && link->prog != old_prog) {
602 ret = -EPERM;
603 goto out_unlock;
604 }
605 ret = __cgroup_bpf_replace(cg_link->cgroup, cg_link, new_prog);
606out_unlock:
607 mutex_unlock(&cgroup_mutex);
608 return ret;
609}
610
586static struct bpf_prog_list *find_detach_entry(struct list_head *progs,611static struct bpf_prog_list *find_detach_entry(struct list_head *progs,
587 struct bpf_prog *prog,612 struct bpf_prog *prog,
588 struct bpf_cgroup_link *link,613 struct bpf_cgroup_link *link,
@@ -808,17 +833,56 @@ static void bpf_cgroup_link_dealloc(struct bpf_link *link) · 219 unmodified lines
808 kfree(cg_link);833 kfree(cg_link);
809}834}
810 835
811const struct bpf_link_ops bpf_cgroup_link_lops = {836static void bpf_cgroup_link_show_fdinfo(const struct bpf_link *link,
837 struct seq_file *seq)
838{
839 struct bpf_cgroup_link *cg_link =
840 container_of(link, struct bpf_cgroup_link, link);
841 u64 cg_id = 0;
842
843 mutex_lock(&cgroup_mutex);
844 if (cg_link->cgroup)
845 cg_id = cgroup_id(cg_link->cgroup);
846 mutex_unlock(&cgroup_mutex);
847
848 seq_printf(seq,
849 "cgroup_id:\t%llu\n"
850 "attach_type:\t%d\n",
851 cg_id,
852 cg_link->type);
853}
854
855static int bpf_cgroup_link_fill_link_info(const struct bpf_link *link,
856 struct bpf_link_info *info)
857{
858 struct bpf_cgroup_link *cg_link =
859 container_of(link, struct bpf_cgroup_link, link);
860 u64 cg_id = 0;
861
862 mutex_lock(&cgroup_mutex);
863 if (cg_link->cgroup)
864 cg_id = cgroup_id(cg_link->cgroup);
865 mutex_unlock(&cgroup_mutex);
866
867 info->cgroup.cgroup_id = cg_id;
868 info->cgroup.attach_type = cg_link->type;
869 return 0;
870}
871
872static const struct bpf_link_ops bpf_cgroup_link_lops = {
812 .release = bpf_cgroup_link_release,873 .release = bpf_cgroup_link_release,
813 .dealloc = bpf_cgroup_link_dealloc,874 .dealloc = bpf_cgroup_link_dealloc,
875 .update_prog = cgroup_bpf_replace,
876 .show_fdinfo = bpf_cgroup_link_show_fdinfo,
877 .fill_link_info = bpf_cgroup_link_fill_link_info,
814};878};
815 879
816int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)880int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
817{881{
882 struct bpf_link_primer link_primer;
818 struct bpf_cgroup_link *link;883 struct bpf_cgroup_link *link;
819 struct file *link_file;
820 struct cgroup *cgrp;884 struct cgroup *cgrp;
821 int err, link_fd;885 int err;
822 886
823 if (attr->link_create.flags)887 if (attr->link_create.flags)
824 return -EINVAL;888 return -EINVAL;
@@ -832,26 +896,25 @@ int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) · 7 unmodified lines
832 err = -ENOMEM;896 err = -ENOMEM;
833 goto out_put_cgroup;897 goto out_put_cgroup;
834 }898 }
835 bpf_link_init(&link->link, &bpf_cgroup_link_lops, prog);899 bpf_link_init(&link->link, BPF_LINK_TYPE_CGROUP, &bpf_cgroup_link_lops,
900 prog);
836 link->cgroup = cgrp;901 link->cgroup = cgrp;
837 link->type = attr->link_create.attach_type;902 link->type = attr->link_create.attach_type;
838 903
839 link_file = bpf_link_new_file(&link->link, &link_fd);904 err = bpf_link_prime(&link->link, &link_primer);
840 if (IS_ERR(link_file)) {905 if (err) {
841 kfree(link);906 kfree(link);
842 err = PTR_ERR(link_file);
843 goto out_put_cgroup;907 goto out_put_cgroup;
844 }908 }
845 909
846 err = cgroup_bpf_attach(cgrp, NULL, NULL, link, link->type,910 err = cgroup_bpf_attach(cgrp, NULL, NULL, link, link->type,
847 BPF_F_ALLOW_MULTI);911 BPF_F_ALLOW_MULTI);
848 if (err) {912 if (err) {
849 bpf_link_cleanup(&link->link, link_file, link_fd);913 bpf_link_cleanup(&link_primer);
850 goto out_put_cgroup;914 goto out_put_cgroup;
851 }915 }
852 916
853 fd_install(link_fd, link_file);917 return bpf_link_settle(&link_primer);
854 return link_fd;
855 918
856out_put_cgroup:919out_put_cgroup:
857 cgroup_put(cgrp);920 cgroup_put(cgrp);
@@ -1054,36 +1117,21 @@ int __cgroup_bpf_check_dev_permission(short dev_type, u32 major, u32 minor, · 196 unmodified lines
1054 1117
1055 return !allow;1118 return !allow;
1056}1119}
1057EXPORT_SYMBOL(__cgroup_bpf_check_dev_permission);
1058 1120
1059static const struct bpf_func_proto *1121static const struct bpf_func_proto *
1060cgroup_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)1122cgroup_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
1061{1123{
1062 switch (func_id) {1124 switch (func_id) {
1063 case BPF_FUNC_map_lookup_elem:
1064 return &bpf_map_lookup_elem_proto;
1065 case BPF_FUNC_map_update_elem:
1066 return &bpf_map_update_elem_proto;
1067 case BPF_FUNC_map_delete_elem:
1068 return &bpf_map_delete_elem_proto;
1069 case BPF_FUNC_map_push_elem:
1070 return &bpf_map_push_elem_proto;
1071 case BPF_FUNC_map_pop_elem:
1072 return &bpf_map_pop_elem_proto;
1073 case BPF_FUNC_map_peek_elem:
1074 return &bpf_map_peek_elem_proto;
1075 case BPF_FUNC_get_current_uid_gid:1125 case BPF_FUNC_get_current_uid_gid:
1076 return &bpf_get_current_uid_gid_proto;1126 return &bpf_get_current_uid_gid_proto;
1077 case BPF_FUNC_get_local_storage:1127 case BPF_FUNC_get_local_storage:
1078 return &bpf_get_local_storage_proto;1128 return &bpf_get_local_storage_proto;
1079 case BPF_FUNC_get_current_cgroup_id:1129 case BPF_FUNC_get_current_cgroup_id:
1080 return &bpf_get_current_cgroup_id_proto;1130 return &bpf_get_current_cgroup_id_proto;
1081 case BPF_FUNC_trace_printk:1131 case BPF_FUNC_perf_event_output:
1082 if (capable(CAP_SYS_ADMIN))1132 return &bpf_event_output_data_proto;
1083 return bpf_get_trace_printk_proto();
1084 /* fall through */
1085 default:1133 default:
1086 return NULL;1134 return bpf_base_func_proto(func_id);
1087 }1135 }
1088}1136}
1089 1137
@@ -1137,16 +1185,13 @@ const struct bpf_verifier_ops cg_dev_verifier_ops = { · 47 unmodified lines
1137 * @head: sysctl table header1185 * @head: sysctl table header
1138 * @table: sysctl table1186 * @table: sysctl table
1139 * @write: sysctl is being read (= 0) or written (= 1)1187 * @write: sysctl is being read (= 0) or written (= 1)
1140 * @buf: pointer to buffer passed by user space1188 * @buf: pointer to buffer (in and out)
1141 * @pcount: value-result argument: value is size of buffer pointed to by @buf,1189 * @pcount: value-result argument: value is size of buffer pointed to by @buf,
1142 * result is size of @new_buf if program set new value, initial value1190 * result is size of @new_buf if program set new value, initial value
1143 * otherwise1191 * otherwise
1144 * @ppos: value-result argument: value is position at which read from or write1192 * @ppos: value-result argument: value is position at which read from or write
1145 * to sysctl is happening, result is new position if program overrode it,1193 * to sysctl is happening, result is new position if program overrode it,
1146 * initial value otherwise1194 * initial value otherwise
1147 * @new_buf: pointer to pointer to new buffer that will be allocated if program
1148 * overrides new value provided by user space on sysctl write
1149 * NOTE: it's caller responsibility to free *new_buf if it was set
1150 * @type: type of program to be executed1195 * @type: type of program to be executed
1151 *1196 *
1152 * Program is run when sysctl is being accessed, either read or written, and1197 * Program is run when sysctl is being accessed, either read or written, and
@@ -1157,8 +1202,7 @@ const struct bpf_verifier_ops cg_dev_verifier_ops = { · 4 unmodified lines
1157 */1202 */
1158int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head,1203int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head,
1159 struct ctl_table *table, int write,1204 struct ctl_table *table, int write,
1160 void __user *buf, size_t *pcount,1205 void **buf, size_t *pcount, loff_t *ppos,
1161 loff_t *ppos, void **new_buf,
1162 enum bpf_attach_type type)1206 enum bpf_attach_type type)
1163{1207{
1164 struct bpf_sysctl_kern ctx = {1208 struct bpf_sysctl_kern ctx = {
@@ -1173,36 +1217,28 @@ int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head, · 8 unmodified lines
1173 .new_updated = 0,1217 .new_updated = 0,
1174 };1218 };
1175 struct cgroup *cgrp;1219 struct cgroup *cgrp;
1220 loff_t pos = 0;
1176 int ret;1221 int ret;
1177 1222
1178 ctx.cur_val = kmalloc_track_caller(ctx.cur_len, GFP_KERNEL);1223 ctx.cur_val = kmalloc_track_caller(ctx.cur_len, GFP_KERNEL);
1179 if (ctx.cur_val) {1224 if (!ctx.cur_val ||
1180 mm_segment_t old_fs;1225 table->proc_handler(table, 0, ctx.cur_val, &ctx.cur_len, &pos)) {
1181 loff_t pos = 0;
1182
1183 old_fs = get_fs();
1184 set_fs(KERNEL_DS);
1185 if (table->proc_handler(table, 0, (void __user *)ctx.cur_val,
1186 &ctx.cur_len, &pos)) {
1187 /* Let BPF program decide how to proceed. */
1188 ctx.cur_len = 0;
1189 }
1190 set_fs(old_fs);
1191 } else {
1192 /* Let BPF program decide how to proceed. */1226 /* Let BPF program decide how to proceed. */
1193 ctx.cur_len = 0;1227 ctx.cur_len = 0;
1194 }1228 }
1195 1229
1196 if (write && buf && *pcount) {1230 if (write && *buf && *pcount) {
1197 /* BPF program should be able to override new value with a1231 /* BPF program should be able to override new value with a
1198 * buffer bigger than provided by user.1232 * buffer bigger than provided by user.
1199 */1233 */
1200 ctx.new_val = kmalloc_track_caller(PAGE_SIZE, GFP_KERNEL);1234 ctx.new_val = kmalloc_track_caller(PAGE_SIZE, GFP_KERNEL);
1201 ctx.new_len = min_t(size_t, PAGE_SIZE, *pcount);1235 ctx.new_len = min_t(size_t, PAGE_SIZE, *pcount);
1202 if (!ctx.new_val ||1236 if (ctx.new_val) {
1203 copy_from_user(ctx.new_val, buf, ctx.new_len))1237 memcpy(ctx.new_val, *buf, ctx.new_len);
1238 } else {
1204 /* Let BPF program decide how to proceed. */1239 /* Let BPF program decide how to proceed. */
1205 ctx.new_len = 0;1240 ctx.new_len = 0;
1241 }
1206 }1242 }
1207 1243
1208 rcu_read_lock();1244 rcu_read_lock();
@@ -1213,7 +1249,8 @@ int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head, · 4 unmodified lines
1213 kfree(ctx.cur_val);1249 kfree(ctx.cur_val);
1214 1250
1215 if (ret == 1 && ctx.new_updated) {1251 if (ret == 1 && ctx.new_updated) {
1216 *new_buf = ctx.new_val;1252 kfree(*buf);
1253 *buf = ctx.new_val;
1217 *pcount = ctx.new_len;1254 *pcount = ctx.new_len;
1218 } else {1255 } else {
1219 kfree(ctx.new_val);1256 kfree(ctx.new_val);
@@ -1221,7 +1258,6 @@ int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head, · 1 unmodified line
1221 1258
1222 return ret == 1 ? 0 : -EPERM;1259 return ret == 1 ? 0 : -EPERM;
1223}1260}
1224EXPORT_SYMBOL(__cgroup_bpf_run_filter_sysctl);
1225 1261
1226#ifdef CONFIG_NET1262#ifdef CONFIG_NET
1227static bool __cgroup_bpf_prog_array_is_empty(struct cgroup *cgrp,1263static bool __cgroup_bpf_prog_array_is_empty(struct cgroup *cgrp,
@@ -1326,7 +1362,6 @@ out: · 98 unmodified lines
1326 sockopt_free_buf(&ctx);1362 sockopt_free_buf(&ctx);
1327 return ret;1363 return ret;
1328}1364}
1329EXPORT_SYMBOL(__cgroup_bpf_run_filter_setsockopt);
1330 1365
1331int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level,1366int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level,
1332 int optname, char __user *optval,1367 int optname, char __user *optval,
@@ -1413,7 +1448,6 @@ out: · 80 unmodified lines
1413 sockopt_free_buf(&ctx);1448 sockopt_free_buf(&ctx);
1414 return ret;1449 return ret;
1415}1450}
1416EXPORT_SYMBOL(__cgroup_bpf_run_filter_getsockopt);
1417#endif1451#endif
1418 1452
1419static ssize_t sysctl_cpy_dir(const struct ctl_dir *dir, char **bufp,1453static ssize_t sysctl_cpy_dir(const struct ctl_dir *dir, char **bufp,
kernel/bpf/core.c+8−2
@@ -646,7 +646,7 @@ static bool bpf_prog_kallsyms_verify_off(const struct bpf_prog *fp) · 645 unmodified lines
646void bpf_prog_kallsyms_add(struct bpf_prog *fp)646void bpf_prog_kallsyms_add(struct bpf_prog *fp)
647{647{
648 if (!bpf_prog_kallsyms_candidate(fp) ||648 if (!bpf_prog_kallsyms_candidate(fp) ||
649 !capable(CAP_SYS_ADMIN))649 !bpf_capable())
650 return;650 return;
651 651
652 bpf_prog_ksym_set_addr(fp);652 bpf_prog_ksym_set_addr(fp);
@@ -1543,7 +1543,7 @@ select_insn: · 890 unmodified lines
1543 1543
1544 /* ARG1 at this point is guaranteed to point to CTX from1544 /* ARG1 at this point is guaranteed to point to CTX from
1545 * the verifier side due to the fact that the tail call is1545 * the verifier side due to the fact that the tail call is
1546 * handeled like a helper, that is, bpf_tail_call_proto,1546 * handled like a helper, that is, bpf_tail_call_proto,
1547 * where arg1_type is ARG_PTR_TO_CTX.1547 * where arg1_type is ARG_PTR_TO_CTX.
1548 */1548 */
1549 insn = prog->insnsi;1549 insn = prog->insnsi;
@@ -2136,6 +2136,11 @@ BPF_CALL_0(bpf_user_rnd_u32) · 586 unmodified lines
2136 return res;2136 return res;
2137}2137}
2138 2138
2139BPF_CALL_0(bpf_get_raw_cpu_id)
2140{
2141 return raw_smp_processor_id();
2142}
2143
2139/* Weak definitions of helper functions in case we don't have bpf syscall. */2144/* Weak definitions of helper functions in case we don't have bpf syscall. */
2140const struct bpf_func_proto bpf_map_lookup_elem_proto __weak;2145const struct bpf_func_proto bpf_map_lookup_elem_proto __weak;
2141const struct bpf_func_proto bpf_map_update_elem_proto __weak;2146const struct bpf_func_proto bpf_map_update_elem_proto __weak;
@@ -2151,6 +2156,7 @@ const struct bpf_func_proto bpf_get_prandom_u32_proto __weak; · 9 unmodified lines
2151const struct bpf_func_proto bpf_get_smp_processor_id_proto __weak;2156const struct bpf_func_proto bpf_get_smp_processor_id_proto __weak;
2152const struct bpf_func_proto bpf_get_numa_node_id_proto __weak;2157const struct bpf_func_proto bpf_get_numa_node_id_proto __weak;
2153const struct bpf_func_proto bpf_ktime_get_ns_proto __weak;2158const struct bpf_func_proto bpf_ktime_get_ns_proto __weak;
2159const struct bpf_func_proto bpf_ktime_get_boot_ns_proto __weak;
2154 2160
2155const struct bpf_func_proto bpf_get_current_pid_tgid_proto __weak;2161const struct bpf_func_proto bpf_get_current_pid_tgid_proto __weak;
2156const struct bpf_func_proto bpf_get_current_uid_gid_proto __weak;2162const struct bpf_func_proto bpf_get_current_uid_gid_proto __weak;
kernel/bpf/cpumap.c+5−20
@@ -85,7 +85,7 @@ static struct bpf_map *cpu_map_alloc(union bpf_attr *attr) · 84 unmodified lines
85 u64 cost;85 u64 cost;
86 int ret;86 int ret;
87 87
88 if (!capable(CAP_SYS_ADMIN))88 if (!bpf_capable())
89 return ERR_PTR(-EPERM);89 return ERR_PTR(-EPERM);
90 90
91 /* check sanity of attributes */91 /* check sanity of attributes */
@@ -162,25 +162,10 @@ static struct sk_buff *cpu_map_build_skb(struct bpf_cpu_map_entry *rcpu, · 70 unmodified lines
162 /* Part of headroom was reserved to xdpf */162 /* Part of headroom was reserved to xdpf */
163 hard_start_headroom = sizeof(struct xdp_frame) + xdpf->headroom;163 hard_start_headroom = sizeof(struct xdp_frame) + xdpf->headroom;
164 164
165 /* build_skb need to place skb_shared_info after SKB end, and165 /* Memory size backing xdp_frame data already have reserved
166 * also want to know the memory "truesize". Thus, need to166 * room for build_skb to place skb_shared_info in tailroom.
167 * know the memory frame size backing xdp_buff.
168 *
169 * XDP was designed to have PAGE_SIZE frames, but this
170 * assumption is not longer true with ixgbe and i40e. It
171 * would be preferred to set frame_size to 2048 or 4096
172 * depending on the driver.
173 * frame_size = 2048;
174 * frame_len = frame_size - sizeof(*xdp_frame);
175 *
176 * Instead, with info avail, skb_shared_info in placed after
177 * packet len. This, unfortunately fakes the truesize.
178 * Another disadvantage of this approach, the skb_shared_info
179 * is not at a fixed memory location, with mixed length
180 * packets, which is bad for cache-line hotness.
181 */167 */
182 frame_size = SKB_DATA_ALIGN(xdpf->len + hard_start_headroom) +168 frame_size = xdpf->frame_sz;
183 SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
184 169
185 pkt_data_start = xdpf->data - hard_start_headroom;170 pkt_data_start = xdpf->data - hard_start_headroom;
186 skb = build_skb_around(skb, pkt_data_start, frame_size);171 skb = build_skb_around(skb, pkt_data_start, frame_size);
@@ -636,7 +621,7 @@ int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp, · 449 unmodified lines
636{621{
637 struct xdp_frame *xdpf;622 struct xdp_frame *xdpf;
638 623
639 xdpf = convert_to_xdp_frame(xdp);624 xdpf = xdp_convert_buff_to_frame(xdp);
640 if (unlikely(!xdpf))625 if (unlikely(!xdpf))
641 return -EOVERFLOW;626 return -EOVERFLOW;
642 627
kernel/bpf/devmap.c+113−20
@@ -52,7 +52,6 @@ · 51 unmodified lines
52#define DEV_CREATE_FLAG_MASK \52#define DEV_CREATE_FLAG_MASK \
53 (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY)53 (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY)
54 54
55#define DEV_MAP_BULK_SIZE 16
56struct xdp_dev_bulk_queue {55struct xdp_dev_bulk_queue {
57 struct xdp_frame *q[DEV_MAP_BULK_SIZE];56 struct xdp_frame *q[DEV_MAP_BULK_SIZE];
58 struct list_head flush_node;57 struct list_head flush_node;
@@ -61,12 +60,23 @@ struct xdp_dev_bulk_queue { · 2 unmodified lines
61 unsigned int count;60 unsigned int count;
62};61};
63 62
63/* DEVMAP values */
64struct bpf_devmap_val {
65 u32 ifindex; /* device index */
66 union {
67 int fd; /* prog fd on map write */
68 u32 id; /* prog id on map read */
69 } bpf_prog;
70};
71
64struct bpf_dtab_netdev {72struct bpf_dtab_netdev {
65 struct net_device *dev; /* must be first member, due to tracepoint */73 struct net_device *dev; /* must be first member, due to tracepoint */
66 struct hlist_node index_hlist;74 struct hlist_node index_hlist;
67 struct bpf_dtab *dtab;75 struct bpf_dtab *dtab;
76 struct bpf_prog *xdp_prog;
68 struct rcu_head rcu;77 struct rcu_head rcu;
69 unsigned int idx;78 unsigned int idx;
79 struct bpf_devmap_val val;
70};80};
71 81
72struct bpf_dtab {82struct bpf_dtab {
@@ -106,12 +116,18 @@ static inline struct hlist_head *dev_map_index_hash(struct bpf_dtab *dtab, · 33 unmodified lines
106 116
107static int dev_map_init_map(struct bpf_dtab *dtab, union bpf_attr *attr)117static int dev_map_init_map(struct bpf_dtab *dtab, union bpf_attr *attr)
108{118{
119 u32 valsize = attr->value_size;
109 u64 cost = 0;120 u64 cost = 0;
110 int err;121 int err;
111 122
112 /* check sanity of attributes */123 /* check sanity of attributes. 2 value sizes supported:
124 * 4 bytes: ifindex
125 * 8 bytes: ifindex + prog fd
126 */
113 if (attr->max_entries == 0 || attr->key_size != 4 ||127 if (attr->max_entries == 0 || attr->key_size != 4 ||
114 attr->value_size != 4 || attr->map_flags & ~DEV_CREATE_FLAG_MASK)128 (valsize != offsetofend(struct bpf_devmap_val, ifindex) &&
129 valsize != offsetofend(struct bpf_devmap_val, bpf_prog.fd)) ||
130 attr->map_flags & ~DEV_CREATE_FLAG_MASK)
115 return -EINVAL;131 return -EINVAL;
116 132
117 /* Lookup returns a pointer straight to dev->ifindex, so make sure the133 /* Lookup returns a pointer straight to dev->ifindex, so make sure the
@@ -218,6 +234,8 @@ static void dev_map_free(struct bpf_map *map) · 100 unmodified lines
218 234
219 hlist_for_each_entry_safe(dev, next, head, index_hlist) {235 hlist_for_each_entry_safe(dev, next, head, index_hlist) {
220 hlist_del_rcu(&dev->index_hlist);236 hlist_del_rcu(&dev->index_hlist);
237 if (dev->xdp_prog)
238 bpf_prog_put(dev->xdp_prog);
221 dev_put(dev->dev);239 dev_put(dev->dev);
222 kfree(dev);240 kfree(dev);
223 }241 }
@@ -232,6 +250,8 @@ static void dev_map_free(struct bpf_map *map) · 8 unmodified lines
232 if (!dev)250 if (!dev)
233 continue;251 continue;
234 252
253 if (dev->xdp_prog)
254 bpf_prog_put(dev->xdp_prog);
235 dev_put(dev->dev);255 dev_put(dev->dev);
236 kfree(dev);256 kfree(dev);
237 }257 }
@@ -318,6 +338,16 @@ static int dev_map_hash_get_next_key(struct bpf_map *map, void *key, · 80 unmodified lines
318 return -ENOENT;338 return -ENOENT;
319}339}
320 340
341bool dev_map_can_have_prog(struct bpf_map *map)
342{
343 if ((map->map_type == BPF_MAP_TYPE_DEVMAP ||
344 map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) &&
345 map->value_size != offsetofend(struct bpf_devmap_val, ifindex))
346 return true;
347
348 return false;
349}
350
321static int bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)351static int bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
322{352{
323 struct net_device *dev = bq->dev;353 struct net_device *dev = bq->dev;
@@ -435,13 +465,40 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, · 111 unmodified lines
435 if (unlikely(err))465 if (unlikely(err))
436 return err;466 return err;
437 467
438 xdpf = convert_to_xdp_frame(xdp);468 xdpf = xdp_convert_buff_to_frame(xdp);
439 if (unlikely(!xdpf))469 if (unlikely(!xdpf))
440 return -EOVERFLOW;470 return -EOVERFLOW;
441 471
442 return bq_enqueue(dev, xdpf, dev_rx);472 return bq_enqueue(dev, xdpf, dev_rx);
443}473}
444 474
475static struct xdp_buff *dev_map_run_prog(struct net_device *dev,
476 struct xdp_buff *xdp,
477 struct bpf_prog *xdp_prog)
478{
479 struct xdp_txq_info txq = { .dev = dev };
480 u32 act;
481
482 xdp->txq = &txq;
483
484 act = bpf_prog_run_xdp(xdp_prog, xdp);
485 switch (act) {
486 case XDP_PASS:
487 return xdp;
488 case XDP_DROP:
489 break;
490 default:
491 bpf_warn_invalid_xdp_action(act);
492 fallthrough;
493 case XDP_ABORTED:
494 trace_xdp_exception(dev, xdp_prog, act);
495 break;
496 }
497
498 xdp_return_buff(xdp);
499 return NULL;
500}
501
445int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp,502int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp,
446 struct net_device *dev_rx)503 struct net_device *dev_rx)
447{504{
@@ -453,6 +510,11 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, · 5 unmodified lines
453{510{
454 struct net_device *dev = dst->dev;511 struct net_device *dev = dst->dev;
455 512
513 if (dst->xdp_prog) {
514 xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog);
515 if (!xdp)
516 return 0;
517 }
456 return __xdp_enqueue(dev, xdp, dev_rx);518 return __xdp_enqueue(dev, xdp, dev_rx);
457}519}
458 520
@@ -473,18 +535,15 @@ int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, · 14 unmodified lines
473static void *dev_map_lookup_elem(struct bpf_map *map, void *key)535static void *dev_map_lookup_elem(struct bpf_map *map, void *key)
474{536{
475 struct bpf_dtab_netdev *obj = __dev_map_lookup_elem(map, *(u32 *)key);537 struct bpf_dtab_netdev *obj = __dev_map_lookup_elem(map, *(u32 *)key);
476 struct net_device *dev = obj ? obj->dev : NULL;
477 538
478 return dev ? &dev->ifindex : NULL;539 return obj ? &obj->val : NULL;
479}540}
480 541
481static void *dev_map_hash_lookup_elem(struct bpf_map *map, void *key)542static void *dev_map_hash_lookup_elem(struct bpf_map *map, void *key)
482{543{
483 struct bpf_dtab_netdev *obj = __dev_map_hash_lookup_elem(map,544 struct bpf_dtab_netdev *obj = __dev_map_hash_lookup_elem(map,
484 *(u32 *)key);545 *(u32 *)key);
485 struct net_device *dev = obj ? obj->dev : NULL;546 return obj ? &obj->val : NULL;
486
487 return dev ? &dev->ifindex : NULL;
488}547}
489 548
490static void __dev_map_entry_free(struct rcu_head *rcu)549static void __dev_map_entry_free(struct rcu_head *rcu)
@@ -492,6 +551,8 @@ static void __dev_map_entry_free(struct rcu_head *rcu) · 1 unmodified line
492 struct bpf_dtab_netdev *dev;551 struct bpf_dtab_netdev *dev;
493 552
494 dev = container_of(rcu, struct bpf_dtab_netdev, rcu);553 dev = container_of(rcu, struct bpf_dtab_netdev, rcu);
554 if (dev->xdp_prog)
555 bpf_prog_put(dev->xdp_prog);
495 dev_put(dev->dev);556 dev_put(dev->dev);
496 kfree(dev);557 kfree(dev);
497}558}
@@ -542,9 +603,10 @@ static int dev_map_hash_delete_elem(struct bpf_map *map, void *key) · 44 unmodified lines
542 603
543static struct bpf_dtab_netdev *__dev_map_alloc_node(struct net *net,604static struct bpf_dtab_netdev *__dev_map_alloc_node(struct net *net,
544 struct bpf_dtab *dtab,605 struct bpf_dtab *dtab,
545 u32 ifindex,606 struct bpf_devmap_val *val,
546 unsigned int idx)607 unsigned int idx)
547{608{
609 struct bpf_prog *prog = NULL;
548 struct bpf_dtab_netdev *dev;610 struct bpf_dtab_netdev *dev;
549 611
550 dev = kmalloc_node(sizeof(*dev), GFP_ATOMIC | __GFP_NOWARN,612 dev = kmalloc_node(sizeof(*dev), GFP_ATOMIC | __GFP_NOWARN,
@@ -552,24 +614,46 @@ static struct bpf_dtab_netdev *__dev_map_alloc_node(struct net *net, · 1 unmodified line
552 if (!dev)614 if (!dev)
553 return ERR_PTR(-ENOMEM);615 return ERR_PTR(-ENOMEM);
554 616
555 dev->dev = dev_get_by_index(net, ifindex);617 dev->dev = dev_get_by_index(net, val->ifindex);
556 if (!dev->dev) {618 if (!dev->dev)
557 kfree(dev);619 goto err_out;
558 return ERR_PTR(-EINVAL);620
621 if (val->bpf_prog.fd >= 0) {
622 prog = bpf_prog_get_type_dev(val->bpf_prog.fd,
623 BPF_PROG_TYPE_XDP, false);
624 if (IS_ERR(prog))
625 goto err_put_dev;
626 if (prog->expected_attach_type != BPF_XDP_DEVMAP)
627 goto err_put_prog;
559 }628 }
560 629
561 dev->idx = idx;630 dev->idx = idx;
562 dev->dtab = dtab;631 dev->dtab = dtab;
632 if (prog) {
633 dev->xdp_prog = prog;
634 dev->val.bpf_prog.id = prog->aux->id;
635 } else {
636 dev->xdp_prog = NULL;
637 dev->val.bpf_prog.id = 0;
638 }
639 dev->val.ifindex = val->ifindex;
563 640
564 return dev;641 return dev;
642err_put_prog:
643 bpf_prog_put(prog);
644err_put_dev:
645 dev_put(dev->dev);
646err_out:
647 kfree(dev);
648 return ERR_PTR(-EINVAL);
565}649}
566 650
567static int __dev_map_update_elem(struct net *net, struct bpf_map *map,651static int __dev_map_update_elem(struct net *net, struct bpf_map *map,
568 void *key, void *value, u64 map_flags)652 void *key, void *value, u64 map_flags)
569{653{
570 struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);654 struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
655 struct bpf_devmap_val val = { .bpf_prog.fd = -1 };
571 struct bpf_dtab_netdev *dev, *old_dev;656 struct bpf_dtab_netdev *dev, *old_dev;
572 u32 ifindex = *(u32 *)value;
573 u32 i = *(u32 *)key;657 u32 i = *(u32 *)key;
574 658
575 if (unlikely(map_flags > BPF_EXIST))659 if (unlikely(map_flags > BPF_EXIST))
@@ -579,10 +663,16 @@ static int __dev_map_update_elem(struct net *net, struct bpf_map *map, · 3 unmodified lines
579 if (unlikely(map_flags == BPF_NOEXIST))663 if (unlikely(map_flags == BPF_NOEXIST))
580 return -EEXIST;664 return -EEXIST;
581 665
582 if (!ifindex) {666 /* already verified value_size <= sizeof val */
667 memcpy(&val, value, map->value_size);
668
669 if (!val.ifindex) {
583 dev = NULL;670 dev = NULL;
671 /* can not specify fd if ifindex is 0 */
672 if (val.bpf_prog.fd != -1)
673 return -EINVAL;
584 } else {674 } else {
585 dev = __dev_map_alloc_node(net, dtab, ifindex, i);675 dev = __dev_map_alloc_node(net, dtab, &val, i);
586 if (IS_ERR(dev))676 if (IS_ERR(dev))
587 return PTR_ERR(dev);677 return PTR_ERR(dev);
588 }678 }
@@ -609,13 +699,16 @@ static int __dev_map_hash_update_elem(struct net *net, struct bpf_map *map, · 20 unmodified lines
609 void *key, void *value, u64 map_flags)699 void *key, void *value, u64 map_flags)
610{700{
611 struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);701 struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
702 struct bpf_devmap_val val = { .bpf_prog.fd = -1 };
612 struct bpf_dtab_netdev *dev, *old_dev;703 struct bpf_dtab_netdev *dev, *old_dev;
613 u32 ifindex = *(u32 *)value;
614 u32 idx = *(u32 *)key;704 u32 idx = *(u32 *)key;
615 unsigned long flags;705 unsigned long flags;
616 int err = -EEXIST;706 int err = -EEXIST;
617 707
618 if (unlikely(map_flags > BPF_EXIST || !ifindex))708 /* already verified value_size <= sizeof val */
709 memcpy(&val, value, map->value_size);
710
711 if (unlikely(map_flags > BPF_EXIST || !val.ifindex))
619 return -EINVAL;712 return -EINVAL;
620 713
621 spin_lock_irqsave(&dtab->index_lock, flags);714 spin_lock_irqsave(&dtab->index_lock, flags);
@@ -624,7 +717,7 @@ static int __dev_map_hash_update_elem(struct net *net, struct bpf_map *map, · 2 unmodified lines
624 if (old_dev && (map_flags & BPF_NOEXIST))717 if (old_dev && (map_flags & BPF_NOEXIST))
625 goto out_err;718 goto out_err;
626 719
627 dev = __dev_map_alloc_node(net, dtab, ifindex, idx);720 dev = __dev_map_alloc_node(net, dtab, &val, idx);
628 if (IS_ERR(dev)) {721 if (IS_ERR(dev)) {
629 err = PTR_ERR(dev);722 err = PTR_ERR(dev);
630 goto out_err;723 goto out_err;
kernel/bpf/hashtab.c+2−2
@@ -359,9 +359,9 @@ static int htab_map_alloc_check(union bpf_attr *attr) · 358 unmodified lines
359 BUILD_BUG_ON(offsetof(struct htab_elem, fnode.next) !=359 BUILD_BUG_ON(offsetof(struct htab_elem, fnode.next) !=
360 offsetof(struct htab_elem, hash_node.pprev));360 offsetof(struct htab_elem, hash_node.pprev));
361 361
362 if (lru && !capable(CAP_SYS_ADMIN))362 if (lru && !bpf_capable())
363 /* LRU implementation is much complicated than other363 /* LRU implementation is much complicated than other
364 * maps. Hence, limit to CAP_SYS_ADMIN for now.364 * maps. Hence, limit to CAP_BPF.
365 */365 */
366 return -EPERM;366 return -EPERM;
367 367
kernel/bpf/helpers.c+124−1
@@ -151,7 +151,19 @@ BPF_CALL_0(bpf_ktime_get_ns) · 150 unmodified lines
151 151
152const struct bpf_func_proto bpf_ktime_get_ns_proto = {152const struct bpf_func_proto bpf_ktime_get_ns_proto = {
153 .func = bpf_ktime_get_ns,153 .func = bpf_ktime_get_ns,
154 .gpl_only = true,154 .gpl_only = false,
155 .ret_type = RET_INTEGER,
156};
157
158BPF_CALL_0(bpf_ktime_get_boot_ns)
159{
160 /* NMI safe access to clock boottime */
161 return ktime_get_boot_fast_ns();
162}
163
164const struct bpf_func_proto bpf_ktime_get_boot_ns_proto = {
165 .func = bpf_ktime_get_boot_ns,
166 .gpl_only = false,
155 .ret_type = RET_INTEGER,167 .ret_type = RET_INTEGER,
156};168};
157 169
@@ -562,3 +574,114 @@ const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto = { · 404 unmodified lines
562 .arg3_type = ARG_PTR_TO_UNINIT_MEM,574 .arg3_type = ARG_PTR_TO_UNINIT_MEM,
563 .arg4_type = ARG_CONST_SIZE,575 .arg4_type = ARG_CONST_SIZE,
564};576};
577
578static const struct bpf_func_proto bpf_get_raw_smp_processor_id_proto = {
579 .func = bpf_get_raw_cpu_id,
580 .gpl_only = false,
581 .ret_type = RET_INTEGER,
582};
583
584BPF_CALL_5(bpf_event_output_data, void *, ctx, struct bpf_map *, map,
585 u64, flags, void *, data, u64, size)
586{
587 if (unlikely(flags & ~(BPF_F_INDEX_MASK)))
588 return -EINVAL;
589
590 return bpf_event_output(map, flags, data, size, NULL, 0, NULL);
591}
592
593const struct bpf_func_proto bpf_event_output_data_proto = {
594 .func = bpf_event_output_data,
595 .gpl_only = true,
596 .ret_type = RET_INTEGER,
597 .arg1_type = ARG_PTR_TO_CTX,
598 .arg2_type = ARG_CONST_MAP_PTR,
599 .arg3_type = ARG_ANYTHING,
600 .arg4_type = ARG_PTR_TO_MEM,
601 .arg5_type = ARG_CONST_SIZE_OR_ZERO,
602};
603
604const struct bpf_func_proto bpf_get_current_task_proto __weak;
605const struct bpf_func_proto bpf_probe_read_user_proto __weak;
606const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;
607const struct bpf_func_proto bpf_probe_read_kernel_proto __weak;
608const struct bpf_func_proto bpf_probe_read_kernel_str_proto __weak;
609
610const struct bpf_func_proto *
611bpf_base_func_proto(enum bpf_func_id func_id)
612{
613 switch (func_id) {
614 case BPF_FUNC_map_lookup_elem:
615 return &bpf_map_lookup_elem_proto;
616 case BPF_FUNC_map_update_elem:
617 return &bpf_map_update_elem_proto;
618 case BPF_FUNC_map_delete_elem:
619 return &bpf_map_delete_elem_proto;
620 case BPF_FUNC_map_push_elem:
621 return &bpf_map_push_elem_proto;
622 case BPF_FUNC_map_pop_elem:
623 return &bpf_map_pop_elem_proto;
624 case BPF_FUNC_map_peek_elem:
625 return &bpf_map_peek_elem_proto;
626 case BPF_FUNC_get_prandom_u32:
627 return &bpf_get_prandom_u32_proto;
628 case BPF_FUNC_get_smp_processor_id:
629 return &bpf_get_raw_smp_processor_id_proto;
630 case BPF_FUNC_get_numa_node_id:
631 return &bpf_get_numa_node_id_proto;
632 case BPF_FUNC_tail_call:
633 return &bpf_tail_call_proto;
634 case BPF_FUNC_ktime_get_ns:
635 return &bpf_ktime_get_ns_proto;
636 case BPF_FUNC_ktime_get_boot_ns:
637 return &bpf_ktime_get_boot_ns_proto;
638 case BPF_FUNC_ringbuf_output:
639 return &bpf_ringbuf_output_proto;
640 case BPF_FUNC_ringbuf_reserve:
641 return &bpf_ringbuf_reserve_proto;
642 case BPF_FUNC_ringbuf_submit:
643 return &bpf_ringbuf_submit_proto;
644 case BPF_FUNC_ringbuf_discard:
645 return &bpf_ringbuf_discard_proto;
646 case BPF_FUNC_ringbuf_query:
647 return &bpf_ringbuf_query_proto;
648 default:
649 break;
650 }
651
652 if (!bpf_capable())
653 return NULL;
654
655 switch (func_id) {
656 case BPF_FUNC_spin_lock:
657 return &bpf_spin_lock_proto;
658 case BPF_FUNC_spin_unlock:
659 return &bpf_spin_unlock_proto;
660 case BPF_FUNC_trace_printk:
661 if (!perfmon_capable())
662 return NULL;
663 return bpf_get_trace_printk_proto();
664 case BPF_FUNC_jiffies64:
665 return &bpf_jiffies64_proto;
666 default:
667 break;
668 }
669
670 if (!perfmon_capable())
671 return NULL;
672
673 switch (func_id) {
674 case BPF_FUNC_get_current_task:
675 return &bpf_get_current_task_proto;
676 case BPF_FUNC_probe_read_user:
677 return &bpf_probe_read_user_proto;
678 case BPF_FUNC_probe_read_kernel:
679 return &bpf_probe_read_kernel_proto;
680 case BPF_FUNC_probe_read_user_str:
681 return &bpf_probe_read_user_str_proto;
682 case BPF_FUNC_probe_read_kernel_str:
683 return &bpf_probe_read_kernel_str_proto;
684 default:
685 return NULL;
686 }
687}
kernel/bpf/inode.c+4−1
@@ -358,8 +358,11 @@ static int bpf_mkmap(struct dentry *dentry, umode_t mode, void *arg) · 357 unmodified lines
358 358
359static int bpf_mklink(struct dentry *dentry, umode_t mode, void *arg)359static int bpf_mklink(struct dentry *dentry, umode_t mode, void *arg)
360{360{
361 struct bpf_link *link = arg;
362
361 return bpf_mkobj_ops(dentry, mode, arg, &bpf_link_iops,363 return bpf_mkobj_ops(dentry, mode, arg, &bpf_link_iops,
362 &bpffs_obj_fops);364 bpf_link_is_iter(link) ?
365 &bpf_iter_fops : &bpffs_obj_fops);
363}366}
364 367
365static struct dentry *368static struct dentry *
kernel/bpf/lpm_trie.c+1−1
@@ -543,7 +543,7 @@ static struct bpf_map *trie_alloc(union bpf_attr *attr) · 542 unmodified lines
543 u64 cost = sizeof(*trie), cost_per_node;543 u64 cost = sizeof(*trie), cost_per_node;
544 int ret;544 int ret;
545 545
546 if (!capable(CAP_SYS_ADMIN))546 if (!bpf_capable())
547 return ERR_PTR(-EPERM);547 return ERR_PTR(-EPERM);
548 548
549 /* check sanity of attributes */549 /* check sanity of attributes */
kernel/bpf/map_in_map.c+1−1
@@ -60,7 +60,7 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) · 59 unmodified lines
60 /* Misc members not needed in bpf_map_meta_equal() check. */60 /* Misc members not needed in bpf_map_meta_equal() check. */
61 inner_map_meta->ops = inner_map->ops;61 inner_map_meta->ops = inner_map->ops;
62 if (inner_map->ops == &array_map_ops) {62 if (inner_map->ops == &array_map_ops) {
63 inner_map_meta->unpriv_array = inner_map->unpriv_array;63 inner_map_meta->bypass_spec_v1 = inner_map->bypass_spec_v1;
64 container_of(inner_map_meta, struct bpf_array, map)->index_mask =64 container_of(inner_map_meta, struct bpf_array, map)->index_mask =
65 container_of(inner_map, struct bpf_array, map)->index_mask;65 container_of(inner_map, struct bpf_array, map)->index_mask;
66 }66 }
kernel/bpf/map_iter.c+102−0
@@ -0,0 +1,102 @@
1// SPDX-License-Identifier: GPL-2.0-only
2/* Copyright (c) 2020 Facebook */
3#include <linux/bpf.h>
4#include <linux/fs.h>
5#include <linux/filter.h>
6#include <linux/kernel.h>
7
8struct bpf_iter_seq_map_info {
9 u32 mid;
10};
11
12static void *bpf_map_seq_start(struct seq_file *seq, loff_t *pos)
13{
14 struct bpf_iter_seq_map_info *info = seq->private;
15 struct bpf_map *map;
16
17 map = bpf_map_get_curr_or_next(&info->mid);
18 if (!map)
19 return NULL;
20
21 ++*pos;
22 return map;
23}
24
25static void *bpf_map_seq_next(struct seq_file *seq, void *v, loff_t *pos)
26{
27 struct bpf_iter_seq_map_info *info = seq->private;
28 struct bpf_map *map;
29
30 ++*pos;
31 ++info->mid;
32 bpf_map_put((struct bpf_map *)v);
33 map = bpf_map_get_curr_or_next(&info->mid);
34 if (!map)
35 return NULL;
36
37 return map;
38}
39
40struct bpf_iter__bpf_map {
41 __bpf_md_ptr(struct bpf_iter_meta *, meta);
42 __bpf_md_ptr(struct bpf_map *, map);
43};
44
45DEFINE_BPF_ITER_FUNC(bpf_map, struct bpf_iter_meta *meta, struct bpf_map *map)
46
47static int __bpf_map_seq_show(struct seq_file *seq, void *v, bool in_stop)
48{
49 struct bpf_iter__bpf_map ctx;
50 struct bpf_iter_meta meta;
51 struct bpf_prog *prog;
52 int ret = 0;
53
54 ctx.meta = &meta;
55 ctx.map = v;
56 meta.seq = seq;
57 prog = bpf_iter_get_info(&meta, in_stop);
58 if (prog)
59 ret = bpf_iter_run_prog(prog, &ctx);
60
61 return ret;
62}
63
64static int bpf_map_seq_show(struct seq_file *seq, void *v)
65{
66 return __bpf_map_seq_show(seq, v, false);
67}
68
69static void bpf_map_seq_stop(struct seq_file *seq, void *v)
70{
71 if (!v)
72 (void)__bpf_map_seq_show(seq, v, true);
73 else
74 bpf_map_put((struct bpf_map *)v);
75}
76
77static const struct seq_operations bpf_map_seq_ops = {
78 .start = bpf_map_seq_start,
79 .next = bpf_map_seq_next,
80 .stop = bpf_map_seq_stop,
81 .show = bpf_map_seq_show,
82};
83
84static const struct bpf_iter_reg bpf_map_reg_info = {
85 .target = "bpf_map",
86 .seq_ops = &bpf_map_seq_ops,
87 .init_seq_private = NULL,
88 .fini_seq_private = NULL,
89 .seq_priv_size = sizeof(struct bpf_iter_seq_map_info),
90 .ctx_arg_info_size = 1,
91 .ctx_arg_info = {
92 { offsetof(struct bpf_iter__bpf_map, map),
93 PTR_TO_BTF_ID_OR_NULL },
94 },
95};
96
97static int __init bpf_map_iter_init(void)
98{
99 return bpf_iter_reg_target(&bpf_map_reg_info);
100}
101
102late_initcall(bpf_map_iter_init);
kernel/bpf/net_namespace.c+373−0
@@ -0,0 +1,373 @@
1// SPDX-License-Identifier: GPL-2.0
2
3#include <linux/bpf.h>
4#include <linux/filter.h>
5#include <net/net_namespace.h>
6
7/*
8 * Functions to manage BPF programs attached to netns
9 */
10
11struct bpf_netns_link {
12 struct bpf_link link;
13 enum bpf_attach_type type;
14 enum netns_bpf_attach_type netns_type;
15
16 /* We don't hold a ref to net in order to auto-detach the link
17 * when netns is going away. Instead we rely on pernet
18 * pre_exit callback to clear this pointer. Must be accessed
19 * with netns_bpf_mutex held.
20 */
21 struct net *net;
22};
23
24/* Protects updates to netns_bpf */
25DEFINE_MUTEX(netns_bpf_mutex);
26
27/* Must be called with netns_bpf_mutex held. */
28static void __net_exit bpf_netns_link_auto_detach(struct bpf_link *link)
29{
30 struct bpf_netns_link *net_link =
31 container_of(link, struct bpf_netns_link, link);
32
33 net_link->net = NULL;
34}
35
36static void bpf_netns_link_release(struct bpf_link *link)
37{
38 struct bpf_netns_link *net_link =
39 container_of(link, struct bpf_netns_link, link);
40 enum netns_bpf_attach_type type = net_link->netns_type;
41 struct net *net;
42
43 /* Link auto-detached by dying netns. */
44 if (!net_link->net)
45 return;
46
47 mutex_lock(&netns_bpf_mutex);
48
49 /* Recheck after potential sleep. We can race with cleanup_net
50 * here, but if we see a non-NULL struct net pointer pre_exit
51 * has not happened yet and will block on netns_bpf_mutex.
52 */
53 net = net_link->net;
54 if (!net)
55 goto out_unlock;
56
57 net->bpf.links[type] = NULL;
58 RCU_INIT_POINTER(net->bpf.progs[type], NULL);
59
60out_unlock:
61 mutex_unlock(&netns_bpf_mutex);
62}
63
64static void bpf_netns_link_dealloc(struct bpf_link *link)
65{
66 struct bpf_netns_link *net_link =
67 container_of(link, struct bpf_netns_link, link);
68
69 kfree(net_link);
70}
71
72static int bpf_netns_link_update_prog(struct bpf_link *link,
73 struct bpf_prog *new_prog,
74 struct bpf_prog *old_prog)
75{
76 struct bpf_netns_link *net_link =
77 container_of(link, struct bpf_netns_link, link);
78 enum netns_bpf_attach_type type = net_link->netns_type;
79 struct net *net;
80 int ret = 0;
81
82 if (old_prog && old_prog != link->prog)
83 return -EPERM;
84 if (new_prog->type != link->prog->type)
85 return -EINVAL;
86
87 mutex_lock(&netns_bpf_mutex);
88
89 net = net_link->net;
90 if (!net || !check_net(net)) {
91 /* Link auto-detached or netns dying */
92 ret = -ENOLINK;
93 goto out_unlock;
94 }
95
96 old_prog = xchg(&link->prog, new_prog);
97 rcu_assign_pointer(net->bpf.progs[type], new_prog);
98 bpf_prog_put(old_prog);
99
100out_unlock:
101 mutex_unlock(&netns_bpf_mutex);
102 return ret;
103}
104
105static int bpf_netns_link_fill_info(const struct bpf_link *link,
106 struct bpf_link_info *info)
107{
108 const struct bpf_netns_link *net_link =
109 container_of(link, struct bpf_netns_link, link);
110 unsigned int inum = 0;
111 struct net *net;
112
113 mutex_lock(&netns_bpf_mutex);
114 net = net_link->net;
115 if (net && check_net(net))
116 inum = net->ns.inum;
117 mutex_unlock(&netns_bpf_mutex);
118
119 info->netns.netns_ino = inum;
120 info->netns.attach_type = net_link->type;
121 return 0;
122}
123
124static void bpf_netns_link_show_fdinfo(const struct bpf_link *link,
125 struct seq_file *seq)
126{
127 struct bpf_link_info info = {};
128
129 bpf_netns_link_fill_info(link, &info);
130 seq_printf(seq,
131 "netns_ino:\t%u\n"
132 "attach_type:\t%u\n",
133 info.netns.netns_ino,
134 info.netns.attach_type);
135}
136
137static const struct bpf_link_ops bpf_netns_link_ops = {
138 .release = bpf_netns_link_release,
139 .dealloc = bpf_netns_link_dealloc,
140 .update_prog = bpf_netns_link_update_prog,
141 .fill_link_info = bpf_netns_link_fill_info,
142 .show_fdinfo = bpf_netns_link_show_fdinfo,
143};
144
145int netns_bpf_prog_query(const union bpf_attr *attr,
146 union bpf_attr __user *uattr)
147{
148 __u32 __user *prog_ids = u64_to_user_ptr(attr->query.prog_ids);
149 u32 prog_id, prog_cnt = 0, flags = 0;
150 enum netns_bpf_attach_type type;
151 struct bpf_prog *attached;
152 struct net *net;
153
154 if (attr->query.query_flags)
155 return -EINVAL;
156
157 type = to_netns_bpf_attach_type(attr->query.attach_type);
158 if (type < 0)
159 return -EINVAL;
160
161 net = get_net_ns_by_fd(attr->query.target_fd);
162 if (IS_ERR(net))
163 return PTR_ERR(net);
164
165 rcu_read_lock();
166 attached = rcu_dereference(net->bpf.progs[type]);
167 if (attached) {
168 prog_cnt = 1;
169 prog_id = attached->aux->id;
170 }
171 rcu_read_unlock();
172
173 put_net(net);
174
175 if (copy_to_user(&uattr->query.attach_flags, &flags, sizeof(flags)))
176 return -EFAULT;
177 if (copy_to_user(&uattr->query.prog_cnt, &prog_cnt, sizeof(prog_cnt)))
178 return -EFAULT;
179
180 if (!attr->query.prog_cnt || !prog_ids || !prog_cnt)
181 return 0;
182
183 if (copy_to_user(prog_ids, &prog_id, sizeof(u32)))
184 return -EFAULT;
185
186 return 0;
187}
188
189int netns_bpf_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog)
190{
191 enum netns_bpf_attach_type type;
192 struct net *net;
193 int ret;
194
195 type = to_netns_bpf_attach_type(attr->attach_type);
196 if (type < 0)
197 return -EINVAL;
198
199 net = current->nsproxy->net_ns;
200 mutex_lock(&netns_bpf_mutex);
201
202 /* Attaching prog directly is not compatible with links */
203 if (net->bpf.links[type]) {
204 ret = -EEXIST;
205 goto out_unlock;
206 }
207
208 switch (type) {
209 case NETNS_BPF_FLOW_DISSECTOR:
210 ret = flow_dissector_bpf_prog_attach(net, prog);
211 break;
212 default:
213 ret = -EINVAL;
214 break;
215 }
216out_unlock:
217 mutex_unlock(&netns_bpf_mutex);
218
219 return ret;
220}
221
222/* Must be called with netns_bpf_mutex held. */
223static int __netns_bpf_prog_detach(struct net *net,
224 enum netns_bpf_attach_type type)
225{
226 struct bpf_prog *attached;
227
228 /* Progs attached via links cannot be detached */
229 if (net->bpf.links[type])
230 return -EINVAL;
231
232 attached = rcu_dereference_protected(net->bpf.progs[type],
233 lockdep_is_held(&netns_bpf_mutex));
234 if (!attached)
235 return -ENOENT;
236 RCU_INIT_POINTER(net->bpf.progs[type], NULL);
237 bpf_prog_put(attached);
238 return 0;
239}
240
241int netns_bpf_prog_detach(const union bpf_attr *attr)
242{
243 enum netns_bpf_attach_type type;
244 int ret;
245
246 type = to_netns_bpf_attach_type(attr->attach_type);
247 if (type < 0)
248 return -EINVAL;
249
250 mutex_lock(&netns_bpf_mutex);
251 ret = __netns_bpf_prog_detach(current->nsproxy->net_ns, type);
252 mutex_unlock(&netns_bpf_mutex);
253
254 return ret;
255}
256
257static int netns_bpf_link_attach(struct net *net, struct bpf_link *link,
258 enum netns_bpf_attach_type type)
259{
260 struct bpf_prog *prog;
261 int err;
262
263 mutex_lock(&netns_bpf_mutex);
264
265 /* Allow attaching only one prog or link for now */
266 if (net->bpf.links[type]) {
267 err = -E2BIG;
268 goto out_unlock;
269 }
270 /* Links are not compatible with attaching prog directly */
271 prog = rcu_dereference_protected(net->bpf.progs[type],
272 lockdep_is_held(&netns_bpf_mutex));
273 if (prog) {
274 err = -EEXIST;
275 goto out_unlock;
276 }
277
278 switch (type) {
279 case NETNS_BPF_FLOW_DISSECTOR:
280 err = flow_dissector_bpf_prog_attach(net, link->prog);
281 break;
282 default:
283 err = -EINVAL;
284 break;
285 }
286 if (err)
287 goto out_unlock;
288
289 net->bpf.links[type] = link;
290
291out_unlock:
292 mutex_unlock(&netns_bpf_mutex);
293 return err;
294}
295
296int netns_bpf_link_create(const union bpf_attr *attr, struct bpf_prog *prog)
297{
298 enum netns_bpf_attach_type netns_type;
299 struct bpf_link_primer link_primer;
300 struct bpf_netns_link *net_link;
301 enum bpf_attach_type type;
302 struct net *net;
303 int err;
304
305 if (attr->link_create.flags)
306 return -EINVAL;
307
308 type = attr->link_create.attach_type;
309 netns_type = to_netns_bpf_attach_type(type);
310 if (netns_type < 0)
311 return -EINVAL;
312
313 net = get_net_ns_by_fd(attr->link_create.target_fd);
314 if (IS_ERR(net))
315 return PTR_ERR(net);
316
317 net_link = kzalloc(sizeof(*net_link), GFP_USER);
318 if (!net_link) {
319 err = -ENOMEM;
320 goto out_put_net;
321 }
322 bpf_link_init(&net_link->link, BPF_LINK_TYPE_NETNS,
323 &bpf_netns_link_ops, prog);
324 net_link->net = net;
325 net_link->type = type;
326 net_link->netns_type = netns_type;
327
328 err = bpf_link_prime(&net_link->link, &link_primer);
329 if (err) {
330 kfree(net_link);
331 goto out_put_net;
332 }
333
334 err = netns_bpf_link_attach(net, &net_link->link, netns_type);
335 if (err) {
336 bpf_link_cleanup(&link_primer);
337 goto out_put_net;
338 }
339
340 put_net(net);
341 return bpf_link_settle(&link_primer);
342
343out_put_net:
344 put_net(net);
345 return err;
346}
347
348static void __net_exit netns_bpf_pernet_pre_exit(struct net *net)
349{
350 enum netns_bpf_attach_type type;
351 struct bpf_link *link;
352
353 mutex_lock(&netns_bpf_mutex);
354 for (type = 0; type < MAX_NETNS_BPF_ATTACH_TYPE; type++) {
355 link = net->bpf.links[type];
356 if (link)
357 bpf_netns_link_auto_detach(link);
358 else
359 __netns_bpf_prog_detach(net, type);
360 }
361 mutex_unlock(&netns_bpf_mutex);
362}
363
364static struct pernet_operations netns_bpf_pernet_ops __net_initdata = {
365 .pre_exit = netns_bpf_pernet_pre_exit,
366};
367
368static int __init netns_bpf_init(void)
369{
370 return register_pernet_subsys(&netns_bpf_pernet_ops);
371}
372
373subsys_initcall(netns_bpf_init);
kernel/bpf/queue_stack_maps.c+2−2
@@ -19,7 +19,7 @@ struct bpf_queue_stack { · 18 unmodified lines
19 u32 head, tail;19 u32 head, tail;
20 u32 size; /* max_entries + 1 */20 u32 size; /* max_entries + 1 */
21 21
22 char elements[0] __aligned(8);22 char elements[] __aligned(8);
23};23};
24 24
25static struct bpf_queue_stack *bpf_queue_stack(struct bpf_map *map)25static struct bpf_queue_stack *bpf_queue_stack(struct bpf_map *map)
@@ -45,7 +45,7 @@ static bool queue_stack_map_is_full(struct bpf_queue_stack *qs) · 19 unmodified lines
45/* Called from syscall */45/* Called from syscall */
46static int queue_stack_map_alloc_check(union bpf_attr *attr)46static int queue_stack_map_alloc_check(union bpf_attr *attr)
47{47{
48 if (!capable(CAP_SYS_ADMIN))48 if (!bpf_capable())
49 return -EPERM;49 return -EPERM;
50 50
51 /* check sanity of attributes */51 /* check sanity of attributes */
kernel/bpf/reuseport_array.c+1−1
@@ -154,7 +154,7 @@ static struct bpf_map *reuseport_array_alloc(union bpf_attr *attr) · 153 unmodified lines
154 struct bpf_map_memory mem;154 struct bpf_map_memory mem;
155 u64 array_size;155 u64 array_size;
156 156
157 if (!capable(CAP_SYS_ADMIN))157 if (!bpf_capable())
158 return ERR_PTR(-EPERM);158 return ERR_PTR(-EPERM);
159 159
160 array_size = sizeof(*array);160 array_size = sizeof(*array);
kernel/bpf/ringbuf.c+501−0
@@ -0,0 +1,501 @@
1#include <linux/bpf.h>
2#include <linux/btf.h>
3#include <linux/err.h>
4#include <linux/irq_work.h>
5#include <linux/slab.h>
6#include <linux/filter.h>
7#include <linux/mm.h>
8#include <linux/vmalloc.h>
9#include <linux/wait.h>
10#include <linux/poll.h>
11#include <uapi/linux/btf.h>
12
13#define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)
14
15/* non-mmap()'able part of bpf_ringbuf (everything up to consumer page) */
16#define RINGBUF_PGOFF \
17 (offsetof(struct bpf_ringbuf, consumer_pos) >> PAGE_SHIFT)
18/* consumer page and producer page */
19#define RINGBUF_POS_PAGES 2
20
21#define RINGBUF_MAX_RECORD_SZ (UINT_MAX/4)
22
23/* Maximum size of ring buffer area is limited by 32-bit page offset within
24 * record header, counted in pages. Reserve 8 bits for extensibility, and take
25 * into account few extra pages for consumer/producer pages and
26 * non-mmap()'able parts. This gives 64GB limit, which seems plenty for single
27 * ring buffer.
28 */
29#define RINGBUF_MAX_DATA_SZ \
30 (((1ULL << 24) - RINGBUF_POS_PAGES - RINGBUF_PGOFF) * PAGE_SIZE)
31
32struct bpf_ringbuf {
33 wait_queue_head_t waitq;
34 struct irq_work work;
35 u64 mask;
36 struct page **pages;
37 int nr_pages;
38 spinlock_t spinlock ____cacheline_aligned_in_smp;
39 /* Consumer and producer counters are put into separate pages to allow
40 * mapping consumer page as r/w, but restrict producer page to r/o.
41 * This protects producer position from being modified by user-space
42 * application and ruining in-kernel position tracking.
43 */
44 unsigned long consumer_pos __aligned(PAGE_SIZE);
45 unsigned long producer_pos __aligned(PAGE_SIZE);
46 char data[] __aligned(PAGE_SIZE);
47};
48
49struct bpf_ringbuf_map {
50 struct bpf_map map;
51 struct bpf_map_memory memory;
52 struct bpf_ringbuf *rb;
53};
54
55/* 8-byte ring buffer record header structure */
56struct bpf_ringbuf_hdr {
57 u32 len;
58 u32 pg_off;
59};
60
61static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node)
62{
63 const gfp_t flags = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN |
64 __GFP_ZERO;
65 int nr_meta_pages = RINGBUF_PGOFF + RINGBUF_POS_PAGES;
66 int nr_data_pages = data_sz >> PAGE_SHIFT;
67 int nr_pages = nr_meta_pages + nr_data_pages;
68 struct page **pages, *page;
69 struct bpf_ringbuf *rb;
70 size_t array_size;
71 int i;
72
73 /* Each data page is mapped twice to allow "virtual"
74 * continuous read of samples wrapping around the end of ring
75 * buffer area:
76 * ------------------------------------------------------
77 * | meta pages | real data pages | same data pages |
78 * ------------------------------------------------------
79 * | | 1 2 3 4 5 6 7 8 9 | 1 2 3 4 5 6 7 8 9 |
80 * ------------------------------------------------------
81 * | | TA DA | TA DA |
82 * ------------------------------------------------------
83 * ^^^^^^^
84 * |
85 * Here, no need to worry about special handling of wrapped-around
86 * data due to double-mapped data pages. This works both in kernel and
87 * when mmap()'ed in user-space, simplifying both kernel and
88 * user-space implementations significantly.
89 */
90 array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages);
91 if (array_size > PAGE_SIZE)
92 pages = vmalloc_node(array_size, numa_node);
93 else
94 pages = kmalloc_node(array_size, flags, numa_node);
95 if (!pages)
96 return NULL;
97
98 for (i = 0; i < nr_pages; i++) {
99 page = alloc_pages_node(numa_node, flags, 0);
100 if (!page) {
101 nr_pages = i;
102 goto err_free_pages;
103 }
104 pages[i] = page;
105 if (i >= nr_meta_pages)
106 pages[nr_data_pages + i] = page;
107 }
108
109 rb = vmap(pages, nr_meta_pages + 2 * nr_data_pages,
110 VM_ALLOC | VM_USERMAP, PAGE_KERNEL);
111 if (rb) {
112 rb->pages = pages;
113 rb->nr_pages = nr_pages;
114 return rb;
115 }
116
117err_free_pages:
118 for (i = 0; i < nr_pages; i++)
119 __free_page(pages[i]);
120 kvfree(pages);
121 return NULL;
122}
123
124static void bpf_ringbuf_notify(struct irq_work *work)
125{
126 struct bpf_ringbuf *rb = container_of(work, struct bpf_ringbuf, work);
127
128 wake_up_all(&rb->waitq);
129}
130
131static struct bpf_ringbuf *bpf_ringbuf_alloc(size_t data_sz, int numa_node)
132{
133 struct bpf_ringbuf *rb;
134
135 if (!data_sz || !PAGE_ALIGNED(data_sz))
136 return ERR_PTR(-EINVAL);
137
138#ifdef CONFIG_64BIT
139 /* on 32-bit arch, it's impossible to overflow record's hdr->pgoff */
140 if (data_sz > RINGBUF_MAX_DATA_SZ)
141 return ERR_PTR(-E2BIG);
142#endif
143
144 rb = bpf_ringbuf_area_alloc(data_sz, numa_node);
145 if (!rb)
146 return ERR_PTR(-ENOMEM);
147
148 spin_lock_init(&rb->spinlock);
149 init_waitqueue_head(&rb->waitq);
150 init_irq_work(&rb->work, bpf_ringbuf_notify);
151
152 rb->mask = data_sz - 1;
153 rb->consumer_pos = 0;
154 rb->producer_pos = 0;
155
156 return rb;
157}
158
159static struct bpf_map *ringbuf_map_alloc(union bpf_attr *attr)
160{
161 struct bpf_ringbuf_map *rb_map;
162 u64 cost;
163 int err;
164
165 if (attr->map_flags & ~RINGBUF_CREATE_FLAG_MASK)
166 return ERR_PTR(-EINVAL);
167
168 if (attr->key_size || attr->value_size ||
169 attr->max_entries == 0 || !PAGE_ALIGNED(attr->max_entries))
170 return ERR_PTR(-EINVAL);
171
172 rb_map = kzalloc(sizeof(*rb_map), GFP_USER);
173 if (!rb_map)
174 return ERR_PTR(-ENOMEM);
175
176 bpf_map_init_from_attr(&rb_map->map, attr);
177
178 cost = sizeof(struct bpf_ringbuf_map) +
179 sizeof(struct bpf_ringbuf) +
180 attr->max_entries;
181 err = bpf_map_charge_init(&rb_map->map.memory, cost);
182 if (err)
183 goto err_free_map;
184
185 rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, rb_map->map.numa_node);
186 if (IS_ERR(rb_map->rb)) {
187 err = PTR_ERR(rb_map->rb);
188 goto err_uncharge;
189 }
190
191 return &rb_map->map;
192
193err_uncharge:
194 bpf_map_charge_finish(&rb_map->map.memory);
195err_free_map:
196 kfree(rb_map);
197 return ERR_PTR(err);
198}
199
200static void bpf_ringbuf_free(struct bpf_ringbuf *rb)
201{
202 /* copy pages pointer and nr_pages to local variable, as we are going
203 * to unmap rb itself with vunmap() below
204 */
205 struct page **pages = rb->pages;
206 int i, nr_pages = rb->nr_pages;
207
208 vunmap(rb);
209 for (i = 0; i < nr_pages; i++)
210 __free_page(pages[i]);
211 kvfree(pages);
212}
213
214static void ringbuf_map_free(struct bpf_map *map)
215{
216 struct bpf_ringbuf_map *rb_map;
217
218 /* at this point bpf_prog->aux->refcnt == 0 and this map->refcnt == 0,
219 * so the programs (can be more than one that used this map) were
220 * disconnected from events. Wait for outstanding critical sections in
221 * these programs to complete
222 */
223 synchronize_rcu();
224
225 rb_map = container_of(map, struct bpf_ringbuf_map, map);
226 bpf_ringbuf_free(rb_map->rb);
227 kfree(rb_map);
228}
229
230static void *ringbuf_map_lookup_elem(struct bpf_map *map, void *key)
231{
232 return ERR_PTR(-ENOTSUPP);
233}
234
235static int ringbuf_map_update_elem(struct bpf_map *map, void *key, void *value,
236 u64 flags)
237{
238 return -ENOTSUPP;
239}
240
241static int ringbuf_map_delete_elem(struct bpf_map *map, void *key)
242{
243 return -ENOTSUPP;
244}
245
246static int ringbuf_map_get_next_key(struct bpf_map *map, void *key,
247 void *next_key)
248{
249 return -ENOTSUPP;
250}
251
252static size_t bpf_ringbuf_mmap_page_cnt(const struct bpf_ringbuf *rb)
253{
254 size_t data_pages = (rb->mask + 1) >> PAGE_SHIFT;
255
256 /* consumer page + producer page + 2 x data pages */
257 return RINGBUF_POS_PAGES + 2 * data_pages;
258}
259
260static int ringbuf_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
261{
262 struct bpf_ringbuf_map *rb_map;
263 size_t mmap_sz;
264
265 rb_map = container_of(map, struct bpf_ringbuf_map, map);
266 mmap_sz = bpf_ringbuf_mmap_page_cnt(rb_map->rb) << PAGE_SHIFT;
267
268 if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) > mmap_sz)
269 return -EINVAL;
270
271 return remap_vmalloc_range(vma, rb_map->rb,
272 vma->vm_pgoff + RINGBUF_PGOFF);
273}
274
275static unsigned long ringbuf_avail_data_sz(struct bpf_ringbuf *rb)
276{
277 unsigned long cons_pos, prod_pos;
278
279 cons_pos = smp_load_acquire(&rb->consumer_pos);
280 prod_pos = smp_load_acquire(&rb->producer_pos);
281 return prod_pos - cons_pos;
282}
283
284static __poll_t ringbuf_map_poll(struct bpf_map *map, struct file *filp,
285 struct poll_table_struct *pts)
286{
287 struct bpf_ringbuf_map *rb_map;
288
289 rb_map = container_of(map, struct bpf_ringbuf_map, map);
290 poll_wait(filp, &rb_map->rb->waitq, pts);
291
292 if (ringbuf_avail_data_sz(rb_map->rb))
293 return EPOLLIN | EPOLLRDNORM;
294 return 0;
295}
296
297const struct bpf_map_ops ringbuf_map_ops = {
298 .map_alloc = ringbuf_map_alloc,
299 .map_free = ringbuf_map_free,
300 .map_mmap = ringbuf_map_mmap,
301 .map_poll = ringbuf_map_poll,
302 .map_lookup_elem = ringbuf_map_lookup_elem,
303 .map_update_elem = ringbuf_map_update_elem,
304 .map_delete_elem = ringbuf_map_delete_elem,
305 .map_get_next_key = ringbuf_map_get_next_key,
306};
307
308/* Given pointer to ring buffer record metadata and struct bpf_ringbuf itself,
309 * calculate offset from record metadata to ring buffer in pages, rounded
310 * down. This page offset is stored as part of record metadata and allows to
311 * restore struct bpf_ringbuf * from record pointer. This page offset is
312 * stored at offset 4 of record metadata header.
313 */
314static size_t bpf_ringbuf_rec_pg_off(struct bpf_ringbuf *rb,
315 struct bpf_ringbuf_hdr *hdr)
316{
317 return ((void *)hdr - (void *)rb) >> PAGE_SHIFT;
318}
319
320/* Given pointer to ring buffer record header, restore pointer to struct
321 * bpf_ringbuf itself by using page offset stored at offset 4
322 */
323static struct bpf_ringbuf *
324bpf_ringbuf_restore_from_rec(struct bpf_ringbuf_hdr *hdr)
325{
326 unsigned long addr = (unsigned long)(void *)hdr;
327 unsigned long off = (unsigned long)hdr->pg_off << PAGE_SHIFT;
328
329 return (void*)((addr & PAGE_MASK) - off);
330}
331
332static void *__bpf_ringbuf_reserve(struct bpf_ringbuf *rb, u64 size)
333{
334 unsigned long cons_pos, prod_pos, new_prod_pos, flags;
335 u32 len, pg_off;
336 struct bpf_ringbuf_hdr *hdr;
337
338 if (unlikely(size > RINGBUF_MAX_RECORD_SZ))
339 return NULL;
340
341 len = round_up(size + BPF_RINGBUF_HDR_SZ, 8);
342 cons_pos = smp_load_acquire(&rb->consumer_pos);
343
344 if (in_nmi()) {
345 if (!spin_trylock_irqsave(&rb->spinlock, flags))
346 return NULL;
347 } else {
348 spin_lock_irqsave(&rb->spinlock, flags);
349 }
350
351 prod_pos = rb->producer_pos;
352 new_prod_pos = prod_pos + len;
353
354 /* check for out of ringbuf space by ensuring producer position
355 * doesn't advance more than (ringbuf_size - 1) ahead
356 */
357 if (new_prod_pos - cons_pos > rb->mask) {
358 spin_unlock_irqrestore(&rb->spinlock, flags);
359 return NULL;
360 }
361
362 hdr = (void *)rb->data + (prod_pos & rb->mask);
363 pg_off = bpf_ringbuf_rec_pg_off(rb, hdr);
364 hdr->len = size | BPF_RINGBUF_BUSY_BIT;
365 hdr->pg_off = pg_off;
366
367 /* pairs with consumer's smp_load_acquire() */
368 smp_store_release(&rb->producer_pos, new_prod_pos);
369
370 spin_unlock_irqrestore(&rb->spinlock, flags);
371
372 return (void *)hdr + BPF_RINGBUF_HDR_SZ;
373}
374
375BPF_CALL_3(bpf_ringbuf_reserve, struct bpf_map *, map, u64, size, u64, flags)
376{
377 struct bpf_ringbuf_map *rb_map;
378
379 if (unlikely(flags))
380 return 0;
381
382 rb_map = container_of(map, struct bpf_ringbuf_map, map);
383 return (unsigned long)__bpf_ringbuf_reserve(rb_map->rb, size);
384}
385
386const struct bpf_func_proto bpf_ringbuf_reserve_proto = {
387 .func = bpf_ringbuf_reserve,
388 .ret_type = RET_PTR_TO_ALLOC_MEM_OR_NULL,
389 .arg1_type = ARG_CONST_MAP_PTR,
390 .arg2_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
391 .arg3_type = ARG_ANYTHING,
392};
393
394static void bpf_ringbuf_commit(void *sample, u64 flags, bool discard)
395{
396 unsigned long rec_pos, cons_pos;
397 struct bpf_ringbuf_hdr *hdr;
398 struct bpf_ringbuf *rb;
399 u32 new_len;
400
401 hdr = sample - BPF_RINGBUF_HDR_SZ;
402 rb = bpf_ringbuf_restore_from_rec(hdr);
403 new_len = hdr->len ^ BPF_RINGBUF_BUSY_BIT;
404 if (discard)
405 new_len |= BPF_RINGBUF_DISCARD_BIT;
406
407 /* update record header with correct final size prefix */
408 xchg(&hdr->len, new_len);
409
410 /* if consumer caught up and is waiting for our record, notify about
411 * new data availability
412 */
413 rec_pos = (void *)hdr - (void *)rb->data;
414 cons_pos = smp_load_acquire(&rb->consumer_pos) & rb->mask;
415
416 if (flags & BPF_RB_FORCE_WAKEUP)
417 irq_work_queue(&rb->work);
418 else if (cons_pos == rec_pos && !(flags & BPF_RB_NO_WAKEUP))
419 irq_work_queue(&rb->work);
420}
421
422BPF_CALL_2(bpf_ringbuf_submit, void *, sample, u64, flags)
423{
424 bpf_ringbuf_commit(sample, flags, false /* discard */);
425 return 0;
426}
427
428const struct bpf_func_proto bpf_ringbuf_submit_proto = {
429 .func = bpf_ringbuf_submit,
430 .ret_type = RET_VOID,
431 .arg1_type = ARG_PTR_TO_ALLOC_MEM,
432 .arg2_type = ARG_ANYTHING,
433};
434
435BPF_CALL_2(bpf_ringbuf_discard, void *, sample, u64, flags)
436{
437 bpf_ringbuf_commit(sample, flags, true /* discard */);
438 return 0;
439}
440
441const struct bpf_func_proto bpf_ringbuf_discard_proto = {
442 .func = bpf_ringbuf_discard,
443 .ret_type = RET_VOID,
444 .arg1_type = ARG_PTR_TO_ALLOC_MEM,
445 .arg2_type = ARG_ANYTHING,
446};
447
448BPF_CALL_4(bpf_ringbuf_output, struct bpf_map *, map, void *, data, u64, size,
449 u64, flags)
450{
451 struct bpf_ringbuf_map *rb_map;
452 void *rec;
453
454 if (unlikely(flags & ~(BPF_RB_NO_WAKEUP | BPF_RB_FORCE_WAKEUP)))
455 return -EINVAL;
456
457 rb_map = container_of(map, struct bpf_ringbuf_map, map);
458 rec = __bpf_ringbuf_reserve(rb_map->rb, size);
459 if (!rec)
460 return -EAGAIN;
461
462 memcpy(rec, data, size);
463 bpf_ringbuf_commit(rec, flags, false /* discard */);
464 return 0;
465}
466
467const struct bpf_func_proto bpf_ringbuf_output_proto = {
468 .func = bpf_ringbuf_output,
469 .ret_type = RET_INTEGER,
470 .arg1_type = ARG_CONST_MAP_PTR,
471 .arg2_type = ARG_PTR_TO_MEM,
472 .arg3_type = ARG_CONST_SIZE_OR_ZERO,
473 .arg4_type = ARG_ANYTHING,
474};
475
476BPF_CALL_2(bpf_ringbuf_query, struct bpf_map *, map, u64, flags)
477{
478 struct bpf_ringbuf *rb;
479
480 rb = container_of(map, struct bpf_ringbuf_map, map)->rb;
481
482 switch (flags) {
483 case BPF_RB_AVAIL_DATA:
484 return ringbuf_avail_data_sz(rb);
485 case BPF_RB_RING_SIZE:
486 return rb->mask + 1;
487 case BPF_RB_CONS_POS:
488 return smp_load_acquire(&rb->consumer_pos);
489 case BPF_RB_PROD_POS:
490 return smp_load_acquire(&rb->producer_pos);
491 default:
492 return 0;
493 }
494}
495
496const struct bpf_func_proto bpf_ringbuf_query_proto = {
497 .func = bpf_ringbuf_query,
498 .ret_type = RET_INTEGER,
499 .arg1_type = ARG_CONST_MAP_PTR,
500 .arg2_type = ARG_ANYTHING,
501};
kernel/bpf/stackmap.c+1−1
@@ -93,7 +93,7 @@ static struct bpf_map *stack_map_alloc(union bpf_attr *attr) · 92 unmodified lines
93 u64 cost, n_buckets;93 u64 cost, n_buckets;
94 int err;94 int err;
95 95
96 if (!capable(CAP_SYS_ADMIN))96 if (!bpf_capable())
97 return ERR_PTR(-EPERM);97 return ERR_PTR(-EPERM);
98 98
99 if (attr->map_flags & ~STACK_CREATE_FLAG_MASK)99 if (attr->map_flags & ~STACK_CREATE_FLAG_MASK)
kernel/bpf/syscall.c+503−94
@@ -27,6 +27,8 @@ · 26 unmodified lines
27#include <uapi/linux/btf.h>27#include <uapi/linux/btf.h>
28#include <asm/pgtable.h>28#include <asm/pgtable.h>
29#include <linux/bpf_lsm.h>29#include <linux/bpf_lsm.h>
30#include <linux/poll.h>
31#include <linux/bpf-netns.h>
30 32
31#define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \33#define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \
32 (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \34 (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \
@@ -43,6 +45,8 @@ static DEFINE_IDR(prog_idr); · 10 unmodified lines
43static DEFINE_SPINLOCK(prog_idr_lock);45static DEFINE_SPINLOCK(prog_idr_lock);
44static DEFINE_IDR(map_idr);46static DEFINE_IDR(map_idr);
45static DEFINE_SPINLOCK(map_idr_lock);47static DEFINE_SPINLOCK(map_idr_lock);
48static DEFINE_IDR(link_idr);
49static DEFINE_SPINLOCK(link_idr_lock);
46 50
47int sysctl_unprivileged_bpf_disabled __read_mostly;51int sysctl_unprivileged_bpf_disabled __read_mostly;
48 52
@@ -50,9 +54,11 @@ static const struct bpf_map_ops * const bpf_map_types[] = { · 1 unmodified line
50#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type)54#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type)
51#define BPF_MAP_TYPE(_id, _ops) \55#define BPF_MAP_TYPE(_id, _ops) \
52 [_id] = &_ops,56 [_id] = &_ops,
57#define BPF_LINK_TYPE(_id, _name)
53#include <linux/bpf_types.h>58#include <linux/bpf_types.h>
54#undef BPF_PROG_TYPE59#undef BPF_PROG_TYPE
55#undef BPF_MAP_TYPE60#undef BPF_MAP_TYPE
61#undef BPF_LINK_TYPE
56};62};
57 63
58/*64/*
@@ -661,6 +667,16 @@ out: · 602 unmodified lines
661 return err;667 return err;
662}668}
663 669
670static __poll_t bpf_map_poll(struct file *filp, struct poll_table_struct *pts)
671{
672 struct bpf_map *map = filp->private_data;
673
674 if (map->ops->map_poll)
675 return map->ops->map_poll(map, filp, pts);
676
677 return EPOLLERR;
678}
679
664const struct file_operations bpf_map_fops = {680const struct file_operations bpf_map_fops = {
665#ifdef CONFIG_PROC_FS681#ifdef CONFIG_PROC_FS
666 .show_fdinfo = bpf_map_show_fdinfo,682 .show_fdinfo = bpf_map_show_fdinfo,
@@ -669,6 +685,7 @@ const struct file_operations bpf_map_fops = { · 2 unmodified lines
669 .read = bpf_dummy_read,685 .read = bpf_dummy_read,
670 .write = bpf_dummy_write,686 .write = bpf_dummy_write,
671 .mmap = bpf_map_mmap,687 .mmap = bpf_map_mmap,
688 .poll = bpf_map_poll,
672};689};
673 690
674int bpf_map_new_fd(struct bpf_map *map, int flags)691int bpf_map_new_fd(struct bpf_map *map, int flags)
@@ -1386,7 +1403,7 @@ int generic_map_lookup_batch(struct bpf_map *map, · 711 unmodified lines
1386 1403
1387 buf = kmalloc(map->key_size + value_size, GFP_USER | __GFP_NOWARN);1404 buf = kmalloc(map->key_size + value_size, GFP_USER | __GFP_NOWARN);
1388 if (!buf) {1405 if (!buf) {
1389 kvfree(buf_prevkey);1406 kfree(buf_prevkey);
1390 return -ENOMEM;1407 return -ENOMEM;
1391 }1408 }
1392 1409
@@ -1471,7 +1488,8 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr) · 78 unmodified lines
1471 map = __bpf_map_get(f);1488 map = __bpf_map_get(f);
1472 if (IS_ERR(map))1489 if (IS_ERR(map))
1473 return PTR_ERR(map);1490 return PTR_ERR(map);
1474 if (!(map_get_sys_perms(map, f) & FMODE_CAN_WRITE)) {1491 if (!(map_get_sys_perms(map, f) & FMODE_CAN_READ) ||
1492 !(map_get_sys_perms(map, f) & FMODE_CAN_WRITE)) {
1475 err = -EPERM;1493 err = -EPERM;
1476 goto err_put;1494 goto err_put;
1477 }1495 }
@@ -1546,7 +1564,7 @@ static int map_freeze(const union bpf_attr *attr) · 68 unmodified lines
1546 err = -EBUSY;1564 err = -EBUSY;
1547 goto err_put;1565 goto err_put;
1548 }1566 }
1549 if (!capable(CAP_SYS_ADMIN)) {1567 if (!bpf_capable()) {
1550 err = -EPERM;1568 err = -EPERM;
1551 goto err_put;1569 goto err_put;
1552 }1570 }
@@ -1562,9 +1580,11 @@ static const struct bpf_prog_ops * const bpf_prog_types[] = { · 9 unmodified lines
1562#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \1580#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
1563 [_id] = & _name ## _prog_ops,1581 [_id] = & _name ## _prog_ops,
1564#define BPF_MAP_TYPE(_id, _ops)1582#define BPF_MAP_TYPE(_id, _ops)
1583#define BPF_LINK_TYPE(_id, _name)
1565#include <linux/bpf_types.h>1584#include <linux/bpf_types.h>
1566#undef BPF_PROG_TYPE1585#undef BPF_PROG_TYPE
1567#undef BPF_MAP_TYPE1586#undef BPF_MAP_TYPE
1587#undef BPF_LINK_TYPE
1568};1588};
1569 1589
1570static int find_prog_type(enum bpf_prog_type type, struct bpf_prog *prog)1590static int find_prog_type(enum bpf_prog_type type, struct bpf_prog *prog)
@@ -1986,6 +2006,10 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type, · 415 unmodified lines
1986 case BPF_CGROUP_INET6_BIND:2006 case BPF_CGROUP_INET6_BIND:
1987 case BPF_CGROUP_INET4_CONNECT:2007 case BPF_CGROUP_INET4_CONNECT:
1988 case BPF_CGROUP_INET6_CONNECT:2008 case BPF_CGROUP_INET6_CONNECT:
2009 case BPF_CGROUP_INET4_GETPEERNAME:
2010 case BPF_CGROUP_INET6_GETPEERNAME:
2011 case BPF_CGROUP_INET4_GETSOCKNAME:
2012 case BPF_CGROUP_INET6_GETSOCKNAME:
1989 case BPF_CGROUP_UDP4_SENDMSG:2013 case BPF_CGROUP_UDP4_SENDMSG:
1990 case BPF_CGROUP_UDP6_SENDMSG:2014 case BPF_CGROUP_UDP6_SENDMSG:
1991 case BPF_CGROUP_UDP4_RECVMSG:2015 case BPF_CGROUP_UDP4_RECVMSG:
@@ -2019,6 +2043,55 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type, · 27 unmodified lines
2019 }2043 }
2020}2044}
2021 2045
2046static bool is_net_admin_prog_type(enum bpf_prog_type prog_type)
2047{
2048 switch (prog_type) {
2049 case BPF_PROG_TYPE_SCHED_CLS:
2050 case BPF_PROG_TYPE_SCHED_ACT:
2051 case BPF_PROG_TYPE_XDP:
2052 case BPF_PROG_TYPE_LWT_IN:
2053 case BPF_PROG_TYPE_LWT_OUT:
2054 case BPF_PROG_TYPE_LWT_XMIT:
2055 case BPF_PROG_TYPE_LWT_SEG6LOCAL:
2056 case BPF_PROG_TYPE_SK_SKB:
2057 case BPF_PROG_TYPE_SK_MSG:
2058 case BPF_PROG_TYPE_LIRC_MODE2:
2059 case BPF_PROG_TYPE_FLOW_DISSECTOR:
2060 case BPF_PROG_TYPE_CGROUP_DEVICE:
2061 case BPF_PROG_TYPE_CGROUP_SOCK:
2062 case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
2063 case BPF_PROG_TYPE_CGROUP_SOCKOPT:
2064 case BPF_PROG_TYPE_CGROUP_SYSCTL:
2065 case BPF_PROG_TYPE_SOCK_OPS:
2066 case BPF_PROG_TYPE_EXT: /* extends any prog */
2067 return true;
2068 case BPF_PROG_TYPE_CGROUP_SKB:
2069 /* always unpriv */
2070 case BPF_PROG_TYPE_SK_REUSEPORT:
2071 /* equivalent to SOCKET_FILTER. need CAP_BPF only */
2072 default:
2073 return false;
2074 }
2075}
2076
2077static bool is_perfmon_prog_type(enum bpf_prog_type prog_type)
2078{
2079 switch (prog_type) {
2080 case BPF_PROG_TYPE_KPROBE:
2081 case BPF_PROG_TYPE_TRACEPOINT:
2082 case BPF_PROG_TYPE_PERF_EVENT:
2083 case BPF_PROG_TYPE_RAW_TRACEPOINT:
2084 case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE:
2085 case BPF_PROG_TYPE_TRACING:
2086 case BPF_PROG_TYPE_LSM:
2087 case BPF_PROG_TYPE_STRUCT_OPS: /* has access to struct sock */
2088 case BPF_PROG_TYPE_EXT: /* extends any prog */
2089 return true;
2090 default:
2091 return false;
2092 }
2093}
2094
2022/* last field in 'union bpf_attr' used by this command */2095/* last field in 'union bpf_attr' used by this command */
2023#define BPF_PROG_LOAD_LAST_FIELD attach_prog_fd2096#define BPF_PROG_LOAD_LAST_FIELD attach_prog_fd
2024 2097
@@ -2041,7 +2114,7 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) · 16 unmodified lines
2041 2114
2042 if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&2115 if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
2043 (attr->prog_flags & BPF_F_ANY_ALIGNMENT) &&2116 (attr->prog_flags & BPF_F_ANY_ALIGNMENT) &&
2044 !capable(CAP_SYS_ADMIN))2117 !bpf_capable())
2045 return -EPERM;2118 return -EPERM;
2046 2119
2047 /* copy eBPF program license from user space */2120 /* copy eBPF program license from user space */
@@ -2054,11 +2127,16 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) · 6 unmodified lines
2054 is_gpl = license_is_gpl_compatible(license);2127 is_gpl = license_is_gpl_compatible(license);
2055 2128
2056 if (attr->insn_cnt == 0 ||2129 if (attr->insn_cnt == 0 ||
2057 attr->insn_cnt > (capable(CAP_SYS_ADMIN) ? BPF_COMPLEXITY_LIMIT_INSNS : BPF_MAXINSNS))2130 attr->insn_cnt > (bpf_capable() ? BPF_COMPLEXITY_LIMIT_INSNS : BPF_MAXINSNS))
2058 return -E2BIG;2131 return -E2BIG;
2059 if (type != BPF_PROG_TYPE_SOCKET_FILTER &&2132 if (type != BPF_PROG_TYPE_SOCKET_FILTER &&
2060 type != BPF_PROG_TYPE_CGROUP_SKB &&2133 type != BPF_PROG_TYPE_CGROUP_SKB &&
2061 !capable(CAP_SYS_ADMIN))2134 !bpf_capable())
2135 return -EPERM;
2136
2137 if (is_net_admin_prog_type(type) && !capable(CAP_NET_ADMIN))
2138 return -EPERM;
2139 if (is_perfmon_prog_type(type) && !perfmon_capable())
2062 return -EPERM;2140 return -EPERM;
2063 2141
2064 bpf_prog_load_fixup_attach_type(attr);2142 bpf_prog_load_fixup_attach_type(attr);
@@ -2197,25 +2275,39 @@ static int bpf_obj_get(const union bpf_attr *attr) · 132 unmodified lines
2197 attr->file_flags);2275 attr->file_flags);
2198}2276}
2199 2277
2200void bpf_link_init(struct bpf_link *link, const struct bpf_link_ops *ops,2278void bpf_link_init(struct bpf_link *link, enum bpf_link_type type,
2201 struct bpf_prog *prog)2279 const struct bpf_link_ops *ops, struct bpf_prog *prog)
2202{2280{
2203 atomic64_set(&link->refcnt, 1);2281 atomic64_set(&link->refcnt, 1);
2282 link->type = type;
2283 link->id = 0;
2204 link->ops = ops;2284 link->ops = ops;
2205 link->prog = prog;2285 link->prog = prog;
2206}2286}
2207 2287
2288static void bpf_link_free_id(int id)
2289{
2290 if (!id)
2291 return;
2292
2293 spin_lock_bh(&link_idr_lock);
2294 idr_remove(&link_idr, id);
2295 spin_unlock_bh(&link_idr_lock);
2296}
2297
2208/* Clean up bpf_link and corresponding anon_inode file and FD. After2298/* Clean up bpf_link and corresponding anon_inode file and FD. After
2209 * anon_inode is created, bpf_link can't be just kfree()'d due to deferred2299 * anon_inode is created, bpf_link can't be just kfree()'d due to deferred
2210 * anon_inode's release() call. This helper manages marking bpf_link as2300 * anon_inode's release() call. This helper marksbpf_link as
2211 * defunct, releases anon_inode file and puts reserved FD.2301 * defunct, releases anon_inode file and puts reserved FD. bpf_prog's refcnt
2302 * is not decremented, it's the responsibility of a calling code that failed
2303 * to complete bpf_link initialization.
2212 */2304 */
2213void bpf_link_cleanup(struct bpf_link *link, struct file *link_file,2305void bpf_link_cleanup(struct bpf_link_primer *primer)
2214 int link_fd)
2215{2306{
2216 link->prog = NULL;2307 primer->link->prog = NULL;
2217 fput(link_file);2308 bpf_link_free_id(primer->id);
2218 put_unused_fd(link_fd);2309 fput(primer->file);
2310 put_unused_fd(primer->fd);
2219}2311}
2220 2312
2221void bpf_link_inc(struct bpf_link *link)2313void bpf_link_inc(struct bpf_link *link)
@@ -2226,6 +2318,7 @@ void bpf_link_inc(struct bpf_link *link) · 4 unmodified lines
2226/* bpf_link_free is guaranteed to be called from process context */2318/* bpf_link_free is guaranteed to be called from process context */
2227static void bpf_link_free(struct bpf_link *link)2319static void bpf_link_free(struct bpf_link *link)
2228{2320{
2321 bpf_link_free_id(link->id);
2229 if (link->prog) {2322 if (link->prog) {
2230 /* detach BPF program, clean up used resources */2323 /* detach BPF program, clean up used resources */
2231 link->ops->release(link);2324 link->ops->release(link);
@@ -2267,35 +2360,35 @@ static int bpf_link_release(struct inode *inode, struct file *filp) · 35 unmodified lines
2267}2360}
2268 2361
2269#ifdef CONFIG_PROC_FS2362#ifdef CONFIG_PROC_FS
2270static const struct bpf_link_ops bpf_raw_tp_lops;2363#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type)
2271static const struct bpf_link_ops bpf_tracing_link_lops;2364#define BPF_MAP_TYPE(_id, _ops)
2365#define BPF_LINK_TYPE(_id, _name) [_id] = #_name,
2366static const char *bpf_link_type_strs[] = {
2367 [BPF_LINK_TYPE_UNSPEC] = "<invalid>",
2368#include <linux/bpf_types.h>
2369};
2370#undef BPF_PROG_TYPE
2371#undef BPF_MAP_TYPE
2372#undef BPF_LINK_TYPE
2272 2373
2273static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp)2374static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp)
2274{2375{
2275 const struct bpf_link *link = filp->private_data;2376 const struct bpf_link *link = filp->private_data;
2276 const struct bpf_prog *prog = link->prog;2377 const struct bpf_prog *prog = link->prog;
2277 char prog_tag[sizeof(prog->tag) * 2 + 1] = { };2378 char prog_tag[sizeof(prog->tag) * 2 + 1] = { };
2278 const char *link_type;
2279
2280 if (link->ops == &bpf_raw_tp_lops)
2281 link_type = "raw_tracepoint";
2282 else if (link->ops == &bpf_tracing_link_lops)
2283 link_type = "tracing";
2284#ifdef CONFIG_CGROUP_BPF
2285 else if (link->ops == &bpf_cgroup_link_lops)
2286 link_type = "cgroup";
2287#endif
2288 else
2289 link_type = "unknown";
2290 2379
2291 bin2hex(prog_tag, prog->tag, sizeof(prog->tag));2380 bin2hex(prog_tag, prog->tag, sizeof(prog->tag));
2292 seq_printf(m,2381 seq_printf(m,
2293 "link_type:\t%s\n"2382 "link_type:\t%s\n"
2383 "link_id:\t%u\n"
2294 "prog_tag:\t%s\n"2384 "prog_tag:\t%s\n"
2295 "prog_id:\t%u\n",2385 "prog_id:\t%u\n",
2296 link_type,2386 bpf_link_type_strs[link->type],
2387 link->id,
2297 prog_tag,2388 prog_tag,
2298 prog->aux->id);2389 prog->aux->id);
2390 if (link->ops->show_fdinfo)
2391 link->ops->show_fdinfo(link, m);
2299}2392}
2300#endif2393#endif
2301 2394
@@ -2308,36 +2401,77 @@ static const struct file_operations bpf_link_fops = { · 6 unmodified lines
2308 .write = bpf_dummy_write,2401 .write = bpf_dummy_write,
2309};2402};
2310 2403
2311int bpf_link_new_fd(struct bpf_link *link)2404static int bpf_link_alloc_id(struct bpf_link *link)
2312{2405{
2313 return anon_inode_getfd("bpf-link", &bpf_link_fops, link, O_CLOEXEC);2406 int id;
2314}
2315 2407
2316/* Similar to bpf_link_new_fd, create anon_inode for given bpf_link, but2408 idr_preload(GFP_KERNEL);
2317 * instead of immediately installing fd in fdtable, just reserve it and2409 spin_lock_bh(&link_idr_lock);
2318 * return. Caller then need to either install it with fd_install(fd, file) or2410 id = idr_alloc_cyclic(&link_idr, link, 1, INT_MAX, GFP_ATOMIC);
2319 * release with put_unused_fd(fd).2411 spin_unlock_bh(&link_idr_lock);
2320 * This is useful for cases when bpf_link attachment/detachment are2412 idr_preload_end();
2321 * complicated and expensive operations and should be delayed until all the fd2413
2322 * reservation and anon_inode creation succeeds.2414 return id;
2415}
2416
2417/* Prepare bpf_link to be exposed to user-space by allocating anon_inode file,
2418 * reserving unused FD and allocating ID from link_idr. This is to be paired
2419 * with bpf_link_settle() to install FD and ID and expose bpf_link to
2420 * user-space, if bpf_link is successfully attached. If not, bpf_link and
2421 * pre-allocated resources are to be freed with bpf_cleanup() call. All the
2422 * transient state is passed around in struct bpf_link_primer.
2423 * This is preferred way to create and initialize bpf_link, especially when
2424 * there are complicated and expensive operations inbetween creating bpf_link
2425 * itself and attaching it to BPF hook. By using bpf_link_prime() and
2426 * bpf_link_settle() kernel code using bpf_link doesn't have to perform
2427 * expensive (and potentially failing) roll back operations in a rare case
2428 * that file, FD, or ID can't be allocated.
2323 */2429 */
2324struct file *bpf_link_new_file(struct bpf_link *link, int *reserved_fd)2430int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer)
2325{2431{
2326 struct file *file;2432 struct file *file;
2327 int fd;2433 int fd, id;
2328 2434
2329 fd = get_unused_fd_flags(O_CLOEXEC);2435 fd = get_unused_fd_flags(O_CLOEXEC);
2330 if (fd < 0)2436 if (fd < 0)
2331 return ERR_PTR(fd);2437 return fd;
2438
2439
2440 id = bpf_link_alloc_id(link);
2441 if (id < 0) {
2442 put_unused_fd(fd);
2443 return id;
2444 }
2332 2445
2333 file = anon_inode_getfile("bpf_link", &bpf_link_fops, link, O_CLOEXEC);2446 file = anon_inode_getfile("bpf_link", &bpf_link_fops, link, O_CLOEXEC);
2334 if (IS_ERR(file)) {2447 if (IS_ERR(file)) {
2448 bpf_link_free_id(id);
2335 put_unused_fd(fd);2449 put_unused_fd(fd);
2336 return file;2450 return PTR_ERR(file);
2337 }2451 }
2338 2452
2339 *reserved_fd = fd;2453 primer->link = link;
2340 return file;2454 primer->file = file;
2455 primer->fd = fd;
2456 primer->id = id;
2457 return 0;
2458}
2459
2460int bpf_link_settle(struct bpf_link_primer *primer)
2461{
2462 /* make bpf_link fetchable by ID */
2463 spin_lock_bh(&link_idr_lock);
2464 primer->link->id = primer->id;
2465 spin_unlock_bh(&link_idr_lock);
2466 /* make bpf_link fetchable by FD */
2467 fd_install(primer->fd, primer->file);
2468 /* pass through installed FD */
2469 return primer->fd;
2470}
2471
2472int bpf_link_new_fd(struct bpf_link *link)
2473{
2474 return anon_inode_getfd("bpf-link", &bpf_link_fops, link, O_CLOEXEC);
2341}2475}
2342 2476
2343struct bpf_link *bpf_link_get_from_fd(u32 ufd)2477struct bpf_link *bpf_link_get_from_fd(u32 ufd)
@@ -2361,6 +2495,7 @@ struct bpf_link *bpf_link_get_from_fd(u32 ufd) · 17 unmodified lines
2361 2495
2362struct bpf_tracing_link {2496struct bpf_tracing_link {
2363 struct bpf_link link;2497 struct bpf_link link;
2498 enum bpf_attach_type attach_type;
2364};2499};
2365 2500
2366static void bpf_tracing_link_release(struct bpf_link *link)2501static void bpf_tracing_link_release(struct bpf_link *link)
@@ -2376,16 +2511,40 @@ static void bpf_tracing_link_dealloc(struct bpf_link *link) · 9 unmodified lines
2376 kfree(tr_link);2511 kfree(tr_link);
2377}2512}
2378 2513
2514static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link,
2515 struct seq_file *seq)
2516{
2517 struct bpf_tracing_link *tr_link =
2518 container_of(link, struct bpf_tracing_link, link);
2519
2520 seq_printf(seq,
2521 "attach_type:\t%d\n",
2522 tr_link->attach_type);
2523}
2524
2525static int bpf_tracing_link_fill_link_info(const struct bpf_link *link,
2526 struct bpf_link_info *info)
2527{
2528 struct bpf_tracing_link *tr_link =
2529 container_of(link, struct bpf_tracing_link, link);
2530
2531 info->tracing.attach_type = tr_link->attach_type;
2532
2533 return 0;
2534}
2535
2379static const struct bpf_link_ops bpf_tracing_link_lops = {2536static const struct bpf_link_ops bpf_tracing_link_lops = {
2380 .release = bpf_tracing_link_release,2537 .release = bpf_tracing_link_release,
2381 .dealloc = bpf_tracing_link_dealloc,2538 .dealloc = bpf_tracing_link_dealloc,
2539 .show_fdinfo = bpf_tracing_link_show_fdinfo,
2540 .fill_link_info = bpf_tracing_link_fill_link_info,
2382};2541};
2383 2542
2384static int bpf_tracing_prog_attach(struct bpf_prog *prog)2543static int bpf_tracing_prog_attach(struct bpf_prog *prog)
2385{2544{
2545 struct bpf_link_primer link_primer;
2386 struct bpf_tracing_link *link;2546 struct bpf_tracing_link *link;
2387 struct file *link_file;2547 int err;
2388 int link_fd, err;
2389 2548
2390 switch (prog->type) {2549 switch (prog->type) {
2391 case BPF_PROG_TYPE_TRACING:2550 case BPF_PROG_TYPE_TRACING:
@@ -2418,24 +2577,23 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog) · 26 unmodified lines
2418 err = -ENOMEM;2577 err = -ENOMEM;
2419 goto out_put_prog;2578 goto out_put_prog;
2420 }2579 }
2421 bpf_link_init(&link->link, &bpf_tracing_link_lops, prog);2580 bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING,
2581 &bpf_tracing_link_lops, prog);
2582 link->attach_type = prog->expected_attach_type;
2422 2583
2423 link_file = bpf_link_new_file(&link->link, &link_fd);2584 err = bpf_link_prime(&link->link, &link_primer);
2424 if (IS_ERR(link_file)) {2585 if (err) {
2425 kfree(link);2586 kfree(link);
2426 err = PTR_ERR(link_file);
2427 goto out_put_prog;2587 goto out_put_prog;
2428 }2588 }
2429 2589
2430 err = bpf_trampoline_link_prog(prog);2590 err = bpf_trampoline_link_prog(prog);
2431 if (err) {2591 if (err) {
2432 bpf_link_cleanup(&link->link, link_file, link_fd);2592 bpf_link_cleanup(&link_primer);
2433 goto out_put_prog;2593 goto out_put_prog;
2434 }2594 }
2435 2595
2436 fd_install(link_fd, link_file);2596 return bpf_link_settle(&link_primer);
2437 return link_fd;
2438
2439out_put_prog:2597out_put_prog:
2440 bpf_prog_put(prog);2598 bpf_prog_put(prog);
2441 return err;2599 return err;
@@ -2463,22 +2621,69 @@ static void bpf_raw_tp_link_dealloc(struct bpf_link *link) · 21 unmodified lines
2463 kfree(raw_tp);2621 kfree(raw_tp);
2464}2622}
2465 2623
2466static const struct bpf_link_ops bpf_raw_tp_lops = {2624static void bpf_raw_tp_link_show_fdinfo(const struct bpf_link *link,
2625 struct seq_file *seq)
2626{
2627 struct bpf_raw_tp_link *raw_tp_link =
2628 container_of(link, struct bpf_raw_tp_link, link);
2629
2630 seq_printf(seq,
2631 "tp_name:\t%s\n",
2632 raw_tp_link->btp->tp->name);
2633}
2634
2635static int bpf_raw_tp_link_fill_link_info(const struct bpf_link *link,
2636 struct bpf_link_info *info)
2637{
2638 struct bpf_raw_tp_link *raw_tp_link =
2639 container_of(link, struct bpf_raw_tp_link, link);
2640 char __user *ubuf = u64_to_user_ptr(info->raw_tracepoint.tp_name);
2641 const char *tp_name = raw_tp_link->btp->tp->name;
2642 u32 ulen = info->raw_tracepoint.tp_name_len;
2643 size_t tp_len = strlen(tp_name);
2644
2645 if (ulen && !ubuf)
2646 return -EINVAL;
2647
2648 info->raw_tracepoint.tp_name_len = tp_len + 1;
2649
2650 if (!ubuf)
2651 return 0;
2652
2653 if (ulen >= tp_len + 1) {
2654 if (copy_to_user(ubuf, tp_name, tp_len + 1))
2655 return -EFAULT;
2656 } else {
2657 char zero = '\0';
2658
2659 if (copy_to_user(ubuf, tp_name, ulen - 1))
2660 return -EFAULT;
2661 if (put_user(zero, ubuf + ulen - 1))
2662 return -EFAULT;
2663 return -ENOSPC;
2664 }
2665
2666 return 0;
2667}
2668
2669static const struct bpf_link_ops bpf_raw_tp_link_lops = {
2467 .release = bpf_raw_tp_link_release,2670 .release = bpf_raw_tp_link_release,
2468 .dealloc = bpf_raw_tp_link_dealloc,2671 .dealloc = bpf_raw_tp_link_dealloc,
2672 .show_fdinfo = bpf_raw_tp_link_show_fdinfo,
2673 .fill_link_info = bpf_raw_tp_link_fill_link_info,
2469};2674};
2470 2675
2471#define BPF_RAW_TRACEPOINT_OPEN_LAST_FIELD raw_tracepoint.prog_fd2676#define BPF_RAW_TRACEPOINT_OPEN_LAST_FIELD raw_tracepoint.prog_fd
2472 2677
2473static int bpf_raw_tracepoint_open(const union bpf_attr *attr)2678static int bpf_raw_tracepoint_open(const union bpf_attr *attr)
2474{2679{
2680 struct bpf_link_primer link_primer;
2475 struct bpf_raw_tp_link *link;2681 struct bpf_raw_tp_link *link;
2476 struct bpf_raw_event_map *btp;2682 struct bpf_raw_event_map *btp;
2477 struct file *link_file;
2478 struct bpf_prog *prog;2683 struct bpf_prog *prog;
2479 const char *tp_name;2684 const char *tp_name;
2480 char buf[128];2685 char buf[128];
2481 int link_fd, err;2686 int err;
2482 2687
2483 if (CHECK_ATTR(BPF_RAW_TRACEPOINT_OPEN))2688 if (CHECK_ATTR(BPF_RAW_TRACEPOINT_OPEN))
2484 return -EINVAL;2689 return -EINVAL;
@@ -2531,24 +2736,23 @@ static int bpf_raw_tracepoint_open(const union bpf_attr *attr) · 46 unmodified lines
2531 err = -ENOMEM;2736 err = -ENOMEM;
2532 goto out_put_btp;2737 goto out_put_btp;
2533 }2738 }
2534 bpf_link_init(&link->link, &bpf_raw_tp_lops, prog);2739 bpf_link_init(&link->link, BPF_LINK_TYPE_RAW_TRACEPOINT,
2740 &bpf_raw_tp_link_lops, prog);
2535 link->btp = btp;2741 link->btp = btp;
2536 2742
2537 link_file = bpf_link_new_file(&link->link, &link_fd);2743 err = bpf_link_prime(&link->link, &link_primer);
2538 if (IS_ERR(link_file)) {2744 if (err) {
2539 kfree(link);2745 kfree(link);
2540 err = PTR_ERR(link_file);
2541 goto out_put_btp;2746 goto out_put_btp;
2542 }2747 }
2543 2748
2544 err = bpf_probe_register(link->btp, prog);2749 err = bpf_probe_register(link->btp, prog);
2545 if (err) {2750 if (err) {
2546 bpf_link_cleanup(&link->link, link_file, link_fd);2751 bpf_link_cleanup(&link_primer);
2547 goto out_put_btp;2752 goto out_put_btp;
2548 }2753 }
2549 2754
2550 fd_install(link_fd, link_file);2755 return bpf_link_settle(&link_primer);
2551 return link_fd;
2552 2756
2553out_put_btp:2757out_put_btp:
2554 bpf_put_raw_tracepoint(btp);2758 bpf_put_raw_tracepoint(btp);
@@ -2566,6 +2770,11 @@ static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog, · 11 unmodified lines
2566 case BPF_PROG_TYPE_CGROUP_SOCKOPT:2770 case BPF_PROG_TYPE_CGROUP_SOCKOPT:
2567 return attach_type == prog->expected_attach_type ? 0 : -EINVAL;2771 return attach_type == prog->expected_attach_type ? 0 : -EINVAL;
2568 case BPF_PROG_TYPE_CGROUP_SKB:2772 case BPF_PROG_TYPE_CGROUP_SKB:
2773 if (!capable(CAP_NET_ADMIN))
2774 /* cg-skb progs can be loaded by unpriv user.
2775 * check permissions at attach time.
2776 */
2777 return -EPERM;
2569 return prog->enforce_expected_attach_type &&2778 return prog->enforce_expected_attach_type &&
2570 prog->expected_attach_type != attach_type ?2779 prog->expected_attach_type != attach_type ?
2571 -EINVAL : 0;2780 -EINVAL : 0;
@@ -2590,6 +2799,10 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) · 18 unmodified lines
2590 case BPF_CGROUP_INET6_BIND:2799 case BPF_CGROUP_INET6_BIND:
2591 case BPF_CGROUP_INET4_CONNECT:2800 case BPF_CGROUP_INET4_CONNECT:
2592 case BPF_CGROUP_INET6_CONNECT:2801 case BPF_CGROUP_INET6_CONNECT:
2802 case BPF_CGROUP_INET4_GETPEERNAME:
2803 case BPF_CGROUP_INET6_GETPEERNAME:
2804 case BPF_CGROUP_INET4_GETSOCKNAME:
2805 case BPF_CGROUP_INET6_GETSOCKNAME:
2593 case BPF_CGROUP_UDP4_SENDMSG:2806 case BPF_CGROUP_UDP4_SENDMSG:
2594 case BPF_CGROUP_UDP6_SENDMSG:2807 case BPF_CGROUP_UDP6_SENDMSG:
2595 case BPF_CGROUP_UDP4_RECVMSG:2808 case BPF_CGROUP_UDP4_RECVMSG:
@@ -2613,6 +2826,8 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) · 17 unmodified lines
2613 case BPF_CGROUP_GETSOCKOPT:2826 case BPF_CGROUP_GETSOCKOPT:
2614 case BPF_CGROUP_SETSOCKOPT:2827 case BPF_CGROUP_SETSOCKOPT:
2615 return BPF_PROG_TYPE_CGROUP_SOCKOPT;2828 return BPF_PROG_TYPE_CGROUP_SOCKOPT;
2829 case BPF_TRACE_ITER:
2830 return BPF_PROG_TYPE_TRACING;
2616 default:2831 default:
2617 return BPF_PROG_TYPE_UNSPEC;2832 return BPF_PROG_TYPE_UNSPEC;
2618 }2833 }
@@ -2629,9 +2844,6 @@ static int bpf_prog_attach(const union bpf_attr *attr) · 10 unmodified lines
2629 struct bpf_prog *prog;2844 struct bpf_prog *prog;
2630 int ret;2845 int ret;
2631 2846
2632 if (!capable(CAP_NET_ADMIN))
2633 return -EPERM;
2634
2635 if (CHECK_ATTR(BPF_PROG_ATTACH))2847 if (CHECK_ATTR(BPF_PROG_ATTACH))
2636 return -EINVAL;2848 return -EINVAL;
2637 2849
@@ -2660,7 +2872,7 @@ static int bpf_prog_attach(const union bpf_attr *attr) · 22 unmodified lines
2660 ret = lirc_prog_attach(attr, prog);2872 ret = lirc_prog_attach(attr, prog);
2661 break;2873 break;
2662 case BPF_PROG_TYPE_FLOW_DISSECTOR:2874 case BPF_PROG_TYPE_FLOW_DISSECTOR:
2663 ret = skb_flow_dissector_bpf_prog_attach(attr, prog);2875 ret = netns_bpf_prog_attach(attr, prog);
2664 break;2876 break;
2665 case BPF_PROG_TYPE_CGROUP_DEVICE:2877 case BPF_PROG_TYPE_CGROUP_DEVICE:
2666 case BPF_PROG_TYPE_CGROUP_SKB:2878 case BPF_PROG_TYPE_CGROUP_SKB:
@@ -2686,9 +2898,6 @@ static int bpf_prog_detach(const union bpf_attr *attr) · 19 unmodified lines
2686{2898{
2687 enum bpf_prog_type ptype;2899 enum bpf_prog_type ptype;
2688 2900
2689 if (!capable(CAP_NET_ADMIN))
2690 return -EPERM;
2691
2692 if (CHECK_ATTR(BPF_PROG_DETACH))2901 if (CHECK_ATTR(BPF_PROG_DETACH))
2693 return -EINVAL;2902 return -EINVAL;
2694 2903
@@ -2701,7 +2910,9 @@ static int bpf_prog_detach(const union bpf_attr *attr) · 6 unmodified lines
2701 case BPF_PROG_TYPE_LIRC_MODE2:2910 case BPF_PROG_TYPE_LIRC_MODE2:
2702 return lirc_prog_detach(attr);2911 return lirc_prog_detach(attr);
2703 case BPF_PROG_TYPE_FLOW_DISSECTOR:2912 case BPF_PROG_TYPE_FLOW_DISSECTOR:
2704 return skb_flow_dissector_bpf_prog_detach(attr);2913 if (!capable(CAP_NET_ADMIN))
2914 return -EPERM;
2915 return netns_bpf_prog_detach(attr);
2705 case BPF_PROG_TYPE_CGROUP_DEVICE:2916 case BPF_PROG_TYPE_CGROUP_DEVICE:
2706 case BPF_PROG_TYPE_CGROUP_SKB:2917 case BPF_PROG_TYPE_CGROUP_SKB:
2707 case BPF_PROG_TYPE_CGROUP_SOCK:2918 case BPF_PROG_TYPE_CGROUP_SOCK:
@@ -2737,6 +2948,10 @@ static int bpf_prog_query(const union bpf_attr *attr, · 29 unmodified lines
2737 case BPF_CGROUP_INET6_POST_BIND:2948 case BPF_CGROUP_INET6_POST_BIND:
2738 case BPF_CGROUP_INET4_CONNECT:2949 case BPF_CGROUP_INET4_CONNECT:
2739 case BPF_CGROUP_INET6_CONNECT:2950 case BPF_CGROUP_INET6_CONNECT:
2951 case BPF_CGROUP_INET4_GETPEERNAME:
2952 case BPF_CGROUP_INET6_GETPEERNAME:
2953 case BPF_CGROUP_INET4_GETSOCKNAME:
2954 case BPF_CGROUP_INET6_GETSOCKNAME:
2740 case BPF_CGROUP_UDP4_SENDMSG:2955 case BPF_CGROUP_UDP4_SENDMSG:
2741 case BPF_CGROUP_UDP6_SENDMSG:2956 case BPF_CGROUP_UDP6_SENDMSG:
2742 case BPF_CGROUP_UDP4_RECVMSG:2957 case BPF_CGROUP_UDP4_RECVMSG:
@@ -2750,7 +2965,7 @@ static int bpf_prog_query(const union bpf_attr *attr, · 7 unmodified lines
2750 case BPF_LIRC_MODE2:2965 case BPF_LIRC_MODE2:
2751 return lirc_prog_query(attr, uattr);2966 return lirc_prog_query(attr, uattr);
2752 case BPF_FLOW_DISSECTOR:2967 case BPF_FLOW_DISSECTOR:
2753 return skb_flow_dissector_prog_query(attr, uattr);2968 return netns_bpf_prog_query(attr, uattr);
2754 default:2969 default:
2755 return -EINVAL;2970 return -EINVAL;
2756 }2971 }
@@ -2764,8 +2979,6 @@ static int bpf_prog_test_run(const union bpf_attr *attr, · 7 unmodified lines
2764 struct bpf_prog *prog;2979 struct bpf_prog *prog;
2765 int ret = -ENOTSUPP;2980 int ret = -ENOTSUPP;
2766 2981
2767 if (!capable(CAP_SYS_ADMIN))
2768 return -EPERM;
2769 if (CHECK_ATTR(BPF_PROG_TEST_RUN))2982 if (CHECK_ATTR(BPF_PROG_TEST_RUN))
2770 return -EINVAL;2983 return -EINVAL;
2771 2984
@@ -2816,6 +3029,25 @@ static int bpf_obj_get_next_id(const union bpf_attr *attr, · 44 unmodified lines
2816 return err;3029 return err;
2817}3030}
2818 3031
3032struct bpf_map *bpf_map_get_curr_or_next(u32 *id)
3033{
3034 struct bpf_map *map;
3035
3036 spin_lock_bh(&map_idr_lock);
3037again:
3038 map = idr_get_next(&map_idr, id);
3039 if (map) {
3040 map = __bpf_map_inc_not_zero(map, false);
3041 if (IS_ERR(map)) {
3042 (*id)++;
3043 goto again;
3044 }
3045 }
3046 spin_unlock_bh(&map_idr_lock);
3047
3048 return map;
3049}
3050
2819#define BPF_PROG_GET_FD_BY_ID_LAST_FIELD prog_id3051#define BPF_PROG_GET_FD_BY_ID_LAST_FIELD prog_id
2820 3052
2821struct bpf_prog *bpf_prog_by_id(u32 id)3053struct bpf_prog *bpf_prog_by_id(u32 id)
@@ -3047,7 +3279,7 @@ static int bpf_prog_get_info_by_fd(struct bpf_prog *prog, · 225 unmodified lines
3047 info.run_time_ns = stats.nsecs;3279 info.run_time_ns = stats.nsecs;
3048 info.run_cnt = stats.cnt;3280 info.run_cnt = stats.cnt;
3049 3281
3050 if (!capable(CAP_SYS_ADMIN)) {3282 if (!bpf_capable()) {
3051 info.jited_prog_len = 0;3283 info.jited_prog_len = 0;
3052 info.xlated_prog_len = 0;3284 info.xlated_prog_len = 0;
3053 info.nr_jited_ksyms = 0;3285 info.nr_jited_ksyms = 0;
@@ -3329,6 +3561,42 @@ static int bpf_btf_get_info_by_fd(struct btf *btf, · 275 unmodified lines
3329 return btf_get_info_by_fd(btf, attr, uattr);3561 return btf_get_info_by_fd(btf, attr, uattr);
3330}3562}
3331 3563
3564static int bpf_link_get_info_by_fd(struct bpf_link *link,
3565 const union bpf_attr *attr,
3566 union bpf_attr __user *uattr)
3567{
3568 struct bpf_link_info __user *uinfo = u64_to_user_ptr(attr->info.info);
3569 struct bpf_link_info info;
3570 u32 info_len = attr->info.info_len;
3571 int err;
3572
3573 err = bpf_check_uarg_tail_zero(uinfo, sizeof(info), info_len);
3574 if (err)
3575 return err;
3576 info_len = min_t(u32, sizeof(info), info_len);
3577
3578 memset(&info, 0, sizeof(info));
3579 if (copy_from_user(&info, uinfo, info_len))
3580 return -EFAULT;
3581
3582 info.type = link->type;
3583 info.id = link->id;
3584 info.prog_id = link->prog->aux->id;
3585
3586 if (link->ops->fill_link_info) {
3587 err = link->ops->fill_link_info(link, &info);
3588 if (err)
3589 return err;
3590 }
3591
3592 if (copy_to_user(uinfo, &info, info_len) ||
3593 put_user(info_len, &uattr->info.info_len))
3594 return -EFAULT;
3595
3596 return 0;
3597}
3598
3599
3332#define BPF_OBJ_GET_INFO_BY_FD_LAST_FIELD info.info3600#define BPF_OBJ_GET_INFO_BY_FD_LAST_FIELD info.info
3333 3601
3334static int bpf_obj_get_info_by_fd(const union bpf_attr *attr,3602static int bpf_obj_get_info_by_fd(const union bpf_attr *attr,
@@ -3353,6 +3621,9 @@ static int bpf_obj_get_info_by_fd(const union bpf_attr *attr, · 18 unmodified lines
3353 uattr);3621 uattr);
3354 else if (f.file->f_op == &btf_fops)3622 else if (f.file->f_op == &btf_fops)
3355 err = bpf_btf_get_info_by_fd(f.file->private_data, attr, uattr);3623 err = bpf_btf_get_info_by_fd(f.file->private_data, attr, uattr);
3624 else if (f.file->f_op == &bpf_link_fops)
3625 err = bpf_link_get_info_by_fd(f.file->private_data,
3626 attr, uattr);
3356 else3627 else
3357 err = -EINVAL;3628 err = -EINVAL;
3358 3629
@@ -3367,7 +3638,7 @@ static int bpf_btf_load(const union bpf_attr *attr) · 8 unmodified lines
3367 if (CHECK_ATTR(BPF_BTF_LOAD))3638 if (CHECK_ATTR(BPF_BTF_LOAD))
3368 return -EINVAL;3639 return -EINVAL;
3369 3640
3370 if (!capable(CAP_SYS_ADMIN))3641 if (!bpf_capable())
3371 return -EPERM;3642 return -EPERM;
3372 3643
3373 return btf_new_fd(attr);3644 return btf_new_fd(attr);
@@ -3480,7 +3751,7 @@ static int bpf_task_fd_query(const union bpf_attr *attr, · 106 unmodified lines
3480 if (file->f_op == &bpf_link_fops) {3751 if (file->f_op == &bpf_link_fops) {
3481 struct bpf_link *link = file->private_data;3752 struct bpf_link *link = file->private_data;
3482 3753
3483 if (link->ops == &bpf_raw_tp_lops) {3754 if (link->ops == &bpf_raw_tp_link_lops) {
3484 struct bpf_raw_tp_link *raw_tp =3755 struct bpf_raw_tp_link *raw_tp =
3485 container_of(link, struct bpf_raw_tp_link, link);3756 container_of(link, struct bpf_raw_tp_link, link);
3486 struct bpf_raw_event_map *btp = raw_tp->btp;3757 struct bpf_raw_event_map *btp = raw_tp->btp;
@@ -3574,6 +3845,15 @@ err_put: · 87 unmodified lines
3574 return err;3845 return err;
3575}3846}
3576 3847
3848static int tracing_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
3849{
3850 if (attr->link_create.attach_type == BPF_TRACE_ITER &&
3851 prog->expected_attach_type == BPF_TRACE_ITER)
3852 return bpf_iter_link_attach(attr, prog);
3853
3854 return -EINVAL;
3855}
3856
3577#define BPF_LINK_CREATE_LAST_FIELD link_create.flags3857#define BPF_LINK_CREATE_LAST_FIELD link_create.flags
3578static int link_create(union bpf_attr *attr)3858static int link_create(union bpf_attr *attr)
3579{3859{
@@ -3581,9 +3861,6 @@ static int link_create(union bpf_attr *attr) · 1 unmodified line
3581 struct bpf_prog *prog;3861 struct bpf_prog *prog;
3582 int ret;3862 int ret;
3583 3863
3584 if (!capable(CAP_NET_ADMIN))
3585 return -EPERM;
3586
3587 if (CHECK_ATTR(BPF_LINK_CREATE))3864 if (CHECK_ATTR(BPF_LINK_CREATE))
3588 return -EINVAL;3865 return -EINVAL;
3589 3866
@@ -3610,6 +3887,12 @@ static int link_create(union bpf_attr *attr) · 20 unmodified lines
3610 case BPF_PROG_TYPE_CGROUP_SOCKOPT:3887 case BPF_PROG_TYPE_CGROUP_SOCKOPT:
3611 ret = cgroup_bpf_link_attach(attr, prog);3888 ret = cgroup_bpf_link_attach(attr, prog);
3612 break;3889 break;
3890 case BPF_PROG_TYPE_TRACING:
3891 ret = tracing_bpf_link_attach(attr, prog);
3892 break;
3893 case BPF_PROG_TYPE_FLOW_DISSECTOR:
3894 ret = netns_bpf_link_create(attr, prog);
3895 break;
3613 default:3896 default:
3614 ret = -EINVAL;3897 ret = -EINVAL;
3615 }3898 }
@@ -3629,9 +3912,6 @@ static int link_update(union bpf_attr *attr) · 13 unmodified lines
3629 u32 flags;3912 u32 flags;
3630 int ret;3913 int ret;
3631 3914
3632 if (!capable(CAP_NET_ADMIN))
3633 return -EPERM;
3634
3635 if (CHECK_ATTR(BPF_LINK_UPDATE))3915 if (CHECK_ATTR(BPF_LINK_UPDATE))
3636 return -EINVAL;3916 return -EINVAL;
3637 3917
@@ -3661,13 +3941,10 @@ static int link_update(union bpf_attr *attr) · 23 unmodified lines
3661 goto out_put_progs;3941 goto out_put_progs;
3662 }3942 }
3663 3943
3664#ifdef CONFIG_CGROUP_BPF3944 if (link->ops->update_prog)
3665 if (link->ops == &bpf_cgroup_link_lops) {3945 ret = link->ops->update_prog(link, new_prog, old_prog);
3666 ret = cgroup_bpf_replace(link, old_prog, new_prog);3946 else
3667 goto out_put_progs;3947 ret = -EINVAL;
3668 }
3669#endif
3670 ret = -EINVAL;
3671 3948
3672out_put_progs:3949out_put_progs:
3673 if (old_prog)3950 if (old_prog)
@@ -3679,12 +3956,131 @@ out_put_link: · 5 unmodified lines
3679 return ret;3956 return ret;
3680}3957}
3681 3958
3959static int bpf_link_inc_not_zero(struct bpf_link *link)
3960{
3961 return atomic64_fetch_add_unless(&link->refcnt, 1, 0) ? 0 : -ENOENT;
3962}
3963
3964#define BPF_LINK_GET_FD_BY_ID_LAST_FIELD link_id
3965
3966static int bpf_link_get_fd_by_id(const union bpf_attr *attr)
3967{
3968 struct bpf_link *link;
3969 u32 id = attr->link_id;
3970 int fd, err;
3971
3972 if (CHECK_ATTR(BPF_LINK_GET_FD_BY_ID))
3973 return -EINVAL;
3974
3975 if (!capable(CAP_SYS_ADMIN))
3976 return -EPERM;
3977
3978 spin_lock_bh(&link_idr_lock);
3979 link = idr_find(&link_idr, id);
3980 /* before link is "settled", ID is 0, pretend it doesn't exist yet */
3981 if (link) {
3982 if (link->id)
3983 err = bpf_link_inc_not_zero(link);
3984 else
3985 err = -EAGAIN;
3986 } else {
3987 err = -ENOENT;
3988 }
3989 spin_unlock_bh(&link_idr_lock);
3990
3991 if (err)
3992 return err;
3993
3994 fd = bpf_link_new_fd(link);
3995 if (fd < 0)
3996 bpf_link_put(link);
3997
3998 return fd;
3999}
4000
4001DEFINE_MUTEX(bpf_stats_enabled_mutex);
4002
4003static int bpf_stats_release(struct inode *inode, struct file *file)
4004{
4005 mutex_lock(&bpf_stats_enabled_mutex);
4006 static_key_slow_dec(&bpf_stats_enabled_key.key);
4007 mutex_unlock(&bpf_stats_enabled_mutex);
4008 return 0;
4009}
4010
4011static const struct file_operations bpf_stats_fops = {
4012 .release = bpf_stats_release,
4013};
4014
4015static int bpf_enable_runtime_stats(void)
4016{
4017 int fd;
4018
4019 mutex_lock(&bpf_stats_enabled_mutex);
4020
4021 /* Set a very high limit to avoid overflow */
4022 if (static_key_count(&bpf_stats_enabled_key.key) > INT_MAX / 2) {
4023 mutex_unlock(&bpf_stats_enabled_mutex);
4024 return -EBUSY;
4025 }
4026
4027 fd = anon_inode_getfd("bpf-stats", &bpf_stats_fops, NULL, O_CLOEXEC);
4028 if (fd >= 0)
4029 static_key_slow_inc(&bpf_stats_enabled_key.key);
4030
4031 mutex_unlock(&bpf_stats_enabled_mutex);
4032 return fd;
4033}
4034
4035#define BPF_ENABLE_STATS_LAST_FIELD enable_stats.type
4036
4037static int bpf_enable_stats(union bpf_attr *attr)
4038{
4039
4040 if (CHECK_ATTR(BPF_ENABLE_STATS))
4041 return -EINVAL;
4042
4043 if (!capable(CAP_SYS_ADMIN))
4044 return -EPERM;
4045
4046 switch (attr->enable_stats.type) {
4047 case BPF_STATS_RUN_TIME:
4048 return bpf_enable_runtime_stats();
4049 default:
4050 break;
4051 }
4052 return -EINVAL;
4053}
4054
4055#define BPF_ITER_CREATE_LAST_FIELD iter_create.flags
4056
4057static int bpf_iter_create(union bpf_attr *attr)
4058{
4059 struct bpf_link *link;
4060 int err;
4061
4062 if (CHECK_ATTR(BPF_ITER_CREATE))
4063 return -EINVAL;
4064
4065 if (attr->iter_create.flags)
4066 return -EINVAL;
4067
4068 link = bpf_link_get_from_fd(attr->iter_create.link_fd);
4069 if (IS_ERR(link))
4070 return PTR_ERR(link);
4071
4072 err = bpf_iter_new_fd(link);
4073 bpf_link_put(link);
4074
4075 return err;
4076}
4077
3682SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)4078SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
3683{4079{
3684 union bpf_attr attr;4080 union bpf_attr attr;
3685 int err;4081 int err;
3686 4082
3687 if (sysctl_unprivileged_bpf_disabled && !capable(CAP_SYS_ADMIN))4083 if (sysctl_unprivileged_bpf_disabled && !bpf_capable())
3688 return -EPERM;4084 return -EPERM;
3689 4085
3690 err = bpf_check_uarg_tail_zero(uattr, sizeof(attr), size);4086 err = bpf_check_uarg_tail_zero(uattr, sizeof(attr), size);
@@ -3796,6 +4192,19 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz · 105 unmodified lines
3796 case BPF_LINK_UPDATE:4192 case BPF_LINK_UPDATE:
3797 err = link_update(&attr);4193 err = link_update(&attr);
3798 break;4194 break;
4195 case BPF_LINK_GET_FD_BY_ID:
4196 err = bpf_link_get_fd_by_id(&attr);
4197 break;
4198 case BPF_LINK_GET_NEXT_ID:
4199 err = bpf_obj_get_next_id(&attr, uattr,
4200 &link_idr, &link_idr_lock);
4201 break;
4202 case BPF_ENABLE_STATS:
4203 err = bpf_enable_stats(&attr);
4204 break;
4205 case BPF_ITER_CREATE:
4206 err = bpf_iter_create(&attr);
4207 break;
3799 default:4208 default:
3800 err = -EINVAL;4209 err = -EINVAL;
3801 break;4210 break;
kernel/bpf/task_iter.c+353−0
@@ -0,0 +1,353 @@
1// SPDX-License-Identifier: GPL-2.0-only
2/* Copyright (c) 2020 Facebook */
3
4#include <linux/init.h>
5#include <linux/namei.h>
6#include <linux/pid_namespace.h>
7#include <linux/fs.h>
8#include <linux/fdtable.h>
9#include <linux/filter.h>
10
11struct bpf_iter_seq_task_common {
12 struct pid_namespace *ns;
13};
14
15struct bpf_iter_seq_task_info {
16 /* The first field must be struct bpf_iter_seq_task_common.
17 * this is assumed by {init, fini}_seq_pidns() callback functions.
18 */
19 struct bpf_iter_seq_task_common common;
20 u32 tid;
21};
22
23static struct task_struct *task_seq_get_next(struct pid_namespace *ns,
24 u32 *tid)
25{
26 struct task_struct *task = NULL;
27 struct pid *pid;
28
29 rcu_read_lock();
30retry:
31 pid = idr_get_next(&ns->idr, tid);
32 if (pid) {
33 task = get_pid_task(pid, PIDTYPE_PID);
34 if (!task) {
35 ++*tid;
36 goto retry;
37 }
38 }
39 rcu_read_unlock();
40
41 return task;
42}
43
44static void *task_seq_start(struct seq_file *seq, loff_t *pos)
45{
46 struct bpf_iter_seq_task_info *info = seq->private;
47 struct task_struct *task;
48
49 task = task_seq_get_next(info->common.ns, &info->tid);
50 if (!task)
51 return NULL;
52
53 ++*pos;
54 return task;
55}
56
57static void *task_seq_next(struct seq_file *seq, void *v, loff_t *pos)
58{
59 struct bpf_iter_seq_task_info *info = seq->private;
60 struct task_struct *task;
61
62 ++*pos;
63 ++info->tid;
64 put_task_struct((struct task_struct *)v);
65 task = task_seq_get_next(info->common.ns, &info->tid);
66 if (!task)
67 return NULL;
68
69 return task;
70}
71
72struct bpf_iter__task {
73 __bpf_md_ptr(struct bpf_iter_meta *, meta);
74 __bpf_md_ptr(struct task_struct *, task);
75};
76
77DEFINE_BPF_ITER_FUNC(task, struct bpf_iter_meta *meta, struct task_struct *task)
78
79static int __task_seq_show(struct seq_file *seq, struct task_struct *task,
80 bool in_stop)
81{
82 struct bpf_iter_meta meta;
83 struct bpf_iter__task ctx;
84 struct bpf_prog *prog;
85
86 meta.seq = seq;
87 prog = bpf_iter_get_info(&meta, in_stop);
88 if (!prog)
89 return 0;
90
91 meta.seq = seq;
92 ctx.meta = &meta;
93 ctx.task = task;
94 return bpf_iter_run_prog(prog, &ctx);
95}
96
97static int task_seq_show(struct seq_file *seq, void *v)
98{
99 return __task_seq_show(seq, v, false);
100}
101
102static void task_seq_stop(struct seq_file *seq, void *v)
103{
104 if (!v)
105 (void)__task_seq_show(seq, v, true);
106 else
107 put_task_struct((struct task_struct *)v);
108}
109
110static const struct seq_operations task_seq_ops = {
111 .start = task_seq_start,
112 .next = task_seq_next,
113 .stop = task_seq_stop,
114 .show = task_seq_show,
115};
116
117struct bpf_iter_seq_task_file_info {
118 /* The first field must be struct bpf_iter_seq_task_common.
119 * this is assumed by {init, fini}_seq_pidns() callback functions.
120 */
121 struct bpf_iter_seq_task_common common;
122 struct task_struct *task;
123 struct files_struct *files;
124 u32 tid;
125 u32 fd;
126};
127
128static struct file *
129task_file_seq_get_next(struct bpf_iter_seq_task_file_info *info,
130 struct task_struct **task, struct files_struct **fstruct)
131{
132 struct pid_namespace *ns = info->common.ns;
133 u32 curr_tid = info->tid, max_fds;
134 struct files_struct *curr_files;
135 struct task_struct *curr_task;
136 int curr_fd = info->fd;
137
138 /* If this function returns a non-NULL file object,
139 * it held a reference to the task/files_struct/file.
140 * Otherwise, it does not hold any reference.
141 */
142again:
143 if (*task) {
144 curr_task = *task;
145 curr_files = *fstruct;
146 curr_fd = info->fd;
147 } else {
148 curr_task = task_seq_get_next(ns, &curr_tid);
149 if (!curr_task)
150 return NULL;
151
152 curr_files = get_files_struct(curr_task);
153 if (!curr_files) {
154 put_task_struct(curr_task);
155 curr_tid = ++(info->tid);
156 info->fd = 0;
157 goto again;
158 }
159
160 /* set *fstruct, *task and info->tid */
161 *fstruct = curr_files;
162 *task = curr_task;
163 if (curr_tid == info->tid) {
164 curr_fd = info->fd;
165 } else {
166 info->tid = curr_tid;
167 curr_fd = 0;
168 }
169 }
170
171 rcu_read_lock();
172 max_fds = files_fdtable(curr_files)->max_fds;
173 for (; curr_fd < max_fds; curr_fd++) {
174 struct file *f;
175
176 f = fcheck_files(curr_files, curr_fd);
177 if (!f)
178 continue;
179
180 /* set info->fd */
181 info->fd = curr_fd;
182 get_file(f);
183 rcu_read_unlock();
184 return f;
185 }
186
187 /* the current task is done, go to the next task */
188 rcu_read_unlock();
189 put_files_struct(curr_files);
190 put_task_struct(curr_task);
191 *task = NULL;
192 *fstruct = NULL;
193 info->fd = 0;
194 curr_tid = ++(info->tid);
195 goto again;
196}
197
198static void *task_file_seq_start(struct seq_file *seq, loff_t *pos)
199{
200 struct bpf_iter_seq_task_file_info *info = seq->private;
201 struct files_struct *files = NULL;
202 struct task_struct *task = NULL;
203 struct file *file;
204
205 file = task_file_seq_get_next(info, &task, &files);
206 if (!file) {
207 info->files = NULL;
208 info->task = NULL;
209 return NULL;
210 }
211
212 ++*pos;
213 info->task = task;
214 info->files = files;
215
216 return file;
217}
218
219static void *task_file_seq_next(struct seq_file *seq, void *v, loff_t *pos)
220{
221 struct bpf_iter_seq_task_file_info *info = seq->private;
222 struct files_struct *files = info->files;
223 struct task_struct *task = info->task;
224 struct file *file;
225
226 ++*pos;
227 ++info->fd;
228 fput((struct file *)v);
229 file = task_file_seq_get_next(info, &task, &files);
230 if (!file) {
231 info->files = NULL;
232 info->task = NULL;
233 return NULL;
234 }
235
236 info->task = task;
237 info->files = files;
238
239 return file;
240}
241
242struct bpf_iter__task_file {
243 __bpf_md_ptr(struct bpf_iter_meta *, meta);
244 __bpf_md_ptr(struct task_struct *, task);
245 u32 fd __aligned(8);
246 __bpf_md_ptr(struct file *, file);
247};
248
249DEFINE_BPF_ITER_FUNC(task_file, struct bpf_iter_meta *meta,
250 struct task_struct *task, u32 fd,
251 struct file *file)
252
253static int __task_file_seq_show(struct seq_file *seq, struct file *file,
254 bool in_stop)
255{
256 struct bpf_iter_seq_task_file_info *info = seq->private;
257 struct bpf_iter__task_file ctx;
258 struct bpf_iter_meta meta;
259 struct bpf_prog *prog;
260
261 meta.seq = seq;
262 prog = bpf_iter_get_info(&meta, in_stop);
263 if (!prog)
264 return 0;
265
266 ctx.meta = &meta;
267 ctx.task = info->task;
268 ctx.fd = info->fd;
269 ctx.file = file;
270 return bpf_iter_run_prog(prog, &ctx);
271}
272
273static int task_file_seq_show(struct seq_file *seq, void *v)
274{
275 return __task_file_seq_show(seq, v, false);
276}
277
278static void task_file_seq_stop(struct seq_file *seq, void *v)
279{
280 struct bpf_iter_seq_task_file_info *info = seq->private;
281
282 if (!v) {
283 (void)__task_file_seq_show(seq, v, true);
284 } else {
285 fput((struct file *)v);
286 put_files_struct(info->files);
287 put_task_struct(info->task);
288 info->files = NULL;
289 info->task = NULL;
290 }
291}
292
293static int init_seq_pidns(void *priv_data)
294{
295 struct bpf_iter_seq_task_common *common = priv_data;
296
297 common->ns = get_pid_ns(task_active_pid_ns(current));
298 return 0;
299}
300
301static void fini_seq_pidns(void *priv_data)
302{
303 struct bpf_iter_seq_task_common *common = priv_data;
304
305 put_pid_ns(common->ns);
306}
307
308static const struct seq_operations task_file_seq_ops = {
309 .start = task_file_seq_start,
310 .next = task_file_seq_next,
311 .stop = task_file_seq_stop,
312 .show = task_file_seq_show,
313};
314
315static const struct bpf_iter_reg task_reg_info = {
316 .target = "task",
317 .seq_ops = &task_seq_ops,
318 .init_seq_private = init_seq_pidns,
319 .fini_seq_private = fini_seq_pidns,
320 .seq_priv_size = sizeof(struct bpf_iter_seq_task_info),
321 .ctx_arg_info_size = 1,
322 .ctx_arg_info = {
323 { offsetof(struct bpf_iter__task, task),
324 PTR_TO_BTF_ID_OR_NULL },
325 },
326};
327
328static const struct bpf_iter_reg task_file_reg_info = {
329 .target = "task_file",
330 .seq_ops = &task_file_seq_ops,
331 .init_seq_private = init_seq_pidns,
332 .fini_seq_private = fini_seq_pidns,
333 .seq_priv_size = sizeof(struct bpf_iter_seq_task_file_info),
334 .ctx_arg_info_size = 2,
335 .ctx_arg_info = {
336 { offsetof(struct bpf_iter__task_file, task),
337 PTR_TO_BTF_ID_OR_NULL },
338 { offsetof(struct bpf_iter__task_file, file),
339 PTR_TO_BTF_ID_OR_NULL },
340 },
341};
342
343static int __init task_iter_init(void)
344{
345 int ret;
346
347 ret = bpf_iter_reg_target(&task_reg_info);
348 if (ret)
349 return ret;
350
351 return bpf_iter_reg_target(&task_file_reg_info);
352}
353late_initcall(task_iter_init);
kernel/bpf/verifier.c+299−88
@@ -28,9 +28,11 @@ static const struct bpf_verifier_ops * const bpf_verifier_ops[] = { · 27 unmodified lines
28#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \28#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
29 [_id] = & _name ## _verifier_ops,29 [_id] = & _name ## _verifier_ops,
30#define BPF_MAP_TYPE(_id, _ops)30#define BPF_MAP_TYPE(_id, _ops)
31#define BPF_LINK_TYPE(_id, _name)
31#include <linux/bpf_types.h>32#include <linux/bpf_types.h>
32#undef BPF_PROG_TYPE33#undef BPF_PROG_TYPE
33#undef BPF_MAP_TYPE34#undef BPF_MAP_TYPE
35#undef BPF_LINK_TYPE
34};36};
35 37
36/* bpf_check() is a static code analyzer that walks eBPF program38/* bpf_check() is a static code analyzer that walks eBPF program
@@ -168,6 +170,8 @@ struct bpf_verifier_stack_elem { · 131 unmodified lines
168 int insn_idx;170 int insn_idx;
169 int prev_insn_idx;171 int prev_insn_idx;
170 struct bpf_verifier_stack_elem *next;172 struct bpf_verifier_stack_elem *next;
173 /* length of verifier log at the time this state was pushed on stack */
174 u32 log_pos;
171};175};
172 176
173#define BPF_COMPLEXITY_LIMIT_JMP_SEQ 8192177#define BPF_COMPLEXITY_LIMIT_JMP_SEQ 8192
@@ -229,6 +233,7 @@ struct bpf_call_arg_meta { · 55 unmodified lines
229 bool pkt_access;233 bool pkt_access;
230 int regno;234 int regno;
231 int access_size;235 int access_size;
236 int mem_size;
232 u64 msize_max_value;237 u64 msize_max_value;
233 int ref_obj_id;238 int ref_obj_id;
234 int func_id;239 int func_id;
@@ -283,6 +288,18 @@ void bpf_verifier_vlog(struct bpf_verifier_log *log, const char *fmt, · 48 unmodified lines
283 log->ubuf = NULL;288 log->ubuf = NULL;
284}289}
285 290
291static void bpf_vlog_reset(struct bpf_verifier_log *log, u32 new_pos)
292{
293 char zero = 0;
294
295 if (!bpf_verifier_log_needed(log))
296 return;
297
298 log->len_used = new_pos;
299 if (put_user(zero, log->ubuf + new_pos))
300 log->ubuf = NULL;
301}
302
286/* log_level controls verbosity level of eBPF verifier.303/* log_level controls verbosity level of eBPF verifier.
287 * bpf_verifier_log_write() is used to dump the verification trace to the log,304 * bpf_verifier_log_write() is used to dump the verification trace to the log,
288 * so the user can figure out what's wrong with the program305 * so the user can figure out what's wrong with the program
@@ -377,12 +394,23 @@ static bool type_is_sk_pointer(enum bpf_reg_type type) · 88 unmodified lines
377 type == PTR_TO_XDP_SOCK;394 type == PTR_TO_XDP_SOCK;
378}395}
379 396
397static bool reg_type_not_null(enum bpf_reg_type type)
398{
399 return type == PTR_TO_SOCKET ||
400 type == PTR_TO_TCP_SOCK ||
401 type == PTR_TO_MAP_VALUE ||
402 type == PTR_TO_SOCK_COMMON ||
403 type == PTR_TO_BTF_ID;
404}
405
380static bool reg_type_may_be_null(enum bpf_reg_type type)406static bool reg_type_may_be_null(enum bpf_reg_type type)
381{407{
382 return type == PTR_TO_MAP_VALUE_OR_NULL ||408 return type == PTR_TO_MAP_VALUE_OR_NULL ||
383 type == PTR_TO_SOCKET_OR_NULL ||409 type == PTR_TO_SOCKET_OR_NULL ||
384 type == PTR_TO_SOCK_COMMON_OR_NULL ||410 type == PTR_TO_SOCK_COMMON_OR_NULL ||
385 type == PTR_TO_TCP_SOCK_OR_NULL;411 type == PTR_TO_TCP_SOCK_OR_NULL ||
412 type == PTR_TO_BTF_ID_OR_NULL ||
413 type == PTR_TO_MEM_OR_NULL;
386}414}
387 415
388static bool reg_may_point_to_spin_lock(const struct bpf_reg_state *reg)416static bool reg_may_point_to_spin_lock(const struct bpf_reg_state *reg)
@@ -396,7 +424,9 @@ static bool reg_type_may_be_refcounted_or_null(enum bpf_reg_type type) · 7 unmodified lines
396 return type == PTR_TO_SOCKET ||424 return type == PTR_TO_SOCKET ||
397 type == PTR_TO_SOCKET_OR_NULL ||425 type == PTR_TO_SOCKET_OR_NULL ||
398 type == PTR_TO_TCP_SOCK ||426 type == PTR_TO_TCP_SOCK ||
399 type == PTR_TO_TCP_SOCK_OR_NULL;427 type == PTR_TO_TCP_SOCK_OR_NULL ||
428 type == PTR_TO_MEM ||
429 type == PTR_TO_MEM_OR_NULL;
400}430}
401 431
402static bool arg_type_may_be_refcounted(enum bpf_arg_type type)432static bool arg_type_may_be_refcounted(enum bpf_arg_type type)
@@ -410,14 +440,37 @@ static bool arg_type_may_be_refcounted(enum bpf_arg_type type) · 7 unmodified lines
410 */440 */
411static bool is_release_function(enum bpf_func_id func_id)441static bool is_release_function(enum bpf_func_id func_id)
412{442{
413 return func_id == BPF_FUNC_sk_release;443 return func_id == BPF_FUNC_sk_release ||
444 func_id == BPF_FUNC_ringbuf_submit ||
445 func_id == BPF_FUNC_ringbuf_discard;
414}446}
415 447
416static bool is_acquire_function(enum bpf_func_id func_id)448static bool may_be_acquire_function(enum bpf_func_id func_id)
417{449{
418 return func_id == BPF_FUNC_sk_lookup_tcp ||450 return func_id == BPF_FUNC_sk_lookup_tcp ||
419 func_id == BPF_FUNC_sk_lookup_udp ||451 func_id == BPF_FUNC_sk_lookup_udp ||
420 func_id == BPF_FUNC_skc_lookup_tcp;452 func_id == BPF_FUNC_skc_lookup_tcp ||
453 func_id == BPF_FUNC_map_lookup_elem ||
454 func_id == BPF_FUNC_ringbuf_reserve;
455}
456
457static bool is_acquire_function(enum bpf_func_id func_id,
458 const struct bpf_map *map)
459{
460 enum bpf_map_type map_type = map ? map->map_type : BPF_MAP_TYPE_UNSPEC;
461
462 if (func_id == BPF_FUNC_sk_lookup_tcp ||
463 func_id == BPF_FUNC_sk_lookup_udp ||
464 func_id == BPF_FUNC_skc_lookup_tcp ||
465 func_id == BPF_FUNC_ringbuf_reserve)
466 return true;
467
468 if (func_id == BPF_FUNC_map_lookup_elem &&
469 (map_type == BPF_MAP_TYPE_SOCKMAP ||
470 map_type == BPF_MAP_TYPE_SOCKHASH))
471 return true;
472
473 return false;
421}474}
422 475
423static bool is_ptr_cast_function(enum bpf_func_id func_id)476static bool is_ptr_cast_function(enum bpf_func_id func_id)
@@ -448,6 +501,9 @@ static const char * const reg_type_str[] = { · 24 unmodified lines
448 [PTR_TO_TP_BUFFER] = "tp_buffer",501 [PTR_TO_TP_BUFFER] = "tp_buffer",
449 [PTR_TO_XDP_SOCK] = "xdp_sock",502 [PTR_TO_XDP_SOCK] = "xdp_sock",
450 [PTR_TO_BTF_ID] = "ptr_",503 [PTR_TO_BTF_ID] = "ptr_",
504 [PTR_TO_BTF_ID_OR_NULL] = "ptr_or_null_",
505 [PTR_TO_MEM] = "mem",
506 [PTR_TO_MEM_OR_NULL] = "mem_or_null",
451};507};
452 508
453static char slot_type_char[] = {509static char slot_type_char[] = {
@@ -508,7 +564,7 @@ static void print_verifier_state(struct bpf_verifier_env *env, · 54 unmodified lines
508 /* reg->off should be 0 for SCALAR_VALUE */564 /* reg->off should be 0 for SCALAR_VALUE */
509 verbose(env, "%lld", reg->var_off.value + reg->off);565 verbose(env, "%lld", reg->var_off.value + reg->off);
510 } else {566 } else {
511 if (t == PTR_TO_BTF_ID)567 if (t == PTR_TO_BTF_ID || t == PTR_TO_BTF_ID_OR_NULL)
512 verbose(env, "%s", kernel_type_name(reg->btf_id));568 verbose(env, "%s", kernel_type_name(reg->btf_id));
513 verbose(env, "(id=%d", reg->id);569 verbose(env, "(id=%d", reg->id);
514 if (reg_type_may_be_refcounted_or_null(t))570 if (reg_type_may_be_refcounted_or_null(t))
@@ -846,7 +902,7 @@ static void update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifi · 331 unmodified lines
846}902}
847 903
848static int pop_stack(struct bpf_verifier_env *env, int *prev_insn_idx,904static int pop_stack(struct bpf_verifier_env *env, int *prev_insn_idx,
849 int *insn_idx)905 int *insn_idx, bool pop_log)
850{906{
851 struct bpf_verifier_state *cur = env->cur_state;907 struct bpf_verifier_state *cur = env->cur_state;
852 struct bpf_verifier_stack_elem *elem, *head = env->head;908 struct bpf_verifier_stack_elem *elem, *head = env->head;
@@ -860,6 +916,8 @@ static int pop_stack(struct bpf_verifier_env *env, int *prev_insn_idx, · 7 unmodified lines
860 if (err)916 if (err)
861 return err;917 return err;
862 }918 }
919 if (pop_log)
920 bpf_vlog_reset(&env->log, head->log_pos);
863 if (insn_idx)921 if (insn_idx)
864 *insn_idx = head->insn_idx;922 *insn_idx = head->insn_idx;
865 if (prev_insn_idx)923 if (prev_insn_idx)
@@ -887,6 +945,7 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env, · 21 unmodified lines
887 elem->insn_idx = insn_idx;945 elem->insn_idx = insn_idx;
888 elem->prev_insn_idx = prev_insn_idx;946 elem->prev_insn_idx = prev_insn_idx;
889 elem->next = env->head;947 elem->next = env->head;
948 elem->log_pos = env->log.len_used;
890 env->head = elem;949 env->head = elem;
891 env->stack_size++;950 env->stack_size++;
892 err = copy_verifier_state(&elem->st, cur);951 err = copy_verifier_state(&elem->st, cur);
@@ -915,7 +974,7 @@ err: · 22 unmodified lines
915 free_verifier_state(env->cur_state, true);974 free_verifier_state(env->cur_state, true);
916 env->cur_state = NULL;975 env->cur_state = NULL;
917 /* pop all elements and return */976 /* pop all elements and return */
918 while (!pop_stack(env, NULL, NULL));977 while (!pop_stack(env, NULL, NULL, false));
919 return NULL;978 return NULL;
920}979}
921 980
@@ -1255,7 +1314,7 @@ static void __mark_reg_unknown(const struct bpf_verifier_env *env, · 333 unmodified lines
1255 reg->type = SCALAR_VALUE;1314 reg->type = SCALAR_VALUE;
1256 reg->var_off = tnum_unknown;1315 reg->var_off = tnum_unknown;
1257 reg->frameno = 0;1316 reg->frameno = 0;
1258 reg->precise = env->subprog_cnt > 1 || !env->allow_ptr_leaks;1317 reg->precise = env->subprog_cnt > 1 || !env->bpf_capable;
1259 __mark_reg_unbounded(reg);1318 __mark_reg_unbounded(reg);
1260}1319}
1261 1320
@@ -1387,8 +1446,9 @@ static int check_subprogs(struct bpf_verifier_env *env) · 125 unmodified lines
1387 continue;1446 continue;
1388 if (insn[i].src_reg != BPF_PSEUDO_CALL)1447 if (insn[i].src_reg != BPF_PSEUDO_CALL)
1389 continue;1448 continue;
1390 if (!env->allow_ptr_leaks) {1449 if (!env->bpf_capable) {
1391 verbose(env, "function calls to other bpf functions are allowed for root only\n");1450 verbose(env,
1451 "function calls to other bpf functions are allowed for CAP_BPF and CAP_SYS_ADMIN\n");
1392 return -EPERM;1452 return -EPERM;
1393 }1453 }
1394 ret = add_subprog(env, i + insn[i].imm + 1);1454 ret = add_subprog(env, i + insn[i].imm + 1);
@@ -1922,8 +1982,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno, · 527 unmodified lines
1922 bool new_marks = false;1982 bool new_marks = false;
1923 int i, err;1983 int i, err;
1924 1984
1925 if (!env->allow_ptr_leaks)1985 if (!env->bpf_capable)
1926 /* backtracking is root only for now */
1927 return 0;1986 return 0;
1928 1987
1929 func = st->frame[st->curframe];1988 func = st->frame[st->curframe];
@@ -2101,6 +2160,7 @@ static bool is_spillable_regtype(enum bpf_reg_type type) · 171 unmodified lines
2101 case PTR_TO_TCP_SOCK_OR_NULL:2160 case PTR_TO_TCP_SOCK_OR_NULL:
2102 case PTR_TO_XDP_SOCK:2161 case PTR_TO_XDP_SOCK:
2103 case PTR_TO_BTF_ID:2162 case PTR_TO_BTF_ID:
2163 case PTR_TO_BTF_ID_OR_NULL:
2104 return true;2164 return true;
2105 default:2165 default:
2106 return false;2166 return false;
@@ -2170,7 +2230,7 @@ static int check_stack_write(struct bpf_verifier_env *env, · 63 unmodified lines
2170 reg = &cur->regs[value_regno];2230 reg = &cur->regs[value_regno];
2171 2231
2172 if (reg && size == BPF_REG_SIZE && register_is_const(reg) &&2232 if (reg && size == BPF_REG_SIZE && register_is_const(reg) &&
2173 !register_is_null(reg) && env->allow_ptr_leaks) {2233 !register_is_null(reg) && env->bpf_capable) {
2174 if (dst_reg != BPF_REG_FP) {2234 if (dst_reg != BPF_REG_FP) {
2175 /* The backtracking logic can only recognize explicit2235 /* The backtracking logic can only recognize explicit
2176 * stack slot address like [fp - 8]. Other spill of2236 * stack slot address like [fp - 8]. Other spill of
@@ -2196,7 +2256,7 @@ static int check_stack_write(struct bpf_verifier_env *env, · 19 unmodified lines
2196 return -EINVAL;2256 return -EINVAL;
2197 }2257 }
2198 2258
2199 if (!env->allow_ptr_leaks) {2259 if (!env->bypass_spec_v4) {
2200 bool sanitize = false;2260 bool sanitize = false;
2201 2261
2202 if (state->stack[spi].slot_type[0] == STACK_SPILL &&2262 if (state->stack[spi].slot_type[0] == STACK_SPILL &&
@@ -2418,32 +2478,49 @@ static int check_map_access_type(struct bpf_verifier_env *env, u32 regno, · 215 unmodified lines
2418 return 0;2478 return 0;
2419}2479}
2420 2480
2421/* check read/write into map element returned by bpf_map_lookup_elem() */2481/* check read/write into memory region (e.g., map value, ringbuf sample, etc) */
2422static int __check_map_access(struct bpf_verifier_env *env, u32 regno, int off,2482static int __check_mem_access(struct bpf_verifier_env *env, int regno,
2423 int size, bool zero_size_allowed)2483 int off, int size, u32 mem_size,
2484 bool zero_size_allowed)
2424{2485{
2425 struct bpf_reg_state *regs = cur_regs(env);2486 bool size_ok = size > 0 || (size == 0 && zero_size_allowed);
2426 struct bpf_map *map = regs[regno].map_ptr;2487 struct bpf_reg_state *reg;
2488
2489 if (off >= 0 && size_ok && (u64)off + size <= mem_size)
2490 return 0;
2427 2491
2428 if (off < 0 || size < 0 || (size == 0 && !zero_size_allowed) ||2492 reg = &cur_regs(env)[regno];
2429 off + size > map->value_size) {2493 switch (reg->type) {
2494 case PTR_TO_MAP_VALUE:
2430 verbose(env, "invalid access to map value, value_size=%d off=%d size=%d\n",2495 verbose(env, "invalid access to map value, value_size=%d off=%d size=%d\n",
2431 map->value_size, off, size);2496 mem_size, off, size);
2432 return -EACCES;2497 break;
2498 case PTR_TO_PACKET:
2499 case PTR_TO_PACKET_META:
2500 case PTR_TO_PACKET_END:
2501 verbose(env, "invalid access to packet, off=%d size=%d, R%d(id=%d,off=%d,r=%d)\n",
2502 off, size, regno, reg->id, off, mem_size);
2503 break;
2504 case PTR_TO_MEM:
2505 default:
2506 verbose(env, "invalid access to memory, mem_size=%u off=%d size=%d\n",
2507 mem_size, off, size);
2433 }2508 }
2434 return 0;2509
2510 return -EACCES;
2435}2511}
2436 2512
2437/* check read/write into a map element with possible variable offset */2513/* check read/write into a memory region with possible variable offset */
2438static int check_map_access(struct bpf_verifier_env *env, u32 regno,2514static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno,
2439 int off, int size, bool zero_size_allowed)2515 int off, int size, u32 mem_size,
2516 bool zero_size_allowed)
2440{2517{
2441 struct bpf_verifier_state *vstate = env->cur_state;2518 struct bpf_verifier_state *vstate = env->cur_state;
2442 struct bpf_func_state *state = vstate->frame[vstate->curframe];2519 struct bpf_func_state *state = vstate->frame[vstate->curframe];
2443 struct bpf_reg_state *reg = &state->regs[regno];2520 struct bpf_reg_state *reg = &state->regs[regno];
2444 int err;2521 int err;
2445 2522
2446 /* We may have adjusted the register to this map value, so we2523 /* We may have adjusted the register pointing to memory region, so we
2447 * need to try adding each of min_value and max_value to off2524 * need to try adding each of min_value and max_value to off
2448 * to make sure our theoretical access will be safe.2525 * to make sure our theoretical access will be safe.
2449 */2526 */
@@ -2464,10 +2541,10 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 14 unmodified lines
2464 regno);2541 regno);
2465 return -EACCES;2542 return -EACCES;
2466 }2543 }
2467 err = __check_map_access(env, regno, reg->smin_value + off, size,2544 err = __check_mem_access(env, regno, reg->smin_value + off, size,
2468 zero_size_allowed);2545 mem_size, zero_size_allowed);
2469 if (err) {2546 if (err) {
2470 verbose(env, "R%d min value is outside of the array range\n",2547 verbose(env, "R%d min value is outside of the allowed memory range\n",
2471 regno);2548 regno);
2472 return err;2549 return err;
2473 }2550 }
@@ -2477,18 +2554,38 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 3 unmodified lines
2477 * If reg->umax_value + off could overflow, treat that as unbounded too.2554 * If reg->umax_value + off could overflow, treat that as unbounded too.
2478 */2555 */
2479 if (reg->umax_value >= BPF_MAX_VAR_OFF) {2556 if (reg->umax_value >= BPF_MAX_VAR_OFF) {
2480 verbose(env, "R%d unbounded memory access, make sure to bounds check any array access into a map\n",2557 verbose(env, "R%d unbounded memory access, make sure to bounds check any such access\n",
2481 regno);2558 regno);
2482 return -EACCES;2559 return -EACCES;
2483 }2560 }
2484 err = __check_map_access(env, regno, reg->umax_value + off, size,2561 err = __check_mem_access(env, regno, reg->umax_value + off, size,
2485 zero_size_allowed);2562 mem_size, zero_size_allowed);
2486 if (err)2563 if (err) {
2487 verbose(env, "R%d max value is outside of the array range\n",2564 verbose(env, "R%d max value is outside of the allowed memory range\n",
2488 regno);2565 regno);
2566 return err;
2567 }
2568
2569 return 0;
2570}
2571
2572/* check read/write into a map element with possible variable offset */
2573static int check_map_access(struct bpf_verifier_env *env, u32 regno,
2574 int off, int size, bool zero_size_allowed)
2575{
2576 struct bpf_verifier_state *vstate = env->cur_state;
2577 struct bpf_func_state *state = vstate->frame[vstate->curframe];
2578 struct bpf_reg_state *reg = &state->regs[regno];
2579 struct bpf_map *map = reg->map_ptr;
2580 int err;
2489 2581
2490 if (map_value_has_spin_lock(reg->map_ptr)) {2582 err = check_mem_region_access(env, regno, off, size, map->value_size,
2491 u32 lock = reg->map_ptr->spin_lock_off;2583 zero_size_allowed);
2584 if (err)
2585 return err;
2586
2587 if (map_value_has_spin_lock(map)) {
2588 u32 lock = map->spin_lock_off;
2492 2589
2493 /* if any part of struct bpf_spin_lock can be touched by2590 /* if any part of struct bpf_spin_lock can be touched by
2494 * load/store reject this program.2591 * load/store reject this program.
@@ -2546,21 +2643,6 @@ static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, · 51 unmodified lines
2546 }2643 }
2547}2644}
2548 2645
2549static int __check_packet_access(struct bpf_verifier_env *env, u32 regno,
2550 int off, int size, bool zero_size_allowed)
2551{
2552 struct bpf_reg_state *regs = cur_regs(env);
2553 struct bpf_reg_state *reg = &regs[regno];
2554
2555 if (off < 0 || size < 0 || (size == 0 && !zero_size_allowed) ||
2556 (u64)off + size > reg->range) {
2557 verbose(env, "invalid access to packet, off=%d size=%d, R%d(id=%d,off=%d,r=%d)\n",
2558 off, size, regno, reg->id, reg->off, reg->range);
2559 return -EACCES;
2560 }
2561 return 0;
2562}
2563
2564static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off,2646static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off,
2565 int size, bool zero_size_allowed)2647 int size, bool zero_size_allowed)
2566{2648{
@@ -2581,16 +2663,17 @@ static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off, · 14 unmodified lines
2581 regno);2663 regno);
2582 return -EACCES;2664 return -EACCES;
2583 }2665 }
2584 err = __check_packet_access(env, regno, off, size, zero_size_allowed);2666 err = __check_mem_access(env, regno, off, size, reg->range,
2667 zero_size_allowed);
2585 if (err) {2668 if (err) {
2586 verbose(env, "R%d offset is outside of the packet\n", regno);2669 verbose(env, "R%d offset is outside of the packet\n", regno);
2587 return err;2670 return err;
2588 }2671 }
2589 2672
2590 /* __check_packet_access has made sure "off + size - 1" is within u16.2673 /* __check_mem_access has made sure "off + size - 1" is within u16.
2591 * reg->umax_value can't be bigger than MAX_PACKET_OFF which is 0xffff,2674 * reg->umax_value can't be bigger than MAX_PACKET_OFF which is 0xffff,
2592 * otherwise find_good_pkt_pointers would have refused to set range info2675 * otherwise find_good_pkt_pointers would have refused to set range info
2593 * that __check_packet_access would have rejected this pkt access.2676 * that __check_mem_access would have rejected this pkt access.
2594 * Therefore, "off + reg->umax_value + size - 1" won't overflow u32.2677 * Therefore, "off + reg->umax_value + size - 1" won't overflow u32.
2595 */2678 */
2596 env->prog->aux->max_pkt_offset =2679 env->prog->aux->max_pkt_offset =
@@ -2621,7 +2704,7 @@ static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, int off, · 24 unmodified lines
2621 */2704 */
2622 *reg_type = info.reg_type;2705 *reg_type = info.reg_type;
2623 2706
2624 if (*reg_type == PTR_TO_BTF_ID)2707 if (*reg_type == PTR_TO_BTF_ID || *reg_type == PTR_TO_BTF_ID_OR_NULL)
2625 *btf_id = info.btf_id;2708 *btf_id = info.btf_id;
2626 else2709 else
2627 env->insn_aux_data[insn_idx].ctx_field_size = info.ctx_field_size;2710 env->insn_aux_data[insn_idx].ctx_field_size = info.ctx_field_size;
@@ -3170,6 +3253,16 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 542 unmodified lines
3170 mark_reg_unknown(env, regs, value_regno);3253 mark_reg_unknown(env, regs, value_regno);
3171 }3254 }
3172 }3255 }
3256 } else if (reg->type == PTR_TO_MEM) {
3257 if (t == BPF_WRITE && value_regno >= 0 &&
3258 is_pointer_value(env, value_regno)) {
3259 verbose(env, "R%d leaks addr into mem\n", value_regno);
3260 return -EACCES;
3261 }
3262 err = check_mem_region_access(env, regno, off, size,
3263 reg->mem_size, false);
3264 if (!err && t == BPF_READ && value_regno >= 0)
3265 mark_reg_unknown(env, regs, value_regno);
3173 } else if (reg->type == PTR_TO_CTX) {3266 } else if (reg->type == PTR_TO_CTX) {
3174 enum bpf_reg_type reg_type = SCALAR_VALUE;3267 enum bpf_reg_type reg_type = SCALAR_VALUE;
3175 u32 btf_id = 0;3268 u32 btf_id = 0;
@@ -3205,7 +3298,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 29 unmodified lines
3205 * a sub-register.3298 * a sub-register.
3206 */3299 */
3207 regs[value_regno].subreg_def = DEF_NOT_SUBREG;3300 regs[value_regno].subreg_def = DEF_NOT_SUBREG;
3208 if (reg_type == PTR_TO_BTF_ID)3301 if (reg_type == PTR_TO_BTF_ID ||
3302 reg_type == PTR_TO_BTF_ID_OR_NULL)
3209 regs[value_regno].btf_id = btf_id;3303 regs[value_regno].btf_id = btf_id;
3210 }3304 }
3211 regs[value_regno].type = reg_type;3305 regs[value_regno].type = reg_type;
@@ -3390,7 +3484,7 @@ static int check_stack_boundary(struct bpf_verifier_env *env, int regno, · 178 unmodified lines
3390 * Spectre masking for stack ALU.3484 * Spectre masking for stack ALU.
3391 * See also retrieve_ptr_limit().3485 * See also retrieve_ptr_limit().
3392 */3486 */
3393 if (!env->allow_ptr_leaks) {3487 if (!env->bypass_spec_v1) {
3394 char tn_buf[48];3488 char tn_buf[48];
3395 3489
3396 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);3490 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
@@ -3452,6 +3546,11 @@ static int check_stack_boundary(struct bpf_verifier_env *env, int regno, · 55 unmodified lines
3452 *stype = STACK_MISC;3546 *stype = STACK_MISC;
3453 goto mark;3547 goto mark;
3454 }3548 }
3549
3550 if (state->stack[spi].slot_type[0] == STACK_SPILL &&
3551 state->stack[spi].spilled_ptr.type == PTR_TO_BTF_ID)
3552 goto mark;
3553
3455 if (state->stack[spi].slot_type[0] == STACK_SPILL &&3554 if (state->stack[spi].slot_type[0] == STACK_SPILL &&
3456 state->stack[spi].spilled_ptr.type == SCALAR_VALUE) {3555 state->stack[spi].spilled_ptr.type == SCALAR_VALUE) {
3457 __mark_reg_unknown(env, &state->stack[spi].spilled_ptr);3556 __mark_reg_unknown(env, &state->stack[spi].spilled_ptr);
@@ -3501,6 +3600,10 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 43 unmodified lines
3501 return -EACCES;3600 return -EACCES;
3502 return check_map_access(env, regno, reg->off, access_size,3601 return check_map_access(env, regno, reg->off, access_size,
3503 zero_size_allowed);3602 zero_size_allowed);
3603 case PTR_TO_MEM:
3604 return check_mem_region_access(env, regno, reg->off,
3605 access_size, reg->mem_size,
3606 zero_size_allowed);
3504 default: /* scalar_value|ptr_to_stack or invalid ptr */3607 default: /* scalar_value|ptr_to_stack or invalid ptr */
3505 return check_stack_boundary(env, regno, access_size,3608 return check_stack_boundary(env, regno, access_size,
3506 zero_size_allowed, meta);3609 zero_size_allowed, meta);
@@ -3605,6 +3708,17 @@ static bool arg_type_is_mem_size(enum bpf_arg_type type) · 98 unmodified lines
3605 type == ARG_CONST_SIZE_OR_ZERO;3708 type == ARG_CONST_SIZE_OR_ZERO;
3606}3709}
3607 3710
3711static bool arg_type_is_alloc_mem_ptr(enum bpf_arg_type type)
3712{
3713 return type == ARG_PTR_TO_ALLOC_MEM ||
3714 type == ARG_PTR_TO_ALLOC_MEM_OR_NULL;
3715}
3716
3717static bool arg_type_is_alloc_size(enum bpf_arg_type type)
3718{
3719 return type == ARG_CONST_ALLOC_SIZE_OR_ZERO;
3720}
3721
3608static bool arg_type_is_int_ptr(enum bpf_arg_type type)3722static bool arg_type_is_int_ptr(enum bpf_arg_type type)
3609{3723{
3610 return type == ARG_PTR_TO_INT ||3724 return type == ARG_PTR_TO_INT ||
@@ -3664,7 +3778,8 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, · 53 unmodified lines
3664 type != expected_type)3778 type != expected_type)
3665 goto err_type;3779 goto err_type;
3666 } else if (arg_type == ARG_CONST_SIZE ||3780 } else if (arg_type == ARG_CONST_SIZE ||
3667 arg_type == ARG_CONST_SIZE_OR_ZERO) {3781 arg_type == ARG_CONST_SIZE_OR_ZERO ||
3782 arg_type == ARG_CONST_ALLOC_SIZE_OR_ZERO) {
3668 expected_type = SCALAR_VALUE;3783 expected_type = SCALAR_VALUE;
3669 if (type != expected_type)3784 if (type != expected_type)
3670 goto err_type;3785 goto err_type;
@@ -3735,13 +3850,29 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, · 64 unmodified lines
3735 * happens during stack boundary checking.3850 * happens during stack boundary checking.
3736 */3851 */
3737 if (register_is_null(reg) &&3852 if (register_is_null(reg) &&
3738 arg_type == ARG_PTR_TO_MEM_OR_NULL)3853 (arg_type == ARG_PTR_TO_MEM_OR_NULL ||
3854 arg_type == ARG_PTR_TO_ALLOC_MEM_OR_NULL))
3739 /* final test in check_stack_boundary() */;3855 /* final test in check_stack_boundary() */;
3740 else if (!type_is_pkt_pointer(type) &&3856 else if (!type_is_pkt_pointer(type) &&
3741 type != PTR_TO_MAP_VALUE &&3857 type != PTR_TO_MAP_VALUE &&
3858 type != PTR_TO_MEM &&
3742 type != expected_type)3859 type != expected_type)
3743 goto err_type;3860 goto err_type;
3744 meta->raw_mode = arg_type == ARG_PTR_TO_UNINIT_MEM;3861 meta->raw_mode = arg_type == ARG_PTR_TO_UNINIT_MEM;
3862 } else if (arg_type_is_alloc_mem_ptr(arg_type)) {
3863 expected_type = PTR_TO_MEM;
3864 if (register_is_null(reg) &&
3865 arg_type == ARG_PTR_TO_ALLOC_MEM_OR_NULL)
3866 /* final test in check_stack_boundary() */;
3867 else if (type != expected_type)
3868 goto err_type;
3869 if (meta->ref_obj_id) {
3870 verbose(env, "verifier internal error: more than one arg with ref_obj_id R%d %u %u\n",
3871 regno, reg->ref_obj_id,
3872 meta->ref_obj_id);
3873 return -EFAULT;
3874 }
3875 meta->ref_obj_id = reg->ref_obj_id;
3745 } else if (arg_type_is_int_ptr(arg_type)) {3876 } else if (arg_type_is_int_ptr(arg_type)) {
3746 expected_type = PTR_TO_STACK;3877 expected_type = PTR_TO_STACK;
3747 if (!type_is_pkt_pointer(type) &&3878 if (!type_is_pkt_pointer(type) &&
@@ -3837,6 +3968,13 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, · 89 unmodified lines
3837 zero_size_allowed, meta);3968 zero_size_allowed, meta);
3838 if (!err)3969 if (!err)
3839 err = mark_chain_precision(env, regno);3970 err = mark_chain_precision(env, regno);
3971 } else if (arg_type_is_alloc_size(arg_type)) {
3972 if (!tnum_is_const(reg->var_off)) {
3973 verbose(env, "R%d unbounded size, use 'var &= const' or 'if (var < const)'\n",
3974 regno);
3975 return -EACCES;
3976 }
3977 meta->mem_size = reg->var_off.value;
3840 } else if (arg_type_is_int_ptr(arg_type)) {3978 } else if (arg_type_is_int_ptr(arg_type)) {
3841 int size = int_ptr_type_to_size(arg_type);3979 int size = int_ptr_type_to_size(arg_type);
3842 3980
@@ -3873,6 +4011,14 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 30 unmodified lines
3873 func_id != BPF_FUNC_xdp_output)4011 func_id != BPF_FUNC_xdp_output)
3874 goto error;4012 goto error;
3875 break;4013 break;
4014 case BPF_MAP_TYPE_RINGBUF:
4015 if (func_id != BPF_FUNC_ringbuf_output &&
4016 func_id != BPF_FUNC_ringbuf_reserve &&
4017 func_id != BPF_FUNC_ringbuf_submit &&
4018 func_id != BPF_FUNC_ringbuf_discard &&
4019 func_id != BPF_FUNC_ringbuf_query)
4020 goto error;
4021 break;
3876 case BPF_MAP_TYPE_STACK_TRACE:4022 case BPF_MAP_TYPE_STACK_TRACE:
3877 if (func_id != BPF_FUNC_get_stackid)4023 if (func_id != BPF_FUNC_get_stackid)
3878 goto error;4024 goto error;
@@ -3915,7 +4061,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 36 unmodified lines
3915 func_id != BPF_FUNC_sock_map_update &&4061 func_id != BPF_FUNC_sock_map_update &&
3916 func_id != BPF_FUNC_map_delete_elem &&4062 func_id != BPF_FUNC_map_delete_elem &&
3917 func_id != BPF_FUNC_msg_redirect_map &&4063 func_id != BPF_FUNC_msg_redirect_map &&
3918 func_id != BPF_FUNC_sk_select_reuseport)4064 func_id != BPF_FUNC_sk_select_reuseport &&
4065 func_id != BPF_FUNC_map_lookup_elem)
3919 goto error;4066 goto error;
3920 break;4067 break;
3921 case BPF_MAP_TYPE_SOCKHASH:4068 case BPF_MAP_TYPE_SOCKHASH:
@@ -3923,7 +4070,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 1 unmodified line
3923 func_id != BPF_FUNC_sock_hash_update &&4070 func_id != BPF_FUNC_sock_hash_update &&
3924 func_id != BPF_FUNC_map_delete_elem &&4071 func_id != BPF_FUNC_map_delete_elem &&
3925 func_id != BPF_FUNC_msg_redirect_hash &&4072 func_id != BPF_FUNC_msg_redirect_hash &&
3926 func_id != BPF_FUNC_sk_select_reuseport)4073 func_id != BPF_FUNC_sk_select_reuseport &&
4074 func_id != BPF_FUNC_map_lookup_elem)
3927 goto error;4075 goto error;
3928 break;4076 break;
3929 case BPF_MAP_TYPE_REUSEPORT_SOCKARRAY:4077 case BPF_MAP_TYPE_REUSEPORT_SOCKARRAY:
@@ -4093,7 +4241,7 @@ static bool check_refcount_ok(const struct bpf_func_proto *fn, int func_id) · 163 unmodified lines
4093 /* A reference acquiring function cannot acquire4241 /* A reference acquiring function cannot acquire
4094 * another refcounted ptr.4242 * another refcounted ptr.
4095 */4243 */
4096 if (is_acquire_function(func_id) && count)4244 if (may_be_acquire_function(func_id) && count)
4097 return false;4245 return false;
4098 4246
4099 /* We only support one arg being unreferenced at the moment,4247 /* We only support one arg being unreferenced at the moment,
@@ -4388,10 +4536,10 @@ record_func_map(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, · 288 unmodified lines
4388 4536
4389 if (!BPF_MAP_PTR(aux->map_ptr_state))4537 if (!BPF_MAP_PTR(aux->map_ptr_state))
4390 bpf_map_ptr_store(aux, meta->map_ptr,4538 bpf_map_ptr_store(aux, meta->map_ptr,
4391 meta->map_ptr->unpriv_array);4539 !meta->map_ptr->bypass_spec_v1);
4392 else if (BPF_MAP_PTR(aux->map_ptr_state) != meta->map_ptr)4540 else if (BPF_MAP_PTR(aux->map_ptr_state) != meta->map_ptr)
4393 bpf_map_ptr_store(aux, BPF_MAP_PTR_POISON,4541 bpf_map_ptr_store(aux, BPF_MAP_PTR_POISON,
4394 meta->map_ptr->unpriv_array);4542 !meta->map_ptr->bypass_spec_v1);
4395 return 0;4543 return 0;
4396}4544}
4397 4545
@@ -4597,6 +4745,11 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 199 unmodified lines
4597 mark_reg_known_zero(env, regs, BPF_REG_0);4745 mark_reg_known_zero(env, regs, BPF_REG_0);
4598 regs[BPF_REG_0].type = PTR_TO_TCP_SOCK_OR_NULL;4746 regs[BPF_REG_0].type = PTR_TO_TCP_SOCK_OR_NULL;
4599 regs[BPF_REG_0].id = ++env->id_gen;4747 regs[BPF_REG_0].id = ++env->id_gen;
4748 } else if (fn->ret_type == RET_PTR_TO_ALLOC_MEM_OR_NULL) {
4749 mark_reg_known_zero(env, regs, BPF_REG_0);
4750 regs[BPF_REG_0].type = PTR_TO_MEM_OR_NULL;
4751 regs[BPF_REG_0].id = ++env->id_gen;
4752 regs[BPF_REG_0].mem_size = meta.mem_size;
4600 } else {4753 } else {
4601 verbose(env, "unknown return type %d of func %s#%d\n",4754 verbose(env, "unknown return type %d of func %s#%d\n",
4602 fn->ret_type, func_id_name(func_id), func_id);4755 fn->ret_type, func_id_name(func_id), func_id);
@@ -4606,7 +4759,7 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 3 unmodified lines
4606 if (is_ptr_cast_function(func_id)) {4759 if (is_ptr_cast_function(func_id)) {
4607 /* For release_reference() */4760 /* For release_reference() */
4608 regs[BPF_REG_0].ref_obj_id = meta.ref_obj_id;4761 regs[BPF_REG_0].ref_obj_id = meta.ref_obj_id;
4609 } else if (is_acquire_function(func_id)) {4762 } else if (is_acquire_function(func_id, meta.map_ptr)) {
4610 int id = acquire_reference_state(env, insn_idx);4763 int id = acquire_reference_state(env, insn_idx);
4611 4764
4612 if (id < 0)4765 if (id < 0)
@@ -4760,7 +4913,7 @@ static int retrieve_ptr_limit(const struct bpf_reg_state *ptr_reg, · 147 unmodified lines
4760static bool can_skip_alu_sanitation(const struct bpf_verifier_env *env,4913static bool can_skip_alu_sanitation(const struct bpf_verifier_env *env,
4761 const struct bpf_insn *insn)4914 const struct bpf_insn *insn)
4762{4915{
4763 return env->allow_ptr_leaks || BPF_SRC(insn->code) == BPF_K;4916 return env->bypass_spec_v1 || BPF_SRC(insn->code) == BPF_K;
4764}4917}
4765 4918
4766static int update_alu_sanitation_state(struct bpf_insn_aux_data *aux,4919static int update_alu_sanitation_state(struct bpf_insn_aux_data *aux,
@@ -5070,7 +5223,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, · 303 unmodified lines
5070 /* For unprivileged we require that resulting offset must be in bounds5223 /* For unprivileged we require that resulting offset must be in bounds
5071 * in order to be able to sanitize access later on.5224 * in order to be able to sanitize access later on.
5072 */5225 */
5073 if (!env->allow_ptr_leaks) {5226 if (!env->bypass_spec_v1) {
5074 if (dst_reg->type == PTR_TO_MAP_VALUE &&5227 if (dst_reg->type == PTR_TO_MAP_VALUE &&
5075 check_map_access(env, dst, dst_reg->off, 1, false)) {5228 check_map_access(env, dst, dst_reg->off, 1, false)) {
5076 verbose(env, "R%d pointer arithmetic of map value goes out of range, "5229 verbose(env, "R%d pointer arithmetic of map value goes out of range, "
@@ -5611,7 +5764,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env, · 534 unmodified lines
5611{5764{
5612 struct bpf_reg_state *regs = cur_regs(env);5765 struct bpf_reg_state *regs = cur_regs(env);
5613 u8 opcode = BPF_OP(insn->code);5766 u8 opcode = BPF_OP(insn->code);
5614 bool src_known, dst_known;5767 bool src_known;
5615 s64 smin_val, smax_val;5768 s64 smin_val, smax_val;
5616 u64 umin_val, umax_val;5769 u64 umin_val, umax_val;
5617 s32 s32_min_val, s32_max_val;5770 s32 s32_min_val, s32_max_val;
@@ -5633,7 +5786,6 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env, · 15 unmodified lines
5633 5786
5634 if (alu32) {5787 if (alu32) {
5635 src_known = tnum_subreg_is_const(src_reg.var_off);5788 src_known = tnum_subreg_is_const(src_reg.var_off);
5636 dst_known = tnum_subreg_is_const(dst_reg->var_off);
5637 if ((src_known &&5789 if ((src_known &&
5638 (s32_min_val != s32_max_val || u32_min_val != u32_max_val)) ||5790 (s32_min_val != s32_max_val || u32_min_val != u32_max_val)) ||
5639 s32_min_val > s32_max_val || u32_min_val > u32_max_val) {5791 s32_min_val > s32_max_val || u32_min_val > u32_max_val) {
@@ -5645,7 +5797,6 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env, · 5 unmodified lines
5645 }5797 }
5646 } else {5798 } else {
5647 src_known = tnum_is_const(src_reg.var_off);5799 src_known = tnum_is_const(src_reg.var_off);
5648 dst_known = tnum_is_const(dst_reg->var_off);
5649 if ((src_known &&5800 if ((src_known &&
5650 (smin_val != smax_val || umin_val != umax_val)) ||5801 (smin_val != smax_val || umin_val != umax_val)) ||
5651 smin_val > smax_val || umin_val > umax_val) {5802 smin_val > smax_val || umin_val > umax_val) {
@@ -6261,8 +6412,25 @@ static int is_branch64_taken(struct bpf_reg_state *reg, u64 val, u8 opcode) · 609 unmodified lines
6261static int is_branch_taken(struct bpf_reg_state *reg, u64 val, u8 opcode,6412static int is_branch_taken(struct bpf_reg_state *reg, u64 val, u8 opcode,
6262 bool is_jmp32)6413 bool is_jmp32)
6263{6414{
6264 if (__is_pointer_value(false, reg))6415 if (__is_pointer_value(false, reg)) {
6265 return -1;6416 if (!reg_type_not_null(reg->type))
6417 return -1;
6418
6419 /* If pointer is valid tests against zero will fail so we can
6420 * use this to direct branch taken.
6421 */
6422 if (val != 0)
6423 return -1;
6424
6425 switch (opcode) {
6426 case BPF_JEQ:
6427 return 0;
6428 case BPF_JNE:
6429 return 1;
6430 default:
6431 return -1;
6432 }
6433 }
6266 6434
6267 if (is_jmp32)6435 if (is_jmp32)
6268 return is_branch32_taken(reg, val, opcode);6436 return is_branch32_taken(reg, val, opcode);
@@ -6517,12 +6685,16 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state, · 248 unmodified lines
6517 if (is_null) {6685 if (is_null) {
6518 reg->type = SCALAR_VALUE;6686 reg->type = SCALAR_VALUE;
6519 } else if (reg->type == PTR_TO_MAP_VALUE_OR_NULL) {6687 } else if (reg->type == PTR_TO_MAP_VALUE_OR_NULL) {
6520 if (reg->map_ptr->inner_map_meta) {6688 const struct bpf_map *map = reg->map_ptr;
6689
6690 if (map->inner_map_meta) {
6521 reg->type = CONST_PTR_TO_MAP;6691 reg->type = CONST_PTR_TO_MAP;
6522 reg->map_ptr = reg->map_ptr->inner_map_meta;6692 reg->map_ptr = map->inner_map_meta;
6523 } else if (reg->map_ptr->map_type ==6693 } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) {
6524 BPF_MAP_TYPE_XSKMAP) {
6525 reg->type = PTR_TO_XDP_SOCK;6694 reg->type = PTR_TO_XDP_SOCK;
6695 } else if (map->map_type == BPF_MAP_TYPE_SOCKMAP ||
6696 map->map_type == BPF_MAP_TYPE_SOCKHASH) {
6697 reg->type = PTR_TO_SOCKET;
6526 } else {6698 } else {
6527 reg->type = PTR_TO_MAP_VALUE;6699 reg->type = PTR_TO_MAP_VALUE;
6528 }6700 }
@@ -6532,6 +6704,10 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state, · 3 unmodified lines
6532 reg->type = PTR_TO_SOCK_COMMON;6704 reg->type = PTR_TO_SOCK_COMMON;
6533 } else if (reg->type == PTR_TO_TCP_SOCK_OR_NULL) {6705 } else if (reg->type == PTR_TO_TCP_SOCK_OR_NULL) {
6534 reg->type = PTR_TO_TCP_SOCK;6706 reg->type = PTR_TO_TCP_SOCK;
6707 } else if (reg->type == PTR_TO_BTF_ID_OR_NULL) {
6708 reg->type = PTR_TO_BTF_ID;
6709 } else if (reg->type == PTR_TO_MEM_OR_NULL) {
6710 reg->type = PTR_TO_MEM;
6535 }6711 }
6536 if (is_null) {6712 if (is_null) {
6537 /* We don't need id and ref_obj_id from this point6713 /* We don't need id and ref_obj_id from this point
@@ -6755,7 +6931,11 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, · 217 unmodified lines
6755 }6931 }
6756 6932
6757 if (pred >= 0) {6933 if (pred >= 0) {
6758 err = mark_chain_precision(env, insn->dst_reg);6934 /* If we get here with a dst_reg pointer type it is because
6935 * above is_branch_taken() special cased the 0 comparison.
6936 */
6937 if (!__is_pointer_value(false, dst_reg))
6938 err = mark_chain_precision(env, insn->dst_reg);
6759 if (BPF_SRC(insn->code) == BPF_X && !err)6939 if (BPF_SRC(insn->code) == BPF_X && !err)
6760 err = mark_chain_precision(env, insn->src_reg);6940 err = mark_chain_precision(env, insn->src_reg);
6761 if (err)6941 if (err)
@@ -7041,7 +7221,11 @@ static int check_return_code(struct bpf_verifier_env *env) · 279 unmodified lines
7041 switch (env->prog->type) {7221 switch (env->prog->type) {
7042 case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:7222 case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
7043 if (env->prog->expected_attach_type == BPF_CGROUP_UDP4_RECVMSG ||7223 if (env->prog->expected_attach_type == BPF_CGROUP_UDP4_RECVMSG ||
7044 env->prog->expected_attach_type == BPF_CGROUP_UDP6_RECVMSG)7224 env->prog->expected_attach_type == BPF_CGROUP_UDP6_RECVMSG ||
7225 env->prog->expected_attach_type == BPF_CGROUP_INET4_GETPEERNAME ||
7226 env->prog->expected_attach_type == BPF_CGROUP_INET6_GETPEERNAME ||
7227 env->prog->expected_attach_type == BPF_CGROUP_INET4_GETSOCKNAME ||
7228 env->prog->expected_attach_type == BPF_CGROUP_INET6_GETSOCKNAME)
7045 range = tnum_range(1, 1);7229 range = tnum_range(1, 1);
7046 break;7230 break;
7047 case BPF_PROG_TYPE_CGROUP_SKB:7231 case BPF_PROG_TYPE_CGROUP_SKB:
@@ -7070,6 +7254,8 @@ static int check_return_code(struct bpf_verifier_env *env) · 22 unmodified lines
7070 case BPF_TRACE_RAW_TP:7254 case BPF_TRACE_RAW_TP:
7071 case BPF_MODIFY_RETURN:7255 case BPF_MODIFY_RETURN:
7072 return 0;7256 return 0;
7257 case BPF_TRACE_ITER:
7258 break;
7073 default:7259 default:
7074 return -ENOTSUPP;7260 return -ENOTSUPP;
7075 }7261 }
@@ -7206,7 +7392,7 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env, · 130 unmodified lines
7206 insn_stack[env->cfg.cur_stack++] = w;7392 insn_stack[env->cfg.cur_stack++] = w;
7207 return 1;7393 return 1;
7208 } else if ((insn_state[w] & 0xF0) == DISCOVERED) {7394 } else if ((insn_state[w] & 0xF0) == DISCOVERED) {
7209 if (loop_ok && env->allow_ptr_leaks)7395 if (loop_ok && env->bpf_capable)
7210 return 0;7396 return 0;
7211 verbose_linfo(env, t, "%d: ", t);7397 verbose_linfo(env, t, "%d: ", t);
7212 verbose_linfo(env, w, "%d: ", w);7398 verbose_linfo(env, w, "%d: ", w);
@@ -8315,7 +8501,7 @@ next: · 1102 unmodified lines
8315 if (env->max_states_per_insn < states_cnt)8501 if (env->max_states_per_insn < states_cnt)
8316 env->max_states_per_insn = states_cnt;8502 env->max_states_per_insn = states_cnt;
8317 8503
8318 if (!env->allow_ptr_leaks && states_cnt > BPF_COMPLEXITY_LIMIT_STATES)8504 if (!env->bpf_capable && states_cnt > BPF_COMPLEXITY_LIMIT_STATES)
8319 return push_jmp_history(env, cur);8505 return push_jmp_history(env, cur);
8320 8506
8321 if (!add_new_state)8507 if (!add_new_state)
@@ -8402,6 +8588,7 @@ static bool reg_type_mismatch_ok(enum bpf_reg_type type) · 80 unmodified lines
8402 case PTR_TO_TCP_SOCK_OR_NULL:8588 case PTR_TO_TCP_SOCK_OR_NULL:
8403 case PTR_TO_XDP_SOCK:8589 case PTR_TO_XDP_SOCK:
8404 case PTR_TO_BTF_ID:8590 case PTR_TO_BTF_ID:
8591 case PTR_TO_BTF_ID_OR_NULL:
8405 return false;8592 return false;
8406 default:8593 default:
8407 return true;8594 return true;
@@ -8428,6 +8615,7 @@ static bool reg_type_mismatch(enum bpf_reg_type src, enum bpf_reg_type prev) · 20 unmodified lines
8428 8615
8429static int do_check(struct bpf_verifier_env *env)8616static int do_check(struct bpf_verifier_env *env)
8430{8617{
8618 bool pop_log = !(env->log.level & BPF_LOG_LEVEL2);
8431 struct bpf_verifier_state *state = env->cur_state;8619 struct bpf_verifier_state *state = env->cur_state;
8432 struct bpf_insn *insns = env->prog->insnsi;8620 struct bpf_insn *insns = env->prog->insnsi;
8433 struct bpf_reg_state *regs;8621 struct bpf_reg_state *regs;
@@ -8704,7 +8892,7 @@ static int do_check(struct bpf_verifier_env *env) · 270 unmodified lines
8704process_bpf_exit:8892process_bpf_exit:
8705 update_branch_counts(env, env->cur_state);8893 update_branch_counts(env, env->cur_state);
8706 err = pop_stack(env, &prev_insn_idx,8894 err = pop_stack(env, &prev_insn_idx,
8707 &env->insn_idx);8895 &env->insn_idx, pop_log);
8708 if (err < 0) {8896 if (err < 0) {
8709 if (err != -ENOENT)8897 if (err != -ENOENT)
8710 return err;8898 return err;
@@ -9974,7 +10162,7 @@ static int fixup_bpf_calls(struct bpf_verifier_env *env) · 1263 unmodified lines
9974 insn->code = BPF_JMP | BPF_TAIL_CALL;10162 insn->code = BPF_JMP | BPF_TAIL_CALL;
9975 10163
9976 aux = &env->insn_aux_data[i + delta];10164 aux = &env->insn_aux_data[i + delta];
9977 if (env->allow_ptr_leaks && !expect_blinding &&10165 if (env->bpf_capable && !expect_blinding &&
9978 prog->jit_requested &&10166 prog->jit_requested &&
9979 !bpf_map_key_poisoned(aux) &&10167 !bpf_map_key_poisoned(aux) &&
9980 !bpf_map_ptr_poisoned(aux) &&10168 !bpf_map_ptr_poisoned(aux) &&
@@ -10227,6 +10415,7 @@ static void sanitize_insn_aux_data(struct bpf_verifier_env *env) · 246 unmodified lines
10227 10415
10228static int do_check_common(struct bpf_verifier_env *env, int subprog)10416static int do_check_common(struct bpf_verifier_env *env, int subprog)
10229{10417{
10418 bool pop_log = !(env->log.level & BPF_LOG_LEVEL2);
10230 struct bpf_verifier_state *state;10419 struct bpf_verifier_state *state;
10231 struct bpf_reg_state *regs;10420 struct bpf_reg_state *regs;
10232 int ret, i;10421 int ret, i;
@@ -10289,7 +10478,9 @@ out: · 56 unmodified lines
10289 free_verifier_state(env->cur_state, true);10478 free_verifier_state(env->cur_state, true);
10290 env->cur_state = NULL;10479 env->cur_state = NULL;
10291 }10480 }
10292 while (!pop_stack(env, NULL, NULL));10481 while (!pop_stack(env, NULL, NULL, false));
10482 if (!ret && pop_log)
10483 bpf_vlog_reset(&env->log, 0);
10293 free_states(env);10484 free_states(env);
10294 if (ret)10485 if (ret)
10295 /* clean aux data in case subprog was rejected */10486 /* clean aux data in case subprog was rejected */
@@ -10445,6 +10636,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 149 unmodified lines
10445 struct bpf_prog *tgt_prog = prog->aux->linked_prog;10636 struct bpf_prog *tgt_prog = prog->aux->linked_prog;
10446 u32 btf_id = prog->aux->attach_btf_id;10637 u32 btf_id = prog->aux->attach_btf_id;
10447 const char prefix[] = "btf_trace_";10638 const char prefix[] = "btf_trace_";
10639 struct btf_func_model fmodel;
10448 int ret = 0, subprog = -1, i;10640 int ret = 0, subprog = -1, i;
10449 struct bpf_trampoline *tr;10641 struct bpf_trampoline *tr;
10450 const struct btf_type *t;10642 const struct btf_type *t;
@@ -10586,6 +10778,22 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 135 unmodified lines
10586 prog->aux->attach_func_proto = t;10778 prog->aux->attach_func_proto = t;
10587 prog->aux->attach_btf_trace = true;10779 prog->aux->attach_btf_trace = true;
10588 return 0;10780 return 0;
10781 case BPF_TRACE_ITER:
10782 if (!btf_type_is_func(t)) {
10783 verbose(env, "attach_btf_id %u is not a function\n",
10784 btf_id);
10785 return -EINVAL;
10786 }
10787 t = btf_type_by_id(btf, t->type);
10788 if (!btf_type_is_func_proto(t))
10789 return -EINVAL;
10790 prog->aux->attach_func_name = tname;
10791 prog->aux->attach_func_proto = t;
10792 if (!bpf_iter_prog_supported(prog))
10793 return -EINVAL;
10794 ret = btf_distill_func_proto(&env->log, btf, t,
10795 tname, &fmodel);
10796 return ret;
10589 default:10797 default:
10590 if (!prog_extension)10798 if (!prog_extension)
10591 return -EINVAL;10799 return -EINVAL;
@@ -10696,7 +10904,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, · 104 unmodified lines
10696 env->insn_aux_data[i].orig_idx = i;10904 env->insn_aux_data[i].orig_idx = i;
10697 env->prog = *prog;10905 env->prog = *prog;
10698 env->ops = bpf_verifier_ops[env->prog->type];10906 env->ops = bpf_verifier_ops[env->prog->type];
10699 is_priv = capable(CAP_SYS_ADMIN);10907 is_priv = bpf_capable();
10700 10908
10701 if (!btf_vmlinux && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {10909 if (!btf_vmlinux && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {
10702 mutex_lock(&bpf_verifier_lock);10910 mutex_lock(&bpf_verifier_lock);
@@ -10737,7 +10945,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, · 34 unmodified lines
10737 if (attr->prog_flags & BPF_F_ANY_ALIGNMENT)10945 if (attr->prog_flags & BPF_F_ANY_ALIGNMENT)
10738 env->strict_alignment = false;10946 env->strict_alignment = false;
10739 10947
10740 env->allow_ptr_leaks = is_priv;10948 env->allow_ptr_leaks = bpf_allow_ptr_leaks();
10949 env->bypass_spec_v1 = bpf_bypass_spec_v1();
10950 env->bypass_spec_v4 = bpf_bypass_spec_v4();
10951 env->bpf_capable = bpf_capable();
10741 10952
10742 if (is_priv)10953 if (is_priv)
10743 env->test_state_freq = attr->prog_flags & BPF_F_TEST_STATE_FREQ;10954 env->test_state_freq = attr->prog_flags & BPF_F_TEST_STATE_FREQ;
kernel/bpf/xskmap.c+0−265
@@ -1,265 +0,0 @@
1// SPDX-License-Identifier: GPL-2.0
2/* XSKMAP used for AF_XDP sockets
3 * Copyright(c) 2018 Intel Corporation.
4 */
5
6#include <linux/bpf.h>
7#include <linux/capability.h>
8#include <net/xdp_sock.h>
9#include <linux/slab.h>
10#include <linux/sched.h>
11
12int xsk_map_inc(struct xsk_map *map)
13{
14 bpf_map_inc(&map->map);
15 return 0;
16}
17
18void xsk_map_put(struct xsk_map *map)
19{
20 bpf_map_put(&map->map);
21}
22
23static struct xsk_map_node *xsk_map_node_alloc(struct xsk_map *map,
24 struct xdp_sock **map_entry)
25{
26 struct xsk_map_node *node;
27 int err;
28
29 node = kzalloc(sizeof(*node), GFP_ATOMIC | __GFP_NOWARN);
30 if (!node)
31 return ERR_PTR(-ENOMEM);
32
33 err = xsk_map_inc(map);
34 if (err) {
35 kfree(node);
36 return ERR_PTR(err);
37 }
38
39 node->map = map;
40 node->map_entry = map_entry;
41 return node;
42}
43
44static void xsk_map_node_free(struct xsk_map_node *node)
45{
46 xsk_map_put(node->map);
47 kfree(node);
48}
49
50static void xsk_map_sock_add(struct xdp_sock *xs, struct xsk_map_node *node)
51{
52 spin_lock_bh(&xs->map_list_lock);
53 list_add_tail(&node->node, &xs->map_list);
54 spin_unlock_bh(&xs->map_list_lock);
55}
56
57static void xsk_map_sock_delete(struct xdp_sock *xs,
58 struct xdp_sock **map_entry)
59{
60 struct xsk_map_node *n, *tmp;
61
62 spin_lock_bh(&xs->map_list_lock);
63 list_for_each_entry_safe(n, tmp, &xs->map_list, node) {
64 if (map_entry == n->map_entry) {
65 list_del(&n->node);
66 xsk_map_node_free(n);
67 }
68 }
69 spin_unlock_bh(&xs->map_list_lock);
70}
71
72static struct bpf_map *xsk_map_alloc(union bpf_attr *attr)
73{
74 struct bpf_map_memory mem;
75 int err, numa_node;
76 struct xsk_map *m;
77 u64 size;
78
79 if (!capable(CAP_NET_ADMIN))
80 return ERR_PTR(-EPERM);
81
82 if (attr->max_entries == 0 || attr->key_size != 4 ||
83 attr->value_size != 4 ||
84 attr->map_flags & ~(BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY))
85 return ERR_PTR(-EINVAL);
86
87 numa_node = bpf_map_attr_numa_node(attr);
88 size = struct_size(m, xsk_map, attr->max_entries);
89
90 err = bpf_map_charge_init(&mem, size);
91 if (err < 0)
92 return ERR_PTR(err);
93
94 m = bpf_map_area_alloc(size, numa_node);
95 if (!m) {
96 bpf_map_charge_finish(&mem);
97 return ERR_PTR(-ENOMEM);
98 }
99
100 bpf_map_init_from_attr(&m->map, attr);
101 bpf_map_charge_move(&m->map.memory, &mem);
102 spin_lock_init(&m->lock);
103
104 return &m->map;
105}
106
107static void xsk_map_free(struct bpf_map *map)
108{
109 struct xsk_map *m = container_of(map, struct xsk_map, map);
110
111 bpf_clear_redirect_map(map);
112 synchronize_net();
113 bpf_map_area_free(m);
114}
115
116static int xsk_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
117{
118 struct xsk_map *m = container_of(map, struct xsk_map, map);
119 u32 index = key ? *(u32 *)key : U32_MAX;
120 u32 *next = next_key;
121
122 if (index >= m->map.max_entries) {
123 *next = 0;
124 return 0;
125 }
126
127 if (index == m->map.max_entries - 1)
128 return -ENOENT;
129 *next = index + 1;
130 return 0;
131}
132
133static u32 xsk_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
134{
135 const int ret = BPF_REG_0, mp = BPF_REG_1, index = BPF_REG_2;
136 struct bpf_insn *insn = insn_buf;
137
138 *insn++ = BPF_LDX_MEM(BPF_W, ret, index, 0);
139 *insn++ = BPF_JMP_IMM(BPF_JGE, ret, map->max_entries, 5);
140 *insn++ = BPF_ALU64_IMM(BPF_LSH, ret, ilog2(sizeof(struct xsk_sock *)));
141 *insn++ = BPF_ALU64_IMM(BPF_ADD, mp, offsetof(struct xsk_map, xsk_map));
142 *insn++ = BPF_ALU64_REG(BPF_ADD, ret, mp);
143 *insn++ = BPF_LDX_MEM(BPF_SIZEOF(struct xsk_sock *), ret, ret, 0);
144 *insn++ = BPF_JMP_IMM(BPF_JA, 0, 0, 1);
145 *insn++ = BPF_MOV64_IMM(ret, 0);
146 return insn - insn_buf;
147}
148
149static void *xsk_map_lookup_elem(struct bpf_map *map, void *key)
150{
151 WARN_ON_ONCE(!rcu_read_lock_held());
152 return __xsk_map_lookup_elem(map, *(u32 *)key);
153}
154
155static void *xsk_map_lookup_elem_sys_only(struct bpf_map *map, void *key)
156{
157 return ERR_PTR(-EOPNOTSUPP);
158}
159
160static int xsk_map_update_elem(struct bpf_map *map, void *key, void *value,
161 u64 map_flags)
162{
163 struct xsk_map *m = container_of(map, struct xsk_map, map);
164 struct xdp_sock *xs, *old_xs, **map_entry;
165 u32 i = *(u32 *)key, fd = *(u32 *)value;
166 struct xsk_map_node *node;
167 struct socket *sock;
168 int err;
169
170 if (unlikely(map_flags > BPF_EXIST))
171 return -EINVAL;
172 if (unlikely(i >= m->map.max_entries))
173 return -E2BIG;
174
175 sock = sockfd_lookup(fd, &err);
176 if (!sock)
177 return err;
178
179 if (sock->sk->sk_family != PF_XDP) {
180 sockfd_put(sock);
181 return -EOPNOTSUPP;
182 }
183
184 xs = (struct xdp_sock *)sock->sk;
185
186 if (!xsk_is_setup_for_bpf_map(xs)) {
187 sockfd_put(sock);
188 return -EOPNOTSUPP;
189 }
190
191 map_entry = &m->xsk_map[i];
192 node = xsk_map_node_alloc(m, map_entry);
193 if (IS_ERR(node)) {
194 sockfd_put(sock);
195 return PTR_ERR(node);
196 }
197
198 spin_lock_bh(&m->lock);
199 old_xs = READ_ONCE(*map_entry);
200 if (old_xs == xs) {
201 err = 0;
202 goto out;
203 } else if (old_xs && map_flags == BPF_NOEXIST) {
204 err = -EEXIST;
205 goto out;
206 } else if (!old_xs && map_flags == BPF_EXIST) {
207 err = -ENOENT;
208 goto out;
209 }
210 xsk_map_sock_add(xs, node);
211 WRITE_ONCE(*map_entry, xs);
212 if (old_xs)
213 xsk_map_sock_delete(old_xs, map_entry);
214 spin_unlock_bh(&m->lock);
215 sockfd_put(sock);
216 return 0;
217
218out:
219 spin_unlock_bh(&m->lock);
220 sockfd_put(sock);
221 xsk_map_node_free(node);
222 return err;
223}
224
225static int xsk_map_delete_elem(struct bpf_map *map, void *key)
226{
227 struct xsk_map *m = container_of(map, struct xsk_map, map);
228 struct xdp_sock *old_xs, **map_entry;
229 int k = *(u32 *)key;
230
231 if (k >= map->max_entries)
232 return -EINVAL;
233
234 spin_lock_bh(&m->lock);
235 map_entry = &m->xsk_map[k];
236 old_xs = xchg(map_entry, NULL);
237 if (old_xs)
238 xsk_map_sock_delete(old_xs, map_entry);
239 spin_unlock_bh(&m->lock);
240
241 return 0;
242}
243
244void xsk_map_try_sock_delete(struct xsk_map *map, struct xdp_sock *xs,
245 struct xdp_sock **map_entry)
246{
247 spin_lock_bh(&map->lock);
248 if (READ_ONCE(*map_entry) == xs) {
249 WRITE_ONCE(*map_entry, NULL);
250 xsk_map_sock_delete(xs, map_entry);
251 }
252 spin_unlock_bh(&map->lock);
253}
254
255const struct bpf_map_ops xsk_map_ops = {
256 .map_alloc = xsk_map_alloc,
257 .map_free = xsk_map_free,
258 .map_get_next_key = xsk_map_get_next_key,
259 .map_lookup_elem = xsk_map_lookup_elem,
260 .map_gen_lookup = xsk_map_gen_lookup,
261 .map_lookup_elem_sys_only = xsk_map_lookup_elem_sys_only,
262 .map_update_elem = xsk_map_update_elem,
263 .map_delete_elem = xsk_map_delete_elem,
264 .map_check_btf = map_check_no_btf,
265};
kernel/cgroup/cgroup.c+0−27
@@ -6508,33 +6508,6 @@ int cgroup_bpf_attach(struct cgroup *cgrp, · 6507 unmodified lines
6508 return ret;6508 return ret;
6509}6509}
6510 6510
6511int cgroup_bpf_replace(struct bpf_link *link, struct bpf_prog *old_prog,
6512 struct bpf_prog *new_prog)
6513{
6514 struct bpf_cgroup_link *cg_link;
6515 int ret;
6516
6517 if (link->ops != &bpf_cgroup_link_lops)
6518 return -EINVAL;
6519
6520 cg_link = container_of(link, struct bpf_cgroup_link, link);
6521
6522 mutex_lock(&cgroup_mutex);
6523 /* link might have been auto-released by dying cgroup, so fail */
6524 if (!cg_link->cgroup) {
6525 ret = -EINVAL;
6526 goto out_unlock;
6527 }
6528 if (old_prog && link->prog != old_prog) {
6529 ret = -EPERM;
6530 goto out_unlock;
6531 }
6532 ret = __cgroup_bpf_replace(cg_link->cgroup, cg_link, new_prog);
6533out_unlock:
6534 mutex_unlock(&cgroup_mutex);
6535 return ret;
6536}
6537
6538int cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog,6511int cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog,
6539 enum bpf_attach_type type)6512 enum bpf_attach_type type)
6540{6513{
kernel/events/callchain.c+1−1
@@ -236,7 +236,7 @@ exit_put: · 235 unmodified lines
236 * sysctl_perf_event_max_contexts_per_stack.236 * sysctl_perf_event_max_contexts_per_stack.
237 */237 */
238int perf_event_max_stack_handler(struct ctl_table *table, int write,238int perf_event_max_stack_handler(struct ctl_table *table, int write,
239 void __user *buffer, size_t *lenp, loff_t *ppos)239 void *buffer, size_t *lenp, loff_t *ppos)
240{240{
241 int *value = table->data;241 int *value = table->data;
242 int new_value = *value, ret;242 int new_value = *value, ret;
kernel/events/core.c+2−4
@@ -442,8 +442,7 @@ static void update_perf_cpu_limits(void) · 441 unmodified lines
442static bool perf_rotate_context(struct perf_cpu_context *cpuctx);442static bool perf_rotate_context(struct perf_cpu_context *cpuctx);
443 443
444int perf_proc_update_handler(struct ctl_table *table, int write,444int perf_proc_update_handler(struct ctl_table *table, int write,
445 void __user *buffer, size_t *lenp,445 void *buffer, size_t *lenp, loff_t *ppos)
446 loff_t *ppos)
447{446{
448 int ret;447 int ret;
449 int perf_cpu = sysctl_perf_cpu_time_max_percent;448 int perf_cpu = sysctl_perf_cpu_time_max_percent;
@@ -467,8 +466,7 @@ int perf_proc_update_handler(struct ctl_table *table, int write, · 17 unmodified lines
467int sysctl_perf_cpu_time_max_percent __read_mostly = DEFAULT_CPU_TIME_MAX_PERCENT;466int sysctl_perf_cpu_time_max_percent __read_mostly = DEFAULT_CPU_TIME_MAX_PERCENT;
468 467
469int perf_cpu_time_max_percent_handler(struct ctl_table *table, int write,468int perf_cpu_time_max_percent_handler(struct ctl_table *table, int write,
470 void __user *buffer, size_t *lenp,469 void *buffer, size_t *lenp, loff_t *ppos)
471 loff_t *ppos)
472{470{
473 int ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);471 int ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
474 472
kernel/kprobes.c+1−1
@@ -892,7 +892,7 @@ static void unoptimize_all_kprobes(void) · 891 unmodified lines
892static DEFINE_MUTEX(kprobe_sysctl_mutex);892static DEFINE_MUTEX(kprobe_sysctl_mutex);
893int sysctl_kprobes_optimization;893int sysctl_kprobes_optimization;
894int proc_kprobes_optimization_handler(struct ctl_table *table, int write,894int proc_kprobes_optimization_handler(struct ctl_table *table, int write,
895 void __user *buffer, size_t *length,895 void *buffer, size_t *length,
896 loff_t *ppos)896 loff_t *ppos)
897{897{
898 int ret;898 int ret;
kernel/latencytop.c+2−2
@@ -269,8 +269,8 @@ static int __init init_lstats_procfs(void) · 268 unmodified lines
269 return 0;269 return 0;
270}270}
271 271
272int sysctl_latencytop(struct ctl_table *table, int write,272int sysctl_latencytop(struct ctl_table *table, int write, void *buffer,
273 void __user *buffer, size_t *lenp, loff_t *ppos)273 size_t *lenp, loff_t *ppos)
274{274{
275 int err;275 int err;
276 276
kernel/module.c+3−0
@@ -4,6 +4,9 @@ · 3 unmodified lines
4 Copyright (C) 2001 Rusty Russell, 2002, 2010 Rusty Russell IBM.4 Copyright (C) 2001 Rusty Russell, 2002, 2010 Rusty Russell IBM.
5 5
6*/6*/
7
8#define INCLUDE_VERMAGIC
9
7#include <linux/export.h>10#include <linux/export.h>
8#include <linux/extable.h>11#include <linux/extable.h>
9#include <linux/moduleloader.h>12#include <linux/moduleloader.h>
kernel/pid_namespace.c+1−1
@@ -263,7 +263,7 @@ void zap_pid_ns_processes(struct pid_namespace *pid_ns) · 262 unmodified lines
263 263
264#ifdef CONFIG_CHECKPOINT_RESTORE264#ifdef CONFIG_CHECKPOINT_RESTORE
265static int pid_ns_ctl_handler(struct ctl_table *table, int write,265static int pid_ns_ctl_handler(struct ctl_table *table, int write,
266 void __user *buffer, size_t *lenp, loff_t *ppos)266 void *buffer, size_t *lenp, loff_t *ppos)
267{267{
268 struct pid_namespace *pid_ns = task_active_pid_ns(current);268 struct pid_namespace *pid_ns = task_active_pid_ns(current);
269 struct ctl_table tmp = *table;269 struct ctl_table tmp = *table;
kernel/printk/printk.c+1−1
@@ -173,7 +173,7 @@ __setup("printk.devkmsg=", control_devkmsg); · 172 unmodified lines
173char devkmsg_log_str[DEVKMSG_STR_MAX_SIZE] = "ratelimit";173char devkmsg_log_str[DEVKMSG_STR_MAX_SIZE] = "ratelimit";
174 174
175int devkmsg_sysctl_set_loglvl(struct ctl_table *table, int write,175int devkmsg_sysctl_set_loglvl(struct ctl_table *table, int write,
176 void __user *buffer, size_t *lenp, loff_t *ppos)176 void *buffer, size_t *lenp, loff_t *ppos)
177{177{
178 char old_str[DEVKMSG_STR_MAX_SIZE];178 char old_str[DEVKMSG_STR_MAX_SIZE];
179 unsigned int old;179 unsigned int old;
kernel/sched/core.c+4−5
@@ -1111,8 +1111,7 @@ static void uclamp_update_root_tg(void) { } · 1110 unmodified lines
1111#endif1111#endif
1112 1112
1113int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,1113int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,
1114 void __user *buffer, size_t *lenp,1114 void *buffer, size_t *lenp, loff_t *ppos)
1115 loff_t *ppos)
1116{1115{
1117 bool update_root_tg = false;1116 bool update_root_tg = false;
1118 int old_min, old_max;1117 int old_min, old_max;
@@ -2782,7 +2781,7 @@ void set_numabalancing_state(bool enabled) · 1663 unmodified lines
2782 2781
2783#ifdef CONFIG_PROC_SYSCTL2782#ifdef CONFIG_PROC_SYSCTL
2784int sysctl_numa_balancing(struct ctl_table *table, int write,2783int sysctl_numa_balancing(struct ctl_table *table, int write,
2785 void __user *buffer, size_t *lenp, loff_t *ppos)2784 void *buffer, size_t *lenp, loff_t *ppos)
2786{2785{
2787 struct ctl_table t;2786 struct ctl_table t;
2788 int err;2787 int err;
@@ -2856,8 +2855,8 @@ static void __init init_schedstats(void) · 67 unmodified lines
2856}2855}
2857 2856
2858#ifdef CONFIG_PROC_SYSCTL2857#ifdef CONFIG_PROC_SYSCTL
2859int sysctl_schedstats(struct ctl_table *table, int write,2858int sysctl_schedstats(struct ctl_table *table, int write, void *buffer,
2860 void __user *buffer, size_t *lenp, loff_t *ppos)2859 size_t *lenp, loff_t *ppos)
2861{2860{
2862 struct ctl_table t;2861 struct ctl_table t;
2863 int err;2862 int err;
kernel/sched/fair.c+1−2
@@ -645,8 +645,7 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq) · 644 unmodified lines
645 */645 */
646 646
647int sched_proc_update_handler(struct ctl_table *table, int write,647int sched_proc_update_handler(struct ctl_table *table, int write,
648 void __user *buffer, size_t *lenp,648 void *buffer, size_t *lenp, loff_t *ppos)
649 loff_t *ppos)
650{649{
651 int ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);650 int ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
652 unsigned int factor = get_update_sysctl_factor();651 unsigned int factor = get_update_sysctl_factor();
kernel/sched/rt.c+4−6
@@ -2724,9 +2724,8 @@ static void sched_rt_do_global(void) · 2723 unmodified lines
2724 def_rt_bandwidth.rt_period = ns_to_ktime(global_rt_period());2724 def_rt_bandwidth.rt_period = ns_to_ktime(global_rt_period());
2725}2725}
2726 2726
2727int sched_rt_handler(struct ctl_table *table, int write,2727int sched_rt_handler(struct ctl_table *table, int write, void *buffer,
2728 void __user *buffer, size_t *lenp,2728 size_t *lenp, loff_t *ppos)
2729 loff_t *ppos)
2730{2729{
2731 int old_period, old_runtime;2730 int old_period, old_runtime;
2732 static DEFINE_MUTEX(mutex);2731 static DEFINE_MUTEX(mutex);
@@ -2764,9 +2763,8 @@ undo: · 31 unmodified lines
2764 return ret;2763 return ret;
2765}2764}
2766 2765
2767int sched_rr_handler(struct ctl_table *table, int write,2766int sched_rr_handler(struct ctl_table *table, int write, void *buffer,
2768 void __user *buffer, size_t *lenp,2767 size_t *lenp, loff_t *ppos)
2769 loff_t *ppos)
2770{2768{
2771 int ret;2769 int ret;
2772 static DEFINE_MUTEX(mutex);2770 static DEFINE_MUTEX(mutex);
kernel/sched/topology.c+1−1
@@ -199,7 +199,7 @@ bool sched_energy_update; · 198 unmodified lines
199 199
200#ifdef CONFIG_PROC_SYSCTL200#ifdef CONFIG_PROC_SYSCTL
201int sched_energy_aware_handler(struct ctl_table *table, int write,201int sched_energy_aware_handler(struct ctl_table *table, int write,
202 void __user *buffer, size_t *lenp, loff_t *ppos)202 void *buffer, size_t *lenp, loff_t *ppos)
203{203{
204 int ret, state;204 int ret, state;
205 205
kernel/seccomp.c+1−1
@@ -1776,7 +1776,7 @@ static void audit_actions_logged(u32 actions_logged, u32 old_actions_logged, · 1775 unmodified lines
1776}1776}
1777 1777
1778static int seccomp_actions_logged_handler(struct ctl_table *ro_table, int write,1778static int seccomp_actions_logged_handler(struct ctl_table *ro_table, int write,
1779 void __user *buffer, size_t *lenp,1779 void *buffer, size_t *lenp,
1780 loff_t *ppos)1780 loff_t *ppos)
1781{1781{
1782 int ret;1782 int ret;
kernel/sysctl.c+1891−1990
@@ -68,6 +68,9 @@ · 67 unmodified lines
68#include <linux/bpf.h>68#include <linux/bpf.h>
69#include <linux/mount.h>69#include <linux/mount.h>
70#include <linux/userfaultfd_k.h>70#include <linux/userfaultfd_k.h>
71#include <linux/coredump.h>
72#include <linux/latencytop.h>
73#include <linux/pid.h>
71 74
72#include "../lib/kstrtox.h"75#include "../lib/kstrtox.h"
73 76
@@ -103,22 +106,6 @@ · 29 unmodified lines
103 106
104#if defined(CONFIG_SYSCTL)107#if defined(CONFIG_SYSCTL)
105 108
106/* External variables not in a header file. */
107extern int suid_dumpable;
108#ifdef CONFIG_COREDUMP
109extern int core_uses_pid;
110extern char core_pattern[];
111extern unsigned int core_pipe_limit;
112#endif
113extern int pid_max;
114extern int pid_max_min, pid_max_max;
115extern int percpu_pagelist_fraction;
116extern int latencytop_enabled;
117extern unsigned int sysctl_nr_open_min, sysctl_nr_open_max;
118#ifndef CONFIG_MMU
119extern int sysctl_nr_trim_pages;
120#endif
121
122/* Constants used for minimum and maximum */109/* Constants used for minimum and maximum */
123#ifdef CONFIG_LOCKUP_DETECTOR110#ifdef CONFIG_LOCKUP_DETECTOR
124static int sixty = 60;111static int sixty = 60;
@@ -160,24 +147,6 @@ static unsigned long hung_task_timeout_max = (LONG_MAX/HZ); · 35 unmodified lines
160#ifdef CONFIG_INOTIFY_USER147#ifdef CONFIG_INOTIFY_USER
161#include <linux/inotify.h>148#include <linux/inotify.h>
162#endif149#endif
163#ifdef CONFIG_SPARC
164#endif
165
166#ifdef CONFIG_PARISC
167extern int pwrsw_enabled;
168#endif
169
170#ifdef CONFIG_SYSCTL_ARCH_UNALIGN_ALLOW
171extern int unaligned_enabled;
172#endif
173
174#ifdef CONFIG_IA64
175extern int unaligned_dump_stack;
176#endif
177
178#ifdef CONFIG_SYSCTL_ARCH_UNALIGN_NO_WARN
179extern int no_unaligned_warning;
180#endif
181 150
182#ifdef CONFIG_PROC_SYSCTL151#ifdef CONFIG_PROC_SYSCTL
183 152
@@ -207,87 +176,13 @@ enum sysctl_writes_mode { · 23 unmodified lines
207};176};
208 177
209static enum sysctl_writes_mode sysctl_writes_strict = SYSCTL_WRITES_STRICT;178static enum sysctl_writes_mode sysctl_writes_strict = SYSCTL_WRITES_STRICT;
210 179#endif /* CONFIG_PROC_SYSCTL */
211static int proc_do_cad_pid(struct ctl_table *table, int write,
212 void __user *buffer, size_t *lenp, loff_t *ppos);
213static int proc_taint(struct ctl_table *table, int write,
214 void __user *buffer, size_t *lenp, loff_t *ppos);
215#ifdef CONFIG_COMPACTION
216static int proc_dointvec_minmax_warn_RT_change(struct ctl_table *table,
217 int write, void __user *buffer,
218 size_t *lenp, loff_t *ppos);
219#endif
220#endif
221
222#ifdef CONFIG_PRINTK
223static int proc_dointvec_minmax_sysadmin(struct ctl_table *table, int write,
224 void __user *buffer, size_t *lenp, loff_t *ppos);
225#endif
226
227static int proc_dointvec_minmax_coredump(struct ctl_table *table, int write,
228 void __user *buffer, size_t *lenp, loff_t *ppos);
229#ifdef CONFIG_COREDUMP
230static int proc_dostring_coredump(struct ctl_table *table, int write,
231 void __user *buffer, size_t *lenp, loff_t *ppos);
232#endif
233static int proc_dopipe_max_size(struct ctl_table *table, int write,
234 void __user *buffer, size_t *lenp, loff_t *ppos);
235
236#ifdef CONFIG_MAGIC_SYSRQ
237static int sysrq_sysctl_handler(struct ctl_table *table, int write,
238 void __user *buffer, size_t *lenp, loff_t *ppos);
239#endif
240
241static struct ctl_table kern_table[];
242static struct ctl_table vm_table[];
243static struct ctl_table fs_table[];
244static struct ctl_table debug_table[];
245static struct ctl_table dev_table[];
246extern struct ctl_table random_table[];
247#ifdef CONFIG_EPOLL
248extern struct ctl_table epoll_table[];
249#endif
250
251#ifdef CONFIG_FW_LOADER_USER_HELPER
252extern struct ctl_table firmware_config_table[];
253#endif
254 180
255#if defined(HAVE_ARCH_PICK_MMAP_LAYOUT) || \181#if defined(HAVE_ARCH_PICK_MMAP_LAYOUT) || \
256 defined(CONFIG_ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT)182 defined(CONFIG_ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT)
257int sysctl_legacy_va_layout;183int sysctl_legacy_va_layout;
258#endif184#endif
259 185
260/* The default sysctl tables: */
261
262static struct ctl_table sysctl_base_table[] = {
263 {
264 .procname = "kernel",
265 .mode = 0555,
266 .child = kern_table,
267 },
268 {
269 .procname = "vm",
270 .mode = 0555,
271 .child = vm_table,
272 },
273 {
274 .procname = "fs",
275 .mode = 0555,
276 .child = fs_table,
277 },
278 {
279 .procname = "debug",
280 .mode = 0555,
281 .child = debug_table,
282 },
283 {
284 .procname = "dev",
285 .mode = 0555,
286 .child = dev_table,
287 },
288 { }
289};
290
291#ifdef CONFIG_SCHED_DEBUG186#ifdef CONFIG_SCHED_DEBUG
292static int min_sched_granularity_ns = 100000; /* 100 usecs */187static int min_sched_granularity_ns = 100000; /* 100 usecs */
293static int max_sched_granularity_ns = NSEC_PER_SEC; /* 1 second */188static int max_sched_granularity_ns = NSEC_PER_SEC; /* 1 second */
@@ -304,1687 +199,53 @@ static int min_extfrag_threshold; · 10 unmodified lines
304static int max_extfrag_threshold = 1000;199static int max_extfrag_threshold = 1000;
305#endif200#endif
306 201
307static struct ctl_table kern_table[] = {202#endif /* CONFIG_SYSCTL */
308 {203
309 .procname = "sched_child_runs_first",204#if defined(CONFIG_BPF_SYSCALL) && defined(CONFIG_SYSCTL)
310 .data = &sysctl_sched_child_runs_first,205static int bpf_stats_handler(struct ctl_table *table, int write,
311 .maxlen = sizeof(unsigned int),206 void __user *buffer, size_t *lenp,
312 .mode = 0644,207 loff_t *ppos)
313 .proc_handler = proc_dointvec,208{
314 },209 struct static_key *key = (struct static_key *)table->data;
315#ifdef CONFIG_SCHED_DEBUG210 static int saved_val;
316 {211 int val, ret;
317 .procname = "sched_min_granularity_ns",212 struct ctl_table tmp = {
318 .data = &sysctl_sched_min_granularity,213 .data = &val,
319 .maxlen = sizeof(unsigned int),214 .maxlen = sizeof(val),
320 .mode = 0644,215 .mode = table->mode,
321 .proc_handler = sched_proc_update_handler,216 .extra1 = SYSCTL_ZERO,
322 .extra1 = &min_sched_granularity_ns,217 .extra2 = SYSCTL_ONE,
323 .extra2 = &max_sched_granularity_ns,218 };
324 },219
325 {220 if (write && !capable(CAP_SYS_ADMIN))
326 .procname = "sched_latency_ns",221 return -EPERM;
327 .data = &sysctl_sched_latency,222
328 .maxlen = sizeof(unsigned int),223 mutex_lock(&bpf_stats_enabled_mutex);
329 .mode = 0644,224 val = saved_val;
330 .proc_handler = sched_proc_update_handler,225 ret = proc_dointvec_minmax(&tmp, write, buffer, lenp, ppos);
331 .extra1 = &min_sched_granularity_ns,226 if (write && !ret && val != saved_val) {
332 .extra2 = &max_sched_granularity_ns,227 if (val)
333 },228 static_key_slow_inc(key);
334 {229 else
335 .procname = "sched_wakeup_granularity_ns",230 static_key_slow_dec(key);
336 .data = &sysctl_sched_wakeup_granularity,231 saved_val = val;
337 .maxlen = sizeof(unsigned int),232 }
338 .mode = 0644,233 mutex_unlock(&bpf_stats_enabled_mutex);
339 .proc_handler = sched_proc_update_handler,234 return ret;
340 .extra1 = &min_wakeup_granularity_ns,235}
341 .extra2 = &max_wakeup_granularity_ns,
342 },
343#ifdef CONFIG_SMP
344 {
345 .procname = "sched_tunable_scaling",
346 .data = &sysctl_sched_tunable_scaling,
347 .maxlen = sizeof(enum sched_tunable_scaling),
348 .mode = 0644,
349 .proc_handler = sched_proc_update_handler,
350 .extra1 = &min_sched_tunable_scaling,
351 .extra2 = &max_sched_tunable_scaling,
352 },
353 {
354 .procname = "sched_migration_cost_ns",
355 .data = &sysctl_sched_migration_cost,
356 .maxlen = sizeof(unsigned int),
357 .mode = 0644,
358 .proc_handler = proc_dointvec,
359 },
360 {
361 .procname = "sched_nr_migrate",
362 .data = &sysctl_sched_nr_migrate,
363 .maxlen = sizeof(unsigned int),
364 .mode = 0644,
365 .proc_handler = proc_dointvec,
366 },
367#ifdef CONFIG_SCHEDSTATS
368 {
369 .procname = "sched_schedstats",
370 .data = NULL,
371 .maxlen = sizeof(unsigned int),
372 .mode = 0644,
373 .proc_handler = sysctl_schedstats,
374 .extra1 = SYSCTL_ZERO,
375 .extra2 = SYSCTL_ONE,
376 },
377#endif /* CONFIG_SCHEDSTATS */
378#endif /* CONFIG_SMP */
379#ifdef CONFIG_NUMA_BALANCING
380 {
381 .procname = "numa_balancing_scan_delay_ms",
382 .data = &sysctl_numa_balancing_scan_delay,
383 .maxlen = sizeof(unsigned int),
384 .mode = 0644,
385 .proc_handler = proc_dointvec,
386 },
387 {
388 .procname = "numa_balancing_scan_period_min_ms",
389 .data = &sysctl_numa_balancing_scan_period_min,
390 .maxlen = sizeof(unsigned int),
391 .mode = 0644,
392 .proc_handler = proc_dointvec,
393 },
394 {
395 .procname = "numa_balancing_scan_period_max_ms",
396 .data = &sysctl_numa_balancing_scan_period_max,
397 .maxlen = sizeof(unsigned int),
398 .mode = 0644,
399 .proc_handler = proc_dointvec,
400 },
401 {
402 .procname = "numa_balancing_scan_size_mb",
403 .data = &sysctl_numa_balancing_scan_size,
404 .maxlen = sizeof(unsigned int),
405 .mode = 0644,
406 .proc_handler = proc_dointvec_minmax,
407 .extra1 = SYSCTL_ONE,
408 },
409 {
410 .procname = "numa_balancing",
411 .data = NULL, /* filled in by handler */
412 .maxlen = sizeof(unsigned int),
413 .mode = 0644,
414 .proc_handler = sysctl_numa_balancing,
415 .extra1 = SYSCTL_ZERO,
416 .extra2 = SYSCTL_ONE,
417 },
418#endif /* CONFIG_NUMA_BALANCING */
419#endif /* CONFIG_SCHED_DEBUG */
420 {
421 .procname = "sched_rt_period_us",
422 .data = &sysctl_sched_rt_period,
423 .maxlen = sizeof(unsigned int),
424 .mode = 0644,
425 .proc_handler = sched_rt_handler,
426 },
427 {
428 .procname = "sched_rt_runtime_us",
429 .data = &sysctl_sched_rt_runtime,
430 .maxlen = sizeof(int),
431 .mode = 0644,
432 .proc_handler = sched_rt_handler,
433 },
434 {
435 .procname = "sched_rr_timeslice_ms",
436 .data = &sysctl_sched_rr_timeslice,
437 .maxlen = sizeof(int),
438 .mode = 0644,
439 .proc_handler = sched_rr_handler,
440 },
441#ifdef CONFIG_UCLAMP_TASK
442 {
443 .procname = "sched_util_clamp_min",
444 .data = &sysctl_sched_uclamp_util_min,
445 .maxlen = sizeof(unsigned int),
446 .mode = 0644,
447 .proc_handler = sysctl_sched_uclamp_handler,
448 },
449 {
450 .procname = "sched_util_clamp_max",
451 .data = &sysctl_sched_uclamp_util_max,
452 .maxlen = sizeof(unsigned int),
453 .mode = 0644,
454 .proc_handler = sysctl_sched_uclamp_handler,
455 },
456#endif
457#ifdef CONFIG_SCHED_AUTOGROUP
458 {
459 .procname = "sched_autogroup_enabled",
460 .data = &sysctl_sched_autogroup_enabled,
461 .maxlen = sizeof(unsigned int),
462 .mode = 0644,
463 .proc_handler = proc_dointvec_minmax,
464 .extra1 = SYSCTL_ZERO,
465 .extra2 = SYSCTL_ONE,
466 },
467#endif
468#ifdef CONFIG_CFS_BANDWIDTH
469 {
470 .procname = "sched_cfs_bandwidth_slice_us",
471 .data = &sysctl_sched_cfs_bandwidth_slice,
472 .maxlen = sizeof(unsigned int),
473 .mode = 0644,
474 .proc_handler = proc_dointvec_minmax,
475 .extra1 = SYSCTL_ONE,
476 },
477#endif
478#if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL)
479 {
480 .procname = "sched_energy_aware",
481 .data = &sysctl_sched_energy_aware,
482 .maxlen = sizeof(unsigned int),
483 .mode = 0644,
484 .proc_handler = sched_energy_aware_handler,
485 .extra1 = SYSCTL_ZERO,
486 .extra2 = SYSCTL_ONE,
487 },
488#endif
489#ifdef CONFIG_PROVE_LOCKING
490 {
491 .procname = "prove_locking",
492 .data = &prove_locking,
493 .maxlen = sizeof(int),
494 .mode = 0644,
495 .proc_handler = proc_dointvec,
496 },
497#endif
498#ifdef CONFIG_LOCK_STAT
499 {
500 .procname = "lock_stat",
501 .data = &lock_stat,
502 .maxlen = sizeof(int),
503 .mode = 0644,
504 .proc_handler = proc_dointvec,
505 },
506#endif
507 {
508 .procname = "panic",
509 .data = &panic_timeout,
510 .maxlen = sizeof(int),
511 .mode = 0644,
512 .proc_handler = proc_dointvec,
513 },
514#ifdef CONFIG_COREDUMP
515 {
516 .procname = "core_uses_pid",
517 .data = &core_uses_pid,
518 .maxlen = sizeof(int),
519 .mode = 0644,
520 .proc_handler = proc_dointvec,
521 },
522 {
523 .procname = "core_pattern",
524 .data = core_pattern,
525 .maxlen = CORENAME_MAX_SIZE,
526 .mode = 0644,
527 .proc_handler = proc_dostring_coredump,
528 },
529 {
530 .procname = "core_pipe_limit",
531 .data = &core_pipe_limit,
532 .maxlen = sizeof(unsigned int),
533 .mode = 0644,
534 .proc_handler = proc_dointvec,
535 },
536#endif236#endif
537#ifdef CONFIG_PROC_SYSCTL237
538 {238/*
539 .procname = "tainted",239 * /proc/sys support
540 .maxlen = sizeof(long),240 */
541 .mode = 0644,241
542 .proc_handler = proc_taint,
543 },
544 {
545 .procname = "sysctl_writes_strict",
546 .data = &sysctl_writes_strict,
547 .maxlen = sizeof(int),
548 .mode = 0644,
549 .proc_handler = proc_dointvec_minmax,
550 .extra1 = &neg_one,
551 .extra2 = SYSCTL_ONE,
552 },
553#endif
554#ifdef CONFIG_LATENCYTOP
555 {
556 .procname = "latencytop",
557 .data = &latencytop_enabled,
558 .maxlen = sizeof(int),
559 .mode = 0644,
560 .proc_handler = sysctl_latencytop,
561 },
562#endif
563#ifdef CONFIG_BLK_DEV_INITRD
564 {
565 .procname = "real-root-dev",
566 .data = &real_root_dev,
567 .maxlen = sizeof(int),
568 .mode = 0644,
569 .proc_handler = proc_dointvec,
570 },
571#endif
572 {
573 .procname = "print-fatal-signals",
574 .data = &print_fatal_signals,
575 .maxlen = sizeof(int),
576 .mode = 0644,
577 .proc_handler = proc_dointvec,
578 },
579#ifdef CONFIG_SPARC
580 {
581 .procname = "reboot-cmd",
582 .data = reboot_command,
583 .maxlen = 256,
584 .mode = 0644,
585 .proc_handler = proc_dostring,
586 },
587 {
588 .procname = "stop-a",
589 .data = &stop_a_enabled,
590 .maxlen = sizeof (int),
591 .mode = 0644,
592 .proc_handler = proc_dointvec,
593 },
594 {
595 .procname = "scons-poweroff",
596 .data = &scons_pwroff,
597 .maxlen = sizeof (int),
598 .mode = 0644,
599 .proc_handler = proc_dointvec,
600 },
601#endif
602#ifdef CONFIG_SPARC64
603 {
604 .procname = "tsb-ratio",
605 .data = &sysctl_tsb_ratio,
606 .maxlen = sizeof (int),
607 .mode = 0644,
608 .proc_handler = proc_dointvec,
609 },
610#endif
611#ifdef CONFIG_PARISC
612 {
613 .procname = "soft-power",
614 .data = &pwrsw_enabled,
615 .maxlen = sizeof (int),
616 .mode = 0644,
617 .proc_handler = proc_dointvec,
618 },
619#endif
620#ifdef CONFIG_SYSCTL_ARCH_UNALIGN_ALLOW
621 {
622 .procname = "unaligned-trap",
623 .data = &unaligned_enabled,
624 .maxlen = sizeof (int),
625 .mode = 0644,
626 .proc_handler = proc_dointvec,
627 },
628#endif
629 {
630 .procname = "ctrl-alt-del",
631 .data = &C_A_D,
632 .maxlen = sizeof(int),
633 .mode = 0644,
634 .proc_handler = proc_dointvec,
635 },
636#ifdef CONFIG_FUNCTION_TRACER
637 {
638 .procname = "ftrace_enabled",
639 .data = &ftrace_enabled,
640 .maxlen = sizeof(int),
641 .mode = 0644,
642 .proc_handler = ftrace_enable_sysctl,
643 },
644#endif
645#ifdef CONFIG_STACK_TRACER
646 {
647 .procname = "stack_tracer_enabled",
648 .data = &stack_tracer_enabled,
649 .maxlen = sizeof(int),
650 .mode = 0644,
651 .proc_handler = stack_trace_sysctl,
652 },
653#endif
654#ifdef CONFIG_TRACING
655 {
656 .procname = "ftrace_dump_on_oops",
657 .data = &ftrace_dump_on_oops,
658 .maxlen = sizeof(int),
659 .mode = 0644,
660 .proc_handler = proc_dointvec,
661 },
662 {
663 .procname = "traceoff_on_warning",
664 .data = &__disable_trace_on_warning,
665 .maxlen = sizeof(__disable_trace_on_warning),
666 .mode = 0644,
667 .proc_handler = proc_dointvec,
668 },
669 {
670 .procname = "tracepoint_printk",
671 .data = &tracepoint_printk,
672 .maxlen = sizeof(tracepoint_printk),
673 .mode = 0644,
674 .proc_handler = tracepoint_printk_sysctl,
675 },
676#endif
677#ifdef CONFIG_KEXEC_CORE
678 {
679 .procname = "kexec_load_disabled",
680 .data = &kexec_load_disabled,
681 .maxlen = sizeof(int),
682 .mode = 0644,
683 /* only handle a transition from default "0" to "1" */
684 .proc_handler = proc_dointvec_minmax,
685 .extra1 = SYSCTL_ONE,
686 .extra2 = SYSCTL_ONE,
687 },
688#endif
689#ifdef CONFIG_MODULES
690 {
691 .procname = "modprobe",
692 .data = &modprobe_path,
693 .maxlen = KMOD_PATH_LEN,
694 .mode = 0644,
695 .proc_handler = proc_dostring,
696 },
697 {
698 .procname = "modules_disabled",
699 .data = &modules_disabled,
700 .maxlen = sizeof(int),
701 .mode = 0644,
702 /* only handle a transition from default "0" to "1" */
703 .proc_handler = proc_dointvec_minmax,
704 .extra1 = SYSCTL_ONE,
705 .extra2 = SYSCTL_ONE,
706 },
707#endif
708#ifdef CONFIG_UEVENT_HELPER
709 {
710 .procname = "hotplug",
711 .data = &uevent_helper,
712 .maxlen = UEVENT_HELPER_PATH_LEN,
713 .mode = 0644,
714 .proc_handler = proc_dostring,
715 },
716#endif
717#ifdef CONFIG_CHR_DEV_SG
718 {
719 .procname = "sg-big-buff",
720 .data = &sg_big_buff,
721 .maxlen = sizeof (int),
722 .mode = 0444,
723 .proc_handler = proc_dointvec,
724 },
725#endif
726#ifdef CONFIG_BSD_PROCESS_ACCT
727 {
728 .procname = "acct",
729 .data = &acct_parm,
730 .maxlen = 3*sizeof(int),
731 .mode = 0644,
732 .proc_handler = proc_dointvec,
733 },
734#endif
735#ifdef CONFIG_MAGIC_SYSRQ
736 {
737 .procname = "sysrq",
738 .data = NULL,
739 .maxlen = sizeof (int),
740 .mode = 0644,
741 .proc_handler = sysrq_sysctl_handler,
742 },
743#endif
744#ifdef CONFIG_PROC_SYSCTL
745 {
746 .procname = "cad_pid",
747 .data = NULL,
748 .maxlen = sizeof (int),
749 .mode = 0600,
750 .proc_handler = proc_do_cad_pid,
751 },
752#endif
753 {
754 .procname = "threads-max",
755 .data = NULL,
756 .maxlen = sizeof(int),
757 .mode = 0644,
758 .proc_handler = sysctl_max_threads,
759 },
760 {
761 .procname = "random",
762 .mode = 0555,
763 .child = random_table,
764 },
765 {
766 .procname = "usermodehelper",
767 .mode = 0555,
768 .child = usermodehelper_table,
769 },
770#ifdef CONFIG_FW_LOADER_USER_HELPER
771 {
772 .procname = "firmware_config",
773 .mode = 0555,
774 .child = firmware_config_table,
775 },
776#endif
777 {
778 .procname = "overflowuid",
779 .data = &overflowuid,
780 .maxlen = sizeof(int),
781 .mode = 0644,
782 .proc_handler = proc_dointvec_minmax,
783 .extra1 = &minolduid,
784 .extra2 = &maxolduid,
785 },
786 {
787 .procname = "overflowgid",
788 .data = &overflowgid,
789 .maxlen = sizeof(int),
790 .mode = 0644,
791 .proc_handler = proc_dointvec_minmax,
792 .extra1 = &minolduid,
793 .extra2 = &maxolduid,
794 },
795#ifdef CONFIG_S390
796 {
797 .procname = "userprocess_debug",
798 .data = &show_unhandled_signals,
799 .maxlen = sizeof(int),
800 .mode = 0644,
801 .proc_handler = proc_dointvec,
802 },
803#endif
804 {
805 .procname = "pid_max",
806 .data = &pid_max,
807 .maxlen = sizeof (int),
808 .mode = 0644,
809 .proc_handler = proc_dointvec_minmax,
810 .extra1 = &pid_max_min,
811 .extra2 = &pid_max_max,
812 },
813 {
814 .procname = "panic_on_oops",
815 .data = &panic_on_oops,
816 .maxlen = sizeof(int),
817 .mode = 0644,
818 .proc_handler = proc_dointvec,
819 },
820 {
821 .procname = "panic_print",
822 .data = &panic_print,
823 .maxlen = sizeof(unsigned long),
824 .mode = 0644,
825 .proc_handler = proc_doulongvec_minmax,
826 },
827#if defined CONFIG_PRINTK
828 {
829 .procname = "printk",
830 .data = &console_loglevel,
831 .maxlen = 4*sizeof(int),
832 .mode = 0644,
833 .proc_handler = proc_dointvec,
834 },
835 {
836 .procname = "printk_ratelimit",
837 .data = &printk_ratelimit_state.interval,
838 .maxlen = sizeof(int),
839 .mode = 0644,
840 .proc_handler = proc_dointvec_jiffies,
841 },
842 {
843 .procname = "printk_ratelimit_burst",
844 .data = &printk_ratelimit_state.burst,
845 .maxlen = sizeof(int),
846 .mode = 0644,
847 .proc_handler = proc_dointvec,
848 },
849 {
850 .procname = "printk_delay",
851 .data = &printk_delay_msec,
852 .maxlen = sizeof(int),
853 .mode = 0644,
854 .proc_handler = proc_dointvec_minmax,
855 .extra1 = SYSCTL_ZERO,
856 .extra2 = &ten_thousand,
857 },
858 {
859 .procname = "printk_devkmsg",
860 .data = devkmsg_log_str,
861 .maxlen = DEVKMSG_STR_MAX_SIZE,
862 .mode = 0644,
863 .proc_handler = devkmsg_sysctl_set_loglvl,
864 },
865 {
866 .procname = "dmesg_restrict",
867 .data = &dmesg_restrict,
868 .maxlen = sizeof(int),
869 .mode = 0644,
870 .proc_handler = proc_dointvec_minmax_sysadmin,
871 .extra1 = SYSCTL_ZERO,
872 .extra2 = SYSCTL_ONE,
873 },
874 {
875 .procname = "kptr_restrict",
876 .data = &kptr_restrict,
877 .maxlen = sizeof(int),
878 .mode = 0644,
879 .proc_handler = proc_dointvec_minmax_sysadmin,
880 .extra1 = SYSCTL_ZERO,
881 .extra2 = &two,
882 },
883#endif
884 {
885 .procname = "ngroups_max",
886 .data = &ngroups_max,
887 .maxlen = sizeof (int),
888 .mode = 0444,
889 .proc_handler = proc_dointvec,
890 },
891 {
892 .procname = "cap_last_cap",
893 .data = (void *)&cap_last_cap,
894 .maxlen = sizeof(int),
895 .mode = 0444,
896 .proc_handler = proc_dointvec,
897 },
898#if defined(CONFIG_LOCKUP_DETECTOR)
899 {
900 .procname = "watchdog",
901 .data = &watchdog_user_enabled,
902 .maxlen = sizeof(int),
903 .mode = 0644,
904 .proc_handler = proc_watchdog,
905 .extra1 = SYSCTL_ZERO,
906 .extra2 = SYSCTL_ONE,
907 },
908 {
909 .procname = "watchdog_thresh",
910 .data = &watchdog_thresh,
911 .maxlen = sizeof(int),
912 .mode = 0644,
913 .proc_handler = proc_watchdog_thresh,
914 .extra1 = SYSCTL_ZERO,
915 .extra2 = &sixty,
916 },
917 {
918 .procname = "nmi_watchdog",
919 .data = &nmi_watchdog_user_enabled,
920 .maxlen = sizeof(int),
921 .mode = NMI_WATCHDOG_SYSCTL_PERM,
922 .proc_handler = proc_nmi_watchdog,
923 .extra1 = SYSCTL_ZERO,
924 .extra2 = SYSCTL_ONE,
925 },
926 {
927 .procname = "watchdog_cpumask",
928 .data = &watchdog_cpumask_bits,
929 .maxlen = NR_CPUS,
930 .mode = 0644,
931 .proc_handler = proc_watchdog_cpumask,
932 },
933#ifdef CONFIG_SOFTLOCKUP_DETECTOR
934 {
935 .procname = "soft_watchdog",
936 .data = &soft_watchdog_user_enabled,
937 .maxlen = sizeof(int),
938 .mode = 0644,
939 .proc_handler = proc_soft_watchdog,
940 .extra1 = SYSCTL_ZERO,
941 .extra2 = SYSCTL_ONE,
942 },
943 {
944 .procname = "softlockup_panic",
945 .data = &softlockup_panic,
946 .maxlen = sizeof(int),
947 .mode = 0644,
948 .proc_handler = proc_dointvec_minmax,
949 .extra1 = SYSCTL_ZERO,
950 .extra2 = SYSCTL_ONE,
951 },
952#ifdef CONFIG_SMP
953 {
954 .procname = "softlockup_all_cpu_backtrace",
955 .data = &sysctl_softlockup_all_cpu_backtrace,
956 .maxlen = sizeof(int),
957 .mode = 0644,
958 .proc_handler = proc_dointvec_minmax,
959 .extra1 = SYSCTL_ZERO,
960 .extra2 = SYSCTL_ONE,
961 },
962#endif /* CONFIG_SMP */
963#endif
964#ifdef CONFIG_HARDLOCKUP_DETECTOR
965 {
966 .procname = "hardlockup_panic",
967 .data = &hardlockup_panic,
968 .maxlen = sizeof(int),
969 .mode = 0644,
970 .proc_handler = proc_dointvec_minmax,
971 .extra1 = SYSCTL_ZERO,
972 .extra2 = SYSCTL_ONE,
973 },
974#ifdef CONFIG_SMP
975 {
976 .procname = "hardlockup_all_cpu_backtrace",
977 .data = &sysctl_hardlockup_all_cpu_backtrace,
978 .maxlen = sizeof(int),
979 .mode = 0644,
980 .proc_handler = proc_dointvec_minmax,
981 .extra1 = SYSCTL_ZERO,
982 .extra2 = SYSCTL_ONE,
983 },
984#endif /* CONFIG_SMP */
985#endif
986#endif
987
988#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
989 {
990 .procname = "unknown_nmi_panic",
991 .data = &unknown_nmi_panic,
992 .maxlen = sizeof (int),
993 .mode = 0644,
994 .proc_handler = proc_dointvec,
995 },
996#endif
997
998#if (defined(CONFIG_X86_32) || defined(CONFIG_PARISC)) && \
999 defined(CONFIG_DEBUG_STACKOVERFLOW)
1000 {
1001 .procname = "panic_on_stackoverflow",
1002 .data = &sysctl_panic_on_stackoverflow,
1003 .maxlen = sizeof(int),
1004 .mode = 0644,
1005 .proc_handler = proc_dointvec,
1006 },
1007#endif
1008#if defined(CONFIG_X86)
1009 {
1010 .procname = "panic_on_unrecovered_nmi",
1011 .data = &panic_on_unrecovered_nmi,
1012 .maxlen = sizeof(int),
1013 .mode = 0644,
1014 .proc_handler = proc_dointvec,
1015 },
1016 {
1017 .procname = "panic_on_io_nmi",
1018 .data = &panic_on_io_nmi,
1019 .maxlen = sizeof(int),
1020 .mode = 0644,
1021 .proc_handler = proc_dointvec,
1022 },
1023 {
1024 .procname = "bootloader_type",
1025 .data = &bootloader_type,
1026 .maxlen = sizeof (int),
1027 .mode = 0444,
1028 .proc_handler = proc_dointvec,
1029 },
1030 {
1031 .procname = "bootloader_version",
1032 .data = &bootloader_version,
1033 .maxlen = sizeof (int),
1034 .mode = 0444,
1035 .proc_handler = proc_dointvec,
1036 },
1037 {
1038 .procname = "io_delay_type",
1039 .data = &io_delay_type,
1040 .maxlen = sizeof(int),
1041 .mode = 0644,
1042 .proc_handler = proc_dointvec,
1043 },
1044#endif
1045#if defined(CONFIG_MMU)
1046 {
1047 .procname = "randomize_va_space",
1048 .data = &randomize_va_space,
1049 .maxlen = sizeof(int),
1050 .mode = 0644,
1051 .proc_handler = proc_dointvec,
1052 },
1053#endif
1054#if defined(CONFIG_S390) && defined(CONFIG_SMP)
1055 {
1056 .procname = "spin_retry",
1057 .data = &spin_retry,
1058 .maxlen = sizeof (int),
1059 .mode = 0644,
1060 .proc_handler = proc_dointvec,
1061 },
1062#endif
1063#if defined(CONFIG_ACPI_SLEEP) && defined(CONFIG_X86)
1064 {
1065 .procname = "acpi_video_flags",
1066 .data = &acpi_realmode_flags,
1067 .maxlen = sizeof (unsigned long),
1068 .mode = 0644,
1069 .proc_handler = proc_doulongvec_minmax,
1070 },
1071#endif
1072#ifdef CONFIG_SYSCTL_ARCH_UNALIGN_NO_WARN
1073 {
1074 .procname = "ignore-unaligned-usertrap",
1075 .data = &no_unaligned_warning,
1076 .maxlen = sizeof (int),
1077 .mode = 0644,
1078 .proc_handler = proc_dointvec,
1079 },
1080#endif
1081#ifdef CONFIG_IA64
1082 {
1083 .procname = "unaligned-dump-stack",
1084 .data = &unaligned_dump_stack,
1085 .maxlen = sizeof (int),
1086 .mode = 0644,
1087 .proc_handler = proc_dointvec,
1088 },
1089#endif
1090#ifdef CONFIG_DETECT_HUNG_TASK
1091 {
1092 .procname = "hung_task_panic",
1093 .data = &sysctl_hung_task_panic,
1094 .maxlen = sizeof(int),
1095 .mode = 0644,
1096 .proc_handler = proc_dointvec_minmax,
1097 .extra1 = SYSCTL_ZERO,
1098 .extra2 = SYSCTL_ONE,
1099 },
1100 {
1101 .procname = "hung_task_check_count",
1102 .data = &sysctl_hung_task_check_count,
1103 .maxlen = sizeof(int),
1104 .mode = 0644,
1105 .proc_handler = proc_dointvec_minmax,
1106 .extra1 = SYSCTL_ZERO,
1107 },
1108 {
1109 .procname = "hung_task_timeout_secs",
1110 .data = &sysctl_hung_task_timeout_secs,
1111 .maxlen = sizeof(unsigned long),
1112 .mode = 0644,
1113 .proc_handler = proc_dohung_task_timeout_secs,
1114 .extra2 = &hung_task_timeout_max,
1115 },
1116 {
1117 .procname = "hung_task_check_interval_secs",
1118 .data = &sysctl_hung_task_check_interval_secs,
1119 .maxlen = sizeof(unsigned long),
1120 .mode = 0644,
1121 .proc_handler = proc_dohung_task_timeout_secs,
1122 .extra2 = &hung_task_timeout_max,
1123 },
1124 {
1125 .procname = "hung_task_warnings",
1126 .data = &sysctl_hung_task_warnings,
1127 .maxlen = sizeof(int),
1128 .mode = 0644,
1129 .proc_handler = proc_dointvec_minmax,
1130 .extra1 = &neg_one,
1131 },
1132#endif
1133#ifdef CONFIG_RT_MUTEXES
1134 {
1135 .procname = "max_lock_depth",
1136 .data = &max_lock_depth,
1137 .maxlen = sizeof(int),
1138 .mode = 0644,
1139 .proc_handler = proc_dointvec,
1140 },
1141#endif
1142 {
1143 .procname = "poweroff_cmd",
1144 .data = &poweroff_cmd,
1145 .maxlen = POWEROFF_CMD_PATH_LEN,
1146 .mode = 0644,
1147 .proc_handler = proc_dostring,
1148 },
1149#ifdef CONFIG_KEYS
1150 {
1151 .procname = "keys",
1152 .mode = 0555,
1153 .child = key_sysctls,
1154 },
1155#endif
1156#ifdef CONFIG_PERF_EVENTS
1157 /*
1158 * User-space scripts rely on the existence of this file
1159 * as a feature check for perf_events being enabled.
1160 *
1161 * So it's an ABI, do not remove!
1162 */
1163 {
1164 .procname = "perf_event_paranoid",
1165 .data = &sysctl_perf_event_paranoid,
1166 .maxlen = sizeof(sysctl_perf_event_paranoid),
1167 .mode = 0644,
1168 .proc_handler = proc_dointvec,
1169 },
1170 {
1171 .procname = "perf_event_mlock_kb",
1172 .data = &sysctl_perf_event_mlock,
1173 .maxlen = sizeof(sysctl_perf_event_mlock),
1174 .mode = 0644,
1175 .proc_handler = proc_dointvec,
1176 },
1177 {
1178 .procname = "perf_event_max_sample_rate",
1179 .data = &sysctl_perf_event_sample_rate,
1180 .maxlen = sizeof(sysctl_perf_event_sample_rate),
1181 .mode = 0644,
1182 .proc_handler = perf_proc_update_handler,
1183 .extra1 = SYSCTL_ONE,
1184 },
1185 {
1186 .procname = "perf_cpu_time_max_percent",
1187 .data = &sysctl_perf_cpu_time_max_percent,
1188 .maxlen = sizeof(sysctl_perf_cpu_time_max_percent),
1189 .mode = 0644,
1190 .proc_handler = perf_cpu_time_max_percent_handler,
1191 .extra1 = SYSCTL_ZERO,
1192 .extra2 = &one_hundred,
1193 },
1194 {
1195 .procname = "perf_event_max_stack",
1196 .data = &sysctl_perf_event_max_stack,
1197 .maxlen = sizeof(sysctl_perf_event_max_stack),
1198 .mode = 0644,
1199 .proc_handler = perf_event_max_stack_handler,
1200 .extra1 = SYSCTL_ZERO,
1201 .extra2 = &six_hundred_forty_kb,
1202 },
1203 {
1204 .procname = "perf_event_max_contexts_per_stack",
1205 .data = &sysctl_perf_event_max_contexts_per_stack,
1206 .maxlen = sizeof(sysctl_perf_event_max_contexts_per_stack),
1207 .mode = 0644,
1208 .proc_handler = perf_event_max_stack_handler,
1209 .extra1 = SYSCTL_ZERO,
1210 .extra2 = &one_thousand,
1211 },
1212#endif
1213 {
1214 .procname = "panic_on_warn",
1215 .data = &panic_on_warn,
1216 .maxlen = sizeof(int),
1217 .mode = 0644,
1218 .proc_handler = proc_dointvec_minmax,
1219 .extra1 = SYSCTL_ZERO,
1220 .extra2 = SYSCTL_ONE,
1221 },
1222#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
1223 {
1224 .procname = "timer_migration",
1225 .data = &sysctl_timer_migration,
1226 .maxlen = sizeof(unsigned int),
1227 .mode = 0644,
1228 .proc_handler = timer_migration_handler,
1229 .extra1 = SYSCTL_ZERO,
1230 .extra2 = SYSCTL_ONE,
1231 },
1232#endif
1233#ifdef CONFIG_BPF_SYSCALL
1234 {
1235 .procname = "unprivileged_bpf_disabled",
1236 .data = &sysctl_unprivileged_bpf_disabled,
1237 .maxlen = sizeof(sysctl_unprivileged_bpf_disabled),
1238 .mode = 0644,
1239 /* only handle a transition from default "0" to "1" */
1240 .proc_handler = proc_dointvec_minmax,
1241 .extra1 = SYSCTL_ONE,
1242 .extra2 = SYSCTL_ONE,
1243 },
1244 {
1245 .procname = "bpf_stats_enabled",
1246 .data = &bpf_stats_enabled_key.key,
1247 .maxlen = sizeof(bpf_stats_enabled_key),
1248 .mode = 0644,
1249 .proc_handler = proc_do_static_key,
1250 },
1251#endif
1252#if defined(CONFIG_TREE_RCU)
1253 {
1254 .procname = "panic_on_rcu_stall",
1255 .data = &sysctl_panic_on_rcu_stall,
1256 .maxlen = sizeof(sysctl_panic_on_rcu_stall),
1257 .mode = 0644,
1258 .proc_handler = proc_dointvec_minmax,
1259 .extra1 = SYSCTL_ZERO,
1260 .extra2 = SYSCTL_ONE,
1261 },
1262#endif
1263#ifdef CONFIG_STACKLEAK_RUNTIME_DISABLE
1264 {
1265 .procname = "stack_erasing",
1266 .data = NULL,
1267 .maxlen = sizeof(int),
1268 .mode = 0600,
1269 .proc_handler = stack_erasing_sysctl,
1270 .extra1 = SYSCTL_ZERO,
1271 .extra2 = SYSCTL_ONE,
1272 },
1273#endif
1274 { }
1275};
1276
1277static struct ctl_table vm_table[] = {
1278 {
1279 .procname = "overcommit_memory",
1280 .data = &sysctl_overcommit_memory,
1281 .maxlen = sizeof(sysctl_overcommit_memory),
1282 .mode = 0644,
1283 .proc_handler = proc_dointvec_minmax,
1284 .extra1 = SYSCTL_ZERO,
1285 .extra2 = &two,
1286 },
1287 {
1288 .procname = "panic_on_oom",
1289 .data = &sysctl_panic_on_oom,
1290 .maxlen = sizeof(sysctl_panic_on_oom),
1291 .mode = 0644,
1292 .proc_handler = proc_dointvec_minmax,
1293 .extra1 = SYSCTL_ZERO,
1294 .extra2 = &two,
1295 },
1296 {
1297 .procname = "oom_kill_allocating_task",
1298 .data = &sysctl_oom_kill_allocating_task,
1299 .maxlen = sizeof(sysctl_oom_kill_allocating_task),
1300 .mode = 0644,
1301 .proc_handler = proc_dointvec,
1302 },
1303 {
1304 .procname = "oom_dump_tasks",
1305 .data = &sysctl_oom_dump_tasks,
1306 .maxlen = sizeof(sysctl_oom_dump_tasks),
1307 .mode = 0644,
1308 .proc_handler = proc_dointvec,
1309 },
1310 {
1311 .procname = "overcommit_ratio",
1312 .data = &sysctl_overcommit_ratio,
1313 .maxlen = sizeof(sysctl_overcommit_ratio),
1314 .mode = 0644,
1315 .proc_handler = overcommit_ratio_handler,
1316 },
1317 {
1318 .procname = "overcommit_kbytes",
1319 .data = &sysctl_overcommit_kbytes,
1320 .maxlen = sizeof(sysctl_overcommit_kbytes),
1321 .mode = 0644,
1322 .proc_handler = overcommit_kbytes_handler,
1323 },
1324 {
1325 .procname = "page-cluster",
1326 .data = &page_cluster,
1327 .maxlen = sizeof(int),
1328 .mode = 0644,
1329 .proc_handler = proc_dointvec_minmax,
1330 .extra1 = SYSCTL_ZERO,
1331 },
1332 {
1333 .procname = "dirty_background_ratio",
1334 .data = &dirty_background_ratio,
1335 .maxlen = sizeof(dirty_background_ratio),
1336 .mode = 0644,
1337 .proc_handler = dirty_background_ratio_handler,
1338 .extra1 = SYSCTL_ZERO,
1339 .extra2 = &one_hundred,
1340 },
1341 {
1342 .procname = "dirty_background_bytes",
1343 .data = &dirty_background_bytes,
1344 .maxlen = sizeof(dirty_background_bytes),
1345 .mode = 0644,
1346 .proc_handler = dirty_background_bytes_handler,
1347 .extra1 = &one_ul,
1348 },
1349 {
1350 .procname = "dirty_ratio",
1351 .data = &vm_dirty_ratio,
1352 .maxlen = sizeof(vm_dirty_ratio),
1353 .mode = 0644,
1354 .proc_handler = dirty_ratio_handler,
1355 .extra1 = SYSCTL_ZERO,
1356 .extra2 = &one_hundred,
1357 },
1358 {
1359 .procname = "dirty_bytes",
1360 .data = &vm_dirty_bytes,
1361 .maxlen = sizeof(vm_dirty_bytes),
1362 .mode = 0644,
1363 .proc_handler = dirty_bytes_handler,
1364 .extra1 = &dirty_bytes_min,
1365 },
1366 {
1367 .procname = "dirty_writeback_centisecs",
1368 .data = &dirty_writeback_interval,
1369 .maxlen = sizeof(dirty_writeback_interval),
1370 .mode = 0644,
1371 .proc_handler = dirty_writeback_centisecs_handler,
1372 },
1373 {
1374 .procname = "dirty_expire_centisecs",
1375 .data = &dirty_expire_interval,
1376 .maxlen = sizeof(dirty_expire_interval),
1377 .mode = 0644,
1378 .proc_handler = proc_dointvec_minmax,
1379 .extra1 = SYSCTL_ZERO,
1380 },
1381 {
1382 .procname = "dirtytime_expire_seconds",
1383 .data = &dirtytime_expire_interval,
1384 .maxlen = sizeof(dirtytime_expire_interval),
1385 .mode = 0644,
1386 .proc_handler = dirtytime_interval_handler,
1387 .extra1 = SYSCTL_ZERO,
1388 },
1389 {
1390 .procname = "swappiness",
1391 .data = &vm_swappiness,
1392 .maxlen = sizeof(vm_swappiness),
1393 .mode = 0644,
1394 .proc_handler = proc_dointvec_minmax,
1395 .extra1 = SYSCTL_ZERO,
1396 .extra2 = &one_hundred,
1397 },
1398#ifdef CONFIG_HUGETLB_PAGE
1399 {
1400 .procname = "nr_hugepages",
1401 .data = NULL,
1402 .maxlen = sizeof(unsigned long),
1403 .mode = 0644,
1404 .proc_handler = hugetlb_sysctl_handler,
1405 },
1406#ifdef CONFIG_NUMA
1407 {
1408 .procname = "nr_hugepages_mempolicy",
1409 .data = NULL,
1410 .maxlen = sizeof(unsigned long),
1411 .mode = 0644,
1412 .proc_handler = &hugetlb_mempolicy_sysctl_handler,
1413 },
1414 {
1415 .procname = "numa_stat",
1416 .data = &sysctl_vm_numa_stat,
1417 .maxlen = sizeof(int),
1418 .mode = 0644,
1419 .proc_handler = sysctl_vm_numa_stat_handler,
1420 .extra1 = SYSCTL_ZERO,
1421 .extra2 = SYSCTL_ONE,
1422 },
1423#endif
1424 {
1425 .procname = "hugetlb_shm_group",
1426 .data = &sysctl_hugetlb_shm_group,
1427 .maxlen = sizeof(gid_t),
1428 .mode = 0644,
1429 .proc_handler = proc_dointvec,
1430 },
1431 {
1432 .procname = "nr_overcommit_hugepages",
1433 .data = NULL,
1434 .maxlen = sizeof(unsigned long),
1435 .mode = 0644,
1436 .proc_handler = hugetlb_overcommit_handler,
1437 },
1438#endif
1439 {
1440 .procname = "lowmem_reserve_ratio",
1441 .data = &sysctl_lowmem_reserve_ratio,
1442 .maxlen = sizeof(sysctl_lowmem_reserve_ratio),
1443 .mode = 0644,
1444 .proc_handler = lowmem_reserve_ratio_sysctl_handler,
1445 },
1446 {
1447 .procname = "drop_caches",
1448 .data = &sysctl_drop_caches,
1449 .maxlen = sizeof(int),
1450 .mode = 0200,
1451 .proc_handler = drop_caches_sysctl_handler,
1452 .extra1 = SYSCTL_ONE,
1453 .extra2 = &four,
1454 },
1455#ifdef CONFIG_COMPACTION
1456 {
1457 .procname = "compact_memory",
1458 .data = &sysctl_compact_memory,
1459 .maxlen = sizeof(int),
1460 .mode = 0200,
1461 .proc_handler = sysctl_compaction_handler,
1462 },
1463 {
1464 .procname = "extfrag_threshold",
1465 .data = &sysctl_extfrag_threshold,
1466 .maxlen = sizeof(int),
1467 .mode = 0644,
1468 .proc_handler = proc_dointvec_minmax,
1469 .extra1 = &min_extfrag_threshold,
1470 .extra2 = &max_extfrag_threshold,
1471 },
1472 {
1473 .procname = "compact_unevictable_allowed",
1474 .data = &sysctl_compact_unevictable_allowed,
1475 .maxlen = sizeof(int),
1476 .mode = 0644,
1477 .proc_handler = proc_dointvec_minmax_warn_RT_change,
1478 .extra1 = SYSCTL_ZERO,
1479 .extra2 = SYSCTL_ONE,
1480 },
1481
1482#endif /* CONFIG_COMPACTION */
1483 {
1484 .procname = "min_free_kbytes",
1485 .data = &min_free_kbytes,
1486 .maxlen = sizeof(min_free_kbytes),
1487 .mode = 0644,
1488 .proc_handler = min_free_kbytes_sysctl_handler,
1489 .extra1 = SYSCTL_ZERO,
1490 },
1491 {
1492 .procname = "watermark_boost_factor",
1493 .data = &watermark_boost_factor,
1494 .maxlen = sizeof(watermark_boost_factor),
1495 .mode = 0644,
1496 .proc_handler = watermark_boost_factor_sysctl_handler,
1497 .extra1 = SYSCTL_ZERO,
1498 },
1499 {
1500 .procname = "watermark_scale_factor",
1501 .data = &watermark_scale_factor,
1502 .maxlen = sizeof(watermark_scale_factor),
1503 .mode = 0644,
1504 .proc_handler = watermark_scale_factor_sysctl_handler,
1505 .extra1 = SYSCTL_ONE,
1506 .extra2 = &one_thousand,
1507 },
1508 {
1509 .procname = "percpu_pagelist_fraction",
1510 .data = &percpu_pagelist_fraction,
1511 .maxlen = sizeof(percpu_pagelist_fraction),
1512 .mode = 0644,
1513 .proc_handler = percpu_pagelist_fraction_sysctl_handler,
1514 .extra1 = SYSCTL_ZERO,
1515 },
1516#ifdef CONFIG_MMU
1517 {
1518 .procname = "max_map_count",
1519 .data = &sysctl_max_map_count,
1520 .maxlen = sizeof(sysctl_max_map_count),
1521 .mode = 0644,
1522 .proc_handler = proc_dointvec_minmax,
1523 .extra1 = SYSCTL_ZERO,
1524 },
1525#else
1526 {
1527 .procname = "nr_trim_pages",
1528 .data = &sysctl_nr_trim_pages,
1529 .maxlen = sizeof(sysctl_nr_trim_pages),
1530 .mode = 0644,
1531 .proc_handler = proc_dointvec_minmax,
1532 .extra1 = SYSCTL_ZERO,
1533 },
1534#endif
1535 {
1536 .procname = "laptop_mode",
1537 .data = &laptop_mode,
1538 .maxlen = sizeof(laptop_mode),
1539 .mode = 0644,
1540 .proc_handler = proc_dointvec_jiffies,
1541 },
1542 {
1543 .procname = "block_dump",
1544 .data = &block_dump,
1545 .maxlen = sizeof(block_dump),
1546 .mode = 0644,
1547 .proc_handler = proc_dointvec,
1548 .extra1 = SYSCTL_ZERO,
1549 },
1550 {
1551 .procname = "vfs_cache_pressure",
1552 .data = &sysctl_vfs_cache_pressure,
1553 .maxlen = sizeof(sysctl_vfs_cache_pressure),
1554 .mode = 0644,
1555 .proc_handler = proc_dointvec,
1556 .extra1 = SYSCTL_ZERO,
1557 },
1558#if defined(HAVE_ARCH_PICK_MMAP_LAYOUT) || \
1559 defined(CONFIG_ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT)
1560 {
1561 .procname = "legacy_va_layout",
1562 .data = &sysctl_legacy_va_layout,
1563 .maxlen = sizeof(sysctl_legacy_va_layout),
1564 .mode = 0644,
1565 .proc_handler = proc_dointvec,
1566 .extra1 = SYSCTL_ZERO,
1567 },
1568#endif
1569#ifdef CONFIG_NUMA
1570 {
1571 .procname = "zone_reclaim_mode",
1572 .data = &node_reclaim_mode,
1573 .maxlen = sizeof(node_reclaim_mode),
1574 .mode = 0644,
1575 .proc_handler = proc_dointvec,
1576 .extra1 = SYSCTL_ZERO,
1577 },
1578 {
1579 .procname = "min_unmapped_ratio",
1580 .data = &sysctl_min_unmapped_ratio,
1581 .maxlen = sizeof(sysctl_min_unmapped_ratio),
1582 .mode = 0644,
1583 .proc_handler = sysctl_min_unmapped_ratio_sysctl_handler,
1584 .extra1 = SYSCTL_ZERO,
1585 .extra2 = &one_hundred,
1586 },
1587 {
1588 .procname = "min_slab_ratio",
1589 .data = &sysctl_min_slab_ratio,
1590 .maxlen = sizeof(sysctl_min_slab_ratio),
1591 .mode = 0644,
1592 .proc_handler = sysctl_min_slab_ratio_sysctl_handler,
1593 .extra1 = SYSCTL_ZERO,
1594 .extra2 = &one_hundred,
1595 },
1596#endif
1597#ifdef CONFIG_SMP
1598 {
1599 .procname = "stat_interval",
1600 .data = &sysctl_stat_interval,
1601 .maxlen = sizeof(sysctl_stat_interval),
1602 .mode = 0644,
1603 .proc_handler = proc_dointvec_jiffies,
1604 },
1605 {
1606 .procname = "stat_refresh",
1607 .data = NULL,
1608 .maxlen = 0,
1609 .mode = 0600,
1610 .proc_handler = vmstat_refresh,
1611 },
1612#endif
1613#ifdef CONFIG_MMU
1614 {
1615 .procname = "mmap_min_addr",
1616 .data = &dac_mmap_min_addr,
1617 .maxlen = sizeof(unsigned long),
1618 .mode = 0644,
1619 .proc_handler = mmap_min_addr_handler,
1620 },
1621#endif
1622#ifdef CONFIG_NUMA
1623 {
1624 .procname = "numa_zonelist_order",
1625 .data = &numa_zonelist_order,
1626 .maxlen = NUMA_ZONELIST_ORDER_LEN,
1627 .mode = 0644,
1628 .proc_handler = numa_zonelist_order_handler,
1629 },
1630#endif
1631#if (defined(CONFIG_X86_32) && !defined(CONFIG_UML))|| \
1632 (defined(CONFIG_SUPERH) && defined(CONFIG_VSYSCALL))
1633 {
1634 .procname = "vdso_enabled",
1635#ifdef CONFIG_X86_32
1636 .data = &vdso32_enabled,
1637 .maxlen = sizeof(vdso32_enabled),
1638#else
1639 .data = &vdso_enabled,
1640 .maxlen = sizeof(vdso_enabled),
1641#endif
1642 .mode = 0644,
1643 .proc_handler = proc_dointvec,
1644 .extra1 = SYSCTL_ZERO,
1645 },
1646#endif
1647#ifdef CONFIG_HIGHMEM
1648 {
1649 .procname = "highmem_is_dirtyable",
1650 .data = &vm_highmem_is_dirtyable,
1651 .maxlen = sizeof(vm_highmem_is_dirtyable),
1652 .mode = 0644,
1653 .proc_handler = proc_dointvec_minmax,
1654 .extra1 = SYSCTL_ZERO,
1655 .extra2 = SYSCTL_ONE,
1656 },
1657#endif
1658#ifdef CONFIG_MEMORY_FAILURE
1659 {
1660 .procname = "memory_failure_early_kill",
1661 .data = &sysctl_memory_failure_early_kill,
1662 .maxlen = sizeof(sysctl_memory_failure_early_kill),
1663 .mode = 0644,
1664 .proc_handler = proc_dointvec_minmax,
1665 .extra1 = SYSCTL_ZERO,
1666 .extra2 = SYSCTL_ONE,
1667 },
1668 {
1669 .procname = "memory_failure_recovery",
1670 .data = &sysctl_memory_failure_recovery,
1671 .maxlen = sizeof(sysctl_memory_failure_recovery),
1672 .mode = 0644,
1673 .proc_handler = proc_dointvec_minmax,
1674 .extra1 = SYSCTL_ZERO,
1675 .extra2 = SYSCTL_ONE,
1676 },
1677#endif
1678 {
1679 .procname = "user_reserve_kbytes",
1680 .data = &sysctl_user_reserve_kbytes,
1681 .maxlen = sizeof(sysctl_user_reserve_kbytes),
1682 .mode = 0644,
1683 .proc_handler = proc_doulongvec_minmax,
1684 },
1685 {
1686 .procname = "admin_reserve_kbytes",
1687 .data = &sysctl_admin_reserve_kbytes,
1688 .maxlen = sizeof(sysctl_admin_reserve_kbytes),
1689 .mode = 0644,
1690 .proc_handler = proc_doulongvec_minmax,
1691 },
1692#ifdef CONFIG_HAVE_ARCH_MMAP_RND_BITS
1693 {
1694 .procname = "mmap_rnd_bits",
1695 .data = &mmap_rnd_bits,
1696 .maxlen = sizeof(mmap_rnd_bits),
1697 .mode = 0600,
1698 .proc_handler = proc_dointvec_minmax,
1699 .extra1 = (void *)&mmap_rnd_bits_min,
1700 .extra2 = (void *)&mmap_rnd_bits_max,
1701 },
1702#endif
1703#ifdef CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS
1704 {
1705 .procname = "mmap_rnd_compat_bits",
1706 .data = &mmap_rnd_compat_bits,
1707 .maxlen = sizeof(mmap_rnd_compat_bits),
1708 .mode = 0600,
1709 .proc_handler = proc_dointvec_minmax,
1710 .extra1 = (void *)&mmap_rnd_compat_bits_min,
1711 .extra2 = (void *)&mmap_rnd_compat_bits_max,
1712 },
1713#endif
1714#ifdef CONFIG_USERFAULTFD
1715 {
1716 .procname = "unprivileged_userfaultfd",
1717 .data = &sysctl_unprivileged_userfaultfd,
1718 .maxlen = sizeof(sysctl_unprivileged_userfaultfd),
1719 .mode = 0644,
1720 .proc_handler = proc_dointvec_minmax,
1721 .extra1 = SYSCTL_ZERO,
1722 .extra2 = SYSCTL_ONE,
1723 },
1724#endif
1725 { }
1726};
1727
1728static struct ctl_table fs_table[] = {
1729 {
1730 .procname = "inode-nr",
1731 .data = &inodes_stat,
1732 .maxlen = 2*sizeof(long),
1733 .mode = 0444,
1734 .proc_handler = proc_nr_inodes,
1735 },
1736 {
1737 .procname = "inode-state",
1738 .data = &inodes_stat,
1739 .maxlen = 7*sizeof(long),
1740 .mode = 0444,
1741 .proc_handler = proc_nr_inodes,
1742 },
1743 {
1744 .procname = "file-nr",
1745 .data = &files_stat,
1746 .maxlen = sizeof(files_stat),
1747 .mode = 0444,
1748 .proc_handler = proc_nr_files,
1749 },
1750 {
1751 .procname = "file-max",
1752 .data = &files_stat.max_files,
1753 .maxlen = sizeof(files_stat.max_files),
1754 .mode = 0644,
1755 .proc_handler = proc_doulongvec_minmax,
1756 .extra1 = &zero_ul,
1757 .extra2 = &long_max,
1758 },
1759 {
1760 .procname = "nr_open",
1761 .data = &sysctl_nr_open,
1762 .maxlen = sizeof(unsigned int),
1763 .mode = 0644,
1764 .proc_handler = proc_dointvec_minmax,
1765 .extra1 = &sysctl_nr_open_min,
1766 .extra2 = &sysctl_nr_open_max,
1767 },
1768 {
1769 .procname = "dentry-state",
1770 .data = &dentry_stat,
1771 .maxlen = 6*sizeof(long),
1772 .mode = 0444,
1773 .proc_handler = proc_nr_dentry,
1774 },
1775 {
1776 .procname = "overflowuid",
1777 .data = &fs_overflowuid,
1778 .maxlen = sizeof(int),
1779 .mode = 0644,
1780 .proc_handler = proc_dointvec_minmax,
1781 .extra1 = &minolduid,
1782 .extra2 = &maxolduid,
1783 },
1784 {
1785 .procname = "overflowgid",
1786 .data = &fs_overflowgid,
1787 .maxlen = sizeof(int),
1788 .mode = 0644,
1789 .proc_handler = proc_dointvec_minmax,
1790 .extra1 = &minolduid,
1791 .extra2 = &maxolduid,
1792 },
1793#ifdef CONFIG_FILE_LOCKING
1794 {
1795 .procname = "leases-enable",
1796 .data = &leases_enable,
1797 .maxlen = sizeof(int),
1798 .mode = 0644,
1799 .proc_handler = proc_dointvec,
1800 },
1801#endif
1802#ifdef CONFIG_DNOTIFY
1803 {
1804 .procname = "dir-notify-enable",
1805 .data = &dir_notify_enable,
1806 .maxlen = sizeof(int),
1807 .mode = 0644,
1808 .proc_handler = proc_dointvec,
1809 },
1810#endif
1811#ifdef CONFIG_MMU
1812#ifdef CONFIG_FILE_LOCKING
1813 {
1814 .procname = "lease-break-time",
1815 .data = &lease_break_time,
1816 .maxlen = sizeof(int),
1817 .mode = 0644,
1818 .proc_handler = proc_dointvec,
1819 },
1820#endif
1821#ifdef CONFIG_AIO
1822 {
1823 .procname = "aio-nr",
1824 .data = &aio_nr,
1825 .maxlen = sizeof(aio_nr),
1826 .mode = 0444,
1827 .proc_handler = proc_doulongvec_minmax,
1828 },
1829 {
1830 .procname = "aio-max-nr",
1831 .data = &aio_max_nr,
1832 .maxlen = sizeof(aio_max_nr),
1833 .mode = 0644,
1834 .proc_handler = proc_doulongvec_minmax,
1835 },
1836#endif /* CONFIG_AIO */
1837#ifdef CONFIG_INOTIFY_USER
1838 {
1839 .procname = "inotify",
1840 .mode = 0555,
1841 .child = inotify_table,
1842 },
1843#endif
1844#ifdef CONFIG_EPOLL
1845 {
1846 .procname = "epoll",
1847 .mode = 0555,
1848 .child = epoll_table,
1849 },
1850#endif
1851#endif
1852 {
1853 .procname = "protected_symlinks",
1854 .data = &sysctl_protected_symlinks,
1855 .maxlen = sizeof(int),
1856 .mode = 0600,
1857 .proc_handler = proc_dointvec_minmax,
1858 .extra1 = SYSCTL_ZERO,
1859 .extra2 = SYSCTL_ONE,
1860 },
1861 {
1862 .procname = "protected_hardlinks",
1863 .data = &sysctl_protected_hardlinks,
1864 .maxlen = sizeof(int),
1865 .mode = 0600,
1866 .proc_handler = proc_dointvec_minmax,
1867 .extra1 = SYSCTL_ZERO,
1868 .extra2 = SYSCTL_ONE,
1869 },
1870 {
1871 .procname = "protected_fifos",
1872 .data = &sysctl_protected_fifos,
1873 .maxlen = sizeof(int),
1874 .mode = 0600,
1875 .proc_handler = proc_dointvec_minmax,
1876 .extra1 = SYSCTL_ZERO,
1877 .extra2 = &two,
1878 },
1879 {
1880 .procname = "protected_regular",
1881 .data = &sysctl_protected_regular,
1882 .maxlen = sizeof(int),
1883 .mode = 0600,
1884 .proc_handler = proc_dointvec_minmax,
1885 .extra1 = SYSCTL_ZERO,
1886 .extra2 = &two,
1887 },
1888 {
1889 .procname = "suid_dumpable",
1890 .data = &suid_dumpable,
1891 .maxlen = sizeof(int),
1892 .mode = 0644,
1893 .proc_handler = proc_dointvec_minmax_coredump,
1894 .extra1 = SYSCTL_ZERO,
1895 .extra2 = &two,
1896 },
1897#if defined(CONFIG_BINFMT_MISC) || defined(CONFIG_BINFMT_MISC_MODULE)
1898 {
1899 .procname = "binfmt_misc",
1900 .mode = 0555,
1901 .child = sysctl_mount_point,
1902 },
1903#endif
1904 {
1905 .procname = "pipe-max-size",
1906 .data = &pipe_max_size,
1907 .maxlen = sizeof(pipe_max_size),
1908 .mode = 0644,
1909 .proc_handler = proc_dopipe_max_size,
1910 },
1911 {
1912 .procname = "pipe-user-pages-hard",
1913 .data = &pipe_user_pages_hard,
1914 .maxlen = sizeof(pipe_user_pages_hard),
1915 .mode = 0644,
1916 .proc_handler = proc_doulongvec_minmax,
1917 },
1918 {
1919 .procname = "pipe-user-pages-soft",
1920 .data = &pipe_user_pages_soft,
1921 .maxlen = sizeof(pipe_user_pages_soft),
1922 .mode = 0644,
1923 .proc_handler = proc_doulongvec_minmax,
1924 },
1925 {
1926 .procname = "mount-max",
1927 .data = &sysctl_mount_max,
1928 .maxlen = sizeof(unsigned int),
1929 .mode = 0644,
1930 .proc_handler = proc_dointvec_minmax,
1931 .extra1 = SYSCTL_ONE,
1932 },
1933 { }
1934};
1935
1936static struct ctl_table debug_table[] = {
1937#ifdef CONFIG_SYSCTL_EXCEPTION_TRACE
1938 {
1939 .procname = "exception-trace",
1940 .data = &show_unhandled_signals,
1941 .maxlen = sizeof(int),
1942 .mode = 0644,
1943 .proc_handler = proc_dointvec
1944 },
1945#endif
1946#if defined(CONFIG_OPTPROBES)
1947 {
1948 .procname = "kprobes-optimization",
1949 .data = &sysctl_kprobes_optimization,
1950 .maxlen = sizeof(int),
1951 .mode = 0644,
1952 .proc_handler = proc_kprobes_optimization_handler,
1953 .extra1 = SYSCTL_ZERO,
1954 .extra2 = SYSCTL_ONE,
1955 },
1956#endif
1957 { }
1958};
1959
1960static struct ctl_table dev_table[] = {
1961 { }
1962};
1963
1964int __init sysctl_init(void)
1965{
1966 struct ctl_table_header *hdr;
1967
1968 hdr = register_sysctl_table(sysctl_base_table);
1969 kmemleak_not_leak(hdr);
1970 return 0;
1971}
1972
1973#endif /* CONFIG_SYSCTL */
1974
1975/*
1976 * /proc/sys support
1977 */
1978
1979#ifdef CONFIG_PROC_SYSCTL242#ifdef CONFIG_PROC_SYSCTL
1980 243
1981static int _proc_do_string(char *data, int maxlen, int write,244static int _proc_do_string(char *data, int maxlen, int write,
1982 char __user *buffer,245 char *buffer, size_t *lenp, loff_t *ppos)
1983 size_t *lenp, loff_t *ppos)
1984{246{
1985 size_t len;247 size_t len;
1986 char __user *p;248 char c, *p;
1987 char c;
1988 249
1989 if (!data || !maxlen || !*lenp) {250 if (!data || !maxlen || !*lenp) {
1990 *lenp = 0;251 *lenp = 0;
@@ -2009,8 +270,7 @@ static int _proc_do_string(char *data, int maxlen, int write, · 18 unmodified lines
2009 *ppos += *lenp;270 *ppos += *lenp;
2010 p = buffer;271 p = buffer;
2011 while ((p - buffer) < *lenp && len < maxlen - 1) {272 while ((p - buffer) < *lenp && len < maxlen - 1) {
2012 if (get_user(c, p++))273 c = *(p++);
2013 return -EFAULT;
2014 if (c == 0 || c == '\n')274 if (c == 0 || c == '\n')
2015 break;275 break;
2016 data[len++] = c;276 data[len++] = c;
@@ -2032,11 +292,9 @@ static int _proc_do_string(char *data, int maxlen, int write, · 15 unmodified lines
2032 if (len > *lenp)292 if (len > *lenp)
2033 len = *lenp;293 len = *lenp;
2034 if (len)294 if (len)
2035 if (copy_to_user(buffer, data, len))295 memcpy(buffer, data, len);
2036 return -EFAULT;
2037 if (len < *lenp) {296 if (len < *lenp) {
2038 if (put_user('\n', buffer + len))297 buffer[len] = '\n';
2039 return -EFAULT;
2040 len++;298 len++;
2041 }299 }
2042 *lenp = len;300 *lenp = len;
@@ -2097,13 +355,13 @@ static bool proc_first_pos_non_zero_ignore(loff_t *ppos, · 54 unmodified lines
2097 * Returns 0 on success.355 * Returns 0 on success.
2098 */356 */
2099int proc_dostring(struct ctl_table *table, int write,357int proc_dostring(struct ctl_table *table, int write,
2100 void __user *buffer, size_t *lenp, loff_t *ppos)358 void *buffer, size_t *lenp, loff_t *ppos)
2101{359{
2102 if (write)360 if (write)
2103 proc_first_pos_non_zero_ignore(ppos, table);361 proc_first_pos_non_zero_ignore(ppos, table);
2104 362
2105 return _proc_do_string((char *)(table->data), table->maxlen, write,363 return _proc_do_string(table->data, table->maxlen, write, buffer, lenp,
2106 (char __user *)buffer, lenp, ppos);364 ppos);
2107}365}
2108 366
2109static size_t proc_skip_spaces(char **buf)367static size_t proc_skip_spaces(char **buf)
@@ -2234,11 +492,10 @@ static int proc_get_long(char **buf, size_t *size, · 124 unmodified lines
2234 * @val: the integer to be converted492 * @val: the integer to be converted
2235 * @neg: sign of the number, %TRUE for negative493 * @neg: sign of the number, %TRUE for negative
2236 *494 *
2237 * In case of success %0 is returned and @buf and @size are updated with495 * In case of success @buf and @size are updated with the amount of bytes
2238 * the amount of bytes written.496 * written.
2239 */497 */
2240static int proc_put_long(void __user **buf, size_t *size, unsigned long val,498static void proc_put_long(void **buf, size_t *size, unsigned long val, bool neg)
2241 bool neg)
2242{499{
2243 int len;500 int len;
2244 char tmp[TMPBUFLEN], *p = tmp;501 char tmp[TMPBUFLEN], *p = tmp;
@@ -2247,24 +504,22 @@ static int proc_put_long(void __user **buf, size_t *size, unsigned long val, · 2 unmodified lines
2247 len = strlen(tmp);504 len = strlen(tmp);
2248 if (len > *size)505 if (len > *size)
2249 len = *size;506 len = *size;
2250 if (copy_to_user(*buf, tmp, len))507 memcpy(*buf, tmp, len);
2251 return -EFAULT;
2252 *size -= len;508 *size -= len;
2253 *buf += len;509 *buf += len;
2254 return 0;
2255}510}
2256#undef TMPBUFLEN511#undef TMPBUFLEN
2257 512
2258static int proc_put_char(void __user **buf, size_t *size, char c)513static void proc_put_char(void **buf, size_t *size, char c)
2259{514{
2260 if (*size) {515 if (*size) {
2261 char __user **buffer = (char __user **)buf;516 char **buffer = (char **)buf;
2262 if (put_user(c, *buffer))517 **buffer = c;
2263 return -EFAULT;518
2264 (*size)--, (*buffer)++;519 (*size)--;
520 (*buffer)++;
2265 *buf = *buffer;521 *buf = *buffer;
2266 }522 }
2267 return 0;
2268}523}
2269 524
2270static int do_proc_dointvec_conv(bool *negp, unsigned long *lvalp,525static int do_proc_dointvec_conv(bool *negp, unsigned long *lvalp,
@@ -2312,7 +567,7 @@ static int do_proc_douintvec_conv(unsigned long *lvalp, · 41 unmodified lines
2312static const char proc_wspace_sep[] = { ' ', '\t', '\n' };567static const char proc_wspace_sep[] = { ' ', '\t', '\n' };
2313 568
2314static int __do_proc_dointvec(void *tbl_data, struct ctl_table *table,569static int __do_proc_dointvec(void *tbl_data, struct ctl_table *table,
2315 int write, void __user *buffer,570 int write, void *buffer,
2316 size_t *lenp, loff_t *ppos,571 size_t *lenp, loff_t *ppos,
2317 int (*conv)(bool *negp, unsigned long *lvalp, int *valp,572 int (*conv)(bool *negp, unsigned long *lvalp, int *valp,
2318 int write, void *data),573 int write, void *data),
@@ -2320,7 +575,7 @@ static int __do_proc_dointvec(void *tbl_data, struct ctl_table *table, · 1 unmodified line
2320{575{
2321 int *i, vleft, first = 1, err = 0;576 int *i, vleft, first = 1, err = 0;
2322 size_t left;577 size_t left;
2323 char *kbuf = NULL, *p;578 char *p;
2324 579
2325 if (!tbl_data || !table->maxlen || !*lenp || (*ppos && !write)) {580 if (!tbl_data || !table->maxlen || !*lenp || (*ppos && !write)) {
2326 *lenp = 0;581 *lenp = 0;
@@ -2340,9 +595,7 @@ static int __do_proc_dointvec(void *tbl_data, struct ctl_table *table, · 13 unmodified lines
2340 595
2341 if (left > PAGE_SIZE - 1)596 if (left > PAGE_SIZE - 1)
2342 left = PAGE_SIZE - 1;597 left = PAGE_SIZE - 1;
2343 p = kbuf = memdup_user_nul(buffer, left);598 p = buffer;
2344 if (IS_ERR(kbuf))
2345 return PTR_ERR(kbuf);
2346 }599 }
2347 600
2348 for (; left && vleft--; i++, first=0) {601 for (; left && vleft--; i++, first=0) {
@@ -2369,24 +622,17 @@ static int __do_proc_dointvec(void *tbl_data, struct ctl_table *table, · 20 unmodified lines
2369 break;622 break;
2370 }623 }
2371 if (!first)624 if (!first)
2372 err = proc_put_char(&buffer, &left, '\t');625 proc_put_char(&buffer, &left, '\t');
2373 if (err)626 proc_put_long(&buffer, &left, lval, neg);
2374 break;
2375 err = proc_put_long(&buffer, &left, lval, neg);
2376 if (err)
2377 break;
2378 }627 }
2379 }628 }
2380 629
2381 if (!write && !first && left && !err)630 if (!write && !first && left && !err)
2382 err = proc_put_char(&buffer, &left, '\n');631 proc_put_char(&buffer, &left, '\n');
2383 if (write && !err && left)632 if (write && !err && left)
2384 left -= proc_skip_spaces(&p);633 left -= proc_skip_spaces(&p);
2385 if (write) {634 if (write && first)
2386 kfree(kbuf);635 return err ? : -EINVAL;
2387 if (first)
2388 return err ? : -EINVAL;
2389 }
2390 *lenp -= left;636 *lenp -= left;
2391out:637out:
2392 *ppos += *lenp;638 *ppos += *lenp;
@@ -2394,7 +640,7 @@ out: · 1 unmodified line
2394}640}
2395 641
2396static int do_proc_dointvec(struct ctl_table *table, int write,642static int do_proc_dointvec(struct ctl_table *table, int write,
2397 void __user *buffer, size_t *lenp, loff_t *ppos,643 void *buffer, size_t *lenp, loff_t *ppos,
2398 int (*conv)(bool *negp, unsigned long *lvalp, int *valp,644 int (*conv)(bool *negp, unsigned long *lvalp, int *valp,
2399 int write, void *data),645 int write, void *data),
2400 void *data)646 void *data)
@@ -2405,7 +651,7 @@ static int do_proc_dointvec(struct ctl_table *table, int write, · 4 unmodified lines
2405 651
2406static int do_proc_douintvec_w(unsigned int *tbl_data,652static int do_proc_douintvec_w(unsigned int *tbl_data,
2407 struct ctl_table *table,653 struct ctl_table *table,
2408 void __user *buffer,654 void *buffer,
2409 size_t *lenp, loff_t *ppos,655 size_t *lenp, loff_t *ppos,
2410 int (*conv)(unsigned long *lvalp,656 int (*conv)(unsigned long *lvalp,
2411 unsigned int *valp,657 unsigned int *valp,
@@ -2416,7 +662,7 @@ static int do_proc_douintvec_w(unsigned int *tbl_data, · 4 unmodified lines
2416 int err = 0;662 int err = 0;
2417 size_t left;663 size_t left;
2418 bool neg;664 bool neg;
2419 char *kbuf = NULL, *p;665 char *p = buffer;
2420 666
2421 left = *lenp;667 left = *lenp;
2422 668
@@ -2426,10 +672,6 @@ static int do_proc_douintvec_w(unsigned int *tbl_data, · 3 unmodified lines
2426 if (left > PAGE_SIZE - 1)672 if (left > PAGE_SIZE - 1)
2427 left = PAGE_SIZE - 1;673 left = PAGE_SIZE - 1;
2428 674
2429 p = kbuf = memdup_user_nul(buffer, left);
2430 if (IS_ERR(kbuf))
2431 return -EINVAL;
2432
2433 left -= proc_skip_spaces(&p);675 left -= proc_skip_spaces(&p);
2434 if (!left) {676 if (!left) {
2435 err = -EINVAL;677 err = -EINVAL;
@@ -2453,7 +695,6 @@ static int do_proc_douintvec_w(unsigned int *tbl_data, · 17 unmodified lines
2453 left -= proc_skip_spaces(&p);695 left -= proc_skip_spaces(&p);
2454 696
2455out_free:697out_free:
2456 kfree(kbuf);
2457 if (err)698 if (err)
2458 return -EINVAL;699 return -EINVAL;
2459 700
@@ -2465,7 +706,7 @@ bail_early: · 5 unmodified lines
2465 return err;706 return err;
2466}707}
2467 708
2468static int do_proc_douintvec_r(unsigned int *tbl_data, void __user *buffer,709static int do_proc_douintvec_r(unsigned int *tbl_data, void *buffer,
2469 size_t *lenp, loff_t *ppos,710 size_t *lenp, loff_t *ppos,
2470 int (*conv)(unsigned long *lvalp,711 int (*conv)(unsigned long *lvalp,
2471 unsigned int *valp,712 unsigned int *valp,
@@ -2483,11 +724,11 @@ static int do_proc_douintvec_r(unsigned int *tbl_data, void __user *buffer, · 11 unmodified lines
2483 goto out;724 goto out;
2484 }725 }
2485 726
2486 err = proc_put_long(&buffer, &left, lval, false);727 proc_put_long(&buffer, &left, lval, false);
2487 if (err || !left)728 if (!left)
2488 goto out;729 goto out;
2489 730
2490 err = proc_put_char(&buffer, &left, '\n');731 proc_put_char(&buffer, &left, '\n');
2491 732
2492out:733out:
2493 *lenp -= left;734 *lenp -= left;
@@ -2497,7 +738,7 @@ out: · 3 unmodified lines
2497}738}
2498 739
2499static int __do_proc_douintvec(void *tbl_data, struct ctl_table *table,740static int __do_proc_douintvec(void *tbl_data, struct ctl_table *table,
2500 int write, void __user *buffer,741 int write, void *buffer,
2501 size_t *lenp, loff_t *ppos,742 size_t *lenp, loff_t *ppos,
2502 int (*conv)(unsigned long *lvalp,743 int (*conv)(unsigned long *lvalp,
2503 unsigned int *valp,744 unsigned int *valp,
@@ -2533,7 +774,7 @@ static int __do_proc_douintvec(void *tbl_data, struct ctl_table *table, · 29 unmodified lines
2533}774}
2534 775
2535static int do_proc_douintvec(struct ctl_table *table, int write,776static int do_proc_douintvec(struct ctl_table *table, int write,
2536 void __user *buffer, size_t *lenp, loff_t *ppos,777 void *buffer, size_t *lenp, loff_t *ppos,
2537 int (*conv)(unsigned long *lvalp,778 int (*conv)(unsigned long *lvalp,
2538 unsigned int *valp,779 unsigned int *valp,
2539 int write, void *data),780 int write, void *data),
@@ -2556,16 +797,15 @@ static int do_proc_douintvec(struct ctl_table *table, int write, · 16 unmodified lines
2556 *797 *
2557 * Returns 0 on success.798 * Returns 0 on success.
2558 */799 */
2559int proc_dointvec(struct ctl_table *table, int write,800int proc_dointvec(struct ctl_table *table, int write, void *buffer,
2560 void __user *buffer, size_t *lenp, loff_t *ppos)801 size_t *lenp, loff_t *ppos)
2561{802{
2562 return do_proc_dointvec(table, write, buffer, lenp, ppos, NULL, NULL);803 return do_proc_dointvec(table, write, buffer, lenp, ppos, NULL, NULL);
2563}804}
2564 805
2565#ifdef CONFIG_COMPACTION806#ifdef CONFIG_COMPACTION
2566static int proc_dointvec_minmax_warn_RT_change(struct ctl_table *table,807static int proc_dointvec_minmax_warn_RT_change(struct ctl_table *table,
2567 int write, void __user *buffer,808 int write, void *buffer, size_t *lenp, loff_t *ppos)
2568 size_t *lenp, loff_t *ppos)
2569{809{
2570 int ret, old;810 int ret, old;
2571 811
@@ -2597,8 +837,8 @@ static int proc_dointvec_minmax_warn_RT_change(struct ctl_table *table, · 25 unmodified lines
2597 *837 *
2598 * Returns 0 on success.838 * Returns 0 on success.
2599 */839 */
2600int proc_douintvec(struct ctl_table *table, int write,840int proc_douintvec(struct ctl_table *table, int write, void *buffer,
2601 void __user *buffer, size_t *lenp, loff_t *ppos)841 size_t *lenp, loff_t *ppos)
2602{842{
2603 return do_proc_douintvec(table, write, buffer, lenp, ppos,843 return do_proc_douintvec(table, write, buffer, lenp, ppos,
2604 do_proc_douintvec_conv, NULL);844 do_proc_douintvec_conv, NULL);
@@ -2609,7 +849,7 @@ int proc_douintvec(struct ctl_table *table, int write, · 4 unmodified lines
2609 * This means we can safely use a temporary.849 * This means we can safely use a temporary.
2610 */850 */
2611static int proc_taint(struct ctl_table *table, int write,851static int proc_taint(struct ctl_table *table, int write,
2612 void __user *buffer, size_t *lenp, loff_t *ppos)852 void *buffer, size_t *lenp, loff_t *ppos)
2613{853{
2614 struct ctl_table t;854 struct ctl_table t;
2615 unsigned long tmptaint = get_taint();855 unsigned long tmptaint = get_taint();
@@ -2641,7 +881,7 @@ static int proc_taint(struct ctl_table *table, int write, · 25 unmodified lines
2641 881
2642#ifdef CONFIG_PRINTK882#ifdef CONFIG_PRINTK
2643static int proc_dointvec_minmax_sysadmin(struct ctl_table *table, int write,883static int proc_dointvec_minmax_sysadmin(struct ctl_table *table, int write,
2644 void __user *buffer, size_t *lenp, loff_t *ppos)884 void *buffer, size_t *lenp, loff_t *ppos)
2645{885{
2646 if (write && !capable(CAP_SYS_ADMIN))886 if (write && !capable(CAP_SYS_ADMIN))
2647 return -EPERM;887 return -EPERM;
@@ -2707,7 +947,7 @@ static int do_proc_dointvec_minmax_conv(bool *negp, unsigned long *lvalp, · 59 unmodified lines
2707 * Returns 0 on success or -EINVAL on write when the range check fails.947 * Returns 0 on success or -EINVAL on write when the range check fails.
2708 */948 */
2709int proc_dointvec_minmax(struct ctl_table *table, int write,949int proc_dointvec_minmax(struct ctl_table *table, int write,
2710 void __user *buffer, size_t *lenp, loff_t *ppos)950 void *buffer, size_t *lenp, loff_t *ppos)
2711{951{
2712 struct do_proc_dointvec_minmax_conv_param param = {952 struct do_proc_dointvec_minmax_conv_param param = {
2713 .min = (int *) table->extra1,953 .min = (int *) table->extra1,
@@ -2776,7 +1016,7 @@ static int do_proc_douintvec_minmax_conv(unsigned long *lvalp, · 62 unmodified lines
2776 * Returns 0 on success or -ERANGE on write when the range check fails.1016 * Returns 0 on success or -ERANGE on write when the range check fails.
2777 */1017 */
2778int proc_douintvec_minmax(struct ctl_table *table, int write,1018int proc_douintvec_minmax(struct ctl_table *table, int write,
2779 void __user *buffer, size_t *lenp, loff_t *ppos)1019 void *buffer, size_t *lenp, loff_t *ppos)
2780{1020{
2781 struct do_proc_douintvec_minmax_conv_param param = {1021 struct do_proc_douintvec_minmax_conv_param param = {
2782 .min = (unsigned int *) table->extra1,1022 .min = (unsigned int *) table->extra1,
@@ -2807,7 +1047,7 @@ static int do_proc_dopipe_max_size_conv(unsigned long *lvalp, · 24 unmodified lines
2807}1047}
2808 1048
2809static int proc_dopipe_max_size(struct ctl_table *table, int write,1049static int proc_dopipe_max_size(struct ctl_table *table, int write,
2810 void __user *buffer, size_t *lenp, loff_t *ppos)1050 void *buffer, size_t *lenp, loff_t *ppos)
2811{1051{
2812 return do_proc_douintvec(table, write, buffer, lenp, ppos,1052 return do_proc_douintvec(table, write, buffer, lenp, ppos,
2813 do_proc_dopipe_max_size_conv, NULL);1053 do_proc_dopipe_max_size_conv, NULL);
@@ -2828,7 +1068,7 @@ static void validate_coredump_safety(void) · 14 unmodified lines
2828}1068}
2829 1069
2830static int proc_dointvec_minmax_coredump(struct ctl_table *table, int write,1070static int proc_dointvec_minmax_coredump(struct ctl_table *table, int write,
2831 void __user *buffer, size_t *lenp, loff_t *ppos)1071 void *buffer, size_t *lenp, loff_t *ppos)
2832{1072{
2833 int error = proc_dointvec_minmax(table, write, buffer, lenp, ppos);1073 int error = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
2834 if (!error)1074 if (!error)
@@ -2838,7 +1078,7 @@ static int proc_dointvec_minmax_coredump(struct ctl_table *table, int write, · 3 unmodified lines
2838 1078
2839#ifdef CONFIG_COREDUMP1079#ifdef CONFIG_COREDUMP
2840static int proc_dostring_coredump(struct ctl_table *table, int write,1080static int proc_dostring_coredump(struct ctl_table *table, int write,
2841 void __user *buffer, size_t *lenp, loff_t *ppos)1081 void *buffer, size_t *lenp, loff_t *ppos)
2842{1082{
2843 int error = proc_dostring(table, write, buffer, lenp, ppos);1083 int error = proc_dostring(table, write, buffer, lenp, ppos);
2844 if (!error)1084 if (!error)
@@ -2849,7 +1089,7 @@ static int proc_dostring_coredump(struct ctl_table *table, int write, · 4 unmodified lines
2849 1089
2850#ifdef CONFIG_MAGIC_SYSRQ1090#ifdef CONFIG_MAGIC_SYSRQ
2851static int sysrq_sysctl_handler(struct ctl_table *table, int write,1091static int sysrq_sysctl_handler(struct ctl_table *table, int write,
2852 void __user *buffer, size_t *lenp, loff_t *ppos)1092 void *buffer, size_t *lenp, loff_t *ppos)
2853{1093{
2854 int tmp, ret;1094 int tmp, ret;
2855 1095
@@ -2867,16 +1107,14 @@ static int sysrq_sysctl_handler(struct ctl_table *table, int write, · 11 unmodified lines
2867}1107}
2868#endif1108#endif
2869 1109
2870static int __do_proc_doulongvec_minmax(void *data, struct ctl_table *table, int write,1110static int __do_proc_doulongvec_minmax(void *data, struct ctl_table *table,
2871 void __user *buffer,1111 int write, void *buffer, size_t *lenp, loff_t *ppos,
2872 size_t *lenp, loff_t *ppos,1112 unsigned long convmul, unsigned long convdiv)
2873 unsigned long convmul,
2874 unsigned long convdiv)
2875{1113{
2876 unsigned long *i, *min, *max;1114 unsigned long *i, *min, *max;
2877 int vleft, first = 1, err = 0;1115 int vleft, first = 1, err = 0;
2878 size_t left;1116 size_t left;
2879 char *kbuf = NULL, *p;1117 char *p;
2880 1118
2881 if (!data || !table->maxlen || !*lenp || (*ppos && !write)) {1119 if (!data || !table->maxlen || !*lenp || (*ppos && !write)) {
2882 *lenp = 0;1120 *lenp = 0;
@@ -2895,9 +1133,7 @@ static int __do_proc_doulongvec_minmax(void *data, struct ctl_table *table, int · 12 unmodified lines
2895 1133
2896 if (left > PAGE_SIZE - 1)1134 if (left > PAGE_SIZE - 1)
2897 left = PAGE_SIZE - 1;1135 left = PAGE_SIZE - 1;
2898 p = kbuf = memdup_user_nul(buffer, left);1136 p = buffer;
2899 if (IS_ERR(kbuf))
2900 return PTR_ERR(kbuf);
2901 }1137 }
2902 1138
2903 for (; left && vleft--; i++, first = 0) {1139 for (; left && vleft--; i++, first = 0) {
@@ -2925,26 +1161,18 @@ static int __do_proc_doulongvec_minmax(void *data, struct ctl_table *table, int · 21 unmodified lines
2925 *i = val;1161 *i = val;
2926 } else {1162 } else {
2927 val = convdiv * (*i) / convmul;1163 val = convdiv * (*i) / convmul;
2928 if (!first) {1164 if (!first)
2929 err = proc_put_char(&buffer, &left, '\t');1165 proc_put_char(&buffer, &left, '\t');
2930 if (err)1166 proc_put_long(&buffer, &left, val, false);
2931 break;
2932 }
2933 err = proc_put_long(&buffer, &left, val, false);
2934 if (err)
2935 break;
2936 }1167 }
2937 }1168 }
2938 1169
2939 if (!write && !first && left && !err)1170 if (!write && !first && left && !err)
2940 err = proc_put_char(&buffer, &left, '\n');1171 proc_put_char(&buffer, &left, '\n');
2941 if (write && !err)1172 if (write && !err)
2942 left -= proc_skip_spaces(&p);1173 left -= proc_skip_spaces(&p);
2943 if (write) {1174 if (write && first)
2944 kfree(kbuf);1175 return err ? : -EINVAL;
2945 if (first)
2946 return err ? : -EINVAL;
2947 }
2948 *lenp -= left;1176 *lenp -= left;
2949out:1177out:
2950 *ppos += *lenp;1178 *ppos += *lenp;
@@ -2952,10 +1180,8 @@ out: · 1 unmodified line
2952}1180}
2953 1181
2954static int do_proc_doulongvec_minmax(struct ctl_table *table, int write,1182static int do_proc_doulongvec_minmax(struct ctl_table *table, int write,
2955 void __user *buffer,1183 void *buffer, size_t *lenp, loff_t *ppos, unsigned long convmul,
2956 size_t *lenp, loff_t *ppos,1184 unsigned long convdiv)
2957 unsigned long convmul,
2958 unsigned long convdiv)
2959{1185{
2960 return __do_proc_doulongvec_minmax(table->data, table, write,1186 return __do_proc_doulongvec_minmax(table->data, table, write,
2961 buffer, lenp, ppos, convmul, convdiv);1187 buffer, lenp, ppos, convmul, convdiv);
@@ -2978,7 +1204,7 @@ static int do_proc_doulongvec_minmax(struct ctl_table *table, int write, · 16 unmodified lines
2978 * Returns 0 on success.1204 * Returns 0 on success.
2979 */1205 */
2980int proc_doulongvec_minmax(struct ctl_table *table, int write,1206int proc_doulongvec_minmax(struct ctl_table *table, int write,
2981 void __user *buffer, size_t *lenp, loff_t *ppos)1207 void *buffer, size_t *lenp, loff_t *ppos)
2982{1208{
2983 return do_proc_doulongvec_minmax(table, write, buffer, lenp, ppos, 1l, 1l);1209 return do_proc_doulongvec_minmax(table, write, buffer, lenp, ppos, 1l, 1l);
2984}1210}
@@ -3001,8 +1227,7 @@ int proc_doulongvec_minmax(struct ctl_table *table, int write, · 16 unmodified lines
3001 * Returns 0 on success.1227 * Returns 0 on success.
3002 */1228 */
3003int proc_doulongvec_ms_jiffies_minmax(struct ctl_table *table, int write,1229int proc_doulongvec_ms_jiffies_minmax(struct ctl_table *table, int write,
3004 void __user *buffer,1230 void *buffer, size_t *lenp, loff_t *ppos)
3005 size_t *lenp, loff_t *ppos)
3006{1231{
3007 return do_proc_doulongvec_minmax(table, write, buffer,1232 return do_proc_doulongvec_minmax(table, write, buffer,
3008 lenp, ppos, HZ, 1000l);1233 lenp, ppos, HZ, 1000l);
@@ -3096,7 +1321,7 @@ static int do_proc_dointvec_ms_jiffies_conv(bool *negp, unsigned long *lvalp, · 87 unmodified lines
3096 * Returns 0 on success.1321 * Returns 0 on success.
3097 */1322 */
3098int proc_dointvec_jiffies(struct ctl_table *table, int write,1323int proc_dointvec_jiffies(struct ctl_table *table, int write,
3099 void __user *buffer, size_t *lenp, loff_t *ppos)1324 void *buffer, size_t *lenp, loff_t *ppos)
3100{1325{
3101 return do_proc_dointvec(table,write,buffer,lenp,ppos,1326 return do_proc_dointvec(table,write,buffer,lenp,ppos,
3102 do_proc_dointvec_jiffies_conv,NULL);1327 do_proc_dointvec_jiffies_conv,NULL);
@@ -3118,7 +1343,7 @@ int proc_dointvec_jiffies(struct ctl_table *table, int write, · 15 unmodified lines
3118 * Returns 0 on success.1343 * Returns 0 on success.
3119 */1344 */
3120int proc_dointvec_userhz_jiffies(struct ctl_table *table, int write,1345int proc_dointvec_userhz_jiffies(struct ctl_table *table, int write,
3121 void __user *buffer, size_t *lenp, loff_t *ppos)1346 void *buffer, size_t *lenp, loff_t *ppos)
3122{1347{
3123 return do_proc_dointvec(table,write,buffer,lenp,ppos,1348 return do_proc_dointvec(table,write,buffer,lenp,ppos,
3124 do_proc_dointvec_userhz_jiffies_conv,NULL);1349 do_proc_dointvec_userhz_jiffies_conv,NULL);
@@ -3140,15 +1365,15 @@ int proc_dointvec_userhz_jiffies(struct ctl_table *table, int write, · 15 unmodified lines
3140 *1365 *
3141 * Returns 0 on success.1366 * Returns 0 on success.
3142 */1367 */
3143int proc_dointvec_ms_jiffies(struct ctl_table *table, int write,1368int proc_dointvec_ms_jiffies(struct ctl_table *table, int write, void *buffer,
3144 void __user *buffer, size_t *lenp, loff_t *ppos)1369 size_t *lenp, loff_t *ppos)
3145{1370{
3146 return do_proc_dointvec(table, write, buffer, lenp, ppos,1371 return do_proc_dointvec(table, write, buffer, lenp, ppos,
3147 do_proc_dointvec_ms_jiffies_conv, NULL);1372 do_proc_dointvec_ms_jiffies_conv, NULL);
3148}1373}
3149 1374
3150static int proc_do_cad_pid(struct ctl_table *table, int write,1375static int proc_do_cad_pid(struct ctl_table *table, int write, void *buffer,
3151 void __user *buffer, size_t *lenp, loff_t *ppos)1376 size_t *lenp, loff_t *ppos)
3152{1377{
3153 struct pid *new_pid;1378 struct pid *new_pid;
3154 pid_t tmp;1379 pid_t tmp;
@@ -3187,7 +1412,7 @@ static int proc_do_cad_pid(struct ctl_table *table, int write, · 32 unmodified lines
3187 * Returns 0 on success.1412 * Returns 0 on success.
3188 */1413 */
3189int proc_do_large_bitmap(struct ctl_table *table, int write,1414int proc_do_large_bitmap(struct ctl_table *table, int write,
3190 void __user *buffer, size_t *lenp, loff_t *ppos)1415 void *buffer, size_t *lenp, loff_t *ppos)
3191{1416{
3192 int err = 0;1417 int err = 0;
3193 bool first = 1;1418 bool first = 1;
@@ -3203,7 +1428,7 @@ int proc_do_large_bitmap(struct ctl_table *table, int write, · 9 unmodified lines
3203 }1428 }
3204 1429
3205 if (write) {1430 if (write) {
3206 char *kbuf, *p;1431 char *p = buffer;
3207 size_t skipped = 0;1432 size_t skipped = 0;
3208 1433
3209 if (left > PAGE_SIZE - 1) {1434 if (left > PAGE_SIZE - 1) {
@@ -3212,15 +1437,9 @@ int proc_do_large_bitmap(struct ctl_table *table, int write, · 2 unmodified lines
3212 skipped = *lenp - left;1437 skipped = *lenp - left;
3213 }1438 }
3214 1439
3215 p = kbuf = memdup_user_nul(buffer, left);
3216 if (IS_ERR(kbuf))
3217 return PTR_ERR(kbuf);
3218
3219 tmp_bitmap = bitmap_zalloc(bitmap_len, GFP_KERNEL);1440 tmp_bitmap = bitmap_zalloc(bitmap_len, GFP_KERNEL);
3220 if (!tmp_bitmap) {1441 if (!tmp_bitmap)
3221 kfree(kbuf);
3222 return -ENOMEM;1442 return -ENOMEM;
3223 }
3224 proc_skip_char(&p, &left, '\n');1443 proc_skip_char(&p, &left, '\n');
3225 while (!err && left) {1444 while (!err && left) {
3226 unsigned long val_a, val_b;1445 unsigned long val_a, val_b;
@@ -3284,7 +1503,6 @@ int proc_do_large_bitmap(struct ctl_table *table, int write, · 57 unmodified lines
3284 first = 0;1503 first = 0;
3285 proc_skip_char(&p, &left, '\n');1504 proc_skip_char(&p, &left, '\n');
3286 }1505 }
3287 kfree(kbuf);
3288 left += skipped;1506 left += skipped;
3289 } else {1507 } else {
3290 unsigned long bit_a, bit_b = 0;1508 unsigned long bit_a, bit_b = 0;
@@ -3296,27 +1514,17 @@ int proc_do_large_bitmap(struct ctl_table *table, int write, · 5 unmodified lines
3296 bit_b = find_next_zero_bit(bitmap, bitmap_len,1514 bit_b = find_next_zero_bit(bitmap, bitmap_len,
3297 bit_a + 1) - 1;1515 bit_a + 1) - 1;
3298 1516
3299 if (!first) {1517 if (!first)
3300 err = proc_put_char(&buffer, &left, ',');1518 proc_put_char(&buffer, &left, ',');
3301 if (err)1519 proc_put_long(&buffer, &left, bit_a, false);
3302 break;
3303 }
3304 err = proc_put_long(&buffer, &left, bit_a, false);
3305 if (err)
3306 break;
3307 if (bit_a != bit_b) {1520 if (bit_a != bit_b) {
3308 err = proc_put_char(&buffer, &left, '-');1521 proc_put_char(&buffer, &left, '-');
3309 if (err)1522 proc_put_long(&buffer, &left, bit_b, false);
3310 break;
3311 err = proc_put_long(&buffer, &left, bit_b, false);
3312 if (err)
3313 break;
3314 }1523 }
3315 1524
3316 first = 0; bit_b++;1525 first = 0; bit_b++;
3317 }1526 }
3318 if (!err)1527 proc_put_char(&buffer, &left, '\n');
3319 err = proc_put_char(&buffer, &left, '\n');
3320 }1528 }
3321 1529
3322 if (!err) {1530 if (!err) {
@@ -3330,113 +1538,1806 @@ int proc_do_large_bitmap(struct ctl_table *table, int write, · 7 unmodified lines
3330 *ppos += *lenp;1538 *ppos += *lenp;
3331 }1539 }
3332 1540
3333 bitmap_free(tmp_bitmap);1541 bitmap_free(tmp_bitmap);
3334 return err;1542 return err;
1543}
1544
1545#else /* CONFIG_PROC_SYSCTL */
1546
1547int proc_dostring(struct ctl_table *table, int write,
1548 void *buffer, size_t *lenp, loff_t *ppos)
1549{
1550 return -ENOSYS;
1551}
1552
1553int proc_dointvec(struct ctl_table *table, int write,
1554 void *buffer, size_t *lenp, loff_t *ppos)
1555{
1556 return -ENOSYS;
1557}
1558
1559int proc_douintvec(struct ctl_table *table, int write,
1560 void *buffer, size_t *lenp, loff_t *ppos)
1561{
1562 return -ENOSYS;
1563}
1564
1565int proc_dointvec_minmax(struct ctl_table *table, int write,
1566 void *buffer, size_t *lenp, loff_t *ppos)
1567{
1568 return -ENOSYS;
1569}
1570
1571int proc_douintvec_minmax(struct ctl_table *table, int write,
1572 void *buffer, size_t *lenp, loff_t *ppos)
1573{
1574 return -ENOSYS;
3335}1575}
3336 1576
3337#else /* CONFIG_PROC_SYSCTL */1577int proc_dointvec_jiffies(struct ctl_table *table, int write,
1578 void *buffer, size_t *lenp, loff_t *ppos)
1579{
1580 return -ENOSYS;
1581}
3338 1582
3339int proc_dostring(struct ctl_table *table, int write,1583int proc_dointvec_userhz_jiffies(struct ctl_table *table, int write,
3340 void __user *buffer, size_t *lenp, loff_t *ppos)1584 void *buffer, size_t *lenp, loff_t *ppos)
3341{1585{
3342 return -ENOSYS;1586 return -ENOSYS;
3343}1587}
3344 1588
3345int proc_dointvec(struct ctl_table *table, int write,1589int proc_dointvec_ms_jiffies(struct ctl_table *table, int write,
3346 void __user *buffer, size_t *lenp, loff_t *ppos)1590 void *buffer, size_t *lenp, loff_t *ppos)
3347{1591{
3348 return -ENOSYS;1592 return -ENOSYS;
3349}1593}
3350 1594
3351int proc_douintvec(struct ctl_table *table, int write,1595int proc_doulongvec_minmax(struct ctl_table *table, int write,
3352 void __user *buffer, size_t *lenp, loff_t *ppos)1596 void *buffer, size_t *lenp, loff_t *ppos)
3353{1597{
3354 return -ENOSYS;1598 return -ENOSYS;
3355}1599}
3356 1600
3357int proc_dointvec_minmax(struct ctl_table *table, int write,1601int proc_doulongvec_ms_jiffies_minmax(struct ctl_table *table, int write,
3358 void __user *buffer, size_t *lenp, loff_t *ppos)1602 void *buffer, size_t *lenp, loff_t *ppos)
1603{
1604 return -ENOSYS;
1605}
1606
1607int proc_do_large_bitmap(struct ctl_table *table, int write,
1608 void *buffer, size_t *lenp, loff_t *ppos)
3359{1609{
3360 return -ENOSYS;1610 return -ENOSYS;
3361}1611}
3362 1612
3363int proc_douintvec_minmax(struct ctl_table *table, int write,1613#endif /* CONFIG_PROC_SYSCTL */
3364 void __user *buffer, size_t *lenp, loff_t *ppos)1614
3365{1615#if defined(CONFIG_SYSCTL)
3366 return -ENOSYS;1616int proc_do_static_key(struct ctl_table *table, int write,
3367}1617 void *buffer, size_t *lenp, loff_t *ppos)
1618{
1619 struct static_key *key = (struct static_key *)table->data;
1620 static DEFINE_MUTEX(static_key_mutex);
1621 int val, ret;
1622 struct ctl_table tmp = {
1623 .data = &val,
1624 .maxlen = sizeof(val),
1625 .mode = table->mode,
1626 .extra1 = SYSCTL_ZERO,
1627 .extra2 = SYSCTL_ONE,
1628 };
1629
1630 if (write && !capable(CAP_SYS_ADMIN))
1631 return -EPERM;
1632
1633 mutex_lock(&static_key_mutex);
1634 val = static_key_enabled(key);
1635 ret = proc_dointvec_minmax(&tmp, write, buffer, lenp, ppos);
1636 if (write && !ret) {
1637 if (val)
1638 static_key_enable(key);
1639 else
1640 static_key_disable(key);
1641 }
1642 mutex_unlock(&static_key_mutex);
1643 return ret;
1644}
1645
1646static struct ctl_table kern_table[] = {
1647 {
1648 .procname = "sched_child_runs_first",
1649 .data = &sysctl_sched_child_runs_first,
1650 .maxlen = sizeof(unsigned int),
1651 .mode = 0644,
1652 .proc_handler = proc_dointvec,
1653 },
1654#ifdef CONFIG_SCHED_DEBUG
1655 {
1656 .procname = "sched_min_granularity_ns",
1657 .data = &sysctl_sched_min_granularity,
1658 .maxlen = sizeof(unsigned int),
1659 .mode = 0644,
1660 .proc_handler = sched_proc_update_handler,
1661 .extra1 = &min_sched_granularity_ns,
1662 .extra2 = &max_sched_granularity_ns,
1663 },
1664 {
1665 .procname = "sched_latency_ns",
1666 .data = &sysctl_sched_latency,
1667 .maxlen = sizeof(unsigned int),
1668 .mode = 0644,
1669 .proc_handler = sched_proc_update_handler,
1670 .extra1 = &min_sched_granularity_ns,
1671 .extra2 = &max_sched_granularity_ns,
1672 },
1673 {
1674 .procname = "sched_wakeup_granularity_ns",
1675 .data = &sysctl_sched_wakeup_granularity,
1676 .maxlen = sizeof(unsigned int),
1677 .mode = 0644,
1678 .proc_handler = sched_proc_update_handler,
1679 .extra1 = &min_wakeup_granularity_ns,
1680 .extra2 = &max_wakeup_granularity_ns,
1681 },
1682#ifdef CONFIG_SMP
1683 {
1684 .procname = "sched_tunable_scaling",
1685 .data = &sysctl_sched_tunable_scaling,
1686 .maxlen = sizeof(enum sched_tunable_scaling),
1687 .mode = 0644,
1688 .proc_handler = sched_proc_update_handler,
1689 .extra1 = &min_sched_tunable_scaling,
1690 .extra2 = &max_sched_tunable_scaling,
1691 },
1692 {
1693 .procname = "sched_migration_cost_ns",
1694 .data = &sysctl_sched_migration_cost,
1695 .maxlen = sizeof(unsigned int),
1696 .mode = 0644,
1697 .proc_handler = proc_dointvec,
1698 },
1699 {
1700 .procname = "sched_nr_migrate",
1701 .data = &sysctl_sched_nr_migrate,
1702 .maxlen = sizeof(unsigned int),
1703 .mode = 0644,
1704 .proc_handler = proc_dointvec,
1705 },
1706#ifdef CONFIG_SCHEDSTATS
1707 {
1708 .procname = "sched_schedstats",
1709 .data = NULL,
1710 .maxlen = sizeof(unsigned int),
1711 .mode = 0644,
1712 .proc_handler = sysctl_schedstats,
1713 .extra1 = SYSCTL_ZERO,
1714 .extra2 = SYSCTL_ONE,
1715 },
1716#endif /* CONFIG_SCHEDSTATS */
1717#endif /* CONFIG_SMP */
1718#ifdef CONFIG_NUMA_BALANCING
1719 {
1720 .procname = "numa_balancing_scan_delay_ms",
1721 .data = &sysctl_numa_balancing_scan_delay,
1722 .maxlen = sizeof(unsigned int),
1723 .mode = 0644,
1724 .proc_handler = proc_dointvec,
1725 },
1726 {
1727 .procname = "numa_balancing_scan_period_min_ms",
1728 .data = &sysctl_numa_balancing_scan_period_min,
1729 .maxlen = sizeof(unsigned int),
1730 .mode = 0644,
1731 .proc_handler = proc_dointvec,
1732 },
1733 {
1734 .procname = "numa_balancing_scan_period_max_ms",
1735 .data = &sysctl_numa_balancing_scan_period_max,
1736 .maxlen = sizeof(unsigned int),
1737 .mode = 0644,
1738 .proc_handler = proc_dointvec,
1739 },
1740 {
1741 .procname = "numa_balancing_scan_size_mb",
1742 .data = &sysctl_numa_balancing_scan_size,
1743 .maxlen = sizeof(unsigned int),
1744 .mode = 0644,
1745 .proc_handler = proc_dointvec_minmax,
1746 .extra1 = SYSCTL_ONE,
1747 },
1748 {
1749 .procname = "numa_balancing",
1750 .data = NULL, /* filled in by handler */
1751 .maxlen = sizeof(unsigned int),
1752 .mode = 0644,
1753 .proc_handler = sysctl_numa_balancing,
1754 .extra1 = SYSCTL_ZERO,
1755 .extra2 = SYSCTL_ONE,
1756 },
1757#endif /* CONFIG_NUMA_BALANCING */
1758#endif /* CONFIG_SCHED_DEBUG */
1759 {
1760 .procname = "sched_rt_period_us",
1761 .data = &sysctl_sched_rt_period,
1762 .maxlen = sizeof(unsigned int),
1763 .mode = 0644,
1764 .proc_handler = sched_rt_handler,
1765 },
1766 {
1767 .procname = "sched_rt_runtime_us",
1768 .data = &sysctl_sched_rt_runtime,
1769 .maxlen = sizeof(int),
1770 .mode = 0644,
1771 .proc_handler = sched_rt_handler,
1772 },
1773 {
1774 .procname = "sched_rr_timeslice_ms",
1775 .data = &sysctl_sched_rr_timeslice,
1776 .maxlen = sizeof(int),
1777 .mode = 0644,
1778 .proc_handler = sched_rr_handler,
1779 },
1780#ifdef CONFIG_UCLAMP_TASK
1781 {
1782 .procname = "sched_util_clamp_min",
1783 .data = &sysctl_sched_uclamp_util_min,
1784 .maxlen = sizeof(unsigned int),
1785 .mode = 0644,
1786 .proc_handler = sysctl_sched_uclamp_handler,
1787 },
1788 {
1789 .procname = "sched_util_clamp_max",
1790 .data = &sysctl_sched_uclamp_util_max,
1791 .maxlen = sizeof(unsigned int),
1792 .mode = 0644,
1793 .proc_handler = sysctl_sched_uclamp_handler,
1794 },
1795#endif
1796#ifdef CONFIG_SCHED_AUTOGROUP
1797 {
1798 .procname = "sched_autogroup_enabled",
1799 .data = &sysctl_sched_autogroup_enabled,
1800 .maxlen = sizeof(unsigned int),
1801 .mode = 0644,
1802 .proc_handler = proc_dointvec_minmax,
1803 .extra1 = SYSCTL_ZERO,
1804 .extra2 = SYSCTL_ONE,
1805 },
1806#endif
1807#ifdef CONFIG_CFS_BANDWIDTH
1808 {
1809 .procname = "sched_cfs_bandwidth_slice_us",
1810 .data = &sysctl_sched_cfs_bandwidth_slice,
1811 .maxlen = sizeof(unsigned int),
1812 .mode = 0644,
1813 .proc_handler = proc_dointvec_minmax,
1814 .extra1 = SYSCTL_ONE,
1815 },
1816#endif
1817#if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL)
1818 {
1819 .procname = "sched_energy_aware",
1820 .data = &sysctl_sched_energy_aware,
1821 .maxlen = sizeof(unsigned int),
1822 .mode = 0644,
1823 .proc_handler = sched_energy_aware_handler,
1824 .extra1 = SYSCTL_ZERO,
1825 .extra2 = SYSCTL_ONE,
1826 },
1827#endif
1828#ifdef CONFIG_PROVE_LOCKING
1829 {
1830 .procname = "prove_locking",
1831 .data = &prove_locking,
1832 .maxlen = sizeof(int),
1833 .mode = 0644,
1834 .proc_handler = proc_dointvec,
1835 },
1836#endif
1837#ifdef CONFIG_LOCK_STAT
1838 {
1839 .procname = "lock_stat",
1840 .data = &lock_stat,
1841 .maxlen = sizeof(int),
1842 .mode = 0644,
1843 .proc_handler = proc_dointvec,
1844 },
1845#endif
1846 {
1847 .procname = "panic",
1848 .data = &panic_timeout,
1849 .maxlen = sizeof(int),
1850 .mode = 0644,
1851 .proc_handler = proc_dointvec,
1852 },
1853#ifdef CONFIG_COREDUMP
1854 {
1855 .procname = "core_uses_pid",
1856 .data = &core_uses_pid,
1857 .maxlen = sizeof(int),
1858 .mode = 0644,
1859 .proc_handler = proc_dointvec,
1860 },
1861 {
1862 .procname = "core_pattern",
1863 .data = core_pattern,
1864 .maxlen = CORENAME_MAX_SIZE,
1865 .mode = 0644,
1866 .proc_handler = proc_dostring_coredump,
1867 },
1868 {
1869 .procname = "core_pipe_limit",
1870 .data = &core_pipe_limit,
1871 .maxlen = sizeof(unsigned int),
1872 .mode = 0644,
1873 .proc_handler = proc_dointvec,
1874 },
1875#endif
1876#ifdef CONFIG_PROC_SYSCTL
1877 {
1878 .procname = "tainted",
1879 .maxlen = sizeof(long),
1880 .mode = 0644,
1881 .proc_handler = proc_taint,
1882 },
1883 {
1884 .procname = "sysctl_writes_strict",
1885 .data = &sysctl_writes_strict,
1886 .maxlen = sizeof(int),
1887 .mode = 0644,
1888 .proc_handler = proc_dointvec_minmax,
1889 .extra1 = &neg_one,
1890 .extra2 = SYSCTL_ONE,
1891 },
1892#endif
1893#ifdef CONFIG_LATENCYTOP
1894 {
1895 .procname = "latencytop",
1896 .data = &latencytop_enabled,
1897 .maxlen = sizeof(int),
1898 .mode = 0644,
1899 .proc_handler = sysctl_latencytop,
1900 },
1901#endif
1902#ifdef CONFIG_BLK_DEV_INITRD
1903 {
1904 .procname = "real-root-dev",
1905 .data = &real_root_dev,
1906 .maxlen = sizeof(int),
1907 .mode = 0644,
1908 .proc_handler = proc_dointvec,
1909 },
1910#endif
1911 {
1912 .procname = "print-fatal-signals",
1913 .data = &print_fatal_signals,
1914 .maxlen = sizeof(int),
1915 .mode = 0644,
1916 .proc_handler = proc_dointvec,
1917 },
1918#ifdef CONFIG_SPARC
1919 {
1920 .procname = "reboot-cmd",
1921 .data = reboot_command,
1922 .maxlen = 256,
1923 .mode = 0644,
1924 .proc_handler = proc_dostring,
1925 },
1926 {
1927 .procname = "stop-a",
1928 .data = &stop_a_enabled,
1929 .maxlen = sizeof (int),
1930 .mode = 0644,
1931 .proc_handler = proc_dointvec,
1932 },
1933 {
1934 .procname = "scons-poweroff",
1935 .data = &scons_pwroff,
1936 .maxlen = sizeof (int),
1937 .mode = 0644,
1938 .proc_handler = proc_dointvec,
1939 },
1940#endif
1941#ifdef CONFIG_SPARC64
1942 {
1943 .procname = "tsb-ratio",
1944 .data = &sysctl_tsb_ratio,
1945 .maxlen = sizeof (int),
1946 .mode = 0644,
1947 .proc_handler = proc_dointvec,
1948 },
1949#endif
1950#ifdef CONFIG_PARISC
1951 {
1952 .procname = "soft-power",
1953 .data = &pwrsw_enabled,
1954 .maxlen = sizeof (int),
1955 .mode = 0644,
1956 .proc_handler = proc_dointvec,
1957 },
1958#endif
1959#ifdef CONFIG_SYSCTL_ARCH_UNALIGN_ALLOW
1960 {
1961 .procname = "unaligned-trap",
1962 .data = &unaligned_enabled,
1963 .maxlen = sizeof (int),
1964 .mode = 0644,
1965 .proc_handler = proc_dointvec,
1966 },
1967#endif
1968 {
1969 .procname = "ctrl-alt-del",
1970 .data = &C_A_D,
1971 .maxlen = sizeof(int),
1972 .mode = 0644,
1973 .proc_handler = proc_dointvec,
1974 },
1975#ifdef CONFIG_FUNCTION_TRACER
1976 {
1977 .procname = "ftrace_enabled",
1978 .data = &ftrace_enabled,
1979 .maxlen = sizeof(int),
1980 .mode = 0644,
1981 .proc_handler = ftrace_enable_sysctl,
1982 },
1983#endif
1984#ifdef CONFIG_STACK_TRACER
1985 {
1986 .procname = "stack_tracer_enabled",
1987 .data = &stack_tracer_enabled,
1988 .maxlen = sizeof(int),
1989 .mode = 0644,
1990 .proc_handler = stack_trace_sysctl,
1991 },
1992#endif
1993#ifdef CONFIG_TRACING
1994 {
1995 .procname = "ftrace_dump_on_oops",
1996 .data = &ftrace_dump_on_oops,
1997 .maxlen = sizeof(int),
1998 .mode = 0644,
1999 .proc_handler = proc_dointvec,
2000 },
2001 {
2002 .procname = "traceoff_on_warning",
2003 .data = &__disable_trace_on_warning,
2004 .maxlen = sizeof(__disable_trace_on_warning),
2005 .mode = 0644,
2006 .proc_handler = proc_dointvec,
2007 },
2008 {
2009 .procname = "tracepoint_printk",
2010 .data = &tracepoint_printk,
2011 .maxlen = sizeof(tracepoint_printk),
2012 .mode = 0644,
2013 .proc_handler = tracepoint_printk_sysctl,
2014 },
2015#endif
2016#ifdef CONFIG_KEXEC_CORE
2017 {
2018 .procname = "kexec_load_disabled",
2019 .data = &kexec_load_disabled,
2020 .maxlen = sizeof(int),
2021 .mode = 0644,
2022 /* only handle a transition from default "0" to "1" */
2023 .proc_handler = proc_dointvec_minmax,
2024 .extra1 = SYSCTL_ONE,
2025 .extra2 = SYSCTL_ONE,
2026 },
2027#endif
2028#ifdef CONFIG_MODULES
2029 {
2030 .procname = "modprobe",
2031 .data = &modprobe_path,
2032 .maxlen = KMOD_PATH_LEN,
2033 .mode = 0644,
2034 .proc_handler = proc_dostring,
2035 },
2036 {
2037 .procname = "modules_disabled",
2038 .data = &modules_disabled,
2039 .maxlen = sizeof(int),
2040 .mode = 0644,
2041 /* only handle a transition from default "0" to "1" */
2042 .proc_handler = proc_dointvec_minmax,
2043 .extra1 = SYSCTL_ONE,
2044 .extra2 = SYSCTL_ONE,
2045 },
2046#endif
2047#ifdef CONFIG_UEVENT_HELPER
2048 {
2049 .procname = "hotplug",
2050 .data = &uevent_helper,
2051 .maxlen = UEVENT_HELPER_PATH_LEN,
2052 .mode = 0644,
2053 .proc_handler = proc_dostring,
2054 },
2055#endif
2056#ifdef CONFIG_CHR_DEV_SG
2057 {
2058 .procname = "sg-big-buff",
2059 .data = &sg_big_buff,
2060 .maxlen = sizeof (int),
2061 .mode = 0444,
2062 .proc_handler = proc_dointvec,
2063 },
2064#endif
2065#ifdef CONFIG_BSD_PROCESS_ACCT
2066 {
2067 .procname = "acct",
2068 .data = &acct_parm,
2069 .maxlen = 3*sizeof(int),
2070 .mode = 0644,
2071 .proc_handler = proc_dointvec,
2072 },
2073#endif
2074#ifdef CONFIG_MAGIC_SYSRQ
2075 {
2076 .procname = "sysrq",
2077 .data = NULL,
2078 .maxlen = sizeof (int),
2079 .mode = 0644,
2080 .proc_handler = sysrq_sysctl_handler,
2081 },
2082#endif
2083#ifdef CONFIG_PROC_SYSCTL
2084 {
2085 .procname = "cad_pid",
2086 .data = NULL,
2087 .maxlen = sizeof (int),
2088 .mode = 0600,
2089 .proc_handler = proc_do_cad_pid,
2090 },
2091#endif
2092 {
2093 .procname = "threads-max",
2094 .data = NULL,
2095 .maxlen = sizeof(int),
2096 .mode = 0644,
2097 .proc_handler = sysctl_max_threads,
2098 },
2099 {
2100 .procname = "random",
2101 .mode = 0555,
2102 .child = random_table,
2103 },
2104 {
2105 .procname = "usermodehelper",
2106 .mode = 0555,
2107 .child = usermodehelper_table,
2108 },
2109#ifdef CONFIG_FW_LOADER_USER_HELPER
2110 {
2111 .procname = "firmware_config",
2112 .mode = 0555,
2113 .child = firmware_config_table,
2114 },
2115#endif
2116 {
2117 .procname = "overflowuid",
2118 .data = &overflowuid,
2119 .maxlen = sizeof(int),
2120 .mode = 0644,
2121 .proc_handler = proc_dointvec_minmax,
2122 .extra1 = &minolduid,
2123 .extra2 = &maxolduid,
2124 },
2125 {
2126 .procname = "overflowgid",
2127 .data = &overflowgid,
2128 .maxlen = sizeof(int),
2129 .mode = 0644,
2130 .proc_handler = proc_dointvec_minmax,
2131 .extra1 = &minolduid,
2132 .extra2 = &maxolduid,
2133 },
2134#ifdef CONFIG_S390
2135 {
2136 .procname = "userprocess_debug",
2137 .data = &show_unhandled_signals,
2138 .maxlen = sizeof(int),
2139 .mode = 0644,
2140 .proc_handler = proc_dointvec,
2141 },
2142#endif
2143 {
2144 .procname = "pid_max",
2145 .data = &pid_max,
2146 .maxlen = sizeof (int),
2147 .mode = 0644,
2148 .proc_handler = proc_dointvec_minmax,
2149 .extra1 = &pid_max_min,
2150 .extra2 = &pid_max_max,
2151 },
2152 {
2153 .procname = "panic_on_oops",
2154 .data = &panic_on_oops,
2155 .maxlen = sizeof(int),
2156 .mode = 0644,
2157 .proc_handler = proc_dointvec,
2158 },
2159 {
2160 .procname = "panic_print",
2161 .data = &panic_print,
2162 .maxlen = sizeof(unsigned long),
2163 .mode = 0644,
2164 .proc_handler = proc_doulongvec_minmax,
2165 },
2166#if defined CONFIG_PRINTK
2167 {
2168 .procname = "printk",
2169 .data = &console_loglevel,
2170 .maxlen = 4*sizeof(int),
2171 .mode = 0644,
2172 .proc_handler = proc_dointvec,
2173 },
2174 {
2175 .procname = "printk_ratelimit",
2176 .data = &printk_ratelimit_state.interval,
2177 .maxlen = sizeof(int),
2178 .mode = 0644,
2179 .proc_handler = proc_dointvec_jiffies,
2180 },
2181 {
2182 .procname = "printk_ratelimit_burst",
2183 .data = &printk_ratelimit_state.burst,
2184 .maxlen = sizeof(int),
2185 .mode = 0644,
2186 .proc_handler = proc_dointvec,
2187 },
2188 {
2189 .procname = "printk_delay",
2190 .data = &printk_delay_msec,
2191 .maxlen = sizeof(int),
2192 .mode = 0644,
2193 .proc_handler = proc_dointvec_minmax,
2194 .extra1 = SYSCTL_ZERO,
2195 .extra2 = &ten_thousand,
2196 },
2197 {
2198 .procname = "printk_devkmsg",
2199 .data = devkmsg_log_str,
2200 .maxlen = DEVKMSG_STR_MAX_SIZE,
2201 .mode = 0644,
2202 .proc_handler = devkmsg_sysctl_set_loglvl,
2203 },
2204 {
2205 .procname = "dmesg_restrict",
2206 .data = &dmesg_restrict,
2207 .maxlen = sizeof(int),
2208 .mode = 0644,
2209 .proc_handler = proc_dointvec_minmax_sysadmin,
2210 .extra1 = SYSCTL_ZERO,
2211 .extra2 = SYSCTL_ONE,
2212 },
2213 {
2214 .procname = "kptr_restrict",
2215 .data = &kptr_restrict,
2216 .maxlen = sizeof(int),
2217 .mode = 0644,
2218 .proc_handler = proc_dointvec_minmax_sysadmin,
2219 .extra1 = SYSCTL_ZERO,
2220 .extra2 = &two,
2221 },
2222#endif
2223 {
2224 .procname = "ngroups_max",
2225 .data = &ngroups_max,
2226 .maxlen = sizeof (int),
2227 .mode = 0444,
2228 .proc_handler = proc_dointvec,
2229 },
2230 {
2231 .procname = "cap_last_cap",
2232 .data = (void *)&cap_last_cap,
2233 .maxlen = sizeof(int),
2234 .mode = 0444,
2235 .proc_handler = proc_dointvec,
2236 },
2237#if defined(CONFIG_LOCKUP_DETECTOR)
2238 {
2239 .procname = "watchdog",
2240 .data = &watchdog_user_enabled,
2241 .maxlen = sizeof(int),
2242 .mode = 0644,
2243 .proc_handler = proc_watchdog,
2244 .extra1 = SYSCTL_ZERO,
2245 .extra2 = SYSCTL_ONE,
2246 },
2247 {
2248 .procname = "watchdog_thresh",
2249 .data = &watchdog_thresh,
2250 .maxlen = sizeof(int),
2251 .mode = 0644,
2252 .proc_handler = proc_watchdog_thresh,
2253 .extra1 = SYSCTL_ZERO,
2254 .extra2 = &sixty,
2255 },
2256 {
2257 .procname = "nmi_watchdog",
2258 .data = &nmi_watchdog_user_enabled,
2259 .maxlen = sizeof(int),
2260 .mode = NMI_WATCHDOG_SYSCTL_PERM,
2261 .proc_handler = proc_nmi_watchdog,
2262 .extra1 = SYSCTL_ZERO,
2263 .extra2 = SYSCTL_ONE,
2264 },
2265 {
2266 .procname = "watchdog_cpumask",
2267 .data = &watchdog_cpumask_bits,
2268 .maxlen = NR_CPUS,
2269 .mode = 0644,
2270 .proc_handler = proc_watchdog_cpumask,
2271 },
2272#ifdef CONFIG_SOFTLOCKUP_DETECTOR
2273 {
2274 .procname = "soft_watchdog",
2275 .data = &soft_watchdog_user_enabled,
2276 .maxlen = sizeof(int),
2277 .mode = 0644,
2278 .proc_handler = proc_soft_watchdog,
2279 .extra1 = SYSCTL_ZERO,
2280 .extra2 = SYSCTL_ONE,
2281 },
2282 {
2283 .procname = "softlockup_panic",
2284 .data = &softlockup_panic,
2285 .maxlen = sizeof(int),
2286 .mode = 0644,
2287 .proc_handler = proc_dointvec_minmax,
2288 .extra1 = SYSCTL_ZERO,
2289 .extra2 = SYSCTL_ONE,
2290 },
2291#ifdef CONFIG_SMP
2292 {
2293 .procname = "softlockup_all_cpu_backtrace",
2294 .data = &sysctl_softlockup_all_cpu_backtrace,
2295 .maxlen = sizeof(int),
2296 .mode = 0644,
2297 .proc_handler = proc_dointvec_minmax,
2298 .extra1 = SYSCTL_ZERO,
2299 .extra2 = SYSCTL_ONE,
2300 },
2301#endif /* CONFIG_SMP */
2302#endif
2303#ifdef CONFIG_HARDLOCKUP_DETECTOR
2304 {
2305 .procname = "hardlockup_panic",
2306 .data = &hardlockup_panic,
2307 .maxlen = sizeof(int),
2308 .mode = 0644,
2309 .proc_handler = proc_dointvec_minmax,
2310 .extra1 = SYSCTL_ZERO,
2311 .extra2 = SYSCTL_ONE,
2312 },
2313#ifdef CONFIG_SMP
2314 {
2315 .procname = "hardlockup_all_cpu_backtrace",
2316 .data = &sysctl_hardlockup_all_cpu_backtrace,
2317 .maxlen = sizeof(int),
2318 .mode = 0644,
2319 .proc_handler = proc_dointvec_minmax,
2320 .extra1 = SYSCTL_ZERO,
2321 .extra2 = SYSCTL_ONE,
2322 },
2323#endif /* CONFIG_SMP */
2324#endif
2325#endif
2326
2327#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
2328 {
2329 .procname = "unknown_nmi_panic",
2330 .data = &unknown_nmi_panic,
2331 .maxlen = sizeof (int),
2332 .mode = 0644,
2333 .proc_handler = proc_dointvec,
2334 },
2335#endif
3368 2336
3369int proc_dointvec_jiffies(struct ctl_table *table, int write,2337#if (defined(CONFIG_X86_32) || defined(CONFIG_PARISC)) && \
3370 void __user *buffer, size_t *lenp, loff_t *ppos)2338 defined(CONFIG_DEBUG_STACKOVERFLOW)
3371{2339 {
3372 return -ENOSYS;2340 .procname = "panic_on_stackoverflow",
3373}2341 .data = &sysctl_panic_on_stackoverflow,
2342 .maxlen = sizeof(int),
2343 .mode = 0644,
2344 .proc_handler = proc_dointvec,
2345 },
2346#endif
2347#if defined(CONFIG_X86)
2348 {
2349 .procname = "panic_on_unrecovered_nmi",
2350 .data = &panic_on_unrecovered_nmi,
2351 .maxlen = sizeof(int),
2352 .mode = 0644,
2353 .proc_handler = proc_dointvec,
2354 },
2355 {
2356 .procname = "panic_on_io_nmi",
2357 .data = &panic_on_io_nmi,
2358 .maxlen = sizeof(int),
2359 .mode = 0644,
2360 .proc_handler = proc_dointvec,
2361 },
2362 {
2363 .procname = "bootloader_type",
2364 .data = &bootloader_type,
2365 .maxlen = sizeof (int),
2366 .mode = 0444,
2367 .proc_handler = proc_dointvec,
2368 },
2369 {
2370 .procname = "bootloader_version",
2371 .data = &bootloader_version,
2372 .maxlen = sizeof (int),
2373 .mode = 0444,
2374 .proc_handler = proc_dointvec,
2375 },
2376 {
2377 .procname = "io_delay_type",
2378 .data = &io_delay_type,
2379 .maxlen = sizeof(int),
2380 .mode = 0644,
2381 .proc_handler = proc_dointvec,
2382 },
2383#endif
2384#if defined(CONFIG_MMU)
2385 {
2386 .procname = "randomize_va_space",
2387 .data = &randomize_va_space,
2388 .maxlen = sizeof(int),
2389 .mode = 0644,
2390 .proc_handler = proc_dointvec,
2391 },
2392#endif
2393#if defined(CONFIG_S390) && defined(CONFIG_SMP)
2394 {
2395 .procname = "spin_retry",
2396 .data = &spin_retry,
2397 .maxlen = sizeof (int),
2398 .mode = 0644,
2399 .proc_handler = proc_dointvec,
2400 },
2401#endif
2402#if defined(CONFIG_ACPI_SLEEP) && defined(CONFIG_X86)
2403 {
2404 .procname = "acpi_video_flags",
2405 .data = &acpi_realmode_flags,
2406 .maxlen = sizeof (unsigned long),
2407 .mode = 0644,
2408 .proc_handler = proc_doulongvec_minmax,
2409 },
2410#endif
2411#ifdef CONFIG_SYSCTL_ARCH_UNALIGN_NO_WARN
2412 {
2413 .procname = "ignore-unaligned-usertrap",
2414 .data = &no_unaligned_warning,
2415 .maxlen = sizeof (int),
2416 .mode = 0644,
2417 .proc_handler = proc_dointvec,
2418 },
2419#endif
2420#ifdef CONFIG_IA64
2421 {
2422 .procname = "unaligned-dump-stack",
2423 .data = &unaligned_dump_stack,
2424 .maxlen = sizeof (int),
2425 .mode = 0644,
2426 .proc_handler = proc_dointvec,
2427 },
2428#endif
2429#ifdef CONFIG_DETECT_HUNG_TASK
2430 {
2431 .procname = "hung_task_panic",
2432 .data = &sysctl_hung_task_panic,
2433 .maxlen = sizeof(int),
2434 .mode = 0644,
2435 .proc_handler = proc_dointvec_minmax,
2436 .extra1 = SYSCTL_ZERO,
2437 .extra2 = SYSCTL_ONE,
2438 },
2439 {
2440 .procname = "hung_task_check_count",
2441 .data = &sysctl_hung_task_check_count,
2442 .maxlen = sizeof(int),
2443 .mode = 0644,
2444 .proc_handler = proc_dointvec_minmax,
2445 .extra1 = SYSCTL_ZERO,
2446 },
2447 {
2448 .procname = "hung_task_timeout_secs",
2449 .data = &sysctl_hung_task_timeout_secs,
2450 .maxlen = sizeof(unsigned long),
2451 .mode = 0644,
2452 .proc_handler = proc_dohung_task_timeout_secs,
2453 .extra2 = &hung_task_timeout_max,
2454 },
2455 {
2456 .procname = "hung_task_check_interval_secs",
2457 .data = &sysctl_hung_task_check_interval_secs,
2458 .maxlen = sizeof(unsigned long),
2459 .mode = 0644,
2460 .proc_handler = proc_dohung_task_timeout_secs,
2461 .extra2 = &hung_task_timeout_max,
2462 },
2463 {
2464 .procname = "hung_task_warnings",
2465 .data = &sysctl_hung_task_warnings,
2466 .maxlen = sizeof(int),
2467 .mode = 0644,
2468 .proc_handler = proc_dointvec_minmax,
2469 .extra1 = &neg_one,
2470 },
2471#endif
2472#ifdef CONFIG_RT_MUTEXES
2473 {
2474 .procname = "max_lock_depth",
2475 .data = &max_lock_depth,
2476 .maxlen = sizeof(int),
2477 .mode = 0644,
2478 .proc_handler = proc_dointvec,
2479 },
2480#endif
2481 {
2482 .procname = "poweroff_cmd",
2483 .data = &poweroff_cmd,
2484 .maxlen = POWEROFF_CMD_PATH_LEN,
2485 .mode = 0644,
2486 .proc_handler = proc_dostring,
2487 },
2488#ifdef CONFIG_KEYS
2489 {
2490 .procname = "keys",
2491 .mode = 0555,
2492 .child = key_sysctls,
2493 },
2494#endif
2495#ifdef CONFIG_PERF_EVENTS
2496 /*
2497 * User-space scripts rely on the existence of this file
2498 * as a feature check for perf_events being enabled.
2499 *
2500 * So it's an ABI, do not remove!
2501 */
2502 {
2503 .procname = "perf_event_paranoid",
2504 .data = &sysctl_perf_event_paranoid,
2505 .maxlen = sizeof(sysctl_perf_event_paranoid),
2506 .mode = 0644,
2507 .proc_handler = proc_dointvec,
2508 },
2509 {
2510 .procname = "perf_event_mlock_kb",
2511 .data = &sysctl_perf_event_mlock,
2512 .maxlen = sizeof(sysctl_perf_event_mlock),
2513 .mode = 0644,
2514 .proc_handler = proc_dointvec,
2515 },
2516 {
2517 .procname = "perf_event_max_sample_rate",
2518 .data = &sysctl_perf_event_sample_rate,
2519 .maxlen = sizeof(sysctl_perf_event_sample_rate),
2520 .mode = 0644,
2521 .proc_handler = perf_proc_update_handler,
2522 .extra1 = SYSCTL_ONE,
2523 },
2524 {
2525 .procname = "perf_cpu_time_max_percent",
2526 .data = &sysctl_perf_cpu_time_max_percent,
2527 .maxlen = sizeof(sysctl_perf_cpu_time_max_percent),
2528 .mode = 0644,
2529 .proc_handler = perf_cpu_time_max_percent_handler,
2530 .extra1 = SYSCTL_ZERO,
2531 .extra2 = &one_hundred,
2532 },
2533 {
2534 .procname = "perf_event_max_stack",
2535 .data = &sysctl_perf_event_max_stack,
2536 .maxlen = sizeof(sysctl_perf_event_max_stack),
2537 .mode = 0644,
2538 .proc_handler = perf_event_max_stack_handler,
2539 .extra1 = SYSCTL_ZERO,
2540 .extra2 = &six_hundred_forty_kb,
2541 },
2542 {
2543 .procname = "perf_event_max_contexts_per_stack",
2544 .data = &sysctl_perf_event_max_contexts_per_stack,
2545 .maxlen = sizeof(sysctl_perf_event_max_contexts_per_stack),
2546 .mode = 0644,
2547 .proc_handler = perf_event_max_stack_handler,
2548 .extra1 = SYSCTL_ZERO,
2549 .extra2 = &one_thousand,
2550 },
2551#endif
2552 {
2553 .procname = "panic_on_warn",
2554 .data = &panic_on_warn,
2555 .maxlen = sizeof(int),
2556 .mode = 0644,
2557 .proc_handler = proc_dointvec_minmax,
2558 .extra1 = SYSCTL_ZERO,
2559 .extra2 = SYSCTL_ONE,
2560 },
2561#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
2562 {
2563 .procname = "timer_migration",
2564 .data = &sysctl_timer_migration,
2565 .maxlen = sizeof(unsigned int),
2566 .mode = 0644,
2567 .proc_handler = timer_migration_handler,
2568 .extra1 = SYSCTL_ZERO,
2569 .extra2 = SYSCTL_ONE,
2570 },
2571#endif
2572#ifdef CONFIG_BPF_SYSCALL
2573 {
2574 .procname = "unprivileged_bpf_disabled",
2575 .data = &sysctl_unprivileged_bpf_disabled,
2576 .maxlen = sizeof(sysctl_unprivileged_bpf_disabled),
2577 .mode = 0644,
2578 /* only handle a transition from default "0" to "1" */
2579 .proc_handler = proc_dointvec_minmax,
2580 .extra1 = SYSCTL_ONE,
2581 .extra2 = SYSCTL_ONE,
2582 },
2583 {
2584 .procname = "bpf_stats_enabled",
2585 .data = &bpf_stats_enabled_key.key,
2586 .maxlen = sizeof(bpf_stats_enabled_key),
2587 .mode = 0644,
2588 .proc_handler = bpf_stats_handler,
2589 },
2590#endif
2591#if defined(CONFIG_TREE_RCU)
2592 {
2593 .procname = "panic_on_rcu_stall",
2594 .data = &sysctl_panic_on_rcu_stall,
2595 .maxlen = sizeof(sysctl_panic_on_rcu_stall),
2596 .mode = 0644,
2597 .proc_handler = proc_dointvec_minmax,
2598 .extra1 = SYSCTL_ZERO,
2599 .extra2 = SYSCTL_ONE,
2600 },
2601#endif
2602#ifdef CONFIG_STACKLEAK_RUNTIME_DISABLE
2603 {
2604 .procname = "stack_erasing",
2605 .data = NULL,
2606 .maxlen = sizeof(int),
2607 .mode = 0600,
2608 .proc_handler = stack_erasing_sysctl,
2609 .extra1 = SYSCTL_ZERO,
2610 .extra2 = SYSCTL_ONE,
2611 },
2612#endif
2613 { }
2614};
3374 2615
3375int proc_dointvec_userhz_jiffies(struct ctl_table *table, int write,2616static struct ctl_table vm_table[] = {
3376 void __user *buffer, size_t *lenp, loff_t *ppos)2617 {
3377{2618 .procname = "overcommit_memory",
3378 return -ENOSYS;2619 .data = &sysctl_overcommit_memory,
3379}2620 .maxlen = sizeof(sysctl_overcommit_memory),
2621 .mode = 0644,
2622 .proc_handler = proc_dointvec_minmax,
2623 .extra1 = SYSCTL_ZERO,
2624 .extra2 = &two,
2625 },
2626 {
2627 .procname = "panic_on_oom",
2628 .data = &sysctl_panic_on_oom,
2629 .maxlen = sizeof(sysctl_panic_on_oom),
2630 .mode = 0644,
2631 .proc_handler = proc_dointvec_minmax,
2632 .extra1 = SYSCTL_ZERO,
2633 .extra2 = &two,
2634 },
2635 {
2636 .procname = "oom_kill_allocating_task",
2637 .data = &sysctl_oom_kill_allocating_task,
2638 .maxlen = sizeof(sysctl_oom_kill_allocating_task),
2639 .mode = 0644,
2640 .proc_handler = proc_dointvec,
2641 },
2642 {
2643 .procname = "oom_dump_tasks",
2644 .data = &sysctl_oom_dump_tasks,
2645 .maxlen = sizeof(sysctl_oom_dump_tasks),
2646 .mode = 0644,
2647 .proc_handler = proc_dointvec,
2648 },
2649 {
2650 .procname = "overcommit_ratio",
2651 .data = &sysctl_overcommit_ratio,
2652 .maxlen = sizeof(sysctl_overcommit_ratio),
2653 .mode = 0644,
2654 .proc_handler = overcommit_ratio_handler,
2655 },
2656 {
2657 .procname = "overcommit_kbytes",
2658 .data = &sysctl_overcommit_kbytes,
2659 .maxlen = sizeof(sysctl_overcommit_kbytes),
2660 .mode = 0644,
2661 .proc_handler = overcommit_kbytes_handler,
2662 },
2663 {
2664 .procname = "page-cluster",
2665 .data = &page_cluster,
2666 .maxlen = sizeof(int),
2667 .mode = 0644,
2668 .proc_handler = proc_dointvec_minmax,
2669 .extra1 = SYSCTL_ZERO,
2670 },
2671 {
2672 .procname = "dirty_background_ratio",
2673 .data = &dirty_background_ratio,
2674 .maxlen = sizeof(dirty_background_ratio),
2675 .mode = 0644,
2676 .proc_handler = dirty_background_ratio_handler,
2677 .extra1 = SYSCTL_ZERO,
2678 .extra2 = &one_hundred,
2679 },
2680 {
2681 .procname = "dirty_background_bytes",
2682 .data = &dirty_background_bytes,
2683 .maxlen = sizeof(dirty_background_bytes),
2684 .mode = 0644,
2685 .proc_handler = dirty_background_bytes_handler,
2686 .extra1 = &one_ul,
2687 },
2688 {
2689 .procname = "dirty_ratio",
2690 .data = &vm_dirty_ratio,
2691 .maxlen = sizeof(vm_dirty_ratio),
2692 .mode = 0644,
2693 .proc_handler = dirty_ratio_handler,
2694 .extra1 = SYSCTL_ZERO,
2695 .extra2 = &one_hundred,
2696 },
2697 {
2698 .procname = "dirty_bytes",
2699 .data = &vm_dirty_bytes,
2700 .maxlen = sizeof(vm_dirty_bytes),
2701 .mode = 0644,
2702 .proc_handler = dirty_bytes_handler,
2703 .extra1 = &dirty_bytes_min,
2704 },
2705 {
2706 .procname = "dirty_writeback_centisecs",
2707 .data = &dirty_writeback_interval,
2708 .maxlen = sizeof(dirty_writeback_interval),
2709 .mode = 0644,
2710 .proc_handler = dirty_writeback_centisecs_handler,
2711 },
2712 {
2713 .procname = "dirty_expire_centisecs",
2714 .data = &dirty_expire_interval,
2715 .maxlen = sizeof(dirty_expire_interval),
2716 .mode = 0644,
2717 .proc_handler = proc_dointvec_minmax,
2718 .extra1 = SYSCTL_ZERO,
2719 },
2720 {
2721 .procname = "dirtytime_expire_seconds",
2722 .data = &dirtytime_expire_interval,
2723 .maxlen = sizeof(dirtytime_expire_interval),
2724 .mode = 0644,
2725 .proc_handler = dirtytime_interval_handler,
2726 .extra1 = SYSCTL_ZERO,
2727 },
2728 {
2729 .procname = "swappiness",
2730 .data = &vm_swappiness,
2731 .maxlen = sizeof(vm_swappiness),
2732 .mode = 0644,
2733 .proc_handler = proc_dointvec_minmax,
2734 .extra1 = SYSCTL_ZERO,
2735 .extra2 = &one_hundred,
2736 },
2737#ifdef CONFIG_HUGETLB_PAGE
2738 {
2739 .procname = "nr_hugepages",
2740 .data = NULL,
2741 .maxlen = sizeof(unsigned long),
2742 .mode = 0644,
2743 .proc_handler = hugetlb_sysctl_handler,
2744 },
2745#ifdef CONFIG_NUMA
2746 {
2747 .procname = "nr_hugepages_mempolicy",
2748 .data = NULL,
2749 .maxlen = sizeof(unsigned long),
2750 .mode = 0644,
2751 .proc_handler = &hugetlb_mempolicy_sysctl_handler,
2752 },
2753 {
2754 .procname = "numa_stat",
2755 .data = &sysctl_vm_numa_stat,
2756 .maxlen = sizeof(int),
2757 .mode = 0644,
2758 .proc_handler = sysctl_vm_numa_stat_handler,
2759 .extra1 = SYSCTL_ZERO,
2760 .extra2 = SYSCTL_ONE,
2761 },
2762#endif
2763 {
2764 .procname = "hugetlb_shm_group",
2765 .data = &sysctl_hugetlb_shm_group,
2766 .maxlen = sizeof(gid_t),
2767 .mode = 0644,
2768 .proc_handler = proc_dointvec,
2769 },
2770 {
2771 .procname = "nr_overcommit_hugepages",
2772 .data = NULL,
2773 .maxlen = sizeof(unsigned long),
2774 .mode = 0644,
2775 .proc_handler = hugetlb_overcommit_handler,
2776 },
2777#endif
2778 {
2779 .procname = "lowmem_reserve_ratio",
2780 .data = &sysctl_lowmem_reserve_ratio,
2781 .maxlen = sizeof(sysctl_lowmem_reserve_ratio),
2782 .mode = 0644,
2783 .proc_handler = lowmem_reserve_ratio_sysctl_handler,
2784 },
2785 {
2786 .procname = "drop_caches",
2787 .data = &sysctl_drop_caches,
2788 .maxlen = sizeof(int),
2789 .mode = 0200,
2790 .proc_handler = drop_caches_sysctl_handler,
2791 .extra1 = SYSCTL_ONE,
2792 .extra2 = &four,
2793 },
2794#ifdef CONFIG_COMPACTION
2795 {
2796 .procname = "compact_memory",
2797 .data = &sysctl_compact_memory,
2798 .maxlen = sizeof(int),
2799 .mode = 0200,
2800 .proc_handler = sysctl_compaction_handler,
2801 },
2802 {
2803 .procname = "extfrag_threshold",
2804 .data = &sysctl_extfrag_threshold,
2805 .maxlen = sizeof(int),
2806 .mode = 0644,
2807 .proc_handler = proc_dointvec_minmax,
2808 .extra1 = &min_extfrag_threshold,
2809 .extra2 = &max_extfrag_threshold,
2810 },
2811 {
2812 .procname = "compact_unevictable_allowed",
2813 .data = &sysctl_compact_unevictable_allowed,
2814 .maxlen = sizeof(int),
2815 .mode = 0644,
2816 .proc_handler = proc_dointvec_minmax_warn_RT_change,
2817 .extra1 = SYSCTL_ZERO,
2818 .extra2 = SYSCTL_ONE,
2819 },
3380 2820
3381int proc_dointvec_ms_jiffies(struct ctl_table *table, int write,2821#endif /* CONFIG_COMPACTION */
3382 void __user *buffer, size_t *lenp, loff_t *ppos)2822 {
3383{2823 .procname = "min_free_kbytes",
3384 return -ENOSYS;2824 .data = &min_free_kbytes,
3385}2825 .maxlen = sizeof(min_free_kbytes),
2826 .mode = 0644,
2827 .proc_handler = min_free_kbytes_sysctl_handler,
2828 .extra1 = SYSCTL_ZERO,
2829 },
2830 {
2831 .procname = "watermark_boost_factor",
2832 .data = &watermark_boost_factor,
2833 .maxlen = sizeof(watermark_boost_factor),
2834 .mode = 0644,
2835 .proc_handler = proc_dointvec_minmax,
2836 .extra1 = SYSCTL_ZERO,
2837 },
2838 {
2839 .procname = "watermark_scale_factor",
2840 .data = &watermark_scale_factor,
2841 .maxlen = sizeof(watermark_scale_factor),
2842 .mode = 0644,
2843 .proc_handler = watermark_scale_factor_sysctl_handler,
2844 .extra1 = SYSCTL_ONE,
2845 .extra2 = &one_thousand,
2846 },
2847 {
2848 .procname = "percpu_pagelist_fraction",
2849 .data = &percpu_pagelist_fraction,
2850 .maxlen = sizeof(percpu_pagelist_fraction),
2851 .mode = 0644,
2852 .proc_handler = percpu_pagelist_fraction_sysctl_handler,
2853 .extra1 = SYSCTL_ZERO,
2854 },
2855#ifdef CONFIG_MMU
2856 {
2857 .procname = "max_map_count",
2858 .data = &sysctl_max_map_count,
2859 .maxlen = sizeof(sysctl_max_map_count),
2860 .mode = 0644,
2861 .proc_handler = proc_dointvec_minmax,
2862 .extra1 = SYSCTL_ZERO,
2863 },
2864#else
2865 {
2866 .procname = "nr_trim_pages",
2867 .data = &sysctl_nr_trim_pages,
2868 .maxlen = sizeof(sysctl_nr_trim_pages),
2869 .mode = 0644,
2870 .proc_handler = proc_dointvec_minmax,
2871 .extra1 = SYSCTL_ZERO,
2872 },
2873#endif
2874 {
2875 .procname = "laptop_mode",
2876 .data = &laptop_mode,
2877 .maxlen = sizeof(laptop_mode),
2878 .mode = 0644,
2879 .proc_handler = proc_dointvec_jiffies,
2880 },
2881 {
2882 .procname = "block_dump",
2883 .data = &block_dump,
2884 .maxlen = sizeof(block_dump),
2885 .mode = 0644,
2886 .proc_handler = proc_dointvec,
2887 .extra1 = SYSCTL_ZERO,
2888 },
2889 {
2890 .procname = "vfs_cache_pressure",
2891 .data = &sysctl_vfs_cache_pressure,
2892 .maxlen = sizeof(sysctl_vfs_cache_pressure),
2893 .mode = 0644,
2894 .proc_handler = proc_dointvec,
2895 .extra1 = SYSCTL_ZERO,
2896 },
2897#if defined(HAVE_ARCH_PICK_MMAP_LAYOUT) || \
2898 defined(CONFIG_ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT)
2899 {
2900 .procname = "legacy_va_layout",
2901 .data = &sysctl_legacy_va_layout,
2902 .maxlen = sizeof(sysctl_legacy_va_layout),
2903 .mode = 0644,
2904 .proc_handler = proc_dointvec,
2905 .extra1 = SYSCTL_ZERO,
2906 },
2907#endif
2908#ifdef CONFIG_NUMA
2909 {
2910 .procname = "zone_reclaim_mode",
2911 .data = &node_reclaim_mode,
2912 .maxlen = sizeof(node_reclaim_mode),
2913 .mode = 0644,
2914 .proc_handler = proc_dointvec,
2915 .extra1 = SYSCTL_ZERO,
2916 },
2917 {
2918 .procname = "min_unmapped_ratio",
2919 .data = &sysctl_min_unmapped_ratio,
2920 .maxlen = sizeof(sysctl_min_unmapped_ratio),
2921 .mode = 0644,
2922 .proc_handler = sysctl_min_unmapped_ratio_sysctl_handler,
2923 .extra1 = SYSCTL_ZERO,
2924 .extra2 = &one_hundred,
2925 },
2926 {
2927 .procname = "min_slab_ratio",
2928 .data = &sysctl_min_slab_ratio,
2929 .maxlen = sizeof(sysctl_min_slab_ratio),
2930 .mode = 0644,
2931 .proc_handler = sysctl_min_slab_ratio_sysctl_handler,
2932 .extra1 = SYSCTL_ZERO,
2933 .extra2 = &one_hundred,
2934 },
2935#endif
2936#ifdef CONFIG_SMP
2937 {
2938 .procname = "stat_interval",
2939 .data = &sysctl_stat_interval,
2940 .maxlen = sizeof(sysctl_stat_interval),
2941 .mode = 0644,
2942 .proc_handler = proc_dointvec_jiffies,
2943 },
2944 {
2945 .procname = "stat_refresh",
2946 .data = NULL,
2947 .maxlen = 0,
2948 .mode = 0600,
2949 .proc_handler = vmstat_refresh,
2950 },
2951#endif
2952#ifdef CONFIG_MMU
2953 {
2954 .procname = "mmap_min_addr",
2955 .data = &dac_mmap_min_addr,
2956 .maxlen = sizeof(unsigned long),
2957 .mode = 0644,
2958 .proc_handler = mmap_min_addr_handler,
2959 },
2960#endif
2961#ifdef CONFIG_NUMA
2962 {
2963 .procname = "numa_zonelist_order",
2964 .data = &numa_zonelist_order,
2965 .maxlen = NUMA_ZONELIST_ORDER_LEN,
2966 .mode = 0644,
2967 .proc_handler = numa_zonelist_order_handler,
2968 },
2969#endif
2970#if (defined(CONFIG_X86_32) && !defined(CONFIG_UML))|| \
2971 (defined(CONFIG_SUPERH) && defined(CONFIG_VSYSCALL))
2972 {
2973 .procname = "vdso_enabled",
2974#ifdef CONFIG_X86_32
2975 .data = &vdso32_enabled,
2976 .maxlen = sizeof(vdso32_enabled),
2977#else
2978 .data = &vdso_enabled,
2979 .maxlen = sizeof(vdso_enabled),
2980#endif
2981 .mode = 0644,
2982 .proc_handler = proc_dointvec,
2983 .extra1 = SYSCTL_ZERO,
2984 },
2985#endif
2986#ifdef CONFIG_HIGHMEM
2987 {
2988 .procname = "highmem_is_dirtyable",
2989 .data = &vm_highmem_is_dirtyable,
2990 .maxlen = sizeof(vm_highmem_is_dirtyable),
2991 .mode = 0644,
2992 .proc_handler = proc_dointvec_minmax,
2993 .extra1 = SYSCTL_ZERO,
2994 .extra2 = SYSCTL_ONE,
2995 },
2996#endif
2997#ifdef CONFIG_MEMORY_FAILURE
2998 {
2999 .procname = "memory_failure_early_kill",
3000 .data = &sysctl_memory_failure_early_kill,
3001 .maxlen = sizeof(sysctl_memory_failure_early_kill),
3002 .mode = 0644,
3003 .proc_handler = proc_dointvec_minmax,
3004 .extra1 = SYSCTL_ZERO,
3005 .extra2 = SYSCTL_ONE,
3006 },
3007 {
3008 .procname = "memory_failure_recovery",
3009 .data = &sysctl_memory_failure_recovery,
3010 .maxlen = sizeof(sysctl_memory_failure_recovery),
3011 .mode = 0644,
3012 .proc_handler = proc_dointvec_minmax,
3013 .extra1 = SYSCTL_ZERO,
3014 .extra2 = SYSCTL_ONE,
3015 },
3016#endif
3017 {
3018 .procname = "user_reserve_kbytes",
3019 .data = &sysctl_user_reserve_kbytes,
3020 .maxlen = sizeof(sysctl_user_reserve_kbytes),
3021 .mode = 0644,
3022 .proc_handler = proc_doulongvec_minmax,
3023 },
3024 {
3025 .procname = "admin_reserve_kbytes",
3026 .data = &sysctl_admin_reserve_kbytes,
3027 .maxlen = sizeof(sysctl_admin_reserve_kbytes),
3028 .mode = 0644,
3029 .proc_handler = proc_doulongvec_minmax,
3030 },
3031#ifdef CONFIG_HAVE_ARCH_MMAP_RND_BITS
3032 {
3033 .procname = "mmap_rnd_bits",
3034 .data = &mmap_rnd_bits,
3035 .maxlen = sizeof(mmap_rnd_bits),
3036 .mode = 0600,
3037 .proc_handler = proc_dointvec_minmax,
3038 .extra1 = (void *)&mmap_rnd_bits_min,
3039 .extra2 = (void *)&mmap_rnd_bits_max,
3040 },
3041#endif
3042#ifdef CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS
3043 {
3044 .procname = "mmap_rnd_compat_bits",
3045 .data = &mmap_rnd_compat_bits,
3046 .maxlen = sizeof(mmap_rnd_compat_bits),
3047 .mode = 0600,
3048 .proc_handler = proc_dointvec_minmax,
3049 .extra1 = (void *)&mmap_rnd_compat_bits_min,
3050 .extra2 = (void *)&mmap_rnd_compat_bits_max,
3051 },
3052#endif
3053#ifdef CONFIG_USERFAULTFD
3054 {
3055 .procname = "unprivileged_userfaultfd",
3056 .data = &sysctl_unprivileged_userfaultfd,
3057 .maxlen = sizeof(sysctl_unprivileged_userfaultfd),
3058 .mode = 0644,
3059 .proc_handler = proc_dointvec_minmax,
3060 .extra1 = SYSCTL_ZERO,
3061 .extra2 = SYSCTL_ONE,
3062 },
3063#endif
3064 { }
3065};
3386 3066
3387int proc_doulongvec_minmax(struct ctl_table *table, int write,3067static struct ctl_table fs_table[] = {
3388 void __user *buffer, size_t *lenp, loff_t *ppos)3068 {
3389{3069 .procname = "inode-nr",
3390 return -ENOSYS;3070 .data = &inodes_stat,
3391}3071 .maxlen = 2*sizeof(long),
3072 .mode = 0444,
3073 .proc_handler = proc_nr_inodes,
3074 },
3075 {
3076 .procname = "inode-state",
3077 .data = &inodes_stat,
3078 .maxlen = 7*sizeof(long),
3079 .mode = 0444,
3080 .proc_handler = proc_nr_inodes,
3081 },
3082 {
3083 .procname = "file-nr",
3084 .data = &files_stat,
3085 .maxlen = sizeof(files_stat),
3086 .mode = 0444,
3087 .proc_handler = proc_nr_files,
3088 },
3089 {
3090 .procname = "file-max",
3091 .data = &files_stat.max_files,
3092 .maxlen = sizeof(files_stat.max_files),
3093 .mode = 0644,
3094 .proc_handler = proc_doulongvec_minmax,
3095 .extra1 = &zero_ul,
3096 .extra2 = &long_max,
3097 },
3098 {
3099 .procname = "nr_open",
3100 .data = &sysctl_nr_open,
3101 .maxlen = sizeof(unsigned int),
3102 .mode = 0644,
3103 .proc_handler = proc_dointvec_minmax,
3104 .extra1 = &sysctl_nr_open_min,
3105 .extra2 = &sysctl_nr_open_max,
3106 },
3107 {
3108 .procname = "dentry-state",
3109 .data = &dentry_stat,
3110 .maxlen = 6*sizeof(long),
3111 .mode = 0444,
3112 .proc_handler = proc_nr_dentry,
3113 },
3114 {
3115 .procname = "overflowuid",
3116 .data = &fs_overflowuid,
3117 .maxlen = sizeof(int),
3118 .mode = 0644,
3119 .proc_handler = proc_dointvec_minmax,
3120 .extra1 = &minolduid,
3121 .extra2 = &maxolduid,
3122 },
3123 {
3124 .procname = "overflowgid",
3125 .data = &fs_overflowgid,
3126 .maxlen = sizeof(int),
3127 .mode = 0644,
3128 .proc_handler = proc_dointvec_minmax,
3129 .extra1 = &minolduid,
3130 .extra2 = &maxolduid,
3131 },
3132#ifdef CONFIG_FILE_LOCKING
3133 {
3134 .procname = "leases-enable",
3135 .data = &leases_enable,
3136 .maxlen = sizeof(int),
3137 .mode = 0644,
3138 .proc_handler = proc_dointvec,
3139 },
3140#endif
3141#ifdef CONFIG_DNOTIFY
3142 {
3143 .procname = "dir-notify-enable",
3144 .data = &dir_notify_enable,
3145 .maxlen = sizeof(int),
3146 .mode = 0644,
3147 .proc_handler = proc_dointvec,
3148 },
3149#endif
3150#ifdef CONFIG_MMU
3151#ifdef CONFIG_FILE_LOCKING
3152 {
3153 .procname = "lease-break-time",
3154 .data = &lease_break_time,
3155 .maxlen = sizeof(int),
3156 .mode = 0644,
3157 .proc_handler = proc_dointvec,
3158 },
3159#endif
3160#ifdef CONFIG_AIO
3161 {
3162 .procname = "aio-nr",
3163 .data = &aio_nr,
3164 .maxlen = sizeof(aio_nr),
3165 .mode = 0444,
3166 .proc_handler = proc_doulongvec_minmax,
3167 },
3168 {
3169 .procname = "aio-max-nr",
3170 .data = &aio_max_nr,
3171 .maxlen = sizeof(aio_max_nr),
3172 .mode = 0644,
3173 .proc_handler = proc_doulongvec_minmax,
3174 },
3175#endif /* CONFIG_AIO */
3176#ifdef CONFIG_INOTIFY_USER
3177 {
3178 .procname = "inotify",
3179 .mode = 0555,
3180 .child = inotify_table,
3181 },
3182#endif
3183#ifdef CONFIG_EPOLL
3184 {
3185 .procname = "epoll",
3186 .mode = 0555,
3187 .child = epoll_table,
3188 },
3189#endif
3190#endif
3191 {
3192 .procname = "protected_symlinks",
3193 .data = &sysctl_protected_symlinks,
3194 .maxlen = sizeof(int),
3195 .mode = 0600,
3196 .proc_handler = proc_dointvec_minmax,
3197 .extra1 = SYSCTL_ZERO,
3198 .extra2 = SYSCTL_ONE,
3199 },
3200 {
3201 .procname = "protected_hardlinks",
3202 .data = &sysctl_protected_hardlinks,
3203 .maxlen = sizeof(int),
3204 .mode = 0600,
3205 .proc_handler = proc_dointvec_minmax,
3206 .extra1 = SYSCTL_ZERO,
3207 .extra2 = SYSCTL_ONE,
3208 },
3209 {
3210 .procname = "protected_fifos",
3211 .data = &sysctl_protected_fifos,
3212 .maxlen = sizeof(int),
3213 .mode = 0600,
3214 .proc_handler = proc_dointvec_minmax,
3215 .extra1 = SYSCTL_ZERO,
3216 .extra2 = &two,
3217 },
3218 {
3219 .procname = "protected_regular",
3220 .data = &sysctl_protected_regular,
3221 .maxlen = sizeof(int),
3222 .mode = 0600,
3223 .proc_handler = proc_dointvec_minmax,
3224 .extra1 = SYSCTL_ZERO,
3225 .extra2 = &two,
3226 },
3227 {
3228 .procname = "suid_dumpable",
3229 .data = &suid_dumpable,
3230 .maxlen = sizeof(int),
3231 .mode = 0644,
3232 .proc_handler = proc_dointvec_minmax_coredump,
3233 .extra1 = SYSCTL_ZERO,
3234 .extra2 = &two,
3235 },
3236#if defined(CONFIG_BINFMT_MISC) || defined(CONFIG_BINFMT_MISC_MODULE)
3237 {
3238 .procname = "binfmt_misc",
3239 .mode = 0555,
3240 .child = sysctl_mount_point,
3241 },
3242#endif
3243 {
3244 .procname = "pipe-max-size",
3245 .data = &pipe_max_size,
3246 .maxlen = sizeof(pipe_max_size),
3247 .mode = 0644,
3248 .proc_handler = proc_dopipe_max_size,
3249 },
3250 {
3251 .procname = "pipe-user-pages-hard",
3252 .data = &pipe_user_pages_hard,
3253 .maxlen = sizeof(pipe_user_pages_hard),
3254 .mode = 0644,
3255 .proc_handler = proc_doulongvec_minmax,
3256 },
3257 {
3258 .procname = "pipe-user-pages-soft",
3259 .data = &pipe_user_pages_soft,
3260 .maxlen = sizeof(pipe_user_pages_soft),
3261 .mode = 0644,
3262 .proc_handler = proc_doulongvec_minmax,
3263 },
3264 {
3265 .procname = "mount-max",
3266 .data = &sysctl_mount_max,
3267 .maxlen = sizeof(unsigned int),
3268 .mode = 0644,
3269 .proc_handler = proc_dointvec_minmax,
3270 .extra1 = SYSCTL_ONE,
3271 },
3272 { }
3273};
3392 3274
3393int proc_doulongvec_ms_jiffies_minmax(struct ctl_table *table, int write,3275static struct ctl_table debug_table[] = {
3394 void __user *buffer,3276#ifdef CONFIG_SYSCTL_EXCEPTION_TRACE
3395 size_t *lenp, loff_t *ppos)3277 {
3396{3278 .procname = "exception-trace",
3397 return -ENOSYS;3279 .data = &show_unhandled_signals,
3398}3280 .maxlen = sizeof(int),
3281 .mode = 0644,
3282 .proc_handler = proc_dointvec
3283 },
3284#endif
3285#if defined(CONFIG_OPTPROBES)
3286 {
3287 .procname = "kprobes-optimization",
3288 .data = &sysctl_kprobes_optimization,
3289 .maxlen = sizeof(int),
3290 .mode = 0644,
3291 .proc_handler = proc_kprobes_optimization_handler,
3292 .extra1 = SYSCTL_ZERO,
3293 .extra2 = SYSCTL_ONE,
3294 },
3295#endif
3296 { }
3297};
3399 3298
3400int proc_do_large_bitmap(struct ctl_table *table, int write,3299static struct ctl_table dev_table[] = {
3401 void __user *buffer, size_t *lenp, loff_t *ppos)3300 { }
3402{3301};
3403 return -ENOSYS;
3404}
3405 3302
3406#endif /* CONFIG_PROC_SYSCTL */3303static struct ctl_table sysctl_base_table[] = {
3304 {
3305 .procname = "kernel",
3306 .mode = 0555,
3307 .child = kern_table,
3308 },
3309 {
3310 .procname = "vm",
3311 .mode = 0555,
3312 .child = vm_table,
3313 },
3314 {
3315 .procname = "fs",
3316 .mode = 0555,
3317 .child = fs_table,
3318 },
3319 {
3320 .procname = "debug",
3321 .mode = 0555,
3322 .child = debug_table,
3323 },
3324 {
3325 .procname = "dev",
3326 .mode = 0555,
3327 .child = dev_table,
3328 },
3329 { }
3330};
3407 3331
3408#if defined(CONFIG_SYSCTL)3332int __init sysctl_init(void)
3409int proc_do_static_key(struct ctl_table *table, int write,
3410 void __user *buffer, size_t *lenp,
3411 loff_t *ppos)
3412{3333{
3413 struct static_key *key = (struct static_key *)table->data;3334 struct ctl_table_header *hdr;
3414 static DEFINE_MUTEX(static_key_mutex);
3415 int val, ret;
3416 struct ctl_table tmp = {
3417 .data = &val,
3418 .maxlen = sizeof(val),
3419 .mode = table->mode,
3420 .extra1 = SYSCTL_ZERO,
3421 .extra2 = SYSCTL_ONE,
3422 };
3423
3424 if (write && !capable(CAP_SYS_ADMIN))
3425 return -EPERM;
3426 3335
3427 mutex_lock(&static_key_mutex);3336 hdr = register_sysctl_table(sysctl_base_table);
3428 val = static_key_enabled(key);3337 kmemleak_not_leak(hdr);
3429 ret = proc_dointvec_minmax(&tmp, write, buffer, lenp, ppos);3338 return 0;
3430 if (write && !ret) {
3431 if (val)
3432 static_key_enable(key);
3433 else
3434 static_key_disable(key);
3435 }
3436 mutex_unlock(&static_key_mutex);
3437 return ret;
3438}3339}
3439#endif3340#endif /* CONFIG_SYSCTL */
3440/*3341/*
3441 * No sense putting this after each symbol definition, twice,3342 * No sense putting this after each symbol definition, twice,
3442 * exception granted :-)3343 * exception granted :-)
kernel/time/timer.c+1−2
@@ -249,8 +249,7 @@ void timers_update_nohz(void) · 248 unmodified lines
249}249}
250 250
251int timer_migration_handler(struct ctl_table *table, int write,251int timer_migration_handler(struct ctl_table *table, int write,
252 void __user *buffer, size_t *lenp,252 void *buffer, size_t *lenp, loff_t *ppos)
253 loff_t *ppos)
254{253{
255 int ret;254 int ret;
256 255
kernel/trace/bpf_trace.c+235−6
@@ -147,7 +147,7 @@ BPF_CALL_3(bpf_probe_read_user, void *, dst, u32, size, · 146 unmodified lines
147 return ret;147 return ret;
148}148}
149 149
150static const struct bpf_func_proto bpf_probe_read_user_proto = {150const struct bpf_func_proto bpf_probe_read_user_proto = {
151 .func = bpf_probe_read_user,151 .func = bpf_probe_read_user,
152 .gpl_only = true,152 .gpl_only = true,
153 .ret_type = RET_INTEGER,153 .ret_type = RET_INTEGER,
@@ -167,7 +167,7 @@ BPF_CALL_3(bpf_probe_read_user_str, void *, dst, u32, size, · 13 unmodified lines
167 return ret;167 return ret;
168}168}
169 169
170static const struct bpf_func_proto bpf_probe_read_user_str_proto = {170const struct bpf_func_proto bpf_probe_read_user_str_proto = {
171 .func = bpf_probe_read_user_str,171 .func = bpf_probe_read_user_str,
172 .gpl_only = true,172 .gpl_only = true,
173 .ret_type = RET_INTEGER,173 .ret_type = RET_INTEGER,
@@ -198,7 +198,7 @@ BPF_CALL_3(bpf_probe_read_kernel, void *, dst, u32, size, · 24 unmodified lines
198 return bpf_probe_read_kernel_common(dst, size, unsafe_ptr, false);198 return bpf_probe_read_kernel_common(dst, size, unsafe_ptr, false);
199}199}
200 200
201static const struct bpf_func_proto bpf_probe_read_kernel_proto = {201const struct bpf_func_proto bpf_probe_read_kernel_proto = {
202 .func = bpf_probe_read_kernel,202 .func = bpf_probe_read_kernel,
203 .gpl_only = true,203 .gpl_only = true,
204 .ret_type = RET_INTEGER,204 .ret_type = RET_INTEGER,
@@ -253,7 +253,7 @@ BPF_CALL_3(bpf_probe_read_kernel_str, void *, dst, u32, size, · 48 unmodified lines
253 return bpf_probe_read_kernel_str_common(dst, size, unsafe_ptr, false);253 return bpf_probe_read_kernel_str_common(dst, size, unsafe_ptr, false);
254}254}
255 255
256static const struct bpf_func_proto bpf_probe_read_kernel_str_proto = {256const struct bpf_func_proto bpf_probe_read_kernel_str_proto = {
257 .func = bpf_probe_read_kernel_str,257 .func = bpf_probe_read_kernel_str,
258 .gpl_only = true,258 .gpl_only = true,
259 .ret_type = RET_INTEGER,259 .ret_type = RET_INTEGER,
@@ -315,6 +315,9 @@ static const struct bpf_func_proto bpf_probe_write_user_proto = { · 55 unmodified lines
315 315
316static const struct bpf_func_proto *bpf_get_probe_write_proto(void)316static const struct bpf_func_proto *bpf_get_probe_write_proto(void)
317{317{
318 if (!capable(CAP_SYS_ADMIN))
319 return NULL;
320
318 pr_warn_ratelimited("%s[%d] is installing a program with bpf_probe_write_user helper that may corrupt user memory!",321 pr_warn_ratelimited("%s[%d] is installing a program with bpf_probe_write_user helper that may corrupt user memory!",
319 current->comm, task_pid_nr(current));322 current->comm, task_pid_nr(current));
320 323
@@ -487,6 +490,212 @@ const struct bpf_func_proto *bpf_get_trace_printk_proto(void) · 166 unmodified lines
487 return &bpf_trace_printk_proto;490 return &bpf_trace_printk_proto;
488}491}
489 492
493#define MAX_SEQ_PRINTF_VARARGS 12
494#define MAX_SEQ_PRINTF_MAX_MEMCPY 6
495#define MAX_SEQ_PRINTF_STR_LEN 128
496
497struct bpf_seq_printf_buf {
498 char buf[MAX_SEQ_PRINTF_MAX_MEMCPY][MAX_SEQ_PRINTF_STR_LEN];
499};
500static DEFINE_PER_CPU(struct bpf_seq_printf_buf, bpf_seq_printf_buf);
501static DEFINE_PER_CPU(int, bpf_seq_printf_buf_used);
502
503BPF_CALL_5(bpf_seq_printf, struct seq_file *, m, char *, fmt, u32, fmt_size,
504 const void *, data, u32, data_len)
505{
506 int err = -EINVAL, fmt_cnt = 0, memcpy_cnt = 0;
507 int i, buf_used, copy_size, num_args;
508 u64 params[MAX_SEQ_PRINTF_VARARGS];
509 struct bpf_seq_printf_buf *bufs;
510 const u64 *args = data;
511
512 buf_used = this_cpu_inc_return(bpf_seq_printf_buf_used);
513 if (WARN_ON_ONCE(buf_used > 1)) {
514 err = -EBUSY;
515 goto out;
516 }
517
518 bufs = this_cpu_ptr(&bpf_seq_printf_buf);
519
520 /*
521 * bpf_check()->check_func_arg()->check_stack_boundary()
522 * guarantees that fmt points to bpf program stack,
523 * fmt_size bytes of it were initialized and fmt_size > 0
524 */
525 if (fmt[--fmt_size] != 0)
526 goto out;
527
528 if (data_len & 7)
529 goto out;
530
531 for (i = 0; i < fmt_size; i++) {
532 if (fmt[i] == '%') {
533 if (fmt[i + 1] == '%')
534 i++;
535 else if (!data || !data_len)
536 goto out;
537 }
538 }
539
540 num_args = data_len / 8;
541
542 /* check format string for allowed specifiers */
543 for (i = 0; i < fmt_size; i++) {
544 /* only printable ascii for now. */
545 if ((!isprint(fmt[i]) && !isspace(fmt[i])) || !isascii(fmt[i])) {
546 err = -EINVAL;
547 goto out;
548 }
549
550 if (fmt[i] != '%')
551 continue;
552
553 if (fmt[i + 1] == '%') {
554 i++;
555 continue;
556 }
557
558 if (fmt_cnt >= MAX_SEQ_PRINTF_VARARGS) {
559 err = -E2BIG;
560 goto out;
561 }
562
563 if (fmt_cnt >= num_args) {
564 err = -EINVAL;
565 goto out;
566 }
567
568 /* fmt[i] != 0 && fmt[last] == 0, so we can access fmt[i + 1] */
569 i++;
570
571 /* skip optional "[0 +-][num]" width formating field */
572 while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
573 fmt[i] == ' ')
574 i++;
575 if (fmt[i] >= '1' && fmt[i] <= '9') {
576 i++;
577 while (fmt[i] >= '0' && fmt[i] <= '9')
578 i++;
579 }
580
581 if (fmt[i] == 's') {
582 /* try our best to copy */
583 if (memcpy_cnt >= MAX_SEQ_PRINTF_MAX_MEMCPY) {
584 err = -E2BIG;
585 goto out;
586 }
587
588 err = strncpy_from_unsafe_strict(bufs->buf[memcpy_cnt],
589 (void *) (long) args[fmt_cnt],
590 MAX_SEQ_PRINTF_STR_LEN);
591 if (err < 0)
592 bufs->buf[memcpy_cnt][0] = '\0';
593 params[fmt_cnt] = (u64)(long)bufs->buf[memcpy_cnt];
594
595 fmt_cnt++;
596 memcpy_cnt++;
597 continue;
598 }
599
600 if (fmt[i] == 'p') {
601 if (fmt[i + 1] == 0 ||
602 fmt[i + 1] == 'K' ||
603 fmt[i + 1] == 'x') {
604 /* just kernel pointers */
605 params[fmt_cnt] = args[fmt_cnt];
606 fmt_cnt++;
607 continue;
608 }
609
610 /* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
611 if (fmt[i + 1] != 'i' && fmt[i + 1] != 'I') {
612 err = -EINVAL;
613 goto out;
614 }
615 if (fmt[i + 2] != '4' && fmt[i + 2] != '6') {
616 err = -EINVAL;
617 goto out;
618 }
619
620 if (memcpy_cnt >= MAX_SEQ_PRINTF_MAX_MEMCPY) {
621 err = -E2BIG;
622 goto out;
623 }
624
625
626 copy_size = (fmt[i + 2] == '4') ? 4 : 16;
627
628 err = probe_kernel_read(bufs->buf[memcpy_cnt],
629 (void *) (long) args[fmt_cnt],
630 copy_size);
631 if (err < 0)
632 memset(bufs->buf[memcpy_cnt], 0, copy_size);
633 params[fmt_cnt] = (u64)(long)bufs->buf[memcpy_cnt];
634
635 i += 2;
636 fmt_cnt++;
637 memcpy_cnt++;
638 continue;
639 }
640
641 if (fmt[i] == 'l') {
642 i++;
643 if (fmt[i] == 'l')
644 i++;
645 }
646
647 if (fmt[i] != 'i' && fmt[i] != 'd' &&
648 fmt[i] != 'u' && fmt[i] != 'x') {
649 err = -EINVAL;
650 goto out;
651 }
652
653 params[fmt_cnt] = args[fmt_cnt];
654 fmt_cnt++;
655 }
656
657 /* Maximumly we can have MAX_SEQ_PRINTF_VARARGS parameter, just give
658 * all of them to seq_printf().
659 */
660 seq_printf(m, fmt, params[0], params[1], params[2], params[3],
661 params[4], params[5], params[6], params[7], params[8],
662 params[9], params[10], params[11]);
663
664 err = seq_has_overflowed(m) ? -EOVERFLOW : 0;
665out:
666 this_cpu_dec(bpf_seq_printf_buf_used);
667 return err;
668}
669
670static int bpf_seq_printf_btf_ids[5];
671static const struct bpf_func_proto bpf_seq_printf_proto = {
672 .func = bpf_seq_printf,
673 .gpl_only = true,
674 .ret_type = RET_INTEGER,
675 .arg1_type = ARG_PTR_TO_BTF_ID,
676 .arg2_type = ARG_PTR_TO_MEM,
677 .arg3_type = ARG_CONST_SIZE,
678 .arg4_type = ARG_PTR_TO_MEM_OR_NULL,
679 .arg5_type = ARG_CONST_SIZE_OR_ZERO,
680 .btf_id = bpf_seq_printf_btf_ids,
681};
682
683BPF_CALL_3(bpf_seq_write, struct seq_file *, m, const void *, data, u32, len)
684{
685 return seq_write(m, data, len) ? -EOVERFLOW : 0;
686}
687
688static int bpf_seq_write_btf_ids[5];
689static const struct bpf_func_proto bpf_seq_write_proto = {
690 .func = bpf_seq_write,
691 .gpl_only = true,
692 .ret_type = RET_INTEGER,
693 .arg1_type = ARG_PTR_TO_BTF_ID,
694 .arg2_type = ARG_PTR_TO_MEM,
695 .arg3_type = ARG_CONST_SIZE_OR_ZERO,
696 .btf_id = bpf_seq_write_btf_ids,
697};
698
490static __always_inline int699static __always_inline int
491get_map_perf_counter(struct bpf_map *map, u64 flags,700get_map_perf_counter(struct bpf_map *map, u64 flags,
492 u64 *value, u64 *enabled, u64 *running)701 u64 *value, u64 *enabled, u64 *running)
@@ -698,7 +907,7 @@ BPF_CALL_0(bpf_get_current_task) · 205 unmodified lines
698 return (long) current;907 return (long) current;
699}908}
700 909
701static const struct bpf_func_proto bpf_get_current_task_proto = {910const struct bpf_func_proto bpf_get_current_task_proto = {
702 .func = bpf_get_current_task,911 .func = bpf_get_current_task,
703 .gpl_only = true,912 .gpl_only = true,
704 .ret_type = RET_INTEGER,913 .ret_type = RET_INTEGER,
@@ -827,6 +1036,8 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 122 unmodified lines
827 return &bpf_map_peek_elem_proto;1036 return &bpf_map_peek_elem_proto;
828 case BPF_FUNC_ktime_get_ns:1037 case BPF_FUNC_ktime_get_ns:
829 return &bpf_ktime_get_ns_proto;1038 return &bpf_ktime_get_ns_proto;
1039 case BPF_FUNC_ktime_get_boot_ns:
1040 return &bpf_ktime_get_boot_ns_proto;
830 case BPF_FUNC_tail_call:1041 case BPF_FUNC_tail_call:
831 return &bpf_tail_call_proto;1042 return &bpf_tail_call_proto;
832 case BPF_FUNC_get_current_pid_tgid:1043 case BPF_FUNC_get_current_pid_tgid:
@@ -877,6 +1088,16 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 44 unmodified lines
877 return &bpf_perf_event_read_value_proto;1088 return &bpf_perf_event_read_value_proto;
878 case BPF_FUNC_get_ns_current_pid_tgid:1089 case BPF_FUNC_get_ns_current_pid_tgid:
879 return &bpf_get_ns_current_pid_tgid_proto;1090 return &bpf_get_ns_current_pid_tgid_proto;
1091 case BPF_FUNC_ringbuf_output:
1092 return &bpf_ringbuf_output_proto;
1093 case BPF_FUNC_ringbuf_reserve:
1094 return &bpf_ringbuf_reserve_proto;
1095 case BPF_FUNC_ringbuf_submit:
1096 return &bpf_ringbuf_submit_proto;
1097 case BPF_FUNC_ringbuf_discard:
1098 return &bpf_ringbuf_discard_proto;
1099 case BPF_FUNC_ringbuf_query:
1100 return &bpf_ringbuf_query_proto;
880 default:1101 default:
881 return NULL;1102 return NULL;
882 }1103 }
@@ -1246,7 +1467,7 @@ raw_tp_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 363 unmodified lines
1246 }1467 }
1247}1468}
1248 1469
1249static const struct bpf_func_proto *1470const struct bpf_func_proto *
1250tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)1471tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
1251{1472{
1252 switch (func_id) {1473 switch (func_id) {
@@ -1256,6 +1477,14 @@ tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 3 unmodified lines
1256 case BPF_FUNC_xdp_output:1477 case BPF_FUNC_xdp_output:
1257 return &bpf_xdp_output_proto;1478 return &bpf_xdp_output_proto;
1258#endif1479#endif
1480 case BPF_FUNC_seq_printf:
1481 return prog->expected_attach_type == BPF_TRACE_ITER ?
1482 &bpf_seq_printf_proto :
1483 NULL;
1484 case BPF_FUNC_seq_write:
1485 return prog->expected_attach_type == BPF_TRACE_ITER ?
1486 &bpf_seq_write_proto :
1487 NULL;
1259 default:1488 default:
1260 return raw_tp_prog_func_proto(func_id, prog);1489 return raw_tp_prog_func_proto(func_id, prog);
1261 }1490 }
kernel/trace/trace.c+1−1
@@ -2662,7 +2662,7 @@ static void output_printk(struct trace_event_buffer *fbuffer) · 2661 unmodified lines
2662}2662}
2663 2663
2664int tracepoint_printk_sysctl(struct ctl_table *table, int write,2664int tracepoint_printk_sysctl(struct ctl_table *table, int write,
2665 void __user *buffer, size_t *lenp,2665 void *buffer, size_t *lenp,
2666 loff_t *ppos)2666 loff_t *ppos)
2667{2667{
2668 int save_tracepoint_printk;2668 int save_tracepoint_printk;
kernel/umh.c+1−1
@@ -641,7 +641,7 @@ int call_usermodehelper(const char *path, char **argv, char **envp, int wait) · 640 unmodified lines
641EXPORT_SYMBOL(call_usermodehelper);641EXPORT_SYMBOL(call_usermodehelper);
642 642
643static int proc_cap_handler(struct ctl_table *table, int write,643static int proc_cap_handler(struct ctl_table *table, int write,
644 void __user *buffer, size_t *lenp, loff_t *ppos)644 void *buffer, size_t *lenp, loff_t *ppos)
645{645{
646 struct ctl_table t;646 struct ctl_table t;
647 unsigned long cap_array[_KERNEL_CAPABILITY_U32S];647 unsigned long cap_array[_KERNEL_CAPABILITY_U32S];
kernel/utsname_sysctl.c+1−1
@@ -30,7 +30,7 @@ static void *get_uts(struct ctl_table *table) · 29 unmodified lines
30 * to observe. Should this be in kernel/sys.c ????30 * to observe. Should this be in kernel/sys.c ????
31 */31 */
32static int proc_do_uts_string(struct ctl_table *table, int write,32static int proc_do_uts_string(struct ctl_table *table, int write,
33 void __user *buffer, size_t *lenp, loff_t *ppos)33 void *buffer, size_t *lenp, loff_t *ppos)
34{34{
35 struct ctl_table uts_table;35 struct ctl_table uts_table;
36 int r;36 int r;
kernel/watchdog.c+6−6
@@ -661,7 +661,7 @@ static void proc_watchdog_update(void) · 660 unmodified lines
661 * proc_soft_watchdog | soft_watchdog_user_enabled | SOFT_WATCHDOG_ENABLED661 * proc_soft_watchdog | soft_watchdog_user_enabled | SOFT_WATCHDOG_ENABLED
662 */662 */
663static int proc_watchdog_common(int which, struct ctl_table *table, int write,663static int proc_watchdog_common(int which, struct ctl_table *table, int write,
664 void __user *buffer, size_t *lenp, loff_t *ppos)664 void *buffer, size_t *lenp, loff_t *ppos)
665{665{
666 int err, old, *param = table->data;666 int err, old, *param = table->data;
667 667
@@ -688,7 +688,7 @@ static int proc_watchdog_common(int which, struct ctl_table *table, int write, · 20 unmodified lines
688 * /proc/sys/kernel/watchdog688 * /proc/sys/kernel/watchdog
689 */689 */
690int proc_watchdog(struct ctl_table *table, int write,690int proc_watchdog(struct ctl_table *table, int write,
691 void __user *buffer, size_t *lenp, loff_t *ppos)691 void *buffer, size_t *lenp, loff_t *ppos)
692{692{
693 return proc_watchdog_common(NMI_WATCHDOG_ENABLED|SOFT_WATCHDOG_ENABLED,693 return proc_watchdog_common(NMI_WATCHDOG_ENABLED|SOFT_WATCHDOG_ENABLED,
694 table, write, buffer, lenp, ppos);694 table, write, buffer, lenp, ppos);
@@ -698,7 +698,7 @@ int proc_watchdog(struct ctl_table *table, int write, · 3 unmodified lines
698 * /proc/sys/kernel/nmi_watchdog698 * /proc/sys/kernel/nmi_watchdog
699 */699 */
700int proc_nmi_watchdog(struct ctl_table *table, int write,700int proc_nmi_watchdog(struct ctl_table *table, int write,
701 void __user *buffer, size_t *lenp, loff_t *ppos)701 void *buffer, size_t *lenp, loff_t *ppos)
702{702{
703 if (!nmi_watchdog_available && write)703 if (!nmi_watchdog_available && write)
704 return -ENOTSUPP;704 return -ENOTSUPP;
@@ -710,7 +710,7 @@ int proc_nmi_watchdog(struct ctl_table *table, int write, · 5 unmodified lines
710 * /proc/sys/kernel/soft_watchdog710 * /proc/sys/kernel/soft_watchdog
711 */711 */
712int proc_soft_watchdog(struct ctl_table *table, int write,712int proc_soft_watchdog(struct ctl_table *table, int write,
713 void __user *buffer, size_t *lenp, loff_t *ppos)713 void *buffer, size_t *lenp, loff_t *ppos)
714{714{
715 return proc_watchdog_common(SOFT_WATCHDOG_ENABLED,715 return proc_watchdog_common(SOFT_WATCHDOG_ENABLED,
716 table, write, buffer, lenp, ppos);716 table, write, buffer, lenp, ppos);
@@ -720,7 +720,7 @@ int proc_soft_watchdog(struct ctl_table *table, int write, · 3 unmodified lines
720 * /proc/sys/kernel/watchdog_thresh720 * /proc/sys/kernel/watchdog_thresh
721 */721 */
722int proc_watchdog_thresh(struct ctl_table *table, int write,722int proc_watchdog_thresh(struct ctl_table *table, int write,
723 void __user *buffer, size_t *lenp, loff_t *ppos)723 void *buffer, size_t *lenp, loff_t *ppos)
724{724{
725 int err, old;725 int err, old;
726 726
@@ -743,7 +743,7 @@ int proc_watchdog_thresh(struct ctl_table *table, int write, · 16 unmodified lines
743 * been brought online, if desired.743 * been brought online, if desired.
744 */744 */
745int proc_watchdog_cpumask(struct ctl_table *table, int write,745int proc_watchdog_cpumask(struct ctl_table *table, int write,
746 void __user *buffer, size_t *lenp, loff_t *ppos)746 void *buffer, size_t *lenp, loff_t *ppos)
747{747{
748 int err;748 int err;
749 749

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t