History

Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next

Pull networking updates from David Miller: 1) Allow setting bluetooth L2CAP modes via socket option, from Luiz Augusto von Dentz. 2) Add GSO partial support to igc, from Sasha Neftin. 3) Several cleanups and improvements to r8169 from Heiner Kallweit. 4) Add IF_OPER_TESTING link state and use it when ethtool triggers a device self-test. From Andrew Lunn. 5) Start moving away from custom driver versions, use the globally defined kernel version instead, from Leon Romanovsky. 6) Support GRO vis gro_cells in DSA layer, from Alexander Lobakin. 7) Allow hard IRQ deferral during NAPI, from Eric Dumazet. 8) Add sriov and vf support to hinic, from Luo bin. 9) Support Media Redundancy Protocol (MRP) in the bridging code, from Horatiu Vultur. 10) Support netmap in the nft_nat code, from Pablo Neira Ayuso. 11) Allow UDPv6 encapsulation of ESP in the ipsec code, from Sabrina Dubroca. Also add ipv6 support for espintcp. 12) Lots of ReST conversions of the networking documentation, from Mauro Carvalho Chehab. 13) Support configuration of ethtool rxnfc flows in bcmgenet driver, from Doug Berger. 14) Allow to dump cgroup id and filter by it in inet_diag code, from Dmitry Yakunin. 15) Add infrastructure to export netlink attribute policies to userspace, from Johannes Berg. 16) Several optimizations to sch_fq scheduler, from Eric Dumazet. 17) Fallback to the default qdisc if qdisc init fails because otherwise a packet scheduler init failure will make a device inoperative. From Jesper Dangaard Brouer. 18) Several RISCV bpf jit optimizations, from Luke Nelson. 19) Correct the return type of the ->ndo_start_xmit() method in several drivers, it's netdev_tx_t but many drivers were using 'int'. From Yunjian Wang. 20) Add an ethtool interface for PHY master/slave config, from Oleksij Rempel. 21) Add BPF iterators, from Yonghang Song. 22) Add cable test infrastructure, including ethool interfaces, from Andrew Lunn. Marvell PHY driver is the first to support this facility. 23) Remove zero-length arrays all over, from Gustavo A. R. Silva. 24) Calculate and maintain an explicit frame size in XDP, from Jesper Dangaard Brouer. 25) Add CAP_BPF, from Alexei Starovoitov. 26) Support terse dumps in the packet scheduler, from Vlad Buslov. 27) Support XDP_TX bulking in dpaa2 driver, from Ioana Ciornei. 28) Add devm_register_netdev(), from Bartosz Golaszewski. 29) Minimize qdisc resets, from Cong Wang. 30) Get rid of kernel_getsockopt and kernel_setsockopt in order to eliminate set_fs/get_fs calls. From Christoph Hellwig. * git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next: (2517 commits) selftests: net: ip_defrag: ignore EPERM net_failover: fixed rollback in net_failover_open() Revert "tipc: Fix potential tipc_aead refcnt leak in tipc_crypto_rcv" Revert "tipc: Fix potential tipc_node refcnt leak in tipc_rcv" vmxnet3: allow rx flow hash ops only when rss is enabled hinic: add set_channels ethtool_ops support selftests/bpf: Add a default $(CXX) value tools/bpf: Don't use $(COMPILE.c) bpf, selftests: Use bpf_probe_read_kernel s390/bpf: Use bcr 0,%0 as tail call nop filler s390/bpf: Maintain 8-byte stack alignment selftests/bpf: Fix verifier test selftests/bpf: Fix sample_cnt shared between two threads bpf, selftests: Adapt cls_redirect to call csum_level helper bpf: Add csum_level helper for fixing up csum levels bpf: Fix up bpf_skb_adjust_room helper's skb csum setting sfc: add missing annotation for efx_ef10_try_update_nic_stats_vf() crypto/chtls: IPv6 support for inline TLS Crypto/chcr: Fixes a coccinile check error Crypto/chcr: Fixes compilations warnings ...

Linus Torvalds committed 6 years ago

Commit cb8e59c2 parents 2e63f6c 065fcfd2097 files · +151766 −46213Browse files

Showing the changes to kernel (45 of 2097 files). Show all

Unified · Split

45 files
  1. kernel/bpf/Makefile +3 −5
  2. kernel/bpf/arraymap.c +5 −5
  3. kernel/bpf/bpf_iter.c +539 −0
  4. kernel/bpf/bpf_lsm.c +1 −1
  5. kernel/bpf/bpf_struct_ops.c +1 −1
  6. kernel/bpf/btf.c +48 −1
  7. kernel/bpf/cgroup.c +90 −56
  8. kernel/bpf/core.c +8 −2
  9. kernel/bpf/cpumap.c +5 −20
  10. kernel/bpf/devmap.c +113 −20
  11. kernel/bpf/hashtab.c +2 −2
  12. kernel/bpf/helpers.c +124 −1
  13. kernel/bpf/inode.c +4 −1
  14. kernel/bpf/lpm_trie.c +1 −1
  15. kernel/bpf/map_in_map.c +1 −1
  16. kernel/bpf/map_iter.c +102 −0
  17. kernel/bpf/net_namespace.c +373 −0
  18. kernel/bpf/queue_stack_maps.c +2 −2
  19. kernel/bpf/reuseport_array.c +1 −1
  20. kernel/bpf/ringbuf.c +501 −0
  21. kernel/bpf/stackmap.c +1 −1
  22. kernel/bpf/syscall.c +503 −94
  23. kernel/bpf/task_iter.c +353 −0
  24. kernel/bpf/verifier.c +299 −88
  25. kernel/bpf/xskmap.c +0 −265
  26. kernel/cgroup/cgroup.c +0 −27
  27. kernel/events/callchain.c +1 −1
  28. kernel/events/core.c +2 −4
  29. kernel/kprobes.c +1 −1
  30. kernel/latencytop.c +2 −2
  31. kernel/module.c +3 −0
  32. kernel/pid_namespace.c +1 −1
  33. kernel/printk/printk.c +1 −1
  34. kernel/sched/core.c +4 −5
  35. kernel/sched/fair.c +1 −2
  36. kernel/sched/rt.c +4 −6
  37. kernel/sched/topology.c +1 −1
  38. kernel/seccomp.c +1 −1
  39. kernel/sysctl.c +1891 −1990
  40. kernel/time/timer.c +1 −2
  41. kernel/trace/bpf_trace.c +235 −6
  42. kernel/trace/trace.c +1 −1
  43. kernel/umh.c +1 −1
  44. kernel/utsname_sysctl.c +1 −1
  45. kernel/watchdog.c +6 −6
kernel/bpf/Makefile+3−5
@@ -2,9 +2,9 @@ · 1 unmodified line
22 obj-y := core.o
33 CFLAGS_core.o += $(call cc-disable-warning, override-init)
44
5−obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o
5+obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o map_iter.o task_iter.o
66 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o
7−obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o
7+obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
88 obj-$(CONFIG_BPF_SYSCALL) += disasm.o
99 obj-$(CONFIG_BPF_JIT) += trampoline.o
1010 obj-$(CONFIG_BPF_SYSCALL) += btf.o
@@ -12,10 +12,8 @@ obj-$(CONFIG_BPF_JIT) += dispatcher.o · 1 unmodified line
1212 ifeq ($(CONFIG_NET),y)
1313 obj-$(CONFIG_BPF_SYSCALL) += devmap.o
1414 obj-$(CONFIG_BPF_SYSCALL) += cpumap.o
15−ifeq ($(CONFIG_XDP_SOCKETS),y)
16−obj-$(CONFIG_BPF_SYSCALL) += xskmap.o
17−endif
1815 obj-$(CONFIG_BPF_SYSCALL) += offload.o
16+obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o
1917 endif
2018 ifeq ($(CONFIG_PERF_EVENTS),y)
2119 obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
kernel/bpf/arraymap.c+5−5
@@ -77,7 +77,7 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) · 76 unmodified lines
7777 bool percpu = attr->map_type == BPF_MAP_TYPE_PERCPU_ARRAY;
7878 int ret, numa_node = bpf_map_attr_numa_node(attr);
7979 u32 elem_size, index_mask, max_entries;
80− bool unpriv = !capable(CAP_SYS_ADMIN);
80+ bool bypass_spec_v1 = bpf_bypass_spec_v1();
8181 u64 cost, array_size, mask64;
8282 struct bpf_map_memory mem;
8383 struct bpf_array *array;
@@ -95,7 +95,7 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) · 11 unmodified lines
9595 mask64 -= 1;
9696
9797 index_mask = mask64;
98− if (unpriv) {
98+ if (!bypass_spec_v1) {
9999 /* round up array size to nearest power of 2,
100100 * since cpu will speculate within index_mask limits
101101 */
@@ -149,7 +149,7 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) · 47 unmodified lines
149149 return ERR_PTR(-ENOMEM);
150150 }
151151 array->index_mask = index_mask;
152− array->map.unpriv_array = unpriv;
152+ array->map.bypass_spec_v1 = bypass_spec_v1;
153153
154154 /* copy mandatory map attributes */
155155 bpf_map_init_from_attr(&array->map, attr);
@@ -219,7 +219,7 @@ static u32 array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) · 63 unmodified lines
219219
220220 *insn++ = BPF_ALU64_IMM(BPF_ADD, map_ptr, offsetof(struct bpf_array, value));
221221 *insn++ = BPF_LDX_MEM(BPF_W, ret, index, 0);
222− if (map->unpriv_array) {
222+ if (!map->bypass_spec_v1) {
223223 *insn++ = BPF_JMP_IMM(BPF_JGE, ret, map->max_entries, 4);
224224 *insn++ = BPF_ALU32_IMM(BPF_AND, ret, array->index_mask);
225225 } else {
@@ -1058,7 +1058,7 @@ static u32 array_of_map_gen_lookup(struct bpf_map *map, · 832 unmodified lines
10581058
10591059 *insn++ = BPF_ALU64_IMM(BPF_ADD, map_ptr, offsetof(struct bpf_array, value));
10601060 *insn++ = BPF_LDX_MEM(BPF_W, ret, index, 0);
1061− if (map->unpriv_array) {
1061+ if (!map->bypass_spec_v1) {
10621062 *insn++ = BPF_JMP_IMM(BPF_JGE, ret, map->max_entries, 6);
10631063 *insn++ = BPF_ALU32_IMM(BPF_AND, ret, array->index_mask);
10641064 } else {
kernel/bpf/bpf_iter.c+539−0
@@ -0,0 +1,539 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/* Copyright (c) 2020 Facebook */
3+
4+#include <linux/fs.h>
5+#include <linux/anon_inodes.h>
6+#include <linux/filter.h>
7+#include <linux/bpf.h>
8+
9+struct bpf_iter_target_info {
10+ struct list_head list;
11+ const struct bpf_iter_reg *reg_info;
12+ u32 btf_id; /* cached value */
13+};
14+
15+struct bpf_iter_link {
16+ struct bpf_link link;
17+ struct bpf_iter_target_info *tinfo;
18+};
19+
20+struct bpf_iter_priv_data {
21+ struct bpf_iter_target_info *tinfo;
22+ struct bpf_prog *prog;
23+ u64 session_id;
24+ u64 seq_num;
25+ bool done_stop;
26+ u8 target_private[] __aligned(8);
27+};
28+
29+static struct list_head targets = LIST_HEAD_INIT(targets);
30+static DEFINE_MUTEX(targets_mutex);
31+
32+/* protect bpf_iter_link changes */
33+static DEFINE_MUTEX(link_mutex);
34+
35+/* incremented on every opened seq_file */
36+static atomic64_t session_id;
37+
38+static int prepare_seq_file(struct file *file, struct bpf_iter_link *link);
39+
40+static void bpf_iter_inc_seq_num(struct seq_file *seq)
41+{
42+ struct bpf_iter_priv_data *iter_priv;
43+
44+ iter_priv = container_of(seq->private, struct bpf_iter_priv_data,
45+ target_private);
46+ iter_priv->seq_num++;
47+}
48+
49+static void bpf_iter_dec_seq_num(struct seq_file *seq)
50+{
51+ struct bpf_iter_priv_data *iter_priv;
52+
53+ iter_priv = container_of(seq->private, struct bpf_iter_priv_data,
54+ target_private);
55+ iter_priv->seq_num--;
56+}
57+
58+static void bpf_iter_done_stop(struct seq_file *seq)
59+{
60+ struct bpf_iter_priv_data *iter_priv;
61+
62+ iter_priv = container_of(seq->private, struct bpf_iter_priv_data,
63+ target_private);
64+ iter_priv->done_stop = true;
65+}
66+
67+/* bpf_seq_read, a customized and simpler version for bpf iterator.
68+ * no_llseek is assumed for this file.
69+ * The following are differences from seq_read():
70+ * . fixed buffer size (PAGE_SIZE)
71+ * . assuming no_llseek
72+ * . stop() may call bpf program, handling potential overflow there
73+ */
74+static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size,
75+ loff_t *ppos)
76+{
77+ struct seq_file *seq = file->private_data;
78+ size_t n, offs, copied = 0;
79+ int err = 0;
80+ void *p;
81+
82+ mutex_lock(&seq->lock);
83+
84+ if (!seq->buf) {
85+ seq->size = PAGE_SIZE;
86+ seq->buf = kmalloc(seq->size, GFP_KERNEL);
87+ if (!seq->buf) {
88+ err = -ENOMEM;
89+ goto done;
90+ }
91+ }
92+
93+ if (seq->count) {
94+ n = min(seq->count, size);
95+ err = copy_to_user(buf, seq->buf + seq->from, n);
96+ if (err) {
97+ err = -EFAULT;
98+ goto done;
99+ }
100+ seq->count -= n;
101+ seq->from += n;
102+ copied = n;
103+ goto done;
104+ }
105+
106+ seq->from = 0;
107+ p = seq->op->start(seq, &seq->index);
108+ if (!p)
109+ goto stop;
110+ if (IS_ERR(p)) {
111+ err = PTR_ERR(p);
112+ seq->op->stop(seq, p);
113+ seq->count = 0;
114+ goto done;
115+ }
116+
117+ err = seq->op->show(seq, p);
118+ if (err > 0) {
119+ /* object is skipped, decrease seq_num, so next
120+ * valid object can reuse the same seq_num.
121+ */
122+ bpf_iter_dec_seq_num(seq);
123+ seq->count = 0;
124+ } else if (err < 0 || seq_has_overflowed(seq)) {
125+ if (!err)
126+ err = -E2BIG;
127+ seq->op->stop(seq, p);
128+ seq->count = 0;
129+ goto done;
130+ }
131+
132+ while (1) {
133+ loff_t pos = seq->index;
134+
135+ offs = seq->count;
136+ p = seq->op->next(seq, p, &seq->index);
137+ if (pos == seq->index) {
138+ pr_info_ratelimited("buggy seq_file .next function %ps "
139+ "did not updated position index\n",
140+ seq->op->next);
141+ seq->index++;
142+ }
143+
144+ if (IS_ERR_OR_NULL(p))
145+ break;
146+
147+ /* got a valid next object, increase seq_num */
148+ bpf_iter_inc_seq_num(seq);
149+
150+ if (seq->count >= size)
151+ break;
152+
153+ err = seq->op->show(seq, p);
154+ if (err > 0) {
155+ bpf_iter_dec_seq_num(seq);
156+ seq->count = offs;
157+ } else if (err < 0 || seq_has_overflowed(seq)) {
158+ seq->count = offs;
159+ if (offs == 0) {
160+ if (!err)
161+ err = -E2BIG;
162+ seq->op->stop(seq, p);
163+ goto done;
164+ }
165+ break;
166+ }
167+ }
168+stop:
169+ offs = seq->count;
170+ /* bpf program called if !p */
171+ seq->op->stop(seq, p);
172+ if (!p) {
173+ if (!seq_has_overflowed(seq)) {
174+ bpf_iter_done_stop(seq);
175+ } else {
176+ seq->count = offs;
177+ if (offs == 0) {
178+ err = -E2BIG;
179+ goto done;
180+ }
181+ }
182+ }
183+
184+ n = min(seq->count, size);
185+ err = copy_to_user(buf, seq->buf, n);
186+ if (err) {
187+ err = -EFAULT;
188+ goto done;
189+ }
190+ copied = n;
191+ seq->count -= n;
192+ seq->from = n;
193+done:
194+ if (!copied)
195+ copied = err;
196+ else
197+ *ppos += copied;
198+ mutex_unlock(&seq->lock);
199+ return copied;
200+}
201+
202+static int iter_open(struct inode *inode, struct file *file)
203+{
204+ struct bpf_iter_link *link = inode->i_private;
205+
206+ return prepare_seq_file(file, link);
207+}
208+
209+static int iter_release(struct inode *inode, struct file *file)
210+{
211+ struct bpf_iter_priv_data *iter_priv;
212+ struct seq_file *seq;
213+
214+ seq = file->private_data;
215+ if (!seq)
216+ return 0;
217+
218+ iter_priv = container_of(seq->private, struct bpf_iter_priv_data,
219+ target_private);
220+
221+ if (iter_priv->tinfo->reg_info->fini_seq_private)
222+ iter_priv->tinfo->reg_info->fini_seq_private(seq->private);
223+
224+ bpf_prog_put(iter_priv->prog);
225+ seq->private = iter_priv;
226+
227+ return seq_release_private(inode, file);
228+}
229+
230+const struct file_operations bpf_iter_fops = {
231+ .open = iter_open,
232+ .llseek = no_llseek,
233+ .read = bpf_seq_read,
234+ .release = iter_release,
235+};
236+
237+/* The argument reg_info will be cached in bpf_iter_target_info.
238+ * The common practice is to declare target reg_info as
239+ * a const static variable and passed as an argument to
240+ * bpf_iter_reg_target().
241+ */
242+int bpf_iter_reg_target(const struct bpf_iter_reg *reg_info)
243+{
244+ struct bpf_iter_target_info *tinfo;
245+
246+ tinfo = kmalloc(sizeof(*tinfo), GFP_KERNEL);
247+ if (!tinfo)
248+ return -ENOMEM;
249+
250+ tinfo->reg_info = reg_info;
251+ INIT_LIST_HEAD(&tinfo->list);
252+
253+ mutex_lock(&targets_mutex);
254+ list_add(&tinfo->list, &targets);
255+ mutex_unlock(&targets_mutex);
256+
257+ return 0;
258+}
259+
260+void bpf_iter_unreg_target(const struct bpf_iter_reg *reg_info)
261+{
262+ struct bpf_iter_target_info *tinfo;
263+ bool found = false;
264+
265+ mutex_lock(&targets_mutex);
266+ list_for_each_entry(tinfo, &targets, list) {
267+ if (reg_info == tinfo->reg_info) {
268+ list_del(&tinfo->list);
269+ kfree(tinfo);
270+ found = true;
271+ break;
272+ }
273+ }
274+ mutex_unlock(&targets_mutex);
275+
276+ WARN_ON(found == false);
277+}
278+
279+static void cache_btf_id(struct bpf_iter_target_info *tinfo,
280+ struct bpf_prog *prog)
281+{
282+ tinfo->btf_id = prog->aux->attach_btf_id;
283+}
284+
285+bool bpf_iter_prog_supported(struct bpf_prog *prog)
286+{
287+ const char *attach_fname = prog->aux->attach_func_name;
288+ u32 prog_btf_id = prog->aux->attach_btf_id;
289+ const char *prefix = BPF_ITER_FUNC_PREFIX;
290+ struct bpf_iter_target_info *tinfo;
291+ int prefix_len = strlen(prefix);
292+ bool supported = false;
293+
294+ if (strncmp(attach_fname, prefix, prefix_len))
295+ return false;
296+
297+ mutex_lock(&targets_mutex);
298+ list_for_each_entry(tinfo, &targets, list) {
299+ if (tinfo->btf_id && tinfo->btf_id == prog_btf_id) {
300+ supported = true;
301+ break;
302+ }
303+ if (!strcmp(attach_fname + prefix_len, tinfo->reg_info->target)) {
304+ cache_btf_id(tinfo, prog);
305+ supported = true;
306+ break;
307+ }
308+ }
309+ mutex_unlock(&targets_mutex);
310+
311+ if (supported) {
312+ prog->aux->ctx_arg_info_size = tinfo->reg_info->ctx_arg_info_size;
313+ prog->aux->ctx_arg_info = tinfo->reg_info->ctx_arg_info;
314+ }
315+
316+ return supported;
317+}
318+
319+static void bpf_iter_link_release(struct bpf_link *link)
320+{
321+}
322+
323+static void bpf_iter_link_dealloc(struct bpf_link *link)
324+{
325+ struct bpf_iter_link *iter_link =
326+ container_of(link, struct bpf_iter_link, link);
327+
328+ kfree(iter_link);
329+}
330+
331+static int bpf_iter_link_replace(struct bpf_link *link,
332+ struct bpf_prog *new_prog,
333+ struct bpf_prog *old_prog)
334+{
335+ int ret = 0;
336+
337+ mutex_lock(&link_mutex);
338+ if (old_prog && link->prog != old_prog) {
339+ ret = -EPERM;
340+ goto out_unlock;
341+ }
342+
343+ if (link->prog->type != new_prog->type ||
344+ link->prog->expected_attach_type != new_prog->expected_attach_type ||
345+ link->prog->aux->attach_btf_id != new_prog->aux->attach_btf_id) {
346+ ret = -EINVAL;
347+ goto out_unlock;
348+ }
349+
350+ old_prog = xchg(&link->prog, new_prog);
351+ bpf_prog_put(old_prog);
352+
353+out_unlock:
354+ mutex_unlock(&link_mutex);
355+ return ret;
356+}
357+
358+static const struct bpf_link_ops bpf_iter_link_lops = {
359+ .release = bpf_iter_link_release,
360+ .dealloc = bpf_iter_link_dealloc,
361+ .update_prog = bpf_iter_link_replace,
362+};
363+
364+bool bpf_link_is_iter(struct bpf_link *link)
365+{
366+ return link->ops == &bpf_iter_link_lops;
367+}
368+
369+int bpf_iter_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
370+{
371+ struct bpf_link_primer link_primer;
372+ struct bpf_iter_target_info *tinfo;
373+ struct bpf_iter_link *link;
374+ bool existed = false;
375+ u32 prog_btf_id;
376+ int err;
377+
378+ if (attr->link_create.target_fd || attr->link_create.flags)
379+ return -EINVAL;
380+
381+ prog_btf_id = prog->aux->attach_btf_id;
382+ mutex_lock(&targets_mutex);
383+ list_for_each_entry(tinfo, &targets, list) {
384+ if (tinfo->btf_id == prog_btf_id) {
385+ existed = true;
386+ break;
387+ }
388+ }
389+ mutex_unlock(&targets_mutex);
390+ if (!existed)
391+ return -ENOENT;
392+
393+ link = kzalloc(sizeof(*link), GFP_USER | __GFP_NOWARN);
394+ if (!link)
395+ return -ENOMEM;
396+
397+ bpf_link_init(&link->link, BPF_LINK_TYPE_ITER, &bpf_iter_link_lops, prog);
398+ link->tinfo = tinfo;
399+
400+ err = bpf_link_prime(&link->link, &link_primer);
401+ if (err) {
402+ kfree(link);
403+ return err;
404+ }
405+
406+ return bpf_link_settle(&link_primer);
407+}
408+
409+static void init_seq_meta(struct bpf_iter_priv_data *priv_data,
410+ struct bpf_iter_target_info *tinfo,
411+ struct bpf_prog *prog)
412+{
413+ priv_data->tinfo = tinfo;
414+ priv_data->prog = prog;
415+ priv_data->session_id = atomic64_inc_return(&session_id);
416+ priv_data->seq_num = 0;
417+ priv_data->done_stop = false;
418+}
419+
420+static int prepare_seq_file(struct file *file, struct bpf_iter_link *link)
421+{
422+ struct bpf_iter_priv_data *priv_data;
423+ struct bpf_iter_target_info *tinfo;
424+ struct bpf_prog *prog;
425+ u32 total_priv_dsize;
426+ struct seq_file *seq;
427+ int err = 0;
428+
429+ mutex_lock(&link_mutex);
430+ prog = link->link.prog;
431+ bpf_prog_inc(prog);
432+ mutex_unlock(&link_mutex);
433+
434+ tinfo = link->tinfo;
435+ total_priv_dsize = offsetof(struct bpf_iter_priv_data, target_private) +
436+ tinfo->reg_info->seq_priv_size;
437+ priv_data = __seq_open_private(file, tinfo->reg_info->seq_ops,
438+ total_priv_dsize);
439+ if (!priv_data) {
440+ err = -ENOMEM;
441+ goto release_prog;
442+ }
443+
444+ if (tinfo->reg_info->init_seq_private) {
445+ err = tinfo->reg_info->init_seq_private(priv_data->target_private);
446+ if (err)
447+ goto release_seq_file;
448+ }
449+
450+ init_seq_meta(priv_data, tinfo, prog);
451+ seq = file->private_data;
452+ seq->private = priv_data->target_private;
453+
454+ return 0;
455+
456+release_seq_file:
457+ seq_release_private(file->f_inode, file);
458+ file->private_data = NULL;
459+release_prog:
460+ bpf_prog_put(prog);
461+ return err;
462+}
463+
464+int bpf_iter_new_fd(struct bpf_link *link)
465+{
466+ struct file *file;
467+ unsigned int flags;
468+ int err, fd;
469+
470+ if (link->ops != &bpf_iter_link_lops)
471+ return -EINVAL;
472+
473+ flags = O_RDONLY | O_CLOEXEC;
474+ fd = get_unused_fd_flags(flags);
475+ if (fd < 0)
476+ return fd;
477+
478+ file = anon_inode_getfile("bpf_iter", &bpf_iter_fops, NULL, flags);
479+ if (IS_ERR(file)) {
480+ err = PTR_ERR(file);
481+ goto free_fd;
482+ }
483+
484+ err = prepare_seq_file(file,
485+ container_of(link, struct bpf_iter_link, link));
486+ if (err)
487+ goto free_file;
488+
489+ fd_install(fd, file);
490+ return fd;
491+
492+free_file:
493+ fput(file);
494+free_fd:
495+ put_unused_fd(fd);
496+ return err;
497+}
498+
499+struct bpf_prog *bpf_iter_get_info(struct bpf_iter_meta *meta, bool in_stop)
500+{
501+ struct bpf_iter_priv_data *iter_priv;
502+ struct seq_file *seq;
503+ void *seq_priv;
504+
505+ seq = meta->seq;
506+ if (seq->file->f_op != &bpf_iter_fops)
507+ return NULL;
508+
509+ seq_priv = seq->private;
510+ iter_priv = container_of(seq_priv, struct bpf_iter_priv_data,
511+ target_private);
512+
513+ if (in_stop && iter_priv->done_stop)
514+ return NULL;
515+
516+ meta->session_id = iter_priv->session_id;
517+ meta->seq_num = iter_priv->seq_num;
518+
519+ return iter_priv->prog;
520+}
521+
522+int bpf_iter_run_prog(struct bpf_prog *prog, void *ctx)
523+{
524+ int ret;
525+
526+ rcu_read_lock();
527+ migrate_disable();
528+ ret = BPF_PROG_RUN(prog, ctx);
529+ migrate_enable();
530+ rcu_read_unlock();
531+
532+ /* bpf program can only return 0 or 1:
533+ * 0 : okay
534+ * 1 : retry the same object
535+ * The bpf_iter_run_prog() return value
536+ * will be seq_ops->show() return value.
537+ */
538+ return ret == 0 ? 0 : -EAGAIN;
539+}
kernel/bpf/bpf_lsm.c+1−1
@@ -49,6 +49,6 @@ const struct bpf_prog_ops lsm_prog_ops = { · 48 unmodified lines
4949 };
5050
5151 const struct bpf_verifier_ops lsm_verifier_ops = {
52− .get_func_proto = bpf_tracing_func_proto,
52+ .get_func_proto = tracing_prog_func_proto,
5353 .is_valid_access = btf_ctx_access,
5454 };
kernel/bpf/bpf_struct_ops.c+1−1
@@ -557,7 +557,7 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) · 556 unmodified lines
557557 struct bpf_map *map;
558558 int err;
559559
560− if (!capable(CAP_SYS_ADMIN))
560+ if (!bpf_capable())
561561 return ERR_PTR(-EPERM);
562562
563563 st_ops = bpf_struct_ops_find_value(attr->btf_vmlinux_value_type_id);
kernel/bpf/btf.c+48−1
@@ -3482,6 +3482,7 @@ extern char __weak __stop_BTF[]; · 3481 unmodified lines
34823482 extern struct btf *btf_vmlinux;
34833483
34843484 #define BPF_MAP_TYPE(_id, _ops)
3485+#define BPF_LINK_TYPE(_id, _name)
34853486 static union {
34863487 struct bpf_ctx_convert {
34873488 #define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
@@ -3508,6 +3509,7 @@ static u8 bpf_ctx_convert_map[] = { · 20 unmodified lines
35083509 0, /* avoid empty array */
35093510 };
35103511 #undef BPF_MAP_TYPE
3512+#undef BPF_LINK_TYPE
35113513
35123514 static const struct btf_member *
35133515 btf_get_prog_ctx_type(struct bpf_verifier_log *log, struct btf *btf,
@@ -3692,7 +3694,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 178 unmodified lines
36923694 struct bpf_verifier_log *log = info->log;
36933695 const struct btf_param *args;
36943696 u32 nr_args, arg;
3695− int ret;
3697+ int i, ret;
36963698
36973699 if (off % 8) {
36983700 bpf_log(log, "func '%s' offset %d is not multiple of 8\n",
@@ -3789,6 +3791,14 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 90 unmodified lines
37893791
37903792 /* this is a pointer to another type */
37913793 info->reg_type = PTR_TO_BTF_ID;
3794+ for (i = 0; i < prog->aux->ctx_arg_info_size; i++) {
3795+ const struct bpf_ctx_arg_aux *ctx_arg_info = &prog->aux->ctx_arg_info[i];
3796+
3797+ if (ctx_arg_info->offset == off) {
3798+ info->reg_type = ctx_arg_info->reg_type;
3799+ break;
3800+ }
3801+ }
37923802
37933803 if (tgt_prog) {
37943804 ret = btf_translate_to_vmlinux(log, btf, t, tgt_prog->type, arg);
@@ -3828,6 +3838,7 @@ int btf_struct_access(struct bpf_verifier_log *log, · 33 unmodified lines
38283838 const struct btf_type *mtype, *elem_type = NULL;
38293839 const struct btf_member *member;
38303840 const char *tname, *mname;
3841+ u32 vlen;
38313842
38323843 again:
38333844 tname = __btf_name_by_offset(btf_vmlinux, t->name_off);
@@ -3836,7 +3847,43 @@ again: · 2 unmodified lines
38363847 return -EINVAL;
38373848 }
38383849
3850+ vlen = btf_type_vlen(t);
38393851 if (off + size > t->size) {
3852+ /* If the last element is a variable size array, we may
3853+ * need to relax the rule.
3854+ */
3855+ struct btf_array *array_elem;
3856+
3857+ if (vlen == 0)
3858+ goto error;
3859+
3860+ member = btf_type_member(t) + vlen - 1;
3861+ mtype = btf_type_skip_modifiers(btf_vmlinux, member->type,
3862+ NULL);
3863+ if (!btf_type_is_array(mtype))
3864+ goto error;
3865+
3866+ array_elem = (struct btf_array *)(mtype + 1);
3867+ if (array_elem->nelems != 0)
3868+ goto error;
3869+
3870+ moff = btf_member_bit_offset(t, member) / 8;
3871+ if (off < moff)
3872+ goto error;
3873+
3874+ /* Only allow structure for now, can be relaxed for
3875+ * other types later.
3876+ */
3877+ elem_type = btf_type_skip_modifiers(btf_vmlinux,
3878+ array_elem->type, NULL);
3879+ if (!btf_type_is_struct(elem_type))
3880+ goto error;
3881+
3882+ off = (off - moff) % elem_type->size;
3883+ return btf_struct_access(log, elem_type, off, size, atype,
3884+ next_btf_id);
3885+
3886+error:
38403887 bpf_log(log, "access beyond struct %s at off %u size %u\n",
38413888 tname, off, size);
38423889 return -EACCES;
kernel/bpf/cgroup.c+90−56
@@ -557,8 +557,9 @@ found: · 556 unmodified lines
557557 *
558558 * Must be called with cgroup_mutex held.
559559 */
560−int __cgroup_bpf_replace(struct cgroup *cgrp, struct bpf_cgroup_link *link,
561− struct bpf_prog *new_prog)
560+static int __cgroup_bpf_replace(struct cgroup *cgrp,
561+ struct bpf_cgroup_link *link,
562+ struct bpf_prog *new_prog)
562563 {
563564 struct list_head *progs = &cgrp->bpf.progs[link->type];
564565 struct bpf_prog *old_prog;
@@ -583,6 +584,30 @@ int __cgroup_bpf_replace(struct cgroup *cgrp, struct bpf_cgroup_link *link, · 18 unmodified lines
583584 return 0;
584585 }
585586
587+static int cgroup_bpf_replace(struct bpf_link *link, struct bpf_prog *new_prog,
588+ struct bpf_prog *old_prog)
589+{
590+ struct bpf_cgroup_link *cg_link;
591+ int ret;
592+
593+ cg_link = container_of(link, struct bpf_cgroup_link, link);
594+
595+ mutex_lock(&cgroup_mutex);
596+ /* link might have been auto-released by dying cgroup, so fail */
597+ if (!cg_link->cgroup) {
598+ ret = -ENOLINK;
599+ goto out_unlock;
600+ }
601+ if (old_prog && link->prog != old_prog) {
602+ ret = -EPERM;
603+ goto out_unlock;
604+ }
605+ ret = __cgroup_bpf_replace(cg_link->cgroup, cg_link, new_prog);
606+out_unlock:
607+ mutex_unlock(&cgroup_mutex);
608+ return ret;
609+}
610+
586611 static struct bpf_prog_list *find_detach_entry(struct list_head *progs,
587612 struct bpf_prog *prog,
588613 struct bpf_cgroup_link *link,
@@ -808,17 +833,56 @@ static void bpf_cgroup_link_dealloc(struct bpf_link *link) · 219 unmodified lines
808833 kfree(cg_link);
809834 }
810835
811−const struct bpf_link_ops bpf_cgroup_link_lops = {
836+static void bpf_cgroup_link_show_fdinfo(const struct bpf_link *link,
837+ struct seq_file *seq)
838+{
839+ struct bpf_cgroup_link *cg_link =
840+ container_of(link, struct bpf_cgroup_link, link);
841+ u64 cg_id = 0;
842+
843+ mutex_lock(&cgroup_mutex);
844+ if (cg_link->cgroup)
845+ cg_id = cgroup_id(cg_link->cgroup);
846+ mutex_unlock(&cgroup_mutex);
847+
848+ seq_printf(seq,
849+ "cgroup_id:\t%llu\n"
850+ "attach_type:\t%d\n",
851+ cg_id,
852+ cg_link->type);
853+}
854+
855+static int bpf_cgroup_link_fill_link_info(const struct bpf_link *link,
856+ struct bpf_link_info *info)
857+{
858+ struct bpf_cgroup_link *cg_link =
859+ container_of(link, struct bpf_cgroup_link, link);
860+ u64 cg_id = 0;
861+
862+ mutex_lock(&cgroup_mutex);
863+ if (cg_link->cgroup)
864+ cg_id = cgroup_id(cg_link->cgroup);
865+ mutex_unlock(&cgroup_mutex);
866+
867+ info->cgroup.cgroup_id = cg_id;
868+ info->cgroup.attach_type = cg_link->type;
869+ return 0;
870+}
871+
872+static const struct bpf_link_ops bpf_cgroup_link_lops = {
812873 .release = bpf_cgroup_link_release,
813874 .dealloc = bpf_cgroup_link_dealloc,
875+ .update_prog = cgroup_bpf_replace,
876+ .show_fdinfo = bpf_cgroup_link_show_fdinfo,
877+ .fill_link_info = bpf_cgroup_link_fill_link_info,
814878 };
815879
816880 int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
817881 {
882+ struct bpf_link_primer link_primer;
818883 struct bpf_cgroup_link *link;
819− struct file *link_file;
820884 struct cgroup *cgrp;
821− int err, link_fd;
885+ int err;
822886
823887 if (attr->link_create.flags)
824888 return -EINVAL;
@@ -832,26 +896,25 @@ int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) · 7 unmodified lines
832896 err = -ENOMEM;
833897 goto out_put_cgroup;
834898 }
835− bpf_link_init(&link->link, &bpf_cgroup_link_lops, prog);
899+ bpf_link_init(&link->link, BPF_LINK_TYPE_CGROUP, &bpf_cgroup_link_lops,
900+ prog);
836901 link->cgroup = cgrp;
837902 link->type = attr->link_create.attach_type;
838903
839− link_file = bpf_link_new_file(&link->link, &link_fd);
840− if (IS_ERR(link_file)) {
904+ err = bpf_link_prime(&link->link, &link_primer);
905+ if (err) {
841906 kfree(link);
842− err = PTR_ERR(link_file);
843907 goto out_put_cgroup;
844908 }
845909
846910 err = cgroup_bpf_attach(cgrp, NULL, NULL, link, link->type,
847911 BPF_F_ALLOW_MULTI);
848912 if (err) {
849− bpf_link_cleanup(&link->link, link_file, link_fd);
913+ bpf_link_cleanup(&link_primer);
850914 goto out_put_cgroup;
851915 }
852916
853− fd_install(link_fd, link_file);
854− return link_fd;
917+ return bpf_link_settle(&link_primer);
855918
856919 out_put_cgroup:
857920 cgroup_put(cgrp);
@@ -1054,36 +1117,21 @@ int __cgroup_bpf_check_dev_permission(short dev_type, u32 major, u32 minor, · 196 unmodified lines
10541117
10551118 return !allow;
10561119 }
1057−EXPORT_SYMBOL(__cgroup_bpf_check_dev_permission);
10581120
10591121 static const struct bpf_func_proto *
10601122 cgroup_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
10611123 {
10621124 switch (func_id) {
1063− case BPF_FUNC_map_lookup_elem:
1064− return &bpf_map_lookup_elem_proto;
1065− case BPF_FUNC_map_update_elem:
1066− return &bpf_map_update_elem_proto;
1067− case BPF_FUNC_map_delete_elem:
1068− return &bpf_map_delete_elem_proto;
1069− case BPF_FUNC_map_push_elem:
1070− return &bpf_map_push_elem_proto;
1071− case BPF_FUNC_map_pop_elem:
1072− return &bpf_map_pop_elem_proto;
1073− case BPF_FUNC_map_peek_elem:
1074− return &bpf_map_peek_elem_proto;
10751125 case BPF_FUNC_get_current_uid_gid:
10761126 return &bpf_get_current_uid_gid_proto;
10771127 case BPF_FUNC_get_local_storage:
10781128 return &bpf_get_local_storage_proto;
10791129 case BPF_FUNC_get_current_cgroup_id:
10801130 return &bpf_get_current_cgroup_id_proto;
1081− case BPF_FUNC_trace_printk:
1082− if (capable(CAP_SYS_ADMIN))
1083− return bpf_get_trace_printk_proto();
1084− /* fall through */
1131+ case BPF_FUNC_perf_event_output:
1132+ return &bpf_event_output_data_proto;
10851133 default:
1086− return NULL;
1134+ return bpf_base_func_proto(func_id);
10871135 }
10881136 }
10891137
@@ -1137,16 +1185,13 @@ const struct bpf_verifier_ops cg_dev_verifier_ops = { · 47 unmodified lines
11371185 * @head: sysctl table header
11381186 * @table: sysctl table
11391187 * @write: sysctl is being read (= 0) or written (= 1)
1140− * @buf: pointer to buffer passed by user space
1188+ * @buf: pointer to buffer (in and out)
11411189 * @pcount: value-result argument: value is size of buffer pointed to by @buf,
11421190 * result is size of @new_buf if program set new value, initial value
11431191 * otherwise
11441192 * @ppos: value-result argument: value is position at which read from or write
11451193 * to sysctl is happening, result is new position if program overrode it,
11461194 * initial value otherwise
1147− * @new_buf: pointer to pointer to new buffer that will be allocated if program
1148− * overrides new value provided by user space on sysctl write
1149− * NOTE: it's caller responsibility to free *new_buf if it was set
11501195 * @type: type of program to be executed
11511196 *
11521197 * Program is run when sysctl is being accessed, either read or written, and
@@ -1157,8 +1202,7 @@ const struct bpf_verifier_ops cg_dev_verifier_ops = { · 4 unmodified lines
11571202 */
11581203 int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head,
11591204 struct ctl_table *table, int write,
1160− void __user *buf, size_t *pcount,
1161− loff_t *ppos, void **new_buf,
1205+ void **buf, size_t *pcount, loff_t *ppos,
11621206 enum bpf_attach_type type)
11631207 {
11641208 struct bpf_sysctl_kern ctx = {
@@ -1173,36 +1217,28 @@ int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head, · 8 unmodified lines
11731217 .new_updated = 0,
11741218 };
11751219 struct cgroup *cgrp;
1220+ loff_t pos = 0;
11761221 int ret;
11771222
11781223 ctx.cur_val = kmalloc_track_caller(ctx.cur_len, GFP_KERNEL);
1179− if (ctx.cur_val) {
1180− mm_segment_t old_fs;
1181− loff_t pos = 0;
1182−
1183− old_fs = get_fs();
1184− set_fs(KERNEL_DS);
1185− if (table->proc_handler(table, 0, (void __user *)ctx.cur_val,
1186− &ctx.cur_len, &pos)) {
1187− /* Let BPF program decide how to proceed. */
1188− ctx.cur_len = 0;
1189− }
1190− set_fs(old_fs);
1191− } else {
1224+ if (!ctx.cur_val ||
1225+ table->proc_handler(table, 0, ctx.cur_val, &ctx.cur_len, &pos)) {
11921226 /* Let BPF program decide how to proceed. */
11931227 ctx.cur_len = 0;
11941228 }
11951229
1196− if (write && buf && *pcount) {
1230+ if (write && *buf && *pcount) {
11971231 /* BPF program should be able to override new value with a
11981232 * buffer bigger than provided by user.
11991233 */
12001234 ctx.new_val = kmalloc_track_caller(PAGE_SIZE, GFP_KERNEL);
12011235 ctx.new_len = min_t(size_t, PAGE_SIZE, *pcount);
1202− if (!ctx.new_val ||
1203− copy_from_user(ctx.new_val, buf, ctx.new_len))
1236+ if (ctx.new_val) {
1237+ memcpy(ctx.new_val, *buf, ctx.new_len);
1238+ } else {
12041239 /* Let BPF program decide how to proceed. */
12051240 ctx.new_len = 0;
1241+ }
12061242 }
12071243
12081244 rcu_read_lock();
@@ -1213,7 +1249,8 @@ int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head, · 4 unmodified lines
12131249 kfree(ctx.cur_val);
12141250
12151251 if (ret == 1 && ctx.new_updated) {
1216− *new_buf = ctx.new_val;
1252+ kfree(*buf);
1253+ *buf = ctx.new_val;
12171254 *pcount = ctx.new_len;
12181255 } else {
12191256 kfree(ctx.new_val);
@@ -1221,7 +1258,6 @@ int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head, · 1 unmodified line
12211258
12221259 return ret == 1 ? 0 : -EPERM;
12231260 }
1224−EXPORT_SYMBOL(__cgroup_bpf_run_filter_sysctl);
12251261
12261262 #ifdef CONFIG_NET
12271263 static bool __cgroup_bpf_prog_array_is_empty(struct cgroup *cgrp,
@@ -1326,7 +1362,6 @@ out: · 98 unmodified lines
13261362 sockopt_free_buf(&ctx);
13271363 return ret;
13281364 }
1329−EXPORT_SYMBOL(__cgroup_bpf_run_filter_setsockopt);
13301365
13311366 int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level,
13321367 int optname, char __user *optval,
@@ -1413,7 +1448,6 @@ out: · 80 unmodified lines
14131448 sockopt_free_buf(&ctx);
14141449 return ret;
14151450 }
1416−EXPORT_SYMBOL(__cgroup_bpf_run_filter_getsockopt);
14171451 #endif
14181452
14191453 static ssize_t sysctl_cpy_dir(const struct ctl_dir *dir, char **bufp,
kernel/bpf/core.c+8−2
@@ -646,7 +646,7 @@ static bool bpf_prog_kallsyms_verify_off(const struct bpf_prog *fp) · 645 unmodified lines
646646 void bpf_prog_kallsyms_add(struct bpf_prog *fp)
647647 {
648648 if (!bpf_prog_kallsyms_candidate(fp) ||
649− !capable(CAP_SYS_ADMIN))
649+ !bpf_capable())
650650 return;
651651
652652 bpf_prog_ksym_set_addr(fp);
@@ -1543,7 +1543,7 @@ select_insn: · 890 unmodified lines
15431543
15441544 /* ARG1 at this point is guaranteed to point to CTX from
15451545 * the verifier side due to the fact that the tail call is
1546− * handeled like a helper, that is, bpf_tail_call_proto,
1546+ * handled like a helper, that is, bpf_tail_call_proto,
15471547 * where arg1_type is ARG_PTR_TO_CTX.
15481548 */
15491549 insn = prog->insnsi;
@@ -2136,6 +2136,11 @@ BPF_CALL_0(bpf_user_rnd_u32) · 586 unmodified lines
21362136 return res;
21372137 }
21382138
2139+BPF_CALL_0(bpf_get_raw_cpu_id)
2140+{
2141+ return raw_smp_processor_id();
2142+}
2143+
21392144 /* Weak definitions of helper functions in case we don't have bpf syscall. */
21402145 const struct bpf_func_proto bpf_map_lookup_elem_proto __weak;
21412146 const struct bpf_func_proto bpf_map_update_elem_proto __weak;
@@ -2151,6 +2156,7 @@ const struct bpf_func_proto bpf_get_prandom_u32_proto __weak; · 9 unmodified lines
21512156 const struct bpf_func_proto bpf_get_smp_processor_id_proto __weak;
21522157 const struct bpf_func_proto bpf_get_numa_node_id_proto __weak;
21532158 const struct bpf_func_proto bpf_ktime_get_ns_proto __weak;
2159+const struct bpf_func_proto bpf_ktime_get_boot_ns_proto __weak;
21542160
21552161 const struct bpf_func_proto bpf_get_current_pid_tgid_proto __weak;
21562162 const struct bpf_func_proto bpf_get_current_uid_gid_proto __weak;
kernel/bpf/cpumap.c+5−20
@@ -85,7 +85,7 @@ static struct bpf_map *cpu_map_alloc(union bpf_attr *attr) · 84 unmodified lines
8585 u64 cost;
8686 int ret;
8787
88− if (!capable(CAP_SYS_ADMIN))
88+ if (!bpf_capable())
8989 return ERR_PTR(-EPERM);
9090
9191 /* check sanity of attributes */
@@ -162,25 +162,10 @@ static struct sk_buff *cpu_map_build_skb(struct bpf_cpu_map_entry *rcpu, · 70 unmodified lines
162162 /* Part of headroom was reserved to xdpf */
163163 hard_start_headroom = sizeof(struct xdp_frame) + xdpf->headroom;
164164
165− /* build_skb need to place skb_shared_info after SKB end, and
166− * also want to know the memory "truesize". Thus, need to
167− * know the memory frame size backing xdp_buff.
168− *
169− * XDP was designed to have PAGE_SIZE frames, but this
170− * assumption is not longer true with ixgbe and i40e. It
171− * would be preferred to set frame_size to 2048 or 4096
172− * depending on the driver.
173− * frame_size = 2048;
174− * frame_len = frame_size - sizeof(*xdp_frame);
175− *
176− * Instead, with info avail, skb_shared_info in placed after
177− * packet len. This, unfortunately fakes the truesize.
178− * Another disadvantage of this approach, the skb_shared_info
179− * is not at a fixed memory location, with mixed length
180− * packets, which is bad for cache-line hotness.
165+ /* Memory size backing xdp_frame data already have reserved
166+ * room for build_skb to place skb_shared_info in tailroom.
181167 */
182− frame_size = SKB_DATA_ALIGN(xdpf->len + hard_start_headroom) +
183− SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
168+ frame_size = xdpf->frame_sz;
184169
185170 pkt_data_start = xdpf->data - hard_start_headroom;
186171 skb = build_skb_around(skb, pkt_data_start, frame_size);
@@ -636,7 +621,7 @@ int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp, · 449 unmodified lines
636621 {
637622 struct xdp_frame *xdpf;
638623
639− xdpf = convert_to_xdp_frame(xdp);
624+ xdpf = xdp_convert_buff_to_frame(xdp);
640625 if (unlikely(!xdpf))
641626 return -EOVERFLOW;
642627
kernel/bpf/devmap.c+113−20
@@ -52,7 +52,6 @@ · 51 unmodified lines
5252 #define DEV_CREATE_FLAG_MASK \
5353 (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY)
5454
55−#define DEV_MAP_BULK_SIZE 16
5655 struct xdp_dev_bulk_queue {
5756 struct xdp_frame *q[DEV_MAP_BULK_SIZE];
5857 struct list_head flush_node;
@@ -61,12 +60,23 @@ struct xdp_dev_bulk_queue { · 2 unmodified lines
6160 unsigned int count;
6261 };
6362
63+/* DEVMAP values */
64+struct bpf_devmap_val {
65+ u32 ifindex; /* device index */
66+ union {
67+ int fd; /* prog fd on map write */
68+ u32 id; /* prog id on map read */
69+ } bpf_prog;
70+};
71+
6472 struct bpf_dtab_netdev {
6573 struct net_device *dev; /* must be first member, due to tracepoint */
6674 struct hlist_node index_hlist;
6775 struct bpf_dtab *dtab;
76+ struct bpf_prog *xdp_prog;
6877 struct rcu_head rcu;
6978 unsigned int idx;
79+ struct bpf_devmap_val val;
7080 };
7181
7282 struct bpf_dtab {
@@ -106,12 +116,18 @@ static inline struct hlist_head *dev_map_index_hash(struct bpf_dtab *dtab, · 33 unmodified lines
106116
107117 static int dev_map_init_map(struct bpf_dtab *dtab, union bpf_attr *attr)
108118 {
119+ u32 valsize = attr->value_size;
109120 u64 cost = 0;
110121 int err;
111122
112− /* check sanity of attributes */
123+ /* check sanity of attributes. 2 value sizes supported:
124+ * 4 bytes: ifindex
125+ * 8 bytes: ifindex + prog fd
126+ */
113127 if (attr->max_entries == 0 || attr->key_size != 4 ||
114− attr->value_size != 4 || attr->map_flags & ~DEV_CREATE_FLAG_MASK)
128+ (valsize != offsetofend(struct bpf_devmap_val, ifindex) &&
129+ valsize != offsetofend(struct bpf_devmap_val, bpf_prog.fd)) ||
130+ attr->map_flags & ~DEV_CREATE_FLAG_MASK)
115131 return -EINVAL;
116132
117133 /* Lookup returns a pointer straight to dev->ifindex, so make sure the
@@ -218,6 +234,8 @@ static void dev_map_free(struct bpf_map *map) · 100 unmodified lines
218234
219235 hlist_for_each_entry_safe(dev, next, head, index_hlist) {
220236 hlist_del_rcu(&dev->index_hlist);
237+ if (dev->xdp_prog)
238+ bpf_prog_put(dev->xdp_prog);
221239 dev_put(dev->dev);
222240 kfree(dev);
223241 }
@@ -232,6 +250,8 @@ static void dev_map_free(struct bpf_map *map) · 8 unmodified lines
232250 if (!dev)
233251 continue;
234252
253+ if (dev->xdp_prog)
254+ bpf_prog_put(dev->xdp_prog);
235255 dev_put(dev->dev);
236256 kfree(dev);
237257 }
@@ -318,6 +338,16 @@ static int dev_map_hash_get_next_key(struct bpf_map *map, void *key, · 80 unmodified lines
318338 return -ENOENT;
319339 }
320340
341+bool dev_map_can_have_prog(struct bpf_map *map)
342+{
343+ if ((map->map_type == BPF_MAP_TYPE_DEVMAP ||
344+ map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) &&
345+ map->value_size != offsetofend(struct bpf_devmap_val, ifindex))
346+ return true;
347+
348+ return false;
349+}
350+
321351 static int bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
322352 {
323353 struct net_device *dev = bq->dev;
@@ -435,13 +465,40 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, · 111 unmodified lines
435465 if (unlikely(err))
436466 return err;
437467
438− xdpf = convert_to_xdp_frame(xdp);
468+ xdpf = xdp_convert_buff_to_frame(xdp);
439469 if (unlikely(!xdpf))
440470 return -EOVERFLOW;
441471
442472 return bq_enqueue(dev, xdpf, dev_rx);
443473 }
444474
475+static struct xdp_buff *dev_map_run_prog(struct net_device *dev,
476+ struct xdp_buff *xdp,
477+ struct bpf_prog *xdp_prog)
478+{
479+ struct xdp_txq_info txq = { .dev = dev };
480+ u32 act;
481+
482+ xdp->txq = &txq;
483+
484+ act = bpf_prog_run_xdp(xdp_prog, xdp);
485+ switch (act) {
486+ case XDP_PASS:
487+ return xdp;
488+ case XDP_DROP:
489+ break;
490+ default:
491+ bpf_warn_invalid_xdp_action(act);
492+ fallthrough;
493+ case XDP_ABORTED:
494+ trace_xdp_exception(dev, xdp_prog, act);
495+ break;
496+ }
497+
498+ xdp_return_buff(xdp);
499+ return NULL;
500+}
501+
445502 int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp,
446503 struct net_device *dev_rx)
447504 {
@@ -453,6 +510,11 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, · 5 unmodified lines
453510 {
454511 struct net_device *dev = dst->dev;
455512
513+ if (dst->xdp_prog) {
514+ xdp = dev_map_run_prog(dev, xdp, dst->xdp_prog);
515+ if (!xdp)
516+ return 0;
517+ }
456518 return __xdp_enqueue(dev, xdp, dev_rx);
457519 }
458520
@@ -473,18 +535,15 @@ int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, · 14 unmodified lines
473535 static void *dev_map_lookup_elem(struct bpf_map *map, void *key)
474536 {
475537 struct bpf_dtab_netdev *obj = __dev_map_lookup_elem(map, *(u32 *)key);
476− struct net_device *dev = obj ? obj->dev : NULL;
477538
478− return dev ? &dev->ifindex : NULL;
539+ return obj ? &obj->val : NULL;
479540 }
480541
481542 static void *dev_map_hash_lookup_elem(struct bpf_map *map, void *key)
482543 {
483544 struct bpf_dtab_netdev *obj = __dev_map_hash_lookup_elem(map,
484545 *(u32 *)key);
485− struct net_device *dev = obj ? obj->dev : NULL;
486−
487− return dev ? &dev->ifindex : NULL;
546+ return obj ? &obj->val : NULL;
488547 }
489548
490549 static void __dev_map_entry_free(struct rcu_head *rcu)
@@ -492,6 +551,8 @@ static void __dev_map_entry_free(struct rcu_head *rcu) · 1 unmodified line
492551 struct bpf_dtab_netdev *dev;
493552
494553 dev = container_of(rcu, struct bpf_dtab_netdev, rcu);
554+ if (dev->xdp_prog)
555+ bpf_prog_put(dev->xdp_prog);
495556 dev_put(dev->dev);
496557 kfree(dev);
497558 }
@@ -542,9 +603,10 @@ static int dev_map_hash_delete_elem(struct bpf_map *map, void *key) · 44 unmodified lines
542603
543604 static struct bpf_dtab_netdev *__dev_map_alloc_node(struct net *net,
544605 struct bpf_dtab *dtab,
545− u32 ifindex,
606+ struct bpf_devmap_val *val,
546607 unsigned int idx)
547608 {
609+ struct bpf_prog *prog = NULL;
548610 struct bpf_dtab_netdev *dev;
549611
550612 dev = kmalloc_node(sizeof(*dev), GFP_ATOMIC | __GFP_NOWARN,
@@ -552,24 +614,46 @@ static struct bpf_dtab_netdev *__dev_map_alloc_node(struct net *net, · 1 unmodified line
552614 if (!dev)
553615 return ERR_PTR(-ENOMEM);
554616
555− dev->dev = dev_get_by_index(net, ifindex);
556− if (!dev->dev) {
557− kfree(dev);
558− return ERR_PTR(-EINVAL);
617+ dev->dev = dev_get_by_index(net, val->ifindex);
618+ if (!dev->dev)
619+ goto err_out;
620+
621+ if (val->bpf_prog.fd >= 0) {
622+ prog = bpf_prog_get_type_dev(val->bpf_prog.fd,
623+ BPF_PROG_TYPE_XDP, false);
624+ if (IS_ERR(prog))
625+ goto err_put_dev;
626+ if (prog->expected_attach_type != BPF_XDP_DEVMAP)
627+ goto err_put_prog;
559628 }
560629
561630 dev->idx = idx;
562631 dev->dtab = dtab;
632+ if (prog) {
633+ dev->xdp_prog = prog;
634+ dev->val.bpf_prog.id = prog->aux->id;
635+ } else {
636+ dev->xdp_prog = NULL;
637+ dev->val.bpf_prog.id = 0;
638+ }
639+ dev->val.ifindex = val->ifindex;
563640
564641 return dev;
642+err_put_prog:
643+ bpf_prog_put(prog);
644+err_put_dev:
645+ dev_put(dev->dev);
646+err_out:
647+ kfree(dev);
648+ return ERR_PTR(-EINVAL);
565649 }
566650
567651 static int __dev_map_update_elem(struct net *net, struct bpf_map *map,
568652 void *key, void *value, u64 map_flags)
569653 {
570654 struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
655+ struct bpf_devmap_val val = { .bpf_prog.fd = -1 };
571656 struct bpf_dtab_netdev *dev, *old_dev;
572− u32 ifindex = *(u32 *)value;
573657 u32 i = *(u32 *)key;
574658
575659 if (unlikely(map_flags > BPF_EXIST))
@@ -579,10 +663,16 @@ static int __dev_map_update_elem(struct net *net, struct bpf_map *map, · 3 unmodified lines
579663 if (unlikely(map_flags == BPF_NOEXIST))
580664 return -EEXIST;
581665
582− if (!ifindex) {
666+ /* already verified value_size <= sizeof val */
667+ memcpy(&val, value, map->value_size);
668+
669+ if (!val.ifindex) {
583670 dev = NULL;
671+ /* can not specify fd if ifindex is 0 */
672+ if (val.bpf_prog.fd != -1)
673+ return -EINVAL;
584674 } else {
585− dev = __dev_map_alloc_node(net, dtab, ifindex, i);
675+ dev = __dev_map_alloc_node(net, dtab, &val, i);
586676 if (IS_ERR(dev))
587677 return PTR_ERR(dev);
588678 }
@@ -609,13 +699,16 @@ static int __dev_map_hash_update_elem(struct net *net, struct bpf_map *map, · 20 unmodified lines
609699 void *key, void *value, u64 map_flags)
610700 {
611701 struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map);
702+ struct bpf_devmap_val val = { .bpf_prog.fd = -1 };
612703 struct bpf_dtab_netdev *dev, *old_dev;
613− u32 ifindex = *(u32 *)value;
614704 u32 idx = *(u32 *)key;
615705 unsigned long flags;
616706 int err = -EEXIST;
617707
618− if (unlikely(map_flags > BPF_EXIST || !ifindex))
708+ /* already verified value_size <= sizeof val */
709+ memcpy(&val, value, map->value_size);
710+
711+ if (unlikely(map_flags > BPF_EXIST || !val.ifindex))
619712 return -EINVAL;
620713
621714 spin_lock_irqsave(&dtab->index_lock, flags);
@@ -624,7 +717,7 @@ static int __dev_map_hash_update_elem(struct net *net, struct bpf_map *map, · 2 unmodified lines
624717 if (old_dev && (map_flags & BPF_NOEXIST))
625718 goto out_err;
626719
627− dev = __dev_map_alloc_node(net, dtab, ifindex, idx);
720+ dev = __dev_map_alloc_node(net, dtab, &val, idx);
628721 if (IS_ERR(dev)) {
629722 err = PTR_ERR(dev);
630723 goto out_err;
kernel/bpf/hashtab.c+2−2
@@ -359,9 +359,9 @@ static int htab_map_alloc_check(union bpf_attr *attr) · 358 unmodified lines
359359 BUILD_BUG_ON(offsetof(struct htab_elem, fnode.next) !=
360360 offsetof(struct htab_elem, hash_node.pprev));
361361
362− if (lru && !capable(CAP_SYS_ADMIN))
362+ if (lru && !bpf_capable())
363363 /* LRU implementation is much complicated than other
364− * maps. Hence, limit to CAP_SYS_ADMIN for now.
364+ * maps. Hence, limit to CAP_BPF.
365365 */
366366 return -EPERM;
367367
kernel/bpf/helpers.c+124−1
@@ -151,7 +151,19 @@ BPF_CALL_0(bpf_ktime_get_ns) · 150 unmodified lines
151151
152152 const struct bpf_func_proto bpf_ktime_get_ns_proto = {
153153 .func = bpf_ktime_get_ns,
154− .gpl_only = true,
154+ .gpl_only = false,
155+ .ret_type = RET_INTEGER,
156+};
157+
158+BPF_CALL_0(bpf_ktime_get_boot_ns)
159+{
160+ /* NMI safe access to clock boottime */
161+ return ktime_get_boot_fast_ns();
162+}
163+
164+const struct bpf_func_proto bpf_ktime_get_boot_ns_proto = {
165+ .func = bpf_ktime_get_boot_ns,
166+ .gpl_only = false,
155167 .ret_type = RET_INTEGER,
156168 };
157169
@@ -562,3 +574,114 @@ const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto = { · 404 unmodified lines
562574 .arg3_type = ARG_PTR_TO_UNINIT_MEM,
563575 .arg4_type = ARG_CONST_SIZE,
564576 };
577+
578+static const struct bpf_func_proto bpf_get_raw_smp_processor_id_proto = {
579+ .func = bpf_get_raw_cpu_id,
580+ .gpl_only = false,
581+ .ret_type = RET_INTEGER,
582+};
583+
584+BPF_CALL_5(bpf_event_output_data, void *, ctx, struct bpf_map *, map,
585+ u64, flags, void *, data, u64, size)
586+{
587+ if (unlikely(flags & ~(BPF_F_INDEX_MASK)))
588+ return -EINVAL;
589+
590+ return bpf_event_output(map, flags, data, size, NULL, 0, NULL);
591+}
592+
593+const struct bpf_func_proto bpf_event_output_data_proto = {
594+ .func = bpf_event_output_data,
595+ .gpl_only = true,
596+ .ret_type = RET_INTEGER,
597+ .arg1_type = ARG_PTR_TO_CTX,
598+ .arg2_type = ARG_CONST_MAP_PTR,
599+ .arg3_type = ARG_ANYTHING,
600+ .arg4_type = ARG_PTR_TO_MEM,
601+ .arg5_type = ARG_CONST_SIZE_OR_ZERO,
602+};
603+
604+const struct bpf_func_proto bpf_get_current_task_proto __weak;
605+const struct bpf_func_proto bpf_probe_read_user_proto __weak;
606+const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;
607+const struct bpf_func_proto bpf_probe_read_kernel_proto __weak;
608+const struct bpf_func_proto bpf_probe_read_kernel_str_proto __weak;
609+
610+const struct bpf_func_proto *
611+bpf_base_func_proto(enum bpf_func_id func_id)
612+{
613+ switch (func_id) {
614+ case BPF_FUNC_map_lookup_elem:
615+ return &bpf_map_lookup_elem_proto;
616+ case BPF_FUNC_map_update_elem:
617+ return &bpf_map_update_elem_proto;
618+ case BPF_FUNC_map_delete_elem:
619+ return &bpf_map_delete_elem_proto;
620+ case BPF_FUNC_map_push_elem:
621+ return &bpf_map_push_elem_proto;
622+ case BPF_FUNC_map_pop_elem:
623+ return &bpf_map_pop_elem_proto;
624+ case BPF_FUNC_map_peek_elem:
625+ return &bpf_map_peek_elem_proto;
626+ case BPF_FUNC_get_prandom_u32:
627+ return &bpf_get_prandom_u32_proto;
628+ case BPF_FUNC_get_smp_processor_id:
629+ return &bpf_get_raw_smp_processor_id_proto;
630+ case BPF_FUNC_get_numa_node_id:
631+ return &bpf_get_numa_node_id_proto;
632+ case BPF_FUNC_tail_call:
633+ return &bpf_tail_call_proto;
634+ case BPF_FUNC_ktime_get_ns:
635+ return &bpf_ktime_get_ns_proto;
636+ case BPF_FUNC_ktime_get_boot_ns:
637+ return &bpf_ktime_get_boot_ns_proto;
638+ case BPF_FUNC_ringbuf_output:
639+ return &bpf_ringbuf_output_proto;
640+ case BPF_FUNC_ringbuf_reserve:
641+ return &bpf_ringbuf_reserve_proto;
642+ case BPF_FUNC_ringbuf_submit:
643+ return &bpf_ringbuf_submit_proto;
644+ case BPF_FUNC_ringbuf_discard:
645+ return &bpf_ringbuf_discard_proto;
646+ case BPF_FUNC_ringbuf_query:
647+ return &bpf_ringbuf_query_proto;
648+ default:
649+ break;
650+ }
651+
652+ if (!bpf_capable())
653+ return NULL;
654+
655+ switch (func_id) {
656+ case BPF_FUNC_spin_lock:
657+ return &bpf_spin_lock_proto;
658+ case BPF_FUNC_spin_unlock:
659+ return &bpf_spin_unlock_proto;
660+ case BPF_FUNC_trace_printk:
661+ if (!perfmon_capable())
662+ return NULL;
663+ return bpf_get_trace_printk_proto();
664+ case BPF_FUNC_jiffies64:
665+ return &bpf_jiffies64_proto;
666+ default:
667+ break;
668+ }
669+
670+ if (!perfmon_capable())
671+ return NULL;
672+
673+ switch (func_id) {
674+ case BPF_FUNC_get_current_task:
675+ return &bpf_get_current_task_proto;
676+ case BPF_FUNC_probe_read_user:
677+ return &bpf_probe_read_user_proto;
678+ case BPF_FUNC_probe_read_kernel:
679+ return &bpf_probe_read_kernel_proto;
680+ case BPF_FUNC_probe_read_user_str:
681+ return &bpf_probe_read_user_str_proto;
682+ case BPF_FUNC_probe_read_kernel_str:
683+ return &bpf_probe_read_kernel_str_proto;
684+ default:
685+ return NULL;
686+ }
687+}
kernel/bpf/inode.c+4−1
@@ -358,8 +358,11 @@ static int bpf_mkmap(struct dentry *dentry, umode_t mode, void *arg) · 357 unmodified lines
358358
359359 static int bpf_mklink(struct dentry *dentry, umode_t mode, void *arg)
360360 {
361+ struct bpf_link *link = arg;
362+
361363 return bpf_mkobj_ops(dentry, mode, arg, &bpf_link_iops,
362− &bpffs_obj_fops);
364+ bpf_link_is_iter(link) ?
365+ &bpf_iter_fops : &bpffs_obj_fops);
363366 }
364367
365368 static struct dentry *
kernel/bpf/lpm_trie.c+1−1
@@ -543,7 +543,7 @@ static struct bpf_map *trie_alloc(union bpf_attr *attr) · 542 unmodified lines
543543 u64 cost = sizeof(*trie), cost_per_node;
544544 int ret;
545545
546− if (!capable(CAP_SYS_ADMIN))
546+ if (!bpf_capable())
547547 return ERR_PTR(-EPERM);
548548
549549 /* check sanity of attributes */
kernel/bpf/map_in_map.c+1−1
@@ -60,7 +60,7 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) · 59 unmodified lines
6060 /* Misc members not needed in bpf_map_meta_equal() check. */
6161 inner_map_meta->ops = inner_map->ops;
6262 if (inner_map->ops == &array_map_ops) {
63− inner_map_meta->unpriv_array = inner_map->unpriv_array;
63+ inner_map_meta->bypass_spec_v1 = inner_map->bypass_spec_v1;
6464 container_of(inner_map_meta, struct bpf_array, map)->index_mask =
6565 container_of(inner_map, struct bpf_array, map)->index_mask;
6666 }
kernel/bpf/map_iter.c+102−0
@@ -0,0 +1,102 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/* Copyright (c) 2020 Facebook */
3+#include <linux/bpf.h>
4+#include <linux/fs.h>
5+#include <linux/filter.h>
6+#include <linux/kernel.h>
7+
8+struct bpf_iter_seq_map_info {
9+ u32 mid;
10+};
11+
12+static void *bpf_map_seq_start(struct seq_file *seq, loff_t *pos)
13+{
14+ struct bpf_iter_seq_map_info *info = seq->private;
15+ struct bpf_map *map;
16+
17+ map = bpf_map_get_curr_or_next(&info->mid);
18+ if (!map)
19+ return NULL;
20+
21+ ++*pos;
22+ return map;
23+}
24+
25+static void *bpf_map_seq_next(struct seq_file *seq, void *v, loff_t *pos)
26+{
27+ struct bpf_iter_seq_map_info *info = seq->private;
28+ struct bpf_map *map;
29+
30+ ++*pos;
31+ ++info->mid;
32+ bpf_map_put((struct bpf_map *)v);
33+ map = bpf_map_get_curr_or_next(&info->mid);
34+ if (!map)
35+ return NULL;
36+
37+ return map;
38+}
39+
40+struct bpf_iter__bpf_map {
41+ __bpf_md_ptr(struct bpf_iter_meta *, meta);
42+ __bpf_md_ptr(struct bpf_map *, map);
43+};
44+
45+DEFINE_BPF_ITER_FUNC(bpf_map, struct bpf_iter_meta *meta, struct bpf_map *map)
46+
47+static int __bpf_map_seq_show(struct seq_file *seq, void *v, bool in_stop)
48+{
49+ struct bpf_iter__bpf_map ctx;
50+ struct bpf_iter_meta meta;
51+ struct bpf_prog *prog;
52+ int ret = 0;
53+
54+ ctx.meta = &meta;
55+ ctx.map = v;
56+ meta.seq = seq;
57+ prog = bpf_iter_get_info(&meta, in_stop);
58+ if (prog)
59+ ret = bpf_iter_run_prog(prog, &ctx);
60+
61+ return ret;
62+}
63+
64+static int bpf_map_seq_show(struct seq_file *seq, void *v)
65+{
66+ return __bpf_map_seq_show(seq, v, false);
67+}
68+
69+static void bpf_map_seq_stop(struct seq_file *seq, void *v)
70+{
71+ if (!v)
72+ (void)__bpf_map_seq_show(seq, v, true);
73+ else
74+ bpf_map_put((struct bpf_map *)v);
75+}
76+
77+static const struct seq_operations bpf_map_seq_ops = {
78+ .start = bpf_map_seq_start,
79+ .next = bpf_map_seq_next,
80+ .stop = bpf_map_seq_stop,
81+ .show = bpf_map_seq_show,
82+};
83+
84+static const struct bpf_iter_reg bpf_map_reg_info = {
85+ .target = "bpf_map",
86+ .seq_ops = &bpf_map_seq_ops,
87+ .init_seq_private = NULL,
88+ .fini_seq_private = NULL,
89+ .seq_priv_size = sizeof(struct bpf_iter_seq_map_info),
90+ .ctx_arg_info_size = 1,
91+ .ctx_arg_info = {
92+ { offsetof(struct bpf_iter__bpf_map, map),
93+ PTR_TO_BTF_ID_OR_NULL },
94+ },
95+};
96+
97+static int __init bpf_map_iter_init(void)
98+{
99+ return bpf_iter_reg_target(&bpf_map_reg_info);
100+}
101+
102+late_initcall(bpf_map_iter_init);
kernel/bpf/net_namespace.c+373−0
@@ -0,0 +1,373 @@
1+// SPDX-License-Identifier: GPL-2.0
2+
3+#include <linux/bpf.h>
4+#include <linux/filter.h>
5+#include <net/net_namespace.h>
6+
7+/*
8+ * Functions to manage BPF programs attached to netns
9+ */
10+
11+struct bpf_netns_link {
12+ struct bpf_link link;
13+ enum bpf_attach_type type;
14+ enum netns_bpf_attach_type netns_type;
15+
16+ /* We don't hold a ref to net in order to auto-detach the link
17+ * when netns is going away. Instead we rely on pernet
18+ * pre_exit callback to clear this pointer. Must be accessed
19+ * with netns_bpf_mutex held.
20+ */
21+ struct net *net;
22+};
23+
24+/* Protects updates to netns_bpf */
25+DEFINE_MUTEX(netns_bpf_mutex);
26+
27+/* Must be called with netns_bpf_mutex held. */
28+static void __net_exit bpf_netns_link_auto_detach(struct bpf_link *link)
29+{
30+ struct bpf_netns_link *net_link =
31+ container_of(link, struct bpf_netns_link, link);
32+
33+ net_link->net = NULL;
34+}
35+
36+static void bpf_netns_link_release(struct bpf_link *link)
37+{
38+ struct bpf_netns_link *net_link =
39+ container_of(link, struct bpf_netns_link, link);
40+ enum netns_bpf_attach_type type = net_link->netns_type;
41+ struct net *net;
42+
43+ /* Link auto-detached by dying netns. */
44+ if (!net_link->net)
45+ return;
46+
47+ mutex_lock(&netns_bpf_mutex);
48+
49+ /* Recheck after potential sleep. We can race with cleanup_net
50+ * here, but if we see a non-NULL struct net pointer pre_exit
51+ * has not happened yet and will block on netns_bpf_mutex.
52+ */
53+ net = net_link->net;
54+ if (!net)
55+ goto out_unlock;
56+
57+ net->bpf.links[type] = NULL;
58+ RCU_INIT_POINTER(net->bpf.progs[type], NULL);
59+
60+out_unlock:
61+ mutex_unlock(&netns_bpf_mutex);
62+}
63+
64+static void bpf_netns_link_dealloc(struct bpf_link *link)
65+{
66+ struct bpf_netns_link *net_link =
67+ container_of(link, struct bpf_netns_link, link);
68+
69+ kfree(net_link);
70+}
71+
72+static int bpf_netns_link_update_prog(struct bpf_link *link,
73+ struct bpf_prog *new_prog,
74+ struct bpf_prog *old_prog)
75+{
76+ struct bpf_netns_link *net_link =
77+ container_of(link, struct bpf_netns_link, link);
78+ enum netns_bpf_attach_type type = net_link->netns_type;
79+ struct net *net;
80+ int ret = 0;
81+
82+ if (old_prog && old_prog != link->prog)
83+ return -EPERM;
84+ if (new_prog->type != link->prog->type)
85+ return -EINVAL;
86+
87+ mutex_lock(&netns_bpf_mutex);
88+
89+ net = net_link->net;
90+ if (!net || !check_net(net)) {
91+ /* Link auto-detached or netns dying */
92+ ret = -ENOLINK;
93+ goto out_unlock;
94+ }
95+
96+ old_prog = xchg(&link->prog, new_prog);
97+ rcu_assign_pointer(net->bpf.progs[type], new_prog);
98+ bpf_prog_put(old_prog);
99+
100+out_unlock:
101+ mutex_unlock(&netns_bpf_mutex);
102+ return ret;
103+}
104+
105+static int bpf_netns_link_fill_info(const struct bpf_link *link,
106+ struct bpf_link_info *info)
107+{
108+ const struct bpf_netns_link *net_link =
109+ container_of(link, struct bpf_netns_link, link);
110+ unsigned int inum = 0;
111+ struct net *net;
112+
113+ mutex_lock(&netns_bpf_mutex);
114+ net = net_link->net;
115+ if (net && check_net(net))
116+ inum = net->ns.inum;
117+ mutex_unlock(&netns_bpf_mutex);
118+
119+ info->netns.netns_ino = inum;
120+ info->netns.attach_type = net_link->type;
121+ return 0;
122+}
123+
124+static void bpf_netns_link_show_fdinfo(const struct bpf_link *link,
125+ struct seq_file *seq)
126+{
127+ struct bpf_link_info info = {};
128+
129+ bpf_netns_link_fill_info(link, &info);
130+ seq_printf(seq,
131+ "netns_ino:\t%u\n"
132+ "attach_type:\t%u\n",
133+ info.netns.netns_ino,
134+ info.netns.attach_type);
135+}
136+
137+static const struct bpf_link_ops bpf_netns_link_ops = {
138+ .release = bpf_netns_link_release,
139+ .dealloc = bpf_netns_link_dealloc,
140+ .update_prog = bpf_netns_link_update_prog,
141+ .fill_link_info = bpf_netns_link_fill_info,
142+ .show_fdinfo = bpf_netns_link_show_fdinfo,
143+};
144+
145+int netns_bpf_prog_query(const union bpf_attr *attr,
146+ union bpf_attr __user *uattr)
147+{
148+ __u32 __user *prog_ids = u64_to_user_ptr(attr->query.prog_ids);
149+ u32 prog_id, prog_cnt = 0, flags = 0;
150+ enum netns_bpf_attach_type type;
151+ struct bpf_prog *attached;
152+ struct net *net;
153+
154+ if (attr->query.query_flags)
155+ return -EINVAL;
156+
157+ type = to_netns_bpf_attach_type(attr->query.attach_type);
158+ if (type < 0)
159+ return -EINVAL;
160+
161+ net = get_net_ns_by_fd(attr->query.target_fd);
162+ if (IS_ERR(net))
163+ return PTR_ERR(net);
164+
165+ rcu_read_lock();
166+ attached = rcu_dereference(net->bpf.progs[type]);
167+ if (attached) {
168+ prog_cnt = 1;
169+ prog_id = attached->aux->id;
170+ }
171+ rcu_read_unlock();
172+
173+ put_net(net);
174+
175+ if (copy_to_user(&uattr->query.attach_flags, &flags, sizeof(flags)))
176+ return -EFAULT;
177+ if (copy_to_user(&uattr->query.prog_cnt, &prog_cnt, sizeof(prog_cnt)))
178+ return -EFAULT;
179+
180+ if (!attr->query.prog_cnt || !prog_ids || !prog_cnt)
181+ return 0;
182+
183+ if (copy_to_user(prog_ids, &prog_id, sizeof(u32)))
184+ return -EFAULT;
185+
186+ return 0;
187+}
188+
189+int netns_bpf_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog)
190+{
191+ enum netns_bpf_attach_type type;
192+ struct net *net;
193+ int ret;
194+
195+ type = to_netns_bpf_attach_type(attr->attach_type);
196+ if (type < 0)
197+ return -EINVAL;
198+
199+ net = current->nsproxy->net_ns;
200+ mutex_lock(&netns_bpf_mutex);
201+
202+ /* Attaching prog directly is not compatible with links */
203+ if (net->bpf.links[type]) {
204+ ret = -EEXIST;
205+ goto out_unlock;
206+ }
207+
208+ switch (type) {
209+ case NETNS_BPF_FLOW_DISSECTOR:
210+ ret = flow_dissector_bpf_prog_attach(net, prog);
211+ break;
212+ default:
213+ ret = -EINVAL;
214+ break;
215+ }
216+out_unlock:
217+ mutex_unlock(&netns_bpf_mutex);
218+
219+ return ret;
220+}
221+
222+/* Must be called with netns_bpf_mutex held. */
223+static int __netns_bpf_prog_detach(struct net *net,
224+ enum netns_bpf_attach_type type)
225+{
226+ struct bpf_prog *attached;
227+
228+ /* Progs attached via links cannot be detached */
229+ if (net->bpf.links[type])
230+ return -EINVAL;
231+
232+ attached = rcu_dereference_protected(net->bpf.progs[type],
233+ lockdep_is_held(&netns_bpf_mutex));
234+ if (!attached)
235+ return -ENOENT;
236+ RCU_INIT_POINTER(net->bpf.progs[type], NULL);
237+ bpf_prog_put(attached);
238+ return 0;
239+}
240+
241+int netns_bpf_prog_detach(const union bpf_attr *attr)
242+{
243+ enum netns_bpf_attach_type type;
244+ int ret;
245+
246+ type = to_netns_bpf_attach_type(attr->attach_type);
247+ if (type < 0)
248+ return -EINVAL;
249+
250+ mutex_lock(&netns_bpf_mutex);
251+ ret = __netns_bpf_prog_detach(current->nsproxy->net_ns, type);
252+ mutex_unlock(&netns_bpf_mutex);
253+
254+ return ret;
255+}
256+
257+static int netns_bpf_link_attach(struct net *net, struct bpf_link *link,
258+ enum netns_bpf_attach_type type)
259+{
260+ struct bpf_prog *prog;
261+ int err;
262+
263+ mutex_lock(&netns_bpf_mutex);
264+
265+ /* Allow attaching only one prog or link for now */
266+ if (net->bpf.links[type]) {
267+ err = -E2BIG;
268+ goto out_unlock;
269+ }
270+ /* Links are not compatible with attaching prog directly */
271+ prog = rcu_dereference_protected(net->bpf.progs[type],
272+ lockdep_is_held(&netns_bpf_mutex));
273+ if (prog) {
274+ err = -EEXIST;
275+ goto out_unlock;
276+ }
277+
278+ switch (type) {
279+ case NETNS_BPF_FLOW_DISSECTOR:
280+ err = flow_dissector_bpf_prog_attach(net, link->prog);
281+ break;
282+ default:
283+ err = -EINVAL;
284+ break;
285+ }
286+ if (err)
287+ goto out_unlock;
288+
289+ net->bpf.links[type] = link;
290+
291+out_unlock:
292+ mutex_unlock(&netns_bpf_mutex);
293+ return err;
294+}
295+
296+int netns_bpf_link_create(const union bpf_attr *attr, struct bpf_prog *prog)
297+{
298+ enum netns_bpf_attach_type netns_type;
299+ struct bpf_link_primer link_primer;
300+ struct bpf_netns_link *net_link;
301+ enum bpf_attach_type type;
302+ struct net *net;
303+ int err;
304+
305+ if (attr->link_create.flags)
306+ return -EINVAL;
307+
308+ type = attr->link_create.attach_type;
309+ netns_type = to_netns_bpf_attach_type(type);
310+ if (netns_type < 0)
311+ return -EINVAL;
312+
313+ net = get_net_ns_by_fd(attr->link_create.target_fd);
314+ if (IS_ERR(net))
315+ return PTR_ERR(net);
316+
317+ net_link = kzalloc(sizeof(*net_link), GFP_USER);
318+ if (!net_link) {
319+ err = -ENOMEM;
320+ goto out_put_net;
321+ }
322+ bpf_link_init(&net_link->link, BPF_LINK_TYPE_NETNS,
323+ &bpf_netns_link_ops, prog);
324+ net_link->net = net;
325+ net_link->type = type;
326+ net_link->netns_type = netns_type;
327+
328+ err = bpf_link_prime(&net_link->link, &link_primer);
329+ if (err) {
330+ kfree(net_link);
331+ goto out_put_net;
332+ }
333+
334+ err = netns_bpf_link_attach(net, &net_link->link, netns_type);
335+ if (err) {
336+ bpf_link_cleanup(&link_primer);
337+ goto out_put_net;
338+ }
339+
340+ put_net(net);
341+ return bpf_link_settle(&link_primer);
342+
343+out_put_net:
344+ put_net(net);
345+ return err;
346+}
347+
348+static void __net_exit netns_bpf_pernet_pre_exit(struct net *net)
349+{
350+ enum netns_bpf_attach_type type;
351+ struct bpf_link *link;
352+
353+ mutex_lock(&netns_bpf_mutex);
354+ for (type = 0; type < MAX_NETNS_BPF_ATTACH_TYPE; type++) {
355+ link = net->bpf.links[type];
356+ if (link)
357+ bpf_netns_link_auto_detach(link);
358+ else
359+ __netns_bpf_prog_detach(net, type);
360+ }
361+ mutex_unlock(&netns_bpf_mutex);
362+}
363+
364+static struct pernet_operations netns_bpf_pernet_ops __net_initdata = {
365+ .pre_exit = netns_bpf_pernet_pre_exit,
366+};
367+
368+static int __init netns_bpf_init(void)
369+{
370+ return register_pernet_subsys(&netns_bpf_pernet_ops);
371+}
372+
373+subsys_initcall(netns_bpf_init);
kernel/bpf/queue_stack_maps.c+2−2
@@ -19,7 +19,7 @@ struct bpf_queue_stack { · 18 unmodified lines
1919 u32 head, tail;
2020 u32 size; /* max_entries + 1 */
2121
22− char elements[0] __aligned(8);
22+ char elements[] __aligned(8);
2323 };
2424
2525 static struct bpf_queue_stack *bpf_queue_stack(struct bpf_map *map)
@@ -45,7 +45,7 @@ static bool queue_stack_map_is_full(struct bpf_queue_stack *qs) · 19 unmodified lines
4545 /* Called from syscall */
4646 static int queue_stack_map_alloc_check(union bpf_attr *attr)
4747 {
48− if (!capable(CAP_SYS_ADMIN))
48+ if (!bpf_capable())
4949 return -EPERM;
5050
5151 /* check sanity of attributes */
kernel/bpf/reuseport_array.c+1−1
@@ -154,7 +154,7 @@ static struct bpf_map *reuseport_array_alloc(union bpf_attr *attr) · 153 unmodified lines
154154 struct bpf_map_memory mem;
155155 u64 array_size;
156156
157− if (!capable(CAP_SYS_ADMIN))
157+ if (!bpf_capable())
158158 return ERR_PTR(-EPERM);
159159
160160 array_size = sizeof(*array);
kernel/bpf/ringbuf.c+501−0
@@ -0,0 +1,501 @@
1+#include <linux/bpf.h>
2+#include <linux/btf.h>
3+#include <linux/err.h>
4+#include <linux/irq_work.h>
5+#include <linux/slab.h>
6+#include <linux/filter.h>
7+#include <linux/mm.h>
8+#include <linux/vmalloc.h>
9+#include <linux/wait.h>
10+#include <linux/poll.h>
11+#include <uapi/linux/btf.h>
12+
13+#define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)
14+
15+/* non-mmap()'able part of bpf_ringbuf (everything up to consumer page) */
16+#define RINGBUF_PGOFF \
17+ (offsetof(struct bpf_ringbuf, consumer_pos) >> PAGE_SHIFT)
18+/* consumer page and producer page */
19+#define RINGBUF_POS_PAGES 2
20+
21+#define RINGBUF_MAX_RECORD_SZ (UINT_MAX/4)
22+
23+/* Maximum size of ring buffer area is limited by 32-bit page offset within
24+ * record header, counted in pages. Reserve 8 bits for extensibility, and take
25+ * into account few extra pages for consumer/producer pages and
26+ * non-mmap()'able parts. This gives 64GB limit, which seems plenty for single
27+ * ring buffer.
28+ */
29+#define RINGBUF_MAX_DATA_SZ \
30+ (((1ULL << 24) - RINGBUF_POS_PAGES - RINGBUF_PGOFF) * PAGE_SIZE)
31+
32+struct bpf_ringbuf {
33+ wait_queue_head_t waitq;
34+ struct irq_work work;
35+ u64 mask;
36+ struct page **pages;
37+ int nr_pages;
38+ spinlock_t spinlock ____cacheline_aligned_in_smp;
39+ /* Consumer and producer counters are put into separate pages to allow
40+ * mapping consumer page as r/w, but restrict producer page to r/o.
41+ * This protects producer position from being modified by user-space
42+ * application and ruining in-kernel position tracking.
43+ */
44+ unsigned long consumer_pos __aligned(PAGE_SIZE);
45+ unsigned long producer_pos __aligned(PAGE_SIZE);
46+ char data[] __aligned(PAGE_SIZE);
47+};
48+
49+struct bpf_ringbuf_map {
50+ struct bpf_map map;
51+ struct bpf_map_memory memory;
52+ struct bpf_ringbuf *rb;
53+};
54+
55+/* 8-byte ring buffer record header structure */
56+struct bpf_ringbuf_hdr {
57+ u32 len;
58+ u32 pg_off;
59+};
60+
61+static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node)
62+{
63+ const gfp_t flags = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN |
64+ __GFP_ZERO;
65+ int nr_meta_pages = RINGBUF_PGOFF + RINGBUF_POS_PAGES;
66+ int nr_data_pages = data_sz >> PAGE_SHIFT;
67+ int nr_pages = nr_meta_pages + nr_data_pages;
68+ struct page **pages, *page;
69+ struct bpf_ringbuf *rb;
70+ size_t array_size;
71+ int i;
72+
73+ /* Each data page is mapped twice to allow "virtual"
74+ * continuous read of samples wrapping around the end of ring
75+ * buffer area:
76+ * ------------------------------------------------------
77+ * | meta pages | real data pages | same data pages |
78+ * ------------------------------------------------------
79+ * | | 1 2 3 4 5 6 7 8 9 | 1 2 3 4 5 6 7 8 9 |
80+ * ------------------------------------------------------
81+ * | | TA DA | TA DA |
82+ * ------------------------------------------------------
83+ * ^^^^^^^
84+ * |
85+ * Here, no need to worry about special handling of wrapped-around
86+ * data due to double-mapped data pages. This works both in kernel and
87+ * when mmap()'ed in user-space, simplifying both kernel and
88+ * user-space implementations significantly.
89+ */
90+ array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages);
91+ if (array_size > PAGE_SIZE)
92+ pages = vmalloc_node(array_size, numa_node);
93+ else
94+ pages = kmalloc_node(array_size, flags, numa_node);
95+ if (!pages)
96+ return NULL;
97+
98+ for (i = 0; i < nr_pages; i++) {
99+ page = alloc_pages_node(numa_node, flags, 0);
100+ if (!page) {
101+ nr_pages = i;
102+ goto err_free_pages;
103+ }
104+ pages[i] = page;
105+ if (i >= nr_meta_pages)
106+ pages[nr_data_pages + i] = page;
107+ }
108+
109+ rb = vmap(pages, nr_meta_pages + 2 * nr_data_pages,
110+ VM_ALLOC | VM_USERMAP, PAGE_KERNEL);
111+ if (rb) {
112+ rb->pages = pages;
113+ rb->nr_pages = nr_pages;
114+ return rb;
115+ }
116+
117+err_free_pages:
118+ for (i = 0; i < nr_pages; i++)
119+ __free_page(pages[i]);
120+ kvfree(pages);
121+ return NULL;
122+}
123+
124+static void bpf_ringbuf_notify(struct irq_work *work)
125+{
126+ struct bpf_ringbuf *rb = container_of(work, struct bpf_ringbuf, work);
127+
128+ wake_up_all(&rb->waitq);
129+}
130+
131+static struct bpf_ringbuf *bpf_ringbuf_alloc(size_t data_sz, int numa_node)
132+{
133+ struct bpf_ringbuf *rb;
134+
135+ if (!data_sz || !PAGE_ALIGNED(data_sz))
136+ return ERR_PTR(-EINVAL);
137+
138+#ifdef CONFIG_64BIT
139+ /* on 32-bit arch, it's impossible to overflow record's hdr->pgoff */
140+ if (data_sz > RINGBUF_MAX_DATA_SZ)
141+ return ERR_PTR(-E2BIG);
142+#endif
143+
144+ rb = bpf_ringbuf_area_alloc(data_sz, numa_node);
145+ if (!rb)
146+ return ERR_PTR(-ENOMEM);
147+
148+ spin_lock_init(&rb->spinlock);
149+ init_waitqueue_head(&rb->waitq);
150+ init_irq_work(&rb->work, bpf_ringbuf_notify);
151+
152+ rb->mask = data_sz - 1;
153+ rb->consumer_pos = 0;
154+ rb->producer_pos = 0;
155+
156+ return rb;
157+}
158+
159+static struct bpf_map *ringbuf_map_alloc(union bpf_attr *attr)
160+{
161+ struct bpf_ringbuf_map *rb_map;
162+ u64 cost;
163+ int err;
164+
165+ if (attr->map_flags & ~RINGBUF_CREATE_FLAG_MASK)
166+ return ERR_PTR(-EINVAL);
167+
168+ if (attr->key_size || attr->value_size ||
169+ attr->max_entries == 0 || !PAGE_ALIGNED(attr->max_entries))
170+ return ERR_PTR(-EINVAL);
171+
172+ rb_map = kzalloc(sizeof(*rb_map), GFP_USER);
173+ if (!rb_map)
174+ return ERR_PTR(-ENOMEM);
175+
176+ bpf_map_init_from_attr(&rb_map->map, attr);
177+
178+ cost = sizeof(struct bpf_ringbuf_map) +
179+ sizeof(struct bpf_ringbuf) +
180+ attr->max_entries;
181+ err = bpf_map_charge_init(&rb_map->map.memory, cost);
182+ if (err)
183+ goto err_free_map;
184+
185+ rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, rb_map->map.numa_node);
186+ if (IS_ERR(rb_map->rb)) {
187+ err = PTR_ERR(rb_map->rb);
188+ goto err_uncharge;
189+ }
190+
191+ return &rb_map->map;
192+
193+err_uncharge:
194+ bpf_map_charge_finish(&rb_map->map.memory);
195+err_free_map:
196+ kfree(rb_map);
197+ return ERR_PTR(err);
198+}
199+
200+static void bpf_ringbuf_free(struct bpf_ringbuf *rb)
201+{
202+ /* copy pages pointer and nr_pages to local variable, as we are going
203+ * to unmap rb itself with vunmap() below
204+ */
205+ struct page **pages = rb->pages;
206+ int i, nr_pages = rb->nr_pages;
207+
208+ vunmap(rb);
209+ for (i = 0; i < nr_pages; i++)
210+ __free_page(pages[i]);
211+ kvfree(pages);
212+}
213+
214+static void ringbuf_map_free(struct bpf_map *map)
215+{
216+ struct bpf_ringbuf_map *rb_map;
217+
218+ /* at this point bpf_prog->aux->refcnt == 0 and this map->refcnt == 0,
219+ * so the programs (can be more than one that used this map) were
220+ * disconnected from events. Wait for outstanding critical sections in
221+ * these programs to complete
222+ */
223+ synchronize_rcu();
224+
225+ rb_map = container_of(map, struct bpf_ringbuf_map, map);
226+ bpf_ringbuf_free(rb_map->rb);
227+ kfree(rb_map);
228+}
229+
230+static void *ringbuf_map_lookup_elem(struct bpf_map *map, void *key)
231+{
232+ return ERR_PTR(-ENOTSUPP);
233+}
234+
235+static int ringbuf_map_update_elem(struct bpf_map *map, void *key, void *value,
236+ u64 flags)
237+{
238+ return -ENOTSUPP;
239+}
240+
241+static int ringbuf_map_delete_elem(struct bpf_map *map, void *key)
242+{
243+ return -ENOTSUPP;
244+}
245+
246+static int ringbuf_map_get_next_key(struct bpf_map *map, void *key,
247+ void *next_key)
248+{
249+ return -ENOTSUPP;
250+}
251+
252+static size_t bpf_ringbuf_mmap_page_cnt(const struct bpf_ringbuf *rb)
253+{
254+ size_t data_pages = (rb->mask + 1) >> PAGE_SHIFT;
255+
256+ /* consumer page + producer page + 2 x data pages */
257+ return RINGBUF_POS_PAGES + 2 * data_pages;
258+}
259+
260+static int ringbuf_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
261+{
262+ struct bpf_ringbuf_map *rb_map;
263+ size_t mmap_sz;
264+
265+ rb_map = container_of(map, struct bpf_ringbuf_map, map);
266+ mmap_sz = bpf_ringbuf_mmap_page_cnt(rb_map->rb) << PAGE_SHIFT;
267+
268+ if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) > mmap_sz)
269+ return -EINVAL;
270+
271+ return remap_vmalloc_range(vma, rb_map->rb,
272+ vma->vm_pgoff + RINGBUF_PGOFF);
273+}
274+
275+static unsigned long ringbuf_avail_data_sz(struct bpf_ringbuf *rb)
276+{
277+ unsigned long cons_pos, prod_pos;
278+
279+ cons_pos = smp_load_acquire(&rb->consumer_pos);
280+ prod_pos = smp_load_acquire(&rb->producer_pos);
281+ return prod_pos - cons_pos;
282+}
283+
284+static __poll_t ringbuf_map_poll(struct bpf_map *map, struct file *filp,
285+ struct poll_table_struct *pts)
286+{
287+ struct bpf_ringbuf_map *rb_map;
288+
289+ rb_map = container_of(map, struct bpf_ringbuf_map, map);
290+ poll_wait(filp, &rb_map->rb->waitq, pts);
291+
292+ if (ringbuf_avail_data_sz(rb_map->rb))
293+ return EPOLLIN | EPOLLRDNORM;
294+ return 0;
295+}
296+
297+const struct bpf_map_ops ringbuf_map_ops = {
298+ .map_alloc = ringbuf_map_alloc,
299+ .map_free = ringbuf_map_free,
300+ .map_mmap = ringbuf_map_mmap,
301+ .map_poll = ringbuf_map_poll,
302+ .map_lookup_elem = ringbuf_map_lookup_elem,
303+ .map_update_elem = ringbuf_map_update_elem,
304+ .map_delete_elem = ringbuf_map_delete_elem,
305+ .map_get_next_key = ringbuf_map_get_next_key,
306+};
307+
308+/* Given pointer to ring buffer record metadata and struct bpf_ringbuf itself,
309+ * calculate offset from record metadata to ring buffer in pages, rounded
310+ * down. This page offset is stored as part of record metadata and allows to
311+ * restore struct bpf_ringbuf * from record pointer. This page offset is
312+ * stored at offset 4 of record metadata header.
313+ */
314+static size_t bpf_ringbuf_rec_pg_off(struct bpf_ringbuf *rb,
315+ struct bpf_ringbuf_hdr *hdr)
316+{
317+ return ((void *)hdr - (void *)rb) >> PAGE_SHIFT;
318+}
319+
320+/* Given pointer to ring buffer record header, restore pointer to struct
321+ * bpf_ringbuf itself by using page offset stored at offset 4
322+ */
323+static struct bpf_ringbuf *
324+bpf_ringbuf_restore_from_rec(struct bpf_ringbuf_hdr *hdr)
325+{
326+ unsigned long addr = (unsigned long)(void *)hdr;
327+ unsigned long off = (unsigned long)hdr->pg_off << PAGE_SHIFT;
328+
329+ return (void*)((addr & PAGE_MASK) - off);
330+}
331+
332+static void *__bpf_ringbuf_reserve(struct bpf_ringbuf *rb, u64 size)
333+{
334+ unsigned long cons_pos, prod_pos, new_prod_pos, flags;
335+ u32 len, pg_off;
336+ struct bpf_ringbuf_hdr *hdr;
337+
338+ if (unlikely(size > RINGBUF_MAX_RECORD_SZ))
339+ return NULL;
340+
341+ len = round_up(size + BPF_RINGBUF_HDR_SZ, 8);
342+ cons_pos = smp_load_acquire(&rb->consumer_pos);
343+
344+ if (in_nmi()) {
345+ if (!spin_trylock_irqsave(&rb->spinlock, flags))
346+ return NULL;
347+ } else {
348+ spin_lock_irqsave(&rb->spinlock, flags);
349+ }
350+
351+ prod_pos = rb->producer_pos;
352+ new_prod_pos = prod_pos + len;
353+
354+ /* check for out of ringbuf space by ensuring producer position
355+ * doesn't advance more than (ringbuf_size - 1) ahead
356+ */
357+ if (new_prod_pos - cons_pos > rb->mask) {
358+ spin_unlock_irqrestore(&rb->spinlock, flags);
359+ return NULL;
360+ }
361+
362+ hdr = (void *)rb->data + (prod_pos & rb->mask);
363+ pg_off = bpf_ringbuf_rec_pg_off(rb, hdr);
364+ hdr->len = size | BPF_RINGBUF_BUSY_BIT;
365+ hdr->pg_off = pg_off;
366+
367+ /* pairs with consumer's smp_load_acquire() */
368+ smp_store_release(&rb->producer_pos, new_prod_pos);
369+
370+ spin_unlock_irqrestore(&rb->spinlock, flags);
371+
372+ return (void *)hdr + BPF_RINGBUF_HDR_SZ;
373+}
374+
375+BPF_CALL_3(bpf_ringbuf_reserve, struct bpf_map *, map, u64, size, u64, flags)
376+{
377+ struct bpf_ringbuf_map *rb_map;
378+
379+ if (unlikely(flags))
380+ return 0;
381+
382+ rb_map = container_of(map, struct bpf_ringbuf_map, map);
383+ return (unsigned long)__bpf_ringbuf_reserve(rb_map->rb, size);
384+}
385+
386+const struct bpf_func_proto bpf_ringbuf_reserve_proto = {
387+ .func = bpf_ringbuf_reserve,
388+ .ret_type = RET_PTR_TO_ALLOC_MEM_OR_NULL,
389+ .arg1_type = ARG_CONST_MAP_PTR,
390+ .arg2_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
391+ .arg3_type = ARG_ANYTHING,
392+};
393+
394+static void bpf_ringbuf_commit(void *sample, u64 flags, bool discard)
395+{
396+ unsigned long rec_pos, cons_pos;
397+ struct bpf_ringbuf_hdr *hdr;
398+ struct bpf_ringbuf *rb;
399+ u32 new_len;
400+
401+ hdr = sample - BPF_RINGBUF_HDR_SZ;
402+ rb = bpf_ringbuf_restore_from_rec(hdr);
403+ new_len = hdr->len ^ BPF_RINGBUF_BUSY_BIT;
404+ if (discard)
405+ new_len |= BPF_RINGBUF_DISCARD_BIT;
406+
407+ /* update record header with correct final size prefix */
408+ xchg(&hdr->len, new_len);
409+
410+ /* if consumer caught up and is waiting for our record, notify about
411+ * new data availability
412+ */
413+ rec_pos = (void *)hdr - (void *)rb->data;
414+ cons_pos = smp_load_acquire(&rb->consumer_pos) & rb->mask;
415+
416+ if (flags & BPF_RB_FORCE_WAKEUP)
417+ irq_work_queue(&rb->work);
418+ else if (cons_pos == rec_pos && !(flags & BPF_RB_NO_WAKEUP))
419+ irq_work_queue(&rb->work);
420+}
421+
422+BPF_CALL_2(bpf_ringbuf_submit, void *, sample, u64, flags)
423+{
424+ bpf_ringbuf_commit(sample, flags, false /* discard */);
425+ return 0;
426+}
427+
428+const struct bpf_func_proto bpf_ringbuf_submit_proto = {
429+ .func = bpf_ringbuf_submit,
430+ .ret_type = RET_VOID,
431+ .arg1_type = ARG_PTR_TO_ALLOC_MEM,
432+ .arg2_type = ARG_ANYTHING,
433+};
434+
435+BPF_CALL_2(bpf_ringbuf_discard, void *, sample, u64, flags)
436+{
437+ bpf_ringbuf_commit(sample, flags, true /* discard */);
438+ return 0;
439+}
440+
441+const struct bpf_func_proto bpf_ringbuf_discard_proto = {
442+ .func = bpf_ringbuf_discard,
443+ .ret_type = RET_VOID,
444+ .arg1_type = ARG_PTR_TO_ALLOC_MEM,
445+ .arg2_type = ARG_ANYTHING,
446+};
447+
448+BPF_CALL_4(bpf_ringbuf_output, struct bpf_map *, map, void *, data, u64, size,
449+ u64, flags)
450+{
451+ struct bpf_ringbuf_map *rb_map;
452+ void *rec;
453+
454+ if (unlikely(flags & ~(BPF_RB_NO_WAKEUP | BPF_RB_FORCE_WAKEUP)))
455+ return -EINVAL;
456+
457+ rb_map = container_of(map, struct bpf_ringbuf_map, map);
458+ rec = __bpf_ringbuf_reserve(rb_map->rb, size);
459+ if (!rec)
460+ return -EAGAIN;
461+
462+ memcpy(rec, data, size);
463+ bpf_ringbuf_commit(rec, flags, false /* discard */);
464+ return 0;
465+}
466+
467+const struct bpf_func_proto bpf_ringbuf_output_proto = {
468+ .func = bpf_ringbuf_output,
469+ .ret_type = RET_INTEGER,
470+ .arg1_type = ARG_CONST_MAP_PTR,
471+ .arg2_type = ARG_PTR_TO_MEM,
472+ .arg3_type = ARG_CONST_SIZE_OR_ZERO,
473+ .arg4_type = ARG_ANYTHING,
474+};
475+
476+BPF_CALL_2(bpf_ringbuf_query, struct bpf_map *, map, u64, flags)
477+{
478+ struct bpf_ringbuf *rb;
479+
480+ rb = container_of(map, struct bpf_ringbuf_map, map)->rb;
481+
482+ switch (flags) {
483+ case BPF_RB_AVAIL_DATA:
484+ return ringbuf_avail_data_sz(rb);
485+ case BPF_RB_RING_SIZE:
486+ return rb->mask + 1;
487+ case BPF_RB_CONS_POS:
488+ return smp_load_acquire(&rb->consumer_pos);
489+ case BPF_RB_PROD_POS:
490+ return smp_load_acquire(&rb->producer_pos);
491+ default:
492+ return 0;
493+ }
494+}
495+
496+const struct bpf_func_proto bpf_ringbuf_query_proto = {
497+ .func = bpf_ringbuf_query,
498+ .ret_type = RET_INTEGER,
499+ .arg1_type = ARG_CONST_MAP_PTR,
500+ .arg2_type = ARG_ANYTHING,
501+};
kernel/bpf/stackmap.c+1−1
@@ -93,7 +93,7 @@ static struct bpf_map *stack_map_alloc(union bpf_attr *attr) · 92 unmodified lines
9393 u64 cost, n_buckets;
9494 int err;
9595
96− if (!capable(CAP_SYS_ADMIN))
96+ if (!bpf_capable())
9797 return ERR_PTR(-EPERM);
9898
9999 if (attr->map_flags & ~STACK_CREATE_FLAG_MASK)
kernel/bpf/syscall.c+503−94
@@ -27,6 +27,8 @@ · 26 unmodified lines
2727 #include <uapi/linux/btf.h>
2828 #include <asm/pgtable.h>
2929 #include <linux/bpf_lsm.h>
30+#include <linux/poll.h>
31+#include <linux/bpf-netns.h>
3032
3133 #define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \
3234 (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \
@@ -43,6 +45,8 @@ static DEFINE_IDR(prog_idr); · 10 unmodified lines
4345 static DEFINE_SPINLOCK(prog_idr_lock);
4446 static DEFINE_IDR(map_idr);
4547 static DEFINE_SPINLOCK(map_idr_lock);
48+static DEFINE_IDR(link_idr);
49+static DEFINE_SPINLOCK(link_idr_lock);
4650
4751 int sysctl_unprivileged_bpf_disabled __read_mostly;
4852
@@ -50,9 +54,11 @@ static const struct bpf_map_ops * const bpf_map_types[] = { · 1 unmodified line
5054 #define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type)
5155 #define BPF_MAP_TYPE(_id, _ops) \
5256 [_id] = &_ops,
57+#define BPF_LINK_TYPE(_id, _name)
5358 #include <linux/bpf_types.h>
5459 #undef BPF_PROG_TYPE
5560 #undef BPF_MAP_TYPE
61+#undef BPF_LINK_TYPE
5662 };
5763
5864 /*
@@ -661,6 +667,16 @@ out: · 602 unmodified lines
661667 return err;
662668 }
663669
670+static __poll_t bpf_map_poll(struct file *filp, struct poll_table_struct *pts)
671+{
672+ struct bpf_map *map = filp->private_data;
673+
674+ if (map->ops->map_poll)
675+ return map->ops->map_poll(map, filp, pts);
676+
677+ return EPOLLERR;
678+}
679+
664680 const struct file_operations bpf_map_fops = {
665681 #ifdef CONFIG_PROC_FS
666682 .show_fdinfo = bpf_map_show_fdinfo,
@@ -669,6 +685,7 @@ const struct file_operations bpf_map_fops = { · 2 unmodified lines
669685 .read = bpf_dummy_read,
670686 .write = bpf_dummy_write,
671687 .mmap = bpf_map_mmap,
688+ .poll = bpf_map_poll,
672689 };
673690
674691 int bpf_map_new_fd(struct bpf_map *map, int flags)
@@ -1386,7 +1403,7 @@ int generic_map_lookup_batch(struct bpf_map *map, · 711 unmodified lines
13861403
13871404 buf = kmalloc(map->key_size + value_size, GFP_USER | __GFP_NOWARN);
13881405 if (!buf) {
1389− kvfree(buf_prevkey);
1406+ kfree(buf_prevkey);
13901407 return -ENOMEM;
13911408 }
13921409
@@ -1471,7 +1488,8 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr) · 78 unmodified lines
14711488 map = __bpf_map_get(f);
14721489 if (IS_ERR(map))
14731490 return PTR_ERR(map);
1474− if (!(map_get_sys_perms(map, f) & FMODE_CAN_WRITE)) {
1491+ if (!(map_get_sys_perms(map, f) & FMODE_CAN_READ) ||
1492+ !(map_get_sys_perms(map, f) & FMODE_CAN_WRITE)) {
14751493 err = -EPERM;
14761494 goto err_put;
14771495 }
@@ -1546,7 +1564,7 @@ static int map_freeze(const union bpf_attr *attr) · 68 unmodified lines
15461564 err = -EBUSY;
15471565 goto err_put;
15481566 }
1549− if (!capable(CAP_SYS_ADMIN)) {
1567+ if (!bpf_capable()) {
15501568 err = -EPERM;
15511569 goto err_put;
15521570 }
@@ -1562,9 +1580,11 @@ static const struct bpf_prog_ops * const bpf_prog_types[] = { · 9 unmodified lines
15621580 #define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
15631581 [_id] = & _name ## _prog_ops,
15641582 #define BPF_MAP_TYPE(_id, _ops)
1583+#define BPF_LINK_TYPE(_id, _name)
15651584 #include <linux/bpf_types.h>
15661585 #undef BPF_PROG_TYPE
15671586 #undef BPF_MAP_TYPE
1587+#undef BPF_LINK_TYPE
15681588 };
15691589
15701590 static int find_prog_type(enum bpf_prog_type type, struct bpf_prog *prog)
@@ -1986,6 +2006,10 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type, · 415 unmodified lines
19862006 case BPF_CGROUP_INET6_BIND:
19872007 case BPF_CGROUP_INET4_CONNECT:
19882008 case BPF_CGROUP_INET6_CONNECT:
2009+ case BPF_CGROUP_INET4_GETPEERNAME:
2010+ case BPF_CGROUP_INET6_GETPEERNAME:
2011+ case BPF_CGROUP_INET4_GETSOCKNAME:
2012+ case BPF_CGROUP_INET6_GETSOCKNAME:
19892013 case BPF_CGROUP_UDP4_SENDMSG:
19902014 case BPF_CGROUP_UDP6_SENDMSG:
19912015 case BPF_CGROUP_UDP4_RECVMSG:
@@ -2019,6 +2043,55 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type, · 27 unmodified lines
20192043 }
20202044 }
20212045
2046+static bool is_net_admin_prog_type(enum bpf_prog_type prog_type)
2047+{
2048+ switch (prog_type) {
2049+ case BPF_PROG_TYPE_SCHED_CLS:
2050+ case BPF_PROG_TYPE_SCHED_ACT:
2051+ case BPF_PROG_TYPE_XDP:
2052+ case BPF_PROG_TYPE_LWT_IN:
2053+ case BPF_PROG_TYPE_LWT_OUT:
2054+ case BPF_PROG_TYPE_LWT_XMIT:
2055+ case BPF_PROG_TYPE_LWT_SEG6LOCAL:
2056+ case BPF_PROG_TYPE_SK_SKB:
2057+ case BPF_PROG_TYPE_SK_MSG:
2058+ case BPF_PROG_TYPE_LIRC_MODE2:
2059+ case BPF_PROG_TYPE_FLOW_DISSECTOR:
2060+ case BPF_PROG_TYPE_CGROUP_DEVICE:
2061+ case BPF_PROG_TYPE_CGROUP_SOCK:
2062+ case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
2063+ case BPF_PROG_TYPE_CGROUP_SOCKOPT:
2064+ case BPF_PROG_TYPE_CGROUP_SYSCTL:
2065+ case BPF_PROG_TYPE_SOCK_OPS:
2066+ case BPF_PROG_TYPE_EXT: /* extends any prog */
2067+ return true;
2068+ case BPF_PROG_TYPE_CGROUP_SKB:
2069+ /* always unpriv */
2070+ case BPF_PROG_TYPE_SK_REUSEPORT:
2071+ /* equivalent to SOCKET_FILTER. need CAP_BPF only */
2072+ default:
2073+ return false;
2074+ }
2075+}
2076+
2077+static bool is_perfmon_prog_type(enum bpf_prog_type prog_type)
2078+{
2079+ switch (prog_type) {
2080+ case BPF_PROG_TYPE_KPROBE:
2081+ case BPF_PROG_TYPE_TRACEPOINT:
2082+ case BPF_PROG_TYPE_PERF_EVENT:
2083+ case BPF_PROG_TYPE_RAW_TRACEPOINT:
2084+ case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE:
2085+ case BPF_PROG_TYPE_TRACING:
2086+ case BPF_PROG_TYPE_LSM:
2087+ case BPF_PROG_TYPE_STRUCT_OPS: /* has access to struct sock */
2088+ case BPF_PROG_TYPE_EXT: /* extends any prog */
2089+ return true;
2090+ default:
2091+ return false;
2092+ }
2093+}
2094+
20222095 /* last field in 'union bpf_attr' used by this command */
20232096 #define BPF_PROG_LOAD_LAST_FIELD attach_prog_fd
20242097
@@ -2041,7 +2114,7 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) · 16 unmodified lines
20412114
20422115 if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
20432116 (attr->prog_flags & BPF_F_ANY_ALIGNMENT) &&
2044− !capable(CAP_SYS_ADMIN))
2117+ !bpf_capable())
20452118 return -EPERM;
20462119
20472120 /* copy eBPF program license from user space */
@@ -2054,11 +2127,16 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) · 6 unmodified lines
20542127 is_gpl = license_is_gpl_compatible(license);
20552128
20562129 if (attr->insn_cnt == 0 ||
2057− attr->insn_cnt > (capable(CAP_SYS_ADMIN) ? BPF_COMPLEXITY_LIMIT_INSNS : BPF_MAXINSNS))
2130+ attr->insn_cnt > (bpf_capable() ? BPF_COMPLEXITY_LIMIT_INSNS : BPF_MAXINSNS))
20582131 return -E2BIG;
20592132 if (type != BPF_PROG_TYPE_SOCKET_FILTER &&
20602133 type != BPF_PROG_TYPE_CGROUP_SKB &&
2061− !capable(CAP_SYS_ADMIN))
2134+ !bpf_capable())
2135+ return -EPERM;
2136+
2137+ if (is_net_admin_prog_type(type) && !capable(CAP_NET_ADMIN))
2138+ return -EPERM;
2139+ if (is_perfmon_prog_type(type) && !perfmon_capable())
20622140 return -EPERM;
20632141
20642142 bpf_prog_load_fixup_attach_type(attr);
@@ -2197,25 +2275,39 @@ static int bpf_obj_get(const union bpf_attr *attr) · 132 unmodified lines
21972275 attr->file_flags);
21982276 }
21992277
2200−void bpf_link_init(struct bpf_link *link, const struct bpf_link_ops *ops,
2201− struct bpf_prog *prog)
2278+void bpf_link_init(struct bpf_link *link, enum bpf_link_type type,
2279+ const struct bpf_link_ops *ops, struct bpf_prog *prog)
22022280 {
22032281 atomic64_set(&link->refcnt, 1);
2282+ link->type = type;
2283+ link->id = 0;
22042284 link->ops = ops;
22052285 link->prog = prog;
22062286 }
22072287
2288+static void bpf_link_free_id(int id)
2289+{
2290+ if (!id)
2291+ return;
2292+
2293+ spin_lock_bh(&link_idr_lock);
2294+ idr_remove(&link_idr, id);
2295+ spin_unlock_bh(&link_idr_lock);
2296+}
2297+
22082298 /* Clean up bpf_link and corresponding anon_inode file and FD. After
22092299 * anon_inode is created, bpf_link can't be just kfree()'d due to deferred
2210− * anon_inode's release() call. This helper manages marking bpf_link as
2211− * defunct, releases anon_inode file and puts reserved FD.
2300+ * anon_inode's release() call. This helper marksbpf_link as
2301+ * defunct, releases anon_inode file and puts reserved FD. bpf_prog's refcnt
2302+ * is not decremented, it's the responsibility of a calling code that failed
2303+ * to complete bpf_link initialization.
22122304 */
2213−void bpf_link_cleanup(struct bpf_link *link, struct file *link_file,
2214− int link_fd)
2305+void bpf_link_cleanup(struct bpf_link_primer *primer)
22152306 {
2216− link->prog = NULL;
2217− fput(link_file);
2218− put_unused_fd(link_fd);
2307+ primer->link->prog = NULL;
2308+ bpf_link_free_id(primer->id);
2309+ fput(primer->file);
2310+ put_unused_fd(primer->fd);
22192311 }
22202312
22212313 void bpf_link_inc(struct bpf_link *link)
@@ -2226,6 +2318,7 @@ void bpf_link_inc(struct bpf_link *link) · 4 unmodified lines
22262318 /* bpf_link_free is guaranteed to be called from process context */
22272319 static void bpf_link_free(struct bpf_link *link)
22282320 {
2321+ bpf_link_free_id(link->id);
22292322 if (link->prog) {
22302323 /* detach BPF program, clean up used resources */
22312324 link->ops->release(link);
@@ -2267,35 +2360,35 @@ static int bpf_link_release(struct inode *inode, struct file *filp) · 35 unmodified lines
22672360 }
22682361
22692362 #ifdef CONFIG_PROC_FS
2270−static const struct bpf_link_ops bpf_raw_tp_lops;
2271−static const struct bpf_link_ops bpf_tracing_link_lops;
2363+#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type)
2364+#define BPF_MAP_TYPE(_id, _ops)
2365+#define BPF_LINK_TYPE(_id, _name) [_id] = #_name,
2366+static const char *bpf_link_type_strs[] = {
2367+ [BPF_LINK_TYPE_UNSPEC] = "<invalid>",
2368+#include <linux/bpf_types.h>
2369+};
2370+#undef BPF_PROG_TYPE
2371+#undef BPF_MAP_TYPE
2372+#undef BPF_LINK_TYPE
22722373
22732374 static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp)
22742375 {
22752376 const struct bpf_link *link = filp->private_data;
22762377 const struct bpf_prog *prog = link->prog;
22772378 char prog_tag[sizeof(prog->tag) * 2 + 1] = { };
2278− const char *link_type;
2279−
2280− if (link->ops == &bpf_raw_tp_lops)
2281− link_type = "raw_tracepoint";
2282− else if (link->ops == &bpf_tracing_link_lops)
2283− link_type = "tracing";
2284−#ifdef CONFIG_CGROUP_BPF
2285− else if (link->ops == &bpf_cgroup_link_lops)
2286− link_type = "cgroup";
2287−#endif
2288− else
2289− link_type = "unknown";
22902379
22912380 bin2hex(prog_tag, prog->tag, sizeof(prog->tag));
22922381 seq_printf(m,
22932382 "link_type:\t%s\n"
2383+ "link_id:\t%u\n"
22942384 "prog_tag:\t%s\n"
22952385 "prog_id:\t%u\n",
2296− link_type,
2386+ bpf_link_type_strs[link->type],
2387+ link->id,
22972388 prog_tag,
22982389 prog->aux->id);
2390+ if (link->ops->show_fdinfo)
2391+ link->ops->show_fdinfo(link, m);
22992392 }
23002393 #endif
23012394
@@ -2308,36 +2401,77 @@ static const struct file_operations bpf_link_fops = { · 6 unmodified lines
23082401 .write = bpf_dummy_write,
23092402 };
23102403
2311−int bpf_link_new_fd(struct bpf_link *link)
2404+static int bpf_link_alloc_id(struct bpf_link *link)
23122405 {
2313− return anon_inode_getfd("bpf-link", &bpf_link_fops, link, O_CLOEXEC);
2314−}
2406+ int id;
23152407
2316−/* Similar to bpf_link_new_fd, create anon_inode for given bpf_link, but
2317− * instead of immediately installing fd in fdtable, just reserve it and
2318− * return. Caller then need to either install it with fd_install(fd, file) or
2319− * release with put_unused_fd(fd).
2320− * This is useful for cases when bpf_link attachment/detachment are
2321− * complicated and expensive operations and should be delayed until all the fd
2322− * reservation and anon_inode creation succeeds.
2408+ idr_preload(GFP_KERNEL);
2409+ spin_lock_bh(&link_idr_lock);
2410+ id = idr_alloc_cyclic(&link_idr, link, 1, INT_MAX, GFP_ATOMIC);
2411+ spin_unlock_bh(&link_idr_lock);
2412+ idr_preload_end();
2413+
2414+ return id;
2415+}
2416+
2417+/* Prepare bpf_link to be exposed to user-space by allocating anon_inode file,
2418+ * reserving unused FD and allocating ID from link_idr. This is to be paired
2419+ * with bpf_link_settle() to install FD and ID and expose bpf_link to
2420+ * user-space, if bpf_link is successfully attached. If not, bpf_link and
2421+ * pre-allocated resources are to be freed with bpf_cleanup() call. All the
2422+ * transient state is passed around in struct bpf_link_primer.
2423+ * This is preferred way to create and initialize bpf_link, especially when
2424+ * there are complicated and expensive operations inbetween creating bpf_link
2425+ * itself and attaching it to BPF hook. By using bpf_link_prime() and
2426+ * bpf_link_settle() kernel code using bpf_link doesn't have to perform
2427+ * expensive (and potentially failing) roll back operations in a rare case
2428+ * that file, FD, or ID can't be allocated.
23232429 */
2324−struct file *bpf_link_new_file(struct bpf_link *link, int *reserved_fd)
2430+int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer)
23252431 {
23262432 struct file *file;
2327− int fd;
2433+ int fd, id;
23282434
23292435 fd = get_unused_fd_flags(O_CLOEXEC);
23302436 if (fd < 0)
2331− return ERR_PTR(fd);
2437+ return fd;
2438+
2439+
2440+ id = bpf_link_alloc_id(link);
2441+ if (id < 0) {
2442+ put_unused_fd(fd);
2443+ return id;
2444+ }
23322445
23332446 file = anon_inode_getfile("bpf_link", &bpf_link_fops, link, O_CLOEXEC);
23342447 if (IS_ERR(file)) {
2448+ bpf_link_free_id(id);
23352449 put_unused_fd(fd);
2336− return file;
2450+ return PTR_ERR(file);
23372451 }
23382452
2339− *reserved_fd = fd;
2340− return file;
2453+ primer->link = link;
2454+ primer->file = file;
2455+ primer->fd = fd;
2456+ primer->id = id;
2457+ return 0;
2458+}
2459+
2460+int bpf_link_settle(struct bpf_link_primer *primer)
2461+{
2462+ /* make bpf_link fetchable by ID */
2463+ spin_lock_bh(&link_idr_lock);
2464+ primer->link->id = primer->id;
2465+ spin_unlock_bh(&link_idr_lock);
2466+ /* make bpf_link fetchable by FD */
2467+ fd_install(primer->fd, primer->file);
2468+ /* pass through installed FD */
2469+ return primer->fd;
2470+}
2471+
2472+int bpf_link_new_fd(struct bpf_link *link)
2473+{
2474+ return anon_inode_getfd("bpf-link", &bpf_link_fops, link, O_CLOEXEC);
23412475 }
23422476
23432477 struct bpf_link *bpf_link_get_from_fd(u32 ufd)
@@ -2361,6 +2495,7 @@ struct bpf_link *bpf_link_get_from_fd(u32 ufd) · 17 unmodified lines
23612495
23622496 struct bpf_tracing_link {
23632497 struct bpf_link link;
2498+ enum bpf_attach_type attach_type;
23642499 };
23652500
23662501 static void bpf_tracing_link_release(struct bpf_link *link)
@@ -2376,16 +2511,40 @@ static void bpf_tracing_link_dealloc(struct bpf_link *link) · 9 unmodified lines
23762511 kfree(tr_link);
23772512 }
23782513
2514+static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link,
2515+ struct seq_file *seq)
2516+{
2517+ struct bpf_tracing_link *tr_link =
2518+ container_of(link, struct bpf_tracing_link, link);
2519+
2520+ seq_printf(seq,
2521+ "attach_type:\t%d\n",
2522+ tr_link->attach_type);
2523+}
2524+
2525+static int bpf_tracing_link_fill_link_info(const struct bpf_link *link,
2526+ struct bpf_link_info *info)
2527+{
2528+ struct bpf_tracing_link *tr_link =
2529+ container_of(link, struct bpf_tracing_link, link);
2530+
2531+ info->tracing.attach_type = tr_link->attach_type;
2532+
2533+ return 0;
2534+}
2535+
23792536 static const struct bpf_link_ops bpf_tracing_link_lops = {
23802537 .release = bpf_tracing_link_release,
23812538 .dealloc = bpf_tracing_link_dealloc,
2539+ .show_fdinfo = bpf_tracing_link_show_fdinfo,
2540+ .fill_link_info = bpf_tracing_link_fill_link_info,
23822541 };
23832542
23842543 static int bpf_tracing_prog_attach(struct bpf_prog *prog)
23852544 {
2545+ struct bpf_link_primer link_primer;
23862546 struct bpf_tracing_link *link;
2387− struct file *link_file;
2388− int link_fd, err;
2547+ int err;
23892548
23902549 switch (prog->type) {
23912550 case BPF_PROG_TYPE_TRACING:
@@ -2418,24 +2577,23 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog) · 26 unmodified lines
24182577 err = -ENOMEM;
24192578 goto out_put_prog;
24202579 }
2421− bpf_link_init(&link->link, &bpf_tracing_link_lops, prog);
2580+ bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING,
2581+ &bpf_tracing_link_lops, prog);
2582+ link->attach_type = prog->expected_attach_type;
24222583
2423− link_file = bpf_link_new_file(&link->link, &link_fd);
2424− if (IS_ERR(link_file)) {
2584+ err = bpf_link_prime(&link->link, &link_primer);
2585+ if (err) {
24252586 kfree(link);
2426− err = PTR_ERR(link_file);
24272587 goto out_put_prog;
24282588 }
24292589
24302590 err = bpf_trampoline_link_prog(prog);
24312591 if (err) {
2432− bpf_link_cleanup(&link->link, link_file, link_fd);
2592+ bpf_link_cleanup(&link_primer);
24332593 goto out_put_prog;
24342594 }
24352595
2436− fd_install(link_fd, link_file);
2437− return link_fd;
2438−
2596+ return bpf_link_settle(&link_primer);
24392597 out_put_prog:
24402598 bpf_prog_put(prog);
24412599 return err;
@@ -2463,22 +2621,69 @@ static void bpf_raw_tp_link_dealloc(struct bpf_link *link) · 21 unmodified lines
24632621 kfree(raw_tp);
24642622 }
24652623
2466−static const struct bpf_link_ops bpf_raw_tp_lops = {
2624+static void bpf_raw_tp_link_show_fdinfo(const struct bpf_link *link,
2625+ struct seq_file *seq)
2626+{
2627+ struct bpf_raw_tp_link *raw_tp_link =
2628+ container_of(link, struct bpf_raw_tp_link, link);
2629+
2630+ seq_printf(seq,
2631+ "tp_name:\t%s\n",
2632+ raw_tp_link->btp->tp->name);
2633+}
2634+
2635+static int bpf_raw_tp_link_fill_link_info(const struct bpf_link *link,
2636+ struct bpf_link_info *info)
2637+{
2638+ struct bpf_raw_tp_link *raw_tp_link =
2639+ container_of(link, struct bpf_raw_tp_link, link);
2640+ char __user *ubuf = u64_to_user_ptr(info->raw_tracepoint.tp_name);
2641+ const char *tp_name = raw_tp_link->btp->tp->name;
2642+ u32 ulen = info->raw_tracepoint.tp_name_len;
2643+ size_t tp_len = strlen(tp_name);
2644+
2645+ if (ulen && !ubuf)
2646+ return -EINVAL;
2647+
2648+ info->raw_tracepoint.tp_name_len = tp_len + 1;
2649+
2650+ if (!ubuf)
2651+ return 0;
2652+
2653+ if (ulen >= tp_len + 1) {
2654+ if (copy_to_user(ubuf, tp_name, tp_len + 1))
2655+ return -EFAULT;
2656+ } else {
2657+ char zero = '\0';
2658+
2659+ if (copy_to_user(ubuf, tp_name, ulen - 1))
2660+ return -EFAULT;
2661+ if (put_user(zero, ubuf + ulen - 1))
2662+ return -EFAULT;
2663+ return -ENOSPC;
2664+ }
2665+
2666+ return 0;
2667+}
2668+
2669+static const struct bpf_link_ops bpf_raw_tp_link_lops = {
24672670 .release = bpf_raw_tp_link_release,
24682671 .dealloc = bpf_raw_tp_link_dealloc,
2672+ .show_fdinfo = bpf_raw_tp_link_show_fdinfo,
2673+ .fill_link_info = bpf_raw_tp_link_fill_link_info,
24692674 };
24702675
24712676 #define BPF_RAW_TRACEPOINT_OPEN_LAST_FIELD raw_tracepoint.prog_fd
24722677
24732678 static int bpf_raw_tracepoint_open(const union bpf_attr *attr)
24742679 {
2680+ struct bpf_link_primer link_primer;
24752681 struct bpf_raw_tp_link *link;
24762682 struct bpf_raw_event_map *btp;
2477− struct file *link_file;
24782683 struct bpf_prog *prog;
24792684 const char *tp_name;
24802685 char buf[128];
2481− int link_fd, err;
2686+ int err;
24822687
24832688 if (CHECK_ATTR(BPF_RAW_TRACEPOINT_OPEN))
24842689 return -EINVAL;
@@ -2531,24 +2736,23 @@ static int bpf_raw_tracepoint_open(const union bpf_attr *attr) · 46 unmodified lines
25312736 err = -ENOMEM;
25322737 goto out_put_btp;
25332738 }
2534− bpf_link_init(&link->link, &bpf_raw_tp_lops, prog);
2739+ bpf_link_init(&link->link, BPF_LINK_TYPE_RAW_TRACEPOINT,
2740+ &bpf_raw_tp_link_lops, prog);
25352741 link->btp = btp;
25362742
2537− link_file = bpf_link_new_file(&link->link, &link_fd);
2538− if (IS_ERR(link_file)) {
2743+ err = bpf_link_prime(&link->link, &link_primer);
2744+ if (err) {
25392745 kfree(link);
2540− err = PTR_ERR(link_file);
25412746 goto out_put_btp;
25422747 }
25432748
25442749 err = bpf_probe_register(link->btp, prog);
25452750 if (err) {
2546− bpf_link_cleanup(&link->link, link_file, link_fd);
2751+ bpf_link_cleanup(&link_primer);
25472752 goto out_put_btp;
25482753 }
25492754
2550− fd_install(link_fd, link_file);
2551− return link_fd;
2755+ return bpf_link_settle(&link_primer);
25522756
25532757 out_put_btp:
25542758 bpf_put_raw_tracepoint(btp);
@@ -2566,6 +2770,11 @@ static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog, · 11 unmodified lines
25662770 case BPF_PROG_TYPE_CGROUP_SOCKOPT:
25672771 return attach_type == prog->expected_attach_type ? 0 : -EINVAL;
25682772 case BPF_PROG_TYPE_CGROUP_SKB:
2773+ if (!capable(CAP_NET_ADMIN))
2774+ /* cg-skb progs can be loaded by unpriv user.
2775+ * check permissions at attach time.
2776+ */
2777+ return -EPERM;
25692778 return prog->enforce_expected_attach_type &&
25702779 prog->expected_attach_type != attach_type ?
25712780 -EINVAL : 0;
@@ -2590,6 +2799,10 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) · 18 unmodified lines
25902799 case BPF_CGROUP_INET6_BIND:
25912800 case BPF_CGROUP_INET4_CONNECT:
25922801 case BPF_CGROUP_INET6_CONNECT:
2802+ case BPF_CGROUP_INET4_GETPEERNAME:
2803+ case BPF_CGROUP_INET6_GETPEERNAME:
2804+ case BPF_CGROUP_INET4_GETSOCKNAME:
2805+ case BPF_CGROUP_INET6_GETSOCKNAME:
25932806 case BPF_CGROUP_UDP4_SENDMSG:
25942807 case BPF_CGROUP_UDP6_SENDMSG:
25952808 case BPF_CGROUP_UDP4_RECVMSG:
@@ -2613,6 +2826,8 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) · 17 unmodified lines
26132826 case BPF_CGROUP_GETSOCKOPT:
26142827 case BPF_CGROUP_SETSOCKOPT:
26152828 return BPF_PROG_TYPE_CGROUP_SOCKOPT;
2829+ case BPF_TRACE_ITER:
2830+ return BPF_PROG_TYPE_TRACING;
26162831 default:
26172832 return BPF_PROG_TYPE_UNSPEC;
26182833 }
@@ -2629,9 +2844,6 @@ static int bpf_prog_attach(const union bpf_attr *attr) · 10 unmodified lines
26292844 struct bpf_prog *prog;
26302845 int ret;
26312846
2632− if (!capable(CAP_NET_ADMIN))
2633− return -EPERM;
2634−
26352847 if (CHECK_ATTR(BPF_PROG_ATTACH))
26362848 return -EINVAL;
26372849
@@ -2660,7 +2872,7 @@ static int bpf_prog_attach(const union bpf_attr *attr) · 22 unmodified lines
26602872 ret = lirc_prog_attach(attr, prog);
26612873 break;
26622874 case BPF_PROG_TYPE_FLOW_DISSECTOR:
2663− ret = skb_flow_dissector_bpf_prog_attach(attr, prog);
2875+ ret = netns_bpf_prog_attach(attr, prog);
26642876 break;
26652877 case BPF_PROG_TYPE_CGROUP_DEVICE:
26662878 case BPF_PROG_TYPE_CGROUP_SKB:
@@ -2686,9 +2898,6 @@ static int bpf_prog_detach(const union bpf_attr *attr) · 19 unmodified lines
26862898 {
26872899 enum bpf_prog_type ptype;
26882900
2689− if (!capable(CAP_NET_ADMIN))
2690− return -EPERM;
2691−
26922901 if (CHECK_ATTR(BPF_PROG_DETACH))
26932902 return -EINVAL;
26942903
@@ -2701,7 +2910,9 @@ static int bpf_prog_detach(const union bpf_attr *attr) · 6 unmodified lines
27012910 case BPF_PROG_TYPE_LIRC_MODE2:
27022911 return lirc_prog_detach(attr);
27032912 case BPF_PROG_TYPE_FLOW_DISSECTOR:
2704− return skb_flow_dissector_bpf_prog_detach(attr);
2913+ if (!capable(CAP_NET_ADMIN))
2914+ return -EPERM;
2915+ return netns_bpf_prog_detach(attr);
27052916 case BPF_PROG_TYPE_CGROUP_DEVICE:
27062917 case BPF_PROG_TYPE_CGROUP_SKB:
27072918 case BPF_PROG_TYPE_CGROUP_SOCK:
@@ -2737,6 +2948,10 @@ static int bpf_prog_query(const union bpf_attr *attr, · 29 unmodified lines
27372948 case BPF_CGROUP_INET6_POST_BIND:
27382949 case BPF_CGROUP_INET4_CONNECT:
27392950 case BPF_CGROUP_INET6_CONNECT:
2951+ case BPF_CGROUP_INET4_GETPEERNAME:
2952+ case BPF_CGROUP_INET6_GETPEERNAME:
2953+ case BPF_CGROUP_INET4_GETSOCKNAME:
2954+ case BPF_CGROUP_INET6_GETSOCKNAME:
27402955 case BPF_CGROUP_UDP4_SENDMSG:
27412956 case BPF_CGROUP_UDP6_SENDMSG:
27422957 case BPF_CGROUP_UDP4_RECVMSG:
@@ -2750,7 +2965,7 @@ static int bpf_prog_query(const union bpf_attr *attr, · 7 unmodified lines
27502965 case BPF_LIRC_MODE2:
27512966 return lirc_prog_query(attr, uattr);
27522967 case BPF_FLOW_DISSECTOR:
2753− return skb_flow_dissector_prog_query(attr, uattr);
2968+ return netns_bpf_prog_query(attr, uattr);
27542969 default:
27552970 return -EINVAL;
27562971 }
@@ -2764,8 +2979,6 @@ static int bpf_prog_test_run(const union bpf_attr *attr, · 7 unmodified lines
27642979 struct bpf_prog *prog;
27652980 int ret = -ENOTSUPP;
27662981
2767− if (!capable(CAP_SYS_ADMIN))
2768− return -EPERM;
27692982 if (CHECK_ATTR(BPF_PROG_TEST_RUN))
27702983 return -EINVAL;
27712984
@@ -2816,6 +3029,25 @@ static int bpf_obj_get_next_id(const union bpf_attr *attr, · 44 unmodified lines
28163029 return err;
28173030 }
28183031
3032+struct bpf_map *bpf_map_get_curr_or_next(u32 *id)
3033+{
3034+ struct bpf_map *map;
3035+
3036+ spin_lock_bh(&map_idr_lock);
3037+again:
3038+ map = idr_get_next(&map_idr, id);
3039+ if (map) {
3040+ map = __bpf_map_inc_not_zero(map, false);
3041+ if (IS_ERR(map)) {
3042+ (*id)++;
3043+ goto again;
3044+ }
3045+ }
3046+ spin_unlock_bh(&map_idr_lock);
3047+
3048+ return map;
3049+}
3050+
28193051 #define BPF_PROG_GET_FD_BY_ID_LAST_FIELD prog_id
28203052
28213053 struct bpf_prog *bpf_prog_by_id(u32 id)
@@ -3047,7 +3279,7 @@ static int bpf_prog_get_info_by_fd(struct bpf_prog *prog, · 225 unmodified lines
30473279 info.run_time_ns = stats.nsecs;
30483280 info.run_cnt = stats.cnt;
30493281
3050− if (!capable(CAP_SYS_ADMIN)) {
3282+ if (!bpf_capable()) {
30513283 info.jited_prog_len = 0;
30523284 info.xlated_prog_len = 0;
30533285 info.nr_jited_ksyms = 0;
@@ -3329,6 +3561,42 @@ static int bpf_btf_get_info_by_fd(struct btf *btf, · 275 unmodified lines
33293561 return btf_get_info_by_fd(btf, attr, uattr);
33303562 }
33313563
3564+static int bpf_link_get_info_by_fd(struct bpf_link *link,
3565+ const union bpf_attr *attr,
3566+ union bpf_attr __user *uattr)
3567+{
3568+ struct bpf_link_info __user *uinfo = u64_to_user_ptr(attr->info.info);
3569+ struct bpf_link_info info;
3570+ u32 info_len = attr->info.info_len;
3571+ int err;
3572+
3573+ err = bpf_check_uarg_tail_zero(uinfo, sizeof(info), info_len);
3574+ if (err)
3575+ return err;
3576+ info_len = min_t(u32, sizeof(info), info_len);
3577+
3578+ memset(&info, 0, sizeof(info));
3579+ if (copy_from_user(&info, uinfo, info_len))
3580+ return -EFAULT;
3581+
3582+ info.type = link->type;
3583+ info.id = link->id;
3584+ info.prog_id = link->prog->aux->id;
3585+
3586+ if (link->ops->fill_link_info) {
3587+ err = link->ops->fill_link_info(link, &info);
3588+ if (err)
3589+ return err;
3590+ }
3591+
3592+ if (copy_to_user(uinfo, &info, info_len) ||
3593+ put_user(info_len, &uattr->info.info_len))
3594+ return -EFAULT;
3595+
3596+ return 0;
3597+}
3598+
3599+
33323600 #define BPF_OBJ_GET_INFO_BY_FD_LAST_FIELD info.info
33333601
33343602 static int bpf_obj_get_info_by_fd(const union bpf_attr *attr,
@@ -3353,6 +3621,9 @@ static int bpf_obj_get_info_by_fd(const union bpf_attr *attr, · 18 unmodified lines
33533621 uattr);
33543622 else if (f.file->f_op == &btf_fops)
33553623 err = bpf_btf_get_info_by_fd(f.file->private_data, attr, uattr);
3624+ else if (f.file->f_op == &bpf_link_fops)
3625+ err = bpf_link_get_info_by_fd(f.file->private_data,
3626+ attr, uattr);
33563627 else
33573628 err = -EINVAL;
33583629
@@ -3367,7 +3638,7 @@ static int bpf_btf_load(const union bpf_attr *attr) · 8 unmodified lines
33673638 if (CHECK_ATTR(BPF_BTF_LOAD))
33683639 return -EINVAL;
33693640
3370− if (!capable(CAP_SYS_ADMIN))
3641+ if (!bpf_capable())
33713642 return -EPERM;
33723643
33733644 return btf_new_fd(attr);
@@ -3480,7 +3751,7 @@ static int bpf_task_fd_query(const union bpf_attr *attr, · 106 unmodified lines
34803751 if (file->f_op == &bpf_link_fops) {
34813752 struct bpf_link *link = file->private_data;
34823753
3483− if (link->ops == &bpf_raw_tp_lops) {
3754+ if (link->ops == &bpf_raw_tp_link_lops) {
34843755 struct bpf_raw_tp_link *raw_tp =
34853756 container_of(link, struct bpf_raw_tp_link, link);
34863757 struct bpf_raw_event_map *btp = raw_tp->btp;
@@ -3574,6 +3845,15 @@ err_put: · 87 unmodified lines
35743845 return err;
35753846 }
35763847
3848+static int tracing_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
3849+{
3850+ if (attr->link_create.attach_type == BPF_TRACE_ITER &&
3851+ prog->expected_attach_type == BPF_TRACE_ITER)
3852+ return bpf_iter_link_attach(attr, prog);
3853+
3854+ return -EINVAL;
3855+}
3856+
35773857 #define BPF_LINK_CREATE_LAST_FIELD link_create.flags
35783858 static int link_create(union bpf_attr *attr)
35793859 {
@@ -3581,9 +3861,6 @@ static int link_create(union bpf_attr *attr) · 1 unmodified line
35813861 struct bpf_prog *prog;
35823862 int ret;
35833863
3584− if (!capable(CAP_NET_ADMIN))
3585− return -EPERM;
3586−
35873864 if (CHECK_ATTR(BPF_LINK_CREATE))
35883865 return -EINVAL;
35893866
@@ -3610,6 +3887,12 @@ static int link_create(union bpf_attr *attr) · 20 unmodified lines
36103887 case BPF_PROG_TYPE_CGROUP_SOCKOPT:
36113888 ret = cgroup_bpf_link_attach(attr, prog);
36123889 break;
3890+ case BPF_PROG_TYPE_TRACING:
3891+ ret = tracing_bpf_link_attach(attr, prog);
3892+ break;
3893+ case BPF_PROG_TYPE_FLOW_DISSECTOR:
3894+ ret = netns_bpf_link_create(attr, prog);
3895+ break;
36133896 default:
36143897 ret = -EINVAL;
36153898 }
@@ -3629,9 +3912,6 @@ static int link_update(union bpf_attr *attr) · 13 unmodified lines
36293912 u32 flags;
36303913 int ret;
36313914
3632− if (!capable(CAP_NET_ADMIN))
3633− return -EPERM;
3634−
36353915 if (CHECK_ATTR(BPF_LINK_UPDATE))
36363916 return -EINVAL;
36373917
@@ -3661,13 +3941,10 @@ static int link_update(union bpf_attr *attr) · 23 unmodified lines
36613941 goto out_put_progs;
36623942 }
36633943
3664−#ifdef CONFIG_CGROUP_BPF
3665− if (link->ops == &bpf_cgroup_link_lops) {
3666− ret = cgroup_bpf_replace(link, old_prog, new_prog);
3667− goto out_put_progs;
3668− }
3669−#endif
3670− ret = -EINVAL;
3944+ if (link->ops->update_prog)
3945+ ret = link->ops->update_prog(link, new_prog, old_prog);
3946+ else
3947+ ret = -EINVAL;
36713948
36723949 out_put_progs:
36733950 if (old_prog)
@@ -3679,12 +3956,131 @@ out_put_link: · 5 unmodified lines
36793956 return ret;
36803957 }
36813958
3959+static int bpf_link_inc_not_zero(struct bpf_link *link)
3960+{
3961+ return atomic64_fetch_add_unless(&link->refcnt, 1, 0) ? 0 : -ENOENT;
3962+}
3963+
3964+#define BPF_LINK_GET_FD_BY_ID_LAST_FIELD link_id
3965+
3966+static int bpf_link_get_fd_by_id(const union bpf_attr *attr)
3967+{
3968+ struct bpf_link *link;
3969+ u32 id = attr->link_id;
3970+ int fd, err;
3971+
3972+ if (CHECK_ATTR(BPF_LINK_GET_FD_BY_ID))
3973+ return -EINVAL;
3974+
3975+ if (!capable(CAP_SYS_ADMIN))
3976+ return -EPERM;
3977+
3978+ spin_lock_bh(&link_idr_lock);
3979+ link = idr_find(&link_idr, id);
3980+ /* before link is "settled", ID is 0, pretend it doesn't exist yet */
3981+ if (link) {
3982+ if (link->id)
3983+ err = bpf_link_inc_not_zero(link);
3984+ else
3985+ err = -EAGAIN;
3986+ } else {
3987+ err = -ENOENT;
3988+ }
3989+ spin_unlock_bh(&link_idr_lock);
3990+
3991+ if (err)
3992+ return err;
3993+
3994+ fd = bpf_link_new_fd(link);
3995+ if (fd < 0)
3996+ bpf_link_put(link);
3997+
3998+ return fd;
3999+}
4000+
4001+DEFINE_MUTEX(bpf_stats_enabled_mutex);
4002+
4003+static int bpf_stats_release(struct inode *inode, struct file *file)
4004+{
4005+ mutex_lock(&bpf_stats_enabled_mutex);
4006+ static_key_slow_dec(&bpf_stats_enabled_key.key);
4007+ mutex_unlock(&bpf_stats_enabled_mutex);
4008+ return 0;
4009+}
4010+
4011+static const struct file_operations bpf_stats_fops = {
4012+ .release = bpf_stats_release,
4013+};
4014+
4015+static int bpf_enable_runtime_stats(void)
4016+{
4017+ int fd;
4018+
4019+ mutex_lock(&bpf_stats_enabled_mutex);
4020+
4021+ /* Set a very high limit to avoid overflow */
4022+ if (static_key_count(&bpf_stats_enabled_key.key) > INT_MAX / 2) {
4023+ mutex_unlock(&bpf_stats_enabled_mutex);
4024+ return -EBUSY;
4025+ }
4026+
4027+ fd = anon_inode_getfd("bpf-stats", &bpf_stats_fops, NULL, O_CLOEXEC);
4028+ if (fd >= 0)
4029+ static_key_slow_inc(&bpf_stats_enabled_key.key);
4030+
4031+ mutex_unlock(&bpf_stats_enabled_mutex);
4032+ return fd;
4033+}
4034+
4035+#define BPF_ENABLE_STATS_LAST_FIELD enable_stats.type
4036+
4037+static int bpf_enable_stats(union bpf_attr *attr)
4038+{
4039+
4040+ if (CHECK_ATTR(BPF_ENABLE_STATS))
4041+ return -EINVAL;
4042+
4043+ if (!capable(CAP_SYS_ADMIN))
4044+ return -EPERM;
4045+
4046+ switch (attr->enable_stats.type) {
4047+ case BPF_STATS_RUN_TIME:
4048+ return bpf_enable_runtime_stats();
4049+ default:
4050+ break;
4051+ }
4052+ return -EINVAL;
4053+}
4054+
4055+#define BPF_ITER_CREATE_LAST_FIELD iter_create.flags
4056+
4057+static int bpf_iter_create(union bpf_attr *attr)
4058+{
4059+ struct bpf_link *link;
4060+ int err;
4061+
4062+ if (CHECK_ATTR(BPF_ITER_CREATE))
4063+ return -EINVAL;
4064+
4065+ if (attr->iter_create.flags)
4066+ return -EINVAL;
4067+
4068+ link = bpf_link_get_from_fd(attr->iter_create.link_fd);
4069+ if (IS_ERR(link))
4070+ return PTR_ERR(link);
4071+
4072+ err = bpf_iter_new_fd(link);
4073+ bpf_link_put(link);
4074+
4075+ return err;
4076+}
4077+
36824078 SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
36834079 {
36844080 union bpf_attr attr;
36854081 int err;
36864082
3687− if (sysctl_unprivileged_bpf_disabled && !capable(CAP_SYS_ADMIN))
4083+ if (sysctl_unprivileged_bpf_disabled && !bpf_capable())
36884084 return -EPERM;
36894085
36904086 err = bpf_check_uarg_tail_zero(uattr, sizeof(attr), size);
@@ -3796,6 +4192,19 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz · 105 unmodified lines
37964192 case BPF_LINK_UPDATE:
37974193 err = link_update(&attr);
37984194 break;
4195+ case BPF_LINK_GET_FD_BY_ID:
4196+ err = bpf_link_get_fd_by_id(&attr);
4197+ break;
4198+ case BPF_LINK_GET_NEXT_ID:
4199+ err = bpf_obj_get_next_id(&attr, uattr,
4200+ &link_idr, &link_idr_lock);
4201+ break;
4202+ case BPF_ENABLE_STATS:
4203+ err = bpf_enable_stats(&attr);
4204+ break;
4205+ case BPF_ITER_CREATE:
4206+ err = bpf_iter_create(&attr);
4207+ break;
37994208 default:
38004209 err = -EINVAL;
38014210 break;
kernel/bpf/task_iter.c+353−0
@@ -0,0 +1,353 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/* Copyright (c) 2020 Facebook */
3+
4+#include <linux/init.h>
5+#include <linux/namei.h>
6+#include <linux/pid_namespace.h>
7+#include <linux/fs.h>
8+#include <linux/fdtable.h>
9+#include <linux/filter.h>
10+
11+struct bpf_iter_seq_task_common {
12+ struct pid_namespace *ns;
13+};
14+
15+struct bpf_iter_seq_task_info {
16+ /* The first field must be struct bpf_iter_seq_task_common.
17+ * this is assumed by {init, fini}_seq_pidns() callback functions.
18+ */
19+ struct bpf_iter_seq_task_common common;
20+ u32 tid;
21+};
22+
23+static struct task_struct *task_seq_get_next(struct pid_namespace *ns,
24+ u32 *tid)
25+{
26+ struct task_struct *task = NULL;
27+ struct pid *pid;
28+
29+ rcu_read_lock();
30+retry:
31+ pid = idr_get_next(&ns->idr, tid);
32+ if (pid) {
33+ task = get_pid_task(pid, PIDTYPE_PID);
34+ if (!task) {
35+ ++*tid;
36+ goto retry;
37+ }
38+ }
39+ rcu_read_unlock();
40+
41+ return task;
42+}
43+
44+static void *task_seq_start(struct seq_file *seq, loff_t *pos)
45+{
46+ struct bpf_iter_seq_task_info *info = seq->private;
47+ struct task_struct *task;
48+
49+ task = task_seq_get_next(info->common.ns, &info->tid);
50+ if (!task)
51+ return NULL;
52+
53+ ++*pos;
54+ return task;
55+}
56+
57+static void *task_seq_next(struct seq_file *seq, void *v, loff_t *pos)
58+{
59+ struct bpf_iter_seq_task_info *info = seq->private;
60+ struct task_struct *task;
61+
62+ ++*pos;
63+ ++info->tid;
64+ put_task_struct((struct task_struct *)v);
65+ task = task_seq_get_next(info->common.ns, &info->tid);
66+ if (!task)
67+ return NULL;
68+
69+ return task;
70+}
71+
72+struct bpf_iter__task {
73+ __bpf_md_ptr(struct bpf_iter_meta *, meta);
74+ __bpf_md_ptr(struct task_struct *, task);
75+};
76+
77+DEFINE_BPF_ITER_FUNC(task, struct bpf_iter_meta *meta, struct task_struct *task)
78+
79+static int __task_seq_show(struct seq_file *seq, struct task_struct *task,
80+ bool in_stop)
81+{
82+ struct bpf_iter_meta meta;
83+ struct bpf_iter__task ctx;
84+ struct bpf_prog *prog;
85+
86+ meta.seq = seq;
87+ prog = bpf_iter_get_info(&meta, in_stop);
88+ if (!prog)
89+ return 0;
90+
91+ meta.seq = seq;
92+ ctx.meta = &meta;
93+ ctx.task = task;
94+ return bpf_iter_run_prog(prog, &ctx);
95+}
96+
97+static int task_seq_show(struct seq_file *seq, void *v)
98+{
99+ return __task_seq_show(seq, v, false);
100+}
101+
102+static void task_seq_stop(struct seq_file *seq, void *v)
103+{
104+ if (!v)
105+ (void)__task_seq_show(seq, v, true);
106+ else
107+ put_task_struct((struct task_struct *)v);
108+}
109+
110+static const struct seq_operations task_seq_ops = {
111+ .start = task_seq_start,
112+ .next = task_seq_next,
113+ .stop = task_seq_stop,
114+ .show = task_seq_show,
115+};
116+
117+struct bpf_iter_seq_task_file_info {
118+ /* The first field must be struct bpf_iter_seq_task_common.
119+ * this is assumed by {init, fini}_seq_pidns() callback functions.
120+ */
121+ struct bpf_iter_seq_task_common common;
122+ struct task_struct *task;
123+ struct files_struct *files;
124+ u32 tid;
125+ u32 fd;
126+};
127+
128+static struct file *
129+task_file_seq_get_next(struct bpf_iter_seq_task_file_info *info,
130+ struct task_struct **task, struct files_struct **fstruct)
131+{
132+ struct pid_namespace *ns = info->common.ns;
133+ u32 curr_tid = info->tid, max_fds;
134+ struct files_struct *curr_files;
135+ struct task_struct *curr_task;
136+ int curr_fd = info->fd;
137+
138+ /* If this function returns a non-NULL file object,
139+ * it held a reference to the task/files_struct/file.
140+ * Otherwise, it does not hold any reference.
141+ */
142+again:
143+ if (*task) {
144+ curr_task = *task;
145+ curr_files = *fstruct;
146+ curr_fd = info->fd;
147+ } else {
148+ curr_task = task_seq_get_next(ns, &curr_tid);
149+ if (!curr_task)
150+ return NULL;
151+
152+ curr_files = get_files_struct(curr_task);
153+ if (!curr_files) {
154+ put_task_struct(curr_task);
155+ curr_tid = ++(info->tid);
156+ info->fd = 0;
157+ goto again;
158+ }
159+
160+ /* set *fstruct, *task and info->tid */
161+ *fstruct = curr_files;
162+ *task = curr_task;
163+ if (curr_tid == info->tid) {
164+ curr_fd = info->fd;
165+ } else {
166+ info->tid = curr_tid;
167+ curr_fd = 0;
168+ }
169+ }
170+
171+ rcu_read_lock();
172+ max_fds = files_fdtable(curr_files)->max_fds;
173+ for (; curr_fd < max_fds; curr_fd++) {
174+ struct file *f;
175+
176+ f = fcheck_files(curr_files, curr_fd);
177+ if (!f)
178+ continue;
179+
180+ /* set info->fd */
181+ info->fd = curr_fd;
182+ get_file(f);
183+ rcu_read_unlock();
184+ return f;
185+ }
186+
187+ /* the current task is done, go to the next task */
188+ rcu_read_unlock();
189+ put_files_struct(curr_files);
190+ put_task_struct(curr_task);
191+ *task = NULL;
192+ *fstruct = NULL;
193+ info->fd = 0;
194+ curr_tid = ++(info->tid);
195+ goto again;
196+}
197+
198+static void *task_file_seq_start(struct seq_file *seq, loff_t *pos)
199+{
200+ struct bpf_iter_seq_task_file_info *info = seq->private;
201+ struct files_struct *files = NULL;
202+ struct task_struct *task = NULL;
203+ struct file *file;
204+
205+ file = task_file_seq_get_next(info, &task, &files);
206+ if (!file) {
207+ info->files = NULL;
208+ info->task = NULL;
209+ return NULL;
210+ }
211+
212+ ++*pos;
213+ info->task = task;
214+ info->files = files;
215+
216+ return file;
217+}
218+
219+static void *task_file_seq_next(struct seq_file *seq, void *v, loff_t *pos)
220+{
221+ struct bpf_iter_seq_task_file_info *info = seq->private;
222+ struct files_struct *files = info->files;
223+ struct task_struct *task = info->task;
224+ struct file *file;
225+
226+ ++*pos;
227+ ++info->fd;
228+ fput((struct file *)v);
229+ file = task_file_seq_get_next(info, &task, &files);
230+ if (!file) {
231+ info->files = NULL;
232+ info->task = NULL;
233+ return NULL;
234+ }
235+
236+ info->task = task;
237+ info->files = files;
238+
239+ return file;
240+}
241+
242+struct bpf_iter__task_file {
243+ __bpf_md_ptr(struct bpf_iter_meta *, meta);
244+ __bpf_md_ptr(struct task_struct *, task);
245+ u32 fd __aligned(8);
246+ __bpf_md_ptr(struct file *, file);
247+};
248+
249+DEFINE_BPF_ITER_FUNC(task_file, struct bpf_iter_meta *meta,
250+ struct task_struct *task, u32 fd,
251+ struct file *file)
252+
253+static int __task_file_seq_show(struct seq_file *seq, struct file *file,
254+ bool in_stop)
255+{
256+ struct bpf_iter_seq_task_file_info *info = seq->private;
257+ struct bpf_iter__task_file ctx;
258+ struct bpf_iter_meta meta;
259+ struct bpf_prog *prog;
260+
261+ meta.seq = seq;
262+ prog = bpf_iter_get_info(&meta, in_stop);
263+ if (!prog)
264+ return 0;
265+
266+ ctx.meta = &meta;
267+ ctx.task = info->task;
268+ ctx.fd = info->fd;
269+ ctx.file = file;
270+ return bpf_iter_run_prog(prog, &ctx);
271+}
272+
273+static int task_file_seq_show(struct seq_file *seq, void *v)
274+{
275+ return __task_file_seq_show(seq, v, false);
276+}
277+
278+static void task_file_seq_stop(struct seq_file *seq, void *v)
279+{
280+ struct bpf_iter_seq_task_file_info *info = seq->private;
281+
282+ if (!v) {
283+ (void)__task_file_seq_show(seq, v, true);
284+ } else {
285+ fput((struct file *)v);
286+ put_files_struct(info->files);
287+ put_task_struct(info->task);
288+ info->files = NULL;
289+ info->task = NULL;
290+ }
291+}
292+
293+static int init_seq_pidns(void *priv_data)
294+{
295+ struct bpf_iter_seq_task_common *common = priv_data;
296+
297+ common->ns = get_pid_ns(task_active_pid_ns(current));
298+ return 0;
299+}
300+
301+static void fini_seq_pidns(void *priv_data)
302+{
303+ struct bpf_iter_seq_task_common *common = priv_data;
304+
305+ put_pid_ns(common->ns);
306+}
307+
308+static const struct seq_operations task_file_seq_ops = {
309+ .start = task_file_seq_start,
310+ .next = task_file_seq_next,
311+ .stop = task_file_seq_stop,
312+ .show = task_file_seq_show,
313+};
314+
315+static const struct bpf_iter_reg task_reg_info = {
316+ .target = "task",
317+ .seq_ops = &task_seq_ops,
318+ .init_seq_private = init_seq_pidns,
319+ .fini_seq_private = fini_seq_pidns,
320+ .seq_priv_size = sizeof(struct bpf_iter_seq_task_info),
321+ .ctx_arg_info_size = 1,
322+ .ctx_arg_info = {
323+ { offsetof(struct bpf_iter__task, task),
324+ PTR_TO_BTF_ID_OR_NULL },
325+ },
326+};
327+
328+static const struct bpf_iter_reg task_file_reg_info = {
329+ .target = "task_file",
330+ .seq_ops = &task_file_seq_ops,
331+ .init_seq_private = init_seq_pidns,
332+ .fini_seq_private = fini_seq_pidns,
333+ .seq_priv_size = sizeof(struct bpf_iter_seq_task_file_info),
334+ .ctx_arg_info_size = 2,
335+ .ctx_arg_info = {
336+ { offsetof(struct bpf_iter__task_file, task),
337+ PTR_TO_BTF_ID_OR_NULL },
338+ { offsetof(struct bpf_iter__task_file, file),
339+ PTR_TO_BTF_ID_OR_NULL },
340+ },
341+};
342+
343+static int __init task_iter_init(void)
344+{
345+ int ret;
346+
347+ ret = bpf_iter_reg_target(&task_reg_info);
348+ if (ret)
349+ return ret;
350+
351+ return bpf_iter_reg_target(&task_file_reg_info);
352+}
353+late_initcall(task_iter_init);
kernel/bpf/verifier.c+299−88
@@ -28,9 +28,11 @@ static const struct bpf_verifier_ops * const bpf_verifier_ops[] = { · 27 unmodified lines
2828 #define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
2929 [_id] = & _name ## _verifier_ops,
3030 #define BPF_MAP_TYPE(_id, _ops)
31+#define BPF_LINK_TYPE(_id, _name)
3132 #include <linux/bpf_types.h>
3233 #undef BPF_PROG_TYPE
3334 #undef BPF_MAP_TYPE
35+#undef BPF_LINK_TYPE
3436 };
3537
3638 /* bpf_check() is a static code analyzer that walks eBPF program
@@ -168,6 +170,8 @@ struct bpf_verifier_stack_elem { · 131 unmodified lines
168170 int insn_idx;
169171 int prev_insn_idx;
170172 struct bpf_verifier_stack_elem *next;
173+ /* length of verifier log at the time this state was pushed on stack */
174+ u32 log_pos;
171175 };
172176
173177 #define BPF_COMPLEXITY_LIMIT_JMP_SEQ 8192
@@ -229,6 +233,7 @@ struct bpf_call_arg_meta { · 55 unmodified lines
229233 bool pkt_access;
230234 int regno;
231235 int access_size;
236+ int mem_size;
232237 u64 msize_max_value;
233238 int ref_obj_id;
234239 int func_id;
@@ -283,6 +288,18 @@ void bpf_verifier_vlog(struct bpf_verifier_log *log, const char *fmt, · 48 unmodified lines
283288 log->ubuf = NULL;
284289 }
285290
291+static void bpf_vlog_reset(struct bpf_verifier_log *log, u32 new_pos)
292+{
293+ char zero = 0;
294+
295+ if (!bpf_verifier_log_needed(log))
296+ return;
297+
298+ log->len_used = new_pos;
299+ if (put_user(zero, log->ubuf + new_pos))
300+ log->ubuf = NULL;
301+}
302+
286303 /* log_level controls verbosity level of eBPF verifier.
287304 * bpf_verifier_log_write() is used to dump the verification trace to the log,
288305 * so the user can figure out what's wrong with the program
@@ -377,12 +394,23 @@ static bool type_is_sk_pointer(enum bpf_reg_type type) · 88 unmodified lines
377394 type == PTR_TO_XDP_SOCK;
378395 }
379396
397+static bool reg_type_not_null(enum bpf_reg_type type)
398+{
399+ return type == PTR_TO_SOCKET ||
400+ type == PTR_TO_TCP_SOCK ||
401+ type == PTR_TO_MAP_VALUE ||
402+ type == PTR_TO_SOCK_COMMON ||
403+ type == PTR_TO_BTF_ID;
404+}
405+
380406 static bool reg_type_may_be_null(enum bpf_reg_type type)
381407 {
382408 return type == PTR_TO_MAP_VALUE_OR_NULL ||
383409 type == PTR_TO_SOCKET_OR_NULL ||
384410 type == PTR_TO_SOCK_COMMON_OR_NULL ||
385− type == PTR_TO_TCP_SOCK_OR_NULL;
411+ type == PTR_TO_TCP_SOCK_OR_NULL ||
412+ type == PTR_TO_BTF_ID_OR_NULL ||
413+ type == PTR_TO_MEM_OR_NULL;
386414 }
387415
388416 static bool reg_may_point_to_spin_lock(const struct bpf_reg_state *reg)
@@ -396,7 +424,9 @@ static bool reg_type_may_be_refcounted_or_null(enum bpf_reg_type type) · 7 unmodified lines
396424 return type == PTR_TO_SOCKET ||
397425 type == PTR_TO_SOCKET_OR_NULL ||
398426 type == PTR_TO_TCP_SOCK ||
399− type == PTR_TO_TCP_SOCK_OR_NULL;
427+ type == PTR_TO_TCP_SOCK_OR_NULL ||
428+ type == PTR_TO_MEM ||
429+ type == PTR_TO_MEM_OR_NULL;
400430 }
401431
402432 static bool arg_type_may_be_refcounted(enum bpf_arg_type type)
@@ -410,14 +440,37 @@ static bool arg_type_may_be_refcounted(enum bpf_arg_type type) · 7 unmodified lines
410440 */
411441 static bool is_release_function(enum bpf_func_id func_id)
412442 {
413− return func_id == BPF_FUNC_sk_release;
443+ return func_id == BPF_FUNC_sk_release ||
444+ func_id == BPF_FUNC_ringbuf_submit ||
445+ func_id == BPF_FUNC_ringbuf_discard;
414446 }
415447
416−static bool is_acquire_function(enum bpf_func_id func_id)
448+static bool may_be_acquire_function(enum bpf_func_id func_id)
417449 {
418450 return func_id == BPF_FUNC_sk_lookup_tcp ||
419451 func_id == BPF_FUNC_sk_lookup_udp ||
420− func_id == BPF_FUNC_skc_lookup_tcp;
452+ func_id == BPF_FUNC_skc_lookup_tcp ||
453+ func_id == BPF_FUNC_map_lookup_elem ||
454+ func_id == BPF_FUNC_ringbuf_reserve;
455+}
456+
457+static bool is_acquire_function(enum bpf_func_id func_id,
458+ const struct bpf_map *map)
459+{
460+ enum bpf_map_type map_type = map ? map->map_type : BPF_MAP_TYPE_UNSPEC;
461+
462+ if (func_id == BPF_FUNC_sk_lookup_tcp ||
463+ func_id == BPF_FUNC_sk_lookup_udp ||
464+ func_id == BPF_FUNC_skc_lookup_tcp ||
465+ func_id == BPF_FUNC_ringbuf_reserve)
466+ return true;
467+
468+ if (func_id == BPF_FUNC_map_lookup_elem &&
469+ (map_type == BPF_MAP_TYPE_SOCKMAP ||
470+ map_type == BPF_MAP_TYPE_SOCKHASH))
471+ return true;
472+
473+ return false;
421474 }
422475
423476 static bool is_ptr_cast_function(enum bpf_func_id func_id)
@@ -448,6 +501,9 @@ static const char * const reg_type_str[] = { · 24 unmodified lines
448501 [PTR_TO_TP_BUFFER] = "tp_buffer",
449502 [PTR_TO_XDP_SOCK] = "xdp_sock",
450503 [PTR_TO_BTF_ID] = "ptr_",
504+ [PTR_TO_BTF_ID_OR_NULL] = "ptr_or_null_",
505+ [PTR_TO_MEM] = "mem",
506+ [PTR_TO_MEM_OR_NULL] = "mem_or_null",
451507 };
452508
453509 static char slot_type_char[] = {
@@ -508,7 +564,7 @@ static void print_verifier_state(struct bpf_verifier_env *env, · 54 unmodified lines
508564 /* reg->off should be 0 for SCALAR_VALUE */
509565 verbose(env, "%lld", reg->var_off.value + reg->off);
510566 } else {
511− if (t == PTR_TO_BTF_ID)
567+ if (t == PTR_TO_BTF_ID || t == PTR_TO_BTF_ID_OR_NULL)
512568 verbose(env, "%s", kernel_type_name(reg->btf_id));
513569 verbose(env, "(id=%d", reg->id);
514570 if (reg_type_may_be_refcounted_or_null(t))
@@ -846,7 +902,7 @@ static void update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifi · 331 unmodified lines
846902 }
847903
848904 static int pop_stack(struct bpf_verifier_env *env, int *prev_insn_idx,
849− int *insn_idx)
905+ int *insn_idx, bool pop_log)
850906 {
851907 struct bpf_verifier_state *cur = env->cur_state;
852908 struct bpf_verifier_stack_elem *elem, *head = env->head;
@@ -860,6 +916,8 @@ static int pop_stack(struct bpf_verifier_env *env, int *prev_insn_idx, · 7 unmodified lines
860916 if (err)
861917 return err;
862918 }
919+ if (pop_log)
920+ bpf_vlog_reset(&env->log, head->log_pos);
863921 if (insn_idx)
864922 *insn_idx = head->insn_idx;
865923 if (prev_insn_idx)
@@ -887,6 +945,7 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env, · 21 unmodified lines
887945 elem->insn_idx = insn_idx;
888946 elem->prev_insn_idx = prev_insn_idx;
889947 elem->next = env->head;
948+ elem->log_pos = env->log.len_used;
890949 env->head = elem;
891950 env->stack_size++;
892951 err = copy_verifier_state(&elem->st, cur);
@@ -915,7 +974,7 @@ err: · 22 unmodified lines
915974 free_verifier_state(env->cur_state, true);
916975 env->cur_state = NULL;
917976 /* pop all elements and return */
918− while (!pop_stack(env, NULL, NULL));
977+ while (!pop_stack(env, NULL, NULL, false));
919978 return NULL;
920979 }
921980
@@ -1255,7 +1314,7 @@ static void __mark_reg_unknown(const struct bpf_verifier_env *env, · 333 unmodified lines
12551314 reg->type = SCALAR_VALUE;
12561315 reg->var_off = tnum_unknown;
12571316 reg->frameno = 0;
1258− reg->precise = env->subprog_cnt > 1 || !env->allow_ptr_leaks;
1317+ reg->precise = env->subprog_cnt > 1 || !env->bpf_capable;
12591318 __mark_reg_unbounded(reg);
12601319 }
12611320
@@ -1387,8 +1446,9 @@ static int check_subprogs(struct bpf_verifier_env *env) · 125 unmodified lines
13871446 continue;
13881447 if (insn[i].src_reg != BPF_PSEUDO_CALL)
13891448 continue;
1390− if (!env->allow_ptr_leaks) {
1391− verbose(env, "function calls to other bpf functions are allowed for root only\n");
1449+ if (!env->bpf_capable) {
1450+ verbose(env,
1451+ "function calls to other bpf functions are allowed for CAP_BPF and CAP_SYS_ADMIN\n");
13921452 return -EPERM;
13931453 }
13941454 ret = add_subprog(env, i + insn[i].imm + 1);
@@ -1922,8 +1982,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int regno, · 527 unmodified lines
19221982 bool new_marks = false;
19231983 int i, err;
19241984
1925− if (!env->allow_ptr_leaks)
1926− /* backtracking is root only for now */
1985+ if (!env->bpf_capable)
19271986 return 0;
19281987
19291988 func = st->frame[st->curframe];
@@ -2101,6 +2160,7 @@ static bool is_spillable_regtype(enum bpf_reg_type type) · 171 unmodified lines
21012160 case PTR_TO_TCP_SOCK_OR_NULL:
21022161 case PTR_TO_XDP_SOCK:
21032162 case PTR_TO_BTF_ID:
2163+ case PTR_TO_BTF_ID_OR_NULL:
21042164 return true;
21052165 default:
21062166 return false;
@@ -2170,7 +2230,7 @@ static int check_stack_write(struct bpf_verifier_env *env, · 63 unmodified lines
21702230 reg = &cur->regs[value_regno];
21712231
21722232 if (reg && size == BPF_REG_SIZE && register_is_const(reg) &&
2173− !register_is_null(reg) && env->allow_ptr_leaks) {
2233+ !register_is_null(reg) && env->bpf_capable) {
21742234 if (dst_reg != BPF_REG_FP) {
21752235 /* The backtracking logic can only recognize explicit
21762236 * stack slot address like [fp - 8]. Other spill of
@@ -2196,7 +2256,7 @@ static int check_stack_write(struct bpf_verifier_env *env, · 19 unmodified lines
21962256 return -EINVAL;
21972257 }
21982258
2199− if (!env->allow_ptr_leaks) {
2259+ if (!env->bypass_spec_v4) {
22002260 bool sanitize = false;
22012261
22022262 if (state->stack[spi].slot_type[0] == STACK_SPILL &&
@@ -2418,32 +2478,49 @@ static int check_map_access_type(struct bpf_verifier_env *env, u32 regno, · 215 unmodified lines
24182478 return 0;
24192479 }
24202480
2421−/* check read/write into map element returned by bpf_map_lookup_elem() */
2422−static int __check_map_access(struct bpf_verifier_env *env, u32 regno, int off,
2423− int size, bool zero_size_allowed)
2481+/* check read/write into memory region (e.g., map value, ringbuf sample, etc) */
2482+static int __check_mem_access(struct bpf_verifier_env *env, int regno,
2483+ int off, int size, u32 mem_size,
2484+ bool zero_size_allowed)
24242485 {
2425− struct bpf_reg_state *regs = cur_regs(env);
2426− struct bpf_map *map = regs[regno].map_ptr;
2486+ bool size_ok = size > 0 || (size == 0 && zero_size_allowed);
2487+ struct bpf_reg_state *reg;
2488+
2489+ if (off >= 0 && size_ok && (u64)off + size <= mem_size)
2490+ return 0;
24272491
2428− if (off < 0 || size < 0 || (size == 0 && !zero_size_allowed) ||
2429− off + size > map->value_size) {
2492+ reg = &cur_regs(env)[regno];
2493+ switch (reg->type) {
2494+ case PTR_TO_MAP_VALUE:
24302495 verbose(env, "invalid access to map value, value_size=%d off=%d size=%d\n",
2431− map->value_size, off, size);
2432− return -EACCES;
2496+ mem_size, off, size);
2497+ break;
2498+ case PTR_TO_PACKET:
2499+ case PTR_TO_PACKET_META:
2500+ case PTR_TO_PACKET_END:
2501+ verbose(env, "invalid access to packet, off=%d size=%d, R%d(id=%d,off=%d,r=%d)\n",
2502+ off, size, regno, reg->id, off, mem_size);
2503+ break;
2504+ case PTR_TO_MEM:
2505+ default:
2506+ verbose(env, "invalid access to memory, mem_size=%u off=%d size=%d\n",
2507+ mem_size, off, size);
24332508 }
2434− return 0;
2509+
2510+ return -EACCES;
24352511 }
24362512
2437−/* check read/write into a map element with possible variable offset */
2438−static int check_map_access(struct bpf_verifier_env *env, u32 regno,
2439− int off, int size, bool zero_size_allowed)
2513+/* check read/write into a memory region with possible variable offset */
2514+static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno,
2515+ int off, int size, u32 mem_size,
2516+ bool zero_size_allowed)
24402517 {
24412518 struct bpf_verifier_state *vstate = env->cur_state;
24422519 struct bpf_func_state *state = vstate->frame[vstate->curframe];
24432520 struct bpf_reg_state *reg = &state->regs[regno];
24442521 int err;
24452522
2446− /* We may have adjusted the register to this map value, so we
2523+ /* We may have adjusted the register pointing to memory region, so we
24472524 * need to try adding each of min_value and max_value to off
24482525 * to make sure our theoretical access will be safe.
24492526 */
@@ -2464,10 +2541,10 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 14 unmodified lines
24642541 regno);
24652542 return -EACCES;
24662543 }
2467− err = __check_map_access(env, regno, reg->smin_value + off, size,
2468− zero_size_allowed);
2544+ err = __check_mem_access(env, regno, reg->smin_value + off, size,
2545+ mem_size, zero_size_allowed);
24692546 if (err) {
2470− verbose(env, "R%d min value is outside of the array range\n",
2547+ verbose(env, "R%d min value is outside of the allowed memory range\n",
24712548 regno);
24722549 return err;
24732550 }
@@ -2477,18 +2554,38 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 3 unmodified lines
24772554 * If reg->umax_value + off could overflow, treat that as unbounded too.
24782555 */
24792556 if (reg->umax_value >= BPF_MAX_VAR_OFF) {
2480− verbose(env, "R%d unbounded memory access, make sure to bounds check any array access into a map\n",
2557+ verbose(env, "R%d unbounded memory access, make sure to bounds check any such access\n",
24812558 regno);
24822559 return -EACCES;
24832560 }
2484− err = __check_map_access(env, regno, reg->umax_value + off, size,
2485− zero_size_allowed);
2486− if (err)
2487− verbose(env, "R%d max value is outside of the array range\n",
2561+ err = __check_mem_access(env, regno, reg->umax_value + off, size,
2562+ mem_size, zero_size_allowed);
2563+ if (err) {
2564+ verbose(env, "R%d max value is outside of the allowed memory range\n",
24882565 regno);
2566+ return err;
2567+ }
2568+
2569+ return 0;
2570+}
2571+
2572+/* check read/write into a map element with possible variable offset */
2573+static int check_map_access(struct bpf_verifier_env *env, u32 regno,
2574+ int off, int size, bool zero_size_allowed)
2575+{
2576+ struct bpf_verifier_state *vstate = env->cur_state;
2577+ struct bpf_func_state *state = vstate->frame[vstate->curframe];
2578+ struct bpf_reg_state *reg = &state->regs[regno];
2579+ struct bpf_map *map = reg->map_ptr;
2580+ int err;
24892581
2490− if (map_value_has_spin_lock(reg->map_ptr)) {
2491− u32 lock = reg->map_ptr->spin_lock_off;
2582+ err = check_mem_region_access(env, regno, off, size, map->value_size,
2583+ zero_size_allowed);
2584+ if (err)
2585+ return err;
2586+
2587+ if (map_value_has_spin_lock(map)) {
2588+ u32 lock = map->spin_lock_off;
24922589
24932590 /* if any part of struct bpf_spin_lock can be touched by
24942591 * load/store reject this program.
@@ -2546,21 +2643,6 @@ static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, · 51 unmodified lines
25462643 }
25472644 }
25482645
2549−static int __check_packet_access(struct bpf_verifier_env *env, u32 regno,
2550− int off, int size, bool zero_size_allowed)
2551−{
2552− struct bpf_reg_state *regs = cur_regs(env);
2553− struct bpf_reg_state *reg = &regs[regno];
2554−
2555− if (off < 0 || size < 0 || (size == 0 && !zero_size_allowed) ||
2556− (u64)off + size > reg->range) {
2557− verbose(env, "invalid access to packet, off=%d size=%d, R%d(id=%d,off=%d,r=%d)\n",
2558− off, size, regno, reg->id, reg->off, reg->range);
2559− return -EACCES;
2560− }
2561− return 0;
2562−}
2563−
25642646 static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off,
25652647 int size, bool zero_size_allowed)
25662648 {
@@ -2581,16 +2663,17 @@ static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off, · 14 unmodified lines
25812663 regno);
25822664 return -EACCES;
25832665 }
2584− err = __check_packet_access(env, regno, off, size, zero_size_allowed);
2666+ err = __check_mem_access(env, regno, off, size, reg->range,
2667+ zero_size_allowed);
25852668 if (err) {
25862669 verbose(env, "R%d offset is outside of the packet\n", regno);
25872670 return err;
25882671 }
25892672
2590− /* __check_packet_access has made sure "off + size - 1" is within u16.
2673+ /* __check_mem_access has made sure "off + size - 1" is within u16.
25912674 * reg->umax_value can't be bigger than MAX_PACKET_OFF which is 0xffff,
25922675 * otherwise find_good_pkt_pointers would have refused to set range info
2593− * that __check_packet_access would have rejected this pkt access.
2676+ * that __check_mem_access would have rejected this pkt access.
25942677 * Therefore, "off + reg->umax_value + size - 1" won't overflow u32.
25952678 */
25962679 env->prog->aux->max_pkt_offset =
@@ -2621,7 +2704,7 @@ static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, int off, · 24 unmodified lines
26212704 */
26222705 *reg_type = info.reg_type;
26232706
2624− if (*reg_type == PTR_TO_BTF_ID)
2707+ if (*reg_type == PTR_TO_BTF_ID || *reg_type == PTR_TO_BTF_ID_OR_NULL)
26252708 *btf_id = info.btf_id;
26262709 else
26272710 env->insn_aux_data[insn_idx].ctx_field_size = info.ctx_field_size;
@@ -3170,6 +3253,16 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 542 unmodified lines
31703253 mark_reg_unknown(env, regs, value_regno);
31713254 }
31723255 }
3256+ } else if (reg->type == PTR_TO_MEM) {
3257+ if (t == BPF_WRITE && value_regno >= 0 &&
3258+ is_pointer_value(env, value_regno)) {
3259+ verbose(env, "R%d leaks addr into mem\n", value_regno);
3260+ return -EACCES;
3261+ }
3262+ err = check_mem_region_access(env, regno, off, size,
3263+ reg->mem_size, false);
3264+ if (!err && t == BPF_READ && value_regno >= 0)
3265+ mark_reg_unknown(env, regs, value_regno);
31733266 } else if (reg->type == PTR_TO_CTX) {
31743267 enum bpf_reg_type reg_type = SCALAR_VALUE;
31753268 u32 btf_id = 0;
@@ -3205,7 +3298,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 29 unmodified lines
32053298 * a sub-register.
32063299 */
32073300 regs[value_regno].subreg_def = DEF_NOT_SUBREG;
3208− if (reg_type == PTR_TO_BTF_ID)
3301+ if (reg_type == PTR_TO_BTF_ID ||
3302+ reg_type == PTR_TO_BTF_ID_OR_NULL)
32093303 regs[value_regno].btf_id = btf_id;
32103304 }
32113305 regs[value_regno].type = reg_type;
@@ -3390,7 +3484,7 @@ static int check_stack_boundary(struct bpf_verifier_env *env, int regno, · 178 unmodified lines
33903484 * Spectre masking for stack ALU.
33913485 * See also retrieve_ptr_limit().
33923486 */
3393− if (!env->allow_ptr_leaks) {
3487+ if (!env->bypass_spec_v1) {
33943488 char tn_buf[48];
33953489
33963490 tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
@@ -3452,6 +3546,11 @@ static int check_stack_boundary(struct bpf_verifier_env *env, int regno, · 55 unmodified lines
34523546 *stype = STACK_MISC;
34533547 goto mark;
34543548 }
3549+
3550+ if (state->stack[spi].slot_type[0] == STACK_SPILL &&
3551+ state->stack[spi].spilled_ptr.type == PTR_TO_BTF_ID)
3552+ goto mark;
3553+
34553554 if (state->stack[spi].slot_type[0] == STACK_SPILL &&
34563555 state->stack[spi].spilled_ptr.type == SCALAR_VALUE) {
34573556 __mark_reg_unknown(env, &state->stack[spi].spilled_ptr);
@@ -3501,6 +3600,10 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 43 unmodified lines
35013600 return -EACCES;
35023601 return check_map_access(env, regno, reg->off, access_size,
35033602 zero_size_allowed);
3603+ case PTR_TO_MEM:
3604+ return check_mem_region_access(env, regno, reg->off,
3605+ access_size, reg->mem_size,
3606+ zero_size_allowed);
35043607 default: /* scalar_value|ptr_to_stack or invalid ptr */
35053608 return check_stack_boundary(env, regno, access_size,
35063609 zero_size_allowed, meta);
@@ -3605,6 +3708,17 @@ static bool arg_type_is_mem_size(enum bpf_arg_type type) · 98 unmodified lines
36053708 type == ARG_CONST_SIZE_OR_ZERO;
36063709 }
36073710
3711+static bool arg_type_is_alloc_mem_ptr(enum bpf_arg_type type)
3712+{
3713+ return type == ARG_PTR_TO_ALLOC_MEM ||
3714+ type == ARG_PTR_TO_ALLOC_MEM_OR_NULL;
3715+}
3716+
3717+static bool arg_type_is_alloc_size(enum bpf_arg_type type)
3718+{
3719+ return type == ARG_CONST_ALLOC_SIZE_OR_ZERO;
3720+}
3721+
36083722 static bool arg_type_is_int_ptr(enum bpf_arg_type type)
36093723 {
36103724 return type == ARG_PTR_TO_INT ||
@@ -3664,7 +3778,8 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, · 53 unmodified lines
36643778 type != expected_type)
36653779 goto err_type;
36663780 } else if (arg_type == ARG_CONST_SIZE ||
3667− arg_type == ARG_CONST_SIZE_OR_ZERO) {
3781+ arg_type == ARG_CONST_SIZE_OR_ZERO ||
3782+ arg_type == ARG_CONST_ALLOC_SIZE_OR_ZERO) {
36683783 expected_type = SCALAR_VALUE;
36693784 if (type != expected_type)
36703785 goto err_type;
@@ -3735,13 +3850,29 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, · 64 unmodified lines
37353850 * happens during stack boundary checking.
37363851 */
37373852 if (register_is_null(reg) &&
3738− arg_type == ARG_PTR_TO_MEM_OR_NULL)
3853+ (arg_type == ARG_PTR_TO_MEM_OR_NULL ||
3854+ arg_type == ARG_PTR_TO_ALLOC_MEM_OR_NULL))
37393855 /* final test in check_stack_boundary() */;
37403856 else if (!type_is_pkt_pointer(type) &&
37413857 type != PTR_TO_MAP_VALUE &&
3858+ type != PTR_TO_MEM &&
37423859 type != expected_type)
37433860 goto err_type;
37443861 meta->raw_mode = arg_type == ARG_PTR_TO_UNINIT_MEM;
3862+ } else if (arg_type_is_alloc_mem_ptr(arg_type)) {
3863+ expected_type = PTR_TO_MEM;
3864+ if (register_is_null(reg) &&
3865+ arg_type == ARG_PTR_TO_ALLOC_MEM_OR_NULL)
3866+ /* final test in check_stack_boundary() */;
3867+ else if (type != expected_type)
3868+ goto err_type;
3869+ if (meta->ref_obj_id) {
3870+ verbose(env, "verifier internal error: more than one arg with ref_obj_id R%d %u %u\n",
3871+ regno, reg->ref_obj_id,
3872+ meta->ref_obj_id);
3873+ return -EFAULT;
3874+ }
3875+ meta->ref_obj_id = reg->ref_obj_id;
37453876 } else if (arg_type_is_int_ptr(arg_type)) {
37463877 expected_type = PTR_TO_STACK;
37473878 if (!type_is_pkt_pointer(type) &&
@@ -3837,6 +3968,13 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, · 89 unmodified lines
38373968 zero_size_allowed, meta);
38383969 if (!err)
38393970 err = mark_chain_precision(env, regno);
3971+ } else if (arg_type_is_alloc_size(arg_type)) {
3972+ if (!tnum_is_const(reg->var_off)) {
3973+ verbose(env, "R%d unbounded size, use 'var &= const' or 'if (var < const)'\n",
3974+ regno);
3975+ return -EACCES;
3976+ }
3977+ meta->mem_size = reg->var_off.value;
38403978 } else if (arg_type_is_int_ptr(arg_type)) {
38413979 int size = int_ptr_type_to_size(arg_type);
38423980
@@ -3873,6 +4011,14 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 30 unmodified lines
38734011 func_id != BPF_FUNC_xdp_output)
38744012 goto error;
38754013 break;
4014+ case BPF_MAP_TYPE_RINGBUF:
4015+ if (func_id != BPF_FUNC_ringbuf_output &&
4016+ func_id != BPF_FUNC_ringbuf_reserve &&
4017+ func_id != BPF_FUNC_ringbuf_submit &&
4018+ func_id != BPF_FUNC_ringbuf_discard &&
4019+ func_id != BPF_FUNC_ringbuf_query)
4020+ goto error;
4021+ break;
38764022 case BPF_MAP_TYPE_STACK_TRACE:
38774023 if (func_id != BPF_FUNC_get_stackid)
38784024 goto error;
@@ -3915,7 +4061,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 36 unmodified lines
39154061 func_id != BPF_FUNC_sock_map_update &&
39164062 func_id != BPF_FUNC_map_delete_elem &&
39174063 func_id != BPF_FUNC_msg_redirect_map &&
3918− func_id != BPF_FUNC_sk_select_reuseport)
4064+ func_id != BPF_FUNC_sk_select_reuseport &&
4065+ func_id != BPF_FUNC_map_lookup_elem)
39194066 goto error;
39204067 break;
39214068 case BPF_MAP_TYPE_SOCKHASH:
@@ -3923,7 +4070,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 1 unmodified line
39234070 func_id != BPF_FUNC_sock_hash_update &&
39244071 func_id != BPF_FUNC_map_delete_elem &&
39254072 func_id != BPF_FUNC_msg_redirect_hash &&
3926− func_id != BPF_FUNC_sk_select_reuseport)
4073+ func_id != BPF_FUNC_sk_select_reuseport &&
4074+ func_id != BPF_FUNC_map_lookup_elem)
39274075 goto error;
39284076 break;
39294077 case BPF_MAP_TYPE_REUSEPORT_SOCKARRAY:
@@ -4093,7 +4241,7 @@ static bool check_refcount_ok(const struct bpf_func_proto *fn, int func_id) · 163 unmodified lines
40934241 /* A reference acquiring function cannot acquire
40944242 * another refcounted ptr.
40954243 */
4096− if (is_acquire_function(func_id) && count)
4244+ if (may_be_acquire_function(func_id) && count)
40974245 return false;
40984246
40994247 /* We only support one arg being unreferenced at the moment,
@@ -4388,10 +4536,10 @@ record_func_map(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, · 288 unmodified lines
43884536
43894537 if (!BPF_MAP_PTR(aux->map_ptr_state))
43904538 bpf_map_ptr_store(aux, meta->map_ptr,
4391− meta->map_ptr->unpriv_array);
4539+ !meta->map_ptr->bypass_spec_v1);
43924540 else if (BPF_MAP_PTR(aux->map_ptr_state) != meta->map_ptr)
43934541 bpf_map_ptr_store(aux, BPF_MAP_PTR_POISON,
4394− meta->map_ptr->unpriv_array);
4542+ !meta->map_ptr->bypass_spec_v1);
43954543 return 0;
43964544 }
43974545
@@ -4597,6 +4745,11 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 199 unmodified lines
45974745 mark_reg_known_zero(env, regs, BPF_REG_0);
45984746 regs[BPF_REG_0].type = PTR_TO_TCP_SOCK_OR_NULL;
45994747 regs[BPF_REG_0].id = ++env->id_gen;
4748+ } else if (fn->ret_type == RET_PTR_TO_ALLOC_MEM_OR_NULL) {
4749+ mark_reg_known_zero(env, regs, BPF_REG_0);
4750+ regs[BPF_REG_0].type = PTR_TO_MEM_OR_NULL;
4751+ regs[BPF_REG_0].id = ++env->id_gen;
4752+ regs[BPF_REG_0].mem_size = meta.mem_size;
46004753 } else {
46014754 verbose(env, "unknown return type %d of func %s#%d\n",
46024755 fn->ret_type, func_id_name(func_id), func_id);
@@ -4606,7 +4759,7 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 3 unmodified lines
46064759 if (is_ptr_cast_function(func_id)) {
46074760 /* For release_reference() */
46084761 regs[BPF_REG_0].ref_obj_id = meta.ref_obj_id;
4609− } else if (is_acquire_function(func_id)) {
4762+ } else if (is_acquire_function(func_id, meta.map_ptr)) {
46104763 int id = acquire_reference_state(env, insn_idx);
46114764
46124765 if (id < 0)
@@ -4760,7 +4913,7 @@ static int retrieve_ptr_limit(const struct bpf_reg_state *ptr_reg, · 147 unmodified lines
47604913 static bool can_skip_alu_sanitation(const struct bpf_verifier_env *env,
47614914 const struct bpf_insn *insn)
47624915 {
4763− return env->allow_ptr_leaks || BPF_SRC(insn->code) == BPF_K;
4916+ return env->bypass_spec_v1 || BPF_SRC(insn->code) == BPF_K;
47644917 }
47654918
47664919 static int update_alu_sanitation_state(struct bpf_insn_aux_data *aux,
@@ -5070,7 +5223,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, · 303 unmodified lines
50705223 /* For unprivileged we require that resulting offset must be in bounds
50715224 * in order to be able to sanitize access later on.
50725225 */
5073− if (!env->allow_ptr_leaks) {
5226+ if (!env->bypass_spec_v1) {
50745227 if (dst_reg->type == PTR_TO_MAP_VALUE &&
50755228 check_map_access(env, dst, dst_reg->off, 1, false)) {
50765229 verbose(env, "R%d pointer arithmetic of map value goes out of range, "
@@ -5611,7 +5764,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env, · 534 unmodified lines
56115764 {
56125765 struct bpf_reg_state *regs = cur_regs(env);
56135766 u8 opcode = BPF_OP(insn->code);
5614− bool src_known, dst_known;
5767+ bool src_known;
56155768 s64 smin_val, smax_val;
56165769 u64 umin_val, umax_val;
56175770 s32 s32_min_val, s32_max_val;
@@ -5633,7 +5786,6 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env, · 15 unmodified lines
56335786
56345787 if (alu32) {
56355788 src_known = tnum_subreg_is_const(src_reg.var_off);
5636− dst_known = tnum_subreg_is_const(dst_reg->var_off);
56375789 if ((src_known &&
56385790 (s32_min_val != s32_max_val || u32_min_val != u32_max_val)) ||
56395791 s32_min_val > s32_max_val || u32_min_val > u32_max_val) {
@@ -5645,7 +5797,6 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env, · 5 unmodified lines
56455797 }
56465798 } else {
56475799 src_known = tnum_is_const(src_reg.var_off);
5648− dst_known = tnum_is_const(dst_reg->var_off);
56495800 if ((src_known &&
56505801 (smin_val != smax_val || umin_val != umax_val)) ||
56515802 smin_val > smax_val || umin_val > umax_val) {
@@ -6261,8 +6412,25 @@ static int is_branch64_taken(struct bpf_reg_state *reg, u64 val, u8 opcode) · 609 unmodified lines
62616412 static int is_branch_taken(struct bpf_reg_state *reg, u64 val, u8 opcode,
62626413 bool is_jmp32)
62636414 {
6264− if (__is_pointer_value(false, reg))
6265− return -1;
6415+ if (__is_pointer_value(false, reg)) {
6416+ if (!reg_type_not_null(reg->type))
6417+ return -1;
6418+
6419+ /* If pointer is valid tests against zero will fail so we can
6420+ * use this to direct branch taken.
6421+ */
6422+ if (val != 0)
6423+ return -1;
6424+
6425+ switch (opcode) {
6426+ case BPF_JEQ:
6427+ return 0;
6428+ case BPF_JNE:
6429+ return 1;
6430+ default:
6431+ return -1;
6432+ }
6433+ }
62666434
62676435 if (is_jmp32)
62686436 return is_branch32_taken(reg, val, opcode);
@@ -6517,12 +6685,16 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state, · 248 unmodified lines
65176685 if (is_null) {
65186686 reg->type = SCALAR_VALUE;
65196687 } else if (reg->type == PTR_TO_MAP_VALUE_OR_NULL) {
6520− if (reg->map_ptr->inner_map_meta) {
6688+ const struct bpf_map *map = reg->map_ptr;
6689+
6690+ if (map->inner_map_meta) {
65216691 reg->type = CONST_PTR_TO_MAP;
6522− reg->map_ptr = reg->map_ptr->inner_map_meta;
6523− } else if (reg->map_ptr->map_type ==
6524− BPF_MAP_TYPE_XSKMAP) {
6692+ reg->map_ptr = map->inner_map_meta;
6693+ } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) {
65256694 reg->type = PTR_TO_XDP_SOCK;
6695+ } else if (map->map_type == BPF_MAP_TYPE_SOCKMAP ||
6696+ map->map_type == BPF_MAP_TYPE_SOCKHASH) {
6697+ reg->type = PTR_TO_SOCKET;
65266698 } else {
65276699 reg->type = PTR_TO_MAP_VALUE;
65286700 }
@@ -6532,6 +6704,10 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state, · 3 unmodified lines
65326704 reg->type = PTR_TO_SOCK_COMMON;
65336705 } else if (reg->type == PTR_TO_TCP_SOCK_OR_NULL) {
65346706 reg->type = PTR_TO_TCP_SOCK;
6707+ } else if (reg->type == PTR_TO_BTF_ID_OR_NULL) {
6708+ reg->type = PTR_TO_BTF_ID;
6709+ } else if (reg->type == PTR_TO_MEM_OR_NULL) {
6710+ reg->type = PTR_TO_MEM;
65356711 }
65366712 if (is_null) {
65376713 /* We don't need id and ref_obj_id from this point
@@ -6755,7 +6931,11 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, · 217 unmodified lines
67556931 }
67566932
67576933 if (pred >= 0) {
6758− err = mark_chain_precision(env, insn->dst_reg);
6934+ /* If we get here with a dst_reg pointer type it is because
6935+ * above is_branch_taken() special cased the 0 comparison.
6936+ */
6937+ if (!__is_pointer_value(false, dst_reg))
6938+ err = mark_chain_precision(env, insn->dst_reg);
67596939 if (BPF_SRC(insn->code) == BPF_X && !err)
67606940 err = mark_chain_precision(env, insn->src_reg);
67616941 if (err)
@@ -7041,7 +7221,11 @@ static int check_return_code(struct bpf_verifier_env *env) · 279 unmodified lines
70417221 switch (env->prog->type) {
70427222 case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
70437223 if (env->prog->expected_attach_type == BPF_CGROUP_UDP4_RECVMSG ||
7044− env->prog->expected_attach_type == BPF_CGROUP_UDP6_RECVMSG)
7224+ env->prog->expected_attach_type == BPF_CGROUP_UDP6_RECVMSG ||
7225+ env->prog->expected_attach_type == BPF_CGROUP_INET4_GETPEERNAME ||
7226+ env->prog->expected_attach_type == BPF_CGROUP_INET6_GETPEERNAME ||
7227+ env->prog->expected_attach_type == BPF_CGROUP_INET4_GETSOCKNAME ||
7228+ env->prog->expected_attach_type == BPF_CGROUP_INET6_GETSOCKNAME)
70457229 range = tnum_range(1, 1);
70467230 break;
70477231 case BPF_PROG_TYPE_CGROUP_SKB:
@@ -7070,6 +7254,8 @@ static int check_return_code(struct bpf_verifier_env *env) · 22 unmodified lines
70707254 case BPF_TRACE_RAW_TP:
70717255 case BPF_MODIFY_RETURN:
70727256 return 0;
7257+ case BPF_TRACE_ITER:
7258+ break;
70737259 default:
70747260 return -ENOTSUPP;
70757261 }
@@ -7206,7 +7392,7 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env, · 130 unmodified lines
72067392 insn_stack[env->cfg.cur_stack++] = w;
72077393 return 1;
72087394 } else if ((insn_state[w] & 0xF0) == DISCOVERED) {
7209− if (loop_ok && env->allow_ptr_leaks)
7395+ if (loop_ok && env->bpf_capable)
72107396 return 0;
72117397 verbose_linfo(env, t, "%d: ", t);
72127398 verbose_linfo(env, w, "%d: ", w);
@@ -8315,7 +8501,7 @@ next: · 1102 unmodified lines
83158501 if (env->max_states_per_insn < states_cnt)
83168502 env->max_states_per_insn = states_cnt;
83178503
8318− if (!env->allow_ptr_leaks && states_cnt > BPF_COMPLEXITY_LIMIT_STATES)
8504+ if (!env->bpf_capable && states_cnt > BPF_COMPLEXITY_LIMIT_STATES)
83198505 return push_jmp_history(env, cur);
83208506
83218507 if (!add_new_state)
@@ -8402,6 +8588,7 @@ static bool reg_type_mismatch_ok(enum bpf_reg_type type) · 80 unmodified lines
84028588 case PTR_TO_TCP_SOCK_OR_NULL:
84038589 case PTR_TO_XDP_SOCK:
84048590 case PTR_TO_BTF_ID:
8591+ case PTR_TO_BTF_ID_OR_NULL:
84058592 return false;
84068593 default:
84078594 return true;
@@ -8428,6 +8615,7 @@ static bool reg_type_mismatch(enum bpf_reg_type src, enum bpf_reg_type prev) · 20 unmodified lines
84288615
84298616 static int do_check(struct bpf_verifier_env *env)
84308617 {
8618+ bool pop_log = !(env->log.level & BPF_LOG_LEVEL2);
84318619 struct bpf_verifier_state *state = env->cur_state;
84328620 struct bpf_insn *insns = env->prog->insnsi;
84338621 struct bpf_reg_state *regs;
@@ -8704,7 +8892,7 @@ static int do_check(struct bpf_verifier_env *env) · 270 unmodified lines
87048892 process_bpf_exit:
87058893 update_branch_counts(env, env->cur_state);
87068894 err = pop_stack(env, &prev_insn_idx,
8707− &env->insn_idx);
8895+ &env->insn_idx, pop_log);
87088896 if (err < 0) {
87098897 if (err != -ENOENT)
87108898 return err;
@@ -9974,7 +10162,7 @@ static int fixup_bpf_calls(struct bpf_verifier_env *env) · 1263 unmodified lines
997410162 insn->code = BPF_JMP | BPF_TAIL_CALL;
997510163
997610164 aux = &env->insn_aux_data[i + delta];
9977− if (env->allow_ptr_leaks && !expect_blinding &&
10165+ if (env->bpf_capable && !expect_blinding &&
997810166 prog->jit_requested &&
997910167 !bpf_map_key_poisoned(aux) &&
998010168 !bpf_map_ptr_poisoned(aux) &&
@@ -10227,6 +10415,7 @@ static void sanitize_insn_aux_data(struct bpf_verifier_env *env) · 246 unmodified lines
1022710415
1022810416 static int do_check_common(struct bpf_verifier_env *env, int subprog)
1022910417 {
10418+ bool pop_log = !(env->log.level & BPF_LOG_LEVEL2);
1023010419 struct bpf_verifier_state *state;
1023110420 struct bpf_reg_state *regs;
1023210421 int ret, i;
@@ -10289,7 +10478,9 @@ out: · 56 unmodified lines
1028910478 free_verifier_state(env->cur_state, true);
1029010479 env->cur_state = NULL;
1029110480 }
10292− while (!pop_stack(env, NULL, NULL));
10481+ while (!pop_stack(env, NULL, NULL, false));
10482+ if (!ret && pop_log)
10483+ bpf_vlog_reset(&env->log, 0);
1029310484 free_states(env);
1029410485 if (ret)
1029510486 /* clean aux data in case subprog was rejected */
@@ -10445,6 +10636,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 149 unmodified lines
1044510636 struct bpf_prog *tgt_prog = prog->aux->linked_prog;
1044610637 u32 btf_id = prog->aux->attach_btf_id;
1044710638 const char prefix[] = "btf_trace_";
10639+ struct btf_func_model fmodel;
1044810640 int ret = 0, subprog = -1, i;
1044910641 struct bpf_trampoline *tr;
1045010642 const struct btf_type *t;
@@ -10586,6 +10778,22 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 135 unmodified lines
1058610778 prog->aux->attach_func_proto = t;
1058710779 prog->aux->attach_btf_trace = true;
1058810780 return 0;
10781+ case BPF_TRACE_ITER:
10782+ if (!btf_type_is_func(t)) {
10783+ verbose(env, "attach_btf_id %u is not a function\n",
10784+ btf_id);
10785+ return -EINVAL;
10786+ }
10787+ t = btf_type_by_id(btf, t->type);
10788+ if (!btf_type_is_func_proto(t))
10789+ return -EINVAL;
10790+ prog->aux->attach_func_name = tname;
10791+ prog->aux->attach_func_proto = t;
10792+ if (!bpf_iter_prog_supported(prog))
10793+ return -EINVAL;
10794+ ret = btf_distill_func_proto(&env->log, btf, t,
10795+ tname, &fmodel);
10796+ return ret;
1058910797 default:
1059010798 if (!prog_extension)
1059110799 return -EINVAL;
@@ -10696,7 +10904,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, · 104 unmodified lines
1069610904 env->insn_aux_data[i].orig_idx = i;
1069710905 env->prog = *prog;
1069810906 env->ops = bpf_verifier_ops[env->prog->type];
10699− is_priv = capable(CAP_SYS_ADMIN);
10907+ is_priv = bpf_capable();
1070010908
1070110909 if (!btf_vmlinux && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {
1070210910 mutex_lock(&bpf_verifier_lock);
@@ -10737,7 +10945,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, · 34 unmodified lines
1073710945 if (attr->prog_flags & BPF_F_ANY_ALIGNMENT)
1073810946 env->strict_alignment = false;
1073910947
10740− env->allow_ptr_leaks = is_priv;
10948+ env->allow_ptr_leaks = bpf_allow_ptr_leaks();
10949+ env->bypass_spec_v1 = bpf_bypass_spec_v1();
10950+ env->bypass_spec_v4 = bpf_bypass_spec_v4();
10951+ env->bpf_capable = bpf_capable();
1074110952
1074210953 if (is_priv)
1074310954 env->test_state_freq = attr->prog_flags & BPF_F_TEST_STATE_FREQ;
kernel/bpf/xskmap.c+0−265
@@ -1,265 +0,0 @@
1−// SPDX-License-Identifier: GPL-2.0
2−/* XSKMAP used for AF_XDP sockets
3− * Copyright(c) 2018 Intel Corporation.
4− */
5−
6−#include <linux/bpf.h>
7−#include <linux/capability.h>
8−#include <net/xdp_sock.h>
9−#include <linux/slab.h>
10−#include <linux/sched.h>
11−
12−int xsk_map_inc(struct xsk_map *map)
13−{
14− bpf_map_inc(&map->map);
15− return 0;
16−}
17−
18−void xsk_map_put(struct xsk_map *map)
19−{
20− bpf_map_put(&map->map);
21−}
22−
23−static struct xsk_map_node *xsk_map_node_alloc(struct xsk_map *map,
24− struct xdp_sock **map_entry)
25−{
26− struct xsk_map_node *node;
27− int err;
28−
29− node = kzalloc(sizeof(*node), GFP_ATOMIC | __GFP_NOWARN);
30− if (!node)
31− return ERR_PTR(-ENOMEM);
32−
33− err = xsk_map_inc(map);
34− if (err) {
35− kfree(node);
36− return ERR_PTR(err);
37− }
38−
39− node->map = map;
40− node->map_entry = map_entry;
41− return node;
42−}
43−
44−static void xsk_map_node_free(struct xsk_map_node *node)
45−{
46− xsk_map_put(node->map);
47− kfree(node);
48−}
49−
50−static void xsk_map_sock_add(struct xdp_sock *xs, struct xsk_map_node *node)
51−{
52− spin_lock_bh(&xs->map_list_lock);
53− list_add_tail(&node->node, &xs->map_list);
54− spin_unlock_bh(&xs->map_list_lock);
55−}
56−
57−static void xsk_map_sock_delete(struct xdp_sock *xs,
58− struct xdp_sock **map_entry)
59−{
60− struct xsk_map_node *n, *tmp;
61−
62− spin_lock_bh(&xs->map_list_lock);
63− list_for_each_entry_safe(n, tmp, &xs->map_list, node) {
64− if (map_entry == n->map_entry) {
65− list_del(&n->node);
66− xsk_map_node_free(n);
67− }
68− }
69− spin_unlock_bh(&xs->map_list_lock);
70−}
71−
72−static struct bpf_map *xsk_map_alloc(union bpf_attr *attr)
73−{
74− struct bpf_map_memory mem;
75− int err, numa_node;
76− struct xsk_map *m;
77− u64 size;
78−
79− if (!capable(CAP_NET_ADMIN))
80− return ERR_PTR(-EPERM);
81−
82− if (attr->max_entries == 0 || attr->key_size != 4 ||
83− attr->value_size != 4 ||
84− attr->map_flags & ~(BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY))
85− return ERR_PTR(-EINVAL);
86−
87− numa_node = bpf_map_attr_numa_node(attr);
88− size = struct_size(m, xsk_map, attr->max_entries);
89−
90− err = bpf_map_charge_init(&mem, size);
91− if (err < 0)
92− return ERR_PTR(err);
93−
94− m = bpf_map_area_alloc(size, numa_node);
95− if (!m) {
96− bpf_map_charge_finish(&mem);
97− return ERR_PTR(-ENOMEM);
98− }
99−
100− bpf_map_init_from_attr(&m->map, attr);
101− bpf_map_charge_move(&m->map.memory, &mem);
102− spin_lock_init(&m->lock);
103−
104− return &m->map;
105−}
106−
107−static void xsk_map_free(struct bpf_map *map)
108−{
109− struct xsk_map *m = container_of(map, struct xsk_map, map);
110−
111− bpf_clear_redirect_map(map);
112− synchronize_net();
113− bpf_map_area_free(m);
114−}
115−
116−static int xsk_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
117−{
118− struct xsk_map *m = container_of(map, struct xsk_map, map);
119− u32 index = key ? *(u32 *)key : U32_MAX;
120− u32 *next = next_key;
121−
122− if (index >= m->map.max_entries) {
123− *next = 0;
124− return 0;
125− }
126−
127− if (index == m->map.max_entries - 1)
128− return -ENOENT;
129− *next = index + 1;
130− return 0;
131−}
132−
133−static u32 xsk_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
134−{
135− const int ret = BPF_REG_0, mp = BPF_REG_1, index = BPF_REG_2;
136− struct bpf_insn *insn = insn_buf;
137−
138− *insn++ = BPF_LDX_MEM(BPF_W, ret, index, 0);
139− *insn++ = BPF_JMP_IMM(BPF_JGE, ret, map->max_entries, 5);
140− *insn++ = BPF_ALU64_IMM(BPF_LSH, ret, ilog2(sizeof(struct xsk_sock *)));
141− *insn++ = BPF_ALU64_IMM(BPF_ADD, mp, offsetof(struct xsk_map, xsk_map));
142− *insn++ = BPF_ALU64_REG(BPF_ADD, ret, mp);
143− *insn++ = BPF_LDX_MEM(BPF_SIZEOF(struct xsk_sock *), ret, ret, 0);
144− *insn++ = BPF_JMP_IMM(BPF_JA, 0, 0, 1);
145− *insn++ = BPF_MOV64_IMM(ret, 0);
146− return insn - insn_buf;
147−}
148−
149−static void *xsk_map_lookup_elem(struct bpf_map *map, void *key)
150−{
151− WARN_ON_ONCE(!rcu_read_lock_held());
152− return __xsk_map_lookup_elem(map, *(u32 *)key);
153−}
154−
155−static void *xsk_map_lookup_elem_sys_only(struct bpf_map *map, void *key)
156−{
157− return ERR_PTR(-EOPNOTSUPP);
158−}
159−
160−static int xsk_map_update_elem(struct bpf_map *map, void *key, void *value,
161− u64 map_flags)
162−{
163− struct xsk_map *m = container_of(map, struct xsk_map, map);
164− struct xdp_sock *xs, *old_xs, **map_entry;
165− u32 i = *(u32 *)key, fd = *(u32 *)value;
166− struct xsk_map_node *node;
167− struct socket *sock;
168− int err;
169−
170− if (unlikely(map_flags > BPF_EXIST))
171− return -EINVAL;
172− if (unlikely(i >= m->map.max_entries))
173− return -E2BIG;
174−
175− sock = sockfd_lookup(fd, &err);
176− if (!sock)
177− return err;
178−
179− if (sock->sk->sk_family != PF_XDP) {
180− sockfd_put(sock);
181− return -EOPNOTSUPP;
182− }
183−
184− xs = (struct xdp_sock *)sock->sk;
185−
186− if (!xsk_is_setup_for_bpf_map(xs)) {
187− sockfd_put(sock);
188− return -EOPNOTSUPP;
189− }
190−
191− map_entry = &m->xsk_map[i];
192− node = xsk_map_node_alloc(m, map_entry);
193− if (IS_ERR(node)) {
194− sockfd_put(sock);
195− return PTR_ERR(node);
196− }
197−
198− spin_lock_bh(&m->lock);
199− old_xs = READ_ONCE(*map_entry);
200− if (old_xs == xs) {
201− err = 0;
202− goto out;
203− } else if (old_xs && map_flags == BPF_NOEXIST) {
204− err = -EEXIST;
205− goto out;
206− } else if (!old_xs && map_flags == BPF_EXIST) {
207− err = -ENOENT;
208− goto out;
209− }
210− xsk_map_sock_add(xs, node);
211− WRITE_ONCE(*map_entry, xs);
212− if (old_xs)
213− xsk_map_sock_delete(old_xs, map_entry);
214− spin_unlock_bh(&m->lock);
215− sockfd_put(sock);
216− return 0;
217−
218−out:
219− spin_unlock_bh(&m->lock);
220− sockfd_put(sock);
221− xsk_map_node_free(node);
222− return err;
223−}
224−
225−static int xsk_map_delete_elem(struct bpf_map *map, void *key)
226−{
227− struct xsk_map *m = container_of(map, struct xsk_map, map);
228− struct xdp_sock *old_xs, **map_entry;
229− int k = *(u32 *)key;
230−
231− if (k >= map->max_entries)
232− return -EINVAL;
233−
234− spin_lock_bh(&m->lock);
235− map_entry = &m->xsk_map[k];
236− old_xs = xchg(map_entry, NULL);
237− if (old_xs)
238− xsk_map_sock_delete(old_xs, map_entry);
239− spin_unlock_bh(&m->lock);
240−
241− return 0;
242−}
243−
244−void xsk_map_try_sock_delete(struct xsk_map *map, struct xdp_sock *xs,
245− struct xdp_sock **map_entry)
246−{
247− spin_lock_bh(&map->lock);
248− if (READ_ONCE(*map_entry) == xs) {
249− WRITE_ONCE(*map_entry, NULL);
250− xsk_map_sock_delete(xs, map_entry);
251− }
252− spin_unlock_bh(&map->lock);
253−}
254−
255−const struct bpf_map_ops xsk_map_ops = {
256− .map_alloc = xsk_map_alloc,
257− .map_free = xsk_map_free,
258− .map_get_next_key = xsk_map_get_next_key,
259− .map_lookup_elem = xsk_map_lookup_elem,
260− .map_gen_lookup = xsk_map_gen_lookup,
261− .map_lookup_elem_sys_only = xsk_map_lookup_elem_sys_only,
262− .map_update_elem = xsk_map_update_elem,
263− .map_delete_elem = xsk_map_delete_elem,
264− .map_check_btf = map_check_no_btf,
265−};
kernel/cgroup/cgroup.c+0−27
@@ -6508,33 +6508,6 @@ int cgroup_bpf_attach(struct cgroup *cgrp, · 6507 unmodified lines
65086508 return ret;
65096509 }
65106510
6511−int cgroup_bpf_replace(struct bpf_link *link, struct bpf_prog *old_prog,
6512− struct bpf_prog *new_prog)
6513−{
6514− struct bpf_cgroup_link *cg_link;
6515− int ret;
6516−
6517− if (link->ops != &bpf_cgroup_link_lops)
6518− return -EINVAL;
6519−
6520− cg_link = container_of(link, struct bpf_cgroup_link, link);
6521−
6522− mutex_lock(&cgroup_mutex);
6523− /* link might have been auto-released by dying cgroup, so fail */
6524− if (!cg_link->cgroup) {
6525− ret = -EINVAL;
6526− goto out_unlock;
6527− }
6528− if (old_prog && link->prog != old_prog) {
6529− ret = -EPERM;
6530− goto out_unlock;
6531− }
6532− ret = __cgroup_bpf_replace(cg_link->cgroup, cg_link, new_prog);
6533−out_unlock:
6534− mutex_unlock(&cgroup_mutex);
6535− return ret;
6536−}
6537−
65386511 int cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog,
65396512 enum bpf_attach_type type)
65406513 {
kernel/events/callchain.c+1−1
@@ -236,7 +236,7 @@ exit_put: · 235 unmodified lines
236236 * sysctl_perf_event_max_contexts_per_stack.
237237 */
238238 int perf_event_max_stack_handler(struct ctl_table *table, int write,
239− void __user *buffer, size_t *lenp, loff_t *ppos)
239+ void *buffer, size_t *lenp, loff_t *ppos)
240240 {
241241 int *value = table->data;
242242 int new_value = *value, ret;
kernel/events/core.c+2−4
@@ -442,8 +442,7 @@ static void update_perf_cpu_limits(void) · 441 unmodified lines
442442 static bool perf_rotate_context(struct perf_cpu_context *cpuctx);
443443
444444 int perf_proc_update_handler(struct ctl_table *table, int write,
445− void __user *buffer, size_t *lenp,
446− loff_t *ppos)
445+ void *buffer, size_t *lenp, loff_t *ppos)
447446 {
448447 int ret;
449448 int perf_cpu = sysctl_perf_cpu_time_max_percent;
@@ -467,8 +466,7 @@ int perf_proc_update_handler(struct ctl_table *table, int write, · 17 unmodified lines
467466 int sysctl_perf_cpu_time_max_percent __read_mostly = DEFAULT_CPU_TIME_MAX_PERCENT;
468467
469468 int perf_cpu_time_max_percent_handler(struct ctl_table *table, int write,
470− void __user *buffer, size_t *lenp,
471− loff_t *ppos)
469+ void *buffer, size_t *lenp, loff_t *ppos)
472470 {
473471 int ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
474472
kernel/kprobes.c+1−1
@@ -892,7 +892,7 @@ static void unoptimize_all_kprobes(void) · 891 unmodified lines
892892 static DEFINE_MUTEX(kprobe_sysctl_mutex);
893893 int sysctl_kprobes_optimization;
894894 int proc_kprobes_optimization_handler(struct ctl_table *table, int write,
895− void __user *buffer, size_t *length,
895+ void *buffer, size_t *length,
896896 loff_t *ppos)
897897 {
898898 int ret;
kernel/latencytop.c+2−2
@@ -269,8 +269,8 @@ static int __init init_lstats_procfs(void) · 268 unmodified lines
269269 return 0;
270270 }
271271
272−int sysctl_latencytop(struct ctl_table *table, int write,
273− void __user *buffer, size_t *lenp, loff_t *ppos)
272+int sysctl_latencytop(struct ctl_table *table, int write, void *buffer,
273+ size_t *lenp, loff_t *ppos)
274274 {
275275 int err;
276276
kernel/module.c+3−0
@@ -4,6 +4,9 @@ · 3 unmodified lines
44 Copyright (C) 2001 Rusty Russell, 2002, 2010 Rusty Russell IBM.
55
66 */
7+
8+#define INCLUDE_VERMAGIC
9+
710 #include <linux/export.h>
811 #include <linux/extable.h>
912 #include <linux/moduleloader.h>
kernel/pid_namespace.c+1−1
@@ -263,7 +263,7 @@ void zap_pid_ns_processes(struct pid_namespace *pid_ns) · 262 unmodified lines
263263
264264 #ifdef CONFIG_CHECKPOINT_RESTORE
265265 static int pid_ns_ctl_handler(struct ctl_table *table, int write,
266− void __user *buffer, size_t *lenp, loff_t *ppos)
266+ void *buffer, size_t *lenp, loff_t *ppos)
267267 {
268268 struct pid_namespace *pid_ns = task_active_pid_ns(current);
269269 struct ctl_table tmp = *table;
kernel/printk/printk.c+1−1
@@ -173,7 +173,7 @@ __setup("printk.devkmsg=", control_devkmsg); · 172 unmodified lines
173173 char devkmsg_log_str[DEVKMSG_STR_MAX_SIZE] = "ratelimit";
174174
175175 int devkmsg_sysctl_set_loglvl(struct ctl_table *table, int write,
176− void __user *buffer, size_t *lenp, loff_t *ppos)
176+ void *buffer, size_t *lenp, loff_t *ppos)
177177 {
178178 char old_str[DEVKMSG_STR_MAX_SIZE];
179179 unsigned int old;
kernel/sched/core.c+4−5
@@ -1111,8 +1111,7 @@ static void uclamp_update_root_tg(void) { } · 1110 unmodified lines
11111111 #endif
11121112
11131113 int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,
1114− void __user *buffer, size_t *lenp,
1115− loff_t *ppos)
1114+ void *buffer, size_t *lenp, loff_t *ppos)
11161115 {
11171116 bool update_root_tg = false;
11181117 int old_min, old_max;
@@ -2782,7 +2781,7 @@ void set_numabalancing_state(bool enabled) · 1663 unmodified lines
27822781
27832782 #ifdef CONFIG_PROC_SYSCTL
27842783 int sysctl_numa_balancing(struct ctl_table *table, int write,
2785− void __user *buffer, size_t *lenp, loff_t *ppos)
2784+ void *buffer, size_t *lenp, loff_t *ppos)
27862785 {
27872786 struct ctl_table t;
27882787 int err;
@@ -2856,8 +2855,8 @@ static void __init init_schedstats(void) · 67 unmodified lines
28562855 }
28572856
28582857 #ifdef CONFIG_PROC_SYSCTL
2859−int sysctl_schedstats(struct ctl_table *table, int write,
2860− void __user *buffer, size_t *lenp, loff_t *ppos)
2858+int sysctl_schedstats(struct ctl_table *table, int write, void *buffer,
2859+ size_t *lenp, loff_t *ppos)
28612860 {
28622861 struct ctl_table t;
28632862 int err;
kernel/sched/fair.c+1−2
@@ -645,8 +645,7 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq) · 644 unmodified lines
645645 */
646646
647647 int sched_proc_update_handler(struct ctl_table *table, int write,
648− void __user *buffer, size_t *lenp,
649− loff_t *ppos)
648+ void *buffer, size_t *lenp, loff_t *ppos)
650649 {
651650 int ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
652651 unsigned int factor = get_update_sysctl_factor();
kernel/sched/rt.c+4−6
@@ -2724,9 +2724,8 @@ static void sched_rt_do_global(void) · 2723 unmodified lines
27242724 def_rt_bandwidth.rt_period = ns_to_ktime(global_rt_period());
27252725 }
27262726
2727−int sched_rt_handler(struct ctl_table *table, int write,
2728− void __user *buffer, size_t *lenp,
2729− loff_t *ppos)
2727+int sched_rt_handler(struct ctl_table *table, int write, void *buffer,
2728+ size_t *lenp, loff_t *ppos)
27302729 {
27312730 int old_period, old_runtime;
27322731 static DEFINE_MUTEX(mutex);
@@ -2764,9 +2763,8 @@ undo: · 31 unmodified lines
27642763 return ret;
27652764 }
27662765
2767−int sched_rr_handler(struct ctl_table *table, int write,
2768− void __user *buffer, size_t *lenp,
2769− loff_t *ppos)
2766+int sched_rr_handler(struct ctl_table *table, int write, void *buffer,
2767+ size_t *lenp, loff_t *ppos)
27702768 {
27712769 int ret;
27722770 static DEFINE_MUTEX(mutex);
kernel/sched/topology.c+1−1
@@ -199,7 +199,7 @@ bool sched_energy_update; · 198 unmodified lines
199199
200200 #ifdef CONFIG_PROC_SYSCTL
201201 int sched_energy_aware_handler(struct ctl_table *table, int write,
202− void __user *buffer, size_t *lenp, loff_t *ppos)
202+ void *buffer, size_t *lenp, loff_t *ppos)
203203 {
204204 int ret, state;
205205
kernel/seccomp.c+1−1
@@ -1776,7 +1776,7 @@ static void audit_actions_logged(u32 actions_logged, u32 old_actions_logged, · 1775 unmodified lines
17761776 }
17771777
17781778 static int seccomp_actions_logged_handler(struct ctl_table *ro_table, int write,
1779− void __user *buffer, size_t *lenp,
1779+ void *buffer, size_t *lenp,
17801780 loff_t *ppos)
17811781 {
17821782 int ret;
kernel/sysctl.c+1891−1990
@@ -68,6 +68,9 @@ · 67 unmodified lines
6868 #include <linux/bpf.h>
6969 #include <linux/mount.h>
7070 #include <linux/userfaultfd_k.h>
71+#include <linux/coredump.h>
72+#include <linux/latencytop.h>
73+#include <linux/pid.h>
7174
7275 #include "../lib/kstrtox.h"
7376
@@ -103,22 +106,6 @@ · 29 unmodified lines
103106
104107 #if defined(CONFIG_SYSCTL)
105108
106−/* External variables not in a header file. */
107−extern int suid_dumpable;
108−#ifdef CONFIG_COREDUMP
109−extern int core_uses_pid;
110−extern char core_pattern[];
111−extern unsigned int core_pipe_limit;
112−#endif
113−extern int pid_max;
114−extern int pid_max_min, pid_max_max;
115−extern int percpu_pagelist_fraction;
116−extern int latencytop_enabled;
117−extern unsigned int sysctl_nr_open_min, sysctl_nr_open_max;
118−#ifndef CONFIG_MMU
119−extern int sysctl_nr_trim_pages;
120−#endif
121−
122109 /* Constants used for minimum and maximum */
123110 #ifdef CONFIG_LOCKUP_DETECTOR
124111 static int sixty = 60;
@@ -160,24 +147,6 @@ static unsigned long hung_task_timeout_max = (LONG_MAX/HZ); · 35 unmodified lines
160147 #ifdef CONFIG_INOTIFY_USER
161148 #include <linux/inotify.h>
162149 #endif
163−#ifdef CONFIG_SPARC
164−#endif
165−
166−#ifdef CONFIG_PARISC
167−extern int pwrsw_enabled;
168−#endif
169−
170−#ifdef CONFIG_SYSCTL_ARCH_UNALIGN_ALLOW
171−extern int unaligned_enabled;
172−#endif
173−
174−#ifdef CONFIG_IA64
175−extern int unaligned_dump_stack;
176−#endif
177−
178−#ifdef CONFIG_SYSCTL_ARCH_UNALIGN_NO_WARN
179−extern int no_unaligned_warning;
180−#endif
181150
182151 #ifdef CONFIG_PROC_SYSCTL
183152
@@ -207,87 +176,13 @@ enum sysctl_writes_mode { · 23 unmodified lines
207176 };
208177
209178 static enum sysctl_writes_mode sysctl_writes_strict = SYSCTL_WRITES_STRICT;
210−
211−static int proc_do_cad_pid(struct ctl_table *table, int write,
212− void __user *buffer, size_t *lenp, loff_t *ppos);
213−static int proc_taint(struct ctl_table *table, int write,
214− void __user *buffer, size_t *lenp, loff_t *ppos);
215−#ifdef CONFIG_COMPACTION
216−static int proc_dointvec_minmax_warn_RT_change(struct ctl_table *table,
217− int write, void __user *buffer,
218− size_t *lenp, loff_t *ppos);
219−#endif
220−#endif
221−
222−#ifdef CONFIG_PRINTK
223−static int proc_dointvec_minmax_sysadmin(struct ctl_table *table, int write,
224− void __user *buffer, size_t *lenp, loff_t *ppos);
225−#endif
226−
227−static int proc_dointvec_minmax_coredump(struct ctl_table *table, int write,
228− void __user *buffer, size_t *lenp, loff_t *ppos);
229−#ifdef CONFIG_COREDUMP
230−static int proc_dostring_coredump(struct ctl_table *table, int write,
231− void __user *buffer, size_t *lenp, loff_t *ppos);
232−#endif
233−static int proc_dopipe_max_size(struct ctl_table *table, int write,
234− void __user *buffer, size_t *lenp, loff_t *ppos);
235−
236−#ifdef CONFIG_MAGIC_SYSRQ
237−static int sysrq_sysctl_handler(struct ctl_table *table, int write,
238− void __user *buffer, size_t *lenp, loff_t *ppos);
239−#endif
240−
241−static struct ctl_table kern_table[];
242−static struct ctl_table vm_table[];
243−static struct ctl_table fs_table[];
244−static struct ctl_table debug_table[];
245−static struct ctl_table dev_table[];
246−extern struct ctl_table random_table[];
247−#ifdef CONFIG_EPOLL
248−extern struct ctl_table epoll_table[];
249−#endif
250−
251−#ifdef CONFIG_FW_LOADER_USER_HELPER
252−extern struct ctl_table firmware_config_table[];
253−#endif
179+#endif /* CONFIG_PROC_SYSCTL */
254180
255181 #if defined(HAVE_ARCH_PICK_MMAP_LAYOUT) || \
256182 defined(CONFIG_ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT)
257183 int sysctl_legacy_va_layout;
258184 #endif
259185
260−/* The default sysctl tables: */
261−
262−static struct ctl_table sysctl_base_table[] = {
263− {
264− .procname = "kernel",
265− .mode = 0555,
266− .child = kern_table,
267− },
268− {
269− .procname = "vm",
270− .mode = 0555,
271− .child = vm_table,
272− },
273− {
274− .procname = "fs",
275− .mode = 0555,
276− .child = fs_table,
277− },
278− {
279− .procname = "debug",
280− .mode = 0555,
281− .child = debug_table,
282− },
283− {
284− .procname = "dev",
285− .mode = 0555,
286− .child = dev_table,
287− },
288− { }
289−};
290−
291186 #ifdef CONFIG_SCHED_DEBUG
292187 static int min_sched_granularity_ns = 100000; /* 100 usecs */
293188 static int max_sched_granularity_ns = NSEC_PER_SEC; /* 1 second */
@@ -304,1687 +199,53 @@ static int min_extfrag_threshold; · 10 unmodified lines
304199 static int max_extfrag_threshold = 1000;
305200 #endif
306201
307−static struct ctl_table kern_table[] = {
308− {
309− .procname = "sched_child_runs_first",
310− .data = &sysctl_sched_child_runs_first,
311− .maxlen = sizeof(unsigned int),
312− .mode = 0644,
313− .proc_handler = proc_dointvec,
314− },
315−#ifdef CONFIG_SCHED_DEBUG
316− {
317− .procname = "sched_min_granularity_ns",
318− .data = &sysctl_sched_min_granularity,
319− .maxlen = sizeof(unsigned int),
320− .mode = 0644,
321− .proc_handler = sched_proc_update_handler,
322− .extra1 = &min_sched_granularity_ns,
323− .extra2 = &max_sched_granularity_ns,
324− },
325− {
326− .procname = "sched_latency_ns",
327− .data = &sysctl_sched_latency,
328− .maxlen = sizeof(unsigned int),
329− .mode = 0644,
330− .proc_handler = sched_proc_update_handler,
331− .extra1 = &min_sched_granularity_ns,
332− .extra2 = &max_sched_granularity_ns,
333− },
334− {
335− .procname = "sched_wakeup_granularity_ns",
336− .data = &sysctl_sched_wakeup_granularity,
337− .maxlen = sizeof(unsigned int),
338− .mode = 0644,
339− .proc_handler = sched_proc_update_handler,
340− .extra1 = &min_wakeup_granularity_ns,
341− .extra2 = &max_wakeup_granularity_ns,
342− },
343−#ifdef CONFIG_SMP
344− {
345− .procname = "sched_tunable_scaling",
346− .data = &sysctl_sched_tunable_scaling,
347− .maxlen = sizeof(enum sched_tunable_scaling),
348− .mode = 0644,
349− .proc_handler = sched_proc_update_handler,
350− .extra1 = &min_sched_tunable_scaling,
351− .extra2 = &max_sched_tunable_scaling,
352− },
353− {
354− .procname = "sched_migration_cost_ns",
355− .data = &sysctl_sched_migration_cost,
356− .maxlen = sizeof(unsigned int),
357− .mode = 0644,
358− .proc_handler = proc_dointvec,
359− },
360− {
361− .procname = "sched_nr_migrate",
362− .data = &sysctl_sched_nr_migrate,
363− .maxlen = sizeof(unsigned int),
364− .mode = 0644,
365− .proc_handler = proc_dointvec,
366− },
367−#ifdef CONFIG_SCHEDSTATS
368− {
369− .procname = "sched_schedstats",
370− .data = NULL,
371− .maxlen = sizeof(unsigned int),
372− .mode = 0644,
373− .proc_handler = sysctl_schedstats,
374− .extra1 = SYSCTL_ZERO,
375− .extra2 = SYSCTL_ONE,
376− },
377−#endif /* CONFIG_SCHEDSTATS */
378−#endif /* CONFIG_SMP */
379−#ifdef CONFIG_NUMA_BALANCING
380− {
381− .procname = "numa_balancing_scan_delay_ms",
382− .data = &sysctl_numa_balancing_scan_delay,
383− .maxlen = sizeof(unsigned int),
384− .mode = 0644,
385− .proc_handler = proc_dointvec,
386− },
387− {
388− .procname = "numa_balancing_scan_period_min_ms",
389− .data = &sysctl_numa_balancing_scan_period_min,
390− .maxlen = sizeof(unsigned int),
391− .mode = 0644,
392− .proc_handler = proc_dointvec,
393− },
394− {
395− .procname = "numa_balancing_scan_period_max_ms",
396− .data = &sysctl_numa_balancing_scan_period_max,
397− .maxlen = sizeof(unsigned int),
398− .mode = 0644,
399− .proc_handler = proc_dointvec,
400− },
401− {
402− .procname = "numa_balancing_scan_size_mb",
403− .data = &sysctl_numa_balancing_scan_size,
404− .maxlen = sizeof(unsigned int),
405− .mode = 0644,
406− .proc_handler = proc_dointvec_minmax,
407− .extra1 = SYSCTL_ONE,
408− },
409− {
410− .procname = "numa_balancing",
411− .data = NULL, /* filled in by handler */
412− .maxlen = sizeof(unsigned int),
413− .mode = 0644,
414− .proc_handler = sysctl_numa_balancing,
415− .extra1 = SYSCTL_ZERO,
416− .extra2 = SYSCTL_ONE,
417− },
418−#endif /* CONFIG_NUMA_BALANCING */
419−#endif /* CONFIG_SCHED_DEBUG */
420− {
421− .procname = "sched_rt_period_us",
422− .data = &sysctl_sched_rt_period,
423− .maxlen = sizeof(unsigned int),
424− .mode = 0644,
425− .proc_handler = sched_rt_handler,
426− },
427− {
428− .procname = "sched_rt_runtime_us",
429− .data = &sysctl_sched_rt_runtime,
430− .maxlen = sizeof(int),
431− .mode = 0644,
432− .proc_handler = sched_rt_handler,
433− },
434− {
435− .procname = "sched_rr_timeslice_ms",
436− .data = &sysctl_sched_rr_timeslice,
437− .maxlen = sizeof(int),
438− .mode = 0644,
439− .proc_handler = sched_rr_handler,
440− },
441−#ifdef CONFIG_UCLAMP_TASK
442− {
443− .procname = "sched_util_clamp_min",
444− .data = &sysctl_sched_uclamp_util_min,
445− .maxlen = sizeof(unsigned int),
446− .mode = 0644,
447− .proc_handler = sysctl_sched_uclamp_handler,
448− },
449− {
450− .procname = "sched_util_clamp_max",
451− .data = &sysctl_sched_uclamp_util_max,
452− .maxlen = sizeof(unsigned int),
453− .mode = 0644,
454− .proc_handler = sysctl_sched_uclamp_handler,
455− },
456−#endif
457−#ifdef CONFIG_SCHED_AUTOGROUP
458− {
459− .procname = "sched_autogroup_enabled",
460− .data = &sysctl_sched_autogroup_enabled,
461− .maxlen = sizeof(unsigned int),
462− .mode = 0644,
463− .proc_handler = proc_dointvec_minmax,
464− .extra1 = SYSCTL_ZERO,
465− .extra2 = SYSCTL_ONE,
466− },
467−#endif
468−#ifdef CONFIG_CFS_BANDWIDTH
469− {
470− .procname = "sched_cfs_bandwidth_slice_us",
471− .data = &sysctl_sched_cfs_bandwidth_slice,
472− .maxlen = sizeof(unsigned int),
473− .mode = 0644,
474− .proc_handler = proc_dointvec_minmax,
475− .extra1 = SYSCTL_ONE,
476− },
477−#endif
478−#if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL)
479− {
480− .procname = "sched_energy_aware",
481− .data = &sysctl_sched_energy_aware,
482− .maxlen = sizeof(unsigned int),
483− .mode = 0644,
484− .proc_handler = sched_energy_aware_handler,
485− .extra1 = SYSCTL_ZERO,
486− .extra2 = SYSCTL_ONE,
487− },
488−#endif
489−#ifdef CONFIG_PROVE_LOCKING
490− {
491− .procname = "prove_locking",
492− .data = &prove_locking,
493− .maxlen = sizeof(int),
494− .mode = 0644,
495− .proc_handler = proc_dointvec,
496− },
497−#endif
498−#ifdef CONFIG_LOCK_STAT
499− {
500− .procname = "lock_stat",
501− .data = &lock_stat,
502− .maxlen = sizeof(int),
503− .mode = 0644,
504− .proc_handler = proc_dointvec,
505− },
506−#endif
507− {
508− .procname = "panic",
509− .data = &panic_timeout,
510− .maxlen = sizeof(int),
511− .mode = 0644,
512− .proc_handler = proc_dointvec,
513− },
514−#ifdef CONFIG_COREDUMP
515− {
516− .procname = "core_uses_pid",
517− .data = &core_uses_pid,
518− .maxlen = sizeof(int),
519− .mode = 0644,
520− .proc_handler = proc_dointvec,
521− },
522− {
523− .procname = "core_pattern",
524− .data = core_pattern,
525− .maxlen = CORENAME_MAX_SIZE,
526− .mode = 0644,
527− .proc_handler = proc_dostring_coredump,
528− },
529− {
530− .procname = "core_pipe_limit",
531− .data = &core_pipe_limit,
532− .maxlen = sizeof(unsigned int),
533− .mode = 0644,
534− .proc_handler = proc_dointvec,
535− },
202+#endif /* CONFIG_SYSCTL */
203+
204+#if defined(CONFIG_BPF_SYSCALL) && defined(CONFIG_SYSCTL)
205+static int bpf_stats_handler(struct ctl_table *table, int write,
206+ void __user *buffer, size_t *lenp,
207+ loff_t *ppos)
208+{
209+ struct static_key *key = (struct static_key *)table->data;
210+ static int saved_val;
211+ int val, ret;
212+ struct ctl_table tmp = {
213+ .data = &val,
214+ .maxlen = sizeof(val),
215+ .mode = table->mode,
216+ .extra1 = SYSCTL_ZERO,
217+ .extra2 = SYSCTL_ONE,
218+ };
219+
220+ if (write && !capable(CAP_SYS_ADMIN))
221+ return -EPERM;
222+
223+ mutex_lock(&bpf_stats_enabled_mutex);
224+ val = saved_val;
225+ ret = proc_dointvec_minmax(&tmp, write, buffer, lenp, ppos);
226+ if (write && !ret && val != saved_val) {
227+ if (val)
228+ static_key_slow_inc(key);
229+ else
230+ static_key_slow_dec(key);
231+ saved_val = val;
232+ }
233+ mutex_unlock(&bpf_stats_enabled_mutex);
234+ return ret;
235+}
536236 #endif
537−#ifdef CONFIG_PROC_SYSCTL
538− {
539− .procname = "tainted",
540− .maxlen = sizeof(long),
541− .mode = 0644,
542− .proc_handler = proc_taint,
543− },
544− {
545− .procname = "sysctl_writes_strict",
546− .data = &sysctl_writes_strict,
547− .maxlen = sizeof(int),
548− .mode = 0644,
549− .proc_handler = proc_dointvec_minmax,
550− .extra1 = &neg_one,
551− .extra2 = SYSCTL_ONE,
552− },
553−#endif
554−#ifdef CONFIG_LATENCYTOP
555− {
556− .procname = "latencytop",
557− .data = &latencytop_enabled,
558− .maxlen = sizeof(int),
559− .mode = 0644,
560− .proc_handler = sysctl_latencytop,
561− },
562−#endif
563−#ifdef CONFIG_BLK_DEV_INITRD
564− {
565− .procname = "real-root-dev",
566− .data = &real_root_dev,
567− .maxlen = sizeof(int),
568− .mode = 0644,
569− .proc_handler = proc_dointvec,
570− },
571−#endif
572− {
573− .procname = "print-fatal-signals",
574− .data = &print_fatal_signals,
575− .maxlen = sizeof(int),
576− .mode = 0644,
577− .proc_handler = proc_dointvec,
578− },
579−#ifdef CONFIG_SPARC
580− {
581− .procname = "reboot-cmd",
582− .data = reboot_command,
583− .maxlen = 256,
584− .mode = 0644,
585− .proc_handler = proc_dostring,
586− },
587− {
588− .procname = "stop-a",
589− .data = &stop_a_enabled,
590− .maxlen = sizeof (int),
591− .mode = 0644,
592− .proc_handler = proc_dointvec,
593− },
594− {
595− .procname = "scons-poweroff",
596− .data = &scons_pwroff,
597− .maxlen = sizeof (int),
598− .mode = 0644,
599− .proc_handler = proc_dointvec,
600− },
601−#endif
602−#ifdef CONFIG_SPARC64
603− {
604− .procname = "tsb-ratio",
605− .data = &sysctl_tsb_ratio,
606− .maxlen = sizeof (int),
607− .mode = 0644,
608− .proc_handler = proc_dointvec,
609− },
610−#endif
611−#ifdef CONFIG_PARISC
612− {
613− .procname = "soft-power",
614− .data = &pwrsw_enabled,
615− .maxlen = sizeof (int),
616− .mode = 0644,
617− .proc_handler = proc_dointvec,
618− },
619−#endif
620−#ifdef CONFIG_SYSCTL_ARCH_UNALIGN_ALLOW
621− {
622− .procname = "unaligned-trap",
623− .data = &unaligned_enabled,
624− .maxlen = sizeof (int),
625− .mode = 0644,
626− .proc_handler = proc_dointvec,
627− },
628−#endif
629− {
630− .procname = "ctrl-alt-del",
631− .data = &C_A_D,
632− .maxlen = sizeof(int),
633− .mode = 0644,
634− .proc_handler = proc_dointvec,
635− },
636−#ifdef CONFIG_FUNCTION_TRACER
637− {
638− .procname = "ftrace_enabled",
639− .data = &ftrace_enabled,
640− .maxlen = sizeof(int),
641− .mode = 0644,
642− .proc_handler = ftrace_enable_sysctl,
643− },
644−#endif
645−#ifdef CONFIG_STACK_TRACER
646− {
647− .procname = "stack_tracer_enabled",
648− .data = &stack_tracer_enabled,
649− .maxlen = sizeof(int),
650− .mode = 0644,
651− .proc_handler = stack_trace_sysctl,
652− },
653−#endif
654−#ifdef CONFIG_TRACING
655− {
656− .procname = "ftrace_dump_on_oops",
657− .data = &ftrace_dump_on_oops,
658− .maxlen = sizeof(int),
659− .mode = 0644,
660− .proc_handler = proc_dointvec,
661− },
662− {
663− .procname = "traceoff_on_warning",
664− .data = &__disable_trace_on_warning,
665− .maxlen = sizeof(__disable_trace_on_warning),
666− .mode = 0644,
667− .proc_handler = proc_dointvec,
668− },
669− {
670− .procname = "tracepoint_printk",
671− .data = &tracepoint_printk,
672− .maxlen = sizeof(tracepoint_printk),
673− .mode = 0644,
674− .proc_handler = tracepoint_printk_sysctl,
675− },
676−#endif
677−#ifdef CONFIG_KEXEC_CORE
678− {
679− .procname = "kexec_load_disabled",
680− .data = &kexec_load_disabled,
681− .maxlen = sizeof(int),
682− .mode = 0644,
683− /* only handle a transition from default "0" to "1" */
684− .proc_handler = proc_dointvec_minmax,
685− .extra1 = SYSCTL_ONE,
686− .extra2 = SYSCTL_ONE,
687− },
688−#endif
689−#ifdef CONFIG_MODULES
690− {
691− .procname = "modprobe",
692− .data = &modprobe_path,
693− .maxlen = KMOD_PATH_LEN,
694− .mode = 0644,
695− .proc_handler = proc_dostring,
696− },
697− {
698− .procname = "modules_disabled",
699− .data = &modules_disabled,
700− .maxlen = sizeof(int),
701− .mode = 0644,
702− /* only handle a transition from default "0" to "1" */
703− .proc_handler = proc_dointvec_minmax,
704− .extra1 = SYSCTL_ONE,
705− .extra2 = SYSCTL_ONE,
706− },
707−#endif
708−#ifdef CONFIG_UEVENT_HELPER
709− {
710− .procname = "hotplug",
711− .data = &uevent_helper,
712− .maxlen = UEVENT_HELPER_PATH_LEN,
713− .mode = 0644,
714− .proc_handler = proc_dostring,
715− },
716−#endif
717−#ifdef CONFIG_CHR_DEV_SG
718− {
719− .procname = "sg-big-buff",
720− .data = &sg_big_buff,
721− .maxlen = sizeof (int),
722− .mode = 0444,
723− .proc_handler = proc_dointvec,
724− },
725−#endif
726−#ifdef CONFIG_BSD_PROCESS_ACCT
727− {
728− .procname = "acct",
729− .data = &acct_parm,
730− .maxlen = 3*sizeof(int),
731− .mode = 0644,
732− .proc_handler = proc_dointvec,
733− },
734−#endif
735−#ifdef CONFIG_MAGIC_SYSRQ
736− {
737− .procname = "sysrq",
738− .data = NULL,
739− .maxlen = sizeof (int),
740− .mode = 0644,
741− .proc_handler = sysrq_sysctl_handler,
742− },
743−#endif
744−#ifdef CONFIG_PROC_SYSCTL
745− {
746− .procname = "cad_pid",
747− .data = NULL,
748− .maxlen = sizeof (int),
749− .mode = 0600,
750− .proc_handler = proc_do_cad_pid,
751− },
752−#endif
753− {
754− .procname = "threads-max",
755− .data = NULL,
756− .maxlen = sizeof(int),
757− .mode = 0644,
758− .proc_handler = sysctl_max_threads,
759− },
760− {
761− .procname = "random",
762− .mode = 0555,
763− .child = random_table,
764− },
765− {
766− .procname = "usermodehelper",
767− .mode = 0555,
768− .child = usermodehelper_table,
769− },
770−#ifdef CONFIG_FW_LOADER_USER_HELPER
771− {
772− .procname = "firmware_config",
773− .mode = 0555,
774− .child = firmware_config_table,
775− },
776−#endif
777− {
778− .procname = "overflowuid",
779− .data = &overflowuid,
780− .maxlen = sizeof(int),
781− .mode = 0644,
782− .proc_handler = proc_dointvec_minmax,
783− .extra1 = &minolduid,
784− .extra2 = &maxolduid,
785− },
786− {
787− .procname = "overflowgid",
788− .data = &overflowgid,
789− .maxlen = sizeof(int),
790− .mode = 0644,
791− .proc_handler = proc_dointvec_minmax,
792− .extra1 = &minolduid,
793− .extra2 = &maxolduid,
794− },
795−#ifdef CONFIG_S390
796− {
797− .procname = "userprocess_debug",
798− .data = &show_unhandled_signals,
799− .maxlen = sizeof(int),
800− .mode = 0644,
801− .proc_handler = proc_dointvec,
802− },
803−#endif
804− {
805− .procname = "pid_max",
806− .data = &pid_max,
807− .maxlen = sizeof (int),
808− .mode = 0644,
809− .proc_handler = proc_dointvec_minmax,
810− .extra1 = &pid_max_min,
811− .extra2 = &pid_max_max,
812− },
813− {
814− .procname = "panic_on_oops",
815− .data = &panic_on_oops,
816− .maxlen = sizeof(int),
817− .mode = 0644,
818− .proc_handler = proc_dointvec,
819− },
820− {
821− .procname = "panic_print",
822− .data = &panic_print,
823− .maxlen = sizeof(unsigned long),
824− .mode = 0644,
825− .proc_handler = proc_doulongvec_minmax,
826− },
827−#if defined CONFIG_PRINTK
828− {
829− .procname = "printk",
830− .data = &console_loglevel,
831− .maxlen = 4*sizeof(int),
832− .mode = 0644,
833− .proc_handler = proc_dointvec,
834− },
835− {
836− .procname = "printk_ratelimit",
837− .data = &printk_ratelimit_state.interval,
838− .maxlen = sizeof(int),
839− .mode = 0644,
840− .proc_handler = proc_dointvec_jiffies,
841− },
842− {
843− .procname = "printk_ratelimit_burst",
844− .data = &printk_ratelimit_state.burst,
845− .maxlen = sizeof(int),
846− .mode = 0644,
847− .proc_handler = proc_dointvec,
848− },
849− {
850− .procname = "printk_delay",
851− .data = &printk_delay_msec,
852− .maxlen = sizeof(int),
853− .mode = 0644,
854− .proc_handler = proc_dointvec_minmax,
855− .extra1 = SYSCTL_ZERO,
856− .extra2 = &ten_thousand,
857− },
858− {
859− .procname = "printk_devkmsg",
860− .data = devkmsg_log_str,
861− .maxlen = DEVKMSG_STR_MAX_SIZE,
862− .mode = 0644,
863− .proc_handler = devkmsg_sysctl_set_loglvl,
864− },
865− {
866− .procname = "dmesg_restrict",
867− .data = &dmesg_restrict,
868− .maxlen = sizeof(int),
869− .mode = 0644,
870− .proc_handler = proc_dointvec_minmax_sysadmin,
871− .extra1 = SYSCTL_ZERO,
872− .extra2 = SYSCTL_ONE,
873− },
874− {
875− .procname = "kptr_restrict",
876− .data = &kptr_restrict,
877− .maxlen = sizeof(int),
878− .mode = 0644,
879− .proc_handler = proc_dointvec_minmax_sysadmin,
880− .extra1 = SYSCTL_ZERO,
881− .extra2 = &two,
882− },
883−#endif
884− {
885− .procname = "ngroups_max",
886− .data = &ngroups_max,
887− .maxlen = sizeof (int),
888− .mode = 0444,
889− .proc_handler = proc_dointvec,
890− },
891− {
892− .procname = "cap_last_cap",
893− .data = (void *)&cap_last_cap,
894− .maxlen = sizeof(int),
895− .mode = 0444,
896− .proc_handler = proc_dointvec,
897− },
898−#if defined(CONFIG_LOCKUP_DETECTOR)
899− {
900− .procname = "watchdog",
901− .data = &watchdog_user_enabled,
902− .maxlen = sizeof(int),
903− .mode = 0644,
904− .proc_handler = proc_watchdog,
905− .extra1 = SYSCTL_ZERO,
906− .extra2 = SYSCTL_ONE,
907− },
908− {
909− .procname = "watchdog_thresh",
910− .data = &watchdog_thresh,
911− .maxlen = sizeof(int),
912− .mode = 0644,
913− .proc_handler = proc_watchdog_thresh,
914− .extra1 = SYSCTL_ZERO,
915− .extra2 = &sixty,
916− },
917− {
918− .procname = "nmi_watchdog",
919− .data = &nmi_watchdog_user_enabled,
920− .maxlen = sizeof(int),
921− .mode = NMI_WATCHDOG_SYSCTL_PERM,
922− .proc_handler = proc_nmi_watchdog,
923− .extra1 = SYSCTL_ZERO,
924− .extra2 = SYSCTL_ONE,
925− },
926− {
927− .procname = "watchdog_cpumask",
928− .data = &watchdog_cpumask_bits,
929− .maxlen = NR_CPUS,
930− .mode = 0644,
931− .proc_handler = proc_watchdog_cpumask,
932− },
933−#ifdef CONFIG_SOFTLOCKUP_DETECTOR
934− {
935− .procname = "soft_watchdog",
936− .data = &soft_watchdog_user_enabled,
937− .maxlen = sizeof(int),
938− .mode = 0644,
939− .proc_handler = proc_soft_watchdog,
940− .extra1 = SYSCTL_ZERO,
941− .extra2 = SYSCTL_ONE,
942− },
943− {
944− .procname = "softlockup_panic",
945− .data = &softlockup_panic,
946− .maxlen = sizeof(int),
947− .mode = 0644,
948− .proc_handler = proc_dointvec_minmax,
949− .extra1 = SYSCTL_ZERO,
950− .extra2 = SYSCTL_ONE,
951− },
952−#ifdef CONFIG_SMP
953− {
954− .procname = "softlockup_all_cpu_backtrace",
955− .data = &sysctl_softlockup_all_cpu_backtrace,
956− .maxlen = sizeof(int),
957− .mode = 0644,
958− .proc_handler = proc_dointvec_minmax,
959− .extra1 = SYSCTL_ZERO,
960− .extra2 = SYSCTL_ONE,
961− },
962−#endif /* CONFIG_SMP */
963−#endif
964−#ifdef CONFIG_HARDLOCKUP_DETECTOR
965− {
966− .procname = "hardlockup_panic",
967− .data = &hardlockup_panic,
968− .maxlen = sizeof(int),
969− .mode = 0644,
970− .proc_handler = proc_dointvec_minmax,
971− .extra1 = SYSCTL_ZERO,
972− .extra2 = SYSCTL_ONE,
973− },
974−#ifdef CONFIG_SMP
975− {
976− .procname = "hardlockup_all_cpu_backtrace",
977− .data = &sysctl_hardlockup_all_cpu_backtrace,
978− .maxlen = sizeof(int),
979− .mode = 0644,
980− .proc_handler = proc_dointvec_minmax,
981− .extra1 = SYSCTL_ZERO,
982− .extra2 = SYSCTL_ONE,
983− },
984−#endif /* CONFIG_SMP */
985−#endif
986−#endif
987−
988−#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
989− {
990− .procname = "unknown_nmi_panic",
991− .data = &unknown_nmi_panic,
992− .maxlen = sizeof (int),
993− .mode = 0644,
994− .proc_handler = proc_dointvec,
995− },
996−#endif
997−
998−#if (defined(CONFIG_X86_32) || defined(CONFIG_PARISC)) && \
999− defined(CONFIG_DEBUG_STACKOVERFLOW)
1000− {
1001− .procname = "panic_on_stackoverflow",
1002− .data = &sysctl_panic_on_stackoverflow,
1003− .maxlen = sizeof(int),
1004− .mode = 0644,
1005− .proc_handler = proc_dointvec,
1006− },
1007−#endif
1008−#if defined(CONFIG_X86)
1009− {
1010− .procname = "panic_on_unrecovered_nmi",
1011− .data = &panic_on_unrecovered_nmi,
1012− .maxlen = sizeof(int),
1013− .mode = 0644,
1014− .proc_handler = proc_dointvec,
1015− },
1016− {
1017− .procname = "panic_on_io_nmi",
1018− .data = &panic_on_io_nmi,
1019− .maxlen = sizeof(int),
1020− .mode = 0644,
1021− .proc_handler = proc_dointvec,
1022− },
1023− {
1024− .procname = "bootloader_type",
1025− .data = &bootloader_type,
1026− .maxlen = sizeof (int),
1027− .mode = 0444,
1028− .proc_handler = proc_dointvec,
1029− },
1030− {
1031− .procname = "bootloader_version",
1032− .data = &bootloader_version,
1033− .maxlen = sizeof (int),
1034− .mode = 0444,
1035− .proc_handler = proc_dointvec,
1036− },
1037− {
1038− .procname = "io_delay_type",
1039− .data = &io_delay_type,
1040− .maxlen = sizeof(int),
1041− .mode = 0644,
1042− .proc_handler = proc_dointvec,
1043− },
1044−#endif
1045−#if defined(CONFIG_MMU)
1046− {
1047− .procname = "randomize_va_space",
1048− .data = &randomize_va_space,
1049− .maxlen = sizeof(int),
1050− .mode = 0644,
1051− .proc_handler = proc_dointvec,
1052− },
1053−#endif
1054−#if defined(CONFIG_S390) && defined(CONFIG_SMP)
1055− {
1056− .procname = "spin_retry",
1057− .data = &spin_retry,
1058− .maxlen = sizeof (int),
1059− .mode = 0644,
1060− .proc_handler = proc_dointvec,
1061− },
1062−#endif
1063−#if defined(CONFIG_ACPI_SLEEP) && defined(CONFIG_X86)
1064− {
1065− .procname = "acpi_video_flags",
1066− .data = &acpi_realmode_flags,
1067− .maxlen = sizeof (unsigned long),
1068− .mode = 0644,
1069− .proc_handler = proc_doulongvec_minmax,
1070− },
1071−#endif
1072−#ifdef CONFIG_SYSCTL_ARCH_UNALIGN_NO_WARN
1073− {
1074− .procname = "ignore-unaligned-usertrap",
1075− .data = &no_unaligned_warning,
1076− .maxlen = sizeof (int),
1077− .mode = 0644,
1078− .proc_handler = proc_dointvec,
1079− },
1080−#endif
1081−#ifdef CONFIG_IA64
1082− {
1083− .procname = "unaligned-dump-stack",
1084− .data = &unaligned_dump_stack,
1085− .maxlen = sizeof (int),
1086− .mode = 0644,
1087− .proc_handler = proc_dointvec,
1088− },
1089−#endif
1090−#ifdef CONFIG_DETECT_HUNG_TASK
1091− {
1092− .procname = "hung_task_panic",
1093− .data = &sysctl_hung_task_panic,
1094− .maxlen = sizeof(int),
1095− .mode = 0644,
1096− .proc_handler = proc_dointvec_minmax,
1097− .extra1 = SYSCTL_ZERO,
1098− .extra2 = SYSCTL_ONE,
1099− },
1100− {
1101− .procname = "hung_task_check_count",
1102− .data = &sysctl_hung_task_check_count,
1103− .maxlen = sizeof(int),
1104− .mode = 0644,
1105− .proc_handler = proc_dointvec_minmax,
1106− .extra1 = SYSCTL_ZERO,
1107− },
1108− {
1109− .procname = "hung_task_timeout_secs",
1110− .data = &sysctl_hung_task_timeout_secs,
1111− .maxlen = sizeof(unsigned long),
1112− .mode = 0644,
1113− .proc_handler = proc_dohung_task_timeout_secs,
1114− .extra2 = &hung_task_timeout_max,
1115− },
1116− {
1117− .procname = "hung_task_check_interval_secs",
1118− .data = &sysctl_hung_task_check_interval_secs,
1119− .maxlen = sizeof(unsigned long),
1120− .mode = 0644,
1121− .proc_handler = proc_dohung_task_timeout_secs,
1122− .extra2 = &hung_task_timeout_max,
1123− },
1124− {
1125− .procname = "hung_task_warnings",
1126− .data = &sysctl_hung_task_warnings,
1127− .maxlen = sizeof(int),
1128− .mode = 0644,
1129− .proc_handler = proc_dointvec_minmax,
1130− .extra1 = &neg_one,
1131− },
1132−#endif
1133−#ifdef CONFIG_RT_MUTEXES
1134− {
1135− .procname = "max_lock_depth",
1136− .data = &max_lock_depth,
1137− .maxlen = sizeof(int),
1138− .mode = 0644,
1139− .proc_handler = proc_dointvec,
1140− },
1141−#endif
1142− {
1143− .procname = "poweroff_cmd",
1144− .data = &poweroff_cmd,
1145− .maxlen = POWEROFF_CMD_PATH_LEN,
1146− .mode = 0644,
1147− .proc_handler = proc_dostring,
1148− },
1149−#ifdef CONFIG_KEYS
1150− {
1151− .procname = "keys",
1152− .mode = 0555,
1153− .child = key_sysctls,
1154− },
1155−#endif
1156−#ifdef CONFIG_PERF_EVENTS
1157− /*
1158− * User-space scripts rely on the existence of this file
1159− * as a feature check for perf_events being enabled.
1160− *
1161− * So it's an ABI, do not remove!
1162− */
1163− {
1164− .procname = "perf_event_paranoid",
1165− .data = &sysctl_perf_event_paranoid,
1166− .maxlen = sizeof(sysctl_perf_event_paranoid),
1167− .mode = 0644,
1168− .proc_handler = proc_dointvec,
1169− },
1170− {
1171− .procname = "perf_event_mlock_kb",
1172− .data = &sysctl_perf_event_mlock,
1173− .maxlen = sizeof(sysctl_perf_event_mlock),
1174− .mode = 0644,
1175− .proc_handler = proc_dointvec,
1176− },
1177− {
1178− .procname = "perf_event_max_sample_rate",
1179− .data = &sysctl_perf_event_sample_rate,
1180− .maxlen = sizeof(sysctl_perf_event_sample_rate),
1181− .mode = 0644,
1182− .proc_handler = perf_proc_update_handler,
1183− .extra1 = SYSCTL_ONE,
1184− },
1185− {
1186− .procname = "perf_cpu_time_max_percent",
1187− .data = &sysctl_perf_cpu_time_max_percent,
1188− .maxlen = sizeof(sysctl_perf_cpu_time_max_percent),
1189− .mode = 0644,
1190− .proc_handler = perf_cpu_time_max_percent_handler,
1191− .extra1 = SYSCTL_ZERO,
1192− .extra2 = &one_hundred,
1193− },
1194− {
1195− .procname = "perf_event_max_stack",
1196− .data = &sysctl_perf_event_max_stack,
1197− .maxlen = sizeof(sysctl_perf_event_max_stack),
1198− .mode = 0644,
1199− .proc_handler = perf_event_max_stack_handler,
1200− .extra1 = SYSCTL_ZERO,
1201− .extra2 = &six_hundred_forty_kb,
1202− },
1203− {
1204− .procname = "perf_event_max_contexts_per_stack",
1205− .data = &sysctl_perf_event_max_contexts_per_stack,
1206− .maxlen = sizeof(sysctl_perf_event_max_contexts_per_stack),
1207− .mode = 0644,
1208− .proc_handler = perf_event_max_stack_handler,
1209− .extra1 = SYSCTL_ZERO,
1210− .extra2 = &one_thousand,
1211− },
1212−#endif
1213− {
1214− .procname = "panic_on_warn",
1215− .data = &panic_on_warn,
1216− .maxlen = sizeof(int),
1217− .mode = 0644,
1218− .proc_handler = proc_dointvec_minmax,
1219− .extra1 = SYSCTL_ZERO,
1220− .extra2 = SYSCTL_ONE,
1221− },
1222−#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
1223− {
1224− .procname = "timer_migration",
1225− .data = &sysctl_timer_migration,
1226− .maxlen = sizeof(unsigned int),
1227− .mode = 0644,
1228− .proc_handler = timer_migration_handler,
1229− .extra1 = SYSCTL_ZERO,
1230− .extra2 = SYSCTL_ONE,
1231− },
1232−#endif
1233−#ifdef CONFIG_BPF_SYSCALL
1234− {
1235− .procname = "unprivileged_bpf_disabled",
1236− .data = &sysctl_unprivileged_bpf_disabled,
1237− .maxlen = sizeof(sysctl_unprivileged_bpf_disabled),
1238− .mode = 0644,
1239− /* only handle a transition from default "0" to "1" */
1240− .proc_handler = proc_dointvec_minmax,
1241− .extra1 = SYSCTL_ONE,
1242− .extra2 = SYSCTL_ONE,
1243− },
1244− {
1245− .procname = "bpf_stats_enabled",
1246− .data = &bpf_stats_enabled_key.key,
1247− .maxlen = sizeof(bpf_stats_enabled_key),
1248− .mode = 0644,
1249− .proc_handler = proc_do_static_key,
1250− },
1251−#endif
1252−#if defined(CONFIG_TREE_RCU)
1253− {
1254− .procname = "panic_on_rcu_stall",
1255− .data = &sysctl_panic_on_rcu_stall,
1256− .maxlen = sizeof(sysctl_panic_on_rcu_stall),
1257− .mode = 0644,
1258− .proc_handler = proc_dointvec_minmax,
1259− .extra1 = SYSCTL_ZERO,
1260− .extra2 = SYSCTL_ONE,
1261− },
1262−#endif
1263−#ifdef CONFIG_STACKLEAK_RUNTIME_DISABLE
1264− {
1265− .procname = "stack_erasing",
1266− .data = NULL,
1267− .maxlen = sizeof(int),
1268− .mode = 0600,
1269− .proc_handler = stack_erasing_sysctl,
1270− .extra1 = SYSCTL_ZERO,
1271− .extra2 = SYSCTL_ONE,
1272− },
1273−#endif
1274− { }
1275−};
1276−
1277−static struct ctl_table vm_table[] = {
1278− {
1279− .procname = "overcommit_memory",
1280− .data = &sysctl_overcommit_memory,
1281− .maxlen = sizeof(sysctl_overcommit_memory),
1282− .mode = 0644,
1283− .proc_handler = proc_dointvec_minmax,
1284− .extra1 = SYSCTL_ZERO,
1285− .extra2 = &two,
1286− },
1287− {
1288− .procname = "panic_on_oom",
1289− .data = &sysctl_panic_on_oom,
1290− .maxlen = sizeof(sysctl_panic_on_oom),
1291− .mode = 0644,
1292− .proc_handler = proc_dointvec_minmax,
1293− .extra1 = SYSCTL_ZERO,
1294− .extra2 = &two,
1295− },
1296− {
1297− .procname = "oom_kill_allocating_task",
1298− .data = &sysctl_oom_kill_allocating_task,
1299− .maxlen = sizeof(sysctl_oom_kill_allocating_task),
1300− .mode = 0644,
1301− .proc_handler = proc_dointvec,
1302− },
1303− {
1304− .procname = "oom_dump_tasks",
1305− .data = &sysctl_oom_dump_tasks,
1306− .maxlen = sizeof(sysctl_oom_dump_tasks),
1307− .mode = 0644,
1308− .proc_handler = proc_dointvec,
1309− },
1310− {
1311− .procname = "overcommit_ratio",
1312− .data = &sysctl_overcommit_ratio,
1313− .maxlen = sizeof(sysctl_overcommit_ratio),
1314− .mode = 0644,
1315− .proc_handler = overcommit_ratio_handler,
1316− },
1317− {
1318− .procname = "overcommit_kbytes",
1319− .data = &sysctl_overcommit_kbytes,
1320− .maxlen = sizeof(sysctl_overcommit_kbytes),
1321− .mode = 0644,
1322− .proc_handler = overcommit_kbytes_handler,
1323− },
1324− {
1325− .procname = "page-cluster",
1326− .data = &page_cluster,
1327− .maxlen = sizeof(int),
1328− .mode = 0644,
1329− .proc_handler = proc_dointvec_minmax,
1330− .extra1 = SYSCTL_ZERO,
1331− },
1332− {
1333− .procname = "dirty_background_ratio",
1334− .data = &dirty_background_ratio,
1335− .maxlen = sizeof(dirty_background_ratio),
1336− .mode = 0644,
1337− .proc_handler = dirty_background_ratio_handler,
1338− .extra1 = SYSCTL_ZERO,
1339− .extra2 = &one_hundred,
1340− },
1341− {
1342− .procname = "dirty_background_bytes",
1343− .data = &dirty_background_bytes,
1344− .maxlen = sizeof(dirty_background_bytes),
1345− .mode = 0644,
1346− .proc_handler = dirty_background_bytes_handler,
1347− .extra1 = &one_ul,
1348− },
1349− {
1350− .procname = "dirty_ratio",
1351− .data = &vm_dirty_ratio,
1352− .maxlen = sizeof(vm_dirty_ratio),
1353− .mode = 0644,
1354− .proc_handler = dirty_ratio_handler,
1355− .extra1 = SYSCTL_ZERO,
1356− .extra2 = &one_hundred,
1357− },
1358− {
1359− .procname = "dirty_bytes",
1360− .data = &vm_dirty_bytes,
1361− .maxlen = sizeof(vm_dirty_bytes),
1362− .mode = 0644,
1363− .proc_handler = dirty_bytes_handler,
1364− .extra1 = &dirty_bytes_min,
1365− },
1366− {
1367− .procname = "dirty_writeback_centisecs",
1368− .data = &dirty_writeback_interval,
1369− .maxlen = sizeof(dirty_writeback_interval),
1370− .mode = 0644,
1371− .proc_handler = dirty_writeback_centisecs_handler,
1372− },
1373− {
1374− .procname = "dirty_expire_centisecs",
1375− .data = &dirty_expire_interval,
1376− .maxlen = sizeof(dirty_expire_interval),
1377− .mode = 0644,
1378− .proc_handler = proc_dointvec_minmax,
1379− .extra1 = SYSCTL_ZERO,
1380− },
1381− {
1382− .procname = "dirtytime_expire_seconds",
1383− .data = &dirtytime_expire_interval,
1384− .maxlen = sizeof(dirtytime_expire_interval),
1385− .mode = 0644,
1386− .proc_handler = dirtytime_interval_handler,
1387− .extra1 = SYSCTL_ZERO,
1388− },
1389− {
1390− .procname = "swappiness",
1391− .data = &vm_swappiness,
1392− .maxlen = sizeof(vm_swappiness),
1393− .mode = 0644,
1394− .proc_handler = proc_dointvec_minmax,
1395− .extra1 = SYSCTL_ZERO,
1396− .extra2 = &one_hundred,
1397− },
1398−#ifdef CONFIG_HUGETLB_PAGE
1399− {
1400− .procname = "nr_hugepages",
1401− .data = NULL,
1402− .maxlen = sizeof(unsigned long),
1403− .mode = 0644,
1404− .proc_handler = hugetlb_sysctl_handler,
1405− },
1406−#ifdef CONFIG_NUMA
1407− {
1408− .procname = "nr_hugepages_mempolicy",
1409− .data = NULL,
1410− .maxlen = sizeof(unsigned long),
1411− .mode = 0644,
1412− .proc_handler = &hugetlb_mempolicy_sysctl_handler,
1413− },
1414− {
1415− .procname = "numa_stat",
1416− .data = &sysctl_vm_numa_stat,
1417− .maxlen = sizeof(int),
1418− .mode = 0644,
1419− .proc_handler = sysctl_vm_numa_stat_handler,
1420− .extra1 = SYSCTL_ZERO,
1421− .extra2 = SYSCTL_ONE,
1422− },
1423−#endif
1424− {
1425− .procname = "hugetlb_shm_group",
1426− .data = &sysctl_hugetlb_shm_group,
1427− .maxlen = sizeof(gid_t),
1428− .mode = 0644,
1429− .proc_handler = proc_dointvec,
1430− },
1431− {
1432− .procname = "nr_overcommit_hugepages",
1433− .data = NULL,
1434− .maxlen = sizeof(unsigned long),
1435− .mode = 0644,
1436− .proc_handler = hugetlb_overcommit_handler,
1437− },
1438−#endif
1439− {
1440− .procname = "lowmem_reserve_ratio",
1441− .data = &sysctl_lowmem_reserve_ratio,
1442− .maxlen = sizeof(sysctl_lowmem_reserve_ratio),
1443− .mode = 0644,
1444− .proc_handler = lowmem_reserve_ratio_sysctl_handler,
1445− },
1446− {
1447− .procname = "drop_caches",
1448− .data = &sysctl_drop_caches,
1449− .maxlen = sizeof(int),
1450− .mode = 0200,
1451− .proc_handler = drop_caches_sysctl_handler,
1452− .extra1 = SYSCTL_ONE,
1453− .extra2 = &four,
1454− },
1455−#ifdef CONFIG_COMPACTION
1456− {
1457− .procname = "compact_memory",
1458− .data = &sysctl_compact_memory,
1459− .maxlen = sizeof(int),
1460− .mode = 0200,
1461− .proc_handler = sysctl_compaction_handler,
1462− },
1463− {
1464− .procname = "extfrag_threshold",
1465− .data = &sysctl_extfrag_threshold,
1466− .maxlen = sizeof(int),
1467− .mode = 0644,
1468− .proc_handler = proc_dointvec_minmax,
1469− .extra1 = &min_extfrag_threshold,
1470− .extra2 = &max_extfrag_threshold,
1471− },
1472− {
1473− .procname = "compact_unevictable_allowed",
1474− .data = &sysctl_compact_unevictable_allowed,
1475− .maxlen = sizeof(int),
1476− .mode = 0644,
1477− .proc_handler = proc_dointvec_minmax_warn_RT_change,
1478− .extra1 = SYSCTL_ZERO,
1479− .extra2 = SYSCTL_ONE,
1480− },
1481−
1482−#endif /* CONFIG_COMPACTION */
1483− {
1484− .procname = "min_free_kbytes",
1485− .data = &min_free_kbytes,
1486− .maxlen = sizeof(min_free_kbytes),
1487− .mode = 0644,
1488− .proc_handler = min_free_kbytes_sysctl_handler,
1489− .extra1 = SYSCTL_ZERO,
1490− },
1491− {
1492− .procname = "watermark_boost_factor",
1493− .data = &watermark_boost_factor,
1494− .maxlen = sizeof(watermark_boost_factor),
1495− .mode = 0644,
1496− .proc_handler = watermark_boost_factor_sysctl_handler,
1497− .extra1 = SYSCTL_ZERO,
1498− },
1499− {
1500− .procname = "watermark_scale_factor",
1501− .data = &watermark_scale_factor,
1502− .maxlen = sizeof(watermark_scale_factor),
1503− .mode = 0644,
1504− .proc_handler = watermark_scale_factor_sysctl_handler,
1505− .extra1 = SYSCTL_ONE,
1506− .extra2 = &one_thousand,
1507− },
1508− {
1509− .procname = "percpu_pagelist_fraction",
1510− .data = &percpu_pagelist_fraction,
1511− .maxlen = sizeof(percpu_pagelist_fraction),
1512− .mode = 0644,
1513− .proc_handler = percpu_pagelist_fraction_sysctl_handler,
1514− .extra1 = SYSCTL_ZERO,
1515− },
1516−#ifdef CONFIG_MMU
1517− {
1518− .procname = "max_map_count",
1519− .data = &sysctl_max_map_count,
1520− .maxlen = sizeof(sysctl_max_map_count),
1521− .mode = 0644,
1522− .proc_handler = proc_dointvec_minmax,
1523− .extra1 = SYSCTL_ZERO,
1524− },
1525−#else
1526− {
1527− .procname = "nr_trim_pages",
1528− .data = &sysctl_nr_trim_pages,
1529− .maxlen = sizeof(sysctl_nr_trim_pages),
1530− .mode = 0644,
1531− .proc_handler = proc_dointvec_minmax,
1532− .extra1 = SYSCTL_ZERO,
1533− },
1534−#endif
1535− {
1536− .procname = "laptop_mode",
1537− .data = &laptop_mode,
1538− .maxlen = sizeof(laptop_mode),
1539− .mode = 0644,
1540− .proc_handler = proc_dointvec_jiffies,
1541− },
1542− {
1543− .procname = "block_dump",
1544− .data = &block_dump,
1545− .maxlen = sizeof(block_dump),
1546− .mode = 0644,
1547− .proc_handler = proc_dointvec,
1548− .extra1 = SYSCTL_ZERO,
1549− },
1550− {
1551− .procname = "vfs_cache_pressure",
1552− .data = &sysctl_vfs_cache_pressure,
1553− .maxlen = sizeof(sysctl_vfs_cache_pressure),
1554− .mode = 0644,
1555− .proc_handler = proc_dointvec,
1556− .extra1 = SYSCTL_ZERO,
1557− },
1558−#if defined(HAVE_ARCH_PICK_MMAP_LAYOUT) || \
1559− defined(CONFIG_ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT)
1560− {
1561− .procname = "legacy_va_layout",
1562− .data = &sysctl_legacy_va_layout,
1563− .maxlen = sizeof(sysctl_legacy_va_layout),
1564− .mode = 0644,
1565− .proc_handler = proc_dointvec,
1566− .extra1 = SYSCTL_ZERO,
1567− },
1568−#endif
1569−#ifdef CONFIG_NUMA
1570− {
1571− .procname = "zone_reclaim_mode",
1572− .data = &node_reclaim_mode,
1573− .maxlen = sizeof(node_reclaim_mode),
1574− .mode = 0644,
1575− .proc_handler = proc_dointvec,
1576− .extra1 = SYSCTL_ZERO,
1577− },
1578− {
1579− .procname = "min_unmapped_ratio",
1580− .data = &sysctl_min_unmapped_ratio,
1581− .maxlen = sizeof(sysctl_min_unmapped_ratio),
1582− .mode = 0644,
1583− .proc_handler = sysctl_min_unmapped_ratio_sysctl_handler,
1584− .extra1 = SYSCTL_ZERO,
1585− .extra2 = &one_hundred,
1586− },
1587− {
1588− .procname = "min_slab_ratio",
1589− .data = &sysctl_min_slab_ratio,
1590− .maxlen = sizeof(sysctl_min_slab_ratio),
1591− .mode = 0644,
1592− .proc_handler = sysctl_min_slab_ratio_sysctl_handler,
1593− .extra1 = SYSCTL_ZERO,
1594− .extra2 = &one_hundred,
1595− },
1596−#endif
1597−#ifdef CONFIG_SMP
1598− {
1599− .procname = "stat_interval",
1600− .data = &sysctl_stat_interval,
1601− .maxlen = sizeof(sysctl_stat_interval),
1602− .mode = 0644,
1603− .proc_handler = proc_dointvec_jiffies,
1604− },
1605− {
1606− .procname = "stat_refresh",
1607− .data = NULL,
1608− .maxlen = 0,
1609− .mode = 0600,
1610− .proc_handler = vmstat_refresh,
1611− },
1612−#endif
1613−#ifdef CONFIG_MMU
1614− {
1615− .procname = "mmap_min_addr",
1616− .data = &dac_mmap_min_addr,
1617− .maxlen = sizeof(unsigned long),
1618− .mode = 0644,
1619− .proc_handler = mmap_min_addr_handler,
1620− },
1621−#endif
1622−#ifdef CONFIG_NUMA
1623− {
1624− .procname = "numa_zonelist_order",
1625− .data = &numa_zonelist_order,
1626− .maxlen = NUMA_ZONELIST_ORDER_LEN,
1627− .mode = 0644,
1628− .proc_handler = numa_zonelist_order_handler,
1629− },
1630−#endif
1631−#if (defined(CONFIG_X86_32) && !defined(CONFIG_UML))|| \
1632− (defined(CONFIG_SUPERH) && defined(CONFIG_VSYSCALL))
1633− {
1634− .procname = "vdso_enabled",
1635−#ifdef CONFIG_X86_32
1636− .data = &vdso32_enabled,
1637− .maxlen = sizeof(vdso32_enabled),
1638−#else
1639− .data = &vdso_enabled,
1640− .maxlen = sizeof(vdso_enabled),
1641−#endif
1642− .mode = 0644,
1643− .proc_handler = proc_dointvec,
1644− .extra1 = SYSCTL_ZERO,
1645− },
1646−#endif
1647−#ifdef CONFIG_HIGHMEM
1648− {
1649− .procname = "highmem_is_dirtyable",
1650− .data = &vm_highmem_is_dirtyable,
1651− .maxlen = sizeof(vm_highmem_is_dirtyable),
1652− .mode = 0644,
1653− .proc_handler = proc_dointvec_minmax,
1654− .extra1 = SYSCTL_ZERO,
1655− .extra2 = SYSCTL_ONE,
1656− },
1657−#endif
1658−#ifdef CONFIG_MEMORY_FAILURE
1659− {
1660− .procname = "memory_failure_early_kill",
1661− .data = &sysctl_memory_failure_early_kill,
1662− .maxlen = sizeof(sysctl_memory_failure_early_kill),
1663− .mode = 0644,
1664− .proc_handler = proc_dointvec_minmax,
1665− .extra1 = SYSCTL_ZERO,
1666− .extra2 = SYSCTL_ONE,
1667− },
1668− {
1669− .procname = "memory_failure_recovery",
1670− .data = &sysctl_memory_failure_recovery,
1671− .maxlen = sizeof(sysctl_memory_failure_recovery),
1672− .mode = 0644,
1673− .proc_handler = proc_dointvec_minmax,
1674− .extra1 = SYSCTL_ZERO,
1675− .extra2 = SYSCTL_ONE,
1676− },
1677−#endif
1678− {
1679− .procname = "user_reserve_kbytes",
1680− .data = &sysctl_user_reserve_kbytes,
1681− .maxlen = sizeof(sysctl_user_reserve_kbytes),
1682− .mode = 0644,
1683− .proc_handler = proc_doulongvec_minmax,
1684− },
1685− {
1686− .procname = "admin_reserve_kbytes",
1687− .data = &sysctl_admin_reserve_kbytes,
1688− .maxlen = sizeof(sysctl_admin_reserve_kbytes),
1689− .mode = 0644,
1690− .proc_handler = proc_doulongvec_minmax,
1691− },
1692−#ifdef CONFIG_HAVE_ARCH_MMAP_RND_BITS
1693− {
1694− .procname = "mmap_rnd_bits",
1695− .data = &mmap_rnd_bits,
1696− .maxlen = sizeof(mmap_rnd_bits),
1697− .mode = 0600,
1698− .proc_handler = proc_dointvec_minmax,
1699− .extra1 = (void *)&mmap_rnd_bits_min,
1700− .extra2 = (void *)&mmap_rnd_bits_max,
1701− },
1702−#endif
1703−#ifdef CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS
1704− {
1705− .procname = "mmap_rnd_compat_bits",
1706− .data = &mmap_rnd_compat_bits,
1707− .maxlen = sizeof(mmap_rnd_compat_bits),
1708− .mode = 0600,
1709− .proc_handler = proc_dointvec_minmax,
1710− .extra1 = (void *)&mmap_rnd_compat_bits_min,
1711− .extra2 = (void *)&mmap_rnd_compat_bits_max,
1712− },
1713−#endif
1714−#ifdef CONFIG_USERFAULTFD
1715− {
1716− .procname = "unprivileged_userfaultfd",
1717− .data = &sysctl_unprivileged_userfaultfd,
1718− .maxlen = sizeof(sysctl_unprivileged_userfaultfd),
1719− .mode = 0644,
1720− .proc_handler = proc_dointvec_minmax,
1721− .extra1 = SYSCTL_ZERO,
1722− .extra2 = SYSCTL_ONE,
1723− },
1724−#endif
1725− { }
1726−};
1727−
1728−static struct ctl_table fs_table[] = {
1729− {
1730− .procname = "inode-nr",
1731− .data = &inodes_stat,
1732− .maxlen = 2*sizeof(long),
1733− .mode = 0444,
1734− .proc_handler = proc_nr_inodes,
1735− },
1736− {
1737− .procname = "inode-state",
1738− .data = &inodes_stat,
1739− .maxlen = 7*sizeof(long),
1740− .mode = 0444,
1741− .proc_handler = proc_nr_inodes,
1742− },
1743− {
1744− .procname = "file-nr",
1745− .data = &files_stat,
1746− .maxlen = sizeof(files_stat),
1747− .mode = 0444,
1748− .proc_handler = proc_nr_files,
1749− },
1750− {
1751− .procname = "file-max",
1752− .data = &files_stat.max_files,
1753− .maxlen = sizeof(files_stat.max_files),
1754− .mode = 0644,
1755− .proc_handler = proc_doulongvec_minmax,
1756− .extra1 = &zero_ul,
1757− .extra2 = &long_max,
1758− },
1759− {
1760− .procname = "nr_open",
1761− .data = &sysctl_nr_open,
1762− .maxlen = sizeof(unsigned int),
1763− .mode = 0644,
1764− .proc_handler = proc_dointvec_minmax,
1765− .extra1 = &sysctl_nr_open_min,
1766− .extra2 = &sysctl_nr_open_max,
1767− },
1768− {
1769− .procname = "dentry-state",
1770− .data = &dentry_stat,
1771− .maxlen = 6*sizeof(long),
1772− .mode = 0444,
1773− .proc_handler = proc_nr_dentry,
1774− },
1775− {
1776− .procname = "overflowuid",
1777− .data = &fs_overflowuid,
1778− .maxlen = sizeof(int),
1779− .mode = 0644,
1780− .proc_handler = proc_dointvec_minmax,
1781− .extra1 = &minolduid,
1782− .extra2 = &maxolduid,
1783− },
1784− {
1785− .procname = "overflowgid",
1786− .data = &fs_overflowgid,
1787− .maxlen = sizeof(int),
1788− .mode = 0644,
1789− .proc_handler = proc_dointvec_minmax,
1790− .extra1 = &minolduid,
1791− .extra2 = &maxolduid,
1792− },
1793−#ifdef CONFIG_FILE_LOCKING
1794− {
1795− .procname = "leases-enable",
1796− .data = &leases_enable,
1797− .maxlen = sizeof(int),
1798− .mode = 0644,
1799− .proc_handler = proc_dointvec,
1800− },
1801−#endif
1802−#ifdef CONFIG_DNOTIFY
1803− {
1804− .procname = "dir-notify-enable",
1805− .data = &dir_notify_enable,
1806− .maxlen = sizeof(int),
1807− .mode = 0644,
1808− .proc_handler = proc_dointvec,
1809− },
1810−#endif
1811−#ifdef CONFIG_MMU
1812−#ifdef CONFIG_FILE_LOCKING
1813− {
1814− .procname = "lease-break-time",
1815− .data = &lease_break_time,
1816− .maxlen = sizeof(int),
1817− .mode = 0644,
1818− .proc_handler = proc_dointvec,
1819− },
1820−#endif
1821−#ifdef CONFIG_AIO
1822− {
1823− .procname = "aio-nr",
1824− .data = &aio_nr,
1825− .maxlen = sizeof(aio_nr),
1826− .mode = 0444,
1827− .proc_handler = proc_doulongvec_minmax,
1828− },
1829− {
1830− .procname = "aio-max-nr",
1831− .data = &aio_max_nr,
1832− .maxlen = sizeof(aio_max_nr),
1833− .mode = 0644,
1834− .proc_handler = proc_doulongvec_minmax,
1835− },
1836−#endif /* CONFIG_AIO */
1837−#ifdef CONFIG_INOTIFY_USER
1838− {
1839− .procname = "inotify",
1840− .mode = 0555,
1841− .child = inotify_table,
1842− },
1843−#endif
1844−#ifdef CONFIG_EPOLL
1845− {
1846− .procname = "epoll",
1847− .mode = 0555,
1848− .child = epoll_table,
1849− },
1850−#endif
1851−#endif
1852− {
1853− .procname = "protected_symlinks",
1854− .data = &sysctl_protected_symlinks,
1855− .maxlen = sizeof(int),
1856− .mode = 0600,
1857− .proc_handler = proc_dointvec_minmax,
1858− .extra1 = SYSCTL_ZERO,
1859− .extra2 = SYSCTL_ONE,
1860− },
1861− {
1862− .procname = "protected_hardlinks",
1863− .data = &sysctl_protected_hardlinks,
1864− .maxlen = sizeof(int),
1865− .mode = 0600,
1866− .proc_handler = proc_dointvec_minmax,
1867− .extra1 = SYSCTL_ZERO,
1868− .extra2 = SYSCTL_ONE,
1869− },
1870− {
1871− .procname = "protected_fifos",
1872− .data = &sysctl_protected_fifos,
1873− .maxlen = sizeof(int),
1874− .mode = 0600,
1875− .proc_handler = proc_dointvec_minmax,
1876− .extra1 = SYSCTL_ZERO,
1877− .extra2 = &two,
1878− },
1879− {
1880− .procname = "protected_regular",
1881− .data = &sysctl_protected_regular,
1882− .maxlen = sizeof(int),
1883− .mode = 0600,
1884− .proc_handler = proc_dointvec_minmax,
1885− .extra1 = SYSCTL_ZERO,
1886− .extra2 = &two,
1887− },
1888− {
1889− .procname = "suid_dumpable",
1890− .data = &suid_dumpable,
1891− .maxlen = sizeof(int),
1892− .mode = 0644,
1893− .proc_handler = proc_dointvec_minmax_coredump,
1894− .extra1 = SYSCTL_ZERO,
1895− .extra2 = &two,
1896− },
1897−#if defined(CONFIG_BINFMT_MISC) || defined(CONFIG_BINFMT_MISC_MODULE)
1898− {
1899− .procname = "binfmt_misc",
1900− .mode = 0555,
1901− .child = sysctl_mount_point,
1902− },
1903−#endif
1904− {
1905− .procname = "pipe-max-size",
1906− .data = &pipe_max_size,
1907− .maxlen = sizeof(pipe_max_size),
1908− .mode = 0644,
1909− .proc_handler = proc_dopipe_max_size,
1910− },
1911− {
1912− .procname = "pipe-user-pages-hard",
1913− .data = &pipe_user_pages_hard,
1914− .maxlen = sizeof(pipe_user_pages_hard),
1915− .mode = 0644,
1916− .proc_handler = proc_doulongvec_minmax,
1917− },
1918− {
1919− .procname = "pipe-user-pages-soft",
1920− .data = &pipe_user_pages_soft,
1921− .maxlen = sizeof(pipe_user_pages_soft),
1922− .mode = 0644,
1923− .proc_handler = proc_doulongvec_minmax,
1924− },
1925− {
1926− .procname = "mount-max",
1927− .data = &sysctl_mount_max,
1928− .maxlen = sizeof(unsigned int),
1929− .mode = 0644,
1930− .proc_handler = proc_dointvec_minmax,
1931− .extra1 = SYSCTL_ONE,
1932− },
1933− { }
1934−};
1935−
1936−static struct ctl_table debug_table[] = {
1937−#ifdef CONFIG_SYSCTL_EXCEPTION_TRACE
1938− {
1939− .procname = "exception-trace",
1940− .data = &show_unhandled_signals,
1941− .maxlen = sizeof(int),
1942− .mode = 0644,
1943− .proc_handler = proc_dointvec
1944− },
1945−#endif
1946−#if defined(CONFIG_OPTPROBES)
1947− {
1948− .procname = "kprobes-optimization",
1949− .data = &sysctl_kprobes_optimization,
1950− .maxlen = sizeof(int),
1951− .mode = 0644,
1952− .proc_handler = proc_kprobes_optimization_handler,
1953− .extra1 = SYSCTL_ZERO,
1954− .extra2 = SYSCTL_ONE,
1955− },
1956−#endif
1957− { }
1958−};
1959−
1960−static struct ctl_table dev_table[] = {
1961− { }
1962−};
1963−
1964−int __init sysctl_init(void)
1965−{
1966− struct ctl_table_header *hdr;
1967−
1968− hdr = register_sysctl_table(sysctl_base_table);
1969− kmemleak_not_leak(hdr);
1970− return 0;
1971−}
1972−
1973−#endif /* CONFIG_SYSCTL */
1974−
1975−/*
1976− * /proc/sys support
1977− */
1978−
237+
238+/*
239+ * /proc/sys support
240+ */
241+
1979242 #ifdef CONFIG_PROC_SYSCTL
1980243
1981244 static int _proc_do_string(char *data, int maxlen, int write,
1982− char __user *buffer,
1983− size_t *lenp, loff_t *ppos)
245+ char *buffer, size_t *lenp, loff_t *ppos)
1984246 {
1985247 size_t len;
1986− char __user *p;
1987− char c;
248+ char c, *p;
1988249
1989250 if (!data || !maxlen || !*lenp) {
1990251 *lenp = 0;
@@ -2009,8 +270,7 @@ static int _proc_do_string(char *data, int maxlen, int write, · 18 unmodified lines
2009270 *ppos += *lenp;
2010271 p = buffer;
2011272 while ((p - buffer) < *lenp && len < maxlen - 1) {
2012− if (get_user(c, p++))
2013− return -EFAULT;
273+ c = *(p++);
2014274 if (c == 0 || c == '\n')
2015275 break;
2016276 data[len++] = c;
@@ -2032,11 +292,9 @@ static int _proc_do_string(char *data, int maxlen, int write, · 15 unmodified lines
2032292 if (len > *lenp)
2033293 len = *lenp;
2034294 if (len)
2035− if (copy_to_user(buffer, data, len))
2036− return -EFAULT;
295+ memcpy(buffer, data, len);
2037296 if (len < *lenp) {
2038− if (put_user('\n', buffer + len))
2039− return -EFAULT;
297+ buffer[len] = '\n';
2040298 len++;
2041299 }
2042300 *lenp = len;
@@ -2097,13 +355,13 @@ static bool proc_first_pos_non_zero_ignore(loff_t *ppos, · 54 unmodified lines
2097355 * Returns 0 on success.
2098356 */
2099357 int proc_dostring(struct ctl_table *table, int write,
2100− void __user *buffer, size_t *lenp, loff_t *ppos)
358+ void *buffer, size_t *lenp, loff_t *ppos)
2101359 {
2102360 if (write)
2103361 proc_first_pos_non_zero_ignore(ppos, table);
2104362
2105− return _proc_do_string((char *)(table->data), table->maxlen, write,
2106− (char __user *)buffer, lenp, ppos);
363+ return _proc_do_string(table->data, table->maxlen, write, buffer, lenp,
364+ ppos);
2107365 }
2108366
2109367 static size_t proc_skip_spaces(char **buf)
@@ -2234,11 +492,10 @@ static int proc_get_long(char **buf, size_t *size, · 124 unmodified lines
2234492 * @val: the integer to be converted
2235493 * @neg: sign of the number, %TRUE for negative
2236494 *
2237− * In case of success %0 is returned and @buf and @size are updated with
2238− * the amount of bytes written.
495+ * In case of success @buf and @size are updated with the amount of bytes
496+ * written.
2239497 */
2240−static int proc_put_long(void __user **buf, size_t *size, unsigned long val,
2241− bool neg)
498+static void proc_put_long(void **buf, size_t *size, unsigned long val, bool neg)
2242499 {
2243500 int len;
2244501 char tmp[TMPBUFLEN], *p = tmp;
@@ -2247,24 +504,22 @@ static int proc_put_long(void __user **buf, size_t *size, unsigned long val, · 2 unmodified lines
2247504 len = strlen(tmp);
2248505 if (len > *size)
2249506 len = *size;
2250− if (copy_to_user(*buf, tmp, len))
2251− return -EFAULT;
507+ memcpy(*buf, tmp, len);
2252508 *size -= len;
2253509 *buf += len;
2254− return 0;
2255510 }
2256511 #undef TMPBUFLEN
2257512
2258−static int proc_put_char(void __user **buf, size_t *size, char c)
513+static void proc_put_char(void **buf, size_t *size, char c)
2259514 {
2260515 if (*size) {
2261− char __user **buffer = (char __user **)buf;
2262− if (put_user(c, *buffer))
2263− return -EFAULT;
2264− (*size)--, (*buffer)++;
516+ char **buffer = (char **)buf;
517+ **buffer = c;
518+
519+ (*size)--;
520+ (*buffer)++;
2265521 *buf = *buffer;
2266522 }
2267− return 0;
2268523 }
2269524
2270525 static int do_proc_dointvec_conv(bool *negp, unsigned long *lvalp,
@@ -2312,7 +567,7 @@ static int do_proc_douintvec_conv(unsigned long *lvalp, · 41 unmodified lines
2312567 static const char proc_wspace_sep[] = { ' ', '\t', '\n' };
2313568
2314569 static int __do_proc_dointvec(void *tbl_data, struct ctl_table *table,
2315− int write, void __user *buffer,
570+ int write, void *buffer,
2316571 size_t *lenp, loff_t *ppos,
2317572 int (*conv)(bool *negp, unsigned long *lvalp, int *valp,
2318573 int write, void *data),
@@ -2320,7 +575,7 @@ static int __do_proc_dointvec(void *tbl_data, struct ctl_table *table, · 1 unmodified line
2320575 {
2321576 int *i, vleft, first = 1, err = 0;
2322577 size_t left;
2323− char *kbuf = NULL, *p;
578+ char *p;
2324579
2325580 if (!tbl_data || !table->maxlen || !*lenp || (*ppos && !write)) {
2326581 *lenp = 0;
@@ -2340,9 +595,7 @@ static int __do_proc_dointvec(void *tbl_data, struct ctl_table *table, · 13 unmodified lines
2340595
2341596 if (left > PAGE_SIZE - 1)
2342597 left = PAGE_SIZE - 1;
2343− p = kbuf = memdup_user_nul(buffer, left);
2344− if (IS_ERR(kbuf))
2345− return PTR_ERR(kbuf);
598+ p = buffer;
2346599 }
2347600
2348601 for (; left && vleft--; i++, first=0) {
@@ -2369,24 +622,17 @@ static int __do_proc_dointvec(void *tbl_data, struct ctl_table *table, · 20 unmodified lines
2369622 break;
2370623 }
2371624 if (!first)
2372− err = proc_put_char(&buffer, &left, '\t');
2373− if (err)
2374− break;
2375− err = proc_put_long(&buffer, &left, lval, neg);
2376− if (err)
2377− break;
625+ proc_put_char(&buffer, &left, '\t');
626+ proc_put_long(&buffer, &left, lval, neg);
2378627 }
2379628 }
2380629
2381630 if (!write && !first && left && !err)
2382− err = proc_put_char(&buffer, &left, '\n');
631+ proc_put_char(&buffer, &left, '\n');
2383632 if (write && !err && left)
2384633 left -= proc_skip_spaces(&p);
2385− if (write) {
2386− kfree(kbuf);
2387− if (first)
2388− return err ? : -EINVAL;
2389− }
634+ if (write && first)
635+ return err ? : -EINVAL;
2390636 *lenp -= left;
2391637 out:
2392638 *ppos += *lenp;
@@ -2394,7 +640,7 @@ out: · 1 unmodified line
2394640 }
2395641
2396642 static int do_proc_dointvec(struct ctl_table *table, int write,
2397− void __user *buffer, size_t *lenp, loff_t *ppos,
643+ void *buffer, size_t *lenp, loff_t *ppos,
2398644 int (*conv)(bool *negp, unsigned long *lvalp, int *valp,
2399645 int write, void *data),
2400646 void *data)
@@ -2405,7 +651,7 @@ static int do_proc_dointvec(struct ctl_table *table, int write, · 4 unmodified lines
2405651
2406652 static int do_proc_douintvec_w(unsigned int *tbl_data,
2407653 struct ctl_table *table,
2408− void __user *buffer,
654+ void *buffer,
2409655 size_t *lenp, loff_t *ppos,
2410656 int (*conv)(unsigned long *lvalp,
2411657 unsigned int *valp,
@@ -2416,7 +662,7 @@ static int do_proc_douintvec_w(unsigned int *tbl_data, · 4 unmodified lines
2416662 int err = 0;
2417663 size_t left;
2418664 bool neg;
2419− char *kbuf = NULL, *p;
665+ char *p = buffer;
2420666
2421667 left = *lenp;
2422668
@@ -2426,10 +672,6 @@ static int do_proc_douintvec_w(unsigned int *tbl_data, · 3 unmodified lines
2426672 if (left > PAGE_SIZE - 1)
2427673 left = PAGE_SIZE - 1;
2428674
2429− p = kbuf = memdup_user_nul(buffer, left);
2430− if (IS_ERR(kbuf))
2431− return -EINVAL;
2432−
2433675 left -= proc_skip_spaces(&p);
2434676 if (!left) {
2435677 err = -EINVAL;
@@ -2453,7 +695,6 @@ static int do_proc_douintvec_w(unsigned int *tbl_data, · 17 unmodified lines
2453695 left -= proc_skip_spaces(&p);
2454696
2455697 out_free:
2456− kfree(kbuf);
2457698 if (err)
2458699 return -EINVAL;
2459700
@@ -2465,7 +706,7 @@ bail_early: · 5 unmodified lines
2465706 return err;
2466707 }
2467708
2468−static int do_proc_douintvec_r(unsigned int *tbl_data, void __user *buffer,
709+static int do_proc_douintvec_r(unsigned int *tbl_data, void *buffer,
2469710 size_t *lenp, loff_t *ppos,
2470711 int (*conv)(unsigned long *lvalp,
2471712 unsigned int *valp,
@@ -2483,11 +724,11 @@ static int do_proc_douintvec_r(unsigned int *tbl_data, void __user *buffer, · 11 unmodified lines
2483724 goto out;
2484725 }
2485726
2486− err = proc_put_long(&buffer, &left, lval, false);
2487− if (err || !left)
727+ proc_put_long(&buffer, &left, lval, false);
728+ if (!left)
2488729 goto out;
2489730
2490− err = proc_put_char(&buffer, &left, '\n');
731+ proc_put_char(&buffer, &left, '\n');
2491732
2492733 out:
2493734 *lenp -= left;
@@ -2497,7 +738,7 @@ out: · 3 unmodified lines
2497738 }
2498739
2499740 static int __do_proc_douintvec(void *tbl_data, struct ctl_table *table,
2500− int write, void __user *buffer,
741+ int write, void *buffer,
2501742 size_t *lenp, loff_t *ppos,
2502743 int (*conv)(unsigned long *lvalp,
2503744 unsigned int *valp,
@@ -2533,7 +774,7 @@ static int __do_proc_douintvec(void *tbl_data, struct ctl_table *table, · 29 unmodified lines
2533774 }
2534775
2535776 static int do_proc_douintvec(struct ctl_table *table, int write,
2536− void __user *buffer, size_t *lenp, loff_t *ppos,
777+ void *buffer, size_t *lenp, loff_t *ppos,
2537778 int (*conv)(unsigned long *lvalp,
2538779 unsigned int *valp,
2539780 int write, void *data),
@@ -2556,16 +797,15 @@ static int do_proc_douintvec(struct ctl_table *table, int write, · 16 unmodified lines
2556797 *
2557798 * Returns 0 on success.
2558799 */
2559−int proc_dointvec(struct ctl_table *table, int write,
2560− void __user *buffer, size_t *lenp, loff_t *ppos)
800+int proc_dointvec(struct ctl_table *table, int write, void *buffer,
801+ size_t *lenp, loff_t *ppos)
2561802 {
2562803 return do_proc_dointvec(table, write, buffer, lenp, ppos, NULL, NULL);
2563804 }
2564805
2565806 #ifdef CONFIG_COMPACTION
2566807 static int proc_dointvec_minmax_warn_RT_change(struct ctl_table *table,
2567− int write, void __user *buffer,
2568− size_t *lenp, loff_t *ppos)
808+ int write, void *buffer, size_t *lenp, loff_t *ppos)
2569809 {
2570810 int ret, old;
2571811
@@ -2597,8 +837,8 @@ static int proc_dointvec_minmax_warn_RT_change(struct ctl_table *table, · 25 unmodified lines
2597837 *
2598838 * Returns 0 on success.
2599839 */
2600−int proc_douintvec(struct ctl_table *table, int write,
2601− void __user *buffer, size_t *lenp, loff_t *ppos)
840+int proc_douintvec(struct ctl_table *table, int write, void *buffer,
841+ size_t *lenp, loff_t *ppos)
2602842 {
2603843 return do_proc_douintvec(table, write, buffer, lenp, ppos,
2604844 do_proc_douintvec_conv, NULL);
@@ -2609,7 +849,7 @@ int proc_douintvec(struct ctl_table *table, int write, · 4 unmodified lines
2609849 * This means we can safely use a temporary.
2610850 */
2611851 static int proc_taint(struct ctl_table *table, int write,
2612− void __user *buffer, size_t *lenp, loff_t *ppos)
852+ void *buffer, size_t *lenp, loff_t *ppos)
2613853 {
2614854 struct ctl_table t;
2615855 unsigned long tmptaint = get_taint();
@@ -2641,7 +881,7 @@ static int proc_taint(struct ctl_table *table, int write, · 25 unmodified lines
2641881
2642882 #ifdef CONFIG_PRINTK
2643883 static int proc_dointvec_minmax_sysadmin(struct ctl_table *table, int write,
2644− void __user *buffer, size_t *lenp, loff_t *ppos)
884+ void *buffer, size_t *lenp, loff_t *ppos)
2645885 {
2646886 if (write && !capable(CAP_SYS_ADMIN))
2647887 return -EPERM;
@@ -2707,7 +947,7 @@ static int do_proc_dointvec_minmax_conv(bool *negp, unsigned long *lvalp, · 59 unmodified lines
2707947 * Returns 0 on success or -EINVAL on write when the range check fails.
2708948 */
2709949 int proc_dointvec_minmax(struct ctl_table *table, int write,
2710− void __user *buffer, size_t *lenp, loff_t *ppos)
950+ void *buffer, size_t *lenp, loff_t *ppos)
2711951 {
2712952 struct do_proc_dointvec_minmax_conv_param param = {
2713953 .min = (int *) table->extra1,
@@ -2776,7 +1016,7 @@ static int do_proc_douintvec_minmax_conv(unsigned long *lvalp, · 62 unmodified lines
27761016 * Returns 0 on success or -ERANGE on write when the range check fails.
27771017 */
27781018 int proc_douintvec_minmax(struct ctl_table *table, int write,
2779− void __user *buffer, size_t *lenp, loff_t *ppos)
1019+ void *buffer, size_t *lenp, loff_t *ppos)
27801020 {
27811021 struct do_proc_douintvec_minmax_conv_param param = {
27821022 .min = (unsigned int *) table->extra1,
@@ -2807,7 +1047,7 @@ static int do_proc_dopipe_max_size_conv(unsigned long *lvalp, · 24 unmodified lines
28071047 }
28081048
28091049 static int proc_dopipe_max_size(struct ctl_table *table, int write,
2810− void __user *buffer, size_t *lenp, loff_t *ppos)
1050+ void *buffer, size_t *lenp, loff_t *ppos)
28111051 {
28121052 return do_proc_douintvec(table, write, buffer, lenp, ppos,
28131053 do_proc_dopipe_max_size_conv, NULL);
@@ -2828,7 +1068,7 @@ static void validate_coredump_safety(void) · 14 unmodified lines
28281068 }
28291069
28301070 static int proc_dointvec_minmax_coredump(struct ctl_table *table, int write,
2831− void __user *buffer, size_t *lenp, loff_t *ppos)
1071+ void *buffer, size_t *lenp, loff_t *ppos)
28321072 {
28331073 int error = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
28341074 if (!error)
@@ -2838,7 +1078,7 @@ static int proc_dointvec_minmax_coredump(struct ctl_table *table, int write, · 3 unmodified lines
28381078
28391079 #ifdef CONFIG_COREDUMP
28401080 static int proc_dostring_coredump(struct ctl_table *table, int write,
2841− void __user *buffer, size_t *lenp, loff_t *ppos)
1081+ void *buffer, size_t *lenp, loff_t *ppos)
28421082 {
28431083 int error = proc_dostring(table, write, buffer, lenp, ppos);
28441084 if (!error)
@@ -2849,7 +1089,7 @@ static int proc_dostring_coredump(struct ctl_table *table, int write, · 4 unmodified lines
28491089
28501090 #ifdef CONFIG_MAGIC_SYSRQ
28511091 static int sysrq_sysctl_handler(struct ctl_table *table, int write,
2852− void __user *buffer, size_t *lenp, loff_t *ppos)
1092+ void *buffer, size_t *lenp, loff_t *ppos)
28531093 {
28541094 int tmp, ret;
28551095
@@ -2867,16 +1107,14 @@ static int sysrq_sysctl_handler(struct ctl_table *table, int write, · 11 unmodified lines
28671107 }
28681108 #endif
28691109
2870−static int __do_proc_doulongvec_minmax(void *data, struct ctl_table *table, int write,
2871− void __user *buffer,
2872− size_t *lenp, loff_t *ppos,
2873− unsigned long convmul,
2874− unsigned long convdiv)
1110+static int __do_proc_doulongvec_minmax(void *data, struct ctl_table *table,
1111+ int write, void *buffer, size_t *lenp, loff_t *ppos,
1112+ unsigned long convmul, unsigned long convdiv)
28751113 {
28761114 unsigned long *i, *min, *max;
28771115 int vleft, first = 1, err = 0;
28781116 size_t left;
2879− char *kbuf = NULL, *p;
1117+ char *p;
28801118
28811119 if (!data || !table->maxlen || !*lenp || (*ppos && !write)) {
28821120 *lenp = 0;
@@ -2895,9 +1133,7 @@ static int __do_proc_doulongvec_minmax(void *data, struct ctl_table *table, int · 12 unmodified lines
28951133
28961134 if (left > PAGE_SIZE - 1)
28971135 left = PAGE_SIZE - 1;
2898− p = kbuf = memdup_user_nul(buffer, left);
2899− if (IS_ERR(kbuf))
2900− return PTR_ERR(kbuf);
1136+ p = buffer;
29011137 }
29021138
29031139 for (; left && vleft--; i++, first = 0) {
@@ -2925,26 +1161,18 @@ static int __do_proc_doulongvec_minmax(void *data, struct ctl_table *table, int · 21 unmodified lines
29251161 *i = val;
29261162 } else {
29271163 val = convdiv * (*i) / convmul;
2928− if (!first) {
2929− err = proc_put_char(&buffer, &left, '\t');
2930− if (err)
2931− break;
2932− }
2933− err = proc_put_long(&buffer, &left, val, false);
2934− if (err)
2935− break;
1164+ if (!first)
1165+ proc_put_char(&buffer, &left, '\t');
1166+ proc_put_long(&buffer, &left, val, false);
29361167 }
29371168 }
29381169
29391170 if (!write && !first && left && !err)
2940− err = proc_put_char(&buffer, &left, '\n');
1171+ proc_put_char(&buffer, &left, '\n');
29411172 if (write && !err)
29421173 left -= proc_skip_spaces(&p);
2943− if (write) {
2944− kfree(kbuf);
2945− if (first)
2946− return err ? : -EINVAL;
2947− }
1174+ if (write && first)
1175+ return err ? : -EINVAL;
29481176 *lenp -= left;
29491177 out:
29501178 *ppos += *lenp;
@@ -2952,10 +1180,8 @@ out: · 1 unmodified line
29521180 }
29531181
29541182 static int do_proc_doulongvec_minmax(struct ctl_table *table, int write,
2955− void __user *buffer,
2956− size_t *lenp, loff_t *ppos,
2957− unsigned long convmul,
2958− unsigned long convdiv)
1183+ void *buffer, size_t *lenp, loff_t *ppos, unsigned long convmul,
1184+ unsigned long convdiv)
29591185 {
29601186 return __do_proc_doulongvec_minmax(table->data, table, write,
29611187 buffer, lenp, ppos, convmul, convdiv);
@@ -2978,7 +1204,7 @@ static int do_proc_doulongvec_minmax(struct ctl_table *table, int write, · 16 unmodified lines
29781204 * Returns 0 on success.
29791205 */
29801206 int proc_doulongvec_minmax(struct ctl_table *table, int write,
2981− void __user *buffer, size_t *lenp, loff_t *ppos)
1207+ void *buffer, size_t *lenp, loff_t *ppos)
29821208 {
29831209 return do_proc_doulongvec_minmax(table, write, buffer, lenp, ppos, 1l, 1l);
29841210 }
@@ -3001,8 +1227,7 @@ int proc_doulongvec_minmax(struct ctl_table *table, int write, · 16 unmodified lines
30011227 * Returns 0 on success.
30021228 */
30031229 int proc_doulongvec_ms_jiffies_minmax(struct ctl_table *table, int write,
3004− void __user *buffer,
3005− size_t *lenp, loff_t *ppos)
1230+ void *buffer, size_t *lenp, loff_t *ppos)
30061231 {
30071232 return do_proc_doulongvec_minmax(table, write, buffer,
30081233 lenp, ppos, HZ, 1000l);
@@ -3096,7 +1321,7 @@ static int do_proc_dointvec_ms_jiffies_conv(bool *negp, unsigned long *lvalp, · 87 unmodified lines
30961321 * Returns 0 on success.
30971322 */
30981323 int proc_dointvec_jiffies(struct ctl_table *table, int write,
3099− void __user *buffer, size_t *lenp, loff_t *ppos)
1324+ void *buffer, size_t *lenp, loff_t *ppos)
31001325 {
31011326 return do_proc_dointvec(table,write,buffer,lenp,ppos,
31021327 do_proc_dointvec_jiffies_conv,NULL);
@@ -3118,7 +1343,7 @@ int proc_dointvec_jiffies(struct ctl_table *table, int write, · 15 unmodified lines
31181343 * Returns 0 on success.
31191344 */
31201345 int proc_dointvec_userhz_jiffies(struct ctl_table *table, int write,
3121− void __user *buffer, size_t *lenp, loff_t *ppos)
1346+ void *buffer, size_t *lenp, loff_t *ppos)
31221347 {
31231348 return do_proc_dointvec(table,write,buffer,lenp,ppos,
31241349 do_proc_dointvec_userhz_jiffies_conv,NULL);
@@ -3140,15 +1365,15 @@ int proc_dointvec_userhz_jiffies(struct ctl_table *table, int write, · 15 unmodified lines
31401365 *
31411366 * Returns 0 on success.
31421367 */
3143−int proc_dointvec_ms_jiffies(struct ctl_table *table, int write,
3144− void __user *buffer, size_t *lenp, loff_t *ppos)
1368+int proc_dointvec_ms_jiffies(struct ctl_table *table, int write, void *buffer,
1369+ size_t *lenp, loff_t *ppos)
31451370 {
31461371 return do_proc_dointvec(table, write, buffer, lenp, ppos,
31471372 do_proc_dointvec_ms_jiffies_conv, NULL);
31481373 }
31491374
3150−static int proc_do_cad_pid(struct ctl_table *table, int write,
3151− void __user *buffer, size_t *lenp, loff_t *ppos)
1375+static int proc_do_cad_pid(struct ctl_table *table, int write, void *buffer,
1376+ size_t *lenp, loff_t *ppos)
31521377 {
31531378 struct pid *new_pid;
31541379 pid_t tmp;
@@ -3187,7 +1412,7 @@ static int proc_do_cad_pid(struct ctl_table *table, int write, · 32 unmodified lines
31871412 * Returns 0 on success.
31881413 */
31891414 int proc_do_large_bitmap(struct ctl_table *table, int write,
3190− void __user *buffer, size_t *lenp, loff_t *ppos)
1415+ void *buffer, size_t *lenp, loff_t *ppos)
31911416 {
31921417 int err = 0;
31931418 bool first = 1;
@@ -3203,7 +1428,7 @@ int proc_do_large_bitmap(struct ctl_table *table, int write, · 9 unmodified lines
32031428 }
32041429
32051430 if (write) {
3206− char *kbuf, *p;
1431+ char *p = buffer;
32071432 size_t skipped = 0;
32081433
32091434 if (left > PAGE_SIZE - 1) {
@@ -3212,15 +1437,9 @@ int proc_do_large_bitmap(struct ctl_table *table, int write, · 2 unmodified lines
32121437 skipped = *lenp - left;
32131438 }
32141439
3215− p = kbuf = memdup_user_nul(buffer, left);
3216− if (IS_ERR(kbuf))
3217− return PTR_ERR(kbuf);
3218−
32191440 tmp_bitmap = bitmap_zalloc(bitmap_len, GFP_KERNEL);
3220− if (!tmp_bitmap) {
3221− kfree(kbuf);
1441+ if (!tmp_bitmap)
32221442 return -ENOMEM;
3223− }
32241443 proc_skip_char(&p, &left, '\n');
32251444 while (!err && left) {
32261445 unsigned long val_a, val_b;
@@ -3284,7 +1503,6 @@ int proc_do_large_bitmap(struct ctl_table *table, int write, · 57 unmodified lines
32841503 first = 0;
32851504 proc_skip_char(&p, &left, '\n');
32861505 }
3287− kfree(kbuf);
32881506 left += skipped;
32891507 } else {
32901508 unsigned long bit_a, bit_b = 0;
@@ -3296,27 +1514,17 @@ int proc_do_large_bitmap(struct ctl_table *table, int write, · 5 unmodified lines
32961514 bit_b = find_next_zero_bit(bitmap, bitmap_len,
32971515 bit_a + 1) - 1;
32981516
3299− if (!first) {
3300− err = proc_put_char(&buffer, &left, ',');
3301− if (err)
3302− break;
3303− }
3304− err = proc_put_long(&buffer, &left, bit_a, false);
3305− if (err)
3306− break;
1517+ if (!first)
1518+ proc_put_char(&buffer, &left, ',');
1519+ proc_put_long(&buffer, &left, bit_a, false);
33071520 if (bit_a != bit_b) {
3308− err = proc_put_char(&buffer, &left, '-');
3309− if (err)
3310− break;
3311− err = proc_put_long(&buffer, &left, bit_b, false);
3312− if (err)
3313− break;
1521+ proc_put_char(&buffer, &left, '-');
1522+ proc_put_long(&buffer, &left, bit_b, false);
33141523 }
33151524
33161525 first = 0; bit_b++;
33171526 }
3318− if (!err)
3319− err = proc_put_char(&buffer, &left, '\n');
1527+ proc_put_char(&buffer, &left, '\n');
33201528 }
33211529
33221530 if (!err) {
@@ -3330,113 +1538,1806 @@ int proc_do_large_bitmap(struct ctl_table *table, int write, · 7 unmodified lines
33301538 *ppos += *lenp;
33311539 }
33321540
3333− bitmap_free(tmp_bitmap);
3334− return err;
1541+ bitmap_free(tmp_bitmap);
1542+ return err;
1543+}
1544+
1545+#else /* CONFIG_PROC_SYSCTL */
1546+
1547+int proc_dostring(struct ctl_table *table, int write,
1548+ void *buffer, size_t *lenp, loff_t *ppos)
1549+{
1550+ return -ENOSYS;
1551+}
1552+
1553+int proc_dointvec(struct ctl_table *table, int write,
1554+ void *buffer, size_t *lenp, loff_t *ppos)
1555+{
1556+ return -ENOSYS;
1557+}
1558+
1559+int proc_douintvec(struct ctl_table *table, int write,
1560+ void *buffer, size_t *lenp, loff_t *ppos)
1561+{
1562+ return -ENOSYS;
1563+}
1564+
1565+int proc_dointvec_minmax(struct ctl_table *table, int write,
1566+ void *buffer, size_t *lenp, loff_t *ppos)
1567+{
1568+ return -ENOSYS;
1569+}
1570+
1571+int proc_douintvec_minmax(struct ctl_table *table, int write,
1572+ void *buffer, size_t *lenp, loff_t *ppos)
1573+{
1574+ return -ENOSYS;
33351575 }
33361576
3337−#else /* CONFIG_PROC_SYSCTL */
1577+int proc_dointvec_jiffies(struct ctl_table *table, int write,
1578+ void *buffer, size_t *lenp, loff_t *ppos)
1579+{
1580+ return -ENOSYS;
1581+}
33381582
3339−int proc_dostring(struct ctl_table *table, int write,
3340− void __user *buffer, size_t *lenp, loff_t *ppos)
1583+int proc_dointvec_userhz_jiffies(struct ctl_table *table, int write,
1584+ void *buffer, size_t *lenp, loff_t *ppos)
33411585 {
33421586 return -ENOSYS;
33431587 }
33441588
3345−int proc_dointvec(struct ctl_table *table, int write,
3346− void __user *buffer, size_t *lenp, loff_t *ppos)
1589+int proc_dointvec_ms_jiffies(struct ctl_table *table, int write,
1590+ void *buffer, size_t *lenp, loff_t *ppos)
33471591 {
33481592 return -ENOSYS;
33491593 }
33501594
3351−int proc_douintvec(struct ctl_table *table, int write,
3352− void __user *buffer, size_t *lenp, loff_t *ppos)
1595+int proc_doulongvec_minmax(struct ctl_table *table, int write,
1596+ void *buffer, size_t *lenp, loff_t *ppos)
33531597 {
33541598 return -ENOSYS;
33551599 }
33561600
3357−int proc_dointvec_minmax(struct ctl_table *table, int write,
3358− void __user *buffer, size_t *lenp, loff_t *ppos)
1601+int proc_doulongvec_ms_jiffies_minmax(struct ctl_table *table, int write,
1602+ void *buffer, size_t *lenp, loff_t *ppos)
1603+{
1604+ return -ENOSYS;
1605+}
1606+
1607+int proc_do_large_bitmap(struct ctl_table *table, int write,
1608+ void *buffer, size_t *lenp, loff_t *ppos)
33591609 {
33601610 return -ENOSYS;
33611611 }
33621612
3363−int proc_douintvec_minmax(struct ctl_table *table, int write,
3364− void __user *buffer, size_t *lenp, loff_t *ppos)
3365−{
3366− return -ENOSYS;
3367−}
1613+#endif /* CONFIG_PROC_SYSCTL */
1614+
1615+#if defined(CONFIG_SYSCTL)
1616+int proc_do_static_key(struct ctl_table *table, int write,
1617+ void *buffer, size_t *lenp, loff_t *ppos)
1618+{
1619+ struct static_key *key = (struct static_key *)table->data;
1620+ static DEFINE_MUTEX(static_key_mutex);
1621+ int val, ret;
1622+ struct ctl_table tmp = {
1623+ .data = &val,
1624+ .maxlen = sizeof(val),
1625+ .mode = table->mode,
1626+ .extra1 = SYSCTL_ZERO,
1627+ .extra2 = SYSCTL_ONE,
1628+ };
1629+
1630+ if (write && !capable(CAP_SYS_ADMIN))
1631+ return -EPERM;
1632+
1633+ mutex_lock(&static_key_mutex);
1634+ val = static_key_enabled(key);
1635+ ret = proc_dointvec_minmax(&tmp, write, buffer, lenp, ppos);
1636+ if (write && !ret) {
1637+ if (val)
1638+ static_key_enable(key);
1639+ else
1640+ static_key_disable(key);
1641+ }
1642+ mutex_unlock(&static_key_mutex);
1643+ return ret;
1644+}
1645+
1646+static struct ctl_table kern_table[] = {
1647+ {
1648+ .procname = "sched_child_runs_first",
1649+ .data = &sysctl_sched_child_runs_first,
1650+ .maxlen = sizeof(unsigned int),
1651+ .mode = 0644,
1652+ .proc_handler = proc_dointvec,
1653+ },
1654+#ifdef CONFIG_SCHED_DEBUG
1655+ {
1656+ .procname = "sched_min_granularity_ns",
1657+ .data = &sysctl_sched_min_granularity,
1658+ .maxlen = sizeof(unsigned int),
1659+ .mode = 0644,
1660+ .proc_handler = sched_proc_update_handler,
1661+ .extra1 = &min_sched_granularity_ns,
1662+ .extra2 = &max_sched_granularity_ns,
1663+ },
1664+ {
1665+ .procname = "sched_latency_ns",
1666+ .data = &sysctl_sched_latency,
1667+ .maxlen = sizeof(unsigned int),
1668+ .mode = 0644,
1669+ .proc_handler = sched_proc_update_handler,
1670+ .extra1 = &min_sched_granularity_ns,
1671+ .extra2 = &max_sched_granularity_ns,
1672+ },
1673+ {
1674+ .procname = "sched_wakeup_granularity_ns",
1675+ .data = &sysctl_sched_wakeup_granularity,
1676+ .maxlen = sizeof(unsigned int),
1677+ .mode = 0644,
1678+ .proc_handler = sched_proc_update_handler,
1679+ .extra1 = &min_wakeup_granularity_ns,
1680+ .extra2 = &max_wakeup_granularity_ns,
1681+ },
1682+#ifdef CONFIG_SMP
1683+ {
1684+ .procname = "sched_tunable_scaling",
1685+ .data = &sysctl_sched_tunable_scaling,
1686+ .maxlen = sizeof(enum sched_tunable_scaling),
1687+ .mode = 0644,
1688+ .proc_handler = sched_proc_update_handler,
1689+ .extra1 = &min_sched_tunable_scaling,
1690+ .extra2 = &max_sched_tunable_scaling,
1691+ },
1692+ {
1693+ .procname = "sched_migration_cost_ns",
1694+ .data = &sysctl_sched_migration_cost,
1695+ .maxlen = sizeof(unsigned int),
1696+ .mode = 0644,
1697+ .proc_handler = proc_dointvec,
1698+ },
1699+ {
1700+ .procname = "sched_nr_migrate",
1701+ .data = &sysctl_sched_nr_migrate,
1702+ .maxlen = sizeof(unsigned int),
1703+ .mode = 0644,
1704+ .proc_handler = proc_dointvec,
1705+ },
1706+#ifdef CONFIG_SCHEDSTATS
1707+ {
1708+ .procname = "sched_schedstats",
1709+ .data = NULL,
1710+ .maxlen = sizeof(unsigned int),
1711+ .mode = 0644,
1712+ .proc_handler = sysctl_schedstats,
1713+ .extra1 = SYSCTL_ZERO,
1714+ .extra2 = SYSCTL_ONE,
1715+ },
1716+#endif /* CONFIG_SCHEDSTATS */
1717+#endif /* CONFIG_SMP */
1718+#ifdef CONFIG_NUMA_BALANCING
1719+ {
1720+ .procname = "numa_balancing_scan_delay_ms",
1721+ .data = &sysctl_numa_balancing_scan_delay,
1722+ .maxlen = sizeof(unsigned int),
1723+ .mode = 0644,
1724+ .proc_handler = proc_dointvec,
1725+ },
1726+ {
1727+ .procname = "numa_balancing_scan_period_min_ms",
1728+ .data = &sysctl_numa_balancing_scan_period_min,
1729+ .maxlen = sizeof(unsigned int),
1730+ .mode = 0644,
1731+ .proc_handler = proc_dointvec,
1732+ },
1733+ {
1734+ .procname = "numa_balancing_scan_period_max_ms",
1735+ .data = &sysctl_numa_balancing_scan_period_max,
1736+ .maxlen = sizeof(unsigned int),
1737+ .mode = 0644,
1738+ .proc_handler = proc_dointvec,
1739+ },
1740+ {
1741+ .procname = "numa_balancing_scan_size_mb",
1742+ .data = &sysctl_numa_balancing_scan_size,
1743+ .maxlen = sizeof(unsigned int),
1744+ .mode = 0644,
1745+ .proc_handler = proc_dointvec_minmax,
1746+ .extra1 = SYSCTL_ONE,
1747+ },
1748+ {
1749+ .procname = "numa_balancing",
1750+ .data = NULL, /* filled in by handler */
1751+ .maxlen = sizeof(unsigned int),
1752+ .mode = 0644,
1753+ .proc_handler = sysctl_numa_balancing,
1754+ .extra1 = SYSCTL_ZERO,
1755+ .extra2 = SYSCTL_ONE,
1756+ },
1757+#endif /* CONFIG_NUMA_BALANCING */
1758+#endif /* CONFIG_SCHED_DEBUG */
1759+ {
1760+ .procname = "sched_rt_period_us",
1761+ .data = &sysctl_sched_rt_period,
1762+ .maxlen = sizeof(unsigned int),
1763+ .mode = 0644,
1764+ .proc_handler = sched_rt_handler,
1765+ },
1766+ {
1767+ .procname = "sched_rt_runtime_us",
1768+ .data = &sysctl_sched_rt_runtime,
1769+ .maxlen = sizeof(int),
1770+ .mode = 0644,
1771+ .proc_handler = sched_rt_handler,
1772+ },
1773+ {
1774+ .procname = "sched_rr_timeslice_ms",
1775+ .data = &sysctl_sched_rr_timeslice,
1776+ .maxlen = sizeof(int),
1777+ .mode = 0644,
1778+ .proc_handler = sched_rr_handler,
1779+ },
1780+#ifdef CONFIG_UCLAMP_TASK
1781+ {
1782+ .procname = "sched_util_clamp_min",
1783+ .data = &sysctl_sched_uclamp_util_min,
1784+ .maxlen = sizeof(unsigned int),
1785+ .mode = 0644,
1786+ .proc_handler = sysctl_sched_uclamp_handler,
1787+ },
1788+ {
1789+ .procname = "sched_util_clamp_max",
1790+ .data = &sysctl_sched_uclamp_util_max,
1791+ .maxlen = sizeof(unsigned int),
1792+ .mode = 0644,
1793+ .proc_handler = sysctl_sched_uclamp_handler,
1794+ },
1795+#endif
1796+#ifdef CONFIG_SCHED_AUTOGROUP
1797+ {
1798+ .procname = "sched_autogroup_enabled",
1799+ .data = &sysctl_sched_autogroup_enabled,
1800+ .maxlen = sizeof(unsigned int),
1801+ .mode = 0644,
1802+ .proc_handler = proc_dointvec_minmax,
1803+ .extra1 = SYSCTL_ZERO,
1804+ .extra2 = SYSCTL_ONE,
1805+ },
1806+#endif
1807+#ifdef CONFIG_CFS_BANDWIDTH
1808+ {
1809+ .procname = "sched_cfs_bandwidth_slice_us",
1810+ .data = &sysctl_sched_cfs_bandwidth_slice,
1811+ .maxlen = sizeof(unsigned int),
1812+ .mode = 0644,
1813+ .proc_handler = proc_dointvec_minmax,
1814+ .extra1 = SYSCTL_ONE,
1815+ },
1816+#endif
1817+#if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL)
1818+ {
1819+ .procname = "sched_energy_aware",
1820+ .data = &sysctl_sched_energy_aware,
1821+ .maxlen = sizeof(unsigned int),
1822+ .mode = 0644,
1823+ .proc_handler = sched_energy_aware_handler,
1824+ .extra1 = SYSCTL_ZERO,
1825+ .extra2 = SYSCTL_ONE,
1826+ },
1827+#endif
1828+#ifdef CONFIG_PROVE_LOCKING
1829+ {
1830+ .procname = "prove_locking",
1831+ .data = &prove_locking,
1832+ .maxlen = sizeof(int),
1833+ .mode = 0644,
1834+ .proc_handler = proc_dointvec,
1835+ },
1836+#endif
1837+#ifdef CONFIG_LOCK_STAT
1838+ {
1839+ .procname = "lock_stat",
1840+ .data = &lock_stat,
1841+ .maxlen = sizeof(int),
1842+ .mode = 0644,
1843+ .proc_handler = proc_dointvec,
1844+ },
1845+#endif
1846+ {
1847+ .procname = "panic",
1848+ .data = &panic_timeout,
1849+ .maxlen = sizeof(int),
1850+ .mode = 0644,
1851+ .proc_handler = proc_dointvec,
1852+ },
1853+#ifdef CONFIG_COREDUMP
1854+ {
1855+ .procname = "core_uses_pid",
1856+ .data = &core_uses_pid,
1857+ .maxlen = sizeof(int),
1858+ .mode = 0644,
1859+ .proc_handler = proc_dointvec,
1860+ },
1861+ {
1862+ .procname = "core_pattern",
1863+ .data = core_pattern,
1864+ .maxlen = CORENAME_MAX_SIZE,
1865+ .mode = 0644,
1866+ .proc_handler = proc_dostring_coredump,
1867+ },
1868+ {
1869+ .procname = "core_pipe_limit",
1870+ .data = &core_pipe_limit,
1871+ .maxlen = sizeof(unsigned int),
1872+ .mode = 0644,
1873+ .proc_handler = proc_dointvec,
1874+ },
1875+#endif
1876+#ifdef CONFIG_PROC_SYSCTL
1877+ {
1878+ .procname = "tainted",
1879+ .maxlen = sizeof(long),
1880+ .mode = 0644,
1881+ .proc_handler = proc_taint,
1882+ },
1883+ {
1884+ .procname = "sysctl_writes_strict",
1885+ .data = &sysctl_writes_strict,
1886+ .maxlen = sizeof(int),
1887+ .mode = 0644,
1888+ .proc_handler = proc_dointvec_minmax,
1889+ .extra1 = &neg_one,
1890+ .extra2 = SYSCTL_ONE,
1891+ },
1892+#endif
1893+#ifdef CONFIG_LATENCYTOP
1894+ {
1895+ .procname = "latencytop",
1896+ .data = &latencytop_enabled,
1897+ .maxlen = sizeof(int),
1898+ .mode = 0644,
1899+ .proc_handler = sysctl_latencytop,
1900+ },
1901+#endif
1902+#ifdef CONFIG_BLK_DEV_INITRD
1903+ {
1904+ .procname = "real-root-dev",
1905+ .data = &real_root_dev,
1906+ .maxlen = sizeof(int),
1907+ .mode = 0644,
1908+ .proc_handler = proc_dointvec,
1909+ },
1910+#endif
1911+ {
1912+ .procname = "print-fatal-signals",
1913+ .data = &print_fatal_signals,
1914+ .maxlen = sizeof(int),
1915+ .mode = 0644,
1916+ .proc_handler = proc_dointvec,
1917+ },
1918+#ifdef CONFIG_SPARC
1919+ {
1920+ .procname = "reboot-cmd",
1921+ .data = reboot_command,
1922+ .maxlen = 256,
1923+ .mode = 0644,
1924+ .proc_handler = proc_dostring,
1925+ },
1926+ {
1927+ .procname = "stop-a",
1928+ .data = &stop_a_enabled,
1929+ .maxlen = sizeof (int),
1930+ .mode = 0644,
1931+ .proc_handler = proc_dointvec,
1932+ },
1933+ {
1934+ .procname = "scons-poweroff",
1935+ .data = &scons_pwroff,
1936+ .maxlen = sizeof (int),
1937+ .mode = 0644,
1938+ .proc_handler = proc_dointvec,
1939+ },
1940+#endif
1941+#ifdef CONFIG_SPARC64
1942+ {
1943+ .procname = "tsb-ratio",
1944+ .data = &sysctl_tsb_ratio,
1945+ .maxlen = sizeof (int),
1946+ .mode = 0644,
1947+ .proc_handler = proc_dointvec,
1948+ },
1949+#endif
1950+#ifdef CONFIG_PARISC
1951+ {
1952+ .procname = "soft-power",
1953+ .data = &pwrsw_enabled,
1954+ .maxlen = sizeof (int),
1955+ .mode = 0644,
1956+ .proc_handler = proc_dointvec,
1957+ },
1958+#endif
1959+#ifdef CONFIG_SYSCTL_ARCH_UNALIGN_ALLOW
1960+ {
1961+ .procname = "unaligned-trap",
1962+ .data = &unaligned_enabled,
1963+ .maxlen = sizeof (int),
1964+ .mode = 0644,
1965+ .proc_handler = proc_dointvec,
1966+ },
1967+#endif
1968+ {
1969+ .procname = "ctrl-alt-del",
1970+ .data = &C_A_D,
1971+ .maxlen = sizeof(int),
1972+ .mode = 0644,
1973+ .proc_handler = proc_dointvec,
1974+ },
1975+#ifdef CONFIG_FUNCTION_TRACER
1976+ {
1977+ .procname = "ftrace_enabled",
1978+ .data = &ftrace_enabled,
1979+ .maxlen = sizeof(int),
1980+ .mode = 0644,
1981+ .proc_handler = ftrace_enable_sysctl,
1982+ },
1983+#endif
1984+#ifdef CONFIG_STACK_TRACER
1985+ {
1986+ .procname = "stack_tracer_enabled",
1987+ .data = &stack_tracer_enabled,
1988+ .maxlen = sizeof(int),
1989+ .mode = 0644,
1990+ .proc_handler = stack_trace_sysctl,
1991+ },
1992+#endif
1993+#ifdef CONFIG_TRACING
1994+ {
1995+ .procname = "ftrace_dump_on_oops",
1996+ .data = &ftrace_dump_on_oops,
1997+ .maxlen = sizeof(int),
1998+ .mode = 0644,
1999+ .proc_handler = proc_dointvec,
2000+ },
2001+ {
2002+ .procname = "traceoff_on_warning",
2003+ .data = &__disable_trace_on_warning,
2004+ .maxlen = sizeof(__disable_trace_on_warning),
2005+ .mode = 0644,
2006+ .proc_handler = proc_dointvec,
2007+ },
2008+ {
2009+ .procname = "tracepoint_printk",
2010+ .data = &tracepoint_printk,
2011+ .maxlen = sizeof(tracepoint_printk),
2012+ .mode = 0644,
2013+ .proc_handler = tracepoint_printk_sysctl,
2014+ },
2015+#endif
2016+#ifdef CONFIG_KEXEC_CORE
2017+ {
2018+ .procname = "kexec_load_disabled",
2019+ .data = &kexec_load_disabled,
2020+ .maxlen = sizeof(int),
2021+ .mode = 0644,
2022+ /* only handle a transition from default "0" to "1" */
2023+ .proc_handler = proc_dointvec_minmax,
2024+ .extra1 = SYSCTL_ONE,
2025+ .extra2 = SYSCTL_ONE,
2026+ },
2027+#endif
2028+#ifdef CONFIG_MODULES
2029+ {
2030+ .procname = "modprobe",
2031+ .data = &modprobe_path,
2032+ .maxlen = KMOD_PATH_LEN,
2033+ .mode = 0644,
2034+ .proc_handler = proc_dostring,
2035+ },
2036+ {
2037+ .procname = "modules_disabled",
2038+ .data = &modules_disabled,
2039+ .maxlen = sizeof(int),
2040+ .mode = 0644,
2041+ /* only handle a transition from default "0" to "1" */
2042+ .proc_handler = proc_dointvec_minmax,
2043+ .extra1 = SYSCTL_ONE,
2044+ .extra2 = SYSCTL_ONE,
2045+ },
2046+#endif
2047+#ifdef CONFIG_UEVENT_HELPER
2048+ {
2049+ .procname = "hotplug",
2050+ .data = &uevent_helper,
2051+ .maxlen = UEVENT_HELPER_PATH_LEN,
2052+ .mode = 0644,
2053+ .proc_handler = proc_dostring,
2054+ },
2055+#endif
2056+#ifdef CONFIG_CHR_DEV_SG
2057+ {
2058+ .procname = "sg-big-buff",
2059+ .data = &sg_big_buff,
2060+ .maxlen = sizeof (int),
2061+ .mode = 0444,
2062+ .proc_handler = proc_dointvec,
2063+ },
2064+#endif
2065+#ifdef CONFIG_BSD_PROCESS_ACCT
2066+ {
2067+ .procname = "acct",
2068+ .data = &acct_parm,
2069+ .maxlen = 3*sizeof(int),
2070+ .mode = 0644,
2071+ .proc_handler = proc_dointvec,
2072+ },
2073+#endif
2074+#ifdef CONFIG_MAGIC_SYSRQ
2075+ {
2076+ .procname = "sysrq",
2077+ .data = NULL,
2078+ .maxlen = sizeof (int),
2079+ .mode = 0644,
2080+ .proc_handler = sysrq_sysctl_handler,
2081+ },
2082+#endif
2083+#ifdef CONFIG_PROC_SYSCTL
2084+ {
2085+ .procname = "cad_pid",
2086+ .data = NULL,
2087+ .maxlen = sizeof (int),
2088+ .mode = 0600,
2089+ .proc_handler = proc_do_cad_pid,
2090+ },
2091+#endif
2092+ {
2093+ .procname = "threads-max",
2094+ .data = NULL,
2095+ .maxlen = sizeof(int),
2096+ .mode = 0644,
2097+ .proc_handler = sysctl_max_threads,
2098+ },
2099+ {
2100+ .procname = "random",
2101+ .mode = 0555,
2102+ .child = random_table,
2103+ },
2104+ {
2105+ .procname = "usermodehelper",
2106+ .mode = 0555,
2107+ .child = usermodehelper_table,
2108+ },
2109+#ifdef CONFIG_FW_LOADER_USER_HELPER
2110+ {
2111+ .procname = "firmware_config",
2112+ .mode = 0555,
2113+ .child = firmware_config_table,
2114+ },
2115+#endif
2116+ {
2117+ .procname = "overflowuid",
2118+ .data = &overflowuid,
2119+ .maxlen = sizeof(int),
2120+ .mode = 0644,
2121+ .proc_handler = proc_dointvec_minmax,
2122+ .extra1 = &minolduid,
2123+ .extra2 = &maxolduid,
2124+ },
2125+ {
2126+ .procname = "overflowgid",
2127+ .data = &overflowgid,
2128+ .maxlen = sizeof(int),
2129+ .mode = 0644,
2130+ .proc_handler = proc_dointvec_minmax,
2131+ .extra1 = &minolduid,
2132+ .extra2 = &maxolduid,
2133+ },
2134+#ifdef CONFIG_S390
2135+ {
2136+ .procname = "userprocess_debug",
2137+ .data = &show_unhandled_signals,
2138+ .maxlen = sizeof(int),
2139+ .mode = 0644,
2140+ .proc_handler = proc_dointvec,
2141+ },
2142+#endif
2143+ {
2144+ .procname = "pid_max",
2145+ .data = &pid_max,
2146+ .maxlen = sizeof (int),
2147+ .mode = 0644,
2148+ .proc_handler = proc_dointvec_minmax,
2149+ .extra1 = &pid_max_min,
2150+ .extra2 = &pid_max_max,
2151+ },
2152+ {
2153+ .procname = "panic_on_oops",
2154+ .data = &panic_on_oops,
2155+ .maxlen = sizeof(int),
2156+ .mode = 0644,
2157+ .proc_handler = proc_dointvec,
2158+ },
2159+ {
2160+ .procname = "panic_print",
2161+ .data = &panic_print,
2162+ .maxlen = sizeof(unsigned long),
2163+ .mode = 0644,
2164+ .proc_handler = proc_doulongvec_minmax,
2165+ },
2166+#if defined CONFIG_PRINTK
2167+ {
2168+ .procname = "printk",
2169+ .data = &console_loglevel,
2170+ .maxlen = 4*sizeof(int),
2171+ .mode = 0644,
2172+ .proc_handler = proc_dointvec,
2173+ },
2174+ {
2175+ .procname = "printk_ratelimit",
2176+ .data = &printk_ratelimit_state.interval,
2177+ .maxlen = sizeof(int),
2178+ .mode = 0644,
2179+ .proc_handler = proc_dointvec_jiffies,
2180+ },
2181+ {
2182+ .procname = "printk_ratelimit_burst",
2183+ .data = &printk_ratelimit_state.burst,
2184+ .maxlen = sizeof(int),
2185+ .mode = 0644,
2186+ .proc_handler = proc_dointvec,
2187+ },
2188+ {
2189+ .procname = "printk_delay",
2190+ .data = &printk_delay_msec,
2191+ .maxlen = sizeof(int),
2192+ .mode = 0644,
2193+ .proc_handler = proc_dointvec_minmax,
2194+ .extra1 = SYSCTL_ZERO,
2195+ .extra2 = &ten_thousand,
2196+ },
2197+ {
2198+ .procname = "printk_devkmsg",
2199+ .data = devkmsg_log_str,
2200+ .maxlen = DEVKMSG_STR_MAX_SIZE,
2201+ .mode = 0644,
2202+ .proc_handler = devkmsg_sysctl_set_loglvl,
2203+ },
2204+ {
2205+ .procname = "dmesg_restrict",
2206+ .data = &dmesg_restrict,
2207+ .maxlen = sizeof(int),
2208+ .mode = 0644,
2209+ .proc_handler = proc_dointvec_minmax_sysadmin,
2210+ .extra1 = SYSCTL_ZERO,
2211+ .extra2 = SYSCTL_ONE,
2212+ },
2213+ {
2214+ .procname = "kptr_restrict",
2215+ .data = &kptr_restrict,
2216+ .maxlen = sizeof(int),
2217+ .mode = 0644,
2218+ .proc_handler = proc_dointvec_minmax_sysadmin,
2219+ .extra1 = SYSCTL_ZERO,
2220+ .extra2 = &two,
2221+ },
2222+#endif
2223+ {
2224+ .procname = "ngroups_max",
2225+ .data = &ngroups_max,
2226+ .maxlen = sizeof (int),
2227+ .mode = 0444,
2228+ .proc_handler = proc_dointvec,
2229+ },
2230+ {
2231+ .procname = "cap_last_cap",
2232+ .data = (void *)&cap_last_cap,
2233+ .maxlen = sizeof(int),
2234+ .mode = 0444,
2235+ .proc_handler = proc_dointvec,
2236+ },
2237+#if defined(CONFIG_LOCKUP_DETECTOR)
2238+ {
2239+ .procname = "watchdog",
2240+ .data = &watchdog_user_enabled,
2241+ .maxlen = sizeof(int),
2242+ .mode = 0644,
2243+ .proc_handler = proc_watchdog,
2244+ .extra1 = SYSCTL_ZERO,
2245+ .extra2 = SYSCTL_ONE,
2246+ },
2247+ {
2248+ .procname = "watchdog_thresh",
2249+ .data = &watchdog_thresh,
2250+ .maxlen = sizeof(int),
2251+ .mode = 0644,
2252+ .proc_handler = proc_watchdog_thresh,
2253+ .extra1 = SYSCTL_ZERO,
2254+ .extra2 = &sixty,
2255+ },
2256+ {
2257+ .procname = "nmi_watchdog",
2258+ .data = &nmi_watchdog_user_enabled,
2259+ .maxlen = sizeof(int),
2260+ .mode = NMI_WATCHDOG_SYSCTL_PERM,
2261+ .proc_handler = proc_nmi_watchdog,
2262+ .extra1 = SYSCTL_ZERO,
2263+ .extra2 = SYSCTL_ONE,
2264+ },
2265+ {
2266+ .procname = "watchdog_cpumask",
2267+ .data = &watchdog_cpumask_bits,
2268+ .maxlen = NR_CPUS,
2269+ .mode = 0644,
2270+ .proc_handler = proc_watchdog_cpumask,
2271+ },
2272+#ifdef CONFIG_SOFTLOCKUP_DETECTOR
2273+ {
2274+ .procname = "soft_watchdog",
2275+ .data = &soft_watchdog_user_enabled,
2276+ .maxlen = sizeof(int),
2277+ .mode = 0644,
2278+ .proc_handler = proc_soft_watchdog,
2279+ .extra1 = SYSCTL_ZERO,
2280+ .extra2 = SYSCTL_ONE,
2281+ },
2282+ {
2283+ .procname = "softlockup_panic",
2284+ .data = &softlockup_panic,
2285+ .maxlen = sizeof(int),
2286+ .mode = 0644,
2287+ .proc_handler = proc_dointvec_minmax,
2288+ .extra1 = SYSCTL_ZERO,
2289+ .extra2 = SYSCTL_ONE,
2290+ },
2291+#ifdef CONFIG_SMP
2292+ {
2293+ .procname = "softlockup_all_cpu_backtrace",
2294+ .data = &sysctl_softlockup_all_cpu_backtrace,
2295+ .maxlen = sizeof(int),
2296+ .mode = 0644,
2297+ .proc_handler = proc_dointvec_minmax,
2298+ .extra1 = SYSCTL_ZERO,
2299+ .extra2 = SYSCTL_ONE,
2300+ },
2301+#endif /* CONFIG_SMP */
2302+#endif
2303+#ifdef CONFIG_HARDLOCKUP_DETECTOR
2304+ {
2305+ .procname = "hardlockup_panic",
2306+ .data = &hardlockup_panic,
2307+ .maxlen = sizeof(int),
2308+ .mode = 0644,
2309+ .proc_handler = proc_dointvec_minmax,
2310+ .extra1 = SYSCTL_ZERO,
2311+ .extra2 = SYSCTL_ONE,
2312+ },
2313+#ifdef CONFIG_SMP
2314+ {
2315+ .procname = "hardlockup_all_cpu_backtrace",
2316+ .data = &sysctl_hardlockup_all_cpu_backtrace,
2317+ .maxlen = sizeof(int),
2318+ .mode = 0644,
2319+ .proc_handler = proc_dointvec_minmax,
2320+ .extra1 = SYSCTL_ZERO,
2321+ .extra2 = SYSCTL_ONE,
2322+ },
2323+#endif /* CONFIG_SMP */
2324+#endif
2325+#endif
2326+
2327+#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
2328+ {
2329+ .procname = "unknown_nmi_panic",
2330+ .data = &unknown_nmi_panic,
2331+ .maxlen = sizeof (int),
2332+ .mode = 0644,
2333+ .proc_handler = proc_dointvec,
2334+ },
2335+#endif
33682336
3369−int proc_dointvec_jiffies(struct ctl_table *table, int write,
3370− void __user *buffer, size_t *lenp, loff_t *ppos)
3371−{
3372− return -ENOSYS;
3373−}
2337+#if (defined(CONFIG_X86_32) || defined(CONFIG_PARISC)) && \
2338+ defined(CONFIG_DEBUG_STACKOVERFLOW)
2339+ {
2340+ .procname = "panic_on_stackoverflow",
2341+ .data = &sysctl_panic_on_stackoverflow,
2342+ .maxlen = sizeof(int),
2343+ .mode = 0644,
2344+ .proc_handler = proc_dointvec,
2345+ },
2346+#endif
2347+#if defined(CONFIG_X86)
2348+ {
2349+ .procname = "panic_on_unrecovered_nmi",
2350+ .data = &panic_on_unrecovered_nmi,
2351+ .maxlen = sizeof(int),
2352+ .mode = 0644,
2353+ .proc_handler = proc_dointvec,
2354+ },
2355+ {
2356+ .procname = "panic_on_io_nmi",
2357+ .data = &panic_on_io_nmi,
2358+ .maxlen = sizeof(int),
2359+ .mode = 0644,
2360+ .proc_handler = proc_dointvec,
2361+ },
2362+ {
2363+ .procname = "bootloader_type",
2364+ .data = &bootloader_type,
2365+ .maxlen = sizeof (int),
2366+ .mode = 0444,
2367+ .proc_handler = proc_dointvec,
2368+ },
2369+ {
2370+ .procname = "bootloader_version",
2371+ .data = &bootloader_version,
2372+ .maxlen = sizeof (int),
2373+ .mode = 0444,
2374+ .proc_handler = proc_dointvec,
2375+ },
2376+ {
2377+ .procname = "io_delay_type",
2378+ .data = &io_delay_type,
2379+ .maxlen = sizeof(int),
2380+ .mode = 0644,
2381+ .proc_handler = proc_dointvec,
2382+ },
2383+#endif
2384+#if defined(CONFIG_MMU)
2385+ {
2386+ .procname = "randomize_va_space",
2387+ .data = &randomize_va_space,
2388+ .maxlen = sizeof(int),
2389+ .mode = 0644,
2390+ .proc_handler = proc_dointvec,
2391+ },
2392+#endif
2393+#if defined(CONFIG_S390) && defined(CONFIG_SMP)
2394+ {
2395+ .procname = "spin_retry",
2396+ .data = &spin_retry,
2397+ .maxlen = sizeof (int),
2398+ .mode = 0644,
2399+ .proc_handler = proc_dointvec,
2400+ },
2401+#endif
2402+#if defined(CONFIG_ACPI_SLEEP) && defined(CONFIG_X86)
2403+ {
2404+ .procname = "acpi_video_flags",
2405+ .data = &acpi_realmode_flags,
2406+ .maxlen = sizeof (unsigned long),
2407+ .mode = 0644,
2408+ .proc_handler = proc_doulongvec_minmax,
2409+ },
2410+#endif
2411+#ifdef CONFIG_SYSCTL_ARCH_UNALIGN_NO_WARN
2412+ {
2413+ .procname = "ignore-unaligned-usertrap",
2414+ .data = &no_unaligned_warning,
2415+ .maxlen = sizeof (int),
2416+ .mode = 0644,
2417+ .proc_handler = proc_dointvec,
2418+ },
2419+#endif
2420+#ifdef CONFIG_IA64
2421+ {
2422+ .procname = "unaligned-dump-stack",
2423+ .data = &unaligned_dump_stack,
2424+ .maxlen = sizeof (int),
2425+ .mode = 0644,
2426+ .proc_handler = proc_dointvec,
2427+ },
2428+#endif
2429+#ifdef CONFIG_DETECT_HUNG_TASK
2430+ {
2431+ .procname = "hung_task_panic",
2432+ .data = &sysctl_hung_task_panic,
2433+ .maxlen = sizeof(int),
2434+ .mode = 0644,
2435+ .proc_handler = proc_dointvec_minmax,
2436+ .extra1 = SYSCTL_ZERO,
2437+ .extra2 = SYSCTL_ONE,
2438+ },
2439+ {
2440+ .procname = "hung_task_check_count",
2441+ .data = &sysctl_hung_task_check_count,
2442+ .maxlen = sizeof(int),
2443+ .mode = 0644,
2444+ .proc_handler = proc_dointvec_minmax,
2445+ .extra1 = SYSCTL_ZERO,
2446+ },
2447+ {
2448+ .procname = "hung_task_timeout_secs",
2449+ .data = &sysctl_hung_task_timeout_secs,
2450+ .maxlen = sizeof(unsigned long),
2451+ .mode = 0644,
2452+ .proc_handler = proc_dohung_task_timeout_secs,
2453+ .extra2 = &hung_task_timeout_max,
2454+ },
2455+ {
2456+ .procname = "hung_task_check_interval_secs",
2457+ .data = &sysctl_hung_task_check_interval_secs,
2458+ .maxlen = sizeof(unsigned long),
2459+ .mode = 0644,
2460+ .proc_handler = proc_dohung_task_timeout_secs,
2461+ .extra2 = &hung_task_timeout_max,
2462+ },
2463+ {
2464+ .procname = "hung_task_warnings",
2465+ .data = &sysctl_hung_task_warnings,
2466+ .maxlen = sizeof(int),
2467+ .mode = 0644,
2468+ .proc_handler = proc_dointvec_minmax,
2469+ .extra1 = &neg_one,
2470+ },
2471+#endif
2472+#ifdef CONFIG_RT_MUTEXES
2473+ {
2474+ .procname = "max_lock_depth",
2475+ .data = &max_lock_depth,
2476+ .maxlen = sizeof(int),
2477+ .mode = 0644,
2478+ .proc_handler = proc_dointvec,
2479+ },
2480+#endif
2481+ {
2482+ .procname = "poweroff_cmd",
2483+ .data = &poweroff_cmd,
2484+ .maxlen = POWEROFF_CMD_PATH_LEN,
2485+ .mode = 0644,
2486+ .proc_handler = proc_dostring,
2487+ },
2488+#ifdef CONFIG_KEYS
2489+ {
2490+ .procname = "keys",
2491+ .mode = 0555,
2492+ .child = key_sysctls,
2493+ },
2494+#endif
2495+#ifdef CONFIG_PERF_EVENTS
2496+ /*
2497+ * User-space scripts rely on the existence of this file
2498+ * as a feature check for perf_events being enabled.
2499+ *
2500+ * So it's an ABI, do not remove!
2501+ */
2502+ {
2503+ .procname = "perf_event_paranoid",
2504+ .data = &sysctl_perf_event_paranoid,
2505+ .maxlen = sizeof(sysctl_perf_event_paranoid),
2506+ .mode = 0644,
2507+ .proc_handler = proc_dointvec,
2508+ },
2509+ {
2510+ .procname = "perf_event_mlock_kb",
2511+ .data = &sysctl_perf_event_mlock,
2512+ .maxlen = sizeof(sysctl_perf_event_mlock),
2513+ .mode = 0644,
2514+ .proc_handler = proc_dointvec,
2515+ },
2516+ {
2517+ .procname = "perf_event_max_sample_rate",
2518+ .data = &sysctl_perf_event_sample_rate,
2519+ .maxlen = sizeof(sysctl_perf_event_sample_rate),
2520+ .mode = 0644,
2521+ .proc_handler = perf_proc_update_handler,
2522+ .extra1 = SYSCTL_ONE,
2523+ },
2524+ {
2525+ .procname = "perf_cpu_time_max_percent",
2526+ .data = &sysctl_perf_cpu_time_max_percent,
2527+ .maxlen = sizeof(sysctl_perf_cpu_time_max_percent),
2528+ .mode = 0644,
2529+ .proc_handler = perf_cpu_time_max_percent_handler,
2530+ .extra1 = SYSCTL_ZERO,
2531+ .extra2 = &one_hundred,
2532+ },
2533+ {
2534+ .procname = "perf_event_max_stack",
2535+ .data = &sysctl_perf_event_max_stack,
2536+ .maxlen = sizeof(sysctl_perf_event_max_stack),
2537+ .mode = 0644,
2538+ .proc_handler = perf_event_max_stack_handler,
2539+ .extra1 = SYSCTL_ZERO,
2540+ .extra2 = &six_hundred_forty_kb,
2541+ },
2542+ {
2543+ .procname = "perf_event_max_contexts_per_stack",
2544+ .data = &sysctl_perf_event_max_contexts_per_stack,
2545+ .maxlen = sizeof(sysctl_perf_event_max_contexts_per_stack),
2546+ .mode = 0644,
2547+ .proc_handler = perf_event_max_stack_handler,
2548+ .extra1 = SYSCTL_ZERO,
2549+ .extra2 = &one_thousand,
2550+ },
2551+#endif
2552+ {
2553+ .procname = "panic_on_warn",
2554+ .data = &panic_on_warn,
2555+ .maxlen = sizeof(int),
2556+ .mode = 0644,
2557+ .proc_handler = proc_dointvec_minmax,
2558+ .extra1 = SYSCTL_ZERO,
2559+ .extra2 = SYSCTL_ONE,
2560+ },
2561+#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
2562+ {
2563+ .procname = "timer_migration",
2564+ .data = &sysctl_timer_migration,
2565+ .maxlen = sizeof(unsigned int),
2566+ .mode = 0644,
2567+ .proc_handler = timer_migration_handler,
2568+ .extra1 = SYSCTL_ZERO,
2569+ .extra2 = SYSCTL_ONE,
2570+ },
2571+#endif
2572+#ifdef CONFIG_BPF_SYSCALL
2573+ {
2574+ .procname = "unprivileged_bpf_disabled",
2575+ .data = &sysctl_unprivileged_bpf_disabled,
2576+ .maxlen = sizeof(sysctl_unprivileged_bpf_disabled),
2577+ .mode = 0644,
2578+ /* only handle a transition from default "0" to "1" */
2579+ .proc_handler = proc_dointvec_minmax,
2580+ .extra1 = SYSCTL_ONE,
2581+ .extra2 = SYSCTL_ONE,
2582+ },
2583+ {
2584+ .procname = "bpf_stats_enabled",
2585+ .data = &bpf_stats_enabled_key.key,
2586+ .maxlen = sizeof(bpf_stats_enabled_key),
2587+ .mode = 0644,
2588+ .proc_handler = bpf_stats_handler,
2589+ },
2590+#endif
2591+#if defined(CONFIG_TREE_RCU)
2592+ {
2593+ .procname = "panic_on_rcu_stall",
2594+ .data = &sysctl_panic_on_rcu_stall,
2595+ .maxlen = sizeof(sysctl_panic_on_rcu_stall),
2596+ .mode = 0644,
2597+ .proc_handler = proc_dointvec_minmax,
2598+ .extra1 = SYSCTL_ZERO,
2599+ .extra2 = SYSCTL_ONE,
2600+ },
2601+#endif
2602+#ifdef CONFIG_STACKLEAK_RUNTIME_DISABLE
2603+ {
2604+ .procname = "stack_erasing",
2605+ .data = NULL,
2606+ .maxlen = sizeof(int),
2607+ .mode = 0600,
2608+ .proc_handler = stack_erasing_sysctl,
2609+ .extra1 = SYSCTL_ZERO,
2610+ .extra2 = SYSCTL_ONE,
2611+ },
2612+#endif
2613+ { }
2614+};
33742615
3375−int proc_dointvec_userhz_jiffies(struct ctl_table *table, int write,
3376− void __user *buffer, size_t *lenp, loff_t *ppos)
3377−{
3378− return -ENOSYS;
3379−}
2616+static struct ctl_table vm_table[] = {
2617+ {
2618+ .procname = "overcommit_memory",
2619+ .data = &sysctl_overcommit_memory,
2620+ .maxlen = sizeof(sysctl_overcommit_memory),
2621+ .mode = 0644,
2622+ .proc_handler = proc_dointvec_minmax,
2623+ .extra1 = SYSCTL_ZERO,
2624+ .extra2 = &two,
2625+ },
2626+ {
2627+ .procname = "panic_on_oom",
2628+ .data = &sysctl_panic_on_oom,
2629+ .maxlen = sizeof(sysctl_panic_on_oom),
2630+ .mode = 0644,
2631+ .proc_handler = proc_dointvec_minmax,
2632+ .extra1 = SYSCTL_ZERO,
2633+ .extra2 = &two,
2634+ },
2635+ {
2636+ .procname = "oom_kill_allocating_task",
2637+ .data = &sysctl_oom_kill_allocating_task,
2638+ .maxlen = sizeof(sysctl_oom_kill_allocating_task),
2639+ .mode = 0644,
2640+ .proc_handler = proc_dointvec,
2641+ },
2642+ {
2643+ .procname = "oom_dump_tasks",
2644+ .data = &sysctl_oom_dump_tasks,
2645+ .maxlen = sizeof(sysctl_oom_dump_tasks),
2646+ .mode = 0644,
2647+ .proc_handler = proc_dointvec,
2648+ },
2649+ {
2650+ .procname = "overcommit_ratio",
2651+ .data = &sysctl_overcommit_ratio,
2652+ .maxlen = sizeof(sysctl_overcommit_ratio),
2653+ .mode = 0644,
2654+ .proc_handler = overcommit_ratio_handler,
2655+ },
2656+ {
2657+ .procname = "overcommit_kbytes",
2658+ .data = &sysctl_overcommit_kbytes,
2659+ .maxlen = sizeof(sysctl_overcommit_kbytes),
2660+ .mode = 0644,
2661+ .proc_handler = overcommit_kbytes_handler,
2662+ },
2663+ {
2664+ .procname = "page-cluster",
2665+ .data = &page_cluster,
2666+ .maxlen = sizeof(int),
2667+ .mode = 0644,
2668+ .proc_handler = proc_dointvec_minmax,
2669+ .extra1 = SYSCTL_ZERO,
2670+ },
2671+ {
2672+ .procname = "dirty_background_ratio",
2673+ .data = &dirty_background_ratio,
2674+ .maxlen = sizeof(dirty_background_ratio),
2675+ .mode = 0644,
2676+ .proc_handler = dirty_background_ratio_handler,
2677+ .extra1 = SYSCTL_ZERO,
2678+ .extra2 = &one_hundred,
2679+ },
2680+ {
2681+ .procname = "dirty_background_bytes",
2682+ .data = &dirty_background_bytes,
2683+ .maxlen = sizeof(dirty_background_bytes),
2684+ .mode = 0644,
2685+ .proc_handler = dirty_background_bytes_handler,
2686+ .extra1 = &one_ul,
2687+ },
2688+ {
2689+ .procname = "dirty_ratio",
2690+ .data = &vm_dirty_ratio,
2691+ .maxlen = sizeof(vm_dirty_ratio),
2692+ .mode = 0644,
2693+ .proc_handler = dirty_ratio_handler,
2694+ .extra1 = SYSCTL_ZERO,
2695+ .extra2 = &one_hundred,
2696+ },
2697+ {
2698+ .procname = "dirty_bytes",
2699+ .data = &vm_dirty_bytes,
2700+ .maxlen = sizeof(vm_dirty_bytes),
2701+ .mode = 0644,
2702+ .proc_handler = dirty_bytes_handler,
2703+ .extra1 = &dirty_bytes_min,
2704+ },
2705+ {
2706+ .procname = "dirty_writeback_centisecs",
2707+ .data = &dirty_writeback_interval,
2708+ .maxlen = sizeof(dirty_writeback_interval),
2709+ .mode = 0644,
2710+ .proc_handler = dirty_writeback_centisecs_handler,
2711+ },
2712+ {
2713+ .procname = "dirty_expire_centisecs",
2714+ .data = &dirty_expire_interval,
2715+ .maxlen = sizeof(dirty_expire_interval),
2716+ .mode = 0644,
2717+ .proc_handler = proc_dointvec_minmax,
2718+ .extra1 = SYSCTL_ZERO,
2719+ },
2720+ {
2721+ .procname = "dirtytime_expire_seconds",
2722+ .data = &dirtytime_expire_interval,
2723+ .maxlen = sizeof(dirtytime_expire_interval),
2724+ .mode = 0644,
2725+ .proc_handler = dirtytime_interval_handler,
2726+ .extra1 = SYSCTL_ZERO,
2727+ },
2728+ {
2729+ .procname = "swappiness",
2730+ .data = &vm_swappiness,
2731+ .maxlen = sizeof(vm_swappiness),
2732+ .mode = 0644,
2733+ .proc_handler = proc_dointvec_minmax,
2734+ .extra1 = SYSCTL_ZERO,
2735+ .extra2 = &one_hundred,
2736+ },
2737+#ifdef CONFIG_HUGETLB_PAGE
2738+ {
2739+ .procname = "nr_hugepages",
2740+ .data = NULL,
2741+ .maxlen = sizeof(unsigned long),
2742+ .mode = 0644,
2743+ .proc_handler = hugetlb_sysctl_handler,
2744+ },
2745+#ifdef CONFIG_NUMA
2746+ {
2747+ .procname = "nr_hugepages_mempolicy",
2748+ .data = NULL,
2749+ .maxlen = sizeof(unsigned long),
2750+ .mode = 0644,
2751+ .proc_handler = &hugetlb_mempolicy_sysctl_handler,
2752+ },
2753+ {
2754+ .procname = "numa_stat",
2755+ .data = &sysctl_vm_numa_stat,
2756+ .maxlen = sizeof(int),
2757+ .mode = 0644,
2758+ .proc_handler = sysctl_vm_numa_stat_handler,
2759+ .extra1 = SYSCTL_ZERO,
2760+ .extra2 = SYSCTL_ONE,
2761+ },
2762+#endif
2763+ {
2764+ .procname = "hugetlb_shm_group",
2765+ .data = &sysctl_hugetlb_shm_group,
2766+ .maxlen = sizeof(gid_t),
2767+ .mode = 0644,
2768+ .proc_handler = proc_dointvec,
2769+ },
2770+ {
2771+ .procname = "nr_overcommit_hugepages",
2772+ .data = NULL,
2773+ .maxlen = sizeof(unsigned long),
2774+ .mode = 0644,
2775+ .proc_handler = hugetlb_overcommit_handler,
2776+ },
2777+#endif
2778+ {
2779+ .procname = "lowmem_reserve_ratio",
2780+ .data = &sysctl_lowmem_reserve_ratio,
2781+ .maxlen = sizeof(sysctl_lowmem_reserve_ratio),
2782+ .mode = 0644,
2783+ .proc_handler = lowmem_reserve_ratio_sysctl_handler,
2784+ },
2785+ {
2786+ .procname = "drop_caches",
2787+ .data = &sysctl_drop_caches,
2788+ .maxlen = sizeof(int),
2789+ .mode = 0200,
2790+ .proc_handler = drop_caches_sysctl_handler,
2791+ .extra1 = SYSCTL_ONE,
2792+ .extra2 = &four,
2793+ },
2794+#ifdef CONFIG_COMPACTION
2795+ {
2796+ .procname = "compact_memory",
2797+ .data = &sysctl_compact_memory,
2798+ .maxlen = sizeof(int),
2799+ .mode = 0200,
2800+ .proc_handler = sysctl_compaction_handler,
2801+ },
2802+ {
2803+ .procname = "extfrag_threshold",
2804+ .data = &sysctl_extfrag_threshold,
2805+ .maxlen = sizeof(int),
2806+ .mode = 0644,
2807+ .proc_handler = proc_dointvec_minmax,
2808+ .extra1 = &min_extfrag_threshold,
2809+ .extra2 = &max_extfrag_threshold,
2810+ },
2811+ {
2812+ .procname = "compact_unevictable_allowed",
2813+ .data = &sysctl_compact_unevictable_allowed,
2814+ .maxlen = sizeof(int),
2815+ .mode = 0644,
2816+ .proc_handler = proc_dointvec_minmax_warn_RT_change,
2817+ .extra1 = SYSCTL_ZERO,
2818+ .extra2 = SYSCTL_ONE,
2819+ },
33802820
3381−int proc_dointvec_ms_jiffies(struct ctl_table *table, int write,
3382− void __user *buffer, size_t *lenp, loff_t *ppos)
3383−{
3384− return -ENOSYS;
3385−}
2821+#endif /* CONFIG_COMPACTION */
2822+ {
2823+ .procname = "min_free_kbytes",
2824+ .data = &min_free_kbytes,
2825+ .maxlen = sizeof(min_free_kbytes),
2826+ .mode = 0644,
2827+ .proc_handler = min_free_kbytes_sysctl_handler,
2828+ .extra1 = SYSCTL_ZERO,
2829+ },
2830+ {
2831+ .procname = "watermark_boost_factor",
2832+ .data = &watermark_boost_factor,
2833+ .maxlen = sizeof(watermark_boost_factor),
2834+ .mode = 0644,
2835+ .proc_handler = proc_dointvec_minmax,
2836+ .extra1 = SYSCTL_ZERO,
2837+ },
2838+ {
2839+ .procname = "watermark_scale_factor",
2840+ .data = &watermark_scale_factor,
2841+ .maxlen = sizeof(watermark_scale_factor),
2842+ .mode = 0644,
2843+ .proc_handler = watermark_scale_factor_sysctl_handler,
2844+ .extra1 = SYSCTL_ONE,
2845+ .extra2 = &one_thousand,
2846+ },
2847+ {
2848+ .procname = "percpu_pagelist_fraction",
2849+ .data = &percpu_pagelist_fraction,
2850+ .maxlen = sizeof(percpu_pagelist_fraction),
2851+ .mode = 0644,
2852+ .proc_handler = percpu_pagelist_fraction_sysctl_handler,
2853+ .extra1 = SYSCTL_ZERO,
2854+ },
2855+#ifdef CONFIG_MMU
2856+ {
2857+ .procname = "max_map_count",
2858+ .data = &sysctl_max_map_count,
2859+ .maxlen = sizeof(sysctl_max_map_count),
2860+ .mode = 0644,
2861+ .proc_handler = proc_dointvec_minmax,
2862+ .extra1 = SYSCTL_ZERO,
2863+ },
2864+#else
2865+ {
2866+ .procname = "nr_trim_pages",
2867+ .data = &sysctl_nr_trim_pages,
2868+ .maxlen = sizeof(sysctl_nr_trim_pages),
2869+ .mode = 0644,
2870+ .proc_handler = proc_dointvec_minmax,
2871+ .extra1 = SYSCTL_ZERO,
2872+ },
2873+#endif
2874+ {
2875+ .procname = "laptop_mode",
2876+ .data = &laptop_mode,
2877+ .maxlen = sizeof(laptop_mode),
2878+ .mode = 0644,
2879+ .proc_handler = proc_dointvec_jiffies,
2880+ },
2881+ {
2882+ .procname = "block_dump",
2883+ .data = &block_dump,
2884+ .maxlen = sizeof(block_dump),
2885+ .mode = 0644,
2886+ .proc_handler = proc_dointvec,
2887+ .extra1 = SYSCTL_ZERO,
2888+ },
2889+ {
2890+ .procname = "vfs_cache_pressure",
2891+ .data = &sysctl_vfs_cache_pressure,
2892+ .maxlen = sizeof(sysctl_vfs_cache_pressure),
2893+ .mode = 0644,
2894+ .proc_handler = proc_dointvec,
2895+ .extra1 = SYSCTL_ZERO,
2896+ },
2897+#if defined(HAVE_ARCH_PICK_MMAP_LAYOUT) || \
2898+ defined(CONFIG_ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT)
2899+ {
2900+ .procname = "legacy_va_layout",
2901+ .data = &sysctl_legacy_va_layout,
2902+ .maxlen = sizeof(sysctl_legacy_va_layout),
2903+ .mode = 0644,
2904+ .proc_handler = proc_dointvec,
2905+ .extra1 = SYSCTL_ZERO,
2906+ },
2907+#endif
2908+#ifdef CONFIG_NUMA
2909+ {
2910+ .procname = "zone_reclaim_mode",
2911+ .data = &node_reclaim_mode,
2912+ .maxlen = sizeof(node_reclaim_mode),
2913+ .mode = 0644,
2914+ .proc_handler = proc_dointvec,
2915+ .extra1 = SYSCTL_ZERO,
2916+ },
2917+ {
2918+ .procname = "min_unmapped_ratio",
2919+ .data = &sysctl_min_unmapped_ratio,
2920+ .maxlen = sizeof(sysctl_min_unmapped_ratio),
2921+ .mode = 0644,
2922+ .proc_handler = sysctl_min_unmapped_ratio_sysctl_handler,
2923+ .extra1 = SYSCTL_ZERO,
2924+ .extra2 = &one_hundred,
2925+ },
2926+ {
2927+ .procname = "min_slab_ratio",
2928+ .data = &sysctl_min_slab_ratio,
2929+ .maxlen = sizeof(sysctl_min_slab_ratio),
2930+ .mode = 0644,
2931+ .proc_handler = sysctl_min_slab_ratio_sysctl_handler,
2932+ .extra1 = SYSCTL_ZERO,
2933+ .extra2 = &one_hundred,
2934+ },
2935+#endif
2936+#ifdef CONFIG_SMP
2937+ {
2938+ .procname = "stat_interval",
2939+ .data = &sysctl_stat_interval,
2940+ .maxlen = sizeof(sysctl_stat_interval),
2941+ .mode = 0644,
2942+ .proc_handler = proc_dointvec_jiffies,
2943+ },
2944+ {
2945+ .procname = "stat_refresh",
2946+ .data = NULL,
2947+ .maxlen = 0,
2948+ .mode = 0600,
2949+ .proc_handler = vmstat_refresh,
2950+ },
2951+#endif
2952+#ifdef CONFIG_MMU
2953+ {
2954+ .procname = "mmap_min_addr",
2955+ .data = &dac_mmap_min_addr,
2956+ .maxlen = sizeof(unsigned long),
2957+ .mode = 0644,
2958+ .proc_handler = mmap_min_addr_handler,
2959+ },
2960+#endif
2961+#ifdef CONFIG_NUMA
2962+ {
2963+ .procname = "numa_zonelist_order",
2964+ .data = &numa_zonelist_order,
2965+ .maxlen = NUMA_ZONELIST_ORDER_LEN,
2966+ .mode = 0644,
2967+ .proc_handler = numa_zonelist_order_handler,
2968+ },
2969+#endif
2970+#if (defined(CONFIG_X86_32) && !defined(CONFIG_UML))|| \
2971+ (defined(CONFIG_SUPERH) && defined(CONFIG_VSYSCALL))
2972+ {
2973+ .procname = "vdso_enabled",
2974+#ifdef CONFIG_X86_32
2975+ .data = &vdso32_enabled,
2976+ .maxlen = sizeof(vdso32_enabled),
2977+#else
2978+ .data = &vdso_enabled,
2979+ .maxlen = sizeof(vdso_enabled),
2980+#endif
2981+ .mode = 0644,
2982+ .proc_handler = proc_dointvec,
2983+ .extra1 = SYSCTL_ZERO,
2984+ },
2985+#endif
2986+#ifdef CONFIG_HIGHMEM
2987+ {
2988+ .procname = "highmem_is_dirtyable",
2989+ .data = &vm_highmem_is_dirtyable,
2990+ .maxlen = sizeof(vm_highmem_is_dirtyable),
2991+ .mode = 0644,
2992+ .proc_handler = proc_dointvec_minmax,
2993+ .extra1 = SYSCTL_ZERO,
2994+ .extra2 = SYSCTL_ONE,
2995+ },
2996+#endif
2997+#ifdef CONFIG_MEMORY_FAILURE
2998+ {
2999+ .procname = "memory_failure_early_kill",
3000+ .data = &sysctl_memory_failure_early_kill,
3001+ .maxlen = sizeof(sysctl_memory_failure_early_kill),
3002+ .mode = 0644,
3003+ .proc_handler = proc_dointvec_minmax,
3004+ .extra1 = SYSCTL_ZERO,
3005+ .extra2 = SYSCTL_ONE,
3006+ },
3007+ {
3008+ .procname = "memory_failure_recovery",
3009+ .data = &sysctl_memory_failure_recovery,
3010+ .maxlen = sizeof(sysctl_memory_failure_recovery),
3011+ .mode = 0644,
3012+ .proc_handler = proc_dointvec_minmax,
3013+ .extra1 = SYSCTL_ZERO,
3014+ .extra2 = SYSCTL_ONE,
3015+ },
3016+#endif
3017+ {
3018+ .procname = "user_reserve_kbytes",
3019+ .data = &sysctl_user_reserve_kbytes,
3020+ .maxlen = sizeof(sysctl_user_reserve_kbytes),
3021+ .mode = 0644,
3022+ .proc_handler = proc_doulongvec_minmax,
3023+ },
3024+ {
3025+ .procname = "admin_reserve_kbytes",
3026+ .data = &sysctl_admin_reserve_kbytes,
3027+ .maxlen = sizeof(sysctl_admin_reserve_kbytes),
3028+ .mode = 0644,
3029+ .proc_handler = proc_doulongvec_minmax,
3030+ },
3031+#ifdef CONFIG_HAVE_ARCH_MMAP_RND_BITS
3032+ {
3033+ .procname = "mmap_rnd_bits",
3034+ .data = &mmap_rnd_bits,
3035+ .maxlen = sizeof(mmap_rnd_bits),
3036+ .mode = 0600,
3037+ .proc_handler = proc_dointvec_minmax,
3038+ .extra1 = (void *)&mmap_rnd_bits_min,
3039+ .extra2 = (void *)&mmap_rnd_bits_max,
3040+ },
3041+#endif
3042+#ifdef CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS
3043+ {
3044+ .procname = "mmap_rnd_compat_bits",
3045+ .data = &mmap_rnd_compat_bits,
3046+ .maxlen = sizeof(mmap_rnd_compat_bits),
3047+ .mode = 0600,
3048+ .proc_handler = proc_dointvec_minmax,
3049+ .extra1 = (void *)&mmap_rnd_compat_bits_min,
3050+ .extra2 = (void *)&mmap_rnd_compat_bits_max,
3051+ },
3052+#endif
3053+#ifdef CONFIG_USERFAULTFD
3054+ {
3055+ .procname = "unprivileged_userfaultfd",
3056+ .data = &sysctl_unprivileged_userfaultfd,
3057+ .maxlen = sizeof(sysctl_unprivileged_userfaultfd),
3058+ .mode = 0644,
3059+ .proc_handler = proc_dointvec_minmax,
3060+ .extra1 = SYSCTL_ZERO,
3061+ .extra2 = SYSCTL_ONE,
3062+ },
3063+#endif
3064+ { }
3065+};
33863066
3387−int proc_doulongvec_minmax(struct ctl_table *table, int write,
3388− void __user *buffer, size_t *lenp, loff_t *ppos)
3389−{
3390− return -ENOSYS;
3391−}
3067+static struct ctl_table fs_table[] = {
3068+ {
3069+ .procname = "inode-nr",
3070+ .data = &inodes_stat,
3071+ .maxlen = 2*sizeof(long),
3072+ .mode = 0444,
3073+ .proc_handler = proc_nr_inodes,
3074+ },
3075+ {
3076+ .procname = "inode-state",
3077+ .data = &inodes_stat,
3078+ .maxlen = 7*sizeof(long),
3079+ .mode = 0444,
3080+ .proc_handler = proc_nr_inodes,
3081+ },
3082+ {
3083+ .procname = "file-nr",
3084+ .data = &files_stat,
3085+ .maxlen = sizeof(files_stat),
3086+ .mode = 0444,
3087+ .proc_handler = proc_nr_files,
3088+ },
3089+ {
3090+ .procname = "file-max",
3091+ .data = &files_stat.max_files,
3092+ .maxlen = sizeof(files_stat.max_files),
3093+ .mode = 0644,
3094+ .proc_handler = proc_doulongvec_minmax,
3095+ .extra1 = &zero_ul,
3096+ .extra2 = &long_max,
3097+ },
3098+ {
3099+ .procname = "nr_open",
3100+ .data = &sysctl_nr_open,
3101+ .maxlen = sizeof(unsigned int),
3102+ .mode = 0644,
3103+ .proc_handler = proc_dointvec_minmax,
3104+ .extra1 = &sysctl_nr_open_min,
3105+ .extra2 = &sysctl_nr_open_max,
3106+ },
3107+ {
3108+ .procname = "dentry-state",
3109+ .data = &dentry_stat,
3110+ .maxlen = 6*sizeof(long),
3111+ .mode = 0444,
3112+ .proc_handler = proc_nr_dentry,
3113+ },
3114+ {
3115+ .procname = "overflowuid",
3116+ .data = &fs_overflowuid,
3117+ .maxlen = sizeof(int),
3118+ .mode = 0644,
3119+ .proc_handler = proc_dointvec_minmax,
3120+ .extra1 = &minolduid,
3121+ .extra2 = &maxolduid,
3122+ },
3123+ {
3124+ .procname = "overflowgid",
3125+ .data = &fs_overflowgid,
3126+ .maxlen = sizeof(int),
3127+ .mode = 0644,
3128+ .proc_handler = proc_dointvec_minmax,
3129+ .extra1 = &minolduid,
3130+ .extra2 = &maxolduid,
3131+ },
3132+#ifdef CONFIG_FILE_LOCKING
3133+ {
3134+ .procname = "leases-enable",
3135+ .data = &leases_enable,
3136+ .maxlen = sizeof(int),
3137+ .mode = 0644,
3138+ .proc_handler = proc_dointvec,
3139+ },
3140+#endif
3141+#ifdef CONFIG_DNOTIFY
3142+ {
3143+ .procname = "dir-notify-enable",
3144+ .data = &dir_notify_enable,
3145+ .maxlen = sizeof(int),
3146+ .mode = 0644,
3147+ .proc_handler = proc_dointvec,
3148+ },
3149+#endif
3150+#ifdef CONFIG_MMU
3151+#ifdef CONFIG_FILE_LOCKING
3152+ {
3153+ .procname = "lease-break-time",
3154+ .data = &lease_break_time,
3155+ .maxlen = sizeof(int),
3156+ .mode = 0644,
3157+ .proc_handler = proc_dointvec,
3158+ },
3159+#endif
3160+#ifdef CONFIG_AIO
3161+ {
3162+ .procname = "aio-nr",
3163+ .data = &aio_nr,
3164+ .maxlen = sizeof(aio_nr),
3165+ .mode = 0444,
3166+ .proc_handler = proc_doulongvec_minmax,
3167+ },
3168+ {
3169+ .procname = "aio-max-nr",
3170+ .data = &aio_max_nr,
3171+ .maxlen = sizeof(aio_max_nr),
3172+ .mode = 0644,
3173+ .proc_handler = proc_doulongvec_minmax,
3174+ },
3175+#endif /* CONFIG_AIO */
3176+#ifdef CONFIG_INOTIFY_USER
3177+ {
3178+ .procname = "inotify",
3179+ .mode = 0555,
3180+ .child = inotify_table,
3181+ },
3182+#endif
3183+#ifdef CONFIG_EPOLL
3184+ {
3185+ .procname = "epoll",
3186+ .mode = 0555,
3187+ .child = epoll_table,
3188+ },
3189+#endif
3190+#endif
3191+ {
3192+ .procname = "protected_symlinks",
3193+ .data = &sysctl_protected_symlinks,
3194+ .maxlen = sizeof(int),
3195+ .mode = 0600,
3196+ .proc_handler = proc_dointvec_minmax,
3197+ .extra1 = SYSCTL_ZERO,
3198+ .extra2 = SYSCTL_ONE,
3199+ },
3200+ {
3201+ .procname = "protected_hardlinks",
3202+ .data = &sysctl_protected_hardlinks,
3203+ .maxlen = sizeof(int),
3204+ .mode = 0600,
3205+ .proc_handler = proc_dointvec_minmax,
3206+ .extra1 = SYSCTL_ZERO,
3207+ .extra2 = SYSCTL_ONE,
3208+ },
3209+ {
3210+ .procname = "protected_fifos",
3211+ .data = &sysctl_protected_fifos,
3212+ .maxlen = sizeof(int),
3213+ .mode = 0600,
3214+ .proc_handler = proc_dointvec_minmax,
3215+ .extra1 = SYSCTL_ZERO,
3216+ .extra2 = &two,
3217+ },
3218+ {
3219+ .procname = "protected_regular",
3220+ .data = &sysctl_protected_regular,
3221+ .maxlen = sizeof(int),
3222+ .mode = 0600,
3223+ .proc_handler = proc_dointvec_minmax,
3224+ .extra1 = SYSCTL_ZERO,
3225+ .extra2 = &two,
3226+ },
3227+ {
3228+ .procname = "suid_dumpable",
3229+ .data = &suid_dumpable,
3230+ .maxlen = sizeof(int),
3231+ .mode = 0644,
3232+ .proc_handler = proc_dointvec_minmax_coredump,
3233+ .extra1 = SYSCTL_ZERO,
3234+ .extra2 = &two,
3235+ },
3236+#if defined(CONFIG_BINFMT_MISC) || defined(CONFIG_BINFMT_MISC_MODULE)
3237+ {
3238+ .procname = "binfmt_misc",
3239+ .mode = 0555,
3240+ .child = sysctl_mount_point,
3241+ },
3242+#endif
3243+ {
3244+ .procname = "pipe-max-size",
3245+ .data = &pipe_max_size,
3246+ .maxlen = sizeof(pipe_max_size),
3247+ .mode = 0644,
3248+ .proc_handler = proc_dopipe_max_size,
3249+ },
3250+ {
3251+ .procname = "pipe-user-pages-hard",
3252+ .data = &pipe_user_pages_hard,
3253+ .maxlen = sizeof(pipe_user_pages_hard),
3254+ .mode = 0644,
3255+ .proc_handler = proc_doulongvec_minmax,
3256+ },
3257+ {
3258+ .procname = "pipe-user-pages-soft",
3259+ .data = &pipe_user_pages_soft,
3260+ .maxlen = sizeof(pipe_user_pages_soft),
3261+ .mode = 0644,
3262+ .proc_handler = proc_doulongvec_minmax,
3263+ },
3264+ {
3265+ .procname = "mount-max",
3266+ .data = &sysctl_mount_max,
3267+ .maxlen = sizeof(unsigned int),
3268+ .mode = 0644,
3269+ .proc_handler = proc_dointvec_minmax,
3270+ .extra1 = SYSCTL_ONE,
3271+ },
3272+ { }
3273+};
33923274
3393−int proc_doulongvec_ms_jiffies_minmax(struct ctl_table *table, int write,
3394− void __user *buffer,
3395− size_t *lenp, loff_t *ppos)
3396−{
3397− return -ENOSYS;
3398−}
3275+static struct ctl_table debug_table[] = {
3276+#ifdef CONFIG_SYSCTL_EXCEPTION_TRACE
3277+ {
3278+ .procname = "exception-trace",
3279+ .data = &show_unhandled_signals,
3280+ .maxlen = sizeof(int),
3281+ .mode = 0644,
3282+ .proc_handler = proc_dointvec
3283+ },
3284+#endif
3285+#if defined(CONFIG_OPTPROBES)
3286+ {
3287+ .procname = "kprobes-optimization",
3288+ .data = &sysctl_kprobes_optimization,
3289+ .maxlen = sizeof(int),
3290+ .mode = 0644,
3291+ .proc_handler = proc_kprobes_optimization_handler,
3292+ .extra1 = SYSCTL_ZERO,
3293+ .extra2 = SYSCTL_ONE,
3294+ },
3295+#endif
3296+ { }
3297+};
33993298
3400−int proc_do_large_bitmap(struct ctl_table *table, int write,
3401− void __user *buffer, size_t *lenp, loff_t *ppos)
3402−{
3403− return -ENOSYS;
3404−}
3299+static struct ctl_table dev_table[] = {
3300+ { }
3301+};
34053302
3406−#endif /* CONFIG_PROC_SYSCTL */
3303+static struct ctl_table sysctl_base_table[] = {
3304+ {
3305+ .procname = "kernel",
3306+ .mode = 0555,
3307+ .child = kern_table,
3308+ },
3309+ {
3310+ .procname = "vm",
3311+ .mode = 0555,
3312+ .child = vm_table,
3313+ },
3314+ {
3315+ .procname = "fs",
3316+ .mode = 0555,
3317+ .child = fs_table,
3318+ },
3319+ {
3320+ .procname = "debug",
3321+ .mode = 0555,
3322+ .child = debug_table,
3323+ },
3324+ {
3325+ .procname = "dev",
3326+ .mode = 0555,
3327+ .child = dev_table,
3328+ },
3329+ { }
3330+};
34073331
3408−#if defined(CONFIG_SYSCTL)
3409−int proc_do_static_key(struct ctl_table *table, int write,
3410− void __user *buffer, size_t *lenp,
3411− loff_t *ppos)
3332+int __init sysctl_init(void)
34123333 {
3413− struct static_key *key = (struct static_key *)table->data;
3414− static DEFINE_MUTEX(static_key_mutex);
3415− int val, ret;
3416− struct ctl_table tmp = {
3417− .data = &val,
3418− .maxlen = sizeof(val),
3419− .mode = table->mode,
3420− .extra1 = SYSCTL_ZERO,
3421− .extra2 = SYSCTL_ONE,
3422− };
3423−
3424− if (write && !capable(CAP_SYS_ADMIN))
3425− return -EPERM;
3334+ struct ctl_table_header *hdr;
34263335
3427− mutex_lock(&static_key_mutex);
3428− val = static_key_enabled(key);
3429− ret = proc_dointvec_minmax(&tmp, write, buffer, lenp, ppos);
3430− if (write && !ret) {
3431− if (val)
3432− static_key_enable(key);
3433− else
3434− static_key_disable(key);
3435− }
3436− mutex_unlock(&static_key_mutex);
3437− return ret;
3336+ hdr = register_sysctl_table(sysctl_base_table);
3337+ kmemleak_not_leak(hdr);
3338+ return 0;
34383339 }
3439−#endif
3340+#endif /* CONFIG_SYSCTL */
34403341 /*
34413342 * No sense putting this after each symbol definition, twice,
34423343 * exception granted :-)
kernel/time/timer.c+1−2
@@ -249,8 +249,7 @@ void timers_update_nohz(void) · 248 unmodified lines
249249 }
250250
251251 int timer_migration_handler(struct ctl_table *table, int write,
252− void __user *buffer, size_t *lenp,
253− loff_t *ppos)
252+ void *buffer, size_t *lenp, loff_t *ppos)
254253 {
255254 int ret;
256255
kernel/trace/bpf_trace.c+235−6
@@ -147,7 +147,7 @@ BPF_CALL_3(bpf_probe_read_user, void *, dst, u32, size, · 146 unmodified lines
147147 return ret;
148148 }
149149
150−static const struct bpf_func_proto bpf_probe_read_user_proto = {
150+const struct bpf_func_proto bpf_probe_read_user_proto = {
151151 .func = bpf_probe_read_user,
152152 .gpl_only = true,
153153 .ret_type = RET_INTEGER,
@@ -167,7 +167,7 @@ BPF_CALL_3(bpf_probe_read_user_str, void *, dst, u32, size, · 13 unmodified lines
167167 return ret;
168168 }
169169
170−static const struct bpf_func_proto bpf_probe_read_user_str_proto = {
170+const struct bpf_func_proto bpf_probe_read_user_str_proto = {
171171 .func = bpf_probe_read_user_str,
172172 .gpl_only = true,
173173 .ret_type = RET_INTEGER,
@@ -198,7 +198,7 @@ BPF_CALL_3(bpf_probe_read_kernel, void *, dst, u32, size, · 24 unmodified lines
198198 return bpf_probe_read_kernel_common(dst, size, unsafe_ptr, false);
199199 }
200200
201−static const struct bpf_func_proto bpf_probe_read_kernel_proto = {
201+const struct bpf_func_proto bpf_probe_read_kernel_proto = {
202202 .func = bpf_probe_read_kernel,
203203 .gpl_only = true,
204204 .ret_type = RET_INTEGER,
@@ -253,7 +253,7 @@ BPF_CALL_3(bpf_probe_read_kernel_str, void *, dst, u32, size, · 48 unmodified lines
253253 return bpf_probe_read_kernel_str_common(dst, size, unsafe_ptr, false);
254254 }
255255
256−static const struct bpf_func_proto bpf_probe_read_kernel_str_proto = {
256+const struct bpf_func_proto bpf_probe_read_kernel_str_proto = {
257257 .func = bpf_probe_read_kernel_str,
258258 .gpl_only = true,
259259 .ret_type = RET_INTEGER,
@@ -315,6 +315,9 @@ static const struct bpf_func_proto bpf_probe_write_user_proto = { · 55 unmodified lines
315315
316316 static const struct bpf_func_proto *bpf_get_probe_write_proto(void)
317317 {
318+ if (!capable(CAP_SYS_ADMIN))
319+ return NULL;
320+
318321 pr_warn_ratelimited("%s[%d] is installing a program with bpf_probe_write_user helper that may corrupt user memory!",
319322 current->comm, task_pid_nr(current));
320323
@@ -487,6 +490,212 @@ const struct bpf_func_proto *bpf_get_trace_printk_proto(void) · 166 unmodified lines
487490 return &bpf_trace_printk_proto;
488491 }
489492
493+#define MAX_SEQ_PRINTF_VARARGS 12
494+#define MAX_SEQ_PRINTF_MAX_MEMCPY 6
495+#define MAX_SEQ_PRINTF_STR_LEN 128
496+
497+struct bpf_seq_printf_buf {
498+ char buf[MAX_SEQ_PRINTF_MAX_MEMCPY][MAX_SEQ_PRINTF_STR_LEN];
499+};
500+static DEFINE_PER_CPU(struct bpf_seq_printf_buf, bpf_seq_printf_buf);
501+static DEFINE_PER_CPU(int, bpf_seq_printf_buf_used);
502+
503+BPF_CALL_5(bpf_seq_printf, struct seq_file *, m, char *, fmt, u32, fmt_size,
504+ const void *, data, u32, data_len)
505+{
506+ int err = -EINVAL, fmt_cnt = 0, memcpy_cnt = 0;
507+ int i, buf_used, copy_size, num_args;
508+ u64 params[MAX_SEQ_PRINTF_VARARGS];
509+ struct bpf_seq_printf_buf *bufs;
510+ const u64 *args = data;
511+
512+ buf_used = this_cpu_inc_return(bpf_seq_printf_buf_used);
513+ if (WARN_ON_ONCE(buf_used > 1)) {
514+ err = -EBUSY;
515+ goto out;
516+ }
517+
518+ bufs = this_cpu_ptr(&bpf_seq_printf_buf);
519+
520+ /*
521+ * bpf_check()->check_func_arg()->check_stack_boundary()
522+ * guarantees that fmt points to bpf program stack,
523+ * fmt_size bytes of it were initialized and fmt_size > 0
524+ */
525+ if (fmt[--fmt_size] != 0)
526+ goto out;
527+
528+ if (data_len & 7)
529+ goto out;
530+
531+ for (i = 0; i < fmt_size; i++) {
532+ if (fmt[i] == '%') {
533+ if (fmt[i + 1] == '%')
534+ i++;
535+ else if (!data || !data_len)
536+ goto out;
537+ }
538+ }
539+
540+ num_args = data_len / 8;
541+
542+ /* check format string for allowed specifiers */
543+ for (i = 0; i < fmt_size; i++) {
544+ /* only printable ascii for now. */
545+ if ((!isprint(fmt[i]) && !isspace(fmt[i])) || !isascii(fmt[i])) {
546+ err = -EINVAL;
547+ goto out;
548+ }
549+
550+ if (fmt[i] != '%')
551+ continue;
552+
553+ if (fmt[i + 1] == '%') {
554+ i++;
555+ continue;
556+ }
557+
558+ if (fmt_cnt >= MAX_SEQ_PRINTF_VARARGS) {
559+ err = -E2BIG;
560+ goto out;
561+ }
562+
563+ if (fmt_cnt >= num_args) {
564+ err = -EINVAL;
565+ goto out;
566+ }
567+
568+ /* fmt[i] != 0 && fmt[last] == 0, so we can access fmt[i + 1] */
569+ i++;
570+
571+ /* skip optional "[0 +-][num]" width formating field */
572+ while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
573+ fmt[i] == ' ')
574+ i++;
575+ if (fmt[i] >= '1' && fmt[i] <= '9') {
576+ i++;
577+ while (fmt[i] >= '0' && fmt[i] <= '9')
578+ i++;
579+ }
580+
581+ if (fmt[i] == 's') {
582+ /* try our best to copy */
583+ if (memcpy_cnt >= MAX_SEQ_PRINTF_MAX_MEMCPY) {
584+ err = -E2BIG;
585+ goto out;
586+ }
587+
588+ err = strncpy_from_unsafe_strict(bufs->buf[memcpy_cnt],
589+ (void *) (long) args[fmt_cnt],
590+ MAX_SEQ_PRINTF_STR_LEN);
591+ if (err < 0)
592+ bufs->buf[memcpy_cnt][0] = '\0';
593+ params[fmt_cnt] = (u64)(long)bufs->buf[memcpy_cnt];
594+
595+ fmt_cnt++;
596+ memcpy_cnt++;
597+ continue;
598+ }
599+
600+ if (fmt[i] == 'p') {
601+ if (fmt[i + 1] == 0 ||
602+ fmt[i + 1] == 'K' ||
603+ fmt[i + 1] == 'x') {
604+ /* just kernel pointers */
605+ params[fmt_cnt] = args[fmt_cnt];
606+ fmt_cnt++;
607+ continue;
608+ }
609+
610+ /* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
611+ if (fmt[i + 1] != 'i' && fmt[i + 1] != 'I') {
612+ err = -EINVAL;
613+ goto out;
614+ }
615+ if (fmt[i + 2] != '4' && fmt[i + 2] != '6') {
616+ err = -EINVAL;
617+ goto out;
618+ }
619+
620+ if (memcpy_cnt >= MAX_SEQ_PRINTF_MAX_MEMCPY) {
621+ err = -E2BIG;
622+ goto out;
623+ }
624+
625+
626+ copy_size = (fmt[i + 2] == '4') ? 4 : 16;
627+
628+ err = probe_kernel_read(bufs->buf[memcpy_cnt],
629+ (void *) (long) args[fmt_cnt],
630+ copy_size);
631+ if (err < 0)
632+ memset(bufs->buf[memcpy_cnt], 0, copy_size);
633+ params[fmt_cnt] = (u64)(long)bufs->buf[memcpy_cnt];
634+
635+ i += 2;
636+ fmt_cnt++;
637+ memcpy_cnt++;
638+ continue;
639+ }
640+
641+ if (fmt[i] == 'l') {
642+ i++;
643+ if (fmt[i] == 'l')
644+ i++;
645+ }
646+
647+ if (fmt[i] != 'i' && fmt[i] != 'd' &&
648+ fmt[i] != 'u' && fmt[i] != 'x') {
649+ err = -EINVAL;
650+ goto out;
651+ }
652+
653+ params[fmt_cnt] = args[fmt_cnt];
654+ fmt_cnt++;
655+ }
656+
657+ /* Maximumly we can have MAX_SEQ_PRINTF_VARARGS parameter, just give
658+ * all of them to seq_printf().
659+ */
660+ seq_printf(m, fmt, params[0], params[1], params[2], params[3],
661+ params[4], params[5], params[6], params[7], params[8],
662+ params[9], params[10], params[11]);
663+
664+ err = seq_has_overflowed(m) ? -EOVERFLOW : 0;
665+out:
666+ this_cpu_dec(bpf_seq_printf_buf_used);
667+ return err;
668+}
669+
670+static int bpf_seq_printf_btf_ids[5];
671+static const struct bpf_func_proto bpf_seq_printf_proto = {
672+ .func = bpf_seq_printf,
673+ .gpl_only = true,
674+ .ret_type = RET_INTEGER,
675+ .arg1_type = ARG_PTR_TO_BTF_ID,
676+ .arg2_type = ARG_PTR_TO_MEM,
677+ .arg3_type = ARG_CONST_SIZE,
678+ .arg4_type = ARG_PTR_TO_MEM_OR_NULL,
679+ .arg5_type = ARG_CONST_SIZE_OR_ZERO,
680+ .btf_id = bpf_seq_printf_btf_ids,
681+};
682+
683+BPF_CALL_3(bpf_seq_write, struct seq_file *, m, const void *, data, u32, len)
684+{
685+ return seq_write(m, data, len) ? -EOVERFLOW : 0;
686+}
687+
688+static int bpf_seq_write_btf_ids[5];
689+static const struct bpf_func_proto bpf_seq_write_proto = {
690+ .func = bpf_seq_write,
691+ .gpl_only = true,
692+ .ret_type = RET_INTEGER,
693+ .arg1_type = ARG_PTR_TO_BTF_ID,
694+ .arg2_type = ARG_PTR_TO_MEM,
695+ .arg3_type = ARG_CONST_SIZE_OR_ZERO,
696+ .btf_id = bpf_seq_write_btf_ids,
697+};
698+
490699 static __always_inline int
491700 get_map_perf_counter(struct bpf_map *map, u64 flags,
492701 u64 *value, u64 *enabled, u64 *running)
@@ -698,7 +907,7 @@ BPF_CALL_0(bpf_get_current_task) · 205 unmodified lines
698907 return (long) current;
699908 }
700909
701−static const struct bpf_func_proto bpf_get_current_task_proto = {
910+const struct bpf_func_proto bpf_get_current_task_proto = {
702911 .func = bpf_get_current_task,
703912 .gpl_only = true,
704913 .ret_type = RET_INTEGER,
@@ -827,6 +1036,8 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 122 unmodified lines
8271036 return &bpf_map_peek_elem_proto;
8281037 case BPF_FUNC_ktime_get_ns:
8291038 return &bpf_ktime_get_ns_proto;
1039+ case BPF_FUNC_ktime_get_boot_ns:
1040+ return &bpf_ktime_get_boot_ns_proto;
8301041 case BPF_FUNC_tail_call:
8311042 return &bpf_tail_call_proto;
8321043 case BPF_FUNC_get_current_pid_tgid:
@@ -877,6 +1088,16 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 44 unmodified lines
8771088 return &bpf_perf_event_read_value_proto;
8781089 case BPF_FUNC_get_ns_current_pid_tgid:
8791090 return &bpf_get_ns_current_pid_tgid_proto;
1091+ case BPF_FUNC_ringbuf_output:
1092+ return &bpf_ringbuf_output_proto;
1093+ case BPF_FUNC_ringbuf_reserve:
1094+ return &bpf_ringbuf_reserve_proto;
1095+ case BPF_FUNC_ringbuf_submit:
1096+ return &bpf_ringbuf_submit_proto;
1097+ case BPF_FUNC_ringbuf_discard:
1098+ return &bpf_ringbuf_discard_proto;
1099+ case BPF_FUNC_ringbuf_query:
1100+ return &bpf_ringbuf_query_proto;
8801101 default:
8811102 return NULL;
8821103 }
@@ -1246,7 +1467,7 @@ raw_tp_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 363 unmodified lines
12461467 }
12471468 }
12481469
1249−static const struct bpf_func_proto *
1470+const struct bpf_func_proto *
12501471 tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
12511472 {
12521473 switch (func_id) {
@@ -1256,6 +1477,14 @@ tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 3 unmodified lines
12561477 case BPF_FUNC_xdp_output:
12571478 return &bpf_xdp_output_proto;
12581479 #endif
1480+ case BPF_FUNC_seq_printf:
1481+ return prog->expected_attach_type == BPF_TRACE_ITER ?
1482+ &bpf_seq_printf_proto :
1483+ NULL;
1484+ case BPF_FUNC_seq_write:
1485+ return prog->expected_attach_type == BPF_TRACE_ITER ?
1486+ &bpf_seq_write_proto :
1487+ NULL;
12591488 default:
12601489 return raw_tp_prog_func_proto(func_id, prog);
12611490 }
kernel/trace/trace.c+1−1
@@ -2662,7 +2662,7 @@ static void output_printk(struct trace_event_buffer *fbuffer) · 2661 unmodified lines
26622662 }
26632663
26642664 int tracepoint_printk_sysctl(struct ctl_table *table, int write,
2665− void __user *buffer, size_t *lenp,
2665+ void *buffer, size_t *lenp,
26662666 loff_t *ppos)
26672667 {
26682668 int save_tracepoint_printk;
kernel/umh.c+1−1
@@ -641,7 +641,7 @@ int call_usermodehelper(const char *path, char **argv, char **envp, int wait) · 640 unmodified lines
641641 EXPORT_SYMBOL(call_usermodehelper);
642642
643643 static int proc_cap_handler(struct ctl_table *table, int write,
644− void __user *buffer, size_t *lenp, loff_t *ppos)
644+ void *buffer, size_t *lenp, loff_t *ppos)
645645 {
646646 struct ctl_table t;
647647 unsigned long cap_array[_KERNEL_CAPABILITY_U32S];
kernel/utsname_sysctl.c+1−1
@@ -30,7 +30,7 @@ static void *get_uts(struct ctl_table *table) · 29 unmodified lines
3030 * to observe. Should this be in kernel/sys.c ????
3131 */
3232 static int proc_do_uts_string(struct ctl_table *table, int write,
33− void __user *buffer, size_t *lenp, loff_t *ppos)
33+ void *buffer, size_t *lenp, loff_t *ppos)
3434 {
3535 struct ctl_table uts_table;
3636 int r;
kernel/watchdog.c+6−6
@@ -661,7 +661,7 @@ static void proc_watchdog_update(void) · 660 unmodified lines
661661 * proc_soft_watchdog | soft_watchdog_user_enabled | SOFT_WATCHDOG_ENABLED
662662 */
663663 static int proc_watchdog_common(int which, struct ctl_table *table, int write,
664− void __user *buffer, size_t *lenp, loff_t *ppos)
664+ void *buffer, size_t *lenp, loff_t *ppos)
665665 {
666666 int err, old, *param = table->data;
667667
@@ -688,7 +688,7 @@ static int proc_watchdog_common(int which, struct ctl_table *table, int write, · 20 unmodified lines
688688 * /proc/sys/kernel/watchdog
689689 */
690690 int proc_watchdog(struct ctl_table *table, int write,
691− void __user *buffer, size_t *lenp, loff_t *ppos)
691+ void *buffer, size_t *lenp, loff_t *ppos)
692692 {
693693 return proc_watchdog_common(NMI_WATCHDOG_ENABLED|SOFT_WATCHDOG_ENABLED,
694694 table, write, buffer, lenp, ppos);
@@ -698,7 +698,7 @@ int proc_watchdog(struct ctl_table *table, int write, · 3 unmodified lines
698698 * /proc/sys/kernel/nmi_watchdog
699699 */
700700 int proc_nmi_watchdog(struct ctl_table *table, int write,
701− void __user *buffer, size_t *lenp, loff_t *ppos)
701+ void *buffer, size_t *lenp, loff_t *ppos)
702702 {
703703 if (!nmi_watchdog_available && write)
704704 return -ENOTSUPP;
@@ -710,7 +710,7 @@ int proc_nmi_watchdog(struct ctl_table *table, int write, · 5 unmodified lines
710710 * /proc/sys/kernel/soft_watchdog
711711 */
712712 int proc_soft_watchdog(struct ctl_table *table, int write,
713− void __user *buffer, size_t *lenp, loff_t *ppos)
713+ void *buffer, size_t *lenp, loff_t *ppos)
714714 {
715715 return proc_watchdog_common(SOFT_WATCHDOG_ENABLED,
716716 table, write, buffer, lenp, ppos);
@@ -720,7 +720,7 @@ int proc_soft_watchdog(struct ctl_table *table, int write, · 3 unmodified lines
720720 * /proc/sys/kernel/watchdog_thresh
721721 */
722722 int proc_watchdog_thresh(struct ctl_table *table, int write,
723− void __user *buffer, size_t *lenp, loff_t *ppos)
723+ void *buffer, size_t *lenp, loff_t *ppos)
724724 {
725725 int err, old;
726726
@@ -743,7 +743,7 @@ int proc_watchdog_thresh(struct ctl_table *table, int write, · 16 unmodified lines
743743 * been brought online, if desired.
744744 */
745745 int proc_watchdog_cpumask(struct ctl_table *table, int write,
746− void __user *buffer, size_t *lenp, loff_t *ppos)
746+ void *buffer, size_t *lenp, loff_t *ppos)
747747 {
748748 int err;
749749

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t