To access per-task data, BPF programs usually creates a hash table with pid as the key. This is not ideal because: 1. The user need to estimate the proper size of the hash table, which may be inaccurate; 2. Big hash tables are slow; 3. To clean up the data properly during task terminations, the user need to write extra logic. Task local storage overcomes these issues and offers a better option for these per-task data. Task local storage is only available to BPF_LSM. Now enable it for tracing programs. Unlike LSM programs, tracing programs can be called in IRQ contexts. Helpers that access task local storage are updated to use raw_spin_lock_irqsave() instead of raw_spin_lock_bh(). Tracing programs can attach to functions on the task free path, e.g. exit_creds(). To avoid allocating task local storage after bpf_task_storage_free(). bpf_task_storage_get() is updated to not allocate new storage when the task is not refcounted (task->usage == 0).
Showing the changes to kernel/fork.c (1 of 10 files). Show all
Unified · Split
| @@ -96,6 +96,7 @@ · 95 unmodified lines | |||
| 96 | #include <linux/kasan.h> | 96 | #include <linux/kasan.h> |
| 97 | #include <linux/scs.h> | 97 | #include <linux/scs.h> |
| 98 | #include <linux/io_uring.h> | 98 | #include <linux/io_uring.h> |
| 99 | #include <linux/bpf.h> | ||
| 99 | 100 | ||
| 100 | #include <asm/pgalloc.h> | 101 | #include <asm/pgalloc.h> |
| 101 | #include <linux/uaccess.h> | 102 | #include <linux/uaccess.h> |
| @@ -734,6 +735,7 @@ void __put_task_struct(struct task_struct *tsk) · 632 unmodified lines | |||
| 734 | cgroup_free(tsk); | 735 | cgroup_free(tsk); |
| 735 | task_numa_free(tsk, true); | 736 | task_numa_free(tsk, true); |
| 736 | security_task_free(tsk); | 737 | security_task_free(tsk); |
| 738 | bpf_task_storage_free(tsk); | ||
| 737 | exit_creds(tsk); | 739 | exit_creds(tsk); |
| 738 | delayacct_tsk_free(tsk); | 740 | delayacct_tsk_free(tsk); |
| 739 | put_signal_struct(tsk->signal); | 741 | put_signal_struct(tsk->signal); |
| @@ -2062,6 +2064,9 @@ static __latent_entropy struct task_struct *copy_process( · 1322 unmodified lines | |||
| 2062 | p->sequential_io = 0; | 2064 | p->sequential_io = 0; |
| 2063 | p->sequential_io_avg = 0; | 2065 | p->sequential_io_avg = 0; |
| 2064 | #endif | 2066 | #endif |
| 2067 | #ifdef CONFIG_BPF_SYSCALL | ||
| 2068 | RCU_INIT_POINTER(p->bpf_storage, NULL); | ||
| 2069 | #endif | ||
| 2065 | 2070 | ||
| 2066 | /* Perform scheduler related setup. Assign this task to a CPU. */ | 2071 | /* Perform scheduler related setup. Assign this task to a CPU. */ |
| 2067 | retval = sched_fork(clone_flags, p); | 2072 | retval = sched_fork(clone_flags, p); |