bpf: Introduce load-acquire and store-release instructions

Introduce BPF instructions with load-acquire and store-release semantics, as discussed in [1]. Define 2 new flags: #define BPF_LOAD_ACQ 0x100 #define BPF_STORE_REL 0x110 A "load-acquire" is a BPF_STX | BPF_ATOMIC instruction with the 'imm' field set to BPF_LOAD_ACQ (0x100). Similarly, a "store-release" is a BPF_STX | BPF_ATOMIC instruction with the 'imm' field set to BPF_STORE_REL (0x110). Unlike existing atomic read-modify-write operations that only support BPF_W (32-bit) and BPF_DW (64-bit) size modifiers, load-acquires and store-releases also support BPF_B (8-bit) and BPF_H (16-bit). As an exception, however, 64-bit load-acquires/store-releases are not supported on 32-bit architectures (to fix a build error reported by the kernel test robot). An 8- or 16-bit load-acquire zero-extends the value before writing it to a 32-bit register, just like ARM64 instruction LDARH and friends. Similar to existing atomic read-modify-write operations, misaligned load-acquires/store-releases are not allowed (even if BPF_F_ANY_ALIGNMENT is set). As an example, consider the following 64-bit load-acquire BPF instruction (assuming little-endian): db 10 00 00 00 01 00 00 r0 = load_acquire((u64 *)(r1 + 0x0)) opcode (0xdb): BPF_ATOMIC | BPF_DW | BPF_STX imm (0x00000100): BPF_LOAD_ACQ Similarly, a 16-bit BPF store-release: cb 21 00 00 10 01 00 00 store_release((u16 *)(r1 + 0x0), w2) opcode (0xcb): BPF_ATOMIC | BPF_H | BPF_STX imm (0x00000110): BPF_STORE_REL In arch/{arm64,s390,x86}/net/bpf_jit_comp.c, have bpf_jit_supports_insn(..., /*in_arena=*/true) return false for the new instructions, until the corresponding JIT compiler supports them in arena. [1] https://lore.kernel.org/all/20240729183246.4110549-1-yepeilin@google.com/

Peilin Ye committed 1 year ago

Commit 88044231 parent 3a6fa5710 files · +166 −13Browse files

Showing the changes to include/uapi (1 of 10 files). Show all

Unified · Split

include/uapi/linux/bpf.h+3−0

@@ -51,6 +51,9 @@ · 50 unmodified lines
51	#define BPF_XCHG (0xe0 \| BPF_FETCH) /* atomic exchange */	51	#define BPF_XCHG (0xe0 \| BPF_FETCH) /* atomic exchange */
52	#define BPF_CMPXCHG (0xf0 \| BPF_FETCH) /* atomic compare-and-write */	52	#define BPF_CMPXCHG (0xf0 \| BPF_FETCH) /* atomic compare-and-write */
53		53
		54	#define BPF_LOAD_ACQ 0x100 /* load-acquire */
		55	#define BPF_STORE_REL 0x110 /* store-release */
		56
54	enum bpf_cond_pseudo_jmp {	57	enum bpf_cond_pseudo_jmp {
55	BPF_MAY_GOTO = 0,	58	BPF_MAY_GOTO = 0,
56	};	59	};