Merge tag 'x86-fpu-2021-11-01' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull x86 fpu updates from Thomas Gleixner: - Cleanup of extable fixup handling to be more robust, which in turn allows to make the FPU exception fixups more robust as well. - Change the return code for signal frame related failures from explicit error codes to a boolean fail/success as that's all what the calling code evaluates. - A large refactoring of the FPU code to prepare for adding AMX support: - Distangle the public header maze and remove especially the misnomed kitchen sink internal.h which is despite it's name included all over the place. - Add a proper abstraction for the register buffer storage (struct fpstate) which allows to dynamically size the buffer at runtime by flipping the pointer to the buffer container from the default container which is embedded in task_struct::tread::fpu to a dynamically allocated container with a larger register buffer. - Convert the code over to the new fpstate mechanism. - Consolidate the KVM FPU handling by moving the FPU related code into the FPU core which removes the number of exports and avoids adding even more export when AMX has to be supported in KVM. This also removes duplicated code which was of course unnecessary different and incomplete in the KVM copy. - Simplify the KVM FPU buffer handling by utilizing the new fpstate container and just switching the buffer pointer from the user space buffer to the KVM guest buffer when entering vcpu_run() and flipping it back when leaving the function. This cuts the memory requirements of a vCPU for FPU buffers in half and avoids pointless memory copy operations. This also solves the so far unresolved problem of adding AMX support because the current FPU buffer handling of KVM inflicted a circular dependency between adding AMX support to the core and to KVM. With the new scheme of switching fpstate AMX support can be added to the core code without affecting KVM. - Replace various variables with proper data structures so the extra information required for adding dynamically enabled FPU features (AMX) can be added in one place - Add AMX (Advanced Matrix eXtensions) support (finally): AMX is a large XSTATE component which is going to be available with Saphire Rapids XEON CPUs. The feature comes with an extra MSR (MSR_XFD) which allows to trap the (first) use of an AMX related instruction, which has two benefits: 1) It allows the kernel to control access to the feature 2) It allows the kernel to dynamically allocate the large register state buffer instead of burdening every task with the the extra 8K or larger state storage. It would have been great to gain this kind of control already with AVX512. The support comes with the following infrastructure components: 1) arch_prctl() to - read the supported features (equivalent to XGETBV(0)) - read the permitted features for a task - request permission for a dynamically enabled feature Permission is granted per process, inherited on fork() and cleared on exec(). The permission policy of the kernel is restricted to sigaltstack size validation, but the syscall obviously allows further restrictions via seccomp etc. 2) A stronger sigaltstack size validation for sys_sigaltstack(2) which takes granted permissions and the potentially resulting larger signal frame into account. This mechanism can also be used to enforce factual sigaltstack validation independent of dynamic features to help with finding potential victims of the 2K sigaltstack size constant which is broken since AVX512 support was added. 3) Exception handling for #NM traps to catch first use of a extended feature via a new cause MSR. If the exception was caused by the use of such a feature, the handler checks permission for that feature. If permission has not been granted, the handler sends a SIGILL like the #UD handler would do if the feature would have been disabled in XCR0. If permission has been granted, then a new fpstate which fits the larger buffer requirement is allocated. In the unlikely case that this allocation fails, the handler sends SIGSEGV to the task. That's not elegant, but unavoidable as the other discussed options of preallocation or full per task permissions come with their own set of horrors for kernel and/or userspace. So this is the lesser of the evils and SIGSEGV caused by unexpected memory allocation failures is not a fundamentally new concept either. When allocation succeeds, the fpstate properties are filled in to reflect the extended feature set and the resulting sizes, the fpu::fpstate pointer is updated accordingly and the trap is disarmed for this task permanently. 4) Enumeration and size calculations 5) Trap switching via MSR_XFD The XFD (eXtended Feature Disable) MSR is context switched with the same life time rules as the FPU register state itself. The mechanism is keyed off with a static key which is default disabled so !AMX equipped CPUs have zero overhead. On AMX enabled CPUs the overhead is limited by comparing the tasks XFD value with a per CPU shadow variable to avoid redundant MSR writes. In case of switching from a AMX using task to a non AMX using task or vice versa, the extra MSR write is obviously inevitable. All other places which need to be aware of the variable feature sets and resulting variable sizes are not affected at all because they retrieve the information (feature set, sizes) unconditonally from the fpstate properties. 6) Enable the new AMX states Note, this is relatively new code despite the fact that AMX support is in the works for more than a year now. The big refactoring of the FPU code, which allowed to do a proper integration has been started exactly 3 weeks ago. Refactoring of the existing FPU code and of the original AMX patches took a week and has been subject to extensive review and testing. The only fallout which has not been caught in review and testing right away was restricted to AMX enabled systems, which is completely irrelevant for anyone outside Intel and their early access program. There might be dragons lurking as usual, but so far the fine grained refactoring has held up and eventual yet undetected fallout is bisectable and should be easily addressable before the 5.16 release. Famous last words... Many thanks to Chang Bae and Dave Hansen for working hard on this and also to the various test teams at Intel who reserved extra capacity to follow the rapid development of this closely which provides the confidence level required to offer this rather large update for inclusion into 5.16-rc1 * tag 'x86-fpu-2021-11-01' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (110 commits) Documentation/x86: Add documentation for using dynamic XSTATE features x86/fpu: Include vmalloc.h for vzalloc() selftests/x86/amx: Add context switch test selftests/x86/amx: Add test cases for AMX state management x86/fpu/amx: Enable the AMX feature in 64-bit mode x86/fpu: Add XFD handling for dynamic states x86/fpu: Calculate the default sizes independently x86/fpu/amx: Define AMX state components and have it used for boot-time checks x86/fpu/xstate: Prepare XSAVE feature table for gaps in state component numbers x86/fpu/xstate: Add fpstate_realloc()/free() x86/fpu/xstate: Add XFD #NM handler x86/fpu: Update XFD state where required x86/fpu: Add sanity checks for XFD x86/fpu: Add XFD state to fpstate x86/msr-index: Add MSRs for XFD x86/cpufeatures: Add eXtended Feature Disabling (XFD) feature bit x86/fpu: Reset permission and fpstate on exec() x86/fpu: Prepare fpu_clone() for dynamically enabled features x86/fpu/signal: Prepare for variable sigframe length x86/signal: Use fpu::__state_user_size for sigalt stack validation ...

Linus Torvalds committed 5 years ago

Commit 8cb1ae12 parents 7d20dd3 d7a959067 files · +3458 −1575Browse files

Unified · Split

67 files

Documentation/admin-guide/kernel-parameters.txt +9 −0
Documentation/x86/index.rst +1 −0
Documentation/x86/xstate.rst +65 −0
arch/Kconfig +3 −0
arch/x86/Kconfig +17 −0
arch/x86/events/perf_event.h +1 −0
arch/x86/ia32/ia32_signal.c +7 −8
arch/x86/include/asm/asm.h +20 −30
arch/x86/include/asm/cpufeatures.h +2 −0
arch/x86/include/asm/extable.h +28 −16
arch/x86/include/asm/extable_fixup_types.h +22 −0
arch/x86/include/asm/fpu/api.h +56 −2
arch/x86/include/asm/fpu/internal.h +0 −540
arch/x86/include/asm/fpu/sched.h +68 −0
arch/x86/include/asm/fpu/signal.h +12 −1
arch/x86/include/asm/fpu/types.h +206 −8
arch/x86/include/asm/fpu/xcr.h +0 −11
arch/x86/include/asm/fpu/xstate.h +30 −60
arch/x86/include/asm/kvm_host.h +2 −5
arch/x86/include/asm/msr-index.h +2 −0
arch/x86/include/asm/msr.h +2 −2
arch/x86/include/asm/pkru.h +1 −1
arch/x86/include/asm/processor.h +3 −6
arch/x86/include/asm/proto.h +1 −1
arch/x86/include/asm/segment.h +1 −1
arch/x86/include/asm/trace/fpu.h +2 −2
arch/x86/include/uapi/asm/prctl.h +4 −0
arch/x86/kernel/cpu/bugs.c +1 −1
arch/x86/kernel/cpu/common.c +1 −1
arch/x86/kernel/cpu/cpuid-deps.c +2 −0
arch/x86/kernel/cpu/mce/core.c +11 −29
arch/x86/kernel/cpu/mce/internal.h +2 −12
arch/x86/kernel/cpu/mce/severity.c +11 −11
arch/x86/kernel/fpu/bugs.c +1 −1
arch/x86/kernel/fpu/context.h +83 −0
arch/x86/kernel/fpu/core.c +315 −77
arch/x86/kernel/fpu/init.c +34 −42
arch/x86/kernel/fpu/internal.h +28 −0
arch/x86/kernel/fpu/legacy.h +115 −0
arch/x86/kernel/fpu/regset.c +20 −16
arch/x86/kernel/fpu/signal.c +145 −140
arch/x86/kernel/fpu/xstate.c +712 −186
arch/x86/kernel/fpu/xstate.h +278 −0
arch/x86/kernel/process.c +23 −4
arch/x86/kernel/process_32.c +2 −3
arch/x86/kernel/process_64.c +2 −3
arch/x86/kernel/ptrace.c +1 −1
arch/x86/kernel/sev.c +1 −1
arch/x86/kernel/signal.c +71 −12
arch/x86/kernel/smpboot.c +1 −1
arch/x86/kernel/traps.c +39 −1
arch/x86/kvm/svm/sev.c +1 −1
arch/x86/kvm/svm/svm.c +4 −3
arch/x86/kvm/vmx/vmx.c +1 −1
arch/x86/kvm/x86.c +31 −227
arch/x86/lib/copy_mc_64.S +4 −4
arch/x86/math-emu/fpu_aux.c +1 −1
arch/x86/math-emu/fpu_entry.c +3 −3
arch/x86/math-emu/fpu_system.h +1 −1
arch/x86/mm/extable.c +56 −79
arch/x86/net/bpf_jit_comp.c +2 −9
arch/x86/power/cpu.c +1 −1
include/linux/signal.h +6 −0
kernel/signal.c +29 −6
scripts/sorttable.c +2 −2
tools/testing/selftests/x86/Makefile +1 −1
tools/testing/selftests/x86/amx.c +851 −0

Documentation/admin-guide/kernel-parameters.txt+9−0

@@ -5497,6 +5497,15 @@ · 5496 unmodified lines
5497	5497	stifb= [HW]
5498	5498	Format: bpp:<bpp1>[:<bpp2>[:<bpp3>...]]
5499	5499
	5500	+ strict_sas_size=
	5501	+ [X86]
	5502	+ Format: <bool>
	5503	+ Enable or disable strict sigaltstack size checks
	5504	+ against the required signal frame size which
	5505	+ depends on the supported FPU features. This can
	5506	+ be used to filter out binaries which have
	5507	+ not yet been made aware of AT_MINSIGSTKSZ.
	5508	+
5500	5509	sunrpc.min_resvport=
5501	5510	sunrpc.max_resvport=
5502	5511	[NFS,SUNRPC]

Documentation/x86/index.rst+1−0

@@ -37,3 +37,4 @@ x86-specific Documentation · 36 unmodified lines
37	37	sgx
38	38	features
39	39	elf_auxvec
	40	+ xstate

Documentation/x86/xstate.rst+65−0

@@ -0,0 +1,65 @@
	1	+Using XSTATE features in user space applications
	2	+================================================
	3	+
	4	+The x86 architecture supports floating-point extensions which are
	5	+enumerated via CPUID. Applications consult CPUID and use XGETBV to
	6	+evaluate which features have been enabled by the kernel XCR0.
	7	+
	8	+Up to AVX-512 and PKRU states, these features are automatically enabled by
	9	+the kernel if available. Features like AMX TILE_DATA (XSTATE component 18)
	10	+are enabled by XCR0 as well, but the first use of related instruction is
	11	+trapped by the kernel because by default the required large XSTATE buffers
	12	+are not allocated automatically.
	13	+
	14	+Using dynamically enabled XSTATE features in user space applications
	15	+--------------------------------------------------------------------
	16	+
	17	+The kernel provides an arch_prctl(2) based mechanism for applications to
	18	+request the usage of such features. The arch_prctl(2) options related to
	19	+this are:
	20	+
	21	+-ARCH_GET_XCOMP_SUPP
	22	+
	23	+ arch_prctl(ARCH_GET_XCOMP_SUPP, &features);
	24	+
	25	+ ARCH_GET_XCOMP_SUPP stores the supported features in userspace storage of
	26	+ type uint64_t. The second argument is a pointer to that storage.
	27	+
	28	+-ARCH_GET_XCOMP_PERM
	29	+
	30	+ arch_prctl(ARCH_GET_XCOMP_PERM, &features);
	31	+
	32	+ ARCH_GET_XCOMP_PERM stores the features for which the userspace process
	33	+ has permission in userspace storage of type uint64_t. The second argument
	34	+ is a pointer to that storage.
	35	+
	36	+-ARCH_REQ_XCOMP_PERM
	37	+
	38	+ arch_prctl(ARCH_REQ_XCOMP_PERM, feature_nr);
	39	+
	40	+ ARCH_REQ_XCOMP_PERM allows to request permission for a dynamically enabled
	41	+ feature or a feature set. A feature set can be mapped to a facility, e.g.
	42	+ AMX, and can require one or more XSTATE components to be enabled.
	43	+
	44	+ The feature argument is the number of the highest XSTATE component which
	45	+ is required for a facility to work.
	46	+
	47	+When requesting permission for a feature, the kernel checks the
	48	+availability. The kernel ensures that sigaltstacks in the process's tasks
	49	+are large enough to accommodate the resulting large signal frame. It
	50	+enforces this both during ARCH_REQ_XCOMP_SUPP and during any subsequent
	51	+sigaltstack(2) calls. If an installed sigaltstack is smaller than the
	52	+resulting sigframe size, ARCH_REQ_XCOMP_SUPP results in -ENOSUPP. Also,
	53	+sigaltstack(2) results in -ENOMEM if the requested altstack is too small
	54	+for the permitted features.
	55	+
	56	+Permission, when granted, is valid per process. Permissions are inherited
	57	+on fork(2) and cleared on exec(3).
	58	+
	59	+The first use of an instruction related to a dynamically enabled feature is
	60	+trapped by the kernel. The trap handler checks whether the process has
	61	+permission to use the feature. If the process has no permission then the
	62	+kernel sends SIGILL to the application. If the process has permission then
	63	+the handler allocates a larger xstate buffer for the task so the large
	64	+state can be context switched. In the unlikely cases that the allocation
	65	+fails, the kernel sends SIGSEGV.

arch/Kconfig+3−0

@@ -1288,6 +1288,9 @@ config ARCH_HAS_ELFCORE_COMPAT · 1287 unmodified lines
1288	1288	config ARCH_HAS_PARANOID_L1D_FLUSH
1289	1289	bool
1290	1290
	1291	+config DYNAMIC_SIGFRAME
	1292	+ bool
	1293	+
1291	1294	source "kernel/gcov/Kconfig"
1292	1295
1293	1296	source "scripts/gcc-plugins/Kconfig"

arch/x86/Kconfig+17−0

@@ -125,6 +125,7 @@ config X86 · 124 unmodified lines
125	125	select CLOCKSOURCE_VALIDATE_LAST_CYCLE
126	126	select CLOCKSOURCE_WATCHDOG
127	127	select DCACHE_WORD_ACCESS
	128	+ select DYNAMIC_SIGFRAME
128	129	select EDAC_ATOMIC_SCRUB
129	130	select EDAC_SUPPORT
130	131	select GENERIC_CLOCKEVENTS_BROADCAST if X86_64 \|\| (X86_32 && X86_LOCAL_APIC)
@@ -2399,6 +2400,22 @@ config MODIFY_LDT_SYSCALL · 2268 unmodified lines
2399	2400
2400	2401	Saying 'N' here may make sense for embedded or server kernels.
2401	2402
	2403	+config STRICT_SIGALTSTACK_SIZE
	2404	+ bool "Enforce strict size checking for sigaltstack"
	2405	+ depends on DYNAMIC_SIGFRAME
	2406	+ help
	2407	+ For historical reasons MINSIGSTKSZ is a constant which became
	2408	+ already too small with AVX512 support. Add a mechanism to
	2409	+ enforce strict checking of the sigaltstack size against the
	2410	+ real size of the FPU frame. This option enables the check
	2411	+ by default. It can also be controlled via the kernel command
	2412	+ line option 'strict_sas_size' independent of this config
	2413	+ switch. Enabling it might break existing applications which
	2414	+ allocate a too small sigaltstack but 'work' because they
	2415	+ never get a signal delivered.
	2416	+
	2417	+ Say 'N' unless you want to really enforce this check.
	2418	+
2402	2419	source "kernel/livepatch/Kconfig"
2403	2420
2404	2421	endmenu

arch/x86/events/perf_event.h+1−0

@@ -14,6 +14,7 @@ · 13 unmodified lines
14	14
15	15	#include <linux/perf_event.h>
16	16
	17	+#include <asm/fpu/xstate.h>
17	18	#include <asm/intel_ds.h>
18	19	#include <asm/cpu.h>
19	20

arch/x86/ia32/ia32_signal.c+7−8

@@ -24,7 +24,6 @@ · 23 unmodified lines
24	24	#include <linux/syscalls.h>
25	25	#include <asm/ucontext.h>
26	26	#include <linux/uaccess.h>
27		−#include <asm/fpu/internal.h>
28	27	#include <asm/fpu/signal.h>
29	28	#include <asm/ptrace.h>
30	29	#include <asm/ia32_unistd.h>
@@ -57,8 +56,8 @@ static inline void reload_segments(struct sigcontext_32 *sc) · 26 unmodified lines
57	56	/*
58	57	* Do a signal return; undo the signal stack.
59	58	*/
60		−static int ia32_restore_sigcontext(struct pt_regs *regs,
61		− struct sigcontext_32 __user *usc)
	59	+static bool ia32_restore_sigcontext(struct pt_regs *regs,
	60	+ struct sigcontext_32 __user *usc)
62	61	{
63	62	struct sigcontext_32 sc;
64	63
@@ -66,7 +65,7 @@ static int ia32_restore_sigcontext(struct pt_regs *regs, · 1 unmodified line
66	65	current->restart_block.fn = do_no_restart_syscall;
67	66
68	67	if (unlikely(copy_from_user(&sc, usc, sizeof(sc))))
69		− return -EFAULT;
	68	+ return false;
70	69
71	70	/* Get only the ia32 registers. */
72	71	regs->bx = sc.bx;
@@ -111,7 +110,7 @@ COMPAT_SYSCALL_DEFINE0(sigreturn) · 38 unmodified lines
111	110
112	111	set_current_blocked(&set);
113	112
114		− if (ia32_restore_sigcontext(regs, &frame->sc))
	113	+ if (!ia32_restore_sigcontext(regs, &frame->sc))
115	114	goto badframe;
116	115	return regs->ax;
117	116
@@ -135,7 +134,7 @@ COMPAT_SYSCALL_DEFINE0(rt_sigreturn) · 17 unmodified lines
135	134
136	135	set_current_blocked(&set);
137	136
138		− if (ia32_restore_sigcontext(regs, &frame->uc.uc_mcontext))
	137	+ if (!ia32_restore_sigcontext(regs, &frame->uc.uc_mcontext))
139	138	goto badframe;
140	139
141	140	if (compat_restore_altstack(&frame->uc.uc_stack))
@@ -220,8 +219,8 @@ static void __user get_sigframe(struct ksignal ksig, struct pt_regs *regs, · 78 unmodified lines
220	219
221	220	sp = fpu__alloc_mathframe(sp, 1, &fx_aligned, &math_size);
222	221	fpstate = (struct _fpstate_32 __user ) sp;
223		− if (copy_fpstate_to_sigframe(fpstate, (void __user )fx_aligned,
224		− math_size) < 0)
	222	+ if (!copy_fpstate_to_sigframe(fpstate, (void __user )fx_aligned,
	223	+ math_size))
225	224	return (void __user *) -1L;
226	225
227	226	sp -= frame_size;

arch/x86/include/asm/asm.h+20−30

@@ -122,28 +122,19 @@ · 121 unmodified lines
122	122
123	123	#ifdef __KERNEL__
124	124
	125	+# include <asm/extable_fixup_types.h>
	126	+
125	127	/* Exception table entry */
126	128	#ifdef __ASSEMBLY__
127		−# define _ASM_EXTABLE_HANDLE(from, to, handler) \
	129	+
	130	+# define _ASM_EXTABLE_TYPE(from, to, type) \
128	131	.pushsection "__ex_table","a" ; \
129	132	.balign 4 ; \
130	133	.long (from) - . ; \
131	134	.long (to) - . ; \
132		− .long (handler) - . ; \
	135	+ .long type ; \
133	136	.popsection
134	137
135		−# define _ASM_EXTABLE(from, to) \
136		− _ASM_EXTABLE_HANDLE(from, to, ex_handler_default)
137		−
138		−# define _ASM_EXTABLE_UA(from, to) \
139		− _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess)
140		−
141		−# define _ASM_EXTABLE_CPY(from, to) \
142		− _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy)
143		−
144		−# define _ASM_EXTABLE_FAULT(from, to) \
145		− _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault)
146		−
147	138	# ifdef CONFIG_KPROBES
148	139	# define _ASM_NOKPROBE(entry) \
149	140	.pushsection "_kprobe_blacklist","aw" ; \
@@ -155,27 +146,15 @@ · 5 unmodified lines
155	146	# endif
156	147
157	148	#else /* ! __ASSEMBLY__ */
158		−# define _EXPAND_EXTABLE_HANDLE(x) #x
159		−# define _ASM_EXTABLE_HANDLE(from, to, handler) \
	149	+
	150	+# define _ASM_EXTABLE_TYPE(from, to, type) \
160	151	" .pushsection \"__ex_table\",\"a\"\n" \
161	152	" .balign 4\n" \
162	153	" .long (" #from ") - .\n" \
163	154	" .long (" #to ") - .\n" \
164		− " .long (" _EXPAND_EXTABLE_HANDLE(handler) ") - .\n" \
	155	+ " .long " __stringify(type) " \n" \
165	156	" .popsection\n"
166	157
167		−# define _ASM_EXTABLE(from, to) \
168		− _ASM_EXTABLE_HANDLE(from, to, ex_handler_default)
169		−
170		−# define _ASM_EXTABLE_UA(from, to) \
171		− _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess)
172		−
173		−# define _ASM_EXTABLE_CPY(from, to) \
174		− _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy)
175		−
176		−# define _ASM_EXTABLE_FAULT(from, to) \
177		− _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault)
178		−
179	158	/* For C file, we already have NOKPROBE_SYMBOL macro */
180	159
181	160	/*
@@ -188,6 +167,17 @@ register unsigned long current_stack_pointer asm(_ASM_SP); · 6 unmodified lines
188	167	#define ASM_CALL_CONSTRAINT "+r" (current_stack_pointer)
189	168	#endif /* __ASSEMBLY__ */
190	169
191		−#endif /* __KERNEL__ */
	170	+#define _ASM_EXTABLE(from, to) \
	171	+ _ASM_EXTABLE_TYPE(from, to, EX_TYPE_DEFAULT)
192	172
	173	+#define _ASM_EXTABLE_UA(from, to) \
	174	+ _ASM_EXTABLE_TYPE(from, to, EX_TYPE_UACCESS)
	175	+
	176	+#define _ASM_EXTABLE_CPY(from, to) \
	177	+ _ASM_EXTABLE_TYPE(from, to, EX_TYPE_COPY)
	178	+
	179	+#define _ASM_EXTABLE_FAULT(from, to) \
	180	+ _ASM_EXTABLE_TYPE(from, to, EX_TYPE_FAULT)
	181	+
	182	+#endif /* __KERNEL__ */
193	183	#endif /* _ASM_X86_ASM_H */

arch/x86/include/asm/cpufeatures.h+2−0

@@ -277,6 +277,7 @@ · 276 unmodified lines
277	277	#define X86_FEATURE_XSAVEC (1032+ 1) / XSAVEC instruction */
278	278	#define X86_FEATURE_XGETBV1 (1032+ 2) / XGETBV with ECX = 1 instruction */
279	279	#define X86_FEATURE_XSAVES (1032+ 3) / XSAVES/XRSTORS instructions */
	280	+#define X86_FEATURE_XFD (1032+ 4) / "" eXtended Feature Disabling */
280	281
281	282	/*
282	283	* Extended auxiliary flags: Linux defined - for features scattered in various
@@ -298,6 +299,7 @@ · 15 unmodified lines
298	299	/* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */
299	300	#define X86_FEATURE_AVX_VNNI (1232+ 4) / AVX VNNI instructions */
300	301	#define X86_FEATURE_AVX512_BF16 (1232+ 5) / AVX512 BFLOAT16 instructions */
	302	+#define X86_FEATURE_AMX_TILE (1832+24) / AMX tile Support */
301	303
302	304	/* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */
303	305	#define X86_FEATURE_CLZERO (1332+ 0) / CLZERO instruction */

arch/x86/include/asm/extable.h+28−16

@@ -1,12 +1,18 @@
1	1	/* SPDX-License-Identifier: GPL-2.0 */
2	2	#ifndef _ASM_X86_EXTABLE_H
3	3	#define _ASM_X86_EXTABLE_H
	4	+
	5	+#include <asm/extable_fixup_types.h>
	6	+
4	7	/*
5		− * The exception table consists of triples of addresses relative to the
6		− * exception table entry itself. The first address is of an instruction
7		− * that is allowed to fault, the second is the target at which the program
8		− * should continue. The third is a handler function to deal with the fault
9		− * caused by the instruction in the first field.
	8	+ * The exception table consists of two addresses relative to the
	9	+ * exception table entry itself and a type selector field.
	10	+ *
	11	+ * The first address is of an instruction that is allowed to fault, the
	12	+ * second is the target at which the program should continue.
	13	+ *
	14	+ * The type entry is used by fixup_exception() to select the handler to
	15	+ * deal with the fault caused by the instruction in the first field.
10	16	*
11	17	* All the routines below use bits of fixup code that are out of line
12	18	* with the main instruction path. This means when everything is well,
@@ -15,7 +21,7 @@ · 2 unmodified lines
15	21	*/
16	22
17	23	struct exception_table_entry {
18		− int insn, fixup, handler;
	24	+ int insn, fixup, type;
19	25	};
20	26	struct pt_regs;
21	27
@@ -25,21 +31,27 @@ struct pt_regs; · 3 unmodified lines
25	31	do { \
26	32	(a)->fixup = (b)->fixup + (delta); \
27	33	(b)->fixup = (tmp).fixup - (delta); \
28		− (a)->handler = (b)->handler + (delta); \
29		− (b)->handler = (tmp).handler - (delta); \
	34	+ (a)->type = (b)->type; \
	35	+ (b)->type = (tmp).type; \
30	36	} while (0)
31	37
32		−enum handler_type {
33		− EX_HANDLER_NONE,
34		− EX_HANDLER_FAULT,
35		− EX_HANDLER_UACCESS,
36		− EX_HANDLER_OTHER
37		−};
38		−
39	38	extern int fixup_exception(struct pt_regs *regs, int trapnr,
40	39	unsigned long error_code, unsigned long fault_addr);
41	40	extern int fixup_bug(struct pt_regs *regs, int trapnr);
42		−extern enum handler_type ex_get_fault_handler_type(unsigned long ip);
	41	+extern int ex_get_fixup_type(unsigned long ip);
43	42	extern void early_fixup_exception(struct pt_regs *regs, int trapnr);
44	43
	44	+#ifdef CONFIG_X86_MCE
	45	+extern void ex_handler_msr_mce(struct pt_regs *regs, bool wrmsr);
	46	+#else
	47	+static inline void ex_handler_msr_mce(struct pt_regs *regs, bool wrmsr) { }
	48	+#endif
	49	+
	50	+#if defined(CONFIG_BPF_JIT) && defined(CONFIG_X86_64)
	51	+bool ex_handler_bpf(const struct exception_table_entry x, struct pt_regs regs);
	52	+#else
	53	+static inline bool ex_handler_bpf(const struct exception_table_entry *x,
	54	+ struct pt_regs *regs) { return false; }
	55	+#endif
	56	+
45	57	#endif

arch/x86/include/asm/extable_fixup_types.h+22−0

@@ -0,0 +1,22 @@
	1	+/* SPDX-License-Identifier: GPL-2.0 */
	2	+#ifndef _ASM_X86_EXTABLE_FIXUP_TYPES_H
	3	+#define _ASM_X86_EXTABLE_FIXUP_TYPES_H
	4	+
	5	+#define EX_TYPE_NONE 0
	6	+#define EX_TYPE_DEFAULT 1
	7	+#define EX_TYPE_FAULT 2
	8	+#define EX_TYPE_UACCESS 3
	9	+#define EX_TYPE_COPY 4
	10	+#define EX_TYPE_CLEAR_FS 5
	11	+#define EX_TYPE_FPU_RESTORE 6
	12	+#define EX_TYPE_WRMSR 7
	13	+#define EX_TYPE_RDMSR 8
	14	+#define EX_TYPE_BPF 9
	15	+
	16	+#define EX_TYPE_WRMSR_IN_MCE 10
	17	+#define EX_TYPE_RDMSR_IN_MCE 11
	18	+
	19	+#define EX_TYPE_DEFAULT_MCE_SAFE 12
	20	+#define EX_TYPE_FAULT_MCE_SAFE 13
	21	+
	22	+#endif

arch/x86/include/asm/fpu/api.h+56−2

@@ -12,6 +12,8 @@ · 11 unmodified lines
12	12	#define _ASM_X86_FPU_API_H
13	13	#include <linux/bottom_half.h>
14	14
	15	+#include <asm/fpu/types.h>
	16	+
15	17	/*
16	18	* Use kernel_fpu_begin/end() if you intend to use FPU in kernel context. It
17	19	* disables preemption so be careful if you intend to use it for long periods
@@ -48,9 +50,9 @@ static inline void kernel_fpu_begin(void) · 30 unmodified lines
48	50	}
49	51
50	52	/*
51		− * Use fpregs_lock() while editing CPU's FPU registers or fpu->state.
	53	+ * Use fpregs_lock() while editing CPU's FPU registers or fpu->fpstate.
52	54	* A context switch will (and softirq might) save CPU's FPU registers to
53		− * fpu->state and set TIF_NEED_FPU_LOAD leaving CPU's FPU registers in
	55	+ * fpu->fpstate.regs and set TIF_NEED_FPU_LOAD leaving CPU's FPU registers in
54	56	* a random state.
55	57	*
56	58	* local_bh_disable() protects against both preemption and soft interrupts
@@ -108,4 +110,56 @@ extern int cpu_has_xfeatures(u64 xfeatures_mask, const char **feature_name); · 51 unmodified lines
108	110
109	111	static inline void update_pasid(void) { }
110	112
	113	+/* Trap handling */
	114	+extern int fpu__exception_code(struct fpu *fpu, int trap_nr);
	115	+extern void fpu_sync_fpstate(struct fpu *fpu);
	116	+extern void fpu_reset_from_exception_fixup(void);
	117	+
	118	+/* Boot, hotplug and resume */
	119	+extern void fpu__init_cpu(void);
	120	+extern void fpu__init_system(struct cpuinfo_x86 *c);
	121	+extern void fpu__init_check_bugs(void);
	122	+extern void fpu__resume_cpu(void);
	123	+
	124	+#ifdef CONFIG_MATH_EMULATION
	125	+extern void fpstate_init_soft(struct swregs_state *soft);
	126	+#else
	127	+static inline void fpstate_init_soft(struct swregs_state *soft) {}
	128	+#endif
	129	+
	130	+/* State tracking */
	131	+DECLARE_PER_CPU(struct fpu *, fpu_fpregs_owner_ctx);
	132	+
	133	+/* Process cleanup */
	134	+#ifdef CONFIG_X86_64
	135	+extern void fpstate_free(struct fpu *fpu);
	136	+#else
	137	+static inline void fpstate_free(struct fpu *fpu) { }
	138	+#endif
	139	+
	140	+/* fpstate-related functions which are exported to KVM */
	141	+extern void fpstate_clear_xstate_component(struct fpstate *fps, unsigned int xfeature);
	142	+
	143	+/* KVM specific functions */
	144	+extern bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu);
	145	+extern void fpu_free_guest_fpstate(struct fpu_guest *gfpu);
	146	+extern int fpu_swap_kvm_fpstate(struct fpu_guest *gfpu, bool enter_guest);
	147	+
	148	+extern void fpu_copy_guest_fpstate_to_uabi(struct fpu_guest gfpu, void buf, unsigned int size, u32 pkru);
	149	+extern int fpu_copy_uabi_to_guest_fpstate(struct fpu_guest gfpu, const void buf, u64 xcr0, u32 *vpkru);
	150	+
	151	+static inline void fpstate_set_confidential(struct fpu_guest *gfpu)
	152	+{
	153	+ gfpu->fpstate->is_confidential = true;
	154	+}
	155	+
	156	+static inline bool fpstate_is_confidential(struct fpu_guest *gfpu)
	157	+{
	158	+ return gfpu->fpstate->is_confidential;
	159	+}
	160	+
	161	+/* prctl */
	162	+struct task_struct;
	163	+extern long fpu_xstate_prctl(struct task_struct *tsk, int option, unsigned long arg2);
	164	+
111	165	#endif /* _ASM_X86_FPU_API_H */

arch/x86/include/asm/fpu/internal.h+0−540

@@ -1,540 +0,0 @@
1		−/* SPDX-License-Identifier: GPL-2.0 */
2		−/*
3		− * Copyright (C) 1994 Linus Torvalds
4		− *
5		− * Pentium III FXSR, SSE support
6		− * General FPU state handling cleanups
7		− * Gareth Hughes <gareth@valinux.com>, May 2000
8		− * x86-64 work by Andi Kleen 2002
9		− */
10		−
11		−#ifndef _ASM_X86_FPU_INTERNAL_H
12		−#define _ASM_X86_FPU_INTERNAL_H
13		−
14		−#include <linux/compat.h>
15		−#include <linux/sched.h>
16		−#include <linux/slab.h>
17		−#include <linux/mm.h>
18		−
19		−#include <asm/user.h>
20		−#include <asm/fpu/api.h>
21		−#include <asm/fpu/xstate.h>
22		−#include <asm/fpu/xcr.h>
23		−#include <asm/cpufeature.h>
24		−#include <asm/trace/fpu.h>
25		−
26		−/*
27		− * High level FPU state handling functions:
28		− */
29		−extern int fpu__restore_sig(void __user *buf, int ia32_frame);
30		−extern void fpu__drop(struct fpu *fpu);
31		−extern void fpu__clear_user_states(struct fpu *fpu);
32		−extern int fpu__exception_code(struct fpu *fpu, int trap_nr);
33		−
34		−extern void fpu_sync_fpstate(struct fpu *fpu);
35		−
36		−/* Clone and exit operations */
37		−extern int fpu_clone(struct task_struct *dst);
38		−extern void fpu_flush_thread(void);
39		−
40		−/*
41		− * Boot time FPU initialization functions:
42		− */
43		−extern void fpu__init_cpu(void);
44		−extern void fpu__init_system_xstate(void);
45		−extern void fpu__init_cpu_xstate(void);
46		−extern void fpu__init_system(struct cpuinfo_x86 *c);
47		−extern void fpu__init_check_bugs(void);
48		−extern void fpu__resume_cpu(void);
49		−
50		−/*
51		− * Debugging facility:
52		− */
53		−#ifdef CONFIG_X86_DEBUG_FPU
54		−# define WARN_ON_FPU(x) WARN_ON_ONCE(x)
55		−#else
56		−# define WARN_ON_FPU(x) ({ (void)(x); 0; })
57		−#endif
58		−
59		−/*
60		− * FPU related CPU feature flag helper routines:
61		− */
62		−static __always_inline __pure bool use_xsaveopt(void)
63		−{
64		− return static_cpu_has(X86_FEATURE_XSAVEOPT);
65		−}
66		−
67		−static __always_inline __pure bool use_xsave(void)
68		−{
69		− return static_cpu_has(X86_FEATURE_XSAVE);
70		−}
71		−
72		−static __always_inline __pure bool use_fxsr(void)
73		−{
74		− return static_cpu_has(X86_FEATURE_FXSR);
75		−}
76		−
77		−/*
78		− * fpstate handling functions:
79		− */
80		−
81		−extern union fpregs_state init_fpstate;
82		−
83		−extern void fpstate_init(union fpregs_state *state);
84		−#ifdef CONFIG_MATH_EMULATION
85		−extern void fpstate_init_soft(struct swregs_state *soft);
86		−#else
87		−static inline void fpstate_init_soft(struct swregs_state *soft) {}
88		−#endif
89		−extern void save_fpregs_to_fpstate(struct fpu *fpu);
90		−
91		−/* Returns 0 or the negated trap number, which results in -EFAULT for #PF */
92		−#define user_insn(insn, output, input...) \
93		−({ \
94		− int err; \
95		− \
96		− might_fault(); \
97		− \
98		− asm volatile(ASM_STAC "\n" \
99		− "1: " #insn "\n" \
100		− "2: " ASM_CLAC "\n" \
101		− ".section .fixup,\"ax\"\n" \
102		− "3: negl %%eax\n" \
103		− " jmp 2b\n" \
104		− ".previous\n" \
105		− _ASM_EXTABLE_FAULT(1b, 3b) \
106		− : [err] "=a" (err), output \
107		− : "0"(0), input); \
108		− err; \
109		−})
110		−
111		−#define kernel_insn_err(insn, output, input...) \
112		−({ \
113		− int err; \
114		− asm volatile("1:" #insn "\n\t" \
115		− "2:\n" \
116		− ".section .fixup,\"ax\"\n" \
117		− "3: movl $-1,%[err]\n" \
118		− " jmp 2b\n" \
119		− ".previous\n" \
120		− _ASM_EXTABLE(1b, 3b) \
121		− : [err] "=r" (err), output \
122		− : "0"(0), input); \
123		− err; \
124		−})
125		−
126		−#define kernel_insn(insn, output, input...) \
127		− asm volatile("1:" #insn "\n\t" \
128		− "2:\n" \
129		− _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_fprestore) \
130		− : output : input)
131		−
132		−static inline int fnsave_to_user_sigframe(struct fregs_state __user *fx)
133		−{
134		− return user_insn(fnsave %[fx]; fwait, [fx] "=m" (fx), "m" (fx));
135		−}
136		−
137		−static inline int fxsave_to_user_sigframe(struct fxregs_state __user *fx)
138		−{
139		− if (IS_ENABLED(CONFIG_X86_32))
140		− return user_insn(fxsave %[fx], [fx] "=m" (fx), "m" (fx));
141		− else
142		− return user_insn(fxsaveq %[fx], [fx] "=m" (fx), "m" (fx));
143		−
144		−}
145		−
146		−static inline void fxrstor(struct fxregs_state *fx)
147		−{
148		− if (IS_ENABLED(CONFIG_X86_32))
149		− kernel_insn(fxrstor %[fx], "=m" (fx), [fx] "m" (fx));
150		− else
151		− kernel_insn(fxrstorq %[fx], "=m" (fx), [fx] "m" (fx));
152		−}
153		−
154		−static inline int fxrstor_safe(struct fxregs_state *fx)
155		−{
156		− if (IS_ENABLED(CONFIG_X86_32))
157		− return kernel_insn_err(fxrstor %[fx], "=m" (fx), [fx] "m" (fx));
158		− else
159		− return kernel_insn_err(fxrstorq %[fx], "=m" (fx), [fx] "m" (fx));
160		−}
161		−
162		−static inline int fxrstor_from_user_sigframe(struct fxregs_state __user *fx)
163		−{
164		− if (IS_ENABLED(CONFIG_X86_32))
165		− return user_insn(fxrstor %[fx], "=m" (fx), [fx] "m" (fx));
166		− else
167		− return user_insn(fxrstorq %[fx], "=m" (fx), [fx] "m" (fx));
168		−}
169		−
170		−static inline void frstor(struct fregs_state *fx)
171		−{
172		− kernel_insn(frstor %[fx], "=m" (fx), [fx] "m" (fx));
173		−}
174		−
175		−static inline int frstor_safe(struct fregs_state *fx)
176		−{
177		− return kernel_insn_err(frstor %[fx], "=m" (fx), [fx] "m" (fx));
178		−}
179		−
180		−static inline int frstor_from_user_sigframe(struct fregs_state __user *fx)
181		−{
182		− return user_insn(frstor %[fx], "=m" (fx), [fx] "m" (fx));
183		−}
184		−
185		−static inline void fxsave(struct fxregs_state *fx)
186		−{
187		− if (IS_ENABLED(CONFIG_X86_32))
188		− asm volatile( "fxsave %[fx]" : [fx] "=m" (*fx));
189		− else
190		− asm volatile("fxsaveq %[fx]" : [fx] "=m" (*fx));
191		−}
192		−
193		−/* These macros all use (%edi)/(%rdi) as the single memory argument. */
194		−#define XSAVE ".byte " REX_PREFIX "0x0f,0xae,0x27"
195		−#define XSAVEOPT ".byte " REX_PREFIX "0x0f,0xae,0x37"
196		−#define XSAVES ".byte " REX_PREFIX "0x0f,0xc7,0x2f"
197		−#define XRSTOR ".byte " REX_PREFIX "0x0f,0xae,0x2f"
198		−#define XRSTORS ".byte " REX_PREFIX "0x0f,0xc7,0x1f"
199		−
200		−/*
201		− * After this @err contains 0 on success or the negated trap number when
202		− * the operation raises an exception. For faults this results in -EFAULT.
203		− */
204		−#define XSTATE_OP(op, st, lmask, hmask, err) \
205		− asm volatile("1:" op "\n\t" \
206		− "xor %[err], %[err]\n" \
207		− "2:\n\t" \
208		− ".pushsection .fixup,\"ax\"\n\t" \
209		− "3: negl %%eax\n\t" \
210		− "jmp 2b\n\t" \
211		− ".popsection\n\t" \
212		− _ASM_EXTABLE_FAULT(1b, 3b) \
213		− : [err] "=a" (err) \
214		− : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \
215		− : "memory")
216		−
217		−/*
218		− * If XSAVES is enabled, it replaces XSAVEOPT because it supports a compact
219		− * format and supervisor states in addition to modified optimization in
220		− * XSAVEOPT.
221		− *
222		− * Otherwise, if XSAVEOPT is enabled, XSAVEOPT replaces XSAVE because XSAVEOPT
223		− * supports modified optimization which is not supported by XSAVE.
224		− *
225		− * We use XSAVE as a fallback.
226		− *
227		− * The 661 label is defined in the ALTERNATIVE* macros as the address of the
228		− * original instruction which gets replaced. We need to use it here as the
229		− * address of the instruction where we might get an exception at.
230		− */
231		−#define XSTATE_XSAVE(st, lmask, hmask, err) \
232		− asm volatile(ALTERNATIVE_2(XSAVE, \
233		− XSAVEOPT, X86_FEATURE_XSAVEOPT, \
234		− XSAVES, X86_FEATURE_XSAVES) \
235		− "\n" \
236		− "xor %[err], %[err]\n" \
237		− "3:\n" \
238		− ".pushsection .fixup,\"ax\"\n" \
239		− "4: movl $-2, %[err]\n" \
240		− "jmp 3b\n" \
241		− ".popsection\n" \
242		− _ASM_EXTABLE(661b, 4b) \
243		− : [err] "=r" (err) \
244		− : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \
245		− : "memory")
246		−
247		−/*
248		− * Use XRSTORS to restore context if it is enabled. XRSTORS supports compact
249		− * XSAVE area format.
250		− */
251		−#define XSTATE_XRESTORE(st, lmask, hmask) \
252		− asm volatile(ALTERNATIVE(XRSTOR, \
253		− XRSTORS, X86_FEATURE_XSAVES) \
254		− "\n" \
255		− "3:\n" \
256		− _ASM_EXTABLE_HANDLE(661b, 3b, ex_handler_fprestore)\
257		− : \
258		− : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \
259		− : "memory")
260		−
261		−/*
262		− * This function is called only during boot time when x86 caps are not set
263		− * up and alternative can not be used yet.
264		− */
265		−static inline void os_xrstor_booting(struct xregs_state *xstate)
266		−{
267		− u64 mask = xfeatures_mask_fpstate();
268		− u32 lmask = mask;
269		− u32 hmask = mask >> 32;
270		− int err;
271		−
272		− WARN_ON(system_state != SYSTEM_BOOTING);
273		−
274		− if (boot_cpu_has(X86_FEATURE_XSAVES))
275		− XSTATE_OP(XRSTORS, xstate, lmask, hmask, err);
276		− else
277		− XSTATE_OP(XRSTOR, xstate, lmask, hmask, err);
278		−
279		− /*
280		− * We should never fault when copying from a kernel buffer, and the FPU
281		− * state we set at boot time should be valid.
282		− */
283		− WARN_ON_FPU(err);
284		−}
285		−
286		−/*
287		− * Save processor xstate to xsave area.
288		− *
289		− * Uses either XSAVE or XSAVEOPT or XSAVES depending on the CPU features
290		− * and command line options. The choice is permanent until the next reboot.
291		− */
292		−static inline void os_xsave(struct xregs_state *xstate)
293		−{
294		− u64 mask = xfeatures_mask_all;
295		− u32 lmask = mask;
296		− u32 hmask = mask >> 32;
297		− int err;
298		−
299		− WARN_ON_FPU(!alternatives_patched);
300		−
301		− XSTATE_XSAVE(xstate, lmask, hmask, err);
302		−
303		− /* We should never fault when copying to a kernel buffer: */
304		− WARN_ON_FPU(err);
305		−}
306		−
307		−/*
308		− * Restore processor xstate from xsave area.
309		− *
310		− * Uses XRSTORS when XSAVES is used, XRSTOR otherwise.
311		− */
312		−static inline void os_xrstor(struct xregs_state *xstate, u64 mask)
313		−{
314		− u32 lmask = mask;
315		− u32 hmask = mask >> 32;
316		−
317		− XSTATE_XRESTORE(xstate, lmask, hmask);
318		−}
319		−
320		−/*
321		− * Save xstate to user space xsave area.
322		− *
323		− * We don't use modified optimization because xrstor/xrstors might track
324		− * a different application.
325		− *
326		− * We don't use compacted format xsave area for
327		− * backward compatibility for old applications which don't understand
328		− * compacted format of xsave area.
329		− */
330		−static inline int xsave_to_user_sigframe(struct xregs_state __user *buf)
331		−{
332		− /*
333		− * Include the features which are not xsaved/rstored by the kernel
334		− * internally, e.g. PKRU. That's user space ABI and also required
335		− * to allow the signal handler to modify PKRU.
336		− */
337		− u64 mask = xfeatures_mask_uabi();
338		− u32 lmask = mask;
339		− u32 hmask = mask >> 32;
340		− int err;
341		−
342		− /*
343		− * Clear the xsave header first, so that reserved fields are
344		− * initialized to zero.
345		− */
346		− err = __clear_user(&buf->header, sizeof(buf->header));
347		− if (unlikely(err))
348		− return -EFAULT;
349		−
350		− stac();
351		− XSTATE_OP(XSAVE, buf, lmask, hmask, err);
352		− clac();
353		−
354		− return err;
355		−}
356		−
357		−/*
358		− * Restore xstate from user space xsave area.
359		− */
360		−static inline int xrstor_from_user_sigframe(struct xregs_state __user *buf, u64 mask)
361		−{
362		− struct xregs_state xstate = ((__force struct xregs_state )buf);
363		− u32 lmask = mask;
364		− u32 hmask = mask >> 32;
365		− int err;
366		−
367		− stac();
368		− XSTATE_OP(XRSTOR, xstate, lmask, hmask, err);
369		− clac();
370		−
371		− return err;
372		−}
373		−
374		−/*
375		− * Restore xstate from kernel space xsave area, return an error code instead of
376		− * an exception.
377		− */
378		−static inline int os_xrstor_safe(struct xregs_state *xstate, u64 mask)
379		−{
380		− u32 lmask = mask;
381		− u32 hmask = mask >> 32;
382		− int err;
383		−
384		− if (cpu_feature_enabled(X86_FEATURE_XSAVES))
385		− XSTATE_OP(XRSTORS, xstate, lmask, hmask, err);
386		− else
387		− XSTATE_OP(XRSTOR, xstate, lmask, hmask, err);
388		−
389		− return err;
390		−}
391		−
392		−extern void __restore_fpregs_from_fpstate(union fpregs_state *fpstate, u64 mask);
393		−
394		−static inline void restore_fpregs_from_fpstate(union fpregs_state *fpstate)
395		−{
396		− __restore_fpregs_from_fpstate(fpstate, xfeatures_mask_fpstate());
397		−}
398		−
399		−extern int copy_fpstate_to_sigframe(void __user buf, void __user fp, int size);
400		−
401		−/*
402		− * FPU context switch related helper methods:
403		− */
404		−
405		−DECLARE_PER_CPU(struct fpu *, fpu_fpregs_owner_ctx);
406		−
407		−/*
408		− * The in-register FPU state for an FPU context on a CPU is assumed to be
409		− * valid if the fpu->last_cpu matches the CPU, and the fpu_fpregs_owner_ctx
410		− * matches the FPU.
411		− *
412		− * If the FPU register state is valid, the kernel can skip restoring the
413		− * FPU state from memory.
414		− *
415		− * Any code that clobbers the FPU registers or updates the in-memory
416		− * FPU state for a task MUST let the rest of the kernel know that the
417		− * FPU registers are no longer valid for this task.
418		− *
419		− * Either one of these invalidation functions is enough. Invalidate
420		− * a resource you control: CPU if using the CPU for something else
421		− * (with preemption disabled), FPU for the current task, or a task that
422		− * is prevented from running by the current task.
423		− */
424		−static inline void __cpu_invalidate_fpregs_state(void)
425		−{
426		− __this_cpu_write(fpu_fpregs_owner_ctx, NULL);
427		−}
428		−
429		−static inline void __fpu_invalidate_fpregs_state(struct fpu *fpu)
430		−{
431		− fpu->last_cpu = -1;
432		−}
433		−
434		−static inline int fpregs_state_valid(struct fpu *fpu, unsigned int cpu)
435		−{
436		− return fpu == this_cpu_read(fpu_fpregs_owner_ctx) && cpu == fpu->last_cpu;
437		−}
438		−
439		−/*
440		− * These generally need preemption protection to work,
441		− * do try to avoid using these on their own:
442		− */
443		−static inline void fpregs_deactivate(struct fpu *fpu)
444		−{
445		− this_cpu_write(fpu_fpregs_owner_ctx, NULL);
446		− trace_x86_fpu_regs_deactivated(fpu);
447		−}
448		−
449		−static inline void fpregs_activate(struct fpu *fpu)
450		−{
451		− this_cpu_write(fpu_fpregs_owner_ctx, fpu);
452		− trace_x86_fpu_regs_activated(fpu);
453		−}
454		−
455		−/* Internal helper for switch_fpu_return() and signal frame setup */
456		−static inline void fpregs_restore_userregs(void)
457		−{
458		− struct fpu *fpu = &current->thread.fpu;
459		− int cpu = smp_processor_id();
460		−
461		− if (WARN_ON_ONCE(current->flags & PF_KTHREAD))
462		− return;
463		−
464		− if (!fpregs_state_valid(fpu, cpu)) {
465		− u64 mask;
466		−
467		− /*
468		− * This restores _all_ xstate which has not been
469		− * established yet.
470		− *
471		− * If PKRU is enabled, then the PKRU value is already
472		− * correct because it was either set in switch_to() or in
473		− * flush_thread(). So it is excluded because it might be
474		− * not up to date in current->thread.fpu.xsave state.
475		− */
476		− mask = xfeatures_mask_restore_user() \|
477		− xfeatures_mask_supervisor();
478		− __restore_fpregs_from_fpstate(&fpu->state, mask);
479		−
480		− fpregs_activate(fpu);
481		− fpu->last_cpu = cpu;
482		− }
483		− clear_thread_flag(TIF_NEED_FPU_LOAD);
484		−}
485		−
486		−/*
487		− * FPU state switching for scheduling.
488		− *
489		− * This is a two-stage process:
490		− *
491		− * - switch_fpu_prepare() saves the old state.
492		− * This is done within the context of the old process.
493		− *
494		− * - switch_fpu_finish() sets TIF_NEED_FPU_LOAD; the floating point state
495		− * will get loaded on return to userspace, or when the kernel needs it.
496		− *
497		− * If TIF_NEED_FPU_LOAD is cleared then the CPU's FPU registers
498		− * are saved in the current thread's FPU register state.
499		− *
500		− * If TIF_NEED_FPU_LOAD is set then CPU's FPU registers may not
501		− * hold current()'s FPU registers. It is required to load the
502		− * registers before returning to userland or using the content
503		− * otherwise.
504		− *
505		− * The FPU context is only stored/restored for a user task and
506		− * PF_KTHREAD is used to distinguish between kernel and user threads.
507		− */
508		−static inline void switch_fpu_prepare(struct fpu *old_fpu, int cpu)
509		−{
510		− if (static_cpu_has(X86_FEATURE_FPU) && !(current->flags & PF_KTHREAD)) {
511		− save_fpregs_to_fpstate(old_fpu);
512		− /*
513		− * The save operation preserved register state, so the
514		− * fpu_fpregs_owner_ctx is still @old_fpu. Store the
515		− * current CPU number in @old_fpu, so the next return
516		− * to user space can avoid the FPU register restore
517		− * when is returns on the same CPU and still owns the
518		− * context.
519		− */
520		− old_fpu->last_cpu = cpu;
521		−
522		− trace_x86_fpu_regs_deactivated(old_fpu);
523		− }
524		−}
525		−
526		−/*
527		− * Misc helper functions:
528		− */
529		−
530		−/*
531		− * Delay loading of the complete FPU state until the return to userland.
532		− * PKRU is handled separately.
533		− */
534		−static inline void switch_fpu_finish(struct fpu *new_fpu)
535		−{
536		− if (cpu_feature_enabled(X86_FEATURE_FPU))
537		− set_thread_flag(TIF_NEED_FPU_LOAD);
538		−}
539		−
540		−#endif /* _ASM_X86_FPU_INTERNAL_H */

arch/x86/include/asm/fpu/sched.h+68−0

@@ -0,0 +1,68 @@
	1	+/* SPDX-License-Identifier: GPL-2.0 */
	2	+#ifndef _ASM_X86_FPU_SCHED_H
	3	+#define _ASM_X86_FPU_SCHED_H
	4	+
	5	+#include <linux/sched.h>
	6	+
	7	+#include <asm/cpufeature.h>
	8	+#include <asm/fpu/types.h>
	9	+
	10	+#include <asm/trace/fpu.h>
	11	+
	12	+extern void save_fpregs_to_fpstate(struct fpu *fpu);
	13	+extern void fpu__drop(struct fpu *fpu);
	14	+extern int fpu_clone(struct task_struct *dst, unsigned long clone_flags);
	15	+extern void fpu_flush_thread(void);
	16	+
	17	+/*
	18	+ * FPU state switching for scheduling.
	19	+ *
	20	+ * This is a two-stage process:
	21	+ *
	22	+ * - switch_fpu_prepare() saves the old state.
	23	+ * This is done within the context of the old process.
	24	+ *
	25	+ * - switch_fpu_finish() sets TIF_NEED_FPU_LOAD; the floating point state
	26	+ * will get loaded on return to userspace, or when the kernel needs it.
	27	+ *
	28	+ * If TIF_NEED_FPU_LOAD is cleared then the CPU's FPU registers
	29	+ * are saved in the current thread's FPU register state.
	30	+ *
	31	+ * If TIF_NEED_FPU_LOAD is set then CPU's FPU registers may not
	32	+ * hold current()'s FPU registers. It is required to load the
	33	+ * registers before returning to userland or using the content
	34	+ * otherwise.
	35	+ *
	36	+ * The FPU context is only stored/restored for a user task and
	37	+ * PF_KTHREAD is used to distinguish between kernel and user threads.
	38	+ */
	39	+static inline void switch_fpu_prepare(struct fpu *old_fpu, int cpu)
	40	+{
	41	+ if (cpu_feature_enabled(X86_FEATURE_FPU) &&
	42	+ !(current->flags & PF_KTHREAD)) {
	43	+ save_fpregs_to_fpstate(old_fpu);
	44	+ /*
	45	+ * The save operation preserved register state, so the
	46	+ * fpu_fpregs_owner_ctx is still @old_fpu. Store the
	47	+ * current CPU number in @old_fpu, so the next return
	48	+ * to user space can avoid the FPU register restore
	49	+ * when is returns on the same CPU and still owns the
	50	+ * context.
	51	+ */
	52	+ old_fpu->last_cpu = cpu;
	53	+
	54	+ trace_x86_fpu_regs_deactivated(old_fpu);
	55	+ }
	56	+}
	57	+
	58	+/*
	59	+ * Delay loading of the complete FPU state until the return to userland.
	60	+ * PKRU is handled separately.
	61	+ */
	62	+static inline void switch_fpu_finish(void)
	63	+{
	64	+ if (cpu_feature_enabled(X86_FEATURE_FPU))
	65	+ set_thread_flag(TIF_NEED_FPU_LOAD);
	66	+}
	67	+
	68	+#endif /* _ASM_X86_FPU_SCHED_H */

arch/x86/include/asm/fpu/signal.h+12−1

@@ -5,6 +5,11 @@ · 4 unmodified lines
5	5	#ifndef _ASM_X86_FPU_SIGNAL_H
6	6	#define _ASM_X86_FPU_SIGNAL_H
7	7
	8	+#include <linux/compat.h>
	9	+#include <linux/user.h>
	10	+
	11	+#include <asm/fpu/types.h>
	12	+
8	13	#ifdef CONFIG_X86_64
9	14	# include <uapi/asm/sigcontext.h>
10	15	# include <asm/user32.h>
@@ -31,6 +36,12 @@ fpu__alloc_mathframe(unsigned long sp, int ia32_frame, · 20 unmodified lines
31	36
32	37	unsigned long fpu__get_fpstate_size(void);
33	38
34		−extern void fpu__init_prepare_fx_sw_frame(void);
	39	+extern bool copy_fpstate_to_sigframe(void __user buf, void __user fp, int size);
	40	+extern void fpu__clear_user_states(struct fpu *fpu);
	41	+extern bool fpu__restore_sig(void __user *buf, int ia32_frame);
	42	+
	43	+extern void restore_fpregs_from_fpstate(struct fpstate *fpstate, u64 mask);
	44	+
	45	+extern bool copy_fpstate_to_sigframe(void __user buf, void __user fp, int size);
35	46
36	47	#endif /* _ASM_X86_FPU_SIGNAL_H */

arch/x86/include/asm/fpu/types.h+206−8

@@ -120,6 +120,9 @@ enum xfeature { · 119 unmodified lines
120	120	XFEATURE_RSRVD_COMP_13,
121	121	XFEATURE_RSRVD_COMP_14,
122	122	XFEATURE_LBR,
	123	+ XFEATURE_RSRVD_COMP_16,
	124	+ XFEATURE_XTILE_CFG,
	125	+ XFEATURE_XTILE_DATA,
123	126
124	127	XFEATURE_MAX,
125	128	};
@@ -136,12 +139,21 @@ enum xfeature { · 10 unmodified lines
136	139	#define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU)
137	140	#define XFEATURE_MASK_PASID (1 << XFEATURE_PASID)
138	141	#define XFEATURE_MASK_LBR (1 << XFEATURE_LBR)
	142	+#define XFEATURE_MASK_XTILE_CFG (1 << XFEATURE_XTILE_CFG)
	143	+#define XFEATURE_MASK_XTILE_DATA (1 << XFEATURE_XTILE_DATA)
139	144
140	145	#define XFEATURE_MASK_FPSSE (XFEATURE_MASK_FP \| XFEATURE_MASK_SSE)
141	146	#define XFEATURE_MASK_AVX512 (XFEATURE_MASK_OPMASK \
142	147	\| XFEATURE_MASK_ZMM_Hi256 \
143	148	\| XFEATURE_MASK_Hi16_ZMM)
144	149
	150	+#ifdef CONFIG_X86_64
	151	+# define XFEATURE_MASK_XTILE (XFEATURE_MASK_XTILE_DATA \
	152	+ \| XFEATURE_MASK_XTILE_CFG)
	153	+#else
	154	+# define XFEATURE_MASK_XTILE (0)
	155	+#endif
	156	+
145	157	#define FIRST_EXTENDED_XFEATURE XFEATURE_YMM
146	158
147	159	struct reg_128_bit {
@@ -153,6 +165,9 @@ struct reg_256_bit { · 5 unmodified lines
153	165	struct reg_512_bit {
154	166	u8 regbytes[512/8];
155	167	};
	168	+struct reg_1024_byte {
	169	+ u8 regbytes[1024];
	170	+};
156	171
157	172	/*
158	173	* State component 2:
@@ -255,6 +270,23 @@ struct arch_lbr_state { · 96 unmodified lines
255	270	u64 ler_to;
256	271	u64 ler_info;
257	272	struct lbr_entry entries[];
	273	+};
	274	+
	275	+/*
	276	+ * State component 17: 64-byte tile configuration register.
	277	+ */
	278	+struct xtile_cfg {
	279	+ u64 tcfg[8];
	280	+} __packed;
	281	+
	282	+/*
	283	+ * State component 18: 1KB tile data register.
	284	+ * Each register represents 16 64-byte rows of the matrix
	285	+ * data. But the number of registers depends on the actual
	286	+ * implementation.
	287	+ */
	288	+struct xtile_data {
	289	+ struct reg_1024_byte tmm;
258	290	} __packed;
259	291
260	292	/*
@@ -309,6 +341,91 @@ union fpregs_state { · 48 unmodified lines
309	341	u8 __padding[PAGE_SIZE];
310	342	};
311	343
	344	+struct fpstate {
	345	+ /* @kernel_size: The size of the kernel register image */
	346	+ unsigned int size;
	347	+
	348	+ /* @user_size: The size in non-compacted UABI format */
	349	+ unsigned int user_size;
	350	+
	351	+ /* @xfeatures: xfeatures for which the storage is sized */
	352	+ u64 xfeatures;
	353	+
	354	+ /* @user_xfeatures: xfeatures valid in UABI buffers */
	355	+ u64 user_xfeatures;
	356	+
	357	+ /* @xfd: xfeatures disabled to trap userspace use. */
	358	+ u64 xfd;
	359	+
	360	+ /* @is_valloc: Indicator for dynamically allocated state */
	361	+ unsigned int is_valloc : 1;
	362	+
	363	+ /* @is_guest: Indicator for guest state (KVM) */
	364	+ unsigned int is_guest : 1;
	365	+
	366	+ /*
	367	+ * @is_confidential: Indicator for KVM confidential mode.
	368	+ * The FPU registers are restored by the
	369	+ * vmentry firmware from encrypted guest
	370	+ * memory. On vmexit the FPU registers are
	371	+ * saved by firmware to encrypted guest memory
	372	+ * and the registers are scrubbed before
	373	+ * returning to the host. So there is no
	374	+ * content which is worth saving and restoring.
	375	+ * The fpstate has to be there so that
	376	+ * preemption and softirq FPU usage works
	377	+ * without special casing.
	378	+ */
	379	+ unsigned int is_confidential : 1;
	380	+
	381	+ /* @in_use: State is in use */
	382	+ unsigned int in_use : 1;
	383	+
	384	+ /* @regs: The register state union for all supported formats */
	385	+ union fpregs_state regs;
	386	+
	387	+ /* @regs is dynamically sized! Don't add anything after @regs! */
	388	+} __aligned(64);
	389	+
	390	+struct fpu_state_perm {
	391	+ /*
	392	+ * @__state_perm:
	393	+ *
	394	+ * This bitmap indicates the permission for state components, which
	395	+ * are available to a thread group. The permission prctl() sets the
	396	+ * enabled state bits in thread_group_leader()->thread.fpu.
	397	+ *
	398	+ * All run time operations use the per thread information in the
	399	+ * currently active fpu.fpstate which contains the xfeature masks
	400	+ * and sizes for kernel and user space.
	401	+ *
	402	+ * This master permission field is only to be used when
	403	+ * task.fpu.fpstate based checks fail to validate whether the task
	404	+ * is allowed to expand it's xfeatures set which requires to
	405	+ * allocate a larger sized fpstate buffer.
	406	+ *
	407	+ * Do not access this field directly. Use the provided helper
	408	+ * function. Unlocked access is possible for quick checks.
	409	+ */
	410	+ u64 __state_perm;
	411	+
	412	+ /*
	413	+ * @__state_size:
	414	+ *
	415	+ * The size required for @__state_perm. Only valid to access
	416	+ * with sighand locked.
	417	+ */
	418	+ unsigned int __state_size;
	419	+
	420	+ /*
	421	+ * @__user_state_size:
	422	+ *
	423	+ * The size required for @__state_perm user part. Only valid to
	424	+ * access with sighand locked.
	425	+ */
	426	+ unsigned int __user_state_size;
	427	+};
	428	+
312	429	/*
313	430	* Highest level per task FPU state data structure that
314	431	* contains the FPU register state plus various FPU
@@ -337,19 +454,100 @@ struct fpu { · 22 unmodified lines
337	454	unsigned long avx512_timestamp;
338	455
339	456	/*
340		− * @state:
	457	+ * @fpstate:
	458	+ *
	459	+ * Pointer to the active struct fpstate. Initialized to
	460	+ * point at @__fpstate below.
	461	+ */
	462	+ struct fpstate *fpstate;
	463	+
	464	+ /*
	465	+ * @__task_fpstate:
	466	+ *
	467	+ * Pointer to an inactive struct fpstate. Initialized to NULL. Is
	468	+ * used only for KVM support to swap out the regular task fpstate.
	469	+ */
	470	+ struct fpstate *__task_fpstate;
	471	+
	472	+ /*
	473	+ * @perm:
	474	+ *
	475	+ * Permission related information
	476	+ */
	477	+ struct fpu_state_perm perm;
	478	+
	479	+ /*
	480	+ * @__fpstate:
341	481	*
342		− * In-memory copy of all FPU registers that we save/restore
343		− * over context switches. If the task is using the FPU then
344		− * the registers in the FPU are more recent than this state
345		− * copy. If the task context-switches away then they get
346		− * saved here and represent the FPU state.
	482	+ * Initial in-memory storage for FPU registers which are saved in
	483	+ * context switch and when the kernel uses the FPU. The registers
	484	+ * are restored from this storage on return to user space if they
	485	+ * are not longer containing the tasks FPU register state.
347	486	*/
348		− union fpregs_state state;
	487	+ struct fpstate __fpstate;
349	488	/*
350		− * WARNING: 'state' is dynamically-sized. Do not put
	489	+ * WARNING: '__fpstate' is dynamically-sized. Do not put
351	490	* anything after it here.
352	491	*/
353	492	};
354	493
	494	+/*
	495	+ * Guest pseudo FPU container
	496	+ */
	497	+struct fpu_guest {
	498	+ /*
	499	+ * @fpstate: Pointer to the allocated guest fpstate
	500	+ */
	501	+ struct fpstate *fpstate;
	502	+};
	503	+
	504	+/*
	505	+ * FPU state configuration data. Initialized at boot time. Read only after init.
	506	+ */
	507	+struct fpu_state_config {
	508	+ /*
	509	+ * @max_size:
	510	+ *
	511	+ * The maximum size of the register state buffer. Includes all
	512	+ * supported features except independent managed features.
	513	+ */
	514	+ unsigned int max_size;
	515	+
	516	+ /*
	517	+ * @default_size:
	518	+ *
	519	+ * The default size of the register state buffer. Includes all
	520	+ * supported features except independent managed features and
	521	+ * features which have to be requested by user space before usage.
	522	+ */
	523	+ unsigned int default_size;
	524	+
	525	+ /*
	526	+ * @max_features:
	527	+ *
	528	+ * The maximum supported features bitmap. Does not include
	529	+ * independent managed features.
	530	+ */
	531	+ u64 max_features;
	532	+
	533	+ /*
	534	+ * @default_features:
	535	+ *
	536	+ * The default supported features bitmap. Does not include
	537	+ * independent managed features and features which have to
	538	+ * be requested by user space before usage.
	539	+ */
	540	+ u64 default_features;
	541	+ /*
	542	+ * @legacy_features:
	543	+ *
	544	+ * Features which can be reported back to user space
	545	+ * even without XSAVE support, i.e. legacy features FP + SSE
	546	+ */
	547	+ u64 legacy_features;
	548	+};
	549	+
	550	+/* FPU state configuration information */
	551	+extern struct fpu_state_config fpu_kernel_cfg, fpu_user_cfg;
	552	+
355	553	#endif /* _ASM_X86_FPU_H */

arch/x86/include/asm/fpu/xcr.h+0−11

@@ -2,17 +2,6 @@ · 1 unmodified line
2	2	#ifndef _ASM_X86_FPU_XCR_H
3	3	#define _ASM_X86_FPU_XCR_H
4	4
5		−/*
6		− * MXCSR and XCR definitions:
7		− */
8		−
9		−static inline void ldmxcsr(u32 mxcsr)
10		−{
11		− asm volatile("ldmxcsr %0" :: "m" (mxcsr));
12		−}
13		−
14		−extern unsigned int mxcsr_feature_mask;
15		−
16	5	#define XCR_XFEATURE_ENABLED_MASK 0x00000000
17	6
18	7	static inline u64 xgetbv(u32 index)

arch/x86/include/asm/fpu/xstate.h+30−60

@@ -14,6 +14,8 @@ · 13 unmodified lines
14	14
15	15	#define XSTATE_CPUID 0x0000000d
16	16
	17	+#define TILE_CPUID 0x0000001d
	18	+
17	19	#define FXSAVE_SIZE 512
18	20
19	21	#define XSAVE_HDR_SIZE 64
@@ -33,7 +35,8 @@ · 13 unmodified lines
33	35	XFEATURE_MASK_Hi16_ZMM \| \
34	36	XFEATURE_MASK_PKRU \| \
35	37	XFEATURE_MASK_BNDREGS \| \
36		− XFEATURE_MASK_BNDCSR)
	38	+ XFEATURE_MASK_BNDCSR \| \
	39	+ XFEATURE_MASK_XTILE)
37	40
38	41	/*
39	42	* Features which are restored when returning to user space.
@@ -43,6 +46,9 @@ · 3 unmodified lines
43	46	#define XFEATURE_MASK_USER_RESTORE \
44	47	(XFEATURE_MASK_USER_SUPPORTED & ~XFEATURE_MASK_PKRU)
45	48
	49	+/* Features which are dynamically enabled for a process on request */
	50	+#define XFEATURE_MASK_USER_DYNAMIC XFEATURE_MASK_XTILE_DATA
	51	+
46	52	/* All currently supported supervisor features */
47	53	#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID)
48	54
@@ -78,78 +84,42 @@ · 29 unmodified lines
78	84	XFEATURE_MASK_INDEPENDENT \| \
79	85	XFEATURE_MASK_SUPERVISOR_UNSUPPORTED)
80	86
81		−#ifdef CONFIG_X86_64
82		−#define REX_PREFIX "0x48, "
83		−#else
84		−#define REX_PREFIX
85		−#endif
86		−
87		−extern u64 xfeatures_mask_all;
88		−
89		−static inline u64 xfeatures_mask_supervisor(void)
90		−{
91		− return xfeatures_mask_all & XFEATURE_MASK_SUPERVISOR_SUPPORTED;
92		−}
93		−
94	87	/*
95		− * The xfeatures which are enabled in XCR0 and expected to be in ptrace
96		− * buffers and signal frames.
	88	+ * The feature mask required to restore FPU state:
	89	+ * - All user states which are not eagerly switched in switch_to()/exec()
	90	+ * - The suporvisor states
97	91	*/
98		−static inline u64 xfeatures_mask_uabi(void)
99		−{
100		− return xfeatures_mask_all & XFEATURE_MASK_USER_SUPPORTED;
101		−}
102		−
103		−/*
104		− * The xfeatures which are restored by the kernel when returning to user
105		− * mode. This is not necessarily the same as xfeatures_mask_uabi() as the
106		− * kernel does not manage all XCR0 enabled features via xsave/xrstor as
107		− * some of them have to be switched eagerly on context switch and exec().
108		− */
109		−static inline u64 xfeatures_mask_restore_user(void)
110		−{
111		− return xfeatures_mask_all & XFEATURE_MASK_USER_RESTORE;
112		−}
113		−
114		−/*
115		− * Like xfeatures_mask_restore_user() but additionally restors the
116		− * supported supervisor states.
117		− */
118		−static inline u64 xfeatures_mask_fpstate(void)
119		−{
120		− return xfeatures_mask_all & \
121		− (XFEATURE_MASK_USER_RESTORE \| XFEATURE_MASK_SUPERVISOR_SUPPORTED);
122		−}
123		−
124		−static inline u64 xfeatures_mask_independent(void)
125		−{
126		− if (!boot_cpu_has(X86_FEATURE_ARCH_LBR))
127		− return XFEATURE_MASK_INDEPENDENT & ~XFEATURE_MASK_LBR;
128		−
129		− return XFEATURE_MASK_INDEPENDENT;
130		−}
	92	+#define XFEATURE_MASK_FPSTATE (XFEATURE_MASK_USER_RESTORE \| \
	93	+ XFEATURE_MASK_SUPERVISOR_SUPPORTED)
131	94
132	95	extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS];
133	96
134	97	extern void __init update_regset_xstate_info(unsigned int size,
135	98	u64 xstate_mask);
136	99
137		−void get_xsave_addr(struct xregs_state xsave, int xfeature_nr);
138	100	int xfeature_size(int xfeature_nr);
139		−int copy_uabi_from_kernel_to_xstate(struct xregs_state xsave, const void kbuf);
140		−int copy_sigframe_from_user_to_xstate(struct xregs_state xsave, const void __user ubuf);
141	101
142	102	void xsaves(struct xregs_state *xsave, u64 mask);
143	103	void xrstors(struct xregs_state *xsave, u64 mask);
144	104
145		−enum xstate_copy_mode {
146		− XSTATE_COPY_FP,
147		− XSTATE_COPY_FX,
148		− XSTATE_COPY_XSAVE,
149		−};
	105	+int xfd_enable_feature(u64 xfd_err);
	106	+
	107	+#ifdef CONFIG_X86_64
	108	+DECLARE_STATIC_KEY_FALSE(__fpu_state_size_dynamic);
	109	+#endif
	110	+
	111	+#ifdef CONFIG_X86_64
	112	+DECLARE_STATIC_KEY_FALSE(__fpu_state_size_dynamic);
150	113
151		−struct membuf;
152		−void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk,
153		− enum xstate_copy_mode mode);
	114	+static __always_inline __pure bool fpu_state_size_dynamic(void)
	115	+{
	116	+ return static_branch_unlikely(&__fpu_state_size_dynamic);
	117	+}
	118	+#else
	119	+static __always_inline __pure bool fpu_state_size_dynamic(void)
	120	+{
	121	+ return false;
	122	+}
	123	+#endif
154	124
155	125	#endif

arch/x86/include/asm/kvm_host.h+2−5

@@ -691,11 +691,10 @@ struct kvm_vcpu_arch { · 690 unmodified lines
691	691	*
692	692	* Note that while the PKRU state lives inside the fpu registers,
693	693	* it is switched out separately at VMENTER and VMEXIT time. The
694		− * "guest_fpu" state here contains the guest FPU context, with the
	694	+ * "guest_fpstate" state here contains the guest FPU context, with the
695	695	* host PRKU bits.
696	696	*/
697		− struct fpu *user_fpu;
698		− struct fpu *guest_fpu;
	697	+ struct fpu_guest guest_fpu;
699	698
700	699	u64 xcr0;
701	700	u64 guest_supported_xcr0;
@@ -1686,8 +1685,6 @@ void kvm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector); · 984 unmodified lines
1686	1685	int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index,
1687	1686	int reason, bool has_error_code, u32 error_code);
1688	1687
1689		−void kvm_free_guest_fpu(struct kvm_vcpu *vcpu);
1690		−
1691	1688	void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0);
1692	1689	void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned long cr4);
1693	1690	int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0);

arch/x86/include/asm/msr-index.h+2−0

@@ -625,6 +625,8 @@ · 624 unmodified lines
625	625
626	626	#define MSR_IA32_BNDCFGS_RSVD 0x00000ffc
627	627
	628	+#define MSR_IA32_XFD 0x000001c4
	629	+#define MSR_IA32_XFD_ERR 0x000001c5
628	630	#define MSR_IA32_XSS 0x00000da0
629	631
630	632	#define MSR_IA32_APICBASE 0x0000001b

arch/x86/include/asm/msr.h+2−2

@@ -92,7 +92,7 @@ static __always_inline unsigned long long __rdmsr(unsigned int msr) · 91 unmodified lines
92	92
93	93	asm volatile("1: rdmsr\n"
94	94	"2:\n"
95		− _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_rdmsr_unsafe)
	95	+ _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_RDMSR)
96	96	: EAX_EDX_RET(val, low, high) : "c" (msr));
97	97
98	98	return EAX_EDX_VAL(val, low, high);
@@ -102,7 +102,7 @@ static __always_inline void __wrmsr(unsigned int msr, u32 low, u32 high) · 3 unmodified lines
102	102	{
103	103	asm volatile("1: wrmsr\n"
104	104	"2:\n"
105		− _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_wrmsr_unsafe)
	105	+ _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_WRMSR)
106	106	: : "c" (msr), "a"(low), "d" (high) : "memory");
107	107	}
108	108

arch/x86/include/asm/pkru.h+1−1

@@ -2,7 +2,7 @@ · 1 unmodified line
2	2	#ifndef _ASM_X86_PKRU_H
3	3	#define _ASM_X86_PKRU_H
4	4
5		−#include <asm/fpu/xstate.h>
	5	+#include <asm/cpufeature.h>
6	6
7	7	#define PKRU_AD_BIT 0x1
8	8	#define PKRU_WD_BIT 0x2

arch/x86/include/asm/processor.h+3−6

@@ -461,9 +461,6 @@ DECLARE_PER_CPU(struct irq_stack *, hardirq_stack_ptr); · 460 unmodified lines
461	461	DECLARE_PER_CPU(struct irq_stack *, softirq_stack_ptr);
462	462	#endif /* !X86_64 */
463	463
464		−extern unsigned int fpu_kernel_xstate_size;
465		−extern unsigned int fpu_user_xstate_size;
466		−
467	464	struct perf_event;
468	465
469	466	struct thread_struct {
@@ -537,12 +534,12 @@ struct thread_struct { · 67 unmodified lines
537	534	*/
538	535	};
539	536
540		−/* Whitelist the FPU state from the task_struct for hardened usercopy. */
	537	+extern void fpu_thread_struct_whitelist(unsigned long offset, unsigned long size);
	538	+
541	539	static inline void arch_thread_struct_whitelist(unsigned long *offset,
542	540	unsigned long *size)
543	541	{
544		− *offset = offsetof(struct thread_struct, fpu.state);
545		− *size = fpu_kernel_xstate_size;
	542	+ fpu_thread_struct_whitelist(offset, size);
546	543	}
547	544
548	545	static inline void

arch/x86/include/asm/proto.h+1−1

@@ -40,6 +40,6 @@ void x86_report_nx(void); · 39 unmodified lines
40	40	extern int reboot_force;
41	41
42	42	long do_arch_prctl_common(struct task_struct *task, int option,
43		− unsigned long cpuid_enabled);
	43	+ unsigned long arg2);
44	44
45	45	#endif /* _ASM_X86_PROTO_H */

arch/x86/include/asm/segment.h+1−1

@@ -339,7 +339,7 @@ static inline void __loadsegment_fs(unsigned short value) · 338 unmodified lines
339	339	"1: movw %0, %%fs \n"
340	340	"2: \n"
341	341
342		− _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_clear_fs)
	342	+ _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_CLEAR_FS)
343	343
344	344	: : "rm" (value) : "memory");
345	345	}

arch/x86/include/asm/trace/fpu.h+2−2

@@ -22,8 +22,8 @@ DECLARE_EVENT_CLASS(x86_fpu, · 21 unmodified lines
22	22	__entry->fpu = fpu;
23	23	__entry->load_fpu = test_thread_flag(TIF_NEED_FPU_LOAD);
24	24	if (boot_cpu_has(X86_FEATURE_OSXSAVE)) {
25		− __entry->xfeatures = fpu->state.xsave.header.xfeatures;
26		− __entry->xcomp_bv = fpu->state.xsave.header.xcomp_bv;
	25	+ __entry->xfeatures = fpu->fpstate->regs.xsave.header.xfeatures;
	26	+ __entry->xcomp_bv = fpu->fpstate->regs.xsave.header.xcomp_bv;
27	27	}
28	28	),
29	29	TP_printk("x86/fpu: %p load: %d xfeatures: %llx xcomp_bv: %llx",

arch/x86/include/uapi/asm/prctl.h+4−0

@@ -10,6 +10,10 @@ · 9 unmodified lines
10	10	#define ARCH_GET_CPUID 0x1011
11	11	#define ARCH_SET_CPUID 0x1012
12	12
	13	+#define ARCH_GET_XCOMP_SUPP 0x1021
	14	+#define ARCH_GET_XCOMP_PERM 0x1022
	15	+#define ARCH_REQ_XCOMP_PERM 0x1023
	16	+
13	17	#define ARCH_MAP_VDSO_X32 0x2001
14	18	#define ARCH_MAP_VDSO_32 0x2002
15	19	#define ARCH_MAP_VDSO_64 0x2003

arch/x86/kernel/cpu/bugs.c+1−1

@@ -22,7 +22,7 @@ · 21 unmodified lines
22	22	#include <asm/bugs.h>
23	23	#include <asm/processor.h>
24	24	#include <asm/processor-flags.h>
25		−#include <asm/fpu/internal.h>
	25	+#include <asm/fpu/api.h>
26	26	#include <asm/msr.h>
27	27	#include <asm/vmx.h>
28	28	#include <asm/paravirt.h>

arch/x86/kernel/cpu/common.c+1−1

@@ -42,7 +42,7 @@ · 41 unmodified lines
42	42	#include <asm/setup.h>
43	43	#include <asm/apic.h>
44	44	#include <asm/desc.h>
45		−#include <asm/fpu/internal.h>
	45	+#include <asm/fpu/api.h>
46	46	#include <asm/mtrr.h>
47	47	#include <asm/hwcap2.h>
48	48	#include <linux/numa.h>

arch/x86/kernel/cpu/cpuid-deps.c+2−0

@@ -75,6 +75,8 @@ static const struct cpuid_dep cpuid_deps[] = { · 74 unmodified lines
75	75	{ X86_FEATURE_SGX_LC, X86_FEATURE_SGX },
76	76	{ X86_FEATURE_SGX1, X86_FEATURE_SGX },
77	77	{ X86_FEATURE_SGX2, X86_FEATURE_SGX1 },
	78	+ { X86_FEATURE_XFD, X86_FEATURE_XSAVES },
	79	+ { X86_FEATURE_AMX_TILE, X86_FEATURE_XFD },
78	80	{}
79	81	};
80	82

arch/x86/kernel/cpu/mce/core.c+11−29

@@ -373,13 +373,16 @@ static int msr_to_offset(u32 msr) · 372 unmodified lines
373	373	return -1;
374	374	}
375	375
376		−__visible bool ex_handler_rdmsr_fault(const struct exception_table_entry *fixup,
377		− struct pt_regs *regs, int trapnr,
378		− unsigned long error_code,
379		− unsigned long fault_addr)
	376	+void ex_handler_msr_mce(struct pt_regs *regs, bool wrmsr)
380	377	{
381		− pr_emerg("MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n",
382		− (unsigned int)regs->cx, regs->ip, (void *)regs->ip);
	378	+ if (wrmsr) {
	379	+ pr_emerg("MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n",
	380	+ (unsigned int)regs->cx, (unsigned int)regs->dx, (unsigned int)regs->ax,
	381	+ regs->ip, (void *)regs->ip);
	382	+ } else {
	383	+ pr_emerg("MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n",
	384	+ (unsigned int)regs->cx, regs->ip, (void *)regs->ip);
	385	+ }
383	386
384	387	show_stack_regs(regs);
385	388
@@ -387,8 +390,6 @@ __visible bool ex_handler_rdmsr_fault(const struct exception_table_entry *fixup, · 1 unmodified line
387	390
388	391	while (true)
389	392	cpu_relax();
390		−
391		− return true;
392	393	}
393	394
394	395	/* MSR access wrappers used for error injection */
@@ -420,32 +421,13 @@ static noinstr u64 mce_rdmsrl(u32 msr) · 25 unmodified lines
420	421	*/
421	422	asm volatile("1: rdmsr\n"
422	423	"2:\n"
423		− _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_rdmsr_fault)
	424	+ _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_RDMSR_IN_MCE)
424	425	: EAX_EDX_RET(val, low, high) : "c" (msr));
425	426
426	427
427	428	return EAX_EDX_VAL(val, low, high);
428	429	}
429	430
430		−__visible bool ex_handler_wrmsr_fault(const struct exception_table_entry *fixup,
431		− struct pt_regs *regs, int trapnr,
432		− unsigned long error_code,
433		− unsigned long fault_addr)
434		−{
435		− pr_emerg("MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n",
436		− (unsigned int)regs->cx, (unsigned int)regs->dx, (unsigned int)regs->ax,
437		− regs->ip, (void *)regs->ip);
438		−
439		− show_stack_regs(regs);
440		−
441		− panic("MCA architectural violation!\n");
442		−
443		− while (true)
444		− cpu_relax();
445		−
446		− return true;
447		−}
448		−
449	431	static noinstr void mce_wrmsrl(u32 msr, u64 v)
450	432	{
451	433	u32 low, high;
@@ -470,7 +452,7 @@ static noinstr void mce_wrmsrl(u32 msr, u64 v) · 18 unmodified lines
470	452	/* See comment in mce_rdmsrl() */
471	453	asm volatile("1: wrmsr\n"
472	454	"2:\n"
473		− _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_wrmsr_fault)
	455	+ _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_WRMSR_IN_MCE)
474	456	: : "c" (msr), "a"(low), "d" (high) : "memory");
475	457	}
476	458

arch/x86/kernel/cpu/mce/internal.h+2−12

@@ -61,7 +61,7 @@ static inline void cmci_disable_bank(int bank) { } · 60 unmodified lines
61	61	static inline void intel_init_cmci(void) { }
62	62	static inline void intel_init_lmce(void) { }
63	63	static inline void intel_clear_lmce(void) { }
64		−static inline bool intel_filter_mce(struct mce *m) { return false; };
	64	+static inline bool intel_filter_mce(struct mce *m) { return false; }
65	65	#endif
66	66
67	67	void mce_timer_kick(unsigned long interval);
@@ -183,17 +183,7 @@ extern bool filter_mce(struct mce *m); · 115 unmodified lines
183	183	#ifdef CONFIG_X86_MCE_AMD
184	184	extern bool amd_filter_mce(struct mce *m);
185	185	#else
186		−static inline bool amd_filter_mce(struct mce *m) { return false; };
	186	+static inline bool amd_filter_mce(struct mce *m) { return false; }
187	187	#endif
188	188
189		−__visible bool ex_handler_rdmsr_fault(const struct exception_table_entry *fixup,
190		− struct pt_regs *regs, int trapnr,
191		− unsigned long error_code,
192		− unsigned long fault_addr);
193		−
194		−__visible bool ex_handler_wrmsr_fault(const struct exception_table_entry *fixup,
195		− struct pt_regs *regs, int trapnr,
196		− unsigned long error_code,
197		− unsigned long fault_addr);
198		−
199	189	#endif /* __X86_MCE_INTERNAL_H__ */

arch/x86/kernel/cpu/mce/severity.c+11−11

@@ -265,25 +265,25 @@ static bool is_copy_from_user(struct pt_regs *regs) · 264 unmodified lines
265	265	*/
266	266	static int error_context(struct mce m, struct pt_regs regs)
267	267	{
268		− enum handler_type t;
269		−
270	268	if ((m->cs & 3) == 3)
271	269	return IN_USER;
272	270	if (!mc_recoverable(m->mcgstatus))
273	271	return IN_KERNEL;
274	272
275		− t = ex_get_fault_handler_type(m->ip);
276		− if (t == EX_HANDLER_FAULT) {
277		− m->kflags \|= MCE_IN_KERNEL_RECOV;
278		− return IN_KERNEL_RECOV;
279		− }
280		− if (t == EX_HANDLER_UACCESS && regs && is_copy_from_user(regs)) {
281		− m->kflags \|= MCE_IN_KERNEL_RECOV;
	273	+ switch (ex_get_fixup_type(m->ip)) {
	274	+ case EX_TYPE_UACCESS:
	275	+ case EX_TYPE_COPY:
	276	+ if (!regs \|\| !is_copy_from_user(regs))
	277	+ return IN_KERNEL;
282	278	m->kflags \|= MCE_IN_KERNEL_COPYIN;
	279	+ fallthrough;
	280	+ case EX_TYPE_FAULT_MCE_SAFE:
	281	+ case EX_TYPE_DEFAULT_MCE_SAFE:
	282	+ m->kflags \|= MCE_IN_KERNEL_RECOV;
283	283	return IN_KERNEL_RECOV;
	284	+ default:
	285	+ return IN_KERNEL;
284	286	}
285		−
286		− return IN_KERNEL;
287	287	}
288	288
289	289	static int mce_severity_amd_smca(struct mce *m, enum context err_ctx)

arch/x86/kernel/fpu/bugs.c+1−1

@@ -2,7 +2,7 @@ · 1 unmodified line
2	2	/*
3	3	* x86 FPU bug checks:
4	4	*/
5		−#include <asm/fpu/internal.h>
	5	+#include <asm/fpu/api.h>
6	6
7	7	/*
8	8	* Boot time CPU/FPU FDIV bug detection code:

arch/x86/kernel/fpu/context.h+83−0

@@ -0,0 +1,83 @@
	1	+/* SPDX-License-Identifier: GPL-2.0 */
	2	+#ifndef __X86_KERNEL_FPU_CONTEXT_H
	3	+#define __X86_KERNEL_FPU_CONTEXT_H
	4	+
	5	+#include <asm/fpu/xstate.h>
	6	+#include <asm/trace/fpu.h>
	7	+
	8	+/* Functions related to FPU context tracking */
	9	+
	10	+/*
	11	+ * The in-register FPU state for an FPU context on a CPU is assumed to be
	12	+ * valid if the fpu->last_cpu matches the CPU, and the fpu_fpregs_owner_ctx
	13	+ * matches the FPU.
	14	+ *
	15	+ * If the FPU register state is valid, the kernel can skip restoring the
	16	+ * FPU state from memory.
	17	+ *
	18	+ * Any code that clobbers the FPU registers or updates the in-memory
	19	+ * FPU state for a task MUST let the rest of the kernel know that the
	20	+ * FPU registers are no longer valid for this task.
	21	+ *
	22	+ * Either one of these invalidation functions is enough. Invalidate
	23	+ * a resource you control: CPU if using the CPU for something else
	24	+ * (with preemption disabled), FPU for the current task, or a task that
	25	+ * is prevented from running by the current task.
	26	+ */
	27	+static inline void __cpu_invalidate_fpregs_state(void)
	28	+{
	29	+ __this_cpu_write(fpu_fpregs_owner_ctx, NULL);
	30	+}
	31	+
	32	+static inline void __fpu_invalidate_fpregs_state(struct fpu *fpu)
	33	+{
	34	+ fpu->last_cpu = -1;
	35	+}
	36	+
	37	+static inline int fpregs_state_valid(struct fpu *fpu, unsigned int cpu)
	38	+{
	39	+ return fpu == this_cpu_read(fpu_fpregs_owner_ctx) && cpu == fpu->last_cpu;
	40	+}
	41	+
	42	+static inline void fpregs_deactivate(struct fpu *fpu)
	43	+{
	44	+ __this_cpu_write(fpu_fpregs_owner_ctx, NULL);
	45	+ trace_x86_fpu_regs_deactivated(fpu);
	46	+}
	47	+
	48	+static inline void fpregs_activate(struct fpu *fpu)
	49	+{
	50	+ __this_cpu_write(fpu_fpregs_owner_ctx, fpu);
	51	+ trace_x86_fpu_regs_activated(fpu);
	52	+}
	53	+
	54	+/* Internal helper for switch_fpu_return() and signal frame setup */
	55	+static inline void fpregs_restore_userregs(void)
	56	+{
	57	+ struct fpu *fpu = &current->thread.fpu;
	58	+ int cpu = smp_processor_id();
	59	+
	60	+ if (WARN_ON_ONCE(current->flags & PF_KTHREAD))
	61	+ return;
	62	+
	63	+ if (!fpregs_state_valid(fpu, cpu)) {
	64	+ /*
	65	+ * This restores _all_ xstate which has not been
	66	+ * established yet.
	67	+ *
	68	+ * If PKRU is enabled, then the PKRU value is already
	69	+ * correct because it was either set in switch_to() or in
	70	+ * flush_thread(). So it is excluded because it might be
	71	+ * not up to date in current->thread.fpu.xsave state.
	72	+ *
	73	+ * XFD state is handled in restore_fpregs_from_fpstate().
	74	+ */
	75	+ restore_fpregs_from_fpstate(fpu->fpstate, XFEATURE_MASK_FPSTATE);
	76	+
	77	+ fpregs_activate(fpu);
	78	+ fpu->last_cpu = cpu;
	79	+ }
	80	+ clear_thread_flag(TIF_NEED_FPU_LOAD);
	81	+}
	82	+
	83	+#endif

arch/x86/kernel/fpu/core.c+315−77

@@ -6,8 +6,9 @@ · 5 unmodified lines
6	6	* General FPU state handling cleanups
7	7	* Gareth Hughes <gareth@valinux.com>, May 2000
8	8	*/
9		−#include <asm/fpu/internal.h>
	9	+#include <asm/fpu/api.h>
10	10	#include <asm/fpu/regset.h>
	11	+#include <asm/fpu/sched.h>
11	12	#include <asm/fpu/signal.h>
12	13	#include <asm/fpu/types.h>
13	14	#include <asm/traps.h>
@@ -15,15 +16,30 @@ · 1 unmodified line
15	16
16	17	#include <linux/hardirq.h>
17	18	#include <linux/pkeys.h>
	19	+#include <linux/vmalloc.h>
	20	+
	21	+#include "context.h"
	22	+#include "internal.h"
	23	+#include "legacy.h"
	24	+#include "xstate.h"
18	25
19	26	#define CREATE_TRACE_POINTS
20	27	#include <asm/trace/fpu.h>
21	28
	29	+#ifdef CONFIG_X86_64
	30	+DEFINE_STATIC_KEY_FALSE(__fpu_state_size_dynamic);
	31	+DEFINE_PER_CPU(u64, xfd_state);
	32	+#endif
	33	+
	34	+/* The FPU state configuration data for kernel and user space */
	35	+struct fpu_state_config fpu_kernel_cfg __ro_after_init;
	36	+struct fpu_state_config fpu_user_cfg __ro_after_init;
	37	+
22	38	/*
23	39	* Represents the initial FPU state. It's mostly (but not completely) zeroes,
24	40	* depending on the FPU hardware format:
25	41	*/
26		−union fpregs_state init_fpstate __ro_after_init;
	42	+struct fpstate init_fpstate __ro_after_init;
27	43
28	44	/*
29	45	* Track whether the kernel is using the FPU state
@@ -83,7 +99,7 @@ bool irq_fpu_usable(void) · 53 unmodified lines
83	99	EXPORT_SYMBOL(irq_fpu_usable);
84	100
85	101	/*
86		− * Save the FPU register state in fpu->state. The register state is
	102	+ * Save the FPU register state in fpu->fpstate->regs. The register state is
87	103	* preserved.
88	104	*
89	105	* Must be called with fpregs_lock() held.
@@ -99,19 +115,19 @@ EXPORT_SYMBOL(irq_fpu_usable); · 9 unmodified lines
99	115	void save_fpregs_to_fpstate(struct fpu *fpu)
100	116	{
101	117	if (likely(use_xsave())) {
102		− os_xsave(&fpu->state.xsave);
	118	+ os_xsave(fpu->fpstate);
103	119
104	120	/*
105	121	* AVX512 state is tracked here because its use is
106	122	* known to slow the max clock speed of the core.
107	123	*/
108		− if (fpu->state.xsave.header.xfeatures & XFEATURE_MASK_AVX512)
	124	+ if (fpu->fpstate->regs.xsave.header.xfeatures & XFEATURE_MASK_AVX512)
109	125	fpu->avx512_timestamp = jiffies;
110	126	return;
111	127	}
112	128
113	129	if (likely(use_fxsr())) {
114		− fxsave(&fpu->state.fxsave);
	130	+ fxsave(&fpu->fpstate->regs.fxsave);
115	131	return;
116	132	}
117	133
@@ -119,12 +135,11 @@ void save_fpregs_to_fpstate(struct fpu *fpu) · 1 unmodified line
119	135	* Legacy FPU register saving, FNSAVE always clears FPU registers,
120	136	* so we have to reload them from the memory state.
121	137	*/
122		− asm volatile("fnsave %[fp]; fwait" : [fp] "=m" (fpu->state.fsave));
123		− frstor(&fpu->state.fsave);
	138	+ asm volatile("fnsave %[fp]; fwait" : [fp] "=m" (fpu->fpstate->regs.fsave));
	139	+ frstor(&fpu->fpstate->regs.fsave);
124	140	}
125		−EXPORT_SYMBOL(save_fpregs_to_fpstate);
126	141
127		−void __restore_fpregs_from_fpstate(union fpregs_state *fpstate, u64 mask)
	142	+void restore_fpregs_from_fpstate(struct fpstate *fpstate, u64 mask)
128	143	{
129	144	/*
130	145	* AMD K7/K8 and later CPUs up to Zen don't save/restore
@@ -141,15 +156,181 @@ void __restore_fpregs_from_fpstate(union fpregs_state *fpstate, u64 mask) · 10 unmodified lines
141	156	}
142	157
143	158	if (use_xsave()) {
144		− os_xrstor(&fpstate->xsave, mask);
	159	+ /*
	160	+ * Dynamically enabled features are enabled in XCR0, but
	161	+ * usage requires also that the corresponding bits in XFD
	162	+ * are cleared. If the bits are set then using a related
	163	+ * instruction will raise #NM. This allows to do the
	164	+ * allocation of the larger FPU buffer lazy from #NM or if
	165	+ * the task has no permission to kill it which would happen
	166	+ * via #UD if the feature is disabled in XCR0.
	167	+ *
	168	+ * XFD state is following the same life time rules as
	169	+ * XSTATE and to restore state correctly XFD has to be
	170	+ * updated before XRSTORS otherwise the component would
	171	+ * stay in or go into init state even if the bits are set
	172	+ * in fpstate::regs::xsave::xfeatures.
	173	+ */
	174	+ xfd_update_state(fpstate);
	175	+
	176	+ /*
	177	+ * Restoring state always needs to modify all features
	178	+ * which are in @mask even if the current task cannot use
	179	+ * extended features.
	180	+ *
	181	+ * So fpstate->xfeatures cannot be used here, because then
	182	+ * a feature for which the task has no permission but was
	183	+ * used by the previous task would not go into init state.
	184	+ */
	185	+ mask = fpu_kernel_cfg.max_features & mask;
	186	+
	187	+ os_xrstor(fpstate, mask);
145	188	} else {
146	189	if (use_fxsr())
147		− fxrstor(&fpstate->fxsave);
	190	+ fxrstor(&fpstate->regs.fxsave);
148	191	else
149		− frstor(&fpstate->fsave);
	192	+ frstor(&fpstate->regs.fsave);
150	193	}
151	194	}
152		−EXPORT_SYMBOL_GPL(__restore_fpregs_from_fpstate);
	195	+
	196	+void fpu_reset_from_exception_fixup(void)
	197	+{
	198	+ restore_fpregs_from_fpstate(&init_fpstate, XFEATURE_MASK_FPSTATE);
	199	+}
	200	+
	201	+#if IS_ENABLED(CONFIG_KVM)
	202	+static void __fpstate_reset(struct fpstate *fpstate);
	203	+
	204	+bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu)
	205	+{
	206	+ struct fpstate *fpstate;
	207	+ unsigned int size;
	208	+
	209	+ size = fpu_user_cfg.default_size + ALIGN(offsetof(struct fpstate, regs), 64);
	210	+ fpstate = vzalloc(size);
	211	+ if (!fpstate)
	212	+ return false;
	213	+
	214	+ __fpstate_reset(fpstate);
	215	+ fpstate_init_user(fpstate);
	216	+ fpstate->is_valloc = true;
	217	+ fpstate->is_guest = true;
	218	+
	219	+ gfpu->fpstate = fpstate;
	220	+ return true;
	221	+}
	222	+EXPORT_SYMBOL_GPL(fpu_alloc_guest_fpstate);
	223	+
	224	+void fpu_free_guest_fpstate(struct fpu_guest *gfpu)
	225	+{
	226	+ struct fpstate *fps = gfpu->fpstate;
	227	+
	228	+ if (!fps)
	229	+ return;
	230	+
	231	+ if (WARN_ON_ONCE(!fps->is_valloc \|\| !fps->is_guest \|\| fps->in_use))
	232	+ return;
	233	+
	234	+ gfpu->fpstate = NULL;
	235	+ vfree(fps);
	236	+}
	237	+EXPORT_SYMBOL_GPL(fpu_free_guest_fpstate);
	238	+
	239	+int fpu_swap_kvm_fpstate(struct fpu_guest *guest_fpu, bool enter_guest)
	240	+{
	241	+ struct fpstate *guest_fps = guest_fpu->fpstate;
	242	+ struct fpu *fpu = &current->thread.fpu;
	243	+ struct fpstate *cur_fps = fpu->fpstate;
	244	+
	245	+ fpregs_lock();
	246	+ if (!cur_fps->is_confidential && !test_thread_flag(TIF_NEED_FPU_LOAD))
	247	+ save_fpregs_to_fpstate(fpu);
	248	+
	249	+ /* Swap fpstate */
	250	+ if (enter_guest) {
	251	+ fpu->__task_fpstate = cur_fps;
	252	+ fpu->fpstate = guest_fps;
	253	+ guest_fps->in_use = true;
	254	+ } else {
	255	+ guest_fps->in_use = false;
	256	+ fpu->fpstate = fpu->__task_fpstate;
	257	+ fpu->__task_fpstate = NULL;
	258	+ }
	259	+
	260	+ cur_fps = fpu->fpstate;
	261	+
	262	+ if (!cur_fps->is_confidential) {
	263	+ /* Includes XFD update */
	264	+ restore_fpregs_from_fpstate(cur_fps, XFEATURE_MASK_FPSTATE);
	265	+ } else {
	266	+ /*
	267	+ * XSTATE is restored by firmware from encrypted
	268	+ * memory. Make sure XFD state is correct while
	269	+ * running with guest fpstate
	270	+ */
	271	+ xfd_update_state(cur_fps);
	272	+ }
	273	+
	274	+ fpregs_mark_activate();
	275	+ fpregs_unlock();
	276	+ return 0;
	277	+}
	278	+EXPORT_SYMBOL_GPL(fpu_swap_kvm_fpstate);
	279	+
	280	+void fpu_copy_guest_fpstate_to_uabi(struct fpu_guest gfpu, void buf,
	281	+ unsigned int size, u32 pkru)
	282	+{
	283	+ struct fpstate *kstate = gfpu->fpstate;
	284	+ union fpregs_state *ustate = buf;
	285	+ struct membuf mb = { .p = buf, .left = size };
	286	+
	287	+ if (cpu_feature_enabled(X86_FEATURE_XSAVE)) {
	288	+ __copy_xstate_to_uabi_buf(mb, kstate, pkru, XSTATE_COPY_XSAVE);
	289	+ } else {
	290	+ memcpy(&ustate->fxsave, &kstate->regs.fxsave,
	291	+ sizeof(ustate->fxsave));
	292	+ /* Make it restorable on a XSAVE enabled host */
	293	+ ustate->xsave.header.xfeatures = XFEATURE_MASK_FPSSE;
	294	+ }
	295	+}
	296	+EXPORT_SYMBOL_GPL(fpu_copy_guest_fpstate_to_uabi);
	297	+
	298	+int fpu_copy_uabi_to_guest_fpstate(struct fpu_guest gfpu, const void buf,
	299	+ u64 xcr0, u32 *vpkru)
	300	+{
	301	+ struct fpstate *kstate = gfpu->fpstate;
	302	+ const union fpregs_state *ustate = buf;
	303	+ struct pkru_state *xpkru;
	304	+ int ret;
	305	+
	306	+ if (!cpu_feature_enabled(X86_FEATURE_XSAVE)) {
	307	+ if (ustate->xsave.header.xfeatures & ~XFEATURE_MASK_FPSSE)
	308	+ return -EINVAL;
	309	+ if (ustate->fxsave.mxcsr & ~mxcsr_feature_mask)
	310	+ return -EINVAL;
	311	+ memcpy(&kstate->regs.fxsave, &ustate->fxsave, sizeof(ustate->fxsave));
	312	+ return 0;
	313	+ }
	314	+
	315	+ if (ustate->xsave.header.xfeatures & ~xcr0)
	316	+ return -EINVAL;
	317	+
	318	+ ret = copy_uabi_from_kernel_to_xstate(kstate, ustate);
	319	+ if (ret)
	320	+ return ret;
	321	+
	322	+ /* Retrieve PKRU if not in init state */
	323	+ if (kstate->regs.xsave.header.xfeatures & XFEATURE_MASK_PKRU) {
	324	+ xpkru = get_xsave_addr(&kstate->regs.xsave, XFEATURE_PKRU);
	325	+ *vpkru = xpkru->pkru;
	326	+ }
	327	+
	328	+ /* Ensure that XCOMP_BV is set up for XSAVES */
	329	+ xstate_init_xcomp_bv(&kstate->regs.xsave, kstate->xfeatures);
	330	+ return 0;
	331	+}
	332	+EXPORT_SYMBOL_GPL(fpu_copy_uabi_to_guest_fpstate);
	333	+#endif /* CONFIG_KVM */
153	334
154	335	void kernel_fpu_begin_mask(unsigned int kfpu_mask)
155	336	{
@@ -203,52 +384,88 @@ void fpu_sync_fpstate(struct fpu *fpu) · 47 unmodified lines
203	384	fpregs_unlock();
204	385	}
205	386
206		−static inline void fpstate_init_xstate(struct xregs_state *xsave)
	387	+static inline unsigned int init_fpstate_copy_size(void)
207	388	{
208		− /*
209		− * XRSTORS requires these bits set in xcomp_bv, or it will
210		− * trigger #GP:
211		− */
212		− xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT \| xfeatures_mask_all;
	389	+ if (!use_xsave())
	390	+ return fpu_kernel_cfg.default_size;
	391	+
	392	+ /* XSAVE(S) just needs the legacy and the xstate header part */
	393	+ return sizeof(init_fpstate.regs.xsave);
213	394	}
214	395
215		−static inline void fpstate_init_fxstate(struct fxregs_state *fx)
	396	+static inline void fpstate_init_fxstate(struct fpstate *fpstate)
216	397	{
217		− fx->cwd = 0x37f;
218		− fx->mxcsr = MXCSR_DEFAULT;
	398	+ fpstate->regs.fxsave.cwd = 0x37f;
	399	+ fpstate->regs.fxsave.mxcsr = MXCSR_DEFAULT;
219	400	}
220	401
221	402	/*
222	403	* Legacy x87 fpstate state init:
223	404	*/
224		−static inline void fpstate_init_fstate(struct fregs_state *fp)
	405	+static inline void fpstate_init_fstate(struct fpstate *fpstate)
225	406	{
226		− fp->cwd = 0xffff037fu;
227		− fp->swd = 0xffff0000u;
228		− fp->twd = 0xffffffffu;
229		− fp->fos = 0xffff0000u;
	407	+ fpstate->regs.fsave.cwd = 0xffff037fu;
	408	+ fpstate->regs.fsave.swd = 0xffff0000u;
	409	+ fpstate->regs.fsave.twd = 0xffffffffu;
	410	+ fpstate->regs.fsave.fos = 0xffff0000u;
230	411	}
231	412
232		−void fpstate_init(union fpregs_state *state)
	413	+/*
	414	+ * Used in two places:
	415	+ * 1) Early boot to setup init_fpstate for non XSAVE systems
	416	+ * 2) fpu_init_fpstate_user() which is invoked from KVM
	417	+ */
	418	+void fpstate_init_user(struct fpstate *fpstate)
233	419	{
234		− if (!static_cpu_has(X86_FEATURE_FPU)) {
235		− fpstate_init_soft(&state->soft);
	420	+ if (!cpu_feature_enabled(X86_FEATURE_FPU)) {
	421	+ fpstate_init_soft(&fpstate->regs.soft);
236	422	return;
237	423	}
238	424
239		− memset(state, 0, fpu_kernel_xstate_size);
	425	+ xstate_init_xcomp_bv(&fpstate->regs.xsave, fpstate->xfeatures);
240	426
241		− if (static_cpu_has(X86_FEATURE_XSAVES))
242		− fpstate_init_xstate(&state->xsave);
243		− if (static_cpu_has(X86_FEATURE_FXSR))
244		− fpstate_init_fxstate(&state->fxsave);
	427	+ if (cpu_feature_enabled(X86_FEATURE_FXSR))
	428	+ fpstate_init_fxstate(fpstate);
245	429	else
246		− fpstate_init_fstate(&state->fsave);
	430	+ fpstate_init_fstate(fpstate);
	431	+}
	432	+
	433	+static void __fpstate_reset(struct fpstate *fpstate)
	434	+{
	435	+ /* Initialize sizes and feature masks */
	436	+ fpstate->size = fpu_kernel_cfg.default_size;
	437	+ fpstate->user_size = fpu_user_cfg.default_size;
	438	+ fpstate->xfeatures = fpu_kernel_cfg.default_features;
	439	+ fpstate->user_xfeatures = fpu_user_cfg.default_features;
	440	+ fpstate->xfd = init_fpstate.xfd;
	441	+}
	442	+
	443	+void fpstate_reset(struct fpu *fpu)
	444	+{
	445	+ /* Set the fpstate pointer to the default fpstate */
	446	+ fpu->fpstate = &fpu->__fpstate;
	447	+ __fpstate_reset(fpu->fpstate);
	448	+
	449	+ /* Initialize the permission related info in fpu */
	450	+ fpu->perm.__state_perm = fpu_kernel_cfg.default_features;
	451	+ fpu->perm.__state_size = fpu_kernel_cfg.default_size;
	452	+ fpu->perm.__user_state_size = fpu_user_cfg.default_size;
	453	+}
	454	+
	455	+static inline void fpu_inherit_perms(struct fpu *dst_fpu)
	456	+{
	457	+ if (fpu_state_size_dynamic()) {
	458	+ struct fpu *src_fpu = &current->group_leader->thread.fpu;
	459	+
	460	+ spin_lock_irq(&current->sighand->siglock);
	461	+ /* Fork also inherits the permissions of the parent */
	462	+ dst_fpu->perm = src_fpu->perm;
	463	+ spin_unlock_irq(&current->sighand->siglock);
	464	+ }
247	465	}
248		−EXPORT_SYMBOL_GPL(fpstate_init);
249	466
250	467	/* Clone current's FPU state on fork */
251		−int fpu_clone(struct task_struct *dst)
	468	+int fpu_clone(struct task_struct *dst, unsigned long clone_flags)
252	469	{
253	470	struct fpu *src_fpu = &current->thread.fpu;
254	471	struct fpu *dst_fpu = &dst->thread.fpu;
@@ -256,36 +473,67 @@ int fpu_clone(struct task_struct *dst) · 1 unmodified line
256	473	/* The new task's FPU state cannot be valid in the hardware. */
257	474	dst_fpu->last_cpu = -1;
258	475
	476	+ fpstate_reset(dst_fpu);
	477	+
259	478	if (!cpu_feature_enabled(X86_FEATURE_FPU))
260	479	return 0;
261	480
262	481	/*
263		− * Don't let 'init optimized' areas of the XSAVE area
264		− * leak into the child task:
	482	+ * Enforce reload for user space tasks and prevent kernel threads
	483	+ * from trying to save the FPU registers on context switch.
	484	+ */
	485	+ set_tsk_thread_flag(dst, TIF_NEED_FPU_LOAD);
	486	+
	487	+ /*
	488	+ * No FPU state inheritance for kernel threads and IO
	489	+ * worker threads.
	490	+ */
	491	+ if (dst->flags & (PF_KTHREAD \| PF_IO_WORKER)) {
	492	+ /* Clear out the minimal state */
	493	+ memcpy(&dst_fpu->fpstate->regs, &init_fpstate.regs,
	494	+ init_fpstate_copy_size());
	495	+ return 0;
	496	+ }
	497	+
	498	+ /*
	499	+ * If a new feature is added, ensure all dynamic features are
	500	+ * caller-saved from here!
265	501	*/
266		− memset(&dst_fpu->state.xsave, 0, fpu_kernel_xstate_size);
	502	+ BUILD_BUG_ON(XFEATURE_MASK_USER_DYNAMIC != XFEATURE_MASK_XTILE_DATA);
267	503
268	504	/*
269		− * If the FPU registers are not owned by current just memcpy() the
270		− * state. Otherwise save the FPU registers directly into the
271		− * child's FPU context, without any memory-to-memory copying.
	505	+ * Save the default portion of the current FPU state into the
	506	+ * clone. Assume all dynamic features to be defined as caller-
	507	+ * saved, which enables skipping both the expansion of fpstate
	508	+ * and the copying of any dynamic state.
	509	+ *
	510	+ * Do not use memcpy() when TIF_NEED_FPU_LOAD is set because
	511	+ * copying is not valid when current uses non-default states.
272	512	*/
273	513	fpregs_lock();
274	514	if (test_thread_flag(TIF_NEED_FPU_LOAD))
275		− memcpy(&dst_fpu->state, &src_fpu->state, fpu_kernel_xstate_size);
276		−
277		− else
278		− save_fpregs_to_fpstate(dst_fpu);
	515	+ fpregs_restore_userregs();
	516	+ save_fpregs_to_fpstate(dst_fpu);
	517	+ if (!(clone_flags & CLONE_THREAD))
	518	+ fpu_inherit_perms(dst_fpu);
279	519	fpregs_unlock();
280	520
281		− set_tsk_thread_flag(dst, TIF_NEED_FPU_LOAD);
282		−
283	521	trace_x86_fpu_copy_src(src_fpu);
284	522	trace_x86_fpu_copy_dst(dst_fpu);
285	523
286	524	return 0;
287	525	}
288	526
	527	+/*
	528	+ * Whitelist the FPU register state embedded into task_struct for hardened
	529	+ * usercopy.
	530	+ */
	531	+void fpu_thread_struct_whitelist(unsigned long offset, unsigned long size)
	532	+{
	533	+ *offset = offsetof(struct thread_struct, fpu.__fpstate.regs);
	534	+ *size = fpu_kernel_cfg.default_size;
	535	+}
	536	+
289	537	/*
290	538	* Drops current FPU state: deactivates the fpregs and
291	539	* the fpstate. NOTE: it still leaves previous contents
@@ -319,28 +567,19 @@ void fpu__drop(struct fpu *fpu) · 27 unmodified lines
319	567	static inline void restore_fpregs_from_init_fpstate(u64 features_mask)
320	568	{
321	569	if (use_xsave())
322		− os_xrstor(&init_fpstate.xsave, features_mask);
	570	+ os_xrstor(&init_fpstate, features_mask);
323	571	else if (use_fxsr())
324		− fxrstor(&init_fpstate.fxsave);
	572	+ fxrstor(&init_fpstate.regs.fxsave);
325	573	else
326		− frstor(&init_fpstate.fsave);
	574	+ frstor(&init_fpstate.regs.fsave);
327	575
328	576	pkru_write_default();
329	577	}
330	578
331		−static inline unsigned int init_fpstate_copy_size(void)
332		−{
333		− if (!use_xsave())
334		− return fpu_kernel_xstate_size;
335		−
336		− /* XSAVE(S) just needs the legacy and the xstate header part */
337		− return sizeof(init_fpstate.xsave);
338		−}
339		−
340	579	/*
341	580	* Reset current->fpu memory state to the init values.
342	581	*/
343		−static void fpu_reset_fpstate(void)
	582	+static void fpu_reset_fpregs(void)
344	583	{
345	584	struct fpu *fpu = &current->thread.fpu;
346	585
@@ -359,7 +598,7 @@ static void fpu_reset_fpstate(void) · 12 unmodified lines
359	598	* user space as PKRU is eagerly written in switch_to() and
360	599	* flush_thread().
361	600	*/
362		− memcpy(&fpu->state, &init_fpstate, init_fpstate_copy_size());
	601	+ memcpy(&fpu->fpstate->regs, &init_fpstate.regs, init_fpstate_copy_size());
363	602	set_thread_flag(TIF_NEED_FPU_LOAD);
364	603	fpregs_unlock();
365	604	}
@@ -375,7 +614,7 @@ void fpu__clear_user_states(struct fpu *fpu) · 9 unmodified lines
375	614
376	615	fpregs_lock();
377	616	if (!cpu_feature_enabled(X86_FEATURE_FPU)) {
378		− fpu_reset_fpstate();
	617	+ fpu_reset_fpregs();
379	618	fpregs_unlock();
380	619	return;
381	620	}
@@ -385,12 +624,11 @@ void fpu__clear_user_states(struct fpu *fpu) · 3 unmodified lines
385	624	* corresponding registers.
386	625	*/
387	626	if (xfeatures_mask_supervisor() &&
388		− !fpregs_state_valid(fpu, smp_processor_id())) {
389		− os_xrstor(&fpu->state.xsave, xfeatures_mask_supervisor());
390		− }
	627	+ !fpregs_state_valid(fpu, smp_processor_id()))
	628	+ os_xrstor_supervisor(fpu->fpstate);
391	629
392	630	/* Reset user states in registers. */
393		− restore_fpregs_from_init_fpstate(xfeatures_mask_restore_user());
	631	+ restore_fpregs_from_init_fpstate(XFEATURE_MASK_USER_RESTORE);
394	632
395	633	/*
396	634	* Now all FPU registers have their desired values. Inform the FPU
@@ -405,7 +643,8 @@ void fpu__clear_user_states(struct fpu *fpu) · 8 unmodified lines
405	643
406	644	void fpu_flush_thread(void)
407	645	{
408		− fpu_reset_fpstate();
	646	+ fpstate_reset(&current->thread.fpu);
	647	+ fpu_reset_fpregs();
409	648	}
410	649	/*
411	650	* Load FPU context before returning to userspace.
@@ -445,7 +684,6 @@ void fpregs_mark_activate(void) · 33 unmodified lines
445	684	fpu->last_cpu = smp_processor_id();
446	685	clear_thread_flag(TIF_NEED_FPU_LOAD);
447	686	}
448		−EXPORT_SYMBOL_GPL(fpregs_mark_activate);
449	687
450	688	/*
451	689	* x87 math exception handling:
@@ -468,11 +706,11 @@ int fpu__exception_code(struct fpu *fpu, int trap_nr) · 16 unmodified lines
468	706	* fully reproduce the context of the exception.
469	707	*/
470	708	if (boot_cpu_has(X86_FEATURE_FXSR)) {
471		− cwd = fpu->state.fxsave.cwd;
472		− swd = fpu->state.fxsave.swd;
	709	+ cwd = fpu->fpstate->regs.fxsave.cwd;
	710	+ swd = fpu->fpstate->regs.fxsave.swd;
473	711	} else {
474		− cwd = (unsigned short)fpu->state.fsave.cwd;
475		− swd = (unsigned short)fpu->state.fsave.swd;
	712	+ cwd = (unsigned short)fpu->fpstate->regs.fsave.cwd;
	713	+ swd = (unsigned short)fpu->fpstate->regs.fsave.swd;
476	714	}
477	715
478	716	err = swd & ~cwd;
@@ -486,7 +724,7 @@ int fpu__exception_code(struct fpu *fpu, int trap_nr) · 7 unmodified lines
486	724	unsigned short mxcsr = MXCSR_DEFAULT;
487	725
488	726	if (boot_cpu_has(X86_FEATURE_XMM))
489		− mxcsr = fpu->state.fxsave.mxcsr;
	727	+ mxcsr = fpu->fpstate->regs.fxsave.mxcsr;
490	728
491	729	err = ~(mxcsr >> 7) & mxcsr;
492	730	}

arch/x86/kernel/fpu/init.c+34−42

@@ -2,7 +2,7 @@ · 1 unmodified line
2	2	/*
3	3	* x86 FPU boot time init code:
4	4	*/
5		−#include <asm/fpu/internal.h>
	5	+#include <asm/fpu/api.h>
6	6	#include <asm/tlbflush.h>
7	7	#include <asm/setup.h>
8	8
@@ -10,6 +10,10 @@ · 1 unmodified line
10	10	#include <linux/sched/task.h>
11	11	#include <linux/init.h>
12	12
	13	+#include "internal.h"
	14	+#include "legacy.h"
	15	+#include "xstate.h"
	16	+
13	17	/*
14	18	* Initialize the registers found in all CPUs, CR0 and CR4:
15	19	*/
@@ -34,7 +38,7 @@ static void fpu__init_cpu_generic(void) · 18 unmodified lines
34	38	/* Flush out any pending x87 state: */
35	39	#ifdef CONFIG_MATH_EMULATION
36	40	if (!boot_cpu_has(X86_FEATURE_FPU))
37		− fpstate_init_soft(&current->thread.fpu.state.soft);
	41	+ fpstate_init_soft(&current->thread.fpu.fpstate->regs.soft);
38	42	else
39	43	#endif
40	44	asm volatile ("fninit");
@@ -121,23 +125,14 @@ static void __init fpu__init_system_mxcsr(void) · 80 unmodified lines
121	125	static void __init fpu__init_system_generic(void)
122	126	{
123	127	/*
124		− * Set up the legacy init FPU context. (xstate init might overwrite this
125		− * with a more modern format, if the CPU supports it.)
	128	+ * Set up the legacy init FPU context. Will be updated when the
	129	+ * CPU supports XSAVE[S].
126	130	*/
127		− fpstate_init(&init_fpstate);
	131	+ fpstate_init_user(&init_fpstate);
128	132
129	133	fpu__init_system_mxcsr();
130	134	}
131	135
132		−/*
133		− * Size of the FPU context state. All tasks in the system use the
134		− * same context size, regardless of what portion they use.
135		− * This is inherent to the XSAVE architecture which puts all state
136		− * components into a single, continuous memory block:
137		− */
138		−unsigned int fpu_kernel_xstate_size __ro_after_init;
139		−EXPORT_SYMBOL_GPL(fpu_kernel_xstate_size);
140		−
141	136	/* Get alignment of the TYPE. */
142	137	#define TYPE_ALIGN(TYPE) offsetof(struct { char x; TYPE test; }, test)
143	138
@@ -162,13 +157,13 @@ static void __init fpu__init_task_struct_size(void) · 18 unmodified lines
162	157	* Subtract off the static size of the register state.
163	158	* It potentially has a bunch of padding.
164	159	*/
165		− task_size -= sizeof(((struct task_struct *)0)->thread.fpu.state);
	160	+ task_size -= sizeof(current->thread.fpu.__fpstate.regs);
166	161
167	162	/*
168	163	* Add back the dynamically-calculated register state
169	164	* size.
170	165	*/
171		− task_size += fpu_kernel_xstate_size;
	166	+ task_size += fpu_kernel_cfg.default_size;
172	167
173	168	/*
174	169	* We dynamically size 'struct fpu', so we require that
@@ -177,7 +172,7 @@ static void __init fpu__init_task_struct_size(void) · 2 unmodified lines
177	172	* you hit a compile error here, check the structure to
178	173	* see if something got added to the end.
179	174	*/
180		− CHECK_MEMBER_AT_END_OF(struct fpu, state);
	175	+ CHECK_MEMBER_AT_END_OF(struct fpu, __fpstate);
181	176	CHECK_MEMBER_AT_END_OF(struct thread_struct, fpu);
182	177	CHECK_MEMBER_AT_END_OF(struct task_struct, thread);
183	178
@@ -192,37 +187,34 @@ static void __init fpu__init_task_struct_size(void) · 8 unmodified lines
192	187	*/
193	188	static void __init fpu__init_system_xstate_size_legacy(void)
194	189	{
195		− static int on_boot_cpu __initdata = 1;
196		−
197		− WARN_ON_FPU(!on_boot_cpu);
198		− on_boot_cpu = 0;
	190	+ unsigned int size;
199	191
200	192	/*
201		− * Note that xstate sizes might be overwritten later during
202		− * fpu__init_system_xstate().
	193	+ * Note that the size configuration might be overwritten later
	194	+ * during fpu__init_system_xstate().
203	195	*/
204		−
205		− if (!boot_cpu_has(X86_FEATURE_FPU)) {
206		− fpu_kernel_xstate_size = sizeof(struct swregs_state);
	196	+ if (!cpu_feature_enabled(X86_FEATURE_FPU)) {
	197	+ size = sizeof(struct swregs_state);
	198	+ } else if (cpu_feature_enabled(X86_FEATURE_FXSR)) {
	199	+ size = sizeof(struct fxregs_state);
	200	+ fpu_user_cfg.legacy_features = XFEATURE_MASK_FPSSE;
207	201	} else {
208		− if (boot_cpu_has(X86_FEATURE_FXSR))
209		− fpu_kernel_xstate_size =
210		− sizeof(struct fxregs_state);
211		− else
212		− fpu_kernel_xstate_size =
213		− sizeof(struct fregs_state);
	202	+ size = sizeof(struct fregs_state);
	203	+ fpu_user_cfg.legacy_features = XFEATURE_MASK_FP;
214	204	}
215	205
216		− fpu_user_xstate_size = fpu_kernel_xstate_size;
	206	+ fpu_kernel_cfg.max_size = size;
	207	+ fpu_kernel_cfg.default_size = size;
	208	+ fpu_user_cfg.max_size = size;
	209	+ fpu_user_cfg.default_size = size;
	210	+ fpstate_reset(&current->thread.fpu);
217	211	}
218	212
219		−/* Legacy code to initialize eager fpu mode. */
220		−static void __init fpu__init_system_ctx_switch(void)
	213	+static void __init fpu__init_init_fpstate(void)
221	214	{
222		− static bool on_boot_cpu __initdata = 1;
223		−
224		− WARN_ON_FPU(!on_boot_cpu);
225		− on_boot_cpu = 0;
	215	+ /* Bring init_fpstate size and features up to date */
	216	+ init_fpstate.size = fpu_kernel_cfg.max_size;
	217	+ init_fpstate.xfeatures = fpu_kernel_cfg.max_features;
226	218	}
227	219
228	220	/*
@@ -231,6 +223,7 @@ static void __init fpu__init_system_ctx_switch(void) · 2 unmodified lines
231	223	*/
232	224	void __init fpu__init_system(struct cpuinfo_x86 *c)
233	225	{
	226	+ fpstate_reset(&current->thread.fpu);
234	227	fpu__init_system_early_generic(c);
235	228
236	229	/*
@@ -241,8 +234,7 @@ void __init fpu__init_system(struct cpuinfo_x86 *c) · 4 unmodified lines
241	234
242	235	fpu__init_system_generic();
243	236	fpu__init_system_xstate_size_legacy();
244		− fpu__init_system_xstate();
	237	+ fpu__init_system_xstate(fpu_kernel_cfg.max_size);
245	238	fpu__init_task_struct_size();
246		−
247		− fpu__init_system_ctx_switch();
	239	+ fpu__init_init_fpstate();
248	240	}

arch/x86/kernel/fpu/internal.h+28−0

@@ -0,0 +1,28 @@
	1	+/* SPDX-License-Identifier: GPL-2.0 */
	2	+#ifndef __X86_KERNEL_FPU_INTERNAL_H
	3	+#define __X86_KERNEL_FPU_INTERNAL_H
	4	+
	5	+extern struct fpstate init_fpstate;
	6	+
	7	+/* CPU feature check wrappers */
	8	+static __always_inline __pure bool use_xsave(void)
	9	+{
	10	+ return cpu_feature_enabled(X86_FEATURE_XSAVE);
	11	+}
	12	+
	13	+static __always_inline __pure bool use_fxsr(void)
	14	+{
	15	+ return cpu_feature_enabled(X86_FEATURE_FXSR);
	16	+}
	17	+
	18	+#ifdef CONFIG_X86_DEBUG_FPU
	19	+# define WARN_ON_FPU(x) WARN_ON_ONCE(x)
	20	+#else
	21	+# define WARN_ON_FPU(x) ({ (void)(x); 0; })
	22	+#endif
	23	+
	24	+/* Used in init.c */
	25	+extern void fpstate_init_user(struct fpstate *fpstate);
	26	+extern void fpstate_reset(struct fpu *fpu);
	27	+
	28	+#endif

arch/x86/kernel/fpu/legacy.h+115−0

@@ -0,0 +1,115 @@
	1	+/* SPDX-License-Identifier: GPL-2.0 */
	2	+#ifndef __X86_KERNEL_FPU_LEGACY_H
	3	+#define __X86_KERNEL_FPU_LEGACY_H
	4	+
	5	+#include <asm/fpu/types.h>
	6	+
	7	+extern unsigned int mxcsr_feature_mask;
	8	+
	9	+static inline void ldmxcsr(u32 mxcsr)
	10	+{
	11	+ asm volatile("ldmxcsr %0" :: "m" (mxcsr));
	12	+}
	13	+
	14	+/*
	15	+ * Returns 0 on success or the trap number when the operation raises an
	16	+ * exception.
	17	+ */
	18	+#define user_insn(insn, output, input...) \
	19	+({ \
	20	+ int err; \
	21	+ \
	22	+ might_fault(); \
	23	+ \
	24	+ asm volatile(ASM_STAC "\n" \
	25	+ "1: " #insn "\n" \
	26	+ "2: " ASM_CLAC "\n" \
	27	+ _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_FAULT_MCE_SAFE) \
	28	+ : [err] "=a" (err), output \
	29	+ : "0"(0), input); \
	30	+ err; \
	31	+})
	32	+
	33	+#define kernel_insn_err(insn, output, input...) \
	34	+({ \
	35	+ int err; \
	36	+ asm volatile("1:" #insn "\n\t" \
	37	+ "2:\n" \
	38	+ ".section .fixup,\"ax\"\n" \
	39	+ "3: movl $-1,%[err]\n" \
	40	+ " jmp 2b\n" \
	41	+ ".previous\n" \
	42	+ _ASM_EXTABLE(1b, 3b) \
	43	+ : [err] "=r" (err), output \
	44	+ : "0"(0), input); \
	45	+ err; \
	46	+})
	47	+
	48	+#define kernel_insn(insn, output, input...) \
	49	+ asm volatile("1:" #insn "\n\t" \
	50	+ "2:\n" \
	51	+ _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_FPU_RESTORE) \
	52	+ : output : input)
	53	+
	54	+static inline int fnsave_to_user_sigframe(struct fregs_state __user *fx)
	55	+{
	56	+ return user_insn(fnsave %[fx]; fwait, [fx] "=m" (fx), "m" (fx));
	57	+}
	58	+
	59	+static inline int fxsave_to_user_sigframe(struct fxregs_state __user *fx)
	60	+{
	61	+ if (IS_ENABLED(CONFIG_X86_32))
	62	+ return user_insn(fxsave %[fx], [fx] "=m" (fx), "m" (fx));
	63	+ else
	64	+ return user_insn(fxsaveq %[fx], [fx] "=m" (fx), "m" (fx));
	65	+
	66	+}
	67	+
	68	+static inline void fxrstor(struct fxregs_state *fx)
	69	+{
	70	+ if (IS_ENABLED(CONFIG_X86_32))
	71	+ kernel_insn(fxrstor %[fx], "=m" (fx), [fx] "m" (fx));
	72	+ else
	73	+ kernel_insn(fxrstorq %[fx], "=m" (fx), [fx] "m" (fx));
	74	+}
	75	+
	76	+static inline int fxrstor_safe(struct fxregs_state *fx)
	77	+{
	78	+ if (IS_ENABLED(CONFIG_X86_32))
	79	+ return kernel_insn_err(fxrstor %[fx], "=m" (fx), [fx] "m" (fx));
	80	+ else
	81	+ return kernel_insn_err(fxrstorq %[fx], "=m" (fx), [fx] "m" (fx));
	82	+}
	83	+
	84	+static inline int fxrstor_from_user_sigframe(struct fxregs_state __user *fx)
	85	+{
	86	+ if (IS_ENABLED(CONFIG_X86_32))
	87	+ return user_insn(fxrstor %[fx], "=m" (fx), [fx] "m" (fx));
	88	+ else
	89	+ return user_insn(fxrstorq %[fx], "=m" (fx), [fx] "m" (fx));
	90	+}
	91	+
	92	+static inline void frstor(struct fregs_state *fx)
	93	+{
	94	+ kernel_insn(frstor %[fx], "=m" (fx), [fx] "m" (fx));
	95	+}
	96	+
	97	+static inline int frstor_safe(struct fregs_state *fx)
	98	+{
	99	+ return kernel_insn_err(frstor %[fx], "=m" (fx), [fx] "m" (fx));
	100	+}
	101	+
	102	+static inline int frstor_from_user_sigframe(struct fregs_state __user *fx)
	103	+{
	104	+ return user_insn(frstor %[fx], "=m" (fx), [fx] "m" (fx));
	105	+}
	106	+
	107	+static inline void fxsave(struct fxregs_state *fx)
	108	+{
	109	+ if (IS_ENABLED(CONFIG_X86_32))
	110	+ asm volatile( "fxsave %[fx]" : [fx] "=m" (*fx));
	111	+ else
	112	+ asm volatile("fxsaveq %[fx]" : [fx] "=m" (*fx));
	113	+}
	114	+
	115	+#endif

arch/x86/kernel/fpu/regset.c+20−16

@@ -5,10 +5,14 @@ · 4 unmodified lines
5	5	#include <linux/sched/task_stack.h>
6	6	#include <linux/vmalloc.h>
7	7
8		−#include <asm/fpu/internal.h>
	8	+#include <asm/fpu/api.h>
9	9	#include <asm/fpu/signal.h>
10	10	#include <asm/fpu/regset.h>
11		−#include <asm/fpu/xstate.h>
	11	+
	12	+#include "context.h"
	13	+#include "internal.h"
	14	+#include "legacy.h"
	15	+#include "xstate.h"
12	16
13	17	/*
14	18	* The xstateregs_active() routine is the same as the regset_fpregs_active() routine,
@@ -74,8 +78,8 @@ int xfpregs_get(struct task_struct target, const struct user_regset regset, · 59 unmodified lines
74	78	sync_fpstate(fpu);
75	79
76	80	if (!use_xsave()) {
77		− return membuf_write(&to, &fpu->state.fxsave,
78		− sizeof(fpu->state.fxsave));
	81	+ return membuf_write(&to, &fpu->fpstate->regs.fxsave,
	82	+ sizeof(fpu->fpstate->regs.fxsave));
79	83	}
80	84
81	85	copy_xstate_to_uabi_buf(to, target, XSTATE_COPY_FX);
@@ -110,15 +114,15 @@ int xfpregs_set(struct task_struct target, const struct user_regset regset, · 28 unmodified lines
110	114	fpu_force_restore(fpu);
111	115
112	116	/* Copy the state */
113		− memcpy(&fpu->state.fxsave, &newstate, sizeof(newstate));
	117	+ memcpy(&fpu->fpstate->regs.fxsave, &newstate, sizeof(newstate));
114	118
115	119	/* Clear xmm8..15 */
116		− BUILD_BUG_ON(sizeof(fpu->state.fxsave.xmm_space) != 16 * 16);
117		− memset(&fpu->state.fxsave.xmm_space[8], 0, 8 * 16);
	120	+ BUILD_BUG_ON(sizeof(fpu->__fpstate.regs.fxsave.xmm_space) != 16 * 16);
	121	+ memset(&fpu->fpstate->regs.fxsave.xmm_space[8], 0, 8 * 16);
118	122
119	123	/* Mark FP and SSE as in use when XSAVE is enabled */
120	124	if (use_xsave())
121		− fpu->state.xsave.header.xfeatures \|= XFEATURE_MASK_FPSSE;
	125	+ fpu->fpstate->regs.xsave.header.xfeatures \|= XFEATURE_MASK_FPSSE;
122	126
123	127	return 0;
124	128	}
@@ -149,7 +153,7 @@ int xstateregs_set(struct task_struct target, const struct user_regset regset, · 24 unmodified lines
149	153	/*
150	154	* A whole standard-format XSAVE buffer is needed:
151	155	*/
152		− if (pos != 0 \|\| count != fpu_user_xstate_size)
	156	+ if (pos != 0 \|\| count != fpu_user_cfg.max_size)
153	157	return -EFAULT;
154	158
155	159	if (!kbuf) {
@@ -164,7 +168,7 @@ int xstateregs_set(struct task_struct target, const struct user_regset regset, · 8 unmodified lines
164	168	}
165	169
166	170	fpu_force_restore(fpu);
167		− ret = copy_uabi_from_kernel_to_xstate(&fpu->state.xsave, kbuf ?: tmpbuf);
	171	+ ret = copy_uabi_from_kernel_to_xstate(fpu->fpstate, kbuf ?: tmpbuf);
168	172
169	173	out:
170	174	vfree(tmpbuf);
@@ -283,7 +287,7 @@ static void __convert_from_fxsr(struct user_i387_ia32_struct *env, · 112 unmodified lines
283	287	void
284	288	convert_from_fxsr(struct user_i387_ia32_struct env, struct task_struct tsk)
285	289	{
286		− __convert_from_fxsr(env, tsk, &tsk->thread.fpu.state.fxsave);
	290	+ __convert_from_fxsr(env, tsk, &tsk->thread.fpu.fpstate->regs.fxsave);
287	291	}
288	292
289	293	void convert_to_fxsr(struct fxregs_state *fxsave,
@@ -326,7 +330,7 @@ int fpregs_get(struct task_struct target, const struct user_regset regset, · 36 unmodified lines
326	330	return fpregs_soft_get(target, regset, to);
327	331
328	332	if (!cpu_feature_enabled(X86_FEATURE_FXSR)) {
329		− return membuf_write(&to, &fpu->state.fsave,
	333	+ return membuf_write(&to, &fpu->fpstate->regs.fsave,
330	334	sizeof(struct fregs_state));
331	335	}
332	336
@@ -337,7 +341,7 @@ int fpregs_get(struct task_struct target, const struct user_regset regset, · 4 unmodified lines
337	341	copy_xstate_to_uabi_buf(mb, target, XSTATE_COPY_FP);
338	342	fx = &fxsave;
339	343	} else {
340		− fx = &fpu->state.fxsave;
	344	+ fx = &fpu->fpstate->regs.fxsave;
341	345	}
342	346
343	347	__convert_from_fxsr(&env, target, fx);
@@ -366,16 +370,16 @@ int fpregs_set(struct task_struct target, const struct user_regset regset, · 22 unmodified lines
366	370	fpu_force_restore(fpu);
367	371
368	372	if (cpu_feature_enabled(X86_FEATURE_FXSR))
369		− convert_to_fxsr(&fpu->state.fxsave, &env);
	373	+ convert_to_fxsr(&fpu->fpstate->regs.fxsave, &env);
370	374	else
371		− memcpy(&fpu->state.fsave, &env, sizeof(env));
	375	+ memcpy(&fpu->fpstate->regs.fsave, &env, sizeof(env));
372	376
373	377	/*
374	378	* Update the header bit in the xsave header, indicating the
375	379	* presence of FP.
376	380	*/
377	381	if (cpu_feature_enabled(X86_FEATURE_XSAVE))
378		− fpu->state.xsave.header.xfeatures \|= XFEATURE_MASK_FP;
	382	+ fpu->fpstate->regs.xsave.header.xfeatures \|= XFEATURE_MASK_FP;
379	383
380	384	return 0;
381	385	}

arch/x86/kernel/fpu/signal.c+145−140

@@ -7,23 +7,25 @@ · 6 unmodified lines
7	7	#include <linux/cpu.h>
8	8	#include <linux/pagemap.h>
9	9
10		−#include <asm/fpu/internal.h>
11	10	#include <asm/fpu/signal.h>
12	11	#include <asm/fpu/regset.h>
13	12	#include <asm/fpu/xstate.h>
14	13
15	14	#include <asm/sigframe.h>
	15	+#include <asm/trapnr.h>
16	16	#include <asm/trace/fpu.h>
17	17
18		−static struct _fpx_sw_bytes fx_sw_reserved __ro_after_init;
19		−static struct _fpx_sw_bytes fx_sw_reserved_ia32 __ro_after_init;
	18	+#include "context.h"
	19	+#include "internal.h"
	20	+#include "legacy.h"
	21	+#include "xstate.h"
20	22
21	23	/*
22	24	* Check for the presence of extended state information in the
23	25	* user fpstate pointer in the sigcontext.
24	26	*/
25		−static inline int check_xstate_in_sigframe(struct fxregs_state __user *fxbuf,
26		− struct _fpx_sw_bytes *fx_sw)
	27	+static inline bool check_xstate_in_sigframe(struct fxregs_state __user *fxbuf,
	28	+ struct _fpx_sw_bytes *fx_sw)
27	29	{
28	30	int min_xstate_size = sizeof(struct fxregs_state) +
29	31	sizeof(struct xstate_header);
@@ -31,12 +33,12 @@ static inline int check_xstate_in_sigframe(struct fxregs_state __user *fxbuf, · 1 unmodified line
31	33	unsigned int magic2;
32	34
33	35	if (__copy_from_user(fx_sw, &fxbuf->sw_reserved[0], sizeof(*fx_sw)))
34		− return -EFAULT;
	36	+ return false;
35	37
36	38	/* Check for the first magic field and other error scenarios. */
37	39	if (fx_sw->magic1 != FP_XSTATE_MAGIC1 \|\|
38	40	fx_sw->xstate_size < min_xstate_size \|\|
39		− fx_sw->xstate_size > fpu_user_xstate_size \|\|
	41	+ fx_sw->xstate_size > current->thread.fpu.fpstate->user_size \|\|
40	42	fx_sw->xstate_size > fx_sw->extended_size)
41	43	goto setfx;
42	44
@@ -47,10 +49,10 @@ static inline int check_xstate_in_sigframe(struct fxregs_state __user *fxbuf, · 4 unmodified lines
47	49	* in the memory layout.
48	50	*/
49	51	if (__get_user(magic2, (__u32 __user *)(fpstate + fx_sw->xstate_size)))
50		− return -EFAULT;
	52	+ return false;
51	53
52	54	if (likely(magic2 == FP_XSTATE_MAGIC2))
53		− return 0;
	55	+ return true;
54	56	setfx:
55	57	trace_x86_fpu_xstate_check_failed(&current->thread.fpu);
56	58
@@ -58,22 +60,22 @@ setfx: · 1 unmodified line
58	60	fx_sw->magic1 = 0;
59	61	fx_sw->xstate_size = sizeof(struct fxregs_state);
60	62	fx_sw->xfeatures = XFEATURE_MASK_FPSSE;
61		− return 0;
	63	+ return true;
62	64	}
63	65
64	66	/*
65	67	* Signal frame handlers.
66	68	*/
67		−static inline int save_fsave_header(struct task_struct tsk, void __user buf)
	69	+static inline bool save_fsave_header(struct task_struct tsk, void __user buf)
68	70	{
69	71	if (use_fxsr()) {
70		− struct xregs_state *xsave = &tsk->thread.fpu.state.xsave;
	72	+ struct xregs_state *xsave = &tsk->thread.fpu.fpstate->regs.xsave;
71	73	struct user_i387_ia32_struct env;
72	74	struct _fpstate_32 __user *fp = buf;
73	75
74	76	fpregs_lock();
75	77	if (!test_thread_flag(TIF_NEED_FPU_LOAD))
76		− fxsave(&tsk->thread.fpu.state.fxsave);
	78	+ fxsave(&tsk->thread.fpu.fpstate->regs.fxsave);
77	79	fpregs_unlock();
78	80
79	81	convert_from_fxsr(&env, tsk);
@@ -81,33 +83,54 @@ static inline int save_fsave_header(struct task_struct tsk, void __user buf) · 1 unmodified line
81	83	if (__copy_to_user(buf, &env, sizeof(env)) \|\|
82	84	__put_user(xsave->i387.swd, &fp->status) \|\|
83	85	__put_user(X86_FXSR_MAGIC, &fp->magic))
84		− return -1;
	86	+ return false;
85	87	} else {
86	88	struct fregs_state __user *fp = buf;
87	89	u32 swd;
	90	+
88	91	if (__get_user(swd, &fp->swd) \|\| __put_user(swd, &fp->status))
89		− return -1;
	92	+ return false;
90	93	}
91	94
92		− return 0;
	95	+ return true;
	96	+}
	97	+
	98	+/*
	99	+ * Prepare the SW reserved portion of the fxsave memory layout, indicating
	100	+ * the presence of the extended state information in the memory layout
	101	+ * pointed to by the fpstate pointer in the sigcontext.
	102	+ * This is saved when ever the FP and extended state context is
	103	+ * saved on the user stack during the signal handler delivery to the user.
	104	+ */
	105	+static inline void save_sw_bytes(struct _fpx_sw_bytes *sw_bytes, bool ia32_frame,
	106	+ struct fpstate *fpstate)
	107	+{
	108	+ sw_bytes->magic1 = FP_XSTATE_MAGIC1;
	109	+ sw_bytes->extended_size = fpstate->user_size + FP_XSTATE_MAGIC2_SIZE;
	110	+ sw_bytes->xfeatures = fpstate->user_xfeatures;
	111	+ sw_bytes->xstate_size = fpstate->user_size;
	112	+
	113	+ if (ia32_frame)
	114	+ sw_bytes->extended_size += sizeof(struct fregs_state);
93	115	}
94	116
95		−static inline int save_xstate_epilog(void __user *buf, int ia32_frame)
	117	+static inline bool save_xstate_epilog(void __user *buf, int ia32_frame,
	118	+ struct fpstate *fpstate)
96	119	{
97	120	struct xregs_state __user *x = buf;
98		− struct _fpx_sw_bytes *sw_bytes;
	121	+ struct _fpx_sw_bytes sw_bytes;
99	122	u32 xfeatures;
100	123	int err;
101	124
102	125	/* Setup the bytes not touched by the [f]xsave and reserved for SW. */
103		− sw_bytes = ia32_frame ? &fx_sw_reserved_ia32 : &fx_sw_reserved;
104		− err = __copy_to_user(&x->i387.sw_reserved, sw_bytes, sizeof(*sw_bytes));
	126	+ save_sw_bytes(&sw_bytes, ia32_frame, fpstate);
	127	+ err = __copy_to_user(&x->i387.sw_reserved, &sw_bytes, sizeof(sw_bytes));
105	128
106	129	if (!use_xsave())
107		− return err;
	130	+ return !err;
108	131
109	132	err \|= __put_user(FP_XSTATE_MAGIC2,
110		− (__u32 __user *)(buf + fpu_user_xstate_size));
	133	+ (__u32 __user *)(buf + fpstate->user_size));
111	134
112	135	/*
113	136	* Read the xfeatures which we copied (directly from the cpu or
@@ -130,23 +153,17 @@ static inline int save_xstate_epilog(void __user *buf, int ia32_frame) · 16 unmodified lines
130	153
131	154	err \|= __put_user(xfeatures, (__u32 __user *)&x->header.xfeatures);
132	155
133		− return err;
	156	+ return !err;
134	157	}
135	158
136	159	static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf)
137	160	{
138		− int err;
139		−
140	161	if (use_xsave())
141		− err = xsave_to_user_sigframe(buf);
142		− else if (use_fxsr())
143		− err = fxsave_to_user_sigframe((struct fxregs_state __user *) buf);
	162	+ return xsave_to_user_sigframe(buf);
	163	+ if (use_fxsr())
	164	+ return fxsave_to_user_sigframe((struct fxregs_state __user *) buf);
144	165	else
145		− err = fnsave_to_user_sigframe((struct fregs_state __user *) buf);
146		−
147		− if (unlikely(err) && __clear_user(buf, fpu_user_xstate_size))
148		− err = -EFAULT;
149		− return err;
	166	+ return fnsave_to_user_sigframe((struct fregs_state __user *) buf);
150	167	}
151	168
152	169	/*
@@ -159,10 +176,8 @@ static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf) · 6 unmodified lines
159	176	* buf == buf_fx for 64-bit frames and 32-bit fsave frame.
160	177	* buf != buf_fx for 32-bit frames with fxstate.
161	178	*
162		− * Try to save it directly to the user frame with disabled page fault handler.
163		− * If this fails then do the slow path where the FPU state is first saved to
164		− * task's fpu->state and then copy it to the user frame pointed to by the
165		− * aligned pointer 'buf_fx'.
	179	+ * Save it directly to the user frame with disabled page fault handler. If
	180	+ * that faults, try to clear the frame which handles the page fault.
166	181	*
167	182	* If this is a 32-bit frame with fxstate, put a fsave header before
168	183	* the aligned state at 'buf_fx'.
@@ -170,10 +185,11 @@ static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf) · 1 unmodified line
170	185	* For [f]xsave state, update the SW reserved fields in the [f]xsave frame
171	186	* indicating the absence/presence of the extended state to the user.
172	187	*/
173		−int copy_fpstate_to_sigframe(void __user buf, void __user buf_fx, int size)
	188	+bool copy_fpstate_to_sigframe(void __user buf, void __user buf_fx, int size)
174	189	{
175	190	struct task_struct *tsk = current;
176		− int ia32_fxstate = (buf != buf_fx);
	191	+ struct fpstate *fpstate = tsk->thread.fpu.fpstate;
	192	+ bool ia32_fxstate = (buf != buf_fx);
177	193	int ret;
178	194
179	195	ia32_fxstate &= (IS_ENABLED(CONFIG_X86_32) \|\|
@@ -181,13 +197,25 @@ int copy_fpstate_to_sigframe(void __user buf, void __user buf_fx, int size) · 1 unmodified line
181	197
182	198	if (!static_cpu_has(X86_FEATURE_FPU)) {
183	199	struct user_i387_ia32_struct fp;
	200	+
184	201	fpregs_soft_get(current, NULL, (struct membuf){.p = &fp,
185	202	.left = sizeof(fp)});
186		− return copy_to_user(buf, &fp, sizeof(fp)) ? -EFAULT : 0;
	203	+ return !copy_to_user(buf, &fp, sizeof(fp));
187	204	}
188	205
189	206	if (!access_ok(buf, size))
190		− return -EACCES;
	207	+ return false;
	208	+
	209	+ if (use_xsave()) {
	210	+ struct xregs_state __user *xbuf = buf_fx;
	211	+
	212	+ /*
	213	+ * Clear the xsave header first, so that reserved fields are
	214	+ * initialized to zero.
	215	+ */
	216	+ if (__clear_user(&xbuf->header, sizeof(xbuf->header)))
	217	+ return false;
	218	+ }
191	219	retry:
192	220	/*
193	221	* Load the FPU registers if they are not valid for the current task.
@@ -205,26 +233,26 @@ retry: · 11 unmodified lines
205	233	fpregs_unlock();
206	234
207	235	if (ret) {
208		− if (!fault_in_pages_writeable(buf_fx, fpu_user_xstate_size))
	236	+ if (!__clear_user(buf_fx, fpstate->user_size))
209	237	goto retry;
210		− return -EFAULT;
	238	+ return false;
211	239	}
212	240
213	241	/* Save the fsave header for the 32-bit frames. */
214		− if ((ia32_fxstate \|\| !use_fxsr()) && save_fsave_header(tsk, buf))
215		− return -1;
	242	+ if ((ia32_fxstate \|\| !use_fxsr()) && !save_fsave_header(tsk, buf))
	243	+ return false;
216	244
217		− if (use_fxsr() && save_xstate_epilog(buf_fx, ia32_fxstate))
218		− return -1;
	245	+ if (use_fxsr() && !save_xstate_epilog(buf_fx, ia32_fxstate, fpstate))
	246	+ return false;
219	247
220		− return 0;
	248	+ return true;
221	249	}
222	250
223		−static int __restore_fpregs_from_user(void __user *buf, u64 xrestore,
224		− bool fx_only)
	251	+static int __restore_fpregs_from_user(void __user *buf, u64 ufeatures,
	252	+ u64 xrestore, bool fx_only)
225	253	{
226	254	if (use_xsave()) {
227		− u64 init_bv = xfeatures_mask_uabi() & ~xrestore;
	255	+ u64 init_bv = ufeatures & ~xrestore;
228	256	int ret;
229	257
230	258	if (likely(!fx_only))
@@ -233,7 +261,7 @@ static int __restore_fpregs_from_user(void __user *buf, u64 xrestore, · 2 unmodified lines
233	261	ret = fxrstor_from_user_sigframe(buf);
234	262
235	263	if (!ret && unlikely(init_bv))
236		− os_xrstor(&init_fpstate.xsave, init_bv);
	264	+ os_xrstor(&init_fpstate, init_bv);
237	265	return ret;
238	266	} else if (use_fxsr()) {
239	267	return fxrstor_from_user_sigframe(buf);
@@ -246,16 +274,19 @@ static int __restore_fpregs_from_user(void __user *buf, u64 xrestore, · 6 unmodified lines
246	274	* Attempt to restore the FPU registers directly from user memory.
247	275	* Pagefaults are handled and any errors returned are fatal.
248	276	*/
249		−static int restore_fpregs_from_user(void __user *buf, u64 xrestore,
250		− bool fx_only, unsigned int size)
	277	+static bool restore_fpregs_from_user(void __user *buf, u64 xrestore,
	278	+ bool fx_only, unsigned int size)
251	279	{
252	280	struct fpu *fpu = &current->thread.fpu;
253	281	int ret;
254	282
255	283	retry:
256	284	fpregs_lock();
	285	+ /* Ensure that XFD is up to date */
	286	+ xfd_update_state(fpu->fpstate);
257	287	pagefault_disable();
258		− ret = __restore_fpregs_from_user(buf, xrestore, fx_only);
	288	+ ret = __restore_fpregs_from_user(buf, fpu->fpstate->user_xfeatures,
	289	+ xrestore, fx_only);
259	290	pagefault_enable();
260	291
261	292	if (unlikely(ret)) {
@@ -275,13 +306,12 @@ retry: · 13 unmodified lines
275	306	fpregs_unlock();
276	307
277	308	/* Try to handle #PF, but anything else is fatal. */
278		− if (ret != -EFAULT)
279		− return -EINVAL;
	309	+ if (ret != X86_TRAP_PF)
	310	+ return false;
280	311
281		− ret = fault_in_pages_readable(buf, size);
282		− if (!ret)
	312	+ if (!fault_in_pages_readable(buf, size))
283	313	goto retry;
284		− return ret;
	314	+ return false;
285	315	}
286	316
287	317	/*
@@ -294,45 +324,40 @@ retry: · 6 unmodified lines
294	324	* been restored from a user buffer directly.
295	325	*/
296	326	if (test_thread_flag(TIF_NEED_FPU_LOAD) && xfeatures_mask_supervisor())
297		− os_xrstor(&fpu->state.xsave, xfeatures_mask_supervisor());
	327	+ os_xrstor_supervisor(fpu->fpstate);
298	328
299	329	fpregs_mark_activate();
300	330	fpregs_unlock();
301		− return 0;
	331	+ return true;
302	332	}
303	333
304		−static int __fpu_restore_sig(void __user buf, void __user buf_fx,
305		− bool ia32_fxstate)
	334	+static bool __fpu_restore_sig(void __user buf, void __user buf_fx,
	335	+ bool ia32_fxstate)
306	336	{
307		− int state_size = fpu_kernel_xstate_size;
308	337	struct task_struct *tsk = current;
309	338	struct fpu *fpu = &tsk->thread.fpu;
310	339	struct user_i387_ia32_struct env;
	340	+ bool success, fx_only = false;
	341	+ union fpregs_state *fpregs;
	342	+ unsigned int state_size;
311	343	u64 user_xfeatures = 0;
312		− bool fx_only = false;
313		− int ret;
314	344
315	345	if (use_xsave()) {
316	346	struct _fpx_sw_bytes fx_sw_user;
317	347
318		− ret = check_xstate_in_sigframe(buf_fx, &fx_sw_user);
319		− if (unlikely(ret))
320		− return ret;
	348	+ if (!check_xstate_in_sigframe(buf_fx, &fx_sw_user))
	349	+ return false;
321	350
322	351	fx_only = !fx_sw_user.magic1;
323	352	state_size = fx_sw_user.xstate_size;
324	353	user_xfeatures = fx_sw_user.xfeatures;
325	354	} else {
326	355	user_xfeatures = XFEATURE_MASK_FPSSE;
	356	+ state_size = fpu->fpstate->user_size;
327	357	}
328	358
329	359	if (likely(!ia32_fxstate)) {
330		− /*
331		− * Attempt to restore the FPU registers directly from user
332		− * memory. For that to succeed, the user access cannot cause page
333		− * faults. If it does, fall back to the slow path below, going
334		− * through the kernel buffer with the enabled pagefault handler.
335		− */
	360	+ /* Restore the FPU registers directly from user memory. */
336	361	return restore_fpregs_from_user(buf_fx, user_xfeatures, fx_only,
337	362	state_size);
338	363	}
@@ -342,9 +367,8 @@ static int __fpu_restore_sig(void __user buf, void __user buf_fx, · 3 unmodified lines
342	367	* to be ignored for histerical raisins. The legacy state is folded
343	368	* in once the larger state has been copied.
344	369	*/
345		− ret = __copy_from_user(&env, buf, sizeof(env));
346		− if (ret)
347		− return ret;
	370	+ if (__copy_from_user(&env, buf, sizeof(env)))
	371	+ return false;
348	372
349	373	/*
350	374	* By setting TIF_NEED_FPU_LOAD it is ensured that our xstate is
@@ -363,38 +387,38 @@ static int __fpu_restore_sig(void __user buf, void __user buf_fx, · 12 unmodified lines
363	387	* the right place in memory. It's ia32 mode. Shrug.
364	388	*/
365	389	if (xfeatures_mask_supervisor())
366		− os_xsave(&fpu->state.xsave);
	390	+ os_xsave(fpu->fpstate);
367	391	set_thread_flag(TIF_NEED_FPU_LOAD);
368	392	}
369	393	__fpu_invalidate_fpregs_state(fpu);
370	394	__cpu_invalidate_fpregs_state();
371	395	fpregs_unlock();
372	396
	397	+ fpregs = &fpu->fpstate->regs;
373	398	if (use_xsave() && !fx_only) {
374		− ret = copy_sigframe_from_user_to_xstate(&fpu->state.xsave, buf_fx);
375		− if (ret)
376		− return ret;
	399	+ if (copy_sigframe_from_user_to_xstate(fpu->fpstate, buf_fx))
	400	+ return false;
377	401	} else {
378		− if (__copy_from_user(&fpu->state.fxsave, buf_fx,
379		− sizeof(fpu->state.fxsave)))
380		− return -EFAULT;
	402	+ if (__copy_from_user(&fpregs->fxsave, buf_fx,
	403	+ sizeof(fpregs->fxsave)))
	404	+ return false;
381	405
382	406	if (IS_ENABLED(CONFIG_X86_64)) {
383	407	/* Reject invalid MXCSR values. */
384		− if (fpu->state.fxsave.mxcsr & ~mxcsr_feature_mask)
385		− return -EINVAL;
	408	+ if (fpregs->fxsave.mxcsr & ~mxcsr_feature_mask)
	409	+ return false;
386	410	} else {
387	411	/* Mask invalid bits out for historical reasons (broken hardware). */
388		− fpu->state.fxsave.mxcsr &= mxcsr_feature_mask;
	412	+ fpregs->fxsave.mxcsr &= mxcsr_feature_mask;
389	413	}
390	414
391	415	/* Enforce XFEATURE_MASK_FPSSE when XSAVE is enabled */
392	416	if (use_xsave())
393		− fpu->state.xsave.header.xfeatures \|= XFEATURE_MASK_FPSSE;
	417	+ fpregs->xsave.header.xfeatures \|= XFEATURE_MASK_FPSSE;
394	418	}
395	419
396	420	/* Fold the legacy FP storage */
397		− convert_to_fxsr(&fpu->state.fxsave, &env);
	421	+ convert_to_fxsr(&fpregs->fxsave, &env);
398	422
399	423	fpregs_lock();
400	424	if (use_xsave()) {
@@ -409,40 +433,45 @@ static int __fpu_restore_sig(void __user buf, void __user buf_fx, · 8 unmodified lines
409	433	*/
410	434	u64 mask = user_xfeatures \| xfeatures_mask_supervisor();
411	435
412		− fpu->state.xsave.header.xfeatures &= mask;
413		− ret = os_xrstor_safe(&fpu->state.xsave, xfeatures_mask_all);
	436	+ fpregs->xsave.header.xfeatures &= mask;
	437	+ success = !os_xrstor_safe(fpu->fpstate,
	438	+ fpu_kernel_cfg.max_features);
414	439	} else {
415		− ret = fxrstor_safe(&fpu->state.fxsave);
	440	+ success = !fxrstor_safe(&fpregs->fxsave);
416	441	}
417	442
418		− if (likely(!ret))
	443	+ if (likely(success))
419	444	fpregs_mark_activate();
420	445
421	446	fpregs_unlock();
422		− return ret;
	447	+ return success;
423	448	}
424		−static inline int xstate_sigframe_size(void)
	449	+
	450	+static inline unsigned int xstate_sigframe_size(struct fpstate *fpstate)
425	451	{
426		− return use_xsave() ? fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE :
427		− fpu_user_xstate_size;
	452	+ unsigned int size = fpstate->user_size;
	453	+
	454	+ return use_xsave() ? size + FP_XSTATE_MAGIC2_SIZE : size;
428	455	}
429	456
430	457	/*
431	458	* Restore FPU state from a sigframe:
432	459	*/
433		−int fpu__restore_sig(void __user *buf, int ia32_frame)
	460	+bool fpu__restore_sig(void __user *buf, int ia32_frame)
434	461	{
435		− unsigned int size = xstate_sigframe_size();
436	462	struct fpu *fpu = &current->thread.fpu;
437	463	void __user *buf_fx = buf;
438	464	bool ia32_fxstate = false;
439		− int ret;
	465	+ bool success = false;
	466	+ unsigned int size;
440	467
441	468	if (unlikely(!buf)) {
442	469	fpu__clear_user_states(fpu);
443		− return 0;
	470	+ return true;
444	471	}
445	472
	473	+ size = xstate_sigframe_size(fpu->fpstate);
	474	+
446	475	ia32_frame &= (IS_ENABLED(CONFIG_X86_32) \|\|
447	476	IS_ENABLED(CONFIG_IA32_EMULATION));
448	477
@@ -456,30 +485,28 @@ int fpu__restore_sig(void __user *buf, int ia32_frame) · 7 unmodified lines
456	485	ia32_fxstate = true;
457	486	}
458	487
459		− if (!access_ok(buf, size)) {
460		− ret = -EACCES;
	488	+ if (!access_ok(buf, size))
461	489	goto out;
462		− }
463	490
464	491	if (!IS_ENABLED(CONFIG_X86_64) && !cpu_feature_enabled(X86_FEATURE_FPU)) {
465		− ret = fpregs_soft_set(current, NULL, 0,
466		− sizeof(struct user_i387_ia32_struct),
467		− NULL, buf);
	492	+ success = !fpregs_soft_set(current, NULL, 0,
	493	+ sizeof(struct user_i387_ia32_struct),
	494	+ NULL, buf);
468	495	} else {
469		− ret = __fpu_restore_sig(buf, buf_fx, ia32_fxstate);
	496	+ success = __fpu_restore_sig(buf, buf_fx, ia32_fxstate);
470	497	}
471	498
472	499	out:
473		− if (unlikely(ret))
	500	+ if (unlikely(!success))
474	501	fpu__clear_user_states(fpu);
475		− return ret;
	502	+ return success;
476	503	}
477	504
478	505	unsigned long
479	506	fpu__alloc_mathframe(unsigned long sp, int ia32_frame,
480	507	unsigned long buf_fx, unsigned long size)
481	508	{
482		− unsigned long frame_size = xstate_sigframe_size();
	509	+ unsigned long frame_size = xstate_sigframe_size(current->thread.fpu.fpstate);
483	510
484	511	*buf_fx = sp = round_down(sp - frame_size, 64);
485	512	if (ia32_frame && use_fxsr()) {
@@ -492,9 +519,12 @@ fpu__alloc_mathframe(unsigned long sp, int ia32_frame, · 6 unmodified lines
492	519	return sp;
493	520	}
494	521
495		−unsigned long fpu__get_fpstate_size(void)
	522	+unsigned long __init fpu__get_fpstate_size(void)
496	523	{
497		− unsigned long ret = xstate_sigframe_size();
	524	+ unsigned long ret = fpu_user_cfg.max_size;
	525	+
	526	+ if (use_xsave())
	527	+ ret += FP_XSTATE_MAGIC2_SIZE;
498	528
499	529	/*
500	530	* This space is needed on (most) 32-bit kernels, or when a 32-bit
@@ -510,28 +540,3 @@ unsigned long fpu__get_fpstate_size(void) · 9 unmodified lines
510	540	return ret;
511	541	}
512	542
513		−/*
514		− * Prepare the SW reserved portion of the fxsave memory layout, indicating
515		− * the presence of the extended state information in the memory layout
516		− * pointed by the fpstate pointer in the sigcontext.
517		− * This will be saved when ever the FP and extended state context is
518		− * saved on the user stack during the signal handler delivery to the user.
519		− */
520		−void fpu__init_prepare_fx_sw_frame(void)
521		−{
522		− int size = fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE;
523		−
524		− fx_sw_reserved.magic1 = FP_XSTATE_MAGIC1;
525		− fx_sw_reserved.extended_size = size;
526		− fx_sw_reserved.xfeatures = xfeatures_mask_uabi();
527		− fx_sw_reserved.xstate_size = fpu_user_xstate_size;
528		−
529		− if (IS_ENABLED(CONFIG_IA32_EMULATION) \|\|
530		− IS_ENABLED(CONFIG_X86_32)) {
531		− int fsave_header_size = sizeof(struct fregs_state);
532		−
533		− fx_sw_reserved_ia32 = fx_sw_reserved;
534		− fx_sw_reserved_ia32.extended_size = size + fsave_header_size;
535		− }
536		−}
537		−

arch/x86/kernel/fpu/xstate.c+712−186

@@ -4,21 +4,33 @@ · 3 unmodified lines
4	4	*
5	5	* Author: Suresh Siddha <suresh.b.siddha@intel.com>
6	6	*/
	7	+#include <linux/bitops.h>
7	8	#include <linux/compat.h>
8	9	#include <linux/cpu.h>
9	10	#include <linux/mman.h>
	11	+#include <linux/nospec.h>
10	12	#include <linux/pkeys.h>
11	13	#include <linux/seq_file.h>
12	14	#include <linux/proc_fs.h>
	15	+#include <linux/vmalloc.h>
13	16
14	17	#include <asm/fpu/api.h>
15		−#include <asm/fpu/internal.h>
16		−#include <asm/fpu/signal.h>
17	18	#include <asm/fpu/regset.h>
18		−#include <asm/fpu/xstate.h>
	19	+#include <asm/fpu/signal.h>
	20	+#include <asm/fpu/xcr.h>
19	21
20	22	#include <asm/tlbflush.h>
21		−#include <asm/cpufeature.h>
	23	+#include <asm/prctl.h>
	24	+#include <asm/elf.h>
	25	+
	26	+#include "context.h"
	27	+#include "internal.h"
	28	+#include "legacy.h"
	29	+#include "xstate.h"
	30	+
	31	+#define for_each_extended_xfeature(bit, mask) \
	32	+ (bit) = FIRST_EXTENDED_XFEATURE; \
	33	+ for_each_set_bit_from(bit, (unsigned long )&(mask), 8 sizeof(mask))
22	34
23	35	/*
24	36	* Although we spell it out in here, the Processor Trace
@@ -39,29 +51,32 @@ static const char *xfeature_names[] = · 14 unmodified lines
39	51	"Protection Keys User registers",
40	52	"PASID state",
41	53	"unknown xstate feature" ,
	54	+ "unknown xstate feature" ,
	55	+ "unknown xstate feature" ,
	56	+ "unknown xstate feature" ,
	57	+ "unknown xstate feature" ,
	58	+ "unknown xstate feature" ,
	59	+ "AMX Tile config" ,
	60	+ "AMX Tile data" ,
	61	+ "unknown xstate feature" ,
42	62	};
43	63
44		−static short xsave_cpuid_features[] __initdata = {
45		− X86_FEATURE_FPU,
46		− X86_FEATURE_XMM,
47		− X86_FEATURE_AVX,
48		− X86_FEATURE_MPX,
49		− X86_FEATURE_MPX,
50		− X86_FEATURE_AVX512F,
51		− X86_FEATURE_AVX512F,
52		− X86_FEATURE_AVX512F,
53		− X86_FEATURE_INTEL_PT,
54		− X86_FEATURE_PKU,
55		− X86_FEATURE_ENQCMD,
	64	+static unsigned short xsave_cpuid_features[] __initdata = {
	65	+ [XFEATURE_FP] = X86_FEATURE_FPU,
	66	+ [XFEATURE_SSE] = X86_FEATURE_XMM,
	67	+ [XFEATURE_YMM] = X86_FEATURE_AVX,
	68	+ [XFEATURE_BNDREGS] = X86_FEATURE_MPX,
	69	+ [XFEATURE_BNDCSR] = X86_FEATURE_MPX,
	70	+ [XFEATURE_OPMASK] = X86_FEATURE_AVX512F,
	71	+ [XFEATURE_ZMM_Hi256] = X86_FEATURE_AVX512F,
	72	+ [XFEATURE_Hi16_ZMM] = X86_FEATURE_AVX512F,
	73	+ [XFEATURE_PT_UNIMPLEMENTED_SO_FAR] = X86_FEATURE_INTEL_PT,
	74	+ [XFEATURE_PKRU] = X86_FEATURE_PKU,
	75	+ [XFEATURE_PASID] = X86_FEATURE_ENQCMD,
	76	+ [XFEATURE_XTILE_CFG] = X86_FEATURE_AMX_TILE,
	77	+ [XFEATURE_XTILE_DATA] = X86_FEATURE_AMX_TILE,
56	78	};
57	79
58		−/*
59		− * This represents the full set of bits that should ever be set in a kernel
60		− * XSAVE buffer, both supervisor and user xstates.
61		− */
62		−u64 xfeatures_mask_all __ro_after_init;
63		−EXPORT_SYMBOL_GPL(xfeatures_mask_all);
64		−
65	80	static unsigned int xstate_offsets[XFEATURE_MAX] __ro_after_init =
66	81	{ [ 0 ... XFEATURE_MAX - 1] = -1};
67	82	static unsigned int xstate_sizes[XFEATURE_MAX] __ro_after_init =
@@ -71,13 +86,6 @@ static unsigned int xstate_comp_offsets[XFEATURE_MAX] __ro_after_init = · 3 unmodified lines
71	86	static unsigned int xstate_supervisor_only_offsets[XFEATURE_MAX] __ro_after_init =
72	87	{ [ 0 ... XFEATURE_MAX - 1] = -1};
73	88
74		−/*
75		− * The XSAVE area of kernel can be in standard or compacted format;
76		− * it is always in standard format for user mode. This is the user
77		− * mode standard format size used for signal and ptrace frames.
78		− */
79		−unsigned int fpu_user_xstate_size __ro_after_init;
80		−
81	89	/*
82	90	* Return whether the system supports a given xfeature.
83	91	*
@@ -85,7 +93,7 @@ unsigned int fpu_user_xstate_size __ro_after_init; · 1 unmodified line
85	93	*/
86	94	int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name)
87	95	{
88		− u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_all;
	96	+ u64 xfeatures_missing = xfeatures_needed & ~fpu_kernel_cfg.max_features;
89	97
90	98	if (unlikely(feature_name)) {
91	99	long xfeature_idx, max_idx;
@@ -135,17 +143,26 @@ static bool xfeature_is_supervisor(int xfeature_nr) · 43 unmodified lines
135	143	*/
136	144	void fpu__init_cpu_xstate(void)
137	145	{
138		− if (!boot_cpu_has(X86_FEATURE_XSAVE) \|\| !xfeatures_mask_all)
	146	+ if (!boot_cpu_has(X86_FEATURE_XSAVE) \|\| !fpu_kernel_cfg.max_features)
139	147	return;
140	148
141	149	cr4_set_bits(X86_CR4_OSXSAVE);
142	150
	151	+ /*
	152	+ * Must happen after CR4 setup and before xsetbv() to allow KVM
	153	+ * lazy passthrough. Write independent of the dynamic state static
	154	+ * key as that does not work on the boot CPU. This also ensures
	155	+ * that any stale state is wiped out from XFD.
	156	+ */
	157	+ if (cpu_feature_enabled(X86_FEATURE_XFD))
	158	+ wrmsrl(MSR_IA32_XFD, init_fpstate.xfd);
	159	+
143	160	/*
144	161	* XCR_XFEATURE_ENABLED_MASK (aka. XCR0) sets user features
145	162	* managed by XSAVE{C, OPT, S} and XRSTOR{S}. Only XSAVE user
146	163	* states can be set here.
147	164	*/
148		− xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_uabi());
	165	+ xsetbv(XCR_XFEATURE_ENABLED_MASK, fpu_user_cfg.max_features);
149	166
150	167	/*
151	168	* MSR_IA32_XSS sets supervisor states managed by XSAVES.
@@ -158,7 +175,7 @@ void fpu__init_cpu_xstate(void) · 6 unmodified lines
158	175
159	176	static bool xfeature_enabled(enum xfeature xfeature)
160	177	{
161		− return xfeatures_mask_all & BIT_ULL(xfeature);
	178	+ return fpu_kernel_cfg.max_features & BIT_ULL(xfeature);
162	179	}
163	180
164	181	/*
@@ -184,10 +201,7 @@ static void __init setup_xstate_features(void) · 19 unmodified lines
184	201	xstate_sizes[XFEATURE_SSE] = sizeof_field(struct fxregs_state,
185	202	xmm_space);
186	203
187		− for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {
188		− if (!xfeature_enabled(i))
189		− continue;
190		−
	204	+ for_each_extended_xfeature(i, fpu_kernel_cfg.max_features) {
191	205	cpuid_count(XSTATE_CPUID, i, &eax, &ebx, &ecx, &edx);
192	206
193	207	xstate_sizes[i] = eax;
@@ -236,6 +250,8 @@ static void __init print_xstate_features(void) · 42 unmodified lines
236	250	print_xstate_feature(XFEATURE_MASK_Hi16_ZMM);
237	251	print_xstate_feature(XFEATURE_MASK_PKRU);
238	252	print_xstate_feature(XFEATURE_MASK_PASID);
	253	+ print_xstate_feature(XFEATURE_MASK_XTILE_CFG);
	254	+ print_xstate_feature(XFEATURE_MASK_XTILE_DATA);
239	255	}
240	256
241	257	/*
@@ -291,20 +307,15 @@ static void __init setup_xstate_comp_offsets(void) · 49 unmodified lines
291	307	xstate_comp_offsets[XFEATURE_SSE] = offsetof(struct fxregs_state,
292	308	xmm_space);
293	309
294		− if (!boot_cpu_has(X86_FEATURE_XSAVES)) {
295		− for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {
296		− if (xfeature_enabled(i))
297		− xstate_comp_offsets[i] = xstate_offsets[i];
298		− }
	310	+ if (!cpu_feature_enabled(X86_FEATURE_XSAVES)) {
	311	+ for_each_extended_xfeature(i, fpu_kernel_cfg.max_features)
	312	+ xstate_comp_offsets[i] = xstate_offsets[i];
299	313	return;
300	314	}
301	315
302	316	next_offset = FXSAVE_SIZE + XSAVE_HDR_SIZE;
303	317
304		− for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {
305		− if (!xfeature_enabled(i))
306		− continue;
307		−
	318	+ for_each_extended_xfeature(i, fpu_kernel_cfg.max_features) {
308	319	if (xfeature_is_aligned(i))
309	320	next_offset = ALIGN(next_offset, 64);
310	321
@@ -328,8 +339,8 @@ static void __init setup_supervisor_only_offsets(void) · 17 unmodified lines
328	339
329	340	next_offset = FXSAVE_SIZE + XSAVE_HDR_SIZE;
330	341
331		− for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {
332		− if (!xfeature_enabled(i) \|\| !xfeature_is_supervisor(i))
	342	+ for_each_extended_xfeature(i, fpu_kernel_cfg.max_features) {
	343	+ if (!xfeature_is_supervisor(i))
333	344	continue;
334	345
335	346	if (xfeature_is_aligned(i))
@@ -347,14 +358,35 @@ static void __init print_xstate_offset_size(void) · 11 unmodified lines
347	358	{
348	359	int i;
349	360
350		− for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {
351		− if (!xfeature_enabled(i))
352		− continue;
	361	+ for_each_extended_xfeature(i, fpu_kernel_cfg.max_features) {
353	362	pr_info("x86/fpu: xstate_offset[%d]: %4d, xstate_sizes[%d]: %4d\n",
354	363	i, xstate_comp_offsets[i], i, xstate_sizes[i]);
355	364	}
356	365	}
357	366
	367	+/*
	368	+ * This function is called only during boot time when x86 caps are not set
	369	+ * up and alternative can not be used yet.
	370	+ */
	371	+static __init void os_xrstor_booting(struct xregs_state *xstate)
	372	+{
	373	+ u64 mask = fpu_kernel_cfg.max_features & XFEATURE_MASK_FPSTATE;
	374	+ u32 lmask = mask;
	375	+ u32 hmask = mask >> 32;
	376	+ int err;
	377	+
	378	+ if (cpu_feature_enabled(X86_FEATURE_XSAVES))
	379	+ XSTATE_OP(XRSTORS, xstate, lmask, hmask, err);
	380	+ else
	381	+ XSTATE_OP(XRSTOR, xstate, lmask, hmask, err);
	382	+
	383	+ /*
	384	+ * We should never fault when copying from a kernel buffer, and the FPU
	385	+ * state we set at boot time should be valid.
	386	+ */
	387	+ WARN_ON_FPU(err);
	388	+}
	389	+
358	390	/*
359	391	* All supported features have either init state all zeros or are
360	392	* handled in setup_init_fpu() individually. This is an explicit
@@ -372,36 +404,30 @@ static void __init print_xstate_offset_size(void) · 11 unmodified lines
372	404	XFEATURE_MASK_PKRU \| \
373	405	XFEATURE_MASK_BNDREGS \| \
374	406	XFEATURE_MASK_BNDCSR \| \
375		− XFEATURE_MASK_PASID)
	407	+ XFEATURE_MASK_PASID \| \
	408	+ XFEATURE_MASK_XTILE)
376	409
377	410	/*
378	411	* setup the xstate image representing the init state
379	412	*/
380	413	static void __init setup_init_fpu_buf(void)
381	414	{
382		− static int on_boot_cpu __initdata = 1;
383		−
384	415	BUILD_BUG_ON((XFEATURE_MASK_USER_SUPPORTED \|
385	416	XFEATURE_MASK_SUPERVISOR_SUPPORTED) !=
386	417	XFEATURES_INIT_FPSTATE_HANDLED);
387	418
388		− WARN_ON_FPU(!on_boot_cpu);
389		− on_boot_cpu = 0;
390		−
391	419	if (!boot_cpu_has(X86_FEATURE_XSAVE))
392	420	return;
393	421
394	422	setup_xstate_features();
395	423	print_xstate_features();
396	424
397		− if (boot_cpu_has(X86_FEATURE_XSAVES))
398		− init_fpstate.xsave.header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT \|
399		− xfeatures_mask_all;
	425	+ xstate_init_xcomp_bv(&init_fpstate.regs.xsave, fpu_kernel_cfg.max_features);
400	426
401	427	/*
402	428	* Init all the features state with header.xfeatures being 0x0
403	429	*/
404		− os_xrstor_booting(&init_fpstate.xsave);
	430	+ os_xrstor_booting(&init_fpstate.regs.xsave);
405	431
406	432	/*
407	433	* All components are now in init state. Read the state back so
@@ -419,7 +445,7 @@ static void __init setup_init_fpu_buf(void) · 11 unmodified lines
419	445	* state is all zeroes or if not to add the necessary handling
420	446	* here.
421	447	*/
422		− fxsave(&init_fpstate.fxsave);
	448	+ fxsave(&init_fpstate.regs.fxsave);
423	449	}
424	450
425	451	static int xfeature_uncompacted_offset(int xfeature_nr)
@@ -451,10 +477,11 @@ int xfeature_size(int xfeature_nr) · 25 unmodified lines
451	477	}
452	478
453	479	/* Validate an xstate header supplied by userspace (ptrace or sigreturn) */
454		−static int validate_user_xstate_header(const struct xstate_header *hdr)
	480	+static int validate_user_xstate_header(const struct xstate_header *hdr,
	481	+ struct fpstate *fpstate)
455	482	{
456	483	/* No unknown or supervisor features may be set */
457		− if (hdr->xfeatures & ~xfeatures_mask_uabi())
	484	+ if (hdr->xfeatures & ~fpstate->user_xfeatures)
458	485	return -EINVAL;
459	486
460	487	/* Userspace must use the uncompacted format */
@@ -474,7 +501,7 @@ static int validate_user_xstate_header(const struct xstate_header *hdr) · 13 unmodified lines
474	501	return 0;
475	502	}
476	503
477		−static void __xstate_dump_leaves(void)
	504	+static void __init __xstate_dump_leaves(void)
478	505	{
479	506	int i;
480	507	u32 eax, ebx, ecx, edx;
@@ -509,12 +536,73 @@ static void __xstate_dump_leaves(void) · 28 unmodified lines
509	536	} \
510	537	} while (0)
511	538
	539	+/**
	540	+ * check_xtile_data_against_struct - Check tile data state size.
	541	+ *
	542	+ * Calculate the state size by multiplying the single tile size which is
	543	+ * recorded in a C struct, and the number of tiles that the CPU informs.
	544	+ * Compare the provided size with the calculation.
	545	+ *
	546	+ * @size: The tile data state size
	547	+ *
	548	+ * Returns: 0 on success, -EINVAL on mismatch.
	549	+ */
	550	+static int __init check_xtile_data_against_struct(int size)
	551	+{
	552	+ u32 max_palid, palid, state_size;
	553	+ u32 eax, ebx, ecx, edx;
	554	+ u16 max_tile;
	555	+
	556	+ /*
	557	+ * Check the maximum palette id:
	558	+ * eax: the highest numbered palette subleaf.
	559	+ */
	560	+ cpuid_count(TILE_CPUID, 0, &max_palid, &ebx, &ecx, &edx);
	561	+
	562	+ /*
	563	+ * Cross-check each tile size and find the maximum number of
	564	+ * supported tiles.
	565	+ */
	566	+ for (palid = 1, max_tile = 0; palid <= max_palid; palid++) {
	567	+ u16 tile_size, max;
	568	+
	569	+ /*
	570	+ * Check the tile size info:
	571	+ * eax[31:16]: bytes per title
	572	+ * ebx[31:16]: the max names (or max number of tiles)
	573	+ */
	574	+ cpuid_count(TILE_CPUID, palid, &eax, &ebx, &edx, &edx);
	575	+ tile_size = eax >> 16;
	576	+ max = ebx >> 16;
	577	+
	578	+ if (tile_size != sizeof(struct xtile_data)) {
	579	+ pr_err("%s: struct is %zu bytes, cpu xtile %d bytes\n",
	580	+ __stringify(XFEATURE_XTILE_DATA),
	581	+ sizeof(struct xtile_data), tile_size);
	582	+ __xstate_dump_leaves();
	583	+ return -EINVAL;
	584	+ }
	585	+
	586	+ if (max > max_tile)
	587	+ max_tile = max;
	588	+ }
	589	+
	590	+ state_size = sizeof(struct xtile_data) * max_tile;
	591	+ if (size != state_size) {
	592	+ pr_err("%s: calculated size is %u bytes, cpu state %d bytes\n",
	593	+ __stringify(XFEATURE_XTILE_DATA), state_size, size);
	594	+ __xstate_dump_leaves();
	595	+ return -EINVAL;
	596	+ }
	597	+ return 0;
	598	+}
	599	+
512	600	/*
513	601	* We have a C struct for each 'xstate'. We need to ensure
514	602	* that our software representation matches what the CPU
515	603	* tells us about the state's size.
516	604	*/
517		−static void check_xstate_against_struct(int nr)
	605	+static bool __init check_xstate_against_struct(int nr)
518	606	{
519	607	/*
520	608	* Ask the CPU for the size of the state.
@@ -532,6 +620,11 @@ static void check_xstate_against_struct(int nr) · 11 unmodified lines
532	620	XCHECK_SZ(sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state);
533	621	XCHECK_SZ(sz, nr, XFEATURE_PKRU, struct pkru_state);
534	622	XCHECK_SZ(sz, nr, XFEATURE_PASID, struct ia32_pasid_state);
	623	+ XCHECK_SZ(sz, nr, XFEATURE_XTILE_CFG, struct xtile_cfg);
	624	+
	625	+ /* The tile data size varies between implementations. */
	626	+ if (nr == XFEATURE_XTILE_DATA)
	627	+ check_xtile_data_against_struct(sz);
535	628
536	629	/*
537	630	* Make SURE to add any feature numbers in below if
@@ -541,10 +634,39 @@ static void check_xstate_against_struct(int nr) · 3 unmodified lines
541	634	if ((nr < XFEATURE_YMM) \|\|
542	635	(nr >= XFEATURE_MAX) \|\|
543	636	(nr == XFEATURE_PT_UNIMPLEMENTED_SO_FAR) \|\|
544		− ((nr >= XFEATURE_RSRVD_COMP_11) && (nr <= XFEATURE_LBR))) {
	637	+ ((nr >= XFEATURE_RSRVD_COMP_11) && (nr <= XFEATURE_RSRVD_COMP_16))) {
545	638	WARN_ONCE(1, "no structure for xstate: %d\n", nr);
546	639	XSTATE_WARN_ON(1);
	640	+ return false;
	641	+ }
	642	+ return true;
	643	+}
	644	+
	645	+static unsigned int xstate_calculate_size(u64 xfeatures, bool compacted)
	646	+{
	647	+ unsigned int size = FXSAVE_SIZE + XSAVE_HDR_SIZE;
	648	+ int i;
	649	+
	650	+ for_each_extended_xfeature(i, xfeatures) {
	651	+ /* Align from the end of the previous feature */
	652	+ if (xfeature_is_aligned(i))
	653	+ size = ALIGN(size, 64);
	654	+ /*
	655	+ * In compacted format the enabled features are packed,
	656	+ * i.e. disabled features do not occupy space.
	657	+ *
	658	+ * In non-compacted format the offsets are fixed and
	659	+ * disabled states still occupy space in the memory buffer.
	660	+ */
	661	+ if (!compacted)
	662	+ size = xfeature_uncompacted_offset(i);
	663	+ /*
	664	+ * Add the feature size even for non-compacted format
	665	+ * to make the end result correct
	666	+ */
	667	+ size += xfeature_size(i);
547	668	}
	669	+ return size;
548	670	}
549	671
550	672	/*
@@ -556,44 +678,29 @@ static void check_xstate_against_struct(int nr) · 5 unmodified lines
556	678	* covered by these checks. Only the size of the buffer for task->fpu
557	679	* is checked here.
558	680	*/
559		−static void do_extra_xstate_size_checks(void)
	681	+static bool __init paranoid_xstate_size_valid(unsigned int kernel_size)
560	682	{
561		− int paranoid_xstate_size = FXSAVE_SIZE + XSAVE_HDR_SIZE;
	683	+ bool compacted = cpu_feature_enabled(X86_FEATURE_XSAVES);
	684	+ unsigned int size = FXSAVE_SIZE + XSAVE_HDR_SIZE;
562	685	int i;
563	686
564		− for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {
565		− if (!xfeature_enabled(i))
566		− continue;
567		−
568		− check_xstate_against_struct(i);
	687	+ for_each_extended_xfeature(i, fpu_kernel_cfg.max_features) {
	688	+ if (!check_xstate_against_struct(i))
	689	+ return false;
569	690	/*
570	691	* Supervisor state components can be managed only by
571	692	* XSAVES.
572	693	*/
573		− if (!cpu_feature_enabled(X86_FEATURE_XSAVES))
574		− XSTATE_WARN_ON(xfeature_is_supervisor(i));
575		−
576		− /* Align from the end of the previous feature */
577		− if (xfeature_is_aligned(i))
578		− paranoid_xstate_size = ALIGN(paranoid_xstate_size, 64);
579		− /*
580		− * The offset of a given state in the non-compacted
581		− * format is given to us in a CPUID leaf. We check
582		− * them for being ordered (increasing offsets) in
583		− * setup_xstate_features(). XSAVES uses compacted format.
584		− */
585		− if (!cpu_feature_enabled(X86_FEATURE_XSAVES))
586		− paranoid_xstate_size = xfeature_uncompacted_offset(i);
587		− /*
588		− * The compacted-format offset always depends on where
589		− * the previous state ended.
590		− */
591		− paranoid_xstate_size += xfeature_size(i);
	694	+ if (!compacted && xfeature_is_supervisor(i)) {
	695	+ XSTATE_WARN_ON(1);
	696	+ return false;
	697	+ }
592	698	}
593		− XSTATE_WARN_ON(paranoid_xstate_size != fpu_kernel_xstate_size);
	699	+ size = xstate_calculate_size(fpu_kernel_cfg.max_features, compacted);
	700	+ XSTATE_WARN_ON(size != kernel_size);
	701	+ return size == kernel_size;
594	702	}
595	703
596		−
597	704	/*
598	705	* Get total size of enabled xstates in XCR0 \| IA32_XSS.
599	706	*
@@ -644,7 +751,7 @@ static unsigned int __init get_xsaves_size_no_independent(void) · 44 unmodified lines
644	751	return size;
645	752	}
646	753
647		−static unsigned int __init get_xsave_size(void)
	754	+static unsigned int __init get_xsave_size_user(void)
648	755	{
649	756	unsigned int eax, ebx, ecx, edx;
650	757	/*
@@ -662,44 +769,54 @@ static unsigned int __init get_xsave_size(void) · 11 unmodified lines
662	769	* Will the runtime-enumerated 'xstate_size' fit in the init
663	770	* task's statically-allocated buffer?
664	771	*/
665		−static bool is_supported_xstate_size(unsigned int test_xstate_size)
	772	+static bool __init is_supported_xstate_size(unsigned int test_xstate_size)
666	773	{
667		− if (test_xstate_size <= sizeof(union fpregs_state))
	774	+ if (test_xstate_size <= sizeof(init_fpstate.regs))
668	775	return true;
669	776
670	777	pr_warn("x86/fpu: xstate buffer too small (%zu < %d), disabling xsave\n",
671		− sizeof(union fpregs_state), test_xstate_size);
	778	+ sizeof(init_fpstate.regs), test_xstate_size);
672	779	return false;
673	780	}
674	781
675	782	static int __init init_xstate_size(void)
676	783	{
677	784	/* Recompute the context size for enabled features: */
678		− unsigned int possible_xstate_size;
679		− unsigned int xsave_size;
	785	+ unsigned int user_size, kernel_size, kernel_default_size;
	786	+ bool compacted = cpu_feature_enabled(X86_FEATURE_XSAVES);
680	787
681		− xsave_size = get_xsave_size();
	788	+ /* Uncompacted user space size */
	789	+ user_size = get_xsave_size_user();
682	790
683		− if (boot_cpu_has(X86_FEATURE_XSAVES))
684		− possible_xstate_size = get_xsaves_size_no_independent();
	791	+ /*
	792	+ * XSAVES kernel size includes supervisor states and
	793	+ * uses compacted format when available.
	794	+ *
	795	+ * XSAVE does not support supervisor states so
	796	+ * kernel and user size is identical.
	797	+ */
	798	+ if (compacted)
	799	+ kernel_size = get_xsaves_size_no_independent();
685	800	else
686		− possible_xstate_size = xsave_size;
	801	+ kernel_size = user_size;
	802	+
	803	+ kernel_default_size =
	804	+ xstate_calculate_size(fpu_kernel_cfg.default_features, compacted);
687	805
688		− /* Ensure we have the space to store all enabled: */
689		− if (!is_supported_xstate_size(possible_xstate_size))
	806	+ /* Ensure we have the space to store all default enabled features. */
	807	+ if (!is_supported_xstate_size(kernel_default_size))
690	808	return -EINVAL;
691	809
692		− /*
693		− * The size is OK, we are definitely going to use xsave,
694		− * make it known to the world that we need more space.
695		− */
696		− fpu_kernel_xstate_size = possible_xstate_size;
697		− do_extra_xstate_size_checks();
	810	+ if (!paranoid_xstate_size_valid(kernel_size))
	811	+ return -EINVAL;
	812	+
	813	+ fpu_kernel_cfg.max_size = kernel_size;
	814	+ fpu_user_cfg.max_size = user_size;
	815	+
	816	+ fpu_kernel_cfg.default_size = kernel_default_size;
	817	+ fpu_user_cfg.default_size =
	818	+ xstate_calculate_size(fpu_user_cfg.default_features, false);
698	819
699		− /*
700		− * User space is always in standard format.
701		− */
702		− fpu_user_xstate_size = xsave_size;
703	820	return 0;
704	821	}
705	822
@@ -707,28 +824,38 @@ static int __init init_xstate_size(void) · 1 unmodified line
707	824	* We enabled the XSAVE hardware, but something went wrong and
708	825	* we can not use it. Disable it.
709	826	*/
710		−static void fpu__init_disable_system_xstate(void)
	827	+static void __init fpu__init_disable_system_xstate(unsigned int legacy_size)
711	828	{
712		− xfeatures_mask_all = 0;
	829	+ fpu_kernel_cfg.max_features = 0;
713	830	cr4_clear_bits(X86_CR4_OSXSAVE);
714	831	setup_clear_cpu_cap(X86_FEATURE_XSAVE);
	832	+
	833	+ /* Restore the legacy size.*/
	834	+ fpu_kernel_cfg.max_size = legacy_size;
	835	+ fpu_kernel_cfg.default_size = legacy_size;
	836	+ fpu_user_cfg.max_size = legacy_size;
	837	+ fpu_user_cfg.default_size = legacy_size;
	838	+
	839	+ /*
	840	+ * Prevent enabling the static branch which enables writes to the
	841	+ * XFD MSR.
	842	+ */
	843	+ init_fpstate.xfd = 0;
	844	+
	845	+ fpstate_reset(&current->thread.fpu);
715	846	}
716	847
717	848	/*
718	849	* Enable and initialize the xsave feature.
719	850	* Called once per system bootup.
720	851	*/
721		−void __init fpu__init_system_xstate(void)
	852	+void __init fpu__init_system_xstate(unsigned int legacy_size)
722	853	{
723	854	unsigned int eax, ebx, ecx, edx;
724		− static int on_boot_cpu __initdata = 1;
725	855	u64 xfeatures;
726	856	int err;
727	857	int i;
728	858
729		− WARN_ON_FPU(!on_boot_cpu);
730		− on_boot_cpu = 0;
731		−
732	859	if (!boot_cpu_has(X86_FEATURE_FPU)) {
733	860	pr_info("x86/fpu: No FPU detected\n");
734	861	return;
@@ -749,22 +876,22 @@ void __init fpu__init_system_xstate(void) · 14 unmodified lines
749	876	* Find user xstates supported by the processor.
750	877	*/
751	878	cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx);
752		− xfeatures_mask_all = eax + ((u64)edx << 32);
	879	+ fpu_kernel_cfg.max_features = eax + ((u64)edx << 32);
753	880
754	881	/*
755	882	* Find supervisor xstates supported by the processor.
756	883	*/
757	884	cpuid_count(XSTATE_CPUID, 1, &eax, &ebx, &ecx, &edx);
758		− xfeatures_mask_all \|= ecx + ((u64)edx << 32);
	885	+ fpu_kernel_cfg.max_features \|= ecx + ((u64)edx << 32);
759	886
760		− if ((xfeatures_mask_uabi() & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) {
	887	+ if ((fpu_kernel_cfg.max_features & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) {
761	888	/*
762	889	* This indicates that something really unexpected happened
763	890	* with the enumeration. Disable XSAVE and try to continue
764	891	* booting without it. This is too early to BUG().
765	892	*/
766	893	pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n",
767		− xfeatures_mask_all);
	894	+ fpu_kernel_cfg.max_features);
768	895	goto out_disable;
769	896	}
770	897
@@ -772,15 +899,39 @@ void __init fpu__init_system_xstate(void) · 1 unmodified line
772	899	* Clear XSAVE features that are disabled in the normal CPUID.
773	900	*/
774	901	for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) {
775		− if (!boot_cpu_has(xsave_cpuid_features[i]))
776		− xfeatures_mask_all &= ~BIT_ULL(i);
	902	+ unsigned short cid = xsave_cpuid_features[i];
	903	+
	904	+ /* Careful: X86_FEATURE_FPU is 0! */
	905	+ if ((i != XFEATURE_FP && !cid) \|\| !boot_cpu_has(cid))
	906	+ fpu_kernel_cfg.max_features &= ~BIT_ULL(i);
777	907	}
778	908
779		− xfeatures_mask_all &= XFEATURE_MASK_USER_SUPPORTED \|
	909	+ if (!cpu_feature_enabled(X86_FEATURE_XFD))
	910	+ fpu_kernel_cfg.max_features &= ~XFEATURE_MASK_USER_DYNAMIC;
	911	+
	912	+ fpu_kernel_cfg.max_features &= XFEATURE_MASK_USER_SUPPORTED \|
780	913	XFEATURE_MASK_SUPERVISOR_SUPPORTED;
781	914
	915	+ fpu_user_cfg.max_features = fpu_kernel_cfg.max_features;
	916	+ fpu_user_cfg.max_features &= XFEATURE_MASK_USER_SUPPORTED;
	917	+
	918	+ /* Clean out dynamic features from default */
	919	+ fpu_kernel_cfg.default_features = fpu_kernel_cfg.max_features;
	920	+ fpu_kernel_cfg.default_features &= ~XFEATURE_MASK_USER_DYNAMIC;
	921	+
	922	+ fpu_user_cfg.default_features = fpu_user_cfg.max_features;
	923	+ fpu_user_cfg.default_features &= ~XFEATURE_MASK_USER_DYNAMIC;
	924	+
782	925	/* Store it for paranoia check at the end */
783		− xfeatures = xfeatures_mask_all;
	926	+ xfeatures = fpu_kernel_cfg.max_features;
	927	+
	928	+ /*
	929	+ * Initialize the default XFD state in initfp_state and enable the
	930	+ * dynamic sizing mechanism if dynamic states are available. The
	931	+ * static key cannot be enabled here because this runs before
	932	+ * jump_label_init(). This is delayed to an initcall.
	933	+ */
	934	+ init_fpstate.xfd = fpu_user_cfg.max_features & XFEATURE_MASK_USER_DYNAMIC;
784	935
785	936	/* Enable xstate instructions to be able to continue with initialization: */
786	937	fpu__init_cpu_xstate();
@@ -788,13 +939,16 @@ void __init fpu__init_system_xstate(void) · 1 unmodified line
788	939	if (err)
789	940	goto out_disable;
790	941
	942	+ /* Reset the state for the current task */
	943	+ fpstate_reset(&current->thread.fpu);
	944	+
791	945	/*
792	946	* Update info used for ptrace frames; use standard-format size and no
793	947	* supervisor xstates:
794	948	*/
795		− update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_uabi());
	949	+ update_regset_xstate_info(fpu_user_cfg.max_size,
	950	+ fpu_user_cfg.max_features);
796	951
797		− fpu__init_prepare_fx_sw_frame();
798	952	setup_init_fpu_buf();
799	953	setup_xstate_comp_offsets();
800	954	setup_supervisor_only_offsets();
@@ -803,22 +957,22 @@ void __init fpu__init_system_xstate(void) · 2 unmodified lines
803	957	* Paranoia check whether something in the setup modified the
804	958	* xfeatures mask.
805	959	*/
806		− if (xfeatures != xfeatures_mask_all) {
	960	+ if (xfeatures != fpu_kernel_cfg.max_features) {
807	961	pr_err("x86/fpu: xfeatures modified from 0x%016llx to 0x%016llx during init, disabling XSAVE\n",
808		− xfeatures, xfeatures_mask_all);
	962	+ xfeatures, fpu_kernel_cfg.max_features);
809	963	goto out_disable;
810	964	}
811	965
812	966	print_xstate_offset_size();
813	967	pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n",
814		− xfeatures_mask_all,
815		− fpu_kernel_xstate_size,
	968	+ fpu_kernel_cfg.max_features,
	969	+ fpu_kernel_cfg.max_size,
816	970	boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard");
817	971	return;
818	972
819	973	out_disable:
820	974	/* something went wrong, try to boot without any XSAVE support */
821		− fpu__init_disable_system_xstate();
	975	+ fpu__init_disable_system_xstate(legacy_size);
822	976	}
823	977
824	978	/*
@@ -830,7 +984,7 @@ void fpu__resume_cpu(void) · 5 unmodified lines
830	984	* Restore XCR0 on xsave capable CPUs:
831	985	*/
832	986	if (cpu_feature_enabled(X86_FEATURE_XSAVE))
833		− xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_uabi());
	987	+ xsetbv(XCR_XFEATURE_ENABLED_MASK, fpu_user_cfg.max_features);
834	988
835	989	/*
836	990	* Restore IA32_XSS. The same CPUID bit enumerates support
@@ -840,6 +994,9 @@ void fpu__resume_cpu(void) · 3 unmodified lines
840	994	wrmsrl(MSR_IA32_XSS, xfeatures_mask_supervisor() \|
841	995	xfeatures_mask_independent());
842	996	}
	997	+
	998	+ if (fpu_state_size_dynamic())
	999	+ wrmsrl(MSR_IA32_XFD, current->thread.fpu.fpstate->xfd);
843	1000	}
844	1001
845	1002	/*
@@ -886,7 +1043,7 @@ void get_xsave_addr(struct xregs_state xsave, int xfeature_nr) · 40 unmodified lines
886	1043	* We should not ever be requesting features that we
887	1044	* have not enabled.
888	1045	*/
889		− WARN_ONCE(!(xfeatures_mask_all & BIT_ULL(xfeature_nr)),
	1046	+ WARN_ONCE(!(fpu_kernel_cfg.max_features & BIT_ULL(xfeature_nr)),
890	1047	"get of unsupported state");
891	1048	/*
892	1049	* This assumes the last 'xsave*' instruction to
@@ -904,7 +1061,6 @@ void get_xsave_addr(struct xregs_state xsave, int xfeature_nr) · 11 unmodified lines
904	1061
905	1062	return __raw_xsave_addr(xsave, xfeature_nr);
906	1063	}
907		−EXPORT_SYMBOL_GPL(get_xsave_addr);
908	1064
909	1065	#ifdef CONFIG_ARCH_HAS_PKEYS
910	1066
@@ -961,9 +1117,10 @@ static void copy_feature(bool from_xstate, struct membuf to, void xstate, · 50 unmodified lines
961	1117	}
962	1118
963	1119	/**
964		− * copy_xstate_to_uabi_buf - Copy kernel saved xstate to a UABI buffer
	1120	+ * __copy_xstate_to_uabi_buf - Copy kernel saved xstate to a UABI buffer
965	1121	* @to: membuf descriptor
966		− * @tsk: The task from which to copy the saved xstate
	1122	+ * @fpstate: The fpstate buffer from which to copy
	1123	+ * @pkru_val: The PKRU value to store in the PKRU component
967	1124	* @copy_mode: The requested copy mode
968	1125	*
969	1126	* Converts from kernel XSAVE or XSAVES compacted format to UABI conforming
@@ -972,14 +1129,15 @@ static void copy_feature(bool from_xstate, struct membuf to, void xstate, · 2 unmodified lines
972	1129	*
973	1130	* It supports partial copy but @to.pos always starts from zero.
974	1131	*/
975		−void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk,
976		− enum xstate_copy_mode copy_mode)
	1132	+void __copy_xstate_to_uabi_buf(struct membuf to, struct fpstate *fpstate,
	1133	+ u32 pkru_val, enum xstate_copy_mode copy_mode)
977	1134	{
978	1135	const unsigned int off_mxcsr = offsetof(struct fxregs_state, mxcsr);
979		− struct xregs_state *xsave = &tsk->thread.fpu.state.xsave;
980		− struct xregs_state *xinit = &init_fpstate.xsave;
	1136	+ struct xregs_state *xinit = &init_fpstate.regs.xsave;
	1137	+ struct xregs_state *xsave = &fpstate->regs.xsave;
981	1138	struct xstate_header header;
982	1139	unsigned int zerofrom;
	1140	+ u64 mask;
983	1141	int i;
984	1142
985	1143	memset(&header, 0, sizeof(header));
@@ -996,7 +1154,7 @@ void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk, · 10 unmodified lines
996	1154	break;
997	1155
998	1156	case XSTATE_COPY_XSAVE:
999		− header.xfeatures &= xfeatures_mask_uabi();
	1157	+ header.xfeatures &= fpstate->user_xfeatures;
1000	1158	break;
1001	1159	}
1002	1160
@@ -1033,17 +1191,15 @@ void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk, · 30 unmodified lines
1033	1191
1034	1192	zerofrom = offsetof(struct xregs_state, extended_state_area);
1035	1193
1036		− for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {
1037		− /*
1038		− * The ptrace buffer is in non-compacted XSAVE format.
1039		− * In non-compacted format disabled features still occupy
1040		− * state space, but there is no state to copy from in the
1041		− * compacted init_fpstate. The gap tracking will zero this
1042		− * later.
1043		− */
1044		− if (!(xfeatures_mask_uabi() & BIT_ULL(i)))
1045		− continue;
	1194	+ /*
	1195	+ * The ptrace buffer is in non-compacted XSAVE format. In
	1196	+ * non-compacted format disabled features still occupy state space,
	1197	+ * but there is no state to copy from in the compacted
	1198	+ * init_fpstate. The gap tracking will zero these states.
	1199	+ */
	1200	+ mask = fpstate->user_xfeatures;
1046	1201
	1202	+ for_each_extended_xfeature(i, mask) {
1047	1203	/*
1048	1204	* If there was a feature or alignment gap, zero the space
1049	1205	* in the destination buffer.
@@ -1055,10 +1211,9 @@ void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk, · 5 unmodified lines
1055	1211	struct pkru_state pkru = {0};
1056	1212	/*
1057	1213	* PKRU is not necessarily up to date in the
1058		− * thread's XSAVE buffer. Fill this part from the
1059		− * per-thread storage.
	1214	+ * XSAVE buffer. Use the provided value.
1060	1215	*/
1061		− pkru.pkru = tsk->thread.pkru;
	1216	+ pkru.pkru = pkru_val;
1062	1217	membuf_write(&to, &pkru, sizeof(pkru));
1063	1218	} else {
1064	1219	copy_feature(header.xfeatures & BIT_ULL(i), &to,
@@ -1078,6 +1233,25 @@ out: · 13 unmodified lines
1078	1233	membuf_zero(&to, to.left);
1079	1234	}
1080	1235
	1236	+/**
	1237	+ * copy_xstate_to_uabi_buf - Copy kernel saved xstate to a UABI buffer
	1238	+ * @to: membuf descriptor
	1239	+ * @tsk: The task from which to copy the saved xstate
	1240	+ * @copy_mode: The requested copy mode
	1241	+ *
	1242	+ * Converts from kernel XSAVE or XSAVES compacted format to UABI conforming
	1243	+ * format, i.e. from the kernel internal hardware dependent storage format
	1244	+ * to the requested @mode. UABI XSTATE is always uncompacted!
	1245	+ *
	1246	+ * It supports partial copy but @to.pos always starts from zero.
	1247	+ */
	1248	+void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk,
	1249	+ enum xstate_copy_mode copy_mode)
	1250	+{
	1251	+ __copy_xstate_to_uabi_buf(to, tsk->thread.fpu.fpstate,
	1252	+ tsk->thread.pkru, copy_mode);
	1253	+}
	1254	+
1081	1255	static int copy_from_buffer(void *dst, unsigned int offset, unsigned int size,
1082	1256	const void kbuf, const void __user ubuf)
1083	1257	{
@@ -1091,9 +1265,10 @@ static int copy_from_buffer(void *dst, unsigned int offset, unsigned int size, · 7 unmodified lines
1091	1265	}
1092	1266
1093	1267
1094		−static int copy_uabi_to_xstate(struct xregs_state xsave, const void kbuf,
	1268	+static int copy_uabi_to_xstate(struct fpstate fpstate, const void kbuf,
1095	1269	const void __user *ubuf)
1096	1270	{
	1271	+ struct xregs_state *xsave = &fpstate->regs.xsave;
1097	1272	unsigned int offset, size;
1098	1273	struct xstate_header hdr;
1099	1274	u64 mask;
@@ -1103,7 +1278,7 @@ static int copy_uabi_to_xstate(struct xregs_state xsave, const void kbuf, · 3 unmodified lines
1103	1278	if (copy_from_buffer(&hdr, offset, sizeof(hdr), kbuf, ubuf))
1104	1279	return -EFAULT;
1105	1280
1106		− if (validate_user_xstate_header(&hdr))
	1281	+ if (validate_user_xstate_header(&hdr, fpstate))
1107	1282	return -EINVAL;
1108	1283
1109	1284	/* Validate MXCSR when any of the related features is in use */
@@ -1156,12 +1331,11 @@ static int copy_uabi_to_xstate(struct xregs_state xsave, const void kbuf, · 46 unmodified lines
1156	1331
1157	1332	/*
1158	1333	* Convert from a ptrace standard-format kernel buffer to kernel XSAVE[S]
1159		− * format and copy to the target thread. This is called from
1160		− * xstateregs_set().
	1334	+ * format and copy to the target thread. Used by ptrace and KVM.
1161	1335	*/
1162		−int copy_uabi_from_kernel_to_xstate(struct xregs_state xsave, const void kbuf)
	1336	+int copy_uabi_from_kernel_to_xstate(struct fpstate fpstate, const void kbuf)
1163	1337	{
1164		− return copy_uabi_to_xstate(xsave, kbuf, NULL);
	1338	+ return copy_uabi_to_xstate(fpstate, kbuf, NULL);
1165	1339	}
1166	1340
1167	1341	/*
@@ -1169,26 +1343,20 @@ int copy_uabi_from_kernel_to_xstate(struct xregs_state xsave, const void kbuf) · 1 unmodified line
1169	1343	* XSAVE[S] format and copy to the target thread. This is called from the
1170	1344	* sigreturn() and rt_sigreturn() system calls.
1171	1345	*/
1172		−int copy_sigframe_from_user_to_xstate(struct xregs_state *xsave,
	1346	+int copy_sigframe_from_user_to_xstate(struct fpstate *fpstate,
1173	1347	const void __user *ubuf)
1174	1348	{
1175		− return copy_uabi_to_xstate(xsave, NULL, ubuf);
	1349	+ return copy_uabi_to_xstate(fpstate, NULL, ubuf);
1176	1350	}
1177	1351
1178		−static bool validate_xsaves_xrstors(u64 mask)
	1352	+static bool validate_independent_components(u64 mask)
1179	1353	{
1180	1354	u64 xchk;
1181	1355
1182	1356	if (WARN_ON_FPU(!cpu_feature_enabled(X86_FEATURE_XSAVES)))
1183	1357	return false;
1184		− /*
1185		− * Validate that this is either a task->fpstate related component
1186		− * subset or an independent one.
1187		− */
1188		− if (mask & xfeatures_mask_independent())
1189		− xchk = ~xfeatures_mask_independent();
1190		− else
1191		− xchk = ~xfeatures_mask_all;
	1358	+
	1359	+ xchk = ~xfeatures_mask_independent();
1192	1360
1193	1361	if (WARN_ON_ONCE(!mask \|\| mask & xchk))
1194	1362	return false;
@@ -1206,14 +1374,13 @@ static bool validate_xsaves_xrstors(u64 mask) · 11 unmodified lines
1206	1374	* buffer should be zeroed otherwise a consecutive XRSTORS from that buffer
1207	1375	* can #GP.
1208	1376	*
1209		− * The feature mask must either be a subset of the independent features or
1210		− * a subset of the task->fpstate related features.
	1377	+ * The feature mask must be a subset of the independent features.
1211	1378	*/
1212	1379	void xsaves(struct xregs_state *xstate, u64 mask)
1213	1380	{
1214	1381	int err;
1215	1382
1216		− if (!validate_xsaves_xrstors(mask))
	1383	+ if (!validate_independent_components(mask))
1217	1384	return;
1218	1385
1219	1386	XSTATE_OP(XSAVES, xstate, (u32)mask, (u32)(mask >> 32), err);
@@ -1231,20 +1398,379 @@ void xsaves(struct xregs_state *xstate, u64 mask) · 11 unmodified lines
1231	1398	* Proper usage is to restore the state which was saved with
1232	1399	* xsaves() into @xstate.
1233	1400	*
1234		− * The feature mask must either be a subset of the independent features or
1235		− * a subset of the task->fpstate related features.
	1401	+ * The feature mask must be a subset of the independent features.
1236	1402	*/
1237	1403	void xrstors(struct xregs_state *xstate, u64 mask)
1238	1404	{
1239	1405	int err;
1240	1406
1241		− if (!validate_xsaves_xrstors(mask))
	1407	+ if (!validate_independent_components(mask))
1242	1408	return;
1243	1409
1244	1410	XSTATE_OP(XRSTORS, xstate, (u32)mask, (u32)(mask >> 32), err);
1245	1411	WARN_ON_ONCE(err);
1246	1412	}
1247	1413
	1414	+#if IS_ENABLED(CONFIG_KVM)
	1415	+void fpstate_clear_xstate_component(struct fpstate *fps, unsigned int xfeature)
	1416	+{
	1417	+ void *addr = get_xsave_addr(&fps->regs.xsave, xfeature);
	1418	+
	1419	+ if (addr)
	1420	+ memset(addr, 0, xstate_sizes[xfeature]);
	1421	+}
	1422	+EXPORT_SYMBOL_GPL(fpstate_clear_xstate_component);
	1423	+#endif
	1424	+
	1425	+#ifdef CONFIG_X86_64
	1426	+
	1427	+#ifdef CONFIG_X86_DEBUG_FPU
	1428	+/*
	1429	+ * Ensure that a subsequent XSAVE* or XRSTOR* instruction with RFBM=@mask
	1430	+ * can safely operate on the @fpstate buffer.
	1431	+ */
	1432	+static bool xstate_op_valid(struct fpstate *fpstate, u64 mask, bool rstor)
	1433	+{
	1434	+ u64 xfd = __this_cpu_read(xfd_state);
	1435	+
	1436	+ if (fpstate->xfd == xfd)
	1437	+ return true;
	1438	+
	1439	+ /*
	1440	+ * The XFD MSR does not match fpstate->xfd. That's invalid when
	1441	+ * the passed in fpstate is current's fpstate.
	1442	+ */
	1443	+ if (fpstate->xfd == current->thread.fpu.fpstate->xfd)
	1444	+ return false;
	1445	+
	1446	+ /*
	1447	+ * XRSTOR(S) from init_fpstate are always correct as it will just
	1448	+ * bring all components into init state and not read from the
	1449	+ * buffer. XSAVE(S) raises #PF after init.
	1450	+ */
	1451	+ if (fpstate == &init_fpstate)
	1452	+ return rstor;
	1453	+
	1454	+ /*
	1455	+ * XSAVE(S): clone(), fpu_swap_kvm_fpu()
	1456	+ * XRSTORS(S): fpu_swap_kvm_fpu()
	1457	+ */
	1458	+
	1459	+ /*
	1460	+ * No XSAVE/XRSTOR instructions (except XSAVE itself) touch
	1461	+ * the buffer area for XFD-disabled state components.
	1462	+ */
	1463	+ mask &= ~xfd;
	1464	+
	1465	+ /*
	1466	+ * Remove features which are valid in fpstate. They
	1467	+ * have space allocated in fpstate.
	1468	+ */
	1469	+ mask &= ~fpstate->xfeatures;
	1470	+
	1471	+ /*
	1472	+ * Any remaining state components in 'mask' might be written
	1473	+ * by XSAVE/XRSTOR. Fail validation it found.
	1474	+ */
	1475	+ return !mask;
	1476	+}
	1477	+
	1478	+void xfd_validate_state(struct fpstate *fpstate, u64 mask, bool rstor)
	1479	+{
	1480	+ WARN_ON_ONCE(!xstate_op_valid(fpstate, mask, rstor));
	1481	+}
	1482	+#endif /* CONFIG_X86_DEBUG_FPU */
	1483	+
	1484	+static int __init xfd_update_static_branch(void)
	1485	+{
	1486	+ /*
	1487	+ * If init_fpstate.xfd has bits set then dynamic features are
	1488	+ * available and the dynamic sizing must be enabled.
	1489	+ */
	1490	+ if (init_fpstate.xfd)
	1491	+ static_branch_enable(&__fpu_state_size_dynamic);
	1492	+ return 0;
	1493	+}
	1494	+arch_initcall(xfd_update_static_branch)
	1495	+
	1496	+void fpstate_free(struct fpu *fpu)
	1497	+{
	1498	+ if (fpu->fpstate && fpu->fpstate != &fpu->__fpstate)
	1499	+ vfree(fpu->fpstate);
	1500	+}
	1501	+
	1502	+/**
	1503	+ * fpu_install_fpstate - Update the active fpstate in the FPU
	1504	+ *
	1505	+ * @fpu: A struct fpu * pointer
	1506	+ * @newfps: A struct fpstate * pointer
	1507	+ *
	1508	+ * Returns: A null pointer if the last active fpstate is the embedded
	1509	+ * one or the new fpstate is already installed;
	1510	+ * otherwise, a pointer to the old fpstate which has to
	1511	+ * be freed by the caller.
	1512	+ */
	1513	+static struct fpstate fpu_install_fpstate(struct fpu fpu,
	1514	+ struct fpstate *newfps)
	1515	+{
	1516	+ struct fpstate *oldfps = fpu->fpstate;
	1517	+
	1518	+ if (fpu->fpstate == newfps)
	1519	+ return NULL;
	1520	+
	1521	+ fpu->fpstate = newfps;
	1522	+ return oldfps != &fpu->__fpstate ? oldfps : NULL;
	1523	+}
	1524	+
	1525	+/**
	1526	+ * fpstate_realloc - Reallocate struct fpstate for the requested new features
	1527	+ *
	1528	+ * @xfeatures: A bitmap of xstate features which extend the enabled features
	1529	+ * of that task
	1530	+ * @ksize: The required size for the kernel buffer
	1531	+ * @usize: The required size for user space buffers
	1532	+ *
	1533	+ * Note vs. vmalloc(): If the task with a vzalloc()-allocated buffer
	1534	+ * terminates quickly, vfree()-induced IPIs may be a concern, but tasks
	1535	+ * with large states are likely to live longer.
	1536	+ *
	1537	+ * Returns: 0 on success, -ENOMEM on allocation error.
	1538	+ */
	1539	+static int fpstate_realloc(u64 xfeatures, unsigned int ksize,
	1540	+ unsigned int usize)
	1541	+{
	1542	+ struct fpu *fpu = &current->thread.fpu;
	1543	+ struct fpstate curfps, newfps = NULL;
	1544	+ unsigned int fpsize;
	1545	+
	1546	+ curfps = fpu->fpstate;
	1547	+ fpsize = ksize + ALIGN(offsetof(struct fpstate, regs), 64);
	1548	+
	1549	+ newfps = vzalloc(fpsize);
	1550	+ if (!newfps)
	1551	+ return -ENOMEM;
	1552	+ newfps->size = ksize;
	1553	+ newfps->user_size = usize;
	1554	+ newfps->is_valloc = true;
	1555	+
	1556	+ fpregs_lock();
	1557	+ /*
	1558	+ * Ensure that the current state is in the registers before
	1559	+ * swapping fpstate as that might invalidate it due to layout
	1560	+ * changes.
	1561	+ */
	1562	+ if (test_thread_flag(TIF_NEED_FPU_LOAD))
	1563	+ fpregs_restore_userregs();
	1564	+
	1565	+ newfps->xfeatures = curfps->xfeatures \| xfeatures;
	1566	+ newfps->user_xfeatures = curfps->user_xfeatures \| xfeatures;
	1567	+ newfps->xfd = curfps->xfd & ~xfeatures;
	1568	+
	1569	+ curfps = fpu_install_fpstate(fpu, newfps);
	1570	+
	1571	+ /* Do the final updates within the locked region */
	1572	+ xstate_init_xcomp_bv(&newfps->regs.xsave, newfps->xfeatures);
	1573	+ xfd_update_state(newfps);
	1574	+
	1575	+ fpregs_unlock();
	1576	+
	1577	+ vfree(curfps);
	1578	+ return 0;
	1579	+}
	1580	+
	1581	+static int validate_sigaltstack(unsigned int usize)
	1582	+{
	1583	+ struct task_struct thread, leader = current->group_leader;
	1584	+ unsigned long framesize = get_sigframe_size();
	1585	+
	1586	+ lockdep_assert_held(&current->sighand->siglock);
	1587	+
	1588	+ /* get_sigframe_size() is based on fpu_user_cfg.max_size */
	1589	+ framesize -= fpu_user_cfg.max_size;
	1590	+ framesize += usize;
	1591	+ for_each_thread(leader, thread) {
	1592	+ if (thread->sas_ss_size && thread->sas_ss_size < framesize)
	1593	+ return -ENOSPC;
	1594	+ }
	1595	+ return 0;
	1596	+}
	1597	+
	1598	+static int __xstate_request_perm(u64 permitted, u64 requested)
	1599	+{
	1600	+ /*
	1601	+ * This deliberately does not exclude !XSAVES as we still might
	1602	+ * decide to optionally context switch XCR0 or talk the silicon
	1603	+ * vendors into extending XFD for the pre AMX states, especially
	1604	+ * AVX512.
	1605	+ */
	1606	+ bool compacted = cpu_feature_enabled(X86_FEATURE_XSAVES);
	1607	+ struct fpu *fpu = &current->group_leader->thread.fpu;
	1608	+ unsigned int ksize, usize;
	1609	+ u64 mask;
	1610	+ int ret;
	1611	+
	1612	+ /* Check whether fully enabled */
	1613	+ if ((permitted & requested) == requested)
	1614	+ return 0;
	1615	+
	1616	+ /* Calculate the resulting kernel state size */
	1617	+ mask = permitted \| requested;
	1618	+ ksize = xstate_calculate_size(mask, compacted);
	1619	+
	1620	+ /* Calculate the resulting user state size */
	1621	+ mask &= XFEATURE_MASK_USER_SUPPORTED;
	1622	+ usize = xstate_calculate_size(mask, false);
	1623	+
	1624	+ ret = validate_sigaltstack(usize);
	1625	+ if (ret)
	1626	+ return ret;
	1627	+
	1628	+ /* Pairs with the READ_ONCE() in xstate_get_group_perm() */
	1629	+ WRITE_ONCE(fpu->perm.__state_perm, requested);
	1630	+ /* Protected by sighand lock */
	1631	+ fpu->perm.__state_size = ksize;
	1632	+ fpu->perm.__user_state_size = usize;
	1633	+ return ret;
	1634	+}
	1635	+
	1636	+/*
	1637	+ * Permissions array to map facilities with more than one component
	1638	+ */
	1639	+static const u64 xstate_prctl_req[XFEATURE_MAX] = {
	1640	+ [XFEATURE_XTILE_DATA] = XFEATURE_MASK_XTILE_DATA,
	1641	+};
	1642	+
	1643	+static int xstate_request_perm(unsigned long idx)
	1644	+{
	1645	+ u64 permitted, requested;
	1646	+ int ret;
	1647	+
	1648	+ if (idx >= XFEATURE_MAX)
	1649	+ return -EINVAL;
	1650	+
	1651	+ /*
	1652	+ * Look up the facility mask which can require more than
	1653	+ * one xstate component.
	1654	+ */
	1655	+ idx = array_index_nospec(idx, ARRAY_SIZE(xstate_prctl_req));
	1656	+ requested = xstate_prctl_req[idx];
	1657	+ if (!requested)
	1658	+ return -EOPNOTSUPP;
	1659	+
	1660	+ if ((fpu_user_cfg.max_features & requested) != requested)
	1661	+ return -EOPNOTSUPP;
	1662	+
	1663	+ /* Lockless quick check */
	1664	+ permitted = xstate_get_host_group_perm();
	1665	+ if ((permitted & requested) == requested)
	1666	+ return 0;
	1667	+
	1668	+ /* Protect against concurrent modifications */
	1669	+ spin_lock_irq(&current->sighand->siglock);
	1670	+ permitted = xstate_get_host_group_perm();
	1671	+ ret = __xstate_request_perm(permitted, requested);
	1672	+ spin_unlock_irq(&current->sighand->siglock);
	1673	+ return ret;
	1674	+}
	1675	+
	1676	+int xfd_enable_feature(u64 xfd_err)
	1677	+{
	1678	+ u64 xfd_event = xfd_err & XFEATURE_MASK_USER_DYNAMIC;
	1679	+ unsigned int ksize, usize;
	1680	+ struct fpu *fpu;
	1681	+
	1682	+ if (!xfd_event) {
	1683	+ pr_err_once("XFD: Invalid xfd error: %016llx\n", xfd_err);
	1684	+ return 0;
	1685	+ }
	1686	+
	1687	+ /* Protect against concurrent modifications */
	1688	+ spin_lock_irq(&current->sighand->siglock);
	1689	+
	1690	+ /* If not permitted let it die */
	1691	+ if ((xstate_get_host_group_perm() & xfd_event) != xfd_event) {
	1692	+ spin_unlock_irq(&current->sighand->siglock);
	1693	+ return -EPERM;
	1694	+ }
	1695	+
	1696	+ fpu = &current->group_leader->thread.fpu;
	1697	+ ksize = fpu->perm.__state_size;
	1698	+ usize = fpu->perm.__user_state_size;
	1699	+ /*
	1700	+ * The feature is permitted. State size is sufficient. Dropping
	1701	+ * the lock is safe here even if more features are added from
	1702	+ * another task, the retrieved buffer sizes are valid for the
	1703	+ * currently requested feature(s).
	1704	+ */
	1705	+ spin_unlock_irq(&current->sighand->siglock);
	1706	+
	1707	+ /*
	1708	+ * Try to allocate a new fpstate. If that fails there is no way
	1709	+ * out.
	1710	+ */
	1711	+ if (fpstate_realloc(xfd_event, ksize, usize))
	1712	+ return -EFAULT;
	1713	+ return 0;
	1714	+}
	1715	+#else /* CONFIG_X86_64 */
	1716	+static inline int xstate_request_perm(unsigned long idx)
	1717	+{
	1718	+ return -EPERM;
	1719	+}
	1720	+#endif /* !CONFIG_X86_64 */
	1721	+
	1722	+/**
	1723	+ * fpu_xstate_prctl - xstate permission operations
	1724	+ * @tsk: Redundant pointer to current
	1725	+ * @option: A subfunction of arch_prctl()
	1726	+ * @arg2: option argument
	1727	+ * Return: 0 if successful; otherwise, an error code
	1728	+ *
	1729	+ * Option arguments:
	1730	+ *
	1731	+ * ARCH_GET_XCOMP_SUPP: Pointer to user space u64 to store the info
	1732	+ * ARCH_GET_XCOMP_PERM: Pointer to user space u64 to store the info
	1733	+ * ARCH_REQ_XCOMP_PERM: Facility number requested
	1734	+ *
	1735	+ * For facilities which require more than one XSTATE component, the request
	1736	+ * must be the highest state component number related to that facility,
	1737	+ * e.g. for AMX which requires XFEATURE_XTILE_CFG(17) and
	1738	+ * XFEATURE_XTILE_DATA(18) this would be XFEATURE_XTILE_DATA(18).
	1739	+ */
	1740	+long fpu_xstate_prctl(struct task_struct *tsk, int option, unsigned long arg2)
	1741	+{
	1742	+ u64 __user uptr = (u64 __user )arg2;
	1743	+ u64 permitted, supported;
	1744	+ unsigned long idx = arg2;
	1745	+
	1746	+ if (tsk != current)
	1747	+ return -EPERM;
	1748	+
	1749	+ switch (option) {
	1750	+ case ARCH_GET_XCOMP_SUPP:
	1751	+ supported = fpu_user_cfg.max_features \| fpu_user_cfg.legacy_features;
	1752	+ return put_user(supported, uptr);
	1753	+
	1754	+ case ARCH_GET_XCOMP_PERM:
	1755	+ /*
	1756	+ * Lockless snapshot as it can also change right after the
	1757	+ * dropping the lock.
	1758	+ */
	1759	+ permitted = xstate_get_host_group_perm();
	1760	+ permitted &= XFEATURE_MASK_USER_SUPPORTED;
	1761	+ return put_user(permitted, uptr);
	1762	+
	1763	+ case ARCH_REQ_XCOMP_PERM:
	1764	+ if (!IS_ENABLED(CONFIG_X86_64))
	1765	+ return -EOPNOTSUPP;
	1766	+
	1767	+ return xstate_request_perm(idx);
	1768	+
	1769	+ default:
	1770	+ return -EINVAL;
	1771	+ }
	1772	+}
	1773	+
1248	1774	#ifdef CONFIG_PROC_PID_ARCH_STATUS
1249	1775	/*
1250	1776	* Report the amount of time elapsed in millisecond since last AVX512

arch/x86/kernel/fpu/xstate.h+278−0

@@ -0,0 +1,278 @@
	1	+/* SPDX-License-Identifier: GPL-2.0 */
	2	+#ifndef __X86_KERNEL_FPU_XSTATE_H
	3	+#define __X86_KERNEL_FPU_XSTATE_H
	4	+
	5	+#include <asm/cpufeature.h>
	6	+#include <asm/fpu/xstate.h>
	7	+
	8	+#ifdef CONFIG_X86_64
	9	+DECLARE_PER_CPU(u64, xfd_state);
	10	+#endif
	11	+
	12	+static inline void xstate_init_xcomp_bv(struct xregs_state *xsave, u64 mask)
	13	+{
	14	+ /*
	15	+ * XRSTORS requires these bits set in xcomp_bv, or it will
	16	+ * trigger #GP:
	17	+ */
	18	+ if (cpu_feature_enabled(X86_FEATURE_XSAVES))
	19	+ xsave->header.xcomp_bv = mask \| XCOMP_BV_COMPACTED_FORMAT;
	20	+}
	21	+
	22	+static inline u64 xstate_get_host_group_perm(void)
	23	+{
	24	+ /* Pairs with WRITE_ONCE() in xstate_request_perm() */
	25	+ return READ_ONCE(current->group_leader->thread.fpu.perm.__state_perm);
	26	+}
	27	+
	28	+enum xstate_copy_mode {
	29	+ XSTATE_COPY_FP,
	30	+ XSTATE_COPY_FX,
	31	+ XSTATE_COPY_XSAVE,
	32	+};
	33	+
	34	+struct membuf;
	35	+extern void __copy_xstate_to_uabi_buf(struct membuf to, struct fpstate *fpstate,
	36	+ u32 pkru_val, enum xstate_copy_mode copy_mode);
	37	+extern void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk,
	38	+ enum xstate_copy_mode mode);
	39	+extern int copy_uabi_from_kernel_to_xstate(struct fpstate fpstate, const void kbuf);
	40	+extern int copy_sigframe_from_user_to_xstate(struct fpstate fpstate, const void __user ubuf);
	41	+
	42	+
	43	+extern void fpu__init_cpu_xstate(void);
	44	+extern void fpu__init_system_xstate(unsigned int legacy_size);
	45	+
	46	+extern void get_xsave_addr(struct xregs_state xsave, int xfeature_nr);
	47	+
	48	+static inline u64 xfeatures_mask_supervisor(void)
	49	+{
	50	+ return fpu_kernel_cfg.max_features & XFEATURE_MASK_SUPERVISOR_SUPPORTED;
	51	+}
	52	+
	53	+static inline u64 xfeatures_mask_independent(void)
	54	+{
	55	+ if (!cpu_feature_enabled(X86_FEATURE_ARCH_LBR))
	56	+ return XFEATURE_MASK_INDEPENDENT & ~XFEATURE_MASK_LBR;
	57	+
	58	+ return XFEATURE_MASK_INDEPENDENT;
	59	+}
	60	+
	61	+/* XSAVE/XRSTOR wrapper functions */
	62	+
	63	+#ifdef CONFIG_X86_64
	64	+#define REX_PREFIX "0x48, "
	65	+#else
	66	+#define REX_PREFIX
	67	+#endif
	68	+
	69	+/* These macros all use (%edi)/(%rdi) as the single memory argument. */
	70	+#define XSAVE ".byte " REX_PREFIX "0x0f,0xae,0x27"
	71	+#define XSAVEOPT ".byte " REX_PREFIX "0x0f,0xae,0x37"
	72	+#define XSAVES ".byte " REX_PREFIX "0x0f,0xc7,0x2f"
	73	+#define XRSTOR ".byte " REX_PREFIX "0x0f,0xae,0x2f"
	74	+#define XRSTORS ".byte " REX_PREFIX "0x0f,0xc7,0x1f"
	75	+
	76	+/*
	77	+ * After this @err contains 0 on success or the trap number when the
	78	+ * operation raises an exception.
	79	+ */
	80	+#define XSTATE_OP(op, st, lmask, hmask, err) \
	81	+ asm volatile("1:" op "\n\t" \
	82	+ "xor %[err], %[err]\n" \
	83	+ "2:\n\t" \
	84	+ _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_FAULT_MCE_SAFE) \
	85	+ : [err] "=a" (err) \
	86	+ : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \
	87	+ : "memory")
	88	+
	89	+/*
	90	+ * If XSAVES is enabled, it replaces XSAVEOPT because it supports a compact
	91	+ * format and supervisor states in addition to modified optimization in
	92	+ * XSAVEOPT.
	93	+ *
	94	+ * Otherwise, if XSAVEOPT is enabled, XSAVEOPT replaces XSAVE because XSAVEOPT
	95	+ * supports modified optimization which is not supported by XSAVE.
	96	+ *
	97	+ * We use XSAVE as a fallback.
	98	+ *
	99	+ * The 661 label is defined in the ALTERNATIVE* macros as the address of the
	100	+ * original instruction which gets replaced. We need to use it here as the
	101	+ * address of the instruction where we might get an exception at.
	102	+ */
	103	+#define XSTATE_XSAVE(st, lmask, hmask, err) \
	104	+ asm volatile(ALTERNATIVE_2(XSAVE, \
	105	+ XSAVEOPT, X86_FEATURE_XSAVEOPT, \
	106	+ XSAVES, X86_FEATURE_XSAVES) \
	107	+ "\n" \
	108	+ "xor %[err], %[err]\n" \
	109	+ "3:\n" \
	110	+ ".pushsection .fixup,\"ax\"\n" \
	111	+ "4: movl $-2, %[err]\n" \
	112	+ "jmp 3b\n" \
	113	+ ".popsection\n" \
	114	+ _ASM_EXTABLE(661b, 4b) \
	115	+ : [err] "=r" (err) \
	116	+ : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \
	117	+ : "memory")
	118	+
	119	+/*
	120	+ * Use XRSTORS to restore context if it is enabled. XRSTORS supports compact
	121	+ * XSAVE area format.
	122	+ */
	123	+#define XSTATE_XRESTORE(st, lmask, hmask) \
	124	+ asm volatile(ALTERNATIVE(XRSTOR, \
	125	+ XRSTORS, X86_FEATURE_XSAVES) \
	126	+ "\n" \
	127	+ "3:\n" \
	128	+ _ASM_EXTABLE_TYPE(661b, 3b, EX_TYPE_FPU_RESTORE) \
	129	+ : \
	130	+ : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \
	131	+ : "memory")
	132	+
	133	+#if defined(CONFIG_X86_64) && defined(CONFIG_X86_DEBUG_FPU)
	134	+extern void xfd_validate_state(struct fpstate *fpstate, u64 mask, bool rstor);
	135	+#else
	136	+static inline void xfd_validate_state(struct fpstate *fpstate, u64 mask, bool rstor) { }
	137	+#endif
	138	+
	139	+#ifdef CONFIG_X86_64
	140	+static inline void xfd_update_state(struct fpstate *fpstate)
	141	+{
	142	+ if (fpu_state_size_dynamic()) {
	143	+ u64 xfd = fpstate->xfd;
	144	+
	145	+ if (__this_cpu_read(xfd_state) != xfd) {
	146	+ wrmsrl(MSR_IA32_XFD, xfd);
	147	+ __this_cpu_write(xfd_state, xfd);
	148	+ }
	149	+ }
	150	+}
	151	+#else
	152	+static inline void xfd_update_state(struct fpstate *fpstate) { }
	153	+#endif
	154	+
	155	+/*
	156	+ * Save processor xstate to xsave area.
	157	+ *
	158	+ * Uses either XSAVE or XSAVEOPT or XSAVES depending on the CPU features
	159	+ * and command line options. The choice is permanent until the next reboot.
	160	+ */
	161	+static inline void os_xsave(struct fpstate *fpstate)
	162	+{
	163	+ u64 mask = fpstate->xfeatures;
	164	+ u32 lmask = mask;
	165	+ u32 hmask = mask >> 32;
	166	+ int err;
	167	+
	168	+ WARN_ON_FPU(!alternatives_patched);
	169	+ xfd_validate_state(fpstate, mask, false);
	170	+
	171	+ XSTATE_XSAVE(&fpstate->regs.xsave, lmask, hmask, err);
	172	+
	173	+ /* We should never fault when copying to a kernel buffer: */
	174	+ WARN_ON_FPU(err);
	175	+}
	176	+
	177	+/*
	178	+ * Restore processor xstate from xsave area.
	179	+ *
	180	+ * Uses XRSTORS when XSAVES is used, XRSTOR otherwise.
	181	+ */
	182	+static inline void os_xrstor(struct fpstate *fpstate, u64 mask)
	183	+{
	184	+ u32 lmask = mask;
	185	+ u32 hmask = mask >> 32;
	186	+
	187	+ xfd_validate_state(fpstate, mask, true);
	188	+ XSTATE_XRESTORE(&fpstate->regs.xsave, lmask, hmask);
	189	+}
	190	+
	191	+/* Restore of supervisor state. Does not require XFD */
	192	+static inline void os_xrstor_supervisor(struct fpstate *fpstate)
	193	+{
	194	+ u64 mask = xfeatures_mask_supervisor();
	195	+ u32 lmask = mask;
	196	+ u32 hmask = mask >> 32;
	197	+
	198	+ XSTATE_XRESTORE(&fpstate->regs.xsave, lmask, hmask);
	199	+}
	200	+
	201	+/*
	202	+ * Save xstate to user space xsave area.
	203	+ *
	204	+ * We don't use modified optimization because xrstor/xrstors might track
	205	+ * a different application.
	206	+ *
	207	+ * We don't use compacted format xsave area for backward compatibility for
	208	+ * old applications which don't understand the compacted format of the
	209	+ * xsave area.
	210	+ *
	211	+ * The caller has to zero buf::header before calling this because XSAVE*
	212	+ * does not touch the reserved fields in the header.
	213	+ */
	214	+static inline int xsave_to_user_sigframe(struct xregs_state __user *buf)
	215	+{
	216	+ /*
	217	+ * Include the features which are not xsaved/rstored by the kernel
	218	+ * internally, e.g. PKRU. That's user space ABI and also required
	219	+ * to allow the signal handler to modify PKRU.
	220	+ */
	221	+ struct fpstate *fpstate = current->thread.fpu.fpstate;
	222	+ u64 mask = fpstate->user_xfeatures;
	223	+ u32 lmask = mask;
	224	+ u32 hmask = mask >> 32;
	225	+ int err;
	226	+
	227	+ xfd_validate_state(fpstate, mask, false);
	228	+
	229	+ stac();
	230	+ XSTATE_OP(XSAVE, buf, lmask, hmask, err);
	231	+ clac();
	232	+
	233	+ return err;
	234	+}
	235	+
	236	+/*
	237	+ * Restore xstate from user space xsave area.
	238	+ */
	239	+static inline int xrstor_from_user_sigframe(struct xregs_state __user *buf, u64 mask)
	240	+{
	241	+ struct xregs_state xstate = ((__force struct xregs_state )buf);
	242	+ u32 lmask = mask;
	243	+ u32 hmask = mask >> 32;
	244	+ int err;
	245	+
	246	+ xfd_validate_state(current->thread.fpu.fpstate, mask, true);
	247	+
	248	+ stac();
	249	+ XSTATE_OP(XRSTOR, xstate, lmask, hmask, err);
	250	+ clac();
	251	+
	252	+ return err;
	253	+}
	254	+
	255	+/*
	256	+ * Restore xstate from kernel space xsave area, return an error code instead of
	257	+ * an exception.
	258	+ */
	259	+static inline int os_xrstor_safe(struct fpstate *fpstate, u64 mask)
	260	+{
	261	+ struct xregs_state *xstate = &fpstate->regs.xsave;
	262	+ u32 lmask = mask;
	263	+ u32 hmask = mask >> 32;
	264	+ int err;
	265	+
	266	+ /* Ensure that XFD is up to date */
	267	+ xfd_update_state(fpstate);
	268	+
	269	+ if (cpu_feature_enabled(X86_FEATURE_XSAVES))
	270	+ XSTATE_OP(XRSTORS, xstate, lmask, hmask, err);
	271	+ else
	272	+ XSTATE_OP(XRSTOR, xstate, lmask, hmask, err);
	273	+
	274	+ return err;
	275	+}
	276	+
	277	+
	278	+#endif

arch/x86/kernel/process.c+23−4

@@ -30,7 +30,9 @@ · 29 unmodified lines
30	30	#include <asm/apic.h>
31	31	#include <linux/uaccess.h>
32	32	#include <asm/mwait.h>
33		−#include <asm/fpu/internal.h>
	33	+#include <asm/fpu/api.h>
	34	+#include <asm/fpu/sched.h>
	35	+#include <asm/fpu/xstate.h>
34	36	#include <asm/debugreg.h>
35	37	#include <asm/nmi.h>
36	38	#include <asm/tlbflush.h>
@@ -88,8 +90,19 @@ int arch_dup_task_struct(struct task_struct dst, struct task_struct src) · 51 unmodified lines
88	90	#ifdef CONFIG_VM86
89	91	dst->thread.vm86 = NULL;
90	92	#endif
91		− return fpu_clone(dst);
	93	+ /* Drop the copied pointer to current's fpstate */
	94	+ dst->thread.fpu.fpstate = NULL;
	95	+
	96	+ return 0;
	97	+}
	98	+
	99	+#ifdef CONFIG_X86_64
	100	+void arch_release_task_struct(struct task_struct *tsk)
	101	+{
	102	+ if (fpu_state_size_dynamic())
	103	+ fpstate_free(&tsk->thread.fpu);
92	104	}
	105	+#endif
93	106
94	107	/*
95	108	* Free thread data structures etc..
@@ -155,6 +168,8 @@ int copy_thread(unsigned long clone_flags, unsigned long sp, unsigned long arg, · 59 unmodified lines
155	168	frame->flags = X86_EFLAGS_FIXED;
156	169	#endif
157	170
	171	+ fpu_clone(p, clone_flags);
	172	+
158	173	/* Kernel thread ? */
159	174	if (unlikely(p->flags & PF_KTHREAD)) {
160	175	p->thread.pkru = pkru_get_init_value();
@@ -962,13 +977,17 @@ unsigned long __get_wchan(struct task_struct *p) · 801 unmodified lines
962	977	}
963	978
964	979	long do_arch_prctl_common(struct task_struct *task, int option,
965		− unsigned long cpuid_enabled)
	980	+ unsigned long arg2)
966	981	{
967	982	switch (option) {
968	983	case ARCH_GET_CPUID:
969	984	return get_cpuid_mode();
970	985	case ARCH_SET_CPUID:
971		− return set_cpuid_mode(task, cpuid_enabled);
	986	+ return set_cpuid_mode(task, arg2);
	987	+ case ARCH_GET_XCOMP_SUPP:
	988	+ case ARCH_GET_XCOMP_PERM:
	989	+ case ARCH_REQ_XCOMP_PERM:
	990	+ return fpu_xstate_prctl(task, option, arg2);
972	991	}
973	992
974	993	return -EINVAL;

arch/x86/kernel/process_32.c+2−3

@@ -41,7 +41,7 @@ · 40 unmodified lines
41	41
42	42	#include <asm/ldt.h>
43	43	#include <asm/processor.h>
44		−#include <asm/fpu/internal.h>
	44	+#include <asm/fpu/sched.h>
45	45	#include <asm/desc.h>
46	46
47	47	#include <linux/err.h>
@@ -160,7 +160,6 @@ __switch_to(struct task_struct prev_p, struct task_struct next_p) · 112 unmodified lines
160	160	struct thread_struct *prev = &prev_p->thread,
161	161	*next = &next_p->thread;
162	162	struct fpu *prev_fpu = &prev->fpu;
163		− struct fpu *next_fpu = &next->fpu;
164	163	int cpu = smp_processor_id();
165	164
166	165	/* never put a printk in __switch_to... printk() calls wake_up() indirectly /
@@ -213,7 +212,7 @@ __switch_to(struct task_struct prev_p, struct task_struct next_p) · 46 unmodified lines
213	212
214	213	this_cpu_write(current_task, next_p);
215	214
216		− switch_fpu_finish(next_fpu);
	215	+ switch_fpu_finish();
217	216
218	217	/* Load the Intel cache allocation PQR MSR. */
219	218	resctrl_sched_in();

arch/x86/kernel/process_64.c+2−3

@@ -42,7 +42,7 @@ · 41 unmodified lines
42	42
43	43	#include <asm/processor.h>
44	44	#include <asm/pkru.h>
45		−#include <asm/fpu/internal.h>
	45	+#include <asm/fpu/sched.h>
46	46	#include <asm/mmu_context.h>
47	47	#include <asm/prctl.h>
48	48	#include <asm/desc.h>
@@ -559,7 +559,6 @@ __switch_to(struct task_struct prev_p, struct task_struct next_p) · 510 unmodified lines
559	559	struct thread_struct *prev = &prev_p->thread;
560	560	struct thread_struct *next = &next_p->thread;
561	561	struct fpu *prev_fpu = &prev->fpu;
562		− struct fpu *next_fpu = &next->fpu;
563	562	int cpu = smp_processor_id();
564	563
565	564	WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_ENTRY) &&
@@ -620,7 +619,7 @@ __switch_to(struct task_struct prev_p, struct task_struct next_p) · 54 unmodified lines
620	619	this_cpu_write(current_task, next_p);
621	620	this_cpu_write(cpu_current_top_of_stack, task_top_of_stack(next_p));
622	621
623		− switch_fpu_finish(next_fpu);
	622	+ switch_fpu_finish();
624	623
625	624	/* Reload sp0. */
626	625	update_task_stack(next_p);

arch/x86/kernel/ptrace.c+1−1

@@ -29,9 +29,9 @@ · 28 unmodified lines
29	29
30	30	#include <linux/uaccess.h>
31	31	#include <asm/processor.h>
32		−#include <asm/fpu/internal.h>
33	32	#include <asm/fpu/signal.h>
34	33	#include <asm/fpu/regset.h>
	34	+#include <asm/fpu/xstate.h>
35	35	#include <asm/debugreg.h>
36	36	#include <asm/ldt.h>
37	37	#include <asm/desc.h>

arch/x86/kernel/sev.c+1−1

@@ -23,7 +23,7 @@ · 22 unmodified lines
23	23	#include <asm/stacktrace.h>
24	24	#include <asm/sev.h>
25	25	#include <asm/insn-eval.h>
26		−#include <asm/fpu/internal.h>
	26	+#include <asm/fpu/xcr.h>
27	27	#include <asm/processor.h>
28	28	#include <asm/realmode.h>
29	29	#include <asm/traps.h>

arch/x86/kernel/signal.c+71−12

@@ -15,6 +15,7 @@ · 14 unmodified lines
15	15	#include <linux/mm.h>
16	16	#include <linux/smp.h>
17	17	#include <linux/kernel.h>
	18	+#include <linux/kstrtox.h>
18	19	#include <linux/errno.h>
19	20	#include <linux/wait.h>
20	21	#include <linux/tracehook.h>
@@ -30,8 +31,8 @@ · 9 unmodified lines
30	31
31	32	#include <asm/processor.h>
32	33	#include <asm/ucontext.h>
33		−#include <asm/fpu/internal.h>
34	34	#include <asm/fpu/signal.h>
	35	+#include <asm/fpu/xstate.h>
35	36	#include <asm/vdso.h>
36	37	#include <asm/mce.h>
37	38	#include <asm/sighandling.h>
@@ -41,6 +42,7 @@ · 3 unmodified lines
41	42	#include <linux/compat.h>
42	43	#include <asm/proto.h>
43	44	#include <asm/ia32_unistd.h>
	45	+#include <asm/fpu/xstate.h>
44	46	#endif /* CONFIG_X86_64 */
45	47
46	48	#include <asm/syscall.h>
@@ -79,9 +81,9 @@ static void force_valid_ss(struct pt_regs *regs) · 32 unmodified lines
79	81	# define CONTEXT_COPY_SIZE sizeof(struct sigcontext)
80	82	#endif
81	83
82		−static int restore_sigcontext(struct pt_regs *regs,
83		− struct sigcontext __user *usc,
84		− unsigned long uc_flags)
	84	+static bool restore_sigcontext(struct pt_regs *regs,
	85	+ struct sigcontext __user *usc,
	86	+ unsigned long uc_flags)
85	87	{
86	88	struct sigcontext sc;
87	89
@@ -89,7 +91,7 @@ static int restore_sigcontext(struct pt_regs *regs, · 1 unmodified line
89	91	current->restart_block.fn = do_no_restart_syscall;
90	92
91	93	if (copy_from_user(&sc, usc, CONTEXT_COPY_SIZE))
92		− return -EFAULT;
	94	+ return false;
93	95
94	96	#ifdef CONFIG_X86_32
95	97	set_user_gs(regs, sc.gs);
@@ -244,7 +246,6 @@ get_sigframe(struct k_sigaction ka, struct pt_regs regs, size_t frame_size, · 148 unmodified lines
244	246	unsigned long math_size = 0;
245	247	unsigned long sp = regs->sp;
246	248	unsigned long buf_fx = 0;
247		− int ret;
248	249
249	250	/* redzone */
250	251	if (IS_ENABLED(CONFIG_X86_64))
@@ -292,8 +293,7 @@ get_sigframe(struct k_sigaction ka, struct pt_regs regs, size_t frame_size, · 41 unmodified lines
292	293	}
293	294
294	295	/* save i387 and extended state */
295		− ret = copy_fpstate_to_sigframe(fpstate, (void __user )buf_fx, math_size);
296		− if (ret < 0)
	296	+ if (!copy_fpstate_to_sigframe(fpstate, (void __user )buf_fx, math_size))
297	297	return (void __user *)-1L;
298	298
299	299	return (void __user *)sp;
@@ -643,7 +643,7 @@ SYSCALL_DEFINE0(sigreturn) · 343 unmodified lines
643	643	* x86_32 has no uc_flags bits relevant to restore_sigcontext.
644	644	* Save a few cycles by skipping the __get_user.
645	645	*/
646		− if (restore_sigcontext(regs, &frame->sc, 0))
	646	+ if (!restore_sigcontext(regs, &frame->sc, 0))
647	647	goto badframe;
648	648	return regs->ax;
649	649
@@ -671,7 +671,7 @@ SYSCALL_DEFINE0(rt_sigreturn) · 21 unmodified lines
671	671
672	672	set_current_blocked(&set);
673	673
674		− if (restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags))
	674	+ if (!restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags))
675	675	goto badframe;
676	676
677	677	if (restore_altstack(&frame->uc.uc_stack))
@@ -721,12 +721,15 @@ badframe: · 43 unmodified lines
721	721
722	722	/* max_frame_size tells userspace the worst case signal stack size. */
723	723	static unsigned long __ro_after_init max_frame_size;
	724	+static unsigned int __ro_after_init fpu_default_state_size;
724	725
725	726	void __init init_sigframe_size(void)
726	727	{
	728	+ fpu_default_state_size = fpu__get_fpstate_size();
	729	+
727	730	max_frame_size = MAX_FRAME_SIGINFO_UCTXT_SIZE + MAX_FRAME_PADDING;
728	731
729		− max_frame_size += fpu__get_fpstate_size() + MAX_XSAVE_PADDING;
	732	+ max_frame_size += fpu_default_state_size + MAX_XSAVE_PADDING;
730	733
731	734	/* Userspace expects an aligned size. */
732	735	max_frame_size = round_up(max_frame_size, FRAME_ALIGNMENT);
@@ -910,6 +913,62 @@ void signal_fault(struct pt_regs regs, void __user frame, char *where) · 177 unmodified lines
910	913	force_sig(SIGSEGV);
911	914	}
912	915
	916	+#ifdef CONFIG_DYNAMIC_SIGFRAME
	917	+#ifdef CONFIG_STRICT_SIGALTSTACK_SIZE
	918	+static bool strict_sigaltstack_size __ro_after_init = true;
	919	+#else
	920	+static bool strict_sigaltstack_size __ro_after_init = false;
	921	+#endif
	922	+
	923	+static int __init strict_sas_size(char *arg)
	924	+{
	925	+ return kstrtobool(arg, &strict_sigaltstack_size);
	926	+}
	927	+__setup("strict_sas_size", strict_sas_size);
	928	+
	929	+/*
	930	+ * MINSIGSTKSZ is 2048 and can't be changed despite the fact that AVX512
	931	+ * exceeds that size already. As such programs might never use the
	932	+ * sigaltstack they just continued to work. While always checking against
	933	+ * the real size would be correct, this might be considered a regression.
	934	+ *
	935	+ * Therefore avoid the sanity check, unless enforced by kernel
	936	+ * configuration or command line option.
	937	+ *
	938	+ * When dynamic FPU features are supported, the check is also enforced when
	939	+ * the task has permissions to use dynamic features. Tasks which have no
	940	+ * permission are checked against the size of the non-dynamic feature set
	941	+ * if strict checking is enabled. This avoids forcing all tasks on the
	942	+ * system to allocate large sigaltstacks even if they are never going
	943	+ * to use a dynamic feature. As this is serialized via sighand::siglock
	944	+ * any permission request for a dynamic feature either happened already
	945	+ * or will see the newly install sigaltstack size in the permission checks.
	946	+ */
	947	+bool sigaltstack_size_valid(size_t ss_size)
	948	+{
	949	+ unsigned long fsize = max_frame_size - fpu_default_state_size;
	950	+ u64 mask;
	951	+
	952	+ lockdep_assert_held(&current->sighand->siglock);
	953	+
	954	+ if (!fpu_state_size_dynamic() && !strict_sigaltstack_size)
	955	+ return true;
	956	+
	957	+ fsize += current->group_leader->thread.fpu.perm.__user_state_size;
	958	+ if (likely(ss_size > fsize))
	959	+ return true;
	960	+
	961	+ if (strict_sigaltstack_size)
	962	+ return ss_size > fsize;
	963	+
	964	+ mask = current->group_leader->thread.fpu.perm.__state_perm;
	965	+ if (mask & XFEATURE_MASK_USER_DYNAMIC)
	966	+ return ss_size > fsize;
	967	+
	968	+ return true;
	969	+}
	970	+#endif /* CONFIG_DYNAMIC_SIGFRAME */
	971	+
913	972	#ifdef CONFIG_X86_X32_ABI
914	973	COMPAT_SYSCALL_DEFINE0(x32_rt_sigreturn)
915	974	{
@@ -929,7 +988,7 @@ COMPAT_SYSCALL_DEFINE0(x32_rt_sigreturn) · 13 unmodified lines
929	988
930	989	set_current_blocked(&set);
931	990
932		− if (restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags))
	991	+ if (!restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags))
933	992	goto badframe;
934	993
935	994	if (compat_restore_altstack(&frame->uc.uc_stack))

arch/x86/kernel/smpboot.c+1−1

@@ -70,7 +70,7 @@ · 69 unmodified lines
70	70	#include <asm/mwait.h>
71	71	#include <asm/apic.h>
72	72	#include <asm/io_apic.h>
73		−#include <asm/fpu/internal.h>
	73	+#include <asm/fpu/api.h>
74	74	#include <asm/setup.h>
75	75	#include <asm/uv/uv.h>
76	76	#include <linux/mc146818rtc.h>

arch/x86/kernel/traps.c+39−1

@@ -48,7 +48,7 @@ · 47 unmodified lines
48	48	#include <asm/ftrace.h>
49	49	#include <asm/traps.h>
50	50	#include <asm/desc.h>
51		−#include <asm/fpu/internal.h>
	51	+#include <asm/fpu/api.h>
52	52	#include <asm/cpu.h>
53	53	#include <asm/cpu_entry_area.h>
54	54	#include <asm/mce.h>
@@ -1108,10 +1108,48 @@ DEFINE_IDTENTRY(exc_spurious_interrupt_bug) · 1053 unmodified lines
1108	1108	*/
1109	1109	}
1110	1110
	1111	+static bool handle_xfd_event(struct pt_regs *regs)
	1112	+{
	1113	+ u64 xfd_err;
	1114	+ int err;
	1115	+
	1116	+ if (!IS_ENABLED(CONFIG_X86_64) \|\| !cpu_feature_enabled(X86_FEATURE_XFD))
	1117	+ return false;
	1118	+
	1119	+ rdmsrl(MSR_IA32_XFD_ERR, xfd_err);
	1120	+ if (!xfd_err)
	1121	+ return false;
	1122	+
	1123	+ wrmsrl(MSR_IA32_XFD_ERR, 0);
	1124	+
	1125	+ /* Die if that happens in kernel space */
	1126	+ if (WARN_ON(!user_mode(regs)))
	1127	+ return false;
	1128	+
	1129	+ local_irq_enable();
	1130	+
	1131	+ err = xfd_enable_feature(xfd_err);
	1132	+
	1133	+ switch (err) {
	1134	+ case -EPERM:
	1135	+ force_sig_fault(SIGILL, ILL_ILLOPC, error_get_trap_addr(regs));
	1136	+ break;
	1137	+ case -EFAULT:
	1138	+ force_sig(SIGSEGV);
	1139	+ break;
	1140	+ }
	1141	+
	1142	+ local_irq_disable();
	1143	+ return true;
	1144	+}
	1145	+
1111	1146	DEFINE_IDTENTRY(exc_device_not_available)
1112	1147	{
1113	1148	unsigned long cr0 = read_cr0();
1114	1149
	1150	+ if (handle_xfd_event(regs))
	1151	+ return;
	1152	+
1115	1153	#ifdef CONFIG_MATH_EMULATION
1116	1154	if (!boot_cpu_has(X86_FEATURE_FPU) && (cr0 & X86_CR0_EM)) {
1117	1155	struct math_emu_info info = { };

arch/x86/kvm/svm/sev.c+1−1

@@ -17,10 +17,10 @@ · 16 unmodified lines
17	17	#include <linux/misc_cgroup.h>
18	18	#include <linux/processor.h>
19	19	#include <linux/trace_events.h>
20		−#include <asm/fpu/internal.h>
21	20
22	21	#include <asm/pkru.h>
23	22	#include <asm/trapnr.h>
	23	+#include <asm/fpu/xcr.h>
24	24
25	25	#include "x86.h"
26	26	#include "svm.h"

arch/x86/kvm/svm/svm.c+4−3

@@ -36,6 +36,7 @@ · 35 unmodified lines
36	36	#include <asm/spec-ctrl.h>
37	37	#include <asm/cpu_device_id.h>
38	38	#include <asm/traps.h>
	39	+#include <asm/fpu/api.h>
39	40
40	41	#include <asm/virtext.h>
41	42	#include "trace.h"
@@ -1346,10 +1347,10 @@ static int svm_create_vcpu(struct kvm_vcpu *vcpu) · 1304 unmodified lines
1346	1347	/*
1347	1348	* SEV-ES guests maintain an encrypted version of their FPU
1348	1349	* state which is restored and saved on VMRUN and VMEXIT.
1349		− * Free the fpu structure to prevent KVM from attempting to
1350		− * access the FPU state.
	1350	+ * Mark vcpu->arch.guest_fpu->fpstate as scratch so it won't
	1351	+ * do xsave/xrstor on it.
1351	1352	*/
1352		− kvm_free_guest_fpu(vcpu);
	1353	+ fpstate_set_confidential(&vcpu->arch.guest_fpu);
1353	1354	}
1354	1355
1355	1356	err = avic_init_vcpu(svm);

arch/x86/kvm/vmx/vmx.c+1−1

@@ -35,7 +35,7 @@ · 34 unmodified lines
35	35	#include <asm/cpu_device_id.h>
36	36	#include <asm/debugreg.h>
37	37	#include <asm/desc.h>
38		−#include <asm/fpu/internal.h>
	38	+#include <asm/fpu/api.h>
39	39	#include <asm/idtentry.h>
40	40	#include <asm/io.h>
41	41	#include <asm/irq_remapping.h>

arch/x86/kvm/x86.c+31−227

@@ -68,7 +68,9 @@ · 67 unmodified lines
68	68	#include <asm/mce.h>
69	69	#include <asm/pkru.h>
70	70	#include <linux/kernel_stat.h>
71		−#include <asm/fpu/internal.h> /* Ugh! */
	71	+#include <asm/fpu/api.h>
	72	+#include <asm/fpu/xcr.h>
	73	+#include <asm/fpu/xstate.h>
72	74	#include <asm/pvclock.h>
73	75	#include <asm/div64.h>
74	76	#include <asm/irq_remapping.h>
@@ -293,8 +295,6 @@ u64 __read_mostly host_xcr0; · 218 unmodified lines
293	295	u64 __read_mostly supported_xcr0;
294	296	EXPORT_SYMBOL_GPL(supported_xcr0);
295	297
296		−static struct kmem_cache *x86_fpu_cache;
297		−
298	298	static struct kmem_cache *x86_emulator_cache;
299	299
300	300	/*
@@ -4700,144 +4700,27 @@ static int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu, · 4399 unmodified lines
4700	4700	return 0;
4701	4701	}
4702	4702
4703		−#define XSTATE_COMPACTION_ENABLED (1ULL << 63)
4704		−
4705		−static void fill_xsave(u8 dest, struct kvm_vcpu vcpu)
4706		−{
4707		− struct xregs_state *xsave = &vcpu->arch.guest_fpu->state.xsave;
4708		− u64 xstate_bv = xsave->header.xfeatures;
4709		− u64 valid;
4710		−
4711		− /*
4712		− * Copy legacy XSAVE area, to avoid complications with CPUID
4713		− * leaves 0 and 1 in the loop below.
4714		− */
4715		− memcpy(dest, xsave, XSAVE_HDR_OFFSET);
4716		−
4717		− /* Set XSTATE_BV */
4718		− xstate_bv &= vcpu->arch.guest_supported_xcr0 \| XFEATURE_MASK_FPSSE;
4719		− (u64 )(dest + XSAVE_HDR_OFFSET) = xstate_bv;
4720		−
4721		− /*
4722		− * Copy each region from the possibly compacted offset to the
4723		− * non-compacted offset.
4724		− */
4725		− valid = xstate_bv & ~XFEATURE_MASK_FPSSE;
4726		− while (valid) {
4727		− u32 size, offset, ecx, edx;
4728		− u64 xfeature_mask = valid & -valid;
4729		− int xfeature_nr = fls64(xfeature_mask) - 1;
4730		− void *src;
4731		−
4732		− cpuid_count(XSTATE_CPUID, xfeature_nr,
4733		− &size, &offset, &ecx, &edx);
4734		−
4735		− if (xfeature_nr == XFEATURE_PKRU) {
4736		− memcpy(dest + offset, &vcpu->arch.pkru,
4737		− sizeof(vcpu->arch.pkru));
4738		− } else {
4739		− src = get_xsave_addr(xsave, xfeature_nr);
4740		− if (src)
4741		− memcpy(dest + offset, src, size);
4742		− }
4743		−
4744		− valid -= xfeature_mask;
4745		− }
4746		−}
4747		−
4748		−static void load_xsave(struct kvm_vcpu vcpu, u8 src)
4749		−{
4750		− struct xregs_state *xsave = &vcpu->arch.guest_fpu->state.xsave;
4751		− u64 xstate_bv = (u64 )(src + XSAVE_HDR_OFFSET);
4752		− u64 valid;
4753		−
4754		− /*
4755		− * Copy legacy XSAVE area, to avoid complications with CPUID
4756		− * leaves 0 and 1 in the loop below.
4757		− */
4758		− memcpy(xsave, src, XSAVE_HDR_OFFSET);
4759		−
4760		− /* Set XSTATE_BV and possibly XCOMP_BV. */
4761		− xsave->header.xfeatures = xstate_bv;
4762		− if (boot_cpu_has(X86_FEATURE_XSAVES))
4763		− xsave->header.xcomp_bv = host_xcr0 \| XSTATE_COMPACTION_ENABLED;
4764		−
4765		− /*
4766		− * Copy each region from the non-compacted offset to the
4767		− * possibly compacted offset.
4768		− */
4769		− valid = xstate_bv & ~XFEATURE_MASK_FPSSE;
4770		− while (valid) {
4771		− u32 size, offset, ecx, edx;
4772		− u64 xfeature_mask = valid & -valid;
4773		− int xfeature_nr = fls64(xfeature_mask) - 1;
4774		−
4775		− cpuid_count(XSTATE_CPUID, xfeature_nr,
4776		− &size, &offset, &ecx, &edx);
4777		−
4778		− if (xfeature_nr == XFEATURE_PKRU) {
4779		− memcpy(&vcpu->arch.pkru, src + offset,
4780		− sizeof(vcpu->arch.pkru));
4781		− } else {
4782		− void *dest = get_xsave_addr(xsave, xfeature_nr);
4783		−
4784		− if (dest)
4785		− memcpy(dest, src + offset, size);
4786		− }
4787		−
4788		− valid -= xfeature_mask;
4789		− }
4790		−}
4791		−
4792	4703	static void kvm_vcpu_ioctl_x86_get_xsave(struct kvm_vcpu *vcpu,
4793	4704	struct kvm_xsave *guest_xsave)
4794	4705	{
4795		− if (!vcpu->arch.guest_fpu)
	4706	+ if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
4796	4707	return;
4797	4708
4798		− if (boot_cpu_has(X86_FEATURE_XSAVE)) {
4799		− memset(guest_xsave, 0, sizeof(struct kvm_xsave));
4800		− fill_xsave((u8 *) guest_xsave->region, vcpu);
4801		− } else {
4802		− memcpy(guest_xsave->region,
4803		− &vcpu->arch.guest_fpu->state.fxsave,
4804		− sizeof(struct fxregs_state));
4805		− (u64 )&guest_xsave->region[XSAVE_HDR_OFFSET / sizeof(u32)] =
4806		− XFEATURE_MASK_FPSSE;
4807		− }
	4709	+ fpu_copy_guest_fpstate_to_uabi(&vcpu->arch.guest_fpu,
	4710	+ guest_xsave->region,
	4711	+ sizeof(guest_xsave->region),
	4712	+ vcpu->arch.pkru);
4808	4713	}
4809	4714
4810		−#define XSAVE_MXCSR_OFFSET 24
4811		−
4812	4715	static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu,
4813	4716	struct kvm_xsave *guest_xsave)
4814	4717	{
4815		− u64 xstate_bv;
4816		− u32 mxcsr;
4817		−
4818		− if (!vcpu->arch.guest_fpu)
	4718	+ if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
4819	4719	return 0;
4820	4720
4821		− xstate_bv = (u64 )&guest_xsave->region[XSAVE_HDR_OFFSET / sizeof(u32)];
4822		− mxcsr = (u32 )&guest_xsave->region[XSAVE_MXCSR_OFFSET / sizeof(u32)];
4823		−
4824		− if (boot_cpu_has(X86_FEATURE_XSAVE)) {
4825		− /*
4826		− * Here we allow setting states that are not present in
4827		− * CPUID leaf 0xD, index 0, EDX:EAX. This is for compatibility
4828		− * with old userspace.
4829		− */
4830		− if (xstate_bv & ~supported_xcr0 \|\| mxcsr & ~mxcsr_feature_mask)
4831		− return -EINVAL;
4832		− load_xsave(vcpu, (u8 *)guest_xsave->region);
4833		− } else {
4834		− if (xstate_bv & ~XFEATURE_MASK_FPSSE \|\|
4835		− mxcsr & ~mxcsr_feature_mask)
4836		− return -EINVAL;
4837		− memcpy(&vcpu->arch.guest_fpu->state.fxsave,
4838		− guest_xsave->region, sizeof(struct fxregs_state));
4839		− }
4840		− return 0;
	4721	+ return fpu_copy_uabi_to_guest_fpstate(&vcpu->arch.guest_fpu,
	4722	+ guest_xsave->region,
	4723	+ supported_xcr0, &vcpu->arch.pkru);
4841	4724	}
4842	4725
4843	4726	static void kvm_vcpu_ioctl_x86_get_xcrs(struct kvm_vcpu *vcpu,
@@ -8434,18 +8317,11 @@ int kvm_arch_init(void *opaque) · 3590 unmodified lines
8434	8317	}
8435	8318
8436	8319	r = -ENOMEM;
8437		− x86_fpu_cache = kmem_cache_create("x86_fpu", sizeof(struct fpu),
8438		− __alignof__(struct fpu), SLAB_ACCOUNT,
8439		− NULL);
8440		− if (!x86_fpu_cache) {
8441		− printk(KERN_ERR "kvm: failed to allocate cache for x86 fpu\n");
8442		− goto out;
8443		− }
8444	8320
8445	8321	x86_emulator_cache = kvm_alloc_emulator_cache();
8446	8322	if (!x86_emulator_cache) {
8447	8323	pr_err("kvm: failed to allocate cache for x86 emulator\n");
8448		− goto out_free_x86_fpu_cache;
	8324	+ goto out;
8449	8325	}
8450	8326
8451	8327	user_return_msrs = alloc_percpu(struct kvm_user_return_msrs);
@@ -8483,8 +8359,6 @@ out_free_percpu: · 31 unmodified lines
8483	8359	free_percpu(user_return_msrs);
8484	8360	out_free_x86_emulator_cache:
8485	8361	kmem_cache_destroy(x86_emulator_cache);
8486		−out_free_x86_fpu_cache:
8487		− kmem_cache_destroy(x86_fpu_cache);
8488	8362	out:
8489	8363	return r;
8490	8364	}
@@ -8511,7 +8385,6 @@ void kvm_arch_exit(void) · 20 unmodified lines
8511	8385	kvm_mmu_module_exit();
8512	8386	free_percpu(user_return_msrs);
8513	8387	kmem_cache_destroy(x86_emulator_cache);
8514		− kmem_cache_destroy(x86_fpu_cache);
8515	8388	#ifdef CONFIG_KVM_XEN
8516	8389	static_key_deferred_flush(&kvm_xen_enabled);
8517	8390	WARN_ON(static_branch_unlikely(&kvm_xen_enabled.key));
@@ -9938,58 +9811,21 @@ static int complete_emulated_mmio(struct kvm_vcpu *vcpu) · 1420 unmodified lines
9938	9811	return 0;
9939	9812	}
9940	9813
9941		−static void kvm_save_current_fpu(struct fpu *fpu)
9942		−{
9943		− /*
9944		− * If the target FPU state is not resident in the CPU registers, just
9945		− * memcpy() from current, else save CPU state directly to the target.
9946		− */
9947		− if (test_thread_flag(TIF_NEED_FPU_LOAD))
9948		− memcpy(&fpu->state, &current->thread.fpu.state,
9949		− fpu_kernel_xstate_size);
9950		− else
9951		− save_fpregs_to_fpstate(fpu);
9952		−}
9953		−
9954	9814	/* Swap (qemu) user FPU context for the guest FPU context. */
9955	9815	static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu)
9956	9816	{
9957		− fpregs_lock();
9958		−
9959		− kvm_save_current_fpu(vcpu->arch.user_fpu);
9960		−
9961	9817	/*
9962		− * Guests with protected state can't have it set by the hypervisor,
9963		− * so skip trying to set it.
	9818	+ * Exclude PKRU from restore as restored separately in
	9819	+ * kvm_x86_ops.run().
9964	9820	*/
9965		− if (vcpu->arch.guest_fpu)
9966		− /* PKRU is separately restored in kvm_x86_ops.run. */
9967		− __restore_fpregs_from_fpstate(&vcpu->arch.guest_fpu->state,
9968		− ~XFEATURE_MASK_PKRU);
9969		−
9970		− fpregs_mark_activate();
9971		− fpregs_unlock();
9972		−
	9821	+ fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, true);
9973	9822	trace_kvm_fpu(1);
9974	9823	}
9975	9824
9976	9825	/* When vcpu_run ends, restore user space FPU context. */
9977	9826	static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu)
9978	9827	{
9979		− fpregs_lock();
9980		−
9981		− /*
9982		− * Guests with protected state can't have it read by the hypervisor,
9983		− * so skip trying to save it.
9984		− */
9985		− if (vcpu->arch.guest_fpu)
9986		− kvm_save_current_fpu(vcpu->arch.guest_fpu);
9987		−
9988		− restore_fpregs_from_fpstate(&vcpu->arch.user_fpu->state);
9989		−
9990		− fpregs_mark_activate();
9991		− fpregs_unlock();
9992		−
	9828	+ fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, false);
9993	9829	++vcpu->stat.fpu_reload;
9994	9830	trace_kvm_fpu(0);
9995	9831	}
@@ -10570,12 +10406,12 @@ int kvm_arch_vcpu_ioctl_get_fpu(struct kvm_vcpu vcpu, struct kvm_fpu fpu) · 574 unmodified lines
10570	10406	{
10571	10407	struct fxregs_state *fxsave;
10572	10408
10573		− if (!vcpu->arch.guest_fpu)
	10409	+ if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
10574	10410	return 0;
10575	10411
10576	10412	vcpu_load(vcpu);
10577	10413
10578		− fxsave = &vcpu->arch.guest_fpu->state.fxsave;
	10414	+ fxsave = &vcpu->arch.guest_fpu.fpstate->regs.fxsave;
10579	10415	memcpy(fpu->fpr, fxsave->st_space, 128);
10580	10416	fpu->fcw = fxsave->cwd;
10581	10417	fpu->fsw = fxsave->swd;
@@ -10593,12 +10429,12 @@ int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu vcpu, struct kvm_fpu fpu) · 11 unmodified lines
10593	10429	{
10594	10430	struct fxregs_state *fxsave;
10595	10431
10596		− if (!vcpu->arch.guest_fpu)
	10432	+ if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
10597	10433	return 0;
10598	10434
10599	10435	vcpu_load(vcpu);
10600	10436
10601		− fxsave = &vcpu->arch.guest_fpu->state.fxsave;
	10437	+ fxsave = &vcpu->arch.guest_fpu.fpstate->regs.fxsave;
10602	10438
10603	10439	memcpy(fxsave->st_space, fpu->fpr, 128);
10604	10440	fxsave->cwd = fpu->fcw;
@@ -10651,14 +10487,6 @@ static int sync_regs(struct kvm_vcpu *vcpu) · 46 unmodified lines
10651	10487
10652	10488	static void fx_init(struct kvm_vcpu *vcpu)
10653	10489	{
10654		− if (!vcpu->arch.guest_fpu)
10655		− return;
10656		−
10657		− fpstate_init(&vcpu->arch.guest_fpu->state);
10658		− if (boot_cpu_has(X86_FEATURE_XSAVES))
10659		− vcpu->arch.guest_fpu->state.xsave.header.xcomp_bv =
10660		− host_xcr0 \| XSTATE_COMPACTION_ENABLED;
10661		−
10662	10490	/*
10663	10491	* Ensure guest xcr0 is valid for loading
10664	10492	*/
@@ -10667,15 +10495,6 @@ static void fx_init(struct kvm_vcpu *vcpu) · 2 unmodified lines
10667	10495	vcpu->arch.cr0 \|= X86_CR0_ET;
10668	10496	}
10669	10497
10670		−void kvm_free_guest_fpu(struct kvm_vcpu *vcpu)
10671		−{
10672		− if (vcpu->arch.guest_fpu) {
10673		− kmem_cache_free(x86_fpu_cache, vcpu->arch.guest_fpu);
10674		− vcpu->arch.guest_fpu = NULL;
10675		− }
10676		−}
10677		−EXPORT_SYMBOL_GPL(kvm_free_guest_fpu);
10678		−
10679	10498	int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)
10680	10499	{
10681	10500	if (kvm_check_tsc_unstable() && atomic_read(&kvm->online_vcpus) != 0)
@@ -10732,19 +10551,11 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · 50 unmodified lines
10732	10551	if (!alloc_emulate_ctxt(vcpu))
10733	10552	goto free_wbinvd_dirty_mask;
10734	10553
10735		− vcpu->arch.user_fpu = kmem_cache_zalloc(x86_fpu_cache,
10736		− GFP_KERNEL_ACCOUNT);
10737		− if (!vcpu->arch.user_fpu) {
10738		− pr_err("kvm: failed to allocate userspace's fpu\n");
	10554	+ if (!fpu_alloc_guest_fpstate(&vcpu->arch.guest_fpu)) {
	10555	+ pr_err("kvm: failed to allocate vcpu's fpu\n");
10739	10556	goto free_emulate_ctxt;
10740	10557	}
10741	10558
10742		− vcpu->arch.guest_fpu = kmem_cache_zalloc(x86_fpu_cache,
10743		− GFP_KERNEL_ACCOUNT);
10744		− if (!vcpu->arch.guest_fpu) {
10745		− pr_err("kvm: failed to allocate vcpu's fpu\n");
10746		− goto free_user_fpu;
10747		− }
10748	10559	fx_init(vcpu);
10749	10560
10750	10561	vcpu->arch.maxphyaddr = cpuid_query_maxphyaddr(vcpu);
@@ -10777,9 +10588,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · 26 unmodified lines
10777	10588	return 0;
10778	10589
10779	10590	free_guest_fpu:
10780		− kvm_free_guest_fpu(vcpu);
10781		−free_user_fpu:
10782		− kmem_cache_free(x86_fpu_cache, vcpu->arch.user_fpu);
	10591	+ fpu_free_guest_fpstate(&vcpu->arch.guest_fpu);
10783	10592	free_emulate_ctxt:
10784	10593	kmem_cache_free(x86_emulator_cache, vcpu->arch.emulate_ctxt);
10785	10594	free_wbinvd_dirty_mask:
@@ -10828,8 +10637,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) · 42 unmodified lines
10828	10637
10829	10638	kmem_cache_free(x86_emulator_cache, vcpu->arch.emulate_ctxt);
10830	10639	free_cpumask_var(vcpu->arch.wbinvd_dirty_mask);
10831		− kmem_cache_free(x86_fpu_cache, vcpu->arch.user_fpu);
10832		− kvm_free_guest_fpu(vcpu);
	10640	+ fpu_free_guest_fpstate(&vcpu->arch.guest_fpu);
10833	10641
10834	10642	kvm_hv_vcpu_uninit(vcpu);
10835	10643	kvm_pmu_destroy(vcpu);
@@ -10881,8 +10689,8 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) · 45 unmodified lines
10881	10689	kvm_async_pf_hash_reset(vcpu);
10882	10690	vcpu->arch.apf.halted = false;
10883	10691
10884		− if (vcpu->arch.guest_fpu && kvm_mpx_supported()) {
10885		− void *mpx_state_buffer;
	10692	+ if (vcpu->arch.guest_fpu.fpstate && kvm_mpx_supported()) {
	10693	+ struct fpstate *fpstate = vcpu->arch.guest_fpu.fpstate;
10886	10694
10887	10695	/*
10888	10696	* To avoid have the INIT path from kvm_apic_has_events() that be
@@ -10890,14 +10698,10 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) · 1 unmodified line
10890	10698	*/
10891	10699	if (init_event)
10892	10700	kvm_put_guest_fpu(vcpu);
10893		− mpx_state_buffer = get_xsave_addr(&vcpu->arch.guest_fpu->state.xsave,
10894		− XFEATURE_BNDREGS);
10895		− if (mpx_state_buffer)
10896		− memset(mpx_state_buffer, 0, sizeof(struct mpx_bndreg_state));
10897		− mpx_state_buffer = get_xsave_addr(&vcpu->arch.guest_fpu->state.xsave,
10898		− XFEATURE_BNDCSR);
10899		− if (mpx_state_buffer)
10900		− memset(mpx_state_buffer, 0, sizeof(struct mpx_bndcsr));
	10701	+
	10702	+ fpstate_clear_xstate_component(fpstate, XFEATURE_BNDREGS);
	10703	+ fpstate_clear_xstate_component(fpstate, XFEATURE_BNDCSR);
	10704	+
10901	10705	if (init_event)
10902	10706	kvm_load_guest_fpu(vcpu);
10903	10707	}

arch/x86/lib/copy_mc_64.S+4−4

@@ -107,9 +107,9 @@ SYM_FUNC_END(copy_mc_fragile) · 106 unmodified lines
107	107
108	108	.previous
109	109
110		− _ASM_EXTABLE_FAULT(.L_read_leading_bytes, .E_leading_bytes)
111		− _ASM_EXTABLE_FAULT(.L_read_words, .E_read_words)
112		− _ASM_EXTABLE_FAULT(.L_read_trailing_bytes, .E_trailing_bytes)
	110	+ _ASM_EXTABLE_TYPE(.L_read_leading_bytes, .E_leading_bytes, EX_TYPE_DEFAULT_MCE_SAFE)
	111	+ _ASM_EXTABLE_TYPE(.L_read_words, .E_read_words, EX_TYPE_DEFAULT_MCE_SAFE)
	112	+ _ASM_EXTABLE_TYPE(.L_read_trailing_bytes, .E_trailing_bytes, EX_TYPE_DEFAULT_MCE_SAFE)
113	113	_ASM_EXTABLE(.L_write_leading_bytes, .E_leading_bytes)
114	114	_ASM_EXTABLE(.L_write_words, .E_write_words)
115	115	_ASM_EXTABLE(.L_write_trailing_bytes, .E_trailing_bytes)
@@ -149,5 +149,5 @@ SYM_FUNC_END(copy_mc_enhanced_fast_string) · 33 unmodified lines
149	149
150	150	.previous
151	151
152		− _ASM_EXTABLE_FAULT(.L_copy, .E_copy)
	152	+ _ASM_EXTABLE_TYPE(.L_copy, .E_copy, EX_TYPE_DEFAULT_MCE_SAFE)
153	153	#endif /* !CONFIG_UML */

arch/x86/math-emu/fpu_aux.c+1−1

@@ -53,7 +53,7 @@ void fpstate_init_soft(struct swregs_state *soft) · 52 unmodified lines
53	53
54	54	void finit(void)
55	55	{
56		− fpstate_init_soft(&current->thread.fpu.state.soft);
	56	+ fpstate_init_soft(&current->thread.fpu.fpstate->regs.soft);
57	57	}
58	58
59	59	/*

arch/x86/math-emu/fpu_entry.c+3−3

@@ -31,7 +31,7 @@ · 30 unmodified lines
31	31	#include <linux/uaccess.h>
32	32	#include <asm/traps.h>
33	33	#include <asm/user.h>
34		−#include <asm/fpu/internal.h>
	34	+#include <asm/fpu/api.h>
35	35
36	36	#include "fpu_system.h"
37	37	#include "fpu_emu.h"
@@ -640,7 +640,7 @@ int fpregs_soft_set(struct task_struct *target, · 602 unmodified lines
640	640	unsigned int pos, unsigned int count,
641	641	const void kbuf, const void __user ubuf)
642	642	{
643		− struct swregs_state *s387 = &target->thread.fpu.state.soft;
	643	+ struct swregs_state *s387 = &target->thread.fpu.fpstate->regs.soft;
644	644	void *space = s387->st_space;
645	645	int ret;
646	646	int offset, other, i, tags, regnr, tag, newtop;
@@ -691,7 +691,7 @@ int fpregs_soft_get(struct task_struct *target, · 44 unmodified lines
691	691	const struct user_regset *regset,
692	692	struct membuf to)
693	693	{
694		− struct swregs_state *s387 = &target->thread.fpu.state.soft;
	694	+ struct swregs_state *s387 = &target->thread.fpu.fpstate->regs.soft;
695	695	const void *space = s387->st_space;
696	696	int offset = (S387->ftop & 7) * 10, other = 80 - offset;
697	697

arch/x86/math-emu/fpu_system.h+1−1

@@ -73,7 +73,7 @@ static inline bool seg_writable(struct desc_struct *d) · 72 unmodified lines
73	73	return (d->type & SEG_TYPE_EXECUTE_MASK) == SEG_TYPE_WRITABLE;
74	74	}
75	75
76		−#define I387 (&current->thread.fpu.state)
	76	+#define I387 (&current->thread.fpu.fpstate->regs)
77	77	#define FPU_info (I387->soft.info)
78	78
79	79	#define FPU_CS ((unsigned short ) &(FPU_info->regs->cs))

arch/x86/mm/extable.c+56−79

@@ -4,46 +4,30 @@ · 3 unmodified lines
4	4	#include <linux/sched/debug.h>
5	5	#include <xen/xen.h>
6	6
7		−#include <asm/fpu/internal.h>
	7	+#include <asm/fpu/api.h>
8	8	#include <asm/sev.h>
9	9	#include <asm/traps.h>
10	10	#include <asm/kdebug.h>
11	11
12		−typedef bool (ex_handler_t)(const struct exception_table_entry ,
13		− struct pt_regs *, int, unsigned long,
14		− unsigned long);
15		−
16	12	static inline unsigned long
17	13	ex_fixup_addr(const struct exception_table_entry *x)
18	14	{
19	15	return (unsigned long)&x->fixup + x->fixup;
20	16	}
21		−static inline ex_handler_t
22		−ex_fixup_handler(const struct exception_table_entry *x)
23		−{
24		− return (ex_handler_t)((unsigned long)&x->handler + x->handler);
25		−}
26	17
27		−__visible bool ex_handler_default(const struct exception_table_entry *fixup,
28		− struct pt_regs *regs, int trapnr,
29		− unsigned long error_code,
30		− unsigned long fault_addr)
	18	+static bool ex_handler_default(const struct exception_table_entry *fixup,
	19	+ struct pt_regs *regs)
31	20	{
32	21	regs->ip = ex_fixup_addr(fixup);
33	22	return true;
34	23	}
35		−EXPORT_SYMBOL(ex_handler_default);
36	24
37		−__visible bool ex_handler_fault(const struct exception_table_entry *fixup,
38		− struct pt_regs *regs, int trapnr,
39		− unsigned long error_code,
40		− unsigned long fault_addr)
	25	+static bool ex_handler_fault(const struct exception_table_entry *fixup,
	26	+ struct pt_regs *regs, int trapnr)
41	27	{
42		− regs->ip = ex_fixup_addr(fixup);
43	28	regs->ax = trapnr;
44		− return true;
	29	+ return ex_handler_default(fixup, regs);
45	30	}
46		−EXPORT_SYMBOL_GPL(ex_handler_fault);
47	31
48	32	/*
49	33	* Handler for when we fail to restore a task's FPU state. We should never get
@@ -55,65 +39,47 @@ EXPORT_SYMBOL_GPL(ex_handler_fault); · 5 unmodified lines
55	39	* of vulnerability by restoring from the initial state (essentially, zeroing
56	40	* out all the FPU registers) if we can't restore from the task's FPU state.
57	41	*/
58		−__visible bool ex_handler_fprestore(const struct exception_table_entry *fixup,
59		− struct pt_regs *regs, int trapnr,
60		− unsigned long error_code,
61		− unsigned long fault_addr)
	42	+static bool ex_handler_fprestore(const struct exception_table_entry *fixup,
	43	+ struct pt_regs *regs)
62	44	{
63	45	regs->ip = ex_fixup_addr(fixup);
64	46
65	47	WARN_ONCE(1, "Bad FPU state detected at %pB, reinitializing FPU registers.",
66	48	(void *)instruction_pointer(regs));
67	49
68		− __restore_fpregs_from_fpstate(&init_fpstate, xfeatures_mask_fpstate());
	50	+ fpu_reset_from_exception_fixup();
69	51	return true;
70	52	}
71		−EXPORT_SYMBOL_GPL(ex_handler_fprestore);
72	53
73		−__visible bool ex_handler_uaccess(const struct exception_table_entry *fixup,
74		− struct pt_regs *regs, int trapnr,
75		− unsigned long error_code,
76		− unsigned long fault_addr)
	54	+static bool ex_handler_uaccess(const struct exception_table_entry *fixup,
	55	+ struct pt_regs *regs, int trapnr)
77	56	{
78	57	WARN_ONCE(trapnr == X86_TRAP_GP, "General protection fault in user access. Non-canonical address?");
79		− regs->ip = ex_fixup_addr(fixup);
80		− return true;
	58	+ return ex_handler_default(fixup, regs);
81	59	}
82		−EXPORT_SYMBOL(ex_handler_uaccess);
83	60
84		−__visible bool ex_handler_copy(const struct exception_table_entry *fixup,
85		− struct pt_regs *regs, int trapnr,
86		− unsigned long error_code,
87		− unsigned long fault_addr)
	61	+static bool ex_handler_copy(const struct exception_table_entry *fixup,
	62	+ struct pt_regs *regs, int trapnr)
88	63	{
89	64	WARN_ONCE(trapnr == X86_TRAP_GP, "General protection fault in user access. Non-canonical address?");
90		− regs->ip = ex_fixup_addr(fixup);
91		− regs->ax = trapnr;
92		− return true;
	65	+ return ex_handler_fault(fixup, regs, trapnr);
93	66	}
94		−EXPORT_SYMBOL(ex_handler_copy);
95	67
96		−__visible bool ex_handler_rdmsr_unsafe(const struct exception_table_entry *fixup,
97		− struct pt_regs *regs, int trapnr,
98		− unsigned long error_code,
99		− unsigned long fault_addr)
	68	+static bool ex_handler_rdmsr_unsafe(const struct exception_table_entry *fixup,
	69	+ struct pt_regs *regs)
100	70	{
101	71	if (pr_warn_once("unchecked MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n",
102	72	(unsigned int)regs->cx, regs->ip, (void *)regs->ip))
103	73	show_stack_regs(regs);
104	74
105	75	/* Pretend that the read succeeded and returned 0. */
106		− regs->ip = ex_fixup_addr(fixup);
107	76	regs->ax = 0;
108	77	regs->dx = 0;
109		− return true;
	78	+ return ex_handler_default(fixup, regs);
110	79	}
111		−EXPORT_SYMBOL(ex_handler_rdmsr_unsafe);
112	80
113		−__visible bool ex_handler_wrmsr_unsafe(const struct exception_table_entry *fixup,
114		− struct pt_regs *regs, int trapnr,
115		− unsigned long error_code,
116		− unsigned long fault_addr)
	81	+static bool ex_handler_wrmsr_unsafe(const struct exception_table_entry *fixup,
	82	+ struct pt_regs *regs)
117	83	{
118	84	if (pr_warn_once("unchecked MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n",
119	85	(unsigned int)regs->cx, (unsigned int)regs->dx,
@@ -121,45 +87,29 @@ __visible bool ex_handler_wrmsr_unsafe(const struct exception_table_entry *fixup · 1 unmodified line
121	87	show_stack_regs(regs);
122	88
123	89	/* Pretend that the write succeeded. */
124		− regs->ip = ex_fixup_addr(fixup);
125		− return true;
	90	+ return ex_handler_default(fixup, regs);
126	91	}
127		−EXPORT_SYMBOL(ex_handler_wrmsr_unsafe);
128	92
129		−__visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup,
130		− struct pt_regs *regs, int trapnr,
131		− unsigned long error_code,
132		− unsigned long fault_addr)
	93	+static bool ex_handler_clear_fs(const struct exception_table_entry *fixup,
	94	+ struct pt_regs *regs)
133	95	{
134	96	if (static_cpu_has(X86_BUG_NULL_SEG))
135	97	asm volatile ("mov %0, %%fs" : : "rm" (__USER_DS));
136	98	asm volatile ("mov %0, %%fs" : : "rm" (0));
137		− return ex_handler_default(fixup, regs, trapnr, error_code, fault_addr);
	99	+ return ex_handler_default(fixup, regs);
138	100	}
139		−EXPORT_SYMBOL(ex_handler_clear_fs);
140	101
141		−enum handler_type ex_get_fault_handler_type(unsigned long ip)
	102	+int ex_get_fixup_type(unsigned long ip)
142	103	{
143		− const struct exception_table_entry *e;
144		− ex_handler_t handler;
	104	+ const struct exception_table_entry *e = search_exception_tables(ip);
145	105
146		− e = search_exception_tables(ip);
147		− if (!e)
148		− return EX_HANDLER_NONE;
149		− handler = ex_fixup_handler(e);
150		− if (handler == ex_handler_fault)
151		− return EX_HANDLER_FAULT;
152		− else if (handler == ex_handler_uaccess \|\| handler == ex_handler_copy)
153		− return EX_HANDLER_UACCESS;
154		− else
155		− return EX_HANDLER_OTHER;
	106	+ return e ? e->type : EX_TYPE_NONE;
156	107	}
157	108
158	109	int fixup_exception(struct pt_regs *regs, int trapnr, unsigned long error_code,
159	110	unsigned long fault_addr)
160	111	{
161	112	const struct exception_table_entry *e;
162		− ex_handler_t handler;
163	113
164	114	#ifdef CONFIG_PNPBIOS
165	115	if (unlikely(SEGMENT_IS_PNP_CODE(regs->cs))) {
@@ -179,8 +129,35 @@ int fixup_exception(struct pt_regs *regs, int trapnr, unsigned long error_code, · 13 unmodified lines
179	129	if (!e)
180	130	return 0;
181	131
182		− handler = ex_fixup_handler(e);
183		− return handler(e, regs, trapnr, error_code, fault_addr);
	132	+ switch (e->type) {
	133	+ case EX_TYPE_DEFAULT:
	134	+ case EX_TYPE_DEFAULT_MCE_SAFE:
	135	+ return ex_handler_default(e, regs);
	136	+ case EX_TYPE_FAULT:
	137	+ case EX_TYPE_FAULT_MCE_SAFE:
	138	+ return ex_handler_fault(e, regs, trapnr);
	139	+ case EX_TYPE_UACCESS:
	140	+ return ex_handler_uaccess(e, regs, trapnr);
	141	+ case EX_TYPE_COPY:
	142	+ return ex_handler_copy(e, regs, trapnr);
	143	+ case EX_TYPE_CLEAR_FS:
	144	+ return ex_handler_clear_fs(e, regs);
	145	+ case EX_TYPE_FPU_RESTORE:
	146	+ return ex_handler_fprestore(e, regs);
	147	+ case EX_TYPE_RDMSR:
	148	+ return ex_handler_rdmsr_unsafe(e, regs);
	149	+ case EX_TYPE_WRMSR:
	150	+ return ex_handler_wrmsr_unsafe(e, regs);
	151	+ case EX_TYPE_BPF:
	152	+ return ex_handler_bpf(e, regs);
	153	+ case EX_TYPE_RDMSR_IN_MCE:
	154	+ ex_handler_msr_mce(regs, false);
	155	+ break;
	156	+ case EX_TYPE_WRMSR_IN_MCE:
	157	+ ex_handler_msr_mce(regs, true);
	158	+ break;
	159	+ }
	160	+ BUG();
184	161	}
185	162
186	163	extern unsigned int early_recursion_flag;

arch/x86/net/bpf_jit_comp.c+2−9

@@ -803,9 +803,7 @@ static int emit_atomic(u8 **pprog, u8 atomic_op, · 802 unmodified lines
803	803	return 0;
804	804	}
805	805
806		−static bool ex_handler_bpf(const struct exception_table_entry *x,
807		− struct pt_regs *regs, int trapnr,
808		− unsigned long error_code, unsigned long fault_addr)
	806	+bool ex_handler_bpf(const struct exception_table_entry x, struct pt_regs regs)
809	807	{
810	808	u32 reg = x->fixup >> 8;
811	809
@@ -1288,12 +1286,7 @@ st: if (is_imm8(insn->off)) · 476 unmodified lines
1288	1286	}
1289	1287	ex->insn = delta;
1290	1288
1291		− delta = (u8 )ex_handler_bpf - (u8 )&ex->handler;
1292		− if (!is_simm32(delta)) {
1293		− pr_err("extable->handler doesn't fit into 32-bit\n");
1294		− return -EFAULT;
1295		− }
1296		− ex->handler = delta;
	1289	+ ex->type = EX_TYPE_BPF;
1297	1290
1298	1291	if (dst_reg > BPF_REG_9) {
1299	1292	pr_err("verifier error\n");

arch/x86/power/cpu.c+1−1

@@ -20,7 +20,7 @@ · 19 unmodified lines
20	20	#include <asm/page.h>
21	21	#include <asm/mce.h>
22	22	#include <asm/suspend.h>
23		−#include <asm/fpu/internal.h>
	23	+#include <asm/fpu/api.h>
24	24	#include <asm/debugreg.h>
25	25	#include <asm/cpu.h>
26	26	#include <asm/mmu_context.h>

include/linux/signal.h+6−0

@@ -464,6 +464,12 @@ int __save_altstack(stack_t __user *, unsigned long); · 463 unmodified lines
464	464	unsafe_put_user(t->sas_ss_size, &__uss->ss_size, label); \
465	465	} while (0);
466	466
	467	+#ifdef CONFIG_DYNAMIC_SIGFRAME
	468	+bool sigaltstack_size_valid(size_t ss_size);
	469	+#else
	470	+static inline bool sigaltstack_size_valid(size_t size) { return true; }
	471	+#endif /* !CONFIG_DYNAMIC_SIGFRAME */
	472	+
467	473	#ifdef CONFIG_PROC_FS
468	474	struct seq_file;
469	475	extern void render_sigset_t(struct seq_file , const char , sigset_t *);

kernel/signal.c+29−6

@@ -4138,11 +4138,29 @@ int do_sigaction(int sig, struct k_sigaction act, struct k_sigaction oact) · 4137 unmodified lines
4138	4138	return 0;
4139	4139	}
4140	4140
	4141	+#ifdef CONFIG_DYNAMIC_SIGFRAME
	4142	+static inline void sigaltstack_lock(void)
	4143	+ __acquires(&current->sighand->siglock)
	4144	+{
	4145	+ spin_lock_irq(&current->sighand->siglock);
	4146	+}
	4147	+
	4148	+static inline void sigaltstack_unlock(void)
	4149	+ __releases(&current->sighand->siglock)
	4150	+{
	4151	+ spin_unlock_irq(&current->sighand->siglock);
	4152	+}
	4153	+#else
	4154	+static inline void sigaltstack_lock(void) { }
	4155	+static inline void sigaltstack_unlock(void) { }
	4156	+#endif
	4157	+
4141	4158	static int
4142	4159	do_sigaltstack (const stack_t ss, stack_t oss, unsigned long sp,
4143	4160	size_t min_ss_size)
4144	4161	{
4145	4162	struct task_struct *t = current;
	4163	+ int ret = 0;
4146	4164
4147	4165	if (oss) {
4148	4166	memset(oss, 0, sizeof(stack_t));
@@ -4166,19 +4184,24 @@ do_sigaltstack (const stack_t ss, stack_t oss, unsigned long sp, · 17 unmodified lines
4166	4184	ss_mode != 0))
4167	4185	return -EINVAL;
4168	4186
	4187	+ sigaltstack_lock();
4169	4188	if (ss_mode == SS_DISABLE) {
4170	4189	ss_size = 0;
4171	4190	ss_sp = NULL;
4172	4191	} else {
4173	4192	if (unlikely(ss_size < min_ss_size))
4174		− return -ENOMEM;
	4193	+ ret = -ENOMEM;
	4194	+ if (!sigaltstack_size_valid(ss_size))
	4195	+ ret = -ENOMEM;
4175	4196	}
4176		−
4177		− t->sas_ss_sp = (unsigned long) ss_sp;
4178		− t->sas_ss_size = ss_size;
4179		− t->sas_ss_flags = ss_flags;
	4197	+ if (!ret) {
	4198	+ t->sas_ss_sp = (unsigned long) ss_sp;
	4199	+ t->sas_ss_size = ss_size;
	4200	+ t->sas_ss_flags = ss_flags;
	4201	+ }
	4202	+ sigaltstack_unlock();
4180	4203	}
4181		− return 0;
	4204	+ return ret;
4182	4205	}
4183	4206
4184	4207	SYSCALL_DEFINE2(sigaltstack,const stack_t __user ,uss, stack_t __user ,uoss)

scripts/sorttable.c+2−2

@@ -240,7 +240,7 @@ static void x86_sort_relative_table(char *extab_image, int image_size) · 239 unmodified lines
240	240
241	241	w(r(loc) + i, loc);
242	242	w(r(loc + 1) + i + 4, loc + 1);
243		− w(r(loc + 2) + i + 8, loc + 2);
	243	+ /* Don't touch the fixup type */
244	244
245	245	i += sizeof(uint32_t) * 3;
246	246	}
@@ -253,7 +253,7 @@ static void x86_sort_relative_table(char *extab_image, int image_size) · 6 unmodified lines
253	253
254	254	w(r(loc) - i, loc);
255	255	w(r(loc + 1) - (i + 4), loc + 1);
256		− w(r(loc + 2) - (i + 8), loc + 2);
	256	+ /* Don't touch the fixup type */
257	257
258	258	i += sizeof(uint32_t) * 3;
259	259	}

tools/testing/selftests/x86/Makefile+1−1

@@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ · 17 unmodified lines
18	18	test_FCMOV test_FCOMI test_FISTTP \
19	19	vdso_restorer
20	20	TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \
21		− corrupt_xstate_header
	21	+ corrupt_xstate_header amx
22	22	# Some selftests require 32bit support enabled also on 64bit systems
23	23	TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall
24	24

tools/testing/selftests/x86/amx.c+851−0

@@ -0,0 +1,851 @@
	1	+// SPDX-License-Identifier: GPL-2.0
	2	+
	3	+#define _GNU_SOURCE
	4	+#include <err.h>
	5	+#include <errno.h>
	6	+#include <pthread.h>
	7	+#include <setjmp.h>
	8	+#include <stdio.h>
	9	+#include <string.h>
	10	+#include <stdbool.h>
	11	+#include <unistd.h>
	12	+#include <x86intrin.h>
	13	+
	14	+#include <sys/auxv.h>
	15	+#include <sys/mman.h>
	16	+#include <sys/shm.h>
	17	+#include <sys/syscall.h>
	18	+#include <sys/wait.h>
	19	+
	20	+#ifndef __x86_64__
	21	+# error This test is 64-bit only
	22	+#endif
	23	+
	24	+#define XSAVE_HDR_OFFSET 512
	25	+#define XSAVE_HDR_SIZE 64
	26	+
	27	+struct xsave_buffer {
	28	+ union {
	29	+ struct {
	30	+ char legacy[XSAVE_HDR_OFFSET];
	31	+ char header[XSAVE_HDR_SIZE];
	32	+ char extended[0];
	33	+ };
	34	+ char bytes[0];
	35	+ };
	36	+};
	37	+
	38	+static inline uint64_t xgetbv(uint32_t index)
	39	+{
	40	+ uint32_t eax, edx;
	41	+
	42	+ asm volatile("xgetbv;"
	43	+ : "=a" (eax), "=d" (edx)
	44	+ : "c" (index));
	45	+ return eax + ((uint64_t)edx << 32);
	46	+}
	47	+
	48	+static inline void cpuid(uint32_t eax, uint32_t ebx, uint32_t ecx, uint32_t edx)
	49	+{
	50	+ asm volatile("cpuid;"
	51	+ : "=a" (eax), "=b" (ebx), "=c" (ecx), "=d" (edx)
	52	+ : "0" (eax), "2" (ecx));
	53	+}
	54	+
	55	+static inline void xsave(struct xsave_buffer *xbuf, uint64_t rfbm)
	56	+{
	57	+ uint32_t rfbm_lo = rfbm;
	58	+ uint32_t rfbm_hi = rfbm >> 32;
	59	+
	60	+ asm volatile("xsave (%%rdi)"
	61	+ : : "D" (xbuf), "a" (rfbm_lo), "d" (rfbm_hi)
	62	+ : "memory");
	63	+}
	64	+
	65	+static inline void xrstor(struct xsave_buffer *xbuf, uint64_t rfbm)
	66	+{
	67	+ uint32_t rfbm_lo = rfbm;
	68	+ uint32_t rfbm_hi = rfbm >> 32;
	69	+
	70	+ asm volatile("xrstor (%%rdi)"
	71	+ : : "D" (xbuf), "a" (rfbm_lo), "d" (rfbm_hi));
	72	+}
	73	+
	74	+/* err() exits and will not return */
	75	+#define fatal_error(msg, ...) err(1, "[FAIL]\t" msg, ##__VA_ARGS__)
	76	+
	77	+static void sethandler(int sig, void (handler)(int, siginfo_t , void *),
	78	+ int flags)
	79	+{
	80	+ struct sigaction sa;
	81	+
	82	+ memset(&sa, 0, sizeof(sa));
	83	+ sa.sa_sigaction = handler;
	84	+ sa.sa_flags = SA_SIGINFO \| flags;
	85	+ sigemptyset(&sa.sa_mask);
	86	+ if (sigaction(sig, &sa, 0))
	87	+ fatal_error("sigaction");
	88	+}
	89	+
	90	+static void clearhandler(int sig)
	91	+{
	92	+ struct sigaction sa;
	93	+
	94	+ memset(&sa, 0, sizeof(sa));
	95	+ sa.sa_handler = SIG_DFL;
	96	+ sigemptyset(&sa.sa_mask);
	97	+ if (sigaction(sig, &sa, 0))
	98	+ fatal_error("sigaction");
	99	+}
	100	+
	101	+#define XFEATURE_XTILECFG 17
	102	+#define XFEATURE_XTILEDATA 18
	103	+#define XFEATURE_MASK_XTILECFG (1 << XFEATURE_XTILECFG)
	104	+#define XFEATURE_MASK_XTILEDATA (1 << XFEATURE_XTILEDATA)
	105	+#define XFEATURE_MASK_XTILE (XFEATURE_MASK_XTILECFG \| XFEATURE_MASK_XTILEDATA)
	106	+
	107	+#define CPUID_LEAF1_ECX_XSAVE_MASK (1 << 26)
	108	+#define CPUID_LEAF1_ECX_OSXSAVE_MASK (1 << 27)
	109	+static inline void check_cpuid_xsave(void)
	110	+{
	111	+ uint32_t eax, ebx, ecx, edx;
	112	+
	113	+ /*
	114	+ * CPUID.1:ECX.XSAVE[bit 26] enumerates general
	115	+ * support for the XSAVE feature set, including
	116	+ * XGETBV.
	117	+ */
	118	+ eax = 1;
	119	+ ecx = 0;
	120	+ cpuid(&eax, &ebx, &ecx, &edx);
	121	+ if (!(ecx & CPUID_LEAF1_ECX_XSAVE_MASK))
	122	+ fatal_error("cpuid: no CPU xsave support");
	123	+ if (!(ecx & CPUID_LEAF1_ECX_OSXSAVE_MASK))
	124	+ fatal_error("cpuid: no OS xsave support");
	125	+}
	126	+
	127	+static uint32_t xbuf_size;
	128	+
	129	+static struct {
	130	+ uint32_t xbuf_offset;
	131	+ uint32_t size;
	132	+} xtiledata;
	133	+
	134	+#define CPUID_LEAF_XSTATE 0xd
	135	+#define CPUID_SUBLEAF_XSTATE_USER 0x0
	136	+#define TILE_CPUID 0x1d
	137	+#define TILE_PALETTE_ID 0x1
	138	+
	139	+static void check_cpuid_xtiledata(void)
	140	+{
	141	+ uint32_t eax, ebx, ecx, edx;
	142	+
	143	+ eax = CPUID_LEAF_XSTATE;
	144	+ ecx = CPUID_SUBLEAF_XSTATE_USER;
	145	+ cpuid(&eax, &ebx, &ecx, &edx);
	146	+
	147	+ /*
	148	+ * EBX enumerates the size (in bytes) required by the XSAVE
	149	+ * instruction for an XSAVE area containing all the user state
	150	+ * components corresponding to bits currently set in XCR0.
	151	+ *
	152	+ * Stash that off so it can be used to allocate buffers later.
	153	+ */
	154	+ xbuf_size = ebx;
	155	+
	156	+ eax = CPUID_LEAF_XSTATE;
	157	+ ecx = XFEATURE_XTILEDATA;
	158	+
	159	+ cpuid(&eax, &ebx, &ecx, &edx);
	160	+ /*
	161	+ * eax: XTILEDATA state component size
	162	+ * ebx: XTILEDATA state component offset in user buffer
	163	+ */
	164	+ if (!eax \|\| !ebx)
	165	+ fatal_error("xstate cpuid: invalid tile data size/offset: %d/%d",
	166	+ eax, ebx);
	167	+
	168	+ xtiledata.size = eax;
	169	+ xtiledata.xbuf_offset = ebx;
	170	+}
	171	+
	172	+/* The helpers for managing XSAVE buffer and tile states: */
	173	+
	174	+struct xsave_buffer *alloc_xbuf(void)
	175	+{
	176	+ struct xsave_buffer *xbuf;
	177	+
	178	+ /* XSAVE buffer should be 64B-aligned. */
	179	+ xbuf = aligned_alloc(64, xbuf_size);
	180	+ if (!xbuf)
	181	+ fatal_error("aligned_alloc()");
	182	+ return xbuf;
	183	+}
	184	+
	185	+static inline void clear_xstate_header(struct xsave_buffer *buffer)
	186	+{
	187	+ memset(&buffer->header, 0, sizeof(buffer->header));
	188	+}
	189	+
	190	+static inline uint64_t get_xstatebv(struct xsave_buffer *buffer)
	191	+{
	192	+ /* XSTATE_BV is at the beginning of the header: */
	193	+ return (uint64_t )&buffer->header;
	194	+}
	195	+
	196	+static inline void set_xstatebv(struct xsave_buffer *buffer, uint64_t bv)
	197	+{
	198	+ /* XSTATE_BV is at the beginning of the header: */
	199	+ (uint64_t )(&buffer->header) = bv;
	200	+}
	201	+
	202	+static void set_rand_tiledata(struct xsave_buffer *xbuf)
	203	+{
	204	+ int ptr = (int )&xbuf->bytes[xtiledata.xbuf_offset];
	205	+ int data;
	206	+ int i;
	207	+
	208	+ /*
	209	+ * Ensure that 'data' is never 0. This ensures that
	210	+ * the registers are never in their initial configuration
	211	+ * and thus never tracked as being in the init state.
	212	+ */
	213	+ data = rand() \| 1;
	214	+
	215	+ for (i = 0; i < xtiledata.size / sizeof(int); i++, ptr++)
	216	+ *ptr = data;
	217	+}
	218	+
	219	+struct xsave_buffer *stashed_xsave;
	220	+
	221	+static void init_stashed_xsave(void)
	222	+{
	223	+ stashed_xsave = alloc_xbuf();
	224	+ if (!stashed_xsave)
	225	+ fatal_error("failed to allocate stashed_xsave\n");
	226	+ clear_xstate_header(stashed_xsave);
	227	+}
	228	+
	229	+static void free_stashed_xsave(void)
	230	+{
	231	+ free(stashed_xsave);
	232	+}
	233	+
	234	+/* See 'struct _fpx_sw_bytes' at sigcontext.h */
	235	+#define SW_BYTES_OFFSET 464
	236	+/* N.B. The struct's field name varies so read from the offset. */
	237	+#define SW_BYTES_BV_OFFSET (SW_BYTES_OFFSET + 8)
	238	+
	239	+static inline struct _fpx_sw_bytes get_fpx_sw_bytes(void buffer)
	240	+{
	241	+ return (struct _fpx_sw_bytes *)(buffer + SW_BYTES_OFFSET);
	242	+}
	243	+
	244	+static inline uint64_t get_fpx_sw_bytes_features(void *buffer)
	245	+{
	246	+ return (uint64_t )(buffer + SW_BYTES_BV_OFFSET);
	247	+}
	248	+
	249	+/* Work around printf() being unsafe in signals: */
	250	+#define SIGNAL_BUF_LEN 1000
	251	+char signal_message_buffer[SIGNAL_BUF_LEN];
	252	+void sig_print(char *msg)
	253	+{
	254	+ int left = SIGNAL_BUF_LEN - strlen(signal_message_buffer) - 1;
	255	+
	256	+ strncat(signal_message_buffer, msg, left);
	257	+}
	258	+
	259	+static volatile bool noperm_signaled;
	260	+static int noperm_errs;
	261	+/*
	262	+ * Signal handler for when AMX is used but
	263	+ * permission has not been obtained.
	264	+ */
	265	+static void handle_noperm(int sig, siginfo_t si, void ctx_void)
	266	+{
	267	+ ucontext_t ctx = (ucontext_t )ctx_void;
	268	+ void *xbuf = ctx->uc_mcontext.fpregs;
	269	+ struct _fpx_sw_bytes *sw_bytes;
	270	+ uint64_t features;
	271	+
	272	+ /* Reset the signal message buffer: */
	273	+ signal_message_buffer[0] = '\0';
	274	+ sig_print("\tAt SIGILL handler,\n");
	275	+
	276	+ if (si->si_code != ILL_ILLOPC) {
	277	+ noperm_errs++;
	278	+ sig_print("[FAIL]\tInvalid signal code.\n");
	279	+ } else {
	280	+ sig_print("[OK]\tValid signal code (ILL_ILLOPC).\n");
	281	+ }
	282	+
	283	+ sw_bytes = get_fpx_sw_bytes(xbuf);
	284	+ /*
	285	+ * Without permission, the signal XSAVE buffer should not
	286	+ * have room for AMX register state (aka. xtiledata).
	287	+ * Check that the size does not overlap with where xtiledata
	288	+ * will reside.
	289	+ *
	290	+ * This also implies that no state components PAST
	291	+ * XTILEDATA (features >=19) can be present in the buffer.
	292	+ */
	293	+ if (sw_bytes->xstate_size <= xtiledata.xbuf_offset) {
	294	+ sig_print("[OK]\tValid xstate size\n");
	295	+ } else {
	296	+ noperm_errs++;
	297	+ sig_print("[FAIL]\tInvalid xstate size\n");
	298	+ }
	299	+
	300	+ features = get_fpx_sw_bytes_features(xbuf);
	301	+ /*
	302	+ * Without permission, the XTILEDATA feature
	303	+ * bit should not be set.
	304	+ */
	305	+ if ((features & XFEATURE_MASK_XTILEDATA) == 0) {
	306	+ sig_print("[OK]\tValid xstate mask\n");
	307	+ } else {
	308	+ noperm_errs++;
	309	+ sig_print("[FAIL]\tInvalid xstate mask\n");
	310	+ }
	311	+
	312	+ noperm_signaled = true;
	313	+ ctx->uc_mcontext.gregs[REG_RIP] += 3; /* Skip the faulting XRSTOR */
	314	+}
	315	+
	316	+/* Return true if XRSTOR is successful; otherwise, false. */
	317	+static inline bool xrstor_safe(struct xsave_buffer *xbuf, uint64_t mask)
	318	+{
	319	+ noperm_signaled = false;
	320	+ xrstor(xbuf, mask);
	321	+
	322	+ /* Print any messages produced by the signal code: */
	323	+ printf("%s", signal_message_buffer);
	324	+ /*
	325	+ * Reset the buffer to make sure any future printing
	326	+ * only outputs new messages:
	327	+ */
	328	+ signal_message_buffer[0] = '\0';
	329	+
	330	+ if (noperm_errs)
	331	+ fatal_error("saw %d errors in noperm signal handler\n", noperm_errs);
	332	+
	333	+ return !noperm_signaled;
	334	+}
	335	+
	336	+/*
	337	+ * Use XRSTOR to populate the XTILEDATA registers with
	338	+ * random data.
	339	+ *
	340	+ * Return true if successful; otherwise, false.
	341	+ */
	342	+static inline bool load_rand_tiledata(struct xsave_buffer *xbuf)
	343	+{
	344	+ clear_xstate_header(xbuf);
	345	+ set_xstatebv(xbuf, XFEATURE_MASK_XTILEDATA);
	346	+ set_rand_tiledata(xbuf);
	347	+ return xrstor_safe(xbuf, XFEATURE_MASK_XTILEDATA);
	348	+}
	349	+
	350	+/* Return XTILEDATA to its initial configuration. */
	351	+static inline void init_xtiledata(void)
	352	+{
	353	+ clear_xstate_header(stashed_xsave);
	354	+ xrstor_safe(stashed_xsave, XFEATURE_MASK_XTILEDATA);
	355	+}
	356	+
	357	+enum expected_result { FAIL_EXPECTED, SUCCESS_EXPECTED };
	358	+
	359	+/* arch_prctl() and sigaltstack() test */
	360	+
	361	+#define ARCH_GET_XCOMP_PERM 0x1022
	362	+#define ARCH_REQ_XCOMP_PERM 0x1023
	363	+
	364	+static void req_xtiledata_perm(void)
	365	+{
	366	+ syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA);
	367	+}
	368	+
	369	+static void validate_req_xcomp_perm(enum expected_result exp)
	370	+{
	371	+ unsigned long bitmask;
	372	+ long rc;
	373	+
	374	+ rc = syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA);
	375	+ if (exp == FAIL_EXPECTED) {
	376	+ if (rc) {
	377	+ printf("[OK]\tARCH_REQ_XCOMP_PERM saw expected failure..\n");
	378	+ return;
	379	+ }
	380	+
	381	+ fatal_error("ARCH_REQ_XCOMP_PERM saw unexpected success.\n");
	382	+ } else if (rc) {
	383	+ fatal_error("ARCH_REQ_XCOMP_PERM saw unexpected failure.\n");
	384	+ }
	385	+
	386	+ rc = syscall(SYS_arch_prctl, ARCH_GET_XCOMP_PERM, &bitmask);
	387	+ if (rc) {
	388	+ fatal_error("prctl(ARCH_GET_XCOMP_PERM) error: %ld", rc);
	389	+ } else if (bitmask & XFEATURE_MASK_XTILE) {
	390	+ printf("\tARCH_REQ_XCOMP_PERM is successful.\n");
	391	+ }
	392	+}
	393	+
	394	+static void validate_xcomp_perm(enum expected_result exp)
	395	+{
	396	+ bool load_success = load_rand_tiledata(stashed_xsave);
	397	+
	398	+ if (exp == FAIL_EXPECTED) {
	399	+ if (load_success) {
	400	+ noperm_errs++;
	401	+ printf("[FAIL]\tLoad tiledata succeeded.\n");
	402	+ } else {
	403	+ printf("[OK]\tLoad tiledata failed.\n");
	404	+ }
	405	+ } else if (exp == SUCCESS_EXPECTED) {
	406	+ if (load_success) {
	407	+ printf("[OK]\tLoad tiledata succeeded.\n");
	408	+ } else {
	409	+ noperm_errs++;
	410	+ printf("[FAIL]\tLoad tiledata failed.\n");
	411	+ }
	412	+ }
	413	+}
	414	+
	415	+#ifndef AT_MINSIGSTKSZ
	416	+# define AT_MINSIGSTKSZ 51
	417	+#endif
	418	+
	419	+static void *alloc_altstack(unsigned int size)
	420	+{
	421	+ void *altstack;
	422	+
	423	+ altstack = mmap(NULL, size, PROT_READ \| PROT_WRITE,
	424	+ MAP_PRIVATE \| MAP_ANONYMOUS \| MAP_STACK, -1, 0);
	425	+
	426	+ if (altstack == MAP_FAILED)
	427	+ fatal_error("mmap() for altstack");
	428	+
	429	+ return altstack;
	430	+}
	431	+
	432	+static void setup_altstack(void *addr, unsigned long size, enum expected_result exp)
	433	+{
	434	+ stack_t ss;
	435	+ int rc;
	436	+
	437	+ memset(&ss, 0, sizeof(ss));
	438	+ ss.ss_size = size;
	439	+ ss.ss_sp = addr;
	440	+
	441	+ rc = sigaltstack(&ss, NULL);
	442	+
	443	+ if (exp == FAIL_EXPECTED) {
	444	+ if (rc) {
	445	+ printf("[OK]\tsigaltstack() failed.\n");
	446	+ } else {
	447	+ fatal_error("sigaltstack() succeeded unexpectedly.\n");
	448	+ }
	449	+ } else if (rc) {
	450	+ fatal_error("sigaltstack()");
	451	+ }
	452	+}
	453	+
	454	+static void test_dynamic_sigaltstack(void)
	455	+{
	456	+ unsigned int small_size, enough_size;
	457	+ unsigned long minsigstksz;
	458	+ void *altstack;
	459	+
	460	+ minsigstksz = getauxval(AT_MINSIGSTKSZ);
	461	+ printf("\tAT_MINSIGSTKSZ = %lu\n", minsigstksz);
	462	+ /*
	463	+ * getauxval() itself can return 0 for failure or
	464	+ * success. But, in this case, AT_MINSIGSTKSZ
	465	+ * will always return a >=0 value if implemented.
	466	+ * Just check for 0.
	467	+ */
	468	+ if (minsigstksz == 0) {
	469	+ printf("no support for AT_MINSIGSTKSZ, skipping sigaltstack tests\n");
	470	+ return;
	471	+ }
	472	+
	473	+ enough_size = minsigstksz * 2;
	474	+
	475	+ altstack = alloc_altstack(enough_size);
	476	+ printf("\tAllocate memory for altstack (%u bytes).\n", enough_size);
	477	+
	478	+ /*
	479	+ * Try setup_altstack() with a size which can not fit
	480	+ * XTILEDATA. ARCH_REQ_XCOMP_PERM should fail.
	481	+ */
	482	+ small_size = minsigstksz - xtiledata.size;
	483	+ printf("\tAfter sigaltstack() with small size (%u bytes).\n", small_size);
	484	+ setup_altstack(altstack, small_size, SUCCESS_EXPECTED);
	485	+ validate_req_xcomp_perm(FAIL_EXPECTED);
	486	+
	487	+ /*
	488	+ * Try setup_altstack() with a size derived from
	489	+ * AT_MINSIGSTKSZ. It should be more than large enough
	490	+ * and thus ARCH_REQ_XCOMP_PERM should succeed.
	491	+ */
	492	+ printf("\tAfter sigaltstack() with enough size (%u bytes).\n", enough_size);
	493	+ setup_altstack(altstack, enough_size, SUCCESS_EXPECTED);
	494	+ validate_req_xcomp_perm(SUCCESS_EXPECTED);
	495	+
	496	+ /*
	497	+ * Try to coerce setup_altstack() to again accept a
	498	+ * too-small altstack. This ensures that big-enough
	499	+ * sigaltstacks can not shrink to a too-small value
	500	+ * once XTILEDATA permission is established.
	501	+ */
	502	+ printf("\tThen, sigaltstack() with small size (%u bytes).\n", small_size);
	503	+ setup_altstack(altstack, small_size, FAIL_EXPECTED);
	504	+}
	505	+
	506	+static void test_dynamic_state(void)
	507	+{
	508	+ pid_t parent, child, grandchild;
	509	+
	510	+ parent = fork();
	511	+ if (parent < 0) {
	512	+ /* fork() failed */
	513	+ fatal_error("fork");
	514	+ } else if (parent > 0) {
	515	+ int status;
	516	+ /* fork() succeeded. Now in the parent. */
	517	+
	518	+ wait(&status);
	519	+ if (!WIFEXITED(status) \|\| WEXITSTATUS(status))
	520	+ fatal_error("arch_prctl test parent exit");
	521	+ return;
	522	+ }
	523	+ /* fork() succeeded. Now in the child . */
	524	+
	525	+ printf("[RUN]\tCheck ARCH_REQ_XCOMP_PERM around process fork() and sigaltack() test.\n");
	526	+
	527	+ printf("\tFork a child.\n");
	528	+ child = fork();
	529	+ if (child < 0) {
	530	+ fatal_error("fork");
	531	+ } else if (child > 0) {
	532	+ int status;
	533	+
	534	+ wait(&status);
	535	+ if (!WIFEXITED(status) \|\| WEXITSTATUS(status))
	536	+ fatal_error("arch_prctl test child exit");
	537	+ _exit(0);
	538	+ }
	539	+
	540	+ /*
	541	+ * The permission request should fail without an
	542	+ * XTILEDATA-compatible signal stack
	543	+ */
	544	+ printf("\tTest XCOMP_PERM at child.\n");
	545	+ validate_xcomp_perm(FAIL_EXPECTED);
	546	+
	547	+ /*
	548	+ * Set up an XTILEDATA-compatible signal stack and
	549	+ * also obtain permission to populate XTILEDATA.
	550	+ */
	551	+ printf("\tTest dynamic sigaltstack at child:\n");
	552	+ test_dynamic_sigaltstack();
	553	+
	554	+ /* Ensure that XTILEDATA can be populated. */
	555	+ printf("\tTest XCOMP_PERM again at child.\n");
	556	+ validate_xcomp_perm(SUCCESS_EXPECTED);
	557	+
	558	+ printf("\tFork a grandchild.\n");
	559	+ grandchild = fork();
	560	+ if (grandchild < 0) {
	561	+ /* fork() failed */
	562	+ fatal_error("fork");
	563	+ } else if (!grandchild) {
	564	+ /* fork() succeeded. Now in the (grand)child. */
	565	+ printf("\tTest XCOMP_PERM at grandchild.\n");
	566	+
	567	+ /*
	568	+ * Ensure that the grandchild inherited
	569	+ * permission and a compatible sigaltstack:
	570	+ */
	571	+ validate_xcomp_perm(SUCCESS_EXPECTED);
	572	+ } else {
	573	+ int status;
	574	+ /* fork() succeeded. Now in the parent. */
	575	+
	576	+ wait(&status);
	577	+ if (!WIFEXITED(status) \|\| WEXITSTATUS(status))
	578	+ fatal_error("fork test grandchild");
	579	+ }
	580	+
	581	+ _exit(0);
	582	+}
	583	+
	584	+/*
	585	+ * Save current register state and compare it to @xbuf1.'
	586	+ *
	587	+ * Returns false if @xbuf1 matches the registers.
	588	+ * Returns true if @xbuf1 differs from the registers.
	589	+ */
	590	+static inline bool __validate_tiledata_regs(struct xsave_buffer *xbuf1)
	591	+{
	592	+ struct xsave_buffer *xbuf2;
	593	+ int ret;
	594	+
	595	+ xbuf2 = alloc_xbuf();
	596	+ if (!xbuf2)
	597	+ fatal_error("failed to allocate XSAVE buffer\n");
	598	+
	599	+ xsave(xbuf2, XFEATURE_MASK_XTILEDATA);
	600	+ ret = memcmp(&xbuf1->bytes[xtiledata.xbuf_offset],
	601	+ &xbuf2->bytes[xtiledata.xbuf_offset],
	602	+ xtiledata.size);
	603	+
	604	+ free(xbuf2);
	605	+
	606	+ if (ret == 0)
	607	+ return false;
	608	+ return true;
	609	+}
	610	+
	611	+static inline void validate_tiledata_regs_same(struct xsave_buffer *xbuf)
	612	+{
	613	+ int ret = __validate_tiledata_regs(xbuf);
	614	+
	615	+ if (ret != 0)
	616	+ fatal_error("TILEDATA registers changed");
	617	+}
	618	+
	619	+static inline void validate_tiledata_regs_changed(struct xsave_buffer *xbuf)
	620	+{
	621	+ int ret = __validate_tiledata_regs(xbuf);
	622	+
	623	+ if (ret == 0)
	624	+ fatal_error("TILEDATA registers did not change");
	625	+}
	626	+
	627	+/* tiledata inheritance test */
	628	+
	629	+static void test_fork(void)
	630	+{
	631	+ pid_t child, grandchild;
	632	+
	633	+ child = fork();
	634	+ if (child < 0) {
	635	+ /* fork() failed */
	636	+ fatal_error("fork");
	637	+ } else if (child > 0) {
	638	+ /* fork() succeeded. Now in the parent. */
	639	+ int status;
	640	+
	641	+ wait(&status);
	642	+ if (!WIFEXITED(status) \|\| WEXITSTATUS(status))
	643	+ fatal_error("fork test child");
	644	+ return;
	645	+ }
	646	+ /* fork() succeeded. Now in the child. */
	647	+ printf("[RUN]\tCheck tile data inheritance.\n\tBefore fork(), load tiledata\n");
	648	+
	649	+ load_rand_tiledata(stashed_xsave);
	650	+
	651	+ grandchild = fork();
	652	+ if (grandchild < 0) {
	653	+ /* fork() failed */
	654	+ fatal_error("fork");
	655	+ } else if (grandchild > 0) {
	656	+ /* fork() succeeded. Still in the first child. */
	657	+ int status;
	658	+
	659	+ wait(&status);
	660	+ if (!WIFEXITED(status) \|\| WEXITSTATUS(status))
	661	+ fatal_error("fork test grand child");
	662	+ _exit(0);
	663	+ }
	664	+ /* fork() succeeded. Now in the (grand)child. */
	665	+
	666	+ /*
	667	+ * TILEDATA registers are not preserved across fork().
	668	+ * Ensure that their value has changed:
	669	+ */
	670	+ validate_tiledata_regs_changed(stashed_xsave);
	671	+
	672	+ _exit(0);
	673	+}
	674	+
	675	+/* Context switching test */
	676	+
	677	+static struct _ctxtswtest_cfg {
	678	+ unsigned int iterations;
	679	+ unsigned int num_threads;
	680	+} ctxtswtest_config;
	681	+
	682	+struct futex_info {
	683	+ pthread_t thread;
	684	+ int nr;
	685	+ pthread_mutex_t mutex;
	686	+ struct futex_info *next;
	687	+};
	688	+
	689	+static void check_tiledata(void info)
	690	+{
	691	+ struct futex_info finfo = (struct futex_info )info;
	692	+ struct xsave_buffer *xbuf;
	693	+ int i;
	694	+
	695	+ xbuf = alloc_xbuf();
	696	+ if (!xbuf)
	697	+ fatal_error("unable to allocate XSAVE buffer");
	698	+
	699	+ /*
	700	+ * Load random data into 'xbuf' and then restore
	701	+ * it to the tile registers themselves.
	702	+ */
	703	+ load_rand_tiledata(xbuf);
	704	+ for (i = 0; i < ctxtswtest_config.iterations; i++) {
	705	+ pthread_mutex_lock(&finfo->mutex);
	706	+
	707	+ /*
	708	+ * Ensure the register values have not
	709	+ * diverged from those recorded in 'xbuf'.
	710	+ */
	711	+ validate_tiledata_regs_same(xbuf);
	712	+
	713	+ /* Load new, random values into xbuf and registers */
	714	+ load_rand_tiledata(xbuf);
	715	+
	716	+ /*
	717	+ * The last thread's last unlock will be for
	718	+ * thread 0's mutex. However, thread 0 will
	719	+ * have already exited the loop and the mutex
	720	+ * will already be unlocked.
	721	+ *
	722	+ * Because this is not an ERRORCHECK mutex,
	723	+ * that inconsistency will be silently ignored.
	724	+ */
	725	+ pthread_mutex_unlock(&finfo->next->mutex);
	726	+ }
	727	+
	728	+ free(xbuf);
	729	+ /*
	730	+ * Return this thread's finfo, which is
	731	+ * a unique value for this thread.
	732	+ */
	733	+ return finfo;
	734	+}
	735	+
	736	+static int create_threads(int num, struct futex_info *finfo)
	737	+{
	738	+ int i;
	739	+
	740	+ for (i = 0; i < num; i++) {
	741	+ int next_nr;
	742	+
	743	+ finfo[i].nr = i;
	744	+ /*
	745	+ * Thread 'i' will wait on this mutex to
	746	+ * be unlocked. Lock it immediately after
	747	+ * initialization:
	748	+ */
	749	+ pthread_mutex_init(&finfo[i].mutex, NULL);
	750	+ pthread_mutex_lock(&finfo[i].mutex);
	751	+
	752	+ next_nr = (i + 1) % num;
	753	+ finfo[i].next = &finfo[next_nr];
	754	+
	755	+ if (pthread_create(&finfo[i].thread, NULL, check_tiledata, &finfo[i]))
	756	+ fatal_error("pthread_create()");
	757	+ }
	758	+ return 0;
	759	+}
	760	+
	761	+static void affinitize_cpu0(void)
	762	+{
	763	+ cpu_set_t cpuset;
	764	+
	765	+ CPU_ZERO(&cpuset);
	766	+ CPU_SET(0, &cpuset);
	767	+
	768	+ if (sched_setaffinity(0, sizeof(cpuset), &cpuset) != 0)
	769	+ fatal_error("sched_setaffinity to CPU 0");
	770	+}
	771	+
	772	+static void test_context_switch(void)
	773	+{
	774	+ struct futex_info *finfo;
	775	+ int i;
	776	+
	777	+ /* Affinitize to one CPU to force context switches */
	778	+ affinitize_cpu0();
	779	+
	780	+ req_xtiledata_perm();
	781	+
	782	+ printf("[RUN]\tCheck tiledata context switches, %d iterations, %d threads.\n",
	783	+ ctxtswtest_config.iterations,
	784	+ ctxtswtest_config.num_threads);
	785	+
	786	+
	787	+ finfo = malloc(sizeof(finfo) ctxtswtest_config.num_threads);
	788	+ if (!finfo)
	789	+ fatal_error("malloc()");
	790	+
	791	+ create_threads(ctxtswtest_config.num_threads, finfo);
	792	+
	793	+ /*
	794	+ * This thread wakes up thread 0
	795	+ * Thread 0 will wake up 1
	796	+ * Thread 1 will wake up 2
	797	+ * ...
	798	+ * the last thread will wake up 0
	799	+ *
	800	+ * ... this will repeat for the configured
	801	+ * number of iterations.
	802	+ */
	803	+ pthread_mutex_unlock(&finfo[0].mutex);
	804	+
	805	+ /* Wait for all the threads to finish: */
	806	+ for (i = 0; i < ctxtswtest_config.num_threads; i++) {
	807	+ void *thread_retval;
	808	+ int rc;
	809	+
	810	+ rc = pthread_join(finfo[i].thread, &thread_retval);
	811	+
	812	+ if (rc)
	813	+ fatal_error("pthread_join() failed for thread %d err: %d\n",
	814	+ i, rc);
	815	+
	816	+ if (thread_retval != &finfo[i])
	817	+ fatal_error("unexpected thread retval for thread %d: %p\n",
	818	+ i, thread_retval);
	819	+
	820	+ }
	821	+
	822	+ printf("[OK]\tNo incorrect case was found.\n");
	823	+
	824	+ free(finfo);
	825	+}
	826	+
	827	+int main(void)
	828	+{
	829	+ /* Check hardware availability at first */
	830	+ check_cpuid_xsave();
	831	+ check_cpuid_xtiledata();
	832	+
	833	+ init_stashed_xsave();
	834	+ sethandler(SIGILL, handle_noperm, 0);
	835	+
	836	+ test_dynamic_state();
	837	+
	838	+ /* Request permission for the following tests */
	839	+ req_xtiledata_perm();
	840	+
	841	+ test_fork();
	842	+
	843	+ ctxtswtest_config.iterations = 10;
	844	+ ctxtswtest_config.num_threads = 5;
	845	+ test_context_switch();
	846	+
	847	+ clearhandler(SIGILL);
	848	+ free_stashed_xsave();
	849	+
	850	+ return 0;
	851	+}

Merge tag 'x86-fpu-2021-11-01' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code