Merge tag 'x86-fpu-2021-11-01' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull x86 fpu updates from Thomas Gleixner: - Cleanup of extable fixup handling to be more robust, which in turn allows to make the FPU exception fixups more robust as well. - Change the return code for signal frame related failures from explicit error codes to a boolean fail/success as that's all what the calling code evaluates. - A large refactoring of the FPU code to prepare for adding AMX support: - Distangle the public header maze and remove especially the misnomed kitchen sink internal.h which is despite it's name included all over the place. - Add a proper abstraction for the register buffer storage (struct fpstate) which allows to dynamically size the buffer at runtime by flipping the pointer to the buffer container from the default container which is embedded in task_struct::tread::fpu to a dynamically allocated container with a larger register buffer. - Convert the code over to the new fpstate mechanism. - Consolidate the KVM FPU handling by moving the FPU related code into the FPU core which removes the number of exports and avoids adding even more export when AMX has to be supported in KVM. This also removes duplicated code which was of course unnecessary different and incomplete in the KVM copy. - Simplify the KVM FPU buffer handling by utilizing the new fpstate container and just switching the buffer pointer from the user space buffer to the KVM guest buffer when entering vcpu_run() and flipping it back when leaving the function. This cuts the memory requirements of a vCPU for FPU buffers in half and avoids pointless memory copy operations. This also solves the so far unresolved problem of adding AMX support because the current FPU buffer handling of KVM inflicted a circular dependency between adding AMX support to the core and to KVM. With the new scheme of switching fpstate AMX support can be added to the core code without affecting KVM. - Replace various variables with proper data structures so the extra information required for adding dynamically enabled FPU features (AMX) can be added in one place - Add AMX (Advanced Matrix eXtensions) support (finally): AMX is a large XSTATE component which is going to be available with Saphire Rapids XEON CPUs. The feature comes with an extra MSR (MSR_XFD) which allows to trap the (first) use of an AMX related instruction, which has two benefits: 1) It allows the kernel to control access to the feature 2) It allows the kernel to dynamically allocate the large register state buffer instead of burdening every task with the the extra 8K or larger state storage. It would have been great to gain this kind of control already with AVX512. The support comes with the following infrastructure components: 1) arch_prctl() to - read the supported features (equivalent to XGETBV(0)) - read the permitted features for a task - request permission for a dynamically enabled feature Permission is granted per process, inherited on fork() and cleared on exec(). The permission policy of the kernel is restricted to sigaltstack size validation, but the syscall obviously allows further restrictions via seccomp etc. 2) A stronger sigaltstack size validation for sys_sigaltstack(2) which takes granted permissions and the potentially resulting larger signal frame into account. This mechanism can also be used to enforce factual sigaltstack validation independent of dynamic features to help with finding potential victims of the 2K sigaltstack size constant which is broken since AVX512 support was added. 3) Exception handling for #NM traps to catch first use of a extended feature via a new cause MSR. If the exception was caused by the use of such a feature, the handler checks permission for that feature. If permission has not been granted, the handler sends a SIGILL like the #UD handler would do if the feature would have been disabled in XCR0. If permission has been granted, then a new fpstate which fits the larger buffer requirement is allocated. In the unlikely case that this allocation fails, the handler sends SIGSEGV to the task. That's not elegant, but unavoidable as the other discussed options of preallocation or full per task permissions come with their own set of horrors for kernel and/or userspace. So this is the lesser of the evils and SIGSEGV caused by unexpected memory allocation failures is not a fundamentally new concept either. When allocation succeeds, the fpstate properties are filled in to reflect the extended feature set and the resulting sizes, the fpu::fpstate pointer is updated accordingly and the trap is disarmed for this task permanently. 4) Enumeration and size calculations 5) Trap switching via MSR_XFD The XFD (eXtended Feature Disable) MSR is context switched with the same life time rules as the FPU register state itself. The mechanism is keyed off with a static key which is default disabled so !AMX equipped CPUs have zero overhead. On AMX enabled CPUs the overhead is limited by comparing the tasks XFD value with a per CPU shadow variable to avoid redundant MSR writes. In case of switching from a AMX using task to a non AMX using task or vice versa, the extra MSR write is obviously inevitable. All other places which need to be aware of the variable feature sets and resulting variable sizes are not affected at all because they retrieve the information (feature set, sizes) unconditonally from the fpstate properties. 6) Enable the new AMX states Note, this is relatively new code despite the fact that AMX support is in the works for more than a year now. The big refactoring of the FPU code, which allowed to do a proper integration has been started exactly 3 weeks ago. Refactoring of the existing FPU code and of the original AMX patches took a week and has been subject to extensive review and testing. The only fallout which has not been caught in review and testing right away was restricted to AMX enabled systems, which is completely irrelevant for anyone outside Intel and their early access program. There might be dragons lurking as usual, but so far the fine grained refactoring has held up and eventual yet undetected fallout is bisectable and should be easily addressable before the 5.16 release. Famous last words... Many thanks to Chang Bae and Dave Hansen for working hard on this and also to the various test teams at Intel who reserved extra capacity to follow the rapid development of this closely which provides the confidence level required to offer this rather large update for inclusion into 5.16-rc1 * tag 'x86-fpu-2021-11-01' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (110 commits) Documentation/x86: Add documentation for using dynamic XSTATE features x86/fpu: Include vmalloc.h for vzalloc() selftests/x86/amx: Add context switch test selftests/x86/amx: Add test cases for AMX state management x86/fpu/amx: Enable the AMX feature in 64-bit mode x86/fpu: Add XFD handling for dynamic states x86/fpu: Calculate the default sizes independently x86/fpu/amx: Define AMX state components and have it used for boot-time checks x86/fpu/xstate: Prepare XSAVE feature table for gaps in state component numbers x86/fpu/xstate: Add fpstate_realloc()/free() x86/fpu/xstate: Add XFD #NM handler x86/fpu: Update XFD state where required x86/fpu: Add sanity checks for XFD x86/fpu: Add XFD state to fpstate x86/msr-index: Add MSRs for XFD x86/cpufeatures: Add eXtended Feature Disabling (XFD) feature bit x86/fpu: Reset permission and fpstate on exec() x86/fpu: Prepare fpu_clone() for dynamically enabled features x86/fpu/signal: Prepare for variable sigframe length x86/signal: Use fpu::__state_user_size for sigalt stack validation ...

Linus Torvalds committed 5 years ago

Commit 8cb1ae12 parents 7d20dd3 d7a959067 files · +3458 −1575Browse files

Unified · Split

67 files

Documentation/admin-guide/kernel-parameters.txt +9 −0
Documentation/x86/index.rst +1 −0
Documentation/x86/xstate.rst +65 −0
arch/Kconfig +3 −0
arch/x86/Kconfig +17 −0
arch/x86/events/perf_event.h +1 −0
arch/x86/ia32/ia32_signal.c +7 −8
arch/x86/include/asm/asm.h +20 −30
arch/x86/include/asm/cpufeatures.h +2 −0
arch/x86/include/asm/extable.h +28 −16
arch/x86/include/asm/extable_fixup_types.h +22 −0
arch/x86/include/asm/fpu/api.h +56 −2
arch/x86/include/asm/fpu/internal.h +0 −540
arch/x86/include/asm/fpu/sched.h +68 −0
arch/x86/include/asm/fpu/signal.h +12 −1
arch/x86/include/asm/fpu/types.h +206 −8
arch/x86/include/asm/fpu/xcr.h +0 −11
arch/x86/include/asm/fpu/xstate.h +30 −60
arch/x86/include/asm/kvm_host.h +2 −5
arch/x86/include/asm/msr-index.h +2 −0
arch/x86/include/asm/msr.h +2 −2
arch/x86/include/asm/pkru.h +1 −1
arch/x86/include/asm/processor.h +3 −6
arch/x86/include/asm/proto.h +1 −1
arch/x86/include/asm/segment.h +1 −1
arch/x86/include/asm/trace/fpu.h +2 −2
arch/x86/include/uapi/asm/prctl.h +4 −0
arch/x86/kernel/cpu/bugs.c +1 −1
arch/x86/kernel/cpu/common.c +1 −1
arch/x86/kernel/cpu/cpuid-deps.c +2 −0
arch/x86/kernel/cpu/mce/core.c +11 −29
arch/x86/kernel/cpu/mce/internal.h +2 −12
arch/x86/kernel/cpu/mce/severity.c +11 −11
arch/x86/kernel/fpu/bugs.c +1 −1
arch/x86/kernel/fpu/context.h +83 −0
arch/x86/kernel/fpu/core.c +315 −77
arch/x86/kernel/fpu/init.c +34 −42
arch/x86/kernel/fpu/internal.h +28 −0
arch/x86/kernel/fpu/legacy.h +115 −0
arch/x86/kernel/fpu/regset.c +20 −16
arch/x86/kernel/fpu/signal.c +145 −140
arch/x86/kernel/fpu/xstate.c +712 −186
arch/x86/kernel/fpu/xstate.h +278 −0
arch/x86/kernel/process.c +23 −4
arch/x86/kernel/process_32.c +2 −3
arch/x86/kernel/process_64.c +2 −3
arch/x86/kernel/ptrace.c +1 −1
arch/x86/kernel/sev.c +1 −1
arch/x86/kernel/signal.c +71 −12
arch/x86/kernel/smpboot.c +1 −1
arch/x86/kernel/traps.c +39 −1
arch/x86/kvm/svm/sev.c +1 −1
arch/x86/kvm/svm/svm.c +4 −3
arch/x86/kvm/vmx/vmx.c +1 −1
arch/x86/kvm/x86.c +31 −227
arch/x86/lib/copy_mc_64.S +4 −4
arch/x86/math-emu/fpu_aux.c +1 −1
arch/x86/math-emu/fpu_entry.c +3 −3
arch/x86/math-emu/fpu_system.h +1 −1
arch/x86/mm/extable.c +56 −79
arch/x86/net/bpf_jit_comp.c +2 −9
arch/x86/power/cpu.c +1 −1
include/linux/signal.h +6 −0
kernel/signal.c +29 −6
scripts/sorttable.c +2 −2
tools/testing/selftests/x86/Makefile +1 −1
tools/testing/selftests/x86/amx.c +851 −0

Documentation/admin-guide/kernel-parameters.txt+9−0

@@ -5497,6 +5497,15 @@ · 5496 unmodified lines
5497	stifb= [HW]	5497	stifb= [HW]
5498	Format: bpp:<bpp1>[:<bpp2>[:<bpp3>...]]	5498	Format: bpp:<bpp1>[:<bpp2>[:<bpp3>...]]
5499		5499
		5500	strict_sas_size=
		5501	[X86]
		5502	Format: <bool>
		5503	Enable or disable strict sigaltstack size checks
		5504	against the required signal frame size which
		5505	depends on the supported FPU features. This can
		5506	be used to filter out binaries which have
		5507	not yet been made aware of AT_MINSIGSTKSZ.
		5508
5500	sunrpc.min_resvport=	5509	sunrpc.min_resvport=
5501	sunrpc.max_resvport=	5510	sunrpc.max_resvport=
5502	[NFS,SUNRPC]	5511	[NFS,SUNRPC]

Documentation/x86/index.rst+1−0

@@ -37,3 +37,4 @@ x86-specific Documentation · 36 unmodified lines
37	sgx	37	sgx
38	features	38	features
39	elf_auxvec	39	elf_auxvec
		40	xstate

Documentation/x86/xstate.rst+65−0

@@ -0,0 +1,65 @@
	1	Using XSTATE features in user space applications
	2	================================================
	3
	4	The x86 architecture supports floating-point extensions which are
	5	enumerated via CPUID. Applications consult CPUID and use XGETBV to
	6	evaluate which features have been enabled by the kernel XCR0.
	7
	8	Up to AVX-512 and PKRU states, these features are automatically enabled by
	9	the kernel if available. Features like AMX TILE_DATA (XSTATE component 18)
	10	are enabled by XCR0 as well, but the first use of related instruction is
	11	trapped by the kernel because by default the required large XSTATE buffers
	12	are not allocated automatically.
	13
	14	Using dynamically enabled XSTATE features in user space applications
	15	--------------------------------------------------------------------
	16
	17	The kernel provides an arch_prctl(2) based mechanism for applications to
	18	request the usage of such features. The arch_prctl(2) options related to
	19	this are:
	20
	21	-ARCH_GET_XCOMP_SUPP
	22
	23	arch_prctl(ARCH_GET_XCOMP_SUPP, &features);
	24
	25	ARCH_GET_XCOMP_SUPP stores the supported features in userspace storage of
	26	type uint64_t. The second argument is a pointer to that storage.
	27
	28	-ARCH_GET_XCOMP_PERM
	29
	30	arch_prctl(ARCH_GET_XCOMP_PERM, &features);
	31
	32	ARCH_GET_XCOMP_PERM stores the features for which the userspace process
	33	has permission in userspace storage of type uint64_t. The second argument
	34	is a pointer to that storage.
	35
	36	-ARCH_REQ_XCOMP_PERM
	37
	38	arch_prctl(ARCH_REQ_XCOMP_PERM, feature_nr);
	39
	40	ARCH_REQ_XCOMP_PERM allows to request permission for a dynamically enabled
	41	feature or a feature set. A feature set can be mapped to a facility, e.g.
	42	AMX, and can require one or more XSTATE components to be enabled.
	43
	44	The feature argument is the number of the highest XSTATE component which
	45	is required for a facility to work.
	46
	47	When requesting permission for a feature, the kernel checks the
	48	availability. The kernel ensures that sigaltstacks in the process's tasks
	49	are large enough to accommodate the resulting large signal frame. It
	50	enforces this both during ARCH_REQ_XCOMP_SUPP and during any subsequent
	51	sigaltstack(2) calls. If an installed sigaltstack is smaller than the
	52	resulting sigframe size, ARCH_REQ_XCOMP_SUPP results in -ENOSUPP. Also,
	53	sigaltstack(2) results in -ENOMEM if the requested altstack is too small
	54	for the permitted features.
	55
	56	Permission, when granted, is valid per process. Permissions are inherited
	57	on fork(2) and cleared on exec(3).
	58
	59	The first use of an instruction related to a dynamically enabled feature is
	60	trapped by the kernel. The trap handler checks whether the process has
	61	permission to use the feature. If the process has no permission then the
	62	kernel sends SIGILL to the application. If the process has permission then
	63	the handler allocates a larger xstate buffer for the task so the large
	64	state can be context switched. In the unlikely cases that the allocation
	65	fails, the kernel sends SIGSEGV.

arch/Kconfig+3−0

@@ -1288,6 +1288,9 @@ config ARCH_HAS_ELFCORE_COMPAT · 1287 unmodified lines
1288	config ARCH_HAS_PARANOID_L1D_FLUSH	1288	config ARCH_HAS_PARANOID_L1D_FLUSH
1289	bool	1289	bool
1290		1290
		1291	config DYNAMIC_SIGFRAME
		1292	bool
		1293
1291	source "kernel/gcov/Kconfig"	1294	source "kernel/gcov/Kconfig"
1292		1295
1293	source "scripts/gcc-plugins/Kconfig"	1296	source "scripts/gcc-plugins/Kconfig"

arch/x86/Kconfig+17−0

@@ -125,6 +125,7 @@ config X86 · 124 unmodified lines
125	select CLOCKSOURCE_VALIDATE_LAST_CYCLE	125	select CLOCKSOURCE_VALIDATE_LAST_CYCLE
126	select CLOCKSOURCE_WATCHDOG	126	select CLOCKSOURCE_WATCHDOG
127	select DCACHE_WORD_ACCESS	127	select DCACHE_WORD_ACCESS
		128	select DYNAMIC_SIGFRAME
128	select EDAC_ATOMIC_SCRUB	129	select EDAC_ATOMIC_SCRUB
129	select EDAC_SUPPORT	130	select EDAC_SUPPORT
130	select GENERIC_CLOCKEVENTS_BROADCAST if X86_64 \|\| (X86_32 && X86_LOCAL_APIC)	131	select GENERIC_CLOCKEVENTS_BROADCAST if X86_64 \|\| (X86_32 && X86_LOCAL_APIC)
@@ -2399,6 +2400,22 @@ config MODIFY_LDT_SYSCALL · 2268 unmodified lines
2399		2400
2400	Saying 'N' here may make sense for embedded or server kernels.	2401	Saying 'N' here may make sense for embedded or server kernels.
2401		2402
		2403	config STRICT_SIGALTSTACK_SIZE
		2404	bool "Enforce strict size checking for sigaltstack"
		2405	depends on DYNAMIC_SIGFRAME
		2406	help
		2407	For historical reasons MINSIGSTKSZ is a constant which became
		2408	already too small with AVX512 support. Add a mechanism to
		2409	enforce strict checking of the sigaltstack size against the
		2410	real size of the FPU frame. This option enables the check
		2411	by default. It can also be controlled via the kernel command
		2412	line option 'strict_sas_size' independent of this config
		2413	switch. Enabling it might break existing applications which
		2414	allocate a too small sigaltstack but 'work' because they
		2415	never get a signal delivered.
		2416
		2417	Say 'N' unless you want to really enforce this check.
		2418
2402	source "kernel/livepatch/Kconfig"	2419	source "kernel/livepatch/Kconfig"
2403		2420
2404	endmenu	2421	endmenu

arch/x86/events/perf_event.h+1−0

@@ -14,6 +14,7 @@ · 13 unmodified lines
14		14
15	#include <linux/perf_event.h>	15	#include <linux/perf_event.h>
16		16
		17	#include <asm/fpu/xstate.h>
17	#include <asm/intel_ds.h>	18	#include <asm/intel_ds.h>
18	#include <asm/cpu.h>	19	#include <asm/cpu.h>
19		20

arch/x86/ia32/ia32_signal.c+7−8

 #include <linux/syscalls.h>
 #include <asm/ucontext.h>
 #include <linux/uaccess.h>
-#include <asm/fpu/internal.h>
 #include <asm/fpu/signal.h>
 #include <asm/ptrace.h>
 #include <asm/ia32_unistd.h>
 /*
  * Do a signal return; undo the signal stack.
  */
-static int ia32_restore_sigcontext(struct pt_regs *regs,
+static bool ia32_restore_sigcontext(struct pt_regs *regs,
-				   struct sigcontext_32 __user *usc)
+				    struct sigcontext_32 __user *usc)
 {
 	struct sigcontext_32 sc;
 	current->restart_block.fn = do_no_restart_syscall;
 	if (unlikely(copy_from_user(&sc, usc, sizeof(sc))))
-		return -EFAULT;
+		return false;
 	/* Get only the ia32 registers. */
 	regs->bx = sc.bx;
 	set_current_blocked(&set);
-	if (ia32_restore_sigcontext(regs, &frame->sc))
+	if (!ia32_restore_sigcontext(regs, &frame->sc))
 		goto badframe;
 	return regs->ax;
 	set_current_blocked(&set);
-	if (ia32_restore_sigcontext(regs, &frame->uc.uc_mcontext))
+	if (!ia32_restore_sigcontext(regs, &frame->uc.uc_mcontext))
 		goto badframe;
 	if (compat_restore_altstack(&frame->uc.uc_stack))
 	sp = fpu__alloc_mathframe(sp, 1, &fx_aligned, &math_size);
 	*fpstate = (struct _fpstate_32 __user *) sp;
-	if (copy_fpstate_to_sigframe(*fpstate, (void __user *)fx_aligned,
+	if (!copy_fpstate_to_sigframe(*fpstate, (void __user *)fx_aligned,
-				     math_size) < 0)
+				      math_size))
 		return (void __user *) -1L;
 	sp -= frame_size;

arch/x86/include/asm/asm.h+20−30

@@ -122,28 +122,19 @@ · 121 unmodified lines
122		122
123	#ifdef __KERNEL__	123	#ifdef __KERNEL__
124		124
		125	# include <asm/extable_fixup_types.h>
		126
125	/* Exception table entry */	127	/* Exception table entry */
126	#ifdef __ASSEMBLY__	128	#ifdef __ASSEMBLY__
127	# define _ASM_EXTABLE_HANDLE(from, to, handler) \	129
		130	# define _ASM_EXTABLE_TYPE(from, to, type) \
128	.pushsection "__ex_table","a" ; \	131	.pushsection "__ex_table","a" ; \
129	.balign 4 ; \	132	.balign 4 ; \
130	.long (from) - . ; \	133	.long (from) - . ; \
131	.long (to) - . ; \	134	.long (to) - . ; \
132	.long (handler) - . ; \	135	.long type ; \
133	.popsection	136	.popsection
134		137
135	# define _ASM_EXTABLE(from, to) \
136	_ASM_EXTABLE_HANDLE(from, to, ex_handler_default)
137
138	# define _ASM_EXTABLE_UA(from, to) \
139	_ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess)
140
141	# define _ASM_EXTABLE_CPY(from, to) \
142	_ASM_EXTABLE_HANDLE(from, to, ex_handler_copy)
143
144	# define _ASM_EXTABLE_FAULT(from, to) \
145	_ASM_EXTABLE_HANDLE(from, to, ex_handler_fault)
146
147	# ifdef CONFIG_KPROBES	138	# ifdef CONFIG_KPROBES
148	# define _ASM_NOKPROBE(entry) \	139	# define _ASM_NOKPROBE(entry) \
149	.pushsection "_kprobe_blacklist","aw" ; \	140	.pushsection "_kprobe_blacklist","aw" ; \
@@ -155,27 +146,15 @@ · 5 unmodified lines
155	# endif	146	# endif
156		147
157	#else /* ! __ASSEMBLY__ */	148	#else /* ! __ASSEMBLY__ */
158	# define _EXPAND_EXTABLE_HANDLE(x) #x	149
159	# define _ASM_EXTABLE_HANDLE(from, to, handler) \	150	# define _ASM_EXTABLE_TYPE(from, to, type) \
160	" .pushsection \"__ex_table\",\"a\"\n" \	151	" .pushsection \"__ex_table\",\"a\"\n" \
161	" .balign 4\n" \	152	" .balign 4\n" \
162	" .long (" #from ") - .\n" \	153	" .long (" #from ") - .\n" \
163	" .long (" #to ") - .\n" \	154	" .long (" #to ") - .\n" \
164	" .long (" _EXPAND_EXTABLE_HANDLE(handler) ") - .\n" \	155	" .long " __stringify(type) " \n" \
165	" .popsection\n"	156	" .popsection\n"
166		157
167	# define _ASM_EXTABLE(from, to) \
168	_ASM_EXTABLE_HANDLE(from, to, ex_handler_default)
169
170	# define _ASM_EXTABLE_UA(from, to) \
171	_ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess)
172
173	# define _ASM_EXTABLE_CPY(from, to) \
174	_ASM_EXTABLE_HANDLE(from, to, ex_handler_copy)
175
176	# define _ASM_EXTABLE_FAULT(from, to) \
177	_ASM_EXTABLE_HANDLE(from, to, ex_handler_fault)
178
179	/* For C file, we already have NOKPROBE_SYMBOL macro */	158	/* For C file, we already have NOKPROBE_SYMBOL macro */
180		159
181	/*	160	/*
@@ -188,6 +167,17 @@ register unsigned long current_stack_pointer asm(_ASM_SP); · 6 unmodified lines
188	#define ASM_CALL_CONSTRAINT "+r" (current_stack_pointer)	167	#define ASM_CALL_CONSTRAINT "+r" (current_stack_pointer)
189	#endif /* __ASSEMBLY__ */	168	#endif /* __ASSEMBLY__ */
190		169
191	#endif /* __KERNEL__ */	170	#define _ASM_EXTABLE(from, to) \
		171	_ASM_EXTABLE_TYPE(from, to, EX_TYPE_DEFAULT)
192		172
		173	#define _ASM_EXTABLE_UA(from, to) \
		174	_ASM_EXTABLE_TYPE(from, to, EX_TYPE_UACCESS)
		175
		176	#define _ASM_EXTABLE_CPY(from, to) \
		177	_ASM_EXTABLE_TYPE(from, to, EX_TYPE_COPY)
		178
		179	#define _ASM_EXTABLE_FAULT(from, to) \
		180	_ASM_EXTABLE_TYPE(from, to, EX_TYPE_FAULT)
		181
		182	#endif /* __KERNEL__ */
193	#endif /* _ASM_X86_ASM_H */	183	#endif /* _ASM_X86_ASM_H */

arch/x86/include/asm/cpufeatures.h+2−0

 #define X86_FEATURE_XSAVEC		(10*32+ 1) /* XSAVEC instruction */
 #define X86_FEATURE_XGETBV1		(10*32+ 2) /* XGETBV with ECX = 1 instruction */
 #define X86_FEATURE_XSAVES		(10*32+ 3) /* XSAVES/XRSTORS instructions */
+#define X86_FEATURE_XFD			(10*32+ 4) /* "" eXtended Feature Disabling */
 /*
  * Extended auxiliary flags: Linux defined - for features scattered in various
 /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */
 #define X86_FEATURE_AVX_VNNI		(12*32+ 4) /* AVX VNNI instructions */
 #define X86_FEATURE_AVX512_BF16		(12*32+ 5) /* AVX512 BFLOAT16 instructions */
+#define X86_FEATURE_AMX_TILE		(18*32+24) /* AMX tile Support */
 /* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */
 #define X86_FEATURE_CLZERO		(13*32+ 0) /* CLZERO instruction */

arch/x86/include/asm/extable.h+28−16

 /* SPDX-License-Identifier: GPL-2.0 */
 #ifndef _ASM_X86_EXTABLE_H
 #define _ASM_X86_EXTABLE_H
+#include <asm/extable_fixup_types.h>
 /*
- * The exception table consists of triples of addresses relative to the
+ * The exception table consists of two addresses relative to the
- * exception table entry itself. The first address is of an instruction
+ * exception table entry itself and a type selector field.
- * that is allowed to fault, the second is the target at which the program
+ *
- * should continue. The third is a handler function to deal with the fault
+ * The first address is of an instruction that is allowed to fault, the
- * caused by the instruction in the first field.
+ * second is the target at which the program should continue.
+ *
+ * The type entry is used by fixup_exception() to select the handler to
+ * deal with the fault caused by the instruction in the first field.
  *
  * All the routines below use bits of fixup code that are out of line
  * with the main instruction path.  This means when everything is well,
  */
 struct exception_table_entry {
-	int insn, fixup, handler;
+	int insn, fixup, type;
 };
 struct pt_regs;
 	do {							\
 		(a)->fixup = (b)->fixup + (delta);		\
 		(b)->fixup = (tmp).fixup - (delta);		\
-		(a)->handler = (b)->handler + (delta);		\
+		(a)->type = (b)->type;				\
-		(b)->handler = (tmp).handler - (delta);		\
+		(b)->type = (tmp).type;				\
 	} while (0)
-enum handler_type {
-	EX_HANDLER_NONE,
-	EX_HANDLER_FAULT,
-	EX_HANDLER_UACCESS,
-	EX_HANDLER_OTHER
-};
 extern int fixup_exception(struct pt_regs *regs, int trapnr,
 			   unsigned long error_code, unsigned long fault_addr);
 extern int fixup_bug(struct pt_regs *regs, int trapnr);
-extern enum handler_type ex_get_fault_handler_type(unsigned long ip);
+extern int ex_get_fixup_type(unsigned long ip);
 extern void early_fixup_exception(struct pt_regs *regs, int trapnr);
+#ifdef CONFIG_X86_MCE
+extern void ex_handler_msr_mce(struct pt_regs *regs, bool wrmsr);
+#else
+static inline void ex_handler_msr_mce(struct pt_regs *regs, bool wrmsr) { }
+#endif
+#if defined(CONFIG_BPF_JIT) && defined(CONFIG_X86_64)
+bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs);
+#else
+static inline bool ex_handler_bpf(const struct exception_table_entry *x,
+				  struct pt_regs *regs) { return false; }
+#endif
 #endif

arch/x86/include/asm/extable_fixup_types.h+22−0

@@ -0,0 +1,22 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#ifndef _ASM_X86_EXTABLE_FIXUP_TYPES_H
	3	#define _ASM_X86_EXTABLE_FIXUP_TYPES_H
	4
	5	#define EX_TYPE_NONE 0
	6	#define EX_TYPE_DEFAULT 1
	7	#define EX_TYPE_FAULT 2
	8	#define EX_TYPE_UACCESS 3
	9	#define EX_TYPE_COPY 4
	10	#define EX_TYPE_CLEAR_FS 5
	11	#define EX_TYPE_FPU_RESTORE 6
	12	#define EX_TYPE_WRMSR 7
	13	#define EX_TYPE_RDMSR 8
	14	#define EX_TYPE_BPF 9
	15
	16	#define EX_TYPE_WRMSR_IN_MCE 10
	17	#define EX_TYPE_RDMSR_IN_MCE 11
	18
	19	#define EX_TYPE_DEFAULT_MCE_SAFE 12
	20	#define EX_TYPE_FAULT_MCE_SAFE 13
	21
	22	#endif

arch/x86/include/asm/fpu/api.h+56−2

@@ -12,6 +12,8 @@ · 11 unmodified lines
12	#define _ASM_X86_FPU_API_H	12	#define _ASM_X86_FPU_API_H
13	#include <linux/bottom_half.h>	13	#include <linux/bottom_half.h>
14		14
		15	#include <asm/fpu/types.h>
		16
15	/*	17	/*
16	* Use kernel_fpu_begin/end() if you intend to use FPU in kernel context. It	18	* Use kernel_fpu_begin/end() if you intend to use FPU in kernel context. It
17	* disables preemption so be careful if you intend to use it for long periods	19	* disables preemption so be careful if you intend to use it for long periods
@@ -48,9 +50,9 @@ static inline void kernel_fpu_begin(void) · 30 unmodified lines
48	}	50	}
49		51
50	/*	52	/*
51	* Use fpregs_lock() while editing CPU's FPU registers or fpu->state.	53	* Use fpregs_lock() while editing CPU's FPU registers or fpu->fpstate.
52	* A context switch will (and softirq might) save CPU's FPU registers to	54	* A context switch will (and softirq might) save CPU's FPU registers to
53	* fpu->state and set TIF_NEED_FPU_LOAD leaving CPU's FPU registers in	55	* fpu->fpstate.regs and set TIF_NEED_FPU_LOAD leaving CPU's FPU registers in
54	* a random state.	56	* a random state.
55	*	57	*
56	* local_bh_disable() protects against both preemption and soft interrupts	58	* local_bh_disable() protects against both preemption and soft interrupts
@@ -108,4 +110,56 @@ extern int cpu_has_xfeatures(u64 xfeatures_mask, const char **feature_name); · 51 unmodified lines
108		110
109	static inline void update_pasid(void) { }	111	static inline void update_pasid(void) { }
110		112
		113	/* Trap handling */
		114	extern int fpu__exception_code(struct fpu *fpu, int trap_nr);
		115	extern void fpu_sync_fpstate(struct fpu *fpu);
		116	extern void fpu_reset_from_exception_fixup(void);
		117
		118	/* Boot, hotplug and resume */
		119	extern void fpu__init_cpu(void);
		120	extern void fpu__init_system(struct cpuinfo_x86 *c);
		121	extern void fpu__init_check_bugs(void);
		122	extern void fpu__resume_cpu(void);
		123
		124	#ifdef CONFIG_MATH_EMULATION
		125	extern void fpstate_init_soft(struct swregs_state *soft);
		126	#else
		127	static inline void fpstate_init_soft(struct swregs_state *soft) {}
		128	#endif
		129
		130	/* State tracking */
		131	DECLARE_PER_CPU(struct fpu *, fpu_fpregs_owner_ctx);
		132
		133	/* Process cleanup */
		134	#ifdef CONFIG_X86_64
		135	extern void fpstate_free(struct fpu *fpu);
		136	#else
		137	static inline void fpstate_free(struct fpu *fpu) { }
		138	#endif
		139
		140	/* fpstate-related functions which are exported to KVM */
		141	extern void fpstate_clear_xstate_component(struct fpstate *fps, unsigned int xfeature);
		142
		143	/* KVM specific functions */
		144	extern bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu);
		145	extern void fpu_free_guest_fpstate(struct fpu_guest *gfpu);
		146	extern int fpu_swap_kvm_fpstate(struct fpu_guest *gfpu, bool enter_guest);
		147
		148	extern void fpu_copy_guest_fpstate_to_uabi(struct fpu_guest gfpu, void buf, unsigned int size, u32 pkru);
		149	extern int fpu_copy_uabi_to_guest_fpstate(struct fpu_guest gfpu, const void buf, u64 xcr0, u32 *vpkru);
		150
		151	static inline void fpstate_set_confidential(struct fpu_guest *gfpu)
		152	{
		153	gfpu->fpstate->is_confidential = true;
		154	}
		155
		156	static inline bool fpstate_is_confidential(struct fpu_guest *gfpu)
		157	{
		158	return gfpu->fpstate->is_confidential;
		159	}
		160
		161	/* prctl */
		162	struct task_struct;
		163	extern long fpu_xstate_prctl(struct task_struct *tsk, int option, unsigned long arg2);
		164
111	#endif /* _ASM_X86_FPU_API_H */	165	#endif /* _ASM_X86_FPU_API_H */

arch/x86/include/asm/fpu/internal.h+0−540

@@ -1,540 +0,0 @@
1	/* SPDX-License-Identifier: GPL-2.0 */
2	/*
3	* Copyright (C) 1994 Linus Torvalds
4	*
5	* Pentium III FXSR, SSE support
6	* General FPU state handling cleanups
7	* Gareth Hughes <gareth@valinux.com>, May 2000
8	* x86-64 work by Andi Kleen 2002
9	*/
10
11	#ifndef _ASM_X86_FPU_INTERNAL_H
12	#define _ASM_X86_FPU_INTERNAL_H
13
14	#include <linux/compat.h>
15	#include <linux/sched.h>
16	#include <linux/slab.h>
17	#include <linux/mm.h>
18
19	#include <asm/user.h>
20	#include <asm/fpu/api.h>
21	#include <asm/fpu/xstate.h>
22	#include <asm/fpu/xcr.h>
23	#include <asm/cpufeature.h>
24	#include <asm/trace/fpu.h>
25
26	/*
27	* High level FPU state handling functions:
28	*/
29	extern int fpu__restore_sig(void __user *buf, int ia32_frame);
30	extern void fpu__drop(struct fpu *fpu);
31	extern void fpu__clear_user_states(struct fpu *fpu);
32	extern int fpu__exception_code(struct fpu *fpu, int trap_nr);
33
34	extern void fpu_sync_fpstate(struct fpu *fpu);
35
36	/* Clone and exit operations */
37	extern int fpu_clone(struct task_struct *dst);
38	extern void fpu_flush_thread(void);
39
40	/*
41	* Boot time FPU initialization functions:
42	*/
43	extern void fpu__init_cpu(void);
44	extern void fpu__init_system_xstate(void);
45	extern void fpu__init_cpu_xstate(void);
46	extern void fpu__init_system(struct cpuinfo_x86 *c);
47	extern void fpu__init_check_bugs(void);
48	extern void fpu__resume_cpu(void);
49
50	/*
51	* Debugging facility:
52	*/
53	#ifdef CONFIG_X86_DEBUG_FPU
54	# define WARN_ON_FPU(x) WARN_ON_ONCE(x)
55	#else
56	# define WARN_ON_FPU(x) ({ (void)(x); 0; })
57	#endif
58
59	/*
60	* FPU related CPU feature flag helper routines:
61	*/
62	static __always_inline __pure bool use_xsaveopt(void)
63	{
64	return static_cpu_has(X86_FEATURE_XSAVEOPT);
65	}
66
67	static __always_inline __pure bool use_xsave(void)
68	{
69	return static_cpu_has(X86_FEATURE_XSAVE);
70	}
71
72	static __always_inline __pure bool use_fxsr(void)
73	{
74	return static_cpu_has(X86_FEATURE_FXSR);
75	}
76
77	/*
78	* fpstate handling functions:
79	*/
80
81	extern union fpregs_state init_fpstate;
82
83	extern void fpstate_init(union fpregs_state *state);
84	#ifdef CONFIG_MATH_EMULATION
85	extern void fpstate_init_soft(struct swregs_state *soft);
86	#else
87	static inline void fpstate_init_soft(struct swregs_state *soft) {}
88	#endif
89	extern void save_fpregs_to_fpstate(struct fpu *fpu);
90
91	/* Returns 0 or the negated trap number, which results in -EFAULT for #PF */
92	#define user_insn(insn, output, input...) \
93	({ \
94	int err; \
95	\
96	might_fault(); \
97	\
98	asm volatile(ASM_STAC "\n" \
99	"1: " #insn "\n" \
100	"2: " ASM_CLAC "\n" \
101	".section .fixup,\"ax\"\n" \
102	"3: negl %%eax\n" \
103	" jmp 2b\n" \
104	".previous\n" \
105	_ASM_EXTABLE_FAULT(1b, 3b) \
106	: [err] "=a" (err), output \
107	: "0"(0), input); \
108	err; \
109	})
110
111	#define kernel_insn_err(insn, output, input...) \
112	({ \
113	int err; \
114	asm volatile("1:" #insn "\n\t" \
115	"2:\n" \
116	".section .fixup,\"ax\"\n" \
117	"3: movl $-1,%[err]\n" \
118	" jmp 2b\n" \
119	".previous\n" \
120	_ASM_EXTABLE(1b, 3b) \
121	: [err] "=r" (err), output \
122	: "0"(0), input); \
123	err; \
124	})
125
126	#define kernel_insn(insn, output, input...) \
127	asm volatile("1:" #insn "\n\t" \
128	"2:\n" \
129	_ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_fprestore) \
130	: output : input)
131
132	static inline int fnsave_to_user_sigframe(struct fregs_state __user *fx)
133	{
134	return user_insn(fnsave %[fx]; fwait, [fx] "=m" (fx), "m" (fx));
135	}
136
137	static inline int fxsave_to_user_sigframe(struct fxregs_state __user *fx)
138	{
139	if (IS_ENABLED(CONFIG_X86_32))
140	return user_insn(fxsave %[fx], [fx] "=m" (fx), "m" (fx));
141	else
142	return user_insn(fxsaveq %[fx], [fx] "=m" (fx), "m" (fx));
143
144	}
145
146	static inline void fxrstor(struct fxregs_state *fx)
147	{
148	if (IS_ENABLED(CONFIG_X86_32))
149	kernel_insn(fxrstor %[fx], "=m" (fx), [fx] "m" (fx));
150	else
151	kernel_insn(fxrstorq %[fx], "=m" (fx), [fx] "m" (fx));
152	}
153
154	static inline int fxrstor_safe(struct fxregs_state *fx)
155	{
156	if (IS_ENABLED(CONFIG_X86_32))
157	return kernel_insn_err(fxrstor %[fx], "=m" (fx), [fx] "m" (fx));
158	else
159	return kernel_insn_err(fxrstorq %[fx], "=m" (fx), [fx] "m" (fx));
160	}
161
162	static inline int fxrstor_from_user_sigframe(struct fxregs_state __user *fx)
163	{
164	if (IS_ENABLED(CONFIG_X86_32))
165	return user_insn(fxrstor %[fx], "=m" (fx), [fx] "m" (fx));
166	else
167	return user_insn(fxrstorq %[fx], "=m" (fx), [fx] "m" (fx));
168	}
169
170	static inline void frstor(struct fregs_state *fx)
171	{
172	kernel_insn(frstor %[fx], "=m" (fx), [fx] "m" (fx));
173	}
174
175	static inline int frstor_safe(struct fregs_state *fx)
176	{
177	return kernel_insn_err(frstor %[fx], "=m" (fx), [fx] "m" (fx));
178	}
179
180	static inline int frstor_from_user_sigframe(struct fregs_state __user *fx)
181	{
182	return user_insn(frstor %[fx], "=m" (fx), [fx] "m" (fx));
183	}
184
185	static inline void fxsave(struct fxregs_state *fx)
186	{
187	if (IS_ENABLED(CONFIG_X86_32))
188	asm volatile( "fxsave %[fx]" : [fx] "=m" (*fx));
189	else
190	asm volatile("fxsaveq %[fx]" : [fx] "=m" (*fx));
191	}
192
193	/* These macros all use (%edi)/(%rdi) as the single memory argument. */
194	#define XSAVE ".byte " REX_PREFIX "0x0f,0xae,0x27"
195	#define XSAVEOPT ".byte " REX_PREFIX "0x0f,0xae,0x37"
196	#define XSAVES ".byte " REX_PREFIX "0x0f,0xc7,0x2f"
197	#define XRSTOR ".byte " REX_PREFIX "0x0f,0xae,0x2f"
198	#define XRSTORS ".byte " REX_PREFIX "0x0f,0xc7,0x1f"
199
200	/*
201	* After this @err contains 0 on success or the negated trap number when
202	* the operation raises an exception. For faults this results in -EFAULT.
203	*/
204	#define XSTATE_OP(op, st, lmask, hmask, err) \
205	asm volatile("1:" op "\n\t" \
206	"xor %[err], %[err]\n" \
207	"2:\n\t" \
208	".pushsection .fixup,\"ax\"\n\t" \
209	"3: negl %%eax\n\t" \
210	"jmp 2b\n\t" \
211	".popsection\n\t" \
212	_ASM_EXTABLE_FAULT(1b, 3b) \
213	: [err] "=a" (err) \
214	: "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \
215	: "memory")
216
217	/*
218	* If XSAVES is enabled, it replaces XSAVEOPT because it supports a compact
219	* format and supervisor states in addition to modified optimization in
220	* XSAVEOPT.
221	*
222	* Otherwise, if XSAVEOPT is enabled, XSAVEOPT replaces XSAVE because XSAVEOPT
223	* supports modified optimization which is not supported by XSAVE.
224	*
225	* We use XSAVE as a fallback.
226	*
227	* The 661 label is defined in the ALTERNATIVE* macros as the address of the
228	* original instruction which gets replaced. We need to use it here as the
229	* address of the instruction where we might get an exception at.
230	*/
231	#define XSTATE_XSAVE(st, lmask, hmask, err) \
232	asm volatile(ALTERNATIVE_2(XSAVE, \
233	XSAVEOPT, X86_FEATURE_XSAVEOPT, \
234	XSAVES, X86_FEATURE_XSAVES) \
235	"\n" \
236	"xor %[err], %[err]\n" \
237	"3:\n" \
238	".pushsection .fixup,\"ax\"\n" \
239	"4: movl $-2, %[err]\n" \
240	"jmp 3b\n" \
241	".popsection\n" \
242	_ASM_EXTABLE(661b, 4b) \
243	: [err] "=r" (err) \
244	: "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \
245	: "memory")
246
247	/*
248	* Use XRSTORS to restore context if it is enabled. XRSTORS supports compact
249	* XSAVE area format.
250	*/
251	#define XSTATE_XRESTORE(st, lmask, hmask) \
252	asm volatile(ALTERNATIVE(XRSTOR, \
253	XRSTORS, X86_FEATURE_XSAVES) \
254	"\n" \
255	"3:\n" \
256	_ASM_EXTABLE_HANDLE(661b, 3b, ex_handler_fprestore)\
257	: \
258	: "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \
259	: "memory")
260
261	/*
262	* This function is called only during boot time when x86 caps are not set
263	* up and alternative can not be used yet.
264	*/
265	static inline void os_xrstor_booting(struct xregs_state *xstate)
266	{
267	u64 mask = xfeatures_mask_fpstate();
268	u32 lmask = mask;
269	u32 hmask = mask >> 32;
270	int err;
271
272	WARN_ON(system_state != SYSTEM_BOOTING);
273
274	if (boot_cpu_has(X86_FEATURE_XSAVES))
275	XSTATE_OP(XRSTORS, xstate, lmask, hmask, err);
276	else
277	XSTATE_OP(XRSTOR, xstate, lmask, hmask, err);
278
279	/*
280	* We should never fault when copying from a kernel buffer, and the FPU
281	* state we set at boot time should be valid.
282	*/
283	WARN_ON_FPU(err);
284	}
285
286	/*
287	* Save processor xstate to xsave area.
288	*
289	* Uses either XSAVE or XSAVEOPT or XSAVES depending on the CPU features
290	* and command line options. The choice is permanent until the next reboot.
291	*/
292	static inline void os_xsave(struct xregs_state *xstate)
293	{
294	u64 mask = xfeatures_mask_all;
295	u32 lmask = mask;
296	u32 hmask = mask >> 32;
297	int err;
298
299	WARN_ON_FPU(!alternatives_patched);
300
301	XSTATE_XSAVE(xstate, lmask, hmask, err);
302
303	/* We should never fault when copying to a kernel buffer: */
304	WARN_ON_FPU(err);
305	}
306
307	/*
308	* Restore processor xstate from xsave area.
309	*
310	* Uses XRSTORS when XSAVES is used, XRSTOR otherwise.
311	*/
312	static inline void os_xrstor(struct xregs_state *xstate, u64 mask)
313	{
314	u32 lmask = mask;
315	u32 hmask = mask >> 32;
316
317	XSTATE_XRESTORE(xstate, lmask, hmask);
318	}
319
320	/*
321	* Save xstate to user space xsave area.
322	*
323	* We don't use modified optimization because xrstor/xrstors might track
324	* a different application.
325	*
326	* We don't use compacted format xsave area for
327	* backward compatibility for old applications which don't understand
328	* compacted format of xsave area.
329	*/
330	static inline int xsave_to_user_sigframe(struct xregs_state __user *buf)
331	{
332	/*
333	* Include the features which are not xsaved/rstored by the kernel
334	* internally, e.g. PKRU. That's user space ABI and also required
335	* to allow the signal handler to modify PKRU.
336	*/
337	u64 mask = xfeatures_mask_uabi();
338	u32 lmask = mask;
339	u32 hmask = mask >> 32;
340	int err;
341
342	/*
343	* Clear the xsave header first, so that reserved fields are
344	* initialized to zero.
345	*/
346	err = __clear_user(&buf->header, sizeof(buf->header));
347	if (unlikely(err))
348	return -EFAULT;
349
350	stac();
351	XSTATE_OP(XSAVE, buf, lmask, hmask, err);
352	clac();
353
354	return err;
355	}
356
357	/*
358	* Restore xstate from user space xsave area.
359	*/
360	static inline int xrstor_from_user_sigframe(struct xregs_state __user *buf, u64 mask)
361	{
362	struct xregs_state xstate = ((__force struct xregs_state )buf);
363	u32 lmask = mask;
364	u32 hmask = mask >> 32;
365	int err;
366
367	stac();
368	XSTATE_OP(XRSTOR, xstate, lmask, hmask, err);
369	clac();
370
371	return err;
372	}
373
374	/*
375	* Restore xstate from kernel space xsave area, return an error code instead of
376	* an exception.
377	*/
378	static inline int os_xrstor_safe(struct xregs_state *xstate, u64 mask)
379	{
380	u32 lmask = mask;
381	u32 hmask = mask >> 32;
382	int err;
383
384	if (cpu_feature_enabled(X86_FEATURE_XSAVES))
385	XSTATE_OP(XRSTORS, xstate, lmask, hmask, err);
386	else
387	XSTATE_OP(XRSTOR, xstate, lmask, hmask, err);
388
389	return err;
390	}
391
392	extern void __restore_fpregs_from_fpstate(union fpregs_state *fpstate, u64 mask);
393
394	static inline void restore_fpregs_from_fpstate(union fpregs_state *fpstate)
395	{
396	__restore_fpregs_from_fpstate(fpstate, xfeatures_mask_fpstate());
397	}
398
399	extern int copy_fpstate_to_sigframe(void __user buf, void __user fp, int size);
400
401	/*
402	* FPU context switch related helper methods:
403	*/
404
405	DECLARE_PER_CPU(struct fpu *, fpu_fpregs_owner_ctx);
406
407	/*
408	* The in-register FPU state for an FPU context on a CPU is assumed to be
409	* valid if the fpu->last_cpu matches the CPU, and the fpu_fpregs_owner_ctx
410	* matches the FPU.
411	*
412	* If the FPU register state is valid, the kernel can skip restoring the
413	* FPU state from memory.
414	*
415	* Any code that clobbers the FPU registers or updates the in-memory
416	* FPU state for a task MUST let the rest of the kernel know that the
417	* FPU registers are no longer valid for this task.
418	*
419	* Either one of these invalidation functions is enough. Invalidate
420	* a resource you control: CPU if using the CPU for something else
421	* (with preemption disabled), FPU for the current task, or a task that
422	* is prevented from running by the current task.
423	*/
424	static inline void __cpu_invalidate_fpregs_state(void)
425	{
426	__this_cpu_write(fpu_fpregs_owner_ctx, NULL);
427	}
428
429	static inline void __fpu_invalidate_fpregs_state(struct fpu *fpu)
430	{
431	fpu->last_cpu = -1;
432	}
433
434	static inline int fpregs_state_valid(struct fpu *fpu, unsigned int cpu)
435	{
436	return fpu == this_cpu_read(fpu_fpregs_owner_ctx) && cpu == fpu->last_cpu;
437	}
438
439	/*
440	* These generally need preemption protection to work,
441	* do try to avoid using these on their own:
442	*/
443	static inline void fpregs_deactivate(struct fpu *fpu)
444	{
445	this_cpu_write(fpu_fpregs_owner_ctx, NULL);
446	trace_x86_fpu_regs_deactivated(fpu);
447	}
448
449	static inline void fpregs_activate(struct fpu *fpu)
450	{
451	this_cpu_write(fpu_fpregs_owner_ctx, fpu);
452	trace_x86_fpu_regs_activated(fpu);
453	}
454
455	/* Internal helper for switch_fpu_return() and signal frame setup */
456	static inline void fpregs_restore_userregs(void)
457	{
458	struct fpu *fpu = &current->thread.fpu;
459	int cpu = smp_processor_id();
460
461	if (WARN_ON_ONCE(current->flags & PF_KTHREAD))
462	return;
463
464	if (!fpregs_state_valid(fpu, cpu)) {
465	u64 mask;
466
467	/*
468	* This restores _all_ xstate which has not been
469	* established yet.
470	*
471	* If PKRU is enabled, then the PKRU value is already
472	* correct because it was either set in switch_to() or in
473	* flush_thread(). So it is excluded because it might be
474	* not up to date in current->thread.fpu.xsave state.
475	*/
476	mask = xfeatures_mask_restore_user() \|
477	xfeatures_mask_supervisor();
478	__restore_fpregs_from_fpstate(&fpu->state, mask);
479
480	fpregs_activate(fpu);
481	fpu->last_cpu = cpu;
482	}
483	clear_thread_flag(TIF_NEED_FPU_LOAD);
484	}
485
486	/*
487	* FPU state switching for scheduling.
488	*
489	* This is a two-stage process:
490	*
491	* - switch_fpu_prepare() saves the old state.
492	* This is done within the context of the old process.
493	*
494	* - switch_fpu_finish() sets TIF_NEED_FPU_LOAD; the floating point state
495	* will get loaded on return to userspace, or when the kernel needs it.
496	*
497	* If TIF_NEED_FPU_LOAD is cleared then the CPU's FPU registers
498	* are saved in the current thread's FPU register state.
499	*
500	* If TIF_NEED_FPU_LOAD is set then CPU's FPU registers may not
501	* hold current()'s FPU registers. It is required to load the
502	* registers before returning to userland or using the content
503	* otherwise.
504	*
505	* The FPU context is only stored/restored for a user task and
506	* PF_KTHREAD is used to distinguish between kernel and user threads.
507	*/
508	static inline void switch_fpu_prepare(struct fpu *old_fpu, int cpu)
509	{
510	if (static_cpu_has(X86_FEATURE_FPU) && !(current->flags & PF_KTHREAD)) {
511	save_fpregs_to_fpstate(old_fpu);
512	/*
513	* The save operation preserved register state, so the
514	* fpu_fpregs_owner_ctx is still @old_fpu. Store the
515	* current CPU number in @old_fpu, so the next return
516	* to user space can avoid the FPU register restore
517	* when is returns on the same CPU and still owns the
518	* context.
519	*/
520	old_fpu->last_cpu = cpu;
521
522	trace_x86_fpu_regs_deactivated(old_fpu);
523	}
524	}
525
526	/*
527	* Misc helper functions:
528	*/
529
530	/*
531	* Delay loading of the complete FPU state until the return to userland.
532	* PKRU is handled separately.
533	*/
534	static inline void switch_fpu_finish(struct fpu *new_fpu)
535	{
536	if (cpu_feature_enabled(X86_FEATURE_FPU))
537	set_thread_flag(TIF_NEED_FPU_LOAD);
538	}
539
540	#endif /* _ASM_X86_FPU_INTERNAL_H */

arch/x86/include/asm/fpu/sched.h+68−0

@@ -0,0 +1,68 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#ifndef _ASM_X86_FPU_SCHED_H
	3	#define _ASM_X86_FPU_SCHED_H
	4
	5	#include <linux/sched.h>
	6
	7	#include <asm/cpufeature.h>
	8	#include <asm/fpu/types.h>
	9
	10	#include <asm/trace/fpu.h>
	11
	12	extern void save_fpregs_to_fpstate(struct fpu *fpu);
	13	extern void fpu__drop(struct fpu *fpu);
	14	extern int fpu_clone(struct task_struct *dst, unsigned long clone_flags);
	15	extern void fpu_flush_thread(void);
	16
	17	/*
	18	* FPU state switching for scheduling.
	19	*
	20	* This is a two-stage process:
	21	*
	22	* - switch_fpu_prepare() saves the old state.
	23	* This is done within the context of the old process.
	24	*
	25	* - switch_fpu_finish() sets TIF_NEED_FPU_LOAD; the floating point state
	26	* will get loaded on return to userspace, or when the kernel needs it.
	27	*
	28	* If TIF_NEED_FPU_LOAD is cleared then the CPU's FPU registers
	29	* are saved in the current thread's FPU register state.
	30	*
	31	* If TIF_NEED_FPU_LOAD is set then CPU's FPU registers may not
	32	* hold current()'s FPU registers. It is required to load the
	33	* registers before returning to userland or using the content
	34	* otherwise.
	35	*
	36	* The FPU context is only stored/restored for a user task and
	37	* PF_KTHREAD is used to distinguish between kernel and user threads.
	38	*/
	39	static inline void switch_fpu_prepare(struct fpu *old_fpu, int cpu)
	40	{
	41	if (cpu_feature_enabled(X86_FEATURE_FPU) &&
	42	!(current->flags & PF_KTHREAD)) {
	43	save_fpregs_to_fpstate(old_fpu);
	44	/*
	45	* The save operation preserved register state, so the
	46	* fpu_fpregs_owner_ctx is still @old_fpu. Store the
	47	* current CPU number in @old_fpu, so the next return
	48	* to user space can avoid the FPU register restore
	49	* when is returns on the same CPU and still owns the
	50	* context.
	51	*/
	52	old_fpu->last_cpu = cpu;
	53
	54	trace_x86_fpu_regs_deactivated(old_fpu);
	55	}
	56	}
	57
	58	/*
	59	* Delay loading of the complete FPU state until the return to userland.
	60	* PKRU is handled separately.
	61	*/
	62	static inline void switch_fpu_finish(void)
	63	{
	64	if (cpu_feature_enabled(X86_FEATURE_FPU))
	65	set_thread_flag(TIF_NEED_FPU_LOAD);
	66	}
	67
	68	#endif /* _ASM_X86_FPU_SCHED_H */

arch/x86/include/asm/fpu/signal.h+12−1

 #ifndef _ASM_X86_FPU_SIGNAL_H
 #define _ASM_X86_FPU_SIGNAL_H
+#include <linux/compat.h>
+#include <linux/user.h>
+#include <asm/fpu/types.h>
 #ifdef CONFIG_X86_64
 # include <uapi/asm/sigcontext.h>
 # include <asm/user32.h>
 unsigned long fpu__get_fpstate_size(void);
-extern void fpu__init_prepare_fx_sw_frame(void);
+extern bool copy_fpstate_to_sigframe(void __user *buf, void __user *fp, int size);
+extern void fpu__clear_user_states(struct fpu *fpu);
+extern bool fpu__restore_sig(void __user *buf, int ia32_frame);
+extern void restore_fpregs_from_fpstate(struct fpstate *fpstate, u64 mask);
+extern bool copy_fpstate_to_sigframe(void __user *buf, void __user *fp, int size);
 #endif /* _ASM_X86_FPU_SIGNAL_H */

arch/x86/include/asm/fpu/types.h+206−8

@@ -120,6 +120,9 @@ enum xfeature { · 119 unmodified lines
120	XFEATURE_RSRVD_COMP_13,	120	XFEATURE_RSRVD_COMP_13,
121	XFEATURE_RSRVD_COMP_14,	121	XFEATURE_RSRVD_COMP_14,
122	XFEATURE_LBR,	122	XFEATURE_LBR,
		123	XFEATURE_RSRVD_COMP_16,
		124	XFEATURE_XTILE_CFG,
		125	XFEATURE_XTILE_DATA,
123		126
124	XFEATURE_MAX,	127	XFEATURE_MAX,
125	};	128	};
@@ -136,12 +139,21 @@ enum xfeature { · 10 unmodified lines
136	#define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU)	139	#define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU)
137	#define XFEATURE_MASK_PASID (1 << XFEATURE_PASID)	140	#define XFEATURE_MASK_PASID (1 << XFEATURE_PASID)
138	#define XFEATURE_MASK_LBR (1 << XFEATURE_LBR)	141	#define XFEATURE_MASK_LBR (1 << XFEATURE_LBR)
		142	#define XFEATURE_MASK_XTILE_CFG (1 << XFEATURE_XTILE_CFG)
		143	#define XFEATURE_MASK_XTILE_DATA (1 << XFEATURE_XTILE_DATA)
139		144
140	#define XFEATURE_MASK_FPSSE (XFEATURE_MASK_FP \| XFEATURE_MASK_SSE)	145	#define XFEATURE_MASK_FPSSE (XFEATURE_MASK_FP \| XFEATURE_MASK_SSE)
141	#define XFEATURE_MASK_AVX512 (XFEATURE_MASK_OPMASK \	146	#define XFEATURE_MASK_AVX512 (XFEATURE_MASK_OPMASK \
142	\| XFEATURE_MASK_ZMM_Hi256 \	147	\| XFEATURE_MASK_ZMM_Hi256 \
143	\| XFEATURE_MASK_Hi16_ZMM)	148	\| XFEATURE_MASK_Hi16_ZMM)
144		149
		150	#ifdef CONFIG_X86_64
		151	# define XFEATURE_MASK_XTILE (XFEATURE_MASK_XTILE_DATA \
		152	\| XFEATURE_MASK_XTILE_CFG)
		153	#else
		154	# define XFEATURE_MASK_XTILE (0)
		155	#endif
		156
145	#define FIRST_EXTENDED_XFEATURE XFEATURE_YMM	157	#define FIRST_EXTENDED_XFEATURE XFEATURE_YMM
146		158
147	struct reg_128_bit {	159	struct reg_128_bit {
@@ -153,6 +165,9 @@ struct reg_256_bit { · 5 unmodified lines
153	struct reg_512_bit {	165	struct reg_512_bit {
154	u8 regbytes[512/8];	166	u8 regbytes[512/8];
155	};	167	};
		168	struct reg_1024_byte {
		169	u8 regbytes[1024];
		170	};
156		171
157	/*	172	/*
158	* State component 2:	173	* State component 2:
@@ -255,6 +270,23 @@ struct arch_lbr_state { · 96 unmodified lines
255	u64 ler_to;	270	u64 ler_to;
256	u64 ler_info;	271	u64 ler_info;
257	struct lbr_entry entries[];	272	struct lbr_entry entries[];
		273	};
		274
		275	/*
		276	* State component 17: 64-byte tile configuration register.
		277	*/
		278	struct xtile_cfg {
		279	u64 tcfg[8];
		280	} __packed;
		281
		282	/*
		283	* State component 18: 1KB tile data register.
		284	* Each register represents 16 64-byte rows of the matrix
		285	* data. But the number of registers depends on the actual
		286	* implementation.
		287	*/
		288	struct xtile_data {
		289	struct reg_1024_byte tmm;
258	} __packed;	290	} __packed;
259		291
260	/*	292	/*
@@ -309,6 +341,91 @@ union fpregs_state { · 48 unmodified lines
309	u8 __padding[PAGE_SIZE];	341	u8 __padding[PAGE_SIZE];
310	};	342	};
311		343
		344	struct fpstate {
		345	/* @kernel_size: The size of the kernel register image */
		346	unsigned int size;
		347
		348	/* @user_size: The size in non-compacted UABI format */
		349	unsigned int user_size;
		350
		351	/* @xfeatures: xfeatures for which the storage is sized */
		352	u64 xfeatures;
		353
		354	/* @user_xfeatures: xfeatures valid in UABI buffers */
		355	u64 user_xfeatures;
		356
		357	/* @xfd: xfeatures disabled to trap userspace use. */
		358	u64 xfd;
		359
		360	/* @is_valloc: Indicator for dynamically allocated state */
		361	unsigned int is_valloc : 1;
		362
		363	/* @is_guest: Indicator for guest state (KVM) */
		364	unsigned int is_guest : 1;
		365
		366	/*
		367	* @is_confidential: Indicator for KVM confidential mode.
		368	* The FPU registers are restored by the
		369	* vmentry firmware from encrypted guest
		370	* memory. On vmexit the FPU registers are
		371	* saved by firmware to encrypted guest memory
		372	* and the registers are scrubbed before
		373	* returning to the host. So there is no
		374	* content which is worth saving and restoring.
		375	* The fpstate has to be there so that
		376	* preemption and softirq FPU usage works
		377	* without special casing.
		378	*/
		379	unsigned int is_confidential : 1;
		380
		381	/* @in_use: State is in use */
		382	unsigned int in_use : 1;
		383
		384	/* @regs: The register state union for all supported formats */
		385	union fpregs_state regs;
		386
		387	/* @regs is dynamically sized! Don't add anything after @regs! */
		388	} __aligned(64);
		389
		390	struct fpu_state_perm {
		391	/*
		392	* @__state_perm:
		393	*
		394	* This bitmap indicates the permission for state components, which
		395	* are available to a thread group. The permission prctl() sets the
		396	* enabled state bits in thread_group_leader()->thread.fpu.
		397	*
		398	* All run time operations use the per thread information in the
		399	* currently active fpu.fpstate which contains the xfeature masks
		400	* and sizes for kernel and user space.
		401	*
		402	* This master permission field is only to be used when
		403	* task.fpu.fpstate based checks fail to validate whether the task
		404	* is allowed to expand it's xfeatures set which requires to
		405	* allocate a larger sized fpstate buffer.
		406	*
		407	* Do not access this field directly. Use the provided helper
		408	* function. Unlocked access is possible for quick checks.
		409	*/
		410	u64 __state_perm;
		411
		412	/*
		413	* @__state_size:
		414	*
		415	* The size required for @__state_perm. Only valid to access
		416	* with sighand locked.
		417	*/
		418	unsigned int __state_size;
		419
		420	/*
		421	* @__user_state_size:
		422	*
		423	* The size required for @__state_perm user part. Only valid to
		424	* access with sighand locked.
		425	*/
		426	unsigned int __user_state_size;
		427	};
		428
312	/*	429	/*
313	* Highest level per task FPU state data structure that	430	* Highest level per task FPU state data structure that
314	* contains the FPU register state plus various FPU	431	* contains the FPU register state plus various FPU
@@ -337,19 +454,100 @@ struct fpu { · 22 unmodified lines
337	unsigned long avx512_timestamp;	454	unsigned long avx512_timestamp;
338		455
339	/*	456	/*
340	* @state:	457	* @fpstate:
		458	*
		459	* Pointer to the active struct fpstate. Initialized to
		460	* point at @__fpstate below.
		461	*/
		462	struct fpstate *fpstate;
		463
		464	/*
		465	* @__task_fpstate:
		466	*
		467	* Pointer to an inactive struct fpstate. Initialized to NULL. Is
		468	* used only for KVM support to swap out the regular task fpstate.
		469	*/
		470	struct fpstate *__task_fpstate;
		471
		472	/*
		473	* @perm:
		474	*
		475	* Permission related information
		476	*/
		477	struct fpu_state_perm perm;
		478
		479	/*
		480	* @__fpstate:
341	*	481	*
342	* In-memory copy of all FPU registers that we save/restore	482	* Initial in-memory storage for FPU registers which are saved in
343	* over context switches. If the task is using the FPU then	483	* context switch and when the kernel uses the FPU. The registers
344	* the registers in the FPU are more recent than this state	484	* are restored from this storage on return to user space if they
345	* copy. If the task context-switches away then they get	485	* are not longer containing the tasks FPU register state.
346	* saved here and represent the FPU state.
347	*/	486	*/
348	union fpregs_state state;	487	struct fpstate __fpstate;
349	/*	488	/*
350	* WARNING: 'state' is dynamically-sized. Do not put	489	* WARNING: '__fpstate' is dynamically-sized. Do not put
351	* anything after it here.	490	* anything after it here.
352	*/	491	*/
353	};	492	};
354		493
		494	/*
		495	* Guest pseudo FPU container
		496	*/
		497	struct fpu_guest {
		498	/*
		499	* @fpstate: Pointer to the allocated guest fpstate
		500	*/
		501	struct fpstate *fpstate;
		502	};
		503
		504	/*
		505	* FPU state configuration data. Initialized at boot time. Read only after init.
		506	*/
		507	struct fpu_state_config {
		508	/*
		509	* @max_size:
		510	*
		511	* The maximum size of the register state buffer. Includes all
		512	* supported features except independent managed features.
		513	*/
		514	unsigned int max_size;
		515
		516	/*
		517	* @default_size:
		518	*
		519	* The default size of the register state buffer. Includes all
		520	* supported features except independent managed features and
		521	* features which have to be requested by user space before usage.
		522	*/
		523	unsigned int default_size;
		524
		525	/*
		526	* @max_features:
		527	*
		528	* The maximum supported features bitmap. Does not include
		529	* independent managed features.
		530	*/
		531	u64 max_features;
		532
		533	/*
		534	* @default_features:
		535	*
		536	* The default supported features bitmap. Does not include
		537	* independent managed features and features which have to
		538	* be requested by user space before usage.
		539	*/
		540	u64 default_features;
		541	/*
		542	* @legacy_features:
		543	*
		544	* Features which can be reported back to user space
		545	* even without XSAVE support, i.e. legacy features FP + SSE
		546	*/
		547	u64 legacy_features;
		548	};
		549
		550	/* FPU state configuration information */
		551	extern struct fpu_state_config fpu_kernel_cfg, fpu_user_cfg;
		552
355	#endif /* _ASM_X86_FPU_H */	553	#endif /* _ASM_X86_FPU_H */

arch/x86/include/asm/fpu/xcr.h+0−11

@@ -2,17 +2,6 @@ · 1 unmodified line
2	#ifndef _ASM_X86_FPU_XCR_H	2	#ifndef _ASM_X86_FPU_XCR_H
3	#define _ASM_X86_FPU_XCR_H	3	#define _ASM_X86_FPU_XCR_H
4		4
5	/*
6	* MXCSR and XCR definitions:
7	*/
8
9	static inline void ldmxcsr(u32 mxcsr)
10	{
11	asm volatile("ldmxcsr %0" :: "m" (mxcsr));
12	}
13
14	extern unsigned int mxcsr_feature_mask;
15
16	#define XCR_XFEATURE_ENABLED_MASK 0x00000000	5	#define XCR_XFEATURE_ENABLED_MASK 0x00000000
17		6
18	static inline u64 xgetbv(u32 index)	7	static inline u64 xgetbv(u32 index)

arch/x86/include/asm/fpu/xstate.h+30−60

@@ -14,6 +14,8 @@ · 13 unmodified lines
14		14
15	#define XSTATE_CPUID 0x0000000d	15	#define XSTATE_CPUID 0x0000000d
16		16
		17	#define TILE_CPUID 0x0000001d
		18
17	#define FXSAVE_SIZE 512	19	#define FXSAVE_SIZE 512
18		20
19	#define XSAVE_HDR_SIZE 64	21	#define XSAVE_HDR_SIZE 64
@@ -33,7 +35,8 @@ · 13 unmodified lines
33	XFEATURE_MASK_Hi16_ZMM \| \	35	XFEATURE_MASK_Hi16_ZMM \| \
34	XFEATURE_MASK_PKRU \| \	36	XFEATURE_MASK_PKRU \| \
35	XFEATURE_MASK_BNDREGS \| \	37	XFEATURE_MASK_BNDREGS \| \
36	XFEATURE_MASK_BNDCSR)	38	XFEATURE_MASK_BNDCSR \| \
		39	XFEATURE_MASK_XTILE)
37		40
38	/*	41	/*
39	* Features which are restored when returning to user space.	42	* Features which are restored when returning to user space.
@@ -43,6 +46,9 @@ · 3 unmodified lines
43	#define XFEATURE_MASK_USER_RESTORE \	46	#define XFEATURE_MASK_USER_RESTORE \
44	(XFEATURE_MASK_USER_SUPPORTED & ~XFEATURE_MASK_PKRU)	47	(XFEATURE_MASK_USER_SUPPORTED & ~XFEATURE_MASK_PKRU)
45		48
		49	/* Features which are dynamically enabled for a process on request */
		50	#define XFEATURE_MASK_USER_DYNAMIC XFEATURE_MASK_XTILE_DATA
		51
46	/* All currently supported supervisor features */	52	/* All currently supported supervisor features */
47	#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID)	53	#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID)
48		54
@@ -78,78 +84,42 @@ · 29 unmodified lines
78	XFEATURE_MASK_INDEPENDENT \| \	84	XFEATURE_MASK_INDEPENDENT \| \
79	XFEATURE_MASK_SUPERVISOR_UNSUPPORTED)	85	XFEATURE_MASK_SUPERVISOR_UNSUPPORTED)
80		86
81	#ifdef CONFIG_X86_64
82	#define REX_PREFIX "0x48, "
83	#else
84	#define REX_PREFIX
85	#endif
86
87	extern u64 xfeatures_mask_all;
88
89	static inline u64 xfeatures_mask_supervisor(void)
90	{
91	return xfeatures_mask_all & XFEATURE_MASK_SUPERVISOR_SUPPORTED;
92	}
93
94	/*	87	/*
95	* The xfeatures which are enabled in XCR0 and expected to be in ptrace	88	* The feature mask required to restore FPU state:
96	* buffers and signal frames.	89	* - All user states which are not eagerly switched in switch_to()/exec()
		90	* - The suporvisor states
97	*/	91	*/
98	static inline u64 xfeatures_mask_uabi(void)	92	#define XFEATURE_MASK_FPSTATE (XFEATURE_MASK_USER_RESTORE \| \
99	{	93	XFEATURE_MASK_SUPERVISOR_SUPPORTED)
100	return xfeatures_mask_all & XFEATURE_MASK_USER_SUPPORTED;
101	}
102
103	/*
104	* The xfeatures which are restored by the kernel when returning to user
105	* mode. This is not necessarily the same as xfeatures_mask_uabi() as the
106	* kernel does not manage all XCR0 enabled features via xsave/xrstor as
107	* some of them have to be switched eagerly on context switch and exec().
108	*/
109	static inline u64 xfeatures_mask_restore_user(void)
110	{
111	return xfeatures_mask_all & XFEATURE_MASK_USER_RESTORE;
112	}
113
114	/*
115	* Like xfeatures_mask_restore_user() but additionally restors the
116	* supported supervisor states.
117	*/
118	static inline u64 xfeatures_mask_fpstate(void)
119	{
120	return xfeatures_mask_all & \
121	(XFEATURE_MASK_USER_RESTORE \| XFEATURE_MASK_SUPERVISOR_SUPPORTED);
122	}
123
124	static inline u64 xfeatures_mask_independent(void)
125	{
126	if (!boot_cpu_has(X86_FEATURE_ARCH_LBR))
127	return XFEATURE_MASK_INDEPENDENT & ~XFEATURE_MASK_LBR;
128
129	return XFEATURE_MASK_INDEPENDENT;
130	}
131		94
132	extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS];	95	extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS];
133		96
134	extern void __init update_regset_xstate_info(unsigned int size,	97	extern void __init update_regset_xstate_info(unsigned int size,
135	u64 xstate_mask);	98	u64 xstate_mask);
136		99
137	void get_xsave_addr(struct xregs_state xsave, int xfeature_nr);
138	int xfeature_size(int xfeature_nr);	100	int xfeature_size(int xfeature_nr);
139	int copy_uabi_from_kernel_to_xstate(struct xregs_state xsave, const void kbuf);
140	int copy_sigframe_from_user_to_xstate(struct xregs_state xsave, const void __user ubuf);
141		101
142	void xsaves(struct xregs_state *xsave, u64 mask);	102	void xsaves(struct xregs_state *xsave, u64 mask);
143	void xrstors(struct xregs_state *xsave, u64 mask);	103	void xrstors(struct xregs_state *xsave, u64 mask);
144		104
145	enum xstate_copy_mode {	105	int xfd_enable_feature(u64 xfd_err);
146	XSTATE_COPY_FP,	106
147	XSTATE_COPY_FX,	107	#ifdef CONFIG_X86_64
148	XSTATE_COPY_XSAVE,	108	DECLARE_STATIC_KEY_FALSE(__fpu_state_size_dynamic);
149	};	109	#endif
		110
		111	#ifdef CONFIG_X86_64
		112	DECLARE_STATIC_KEY_FALSE(__fpu_state_size_dynamic);
150		113
151	struct membuf;	114	static __always_inline __pure bool fpu_state_size_dynamic(void)
152	void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk,	115	{
153	enum xstate_copy_mode mode);	116	return static_branch_unlikely(&__fpu_state_size_dynamic);
		117	}
		118	#else
		119	static __always_inline __pure bool fpu_state_size_dynamic(void)
		120	{
		121	return false;
		122	}
		123	#endif
154		124
155	#endif	125	#endif

arch/x86/include/asm/kvm_host.h+2−5

 	 *
 	 * Note that while the PKRU state lives inside the fpu registers,
 	 * it is switched out separately at VMENTER and VMEXIT time. The
-	 * "guest_fpu" state here contains the guest FPU context, with the
+	 * "guest_fpstate" state here contains the guest FPU context, with the
 	 * host PRKU bits.
 	 */
-	struct fpu *user_fpu;
+	struct fpu_guest guest_fpu;
-	struct fpu *guest_fpu;
 	u64 xcr0;
 	u64 guest_supported_xcr0;
 int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index,
 		    int reason, bool has_error_code, u32 error_code);
-void kvm_free_guest_fpu(struct kvm_vcpu *vcpu);
 void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0);
 void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned long cr4);
 int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0);

arch/x86/include/asm/msr-index.h+2−0

@@ -625,6 +625,8 @@ · 624 unmodified lines
625		625
626	#define MSR_IA32_BNDCFGS_RSVD 0x00000ffc	626	#define MSR_IA32_BNDCFGS_RSVD 0x00000ffc
627		627
		628	#define MSR_IA32_XFD 0x000001c4
		629	#define MSR_IA32_XFD_ERR 0x000001c5
628	#define MSR_IA32_XSS 0x00000da0	630	#define MSR_IA32_XSS 0x00000da0
629		631
630	#define MSR_IA32_APICBASE 0x0000001b	632	#define MSR_IA32_APICBASE 0x0000001b

arch/x86/include/asm/msr.h+2−2

@@ -92,7 +92,7 @@ static __always_inline unsigned long long __rdmsr(unsigned int msr) · 91 unmodified lines
92		92
93	asm volatile("1: rdmsr\n"	93	asm volatile("1: rdmsr\n"
94	"2:\n"	94	"2:\n"
95	_ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_rdmsr_unsafe)	95	_ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_RDMSR)
96	: EAX_EDX_RET(val, low, high) : "c" (msr));	96	: EAX_EDX_RET(val, low, high) : "c" (msr));
97		97
98	return EAX_EDX_VAL(val, low, high);	98	return EAX_EDX_VAL(val, low, high);
@@ -102,7 +102,7 @@ static __always_inline void __wrmsr(unsigned int msr, u32 low, u32 high) · 3 unmodified lines
102	{	102	{
103	asm volatile("1: wrmsr\n"	103	asm volatile("1: wrmsr\n"
104	"2:\n"	104	"2:\n"
105	_ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_wrmsr_unsafe)	105	_ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_WRMSR)
106	: : "c" (msr), "a"(low), "d" (high) : "memory");	106	: : "c" (msr), "a"(low), "d" (high) : "memory");
107	}	107	}
108		108

arch/x86/include/asm/pkru.h+1−1

@@ -2,7 +2,7 @@ · 1 unmodified line
2	#ifndef _ASM_X86_PKRU_H	2	#ifndef _ASM_X86_PKRU_H
3	#define _ASM_X86_PKRU_H	3	#define _ASM_X86_PKRU_H
4		4
5	#include <asm/fpu/xstate.h>	5	#include <asm/cpufeature.h>
6		6
7	#define PKRU_AD_BIT 0x1	7	#define PKRU_AD_BIT 0x1
8	#define PKRU_WD_BIT 0x2	8	#define PKRU_WD_BIT 0x2

arch/x86/include/asm/processor.h+3−6

 DECLARE_PER_CPU(struct irq_stack *, softirq_stack_ptr);
 #endif	/* !X86_64 */
-extern unsigned int fpu_kernel_xstate_size;
-extern unsigned int fpu_user_xstate_size;
 struct perf_event;
 struct thread_struct {
 	 */
 };
-/* Whitelist the FPU state from the task_struct for hardened usercopy. */
+extern void fpu_thread_struct_whitelist(unsigned long *offset, unsigned long *size);
 static inline void arch_thread_struct_whitelist(unsigned long *offset,
 						unsigned long *size)
 {
-	*offset = offsetof(struct thread_struct, fpu.state);
+	fpu_thread_struct_whitelist(offset, size);
-	*size = fpu_kernel_xstate_size;
 }
 static inline void

arch/x86/include/asm/proto.h+1−1

@@ -40,6 +40,6 @@ void x86_report_nx(void); · 39 unmodified lines
40	extern int reboot_force;	40	extern int reboot_force;
41		41
42	long do_arch_prctl_common(struct task_struct *task, int option,	42	long do_arch_prctl_common(struct task_struct *task, int option,
43	unsigned long cpuid_enabled);	43	unsigned long arg2);
44		44
45	#endif /* _ASM_X86_PROTO_H */	45	#endif /* _ASM_X86_PROTO_H */

arch/x86/include/asm/segment.h+1−1

@@ -339,7 +339,7 @@ static inline void __loadsegment_fs(unsigned short value) · 338 unmodified lines
339	"1: movw %0, %%fs \n"	339	"1: movw %0, %%fs \n"
340	"2: \n"	340	"2: \n"
341		341
342	_ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_clear_fs)	342	_ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_CLEAR_FS)
343		343
344	: : "rm" (value) : "memory");	344	: : "rm" (value) : "memory");
345	}	345	}

arch/x86/include/asm/trace/fpu.h+2−2

@@ -22,8 +22,8 @@ DECLARE_EVENT_CLASS(x86_fpu, · 21 unmodified lines
22	__entry->fpu = fpu;	22	__entry->fpu = fpu;
23	__entry->load_fpu = test_thread_flag(TIF_NEED_FPU_LOAD);	23	__entry->load_fpu = test_thread_flag(TIF_NEED_FPU_LOAD);
24	if (boot_cpu_has(X86_FEATURE_OSXSAVE)) {	24	if (boot_cpu_has(X86_FEATURE_OSXSAVE)) {
25	__entry->xfeatures = fpu->state.xsave.header.xfeatures;	25	__entry->xfeatures = fpu->fpstate->regs.xsave.header.xfeatures;
26	__entry->xcomp_bv = fpu->state.xsave.header.xcomp_bv;	26	__entry->xcomp_bv = fpu->fpstate->regs.xsave.header.xcomp_bv;
27	}	27	}
28	),	28	),
29	TP_printk("x86/fpu: %p load: %d xfeatures: %llx xcomp_bv: %llx",	29	TP_printk("x86/fpu: %p load: %d xfeatures: %llx xcomp_bv: %llx",

arch/x86/include/uapi/asm/prctl.h+4−0

 #define ARCH_GET_CPUID		0x1011
 #define ARCH_SET_CPUID		0x1012
+#define ARCH_GET_XCOMP_SUPP	0x1021
+#define ARCH_GET_XCOMP_PERM	0x1022
+#define ARCH_REQ_XCOMP_PERM	0x1023
 #define ARCH_MAP_VDSO_X32	0x2001
 #define ARCH_MAP_VDSO_32	0x2002
 #define ARCH_MAP_VDSO_64	0x2003

arch/x86/kernel/cpu/bugs.c+1−1

@@ -22,7 +22,7 @@ · 21 unmodified lines
22	#include <asm/bugs.h>	22	#include <asm/bugs.h>
23	#include <asm/processor.h>	23	#include <asm/processor.h>
24	#include <asm/processor-flags.h>	24	#include <asm/processor-flags.h>
25	#include <asm/fpu/internal.h>	25	#include <asm/fpu/api.h>
26	#include <asm/msr.h>	26	#include <asm/msr.h>
27	#include <asm/vmx.h>	27	#include <asm/vmx.h>
28	#include <asm/paravirt.h>	28	#include <asm/paravirt.h>

arch/x86/kernel/cpu/common.c+1−1

@@ -42,7 +42,7 @@ · 41 unmodified lines
42	#include <asm/setup.h>	42	#include <asm/setup.h>
43	#include <asm/apic.h>	43	#include <asm/apic.h>
44	#include <asm/desc.h>	44	#include <asm/desc.h>
45	#include <asm/fpu/internal.h>	45	#include <asm/fpu/api.h>
46	#include <asm/mtrr.h>	46	#include <asm/mtrr.h>
47	#include <asm/hwcap2.h>	47	#include <asm/hwcap2.h>
48	#include <linux/numa.h>	48	#include <linux/numa.h>

arch/x86/kernel/cpu/cpuid-deps.c+2−0

@@ -75,6 +75,8 @@ static const struct cpuid_dep cpuid_deps[] = { · 74 unmodified lines
75	{ X86_FEATURE_SGX_LC, X86_FEATURE_SGX },	75	{ X86_FEATURE_SGX_LC, X86_FEATURE_SGX },
76	{ X86_FEATURE_SGX1, X86_FEATURE_SGX },	76	{ X86_FEATURE_SGX1, X86_FEATURE_SGX },
77	{ X86_FEATURE_SGX2, X86_FEATURE_SGX1 },	77	{ X86_FEATURE_SGX2, X86_FEATURE_SGX1 },
		78	{ X86_FEATURE_XFD, X86_FEATURE_XSAVES },
		79	{ X86_FEATURE_AMX_TILE, X86_FEATURE_XFD },
78	{}	80	{}
79	};	81	};
80		82

arch/x86/kernel/cpu/mce/core.c+11−29

 	return -1;
 }
-__visible bool ex_handler_rdmsr_fault(const struct exception_table_entry *fixup,
+void ex_handler_msr_mce(struct pt_regs *regs, bool wrmsr)
-				      struct pt_regs *regs, int trapnr,
-				      unsigned long error_code,
-				      unsigned long fault_addr)
 {
-	pr_emerg("MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n",
+	if (wrmsr) {
-		 (unsigned int)regs->cx, regs->ip, (void *)regs->ip);
+		pr_emerg("MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n",
+			 (unsigned int)regs->cx, (unsigned int)regs->dx, (unsigned int)regs->ax,
+			 regs->ip, (void *)regs->ip);
+	} else {
+		pr_emerg("MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n",
+			 (unsigned int)regs->cx, regs->ip, (void *)regs->ip);
+	}
 	show_stack_regs(regs);
 	while (true)
 		cpu_relax();
-	return true;
 }
 /* MSR access wrappers used for error injection */
 	 */
 	asm volatile("1: rdmsr\n"
 		     "2:\n"
-		     _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_rdmsr_fault)
+		     _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_RDMSR_IN_MCE)
 		     : EAX_EDX_RET(val, low, high) : "c" (msr));
 	return EAX_EDX_VAL(val, low, high);
 }
-__visible bool ex_handler_wrmsr_fault(const struct exception_table_entry *fixup,
-				      struct pt_regs *regs, int trapnr,
-				      unsigned long error_code,
-				      unsigned long fault_addr)
-{
-	pr_emerg("MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n",
-		 (unsigned int)regs->cx, (unsigned int)regs->dx, (unsigned int)regs->ax,
-		  regs->ip, (void *)regs->ip);
-	show_stack_regs(regs);
-	panic("MCA architectural violation!\n");
-	while (true)
-		cpu_relax();
-	return true;
-}
 static noinstr void mce_wrmsrl(u32 msr, u64 v)
 {
 	u32 low, high;
 	/* See comment in mce_rdmsrl() */
 	asm volatile("1: wrmsr\n"
 		     "2:\n"
-		     _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_wrmsr_fault)
+		     _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_WRMSR_IN_MCE)
 		     : : "c" (msr), "a"(low), "d" (high) : "memory");
 }

arch/x86/kernel/cpu/mce/internal.h+2−12

 static inline void intel_init_cmci(void) { }
 static inline void intel_init_lmce(void) { }
 static inline void intel_clear_lmce(void) { }
-static inline bool intel_filter_mce(struct mce *m) { return false; };
+static inline bool intel_filter_mce(struct mce *m) { return false; }
 #endif
 void mce_timer_kick(unsigned long interval);
 #ifdef CONFIG_X86_MCE_AMD
 extern bool amd_filter_mce(struct mce *m);
 #else
-static inline bool amd_filter_mce(struct mce *m)			{ return false; };
+static inline bool amd_filter_mce(struct mce *m) { return false; }
 #endif
-__visible bool ex_handler_rdmsr_fault(const struct exception_table_entry *fixup,
-				      struct pt_regs *regs, int trapnr,
-				      unsigned long error_code,
-				      unsigned long fault_addr);
-__visible bool ex_handler_wrmsr_fault(const struct exception_table_entry *fixup,
-				      struct pt_regs *regs, int trapnr,
-				      unsigned long error_code,
-				      unsigned long fault_addr);
 #endif /* __X86_MCE_INTERNAL_H__ */

arch/x86/kernel/cpu/mce/severity.c+11−11

  */
 static int error_context(struct mce *m, struct pt_regs *regs)
 {
-	enum handler_type t;
 	if ((m->cs & 3) == 3)
 		return IN_USER;
 	if (!mc_recoverable(m->mcgstatus))
 		return IN_KERNEL;
-	t = ex_get_fault_handler_type(m->ip);
+	switch (ex_get_fixup_type(m->ip)) {
-	if (t == EX_HANDLER_FAULT) {
+	case EX_TYPE_UACCESS:
-		m->kflags |= MCE_IN_KERNEL_RECOV;
+	case EX_TYPE_COPY:
-		return IN_KERNEL_RECOV;
+		if (!regs || !is_copy_from_user(regs))
-	}
+			return IN_KERNEL;
-	if (t == EX_HANDLER_UACCESS && regs && is_copy_from_user(regs)) {
-		m->kflags |= MCE_IN_KERNEL_RECOV;
 		m->kflags |= MCE_IN_KERNEL_COPYIN;
+		fallthrough;
+	case EX_TYPE_FAULT_MCE_SAFE:
+	case EX_TYPE_DEFAULT_MCE_SAFE:
+		m->kflags |= MCE_IN_KERNEL_RECOV;
 		return IN_KERNEL_RECOV;
+	default:
+		return IN_KERNEL;
 	}
-	return IN_KERNEL;
 }
 static int mce_severity_amd_smca(struct mce *m, enum context err_ctx)

arch/x86/kernel/fpu/bugs.c+1−1

@@ -2,7 +2,7 @@ · 1 unmodified line
2	/*	2	/*
3	* x86 FPU bug checks:	3	* x86 FPU bug checks:
4	*/	4	*/
5	#include <asm/fpu/internal.h>	5	#include <asm/fpu/api.h>
6		6
7	/*	7	/*
8	* Boot time CPU/FPU FDIV bug detection code:	8	* Boot time CPU/FPU FDIV bug detection code:

arch/x86/kernel/fpu/context.h+83−0

@@ -0,0 +1,83 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#ifndef __X86_KERNEL_FPU_CONTEXT_H
	3	#define __X86_KERNEL_FPU_CONTEXT_H
	4
	5	#include <asm/fpu/xstate.h>
	6	#include <asm/trace/fpu.h>
	7
	8	/* Functions related to FPU context tracking */
	9
	10	/*
	11	* The in-register FPU state for an FPU context on a CPU is assumed to be
	12	* valid if the fpu->last_cpu matches the CPU, and the fpu_fpregs_owner_ctx
	13	* matches the FPU.
	14	*
	15	* If the FPU register state is valid, the kernel can skip restoring the
	16	* FPU state from memory.
	17	*
	18	* Any code that clobbers the FPU registers or updates the in-memory
	19	* FPU state for a task MUST let the rest of the kernel know that the
	20	* FPU registers are no longer valid for this task.
	21	*
	22	* Either one of these invalidation functions is enough. Invalidate
	23	* a resource you control: CPU if using the CPU for something else
	24	* (with preemption disabled), FPU for the current task, or a task that
	25	* is prevented from running by the current task.
	26	*/
	27	static inline void __cpu_invalidate_fpregs_state(void)
	28	{
	29	__this_cpu_write(fpu_fpregs_owner_ctx, NULL);
	30	}
	31
	32	static inline void __fpu_invalidate_fpregs_state(struct fpu *fpu)
	33	{
	34	fpu->last_cpu = -1;
	35	}
	36
	37	static inline int fpregs_state_valid(struct fpu *fpu, unsigned int cpu)
	38	{
	39	return fpu == this_cpu_read(fpu_fpregs_owner_ctx) && cpu == fpu->last_cpu;
	40	}
	41
	42	static inline void fpregs_deactivate(struct fpu *fpu)
	43	{
	44	__this_cpu_write(fpu_fpregs_owner_ctx, NULL);
	45	trace_x86_fpu_regs_deactivated(fpu);
	46	}
	47
	48	static inline void fpregs_activate(struct fpu *fpu)
	49	{
	50	__this_cpu_write(fpu_fpregs_owner_ctx, fpu);
	51	trace_x86_fpu_regs_activated(fpu);
	52	}
	53
	54	/* Internal helper for switch_fpu_return() and signal frame setup */
	55	static inline void fpregs_restore_userregs(void)
	56	{
	57	struct fpu *fpu = &current->thread.fpu;
	58	int cpu = smp_processor_id();
	59
	60	if (WARN_ON_ONCE(current->flags & PF_KTHREAD))
	61	return;
	62
	63	if (!fpregs_state_valid(fpu, cpu)) {
	64	/*
	65	* This restores _all_ xstate which has not been
	66	* established yet.
	67	*
	68	* If PKRU is enabled, then the PKRU value is already
	69	* correct because it was either set in switch_to() or in
	70	* flush_thread(). So it is excluded because it might be
	71	* not up to date in current->thread.fpu.xsave state.
	72	*
	73	* XFD state is handled in restore_fpregs_from_fpstate().
	74	*/
	75	restore_fpregs_from_fpstate(fpu->fpstate, XFEATURE_MASK_FPSTATE);
	76
	77	fpregs_activate(fpu);
	78	fpu->last_cpu = cpu;
	79	}
	80	clear_thread_flag(TIF_NEED_FPU_LOAD);
	81	}
	82
	83	#endif

arch/x86/kernel/fpu/core.c+315−77

@@ -6,8 +6,9 @@ · 5 unmodified lines
6	* General FPU state handling cleanups	6	* General FPU state handling cleanups
7	* Gareth Hughes <gareth@valinux.com>, May 2000	7	* Gareth Hughes <gareth@valinux.com>, May 2000
8	*/	8	*/
9	#include <asm/fpu/internal.h>	9	#include <asm/fpu/api.h>
10	#include <asm/fpu/regset.h>	10	#include <asm/fpu/regset.h>
		11	#include <asm/fpu/sched.h>
11	#include <asm/fpu/signal.h>	12	#include <asm/fpu/signal.h>
12	#include <asm/fpu/types.h>	13	#include <asm/fpu/types.h>
13	#include <asm/traps.h>	14	#include <asm/traps.h>
@@ -15,15 +16,30 @@ · 1 unmodified line
15		16
16	#include <linux/hardirq.h>	17	#include <linux/hardirq.h>
17	#include <linux/pkeys.h>	18	#include <linux/pkeys.h>
		19	#include <linux/vmalloc.h>
		20
		21	#include "context.h"
		22	#include "internal.h"
		23	#include "legacy.h"
		24	#include "xstate.h"
18		25
19	#define CREATE_TRACE_POINTS	26	#define CREATE_TRACE_POINTS
20	#include <asm/trace/fpu.h>	27	#include <asm/trace/fpu.h>
21		28
		29	#ifdef CONFIG_X86_64
		30	DEFINE_STATIC_KEY_FALSE(__fpu_state_size_dynamic);
		31	DEFINE_PER_CPU(u64, xfd_state);
		32	#endif
		33
		34	/* The FPU state configuration data for kernel and user space */
		35	struct fpu_state_config fpu_kernel_cfg __ro_after_init;
		36	struct fpu_state_config fpu_user_cfg __ro_after_init;
		37
22	/*	38	/*
23	* Represents the initial FPU state. It's mostly (but not completely) zeroes,	39	* Represents the initial FPU state. It's mostly (but not completely) zeroes,
24	* depending on the FPU hardware format:	40	* depending on the FPU hardware format:
25	*/	41	*/
26	union fpregs_state init_fpstate __ro_after_init;	42	struct fpstate init_fpstate __ro_after_init;
27		43
28	/*	44	/*
29	* Track whether the kernel is using the FPU state	45	* Track whether the kernel is using the FPU state
@@ -83,7 +99,7 @@ bool irq_fpu_usable(void) · 53 unmodified lines
83	EXPORT_SYMBOL(irq_fpu_usable);	99	EXPORT_SYMBOL(irq_fpu_usable);
84		100
85	/*	101	/*
86	* Save the FPU register state in fpu->state. The register state is	102	* Save the FPU register state in fpu->fpstate->regs. The register state is
87	* preserved.	103	* preserved.
88	*	104	*
89	* Must be called with fpregs_lock() held.	105	* Must be called with fpregs_lock() held.
@@ -99,19 +115,19 @@ EXPORT_SYMBOL(irq_fpu_usable); · 9 unmodified lines
99	void save_fpregs_to_fpstate(struct fpu *fpu)	115	void save_fpregs_to_fpstate(struct fpu *fpu)
100	{	116	{
101	if (likely(use_xsave())) {	117	if (likely(use_xsave())) {
102	os_xsave(&fpu->state.xsave);	118	os_xsave(fpu->fpstate);
103		119
104	/*	120	/*
105	* AVX512 state is tracked here because its use is	121	* AVX512 state is tracked here because its use is
106	* known to slow the max clock speed of the core.	122	* known to slow the max clock speed of the core.
107	*/	123	*/
108	if (fpu->state.xsave.header.xfeatures & XFEATURE_MASK_AVX512)	124	if (fpu->fpstate->regs.xsave.header.xfeatures & XFEATURE_MASK_AVX512)
109	fpu->avx512_timestamp = jiffies;	125	fpu->avx512_timestamp = jiffies;
110	return;	126	return;
111	}	127	}
112		128
113	if (likely(use_fxsr())) {	129	if (likely(use_fxsr())) {
114	fxsave(&fpu->state.fxsave);	130	fxsave(&fpu->fpstate->regs.fxsave);
115	return;	131	return;
116	}	132	}
117		133
@@ -119,12 +135,11 @@ void save_fpregs_to_fpstate(struct fpu *fpu) · 1 unmodified line
119	* Legacy FPU register saving, FNSAVE always clears FPU registers,	135	* Legacy FPU register saving, FNSAVE always clears FPU registers,
120	* so we have to reload them from the memory state.	136	* so we have to reload them from the memory state.
121	*/	137	*/
122	asm volatile("fnsave %[fp]; fwait" : [fp] "=m" (fpu->state.fsave));	138	asm volatile("fnsave %[fp]; fwait" : [fp] "=m" (fpu->fpstate->regs.fsave));
123	frstor(&fpu->state.fsave);	139	frstor(&fpu->fpstate->regs.fsave);
124	}	140	}
125	EXPORT_SYMBOL(save_fpregs_to_fpstate);
126		141
127	void __restore_fpregs_from_fpstate(union fpregs_state *fpstate, u64 mask)	142	void restore_fpregs_from_fpstate(struct fpstate *fpstate, u64 mask)
128	{	143	{
129	/*	144	/*
130	* AMD K7/K8 and later CPUs up to Zen don't save/restore	145	* AMD K7/K8 and later CPUs up to Zen don't save/restore
@@ -141,15 +156,181 @@ void __restore_fpregs_from_fpstate(union fpregs_state *fpstate, u64 mask) · 10 unmodified lines
141	}	156	}
142		157
143	if (use_xsave()) {	158	if (use_xsave()) {
144	os_xrstor(&fpstate->xsave, mask);	159	/*
		160	* Dynamically enabled features are enabled in XCR0, but
		161	* usage requires also that the corresponding bits in XFD
		162	* are cleared. If the bits are set then using a related
		163	* instruction will raise #NM. This allows to do the
		164	* allocation of the larger FPU buffer lazy from #NM or if
		165	* the task has no permission to kill it which would happen
		166	* via #UD if the feature is disabled in XCR0.
		167	*
		168	* XFD state is following the same life time rules as
		169	* XSTATE and to restore state correctly XFD has to be
		170	* updated before XRSTORS otherwise the component would
		171	* stay in or go into init state even if the bits are set
		172	* in fpstate::regs::xsave::xfeatures.
		173	*/
		174	xfd_update_state(fpstate);
		175
		176	/*
		177	* Restoring state always needs to modify all features
		178	* which are in @mask even if the current task cannot use
		179	* extended features.
		180	*
		181	* So fpstate->xfeatures cannot be used here, because then
		182	* a feature for which the task has no permission but was
		183	* used by the previous task would not go into init state.
		184	*/
		185	mask = fpu_kernel_cfg.max_features & mask;
		186
		187	os_xrstor(fpstate, mask);
145	} else {	188	} else {
146	if (use_fxsr())	189	if (use_fxsr())
147	fxrstor(&fpstate->fxsave);	190	fxrstor(&fpstate->regs.fxsave);
148	else	191	else
149	frstor(&fpstate->fsave);	192	frstor(&fpstate->regs.fsave);
150	}	193	}
151	}	194	}
152	EXPORT_SYMBOL_GPL(__restore_fpregs_from_fpstate);	195
		196	void fpu_reset_from_exception_fixup(void)
		197	{
		198	restore_fpregs_from_fpstate(&init_fpstate, XFEATURE_MASK_FPSTATE);
		199	}
		200
		201	#if IS_ENABLED(CONFIG_KVM)
		202	static void __fpstate_reset(struct fpstate *fpstate);
		203
		204	bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu)
		205	{
		206	struct fpstate *fpstate;
		207	unsigned int size;
		208
		209	size = fpu_user_cfg.default_size + ALIGN(offsetof(struct fpstate, regs), 64);
		210	fpstate = vzalloc(size);
		211	if (!fpstate)
		212	return false;
		213
		214	__fpstate_reset(fpstate);
		215	fpstate_init_user(fpstate);
		216	fpstate->is_valloc = true;
		217	fpstate->is_guest = true;
		218
		219	gfpu->fpstate = fpstate;
		220	return true;
		221	}
		222	EXPORT_SYMBOL_GPL(fpu_alloc_guest_fpstate);
		223
		224	void fpu_free_guest_fpstate(struct fpu_guest *gfpu)
		225	{
		226	struct fpstate *fps = gfpu->fpstate;
		227
		228	if (!fps)
		229	return;
		230
		231	if (WARN_ON_ONCE(!fps->is_valloc \|\| !fps->is_guest \|\| fps->in_use))
		232	return;
		233
		234	gfpu->fpstate = NULL;
		235	vfree(fps);
		236	}
		237	EXPORT_SYMBOL_GPL(fpu_free_guest_fpstate);
		238
		239	int fpu_swap_kvm_fpstate(struct fpu_guest *guest_fpu, bool enter_guest)
		240	{
		241	struct fpstate *guest_fps = guest_fpu->fpstate;
		242	struct fpu *fpu = &current->thread.fpu;
		243	struct fpstate *cur_fps = fpu->fpstate;
		244
		245	fpregs_lock();
		246	if (!cur_fps->is_confidential && !test_thread_flag(TIF_NEED_FPU_LOAD))
		247	save_fpregs_to_fpstate(fpu);
		248
		249	/* Swap fpstate */
		250	if (enter_guest) {
		251	fpu->__task_fpstate = cur_fps;
		252	fpu->fpstate = guest_fps;
		253	guest_fps->in_use = true;
		254	} else {
		255	guest_fps->in_use = false;
		256	fpu->fpstate = fpu->__task_fpstate;
		257	fpu->__task_fpstate = NULL;
		258	}
		259
		260	cur_fps = fpu->fpstate;
		261
		262	if (!cur_fps->is_confidential) {
		263	/* Includes XFD update */
		264	restore_fpregs_from_fpstate(cur_fps, XFEATURE_MASK_FPSTATE);
		265	} else {
		266	/*
		267	* XSTATE is restored by firmware from encrypted
		268	* memory. Make sure XFD state is correct while
		269	* running with guest fpstate
		270	*/
		271	xfd_update_state(cur_fps);
		272	}
		273
		274	fpregs_mark_activate();
		275	fpregs_unlock();
		276	return 0;
		277	}
		278	EXPORT_SYMBOL_GPL(fpu_swap_kvm_fpstate);
		279
		280	void fpu_copy_guest_fpstate_to_uabi(struct fpu_guest gfpu, void buf,
		281	unsigned int size, u32 pkru)
		282	{
		283	struct fpstate *kstate = gfpu->fpstate;
		284	union fpregs_state *ustate = buf;
		285	struct membuf mb = { .p = buf, .left = size };
		286
		287	if (cpu_feature_enabled(X86_FEATURE_XSAVE)) {
		288	__copy_xstate_to_uabi_buf(mb, kstate, pkru, XSTATE_COPY_XSAVE);
		289	} else {
		290	memcpy(&ustate->fxsave, &kstate->regs.fxsave,
		291	sizeof(ustate->fxsave));
		292	/* Make it restorable on a XSAVE enabled host */
		293	ustate->xsave.header.xfeatures = XFEATURE_MASK_FPSSE;
		294	}
		295	}
		296	EXPORT_SYMBOL_GPL(fpu_copy_guest_fpstate_to_uabi);
		297
		298	int fpu_copy_uabi_to_guest_fpstate(struct fpu_guest gfpu, const void buf,
		299	u64 xcr0, u32 *vpkru)
		300	{
		301	struct fpstate *kstate = gfpu->fpstate;
		302	const union fpregs_state *ustate = buf;
		303	struct pkru_state *xpkru;
		304	int ret;
		305
		306	if (!cpu_feature_enabled(X86_FEATURE_XSAVE)) {
		307	if (ustate->xsave.header.xfeatures & ~XFEATURE_MASK_FPSSE)
		308	return -EINVAL;
		309	if (ustate->fxsave.mxcsr & ~mxcsr_feature_mask)
		310	return -EINVAL;
		311	memcpy(&kstate->regs.fxsave, &ustate->fxsave, sizeof(ustate->fxsave));
		312	return 0;
		313	}
		314
		315	if (ustate->xsave.header.xfeatures & ~xcr0)
		316	return -EINVAL;
		317
		318	ret = copy_uabi_from_kernel_to_xstate(kstate, ustate);
		319	if (ret)
		320	return ret;
		321
		322	/* Retrieve PKRU if not in init state */
		323	if (kstate->regs.xsave.header.xfeatures & XFEATURE_MASK_PKRU) {
		324	xpkru = get_xsave_addr(&kstate->regs.xsave, XFEATURE_PKRU);
		325	*vpkru = xpkru->pkru;
		326	}
		327
		328	/* Ensure that XCOMP_BV is set up for XSAVES */
		329	xstate_init_xcomp_bv(&kstate->regs.xsave, kstate->xfeatures);
		330	return 0;
		331	}
		332	EXPORT_SYMBOL_GPL(fpu_copy_uabi_to_guest_fpstate);
		333	#endif /* CONFIG_KVM */
153		334
154	void kernel_fpu_begin_mask(unsigned int kfpu_mask)	335	void kernel_fpu_begin_mask(unsigned int kfpu_mask)
155	{	336	{
@@ -203,52 +384,88 @@ void fpu_sync_fpstate(struct fpu *fpu) · 47 unmodified lines
203	fpregs_unlock();	384	fpregs_unlock();
204	}	385	}
205		386
206	static inline void fpstate_init_xstate(struct xregs_state *xsave)	387	static inline unsigned int init_fpstate_copy_size(void)
207	{	388	{
208	/*	389	if (!use_xsave())
209	* XRSTORS requires these bits set in xcomp_bv, or it will	390	return fpu_kernel_cfg.default_size;
210	* trigger #GP:	391
211	*/	392	/* XSAVE(S) just needs the legacy and the xstate header part */
212	xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT \| xfeatures_mask_all;	393	return sizeof(init_fpstate.regs.xsave);
213	}	394	}
214		395
215	static inline void fpstate_init_fxstate(struct fxregs_state *fx)	396	static inline void fpstate_init_fxstate(struct fpstate *fpstate)
216	{	397	{
217	fx->cwd = 0x37f;	398	fpstate->regs.fxsave.cwd = 0x37f;
218	fx->mxcsr = MXCSR_DEFAULT;	399	fpstate->regs.fxsave.mxcsr = MXCSR_DEFAULT;
219	}	400	}
220		401
221	/*	402	/*
222	* Legacy x87 fpstate state init:	403	* Legacy x87 fpstate state init:
223	*/	404	*/
224	static inline void fpstate_init_fstate(struct fregs_state *fp)	405	static inline void fpstate_init_fstate(struct fpstate *fpstate)
225	{	406	{
226	fp->cwd = 0xffff037fu;	407	fpstate->regs.fsave.cwd = 0xffff037fu;
227	fp->swd = 0xffff0000u;	408	fpstate->regs.fsave.swd = 0xffff0000u;
228	fp->twd = 0xffffffffu;	409	fpstate->regs.fsave.twd = 0xffffffffu;
229	fp->fos = 0xffff0000u;	410	fpstate->regs.fsave.fos = 0xffff0000u;
230	}	411	}
231		412
232	void fpstate_init(union fpregs_state *state)	413	/*
		414	* Used in two places:
		415	* 1) Early boot to setup init_fpstate for non XSAVE systems
		416	* 2) fpu_init_fpstate_user() which is invoked from KVM
		417	*/
		418	void fpstate_init_user(struct fpstate *fpstate)
233	{	419	{
234	if (!static_cpu_has(X86_FEATURE_FPU)) {	420	if (!cpu_feature_enabled(X86_FEATURE_FPU)) {
235	fpstate_init_soft(&state->soft);	421	fpstate_init_soft(&fpstate->regs.soft);
236	return;	422	return;
237	}	423	}
238		424
239	memset(state, 0, fpu_kernel_xstate_size);	425	xstate_init_xcomp_bv(&fpstate->regs.xsave, fpstate->xfeatures);
240		426
241	if (static_cpu_has(X86_FEATURE_XSAVES))	427	if (cpu_feature_enabled(X86_FEATURE_FXSR))
242	fpstate_init_xstate(&state->xsave);	428	fpstate_init_fxstate(fpstate);
243	if (static_cpu_has(X86_FEATURE_FXSR))
244	fpstate_init_fxstate(&state->fxsave);
245	else	429	else
246	fpstate_init_fstate(&state->fsave);	430	fpstate_init_fstate(fpstate);
		431	}
		432
		433	static void __fpstate_reset(struct fpstate *fpstate)
		434	{
		435	/* Initialize sizes and feature masks */
		436	fpstate->size = fpu_kernel_cfg.default_size;
		437	fpstate->user_size = fpu_user_cfg.default_size;
		438	fpstate->xfeatures = fpu_kernel_cfg.default_features;
		439	fpstate->user_xfeatures = fpu_user_cfg.default_features;
		440	fpstate->xfd = init_fpstate.xfd;
		441	}
		442
		443	void fpstate_reset(struct fpu *fpu)
		444	{
		445	/* Set the fpstate pointer to the default fpstate */
		446	fpu->fpstate = &fpu->__fpstate;
		447	__fpstate_reset(fpu->fpstate);
		448
		449	/* Initialize the permission related info in fpu */
		450	fpu->perm.__state_perm = fpu_kernel_cfg.default_features;
		451	fpu->perm.__state_size = fpu_kernel_cfg.default_size;
		452	fpu->perm.__user_state_size = fpu_user_cfg.default_size;
		453	}
		454
		455	static inline void fpu_inherit_perms(struct fpu *dst_fpu)
		456	{
		457	if (fpu_state_size_dynamic()) {
		458	struct fpu *src_fpu = &current->group_leader->thread.fpu;
		459
		460	spin_lock_irq(&current->sighand->siglock);
		461	/* Fork also inherits the permissions of the parent */
		462	dst_fpu->perm = src_fpu->perm;
		463	spin_unlock_irq(&current->sighand->siglock);
		464	}
247	}	465	}
248	EXPORT_SYMBOL_GPL(fpstate_init);
249		466
250	/* Clone current's FPU state on fork */	467	/* Clone current's FPU state on fork */
251	int fpu_clone(struct task_struct *dst)	468	int fpu_clone(struct task_struct *dst, unsigned long clone_flags)
252	{	469	{
253	struct fpu *src_fpu = &current->thread.fpu;	470	struct fpu *src_fpu = &current->thread.fpu;
254	struct fpu *dst_fpu = &dst->thread.fpu;	471	struct fpu *dst_fpu = &dst->thread.fpu;
@@ -256,36 +473,67 @@ int fpu_clone(struct task_struct *dst) · 1 unmodified line
256	/* The new task's FPU state cannot be valid in the hardware. */	473	/* The new task's FPU state cannot be valid in the hardware. */
257	dst_fpu->last_cpu = -1;	474	dst_fpu->last_cpu = -1;
258		475
		476	fpstate_reset(dst_fpu);
		477
259	if (!cpu_feature_enabled(X86_FEATURE_FPU))	478	if (!cpu_feature_enabled(X86_FEATURE_FPU))
260	return 0;	479	return 0;
261		480
262	/*	481	/*
263	* Don't let 'init optimized' areas of the XSAVE area	482	* Enforce reload for user space tasks and prevent kernel threads
264	* leak into the child task:	483	* from trying to save the FPU registers on context switch.
		484	*/
		485	set_tsk_thread_flag(dst, TIF_NEED_FPU_LOAD);
		486
		487	/*
		488	* No FPU state inheritance for kernel threads and IO
		489	* worker threads.
		490	*/
		491	if (dst->flags & (PF_KTHREAD \| PF_IO_WORKER)) {
		492	/* Clear out the minimal state */
		493	memcpy(&dst_fpu->fpstate->regs, &init_fpstate.regs,
		494	init_fpstate_copy_size());
		495	return 0;
		496	}
		497
		498	/*
		499	* If a new feature is added, ensure all dynamic features are
		500	* caller-saved from here!
265	*/	501	*/
266	memset(&dst_fpu->state.xsave, 0, fpu_kernel_xstate_size);	502	BUILD_BUG_ON(XFEATURE_MASK_USER_DYNAMIC != XFEATURE_MASK_XTILE_DATA);
267		503
268	/*	504	/*
269	* If the FPU registers are not owned by current just memcpy() the	505	* Save the default portion of the current FPU state into the
270	* state. Otherwise save the FPU registers directly into the	506	* clone. Assume all dynamic features to be defined as caller-
271	* child's FPU context, without any memory-to-memory copying.	507	* saved, which enables skipping both the expansion of fpstate
		508	* and the copying of any dynamic state.
		509	*
		510	* Do not use memcpy() when TIF_NEED_FPU_LOAD is set because
		511	* copying is not valid when current uses non-default states.
272	*/	512	*/
273	fpregs_lock();	513	fpregs_lock();
274	if (test_thread_flag(TIF_NEED_FPU_LOAD))	514	if (test_thread_flag(TIF_NEED_FPU_LOAD))
275	memcpy(&dst_fpu->state, &src_fpu->state, fpu_kernel_xstate_size);	515	fpregs_restore_userregs();
276		516	save_fpregs_to_fpstate(dst_fpu);
277	else	517	if (!(clone_flags & CLONE_THREAD))
278	save_fpregs_to_fpstate(dst_fpu);	518	fpu_inherit_perms(dst_fpu);
279	fpregs_unlock();	519	fpregs_unlock();
280		520
281	set_tsk_thread_flag(dst, TIF_NEED_FPU_LOAD);
282
283	trace_x86_fpu_copy_src(src_fpu);	521	trace_x86_fpu_copy_src(src_fpu);
284	trace_x86_fpu_copy_dst(dst_fpu);	522	trace_x86_fpu_copy_dst(dst_fpu);
285		523
286	return 0;	524	return 0;
287	}	525	}
288		526
		527	/*
		528	* Whitelist the FPU register state embedded into task_struct for hardened
		529	* usercopy.
		530	*/
		531	void fpu_thread_struct_whitelist(unsigned long offset, unsigned long size)
		532	{
		533	*offset = offsetof(struct thread_struct, fpu.__fpstate.regs);
		534	*size = fpu_kernel_cfg.default_size;
		535	}
		536
289	/*	537	/*
290	* Drops current FPU state: deactivates the fpregs and	538	* Drops current FPU state: deactivates the fpregs and
291	* the fpstate. NOTE: it still leaves previous contents	539	* the fpstate. NOTE: it still leaves previous contents
@@ -319,28 +567,19 @@ void fpu__drop(struct fpu *fpu) · 27 unmodified lines
319	static inline void restore_fpregs_from_init_fpstate(u64 features_mask)	567	static inline void restore_fpregs_from_init_fpstate(u64 features_mask)
320	{	568	{
321	if (use_xsave())	569	if (use_xsave())
322	os_xrstor(&init_fpstate.xsave, features_mask);	570	os_xrstor(&init_fpstate, features_mask);
323	else if (use_fxsr())	571	else if (use_fxsr())
324	fxrstor(&init_fpstate.fxsave);	572	fxrstor(&init_fpstate.regs.fxsave);
325	else	573	else
326	frstor(&init_fpstate.fsave);	574	frstor(&init_fpstate.regs.fsave);
327		575
328	pkru_write_default();	576	pkru_write_default();
329	}	577	}
330		578
331	static inline unsigned int init_fpstate_copy_size(void)
332	{
333	if (!use_xsave())
334	return fpu_kernel_xstate_size;
335
336	/* XSAVE(S) just needs the legacy and the xstate header part */
337	return sizeof(init_fpstate.xsave);
338	}
339
340	/*	579	/*
341	* Reset current->fpu memory state to the init values.	580	* Reset current->fpu memory state to the init values.
342	*/	581	*/
343	static void fpu_reset_fpstate(void)	582	static void fpu_reset_fpregs(void)
344	{	583	{
345	struct fpu *fpu = &current->thread.fpu;	584	struct fpu *fpu = &current->thread.fpu;
346		585
@@ -359,7 +598,7 @@ static void fpu_reset_fpstate(void) · 12 unmodified lines
359	* user space as PKRU is eagerly written in switch_to() and	598	* user space as PKRU is eagerly written in switch_to() and
360	* flush_thread().	599	* flush_thread().
361	*/	600	*/
362	memcpy(&fpu->state, &init_fpstate, init_fpstate_copy_size());	601	memcpy(&fpu->fpstate->regs, &init_fpstate.regs, init_fpstate_copy_size());
363	set_thread_flag(TIF_NEED_FPU_LOAD);	602	set_thread_flag(TIF_NEED_FPU_LOAD);
364	fpregs_unlock();	603	fpregs_unlock();
365	}	604	}
@@ -375,7 +614,7 @@ void fpu__clear_user_states(struct fpu *fpu) · 9 unmodified lines
375		614
376	fpregs_lock();	615	fpregs_lock();
377	if (!cpu_feature_enabled(X86_FEATURE_FPU)) {	616	if (!cpu_feature_enabled(X86_FEATURE_FPU)) {
378	fpu_reset_fpstate();	617	fpu_reset_fpregs();
379	fpregs_unlock();	618	fpregs_unlock();
380	return;	619	return;
381	}	620	}
@@ -385,12 +624,11 @@ void fpu__clear_user_states(struct fpu *fpu) · 3 unmodified lines
385	* corresponding registers.	624	* corresponding registers.
386	*/	625	*/
387	if (xfeatures_mask_supervisor() &&	626	if (xfeatures_mask_supervisor() &&
388	!fpregs_state_valid(fpu, smp_processor_id())) {	627	!fpregs_state_valid(fpu, smp_processor_id()))
389	os_xrstor(&fpu->state.xsave, xfeatures_mask_supervisor());	628	os_xrstor_supervisor(fpu->fpstate);
390	}
391		629
392	/* Reset user states in registers. */	630	/* Reset user states in registers. */
393	restore_fpregs_from_init_fpstate(xfeatures_mask_restore_user());	631	restore_fpregs_from_init_fpstate(XFEATURE_MASK_USER_RESTORE);
394		632
395	/*	633	/*
396	* Now all FPU registers have their desired values. Inform the FPU	634	* Now all FPU registers have their desired values. Inform the FPU
@@ -405,7 +643,8 @@ void fpu__clear_user_states(struct fpu *fpu) · 8 unmodified lines
405		643
406	void fpu_flush_thread(void)	644	void fpu_flush_thread(void)
407	{	645	{
408	fpu_reset_fpstate();	646	fpstate_reset(&current->thread.fpu);
		647	fpu_reset_fpregs();
409	}	648	}
410	/*	649	/*
411	* Load FPU context before returning to userspace.	650	* Load FPU context before returning to userspace.
@@ -445,7 +684,6 @@ void fpregs_mark_activate(void) · 33 unmodified lines
445	fpu->last_cpu = smp_processor_id();	684	fpu->last_cpu = smp_processor_id();
446	clear_thread_flag(TIF_NEED_FPU_LOAD);	685	clear_thread_flag(TIF_NEED_FPU_LOAD);
447	}	686	}
448	EXPORT_SYMBOL_GPL(fpregs_mark_activate);
449		687
450	/*	688	/*
451	* x87 math exception handling:	689	* x87 math exception handling:
@@ -468,11 +706,11 @@ int fpu__exception_code(struct fpu *fpu, int trap_nr) · 16 unmodified lines
468	* fully reproduce the context of the exception.	706	* fully reproduce the context of the exception.
469	*/	707	*/
470	if (boot_cpu_has(X86_FEATURE_FXSR)) {	708	if (boot_cpu_has(X86_FEATURE_FXSR)) {
471	cwd = fpu->state.fxsave.cwd;	709	cwd = fpu->fpstate->regs.fxsave.cwd;
472	swd = fpu->state.fxsave.swd;	710	swd = fpu->fpstate->regs.fxsave.swd;
473	} else {	711	} else {
474	cwd = (unsigned short)fpu->state.fsave.cwd;	712	cwd = (unsigned short)fpu->fpstate->regs.fsave.cwd;
475	swd = (unsigned short)fpu->state.fsave.swd;	713	swd = (unsigned short)fpu->fpstate->regs.fsave.swd;
476	}	714	}
477		715
478	err = swd & ~cwd;	716	err = swd & ~cwd;
@@ -486,7 +724,7 @@ int fpu__exception_code(struct fpu *fpu, int trap_nr) · 7 unmodified lines
486	unsigned short mxcsr = MXCSR_DEFAULT;	724	unsigned short mxcsr = MXCSR_DEFAULT;
487		725
488	if (boot_cpu_has(X86_FEATURE_XMM))	726	if (boot_cpu_has(X86_FEATURE_XMM))
489	mxcsr = fpu->state.fxsave.mxcsr;	727	mxcsr = fpu->fpstate->regs.fxsave.mxcsr;
490		728
491	err = ~(mxcsr >> 7) & mxcsr;	729	err = ~(mxcsr >> 7) & mxcsr;
492	}	730	}

arch/x86/kernel/fpu/init.c+34−42

@@ -2,7 +2,7 @@ · 1 unmodified line
2	/*	2	/*
3	* x86 FPU boot time init code:	3	* x86 FPU boot time init code:
4	*/	4	*/
5	#include <asm/fpu/internal.h>	5	#include <asm/fpu/api.h>
6	#include <asm/tlbflush.h>	6	#include <asm/tlbflush.h>
7	#include <asm/setup.h>	7	#include <asm/setup.h>
8		8
@@ -10,6 +10,10 @@ · 1 unmodified line
10	#include <linux/sched/task.h>	10	#include <linux/sched/task.h>
11	#include <linux/init.h>	11	#include <linux/init.h>
12		12
		13	#include "internal.h"
		14	#include "legacy.h"
		15	#include "xstate.h"
		16
13	/*	17	/*
14	* Initialize the registers found in all CPUs, CR0 and CR4:	18	* Initialize the registers found in all CPUs, CR0 and CR4:
15	*/	19	*/
@@ -34,7 +38,7 @@ static void fpu__init_cpu_generic(void) · 18 unmodified lines
34	/* Flush out any pending x87 state: */	38	/* Flush out any pending x87 state: */
35	#ifdef CONFIG_MATH_EMULATION	39	#ifdef CONFIG_MATH_EMULATION
36	if (!boot_cpu_has(X86_FEATURE_FPU))	40	if (!boot_cpu_has(X86_FEATURE_FPU))
37	fpstate_init_soft(&current->thread.fpu.state.soft);	41	fpstate_init_soft(&current->thread.fpu.fpstate->regs.soft);
38	else	42	else
39	#endif	43	#endif
40	asm volatile ("fninit");	44	asm volatile ("fninit");
@@ -121,23 +125,14 @@ static void __init fpu__init_system_mxcsr(void) · 80 unmodified lines
121	static void __init fpu__init_system_generic(void)	125	static void __init fpu__init_system_generic(void)
122	{	126	{
123	/*	127	/*
124	* Set up the legacy init FPU context. (xstate init might overwrite this	128	* Set up the legacy init FPU context. Will be updated when the
125	* with a more modern format, if the CPU supports it.)	129	* CPU supports XSAVE[S].
126	*/	130	*/
127	fpstate_init(&init_fpstate);	131	fpstate_init_user(&init_fpstate);
128		132
129	fpu__init_system_mxcsr();	133	fpu__init_system_mxcsr();
130	}	134	}
131		135
132	/*
133	* Size of the FPU context state. All tasks in the system use the
134	* same context size, regardless of what portion they use.
135	* This is inherent to the XSAVE architecture which puts all state
136	* components into a single, continuous memory block:
137	*/
138	unsigned int fpu_kernel_xstate_size __ro_after_init;
139	EXPORT_SYMBOL_GPL(fpu_kernel_xstate_size);
140
141	/* Get alignment of the TYPE. */	136	/* Get alignment of the TYPE. */
142	#define TYPE_ALIGN(TYPE) offsetof(struct { char x; TYPE test; }, test)	137	#define TYPE_ALIGN(TYPE) offsetof(struct { char x; TYPE test; }, test)
143		138
@@ -162,13 +157,13 @@ static void __init fpu__init_task_struct_size(void) · 18 unmodified lines
162	* Subtract off the static size of the register state.	157	* Subtract off the static size of the register state.
163	* It potentially has a bunch of padding.	158	* It potentially has a bunch of padding.
164	*/	159	*/
165	task_size -= sizeof(((struct task_struct *)0)->thread.fpu.state);	160	task_size -= sizeof(current->thread.fpu.__fpstate.regs);
166		161
167	/*	162	/*
168	* Add back the dynamically-calculated register state	163	* Add back the dynamically-calculated register state
169	* size.	164	* size.
170	*/	165	*/
171	task_size += fpu_kernel_xstate_size;	166	task_size += fpu_kernel_cfg.default_size;
172		167
173	/*	168	/*
174	* We dynamically size 'struct fpu', so we require that	169	* We dynamically size 'struct fpu', so we require that
@@ -177,7 +172,7 @@ static void __init fpu__init_task_struct_size(void) · 2 unmodified lines
177	* you hit a compile error here, check the structure to	172	* you hit a compile error here, check the structure to
178	* see if something got added to the end.	173	* see if something got added to the end.
179	*/	174	*/
180	CHECK_MEMBER_AT_END_OF(struct fpu, state);	175	CHECK_MEMBER_AT_END_OF(struct fpu, __fpstate);
181	CHECK_MEMBER_AT_END_OF(struct thread_struct, fpu);	176	CHECK_MEMBER_AT_END_OF(struct thread_struct, fpu);
182	CHECK_MEMBER_AT_END_OF(struct task_struct, thread);	177	CHECK_MEMBER_AT_END_OF(struct task_struct, thread);
183		178
@@ -192,37 +187,34 @@ static void __init fpu__init_task_struct_size(void) · 8 unmodified lines
192	*/	187	*/
193	static void __init fpu__init_system_xstate_size_legacy(void)	188	static void __init fpu__init_system_xstate_size_legacy(void)
194	{	189	{
195	static int on_boot_cpu __initdata = 1;	190	unsigned int size;
196
197	WARN_ON_FPU(!on_boot_cpu);
198	on_boot_cpu = 0;
199		191
200	/*	192	/*
201	* Note that xstate sizes might be overwritten later during	193	* Note that the size configuration might be overwritten later
202	* fpu__init_system_xstate().	194	* during fpu__init_system_xstate().
203	*/	195	*/
204		196	if (!cpu_feature_enabled(X86_FEATURE_FPU)) {
205	if (!boot_cpu_has(X86_FEATURE_FPU)) {	197	size = sizeof(struct swregs_state);
206	fpu_kernel_xstate_size = sizeof(struct swregs_state);	198	} else if (cpu_feature_enabled(X86_FEATURE_FXSR)) {
		199	size = sizeof(struct fxregs_state);
		200	fpu_user_cfg.legacy_features = XFEATURE_MASK_FPSSE;
207	} else {	201	} else {
208	if (boot_cpu_has(X86_FEATURE_FXSR))	202	size = sizeof(struct fregs_state);
209	fpu_kernel_xstate_size =	203	fpu_user_cfg.legacy_features = XFEATURE_MASK_FP;
210	sizeof(struct fxregs_state);
211	else
212	fpu_kernel_xstate_size =
213	sizeof(struct fregs_state);
214	}	204	}
215		205
216	fpu_user_xstate_size = fpu_kernel_xstate_size;	206	fpu_kernel_cfg.max_size = size;
		207	fpu_kernel_cfg.default_size = size;
		208	fpu_user_cfg.max_size = size;
		209	fpu_user_cfg.default_size = size;
		210	fpstate_reset(&current->thread.fpu);
217	}	211	}
218		212
219	/* Legacy code to initialize eager fpu mode. */	213	static void __init fpu__init_init_fpstate(void)
220	static void __init fpu__init_system_ctx_switch(void)
221	{	214	{
222	static bool on_boot_cpu __initdata = 1;	215	/* Bring init_fpstate size and features up to date */
223		216	init_fpstate.size = fpu_kernel_cfg.max_size;
224	WARN_ON_FPU(!on_boot_cpu);	217	init_fpstate.xfeatures = fpu_kernel_cfg.max_features;
225	on_boot_cpu = 0;
226	}	218	}
227		219
228	/*	220	/*
@@ -231,6 +223,7 @@ static void __init fpu__init_system_ctx_switch(void) · 2 unmodified lines
231	*/	223	*/
232	void __init fpu__init_system(struct cpuinfo_x86 *c)	224	void __init fpu__init_system(struct cpuinfo_x86 *c)
233	{	225	{
		226	fpstate_reset(&current->thread.fpu);
234	fpu__init_system_early_generic(c);	227	fpu__init_system_early_generic(c);
235		228
236	/*	229	/*
@@ -241,8 +234,7 @@ void __init fpu__init_system(struct cpuinfo_x86 *c) · 4 unmodified lines
241		234
242	fpu__init_system_generic();	235	fpu__init_system_generic();
243	fpu__init_system_xstate_size_legacy();	236	fpu__init_system_xstate_size_legacy();
244	fpu__init_system_xstate();	237	fpu__init_system_xstate(fpu_kernel_cfg.max_size);
245	fpu__init_task_struct_size();	238	fpu__init_task_struct_size();
246		239	fpu__init_init_fpstate();
247	fpu__init_system_ctx_switch();
248	}	240	}

arch/x86/kernel/fpu/internal.h+28−0

@@ -0,0 +1,28 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#ifndef __X86_KERNEL_FPU_INTERNAL_H
	3	#define __X86_KERNEL_FPU_INTERNAL_H
	4
	5	extern struct fpstate init_fpstate;
	6
	7	/* CPU feature check wrappers */
	8	static __always_inline __pure bool use_xsave(void)
	9	{
	10	return cpu_feature_enabled(X86_FEATURE_XSAVE);
	11	}
	12
	13	static __always_inline __pure bool use_fxsr(void)
	14	{
	15	return cpu_feature_enabled(X86_FEATURE_FXSR);
	16	}
	17
	18	#ifdef CONFIG_X86_DEBUG_FPU
	19	# define WARN_ON_FPU(x) WARN_ON_ONCE(x)
	20	#else
	21	# define WARN_ON_FPU(x) ({ (void)(x); 0; })
	22	#endif
	23
	24	/* Used in init.c */
	25	extern void fpstate_init_user(struct fpstate *fpstate);
	26	extern void fpstate_reset(struct fpu *fpu);
	27
	28	#endif

arch/x86/kernel/fpu/legacy.h+115−0

@@ -0,0 +1,115 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#ifndef __X86_KERNEL_FPU_LEGACY_H
	3	#define __X86_KERNEL_FPU_LEGACY_H
	4
	5	#include <asm/fpu/types.h>
	6
	7	extern unsigned int mxcsr_feature_mask;
	8
	9	static inline void ldmxcsr(u32 mxcsr)
	10	{
	11	asm volatile("ldmxcsr %0" :: "m" (mxcsr));
	12	}
	13
	14	/*
	15	* Returns 0 on success or the trap number when the operation raises an
	16	* exception.
	17	*/
	18	#define user_insn(insn, output, input...) \
	19	({ \
	20	int err; \
	21	\
	22	might_fault(); \
	23	\
	24	asm volatile(ASM_STAC "\n" \
	25	"1: " #insn "\n" \
	26	"2: " ASM_CLAC "\n" \
	27	_ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_FAULT_MCE_SAFE) \
	28	: [err] "=a" (err), output \
	29	: "0"(0), input); \
	30	err; \
	31	})
	32
	33	#define kernel_insn_err(insn, output, input...) \
	34	({ \
	35	int err; \
	36	asm volatile("1:" #insn "\n\t" \
	37	"2:\n" \
	38	".section .fixup,\"ax\"\n" \
	39	"3: movl $-1,%[err]\n" \
	40	" jmp 2b\n" \
	41	".previous\n" \
	42	_ASM_EXTABLE(1b, 3b) \
	43	: [err] "=r" (err), output \
	44	: "0"(0), input); \
	45	err; \
	46	})
	47
	48	#define kernel_insn(insn, output, input...) \
	49	asm volatile("1:" #insn "\n\t" \
	50	"2:\n" \
	51	_ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_FPU_RESTORE) \
	52	: output : input)
	53
	54	static inline int fnsave_to_user_sigframe(struct fregs_state __user *fx)
	55	{
	56	return user_insn(fnsave %[fx]; fwait, [fx] "=m" (fx), "m" (fx));
	57	}
	58
	59	static inline int fxsave_to_user_sigframe(struct fxregs_state __user *fx)
	60	{
	61	if (IS_ENABLED(CONFIG_X86_32))
	62	return user_insn(fxsave %[fx], [fx] "=m" (fx), "m" (fx));
	63	else
	64	return user_insn(fxsaveq %[fx], [fx] "=m" (fx), "m" (fx));
	65
	66	}
	67
	68	static inline void fxrstor(struct fxregs_state *fx)
	69	{
	70	if (IS_ENABLED(CONFIG_X86_32))
	71	kernel_insn(fxrstor %[fx], "=m" (fx), [fx] "m" (fx));
	72	else
	73	kernel_insn(fxrstorq %[fx], "=m" (fx), [fx] "m" (fx));
	74	}
	75
	76	static inline int fxrstor_safe(struct fxregs_state *fx)
	77	{
	78	if (IS_ENABLED(CONFIG_X86_32))
	79	return kernel_insn_err(fxrstor %[fx], "=m" (fx), [fx] "m" (fx));
	80	else
	81	return kernel_insn_err(fxrstorq %[fx], "=m" (fx), [fx] "m" (fx));
	82	}
	83
	84	static inline int fxrstor_from_user_sigframe(struct fxregs_state __user *fx)
	85	{
	86	if (IS_ENABLED(CONFIG_X86_32))
	87	return user_insn(fxrstor %[fx], "=m" (fx), [fx] "m" (fx));
	88	else
	89	return user_insn(fxrstorq %[fx], "=m" (fx), [fx] "m" (fx));
	90	}
	91
	92	static inline void frstor(struct fregs_state *fx)
	93	{
	94	kernel_insn(frstor %[fx], "=m" (fx), [fx] "m" (fx));
	95	}
	96
	97	static inline int frstor_safe(struct fregs_state *fx)
	98	{
	99	return kernel_insn_err(frstor %[fx], "=m" (fx), [fx] "m" (fx));
	100	}
	101
	102	static inline int frstor_from_user_sigframe(struct fregs_state __user *fx)
	103	{
	104	return user_insn(frstor %[fx], "=m" (fx), [fx] "m" (fx));
	105	}
	106
	107	static inline void fxsave(struct fxregs_state *fx)
	108	{
	109	if (IS_ENABLED(CONFIG_X86_32))
	110	asm volatile( "fxsave %[fx]" : [fx] "=m" (*fx));
	111	else
	112	asm volatile("fxsaveq %[fx]" : [fx] "=m" (*fx));
	113	}
	114
	115	#endif

arch/x86/kernel/fpu/regset.c+20−16

@@ -5,10 +5,14 @@ · 4 unmodified lines
5	#include <linux/sched/task_stack.h>	5	#include <linux/sched/task_stack.h>
6	#include <linux/vmalloc.h>	6	#include <linux/vmalloc.h>
7		7
8	#include <asm/fpu/internal.h>	8	#include <asm/fpu/api.h>
9	#include <asm/fpu/signal.h>	9	#include <asm/fpu/signal.h>
10	#include <asm/fpu/regset.h>	10	#include <asm/fpu/regset.h>
11	#include <asm/fpu/xstate.h>	11
		12	#include "context.h"
		13	#include "internal.h"
		14	#include "legacy.h"
		15	#include "xstate.h"
12		16
13	/*	17	/*
14	* The xstateregs_active() routine is the same as the regset_fpregs_active() routine,	18	* The xstateregs_active() routine is the same as the regset_fpregs_active() routine,
@@ -74,8 +78,8 @@ int xfpregs_get(struct task_struct target, const struct user_regset regset, · 59 unmodified lines
74	sync_fpstate(fpu);	78	sync_fpstate(fpu);
75		79
76	if (!use_xsave()) {	80	if (!use_xsave()) {
77	return membuf_write(&to, &fpu->state.fxsave,	81	return membuf_write(&to, &fpu->fpstate->regs.fxsave,
78	sizeof(fpu->state.fxsave));	82	sizeof(fpu->fpstate->regs.fxsave));
79	}	83	}
80		84
81	copy_xstate_to_uabi_buf(to, target, XSTATE_COPY_FX);	85	copy_xstate_to_uabi_buf(to, target, XSTATE_COPY_FX);
@@ -110,15 +114,15 @@ int xfpregs_set(struct task_struct target, const struct user_regset regset, · 28 unmodified lines
110	fpu_force_restore(fpu);	114	fpu_force_restore(fpu);
111		115
112	/* Copy the state */	116	/* Copy the state */
113	memcpy(&fpu->state.fxsave, &newstate, sizeof(newstate));	117	memcpy(&fpu->fpstate->regs.fxsave, &newstate, sizeof(newstate));
114		118
115	/* Clear xmm8..15 */	119	/* Clear xmm8..15 */
116	BUILD_BUG_ON(sizeof(fpu->state.fxsave.xmm_space) != 16 * 16);	120	BUILD_BUG_ON(sizeof(fpu->__fpstate.regs.fxsave.xmm_space) != 16 * 16);
117	memset(&fpu->state.fxsave.xmm_space[8], 0, 8 * 16);	121	memset(&fpu->fpstate->regs.fxsave.xmm_space[8], 0, 8 * 16);
118		122
119	/* Mark FP and SSE as in use when XSAVE is enabled */	123	/* Mark FP and SSE as in use when XSAVE is enabled */
120	if (use_xsave())	124	if (use_xsave())
121	fpu->state.xsave.header.xfeatures \|= XFEATURE_MASK_FPSSE;	125	fpu->fpstate->regs.xsave.header.xfeatures \|= XFEATURE_MASK_FPSSE;
122		126
123	return 0;	127	return 0;
124	}	128	}
@@ -149,7 +153,7 @@ int xstateregs_set(struct task_struct target, const struct user_regset regset, · 24 unmodified lines
149	/*	153	/*
150	* A whole standard-format XSAVE buffer is needed:	154	* A whole standard-format XSAVE buffer is needed:
151	*/	155	*/
152	if (pos != 0 \|\| count != fpu_user_xstate_size)	156	if (pos != 0 \|\| count != fpu_user_cfg.max_size)
153	return -EFAULT;	157	return -EFAULT;
154		158
155	if (!kbuf) {	159	if (!kbuf) {
@@ -164,7 +168,7 @@ int xstateregs_set(struct task_struct target, const struct user_regset regset, · 8 unmodified lines
164	}	168	}
165		169
166	fpu_force_restore(fpu);	170	fpu_force_restore(fpu);
167	ret = copy_uabi_from_kernel_to_xstate(&fpu->state.xsave, kbuf ?: tmpbuf);	171	ret = copy_uabi_from_kernel_to_xstate(fpu->fpstate, kbuf ?: tmpbuf);
168		172
169	out:	173	out:
170	vfree(tmpbuf);	174	vfree(tmpbuf);
@@ -283,7 +287,7 @@ static void __convert_from_fxsr(struct user_i387_ia32_struct *env, · 112 unmodified lines
283	void	287	void
284	convert_from_fxsr(struct user_i387_ia32_struct env, struct task_struct tsk)	288	convert_from_fxsr(struct user_i387_ia32_struct env, struct task_struct tsk)
285	{	289	{
286	__convert_from_fxsr(env, tsk, &tsk->thread.fpu.state.fxsave);	290	__convert_from_fxsr(env, tsk, &tsk->thread.fpu.fpstate->regs.fxsave);
287	}	291	}
288		292
289	void convert_to_fxsr(struct fxregs_state *fxsave,	293	void convert_to_fxsr(struct fxregs_state *fxsave,
@@ -326,7 +330,7 @@ int fpregs_get(struct task_struct target, const struct user_regset regset, · 36 unmodified lines
326	return fpregs_soft_get(target, regset, to);	330	return fpregs_soft_get(target, regset, to);
327		331
328	if (!cpu_feature_enabled(X86_FEATURE_FXSR)) {	332	if (!cpu_feature_enabled(X86_FEATURE_FXSR)) {
329	return membuf_write(&to, &fpu->state.fsave,	333	return membuf_write(&to, &fpu->fpstate->regs.fsave,
330	sizeof(struct fregs_state));	334	sizeof(struct fregs_state));
331	}	335	}
332		336
@@ -337,7 +341,7 @@ int fpregs_get(struct task_struct target, const struct user_regset regset, · 4 unmodified lines
337	copy_xstate_to_uabi_buf(mb, target, XSTATE_COPY_FP);	341	copy_xstate_to_uabi_buf(mb, target, XSTATE_COPY_FP);
338	fx = &fxsave;	342	fx = &fxsave;
339	} else {	343	} else {
340	fx = &fpu->state.fxsave;	344	fx = &fpu->fpstate->regs.fxsave;
341	}	345	}
342		346
343	__convert_from_fxsr(&env, target, fx);	347	__convert_from_fxsr(&env, target, fx);
@@ -366,16 +370,16 @@ int fpregs_set(struct task_struct target, const struct user_regset regset, · 22 unmodified lines
366	fpu_force_restore(fpu);	370	fpu_force_restore(fpu);
367		371
368	if (cpu_feature_enabled(X86_FEATURE_FXSR))	372	if (cpu_feature_enabled(X86_FEATURE_FXSR))
369	convert_to_fxsr(&fpu->state.fxsave, &env);	373	convert_to_fxsr(&fpu->fpstate->regs.fxsave, &env);
370	else	374	else
371	memcpy(&fpu->state.fsave, &env, sizeof(env));	375	memcpy(&fpu->fpstate->regs.fsave, &env, sizeof(env));
372		376
373	/*	377	/*
374	* Update the header bit in the xsave header, indicating the	378	* Update the header bit in the xsave header, indicating the
375	* presence of FP.	379	* presence of FP.
376	*/	380	*/
377	if (cpu_feature_enabled(X86_FEATURE_XSAVE))	381	if (cpu_feature_enabled(X86_FEATURE_XSAVE))
378	fpu->state.xsave.header.xfeatures \|= XFEATURE_MASK_FP;	382	fpu->fpstate->regs.xsave.header.xfeatures \|= XFEATURE_MASK_FP;
379		383
380	return 0;	384	return 0;
381	}	385	}

arch/x86/kernel/fpu/signal.c+145−140

@@ -7,23 +7,25 @@ · 6 unmodified lines
7	#include <linux/cpu.h>	7	#include <linux/cpu.h>
8	#include <linux/pagemap.h>	8	#include <linux/pagemap.h>
9		9
10	#include <asm/fpu/internal.h>
11	#include <asm/fpu/signal.h>	10	#include <asm/fpu/signal.h>
12	#include <asm/fpu/regset.h>	11	#include <asm/fpu/regset.h>
13	#include <asm/fpu/xstate.h>	12	#include <asm/fpu/xstate.h>
14		13
15	#include <asm/sigframe.h>	14	#include <asm/sigframe.h>
		15	#include <asm/trapnr.h>
16	#include <asm/trace/fpu.h>	16	#include <asm/trace/fpu.h>
17		17
18	static struct _fpx_sw_bytes fx_sw_reserved __ro_after_init;	18	#include "context.h"
19	static struct _fpx_sw_bytes fx_sw_reserved_ia32 __ro_after_init;	19	#include "internal.h"
		20	#include "legacy.h"
		21	#include "xstate.h"
20		22
21	/*	23	/*
22	* Check for the presence of extended state information in the	24	* Check for the presence of extended state information in the
23	* user fpstate pointer in the sigcontext.	25	* user fpstate pointer in the sigcontext.
24	*/	26	*/
25	static inline int check_xstate_in_sigframe(struct fxregs_state __user *fxbuf,	27	static inline bool check_xstate_in_sigframe(struct fxregs_state __user *fxbuf,
26	struct _fpx_sw_bytes *fx_sw)	28	struct _fpx_sw_bytes *fx_sw)
27	{	29	{
28	int min_xstate_size = sizeof(struct fxregs_state) +	30	int min_xstate_size = sizeof(struct fxregs_state) +
29	sizeof(struct xstate_header);	31	sizeof(struct xstate_header);
@@ -31,12 +33,12 @@ static inline int check_xstate_in_sigframe(struct fxregs_state __user *fxbuf, · 1 unmodified line
31	unsigned int magic2;	33	unsigned int magic2;
32		34
33	if (__copy_from_user(fx_sw, &fxbuf->sw_reserved[0], sizeof(*fx_sw)))	35	if (__copy_from_user(fx_sw, &fxbuf->sw_reserved[0], sizeof(*fx_sw)))
34	return -EFAULT;	36	return false;
35		37
36	/* Check for the first magic field and other error scenarios. */	38	/* Check for the first magic field and other error scenarios. */
37	if (fx_sw->magic1 != FP_XSTATE_MAGIC1 \|\|	39	if (fx_sw->magic1 != FP_XSTATE_MAGIC1 \|\|
38	fx_sw->xstate_size < min_xstate_size \|\|	40	fx_sw->xstate_size < min_xstate_size \|\|
39	fx_sw->xstate_size > fpu_user_xstate_size \|\|	41	fx_sw->xstate_size > current->thread.fpu.fpstate->user_size \|\|
40	fx_sw->xstate_size > fx_sw->extended_size)	42	fx_sw->xstate_size > fx_sw->extended_size)
41	goto setfx;	43	goto setfx;
42		44
@@ -47,10 +49,10 @@ static inline int check_xstate_in_sigframe(struct fxregs_state __user *fxbuf, · 4 unmodified lines
47	* in the memory layout.	49	* in the memory layout.
48	*/	50	*/
49	if (__get_user(magic2, (__u32 __user *)(fpstate + fx_sw->xstate_size)))	51	if (__get_user(magic2, (__u32 __user *)(fpstate + fx_sw->xstate_size)))
50	return -EFAULT;	52	return false;
51		53
52	if (likely(magic2 == FP_XSTATE_MAGIC2))	54	if (likely(magic2 == FP_XSTATE_MAGIC2))
53	return 0;	55	return true;
54	setfx:	56	setfx:
55	trace_x86_fpu_xstate_check_failed(&current->thread.fpu);	57	trace_x86_fpu_xstate_check_failed(&current->thread.fpu);
56		58
@@ -58,22 +60,22 @@ setfx: · 1 unmodified line
58	fx_sw->magic1 = 0;	60	fx_sw->magic1 = 0;
59	fx_sw->xstate_size = sizeof(struct fxregs_state);	61	fx_sw->xstate_size = sizeof(struct fxregs_state);
60	fx_sw->xfeatures = XFEATURE_MASK_FPSSE;	62	fx_sw->xfeatures = XFEATURE_MASK_FPSSE;
61	return 0;	63	return true;
62	}	64	}
63		65
64	/*	66	/*
65	* Signal frame handlers.	67	* Signal frame handlers.
66	*/	68	*/
67	static inline int save_fsave_header(struct task_struct tsk, void __user buf)	69	static inline bool save_fsave_header(struct task_struct tsk, void __user buf)
68	{	70	{
69	if (use_fxsr()) {	71	if (use_fxsr()) {
70	struct xregs_state *xsave = &tsk->thread.fpu.state.xsave;	72	struct xregs_state *xsave = &tsk->thread.fpu.fpstate->regs.xsave;
71	struct user_i387_ia32_struct env;	73	struct user_i387_ia32_struct env;
72	struct _fpstate_32 __user *fp = buf;	74	struct _fpstate_32 __user *fp = buf;
73		75
74	fpregs_lock();	76	fpregs_lock();
75	if (!test_thread_flag(TIF_NEED_FPU_LOAD))	77	if (!test_thread_flag(TIF_NEED_FPU_LOAD))
76	fxsave(&tsk->thread.fpu.state.fxsave);	78	fxsave(&tsk->thread.fpu.fpstate->regs.fxsave);
77	fpregs_unlock();	79	fpregs_unlock();
78		80
79	convert_from_fxsr(&env, tsk);	81	convert_from_fxsr(&env, tsk);
@@ -81,33 +83,54 @@ static inline int save_fsave_header(struct task_struct tsk, void __user buf) · 1 unmodified line
81	if (__copy_to_user(buf, &env, sizeof(env)) \|\|	83	if (__copy_to_user(buf, &env, sizeof(env)) \|\|
82	__put_user(xsave->i387.swd, &fp->status) \|\|	84	__put_user(xsave->i387.swd, &fp->status) \|\|
83	__put_user(X86_FXSR_MAGIC, &fp->magic))	85	__put_user(X86_FXSR_MAGIC, &fp->magic))
84	return -1;	86	return false;
85	} else {	87	} else {
86	struct fregs_state __user *fp = buf;	88	struct fregs_state __user *fp = buf;
87	u32 swd;	89	u32 swd;
		90
88	if (__get_user(swd, &fp->swd) \|\| __put_user(swd, &fp->status))	91	if (__get_user(swd, &fp->swd) \|\| __put_user(swd, &fp->status))
89	return -1;	92	return false;
90	}	93	}
91		94
92	return 0;	95	return true;
		96	}
		97
		98	/*
		99	* Prepare the SW reserved portion of the fxsave memory layout, indicating
		100	* the presence of the extended state information in the memory layout
		101	* pointed to by the fpstate pointer in the sigcontext.
		102	* This is saved when ever the FP and extended state context is
		103	* saved on the user stack during the signal handler delivery to the user.
		104	*/
		105	static inline void save_sw_bytes(struct _fpx_sw_bytes *sw_bytes, bool ia32_frame,
		106	struct fpstate *fpstate)
		107	{
		108	sw_bytes->magic1 = FP_XSTATE_MAGIC1;
		109	sw_bytes->extended_size = fpstate->user_size + FP_XSTATE_MAGIC2_SIZE;
		110	sw_bytes->xfeatures = fpstate->user_xfeatures;
		111	sw_bytes->xstate_size = fpstate->user_size;
		112
		113	if (ia32_frame)
		114	sw_bytes->extended_size += sizeof(struct fregs_state);
93	}	115	}
94		116
95	static inline int save_xstate_epilog(void __user *buf, int ia32_frame)	117	static inline bool save_xstate_epilog(void __user *buf, int ia32_frame,
		118	struct fpstate *fpstate)
96	{	119	{
97	struct xregs_state __user *x = buf;	120	struct xregs_state __user *x = buf;
98	struct _fpx_sw_bytes *sw_bytes;	121	struct _fpx_sw_bytes sw_bytes;
99	u32 xfeatures;	122	u32 xfeatures;
100	int err;	123	int err;
101		124
102	/* Setup the bytes not touched by the [f]xsave and reserved for SW. */	125	/* Setup the bytes not touched by the [f]xsave and reserved for SW. */
103	sw_bytes = ia32_frame ? &fx_sw_reserved_ia32 : &fx_sw_reserved;	126	save_sw_bytes(&sw_bytes, ia32_frame, fpstate);
104	err = __copy_to_user(&x->i387.sw_reserved, sw_bytes, sizeof(*sw_bytes));	127	err = __copy_to_user(&x->i387.sw_reserved, &sw_bytes, sizeof(sw_bytes));
105		128
106	if (!use_xsave())	129	if (!use_xsave())
107	return err;	130	return !err;
108		131
109	err \|= __put_user(FP_XSTATE_MAGIC2,	132	err \|= __put_user(FP_XSTATE_MAGIC2,
110	(__u32 __user *)(buf + fpu_user_xstate_size));	133	(__u32 __user *)(buf + fpstate->user_size));
111		134
112	/*	135	/*
113	* Read the xfeatures which we copied (directly from the cpu or	136	* Read the xfeatures which we copied (directly from the cpu or
@@ -130,23 +153,17 @@ static inline int save_xstate_epilog(void __user *buf, int ia32_frame) · 16 unmodified lines
130		153
131	err \|= __put_user(xfeatures, (__u32 __user *)&x->header.xfeatures);	154	err \|= __put_user(xfeatures, (__u32 __user *)&x->header.xfeatures);
132		155
133	return err;	156	return !err;
134	}	157	}
135		158
136	static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf)	159	static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf)
137	{	160	{
138	int err;
139
140	if (use_xsave())	161	if (use_xsave())
141	err = xsave_to_user_sigframe(buf);	162	return xsave_to_user_sigframe(buf);
142	else if (use_fxsr())	163	if (use_fxsr())
143	err = fxsave_to_user_sigframe((struct fxregs_state __user *) buf);	164	return fxsave_to_user_sigframe((struct fxregs_state __user *) buf);
144	else	165	else
145	err = fnsave_to_user_sigframe((struct fregs_state __user *) buf);	166	return fnsave_to_user_sigframe((struct fregs_state __user *) buf);
146
147	if (unlikely(err) && __clear_user(buf, fpu_user_xstate_size))
148	err = -EFAULT;
149	return err;
150	}	167	}
151		168
152	/*	169	/*
@@ -159,10 +176,8 @@ static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf) · 6 unmodified lines
159	* buf == buf_fx for 64-bit frames and 32-bit fsave frame.	176	* buf == buf_fx for 64-bit frames and 32-bit fsave frame.
160	* buf != buf_fx for 32-bit frames with fxstate.	177	* buf != buf_fx for 32-bit frames with fxstate.
161	*	178	*
162	* Try to save it directly to the user frame with disabled page fault handler.	179	* Save it directly to the user frame with disabled page fault handler. If
163	* If this fails then do the slow path where the FPU state is first saved to	180	* that faults, try to clear the frame which handles the page fault.
164	* task's fpu->state and then copy it to the user frame pointed to by the
165	* aligned pointer 'buf_fx'.
166	*	181	*
167	* If this is a 32-bit frame with fxstate, put a fsave header before	182	* If this is a 32-bit frame with fxstate, put a fsave header before
168	* the aligned state at 'buf_fx'.	183	* the aligned state at 'buf_fx'.
@@ -170,10 +185,11 @@ static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf) · 1 unmodified line
170	* For [f]xsave state, update the SW reserved fields in the [f]xsave frame	185	* For [f]xsave state, update the SW reserved fields in the [f]xsave frame
171	* indicating the absence/presence of the extended state to the user.	186	* indicating the absence/presence of the extended state to the user.
172	*/	187	*/
173	int copy_fpstate_to_sigframe(void __user buf, void __user buf_fx, int size)	188	bool copy_fpstate_to_sigframe(void __user buf, void __user buf_fx, int size)
174	{	189	{
175	struct task_struct *tsk = current;	190	struct task_struct *tsk = current;
176	int ia32_fxstate = (buf != buf_fx);	191	struct fpstate *fpstate = tsk->thread.fpu.fpstate;
		192	bool ia32_fxstate = (buf != buf_fx);
177	int ret;	193	int ret;
178		194
179	ia32_fxstate &= (IS_ENABLED(CONFIG_X86_32) \|\|	195	ia32_fxstate &= (IS_ENABLED(CONFIG_X86_32) \|\|
@@ -181,13 +197,25 @@ int copy_fpstate_to_sigframe(void __user buf, void __user buf_fx, int size) · 1 unmodified line
181		197
182	if (!static_cpu_has(X86_FEATURE_FPU)) {	198	if (!static_cpu_has(X86_FEATURE_FPU)) {
183	struct user_i387_ia32_struct fp;	199	struct user_i387_ia32_struct fp;
		200
184	fpregs_soft_get(current, NULL, (struct membuf){.p = &fp,	201	fpregs_soft_get(current, NULL, (struct membuf){.p = &fp,
185	.left = sizeof(fp)});	202	.left = sizeof(fp)});
186	return copy_to_user(buf, &fp, sizeof(fp)) ? -EFAULT : 0;	203	return !copy_to_user(buf, &fp, sizeof(fp));
187	}	204	}
188		205
189	if (!access_ok(buf, size))	206	if (!access_ok(buf, size))
190	return -EACCES;	207	return false;
		208
		209	if (use_xsave()) {
		210	struct xregs_state __user *xbuf = buf_fx;
		211
		212	/*
		213	* Clear the xsave header first, so that reserved fields are
		214	* initialized to zero.
		215	*/
		216	if (__clear_user(&xbuf->header, sizeof(xbuf->header)))
		217	return false;
		218	}
191	retry:	219	retry:
192	/*	220	/*
193	* Load the FPU registers if they are not valid for the current task.	221	* Load the FPU registers if they are not valid for the current task.
@@ -205,26 +233,26 @@ retry: · 11 unmodified lines
205	fpregs_unlock();	233	fpregs_unlock();
206		234
207	if (ret) {	235	if (ret) {
208	if (!fault_in_pages_writeable(buf_fx, fpu_user_xstate_size))	236	if (!__clear_user(buf_fx, fpstate->user_size))
209	goto retry;	237	goto retry;
210	return -EFAULT;	238	return false;
211	}	239	}
212		240
213	/* Save the fsave header for the 32-bit frames. */	241	/* Save the fsave header for the 32-bit frames. */
214	if ((ia32_fxstate \|\| !use_fxsr()) && save_fsave_header(tsk, buf))	242	if ((ia32_fxstate \|\| !use_fxsr()) && !save_fsave_header(tsk, buf))
215	return -1;	243	return false;
216		244
217	if (use_fxsr() && save_xstate_epilog(buf_fx, ia32_fxstate))	245	if (use_fxsr() && !save_xstate_epilog(buf_fx, ia32_fxstate, fpstate))
218	return -1;	246	return false;
219		247
220	return 0;	248	return true;
221	}	249	}
222		250
223	static int __restore_fpregs_from_user(void __user *buf, u64 xrestore,	251	static int __restore_fpregs_from_user(void __user *buf, u64 ufeatures,
224	bool fx_only)	252	u64 xrestore, bool fx_only)
225	{	253	{
226	if (use_xsave()) {	254	if (use_xsave()) {
227	u64 init_bv = xfeatures_mask_uabi() & ~xrestore;	255	u64 init_bv = ufeatures & ~xrestore;
228	int ret;	256	int ret;
229		257
230	if (likely(!fx_only))	258	if (likely(!fx_only))
@@ -233,7 +261,7 @@ static int __restore_fpregs_from_user(void __user *buf, u64 xrestore, · 2 unmodified lines
233	ret = fxrstor_from_user_sigframe(buf);	261	ret = fxrstor_from_user_sigframe(buf);
234		262
235	if (!ret && unlikely(init_bv))	263	if (!ret && unlikely(init_bv))
236	os_xrstor(&init_fpstate.xsave, init_bv);	264	os_xrstor(&init_fpstate, init_bv);
237	return ret;	265	return ret;
238	} else if (use_fxsr()) {	266	} else if (use_fxsr()) {
239	return fxrstor_from_user_sigframe(buf);	267	return fxrstor_from_user_sigframe(buf);
@@ -246,16 +274,19 @@ static int __restore_fpregs_from_user(void __user *buf, u64 xrestore, · 6 unmodified lines
246	* Attempt to restore the FPU registers directly from user memory.	274	* Attempt to restore the FPU registers directly from user memory.
247	* Pagefaults are handled and any errors returned are fatal.	275	* Pagefaults are handled and any errors returned are fatal.
248	*/	276	*/
249	static int restore_fpregs_from_user(void __user *buf, u64 xrestore,	277	static bool restore_fpregs_from_user(void __user *buf, u64 xrestore,
250	bool fx_only, unsigned int size)	278	bool fx_only, unsigned int size)
251	{	279	{
252	struct fpu *fpu = &current->thread.fpu;	280	struct fpu *fpu = &current->thread.fpu;
253	int ret;	281	int ret;
254		282
255	retry:	283	retry:
256	fpregs_lock();	284	fpregs_lock();
		285	/* Ensure that XFD is up to date */
		286	xfd_update_state(fpu->fpstate);
257	pagefault_disable();	287	pagefault_disable();
258	ret = __restore_fpregs_from_user(buf, xrestore, fx_only);	288	ret = __restore_fpregs_from_user(buf, fpu->fpstate->user_xfeatures,
		289	xrestore, fx_only);
259	pagefault_enable();	290	pagefault_enable();
260		291
261	if (unlikely(ret)) {	292	if (unlikely(ret)) {
@@ -275,13 +306,12 @@ retry: · 13 unmodified lines
275	fpregs_unlock();	306	fpregs_unlock();
276		307
277	/* Try to handle #PF, but anything else is fatal. */	308	/* Try to handle #PF, but anything else is fatal. */
278	if (ret != -EFAULT)	309	if (ret != X86_TRAP_PF)
279	return -EINVAL;	310	return false;
280		311
281	ret = fault_in_pages_readable(buf, size);	312	if (!fault_in_pages_readable(buf, size))
282	if (!ret)
283	goto retry;	313	goto retry;
284	return ret;	314	return false;
285	}	315	}
286		316
287	/*	317	/*
@@ -294,45 +324,40 @@ retry: · 6 unmodified lines
294	* been restored from a user buffer directly.	324	* been restored from a user buffer directly.
295	*/	325	*/
296	if (test_thread_flag(TIF_NEED_FPU_LOAD) && xfeatures_mask_supervisor())	326	if (test_thread_flag(TIF_NEED_FPU_LOAD) && xfeatures_mask_supervisor())
297	os_xrstor(&fpu->state.xsave, xfeatures_mask_supervisor());	327	os_xrstor_supervisor(fpu->fpstate);
298		328
299	fpregs_mark_activate();	329	fpregs_mark_activate();
300	fpregs_unlock();	330	fpregs_unlock();
301	return 0;	331	return true;
302	}	332	}
303		333
304	static int __fpu_restore_sig(void __user buf, void __user buf_fx,	334	static bool __fpu_restore_sig(void __user buf, void __user buf_fx,
305	bool ia32_fxstate)	335	bool ia32_fxstate)
306	{	336	{
307	int state_size = fpu_kernel_xstate_size;
308	struct task_struct *tsk = current;	337	struct task_struct *tsk = current;
309	struct fpu *fpu = &tsk->thread.fpu;	338	struct fpu *fpu = &tsk->thread.fpu;
310	struct user_i387_ia32_struct env;	339	struct user_i387_ia32_struct env;
		340	bool success, fx_only = false;
		341	union fpregs_state *fpregs;
		342	unsigned int state_size;
311	u64 user_xfeatures = 0;	343	u64 user_xfeatures = 0;
312	bool fx_only = false;
313	int ret;
314		344
315	if (use_xsave()) {	345	if (use_xsave()) {
316	struct _fpx_sw_bytes fx_sw_user;	346	struct _fpx_sw_bytes fx_sw_user;
317		347
318	ret = check_xstate_in_sigframe(buf_fx, &fx_sw_user);	348	if (!check_xstate_in_sigframe(buf_fx, &fx_sw_user))
319	if (unlikely(ret))	349	return false;
320	return ret;
321		350
322	fx_only = !fx_sw_user.magic1;	351	fx_only = !fx_sw_user.magic1;
323	state_size = fx_sw_user.xstate_size;	352	state_size = fx_sw_user.xstate_size;
324	user_xfeatures = fx_sw_user.xfeatures;	353	user_xfeatures = fx_sw_user.xfeatures;
325	} else {	354	} else {
326	user_xfeatures = XFEATURE_MASK_FPSSE;	355	user_xfeatures = XFEATURE_MASK_FPSSE;
		356	state_size = fpu->fpstate->user_size;
327	}	357	}
328		358
329	if (likely(!ia32_fxstate)) {	359	if (likely(!ia32_fxstate)) {
330	/*	360	/* Restore the FPU registers directly from user memory. */
331	* Attempt to restore the FPU registers directly from user
332	* memory. For that to succeed, the user access cannot cause page
333	* faults. If it does, fall back to the slow path below, going
334	* through the kernel buffer with the enabled pagefault handler.
335	*/
336	return restore_fpregs_from_user(buf_fx, user_xfeatures, fx_only,	361	return restore_fpregs_from_user(buf_fx, user_xfeatures, fx_only,
337	state_size);	362	state_size);
338	}	363	}
@@ -342,9 +367,8 @@ static int __fpu_restore_sig(void __user buf, void __user buf_fx, · 3 unmodified lines
342	* to be ignored for histerical raisins. The legacy state is folded	367	* to be ignored for histerical raisins. The legacy state is folded
343	* in once the larger state has been copied.	368	* in once the larger state has been copied.
344	*/	369	*/
345	ret = __copy_from_user(&env, buf, sizeof(env));	370	if (__copy_from_user(&env, buf, sizeof(env)))
346	if (ret)	371	return false;
347	return ret;
348		372
349	/*	373	/*
350	* By setting TIF_NEED_FPU_LOAD it is ensured that our xstate is	374	* By setting TIF_NEED_FPU_LOAD it is ensured that our xstate is
@@ -363,38 +387,38 @@ static int __fpu_restore_sig(void __user buf, void __user buf_fx, · 12 unmodified lines
363	* the right place in memory. It's ia32 mode. Shrug.	387	* the right place in memory. It's ia32 mode. Shrug.
364	*/	388	*/
365	if (xfeatures_mask_supervisor())	389	if (xfeatures_mask_supervisor())
366	os_xsave(&fpu->state.xsave);	390	os_xsave(fpu->fpstate);
367	set_thread_flag(TIF_NEED_FPU_LOAD);	391	set_thread_flag(TIF_NEED_FPU_LOAD);
368	}	392	}
369	__fpu_invalidate_fpregs_state(fpu);	393	__fpu_invalidate_fpregs_state(fpu);
370	__cpu_invalidate_fpregs_state();	394	__cpu_invalidate_fpregs_state();
371	fpregs_unlock();	395	fpregs_unlock();
372		396
		397	fpregs = &fpu->fpstate->regs;
373	if (use_xsave() && !fx_only) {	398	if (use_xsave() && !fx_only) {
374	ret = copy_sigframe_from_user_to_xstate(&fpu->state.xsave, buf_fx);	399	if (copy_sigframe_from_user_to_xstate(fpu->fpstate, buf_fx))
375	if (ret)	400	return false;
376	return ret;
377	} else {	401	} else {
378	if (__copy_from_user(&fpu->state.fxsave, buf_fx,	402	if (__copy_from_user(&fpregs->fxsave, buf_fx,
379	sizeof(fpu->state.fxsave)))	403	sizeof(fpregs->fxsave)))
380	return -EFAULT;	404	return false;
381		405
382	if (IS_ENABLED(CONFIG_X86_64)) {	406	if (IS_ENABLED(CONFIG_X86_64)) {
383	/* Reject invalid MXCSR values. */	407	/* Reject invalid MXCSR values. */
384	if (fpu->state.fxsave.mxcsr & ~mxcsr_feature_mask)	408	if (fpregs->fxsave.mxcsr & ~mxcsr_feature_mask)
385	return -EINVAL;	409	return false;
386	} else {	410	} else {
387	/* Mask invalid bits out for historical reasons (broken hardware). */	411	/* Mask invalid bits out for historical reasons (broken hardware). */
388	fpu->state.fxsave.mxcsr &= mxcsr_feature_mask;	412	fpregs->fxsave.mxcsr &= mxcsr_feature_mask;
389	}	413	}
390		414
391	/* Enforce XFEATURE_MASK_FPSSE when XSAVE is enabled */	415	/* Enforce XFEATURE_MASK_FPSSE when XSAVE is enabled */
392	if (use_xsave())	416	if (use_xsave())
393	fpu->state.xsave.header.xfeatures \|= XFEATURE_MASK_FPSSE;	417	fpregs->xsave.header.xfeatures \|= XFEATURE_MASK_FPSSE;
394	}	418	}
395		419
396	/* Fold the legacy FP storage */	420	/* Fold the legacy FP storage */
397	convert_to_fxsr(&fpu->state.fxsave, &env);	421	convert_to_fxsr(&fpregs->fxsave, &env);
398		422
399	fpregs_lock();	423	fpregs_lock();
400	if (use_xsave()) {	424	if (use_xsave()) {
@@ -409,40 +433,45 @@ static int __fpu_restore_sig(void __user buf, void __user buf_fx, · 8 unmodified lines
409	*/	433	*/
410	u64 mask = user_xfeatures \| xfeatures_mask_supervisor();	434	u64 mask = user_xfeatures \| xfeatures_mask_supervisor();
411		435
412	fpu->state.xsave.header.xfeatures &= mask;	436	fpregs->xsave.header.xfeatures &= mask;
413	ret = os_xrstor_safe(&fpu->state.xsave, xfeatures_mask_all);	437	success = !os_xrstor_safe(fpu->fpstate,
		438	fpu_kernel_cfg.max_features);
414	} else {	439	} else {
415	ret = fxrstor_safe(&fpu->state.fxsave);	440	success = !fxrstor_safe(&fpregs->fxsave);
416	}	441	}
417		442
418	if (likely(!ret))	443	if (likely(success))
419	fpregs_mark_activate();	444	fpregs_mark_activate();
420		445
421	fpregs_unlock();	446	fpregs_unlock();
422	return ret;	447	return success;
423	}	448	}
424	static inline int xstate_sigframe_size(void)	449
		450	static inline unsigned int xstate_sigframe_size(struct fpstate *fpstate)
425	{	451	{
426	return use_xsave() ? fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE :	452	unsigned int size = fpstate->user_size;
427	fpu_user_xstate_size;	453
		454	return use_xsave() ? size + FP_XSTATE_MAGIC2_SIZE : size;
428	}	455	}
429		456
430	/*	457	/*
431	* Restore FPU state from a sigframe:	458	* Restore FPU state from a sigframe:
432	*/	459	*/
433	int fpu__restore_sig(void __user *buf, int ia32_frame)	460	bool fpu__restore_sig(void __user *buf, int ia32_frame)
434	{	461	{
435	unsigned int size = xstate_sigframe_size();
436	struct fpu *fpu = &current->thread.fpu;	462	struct fpu *fpu = &current->thread.fpu;
437	void __user *buf_fx = buf;	463	void __user *buf_fx = buf;
438	bool ia32_fxstate = false;	464	bool ia32_fxstate = false;
439	int ret;	465	bool success = false;
		466	unsigned int size;
440		467
441	if (unlikely(!buf)) {	468	if (unlikely(!buf)) {
442	fpu__clear_user_states(fpu);	469	fpu__clear_user_states(fpu);
443	return 0;	470	return true;
444	}	471	}
445		472
		473	size = xstate_sigframe_size(fpu->fpstate);
		474
446	ia32_frame &= (IS_ENABLED(CONFIG_X86_32) \|\|	475	ia32_frame &= (IS_ENABLED(CONFIG_X86_32) \|\|
447	IS_ENABLED(CONFIG_IA32_EMULATION));	476	IS_ENABLED(CONFIG_IA32_EMULATION));
448		477
@@ -456,30 +485,28 @@ int fpu__restore_sig(void __user *buf, int ia32_frame) · 7 unmodified lines
456	ia32_fxstate = true;	485	ia32_fxstate = true;
457	}	486	}
458		487
459	if (!access_ok(buf, size)) {	488	if (!access_ok(buf, size))
460	ret = -EACCES;
461	goto out;	489	goto out;
462	}
463		490
464	if (!IS_ENABLED(CONFIG_X86_64) && !cpu_feature_enabled(X86_FEATURE_FPU)) {	491	if (!IS_ENABLED(CONFIG_X86_64) && !cpu_feature_enabled(X86_FEATURE_FPU)) {
465	ret = fpregs_soft_set(current, NULL, 0,	492	success = !fpregs_soft_set(current, NULL, 0,
466	sizeof(struct user_i387_ia32_struct),	493	sizeof(struct user_i387_ia32_struct),
467	NULL, buf);	494	NULL, buf);
468	} else {	495	} else {
469	ret = __fpu_restore_sig(buf, buf_fx, ia32_fxstate);	496	success = __fpu_restore_sig(buf, buf_fx, ia32_fxstate);
470	}	497	}
471		498
472	out:	499	out:
473	if (unlikely(ret))	500	if (unlikely(!success))
474	fpu__clear_user_states(fpu);	501	fpu__clear_user_states(fpu);
475	return ret;	502	return success;
476	}	503	}
477		504
478	unsigned long	505	unsigned long
479	fpu__alloc_mathframe(unsigned long sp, int ia32_frame,	506	fpu__alloc_mathframe(unsigned long sp, int ia32_frame,
480	unsigned long buf_fx, unsigned long size)	507	unsigned long buf_fx, unsigned long size)
481	{	508	{
482	unsigned long frame_size = xstate_sigframe_size();	509	unsigned long frame_size = xstate_sigframe_size(current->thread.fpu.fpstate);
483		510
484	*buf_fx = sp = round_down(sp - frame_size, 64);	511	*buf_fx = sp = round_down(sp - frame_size, 64);
485	if (ia32_frame && use_fxsr()) {	512	if (ia32_frame && use_fxsr()) {
@@ -492,9 +519,12 @@ fpu__alloc_mathframe(unsigned long sp, int ia32_frame, · 6 unmodified lines
492	return sp;	519	return sp;
493	}	520	}
494		521
495	unsigned long fpu__get_fpstate_size(void)	522	unsigned long __init fpu__get_fpstate_size(void)
496	{	523	{
497	unsigned long ret = xstate_sigframe_size();	524	unsigned long ret = fpu_user_cfg.max_size;
		525
		526	if (use_xsave())
		527	ret += FP_XSTATE_MAGIC2_SIZE;
498		528
499	/*	529	/*
500	* This space is needed on (most) 32-bit kernels, or when a 32-bit	530	* This space is needed on (most) 32-bit kernels, or when a 32-bit
@@ -510,28 +540,3 @@ unsigned long fpu__get_fpstate_size(void) · 9 unmodified lines
510	return ret;	540	return ret;
511	}	541	}
512		542
513	/*
514	* Prepare the SW reserved portion of the fxsave memory layout, indicating
515	* the presence of the extended state information in the memory layout
516	* pointed by the fpstate pointer in the sigcontext.
517	* This will be saved when ever the FP and extended state context is
518	* saved on the user stack during the signal handler delivery to the user.
519	*/
520	void fpu__init_prepare_fx_sw_frame(void)
521	{
522	int size = fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE;
523
524	fx_sw_reserved.magic1 = FP_XSTATE_MAGIC1;
525	fx_sw_reserved.extended_size = size;
526	fx_sw_reserved.xfeatures = xfeatures_mask_uabi();
527	fx_sw_reserved.xstate_size = fpu_user_xstate_size;
528
529	if (IS_ENABLED(CONFIG_IA32_EMULATION) \|\|
530	IS_ENABLED(CONFIG_X86_32)) {
531	int fsave_header_size = sizeof(struct fregs_state);
532
533	fx_sw_reserved_ia32 = fx_sw_reserved;
534	fx_sw_reserved_ia32.extended_size = size + fsave_header_size;
535	}
536	}
537

arch/x86/kernel/fpu/xstate.c+712−186

@@ -4,21 +4,33 @@ · 3 unmodified lines
4	*	4	*
5	* Author: Suresh Siddha <suresh.b.siddha@intel.com>	5	* Author: Suresh Siddha <suresh.b.siddha@intel.com>
6	*/	6	*/
		7	#include <linux/bitops.h>
7	#include <linux/compat.h>	8	#include <linux/compat.h>
8	#include <linux/cpu.h>	9	#include <linux/cpu.h>
9	#include <linux/mman.h>	10	#include <linux/mman.h>
		11	#include <linux/nospec.h>
10	#include <linux/pkeys.h>	12	#include <linux/pkeys.h>
11	#include <linux/seq_file.h>	13	#include <linux/seq_file.h>
12	#include <linux/proc_fs.h>	14	#include <linux/proc_fs.h>
		15	#include <linux/vmalloc.h>
13		16
14	#include <asm/fpu/api.h>	17	#include <asm/fpu/api.h>
15	#include <asm/fpu/internal.h>
16	#include <asm/fpu/signal.h>
17	#include <asm/fpu/regset.h>	18	#include <asm/fpu/regset.h>
18	#include <asm/fpu/xstate.h>	19	#include <asm/fpu/signal.h>
		20	#include <asm/fpu/xcr.h>
19		21
20	#include <asm/tlbflush.h>	22	#include <asm/tlbflush.h>
21	#include <asm/cpufeature.h>	23	#include <asm/prctl.h>
		24	#include <asm/elf.h>
		25
		26	#include "context.h"
		27	#include "internal.h"
		28	#include "legacy.h"
		29	#include "xstate.h"
		30
		31	#define for_each_extended_xfeature(bit, mask) \
		32	(bit) = FIRST_EXTENDED_XFEATURE; \
		33	for_each_set_bit_from(bit, (unsigned long )&(mask), 8 sizeof(mask))
22		34
23	/*	35	/*
24	* Although we spell it out in here, the Processor Trace	36	* Although we spell it out in here, the Processor Trace
@@ -39,29 +51,32 @@ static const char *xfeature_names[] = · 14 unmodified lines
39	"Protection Keys User registers",	51	"Protection Keys User registers",
40	"PASID state",	52	"PASID state",
41	"unknown xstate feature" ,	53	"unknown xstate feature" ,
		54	"unknown xstate feature" ,
		55	"unknown xstate feature" ,
		56	"unknown xstate feature" ,
		57	"unknown xstate feature" ,
		58	"unknown xstate feature" ,
		59	"AMX Tile config" ,
		60	"AMX Tile data" ,
		61	"unknown xstate feature" ,
42	};	62	};
43		63
44	static short xsave_cpuid_features[] __initdata = {	64	static unsigned short xsave_cpuid_features[] __initdata = {
45	X86_FEATURE_FPU,	65	[XFEATURE_FP] = X86_FEATURE_FPU,
46	X86_FEATURE_XMM,	66	[XFEATURE_SSE] = X86_FEATURE_XMM,
47	X86_FEATURE_AVX,	67	[XFEATURE_YMM] = X86_FEATURE_AVX,
48	X86_FEATURE_MPX,	68	[XFEATURE_BNDREGS] = X86_FEATURE_MPX,
49	X86_FEATURE_MPX,	69	[XFEATURE_BNDCSR] = X86_FEATURE_MPX,
50	X86_FEATURE_AVX512F,	70	[XFEATURE_OPMASK] = X86_FEATURE_AVX512F,
51	X86_FEATURE_AVX512F,	71	[XFEATURE_ZMM_Hi256] = X86_FEATURE_AVX512F,
52	X86_FEATURE_AVX512F,	72	[XFEATURE_Hi16_ZMM] = X86_FEATURE_AVX512F,
53	X86_FEATURE_INTEL_PT,	73	[XFEATURE_PT_UNIMPLEMENTED_SO_FAR] = X86_FEATURE_INTEL_PT,
54	X86_FEATURE_PKU,	74	[XFEATURE_PKRU] = X86_FEATURE_PKU,
55	X86_FEATURE_ENQCMD,	75	[XFEATURE_PASID] = X86_FEATURE_ENQCMD,
		76	[XFEATURE_XTILE_CFG] = X86_FEATURE_AMX_TILE,
		77	[XFEATURE_XTILE_DATA] = X86_FEATURE_AMX_TILE,
56	};	78	};
57		79
58	/*
59	* This represents the full set of bits that should ever be set in a kernel
60	* XSAVE buffer, both supervisor and user xstates.
61	*/
62	u64 xfeatures_mask_all __ro_after_init;
63	EXPORT_SYMBOL_GPL(xfeatures_mask_all);
64
65	static unsigned int xstate_offsets[XFEATURE_MAX] __ro_after_init =	80	static unsigned int xstate_offsets[XFEATURE_MAX] __ro_after_init =
66	{ [ 0 ... XFEATURE_MAX - 1] = -1};	81	{ [ 0 ... XFEATURE_MAX - 1] = -1};
67	static unsigned int xstate_sizes[XFEATURE_MAX] __ro_after_init =	82	static unsigned int xstate_sizes[XFEATURE_MAX] __ro_after_init =
@@ -71,13 +86,6 @@ static unsigned int xstate_comp_offsets[XFEATURE_MAX] __ro_after_init = · 3 unmodified lines
71	static unsigned int xstate_supervisor_only_offsets[XFEATURE_MAX] __ro_after_init =	86	static unsigned int xstate_supervisor_only_offsets[XFEATURE_MAX] __ro_after_init =
72	{ [ 0 ... XFEATURE_MAX - 1] = -1};	87	{ [ 0 ... XFEATURE_MAX - 1] = -1};
73		88
74	/*
75	* The XSAVE area of kernel can be in standard or compacted format;
76	* it is always in standard format for user mode. This is the user
77	* mode standard format size used for signal and ptrace frames.
78	*/
79	unsigned int fpu_user_xstate_size __ro_after_init;
80
81	/*	89	/*
82	* Return whether the system supports a given xfeature.	90	* Return whether the system supports a given xfeature.
83	*	91	*
@@ -85,7 +93,7 @@ unsigned int fpu_user_xstate_size __ro_after_init; · 1 unmodified line
85	*/	93	*/
86	int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name)	94	int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name)
87	{	95	{
88	u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_all;	96	u64 xfeatures_missing = xfeatures_needed & ~fpu_kernel_cfg.max_features;
89		97
90	if (unlikely(feature_name)) {	98	if (unlikely(feature_name)) {
91	long xfeature_idx, max_idx;	99	long xfeature_idx, max_idx;
@@ -135,17 +143,26 @@ static bool xfeature_is_supervisor(int xfeature_nr) · 43 unmodified lines
135	*/	143	*/
136	void fpu__init_cpu_xstate(void)	144	void fpu__init_cpu_xstate(void)
137	{	145	{
138	if (!boot_cpu_has(X86_FEATURE_XSAVE) \|\| !xfeatures_mask_all)	146	if (!boot_cpu_has(X86_FEATURE_XSAVE) \|\| !fpu_kernel_cfg.max_features)
139	return;	147	return;
140		148
141	cr4_set_bits(X86_CR4_OSXSAVE);	149	cr4_set_bits(X86_CR4_OSXSAVE);
142		150
		151	/*
		152	* Must happen after CR4 setup and before xsetbv() to allow KVM
		153	* lazy passthrough. Write independent of the dynamic state static
		154	* key as that does not work on the boot CPU. This also ensures
		155	* that any stale state is wiped out from XFD.
		156	*/
		157	if (cpu_feature_enabled(X86_FEATURE_XFD))
		158	wrmsrl(MSR_IA32_XFD, init_fpstate.xfd);
		159
143	/*	160	/*
144	* XCR_XFEATURE_ENABLED_MASK (aka. XCR0) sets user features	161	* XCR_XFEATURE_ENABLED_MASK (aka. XCR0) sets user features
145	* managed by XSAVE{C, OPT, S} and XRSTOR{S}. Only XSAVE user	162	* managed by XSAVE{C, OPT, S} and XRSTOR{S}. Only XSAVE user
146	* states can be set here.	163	* states can be set here.
147	*/	164	*/
148	xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_uabi());	165	xsetbv(XCR_XFEATURE_ENABLED_MASK, fpu_user_cfg.max_features);
149		166
150	/*	167	/*
151	* MSR_IA32_XSS sets supervisor states managed by XSAVES.	168	* MSR_IA32_XSS sets supervisor states managed by XSAVES.
@@ -158,7 +175,7 @@ void fpu__init_cpu_xstate(void) · 6 unmodified lines
158		175
159	static bool xfeature_enabled(enum xfeature xfeature)	176	static bool xfeature_enabled(enum xfeature xfeature)
160	{	177	{
161	return xfeatures_mask_all & BIT_ULL(xfeature);	178	return fpu_kernel_cfg.max_features & BIT_ULL(xfeature);
162	}	179	}
163		180
164	/*	181	/*
@@ -184,10 +201,7 @@ static void __init setup_xstate_features(void) · 19 unmodified lines
184	xstate_sizes[XFEATURE_SSE] = sizeof_field(struct fxregs_state,	201	xstate_sizes[XFEATURE_SSE] = sizeof_field(struct fxregs_state,
185	xmm_space);	202	xmm_space);
186		203
187	for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {	204	for_each_extended_xfeature(i, fpu_kernel_cfg.max_features) {
188	if (!xfeature_enabled(i))
189	continue;
190
191	cpuid_count(XSTATE_CPUID, i, &eax, &ebx, &ecx, &edx);	205	cpuid_count(XSTATE_CPUID, i, &eax, &ebx, &ecx, &edx);
192		206
193	xstate_sizes[i] = eax;	207	xstate_sizes[i] = eax;
@@ -236,6 +250,8 @@ static void __init print_xstate_features(void) · 42 unmodified lines
236	print_xstate_feature(XFEATURE_MASK_Hi16_ZMM);	250	print_xstate_feature(XFEATURE_MASK_Hi16_ZMM);
237	print_xstate_feature(XFEATURE_MASK_PKRU);	251	print_xstate_feature(XFEATURE_MASK_PKRU);
238	print_xstate_feature(XFEATURE_MASK_PASID);	252	print_xstate_feature(XFEATURE_MASK_PASID);
		253	print_xstate_feature(XFEATURE_MASK_XTILE_CFG);
		254	print_xstate_feature(XFEATURE_MASK_XTILE_DATA);
239	}	255	}
240		256
241	/*	257	/*
@@ -291,20 +307,15 @@ static void __init setup_xstate_comp_offsets(void) · 49 unmodified lines
291	xstate_comp_offsets[XFEATURE_SSE] = offsetof(struct fxregs_state,	307	xstate_comp_offsets[XFEATURE_SSE] = offsetof(struct fxregs_state,
292	xmm_space);	308	xmm_space);
293		309
294	if (!boot_cpu_has(X86_FEATURE_XSAVES)) {	310	if (!cpu_feature_enabled(X86_FEATURE_XSAVES)) {
295	for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {	311	for_each_extended_xfeature(i, fpu_kernel_cfg.max_features)
296	if (xfeature_enabled(i))	312	xstate_comp_offsets[i] = xstate_offsets[i];
297	xstate_comp_offsets[i] = xstate_offsets[i];
298	}
299	return;	313	return;
300	}	314	}
301		315
302	next_offset = FXSAVE_SIZE + XSAVE_HDR_SIZE;	316	next_offset = FXSAVE_SIZE + XSAVE_HDR_SIZE;
303		317
304	for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {	318	for_each_extended_xfeature(i, fpu_kernel_cfg.max_features) {
305	if (!xfeature_enabled(i))
306	continue;
307
308	if (xfeature_is_aligned(i))	319	if (xfeature_is_aligned(i))
309	next_offset = ALIGN(next_offset, 64);	320	next_offset = ALIGN(next_offset, 64);
310		321
@@ -328,8 +339,8 @@ static void __init setup_supervisor_only_offsets(void) · 17 unmodified lines
328		339
329	next_offset = FXSAVE_SIZE + XSAVE_HDR_SIZE;	340	next_offset = FXSAVE_SIZE + XSAVE_HDR_SIZE;
330		341
331	for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {	342	for_each_extended_xfeature(i, fpu_kernel_cfg.max_features) {
332	if (!xfeature_enabled(i) \|\| !xfeature_is_supervisor(i))	343	if (!xfeature_is_supervisor(i))
333	continue;	344	continue;
334		345
335	if (xfeature_is_aligned(i))	346	if (xfeature_is_aligned(i))
@@ -347,14 +358,35 @@ static void __init print_xstate_offset_size(void) · 11 unmodified lines
347	{	358	{
348	int i;	359	int i;
349		360
350	for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {	361	for_each_extended_xfeature(i, fpu_kernel_cfg.max_features) {
351	if (!xfeature_enabled(i))
352	continue;
353	pr_info("x86/fpu: xstate_offset[%d]: %4d, xstate_sizes[%d]: %4d\n",	362	pr_info("x86/fpu: xstate_offset[%d]: %4d, xstate_sizes[%d]: %4d\n",
354	i, xstate_comp_offsets[i], i, xstate_sizes[i]);	363	i, xstate_comp_offsets[i], i, xstate_sizes[i]);
355	}	364	}
356	}	365	}
357		366
		367	/*
		368	* This function is called only during boot time when x86 caps are not set
		369	* up and alternative can not be used yet.
		370	*/
		371	static __init void os_xrstor_booting(struct xregs_state *xstate)
		372	{
		373	u64 mask = fpu_kernel_cfg.max_features & XFEATURE_MASK_FPSTATE;
		374	u32 lmask = mask;
		375	u32 hmask = mask >> 32;
		376	int err;
		377
		378	if (cpu_feature_enabled(X86_FEATURE_XSAVES))
		379	XSTATE_OP(XRSTORS, xstate, lmask, hmask, err);
		380	else
		381	XSTATE_OP(XRSTOR, xstate, lmask, hmask, err);
		382
		383	/*
		384	* We should never fault when copying from a kernel buffer, and the FPU
		385	* state we set at boot time should be valid.
		386	*/
		387	WARN_ON_FPU(err);
		388	}
		389
358	/*	390	/*
359	* All supported features have either init state all zeros or are	391	* All supported features have either init state all zeros or are
360	* handled in setup_init_fpu() individually. This is an explicit	392	* handled in setup_init_fpu() individually. This is an explicit
@@ -372,36 +404,30 @@ static void __init print_xstate_offset_size(void) · 11 unmodified lines
372	XFEATURE_MASK_PKRU \| \	404	XFEATURE_MASK_PKRU \| \
373	XFEATURE_MASK_BNDREGS \| \	405	XFEATURE_MASK_BNDREGS \| \
374	XFEATURE_MASK_BNDCSR \| \	406	XFEATURE_MASK_BNDCSR \| \
375	XFEATURE_MASK_PASID)	407	XFEATURE_MASK_PASID \| \
		408	XFEATURE_MASK_XTILE)
376		409
377	/*	410	/*
378	* setup the xstate image representing the init state	411	* setup the xstate image representing the init state
379	*/	412	*/
380	static void __init setup_init_fpu_buf(void)	413	static void __init setup_init_fpu_buf(void)
381	{	414	{
382	static int on_boot_cpu __initdata = 1;
383
384	BUILD_BUG_ON((XFEATURE_MASK_USER_SUPPORTED \|	415	BUILD_BUG_ON((XFEATURE_MASK_USER_SUPPORTED \|
385	XFEATURE_MASK_SUPERVISOR_SUPPORTED) !=	416	XFEATURE_MASK_SUPERVISOR_SUPPORTED) !=
386	XFEATURES_INIT_FPSTATE_HANDLED);	417	XFEATURES_INIT_FPSTATE_HANDLED);
387		418
388	WARN_ON_FPU(!on_boot_cpu);
389	on_boot_cpu = 0;
390
391	if (!boot_cpu_has(X86_FEATURE_XSAVE))	419	if (!boot_cpu_has(X86_FEATURE_XSAVE))
392	return;	420	return;
393		421
394	setup_xstate_features();	422	setup_xstate_features();
395	print_xstate_features();	423	print_xstate_features();
396		424
397	if (boot_cpu_has(X86_FEATURE_XSAVES))	425	xstate_init_xcomp_bv(&init_fpstate.regs.xsave, fpu_kernel_cfg.max_features);
398	init_fpstate.xsave.header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT \|
399	xfeatures_mask_all;
400		426
401	/*	427	/*
402	* Init all the features state with header.xfeatures being 0x0	428	* Init all the features state with header.xfeatures being 0x0
403	*/	429	*/
404	os_xrstor_booting(&init_fpstate.xsave);	430	os_xrstor_booting(&init_fpstate.regs.xsave);
405		431
406	/*	432	/*
407	* All components are now in init state. Read the state back so	433	* All components are now in init state. Read the state back so
@@ -419,7 +445,7 @@ static void __init setup_init_fpu_buf(void) · 11 unmodified lines
419	* state is all zeroes or if not to add the necessary handling	445	* state is all zeroes or if not to add the necessary handling
420	* here.	446	* here.
421	*/	447	*/
422	fxsave(&init_fpstate.fxsave);	448	fxsave(&init_fpstate.regs.fxsave);
423	}	449	}
424		450
425	static int xfeature_uncompacted_offset(int xfeature_nr)	451	static int xfeature_uncompacted_offset(int xfeature_nr)
@@ -451,10 +477,11 @@ int xfeature_size(int xfeature_nr) · 25 unmodified lines
451	}	477	}
452		478
453	/* Validate an xstate header supplied by userspace (ptrace or sigreturn) */	479	/* Validate an xstate header supplied by userspace (ptrace or sigreturn) */
454	static int validate_user_xstate_header(const struct xstate_header *hdr)	480	static int validate_user_xstate_header(const struct xstate_header *hdr,
		481	struct fpstate *fpstate)
455	{	482	{
456	/* No unknown or supervisor features may be set */	483	/* No unknown or supervisor features may be set */
457	if (hdr->xfeatures & ~xfeatures_mask_uabi())	484	if (hdr->xfeatures & ~fpstate->user_xfeatures)
458	return -EINVAL;	485	return -EINVAL;
459		486
460	/* Userspace must use the uncompacted format */	487	/* Userspace must use the uncompacted format */
@@ -474,7 +501,7 @@ static int validate_user_xstate_header(const struct xstate_header *hdr) · 13 unmodified lines
474	return 0;	501	return 0;
475	}	502	}
476		503
477	static void __xstate_dump_leaves(void)	504	static void __init __xstate_dump_leaves(void)
478	{	505	{
479	int i;	506	int i;
480	u32 eax, ebx, ecx, edx;	507	u32 eax, ebx, ecx, edx;
@@ -509,12 +536,73 @@ static void __xstate_dump_leaves(void) · 28 unmodified lines
509	} \	536	} \
510	} while (0)	537	} while (0)
511		538
		539	/**
		540	* check_xtile_data_against_struct - Check tile data state size.
		541	*
		542	* Calculate the state size by multiplying the single tile size which is
		543	* recorded in a C struct, and the number of tiles that the CPU informs.
		544	* Compare the provided size with the calculation.
		545	*
		546	* @size: The tile data state size
		547	*
		548	* Returns: 0 on success, -EINVAL on mismatch.
		549	*/
		550	static int __init check_xtile_data_against_struct(int size)
		551	{
		552	u32 max_palid, palid, state_size;
		553	u32 eax, ebx, ecx, edx;
		554	u16 max_tile;
		555
		556	/*
		557	* Check the maximum palette id:
		558	* eax: the highest numbered palette subleaf.
		559	*/
		560	cpuid_count(TILE_CPUID, 0, &max_palid, &ebx, &ecx, &edx);
		561
		562	/*
		563	* Cross-check each tile size and find the maximum number of
		564	* supported tiles.
		565	*/
		566	for (palid = 1, max_tile = 0; palid <= max_palid; palid++) {
		567	u16 tile_size, max;
		568
		569	/*
		570	* Check the tile size info:
		571	* eax[31:16]: bytes per title
		572	* ebx[31:16]: the max names (or max number of tiles)
		573	*/
		574	cpuid_count(TILE_CPUID, palid, &eax, &ebx, &edx, &edx);
		575	tile_size = eax >> 16;
		576	max = ebx >> 16;
		577
		578	if (tile_size != sizeof(struct xtile_data)) {
		579	pr_err("%s: struct is %zu bytes, cpu xtile %d bytes\n",
		580	__stringify(XFEATURE_XTILE_DATA),
		581	sizeof(struct xtile_data), tile_size);
		582	__xstate_dump_leaves();
		583	return -EINVAL;
		584	}
		585
		586	if (max > max_tile)
		587	max_tile = max;
		588	}
		589
		590	state_size = sizeof(struct xtile_data) * max_tile;
		591	if (size != state_size) {
		592	pr_err("%s: calculated size is %u bytes, cpu state %d bytes\n",
		593	__stringify(XFEATURE_XTILE_DATA), state_size, size);
		594	__xstate_dump_leaves();
		595	return -EINVAL;
		596	}
		597	return 0;
		598	}
		599
512	/*	600	/*
513	* We have a C struct for each 'xstate'. We need to ensure	601	* We have a C struct for each 'xstate'. We need to ensure
514	* that our software representation matches what the CPU	602	* that our software representation matches what the CPU
515	* tells us about the state's size.	603	* tells us about the state's size.
516	*/	604	*/
517	static void check_xstate_against_struct(int nr)	605	static bool __init check_xstate_against_struct(int nr)
518	{	606	{
519	/*	607	/*
520	* Ask the CPU for the size of the state.	608	* Ask the CPU for the size of the state.
@@ -532,6 +620,11 @@ static void check_xstate_against_struct(int nr) · 11 unmodified lines
532	XCHECK_SZ(sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state);	620	XCHECK_SZ(sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state);
533	XCHECK_SZ(sz, nr, XFEATURE_PKRU, struct pkru_state);	621	XCHECK_SZ(sz, nr, XFEATURE_PKRU, struct pkru_state);
534	XCHECK_SZ(sz, nr, XFEATURE_PASID, struct ia32_pasid_state);	622	XCHECK_SZ(sz, nr, XFEATURE_PASID, struct ia32_pasid_state);
		623	XCHECK_SZ(sz, nr, XFEATURE_XTILE_CFG, struct xtile_cfg);
		624
		625	/* The tile data size varies between implementations. */
		626	if (nr == XFEATURE_XTILE_DATA)
		627	check_xtile_data_against_struct(sz);
535		628
536	/*	629	/*
537	* Make SURE to add any feature numbers in below if	630	* Make SURE to add any feature numbers in below if
@@ -541,10 +634,39 @@ static void check_xstate_against_struct(int nr) · 3 unmodified lines
541	if ((nr < XFEATURE_YMM) \|\|	634	if ((nr < XFEATURE_YMM) \|\|
542	(nr >= XFEATURE_MAX) \|\|	635	(nr >= XFEATURE_MAX) \|\|
543	(nr == XFEATURE_PT_UNIMPLEMENTED_SO_FAR) \|\|	636	(nr == XFEATURE_PT_UNIMPLEMENTED_SO_FAR) \|\|
544	((nr >= XFEATURE_RSRVD_COMP_11) && (nr <= XFEATURE_LBR))) {	637	((nr >= XFEATURE_RSRVD_COMP_11) && (nr <= XFEATURE_RSRVD_COMP_16))) {
545	WARN_ONCE(1, "no structure for xstate: %d\n", nr);	638	WARN_ONCE(1, "no structure for xstate: %d\n", nr);
546	XSTATE_WARN_ON(1);	639	XSTATE_WARN_ON(1);
		640	return false;
		641	}
		642	return true;
		643	}
		644
		645	static unsigned int xstate_calculate_size(u64 xfeatures, bool compacted)
		646	{
		647	unsigned int size = FXSAVE_SIZE + XSAVE_HDR_SIZE;
		648	int i;
		649
		650	for_each_extended_xfeature(i, xfeatures) {
		651	/* Align from the end of the previous feature */
		652	if (xfeature_is_aligned(i))
		653	size = ALIGN(size, 64);
		654	/*
		655	* In compacted format the enabled features are packed,
		656	* i.e. disabled features do not occupy space.
		657	*
		658	* In non-compacted format the offsets are fixed and
		659	* disabled states still occupy space in the memory buffer.
		660	*/
		661	if (!compacted)
		662	size = xfeature_uncompacted_offset(i);
		663	/*
		664	* Add the feature size even for non-compacted format
		665	* to make the end result correct
		666	*/
		667	size += xfeature_size(i);
547	}	668	}
		669	return size;
548	}	670	}
549		671
550	/*	672	/*
@@ -556,44 +678,29 @@ static void check_xstate_against_struct(int nr) · 5 unmodified lines
556	* covered by these checks. Only the size of the buffer for task->fpu	678	* covered by these checks. Only the size of the buffer for task->fpu
557	* is checked here.	679	* is checked here.
558	*/	680	*/
559	static void do_extra_xstate_size_checks(void)	681	static bool __init paranoid_xstate_size_valid(unsigned int kernel_size)
560	{	682	{
561	int paranoid_xstate_size = FXSAVE_SIZE + XSAVE_HDR_SIZE;	683	bool compacted = cpu_feature_enabled(X86_FEATURE_XSAVES);
		684	unsigned int size = FXSAVE_SIZE + XSAVE_HDR_SIZE;
562	int i;	685	int i;
563		686
564	for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {	687	for_each_extended_xfeature(i, fpu_kernel_cfg.max_features) {
565	if (!xfeature_enabled(i))	688	if (!check_xstate_against_struct(i))
566	continue;	689	return false;
567
568	check_xstate_against_struct(i);
569	/*	690	/*
570	* Supervisor state components can be managed only by	691	* Supervisor state components can be managed only by
571	* XSAVES.	692	* XSAVES.
572	*/	693	*/
573	if (!cpu_feature_enabled(X86_FEATURE_XSAVES))	694	if (!compacted && xfeature_is_supervisor(i)) {
574	XSTATE_WARN_ON(xfeature_is_supervisor(i));	695	XSTATE_WARN_ON(1);
575		696	return false;
576	/* Align from the end of the previous feature */	697	}
577	if (xfeature_is_aligned(i))
578	paranoid_xstate_size = ALIGN(paranoid_xstate_size, 64);
579	/*
580	* The offset of a given state in the non-compacted
581	* format is given to us in a CPUID leaf. We check
582	* them for being ordered (increasing offsets) in
583	* setup_xstate_features(). XSAVES uses compacted format.
584	*/
585	if (!cpu_feature_enabled(X86_FEATURE_XSAVES))
586	paranoid_xstate_size = xfeature_uncompacted_offset(i);
587	/*
588	* The compacted-format offset always depends on where
589	* the previous state ended.
590	*/
591	paranoid_xstate_size += xfeature_size(i);
592	}	698	}
593	XSTATE_WARN_ON(paranoid_xstate_size != fpu_kernel_xstate_size);	699	size = xstate_calculate_size(fpu_kernel_cfg.max_features, compacted);
		700	XSTATE_WARN_ON(size != kernel_size);
		701	return size == kernel_size;
594	}	702	}
595		703
596
597	/*	704	/*
598	* Get total size of enabled xstates in XCR0 \| IA32_XSS.	705	* Get total size of enabled xstates in XCR0 \| IA32_XSS.
599	*	706	*
@@ -644,7 +751,7 @@ static unsigned int __init get_xsaves_size_no_independent(void) · 44 unmodified lines
644	return size;	751	return size;
645	}	752	}
646		753
647	static unsigned int __init get_xsave_size(void)	754	static unsigned int __init get_xsave_size_user(void)
648	{	755	{
649	unsigned int eax, ebx, ecx, edx;	756	unsigned int eax, ebx, ecx, edx;
650	/*	757	/*
@@ -662,44 +769,54 @@ static unsigned int __init get_xsave_size(void) · 11 unmodified lines
662	* Will the runtime-enumerated 'xstate_size' fit in the init	769	* Will the runtime-enumerated 'xstate_size' fit in the init
663	* task's statically-allocated buffer?	770	* task's statically-allocated buffer?
664	*/	771	*/
665	static bool is_supported_xstate_size(unsigned int test_xstate_size)	772	static bool __init is_supported_xstate_size(unsigned int test_xstate_size)
666	{	773	{
667	if (test_xstate_size <= sizeof(union fpregs_state))	774	if (test_xstate_size <= sizeof(init_fpstate.regs))
668	return true;	775	return true;
669		776
670	pr_warn("x86/fpu: xstate buffer too small (%zu < %d), disabling xsave\n",	777	pr_warn("x86/fpu: xstate buffer too small (%zu < %d), disabling xsave\n",
671	sizeof(union fpregs_state), test_xstate_size);	778	sizeof(init_fpstate.regs), test_xstate_size);
672	return false;	779	return false;
673	}	780	}
674		781
675	static int __init init_xstate_size(void)	782	static int __init init_xstate_size(void)
676	{	783	{
677	/* Recompute the context size for enabled features: */	784	/* Recompute the context size for enabled features: */
678	unsigned int possible_xstate_size;	785	unsigned int user_size, kernel_size, kernel_default_size;
679	unsigned int xsave_size;	786	bool compacted = cpu_feature_enabled(X86_FEATURE_XSAVES);
680		787
681	xsave_size = get_xsave_size();	788	/* Uncompacted user space size */
		789	user_size = get_xsave_size_user();
682		790
683	if (boot_cpu_has(X86_FEATURE_XSAVES))	791	/*
684	possible_xstate_size = get_xsaves_size_no_independent();	792	* XSAVES kernel size includes supervisor states and
		793	* uses compacted format when available.
		794	*
		795	* XSAVE does not support supervisor states so
		796	* kernel and user size is identical.
		797	*/
		798	if (compacted)
		799	kernel_size = get_xsaves_size_no_independent();
685	else	800	else
686	possible_xstate_size = xsave_size;	801	kernel_size = user_size;
		802
		803	kernel_default_size =
		804	xstate_calculate_size(fpu_kernel_cfg.default_features, compacted);
687		805
688	/* Ensure we have the space to store all enabled: */	806	/* Ensure we have the space to store all default enabled features. */
689	if (!is_supported_xstate_size(possible_xstate_size))	807	if (!is_supported_xstate_size(kernel_default_size))
690	return -EINVAL;	808	return -EINVAL;
691		809
692	/*	810	if (!paranoid_xstate_size_valid(kernel_size))
693	* The size is OK, we are definitely going to use xsave,	811	return -EINVAL;
694	* make it known to the world that we need more space.	812
695	*/	813	fpu_kernel_cfg.max_size = kernel_size;
696	fpu_kernel_xstate_size = possible_xstate_size;	814	fpu_user_cfg.max_size = user_size;
697	do_extra_xstate_size_checks();	815
		816	fpu_kernel_cfg.default_size = kernel_default_size;
		817	fpu_user_cfg.default_size =
		818	xstate_calculate_size(fpu_user_cfg.default_features, false);
698		819
699	/*
700	* User space is always in standard format.
701	*/
702	fpu_user_xstate_size = xsave_size;
703	return 0;	820	return 0;
704	}	821	}
705		822
@@ -707,28 +824,38 @@ static int __init init_xstate_size(void) · 1 unmodified line
707	* We enabled the XSAVE hardware, but something went wrong and	824	* We enabled the XSAVE hardware, but something went wrong and
708	* we can not use it. Disable it.	825	* we can not use it. Disable it.
709	*/	826	*/
710	static void fpu__init_disable_system_xstate(void)	827	static void __init fpu__init_disable_system_xstate(unsigned int legacy_size)
711	{	828	{
712	xfeatures_mask_all = 0;	829	fpu_kernel_cfg.max_features = 0;
713	cr4_clear_bits(X86_CR4_OSXSAVE);	830	cr4_clear_bits(X86_CR4_OSXSAVE);
714	setup_clear_cpu_cap(X86_FEATURE_XSAVE);	831	setup_clear_cpu_cap(X86_FEATURE_XSAVE);
		832
		833	/* Restore the legacy size.*/
		834	fpu_kernel_cfg.max_size = legacy_size;
		835	fpu_kernel_cfg.default_size = legacy_size;
		836	fpu_user_cfg.max_size = legacy_size;
		837	fpu_user_cfg.default_size = legacy_size;
		838
		839	/*
		840	* Prevent enabling the static branch which enables writes to the
		841	* XFD MSR.
		842	*/
		843	init_fpstate.xfd = 0;
		844
		845	fpstate_reset(&current->thread.fpu);
715	}	846	}
716		847
717	/*	848	/*
718	* Enable and initialize the xsave feature.	849	* Enable and initialize the xsave feature.
719	* Called once per system bootup.	850	* Called once per system bootup.
720	*/	851	*/
721	void __init fpu__init_system_xstate(void)	852	void __init fpu__init_system_xstate(unsigned int legacy_size)
722	{	853	{
723	unsigned int eax, ebx, ecx, edx;	854	unsigned int eax, ebx, ecx, edx;
724	static int on_boot_cpu __initdata = 1;
725	u64 xfeatures;	855	u64 xfeatures;
726	int err;	856	int err;
727	int i;	857	int i;
728		858
729	WARN_ON_FPU(!on_boot_cpu);
730	on_boot_cpu = 0;
731
732	if (!boot_cpu_has(X86_FEATURE_FPU)) {	859	if (!boot_cpu_has(X86_FEATURE_FPU)) {
733	pr_info("x86/fpu: No FPU detected\n");	860	pr_info("x86/fpu: No FPU detected\n");
734	return;	861	return;
@@ -749,22 +876,22 @@ void __init fpu__init_system_xstate(void) · 14 unmodified lines
749	* Find user xstates supported by the processor.	876	* Find user xstates supported by the processor.
750	*/	877	*/
751	cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx);	878	cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx);
752	xfeatures_mask_all = eax + ((u64)edx << 32);	879	fpu_kernel_cfg.max_features = eax + ((u64)edx << 32);
753		880
754	/*	881	/*
755	* Find supervisor xstates supported by the processor.	882	* Find supervisor xstates supported by the processor.
756	*/	883	*/
757	cpuid_count(XSTATE_CPUID, 1, &eax, &ebx, &ecx, &edx);	884	cpuid_count(XSTATE_CPUID, 1, &eax, &ebx, &ecx, &edx);
758	xfeatures_mask_all \|= ecx + ((u64)edx << 32);	885	fpu_kernel_cfg.max_features \|= ecx + ((u64)edx << 32);
759		886
760	if ((xfeatures_mask_uabi() & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) {	887	if ((fpu_kernel_cfg.max_features & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) {
761	/*	888	/*
762	* This indicates that something really unexpected happened	889	* This indicates that something really unexpected happened
763	* with the enumeration. Disable XSAVE and try to continue	890	* with the enumeration. Disable XSAVE and try to continue
764	* booting without it. This is too early to BUG().	891	* booting without it. This is too early to BUG().
765	*/	892	*/
766	pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n",	893	pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n",
767	xfeatures_mask_all);	894	fpu_kernel_cfg.max_features);
768	goto out_disable;	895	goto out_disable;
769	}	896	}
770		897
@@ -772,15 +899,39 @@ void __init fpu__init_system_xstate(void) · 1 unmodified line
772	* Clear XSAVE features that are disabled in the normal CPUID.	899	* Clear XSAVE features that are disabled in the normal CPUID.
773	*/	900	*/
774	for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) {	901	for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) {
775	if (!boot_cpu_has(xsave_cpuid_features[i]))	902	unsigned short cid = xsave_cpuid_features[i];
776	xfeatures_mask_all &= ~BIT_ULL(i);	903
		904	/* Careful: X86_FEATURE_FPU is 0! */
		905	if ((i != XFEATURE_FP && !cid) \|\| !boot_cpu_has(cid))
		906	fpu_kernel_cfg.max_features &= ~BIT_ULL(i);
777	}	907	}
778		908
779	xfeatures_mask_all &= XFEATURE_MASK_USER_SUPPORTED \|	909	if (!cpu_feature_enabled(X86_FEATURE_XFD))
		910	fpu_kernel_cfg.max_features &= ~XFEATURE_MASK_USER_DYNAMIC;
		911
		912	fpu_kernel_cfg.max_features &= XFEATURE_MASK_USER_SUPPORTED \|
780	XFEATURE_MASK_SUPERVISOR_SUPPORTED;	913	XFEATURE_MASK_SUPERVISOR_SUPPORTED;
781		914
		915	fpu_user_cfg.max_features = fpu_kernel_cfg.max_features;
		916	fpu_user_cfg.max_features &= XFEATURE_MASK_USER_SUPPORTED;
		917
		918	/* Clean out dynamic features from default */
		919	fpu_kernel_cfg.default_features = fpu_kernel_cfg.max_features;
		920	fpu_kernel_cfg.default_features &= ~XFEATURE_MASK_USER_DYNAMIC;
		921
		922	fpu_user_cfg.default_features = fpu_user_cfg.max_features;
		923	fpu_user_cfg.default_features &= ~XFEATURE_MASK_USER_DYNAMIC;
		924
782	/* Store it for paranoia check at the end */	925	/* Store it for paranoia check at the end */
783	xfeatures = xfeatures_mask_all;	926	xfeatures = fpu_kernel_cfg.max_features;
		927
		928	/*
		929	* Initialize the default XFD state in initfp_state and enable the
		930	* dynamic sizing mechanism if dynamic states are available. The
		931	* static key cannot be enabled here because this runs before
		932	* jump_label_init(). This is delayed to an initcall.
		933	*/
		934	init_fpstate.xfd = fpu_user_cfg.max_features & XFEATURE_MASK_USER_DYNAMIC;
784		935
785	/* Enable xstate instructions to be able to continue with initialization: */	936	/* Enable xstate instructions to be able to continue with initialization: */
786	fpu__init_cpu_xstate();	937	fpu__init_cpu_xstate();
@@ -788,13 +939,16 @@ void __init fpu__init_system_xstate(void) · 1 unmodified line
788	if (err)	939	if (err)
789	goto out_disable;	940	goto out_disable;
790		941
		942	/* Reset the state for the current task */
		943	fpstate_reset(&current->thread.fpu);
		944
791	/*	945	/*
792	* Update info used for ptrace frames; use standard-format size and no	946	* Update info used for ptrace frames; use standard-format size and no
793	* supervisor xstates:	947	* supervisor xstates:
794	*/	948	*/
795	update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_uabi());	949	update_regset_xstate_info(fpu_user_cfg.max_size,
		950	fpu_user_cfg.max_features);
796		951
797	fpu__init_prepare_fx_sw_frame();
798	setup_init_fpu_buf();	952	setup_init_fpu_buf();
799	setup_xstate_comp_offsets();	953	setup_xstate_comp_offsets();
800	setup_supervisor_only_offsets();	954	setup_supervisor_only_offsets();
@@ -803,22 +957,22 @@ void __init fpu__init_system_xstate(void) · 2 unmodified lines
803	* Paranoia check whether something in the setup modified the	957	* Paranoia check whether something in the setup modified the
804	* xfeatures mask.	958	* xfeatures mask.
805	*/	959	*/
806	if (xfeatures != xfeatures_mask_all) {	960	if (xfeatures != fpu_kernel_cfg.max_features) {
807	pr_err("x86/fpu: xfeatures modified from 0x%016llx to 0x%016llx during init, disabling XSAVE\n",	961	pr_err("x86/fpu: xfeatures modified from 0x%016llx to 0x%016llx during init, disabling XSAVE\n",
808	xfeatures, xfeatures_mask_all);	962	xfeatures, fpu_kernel_cfg.max_features);
809	goto out_disable;	963	goto out_disable;
810	}	964	}
811		965
812	print_xstate_offset_size();	966	print_xstate_offset_size();
813	pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n",	967	pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n",
814	xfeatures_mask_all,	968	fpu_kernel_cfg.max_features,
815	fpu_kernel_xstate_size,	969	fpu_kernel_cfg.max_size,
816	boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard");	970	boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard");
817	return;	971	return;
818		972
819	out_disable:	973	out_disable:
820	/* something went wrong, try to boot without any XSAVE support */	974	/* something went wrong, try to boot without any XSAVE support */
821	fpu__init_disable_system_xstate();	975	fpu__init_disable_system_xstate(legacy_size);
822	}	976	}
823		977
824	/*	978	/*
@@ -830,7 +984,7 @@ void fpu__resume_cpu(void) · 5 unmodified lines
830	* Restore XCR0 on xsave capable CPUs:	984	* Restore XCR0 on xsave capable CPUs:
831	*/	985	*/
832	if (cpu_feature_enabled(X86_FEATURE_XSAVE))	986	if (cpu_feature_enabled(X86_FEATURE_XSAVE))
833	xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_uabi());	987	xsetbv(XCR_XFEATURE_ENABLED_MASK, fpu_user_cfg.max_features);
834		988
835	/*	989	/*
836	* Restore IA32_XSS. The same CPUID bit enumerates support	990	* Restore IA32_XSS. The same CPUID bit enumerates support
@@ -840,6 +994,9 @@ void fpu__resume_cpu(void) · 3 unmodified lines
840	wrmsrl(MSR_IA32_XSS, xfeatures_mask_supervisor() \|	994	wrmsrl(MSR_IA32_XSS, xfeatures_mask_supervisor() \|
841	xfeatures_mask_independent());	995	xfeatures_mask_independent());
842	}	996	}
		997
		998	if (fpu_state_size_dynamic())
		999	wrmsrl(MSR_IA32_XFD, current->thread.fpu.fpstate->xfd);
843	}	1000	}
844		1001
845	/*	1002	/*
@@ -886,7 +1043,7 @@ void get_xsave_addr(struct xregs_state xsave, int xfeature_nr) · 40 unmodified lines
886	* We should not ever be requesting features that we	1043	* We should not ever be requesting features that we
887	* have not enabled.	1044	* have not enabled.
888	*/	1045	*/
889	WARN_ONCE(!(xfeatures_mask_all & BIT_ULL(xfeature_nr)),	1046	WARN_ONCE(!(fpu_kernel_cfg.max_features & BIT_ULL(xfeature_nr)),
890	"get of unsupported state");	1047	"get of unsupported state");
891	/*	1048	/*
892	* This assumes the last 'xsave*' instruction to	1049	* This assumes the last 'xsave*' instruction to
@@ -904,7 +1061,6 @@ void get_xsave_addr(struct xregs_state xsave, int xfeature_nr) · 11 unmodified lines
904		1061
905	return __raw_xsave_addr(xsave, xfeature_nr);	1062	return __raw_xsave_addr(xsave, xfeature_nr);
906	}	1063	}
907	EXPORT_SYMBOL_GPL(get_xsave_addr);
908		1064
909	#ifdef CONFIG_ARCH_HAS_PKEYS	1065	#ifdef CONFIG_ARCH_HAS_PKEYS
910		1066
@@ -961,9 +1117,10 @@ static void copy_feature(bool from_xstate, struct membuf to, void xstate, · 50 unmodified lines
961	}	1117	}
962		1118
963	/**	1119	/**
964	* copy_xstate_to_uabi_buf - Copy kernel saved xstate to a UABI buffer	1120	* __copy_xstate_to_uabi_buf - Copy kernel saved xstate to a UABI buffer
965	* @to: membuf descriptor	1121	* @to: membuf descriptor
966	* @tsk: The task from which to copy the saved xstate	1122	* @fpstate: The fpstate buffer from which to copy
		1123	* @pkru_val: The PKRU value to store in the PKRU component
967	* @copy_mode: The requested copy mode	1124	* @copy_mode: The requested copy mode
968	*	1125	*
969	* Converts from kernel XSAVE or XSAVES compacted format to UABI conforming	1126	* Converts from kernel XSAVE or XSAVES compacted format to UABI conforming
@@ -972,14 +1129,15 @@ static void copy_feature(bool from_xstate, struct membuf to, void xstate, · 2 unmodified lines
972	*	1129	*
973	* It supports partial copy but @to.pos always starts from zero.	1130	* It supports partial copy but @to.pos always starts from zero.
974	*/	1131	*/
975	void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk,	1132	void __copy_xstate_to_uabi_buf(struct membuf to, struct fpstate *fpstate,
976	enum xstate_copy_mode copy_mode)	1133	u32 pkru_val, enum xstate_copy_mode copy_mode)
977	{	1134	{
978	const unsigned int off_mxcsr = offsetof(struct fxregs_state, mxcsr);	1135	const unsigned int off_mxcsr = offsetof(struct fxregs_state, mxcsr);
979	struct xregs_state *xsave = &tsk->thread.fpu.state.xsave;	1136	struct xregs_state *xinit = &init_fpstate.regs.xsave;
980	struct xregs_state *xinit = &init_fpstate.xsave;	1137	struct xregs_state *xsave = &fpstate->regs.xsave;
981	struct xstate_header header;	1138	struct xstate_header header;
982	unsigned int zerofrom;	1139	unsigned int zerofrom;
		1140	u64 mask;
983	int i;	1141	int i;
984		1142
985	memset(&header, 0, sizeof(header));	1143	memset(&header, 0, sizeof(header));
@@ -996,7 +1154,7 @@ void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk, · 10 unmodified lines
996	break;	1154	break;
997		1155
998	case XSTATE_COPY_XSAVE:	1156	case XSTATE_COPY_XSAVE:
999	header.xfeatures &= xfeatures_mask_uabi();	1157	header.xfeatures &= fpstate->user_xfeatures;
1000	break;	1158	break;
1001	}	1159	}
1002		1160
@@ -1033,17 +1191,15 @@ void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk, · 30 unmodified lines
1033		1191
1034	zerofrom = offsetof(struct xregs_state, extended_state_area);	1192	zerofrom = offsetof(struct xregs_state, extended_state_area);
1035		1193
1036	for (i = FIRST_EXTENDED_XFEATURE; i < XFEATURE_MAX; i++) {	1194	/*
1037	/*	1195	* The ptrace buffer is in non-compacted XSAVE format. In
1038	* The ptrace buffer is in non-compacted XSAVE format.	1196	* non-compacted format disabled features still occupy state space,
1039	* In non-compacted format disabled features still occupy	1197	* but there is no state to copy from in the compacted
1040	* state space, but there is no state to copy from in the	1198	* init_fpstate. The gap tracking will zero these states.
1041	* compacted init_fpstate. The gap tracking will zero this	1199	*/
1042	* later.	1200	mask = fpstate->user_xfeatures;
1043	*/
1044	if (!(xfeatures_mask_uabi() & BIT_ULL(i)))
1045	continue;
1046		1201
		1202	for_each_extended_xfeature(i, mask) {
1047	/*	1203	/*
1048	* If there was a feature or alignment gap, zero the space	1204	* If there was a feature or alignment gap, zero the space
1049	* in the destination buffer.	1205	* in the destination buffer.
@@ -1055,10 +1211,9 @@ void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk, · 5 unmodified lines
1055	struct pkru_state pkru = {0};	1211	struct pkru_state pkru = {0};
1056	/*	1212	/*
1057	* PKRU is not necessarily up to date in the	1213	* PKRU is not necessarily up to date in the
1058	* thread's XSAVE buffer. Fill this part from the	1214	* XSAVE buffer. Use the provided value.
1059	* per-thread storage.
1060	*/	1215	*/
1061	pkru.pkru = tsk->thread.pkru;	1216	pkru.pkru = pkru_val;
1062	membuf_write(&to, &pkru, sizeof(pkru));	1217	membuf_write(&to, &pkru, sizeof(pkru));
1063	} else {	1218	} else {
1064	copy_feature(header.xfeatures & BIT_ULL(i), &to,	1219	copy_feature(header.xfeatures & BIT_ULL(i), &to,
@@ -1078,6 +1233,25 @@ out: · 13 unmodified lines
1078	membuf_zero(&to, to.left);	1233	membuf_zero(&to, to.left);
1079	}	1234	}
1080		1235
		1236	/**
		1237	* copy_xstate_to_uabi_buf - Copy kernel saved xstate to a UABI buffer
		1238	* @to: membuf descriptor
		1239	* @tsk: The task from which to copy the saved xstate
		1240	* @copy_mode: The requested copy mode
		1241	*
		1242	* Converts from kernel XSAVE or XSAVES compacted format to UABI conforming
		1243	* format, i.e. from the kernel internal hardware dependent storage format
		1244	* to the requested @mode. UABI XSTATE is always uncompacted!
		1245	*
		1246	* It supports partial copy but @to.pos always starts from zero.
		1247	*/
		1248	void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk,
		1249	enum xstate_copy_mode copy_mode)
		1250	{
		1251	__copy_xstate_to_uabi_buf(to, tsk->thread.fpu.fpstate,
		1252	tsk->thread.pkru, copy_mode);
		1253	}
		1254
1081	static int copy_from_buffer(void *dst, unsigned int offset, unsigned int size,	1255	static int copy_from_buffer(void *dst, unsigned int offset, unsigned int size,
1082	const void kbuf, const void __user ubuf)	1256	const void kbuf, const void __user ubuf)
1083	{	1257	{
@@ -1091,9 +1265,10 @@ static int copy_from_buffer(void *dst, unsigned int offset, unsigned int size, · 7 unmodified lines
1091	}	1265	}
1092		1266
1093		1267
1094	static int copy_uabi_to_xstate(struct xregs_state xsave, const void kbuf,	1268	static int copy_uabi_to_xstate(struct fpstate fpstate, const void kbuf,
1095	const void __user *ubuf)	1269	const void __user *ubuf)
1096	{	1270	{
		1271	struct xregs_state *xsave = &fpstate->regs.xsave;
1097	unsigned int offset, size;	1272	unsigned int offset, size;
1098	struct xstate_header hdr;	1273	struct xstate_header hdr;
1099	u64 mask;	1274	u64 mask;
@@ -1103,7 +1278,7 @@ static int copy_uabi_to_xstate(struct xregs_state xsave, const void kbuf, · 3 unmodified lines
1103	if (copy_from_buffer(&hdr, offset, sizeof(hdr), kbuf, ubuf))	1278	if (copy_from_buffer(&hdr, offset, sizeof(hdr), kbuf, ubuf))
1104	return -EFAULT;	1279	return -EFAULT;
1105		1280
1106	if (validate_user_xstate_header(&hdr))	1281	if (validate_user_xstate_header(&hdr, fpstate))
1107	return -EINVAL;	1282	return -EINVAL;
1108		1283
1109	/* Validate MXCSR when any of the related features is in use */	1284	/* Validate MXCSR when any of the related features is in use */
@@ -1156,12 +1331,11 @@ static int copy_uabi_to_xstate(struct xregs_state xsave, const void kbuf, · 46 unmodified lines
1156		1331
1157	/*	1332	/*
1158	* Convert from a ptrace standard-format kernel buffer to kernel XSAVE[S]	1333	* Convert from a ptrace standard-format kernel buffer to kernel XSAVE[S]
1159	* format and copy to the target thread. This is called from	1334	* format and copy to the target thread. Used by ptrace and KVM.
1160	* xstateregs_set().
1161	*/	1335	*/
1162	int copy_uabi_from_kernel_to_xstate(struct xregs_state xsave, const void kbuf)	1336	int copy_uabi_from_kernel_to_xstate(struct fpstate fpstate, const void kbuf)
1163	{	1337	{
1164	return copy_uabi_to_xstate(xsave, kbuf, NULL);	1338	return copy_uabi_to_xstate(fpstate, kbuf, NULL);
1165	}	1339	}
1166		1340
1167	/*	1341	/*
@@ -1169,26 +1343,20 @@ int copy_uabi_from_kernel_to_xstate(struct xregs_state xsave, const void kbuf) · 1 unmodified line
1169	* XSAVE[S] format and copy to the target thread. This is called from the	1343	* XSAVE[S] format and copy to the target thread. This is called from the
1170	* sigreturn() and rt_sigreturn() system calls.	1344	* sigreturn() and rt_sigreturn() system calls.
1171	*/	1345	*/
1172	int copy_sigframe_from_user_to_xstate(struct xregs_state *xsave,	1346	int copy_sigframe_from_user_to_xstate(struct fpstate *fpstate,
1173	const void __user *ubuf)	1347	const void __user *ubuf)
1174	{	1348	{
1175	return copy_uabi_to_xstate(xsave, NULL, ubuf);	1349	return copy_uabi_to_xstate(fpstate, NULL, ubuf);
1176	}	1350	}
1177		1351
1178	static bool validate_xsaves_xrstors(u64 mask)	1352	static bool validate_independent_components(u64 mask)
1179	{	1353	{
1180	u64 xchk;	1354	u64 xchk;
1181		1355
1182	if (WARN_ON_FPU(!cpu_feature_enabled(X86_FEATURE_XSAVES)))	1356	if (WARN_ON_FPU(!cpu_feature_enabled(X86_FEATURE_XSAVES)))
1183	return false;	1357	return false;
1184	/*	1358
1185	* Validate that this is either a task->fpstate related component	1359	xchk = ~xfeatures_mask_independent();
1186	* subset or an independent one.
1187	*/
1188	if (mask & xfeatures_mask_independent())
1189	xchk = ~xfeatures_mask_independent();
1190	else
1191	xchk = ~xfeatures_mask_all;
1192		1360
1193	if (WARN_ON_ONCE(!mask \|\| mask & xchk))	1361	if (WARN_ON_ONCE(!mask \|\| mask & xchk))
1194	return false;	1362	return false;
@@ -1206,14 +1374,13 @@ static bool validate_xsaves_xrstors(u64 mask) · 11 unmodified lines
1206	* buffer should be zeroed otherwise a consecutive XRSTORS from that buffer	1374	* buffer should be zeroed otherwise a consecutive XRSTORS from that buffer
1207	* can #GP.	1375	* can #GP.
1208	*	1376	*
1209	* The feature mask must either be a subset of the independent features or	1377	* The feature mask must be a subset of the independent features.
1210	* a subset of the task->fpstate related features.
1211	*/	1378	*/
1212	void xsaves(struct xregs_state *xstate, u64 mask)	1379	void xsaves(struct xregs_state *xstate, u64 mask)
1213	{	1380	{
1214	int err;	1381	int err;
1215		1382
1216	if (!validate_xsaves_xrstors(mask))	1383	if (!validate_independent_components(mask))
1217	return;	1384	return;
1218		1385
1219	XSTATE_OP(XSAVES, xstate, (u32)mask, (u32)(mask >> 32), err);	1386	XSTATE_OP(XSAVES, xstate, (u32)mask, (u32)(mask >> 32), err);
@@ -1231,20 +1398,379 @@ void xsaves(struct xregs_state *xstate, u64 mask) · 11 unmodified lines
1231	* Proper usage is to restore the state which was saved with	1398	* Proper usage is to restore the state which was saved with
1232	* xsaves() into @xstate.	1399	* xsaves() into @xstate.
1233	*	1400	*
1234	* The feature mask must either be a subset of the independent features or	1401	* The feature mask must be a subset of the independent features.
1235	* a subset of the task->fpstate related features.
1236	*/	1402	*/
1237	void xrstors(struct xregs_state *xstate, u64 mask)	1403	void xrstors(struct xregs_state *xstate, u64 mask)
1238	{	1404	{
1239	int err;	1405	int err;
1240		1406
1241	if (!validate_xsaves_xrstors(mask))	1407	if (!validate_independent_components(mask))
1242	return;	1408	return;
1243		1409
1244	XSTATE_OP(XRSTORS, xstate, (u32)mask, (u32)(mask >> 32), err);	1410	XSTATE_OP(XRSTORS, xstate, (u32)mask, (u32)(mask >> 32), err);
1245	WARN_ON_ONCE(err);	1411	WARN_ON_ONCE(err);
1246	}	1412	}
1247		1413
		1414	#if IS_ENABLED(CONFIG_KVM)
		1415	void fpstate_clear_xstate_component(struct fpstate *fps, unsigned int xfeature)
		1416	{
		1417	void *addr = get_xsave_addr(&fps->regs.xsave, xfeature);
		1418
		1419	if (addr)
		1420	memset(addr, 0, xstate_sizes[xfeature]);
		1421	}
		1422	EXPORT_SYMBOL_GPL(fpstate_clear_xstate_component);
		1423	#endif
		1424
		1425	#ifdef CONFIG_X86_64
		1426
		1427	#ifdef CONFIG_X86_DEBUG_FPU
		1428	/*
		1429	* Ensure that a subsequent XSAVE* or XRSTOR* instruction with RFBM=@mask
		1430	* can safely operate on the @fpstate buffer.
		1431	*/
		1432	static bool xstate_op_valid(struct fpstate *fpstate, u64 mask, bool rstor)
		1433	{
		1434	u64 xfd = __this_cpu_read(xfd_state);
		1435
		1436	if (fpstate->xfd == xfd)
		1437	return true;
		1438
		1439	/*
		1440	* The XFD MSR does not match fpstate->xfd. That's invalid when
		1441	* the passed in fpstate is current's fpstate.
		1442	*/
		1443	if (fpstate->xfd == current->thread.fpu.fpstate->xfd)
		1444	return false;
		1445
		1446	/*
		1447	* XRSTOR(S) from init_fpstate are always correct as it will just
		1448	* bring all components into init state and not read from the
		1449	* buffer. XSAVE(S) raises #PF after init.
		1450	*/
		1451	if (fpstate == &init_fpstate)
		1452	return rstor;
		1453
		1454	/*
		1455	* XSAVE(S): clone(), fpu_swap_kvm_fpu()
		1456	* XRSTORS(S): fpu_swap_kvm_fpu()
		1457	*/
		1458
		1459	/*
		1460	* No XSAVE/XRSTOR instructions (except XSAVE itself) touch
		1461	* the buffer area for XFD-disabled state components.
		1462	*/
		1463	mask &= ~xfd;
		1464
		1465	/*
		1466	* Remove features which are valid in fpstate. They
		1467	* have space allocated in fpstate.
		1468	*/
		1469	mask &= ~fpstate->xfeatures;
		1470
		1471	/*
		1472	* Any remaining state components in 'mask' might be written
		1473	* by XSAVE/XRSTOR. Fail validation it found.
		1474	*/
		1475	return !mask;
		1476	}
		1477
		1478	void xfd_validate_state(struct fpstate *fpstate, u64 mask, bool rstor)
		1479	{
		1480	WARN_ON_ONCE(!xstate_op_valid(fpstate, mask, rstor));
		1481	}
		1482	#endif /* CONFIG_X86_DEBUG_FPU */
		1483
		1484	static int __init xfd_update_static_branch(void)
		1485	{
		1486	/*
		1487	* If init_fpstate.xfd has bits set then dynamic features are
		1488	* available and the dynamic sizing must be enabled.
		1489	*/
		1490	if (init_fpstate.xfd)
		1491	static_branch_enable(&__fpu_state_size_dynamic);
		1492	return 0;
		1493	}
		1494	arch_initcall(xfd_update_static_branch)
		1495
		1496	void fpstate_free(struct fpu *fpu)
		1497	{
		1498	if (fpu->fpstate && fpu->fpstate != &fpu->__fpstate)
		1499	vfree(fpu->fpstate);
		1500	}
		1501
		1502	/**
		1503	* fpu_install_fpstate - Update the active fpstate in the FPU
		1504	*
		1505	* @fpu: A struct fpu * pointer
		1506	* @newfps: A struct fpstate * pointer
		1507	*
		1508	* Returns: A null pointer if the last active fpstate is the embedded
		1509	* one or the new fpstate is already installed;
		1510	* otherwise, a pointer to the old fpstate which has to
		1511	* be freed by the caller.
		1512	*/
		1513	static struct fpstate fpu_install_fpstate(struct fpu fpu,
		1514	struct fpstate *newfps)
		1515	{
		1516	struct fpstate *oldfps = fpu->fpstate;
		1517
		1518	if (fpu->fpstate == newfps)
		1519	return NULL;
		1520
		1521	fpu->fpstate = newfps;
		1522	return oldfps != &fpu->__fpstate ? oldfps : NULL;
		1523	}
		1524
		1525	/**
		1526	* fpstate_realloc - Reallocate struct fpstate for the requested new features
		1527	*
		1528	* @xfeatures: A bitmap of xstate features which extend the enabled features
		1529	* of that task
		1530	* @ksize: The required size for the kernel buffer
		1531	* @usize: The required size for user space buffers
		1532	*
		1533	* Note vs. vmalloc(): If the task with a vzalloc()-allocated buffer
		1534	* terminates quickly, vfree()-induced IPIs may be a concern, but tasks
		1535	* with large states are likely to live longer.
		1536	*
		1537	* Returns: 0 on success, -ENOMEM on allocation error.
		1538	*/
		1539	static int fpstate_realloc(u64 xfeatures, unsigned int ksize,
		1540	unsigned int usize)
		1541	{
		1542	struct fpu *fpu = &current->thread.fpu;
		1543	struct fpstate curfps, newfps = NULL;
		1544	unsigned int fpsize;
		1545
		1546	curfps = fpu->fpstate;
		1547	fpsize = ksize + ALIGN(offsetof(struct fpstate, regs), 64);
		1548
		1549	newfps = vzalloc(fpsize);
		1550	if (!newfps)
		1551	return -ENOMEM;
		1552	newfps->size = ksize;
		1553	newfps->user_size = usize;
		1554	newfps->is_valloc = true;
		1555
		1556	fpregs_lock();
		1557	/*
		1558	* Ensure that the current state is in the registers before
		1559	* swapping fpstate as that might invalidate it due to layout
		1560	* changes.
		1561	*/
		1562	if (test_thread_flag(TIF_NEED_FPU_LOAD))
		1563	fpregs_restore_userregs();
		1564
		1565	newfps->xfeatures = curfps->xfeatures \| xfeatures;
		1566	newfps->user_xfeatures = curfps->user_xfeatures \| xfeatures;
		1567	newfps->xfd = curfps->xfd & ~xfeatures;
		1568
		1569	curfps = fpu_install_fpstate(fpu, newfps);
		1570
		1571	/* Do the final updates within the locked region */
		1572	xstate_init_xcomp_bv(&newfps->regs.xsave, newfps->xfeatures);
		1573	xfd_update_state(newfps);
		1574
		1575	fpregs_unlock();
		1576
		1577	vfree(curfps);
		1578	return 0;
		1579	}
		1580
		1581	static int validate_sigaltstack(unsigned int usize)
		1582	{
		1583	struct task_struct thread, leader = current->group_leader;
		1584	unsigned long framesize = get_sigframe_size();
		1585
		1586	lockdep_assert_held(&current->sighand->siglock);
		1587
		1588	/* get_sigframe_size() is based on fpu_user_cfg.max_size */
		1589	framesize -= fpu_user_cfg.max_size;
		1590	framesize += usize;
		1591	for_each_thread(leader, thread) {
		1592	if (thread->sas_ss_size && thread->sas_ss_size < framesize)
		1593	return -ENOSPC;
		1594	}
		1595	return 0;
		1596	}
		1597
		1598	static int __xstate_request_perm(u64 permitted, u64 requested)
		1599	{
		1600	/*
		1601	* This deliberately does not exclude !XSAVES as we still might
		1602	* decide to optionally context switch XCR0 or talk the silicon
		1603	* vendors into extending XFD for the pre AMX states, especially
		1604	* AVX512.
		1605	*/
		1606	bool compacted = cpu_feature_enabled(X86_FEATURE_XSAVES);
		1607	struct fpu *fpu = &current->group_leader->thread.fpu;
		1608	unsigned int ksize, usize;
		1609	u64 mask;
		1610	int ret;
		1611
		1612	/* Check whether fully enabled */
		1613	if ((permitted & requested) == requested)
		1614	return 0;
		1615
		1616	/* Calculate the resulting kernel state size */
		1617	mask = permitted \| requested;
		1618	ksize = xstate_calculate_size(mask, compacted);
		1619
		1620	/* Calculate the resulting user state size */
		1621	mask &= XFEATURE_MASK_USER_SUPPORTED;
		1622	usize = xstate_calculate_size(mask, false);
		1623
		1624	ret = validate_sigaltstack(usize);
		1625	if (ret)
		1626	return ret;
		1627
		1628	/* Pairs with the READ_ONCE() in xstate_get_group_perm() */
		1629	WRITE_ONCE(fpu->perm.__state_perm, requested);
		1630	/* Protected by sighand lock */
		1631	fpu->perm.__state_size = ksize;
		1632	fpu->perm.__user_state_size = usize;
		1633	return ret;
		1634	}
		1635
		1636	/*
		1637	* Permissions array to map facilities with more than one component
		1638	*/
		1639	static const u64 xstate_prctl_req[XFEATURE_MAX] = {
		1640	[XFEATURE_XTILE_DATA] = XFEATURE_MASK_XTILE_DATA,
		1641	};
		1642
		1643	static int xstate_request_perm(unsigned long idx)
		1644	{
		1645	u64 permitted, requested;
		1646	int ret;
		1647
		1648	if (idx >= XFEATURE_MAX)
		1649	return -EINVAL;
		1650
		1651	/*
		1652	* Look up the facility mask which can require more than
		1653	* one xstate component.
		1654	*/
		1655	idx = array_index_nospec(idx, ARRAY_SIZE(xstate_prctl_req));
		1656	requested = xstate_prctl_req[idx];
		1657	if (!requested)
		1658	return -EOPNOTSUPP;
		1659
		1660	if ((fpu_user_cfg.max_features & requested) != requested)
		1661	return -EOPNOTSUPP;
		1662
		1663	/* Lockless quick check */
		1664	permitted = xstate_get_host_group_perm();
		1665	if ((permitted & requested) == requested)
		1666	return 0;
		1667
		1668	/* Protect against concurrent modifications */
		1669	spin_lock_irq(&current->sighand->siglock);
		1670	permitted = xstate_get_host_group_perm();
		1671	ret = __xstate_request_perm(permitted, requested);
		1672	spin_unlock_irq(&current->sighand->siglock);
		1673	return ret;
		1674	}
		1675
		1676	int xfd_enable_feature(u64 xfd_err)
		1677	{
		1678	u64 xfd_event = xfd_err & XFEATURE_MASK_USER_DYNAMIC;
		1679	unsigned int ksize, usize;
		1680	struct fpu *fpu;
		1681
		1682	if (!xfd_event) {
		1683	pr_err_once("XFD: Invalid xfd error: %016llx\n", xfd_err);
		1684	return 0;
		1685	}
		1686
		1687	/* Protect against concurrent modifications */
		1688	spin_lock_irq(&current->sighand->siglock);
		1689
		1690	/* If not permitted let it die */
		1691	if ((xstate_get_host_group_perm() & xfd_event) != xfd_event) {
		1692	spin_unlock_irq(&current->sighand->siglock);
		1693	return -EPERM;
		1694	}
		1695
		1696	fpu = &current->group_leader->thread.fpu;
		1697	ksize = fpu->perm.__state_size;
		1698	usize = fpu->perm.__user_state_size;
		1699	/*
		1700	* The feature is permitted. State size is sufficient. Dropping
		1701	* the lock is safe here even if more features are added from
		1702	* another task, the retrieved buffer sizes are valid for the
		1703	* currently requested feature(s).
		1704	*/
		1705	spin_unlock_irq(&current->sighand->siglock);
		1706
		1707	/*
		1708	* Try to allocate a new fpstate. If that fails there is no way
		1709	* out.
		1710	*/
		1711	if (fpstate_realloc(xfd_event, ksize, usize))
		1712	return -EFAULT;
		1713	return 0;
		1714	}
		1715	#else /* CONFIG_X86_64 */
		1716	static inline int xstate_request_perm(unsigned long idx)
		1717	{
		1718	return -EPERM;
		1719	}
		1720	#endif /* !CONFIG_X86_64 */
		1721
		1722	/**
		1723	* fpu_xstate_prctl - xstate permission operations
		1724	* @tsk: Redundant pointer to current
		1725	* @option: A subfunction of arch_prctl()
		1726	* @arg2: option argument
		1727	* Return: 0 if successful; otherwise, an error code
		1728	*
		1729	* Option arguments:
		1730	*
		1731	* ARCH_GET_XCOMP_SUPP: Pointer to user space u64 to store the info
		1732	* ARCH_GET_XCOMP_PERM: Pointer to user space u64 to store the info
		1733	* ARCH_REQ_XCOMP_PERM: Facility number requested
		1734	*
		1735	* For facilities which require more than one XSTATE component, the request
		1736	* must be the highest state component number related to that facility,
		1737	* e.g. for AMX which requires XFEATURE_XTILE_CFG(17) and
		1738	* XFEATURE_XTILE_DATA(18) this would be XFEATURE_XTILE_DATA(18).
		1739	*/
		1740	long fpu_xstate_prctl(struct task_struct *tsk, int option, unsigned long arg2)
		1741	{
		1742	u64 __user uptr = (u64 __user )arg2;
		1743	u64 permitted, supported;
		1744	unsigned long idx = arg2;
		1745
		1746	if (tsk != current)
		1747	return -EPERM;
		1748
		1749	switch (option) {
		1750	case ARCH_GET_XCOMP_SUPP:
		1751	supported = fpu_user_cfg.max_features \| fpu_user_cfg.legacy_features;
		1752	return put_user(supported, uptr);
		1753
		1754	case ARCH_GET_XCOMP_PERM:
		1755	/*
		1756	* Lockless snapshot as it can also change right after the
		1757	* dropping the lock.
		1758	*/
		1759	permitted = xstate_get_host_group_perm();
		1760	permitted &= XFEATURE_MASK_USER_SUPPORTED;
		1761	return put_user(permitted, uptr);
		1762
		1763	case ARCH_REQ_XCOMP_PERM:
		1764	if (!IS_ENABLED(CONFIG_X86_64))
		1765	return -EOPNOTSUPP;
		1766
		1767	return xstate_request_perm(idx);
		1768
		1769	default:
		1770	return -EINVAL;
		1771	}
		1772	}
		1773
1248	#ifdef CONFIG_PROC_PID_ARCH_STATUS	1774	#ifdef CONFIG_PROC_PID_ARCH_STATUS
1249	/*	1775	/*
1250	* Report the amount of time elapsed in millisecond since last AVX512	1776	* Report the amount of time elapsed in millisecond since last AVX512

arch/x86/kernel/fpu/xstate.h+278−0

@@ -0,0 +1,278 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#ifndef __X86_KERNEL_FPU_XSTATE_H
	3	#define __X86_KERNEL_FPU_XSTATE_H
	4
	5	#include <asm/cpufeature.h>
	6	#include <asm/fpu/xstate.h>
	7
	8	#ifdef CONFIG_X86_64
	9	DECLARE_PER_CPU(u64, xfd_state);
	10	#endif
	11
	12	static inline void xstate_init_xcomp_bv(struct xregs_state *xsave, u64 mask)
	13	{
	14	/*
	15	* XRSTORS requires these bits set in xcomp_bv, or it will
	16	* trigger #GP:
	17	*/
	18	if (cpu_feature_enabled(X86_FEATURE_XSAVES))
	19	xsave->header.xcomp_bv = mask \| XCOMP_BV_COMPACTED_FORMAT;
	20	}
	21
	22	static inline u64 xstate_get_host_group_perm(void)
	23	{
	24	/* Pairs with WRITE_ONCE() in xstate_request_perm() */
	25	return READ_ONCE(current->group_leader->thread.fpu.perm.__state_perm);
	26	}
	27
	28	enum xstate_copy_mode {
	29	XSTATE_COPY_FP,
	30	XSTATE_COPY_FX,
	31	XSTATE_COPY_XSAVE,
	32	};
	33
	34	struct membuf;
	35	extern void __copy_xstate_to_uabi_buf(struct membuf to, struct fpstate *fpstate,
	36	u32 pkru_val, enum xstate_copy_mode copy_mode);
	37	extern void copy_xstate_to_uabi_buf(struct membuf to, struct task_struct *tsk,
	38	enum xstate_copy_mode mode);
	39	extern int copy_uabi_from_kernel_to_xstate(struct fpstate fpstate, const void kbuf);
	40	extern int copy_sigframe_from_user_to_xstate(struct fpstate fpstate, const void __user ubuf);
	41
	42
	43	extern void fpu__init_cpu_xstate(void);
	44	extern void fpu__init_system_xstate(unsigned int legacy_size);
	45
	46	extern void get_xsave_addr(struct xregs_state xsave, int xfeature_nr);
	47
	48	static inline u64 xfeatures_mask_supervisor(void)
	49	{
	50	return fpu_kernel_cfg.max_features & XFEATURE_MASK_SUPERVISOR_SUPPORTED;
	51	}
	52
	53	static inline u64 xfeatures_mask_independent(void)
	54	{
	55	if (!cpu_feature_enabled(X86_FEATURE_ARCH_LBR))
	56	return XFEATURE_MASK_INDEPENDENT & ~XFEATURE_MASK_LBR;
	57
	58	return XFEATURE_MASK_INDEPENDENT;
	59	}
	60
	61	/* XSAVE/XRSTOR wrapper functions */
	62
	63	#ifdef CONFIG_X86_64
	64	#define REX_PREFIX "0x48, "
	65	#else
	66	#define REX_PREFIX
	67	#endif
	68
	69	/* These macros all use (%edi)/(%rdi) as the single memory argument. */
	70	#define XSAVE ".byte " REX_PREFIX "0x0f,0xae,0x27"
	71	#define XSAVEOPT ".byte " REX_PREFIX "0x0f,0xae,0x37"
	72	#define XSAVES ".byte " REX_PREFIX "0x0f,0xc7,0x2f"
	73	#define XRSTOR ".byte " REX_PREFIX "0x0f,0xae,0x2f"
	74	#define XRSTORS ".byte " REX_PREFIX "0x0f,0xc7,0x1f"
	75
	76	/*
	77	* After this @err contains 0 on success or the trap number when the
	78	* operation raises an exception.
	79	*/
	80	#define XSTATE_OP(op, st, lmask, hmask, err) \
	81	asm volatile("1:" op "\n\t" \
	82	"xor %[err], %[err]\n" \
	83	"2:\n\t" \
	84	_ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_FAULT_MCE_SAFE) \
	85	: [err] "=a" (err) \
	86	: "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \
	87	: "memory")
	88
	89	/*
	90	* If XSAVES is enabled, it replaces XSAVEOPT because it supports a compact
	91	* format and supervisor states in addition to modified optimization in
	92	* XSAVEOPT.
	93	*
	94	* Otherwise, if XSAVEOPT is enabled, XSAVEOPT replaces XSAVE because XSAVEOPT
	95	* supports modified optimization which is not supported by XSAVE.
	96	*
	97	* We use XSAVE as a fallback.
	98	*
	99	* The 661 label is defined in the ALTERNATIVE* macros as the address of the
	100	* original instruction which gets replaced. We need to use it here as the
	101	* address of the instruction where we might get an exception at.
	102	*/
	103	#define XSTATE_XSAVE(st, lmask, hmask, err) \
	104	asm volatile(ALTERNATIVE_2(XSAVE, \
	105	XSAVEOPT, X86_FEATURE_XSAVEOPT, \
	106	XSAVES, X86_FEATURE_XSAVES) \
	107	"\n" \
	108	"xor %[err], %[err]\n" \
	109	"3:\n" \
	110	".pushsection .fixup,\"ax\"\n" \
	111	"4: movl $-2, %[err]\n" \
	112	"jmp 3b\n" \
	113	".popsection\n" \
	114	_ASM_EXTABLE(661b, 4b) \
	115	: [err] "=r" (err) \
	116	: "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \
	117	: "memory")
	118
	119	/*
	120	* Use XRSTORS to restore context if it is enabled. XRSTORS supports compact
	121	* XSAVE area format.
	122	*/
	123	#define XSTATE_XRESTORE(st, lmask, hmask) \
	124	asm volatile(ALTERNATIVE(XRSTOR, \
	125	XRSTORS, X86_FEATURE_XSAVES) \
	126	"\n" \
	127	"3:\n" \
	128	_ASM_EXTABLE_TYPE(661b, 3b, EX_TYPE_FPU_RESTORE) \
	129	: \
	130	: "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \
	131	: "memory")
	132
	133	#if defined(CONFIG_X86_64) && defined(CONFIG_X86_DEBUG_FPU)
	134	extern void xfd_validate_state(struct fpstate *fpstate, u64 mask, bool rstor);
	135	#else
	136	static inline void xfd_validate_state(struct fpstate *fpstate, u64 mask, bool rstor) { }
	137	#endif
	138
	139	#ifdef CONFIG_X86_64
	140	static inline void xfd_update_state(struct fpstate *fpstate)
	141	{
	142	if (fpu_state_size_dynamic()) {
	143	u64 xfd = fpstate->xfd;
	144
	145	if (__this_cpu_read(xfd_state) != xfd) {
	146	wrmsrl(MSR_IA32_XFD, xfd);
	147	__this_cpu_write(xfd_state, xfd);
	148	}
	149	}
	150	}
	151	#else
	152	static inline void xfd_update_state(struct fpstate *fpstate) { }
	153	#endif
	154
	155	/*
	156	* Save processor xstate to xsave area.
	157	*
	158	* Uses either XSAVE or XSAVEOPT or XSAVES depending on the CPU features
	159	* and command line options. The choice is permanent until the next reboot.
	160	*/
	161	static inline void os_xsave(struct fpstate *fpstate)
	162	{
	163	u64 mask = fpstate->xfeatures;
	164	u32 lmask = mask;
	165	u32 hmask = mask >> 32;
	166	int err;
	167
	168	WARN_ON_FPU(!alternatives_patched);
	169	xfd_validate_state(fpstate, mask, false);
	170
	171	XSTATE_XSAVE(&fpstate->regs.xsave, lmask, hmask, err);
	172
	173	/* We should never fault when copying to a kernel buffer: */
	174	WARN_ON_FPU(err);
	175	}
	176
	177	/*
	178	* Restore processor xstate from xsave area.
	179	*
	180	* Uses XRSTORS when XSAVES is used, XRSTOR otherwise.
	181	*/
	182	static inline void os_xrstor(struct fpstate *fpstate, u64 mask)
	183	{
	184	u32 lmask = mask;
	185	u32 hmask = mask >> 32;
	186
	187	xfd_validate_state(fpstate, mask, true);
	188	XSTATE_XRESTORE(&fpstate->regs.xsave, lmask, hmask);
	189	}
	190
	191	/* Restore of supervisor state. Does not require XFD */
	192	static inline void os_xrstor_supervisor(struct fpstate *fpstate)
	193	{
	194	u64 mask = xfeatures_mask_supervisor();
	195	u32 lmask = mask;
	196	u32 hmask = mask >> 32;
	197
	198	XSTATE_XRESTORE(&fpstate->regs.xsave, lmask, hmask);
	199	}
	200
	201	/*
	202	* Save xstate to user space xsave area.
	203	*
	204	* We don't use modified optimization because xrstor/xrstors might track
	205	* a different application.
	206	*
	207	* We don't use compacted format xsave area for backward compatibility for
	208	* old applications which don't understand the compacted format of the
	209	* xsave area.
	210	*
	211	* The caller has to zero buf::header before calling this because XSAVE*
	212	* does not touch the reserved fields in the header.
	213	*/
	214	static inline int xsave_to_user_sigframe(struct xregs_state __user *buf)
	215	{
	216	/*
	217	* Include the features which are not xsaved/rstored by the kernel
	218	* internally, e.g. PKRU. That's user space ABI and also required
	219	* to allow the signal handler to modify PKRU.
	220	*/
	221	struct fpstate *fpstate = current->thread.fpu.fpstate;
	222	u64 mask = fpstate->user_xfeatures;
	223	u32 lmask = mask;
	224	u32 hmask = mask >> 32;
	225	int err;
	226
	227	xfd_validate_state(fpstate, mask, false);
	228
	229	stac();
	230	XSTATE_OP(XSAVE, buf, lmask, hmask, err);
	231	clac();
	232
	233	return err;
	234	}
	235
	236	/*
	237	* Restore xstate from user space xsave area.
	238	*/
	239	static inline int xrstor_from_user_sigframe(struct xregs_state __user *buf, u64 mask)
	240	{
	241	struct xregs_state xstate = ((__force struct xregs_state )buf);
	242	u32 lmask = mask;
	243	u32 hmask = mask >> 32;
	244	int err;
	245
	246	xfd_validate_state(current->thread.fpu.fpstate, mask, true);
	247
	248	stac();
	249	XSTATE_OP(XRSTOR, xstate, lmask, hmask, err);
	250	clac();
	251
	252	return err;
	253	}
	254
	255	/*
	256	* Restore xstate from kernel space xsave area, return an error code instead of
	257	* an exception.
	258	*/
	259	static inline int os_xrstor_safe(struct fpstate *fpstate, u64 mask)
	260	{
	261	struct xregs_state *xstate = &fpstate->regs.xsave;
	262	u32 lmask = mask;
	263	u32 hmask = mask >> 32;
	264	int err;
	265
	266	/* Ensure that XFD is up to date */
	267	xfd_update_state(fpstate);
	268
	269	if (cpu_feature_enabled(X86_FEATURE_XSAVES))
	270	XSTATE_OP(XRSTORS, xstate, lmask, hmask, err);
	271	else
	272	XSTATE_OP(XRSTOR, xstate, lmask, hmask, err);
	273
	274	return err;
	275	}
	276
	277
	278	#endif

arch/x86/kernel/process.c+23−4

 #include <asm/apic.h>
 #include <linux/uaccess.h>
 #include <asm/mwait.h>
-#include <asm/fpu/internal.h>
+#include <asm/fpu/api.h>
+#include <asm/fpu/sched.h>
+#include <asm/fpu/xstate.h>
 #include <asm/debugreg.h>
 #include <asm/nmi.h>
 #include <asm/tlbflush.h>
 #ifdef CONFIG_VM86
 	dst->thread.vm86 = NULL;
 #endif
-	return fpu_clone(dst);
+	/* Drop the copied pointer to current's fpstate */
+	dst->thread.fpu.fpstate = NULL;
+	return 0;
+}
+#ifdef CONFIG_X86_64
+void arch_release_task_struct(struct task_struct *tsk)
+{
+	if (fpu_state_size_dynamic())
+		fpstate_free(&tsk->thread.fpu);
 }
+#endif
 /*
  * Free thread data structures etc..
 	frame->flags = X86_EFLAGS_FIXED;
 #endif
+	fpu_clone(p, clone_flags);
 	/* Kernel thread ? */
 	if (unlikely(p->flags & PF_KTHREAD)) {
 		p->thread.pkru = pkru_get_init_value();
 }
 long do_arch_prctl_common(struct task_struct *task, int option,
-			  unsigned long cpuid_enabled)
+			  unsigned long arg2)
 {
 	switch (option) {
 	case ARCH_GET_CPUID:
 		return get_cpuid_mode();
 	case ARCH_SET_CPUID:
-		return set_cpuid_mode(task, cpuid_enabled);
+		return set_cpuid_mode(task, arg2);
+	case ARCH_GET_XCOMP_SUPP:
+	case ARCH_GET_XCOMP_PERM:
+	case ARCH_REQ_XCOMP_PERM:
+		return fpu_xstate_prctl(task, option, arg2);
 	}
 	return -EINVAL;

arch/x86/kernel/process_32.c+2−3

 #include <asm/ldt.h>
 #include <asm/processor.h>
-#include <asm/fpu/internal.h>
+#include <asm/fpu/sched.h>
 #include <asm/desc.h>
 #include <linux/err.h>
 	struct thread_struct *prev = &prev_p->thread,
 			     *next = &next_p->thread;
 	struct fpu *prev_fpu = &prev->fpu;
-	struct fpu *next_fpu = &next->fpu;
 	int cpu = smp_processor_id();
 	/* never put a printk in __switch_to... printk() calls wake_up*() indirectly */
 	this_cpu_write(current_task, next_p);
-	switch_fpu_finish(next_fpu);
+	switch_fpu_finish();
 	/* Load the Intel cache allocation PQR MSR. */
 	resctrl_sched_in();

arch/x86/kernel/process_64.c+2−3

 #include <asm/processor.h>
 #include <asm/pkru.h>
-#include <asm/fpu/internal.h>
+#include <asm/fpu/sched.h>
 #include <asm/mmu_context.h>
 #include <asm/prctl.h>
 #include <asm/desc.h>
 	struct thread_struct *prev = &prev_p->thread;
 	struct thread_struct *next = &next_p->thread;
 	struct fpu *prev_fpu = &prev->fpu;
-	struct fpu *next_fpu = &next->fpu;
 	int cpu = smp_processor_id();
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_ENTRY) &&
 	this_cpu_write(current_task, next_p);
 	this_cpu_write(cpu_current_top_of_stack, task_top_of_stack(next_p));
-	switch_fpu_finish(next_fpu);
+	switch_fpu_finish();
 	/* Reload sp0. */
 	update_task_stack(next_p);

arch/x86/kernel/ptrace.c+1−1

 #include <linux/uaccess.h>
 #include <asm/processor.h>
-#include <asm/fpu/internal.h>
 #include <asm/fpu/signal.h>
 #include <asm/fpu/regset.h>
+#include <asm/fpu/xstate.h>
 #include <asm/debugreg.h>
 #include <asm/ldt.h>
 #include <asm/desc.h>

arch/x86/kernel/sev.c+1−1

@@ -23,7 +23,7 @@ · 22 unmodified lines
23	#include <asm/stacktrace.h>	23	#include <asm/stacktrace.h>
24	#include <asm/sev.h>	24	#include <asm/sev.h>
25	#include <asm/insn-eval.h>	25	#include <asm/insn-eval.h>
26	#include <asm/fpu/internal.h>	26	#include <asm/fpu/xcr.h>
27	#include <asm/processor.h>	27	#include <asm/processor.h>
28	#include <asm/realmode.h>	28	#include <asm/realmode.h>
29	#include <asm/traps.h>	29	#include <asm/traps.h>

arch/x86/kernel/signal.c+71−12

 #include <linux/mm.h>
 #include <linux/smp.h>
 #include <linux/kernel.h>
+#include <linux/kstrtox.h>
 #include <linux/errno.h>
 #include <linux/wait.h>
 #include <linux/tracehook.h>
 #include <asm/processor.h>
 #include <asm/ucontext.h>
-#include <asm/fpu/internal.h>
 #include <asm/fpu/signal.h>
+#include <asm/fpu/xstate.h>
 #include <asm/vdso.h>
 #include <asm/mce.h>
 #include <asm/sighandling.h>
 #include <linux/compat.h>
 #include <asm/proto.h>
 #include <asm/ia32_unistd.h>
+#include <asm/fpu/xstate.h>
 #endif /* CONFIG_X86_64 */
 #include <asm/syscall.h>
 # define CONTEXT_COPY_SIZE	sizeof(struct sigcontext)
 #endif
-static int restore_sigcontext(struct pt_regs *regs,
+static bool restore_sigcontext(struct pt_regs *regs,
-			      struct sigcontext __user *usc,
+			       struct sigcontext __user *usc,
-			      unsigned long uc_flags)
+			       unsigned long uc_flags)
 {
 	struct sigcontext sc;
 	current->restart_block.fn = do_no_restart_syscall;
 	if (copy_from_user(&sc, usc, CONTEXT_COPY_SIZE))
-		return -EFAULT;
+		return false;
 #ifdef CONFIG_X86_32
 	set_user_gs(regs, sc.gs);
 	unsigned long math_size = 0;
 	unsigned long sp = regs->sp;
 	unsigned long buf_fx = 0;
-	int ret;
 	/* redzone */
 	if (IS_ENABLED(CONFIG_X86_64))
 	}
 	/* save i387 and extended state */
-	ret = copy_fpstate_to_sigframe(*fpstate, (void __user *)buf_fx, math_size);
+	if (!copy_fpstate_to_sigframe(*fpstate, (void __user *)buf_fx, math_size))
-	if (ret < 0)
 		return (void __user *)-1L;
 	return (void __user *)sp;
 	 * x86_32 has no uc_flags bits relevant to restore_sigcontext.
 	 * Save a few cycles by skipping the __get_user.
 	 */
-	if (restore_sigcontext(regs, &frame->sc, 0))
+	if (!restore_sigcontext(regs, &frame->sc, 0))
 		goto badframe;
 	return regs->ax;
 	set_current_blocked(&set);
-	if (restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags))
+	if (!restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags))
 		goto badframe;
 	if (restore_altstack(&frame->uc.uc_stack))
 /* max_frame_size tells userspace the worst case signal stack size. */
 static unsigned long __ro_after_init max_frame_size;
+static unsigned int __ro_after_init fpu_default_state_size;
 void __init init_sigframe_size(void)
 {
+	fpu_default_state_size = fpu__get_fpstate_size();
 	max_frame_size = MAX_FRAME_SIGINFO_UCTXT_SIZE + MAX_FRAME_PADDING;
-	max_frame_size += fpu__get_fpstate_size() + MAX_XSAVE_PADDING;
+	max_frame_size += fpu_default_state_size + MAX_XSAVE_PADDING;
 	/* Userspace expects an aligned size. */
 	max_frame_size = round_up(max_frame_size, FRAME_ALIGNMENT);
 	force_sig(SIGSEGV);
 }
+#ifdef CONFIG_DYNAMIC_SIGFRAME
+#ifdef CONFIG_STRICT_SIGALTSTACK_SIZE
+static bool strict_sigaltstack_size __ro_after_init = true;
+#else
+static bool strict_sigaltstack_size __ro_after_init = false;
+#endif
+static int __init strict_sas_size(char *arg)
+{
+	return kstrtobool(arg, &strict_sigaltstack_size);
+}
+__setup("strict_sas_size", strict_sas_size);
+/*
+ * MINSIGSTKSZ is 2048 and can't be changed despite the fact that AVX512
+ * exceeds that size already. As such programs might never use the
+ * sigaltstack they just continued to work. While always checking against
+ * the real size would be correct, this might be considered a regression.
+ *
+ * Therefore avoid the sanity check, unless enforced by kernel
+ * configuration or command line option.
+ *
+ * When dynamic FPU features are supported, the check is also enforced when
+ * the task has permissions to use dynamic features. Tasks which have no
+ * permission are checked against the size of the non-dynamic feature set
+ * if strict checking is enabled. This avoids forcing all tasks on the
+ * system to allocate large sigaltstacks even if they are never going
+ * to use a dynamic feature. As this is serialized via sighand::siglock
+ * any permission request for a dynamic feature either happened already
+ * or will see the newly install sigaltstack size in the permission checks.
+ */
+bool sigaltstack_size_valid(size_t ss_size)
+{
+	unsigned long fsize = max_frame_size - fpu_default_state_size;
+	u64 mask;
+	lockdep_assert_held(&current->sighand->siglock);
+	if (!fpu_state_size_dynamic() && !strict_sigaltstack_size)
+		return true;
+	fsize += current->group_leader->thread.fpu.perm.__user_state_size;
+	if (likely(ss_size > fsize))
+		return true;
+	if (strict_sigaltstack_size)
+		return ss_size > fsize;
+	mask = current->group_leader->thread.fpu.perm.__state_perm;
+	if (mask & XFEATURE_MASK_USER_DYNAMIC)
+		return ss_size > fsize;
+	return true;
+}
+#endif /* CONFIG_DYNAMIC_SIGFRAME */
 #ifdef CONFIG_X86_X32_ABI
 COMPAT_SYSCALL_DEFINE0(x32_rt_sigreturn)
 {
 	set_current_blocked(&set);
-	if (restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags))
+	if (!restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags))
 		goto badframe;
 	if (compat_restore_altstack(&frame->uc.uc_stack))

arch/x86/kernel/smpboot.c+1−1

@@ -70,7 +70,7 @@ · 69 unmodified lines
70	#include <asm/mwait.h>	70	#include <asm/mwait.h>
71	#include <asm/apic.h>	71	#include <asm/apic.h>
72	#include <asm/io_apic.h>	72	#include <asm/io_apic.h>
73	#include <asm/fpu/internal.h>	73	#include <asm/fpu/api.h>
74	#include <asm/setup.h>	74	#include <asm/setup.h>
75	#include <asm/uv/uv.h>	75	#include <asm/uv/uv.h>
76	#include <linux/mc146818rtc.h>	76	#include <linux/mc146818rtc.h>

arch/x86/kernel/traps.c+39−1

@@ -48,7 +48,7 @@ · 47 unmodified lines
48	#include <asm/ftrace.h>	48	#include <asm/ftrace.h>
49	#include <asm/traps.h>	49	#include <asm/traps.h>
50	#include <asm/desc.h>	50	#include <asm/desc.h>
51	#include <asm/fpu/internal.h>	51	#include <asm/fpu/api.h>
52	#include <asm/cpu.h>	52	#include <asm/cpu.h>
53	#include <asm/cpu_entry_area.h>	53	#include <asm/cpu_entry_area.h>
54	#include <asm/mce.h>	54	#include <asm/mce.h>
@@ -1108,10 +1108,48 @@ DEFINE_IDTENTRY(exc_spurious_interrupt_bug) · 1053 unmodified lines
1108	*/	1108	*/
1109	}	1109	}
1110		1110
		1111	static bool handle_xfd_event(struct pt_regs *regs)
		1112	{
		1113	u64 xfd_err;
		1114	int err;
		1115
		1116	if (!IS_ENABLED(CONFIG_X86_64) \|\| !cpu_feature_enabled(X86_FEATURE_XFD))
		1117	return false;
		1118
		1119	rdmsrl(MSR_IA32_XFD_ERR, xfd_err);
		1120	if (!xfd_err)
		1121	return false;
		1122
		1123	wrmsrl(MSR_IA32_XFD_ERR, 0);
		1124
		1125	/* Die if that happens in kernel space */
		1126	if (WARN_ON(!user_mode(regs)))
		1127	return false;
		1128
		1129	local_irq_enable();
		1130
		1131	err = xfd_enable_feature(xfd_err);
		1132
		1133	switch (err) {
		1134	case -EPERM:
		1135	force_sig_fault(SIGILL, ILL_ILLOPC, error_get_trap_addr(regs));
		1136	break;
		1137	case -EFAULT:
		1138	force_sig(SIGSEGV);
		1139	break;
		1140	}
		1141
		1142	local_irq_disable();
		1143	return true;
		1144	}
		1145
1111	DEFINE_IDTENTRY(exc_device_not_available)	1146	DEFINE_IDTENTRY(exc_device_not_available)
1112	{	1147	{
1113	unsigned long cr0 = read_cr0();	1148	unsigned long cr0 = read_cr0();
1114		1149
		1150	if (handle_xfd_event(regs))
		1151	return;
		1152
1115	#ifdef CONFIG_MATH_EMULATION	1153	#ifdef CONFIG_MATH_EMULATION
1116	if (!boot_cpu_has(X86_FEATURE_FPU) && (cr0 & X86_CR0_EM)) {	1154	if (!boot_cpu_has(X86_FEATURE_FPU) && (cr0 & X86_CR0_EM)) {
1117	struct math_emu_info info = { };	1155	struct math_emu_info info = { };

arch/x86/kvm/svm/sev.c+1−1

 #include <linux/misc_cgroup.h>
 #include <linux/processor.h>
 #include <linux/trace_events.h>
-#include <asm/fpu/internal.h>
 #include <asm/pkru.h>
 #include <asm/trapnr.h>
+#include <asm/fpu/xcr.h>
 #include "x86.h"
 #include "svm.h"

arch/x86/kvm/svm/svm.c+4−3

 #include <asm/spec-ctrl.h>
 #include <asm/cpu_device_id.h>
 #include <asm/traps.h>
+#include <asm/fpu/api.h>
 #include <asm/virtext.h>
 #include "trace.h"
 		/*
 		 * SEV-ES guests maintain an encrypted version of their FPU
 		 * state which is restored and saved on VMRUN and VMEXIT.
-		 * Free the fpu structure to prevent KVM from attempting to
+		 * Mark vcpu->arch.guest_fpu->fpstate as scratch so it won't
-		 * access the FPU state.
+		 * do xsave/xrstor on it.
 		 */
-		kvm_free_guest_fpu(vcpu);
+		fpstate_set_confidential(&vcpu->arch.guest_fpu);
 	}
 	err = avic_init_vcpu(svm);

arch/x86/kvm/vmx/vmx.c+1−1

@@ -35,7 +35,7 @@ · 34 unmodified lines
35	#include <asm/cpu_device_id.h>	35	#include <asm/cpu_device_id.h>
36	#include <asm/debugreg.h>	36	#include <asm/debugreg.h>
37	#include <asm/desc.h>	37	#include <asm/desc.h>
38	#include <asm/fpu/internal.h>	38	#include <asm/fpu/api.h>
39	#include <asm/idtentry.h>	39	#include <asm/idtentry.h>
40	#include <asm/io.h>	40	#include <asm/io.h>
41	#include <asm/irq_remapping.h>	41	#include <asm/irq_remapping.h>

arch/x86/kvm/x86.c+31−227

@@ -68,7 +68,9 @@ · 67 unmodified lines
68	#include <asm/mce.h>	68	#include <asm/mce.h>
69	#include <asm/pkru.h>	69	#include <asm/pkru.h>
70	#include <linux/kernel_stat.h>	70	#include <linux/kernel_stat.h>
71	#include <asm/fpu/internal.h> /* Ugh! */	71	#include <asm/fpu/api.h>
		72	#include <asm/fpu/xcr.h>
		73	#include <asm/fpu/xstate.h>
72	#include <asm/pvclock.h>	74	#include <asm/pvclock.h>
73	#include <asm/div64.h>	75	#include <asm/div64.h>
74	#include <asm/irq_remapping.h>	76	#include <asm/irq_remapping.h>
@@ -293,8 +295,6 @@ u64 __read_mostly host_xcr0; · 218 unmodified lines
293	u64 __read_mostly supported_xcr0;	295	u64 __read_mostly supported_xcr0;
294	EXPORT_SYMBOL_GPL(supported_xcr0);	296	EXPORT_SYMBOL_GPL(supported_xcr0);
295		297
296	static struct kmem_cache *x86_fpu_cache;
297
298	static struct kmem_cache *x86_emulator_cache;	298	static struct kmem_cache *x86_emulator_cache;
299		299
300	/*	300	/*
@@ -4700,144 +4700,27 @@ static int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu, · 4399 unmodified lines
4700	return 0;	4700	return 0;
4701	}	4701	}
4702		4702
4703	#define XSTATE_COMPACTION_ENABLED (1ULL << 63)
4704
4705	static void fill_xsave(u8 dest, struct kvm_vcpu vcpu)
4706	{
4707	struct xregs_state *xsave = &vcpu->arch.guest_fpu->state.xsave;
4708	u64 xstate_bv = xsave->header.xfeatures;
4709	u64 valid;
4710
4711	/*
4712	* Copy legacy XSAVE area, to avoid complications with CPUID
4713	* leaves 0 and 1 in the loop below.
4714	*/
4715	memcpy(dest, xsave, XSAVE_HDR_OFFSET);
4716
4717	/* Set XSTATE_BV */
4718	xstate_bv &= vcpu->arch.guest_supported_xcr0 \| XFEATURE_MASK_FPSSE;
4719	(u64 )(dest + XSAVE_HDR_OFFSET) = xstate_bv;
4720
4721	/*
4722	* Copy each region from the possibly compacted offset to the
4723	* non-compacted offset.
4724	*/
4725	valid = xstate_bv & ~XFEATURE_MASK_FPSSE;
4726	while (valid) {
4727	u32 size, offset, ecx, edx;
4728	u64 xfeature_mask = valid & -valid;
4729	int xfeature_nr = fls64(xfeature_mask) - 1;
4730	void *src;
4731
4732	cpuid_count(XSTATE_CPUID, xfeature_nr,
4733	&size, &offset, &ecx, &edx);
4734
4735	if (xfeature_nr == XFEATURE_PKRU) {
4736	memcpy(dest + offset, &vcpu->arch.pkru,
4737	sizeof(vcpu->arch.pkru));
4738	} else {
4739	src = get_xsave_addr(xsave, xfeature_nr);
4740	if (src)
4741	memcpy(dest + offset, src, size);
4742	}
4743
4744	valid -= xfeature_mask;
4745	}
4746	}
4747
4748	static void load_xsave(struct kvm_vcpu vcpu, u8 src)
4749	{
4750	struct xregs_state *xsave = &vcpu->arch.guest_fpu->state.xsave;
4751	u64 xstate_bv = (u64 )(src + XSAVE_HDR_OFFSET);
4752	u64 valid;
4753
4754	/*
4755	* Copy legacy XSAVE area, to avoid complications with CPUID
4756	* leaves 0 and 1 in the loop below.
4757	*/
4758	memcpy(xsave, src, XSAVE_HDR_OFFSET);
4759
4760	/* Set XSTATE_BV and possibly XCOMP_BV. */
4761	xsave->header.xfeatures = xstate_bv;
4762	if (boot_cpu_has(X86_FEATURE_XSAVES))
4763	xsave->header.xcomp_bv = host_xcr0 \| XSTATE_COMPACTION_ENABLED;
4764
4765	/*
4766	* Copy each region from the non-compacted offset to the
4767	* possibly compacted offset.
4768	*/
4769	valid = xstate_bv & ~XFEATURE_MASK_FPSSE;
4770	while (valid) {
4771	u32 size, offset, ecx, edx;
4772	u64 xfeature_mask = valid & -valid;
4773	int xfeature_nr = fls64(xfeature_mask) - 1;
4774
4775	cpuid_count(XSTATE_CPUID, xfeature_nr,
4776	&size, &offset, &ecx, &edx);
4777
4778	if (xfeature_nr == XFEATURE_PKRU) {
4779	memcpy(&vcpu->arch.pkru, src + offset,
4780	sizeof(vcpu->arch.pkru));
4781	} else {
4782	void *dest = get_xsave_addr(xsave, xfeature_nr);
4783
4784	if (dest)
4785	memcpy(dest, src + offset, size);
4786	}
4787
4788	valid -= xfeature_mask;
4789	}
4790	}
4791
4792	static void kvm_vcpu_ioctl_x86_get_xsave(struct kvm_vcpu *vcpu,	4703	static void kvm_vcpu_ioctl_x86_get_xsave(struct kvm_vcpu *vcpu,
4793	struct kvm_xsave *guest_xsave)	4704	struct kvm_xsave *guest_xsave)
4794	{	4705	{
4795	if (!vcpu->arch.guest_fpu)	4706	if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
4796	return;	4707	return;
4797		4708
4798	if (boot_cpu_has(X86_FEATURE_XSAVE)) {	4709	fpu_copy_guest_fpstate_to_uabi(&vcpu->arch.guest_fpu,
4799	memset(guest_xsave, 0, sizeof(struct kvm_xsave));	4710	guest_xsave->region,
4800	fill_xsave((u8 *) guest_xsave->region, vcpu);	4711	sizeof(guest_xsave->region),
4801	} else {	4712	vcpu->arch.pkru);
4802	memcpy(guest_xsave->region,
4803	&vcpu->arch.guest_fpu->state.fxsave,
4804	sizeof(struct fxregs_state));
4805	(u64 )&guest_xsave->region[XSAVE_HDR_OFFSET / sizeof(u32)] =
4806	XFEATURE_MASK_FPSSE;
4807	}
4808	}	4713	}
4809		4714
4810	#define XSAVE_MXCSR_OFFSET 24
4811
4812	static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu,	4715	static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu,
4813	struct kvm_xsave *guest_xsave)	4716	struct kvm_xsave *guest_xsave)
4814	{	4717	{
4815	u64 xstate_bv;	4718	if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
4816	u32 mxcsr;
4817
4818	if (!vcpu->arch.guest_fpu)
4819	return 0;	4719	return 0;
4820		4720
4821	xstate_bv = (u64 )&guest_xsave->region[XSAVE_HDR_OFFSET / sizeof(u32)];	4721	return fpu_copy_uabi_to_guest_fpstate(&vcpu->arch.guest_fpu,
4822	mxcsr = (u32 )&guest_xsave->region[XSAVE_MXCSR_OFFSET / sizeof(u32)];	4722	guest_xsave->region,
4823		4723	supported_xcr0, &vcpu->arch.pkru);
4824	if (boot_cpu_has(X86_FEATURE_XSAVE)) {
4825	/*
4826	* Here we allow setting states that are not present in
4827	* CPUID leaf 0xD, index 0, EDX:EAX. This is for compatibility
4828	* with old userspace.
4829	*/
4830	if (xstate_bv & ~supported_xcr0 \|\| mxcsr & ~mxcsr_feature_mask)
4831	return -EINVAL;
4832	load_xsave(vcpu, (u8 *)guest_xsave->region);
4833	} else {
4834	if (xstate_bv & ~XFEATURE_MASK_FPSSE \|\|
4835	mxcsr & ~mxcsr_feature_mask)
4836	return -EINVAL;
4837	memcpy(&vcpu->arch.guest_fpu->state.fxsave,
4838	guest_xsave->region, sizeof(struct fxregs_state));
4839	}
4840	return 0;
4841	}	4724	}
4842		4725
4843	static void kvm_vcpu_ioctl_x86_get_xcrs(struct kvm_vcpu *vcpu,	4726	static void kvm_vcpu_ioctl_x86_get_xcrs(struct kvm_vcpu *vcpu,
@@ -8434,18 +8317,11 @@ int kvm_arch_init(void *opaque) · 3590 unmodified lines
8434	}	8317	}
8435		8318
8436	r = -ENOMEM;	8319	r = -ENOMEM;
8437	x86_fpu_cache = kmem_cache_create("x86_fpu", sizeof(struct fpu),
8438	__alignof__(struct fpu), SLAB_ACCOUNT,
8439	NULL);
8440	if (!x86_fpu_cache) {
8441	printk(KERN_ERR "kvm: failed to allocate cache for x86 fpu\n");
8442	goto out;
8443	}
8444		8320
8445	x86_emulator_cache = kvm_alloc_emulator_cache();	8321	x86_emulator_cache = kvm_alloc_emulator_cache();
8446	if (!x86_emulator_cache) {	8322	if (!x86_emulator_cache) {
8447	pr_err("kvm: failed to allocate cache for x86 emulator\n");	8323	pr_err("kvm: failed to allocate cache for x86 emulator\n");
8448	goto out_free_x86_fpu_cache;	8324	goto out;
8449	}	8325	}
8450		8326
8451	user_return_msrs = alloc_percpu(struct kvm_user_return_msrs);	8327	user_return_msrs = alloc_percpu(struct kvm_user_return_msrs);
@@ -8483,8 +8359,6 @@ out_free_percpu: · 31 unmodified lines
8483	free_percpu(user_return_msrs);	8359	free_percpu(user_return_msrs);
8484	out_free_x86_emulator_cache:	8360	out_free_x86_emulator_cache:
8485	kmem_cache_destroy(x86_emulator_cache);	8361	kmem_cache_destroy(x86_emulator_cache);
8486	out_free_x86_fpu_cache:
8487	kmem_cache_destroy(x86_fpu_cache);
8488	out:	8362	out:
8489	return r;	8363	return r;
8490	}	8364	}
@@ -8511,7 +8385,6 @@ void kvm_arch_exit(void) · 20 unmodified lines
8511	kvm_mmu_module_exit();	8385	kvm_mmu_module_exit();
8512	free_percpu(user_return_msrs);	8386	free_percpu(user_return_msrs);
8513	kmem_cache_destroy(x86_emulator_cache);	8387	kmem_cache_destroy(x86_emulator_cache);
8514	kmem_cache_destroy(x86_fpu_cache);
8515	#ifdef CONFIG_KVM_XEN	8388	#ifdef CONFIG_KVM_XEN
8516	static_key_deferred_flush(&kvm_xen_enabled);	8389	static_key_deferred_flush(&kvm_xen_enabled);
8517	WARN_ON(static_branch_unlikely(&kvm_xen_enabled.key));	8390	WARN_ON(static_branch_unlikely(&kvm_xen_enabled.key));
@@ -9938,58 +9811,21 @@ static int complete_emulated_mmio(struct kvm_vcpu *vcpu) · 1420 unmodified lines
9938	return 0;	9811	return 0;
9939	}	9812	}
9940		9813
9941	static void kvm_save_current_fpu(struct fpu *fpu)
9942	{
9943	/*
9944	* If the target FPU state is not resident in the CPU registers, just
9945	* memcpy() from current, else save CPU state directly to the target.
9946	*/
9947	if (test_thread_flag(TIF_NEED_FPU_LOAD))
9948	memcpy(&fpu->state, &current->thread.fpu.state,
9949	fpu_kernel_xstate_size);
9950	else
9951	save_fpregs_to_fpstate(fpu);
9952	}
9953
9954	/* Swap (qemu) user FPU context for the guest FPU context. */	9814	/* Swap (qemu) user FPU context for the guest FPU context. */
9955	static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu)	9815	static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu)
9956	{	9816	{
9957	fpregs_lock();
9958
9959	kvm_save_current_fpu(vcpu->arch.user_fpu);
9960
9961	/*	9817	/*
9962	* Guests with protected state can't have it set by the hypervisor,	9818	* Exclude PKRU from restore as restored separately in
9963	* so skip trying to set it.	9819	* kvm_x86_ops.run().
9964	*/	9820	*/
9965	if (vcpu->arch.guest_fpu)	9821	fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, true);
9966	/* PKRU is separately restored in kvm_x86_ops.run. */
9967	__restore_fpregs_from_fpstate(&vcpu->arch.guest_fpu->state,
9968	~XFEATURE_MASK_PKRU);
9969
9970	fpregs_mark_activate();
9971	fpregs_unlock();
9972
9973	trace_kvm_fpu(1);	9822	trace_kvm_fpu(1);
9974	}	9823	}
9975		9824
9976	/* When vcpu_run ends, restore user space FPU context. */	9825	/* When vcpu_run ends, restore user space FPU context. */
9977	static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu)	9826	static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu)
9978	{	9827	{
9979	fpregs_lock();	9828	fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, false);
9980
9981	/*
9982	* Guests with protected state can't have it read by the hypervisor,
9983	* so skip trying to save it.
9984	*/
9985	if (vcpu->arch.guest_fpu)
9986	kvm_save_current_fpu(vcpu->arch.guest_fpu);
9987
9988	restore_fpregs_from_fpstate(&vcpu->arch.user_fpu->state);
9989
9990	fpregs_mark_activate();
9991	fpregs_unlock();
9992
9993	++vcpu->stat.fpu_reload;	9829	++vcpu->stat.fpu_reload;
9994	trace_kvm_fpu(0);	9830	trace_kvm_fpu(0);
9995	}	9831	}
@@ -10570,12 +10406,12 @@ int kvm_arch_vcpu_ioctl_get_fpu(struct kvm_vcpu vcpu, struct kvm_fpu fpu) · 574 unmodified lines
10570	{	10406	{
10571	struct fxregs_state *fxsave;	10407	struct fxregs_state *fxsave;
10572		10408
10573	if (!vcpu->arch.guest_fpu)	10409	if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
10574	return 0;	10410	return 0;
10575		10411
10576	vcpu_load(vcpu);	10412	vcpu_load(vcpu);
10577		10413
10578	fxsave = &vcpu->arch.guest_fpu->state.fxsave;	10414	fxsave = &vcpu->arch.guest_fpu.fpstate->regs.fxsave;
10579	memcpy(fpu->fpr, fxsave->st_space, 128);	10415	memcpy(fpu->fpr, fxsave->st_space, 128);
10580	fpu->fcw = fxsave->cwd;	10416	fpu->fcw = fxsave->cwd;
10581	fpu->fsw = fxsave->swd;	10417	fpu->fsw = fxsave->swd;
@@ -10593,12 +10429,12 @@ int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu vcpu, struct kvm_fpu fpu) · 11 unmodified lines
10593	{	10429	{
10594	struct fxregs_state *fxsave;	10430	struct fxregs_state *fxsave;
10595		10431
10596	if (!vcpu->arch.guest_fpu)	10432	if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
10597	return 0;	10433	return 0;
10598		10434
10599	vcpu_load(vcpu);	10435	vcpu_load(vcpu);
10600		10436
10601	fxsave = &vcpu->arch.guest_fpu->state.fxsave;	10437	fxsave = &vcpu->arch.guest_fpu.fpstate->regs.fxsave;
10602		10438
10603	memcpy(fxsave->st_space, fpu->fpr, 128);	10439	memcpy(fxsave->st_space, fpu->fpr, 128);
10604	fxsave->cwd = fpu->fcw;	10440	fxsave->cwd = fpu->fcw;
@@ -10651,14 +10487,6 @@ static int sync_regs(struct kvm_vcpu *vcpu) · 46 unmodified lines
10651		10487
10652	static void fx_init(struct kvm_vcpu *vcpu)	10488	static void fx_init(struct kvm_vcpu *vcpu)
10653	{	10489	{
10654	if (!vcpu->arch.guest_fpu)
10655	return;
10656
10657	fpstate_init(&vcpu->arch.guest_fpu->state);
10658	if (boot_cpu_has(X86_FEATURE_XSAVES))
10659	vcpu->arch.guest_fpu->state.xsave.header.xcomp_bv =
10660	host_xcr0 \| XSTATE_COMPACTION_ENABLED;
10661
10662	/*	10490	/*
10663	* Ensure guest xcr0 is valid for loading	10491	* Ensure guest xcr0 is valid for loading
10664	*/	10492	*/
@@ -10667,15 +10495,6 @@ static void fx_init(struct kvm_vcpu *vcpu) · 2 unmodified lines
10667	vcpu->arch.cr0 \|= X86_CR0_ET;	10495	vcpu->arch.cr0 \|= X86_CR0_ET;
10668	}	10496	}
10669		10497
10670	void kvm_free_guest_fpu(struct kvm_vcpu *vcpu)
10671	{
10672	if (vcpu->arch.guest_fpu) {
10673	kmem_cache_free(x86_fpu_cache, vcpu->arch.guest_fpu);
10674	vcpu->arch.guest_fpu = NULL;
10675	}
10676	}
10677	EXPORT_SYMBOL_GPL(kvm_free_guest_fpu);
10678
10679	int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)	10498	int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)
10680	{	10499	{
10681	if (kvm_check_tsc_unstable() && atomic_read(&kvm->online_vcpus) != 0)	10500	if (kvm_check_tsc_unstable() && atomic_read(&kvm->online_vcpus) != 0)
@@ -10732,19 +10551,11 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · 50 unmodified lines
10732	if (!alloc_emulate_ctxt(vcpu))	10551	if (!alloc_emulate_ctxt(vcpu))
10733	goto free_wbinvd_dirty_mask;	10552	goto free_wbinvd_dirty_mask;
10734		10553
10735	vcpu->arch.user_fpu = kmem_cache_zalloc(x86_fpu_cache,	10554	if (!fpu_alloc_guest_fpstate(&vcpu->arch.guest_fpu)) {
10736	GFP_KERNEL_ACCOUNT);	10555	pr_err("kvm: failed to allocate vcpu's fpu\n");
10737	if (!vcpu->arch.user_fpu) {
10738	pr_err("kvm: failed to allocate userspace's fpu\n");
10739	goto free_emulate_ctxt;	10556	goto free_emulate_ctxt;
10740	}	10557	}
10741		10558
10742	vcpu->arch.guest_fpu = kmem_cache_zalloc(x86_fpu_cache,
10743	GFP_KERNEL_ACCOUNT);
10744	if (!vcpu->arch.guest_fpu) {
10745	pr_err("kvm: failed to allocate vcpu's fpu\n");
10746	goto free_user_fpu;
10747	}
10748	fx_init(vcpu);	10559	fx_init(vcpu);
10749		10560
10750	vcpu->arch.maxphyaddr = cpuid_query_maxphyaddr(vcpu);	10561	vcpu->arch.maxphyaddr = cpuid_query_maxphyaddr(vcpu);
@@ -10777,9 +10588,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · 26 unmodified lines
10777	return 0;	10588	return 0;
10778		10589
10779	free_guest_fpu:	10590	free_guest_fpu:
10780	kvm_free_guest_fpu(vcpu);	10591	fpu_free_guest_fpstate(&vcpu->arch.guest_fpu);
10781	free_user_fpu:
10782	kmem_cache_free(x86_fpu_cache, vcpu->arch.user_fpu);
10783	free_emulate_ctxt:	10592	free_emulate_ctxt:
10784	kmem_cache_free(x86_emulator_cache, vcpu->arch.emulate_ctxt);	10593	kmem_cache_free(x86_emulator_cache, vcpu->arch.emulate_ctxt);
10785	free_wbinvd_dirty_mask:	10594	free_wbinvd_dirty_mask:
@@ -10828,8 +10637,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) · 42 unmodified lines
10828		10637
10829	kmem_cache_free(x86_emulator_cache, vcpu->arch.emulate_ctxt);	10638	kmem_cache_free(x86_emulator_cache, vcpu->arch.emulate_ctxt);
10830	free_cpumask_var(vcpu->arch.wbinvd_dirty_mask);	10639	free_cpumask_var(vcpu->arch.wbinvd_dirty_mask);
10831	kmem_cache_free(x86_fpu_cache, vcpu->arch.user_fpu);	10640	fpu_free_guest_fpstate(&vcpu->arch.guest_fpu);
10832	kvm_free_guest_fpu(vcpu);
10833		10641
10834	kvm_hv_vcpu_uninit(vcpu);	10642	kvm_hv_vcpu_uninit(vcpu);
10835	kvm_pmu_destroy(vcpu);	10643	kvm_pmu_destroy(vcpu);
@@ -10881,8 +10689,8 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) · 45 unmodified lines
10881	kvm_async_pf_hash_reset(vcpu);	10689	kvm_async_pf_hash_reset(vcpu);
10882	vcpu->arch.apf.halted = false;	10690	vcpu->arch.apf.halted = false;
10883		10691
10884	if (vcpu->arch.guest_fpu && kvm_mpx_supported()) {	10692	if (vcpu->arch.guest_fpu.fpstate && kvm_mpx_supported()) {
10885	void *mpx_state_buffer;	10693	struct fpstate *fpstate = vcpu->arch.guest_fpu.fpstate;
10886		10694
10887	/*	10695	/*
10888	* To avoid have the INIT path from kvm_apic_has_events() that be	10696	* To avoid have the INIT path from kvm_apic_has_events() that be
@@ -10890,14 +10698,10 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) · 1 unmodified line
10890	*/	10698	*/
10891	if (init_event)	10699	if (init_event)
10892	kvm_put_guest_fpu(vcpu);	10700	kvm_put_guest_fpu(vcpu);
10893	mpx_state_buffer = get_xsave_addr(&vcpu->arch.guest_fpu->state.xsave,	10701
10894	XFEATURE_BNDREGS);	10702	fpstate_clear_xstate_component(fpstate, XFEATURE_BNDREGS);
10895	if (mpx_state_buffer)	10703	fpstate_clear_xstate_component(fpstate, XFEATURE_BNDCSR);
10896	memset(mpx_state_buffer, 0, sizeof(struct mpx_bndreg_state));	10704
10897	mpx_state_buffer = get_xsave_addr(&vcpu->arch.guest_fpu->state.xsave,
10898	XFEATURE_BNDCSR);
10899	if (mpx_state_buffer)
10900	memset(mpx_state_buffer, 0, sizeof(struct mpx_bndcsr));
10901	if (init_event)	10705	if (init_event)
10902	kvm_load_guest_fpu(vcpu);	10706	kvm_load_guest_fpu(vcpu);
10903	}	10707	}

arch/x86/lib/copy_mc_64.S+4−4

@@ -107,9 +107,9 @@ SYM_FUNC_END(copy_mc_fragile) · 106 unmodified lines
107		107
108	.previous	108	.previous
109		109
110	_ASM_EXTABLE_FAULT(.L_read_leading_bytes, .E_leading_bytes)	110	_ASM_EXTABLE_TYPE(.L_read_leading_bytes, .E_leading_bytes, EX_TYPE_DEFAULT_MCE_SAFE)
111	_ASM_EXTABLE_FAULT(.L_read_words, .E_read_words)	111	_ASM_EXTABLE_TYPE(.L_read_words, .E_read_words, EX_TYPE_DEFAULT_MCE_SAFE)
112	_ASM_EXTABLE_FAULT(.L_read_trailing_bytes, .E_trailing_bytes)	112	_ASM_EXTABLE_TYPE(.L_read_trailing_bytes, .E_trailing_bytes, EX_TYPE_DEFAULT_MCE_SAFE)
113	_ASM_EXTABLE(.L_write_leading_bytes, .E_leading_bytes)	113	_ASM_EXTABLE(.L_write_leading_bytes, .E_leading_bytes)
114	_ASM_EXTABLE(.L_write_words, .E_write_words)	114	_ASM_EXTABLE(.L_write_words, .E_write_words)
115	_ASM_EXTABLE(.L_write_trailing_bytes, .E_trailing_bytes)	115	_ASM_EXTABLE(.L_write_trailing_bytes, .E_trailing_bytes)
@@ -149,5 +149,5 @@ SYM_FUNC_END(copy_mc_enhanced_fast_string) · 33 unmodified lines
149		149
150	.previous	150	.previous
151		151
152	_ASM_EXTABLE_FAULT(.L_copy, .E_copy)	152	_ASM_EXTABLE_TYPE(.L_copy, .E_copy, EX_TYPE_DEFAULT_MCE_SAFE)
153	#endif /* !CONFIG_UML */	153	#endif /* !CONFIG_UML */

arch/x86/math-emu/fpu_aux.c+1−1

@@ -53,7 +53,7 @@ void fpstate_init_soft(struct swregs_state *soft) · 52 unmodified lines
53		53
54	void finit(void)	54	void finit(void)
55	{	55	{
56	fpstate_init_soft(&current->thread.fpu.state.soft);	56	fpstate_init_soft(&current->thread.fpu.fpstate->regs.soft);
57	}	57	}
58		58
59	/*	59	/*

arch/x86/math-emu/fpu_entry.c+3−3

@@ -31,7 +31,7 @@ · 30 unmodified lines
31	#include <linux/uaccess.h>	31	#include <linux/uaccess.h>
32	#include <asm/traps.h>	32	#include <asm/traps.h>
33	#include <asm/user.h>	33	#include <asm/user.h>
34	#include <asm/fpu/internal.h>	34	#include <asm/fpu/api.h>
35		35
36	#include "fpu_system.h"	36	#include "fpu_system.h"
37	#include "fpu_emu.h"	37	#include "fpu_emu.h"
@@ -640,7 +640,7 @@ int fpregs_soft_set(struct task_struct *target, · 602 unmodified lines
640	unsigned int pos, unsigned int count,	640	unsigned int pos, unsigned int count,
641	const void kbuf, const void __user ubuf)	641	const void kbuf, const void __user ubuf)
642	{	642	{
643	struct swregs_state *s387 = &target->thread.fpu.state.soft;	643	struct swregs_state *s387 = &target->thread.fpu.fpstate->regs.soft;
644	void *space = s387->st_space;	644	void *space = s387->st_space;
645	int ret;	645	int ret;
646	int offset, other, i, tags, regnr, tag, newtop;	646	int offset, other, i, tags, regnr, tag, newtop;
@@ -691,7 +691,7 @@ int fpregs_soft_get(struct task_struct *target, · 44 unmodified lines
691	const struct user_regset *regset,	691	const struct user_regset *regset,
692	struct membuf to)	692	struct membuf to)
693	{	693	{
694	struct swregs_state *s387 = &target->thread.fpu.state.soft;	694	struct swregs_state *s387 = &target->thread.fpu.fpstate->regs.soft;
695	const void *space = s387->st_space;	695	const void *space = s387->st_space;
696	int offset = (S387->ftop & 7) * 10, other = 80 - offset;	696	int offset = (S387->ftop & 7) * 10, other = 80 - offset;
697		697

arch/x86/math-emu/fpu_system.h+1−1

@@ -73,7 +73,7 @@ static inline bool seg_writable(struct desc_struct *d) · 72 unmodified lines
73	return (d->type & SEG_TYPE_EXECUTE_MASK) == SEG_TYPE_WRITABLE;	73	return (d->type & SEG_TYPE_EXECUTE_MASK) == SEG_TYPE_WRITABLE;
74	}	74	}
75		75
76	#define I387 (&current->thread.fpu.state)	76	#define I387 (&current->thread.fpu.fpstate->regs)
77	#define FPU_info (I387->soft.info)	77	#define FPU_info (I387->soft.info)
78		78
79	#define FPU_CS ((unsigned short ) &(FPU_info->regs->cs))	79	#define FPU_CS ((unsigned short ) &(FPU_info->regs->cs))

arch/x86/mm/extable.c+56−79

 #include <linux/sched/debug.h>
 #include <xen/xen.h>
-#include <asm/fpu/internal.h>
+#include <asm/fpu/api.h>
 #include <asm/sev.h>
 #include <asm/traps.h>
 #include <asm/kdebug.h>
-typedef bool (*ex_handler_t)(const struct exception_table_entry *,
-			    struct pt_regs *, int, unsigned long,
-			    unsigned long);
 static inline unsigned long
 ex_fixup_addr(const struct exception_table_entry *x)
 {
 	return (unsigned long)&x->fixup + x->fixup;
 }
-static inline ex_handler_t
-ex_fixup_handler(const struct exception_table_entry *x)
-{
-	return (ex_handler_t)((unsigned long)&x->handler + x->handler);
-}
-__visible bool ex_handler_default(const struct exception_table_entry *fixup,
+static bool ex_handler_default(const struct exception_table_entry *fixup,
-				  struct pt_regs *regs, int trapnr,
+			       struct pt_regs *regs)
-				  unsigned long error_code,
-				  unsigned long fault_addr)
 {
 	regs->ip = ex_fixup_addr(fixup);
 	return true;
 }
-EXPORT_SYMBOL(ex_handler_default);
-__visible bool ex_handler_fault(const struct exception_table_entry *fixup,
+static bool ex_handler_fault(const struct exception_table_entry *fixup,
-				struct pt_regs *regs, int trapnr,
+			     struct pt_regs *regs, int trapnr)
-				unsigned long error_code,
-				unsigned long fault_addr)
 {
-	regs->ip = ex_fixup_addr(fixup);
 	regs->ax = trapnr;
-	return true;
+	return ex_handler_default(fixup, regs);
 }
-EXPORT_SYMBOL_GPL(ex_handler_fault);
 /*
  * Handler for when we fail to restore a task's FPU state.  We should never get
  * of vulnerability by restoring from the initial state (essentially, zeroing
  * out all the FPU registers) if we can't restore from the task's FPU state.
  */
-__visible bool ex_handler_fprestore(const struct exception_table_entry *fixup,
+static bool ex_handler_fprestore(const struct exception_table_entry *fixup,
-				    struct pt_regs *regs, int trapnr,
+				 struct pt_regs *regs)
-				    unsigned long error_code,
-				    unsigned long fault_addr)
 {
 	regs->ip = ex_fixup_addr(fixup);
 	WARN_ONCE(1, "Bad FPU state detected at %pB, reinitializing FPU registers.",
 		  (void *)instruction_pointer(regs));
-	__restore_fpregs_from_fpstate(&init_fpstate, xfeatures_mask_fpstate());
+	fpu_reset_from_exception_fixup();
 	return true;
 }
-EXPORT_SYMBOL_GPL(ex_handler_fprestore);
-__visible bool ex_handler_uaccess(const struct exception_table_entry *fixup,
+static bool ex_handler_uaccess(const struct exception_table_entry *fixup,
-				  struct pt_regs *regs, int trapnr,
+			       struct pt_regs *regs, int trapnr)
-				  unsigned long error_code,
-				  unsigned long fault_addr)
 {
 	WARN_ONCE(trapnr == X86_TRAP_GP, "General protection fault in user access. Non-canonical address?");
-	regs->ip = ex_fixup_addr(fixup);
+	return ex_handler_default(fixup, regs);
-	return true;
 }
-EXPORT_SYMBOL(ex_handler_uaccess);
-__visible bool ex_handler_copy(const struct exception_table_entry *fixup,
+static bool ex_handler_copy(const struct exception_table_entry *fixup,
-			       struct pt_regs *regs, int trapnr,
+			    struct pt_regs *regs, int trapnr)
-			       unsigned long error_code,
-			       unsigned long fault_addr)
 {
 	WARN_ONCE(trapnr == X86_TRAP_GP, "General protection fault in user access. Non-canonical address?");
-	regs->ip = ex_fixup_addr(fixup);
+	return ex_handler_fault(fixup, regs, trapnr);
-	regs->ax = trapnr;
-	return true;
 }
-EXPORT_SYMBOL(ex_handler_copy);
-__visible bool ex_handler_rdmsr_unsafe(const struct exception_table_entry *fixup,
+static bool ex_handler_rdmsr_unsafe(const struct exception_table_entry *fixup,
-				       struct pt_regs *regs, int trapnr,
+				    struct pt_regs *regs)
-				       unsigned long error_code,
-				       unsigned long fault_addr)
 {
 	if (pr_warn_once("unchecked MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n",
 			 (unsigned int)regs->cx, regs->ip, (void *)regs->ip))
 		show_stack_regs(regs);
 	/* Pretend that the read succeeded and returned 0. */
-	regs->ip = ex_fixup_addr(fixup);
 	regs->ax = 0;
 	regs->dx = 0;
-	return true;
+	return ex_handler_default(fixup, regs);
 }
-EXPORT_SYMBOL(ex_handler_rdmsr_unsafe);
-__visible bool ex_handler_wrmsr_unsafe(const struct exception_table_entry *fixup,
+static bool ex_handler_wrmsr_unsafe(const struct exception_table_entry *fixup,
-				       struct pt_regs *regs, int trapnr,
+				    struct pt_regs *regs)
-				       unsigned long error_code,
-				       unsigned long fault_addr)
 {
 	if (pr_warn_once("unchecked MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n",
 			 (unsigned int)regs->cx, (unsigned int)regs->dx,
 		show_stack_regs(regs);
 	/* Pretend that the write succeeded. */
-	regs->ip = ex_fixup_addr(fixup);
+	return ex_handler_default(fixup, regs);
-	return true;
 }
-EXPORT_SYMBOL(ex_handler_wrmsr_unsafe);
-__visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup,
+static bool ex_handler_clear_fs(const struct exception_table_entry *fixup,
-				   struct pt_regs *regs, int trapnr,
+				struct pt_regs *regs)
-				   unsigned long error_code,
-				   unsigned long fault_addr)
 {
 	if (static_cpu_has(X86_BUG_NULL_SEG))
 		asm volatile ("mov %0, %%fs" : : "rm" (__USER_DS));
 	asm volatile ("mov %0, %%fs" : : "rm" (0));
-	return ex_handler_default(fixup, regs, trapnr, error_code, fault_addr);
+	return ex_handler_default(fixup, regs);
 }
-EXPORT_SYMBOL(ex_handler_clear_fs);
-enum handler_type ex_get_fault_handler_type(unsigned long ip)
+int ex_get_fixup_type(unsigned long ip)
 {
-	const struct exception_table_entry *e;
+	const struct exception_table_entry *e = search_exception_tables(ip);
-	ex_handler_t handler;
-	e = search_exception_tables(ip);
+	return e ? e->type : EX_TYPE_NONE;
-	if (!e)
-		return EX_HANDLER_NONE;
-	handler = ex_fixup_handler(e);
-	if (handler == ex_handler_fault)
-		return EX_HANDLER_FAULT;
-	else if (handler == ex_handler_uaccess || handler == ex_handler_copy)
-		return EX_HANDLER_UACCESS;
-	else
-		return EX_HANDLER_OTHER;
 }
 int fixup_exception(struct pt_regs *regs, int trapnr, unsigned long error_code,
 		    unsigned long fault_addr)
 {
 	const struct exception_table_entry *e;
-	ex_handler_t handler;
 #ifdef CONFIG_PNPBIOS
 	if (unlikely(SEGMENT_IS_PNP_CODE(regs->cs))) {
 	if (!e)
 		return 0;
-	handler = ex_fixup_handler(e);
+	switch (e->type) {
-	return handler(e, regs, trapnr, error_code, fault_addr);
+	case EX_TYPE_DEFAULT:
+	case EX_TYPE_DEFAULT_MCE_SAFE:
+		return ex_handler_default(e, regs);
+	case EX_TYPE_FAULT:
+	case EX_TYPE_FAULT_MCE_SAFE:
+		return ex_handler_fault(e, regs, trapnr);
+	case EX_TYPE_UACCESS:
+		return ex_handler_uaccess(e, regs, trapnr);
+	case EX_TYPE_COPY:
+		return ex_handler_copy(e, regs, trapnr);
+	case EX_TYPE_CLEAR_FS:
+		return ex_handler_clear_fs(e, regs);
+	case EX_TYPE_FPU_RESTORE:
+		return ex_handler_fprestore(e, regs);
+	case EX_TYPE_RDMSR:
+		return ex_handler_rdmsr_unsafe(e, regs);
+	case EX_TYPE_WRMSR:
+		return ex_handler_wrmsr_unsafe(e, regs);
+	case EX_TYPE_BPF:
+		return ex_handler_bpf(e, regs);
+	case EX_TYPE_RDMSR_IN_MCE:
+		ex_handler_msr_mce(regs, false);
+		break;
+	case EX_TYPE_WRMSR_IN_MCE:
+		ex_handler_msr_mce(regs, true);
+		break;
+	}
+	BUG();
 }
 extern unsigned int early_recursion_flag;

arch/x86/net/bpf_jit_comp.c+2−9

 	return 0;
 }
-static bool ex_handler_bpf(const struct exception_table_entry *x,
+bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs)
-			   struct pt_regs *regs, int trapnr,
-			   unsigned long error_code, unsigned long fault_addr)
 {
 	u32 reg = x->fixup >> 8;
 				}
 				ex->insn = delta;
-				delta = (u8 *)ex_handler_bpf - (u8 *)&ex->handler;
+				ex->type = EX_TYPE_BPF;
-				if (!is_simm32(delta)) {
-					pr_err("extable->handler doesn't fit into 32-bit\n");
-					return -EFAULT;
-				}
-				ex->handler = delta;
 				if (dst_reg > BPF_REG_9) {
 					pr_err("verifier error\n");

arch/x86/power/cpu.c+1−1

@@ -20,7 +20,7 @@ · 19 unmodified lines
20	#include <asm/page.h>	20	#include <asm/page.h>
21	#include <asm/mce.h>	21	#include <asm/mce.h>
22	#include <asm/suspend.h>	22	#include <asm/suspend.h>
23	#include <asm/fpu/internal.h>	23	#include <asm/fpu/api.h>
24	#include <asm/debugreg.h>	24	#include <asm/debugreg.h>
25	#include <asm/cpu.h>	25	#include <asm/cpu.h>
26	#include <asm/mmu_context.h>	26	#include <asm/mmu_context.h>

include/linux/signal.h+6−0

 	unsafe_put_user(t->sas_ss_size, &__uss->ss_size, label); \
 } while (0);
+#ifdef CONFIG_DYNAMIC_SIGFRAME
+bool sigaltstack_size_valid(size_t ss_size);
+#else
+static inline bool sigaltstack_size_valid(size_t size) { return true; }
+#endif /* !CONFIG_DYNAMIC_SIGFRAME */
 #ifdef CONFIG_PROC_FS
 struct seq_file;
 extern void render_sigset_t(struct seq_file *, const char *, sigset_t *);

kernel/signal.c+29−6

@@ -4138,11 +4138,29 @@ int do_sigaction(int sig, struct k_sigaction act, struct k_sigaction oact) · 4137 unmodified lines
4138	return 0;	4138	return 0;
4139	}	4139	}
4140		4140
		4141	#ifdef CONFIG_DYNAMIC_SIGFRAME
		4142	static inline void sigaltstack_lock(void)
		4143	__acquires(&current->sighand->siglock)
		4144	{
		4145	spin_lock_irq(&current->sighand->siglock);
		4146	}
		4147
		4148	static inline void sigaltstack_unlock(void)
		4149	__releases(&current->sighand->siglock)
		4150	{
		4151	spin_unlock_irq(&current->sighand->siglock);
		4152	}
		4153	#else
		4154	static inline void sigaltstack_lock(void) { }
		4155	static inline void sigaltstack_unlock(void) { }
		4156	#endif
		4157
4141	static int	4158	static int
4142	do_sigaltstack (const stack_t ss, stack_t oss, unsigned long sp,	4159	do_sigaltstack (const stack_t ss, stack_t oss, unsigned long sp,
4143	size_t min_ss_size)	4160	size_t min_ss_size)
4144	{	4161	{
4145	struct task_struct *t = current;	4162	struct task_struct *t = current;
		4163	int ret = 0;
4146		4164
4147	if (oss) {	4165	if (oss) {
4148	memset(oss, 0, sizeof(stack_t));	4166	memset(oss, 0, sizeof(stack_t));
@@ -4166,19 +4184,24 @@ do_sigaltstack (const stack_t ss, stack_t oss, unsigned long sp, · 17 unmodified lines
4166	ss_mode != 0))	4184	ss_mode != 0))
4167	return -EINVAL;	4185	return -EINVAL;
4168		4186
		4187	sigaltstack_lock();
4169	if (ss_mode == SS_DISABLE) {	4188	if (ss_mode == SS_DISABLE) {
4170	ss_size = 0;	4189	ss_size = 0;
4171	ss_sp = NULL;	4190	ss_sp = NULL;
4172	} else {	4191	} else {
4173	if (unlikely(ss_size < min_ss_size))	4192	if (unlikely(ss_size < min_ss_size))
4174	return -ENOMEM;	4193	ret = -ENOMEM;
		4194	if (!sigaltstack_size_valid(ss_size))
		4195	ret = -ENOMEM;
4175	}	4196	}
4176		4197	if (!ret) {
4177	t->sas_ss_sp = (unsigned long) ss_sp;	4198	t->sas_ss_sp = (unsigned long) ss_sp;
4178	t->sas_ss_size = ss_size;	4199	t->sas_ss_size = ss_size;
4179	t->sas_ss_flags = ss_flags;	4200	t->sas_ss_flags = ss_flags;
		4201	}
		4202	sigaltstack_unlock();
4180	}	4203	}
4181	return 0;	4204	return ret;
4182	}	4205	}
4183		4206
4184	SYSCALL_DEFINE2(sigaltstack,const stack_t __user ,uss, stack_t __user ,uoss)	4207	SYSCALL_DEFINE2(sigaltstack,const stack_t __user ,uss, stack_t __user ,uoss)

scripts/sorttable.c+2−2

@@ -240,7 +240,7 @@ static void x86_sort_relative_table(char *extab_image, int image_size) · 239 unmodified lines
240		240
241	w(r(loc) + i, loc);	241	w(r(loc) + i, loc);
242	w(r(loc + 1) + i + 4, loc + 1);	242	w(r(loc + 1) + i + 4, loc + 1);
243	w(r(loc + 2) + i + 8, loc + 2);	243	/* Don't touch the fixup type */
244		244
245	i += sizeof(uint32_t) * 3;	245	i += sizeof(uint32_t) * 3;
246	}	246	}
@@ -253,7 +253,7 @@ static void x86_sort_relative_table(char *extab_image, int image_size) · 6 unmodified lines
253		253
254	w(r(loc) - i, loc);	254	w(r(loc) - i, loc);
255	w(r(loc + 1) - (i + 4), loc + 1);	255	w(r(loc + 1) - (i + 4), loc + 1);
256	w(r(loc + 2) - (i + 8), loc + 2);	256	/* Don't touch the fixup type */
257		257
258	i += sizeof(uint32_t) * 3;	258	i += sizeof(uint32_t) * 3;
259	}	259	}

tools/testing/selftests/x86/Makefile+1−1

@@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ · 17 unmodified lines
18	test_FCMOV test_FCOMI test_FISTTP \	18	test_FCMOV test_FCOMI test_FISTTP \
19	vdso_restorer	19	vdso_restorer
20	TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \	20	TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \
21	corrupt_xstate_header	21	corrupt_xstate_header amx
22	# Some selftests require 32bit support enabled also on 64bit systems	22	# Some selftests require 32bit support enabled also on 64bit systems
23	TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall	23	TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall
24		24

tools/testing/selftests/x86/amx.c+851−0

@@ -0,0 +1,851 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2
	3	#define _GNU_SOURCE
	4	#include <err.h>
	5	#include <errno.h>
	6	#include <pthread.h>
	7	#include <setjmp.h>
	8	#include <stdio.h>
	9	#include <string.h>
	10	#include <stdbool.h>
	11	#include <unistd.h>
	12	#include <x86intrin.h>
	13
	14	#include <sys/auxv.h>
	15	#include <sys/mman.h>
	16	#include <sys/shm.h>
	17	#include <sys/syscall.h>
	18	#include <sys/wait.h>
	19
	20	#ifndef __x86_64__
	21	# error This test is 64-bit only
	22	#endif
	23
	24	#define XSAVE_HDR_OFFSET 512
	25	#define XSAVE_HDR_SIZE 64
	26
	27	struct xsave_buffer {
	28	union {
	29	struct {
	30	char legacy[XSAVE_HDR_OFFSET];
	31	char header[XSAVE_HDR_SIZE];
	32	char extended[0];
	33	};
	34	char bytes[0];
	35	};
	36	};
	37
	38	static inline uint64_t xgetbv(uint32_t index)
	39	{
	40	uint32_t eax, edx;
	41
	42	asm volatile("xgetbv;"
	43	: "=a" (eax), "=d" (edx)
	44	: "c" (index));
	45	return eax + ((uint64_t)edx << 32);
	46	}
	47
	48	static inline void cpuid(uint32_t eax, uint32_t ebx, uint32_t ecx, uint32_t edx)
	49	{
	50	asm volatile("cpuid;"
	51	: "=a" (eax), "=b" (ebx), "=c" (ecx), "=d" (edx)
	52	: "0" (eax), "2" (ecx));
	53	}
	54
	55	static inline void xsave(struct xsave_buffer *xbuf, uint64_t rfbm)
	56	{
	57	uint32_t rfbm_lo = rfbm;
	58	uint32_t rfbm_hi = rfbm >> 32;
	59
	60	asm volatile("xsave (%%rdi)"
	61	: : "D" (xbuf), "a" (rfbm_lo), "d" (rfbm_hi)
	62	: "memory");
	63	}
	64
	65	static inline void xrstor(struct xsave_buffer *xbuf, uint64_t rfbm)
	66	{
	67	uint32_t rfbm_lo = rfbm;
	68	uint32_t rfbm_hi = rfbm >> 32;
	69
	70	asm volatile("xrstor (%%rdi)"
	71	: : "D" (xbuf), "a" (rfbm_lo), "d" (rfbm_hi));
	72	}
	73
	74	/* err() exits and will not return */
	75	#define fatal_error(msg, ...) err(1, "[FAIL]\t" msg, ##__VA_ARGS__)
	76
	77	static void sethandler(int sig, void (handler)(int, siginfo_t , void *),
	78	int flags)
	79	{
	80	struct sigaction sa;
	81
	82	memset(&sa, 0, sizeof(sa));
	83	sa.sa_sigaction = handler;
	84	sa.sa_flags = SA_SIGINFO \| flags;
	85	sigemptyset(&sa.sa_mask);
	86	if (sigaction(sig, &sa, 0))
	87	fatal_error("sigaction");
	88	}
	89
	90	static void clearhandler(int sig)
	91	{
	92	struct sigaction sa;
	93
	94	memset(&sa, 0, sizeof(sa));
	95	sa.sa_handler = SIG_DFL;
	96	sigemptyset(&sa.sa_mask);
	97	if (sigaction(sig, &sa, 0))
	98	fatal_error("sigaction");
	99	}
	100
	101	#define XFEATURE_XTILECFG 17
	102	#define XFEATURE_XTILEDATA 18
	103	#define XFEATURE_MASK_XTILECFG (1 << XFEATURE_XTILECFG)
	104	#define XFEATURE_MASK_XTILEDATA (1 << XFEATURE_XTILEDATA)
	105	#define XFEATURE_MASK_XTILE (XFEATURE_MASK_XTILECFG \| XFEATURE_MASK_XTILEDATA)
	106
	107	#define CPUID_LEAF1_ECX_XSAVE_MASK (1 << 26)
	108	#define CPUID_LEAF1_ECX_OSXSAVE_MASK (1 << 27)
	109	static inline void check_cpuid_xsave(void)
	110	{
	111	uint32_t eax, ebx, ecx, edx;
	112
	113	/*
	114	* CPUID.1:ECX.XSAVE[bit 26] enumerates general
	115	* support for the XSAVE feature set, including
	116	* XGETBV.
	117	*/
	118	eax = 1;
	119	ecx = 0;
	120	cpuid(&eax, &ebx, &ecx, &edx);
	121	if (!(ecx & CPUID_LEAF1_ECX_XSAVE_MASK))
	122	fatal_error("cpuid: no CPU xsave support");
	123	if (!(ecx & CPUID_LEAF1_ECX_OSXSAVE_MASK))
	124	fatal_error("cpuid: no OS xsave support");
	125	}
	126
	127	static uint32_t xbuf_size;
	128
	129	static struct {
	130	uint32_t xbuf_offset;
	131	uint32_t size;
	132	} xtiledata;
	133
	134	#define CPUID_LEAF_XSTATE 0xd
	135	#define CPUID_SUBLEAF_XSTATE_USER 0x0
	136	#define TILE_CPUID 0x1d
	137	#define TILE_PALETTE_ID 0x1
	138
	139	static void check_cpuid_xtiledata(void)
	140	{
	141	uint32_t eax, ebx, ecx, edx;
	142
	143	eax = CPUID_LEAF_XSTATE;
	144	ecx = CPUID_SUBLEAF_XSTATE_USER;
	145	cpuid(&eax, &ebx, &ecx, &edx);
	146
	147	/*
	148	* EBX enumerates the size (in bytes) required by the XSAVE
	149	* instruction for an XSAVE area containing all the user state
	150	* components corresponding to bits currently set in XCR0.
	151	*
	152	* Stash that off so it can be used to allocate buffers later.
	153	*/
	154	xbuf_size = ebx;
	155
	156	eax = CPUID_LEAF_XSTATE;
	157	ecx = XFEATURE_XTILEDATA;
	158
	159	cpuid(&eax, &ebx, &ecx, &edx);
	160	/*
	161	* eax: XTILEDATA state component size
	162	* ebx: XTILEDATA state component offset in user buffer
	163	*/
	164	if (!eax \|\| !ebx)
	165	fatal_error("xstate cpuid: invalid tile data size/offset: %d/%d",
	166	eax, ebx);
	167
	168	xtiledata.size = eax;
	169	xtiledata.xbuf_offset = ebx;
	170	}
	171
	172	/* The helpers for managing XSAVE buffer and tile states: */
	173
	174	struct xsave_buffer *alloc_xbuf(void)
	175	{
	176	struct xsave_buffer *xbuf;
	177
	178	/* XSAVE buffer should be 64B-aligned. */
	179	xbuf = aligned_alloc(64, xbuf_size);
	180	if (!xbuf)
	181	fatal_error("aligned_alloc()");
	182	return xbuf;
	183	}
	184
	185	static inline void clear_xstate_header(struct xsave_buffer *buffer)
	186	{
	187	memset(&buffer->header, 0, sizeof(buffer->header));
	188	}
	189
	190	static inline uint64_t get_xstatebv(struct xsave_buffer *buffer)
	191	{
	192	/* XSTATE_BV is at the beginning of the header: */
	193	return (uint64_t )&buffer->header;
	194	}
	195
	196	static inline void set_xstatebv(struct xsave_buffer *buffer, uint64_t bv)
	197	{
	198	/* XSTATE_BV is at the beginning of the header: */
	199	(uint64_t )(&buffer->header) = bv;
	200	}
	201
	202	static void set_rand_tiledata(struct xsave_buffer *xbuf)
	203	{
	204	int ptr = (int )&xbuf->bytes[xtiledata.xbuf_offset];
	205	int data;
	206	int i;
	207
	208	/*
	209	* Ensure that 'data' is never 0. This ensures that
	210	* the registers are never in their initial configuration
	211	* and thus never tracked as being in the init state.
	212	*/
	213	data = rand() \| 1;
	214
	215	for (i = 0; i < xtiledata.size / sizeof(int); i++, ptr++)
	216	*ptr = data;
	217	}
	218
	219	struct xsave_buffer *stashed_xsave;
	220
	221	static void init_stashed_xsave(void)
	222	{
	223	stashed_xsave = alloc_xbuf();
	224	if (!stashed_xsave)
	225	fatal_error("failed to allocate stashed_xsave\n");
	226	clear_xstate_header(stashed_xsave);
	227	}
	228
	229	static void free_stashed_xsave(void)
	230	{
	231	free(stashed_xsave);
	232	}
	233
	234	/* See 'struct _fpx_sw_bytes' at sigcontext.h */
	235	#define SW_BYTES_OFFSET 464
	236	/* N.B. The struct's field name varies so read from the offset. */
	237	#define SW_BYTES_BV_OFFSET (SW_BYTES_OFFSET + 8)
	238
	239	static inline struct _fpx_sw_bytes get_fpx_sw_bytes(void buffer)
	240	{
	241	return (struct _fpx_sw_bytes *)(buffer + SW_BYTES_OFFSET);
	242	}
	243
	244	static inline uint64_t get_fpx_sw_bytes_features(void *buffer)
	245	{
	246	return (uint64_t )(buffer + SW_BYTES_BV_OFFSET);
	247	}
	248
	249	/* Work around printf() being unsafe in signals: */
	250	#define SIGNAL_BUF_LEN 1000
	251	char signal_message_buffer[SIGNAL_BUF_LEN];
	252	void sig_print(char *msg)
	253	{
	254	int left = SIGNAL_BUF_LEN - strlen(signal_message_buffer) - 1;
	255
	256	strncat(signal_message_buffer, msg, left);
	257	}
	258
	259	static volatile bool noperm_signaled;
	260	static int noperm_errs;
	261	/*
	262	* Signal handler for when AMX is used but
	263	* permission has not been obtained.
	264	*/
	265	static void handle_noperm(int sig, siginfo_t si, void ctx_void)
	266	{
	267	ucontext_t ctx = (ucontext_t )ctx_void;
	268	void *xbuf = ctx->uc_mcontext.fpregs;
	269	struct _fpx_sw_bytes *sw_bytes;
	270	uint64_t features;
	271
	272	/* Reset the signal message buffer: */
	273	signal_message_buffer[0] = '\0';
	274	sig_print("\tAt SIGILL handler,\n");
	275
	276	if (si->si_code != ILL_ILLOPC) {
	277	noperm_errs++;
	278	sig_print("[FAIL]\tInvalid signal code.\n");
	279	} else {
	280	sig_print("[OK]\tValid signal code (ILL_ILLOPC).\n");
	281	}
	282
	283	sw_bytes = get_fpx_sw_bytes(xbuf);
	284	/*
	285	* Without permission, the signal XSAVE buffer should not
	286	* have room for AMX register state (aka. xtiledata).
	287	* Check that the size does not overlap with where xtiledata
	288	* will reside.
	289	*
	290	* This also implies that no state components PAST
	291	* XTILEDATA (features >=19) can be present in the buffer.
	292	*/
	293	if (sw_bytes->xstate_size <= xtiledata.xbuf_offset) {
	294	sig_print("[OK]\tValid xstate size\n");
	295	} else {
	296	noperm_errs++;
	297	sig_print("[FAIL]\tInvalid xstate size\n");
	298	}
	299
	300	features = get_fpx_sw_bytes_features(xbuf);
	301	/*
	302	* Without permission, the XTILEDATA feature
	303	* bit should not be set.
	304	*/
	305	if ((features & XFEATURE_MASK_XTILEDATA) == 0) {
	306	sig_print("[OK]\tValid xstate mask\n");
	307	} else {
	308	noperm_errs++;
	309	sig_print("[FAIL]\tInvalid xstate mask\n");
	310	}
	311
	312	noperm_signaled = true;
	313	ctx->uc_mcontext.gregs[REG_RIP] += 3; /* Skip the faulting XRSTOR */
	314	}
	315
	316	/* Return true if XRSTOR is successful; otherwise, false. */
	317	static inline bool xrstor_safe(struct xsave_buffer *xbuf, uint64_t mask)
	318	{
	319	noperm_signaled = false;
	320	xrstor(xbuf, mask);
	321
	322	/* Print any messages produced by the signal code: */
	323	printf("%s", signal_message_buffer);
	324	/*
	325	* Reset the buffer to make sure any future printing
	326	* only outputs new messages:
	327	*/
	328	signal_message_buffer[0] = '\0';
	329
	330	if (noperm_errs)
	331	fatal_error("saw %d errors in noperm signal handler\n", noperm_errs);
	332
	333	return !noperm_signaled;
	334	}
	335
	336	/*
	337	* Use XRSTOR to populate the XTILEDATA registers with
	338	* random data.
	339	*
	340	* Return true if successful; otherwise, false.
	341	*/
	342	static inline bool load_rand_tiledata(struct xsave_buffer *xbuf)
	343	{
	344	clear_xstate_header(xbuf);
	345	set_xstatebv(xbuf, XFEATURE_MASK_XTILEDATA);
	346	set_rand_tiledata(xbuf);
	347	return xrstor_safe(xbuf, XFEATURE_MASK_XTILEDATA);
	348	}
	349
	350	/* Return XTILEDATA to its initial configuration. */
	351	static inline void init_xtiledata(void)
	352	{
	353	clear_xstate_header(stashed_xsave);
	354	xrstor_safe(stashed_xsave, XFEATURE_MASK_XTILEDATA);
	355	}
	356
	357	enum expected_result { FAIL_EXPECTED, SUCCESS_EXPECTED };
	358
	359	/* arch_prctl() and sigaltstack() test */
	360
	361	#define ARCH_GET_XCOMP_PERM 0x1022
	362	#define ARCH_REQ_XCOMP_PERM 0x1023
	363
	364	static void req_xtiledata_perm(void)
	365	{
	366	syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA);
	367	}
	368
	369	static void validate_req_xcomp_perm(enum expected_result exp)
	370	{
	371	unsigned long bitmask;
	372	long rc;
	373
	374	rc = syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA);
	375	if (exp == FAIL_EXPECTED) {
	376	if (rc) {
	377	printf("[OK]\tARCH_REQ_XCOMP_PERM saw expected failure..\n");
	378	return;
	379	}
	380
	381	fatal_error("ARCH_REQ_XCOMP_PERM saw unexpected success.\n");
	382	} else if (rc) {
	383	fatal_error("ARCH_REQ_XCOMP_PERM saw unexpected failure.\n");
	384	}
	385
	386	rc = syscall(SYS_arch_prctl, ARCH_GET_XCOMP_PERM, &bitmask);
	387	if (rc) {
	388	fatal_error("prctl(ARCH_GET_XCOMP_PERM) error: %ld", rc);
	389	} else if (bitmask & XFEATURE_MASK_XTILE) {
	390	printf("\tARCH_REQ_XCOMP_PERM is successful.\n");
	391	}
	392	}
	393
	394	static void validate_xcomp_perm(enum expected_result exp)
	395	{
	396	bool load_success = load_rand_tiledata(stashed_xsave);
	397
	398	if (exp == FAIL_EXPECTED) {
	399	if (load_success) {
	400	noperm_errs++;
	401	printf("[FAIL]\tLoad tiledata succeeded.\n");
	402	} else {
	403	printf("[OK]\tLoad tiledata failed.\n");
	404	}
	405	} else if (exp == SUCCESS_EXPECTED) {
	406	if (load_success) {
	407	printf("[OK]\tLoad tiledata succeeded.\n");
	408	} else {
	409	noperm_errs++;
	410	printf("[FAIL]\tLoad tiledata failed.\n");
	411	}
	412	}
	413	}
	414
	415	#ifndef AT_MINSIGSTKSZ
	416	# define AT_MINSIGSTKSZ 51
	417	#endif
	418
	419	static void *alloc_altstack(unsigned int size)
	420	{
	421	void *altstack;
	422
	423	altstack = mmap(NULL, size, PROT_READ \| PROT_WRITE,
	424	MAP_PRIVATE \| MAP_ANONYMOUS \| MAP_STACK, -1, 0);
	425
	426	if (altstack == MAP_FAILED)
	427	fatal_error("mmap() for altstack");
	428
	429	return altstack;
	430	}
	431
	432	static void setup_altstack(void *addr, unsigned long size, enum expected_result exp)
	433	{
	434	stack_t ss;
	435	int rc;
	436
	437	memset(&ss, 0, sizeof(ss));
	438	ss.ss_size = size;
	439	ss.ss_sp = addr;
	440
	441	rc = sigaltstack(&ss, NULL);
	442
	443	if (exp == FAIL_EXPECTED) {
	444	if (rc) {
	445	printf("[OK]\tsigaltstack() failed.\n");
	446	} else {
	447	fatal_error("sigaltstack() succeeded unexpectedly.\n");
	448	}
	449	} else if (rc) {
	450	fatal_error("sigaltstack()");
	451	}
	452	}
	453
	454	static void test_dynamic_sigaltstack(void)
	455	{
	456	unsigned int small_size, enough_size;
	457	unsigned long minsigstksz;
	458	void *altstack;
	459
	460	minsigstksz = getauxval(AT_MINSIGSTKSZ);
	461	printf("\tAT_MINSIGSTKSZ = %lu\n", minsigstksz);
	462	/*
	463	* getauxval() itself can return 0 for failure or
	464	* success. But, in this case, AT_MINSIGSTKSZ
	465	* will always return a >=0 value if implemented.
	466	* Just check for 0.
	467	*/
	468	if (minsigstksz == 0) {
	469	printf("no support for AT_MINSIGSTKSZ, skipping sigaltstack tests\n");
	470	return;
	471	}
	472
	473	enough_size = minsigstksz * 2;
	474
	475	altstack = alloc_altstack(enough_size);
	476	printf("\tAllocate memory for altstack (%u bytes).\n", enough_size);
	477
	478	/*
	479	* Try setup_altstack() with a size which can not fit
	480	* XTILEDATA. ARCH_REQ_XCOMP_PERM should fail.
	481	*/
	482	small_size = minsigstksz - xtiledata.size;
	483	printf("\tAfter sigaltstack() with small size (%u bytes).\n", small_size);
	484	setup_altstack(altstack, small_size, SUCCESS_EXPECTED);
	485	validate_req_xcomp_perm(FAIL_EXPECTED);
	486
	487	/*
	488	* Try setup_altstack() with a size derived from
	489	* AT_MINSIGSTKSZ. It should be more than large enough
	490	* and thus ARCH_REQ_XCOMP_PERM should succeed.
	491	*/
	492	printf("\tAfter sigaltstack() with enough size (%u bytes).\n", enough_size);
	493	setup_altstack(altstack, enough_size, SUCCESS_EXPECTED);
	494	validate_req_xcomp_perm(SUCCESS_EXPECTED);
	495
	496	/*
	497	* Try to coerce setup_altstack() to again accept a
	498	* too-small altstack. This ensures that big-enough
	499	* sigaltstacks can not shrink to a too-small value
	500	* once XTILEDATA permission is established.
	501	*/
	502	printf("\tThen, sigaltstack() with small size (%u bytes).\n", small_size);
	503	setup_altstack(altstack, small_size, FAIL_EXPECTED);
	504	}
	505
	506	static void test_dynamic_state(void)
	507	{
	508	pid_t parent, child, grandchild;
	509
	510	parent = fork();
	511	if (parent < 0) {
	512	/* fork() failed */
	513	fatal_error("fork");
	514	} else if (parent > 0) {
	515	int status;
	516	/* fork() succeeded. Now in the parent. */
	517
	518	wait(&status);
	519	if (!WIFEXITED(status) \|\| WEXITSTATUS(status))
	520	fatal_error("arch_prctl test parent exit");
	521	return;
	522	}
	523	/* fork() succeeded. Now in the child . */
	524
	525	printf("[RUN]\tCheck ARCH_REQ_XCOMP_PERM around process fork() and sigaltack() test.\n");
	526
	527	printf("\tFork a child.\n");
	528	child = fork();
	529	if (child < 0) {
	530	fatal_error("fork");
	531	} else if (child > 0) {
	532	int status;
	533
	534	wait(&status);
	535	if (!WIFEXITED(status) \|\| WEXITSTATUS(status))
	536	fatal_error("arch_prctl test child exit");
	537	_exit(0);
	538	}
	539
	540	/*
	541	* The permission request should fail without an
	542	* XTILEDATA-compatible signal stack
	543	*/
	544	printf("\tTest XCOMP_PERM at child.\n");
	545	validate_xcomp_perm(FAIL_EXPECTED);
	546
	547	/*
	548	* Set up an XTILEDATA-compatible signal stack and
	549	* also obtain permission to populate XTILEDATA.
	550	*/
	551	printf("\tTest dynamic sigaltstack at child:\n");
	552	test_dynamic_sigaltstack();
	553
	554	/* Ensure that XTILEDATA can be populated. */
	555	printf("\tTest XCOMP_PERM again at child.\n");
	556	validate_xcomp_perm(SUCCESS_EXPECTED);
	557
	558	printf("\tFork a grandchild.\n");
	559	grandchild = fork();
	560	if (grandchild < 0) {
	561	/* fork() failed */
	562	fatal_error("fork");
	563	} else if (!grandchild) {
	564	/* fork() succeeded. Now in the (grand)child. */
	565	printf("\tTest XCOMP_PERM at grandchild.\n");
	566
	567	/*
	568	* Ensure that the grandchild inherited
	569	* permission and a compatible sigaltstack:
	570	*/
	571	validate_xcomp_perm(SUCCESS_EXPECTED);
	572	} else {
	573	int status;
	574	/* fork() succeeded. Now in the parent. */
	575
	576	wait(&status);
	577	if (!WIFEXITED(status) \|\| WEXITSTATUS(status))
	578	fatal_error("fork test grandchild");
	579	}
	580
	581	_exit(0);
	582	}
	583
	584	/*
	585	* Save current register state and compare it to @xbuf1.'
	586	*
	587	* Returns false if @xbuf1 matches the registers.
	588	* Returns true if @xbuf1 differs from the registers.
	589	*/
	590	static inline bool __validate_tiledata_regs(struct xsave_buffer *xbuf1)
	591	{
	592	struct xsave_buffer *xbuf2;
	593	int ret;
	594
	595	xbuf2 = alloc_xbuf();
	596	if (!xbuf2)
	597	fatal_error("failed to allocate XSAVE buffer\n");
	598
	599	xsave(xbuf2, XFEATURE_MASK_XTILEDATA);
	600	ret = memcmp(&xbuf1->bytes[xtiledata.xbuf_offset],
	601	&xbuf2->bytes[xtiledata.xbuf_offset],
	602	xtiledata.size);
	603
	604	free(xbuf2);
	605
	606	if (ret == 0)
	607	return false;
	608	return true;
	609	}
	610
	611	static inline void validate_tiledata_regs_same(struct xsave_buffer *xbuf)
	612	{
	613	int ret = __validate_tiledata_regs(xbuf);
	614
	615	if (ret != 0)
	616	fatal_error("TILEDATA registers changed");
	617	}
	618
	619	static inline void validate_tiledata_regs_changed(struct xsave_buffer *xbuf)
	620	{
	621	int ret = __validate_tiledata_regs(xbuf);
	622
	623	if (ret == 0)
	624	fatal_error("TILEDATA registers did not change");
	625	}
	626
	627	/* tiledata inheritance test */
	628
	629	static void test_fork(void)
	630	{
	631	pid_t child, grandchild;
	632
	633	child = fork();
	634	if (child < 0) {
	635	/* fork() failed */
	636	fatal_error("fork");
	637	} else if (child > 0) {
	638	/* fork() succeeded. Now in the parent. */
	639	int status;
	640
	641	wait(&status);
	642	if (!WIFEXITED(status) \|\| WEXITSTATUS(status))
	643	fatal_error("fork test child");
	644	return;
	645	}
	646	/* fork() succeeded. Now in the child. */
	647	printf("[RUN]\tCheck tile data inheritance.\n\tBefore fork(), load tiledata\n");
	648
	649	load_rand_tiledata(stashed_xsave);
	650
	651	grandchild = fork();
	652	if (grandchild < 0) {
	653	/* fork() failed */
	654	fatal_error("fork");
	655	} else if (grandchild > 0) {
	656	/* fork() succeeded. Still in the first child. */
	657	int status;
	658
	659	wait(&status);
	660	if (!WIFEXITED(status) \|\| WEXITSTATUS(status))
	661	fatal_error("fork test grand child");
	662	_exit(0);
	663	}
	664	/* fork() succeeded. Now in the (grand)child. */
	665
	666	/*
	667	* TILEDATA registers are not preserved across fork().
	668	* Ensure that their value has changed:
	669	*/
	670	validate_tiledata_regs_changed(stashed_xsave);
	671
	672	_exit(0);
	673	}
	674
	675	/* Context switching test */
	676
	677	static struct _ctxtswtest_cfg {
	678	unsigned int iterations;
	679	unsigned int num_threads;
	680	} ctxtswtest_config;
	681
	682	struct futex_info {
	683	pthread_t thread;
	684	int nr;
	685	pthread_mutex_t mutex;
	686	struct futex_info *next;
	687	};
	688
	689	static void check_tiledata(void info)
	690	{
	691	struct futex_info finfo = (struct futex_info )info;
	692	struct xsave_buffer *xbuf;
	693	int i;
	694
	695	xbuf = alloc_xbuf();
	696	if (!xbuf)
	697	fatal_error("unable to allocate XSAVE buffer");
	698
	699	/*
	700	* Load random data into 'xbuf' and then restore
	701	* it to the tile registers themselves.
	702	*/
	703	load_rand_tiledata(xbuf);
	704	for (i = 0; i < ctxtswtest_config.iterations; i++) {
	705	pthread_mutex_lock(&finfo->mutex);
	706
	707	/*
	708	* Ensure the register values have not
	709	* diverged from those recorded in 'xbuf'.
	710	*/
	711	validate_tiledata_regs_same(xbuf);
	712
	713	/* Load new, random values into xbuf and registers */
	714	load_rand_tiledata(xbuf);
	715
	716	/*
	717	* The last thread's last unlock will be for
	718	* thread 0's mutex. However, thread 0 will
	719	* have already exited the loop and the mutex
	720	* will already be unlocked.
	721	*
	722	* Because this is not an ERRORCHECK mutex,
	723	* that inconsistency will be silently ignored.
	724	*/
	725	pthread_mutex_unlock(&finfo->next->mutex);
	726	}
	727
	728	free(xbuf);
	729	/*
	730	* Return this thread's finfo, which is
	731	* a unique value for this thread.
	732	*/
	733	return finfo;
	734	}
	735
	736	static int create_threads(int num, struct futex_info *finfo)
	737	{
	738	int i;
	739
	740	for (i = 0; i < num; i++) {
	741	int next_nr;
	742
	743	finfo[i].nr = i;
	744	/*
	745	* Thread 'i' will wait on this mutex to
	746	* be unlocked. Lock it immediately after
	747	* initialization:
	748	*/
	749	pthread_mutex_init(&finfo[i].mutex, NULL);
	750	pthread_mutex_lock(&finfo[i].mutex);
	751
	752	next_nr = (i + 1) % num;
	753	finfo[i].next = &finfo[next_nr];
	754
	755	if (pthread_create(&finfo[i].thread, NULL, check_tiledata, &finfo[i]))
	756	fatal_error("pthread_create()");
	757	}
	758	return 0;
	759	}
	760
	761	static void affinitize_cpu0(void)
	762	{
	763	cpu_set_t cpuset;
	764
	765	CPU_ZERO(&cpuset);
	766	CPU_SET(0, &cpuset);
	767
	768	if (sched_setaffinity(0, sizeof(cpuset), &cpuset) != 0)
	769	fatal_error("sched_setaffinity to CPU 0");
	770	}
	771
	772	static void test_context_switch(void)
	773	{
	774	struct futex_info *finfo;
	775	int i;
	776
	777	/* Affinitize to one CPU to force context switches */
	778	affinitize_cpu0();
	779
	780	req_xtiledata_perm();
	781
	782	printf("[RUN]\tCheck tiledata context switches, %d iterations, %d threads.\n",
	783	ctxtswtest_config.iterations,
	784	ctxtswtest_config.num_threads);
	785
	786
	787	finfo = malloc(sizeof(finfo) ctxtswtest_config.num_threads);
	788	if (!finfo)
	789	fatal_error("malloc()");
	790
	791	create_threads(ctxtswtest_config.num_threads, finfo);
	792
	793	/*
	794	* This thread wakes up thread 0
	795	* Thread 0 will wake up 1
	796	* Thread 1 will wake up 2
	797	* ...
	798	* the last thread will wake up 0
	799	*
	800	* ... this will repeat for the configured
	801	* number of iterations.
	802	*/
	803	pthread_mutex_unlock(&finfo[0].mutex);
	804
	805	/* Wait for all the threads to finish: */
	806	for (i = 0; i < ctxtswtest_config.num_threads; i++) {
	807	void *thread_retval;
	808	int rc;
	809
	810	rc = pthread_join(finfo[i].thread, &thread_retval);
	811
	812	if (rc)
	813	fatal_error("pthread_join() failed for thread %d err: %d\n",
	814	i, rc);
	815
	816	if (thread_retval != &finfo[i])
	817	fatal_error("unexpected thread retval for thread %d: %p\n",
	818	i, thread_retval);
	819
	820	}
	821
	822	printf("[OK]\tNo incorrect case was found.\n");
	823
	824	free(finfo);
	825	}
	826
	827	int main(void)
	828	{
	829	/* Check hardware availability at first */
	830	check_cpuid_xsave();
	831	check_cpuid_xtiledata();
	832
	833	init_stashed_xsave();
	834	sethandler(SIGILL, handle_noperm, 0);
	835
	836	test_dynamic_state();
	837
	838	/* Request permission for the following tests */
	839	req_xtiledata_perm();
	840
	841	test_fork();
	842
	843	ctxtswtest_config.iterations = 10;
	844	ctxtswtest_config.num_threads = 5;
	845	test_context_switch();
	846
	847	clearhandler(SIGILL);
	848	free_stashed_xsave();
	849
	850	return 0;
	851	}

Merge tag 'x86-fpu-2021-11-01' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code