lael-labs/linux

Documentation/arm64/amu.rst+112−0

@@ -0,0 +1,112 @@
	1	=======================================================
	2	Activity Monitors Unit (AMU) extension in AArch64 Linux
	3	=======================================================
	4
	5	Author: Ionela Voinescu <ionela.voinescu@arm.com>
	6
	7	Date: 2019-09-10
	8
	9	This document briefly describes the provision of Activity Monitors Unit
	10	support in AArch64 Linux.
	11
	12
	13	Architecture overview
	14	---------------------
	15
	16	The activity monitors extension is an optional extension introduced by the
	17	ARMv8.4 CPU architecture.
	18
	19	The activity monitors unit, implemented in each CPU, provides performance
	20	counters intended for system management use. The AMU extension provides a
	21	system register interface to the counter registers and also supports an
	22	optional external memory-mapped interface.
	23
	24	Version 1 of the Activity Monitors architecture implements a counter group
	25	of four fixed and architecturally defined 64-bit event counters.
	26	- CPU cycle counter: increments at the frequency of the CPU.
	27	- Constant counter: increments at the fixed frequency of the system
	28	clock.
	29	- Instructions retired: increments with every architecturally executed
	30	instruction.
	31	- Memory stall cycles: counts instruction dispatch stall cycles caused by
	32	misses in the last level cache within the clock domain.
	33
	34	When in WFI or WFE these counters do not increment.
	35
	36	The Activity Monitors architecture provides space for up to 16 architected
	37	event counters. Future versions of the architecture may use this space to
	38	implement additional architected event counters.
	39
	40	Additionally, version 1 implements a counter group of up to 16 auxiliary
	41	64-bit event counters.
	42
	43	On cold reset all counters reset to 0.
	44
	45
	46	Basic support
	47	-------------
	48
	49	The kernel can safely run a mix of CPUs with and without support for the
	50	activity monitors extension. Therefore, when CONFIG_ARM64_AMU_EXTN is
	51	selected we unconditionally enable the capability to allow any late CPU
	52	(secondary or hotplugged) to detect and use the feature.
	53
	54	When the feature is detected on a CPU, we flag the availability of the
	55	feature but this does not guarantee the correct functionality of the
	56	counters, only the presence of the extension.
	57
	58	Firmware (code running at higher exception levels, e.g. arm-tf) support is
	59	needed to:
	60	- Enable access for lower exception levels (EL2 and EL1) to the AMU
	61	registers.
	62	- Enable the counters. If not enabled these will read as 0.
	63	- Save/restore the counters before/after the CPU is being put/brought up
	64	from the 'off' power state.
	65
	66	When using kernels that have this feature enabled but boot with broken
	67	firmware the user may experience panics or lockups when accessing the
	68	counter registers. Even if these symptoms are not observed, the values
	69	returned by the register reads might not correctly reflect reality. Most
	70	commonly, the counters will read as 0, indicating that they are not
	71	enabled.
	72
	73	If proper support is not provided in firmware it's best to disable
	74	CONFIG_ARM64_AMU_EXTN. To be noted that for security reasons, this does not
	75	bypass the setting of AMUSERENR_EL0 to trap accesses from EL0 (userspace) to
	76	EL1 (kernel). Therefore, firmware should still ensure accesses to AMU registers
	77	are not trapped in EL2/EL3.
	78
	79	The fixed counters of AMUv1 are accessible though the following system
	80	register definitions:
	81	- SYS_AMEVCNTR0_CORE_EL0
	82	- SYS_AMEVCNTR0_CONST_EL0
	83	- SYS_AMEVCNTR0_INST_RET_EL0
	84	- SYS_AMEVCNTR0_MEM_STALL_EL0
	85
	86	Auxiliary platform specific counters can be accessed using
	87	SYS_AMEVCNTR1_EL0(n), where n is a value between 0 and 15.
	88
	89	Details can be found in: arch/arm64/include/asm/sysreg.h.
	90
	91
	92	Userspace access
	93	----------------
	94
	95	Currently, access from userspace to the AMU registers is disabled due to:
	96	- Security reasons: they might expose information about code executed in
	97	secure mode.
	98	- Purpose: AMU counters are intended for system management use.
	99
	100	Also, the presence of the feature is not visible to userspace.
	101
	102
	103	Virtualization
	104	--------------
	105
	106	Currently, access from userspace (EL0) and kernelspace (EL1) on the KVM
	107	guest side is disabled due to:
	108	- Security reasons: they might expose information about code executed
	109	by other guests or the host.
	110
	111	Any attempt to access the AMU registers will result in an UNDEFINED
	112	exception being injected into the guest.

Documentation/arm64/booting.rst+14−0

@@ -248,6 +248,20 @@ Before jumping into the kernel, the following conditions must be met: · 247 unmodified lines
248	- HCR_EL2.APK (bit 40) must be initialised to 0b1	248	- HCR_EL2.APK (bit 40) must be initialised to 0b1
249	- HCR_EL2.API (bit 41) must be initialised to 0b1	249	- HCR_EL2.API (bit 41) must be initialised to 0b1
250		250
		251	For CPUs with Activity Monitors Unit v1 (AMUv1) extension present:
		252	- If EL3 is present:
		253	CPTR_EL3.TAM (bit 30) must be initialised to 0b0
		254	CPTR_EL2.TAM (bit 30) must be initialised to 0b0
		255	AMCNTENSET0_EL0 must be initialised to 0b1111
		256	AMCNTENSET1_EL0 must be initialised to a platform specific value
		257	having 0b1 set for the corresponding bit for each of the auxiliary
		258	counters present.
		259	- If the kernel is entered at EL1:
		260	AMCNTENSET0_EL0 must be initialised to 0b1111
		261	AMCNTENSET1_EL0 must be initialised to a platform specific value
		262	having 0b1 set for the corresponding bit for each of the auxiliary
		263	counters present.
		264
251	The requirements described above for CPU mode, caches, MMUs, architected	265	The requirements described above for CPU mode, caches, MMUs, architected
252	timers, coherency and system registers apply to all CPUs. All CPUs must	266	timers, coherency and system registers apply to all CPUs. All CPUs must
253	enter the kernel in the same exception level.	267	enter the kernel in the same exception level.

Documentation/arm64/index.rst+1−0

@@ -6,6 +6,7 @@ ARM64 Architecture · 5 unmodified lines
6	:maxdepth: 1	6	:maxdepth: 1
7		7
8	acpi_object_usage	8	acpi_object_usage
		9	amu
9	arm-acpi	10	arm-acpi
10	booting	11	booting
11	cpu-feature-registers	12	cpu-feature-registers

arch/arm64/Kconfig+66−3

@@ -117,6 +117,7 @@ config ARM64 · 116 unmodified lines
117	select HAVE_ALIGNED_STRUCT_PAGE if SLUB	117	select HAVE_ALIGNED_STRUCT_PAGE if SLUB
118	select HAVE_ARCH_AUDITSYSCALL	118	select HAVE_ARCH_AUDITSYSCALL
119	select HAVE_ARCH_BITREVERSE	119	select HAVE_ARCH_BITREVERSE
		120	select HAVE_ARCH_COMPILER_H
120	select HAVE_ARCH_HUGE_VMAP	121	select HAVE_ARCH_HUGE_VMAP
121	select HAVE_ARCH_JUMP_LABEL	122	select HAVE_ARCH_JUMP_LABEL
122	select HAVE_ARCH_JUMP_LABEL_RELATIVE	123	select HAVE_ARCH_JUMP_LABEL_RELATIVE
@@ -280,6 +281,9 @@ config ZONE_DMA32 · 157 unmodified lines
280	config ARCH_ENABLE_MEMORY_HOTPLUG	281	config ARCH_ENABLE_MEMORY_HOTPLUG
281	def_bool y	282	def_bool y
282		283
		284	config ARCH_ENABLE_MEMORY_HOTREMOVE
		285	def_bool y
		286
283	config SMP	287	config SMP
284	def_bool y	288	def_bool y
285		289
@@ -951,11 +955,11 @@ config HOTPLUG_CPU · 665 unmodified lines
951		955
952	# Common NUMA Features	956	# Common NUMA Features
953	config NUMA	957	config NUMA
954	bool "Numa Memory Allocation and Scheduler Support"	958	bool "NUMA Memory Allocation and Scheduler Support"
955	select ACPI_NUMA if ACPI	959	select ACPI_NUMA if ACPI
956	select OF_NUMA	960	select OF_NUMA
957	help	961	help
958	Enable NUMA (Non Uniform Memory Access) support.	962	Enable NUMA (Non-Uniform Memory Access) support.
959		963
960	The kernel will try to allocate memory used by a CPU on the	964	The kernel will try to allocate memory used by a CPU on the
961	local memory of the CPU and add some more	965	local memory of the CPU and add some more
@@ -1497,6 +1501,9 @@ config ARM64_PTR_AUTH · 535 unmodified lines
1497	bool "Enable support for pointer authentication"	1501	bool "Enable support for pointer authentication"
1498	default y	1502	default y
1499	depends on !KVM \|\| ARM64_VHE	1503	depends on !KVM \|\| ARM64_VHE
		1504	depends on (CC_HAS_SIGN_RETURN_ADDRESS \|\| CC_HAS_BRANCH_PROT_PAC_RET) && AS_HAS_PAC
		1505	depends on CC_IS_GCC \|\| (CC_IS_CLANG && AS_HAS_CFI_NEGATE_RA_STATE)
		1506	depends on (!FUNCTION_GRAPH_TRACER \|\| DYNAMIC_FTRACE_WITH_REGS)
1500	help	1507	help
1501	Pointer authentication (part of the ARMv8.3 Extensions) provides	1508	Pointer authentication (part of the ARMv8.3 Extensions) provides
1502	instructions for signing and authenticating pointers against secret	1509	instructions for signing and authenticating pointers against secret
@@ -1504,16 +1511,72 @@ config ARM64_PTR_AUTH · 1 unmodified line
1504	and other attacks.	1511	and other attacks.
1505		1512
1506	This option enables these instructions at EL0 (i.e. for userspace).	1513	This option enables these instructions at EL0 (i.e. for userspace).
1507
1508	Choosing this option will cause the kernel to initialise secret keys	1514	Choosing this option will cause the kernel to initialise secret keys
1509	for each process at exec() time, with these keys being	1515	for each process at exec() time, with these keys being
1510	context-switched along with the process.	1516	context-switched along with the process.
1511		1517
		1518	If the compiler supports the -mbranch-protection or
		1519	-msign-return-address flag (e.g. GCC 7 or later), then this option
		1520	will also cause the kernel itself to be compiled with return address
		1521	protection. In this case, and if the target hardware is known to
		1522	support pointer authentication, then CONFIG_STACKPROTECTOR can be
		1523	disabled with minimal loss of protection.
		1524
1512	The feature is detected at runtime. If the feature is not present in	1525	The feature is detected at runtime. If the feature is not present in
1513	hardware it will not be advertised to userspace/KVM guest nor will it	1526	hardware it will not be advertised to userspace/KVM guest nor will it
1514	be enabled. However, KVM guest also require VHE mode and hence	1527	be enabled. However, KVM guest also require VHE mode and hence
1515	CONFIG_ARM64_VHE=y option to use this feature.	1528	CONFIG_ARM64_VHE=y option to use this feature.
1516		1529
		1530	If the feature is present on the boot CPU but not on a late CPU, then
		1531	the late CPU will be parked. Also, if the boot CPU does not have
		1532	address auth and the late CPU has then the late CPU will still boot
		1533	but with the feature disabled. On such a system, this option should
		1534	not be selected.
		1535
		1536	This feature works with FUNCTION_GRAPH_TRACER option only if
		1537	DYNAMIC_FTRACE_WITH_REGS is enabled.
		1538
		1539	config CC_HAS_BRANCH_PROT_PAC_RET
		1540	# GCC 9 or later, clang 8 or later
		1541	def_bool $(cc-option,-mbranch-protection=pac-ret+leaf)
		1542
		1543	config CC_HAS_SIGN_RETURN_ADDRESS
		1544	# GCC 7, 8
		1545	def_bool $(cc-option,-msign-return-address=all)
		1546
		1547	config AS_HAS_PAC
		1548	def_bool $(as-option,-Wa$(comma)-march=armv8.3-a)
		1549
		1550	config AS_HAS_CFI_NEGATE_RA_STATE
		1551	def_bool $(as-instr,.cfi_startproc\n.cfi_negate_ra_state\n.cfi_endproc\n)
		1552
		1553	endmenu
		1554
		1555	menu "ARMv8.4 architectural features"
		1556
		1557	config ARM64_AMU_EXTN
		1558	bool "Enable support for the Activity Monitors Unit CPU extension"
		1559	default y
		1560	help
		1561	The activity monitors extension is an optional extension introduced
		1562	by the ARMv8.4 CPU architecture. This enables support for version 1
		1563	of the activity monitors architecture, AMUv1.
		1564
		1565	To enable the use of this extension on CPUs that implement it, say Y.
		1566
		1567	Note that for architectural reasons, firmware _must_ implement AMU
		1568	support when running on CPUs that present the activity monitors
		1569	extension. The required support is present in:
		1570	* Version 1.5 and later of the ARM Trusted Firmware
		1571
		1572	For kernels that have this configuration enabled but boot with broken
		1573	firmware, you may need to say N here until the firmware is fixed.
		1574	Otherwise you may experience firmware panics or lockups when
		1575	accessing the counter registers. Even if you are not observing these
		1576	symptoms, the values returned by the register reads might not
		1577	correctly reflect reality. Most commonly, the value read will be 0,
		1578	indicating that the counter is not enabled.
		1579
1517	endmenu	1580	endmenu
1518		1581
1519	menu "ARMv8.5 architectural features"	1582	menu "ARMv8.5 architectural features"

arch/arm64/Makefile+11−0

@@ -65,6 +65,17 @@ stack_protector_prepare: prepare0 · 64 unmodified lines
65	include/generated/asm-offsets.h))	65	include/generated/asm-offsets.h))
66	endif	66	endif
67		67
		68	ifeq ($(CONFIG_ARM64_PTR_AUTH),y)
		69	branch-prot-flags-$(CONFIG_CC_HAS_SIGN_RETURN_ADDRESS) := -msign-return-address=all
		70	branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET) := -mbranch-protection=pac-ret+leaf
		71	# -march=armv8.3-a enables the non-nops instructions for PAC, to avoid the
		72	# compiler to generate them and consequently to break the single image contract
		73	# we pass it only to the assembler. This option is utilized only in case of non
		74	# integrated assemblers.
		75	branch-prot-flags-$(CONFIG_AS_HAS_PAC) += -Wa,-march=armv8.3-a
		76	KBUILD_CFLAGS += $(branch-prot-flags-y)
		77	endif
		78
68	ifeq ($(CONFIG_CPU_BIG_ENDIAN), y)	79	ifeq ($(CONFIG_CPU_BIG_ENDIAN), y)
69	KBUILD_CPPFLAGS += -mbig-endian	80	KBUILD_CPPFLAGS += -mbig-endian
70	CHECKFLAGS += -D__AARCH64EB__	81	CHECKFLAGS += -D__AARCH64EB__

arch/arm64/crypto/aes-ce.S+2−2

@@ -9,8 +9,8 @@ · 8 unmodified lines
9	#include <linux/linkage.h>	9	#include <linux/linkage.h>
10	#include <asm/assembler.h>	10	#include <asm/assembler.h>
11		11
12	#define AES_ENTRY(func) SYM_FUNC_START(ce_ ## func)	12	#define AES_FUNC_START(func) SYM_FUNC_START(ce_ ## func)
13	#define AES_ENDPROC(func) SYM_FUNC_END(ce_ ## func)	13	#define AES_FUNC_END(func) SYM_FUNC_END(ce_ ## func)
14		14
15	.arch armv8-a+crypto	15	.arch armv8-a+crypto
16		16

arch/arm64/crypto/aes-modes.S+24−24

@@ -51,7 +51,7 @@ SYM_FUNC_END(aes_decrypt_block5x) · 50 unmodified lines
51	* int blocks)	51	* int blocks)
52	*/	52	*/
53		53
54	AES_ENTRY(aes_ecb_encrypt)	54	AES_FUNC_START(aes_ecb_encrypt)
55	stp x29, x30, [sp, #-16]!	55	stp x29, x30, [sp, #-16]!
56	mov x29, sp	56	mov x29, sp
57		57
@@ -79,10 +79,10 @@ ST5( st1 {v4.16b}, [x0], #16 ) · 21 unmodified lines
79	.Lecbencout:	79	.Lecbencout:
80	ldp x29, x30, [sp], #16	80	ldp x29, x30, [sp], #16
81	ret	81	ret
82	AES_ENDPROC(aes_ecb_encrypt)	82	AES_FUNC_END(aes_ecb_encrypt)
83		83
84		84
85	AES_ENTRY(aes_ecb_decrypt)	85	AES_FUNC_START(aes_ecb_decrypt)
86	stp x29, x30, [sp, #-16]!	86	stp x29, x30, [sp, #-16]!
87	mov x29, sp	87	mov x29, sp
88		88
@@ -110,7 +110,7 @@ ST5( st1 {v4.16b}, [x0], #16 ) · 21 unmodified lines
110	.Lecbdecout:	110	.Lecbdecout:
111	ldp x29, x30, [sp], #16	111	ldp x29, x30, [sp], #16
112	ret	112	ret
113	AES_ENDPROC(aes_ecb_decrypt)	113	AES_FUNC_END(aes_ecb_decrypt)
114		114
115		115
116	/*	116	/*
@@ -126,7 +126,7 @@ AES_ENDPROC(aes_ecb_decrypt) · 9 unmodified lines
126	* u32 const rk2[]);	126	* u32 const rk2[]);
127	*/	127	*/
128		128
129	AES_ENTRY(aes_essiv_cbc_encrypt)	129	AES_FUNC_START(aes_essiv_cbc_encrypt)
130	ld1 {v4.16b}, [x5] /* get iv */	130	ld1 {v4.16b}, [x5] /* get iv */
131		131
132	mov w8, #14 /* AES-256: 14 rounds */	132	mov w8, #14 /* AES-256: 14 rounds */
@@ -135,7 +135,7 @@ AES_ENTRY(aes_essiv_cbc_encrypt) · 2 unmodified lines
135	enc_switch_key w3, x2, x6	135	enc_switch_key w3, x2, x6
136	b .Lcbcencloop4x	136	b .Lcbcencloop4x
137		137
138	AES_ENTRY(aes_cbc_encrypt)	138	AES_FUNC_START(aes_cbc_encrypt)
139	ld1 {v4.16b}, [x5] /* get iv */	139	ld1 {v4.16b}, [x5] /* get iv */
140	enc_prepare w3, x2, x6	140	enc_prepare w3, x2, x6
141		141
@@ -167,10 +167,10 @@ AES_ENTRY(aes_cbc_encrypt) · 25 unmodified lines
167	.Lcbcencout:	167	.Lcbcencout:
168	st1 {v4.16b}, [x5] /* return iv */	168	st1 {v4.16b}, [x5] /* return iv */
169	ret	169	ret
170	AES_ENDPROC(aes_cbc_encrypt)	170	AES_FUNC_END(aes_cbc_encrypt)
171	AES_ENDPROC(aes_essiv_cbc_encrypt)	171	AES_FUNC_END(aes_essiv_cbc_encrypt)
172		172
173	AES_ENTRY(aes_essiv_cbc_decrypt)	173	AES_FUNC_START(aes_essiv_cbc_decrypt)
174	stp x29, x30, [sp, #-16]!	174	stp x29, x30, [sp, #-16]!
175	mov x29, sp	175	mov x29, sp
176		176
@@ -181,7 +181,7 @@ AES_ENTRY(aes_essiv_cbc_decrypt) · 4 unmodified lines
181	encrypt_block cbciv, w8, x6, x7, w9	181	encrypt_block cbciv, w8, x6, x7, w9
182	b .Lessivcbcdecstart	182	b .Lessivcbcdecstart
183		183
184	AES_ENTRY(aes_cbc_decrypt)	184	AES_FUNC_START(aes_cbc_decrypt)
185	stp x29, x30, [sp, #-16]!	185	stp x29, x30, [sp, #-16]!
186	mov x29, sp	186	mov x29, sp
187		187
@@ -238,8 +238,8 @@ ST5( st1 {v4.16b}, [x0], #16 ) · 50 unmodified lines
238	st1 {cbciv.16b}, [x5] /* return iv */	238	st1 {cbciv.16b}, [x5] /* return iv */
239	ldp x29, x30, [sp], #16	239	ldp x29, x30, [sp], #16
240	ret	240	ret
241	AES_ENDPROC(aes_cbc_decrypt)	241	AES_FUNC_END(aes_cbc_decrypt)
242	AES_ENDPROC(aes_essiv_cbc_decrypt)	242	AES_FUNC_END(aes_essiv_cbc_decrypt)
243		243
244		244
245	/*	245	/*
@@ -249,7 +249,7 @@ AES_ENDPROC(aes_essiv_cbc_decrypt) · 3 unmodified lines
249	* int rounds, int bytes, u8 const iv[])	249	* int rounds, int bytes, u8 const iv[])
250	*/	250	*/
251		251
252	AES_ENTRY(aes_cbc_cts_encrypt)	252	AES_FUNC_START(aes_cbc_cts_encrypt)
253	adr_l x8, .Lcts_permute_table	253	adr_l x8, .Lcts_permute_table
254	sub x4, x4, #16	254	sub x4, x4, #16
255	add x9, x8, #32	255	add x9, x8, #32
@@ -276,9 +276,9 @@ AES_ENTRY(aes_cbc_cts_encrypt) · 20 unmodified lines
276	st1 {v0.16b}, [x4] /* overlapping stores */	276	st1 {v0.16b}, [x4] /* overlapping stores */
277	st1 {v1.16b}, [x0]	277	st1 {v1.16b}, [x0]
278	ret	278	ret
279	AES_ENDPROC(aes_cbc_cts_encrypt)	279	AES_FUNC_END(aes_cbc_cts_encrypt)
280		280
281	AES_ENTRY(aes_cbc_cts_decrypt)	281	AES_FUNC_START(aes_cbc_cts_decrypt)
282	adr_l x8, .Lcts_permute_table	282	adr_l x8, .Lcts_permute_table
283	sub x4, x4, #16	283	sub x4, x4, #16
284	add x9, x8, #32	284	add x9, x8, #32
@@ -305,7 +305,7 @@ AES_ENTRY(aes_cbc_cts_decrypt) · 20 unmodified lines
305	st1 {v2.16b}, [x4] /* overlapping stores */	305	st1 {v2.16b}, [x4] /* overlapping stores */
306	st1 {v0.16b}, [x0]	306	st1 {v0.16b}, [x0]
307	ret	307	ret
308	AES_ENDPROC(aes_cbc_cts_decrypt)	308	AES_FUNC_END(aes_cbc_cts_decrypt)
309		309
310	.section ".rodata", "a"	310	.section ".rodata", "a"
311	.align 6	311	.align 6
@@ -324,7 +324,7 @@ AES_ENDPROC(aes_cbc_cts_decrypt) · 12 unmodified lines
324	* int blocks, u8 ctr[])	324	* int blocks, u8 ctr[])
325	*/	325	*/
326		326
327	AES_ENTRY(aes_ctr_encrypt)	327	AES_FUNC_START(aes_ctr_encrypt)
328	stp x29, x30, [sp, #-16]!	328	stp x29, x30, [sp, #-16]!
329	mov x29, sp	329	mov x29, sp
330		330
@@ -409,7 +409,7 @@ ST5( st1 {v4.16b}, [x0], #16 ) · 78 unmodified lines
409	rev x7, x7	409	rev x7, x7
410	ins vctr.d[0], x7	410	ins vctr.d[0], x7
411	b .Lctrcarrydone	411	b .Lctrcarrydone
412	AES_ENDPROC(aes_ctr_encrypt)	412	AES_FUNC_END(aes_ctr_encrypt)
413		413
414		414
415	/*	415	/*
@@ -433,7 +433,7 @@ AES_ENDPROC(aes_ctr_encrypt) · 17 unmodified lines
433	uzp1 xtsmask.4s, xtsmask.4s, \tmp\().4s	433	uzp1 xtsmask.4s, xtsmask.4s, \tmp\().4s
434	.endm	434	.endm
435		435
436	AES_ENTRY(aes_xts_encrypt)	436	AES_FUNC_START(aes_xts_encrypt)
437	stp x29, x30, [sp, #-16]!	437	stp x29, x30, [sp, #-16]!
438	mov x29, sp	438	mov x29, sp
439		439
@@ -518,9 +518,9 @@ AES_ENTRY(aes_xts_encrypt) · 78 unmodified lines
518	st1 {v2.16b}, [x4] /* overlapping stores */	518	st1 {v2.16b}, [x4] /* overlapping stores */
519	mov w4, wzr	519	mov w4, wzr
520	b .Lxtsencctsout	520	b .Lxtsencctsout
521	AES_ENDPROC(aes_xts_encrypt)	521	AES_FUNC_END(aes_xts_encrypt)
522		522
523	AES_ENTRY(aes_xts_decrypt)	523	AES_FUNC_START(aes_xts_decrypt)
524	stp x29, x30, [sp, #-16]!	524	stp x29, x30, [sp, #-16]!
525	mov x29, sp	525	mov x29, sp
526		526
@@ -612,13 +612,13 @@ AES_ENTRY(aes_xts_decrypt) · 85 unmodified lines
612	st1 {v2.16b}, [x4] /* overlapping stores */	612	st1 {v2.16b}, [x4] /* overlapping stores */
613	mov w4, wzr	613	mov w4, wzr
614	b .Lxtsdecctsout	614	b .Lxtsdecctsout
615	AES_ENDPROC(aes_xts_decrypt)	615	AES_FUNC_END(aes_xts_decrypt)
616		616
617	/*	617	/*
618	* aes_mac_update(u8 const in[], u32 const rk[], int rounds,	618	* aes_mac_update(u8 const in[], u32 const rk[], int rounds,
619	* int blocks, u8 dg[], int enc_before, int enc_after)	619	* int blocks, u8 dg[], int enc_before, int enc_after)
620	*/	620	*/
621	AES_ENTRY(aes_mac_update)	621	AES_FUNC_START(aes_mac_update)
622	frame_push 6	622	frame_push 6
623		623
624	mov x19, x0	624	mov x19, x0
@@ -676,4 +676,4 @@ AES_ENTRY(aes_mac_update) · 51 unmodified lines
676	ld1 {v0.16b}, [x23] /* get dg */	676	ld1 {v0.16b}, [x23] /* get dg */
677	enc_prepare w21, x20, x0	677	enc_prepare w21, x20, x0
678	b .Lmacloop4x	678	b .Lmacloop4x
679	AES_ENDPROC(aes_mac_update)	679	AES_FUNC_END(aes_mac_update)

arch/arm64/crypto/aes-neon.S+2−2

@@ -8,8 +8,8 @@ · 7 unmodified lines
8	#include <linux/linkage.h>	8	#include <linux/linkage.h>
9	#include <asm/assembler.h>	9	#include <asm/assembler.h>
10		10
11	#define AES_ENTRY(func) SYM_FUNC_START(neon_ ## func)	11	#define AES_FUNC_START(func) SYM_FUNC_START(neon_ ## func)
12	#define AES_ENDPROC(func) SYM_FUNC_END(neon_ ## func)	12	#define AES_FUNC_END(func) SYM_FUNC_END(neon_ ## func)
13		13
14	xtsmask .req v7	14	xtsmask .req v7
15	cbciv .req v7	15	cbciv .req v7

arch/arm64/crypto/ghash-ce-core.S+8−8

@@ -587,20 +587,20 @@ CPU_LE( rev w8, w8 ) · 586 unmodified lines
587	* struct ghash_key const *k, u64 dg[], u8 ctr[],	587	* struct ghash_key const *k, u64 dg[], u8 ctr[],
588	* int rounds, u8 tag)	588	* int rounds, u8 tag)
589	*/	589	*/
590	ENTRY(pmull_gcm_encrypt)	590	SYM_FUNC_START(pmull_gcm_encrypt)
591	pmull_gcm_do_crypt 1	591	pmull_gcm_do_crypt 1
592	ENDPROC(pmull_gcm_encrypt)	592	SYM_FUNC_END(pmull_gcm_encrypt)
593		593
594	/*	594	/*
595	* void pmull_gcm_decrypt(int blocks, u8 dst[], const u8 src[],	595	* void pmull_gcm_decrypt(int blocks, u8 dst[], const u8 src[],
596	* struct ghash_key const *k, u64 dg[], u8 ctr[],	596	* struct ghash_key const *k, u64 dg[], u8 ctr[],
597	* int rounds, u8 tag)	597	* int rounds, u8 tag)
598	*/	598	*/
599	ENTRY(pmull_gcm_decrypt)	599	SYM_FUNC_START(pmull_gcm_decrypt)
600	pmull_gcm_do_crypt 0	600	pmull_gcm_do_crypt 0
601	ENDPROC(pmull_gcm_decrypt)	601	SYM_FUNC_END(pmull_gcm_decrypt)
602		602
603	pmull_gcm_ghash_4x:	603	SYM_FUNC_START_LOCAL(pmull_gcm_ghash_4x)
604	movi MASK.16b, #0xe1	604	movi MASK.16b, #0xe1
605	shl MASK.2d, MASK.2d, #57	605	shl MASK.2d, MASK.2d, #57
606		606
@@ -681,9 +681,9 @@ pmull_gcm_ghash_4x: · 74 unmodified lines
681	eor XL.16b, XL.16b, T2.16b	681	eor XL.16b, XL.16b, T2.16b
682		682
683	ret	683	ret
684	ENDPROC(pmull_gcm_ghash_4x)	684	SYM_FUNC_END(pmull_gcm_ghash_4x)
685		685
686	pmull_gcm_enc_4x:	686	SYM_FUNC_START_LOCAL(pmull_gcm_enc_4x)
687	ld1 {KS0.16b}, [x5] // load upper counter	687	ld1 {KS0.16b}, [x5] // load upper counter
688	sub w10, w8, #4	688	sub w10, w8, #4
689	sub w11, w8, #3	689	sub w11, w8, #3
@@ -746,7 +746,7 @@ pmull_gcm_enc_4x: · 56 unmodified lines
746	eor INP3.16b, INP3.16b, KS3.16b	746	eor INP3.16b, INP3.16b, KS3.16b
747		747
748	ret	748	ret
749	ENDPROC(pmull_gcm_enc_4x)	749	SYM_FUNC_END(pmull_gcm_enc_4x)
750		750
751	.section ".rodata", "a"	751	.section ".rodata", "a"
752	.align 6	752	.align 6

arch/arm64/include/asm/asm_pointer_auth.h+65−0

@@ -0,0 +1,65 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#ifndef __ASM_ASM_POINTER_AUTH_H
	3	#define __ASM_ASM_POINTER_AUTH_H
	4
	5	#include <asm/alternative.h>
	6	#include <asm/asm-offsets.h>
	7	#include <asm/cpufeature.h>
	8	#include <asm/sysreg.h>
	9
	10	#ifdef CONFIG_ARM64_PTR_AUTH
	11	/*
	12	* thread.keys_user.ap* as offset exceeds the #imm offset range
	13	* so use the base value of ldp as thread.keys_user and offset as
	14	* thread.keys_user.ap*.
	15	*/
	16	.macro ptrauth_keys_install_user tsk, tmp1, tmp2, tmp3
	17	mov \tmp1, #THREAD_KEYS_USER
	18	add \tmp1, \tsk, \tmp1
	19	alternative_if_not ARM64_HAS_ADDRESS_AUTH
	20	b .Laddr_auth_skip_\@
	21	alternative_else_nop_endif
	22	ldp \tmp2, \tmp3, [\tmp1, #PTRAUTH_USER_KEY_APIA]
	23	msr_s SYS_APIAKEYLO_EL1, \tmp2
	24	msr_s SYS_APIAKEYHI_EL1, \tmp3
	25	ldp \tmp2, \tmp3, [\tmp1, #PTRAUTH_USER_KEY_APIB]
	26	msr_s SYS_APIBKEYLO_EL1, \tmp2
	27	msr_s SYS_APIBKEYHI_EL1, \tmp3
	28	ldp \tmp2, \tmp3, [\tmp1, #PTRAUTH_USER_KEY_APDA]
	29	msr_s SYS_APDAKEYLO_EL1, \tmp2
	30	msr_s SYS_APDAKEYHI_EL1, \tmp3
	31	ldp \tmp2, \tmp3, [\tmp1, #PTRAUTH_USER_KEY_APDB]
	32	msr_s SYS_APDBKEYLO_EL1, \tmp2
	33	msr_s SYS_APDBKEYHI_EL1, \tmp3
	34	.Laddr_auth_skip_\@:
	35	alternative_if ARM64_HAS_GENERIC_AUTH
	36	ldp \tmp2, \tmp3, [\tmp1, #PTRAUTH_USER_KEY_APGA]
	37	msr_s SYS_APGAKEYLO_EL1, \tmp2
	38	msr_s SYS_APGAKEYHI_EL1, \tmp3
	39	alternative_else_nop_endif
	40	.endm
	41
	42	.macro ptrauth_keys_install_kernel tsk, sync, tmp1, tmp2, tmp3
	43	alternative_if ARM64_HAS_ADDRESS_AUTH
	44	mov \tmp1, #THREAD_KEYS_KERNEL
	45	add \tmp1, \tsk, \tmp1
	46	ldp \tmp2, \tmp3, [\tmp1, #PTRAUTH_KERNEL_KEY_APIA]
	47	msr_s SYS_APIAKEYLO_EL1, \tmp2
	48	msr_s SYS_APIAKEYHI_EL1, \tmp3
	49	.if \sync == 1
	50	isb
	51	.endif
	52	alternative_else_nop_endif
	53	.endm
	54
	55	#else /* CONFIG_ARM64_PTR_AUTH */
	56
	57	.macro ptrauth_keys_install_user tsk, tmp1, tmp2, tmp3
	58	.endm
	59
	60	.macro ptrauth_keys_install_kernel tsk, sync, tmp1, tmp2, tmp3
	61	.endm
	62
	63	#endif /* CONFIG_ARM64_PTR_AUTH */
	64
	65	#endif /* __ASM_ASM_POINTER_AUTH_H */

arch/arm64/include/asm/assembler.h+10−6

 	ldr	\rd, [\rn, #VMA_VM_MM]
 	.endm
-/*
- * mmid - get context id from mm pointer (mm->context.id)
- */
-	.macro	mmid, rd, rn
-	ldr	\rd, [\rn, #MM_CONTEXT_ID]
-	.endm
 /*
  * read_ctr - read CTR_EL0. If the system has mismatched register fields,
  * provide the system wide safe value from arm64_ftr_reg_ctrel0.sys_val
 9000:
 	.endm
+/*
+ * reset_amuserenr_el0 - reset AMUSERENR_EL0 if AMUv1 present
+ */
+	.macro	reset_amuserenr_el0, tmpreg
+	mrs	\tmpreg, id_aa64pfr0_el1	// Check ID_AA64PFR0_EL1
+	ubfx	\tmpreg, \tmpreg, #ID_AA64PFR0_AMU_SHIFT, #4
+	cbz	\tmpreg, .Lskip_\@		// Skip if no AMU present
+	msr_s	SYS_AMUSERENR_EL0, xzr		// Disable AMU access from EL0
+.Lskip_\@:
+	.endm
 /*
  * copy_page - copy src to dest using temp registers t1-t8
  */

arch/arm64/include/asm/checksum.h+6−1

 #ifndef __ASM_CHECKSUM_H
 #define __ASM_CHECKSUM_H
-#include <linux/types.h>
+#include <linux/in6.h>
+#define _HAVE_ARCH_IPV6_CSUM
+__sum16 csum_ipv6_magic(const struct in6_addr *saddr,
+			const struct in6_addr *daddr,
+			__u32 len, __u8 proto, __wsum sum);
 static inline __sum16 csum_fold(__wsum csum)
 {

arch/arm64/include/asm/compiler.h+24−0

@@ -0,0 +1,24 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#ifndef __ASM_COMPILER_H
	3	#define __ASM_COMPILER_H
	4
	5	#if defined(CONFIG_ARM64_PTR_AUTH)
	6
	7	/*
	8	* The EL0/EL1 pointer bits used by a pointer authentication code.
	9	* This is dependent on TBI0/TBI1 being enabled, or bits 63:56 would also apply.
	10	*/
	11	#define ptrauth_user_pac_mask() GENMASK_ULL(54, vabits_actual)
	12	#define ptrauth_kernel_pac_mask() GENMASK_ULL(63, vabits_actual)
	13
	14	/* Valid for EL0 TTBR0 and EL1 TTBR1 instruction pointers */
	15	#define ptrauth_clear_pac(ptr) \
	16	((ptr & BIT_ULL(55)) ? (ptr \| ptrauth_kernel_pac_mask()) : \
	17	(ptr & ~ptrauth_user_pac_mask()))
	18
	19	#define __builtin_return_address(val) \
	20	(void *)(ptrauth_clear_pac((unsigned long)__builtin_return_address(val)))
	21
	22	#endif /* CONFIG_ARM64_PTR_AUTH */
	23
	24	#endif /* __ASM_COMPILER_H */

arch/arm64/include/asm/cpu_ops.h+4−4

@@ -55,12 +55,12 @@ struct cpu_operations { · 54 unmodified lines
55	#endif	55	#endif
56	};	56	};
57		57
58	extern const struct cpu_operations *cpu_ops[NR_CPUS];	58	int __init init_cpu_ops(int cpu);
59	int __init cpu_read_ops(int cpu);	59	extern const struct cpu_operations *get_cpu_ops(int cpu);
60		60
61	static inline void __init cpu_read_bootcpu_ops(void)	61	static inline void __init init_bootcpu_ops(void)
62	{	62	{
63	cpu_read_ops(0);	63	init_cpu_ops(0);
64	}	64	}
65		65
66	#endif /* ifndef __ASM_CPU_OPS_H */	66	#endif /* ifndef __ASM_CPU_OPS_H */

arch/arm64/include/asm/cpucaps.h+4−1

 #define ARM64_WORKAROUND_SPECULATIVE_AT_NVHE	48
 #define ARM64_HAS_E0PD				49
 #define ARM64_HAS_RNG				50
+#define ARM64_HAS_AMU_EXTN			51
+#define ARM64_HAS_ADDRESS_AUTH			52
+#define ARM64_HAS_GENERIC_AUTH			53
-#define ARM64_NCAPS				51
+#define ARM64_NCAPS				54
 #endif /* __ASM_CPUCAPS_H */

arch/arm64/include/asm/cpufeature.h+96−29

  *     In some non-typical cases either both (a) and (b), or neither,
  *     should be permitted. This can be described by including neither
  *     or both flags in the capability's type field.
+ *
+ *     In case of a conflict, the CPU is prevented from booting. If the
+ *     ARM64_CPUCAP_PANIC_ON_CONFLICT flag is specified for the capability,
+ *     then a kernel panic is triggered.
  */
 #define ARM64_CPUCAP_PERMITTED_FOR_LATE_CPU	((u16)BIT(4))
 /* Is it safe for a late CPU to miss this capability when system has it */
 #define ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU	((u16)BIT(5))
+/* Panic when a conflict is detected */
+#define ARM64_CPUCAP_PANIC_ON_CONFLICT		((u16)BIT(6))
 /*
  * CPU errata workarounds that need to be enabled at boot time if one or
 /*
  * CPU feature used early in the boot based on the boot CPU. All secondary
- * CPUs must match the state of the capability as detected by the boot CPU.
+ * CPUs must match the state of the capability as detected by the boot CPU. In
+ * case of a conflict, a kernel panic is triggered.
+ */
+#define ARM64_CPUCAP_STRICT_BOOT_CPU_FEATURE		\
+	(ARM64_CPUCAP_SCOPE_BOOT_CPU | ARM64_CPUCAP_PANIC_ON_CONFLICT)
+/*
+ * CPU feature used early in the boot based on the boot CPU. It is safe for a
+ * late CPU to have this feature even though the boot CPU hasn't enabled it,
+ * although the feature will not be used by Linux in this case. If the boot CPU
+ * has enabled this feature already, then every late CPU must have it.
  */
-#define ARM64_CPUCAP_STRICT_BOOT_CPU_FEATURE ARM64_CPUCAP_SCOPE_BOOT_CPU
+#define ARM64_CPUCAP_BOOT_CPU_FEATURE                  \
+	(ARM64_CPUCAP_SCOPE_BOOT_CPU | ARM64_CPUCAP_PERMITTED_FOR_LATE_CPU)
 struct arm64_cpu_capabilities {
 	const char *desc;
 	return cap->type & ARM64_CPUCAP_SCOPE_MASK;
 }
-static inline bool
-cpucap_late_cpu_optional(const struct arm64_cpu_capabilities *cap)
-{
-	return !!(cap->type & ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU);
-}
-static inline bool
-cpucap_late_cpu_permitted(const struct arm64_cpu_capabilities *cap)
-{
-	return !!(cap->type & ARM64_CPUCAP_PERMITTED_FOR_LATE_CPU);
-}
 /*
  * Generic helper for handling capabilties with multiple (match,enable) pairs
  * of call backs, sharing the same capability bit.
 #define cpu_set_named_feature(name) cpu_set_feature(cpu_feature(name))
 #define cpu_have_named_feature(name) cpu_have_feature(cpu_feature(name))
-/* System capability check for constant caps */
+static __always_inline bool system_capabilities_finalized(void)
-static __always_inline bool __cpus_have_const_cap(int num)
 {
-	if (num >= ARM64_NCAPS)
+	return static_branch_likely(&arm64_const_caps_ready);
-		return false;
-	return static_branch_unlikely(&cpu_hwcap_keys[num]);
 }
+/*
+ * Test for a capability with a runtime check.
+ *
+ * Before the capability is detected, this returns false.
+ */
 static inline bool cpus_have_cap(unsigned int num)
 {
 	if (num >= ARM64_NCAPS)
 	return test_bit(num, cpu_hwcaps);
 }
+/*
+ * Test for a capability without a runtime check.
+ *
+ * Before capabilities are finalized, this returns false.
+ * After capabilities are finalized, this is patched to avoid a runtime check.
+ *
+ * @num must be a compile-time constant.
+ */
+static __always_inline bool __cpus_have_const_cap(int num)
+{
+	if (num >= ARM64_NCAPS)
+		return false;
+	return static_branch_unlikely(&cpu_hwcap_keys[num]);
+}
+/*
+ * Test for a capability, possibly with a runtime check.
+ *
+ * Before capabilities are finalized, this behaves as cpus_have_cap().
+ * After capabilities are finalized, this is patched to avoid a runtime check.
+ *
+ * @num must be a compile-time constant.
+ */
 static __always_inline bool cpus_have_const_cap(int num)
 {
-	if (static_branch_likely(&arm64_const_caps_ready))
+	if (system_capabilities_finalized())
 		return __cpus_have_const_cap(num);
 	else
 		return cpus_have_cap(num);
 }
+/*
+ * Test for a capability without a runtime check.
+ *
+ * Before capabilities are finalized, this will BUG().
+ * After capabilities are finalized, this is patched to avoid a runtime check.
+ *
+ * @num must be a compile-time constant.
+ */
+static __always_inline bool cpus_have_final_cap(int num)
+{
+	if (system_capabilities_finalized())
+		return __cpus_have_const_cap(num);
+	else
+		BUG();
+}
 static inline void cpus_set_cap(unsigned int num)
 {
 	if (num >= ARM64_NCAPS) {
 	return cpuid_feature_extract_unsigned_field_width(features, field, 4);
 }
+/*
+ * Fields that identify the version of the Performance Monitors Extension do
+ * not follow the standard ID scheme. See ARM DDI 0487E.a page D13-2825,
+ * "Alternative ID scheme used for the Performance Monitors Extension version".
+ */
+static inline u64 __attribute_const__
+cpuid_feature_cap_perfmon_field(u64 features, int field, u64 cap)
+{
+	u64 val = cpuid_feature_extract_unsigned_field(features, field);
+	u64 mask = GENMASK_ULL(field + 3, field);
+	/* Treat IMPLEMENTATION DEFINED functionality as unimplemented */
+	if (val == 0xf)
+		val = 0;
+	if (val > cap) {
+		features &= ~mask;
+		features |= (cap << field) & mask;
+	}
+	return features;
+}
 static inline u64 arm64_ftr_mask(const struct arm64_ftr_bits *ftrp)
 {
 	return (u64)GENMASK(ftrp->shift + ftrp->width - 1, ftrp->shift);
 static inline bool system_supports_address_auth(void)
 {
 	return IS_ENABLED(CONFIG_ARM64_PTR_AUTH) &&
-		(cpus_have_const_cap(ARM64_HAS_ADDRESS_AUTH_ARCH) ||
+		cpus_have_const_cap(ARM64_HAS_ADDRESS_AUTH);
-		 cpus_have_const_cap(ARM64_HAS_ADDRESS_AUTH_IMP_DEF));
 }
 static inline bool system_supports_generic_auth(void)
 {
 	return IS_ENABLED(CONFIG_ARM64_PTR_AUTH) &&
-		(cpus_have_const_cap(ARM64_HAS_GENERIC_AUTH_ARCH) ||
+		cpus_have_const_cap(ARM64_HAS_GENERIC_AUTH);
-		 cpus_have_const_cap(ARM64_HAS_GENERIC_AUTH_IMP_DEF));
 }
 static inline bool system_uses_irq_prio_masking(void)
 	       system_uses_irq_prio_masking();
 }
-static inline bool system_capabilities_finalized(void)
-{
-	return static_branch_likely(&arm64_const_caps_ready);
-}
 #define ARM64_BP_HARDEN_UNKNOWN		-1
 #define ARM64_BP_HARDEN_WA_NEEDED	0
 #define ARM64_BP_HARDEN_NOT_REQUIRED	1
 						ID_AA64MMFR1_HADBS_SHIFT);
 }
+#ifdef CONFIG_ARM64_AMU_EXTN
+/* Check whether the cpu supports the Activity Monitors Unit (AMU) */
+extern bool cpu_has_amu_feat(int cpu);
+#endif
 #endif /* __ASSEMBLY__ */
 #endif

arch/arm64/include/asm/esr.h+1−1

@@ -60,7 +60,7 @@ · 59 unmodified lines
60	#define ESR_ELx_EC_BKPT32 (0x38)	60	#define ESR_ELx_EC_BKPT32 (0x38)
61	/* Unallocated EC: 0x39 */	61	/* Unallocated EC: 0x39 */
62	#define ESR_ELx_EC_VECTOR32 (0x3A) /* EL2 only */	62	#define ESR_ELx_EC_VECTOR32 (0x3A) /* EL2 only */
63	/* Unallocted EC: 0x3B */	63	/* Unallocated EC: 0x3B */
64	#define ESR_ELx_EC_BRK64 (0x3C)	64	#define ESR_ELx_EC_BRK64 (0x3C)
65	/* Unallocated EC: 0x3D - 0x3F */	65	/* Unallocated EC: 0x3D - 0x3F */
66	#define ESR_ELx_EC_MAX (0x3F)	66	#define ESR_ELx_EC_MAX (0x3F)

arch/arm64/include/asm/kvm_arm.h+1−0

@@ -267,6 +267,7 @@ · 266 unmodified lines
267		267
268	/* Hyp Coprocessor Trap Register */	268	/* Hyp Coprocessor Trap Register */
269	#define CPTR_EL2_TCPAC (1 << 31)	269	#define CPTR_EL2_TCPAC (1 << 31)
		270	#define CPTR_EL2_TAM (1 << 30)
270	#define CPTR_EL2_TTA (1 << 20)	271	#define CPTR_EL2_TTA (1 << 20)
271	#define CPTR_EL2_TFP (1 << CPTR_EL2_TFP_SHIFT)	272	#define CPTR_EL2_TFP (1 << CPTR_EL2_TFP_SHIFT)
272	#define CPTR_EL2_TZ (1 << 8)	273	#define CPTR_EL2_TZ (1 << 8)

arch/arm64/include/asm/kvm_asm.h+4−0

  */
 #define KVM_VECTOR_PREAMBLE	(2 * AARCH64_INSN_SIZE)
+#define __SMCCC_WORKAROUND_1_SMC_SZ 36
 #ifndef __ASSEMBLY__
 #include <linux/mm.h>
 extern u32 __kvm_get_mdcr_el2(void);
+extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ];
 /* Home-grown __this_cpu_{ptr,read} variants that always work at HYP */
 #define __hyp_this_cpu_ptr(sym)						\
 	({								\

arch/arm64/include/asm/kvm_mmu.h+4−5

 	int slot = -1;
 	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR) && data->fn) {
-		vect = kern_hyp_va(kvm_ksym_ref(__bp_harden_hyp_vecs_start));
+		vect = kern_hyp_va(kvm_ksym_ref(__bp_harden_hyp_vecs));
 		slot = data->hyp_vectors_slot;
 	}
 	 *  HBP +  HEL2 -> use hardened vertors and use exec mapping
 	 */
 	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR)) {
-		__kvm_bp_vect_base = kvm_ksym_ref(__bp_harden_hyp_vecs_start);
+		__kvm_bp_vect_base = kvm_ksym_ref(__bp_harden_hyp_vecs);
 		__kvm_bp_vect_base = kern_hyp_va(__kvm_bp_vect_base);
 	}
 	if (cpus_have_const_cap(ARM64_HARDEN_EL2_VECTORS)) {
-		phys_addr_t vect_pa = __pa_symbol(__bp_harden_hyp_vecs_start);
+		phys_addr_t vect_pa = __pa_symbol(__bp_harden_hyp_vecs);
-		unsigned long size = (__bp_harden_hyp_vecs_end -
+		unsigned long size = __BP_HARDEN_HYP_VECS_SZ;
-				      __bp_harden_hyp_vecs_start);
 		/*
 		 * Always allocate a spare vector slot, as we don't

arch/arm64/include/asm/memory.h+1−0

@@ -54,6 +54,7 @@ · 53 unmodified lines
54	#define MODULES_VADDR (BPF_JIT_REGION_END)	54	#define MODULES_VADDR (BPF_JIT_REGION_END)
55	#define MODULES_VSIZE (SZ_128M)	55	#define MODULES_VSIZE (SZ_128M)
56	#define VMEMMAP_START (-VMEMMAP_SIZE - SZ_2M)	56	#define VMEMMAP_START (-VMEMMAP_SIZE - SZ_2M)
		57	#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE)
57	#define PCI_IO_END (VMEMMAP_START - SZ_2M)	58	#define PCI_IO_END (VMEMMAP_START - SZ_2M)
58	#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE)	59	#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE)
59	#define FIXADDR_TOP (PCI_IO_START - SZ_2M)	60	#define FIXADDR_TOP (PCI_IO_START - SZ_2M)

arch/arm64/include/asm/mmu.h+6−4

@@ -13,6 +13,7 @@ · 12 unmodified lines
13	#define TTBR_ASID_MASK (UL(0xffff) << 48)	13	#define TTBR_ASID_MASK (UL(0xffff) << 48)
14		14
15	#define BP_HARDEN_EL2_SLOTS 4	15	#define BP_HARDEN_EL2_SLOTS 4
		16	#define __BP_HARDEN_HYP_VECS_SZ (BP_HARDEN_EL2_SLOTS * SZ_2K)
16		17
17	#ifndef __ASSEMBLY__	18	#ifndef __ASSEMBLY__
18		19
@@ -23,9 +24,9 @@ typedef struct { · 4 unmodified lines
23	} mm_context_t;	24	} mm_context_t;
24		25
25	/*	26	/*
26	* This macro is only used by the TLBI code, which cannot race with an	27	* This macro is only used by the TLBI and low-level switch_mm() code,
27	* ASID change and therefore doesn't need to reload the counter using	28	* neither of which can race with an ASID change. We therefore don't
28	* atomic64_read.	29	* need to reload the counter using atomic64_read().
29	*/	30	*/
30	#define ASID(mm) ((mm)->context.id.counter & 0xffff)	31	#define ASID(mm) ((mm)->context.id.counter & 0xffff)
31		32
@@ -43,7 +44,8 @@ struct bp_hardening_data { · 11 unmodified lines
43		44
44	#if (defined(CONFIG_HARDEN_BRANCH_PREDICTOR) \|\| \	45	#if (defined(CONFIG_HARDEN_BRANCH_PREDICTOR) \|\| \
45	defined(CONFIG_HARDEN_EL2_VECTORS))	46	defined(CONFIG_HARDEN_EL2_VECTORS))
46	extern char __bp_harden_hyp_vecs_start[], __bp_harden_hyp_vecs_end[];	47
		48	extern char __bp_harden_hyp_vecs[];
47	extern atomic_t arm64_el2_vector_last_slot;	49	extern atomic_t arm64_el2_vector_last_slot;
48	#endif /* CONFIG_HARDEN_BRANCH_PREDICTOR \|\| CONFIG_HARDEN_EL2_VECTORS */	50	#endif /* CONFIG_HARDEN_BRANCH_PREDICTOR \|\| CONFIG_HARDEN_EL2_VECTORS */
49		51

arch/arm64/include/asm/mmu_context.h+2−0

@@ -46,6 +46,8 @@ static inline void cpu_set_reserved_ttbr0(void) · 45 unmodified lines
46	isb();	46	isb();
47	}	47	}
48		48
		49	void cpu_do_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm);
		50
49	static inline void cpu_switch_mm(pgd_t pgd, struct mm_struct mm)	51	static inline void cpu_switch_mm(pgd_t pgd, struct mm_struct mm)
50	{	52	{
51	BUG_ON(pgd == swapper_pg_dir);	53	BUG_ON(pgd == swapper_pg_dir);

arch/arm64/include/asm/page.h+4−0

@@ -21,6 +21,10 @@ extern void __cpu_copy_user_page(void to, const void from, · 20 unmodified lines
21	extern void copy_page(void to, const void from);	21	extern void copy_page(void to, const void from);
22	extern void clear_page(void *to);	22	extern void clear_page(void *to);
23		23
		24	#define __alloc_zeroed_user_highpage(movableflags, vma, vaddr) \
		25	alloc_page_vma(GFP_HIGHUSER \| __GFP_ZERO \| movableflags, vma, vaddr)
		26	#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE
		27
24	#define clear_user_page(addr,vaddr,pg) __cpu_clear_user_page(addr, vaddr)	28	#define clear_user_page(addr,vaddr,pg) __cpu_clear_user_page(addr, vaddr)
25	#define copy_user_page(to,from,vaddr,pg) __cpu_copy_user_page(to, from, vaddr)	29	#define copy_user_page(to,from,vaddr,pg) __cpu_copy_user_page(to, from, vaddr)
26		30

arch/arm64/include/asm/perf_event.h+2−1

 #define ARMV8_PMU_PMCR_X	(1 << 4) /* Export to ETM */
 #define ARMV8_PMU_PMCR_DP	(1 << 5) /* Disable CCNT if non-invasive debug*/
 #define ARMV8_PMU_PMCR_LC	(1 << 6) /* Overflow on 64 bit cycle counter */
+#define ARMV8_PMU_PMCR_LP	(1 << 7) /* Long event counter enable */
 #define	ARMV8_PMU_PMCR_N_SHIFT	11	 /* Number of counters supported */
 #define	ARMV8_PMU_PMCR_N_MASK	0x1f
-#define	ARMV8_PMU_PMCR_MASK	0x7f	 /* Mask for writable bits */
+#define	ARMV8_PMU_PMCR_MASK	0xff	 /* Mask for writable bits */
 /*
  * PMOVSR: counters overflow flag status reg

arch/arm64/include/asm/pointer_auth.h+22−28

  * We give each process its own keys, which are shared by all threads. The keys
  * are inherited upon fork(), and reinitialised upon exec*().
  */
-struct ptrauth_keys {
+struct ptrauth_keys_user {
 	struct ptrauth_key apia;
 	struct ptrauth_key apib;
 	struct ptrauth_key apda;
 	struct ptrauth_key apga;
 };
-static inline void ptrauth_keys_init(struct ptrauth_keys *keys)
+struct ptrauth_keys_kernel {
+	struct ptrauth_key apia;
+};
+static inline void ptrauth_keys_init_user(struct ptrauth_keys_user *keys)
 {
 	if (system_supports_address_auth()) {
 		get_random_bytes(&keys->apia, sizeof(keys->apia));
 	write_sysreg_s(__pki_v.hi, SYS_ ## k ## KEYHI_EL1);	\
 } while (0)
-static inline void ptrauth_keys_switch(struct ptrauth_keys *keys)
+static __always_inline void ptrauth_keys_init_kernel(struct ptrauth_keys_kernel *keys)
 {
-	if (system_supports_address_auth()) {
+	if (system_supports_address_auth())
-		__ptrauth_key_install(APIA, keys->apia);
+		get_random_bytes(&keys->apia, sizeof(keys->apia));
-		__ptrauth_key_install(APIB, keys->apib);
+}
-		__ptrauth_key_install(APDA, keys->apda);
-		__ptrauth_key_install(APDB, keys->apdb);
-	}
-	if (system_supports_generic_auth())
+static __always_inline void ptrauth_keys_switch_kernel(struct ptrauth_keys_kernel *keys)
-		__ptrauth_key_install(APGA, keys->apga);
+{
+	if (system_supports_address_auth())
+		__ptrauth_key_install(APIA, keys->apia);
 }
 extern int ptrauth_prctl_reset_keys(struct task_struct *tsk, unsigned long arg);
-/*
- * The EL0 pointer bits used by a pointer authentication code.
- * This is dependent on TBI0 being enabled, or bits 63:56 would also apply.
- */
-#define ptrauth_user_pac_mask()	GENMASK(54, vabits_actual)
-/* Only valid for EL0 TTBR0 instruction pointers */
 static inline unsigned long ptrauth_strip_insn_pac(unsigned long ptr)
 {
-	return ptr & ~ptrauth_user_pac_mask();
+	return ptrauth_clear_pac(ptr);
 }
 #define ptrauth_thread_init_user(tsk)					\
-do {									\
+	ptrauth_keys_init_user(&(tsk)->thread.keys_user)
-	struct task_struct *__ptiu_tsk = (tsk);				\
+#define ptrauth_thread_init_kernel(tsk)					\
-	ptrauth_keys_init(&__ptiu_tsk->thread.keys_user);		\
+	ptrauth_keys_init_kernel(&(tsk)->thread.keys_kernel)
-	ptrauth_keys_switch(&__ptiu_tsk->thread.keys_user);		\
+#define ptrauth_thread_switch_kernel(tsk)				\
-} while (0)
+	ptrauth_keys_switch_kernel(&(tsk)->thread.keys_kernel)
-#define ptrauth_thread_switch(tsk)	\
-	ptrauth_keys_switch(&(tsk)->thread.keys_user)
 #else /* CONFIG_ARM64_PTR_AUTH */
 #define ptrauth_prctl_reset_keys(tsk, arg)	(-EINVAL)
 #define ptrauth_strip_insn_pac(lr)	(lr)
 #define ptrauth_thread_init_user(tsk)
-#define ptrauth_thread_switch(tsk)
+#define ptrauth_thread_init_kernel(tsk)
+#define ptrauth_thread_switch_kernel(tsk)
 #endif /* CONFIG_ARM64_PTR_AUTH */
 #endif /* __ASM_POINTER_AUTH_H */

arch/arm64/include/asm/proc-fns.h+0−2

@@ -13,11 +13,9 @@ · 12 unmodified lines
13		13
14	#include <asm/page.h>	14	#include <asm/page.h>
15		15
16	struct mm_struct;
17	struct cpu_suspend_ctx;	16	struct cpu_suspend_ctx;
18		17
19	extern void cpu_do_idle(void);	18	extern void cpu_do_idle(void);
20	extern void cpu_do_switch_mm(unsigned long pgd_phys, struct mm_struct *mm);
21	extern void cpu_do_suspend(struct cpu_suspend_ctx *ptr);	19	extern void cpu_do_suspend(struct cpu_suspend_ctx *ptr);
22	extern u64 cpu_do_resume(phys_addr_t ptr, u64 idmap_ttbr);	20	extern u64 cpu_do_resume(phys_addr_t ptr, u64 idmap_ttbr);
23		21

arch/arm64/include/asm/processor.h+2−1

@@ -148,7 +148,8 @@ struct thread_struct { · 147 unmodified lines
148	unsigned long fault_code; /* ESR_EL1 value */	148	unsigned long fault_code; /* ESR_EL1 value */
149	struct debug_info debug; /* debugging */	149	struct debug_info debug; /* debugging */
150	#ifdef CONFIG_ARM64_PTR_AUTH	150	#ifdef CONFIG_ARM64_PTR_AUTH
151	struct ptrauth_keys keys_user;	151	struct ptrauth_keys_user keys_user;
		152	struct ptrauth_keys_kernel keys_kernel;
152	#endif	153	#endif
153	};	154	};
154		155

arch/arm64/include/asm/smp.h+12−0

 #define CPU_STUCK_REASON_52_BIT_VA	(UL(1) << CPU_STUCK_REASON_SHIFT)
 #define CPU_STUCK_REASON_NO_GRAN	(UL(2) << CPU_STUCK_REASON_SHIFT)
+/* Possible options for __cpu_setup */
+/* Option to setup primary cpu */
+#define ARM64_CPU_BOOT_PRIMARY		(1)
+/* Option to setup secondary cpus */
+#define ARM64_CPU_BOOT_SECONDARY	(2)
+/* Option to setup cpus for different cpu run time services */
+#define ARM64_CPU_RUNTIME		(3)
 #ifndef __ASSEMBLY__
 #include <asm/percpu.h>
 #include <linux/threads.h>
 #include <linux/cpumask.h>
 #include <linux/thread_info.h>
+#include <asm/pointer_auth.h>
 DECLARE_PER_CPU_READ_MOSTLY(int, cpu_number);
 struct secondary_data {
 	void *stack;
 	struct task_struct *task;
+#ifdef CONFIG_ARM64_PTR_AUTH
+	struct ptrauth_keys_kernel ptrauth_key;
+#endif
 	long status;
 };

arch/arm64/include/asm/stackprotector.h+5−0

 #include <linux/random.h>
 #include <linux/version.h>
+#include <asm/pointer_auth.h>
 extern unsigned long __stack_chk_guard;
  */
 static __always_inline void boot_init_stack_canary(void)
 {
+#if defined(CONFIG_STACKPROTECTOR)
 	unsigned long canary;
 	/* Try to get a semi random initial value. */
 	current->stack_canary = canary;
 	if (!IS_ENABLED(CONFIG_STACKPROTECTOR_PER_TASK))
 		__stack_chk_guard = current->stack_canary;
+#endif
+	ptrauth_thread_init_kernel(current);
+	ptrauth_thread_switch_kernel(current);
 }
 #endif	/* _ASM_STACKPROTECTOR_H */

arch/arm64/include/asm/sysreg.h+48−0

@@ -386,6 +386,42 @@ · 385 unmodified lines
386	#define SYS_TPIDR_EL0 sys_reg(3, 3, 13, 0, 2)	386	#define SYS_TPIDR_EL0 sys_reg(3, 3, 13, 0, 2)
387	#define SYS_TPIDRRO_EL0 sys_reg(3, 3, 13, 0, 3)	387	#define SYS_TPIDRRO_EL0 sys_reg(3, 3, 13, 0, 3)
388		388
		389	/* Definitions for system register interface to AMU for ARMv8.4 onwards */
		390	#define SYS_AM_EL0(crm, op2) sys_reg(3, 3, 13, (crm), (op2))
		391	#define SYS_AMCR_EL0 SYS_AM_EL0(2, 0)
		392	#define SYS_AMCFGR_EL0 SYS_AM_EL0(2, 1)
		393	#define SYS_AMCGCR_EL0 SYS_AM_EL0(2, 2)
		394	#define SYS_AMUSERENR_EL0 SYS_AM_EL0(2, 3)
		395	#define SYS_AMCNTENCLR0_EL0 SYS_AM_EL0(2, 4)
		396	#define SYS_AMCNTENSET0_EL0 SYS_AM_EL0(2, 5)
		397	#define SYS_AMCNTENCLR1_EL0 SYS_AM_EL0(3, 0)
		398	#define SYS_AMCNTENSET1_EL0 SYS_AM_EL0(3, 1)
		399
		400	/*
		401	* Group 0 of activity monitors (architected):
		402	* op0 op1 CRn CRm op2
		403	* Counter: 11 011 1101 010:n<3> n<2:0>
		404	* Type: 11 011 1101 011:n<3> n<2:0>
		405	* n: 0-15
		406	*
		407	* Group 1 of activity monitors (auxiliary):
		408	* op0 op1 CRn CRm op2
		409	* Counter: 11 011 1101 110:n<3> n<2:0>
		410	* Type: 11 011 1101 111:n<3> n<2:0>
		411	* n: 0-15
		412	*/
		413
		414	#define SYS_AMEVCNTR0_EL0(n) SYS_AM_EL0(4 + ((n) >> 3), (n) & 7)
		415	#define SYS_AMEVTYPE0_EL0(n) SYS_AM_EL0(6 + ((n) >> 3), (n) & 7)
		416	#define SYS_AMEVCNTR1_EL0(n) SYS_AM_EL0(12 + ((n) >> 3), (n) & 7)
		417	#define SYS_AMEVTYPE1_EL0(n) SYS_AM_EL0(14 + ((n) >> 3), (n) & 7)
		418
		419	/* AMU v1: Fixed (architecturally defined) activity monitors */
		420	#define SYS_AMEVCNTR0_CORE_EL0 SYS_AMEVCNTR0_EL0(0)
		421	#define SYS_AMEVCNTR0_CONST_EL0 SYS_AMEVCNTR0_EL0(1)
		422	#define SYS_AMEVCNTR0_INST_RET_EL0 SYS_AMEVCNTR0_EL0(2)
		423	#define SYS_AMEVCNTR0_MEM_STALL SYS_AMEVCNTR0_EL0(3)
		424
389	#define SYS_CNTFRQ_EL0 sys_reg(3, 3, 14, 0, 0)	425	#define SYS_CNTFRQ_EL0 sys_reg(3, 3, 14, 0, 0)
390		426
391	#define SYS_CNTP_TVAL_EL0 sys_reg(3, 3, 14, 2, 0)	427	#define SYS_CNTP_TVAL_EL0 sys_reg(3, 3, 14, 2, 0)
@@ -598,6 +634,7 @@ · 206 unmodified lines
598	#define ID_AA64PFR0_CSV3_SHIFT 60	634	#define ID_AA64PFR0_CSV3_SHIFT 60
599	#define ID_AA64PFR0_CSV2_SHIFT 56	635	#define ID_AA64PFR0_CSV2_SHIFT 56
600	#define ID_AA64PFR0_DIT_SHIFT 48	636	#define ID_AA64PFR0_DIT_SHIFT 48
		637	#define ID_AA64PFR0_AMU_SHIFT 44
601	#define ID_AA64PFR0_SVE_SHIFT 32	638	#define ID_AA64PFR0_SVE_SHIFT 32
602	#define ID_AA64PFR0_RAS_SHIFT 28	639	#define ID_AA64PFR0_RAS_SHIFT 28
603	#define ID_AA64PFR0_GIC_SHIFT 24	640	#define ID_AA64PFR0_GIC_SHIFT 24
@@ -608,6 +645,7 @@ · 4 unmodified lines
608	#define ID_AA64PFR0_EL1_SHIFT 4	645	#define ID_AA64PFR0_EL1_SHIFT 4
609	#define ID_AA64PFR0_EL0_SHIFT 0	646	#define ID_AA64PFR0_EL0_SHIFT 0
610		647
		648	#define ID_AA64PFR0_AMU 0x1
611	#define ID_AA64PFR0_SVE 0x1	649	#define ID_AA64PFR0_SVE 0x1
612	#define ID_AA64PFR0_RAS_V1 0x1	650	#define ID_AA64PFR0_RAS_V1 0x1
613	#define ID_AA64PFR0_FP_NI 0xf	651	#define ID_AA64PFR0_FP_NI 0xf
@@ -702,6 +740,16 @@ · 88 unmodified lines
702	#define ID_AA64DFR0_TRACEVER_SHIFT 4	740	#define ID_AA64DFR0_TRACEVER_SHIFT 4
703	#define ID_AA64DFR0_DEBUGVER_SHIFT 0	741	#define ID_AA64DFR0_DEBUGVER_SHIFT 0
704		742
		743	#define ID_AA64DFR0_PMUVER_8_0 0x1
		744	#define ID_AA64DFR0_PMUVER_8_1 0x4
		745	#define ID_AA64DFR0_PMUVER_8_4 0x5
		746	#define ID_AA64DFR0_PMUVER_8_5 0x6
		747	#define ID_AA64DFR0_PMUVER_IMP_DEF 0xf
		748
		749	#define ID_DFR0_PERFMON_SHIFT 24
		750
		751	#define ID_DFR0_PERFMON_8_1 0x4
		752
705	#define ID_ISAR5_RDM_SHIFT 24	753	#define ID_ISAR5_RDM_SHIFT 24
706	#define ID_ISAR5_CRC32_SHIFT 16	754	#define ID_ISAR5_CRC32_SHIFT 16
707	#define ID_ISAR5_SHA2_SHIFT 12	755	#define ID_ISAR5_SHA2_SHIFT 12

arch/arm64/include/asm/topology.h+9−0

@@ -16,6 +16,15 @@ int pcibus_to_node(struct pci_bus *bus); · 15 unmodified lines
16		16
17	#include <linux/arch_topology.h>	17	#include <linux/arch_topology.h>
18		18
		19	#ifdef CONFIG_ARM64_AMU_EXTN
		20	/*
		21	* Replace task scheduler's default counter-based
		22	* frequency-invariance scale factor setting.
		23	*/
		24	void topology_scale_freq_tick(void);
		25	#define arch_scale_freq_tick topology_scale_freq_tick
		26	#endif /* CONFIG_ARM64_AMU_EXTN */
		27
19	/* Replace task scheduler's default frequency-invariant accounting */	28	/* Replace task scheduler's default frequency-invariant accounting */
20	#define arch_scale_freq_capacity topology_get_freq_scale	29	#define arch_scale_freq_capacity topology_get_freq_scale
21		30

arch/arm64/kernel/Makefile+1−1

@@ -21,7 +21,7 @@ obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ · 20 unmodified lines
21	smp.o smp_spin_table.o topology.o smccc-call.o \	21	smp.o smp_spin_table.o topology.o smccc-call.o \
22	syscall.o	22	syscall.o
23		23
24	extra-$(CONFIG_EFI) := efi-entry.o	24	targets += efi-entry.o
25		25
26	OBJCOPYFLAGS := --prefix-symbols=__efistub_	26	OBJCOPYFLAGS := --prefix-symbols=__efistub_
27	$(obj)/%.stub.o: $(obj)/%.o FORCE	27	$(obj)/%.stub.o: $(obj)/%.o FORCE

arch/arm64/kernel/armv8_deprecated.c+1−1

@@ -630,7 +630,7 @@ static int __init armv8_deprecated_init(void) · 629 unmodified lines
630	register_insn_emulation(&cp15_barrier_ops);	630	register_insn_emulation(&cp15_barrier_ops);
631		631
632	if (IS_ENABLED(CONFIG_SETEND_EMULATION)) {	632	if (IS_ENABLED(CONFIG_SETEND_EMULATION)) {
633	if(system_supports_mixed_endian_el0())	633	if (system_supports_mixed_endian_el0())
634	register_insn_emulation(&setend_ops);	634	register_insn_emulation(&setend_ops);
635	else	635	else
636	pr_info("setend instruction emulation is not supported on this system\n");	636	pr_info("setend instruction emulation is not supported on this system\n");

arch/arm64/kernel/asm-offsets.c+16−0

 #endif
   BLANK();
   DEFINE(THREAD_CPU_CONTEXT,	offsetof(struct task_struct, thread.cpu_context));
+#ifdef CONFIG_ARM64_PTR_AUTH
+  DEFINE(THREAD_KEYS_USER,	offsetof(struct task_struct, thread.keys_user));
+  DEFINE(THREAD_KEYS_KERNEL,	offsetof(struct task_struct, thread.keys_kernel));
+#endif
   BLANK();
   DEFINE(S_X0,			offsetof(struct pt_regs, regs[0]));
   DEFINE(S_X2,			offsetof(struct pt_regs, regs[2]));
   BLANK();
   DEFINE(CPU_BOOT_STACK,	offsetof(struct secondary_data, stack));
   DEFINE(CPU_BOOT_TASK,		offsetof(struct secondary_data, task));
+#ifdef CONFIG_ARM64_PTR_AUTH
+  DEFINE(CPU_BOOT_PTRAUTH_KEY,	offsetof(struct secondary_data, ptrauth_key));
+#endif
   BLANK();
 #ifdef CONFIG_KVM_ARM_HOST
   DEFINE(VCPU_CONTEXT,		offsetof(struct kvm_vcpu, arch.ctxt));
 #ifdef CONFIG_ARM_SDE_INTERFACE
   DEFINE(SDEI_EVENT_INTREGS,	offsetof(struct sdei_registered_event, interrupted_regs));
   DEFINE(SDEI_EVENT_PRIORITY,	offsetof(struct sdei_registered_event, priority));
+#endif
+#ifdef CONFIG_ARM64_PTR_AUTH
+  DEFINE(PTRAUTH_USER_KEY_APIA,		offsetof(struct ptrauth_keys_user, apia));
+  DEFINE(PTRAUTH_USER_KEY_APIB,		offsetof(struct ptrauth_keys_user, apib));
+  DEFINE(PTRAUTH_USER_KEY_APDA,		offsetof(struct ptrauth_keys_user, apda));
+  DEFINE(PTRAUTH_USER_KEY_APDB,		offsetof(struct ptrauth_keys_user, apdb));
+  DEFINE(PTRAUTH_USER_KEY_APGA,		offsetof(struct ptrauth_keys_user, apga));
+  DEFINE(PTRAUTH_KERNEL_KEY_APIA,	offsetof(struct ptrauth_keys_kernel, apia));
+  BLANK();
 #endif
   return 0;
 }

arch/arm64/kernel/cpu-reset.S+1−1

@@ -32,7 +32,7 @@ · 31 unmodified lines
32	ENTRY(__cpu_soft_restart)	32	ENTRY(__cpu_soft_restart)
33	/* Clear sctlr_el1 flags. */	33	/* Clear sctlr_el1 flags. */
34	mrs x12, sctlr_el1	34	mrs x12, sctlr_el1
35	ldr x13, =SCTLR_ELx_FLAGS	35	mov_q x13, SCTLR_ELx_FLAGS
36	bic x12, x12, x13	36	bic x12, x12, x13
37	pre_disable_mmu_workaround	37	pre_disable_mmu_workaround
38	msr sctlr_el1, x12	38	msr sctlr_el1, x12

arch/arm64/kernel/cpu_errata.c+8−10

 #include <asm/cpu.h>
 #include <asm/cputype.h>
 #include <asm/cpufeature.h>
+#include <asm/kvm_asm.h>
 #include <asm/smp_plat.h>
 static bool __maybe_unused
 DEFINE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data);
 #ifdef CONFIG_KVM_INDIRECT_VECTORS
-extern char __smccc_workaround_1_smc_start[];
-extern char __smccc_workaround_1_smc_end[];
 static void __copy_hyp_vect_bpi(int slot, const char *hyp_vecs_start,
 				const char *hyp_vecs_end)
 {
-	void *dst = lm_alias(__bp_harden_hyp_vecs_start + slot * SZ_2K);
+	void *dst = lm_alias(__bp_harden_hyp_vecs + slot * SZ_2K);
 	int i;
 	for (i = 0; i < SZ_2K; i += 0x80)
 	raw_spin_unlock(&bp_lock);
 }
 #else
-#define __smccc_workaround_1_smc_start		NULL
-#define __smccc_workaround_1_smc_end		NULL
 static void install_bp_hardening_cb(bp_hardening_cb_t fn,
 				      const char *hyp_vecs_start,
 				      const char *hyp_vecs_end)
 #include <linux/arm-smccc.h>
-static void call_smc_arch_workaround_1(void)
+static void __maybe_unused call_smc_arch_workaround_1(void)
 {
 	arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_1, NULL);
 }
 		smccc_end = NULL;
 		break;
+#if IS_ENABLED(CONFIG_KVM_ARM_HOST)
 	case SMCCC_CONDUIT_SMC:
 		cb = call_smc_arch_workaround_1;
-		smccc_start = __smccc_workaround_1_smc_start;
+		smccc_start = __smccc_workaround_1_smc;
-		smccc_end = __smccc_workaround_1_smc_end;
+		smccc_end = __smccc_workaround_1_smc +
+			__SMCCC_WORKAROUND_1_SMC_SZ;
 		break;
+#endif
 	default:
 		return -1;

arch/arm64/kernel/cpu_ops.c+9−2

 #include <asm/smp_plat.h>
 extern const struct cpu_operations smp_spin_table_ops;
+#ifdef CONFIG_ARM64_ACPI_PARKING_PROTOCOL
 extern const struct cpu_operations acpi_parking_protocol_ops;
+#endif
 extern const struct cpu_operations cpu_psci_ops;
-const struct cpu_operations *cpu_ops[NR_CPUS] __ro_after_init;
+static const struct cpu_operations *cpu_ops[NR_CPUS] __ro_after_init;
 static const struct cpu_operations *const dt_supported_cpu_ops[] __initconst = {
 	&smp_spin_table_ops,
 /*
  * Read a cpu's enable method and record it in cpu_ops.
  */
-int __init cpu_read_ops(int cpu)
+int __init init_cpu_ops(int cpu)
 {
 	const char *enable_method = cpu_read_enable_method(cpu);
 	return 0;
 }
+const struct cpu_operations *get_cpu_ops(int cpu)
+{
+	return cpu_ops[cpu];
+}

arch/arm64/kernel/cpufeature.c+142−23

@@ -116,6 +116,8 @@ cpufeature_pan_not_uao(const struct arm64_cpu_capabilities *entry, int __unused) · 115 unmodified lines
116		116
117	static void cpu_enable_cnp(struct arm64_cpu_capabilities const *cap);	117	static void cpu_enable_cnp(struct arm64_cpu_capabilities const *cap);
118		118
		119	static bool __system_matches_cap(unsigned int n);
		120
119	/*	121	/*
120	* NOTE: Any changes to the visibility of features should be kept in	122	* NOTE: Any changes to the visibility of features should be kept in
121	* sync with the documentation of the CPU feature register ABI.	123	* sync with the documentation of the CPU feature register ABI.
@@ -163,6 +165,7 @@ static const struct arm64_ftr_bits ftr_id_aa64pfr0[] = { · 41 unmodified lines
163	ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_CSV3_SHIFT, 4, 0),	165	ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_CSV3_SHIFT, 4, 0),
164	ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_CSV2_SHIFT, 4, 0),	166	ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_CSV2_SHIFT, 4, 0),
165	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_DIT_SHIFT, 4, 0),	167	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_DIT_SHIFT, 4, 0),
		168	ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_AMU_SHIFT, 4, 0),
166	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SVE),	169	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SVE),
167	FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_SVE_SHIFT, 4, 0),	170	FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_SVE_SHIFT, 4, 0),
168	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_RAS_SHIFT, 4, 0),	171	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_RAS_SHIFT, 4, 0),
@@ -551,7 +554,7 @@ static void __init init_cpu_ftr_reg(u32 sys_reg, u64 new) · 382 unmodified lines
551		554
552	BUG_ON(!reg);	555	BUG_ON(!reg);
553		556
554	for (ftrp = reg->ftr_bits; ftrp->width; ftrp++) {	557	for (ftrp = reg->ftr_bits; ftrp->width; ftrp++) {
555	u64 ftr_mask = arm64_ftr_mask(ftrp);	558	u64 ftr_mask = arm64_ftr_mask(ftrp);
556	s64 ftr_new = arm64_ftr_value(ftrp, new);	559	s64 ftr_new = arm64_ftr_value(ftrp, new);
557		560
@@ -1222,6 +1225,57 @@ static bool has_hw_dbm(const struct arm64_cpu_capabilities *cap, · 664 unmodified lines
1222		1225
1223	#endif	1226	#endif
1224		1227
		1228	#ifdef CONFIG_ARM64_AMU_EXTN
		1229
		1230	/*
		1231	* The "amu_cpus" cpumask only signals that the CPU implementation for the
		1232	* flagged CPUs supports the Activity Monitors Unit (AMU) but does not provide
		1233	* information regarding all the events that it supports. When a CPU bit is
		1234	* set in the cpumask, the user of this feature can only rely on the presence
		1235	* of the 4 fixed counters for that CPU. But this does not guarantee that the
		1236	* counters are enabled or access to these counters is enabled by code
		1237	* executed at higher exception levels (firmware).
		1238	*/
		1239	static struct cpumask amu_cpus __read_mostly;
		1240
		1241	bool cpu_has_amu_feat(int cpu)
		1242	{
		1243	return cpumask_test_cpu(cpu, &amu_cpus);
		1244	}
		1245
		1246	/* Initialize the use of AMU counters for frequency invariance */
		1247	extern void init_cpu_freq_invariance_counters(void);
		1248
		1249	static void cpu_amu_enable(struct arm64_cpu_capabilities const *cap)
		1250	{
		1251	if (has_cpuid_feature(cap, SCOPE_LOCAL_CPU)) {
		1252	pr_info("detected CPU%d: Activity Monitors Unit (AMU)\n",
		1253	smp_processor_id());
		1254	cpumask_set_cpu(smp_processor_id(), &amu_cpus);
		1255	init_cpu_freq_invariance_counters();
		1256	}
		1257	}
		1258
		1259	static bool has_amu(const struct arm64_cpu_capabilities *cap,
		1260	int __unused)
		1261	{
		1262	/*
		1263	* The AMU extension is a non-conflicting feature: the kernel can
		1264	* safely run a mix of CPUs with and without support for the
		1265	* activity monitors extension. Therefore, unconditionally enable
		1266	* the capability to allow any late CPU to use the feature.
		1267	*
		1268	* With this feature unconditionally enabled, the cpu_enable
		1269	* function will be called for all CPUs that match the criteria,
		1270	* including secondary and hotplugged, marking this feature as
		1271	* present on that respective CPU. The enable function will also
		1272	* print a detection message.
		1273	*/
		1274
		1275	return true;
		1276	}
		1277	#endif
		1278
1225	#ifdef CONFIG_ARM64_VHE	1279	#ifdef CONFIG_ARM64_VHE
1226	static bool runs_at_el2(const struct arm64_cpu_capabilities *entry, int __unused)	1280	static bool runs_at_el2(const struct arm64_cpu_capabilities *entry, int __unused)
1227	{	1281	{
@@ -1316,10 +1370,18 @@ static void cpu_clear_disr(const struct arm64_cpu_capabilities *__unused) · 88 unmodified lines
1316	#endif /* CONFIG_ARM64_RAS_EXTN */	1370	#endif /* CONFIG_ARM64_RAS_EXTN */
1317		1371
1318	#ifdef CONFIG_ARM64_PTR_AUTH	1372	#ifdef CONFIG_ARM64_PTR_AUTH
1319	static void cpu_enable_address_auth(struct arm64_cpu_capabilities const *cap)	1373	static bool has_address_auth(const struct arm64_cpu_capabilities *entry,
		1374	int __unused)
1320	{	1375	{
1321	sysreg_clear_set(sctlr_el1, 0, SCTLR_ELx_ENIA \| SCTLR_ELx_ENIB \|	1376	return __system_matches_cap(ARM64_HAS_ADDRESS_AUTH_ARCH) \|\|
1322	SCTLR_ELx_ENDA \| SCTLR_ELx_ENDB);	1377	__system_matches_cap(ARM64_HAS_ADDRESS_AUTH_IMP_DEF);
		1378	}
		1379
		1380	static bool has_generic_auth(const struct arm64_cpu_capabilities *entry,
		1381	int __unused)
		1382	{
		1383	return __system_matches_cap(ARM64_HAS_GENERIC_AUTH_ARCH) \|\|
		1384	__system_matches_cap(ARM64_HAS_GENERIC_AUTH_IMP_DEF);
1323	}	1385	}
1324	#endif /* CONFIG_ARM64_PTR_AUTH */	1386	#endif /* CONFIG_ARM64_PTR_AUTH */
1325		1387
@@ -1347,6 +1409,25 @@ static bool can_use_gic_priorities(const struct arm64_cpu_capabilities *entry, · 21 unmodified lines
1347	}	1409	}
1348	#endif	1410	#endif
1349		1411
		1412	/* Internal helper functions to match cpu capability type */
		1413	static bool
		1414	cpucap_late_cpu_optional(const struct arm64_cpu_capabilities *cap)
		1415	{
		1416	return !!(cap->type & ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU);
		1417	}
		1418
		1419	static bool
		1420	cpucap_late_cpu_permitted(const struct arm64_cpu_capabilities *cap)
		1421	{
		1422	return !!(cap->type & ARM64_CPUCAP_PERMITTED_FOR_LATE_CPU);
		1423	}
		1424
		1425	static bool
		1426	cpucap_panic_on_conflict(const struct arm64_cpu_capabilities *cap)
		1427	{
		1428	return !!(cap->type & ARM64_CPUCAP_PANIC_ON_CONFLICT);
		1429	}
		1430
1350	static const struct arm64_cpu_capabilities arm64_features[] = {	1431	static const struct arm64_cpu_capabilities arm64_features[] = {
1351	{	1432	{
1352	.desc = "GIC system register CPU interface",	1433	.desc = "GIC system register CPU interface",
@@ -1499,6 +1580,24 @@ static const struct arm64_cpu_capabilities arm64_features[] = { · 146 unmodified lines
1499	.cpu_enable = cpu_clear_disr,	1580	.cpu_enable = cpu_clear_disr,
1500	},	1581	},
1501	#endif /* CONFIG_ARM64_RAS_EXTN */	1582	#endif /* CONFIG_ARM64_RAS_EXTN */
		1583	#ifdef CONFIG_ARM64_AMU_EXTN
		1584	{
		1585	/*
		1586	* The feature is enabled by default if CONFIG_ARM64_AMU_EXTN=y.
		1587	* Therefore, don't provide .desc as we don't want the detection
		1588	* message to be shown until at least one CPU is detected to
		1589	* support the feature.
		1590	*/
		1591	.capability = ARM64_HAS_AMU_EXTN,
		1592	.type = ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE,
		1593	.matches = has_amu,
		1594	.sys_reg = SYS_ID_AA64PFR0_EL1,
		1595	.sign = FTR_UNSIGNED,
		1596	.field_pos = ID_AA64PFR0_AMU_SHIFT,
		1597	.min_field_value = ID_AA64PFR0_AMU,
		1598	.cpu_enable = cpu_amu_enable,
		1599	},
		1600	#endif /* CONFIG_ARM64_AMU_EXTN */
1502	{	1601	{
1503	.desc = "Data cache clean to the PoU not required for I/D coherence",	1602	.desc = "Data cache clean to the PoU not required for I/D coherence",
1504	.capability = ARM64_HAS_CACHE_IDC,	1603	.capability = ARM64_HAS_CACHE_IDC,
@@ -1592,24 +1691,27 @@ static const struct arm64_cpu_capabilities arm64_features[] = { · 87 unmodified lines
1592	{	1691	{
1593	.desc = "Address authentication (architected algorithm)",	1692	.desc = "Address authentication (architected algorithm)",
1594	.capability = ARM64_HAS_ADDRESS_AUTH_ARCH,	1693	.capability = ARM64_HAS_ADDRESS_AUTH_ARCH,
1595	.type = ARM64_CPUCAP_SYSTEM_FEATURE,	1694	.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
1596	.sys_reg = SYS_ID_AA64ISAR1_EL1,	1695	.sys_reg = SYS_ID_AA64ISAR1_EL1,
1597	.sign = FTR_UNSIGNED,	1696	.sign = FTR_UNSIGNED,
1598	.field_pos = ID_AA64ISAR1_APA_SHIFT,	1697	.field_pos = ID_AA64ISAR1_APA_SHIFT,
1599	.min_field_value = ID_AA64ISAR1_APA_ARCHITECTED,	1698	.min_field_value = ID_AA64ISAR1_APA_ARCHITECTED,
1600	.matches = has_cpuid_feature,	1699	.matches = has_cpuid_feature,
1601	.cpu_enable = cpu_enable_address_auth,
1602	},	1700	},
1603	{	1701	{
1604	.desc = "Address authentication (IMP DEF algorithm)",	1702	.desc = "Address authentication (IMP DEF algorithm)",
1605	.capability = ARM64_HAS_ADDRESS_AUTH_IMP_DEF,	1703	.capability = ARM64_HAS_ADDRESS_AUTH_IMP_DEF,
1606	.type = ARM64_CPUCAP_SYSTEM_FEATURE,	1704	.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
1607	.sys_reg = SYS_ID_AA64ISAR1_EL1,	1705	.sys_reg = SYS_ID_AA64ISAR1_EL1,
1608	.sign = FTR_UNSIGNED,	1706	.sign = FTR_UNSIGNED,
1609	.field_pos = ID_AA64ISAR1_API_SHIFT,	1707	.field_pos = ID_AA64ISAR1_API_SHIFT,
1610	.min_field_value = ID_AA64ISAR1_API_IMP_DEF,	1708	.min_field_value = ID_AA64ISAR1_API_IMP_DEF,
1611	.matches = has_cpuid_feature,	1709	.matches = has_cpuid_feature,
1612	.cpu_enable = cpu_enable_address_auth,	1710	},
		1711	{
		1712	.capability = ARM64_HAS_ADDRESS_AUTH,
		1713	.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
		1714	.matches = has_address_auth,
1613	},	1715	},
1614	{	1716	{
1615	.desc = "Generic authentication (architected algorithm)",	1717	.desc = "Generic authentication (architected algorithm)",
@@ -1631,6 +1733,11 @@ static const struct arm64_cpu_capabilities arm64_features[] = { · 15 unmodified lines
1631	.min_field_value = ID_AA64ISAR1_GPI_IMP_DEF,	1733	.min_field_value = ID_AA64ISAR1_GPI_IMP_DEF,
1632	.matches = has_cpuid_feature,	1734	.matches = has_cpuid_feature,
1633	},	1735	},
		1736	{
		1737	.capability = ARM64_HAS_GENERIC_AUTH,
		1738	.type = ARM64_CPUCAP_SYSTEM_FEATURE,
		1739	.matches = has_generic_auth,
		1740	},
1634	#endif /* CONFIG_ARM64_PTR_AUTH */	1741	#endif /* CONFIG_ARM64_PTR_AUTH */
1635	#ifdef CONFIG_ARM64_PSEUDO_NMI	1742	#ifdef CONFIG_ARM64_PSEUDO_NMI
1636	{	1743	{
@@ -1980,10 +2087,8 @@ static void __init enable_cpu_capabilities(u16 scope_mask) · 343 unmodified lines
1980	* Run through the list of capabilities to check for conflicts.	2087	* Run through the list of capabilities to check for conflicts.
1981	* If the system has already detected a capability, take necessary	2088	* If the system has already detected a capability, take necessary
1982	* action on this CPU.	2089	* action on this CPU.
1983	*
1984	* Returns "false" on conflicts.
1985	*/	2090	*/
1986	static bool verify_local_cpu_caps(u16 scope_mask)	2091	static void verify_local_cpu_caps(u16 scope_mask)
1987	{	2092	{
1988	int i;	2093	int i;
1989	bool cpu_has_cap, system_has_cap;	2094	bool cpu_has_cap, system_has_cap;
@@ -2028,10 +2133,12 @@ static bool verify_local_cpu_caps(u16 scope_mask) · 38 unmodified lines
2028	pr_crit("CPU%d: Detected conflict for capability %d (%s), System: %d, CPU: %d\n",	2133	pr_crit("CPU%d: Detected conflict for capability %d (%s), System: %d, CPU: %d\n",
2029	smp_processor_id(), caps->capability,	2134	smp_processor_id(), caps->capability,
2030	caps->desc, system_has_cap, cpu_has_cap);	2135	caps->desc, system_has_cap, cpu_has_cap);
2031	return false;
2032	}
2033		2136
2034	return true;	2137	if (cpucap_panic_on_conflict(caps))
		2138	cpu_panic_kernel();
		2139	else
		2140	cpu_die_early();
		2141	}
2035	}	2142	}
2036		2143
2037	/*	2144	/*
@@ -2041,12 +2148,8 @@ static bool verify_local_cpu_caps(u16 scope_mask) · 3 unmodified lines
2041	static void check_early_cpu_features(void)	2148	static void check_early_cpu_features(void)
2042	{	2149	{
2043	verify_cpu_asid_bits();	2150	verify_cpu_asid_bits();
2044	/*	2151
2045	* Early features are used by the kernel already. If there	2152	verify_local_cpu_caps(SCOPE_BOOT_CPU);
2046	* is a conflict, we cannot proceed further.
2047	*/
2048	if (!verify_local_cpu_caps(SCOPE_BOOT_CPU))
2049	cpu_panic_kernel();
2050	}	2153	}
2051		2154
2052	static void	2155	static void
@@ -2094,8 +2197,7 @@ static void verify_local_cpu_capabilities(void) · 41 unmodified lines
2094	* check_early_cpu_features(), as they need to be verified	2197	* check_early_cpu_features(), as they need to be verified
2095	* on all secondary CPUs.	2198	* on all secondary CPUs.
2096	*/	2199	*/
2097	if (!verify_local_cpu_caps(SCOPE_ALL & ~SCOPE_BOOT_CPU))	2200	verify_local_cpu_caps(SCOPE_ALL & ~SCOPE_BOOT_CPU);
2098	cpu_die_early();
2099		2201
2100	verify_local_elf_hwcaps(arm64_elf_hwcaps);	2202	verify_local_elf_hwcaps(arm64_elf_hwcaps);
2101		2203
@@ -2146,6 +2248,23 @@ bool this_cpu_has_cap(unsigned int n) · 44 unmodified lines
2146	return false;	2248	return false;
2147	}	2249	}
2148		2250
		2251	/*
		2252	* This helper function is used in a narrow window when,
		2253	* - The system wide safe registers are set with all the SMP CPUs and,
		2254	* - The SYSTEM_FEATURE cpu_hwcaps may not have been set.
		2255	* In all other cases cpus_have_{const_}cap() should be used.
		2256	*/
		2257	static bool __system_matches_cap(unsigned int n)
		2258	{
		2259	if (n < ARM64_NCAPS) {
		2260	const struct arm64_cpu_capabilities *cap = cpu_hwcaps_ptrs[n];
		2261
		2262	if (cap)
		2263	return cap->matches(cap, SCOPE_SYSTEM);
		2264	}
		2265	return false;
		2266	}
		2267
2149	void cpu_set_feature(unsigned int num)	2268	void cpu_set_feature(unsigned int num)
2150	{	2269	{
2151	WARN_ON(num >= MAX_CPU_FEATURES);	2270	WARN_ON(num >= MAX_CPU_FEATURES);
@@ -2218,7 +2337,7 @@ void __init setup_cpu_features(void) · 66 unmodified lines
2218	static bool __maybe_unused	2337	static bool __maybe_unused
2219	cpufeature_pan_not_uao(const struct arm64_cpu_capabilities *entry, int __unused)	2338	cpufeature_pan_not_uao(const struct arm64_cpu_capabilities *entry, int __unused)
2220	{	2339	{
2221	return (cpus_have_const_cap(ARM64_HAS_PAN) && !cpus_have_const_cap(ARM64_HAS_UAO));	2340	return (__system_matches_cap(ARM64_HAS_PAN) && !__system_matches_cap(ARM64_HAS_UAO));
2222	}	2341	}
2223		2342
2224	static void __maybe_unused cpu_enable_cnp(struct arm64_cpu_capabilities const *cap)	2343	static void __maybe_unused cpu_enable_cnp(struct arm64_cpu_capabilities const *cap)

arch/arm64/kernel/cpuidle.c+5−4

 int arm_cpuidle_init(unsigned int cpu)
 {
+	const struct cpu_operations *ops = get_cpu_ops(cpu);
 	int ret = -EOPNOTSUPP;
-	if (cpu_ops[cpu] && cpu_ops[cpu]->cpu_suspend &&
+	if (ops && ops->cpu_suspend && ops->cpu_init_idle)
-			cpu_ops[cpu]->cpu_init_idle)
+		ret = ops->cpu_init_idle(cpu);
-		ret = cpu_ops[cpu]->cpu_init_idle(cpu);
 	return ret;
 }
 int arm_cpuidle_suspend(int index)
 {
 	int cpu = smp_processor_id();
+	const struct cpu_operations *ops = get_cpu_ops(cpu);
-	return cpu_ops[cpu]->cpu_suspend(index);
+	return ops->cpu_suspend(index);
 }
 #ifdef CONFIG_ACPI

arch/arm64/kernel/entry-common.c+1−1

@@ -175,7 +175,7 @@ NOKPROBE_SYMBOL(el0_pc); · 174 unmodified lines
175	static void notrace el0_sp(struct pt_regs *regs, unsigned long esr)	175	static void notrace el0_sp(struct pt_regs *regs, unsigned long esr)
176	{	176	{
177	user_exit_irqoff();	177	user_exit_irqoff();
178	local_daif_restore(DAIF_PROCCTX_NOIRQ);	178	local_daif_restore(DAIF_PROCCTX);
179	do_sp_pc_abort(regs->sp, esr, regs);	179	do_sp_pc_abort(regs->sp, esr, regs);
180	}	180	}
181	NOKPROBE_SYMBOL(el0_sp);	181	NOKPROBE_SYMBOL(el0_sp);

arch/arm64/kernel/entry-ftrace.S+24−24

@@ -75,27 +75,27 @@ · 74 unmodified lines
75	add x29, sp, #S_STACKFRAME	75	add x29, sp, #S_STACKFRAME
76	.endm	76	.endm
77		77
78	ENTRY(ftrace_regs_caller)	78	SYM_CODE_START(ftrace_regs_caller)
79	ftrace_regs_entry 1	79	ftrace_regs_entry 1
80	b ftrace_common	80	b ftrace_common
81	ENDPROC(ftrace_regs_caller)	81	SYM_CODE_END(ftrace_regs_caller)
82		82
83	ENTRY(ftrace_caller)	83	SYM_CODE_START(ftrace_caller)
84	ftrace_regs_entry 0	84	ftrace_regs_entry 0
85	b ftrace_common	85	b ftrace_common
86	ENDPROC(ftrace_caller)	86	SYM_CODE_END(ftrace_caller)
87		87
88	ENTRY(ftrace_common)	88	SYM_CODE_START(ftrace_common)
89	sub x0, x30, #AARCH64_INSN_SIZE // ip (callsite's BL insn)	89	sub x0, x30, #AARCH64_INSN_SIZE // ip (callsite's BL insn)
90	mov x1, x9 // parent_ip (callsite's LR)	90	mov x1, x9 // parent_ip (callsite's LR)
91	ldr_l x2, function_trace_op // op	91	ldr_l x2, function_trace_op // op
92	mov x3, sp // regs	92	mov x3, sp // regs
93		93
94	GLOBAL(ftrace_call)	94	SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)
95	bl ftrace_stub	95	bl ftrace_stub
96		96
97	#ifdef CONFIG_FUNCTION_GRAPH_TRACER	97	#ifdef CONFIG_FUNCTION_GRAPH_TRACER
98	GLOBAL(ftrace_graph_call) // ftrace_graph_caller();	98	SYM_INNER_LABEL(ftrace_graph_call, SYM_L_GLOBAL) // ftrace_graph_caller();
99	nop // If enabled, this will be replaced	99	nop // If enabled, this will be replaced
100	// "b ftrace_graph_caller"	100	// "b ftrace_graph_caller"
101	#endif	101	#endif
@@ -122,17 +122,17 @@ ftrace_common_return: · 20 unmodified lines
122	add sp, sp, #S_FRAME_SIZE + 16	122	add sp, sp, #S_FRAME_SIZE + 16
123		123
124	ret x9	124	ret x9
125	ENDPROC(ftrace_common)	125	SYM_CODE_END(ftrace_common)
126		126
127	#ifdef CONFIG_FUNCTION_GRAPH_TRACER	127	#ifdef CONFIG_FUNCTION_GRAPH_TRACER
128	ENTRY(ftrace_graph_caller)	128	SYM_CODE_START(ftrace_graph_caller)
129	ldr x0, [sp, #S_PC]	129	ldr x0, [sp, #S_PC]
130	sub x0, x0, #AARCH64_INSN_SIZE // ip (callsite's BL insn)	130	sub x0, x0, #AARCH64_INSN_SIZE // ip (callsite's BL insn)
131	add x1, sp, #S_LR // parent_ip (callsite's LR)	131	add x1, sp, #S_LR // parent_ip (callsite's LR)
132	ldr x2, [sp, #S_FRAME_SIZE] // parent fp (callsite's FP)	132	ldr x2, [sp, #S_FRAME_SIZE] // parent fp (callsite's FP)
133	bl prepare_ftrace_return	133	bl prepare_ftrace_return
134	b ftrace_common_return	134	b ftrace_common_return
135	ENDPROC(ftrace_graph_caller)	135	SYM_CODE_END(ftrace_graph_caller)
136	#endif	136	#endif
137		137
138	#else /* CONFIG_DYNAMIC_FTRACE_WITH_REGS */	138	#else /* CONFIG_DYNAMIC_FTRACE_WITH_REGS */
@@ -218,7 +218,7 @@ ENDPROC(ftrace_graph_caller) · 79 unmodified lines
218	* - tracer function to probe instrumented function's entry,	218	* - tracer function to probe instrumented function's entry,
219	* - ftrace_graph_caller to set up an exit hook	219	* - ftrace_graph_caller to set up an exit hook
220	*/	220	*/
221	ENTRY(_mcount)	221	SYM_FUNC_START(_mcount)
222	mcount_enter	222	mcount_enter
223		223
224	ldr_l x2, ftrace_trace_function	224	ldr_l x2, ftrace_trace_function
@@ -242,7 +242,7 @@ skip_ftrace_call: // } · 17 unmodified lines
242	b.ne ftrace_graph_caller // ftrace_graph_caller();	242	b.ne ftrace_graph_caller // ftrace_graph_caller();
243	#endif /* CONFIG_FUNCTION_GRAPH_TRACER */	243	#endif /* CONFIG_FUNCTION_GRAPH_TRACER */
244	mcount_exit	244	mcount_exit
245	ENDPROC(_mcount)	245	SYM_FUNC_END(_mcount)
246	EXPORT_SYMBOL(_mcount)	246	EXPORT_SYMBOL(_mcount)
247	NOKPROBE(_mcount)	247	NOKPROBE(_mcount)
248		248
@@ -253,9 +253,9 @@ NOKPROBE(_mcount) · 4 unmodified lines
253	* and later on, NOP to branch to ftrace_caller() when enabled or branch to	253	* and later on, NOP to branch to ftrace_caller() when enabled or branch to
254	* NOP when disabled per-function base.	254	* NOP when disabled per-function base.
255	*/	255	*/
256	ENTRY(_mcount)	256	SYM_FUNC_START(_mcount)
257	ret	257	ret
258	ENDPROC(_mcount)	258	SYM_FUNC_END(_mcount)
259	EXPORT_SYMBOL(_mcount)	259	EXPORT_SYMBOL(_mcount)
260	NOKPROBE(_mcount)	260	NOKPROBE(_mcount)
261		261
@@ -268,24 +268,24 @@ NOKPROBE(_mcount) · 6 unmodified lines
268	* - tracer function to probe instrumented function's entry,	268	* - tracer function to probe instrumented function's entry,
269	* - ftrace_graph_caller to set up an exit hook	269	* - ftrace_graph_caller to set up an exit hook
270	*/	270	*/
271	ENTRY(ftrace_caller)	271	SYM_FUNC_START(ftrace_caller)
272	mcount_enter	272	mcount_enter
273		273
274	mcount_get_pc0 x0 // function's pc	274	mcount_get_pc0 x0 // function's pc
275	mcount_get_lr x1 // function's lr	275	mcount_get_lr x1 // function's lr
276		276
277	GLOBAL(ftrace_call) // tracer(pc, lr);	277	SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) // tracer(pc, lr);
278	nop // This will be replaced with "bl xxx"	278	nop // This will be replaced with "bl xxx"
279	// where xxx can be any kind of tracer.	279	// where xxx can be any kind of tracer.
280		280
281	#ifdef CONFIG_FUNCTION_GRAPH_TRACER	281	#ifdef CONFIG_FUNCTION_GRAPH_TRACER
282	GLOBAL(ftrace_graph_call) // ftrace_graph_caller();	282	SYM_INNER_LABEL(ftrace_graph_call, SYM_L_GLOBAL) // ftrace_graph_caller();
283	nop // If enabled, this will be replaced	283	nop // If enabled, this will be replaced
284	// "b ftrace_graph_caller"	284	// "b ftrace_graph_caller"
285	#endif	285	#endif
286		286
287	mcount_exit	287	mcount_exit
288	ENDPROC(ftrace_caller)	288	SYM_FUNC_END(ftrace_caller)
289	#endif /* CONFIG_DYNAMIC_FTRACE */	289	#endif /* CONFIG_DYNAMIC_FTRACE */
290		290
291	#ifdef CONFIG_FUNCTION_GRAPH_TRACER	291	#ifdef CONFIG_FUNCTION_GRAPH_TRACER
@@ -298,20 +298,20 @@ ENDPROC(ftrace_caller) · 6 unmodified lines
298	* the call stack in order to intercept instrumented function's return path	298	* the call stack in order to intercept instrumented function's return path
299	* and run return_to_handler() later on its exit.	299	* and run return_to_handler() later on its exit.
300	*/	300	*/
301	ENTRY(ftrace_graph_caller)	301	SYM_FUNC_START(ftrace_graph_caller)
302	mcount_get_pc x0 // function's pc	302	mcount_get_pc x0 // function's pc
303	mcount_get_lr_addr x1 // pointer to function's saved lr	303	mcount_get_lr_addr x1 // pointer to function's saved lr
304	mcount_get_parent_fp x2 // parent's fp	304	mcount_get_parent_fp x2 // parent's fp
305	bl prepare_ftrace_return // prepare_ftrace_return(pc, &lr, fp)	305	bl prepare_ftrace_return // prepare_ftrace_return(pc, &lr, fp)
306		306
307	mcount_exit	307	mcount_exit
308	ENDPROC(ftrace_graph_caller)	308	SYM_FUNC_END(ftrace_graph_caller)
309	#endif /* CONFIG_FUNCTION_GRAPH_TRACER */	309	#endif /* CONFIG_FUNCTION_GRAPH_TRACER */
310	#endif /* CONFIG_DYNAMIC_FTRACE_WITH_REGS */	310	#endif /* CONFIG_DYNAMIC_FTRACE_WITH_REGS */
311		311
312	ENTRY(ftrace_stub)	312	SYM_FUNC_START(ftrace_stub)
313	ret	313	ret
314	ENDPROC(ftrace_stub)	314	SYM_FUNC_END(ftrace_stub)
315		315
316	#ifdef CONFIG_FUNCTION_GRAPH_TRACER	316	#ifdef CONFIG_FUNCTION_GRAPH_TRACER
317	/*	317	/*
@@ -320,7 +320,7 @@ ENDPROC(ftrace_stub) · 2 unmodified lines
320	* Run ftrace_return_to_handler() before going back to parent.	320	* Run ftrace_return_to_handler() before going back to parent.
321	* @fp is checked against the value passed by ftrace_graph_caller().	321	* @fp is checked against the value passed by ftrace_graph_caller().
322	*/	322	*/
323	ENTRY(return_to_handler)	323	SYM_CODE_START(return_to_handler)
324	/* save return value regs */	324	/* save return value regs */
325	sub sp, sp, #64	325	sub sp, sp, #64
326	stp x0, x1, [sp]	326	stp x0, x1, [sp]
@@ -340,5 +340,5 @@ ENTRY(return_to_handler) · 13 unmodified lines
340	add sp, sp, #64	340	add sp, sp, #64
341		341
342	ret	342	ret
343	END(return_to_handler)	343	SYM_CODE_END(return_to_handler)
344	#endif /* CONFIG_FUNCTION_GRAPH_TRACER */	344	#endif /* CONFIG_FUNCTION_GRAPH_TRACER */

arch/arm64/kernel/entry.S+64−57

 #include <asm/alternative.h>
 #include <asm/assembler.h>
 #include <asm/asm-offsets.h>
+#include <asm/asm_pointer_auth.h>
 #include <asm/cpufeature.h>
 #include <asm/errno.h>
 #include <asm/esr.h>
 	apply_ssbd 1, x22, x23
+	ptrauth_keys_install_kernel tsk, 1, x20, x22, x23
 	.else
 	add	x21, sp, #S_FRAME_SIZE
 	get_current_task tsk
 	msr	cntkctl_el1, x1
 4:
 #endif
+	/* No kernel C function calls after this as user keys are set. */
+	ptrauth_keys_install_user tsk, x0, x1, x2
 	apply_ssbd 0, x0, x1
 	.endif
 	.pushsection ".entry.text", "ax"
 	.align	11
-ENTRY(vectors)
+SYM_CODE_START(vectors)
 	kernel_ventry	1, sync_invalid			// Synchronous EL1t
 	kernel_ventry	1, irq_invalid			// IRQ EL1t
 	kernel_ventry	1, fiq_invalid			// FIQ EL1t
 	kernel_ventry	0, fiq_invalid, 32		// FIQ 32-bit EL0
 	kernel_ventry	0, error_invalid, 32		// Error 32-bit EL0
 #endif
-END(vectors)
+SYM_CODE_END(vectors)
 #ifdef CONFIG_VMAP_STACK
 	/*
 	ASM_BUG()
 	.endm
-el0_sync_invalid:
+SYM_CODE_START_LOCAL(el0_sync_invalid)
 	inv_entry 0, BAD_SYNC
-ENDPROC(el0_sync_invalid)
+SYM_CODE_END(el0_sync_invalid)
-el0_irq_invalid:
+SYM_CODE_START_LOCAL(el0_irq_invalid)
 	inv_entry 0, BAD_IRQ
-ENDPROC(el0_irq_invalid)
+SYM_CODE_END(el0_irq_invalid)
-el0_fiq_invalid:
+SYM_CODE_START_LOCAL(el0_fiq_invalid)
 	inv_entry 0, BAD_FIQ
-ENDPROC(el0_fiq_invalid)
+SYM_CODE_END(el0_fiq_invalid)
-el0_error_invalid:
+SYM_CODE_START_LOCAL(el0_error_invalid)
 	inv_entry 0, BAD_ERROR
-ENDPROC(el0_error_invalid)
+SYM_CODE_END(el0_error_invalid)
 #ifdef CONFIG_COMPAT
-el0_fiq_invalid_compat:
+SYM_CODE_START_LOCAL(el0_fiq_invalid_compat)
 	inv_entry 0, BAD_FIQ, 32
-ENDPROC(el0_fiq_invalid_compat)
+SYM_CODE_END(el0_fiq_invalid_compat)
 #endif
-el1_sync_invalid:
+SYM_CODE_START_LOCAL(el1_sync_invalid)
 	inv_entry 1, BAD_SYNC
-ENDPROC(el1_sync_invalid)
+SYM_CODE_END(el1_sync_invalid)
-el1_irq_invalid:
+SYM_CODE_START_LOCAL(el1_irq_invalid)
 	inv_entry 1, BAD_IRQ
-ENDPROC(el1_irq_invalid)
+SYM_CODE_END(el1_irq_invalid)
-el1_fiq_invalid:
+SYM_CODE_START_LOCAL(el1_fiq_invalid)
 	inv_entry 1, BAD_FIQ
-ENDPROC(el1_fiq_invalid)
+SYM_CODE_END(el1_fiq_invalid)
-el1_error_invalid:
+SYM_CODE_START_LOCAL(el1_error_invalid)
 	inv_entry 1, BAD_ERROR
-ENDPROC(el1_error_invalid)
+SYM_CODE_END(el1_error_invalid)
 /*
  * EL1 mode handlers.
  */
 	.align	6
-el1_sync:
+SYM_CODE_START_LOCAL_NOALIGN(el1_sync)
 	kernel_entry 1
 	mov	x0, sp
 	bl	el1_sync_handler
 	kernel_exit 1
-ENDPROC(el1_sync)
+SYM_CODE_END(el1_sync)
 	.align	6
-el1_irq:
+SYM_CODE_START_LOCAL_NOALIGN(el1_irq)
 	kernel_entry 1
 	gic_prio_irq_setup pmr=x20, tmp=x1
 	enable_da_f
 #endif
 	kernel_exit 1
-ENDPROC(el1_irq)
+SYM_CODE_END(el1_irq)
 /*
  * EL0 mode handlers.
  */
 	.align	6
-el0_sync:
+SYM_CODE_START_LOCAL_NOALIGN(el0_sync)
 	kernel_entry 0
 	mov	x0, sp
 	bl	el0_sync_handler
 	b	ret_to_user
-ENDPROC(el0_sync)
+SYM_CODE_END(el0_sync)
 #ifdef CONFIG_COMPAT
 	.align	6
-el0_sync_compat:
+SYM_CODE_START_LOCAL_NOALIGN(el0_sync_compat)
 	kernel_entry 0, 32
 	mov	x0, sp
 	bl	el0_sync_compat_handler
 	b	ret_to_user
-ENDPROC(el0_sync_compat)
+SYM_CODE_END(el0_sync_compat)
 	.align	6
-el0_irq_compat:
+SYM_CODE_START_LOCAL_NOALIGN(el0_irq_compat)
 	kernel_entry 0, 32
 	b	el0_irq_naked
-ENDPROC(el0_irq_compat)
+SYM_CODE_END(el0_irq_compat)
-el0_error_compat:
+SYM_CODE_START_LOCAL_NOALIGN(el0_error_compat)
 	kernel_entry 0, 32
 	b	el0_error_naked
-ENDPROC(el0_error_compat)
+SYM_CODE_END(el0_error_compat)
 #endif
 	.align	6
-el0_irq:
+SYM_CODE_START_LOCAL_NOALIGN(el0_irq)
 	kernel_entry 0
 el0_irq_naked:
 	gic_prio_irq_setup pmr=x20, tmp=x0
 	bl	trace_hardirqs_on
 #endif
 	b	ret_to_user
-ENDPROC(el0_irq)
+SYM_CODE_END(el0_irq)
-el1_error:
+SYM_CODE_START_LOCAL(el1_error)
 	kernel_entry 1
 	mrs	x1, esr_el1
 	gic_prio_kentry_setup tmp=x2
 	mov	x0, sp
 	bl	do_serror
 	kernel_exit 1
-ENDPROC(el1_error)
+SYM_CODE_END(el1_error)
-el0_error:
+SYM_CODE_START_LOCAL(el0_error)
 	kernel_entry 0
 el0_error_naked:
 	mrs	x25, esr_el1
 	bl	do_serror
 	enable_da_f
 	b	ret_to_user
-ENDPROC(el0_error)
+SYM_CODE_END(el0_error)
 /*
  * Ok, we need to do extra processing, enter the slow path.
 	.endm
 	.align	11
-ENTRY(tramp_vectors)
+SYM_CODE_START_NOALIGN(tramp_vectors)
 	.space	0x400
 	tramp_ventry
 	tramp_ventry	32
 	tramp_ventry	32
 	tramp_ventry	32
-END(tramp_vectors)
+SYM_CODE_END(tramp_vectors)
-ENTRY(tramp_exit_native)
+SYM_CODE_START(tramp_exit_native)
 	tramp_exit
-END(tramp_exit_native)
+SYM_CODE_END(tramp_exit_native)
-ENTRY(tramp_exit_compat)
+SYM_CODE_START(tramp_exit_compat)
 	tramp_exit	32
-END(tramp_exit_compat)
+SYM_CODE_END(tramp_exit_compat)
 	.ltorg
 	.popsection				// .entry.tramp.text
 #ifdef CONFIG_RANDOMIZE_BASE
 	.pushsection ".rodata", "a"
 	.align PAGE_SHIFT
-	.globl	__entry_tramp_data_start
+SYM_DATA_START(__entry_tramp_data_start)
-__entry_tramp_data_start:
 	.quad	vectors
+SYM_DATA_END(__entry_tramp_data_start)
 	.popsection				// .rodata
 #endif /* CONFIG_RANDOMIZE_BASE */
 #endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */
  * Previous and next are guaranteed not to be the same.
  *
  */
-ENTRY(cpu_switch_to)
+SYM_FUNC_START(cpu_switch_to)
 	mov	x10, #THREAD_CPU_CONTEXT
 	add	x8, x0, x10
 	mov	x9, sp
 	ldr	lr, [x8]
 	mov	sp, x9
 	msr	sp_el0, x1
+	ptrauth_keys_install_kernel x1, 1, x8, x9, x10
 	ret
-ENDPROC(cpu_switch_to)
+SYM_FUNC_END(cpu_switch_to)
 NOKPROBE(cpu_switch_to)
 /*
  * This is how we return from a fork.
  */
-ENTRY(ret_from_fork)
+SYM_CODE_START(ret_from_fork)
 	bl	schedule_tail
 	cbz	x19, 1f				// not a kernel thread
 	mov	x0, x20
 	blr	x19
 1:	get_current_task tsk
 	b	ret_to_user
-ENDPROC(ret_from_fork)
+SYM_CODE_END(ret_from_fork)
 NOKPROBE(ret_from_fork)
 #ifdef CONFIG_ARM_SDE_INTERFACE
  */
 .ltorg
 .pushsection ".entry.tramp.text", "ax"
-ENTRY(__sdei_asm_entry_trampoline)
+SYM_CODE_START(__sdei_asm_entry_trampoline)
 	mrs	x4, ttbr1_el1
 	tbz	x4, #USER_ASID_BIT, 1f
 	ldr	x4, =__sdei_asm_handler
 #endif
 	br	x4
-ENDPROC(__sdei_asm_entry_trampoline)
+SYM_CODE_END(__sdei_asm_entry_trampoline)
 NOKPROBE(__sdei_asm_entry_trampoline)
 /*
  * x2: exit_mode
  * x4: struct sdei_registered_event argument from registration time.
  */
-ENTRY(__sdei_asm_exit_trampoline)
+SYM_CODE_START(__sdei_asm_exit_trampoline)
 	ldr	x4, [x4, #(SDEI_EVENT_INTREGS + S_ORIG_ADDR_LIMIT)]
 	cbnz	x4, 1f
 	tramp_unmap_kernel	tmp=x4
 1:	sdei_handler_exit exit_mode=x2
-ENDPROC(__sdei_asm_exit_trampoline)
+SYM_CODE_END(__sdei_asm_exit_trampoline)
 NOKPROBE(__sdei_asm_exit_trampoline)
 	.ltorg
 .popsection		// .entry.tramp.text
 #ifdef CONFIG_RANDOMIZE_BASE
 .pushsection ".rodata", "a"
-__sdei_asm_trampoline_next_handler:
+SYM_DATA_START(__sdei_asm_trampoline_next_handler)
 	.quad	__sdei_asm_handler
+SYM_DATA_END(__sdei_asm_trampoline_next_handler)
 .popsection		// .rodata
 #endif /* CONFIG_RANDOMIZE_BASE */
 #endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */
  * follow SMC-CC. We save (or retrieve) all the registers as the handler may
  * want them.
  */
-ENTRY(__sdei_asm_handler)
+SYM_CODE_START(__sdei_asm_handler)
 	stp     x2, x3, [x1, #SDEI_EVENT_INTREGS + S_PC]
 	stp     x4, x5, [x1, #SDEI_EVENT_INTREGS + 16 * 2]
 	stp     x6, x7, [x1, #SDEI_EVENT_INTREGS + 16 * 3]
 	tramp_alias	dst=x5, sym=__sdei_asm_exit_trampoline
 	br	x5
 #endif
-ENDPROC(__sdei_asm_handler)
+SYM_CODE_END(__sdei_asm_handler)
 NOKPROBE(__sdei_asm_handler)
 #endif /* CONFIG_ARM_SDE_INTERFACE */

arch/arm64/kernel/head.S+46−40

 	 *  x24        __primary_switch() .. relocate_kernel()
 	 *                                        current RELR displacement
 	 */
-ENTRY(stext)
+SYM_CODE_START(stext)
 	bl	preserve_boot_args
 	bl	el2_setup			// Drop to EL1, w0=cpu_boot_mode
 	adrp	x23, __PHYS_OFFSET
 	 * On return, the CPU will be ready for the MMU to be turned on and
 	 * the TCR will have been set.
 	 */
+	mov	x0, #ARM64_CPU_BOOT_PRIMARY
 	bl	__cpu_setup			// initialise processor
 	b	__primary_switch
-ENDPROC(stext)
+SYM_CODE_END(stext)
 /*
  * Preserve the arguments passed by the bootloader in x0 .. x3
  */
-preserve_boot_args:
+SYM_CODE_START_LOCAL(preserve_boot_args)
 	mov	x21, x0				// x21=FDT
 	adr_l	x0, boot_args			// record the contents of
 	mov	x1, #0x20			// 4 x 8 bytes
 	b	__inval_dcache_area		// tail call
-ENDPROC(preserve_boot_args)
+SYM_CODE_END(preserve_boot_args)
 /*
  * Macro to create a table entry to the next page.
  *   - first few MB of the kernel linear mapping to jump to once the MMU has
  *     been enabled
  */
-__create_page_tables:
+SYM_FUNC_START_LOCAL(__create_page_tables)
 	mov	x28, lr
 	/*
 	bl	__inval_dcache_area
 	ret	x28
-ENDPROC(__create_page_tables)
+SYM_FUNC_END(__create_page_tables)
-	.ltorg
 /*
  * The following fragment of code is executed with the MMU enabled.
  *
  *   x0 = __PHYS_OFFSET
  */
-__primary_switched:
+SYM_FUNC_START_LOCAL(__primary_switched)
 	adrp	x4, init_thread_union
 	add	sp, x4, #THREAD_SIZE
 	adr_l	x5, init_task
 	mov	x29, #0
 	mov	x30, #0
 	b	start_kernel
-ENDPROC(__primary_switched)
+SYM_FUNC_END(__primary_switched)
+	.pushsection ".rodata", "a"
+SYM_DATA_START(kimage_vaddr)
+	.quad		_text - TEXT_OFFSET
+SYM_DATA_END(kimage_vaddr)
+EXPORT_SYMBOL(kimage_vaddr)
+	.popsection
 /*
  * end early head section, begin head code that is also used for
  */
 	.section ".idmap.text","awx"
-ENTRY(kimage_vaddr)
-	.quad		_text - TEXT_OFFSET
-EXPORT_SYMBOL(kimage_vaddr)
 /*
  * If we're fortunate enough to boot at EL2, ensure that the world is
  * sane before dropping to EL1.
  * Returns either BOOT_CPU_MODE_EL1 or BOOT_CPU_MODE_EL2 in w0 if
  * booted in EL1 or EL2 respectively.
  */
-ENTRY(el2_setup)
+SYM_FUNC_START(el2_setup)
 	msr	SPsel, #1			// We want to use SP_EL{1,2}
 	mrs	x0, CurrentEL
 	cmp	x0, #CurrentEL_EL2
 	isb
 	ret
-install_el2_stub:
+SYM_INNER_LABEL(install_el2_stub, SYM_L_LOCAL)
 	/*
 	 * When VHE is not in use, early init of EL2 and EL1 needs to be
 	 * done here.
 	msr	elr_el2, lr
 	mov	w0, #BOOT_CPU_MODE_EL2		// This CPU booted in EL2
 	eret
-ENDPROC(el2_setup)
+SYM_FUNC_END(el2_setup)
 /*
  * Sets the __boot_cpu_mode flag depending on the CPU boot mode passed
  * in w0. See arch/arm64/include/asm/virt.h for more info.
  */
-set_cpu_boot_mode_flag:
+SYM_FUNC_START_LOCAL(set_cpu_boot_mode_flag)
 	adr_l	x1, __boot_cpu_mode
 	cmp	w0, #BOOT_CPU_MODE_EL2
 	b.ne	1f
 	dmb	sy
 	dc	ivac, x1			// Invalidate potentially stale cache line
 	ret
-ENDPROC(set_cpu_boot_mode_flag)
+SYM_FUNC_END(set_cpu_boot_mode_flag)
 /*
  * These values are written with the MMU off, but read with the MMU on.
  * This is not in .bss, because we set it sufficiently early that the boot-time
  * zeroing of .bss would clobber it.
  */
-ENTRY(__boot_cpu_mode)
+SYM_DATA_START(__boot_cpu_mode)
 	.long	BOOT_CPU_MODE_EL2
 	.long	BOOT_CPU_MODE_EL1
+SYM_DATA_END(__boot_cpu_mode)
 /*
  * The booting CPU updates the failed status @__early_cpu_boot_status,
  * with MMU turned off.
  */
-ENTRY(__early_cpu_boot_status)
+SYM_DATA_START(__early_cpu_boot_status)
 	.quad 	0
+SYM_DATA_END(__early_cpu_boot_status)
 	.popsection
 	 * This provides a "holding pen" for platforms to hold all secondary
 	 * cores are held until we're ready for them to initialise.
 	 */
-ENTRY(secondary_holding_pen)
+SYM_FUNC_START(secondary_holding_pen)
 	bl	el2_setup			// Drop to EL1, w0=cpu_boot_mode
 	bl	set_cpu_boot_mode_flag
 	mrs	x0, mpidr_el1
 	b.eq	secondary_startup
 	wfe
 	b	pen
-ENDPROC(secondary_holding_pen)
+SYM_FUNC_END(secondary_holding_pen)
 	/*
 	 * Secondary entry point that jumps straight into the kernel. Only to
 	 * be used where CPUs are brought online dynamically by the kernel.
 	 */
-ENTRY(secondary_entry)
+SYM_FUNC_START(secondary_entry)
 	bl	el2_setup			// Drop to EL1
 	bl	set_cpu_boot_mode_flag
 	b	secondary_startup
-ENDPROC(secondary_entry)
+SYM_FUNC_END(secondary_entry)
-secondary_startup:
+SYM_FUNC_START_LOCAL(secondary_startup)
 	/*
 	 * Common entry point for secondary CPUs.
 	 */
 	bl	__cpu_secondary_check52bitva
+	mov	x0, #ARM64_CPU_BOOT_SECONDARY
 	bl	__cpu_setup			// initialise processor
 	adrp	x1, swapper_pg_dir
 	bl	__enable_mmu
 	ldr	x8, =__secondary_switched
 	br	x8
-ENDPROC(secondary_startup)
+SYM_FUNC_END(secondary_startup)
-__secondary_switched:
+SYM_FUNC_START_LOCAL(__secondary_switched)
 	adr_l	x5, vectors
 	msr	vbar_el1, x5
 	isb
 	mov	x29, #0
 	mov	x30, #0
 	b	secondary_start_kernel
-ENDPROC(__secondary_switched)
+SYM_FUNC_END(__secondary_switched)
-__secondary_too_slow:
+SYM_FUNC_START_LOCAL(__secondary_too_slow)
 	wfe
 	wfi
 	b	__secondary_too_slow
-ENDPROC(__secondary_too_slow)
+SYM_FUNC_END(__secondary_too_slow)
 /*
  * The booting CPU updates the failed status @__early_cpu_boot_status,
  * Checks if the selected granule size is supported by the CPU.
  * If it isn't, park the CPU
  */
-ENTRY(__enable_mmu)
+SYM_FUNC_START(__enable_mmu)
 	mrs	x2, ID_AA64MMFR0_EL1
 	ubfx	x2, x2, #ID_AA64MMFR0_TGRAN_SHIFT, 4
 	cmp	x2, #ID_AA64MMFR0_TGRAN_SUPPORTED
 	dsb	nsh
 	isb
 	ret
-ENDPROC(__enable_mmu)
+SYM_FUNC_END(__enable_mmu)
-ENTRY(__cpu_secondary_check52bitva)
+SYM_FUNC_START(__cpu_secondary_check52bitva)
 #ifdef CONFIG_ARM64_VA_BITS_52
 	ldr_l	x0, vabits_actual
 	cmp	x0, #52
 #endif
 2:	ret
-ENDPROC(__cpu_secondary_check52bitva)
+SYM_FUNC_END(__cpu_secondary_check52bitva)
-__no_granule_support:
+SYM_FUNC_START_LOCAL(__no_granule_support)
 	/* Indicate that this CPU can't boot and is stuck in the kernel */
 	update_early_cpu_boot_status \
 		CPU_STUCK_IN_KERNEL | CPU_STUCK_REASON_NO_GRAN, x1, x2
 	wfe
 	wfi
 	b	1b
-ENDPROC(__no_granule_support)
+SYM_FUNC_END(__no_granule_support)
 #ifdef CONFIG_RELOCATABLE
-__relocate_kernel:
+SYM_FUNC_START_LOCAL(__relocate_kernel)
 	/*
 	 * Iterate over each entry in the relocation table, and apply the
 	 * relocations in place.
 #endif
 	ret
-ENDPROC(__relocate_kernel)
+SYM_FUNC_END(__relocate_kernel)
 #endif
-__primary_switch:
+SYM_FUNC_START_LOCAL(__primary_switch)
 #ifdef CONFIG_RANDOMIZE_BASE
 	mov	x19, x0				// preserve new SCTLR_EL1 value
 	mrs	x20, sctlr_el1			// preserve old SCTLR_EL1 value
 	ldr	x8, =__primary_switched
 	adrp	x0, __PHYS_OFFSET
 	br	x8
-ENDPROC(__primary_switch)
+SYM_FUNC_END(__primary_switch)

arch/arm64/kernel/hibernate-asm.S+0−2

@@ -110,8 +110,6 @@ ENTRY(swsusp_arch_suspend_exit) · 109 unmodified lines
110	cbz x24, 3f /* Do we need to re-initialise EL2? */	110	cbz x24, 3f /* Do we need to re-initialise EL2? */
111	hvc #0	111	hvc #0
112	3: ret	112	3: ret
113
114	.ltorg
115	ENDPROC(swsusp_arch_suspend_exit)	113	ENDPROC(swsusp_arch_suspend_exit)
116		114
117	/*	115	/*

arch/arm64/kernel/hyp-stub.S+1−1

@@ -63,7 +63,7 @@ el1_sync: · 62 unmodified lines
63	beq 9f // Nothing to reset!	63	beq 9f // Nothing to reset!
64		64
65	/* Someone called kvm_call_hyp() against the hyp-stub... */	65	/* Someone called kvm_call_hyp() against the hyp-stub... */
66	ldr x0, =HVC_STUB_ERR	66	mov_q x0, HVC_STUB_ERR
67	eret	67	eret
68		68
69	9: mov x0, xzr	69	9: mov x0, xzr

arch/arm64/kernel/machine_kexec_file.c+1−1

@@ -121,7 +121,7 @@ static int setup_dtb(struct kimage *image, · 120 unmodified lines
121		121
122	/* add kaslr-seed */	122	/* add kaslr-seed */
123	ret = fdt_delprop(dtb, off, FDT_PROP_KASLR_SEED);	123	ret = fdt_delprop(dtb, off, FDT_PROP_KASLR_SEED);
124	if (ret == -FDT_ERR_NOTFOUND)	124	if (ret == -FDT_ERR_NOTFOUND)
125	ret = 0;	125	ret = 0;
126	else if (ret)	126	else if (ret)
127	goto out;	127	goto out;

arch/arm64/kernel/perf_event.c+181−157

@@ -285,6 +285,17 @@ static struct attribute_group armv8_pmuv3_format_attr_group = { · 284 unmodified lines
285	#define ARMV8_IDX_COUNTER_LAST(cpu_pmu) \	285	#define ARMV8_IDX_COUNTER_LAST(cpu_pmu) \
286	(ARMV8_IDX_CYCLE_COUNTER + cpu_pmu->num_events - 1)	286	(ARMV8_IDX_CYCLE_COUNTER + cpu_pmu->num_events - 1)
287		287
		288
		289	/*
		290	* We unconditionally enable ARMv8.5-PMU long event counter support
		291	* (64-bit events) where supported. Indicate if this arm_pmu has long
		292	* event counter support.
		293	*/
		294	static bool armv8pmu_has_long_event(struct arm_pmu *cpu_pmu)
		295	{
		296	return (cpu_pmu->pmuver >= ID_AA64DFR0_PMUVER_8_5);
		297	}
		298
288	/*	299	/*
289	* We must chain two programmable counters for 64 bit events,	300	* We must chain two programmable counters for 64 bit events,
290	* except when we have allocated the 64bit cycle counter (for CPU	301	* except when we have allocated the 64bit cycle counter (for CPU
@@ -294,9 +305,11 @@ static struct attribute_group armv8_pmuv3_format_attr_group = { · 3 unmodified lines
294	static inline bool armv8pmu_event_is_chained(struct perf_event *event)	305	static inline bool armv8pmu_event_is_chained(struct perf_event *event)
295	{	306	{
296	int idx = event->hw.idx;	307	int idx = event->hw.idx;
		308	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
297		309
298	return !WARN_ON(idx < 0) &&	310	return !WARN_ON(idx < 0) &&
299	armv8pmu_event_is_64bit(event) &&	311	armv8pmu_event_is_64bit(event) &&
		312	!armv8pmu_has_long_event(cpu_pmu) &&
300	(idx != ARMV8_IDX_CYCLE_COUNTER);	313	(idx != ARMV8_IDX_CYCLE_COUNTER);
301	}	314	}
302		315
@@ -345,7 +358,7 @@ static inline void armv8pmu_select_counter(int idx) · 42 unmodified lines
345	isb();	358	isb();
346	}	359	}
347		360
348	static inline u32 armv8pmu_read_evcntr(int idx)	361	static inline u64 armv8pmu_read_evcntr(int idx)
349	{	362	{
350	armv8pmu_select_counter(idx);	363	armv8pmu_select_counter(idx);
351	return read_sysreg(pmxevcntr_el0);	364	return read_sysreg(pmxevcntr_el0);
@@ -362,6 +375,44 @@ static inline u64 armv8pmu_read_hw_counter(struct perf_event *event) · 10 unmodified lines
362	return val;	375	return val;
363	}	376	}
364		377
		378	/*
		379	* The cycle counter is always a 64-bit counter. When ARMV8_PMU_PMCR_LP
		380	* is set the event counters also become 64-bit counters. Unless the
		381	* user has requested a long counter (attr.config1) then we want to
		382	* interrupt upon 32-bit overflow - we achieve this by applying a bias.
		383	*/
		384	static bool armv8pmu_event_needs_bias(struct perf_event *event)
		385	{
		386	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
		387	struct hw_perf_event *hwc = &event->hw;
		388	int idx = hwc->idx;
		389
		390	if (armv8pmu_event_is_64bit(event))
		391	return false;
		392
		393	if (armv8pmu_has_long_event(cpu_pmu) \|\|
		394	idx == ARMV8_IDX_CYCLE_COUNTER)
		395	return true;
		396
		397	return false;
		398	}
		399
		400	static u64 armv8pmu_bias_long_counter(struct perf_event *event, u64 value)
		401	{
		402	if (armv8pmu_event_needs_bias(event))
		403	value \|= GENMASK(63, 32);
		404
		405	return value;
		406	}
		407
		408	static u64 armv8pmu_unbias_long_counter(struct perf_event *event, u64 value)
		409	{
		410	if (armv8pmu_event_needs_bias(event))
		411	value &= ~GENMASK(63, 32);
		412
		413	return value;
		414	}
		415
365	static u64 armv8pmu_read_counter(struct perf_event *event)	416	static u64 armv8pmu_read_counter(struct perf_event *event)
366	{	417	{
367	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);	418	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
@@ -377,10 +428,10 @@ static u64 armv8pmu_read_counter(struct perf_event *event) · 9 unmodified lines
377	else	428	else
378	value = armv8pmu_read_hw_counter(event);	429	value = armv8pmu_read_hw_counter(event);
379		430
380	return value;	431	return armv8pmu_unbias_long_counter(event, value);
381	}	432	}
382		433
383	static inline void armv8pmu_write_evcntr(int idx, u32 value)	434	static inline void armv8pmu_write_evcntr(int idx, u64 value)
384	{	435	{
385	armv8pmu_select_counter(idx);	436	armv8pmu_select_counter(idx);
386	write_sysreg(value, pmxevcntr_el0);	437	write_sysreg(value, pmxevcntr_el0);
@@ -405,20 +456,14 @@ static void armv8pmu_write_counter(struct perf_event *event, u64 value) · 18 unmodified lines
405	struct hw_perf_event *hwc = &event->hw;	456	struct hw_perf_event *hwc = &event->hw;
406	int idx = hwc->idx;	457	int idx = hwc->idx;
407		458
		459	value = armv8pmu_bias_long_counter(event, value);
		460
408	if (!armv8pmu_counter_valid(cpu_pmu, idx))	461	if (!armv8pmu_counter_valid(cpu_pmu, idx))
409	pr_err("CPU%u writing wrong counter %d\n",	462	pr_err("CPU%u writing wrong counter %d\n",
410	smp_processor_id(), idx);	463	smp_processor_id(), idx);
411	else if (idx == ARMV8_IDX_CYCLE_COUNTER) {	464	else if (idx == ARMV8_IDX_CYCLE_COUNTER)
412	/*
413	* The cycles counter is really a 64-bit counter.
414	* When treating it as a 32-bit counter, we only count
415	* the lower 32 bits, and set the upper 32-bits so that
416	* we get an interrupt upon 32-bit overflow.
417	*/
418	if (!armv8pmu_event_is_64bit(event))
419	value \|= 0xffffffff00000000ULL;
420	write_sysreg(value, pmccntr_el0);	465	write_sysreg(value, pmccntr_el0);
421	} else	466	else
422	armv8pmu_write_hw_counter(event, value);	467	armv8pmu_write_hw_counter(event, value);
423	}	468	}
424		469
@@ -450,86 +495,74 @@ static inline void armv8pmu_write_event_type(struct perf_event *event) · 25 unmodified lines
450	}	495	}
451	}	496	}
452		497
453	static inline int armv8pmu_enable_counter(int idx)	498	static u32 armv8pmu_event_cnten_mask(struct perf_event *event)
454	{	499	{
455	u32 counter = ARMV8_IDX_TO_COUNTER(idx);	500	int counter = ARMV8_IDX_TO_COUNTER(event->hw.idx);
456	write_sysreg(BIT(counter), pmcntenset_el0);	501	u32 mask = BIT(counter);
457	return idx;	502
		503	if (armv8pmu_event_is_chained(event))
		504	mask \|= BIT(counter - 1);
		505	return mask;
		506	}
		507
		508	static inline void armv8pmu_enable_counter(u32 mask)
		509	{
		510	write_sysreg(mask, pmcntenset_el0);
458	}	511	}
459		512
460	static inline void armv8pmu_enable_event_counter(struct perf_event *event)	513	static inline void armv8pmu_enable_event_counter(struct perf_event *event)
461	{	514	{
462	struct perf_event_attr *attr = &event->attr;	515	struct perf_event_attr *attr = &event->attr;
463	int idx = event->hw.idx;	516	u32 mask = armv8pmu_event_cnten_mask(event);
464	u32 counter_bits = BIT(ARMV8_IDX_TO_COUNTER(idx));
465		517
466	if (armv8pmu_event_is_chained(event))	518	kvm_set_pmu_events(mask, attr);
467	counter_bits \|= BIT(ARMV8_IDX_TO_COUNTER(idx - 1));
468
469	kvm_set_pmu_events(counter_bits, attr);
470		519
471	/* We rely on the hypervisor switch code to enable guest counters */	520	/* We rely on the hypervisor switch code to enable guest counters */
472	if (!kvm_pmu_counter_deferred(attr)) {	521	if (!kvm_pmu_counter_deferred(attr))
473	armv8pmu_enable_counter(idx);	522	armv8pmu_enable_counter(mask);
474	if (armv8pmu_event_is_chained(event))
475	armv8pmu_enable_counter(idx - 1);
476	}
477	}	523	}
478		524
479	static inline int armv8pmu_disable_counter(int idx)	525	static inline void armv8pmu_disable_counter(u32 mask)
480	{	526	{
481	u32 counter = ARMV8_IDX_TO_COUNTER(idx);	527	write_sysreg(mask, pmcntenclr_el0);
482	write_sysreg(BIT(counter), pmcntenclr_el0);
483	return idx;
484	}	528	}
485		529
486	static inline void armv8pmu_disable_event_counter(struct perf_event *event)	530	static inline void armv8pmu_disable_event_counter(struct perf_event *event)
487	{	531	{
488	struct hw_perf_event *hwc = &event->hw;
489	struct perf_event_attr *attr = &event->attr;	532	struct perf_event_attr *attr = &event->attr;
490	int idx = hwc->idx;	533	u32 mask = armv8pmu_event_cnten_mask(event);
491	u32 counter_bits = BIT(ARMV8_IDX_TO_COUNTER(idx));
492
493	if (armv8pmu_event_is_chained(event))
494	counter_bits \|= BIT(ARMV8_IDX_TO_COUNTER(idx - 1));
495		534
496	kvm_clr_pmu_events(counter_bits);	535	kvm_clr_pmu_events(mask);
497		536
498	/* We rely on the hypervisor switch code to disable guest counters */	537	/* We rely on the hypervisor switch code to disable guest counters */
499	if (!kvm_pmu_counter_deferred(attr)) {	538	if (!kvm_pmu_counter_deferred(attr))
500	if (armv8pmu_event_is_chained(event))	539	armv8pmu_disable_counter(mask);
501	armv8pmu_disable_counter(idx - 1);
502	armv8pmu_disable_counter(idx);
503	}
504	}	540	}
505		541
506	static inline int armv8pmu_enable_intens(int idx)	542	static inline void armv8pmu_enable_intens(u32 mask)
507	{	543	{
508	u32 counter = ARMV8_IDX_TO_COUNTER(idx);	544	write_sysreg(mask, pmintenset_el1);
509	write_sysreg(BIT(counter), pmintenset_el1);
510	return idx;
511	}	545	}
512		546
513	static inline int armv8pmu_enable_event_irq(struct perf_event *event)	547	static inline void armv8pmu_enable_event_irq(struct perf_event *event)
514	{	548	{
515	return armv8pmu_enable_intens(event->hw.idx);	549	u32 counter = ARMV8_IDX_TO_COUNTER(event->hw.idx);
		550	armv8pmu_enable_intens(BIT(counter));
516	}	551	}
517		552
518	static inline int armv8pmu_disable_intens(int idx)	553	static inline void armv8pmu_disable_intens(u32 mask)
519	{	554	{
520	u32 counter = ARMV8_IDX_TO_COUNTER(idx);	555	write_sysreg(mask, pmintenclr_el1);
521	write_sysreg(BIT(counter), pmintenclr_el1);
522	isb();	556	isb();
523	/* Clear the overflow flag in case an interrupt is pending. */	557	/* Clear the overflow flag in case an interrupt is pending. */
524	write_sysreg(BIT(counter), pmovsclr_el0);	558	write_sysreg(mask, pmovsclr_el0);
525	isb();	559	isb();
526
527	return idx;
528	}	560	}
529		561
530	static inline int armv8pmu_disable_event_irq(struct perf_event *event)	562	static inline void armv8pmu_disable_event_irq(struct perf_event *event)
531	{	563	{
532	return armv8pmu_disable_intens(event->hw.idx);	564	u32 counter = ARMV8_IDX_TO_COUNTER(event->hw.idx);
		565	armv8pmu_disable_intens(BIT(counter));
533	}	566	}
534		567
535	static inline u32 armv8pmu_getreset_flags(void)	568	static inline u32 armv8pmu_getreset_flags(void)
@@ -743,7 +776,8 @@ static int armv8pmu_get_event_idx(struct pmu_hw_events *cpuc, · 207 unmodified lines
743	/*	776	/*
744	* Otherwise use events counters	777	* Otherwise use events counters
745	*/	778	*/
746	if (armv8pmu_event_is_64bit(event))	779	if (armv8pmu_event_is_64bit(event) &&
		780	!armv8pmu_has_long_event(cpu_pmu))
747	return armv8pmu_get_chain_idx(cpuc, cpu_pmu);	781	return armv8pmu_get_chain_idx(cpuc, cpu_pmu);
748	else	782	else
749	return armv8pmu_get_single_idx(cpuc, cpu_pmu);	783	return armv8pmu_get_single_idx(cpuc, cpu_pmu);
@@ -815,13 +849,11 @@ static int armv8pmu_filter_match(struct perf_event *event) · 65 unmodified lines
815	static void armv8pmu_reset(void *info)	849	static void armv8pmu_reset(void *info)
816	{	850	{
817	struct arm_pmu cpu_pmu = (struct arm_pmu )info;	851	struct arm_pmu cpu_pmu = (struct arm_pmu )info;
818	u32 idx, nb_cnt = cpu_pmu->num_events;	852	u32 pmcr;
819		853
820	/* The counter and interrupt enable registers are unknown at reset. */	854	/* The counter and interrupt enable registers are unknown at reset. */
821	for (idx = ARMV8_IDX_CYCLE_COUNTER; idx < nb_cnt; ++idx) {	855	armv8pmu_disable_counter(U32_MAX);
822	armv8pmu_disable_counter(idx);	856	armv8pmu_disable_intens(U32_MAX);
823	armv8pmu_disable_intens(idx);
824	}
825		857
826	/* Clear the counters we flip at guest entry/exit */	858	/* Clear the counters we flip at guest entry/exit */
827	kvm_clr_pmu_events(U32_MAX);	859	kvm_clr_pmu_events(U32_MAX);
@@ -830,8 +862,13 @@ static void armv8pmu_reset(void *info) · 2 unmodified lines
830	* Initialize & Reset PMNC. Request overflow interrupt for	862	* Initialize & Reset PMNC. Request overflow interrupt for
831	* 64 bit cycle counter but cheat in armv8pmu_write_counter().	863	* 64 bit cycle counter but cheat in armv8pmu_write_counter().
832	*/	864	*/
833	armv8pmu_pmcr_write(ARMV8_PMU_PMCR_P \| ARMV8_PMU_PMCR_C \|	865	pmcr = ARMV8_PMU_PMCR_P \| ARMV8_PMU_PMCR_C \| ARMV8_PMU_PMCR_LC;
834	ARMV8_PMU_PMCR_LC);	866
		867	/* Enable long event counter support where available */
		868	if (armv8pmu_has_long_event(cpu_pmu))
		869	pmcr \|= ARMV8_PMU_PMCR_LP;
		870
		871	armv8pmu_pmcr_write(pmcr);
835	}	872	}
836		873
837	static int __armv8_pmuv3_map_event(struct perf_event *event,	874	static int __armv8_pmuv3_map_event(struct perf_event *event,
@@ -914,6 +951,7 @@ static void __armv8pmu_probe_pmu(void *info) · 76 unmodified lines
914	if (pmuver == 0xf \|\| pmuver == 0)	951	if (pmuver == 0xf \|\| pmuver == 0)
915	return;	952	return;
916		953
		954	cpu_pmu->pmuver = pmuver;
917	probe->present = true;	955	probe->present = true;
918		956
919	/* Read the nb of CNTx counters supported from PMNC */	957	/* Read the nb of CNTx counters supported from PMNC */
@@ -953,7 +991,10 @@ static int armv8pmu_probe_pmu(struct arm_pmu *cpu_pmu) · 33 unmodified lines
953	return probe.present ? 0 : -ENODEV;	991	return probe.present ? 0 : -ENODEV;
954	}	992	}
955		993
956	static int armv8_pmu_init(struct arm_pmu *cpu_pmu)	994	static int armv8_pmu_init(struct arm_pmu cpu_pmu, char name,
		995	int (map_event)(struct perf_event event),
		996	const struct attribute_group *events,
		997	const struct attribute_group *format)
957	{	998	{
958	int ret = armv8pmu_probe_pmu(cpu_pmu);	999	int ret = armv8pmu_probe_pmu(cpu_pmu);
959	if (ret)	1000	if (ret)
@@ -972,144 +1013,127 @@ static int armv8_pmu_init(struct arm_pmu *cpu_pmu) · 12 unmodified lines
972	cpu_pmu->set_event_filter = armv8pmu_set_event_filter;	1013	cpu_pmu->set_event_filter = armv8pmu_set_event_filter;
973	cpu_pmu->filter_match = armv8pmu_filter_match;	1014	cpu_pmu->filter_match = armv8pmu_filter_match;
974		1015
		1016	cpu_pmu->name = name;
		1017	cpu_pmu->map_event = map_event;
		1018	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] = events ?
		1019	events : &armv8_pmuv3_events_attr_group;
		1020	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] = format ?
		1021	format : &armv8_pmuv3_format_attr_group;
		1022
975	return 0;	1023	return 0;
976	}	1024	}
977		1025
978	static int armv8_pmuv3_init(struct arm_pmu *cpu_pmu)	1026	static int armv8_pmuv3_init(struct arm_pmu *cpu_pmu)
979	{	1027	{
980	int ret = armv8_pmu_init(cpu_pmu);	1028	return armv8_pmu_init(cpu_pmu, "armv8_pmuv3",
981	if (ret)	1029	armv8_pmuv3_map_event, NULL, NULL);
982	return ret;	1030	}
983
984	cpu_pmu->name = "armv8_pmuv3";
985	cpu_pmu->map_event = armv8_pmuv3_map_event;
986	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
987	&armv8_pmuv3_events_attr_group;
988	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
989	&armv8_pmuv3_format_attr_group;
990		1031
991	return 0;	1032	static int armv8_a34_pmu_init(struct arm_pmu *cpu_pmu)
		1033	{
		1034	return armv8_pmu_init(cpu_pmu, "armv8_cortex_a34",
		1035	armv8_pmuv3_map_event, NULL, NULL);
992	}	1036	}
993		1037
994	static int armv8_a35_pmu_init(struct arm_pmu *cpu_pmu)	1038	static int armv8_a35_pmu_init(struct arm_pmu *cpu_pmu)
995	{	1039	{
996	int ret = armv8_pmu_init(cpu_pmu);	1040	return armv8_pmu_init(cpu_pmu, "armv8_cortex_a35",
997	if (ret)	1041	armv8_a53_map_event, NULL, NULL);
998	return ret;
999
1000	cpu_pmu->name = "armv8_cortex_a35";
1001	cpu_pmu->map_event = armv8_a53_map_event;
1002	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
1003	&armv8_pmuv3_events_attr_group;
1004	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
1005	&armv8_pmuv3_format_attr_group;
1006
1007	return 0;
1008	}	1042	}
1009		1043
1010	static int armv8_a53_pmu_init(struct arm_pmu *cpu_pmu)	1044	static int armv8_a53_pmu_init(struct arm_pmu *cpu_pmu)
1011	{	1045	{
1012	int ret = armv8_pmu_init(cpu_pmu);	1046	return armv8_pmu_init(cpu_pmu, "armv8_cortex_a53",
1013	if (ret)	1047	armv8_a53_map_event, NULL, NULL);
1014	return ret;	1048	}
1015
1016	cpu_pmu->name = "armv8_cortex_a53";
1017	cpu_pmu->map_event = armv8_a53_map_event;
1018	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
1019	&armv8_pmuv3_events_attr_group;
1020	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
1021	&armv8_pmuv3_format_attr_group;
1022		1049
1023	return 0;	1050	static int armv8_a55_pmu_init(struct arm_pmu *cpu_pmu)
		1051	{
		1052	return armv8_pmu_init(cpu_pmu, "armv8_cortex_a55",
		1053	armv8_pmuv3_map_event, NULL, NULL);
1024	}	1054	}
1025		1055
1026	static int armv8_a57_pmu_init(struct arm_pmu *cpu_pmu)	1056	static int armv8_a57_pmu_init(struct arm_pmu *cpu_pmu)
1027	{	1057	{
1028	int ret = armv8_pmu_init(cpu_pmu);	1058	return armv8_pmu_init(cpu_pmu, "armv8_cortex_a57",
1029	if (ret)	1059	armv8_a57_map_event, NULL, NULL);
1030	return ret;	1060	}
1031
1032	cpu_pmu->name = "armv8_cortex_a57";
1033	cpu_pmu->map_event = armv8_a57_map_event;
1034	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
1035	&armv8_pmuv3_events_attr_group;
1036	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
1037	&armv8_pmuv3_format_attr_group;
1038		1061
1039	return 0;	1062	static int armv8_a65_pmu_init(struct arm_pmu *cpu_pmu)
		1063	{
		1064	return armv8_pmu_init(cpu_pmu, "armv8_cortex_a65",
		1065	armv8_pmuv3_map_event, NULL, NULL);
1040	}	1066	}
1041		1067
1042	static int armv8_a72_pmu_init(struct arm_pmu *cpu_pmu)	1068	static int armv8_a72_pmu_init(struct arm_pmu *cpu_pmu)
1043	{	1069	{
1044	int ret = armv8_pmu_init(cpu_pmu);	1070	return armv8_pmu_init(cpu_pmu, "armv8_cortex_a72",
1045	if (ret)	1071	armv8_a57_map_event, NULL, NULL);
1046	return ret;
1047
1048	cpu_pmu->name = "armv8_cortex_a72";
1049	cpu_pmu->map_event = armv8_a57_map_event;
1050	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
1051	&armv8_pmuv3_events_attr_group;
1052	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
1053	&armv8_pmuv3_format_attr_group;
1054
1055	return 0;
1056	}	1072	}
1057		1073
1058	static int armv8_a73_pmu_init(struct arm_pmu *cpu_pmu)	1074	static int armv8_a73_pmu_init(struct arm_pmu *cpu_pmu)
1059	{	1075	{
1060	int ret = armv8_pmu_init(cpu_pmu);	1076	return armv8_pmu_init(cpu_pmu, "armv8_cortex_a73",
1061	if (ret)	1077	armv8_a73_map_event, NULL, NULL);
1062	return ret;	1078	}
1063
1064	cpu_pmu->name = "armv8_cortex_a73";
1065	cpu_pmu->map_event = armv8_a73_map_event;
1066	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =
1067	&armv8_pmuv3_events_attr_group;
1068	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =
1069	&armv8_pmuv3_format_attr_group;
1070		1079
1071	return 0;	1080	static int armv8_a75_pmu_init(struct arm_pmu *cpu_pmu)
		1081	{
		1082	return armv8_pmu_init(cpu_pmu, "armv8_cortex_a75",
		1083	armv8_pmuv3_map_event, NULL, NULL);
1072	}	1084	}
1073		1085
1074	static int armv8_thunder_pmu_init(struct arm_pmu *cpu_pmu)	1086	static int armv8_a76_pmu_init(struct arm_pmu *cpu_pmu)
1075	{	1087	{
1076	int ret = armv8_pmu_init(cpu_pmu);	1088	return armv8_pmu_init(cpu_pmu, "armv8_cortex_a76",
1077	if (ret)	1089	armv8_pmuv3_map_event, NULL, NULL);
1078	return ret;	1090	}
1079		1091
1080	cpu_pmu->name = "armv8_cavium_thunder";	1092	static int armv8_a77_pmu_init(struct arm_pmu *cpu_pmu)
1081	cpu_pmu->map_event = armv8_thunder_map_event;	1093	{
1082	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =	1094	return armv8_pmu_init(cpu_pmu, "armv8_cortex_a77",
1083	&armv8_pmuv3_events_attr_group;	1095	armv8_pmuv3_map_event, NULL, NULL);
1084	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =	1096	}
1085	&armv8_pmuv3_format_attr_group;
1086		1097
1087	return 0;	1098	static int armv8_e1_pmu_init(struct arm_pmu *cpu_pmu)
		1099	{
		1100	return armv8_pmu_init(cpu_pmu, "armv8_neoverse_e1",
		1101	armv8_pmuv3_map_event, NULL, NULL);
1088	}	1102	}
1089		1103
1090	static int armv8_vulcan_pmu_init(struct arm_pmu *cpu_pmu)	1104	static int armv8_n1_pmu_init(struct arm_pmu *cpu_pmu)
1091	{	1105	{
1092	int ret = armv8_pmu_init(cpu_pmu);	1106	return armv8_pmu_init(cpu_pmu, "armv8_neoverse_n1",
1093	if (ret)	1107	armv8_pmuv3_map_event, NULL, NULL);
1094	return ret;	1108	}
1095		1109
1096	cpu_pmu->name = "armv8_brcm_vulcan";	1110	static int armv8_thunder_pmu_init(struct arm_pmu *cpu_pmu)
1097	cpu_pmu->map_event = armv8_vulcan_map_event;	1111	{
1098	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] =	1112	return armv8_pmu_init(cpu_pmu, "armv8_cavium_thunder",
1099	&armv8_pmuv3_events_attr_group;	1113	armv8_thunder_map_event, NULL, NULL);
1100	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] =	1114	}
1101	&armv8_pmuv3_format_attr_group;
1102		1115
1103	return 0;	1116	static int armv8_vulcan_pmu_init(struct arm_pmu *cpu_pmu)
		1117	{
		1118	return armv8_pmu_init(cpu_pmu, "armv8_brcm_vulcan",
		1119	armv8_vulcan_map_event, NULL, NULL);
1104	}	1120	}
1105		1121
1106	static const struct of_device_id armv8_pmu_of_device_ids[] = {	1122	static const struct of_device_id armv8_pmu_of_device_ids[] = {
1107	{.compatible = "arm,armv8-pmuv3", .data = armv8_pmuv3_init},	1123	{.compatible = "arm,armv8-pmuv3", .data = armv8_pmuv3_init},
		1124	{.compatible = "arm,cortex-a34-pmu", .data = armv8_a34_pmu_init},
1108	{.compatible = "arm,cortex-a35-pmu", .data = armv8_a35_pmu_init},	1125	{.compatible = "arm,cortex-a35-pmu", .data = armv8_a35_pmu_init},
1109	{.compatible = "arm,cortex-a53-pmu", .data = armv8_a53_pmu_init},	1126	{.compatible = "arm,cortex-a53-pmu", .data = armv8_a53_pmu_init},
		1127	{.compatible = "arm,cortex-a55-pmu", .data = armv8_a55_pmu_init},
1110	{.compatible = "arm,cortex-a57-pmu", .data = armv8_a57_pmu_init},	1128	{.compatible = "arm,cortex-a57-pmu", .data = armv8_a57_pmu_init},
		1129	{.compatible = "arm,cortex-a65-pmu", .data = armv8_a65_pmu_init},
1111	{.compatible = "arm,cortex-a72-pmu", .data = armv8_a72_pmu_init},	1130	{.compatible = "arm,cortex-a72-pmu", .data = armv8_a72_pmu_init},
1112	{.compatible = "arm,cortex-a73-pmu", .data = armv8_a73_pmu_init},	1131	{.compatible = "arm,cortex-a73-pmu", .data = armv8_a73_pmu_init},
		1132	{.compatible = "arm,cortex-a75-pmu", .data = armv8_a75_pmu_init},
		1133	{.compatible = "arm,cortex-a76-pmu", .data = armv8_a76_pmu_init},
		1134	{.compatible = "arm,cortex-a77-pmu", .data = armv8_a77_pmu_init},
		1135	{.compatible = "arm,neoverse-e1-pmu", .data = armv8_e1_pmu_init},
		1136	{.compatible = "arm,neoverse-n1-pmu", .data = armv8_n1_pmu_init},
1113	{.compatible = "cavium,thunder-pmu", .data = armv8_thunder_pmu_init},	1137	{.compatible = "cavium,thunder-pmu", .data = armv8_thunder_pmu_init},
1114	{.compatible = "brcm,vulcan-pmu", .data = armv8_vulcan_pmu_init},	1138	{.compatible = "brcm,vulcan-pmu", .data = armv8_vulcan_pmu_init},
1115	{},	1139	{},

arch/arm64/kernel/pointer_auth.c+2−5

 int ptrauth_prctl_reset_keys(struct task_struct *tsk, unsigned long arg)
 {
-	struct ptrauth_keys *keys = &tsk->thread.keys_user;
+	struct ptrauth_keys_user *keys = &tsk->thread.keys_user;
 	unsigned long addr_key_mask = PR_PAC_APIAKEY | PR_PAC_APIBKEY |
 				      PR_PAC_APDAKEY | PR_PAC_APDBKEY;
 	unsigned long key_mask = addr_key_mask | PR_PAC_APGAKEY;
 		return -EINVAL;
 	if (!arg) {
-		ptrauth_keys_init(keys);
+		ptrauth_keys_init_user(keys);
-		ptrauth_keys_switch(keys);
 		return 0;
 	}
 	if (arg & PR_PAC_APGAKEY)
 		get_random_bytes(&keys->apga, sizeof(keys->apga));
-	ptrauth_keys_switch(keys);
 	return 0;
 }

arch/arm64/kernel/process.c+3−2

 	if (!user_mode(regs)) {
 		printk("pc : %pS\n", (void *)regs->pc);
-		printk("lr : %pS\n", (void *)lr);
+		printk("lr : %pS\n", (void *)ptrauth_strip_insn_pac(lr));
 	} else {
 		printk("pc : %016llx\n", regs->pc);
 		printk("lr : %016llx\n", lr);
 	 */
 	fpsimd_flush_task_state(p);
+	ptrauth_thread_init_kernel(p);
 	if (likely(!(p->flags & PF_KTHREAD))) {
 		*childregs = *current_pt_regs();
 		childregs->regs[0] = 0;
 	contextidr_thread_switch(next);
 	entry_task_switch(next);
 	uao_thread_switch(next);
-	ptrauth_thread_switch(next);
 	ssbs_thread_switch(next);
 	/*

arch/arm64/kernel/ptrace.c+8−8

arch/arm64/kernel/relocate_kernel.S+1−3

 	cmp	x0, #CurrentEL_EL2
 	b.ne	1f
 	mrs	x0, sctlr_el2
-	ldr	x1, =SCTLR_ELx_FLAGS
+	mov_q	x1, SCTLR_ELx_FLAGS
 	bic	x0, x0, x1
 	pre_disable_mmu_workaround
 	msr	sctlr_el2, x0
 ENDPROC(arm64_relocate_new_kernel)
-.ltorg
 .align 3	/* To keep the 64-bit values below naturally aligned. */
 .Lcopy_end:

arch/arm64/kernel/setup.c+5−3

@@ -344,7 +344,7 @@ void __init setup_arch(char **cmdline_p) · 343 unmodified lines
344	else	344	else
345	psci_acpi_init();	345	psci_acpi_init();
346		346
347	cpu_read_bootcpu_ops();	347	init_bootcpu_ops();
348	smp_init_cpus();	348	smp_init_cpus();
349	smp_build_mpidr_hash();	349	smp_build_mpidr_hash();
350		350
@@ -371,8 +371,10 @@ void __init setup_arch(char **cmdline_p) · 20 unmodified lines
371	static inline bool cpu_can_disable(unsigned int cpu)	371	static inline bool cpu_can_disable(unsigned int cpu)
372	{	372	{
373	#ifdef CONFIG_HOTPLUG_CPU	373	#ifdef CONFIG_HOTPLUG_CPU
374	if (cpu_ops[cpu] && cpu_ops[cpu]->cpu_can_disable)	374	const struct cpu_operations *ops = get_cpu_ops(cpu);
375	return cpu_ops[cpu]->cpu_can_disable(cpu);	375
		376	if (ops && ops->cpu_can_disable)
		377	return ops->cpu_can_disable(cpu);
376	#endif	378	#endif
377	return false;	379	return false;
378	}	380	}

arch/arm64/kernel/sleep.S+2−0

 #include <linux/linkage.h>
 #include <asm/asm-offsets.h>
 #include <asm/assembler.h>
+#include <asm/smp.h>
 	.text
 /*
 	.pushsection ".idmap.text", "awx"
 ENTRY(cpu_resume)
 	bl	el2_setup		// if in EL2 drop to EL1 cleanly
+	mov	x0, #ARM64_CPU_RUNTIME
 	bl	__cpu_setup
 	/* enable the MMU early - so we can access sleep_save_stash by va */
 	adrp	x1, swapper_pg_dir

arch/arm64/kernel/smp.c+92−67

@@ -93,8 +93,10 @@ static inline int op_cpu_kill(unsigned int cpu) · 92 unmodified lines
93	*/	93	*/
94	static int boot_secondary(unsigned int cpu, struct task_struct *idle)	94	static int boot_secondary(unsigned int cpu, struct task_struct *idle)
95	{	95	{
96	if (cpu_ops[cpu]->cpu_boot)	96	const struct cpu_operations *ops = get_cpu_ops(cpu);
97	return cpu_ops[cpu]->cpu_boot(cpu);	97
		98	if (ops->cpu_boot)
		99	return ops->cpu_boot(cpu);
98		100
99	return -EOPNOTSUPP;	101	return -EOPNOTSUPP;
100	}	102	}
@@ -112,63 +114,66 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) · 11 unmodified lines
112	*/	114	*/
113	secondary_data.task = idle;	115	secondary_data.task = idle;
114	secondary_data.stack = task_stack_page(idle) + THREAD_SIZE;	116	secondary_data.stack = task_stack_page(idle) + THREAD_SIZE;
		117	#if defined(CONFIG_ARM64_PTR_AUTH)
		118	secondary_data.ptrauth_key.apia.lo = idle->thread.keys_kernel.apia.lo;
		119	secondary_data.ptrauth_key.apia.hi = idle->thread.keys_kernel.apia.hi;
		120	#endif
115	update_cpu_boot_status(CPU_MMU_OFF);	121	update_cpu_boot_status(CPU_MMU_OFF);
116	__flush_dcache_area(&secondary_data, sizeof(secondary_data));	122	__flush_dcache_area(&secondary_data, sizeof(secondary_data));
117		123
118	/*	124	/* Now bring the CPU into our world */
119	* Now bring the CPU into our world.
120	*/
121	ret = boot_secondary(cpu, idle);	125	ret = boot_secondary(cpu, idle);
122	if (ret == 0) {	126	if (ret) {
123	/*
124	* CPU was successfully started, wait for it to come online or
125	* time out.
126	*/
127	wait_for_completion_timeout(&cpu_running,
128	msecs_to_jiffies(5000));
129
130	if (!cpu_online(cpu)) {
131	pr_crit("CPU%u: failed to come online\n", cpu);
132	ret = -EIO;
133	}
134	} else {
135	pr_err("CPU%u: failed to boot: %d\n", cpu, ret);	127	pr_err("CPU%u: failed to boot: %d\n", cpu, ret);
136	return ret;	128	return ret;
137	}	129	}
138		130
		131	/*
		132	* CPU was successfully started, wait for it to come online or
		133	* time out.
		134	*/
		135	wait_for_completion_timeout(&cpu_running,
		136	msecs_to_jiffies(5000));
		137	if (cpu_online(cpu))
		138	return 0;
		139
		140	pr_crit("CPU%u: failed to come online\n", cpu);
139	secondary_data.task = NULL;	141	secondary_data.task = NULL;
140	secondary_data.stack = NULL;	142	secondary_data.stack = NULL;
		143	#if defined(CONFIG_ARM64_PTR_AUTH)
		144	secondary_data.ptrauth_key.apia.lo = 0;
		145	secondary_data.ptrauth_key.apia.hi = 0;
		146	#endif
141	__flush_dcache_area(&secondary_data, sizeof(secondary_data));	147	__flush_dcache_area(&secondary_data, sizeof(secondary_data));
142	status = READ_ONCE(secondary_data.status);	148	status = READ_ONCE(secondary_data.status);
143	if (ret && status) {	149	if (status == CPU_MMU_OFF)
144		150	status = READ_ONCE(__early_cpu_boot_status);
145	if (status == CPU_MMU_OFF)
146	status = READ_ONCE(__early_cpu_boot_status);
147		151
148	switch (status & CPU_BOOT_STATUS_MASK) {	152	switch (status & CPU_BOOT_STATUS_MASK) {
149	default:	153	default:
150	pr_err("CPU%u: failed in unknown state : 0x%lx\n",	154	pr_err("CPU%u: failed in unknown state : 0x%lx\n",
151	cpu, status);	155	cpu, status);
152	cpus_stuck_in_kernel++;	156	cpus_stuck_in_kernel++;
153	break;	157	break;
154	case CPU_KILL_ME:	158	case CPU_KILL_ME:
155	if (!op_cpu_kill(cpu)) {	159	if (!op_cpu_kill(cpu)) {
156	pr_crit("CPU%u: died during early boot\n", cpu);	160	pr_crit("CPU%u: died during early boot\n", cpu);
157	break;
158	}
159	pr_crit("CPU%u: may not have shut down cleanly\n", cpu);
160	/* Fall through */
161	case CPU_STUCK_IN_KERNEL:
162	pr_crit("CPU%u: is stuck in kernel\n", cpu);
163	if (status & CPU_STUCK_REASON_52_BIT_VA)
164	pr_crit("CPU%u: does not support 52-bit VAs\n", cpu);
165	if (status & CPU_STUCK_REASON_NO_GRAN)
166	pr_crit("CPU%u: does not support %luK granule \n", cpu, PAGE_SIZE / SZ_1K);
167	cpus_stuck_in_kernel++;
168	break;	161	break;
169	case CPU_PANIC_KERNEL:
170	panic("CPU%u detected unsupported configuration\n", cpu);
171	}	162	}
		163	pr_crit("CPU%u: may not have shut down cleanly\n", cpu);
		164	/* Fall through */
		165	case CPU_STUCK_IN_KERNEL:
		166	pr_crit("CPU%u: is stuck in kernel\n", cpu);
		167	if (status & CPU_STUCK_REASON_52_BIT_VA)
		168	pr_crit("CPU%u: does not support 52-bit VAs\n", cpu);
		169	if (status & CPU_STUCK_REASON_NO_GRAN) {
		170	pr_crit("CPU%u: does not support %luK granule\n",
		171	cpu, PAGE_SIZE / SZ_1K);
		172	}
		173	cpus_stuck_in_kernel++;
		174	break;
		175	case CPU_PANIC_KERNEL:
		176	panic("CPU%u detected unsupported configuration\n", cpu);
172	}	177	}
173		178
174	return ret;	179	return ret;
@@ -196,6 +201,7 @@ asmlinkage notrace void secondary_start_kernel(void) · 21 unmodified lines
196	{	201	{
197	u64 mpidr = read_cpuid_mpidr() & MPIDR_HWID_BITMASK;	202	u64 mpidr = read_cpuid_mpidr() & MPIDR_HWID_BITMASK;
198	struct mm_struct *mm = &init_mm;	203	struct mm_struct *mm = &init_mm;
		204	const struct cpu_operations *ops;
199	unsigned int cpu;	205	unsigned int cpu;
200		206
201	cpu = task_cpu(current);	207	cpu = task_cpu(current);
@@ -227,8 +233,9 @@ asmlinkage notrace void secondary_start_kernel(void) · 25 unmodified lines
227	*/	233	*/
228	check_local_cpu_capabilities();	234	check_local_cpu_capabilities();
229		235
230	if (cpu_ops[cpu]->cpu_postboot)	236	ops = get_cpu_ops(cpu);
231	cpu_ops[cpu]->cpu_postboot();	237	if (ops->cpu_postboot)
		238	ops->cpu_postboot();
232		239
233	/*	240	/*
234	* Log the CPU info before it is marked online and might get read.	241	* Log the CPU info before it is marked online and might get read.
@@ -266,19 +273,21 @@ asmlinkage notrace void secondary_start_kernel(void) · 31 unmodified lines
266	#ifdef CONFIG_HOTPLUG_CPU	273	#ifdef CONFIG_HOTPLUG_CPU
267	static int op_cpu_disable(unsigned int cpu)	274	static int op_cpu_disable(unsigned int cpu)
268	{	275	{
		276	const struct cpu_operations *ops = get_cpu_ops(cpu);
		277
269	/*	278	/*
270	* If we don't have a cpu_die method, abort before we reach the point	279	* If we don't have a cpu_die method, abort before we reach the point
271	* of no return. CPU0 may not have an cpu_ops, so test for it.	280	* of no return. CPU0 may not have an cpu_ops, so test for it.
272	*/	281	*/
273	if (!cpu_ops[cpu] \|\| !cpu_ops[cpu]->cpu_die)	282	if (!ops \|\| !ops->cpu_die)
274	return -EOPNOTSUPP;	283	return -EOPNOTSUPP;
275		284
276	/*	285	/*
277	* We may need to abort a hot unplug for some other mechanism-specific	286	* We may need to abort a hot unplug for some other mechanism-specific
278	* reason.	287	* reason.
279	*/	288	*/
280	if (cpu_ops[cpu]->cpu_disable)	289	if (ops->cpu_disable)
281	return cpu_ops[cpu]->cpu_disable(cpu);	290	return ops->cpu_disable(cpu);
282		291
283	return 0;	292	return 0;
284	}	293	}
@@ -314,15 +323,17 @@ int __cpu_disable(void) · 29 unmodified lines
314		323
315	static int op_cpu_kill(unsigned int cpu)	324	static int op_cpu_kill(unsigned int cpu)
316	{	325	{
		326	const struct cpu_operations *ops = get_cpu_ops(cpu);
		327
317	/*	328	/*
318	* If we have no means of synchronising with the dying CPU, then assume	329	* If we have no means of synchronising with the dying CPU, then assume
319	* that it is really dead. We can only wait for an arbitrary length of	330	* that it is really dead. We can only wait for an arbitrary length of
320	* time and hope that it's dead, so let's skip the wait and just hope.	331	* time and hope that it's dead, so let's skip the wait and just hope.
321	*/	332	*/
322	if (!cpu_ops[cpu]->cpu_kill)	333	if (!ops->cpu_kill)
323	return 0;	334	return 0;
324		335
325	return cpu_ops[cpu]->cpu_kill(cpu);	336	return ops->cpu_kill(cpu);
326	}	337	}
327		338
328	/*	339	/*
@@ -357,6 +368,7 @@ void __cpu_die(unsigned int cpu) · 28 unmodified lines
357	void cpu_die(void)	368	void cpu_die(void)
358	{	369	{
359	unsigned int cpu = smp_processor_id();	370	unsigned int cpu = smp_processor_id();
		371	const struct cpu_operations *ops = get_cpu_ops(cpu);
360		372
361	idle_task_exit();	373	idle_task_exit();
362		374
@@ -370,12 +382,22 @@ void cpu_die(void) · 7 unmodified lines
370	* mechanism must perform all required cache maintenance to ensure that	382	* mechanism must perform all required cache maintenance to ensure that
371	* no dirty lines are lost in the process of shutting down the CPU.	383	* no dirty lines are lost in the process of shutting down the CPU.
372	*/	384	*/
373	cpu_ops[cpu]->cpu_die(cpu);	385	ops->cpu_die(cpu);
374		386
375	BUG();	387	BUG();
376	}	388	}
377	#endif	389	#endif
378		390
		391	static void __cpu_try_die(int cpu)
		392	{
		393	#ifdef CONFIG_HOTPLUG_CPU
		394	const struct cpu_operations *ops = get_cpu_ops(cpu);
		395
		396	if (ops && ops->cpu_die)
		397	ops->cpu_die(cpu);
		398	#endif
		399	}
		400
379	/*	401	/*
380	* Kill the calling secondary CPU, early in bringup before it is turned	402	* Kill the calling secondary CPU, early in bringup before it is turned
381	* online.	403	* online.
@@ -389,12 +411,11 @@ void cpu_die_early(void) · 7 unmodified lines
389	/* Mark this CPU absent */	411	/* Mark this CPU absent */
390	set_cpu_present(cpu, 0);	412	set_cpu_present(cpu, 0);
391		413
392	#ifdef CONFIG_HOTPLUG_CPU	414	if (IS_ENABLED(CONFIG_HOTPLUG_CPU)) {
393	update_cpu_boot_status(CPU_KILL_ME);	415	update_cpu_boot_status(CPU_KILL_ME);
394	/* Check if we can park ourselves */	416	__cpu_try_die(cpu);
395	if (cpu_ops[cpu] && cpu_ops[cpu]->cpu_die)	417	}
396	cpu_ops[cpu]->cpu_die(cpu);	418
397	#endif
398	update_cpu_boot_status(CPU_STUCK_IN_KERNEL);	419	update_cpu_boot_status(CPU_STUCK_IN_KERNEL);
399		420
400	cpu_park_loop();	421	cpu_park_loop();
@@ -488,10 +509,13 @@ static bool __init is_mpidr_duplicate(unsigned int cpu, u64 hwid) · 87 unmodified lines
488	*/	509	*/
489	static int __init smp_cpu_setup(int cpu)	510	static int __init smp_cpu_setup(int cpu)
490	{	511	{
491	if (cpu_read_ops(cpu))	512	const struct cpu_operations *ops;
		513
		514	if (init_cpu_ops(cpu))
492	return -ENODEV;	515	return -ENODEV;
493		516
494	if (cpu_ops[cpu]->cpu_init(cpu))	517	ops = get_cpu_ops(cpu);
		518	if (ops->cpu_init(cpu))
495	return -ENODEV;	519	return -ENODEV;
496		520
497	set_cpu_possible(cpu, true);	521	set_cpu_possible(cpu, true);
@@ -714,6 +738,7 @@ void __init smp_init_cpus(void) · 216 unmodified lines
714		738
715	void __init smp_prepare_cpus(unsigned int max_cpus)	739	void __init smp_prepare_cpus(unsigned int max_cpus)
716	{	740	{
		741	const struct cpu_operations *ops;
717	int err;	742	int err;
718	unsigned int cpu;	743	unsigned int cpu;
719	unsigned int this_cpu;	744	unsigned int this_cpu;
@@ -744,10 +769,11 @@ void __init smp_prepare_cpus(unsigned int max_cpus) · 24 unmodified lines
744	if (cpu == smp_processor_id())	769	if (cpu == smp_processor_id())
745	continue;	770	continue;
746		771
747	if (!cpu_ops[cpu])	772	ops = get_cpu_ops(cpu);
		773	if (!ops)
748	continue;	774	continue;
749		775
750	err = cpu_ops[cpu]->cpu_prepare(cpu);	776	err = ops->cpu_prepare(cpu);
751	if (err)	777	if (err)
752	continue;	778	continue;
753		779
@@ -863,10 +889,8 @@ static void ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs) · 109 unmodified lines
863	local_irq_disable();	889	local_irq_disable();
864	sdei_mask_local_cpu();	890	sdei_mask_local_cpu();
865		891
866	#ifdef CONFIG_HOTPLUG_CPU	892	if (IS_ENABLED(CONFIG_HOTPLUG_CPU))
867	if (cpu_ops[cpu]->cpu_die)	893	__cpu_try_die(cpu);
868	cpu_ops[cpu]->cpu_die(cpu);
869	#endif
870		894
871	/* just in case */	895	/* just in case */
872	cpu_park_loop();	896	cpu_park_loop();
@@ -1059,8 +1083,9 @@ static bool have_cpu_die(void) · 186 unmodified lines
1059	{	1083	{
1060	#ifdef CONFIG_HOTPLUG_CPU	1084	#ifdef CONFIG_HOTPLUG_CPU
1061	int any_cpu = raw_smp_processor_id();	1085	int any_cpu = raw_smp_processor_id();
		1086	const struct cpu_operations *ops = get_cpu_ops(any_cpu);
1062		1087
1063	if (cpu_ops[any_cpu] && cpu_ops[any_cpu]->cpu_die)	1088	if (ops && ops->cpu_die)
1064	return true;	1089	return true;
1065	#endif	1090	#endif
1066	return false;	1091	return false;

arch/arm64/kernel/stacktrace.c+4−1

 #include <linux/stacktrace.h>
 #include <asm/irq.h>
+#include <asm/pointer_auth.h>
 #include <asm/stack_pointer.h>
 #include <asm/stacktrace.h>
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
 	if (tsk->ret_stack &&
-			(frame->pc == (unsigned long)return_to_handler)) {
+		(ptrauth_strip_insn_pac(frame->pc) == (unsigned long)return_to_handler)) {
 		struct ftrace_ret_stack *ret_stack;
 		/*
 		 * This is a case where function graph tracer has
 	}
 #endif /* CONFIG_FUNCTION_GRAPH_TRACER */
+	frame->pc = ptrauth_strip_insn_pac(frame->pc);
 	/*
 	 * Frames created upon entry from EL0 have NULL FP and PC values, so
 	 * don't bother reporting these. Frames created by __noreturn functions

arch/arm64/kernel/topology.c+180−0

@@ -14,6 +14,7 @@ · 13 unmodified lines
14	#include <linux/acpi.h>	14	#include <linux/acpi.h>
15	#include <linux/arch_topology.h>	15	#include <linux/arch_topology.h>
16	#include <linux/cacheinfo.h>	16	#include <linux/cacheinfo.h>
		17	#include <linux/cpufreq.h>
17	#include <linux/init.h>	18	#include <linux/init.h>
18	#include <linux/percpu.h>	19	#include <linux/percpu.h>
19		20
@@ -120,4 +121,183 @@ int __init parse_acpi_topology(void) · 100 unmodified lines
120	}	121	}
121	#endif	122	#endif
122		123
		124	#ifdef CONFIG_ARM64_AMU_EXTN
123		125
		126	#undef pr_fmt
		127	#define pr_fmt(fmt) "AMU: " fmt
		128
		129	static DEFINE_PER_CPU_READ_MOSTLY(unsigned long, arch_max_freq_scale);
		130	static DEFINE_PER_CPU(u64, arch_const_cycles_prev);
		131	static DEFINE_PER_CPU(u64, arch_core_cycles_prev);
		132	static cpumask_var_t amu_fie_cpus;
		133
		134	/* Initialize counter reference per-cpu variables for the current CPU */
		135	void init_cpu_freq_invariance_counters(void)
		136	{
		137	this_cpu_write(arch_core_cycles_prev,
		138	read_sysreg_s(SYS_AMEVCNTR0_CORE_EL0));
		139	this_cpu_write(arch_const_cycles_prev,
		140	read_sysreg_s(SYS_AMEVCNTR0_CONST_EL0));
		141	}
		142
		143	static int validate_cpu_freq_invariance_counters(int cpu)
		144	{
		145	u64 max_freq_hz, ratio;
		146
		147	if (!cpu_has_amu_feat(cpu)) {
		148	pr_debug("CPU%d: counters are not supported.\n", cpu);
		149	return -EINVAL;
		150	}
		151
		152	if (unlikely(!per_cpu(arch_const_cycles_prev, cpu) \|\|
		153	!per_cpu(arch_core_cycles_prev, cpu))) {
		154	pr_debug("CPU%d: cycle counters are not enabled.\n", cpu);
		155	return -EINVAL;
		156	}
		157
		158	/* Convert maximum frequency from KHz to Hz and validate */
		159	max_freq_hz = cpufreq_get_hw_max_freq(cpu) * 1000;
		160	if (unlikely(!max_freq_hz)) {
		161	pr_debug("CPU%d: invalid maximum frequency.\n", cpu);
		162	return -EINVAL;
		163	}
		164
		165	/*
		166	* Pre-compute the fixed ratio between the frequency of the constant
		167	* counter and the maximum frequency of the CPU.
		168	*
		169	* const_freq
		170	* arch_max_freq_scale = ---------------- * SCHED_CAPACITY_SCALE²
		171	* cpuinfo_max_freq
		172	*
		173	* We use a factor of 2 * SCHED_CAPACITY_SHIFT -> SCHED_CAPACITY_SCALE²
		174	* in order to ensure a good resolution for arch_max_freq_scale for
		175	* very low arch timer frequencies (down to the KHz range which should
		176	* be unlikely).
		177	*/
		178	ratio = (u64)arch_timer_get_rate() << (2 * SCHED_CAPACITY_SHIFT);
		179	ratio = div64_u64(ratio, max_freq_hz);
		180	if (!ratio) {
		181	WARN_ONCE(1, "System timer frequency too low.\n");
		182	return -EINVAL;
		183	}
		184
		185	per_cpu(arch_max_freq_scale, cpu) = (unsigned long)ratio;
		186
		187	return 0;
		188	}
		189
		190	static inline bool
		191	enable_policy_freq_counters(int cpu, cpumask_var_t valid_cpus)
		192	{
		193	struct cpufreq_policy *policy = cpufreq_cpu_get(cpu);
		194
		195	if (!policy) {
		196	pr_debug("CPU%d: No cpufreq policy found.\n", cpu);
		197	return false;
		198	}
		199
		200	if (cpumask_subset(policy->related_cpus, valid_cpus))
		201	cpumask_or(amu_fie_cpus, policy->related_cpus,
		202	amu_fie_cpus);
		203
		204	cpufreq_cpu_put(policy);
		205
		206	return true;
		207	}
		208
		209	static DEFINE_STATIC_KEY_FALSE(amu_fie_key);
		210	#define amu_freq_invariant() static_branch_unlikely(&amu_fie_key)
		211
		212	static int __init init_amu_fie(void)
		213	{
		214	cpumask_var_t valid_cpus;
		215	bool have_policy = false;
		216	int ret = 0;
		217	int cpu;
		218
		219	if (!zalloc_cpumask_var(&valid_cpus, GFP_KERNEL))
		220	return -ENOMEM;
		221
		222	if (!zalloc_cpumask_var(&amu_fie_cpus, GFP_KERNEL)) {
		223	ret = -ENOMEM;
		224	goto free_valid_mask;
		225	}
		226
		227	for_each_present_cpu(cpu) {
		228	if (validate_cpu_freq_invariance_counters(cpu))
		229	continue;
		230	cpumask_set_cpu(cpu, valid_cpus);
		231	have_policy \|= enable_policy_freq_counters(cpu, valid_cpus);
		232	}
		233
		234	/*
		235	* If we are not restricted by cpufreq policies, we only enable
		236	* the use of the AMU feature for FIE if all CPUs support AMU.
		237	* Otherwise, enable_policy_freq_counters has already enabled
		238	* policy cpus.
		239	*/
		240	if (!have_policy && cpumask_equal(valid_cpus, cpu_present_mask))
		241	cpumask_or(amu_fie_cpus, amu_fie_cpus, valid_cpus);
		242
		243	if (!cpumask_empty(amu_fie_cpus)) {
		244	pr_info("CPUs[%*pbl]: counters will be used for FIE.",
		245	cpumask_pr_args(amu_fie_cpus));
		246	static_branch_enable(&amu_fie_key);
		247	}
		248
		249	free_valid_mask:
		250	free_cpumask_var(valid_cpus);
		251
		252	return ret;
		253	}
		254	late_initcall_sync(init_amu_fie);
		255
		256	bool arch_freq_counters_available(struct cpumask *cpus)
		257	{
		258	return amu_freq_invariant() &&
		259	cpumask_subset(cpus, amu_fie_cpus);
		260	}
		261
		262	void topology_scale_freq_tick(void)
		263	{
		264	u64 prev_core_cnt, prev_const_cnt;
		265	u64 core_cnt, const_cnt, scale;
		266	int cpu = smp_processor_id();
		267
		268	if (!amu_freq_invariant())
		269	return;
		270
		271	if (!cpumask_test_cpu(cpu, amu_fie_cpus))
		272	return;
		273
		274	const_cnt = read_sysreg_s(SYS_AMEVCNTR0_CONST_EL0);
		275	core_cnt = read_sysreg_s(SYS_AMEVCNTR0_CORE_EL0);
		276	prev_const_cnt = this_cpu_read(arch_const_cycles_prev);
		277	prev_core_cnt = this_cpu_read(arch_core_cycles_prev);
		278
		279	if (unlikely(core_cnt <= prev_core_cnt \|\|
		280	const_cnt <= prev_const_cnt))
		281	goto store_and_exit;
		282
		283	/*
		284	* /\core arch_max_freq_scale
		285	* scale = ------- * --------------------
		286	* /\const SCHED_CAPACITY_SCALE
		287	*
		288	* See validate_cpu_freq_invariance_counters() for details on
		289	* arch_max_freq_scale and the use of SCHED_CAPACITY_SHIFT.
		290	*/
		291	scale = core_cnt - prev_core_cnt;
		292	scale *= this_cpu_read(arch_max_freq_scale);
		293	scale = div64_u64(scale >> SCHED_CAPACITY_SHIFT,
		294	const_cnt - prev_const_cnt);
		295
		296	scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE);
		297	this_cpu_write(freq_scale, (unsigned long)scale);
		298
		299	store_and_exit:
		300	this_cpu_write(arch_core_cycles_prev, core_cnt);
		301	this_cpu_write(arch_const_cycles_prev, const_cnt);
		302	}
		303	#endif /* CONFIG_ARM64_AMU_EXTN */

arch/arm64/kernel/vdso/sigreturn.S+2−2

@@ -14,7 +14,7 @@ · 13 unmodified lines
14	.text	14	.text
15		15
16	nop	16	nop
17	ENTRY(__kernel_rt_sigreturn)	17	SYM_FUNC_START(__kernel_rt_sigreturn)
18	.cfi_startproc	18	.cfi_startproc
19	.cfi_signal_frame	19	.cfi_signal_frame
20	.cfi_def_cfa x29, 0	20	.cfi_def_cfa x29, 0
@@ -23,4 +23,4 @@ ENTRY(__kernel_rt_sigreturn) · 2 unmodified lines
23	mov x8, #__NR_rt_sigreturn	23	mov x8, #__NR_rt_sigreturn
24	svc #0	24	svc #0
25	.cfi_endproc	25	.cfi_endproc
26	ENDPROC(__kernel_rt_sigreturn)	26	SYM_FUNC_END(__kernel_rt_sigreturn)

arch/arm64/kernel/vdso32/sigreturn.S+8−15

 #include <asm/asm-offsets.h>
 #include <asm/unistd.h>
-#define ARM_ENTRY(name)		\
-	ENTRY(name)
-#define ARM_ENDPROC(name)	\
-	.type name, %function;	\
-	END(name)
 	.text
 	.arm
 	.save {r0-r15}
 	.pad #COMPAT_SIGFRAME_REGS_OFFSET
 	nop
-ARM_ENTRY(__kernel_sigreturn_arm)
+SYM_FUNC_START(__kernel_sigreturn_arm)
 	mov r7, #__NR_compat_sigreturn
 	svc #0
 	.fnend
-ARM_ENDPROC(__kernel_sigreturn_arm)
+SYM_FUNC_END(__kernel_sigreturn_arm)
 	.fnstart
 	.save {r0-r15}
 	.pad #COMPAT_RT_SIGFRAME_REGS_OFFSET
 	nop
-ARM_ENTRY(__kernel_rt_sigreturn_arm)
+SYM_FUNC_START(__kernel_rt_sigreturn_arm)
 	mov r7, #__NR_compat_rt_sigreturn
 	svc #0
 	.fnend
-ARM_ENDPROC(__kernel_rt_sigreturn_arm)
+SYM_FUNC_END(__kernel_rt_sigreturn_arm)
 	.thumb
 	.fnstart
 	.save {r0-r15}
 	.pad #COMPAT_SIGFRAME_REGS_OFFSET
 	nop
-ARM_ENTRY(__kernel_sigreturn_thumb)
+SYM_FUNC_START(__kernel_sigreturn_thumb)
 	mov r7, #__NR_compat_sigreturn
 	svc #0
 	.fnend
-ARM_ENDPROC(__kernel_sigreturn_thumb)
+SYM_FUNC_END(__kernel_sigreturn_thumb)
 	.fnstart
 	.save {r0-r15}
 	.pad #COMPAT_RT_SIGFRAME_REGS_OFFSET
 	nop
-ARM_ENTRY(__kernel_rt_sigreturn_thumb)
+SYM_FUNC_START(__kernel_rt_sigreturn_thumb)
 	mov r7, #__NR_compat_rt_sigreturn
 	svc #0
 	.fnend
-ARM_ENDPROC(__kernel_rt_sigreturn_thumb)
+SYM_FUNC_END(__kernel_rt_sigreturn_thumb)

arch/arm64/kvm/hyp-init.S+8−10

 	.align	11
-ENTRY(__kvm_hyp_init)
+SYM_CODE_START(__kvm_hyp_init)
 	ventry	__invalid		// Synchronous EL2t
 	ventry	__invalid		// IRQ EL2t
 	ventry	__invalid		// FIQ EL2t
 	msr	ttbr0_el2, x4
 	mrs	x4, tcr_el1
-	ldr	x5, =TCR_EL2_MASK
+	mov_q	x5, TCR_EL2_MASK
 	and	x4, x4, x5
 	mov	x5, #TCR_EL2_RES1
 	orr	x4, x4, x5
 	 * as well as the EE bit on BE. Drop the A flag since the compiler
 	 * is allowed to generate unaligned accesses.
 	 */
-	ldr	x4, =(SCTLR_EL2_RES1 | (SCTLR_ELx_FLAGS & ~SCTLR_ELx_A))
+	mov_q	x4, (SCTLR_EL2_RES1 | (SCTLR_ELx_FLAGS & ~SCTLR_ELx_A))
 CPU_BE(	orr	x4, x4, #SCTLR_ELx_EE)
 	msr	sctlr_el2, x4
 	isb
 	/* Hello, World! */
 	eret
-ENDPROC(__kvm_hyp_init)
+SYM_CODE_END(__kvm_hyp_init)
-ENTRY(__kvm_handle_stub_hvc)
+SYM_CODE_START(__kvm_handle_stub_hvc)
 	cmp	x0, #HVC_SOFT_RESTART
 	b.ne	1f
 	 * case we coming via HVC_SOFT_RESTART.
 	 */
 	mrs	x5, sctlr_el2
-	ldr	x6, =SCTLR_ELx_FLAGS
+	mov_q	x6, SCTLR_ELx_FLAGS
 	bic	x5, x5, x6		// Clear SCTL_M and etc
 	pre_disable_mmu_workaround
 	msr	sctlr_el2, x5
 	eret
 1:	/* Bad stub call */
-	ldr	x0, =HVC_STUB_ERR
+	mov_q	x0, HVC_STUB_ERR
 	eret
-ENDPROC(__kvm_handle_stub_hvc)
+SYM_CODE_END(__kvm_handle_stub_hvc)
-	.ltorg
 	.popsection

arch/arm64/kvm/hyp.S+2−2

@@ -28,7 +28,7 @@ · 27 unmodified lines
28	* and is used to implement hyp stubs in the same way as in	28	* and is used to implement hyp stubs in the same way as in
29	* arch/arm64/kernel/hyp_stub.S.	29	* arch/arm64/kernel/hyp_stub.S.
30	*/	30	*/
31	ENTRY(__kvm_call_hyp)	31	SYM_FUNC_START(__kvm_call_hyp)
32	hvc #0	32	hvc #0
33	ret	33	ret
34	ENDPROC(__kvm_call_hyp)	34	SYM_FUNC_END(__kvm_call_hyp)

arch/arm64/kvm/hyp/fpsimd.S+4−4

@@ -11,12 +11,12 @@ · 10 unmodified lines
11	.text	11	.text
12	.pushsection .hyp.text, "ax"	12	.pushsection .hyp.text, "ax"
13		13
14	ENTRY(__fpsimd_save_state)	14	SYM_FUNC_START(__fpsimd_save_state)
15	fpsimd_save x0, 1	15	fpsimd_save x0, 1
16	ret	16	ret
17	ENDPROC(__fpsimd_save_state)	17	SYM_FUNC_END(__fpsimd_save_state)
18		18
19	ENTRY(__fpsimd_restore_state)	19	SYM_FUNC_START(__fpsimd_restore_state)
20	fpsimd_restore x0, 1	20	fpsimd_restore x0, 1
21	ret	21	ret
22	ENDPROC(__fpsimd_restore_state)	22	SYM_FUNC_END(__fpsimd_restore_state)

arch/arm64/kvm/hyp/hyp-entry.S+16−11

 	eret
 	sb
-ENTRY(__hyp_do_panic)
+SYM_FUNC_START(__hyp_do_panic)
 	mov	lr, #(PSR_F_BIT | PSR_I_BIT | PSR_A_BIT | PSR_D_BIT |\
 		      PSR_MODE_EL1h)
 	msr	spsr_el2, lr
 	msr	elr_el2, lr
 	eret
 	sb
-ENDPROC(__hyp_do_panic)
+SYM_FUNC_END(__hyp_do_panic)
-ENTRY(__hyp_panic)
+SYM_CODE_START(__hyp_panic)
 	get_host_ctxt x0, x1
 	b	hyp_panic
-ENDPROC(__hyp_panic)
+SYM_CODE_END(__hyp_panic)
 .macro invalid_vector	label, target = __hyp_panic
 	.align	2
+SYM_CODE_START(\label)
 \label:
 	b \target
-ENDPROC(\label)
+SYM_CODE_END(\label)
 .endm
 	/* None of these should ever happen */
 check_preamble_length 661b, 662b
 .endm
-ENTRY(__kvm_hyp_vector)
+SYM_CODE_START(__kvm_hyp_vector)
 	invalid_vect	el2t_sync_invalid	// Synchronous EL2t
 	invalid_vect	el2t_irq_invalid	// IRQ EL2t
 	invalid_vect	el2t_fiq_invalid	// FIQ EL2t
 	valid_vect	el1_irq			// IRQ 32-bit EL1
 	invalid_vect	el1_fiq_invalid		// FIQ 32-bit EL1
 	valid_vect	el1_error		// Error 32-bit EL1
-ENDPROC(__kvm_hyp_vector)
+SYM_CODE_END(__kvm_hyp_vector)
 #ifdef CONFIG_KVM_INDIRECT_VECTORS
 .macro hyp_ventry
 .endm
 	.align	11
-ENTRY(__bp_harden_hyp_vecs_start)
+SYM_CODE_START(__bp_harden_hyp_vecs)
 	.rept BP_HARDEN_EL2_SLOTS
 	generate_vectors
 	.endr
-ENTRY(__bp_harden_hyp_vecs_end)
+1:	.org __bp_harden_hyp_vecs + __BP_HARDEN_HYP_VECS_SZ
+	.org 1b
+SYM_CODE_END(__bp_harden_hyp_vecs)
 	.popsection
-ENTRY(__smccc_workaround_1_smc_start)
+SYM_CODE_START(__smccc_workaround_1_smc)
 	esb
 	sub	sp, sp, #(8 * 4)
 	stp	x2, x3, [sp, #(8 * 0)]
 	ldp	x2, x3, [sp, #(8 * 0)]
 	ldp	x0, x1, [sp, #(8 * 2)]
 	add	sp, sp, #(8 * 4)
-ENTRY(__smccc_workaround_1_smc_end)
+1:	.org __smccc_workaround_1_smc + __SMCCC_WORKAROUND_1_SMC_SZ
+	.org 1b
+SYM_CODE_END(__smccc_workaround_1_smc)
 #endif

arch/arm64/kvm/hyp/switch.c+20−8

 	val = read_sysreg(cpacr_el1);
 	val |= CPACR_EL1_TTA;
 	val &= ~CPACR_EL1_ZEN;
+	/*
+	 * With VHE (HCR.E2H == 1), accesses to CPACR_EL1 are routed to
+	 * CPTR_EL2. In general, CPACR_EL1 has the same layout as CPTR_EL2,
+	 * except for some missing controls, such as TAM.
+	 * In this case, CPTR_EL2.TAM has the same position with or without
+	 * VHE (HCR.E2H == 1) which allows us to use here the CPTR_EL2.TAM
+	 * shift value for trapping the AMU accesses.
+	 */
+	val |= CPTR_EL2_TAM;
 	if (update_fp_enabled(vcpu)) {
 		if (vcpu_has_sve(vcpu))
 			val |= CPACR_EL1_ZEN;
 	__activate_traps_common(vcpu);
 	val = CPTR_EL2_DEFAULT;
-	val |= CPTR_EL2_TTA | CPTR_EL2_TZ;
+	val |= CPTR_EL2_TTA | CPTR_EL2_TZ | CPTR_EL2_TAM;
 	if (!update_fp_enabled(vcpu)) {
 		val |= CPTR_EL2_TFP;
 		__activate_traps_fpsimd32(vcpu);
 	write_sysreg(val, cptr_el2);
-	if (cpus_have_const_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE)) {
+	if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE)) {
 		struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt;
 		isb();
 {
 	u64 hcr = vcpu->arch.hcr_el2;
-	if (cpus_have_const_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM))
+	if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM))
 		hcr |= HCR_TVM;
 	write_sysreg(hcr, hcr_el2);
-	if (cpus_have_const_cap(ARM64_HAS_RAS_EXTN) && (hcr & HCR_VSE))
+	if (cpus_have_final_cap(ARM64_HAS_RAS_EXTN) && (hcr & HCR_VSE))
 		write_sysreg_s(vcpu->arch.vsesr_el2, SYS_VSESR_EL2);
 	if (has_vhe())
 {
 	u64 mdcr_el2 = read_sysreg(mdcr_el2);
-	if (cpus_have_const_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE)) {
+	if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE)) {
 		u64 val;
 		/*
 	 * resolve the IPA using the AT instruction.
 	 */
 	if (!(esr & ESR_ELx_S1PTW) &&
-	    (cpus_have_const_cap(ARM64_WORKAROUND_834220) ||
+	    (cpus_have_final_cap(ARM64_WORKAROUND_834220) ||
 	     (esr & ESR_ELx_FSC_TYPE) == FSC_PERM)) {
 		if (!__translate_far_to_hpfar(far, &hpfar))
 			return false;
 	if (*exit_code != ARM_EXCEPTION_TRAP)
 		goto exit;
-	if (cpus_have_const_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM) &&
+	if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM) &&
 	    kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_SYS64 &&
 	    handle_tx2_tvm(vcpu))
 		return true;
 static inline bool __hyp_text __needs_ssbd_off(struct kvm_vcpu *vcpu)
 {
-	if (!cpus_have_const_cap(ARM64_SSBD))
+	if (!cpus_have_final_cap(ARM64_SSBD))
 		return false;
 	return !(vcpu->arch.workaround_flags & VCPU_WORKAROUND_2_FLAG);

arch/arm64/kvm/hyp/sysreg-sr.c+4−4

arch/arm64/kvm/hyp/tlb.c+4−4

@@ -23,7 +23,7 @@ static void __hyp_text __tlb_switch_to_guest_vhe(struct kvm *kvm, · 22 unmodified lines
23		23
24	local_irq_save(cxt->flags);	24	local_irq_save(cxt->flags);
25		25
26	if (cpus_have_const_cap(ARM64_WORKAROUND_SPECULATIVE_AT_VHE)) {	26	if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT_VHE)) {
27	/*	27	/*
28	* For CPUs that are affected by ARM errata 1165522 or 1530923,	28	* For CPUs that are affected by ARM errata 1165522 or 1530923,
29	* we cannot trust stage-1 to be in a correct state at that	29	* we cannot trust stage-1 to be in a correct state at that
@@ -63,7 +63,7 @@ static void __hyp_text __tlb_switch_to_guest_vhe(struct kvm *kvm, · 33 unmodified lines
63	static void __hyp_text __tlb_switch_to_guest_nvhe(struct kvm *kvm,	63	static void __hyp_text __tlb_switch_to_guest_nvhe(struct kvm *kvm,
64	struct tlb_inv_context *cxt)	64	struct tlb_inv_context *cxt)
65	{	65	{
66	if (cpus_have_const_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE)) {	66	if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE)) {
67	u64 val;	67	u64 val;
68		68
69	/*	69	/*
@@ -103,7 +103,7 @@ static void __hyp_text __tlb_switch_to_host_vhe(struct kvm *kvm, · 33 unmodified lines
103	write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);	103	write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
104	isb();	104	isb();
105		105
106	if (cpus_have_const_cap(ARM64_WORKAROUND_SPECULATIVE_AT_VHE)) {	106	if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT_VHE)) {
107	/* Restore the registers to what they were */	107	/* Restore the registers to what they were */
108	write_sysreg_el1(cxt->tcr, SYS_TCR);	108	write_sysreg_el1(cxt->tcr, SYS_TCR);
109	write_sysreg_el1(cxt->sctlr, SYS_SCTLR);	109	write_sysreg_el1(cxt->sctlr, SYS_SCTLR);
@@ -117,7 +117,7 @@ static void __hyp_text __tlb_switch_to_host_nvhe(struct kvm *kvm, · 7 unmodified lines
117	{	117	{
118	write_sysreg(0, vttbr_el2);	118	write_sysreg(0, vttbr_el2);
119		119
120	if (cpus_have_const_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE)) {	120	if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE)) {
121	/* Ensure write of the host VMID */	121	/* Ensure write of the host VMID */
122	isb();	122	isb();
123	/* Restore the host's TCR_EL1 */	123	/* Restore the host's TCR_EL1 */

arch/arm64/kvm/sys_regs.c+101−2

@@ -1003,6 +1003,20 @@ static bool access_pmuserenr(struct kvm_vcpu vcpu, struct sys_reg_params p, · 1002 unmodified lines
1003	{ SYS_DESC(SYS_PMEVTYPERn_EL0(n)), \	1003	{ SYS_DESC(SYS_PMEVTYPERn_EL0(n)), \
1004	access_pmu_evtyper, reset_unknown, (PMEVTYPER0_EL0 + n), }	1004	access_pmu_evtyper, reset_unknown, (PMEVTYPER0_EL0 + n), }
1005		1005
		1006	static bool access_amu(struct kvm_vcpu vcpu, struct sys_reg_params p,
		1007	const struct sys_reg_desc *r)
		1008	{
		1009	kvm_inject_undefined(vcpu);
		1010
		1011	return false;
		1012	}
		1013
		1014	/* Macro to expand the AMU counter and type registers*/
		1015	#define AMU_AMEVCNTR0_EL0(n) { SYS_DESC(SYS_AMEVCNTR0_EL0(n)), access_amu }
		1016	#define AMU_AMEVTYPE0_EL0(n) { SYS_DESC(SYS_AMEVTYPE0_EL0(n)), access_amu }
		1017	#define AMU_AMEVCNTR1_EL0(n) { SYS_DESC(SYS_AMEVCNTR1_EL0(n)), access_amu }
		1018	#define AMU_AMEVTYPE1_EL0(n) { SYS_DESC(SYS_AMEVTYPE1_EL0(n)), access_amu }
		1019
1006	static bool trap_ptrauth(struct kvm_vcpu *vcpu,	1020	static bool trap_ptrauth(struct kvm_vcpu *vcpu,
1007	struct sys_reg_params *p,	1021	struct sys_reg_params *p,
1008	const struct sys_reg_desc *rd)	1022	const struct sys_reg_desc *rd)
@@ -1078,13 +1092,25 @@ static u64 read_id_reg(const struct kvm_vcpu *vcpu, · 69 unmodified lines
1078	(u32)r->CRn, (u32)r->CRm, (u32)r->Op2);	1092	(u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
1079	u64 val = raz ? 0 : read_sanitised_ftr_reg(id);	1093	u64 val = raz ? 0 : read_sanitised_ftr_reg(id);
1080		1094
1081	if (id == SYS_ID_AA64PFR0_EL1 && !vcpu_has_sve(vcpu)) {	1095	if (id == SYS_ID_AA64PFR0_EL1) {
1082	val &= ~(0xfUL << ID_AA64PFR0_SVE_SHIFT);	1096	if (!vcpu_has_sve(vcpu))
		1097	val &= ~(0xfUL << ID_AA64PFR0_SVE_SHIFT);
		1098	val &= ~(0xfUL << ID_AA64PFR0_AMU_SHIFT);
1083	} else if (id == SYS_ID_AA64ISAR1_EL1 && !vcpu_has_ptrauth(vcpu)) {	1099	} else if (id == SYS_ID_AA64ISAR1_EL1 && !vcpu_has_ptrauth(vcpu)) {
1084	val &= ~((0xfUL << ID_AA64ISAR1_APA_SHIFT) \|	1100	val &= ~((0xfUL << ID_AA64ISAR1_APA_SHIFT) \|
1085	(0xfUL << ID_AA64ISAR1_API_SHIFT) \|	1101	(0xfUL << ID_AA64ISAR1_API_SHIFT) \|
1086	(0xfUL << ID_AA64ISAR1_GPA_SHIFT) \|	1102	(0xfUL << ID_AA64ISAR1_GPA_SHIFT) \|
1087	(0xfUL << ID_AA64ISAR1_GPI_SHIFT));	1103	(0xfUL << ID_AA64ISAR1_GPI_SHIFT));
		1104	} else if (id == SYS_ID_AA64DFR0_EL1) {
		1105	/* Limit guests to PMUv3 for ARMv8.1 */
		1106	val = cpuid_feature_cap_perfmon_field(val,
		1107	ID_AA64DFR0_PMUVER_SHIFT,
		1108	ID_AA64DFR0_PMUVER_8_1);
		1109	} else if (id == SYS_ID_DFR0_EL1) {
		1110	/* Limit guests to PMUv3 for ARMv8.1 */
		1111	val = cpuid_feature_cap_perfmon_field(val,
		1112	ID_DFR0_PERFMON_SHIFT,
		1113	ID_DFR0_PERFMON_8_1);
1088	}	1114	}
1089		1115
1090	return val;	1116	return val;
@@ -1565,6 +1591,79 @@ static const struct sys_reg_desc sys_reg_descs[] = { · 474 unmodified lines
1565	{ SYS_DESC(SYS_TPIDR_EL0), NULL, reset_unknown, TPIDR_EL0 },	1591	{ SYS_DESC(SYS_TPIDR_EL0), NULL, reset_unknown, TPIDR_EL0 },
1566	{ SYS_DESC(SYS_TPIDRRO_EL0), NULL, reset_unknown, TPIDRRO_EL0 },	1592	{ SYS_DESC(SYS_TPIDRRO_EL0), NULL, reset_unknown, TPIDRRO_EL0 },
1567		1593
		1594	{ SYS_DESC(SYS_AMCR_EL0), access_amu },
		1595	{ SYS_DESC(SYS_AMCFGR_EL0), access_amu },
		1596	{ SYS_DESC(SYS_AMCGCR_EL0), access_amu },
		1597	{ SYS_DESC(SYS_AMUSERENR_EL0), access_amu },
		1598	{ SYS_DESC(SYS_AMCNTENCLR0_EL0), access_amu },
		1599	{ SYS_DESC(SYS_AMCNTENSET0_EL0), access_amu },
		1600	{ SYS_DESC(SYS_AMCNTENCLR1_EL0), access_amu },
		1601	{ SYS_DESC(SYS_AMCNTENSET1_EL0), access_amu },
		1602	AMU_AMEVCNTR0_EL0(0),
		1603	AMU_AMEVCNTR0_EL0(1),
		1604	AMU_AMEVCNTR0_EL0(2),
		1605	AMU_AMEVCNTR0_EL0(3),
		1606	AMU_AMEVCNTR0_EL0(4),
		1607	AMU_AMEVCNTR0_EL0(5),
		1608	AMU_AMEVCNTR0_EL0(6),
		1609	AMU_AMEVCNTR0_EL0(7),
		1610	AMU_AMEVCNTR0_EL0(8),
		1611	AMU_AMEVCNTR0_EL0(9),
		1612	AMU_AMEVCNTR0_EL0(10),
		1613	AMU_AMEVCNTR0_EL0(11),
		1614	AMU_AMEVCNTR0_EL0(12),
		1615	AMU_AMEVCNTR0_EL0(13),
		1616	AMU_AMEVCNTR0_EL0(14),
		1617	AMU_AMEVCNTR0_EL0(15),
		1618	AMU_AMEVTYPE0_EL0(0),
		1619	AMU_AMEVTYPE0_EL0(1),
		1620	AMU_AMEVTYPE0_EL0(2),
		1621	AMU_AMEVTYPE0_EL0(3),
		1622	AMU_AMEVTYPE0_EL0(4),
		1623	AMU_AMEVTYPE0_EL0(5),
		1624	AMU_AMEVTYPE0_EL0(6),
		1625	AMU_AMEVTYPE0_EL0(7),
		1626	AMU_AMEVTYPE0_EL0(8),
		1627	AMU_AMEVTYPE0_EL0(9),
		1628	AMU_AMEVTYPE0_EL0(10),
		1629	AMU_AMEVTYPE0_EL0(11),
		1630	AMU_AMEVTYPE0_EL0(12),
		1631	AMU_AMEVTYPE0_EL0(13),
		1632	AMU_AMEVTYPE0_EL0(14),
		1633	AMU_AMEVTYPE0_EL0(15),
		1634	AMU_AMEVCNTR1_EL0(0),
		1635	AMU_AMEVCNTR1_EL0(1),
		1636	AMU_AMEVCNTR1_EL0(2),
		1637	AMU_AMEVCNTR1_EL0(3),
		1638	AMU_AMEVCNTR1_EL0(4),
		1639	AMU_AMEVCNTR1_EL0(5),
		1640	AMU_AMEVCNTR1_EL0(6),
		1641	AMU_AMEVCNTR1_EL0(7),
		1642	AMU_AMEVCNTR1_EL0(8),
		1643	AMU_AMEVCNTR1_EL0(9),
		1644	AMU_AMEVCNTR1_EL0(10),
		1645	AMU_AMEVCNTR1_EL0(11),
		1646	AMU_AMEVCNTR1_EL0(12),
		1647	AMU_AMEVCNTR1_EL0(13),
		1648	AMU_AMEVCNTR1_EL0(14),
		1649	AMU_AMEVCNTR1_EL0(15),
		1650	AMU_AMEVTYPE1_EL0(0),
		1651	AMU_AMEVTYPE1_EL0(1),
		1652	AMU_AMEVTYPE1_EL0(2),
		1653	AMU_AMEVTYPE1_EL0(3),
		1654	AMU_AMEVTYPE1_EL0(4),
		1655	AMU_AMEVTYPE1_EL0(5),
		1656	AMU_AMEVTYPE1_EL0(6),
		1657	AMU_AMEVTYPE1_EL0(7),
		1658	AMU_AMEVTYPE1_EL0(8),
		1659	AMU_AMEVTYPE1_EL0(9),
		1660	AMU_AMEVTYPE1_EL0(10),
		1661	AMU_AMEVTYPE1_EL0(11),
		1662	AMU_AMEVTYPE1_EL0(12),
		1663	AMU_AMEVTYPE1_EL0(13),
		1664	AMU_AMEVTYPE1_EL0(14),
		1665	AMU_AMEVTYPE1_EL0(15),
		1666
1568	{ SYS_DESC(SYS_CNTP_TVAL_EL0), access_arch_timer },	1667	{ SYS_DESC(SYS_CNTP_TVAL_EL0), access_arch_timer },
1569	{ SYS_DESC(SYS_CNTP_CTL_EL0), access_arch_timer },	1668	{ SYS_DESC(SYS_CNTP_CTL_EL0), access_arch_timer },
1570	{ SYS_DESC(SYS_CNTP_CVAL_EL0), access_arch_timer },	1669	{ SYS_DESC(SYS_CNTP_CVAL_EL0), access_arch_timer },

arch/arm64/lib/csum.c+27−0

@@ -124,3 +124,30 @@ unsigned int do_csum(const unsigned char *buff, int len) · 123 unmodified lines
124		124
125	return sum >> 16;	125	return sum >> 16;
126	}	126	}
		127
		128	__sum16 csum_ipv6_magic(const struct in6_addr *saddr,
		129	const struct in6_addr *daddr,
		130	__u32 len, __u8 proto, __wsum csum)
		131	{
		132	__uint128_t src, dst;
		133	u64 sum = (__force u64)csum;
		134
		135	src = (const __uint128_t )saddr->s6_addr;
		136	dst = (const __uint128_t )daddr->s6_addr;
		137
		138	sum += (__force u32)htonl(len);
		139	#ifdef __LITTLE_ENDIAN
		140	sum += (u32)proto << 24;
		141	#else
		142	sum += proto;
		143	#endif
		144	src += (src >> 64) \| (src << 64);
		145	dst += (dst >> 64) \| (dst << 64);
		146
		147	sum = accumulate(sum, src >> 64);
		148	sum = accumulate(sum, dst >> 64);
		149
		150	sum += ((sum >> 32) \| (sum << 32));
		151	return csum_fold((__force __wsum)(sum >> 32));
		152	}
		153	EXPORT_SYMBOL(csum_ipv6_magic);

arch/arm64/lib/strcmp.S+1−1

@@ -186,7 +186,7 @@ CPU_LE( rev data2, data2 ) · 185 unmodified lines
186	* as carry-propagation can corrupt the upper bits if the trailing	186	* as carry-propagation can corrupt the upper bits if the trailing
187	* bytes in the string contain 0x01.	187	* bytes in the string contain 0x01.
188	* However, if there is no NUL byte in the dword, we can generate	188	* However, if there is no NUL byte in the dword, we can generate
189	* the result directly. We ca not just subtract the bytes as the	189	* the result directly. We cannot just subtract the bytes as the
190	* MSB might be significant.	190	* MSB might be significant.
191	*/	191	*/
192	CPU_BE( cbnz has_nul, 1f )	192	CPU_BE( cbnz has_nul, 1f )

arch/arm64/mm/context.c+30−2

@@ -6,6 +6,7 @@ · 5 unmodified lines
6	* Copyright (C) 2012 ARM Ltd.	6	* Copyright (C) 2012 ARM Ltd.
7	*/	7	*/
8		8
		9	#include <linux/bitfield.h>
9	#include <linux/bitops.h>	10	#include <linux/bitops.h>
10	#include <linux/sched.h>	11	#include <linux/sched.h>
11	#include <linux/slab.h>	12	#include <linux/slab.h>
@@ -254,10 +255,37 @@ switch_mm_fastpath: · 242 unmodified lines
254	/* Errata workaround post TTBRx_EL1 update. */	255	/* Errata workaround post TTBRx_EL1 update. */
255	asmlinkage void post_ttbr_update_workaround(void)	256	asmlinkage void post_ttbr_update_workaround(void)
256	{	257	{
		258	if (!IS_ENABLED(CONFIG_CAVIUM_ERRATUM_27456))
		259	return;
		260
257	asm(ALTERNATIVE("nop; nop; nop",	261	asm(ALTERNATIVE("nop; nop; nop",
258	"ic iallu; dsb nsh; isb",	262	"ic iallu; dsb nsh; isb",
259	ARM64_WORKAROUND_CAVIUM_27456,	263	ARM64_WORKAROUND_CAVIUM_27456));
260	CONFIG_CAVIUM_ERRATUM_27456));	264	}
		265
		266	void cpu_do_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm)
		267	{
		268	unsigned long ttbr1 = read_sysreg(ttbr1_el1);
		269	unsigned long asid = ASID(mm);
		270	unsigned long ttbr0 = phys_to_ttbr(pgd_phys);
		271
		272	/* Skip CNP for the reserved ASID */
		273	if (system_supports_cnp() && asid)
		274	ttbr0 \|= TTBR_CNP_BIT;
		275
		276	/* SW PAN needs a copy of the ASID in TTBR0 for entry */
		277	if (IS_ENABLED(CONFIG_ARM64_SW_TTBR0_PAN))
		278	ttbr0 \|= FIELD_PREP(TTBR_ASID_MASK, asid);
		279
		280	/* Set ASID in TTBR1 since TCR.A1 is set */
		281	ttbr1 &= ~TTBR_ASID_MASK;
		282	ttbr1 \|= FIELD_PREP(TTBR_ASID_MASK, asid);
		283
		284	write_sysreg(ttbr1, ttbr1_el1);
		285	isb();
		286	write_sysreg(ttbr0, ttbr0_el1);
		287	isb();
		288	post_ttbr_update_workaround();
261	}	289	}
262		290
263	static int asids_update_limit(void)	291	static int asids_update_limit(void)

arch/arm64/mm/mmu.c+370−9

@@ -17,6 +17,7 @@ · 16 unmodified lines
17	#include <linux/mman.h>	17	#include <linux/mman.h>
18	#include <linux/nodemask.h>	18	#include <linux/nodemask.h>
19	#include <linux/memblock.h>	19	#include <linux/memblock.h>
		20	#include <linux/memory.h>
20	#include <linux/fs.h>	21	#include <linux/fs.h>
21	#include <linux/io.h>	22	#include <linux/io.h>
22	#include <linux/mm.h>	23	#include <linux/mm.h>
@@ -724,6 +725,312 @@ int kern_addr_valid(unsigned long addr) · 701 unmodified lines
724		725
725	return pfn_valid(pte_pfn(pte));	726	return pfn_valid(pte_pfn(pte));
726	}	727	}
		728
		729	#ifdef CONFIG_MEMORY_HOTPLUG
		730	static void free_hotplug_page_range(struct page *page, size_t size)
		731	{
		732	WARN_ON(PageReserved(page));
		733	free_pages((unsigned long)page_address(page), get_order(size));
		734	}
		735
		736	static void free_hotplug_pgtable_page(struct page *page)
		737	{
		738	free_hotplug_page_range(page, PAGE_SIZE);
		739	}
		740
		741	static bool pgtable_range_aligned(unsigned long start, unsigned long end,
		742	unsigned long floor, unsigned long ceiling,
		743	unsigned long mask)
		744	{
		745	start &= mask;
		746	if (start < floor)
		747	return false;
		748
		749	if (ceiling) {
		750	ceiling &= mask;
		751	if (!ceiling)
		752	return false;
		753	}
		754
		755	if (end - 1 > ceiling - 1)
		756	return false;
		757	return true;
		758	}
		759
		760	static void unmap_hotplug_pte_range(pmd_t *pmdp, unsigned long addr,
		761	unsigned long end, bool free_mapped)
		762	{
		763	pte_t *ptep, pte;
		764
		765	do {
		766	ptep = pte_offset_kernel(pmdp, addr);
		767	pte = READ_ONCE(*ptep);
		768	if (pte_none(pte))
		769	continue;
		770
		771	WARN_ON(!pte_present(pte));
		772	pte_clear(&init_mm, addr, ptep);
		773	flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
		774	if (free_mapped)
		775	free_hotplug_page_range(pte_page(pte), PAGE_SIZE);
		776	} while (addr += PAGE_SIZE, addr < end);
		777	}
		778
		779	static void unmap_hotplug_pmd_range(pud_t *pudp, unsigned long addr,
		780	unsigned long end, bool free_mapped)
		781	{
		782	unsigned long next;
		783	pmd_t *pmdp, pmd;
		784
		785	do {
		786	next = pmd_addr_end(addr, end);
		787	pmdp = pmd_offset(pudp, addr);
		788	pmd = READ_ONCE(*pmdp);
		789	if (pmd_none(pmd))
		790	continue;
		791
		792	WARN_ON(!pmd_present(pmd));
		793	if (pmd_sect(pmd)) {
		794	pmd_clear(pmdp);
		795
		796	/*
		797	* One TLBI should be sufficient here as the PMD_SIZE
		798	* range is mapped with a single block entry.
		799	*/
		800	flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
		801	if (free_mapped)
		802	free_hotplug_page_range(pmd_page(pmd),
		803	PMD_SIZE);
		804	continue;
		805	}
		806	WARN_ON(!pmd_table(pmd));
		807	unmap_hotplug_pte_range(pmdp, addr, next, free_mapped);
		808	} while (addr = next, addr < end);
		809	}
		810
		811	static void unmap_hotplug_pud_range(p4d_t *p4dp, unsigned long addr,
		812	unsigned long end, bool free_mapped)
		813	{
		814	unsigned long next;
		815	pud_t *pudp, pud;
		816
		817	do {
		818	next = pud_addr_end(addr, end);
		819	pudp = pud_offset(p4dp, addr);
		820	pud = READ_ONCE(*pudp);
		821	if (pud_none(pud))
		822	continue;
		823
		824	WARN_ON(!pud_present(pud));
		825	if (pud_sect(pud)) {
		826	pud_clear(pudp);
		827
		828	/*
		829	* One TLBI should be sufficient here as the PUD_SIZE
		830	* range is mapped with a single block entry.
		831	*/
		832	flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
		833	if (free_mapped)
		834	free_hotplug_page_range(pud_page(pud),
		835	PUD_SIZE);
		836	continue;
		837	}
		838	WARN_ON(!pud_table(pud));
		839	unmap_hotplug_pmd_range(pudp, addr, next, free_mapped);
		840	} while (addr = next, addr < end);
		841	}
		842
		843	static void unmap_hotplug_p4d_range(pgd_t *pgdp, unsigned long addr,
		844	unsigned long end, bool free_mapped)
		845	{
		846	unsigned long next;
		847	p4d_t *p4dp, p4d;
		848
		849	do {
		850	next = p4d_addr_end(addr, end);
		851	p4dp = p4d_offset(pgdp, addr);
		852	p4d = READ_ONCE(*p4dp);
		853	if (p4d_none(p4d))
		854	continue;
		855
		856	WARN_ON(!p4d_present(p4d));
		857	unmap_hotplug_pud_range(p4dp, addr, next, free_mapped);
		858	} while (addr = next, addr < end);
		859	}
		860
		861	static void unmap_hotplug_range(unsigned long addr, unsigned long end,
		862	bool free_mapped)
		863	{
		864	unsigned long next;
		865	pgd_t *pgdp, pgd;
		866
		867	do {
		868	next = pgd_addr_end(addr, end);
		869	pgdp = pgd_offset_k(addr);
		870	pgd = READ_ONCE(*pgdp);
		871	if (pgd_none(pgd))
		872	continue;
		873
		874	WARN_ON(!pgd_present(pgd));
		875	unmap_hotplug_p4d_range(pgdp, addr, next, free_mapped);
		876	} while (addr = next, addr < end);
		877	}
		878
		879	static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr,
		880	unsigned long end, unsigned long floor,
		881	unsigned long ceiling)
		882	{
		883	pte_t *ptep, pte;
		884	unsigned long i, start = addr;
		885
		886	do {
		887	ptep = pte_offset_kernel(pmdp, addr);
		888	pte = READ_ONCE(*ptep);
		889
		890	/*
		891	* This is just a sanity check here which verifies that
		892	* pte clearing has been done by earlier unmap loops.
		893	*/
		894	WARN_ON(!pte_none(pte));
		895	} while (addr += PAGE_SIZE, addr < end);
		896
		897	if (!pgtable_range_aligned(start, end, floor, ceiling, PMD_MASK))
		898	return;
		899
		900	/*
		901	* Check whether we can free the pte page if the rest of the
		902	* entries are empty. Overlap with other regions have been
		903	* handled by the floor/ceiling check.
		904	*/
		905	ptep = pte_offset_kernel(pmdp, 0UL);
		906	for (i = 0; i < PTRS_PER_PTE; i++) {
		907	if (!pte_none(READ_ONCE(ptep[i])))
		908	return;
		909	}
		910
		911	pmd_clear(pmdp);
		912	__flush_tlb_kernel_pgtable(start);
		913	free_hotplug_pgtable_page(virt_to_page(ptep));
		914	}
		915
		916	static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
		917	unsigned long end, unsigned long floor,
		918	unsigned long ceiling)
		919	{
		920	pmd_t *pmdp, pmd;
		921	unsigned long i, next, start = addr;
		922
		923	do {
		924	next = pmd_addr_end(addr, end);
		925	pmdp = pmd_offset(pudp, addr);
		926	pmd = READ_ONCE(*pmdp);
		927	if (pmd_none(pmd))
		928	continue;
		929
		930	WARN_ON(!pmd_present(pmd) \|\| !pmd_table(pmd) \|\| pmd_sect(pmd));
		931	free_empty_pte_table(pmdp, addr, next, floor, ceiling);
		932	} while (addr = next, addr < end);
		933
		934	if (CONFIG_PGTABLE_LEVELS <= 2)
		935	return;
		936
		937	if (!pgtable_range_aligned(start, end, floor, ceiling, PUD_MASK))
		938	return;
		939
		940	/*
		941	* Check whether we can free the pmd page if the rest of the
		942	* entries are empty. Overlap with other regions have been
		943	* handled by the floor/ceiling check.
		944	*/
		945	pmdp = pmd_offset(pudp, 0UL);
		946	for (i = 0; i < PTRS_PER_PMD; i++) {
		947	if (!pmd_none(READ_ONCE(pmdp[i])))
		948	return;
		949	}
		950
		951	pud_clear(pudp);
		952	__flush_tlb_kernel_pgtable(start);
		953	free_hotplug_pgtable_page(virt_to_page(pmdp));
		954	}
		955
		956	static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr,
		957	unsigned long end, unsigned long floor,
		958	unsigned long ceiling)
		959	{
		960	pud_t *pudp, pud;
		961	unsigned long i, next, start = addr;
		962
		963	do {
		964	next = pud_addr_end(addr, end);
		965	pudp = pud_offset(p4dp, addr);
		966	pud = READ_ONCE(*pudp);
		967	if (pud_none(pud))
		968	continue;
		969
		970	WARN_ON(!pud_present(pud) \|\| !pud_table(pud) \|\| pud_sect(pud));
		971	free_empty_pmd_table(pudp, addr, next, floor, ceiling);
		972	} while (addr = next, addr < end);
		973
		974	if (CONFIG_PGTABLE_LEVELS <= 3)
		975	return;
		976
		977	if (!pgtable_range_aligned(start, end, floor, ceiling, PGDIR_MASK))
		978	return;
		979
		980	/*
		981	* Check whether we can free the pud page if the rest of the
		982	* entries are empty. Overlap with other regions have been
		983	* handled by the floor/ceiling check.
		984	*/
		985	pudp = pud_offset(p4dp, 0UL);
		986	for (i = 0; i < PTRS_PER_PUD; i++) {
		987	if (!pud_none(READ_ONCE(pudp[i])))
		988	return;
		989	}
		990
		991	p4d_clear(p4dp);
		992	__flush_tlb_kernel_pgtable(start);
		993	free_hotplug_pgtable_page(virt_to_page(pudp));
		994	}
		995
		996	static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr,
		997	unsigned long end, unsigned long floor,
		998	unsigned long ceiling)
		999	{
		1000	unsigned long next;
		1001	p4d_t *p4dp, p4d;
		1002
		1003	do {
		1004	next = p4d_addr_end(addr, end);
		1005	p4dp = p4d_offset(pgdp, addr);
		1006	p4d = READ_ONCE(*p4dp);
		1007	if (p4d_none(p4d))
		1008	continue;
		1009
		1010	WARN_ON(!p4d_present(p4d));
		1011	free_empty_pud_table(p4dp, addr, next, floor, ceiling);
		1012	} while (addr = next, addr < end);
		1013	}
		1014
		1015	static void free_empty_tables(unsigned long addr, unsigned long end,
		1016	unsigned long floor, unsigned long ceiling)
		1017	{
		1018	unsigned long next;
		1019	pgd_t *pgdp, pgd;
		1020
		1021	do {
		1022	next = pgd_addr_end(addr, end);
		1023	pgdp = pgd_offset_k(addr);
		1024	pgd = READ_ONCE(*pgdp);
		1025	if (pgd_none(pgd))
		1026	continue;
		1027
		1028	WARN_ON(!pgd_present(pgd));
		1029	free_empty_p4d_table(pgdp, addr, next, floor, ceiling);
		1030	} while (addr = next, addr < end);
		1031	}
		1032	#endif
		1033
727	#ifdef CONFIG_SPARSEMEM_VMEMMAP	1034	#ifdef CONFIG_SPARSEMEM_VMEMMAP
728	#if !ARM64_SWAPPER_USES_SECTION_MAPS	1035	#if !ARM64_SWAPPER_USES_SECTION_MAPS
729	int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,	1036	int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
@@ -771,6 +1078,12 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, · 41 unmodified lines
771	void vmemmap_free(unsigned long start, unsigned long end,	1078	void vmemmap_free(unsigned long start, unsigned long end,
772	struct vmem_altmap *altmap)	1079	struct vmem_altmap *altmap)
773	{	1080	{
		1081	#ifdef CONFIG_MEMORY_HOTPLUG
		1082	WARN_ON((start < VMEMMAP_START) \|\| (end > VMEMMAP_END));
		1083
		1084	unmap_hotplug_range(start, end, true);
		1085	free_empty_tables(start, end, VMEMMAP_START, VMEMMAP_END);
		1086	#endif
774	}	1087	}
775	#endif /* CONFIG_SPARSEMEM_VMEMMAP */	1088	#endif /* CONFIG_SPARSEMEM_VMEMMAP */
776		1089
@@ -1049,10 +1362,21 @@ int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) · 272 unmodified lines
1049	}	1362	}
1050		1363
1051	#ifdef CONFIG_MEMORY_HOTPLUG	1364	#ifdef CONFIG_MEMORY_HOTPLUG
		1365	static void __remove_pgd_mapping(pgd_t *pgdir, unsigned long start, u64 size)
		1366	{
		1367	unsigned long end = start + size;
		1368
		1369	WARN_ON(pgdir != init_mm.pgd);
		1370	WARN_ON((start < PAGE_OFFSET) \|\| (end > PAGE_END));
		1371
		1372	unmap_hotplug_range(start, end, false);
		1373	free_empty_tables(start, end, PAGE_OFFSET, PAGE_END);
		1374	}
		1375
1052	int arch_add_memory(int nid, u64 start, u64 size,	1376	int arch_add_memory(int nid, u64 start, u64 size,
1053	struct mhp_restrictions *restrictions)	1377	struct mhp_restrictions *restrictions)
1054	{	1378	{
1055	int flags = 0;	1379	int ret, flags = 0;
1056		1380
1057	if (rodata_full \|\| debug_pagealloc_enabled())	1381	if (rodata_full \|\| debug_pagealloc_enabled())
1058	flags = NO_BLOCK_MAPPINGS \| NO_CONT_MAPPINGS;	1382	flags = NO_BLOCK_MAPPINGS \| NO_CONT_MAPPINGS;
@@ -1062,22 +1386,59 @@ int arch_add_memory(int nid, u64 start, u64 size, · 3 unmodified lines
1062		1386
1063	memblock_clear_nomap(start, size);	1387	memblock_clear_nomap(start, size);
1064		1388
1065	return __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT,	1389	ret = __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT,
1066	restrictions);	1390	restrictions);
		1391	if (ret)
		1392	__remove_pgd_mapping(swapper_pg_dir,
		1393	__phys_to_virt(start), size);
		1394	return ret;
1067	}	1395	}
		1396
1068	void arch_remove_memory(int nid, u64 start, u64 size,	1397	void arch_remove_memory(int nid, u64 start, u64 size,
1069	struct vmem_altmap *altmap)	1398	struct vmem_altmap *altmap)
1070	{	1399	{
1071	unsigned long start_pfn = start >> PAGE_SHIFT;	1400	unsigned long start_pfn = start >> PAGE_SHIFT;
1072	unsigned long nr_pages = size >> PAGE_SHIFT;	1401	unsigned long nr_pages = size >> PAGE_SHIFT;
1073		1402
1074	/*
1075	* FIXME: Cleanup page tables (also in arch_add_memory() in case
1076	* adding fails). Until then, this function should only be used
1077	* during memory hotplug (adding memory), not for memory
1078	* unplug. ARCH_ENABLE_MEMORY_HOTREMOVE must not be
1079	* unlocked yet.
1080	*/
1081	__remove_pages(start_pfn, nr_pages, altmap);	1403	__remove_pages(start_pfn, nr_pages, altmap);
		1404	__remove_pgd_mapping(swapper_pg_dir, __phys_to_virt(start), size);
		1405	}
		1406
		1407	/*
		1408	* This memory hotplug notifier helps prevent boot memory from being
		1409	* inadvertently removed as it blocks pfn range offlining process in
		1410	* __offline_pages(). Hence this prevents both offlining as well as
		1411	* removal process for boot memory which is initially always online.
		1412	* In future if and when boot memory could be removed, this notifier
		1413	* should be dropped and free_hotplug_page_range() should handle any
		1414	* reserved pages allocated during boot.
		1415	*/
		1416	static int prevent_bootmem_remove_notifier(struct notifier_block *nb,
		1417	unsigned long action, void *data)
		1418	{
		1419	struct mem_section *ms;
		1420	struct memory_notify *arg = data;
		1421	unsigned long end_pfn = arg->start_pfn + arg->nr_pages;
		1422	unsigned long pfn = arg->start_pfn;
		1423
		1424	if (action != MEM_GOING_OFFLINE)
		1425	return NOTIFY_OK;
		1426
		1427	for (; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
		1428	ms = __pfn_to_section(pfn);
		1429	if (early_section(ms))
		1430	return NOTIFY_BAD;
		1431	}
		1432	return NOTIFY_OK;
		1433	}
		1434
		1435	static struct notifier_block prevent_bootmem_remove_nb = {
		1436	.notifier_call = prevent_bootmem_remove_notifier,
		1437	};
		1438
		1439	static int __init prevent_bootmem_remove_init(void)
		1440	{
		1441	return register_memory_notifier(&prevent_bootmem_remove_nb);
1082	}	1442	}
		1443	device_initcall(prevent_bootmem_remove_init);
1083	#endif	1444	#endif

arch/arm64/mm/proc.S+64−40

 #include <linux/linkage.h>
 #include <asm/assembler.h>
 #include <asm/asm-offsets.h>
+#include <asm/asm_pointer_auth.h>
 #include <asm/hwcap.h>
 #include <asm/pgtable.h>
 #include <asm/pgtable-hwdef.h>
 #include <asm/cpufeature.h>
 #include <asm/alternative.h>
+#include <asm/smp.h>
 #ifdef CONFIG_ARM64_64K_PAGES
 #define TCR_TG_FLAGS	TCR_TG0_64K | TCR_TG1_64K
 	ubfx	x11, x11, #1, #1
 	msr	oslar_el1, x11
 	reset_pmuserenr_el0 x0			// Disable PMU access from EL0
+	reset_amuserenr_el0 x0			// Disable AMU access from EL0
 alternative_if ARM64_HAS_RAS_EXTN
 	msr_s	SYS_DISR_EL1, xzr
 alternative_else_nop_endif
+	ptrauth_keys_install_kernel x14, 0, x1, x2, x3
 	isb
 	ret
 SYM_FUNC_END(cpu_do_resume)
 	.popsection
 #endif
-/*
- *	cpu_do_switch_mm(pgd_phys, tsk)
- *
- *	Set the translation table base pointer to be pgd_phys.
- *
- *	- pgd_phys - physical address of new TTB
- */
-SYM_FUNC_START(cpu_do_switch_mm)
-	mrs	x2, ttbr1_el1
-	mmid	x1, x1				// get mm->context.id
-	phys_to_ttbr x3, x0
-alternative_if ARM64_HAS_CNP
-	cbz     x1, 1f                          // skip CNP for reserved ASID
-	orr     x3, x3, #TTBR_CNP_BIT
-1:
-alternative_else_nop_endif
-#ifdef CONFIG_ARM64_SW_TTBR0_PAN
-	bfi	x3, x1, #48, #16		// set the ASID field in TTBR0
-#endif
-	bfi	x2, x1, #48, #16		// set the ASID
-	msr	ttbr1_el1, x2			// in TTBR1 (since TCR.A1 is set)
-	isb
-	msr	ttbr0_el1, x3			// now update TTBR0
-	isb
-	b	post_ttbr_update_workaround	// Back to C code...
-SYM_FUNC_END(cpu_do_switch_mm)
 	.pushsection ".idmap.text", "awx"
 .macro	__idmap_cpu_set_reserved_ttbr1, tmp1, tmp2
 /*
  *	__cpu_setup
  *
- *	Initialise the processor for turning the MMU on.  Return in x0 the
+ *	Initialise the processor for turning the MMU on.
- *	value of the SCTLR_EL1 register.
+ *
+ * Input:
+ *	x0 with a flag ARM64_CPU_BOOT_PRIMARY/ARM64_CPU_BOOT_SECONDARY/ARM64_CPU_RUNTIME.
+ * Output:
+ *	Return in x0 the value of the SCTLR_EL1 register.
  */
 	.pushsection ".idmap.text", "awx"
 SYM_FUNC_START(__cpu_setup)
 	tlbi	vmalle1				// Invalidate local TLB
 	dsb	nsh
-	mov	x0, #3 << 20
+	mov	x1, #3 << 20
-	msr	cpacr_el1, x0			// Enable FP/ASIMD
+	msr	cpacr_el1, x1			// Enable FP/ASIMD
-	mov	x0, #1 << 12			// Reset mdscr_el1 and disable
+	mov	x1, #1 << 12			// Reset mdscr_el1 and disable
-	msr	mdscr_el1, x0			// access to the DCC from EL0
+	msr	mdscr_el1, x1			// access to the DCC from EL0
 	isb					// Unmask debug exceptions now,
 	enable_dbg				// since this is per-cpu
-	reset_pmuserenr_el0 x0			// Disable PMU access from EL0
+	reset_pmuserenr_el0 x1			// Disable PMU access from EL0
+	reset_amuserenr_el0 x1			// Disable AMU access from EL0
 	/*
 	 * Memory region attributes
 	 */
 	mov_q	x5, MAIR_EL1_SET
 	msr	mair_el1, x5
-	/*
-	 * Prepare SCTLR
-	 */
-	mov_q	x0, SCTLR_EL1_SET
 	/*
 	 * Set/prepare TCR and TTBR. We use 512GB (39-bit) address range for
 	 * both user and kernel.
 	 */
-	ldr	x10, =TCR_TxSZ(VA_BITS) | TCR_CACHE_FLAGS | TCR_SMP_FLAGS | \
+	mov_q	x10, TCR_TxSZ(VA_BITS) | TCR_CACHE_FLAGS | TCR_SMP_FLAGS | \
 			TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \
 			TCR_TBI0 | TCR_A1 | TCR_KASAN_FLAGS
 	tcr_clear_errata_bits x10, x9, x5
 1:
 #endif	/* CONFIG_ARM64_HW_AFDBM */
 	msr	tcr_el1, x10
+	mov	x1, x0
+	/*
+	 * Prepare SCTLR
+	 */
+	mov_q	x0, SCTLR_EL1_SET
+#ifdef CONFIG_ARM64_PTR_AUTH
+	/* No ptrauth setup for run time cpus */
+	cmp	x1, #ARM64_CPU_RUNTIME
+	b.eq	3f
+	/* Check if the CPU supports ptrauth */
+	mrs	x2, id_aa64isar1_el1
+	ubfx	x2, x2, #ID_AA64ISAR1_APA_SHIFT, #8
+	cbz	x2, 3f
+	/*
+	 * The primary cpu keys are reset here and can be
+	 * re-initialised with some proper values later.
+	 */
+	msr_s	SYS_APIAKEYLO_EL1, xzr
+	msr_s	SYS_APIAKEYHI_EL1, xzr
+	/* Just enable ptrauth for primary cpu */
+	cmp	x1, #ARM64_CPU_BOOT_PRIMARY
+	b.eq	2f
+	/* if !system_supports_address_auth() then skip enable */
+alternative_if_not ARM64_HAS_ADDRESS_AUTH
+	b	3f
+alternative_else_nop_endif
+	/* Install ptrauth key for secondary cpus */
+	adr_l	x2, secondary_data
+	ldr	x3, [x2, #CPU_BOOT_TASK]	// get secondary_data.task
+	cbz	x3, 2f				// check for slow booting cpus
+	ldp	x3, x4, [x2, #CPU_BOOT_PTRAUTH_KEY]
+	msr_s	SYS_APIAKEYLO_EL1, x3
+	msr_s	SYS_APIAKEYHI_EL1, x4
+2:	/* Enable ptrauth instructions */
+	ldr	x2, =SCTLR_ELx_ENIA | SCTLR_ELx_ENIB | \
+		     SCTLR_ELx_ENDA | SCTLR_ELx_ENDB
+	orr	x0, x0, x2
+3:
+#endif
 	ret					// return to head.S
 SYM_FUNC_END(__cpu_setup)

arch/arm64/mm/ptdump_debugfs.c+4−0

 // SPDX-License-Identifier: GPL-2.0
 #include <linux/debugfs.h>
+#include <linux/memory_hotplug.h>
 #include <linux/seq_file.h>
 #include <asm/ptdump.h>
 static int ptdump_show(struct seq_file *m, void *v)
 {
 	struct ptdump_info *info = m->private;
+	get_online_mems();
 	ptdump_walk(m, info);
+	put_online_mems();
 	return 0;
 }
 DEFINE_SHOW_ATTRIBUTE(ptdump);

drivers/base/arch_topology.c+12−0

 #include <linux/sched.h>
 #include <linux/smp.h>
+__weak bool arch_freq_counters_available(struct cpumask *cpus)
+{
+	return false;
+}
 DEFINE_PER_CPU(unsigned long, freq_scale) = SCHED_CAPACITY_SCALE;
 void arch_set_freq_scale(struct cpumask *cpus, unsigned long cur_freq,
 	unsigned long scale;
 	int i;
+	/*
+	 * If the use of counters for FIE is enabled, just return as we don't
+	 * want to update the scale factor with information from CPUFREQ.
+	 * Instead the scale factor will be updated from arch_scale_freq_tick.
+	 */
+	if (arch_freq_counters_available(cpus))
+		return;
 	scale = (cur_freq << SCHED_CAPACITY_SHIFT) / max_freq;
 	for_each_cpu(i, cpus)

drivers/clocksource/arm_arch_timer.c+15−3

 	return 0;
 }
+static int validate_timer_rate(void)
+{
+	if (!arch_timer_rate)
+		return -EINVAL;
+	/* Arch timer frequency < 1MHz can cause trouble */
+	WARN_ON(arch_timer_rate < 1000000);
+	return 0;
+}
 /*
  * For historical reasons, when probing with DT we use whichever (non-zero)
  * rate was probed first, and don't verify that others match. If the first node
 		arch_timer_rate = rate;
 	/* Check the timer frequency. */
-	if (arch_timer_rate == 0)
+	if (validate_timer_rate())
 		pr_warn("frequency not available\n");
 }
 	 * CNTFRQ value. This *must* be correct.
 	 */
 	arch_timer_rate = arch_timer_get_cntfrq();
-	if (!arch_timer_rate) {
+	ret = validate_timer_rate();
+	if (ret) {
 		pr_err(FW_BUG "frequency not available.\n");
-		return -EINVAL;
+		return ret;
 	}
 	arch_timer_uses_ppi = arch_timer_select_ppi();

drivers/cpufreq/cpufreq.c+20−0

@@ -1733,6 +1733,26 @@ unsigned int cpufreq_quick_get_max(unsigned int cpu) · 1732 unmodified lines
1733	}	1733	}
1734	EXPORT_SYMBOL(cpufreq_quick_get_max);	1734	EXPORT_SYMBOL(cpufreq_quick_get_max);
1735		1735
		1736	/**
		1737	* cpufreq_get_hw_max_freq - get the max hardware frequency of the CPU
		1738	* @cpu: CPU number
		1739	*
		1740	* The default return value is the max_freq field of cpuinfo.
		1741	*/
		1742	__weak unsigned int cpufreq_get_hw_max_freq(unsigned int cpu)
		1743	{
		1744	struct cpufreq_policy *policy = cpufreq_cpu_get(cpu);
		1745	unsigned int ret_freq = 0;
		1746
		1747	if (policy) {
		1748	ret_freq = policy->cpuinfo.max_freq;
		1749	cpufreq_cpu_put(policy);
		1750	}
		1751
		1752	return ret_freq;
		1753	}
		1754	EXPORT_SYMBOL(cpufreq_get_hw_max_freq);
		1755
1736	static unsigned int __cpufreq_get(struct cpufreq_policy *policy)	1756	static unsigned int __cpufreq_get(struct cpufreq_policy *policy)
1737	{	1757	{
1738	if (unlikely(policy_is_inactive(policy)))	1758	if (unlikely(policy_is_inactive(policy)))

drivers/firmware/arm_sdei.c+36−35

 		event->private_registered = regs;
 	}
-	if (sdei_event_find(event_num)) {
+	spin_lock(&sdei_list_lock);
-		kfree(event->registered);
+	list_add(&event->list, &sdei_list);
-		kfree(event);
+	spin_unlock(&sdei_list_lock);
-		event = ERR_PTR(-EBUSY);
-	} else {
-		spin_lock(&sdei_list_lock);
-		list_add(&event->list, &sdei_list);
-		spin_unlock(&sdei_list_lock);
-	}
 	return event;
 }
-static void sdei_event_destroy(struct sdei_event *event)
+static void sdei_event_destroy_llocked(struct sdei_event *event)
 {
 	lockdep_assert_held(&sdei_events_lock);
+	lockdep_assert_held(&sdei_list_lock);
-	spin_lock(&sdei_list_lock);
 	list_del(&event->list);
-	spin_unlock(&sdei_list_lock);
 	if (event->type == SDEI_EVENT_TYPE_SHARED)
 		kfree(event->registered);
 	kfree(event);
 }
+static void sdei_event_destroy(struct sdei_event *event)
+{
+	spin_lock(&sdei_list_lock);
+	sdei_event_destroy_llocked(event);
+	spin_unlock(&sdei_list_lock);
+}
 static int sdei_api_get_version(u64 *version)
 {
 	return invoke_sdei_fn(SDEI_1_0_FN_SDEI_VERSION, 0, 0, 0, 0, 0, version);
 		return -ENOENT;
 	}
-	spin_lock(&sdei_list_lock);
-	event->reenable = true;
-	spin_unlock(&sdei_list_lock);
+	cpus_read_lock();
 	if (event->type == SDEI_EVENT_TYPE_SHARED)
 		err = sdei_api_event_enable(event->event_num);
 	else
 		err = sdei_do_cross_call(_local_event_enable, event);
+	if (!err) {
+		spin_lock(&sdei_list_lock);
+		event->reenable = true;
+		spin_unlock(&sdei_list_lock);
+	}
+	cpus_read_unlock();
 	mutex_unlock(&sdei_events_lock);
 	return err;
 {
 	lockdep_assert_held(&sdei_events_lock);
-	spin_lock(&sdei_list_lock);
-	event->reregister = false;
-	event->reenable = false;
-	spin_unlock(&sdei_list_lock);
 	if (event->type == SDEI_EVENT_TYPE_SHARED)
 		return sdei_api_event_unregister(event->event_num);
 			break;
 		}
+		spin_lock(&sdei_list_lock);
+		event->reregister = false;
+		event->reenable = false;
+		spin_unlock(&sdei_list_lock);
 		err = _sdei_event_unregister(event);
 		if (err)
 			break;
 	lockdep_assert_held(&sdei_events_lock);
-	spin_lock(&sdei_list_lock);
-	event->reregister = true;
-	spin_unlock(&sdei_list_lock);
 	if (event->type == SDEI_EVENT_TYPE_SHARED)
 		return sdei_api_event_register(event->event_num,
 					       sdei_entry_point,
 					       event->registered,
 					       SDEI_EVENT_REGISTER_RM_ANY, 0);
 	err = sdei_do_cross_call(_local_event_register, event);
-	if (err) {
+	if (err)
-		spin_lock(&sdei_list_lock);
-		event->reregister = false;
-		event->reenable = false;
-		spin_unlock(&sdei_list_lock);
 		sdei_do_cross_call(_local_event_unregister, event);
-	}
 	return err;
 }
 			break;
 		}
+		cpus_read_lock();
 		err = _sdei_event_register(event);
 		if (err) {
 			sdei_event_destroy(event);
 			pr_warn("Failed to register event %u: %d\n", event_num,
 				err);
+		} else {
+			spin_lock(&sdei_list_lock);
+			event->reregister = true;
+			spin_unlock(&sdei_list_lock);
 		}
+		cpus_read_unlock();
 	} while (0);
 	mutex_unlock(&sdei_events_lock);
 }
 EXPORT_SYMBOL(sdei_event_register);
-static int sdei_reregister_event(struct sdei_event *event)
+static int sdei_reregister_event_llocked(struct sdei_event *event)
 {
 	int err;
 	lockdep_assert_held(&sdei_events_lock);
+	lockdep_assert_held(&sdei_list_lock);
 	err = _sdei_event_register(event);
 	if (err) {
 		pr_err("Failed to re-register event %u\n", event->event_num);
-		sdei_event_destroy(event);
+		sdei_event_destroy_llocked(event);
 		return err;
 	}
 			continue;
 		if (event->reregister) {
-			err = sdei_reregister_event(event);
+			err = sdei_reregister_event_llocked(event);
 			if (err)
 				break;
 		}

drivers/misc/lkdtm/bugs.c+36−0

@@ -378,3 +378,39 @@ void lkdtm_DOUBLE_FAULT(void) · 377 unmodified lines
378	pr_err("XFAIL: this test is ia32-only\n");	378	pr_err("XFAIL: this test is ia32-only\n");
379	#endif	379	#endif
380	}	380	}
		381
		382	#ifdef CONFIG_ARM64_PTR_AUTH
		383	static noinline void change_pac_parameters(void)
		384	{
		385	/* Reset the keys of current task */
		386	ptrauth_thread_init_kernel(current);
		387	ptrauth_thread_switch_kernel(current);
		388	}
		389
		390	#define CORRUPT_PAC_ITERATE 10
		391	noinline void lkdtm_CORRUPT_PAC(void)
		392	{
		393	int i;
		394
		395	if (!system_supports_address_auth()) {
		396	pr_err("FAIL: arm64 pointer authentication feature not present\n");
		397	return;
		398	}
		399
		400	pr_info("Change the PAC parameters to force function return failure\n");
		401	/*
		402	* Pac is a hash value computed from input keys, return address and
		403	* stack pointer. As pac has fewer bits so there is a chance of
		404	* collision, so iterate few times to reduce the collision probability.
		405	*/
		406	for (i = 0; i < CORRUPT_PAC_ITERATE; i++)
		407	change_pac_parameters();
		408
		409	pr_err("FAIL: %s test failed. Kernel may be unstable from here\n", __func__);
		410	}
		411	#else /* !CONFIG_ARM64_PTR_AUTH */
		412	noinline void lkdtm_CORRUPT_PAC(void)
		413	{
		414	pr_err("FAIL: arm64 pointer authentication config disabled\n");
		415	}
		416	#endif

drivers/misc/lkdtm/core.c+1−0

@@ -116,6 +116,7 @@ static const struct crashtype crashtypes[] = { · 115 unmodified lines
116	CRASHTYPE(STACK_GUARD_PAGE_LEADING),	116	CRASHTYPE(STACK_GUARD_PAGE_LEADING),
117	CRASHTYPE(STACK_GUARD_PAGE_TRAILING),	117	CRASHTYPE(STACK_GUARD_PAGE_TRAILING),
118	CRASHTYPE(UNSET_SMEP),	118	CRASHTYPE(UNSET_SMEP),
		119	CRASHTYPE(CORRUPT_PAC),
119	CRASHTYPE(UNALIGNED_LOAD_STORE_WRITE),	120	CRASHTYPE(UNALIGNED_LOAD_STORE_WRITE),
120	CRASHTYPE(OVERWRITE_ALLOCATION),	121	CRASHTYPE(OVERWRITE_ALLOCATION),
121	CRASHTYPE(WRITE_AFTER_FREE),	122	CRASHTYPE(WRITE_AFTER_FREE),

drivers/misc/lkdtm/lkdtm.h+1−0

@@ -31,6 +31,7 @@ void lkdtm_UNSET_SMEP(void); · 30 unmodified lines
31	#ifdef CONFIG_X86_32	31	#ifdef CONFIG_X86_32
32	void lkdtm_DOUBLE_FAULT(void);	32	void lkdtm_DOUBLE_FAULT(void);
33	#endif	33	#endif
		34	void lkdtm_CORRUPT_PAC(void);
34		35
35	/* lkdtm_heap.c */	36	/* lkdtm_heap.c */
36	void __init lkdtm_heap_init(void);	37	void __init lkdtm_heap_init(void);

drivers/perf/arm-ccn.c+10−10

@@ -328,15 +328,15 @@ static ssize_t arm_ccn_pmu_event_show(struct device *dev, · 327 unmodified lines
328	struct arm_ccn_pmu_event, attr);	328	struct arm_ccn_pmu_event, attr);
329	ssize_t res;	329	ssize_t res;
330		330
331	res = snprintf(buf, PAGE_SIZE, "type=0x%x", event->type);	331	res = scnprintf(buf, PAGE_SIZE, "type=0x%x", event->type);
332	if (event->event)	332	if (event->event)
333	res += snprintf(buf + res, PAGE_SIZE - res, ",event=0x%x",	333	res += scnprintf(buf + res, PAGE_SIZE - res, ",event=0x%x",
334	event->event);	334	event->event);
335	if (event->def)	335	if (event->def)
336	res += snprintf(buf + res, PAGE_SIZE - res, ",%s",	336	res += scnprintf(buf + res, PAGE_SIZE - res, ",%s",
337	event->def);	337	event->def);
338	if (event->mask)	338	if (event->mask)
339	res += snprintf(buf + res, PAGE_SIZE - res, ",mask=0x%x",	339	res += scnprintf(buf + res, PAGE_SIZE - res, ",mask=0x%x",
340	event->mask);	340	event->mask);
341		341
342	/* Arguments required by an event */	342	/* Arguments required by an event */
@@ -344,25 +344,25 @@ static ssize_t arm_ccn_pmu_event_show(struct device *dev, · 1 unmodified line
344	case CCN_TYPE_CYCLES:	344	case CCN_TYPE_CYCLES:
345	break;	345	break;
346	case CCN_TYPE_XP:	346	case CCN_TYPE_XP:
347	res += snprintf(buf + res, PAGE_SIZE - res,	347	res += scnprintf(buf + res, PAGE_SIZE - res,
348	",xp=?,vc=?");	348	",xp=?,vc=?");
349	if (event->event == CCN_EVENT_WATCHPOINT)	349	if (event->event == CCN_EVENT_WATCHPOINT)
350	res += snprintf(buf + res, PAGE_SIZE - res,	350	res += scnprintf(buf + res, PAGE_SIZE - res,
351	",port=?,dir=?,cmp_l=?,cmp_h=?,mask=?");	351	",port=?,dir=?,cmp_l=?,cmp_h=?,mask=?");
352	else	352	else
353	res += snprintf(buf + res, PAGE_SIZE - res,	353	res += scnprintf(buf + res, PAGE_SIZE - res,
354	",bus=?");	354	",bus=?");
355		355
356	break;	356	break;
357	case CCN_TYPE_MN:	357	case CCN_TYPE_MN:
358	res += snprintf(buf + res, PAGE_SIZE - res, ",node=%d", ccn->mn_id);	358	res += scnprintf(buf + res, PAGE_SIZE - res, ",node=%d", ccn->mn_id);
359	break;	359	break;
360	default:	360	default:
361	res += snprintf(buf + res, PAGE_SIZE - res, ",node=?");	361	res += scnprintf(buf + res, PAGE_SIZE - res, ",node=?");
362	break;	362	break;
363	}	363	}
364		364
365	res += snprintf(buf + res, PAGE_SIZE - res, "\n");	365	res += scnprintf(buf + res, PAGE_SIZE - res, "\n");
366		366
367	return res;	367	return res;
368	}	368	}

drivers/perf/arm_spe_pmu.c+1−1

@@ -831,7 +831,7 @@ static void arm_spe_pmu_setup_aux(struct perf_event event, void **pages, · 830 unmodified lines
831	* parts and give userspace a fighting chance of getting some	831	* parts and give userspace a fighting chance of getting some
832	* useful data out of it.	832	* useful data out of it.
833	*/	833	*/
834	if (!nr_pages \|\| (snapshot && (nr_pages & 1)))	834	if (snapshot && (nr_pages & 1))
835	return NULL;	835	return NULL;
836		836
837	if (cpu == -1)	837	if (cpu == -1)

include/linux/arch_topology.h+2−0

@@ -30,6 +30,8 @@ static inline unsigned long topology_get_freq_scale(int cpu) · 29 unmodified lines
30	return per_cpu(freq_scale, cpu);	30	return per_cpu(freq_scale, cpu);
31	}	31	}
32		32
		33	bool arch_freq_counters_available(struct cpumask *cpus);
		34
33	DECLARE_PER_CPU(unsigned long, thermal_pressure);	35	DECLARE_PER_CPU(unsigned long, thermal_pressure);
34		36
35	static inline unsigned long topology_get_thermal_pressure(int cpu)	37	static inline unsigned long topology_get_thermal_pressure(int cpu)

include/linux/cpufreq.h+5−0

 unsigned int cpufreq_get(unsigned int cpu);
 unsigned int cpufreq_quick_get(unsigned int cpu);
 unsigned int cpufreq_quick_get_max(unsigned int cpu);
+unsigned int cpufreq_get_hw_max_freq(unsigned int cpu);
 void disable_cpufreq(void);
 u64 get_cpu_idle_time(unsigned int cpu, u64 *wall, int io_busy);
 {
 	return 0;
 }
+static inline unsigned int cpufreq_get_hw_max_freq(unsigned int cpu)
+{
+	return 0;
+}
 static inline void disable_cpufreq(void) { }
 #endif

include/linux/perf/arm_pmu.h+1−0

@@ -80,6 +80,7 @@ struct arm_pmu { · 79 unmodified lines
80	struct pmu pmu;	80	struct pmu pmu;
81	cpumask_t supported_cpus;	81	cpumask_t supported_cpus;
82	char *name;	82	char *name;
		83	int pmuver;
83	irqreturn_t (handle_irq)(struct arm_pmu pmu);	84	irqreturn_t (handle_irq)(struct arm_pmu pmu);
84	void (enable)(struct perf_event event);	85	void (enable)(struct perf_event event);
85	void (disable)(struct perf_event event);	86	void (disable)(struct perf_event event);

include/linux/stackprotector.h+1−1

@@ -6,7 +6,7 @@ · 5 unmodified lines
6	#include <linux/sched.h>	6	#include <linux/sched.h>
7	#include <linux/random.h>	7	#include <linux/random.h>
8		8
9	#ifdef CONFIG_STACKPROTECTOR	9	#if defined(CONFIG_STACKPROTECTOR) \|\| defined(CONFIG_ARM64_PTR_AUTH)
10	# include <asm/stackprotector.h>	10	# include <asm/stackprotector.h>
11	#else	11	#else
12	static inline void boot_init_stack_canary(void)	12	static inline void boot_init_stack_canary(void)

mm/mremap.c+10−0

@@ -606,6 +606,16 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, · 605 unmodified lines
606	LIST_HEAD(uf_unmap_early);	606	LIST_HEAD(uf_unmap_early);
607	LIST_HEAD(uf_unmap);	607	LIST_HEAD(uf_unmap);
608		608
		609	/*
		610	* There is a deliberate asymmetry here: we strip the pointer tag
		611	* from the old address but leave the new address alone. This is
		612	* for consistency with mmap(), where we prevent the creation of
		613	* aliasing mappings in userspace by leaving the tag bits of the
		614	* mapping address intact. A non-zero tag will cause the subsequent
		615	* range checks to reject the address as invalid.
		616	*
		617	* See Documentation/arm64/tagged-address-abi.rst for more information.
		618	*/
609	addr = untagged_addr(addr);	619	addr = untagged_addr(addr);
610		620
611	if (flags & ~(MREMAP_FIXED \| MREMAP_MAYMOVE))	621	if (flags & ~(MREMAP_FIXED \| MREMAP_MAYMOVE))

scripts/Kconfig.include+6−0

 # Return y if the linker supports <flag>, n otherwise
 ld-option = $(success,$(LD) -v $(1))
+# $(as-option,<flag>)
+# /dev/zero is used as output instead of /dev/null as some assembler cribs when
+# both input and output are same. Also both of them have same write behaviour so
+# can be easily substituted.
+as-option = $(success, $(CC) $(CLANG_FLAGS) $(1) -c -x assembler /dev/null -o /dev/zero)
 # $(as-instr,<instr>)
 # Return y if the assembler supports <instr>, n otherwise
 as-instr = $(success,printf "%b\n" "$(1)" | $(CC) $(CLANG_FLAGS) -c -x assembler -o /dev/null -)

@@ -999,7 +999,7 @@ static struct ptrauth_key pac_key_from_user(__uint128_t ukey) · 998 unmodified lines
999	}	999	}
1000		1000
1001	static void pac_address_keys_to_user(struct user_pac_address_keys *ukeys,	1001	static void pac_address_keys_to_user(struct user_pac_address_keys *ukeys,
1002	const struct ptrauth_keys *keys)	1002	const struct ptrauth_keys_user *keys)
1003	{	1003	{
1004	ukeys->apiakey = pac_key_to_user(&keys->apia);	1004	ukeys->apiakey = pac_key_to_user(&keys->apia);
1005	ukeys->apibkey = pac_key_to_user(&keys->apib);	1005	ukeys->apibkey = pac_key_to_user(&keys->apib);
@@ -1007,7 +1007,7 @@ static void pac_address_keys_to_user(struct user_pac_address_keys *ukeys, · 1 unmodified line
1007	ukeys->apdbkey = pac_key_to_user(&keys->apdb);	1007	ukeys->apdbkey = pac_key_to_user(&keys->apdb);
1008	}	1008	}
1009		1009
1010	static void pac_address_keys_from_user(struct ptrauth_keys *keys,	1010	static void pac_address_keys_from_user(struct ptrauth_keys_user *keys,
1011	const struct user_pac_address_keys *ukeys)	1011	const struct user_pac_address_keys *ukeys)
1012	{	1012	{
1013	keys->apia = pac_key_from_user(ukeys->apiakey);	1013	keys->apia = pac_key_from_user(ukeys->apiakey);
@@ -1021,7 +1021,7 @@ static int pac_address_keys_get(struct task_struct *target, · 7 unmodified lines
1021	unsigned int pos, unsigned int count,	1021	unsigned int pos, unsigned int count,
1022	void kbuf, void __user ubuf)	1022	void kbuf, void __user ubuf)
1023	{	1023	{
1024	struct ptrauth_keys *keys = &target->thread.keys_user;	1024	struct ptrauth_keys_user *keys = &target->thread.keys_user;
1025	struct user_pac_address_keys user_keys;	1025	struct user_pac_address_keys user_keys;
1026		1026
1027	if (!system_supports_address_auth())	1027	if (!system_supports_address_auth())
@@ -1038,7 +1038,7 @@ static int pac_address_keys_set(struct task_struct *target, · 10 unmodified lines
1038	unsigned int pos, unsigned int count,	1038	unsigned int pos, unsigned int count,
1039	const void kbuf, const void __user ubuf)	1039	const void kbuf, const void __user ubuf)
1040	{	1040	{
1041	struct ptrauth_keys *keys = &target->thread.keys_user;	1041	struct ptrauth_keys_user *keys = &target->thread.keys_user;
1042	struct user_pac_address_keys user_keys;	1042	struct user_pac_address_keys user_keys;
1043	int ret;	1043	int ret;
1044		1044
@@ -1056,12 +1056,12 @@ static int pac_address_keys_set(struct task_struct *target, · 11 unmodified lines
1056	}	1056	}
1057		1057
1058	static void pac_generic_keys_to_user(struct user_pac_generic_keys *ukeys,	1058	static void pac_generic_keys_to_user(struct user_pac_generic_keys *ukeys,
1059	const struct ptrauth_keys *keys)	1059	const struct ptrauth_keys_user *keys)
1060	{	1060	{
1061	ukeys->apgakey = pac_key_to_user(&keys->apga);	1061	ukeys->apgakey = pac_key_to_user(&keys->apga);
1062	}	1062	}
1063		1063
1064	static void pac_generic_keys_from_user(struct ptrauth_keys *keys,	1064	static void pac_generic_keys_from_user(struct ptrauth_keys_user *keys,
1065	const struct user_pac_generic_keys *ukeys)	1065	const struct user_pac_generic_keys *ukeys)
1066	{	1066	{
1067	keys->apga = pac_key_from_user(ukeys->apgakey);	1067	keys->apga = pac_key_from_user(ukeys->apgakey);
@@ -1072,7 +1072,7 @@ static int pac_generic_keys_get(struct task_struct *target, · 4 unmodified lines
1072	unsigned int pos, unsigned int count,	1072	unsigned int pos, unsigned int count,
1073	void kbuf, void __user ubuf)	1073	void kbuf, void __user ubuf)
1074	{	1074	{
1075	struct ptrauth_keys *keys = &target->thread.keys_user;	1075	struct ptrauth_keys_user *keys = &target->thread.keys_user;
1076	struct user_pac_generic_keys user_keys;	1076	struct user_pac_generic_keys user_keys;
1077		1077
1078	if (!system_supports_generic_auth())	1078	if (!system_supports_generic_auth())
@@ -1089,7 +1089,7 @@ static int pac_generic_keys_set(struct task_struct *target, · 10 unmodified lines
1089	unsigned int pos, unsigned int count,	1089	unsigned int pos, unsigned int count,
1090	const void kbuf, const void __user ubuf)	1090	const void kbuf, const void __user ubuf)
1091	{	1091	{
1092	struct ptrauth_keys *keys = &target->thread.keys_user;	1092	struct ptrauth_keys_user *keys = &target->thread.keys_user;
1093	struct user_pac_generic_keys user_keys;	1093	struct user_pac_generic_keys user_keys;
1094	int ret;	1094	int ret;
1095		1095

@@ -71,7 +71,7 @@ static void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_context *ct · 70 unmodified lines
71	ctxt->gp_regs.regs.pc = read_sysreg_el2(SYS_ELR);	71	ctxt->gp_regs.regs.pc = read_sysreg_el2(SYS_ELR);
72	ctxt->gp_regs.regs.pstate = read_sysreg_el2(SYS_SPSR);	72	ctxt->gp_regs.regs.pstate = read_sysreg_el2(SYS_SPSR);
73		73
74	if (cpus_have_const_cap(ARM64_HAS_RAS_EXTN))	74	if (cpus_have_final_cap(ARM64_HAS_RAS_EXTN))
75	ctxt->sys_regs[DISR_EL1] = read_sysreg_s(SYS_VDISR_EL2);	75	ctxt->sys_regs[DISR_EL1] = read_sysreg_s(SYS_VDISR_EL2);
76	}	76	}
77		77
@@ -118,7 +118,7 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) · 40 unmodified lines
118	write_sysreg(ctxt->sys_regs[MPIDR_EL1], vmpidr_el2);	118	write_sysreg(ctxt->sys_regs[MPIDR_EL1], vmpidr_el2);
119	write_sysreg(ctxt->sys_regs[CSSELR_EL1], csselr_el1);	119	write_sysreg(ctxt->sys_regs[CSSELR_EL1], csselr_el1);
120		120
121	if (!cpus_have_const_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE)) {	121	if (!cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE)) {
122	write_sysreg_el1(ctxt->sys_regs[SCTLR_EL1], SYS_SCTLR);	122	write_sysreg_el1(ctxt->sys_regs[SCTLR_EL1], SYS_SCTLR);
123	write_sysreg_el1(ctxt->sys_regs[TCR_EL1], SYS_TCR);	123	write_sysreg_el1(ctxt->sys_regs[TCR_EL1], SYS_TCR);
124	} else if (!ctxt->__hyp_running_vcpu) {	124	} else if (!ctxt->__hyp_running_vcpu) {
@@ -149,7 +149,7 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) · 24 unmodified lines
149	write_sysreg(ctxt->sys_regs[PAR_EL1], par_el1);	149	write_sysreg(ctxt->sys_regs[PAR_EL1], par_el1);
150	write_sysreg(ctxt->sys_regs[TPIDR_EL1], tpidr_el1);	150	write_sysreg(ctxt->sys_regs[TPIDR_EL1], tpidr_el1);
151		151
152	if (cpus_have_const_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE) &&	152	if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE) &&
153	ctxt->__hyp_running_vcpu) {	153	ctxt->__hyp_running_vcpu) {
154	/*	154	/*
155	* Must only be done for host registers, hence the context	155	* Must only be done for host registers, hence the context
@@ -194,7 +194,7 @@ __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt) · 38 unmodified lines
194	write_sysreg_el2(ctxt->gp_regs.regs.pc, SYS_ELR);	194	write_sysreg_el2(ctxt->gp_regs.regs.pc, SYS_ELR);
195	write_sysreg_el2(pstate, SYS_SPSR);	195	write_sysreg_el2(pstate, SYS_SPSR);
196		196
197	if (cpus_have_const_cap(ARM64_HAS_RAS_EXTN))	197	if (cpus_have_final_cap(ARM64_HAS_RAS_EXTN))
198	write_sysreg_s(ctxt->sys_regs[DISR_EL1], SYS_VDISR_EL2);	198	write_sysreg_s(ctxt->sys_regs[DISR_EL1], SYS_VDISR_EL2);
199	}	199	}
200		200