History

Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux

Pull arm64 updates from Will Deacon: "There's a reasonable amount here and the juicy details are all below. It's worth noting that the MTE/KASAN changes strayed outside of our usual directories due to core mm changes and some associated changes to some other architectures; Andrew asked for us to carry these [1] rather that take them via the -mm tree. Summary: - Optimise SVE switching for CPUs with 128-bit implementations. - Fix output format from SVE selftest. - Add support for versions v1.2 and 1.3 of the SMC calling convention. - Allow Pointer Authentication to be configured independently for kernel and userspace. - PMU driver cleanups for managing IRQ affinity and exposing event attributes via sysfs. - KASAN optimisations for both hardware tagging (MTE) and out-of-line software tagging implementations. - Relax frame record alignment requirements to facilitate 8-byte alignment with KASAN and Clang. - Cleanup of page-table definitions and removal of unused memory types. - Reduction of ARCH_DMA_MINALIGN back to 64 bytes. - Refactoring of our instruction decoding routines and addition of some missing encodings. - Move entry code moved into C and hardened against harmful compiler instrumentation. - Update booting requirements for the FEAT_HCX feature, added to v8.7 of the architecture. - Fix resume from idle when pNMI is being used. - Additional CPU sanity checks for MTE and preparatory changes for systems where not all of the CPUs support 32-bit EL0. - Update our kernel string routines to the latest Cortex Strings implementation. - Big cleanup of our cache maintenance routines, which were confusingly named and inconsistent in their implementations. - Tweak linker flags so that GDB can understand vmlinux when using RELR relocations. - Boot path cleanups to enable early initialisation of per-cpu operations needed by KCSAN. - Non-critical fixes and miscellaneous cleanup" * tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: (150 commits) arm64: tlb: fix the TTL value of tlb_get_level arm64: Restrict undef hook for cpufeature registers arm64/mm: Rename ARM64_SWAPPER_USES_SECTION_MAPS arm64: insn: avoid circular include dependency arm64: smp: Bump debugging information print down to KERN_DEBUG drivers/perf: fix the missed ida_simple_remove() in ddr_perf_probe() perf/arm-cmn: Fix invalid pointer when access dtc object sharing the same IRQ number arm64: suspend: Use cpuidle context helpers in cpu_suspend() PSCI: Use cpuidle context helpers in psci_cpu_suspend_enter() arm64: Convert cpu_do_idle() to using cpuidle context helpers arm64: Add cpuidle context save/restore helpers arm64: head: fix code comments in set_cpu_boot_mode_flag arm64: mm: drop unused __pa(__idmap_text_start) arm64: mm: fix the count comments in compute_indices arm64/mm: Fix ttbr0 values stored in struct thread_info for software-pan arm64: mm: Pass original fault address to handle_mm_fault() arm64/mm: Drop SECTION_[SHIFT|SIZE|MASK] arm64/mm: Use CONT_PMD_SHIFT for ARM64_MEMSTART_SHIFT arm64/mm: Drop SWAPPER_INIT_MAP_SIZE arm64: Conditionally configure PTR_AUTH key of the kernel. ...

Linus Torvalds committed 5 years ago

Commit 9840cfc2 parents 17cfb9d 3d1bf78158 files · +3370 −2563Browse files

Unified · Split

158 files
  1. Documentation/arm64/booting.rst +6 −0
  2. Makefile +1 −1
  3. arch/alpha/include/asm/page.h +3 −3
  4. arch/arm/include/asm/cpuidle.h +5 −0
  5. arch/arm/kernel/perf_event_v7.c +2 −2
  6. arch/arm64/Kconfig +19 −14
  7. arch/arm64/Makefile +1 −1
  8. arch/arm64/include/asm/alternative-macros.h +1 −8
  9. arch/arm64/include/asm/arch_gicv3.h +2 −1
  10. arch/arm64/include/asm/asm-prototypes.h +6 −0
  11. arch/arm64/include/asm/asm_pointer_auth.h +29 −20
  12. arch/arm64/include/asm/assembler.h +61 −37
  13. arch/arm64/include/asm/cache.h +1 −1
  14. arch/arm64/include/asm/cacheflush.h +42 −29
  15. arch/arm64/include/asm/cpu.h +25 −20
  16. arch/arm64/include/asm/cpufeature.h +14 −1
  17. arch/arm64/include/asm/cpuidle.h +35 −0
  18. arch/arm64/include/asm/efi.h +1 −1
  19. arch/arm64/include/asm/exception.h +26 −8
  20. arch/arm64/include/asm/fpsimd.h +1 −1
  21. arch/arm64/include/asm/fpsimdmacros.h +3 −1
  22. arch/arm64/include/asm/insn-def.h +9 −0
  23. arch/arm64/include/asm/insn.h +61 −6
  24. arch/arm64/include/asm/kernel-pgtable.h +8 −11
  25. arch/arm64/include/asm/kvm_asm.h +1 −0
  26. arch/arm64/include/asm/kvm_mmu.h +4 −3
  27. arch/arm64/include/asm/linkage.h +8 −0
  28. arch/arm64/include/asm/memory.h +2 −4
  29. arch/arm64/include/asm/mmu_context.h +2 −2
  30. arch/arm64/include/asm/module.lds.h +15 −2
  31. arch/arm64/include/asm/mte-kasan.h +67 −26
  32. arch/arm64/include/asm/mte.h +4 −0
  33. arch/arm64/include/asm/page.h +7 −3
  34. arch/arm64/include/asm/patching.h +13 −0
  35. arch/arm64/include/asm/perf_event.h +5 −0
  36. arch/arm64/include/asm/pgtable-hwdef.h +0 −7
  37. arch/arm64/include/asm/pgtable-prot.h +0 −1
  38. arch/arm64/include/asm/pgtable.h +2 −3
  39. arch/arm64/include/asm/pointer_auth.h +33 −26
  40. arch/arm64/include/asm/processor.h +8 −8
  41. arch/arm64/include/asm/scs.h +4 −4
  42. arch/arm64/include/asm/sdei.h +7 −3
  43. arch/arm64/include/asm/smp.h +0 −2
  44. arch/arm64/include/asm/stacktrace.h +16 −16
  45. arch/arm64/include/asm/sysreg.h +0 −2
  46. arch/arm64/include/asm/tlb.h +4 −0
  47. arch/arm64/kernel/Makefile +9 −2
  48. arch/arm64/kernel/acpi.c +17 −5
  49. arch/arm64/kernel/alternative.c +1 −1
  50. arch/arm64/kernel/asm-offsets.c +14 −1
  51. arch/arm64/kernel/cpufeature.c +162 −48
  52. arch/arm64/kernel/cpuinfo.c +32 −26
  53. arch/arm64/kernel/efi-entry.S +5 −4
  54. arch/arm64/kernel/entry-common.c +238 −18
  55. arch/arm64/kernel/entry-fpsimd.S +15 −7
  56. arch/arm64/kernel/entry.S +96 −273
  57. arch/arm64/kernel/fpsimd.c +4 −2
  58. arch/arm64/kernel/ftrace.c +1 −0
  59. arch/arm64/kernel/head.S +44 −32
  60. arch/arm64/kernel/hibernate-asm.S +4 −3
  61. arch/arm64/kernel/hibernate.c +12 −8
  62. arch/arm64/kernel/idle.c +46 −0
  63. arch/arm64/kernel/idreg-override.c +2 −1
  64. arch/arm64/kernel/image-vars.h +1 −1
  65. arch/arm64/kernel/jump_label.c +1 −0
  66. arch/arm64/kernel/kaslr.c +9 −3
  67. arch/arm64/kernel/kgdb.c +1 −0
  68. arch/arm64/kernel/machine_kexec.c +20 −10
  69. arch/arm64/kernel/patching.c +150 −0
  70. arch/arm64/kernel/perf_callchain.c +1 −1
  71. arch/arm64/kernel/perf_event.c +35 −5
  72. arch/arm64/kernel/probes/kprobes.c +10 −8
  73. arch/arm64/kernel/probes/simulate-insn.c +1 −0
  74. arch/arm64/kernel/probes/uprobes.c +1 −1
  75. arch/arm64/kernel/process.c +23 −76
  76. arch/arm64/kernel/ptrace.c +1 −1
  77. arch/arm64/kernel/sdei.c +12 −52
  78. arch/arm64/kernel/setup.c +1 −7
  79. arch/arm64/kernel/signal.c +26 −0
  80. arch/arm64/kernel/smccc-call.S +83 −0
  81. arch/arm64/kernel/smp.c +7 −9
  82. arch/arm64/kernel/smp_spin_table.c +4 −3
  83. arch/arm64/kernel/stacktrace.c +8 −10
  84. arch/arm64/kernel/suspend.c +11 −1
  85. arch/arm64/kernel/sys_compat.c +1 −1
  86. arch/arm64/kernel/traps.c +100 −35
  87. arch/arm64/kvm/arm.c +11 −2
  88. arch/arm64/kvm/hyp/nvhe/cache.S +2 −2
  89. arch/arm64/kvm/hyp/nvhe/setup.c +2 −1
  90. arch/arm64/kvm/hyp/nvhe/tlb.c +1 −1
  91. arch/arm64/kvm/hyp/pgtable.c +10 −3
  92. arch/arm64/lib/Makefile +3 −1
  93. arch/arm64/lib/clear_user.S +27 −20
  94. arch/arm64/lib/insn.c +4 −245
  95. arch/arm64/lib/kasan_sw_tags.S +76 −0
  96. arch/arm64/lib/memchr.S +53 −12
  97. arch/arm64/lib/memcmp.S +119 −227
  98. arch/arm64/lib/memcpy.S +229 −43
  99. arch/arm64/lib/memmove.S +0 −189
  100. arch/arm64/lib/mte.S +20 −0
  101. arch/arm64/lib/strcmp.S +121 −168
  102. arch/arm64/lib/strlen.S +173 −85
  103. arch/arm64/lib/strncmp.S +184 −222
  104. arch/arm64/lib/uaccess_flushcache.c +2 −2
  105. arch/arm64/mm/cache.S +81 −77
  106. arch/arm64/mm/context.c +2 −4
  107. arch/arm64/mm/fault.c +37 −13
  108. arch/arm64/mm/flush.c +14 −15
  109. arch/arm64/mm/init.c +7 −0
  110. arch/arm64/mm/mmu.c +7 −6
  111. arch/arm64/mm/proc.S +9 −15
  112. arch/arm64/mm/ptdump.c +0 −4
  113. arch/arm64/net/bpf_jit_comp.c +1 −0
  114. arch/arm64/tools/cpucaps +2 −1
  115. arch/ia64/include/asm/page.h +3 −3
  116. arch/m68k/include/asm/page_no.h +3 −3
  117. arch/s390/include/asm/page.h +3 −3
  118. arch/x86/include/asm/page.h +3 −3
  119. drivers/firmware/psci/psci.c +7 −2
  120. drivers/firmware/smccc/smccc.c +4 −0
  121. drivers/misc/lkdtm/bugs.c +3 −3
  122. drivers/perf/arm-cci.c +2 −2
  123. drivers/perf/arm-ccn.c +2 −4
  124. drivers/perf/arm-cmn.c +4 −9
  125. drivers/perf/arm_dmc620_pmu.c +2 −3
  126. drivers/perf/arm_dsu_pmu.c +2 −6
  127. drivers/perf/arm_pmu.c +6 −10
  128. drivers/perf/arm_smmuv3_pmu.c +15 −20
  129. drivers/perf/arm_spe_pmu.c +4 −8
  130. drivers/perf/fsl_imx8_ddr_perf.c +8 −10
  131. drivers/perf/hisilicon/hisi_uncore_ddrc_pmu.c +1 −4
  132. drivers/perf/hisilicon/hisi_uncore_hha_pmu.c +3 −6
  133. drivers/perf/hisilicon/hisi_uncore_l3c_pmu.c +1 −4
  134. drivers/perf/hisilicon/hisi_uncore_pa_pmu.c +1 −4
  135. drivers/perf/hisilicon/hisi_uncore_pmu.c +3 −3
  136. drivers/perf/hisilicon/hisi_uncore_pmu.h +1 −1
  137. drivers/perf/hisilicon/hisi_uncore_sllc_pmu.c +2 −5
  138. drivers/perf/qcom_l2_pmu.c +4 −7
  139. drivers/perf/qcom_l3_pmu.c +5 −10
  140. drivers/perf/thunderx2_pmu.c +1 −3
  141. drivers/perf/xgene_pmu.c +7 −10
  142. include/linux/arm-smccc.h +86 −2
  143. include/linux/gfp.h +15 −3
  144. include/linux/highmem.h +16 −27
  145. include/linux/interrupt.h +2 −33
  146. include/linux/kasan.h +38 −26
  147. include/linux/page-flags.h +9 −0
  148. include/linux/perf_event.h +6 −0
  149. include/trace/events/mmflags.h +8 −1
  150. kernel/irq/manage.c +32 −1
  151. mm/kasan/common.c +2 −2
  152. mm/kasan/hw_tags.c +32 −0
  153. mm/kasan/sw_tags.c +7 −0
  154. mm/mempool.c +4 −2
  155. mm/page_alloc.c +39 −27
  156. scripts/Makefile.kasan +1 −0
  157. scripts/tools-support-relr.sh +2 −1
  158. tools/testing/selftests/arm64/fp/sve-probe-vls.c +1 −1
Documentation/arm64/booting.rst+6−0
@@ -277,6 +277,12 @@ Before jumping into the kernel, the following conditions must be met: · 276 unmodified lines
277 277
278 - SCR_EL3.FGTEn (bit 27) must be initialised to 0b1.278 - SCR_EL3.FGTEn (bit 27) must be initialised to 0b1.
279 279
280 For CPUs with support for HCRX_EL2 (FEAT_HCX) present:
281
282 - If EL3 is present and the kernel is entered at EL2:
283
284 - SCR_EL3.HXEn (bit 38) must be initialised to 0b1.
285
280 For CPUs with Advanced SIMD and floating point support:286 For CPUs with Advanced SIMD and floating point support:
281 287
282 - If EL3 is present:288 - If EL3 is present:
Makefile+1−1
@@ -1039,7 +1039,7 @@ LDFLAGS_vmlinux += $(call ld-option, -X,) · 1038 unmodified lines
1039endif1039endif
1040 1040
1041ifeq ($(CONFIG_RELR),y)1041ifeq ($(CONFIG_RELR),y)
1042LDFLAGS_vmlinux += --pack-dyn-relocs=relr1042LDFLAGS_vmlinux += --pack-dyn-relocs=relr --use-android-relr-tags
1043endif1043endif
1044 1044
1045# We never want expected sections to be placed heuristically by the1045# We never want expected sections to be placed heuristically by the
arch/alpha/include/asm/page.h+3−3
@@ -17,9 +17,9 @@ · 16 unmodified lines
17extern void clear_page(void *page);17extern void clear_page(void *page);
18#define clear_user_page(page, vaddr, pg) clear_page(page)18#define clear_user_page(page, vaddr, pg) clear_page(page)
19 19
20#define __alloc_zeroed_user_highpage(movableflags, vma, vaddr) \20#define alloc_zeroed_user_highpage_movable(vma, vaddr) \
21 alloc_page_vma(GFP_HIGHUSER | __GFP_ZERO | movableflags, vma, vmaddr)21 alloc_page_vma(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, vma, vmaddr)
22#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE22#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE
23 23
24extern void copy_page(void * _to, void * _from);24extern void copy_page(void * _to, void * _from);
25#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)25#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
arch/arm/include/asm/cpuidle.h+5−0
@@ -50,4 +50,9 @@ extern int arm_cpuidle_suspend(int index); · 49 unmodified lines
50 50
51extern int arm_cpuidle_init(int cpu);51extern int arm_cpuidle_init(int cpu);
52 52
53struct arm_cpuidle_irq_context { };
54
55#define arm_cpuidle_save_irq_context(c) (void)c
56#define arm_cpuidle_restore_irq_context(c) (void)c
57
53#endif58#endif
arch/arm/kernel/perf_event_v7.c+2−2
@@ -773,10 +773,10 @@ static inline void armv7pmu_write_counter(struct perf_event *event, u64 value) · 772 unmodified lines
773 pr_err("CPU%u writing wrong counter %d\n",773 pr_err("CPU%u writing wrong counter %d\n",
774 smp_processor_id(), idx);774 smp_processor_id(), idx);
775 } else if (idx == ARMV7_IDX_CYCLE_COUNTER) {775 } else if (idx == ARMV7_IDX_CYCLE_COUNTER) {
776 asm volatile("mcr p15, 0, %0, c9, c13, 0" : : "r" (value));776 asm volatile("mcr p15, 0, %0, c9, c13, 0" : : "r" ((u32)value));
777 } else {777 } else {
778 armv7_pmnc_select_counter(idx);778 armv7_pmnc_select_counter(idx);
779 asm volatile("mcr p15, 0, %0, c9, c13, 2" : : "r" (value));779 asm volatile("mcr p15, 0, %0, c9, c13, 2" : : "r" ((u32)value));
780 }780 }
781}781}
782 782
arch/arm64/Kconfig+19−14
@@ -1481,12 +1481,6 @@ menu "ARMv8.3 architectural features" · 1480 unmodified lines
1481config ARM64_PTR_AUTH1481config ARM64_PTR_AUTH
1482 bool "Enable support for pointer authentication"1482 bool "Enable support for pointer authentication"
1483 default y1483 default y
1484 depends on (CC_HAS_SIGN_RETURN_ADDRESS || CC_HAS_BRANCH_PROT_PAC_RET) && AS_HAS_PAC
1485 # Modern compilers insert a .note.gnu.property section note for PAC
1486 # which is only understood by binutils starting with version 2.33.1.
1487 depends on LD_IS_LLD || LD_VERSION >= 23301 || (CC_IS_GCC && GCC_VERSION < 90100)
1488 depends on !CC_IS_CLANG || AS_HAS_CFI_NEGATE_RA_STATE
1489 depends on (!FUNCTION_GRAPH_TRACER || DYNAMIC_FTRACE_WITH_REGS)
1490 help1484 help
1491 Pointer authentication (part of the ARMv8.3 Extensions) provides1485 Pointer authentication (part of the ARMv8.3 Extensions) provides
1492 instructions for signing and authenticating pointers against secret1486 instructions for signing and authenticating pointers against secret
@@ -1498,13 +1492,6 @@ config ARM64_PTR_AUTH · 5 unmodified lines
1498 for each process at exec() time, with these keys being1492 for each process at exec() time, with these keys being
1499 context-switched along with the process.1493 context-switched along with the process.
1500 1494
1501 If the compiler supports the -mbranch-protection or
1502 -msign-return-address flag (e.g. GCC 7 or later), then this option
1503 will also cause the kernel itself to be compiled with return address
1504 protection. In this case, and if the target hardware is known to
1505 support pointer authentication, then CONFIG_STACKPROTECTOR can be
1506 disabled with minimal loss of protection.
1507
1508 The feature is detected at runtime. If the feature is not present in1495 The feature is detected at runtime. If the feature is not present in
1509 hardware it will not be advertised to userspace/KVM guest nor will it1496 hardware it will not be advertised to userspace/KVM guest nor will it
1510 be enabled.1497 be enabled.
@@ -1515,6 +1502,24 @@ config ARM64_PTR_AUTH · 4 unmodified lines
1515 but with the feature disabled. On such a system, this option should1502 but with the feature disabled. On such a system, this option should
1516 not be selected.1503 not be selected.
1517 1504
1505config ARM64_PTR_AUTH_KERNEL
1506 bool "Use pointer authentication for kernel"
1507 default y
1508 depends on ARM64_PTR_AUTH
1509 depends on (CC_HAS_SIGN_RETURN_ADDRESS || CC_HAS_BRANCH_PROT_PAC_RET) && AS_HAS_PAC
1510 # Modern compilers insert a .note.gnu.property section note for PAC
1511 # which is only understood by binutils starting with version 2.33.1.
1512 depends on LD_IS_LLD || LD_VERSION >= 23301 || (CC_IS_GCC && GCC_VERSION < 90100)
1513 depends on !CC_IS_CLANG || AS_HAS_CFI_NEGATE_RA_STATE
1514 depends on (!FUNCTION_GRAPH_TRACER || DYNAMIC_FTRACE_WITH_REGS)
1515 help
1516 If the compiler supports the -mbranch-protection or
1517 -msign-return-address flag (e.g. GCC 7 or later), then this option
1518 will cause the kernel itself to be compiled with return address
1519 protection. In this case, and if the target hardware is known to
1520 support pointer authentication, then CONFIG_STACKPROTECTOR can be
1521 disabled with minimal loss of protection.
1522
1518 This feature works with FUNCTION_GRAPH_TRACER option only if1523 This feature works with FUNCTION_GRAPH_TRACER option only if
1519 DYNAMIC_FTRACE_WITH_REGS is enabled.1524 DYNAMIC_FTRACE_WITH_REGS is enabled.
1520 1525
@@ -1606,7 +1611,7 @@ config ARM64_BTI_KERNEL · 85 unmodified lines
1606 bool "Use Branch Target Identification for kernel"1611 bool "Use Branch Target Identification for kernel"
1607 default y1612 default y
1608 depends on ARM64_BTI1613 depends on ARM64_BTI
1609 depends on ARM64_PTR_AUTH1614 depends on ARM64_PTR_AUTH_KERNEL
1610 depends on CC_HAS_BRANCH_PROT_PAC_RET_BTI1615 depends on CC_HAS_BRANCH_PROT_PAC_RET_BTI
1611 # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=946971616 # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94697
1612 depends on !CC_IS_GCC || GCC_VERSION >= 1001001617 depends on !CC_IS_GCC || GCC_VERSION >= 100100
arch/arm64/Makefile+1−1
@@ -70,7 +70,7 @@ endif · 69 unmodified lines
70# off, this will be overridden if we are using branch protection.70# off, this will be overridden if we are using branch protection.
71branch-prot-flags-y += $(call cc-option,-mbranch-protection=none)71branch-prot-flags-y += $(call cc-option,-mbranch-protection=none)
72 72
73ifeq ($(CONFIG_ARM64_PTR_AUTH),y)73ifeq ($(CONFIG_ARM64_PTR_AUTH_KERNEL),y)
74branch-prot-flags-$(CONFIG_CC_HAS_SIGN_RETURN_ADDRESS) := -msign-return-address=all74branch-prot-flags-$(CONFIG_CC_HAS_SIGN_RETURN_ADDRESS) := -msign-return-address=all
75# We enable additional protection for leaf functions as there is some75# We enable additional protection for leaf functions as there is some
76# narrow potential for ROP protection benefits and no substantial76# narrow potential for ROP protection benefits and no substantial
arch/arm64/include/asm/alternative-macros.h+1−8
@@ -3,12 +3,10 @@ · 2 unmodified lines
3#define __ASM_ALTERNATIVE_MACROS_H3#define __ASM_ALTERNATIVE_MACROS_H
4 4
5#include <asm/cpucaps.h>5#include <asm/cpucaps.h>
6#include <asm/insn-def.h>
6 7
7#define ARM64_CB_PATCH ARM64_NCAPS8#define ARM64_CB_PATCH ARM64_NCAPS
8 9
9/* A64 instructions are always 32 bits. */
10#define AARCH64_INSN_SIZE 4
11
12#ifndef __ASSEMBLY__10#ifndef __ASSEMBLY__
13 11
14#include <linux/stringify.h>12#include <linux/stringify.h>
@@ -197,11 +195,6 @@ alternative_endif · 182 unmodified lines
197#define _ALTERNATIVE_CFG(insn1, insn2, cap, cfg, ...) \195#define _ALTERNATIVE_CFG(insn1, insn2, cap, cfg, ...) \
198 alternative_insn insn1, insn2, cap, IS_ENABLED(cfg)196 alternative_insn insn1, insn2, cap, IS_ENABLED(cfg)
199 197
200.macro user_alt, label, oldinstr, newinstr, cond
2019999: alternative_insn "\oldinstr", "\newinstr", \cond
202 _asm_extable 9999b, \label
203.endm
204
205#endif /* __ASSEMBLY__ */198#endif /* __ASSEMBLY__ */
206 199
207/*200/*
arch/arm64/include/asm/arch_gicv3.h+2−1
@@ -124,7 +124,8 @@ static inline u32 gic_read_rpr(void) · 123 unmodified lines
124#define gic_read_lpir(c) readq_relaxed(c)124#define gic_read_lpir(c) readq_relaxed(c)
125#define gic_write_lpir(v, c) writeq_relaxed(v, c)125#define gic_write_lpir(v, c) writeq_relaxed(v, c)
126 126
127#define gic_flush_dcache_to_poc(a,l) __flush_dcache_area((a), (l))127#define gic_flush_dcache_to_poc(a,l) \
128 dcache_clean_inval_poc((unsigned long)(a), (unsigned long)(a)+(l))
128 129
129#define gits_read_baser(c) readq_relaxed(c)130#define gits_read_baser(c) readq_relaxed(c)
130#define gits_write_baser(v, c) writeq_relaxed(v, c)131#define gits_write_baser(v, c) writeq_relaxed(v, c)
arch/arm64/include/asm/asm-prototypes.h+6−0
@@ -23,4 +23,10 @@ long long __ashlti3(long long a, int b); · 22 unmodified lines
23long long __ashrti3(long long a, int b);23long long __ashrti3(long long a, int b);
24long long __lshrti3(long long a, int b);24long long __lshrti3(long long a, int b);
25 25
26/*
27 * This function uses a custom calling convention and cannot be called from C so
28 * this prototype is not entirely accurate.
29 */
30void __hwasan_tag_mismatch(unsigned long addr, unsigned long access_info);
31
26#endif /* __ASM_PROTOTYPES_H */32#endif /* __ASM_PROTOTYPES_H */
arch/arm64/include/asm/asm_pointer_auth.h+29−20
@@ -7,19 +7,7 @@ · 6 unmodified lines
7#include <asm/cpufeature.h>7#include <asm/cpufeature.h>
8#include <asm/sysreg.h>8#include <asm/sysreg.h>
9 9
10#ifdef CONFIG_ARM64_PTR_AUTH10#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL
11/*
12 * thread.keys_user.ap* as offset exceeds the #imm offset range
13 * so use the base value of ldp as thread.keys_user and offset as
14 * thread.keys_user.ap*.
15 */
16 .macro __ptrauth_keys_install_user tsk, tmp1, tmp2, tmp3
17 mov \tmp1, #THREAD_KEYS_USER
18 add \tmp1, \tsk, \tmp1
19 ldp \tmp2, \tmp3, [\tmp1, #PTRAUTH_USER_KEY_APIA]
20 msr_s SYS_APIAKEYLO_EL1, \tmp2
21 msr_s SYS_APIAKEYHI_EL1, \tmp3
22 .endm
23 11
24 .macro __ptrauth_keys_install_kernel_nosync tsk, tmp1, tmp2, tmp312 .macro __ptrauth_keys_install_kernel_nosync tsk, tmp1, tmp2, tmp3
25 mov \tmp1, #THREAD_KEYS_KERNEL13 mov \tmp1, #THREAD_KEYS_KERNEL
@@ -42,6 +30,33 @@ alternative_if ARM64_HAS_ADDRESS_AUTH · 16 unmodified lines
42alternative_else_nop_endif30alternative_else_nop_endif
43 .endm31 .endm
44 32
33#else /* CONFIG_ARM64_PTR_AUTH_KERNEL */
34
35 .macro __ptrauth_keys_install_kernel_nosync tsk, tmp1, tmp2, tmp3
36 .endm
37
38 .macro ptrauth_keys_install_kernel_nosync tsk, tmp1, tmp2, tmp3
39 .endm
40
41 .macro ptrauth_keys_install_kernel tsk, tmp1, tmp2, tmp3
42 .endm
43
44#endif /* CONFIG_ARM64_PTR_AUTH_KERNEL */
45
46#ifdef CONFIG_ARM64_PTR_AUTH
47/*
48 * thread.keys_user.ap* as offset exceeds the #imm offset range
49 * so use the base value of ldp as thread.keys_user and offset as
50 * thread.keys_user.ap*.
51 */
52 .macro __ptrauth_keys_install_user tsk, tmp1, tmp2, tmp3
53 mov \tmp1, #THREAD_KEYS_USER
54 add \tmp1, \tsk, \tmp1
55 ldp \tmp2, \tmp3, [\tmp1, #PTRAUTH_USER_KEY_APIA]
56 msr_s SYS_APIAKEYLO_EL1, \tmp2
57 msr_s SYS_APIAKEYHI_EL1, \tmp3
58 .endm
59
45 .macro __ptrauth_keys_init_cpu tsk, tmp1, tmp2, tmp360 .macro __ptrauth_keys_init_cpu tsk, tmp1, tmp2, tmp3
46 mrs \tmp1, id_aa64isar1_el161 mrs \tmp1, id_aa64isar1_el1
47 ubfx \tmp1, \tmp1, #ID_AA64ISAR1_APA_SHIFT, #862 ubfx \tmp1, \tmp1, #ID_AA64ISAR1_APA_SHIFT, #8
@@ -64,17 +79,11 @@ alternative_else_nop_endif · 16 unmodified lines
64.Lno_addr_auth\@:79.Lno_addr_auth\@:
65 .endm80 .endm
66 81
67#else /* CONFIG_ARM64_PTR_AUTH */82#else /* !CONFIG_ARM64_PTR_AUTH */
68 83
69 .macro ptrauth_keys_install_user tsk, tmp1, tmp2, tmp384 .macro ptrauth_keys_install_user tsk, tmp1, tmp2, tmp3
70 .endm85 .endm
71 86
72 .macro ptrauth_keys_install_kernel_nosync tsk, tmp1, tmp2, tmp3
73 .endm
74
75 .macro ptrauth_keys_install_kernel tsk, tmp1, tmp2, tmp3
76 .endm
77
78#endif /* CONFIG_ARM64_PTR_AUTH */87#endif /* CONFIG_ARM64_PTR_AUTH */
79 88
80#endif /* __ASM_ASM_POINTER_AUTH_H */89#endif /* __ASM_ASM_POINTER_AUTH_H */
arch/arm64/include/asm/assembler.h+61−37
@@ -130,15 +130,27 @@ alternative_endif · 129 unmodified lines
130 .endm130 .endm
131 131
132/*132/*
133 * Emit an entry into the exception table133 * Create an exception table entry for `insn`, which will branch to `fixup`
134 * when an unhandled fault is taken.
134 */135 */
135 .macro _asm_extable, from, to136 .macro _asm_extable, insn, fixup
136 .pushsection __ex_table, "a"137 .pushsection __ex_table, "a"
137 .align 3138 .align 3
138 .long (\from - .), (\to - .)139 .long (\insn - .), (\fixup - .)
139 .popsection140 .popsection
140 .endm141 .endm
141 142
143/*
144 * Create an exception table entry for `insn` if `fixup` is provided. Otherwise
145 * do nothing.
146 */
147 .macro _cond_extable, insn, fixup
148 .ifnc \fixup,
149 _asm_extable \insn, \fixup
150 .endif
151 .endm
152
153
142#define USER(l, x...) \154#define USER(l, x...) \
1439999: x; \1559999: x; \
144 _asm_extable 9999b, l156 _asm_extable 9999b, l
@@ -232,15 +244,23 @@ lr .req x30 // link register · 87 unmodified lines
232 * @dst: destination register244 * @dst: destination register
233 */245 */
234#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)246#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
235 .macro this_cpu_offset, dst247 .macro get_this_cpu_offset, dst
236 mrs \dst, tpidr_el2248 mrs \dst, tpidr_el2
237 .endm249 .endm
238#else250#else
239 .macro this_cpu_offset, dst251 .macro get_this_cpu_offset, dst
240alternative_if_not ARM64_HAS_VIRT_HOST_EXTN252alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
241 mrs \dst, tpidr_el1253 mrs \dst, tpidr_el1
242alternative_else254alternative_else
243 mrs \dst, tpidr_el2255 mrs \dst, tpidr_el2
256alternative_endif
257 .endm
258
259 .macro set_this_cpu_offset, src
260alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
261 msr tpidr_el1, \src
262alternative_else
263 msr tpidr_el2, \src
244alternative_endif264alternative_endif
245 .endm265 .endm
246#endif266#endif
@@ -253,7 +273,7 @@ alternative_endif · 6 unmodified lines
253 .macro adr_this_cpu, dst, sym, tmp273 .macro adr_this_cpu, dst, sym, tmp
254 adrp \tmp, \sym274 adrp \tmp, \sym
255 add \dst, \tmp, #:lo12:\sym275 add \dst, \tmp, #:lo12:\sym
256 this_cpu_offset \tmp276 get_this_cpu_offset \tmp
257 add \dst, \dst, \tmp277 add \dst, \dst, \tmp
258 .endm278 .endm
259 279
@@ -264,7 +284,7 @@ alternative_endif · 4 unmodified lines
264 */284 */
265 .macro ldr_this_cpu dst, sym, tmp285 .macro ldr_this_cpu dst, sym, tmp
266 adr_l \dst, \sym286 adr_l \dst, \sym
267 this_cpu_offset \tmp287 get_this_cpu_offset \tmp
268 ldr \dst, [\dst, \tmp]288 ldr \dst, [\dst, \tmp]
269 .endm289 .endm
270 290
@@ -375,51 +395,53 @@ alternative_cb_end · 104 unmodified lines
375 bfi \tcr, \tmp0, \pos, #3395 bfi \tcr, \tmp0, \pos, #3
376 .endm396 .endm
377 397
398 .macro __dcache_op_workaround_clean_cache, op, addr
399alternative_if_not ARM64_WORKAROUND_CLEAN_CACHE
400 dc \op, \addr
401alternative_else
402 dc civac, \addr
403alternative_endif
404 .endm
405
378/*406/*
379 * Macro to perform a data cache maintenance for the interval407 * Macro to perform a data cache maintenance for the interval
380 * [kaddr, kaddr + size)408 * [start, end)
381 *409 *
382 * op: operation passed to dc instruction410 * op: operation passed to dc instruction
383 * domain: domain used in dsb instruciton411 * domain: domain used in dsb instruciton
384 * kaddr: starting virtual address of the region412 * start: starting virtual address of the region
385 * size: size of the region413 * end: end virtual address of the region
386 * Corrupts: kaddr, size, tmp1, tmp2414 * fixup: optional label to branch to on user fault
415 * Corrupts: start, end, tmp1, tmp2
387 */416 */
388 .macro __dcache_op_workaround_clean_cache, op, kaddr417 .macro dcache_by_line_op op, domain, start, end, tmp1, tmp2, fixup
389alternative_if_not ARM64_WORKAROUND_CLEAN_CACHE
390 dc \op, \kaddr
391alternative_else
392 dc civac, \kaddr
393alternative_endif
394 .endm
395
396 .macro dcache_by_line_op op, domain, kaddr, size, tmp1, tmp2
397 dcache_line_size \tmp1, \tmp2418 dcache_line_size \tmp1, \tmp2
398 add \size, \kaddr, \size
399 sub \tmp2, \tmp1, #1419 sub \tmp2, \tmp1, #1
400 bic \kaddr, \kaddr, \tmp2420 bic \start, \start, \tmp2
4019998:421.Ldcache_op\@:
402 .ifc \op, cvau422 .ifc \op, cvau
403 __dcache_op_workaround_clean_cache \op, \kaddr423 __dcache_op_workaround_clean_cache \op, \start
404 .else424 .else
405 .ifc \op, cvac425 .ifc \op, cvac
406 __dcache_op_workaround_clean_cache \op, \kaddr426 __dcache_op_workaround_clean_cache \op, \start
407 .else427 .else
408 .ifc \op, cvap428 .ifc \op, cvap
409 sys 3, c7, c12, 1, \kaddr // dc cvap429 sys 3, c7, c12, 1, \start // dc cvap
410 .else430 .else
411 .ifc \op, cvadp431 .ifc \op, cvadp
412 sys 3, c7, c13, 1, \kaddr // dc cvadp432 sys 3, c7, c13, 1, \start // dc cvadp
413 .else433 .else
414 dc \op, \kaddr434 dc \op, \start
415 .endif435 .endif
416 .endif436 .endif
417 .endif437 .endif
418 .endif438 .endif
419 add \kaddr, \kaddr, \tmp1439 add \start, \start, \tmp1
420 cmp \kaddr, \size440 cmp \start, \end
421 b.lo 9998b441 b.lo .Ldcache_op\@
422 dsb \domain442 dsb \domain
443
444 _cond_extable .Ldcache_op\@, \fixup
423 .endm445 .endm
424 446
425/*447/*
@@ -427,20 +449,22 @@ alternative_endif · 1 unmodified line
427 * [start, end)449 * [start, end)
428 *450 *
429 * start, end: virtual addresses describing the region451 * start, end: virtual addresses describing the region
430 * label: A label to branch to on user fault.452 * fixup: optional label to branch to on user fault
431 * Corrupts: tmp1, tmp2453 * Corrupts: tmp1, tmp2
432 */454 */
433 .macro invalidate_icache_by_line start, end, tmp1, tmp2, label455 .macro invalidate_icache_by_line start, end, tmp1, tmp2, fixup
434 icache_line_size \tmp1, \tmp2456 icache_line_size \tmp1, \tmp2
435 sub \tmp2, \tmp1, #1457 sub \tmp2, \tmp1, #1
436 bic \tmp2, \start, \tmp2458 bic \tmp2, \start, \tmp2
4379997:459.Licache_op\@:
438USER(\label, ic ivau, \tmp2) // invalidate I line PoU460 ic ivau, \tmp2 // invalidate I line PoU
439 add \tmp2, \tmp2, \tmp1461 add \tmp2, \tmp2, \tmp1
440 cmp \tmp2, \end462 cmp \tmp2, \end
441 b.lo 9997b463 b.lo .Licache_op\@
442 dsb ish464 dsb ish
443 isb465 isb
466
467 _cond_extable .Licache_op\@, \fixup
444 .endm468 .endm
445 469
446/*470/*
@@ -745,7 +769,7 @@ USER(\label, ic ivau, \tmp2) // invalidate I line PoU · 298 unmodified lines
745 cbz \tmp, \lbl769 cbz \tmp, \lbl
746#endif770#endif
747 adr_l \tmp, irq_stat + IRQ_CPUSTAT_SOFTIRQ_PENDING771 adr_l \tmp, irq_stat + IRQ_CPUSTAT_SOFTIRQ_PENDING
748 this_cpu_offset \tmp2772 get_this_cpu_offset \tmp2
749 ldr w\tmp, [\tmp, \tmp2]773 ldr w\tmp, [\tmp, \tmp2]
750 cbnz w\tmp, \lbl // yield on pending softirq in task context774 cbnz w\tmp, \lbl // yield on pending softirq in task context
751.Lnoyield_\@:775.Lnoyield_\@:
arch/arm64/include/asm/cache.h+1−1
@@ -47,7 +47,7 @@ · 46 unmodified lines
47 * cache before the transfer is done, causing old data to be seen by47 * cache before the transfer is done, causing old data to be seen by
48 * the CPU.48 * the CPU.
49 */49 */
50#define ARCH_DMA_MINALIGN (128)50#define ARCH_DMA_MINALIGN L1_CACHE_BYTES
51 51
52#ifdef CONFIG_KASAN_SW_TAGS52#ifdef CONFIG_KASAN_SW_TAGS
53#define ARCH_SLAB_MINALIGN (1ULL << KASAN_SHADOW_SCALE_SHIFT)53#define ARCH_SLAB_MINALIGN (1ULL << KASAN_SHADOW_SCALE_SHIFT)
arch/arm64/include/asm/cacheflush.h+42−29
@@ -30,45 +30,58 @@ · 29 unmodified lines
30 * the implementation assumes non-aliasing VIPT D-cache and (aliasing)30 * the implementation assumes non-aliasing VIPT D-cache and (aliasing)
31 * VIPT I-cache.31 * VIPT I-cache.
32 *32 *
33 * flush_icache_range(start, end)33 * All functions below apply to the interval [start, end)
34 * - start - virtual start address (inclusive)
35 * - end - virtual end address (exclusive)
34 *36 *
35 * Ensure coherency between the I-cache and the D-cache in the37 * caches_clean_inval_pou(start, end)
36 * region described by start, end.
37 * - start - virtual start address
38 * - end - virtual end address
39 *38 *
40 * invalidate_icache_range(start, end)39 * Ensure coherency between the I-cache and the D-cache region to
40 * the Point of Unification.
41 *41 *
42 * Invalidate the I-cache in the region described by start, end.42 * caches_clean_inval_user_pou(start, end)
43 * - start - virtual start address
44 * - end - virtual end address
45 *43 *
46 * __flush_cache_user_range(start, end)44 * Ensure coherency between the I-cache and the D-cache region to
45 * the Point of Unification.
46 * Use only if the region might access user memory.
47 *47 *
48 * Ensure coherency between the I-cache and the D-cache in the48 * icache_inval_pou(start, end)
49 * region described by start, end.
50 * - start - virtual start address
51 * - end - virtual end address
52 *49 *
53 * __flush_dcache_area(kaddr, size)50 * Invalidate I-cache region to the Point of Unification.
54 *51 *
55 * Ensure that the data held in page is written back.52 * dcache_clean_inval_poc(start, end)
56 * - kaddr - page address53 *
57 * - size - region size54 * Clean and invalidate D-cache region to the Point of Coherency.
55 *
56 * dcache_inval_poc(start, end)
57 *
58 * Invalidate D-cache region to the Point of Coherency.
59 *
60 * dcache_clean_poc(start, end)
61 *
62 * Clean D-cache region to the Point of Coherency.
63 *
64 * dcache_clean_pop(start, end)
65 *
66 * Clean D-cache region to the Point of Persistence.
67 *
68 * dcache_clean_pou(start, end)
69 *
70 * Clean D-cache region to the Point of Unification.
58 */71 */
59extern void __flush_icache_range(unsigned long start, unsigned long end);72extern void caches_clean_inval_pou(unsigned long start, unsigned long end);
60extern int invalidate_icache_range(unsigned long start, unsigned long end);73extern void icache_inval_pou(unsigned long start, unsigned long end);
61extern void __flush_dcache_area(void *addr, size_t len);74extern void dcache_clean_inval_poc(unsigned long start, unsigned long end);
62extern void __inval_dcache_area(void *addr, size_t len);75extern void dcache_inval_poc(unsigned long start, unsigned long end);
63extern void __clean_dcache_area_poc(void *addr, size_t len);76extern void dcache_clean_poc(unsigned long start, unsigned long end);
64extern void __clean_dcache_area_pop(void *addr, size_t len);77extern void dcache_clean_pop(unsigned long start, unsigned long end);
65extern void __clean_dcache_area_pou(void *addr, size_t len);78extern void dcache_clean_pou(unsigned long start, unsigned long end);
66extern long __flush_cache_user_range(unsigned long start, unsigned long end);79extern long caches_clean_inval_user_pou(unsigned long start, unsigned long end);
67extern void sync_icache_aliases(void *kaddr, unsigned long len);80extern void sync_icache_aliases(unsigned long start, unsigned long end);
68 81
69static inline void flush_icache_range(unsigned long start, unsigned long end)82static inline void flush_icache_range(unsigned long start, unsigned long end)
70{83{
71 __flush_icache_range(start, end);84 caches_clean_inval_pou(start, end);
72 85
73 /*86 /*
74 * IPI all online CPUs so that they undergo a context synchronization87 * IPI all online CPUs so that they undergo a context synchronization
@@ -122,7 +135,7 @@ extern void copy_to_user_page(struct vm_area_struct *, struct page *, · 47 unmodified lines
122#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1135#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1
123extern void flush_dcache_page(struct page *);136extern void flush_dcache_page(struct page *);
124 137
125static __always_inline void __flush_icache_all(void)138static __always_inline void icache_inval_all_pou(void)
126{139{
127 if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC))140 if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC))
128 return;141 return;
arch/arm64/include/asm/cpu.h+25−20
@@ -12,26 +12,7 @@ · 11 unmodified lines
12/*12/*
13 * Records attributes of an individual CPU.13 * Records attributes of an individual CPU.
14 */14 */
15struct cpuinfo_arm64 {15struct cpuinfo_32bit {
16 struct cpu cpu;
17 struct kobject kobj;
18 u32 reg_ctr;
19 u32 reg_cntfrq;
20 u32 reg_dczid;
21 u32 reg_midr;
22 u32 reg_revidr;
23
24 u64 reg_id_aa64dfr0;
25 u64 reg_id_aa64dfr1;
26 u64 reg_id_aa64isar0;
27 u64 reg_id_aa64isar1;
28 u64 reg_id_aa64mmfr0;
29 u64 reg_id_aa64mmfr1;
30 u64 reg_id_aa64mmfr2;
31 u64 reg_id_aa64pfr0;
32 u64 reg_id_aa64pfr1;
33 u64 reg_id_aa64zfr0;
34
35 u32 reg_id_dfr0;16 u32 reg_id_dfr0;
36 u32 reg_id_dfr1;17 u32 reg_id_dfr1;
37 u32 reg_id_isar0;18 u32 reg_id_isar0;
@@ -54,6 +35,30 @@ struct cpuinfo_arm64 { · 16 unmodified lines
54 u32 reg_mvfr0;35 u32 reg_mvfr0;
55 u32 reg_mvfr1;36 u32 reg_mvfr1;
56 u32 reg_mvfr2;37 u32 reg_mvfr2;
38};
39
40struct cpuinfo_arm64 {
41 struct cpu cpu;
42 struct kobject kobj;
43 u64 reg_ctr;
44 u64 reg_cntfrq;
45 u64 reg_dczid;
46 u64 reg_midr;
47 u64 reg_revidr;
48 u64 reg_gmid;
49
50 u64 reg_id_aa64dfr0;
51 u64 reg_id_aa64dfr1;
52 u64 reg_id_aa64isar0;
53 u64 reg_id_aa64isar1;
54 u64 reg_id_aa64mmfr0;
55 u64 reg_id_aa64mmfr1;
56 u64 reg_id_aa64mmfr2;
57 u64 reg_id_aa64pfr0;
58 u64 reg_id_aa64pfr1;
59 u64 reg_id_aa64zfr0;
60
61 struct cpuinfo_32bit aarch32;
57 62
58 /* pseudo-ZCR for recording maximum ZCR_EL1 LEN value: */63 /* pseudo-ZCR for recording maximum ZCR_EL1 LEN value: */
59 u64 reg_zcr;64 u64 reg_zcr;
arch/arm64/include/asm/cpufeature.h+14−1
@@ -619,6 +619,13 @@ static inline bool id_aa64pfr0_sve(u64 pfr0) · 618 unmodified lines
619 return val > 0;619 return val > 0;
620}620}
621 621
622static inline bool id_aa64pfr1_mte(u64 pfr1)
623{
624 u32 val = cpuid_feature_extract_unsigned_field(pfr1, ID_AA64PFR1_MTE_SHIFT);
625
626 return val >= ID_AA64PFR1_MTE;
627}
628
622void __init setup_cpu_features(void);629void __init setup_cpu_features(void);
623void check_local_cpu_capabilities(void);630void check_local_cpu_capabilities(void);
624 631
@@ -630,9 +637,15 @@ static inline bool cpu_supports_mixed_endian_el0(void) · 5 unmodified lines
630 return id_aa64mmfr0_mixed_endian_el0(read_cpuid(ID_AA64MMFR0_EL1));637 return id_aa64mmfr0_mixed_endian_el0(read_cpuid(ID_AA64MMFR0_EL1));
631}638}
632 639
640const struct cpumask *system_32bit_el0_cpumask(void);
641DECLARE_STATIC_KEY_FALSE(arm64_mismatched_32bit_el0);
642
633static inline bool system_supports_32bit_el0(void)643static inline bool system_supports_32bit_el0(void)
634{644{
635 return cpus_have_const_cap(ARM64_HAS_32BIT_EL0);645 u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
646
647 return static_branch_unlikely(&arm64_mismatched_32bit_el0) ||
648 id_aa64pfr0_32bit_el0(pfr0);
636}649}
637 650
638static inline bool system_supports_4kb_granule(void)651static inline bool system_supports_4kb_granule(void)
arch/arm64/include/asm/cpuidle.h+35−0
@@ -18,4 +18,39 @@ static inline int arm_cpuidle_suspend(int index) · 17 unmodified lines
18 return -EOPNOTSUPP;18 return -EOPNOTSUPP;
19}19}
20#endif20#endif
21
22#ifdef CONFIG_ARM64_PSEUDO_NMI
23#include <asm/arch_gicv3.h>
24
25struct arm_cpuidle_irq_context {
26 unsigned long pmr;
27 unsigned long daif_bits;
28};
29
30#define arm_cpuidle_save_irq_context(__c) \
31 do { \
32 struct arm_cpuidle_irq_context *c = __c; \
33 if (system_uses_irq_prio_masking()) { \
34 c->daif_bits = read_sysreg(daif); \
35 write_sysreg(c->daif_bits | PSR_I_BIT | PSR_F_BIT, \
36 daif); \
37 c->pmr = gic_read_pmr(); \
38 gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); \
39 } \
40 } while (0)
41
42#define arm_cpuidle_restore_irq_context(__c) \
43 do { \
44 struct arm_cpuidle_irq_context *c = __c; \
45 if (system_uses_irq_prio_masking()) { \
46 gic_write_pmr(c->pmr); \
47 write_sysreg(c->daif_bits, daif); \
48 } \
49 } while (0)
50#else
51struct arm_cpuidle_irq_context { };
52
53#define arm_cpuidle_save_irq_context(c) (void)c
54#define arm_cpuidle_restore_irq_context(c) (void)c
55#endif
21#endif56#endif
arch/arm64/include/asm/efi.h+1−1
@@ -137,7 +137,7 @@ void efi_virtmap_unload(void); · 136 unmodified lines
137 137
138static inline void efi_capsule_flush_cache_range(void *addr, int size)138static inline void efi_capsule_flush_cache_range(void *addr, int size)
139{139{
140 __flush_dcache_area(addr, size);140 dcache_clean_inval_poc((unsigned long)addr, (unsigned long)addr + size);
141}141}
142 142
143#endif /* _ASM_EFI_H */143#endif /* _ASM_EFI_H */
arch/arm64/include/asm/exception.h+26−8
@@ -31,20 +31,35 @@ static inline u32 disr_to_esr(u64 disr) · 30 unmodified lines
31 return esr;31 return esr;
32}32}
33 33
34asmlinkage void el1_sync_handler(struct pt_regs *regs);34asmlinkage void handle_bad_stack(struct pt_regs *regs);
35asmlinkage void el0_sync_handler(struct pt_regs *regs);
36asmlinkage void el0_sync_compat_handler(struct pt_regs *regs);
37 35
38asmlinkage void noinstr enter_el1_irq_or_nmi(struct pt_regs *regs);36asmlinkage void el1t_64_sync_handler(struct pt_regs *regs);
39asmlinkage void noinstr exit_el1_irq_or_nmi(struct pt_regs *regs);37asmlinkage void el1t_64_irq_handler(struct pt_regs *regs);
38asmlinkage void el1t_64_fiq_handler(struct pt_regs *regs);
39asmlinkage void el1t_64_error_handler(struct pt_regs *regs);
40
41asmlinkage void el1h_64_sync_handler(struct pt_regs *regs);
42asmlinkage void el1h_64_irq_handler(struct pt_regs *regs);
43asmlinkage void el1h_64_fiq_handler(struct pt_regs *regs);
44asmlinkage void el1h_64_error_handler(struct pt_regs *regs);
45
46asmlinkage void el0t_64_sync_handler(struct pt_regs *regs);
47asmlinkage void el0t_64_irq_handler(struct pt_regs *regs);
48asmlinkage void el0t_64_fiq_handler(struct pt_regs *regs);
49asmlinkage void el0t_64_error_handler(struct pt_regs *regs);
50
51asmlinkage void el0t_32_sync_handler(struct pt_regs *regs);
52asmlinkage void el0t_32_irq_handler(struct pt_regs *regs);
53asmlinkage void el0t_32_fiq_handler(struct pt_regs *regs);
54asmlinkage void el0t_32_error_handler(struct pt_regs *regs);
55
56asmlinkage void call_on_irq_stack(struct pt_regs *regs,
57 void (*func)(struct pt_regs *));
40asmlinkage void enter_from_user_mode(void);58asmlinkage void enter_from_user_mode(void);
41asmlinkage void exit_to_user_mode(void);59asmlinkage void exit_to_user_mode(void);
42void arm64_enter_nmi(struct pt_regs *regs);
43void arm64_exit_nmi(struct pt_regs *regs);
44void do_mem_abort(unsigned long far, unsigned int esr, struct pt_regs *regs);60void do_mem_abort(unsigned long far, unsigned int esr, struct pt_regs *regs);
45void do_undefinstr(struct pt_regs *regs);61void do_undefinstr(struct pt_regs *regs);
46void do_bti(struct pt_regs *regs);62void do_bti(struct pt_regs *regs);
47asmlinkage void bad_mode(struct pt_regs *regs, int reason, unsigned int esr);
48void do_debug_exception(unsigned long addr_if_watchpoint, unsigned int esr,63void do_debug_exception(unsigned long addr_if_watchpoint, unsigned int esr,
49 struct pt_regs *regs);64 struct pt_regs *regs);
50void do_fpsimd_acc(unsigned int esr, struct pt_regs *regs);65void do_fpsimd_acc(unsigned int esr, struct pt_regs *regs);
@@ -57,4 +72,7 @@ void do_cp15instr(unsigned int esr, struct pt_regs *regs); · 6 unmodified lines
57void do_el0_svc(struct pt_regs *regs);72void do_el0_svc(struct pt_regs *regs);
58void do_el0_svc_compat(struct pt_regs *regs);73void do_el0_svc_compat(struct pt_regs *regs);
59void do_ptrauth_fault(struct pt_regs *regs, unsigned int esr);74void do_ptrauth_fault(struct pt_regs *regs, unsigned int esr);
75void do_serror(struct pt_regs *regs, unsigned int esr);
76
77void panic_bad_stack(struct pt_regs *regs, unsigned int esr, unsigned long far);
60#endif /* __ASM_EXCEPTION_H */78#endif /* __ASM_EXCEPTION_H */
arch/arm64/include/asm/fpsimd.h+1−1
@@ -69,7 +69,7 @@ static inline void *sve_pffr(struct thread_struct *thread) · 68 unmodified lines
69extern void sve_save_state(void *state, u32 *pfpsr);69extern void sve_save_state(void *state, u32 *pfpsr);
70extern void sve_load_state(void const *state, u32 const *pfpsr,70extern void sve_load_state(void const *state, u32 const *pfpsr,
71 unsigned long vq_minus_1);71 unsigned long vq_minus_1);
72extern void sve_flush_live(void);72extern void sve_flush_live(unsigned long vq_minus_1);
73extern void sve_load_from_fpsimd_state(struct user_fpsimd_state const *state,73extern void sve_load_from_fpsimd_state(struct user_fpsimd_state const *state,
74 unsigned long vq_minus_1);74 unsigned long vq_minus_1);
75extern unsigned int sve_get_vl(void);75extern unsigned int sve_get_vl(void);
arch/arm64/include/asm/fpsimdmacros.h+3−1
@@ -213,8 +213,10 @@ · 212 unmodified lines
213 mov v\nz\().16b, v\nz\().16b213 mov v\nz\().16b, v\nz\().16b
214.endm214.endm
215 215
216.macro sve_flush216.macro sve_flush_z
217 _for n, 0, 31, _sve_flush_z \n217 _for n, 0, 31, _sve_flush_z \n
218.endm
219.macro sve_flush_p_ffr
218 _for n, 0, 15, _sve_pfalse \n220 _for n, 0, 15, _sve_pfalse \n
219 _sve_wrffr 0221 _sve_wrffr 0
220.endm222.endm
arch/arm64/include/asm/insn-def.h+9−0
@@ -0,0 +1,9 @@
1/* SPDX-License-Identifier: GPL-2.0-only */
2
3#ifndef __ASM_INSN_DEF_H
4#define __ASM_INSN_DEF_H
5
6/* A64 instructions are always 32 bits. */
7#define AARCH64_INSN_SIZE 4
8
9#endif /* __ASM_INSN_DEF_H */
arch/arm64/include/asm/insn.h+61−6
@@ -10,7 +10,7 @@ · 9 unmodified lines
10#include <linux/build_bug.h>10#include <linux/build_bug.h>
11#include <linux/types.h>11#include <linux/types.h>
12 12
13#include <asm/alternative.h>13#include <asm/insn-def.h>
14 14
15#ifndef __ASSEMBLY__15#ifndef __ASSEMBLY__
16/*16/*
@@ -30,6 +30,7 @@ · 13 unmodified lines
30 */30 */
31enum aarch64_insn_encoding_class {31enum aarch64_insn_encoding_class {
32 AARCH64_INSN_CLS_UNKNOWN, /* UNALLOCATED */32 AARCH64_INSN_CLS_UNKNOWN, /* UNALLOCATED */
33 AARCH64_INSN_CLS_SVE, /* SVE instructions */
33 AARCH64_INSN_CLS_DP_IMM, /* Data processing - immediate */34 AARCH64_INSN_CLS_DP_IMM, /* Data processing - immediate */
34 AARCH64_INSN_CLS_DP_REG, /* Data processing - register */35 AARCH64_INSN_CLS_DP_REG, /* Data processing - register */
35 AARCH64_INSN_CLS_DP_FPSIMD, /* Data processing - SIMD and FP */36 AARCH64_INSN_CLS_DP_FPSIMD, /* Data processing - SIMD and FP */
@@ -294,6 +295,12 @@ __AARCH64_INSN_FUNCS(adr, 0x9F000000, 0x10000000) · 258 unmodified lines
294__AARCH64_INSN_FUNCS(adrp, 0x9F000000, 0x90000000)295__AARCH64_INSN_FUNCS(adrp, 0x9F000000, 0x90000000)
295__AARCH64_INSN_FUNCS(prfm, 0x3FC00000, 0x39800000)296__AARCH64_INSN_FUNCS(prfm, 0x3FC00000, 0x39800000)
296__AARCH64_INSN_FUNCS(prfm_lit, 0xFF000000, 0xD8000000)297__AARCH64_INSN_FUNCS(prfm_lit, 0xFF000000, 0xD8000000)
298__AARCH64_INSN_FUNCS(store_imm, 0x3FC00000, 0x39000000)
299__AARCH64_INSN_FUNCS(load_imm, 0x3FC00000, 0x39400000)
300__AARCH64_INSN_FUNCS(store_pre, 0x3FE00C00, 0x38000C00)
301__AARCH64_INSN_FUNCS(load_pre, 0x3FE00C00, 0x38400C00)
302__AARCH64_INSN_FUNCS(store_post, 0x3FE00C00, 0x38000400)
303__AARCH64_INSN_FUNCS(load_post, 0x3FE00C00, 0x38400400)
297__AARCH64_INSN_FUNCS(str_reg, 0x3FE0EC00, 0x38206800)304__AARCH64_INSN_FUNCS(str_reg, 0x3FE0EC00, 0x38206800)
298__AARCH64_INSN_FUNCS(ldadd, 0x3F20FC00, 0x38200000)305__AARCH64_INSN_FUNCS(ldadd, 0x3F20FC00, 0x38200000)
299__AARCH64_INSN_FUNCS(ldr_reg, 0x3FE0EC00, 0x38606800)306__AARCH64_INSN_FUNCS(ldr_reg, 0x3FE0EC00, 0x38606800)
@@ -302,6 +309,8 @@ __AARCH64_INSN_FUNCS(ldrsw_lit, 0xFF000000, 0x98000000) · 2 unmodified lines
302__AARCH64_INSN_FUNCS(exclusive, 0x3F800000, 0x08000000)309__AARCH64_INSN_FUNCS(exclusive, 0x3F800000, 0x08000000)
303__AARCH64_INSN_FUNCS(load_ex, 0x3F400000, 0x08400000)310__AARCH64_INSN_FUNCS(load_ex, 0x3F400000, 0x08400000)
304__AARCH64_INSN_FUNCS(store_ex, 0x3F400000, 0x08000000)311__AARCH64_INSN_FUNCS(store_ex, 0x3F400000, 0x08000000)
312__AARCH64_INSN_FUNCS(stp, 0x7FC00000, 0x29000000)
313__AARCH64_INSN_FUNCS(ldp, 0x7FC00000, 0x29400000)
305__AARCH64_INSN_FUNCS(stp_post, 0x7FC00000, 0x28800000)314__AARCH64_INSN_FUNCS(stp_post, 0x7FC00000, 0x28800000)
306__AARCH64_INSN_FUNCS(ldp_post, 0x7FC00000, 0x28C00000)315__AARCH64_INSN_FUNCS(ldp_post, 0x7FC00000, 0x28C00000)
307__AARCH64_INSN_FUNCS(stp_pre, 0x7FC00000, 0x29800000)316__AARCH64_INSN_FUNCS(stp_pre, 0x7FC00000, 0x29800000)
@@ -334,6 +343,7 @@ __AARCH64_INSN_FUNCS(rev64, 0x7FFFFC00, 0x5AC00C00) · 26 unmodified lines
334__AARCH64_INSN_FUNCS(and, 0x7F200000, 0x0A000000)343__AARCH64_INSN_FUNCS(and, 0x7F200000, 0x0A000000)
335__AARCH64_INSN_FUNCS(bic, 0x7F200000, 0x0A200000)344__AARCH64_INSN_FUNCS(bic, 0x7F200000, 0x0A200000)
336__AARCH64_INSN_FUNCS(orr, 0x7F200000, 0x2A000000)345__AARCH64_INSN_FUNCS(orr, 0x7F200000, 0x2A000000)
346__AARCH64_INSN_FUNCS(mov_reg, 0x7FE0FFE0, 0x2A0003E0)
337__AARCH64_INSN_FUNCS(orn, 0x7F200000, 0x2A200000)347__AARCH64_INSN_FUNCS(orn, 0x7F200000, 0x2A200000)
338__AARCH64_INSN_FUNCS(eor, 0x7F200000, 0x4A000000)348__AARCH64_INSN_FUNCS(eor, 0x7F200000, 0x4A000000)
339__AARCH64_INSN_FUNCS(eon, 0x7F200000, 0x4A200000)349__AARCH64_INSN_FUNCS(eon, 0x7F200000, 0x4A200000)
@@ -368,6 +378,14 @@ __AARCH64_INSN_FUNCS(eret_auth, 0xFFFFFBFF, 0xD69F0BFF) · 28 unmodified lines
368__AARCH64_INSN_FUNCS(mrs, 0xFFF00000, 0xD5300000)378__AARCH64_INSN_FUNCS(mrs, 0xFFF00000, 0xD5300000)
369__AARCH64_INSN_FUNCS(msr_imm, 0xFFF8F01F, 0xD500401F)379__AARCH64_INSN_FUNCS(msr_imm, 0xFFF8F01F, 0xD500401F)
370__AARCH64_INSN_FUNCS(msr_reg, 0xFFF00000, 0xD5100000)380__AARCH64_INSN_FUNCS(msr_reg, 0xFFF00000, 0xD5100000)
381__AARCH64_INSN_FUNCS(dmb, 0xFFFFF0FF, 0xD50330BF)
382__AARCH64_INSN_FUNCS(dsb_base, 0xFFFFF0FF, 0xD503309F)
383__AARCH64_INSN_FUNCS(dsb_nxs, 0xFFFFF3FF, 0xD503323F)
384__AARCH64_INSN_FUNCS(isb, 0xFFFFF0FF, 0xD50330DF)
385__AARCH64_INSN_FUNCS(sb, 0xFFFFFFFF, 0xD50330FF)
386__AARCH64_INSN_FUNCS(clrex, 0xFFFFF0FF, 0xD503305F)
387__AARCH64_INSN_FUNCS(ssbb, 0xFFFFFFFF, 0xD503309F)
388__AARCH64_INSN_FUNCS(pssbb, 0xFFFFFFFF, 0xD503349F)
371 389
372#undef __AARCH64_INSN_FUNCS390#undef __AARCH64_INSN_FUNCS
373 391
@@ -379,8 +397,47 @@ static inline bool aarch64_insn_is_adr_adrp(u32 insn) · 5 unmodified lines
379 return aarch64_insn_is_adr(insn) || aarch64_insn_is_adrp(insn);397 return aarch64_insn_is_adr(insn) || aarch64_insn_is_adrp(insn);
380}398}
381 399
382int aarch64_insn_read(void *addr, u32 *insnp);400static inline bool aarch64_insn_is_dsb(u32 insn)
383int aarch64_insn_write(void *addr, u32 insn);401{
402 return aarch64_insn_is_dsb_base(insn) || aarch64_insn_is_dsb_nxs(insn);
403}
404
405static inline bool aarch64_insn_is_barrier(u32 insn)
406{
407 return aarch64_insn_is_dmb(insn) || aarch64_insn_is_dsb(insn) ||
408 aarch64_insn_is_isb(insn) || aarch64_insn_is_sb(insn) ||
409 aarch64_insn_is_clrex(insn) || aarch64_insn_is_ssbb(insn) ||
410 aarch64_insn_is_pssbb(insn);
411}
412
413static inline bool aarch64_insn_is_store_single(u32 insn)
414{
415 return aarch64_insn_is_store_imm(insn) ||
416 aarch64_insn_is_store_pre(insn) ||
417 aarch64_insn_is_store_post(insn);
418}
419
420static inline bool aarch64_insn_is_store_pair(u32 insn)
421{
422 return aarch64_insn_is_stp(insn) ||
423 aarch64_insn_is_stp_pre(insn) ||
424 aarch64_insn_is_stp_post(insn);
425}
426
427static inline bool aarch64_insn_is_load_single(u32 insn)
428{
429 return aarch64_insn_is_load_imm(insn) ||
430 aarch64_insn_is_load_pre(insn) ||
431 aarch64_insn_is_load_post(insn);
432}
433
434static inline bool aarch64_insn_is_load_pair(u32 insn)
435{
436 return aarch64_insn_is_ldp(insn) ||
437 aarch64_insn_is_ldp_pre(insn) ||
438 aarch64_insn_is_ldp_post(insn);
439}
440
384enum aarch64_insn_encoding_class aarch64_get_insn_class(u32 insn);441enum aarch64_insn_encoding_class aarch64_get_insn_class(u32 insn);
385bool aarch64_insn_uses_literal(u32 insn);442bool aarch64_insn_uses_literal(u32 insn);
386bool aarch64_insn_is_branch(u32 insn);443bool aarch64_insn_is_branch(u32 insn);
@@ -487,9 +544,6 @@ u32 aarch64_insn_gen_prefetch(enum aarch64_insn_register base, · 100 unmodified lines
487s32 aarch64_get_branch_offset(u32 insn);544s32 aarch64_get_branch_offset(u32 insn);
488u32 aarch64_set_branch_offset(u32 insn, s32 offset);545u32 aarch64_set_branch_offset(u32 insn, s32 offset);
489 546
490int aarch64_insn_patch_text_nosync(void *addr, u32 insn);
491int aarch64_insn_patch_text(void *addrs[], u32 insns[], int cnt);
492
493s32 aarch64_insn_adrp_get_offset(u32 insn);547s32 aarch64_insn_adrp_get_offset(u32 insn);
494u32 aarch64_insn_adrp_set_offset(u32 insn, s32 offset);548u32 aarch64_insn_adrp_set_offset(u32 insn, s32 offset);
495 549
@@ -506,6 +560,7 @@ u32 aarch32_insn_mcr_extract_crm(u32 insn); · 10 unmodified lines
506 560
507typedef bool (pstate_check_t)(unsigned long);561typedef bool (pstate_check_t)(unsigned long);
508extern pstate_check_t * const aarch32_opcode_cond_checks[16];562extern pstate_check_t * const aarch32_opcode_cond_checks[16];
563
509#endif /* __ASSEMBLY__ */564#endif /* __ASSEMBLY__ */
510 565
511#endif /* __ASM_INSN_H */566#endif /* __ASM_INSN_H */
arch/arm64/include/asm/kernel-pgtable.h+8−11
@@ -18,9 +18,9 @@ · 17 unmodified lines
18 * 64K (section size = 512M).18 * 64K (section size = 512M).
19 */19 */
20#ifdef CONFIG_ARM64_4K_PAGES20#ifdef CONFIG_ARM64_4K_PAGES
21#define ARM64_SWAPPER_USES_SECTION_MAPS 121#define ARM64_KERNEL_USES_PMD_MAPS 1
22#else22#else
23#define ARM64_SWAPPER_USES_SECTION_MAPS 023#define ARM64_KERNEL_USES_PMD_MAPS 0
24#endif24#endif
25 25
26/*26/*
@@ -33,7 +33,7 @@ · 6 unmodified lines
33 * VA range, so pages required to map highest possible PA are reserved in all33 * VA range, so pages required to map highest possible PA are reserved in all
34 * cases.34 * cases.
35 */35 */
36#if ARM64_SWAPPER_USES_SECTION_MAPS36#if ARM64_KERNEL_USES_PMD_MAPS
37#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - 1)37#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - 1)
38#define IDMAP_PGTABLE_LEVELS (ARM64_HW_PGTABLE_LEVELS(PHYS_MASK_SHIFT) - 1)38#define IDMAP_PGTABLE_LEVELS (ARM64_HW_PGTABLE_LEVELS(PHYS_MASK_SHIFT) - 1)
39#else39#else
@@ -90,9 +90,9 @@ · 50 unmodified lines
90#define IDMAP_DIR_SIZE (IDMAP_PGTABLE_LEVELS * PAGE_SIZE)90#define IDMAP_DIR_SIZE (IDMAP_PGTABLE_LEVELS * PAGE_SIZE)
91 91
92/* Initial memory map size */92/* Initial memory map size */
93#if ARM64_SWAPPER_USES_SECTION_MAPS93#if ARM64_KERNEL_USES_PMD_MAPS
94#define SWAPPER_BLOCK_SHIFT SECTION_SHIFT94#define SWAPPER_BLOCK_SHIFT PMD_SHIFT
95#define SWAPPER_BLOCK_SIZE SECTION_SIZE95#define SWAPPER_BLOCK_SIZE PMD_SIZE
96#define SWAPPER_TABLE_SHIFT PUD_SHIFT96#define SWAPPER_TABLE_SHIFT PUD_SHIFT
97#else97#else
98#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT98#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT
@@ -100,16 +100,13 @@ · 1 unmodified line
100#define SWAPPER_TABLE_SHIFT PMD_SHIFT100#define SWAPPER_TABLE_SHIFT PMD_SHIFT
101#endif101#endif
102 102
103/* The size of the initial kernel direct mapping */
104#define SWAPPER_INIT_MAP_SIZE (_AC(1, UL) << SWAPPER_TABLE_SHIFT)
105
106/*103/*
107 * Initial memory map attributes.104 * Initial memory map attributes.
108 */105 */
109#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)106#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)
110#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)107#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)
111 108
112#if ARM64_SWAPPER_USES_SECTION_MAPS109#if ARM64_KERNEL_USES_PMD_MAPS
113#define SWAPPER_MM_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS)110#define SWAPPER_MM_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS)
114#else111#else
115#define SWAPPER_MM_MMUFLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS)112#define SWAPPER_MM_MMUFLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS)
@@ -125,7 +122,7 @@ · 9 unmodified lines
125#if defined(CONFIG_ARM64_4K_PAGES)122#if defined(CONFIG_ARM64_4K_PAGES)
126#define ARM64_MEMSTART_SHIFT PUD_SHIFT123#define ARM64_MEMSTART_SHIFT PUD_SHIFT
127#elif defined(CONFIG_ARM64_16K_PAGES)124#elif defined(CONFIG_ARM64_16K_PAGES)
128#define ARM64_MEMSTART_SHIFT (PMD_SHIFT + 5)125#define ARM64_MEMSTART_SHIFT CONT_PMD_SHIFT
129#else126#else
130#define ARM64_MEMSTART_SHIFT PMD_SHIFT127#define ARM64_MEMSTART_SHIFT PMD_SHIFT
131#endif128#endif
arch/arm64/include/asm/kvm_asm.h+1−0
@@ -8,6 +8,7 @@ · 7 unmodified lines
8#define __ARM_KVM_ASM_H__8#define __ARM_KVM_ASM_H__
9 9
10#include <asm/hyp_image.h>10#include <asm/hyp_image.h>
11#include <asm/insn.h>
11#include <asm/virt.h>12#include <asm/virt.h>
12 13
13#define ARM_EXIT_WITH_SERROR_BIT 3114#define ARM_EXIT_WITH_SERROR_BIT 31
arch/arm64/include/asm/kvm_mmu.h+4−3
@@ -180,7 +180,8 @@ static inline void *__kvm_vector_slot2addr(void *base, · 179 unmodified lines
180 180
181struct kvm;181struct kvm;
182 182
183#define kvm_flush_dcache_to_poc(a,l) __flush_dcache_area((a), (l))183#define kvm_flush_dcache_to_poc(a,l) \
184 dcache_clean_inval_poc((unsigned long)(a), (unsigned long)(a)+(l))
184 185
185static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu)186static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu)
186{187{
@@ -208,12 +209,12 @@ static inline void __invalidate_icache_guest_page(kvm_pfn_t pfn, · 21 unmodified lines
208{209{
209 if (icache_is_aliasing()) {210 if (icache_is_aliasing()) {
210 /* any kind of VIPT cache */211 /* any kind of VIPT cache */
211 __flush_icache_all();212 icache_inval_all_pou();
212 } else if (is_kernel_in_hyp_mode() || !icache_is_vpipt()) {213 } else if (is_kernel_in_hyp_mode() || !icache_is_vpipt()) {
213 /* PIPT or VPIPT at EL2 (see comment in __kvm_tlb_flush_vmid_ipa) */214 /* PIPT or VPIPT at EL2 (see comment in __kvm_tlb_flush_vmid_ipa) */
214 void *va = page_address(pfn_to_page(pfn));215 void *va = page_address(pfn_to_page(pfn));
215 216
216 invalidate_icache_range((unsigned long)va,217 icache_inval_pou((unsigned long)va,
217 (unsigned long)va + size);218 (unsigned long)va + size);
218 }219 }
219}220}
arch/arm64/include/asm/linkage.h+8−0
@@ -56,8 +56,16 @@ · 55 unmodified lines
56 SYM_FUNC_START_ALIAS(__pi_##x); \56 SYM_FUNC_START_ALIAS(__pi_##x); \
57 SYM_FUNC_START_WEAK(x)57 SYM_FUNC_START_WEAK(x)
58 58
59#define SYM_FUNC_START_WEAK_ALIAS_PI(x) \
60 SYM_FUNC_START_ALIAS(__pi_##x); \
61 SYM_START(x, SYM_L_WEAK, SYM_A_ALIGN)
62
59#define SYM_FUNC_END_PI(x) \63#define SYM_FUNC_END_PI(x) \
60 SYM_FUNC_END(x); \64 SYM_FUNC_END(x); \
61 SYM_FUNC_END_ALIAS(__pi_##x)65 SYM_FUNC_END_ALIAS(__pi_##x)
62 66
67#define SYM_FUNC_END_ALIAS_PI(x) \
68 SYM_FUNC_END_ALIAS(x); \
69 SYM_FUNC_END_ALIAS(__pi_##x)
70
63#endif71#endif
arch/arm64/include/asm/memory.h+2−4
@@ -135,10 +135,8 @@ · 134 unmodified lines
135#define MT_NORMAL 0135#define MT_NORMAL 0
136#define MT_NORMAL_TAGGED 1136#define MT_NORMAL_TAGGED 1
137#define MT_NORMAL_NC 2137#define MT_NORMAL_NC 2
138#define MT_NORMAL_WT 3138#define MT_DEVICE_nGnRnE 3
139#define MT_DEVICE_nGnRnE 4139#define MT_DEVICE_nGnRE 4
140#define MT_DEVICE_nGnRE 5
141#define MT_DEVICE_GRE 6
142 140
143/*141/*
144 * Memory types for Stage-2 translation142 * Memory types for Stage-2 translation
arch/arm64/include/asm/mmu_context.h+2−2
@@ -177,9 +177,9 @@ static inline void update_saved_ttbr0(struct task_struct *tsk, · 176 unmodified lines
177 return;177 return;
178 178
179 if (mm == &init_mm)179 if (mm == &init_mm)
180 ttbr = __pa_symbol(reserved_pg_dir);180 ttbr = phys_to_ttbr(__pa_symbol(reserved_pg_dir));
181 else181 else
182 ttbr = virt_to_phys(mm->pgd) | ASID(mm) << 48;182 ttbr = phys_to_ttbr(virt_to_phys(mm->pgd)) | ASID(mm) << 48;
183 183
184 WRITE_ONCE(task_thread_info(tsk)->ttbr0, ttbr);184 WRITE_ONCE(task_thread_info(tsk)->ttbr0, ttbr);
185}185}
arch/arm64/include/asm/module.lds.h+15−2
@@ -1,7 +1,20 @@
1#ifdef CONFIG_ARM64_MODULE_PLTS
2SECTIONS {1SECTIONS {
2#ifdef CONFIG_ARM64_MODULE_PLTS
3 .plt 0 (NOLOAD) : { BYTE(0) }3 .plt 0 (NOLOAD) : { BYTE(0) }
4 .init.plt 0 (NOLOAD) : { BYTE(0) }4 .init.plt 0 (NOLOAD) : { BYTE(0) }
5 .text.ftrace_trampoline 0 (NOLOAD) : { BYTE(0) }5 .text.ftrace_trampoline 0 (NOLOAD) : { BYTE(0) }
6}
7#endif6#endif
7
8#ifdef CONFIG_KASAN_SW_TAGS
9 /*
10 * Outlined checks go into comdat-deduplicated sections named .text.hot.
11 * Because they are in comdats they are not combined by the linker and
12 * we otherwise end up with multiple sections with the same .text.hot
13 * name in the .ko file. The kernel module loader warns if it sees
14 * multiple sections with the same name so we use this sections
15 * directive to force them into a single section and silence the
16 * warning.
17 */
18 .text.hot : { *(.text.hot) }
19#endif
20}
arch/arm64/include/asm/mte-kasan.h+67−26
@@ -48,43 +48,84 @@ static inline u8 mte_get_random_tag(void) · 47 unmodified lines
48 return mte_get_ptr_tag(addr);48 return mte_get_ptr_tag(addr);
49}49}
50 50
51static inline u64 __stg_post(u64 p)
52{
53 asm volatile(__MTE_PREAMBLE "stg %0, [%0], #16"
54 : "+r"(p)
55 :
56 : "memory");
57 return p;
58}
59
60static inline u64 __stzg_post(u64 p)
61{
62 asm volatile(__MTE_PREAMBLE "stzg %0, [%0], #16"
63 : "+r"(p)
64 :
65 : "memory");
66 return p;
67}
68
69static inline void __dc_gva(u64 p)
70{
71 asm volatile(__MTE_PREAMBLE "dc gva, %0" : : "r"(p) : "memory");
72}
73
74static inline void __dc_gzva(u64 p)
75{
76 asm volatile(__MTE_PREAMBLE "dc gzva, %0" : : "r"(p) : "memory");
77}
78
51/*79/*
52 * Assign allocation tags for a region of memory based on the pointer tag.80 * Assign allocation tags for a region of memory based on the pointer tag.
53 * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and81 * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and
54 * size must be non-zero and MTE_GRANULE_SIZE aligned.82 * size must be MTE_GRANULE_SIZE aligned.
55 */83 */
56static inline void mte_set_mem_tag_range(void *addr, size_t size,84static inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag,
57 u8 tag, bool init)85 bool init)
58{86{
59 u64 curr, end;87 u64 curr, mask, dczid_bs, end1, end2, end3;
60 88
61 if (!size)89 /* Read DC G(Z)VA block size from the system register. */
62 return;90 dczid_bs = 4ul << (read_cpuid(DCZID_EL0) & 0xf);
63 91
64 curr = (u64)__tag_set(addr, tag);92 curr = (u64)__tag_set(addr, tag);
65 end = curr + size;93 mask = dczid_bs - 1;
94 /* STG/STZG up to the end of the first block. */
95 end1 = curr | mask;
96 end3 = curr + size;
97 /* DC GVA / GZVA in [end1, end2) */
98 end2 = end3 & ~mask;
66 99
67 /*100 /*
68 * 'asm volatile' is required to prevent the compiler to move101 * The following code uses STG on the first DC GVA block even if the
69 * the statement outside of the loop.102 * start address is aligned - it appears to be faster than an alignment
103 * check + conditional branch. Also, if the range size is at least 2 DC
104 * GVA blocks, the first two loops can use post-condition to save one
105 * branch each.
70 */106 */
71 if (init) {107#define SET_MEMTAG_RANGE(stg_post, dc_gva) \
72 do {108 do { \
73 asm volatile(__MTE_PREAMBLE "stzg %0, [%0]"109 if (size >= 2 * dczid_bs) { \
74 :110 do { \
75 : "r" (curr)111 curr = stg_post(curr); \
76 : "memory");112 } while (curr < end1); \
77 curr += MTE_GRANULE_SIZE;113 \
78 } while (curr != end);114 do { \
79 } else {115 dc_gva(curr); \
80 do {116 curr += dczid_bs; \
81 asm volatile(__MTE_PREAMBLE "stg %0, [%0]"117 } while (curr < end2); \
82 :118 } \
83 : "r" (curr)119 \
84 : "memory");120 while (curr < end3) \
85 curr += MTE_GRANULE_SIZE;121 curr = stg_post(curr); \
86 } while (curr != end);122 } while (0)
87 }123
124 if (init)
125 SET_MEMTAG_RANGE(__stzg_post, __dc_gzva);
126 else
127 SET_MEMTAG_RANGE(__stg_post, __dc_gva);
128#undef SET_MEMTAG_RANGE
88}129}
89 130
90void mte_enable_kernel_sync(void);131void mte_enable_kernel_sync(void);
arch/arm64/include/asm/mte.h+4−0
@@ -37,6 +37,7 @@ void mte_free_tag_storage(char *storage); · 36 unmodified lines
37/* track which pages have valid allocation tags */37/* track which pages have valid allocation tags */
38#define PG_mte_tagged PG_arch_238#define PG_mte_tagged PG_arch_2
39 39
40void mte_zero_clear_page_tags(void *addr);
40void mte_sync_tags(pte_t *ptep, pte_t pte);41void mte_sync_tags(pte_t *ptep, pte_t pte);
41void mte_copy_page_tags(void *kto, const void *kfrom);42void mte_copy_page_tags(void *kto, const void *kfrom);
42void mte_thread_init_user(void);43void mte_thread_init_user(void);
@@ -53,6 +54,9 @@ int mte_ptrace_copy_tags(struct task_struct *child, long request, · 10 unmodified lines
53/* unused if !CONFIG_ARM64_MTE, silence the compiler */54/* unused if !CONFIG_ARM64_MTE, silence the compiler */
54#define PG_mte_tagged 055#define PG_mte_tagged 0
55 56
57static inline void mte_zero_clear_page_tags(void *addr)
58{
59}
56static inline void mte_sync_tags(pte_t *ptep, pte_t pte)60static inline void mte_sync_tags(pte_t *ptep, pte_t pte)
57{61{
58}62}
arch/arm64/include/asm/page.h+7−3
@@ -13,6 +13,7 @@ · 12 unmodified lines
13#ifndef __ASSEMBLY__13#ifndef __ASSEMBLY__
14 14
15#include <linux/personality.h> /* for READ_IMPLIES_EXEC */15#include <linux/personality.h> /* for READ_IMPLIES_EXEC */
16#include <linux/types.h> /* for gfp_t */
16#include <asm/pgtable-types.h>17#include <asm/pgtable-types.h>
17 18
18struct page;19struct page;
@@ -28,9 +29,12 @@ void copy_user_highpage(struct page *to, struct page *from, · 9 unmodified lines
28void copy_highpage(struct page *to, struct page *from);29void copy_highpage(struct page *to, struct page *from);
29#define __HAVE_ARCH_COPY_HIGHPAGE30#define __HAVE_ARCH_COPY_HIGHPAGE
30 31
31#define __alloc_zeroed_user_highpage(movableflags, vma, vaddr) \32struct page *alloc_zeroed_user_highpage_movable(struct vm_area_struct *vma,
32 alloc_page_vma(GFP_HIGHUSER | __GFP_ZERO | movableflags, vma, vaddr)33 unsigned long vaddr);
33#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE34#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE
35
36void tag_clear_highpage(struct page *to);
37#define __HAVE_ARCH_TAG_CLEAR_HIGHPAGE
34 38
35#define clear_user_page(page, vaddr, pg) clear_page(page)39#define clear_user_page(page, vaddr, pg) clear_page(page)
36#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)40#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
arch/arm64/include/asm/patching.h+13−0
@@ -0,0 +1,13 @@
1/* SPDX-License-Identifier: GPL-2.0-only */
2#ifndef __ASM_PATCHING_H
3#define __ASM_PATCHING_H
4
5#include <linux/types.h>
6
7int aarch64_insn_read(void *addr, u32 *insnp);
8int aarch64_insn_write(void *addr, u32 insn);
9
10int aarch64_insn_patch_text_nosync(void *addr, u32 insn);
11int aarch64_insn_patch_text(void *addrs[], u32 insns[], int cnt);
12
13#endif /* __ASM_PATCHING_H */
arch/arm64/include/asm/perf_event.h+5−0
@@ -239,6 +239,11 @@ · 238 unmodified lines
239/* PMMIR_EL1.SLOTS mask */239/* PMMIR_EL1.SLOTS mask */
240#define ARMV8_PMU_SLOTS_MASK 0xff240#define ARMV8_PMU_SLOTS_MASK 0xff
241 241
242#define ARMV8_PMU_BUS_SLOTS_SHIFT 8
243#define ARMV8_PMU_BUS_SLOTS_MASK 0xff
244#define ARMV8_PMU_BUS_WIDTH_SHIFT 16
245#define ARMV8_PMU_BUS_WIDTH_MASK 0xf
246
242#ifdef CONFIG_PERF_EVENTS247#ifdef CONFIG_PERF_EVENTS
243struct pt_regs;248struct pt_regs;
244extern unsigned long perf_instruction_pointer(struct pt_regs *regs);249extern unsigned long perf_instruction_pointer(struct pt_regs *regs);
arch/arm64/include/asm/pgtable-hwdef.h+0−7
@@ -71,13 +71,6 @@ · 70 unmodified lines
71#define PGDIR_MASK (~(PGDIR_SIZE-1))71#define PGDIR_MASK (~(PGDIR_SIZE-1))
72#define PTRS_PER_PGD (1 << (VA_BITS - PGDIR_SHIFT))72#define PTRS_PER_PGD (1 << (VA_BITS - PGDIR_SHIFT))
73 73
74/*
75 * Section address mask and size definitions.
76 */
77#define SECTION_SHIFT PMD_SHIFT
78#define SECTION_SIZE (_AC(1, UL) << SECTION_SHIFT)
79#define SECTION_MASK (~(SECTION_SIZE-1))
80
81/*74/*
82 * Contiguous page definitions.75 * Contiguous page definitions.
83 */76 */
arch/arm64/include/asm/pgtable-prot.h+0−1
@@ -55,7 +55,6 @@ extern bool arm64_use_ng_mappings; · 54 unmodified lines
55#define PROT_DEVICE_nGnRnE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRnE))55#define PROT_DEVICE_nGnRnE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRnE))
56#define PROT_DEVICE_nGnRE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRE))56#define PROT_DEVICE_nGnRE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRE))
57#define PROT_NORMAL_NC (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_NC))57#define PROT_NORMAL_NC (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_NC))
58#define PROT_NORMAL_WT (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_WT))
59#define PROT_NORMAL (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL))58#define PROT_NORMAL (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL))
60#define PROT_NORMAL_TAGGED (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_TAGGED))59#define PROT_NORMAL_TAGGED (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_TAGGED))
61 60
arch/arm64/include/asm/pgtable.h+2−3
@@ -511,13 +511,12 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, · 510 unmodified lines
511 511
512#define pmd_none(pmd) (!pmd_val(pmd))512#define pmd_none(pmd) (!pmd_val(pmd))
513 513
514#define pmd_bad(pmd) (!(pmd_val(pmd) & PMD_TABLE_BIT))
515
516#define pmd_table(pmd) ((pmd_val(pmd) & PMD_TYPE_MASK) == \514#define pmd_table(pmd) ((pmd_val(pmd) & PMD_TYPE_MASK) == \
517 PMD_TYPE_TABLE)515 PMD_TYPE_TABLE)
518#define pmd_sect(pmd) ((pmd_val(pmd) & PMD_TYPE_MASK) == \516#define pmd_sect(pmd) ((pmd_val(pmd) & PMD_TYPE_MASK) == \
519 PMD_TYPE_SECT)517 PMD_TYPE_SECT)
520#define pmd_leaf(pmd) pmd_sect(pmd)518#define pmd_leaf(pmd) pmd_sect(pmd)
519#define pmd_bad(pmd) (!pmd_table(pmd))
521 520
522#define pmd_leaf_size(pmd) (pmd_cont(pmd) ? CONT_PMD_SIZE : PMD_SIZE)521#define pmd_leaf_size(pmd) (pmd_cont(pmd) ? CONT_PMD_SIZE : PMD_SIZE)
523#define pte_leaf_size(pte) (pte_cont(pte) ? CONT_PTE_SIZE : PAGE_SIZE)522#define pte_leaf_size(pte) (pte_cont(pte) ? CONT_PTE_SIZE : PAGE_SIZE)
@@ -604,7 +603,7 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) · 80 unmodified lines
604 pr_err("%s:%d: bad pmd %016llx.\n", __FILE__, __LINE__, pmd_val(e))603 pr_err("%s:%d: bad pmd %016llx.\n", __FILE__, __LINE__, pmd_val(e))
605 604
606#define pud_none(pud) (!pud_val(pud))605#define pud_none(pud) (!pud_val(pud))
607#define pud_bad(pud) (!(pud_val(pud) & PUD_TABLE_BIT))606#define pud_bad(pud) (!pud_table(pud))
608#define pud_present(pud) pte_present(pud_pte(pud))607#define pud_present(pud) pte_present(pud_pte(pud))
609#define pud_leaf(pud) pud_sect(pud)608#define pud_leaf(pud) pud_sect(pud)
610#define pud_valid(pud) pte_valid(pud_pte(pud))609#define pud_valid(pud) pte_valid(pud_pte(pud))
arch/arm64/include/asm/pointer_auth.h+33−26
@@ -31,10 +31,6 @@ struct ptrauth_keys_user { · 30 unmodified lines
31 struct ptrauth_key apga;31 struct ptrauth_key apga;
32};32};
33 33
34struct ptrauth_keys_kernel {
35 struct ptrauth_key apia;
36};
37
38#define __ptrauth_key_install_nosync(k, v) \34#define __ptrauth_key_install_nosync(k, v) \
39do { \35do { \
40 struct ptrauth_key __pki_v = (v); \36 struct ptrauth_key __pki_v = (v); \
@@ -42,6 +38,29 @@ do { \ · 1 unmodified line
42 write_sysreg_s(__pki_v.hi, SYS_ ## k ## KEYHI_EL1); \38 write_sysreg_s(__pki_v.hi, SYS_ ## k ## KEYHI_EL1); \
43} while (0)39} while (0)
44 40
41#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL
42
43struct ptrauth_keys_kernel {
44 struct ptrauth_key apia;
45};
46
47static __always_inline void ptrauth_keys_init_kernel(struct ptrauth_keys_kernel *keys)
48{
49 if (system_supports_address_auth())
50 get_random_bytes(&keys->apia, sizeof(keys->apia));
51}
52
53static __always_inline void ptrauth_keys_switch_kernel(struct ptrauth_keys_kernel *keys)
54{
55 if (!system_supports_address_auth())
56 return;
57
58 __ptrauth_key_install_nosync(APIA, keys->apia);
59 isb();
60}
61
62#endif /* CONFIG_ARM64_PTR_AUTH_KERNEL */
63
45static inline void ptrauth_keys_install_user(struct ptrauth_keys_user *keys)64static inline void ptrauth_keys_install_user(struct ptrauth_keys_user *keys)
46{65{
47 if (system_supports_address_auth()) {66 if (system_supports_address_auth()) {
@@ -69,21 +88,6 @@ static inline void ptrauth_keys_init_user(struct ptrauth_keys_user *keys) · 21 unmodified lines
69 ptrauth_keys_install_user(keys);88 ptrauth_keys_install_user(keys);
70}89}
71 90
72static __always_inline void ptrauth_keys_init_kernel(struct ptrauth_keys_kernel *keys)
73{
74 if (system_supports_address_auth())
75 get_random_bytes(&keys->apia, sizeof(keys->apia));
76}
77
78static __always_inline void ptrauth_keys_switch_kernel(struct ptrauth_keys_kernel *keys)
79{
80 if (!system_supports_address_auth())
81 return;
82
83 __ptrauth_key_install_nosync(APIA, keys->apia);
84 isb();
85}
86
87extern int ptrauth_prctl_reset_keys(struct task_struct *tsk, unsigned long arg);91extern int ptrauth_prctl_reset_keys(struct task_struct *tsk, unsigned long arg);
88 92
89extern int ptrauth_set_enabled_keys(struct task_struct *tsk, unsigned long keys,93extern int ptrauth_set_enabled_keys(struct task_struct *tsk, unsigned long keys,
@@ -121,11 +125,6 @@ static __always_inline void ptrauth_enable(void) · 31 unmodified lines
121#define ptrauth_thread_switch_user(tsk) \125#define ptrauth_thread_switch_user(tsk) \
122 ptrauth_keys_install_user(&(tsk)->thread.keys_user)126 ptrauth_keys_install_user(&(tsk)->thread.keys_user)
123 127
124#define ptrauth_thread_init_kernel(tsk) \
125 ptrauth_keys_init_kernel(&(tsk)->thread.keys_kernel)
126#define ptrauth_thread_switch_kernel(tsk) \
127 ptrauth_keys_switch_kernel(&(tsk)->thread.keys_kernel)
128
129#else /* CONFIG_ARM64_PTR_AUTH */128#else /* CONFIG_ARM64_PTR_AUTH */
130#define ptrauth_enable()129#define ptrauth_enable()
131#define ptrauth_prctl_reset_keys(tsk, arg) (-EINVAL)130#define ptrauth_prctl_reset_keys(tsk, arg) (-EINVAL)
@@ -134,11 +133,19 @@ static __always_inline void ptrauth_enable(void) · 2 unmodified lines
134#define ptrauth_strip_insn_pac(lr) (lr)133#define ptrauth_strip_insn_pac(lr) (lr)
135#define ptrauth_suspend_exit()134#define ptrauth_suspend_exit()
136#define ptrauth_thread_init_user()135#define ptrauth_thread_init_user()
137#define ptrauth_thread_init_kernel(tsk)
138#define ptrauth_thread_switch_user(tsk)136#define ptrauth_thread_switch_user(tsk)
139#define ptrauth_thread_switch_kernel(tsk)
140#endif /* CONFIG_ARM64_PTR_AUTH */137#endif /* CONFIG_ARM64_PTR_AUTH */
141 138
139#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL
140#define ptrauth_thread_init_kernel(tsk) \
141 ptrauth_keys_init_kernel(&(tsk)->thread.keys_kernel)
142#define ptrauth_thread_switch_kernel(tsk) \
143 ptrauth_keys_switch_kernel(&(tsk)->thread.keys_kernel)
144#else
145#define ptrauth_thread_init_kernel(tsk)
146#define ptrauth_thread_switch_kernel(tsk)
147#endif /* CONFIG_ARM64_PTR_AUTH_KERNEL */
148
142#define PR_PAC_ENABLED_KEYS_MASK \149#define PR_PAC_ENABLED_KEYS_MASK \
143 (PR_PAC_APIAKEY | PR_PAC_APIBKEY | PR_PAC_APDAKEY | PR_PAC_APDBKEY)150 (PR_PAC_APIAKEY | PR_PAC_APIBKEY | PR_PAC_APDAKEY | PR_PAC_APDBKEY)
144 151
arch/arm64/include/asm/processor.h+8−8
@@ -148,8 +148,10 @@ struct thread_struct { · 147 unmodified lines
148 struct debug_info debug; /* debugging */148 struct debug_info debug; /* debugging */
149#ifdef CONFIG_ARM64_PTR_AUTH149#ifdef CONFIG_ARM64_PTR_AUTH
150 struct ptrauth_keys_user keys_user;150 struct ptrauth_keys_user keys_user;
151#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL
151 struct ptrauth_keys_kernel keys_kernel;152 struct ptrauth_keys_kernel keys_kernel;
152#endif153#endif
154#endif
153#ifdef CONFIG_ARM64_MTE155#ifdef CONFIG_ARM64_MTE
154 u64 gcr_user_excl;156 u64 gcr_user_excl;
155#endif157#endif
@@ -257,8 +259,6 @@ void set_task_sctlr_el1(u64 sctlr); · 101 unmodified lines
257extern struct task_struct *cpu_switch_to(struct task_struct *prev,259extern struct task_struct *cpu_switch_to(struct task_struct *prev,
258 struct task_struct *next);260 struct task_struct *next);
259 261
260asmlinkage void arm64_preempt_schedule_irq(void);
261
262#define task_pt_regs(p) \262#define task_pt_regs(p) \
263 ((struct pt_regs *)(THREAD_SIZE + task_stack_page(p)) - 1)263 ((struct pt_regs *)(THREAD_SIZE + task_stack_page(p)) - 1)
264 264
@@ -329,13 +329,13 @@ long get_tagged_addr_ctrl(struct task_struct *task); · 64 unmodified lines
329 * of header definitions for the use of task_stack_page.329 * of header definitions for the use of task_stack_page.
330 */330 */
331 331
332#define current_top_of_stack() \332#define current_top_of_stack() \
333({ \333({ \
334 struct stack_info _info; \334 struct stack_info _info; \
335 BUG_ON(!on_accessible_stack(current, current_stack_pointer, &_info)); \335 BUG_ON(!on_accessible_stack(current, current_stack_pointer, 1, &_info)); \
336 _info.high; \336 _info.high; \
337})337})
338#define on_thread_stack() (on_task_stack(current, current_stack_pointer, NULL))338#define on_thread_stack() (on_task_stack(current, current_stack_pointer, 1, NULL))
339 339
340#endif /* __ASSEMBLY__ */340#endif /* __ASSEMBLY__ */
341#endif /* __ASM_PROCESSOR_H */341#endif /* __ASM_PROCESSOR_H */
arch/arm64/include/asm/scs.h+4−4
@@ -9,18 +9,18 @@ · 8 unmodified lines
9#ifdef CONFIG_SHADOW_CALL_STACK9#ifdef CONFIG_SHADOW_CALL_STACK
10 scs_sp .req x1810 scs_sp .req x18
11 11
12 .macro scs_load tsk, tmp12 .macro scs_load tsk
13 ldr scs_sp, [\tsk, #TSK_TI_SCS_SP]13 ldr scs_sp, [\tsk, #TSK_TI_SCS_SP]
14 .endm14 .endm
15 15
16 .macro scs_save tsk, tmp16 .macro scs_save tsk
17 str scs_sp, [\tsk, #TSK_TI_SCS_SP]17 str scs_sp, [\tsk, #TSK_TI_SCS_SP]
18 .endm18 .endm
19#else19#else
20 .macro scs_load tsk, tmp20 .macro scs_load tsk
21 .endm21 .endm
22 22
23 .macro scs_save tsk, tmp23 .macro scs_save tsk
24 .endm24 .endm
25#endif /* CONFIG_SHADOW_CALL_STACK */25#endif /* CONFIG_SHADOW_CALL_STACK */
26 26
arch/arm64/include/asm/sdei.h+7−3
@@ -37,13 +37,17 @@ struct sdei_registered_event; · 36 unmodified lines
37asmlinkage unsigned long __sdei_handler(struct pt_regs *regs,37asmlinkage unsigned long __sdei_handler(struct pt_regs *regs,
38 struct sdei_registered_event *arg);38 struct sdei_registered_event *arg);
39 39
40unsigned long do_sdei_event(struct pt_regs *regs,
41 struct sdei_registered_event *arg);
42
40unsigned long sdei_arch_get_entry_point(int conduit);43unsigned long sdei_arch_get_entry_point(int conduit);
41#define sdei_arch_get_entry_point(x) sdei_arch_get_entry_point(x)44#define sdei_arch_get_entry_point(x) sdei_arch_get_entry_point(x)
42 45
43struct stack_info;46struct stack_info;
44 47
45bool _on_sdei_stack(unsigned long sp, struct stack_info *info);48bool _on_sdei_stack(unsigned long sp, unsigned long size,
46static inline bool on_sdei_stack(unsigned long sp,49 struct stack_info *info);
50static inline bool on_sdei_stack(unsigned long sp, unsigned long size,
47 struct stack_info *info)51 struct stack_info *info)
48{52{
49 if (!IS_ENABLED(CONFIG_VMAP_STACK))53 if (!IS_ENABLED(CONFIG_VMAP_STACK))
@@ -51,7 +55,7 @@ static inline bool on_sdei_stack(unsigned long sp, · 1 unmodified line
51 if (!IS_ENABLED(CONFIG_ARM_SDE_INTERFACE))55 if (!IS_ENABLED(CONFIG_ARM_SDE_INTERFACE))
52 return false;56 return false;
53 if (in_nmi())57 if (in_nmi())
54 return _on_sdei_stack(sp, info);58 return _on_sdei_stack(sp, size, info);
55 59
56 return false;60 return false;
57}61}
arch/arm64/include/asm/smp.h+0−2
@@ -73,12 +73,10 @@ asmlinkage void secondary_start_kernel(void); · 72 unmodified lines
73 73
74/*74/*
75 * Initial data for bringing up a secondary CPU.75 * Initial data for bringing up a secondary CPU.
76 * @stack - sp for the secondary CPU
77 * @status - Result passed back from the secondary CPU to76 * @status - Result passed back from the secondary CPU to
78 * indicate failure.77 * indicate failure.
79 */78 */
80struct secondary_data {79struct secondary_data {
81 void *stack;
82 struct task_struct *task;80 struct task_struct *task;
83 long status;81 long status;
84};82};
arch/arm64/include/asm/stacktrace.h+16−16
@@ -69,14 +69,14 @@ extern void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk, · 68 unmodified lines
69 69
70DECLARE_PER_CPU(unsigned long *, irq_stack_ptr);70DECLARE_PER_CPU(unsigned long *, irq_stack_ptr);
71 71
72static inline bool on_stack(unsigned long sp, unsigned long low,72static inline bool on_stack(unsigned long sp, unsigned long size,
73 unsigned long high, enum stack_type type,73 unsigned long low, unsigned long high,
74 struct stack_info *info)74 enum stack_type type, struct stack_info *info)
75{75{
76 if (!low)76 if (!low)
77 return false;77 return false;
78 78
79 if (sp < low || sp >= high)79 if (sp < low || sp + size < sp || sp + size > high)
80 return false;80 return false;
81 81
82 if (info) {82 if (info) {
@@ -87,38 +87,38 @@ static inline bool on_stack(unsigned long sp, unsigned long low, · 4 unmodified lines
87 return true;87 return true;
88}88}
89 89
90static inline bool on_irq_stack(unsigned long sp,90static inline bool on_irq_stack(unsigned long sp, unsigned long size,
91 struct stack_info *info)91 struct stack_info *info)
92{92{
93 unsigned long low = (unsigned long)raw_cpu_read(irq_stack_ptr);93 unsigned long low = (unsigned long)raw_cpu_read(irq_stack_ptr);
94 unsigned long high = low + IRQ_STACK_SIZE;94 unsigned long high = low + IRQ_STACK_SIZE;
95 95
96 return on_stack(sp, low, high, STACK_TYPE_IRQ, info);96 return on_stack(sp, size, low, high, STACK_TYPE_IRQ, info);
97}97}
98 98
99static inline bool on_task_stack(const struct task_struct *tsk,99static inline bool on_task_stack(const struct task_struct *tsk,
100 unsigned long sp,100 unsigned long sp, unsigned long size,
101 struct stack_info *info)101 struct stack_info *info)
102{102{
103 unsigned long low = (unsigned long)task_stack_page(tsk);103 unsigned long low = (unsigned long)task_stack_page(tsk);
104 unsigned long high = low + THREAD_SIZE;104 unsigned long high = low + THREAD_SIZE;
105 105
106 return on_stack(sp, low, high, STACK_TYPE_TASK, info);106 return on_stack(sp, size, low, high, STACK_TYPE_TASK, info);
107}107}
108 108
109#ifdef CONFIG_VMAP_STACK109#ifdef CONFIG_VMAP_STACK
110DECLARE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack);110DECLARE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack);
111 111
112static inline bool on_overflow_stack(unsigned long sp,112static inline bool on_overflow_stack(unsigned long sp, unsigned long size,
113 struct stack_info *info)113 struct stack_info *info)
114{114{
115 unsigned long low = (unsigned long)raw_cpu_ptr(overflow_stack);115 unsigned long low = (unsigned long)raw_cpu_ptr(overflow_stack);
116 unsigned long high = low + OVERFLOW_STACK_SIZE;116 unsigned long high = low + OVERFLOW_STACK_SIZE;
117 117
118 return on_stack(sp, low, high, STACK_TYPE_OVERFLOW, info);118 return on_stack(sp, size, low, high, STACK_TYPE_OVERFLOW, info);
119}119}
120#else120#else
121static inline bool on_overflow_stack(unsigned long sp,121static inline bool on_overflow_stack(unsigned long sp, unsigned long size,
122 struct stack_info *info) { return false; }122 struct stack_info *info) { return false; }
123#endif123#endif
124 124
@@ -128,21 +128,21 @@ static inline bool on_overflow_stack(unsigned long sp, · 3 unmodified lines
128 * context.128 * context.
129 */129 */
130static inline bool on_accessible_stack(const struct task_struct *tsk,130static inline bool on_accessible_stack(const struct task_struct *tsk,
131 unsigned long sp,131 unsigned long sp, unsigned long size,
132 struct stack_info *info)132 struct stack_info *info)
133{133{
134 if (info)134 if (info)
135 info->type = STACK_TYPE_UNKNOWN;135 info->type = STACK_TYPE_UNKNOWN;
136 136
137 if (on_task_stack(tsk, sp, info))137 if (on_task_stack(tsk, sp, size, info))
138 return true;138 return true;
139 if (tsk != current || preemptible())139 if (tsk != current || preemptible())
140 return false;140 return false;
141 if (on_irq_stack(sp, info))141 if (on_irq_stack(sp, size, info))
142 return true;142 return true;
143 if (on_overflow_stack(sp, info))143 if (on_overflow_stack(sp, size, info))
144 return true;144 return true;
145 if (on_sdei_stack(sp, info))145 if (on_sdei_stack(sp, size, info))
146 return true;146 return true;
147 147
148 return false;148 return false;
arch/arm64/include/asm/sysreg.h+0−2
@@ -703,9 +703,7 @@ · 702 unmodified lines
703/* MAIR_ELx memory attributes (used by Linux) */703/* MAIR_ELx memory attributes (used by Linux) */
704#define MAIR_ATTR_DEVICE_nGnRnE UL(0x00)704#define MAIR_ATTR_DEVICE_nGnRnE UL(0x00)
705#define MAIR_ATTR_DEVICE_nGnRE UL(0x04)705#define MAIR_ATTR_DEVICE_nGnRE UL(0x04)
706#define MAIR_ATTR_DEVICE_GRE UL(0x0c)
707#define MAIR_ATTR_NORMAL_NC UL(0x44)706#define MAIR_ATTR_NORMAL_NC UL(0x44)
708#define MAIR_ATTR_NORMAL_WT UL(0xbb)
709#define MAIR_ATTR_NORMAL_TAGGED UL(0xf0)707#define MAIR_ATTR_NORMAL_TAGGED UL(0xf0)
710#define MAIR_ATTR_NORMAL UL(0xff)708#define MAIR_ATTR_NORMAL UL(0xff)
711#define MAIR_ATTR_MASK UL(0xff)709#define MAIR_ATTR_MASK UL(0xff)
arch/arm64/include/asm/tlb.h+4−0
@@ -28,6 +28,10 @@ static void tlb_flush(struct mmu_gather *tlb); · 27 unmodified lines
28 */28 */
29static inline int tlb_get_level(struct mmu_gather *tlb)29static inline int tlb_get_level(struct mmu_gather *tlb)
30{30{
31 /* The TTL field is only valid for the leaf entry. */
32 if (tlb->freed_tables)
33 return 0;
34
31 if (tlb->cleared_ptes && !(tlb->cleared_pmds ||35 if (tlb->cleared_ptes && !(tlb->cleared_pmds ||
32 tlb->cleared_puds ||36 tlb->cleared_puds ||
33 tlb->cleared_p4ds))37 tlb->cleared_p4ds))
arch/arm64/kernel/Makefile+9−2
@@ -14,15 +14,22 @@ CFLAGS_REMOVE_return_address.o = $(CC_FLAGS_FTRACE) · 13 unmodified lines
14CFLAGS_REMOVE_syscall.o = -fstack-protector -fstack-protector-strong14CFLAGS_REMOVE_syscall.o = -fstack-protector -fstack-protector-strong
15CFLAGS_syscall.o += -fno-stack-protector15CFLAGS_syscall.o += -fno-stack-protector
16 16
17# It's not safe to invoke KCOV when portions of the kernel environment aren't
18# available or are out-of-sync with HW state. Since `noinstr` doesn't always
19# inhibit KCOV instrumentation, disable it for the entire compilation unit.
20KCOV_INSTRUMENT_entry.o := n
21KCOV_INSTRUMENT_idle.o := n
22
17# Object file lists.23# Object file lists.
18obj-y := debug-monitors.o entry.o irq.o fpsimd.o \24obj-y := debug-monitors.o entry.o irq.o fpsimd.o \
19 entry-common.o entry-fpsimd.o process.o ptrace.o \25 entry-common.o entry-fpsimd.o process.o ptrace.o \
20 setup.o signal.o sys.o stacktrace.o time.o traps.o \26 setup.o signal.o sys.o stacktrace.o time.o traps.o \
21 io.o vdso.o hyp-stub.o psci.o cpu_ops.o insn.o \27 io.o vdso.o hyp-stub.o psci.o cpu_ops.o \
22 return_address.o cpuinfo.o cpu_errata.o \28 return_address.o cpuinfo.o cpu_errata.o \
23 cpufeature.o alternative.o cacheinfo.o \29 cpufeature.o alternative.o cacheinfo.o \
24 smp.o smp_spin_table.o topology.o smccc-call.o \30 smp.o smp_spin_table.o topology.o smccc-call.o \
25 syscall.o proton-pack.o idreg-override.o31 syscall.o proton-pack.o idreg-override.o idle.o \
32 patching.o
26 33
27targets += efi-entry.o34targets += efi-entry.o
28 35
arch/arm64/kernel/acpi.c+17−5
@@ -239,6 +239,18 @@ done: · 238 unmodified lines
239 }239 }
240}240}
241 241
242static pgprot_t __acpi_get_writethrough_mem_attribute(void)
243{
244 /*
245 * Although UEFI specifies the use of Normal Write-through for
246 * EFI_MEMORY_WT, it is seldom used in practice and not implemented
247 * by most (all?) CPUs. Rather than allocate a MAIR just for this
248 * purpose, emit a warning and use Normal Non-cacheable instead.
249 */
250 pr_warn_once("No MAIR allocation for EFI_MEMORY_WT; treating as Normal Non-cacheable\n");
251 return __pgprot(PROT_NORMAL_NC);
252}
253
242pgprot_t __acpi_get_mem_attribute(phys_addr_t addr)254pgprot_t __acpi_get_mem_attribute(phys_addr_t addr)
243{255{
244 /*256 /*
@@ -246,7 +258,7 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr) · 1 unmodified line
246 * types" of UEFI 2.5 section 2.3.6.1, each EFI memory type is258 * types" of UEFI 2.5 section 2.3.6.1, each EFI memory type is
247 * mapped to a corresponding MAIR attribute encoding.259 * mapped to a corresponding MAIR attribute encoding.
248 * The EFI memory attribute advises all possible capabilities260 * The EFI memory attribute advises all possible capabilities
249 * of a memory region. We use the most efficient capability.261 * of a memory region.
250 */262 */
251 263
252 u64 attr;264 u64 attr;
@@ -254,10 +266,10 @@ pgprot_t __acpi_get_mem_attribute(phys_addr_t addr) · 1 unmodified line
254 attr = efi_mem_attributes(addr);266 attr = efi_mem_attributes(addr);
255 if (attr & EFI_MEMORY_WB)267 if (attr & EFI_MEMORY_WB)
256 return PAGE_KERNEL;268 return PAGE_KERNEL;
257 if (attr & EFI_MEMORY_WT)
258 return __pgprot(PROT_NORMAL_WT);
259 if (attr & EFI_MEMORY_WC)269 if (attr & EFI_MEMORY_WC)
260 return __pgprot(PROT_NORMAL_NC);270 return __pgprot(PROT_NORMAL_NC);
271 if (attr & EFI_MEMORY_WT)
272 return __acpi_get_writethrough_mem_attribute();
261 return __pgprot(PROT_DEVICE_nGnRnE);273 return __pgprot(PROT_DEVICE_nGnRnE);
262}274}
263 275
@@ -340,10 +352,10 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size) · 76 unmodified lines
340 default:352 default:
341 if (region->attribute & EFI_MEMORY_WB)353 if (region->attribute & EFI_MEMORY_WB)
342 prot = PAGE_KERNEL;354 prot = PAGE_KERNEL;
343 else if (region->attribute & EFI_MEMORY_WT)
344 prot = __pgprot(PROT_NORMAL_WT);
345 else if (region->attribute & EFI_MEMORY_WC)355 else if (region->attribute & EFI_MEMORY_WC)
346 prot = __pgprot(PROT_NORMAL_NC);356 prot = __pgprot(PROT_NORMAL_NC);
357 else if (region->attribute & EFI_MEMORY_WT)
358 prot = __acpi_get_writethrough_mem_attribute();
347 }359 }
348 }360 }
349 return __ioremap(phys, size, prot);361 return __ioremap(phys, size, prot);
arch/arm64/kernel/alternative.c+1−1
@@ -181,7 +181,7 @@ static void __nocfi __apply_alternatives(struct alt_region *region, bool is_modu · 180 unmodified lines
181 */181 */
182 if (!is_module) {182 if (!is_module) {
183 dsb(ish);183 dsb(ish);
184 __flush_icache_all();184 icache_inval_all_pou();
185 isb();185 isb();
186 186
187 /* Ignore ARM64_CB bit from feature mask */187 /* Ignore ARM64_CB bit from feature mask */
arch/arm64/kernel/asm-offsets.c+14−1
@@ -27,6 +27,7 @@ · 26 unmodified lines
27int main(void)27int main(void)
28{28{
29 DEFINE(TSK_ACTIVE_MM, offsetof(struct task_struct, active_mm));29 DEFINE(TSK_ACTIVE_MM, offsetof(struct task_struct, active_mm));
30 DEFINE(TSK_CPU, offsetof(struct task_struct, cpu));
30 BLANK();31 BLANK();
31 DEFINE(TSK_TI_FLAGS, offsetof(struct task_struct, thread_info.flags));32 DEFINE(TSK_TI_FLAGS, offsetof(struct task_struct, thread_info.flags));
32 DEFINE(TSK_TI_PREEMPT, offsetof(struct task_struct, thread_info.preempt_count));33 DEFINE(TSK_TI_PREEMPT, offsetof(struct task_struct, thread_info.preempt_count));
@@ -46,6 +47,8 @@ int main(void) · 13 unmodified lines
46 DEFINE(THREAD_SCTLR_USER, offsetof(struct task_struct, thread.sctlr_user));47 DEFINE(THREAD_SCTLR_USER, offsetof(struct task_struct, thread.sctlr_user));
47#ifdef CONFIG_ARM64_PTR_AUTH48#ifdef CONFIG_ARM64_PTR_AUTH
48 DEFINE(THREAD_KEYS_USER, offsetof(struct task_struct, thread.keys_user));49 DEFINE(THREAD_KEYS_USER, offsetof(struct task_struct, thread.keys_user));
50#endif
51#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL
49 DEFINE(THREAD_KEYS_KERNEL, offsetof(struct task_struct, thread.keys_kernel));52 DEFINE(THREAD_KEYS_KERNEL, offsetof(struct task_struct, thread.keys_kernel));
50#endif53#endif
51#ifdef CONFIG_ARM64_MTE54#ifdef CONFIG_ARM64_MTE
@@ -99,7 +102,6 @@ int main(void) · 47 unmodified lines
99 DEFINE(SOFTIRQ_SHIFT, SOFTIRQ_SHIFT);102 DEFINE(SOFTIRQ_SHIFT, SOFTIRQ_SHIFT);
100 DEFINE(IRQ_CPUSTAT_SOFTIRQ_PENDING, offsetof(irq_cpustat_t, __softirq_pending));103 DEFINE(IRQ_CPUSTAT_SOFTIRQ_PENDING, offsetof(irq_cpustat_t, __softirq_pending));
101 BLANK();104 BLANK();
102 DEFINE(CPU_BOOT_STACK, offsetof(struct secondary_data, stack));
103 DEFINE(CPU_BOOT_TASK, offsetof(struct secondary_data, task));105 DEFINE(CPU_BOOT_TASK, offsetof(struct secondary_data, task));
104 BLANK();106 BLANK();
105 DEFINE(FTR_OVR_VAL_OFFSET, offsetof(struct arm64_ftr_override, val));107 DEFINE(FTR_OVR_VAL_OFFSET, offsetof(struct arm64_ftr_override, val));
@@ -138,6 +140,15 @@ int main(void) · 32 unmodified lines
138 DEFINE(ARM_SMCCC_RES_X2_OFFS, offsetof(struct arm_smccc_res, a2));140 DEFINE(ARM_SMCCC_RES_X2_OFFS, offsetof(struct arm_smccc_res, a2));
139 DEFINE(ARM_SMCCC_QUIRK_ID_OFFS, offsetof(struct arm_smccc_quirk, id));141 DEFINE(ARM_SMCCC_QUIRK_ID_OFFS, offsetof(struct arm_smccc_quirk, id));
140 DEFINE(ARM_SMCCC_QUIRK_STATE_OFFS, offsetof(struct arm_smccc_quirk, state));142 DEFINE(ARM_SMCCC_QUIRK_STATE_OFFS, offsetof(struct arm_smccc_quirk, state));
143 DEFINE(ARM_SMCCC_1_2_REGS_X0_OFFS, offsetof(struct arm_smccc_1_2_regs, a0));
144 DEFINE(ARM_SMCCC_1_2_REGS_X2_OFFS, offsetof(struct arm_smccc_1_2_regs, a2));
145 DEFINE(ARM_SMCCC_1_2_REGS_X4_OFFS, offsetof(struct arm_smccc_1_2_regs, a4));
146 DEFINE(ARM_SMCCC_1_2_REGS_X6_OFFS, offsetof(struct arm_smccc_1_2_regs, a6));
147 DEFINE(ARM_SMCCC_1_2_REGS_X8_OFFS, offsetof(struct arm_smccc_1_2_regs, a8));
148 DEFINE(ARM_SMCCC_1_2_REGS_X10_OFFS, offsetof(struct arm_smccc_1_2_regs, a10));
149 DEFINE(ARM_SMCCC_1_2_REGS_X12_OFFS, offsetof(struct arm_smccc_1_2_regs, a12));
150 DEFINE(ARM_SMCCC_1_2_REGS_X14_OFFS, offsetof(struct arm_smccc_1_2_regs, a14));
151 DEFINE(ARM_SMCCC_1_2_REGS_X16_OFFS, offsetof(struct arm_smccc_1_2_regs, a16));
141 BLANK();152 BLANK();
142 DEFINE(HIBERN_PBE_ORIG, offsetof(struct pbe, orig_address));153 DEFINE(HIBERN_PBE_ORIG, offsetof(struct pbe, orig_address));
143 DEFINE(HIBERN_PBE_ADDR, offsetof(struct pbe, address));154 DEFINE(HIBERN_PBE_ADDR, offsetof(struct pbe, address));
@@ -153,7 +164,9 @@ int main(void) · 9 unmodified lines
153#endif164#endif
154#ifdef CONFIG_ARM64_PTR_AUTH165#ifdef CONFIG_ARM64_PTR_AUTH
155 DEFINE(PTRAUTH_USER_KEY_APIA, offsetof(struct ptrauth_keys_user, apia));166 DEFINE(PTRAUTH_USER_KEY_APIA, offsetof(struct ptrauth_keys_user, apia));
167#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL
156 DEFINE(PTRAUTH_KERNEL_KEY_APIA, offsetof(struct ptrauth_keys_kernel, apia));168 DEFINE(PTRAUTH_KERNEL_KEY_APIA, offsetof(struct ptrauth_keys_kernel, apia));
169#endif
157 BLANK();170 BLANK();
158#endif171#endif
159 return 0;172 return 0;
arch/arm64/kernel/cpufeature.c+162−48
@@ -76,6 +76,7 @@ · 75 unmodified lines
76#include <asm/cpufeature.h>76#include <asm/cpufeature.h>
77#include <asm/cpu_ops.h>77#include <asm/cpu_ops.h>
78#include <asm/fpsimd.h>78#include <asm/fpsimd.h>
79#include <asm/insn.h>
79#include <asm/kvm_host.h>80#include <asm/kvm_host.h>
80#include <asm/mmu_context.h>81#include <asm/mmu_context.h>
81#include <asm/mte.h>82#include <asm/mte.h>
@@ -107,6 +108,24 @@ DECLARE_BITMAP(boot_capabilities, ARM64_NPATCHABLE); · 25 unmodified lines
107bool arm64_use_ng_mappings = false;108bool arm64_use_ng_mappings = false;
108EXPORT_SYMBOL(arm64_use_ng_mappings);109EXPORT_SYMBOL(arm64_use_ng_mappings);
109 110
111/*
112 * Permit PER_LINUX32 and execve() of 32-bit binaries even if not all CPUs
113 * support it?
114 */
115static bool __read_mostly allow_mismatched_32bit_el0;
116
117/*
118 * Static branch enabled only if allow_mismatched_32bit_el0 is set and we have
119 * seen at least one CPU capable of 32-bit EL0.
120 */
121DEFINE_STATIC_KEY_FALSE(arm64_mismatched_32bit_el0);
122
123/*
124 * Mask of CPUs supporting 32-bit EL0.
125 * Only valid if arm64_mismatched_32bit_el0 is enabled.
126 */
127static cpumask_var_t cpu_32bit_el0_mask __cpumask_var_read_mostly;
128
110/*129/*
111 * Flag to indicate if we have computed the system wide130 * Flag to indicate if we have computed the system wide
112 * capabilities based on the boot time active CPUs. This131 * capabilities based on the boot time active CPUs. This
@@ -400,6 +419,11 @@ static const struct arm64_ftr_bits ftr_dczid[] = { · 287 unmodified lines
400 ARM64_FTR_END,419 ARM64_FTR_END,
401};420};
402 421
422static const struct arm64_ftr_bits ftr_gmid[] = {
423 ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, SYS_GMID_EL1_BS_SHIFT, 4, 0),
424 ARM64_FTR_END,
425};
426
403static const struct arm64_ftr_bits ftr_id_isar0[] = {427static const struct arm64_ftr_bits ftr_id_isar0[] = {
404 ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_ISAR0_DIVIDE_SHIFT, 4, 0),428 ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_ISAR0_DIVIDE_SHIFT, 4, 0),
405 ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_ISAR0_DEBUG_SHIFT, 4, 0),429 ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_ISAR0_DEBUG_SHIFT, 4, 0),
@@ -617,6 +641,9 @@ static const struct __ftr_reg_entry { · 211 unmodified lines
617 /* Op1 = 0, CRn = 1, CRm = 2 */641 /* Op1 = 0, CRn = 1, CRm = 2 */
618 ARM64_FTR_REG(SYS_ZCR_EL1, ftr_zcr),642 ARM64_FTR_REG(SYS_ZCR_EL1, ftr_zcr),
619 643
644 /* Op1 = 1, CRn = 0, CRm = 0 */
645 ARM64_FTR_REG(SYS_GMID_EL1, ftr_gmid),
646
620 /* Op1 = 3, CRn = 0, CRm = 0 */647 /* Op1 = 3, CRn = 0, CRm = 0 */
621 { SYS_CTR_EL0, &arm64_ftr_reg_ctrel0 },648 { SYS_CTR_EL0, &arm64_ftr_reg_ctrel0 },
622 ARM64_FTR_REG(SYS_DCZID_EL0, ftr_dczid),649 ARM64_FTR_REG(SYS_DCZID_EL0, ftr_dczid),
@@ -767,7 +794,7 @@ static void __init sort_ftr_regs(void) · 144 unmodified lines
767 * Any bits that are not covered by an arm64_ftr_bits entry are considered794 * Any bits that are not covered by an arm64_ftr_bits entry are considered
768 * RES0 for the system-wide value, and must strictly match.795 * RES0 for the system-wide value, and must strictly match.
769 */796 */
770static void __init init_cpu_ftr_reg(u32 sys_reg, u64 new)797static void init_cpu_ftr_reg(u32 sys_reg, u64 new)
771{798{
772 u64 val = 0;799 u64 val = 0;
773 u64 strict_mask = ~0x0ULL;800 u64 strict_mask = ~0x0ULL;
@@ -863,6 +890,31 @@ static void __init init_cpu_hwcaps_indirect_list(void) · 89 unmodified lines
863 890
864static void __init setup_boot_cpu_capabilities(void);891static void __init setup_boot_cpu_capabilities(void);
865 892
893static void init_32bit_cpu_features(struct cpuinfo_32bit *info)
894{
895 init_cpu_ftr_reg(SYS_ID_DFR0_EL1, info->reg_id_dfr0);
896 init_cpu_ftr_reg(SYS_ID_DFR1_EL1, info->reg_id_dfr1);
897 init_cpu_ftr_reg(SYS_ID_ISAR0_EL1, info->reg_id_isar0);
898 init_cpu_ftr_reg(SYS_ID_ISAR1_EL1, info->reg_id_isar1);
899 init_cpu_ftr_reg(SYS_ID_ISAR2_EL1, info->reg_id_isar2);
900 init_cpu_ftr_reg(SYS_ID_ISAR3_EL1, info->reg_id_isar3);
901 init_cpu_ftr_reg(SYS_ID_ISAR4_EL1, info->reg_id_isar4);
902 init_cpu_ftr_reg(SYS_ID_ISAR5_EL1, info->reg_id_isar5);
903 init_cpu_ftr_reg(SYS_ID_ISAR6_EL1, info->reg_id_isar6);
904 init_cpu_ftr_reg(SYS_ID_MMFR0_EL1, info->reg_id_mmfr0);
905 init_cpu_ftr_reg(SYS_ID_MMFR1_EL1, info->reg_id_mmfr1);
906 init_cpu_ftr_reg(SYS_ID_MMFR2_EL1, info->reg_id_mmfr2);
907 init_cpu_ftr_reg(SYS_ID_MMFR3_EL1, info->reg_id_mmfr3);
908 init_cpu_ftr_reg(SYS_ID_MMFR4_EL1, info->reg_id_mmfr4);
909 init_cpu_ftr_reg(SYS_ID_MMFR5_EL1, info->reg_id_mmfr5);
910 init_cpu_ftr_reg(SYS_ID_PFR0_EL1, info->reg_id_pfr0);
911 init_cpu_ftr_reg(SYS_ID_PFR1_EL1, info->reg_id_pfr1);
912 init_cpu_ftr_reg(SYS_ID_PFR2_EL1, info->reg_id_pfr2);
913 init_cpu_ftr_reg(SYS_MVFR0_EL1, info->reg_mvfr0);
914 init_cpu_ftr_reg(SYS_MVFR1_EL1, info->reg_mvfr1);
915 init_cpu_ftr_reg(SYS_MVFR2_EL1, info->reg_mvfr2);
916}
917
866void __init init_cpu_features(struct cpuinfo_arm64 *info)918void __init init_cpu_features(struct cpuinfo_arm64 *info)
867{919{
868 /* Before we start using the tables, make sure it is sorted */920 /* Before we start using the tables, make sure it is sorted */
@@ -882,35 +934,17 @@ void __init init_cpu_features(struct cpuinfo_arm64 *info) · 13 unmodified lines
882 init_cpu_ftr_reg(SYS_ID_AA64PFR1_EL1, info->reg_id_aa64pfr1);934 init_cpu_ftr_reg(SYS_ID_AA64PFR1_EL1, info->reg_id_aa64pfr1);
883 init_cpu_ftr_reg(SYS_ID_AA64ZFR0_EL1, info->reg_id_aa64zfr0);935 init_cpu_ftr_reg(SYS_ID_AA64ZFR0_EL1, info->reg_id_aa64zfr0);
884 936
885 if (id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0)) {937 if (id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0))
886 init_cpu_ftr_reg(SYS_ID_DFR0_EL1, info->reg_id_dfr0);938 init_32bit_cpu_features(&info->aarch32);
887 init_cpu_ftr_reg(SYS_ID_DFR1_EL1, info->reg_id_dfr1);
888 init_cpu_ftr_reg(SYS_ID_ISAR0_EL1, info->reg_id_isar0);
889 init_cpu_ftr_reg(SYS_ID_ISAR1_EL1, info->reg_id_isar1);
890 init_cpu_ftr_reg(SYS_ID_ISAR2_EL1, info->reg_id_isar2);
891 init_cpu_ftr_reg(SYS_ID_ISAR3_EL1, info->reg_id_isar3);
892 init_cpu_ftr_reg(SYS_ID_ISAR4_EL1, info->reg_id_isar4);
893 init_cpu_ftr_reg(SYS_ID_ISAR5_EL1, info->reg_id_isar5);
894 init_cpu_ftr_reg(SYS_ID_ISAR6_EL1, info->reg_id_isar6);
895 init_cpu_ftr_reg(SYS_ID_MMFR0_EL1, info->reg_id_mmfr0);
896 init_cpu_ftr_reg(SYS_ID_MMFR1_EL1, info->reg_id_mmfr1);
897 init_cpu_ftr_reg(SYS_ID_MMFR2_EL1, info->reg_id_mmfr2);
898 init_cpu_ftr_reg(SYS_ID_MMFR3_EL1, info->reg_id_mmfr3);
899 init_cpu_ftr_reg(SYS_ID_MMFR4_EL1, info->reg_id_mmfr4);
900 init_cpu_ftr_reg(SYS_ID_MMFR5_EL1, info->reg_id_mmfr5);
901 init_cpu_ftr_reg(SYS_ID_PFR0_EL1, info->reg_id_pfr0);
902 init_cpu_ftr_reg(SYS_ID_PFR1_EL1, info->reg_id_pfr1);
903 init_cpu_ftr_reg(SYS_ID_PFR2_EL1, info->reg_id_pfr2);
904 init_cpu_ftr_reg(SYS_MVFR0_EL1, info->reg_mvfr0);
905 init_cpu_ftr_reg(SYS_MVFR1_EL1, info->reg_mvfr1);
906 init_cpu_ftr_reg(SYS_MVFR2_EL1, info->reg_mvfr2);
907 }
908 939
909 if (id_aa64pfr0_sve(info->reg_id_aa64pfr0)) {940 if (id_aa64pfr0_sve(info->reg_id_aa64pfr0)) {
910 init_cpu_ftr_reg(SYS_ZCR_EL1, info->reg_zcr);941 init_cpu_ftr_reg(SYS_ZCR_EL1, info->reg_zcr);
911 sve_init_vq_map();942 sve_init_vq_map();
912 }943 }
913 944
945 if (id_aa64pfr1_mte(info->reg_id_aa64pfr1))
946 init_cpu_ftr_reg(SYS_GMID_EL1, info->reg_gmid);
947
914 /*948 /*
915 * Initialize the indirect array of CPU hwcaps capabilities pointers949 * Initialize the indirect array of CPU hwcaps capabilities pointers
916 * before we handle the boot CPU below.950 * before we handle the boot CPU below.
@@ -975,20 +1009,28 @@ static void relax_cpu_ftr_reg(u32 sys_id, int field) · 58 unmodified lines
975 WARN_ON(!ftrp->width);1009 WARN_ON(!ftrp->width);
976}1010}
977 1011
978static int update_32bit_cpu_features(int cpu, struct cpuinfo_arm64 *info,1012static void lazy_init_32bit_cpu_features(struct cpuinfo_arm64 *info,
979 struct cpuinfo_arm64 *boot)1013 struct cpuinfo_arm64 *boot)
1014{
1015 static bool boot_cpu_32bit_regs_overridden = false;
1016
1017 if (!allow_mismatched_32bit_el0 || boot_cpu_32bit_regs_overridden)
1018 return;
1019
1020 if (id_aa64pfr0_32bit_el0(boot->reg_id_aa64pfr0))
1021 return;
1022
1023 boot->aarch32 = info->aarch32;
1024 init_32bit_cpu_features(&boot->aarch32);
1025 boot_cpu_32bit_regs_overridden = true;
1026}
1027
1028static int update_32bit_cpu_features(int cpu, struct cpuinfo_32bit *info,
1029 struct cpuinfo_32bit *boot)
980{1030{
981 int taint = 0;1031 int taint = 0;
982 u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);1032 u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
983 1033
984 /*
985 * If we don't have AArch32 at all then skip the checks entirely
986 * as the register values may be UNKNOWN and we're not going to be
987 * using them for anything.
988 */
989 if (!id_aa64pfr0_32bit_el0(pfr0))
990 return taint;
991
992 /*1034 /*
993 * If we don't have AArch32 at EL1, then relax the strictness of1035 * If we don't have AArch32 at EL1, then relax the strictness of
994 * EL1-dependent register fields to avoid spurious sanity check fails.1036 * EL1-dependent register fields to avoid spurious sanity check fails.
@@ -1135,10 +1177,29 @@ void update_cpu_features(int cpu, · 140 unmodified lines
1135 }1177 }
1136 1178
1137 /*1179 /*
1180 * The kernel uses the LDGM/STGM instructions and the number of tags
1181 * they read/write depends on the GMID_EL1.BS field. Check that the
1182 * value is the same on all CPUs.
1183 */
1184 if (IS_ENABLED(CONFIG_ARM64_MTE) &&
1185 id_aa64pfr1_mte(info->reg_id_aa64pfr1)) {
1186 taint |= check_update_ftr_reg(SYS_GMID_EL1, cpu,
1187 info->reg_gmid, boot->reg_gmid);
1188 }
1189
1190 /*
1191 * If we don't have AArch32 at all then skip the checks entirely
1192 * as the register values may be UNKNOWN and we're not going to be
1193 * using them for anything.
1194 *
1138 * This relies on a sanitised view of the AArch64 ID registers1195 * This relies on a sanitised view of the AArch64 ID registers
1139 * (e.g. SYS_ID_AA64PFR0_EL1), so we call it last.1196 * (e.g. SYS_ID_AA64PFR0_EL1), so we call it last.
1140 */1197 */
1141 taint |= update_32bit_cpu_features(cpu, info, boot);1198 if (id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0)) {
1199 lazy_init_32bit_cpu_features(info, boot);
1200 taint |= update_32bit_cpu_features(cpu, &info->aarch32,
1201 &boot->aarch32);
1202 }
1142 1203
1143 /*1204 /*
1144 * Mismatched CPU features are a recipe for disaster. Don't even1205 * Mismatched CPU features are a recipe for disaster. Don't even
@@ -1248,6 +1309,28 @@ has_cpuid_feature(const struct arm64_cpu_capabilities *entry, int scope) · 103 unmodified lines
1248 return feature_matches(val, entry);1309 return feature_matches(val, entry);
1249}1310}
1250 1311
1312const struct cpumask *system_32bit_el0_cpumask(void)
1313{
1314 if (!system_supports_32bit_el0())
1315 return cpu_none_mask;
1316
1317 if (static_branch_unlikely(&arm64_mismatched_32bit_el0))
1318 return cpu_32bit_el0_mask;
1319
1320 return cpu_possible_mask;
1321}
1322
1323static bool has_32bit_el0(const struct arm64_cpu_capabilities *entry, int scope)
1324{
1325 if (!has_cpuid_feature(entry, scope))
1326 return allow_mismatched_32bit_el0;
1327
1328 if (scope == SCOPE_SYSTEM)
1329 pr_info("detected: 32-bit EL0 Support\n");
1330
1331 return true;
1332}
1333
1251static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)1334static bool has_useable_gicv3_cpuif(const struct arm64_cpu_capabilities *entry, int scope)
1252{1335{
1253 bool has_sre;1336 bool has_sre;
@@ -1866,10 +1949,9 @@ static const struct arm64_cpu_capabilities arm64_features[] = { · 612 unmodified lines
1866 .cpu_enable = cpu_copy_el2regs,1949 .cpu_enable = cpu_copy_el2regs,
1867 },1950 },
1868 {1951 {
1869 .desc = "32-bit EL0 Support",1952 .capability = ARM64_HAS_32BIT_EL0_DO_NOT_USE,
1870 .capability = ARM64_HAS_32BIT_EL0,
1871 .type = ARM64_CPUCAP_SYSTEM_FEATURE,1953 .type = ARM64_CPUCAP_SYSTEM_FEATURE,
1872 .matches = has_cpuid_feature,1954 .matches = has_32bit_el0,
1873 .sys_reg = SYS_ID_AA64PFR0_EL1,1955 .sys_reg = SYS_ID_AA64PFR0_EL1,
1874 .sign = FTR_UNSIGNED,1956 .sign = FTR_UNSIGNED,
1875 .field_pos = ID_AA64PFR0_EL0_SHIFT,1957 .field_pos = ID_AA64PFR0_EL0_SHIFT,
@@ -2378,7 +2460,7 @@ static const struct arm64_cpu_capabilities compat_elf_hwcaps[] = { · 502 unmodified lines
2378 {},2460 {},
2379};2461};
2380 2462
2381static void __init cap_set_elf_hwcap(const struct arm64_cpu_capabilities *cap)2463static void cap_set_elf_hwcap(const struct arm64_cpu_capabilities *cap)
2382{2464{
2383 switch (cap->hwcap_type) {2465 switch (cap->hwcap_type) {
2384 case CAP_HWCAP:2466 case CAP_HWCAP:
@@ -2423,7 +2505,7 @@ static bool cpus_have_elf_hwcap(const struct arm64_cpu_capabilities *cap) · 38 unmodified lines
2423 return rc;2505 return rc;
2424}2506}
2425 2507
2426static void __init setup_elf_hwcaps(const struct arm64_cpu_capabilities *hwcaps)2508static void setup_elf_hwcaps(const struct arm64_cpu_capabilities *hwcaps)
2427{2509{
2428 /* We support emulation of accesses to CPU ID feature registers */2510 /* We support emulation of accesses to CPU ID feature registers */
2429 cpu_set_named_feature(CPUID);2511 cpu_set_named_feature(CPUID);
@@ -2598,7 +2680,7 @@ static void check_early_cpu_features(void) · 168 unmodified lines
2598}2680}
2599 2681
2600static void2682static void
2601verify_local_elf_hwcaps(const struct arm64_cpu_capabilities *caps)2683__verify_local_elf_hwcaps(const struct arm64_cpu_capabilities *caps)
2602{2684{
2603 2685
2604 for (; caps->matches; caps++)2686 for (; caps->matches; caps++)
@@ -2609,6 +2691,14 @@ verify_local_elf_hwcaps(const struct arm64_cpu_capabilities *caps) · 4 unmodified lines
2609 }2691 }
2610}2692}
2611 2693
2694static void verify_local_elf_hwcaps(void)
2695{
2696 __verify_local_elf_hwcaps(arm64_elf_hwcaps);
2697
2698 if (id_aa64pfr0_32bit_el0(read_cpuid(ID_AA64PFR0_EL1)))
2699 __verify_local_elf_hwcaps(compat_elf_hwcaps);
2700}
2701
2612static void verify_sve_features(void)2702static void verify_sve_features(void)
2613{2703{
2614 u64 safe_zcr = read_sanitised_ftr_reg(SYS_ZCR_EL1);2704 u64 safe_zcr = read_sanitised_ftr_reg(SYS_ZCR_EL1);
@@ -2673,11 +2763,7 @@ static void verify_local_cpu_capabilities(void) · 58 unmodified lines
2673 * on all secondary CPUs.2763 * on all secondary CPUs.
2674 */2764 */
2675 verify_local_cpu_caps(SCOPE_ALL & ~SCOPE_BOOT_CPU);2765 verify_local_cpu_caps(SCOPE_ALL & ~SCOPE_BOOT_CPU);
2676 2766 verify_local_elf_hwcaps();
2677 verify_local_elf_hwcaps(arm64_elf_hwcaps);
2678
2679 if (system_supports_32bit_el0())
2680 verify_local_elf_hwcaps(compat_elf_hwcaps);
2681 2767
2682 if (system_supports_sve())2768 if (system_supports_sve())
2683 verify_sve_features();2769 verify_sve_features();
@@ -2812,6 +2898,34 @@ void __init setup_cpu_features(void) · 128 unmodified lines
2812 ARCH_DMA_MINALIGN);2898 ARCH_DMA_MINALIGN);
2813}2899}
2814 2900
2901static int enable_mismatched_32bit_el0(unsigned int cpu)
2902{
2903 struct cpuinfo_arm64 *info = &per_cpu(cpu_data, cpu);
2904 bool cpu_32bit = id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0);
2905
2906 if (cpu_32bit) {
2907 cpumask_set_cpu(cpu, cpu_32bit_el0_mask);
2908 static_branch_enable_cpuslocked(&arm64_mismatched_32bit_el0);
2909 setup_elf_hwcaps(compat_elf_hwcaps);
2910 }
2911
2912 return 0;
2913}
2914
2915static int __init init_32bit_el0_mask(void)
2916{
2917 if (!allow_mismatched_32bit_el0)
2918 return 0;
2919
2920 if (!zalloc_cpumask_var(&cpu_32bit_el0_mask, GFP_KERNEL))
2921 return -ENOMEM;
2922
2923 return cpuhp_setup_state(CPUHP_AP_ONLINE_DYN,
2924 "arm64/mismatched_32bit_el0:online",
2925 enable_mismatched_32bit_el0, NULL);
2926}
2927subsys_initcall_sync(init_32bit_el0_mask);
2928
2815static void __maybe_unused cpu_enable_cnp(struct arm64_cpu_capabilities const *cap)2929static void __maybe_unused cpu_enable_cnp(struct arm64_cpu_capabilities const *cap)
2816{2930{
2817 cpu_replace_ttbr1(lm_alias(swapper_pg_dir));2931 cpu_replace_ttbr1(lm_alias(swapper_pg_dir));
@@ -2905,8 +3019,8 @@ static int emulate_mrs(struct pt_regs *regs, u32 insn) · 87 unmodified lines
2905}3019}
2906 3020
2907static struct undef_hook mrs_hook = {3021static struct undef_hook mrs_hook = {
2908 .instr_mask = 0xfff00000,3022 .instr_mask = 0xffff0000,
2909 .instr_val = 0xd5300000,3023 .instr_val = 0xd5380000,
2910 .pstate_mask = PSR_AA32_MODE_MASK,3024 .pstate_mask = PSR_AA32_MODE_MASK,
2911 .pstate_val = PSR_MODE_EL0t,3025 .pstate_val = PSR_MODE_EL0t,
2912 .fn = emulate_mrs,3026 .fn = emulate_mrs,
arch/arm64/kernel/cpuinfo.c+32−26
@@ -246,7 +246,7 @@ static struct kobj_type cpuregs_kobj_type = { · 245 unmodified lines
246 struct cpuinfo_arm64 *info = kobj_to_cpuinfo(kobj); \246 struct cpuinfo_arm64 *info = kobj_to_cpuinfo(kobj); \
247 \247 \
248 if (info->reg_midr) \248 if (info->reg_midr) \
249 return sprintf(buf, "0x%016x\n", info->reg_##_field); \249 return sprintf(buf, "0x%016llx\n", info->reg_##_field); \
250 else \250 else \
251 return 0; \251 return 0; \
252 } \252 } \
@@ -344,6 +344,32 @@ static void cpuinfo_detect_icache_policy(struct cpuinfo_arm64 *info) · 91 unmodified lines
344 pr_info("Detected %s I-cache on CPU%d\n", icache_policy_str[l1ip], cpu);344 pr_info("Detected %s I-cache on CPU%d\n", icache_policy_str[l1ip], cpu);
345}345}
346 346
347static void __cpuinfo_store_cpu_32bit(struct cpuinfo_32bit *info)
348{
349 info->reg_id_dfr0 = read_cpuid(ID_DFR0_EL1);
350 info->reg_id_dfr1 = read_cpuid(ID_DFR1_EL1);
351 info->reg_id_isar0 = read_cpuid(ID_ISAR0_EL1);
352 info->reg_id_isar1 = read_cpuid(ID_ISAR1_EL1);
353 info->reg_id_isar2 = read_cpuid(ID_ISAR2_EL1);
354 info->reg_id_isar3 = read_cpuid(ID_ISAR3_EL1);
355 info->reg_id_isar4 = read_cpuid(ID_ISAR4_EL1);
356 info->reg_id_isar5 = read_cpuid(ID_ISAR5_EL1);
357 info->reg_id_isar6 = read_cpuid(ID_ISAR6_EL1);
358 info->reg_id_mmfr0 = read_cpuid(ID_MMFR0_EL1);
359 info->reg_id_mmfr1 = read_cpuid(ID_MMFR1_EL1);
360 info->reg_id_mmfr2 = read_cpuid(ID_MMFR2_EL1);
361 info->reg_id_mmfr3 = read_cpuid(ID_MMFR3_EL1);
362 info->reg_id_mmfr4 = read_cpuid(ID_MMFR4_EL1);
363 info->reg_id_mmfr5 = read_cpuid(ID_MMFR5_EL1);
364 info->reg_id_pfr0 = read_cpuid(ID_PFR0_EL1);
365 info->reg_id_pfr1 = read_cpuid(ID_PFR1_EL1);
366 info->reg_id_pfr2 = read_cpuid(ID_PFR2_EL1);
367
368 info->reg_mvfr0 = read_cpuid(MVFR0_EL1);
369 info->reg_mvfr1 = read_cpuid(MVFR1_EL1);
370 info->reg_mvfr2 = read_cpuid(MVFR2_EL1);
371}
372
347static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info)373static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info)
348{374{
349 info->reg_cntfrq = arch_timer_get_cntfrq();375 info->reg_cntfrq = arch_timer_get_cntfrq();
@@ -371,31 +397,11 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) · 21 unmodified lines
371 info->reg_id_aa64pfr1 = read_cpuid(ID_AA64PFR1_EL1);397 info->reg_id_aa64pfr1 = read_cpuid(ID_AA64PFR1_EL1);
372 info->reg_id_aa64zfr0 = read_cpuid(ID_AA64ZFR0_EL1);398 info->reg_id_aa64zfr0 = read_cpuid(ID_AA64ZFR0_EL1);
373 399
374 /* Update the 32bit ID registers only if AArch32 is implemented */400 if (id_aa64pfr1_mte(info->reg_id_aa64pfr1))
375 if (id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0)) {401 info->reg_gmid = read_cpuid(GMID_EL1);
376 info->reg_id_dfr0 = read_cpuid(ID_DFR0_EL1);402
377 info->reg_id_dfr1 = read_cpuid(ID_DFR1_EL1);403 if (id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0))
378 info->reg_id_isar0 = read_cpuid(ID_ISAR0_EL1);404 __cpuinfo_store_cpu_32bit(&info->aarch32);
379 info->reg_id_isar1 = read_cpuid(ID_ISAR1_EL1);
380 info->reg_id_isar2 = read_cpuid(ID_ISAR2_EL1);
381 info->reg_id_isar3 = read_cpuid(ID_ISAR3_EL1);
382 info->reg_id_isar4 = read_cpuid(ID_ISAR4_EL1);
383 info->reg_id_isar5 = read_cpuid(ID_ISAR5_EL1);
384 info->reg_id_isar6 = read_cpuid(ID_ISAR6_EL1);
385 info->reg_id_mmfr0 = read_cpuid(ID_MMFR0_EL1);
386 info->reg_id_mmfr1 = read_cpuid(ID_MMFR1_EL1);
387 info->reg_id_mmfr2 = read_cpuid(ID_MMFR2_EL1);
388 info->reg_id_mmfr3 = read_cpuid(ID_MMFR3_EL1);
389 info->reg_id_mmfr4 = read_cpuid(ID_MMFR4_EL1);
390 info->reg_id_mmfr5 = read_cpuid(ID_MMFR5_EL1);
391 info->reg_id_pfr0 = read_cpuid(ID_PFR0_EL1);
392 info->reg_id_pfr1 = read_cpuid(ID_PFR1_EL1);
393 info->reg_id_pfr2 = read_cpuid(ID_PFR2_EL1);
394
395 info->reg_mvfr0 = read_cpuid(MVFR0_EL1);
396 info->reg_mvfr1 = read_cpuid(MVFR1_EL1);
397 info->reg_mvfr2 = read_cpuid(MVFR2_EL1);
398 }
399 405
400 if (IS_ENABLED(CONFIG_ARM64_SVE) &&406 if (IS_ENABLED(CONFIG_ARM64_SVE) &&
401 id_aa64pfr0_sve(info->reg_id_aa64pfr0))407 id_aa64pfr0_sve(info->reg_id_aa64pfr0))
arch/arm64/kernel/efi-entry.S+5−4
@@ -28,7 +28,8 @@ SYM_CODE_START(efi_enter_kernel) · 27 unmodified lines
28 * stale icache entries from before relocation.28 * stale icache entries from before relocation.
29 */29 */
30 ldr w1, =kernel_size30 ldr w1, =kernel_size
31 bl __clean_dcache_area_poc31 add x1, x0, x1
32 bl dcache_clean_poc
32 ic ialluis33 ic ialluis
33 34
34 /*35 /*
@@ -36,8 +37,8 @@ SYM_CODE_START(efi_enter_kernel) · 1 unmodified line
36 * so that we can safely disable the MMU and caches.37 * so that we can safely disable the MMU and caches.
37 */38 */
38 adr x0, 0f39 adr x0, 0f
39 ldr w1, 3f40 adr x1, 3f
40 bl __clean_dcache_area_poc41 bl dcache_clean_poc
410:420:
42 /* Turn off Dcache and MMU */43 /* Turn off Dcache and MMU */
43 mrs x0, CurrentEL44 mrs x0, CurrentEL
@@ -64,5 +65,5 @@ SYM_CODE_START(efi_enter_kernel) · 20 unmodified lines
64 mov x2, xzr65 mov x2, xzr
65 mov x3, xzr66 mov x3, xzr
66 br x1967 br x19
683:
67SYM_CODE_END(efi_enter_kernel)69SYM_CODE_END(efi_enter_kernel)
683: .long . - 0b
arch/arm64/kernel/entry-common.c+238−18
@@ -6,7 +6,11 @@ · 5 unmodified lines
6 */6 */
7 7
8#include <linux/context_tracking.h>8#include <linux/context_tracking.h>
9#include <linux/linkage.h>
10#include <linux/lockdep.h>
9#include <linux/ptrace.h>11#include <linux/ptrace.h>
12#include <linux/sched.h>
13#include <linux/sched/debug.h>
10#include <linux/thread_info.h>14#include <linux/thread_info.h>
11 15
12#include <asm/cpufeature.h>16#include <asm/cpufeature.h>
@@ -15,7 +19,11 @@ · 2 unmodified lines
15#include <asm/exception.h>19#include <asm/exception.h>
16#include <asm/kprobes.h>20#include <asm/kprobes.h>
17#include <asm/mmu.h>21#include <asm/mmu.h>
22#include <asm/processor.h>
23#include <asm/sdei.h>
24#include <asm/stacktrace.h>
18#include <asm/sysreg.h>25#include <asm/sysreg.h>
26#include <asm/system_misc.h>
19 27
20/*28/*
21 * This is intended to match the logic in irqentry_enter(), handling the kernel29 * This is intended to match the logic in irqentry_enter(), handling the kernel
@@ -67,7 +75,7 @@ static void noinstr exit_to_kernel_mode(struct pt_regs *regs) · 45 unmodified lines
67 }75 }
68}76}
69 77
70void noinstr arm64_enter_nmi(struct pt_regs *regs)78static void noinstr arm64_enter_nmi(struct pt_regs *regs)
71{79{
72 regs->lockdep_hardirqs = lockdep_hardirqs_enabled();80 regs->lockdep_hardirqs = lockdep_hardirqs_enabled();
73 81
@@ -80,7 +88,7 @@ void noinstr arm64_enter_nmi(struct pt_regs *regs) · 6 unmodified lines
80 ftrace_nmi_enter();88 ftrace_nmi_enter();
81}89}
82 90
83void noinstr arm64_exit_nmi(struct pt_regs *regs)91static void noinstr arm64_exit_nmi(struct pt_regs *regs)
84{92{
85 bool restore = regs->lockdep_hardirqs;93 bool restore = regs->lockdep_hardirqs;
86 94
@@ -97,7 +105,7 @@ void noinstr arm64_exit_nmi(struct pt_regs *regs) · 10 unmodified lines
97 __nmi_exit();105 __nmi_exit();
98}106}
99 107
100asmlinkage void noinstr enter_el1_irq_or_nmi(struct pt_regs *regs)108static void noinstr enter_el1_irq_or_nmi(struct pt_regs *regs)
101{109{
102 if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs))110 if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs))
103 arm64_enter_nmi(regs);111 arm64_enter_nmi(regs);
@@ -105,7 +113,7 @@ asmlinkage void noinstr enter_el1_irq_or_nmi(struct pt_regs *regs) · 1 unmodified line
105 enter_from_kernel_mode(regs);113 enter_from_kernel_mode(regs);
106}114}
107 115
108asmlinkage void noinstr exit_el1_irq_or_nmi(struct pt_regs *regs)116static void noinstr exit_el1_irq_or_nmi(struct pt_regs *regs)
109{117{
110 if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs))118 if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs))
111 arm64_exit_nmi(regs);119 arm64_exit_nmi(regs);
@@ -113,6 +121,65 @@ asmlinkage void noinstr exit_el1_irq_or_nmi(struct pt_regs *regs) · 1 unmodified line
113 exit_to_kernel_mode(regs);121 exit_to_kernel_mode(regs);
114}122}
115 123
124static void __sched arm64_preempt_schedule_irq(void)
125{
126 lockdep_assert_irqs_disabled();
127
128 /*
129 * DAIF.DA are cleared at the start of IRQ/FIQ handling, and when GIC
130 * priority masking is used the GIC irqchip driver will clear DAIF.IF
131 * using gic_arch_enable_irqs() for normal IRQs. If anything is set in
132 * DAIF we must have handled an NMI, so skip preemption.
133 */
134 if (system_uses_irq_prio_masking() && read_sysreg(daif))
135 return;
136
137 /*
138 * Preempting a task from an IRQ means we leave copies of PSTATE
139 * on the stack. cpufeature's enable calls may modify PSTATE, but
140 * resuming one of these preempted tasks would undo those changes.
141 *
142 * Only allow a task to be preempted once cpufeatures have been
143 * enabled.
144 */
145 if (system_capabilities_finalized())
146 preempt_schedule_irq();
147}
148
149static void do_interrupt_handler(struct pt_regs *regs,
150 void (*handler)(struct pt_regs *))
151{
152 if (on_thread_stack())
153 call_on_irq_stack(regs, handler);
154 else
155 handler(regs);
156}
157
158extern void (*handle_arch_irq)(struct pt_regs *);
159extern void (*handle_arch_fiq)(struct pt_regs *);
160
161static void noinstr __panic_unhandled(struct pt_regs *regs, const char *vector,
162 unsigned int esr)
163{
164 arm64_enter_nmi(regs);
165
166 console_verbose();
167
168 pr_crit("Unhandled %s exception on CPU%d, ESR 0x%08x -- %s\n",
169 vector, smp_processor_id(), esr,
170 esr_get_class_string(esr));
171
172 __show_regs(regs);
173 panic("Unhandled exception");
174}
175
176#define UNHANDLED(el, regsize, vector) \
177asmlinkage void noinstr el##_##regsize##_##vector##_handler(struct pt_regs *regs) \
178{ \
179 const char *desc = #regsize "-bit " #el " " #vector; \
180 __panic_unhandled(regs, desc, read_sysreg(esr_el1)); \
181}
182
116#ifdef CONFIG_ARM64_ERRATUM_1463225183#ifdef CONFIG_ARM64_ERRATUM_1463225
117static DEFINE_PER_CPU(int, __in_cortex_a76_erratum_1463225_wa);184static DEFINE_PER_CPU(int, __in_cortex_a76_erratum_1463225_wa);
118 185
@@ -162,6 +229,11 @@ static bool cortex_a76_erratum_1463225_debug_handler(struct pt_regs *regs) · 43 unmodified lines
162}229}
163#endif /* CONFIG_ARM64_ERRATUM_1463225 */230#endif /* CONFIG_ARM64_ERRATUM_1463225 */
164 231
232UNHANDLED(el1t, 64, sync)
233UNHANDLED(el1t, 64, irq)
234UNHANDLED(el1t, 64, fiq)
235UNHANDLED(el1t, 64, error)
236
165static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)237static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
166{238{
167 unsigned long far = read_sysreg(far_el1);239 unsigned long far = read_sysreg(far_el1);
@@ -193,15 +265,6 @@ static void noinstr el1_undef(struct pt_regs *regs) · 25 unmodified lines
193 exit_to_kernel_mode(regs);265 exit_to_kernel_mode(regs);
194}266}
195 267
196static void noinstr el1_inv(struct pt_regs *regs, unsigned long esr)
197{
198 enter_from_kernel_mode(regs);
199 local_daif_inherit(regs);
200 bad_mode(regs, 0, esr);
201 local_daif_mask();
202 exit_to_kernel_mode(regs);
203}
204
205static void noinstr arm64_enter_el1_dbg(struct pt_regs *regs)268static void noinstr arm64_enter_el1_dbg(struct pt_regs *regs)
206{269{
207 regs->lockdep_hardirqs = lockdep_hardirqs_enabled();270 regs->lockdep_hardirqs = lockdep_hardirqs_enabled();
@@ -245,7 +308,7 @@ static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr) · 37 unmodified lines
245 exit_to_kernel_mode(regs);308 exit_to_kernel_mode(regs);
246}309}
247 310
248asmlinkage void noinstr el1_sync_handler(struct pt_regs *regs)311asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
249{312{
250 unsigned long esr = read_sysreg(esr_el1);313 unsigned long esr = read_sysreg(esr_el1);
251 314
@@ -275,10 +338,50 @@ asmlinkage void noinstr el1_sync_handler(struct pt_regs *regs) · 23 unmodified lines
275 el1_fpac(regs, esr);338 el1_fpac(regs, esr);
276 break;339 break;
277 default:340 default:
278 el1_inv(regs, esr);341 __panic_unhandled(regs, "64-bit el1h sync", esr);
279 }342 }
280}343}
281 344
345static void noinstr el1_interrupt(struct pt_regs *regs,
346 void (*handler)(struct pt_regs *))
347{
348 write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
349
350 enter_el1_irq_or_nmi(regs);
351 do_interrupt_handler(regs, handler);
352
353 /*
354 * Note: thread_info::preempt_count includes both thread_info::count
355 * and thread_info::need_resched, and is not equivalent to
356 * preempt_count().
357 */
358 if (IS_ENABLED(CONFIG_PREEMPTION) &&
359 READ_ONCE(current_thread_info()->preempt_count) == 0)
360 arm64_preempt_schedule_irq();
361
362 exit_el1_irq_or_nmi(regs);
363}
364
365asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
366{
367 el1_interrupt(regs, handle_arch_irq);
368}
369
370asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
371{
372 el1_interrupt(regs, handle_arch_fiq);
373}
374
375asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
376{
377 unsigned long esr = read_sysreg(esr_el1);
378
379 local_daif_restore(DAIF_ERRCTX);
380 arm64_enter_nmi(regs);
381 do_serror(regs, esr);
382 arm64_exit_nmi(regs);
383}
384
282asmlinkage void noinstr enter_from_user_mode(void)385asmlinkage void noinstr enter_from_user_mode(void)
283{386{
284 lockdep_hardirqs_off(CALLER_ADDR0);387 lockdep_hardirqs_off(CALLER_ADDR0);
@@ -398,7 +501,7 @@ static void noinstr el0_dbg(struct pt_regs *regs, unsigned long esr) · 113 unmodified lines
398 501
399 enter_from_user_mode();502 enter_from_user_mode();
400 do_debug_exception(far, esr, regs);503 do_debug_exception(far, esr, regs);
401 local_daif_restore(DAIF_PROCCTX_NOIRQ);504 local_daif_restore(DAIF_PROCCTX);
402}505}
403 506
404static void noinstr el0_svc(struct pt_regs *regs)507static void noinstr el0_svc(struct pt_regs *regs)
@@ -415,7 +518,7 @@ static void noinstr el0_fpac(struct pt_regs *regs, unsigned long esr) · 10 unmodified lines
415 do_ptrauth_fault(regs, esr);518 do_ptrauth_fault(regs, esr);
416}519}
417 520
418asmlinkage void noinstr el0_sync_handler(struct pt_regs *regs)521asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
419{522{
420 unsigned long esr = read_sysreg(esr_el1);523 unsigned long esr = read_sysreg(esr_el1);
421 524
@@ -468,6 +571,56 @@ asmlinkage void noinstr el0_sync_handler(struct pt_regs *regs) · 46 unmodified lines
468 }571 }
469}572}
470 573
574static void noinstr el0_interrupt(struct pt_regs *regs,
575 void (*handler)(struct pt_regs *))
576{
577 enter_from_user_mode();
578
579 write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
580
581 if (regs->pc & BIT(55))
582 arm64_apply_bp_hardening();
583
584 do_interrupt_handler(regs, handler);
585}
586
587static void noinstr __el0_irq_handler_common(struct pt_regs *regs)
588{
589 el0_interrupt(regs, handle_arch_irq);
590}
591
592asmlinkage void noinstr el0t_64_irq_handler(struct pt_regs *regs)
593{
594 __el0_irq_handler_common(regs);
595}
596
597static void noinstr __el0_fiq_handler_common(struct pt_regs *regs)
598{
599 el0_interrupt(regs, handle_arch_fiq);
600}
601
602asmlinkage void noinstr el0t_64_fiq_handler(struct pt_regs *regs)
603{
604 __el0_fiq_handler_common(regs);
605}
606
607static void __el0_error_handler_common(struct pt_regs *regs)
608{
609 unsigned long esr = read_sysreg(esr_el1);
610
611 enter_from_user_mode();
612 local_daif_restore(DAIF_ERRCTX);
613 arm64_enter_nmi(regs);
614 do_serror(regs, esr);
615 arm64_exit_nmi(regs);
616 local_daif_restore(DAIF_PROCCTX);
617}
618
619asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
620{
621 __el0_error_handler_common(regs);
622}
623
471#ifdef CONFIG_COMPAT624#ifdef CONFIG_COMPAT
472static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)625static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
473{626{
@@ -483,7 +636,7 @@ static void noinstr el0_svc_compat(struct pt_regs *regs) · 9 unmodified lines
483 do_el0_svc_compat(regs);636 do_el0_svc_compat(regs);
484}637}
485 638
486asmlinkage void noinstr el0_sync_compat_handler(struct pt_regs *regs)639asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
487{640{
488 unsigned long esr = read_sysreg(esr_el1);641 unsigned long esr = read_sysreg(esr_el1);
489 642
@@ -526,4 +679,71 @@ asmlinkage void noinstr el0_sync_compat_handler(struct pt_regs *regs) · 36 unmodified lines
526 el0_inv(regs, esr);679 el0_inv(regs, esr);
527 }680 }
528}681}
682
683asmlinkage void noinstr el0t_32_irq_handler(struct pt_regs *regs)
684{
685 __el0_irq_handler_common(regs);
686}
687
688asmlinkage void noinstr el0t_32_fiq_handler(struct pt_regs *regs)
689{
690 __el0_fiq_handler_common(regs);
691}
692
693asmlinkage void noinstr el0t_32_error_handler(struct pt_regs *regs)
694{
695 __el0_error_handler_common(regs);
696}
697#else /* CONFIG_COMPAT */
698UNHANDLED(el0t, 32, sync)
699UNHANDLED(el0t, 32, irq)
700UNHANDLED(el0t, 32, fiq)
701UNHANDLED(el0t, 32, error)
529#endif /* CONFIG_COMPAT */702#endif /* CONFIG_COMPAT */
703
704#ifdef CONFIG_VMAP_STACK
705asmlinkage void noinstr handle_bad_stack(struct pt_regs *regs)
706{
707 unsigned int esr = read_sysreg(esr_el1);
708 unsigned long far = read_sysreg(far_el1);
709
710 arm64_enter_nmi(regs);
711 panic_bad_stack(regs, esr, far);
712}
713#endif /* CONFIG_VMAP_STACK */
714
715#ifdef CONFIG_ARM_SDE_INTERFACE
716asmlinkage noinstr unsigned long
717__sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
718{
719 unsigned long ret;
720
721 /*
722 * We didn't take an exception to get here, so the HW hasn't
723 * set/cleared bits in PSTATE that we may rely on.
724 *
725 * The original SDEI spec (ARM DEN 0054A) can be read ambiguously as to
726 * whether PSTATE bits are inherited unchanged or generated from
727 * scratch, and the TF-A implementation always clears PAN and always
728 * clears UAO. There are no other known implementations.
729 *
730 * Subsequent revisions (ARM DEN 0054B) follow the usual rules for how
731 * PSTATE is modified upon architectural exceptions, and so PAN is
732 * either inherited or set per SCTLR_ELx.SPAN, and UAO is always
733 * cleared.
734 *
735 * We must explicitly reset PAN to the expected state, including
736 * clearing it when the host isn't using it, in case a VM had it set.
737 */
738 if (system_uses_hw_pan())
739 set_pstate_pan(1);
740 else if (cpu_has_pan())
741 set_pstate_pan(0);
742
743 arm64_enter_nmi(regs);
744 ret = do_sdei_event(regs, arg);
745 arm64_exit_nmi(regs);
746
747 return ret;
748}
749#endif /* CONFIG_ARM_SDE_INTERFACE */
arch/arm64/kernel/entry-fpsimd.S+15−7
@@ -63,16 +63,24 @@ SYM_FUNC_END(sve_set_vq) · 62 unmodified lines
63 * and the rest zeroed. All the other SVE registers will be zeroed.63 * and the rest zeroed. All the other SVE registers will be zeroed.
64 */64 */
65SYM_FUNC_START(sve_load_from_fpsimd_state)65SYM_FUNC_START(sve_load_from_fpsimd_state)
66 sve_load_vq x1, x2, x366 sve_load_vq x1, x2, x3
67 fpsimd_restore x0, 867 fpsimd_restore x0, 8
68 _for n, 0, 15, _sve_pfalse \n68 sve_flush_p_ffr
69 _sve_wrffr 069 ret
70 ret
71SYM_FUNC_END(sve_load_from_fpsimd_state)70SYM_FUNC_END(sve_load_from_fpsimd_state)
72 71
73/* Zero all SVE registers but the first 128-bits of each vector */72/*
73 * Zero all SVE registers but the first 128-bits of each vector
74 *
75 * VQ must already be configured by caller, any further updates of VQ
76 * will need to ensure that the register state remains valid.
77 *
78 * x0 = VQ - 1
79 */
74SYM_FUNC_START(sve_flush_live)80SYM_FUNC_START(sve_flush_live)
75 sve_flush81 cbz x0, 1f // A VQ-1 of 0 is 128 bits so no extra Z state
82 sve_flush_z
831: sve_flush_p_ffr
76 ret84 ret
77SYM_FUNC_END(sve_flush_live)85SYM_FUNC_END(sve_flush_live)
78 86
arch/arm64/kernel/entry.S+96−273
@@ -33,12 +33,6 @@ · 32 unmodified lines
33 * Context tracking and irqflag tracing need to instrument transitions between33 * Context tracking and irqflag tracing need to instrument transitions between
34 * user and kernel mode.34 * user and kernel mode.
35 */35 */
36 .macro user_exit_irqoff
37#if defined(CONFIG_CONTEXT_TRACKING) || defined(CONFIG_TRACE_IRQFLAGS)
38 bl enter_from_user_mode
39#endif
40 .endm
41
42 .macro user_enter_irqoff36 .macro user_enter_irqoff
43#if defined(CONFIG_CONTEXT_TRACKING) || defined(CONFIG_TRACE_IRQFLAGS)37#if defined(CONFIG_CONTEXT_TRACKING) || defined(CONFIG_TRACE_IRQFLAGS)
44 bl exit_to_user_mode38 bl exit_to_user_mode
@@ -51,16 +45,7 @@ · 6 unmodified lines
51 .endr45 .endr
52 .endm46 .endm
53 47
54/*48 .macro kernel_ventry, el:req, ht:req, regsize:req, label:req
55 * Bad Abort numbers
56 *-----------------
57 */
58#define BAD_SYNC 0
59#define BAD_IRQ 1
60#define BAD_FIQ 2
61#define BAD_ERROR 3
62
63 .macro kernel_ventry, el, label, regsize = 64
64 .align 749 .align 7
65#ifdef CONFIG_UNMAP_KERNEL_AT_EL050#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
66 .if \el == 051 .if \el == 0
@@ -87,7 +72,7 @@ alternative_else_nop_endif · 20 unmodified lines
87 tbnz x0, #THREAD_SHIFT, 0f72 tbnz x0, #THREAD_SHIFT, 0f
88 sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x073 sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0
89 sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp74 sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp
90 b el\()\el\()_\label75 b el\el\ht\()_\regsize\()_\label
91 76
920:770:
93 /*78 /*
@@ -119,7 +104,7 @@ alternative_else_nop_endif · 25 unmodified lines
119 sub sp, sp, x0104 sub sp, sp, x0
120 mrs x0, tpidrro_el0105 mrs x0, tpidrro_el0
121#endif106#endif
122 b el\()\el\()_\label107 b el\el\ht\()_\regsize\()_\label
123 .endm108 .endm
124 109
125 .macro tramp_alias, dst, sym110 .macro tramp_alias, dst, sym
@@ -275,7 +260,7 @@ alternative_else_nop_endif · 149 unmodified lines
275 260
276 mte_set_kernel_gcr x22, x23261 mte_set_kernel_gcr x22, x23
277 262
278 scs_load tsk, x20263 scs_load tsk
279 .else264 .else
280 add x21, sp, #PT_REGS_SIZE265 add x21, sp, #PT_REGS_SIZE
281 get_current_task tsk266 get_current_task tsk
@@ -285,7 +270,7 @@ alternative_else_nop_endif · 3 unmodified lines
285 stp lr, x21, [sp, #S_LR]270 stp lr, x21, [sp, #S_LR]
286 271
287 /*272 /*
288 * For exceptions from EL0, create a terminal frame record.273 * For exceptions from EL0, create a final frame record.
289 * For exceptions from EL1, create a synthetic frame record so the274 * For exceptions from EL1, create a synthetic frame record so the
290 * interrupted code shows up in the backtrace.275 * interrupted code shows up in the backtrace.
291 */276 */
@@ -375,7 +360,7 @@ alternative_if ARM64_WORKAROUND_845719 · 83 unmodified lines
375alternative_else_nop_endif360alternative_else_nop_endif
376#endif361#endif
3773:3623:
378 scs_save tsk, x0363 scs_save tsk
379 364
380#ifdef CONFIG_ARM64_PTR_AUTH365#ifdef CONFIG_ARM64_PTR_AUTH
381alternative_if ARM64_HAS_ADDRESS_AUTH366alternative_if ARM64_HAS_ADDRESS_AUTH
@@ -486,63 +471,12 @@ SYM_CODE_START_LOCAL(__swpan_exit_el0) · 104 unmodified lines
486SYM_CODE_END(__swpan_exit_el0)471SYM_CODE_END(__swpan_exit_el0)
487#endif472#endif
488 473
489 .macro irq_stack_entry
490 mov x19, sp // preserve the original sp
491#ifdef CONFIG_SHADOW_CALL_STACK
492 mov x24, scs_sp // preserve the original shadow stack
493#endif
494
495 /*
496 * Compare sp with the base of the task stack.
497 * If the top ~(THREAD_SIZE - 1) bits match, we are on a task stack,
498 * and should switch to the irq stack.
499 */
500 ldr x25, [tsk, TSK_STACK]
501 eor x25, x25, x19
502 and x25, x25, #~(THREAD_SIZE - 1)
503 cbnz x25, 9998f
504
505 ldr_this_cpu x25, irq_stack_ptr, x26
506 mov x26, #IRQ_STACK_SIZE
507 add x26, x25, x26
508
509 /* switch to the irq stack */
510 mov sp, x26
511
512#ifdef CONFIG_SHADOW_CALL_STACK
513 /* also switch to the irq shadow stack */
514 ldr_this_cpu scs_sp, irq_shadow_call_stack_ptr, x26
515#endif
516
5179998:
518 .endm
519
520 /*
521 * The callee-saved regs (x19-x29) should be preserved between
522 * irq_stack_entry and irq_stack_exit, but note that kernel_entry
523 * uses x20-x23 to store data for later use.
524 */
525 .macro irq_stack_exit
526 mov sp, x19
527#ifdef CONFIG_SHADOW_CALL_STACK
528 mov scs_sp, x24
529#endif
530 .endm
531
532/* GPRs used by entry code */474/* GPRs used by entry code */
533tsk .req x28 // current thread_info475tsk .req x28 // current thread_info
534 476
535/*477/*
536 * Interrupt handling.478 * Interrupt handling.
537 */479 */
538 .macro irq_handler, handler:req
539 ldr_l x1, \handler
540 mov x0, sp
541 irq_stack_entry
542 blr x1
543 irq_stack_exit
544 .endm
545
546 .macro gic_prio_kentry_setup, tmp:req480 .macro gic_prio_kentry_setup, tmp:req
547#ifdef CONFIG_ARM64_PSEUDO_NMI481#ifdef CONFIG_ARM64_PSEUDO_NMI
548 alternative_if ARM64_HAS_IRQ_PRIO_MASKING482 alternative_if ARM64_HAS_IRQ_PRIO_MASKING
@@ -552,45 +486,6 @@ tsk .req x28 // current thread_info · 3 unmodified lines
552#endif486#endif
553 .endm487 .endm
554 488
555 .macro el1_interrupt_handler, handler:req
556 enable_da
557
558 mov x0, sp
559 bl enter_el1_irq_or_nmi
560
561 irq_handler \handler
562
563#ifdef CONFIG_PREEMPTION
564 ldr x24, [tsk, #TSK_TI_PREEMPT] // get preempt count
565alternative_if ARM64_HAS_IRQ_PRIO_MASKING
566 /*
567 * DA were cleared at start of handling, and IF are cleared by
568 * the GIC irqchip driver using gic_arch_enable_irqs() for
569 * normal IRQs. If anything is set, it means we come back from
570 * an NMI instead of a normal IRQ, so skip preemption
571 */
572 mrs x0, daif
573 orr x24, x24, x0
574alternative_else_nop_endif
575 cbnz x24, 1f // preempt count != 0 || NMI return path
576 bl arm64_preempt_schedule_irq // irq en/disable is done inside
5771:
578#endif
579
580 mov x0, sp
581 bl exit_el1_irq_or_nmi
582 .endm
583
584 .macro el0_interrupt_handler, handler:req
585 user_exit_irqoff
586 enable_da
587
588 tbz x22, #55, 1f
589 bl do_el0_irq_bp_hardening
5901:
591 irq_handler \handler
592 .endm
593
594 .text489 .text
595 490
596/*491/*
@@ -600,32 +495,25 @@ alternative_else_nop_endif · 3 unmodified lines
600 495
601 .align 11496 .align 11
602SYM_CODE_START(vectors)497SYM_CODE_START(vectors)
603 kernel_ventry 1, sync_invalid // Synchronous EL1t498 kernel_ventry 1, t, 64, sync // Synchronous EL1t
604 kernel_ventry 1, irq_invalid // IRQ EL1t499 kernel_ventry 1, t, 64, irq // IRQ EL1t
605 kernel_ventry 1, fiq_invalid // FIQ EL1t500 kernel_ventry 1, t, 64, fiq // FIQ EL1h
606 kernel_ventry 1, error_invalid // Error EL1t501 kernel_ventry 1, t, 64, error // Error EL1t
607 502
608 kernel_ventry 1, sync // Synchronous EL1h503 kernel_ventry 1, h, 64, sync // Synchronous EL1h
609 kernel_ventry 1, irq // IRQ EL1h504 kernel_ventry 1, h, 64, irq // IRQ EL1h
610 kernel_ventry 1, fiq // FIQ EL1h505 kernel_ventry 1, h, 64, fiq // FIQ EL1h
611 kernel_ventry 1, error // Error EL1h506 kernel_ventry 1, h, 64, error // Error EL1h
612 507
613 kernel_ventry 0, sync // Synchronous 64-bit EL0508 kernel_ventry 0, t, 64, sync // Synchronous 64-bit EL0
614 kernel_ventry 0, irq // IRQ 64-bit EL0509 kernel_ventry 0, t, 64, irq // IRQ 64-bit EL0
615 kernel_ventry 0, fiq // FIQ 64-bit EL0510 kernel_ventry 0, t, 64, fiq // FIQ 64-bit EL0
616 kernel_ventry 0, error // Error 64-bit EL0511 kernel_ventry 0, t, 64, error // Error 64-bit EL0
617 512
618#ifdef CONFIG_COMPAT513 kernel_ventry 0, t, 32, sync // Synchronous 32-bit EL0
619 kernel_ventry 0, sync_compat, 32 // Synchronous 32-bit EL0514 kernel_ventry 0, t, 32, irq // IRQ 32-bit EL0
620 kernel_ventry 0, irq_compat, 32 // IRQ 32-bit EL0515 kernel_ventry 0, t, 32, fiq // FIQ 32-bit EL0
621 kernel_ventry 0, fiq_compat, 32 // FIQ 32-bit EL0516 kernel_ventry 0, t, 32, error // Error 32-bit EL0
622 kernel_ventry 0, error_compat, 32 // Error 32-bit EL0
623#else
624 kernel_ventry 0, sync_invalid, 32 // Synchronous 32-bit EL0
625 kernel_ventry 0, irq_invalid, 32 // IRQ 32-bit EL0
626 kernel_ventry 0, fiq_invalid, 32 // FIQ 32-bit EL0
627 kernel_ventry 0, error_invalid, 32 // Error 32-bit EL0
628#endif
629SYM_CODE_END(vectors)517SYM_CODE_END(vectors)
630 518
631#ifdef CONFIG_VMAP_STACK519#ifdef CONFIG_VMAP_STACK
@@ -656,147 +544,46 @@ __bad_stack: · 24 unmodified lines
656 ASM_BUG()544 ASM_BUG()
657#endif /* CONFIG_VMAP_STACK */545#endif /* CONFIG_VMAP_STACK */
658 546
659/*547
660 * Invalid mode handlers548 .macro entry_handler el:req, ht:req, regsize:req, label:req
661 */549SYM_CODE_START_LOCAL(el\el\ht\()_\regsize\()_\label)
662 .macro inv_entry, el, reason, regsize = 64
663 kernel_entry \el, \regsize550 kernel_entry \el, \regsize
664 mov x0, sp551 mov x0, sp
665 mov x1, #\reason552 bl el\el\ht\()_\regsize\()_\label\()_handler
666 mrs x2, esr_el1553 .if \el == 0
667 bl bad_mode554 b ret_to_user
668 ASM_BUG()555 .else
556 b ret_to_kernel
557 .endif
558SYM_CODE_END(el\el\ht\()_\regsize\()_\label)
669 .endm559 .endm
670 560
671SYM_CODE_START_LOCAL(el0_sync_invalid)
672 inv_entry 0, BAD_SYNC
673SYM_CODE_END(el0_sync_invalid)
674
675SYM_CODE_START_LOCAL(el0_irq_invalid)
676 inv_entry 0, BAD_IRQ
677SYM_CODE_END(el0_irq_invalid)
678
679SYM_CODE_START_LOCAL(el0_fiq_invalid)
680 inv_entry 0, BAD_FIQ
681SYM_CODE_END(el0_fiq_invalid)
682
683SYM_CODE_START_LOCAL(el0_error_invalid)
684 inv_entry 0, BAD_ERROR
685SYM_CODE_END(el0_error_invalid)
686
687SYM_CODE_START_LOCAL(el1_sync_invalid)
688 inv_entry 1, BAD_SYNC
689SYM_CODE_END(el1_sync_invalid)
690
691SYM_CODE_START_LOCAL(el1_irq_invalid)
692 inv_entry 1, BAD_IRQ
693SYM_CODE_END(el1_irq_invalid)
694
695SYM_CODE_START_LOCAL(el1_fiq_invalid)
696 inv_entry 1, BAD_FIQ
697SYM_CODE_END(el1_fiq_invalid)
698
699SYM_CODE_START_LOCAL(el1_error_invalid)
700 inv_entry 1, BAD_ERROR
701SYM_CODE_END(el1_error_invalid)
702
703/*561/*
704 * EL1 mode handlers.562 * Early exception handlers
705 */563 */
706 .align 6564 entry_handler 1, t, 64, sync
707SYM_CODE_START_LOCAL_NOALIGN(el1_sync)565 entry_handler 1, t, 64, irq
708 kernel_entry 1566 entry_handler 1, t, 64, fiq
709 mov x0, sp567 entry_handler 1, t, 64, error
710 bl el1_sync_handler568
711 kernel_exit 1569 entry_handler 1, h, 64, sync
712SYM_CODE_END(el1_sync)570 entry_handler 1, h, 64, irq
713 571 entry_handler 1, h, 64, fiq
714 .align 6572 entry_handler 1, h, 64, error
715SYM_CODE_START_LOCAL_NOALIGN(el1_irq)573
716 kernel_entry 1574 entry_handler 0, t, 64, sync
717 el1_interrupt_handler handle_arch_irq575 entry_handler 0, t, 64, irq
718 kernel_exit 1576 entry_handler 0, t, 64, fiq
719SYM_CODE_END(el1_irq)577 entry_handler 0, t, 64, error
720 578
721SYM_CODE_START_LOCAL_NOALIGN(el1_fiq)579 entry_handler 0, t, 32, sync
722 kernel_entry 1580 entry_handler 0, t, 32, irq
723 el1_interrupt_handler handle_arch_fiq581 entry_handler 0, t, 32, fiq
724 kernel_exit 1582 entry_handler 0, t, 32, error
725SYM_CODE_END(el1_fiq)583
726 584SYM_CODE_START_LOCAL(ret_to_kernel)
727/*
728 * EL0 mode handlers.
729 */
730 .align 6
731SYM_CODE_START_LOCAL_NOALIGN(el0_sync)
732 kernel_entry 0
733 mov x0, sp
734 bl el0_sync_handler
735 b ret_to_user
736SYM_CODE_END(el0_sync)
737
738#ifdef CONFIG_COMPAT
739 .align 6
740SYM_CODE_START_LOCAL_NOALIGN(el0_sync_compat)
741 kernel_entry 0, 32
742 mov x0, sp
743 bl el0_sync_compat_handler
744 b ret_to_user
745SYM_CODE_END(el0_sync_compat)
746
747 .align 6
748SYM_CODE_START_LOCAL_NOALIGN(el0_irq_compat)
749 kernel_entry 0, 32
750 b el0_irq_naked
751SYM_CODE_END(el0_irq_compat)
752
753SYM_CODE_START_LOCAL_NOALIGN(el0_fiq_compat)
754 kernel_entry 0, 32
755 b el0_fiq_naked
756SYM_CODE_END(el0_fiq_compat)
757
758SYM_CODE_START_LOCAL_NOALIGN(el0_error_compat)
759 kernel_entry 0, 32
760 b el0_error_naked
761SYM_CODE_END(el0_error_compat)
762#endif
763
764 .align 6
765SYM_CODE_START_LOCAL_NOALIGN(el0_irq)
766 kernel_entry 0
767el0_irq_naked:
768 el0_interrupt_handler handle_arch_irq
769 b ret_to_user
770SYM_CODE_END(el0_irq)
771
772SYM_CODE_START_LOCAL_NOALIGN(el0_fiq)
773 kernel_entry 0
774el0_fiq_naked:
775 el0_interrupt_handler handle_arch_fiq
776 b ret_to_user
777SYM_CODE_END(el0_fiq)
778
779SYM_CODE_START_LOCAL(el1_error)
780 kernel_entry 1
781 mrs x1, esr_el1
782 enable_dbg
783 mov x0, sp
784 bl do_serror
785 kernel_exit 1585 kernel_exit 1
786SYM_CODE_END(el1_error)586SYM_CODE_END(ret_to_kernel)
787
788SYM_CODE_START_LOCAL(el0_error)
789 kernel_entry 0
790el0_error_naked:
791 mrs x25, esr_el1
792 user_exit_irqoff
793 enable_dbg
794 mov x0, sp
795 mov x1, x25
796 bl do_serror
797 enable_da
798 b ret_to_user
799SYM_CODE_END(el0_error)
800 587
801/*588/*
802 * "slow" syscall return path.589 * "slow" syscall return path.
@@ -979,8 +766,8 @@ SYM_FUNC_START(cpu_switch_to) · 176 unmodified lines
979 mov sp, x9766 mov sp, x9
980 msr sp_el0, x1767 msr sp_el0, x1
981 ptrauth_keys_install_kernel x1, x8, x9, x10768 ptrauth_keys_install_kernel x1, x8, x9, x10
982 scs_save x0, x8769 scs_save x0
983 scs_load x1, x8770 scs_load x1
984 ret771 ret
985SYM_FUNC_END(cpu_switch_to)772SYM_FUNC_END(cpu_switch_to)
986NOKPROBE(cpu_switch_to)773NOKPROBE(cpu_switch_to)
@@ -998,6 +785,42 @@ SYM_CODE_START(ret_from_fork) · 11 unmodified lines
998SYM_CODE_END(ret_from_fork)785SYM_CODE_END(ret_from_fork)
999NOKPROBE(ret_from_fork)786NOKPROBE(ret_from_fork)
1000 787
788/*
789 * void call_on_irq_stack(struct pt_regs *regs,
790 * void (*func)(struct pt_regs *));
791 *
792 * Calls func(regs) using this CPU's irq stack and shadow irq stack.
793 */
794SYM_FUNC_START(call_on_irq_stack)
795#ifdef CONFIG_SHADOW_CALL_STACK
796 stp scs_sp, xzr, [sp, #-16]!
797 ldr_this_cpu scs_sp, irq_shadow_call_stack_ptr, x17
798#endif
799 /* Create a frame record to save our LR and SP (implicit in FP) */
800 stp x29, x30, [sp, #-16]!
801 mov x29, sp
802
803 ldr_this_cpu x16, irq_stack_ptr, x17
804 mov x15, #IRQ_STACK_SIZE
805 add x16, x16, x15
806
807 /* Move to the new stack and call the function there */
808 mov sp, x16
809 blr x1
810
811 /*
812 * Restore the SP from the FP, and restore the FP and LR from the frame
813 * record.
814 */
815 mov sp, x29
816 ldp x29, x30, [sp], #16
817#ifdef CONFIG_SHADOW_CALL_STACK
818 ldp scs_sp, xzr, [sp], #16
819#endif
820 ret
821SYM_FUNC_END(call_on_irq_stack)
822NOKPROBE(call_on_irq_stack)
823
1001#ifdef CONFIG_ARM_SDE_INTERFACE824#ifdef CONFIG_ARM_SDE_INTERFACE
1002 825
1003#include <asm/sdei.h>826#include <asm/sdei.h>
arch/arm64/kernel/fpsimd.c+4−2
@@ -957,8 +957,10 @@ void do_sve_acc(unsigned int esr, struct pt_regs *regs) · 956 unmodified lines
957 * disabling the trap, otherwise update our in-memory copy.957 * disabling the trap, otherwise update our in-memory copy.
958 */958 */
959 if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) {959 if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) {
960 sve_set_vq(sve_vq_from_vl(current->thread.sve_vl) - 1);960 unsigned long vq_minus_one =
961 sve_flush_live();961 sve_vq_from_vl(current->thread.sve_vl) - 1;
962 sve_set_vq(vq_minus_one);
963 sve_flush_live(vq_minus_one);
962 fpsimd_bind_task_to_cpu();964 fpsimd_bind_task_to_cpu();
963 } else {965 } else {
964 fpsimd_to_sve(current);966 fpsimd_to_sve(current);
arch/arm64/kernel/ftrace.c+1−0
@@ -15,6 +15,7 @@ · 14 unmodified lines
15#include <asm/debug-monitors.h>15#include <asm/debug-monitors.h>
16#include <asm/ftrace.h>16#include <asm/ftrace.h>
17#include <asm/insn.h>17#include <asm/insn.h>
18#include <asm/patching.h>
18 19
19#ifdef CONFIG_DYNAMIC_FTRACE20#ifdef CONFIG_DYNAMIC_FTRACE
20/*21/*
arch/arm64/kernel/head.S+44−32
@@ -16,6 +16,7 @@ · 15 unmodified lines
16#include <asm/asm_pointer_auth.h>16#include <asm/asm_pointer_auth.h>
17#include <asm/assembler.h>17#include <asm/assembler.h>
18#include <asm/boot.h>18#include <asm/boot.h>
19#include <asm/bug.h>
19#include <asm/ptrace.h>20#include <asm/ptrace.h>
20#include <asm/asm-offsets.h>21#include <asm/asm-offsets.h>
21#include <asm/cache.h>22#include <asm/cache.h>
@@ -117,8 +118,8 @@ SYM_CODE_START_LOCAL(preserve_boot_args) · 95 unmodified lines
117 dmb sy // needed before dc ivac with118 dmb sy // needed before dc ivac with
118 // MMU off119 // MMU off
119 120
120 mov x1, #0x20 // 4 x 8 bytes121 add x1, x0, #0x20 // 4 x 8 bytes
121 b __inval_dcache_area // tail call122 b dcache_inval_poc // tail call
122SYM_CODE_END(preserve_boot_args)123SYM_CODE_END(preserve_boot_args)
123 124
124/*125/*
@@ -195,7 +196,7 @@ SYM_CODE_END(preserve_boot_args) · 70 unmodified lines
195 and \iend, \iend, \istart // iend = (vend >> shift) & (ptrs - 1)196 and \iend, \iend, \istart // iend = (vend >> shift) & (ptrs - 1)
196 mov \istart, \ptrs197 mov \istart, \ptrs
197 mul \istart, \istart, \count198 mul \istart, \istart, \count
198 add \iend, \iend, \istart // iend += (count - 1) * ptrs199 add \iend, \iend, \istart // iend += count * ptrs
199 // our entries span multiple tables200 // our entries span multiple tables
200 201
201 lsr \istart, \vstart, \shift202 lsr \istart, \vstart, \shift
@@ -268,8 +269,7 @@ SYM_FUNC_START_LOCAL(__create_page_tables) · 66 unmodified lines
268 */269 */
269 adrp x0, init_pg_dir270 adrp x0, init_pg_dir
270 adrp x1, init_pg_end271 adrp x1, init_pg_end
271 sub x1, x1, x0272 bl dcache_inval_poc
272 bl __inval_dcache_area
273 273
274 /*274 /*
275 * Clear the init page tables.275 * Clear the init page tables.
@@ -354,7 +354,6 @@ SYM_FUNC_START_LOCAL(__create_page_tables) · 78 unmodified lines
354#endif354#endif
3551:3551:
356 ldr_l x4, idmap_ptrs_per_pgd356 ldr_l x4, idmap_ptrs_per_pgd
357 mov x5, x3 // __pa(__idmap_text_start)
358 adr_l x6, __idmap_text_end // __pa(__idmap_text_end)357 adr_l x6, __idmap_text_end // __pa(__idmap_text_end)
359 358
360 map_memory x0, x1, x3, x6, x7, x3, x4, x10, x11, x12, x13, x14359 map_memory x0, x1, x3, x6, x7, x3, x4, x10, x11, x12, x13, x14
@@ -382,39 +381,57 @@ SYM_FUNC_START_LOCAL(__create_page_tables) · 21 unmodified lines
382 381
383 adrp x0, idmap_pg_dir382 adrp x0, idmap_pg_dir
384 adrp x1, idmap_pg_end383 adrp x1, idmap_pg_end
385 sub x1, x1, x0384 bl dcache_inval_poc
386 bl __inval_dcache_area
387 385
388 adrp x0, init_pg_dir386 adrp x0, init_pg_dir
389 adrp x1, init_pg_end387 adrp x1, init_pg_end
390 sub x1, x1, x0388 bl dcache_inval_poc
391 bl __inval_dcache_area
392 389
393 ret x28390 ret x28
394SYM_FUNC_END(__create_page_tables)391SYM_FUNC_END(__create_page_tables)
395 392
393 /*
394 * Initialize CPU registers with task-specific and cpu-specific context.
395 *
396 * Create a final frame record at task_pt_regs(current)->stackframe, so
397 * that the unwinder can identify the final frame record of any task by
398 * its location in the task stack. We reserve the entire pt_regs space
399 * for consistency with user tasks and kthreads.
400 */
401 .macro init_cpu_task tsk, tmp1, tmp2
402 msr sp_el0, \tsk
403
404 ldr \tmp1, [\tsk, #TSK_STACK]
405 add sp, \tmp1, #THREAD_SIZE
406 sub sp, sp, #PT_REGS_SIZE
407
408 stp xzr, xzr, [sp, #S_STACKFRAME]
409 add x29, sp, #S_STACKFRAME
410
411 scs_load \tsk
412
413 adr_l \tmp1, __per_cpu_offset
414 ldr w\tmp2, [\tsk, #TSK_CPU]
415 ldr \tmp1, [\tmp1, \tmp2, lsl #3]
416 set_this_cpu_offset \tmp1
417 .endm
418
396/*419/*
397 * The following fragment of code is executed with the MMU enabled.420 * The following fragment of code is executed with the MMU enabled.
398 *421 *
399 * x0 = __PHYS_OFFSET422 * x0 = __PHYS_OFFSET
400 */423 */
401SYM_FUNC_START_LOCAL(__primary_switched)424SYM_FUNC_START_LOCAL(__primary_switched)
402 adrp x4, init_thread_union425 adr_l x4, init_task
403 add sp, x4, #THREAD_SIZE426 init_cpu_task x4, x5, x6
404 adr_l x5, init_task
405 msr sp_el0, x5 // Save thread_info
406 427
407 adr_l x8, vectors // load VBAR_EL1 with virtual428 adr_l x8, vectors // load VBAR_EL1 with virtual
408 msr vbar_el1, x8 // vector table address429 msr vbar_el1, x8 // vector table address
409 isb430 isb
410 431
411 stp xzr, x30, [sp, #-16]!432 stp x29, x30, [sp, #-16]!
412 mov x29, sp433 mov x29, sp
413 434
414#ifdef CONFIG_SHADOW_CALL_STACK
415 adr_l scs_sp, init_shadow_call_stack // Set shadow call stack
416#endif
417
418 str_l x21, __fdt_pointer, x5 // Save FDT pointer435 str_l x21, __fdt_pointer, x5 // Save FDT pointer
419 436
420 ldr_l x4, kimage_vaddr // Save the offset between437 ldr_l x4, kimage_vaddr // Save the offset between
@@ -446,10 +463,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) · 25 unmodified lines
4460:4630:
447#endif464#endif
448 bl switch_to_vhe // Prefer VHE if possible465 bl switch_to_vhe // Prefer VHE if possible
449 add sp, sp, #16466 ldp x29, x30, [sp], #16
450 mov x29, #0467 bl start_kernel
451 mov x30, #0468 ASM_BUG()
452 b start_kernel
453SYM_FUNC_END(__primary_switched)469SYM_FUNC_END(__primary_switched)
454 470
455 .pushsection ".rodata", "a"471 .pushsection ".rodata", "a"
@@ -551,7 +567,7 @@ SYM_FUNC_START_LOCAL(set_cpu_boot_mode_flag) · 95 unmodified lines
551 cmp w0, #BOOT_CPU_MODE_EL2567 cmp w0, #BOOT_CPU_MODE_EL2
552 b.ne 1f568 b.ne 1f
553 add x1, x1, #4569 add x1, x1, #4
5541: str w0, [x1] // This CPU has booted in EL15701: str w0, [x1] // Save CPU boot mode
555 dmb sy571 dmb sy
556 dc ivac, x1 // Invalidate potentially stale cache line572 dc ivac, x1 // Invalidate potentially stale cache line
557 ret573 ret
@@ -632,21 +648,17 @@ SYM_FUNC_START_LOCAL(__secondary_switched) · 74 unmodified lines
632 isb648 isb
633 649
634 adr_l x0, secondary_data650 adr_l x0, secondary_data
635 ldr x1, [x0, #CPU_BOOT_STACK] // get secondary_data.stack
636 cbz x1, __secondary_too_slow
637 mov sp, x1
638 ldr x2, [x0, #CPU_BOOT_TASK]651 ldr x2, [x0, #CPU_BOOT_TASK]
639 cbz x2, __secondary_too_slow652 cbz x2, __secondary_too_slow
640 msr sp_el0, x2653
641 scs_load x2, x3654 init_cpu_task x2, x1, x3
642 mov x29, #0
643 mov x30, #0
644 655
645#ifdef CONFIG_ARM64_PTR_AUTH656#ifdef CONFIG_ARM64_PTR_AUTH
646 ptrauth_keys_init_cpu x2, x3, x4, x5657 ptrauth_keys_init_cpu x2, x3, x4, x5
647#endif658#endif
648 659
649 b secondary_start_kernel660 bl secondary_start_kernel
661 ASM_BUG()
650SYM_FUNC_END(__secondary_switched)662SYM_FUNC_END(__secondary_switched)
651 663
652SYM_FUNC_START_LOCAL(__secondary_too_slow)664SYM_FUNC_START_LOCAL(__secondary_too_slow)
arch/arm64/kernel/hibernate-asm.S+4−3
@@ -45,7 +45,7 @@ · 44 unmodified lines
45 * Because this code has to be copied to a 'safe' page, it can't call out to45 * Because this code has to be copied to a 'safe' page, it can't call out to
46 * other functions by PC-relative address. Also remember that it may be46 * other functions by PC-relative address. Also remember that it may be
47 * mid-way through over-writing other functions. For this reason it contains47 * mid-way through over-writing other functions. For this reason it contains
48 * code from flush_icache_range() and uses the copy_page() macro.48 * code from caches_clean_inval_pou() and uses the copy_page() macro.
49 *49 *
50 * This 'safe' page is mapped via ttbr0, and executed from there. This function50 * This 'safe' page is mapped via ttbr0, and executed from there. This function
51 * switches to a copy of the linear map in ttbr1, performs the restore, then51 * switches to a copy of the linear map in ttbr1, performs the restore, then
@@ -87,11 +87,12 @@ SYM_CODE_START(swsusp_arch_suspend_exit) · 35 unmodified lines
87 copy_page x0, x1, x2, x3, x4, x5, x6, x7, x8, x987 copy_page x0, x1, x2, x3, x4, x5, x6, x7, x8, x9
88 88
89 add x1, x10, #PAGE_SIZE89 add x1, x10, #PAGE_SIZE
90 /* Clean the copied page to PoU - based on flush_icache_range() */90 /* Clean the copied page to PoU - based on caches_clean_inval_pou() */
91 raw_dcache_line_size x2, x391 raw_dcache_line_size x2, x3
92 sub x3, x2, #192 sub x3, x2, #1
93 bic x4, x10, x393 bic x4, x10, x3
942: dc cvau, x4 /* clean D line / unified line */942: /* clean D line / unified line */
95alternative_insn "dc cvau, x4", "dc civac, x4", ARM64_WORKAROUND_CLEAN_CACHE
95 add x4, x4, x296 add x4, x4, x2
96 cmp x4, x197 cmp x4, x1
97 b.lo 2b98 b.lo 2b
arch/arm64/kernel/hibernate.c+12−8
@@ -210,7 +210,7 @@ static int create_safe_exec_page(void *src_start, size_t length, · 209 unmodified lines
210 return -ENOMEM;210 return -ENOMEM;
211 211
212 memcpy(page, src_start, length);212 memcpy(page, src_start, length);
213 __flush_icache_range((unsigned long)page, (unsigned long)page + length);213 caches_clean_inval_pou((unsigned long)page, (unsigned long)page + length);
214 rc = trans_pgd_idmap_page(&trans_info, &trans_ttbr0, &t0sz, page);214 rc = trans_pgd_idmap_page(&trans_info, &trans_ttbr0, &t0sz, page);
215 if (rc)215 if (rc)
216 return rc;216 return rc;
@@ -240,8 +240,6 @@ static int create_safe_exec_page(void *src_start, size_t length, · 23 unmodified lines
240 return 0;240 return 0;
241}241}
242 242
243#define dcache_clean_range(start, end) __flush_dcache_area(start, (end - start))
244
245#ifdef CONFIG_ARM64_MTE243#ifdef CONFIG_ARM64_MTE
246 244
247static DEFINE_XARRAY(mte_pages);245static DEFINE_XARRAY(mte_pages);
@@ -383,13 +381,18 @@ int swsusp_arch_suspend(void) · 135 unmodified lines
383 ret = swsusp_save();381 ret = swsusp_save();
384 } else {382 } else {
385 /* Clean kernel core startup/idle code to PoC*/383 /* Clean kernel core startup/idle code to PoC*/
386 dcache_clean_range(__mmuoff_data_start, __mmuoff_data_end);384 dcache_clean_inval_poc((unsigned long)__mmuoff_data_start,
387 dcache_clean_range(__idmap_text_start, __idmap_text_end);385 (unsigned long)__mmuoff_data_end);
386 dcache_clean_inval_poc((unsigned long)__idmap_text_start,
387 (unsigned long)__idmap_text_end);
388 388
389 /* Clean kvm setup code to PoC? */389 /* Clean kvm setup code to PoC? */
390 if (el2_reset_needed()) {390 if (el2_reset_needed()) {
391 dcache_clean_range(__hyp_idmap_text_start, __hyp_idmap_text_end);391 dcache_clean_inval_poc(
392 dcache_clean_range(__hyp_text_start, __hyp_text_end);392 (unsigned long)__hyp_idmap_text_start,
393 (unsigned long)__hyp_idmap_text_end);
394 dcache_clean_inval_poc((unsigned long)__hyp_text_start,
395 (unsigned long)__hyp_text_end);
393 }396 }
394 397
395 swsusp_mte_restore_tags();398 swsusp_mte_restore_tags();
@@ -474,7 +477,8 @@ int swsusp_arch_resume(void) · 78 unmodified lines
474 * The hibernate exit text contains a set of el2 vectors, that will477 * The hibernate exit text contains a set of el2 vectors, that will
475 * be executed at el2 with the mmu off in order to reload hyp-stub.478 * be executed at el2 with the mmu off in order to reload hyp-stub.
476 */479 */
477 __flush_dcache_area(hibernate_exit, exit_size);480 dcache_clean_inval_poc((unsigned long)hibernate_exit,
481 (unsigned long)hibernate_exit + exit_size);
478 482
479 /*483 /*
480 * KASLR will cause the el2 vectors to be in a different location in484 * KASLR will cause the el2 vectors to be in a different location in
arch/arm64/kernel/idle.c+46−0
@@ -0,0 +1,46 @@
1// SPDX-License-Identifier: GPL-2.0-only
2/*
3 * Low-level idle sequences
4 */
5
6#include <linux/cpu.h>
7#include <linux/irqflags.h>
8
9#include <asm/barrier.h>
10#include <asm/cpuidle.h>
11#include <asm/cpufeature.h>
12#include <asm/sysreg.h>
13
14/*
15 * cpu_do_idle()
16 *
17 * Idle the processor (wait for interrupt).
18 *
19 * If the CPU supports priority masking we must do additional work to
20 * ensure that interrupts are not masked at the PMR (because the core will
21 * not wake up if we block the wake up signal in the interrupt controller).
22 */
23void noinstr cpu_do_idle(void)
24{
25 struct arm_cpuidle_irq_context context;
26
27 arm_cpuidle_save_irq_context(&context);
28
29 dsb(sy);
30 wfi();
31
32 arm_cpuidle_restore_irq_context(&context);
33}
34
35/*
36 * This is our default idle handler.
37 */
38void noinstr arch_cpu_idle(void)
39{
40 /*
41 * This should do all the clock switching and wait for interrupt
42 * tricks
43 */
44 cpu_do_idle();
45 raw_local_irq_enable();
46}
arch/arm64/kernel/idreg-override.c+2−1
@@ -237,7 +237,8 @@ asmlinkage void __init init_feature_override(void) · 236 unmodified lines
237 237
238 for (i = 0; i < ARRAY_SIZE(regs); i++) {238 for (i = 0; i < ARRAY_SIZE(regs); i++) {
239 if (regs[i]->override)239 if (regs[i]->override)
240 __flush_dcache_area(regs[i]->override,240 dcache_clean_inval_poc((unsigned long)regs[i]->override,
241 (unsigned long)regs[i]->override +
241 sizeof(*regs[i]->override));242 sizeof(*regs[i]->override));
242 }243 }
243}244}
arch/arm64/kernel/image-vars.h+1−1
@@ -35,7 +35,7 @@ __efistub_strnlen = __pi_strnlen; · 34 unmodified lines
35__efistub_strcmp = __pi_strcmp;35__efistub_strcmp = __pi_strcmp;
36__efistub_strncmp = __pi_strncmp;36__efistub_strncmp = __pi_strncmp;
37__efistub_strrchr = __pi_strrchr;37__efistub_strrchr = __pi_strrchr;
38__efistub___clean_dcache_area_poc = __pi___clean_dcache_area_poc;38__efistub_dcache_clean_poc = __pi_dcache_clean_poc;
39 39
40#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)40#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
41__efistub___memcpy = __pi_memcpy;41__efistub___memcpy = __pi_memcpy;
arch/arm64/kernel/jump_label.c+1−0
@@ -8,6 +8,7 @@ · 7 unmodified lines
8#include <linux/kernel.h>8#include <linux/kernel.h>
9#include <linux/jump_label.h>9#include <linux/jump_label.h>
10#include <asm/insn.h>10#include <asm/insn.h>
11#include <asm/patching.h>
11 12
12void arch_jump_label_transform(struct jump_entry *entry,13void arch_jump_label_transform(struct jump_entry *entry,
13 enum jump_label_type type)14 enum jump_label_type type)
arch/arm64/kernel/kaslr.c+9−3
@@ -72,7 +72,9 @@ u64 __init kaslr_early_init(void) · 71 unmodified lines
72 * we end up running with module randomization disabled.72 * we end up running with module randomization disabled.
73 */73 */
74 module_alloc_base = (u64)_etext - MODULES_VSIZE;74 module_alloc_base = (u64)_etext - MODULES_VSIZE;
75 __flush_dcache_area(&module_alloc_base, sizeof(module_alloc_base));75 dcache_clean_inval_poc((unsigned long)&module_alloc_base,
76 (unsigned long)&module_alloc_base +
77 sizeof(module_alloc_base));
76 78
77 /*79 /*
78 * Try to map the FDT early. If this fails, we simply bail,80 * Try to map the FDT early. If this fails, we simply bail,
@@ -170,8 +172,12 @@ u64 __init kaslr_early_init(void) · 91 unmodified lines
170 module_alloc_base += (module_range * (seed & ((1 << 21) - 1))) >> 21;172 module_alloc_base += (module_range * (seed & ((1 << 21) - 1))) >> 21;
171 module_alloc_base &= PAGE_MASK;173 module_alloc_base &= PAGE_MASK;
172 174
173 __flush_dcache_area(&module_alloc_base, sizeof(module_alloc_base));175 dcache_clean_inval_poc((unsigned long)&module_alloc_base,
174 __flush_dcache_area(&memstart_offset_seed, sizeof(memstart_offset_seed));176 (unsigned long)&module_alloc_base +
177 sizeof(module_alloc_base));
178 dcache_clean_inval_poc((unsigned long)&memstart_offset_seed,
179 (unsigned long)&memstart_offset_seed +
180 sizeof(memstart_offset_seed));
175 181
176 return offset;182 return offset;
177}183}
arch/arm64/kernel/kgdb.c+1−0
@@ -17,6 +17,7 @@ · 16 unmodified lines
17 17
18#include <asm/debug-monitors.h>18#include <asm/debug-monitors.h>
19#include <asm/insn.h>19#include <asm/insn.h>
20#include <asm/patching.h>
20#include <asm/traps.h>21#include <asm/traps.h>
21 22
22struct dbg_reg_def_t dbg_reg_def[DBG_MAX_REG_NUM] = {23struct dbg_reg_def_t dbg_reg_def[DBG_MAX_REG_NUM] = {
arch/arm64/kernel/machine_kexec.c+20−10
@@ -68,10 +68,16 @@ int machine_kexec_post_load(struct kimage *kimage) · 67 unmodified lines
68 kimage->arch.kern_reloc = __pa(reloc_code);68 kimage->arch.kern_reloc = __pa(reloc_code);
69 kexec_image_info(kimage);69 kexec_image_info(kimage);
70 70
71 /* Flush the reloc_code in preparation for its execution. */71 /*
72 __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size);72 * For execution with the MMU off, reloc_code needs to be cleaned to the
73 flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code +73 * PoC and invalidated from the I-cache.
74 arm64_relocate_new_kernel_size);74 */
75 dcache_clean_inval_poc((unsigned long)reloc_code,
76 (unsigned long)reloc_code +
77 arm64_relocate_new_kernel_size);
78 icache_inval_pou((uintptr_t)reloc_code,
79 (uintptr_t)reloc_code +
80 arm64_relocate_new_kernel_size);
75 81
76 return 0;82 return 0;
77}83}
@@ -102,16 +108,18 @@ static void kexec_list_flush(struct kimage *kimage) · 24 unmodified lines
102 108
103 for (entry = &kimage->head; ; entry++) {109 for (entry = &kimage->head; ; entry++) {
104 unsigned int flag;110 unsigned int flag;
105 void *addr;111 unsigned long addr;
106 112
107 /* flush the list entries. */113 /* flush the list entries. */
108 __flush_dcache_area(entry, sizeof(kimage_entry_t));114 dcache_clean_inval_poc((unsigned long)entry,
115 (unsigned long)entry +
116 sizeof(kimage_entry_t));
109 117
110 flag = *entry & IND_FLAGS;118 flag = *entry & IND_FLAGS;
111 if (flag == IND_DONE)119 if (flag == IND_DONE)
112 break;120 break;
113 121
114 addr = phys_to_virt(*entry & PAGE_MASK);122 addr = (unsigned long)phys_to_virt(*entry & PAGE_MASK);
115 123
116 switch (flag) {124 switch (flag) {
117 case IND_INDIRECTION:125 case IND_INDIRECTION:
@@ -120,7 +128,7 @@ static void kexec_list_flush(struct kimage *kimage) · 2 unmodified lines
120 break;128 break;
121 case IND_SOURCE:129 case IND_SOURCE:
122 /* flush the source pages. */130 /* flush the source pages. */
123 __flush_dcache_area(addr, PAGE_SIZE);131 dcache_clean_inval_poc(addr, addr + PAGE_SIZE);
124 break;132 break;
125 case IND_DESTINATION:133 case IND_DESTINATION:
126 break;134 break;
@@ -147,8 +155,10 @@ static void kexec_segment_flush(const struct kimage *kimage) · 20 unmodified lines
147 kimage->segment[i].memsz,155 kimage->segment[i].memsz,
148 kimage->segment[i].memsz / PAGE_SIZE);156 kimage->segment[i].memsz / PAGE_SIZE);
149 157
150 __flush_dcache_area(phys_to_virt(kimage->segment[i].mem),158 dcache_clean_inval_poc(
151 kimage->segment[i].memsz);159 (unsigned long)phys_to_virt(kimage->segment[i].mem),
160 (unsigned long)phys_to_virt(kimage->segment[i].mem) +
161 kimage->segment[i].memsz);
152 }162 }
153}163}
154 164
arch/arm64/kernel/patching.c+150−0
@@ -0,0 +1,150 @@
1// SPDX-License-Identifier: GPL-2.0-only
2#include <linux/kernel.h>
3#include <linux/mm.h>
4#include <linux/smp.h>
5#include <linux/spinlock.h>
6#include <linux/stop_machine.h>
7#include <linux/uaccess.h>
8
9#include <asm/cacheflush.h>
10#include <asm/fixmap.h>
11#include <asm/insn.h>
12#include <asm/kprobes.h>
13#include <asm/patching.h>
14#include <asm/sections.h>
15
16static DEFINE_RAW_SPINLOCK(patch_lock);
17
18static bool is_exit_text(unsigned long addr)
19{
20 /* discarded with init text/data */
21 return system_state < SYSTEM_RUNNING &&
22 addr >= (unsigned long)__exittext_begin &&
23 addr < (unsigned long)__exittext_end;
24}
25
26static bool is_image_text(unsigned long addr)
27{
28 return core_kernel_text(addr) || is_exit_text(addr);
29}
30
31static void __kprobes *patch_map(void *addr, int fixmap)
32{
33 unsigned long uintaddr = (uintptr_t) addr;
34 bool image = is_image_text(uintaddr);
35 struct page *page;
36
37 if (image)
38 page = phys_to_page(__pa_symbol(addr));
39 else if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX))
40 page = vmalloc_to_page(addr);
41 else
42 return addr;
43
44 BUG_ON(!page);
45 return (void *)set_fixmap_offset(fixmap, page_to_phys(page) +
46 (uintaddr & ~PAGE_MASK));
47}
48
49static void __kprobes patch_unmap(int fixmap)
50{
51 clear_fixmap(fixmap);
52}
53/*
54 * In ARMv8-A, A64 instructions have a fixed length of 32 bits and are always
55 * little-endian.
56 */
57int __kprobes aarch64_insn_read(void *addr, u32 *insnp)
58{
59 int ret;
60 __le32 val;
61
62 ret = copy_from_kernel_nofault(&val, addr, AARCH64_INSN_SIZE);
63 if (!ret)
64 *insnp = le32_to_cpu(val);
65
66 return ret;
67}
68
69static int __kprobes __aarch64_insn_write(void *addr, __le32 insn)
70{
71 void *waddr = addr;
72 unsigned long flags = 0;
73 int ret;
74
75 raw_spin_lock_irqsave(&patch_lock, flags);
76 waddr = patch_map(addr, FIX_TEXT_POKE0);
77
78 ret = copy_to_kernel_nofault(waddr, &insn, AARCH64_INSN_SIZE);
79
80 patch_unmap(FIX_TEXT_POKE0);
81 raw_spin_unlock_irqrestore(&patch_lock, flags);
82
83 return ret;
84}
85
86int __kprobes aarch64_insn_write(void *addr, u32 insn)
87{
88 return __aarch64_insn_write(addr, cpu_to_le32(insn));
89}
90
91int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn)
92{
93 u32 *tp = addr;
94 int ret;
95
96 /* A64 instructions must be word aligned */
97 if ((uintptr_t)tp & 0x3)
98 return -EINVAL;
99
100 ret = aarch64_insn_write(tp, insn);
101 if (ret == 0)
102 caches_clean_inval_pou((uintptr_t)tp,
103 (uintptr_t)tp + AARCH64_INSN_SIZE);
104
105 return ret;
106}
107
108struct aarch64_insn_patch {
109 void **text_addrs;
110 u32 *new_insns;
111 int insn_cnt;
112 atomic_t cpu_count;
113};
114
115static int __kprobes aarch64_insn_patch_text_cb(void *arg)
116{
117 int i, ret = 0;
118 struct aarch64_insn_patch *pp = arg;
119
120 /* The first CPU becomes master */
121 if (atomic_inc_return(&pp->cpu_count) == 1) {
122 for (i = 0; ret == 0 && i < pp->insn_cnt; i++)
123 ret = aarch64_insn_patch_text_nosync(pp->text_addrs[i],
124 pp->new_insns[i]);
125 /* Notify other processors with an additional increment. */
126 atomic_inc(&pp->cpu_count);
127 } else {
128 while (atomic_read(&pp->cpu_count) <= num_online_cpus())
129 cpu_relax();
130 isb();
131 }
132
133 return ret;
134}
135
136int __kprobes aarch64_insn_patch_text(void *addrs[], u32 insns[], int cnt)
137{
138 struct aarch64_insn_patch patch = {
139 .text_addrs = addrs,
140 .new_insns = insns,
141 .insn_cnt = cnt,
142 .cpu_count = ATOMIC_INIT(0),
143 };
144
145 if (cnt <= 0)
146 return -EINVAL;
147
148 return stop_machine_cpuslocked(aarch64_insn_patch_text_cb, &patch,
149 cpu_online_mask);
150}
arch/arm64/kernel/perf_callchain.c+1−1
@@ -116,7 +116,7 @@ void perf_callchain_user(struct perf_callchain_entry_ctx *entry, · 115 unmodified lines
116 tail = (struct frame_tail __user *)regs->regs[29];116 tail = (struct frame_tail __user *)regs->regs[29];
117 117
118 while (entry->nr < entry->max_stack &&118 while (entry->nr < entry->max_stack &&
119 tail && !((unsigned long)tail & 0xf))119 tail && !((unsigned long)tail & 0x7))
120 tail = user_backtrace(tail, entry);120 tail = user_backtrace(tail, entry);
121 } else {121 } else {
122#ifdef CONFIG_COMPAT122#ifdef CONFIG_COMPAT
arch/arm64/kernel/perf_event.c+35−5
@@ -165,10 +165,7 @@ armv8pmu_events_sysfs_show(struct device *dev, · 164 unmodified lines
165}165}
166 166
167#define ARMV8_EVENT_ATTR(name, config) \167#define ARMV8_EVENT_ATTR(name, config) \
168 (&((struct perf_pmu_events_attr) { \168 PMU_EVENT_ATTR_ID(name, armv8pmu_events_sysfs_show, config)
169 .attr = __ATTR(name, 0444, armv8pmu_events_sysfs_show, NULL), \
170 .id = config, \
171 }).attr.attr)
172 169
173static struct attribute *armv8_pmuv3_event_attrs[] = {170static struct attribute *armv8_pmuv3_event_attrs[] = {
174 ARMV8_EVENT_ATTR(sw_incr, ARMV8_PMUV3_PERFCTR_SW_INCR),171 ARMV8_EVENT_ATTR(sw_incr, ARMV8_PMUV3_PERFCTR_SW_INCR),
@@ -312,13 +309,46 @@ static ssize_t slots_show(struct device *dev, struct device_attribute *attr, · 137 unmodified lines
312 struct arm_pmu *cpu_pmu = container_of(pmu, struct arm_pmu, pmu);309 struct arm_pmu *cpu_pmu = container_of(pmu, struct arm_pmu, pmu);
313 u32 slots = cpu_pmu->reg_pmmir & ARMV8_PMU_SLOTS_MASK;310 u32 slots = cpu_pmu->reg_pmmir & ARMV8_PMU_SLOTS_MASK;
314 311
315 return snprintf(page, PAGE_SIZE, "0x%08x\n", slots);312 return sysfs_emit(page, "0x%08x\n", slots);
316}313}
317 314
318static DEVICE_ATTR_RO(slots);315static DEVICE_ATTR_RO(slots);
319 316
317static ssize_t bus_slots_show(struct device *dev, struct device_attribute *attr,
318 char *page)
319{
320 struct pmu *pmu = dev_get_drvdata(dev);
321 struct arm_pmu *cpu_pmu = container_of(pmu, struct arm_pmu, pmu);
322 u32 bus_slots = (cpu_pmu->reg_pmmir >> ARMV8_PMU_BUS_SLOTS_SHIFT)
323 & ARMV8_PMU_BUS_SLOTS_MASK;
324
325 return sysfs_emit(page, "0x%08x\n", bus_slots);
326}
327
328static DEVICE_ATTR_RO(bus_slots);
329
330static ssize_t bus_width_show(struct device *dev, struct device_attribute *attr,
331 char *page)
332{
333 struct pmu *pmu = dev_get_drvdata(dev);
334 struct arm_pmu *cpu_pmu = container_of(pmu, struct arm_pmu, pmu);
335 u32 bus_width = (cpu_pmu->reg_pmmir >> ARMV8_PMU_BUS_WIDTH_SHIFT)
336 & ARMV8_PMU_BUS_WIDTH_MASK;
337 u32 val = 0;
338
339 /* Encoded as Log2(number of bytes), plus one */
340 if (bus_width > 2 && bus_width < 13)
341 val = 1 << (bus_width - 1);
342
343 return sysfs_emit(page, "0x%08x\n", val);
344}
345
346static DEVICE_ATTR_RO(bus_width);
347
320static struct attribute *armv8_pmuv3_caps_attrs[] = {348static struct attribute *armv8_pmuv3_caps_attrs[] = {
321 &dev_attr_slots.attr,349 &dev_attr_slots.attr,
350 &dev_attr_bus_slots.attr,
351 &dev_attr_bus_width.attr,
322 NULL,352 NULL,
323};353};
324 354
arch/arm64/kernel/probes/kprobes.c+10−8
@@ -7,26 +7,28 @@ · 6 unmodified lines
7 * Copyright (C) 2013 Linaro Limited.7 * Copyright (C) 2013 Linaro Limited.
8 * Author: Sandeepa Prabhu <sandeepa.prabhu@linaro.org>8 * Author: Sandeepa Prabhu <sandeepa.prabhu@linaro.org>
9 */9 */
10#include <linux/extable.h>
10#include <linux/kasan.h>11#include <linux/kasan.h>
11#include <linux/kernel.h>12#include <linux/kernel.h>
12#include <linux/kprobes.h>13#include <linux/kprobes.h>
13#include <linux/extable.h>
14#include <linux/slab.h>
15#include <linux/stop_machine.h>
16#include <linux/sched/debug.h>14#include <linux/sched/debug.h>
17#include <linux/set_memory.h>15#include <linux/set_memory.h>
16#include <linux/slab.h>
17#include <linux/stop_machine.h>
18#include <linux/stringify.h>18#include <linux/stringify.h>
19#include <linux/uaccess.h>
19#include <linux/vmalloc.h>20#include <linux/vmalloc.h>
20#include <asm/traps.h>21
21#include <asm/ptrace.h>
22#include <asm/cacheflush.h>22#include <asm/cacheflush.h>
23#include <asm/debug-monitors.h>
24#include <asm/daifflags.h>23#include <asm/daifflags.h>
25#include <asm/system_misc.h>24#include <asm/debug-monitors.h>
26#include <asm/insn.h>25#include <asm/insn.h>
27#include <linux/uaccess.h>
28#include <asm/irq.h>26#include <asm/irq.h>
27#include <asm/patching.h>
28#include <asm/ptrace.h>
29#include <asm/sections.h>29#include <asm/sections.h>
30#include <asm/system_misc.h>
31#include <asm/traps.h>
30 32
31#include "decode-insn.h"33#include "decode-insn.h"
32 34
arch/arm64/kernel/probes/simulate-insn.c+1−0
@@ -10,6 +10,7 @@ · 9 unmodified lines
10#include <linux/kprobes.h>10#include <linux/kprobes.h>
11 11
12#include <asm/ptrace.h>12#include <asm/ptrace.h>
13#include <asm/traps.h>
13 14
14#include "simulate-insn.h"15#include "simulate-insn.h"
15 16
arch/arm64/kernel/probes/uprobes.c+1−1
@@ -21,7 +21,7 @@ void arch_uprobe_copy_ixol(struct page *page, unsigned long vaddr, · 20 unmodified lines
21 memcpy(dst, src, len);21 memcpy(dst, src, len);
22 22
23 /* flush caches (dcache/icache) */23 /* flush caches (dcache/icache) */
24 sync_icache_aliases(dst, len);24 sync_icache_aliases((unsigned long)dst, (unsigned long)dst + len);
25 25
26 kunmap_atomic(xol_page_kaddr);26 kunmap_atomic(xol_page_kaddr);
27}27}
arch/arm64/kernel/process.c+23−76
@@ -18,7 +18,6 @@ · 17 unmodified lines
18#include <linux/sched/task.h>18#include <linux/sched/task.h>
19#include <linux/sched/task_stack.h>19#include <linux/sched/task_stack.h>
20#include <linux/kernel.h>20#include <linux/kernel.h>
21#include <linux/lockdep.h>
22#include <linux/mman.h>21#include <linux/mman.h>
23#include <linux/mm.h>22#include <linux/mm.h>
24#include <linux/nospec.h>23#include <linux/nospec.h>
@@ -46,7 +45,6 @@ · 21 unmodified lines
46#include <linux/prctl.h>45#include <linux/prctl.h>
47 46
48#include <asm/alternative.h>47#include <asm/alternative.h>
49#include <asm/arch_gicv3.h>
50#include <asm/compat.h>48#include <asm/compat.h>
51#include <asm/cpufeature.h>49#include <asm/cpufeature.h>
52#include <asm/cacheflush.h>50#include <asm/cacheflush.h>
@@ -74,63 +72,6 @@ EXPORT_SYMBOL_GPL(pm_power_off); · 21 unmodified lines
74 72
75void (*arm_pm_restart)(enum reboot_mode reboot_mode, const char *cmd);73void (*arm_pm_restart)(enum reboot_mode reboot_mode, const char *cmd);
76 74
77static void noinstr __cpu_do_idle(void)
78{
79 dsb(sy);
80 wfi();
81}
82
83static void noinstr __cpu_do_idle_irqprio(void)
84{
85 unsigned long pmr;
86 unsigned long daif_bits;
87
88 daif_bits = read_sysreg(daif);
89 write_sysreg(daif_bits | PSR_I_BIT | PSR_F_BIT, daif);
90
91 /*
92 * Unmask PMR before going idle to make sure interrupts can
93 * be raised.
94 */
95 pmr = gic_read_pmr();
96 gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
97
98 __cpu_do_idle();
99
100 gic_write_pmr(pmr);
101 write_sysreg(daif_bits, daif);
102}
103
104/*
105 * cpu_do_idle()
106 *
107 * Idle the processor (wait for interrupt).
108 *
109 * If the CPU supports priority masking we must do additional work to
110 * ensure that interrupts are not masked at the PMR (because the core will
111 * not wake up if we block the wake up signal in the interrupt controller).
112 */
113void noinstr cpu_do_idle(void)
114{
115 if (system_uses_irq_prio_masking())
116 __cpu_do_idle_irqprio();
117 else
118 __cpu_do_idle();
119}
120
121/*
122 * This is our default idle handler.
123 */
124void noinstr arch_cpu_idle(void)
125{
126 /*
127 * This should do all the clock switching and wait for interrupt
128 * tricks
129 */
130 cpu_do_idle();
131 raw_local_irq_enable();
132}
133
134#ifdef CONFIG_HOTPLUG_CPU75#ifdef CONFIG_HOTPLUG_CPU
135void arch_cpu_idle_dead(void)76void arch_cpu_idle_dead(void)
136{77{
@@ -435,6 +376,11 @@ int copy_thread(unsigned long clone_flags, unsigned long stack_start, · 298 unmodified lines
435 }376 }
436 p->thread.cpu_context.pc = (unsigned long)ret_from_fork;377 p->thread.cpu_context.pc = (unsigned long)ret_from_fork;
437 p->thread.cpu_context.sp = (unsigned long)childregs;378 p->thread.cpu_context.sp = (unsigned long)childregs;
379 /*
380 * For the benefit of the unwinder, set up childregs->stackframe
381 * as the final frame for the new task.
382 */
383 p->thread.cpu_context.fp = (unsigned long)childregs->stackframe;
438 384
439 ptrace_hw_copy_thread(p);385 ptrace_hw_copy_thread(p);
440 386
@@ -527,6 +473,15 @@ static void erratum_1418040_thread_switch(struct task_struct *prev, · 86 unmodified lines
527 write_sysreg(val, cntkctl_el1);473 write_sysreg(val, cntkctl_el1);
528}474}
529 475
476static void compat_thread_switch(struct task_struct *next)
477{
478 if (!is_compat_thread(task_thread_info(next)))
479 return;
480
481 if (static_branch_unlikely(&arm64_mismatched_32bit_el0))
482 set_tsk_thread_flag(next, TIF_NOTIFY_RESUME);
483}
484
530static void update_sctlr_el1(u64 sctlr)485static void update_sctlr_el1(u64 sctlr)
531{486{
532 /*487 /*
@@ -568,6 +523,7 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev, · 35 unmodified lines
568 ssbs_thread_switch(next);523 ssbs_thread_switch(next);
569 erratum_1418040_thread_switch(prev, next);524 erratum_1418040_thread_switch(prev, next);
570 ptrauth_thread_switch_user(next);525 ptrauth_thread_switch_user(next);
526 compat_thread_switch(next);
571 527
572 /*528 /*
573 * Complete any pending TLB or cache maintenance on this CPU in case529 * Complete any pending TLB or cache maintenance on this CPU in case
@@ -633,8 +589,15 @@ unsigned long arch_align_stack(unsigned long sp) · 59 unmodified lines
633 */589 */
634void arch_setup_new_exec(void)590void arch_setup_new_exec(void)
635{591{
636 current->mm->context.flags = is_compat_task() ? MMCF_AARCH32 : 0;592 unsigned long mmflags = 0;
593
594 if (is_compat_task()) {
595 mmflags = MMCF_AARCH32;
596 if (static_branch_unlikely(&arm64_mismatched_32bit_el0))
597 set_tsk_thread_flag(current, TIF_NOTIFY_RESUME);
598 }
637 599
600 current->mm->context.flags = mmflags;
638 ptrauth_thread_init_user();601 ptrauth_thread_init_user();
639 mte_thread_init_user();602 mte_thread_init_user();
640 603
@@ -724,22 +687,6 @@ static int __init tagged_addr_init(void) · 83 unmodified lines
724core_initcall(tagged_addr_init);687core_initcall(tagged_addr_init);
725#endif /* CONFIG_ARM64_TAGGED_ADDR_ABI */688#endif /* CONFIG_ARM64_TAGGED_ADDR_ABI */
726 689
727asmlinkage void __sched arm64_preempt_schedule_irq(void)
728{
729 lockdep_assert_irqs_disabled();
730
731 /*
732 * Preempting a task from an IRQ means we leave copies of PSTATE
733 * on the stack. cpufeature's enable calls may modify PSTATE, but
734 * resuming one of these preempted tasks would undo those changes.
735 *
736 * Only allow a task to be preempted once cpufeatures have been
737 * enabled.
738 */
739 if (system_capabilities_finalized())
740 preempt_schedule_irq();
741}
742
743#ifdef CONFIG_BINFMT_ELF690#ifdef CONFIG_BINFMT_ELF
744int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state,691int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state,
745 bool has_interp, bool is_interp)692 bool has_interp, bool is_interp)
arch/arm64/kernel/ptrace.c+1−1
@@ -122,7 +122,7 @@ static bool regs_within_kernel_stack(struct pt_regs *regs, unsigned long addr) · 121 unmodified lines
122{122{
123 return ((addr & ~(THREAD_SIZE - 1)) ==123 return ((addr & ~(THREAD_SIZE - 1)) ==
124 (kernel_stack_pointer(regs) & ~(THREAD_SIZE - 1))) ||124 (kernel_stack_pointer(regs) & ~(THREAD_SIZE - 1))) ||
125 on_irq_stack(addr, NULL);125 on_irq_stack(addr, sizeof(unsigned long), NULL);
126}126}
127 127
128/**128/**
arch/arm64/kernel/sdei.c+12−52
@@ -162,31 +162,33 @@ static int init_sdei_scs(void) · 161 unmodified lines
162 return err;162 return err;
163}163}
164 164
165static bool on_sdei_normal_stack(unsigned long sp, struct stack_info *info)165static bool on_sdei_normal_stack(unsigned long sp, unsigned long size,
166 struct stack_info *info)
166{167{
167 unsigned long low = (unsigned long)raw_cpu_read(sdei_stack_normal_ptr);168 unsigned long low = (unsigned long)raw_cpu_read(sdei_stack_normal_ptr);
168 unsigned long high = low + SDEI_STACK_SIZE;169 unsigned long high = low + SDEI_STACK_SIZE;
169 170
170 return on_stack(sp, low, high, STACK_TYPE_SDEI_NORMAL, info);171 return on_stack(sp, size, low, high, STACK_TYPE_SDEI_NORMAL, info);
171}172}
172 173
173static bool on_sdei_critical_stack(unsigned long sp, struct stack_info *info)174static bool on_sdei_critical_stack(unsigned long sp, unsigned long size,
175 struct stack_info *info)
174{176{
175 unsigned long low = (unsigned long)raw_cpu_read(sdei_stack_critical_ptr);177 unsigned long low = (unsigned long)raw_cpu_read(sdei_stack_critical_ptr);
176 unsigned long high = low + SDEI_STACK_SIZE;178 unsigned long high = low + SDEI_STACK_SIZE;
177 179
178 return on_stack(sp, low, high, STACK_TYPE_SDEI_CRITICAL, info);180 return on_stack(sp, size, low, high, STACK_TYPE_SDEI_CRITICAL, info);
179}181}
180 182
181bool _on_sdei_stack(unsigned long sp, struct stack_info *info)183bool _on_sdei_stack(unsigned long sp, unsigned long size, struct stack_info *info)
182{184{
183 if (!IS_ENABLED(CONFIG_VMAP_STACK))185 if (!IS_ENABLED(CONFIG_VMAP_STACK))
184 return false;186 return false;
185 187
186 if (on_sdei_critical_stack(sp, info))188 if (on_sdei_critical_stack(sp, size, info))
187 return true;189 return true;
188 190
189 if (on_sdei_normal_stack(sp, info))191 if (on_sdei_normal_stack(sp, size, info))
190 return true;192 return true;
191 193
192 return false;194 return false;
@@ -231,13 +233,13 @@ out_err: · 38 unmodified lines
231}233}
232 234
233/*235/*
234 * __sdei_handler() returns one of:236 * do_sdei_event() returns one of:
235 * SDEI_EV_HANDLED - success, return to the interrupted context.237 * SDEI_EV_HANDLED - success, return to the interrupted context.
236 * SDEI_EV_FAILED - failure, return this error code to firmare.238 * SDEI_EV_FAILED - failure, return this error code to firmare.
237 * virtual-address - success, return to this address.239 * virtual-address - success, return to this address.
238 */240 */
239static __kprobes unsigned long _sdei_handler(struct pt_regs *regs,241unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
240 struct sdei_registered_event *arg)242 struct sdei_registered_event *arg)
241{243{
242 u32 mode;244 u32 mode;
243 int i, err = 0;245 int i, err = 0;
@@ -292,45 +294,3 @@ static __kprobes unsigned long _sdei_handler(struct pt_regs *regs, · 48 unmodified lines
292 294
293 return vbar + 0x480;295 return vbar + 0x480;
294}296}
295
296static void __kprobes notrace __sdei_pstate_entry(void)
297{
298 /*
299 * The original SDEI spec (ARM DEN 0054A) can be read ambiguously as to
300 * whether PSTATE bits are inherited unchanged or generated from
301 * scratch, and the TF-A implementation always clears PAN and always
302 * clears UAO. There are no other known implementations.
303 *
304 * Subsequent revisions (ARM DEN 0054B) follow the usual rules for how
305 * PSTATE is modified upon architectural exceptions, and so PAN is
306 * either inherited or set per SCTLR_ELx.SPAN, and UAO is always
307 * cleared.
308 *
309 * We must explicitly reset PAN to the expected state, including
310 * clearing it when the host isn't using it, in case a VM had it set.
311 */
312 if (system_uses_hw_pan())
313 set_pstate_pan(1);
314 else if (cpu_has_pan())
315 set_pstate_pan(0);
316}
317
318asmlinkage noinstr unsigned long
319__sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
320{
321 unsigned long ret;
322
323 /*
324 * We didn't take an exception to get here, so the HW hasn't
325 * set/cleared bits in PSTATE that we may rely on. Initialize PAN.
326 */
327 __sdei_pstate_entry();
328
329 arm64_enter_nmi(regs);
330
331 ret = _sdei_handler(regs, arg);
332
333 arm64_exit_nmi(regs);
334
335 return ret;
336}
arch/arm64/kernel/setup.c+1−7
@@ -87,12 +87,6 @@ void __init smp_setup_processor_id(void) · 86 unmodified lines
87 u64 mpidr = read_cpuid_mpidr() & MPIDR_HWID_BITMASK;87 u64 mpidr = read_cpuid_mpidr() & MPIDR_HWID_BITMASK;
88 set_cpu_logical_map(0, mpidr);88 set_cpu_logical_map(0, mpidr);
89 89
90 /*
91 * clear __my_cpu_offset on boot CPU to avoid hang caused by
92 * using percpu variable early, for example, lockdep will
93 * access percpu variable inside lock_release
94 */
95 set_my_cpu_offset(0);
96 pr_info("Booting Linux on physical CPU 0x%010lx [0x%08x]\n",90 pr_info("Booting Linux on physical CPU 0x%010lx [0x%08x]\n",
97 (unsigned long)mpidr, read_cpuid_id());91 (unsigned long)mpidr, read_cpuid_id());
98}92}
@@ -381,7 +375,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) · 282 unmodified lines
381 * faults in case uaccess_enable() is inadvertently called by the init375 * faults in case uaccess_enable() is inadvertently called by the init
382 * thread.376 * thread.
383 */377 */
384 init_task.thread_info.ttbr0 = __pa_symbol(reserved_pg_dir);378 init_task.thread_info.ttbr0 = phys_to_ttbr(__pa_symbol(reserved_pg_dir));
385#endif379#endif
386 380
387 if (boot_args[1] || boot_args[2] || boot_args[3]) {381 if (boot_args[1] || boot_args[2] || boot_args[3]) {
arch/arm64/kernel/signal.c+26−0
@@ -911,6 +911,19 @@ static void do_signal(struct pt_regs *regs) · 910 unmodified lines
911 restore_saved_sigmask();911 restore_saved_sigmask();
912}912}
913 913
914static bool cpu_affinity_invalid(struct pt_regs *regs)
915{
916 if (!compat_user_mode(regs))
917 return false;
918
919 /*
920 * We're preemptible, but a reschedule will cause us to check the
921 * affinity again.
922 */
923 return !cpumask_test_cpu(raw_smp_processor_id(),
924 system_32bit_el0_cpumask());
925}
926
914asmlinkage void do_notify_resume(struct pt_regs *regs,927asmlinkage void do_notify_resume(struct pt_regs *regs,
915 unsigned long thread_flags)928 unsigned long thread_flags)
916{929{
@@ -938,6 +951,19 @@ asmlinkage void do_notify_resume(struct pt_regs *regs, · 21 unmodified lines
938 if (thread_flags & _TIF_NOTIFY_RESUME) {951 if (thread_flags & _TIF_NOTIFY_RESUME) {
939 tracehook_notify_resume(regs);952 tracehook_notify_resume(regs);
940 rseq_handle_notify_resume(NULL, regs);953 rseq_handle_notify_resume(NULL, regs);
954
955 /*
956 * If we reschedule after checking the affinity
957 * then we must ensure that TIF_NOTIFY_RESUME
958 * is set so that we check the affinity again.
959 * Since tracehook_notify_resume() clears the
960 * flag, ensure that the compiler doesn't move
961 * it after the affinity check.
962 */
963 barrier();
964
965 if (cpu_affinity_invalid(regs))
966 force_sig(SIGKILL);
941 }967 }
942 968
943 if (thread_flags & _TIF_FOREIGN_FPSTATE)969 if (thread_flags & _TIF_FOREIGN_FPSTATE)
arch/arm64/kernel/smccc-call.S+83−0
@@ -7,8 +7,34 @@ · 6 unmodified lines
7 7
8#include <asm/asm-offsets.h>8#include <asm/asm-offsets.h>
9#include <asm/assembler.h>9#include <asm/assembler.h>
10#include <asm/thread_info.h>
11
12/*
13 * If we have SMCCC v1.3 and (as is likely) no SVE state in
14 * the registers then set the SMCCC hint bit to say there's no
15 * need to preserve it. Do this by directly adjusting the SMCCC
16 * function value which is already stored in x0 ready to be called.
17 */
18SYM_FUNC_START(__arm_smccc_sve_check)
19
20 ldr_l x16, smccc_has_sve_hint
21 cbz x16, 2f
22
23 get_current_task x16
24 ldr x16, [x16, #TSK_TI_FLAGS]
25 tbnz x16, #TIF_FOREIGN_FPSTATE, 1f // Any live FP state?
26 tbnz x16, #TIF_SVE, 2f // Does that state include SVE?
27
281: orr x0, x0, ARM_SMCCC_1_3_SVE_HINT
29
302: ret
31SYM_FUNC_END(__arm_smccc_sve_check)
32EXPORT_SYMBOL(__arm_smccc_sve_check)
10 33
11 .macro SMCCC instr34 .macro SMCCC instr
35alternative_if ARM64_SVE
36 bl __arm_smccc_sve_check
37alternative_else_nop_endif
12 \instr #038 \instr #0
13 ldr x4, [sp]39 ldr x4, [sp]
14 stp x0, x1, [x4, #ARM_SMCCC_RES_X0_OFFS]40 stp x0, x1, [x4, #ARM_SMCCC_RES_X0_OFFS]
@@ -43,3 +69,60 @@ SYM_FUNC_START(__arm_smccc_hvc) · 28 unmodified lines
43 SMCCC hvc69 SMCCC hvc
44SYM_FUNC_END(__arm_smccc_hvc)70SYM_FUNC_END(__arm_smccc_hvc)
45EXPORT_SYMBOL(__arm_smccc_hvc)71EXPORT_SYMBOL(__arm_smccc_hvc)
72
73 .macro SMCCC_1_2 instr
74 /* Save `res` and free a GPR that won't be clobbered */
75 stp x1, x19, [sp, #-16]!
76
77 /* Ensure `args` won't be clobbered while loading regs in next step */
78 mov x19, x0
79
80 /* Load the registers x0 - x17 from the struct arm_smccc_1_2_regs */
81 ldp x0, x1, [x19, #ARM_SMCCC_1_2_REGS_X0_OFFS]
82 ldp x2, x3, [x19, #ARM_SMCCC_1_2_REGS_X2_OFFS]
83 ldp x4, x5, [x19, #ARM_SMCCC_1_2_REGS_X4_OFFS]
84 ldp x6, x7, [x19, #ARM_SMCCC_1_2_REGS_X6_OFFS]
85 ldp x8, x9, [x19, #ARM_SMCCC_1_2_REGS_X8_OFFS]
86 ldp x10, x11, [x19, #ARM_SMCCC_1_2_REGS_X10_OFFS]
87 ldp x12, x13, [x19, #ARM_SMCCC_1_2_REGS_X12_OFFS]
88 ldp x14, x15, [x19, #ARM_SMCCC_1_2_REGS_X14_OFFS]
89 ldp x16, x17, [x19, #ARM_SMCCC_1_2_REGS_X16_OFFS]
90
91 \instr #0
92
93 /* Load the `res` from the stack */
94 ldr x19, [sp]
95
96 /* Store the registers x0 - x17 into the result structure */
97 stp x0, x1, [x19, #ARM_SMCCC_1_2_REGS_X0_OFFS]
98 stp x2, x3, [x19, #ARM_SMCCC_1_2_REGS_X2_OFFS]
99 stp x4, x5, [x19, #ARM_SMCCC_1_2_REGS_X4_OFFS]
100 stp x6, x7, [x19, #ARM_SMCCC_1_2_REGS_X6_OFFS]
101 stp x8, x9, [x19, #ARM_SMCCC_1_2_REGS_X8_OFFS]
102 stp x10, x11, [x19, #ARM_SMCCC_1_2_REGS_X10_OFFS]
103 stp x12, x13, [x19, #ARM_SMCCC_1_2_REGS_X12_OFFS]
104 stp x14, x15, [x19, #ARM_SMCCC_1_2_REGS_X14_OFFS]
105 stp x16, x17, [x19, #ARM_SMCCC_1_2_REGS_X16_OFFS]
106
107 /* Restore original x19 */
108 ldp xzr, x19, [sp], #16
109 ret
110.endm
111
112/*
113 * void arm_smccc_1_2_hvc(const struct arm_smccc_1_2_regs *args,
114 * struct arm_smccc_1_2_regs *res);
115 */
116SYM_FUNC_START(arm_smccc_1_2_hvc)
117 SMCCC_1_2 hvc
118SYM_FUNC_END(arm_smccc_1_2_hvc)
119EXPORT_SYMBOL(arm_smccc_1_2_hvc)
120
121/*
122 * void arm_smccc_1_2_smc(const struct arm_smccc_1_2_regs *args,
123 * struct arm_smccc_1_2_regs *res);
124 */
125SYM_FUNC_START(arm_smccc_1_2_smc)
126 SMCCC_1_2 smc
127SYM_FUNC_END(arm_smccc_1_2_smc)
128EXPORT_SYMBOL(arm_smccc_1_2_smc)
arch/arm64/kernel/smp.c+7−9
@@ -120,9 +120,7 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) · 119 unmodified lines
120 * page tables.120 * page tables.
121 */121 */
122 secondary_data.task = idle;122 secondary_data.task = idle;
123 secondary_data.stack = task_stack_page(idle) + THREAD_SIZE;
124 update_cpu_boot_status(CPU_MMU_OFF);123 update_cpu_boot_status(CPU_MMU_OFF);
125 __flush_dcache_area(&secondary_data, sizeof(secondary_data));
126 124
127 /* Now bring the CPU into our world */125 /* Now bring the CPU into our world */
128 ret = boot_secondary(cpu, idle);126 ret = boot_secondary(cpu, idle);
@@ -142,8 +140,6 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) · 13 unmodified lines
142 140
143 pr_crit("CPU%u: failed to come online\n", cpu);141 pr_crit("CPU%u: failed to come online\n", cpu);
144 secondary_data.task = NULL;142 secondary_data.task = NULL;
145 secondary_data.stack = NULL;
146 __flush_dcache_area(&secondary_data, sizeof(secondary_data));
147 status = READ_ONCE(secondary_data.status);143 status = READ_ONCE(secondary_data.status);
148 if (status == CPU_MMU_OFF)144 if (status == CPU_MMU_OFF)
149 status = READ_ONCE(__early_cpu_boot_status);145 status = READ_ONCE(__early_cpu_boot_status);
@@ -202,10 +198,7 @@ asmlinkage notrace void secondary_start_kernel(void) · 52 unmodified lines
202 u64 mpidr = read_cpuid_mpidr() & MPIDR_HWID_BITMASK;198 u64 mpidr = read_cpuid_mpidr() & MPIDR_HWID_BITMASK;
203 struct mm_struct *mm = &init_mm;199 struct mm_struct *mm = &init_mm;
204 const struct cpu_operations *ops;200 const struct cpu_operations *ops;
205 unsigned int cpu;201 unsigned int cpu = smp_processor_id();
206
207 cpu = task_cpu(current);
208 set_my_cpu_offset(per_cpu_offset(cpu));
209 202
210 /*203 /*
211 * All kernel threads share the same mm context; grab a204 * All kernel threads share the same mm context; grab a
@@ -351,7 +344,7 @@ void __cpu_die(unsigned int cpu) · 139 unmodified lines
351 pr_crit("CPU%u: cpu didn't die\n", cpu);344 pr_crit("CPU%u: cpu didn't die\n", cpu);
352 return;345 return;
353 }346 }
354 pr_notice("CPU%u: shutdown\n", cpu);347 pr_debug("CPU%u: shutdown\n", cpu);
355 348
356 /*349 /*
357 * Now that the dying CPU is beyond the point of no return w.r.t.350 * Now that the dying CPU is beyond the point of no return w.r.t.
@@ -451,6 +444,11 @@ void __init smp_cpus_done(unsigned int max_cpus) · 93 unmodified lines
451 444
452void __init smp_prepare_boot_cpu(void)445void __init smp_prepare_boot_cpu(void)
453{446{
447 /*
448 * The runtime per-cpu areas have been allocated by
449 * setup_per_cpu_areas(), and CPU0's boot time per-cpu area will be
450 * freed shortly, so we must move over to the runtime per-cpu area.
451 */
454 set_my_cpu_offset(per_cpu_offset(smp_processor_id()));452 set_my_cpu_offset(per_cpu_offset(smp_processor_id()));
455 cpuinfo_store_boot_cpu();453 cpuinfo_store_boot_cpu();
456 454
arch/arm64/kernel/smp_spin_table.c+4−3
@@ -36,7 +36,7 @@ static void write_pen_release(u64 val) · 35 unmodified lines
36 unsigned long size = sizeof(secondary_holding_pen_release);36 unsigned long size = sizeof(secondary_holding_pen_release);
37 37
38 secondary_holding_pen_release = val;38 secondary_holding_pen_release = val;
39 __flush_dcache_area(start, size);39 dcache_clean_inval_poc((unsigned long)start, (unsigned long)start + size);
40}40}
41 41
42 42
@@ -90,8 +90,9 @@ static int smp_spin_table_cpu_prepare(unsigned int cpu) · 47 unmodified lines
90 * the boot protocol.90 * the boot protocol.
91 */91 */
92 writeq_relaxed(pa_holding_pen, release_addr);92 writeq_relaxed(pa_holding_pen, release_addr);
93 __flush_dcache_area((__force void *)release_addr,93 dcache_clean_inval_poc((__force unsigned long)release_addr,
94 sizeof(*release_addr));94 (__force unsigned long)release_addr +
95 sizeof(*release_addr));
95 96
96 /*97 /*
97 * Send an event to wake up the secondary CPU.98 * Send an event to wake up the secondary CPU.
arch/arm64/kernel/stacktrace.c+8−10
@@ -68,13 +68,17 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) · 67 unmodified lines
68 unsigned long fp = frame->fp;68 unsigned long fp = frame->fp;
69 struct stack_info info;69 struct stack_info info;
70 70
71 if (fp & 0xf)
72 return -EINVAL;
73
74 if (!tsk)71 if (!tsk)
75 tsk = current;72 tsk = current;
76 73
77 if (!on_accessible_stack(tsk, fp, &info))74 /* Final frame; nothing to unwind */
75 if (fp == (unsigned long)task_pt_regs(tsk)->stackframe)
76 return -ENOENT;
77
78 if (fp & 0x7)
79 return -EINVAL;
80
81 if (!on_accessible_stack(tsk, fp, 16, &info))
78 return -EINVAL;82 return -EINVAL;
79 83
80 if (test_bit(info.type, frame->stacks_done))84 if (test_bit(info.type, frame->stacks_done))
@@ -128,12 +132,6 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) · 47 unmodified lines
128 132
129 frame->pc = ptrauth_strip_insn_pac(frame->pc);133 frame->pc = ptrauth_strip_insn_pac(frame->pc);
130 134
131 /*
132 * This is a terminal record, so we have finished unwinding.
133 */
134 if (!frame->fp && !frame->pc)
135 return -ENOENT;
136
137 return 0;135 return 0;
138}136}
139NOKPROBE_SYMBOL(unwind_frame);137NOKPROBE_SYMBOL(unwind_frame);
arch/arm64/kernel/suspend.c+11−1
@@ -7,6 +7,7 @@ · 6 unmodified lines
7#include <asm/alternative.h>7#include <asm/alternative.h>
8#include <asm/cacheflush.h>8#include <asm/cacheflush.h>
9#include <asm/cpufeature.h>9#include <asm/cpufeature.h>
10#include <asm/cpuidle.h>
10#include <asm/daifflags.h>11#include <asm/daifflags.h>
11#include <asm/debug-monitors.h>12#include <asm/debug-monitors.h>
12#include <asm/exec.h>13#include <asm/exec.h>
@@ -91,6 +92,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long)) · 78 unmodified lines
91 int ret = 0;92 int ret = 0;
92 unsigned long flags;93 unsigned long flags;
93 struct sleep_stack_data state;94 struct sleep_stack_data state;
95 struct arm_cpuidle_irq_context context;
94 96
95 /* Report any MTE async fault before going to suspend */97 /* Report any MTE async fault before going to suspend */
96 mte_suspend_enter();98 mte_suspend_enter();
@@ -103,12 +105,18 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long)) · 6 unmodified lines
103 flags = local_daif_save();105 flags = local_daif_save();
104 106
105 /*107 /*
106 * Function graph tracer state gets incosistent when the kernel108 * Function graph tracer state gets inconsistent when the kernel
107 * calls functions that never return (aka suspend finishers) hence109 * calls functions that never return (aka suspend finishers) hence
108 * disable graph tracing during their execution.110 * disable graph tracing during their execution.
109 */111 */
110 pause_graph_tracing();112 pause_graph_tracing();
111 113
114 /*
115 * Switch to using DAIF.IF instead of PMR in order to reliably
116 * resume if we're using pseudo-NMIs.
117 */
118 arm_cpuidle_save_irq_context(&context);
119
112 if (__cpu_suspend_enter(&state)) {120 if (__cpu_suspend_enter(&state)) {
113 /* Call the suspend finisher */121 /* Call the suspend finisher */
114 ret = fn(arg);122 ret = fn(arg);
@@ -126,6 +134,8 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long)) · 11 unmodified lines
126 RCU_NONIDLE(__cpu_suspend_exit());134 RCU_NONIDLE(__cpu_suspend_exit());
127 }135 }
128 136
137 arm_cpuidle_restore_irq_context(&context);
138
129 unpause_graph_tracing();139 unpause_graph_tracing();
130 140
131 /*141 /*
arch/arm64/kernel/sys_compat.c+1−1
@@ -41,7 +41,7 @@ __do_compat_cache_op(unsigned long start, unsigned long end) · 40 unmodified lines
41 dsb(ish);41 dsb(ish);
42 }42 }
43 43
44 ret = __flush_cache_user_range(start, start + chunk);44 ret = caches_clean_inval_user_pou(start, start + chunk);
45 if (ret)45 if (ret)
46 return ret;46 return ret;
47 47
arch/arm64/kernel/traps.c+100−35
@@ -38,6 +38,7 @@ · 37 unmodified lines
38#include <asm/extable.h>38#include <asm/extable.h>
39#include <asm/insn.h>39#include <asm/insn.h>
40#include <asm/kprobes.h>40#include <asm/kprobes.h>
41#include <asm/patching.h>
41#include <asm/traps.h>42#include <asm/traps.h>
42#include <asm/smp.h>43#include <asm/smp.h>
43#include <asm/stack_pointer.h>44#include <asm/stack_pointer.h>
@@ -45,11 +46,102 @@ · 1 unmodified line
45#include <asm/system_misc.h>46#include <asm/system_misc.h>
46#include <asm/sysreg.h>47#include <asm/sysreg.h>
47 48
48static const char *handler[] = {49static bool __kprobes __check_eq(unsigned long pstate)
49 "Synchronous Abort",50{
50 "IRQ",51 return (pstate & PSR_Z_BIT) != 0;
51 "FIQ",52}
52 "Error"53
54static bool __kprobes __check_ne(unsigned long pstate)
55{
56 return (pstate & PSR_Z_BIT) == 0;
57}
58
59static bool __kprobes __check_cs(unsigned long pstate)
60{
61 return (pstate & PSR_C_BIT) != 0;
62}
63
64static bool __kprobes __check_cc(unsigned long pstate)
65{
66 return (pstate & PSR_C_BIT) == 0;
67}
68
69static bool __kprobes __check_mi(unsigned long pstate)
70{
71 return (pstate & PSR_N_BIT) != 0;
72}
73
74static bool __kprobes __check_pl(unsigned long pstate)
75{
76 return (pstate & PSR_N_BIT) == 0;
77}
78
79static bool __kprobes __check_vs(unsigned long pstate)
80{
81 return (pstate & PSR_V_BIT) != 0;
82}
83
84static bool __kprobes __check_vc(unsigned long pstate)
85{
86 return (pstate & PSR_V_BIT) == 0;
87}
88
89static bool __kprobes __check_hi(unsigned long pstate)
90{
91 pstate &= ~(pstate >> 1); /* PSR_C_BIT &= ~PSR_Z_BIT */
92 return (pstate & PSR_C_BIT) != 0;
93}
94
95static bool __kprobes __check_ls(unsigned long pstate)
96{
97 pstate &= ~(pstate >> 1); /* PSR_C_BIT &= ~PSR_Z_BIT */
98 return (pstate & PSR_C_BIT) == 0;
99}
100
101static bool __kprobes __check_ge(unsigned long pstate)
102{
103 pstate ^= (pstate << 3); /* PSR_N_BIT ^= PSR_V_BIT */
104 return (pstate & PSR_N_BIT) == 0;
105}
106
107static bool __kprobes __check_lt(unsigned long pstate)
108{
109 pstate ^= (pstate << 3); /* PSR_N_BIT ^= PSR_V_BIT */
110 return (pstate & PSR_N_BIT) != 0;
111}
112
113static bool __kprobes __check_gt(unsigned long pstate)
114{
115 /*PSR_N_BIT ^= PSR_V_BIT */
116 unsigned long temp = pstate ^ (pstate << 3);
117
118 temp |= (pstate << 1); /*PSR_N_BIT |= PSR_Z_BIT */
119 return (temp & PSR_N_BIT) == 0;
120}
121
122static bool __kprobes __check_le(unsigned long pstate)
123{
124 /*PSR_N_BIT ^= PSR_V_BIT */
125 unsigned long temp = pstate ^ (pstate << 3);
126
127 temp |= (pstate << 1); /*PSR_N_BIT |= PSR_Z_BIT */
128 return (temp & PSR_N_BIT) != 0;
129}
130
131static bool __kprobes __check_al(unsigned long pstate)
132{
133 return true;
134}
135
136/*
137 * Note that the ARMv8 ARM calls condition code 0b1111 "nv", but states that
138 * it behaves identically to 0b1110 ("al").
139 */
140pstate_check_t * const aarch32_opcode_cond_checks[16] = {
141 __check_eq, __check_ne, __check_cs, __check_cc,
142 __check_mi, __check_pl, __check_vs, __check_vc,
143 __check_hi, __check_ls, __check_ge, __check_lt,
144 __check_gt, __check_le, __check_al, __check_al
53};145};
54 146
55int show_unhandled_signals = 0;147int show_unhandled_signals = 0;
@@ -750,28 +842,9 @@ const char *esr_get_class_string(u32 esr) · 694 unmodified lines
750 return esr_class_str[ESR_ELx_EC(esr)];842 return esr_class_str[ESR_ELx_EC(esr)];
751}843}
752 844
753/*
754 * bad_mode handles the impossible case in the exception vector. This is always
755 * fatal.
756 */
757asmlinkage void notrace bad_mode(struct pt_regs *regs, int reason, unsigned int esr)
758{
759 arm64_enter_nmi(regs);
760
761 console_verbose();
762
763 pr_crit("Bad mode in %s handler detected on CPU%d, code 0x%08x -- %s\n",
764 handler[reason], smp_processor_id(), esr,
765 esr_get_class_string(esr));
766
767 __show_regs(regs);
768 local_daif_mask();
769 panic("bad mode");
770}
771
772/*845/*
773 * bad_el0_sync handles unexpected, but potentially recoverable synchronous846 * bad_el0_sync handles unexpected, but potentially recoverable synchronous
774 * exceptions taken from EL0. Unlike bad_mode, this returns.847 * exceptions taken from EL0.
775 */848 */
776void bad_el0_sync(struct pt_regs *regs, int reason, unsigned int esr)849void bad_el0_sync(struct pt_regs *regs, int reason, unsigned int esr)
777{850{
@@ -789,15 +862,11 @@ void bad_el0_sync(struct pt_regs *regs, int reason, unsigned int esr) · 11 unmodified lines
789DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)862DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)
790 __aligned(16);863 __aligned(16);
791 864
792asmlinkage void noinstr handle_bad_stack(struct pt_regs *regs)865void panic_bad_stack(struct pt_regs *regs, unsigned int esr, unsigned long far)
793{866{
794 unsigned long tsk_stk = (unsigned long)current->stack;867 unsigned long tsk_stk = (unsigned long)current->stack;
795 unsigned long irq_stk = (unsigned long)this_cpu_read(irq_stack_ptr);868 unsigned long irq_stk = (unsigned long)this_cpu_read(irq_stack_ptr);
796 unsigned long ovf_stk = (unsigned long)this_cpu_ptr(overflow_stack);869 unsigned long ovf_stk = (unsigned long)this_cpu_ptr(overflow_stack);
797 unsigned int esr = read_sysreg(esr_el1);
798 unsigned long far = read_sysreg(far_el1);
799
800 arm64_enter_nmi(regs);
801 870
802 console_verbose();871 console_verbose();
803 pr_emerg("Insufficient stack space to handle exception!");872 pr_emerg("Insufficient stack space to handle exception!");
@@ -870,15 +939,11 @@ bool arm64_is_fatal_ras_serror(struct pt_regs *regs, unsigned int esr) · 66 unmodified lines
870 }939 }
871}940}
872 941
873asmlinkage void noinstr do_serror(struct pt_regs *regs, unsigned int esr)942void do_serror(struct pt_regs *regs, unsigned int esr)
874{943{
875 arm64_enter_nmi(regs);
876
877 /* non-RAS errors are not containable */944 /* non-RAS errors are not containable */
878 if (!arm64_is_ras_serror(esr) || arm64_is_fatal_ras_serror(regs, esr))945 if (!arm64_is_ras_serror(esr) || arm64_is_fatal_ras_serror(regs, esr))
879 arm64_serror_panic(regs, esr);946 arm64_serror_panic(regs, esr);
880
881 arm64_exit_nmi(regs);
882}947}
883 948
884/* GENERIC_BUG traps */949/* GENERIC_BUG traps */
arch/arm64/kvm/arm.c+11−2
@@ -692,6 +692,15 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu) · 691 unmodified lines
692 }692 }
693}693}
694 694
695static bool vcpu_mode_is_bad_32bit(struct kvm_vcpu *vcpu)
696{
697 if (likely(!vcpu_mode_is_32bit(vcpu)))
698 return false;
699
700 return !system_supports_32bit_el0() ||
701 static_branch_unlikely(&arm64_mismatched_32bit_el0);
702}
703
695/**704/**
696 * kvm_arch_vcpu_ioctl_run - the main VCPU run function to execute guest code705 * kvm_arch_vcpu_ioctl_run - the main VCPU run function to execute guest code
697 * @vcpu: The VCPU pointer706 * @vcpu: The VCPU pointer
@@ -877,7 +886,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) · 179 unmodified lines
877 * with the asymmetric AArch32 case), return to userspace with886 * with the asymmetric AArch32 case), return to userspace with
878 * a fatal error.887 * a fatal error.
879 */888 */
880 if (!system_supports_32bit_el0() && vcpu_mode_is_32bit(vcpu)) {889 if (vcpu_mode_is_bad_32bit(vcpu)) {
881 /*890 /*
882 * As we have caught the guest red-handed, decide that891 * As we have caught the guest red-handed, decide that
883 * it isn't fit for purpose anymore by making the vcpu892 * it isn't fit for purpose anymore by making the vcpu
@@ -1078,7 +1087,7 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, · 194 unmodified lines
1078 if (!cpus_have_final_cap(ARM64_HAS_STAGE2_FWB))1087 if (!cpus_have_final_cap(ARM64_HAS_STAGE2_FWB))
1079 stage2_unmap_vm(vcpu->kvm);1088 stage2_unmap_vm(vcpu->kvm);
1080 else1089 else
1081 __flush_icache_all();1090 icache_inval_all_pou();
1082 }1091 }
1083 1092
1084 vcpu_reset_hcr(vcpu);1093 vcpu_reset_hcr(vcpu);
arch/arm64/kvm/hyp/nvhe/cache.S+2−2
@@ -7,7 +7,7 @@ · 6 unmodified lines
7#include <asm/assembler.h>7#include <asm/assembler.h>
8#include <asm/alternative.h>8#include <asm/alternative.h>
9 9
10SYM_FUNC_START_PI(__flush_dcache_area)10SYM_FUNC_START_PI(dcache_clean_inval_poc)
11 dcache_by_line_op civac, sy, x0, x1, x2, x311 dcache_by_line_op civac, sy, x0, x1, x2, x3
12 ret12 ret
13SYM_FUNC_END_PI(__flush_dcache_area)13SYM_FUNC_END_PI(dcache_clean_inval_poc)
arch/arm64/kvm/hyp/nvhe/setup.c+2−1
@@ -134,7 +134,8 @@ static void update_nvhe_init_params(void) · 133 unmodified lines
134 for (i = 0; i < hyp_nr_cpus; i++) {134 for (i = 0; i < hyp_nr_cpus; i++) {
135 params = per_cpu_ptr(&kvm_init_params, i);135 params = per_cpu_ptr(&kvm_init_params, i);
136 params->pgd_pa = __hyp_pa(pkvm_pgtable.pgd);136 params->pgd_pa = __hyp_pa(pkvm_pgtable.pgd);
137 __flush_dcache_area(params, sizeof(*params));137 dcache_clean_inval_poc((unsigned long)params,
138 (unsigned long)params + sizeof(*params));
138 }139 }
139}140}
140 141
arch/arm64/kvm/hyp/nvhe/tlb.c+1−1
@@ -104,7 +104,7 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, · 103 unmodified lines
104 * you should be running with VHE enabled.104 * you should be running with VHE enabled.
105 */105 */
106 if (icache_is_vpipt())106 if (icache_is_vpipt())
107 __flush_icache_all();107 icache_inval_all_pou();
108 108
109 __tlb_switch_to_host(&cxt);109 __tlb_switch_to_host(&cxt);
110}110}
arch/arm64/kvm/hyp/pgtable.c+10−3
@@ -839,8 +839,11 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, · 838 unmodified lines
839 stage2_put_pte(ptep, mmu, addr, level, mm_ops);839 stage2_put_pte(ptep, mmu, addr, level, mm_ops);
840 840
841 if (need_flush) {841 if (need_flush) {
842 __flush_dcache_area(kvm_pte_follow(pte, mm_ops),842 kvm_pte_t *pte_follow = kvm_pte_follow(pte, mm_ops);
843 kvm_granule_size(level));843
844 dcache_clean_inval_poc((unsigned long)pte_follow,
845 (unsigned long)pte_follow +
846 kvm_granule_size(level));
844 }847 }
845 848
846 if (childp)849 if (childp)
@@ -988,11 +991,15 @@ static int stage2_flush_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, · 141 unmodified lines
988 struct kvm_pgtable *pgt = arg;991 struct kvm_pgtable *pgt = arg;
989 struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops;992 struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops;
990 kvm_pte_t pte = *ptep;993 kvm_pte_t pte = *ptep;
994 kvm_pte_t *pte_follow;
991 995
992 if (!kvm_pte_valid(pte) || !stage2_pte_cacheable(pgt, pte))996 if (!kvm_pte_valid(pte) || !stage2_pte_cacheable(pgt, pte))
993 return 0;997 return 0;
994 998
995 __flush_dcache_area(kvm_pte_follow(pte, mm_ops), kvm_granule_size(level));999 pte_follow = kvm_pte_follow(pte, mm_ops);
1000 dcache_clean_inval_poc((unsigned long)pte_follow,
1001 (unsigned long)pte_follow +
1002 kvm_granule_size(level));
996 return 0;1003 return 0;
997}1004}
998 1005
arch/arm64/lib/Makefile+3−1
@@ -1,7 +1,7 @@
1# SPDX-License-Identifier: GPL-2.01# SPDX-License-Identifier: GPL-2.0
2lib-y := clear_user.o delay.o copy_from_user.o \2lib-y := clear_user.o delay.o copy_from_user.o \
3 copy_to_user.o copy_in_user.o copy_page.o \3 copy_to_user.o copy_in_user.o copy_page.o \
4 clear_page.o csum.o memchr.o memcpy.o memmove.o \4 clear_page.o csum.o insn.o memchr.o memcpy.o \
5 memset.o memcmp.o strcmp.o strncmp.o strlen.o \5 memset.o memcmp.o strcmp.o strncmp.o strlen.o \
6 strnlen.o strchr.o strrchr.o tishift.o6 strnlen.o strchr.o strrchr.o tishift.o
7 7
@@ -18,3 +18,5 @@ obj-$(CONFIG_CRC32) += crc32.o · 10 unmodified lines
18obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o18obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
19 19
20obj-$(CONFIG_ARM64_MTE) += mte.o20obj-$(CONFIG_ARM64_MTE) += mte.o
21
22obj-$(CONFIG_KASAN_SW_TAGS) += kasan_sw_tags.o
arch/arm64/lib/clear_user.S+27−20
@@ -1,12 +1,9 @@
1/* SPDX-License-Identifier: GPL-2.0-only */1/* SPDX-License-Identifier: GPL-2.0-only */
2/*2/*
3 * Based on arch/arm/lib/clear_user.S3 * Copyright (C) 2021 Arm Ltd.
4 *
5 * Copyright (C) 2012 ARM Ltd.
6 */4 */
7#include <linux/linkage.h>
8 5
9#include <asm/asm-uaccess.h>6#include <linux/linkage.h>
10#include <asm/assembler.h>7#include <asm/assembler.h>
11 8
12 .text9 .text
@@ -19,25 +16,33 @@ · 6 unmodified lines
19 *16 *
20 * Alignment fixed up by hardware.17 * Alignment fixed up by hardware.
21 */18 */
19
20 .p2align 4
21 // Alignment is for the loop, but since the prologue (including BTI)
22 // is also 16 bytes we can keep any padding outside the function
22SYM_FUNC_START(__arch_clear_user)23SYM_FUNC_START(__arch_clear_user)
23 mov x2, x1 // save the size for fixup return24 add x2, x0, x1
24 subs x1, x1, #825 subs x1, x1, #8
25 b.mi 2f26 b.mi 2f
261:271:
27user_ldst 9f, sttr, xzr, x0, 828USER(9f, sttr xzr, [x0])
29 add x0, x0, #8
28 subs x1, x1, #830 subs x1, x1, #8
29 b.pl 1b31 b.hi 1b
302: adds x1, x1, #432USER(9f, sttr xzr, [x2, #-8])
31 b.mi 3f33 mov x0, #0
32user_ldst 9f, sttr, wzr, x0, 434 ret
33 sub x1, x1, #435
343: adds x1, x1, #2362: tbz x1, #2, 3f
35 b.mi 4f37USER(9f, sttr wzr, [x0])
36user_ldst 9f, sttrh, wzr, x0, 238USER(8f, sttr wzr, [x2, #-4])
37 sub x1, x1, #239 mov x0, #0
384: adds x1, x1, #140 ret
39 b.mi 5f41
40user_ldst 9f, sttrb, wzr, x0, 0423: tbz x1, #1, 4f
43USER(9f, sttrh wzr, [x0])
444: tbz x1, #0, 5f
45USER(7f, sttrb wzr, [x2, #-1])
415: mov x0, #0465: mov x0, #0
42 ret47 ret
43SYM_FUNC_END(__arch_clear_user)48SYM_FUNC_END(__arch_clear_user)
@@ -45,6 +50,8 @@ EXPORT_SYMBOL(__arch_clear_user) · 1 unmodified line
45 50
46 .section .fixup,"ax"51 .section .fixup,"ax"
47 .align 252 .align 2
489: mov x0, x2 // return the original size537: sub x0, x2, #5 // Adjust for faulting on the final byte...
548: add x0, x0, #4 // ...or the second word of the 4-7 byte case
559: sub x0, x2, x0
49 ret56 ret
50 .previous57 .previous
arch/arm64/lib/insn.crenamed from arch/arm64/kernel/insn.c+4−245
@@ -7,21 +7,14 @@ · 6 unmodified lines
7 */7 */
8#include <linux/bitops.h>8#include <linux/bitops.h>
9#include <linux/bug.h>9#include <linux/bug.h>
10#include <linux/compiler.h>10#include <linux/printk.h>
11#include <linux/kernel.h>11#include <linux/sizes.h>
12#include <linux/mm.h>
13#include <linux/smp.h>
14#include <linux/spinlock.h>
15#include <linux/stop_machine.h>
16#include <linux/types.h>12#include <linux/types.h>
17#include <linux/uaccess.h>
18 13
19#include <asm/cacheflush.h>
20#include <asm/debug-monitors.h>14#include <asm/debug-monitors.h>
21#include <asm/fixmap.h>15#include <asm/errno.h>
22#include <asm/insn.h>16#include <asm/insn.h>
23#include <asm/kprobes.h>17#include <asm/kprobes.h>
24#include <asm/sections.h>
25 18
26#define AARCH64_INSN_SF_BIT BIT(31)19#define AARCH64_INSN_SF_BIT BIT(31)
27#define AARCH64_INSN_N_BIT BIT(22)20#define AARCH64_INSN_N_BIT BIT(22)
@@ -30,7 +23,7 @@ · 2 unmodified lines
30static const int aarch64_insn_encoding_class[] = {23static const int aarch64_insn_encoding_class[] = {
31 AARCH64_INSN_CLS_UNKNOWN,24 AARCH64_INSN_CLS_UNKNOWN,
32 AARCH64_INSN_CLS_UNKNOWN,25 AARCH64_INSN_CLS_UNKNOWN,
33 AARCH64_INSN_CLS_UNKNOWN,26 AARCH64_INSN_CLS_SVE,
34 AARCH64_INSN_CLS_UNKNOWN,27 AARCH64_INSN_CLS_UNKNOWN,
35 AARCH64_INSN_CLS_LDST,28 AARCH64_INSN_CLS_LDST,
36 AARCH64_INSN_CLS_DP_REG,29 AARCH64_INSN_CLS_DP_REG,
@@ -83,81 +76,6 @@ bool aarch64_insn_is_branch_imm(u32 insn) · 46 unmodified lines
83 aarch64_insn_is_bcond(insn));76 aarch64_insn_is_bcond(insn));
84}77}
85 78
86static DEFINE_RAW_SPINLOCK(patch_lock);
87
88static bool is_exit_text(unsigned long addr)
89{
90 /* discarded with init text/data */
91 return system_state < SYSTEM_RUNNING &&
92 addr >= (unsigned long)__exittext_begin &&
93 addr < (unsigned long)__exittext_end;
94}
95
96static bool is_image_text(unsigned long addr)
97{
98 return core_kernel_text(addr) || is_exit_text(addr);
99}
100
101static void __kprobes *patch_map(void *addr, int fixmap)
102{
103 unsigned long uintaddr = (uintptr_t) addr;
104 bool image = is_image_text(uintaddr);
105 struct page *page;
106
107 if (image)
108 page = phys_to_page(__pa_symbol(addr));
109 else if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX))
110 page = vmalloc_to_page(addr);
111 else
112 return addr;
113
114 BUG_ON(!page);
115 return (void *)set_fixmap_offset(fixmap, page_to_phys(page) +
116 (uintaddr & ~PAGE_MASK));
117}
118
119static void __kprobes patch_unmap(int fixmap)
120{
121 clear_fixmap(fixmap);
122}
123/*
124 * In ARMv8-A, A64 instructions have a fixed length of 32 bits and are always
125 * little-endian.
126 */
127int __kprobes aarch64_insn_read(void *addr, u32 *insnp)
128{
129 int ret;
130 __le32 val;
131
132 ret = copy_from_kernel_nofault(&val, addr, AARCH64_INSN_SIZE);
133 if (!ret)
134 *insnp = le32_to_cpu(val);
135
136 return ret;
137}
138
139static int __kprobes __aarch64_insn_write(void *addr, __le32 insn)
140{
141 void *waddr = addr;
142 unsigned long flags = 0;
143 int ret;
144
145 raw_spin_lock_irqsave(&patch_lock, flags);
146 waddr = patch_map(addr, FIX_TEXT_POKE0);
147
148 ret = copy_to_kernel_nofault(waddr, &insn, AARCH64_INSN_SIZE);
149
150 patch_unmap(FIX_TEXT_POKE0);
151 raw_spin_unlock_irqrestore(&patch_lock, flags);
152
153 return ret;
154}
155
156int __kprobes aarch64_insn_write(void *addr, u32 insn)
157{
158 return __aarch64_insn_write(addr, cpu_to_le32(insn));
159}
160
161bool __kprobes aarch64_insn_uses_literal(u32 insn)79bool __kprobes aarch64_insn_uses_literal(u32 insn)
162{80{
163 /* ldr/ldrsw (literal), prfm */81 /* ldr/ldrsw (literal), prfm */
@@ -187,67 +105,6 @@ bool __kprobes aarch64_insn_is_branch(u32 insn) · 23 unmodified lines
187 aarch64_insn_is_bcond(insn);105 aarch64_insn_is_bcond(insn);
188}106}
189 107
190int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn)
191{
192 u32 *tp = addr;
193 int ret;
194
195 /* A64 instructions must be word aligned */
196 if ((uintptr_t)tp & 0x3)
197 return -EINVAL;
198
199 ret = aarch64_insn_write(tp, insn);
200 if (ret == 0)
201 __flush_icache_range((uintptr_t)tp,
202 (uintptr_t)tp + AARCH64_INSN_SIZE);
203
204 return ret;
205}
206
207struct aarch64_insn_patch {
208 void **text_addrs;
209 u32 *new_insns;
210 int insn_cnt;
211 atomic_t cpu_count;
212};
213
214static int __kprobes aarch64_insn_patch_text_cb(void *arg)
215{
216 int i, ret = 0;
217 struct aarch64_insn_patch *pp = arg;
218
219 /* The first CPU becomes master */
220 if (atomic_inc_return(&pp->cpu_count) == 1) {
221 for (i = 0; ret == 0 && i < pp->insn_cnt; i++)
222 ret = aarch64_insn_patch_text_nosync(pp->text_addrs[i],
223 pp->new_insns[i]);
224 /* Notify other processors with an additional increment. */
225 atomic_inc(&pp->cpu_count);
226 } else {
227 while (atomic_read(&pp->cpu_count) <= num_online_cpus())
228 cpu_relax();
229 isb();
230 }
231
232 return ret;
233}
234
235int __kprobes aarch64_insn_patch_text(void *addrs[], u32 insns[], int cnt)
236{
237 struct aarch64_insn_patch patch = {
238 .text_addrs = addrs,
239 .new_insns = insns,
240 .insn_cnt = cnt,
241 .cpu_count = ATOMIC_INIT(0),
242 };
243
244 if (cnt <= 0)
245 return -EINVAL;
246
247 return stop_machine_cpuslocked(aarch64_insn_patch_text_cb, &patch,
248 cpu_online_mask);
249}
250
251static int __kprobes aarch64_get_imm_shift_mask(enum aarch64_insn_imm_type type,108static int __kprobes aarch64_get_imm_shift_mask(enum aarch64_insn_imm_type type,
252 u32 *maskp, int *shiftp)109 u32 *maskp, int *shiftp)
253{110{
@@ -1432,104 +1289,6 @@ u32 aarch32_insn_mcr_extract_crm(u32 insn) · 1178 unmodified lines
1432 return insn & CRM_MASK;1289 return insn & CRM_MASK;
1433}1290}
1434 1291
1435static bool __kprobes __check_eq(unsigned long pstate)
1436{
1437 return (pstate & PSR_Z_BIT) != 0;
1438}
1439
1440static bool __kprobes __check_ne(unsigned long pstate)
1441{
1442 return (pstate & PSR_Z_BIT) == 0;
1443}
1444
1445static bool __kprobes __check_cs(unsigned long pstate)
1446{
1447 return (pstate & PSR_C_BIT) != 0;
1448}
1449
1450static bool __kprobes __check_cc(unsigned long pstate)
1451{
1452 return (pstate & PSR_C_BIT) == 0;
1453}
1454
1455static bool __kprobes __check_mi(unsigned long pstate)
1456{
1457 return (pstate & PSR_N_BIT) != 0;
1458}
1459
1460static bool __kprobes __check_pl(unsigned long pstate)
1461{
1462 return (pstate & PSR_N_BIT) == 0;
1463}
1464
1465static bool __kprobes __check_vs(unsigned long pstate)
1466{
1467 return (pstate & PSR_V_BIT) != 0;
1468}
1469
1470static bool __kprobes __check_vc(unsigned long pstate)
1471{
1472 return (pstate & PSR_V_BIT) == 0;
1473}
1474
1475static bool __kprobes __check_hi(unsigned long pstate)
1476{
1477 pstate &= ~(pstate >> 1); /* PSR_C_BIT &= ~PSR_Z_BIT */
1478 return (pstate & PSR_C_BIT) != 0;
1479}
1480
1481static bool __kprobes __check_ls(unsigned long pstate)
1482{
1483 pstate &= ~(pstate >> 1); /* PSR_C_BIT &= ~PSR_Z_BIT */
1484 return (pstate & PSR_C_BIT) == 0;
1485}
1486
1487static bool __kprobes __check_ge(unsigned long pstate)
1488{
1489 pstate ^= (pstate << 3); /* PSR_N_BIT ^= PSR_V_BIT */
1490 return (pstate & PSR_N_BIT) == 0;
1491}
1492
1493static bool __kprobes __check_lt(unsigned long pstate)
1494{
1495 pstate ^= (pstate << 3); /* PSR_N_BIT ^= PSR_V_BIT */
1496 return (pstate & PSR_N_BIT) != 0;
1497}
1498
1499static bool __kprobes __check_gt(unsigned long pstate)
1500{
1501 /*PSR_N_BIT ^= PSR_V_BIT */
1502 unsigned long temp = pstate ^ (pstate << 3);
1503
1504 temp |= (pstate << 1); /*PSR_N_BIT |= PSR_Z_BIT */
1505 return (temp & PSR_N_BIT) == 0;
1506}
1507
1508static bool __kprobes __check_le(unsigned long pstate)
1509{
1510 /*PSR_N_BIT ^= PSR_V_BIT */
1511 unsigned long temp = pstate ^ (pstate << 3);
1512
1513 temp |= (pstate << 1); /*PSR_N_BIT |= PSR_Z_BIT */
1514 return (temp & PSR_N_BIT) != 0;
1515}
1516
1517static bool __kprobes __check_al(unsigned long pstate)
1518{
1519 return true;
1520}
1521
1522/*
1523 * Note that the ARMv8 ARM calls condition code 0b1111 "nv", but states that
1524 * it behaves identically to 0b1110 ("al").
1525 */
1526pstate_check_t * const aarch32_opcode_cond_checks[16] = {
1527 __check_eq, __check_ne, __check_cs, __check_cc,
1528 __check_mi, __check_pl, __check_vs, __check_vc,
1529 __check_hi, __check_ls, __check_ge, __check_lt,
1530 __check_gt, __check_le, __check_al, __check_al
1531};
1532
1533static bool range_of_ones(u64 val)1292static bool range_of_ones(u64 val)
1534{1293{
1535 /* Doesn't handle full ones or full zeroes */1294 /* Doesn't handle full ones or full zeroes */
arch/arm64/lib/kasan_sw_tags.S+76−0
@@ -0,0 +1,76 @@
1/* SPDX-License-Identifier: GPL-2.0-only */
2/*
3 * Copyright (C) 2020 Google LLC
4 */
5
6#include <linux/linkage.h>
7#include <asm/assembler.h>
8
9/*
10 * Report a tag mismatch detected by tag-based KASAN.
11 *
12 * A compiler-generated thunk calls this with a non-AAPCS calling
13 * convention. Upon entry to this function, registers are as follows:
14 *
15 * x0: fault address (see below for restore)
16 * x1: fault description (see below for restore)
17 * x2 to x15: callee-saved
18 * x16 to x17: safe to clobber
19 * x18 to x30: callee-saved
20 * sp: pre-decremented by 256 bytes (see below for restore)
21 *
22 * The caller has decremented the SP by 256 bytes, and created a
23 * structure on the stack as follows:
24 *
25 * sp + 0..15: x0 and x1 to be restored
26 * sp + 16..231: free for use
27 * sp + 232..247: x29 and x30 (same as in GPRs)
28 * sp + 248..255: free for use
29 *
30 * Note that this is not a struct pt_regs.
31 *
32 * To call a regular AAPCS function we must save x2 to x15 (which we can
33 * store in the gaps), and create a frame record (for which we can use
34 * x29 and x30 spilled by the caller as those match the GPRs).
35 *
36 * The caller expects x0 and x1 to be restored from the structure, and
37 * for the structure to be removed from the stack (i.e. the SP must be
38 * incremented by 256 prior to return).
39 */
40SYM_CODE_START(__hwasan_tag_mismatch)
41#ifdef BTI_C
42 BTI_C
43#endif
44 add x29, sp, #232
45 stp x2, x3, [sp, #8 * 2]
46 stp x4, x5, [sp, #8 * 4]
47 stp x6, x7, [sp, #8 * 6]
48 stp x8, x9, [sp, #8 * 8]
49 stp x10, x11, [sp, #8 * 10]
50 stp x12, x13, [sp, #8 * 12]
51 stp x14, x15, [sp, #8 * 14]
52#ifndef CONFIG_SHADOW_CALL_STACK
53 str x18, [sp, #8 * 18]
54#endif
55
56 mov x2, x30
57 bl kasan_tag_mismatch
58
59 ldp x0, x1, [sp]
60 ldp x2, x3, [sp, #8 * 2]
61 ldp x4, x5, [sp, #8 * 4]
62 ldp x6, x7, [sp, #8 * 6]
63 ldp x8, x9, [sp, #8 * 8]
64 ldp x10, x11, [sp, #8 * 10]
65 ldp x12, x13, [sp, #8 * 12]
66 ldp x14, x15, [sp, #8 * 14]
67#ifndef CONFIG_SHADOW_CALL_STACK
68 ldr x18, [sp, #8 * 18]
69#endif
70 ldp x29, x30, [sp, #8 * 29]
71
72 /* remove the structure from the stack */
73 add sp, sp, #256
74 ret
75SYM_CODE_END(__hwasan_tag_mismatch)
76EXPORT_SYMBOL(__hwasan_tag_mismatch)
arch/arm64/lib/memchr.S+53−12
@@ -1,9 +1,6 @@
1/* SPDX-License-Identifier: GPL-2.0-only */1/* SPDX-License-Identifier: GPL-2.0-only */
2/*2/*
3 * Based on arch/arm/lib/memchr.S3 * Copyright (C) 2021 Arm Ltd.
4 *
5 * Copyright (C) 1995-2000 Russell King
6 * Copyright (C) 2013 ARM Ltd.
7 */4 */
8 5
9#include <linux/linkage.h>6#include <linux/linkage.h>
@@ -19,16 +16,60 @@ · 9 unmodified lines
19 * Returns:16 * Returns:
20 * x0 - address of first occurrence of 'c' or 017 * x0 - address of first occurrence of 'c' or 0
21 */18 */
19
20#define L(label) .L ## label
21
22#define REP8_01 0x0101010101010101
23#define REP8_7f 0x7f7f7f7f7f7f7f7f
24
25#define srcin x0
26#define chrin w1
27#define cntin x2
28
29#define result x0
30
31#define wordcnt x3
32#define rep01 x4
33#define repchr x5
34#define cur_word x6
35#define cur_byte w6
36#define tmp x7
37#define tmp2 x8
38
39 .p2align 4
40 nop
22SYM_FUNC_START_WEAK_PI(memchr)41SYM_FUNC_START_WEAK_PI(memchr)
23 and w1, w1, #0xff42 and chrin, chrin, #0xff
241: subs x2, x2, #143 lsr wordcnt, cntin, #3
25 b.mi 2f44 cbz wordcnt, L(byte_loop)
26 ldrb w3, [x0], #145 mov rep01, #REP8_01
27 cmp w3, w146 mul repchr, x1, rep01
28 b.ne 1b47 and cntin, cntin, #7
29 sub x0, x0, #148L(word_loop):
49 ldr cur_word, [srcin], #8
50 sub wordcnt, wordcnt, #1
51 eor cur_word, cur_word, repchr
52 sub tmp, cur_word, rep01
53 orr tmp2, cur_word, #REP8_7f
54 bics tmp, tmp, tmp2
55 b.ne L(found_word)
56 cbnz wordcnt, L(word_loop)
57L(byte_loop):
58 cbz cntin, L(not_found)
59 ldrb cur_byte, [srcin], #1
60 sub cntin, cntin, #1
61 cmp cur_byte, chrin
62 b.ne L(byte_loop)
63 sub srcin, srcin, #1
64 ret
65L(found_word):
66CPU_LE( rev tmp, tmp)
67 clz tmp, tmp
68 sub tmp, tmp, #64
69 add result, srcin, tmp, asr #3
30 ret70 ret
312: mov x0, #071L(not_found):
72 mov result, #0
32 ret73 ret
33SYM_FUNC_END_PI(memchr)74SYM_FUNC_END_PI(memchr)
34EXPORT_SYMBOL_NOKASAN(memchr)75EXPORT_SYMBOL_NOKASAN(memchr)
arch/arm64/lib/memcmp.S+119−227
@@ -1,247 +1,139 @@
1/* SPDX-License-Identifier: GPL-2.0-only */1/* SPDX-License-Identifier: GPL-2.0-only */
2/*2/*
3 * Copyright (C) 2013 ARM Ltd.3 * Copyright (c) 2013-2021, Arm Limited.
4 * Copyright (C) 2013 Linaro.
5 *4 *
6 * This code is based on glibc cortex strings work originally authored by Linaro5 * Adapted from the original at:
7 * be found @6 * https://github.com/ARM-software/optimized-routines/blob/e823e3abf5f89ecb/string/aarch64/memcmp.S
8 *
9 * http://bazaar.launchpad.net/~linaro-toolchain-dev/cortex-strings/trunk/
10 * files/head:/src/aarch64/
11 */7 */
12 8
13#include <linux/linkage.h>9#include <linux/linkage.h>
14#include <asm/assembler.h>10#include <asm/assembler.h>
15 11
16/*12/* Assumptions:
17* compare memory areas(when two memory areas' offset are different,13 *
18* alignment handled by the hardware)14 * ARMv8-a, AArch64, unaligned accesses.
19*15 */
20* Parameters:16
21* x0 - const memory area 1 pointer17#define L(label) .L ## label
22* x1 - const memory area 2 pointer
23* x2 - the maximal compare byte length
24* Returns:
25* x0 - a compare result, maybe less than, equal to, or greater than ZERO
26*/
27 18
28/* Parameters and result. */19/* Parameters and result. */
29src1 .req x020#define src1 x0
30src2 .req x121#define src2 x1
31limit .req x222#define limit x2
32result .req x023#define result w0
33 24
34/* Internal variables. */25/* Internal variables. */
35data1 .req x326#define data1 x3
36data1w .req w327#define data1w w3
37data2 .req x428#define data1h x4
38data2w .req w429#define data2 x5
39has_nul .req x530#define data2w w5
40diff .req x631#define data2h x6
41endloop .req x732#define tmp1 x7
42tmp1 .req x833#define tmp2 x8
43tmp2 .req x9
44tmp3 .req x10
45pos .req x11
46limit_wd .req x12
47mask .req x13
48 34
49SYM_FUNC_START_WEAK_PI(memcmp)35SYM_FUNC_START_WEAK_PI(memcmp)
50 cbz limit, .Lret036 subs limit, limit, 8
51 eor tmp1, src1, src237 b.lo L(less8)
52 tst tmp1, #738
53 b.ne .Lmisaligned839 ldr data1, [src1], 8
54 ands tmp1, src1, #740 ldr data2, [src2], 8
55 b.ne .Lmutual_align41 cmp data1, data2
56 sub limit_wd, limit, #1 /* limit != 0, so no underflow. */42 b.ne L(return)
57 lsr limit_wd, limit_wd, #3 /* Convert to Dwords. */43
58 /*44 subs limit, limit, 8
59 * The input source addresses are at alignment boundary.45 b.gt L(more16)
60 * Directly compare eight bytes each time.46
61 */47 ldr data1, [src1, limit]
62.Lloop_aligned:48 ldr data2, [src2, limit]
63 ldr data1, [src1], #849 b L(return)
64 ldr data2, [src2], #850
65.Lstart_realigned:51L(more16):
66 subs limit_wd, limit_wd, #152 ldr data1, [src1], 8
67 eor diff, data1, data2 /* Non-zero if differences found. */53 ldr data2, [src2], 8
68 csinv endloop, diff, xzr, cs /* Last Dword or differences. */54 cmp data1, data2
69 cbz endloop, .Lloop_aligned55 bne L(return)
70 56
71 /* Not reached the limit, must have found a diff. */57 /* Jump directly to comparing the last 16 bytes for 32 byte (or less)
72 tbz limit_wd, #63, .Lnot_limit58 strings. */
73 59 subs limit, limit, 16
74 /* Limit % 8 == 0 => the diff is in the last 8 bytes. */60 b.ls L(last_bytes)
75 ands limit, limit, #761
76 b.eq .Lnot_limit62 /* We overlap loads between 0-32 bytes at either side of SRC1 when we
77 /*63 try to align, so limit it only to strings larger than 128 bytes. */
78 * The remained bytes less than 8. It is needed to extract valid data64 cmp limit, 96
79 * from last eight bytes of the intended memory range.65 b.ls L(loop16)
80 */66
81 lsl limit, limit, #3 /* bytes-> bits. */67 /* Align src1 and adjust src2 with bytes not yet done. */
82 mov mask, #~068 and tmp1, src1, 15
83CPU_BE( lsr mask, mask, limit )69 add limit, limit, tmp1
84CPU_LE( lsl mask, mask, limit )70 sub src1, src1, tmp1
85 bic data1, data1, mask71 sub src2, src2, tmp1
86 bic data2, data2, mask72
87 73 /* Loop performing 16 bytes per iteration using aligned src1.
88 orr diff, diff, mask74 Limit is pre-decremented by 16 and must be larger than zero.
89 b .Lnot_limit75 Exit if <= 16 bytes left to do or if the data is not equal. */
90 76 .p2align 4
91.Lmutual_align:77L(loop16):
92 /*78 ldp data1, data1h, [src1], 16
93 * Sources are mutually aligned, but are not currently at an79 ldp data2, data2h, [src2], 16
94 * alignment boundary. Round down the addresses and then mask off80 subs limit, limit, 16
95 * the bytes that precede the start point.81 ccmp data1, data2, 0, hi
96 */82 ccmp data1h, data2h, 0, eq
97 bic src1, src1, #783 b.eq L(loop16)
98 bic src2, src2, #784
99 ldr data1, [src1], #885 cmp data1, data2
100 ldr data2, [src2], #886 bne L(return)
101 /*87 mov data1, data1h
102 * We can not add limit with alignment offset(tmp1) here. Since the88 mov data2, data2h
103 * addition probably make the limit overflown.89 cmp data1, data2
104 */90 bne L(return)
105 sub limit_wd, limit, #1/*limit != 0, so no underflow.*/91
106 and tmp3, limit_wd, #792 /* Compare last 1-16 bytes using unaligned access. */
107 lsr limit_wd, limit_wd, #393L(last_bytes):
108 add tmp3, tmp3, tmp194 add src1, src1, limit
109 add limit_wd, limit_wd, tmp3, lsr #395 add src2, src2, limit
110 add limit, limit, tmp1/* Adjust the limit for the extra. */96 ldp data1, data1h, [src1]
111 97 ldp data2, data2h, [src2]
112 lsl tmp1, tmp1, #3/* Bytes beyond alignment -> bits.*/98 cmp data1, data2
113 neg tmp1, tmp1/* Bits to alignment -64. */99 bne L(return)
114 mov tmp2, #~0100 mov data1, data1h
115 /*mask off the non-intended bytes before the start address.*/101 mov data2, data2h
116CPU_BE( lsl tmp2, tmp2, tmp1 )/*Big-endian.Early bytes are at MSB*/102 cmp data1, data2
117 /* Little-endian. Early bytes are at LSB. */103
118CPU_LE( lsr tmp2, tmp2, tmp1 )104 /* Compare data bytes and set return value to 0, -1 or 1. */
119 105L(return):
120 orr data1, data1, tmp2106#ifndef __AARCH64EB__
121 orr data2, data2, tmp2107 rev data1, data1
122 b .Lstart_realigned108 rev data2, data2
123 109#endif
124 /*src1 and src2 have different alignment offset.*/110 cmp data1, data2
125.Lmisaligned8:111L(ret_eq):
126 cmp limit, #8112 cset result, ne
127 b.lo .Ltiny8proc /*limit < 8: compare byte by byte*/113 cneg result, result, lo
128
129 and tmp1, src1, #7
130 neg tmp1, tmp1
131 add tmp1, tmp1, #8/*valid length in the first 8 bytes of src1*/
132 and tmp2, src2, #7
133 neg tmp2, tmp2
134 add tmp2, tmp2, #8/*valid length in the first 8 bytes of src2*/
135 subs tmp3, tmp1, tmp2
136 csel pos, tmp1, tmp2, hi /*Choose the maximum.*/
137
138 sub limit, limit, pos
139 /*compare the proceeding bytes in the first 8 byte segment.*/
140.Ltinycmp:
141 ldrb data1w, [src1], #1
142 ldrb data2w, [src2], #1
143 subs pos, pos, #1
144 ccmp data1w, data2w, #0, ne /* NZCV = 0b0000. */
145 b.eq .Ltinycmp
146 cbnz pos, 1f /*diff occurred before the last byte.*/
147 cmp data1w, data2w
148 b.eq .Lstart_align
1491:
150 sub result, data1, data2
151 ret114 ret
152 115
153.Lstart_align:116 .p2align 4
154 lsr limit_wd, limit, #3117 /* Compare up to 8 bytes. Limit is [-8..-1]. */
155 cbz limit_wd, .Lremain8118L(less8):
156 119 adds limit, limit, 4
157 ands xzr, src1, #7120 b.lo L(less4)
158 b.eq .Lrecal_offset121 ldr data1w, [src1], 4
159 /*process more leading bytes to make src1 aligned...*/122 ldr data2w, [src2], 4
160 add src1, src1, tmp3 /*backwards src1 to alignment boundary*/123 cmp data1w, data2w
161 add src2, src2, tmp3124 b.ne L(return)
162 sub limit, limit, tmp3125 sub limit, limit, 4
163 lsr limit_wd, limit, #3126L(less4):
164 cbz limit_wd, .Lremain8127 adds limit, limit, 4
165 /*load 8 bytes from aligned SRC1..*/128 beq L(ret_eq)
166 ldr data1, [src1], #8129L(byte_loop):
167 ldr data2, [src2], #8130 ldrb data1w, [src1], 1
168 131 ldrb data2w, [src2], 1
169 subs limit_wd, limit_wd, #1132 subs limit, limit, 1
170 eor diff, data1, data2 /*Non-zero if differences found.*/133 ccmp data1w, data2w, 0, ne /* NZCV = 0b0000. */
171 csinv endloop, diff, xzr, ne134 b.eq L(byte_loop)
172 cbnz endloop, .Lunequal_proc135 sub result, data1w, data2w
173 /*How far is the current SRC2 from the alignment boundary...*/
174 and tmp3, tmp3, #7
175
176.Lrecal_offset:/*src1 is aligned now..*/
177 neg pos, tmp3
178.Lloopcmp_proc:
179 /*
180 * Divide the eight bytes into two parts. First,backwards the src2
181 * to an alignment boundary,load eight bytes and compare from
182 * the SRC2 alignment boundary. If all 8 bytes are equal,then start
183 * the second part's comparison. Otherwise finish the comparison.
184 * This special handle can garantee all the accesses are in the
185 * thread/task space in avoid to overrange access.
186 */
187 ldr data1, [src1,pos]
188 ldr data2, [src2,pos]
189 eor diff, data1, data2 /* Non-zero if differences found. */
190 cbnz diff, .Lnot_limit
191
192 /*The second part process*/
193 ldr data1, [src1], #8
194 ldr data2, [src2], #8
195 eor diff, data1, data2 /* Non-zero if differences found. */
196 subs limit_wd, limit_wd, #1
197 csinv endloop, diff, xzr, ne/*if limit_wd is 0,will finish the cmp*/
198 cbz endloop, .Lloopcmp_proc
199.Lunequal_proc:
200 cbz diff, .Lremain8
201
202/* There is difference occurred in the latest comparison. */
203.Lnot_limit:
204/*
205* For little endian,reverse the low significant equal bits into MSB,then
206* following CLZ can find how many equal bits exist.
207*/
208CPU_LE( rev diff, diff )
209CPU_LE( rev data1, data1 )
210CPU_LE( rev data2, data2 )
211
212 /*
213 * The MS-non-zero bit of DIFF marks either the first bit
214 * that is different, or the end of the significant data.
215 * Shifting left now will bring the critical information into the
216 * top bits.
217 */
218 clz pos, diff
219 lsl data1, data1, pos
220 lsl data2, data2, pos
221 /*
222 * We need to zero-extend (char is unsigned) the value and then
223 * perform a signed subtraction.
224 */
225 lsr data1, data1, #56
226 sub result, data1, data2, lsr #56
227 ret136 ret
228 137
229.Lremain8:
230 /* Limit % 8 == 0 =>. all data are equal.*/
231 ands limit, limit, #7
232 b.eq .Lret0
233
234.Ltiny8proc:
235 ldrb data1w, [src1], #1
236 ldrb data2w, [src2], #1
237 subs limit, limit, #1
238
239 ccmp data1w, data2w, #0, ne /* NZCV = 0b0000. */
240 b.eq .Ltiny8proc
241 sub result, data1, data2
242 ret
243.Lret0:
244 mov result, #0
245 ret
246SYM_FUNC_END_PI(memcmp)138SYM_FUNC_END_PI(memcmp)
247EXPORT_SYMBOL_NOKASAN(memcmp)139EXPORT_SYMBOL_NOKASAN(memcmp)
arch/arm64/lib/memcpy.S+229−43
@@ -1,66 +1,252 @@
1/* SPDX-License-Identifier: GPL-2.0-only */1/* SPDX-License-Identifier: GPL-2.0-only */
2/*2/*
3 * Copyright (C) 2013 ARM Ltd.3 * Copyright (c) 2012-2021, Arm Limited.
4 * Copyright (C) 2013 Linaro.
5 *4 *
6 * This code is based on glibc cortex strings work originally authored by Linaro5 * Adapted from the original at:
7 * be found @6 * https://github.com/ARM-software/optimized-routines/blob/afd6244a1f8d9229/string/aarch64/memcpy.S
8 *
9 * http://bazaar.launchpad.net/~linaro-toolchain-dev/cortex-strings/trunk/
10 * files/head:/src/aarch64/
11 */7 */
12 8
13#include <linux/linkage.h>9#include <linux/linkage.h>
14#include <asm/assembler.h>10#include <asm/assembler.h>
15#include <asm/cache.h>
16 11
17/*12/* Assumptions:
18 * Copy a buffer from src to dest (alignment handled by the hardware)13 *
14 * ARMv8-a, AArch64, unaligned accesses.
19 *15 *
20 * Parameters:
21 * x0 - dest
22 * x1 - src
23 * x2 - n
24 * Returns:
25 * x0 - dest
26 */16 */
27 .macro ldrb1 reg, ptr, val
28 ldrb \reg, [\ptr], \val
29 .endm
30
31 .macro strb1 reg, ptr, val
32 strb \reg, [\ptr], \val
33 .endm
34 17
35 .macro ldrh1 reg, ptr, val18#define L(label) .L ## label
36 ldrh \reg, [\ptr], \val
37 .endm
38 19
39 .macro strh1 reg, ptr, val20#define dstin x0
40 strh \reg, [\ptr], \val21#define src x1
41 .endm22#define count x2
23#define dst x3
24#define srcend x4
25#define dstend x5
26#define A_l x6
27#define A_lw w6
28#define A_h x7
29#define B_l x8
30#define B_lw w8
31#define B_h x9
32#define C_l x10
33#define C_lw w10
34#define C_h x11
35#define D_l x12
36#define D_h x13
37#define E_l x14
38#define E_h x15
39#define F_l x16
40#define F_h x17
41#define G_l count
42#define G_h dst
43#define H_l src
44#define H_h srcend
45#define tmp1 x14
42 46
43 .macro ldr1 reg, ptr, val47/* This implementation handles overlaps and supports both memcpy and memmove
44 ldr \reg, [\ptr], \val48 from a single entry point. It uses unaligned accesses and branchless
45 .endm49 sequences to keep the code small, simple and improve performance.
46 50
47 .macro str1 reg, ptr, val51 Copies are split into 3 main cases: small copies of up to 32 bytes, medium
48 str \reg, [\ptr], \val52 copies of up to 128 bytes, and large copies. The overhead of the overlap
49 .endm53 check is negligible since it is only required for large copies.
50 54
51 .macro ldp1 reg1, reg2, ptr, val55 Large copies use a software pipelined loop processing 64 bytes per iteration.
52 ldp \reg1, \reg2, [\ptr], \val56 The destination pointer is 16-byte aligned to minimize unaligned accesses.
53 .endm57 The loop tail is handled by always copying 64 bytes from the end.
54 58*/
55 .macro stp1 reg1, reg2, ptr, val
56 stp \reg1, \reg2, [\ptr], \val
57 .endm
58 59
60SYM_FUNC_START_ALIAS(__memmove)
61SYM_FUNC_START_WEAK_ALIAS_PI(memmove)
59SYM_FUNC_START_ALIAS(__memcpy)62SYM_FUNC_START_ALIAS(__memcpy)
60SYM_FUNC_START_WEAK_PI(memcpy)63SYM_FUNC_START_WEAK_PI(memcpy)
61#include "copy_template.S"64 add srcend, src, count
65 add dstend, dstin, count
66 cmp count, 128
67 b.hi L(copy_long)
68 cmp count, 32
69 b.hi L(copy32_128)
70
71 /* Small copies: 0..32 bytes. */
72 cmp count, 16
73 b.lo L(copy16)
74 ldp A_l, A_h, [src]
75 ldp D_l, D_h, [srcend, -16]
76 stp A_l, A_h, [dstin]
77 stp D_l, D_h, [dstend, -16]
78 ret
79
80 /* Copy 8-15 bytes. */
81L(copy16):
82 tbz count, 3, L(copy8)
83 ldr A_l, [src]
84 ldr A_h, [srcend, -8]
85 str A_l, [dstin]
86 str A_h, [dstend, -8]
87 ret
88
89 .p2align 3
90 /* Copy 4-7 bytes. */
91L(copy8):
92 tbz count, 2, L(copy4)
93 ldr A_lw, [src]
94 ldr B_lw, [srcend, -4]
95 str A_lw, [dstin]
96 str B_lw, [dstend, -4]
97 ret
98
99 /* Copy 0..3 bytes using a branchless sequence. */
100L(copy4):
101 cbz count, L(copy0)
102 lsr tmp1, count, 1
103 ldrb A_lw, [src]
104 ldrb C_lw, [srcend, -1]
105 ldrb B_lw, [src, tmp1]
106 strb A_lw, [dstin]
107 strb B_lw, [dstin, tmp1]
108 strb C_lw, [dstend, -1]
109L(copy0):
110 ret
111
112 .p2align 4
113 /* Medium copies: 33..128 bytes. */
114L(copy32_128):
115 ldp A_l, A_h, [src]
116 ldp B_l, B_h, [src, 16]
117 ldp C_l, C_h, [srcend, -32]
118 ldp D_l, D_h, [srcend, -16]
119 cmp count, 64
120 b.hi L(copy128)
121 stp A_l, A_h, [dstin]
122 stp B_l, B_h, [dstin, 16]
123 stp C_l, C_h, [dstend, -32]
124 stp D_l, D_h, [dstend, -16]
62 ret125 ret
126
127 .p2align 4
128 /* Copy 65..128 bytes. */
129L(copy128):
130 ldp E_l, E_h, [src, 32]
131 ldp F_l, F_h, [src, 48]
132 cmp count, 96
133 b.ls L(copy96)
134 ldp G_l, G_h, [srcend, -64]
135 ldp H_l, H_h, [srcend, -48]
136 stp G_l, G_h, [dstend, -64]
137 stp H_l, H_h, [dstend, -48]
138L(copy96):
139 stp A_l, A_h, [dstin]
140 stp B_l, B_h, [dstin, 16]
141 stp E_l, E_h, [dstin, 32]
142 stp F_l, F_h, [dstin, 48]
143 stp C_l, C_h, [dstend, -32]
144 stp D_l, D_h, [dstend, -16]
145 ret
146
147 .p2align 4
148 /* Copy more than 128 bytes. */
149L(copy_long):
150 /* Use backwards copy if there is an overlap. */
151 sub tmp1, dstin, src
152 cbz tmp1, L(copy0)
153 cmp tmp1, count
154 b.lo L(copy_long_backwards)
155
156 /* Copy 16 bytes and then align dst to 16-byte alignment. */
157
158 ldp D_l, D_h, [src]
159 and tmp1, dstin, 15
160 bic dst, dstin, 15
161 sub src, src, tmp1
162 add count, count, tmp1 /* Count is now 16 too large. */
163 ldp A_l, A_h, [src, 16]
164 stp D_l, D_h, [dstin]
165 ldp B_l, B_h, [src, 32]
166 ldp C_l, C_h, [src, 48]
167 ldp D_l, D_h, [src, 64]!
168 subs count, count, 128 + 16 /* Test and readjust count. */
169 b.ls L(copy64_from_end)
170
171L(loop64):
172 stp A_l, A_h, [dst, 16]
173 ldp A_l, A_h, [src, 16]
174 stp B_l, B_h, [dst, 32]
175 ldp B_l, B_h, [src, 32]
176 stp C_l, C_h, [dst, 48]
177 ldp C_l, C_h, [src, 48]
178 stp D_l, D_h, [dst, 64]!
179 ldp D_l, D_h, [src, 64]!
180 subs count, count, 64
181 b.hi L(loop64)
182
183 /* Write the last iteration and copy 64 bytes from the end. */
184L(copy64_from_end):
185 ldp E_l, E_h, [srcend, -64]
186 stp A_l, A_h, [dst, 16]
187 ldp A_l, A_h, [srcend, -48]
188 stp B_l, B_h, [dst, 32]
189 ldp B_l, B_h, [srcend, -32]
190 stp C_l, C_h, [dst, 48]
191 ldp C_l, C_h, [srcend, -16]
192 stp D_l, D_h, [dst, 64]
193 stp E_l, E_h, [dstend, -64]
194 stp A_l, A_h, [dstend, -48]
195 stp B_l, B_h, [dstend, -32]
196 stp C_l, C_h, [dstend, -16]
197 ret
198
199 .p2align 4
200
201 /* Large backwards copy for overlapping copies.
202 Copy 16 bytes and then align dst to 16-byte alignment. */
203L(copy_long_backwards):
204 ldp D_l, D_h, [srcend, -16]
205 and tmp1, dstend, 15
206 sub srcend, srcend, tmp1
207 sub count, count, tmp1
208 ldp A_l, A_h, [srcend, -16]
209 stp D_l, D_h, [dstend, -16]
210 ldp B_l, B_h, [srcend, -32]
211 ldp C_l, C_h, [srcend, -48]
212 ldp D_l, D_h, [srcend, -64]!
213 sub dstend, dstend, tmp1
214 subs count, count, 128
215 b.ls L(copy64_from_start)
216
217L(loop64_backwards):
218 stp A_l, A_h, [dstend, -16]
219 ldp A_l, A_h, [srcend, -16]
220 stp B_l, B_h, [dstend, -32]
221 ldp B_l, B_h, [srcend, -32]
222 stp C_l, C_h, [dstend, -48]
223 ldp C_l, C_h, [srcend, -48]
224 stp D_l, D_h, [dstend, -64]!
225 ldp D_l, D_h, [srcend, -64]!
226 subs count, count, 64
227 b.hi L(loop64_backwards)
228
229 /* Write the last iteration and copy 64 bytes from the start. */
230L(copy64_from_start):
231 ldp G_l, G_h, [src, 48]
232 stp A_l, A_h, [dstend, -16]
233 ldp A_l, A_h, [src, 32]
234 stp B_l, B_h, [dstend, -32]
235 ldp B_l, B_h, [src, 16]
236 stp C_l, C_h, [dstend, -48]
237 ldp C_l, C_h, [src]
238 stp D_l, D_h, [dstend, -64]
239 stp G_l, G_h, [dstin, 48]
240 stp A_l, A_h, [dstin, 32]
241 stp B_l, B_h, [dstin, 16]
242 stp C_l, C_h, [dstin]
243 ret
244
63SYM_FUNC_END_PI(memcpy)245SYM_FUNC_END_PI(memcpy)
64EXPORT_SYMBOL(memcpy)246EXPORT_SYMBOL(memcpy)
65SYM_FUNC_END_ALIAS(__memcpy)247SYM_FUNC_END_ALIAS(__memcpy)
66EXPORT_SYMBOL(__memcpy)248EXPORT_SYMBOL(__memcpy)
249SYM_FUNC_END_ALIAS_PI(memmove)
250EXPORT_SYMBOL(memmove)
251SYM_FUNC_END_ALIAS(__memmove)
252EXPORT_SYMBOL(__memmove)
arch/arm64/lib/memmove.S+0−189
@@ -1,189 +0,0 @@
1/* SPDX-License-Identifier: GPL-2.0-only */
2/*
3 * Copyright (C) 2013 ARM Ltd.
4 * Copyright (C) 2013 Linaro.
5 *
6 * This code is based on glibc cortex strings work originally authored by Linaro
7 * be found @
8 *
9 * http://bazaar.launchpad.net/~linaro-toolchain-dev/cortex-strings/trunk/
10 * files/head:/src/aarch64/
11 */
12
13#include <linux/linkage.h>
14#include <asm/assembler.h>
15#include <asm/cache.h>
16
17/*
18 * Move a buffer from src to test (alignment handled by the hardware).
19 * If dest <= src, call memcpy, otherwise copy in reverse order.
20 *
21 * Parameters:
22 * x0 - dest
23 * x1 - src
24 * x2 - n
25 * Returns:
26 * x0 - dest
27 */
28dstin .req x0
29src .req x1
30count .req x2
31tmp1 .req x3
32tmp1w .req w3
33tmp2 .req x4
34tmp2w .req w4
35tmp3 .req x5
36tmp3w .req w5
37dst .req x6
38
39A_l .req x7
40A_h .req x8
41B_l .req x9
42B_h .req x10
43C_l .req x11
44C_h .req x12
45D_l .req x13
46D_h .req x14
47
48SYM_FUNC_START_ALIAS(__memmove)
49SYM_FUNC_START_WEAK_PI(memmove)
50 cmp dstin, src
51 b.lo __memcpy
52 add tmp1, src, count
53 cmp dstin, tmp1
54 b.hs __memcpy /* No overlap. */
55
56 add dst, dstin, count
57 add src, src, count
58 cmp count, #16
59 b.lo .Ltail15 /*probably non-alignment accesses.*/
60
61 ands tmp2, src, #15 /* Bytes to reach alignment. */
62 b.eq .LSrcAligned
63 sub count, count, tmp2
64 /*
65 * process the aligned offset length to make the src aligned firstly.
66 * those extra instructions' cost is acceptable. It also make the
67 * coming accesses are based on aligned address.
68 */
69 tbz tmp2, #0, 1f
70 ldrb tmp1w, [src, #-1]!
71 strb tmp1w, [dst, #-1]!
721:
73 tbz tmp2, #1, 2f
74 ldrh tmp1w, [src, #-2]!
75 strh tmp1w, [dst, #-2]!
762:
77 tbz tmp2, #2, 3f
78 ldr tmp1w, [src, #-4]!
79 str tmp1w, [dst, #-4]!
803:
81 tbz tmp2, #3, .LSrcAligned
82 ldr tmp1, [src, #-8]!
83 str tmp1, [dst, #-8]!
84
85.LSrcAligned:
86 cmp count, #64
87 b.ge .Lcpy_over64
88
89 /*
90 * Deal with small copies quickly by dropping straight into the
91 * exit block.
92 */
93.Ltail63:
94 /*
95 * Copy up to 48 bytes of data. At this point we only need the
96 * bottom 6 bits of count to be accurate.
97 */
98 ands tmp1, count, #0x30
99 b.eq .Ltail15
100 cmp tmp1w, #0x20
101 b.eq 1f
102 b.lt 2f
103 ldp A_l, A_h, [src, #-16]!
104 stp A_l, A_h, [dst, #-16]!
1051:
106 ldp A_l, A_h, [src, #-16]!
107 stp A_l, A_h, [dst, #-16]!
1082:
109 ldp A_l, A_h, [src, #-16]!
110 stp A_l, A_h, [dst, #-16]!
111
112.Ltail15:
113 tbz count, #3, 1f
114 ldr tmp1, [src, #-8]!
115 str tmp1, [dst, #-8]!
1161:
117 tbz count, #2, 2f
118 ldr tmp1w, [src, #-4]!
119 str tmp1w, [dst, #-4]!
1202:
121 tbz count, #1, 3f
122 ldrh tmp1w, [src, #-2]!
123 strh tmp1w, [dst, #-2]!
1243:
125 tbz count, #0, .Lexitfunc
126 ldrb tmp1w, [src, #-1]
127 strb tmp1w, [dst, #-1]
128
129.Lexitfunc:
130 ret
131
132.Lcpy_over64:
133 subs count, count, #128
134 b.ge .Lcpy_body_large
135 /*
136 * Less than 128 bytes to copy, so handle 64 bytes here and then jump
137 * to the tail.
138 */
139 ldp A_l, A_h, [src, #-16]
140 stp A_l, A_h, [dst, #-16]
141 ldp B_l, B_h, [src, #-32]
142 ldp C_l, C_h, [src, #-48]
143 stp B_l, B_h, [dst, #-32]
144 stp C_l, C_h, [dst, #-48]
145 ldp D_l, D_h, [src, #-64]!
146 stp D_l, D_h, [dst, #-64]!
147
148 tst count, #0x3f
149 b.ne .Ltail63
150 ret
151
152 /*
153 * Critical loop. Start at a new cache line boundary. Assuming
154 * 64 bytes per line this ensures the entire loop is in one line.
155 */
156 .p2align L1_CACHE_SHIFT
157.Lcpy_body_large:
158 /* pre-load 64 bytes data. */
159 ldp A_l, A_h, [src, #-16]
160 ldp B_l, B_h, [src, #-32]
161 ldp C_l, C_h, [src, #-48]
162 ldp D_l, D_h, [src, #-64]!
1631:
164 /*
165 * interlace the load of next 64 bytes data block with store of the last
166 * loaded 64 bytes data.
167 */
168 stp A_l, A_h, [dst, #-16]
169 ldp A_l, A_h, [src, #-16]
170 stp B_l, B_h, [dst, #-32]
171 ldp B_l, B_h, [src, #-32]
172 stp C_l, C_h, [dst, #-48]
173 ldp C_l, C_h, [src, #-48]
174 stp D_l, D_h, [dst, #-64]!
175 ldp D_l, D_h, [src, #-64]!
176 subs count, count, #64
177 b.ge 1b
178 stp A_l, A_h, [dst, #-16]
179 stp B_l, B_h, [dst, #-32]
180 stp C_l, C_h, [dst, #-48]
181 stp D_l, D_h, [dst, #-64]!
182
183 tst count, #0x3f
184 b.ne .Ltail63
185 ret
186SYM_FUNC_END_PI(memmove)
187EXPORT_SYMBOL(memmove)
188SYM_FUNC_END_ALIAS(__memmove)
189EXPORT_SYMBOL(__memmove)
arch/arm64/lib/mte.S+20−0
@@ -36,6 +36,26 @@ SYM_FUNC_START(mte_clear_page_tags) · 35 unmodified lines
36 ret36 ret
37SYM_FUNC_END(mte_clear_page_tags)37SYM_FUNC_END(mte_clear_page_tags)
38 38
39/*
40 * Zero the page and tags at the same time
41 *
42 * Parameters:
43 * x0 - address to the beginning of the page
44 */
45SYM_FUNC_START(mte_zero_clear_page_tags)
46 mrs x1, dczid_el0
47 and w1, w1, #0xf
48 mov x2, #4
49 lsl x1, x2, x1
50 and x0, x0, #(1 << MTE_TAG_SHIFT) - 1 // clear the tag
51
521: dc gzva, x0
53 add x0, x0, x1
54 tst x0, #(PAGE_SIZE - 1)
55 b.ne 1b
56 ret
57SYM_FUNC_END(mte_zero_clear_page_tags)
58
39/*59/*
40 * Copy the tags from the source page to the destination one60 * Copy the tags from the source page to the destination one
41 * x0 - address of the destination page61 * x0 - address of the destination page
arch/arm64/lib/strcmp.S+121−168
@@ -1,84 +1,123 @@
1/* SPDX-License-Identifier: GPL-2.0-only */1/* SPDX-License-Identifier: GPL-2.0-only */
2/*2/*
3 * Copyright (C) 2013 ARM Ltd.3 * Copyright (c) 2012-2021, Arm Limited.
4 * Copyright (C) 2013 Linaro.
5 *4 *
6 * This code is based on glibc cortex strings work originally authored by Linaro5 * Adapted from the original at:
7 * be found @6 * https://github.com/ARM-software/optimized-routines/blob/afd6244a1f8d9229/string/aarch64/strcmp.S
8 *
9 * http://bazaar.launchpad.net/~linaro-toolchain-dev/cortex-strings/trunk/
10 * files/head:/src/aarch64/
11 */7 */
12 8
13#include <linux/linkage.h>9#include <linux/linkage.h>
14#include <asm/assembler.h>10#include <asm/assembler.h>
15 11
16/*12/* Assumptions:
17 * compare two strings
18 *13 *
19 * Parameters:14 * ARMv8-a, AArch64
20 * x0 - const string 1 pointer
21 * x1 - const string 2 pointer
22 * Returns:
23 * x0 - an integer less than, equal to, or greater than zero
24 * if s1 is found, respectively, to be less than, to match,
25 * or be greater than s2.
26 */15 */
27 16
17#define L(label) .L ## label
18
28#define REP8_01 0x010101010101010119#define REP8_01 0x0101010101010101
29#define REP8_7f 0x7f7f7f7f7f7f7f7f20#define REP8_7f 0x7f7f7f7f7f7f7f7f
30#define REP8_80 0x808080808080808021#define REP8_80 0x8080808080808080
31 22
32/* Parameters and result. */23/* Parameters and result. */
33src1 .req x024#define src1 x0
34src2 .req x125#define src2 x1
35result .req x026#define result x0
36 27
37/* Internal variables. */28/* Internal variables. */
38data1 .req x229#define data1 x2
39data1w .req w230#define data1w w2
40data2 .req x331#define data2 x3
41data2w .req w332#define data2w w3
42has_nul .req x433#define has_nul x4
43diff .req x534#define diff x5
44syndrome .req x635#define syndrome x6
45tmp1 .req x736#define tmp1 x7
46tmp2 .req x837#define tmp2 x8
47tmp3 .req x938#define tmp3 x9
48zeroones .req x1039#define zeroones x10
49pos .req x1140#define pos x11
50 41
42 /* Start of performance-critical section -- one 64B cache line. */
43 .align 6
51SYM_FUNC_START_WEAK_PI(strcmp)44SYM_FUNC_START_WEAK_PI(strcmp)
52 eor tmp1, src1, src245 eor tmp1, src1, src2
53 mov zeroones, #REP8_0146 mov zeroones, #REP8_01
54 tst tmp1, #747 tst tmp1, #7
55 b.ne .Lmisaligned848 b.ne L(misaligned8)
56 ands tmp1, src1, #749 ands tmp1, src1, #7
57 b.ne .Lmutual_align50 b.ne L(mutual_align)
58 51 /* NUL detection works on the principle that (X - 1) & (~X) & 0x80
59 /*52 (=> (X - 1) & ~(X | 0x7f)) is non-zero iff a byte is zero, and
60 * NUL detection works on the principle that (X - 1) & (~X) & 0x8053 can be done in parallel across the entire word. */
61 * (=> (X - 1) & ~(X | 0x7f)) is non-zero iff a byte is zero, and54L(loop_aligned):
62 * can be done in parallel across the entire word.
63 */
64.Lloop_aligned:
65 ldr data1, [src1], #855 ldr data1, [src1], #8
66 ldr data2, [src2], #856 ldr data2, [src2], #8
67.Lstart_realigned:57L(start_realigned):
68 sub tmp1, data1, zeroones58 sub tmp1, data1, zeroones
69 orr tmp2, data1, #REP8_7f59 orr tmp2, data1, #REP8_7f
70 eor diff, data1, data2 /* Non-zero if differences found. */60 eor diff, data1, data2 /* Non-zero if differences found. */
71 bic has_nul, tmp1, tmp2 /* Non-zero if NUL terminator. */61 bic has_nul, tmp1, tmp2 /* Non-zero if NUL terminator. */
72 orr syndrome, diff, has_nul62 orr syndrome, diff, has_nul
73 cbz syndrome, .Lloop_aligned63 cbz syndrome, L(loop_aligned)
74 b .Lcal_cmpresult64 /* End of performance-critical section -- one 64B cache line. */
65
66L(end):
67#ifndef __AARCH64EB__
68 rev syndrome, syndrome
69 rev data1, data1
70 /* The MS-non-zero bit of the syndrome marks either the first bit
71 that is different, or the top bit of the first zero byte.
72 Shifting left now will bring the critical information into the
73 top bits. */
74 clz pos, syndrome
75 rev data2, data2
76 lsl data1, data1, pos
77 lsl data2, data2, pos
78 /* But we need to zero-extend (char is unsigned) the value and then
79 perform a signed 32-bit subtraction. */
80 lsr data1, data1, #56
81 sub result, data1, data2, lsr #56
82 ret
83#else
84 /* For big-endian we cannot use the trick with the syndrome value
85 as carry-propagation can corrupt the upper bits if the trailing
86 bytes in the string contain 0x01. */
87 /* However, if there is no NUL byte in the dword, we can generate
88 the result directly. We can't just subtract the bytes as the
89 MSB might be significant. */
90 cbnz has_nul, 1f
91 cmp data1, data2
92 cset result, ne
93 cneg result, result, lo
94 ret
951:
96 /* Re-compute the NUL-byte detection, using a byte-reversed value. */
97 rev tmp3, data1
98 sub tmp1, tmp3, zeroones
99 orr tmp2, tmp3, #REP8_7f
100 bic has_nul, tmp1, tmp2
101 rev has_nul, has_nul
102 orr syndrome, diff, has_nul
103 clz pos, syndrome
104 /* The MS-non-zero bit of the syndrome marks either the first bit
105 that is different, or the top bit of the first zero byte.
106 Shifting left now will bring the critical information into the
107 top bits. */
108 lsl data1, data1, pos
109 lsl data2, data2, pos
110 /* But we need to zero-extend (char is unsigned) the value and then
111 perform a signed 32-bit subtraction. */
112 lsr data1, data1, #56
113 sub result, data1, data2, lsr #56
114 ret
115#endif
75 116
76.Lmutual_align:117L(mutual_align):
77 /*118 /* Sources are mutually aligned, but are not currently at an
78 * Sources are mutually aligned, but are not currently at an119 alignment boundary. Round down the addresses and then mask off
79 * alignment boundary. Round down the addresses and then mask off120 the bytes that preceed the start point. */
80 * the bytes that preceed the start point.
81 */
82 bic src1, src1, #7121 bic src1, src1, #7
83 bic src2, src2, #7122 bic src2, src2, #7
84 lsl tmp1, tmp1, #3 /* Bytes beyond alignment -> bits. */123 lsl tmp1, tmp1, #3 /* Bytes beyond alignment -> bits. */
@@ -86,138 +125,52 @@ SYM_FUNC_START_WEAK_PI(strcmp) · 1 unmodified line
86 neg tmp1, tmp1 /* Bits to alignment -64. */125 neg tmp1, tmp1 /* Bits to alignment -64. */
87 ldr data2, [src2], #8126 ldr data2, [src2], #8
88 mov tmp2, #~0127 mov tmp2, #~0
128#ifdef __AARCH64EB__
89 /* Big-endian. Early bytes are at MSB. */129 /* Big-endian. Early bytes are at MSB. */
90CPU_BE( lsl tmp2, tmp2, tmp1 ) /* Shift (tmp1 & 63). */130 lsl tmp2, tmp2, tmp1 /* Shift (tmp1 & 63). */
131#else
91 /* Little-endian. Early bytes are at LSB. */132 /* Little-endian. Early bytes are at LSB. */
92CPU_LE( lsr tmp2, tmp2, tmp1 ) /* Shift (tmp1 & 63). */133 lsr tmp2, tmp2, tmp1 /* Shift (tmp1 & 63). */
93 134#endif
94 orr data1, data1, tmp2135 orr data1, data1, tmp2
95 orr data2, data2, tmp2136 orr data2, data2, tmp2
96 b .Lstart_realigned137 b L(start_realigned)
97 138
98.Lmisaligned8:139L(misaligned8):
99 /*140 /* Align SRC1 to 8 bytes and then compare 8 bytes at a time, always
100 * Get the align offset length to compare per byte first.141 checking to make sure that we don't access beyond page boundary in
101 * After this process, one string's address will be aligned.142 SRC2. */
102 */143 tst src1, #7
103 and tmp1, src1, #7144 b.eq L(loop_misaligned)
104 neg tmp1, tmp1145L(do_misaligned):
105 add tmp1, tmp1, #8
106 and tmp2, src2, #7
107 neg tmp2, tmp2
108 add tmp2, tmp2, #8
109 subs tmp3, tmp1, tmp2
110 csel pos, tmp1, tmp2, hi /*Choose the maximum. */
111.Ltinycmp:
112 ldrb data1w, [src1], #1146 ldrb data1w, [src1], #1
113 ldrb data2w, [src2], #1147 ldrb data2w, [src2], #1
114 subs pos, pos, #1
115 ccmp data1w, #1, #0, ne /* NZCV = 0b0000. */
116 ccmp data1w, data2w, #0, cs /* NZCV = 0b0000. */
117 b.eq .Ltinycmp
118 cbnz pos, 1f /*find the null or unequal...*/
119 cmp data1w, #1148 cmp data1w, #1
120 ccmp data1w, data2w, #0, cs149 ccmp data1w, data2w, #0, cs /* NZCV = 0b0000. */
121 b.eq .Lstart_align /*the last bytes are equal....*/150 b.ne L(done)
1221:151 tst src1, #7
123 sub result, data1, data2152 b.ne L(do_misaligned)
124 ret153
125 154L(loop_misaligned):
126.Lstart_align:155 /* Test if we are within the last dword of the end of a 4K page. If
127 ands xzr, src1, #7156 yes then jump back to the misaligned loop to copy a byte at a time. */
128 b.eq .Lrecal_offset157 and tmp1, src2, #0xff8
129 /*process more leading bytes to make str1 aligned...*/158 eor tmp1, tmp1, #0xff8
130 add src1, src1, tmp3159 cbz tmp1, L(do_misaligned)
131 add src2, src2, tmp3
132 /*load 8 bytes from aligned str1 and non-aligned str2..*/
133 ldr data1, [src1], #8160 ldr data1, [src1], #8
134 ldr data2, [src2], #8161 ldr data2, [src2], #8
135 162
136 sub tmp1, data1, zeroones163 sub tmp1, data1, zeroones
137 orr tmp2, data1, #REP8_7f164 orr tmp2, data1, #REP8_7f
138 bic has_nul, tmp1, tmp2165 eor diff, data1, data2 /* Non-zero if differences found. */
139 eor diff, data1, data2 /* Non-zero if differences found. */166 bic has_nul, tmp1, tmp2 /* Non-zero if NUL terminator. */
140 orr syndrome, diff, has_nul
141 cbnz syndrome, .Lcal_cmpresult
142 /*How far is the current str2 from the alignment boundary...*/
143 and tmp3, tmp3, #7
144.Lrecal_offset:
145 neg pos, tmp3
146.Lloopcmp_proc:
147 /*
148 * Divide the eight bytes into two parts. First,backwards the src2
149 * to an alignment boundary,load eight bytes from the SRC2 alignment
150 * boundary,then compare with the relative bytes from SRC1.
151 * If all 8 bytes are equal,then start the second part's comparison.
152 * Otherwise finish the comparison.
153 * This special handle can garantee all the accesses are in the
154 * thread/task space in avoid to overrange access.
155 */
156 ldr data1, [src1,pos]
157 ldr data2, [src2,pos]
158 sub tmp1, data1, zeroones
159 orr tmp2, data1, #REP8_7f
160 bic has_nul, tmp1, tmp2
161 eor diff, data1, data2 /* Non-zero if differences found. */
162 orr syndrome, diff, has_nul
163 cbnz syndrome, .Lcal_cmpresult
164
165 /*The second part process*/
166 ldr data1, [src1], #8
167 ldr data2, [src2], #8
168 sub tmp1, data1, zeroones
169 orr tmp2, data1, #REP8_7f
170 bic has_nul, tmp1, tmp2
171 eor diff, data1, data2 /* Non-zero if differences found. */
172 orr syndrome, diff, has_nul167 orr syndrome, diff, has_nul
173 cbz syndrome, .Lloopcmp_proc168 cbz syndrome, L(loop_misaligned)
169 b L(end)
174 170
175.Lcal_cmpresult:171L(done):
176 /*172 sub result, data1, data2
177 * reversed the byte-order as big-endian,then CLZ can find the most
178 * significant zero bits.
179 */
180CPU_LE( rev syndrome, syndrome )
181CPU_LE( rev data1, data1 )
182CPU_LE( rev data2, data2 )
183
184 /*
185 * For big-endian we cannot use the trick with the syndrome value
186 * as carry-propagation can corrupt the upper bits if the trailing
187 * bytes in the string contain 0x01.
188 * However, if there is no NUL byte in the dword, we can generate
189 * the result directly. We cannot just subtract the bytes as the
190 * MSB might be significant.
191 */
192CPU_BE( cbnz has_nul, 1f )
193CPU_BE( cmp data1, data2 )
194CPU_BE( cset result, ne )
195CPU_BE( cneg result, result, lo )
196CPU_BE( ret )
197CPU_BE( 1: )
198 /*Re-compute the NUL-byte detection, using a byte-reversed value. */
199CPU_BE( rev tmp3, data1 )
200CPU_BE( sub tmp1, tmp3, zeroones )
201CPU_BE( orr tmp2, tmp3, #REP8_7f )
202CPU_BE( bic has_nul, tmp1, tmp2 )
203CPU_BE( rev has_nul, has_nul )
204CPU_BE( orr syndrome, diff, has_nul )
205
206 clz pos, syndrome
207 /*
208 * The MS-non-zero bit of the syndrome marks either the first bit
209 * that is different, or the top bit of the first zero byte.
210 * Shifting left now will bring the critical information into the
211 * top bits.
212 */
213 lsl data1, data1, pos
214 lsl data2, data2, pos
215 /*
216 * But we need to zero-extend (char is unsigned) the value and then
217 * perform a signed 32-bit subtraction.
218 */
219 lsr data1, data1, #56
220 sub result, data1, data2, lsr #56
221 ret173 ret
174
222SYM_FUNC_END_PI(strcmp)175SYM_FUNC_END_PI(strcmp)
223EXPORT_SYMBOL_NOKASAN(strcmp)176EXPORT_SYMBOL_NOKASAN(strcmp)
arch/arm64/lib/strlen.S+173−85
@@ -1,115 +1,203 @@
1/* SPDX-License-Identifier: GPL-2.0-only */1/* SPDX-License-Identifier: GPL-2.0-only */
2/*2/*
3 * Copyright (C) 2013 ARM Ltd.3 * Copyright (c) 2013-2021, Arm Limited.
4 * Copyright (C) 2013 Linaro.
5 *4 *
6 * This code is based on glibc cortex strings work originally authored by Linaro5 * Adapted from the original at:
7 * be found @6 * https://github.com/ARM-software/optimized-routines/blob/98e4d6a5c13c8e54/string/aarch64/strlen.S
8 *
9 * http://bazaar.launchpad.net/~linaro-toolchain-dev/cortex-strings/trunk/
10 * files/head:/src/aarch64/
11 */7 */
12 8
13#include <linux/linkage.h>9#include <linux/linkage.h>
14#include <asm/assembler.h>10#include <asm/assembler.h>
15 11
16/*12/* Assumptions:
17 * calculate the length of a string
18 *13 *
19 * Parameters:14 * ARMv8-a, AArch64, unaligned accesses, min page size 4k.
20 * x0 - const string pointer
21 * Returns:
22 * x0 - the return length of specific string
23 */15 */
24 16
17#define L(label) .L ## label
18
25/* Arguments and results. */19/* Arguments and results. */
26srcin .req x020#define srcin x0
27len .req x021#define len x0
28 22
29/* Locals and temporaries. */23/* Locals and temporaries. */
30src .req x124#define src x1
31data1 .req x225#define data1 x2
32data2 .req x326#define data2 x3
33data2a .req x427#define has_nul1 x4
34has_nul1 .req x528#define has_nul2 x5
35has_nul2 .req x629#define tmp1 x4
36tmp1 .req x730#define tmp2 x5
37tmp2 .req x831#define tmp3 x6
38tmp3 .req x932#define tmp4 x7
39tmp4 .req x1033#define zeroones x8
40zeroones .req x1134
41pos .req x1235 /* NUL detection works on the principle that (X - 1) & (~X) & 0x80
36 (=> (X - 1) & ~(X | 0x7f)) is non-zero iff a byte is zero, and
37 can be done in parallel across the entire word. A faster check
38 (X - 1) & 0x80 is zero for non-NUL ASCII characters, but gives
39 false hits for characters 129..255. */
42 40
43#define REP8_01 0x010101010101010141#define REP8_01 0x0101010101010101
44#define REP8_7f 0x7f7f7f7f7f7f7f7f42#define REP8_7f 0x7f7f7f7f7f7f7f7f
45#define REP8_80 0x808080808080808043#define REP8_80 0x8080808080808080
46 44
45#define MIN_PAGE_SIZE 4096
46
47 /* Since strings are short on average, we check the first 16 bytes
48 of the string for a NUL character. In order to do an unaligned ldp
49 safely we have to do a page cross check first. If there is a NUL
50 byte we calculate the length from the 2 8-byte words using
51 conditional select to reduce branch mispredictions (it is unlikely
52 strlen will be repeatedly called on strings with the same length).
53
54 If the string is longer than 16 bytes, we align src so don't need
55 further page cross checks, and process 32 bytes per iteration
56 using the fast NUL check. If we encounter non-ASCII characters,
57 fallback to a second loop using the full NUL check.
58
59 If the page cross check fails, we read 16 bytes from an aligned
60 address, remove any characters before the string, and continue
61 in the main loop using aligned loads. Since strings crossing a
62 page in the first 16 bytes are rare (probability of
63 16/MIN_PAGE_SIZE ~= 0.4%), this case does not need to be optimized.
64
65 AArch64 systems have a minimum page size of 4k. We don't bother
66 checking for larger page sizes - the cost of setting up the correct
67 page size is just not worth the extra gain from a small reduction in
68 the cases taking the slow path. Note that we only care about
69 whether the first fetch, which may be misaligned, crosses a page
70 boundary. */
71
47SYM_FUNC_START_WEAK_PI(strlen)72SYM_FUNC_START_WEAK_PI(strlen)
48 mov zeroones, #REP8_0173 and tmp1, srcin, MIN_PAGE_SIZE - 1
49 bic src, srcin, #1574 mov zeroones, REP8_01
50 ands tmp1, srcin, #1575 cmp tmp1, MIN_PAGE_SIZE - 16
51 b.ne .Lmisaligned76 b.gt L(page_cross)
52 /*77 ldp data1, data2, [srcin]
53 * NUL detection works on the principle that (X - 1) & (~X) & 0x8078#ifdef __AARCH64EB__
54 * (=> (X - 1) & ~(X | 0x7f)) is non-zero iff a byte is zero, and79 /* For big-endian, carry propagation (if the final byte in the
55 * can be done in parallel across the entire word.80 string is 0x01) means we cannot use has_nul1/2 directly.
56 */81 Since we expect strings to be small and early-exit,
57 /*82 byte-swap the data now so has_null1/2 will be correct. */
58 * The inner loop deals with two Dwords at a time. This has a83 rev data1, data1
59 * slightly higher start-up cost, but we should win quite quickly,84 rev data2, data2
60 * especially on cores with a high number of issue slots per85#endif
61 * cycle, as we get much better parallelism out of the operations.
62 */
63.Lloop:
64 ldp data1, data2, [src], #16
65.Lrealigned:
66 sub tmp1, data1, zeroones86 sub tmp1, data1, zeroones
67 orr tmp2, data1, #REP8_7f87 orr tmp2, data1, REP8_7f
68 sub tmp3, data2, zeroones88 sub tmp3, data2, zeroones
69 orr tmp4, data2, #REP8_7f89 orr tmp4, data2, REP8_7f
70 bic has_nul1, tmp1, tmp290 bics has_nul1, tmp1, tmp2
71 bics has_nul2, tmp3, tmp491 bic has_nul2, tmp3, tmp4
72 ccmp has_nul1, #0, #0, eq /* NZCV = 0000 */92 ccmp has_nul2, 0, 0, eq
73 b.eq .Lloop93 beq L(main_loop_entry)
94
95 /* Enter with C = has_nul1 == 0. */
96 csel has_nul1, has_nul1, has_nul2, cc
97 mov len, 8
98 rev has_nul1, has_nul1
99 clz tmp1, has_nul1
100 csel len, xzr, len, cc
101 add len, len, tmp1, lsr 3
102 ret
74 103
104 /* The inner loop processes 32 bytes per iteration and uses the fast
105 NUL check. If we encounter non-ASCII characters, use a second
106 loop with the accurate NUL check. */
107 .p2align 4
108L(main_loop_entry):
109 bic src, srcin, 15
110 sub src, src, 16
111L(main_loop):
112 ldp data1, data2, [src, 32]!
113L(page_cross_entry):
114 sub tmp1, data1, zeroones
115 sub tmp3, data2, zeroones
116 orr tmp2, tmp1, tmp3
117 tst tmp2, zeroones, lsl 7
118 bne 1f
119 ldp data1, data2, [src, 16]
120 sub tmp1, data1, zeroones
121 sub tmp3, data2, zeroones
122 orr tmp2, tmp1, tmp3
123 tst tmp2, zeroones, lsl 7
124 beq L(main_loop)
125 add src, src, 16
1261:
127 /* The fast check failed, so do the slower, accurate NUL check. */
128 orr tmp2, data1, REP8_7f
129 orr tmp4, data2, REP8_7f
130 bics has_nul1, tmp1, tmp2
131 bic has_nul2, tmp3, tmp4
132 ccmp has_nul2, 0, 0, eq
133 beq L(nonascii_loop)
134
135 /* Enter with C = has_nul1 == 0. */
136L(tail):
137#ifdef __AARCH64EB__
138 /* For big-endian, carry propagation (if the final byte in the
139 string is 0x01) means we cannot use has_nul1/2 directly. The
140 easiest way to get the correct byte is to byte-swap the data
141 and calculate the syndrome a second time. */
142 csel data1, data1, data2, cc
143 rev data1, data1
144 sub tmp1, data1, zeroones
145 orr tmp2, data1, REP8_7f
146 bic has_nul1, tmp1, tmp2
147#else
148 csel has_nul1, has_nul1, has_nul2, cc
149#endif
75 sub len, src, srcin150 sub len, src, srcin
76 cbz has_nul1, .Lnul_in_data2151 rev has_nul1, has_nul1
77CPU_BE( mov data2, data1 ) /*prepare data to re-calculate the syndrome*/152 add tmp2, len, 8
78 sub len, len, #8153 clz tmp1, has_nul1
79 mov has_nul2, has_nul1154 csel len, len, tmp2, cc
80.Lnul_in_data2:155 add len, len, tmp1, lsr 3
81 /*
82 * For big-endian, carry propagation (if the final byte in the
83 * string is 0x01) means we cannot use has_nul directly. The
84 * easiest way to get the correct byte is to byte-swap the data
85 * and calculate the syndrome a second time.
86 */
87CPU_BE( rev data2, data2 )
88CPU_BE( sub tmp1, data2, zeroones )
89CPU_BE( orr tmp2, data2, #REP8_7f )
90CPU_BE( bic has_nul2, tmp1, tmp2 )
91
92 sub len, len, #8
93 rev has_nul2, has_nul2
94 clz pos, has_nul2
95 add len, len, pos, lsr #3 /* Bits to bytes. */
96 ret156 ret
97 157
98.Lmisaligned:158L(nonascii_loop):
99 cmp tmp1, #8159 ldp data1, data2, [src, 16]!
100 neg tmp1, tmp1160 sub tmp1, data1, zeroones
101 ldp data1, data2, [src], #16161 orr tmp2, data1, REP8_7f
102 lsl tmp1, tmp1, #3 /* Bytes beyond alignment -> bits. */162 sub tmp3, data2, zeroones
103 mov tmp2, #~0163 orr tmp4, data2, REP8_7f
104 /* Big-endian. Early bytes are at MSB. */164 bics has_nul1, tmp1, tmp2
105CPU_BE( lsl tmp2, tmp2, tmp1 ) /* Shift (tmp1 & 63). */165 bic has_nul2, tmp3, tmp4
166 ccmp has_nul2, 0, 0, eq
167 bne L(tail)
168 ldp data1, data2, [src, 16]!
169 sub tmp1, data1, zeroones
170 orr tmp2, data1, REP8_7f
171 sub tmp3, data2, zeroones
172 orr tmp4, data2, REP8_7f
173 bics has_nul1, tmp1, tmp2
174 bic has_nul2, tmp3, tmp4
175 ccmp has_nul2, 0, 0, eq
176 beq L(nonascii_loop)
177 b L(tail)
178
179 /* Load 16 bytes from [srcin & ~15] and force the bytes that precede
180 srcin to 0x7f, so we ignore any NUL bytes before the string.
181 Then continue in the aligned loop. */
182L(page_cross):
183 bic src, srcin, 15
184 ldp data1, data2, [src]
185 lsl tmp1, srcin, 3
186 mov tmp4, -1
187#ifdef __AARCH64EB__
188 /* Big-endian. Early bytes are at MSB. */
189 lsr tmp1, tmp4, tmp1 /* Shift (tmp1 & 63). */
190#else
106 /* Little-endian. Early bytes are at LSB. */191 /* Little-endian. Early bytes are at LSB. */
107CPU_LE( lsr tmp2, tmp2, tmp1 ) /* Shift (tmp1 & 63). */192 lsl tmp1, tmp4, tmp1 /* Shift (tmp1 & 63). */
193#endif
194 orr tmp1, tmp1, REP8_80
195 orn data1, data1, tmp1
196 orn tmp2, data2, tmp1
197 tst srcin, 8
198 csel data1, data1, tmp4, eq
199 csel data2, data2, tmp2, eq
200 b L(page_cross_entry)
108 201
109 orr data1, data1, tmp2
110 orr data2a, data2, tmp2
111 csinv data1, data1, xzr, le
112 csel data2, data2, data2a, le
113 b .Lrealigned
114SYM_FUNC_END_PI(strlen)202SYM_FUNC_END_PI(strlen)
115EXPORT_SYMBOL_NOKASAN(strlen)203EXPORT_SYMBOL_NOKASAN(strlen)
arch/arm64/lib/strncmp.S+184−222
@@ -1,299 +1,261 @@
1/* SPDX-License-Identifier: GPL-2.0-only */1/* SPDX-License-Identifier: GPL-2.0-only */
2/*2/*
3 * Copyright (C) 2013 ARM Ltd.3 * Copyright (c) 2013-2021, Arm Limited.
4 * Copyright (C) 2013 Linaro.
5 *4 *
6 * This code is based on glibc cortex strings work originally authored by Linaro5 * Adapted from the original at:
7 * be found @6 * https://github.com/ARM-software/optimized-routines/blob/e823e3abf5f89ecb/string/aarch64/strncmp.S
8 *
9 * http://bazaar.launchpad.net/~linaro-toolchain-dev/cortex-strings/trunk/
10 * files/head:/src/aarch64/
11 */7 */
12 8
13#include <linux/linkage.h>9#include <linux/linkage.h>
14#include <asm/assembler.h>10#include <asm/assembler.h>
15 11
16/*12/* Assumptions:
17 * compare two strings
18 *13 *
19 * Parameters:14 * ARMv8-a, AArch64
20 * x0 - const string 1 pointer
21 * x1 - const string 2 pointer
22 * x2 - the maximal length to be compared
23 * Returns:
24 * x0 - an integer less than, equal to, or greater than zero if s1 is found,
25 * respectively, to be less than, to match, or be greater than s2.
26 */15 */
27 16
17#define L(label) .L ## label
18
28#define REP8_01 0x010101010101010119#define REP8_01 0x0101010101010101
29#define REP8_7f 0x7f7f7f7f7f7f7f7f20#define REP8_7f 0x7f7f7f7f7f7f7f7f
30#define REP8_80 0x808080808080808021#define REP8_80 0x8080808080808080
31 22
32/* Parameters and result. */23/* Parameters and result. */
33src1 .req x024#define src1 x0
34src2 .req x125#define src2 x1
35limit .req x226#define limit x2
36result .req x027#define result x0
37 28
38/* Internal variables. */29/* Internal variables. */
39data1 .req x330#define data1 x3
40data1w .req w331#define data1w w3
41data2 .req x432#define data2 x4
42data2w .req w433#define data2w w4
43has_nul .req x534#define has_nul x5
44diff .req x635#define diff x6
45syndrome .req x736#define syndrome x7
46tmp1 .req x837#define tmp1 x8
47tmp2 .req x938#define tmp2 x9
48tmp3 .req x1039#define tmp3 x10
49zeroones .req x1140#define zeroones x11
50pos .req x1241#define pos x12
51limit_wd .req x1342#define limit_wd x13
52mask .req x1443#define mask x14
53endloop .req x1544#define endloop x15
45#define count mask
54 46
55SYM_FUNC_START_WEAK_PI(strncmp)47SYM_FUNC_START_WEAK_PI(strncmp)
56 cbz limit, .Lret048 cbz limit, L(ret0)
57 eor tmp1, src1, src249 eor tmp1, src1, src2
58 mov zeroones, #REP8_0150 mov zeroones, #REP8_01
59 tst tmp1, #751 tst tmp1, #7
60 b.ne .Lmisaligned852 and count, src1, #7
61 ands tmp1, src1, #753 b.ne L(misaligned8)
62 b.ne .Lmutual_align54 cbnz count, L(mutual_align)
63 /* Calculate the number of full and partial words -1. */55 /* Calculate the number of full and partial words -1. */
64 /*56 sub limit_wd, limit, #1 /* limit != 0, so no underflow. */
65 * when limit is mulitply of 8, if not sub 1,57 lsr limit_wd, limit_wd, #3 /* Convert to Dwords. */
66 * the judgement of last dword will wrong.
67 */
68 sub limit_wd, limit, #1 /* limit != 0, so no underflow. */
69 lsr limit_wd, limit_wd, #3 /* Convert to Dwords. */
70 58
71 /*59 /* NUL detection works on the principle that (X - 1) & (~X) & 0x80
72 * NUL detection works on the principle that (X - 1) & (~X) & 0x8060 (=> (X - 1) & ~(X | 0x7f)) is non-zero iff a byte is zero, and
73 * (=> (X - 1) & ~(X | 0x7f)) is non-zero iff a byte is zero, and61 can be done in parallel across the entire word. */
74 * can be done in parallel across the entire word.62 .p2align 4
75 */63L(loop_aligned):
76.Lloop_aligned:
77 ldr data1, [src1], #864 ldr data1, [src1], #8
78 ldr data2, [src2], #865 ldr data2, [src2], #8
79.Lstart_realigned:66L(start_realigned):
80 subs limit_wd, limit_wd, #167 subs limit_wd, limit_wd, #1
81 sub tmp1, data1, zeroones68 sub tmp1, data1, zeroones
82 orr tmp2, data1, #REP8_7f69 orr tmp2, data1, #REP8_7f
83 eor diff, data1, data2 /* Non-zero if differences found. */70 eor diff, data1, data2 /* Non-zero if differences found. */
84 csinv endloop, diff, xzr, pl /* Last Dword or differences.*/71 csinv endloop, diff, xzr, pl /* Last Dword or differences. */
85 bics has_nul, tmp1, tmp2 /* Non-zero if NUL terminator. */72 bics has_nul, tmp1, tmp2 /* Non-zero if NUL terminator. */
86 ccmp endloop, #0, #0, eq73 ccmp endloop, #0, #0, eq
87 b.eq .Lloop_aligned74 b.eq L(loop_aligned)
75 /* End of main loop */
88 76
89 /*Not reached the limit, must have found the end or a diff. */77 /* Not reached the limit, must have found the end or a diff. */
90 tbz limit_wd, #63, .Lnot_limit78 tbz limit_wd, #63, L(not_limit)
91 79
92 /* Limit % 8 == 0 => all bytes significant. */80 /* Limit % 8 == 0 => all bytes significant. */
93 ands limit, limit, #781 ands limit, limit, #7
94 b.eq .Lnot_limit82 b.eq L(not_limit)
95 83
96 lsl limit, limit, #3 /* Bits -> bytes. */84 lsl limit, limit, #3 /* Bits -> bytes. */
97 mov mask, #~085 mov mask, #~0
98CPU_BE( lsr mask, mask, limit )86#ifdef __AARCH64EB__
99CPU_LE( lsl mask, mask, limit )87 lsr mask, mask, limit
88#else
89 lsl mask, mask, limit
90#endif
100 bic data1, data1, mask91 bic data1, data1, mask
101 bic data2, data2, mask92 bic data2, data2, mask
102 93
103 /* Make sure that the NUL byte is marked in the syndrome. */94 /* Make sure that the NUL byte is marked in the syndrome. */
104 orr has_nul, has_nul, mask95 orr has_nul, has_nul, mask
105 96
106.Lnot_limit:97L(not_limit):
107 orr syndrome, diff, has_nul98 orr syndrome, diff, has_nul
108 b .Lcal_cmpresult
109 99
110.Lmutual_align:100#ifndef __AARCH64EB__
111 /*101 rev syndrome, syndrome
112 * Sources are mutually aligned, but are not currently at an102 rev data1, data1
113 * alignment boundary. Round down the addresses and then mask off103 /* The MS-non-zero bit of the syndrome marks either the first bit
114 * the bytes that precede the start point.104 that is different, or the top bit of the first zero byte.
115 * We also need to adjust the limit calculations, but without105 Shifting left now will bring the critical information into the
116 * overflowing if the limit is near ULONG_MAX.106 top bits. */
117 */107 clz pos, syndrome
108 rev data2, data2
109 lsl data1, data1, pos
110 lsl data2, data2, pos
111 /* But we need to zero-extend (char is unsigned) the value and then
112 perform a signed 32-bit subtraction. */
113 lsr data1, data1, #56
114 sub result, data1, data2, lsr #56
115 ret
116#else
117 /* For big-endian we cannot use the trick with the syndrome value
118 as carry-propagation can corrupt the upper bits if the trailing
119 bytes in the string contain 0x01. */
120 /* However, if there is no NUL byte in the dword, we can generate
121 the result directly. We can't just subtract the bytes as the
122 MSB might be significant. */
123 cbnz has_nul, 1f
124 cmp data1, data2
125 cset result, ne
126 cneg result, result, lo
127 ret
1281:
129 /* Re-compute the NUL-byte detection, using a byte-reversed value. */
130 rev tmp3, data1
131 sub tmp1, tmp3, zeroones
132 orr tmp2, tmp3, #REP8_7f
133 bic has_nul, tmp1, tmp2
134 rev has_nul, has_nul
135 orr syndrome, diff, has_nul
136 clz pos, syndrome
137 /* The MS-non-zero bit of the syndrome marks either the first bit
138 that is different, or the top bit of the first zero byte.
139 Shifting left now will bring the critical information into the
140 top bits. */
141 lsl data1, data1, pos
142 lsl data2, data2, pos
143 /* But we need to zero-extend (char is unsigned) the value and then
144 perform a signed 32-bit subtraction. */
145 lsr data1, data1, #56
146 sub result, data1, data2, lsr #56
147 ret
148#endif
149
150L(mutual_align):
151 /* Sources are mutually aligned, but are not currently at an
152 alignment boundary. Round down the addresses and then mask off
153 the bytes that precede the start point.
154 We also need to adjust the limit calculations, but without
155 overflowing if the limit is near ULONG_MAX. */
118 bic src1, src1, #7156 bic src1, src1, #7
119 bic src2, src2, #7157 bic src2, src2, #7
120 ldr data1, [src1], #8158 ldr data1, [src1], #8
121 neg tmp3, tmp1, lsl #3 /* 64 - bits(bytes beyond align). */159 neg tmp3, count, lsl #3 /* 64 - bits(bytes beyond align). */
122 ldr data2, [src2], #8160 ldr data2, [src2], #8
123 mov tmp2, #~0161 mov tmp2, #~0
124 sub limit_wd, limit, #1 /* limit != 0, so no underflow. */162 sub limit_wd, limit, #1 /* limit != 0, so no underflow. */
163#ifdef __AARCH64EB__
125 /* Big-endian. Early bytes are at MSB. */164 /* Big-endian. Early bytes are at MSB. */
126CPU_BE( lsl tmp2, tmp2, tmp3 ) /* Shift (tmp1 & 63). */165 lsl tmp2, tmp2, tmp3 /* Shift (count & 63). */
166#else
127 /* Little-endian. Early bytes are at LSB. */167 /* Little-endian. Early bytes are at LSB. */
128CPU_LE( lsr tmp2, tmp2, tmp3 ) /* Shift (tmp1 & 63). */168 lsr tmp2, tmp2, tmp3 /* Shift (count & 63). */
129 169#endif
130 and tmp3, limit_wd, #7170 and tmp3, limit_wd, #7
131 lsr limit_wd, limit_wd, #3171 lsr limit_wd, limit_wd, #3
132 /* Adjust the limit. Only low 3 bits used, so overflow irrelevant.*/172 /* Adjust the limit. Only low 3 bits used, so overflow irrelevant. */
133 add limit, limit, tmp1173 add limit, limit, count
134 add tmp3, tmp3, tmp1174 add tmp3, tmp3, count
135 orr data1, data1, tmp2175 orr data1, data1, tmp2
136 orr data2, data2, tmp2176 orr data2, data2, tmp2
137 add limit_wd, limit_wd, tmp3, lsr #3177 add limit_wd, limit_wd, tmp3, lsr #3
138 b .Lstart_realigned178 b L(start_realigned)
179
180 .p2align 4
181 /* Don't bother with dwords for up to 16 bytes. */
182L(misaligned8):
183 cmp limit, #16
184 b.hs L(try_misaligned_words)
139 185
140/*when src1 offset is not equal to src2 offset...*/186L(byte_loop):
141.Lmisaligned8:187 /* Perhaps we can do better than this. */
142 cmp limit, #8
143 b.lo .Ltiny8proc /*limit < 8... */
144 /*
145 * Get the align offset length to compare per byte first.
146 * After this process, one string's address will be aligned.*/
147 and tmp1, src1, #7
148 neg tmp1, tmp1
149 add tmp1, tmp1, #8
150 and tmp2, src2, #7
151 neg tmp2, tmp2
152 add tmp2, tmp2, #8
153 subs tmp3, tmp1, tmp2
154 csel pos, tmp1, tmp2, hi /*Choose the maximum. */
155 /*
156 * Here, limit is not less than 8, so directly run .Ltinycmp
157 * without checking the limit.*/
158 sub limit, limit, pos
159.Ltinycmp:
160 ldrb data1w, [src1], #1188 ldrb data1w, [src1], #1
161 ldrb data2w, [src2], #1189 ldrb data2w, [src2], #1
162 subs pos, pos, #1190 subs limit, limit, #1
163 ccmp data1w, #1, #0, ne /* NZCV = 0b0000. */191 ccmp data1w, #1, #0, hi /* NZCV = 0b0000. */
164 ccmp data1w, data2w, #0, cs /* NZCV = 0b0000. */192 ccmp data1w, data2w, #0, cs /* NZCV = 0b0000. */
165 b.eq .Ltinycmp193 b.eq L(byte_loop)
166 cbnz pos, 1f /*find the null or unequal...*/194L(done):
167 cmp data1w, #1
168 ccmp data1w, data2w, #0, cs
169 b.eq .Lstart_align /*the last bytes are equal....*/
1701:
171 sub result, data1, data2195 sub result, data1, data2
172 ret196 ret
173 197 /* Align the SRC1 to a dword by doing a bytewise compare and then do
174.Lstart_align:198 the dword loop. */
199L(try_misaligned_words):
175 lsr limit_wd, limit, #3200 lsr limit_wd, limit, #3
176 cbz limit_wd, .Lremain8201 cbz count, L(do_misaligned)
177 /*process more leading bytes to make str1 aligned...*/
178 ands xzr, src1, #7
179 b.eq .Lrecal_offset
180 add src1, src1, tmp3 /*tmp3 is positive in this branch.*/
181 add src2, src2, tmp3
182 ldr data1, [src1], #8
183 ldr data2, [src2], #8
184 202
185 sub limit, limit, tmp3203 neg count, count
204 and count, count, #7
205 sub limit, limit, count
186 lsr limit_wd, limit, #3206 lsr limit_wd, limit, #3
187 subs limit_wd, limit_wd, #1
188 207
189 sub tmp1, data1, zeroones208L(page_end_loop):
190 orr tmp2, data1, #REP8_7f209 ldrb data1w, [src1], #1
191 eor diff, data1, data2 /* Non-zero if differences found. */210 ldrb data2w, [src2], #1
192 csinv endloop, diff, xzr, ne/*if limit_wd is 0,will finish the cmp*/211 cmp data1w, #1
193 bics has_nul, tmp1, tmp2212 ccmp data1w, data2w, #0, cs /* NZCV = 0b0000. */
194 ccmp endloop, #0, #0, eq /*has_null is ZERO: no null byte*/213 b.ne L(done)
195 b.ne .Lunequal_proc214 subs count, count, #1
196 /*How far is the current str2 from the alignment boundary...*/215 b.hi L(page_end_loop)
197 and tmp3, tmp3, #7216
198.Lrecal_offset:217L(do_misaligned):
199 neg pos, tmp3218 /* Prepare ourselves for the next page crossing. Unlike the aligned
200.Lloopcmp_proc:219 loop, we fetch 1 less dword because we risk crossing bounds on
201 /*220 SRC2. */
202 * Divide the eight bytes into two parts. First,backwards the src2221 mov count, #8
203 * to an alignment boundary,load eight bytes from the SRC2 alignment222 subs limit_wd, limit_wd, #1
204 * boundary,then compare with the relative bytes from SRC1.223 b.lo L(done_loop)
205 * If all 8 bytes are equal,then start the second part's comparison.224L(loop_misaligned):
206 * Otherwise finish the comparison.225 and tmp2, src2, #0xff8
207 * This special handle can garantee all the accesses are in the226 eor tmp2, tmp2, #0xff8
208 * thread/task space in avoid to overrange access.227 cbz tmp2, L(page_end_loop)
209 */
210 ldr data1, [src1,pos]
211 ldr data2, [src2,pos]
212 sub tmp1, data1, zeroones
213 orr tmp2, data1, #REP8_7f
214 bics has_nul, tmp1, tmp2 /* Non-zero if NUL terminator. */
215 eor diff, data1, data2 /* Non-zero if differences found. */
216 csinv endloop, diff, xzr, eq
217 cbnz endloop, .Lunequal_proc
218 228
219 /*The second part process*/
220 ldr data1, [src1], #8229 ldr data1, [src1], #8
221 ldr data2, [src2], #8230 ldr data2, [src2], #8
222 subs limit_wd, limit_wd, #1
223 sub tmp1, data1, zeroones231 sub tmp1, data1, zeroones
224 orr tmp2, data1, #REP8_7f232 orr tmp2, data1, #REP8_7f
225 eor diff, data1, data2 /* Non-zero if differences found. */233 eor diff, data1, data2 /* Non-zero if differences found. */
226 csinv endloop, diff, xzr, ne/*if limit_wd is 0,will finish the cmp*/234 bics has_nul, tmp1, tmp2 /* Non-zero if NUL terminator. */
227 bics has_nul, tmp1, tmp2235 ccmp diff, #0, #0, eq
228 ccmp endloop, #0, #0, eq /*has_null is ZERO: no null byte*/236 b.ne L(not_limit)
229 b.eq .Lloopcmp_proc237 subs limit_wd, limit_wd, #1
230 238 b.pl L(loop_misaligned)
231.Lunequal_proc:
232 orr syndrome, diff, has_nul
233 cbz syndrome, .Lremain8
234.Lcal_cmpresult:
235 /*
236 * reversed the byte-order as big-endian,then CLZ can find the most
237 * significant zero bits.
238 */
239CPU_LE( rev syndrome, syndrome )
240CPU_LE( rev data1, data1 )
241CPU_LE( rev data2, data2 )
242 /*
243 * For big-endian we cannot use the trick with the syndrome value
244 * as carry-propagation can corrupt the upper bits if the trailing
245 * bytes in the string contain 0x01.
246 * However, if there is no NUL byte in the dword, we can generate
247 * the result directly. We can't just subtract the bytes as the
248 * MSB might be significant.
249 */
250CPU_BE( cbnz has_nul, 1f )
251CPU_BE( cmp data1, data2 )
252CPU_BE( cset result, ne )
253CPU_BE( cneg result, result, lo )
254CPU_BE( ret )
255CPU_BE( 1: )
256 /* Re-compute the NUL-byte detection, using a byte-reversed value.*/
257CPU_BE( rev tmp3, data1 )
258CPU_BE( sub tmp1, tmp3, zeroones )
259CPU_BE( orr tmp2, tmp3, #REP8_7f )
260CPU_BE( bic has_nul, tmp1, tmp2 )
261CPU_BE( rev has_nul, has_nul )
262CPU_BE( orr syndrome, diff, has_nul )
263 /*
264 * The MS-non-zero bit of the syndrome marks either the first bit
265 * that is different, or the top bit of the first zero byte.
266 * Shifting left now will bring the critical information into the
267 * top bits.
268 */
269 clz pos, syndrome
270 lsl data1, data1, pos
271 lsl data2, data2, pos
272 /*
273 * But we need to zero-extend (char is unsigned) the value and then
274 * perform a signed 32-bit subtraction.
275 */
276 lsr data1, data1, #56
277 sub result, data1, data2, lsr #56
278 ret
279
280.Lremain8:
281 /* Limit % 8 == 0 => all bytes significant. */
282 ands limit, limit, #7
283 b.eq .Lret0
284.Ltiny8proc:
285 ldrb data1w, [src1], #1
286 ldrb data2w, [src2], #1
287 subs limit, limit, #1
288 239
289 ccmp data1w, #1, #0, ne /* NZCV = 0b0000. */240L(done_loop):
290 ccmp data1w, data2w, #0, cs /* NZCV = 0b0000. */241 /* We found a difference or a NULL before the limit was reached. */
291 b.eq .Ltiny8proc242 and limit, limit, #7
292 sub result, data1, data2243 cbz limit, L(not_limit)
293 ret244 /* Read the last word. */
245 sub src1, src1, 8
246 sub src2, src2, 8
247 ldr data1, [src1, limit]
248 ldr data2, [src2, limit]
249 sub tmp1, data1, zeroones
250 orr tmp2, data1, #REP8_7f
251 eor diff, data1, data2 /* Non-zero if differences found. */
252 bics has_nul, tmp1, tmp2 /* Non-zero if NUL terminator. */
253 ccmp diff, #0, #0, eq
254 b.ne L(not_limit)
294 255
295.Lret0:256L(ret0):
296 mov result, #0257 mov result, #0
297 ret258 ret
259
298SYM_FUNC_END_PI(strncmp)260SYM_FUNC_END_PI(strncmp)
299EXPORT_SYMBOL_NOKASAN(strncmp)261EXPORT_SYMBOL_NOKASAN(strncmp)
arch/arm64/lib/uaccess_flushcache.c+2−2
@@ -15,7 +15,7 @@ void memcpy_flushcache(void *dst, const void *src, size_t cnt) · 14 unmodified lines
15 * barrier to order the cache maintenance against the memcpy.15 * barrier to order the cache maintenance against the memcpy.
16 */16 */
17 memcpy(dst, src, cnt);17 memcpy(dst, src, cnt);
18 __clean_dcache_area_pop(dst, cnt);18 dcache_clean_pop((unsigned long)dst, (unsigned long)dst + cnt);
19}19}
20EXPORT_SYMBOL_GPL(memcpy_flushcache);20EXPORT_SYMBOL_GPL(memcpy_flushcache);
21 21
@@ -33,6 +33,6 @@ unsigned long __copy_user_flushcache(void *to, const void __user *from, · 11 unmodified lines
33 rc = raw_copy_from_user(to, from, n);33 rc = raw_copy_from_user(to, from, n);
34 34
35 /* See above */35 /* See above */
36 __clean_dcache_area_pop(to, n - rc);36 dcache_clean_pop((unsigned long)to, (unsigned long)to + n - rc);
37 return rc;37 return rc;
38}38}
arch/arm64/mm/cache.S+81−77
@@ -15,7 +15,7 @@ · 14 unmodified lines
15#include <asm/asm-uaccess.h>15#include <asm/asm-uaccess.h>
16 16
17/*17/*
18 * flush_icache_range(start,end)18 * caches_clean_inval_pou_macro(start,end) [fixup]
19 *19 *
20 * Ensure that the I and D caches are coherent within specified region.20 * Ensure that the I and D caches are coherent within specified region.
21 * This is typically used when code has been written to a memory region,21 * This is typically used when code has been written to a memory region,
@@ -23,12 +23,27 @@ · 1 unmodified line
23 *23 *
24 * - start - virtual start address of region24 * - start - virtual start address of region
25 * - end - virtual end address of region25 * - end - virtual end address of region
26 * - fixup - optional label to branch to on user fault
26 */27 */
27SYM_FUNC_START(__flush_icache_range)28.macro caches_clean_inval_pou_macro, fixup
28 /* FALLTHROUGH */29alternative_if ARM64_HAS_CACHE_IDC
30 dsb ishst
31 b .Ldc_skip_\@
32alternative_else_nop_endif
33 mov x2, x0
34 mov x3, x1
35 dcache_by_line_op cvau, ish, x2, x3, x4, x5, \fixup
36.Ldc_skip_\@:
37alternative_if ARM64_HAS_CACHE_DIC
38 isb
39 b .Lic_skip_\@
40alternative_else_nop_endif
41 invalidate_icache_by_line x0, x1, x2, x3, \fixup
42.Lic_skip_\@:
43.endm
29 44
30/*45/*
31 * __flush_cache_user_range(start,end)46 * caches_clean_inval_pou(start,end)
32 *47 *
33 * Ensure that the I and D caches are coherent within specified region.48 * Ensure that the I and D caches are coherent within specified region.
34 * This is typically used when code has been written to a memory region,49 * This is typically used when code has been written to a memory region,
@@ -37,117 +52,103 @@ SYM_FUNC_START(__flush_icache_range) · 2 unmodified lines
37 * - start - virtual start address of region52 * - start - virtual start address of region
38 * - end - virtual end address of region53 * - end - virtual end address of region
39 */54 */
40SYM_FUNC_START(__flush_cache_user_range)55SYM_FUNC_START(caches_clean_inval_pou)
56 caches_clean_inval_pou_macro
57 ret
58SYM_FUNC_END(caches_clean_inval_pou)
59
60/*
61 * caches_clean_inval_user_pou(start,end)
62 *
63 * Ensure that the I and D caches are coherent within specified region.
64 * This is typically used when code has been written to a memory region,
65 * and will be executed.
66 *
67 * - start - virtual start address of region
68 * - end - virtual end address of region
69 */
70SYM_FUNC_START(caches_clean_inval_user_pou)
41 uaccess_ttbr0_enable x2, x3, x471 uaccess_ttbr0_enable x2, x3, x4
42alternative_if ARM64_HAS_CACHE_IDC
43 dsb ishst
44 b 7f
45alternative_else_nop_endif
46 dcache_line_size x2, x3
47 sub x3, x2, #1
48 bic x4, x0, x3
491:
50user_alt 9f, "dc cvau, x4", "dc civac, x4", ARM64_WORKAROUND_CLEAN_CACHE
51 add x4, x4, x2
52 cmp x4, x1
53 b.lo 1b
54 dsb ish
55 72
567:73 caches_clean_inval_pou_macro 2f
57alternative_if ARM64_HAS_CACHE_DIC74 mov x0, xzr
58 isb
59 b 8f
60alternative_else_nop_endif
61 invalidate_icache_by_line x0, x1, x2, x3, 9f
628: mov x0, #0
631:751:
64 uaccess_ttbr0_disable x1, x276 uaccess_ttbr0_disable x1, x2
65 ret77 ret
669:782:
67 mov x0, #-EFAULT79 mov x0, #-EFAULT
68 b 1b80 b 1b
69SYM_FUNC_END(__flush_icache_range)81SYM_FUNC_END(caches_clean_inval_user_pou)
70SYM_FUNC_END(__flush_cache_user_range)
71 82
72/*83/*
73 * invalidate_icache_range(start,end)84 * icache_inval_pou(start,end)
74 *85 *
75 * Ensure that the I cache is invalid within specified region.86 * Ensure that the I cache is invalid within specified region.
76 *87 *
77 * - start - virtual start address of region88 * - start - virtual start address of region
78 * - end - virtual end address of region89 * - end - virtual end address of region
79 */90 */
80SYM_FUNC_START(invalidate_icache_range)91SYM_FUNC_START(icache_inval_pou)
81alternative_if ARM64_HAS_CACHE_DIC92alternative_if ARM64_HAS_CACHE_DIC
82 mov x0, xzr
83 isb93 isb
84 ret94 ret
85alternative_else_nop_endif95alternative_else_nop_endif
86 96
87 uaccess_ttbr0_enable x2, x3, x497 invalidate_icache_by_line x0, x1, x2, x3
88
89 invalidate_icache_by_line x0, x1, x2, x3, 2f
90 mov x0, xzr
911:
92 uaccess_ttbr0_disable x1, x2
93 ret98 ret
942:99SYM_FUNC_END(icache_inval_pou)
95 mov x0, #-EFAULT
96 b 1b
97SYM_FUNC_END(invalidate_icache_range)
98 100
99/*101/*
100 * __flush_dcache_area(kaddr, size)102 * dcache_clean_inval_poc(start, end)
101 *103 *
102 * Ensure that any D-cache lines for the interval [kaddr, kaddr+size)104 * Ensure that any D-cache lines for the interval [start, end)
103 * are cleaned and invalidated to the PoC.105 * are cleaned and invalidated to the PoC.
104 *106 *
105 * - kaddr - kernel address107 * - start - virtual start address of region
106 * - size - size in question108 * - end - virtual end address of region
107 */109 */
108SYM_FUNC_START_PI(__flush_dcache_area)110SYM_FUNC_START_PI(dcache_clean_inval_poc)
109 dcache_by_line_op civac, sy, x0, x1, x2, x3111 dcache_by_line_op civac, sy, x0, x1, x2, x3
110 ret112 ret
111SYM_FUNC_END_PI(__flush_dcache_area)113SYM_FUNC_END_PI(dcache_clean_inval_poc)
112 114
113/*115/*
114 * __clean_dcache_area_pou(kaddr, size)116 * dcache_clean_pou(start, end)
115 *117 *
116 * Ensure that any D-cache lines for the interval [kaddr, kaddr+size)118 * Ensure that any D-cache lines for the interval [start, end)
117 * are cleaned to the PoU.119 * are cleaned to the PoU.
118 *120 *
119 * - kaddr - kernel address121 * - start - virtual start address of region
120 * - size - size in question122 * - end - virtual end address of region
121 */123 */
122SYM_FUNC_START(__clean_dcache_area_pou)124SYM_FUNC_START(dcache_clean_pou)
123alternative_if ARM64_HAS_CACHE_IDC125alternative_if ARM64_HAS_CACHE_IDC
124 dsb ishst126 dsb ishst
125 ret127 ret
126alternative_else_nop_endif128alternative_else_nop_endif
127 dcache_by_line_op cvau, ish, x0, x1, x2, x3129 dcache_by_line_op cvau, ish, x0, x1, x2, x3
128 ret130 ret
129SYM_FUNC_END(__clean_dcache_area_pou)131SYM_FUNC_END(dcache_clean_pou)
130 132
131/*133/*
132 * __inval_dcache_area(kaddr, size)134 * dcache_inval_poc(start, end)
133 *135 *
134 * Ensure that any D-cache lines for the interval [kaddr, kaddr+size)136 * Ensure that any D-cache lines for the interval [start, end)
135 * are invalidated. Any partial lines at the ends of the interval are137 * are invalidated. Any partial lines at the ends of the interval are
136 * also cleaned to PoC to prevent data loss.138 * also cleaned to PoC to prevent data loss.
137 *139 *
138 * - kaddr - kernel address140 * - start - kernel start address of region
139 * - size - size in question141 * - end - kernel end address of region
140 */142 */
141SYM_FUNC_START_LOCAL(__dma_inv_area)143SYM_FUNC_START_LOCAL(__dma_inv_area)
142SYM_FUNC_START_PI(__inval_dcache_area)144SYM_FUNC_START_PI(dcache_inval_poc)
143 /* FALLTHROUGH */145 /* FALLTHROUGH */
144 146
145/*147/*
146 * __dma_inv_area(start, size)148 * __dma_inv_area(start, end)
147 * - start - virtual start address of region149 * - start - virtual start address of region
148 * - size - size in question150 * - end - virtual end address of region
149 */151 */
150 add x1, x1, x0
151 dcache_line_size x2, x3152 dcache_line_size x2, x3
152 sub x3, x2, #1153 sub x3, x2, #1
153 tst x1, x3 // end cache line aligned?154 tst x1, x3 // end cache line aligned?
@@ -165,48 +166,48 @@ SYM_FUNC_START_PI(__inval_dcache_area) · 11 unmodified lines
165 b.lo 2b166 b.lo 2b
166 dsb sy167 dsb sy
167 ret168 ret
168SYM_FUNC_END_PI(__inval_dcache_area)169SYM_FUNC_END_PI(dcache_inval_poc)
169SYM_FUNC_END(__dma_inv_area)170SYM_FUNC_END(__dma_inv_area)
170 171
171/*172/*
172 * __clean_dcache_area_poc(kaddr, size)173 * dcache_clean_poc(start, end)
173 *174 *
174 * Ensure that any D-cache lines for the interval [kaddr, kaddr+size)175 * Ensure that any D-cache lines for the interval [start, end)
175 * are cleaned to the PoC.176 * are cleaned to the PoC.
176 *177 *
177 * - kaddr - kernel address178 * - start - virtual start address of region
178 * - size - size in question179 * - end - virtual end address of region
179 */180 */
180SYM_FUNC_START_LOCAL(__dma_clean_area)181SYM_FUNC_START_LOCAL(__dma_clean_area)
181SYM_FUNC_START_PI(__clean_dcache_area_poc)182SYM_FUNC_START_PI(dcache_clean_poc)
182 /* FALLTHROUGH */183 /* FALLTHROUGH */
183 184
184/*185/*
185 * __dma_clean_area(start, size)186 * __dma_clean_area(start, end)
186 * - start - virtual start address of region187 * - start - virtual start address of region
187 * - size - size in question188 * - end - virtual end address of region
188 */189 */
189 dcache_by_line_op cvac, sy, x0, x1, x2, x3190 dcache_by_line_op cvac, sy, x0, x1, x2, x3
190 ret191 ret
191SYM_FUNC_END_PI(__clean_dcache_area_poc)192SYM_FUNC_END_PI(dcache_clean_poc)
192SYM_FUNC_END(__dma_clean_area)193SYM_FUNC_END(__dma_clean_area)
193 194
194/*195/*
195 * __clean_dcache_area_pop(kaddr, size)196 * dcache_clean_pop(start, end)
196 *197 *
197 * Ensure that any D-cache lines for the interval [kaddr, kaddr+size)198 * Ensure that any D-cache lines for the interval [start, end)
198 * are cleaned to the PoP.199 * are cleaned to the PoP.
199 *200 *
200 * - kaddr - kernel address201 * - start - virtual start address of region
201 * - size - size in question202 * - end - virtual end address of region
202 */203 */
203SYM_FUNC_START_PI(__clean_dcache_area_pop)204SYM_FUNC_START_PI(dcache_clean_pop)
204 alternative_if_not ARM64_HAS_DCPOP205 alternative_if_not ARM64_HAS_DCPOP
205 b __clean_dcache_area_poc206 b dcache_clean_poc
206 alternative_else_nop_endif207 alternative_else_nop_endif
207 dcache_by_line_op cvap, sy, x0, x1, x2, x3208 dcache_by_line_op cvap, sy, x0, x1, x2, x3
208 ret209 ret
209SYM_FUNC_END_PI(__clean_dcache_area_pop)210SYM_FUNC_END_PI(dcache_clean_pop)
210 211
211/*212/*
212 * __dma_flush_area(start, size)213 * __dma_flush_area(start, size)
@@ -217,6 +218,7 @@ SYM_FUNC_END_PI(__clean_dcache_area_pop) · 4 unmodified lines
217 * - size - size in question218 * - size - size in question
218 */219 */
219SYM_FUNC_START_PI(__dma_flush_area)220SYM_FUNC_START_PI(__dma_flush_area)
221 add x1, x0, x1
220 dcache_by_line_op civac, sy, x0, x1, x2, x3222 dcache_by_line_op civac, sy, x0, x1, x2, x3
221 ret223 ret
222SYM_FUNC_END_PI(__dma_flush_area)224SYM_FUNC_END_PI(__dma_flush_area)
@@ -228,6 +230,7 @@ SYM_FUNC_END_PI(__dma_flush_area) · 5 unmodified lines
228 * - dir - DMA direction230 * - dir - DMA direction
229 */231 */
230SYM_FUNC_START_PI(__dma_map_area)232SYM_FUNC_START_PI(__dma_map_area)
233 add x1, x0, x1
231 cmp w2, #DMA_FROM_DEVICE234 cmp w2, #DMA_FROM_DEVICE
232 b.eq __dma_inv_area235 b.eq __dma_inv_area
233 b __dma_clean_area236 b __dma_clean_area
@@ -240,6 +243,7 @@ SYM_FUNC_END_PI(__dma_map_area) · 6 unmodified lines
240 * - dir - DMA direction243 * - dir - DMA direction
241 */244 */
242SYM_FUNC_START_PI(__dma_unmap_area)245SYM_FUNC_START_PI(__dma_unmap_area)
246 add x1, x0, x1
243 cmp w2, #DMA_TO_DEVICE247 cmp w2, #DMA_TO_DEVICE
244 b.ne __dma_inv_area248 b.ne __dma_inv_area
245 ret249 ret
arch/arm64/mm/context.c+2−4
@@ -402,14 +402,12 @@ static int asids_init(void) · 401 unmodified lines
402{402{
403 asid_bits = get_cpu_asid_bits();403 asid_bits = get_cpu_asid_bits();
404 atomic64_set(&asid_generation, ASID_FIRST_VERSION);404 atomic64_set(&asid_generation, ASID_FIRST_VERSION);
405 asid_map = kcalloc(BITS_TO_LONGS(NUM_USER_ASIDS), sizeof(*asid_map),405 asid_map = bitmap_zalloc(NUM_USER_ASIDS, GFP_KERNEL);
406 GFP_KERNEL);
407 if (!asid_map)406 if (!asid_map)
408 panic("Failed to allocate bitmap for %lu ASIDs\n",407 panic("Failed to allocate bitmap for %lu ASIDs\n",
409 NUM_USER_ASIDS);408 NUM_USER_ASIDS);
410 409
411 pinned_asid_map = kcalloc(BITS_TO_LONGS(NUM_USER_ASIDS),410 pinned_asid_map = bitmap_zalloc(NUM_USER_ASIDS, GFP_KERNEL);
412 sizeof(*pinned_asid_map), GFP_KERNEL);
413 nr_pinned_asids = 0;411 nr_pinned_asids = 0;
414 412
415 /*413 /*
arch/arm64/mm/fault.c+37−13
@@ -99,6 +99,8 @@ static void mem_abort_decode(unsigned int esr) · 98 unmodified lines
99 pr_alert(" EA = %lu, S1PTW = %lu\n",99 pr_alert(" EA = %lu, S1PTW = %lu\n",
100 (esr & ESR_ELx_EA) >> ESR_ELx_EA_SHIFT,100 (esr & ESR_ELx_EA) >> ESR_ELx_EA_SHIFT,
101 (esr & ESR_ELx_S1PTW) >> ESR_ELx_S1PTW_SHIFT);101 (esr & ESR_ELx_S1PTW) >> ESR_ELx_S1PTW_SHIFT);
102 pr_alert(" FSC = 0x%02x: %s\n", (esr & ESR_ELx_FSC),
103 esr_to_fault_info(esr)->name);
102 104
103 if (esr_is_data_abort(esr))105 if (esr_is_data_abort(esr))
104 data_abort_decode(esr);106 data_abort_decode(esr);
@@ -232,13 +234,17 @@ static bool is_el1_instruction_abort(unsigned int esr) · 127 unmodified lines
232 return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_CUR;234 return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_CUR;
233}235}
234 236
237static bool is_el1_data_abort(unsigned int esr)
238{
239 return ESR_ELx_EC(esr) == ESR_ELx_EC_DABT_CUR;
240}
241
235static inline bool is_el1_permission_fault(unsigned long addr, unsigned int esr,242static inline bool is_el1_permission_fault(unsigned long addr, unsigned int esr,
236 struct pt_regs *regs)243 struct pt_regs *regs)
237{244{
238 unsigned int ec = ESR_ELx_EC(esr);
239 unsigned int fsc_type = esr & ESR_ELx_FSC_TYPE;245 unsigned int fsc_type = esr & ESR_ELx_FSC_TYPE;
240 246
241 if (ec != ESR_ELx_EC_DABT_CUR && ec != ESR_ELx_EC_IABT_CUR)247 if (!is_el1_data_abort(esr) && !is_el1_instruction_abort(esr))
242 return false;248 return false;
243 249
244 if (fsc_type == ESR_ELx_FSC_PERM)250 if (fsc_type == ESR_ELx_FSC_PERM)
@@ -258,7 +264,7 @@ static bool __kprobes is_spurious_el1_translation_fault(unsigned long addr, · 13 unmodified lines
258 unsigned long flags;264 unsigned long flags;
259 u64 par, dfsc;265 u64 par, dfsc;
260 266
261 if (ESR_ELx_EC(esr) != ESR_ELx_EC_DABT_CUR ||267 if (!is_el1_data_abort(esr) ||
262 (esr & ESR_ELx_FSC_TYPE) != ESR_ELx_FSC_FAULT)268 (esr & ESR_ELx_FSC_TYPE) != ESR_ELx_FSC_FAULT)
263 return false;269 return false;
264 270
@@ -346,10 +352,9 @@ static void do_tag_recovery(unsigned long addr, unsigned int esr, · 81 unmodified lines
346 352
347static bool is_el1_mte_sync_tag_check_fault(unsigned int esr)353static bool is_el1_mte_sync_tag_check_fault(unsigned int esr)
348{354{
349 unsigned int ec = ESR_ELx_EC(esr);
350 unsigned int fsc = esr & ESR_ELx_FSC;355 unsigned int fsc = esr & ESR_ELx_FSC;
351 356
352 if (ec != ESR_ELx_EC_DABT_CUR)357 if (!is_el1_data_abort(esr))
353 return false;358 return false;
354 359
355 if (fsc == ESR_ELx_FSC_MTE)360 if (fsc == ESR_ELx_FSC_MTE)
@@ -504,7 +509,7 @@ static vm_fault_t __do_page_fault(struct mm_struct *mm, unsigned long addr, · 148 unmodified lines
504 */509 */
505 if (!(vma->vm_flags & vm_flags))510 if (!(vma->vm_flags & vm_flags))
506 return VM_FAULT_BADACCESS;511 return VM_FAULT_BADACCESS;
507 return handle_mm_fault(vma, addr & PAGE_MASK, mm_flags, regs);512 return handle_mm_fault(vma, addr, mm_flags, regs);
508}513}
509 514
510static bool is_el0_instruction_abort(unsigned int esr)515static bool is_el0_instruction_abort(unsigned int esr)
@@ -836,13 +841,6 @@ void do_mem_abort(unsigned long far, unsigned int esr, struct pt_regs *regs) · 325 unmodified lines
836}841}
837NOKPROBE_SYMBOL(do_mem_abort);842NOKPROBE_SYMBOL(do_mem_abort);
838 843
839void do_el0_irq_bp_hardening(void)
840{
841 /* PC has already been checked in entry.S */
842 arm64_apply_bp_hardening();
843}
844NOKPROBE_SYMBOL(do_el0_irq_bp_hardening);
845
846void do_sp_pc_abort(unsigned long addr, unsigned int esr, struct pt_regs *regs)844void do_sp_pc_abort(unsigned long addr, unsigned int esr, struct pt_regs *regs)
847{845{
848 arm64_notify_die("SP/PC alignment exception", regs, SIGBUS, BUS_ADRALN,846 arm64_notify_die("SP/PC alignment exception", regs, SIGBUS, BUS_ADRALN,
@@ -921,3 +919,29 @@ void do_debug_exception(unsigned long addr_if_watchpoint, unsigned int esr, · 72 unmodified lines
921 debug_exception_exit(regs);919 debug_exception_exit(regs);
922}920}
923NOKPROBE_SYMBOL(do_debug_exception);921NOKPROBE_SYMBOL(do_debug_exception);
922
923/*
924 * Used during anonymous page fault handling.
925 */
926struct page *alloc_zeroed_user_highpage_movable(struct vm_area_struct *vma,
927 unsigned long vaddr)
928{
929 gfp_t flags = GFP_HIGHUSER_MOVABLE | __GFP_ZERO;
930
931 /*
932 * If the page is mapped with PROT_MTE, initialise the tags at the
933 * point of allocation and page zeroing as this is usually faster than
934 * separate DC ZVA and STGM.
935 */
936 if (vma->vm_flags & VM_MTE)
937 flags |= __GFP_ZEROTAGS;
938
939 return alloc_page_vma(flags, vma, vaddr);
940}
941
942void tag_clear_highpage(struct page *page)
943{
944 mte_zero_clear_page_tags(page_address(page));
945 page_kasan_tag_reset(page);
946 set_bit(PG_mte_tagged, &page->flags);
947}
arch/arm64/mm/flush.c+14−15
@@ -14,28 +14,25 @@ · 13 unmodified lines
14#include <asm/cache.h>14#include <asm/cache.h>
15#include <asm/tlbflush.h>15#include <asm/tlbflush.h>
16 16
17void sync_icache_aliases(void *kaddr, unsigned long len)17void sync_icache_aliases(unsigned long start, unsigned long end)
18{18{
19 unsigned long addr = (unsigned long)kaddr;
20
21 if (icache_is_aliasing()) {19 if (icache_is_aliasing()) {
22 __clean_dcache_area_pou(kaddr, len);20 dcache_clean_pou(start, end);
23 __flush_icache_all();21 icache_inval_all_pou();
24 } else {22 } else {
25 /*23 /*
26 * Don't issue kick_all_cpus_sync() after I-cache invalidation24 * Don't issue kick_all_cpus_sync() after I-cache invalidation
27 * for user mappings.25 * for user mappings.
28 */26 */
29 __flush_icache_range(addr, addr + len);27 caches_clean_inval_pou(start, end);
30 }28 }
31}29}
32 30
33static void flush_ptrace_access(struct vm_area_struct *vma, struct page *page,31static void flush_ptrace_access(struct vm_area_struct *vma, unsigned long start,
34 unsigned long uaddr, void *kaddr,32 unsigned long end)
35 unsigned long len)
36{33{
37 if (vma->vm_flags & VM_EXEC)34 if (vma->vm_flags & VM_EXEC)
38 sync_icache_aliases(kaddr, len);35 sync_icache_aliases(start, end);
39}36}
40 37
41/*38/*
@@ -48,7 +45,7 @@ void copy_to_user_page(struct vm_area_struct *vma, struct page *page, · 6 unmodified lines
48 unsigned long len)45 unsigned long len)
49{46{
50 memcpy(dst, src, len);47 memcpy(dst, src, len);
51 flush_ptrace_access(vma, page, uaddr, dst, len);48 flush_ptrace_access(vma, (unsigned long)dst, (unsigned long)dst + len);
52}49}
53 50
54void __sync_icache_dcache(pte_t pte)51void __sync_icache_dcache(pte_t pte)
@@ -56,7 +53,9 @@ void __sync_icache_dcache(pte_t pte) · 1 unmodified line
56 struct page *page = pte_page(pte);53 struct page *page = pte_page(pte);
57 54
58 if (!test_bit(PG_dcache_clean, &page->flags)) {55 if (!test_bit(PG_dcache_clean, &page->flags)) {
59 sync_icache_aliases(page_address(page), page_size(page));56 sync_icache_aliases((unsigned long)page_address(page),
57 (unsigned long)page_address(page) +
58 page_size(page));
60 set_bit(PG_dcache_clean, &page->flags);59 set_bit(PG_dcache_clean, &page->flags);
61 }60 }
62}61}
@@ -77,20 +76,20 @@ EXPORT_SYMBOL(flush_dcache_page); · 14 unmodified lines
77/*76/*
78 * Additional functions defined in assembly.77 * Additional functions defined in assembly.
79 */78 */
80EXPORT_SYMBOL(__flush_icache_range);79EXPORT_SYMBOL(caches_clean_inval_pou);
81 80
82#ifdef CONFIG_ARCH_HAS_PMEM_API81#ifdef CONFIG_ARCH_HAS_PMEM_API
83void arch_wb_cache_pmem(void *addr, size_t size)82void arch_wb_cache_pmem(void *addr, size_t size)
84{83{
85 /* Ensure order against any prior non-cacheable writes */84 /* Ensure order against any prior non-cacheable writes */
86 dmb(osh);85 dmb(osh);
87 __clean_dcache_area_pop(addr, size);86 dcache_clean_pop((unsigned long)addr, (unsigned long)addr + size);
88}87}
89EXPORT_SYMBOL_GPL(arch_wb_cache_pmem);88EXPORT_SYMBOL_GPL(arch_wb_cache_pmem);
90 89
91void arch_invalidate_pmem(void *addr, size_t size)90void arch_invalidate_pmem(void *addr, size_t size)
92{91{
93 __inval_dcache_area(addr, size);92 dcache_inval_poc((unsigned long)addr, (unsigned long)addr + size);
94}93}
95EXPORT_SYMBOL_GPL(arch_invalidate_pmem);94EXPORT_SYMBOL_GPL(arch_invalidate_pmem);
96#endif95#endif
arch/arm64/mm/init.c+7−0
@@ -499,6 +499,13 @@ void __init mem_init(void) · 498 unmodified lines
499 BUILD_BUG_ON(TASK_SIZE_32 > DEFAULT_MAP_WINDOW_64);499 BUILD_BUG_ON(TASK_SIZE_32 > DEFAULT_MAP_WINDOW_64);
500#endif500#endif
501 501
502 /*
503 * Selected page table levels should match when derived from
504 * scratch using the virtual address range and page size.
505 */
506 BUILD_BUG_ON(ARM64_HW_PGTABLE_LEVELS(CONFIG_ARM64_VA_BITS) !=
507 CONFIG_PGTABLE_LEVELS);
508
502 if (PAGE_SIZE >= 16384 && get_num_physpages() <= 128) {509 if (PAGE_SIZE >= 16384 && get_num_physpages() <= 128) {
503 extern int sysctl_overcommit_memory;510 extern int sysctl_overcommit_memory;
504 /*511 /*
arch/arm64/mm/mmu.c+7−6
@@ -228,7 +228,7 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, · 227 unmodified lines
228 next = pmd_addr_end(addr, end);228 next = pmd_addr_end(addr, end);
229 229
230 /* try section mapping first */230 /* try section mapping first */
231 if (((addr | next | phys) & ~SECTION_MASK) == 0 &&231 if (((addr | next | phys) & ~PMD_MASK) == 0 &&
232 (flags & NO_BLOCK_MAPPINGS) == 0) {232 (flags & NO_BLOCK_MAPPINGS) == 0) {
233 pmd_set_huge(pmdp, phys, prot);233 pmd_set_huge(pmdp, phys, prot);
234 234
@@ -1114,14 +1114,14 @@ static void free_empty_tables(unsigned long addr, unsigned long end, · 879 unmodified lines
1114}1114}
1115#endif1115#endif
1116 1116
1117#if !ARM64_SWAPPER_USES_SECTION_MAPS1117#if !ARM64_KERNEL_USES_PMD_MAPS
1118int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,1118int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
1119 struct vmem_altmap *altmap)1119 struct vmem_altmap *altmap)
1120{1120{
1121 WARN_ON((start < VMEMMAP_START) || (end > VMEMMAP_END));1121 WARN_ON((start < VMEMMAP_START) || (end > VMEMMAP_END));
1122 return vmemmap_populate_basepages(start, end, node, altmap);1122 return vmemmap_populate_basepages(start, end, node, altmap);
1123}1123}
1124#else /* !ARM64_SWAPPER_USES_SECTION_MAPS */1124#else /* !ARM64_KERNEL_USES_PMD_MAPS */
1125int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,1125int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
1126 struct vmem_altmap *altmap)1126 struct vmem_altmap *altmap)
1127{1127{
@@ -1166,17 +1166,18 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, · 38 unmodified lines
1166 1166
1167 return 0;1167 return 0;
1168}1168}
1169#endif /* !ARM64_SWAPPER_USES_SECTION_MAPS */1169#endif /* !ARM64_KERNEL_USES_PMD_MAPS */
1170
1171#ifdef CONFIG_MEMORY_HOTPLUG
1170void vmemmap_free(unsigned long start, unsigned long end,1172void vmemmap_free(unsigned long start, unsigned long end,
1171 struct vmem_altmap *altmap)1173 struct vmem_altmap *altmap)
1172{1174{
1173#ifdef CONFIG_MEMORY_HOTPLUG
1174 WARN_ON((start < VMEMMAP_START) || (end > VMEMMAP_END));1175 WARN_ON((start < VMEMMAP_START) || (end > VMEMMAP_END));
1175 1176
1176 unmap_hotplug_range(start, end, true, altmap);1177 unmap_hotplug_range(start, end, true, altmap);
1177 free_empty_tables(start, end, VMEMMAP_START, VMEMMAP_END);1178 free_empty_tables(start, end, VMEMMAP_START, VMEMMAP_END);
1178#endif
1179}1179}
1180#endif /* CONFIG_MEMORY_HOTPLUG */
1180 1181
1181static inline pud_t *fixmap_pud(unsigned long addr)1182static inline pud_t *fixmap_pud(unsigned long addr)
1182{1183{
arch/arm64/mm/proc.S+9−15
@@ -46,9 +46,13 @@ · 45 unmodified lines
46#endif46#endif
47 47
48#ifdef CONFIG_KASAN_HW_TAGS48#ifdef CONFIG_KASAN_HW_TAGS
49#define TCR_KASAN_HW_FLAGS SYS_TCR_EL1_TCMA1 | TCR_TBI1 | TCR_TBID149#define TCR_MTE_FLAGS SYS_TCR_EL1_TCMA1 | TCR_TBI1 | TCR_TBID1
50#else50#else
51#define TCR_KASAN_HW_FLAGS 051/*
52 * The mte_zero_clear_page_tags() implementation uses DC GZVA, which relies on
53 * TBI being enabled at EL1.
54 */
55#define TCR_MTE_FLAGS TCR_TBI1 | TCR_TBID1
52#endif56#endif
53 57
54/*58/*
@@ -58,10 +62,8 @@ · 3 unmodified lines
58#define MAIR_EL1_SET \62#define MAIR_EL1_SET \
59 (MAIR_ATTRIDX(MAIR_ATTR_DEVICE_nGnRnE, MT_DEVICE_nGnRnE) | \63 (MAIR_ATTRIDX(MAIR_ATTR_DEVICE_nGnRnE, MT_DEVICE_nGnRnE) | \
60 MAIR_ATTRIDX(MAIR_ATTR_DEVICE_nGnRE, MT_DEVICE_nGnRE) | \64 MAIR_ATTRIDX(MAIR_ATTR_DEVICE_nGnRE, MT_DEVICE_nGnRE) | \
61 MAIR_ATTRIDX(MAIR_ATTR_DEVICE_GRE, MT_DEVICE_GRE) | \
62 MAIR_ATTRIDX(MAIR_ATTR_NORMAL_NC, MT_NORMAL_NC) | \65 MAIR_ATTRIDX(MAIR_ATTR_NORMAL_NC, MT_NORMAL_NC) | \
63 MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL) | \66 MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL) | \
64 MAIR_ATTRIDX(MAIR_ATTR_NORMAL_WT, MT_NORMAL_WT) | \
65 MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL_TAGGED))67 MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL_TAGGED))
66 68
67#ifdef CONFIG_CPU_PM69#ifdef CONFIG_CPU_PM
@@ -83,11 +85,7 @@ SYM_FUNC_START(cpu_do_suspend) · 15 unmodified lines
83 mrs x9, mdscr_el185 mrs x9, mdscr_el1
84 mrs x10, oslsr_el186 mrs x10, oslsr_el1
85 mrs x11, sctlr_el187 mrs x11, sctlr_el1
86alternative_if_not ARM64_HAS_VIRT_HOST_EXTN88 get_this_cpu_offset x12
87 mrs x12, tpidr_el1
88alternative_else
89 mrs x12, tpidr_el2
90alternative_endif
91 mrs x13, sp_el089 mrs x13, sp_el0
92 stp x2, x3, [x0]90 stp x2, x3, [x0]
93 stp x4, x5, [x0, #16]91 stp x4, x5, [x0, #16]
@@ -145,11 +143,7 @@ SYM_FUNC_START(cpu_do_resume) · 51 unmodified lines
145 msr mdscr_el1, x10143 msr mdscr_el1, x10
146 144
147 msr sctlr_el1, x12145 msr sctlr_el1, x12
148alternative_if_not ARM64_HAS_VIRT_HOST_EXTN146 set_this_cpu_offset x13
149 msr tpidr_el1, x13
150alternative_else
151 msr tpidr_el2, x13
152alternative_endif
153 msr sp_el0, x14147 msr sp_el0, x14
154 /*148 /*
155 * Restore oslsr_el1 by writing oslar_el1149 * Restore oslsr_el1 by writing oslar_el1
@@ -464,7 +458,7 @@ SYM_FUNC_START(__cpu_setup) · 308 unmodified lines
464 msr_s SYS_TFSRE0_EL1, xzr458 msr_s SYS_TFSRE0_EL1, xzr
465 459
466 /* set the TCR_EL1 bits */460 /* set the TCR_EL1 bits */
467 mov_q x10, TCR_KASAN_HW_FLAGS461 mov_q x10, TCR_MTE_FLAGS
468 orr tcr, tcr, x10462 orr tcr, tcr, x10
4691:4631:
470#endif464#endif
arch/arm64/mm/ptdump.c+0−4
@@ -157,10 +157,6 @@ static const struct prot_bits pte_bits[] = { · 156 unmodified lines
157 .mask = PTE_ATTRINDX_MASK,157 .mask = PTE_ATTRINDX_MASK,
158 .val = PTE_ATTRINDX(MT_DEVICE_nGnRE),158 .val = PTE_ATTRINDX(MT_DEVICE_nGnRE),
159 .set = "DEVICE/nGnRE",159 .set = "DEVICE/nGnRE",
160 }, {
161 .mask = PTE_ATTRINDX_MASK,
162 .val = PTE_ATTRINDX(MT_DEVICE_GRE),
163 .set = "DEVICE/GRE",
164 }, {160 }, {
165 .mask = PTE_ATTRINDX_MASK,161 .mask = PTE_ATTRINDX_MASK,
166 .val = PTE_ATTRINDX(MT_NORMAL_NC),162 .val = PTE_ATTRINDX(MT_NORMAL_NC),
arch/arm64/net/bpf_jit_comp.c+1−0
@@ -16,6 +16,7 @@ · 15 unmodified lines
16#include <asm/byteorder.h>16#include <asm/byteorder.h>
17#include <asm/cacheflush.h>17#include <asm/cacheflush.h>
18#include <asm/debug-monitors.h>18#include <asm/debug-monitors.h>
19#include <asm/insn.h>
19#include <asm/set_memory.h>20#include <asm/set_memory.h>
20 21
21#include "bpf_jit.h"22#include "bpf_jit.h"
arch/arm64/tools/cpucaps+2−1
@@ -3,7 +3,8 @@ · 2 unmodified lines
3# Internal CPU capabilities constants, keep this list sorted3# Internal CPU capabilities constants, keep this list sorted
4 4
5BTI5BTI
6HAS_32BIT_EL06# Unreliable: use system_supports_32bit_el0() instead.
7HAS_32BIT_EL0_DO_NOT_USE
7HAS_32BIT_EL18HAS_32BIT_EL1
8HAS_ADDRESS_AUTH9HAS_ADDRESS_AUTH
9HAS_ADDRESS_AUTH_ARCH10HAS_ADDRESS_AUTH_ARCH
arch/ia64/include/asm/page.h+3−3
@@ -82,16 +82,16 @@ do { \ · 81 unmodified lines
82} while (0)82} while (0)
83 83
84 84
85#define __alloc_zeroed_user_highpage(movableflags, vma, vaddr) \85#define alloc_zeroed_user_highpage_movable(vma, vaddr) \
86({ \86({ \
87 struct page *page = alloc_page_vma( \87 struct page *page = alloc_page_vma( \
88 GFP_HIGHUSER | __GFP_ZERO | movableflags, vma, vaddr); \88 GFP_HIGHUSER_MOVABLE | __GFP_ZERO, vma, vaddr); \
89 if (page) \89 if (page) \
90 flush_dcache_page(page); \90 flush_dcache_page(page); \
91 page; \91 page; \
92})92})
93 93
94#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE94#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE
95 95
96#define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)96#define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
97 97
arch/m68k/include/asm/page_no.h+3−3
@@ -13,9 +13,9 @@ extern unsigned long memory_end; · 12 unmodified lines
13#define clear_user_page(page, vaddr, pg) clear_page(page)13#define clear_user_page(page, vaddr, pg) clear_page(page)
14#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)14#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
15 15
16#define __alloc_zeroed_user_highpage(movableflags, vma, vaddr) \16#define alloc_zeroed_user_highpage_movable(vma, vaddr) \
17 alloc_page_vma(GFP_HIGHUSER | __GFP_ZERO | movableflags, vma, vaddr)17 alloc_page_vma(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, vma, vaddr)
18#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE18#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE
19 19
20#define __pa(vaddr) ((unsigned long)(vaddr))20#define __pa(vaddr) ((unsigned long)(vaddr))
21#define __va(paddr) ((void *)((unsigned long)(paddr)))21#define __va(paddr) ((void *)((unsigned long)(paddr)))
arch/s390/include/asm/page.h+3−3
@@ -68,9 +68,9 @@ static inline void copy_page(void *to, void *from) · 67 unmodified lines
68#define clear_user_page(page, vaddr, pg) clear_page(page)68#define clear_user_page(page, vaddr, pg) clear_page(page)
69#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)69#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
70 70
71#define __alloc_zeroed_user_highpage(movableflags, vma, vaddr) \71#define alloc_zeroed_user_highpage_movable(vma, vaddr) \
72 alloc_page_vma(GFP_HIGHUSER | __GFP_ZERO | movableflags, vma, vaddr)72 alloc_page_vma(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, vma, vaddr)
73#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE73#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE
74 74
75/*75/*
76 * These are used to make use of C type-checking..76 * These are used to make use of C type-checking..
arch/x86/include/asm/page.h+3−3
@@ -34,9 +34,9 @@ static inline void copy_user_page(void *to, void *from, unsigned long vaddr, · 33 unmodified lines
34 copy_page(to, from);34 copy_page(to, from);
35}35}
36 36
37#define __alloc_zeroed_user_highpage(movableflags, vma, vaddr) \37#define alloc_zeroed_user_highpage_movable(vma, vaddr) \
38 alloc_page_vma(GFP_HIGHUSER | __GFP_ZERO | movableflags, vma, vaddr)38 alloc_page_vma(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, vma, vaddr)
39#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE39#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE
40 40
41#ifndef __pa41#ifndef __pa
42#define __pa(x) __phys_addr((unsigned long)(x))42#define __pa(x) __phys_addr((unsigned long)(x))
drivers/firmware/psci/psci.c+7−2
@@ -335,10 +335,15 @@ int psci_cpu_suspend_enter(u32 state) · 334 unmodified lines
335{335{
336 int ret;336 int ret;
337 337
338 if (!psci_power_state_loses_context(state))338 if (!psci_power_state_loses_context(state)) {
339 struct arm_cpuidle_irq_context context;
340
341 arm_cpuidle_save_irq_context(&context);
339 ret = psci_ops.cpu_suspend(state, 0);342 ret = psci_ops.cpu_suspend(state, 0);
340 else343 arm_cpuidle_restore_irq_context(&context);
344 } else {
341 ret = cpu_suspend(state, psci_suspend_finisher);345 ret = cpu_suspend(state, psci_suspend_finisher);
346 }
342 347
343 return ret;348 return ret;
344}349}
drivers/firmware/smccc/smccc.c+4−0
@@ -15,6 +15,7 @@ static u32 smccc_version = ARM_SMCCC_VERSION_1_0; · 14 unmodified lines
15static enum arm_smccc_conduit smccc_conduit = SMCCC_CONDUIT_NONE;15static enum arm_smccc_conduit smccc_conduit = SMCCC_CONDUIT_NONE;
16 16
17bool __ro_after_init smccc_trng_available = false;17bool __ro_after_init smccc_trng_available = false;
18u64 __ro_after_init smccc_has_sve_hint = false;
18 19
19void __init arm_smccc_version_init(u32 version, enum arm_smccc_conduit conduit)20void __init arm_smccc_version_init(u32 version, enum arm_smccc_conduit conduit)
20{21{
@@ -22,6 +23,9 @@ void __init arm_smccc_version_init(u32 version, enum arm_smccc_conduit conduit) · 1 unmodified line
22 smccc_conduit = conduit;23 smccc_conduit = conduit;
23 24
24 smccc_trng_available = smccc_probe_trng();25 smccc_trng_available = smccc_probe_trng();
26 if (IS_ENABLED(CONFIG_ARM64_SVE) &&
27 smccc_version >= ARM_SMCCC_VERSION_1_3)
28 smccc_has_sve_hint = true;
25}29}
26 30
27enum arm_smccc_conduit arm_smccc_1_1_get_conduit(void)31enum arm_smccc_conduit arm_smccc_1_1_get_conduit(void)
drivers/misc/lkdtm/bugs.c+3−3
@@ -463,7 +463,7 @@ void lkdtm_DOUBLE_FAULT(void) · 462 unmodified lines
463#ifdef CONFIG_ARM64463#ifdef CONFIG_ARM64
464static noinline void change_pac_parameters(void)464static noinline void change_pac_parameters(void)
465{465{
466 if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH)) {466 if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL)) {
467 /* Reset the keys of current task */467 /* Reset the keys of current task */
468 ptrauth_thread_init_kernel(current);468 ptrauth_thread_init_kernel(current);
469 ptrauth_thread_switch_kernel(current);469 ptrauth_thread_switch_kernel(current);
@@ -477,8 +477,8 @@ noinline void lkdtm_CORRUPT_PAC(void) · 7 unmodified lines
477#define CORRUPT_PAC_ITERATE 10477#define CORRUPT_PAC_ITERATE 10
478 int i;478 int i;
479 479
480 if (!IS_ENABLED(CONFIG_ARM64_PTR_AUTH))480 if (!IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
481 pr_err("FAIL: kernel not built with CONFIG_ARM64_PTR_AUTH\n");481 pr_err("FAIL: kernel not built with CONFIG_ARM64_PTR_AUTH_KERNEL\n");
482 482
483 if (!system_supports_address_auth()) {483 if (!system_supports_address_auth()) {
484 pr_err("FAIL: CPU lacks pointer authentication feature\n");484 pr_err("FAIL: CPU lacks pointer authentication feature\n");
drivers/perf/arm-cci.c+2−2
@@ -37,7 +37,7 @@ · 36 unmodified lines
37 37
38#define CCI_PMU_CNTR_SIZE(model) ((model)->cntr_size)38#define CCI_PMU_CNTR_SIZE(model) ((model)->cntr_size)
39#define CCI_PMU_CNTR_BASE(model, idx) ((idx) * CCI_PMU_CNTR_SIZE(model))39#define CCI_PMU_CNTR_BASE(model, idx) ((idx) * CCI_PMU_CNTR_SIZE(model))
40#define CCI_PMU_CNTR_MASK ((1ULL << 32) -1)40#define CCI_PMU_CNTR_MASK ((1ULL << 32) - 1)
41#define CCI_PMU_CNTR_LAST(cci_pmu) (cci_pmu->num_cntrs - 1)41#define CCI_PMU_CNTR_LAST(cci_pmu) (cci_pmu->num_cntrs - 1)
42 42
43#define CCI_PMU_MAX_HW_CNTRS(model) \43#define CCI_PMU_MAX_HW_CNTRS(model) \
@@ -806,7 +806,7 @@ static int pmu_get_event_idx(struct cci_pmu_hw_events *hw, struct perf_event *ev · 762 unmodified lines
806 return cci_pmu->model->get_event_idx(cci_pmu, hw, cci_event);806 return cci_pmu->model->get_event_idx(cci_pmu, hw, cci_event);
807 807
808 /* Generic code to find an unused idx from the mask */808 /* Generic code to find an unused idx from the mask */
809 for(idx = 0; idx <= CCI_PMU_CNTR_LAST(cci_pmu); idx++)809 for (idx = 0; idx <= CCI_PMU_CNTR_LAST(cci_pmu); idx++)
810 if (!test_and_set_bit(idx, hw->used_mask))810 if (!test_and_set_bit(idx, hw->used_mask))
811 return idx;811 return idx;
812 812
drivers/perf/arm-ccn.c+2−4
@@ -1211,7 +1211,7 @@ static int arm_ccn_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node) · 1210 unmodified lines
1211 perf_pmu_migrate_context(&dt->pmu, cpu, target);1211 perf_pmu_migrate_context(&dt->pmu, cpu, target);
1212 dt->cpu = target;1212 dt->cpu = target;
1213 if (ccn->irq)1213 if (ccn->irq)
1214 WARN_ON(irq_set_affinity_hint(ccn->irq, cpumask_of(dt->cpu)));1214 WARN_ON(irq_set_affinity(ccn->irq, cpumask_of(dt->cpu)));
1215 return 0;1215 return 0;
1216}1216}
1217 1217
@@ -1291,7 +1291,7 @@ static int arm_ccn_pmu_init(struct arm_ccn *ccn) · 73 unmodified lines
1291 1291
1292 /* Also make sure that the overflow interrupt is handled by this CPU */1292 /* Also make sure that the overflow interrupt is handled by this CPU */
1293 if (ccn->irq) {1293 if (ccn->irq) {
1294 err = irq_set_affinity_hint(ccn->irq, cpumask_of(ccn->dt.cpu));1294 err = irq_set_affinity(ccn->irq, cpumask_of(ccn->dt.cpu));
1295 if (err) {1295 if (err) {
1296 dev_err(ccn->dev, "Failed to set interrupt affinity!\n");1296 dev_err(ccn->dev, "Failed to set interrupt affinity!\n");
1297 goto error_set_affinity;1297 goto error_set_affinity;
@@ -1325,8 +1325,6 @@ static void arm_ccn_pmu_cleanup(struct arm_ccn *ccn) · 27 unmodified lines
1325 1325
1326 cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_CCN_ONLINE,1326 cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_CCN_ONLINE,
1327 &ccn->dt.node);1327 &ccn->dt.node);
1328 if (ccn->irq)
1329 irq_set_affinity_hint(ccn->irq, NULL);
1330 for (i = 0; i < ccn->num_xps; i++)1328 for (i = 0; i < ccn->num_xps; i++)
1331 writel(0, ccn->xp[i].base + CCN_XP_DT_CONTROL);1329 writel(0, ccn->xp[i].base + CCN_XP_DT_CONTROL);
1332 writel(0, ccn->dt.base + CCN_DT_PMCR);1330 writel(0, ccn->dt.base + CCN_DT_PMCR);
drivers/perf/arm-cmn.c+4−9
@@ -31,7 +31,7 @@ · 30 unmodified lines
31#define CMN_CI_CHILD_COUNT GENMASK_ULL(15, 0)31#define CMN_CI_CHILD_COUNT GENMASK_ULL(15, 0)
32#define CMN_CI_CHILD_PTR_OFFSET GENMASK_ULL(31, 16)32#define CMN_CI_CHILD_PTR_OFFSET GENMASK_ULL(31, 16)
33 33
34#define CMN_CHILD_NODE_ADDR GENMASK(27,0)34#define CMN_CHILD_NODE_ADDR GENMASK(27, 0)
35#define CMN_CHILD_NODE_EXTERNAL BIT(31)35#define CMN_CHILD_NODE_EXTERNAL BIT(31)
36 36
37#define CMN_ADDR_NODE_PTR GENMASK(27, 14)37#define CMN_ADDR_NODE_PTR GENMASK(27, 14)
@@ -1162,7 +1162,7 @@ static int arm_cmn_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node) · 1124 unmodified lines
1162 1162
1163 perf_pmu_migrate_context(&cmn->pmu, cpu, target);1163 perf_pmu_migrate_context(&cmn->pmu, cpu, target);
1164 for (i = 0; i < cmn->num_dtcs; i++)1164 for (i = 0; i < cmn->num_dtcs; i++)
1165 irq_set_affinity_hint(cmn->dtc[i].irq, cpumask_of(target));1165 irq_set_affinity(cmn->dtc[i].irq, cpumask_of(target));
1166 cmn->cpu = target;1166 cmn->cpu = target;
1167 return 0;1167 return 0;
1168}1168}
@@ -1212,7 +1212,7 @@ static int arm_cmn_init_irqs(struct arm_cmn *cmn) · 43 unmodified lines
1212 irq = cmn->dtc[i].irq;1212 irq = cmn->dtc[i].irq;
1213 for (j = i; j--; ) {1213 for (j = i; j--; ) {
1214 if (cmn->dtc[j].irq == irq) {1214 if (cmn->dtc[j].irq == irq) {
1215 cmn->dtc[j].irq_friend = j - i;1215 cmn->dtc[j].irq_friend = i - j;
1216 goto next;1216 goto next;
1217 }1217 }
1218 }1218 }
@@ -1222,7 +1222,7 @@ static int arm_cmn_init_irqs(struct arm_cmn *cmn) · 3 unmodified lines
1222 if (err)1222 if (err)
1223 return err;1223 return err;
1224 1224
1225 err = irq_set_affinity_hint(irq, cpumask_of(cmn->cpu));1225 err = irq_set_affinity(irq, cpumask_of(cmn->cpu));
1226 if (err)1226 if (err)
1227 return err;1227 return err;
1228 next:1228 next:
@@ -1568,16 +1568,11 @@ static int arm_cmn_probe(struct platform_device *pdev) · 339 unmodified lines
1568static int arm_cmn_remove(struct platform_device *pdev)1568static int arm_cmn_remove(struct platform_device *pdev)
1569{1569{
1570 struct arm_cmn *cmn = platform_get_drvdata(pdev);1570 struct arm_cmn *cmn = platform_get_drvdata(pdev);
1571 int i;
1572 1571
1573 writel_relaxed(0, cmn->dtc[0].base + CMN_DT_DTC_CTL);1572 writel_relaxed(0, cmn->dtc[0].base + CMN_DT_DTC_CTL);
1574 1573
1575 perf_pmu_unregister(&cmn->pmu);1574 perf_pmu_unregister(&cmn->pmu);
1576 cpuhp_state_remove_instance(arm_cmn_hp_state, &cmn->cpuhp_node);1575 cpuhp_state_remove_instance(arm_cmn_hp_state, &cmn->cpuhp_node);
1577
1578 for (i = 0; i < cmn->num_dtcs; i++)
1579 irq_set_affinity_hint(cmn->dtc[i].irq, NULL);
1580
1581 return 0;1576 return 0;
1582}1577}
1583 1578
drivers/perf/arm_dmc620_pmu.c+2−3
@@ -421,7 +421,7 @@ static struct dmc620_pmu_irq *__dmc620_pmu_get_irq(int irq_num) · 420 unmodified lines
421 if (ret)421 if (ret)
422 goto out_free_aff;422 goto out_free_aff;
423 423
424 ret = irq_set_affinity_hint(irq_num, cpumask_of(irq->cpu));424 ret = irq_set_affinity(irq_num, cpumask_of(irq->cpu));
425 if (ret)425 if (ret)
426 goto out_free_irq;426 goto out_free_irq;
427 427
@@ -475,7 +475,6 @@ static void dmc620_pmu_put_irq(struct dmc620_pmu *dmc620_pmu) · 47 unmodified lines
475 list_del(&irq->irqs_node);475 list_del(&irq->irqs_node);
476 mutex_unlock(&dmc620_pmu_irqs_lock);476 mutex_unlock(&dmc620_pmu_irqs_lock);
477 477
478 WARN_ON(irq_set_affinity_hint(irq->irq_num, NULL));
479 free_irq(irq->irq_num, irq);478 free_irq(irq->irq_num, irq);
480 cpuhp_state_remove_instance_nocalls(cpuhp_state_num, &irq->node);479 cpuhp_state_remove_instance_nocalls(cpuhp_state_num, &irq->node);
481 kfree(irq);480 kfree(irq);
@@ -622,7 +621,7 @@ static int dmc620_pmu_cpu_teardown(unsigned int cpu, · 140 unmodified lines
622 perf_pmu_migrate_context(&dmc620_pmu->pmu, irq->cpu, target);621 perf_pmu_migrate_context(&dmc620_pmu->pmu, irq->cpu, target);
623 mutex_unlock(&dmc620_pmu_irqs_lock);622 mutex_unlock(&dmc620_pmu_irqs_lock);
624 623
625 WARN_ON(irq_set_affinity_hint(irq->irq_num, cpumask_of(target)));624 WARN_ON(irq_set_affinity(irq->irq_num, cpumask_of(target)));
626 irq->cpu = target;625 irq->cpu = target;
627 626
628 return 0;627 return 0;
drivers/perf/arm_dsu_pmu.c+2−6
@@ -687,7 +687,7 @@ static void dsu_pmu_probe_pmu(struct dsu_pmu *dsu_pmu) · 686 unmodified lines
687static void dsu_pmu_set_active_cpu(int cpu, struct dsu_pmu *dsu_pmu)687static void dsu_pmu_set_active_cpu(int cpu, struct dsu_pmu *dsu_pmu)
688{688{
689 cpumask_set_cpu(cpu, &dsu_pmu->active_cpu);689 cpumask_set_cpu(cpu, &dsu_pmu->active_cpu);
690 if (irq_set_affinity_hint(dsu_pmu->irq, &dsu_pmu->active_cpu))690 if (irq_set_affinity(dsu_pmu->irq, &dsu_pmu->active_cpu))
691 pr_warn("Failed to set irq affinity to %d\n", cpu);691 pr_warn("Failed to set irq affinity to %d\n", cpu);
692}692}
693 693
@@ -769,7 +769,6 @@ static int dsu_pmu_device_probe(struct platform_device *pdev) · 75 unmodified lines
769 if (rc) {769 if (rc) {
770 cpuhp_state_remove_instance(dsu_pmu_cpuhp_state,770 cpuhp_state_remove_instance(dsu_pmu_cpuhp_state,
771 &dsu_pmu->cpuhp_node);771 &dsu_pmu->cpuhp_node);
772 irq_set_affinity_hint(dsu_pmu->irq, NULL);
773 }772 }
774 773
775 return rc;774 return rc;
@@ -781,7 +780,6 @@ static int dsu_pmu_device_remove(struct platform_device *pdev) · 5 unmodified lines
781 780
782 perf_pmu_unregister(&dsu_pmu->pmu);781 perf_pmu_unregister(&dsu_pmu->pmu);
783 cpuhp_state_remove_instance(dsu_pmu_cpuhp_state, &dsu_pmu->cpuhp_node);782 cpuhp_state_remove_instance(dsu_pmu_cpuhp_state, &dsu_pmu->cpuhp_node);
784 irq_set_affinity_hint(dsu_pmu->irq, NULL);
785 783
786 return 0;784 return 0;
787}785}
@@ -840,10 +838,8 @@ static int dsu_pmu_cpu_teardown(unsigned int cpu, struct hlist_node *node) · 52 unmodified lines
840 838
841 dst = dsu_pmu_get_online_cpu_any_but(dsu_pmu, cpu);839 dst = dsu_pmu_get_online_cpu_any_but(dsu_pmu, cpu);
842 /* If there are no active CPUs in the DSU, leave IRQ disabled */840 /* If there are no active CPUs in the DSU, leave IRQ disabled */
843 if (dst >= nr_cpu_ids) {841 if (dst >= nr_cpu_ids)
844 irq_set_affinity_hint(dsu_pmu->irq, NULL);
845 return 0;842 return 0;
846 }
847 843
848 perf_pmu_migrate_context(&dsu_pmu->pmu, cpu, dst);844 perf_pmu_migrate_context(&dsu_pmu->pmu, cpu, dst);
849 dsu_pmu_set_active_cpu(dst, dsu_pmu);845 dsu_pmu_set_active_cpu(dst, dsu_pmu);
drivers/perf/arm_pmu.c+6−10
@@ -563,14 +563,14 @@ static int armpmu_filter_match(struct perf_event *event) · 562 unmodified lines
563 return ret;563 return ret;
564}564}
565 565
566static ssize_t armpmu_cpumask_show(struct device *dev,566static ssize_t cpus_show(struct device *dev,
567 struct device_attribute *attr, char *buf)567 struct device_attribute *attr, char *buf)
568{568{
569 struct arm_pmu *armpmu = to_arm_pmu(dev_get_drvdata(dev));569 struct arm_pmu *armpmu = to_arm_pmu(dev_get_drvdata(dev));
570 return cpumap_print_to_pagebuf(true, buf, &armpmu->supported_cpus);570 return cpumap_print_to_pagebuf(true, buf, &armpmu->supported_cpus);
571}571}
572 572
573static DEVICE_ATTR(cpus, S_IRUGO, armpmu_cpumask_show, NULL);573static DEVICE_ATTR_RO(cpus);
574 574
575static struct attribute *armpmu_common_attrs[] = {575static struct attribute *armpmu_common_attrs[] = {
576 &dev_attr_cpus.attr,576 &dev_attr_cpus.attr,
@@ -644,11 +644,9 @@ int armpmu_request_irq(int irq, int cpu) · 67 unmodified lines
644 }644 }
645 645
646 irq_flags = IRQF_PERCPU |646 irq_flags = IRQF_PERCPU |
647 IRQF_NOBALANCING |647 IRQF_NOBALANCING | IRQF_NO_AUTOEN |
648 IRQF_NO_THREAD;648 IRQF_NO_THREAD;
649 649
650 irq_set_status_flags(irq, IRQ_NOAUTOEN);
651
652 err = request_nmi(irq, handler, irq_flags, "arm-pmu",650 err = request_nmi(irq, handler, irq_flags, "arm-pmu",
653 per_cpu_ptr(&cpu_armpmu, cpu));651 per_cpu_ptr(&cpu_armpmu, cpu));
654 652
@@ -670,7 +668,7 @@ int armpmu_request_irq(int irq, int cpu) · 15 unmodified lines
670 &cpu_armpmu);668 &cpu_armpmu);
671 irq_ops = &percpu_pmuirq_ops;669 irq_ops = &percpu_pmuirq_ops;
672 } else {670 } else {
673 has_nmi= true;671 has_nmi = true;
674 irq_ops = &percpu_pmunmi_ops;672 irq_ops = &percpu_pmunmi_ops;
675 }673 }
676 } else {674 } else {
@@ -869,10 +867,8 @@ static struct arm_pmu *__armpmu_alloc(gfp_t flags) · 192 unmodified lines
869 int cpu;867 int cpu;
870 868
871 pmu = kzalloc(sizeof(*pmu), flags);869 pmu = kzalloc(sizeof(*pmu), flags);
872 if (!pmu) {870 if (!pmu)
873 pr_info("failed to allocate PMU device!\n");
874 goto out;871 goto out;
875 }
876 872
877 pmu->hw_events = alloc_percpu_gfp(struct pmu_hw_events, flags);873 pmu->hw_events = alloc_percpu_gfp(struct pmu_hw_events, flags);
878 if (!pmu->hw_events) {874 if (!pmu->hw_events) {
drivers/perf/arm_smmuv3_pmu.c+15−20
@@ -277,7 +277,7 @@ static int smmu_pmu_apply_event_filter(struct smmu_pmu *smmu_pmu, · 276 unmodified lines
277 struct perf_event *event, int idx)277 struct perf_event *event, int idx)
278{278{
279 u32 span, sid;279 u32 span, sid;
280 unsigned int num_ctrs = smmu_pmu->num_counters;280 unsigned int cur_idx, num_ctrs = smmu_pmu->num_counters;
281 bool filter_en = !!get_filter_enable(event);281 bool filter_en = !!get_filter_enable(event);
282 282
283 span = filter_en ? get_filter_span(event) :283 span = filter_en ? get_filter_span(event) :
@@ -285,17 +285,19 @@ static int smmu_pmu_apply_event_filter(struct smmu_pmu *smmu_pmu, · 1 unmodified line
285 sid = filter_en ? get_filter_stream_id(event) :285 sid = filter_en ? get_filter_stream_id(event) :
286 SMMU_PMCG_DEFAULT_FILTER_SID;286 SMMU_PMCG_DEFAULT_FILTER_SID;
287 287
288 /* Support individual filter settings */288 cur_idx = find_first_bit(smmu_pmu->used_counters, num_ctrs);
289 if (!smmu_pmu->global_filter) {289 /*
290 * Per-counter filtering, or scheduling the first globally-filtered
291 * event into an empty PMU so idx == 0 and it works out equivalent.
292 */
293 if (!smmu_pmu->global_filter || cur_idx == num_ctrs) {
290 smmu_pmu_set_event_filter(event, idx, span, sid);294 smmu_pmu_set_event_filter(event, idx, span, sid);
291 return 0;295 return 0;
292 }296 }
293 297
294 /* Requested settings same as current global settings*/298 /* Otherwise, must match whatever's currently scheduled */
295 idx = find_first_bit(smmu_pmu->used_counters, num_ctrs);299 if (smmu_pmu_check_global_filter(smmu_pmu->events[cur_idx], event)) {
296 if (idx == num_ctrs ||300 smmu_pmu_set_evtyper(smmu_pmu, idx, get_event(event));
297 smmu_pmu_check_global_filter(smmu_pmu->events[idx], event)) {
298 smmu_pmu_set_event_filter(event, 0, span, sid);
299 return 0;301 return 0;
300 }302 }
301 303
@@ -509,11 +511,8 @@ static ssize_t smmu_pmu_event_show(struct device *dev, · 207 unmodified lines
509 return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id);511 return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id);
510}512}
511 513
512#define SMMU_EVENT_ATTR(name, config) \514#define SMMU_EVENT_ATTR(name, config) \
513 (&((struct perf_pmu_events_attr) { \515 PMU_EVENT_ATTR_ID(name, smmu_pmu_event_show, config)
514 .attr = __ATTR(name, 0444, smmu_pmu_event_show, NULL), \
515 .id = config, \
516 }).attr.attr)
517 516
518static struct attribute *smmu_pmu_events[] = {517static struct attribute *smmu_pmu_events[] = {
519 SMMU_EVENT_ATTR(cycles, 0),518 SMMU_EVENT_ATTR(cycles, 0),
@@ -628,7 +627,7 @@ static int smmu_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node) · 108 unmodified lines
628 627
629 perf_pmu_migrate_context(&smmu_pmu->pmu, cpu, target);628 perf_pmu_migrate_context(&smmu_pmu->pmu, cpu, target);
630 smmu_pmu->on_cpu = target;629 smmu_pmu->on_cpu = target;
631 WARN_ON(irq_set_affinity_hint(smmu_pmu->irq, cpumask_of(target)));630 WARN_ON(irq_set_affinity(smmu_pmu->irq, cpumask_of(target)));
632 631
633 return 0;632 return 0;
634}633}
@@ -839,15 +838,14 @@ static int smmu_pmu_probe(struct platform_device *pdev) · 204 unmodified lines
839 838
840 /* Pick one CPU to be the preferred one to use */839 /* Pick one CPU to be the preferred one to use */
841 smmu_pmu->on_cpu = raw_smp_processor_id();840 smmu_pmu->on_cpu = raw_smp_processor_id();
842 WARN_ON(irq_set_affinity_hint(smmu_pmu->irq,841 WARN_ON(irq_set_affinity(smmu_pmu->irq, cpumask_of(smmu_pmu->on_cpu)));
843 cpumask_of(smmu_pmu->on_cpu)));
844 842
845 err = cpuhp_state_add_instance_nocalls(cpuhp_state_num,843 err = cpuhp_state_add_instance_nocalls(cpuhp_state_num,
846 &smmu_pmu->node);844 &smmu_pmu->node);
847 if (err) {845 if (err) {
848 dev_err(dev, "Error %d registering hotplug, PMU @%pa\n",846 dev_err(dev, "Error %d registering hotplug, PMU @%pa\n",
849 err, &res_0->start);847 err, &res_0->start);
850 goto out_clear_affinity;848 return err;
851 }849 }
852 850
853 err = perf_pmu_register(&smmu_pmu->pmu, name, -1);851 err = perf_pmu_register(&smmu_pmu->pmu, name, -1);
@@ -866,8 +864,6 @@ static int smmu_pmu_probe(struct platform_device *pdev) · 12 unmodified lines
866 864
867out_unregister:865out_unregister:
868 cpuhp_state_remove_instance_nocalls(cpuhp_state_num, &smmu_pmu->node);866 cpuhp_state_remove_instance_nocalls(cpuhp_state_num, &smmu_pmu->node);
869out_clear_affinity:
870 irq_set_affinity_hint(smmu_pmu->irq, NULL);
871 return err;867 return err;
872}868}
873 869
@@ -877,7 +873,6 @@ static int smmu_pmu_remove(struct platform_device *pdev) · 3 unmodified lines
877 873
878 perf_pmu_unregister(&smmu_pmu->pmu);874 perf_pmu_unregister(&smmu_pmu->pmu);
879 cpuhp_state_remove_instance_nocalls(cpuhp_state_num, &smmu_pmu->node);875 cpuhp_state_remove_instance_nocalls(cpuhp_state_num, &smmu_pmu->node);
880 irq_set_affinity_hint(smmu_pmu->irq, NULL);
881 876
882 return 0;877 return 0;
883}878}
drivers/perf/arm_spe_pmu.c+4−8
@@ -231,15 +231,14 @@ static const struct attribute_group arm_spe_pmu_format_group = { · 230 unmodified lines
231 .attrs = arm_spe_pmu_formats_attr,231 .attrs = arm_spe_pmu_formats_attr,
232};232};
233 233
234static ssize_t arm_spe_pmu_get_attr_cpumask(struct device *dev,234static ssize_t cpumask_show(struct device *dev,
235 struct device_attribute *attr,235 struct device_attribute *attr, char *buf)
236 char *buf)
237{236{
238 struct arm_spe_pmu *spe_pmu = dev_get_drvdata(dev);237 struct arm_spe_pmu *spe_pmu = dev_get_drvdata(dev);
239 238
240 return cpumap_print_to_pagebuf(true, buf, &spe_pmu->supported_cpus);239 return cpumap_print_to_pagebuf(true, buf, &spe_pmu->supported_cpus);
241}240}
242static DEVICE_ATTR(cpumask, S_IRUGO, arm_spe_pmu_get_attr_cpumask, NULL);241static DEVICE_ATTR_RO(cpumask);
243 242
244static struct attribute *arm_spe_pmu_attrs[] = {243static struct attribute *arm_spe_pmu_attrs[] = {
245 &dev_attr_cpumask.attr,244 &dev_attr_cpumask.attr,
@@ -1044,7 +1043,6 @@ static void __arm_spe_pmu_dev_probe(void *info) · 798 unmodified lines
1044 spe_pmu->max_record_sz, spe_pmu->align, spe_pmu->features);1043 spe_pmu->max_record_sz, spe_pmu->align, spe_pmu->features);
1045 1044
1046 spe_pmu->features |= SPE_PMU_FEAT_DEV_PROBED;1045 spe_pmu->features |= SPE_PMU_FEAT_DEV_PROBED;
1047 return;
1048}1046}
1049 1047
1050static void __arm_spe_pmu_reset_local(void)1048static void __arm_spe_pmu_reset_local(void)
@@ -1190,10 +1188,8 @@ static int arm_spe_pmu_device_probe(struct platform_device *pdev) · 139 unmodified lines
1190 }1188 }
1191 1189
1192 spe_pmu = devm_kzalloc(dev, sizeof(*spe_pmu), GFP_KERNEL);1190 spe_pmu = devm_kzalloc(dev, sizeof(*spe_pmu), GFP_KERNEL);
1193 if (!spe_pmu) {1191 if (!spe_pmu)
1194 dev_err(dev, "failed to allocate spe_pmu\n");
1195 return -ENOMEM;1192 return -ENOMEM;
1196 }
1197 1193
1198 spe_pmu->handle = alloc_percpu(typeof(*spe_pmu->handle));1194 spe_pmu->handle = alloc_percpu(typeof(*spe_pmu->handle));
1199 if (!spe_pmu->handle)1195 if (!spe_pmu->handle)
drivers/perf/fsl_imx8_ddr_perf.c+8−10
@@ -222,11 +222,8 @@ ddr_pmu_event_show(struct device *dev, struct device_attribute *attr, · 221 unmodified lines
222 return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id);222 return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id);
223}223}
224 224
225#define IMX8_DDR_PMU_EVENT_ATTR(_name, _id) \225#define IMX8_DDR_PMU_EVENT_ATTR(_name, _id) \
226 (&((struct perf_pmu_events_attr[]) { \226 PMU_EVENT_ATTR_ID(_name, ddr_pmu_event_show, _id)
227 { .attr = __ATTR(_name, 0444, ddr_pmu_event_show, NULL),\
228 .id = _id, } \
229 })[0].attr.attr)
230 227
231static struct attribute *ddr_perf_events_attrs[] = {228static struct attribute *ddr_perf_events_attrs[] = {
232 IMX8_DDR_PMU_EVENT_ATTR(cycles, EVENT_CYCLES_ID),229 IMX8_DDR_PMU_EVENT_ATTR(cycles, EVENT_CYCLES_ID),
@@ -674,7 +671,7 @@ static int ddr_perf_offline_cpu(unsigned int cpu, struct hlist_node *node) · 441 unmodified lines
674 perf_pmu_migrate_context(&pmu->pmu, cpu, target);671 perf_pmu_migrate_context(&pmu->pmu, cpu, target);
675 pmu->cpu = target;672 pmu->cpu = target;
676 673
677 WARN_ON(irq_set_affinity_hint(pmu->irq, cpumask_of(pmu->cpu)));674 WARN_ON(irq_set_affinity(pmu->irq, cpumask_of(pmu->cpu)));
678 675
679 return 0;676 return 0;
680}677}
@@ -705,8 +702,10 @@ static int ddr_perf_probe(struct platform_device *pdev) · 24 unmodified lines
705 702
706 name = devm_kasprintf(&pdev->dev, GFP_KERNEL, DDR_PERF_DEV_NAME "%d",703 name = devm_kasprintf(&pdev->dev, GFP_KERNEL, DDR_PERF_DEV_NAME "%d",
707 num);704 num);
708 if (!name)705 if (!name) {
709 return -ENOMEM;706 ret = -ENOMEM;
707 goto cpuhp_state_err;
708 }
710 709
711 pmu->devtype_data = of_device_get_match_data(&pdev->dev);710 pmu->devtype_data = of_device_get_match_data(&pdev->dev);
712 711
@@ -749,7 +748,7 @@ static int ddr_perf_probe(struct platform_device *pdev) · 36 unmodified lines
749 }748 }
750 749
751 pmu->irq = irq;750 pmu->irq = irq;
752 ret = irq_set_affinity_hint(pmu->irq, cpumask_of(pmu->cpu));751 ret = irq_set_affinity(pmu->irq, cpumask_of(pmu->cpu));
753 if (ret) {752 if (ret) {
754 dev_err(pmu->dev, "Failed to set interrupt affinity!\n");753 dev_err(pmu->dev, "Failed to set interrupt affinity!\n");
755 goto ddr_perf_err;754 goto ddr_perf_err;
@@ -777,7 +776,6 @@ static int ddr_perf_remove(struct platform_device *pdev) · 21 unmodified lines
777 776
778 cpuhp_state_remove_instance_nocalls(pmu->cpuhp_state, &pmu->node);777 cpuhp_state_remove_instance_nocalls(pmu->cpuhp_state, &pmu->node);
779 cpuhp_remove_multi_state(pmu->cpuhp_state);778 cpuhp_remove_multi_state(pmu->cpuhp_state);
780 irq_set_affinity_hint(pmu->irq, NULL);
781 779
782 perf_pmu_unregister(&pmu->pmu);780 perf_pmu_unregister(&pmu->pmu);
783 781
drivers/perf/hisilicon/hisi_uncore_ddrc_pmu.c+1−4
@@ -2,7 +2,7 @@ · 1 unmodified line
2/*2/*
3 * HiSilicon SoC DDRC uncore Hardware event counters support3 * HiSilicon SoC DDRC uncore Hardware event counters support
4 *4 *
5 * Copyright (C) 2017 Hisilicon Limited5 * Copyright (C) 2017 HiSilicon Limited
6 * Author: Shaokun Zhang <zhangshaokun@hisilicon.com>6 * Author: Shaokun Zhang <zhangshaokun@hisilicon.com>
7 * Anurup M <anurup.m@huawei.com>7 * Anurup M <anurup.m@huawei.com>
8 *8 *
@@ -537,7 +537,6 @@ static int hisi_ddrc_pmu_probe(struct platform_device *pdev) · 528 unmodified lines
537 dev_err(ddrc_pmu->dev, "DDRC PMU register failed!\n");537 dev_err(ddrc_pmu->dev, "DDRC PMU register failed!\n");
538 cpuhp_state_remove_instance_nocalls(538 cpuhp_state_remove_instance_nocalls(
539 CPUHP_AP_PERF_ARM_HISI_DDRC_ONLINE, &ddrc_pmu->node);539 CPUHP_AP_PERF_ARM_HISI_DDRC_ONLINE, &ddrc_pmu->node);
540 irq_set_affinity_hint(ddrc_pmu->irq, NULL);
541 }540 }
542 541
543 return ret;542 return ret;
@@ -550,8 +549,6 @@ static int hisi_ddrc_pmu_remove(struct platform_device *pdev) · 6 unmodified lines
550 perf_pmu_unregister(&ddrc_pmu->pmu);549 perf_pmu_unregister(&ddrc_pmu->pmu);
551 cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_DDRC_ONLINE,550 cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_DDRC_ONLINE,
552 &ddrc_pmu->node);551 &ddrc_pmu->node);
553 irq_set_affinity_hint(ddrc_pmu->irq, NULL);
554
555 return 0;552 return 0;
556}553}
557 554
drivers/perf/hisilicon/hisi_uncore_hha_pmu.c+3−6
@@ -2,7 +2,7 @@ · 1 unmodified line
2/*2/*
3 * HiSilicon SoC HHA uncore Hardware event counters support3 * HiSilicon SoC HHA uncore Hardware event counters support
4 *4 *
5 * Copyright (C) 2017 Hisilicon Limited5 * Copyright (C) 2017 HiSilicon Limited
6 * Author: Shaokun Zhang <zhangshaokun@hisilicon.com>6 * Author: Shaokun Zhang <zhangshaokun@hisilicon.com>
7 * Anurup M <anurup.m@huawei.com>7 * Anurup M <anurup.m@huawei.com>
8 *8 *
@@ -90,7 +90,7 @@ static void hisi_hha_pmu_config_ds(struct perf_event *event) · 81 unmodified lines
90 90
91 val = readl(hha_pmu->base + HHA_DATSRC_CTRL);91 val = readl(hha_pmu->base + HHA_DATSRC_CTRL);
92 val |= HHA_DATSRC_SKT_EN;92 val |= HHA_DATSRC_SKT_EN;
93 writel(ds_skt, hha_pmu->base + HHA_DATSRC_CTRL);93 writel(val, hha_pmu->base + HHA_DATSRC_CTRL);
94 }94 }
95}95}
96 96
@@ -104,7 +104,7 @@ static void hisi_hha_pmu_clear_ds(struct perf_event *event) · 7 unmodified lines
104 104
105 val = readl(hha_pmu->base + HHA_DATSRC_CTRL);105 val = readl(hha_pmu->base + HHA_DATSRC_CTRL);
106 val &= ~HHA_DATSRC_SKT_EN;106 val &= ~HHA_DATSRC_SKT_EN;
107 writel(ds_skt, hha_pmu->base + HHA_DATSRC_CTRL);107 writel(val, hha_pmu->base + HHA_DATSRC_CTRL);
108 }108 }
109}109}
110 110
@@ -540,7 +540,6 @@ static int hisi_hha_pmu_probe(struct platform_device *pdev) · 429 unmodified lines
540 dev_err(hha_pmu->dev, "HHA PMU register failed!\n");540 dev_err(hha_pmu->dev, "HHA PMU register failed!\n");
541 cpuhp_state_remove_instance_nocalls(541 cpuhp_state_remove_instance_nocalls(
542 CPUHP_AP_PERF_ARM_HISI_HHA_ONLINE, &hha_pmu->node);542 CPUHP_AP_PERF_ARM_HISI_HHA_ONLINE, &hha_pmu->node);
543 irq_set_affinity_hint(hha_pmu->irq, NULL);
544 }543 }
545 544
546 return ret;545 return ret;
@@ -553,8 +552,6 @@ static int hisi_hha_pmu_remove(struct platform_device *pdev) · 6 unmodified lines
553 perf_pmu_unregister(&hha_pmu->pmu);552 perf_pmu_unregister(&hha_pmu->pmu);
554 cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_HHA_ONLINE,553 cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_HHA_ONLINE,
555 &hha_pmu->node);554 &hha_pmu->node);
556 irq_set_affinity_hint(hha_pmu->irq, NULL);
557
558 return 0;555 return 0;
559}556}
560 557
drivers/perf/hisilicon/hisi_uncore_l3c_pmu.c+1−4
@@ -2,7 +2,7 @@ · 1 unmodified line
2/*2/*
3 * HiSilicon SoC L3C uncore Hardware event counters support3 * HiSilicon SoC L3C uncore Hardware event counters support
4 *4 *
5 * Copyright (C) 2017 Hisilicon Limited5 * Copyright (C) 2017 HiSilicon Limited
6 * Author: Anurup M <anurup.m@huawei.com>6 * Author: Anurup M <anurup.m@huawei.com>
7 * Shaokun Zhang <zhangshaokun@hisilicon.com>7 * Shaokun Zhang <zhangshaokun@hisilicon.com>
8 *8 *
@@ -578,7 +578,6 @@ static int hisi_l3c_pmu_probe(struct platform_device *pdev) · 569 unmodified lines
578 dev_err(l3c_pmu->dev, "L3C PMU register failed!\n");578 dev_err(l3c_pmu->dev, "L3C PMU register failed!\n");
579 cpuhp_state_remove_instance_nocalls(579 cpuhp_state_remove_instance_nocalls(
580 CPUHP_AP_PERF_ARM_HISI_L3_ONLINE, &l3c_pmu->node);580 CPUHP_AP_PERF_ARM_HISI_L3_ONLINE, &l3c_pmu->node);
581 irq_set_affinity_hint(l3c_pmu->irq, NULL);
582 }581 }
583 582
584 return ret;583 return ret;
@@ -591,8 +590,6 @@ static int hisi_l3c_pmu_remove(struct platform_device *pdev) · 6 unmodified lines
591 perf_pmu_unregister(&l3c_pmu->pmu);590 perf_pmu_unregister(&l3c_pmu->pmu);
592 cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_L3_ONLINE,591 cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_L3_ONLINE,
593 &l3c_pmu->node);592 &l3c_pmu->node);
594 irq_set_affinity_hint(l3c_pmu->irq, NULL);
595
596 return 0;593 return 0;
597}594}
598 595
drivers/perf/hisilicon/hisi_uncore_pa_pmu.c+1−4
@@ -333,7 +333,7 @@ static struct attribute *hisi_pa_pmu_identifier_attrs[] = { · 332 unmodified lines
333 NULL333 NULL
334};334};
335 335
336static struct attribute_group hisi_pa_pmu_identifier_group = {336static const struct attribute_group hisi_pa_pmu_identifier_group = {
337 .attrs = hisi_pa_pmu_identifier_attrs,337 .attrs = hisi_pa_pmu_identifier_attrs,
338};338};
339 339
@@ -436,7 +436,6 @@ static int hisi_pa_pmu_probe(struct platform_device *pdev) · 96 unmodified lines
436 dev_err(pa_pmu->dev, "PMU register failed, ret = %d\n", ret);436 dev_err(pa_pmu->dev, "PMU register failed, ret = %d\n", ret);
437 cpuhp_state_remove_instance(CPUHP_AP_PERF_ARM_HISI_PA_ONLINE,437 cpuhp_state_remove_instance(CPUHP_AP_PERF_ARM_HISI_PA_ONLINE,
438 &pa_pmu->node);438 &pa_pmu->node);
439 irq_set_affinity_hint(pa_pmu->irq, NULL);
440 return ret;439 return ret;
441 }440 }
442 441
@@ -451,8 +450,6 @@ static int hisi_pa_pmu_remove(struct platform_device *pdev) · 8 unmodified lines
451 perf_pmu_unregister(&pa_pmu->pmu);450 perf_pmu_unregister(&pa_pmu->pmu);
452 cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_PA_ONLINE,451 cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_PA_ONLINE,
453 &pa_pmu->node);452 &pa_pmu->node);
454 irq_set_affinity_hint(pa_pmu->irq, NULL);
455
456 return 0;453 return 0;
457}454}
458 455
drivers/perf/hisilicon/hisi_uncore_pmu.c+3−3
@@ -2,7 +2,7 @@ · 1 unmodified line
2/*2/*
3 * HiSilicon SoC Hardware event counters support3 * HiSilicon SoC Hardware event counters support
4 *4 *
5 * Copyright (C) 2017 Hisilicon Limited5 * Copyright (C) 2017 HiSilicon Limited
6 * Author: Anurup M <anurup.m@huawei.com>6 * Author: Anurup M <anurup.m@huawei.com>
7 * Shaokun Zhang <zhangshaokun@hisilicon.com>7 * Shaokun Zhang <zhangshaokun@hisilicon.com>
8 *8 *
@@ -488,7 +488,7 @@ int hisi_uncore_pmu_online_cpu(unsigned int cpu, struct hlist_node *node) · 479 unmodified lines
488 hisi_pmu->on_cpu = cpu;488 hisi_pmu->on_cpu = cpu;
489 489
490 /* Overflow interrupt also should use the same CPU */490 /* Overflow interrupt also should use the same CPU */
491 WARN_ON(irq_set_affinity_hint(hisi_pmu->irq, cpumask_of(cpu)));491 WARN_ON(irq_set_affinity(hisi_pmu->irq, cpumask_of(cpu)));
492 492
493 return 0;493 return 0;
494}494}
@@ -521,7 +521,7 @@ int hisi_uncore_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node) · 26 unmodified lines
521 perf_pmu_migrate_context(&hisi_pmu->pmu, cpu, target);521 perf_pmu_migrate_context(&hisi_pmu->pmu, cpu, target);
522 /* Use this CPU for event counting */522 /* Use this CPU for event counting */
523 hisi_pmu->on_cpu = target;523 hisi_pmu->on_cpu = target;
524 WARN_ON(irq_set_affinity_hint(hisi_pmu->irq, cpumask_of(target)));524 WARN_ON(irq_set_affinity(hisi_pmu->irq, cpumask_of(target)));
525 525
526 return 0;526 return 0;
527}527}
drivers/perf/hisilicon/hisi_uncore_pmu.h+1−1
@@ -2,7 +2,7 @@ · 1 unmodified line
2/*2/*
3 * HiSilicon SoC Hardware event counters support3 * HiSilicon SoC Hardware event counters support
4 *4 *
5 * Copyright (C) 2017 Hisilicon Limited5 * Copyright (C) 2017 HiSilicon Limited
6 * Author: Anurup M <anurup.m@huawei.com>6 * Author: Anurup M <anurup.m@huawei.com>
7 * Shaokun Zhang <zhangshaokun@hisilicon.com>7 * Shaokun Zhang <zhangshaokun@hisilicon.com>
8 *8 *
drivers/perf/hisilicon/hisi_uncore_sllc_pmu.c+2−5
@@ -2,7 +2,7 @@ · 1 unmodified line
2/*2/*
3 * HiSilicon SLLC uncore Hardware event counters support3 * HiSilicon SLLC uncore Hardware event counters support
4 *4 *
5 * Copyright (C) 2020 Hisilicon Limited5 * Copyright (C) 2020 HiSilicon Limited
6 * Author: Shaokun Zhang <zhangshaokun@hisilicon.com>6 * Author: Shaokun Zhang <zhangshaokun@hisilicon.com>
7 *7 *
8 * This code is based on the uncore PMUs like arm-cci and arm-ccn.8 * This code is based on the uncore PMUs like arm-cci and arm-ccn.
@@ -366,7 +366,7 @@ static struct attribute *hisi_sllc_pmu_identifier_attrs[] = { · 357 unmodified lines
366 NULL366 NULL
367};367};
368 368
369static struct attribute_group hisi_sllc_pmu_identifier_group = {369static const struct attribute_group hisi_sllc_pmu_identifier_group = {
370 .attrs = hisi_sllc_pmu_identifier_attrs,370 .attrs = hisi_sllc_pmu_identifier_attrs,
371};371};
372 372
@@ -465,7 +465,6 @@ static int hisi_sllc_pmu_probe(struct platform_device *pdev) · 92 unmodified lines
465 dev_err(sllc_pmu->dev, "PMU register failed, ret = %d\n", ret);465 dev_err(sllc_pmu->dev, "PMU register failed, ret = %d\n", ret);
466 cpuhp_state_remove_instance(CPUHP_AP_PERF_ARM_HISI_SLLC_ONLINE,466 cpuhp_state_remove_instance(CPUHP_AP_PERF_ARM_HISI_SLLC_ONLINE,
467 &sllc_pmu->node);467 &sllc_pmu->node);
468 irq_set_affinity_hint(sllc_pmu->irq, NULL);
469 return ret;468 return ret;
470 }469 }
471 470
@@ -481,8 +480,6 @@ static int hisi_sllc_pmu_remove(struct platform_device *pdev) · 9 unmodified lines
481 perf_pmu_unregister(&sllc_pmu->pmu);480 perf_pmu_unregister(&sllc_pmu->pmu);
482 cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_SLLC_ONLINE,481 cpuhp_state_remove_instance_nocalls(CPUHP_AP_PERF_ARM_HISI_SLLC_ONLINE,
483 &sllc_pmu->node);482 &sllc_pmu->node);
484 irq_set_affinity_hint(sllc_pmu->irq, NULL);
485
486 return 0;483 return 0;
487}484}
488 485
drivers/perf/qcom_l2_pmu.c+4−7
@@ -679,11 +679,8 @@ static ssize_t l2cache_pmu_event_show(struct device *dev, · 678 unmodified lines
679 return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id);679 return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id);
680}680}
681 681
682#define L2CACHE_EVENT_ATTR(_name, _id) \682#define L2CACHE_EVENT_ATTR(_name, _id) \
683 (&((struct perf_pmu_events_attr[]) { \683 PMU_EVENT_ATTR_ID(_name, l2cache_pmu_event_show, _id)
684 { .attr = __ATTR(_name, 0444, l2cache_pmu_event_show, NULL), \
685 .id = _id, } \
686 })[0].attr.attr)
687 684
688static struct attribute *l2_cache_pmu_events[] = {685static struct attribute *l2_cache_pmu_events[] = {
689 L2CACHE_EVENT_ATTR(cycles, L2_EVENT_CYCLES),686 L2CACHE_EVENT_ATTR(cycles, L2_EVENT_CYCLES),
@@ -869,14 +866,14 @@ static int l2_cache_pmu_probe_cluster(struct device *dev, void *data) · 179 unmodified lines
869 irq = platform_get_irq(sdev, 0);866 irq = platform_get_irq(sdev, 0);
870 if (irq < 0)867 if (irq < 0)
871 return irq;868 return irq;
872 irq_set_status_flags(irq, IRQ_NOAUTOEN);
873 cluster->irq = irq;869 cluster->irq = irq;
874 870
875 cluster->l2cache_pmu = l2cache_pmu;871 cluster->l2cache_pmu = l2cache_pmu;
876 cluster->on_cpu = -1;872 cluster->on_cpu = -1;
877 873
878 err = devm_request_irq(&pdev->dev, irq, l2_cache_handle_irq,874 err = devm_request_irq(&pdev->dev, irq, l2_cache_handle_irq,
879 IRQF_NOBALANCING | IRQF_NO_THREAD,875 IRQF_NOBALANCING | IRQF_NO_THREAD |
876 IRQF_NO_AUTOEN,
880 "l2-cache-pmu", cluster);877 "l2-cache-pmu", cluster);
881 if (err) {878 if (err) {
882 dev_err(&pdev->dev,879 dev_err(&pdev->dev,
drivers/perf/qcom_l3_pmu.c+5−10
@@ -647,10 +647,7 @@ static ssize_t l3cache_pmu_event_show(struct device *dev, · 646 unmodified lines
647}647}
648 648
649#define L3CACHE_EVENT_ATTR(_name, _id) \649#define L3CACHE_EVENT_ATTR(_name, _id) \
650 (&((struct perf_pmu_events_attr[]) { \650 PMU_EVENT_ATTR_ID(_name, l3cache_pmu_event_show, _id)
651 { .attr = __ATTR(_name, 0444, l3cache_pmu_event_show, NULL), \
652 .id = _id, } \
653 })[0].attr.attr)
654 651
655static struct attribute *qcom_l3_cache_pmu_events[] = {652static struct attribute *qcom_l3_cache_pmu_events[] = {
656 L3CACHE_EVENT_ATTR(cycles, L3_EVENT_CYCLES),653 L3CACHE_EVENT_ATTR(cycles, L3_EVENT_CYCLES),
@@ -670,15 +667,15 @@ static const struct attribute_group qcom_l3_cache_pmu_events_group = { · 13 unmodified lines
670 667
671/* cpumask */668/* cpumask */
672 669
673static ssize_t qcom_l3_cache_pmu_cpumask_show(struct device *dev,670static ssize_t cpumask_show(struct device *dev,
674 struct device_attribute *attr, char *buf)671 struct device_attribute *attr, char *buf)
675{672{
676 struct l3cache_pmu *l3pmu = to_l3cache_pmu(dev_get_drvdata(dev));673 struct l3cache_pmu *l3pmu = to_l3cache_pmu(dev_get_drvdata(dev));
677 674
678 return cpumap_print_to_pagebuf(true, buf, &l3pmu->cpumask);675 return cpumap_print_to_pagebuf(true, buf, &l3pmu->cpumask);
679}676}
680 677
681static DEVICE_ATTR(cpumask, 0444, qcom_l3_cache_pmu_cpumask_show, NULL);678static DEVICE_ATTR_RO(cpumask);
682 679
683static struct attribute *qcom_l3_cache_pmu_cpumask_attrs[] = {680static struct attribute *qcom_l3_cache_pmu_cpumask_attrs[] = {
684 &dev_attr_cpumask.attr,681 &dev_attr_cpumask.attr,
@@ -767,10 +764,8 @@ static int qcom_l3_cache_pmu_probe(struct platform_device *pdev) · 82 unmodified lines
767 764
768 memrc = platform_get_resource(pdev, IORESOURCE_MEM, 0);765 memrc = platform_get_resource(pdev, IORESOURCE_MEM, 0);
769 l3pmu->regs = devm_ioremap_resource(&pdev->dev, memrc);766 l3pmu->regs = devm_ioremap_resource(&pdev->dev, memrc);
770 if (IS_ERR(l3pmu->regs)) {767 if (IS_ERR(l3pmu->regs))
771 dev_err(&pdev->dev, "Can't map PMU @%pa\n", &memrc->start);
772 return PTR_ERR(l3pmu->regs);768 return PTR_ERR(l3pmu->regs);
773 }
774 769
775 qcom_l3_cache__init(l3pmu);770 qcom_l3_cache__init(l3pmu);
776 771
drivers/perf/thunderx2_pmu.c+1−3
@@ -817,10 +817,8 @@ static struct tx2_uncore_pmu *tx2_uncore_pmu_init_dev(struct device *dev, · 816 unmodified lines
817 }817 }
818 818
819 base = devm_ioremap_resource(dev, &res);819 base = devm_ioremap_resource(dev, &res);
820 if (IS_ERR(base)) {820 if (IS_ERR(base))
821 dev_err(dev, "PMU type %d: Fail to map resource\n", type);
822 return NULL;821 return NULL;
823 }
824 822
825 tx2_pmu = devm_kzalloc(dev, sizeof(*tx2_pmu), GFP_KERNEL);823 tx2_pmu = devm_kzalloc(dev, sizeof(*tx2_pmu), GFP_KERNEL);
826 if (!tx2_pmu)824 if (!tx2_pmu)
drivers/perf/xgene_pmu.c+7−10
@@ -278,17 +278,14 @@ static const struct attribute_group mc_pmu_v3_format_attr_group = { · 277 unmodified lines
278static ssize_t xgene_pmu_event_show(struct device *dev,278static ssize_t xgene_pmu_event_show(struct device *dev,
279 struct device_attribute *attr, char *buf)279 struct device_attribute *attr, char *buf)
280{280{
281 struct dev_ext_attribute *eattr;281 struct perf_pmu_events_attr *pmu_attr =
282 container_of(attr, struct perf_pmu_events_attr, attr);
282 283
283 eattr = container_of(attr, struct dev_ext_attribute, attr);284 return sysfs_emit(buf, "config=0x%llx\n", pmu_attr->id);
284 return sysfs_emit(buf, "config=0x%lx\n", (unsigned long) eattr->var);
285}285}
286 286
287#define XGENE_PMU_EVENT_ATTR(_name, _config) \287#define XGENE_PMU_EVENT_ATTR(_name, _config) \
288 (&((struct dev_ext_attribute[]) { \288 PMU_EVENT_ATTR_ID(_name, xgene_pmu_event_show, _config)
289 { .attr = __ATTR(_name, S_IRUGO, xgene_pmu_event_show, NULL), \
290 .var = (void *) _config, } \
291 })[0].attr.attr)
292 289
293static struct attribute *l3c_pmu_events_attrs[] = {290static struct attribute *l3c_pmu_events_attrs[] = {
294 XGENE_PMU_EVENT_ATTR(cycle-count, 0x00),291 XGENE_PMU_EVENT_ATTR(cycle-count, 0x00),
@@ -604,15 +601,15 @@ static const struct attribute_group mc_pmu_v3_events_attr_group = { · 309 unmodified lines
604/*601/*
605 * sysfs cpumask attributes602 * sysfs cpumask attributes
606 */603 */
607static ssize_t xgene_pmu_cpumask_show(struct device *dev,604static ssize_t cpumask_show(struct device *dev,
608 struct device_attribute *attr, char *buf)605 struct device_attribute *attr, char *buf)
609{606{
610 struct xgene_pmu_dev *pmu_dev = to_pmu_dev(dev_get_drvdata(dev));607 struct xgene_pmu_dev *pmu_dev = to_pmu_dev(dev_get_drvdata(dev));
611 608
612 return cpumap_print_to_pagebuf(true, buf, &pmu_dev->parent->cpu);609 return cpumap_print_to_pagebuf(true, buf, &pmu_dev->parent->cpu);
613}610}
614 611
615static DEVICE_ATTR(cpumask, S_IRUGO, xgene_pmu_cpumask_show, NULL);612static DEVICE_ATTR_RO(cpumask);
616 613
617static struct attribute *xgene_pmu_cpumask_attrs[] = {614static struct attribute *xgene_pmu_cpumask_attrs[] = {
618 &dev_attr_cpumask.attr,615 &dev_attr_cpumask.attr,
include/linux/arm-smccc.h+86−2
@@ -63,6 +63,9 @@ · 62 unmodified lines
63#define ARM_SMCCC_VERSION_1_0 0x1000063#define ARM_SMCCC_VERSION_1_0 0x10000
64#define ARM_SMCCC_VERSION_1_1 0x1000164#define ARM_SMCCC_VERSION_1_1 0x10001
65#define ARM_SMCCC_VERSION_1_2 0x1000265#define ARM_SMCCC_VERSION_1_2 0x10002
66#define ARM_SMCCC_VERSION_1_3 0x10003
67
68#define ARM_SMCCC_1_3_SVE_HINT 0x10000
66 69
67#define ARM_SMCCC_VERSION_FUNC_ID \70#define ARM_SMCCC_VERSION_FUNC_ID \
68 ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \71 ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \
@@ -216,6 +219,8 @@ u32 arm_smccc_get_version(void); · 147 unmodified lines
216 219
217void __init arm_smccc_version_init(u32 version, enum arm_smccc_conduit conduit);220void __init arm_smccc_version_init(u32 version, enum arm_smccc_conduit conduit);
218 221
222extern u64 smccc_has_sve_hint;
223
219/**224/**
220 * struct arm_smccc_res - Result from SMC/HVC call225 * struct arm_smccc_res - Result from SMC/HVC call
221 * @a0-a3 result values from registers 0 to 3226 * @a0-a3 result values from registers 0 to 3
@@ -227,6 +232,61 @@ struct arm_smccc_res { · 5 unmodified lines
227 unsigned long a3;232 unsigned long a3;
228};233};
229 234
235#ifdef CONFIG_ARM64
236/**
237 * struct arm_smccc_1_2_regs - Arguments for or Results from SMC/HVC call
238 * @a0-a17 argument values from registers 0 to 17
239 */
240struct arm_smccc_1_2_regs {
241 unsigned long a0;
242 unsigned long a1;
243 unsigned long a2;
244 unsigned long a3;
245 unsigned long a4;
246 unsigned long a5;
247 unsigned long a6;
248 unsigned long a7;
249 unsigned long a8;
250 unsigned long a9;
251 unsigned long a10;
252 unsigned long a11;
253 unsigned long a12;
254 unsigned long a13;
255 unsigned long a14;
256 unsigned long a15;
257 unsigned long a16;
258 unsigned long a17;
259};
260
261/**
262 * arm_smccc_1_2_hvc() - make HVC calls
263 * @args: arguments passed via struct arm_smccc_1_2_regs
264 * @res: result values via struct arm_smccc_1_2_regs
265 *
266 * This function is used to make HVC calls following SMC Calling Convention
267 * v1.2 or above. The content of the supplied param are copied from the
268 * structure to registers prior to the HVC instruction. The return values
269 * are updated with the content from registers on return from the HVC
270 * instruction.
271 */
272asmlinkage void arm_smccc_1_2_hvc(const struct arm_smccc_1_2_regs *args,
273 struct arm_smccc_1_2_regs *res);
274
275/**
276 * arm_smccc_1_2_smc() - make SMC calls
277 * @args: arguments passed via struct arm_smccc_1_2_regs
278 * @res: result values via struct arm_smccc_1_2_regs
279 *
280 * This function is used to make SMC calls following SMC Calling Convention
281 * v1.2 or above. The content of the supplied param are copied from the
282 * structure to registers prior to the SMC instruction. The return values
283 * are updated with the content from registers on return from the SMC
284 * instruction.
285 */
286asmlinkage void arm_smccc_1_2_smc(const struct arm_smccc_1_2_regs *args,
287 struct arm_smccc_1_2_regs *res);
288#endif
289
230/**290/**
231 * struct arm_smccc_quirk - Contains quirk information291 * struct arm_smccc_quirk - Contains quirk information
232 * @id: quirk identification292 * @id: quirk identification
@@ -240,6 +300,15 @@ struct arm_smccc_quirk { · 7 unmodified lines
240 } state;300 } state;
241};301};
242 302
303/**
304 * __arm_smccc_sve_check() - Set the SVE hint bit when doing SMC calls
305 *
306 * Sets the SMCCC hint bit to indicate if there is live state in the SVE
307 * registers, this modifies x0 in place and should never be called from C
308 * code.
309 */
310asmlinkage unsigned long __arm_smccc_sve_check(unsigned long x0);
311
243/**312/**
244 * __arm_smccc_smc() - make SMC calls313 * __arm_smccc_smc() - make SMC calls
245 * @a0-a7: arguments passed in registers 0 to 7314 * @a0-a7: arguments passed in registers 0 to 7
@@ -297,6 +366,20 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1, · 51 unmodified lines
297 366
298#endif367#endif
299 368
369/* nVHE hypervisor doesn't have a current thread so needs separate checks */
370#if defined(CONFIG_ARM64_SVE) && !defined(__KVM_NVHE_HYPERVISOR__)
371
372#define SMCCC_SVE_CHECK ALTERNATIVE("nop \n", "bl __arm_smccc_sve_check \n", \
373 ARM64_SVE)
374#define smccc_sve_clobbers "x16", "x30", "cc",
375
376#else
377
378#define SMCCC_SVE_CHECK
379#define smccc_sve_clobbers
380
381#endif
382
300#define ___count_args(_0, _1, _2, _3, _4, _5, _6, _7, _8, x, ...) x383#define ___count_args(_0, _1, _2, _3, _4, _5, _6, _7, _8, x, ...) x
301 384
302#define __count_args(...) \385#define __count_args(...) \
@@ -364,7 +447,7 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1, · 61 unmodified lines
364 447
365#define ___constraints(count) \448#define ___constraints(count) \
366 : __constraint_read_ ## count \449 : __constraint_read_ ## count \
367 : "memory"450 : smccc_sve_clobbers "memory"
368#define __constraints(count) ___constraints(count)451#define __constraints(count) ___constraints(count)
369 452
370/*453/*
@@ -379,7 +462,8 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1, · 8 unmodified lines
379 register unsigned long r2 asm("r2"); \462 register unsigned long r2 asm("r2"); \
380 register unsigned long r3 asm("r3"); \463 register unsigned long r3 asm("r3"); \
381 __declare_args(__count_args(__VA_ARGS__), __VA_ARGS__); \464 __declare_args(__count_args(__VA_ARGS__), __VA_ARGS__); \
382 asm volatile(inst "\n" : \465 asm volatile(SMCCC_SVE_CHECK \
466 inst "\n" : \
383 "=r" (r0), "=r" (r1), "=r" (r2), "=r" (r3) \467 "=r" (r0), "=r" (r1), "=r" (r2), "=r" (r3) \
384 __constraints(__count_args(__VA_ARGS__))); \468 __constraints(__count_args(__VA_ARGS__))); \
385 if (___res) \469 if (___res) \
include/linux/gfp.h+15−3
@@ -53,8 +53,10 @@ struct vm_area_struct; · 52 unmodified lines
53#define ___GFP_HARDWALL 0x100000u53#define ___GFP_HARDWALL 0x100000u
54#define ___GFP_THISNODE 0x200000u54#define ___GFP_THISNODE 0x200000u
55#define ___GFP_ACCOUNT 0x400000u55#define ___GFP_ACCOUNT 0x400000u
56#define ___GFP_ZEROTAGS 0x800000u
57#define ___GFP_SKIP_KASAN_POISON 0x1000000u
56#ifdef CONFIG_LOCKDEP58#ifdef CONFIG_LOCKDEP
57#define ___GFP_NOLOCKDEP 0x800000u59#define ___GFP_NOLOCKDEP 0x2000000u
58#else60#else
59#define ___GFP_NOLOCKDEP 061#define ___GFP_NOLOCKDEP 0
60#endif62#endif
@@ -229,16 +231,25 @@ struct vm_area_struct; · 168 unmodified lines
229 * %__GFP_COMP address compound page metadata.231 * %__GFP_COMP address compound page metadata.
230 *232 *
231 * %__GFP_ZERO returns a zeroed page on success.233 * %__GFP_ZERO returns a zeroed page on success.
234 *
235 * %__GFP_ZEROTAGS returns a page with zeroed memory tags on success, if
236 * __GFP_ZERO is set.
237 *
238 * %__GFP_SKIP_KASAN_POISON returns a page which does not need to be poisoned
239 * on deallocation. Typically used for userspace pages. Currently only has an
240 * effect in HW tags mode.
232 */241 */
233#define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN)242#define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN)
234#define __GFP_COMP ((__force gfp_t)___GFP_COMP)243#define __GFP_COMP ((__force gfp_t)___GFP_COMP)
235#define __GFP_ZERO ((__force gfp_t)___GFP_ZERO)244#define __GFP_ZERO ((__force gfp_t)___GFP_ZERO)
245#define __GFP_ZEROTAGS ((__force gfp_t)___GFP_ZEROTAGS)
246#define __GFP_SKIP_KASAN_POISON ((__force gfp_t)___GFP_SKIP_KASAN_POISON)
236 247
237/* Disable lockdep for GFP context tracking */248/* Disable lockdep for GFP context tracking */
238#define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)249#define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)
239 250
240/* Room for N __GFP_FOO bits */251/* Room for N __GFP_FOO bits */
241#define __GFP_BITS_SHIFT (23 + IS_ENABLED(CONFIG_LOCKDEP))252#define __GFP_BITS_SHIFT (25 + IS_ENABLED(CONFIG_LOCKDEP))
242#define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))253#define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
243 254
244/**255/**
@@ -319,7 +330,8 @@ struct vm_area_struct; · 74 unmodified lines
319#define GFP_DMA __GFP_DMA330#define GFP_DMA __GFP_DMA
320#define GFP_DMA32 __GFP_DMA32331#define GFP_DMA32 __GFP_DMA32
321#define GFP_HIGHUSER (GFP_USER | __GFP_HIGHMEM)332#define GFP_HIGHUSER (GFP_USER | __GFP_HIGHMEM)
322#define GFP_HIGHUSER_MOVABLE (GFP_HIGHUSER | __GFP_MOVABLE)333#define GFP_HIGHUSER_MOVABLE (GFP_HIGHUSER | __GFP_MOVABLE | \
334 __GFP_SKIP_KASAN_POISON)
323#define GFP_TRANSHUGE_LIGHT ((GFP_HIGHUSER_MOVABLE | __GFP_COMP | \335#define GFP_TRANSHUGE_LIGHT ((GFP_HIGHUSER_MOVABLE | __GFP_COMP | \
324 __GFP_NOMEMALLOC | __GFP_NOWARN) & ~__GFP_RECLAIM)336 __GFP_NOMEMALLOC | __GFP_NOWARN) & ~__GFP_RECLAIM)
325#define GFP_TRANSHUGE (GFP_TRANSHUGE_LIGHT | __GFP_DIRECT_RECLAIM)337#define GFP_TRANSHUGE (GFP_TRANSHUGE_LIGHT | __GFP_DIRECT_RECLAIM)
include/linux/highmem.h+16−27
@@ -152,28 +152,24 @@ static inline void clear_user_highpage(struct page *page, unsigned long vaddr) · 151 unmodified lines
152}152}
153#endif153#endif
154 154
155#ifndef __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE155#ifndef __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE
156/**156/**
157 * __alloc_zeroed_user_highpage - Allocate a zeroed HIGHMEM page for a VMA with caller-specified movable GFP flags157 * alloc_zeroed_user_highpage_movable - Allocate a zeroed HIGHMEM page for a VMA that the caller knows can move
158 * @movableflags: The GFP flags related to the pages future ability to move like __GFP_MOVABLE
159 * @vma: The VMA the page is to be allocated for158 * @vma: The VMA the page is to be allocated for
160 * @vaddr: The virtual address the page will be inserted into159 * @vaddr: The virtual address the page will be inserted into
161 *160 *
162 * This function will allocate a page for a VMA but the caller is expected161 * This function will allocate a page for a VMA that the caller knows will
163 * to specify via movableflags whether the page will be movable in the162 * be able to migrate in the future using move_pages() or reclaimed
164 * future or not
165 *163 *
166 * An architecture may override this function by defining164 * An architecture may override this function by defining
167 * __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE and providing their own165 * __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE and providing their own
168 * implementation.166 * implementation.
169 */167 */
170static inline struct page *168static inline struct page *
171__alloc_zeroed_user_highpage(gfp_t movableflags,169alloc_zeroed_user_highpage_movable(struct vm_area_struct *vma,
172 struct vm_area_struct *vma,170 unsigned long vaddr)
173 unsigned long vaddr)
174{171{
175 struct page *page = alloc_page_vma(GFP_HIGHUSER | movableflags,172 struct page *page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vaddr);
176 vma, vaddr);
177 173
178 if (page)174 if (page)
179 clear_user_highpage(page, vaddr);175 clear_user_highpage(page, vaddr);
@@ -182,21 +178,6 @@ __alloc_zeroed_user_highpage(gfp_t movableflags, · 2 unmodified lines
182}178}
183#endif179#endif
184 180
185/**
186 * alloc_zeroed_user_highpage_movable - Allocate a zeroed HIGHMEM page for a VMA that the caller knows can move
187 * @vma: The VMA the page is to be allocated for
188 * @vaddr: The virtual address the page will be inserted into
189 *
190 * This function will allocate a page for a VMA that the caller knows will
191 * be able to migrate in the future using move_pages() or reclaimed
192 */
193static inline struct page *
194alloc_zeroed_user_highpage_movable(struct vm_area_struct *vma,
195 unsigned long vaddr)
196{
197 return __alloc_zeroed_user_highpage(__GFP_MOVABLE, vma, vaddr);
198}
199
200static inline void clear_highpage(struct page *page)181static inline void clear_highpage(struct page *page)
201{182{
202 void *kaddr = kmap_atomic(page);183 void *kaddr = kmap_atomic(page);
@@ -204,6 +185,14 @@ static inline void clear_highpage(struct page *page) · 1 unmodified line
204 kunmap_atomic(kaddr);185 kunmap_atomic(kaddr);
205}186}
206 187
188#ifndef __HAVE_ARCH_TAG_CLEAR_HIGHPAGE
189
190static inline void tag_clear_highpage(struct page *page)
191{
192}
193
194#endif
195
207/*196/*
208 * If we pass in a base or tail page, we can zero up to PAGE_SIZE.197 * If we pass in a base or tail page, we can zero up to PAGE_SIZE.
209 * If we pass in a head page, we can zero up to the size of the compound page.198 * If we pass in a head page, we can zero up to the size of the compound page.
include/linux/interrupt.h+2−33
@@ -319,39 +319,8 @@ struct irq_affinity_desc { · 318 unmodified lines
319 319
320extern cpumask_var_t irq_default_affinity;320extern cpumask_var_t irq_default_affinity;
321 321
322/* Internal implementation. Use the helpers below */322extern int irq_set_affinity(unsigned int irq, const struct cpumask *cpumask);
323extern int __irq_set_affinity(unsigned int irq, const struct cpumask *cpumask,323extern int irq_force_affinity(unsigned int irq, const struct cpumask *cpumask);
324 bool force);
325
326/**
327 * irq_set_affinity - Set the irq affinity of a given irq
328 * @irq: Interrupt to set affinity
329 * @cpumask: cpumask
330 *
331 * Fails if cpumask does not contain an online CPU
332 */
333static inline int
334irq_set_affinity(unsigned int irq, const struct cpumask *cpumask)
335{
336 return __irq_set_affinity(irq, cpumask, false);
337}
338
339/**
340 * irq_force_affinity - Force the irq affinity of a given irq
341 * @irq: Interrupt to set affinity
342 * @cpumask: cpumask
343 *
344 * Same as irq_set_affinity, but without checking the mask against
345 * online cpus.
346 *
347 * Solely for low level cpu hotplug code, where we need to make per
348 * cpu interrupts affine before the cpu becomes online.
349 */
350static inline int
351irq_force_affinity(unsigned int irq, const struct cpumask *cpumask)
352{
353 return __irq_set_affinity(irq, cpumask, true);
354}
355 324
356extern int irq_can_set_affinity(unsigned int irq);325extern int irq_can_set_affinity(unsigned int irq);
357extern int irq_select_affinity(unsigned int irq);326extern int irq_select_affinity(unsigned int irq);
include/linux/kasan.h+38−26
@@ -2,6 +2,7 @@ · 1 unmodified line
2#ifndef _LINUX_KASAN_H2#ifndef _LINUX_KASAN_H
3#define _LINUX_KASAN_H3#define _LINUX_KASAN_H
4 4
5#include <linux/bug.h>
5#include <linux/static_key.h>6#include <linux/static_key.h>
6#include <linux/types.h>7#include <linux/types.h>
7 8
@@ -79,14 +80,6 @@ static inline void kasan_disable_current(void) {} · 71 unmodified lines
79 80
80#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */81#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
81 82
82#ifdef CONFIG_KASAN
83
84struct kasan_cache {
85 int alloc_meta_offset;
86 int free_meta_offset;
87 bool is_kmalloc;
88};
89
90#ifdef CONFIG_KASAN_HW_TAGS83#ifdef CONFIG_KASAN_HW_TAGS
91 84
92DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled);85DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled);
@@ -101,11 +94,14 @@ static inline bool kasan_has_integrated_init(void) · 8 unmodified lines
101 return kasan_enabled();94 return kasan_enabled();
102}95}
103 96
97void kasan_alloc_pages(struct page *page, unsigned int order, gfp_t flags);
98void kasan_free_pages(struct page *page, unsigned int order);
99
104#else /* CONFIG_KASAN_HW_TAGS */100#else /* CONFIG_KASAN_HW_TAGS */
105 101
106static inline bool kasan_enabled(void)102static inline bool kasan_enabled(void)
107{103{
108 return true;104 return IS_ENABLED(CONFIG_KASAN);
109}105}
110 106
111static inline bool kasan_has_integrated_init(void)107static inline bool kasan_has_integrated_init(void)
@@ -113,8 +109,30 @@ static inline bool kasan_has_integrated_init(void) · 1 unmodified line
113 return false;109 return false;
114}110}
115 111
112static __always_inline void kasan_alloc_pages(struct page *page,
113 unsigned int order, gfp_t flags)
114{
115 /* Only available for integrated init. */
116 BUILD_BUG();
117}
118
119static __always_inline void kasan_free_pages(struct page *page,
120 unsigned int order)
121{
122 /* Only available for integrated init. */
123 BUILD_BUG();
124}
125
116#endif /* CONFIG_KASAN_HW_TAGS */126#endif /* CONFIG_KASAN_HW_TAGS */
117 127
128#ifdef CONFIG_KASAN
129
130struct kasan_cache {
131 int alloc_meta_offset;
132 int free_meta_offset;
133 bool is_kmalloc;
134};
135
118slab_flags_t __kasan_never_merge(void);136slab_flags_t __kasan_never_merge(void);
119static __always_inline slab_flags_t kasan_never_merge(void)137static __always_inline slab_flags_t kasan_never_merge(void)
120{138{
@@ -130,20 +148,20 @@ static __always_inline void kasan_unpoison_range(const void *addr, size_t size) · 9 unmodified lines
130 __kasan_unpoison_range(addr, size);148 __kasan_unpoison_range(addr, size);
131}149}
132 150
133void __kasan_alloc_pages(struct page *page, unsigned int order, bool init);151void __kasan_poison_pages(struct page *page, unsigned int order, bool init);
134static __always_inline void kasan_alloc_pages(struct page *page,152static __always_inline void kasan_poison_pages(struct page *page,
135 unsigned int order, bool init)153 unsigned int order, bool init)
136{154{
137 if (kasan_enabled())155 if (kasan_enabled())
138 __kasan_alloc_pages(page, order, init);156 __kasan_poison_pages(page, order, init);
139}157}
140 158
141void __kasan_free_pages(struct page *page, unsigned int order, bool init);159void __kasan_unpoison_pages(struct page *page, unsigned int order, bool init);
142static __always_inline void kasan_free_pages(struct page *page,160static __always_inline void kasan_unpoison_pages(struct page *page,
143 unsigned int order, bool init)161 unsigned int order, bool init)
144{162{
145 if (kasan_enabled())163 if (kasan_enabled())
146 __kasan_free_pages(page, order, init);164 __kasan_unpoison_pages(page, order, init);
147}165}
148 166
149void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,167void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
@@ -285,21 +303,15 @@ void kasan_restore_multi_shot(bool enabled); · 135 unmodified lines
285 303
286#else /* CONFIG_KASAN */304#else /* CONFIG_KASAN */
287 305
288static inline bool kasan_enabled(void)
289{
290 return false;
291}
292static inline bool kasan_has_integrated_init(void)
293{
294 return false;
295}
296static inline slab_flags_t kasan_never_merge(void)306static inline slab_flags_t kasan_never_merge(void)
297{307{
298 return 0;308 return 0;
299}309}
300static inline void kasan_unpoison_range(const void *address, size_t size) {}310static inline void kasan_unpoison_range(const void *address, size_t size) {}
301static inline void kasan_alloc_pages(struct page *page, unsigned int order, bool init) {}311static inline void kasan_poison_pages(struct page *page, unsigned int order,
302static inline void kasan_free_pages(struct page *page, unsigned int order, bool init) {}312 bool init) {}
313static inline void kasan_unpoison_pages(struct page *page, unsigned int order,
314 bool init) {}
303static inline void kasan_cache_create(struct kmem_cache *cache,315static inline void kasan_cache_create(struct kmem_cache *cache,
304 unsigned int *size,316 unsigned int *size,
305 slab_flags_t *flags) {}317 slab_flags_t *flags) {}
include/linux/page-flags.h+9−0
@@ -137,6 +137,9 @@ enum pageflags { · 136 unmodified lines
137#endif137#endif
138#ifdef CONFIG_64BIT138#ifdef CONFIG_64BIT
139 PG_arch_2,139 PG_arch_2,
140#endif
141#ifdef CONFIG_KASAN_HW_TAGS
142 PG_skip_kasan_poison,
140#endif143#endif
141 __NR_PAGEFLAGS,144 __NR_PAGEFLAGS,
142 145
@@ -443,6 +446,12 @@ TESTCLEARFLAG(Young, young, PF_ANY) · 300 unmodified lines
443PAGEFLAG(Idle, idle, PF_ANY)446PAGEFLAG(Idle, idle, PF_ANY)
444#endif447#endif
445 448
449#ifdef CONFIG_KASAN_HW_TAGS
450PAGEFLAG(SkipKASanPoison, skip_kasan_poison, PF_HEAD)
451#else
452PAGEFLAG_FALSE(SkipKASanPoison)
453#endif
454
446/*455/*
447 * PageReported() is used to track reported free pages within the Buddy456 * PageReported() is used to track reported free pages within the Buddy
448 * allocator. We can use the non-atomic version of the test and set457 * allocator. We can use the non-atomic version of the test and set
include/linux/perf_event.h+6−0
@@ -1576,6 +1576,12 @@ static struct perf_pmu_events_attr _var = { \ · 1575 unmodified lines
1576 .event_str = _str, \1576 .event_str = _str, \
1577};1577};
1578 1578
1579#define PMU_EVENT_ATTR_ID(_name, _show, _id) \
1580 (&((struct perf_pmu_events_attr[]) { \
1581 { .attr = __ATTR(_name, 0444, _show, NULL), \
1582 .id = _id, } \
1583 })[0].attr.attr)
1584
1579#define PMU_FORMAT_ATTR(_name, _format) \1585#define PMU_FORMAT_ATTR(_name, _format) \
1580static ssize_t \1586static ssize_t \
1581_name##_show(struct device *dev, \1587_name##_show(struct device *dev, \
include/trace/events/mmflags.h+8−1
@@ -85,6 +85,12 @@ · 84 unmodified lines
85#define IF_HAVE_PG_ARCH_2(flag,string)85#define IF_HAVE_PG_ARCH_2(flag,string)
86#endif86#endif
87 87
88#ifdef CONFIG_KASAN_HW_TAGS
89#define IF_HAVE_PG_SKIP_KASAN_POISON(flag,string) ,{1UL << flag, string}
90#else
91#define IF_HAVE_PG_SKIP_KASAN_POISON(flag,string)
92#endif
93
88#define __def_pageflag_names \94#define __def_pageflag_names \
89 {1UL << PG_locked, "locked" }, \95 {1UL << PG_locked, "locked" }, \
90 {1UL << PG_waiters, "waiters" }, \96 {1UL << PG_waiters, "waiters" }, \
@@ -112,7 +118,8 @@ IF_HAVE_PG_UNCACHED(PG_uncached, "uncached" ) \ · 21 unmodified lines
112IF_HAVE_PG_HWPOISON(PG_hwpoison, "hwpoison" ) \118IF_HAVE_PG_HWPOISON(PG_hwpoison, "hwpoison" ) \
113IF_HAVE_PG_IDLE(PG_young, "young" ) \119IF_HAVE_PG_IDLE(PG_young, "young" ) \
114IF_HAVE_PG_IDLE(PG_idle, "idle" ) \120IF_HAVE_PG_IDLE(PG_idle, "idle" ) \
115IF_HAVE_PG_ARCH_2(PG_arch_2, "arch_2" )121IF_HAVE_PG_ARCH_2(PG_arch_2, "arch_2" ) \
122IF_HAVE_PG_SKIP_KASAN_POISON(PG_skip_kasan_poison, "skip_kasan_poison")
116 123
117#define show_page_flags(flags) \124#define show_page_flags(flags) \
118 (flags) ? __print_flags(flags, "|", \125 (flags) ? __print_flags(flags, "|", \
kernel/irq/manage.c+32−1
@@ -441,7 +441,8 @@ out_unlock: · 440 unmodified lines
441 return ret;441 return ret;
442}442}
443 443
444int __irq_set_affinity(unsigned int irq, const struct cpumask *mask, bool force)444static int __irq_set_affinity(unsigned int irq, const struct cpumask *mask,
445 bool force)
445{446{
446 struct irq_desc *desc = irq_to_desc(irq);447 struct irq_desc *desc = irq_to_desc(irq);
447 unsigned long flags;448 unsigned long flags;
@@ -456,6 +457,36 @@ int __irq_set_affinity(unsigned int irq, const struct cpumask *mask, bool force) · 8 unmodified lines
456 return ret;457 return ret;
457}458}
458 459
460/**
461 * irq_set_affinity - Set the irq affinity of a given irq
462 * @irq: Interrupt to set affinity
463 * @cpumask: cpumask
464 *
465 * Fails if cpumask does not contain an online CPU
466 */
467int irq_set_affinity(unsigned int irq, const struct cpumask *cpumask)
468{
469 return __irq_set_affinity(irq, cpumask, false);
470}
471EXPORT_SYMBOL_GPL(irq_set_affinity);
472
473/**
474 * irq_force_affinity - Force the irq affinity of a given irq
475 * @irq: Interrupt to set affinity
476 * @cpumask: cpumask
477 *
478 * Same as irq_set_affinity, but without checking the mask against
479 * online cpus.
480 *
481 * Solely for low level cpu hotplug code, where we need to make per
482 * cpu interrupts affine before the cpu becomes online.
483 */
484int irq_force_affinity(unsigned int irq, const struct cpumask *cpumask)
485{
486 return __irq_set_affinity(irq, cpumask, true);
487}
488EXPORT_SYMBOL_GPL(irq_force_affinity);
489
459int irq_set_affinity_hint(unsigned int irq, const struct cpumask *m)490int irq_set_affinity_hint(unsigned int irq, const struct cpumask *m)
460{491{
461 unsigned long flags;492 unsigned long flags;
mm/kasan/common.c+2−2
@@ -97,7 +97,7 @@ slab_flags_t __kasan_never_merge(void) · 96 unmodified lines
97 return 0;97 return 0;
98}98}
99 99
100void __kasan_alloc_pages(struct page *page, unsigned int order, bool init)100void __kasan_unpoison_pages(struct page *page, unsigned int order, bool init)
101{101{
102 u8 tag;102 u8 tag;
103 unsigned long i;103 unsigned long i;
@@ -111,7 +111,7 @@ void __kasan_alloc_pages(struct page *page, unsigned int order, bool init) · 7 unmodified lines
111 kasan_unpoison(page_address(page), PAGE_SIZE << order, init);111 kasan_unpoison(page_address(page), PAGE_SIZE << order, init);
112}112}
113 113
114void __kasan_free_pages(struct page *page, unsigned int order, bool init)114void __kasan_poison_pages(struct page *page, unsigned int order, bool init)
115{115{
116 if (likely(!PageHighMem(page)))116 if (likely(!PageHighMem(page)))
117 kasan_poison(page_address(page), PAGE_SIZE << order,117 kasan_poison(page_address(page), PAGE_SIZE << order,
mm/kasan/hw_tags.c+32−0
@@ -238,6 +238,38 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache, · 237 unmodified lines
238 return &alloc_meta->free_track[0];238 return &alloc_meta->free_track[0];
239}239}
240 240
241void kasan_alloc_pages(struct page *page, unsigned int order, gfp_t flags)
242{
243 /*
244 * This condition should match the one in post_alloc_hook() in
245 * page_alloc.c.
246 */
247 bool init = !want_init_on_free() && want_init_on_alloc(flags);
248
249 if (flags & __GFP_SKIP_KASAN_POISON)
250 SetPageSkipKASanPoison(page);
251
252 if (flags & __GFP_ZEROTAGS) {
253 int i;
254
255 for (i = 0; i != 1 << order; ++i)
256 tag_clear_highpage(page + i);
257 } else {
258 kasan_unpoison_pages(page, order, init);
259 }
260}
261
262void kasan_free_pages(struct page *page, unsigned int order)
263{
264 /*
265 * This condition should match the one in free_pages_prepare() in
266 * page_alloc.c.
267 */
268 bool init = want_init_on_free();
269
270 kasan_poison_pages(page, order, init);
271}
272
241#if IS_ENABLED(CONFIG_KASAN_KUNIT_TEST)273#if IS_ENABLED(CONFIG_KASAN_KUNIT_TEST)
242 274
243void kasan_set_tagging_report_once(bool state)275void kasan_set_tagging_report_once(bool state)
mm/kasan/sw_tags.c+7−0
@@ -207,3 +207,10 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache, · 206 unmodified lines
207 207
208 return &alloc_meta->free_track[i];208 return &alloc_meta->free_track[i];
209}209}
210
211void kasan_tag_mismatch(unsigned long addr, unsigned long access_info,
212 unsigned long ret_ip)
213{
214 kasan_report(addr, 1 << (access_info & 0xf), access_info & 0x10,
215 ret_ip);
216}
mm/mempool.c+4−2
@@ -106,7 +106,8 @@ static __always_inline void kasan_poison_element(mempool_t *pool, void *element) · 105 unmodified lines
106 if (pool->alloc == mempool_alloc_slab || pool->alloc == mempool_kmalloc)106 if (pool->alloc == mempool_alloc_slab || pool->alloc == mempool_kmalloc)
107 kasan_slab_free_mempool(element);107 kasan_slab_free_mempool(element);
108 else if (pool->alloc == mempool_alloc_pages)108 else if (pool->alloc == mempool_alloc_pages)
109 kasan_free_pages(element, (unsigned long)pool->pool_data, false);109 kasan_poison_pages(element, (unsigned long)pool->pool_data,
110 false);
110}111}
111 112
112static void kasan_unpoison_element(mempool_t *pool, void *element)113static void kasan_unpoison_element(mempool_t *pool, void *element)
@@ -114,7 +115,8 @@ static void kasan_unpoison_element(mempool_t *pool, void *element) · 1 unmodified line
114 if (pool->alloc == mempool_alloc_slab || pool->alloc == mempool_kmalloc)115 if (pool->alloc == mempool_alloc_slab || pool->alloc == mempool_kmalloc)
115 kasan_unpoison_range(element, __ksize(element));116 kasan_unpoison_range(element, __ksize(element));
116 else if (pool->alloc == mempool_alloc_pages)117 else if (pool->alloc == mempool_alloc_pages)
117 kasan_alloc_pages(element, (unsigned long)pool->pool_data, false);118 kasan_unpoison_pages(element, (unsigned long)pool->pool_data,
119 false);
118}120}
119 121
120static __always_inline void add_element(mempool_t *pool, void *element)122static __always_inline void add_element(mempool_t *pool, void *element)
mm/page_alloc.c+39−27
@@ -382,7 +382,7 @@ int page_group_by_mobility_disabled __read_mostly; · 381 unmodified lines
382static DEFINE_STATIC_KEY_TRUE(deferred_pages);382static DEFINE_STATIC_KEY_TRUE(deferred_pages);
383 383
384/*384/*
385 * Calling kasan_free_pages() only after deferred memory initialization385 * Calling kasan_poison_pages() only after deferred memory initialization
386 * has completed. Poisoning pages during deferred memory init will greatly386 * has completed. Poisoning pages during deferred memory init will greatly
387 * lengthen the process and cause problem in large memory systems as the387 * lengthen the process and cause problem in large memory systems as the
388 * deferred pages initialization is done with interrupt disabled.388 * deferred pages initialization is done with interrupt disabled.
@@ -394,15 +394,12 @@ static DEFINE_STATIC_KEY_TRUE(deferred_pages); · 5 unmodified lines
394 * on-demand allocation and then freed again before the deferred pages394 * on-demand allocation and then freed again before the deferred pages
395 * initialization is done, but this is not likely to happen.395 * initialization is done, but this is not likely to happen.
396 */396 */
397static inline void kasan_free_nondeferred_pages(struct page *page, int order,397static inline bool should_skip_kasan_poison(struct page *page, fpi_t fpi_flags)
398 bool init, fpi_t fpi_flags)
399{398{
400 if (static_branch_unlikely(&deferred_pages))399 return static_branch_unlikely(&deferred_pages) ||
401 return;400 (!IS_ENABLED(CONFIG_KASAN_GENERIC) &&
402 if (!IS_ENABLED(CONFIG_KASAN_GENERIC) &&401 (fpi_flags & FPI_SKIP_KASAN_POISON)) ||
403 (fpi_flags & FPI_SKIP_KASAN_POISON))402 PageSkipKASanPoison(page);
404 return;
405 kasan_free_pages(page, order, init);
406}403}
407 404
408/* Returns true if the struct page for the pfn is uninitialised */405/* Returns true if the struct page for the pfn is uninitialised */
@@ -453,13 +450,11 @@ defer_init(int nid, unsigned long pfn, unsigned long end_pfn) · 44 unmodified lines
453 return false;450 return false;
454}451}
455#else452#else
456static inline void kasan_free_nondeferred_pages(struct page *page, int order,453static inline bool should_skip_kasan_poison(struct page *page, fpi_t fpi_flags)
457 bool init, fpi_t fpi_flags)
458{454{
459 if (!IS_ENABLED(CONFIG_KASAN_GENERIC) &&455 return (!IS_ENABLED(CONFIG_KASAN_GENERIC) &&
460 (fpi_flags & FPI_SKIP_KASAN_POISON))456 (fpi_flags & FPI_SKIP_KASAN_POISON)) ||
461 return;457 PageSkipKASanPoison(page);
462 kasan_free_pages(page, order, init);
463}458}
464 459
465static inline bool early_page_uninitialised(unsigned long pfn)460static inline bool early_page_uninitialised(unsigned long pfn)
@@ -1226,10 +1221,16 @@ out: · 760 unmodified lines
1226 return ret;1221 return ret;
1227}1222}
1228 1223
1229static void kernel_init_free_pages(struct page *page, int numpages)1224static void kernel_init_free_pages(struct page *page, int numpages, bool zero_tags)
1230{1225{
1231 int i;1226 int i;
1232 1227
1228 if (zero_tags) {
1229 for (i = 0; i < numpages; i++)
1230 tag_clear_highpage(page + i);
1231 return;
1232 }
1233
1233 /* s390's use of memset() could override KASAN redzones. */1234 /* s390's use of memset() could override KASAN redzones. */
1234 kasan_disable_current();1235 kasan_disable_current();
1235 for (i = 0; i < numpages; i++) {1236 for (i = 0; i < numpages; i++) {
@@ -1245,7 +1246,7 @@ static __always_inline bool free_pages_prepare(struct page *page, · 9 unmodified lines
1245 unsigned int order, bool check_free, fpi_t fpi_flags)1246 unsigned int order, bool check_free, fpi_t fpi_flags)
1246{1247{
1247 int bad = 0;1248 int bad = 0;
1248 bool init;1249 bool skip_kasan_poison = should_skip_kasan_poison(page, fpi_flags);
1249 1250
1250 VM_BUG_ON_PAGE(PageTail(page), page);1251 VM_BUG_ON_PAGE(PageTail(page), page);
1251 1252
@@ -1314,10 +1315,17 @@ static __always_inline bool free_pages_prepare(struct page *page, · 62 unmodified lines
1314 * With hardware tag-based KASAN, memory tags must be set before the1315 * With hardware tag-based KASAN, memory tags must be set before the
1315 * page becomes unavailable via debug_pagealloc or arch_free_page.1316 * page becomes unavailable via debug_pagealloc or arch_free_page.
1316 */1317 */
1317 init = want_init_on_free();1318 if (kasan_has_integrated_init()) {
1318 if (init && !kasan_has_integrated_init())1319 if (!skip_kasan_poison)
1319 kernel_init_free_pages(page, 1 << order);1320 kasan_free_pages(page, order);
1320 kasan_free_nondeferred_pages(page, order, init, fpi_flags);1321 } else {
1322 bool init = want_init_on_free();
1323
1324 if (init)
1325 kernel_init_free_pages(page, 1 << order, false);
1326 if (!skip_kasan_poison)
1327 kasan_poison_pages(page, order, init);
1328 }
1321 1329
1322 /*1330 /*
1323 * arch_free_page() can make the page's contents inaccessible. s3901331 * arch_free_page() can make the page's contents inaccessible. s390
@@ -2324,8 +2332,6 @@ static bool check_new_pages(struct page *page, unsigned int order) · 1000 unmodified lines
2324inline void post_alloc_hook(struct page *page, unsigned int order,2332inline void post_alloc_hook(struct page *page, unsigned int order,
2325 gfp_t gfp_flags)2333 gfp_t gfp_flags)
2326{2334{
2327 bool init;
2328
2329 set_page_private(page, 0);2335 set_page_private(page, 0);
2330 set_page_refcounted(page);2336 set_page_refcounted(page);
2331 2337
@@ -2344,10 +2350,16 @@ inline void post_alloc_hook(struct page *page, unsigned int order, · 12 unmodified lines
2344 * kasan_alloc_pages and kernel_init_free_pages must be2350 * kasan_alloc_pages and kernel_init_free_pages must be
2345 * kept together to avoid discrepancies in behavior.2351 * kept together to avoid discrepancies in behavior.
2346 */2352 */
2347 init = !want_init_on_free() && want_init_on_alloc(gfp_flags);2353 if (kasan_has_integrated_init()) {
2348 kasan_alloc_pages(page, order, init);2354 kasan_alloc_pages(page, order, gfp_flags);
2349 if (init && !kasan_has_integrated_init())2355 } else {
2350 kernel_init_free_pages(page, 1 << order);2356 bool init = !want_init_on_free() && want_init_on_alloc(gfp_flags);
2357
2358 kasan_unpoison_pages(page, order, init);
2359 if (init)
2360 kernel_init_free_pages(page, 1 << order,
2361 gfp_flags & __GFP_ZEROTAGS);
2362 }
2351 2363
2352 set_page_owner(page, order, gfp_flags);2364 set_page_owner(page, order, gfp_flags);
2353}2365}
scripts/Makefile.kasan+1−0
@@ -50,6 +50,7 @@ endif · 49 unmodified lines
50CFLAGS_KASAN := -fsanitize=kernel-hwaddress \50CFLAGS_KASAN := -fsanitize=kernel-hwaddress \
51 $(call cc-param,hwasan-instrument-stack=$(stack_enable)) \51 $(call cc-param,hwasan-instrument-stack=$(stack_enable)) \
52 $(call cc-param,hwasan-use-short-granules=0) \52 $(call cc-param,hwasan-use-short-granules=0) \
53 $(call cc-param,hwasan-inline-all-checks=0) \
53 $(instrumentation_flags)54 $(instrumentation_flags)
54 55
55endif # CONFIG_KASAN_SW_TAGS56endif # CONFIG_KASAN_SW_TAGS
scripts/tools-support-relr.sh+2−1
@@ -7,7 +7,8 @@ trap "rm -f $tmp_file.o $tmp_file $tmp_file.bin" EXIT · 6 unmodified lines
7cat << "END" | $CC -c -x c - -o $tmp_file.o >/dev/null 2>&17cat << "END" | $CC -c -x c - -o $tmp_file.o >/dev/null 2>&1
8void *p = &p;8void *p = &p;
9END9END
10$LD $tmp_file.o -shared -Bsymbolic --pack-dyn-relocs=relr -o $tmp_file10$LD $tmp_file.o -shared -Bsymbolic --pack-dyn-relocs=relr \
11 --use-android-relr-tags -o $tmp_file
11 12
12# Despite printing an error message, GNU nm still exits with exit code 0 if it13# Despite printing an error message, GNU nm still exits with exit code 0 if it
13# sees a relr section. So we need to check that nothing is printed to stderr.14# sees a relr section. So we need to check that nothing is printed to stderr.
tools/testing/selftests/arm64/fp/sve-probe-vls.c+1−1
@@ -25,7 +25,7 @@ int main(int argc, char **argv) · 24 unmodified lines
25 ksft_set_plan(2);25 ksft_set_plan(2);
26 26
27 if (!(getauxval(AT_HWCAP) & HWCAP_SVE))27 if (!(getauxval(AT_HWCAP) & HWCAP_SVE))
28 ksft_exit_skip("SVE not available");28 ksft_exit_skip("SVE not available\n");
29 29
30 /*30 /*
31 * Enumerate up to SVE_VQ_MAX vector lengths31 * Enumerate up to SVE_VQ_MAX vector lengths

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t