History

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm

Pull KVM updates from Paolo Bonzini: "ARM: - Support for userspace handling of synchronous external aborts (SEAs), allowing the VMM to potentially handle the abort in a non-fatal manner - Large rework of the VGIC's list register handling with the goal of supporting more active/pending IRQs than available list registers in hardware. In addition, the VGIC now supports EOImode==1 style deactivations for IRQs which may occur on a separate vCPU than the one that acked the IRQ - Support for FEAT_XNX (user / privileged execute permissions) and FEAT_HAF (hardware update to the Access Flag) in the software page table walkers and shadow MMU - Allow page table destruction to reschedule, fixing long need_resched latencies observed when destroying a large VM - Minor fixes to KVM and selftests Loongarch: - Get VM PMU capability from HW GCFG register - Add AVEC basic support - Use 64-bit register definition for EIOINTC - Add KVM timer test cases for tools/selftests RISC/V: - SBI message passing (MPXY) support for KVM guest - Give a new, more specific error subcode for the case when in-kernel AIA virtualization fails to allocate IMSIC VS-file - Support KVM_DIRTY_LOG_INITIALLY_SET, enabling dirty log gradually in small chunks - Fix guest page fault within HLV* instructions - Flush VS-stage TLB after VCPU migration for Andes cores s390: - Always allocate ESCA (Extended System Control Area), instead of starting with the basic SCA and converting to ESCA with the addition of the 65th vCPU. The price is increased number of exits (and worse performance) on z10 and earlier processor; ESCA was introduced by z114/z196 in 2010 - VIRT_XFER_TO_GUEST_WORK support - Operation exception forwarding support - Cleanups x86: - Skip the costly "zap all SPTEs" on an MMIO generation wrap if MMIO SPTE caching is disabled, as there can't be any relevant SPTEs to zap - Relocate a misplaced export - Fix an async #PF bug where KVM would clear the completion queue when the guest transitioned in and out of paging mode, e.g. when handling an SMI and then returning to paged mode via RSM - Leave KVM's user-return notifier registered even when disabling virtualization, as long as kvm.ko is loaded. On reboot/shutdown, keeping the notifier registered is ok; the kernel does not use the MSRs and the callback will run cleanly and restore host MSRs if the CPU manages to return to userspace before the system goes down - Use the checked version of {get,put}_user() - Fix a long-lurking bug where KVM's lack of catch-up logic for periodic APIC timers can result in a hard lockup in the host - Revert the periodic kvmclock sync logic now that KVM doesn't use a clocksource that's subject to NTP corrections - Clean up KVM's handling of MMIO Stale Data and L1TF, and bury the latter behind CONFIG_CPU_MITIGATIONS - Context switch XCR0, XSS, and PKRU outside of the entry/exit fast path; the only reason they were handled in the fast path was to paper of a bug in the core #MC code, and that has long since been fixed - Add emulator support for AVX MOV instructions, to play nice with emulated devices whose guest drivers like to access PCI BARs with large multi-byte instructions x86 (AMD): - Fix a few missing "VMCB dirty" bugs - Fix the worst of KVM's lack of EFER.LMSLE emulation - Add AVIC support for addressing 4k vCPUs in x2AVIC mode - Fix incorrect handling of selective CR0 writes when checking intercepts during emulation of L2 instructions - Fix a currently-benign bug where KVM would clobber SPEC_CTRL[63:32] on VMRUN and #VMEXIT - Fix a bug where KVM corrupt the guest code stream when re-injecting a soft interrupt if the guest patched the underlying code after the VM-Exit, e.g. when Linux patches code with a temporary INT3 - Add KVM_X86_SNP_POLICY_BITS to advertise supported SNP policy bits to userspace, and extend KVM "support" to all policy bits that don't require any actual support from KVM x86 (Intel): - Use the root role from kvm_mmu_page to construct EPTPs instead of the current vCPU state, partly as worthwhile cleanup, but mostly to pave the way for tracking per-root TLB flushes, and elide EPT flushes on pCPU migration if the root is clean from a previous flush - Add a few missing nested consistency checks - Rip out support for doing "early" consistency checks via hardware as the functionality hasn't been used in years and is no longer useful in general; replace it with an off-by-default module param to WARN if hardware fails a check that KVM does not perform - Fix a currently-benign bug where KVM would drop the guest's SPEC_CTRL[63:32] on VM-Enter - Misc cleanups - Overhaul the TDX code to address systemic races where KVM (acting on behalf of userspace) could inadvertantly trigger lock contention in the TDX-Module; KVM was either working around these in weird, ugly ways, or was simply oblivious to them (though even Yan's devilish selftests could only break individual VMs, not the host kernel) - Fix a bug where KVM could corrupt a vCPU's cpu_list when freeing a TDX vCPU, if creating said vCPU failed partway through - Fix a few sparse warnings (bad annotation, 0 != NULL) - Use struct_size() to simplify copying TDX capabilities to userspace - Fix a bug where TDX would effectively corrupt user-return MSR values if the TDX Module rejects VP.ENTER and thus doesn't clobber host MSRs as expected Selftests: - Fix a math goof in mmu_stress_test when running on a single-CPU system/VM - Forcefully override ARCH from x86_64 to x86 to play nice with specifying ARCH=x86_64 on the command line - Extend a bunch of nested VMX to validate nested SVM as well - Add support for LA57 in the core VM_MODE_xxx macro, and add a test to verify KVM can save/restore nested VMX state when L1 is using 5-level paging, but L2 is not - Clean up the guest paging code in anticipation of sharing the core logic for nested EPT and nested NPT guest_memfd: - Add NUMA mempolicy support for guest_memfd, and clean up a variety of rough edges in guest_memfd along the way - Define a CLASS to automatically handle get+put when grabbing a guest_memfd from a memslot to make it harder to leak references - Enhance KVM selftests to make it easer to develop and debug selftests like those added for guest_memfd NUMA support, e.g. where test and/or KVM bugs often result in hard-to-debug SIGBUS errors - Misc cleanups Generic: - Use the recently-added WQ_PERCPU when creating the per-CPU workqueue for irqfd cleanup - Fix a goof in the dirty ring documentation - Fix choice of target for directed yield across different calls to kvm_vcpu_on_spin(); the function was always starting from the first vCPU instead of continuing the round-robin search" * tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (260 commits) KVM: arm64: at: Update AF on software walk only if VM has FEAT_HAFDBS KVM: arm64: at: Use correct HA bit in TCR_EL2 when regime is EL2 KVM: arm64: Document KVM_PGTABLE_PROT_{UX,PX} KVM: arm64: Fix spelling mistake "Unexpeced" -> "Unexpected" KVM: arm64: Add break to default case in kvm_pgtable_stage2_pte_prot() KVM: arm64: Add endian casting to kvm_swap_s[12]_desc() KVM: arm64: Fix compilation when CONFIG_ARM64_USE_LSE_ATOMICS=n KVM: arm64: selftests: Add test for AT emulation KVM: arm64: nv: Expose hardware access flag management to NV guests KVM: arm64: nv: Implement HW access flag management in stage-2 SW PTW KVM: arm64: Implement HW access flag management in stage-1 SW PTW KVM: arm64: Propagate PTW errors up to AT emulation KVM: arm64: Add helper for swapping guest descriptor KVM: arm64: nv: Use pgtable definitions in stage-2 walk KVM: arm64: Handle endianness in read helper for emulated PTW KVM: arm64: nv: Stop passing vCPU through void ptr in S2 PTW KVM: arm64: Call helper for reading descriptors directly KVM: arm64: nv: Advertise support for FEAT_XNX KVM: arm64: Teach ptdump about FEAT_XNX permissions KVM: s390: Use generic VIRT_XFER_TO_GUEST_WORK functions ...

Linus Torvalds committed 11 months ago

Commit 51d90a12 parents 399ead3 e0c26d4191 files · +6293 −2625Browse files

Unified · Split

191 files
  1. Documentation/virt/kvm/api.rst +66 −4
  2. Documentation/virt/kvm/x86/errata.rst +8 −1
  3. arch/arm64/include/asm/kvm_arm.h +1 −0
  4. arch/arm64/include/asm/kvm_asm.h +4 −4
  5. arch/arm64/include/asm/kvm_host.h +3 −0
  6. arch/arm64/include/asm/kvm_hyp.h +2 −1
  7. arch/arm64/include/asm/kvm_nested.h +38 −2
  8. arch/arm64/include/asm/kvm_pgtable.h +42 −7
  9. arch/arm64/include/asm/kvm_pkvm.h +3 −1
  10. arch/arm64/include/asm/virt.h +6 −1
  11. arch/arm64/kernel/cpufeature.c +59 −0
  12. arch/arm64/kernel/hyp-stub.S +5 −0
  13. arch/arm64/kernel/image-vars.h +1 −0
  14. arch/arm64/kvm/arm.c +17 −3
  15. arch/arm64/kvm/at.c +176 −20
  16. arch/arm64/kvm/hyp/nvhe/hyp-main.c +4 −3
  17. arch/arm64/kvm/hyp/nvhe/pkvm.c +3 −0
  18. arch/arm64/kvm/hyp/nvhe/sys_regs.c +5 −0
  19. arch/arm64/kvm/hyp/pgtable.c +104 −18
  20. arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c +4 −0
  21. arch/arm64/kvm/hyp/vgic-v3-sr.c +63 −33
  22. arch/arm64/kvm/mmu.c +124 −8
  23. arch/arm64/kvm/nested.c +84 −39
  24. arch/arm64/kvm/pkvm.c +9 −2
  25. arch/arm64/kvm/ptdump.c +27 −8
  26. arch/arm64/kvm/sys_regs.c +23 −5
  27. arch/arm64/kvm/vgic/vgic-init.c +5 −4
  28. arch/arm64/kvm/vgic/vgic-mmio-v2.c +24 −0
  29. arch/arm64/kvm/vgic/vgic-mmio.h +1 −0
  30. arch/arm64/kvm/vgic/vgic-v2.c +220 −71
  31. arch/arm64/kvm/vgic/vgic-v3-nested.c +52 −52
  32. arch/arm64/kvm/vgic/vgic-v3.c +328 −98
  33. arch/arm64/kvm/vgic/vgic-v4.c +4 −1
  34. arch/arm64/kvm/vgic/vgic.c +186 −112
  35. arch/arm64/kvm/vgic/vgic.h +39 −4
  36. arch/arm64/tools/cpucaps +2 −0
  37. arch/loongarch/include/asm/kvm_eiointc.h +8 −47
  38. arch/loongarch/include/asm/kvm_host.h +8 −0
  39. arch/loongarch/include/asm/kvm_vcpu.h +1 −0
  40. arch/loongarch/include/asm/loongarch.h +2 −0
  41. arch/loongarch/include/uapi/asm/kvm.h +1 −0
  42. arch/loongarch/kvm/Kconfig +0 −1
  43. arch/loongarch/kvm/intc/eiointc.c +40 −40
  44. arch/loongarch/kvm/interrupt.c +13 −2
  45. arch/loongarch/kvm/vcpu.c +19 −4
  46. arch/loongarch/kvm/vm.c +27 −13
  47. arch/mips/kvm/Kconfig +0 −1
  48. arch/mips/kvm/mips.c +2 −2
  49. arch/powerpc/kvm/Kconfig +0 −1
  50. arch/powerpc/kvm/powerpc.c +2 −2
  51. arch/riscv/include/asm/kvm_host.h +6 −0
  52. arch/riscv/include/asm/kvm_tlb.h +1 −0
  53. arch/riscv/include/asm/kvm_vcpu_sbi.h +4 −1
  54. arch/riscv/include/asm/kvm_vmid.h +0 −1
  55. arch/riscv/include/uapi/asm/kvm.h +3 −0
  56. arch/riscv/kvm/Kconfig +0 −1
  57. arch/riscv/kvm/Makefile +1 −0
  58. arch/riscv/kvm/aia_imsic.c +1 −1
  59. arch/riscv/kvm/main.c +14 −0
  60. arch/riscv/kvm/mmu.c +4 −1
  61. arch/riscv/kvm/tlb.c +30 −0
  62. arch/riscv/kvm/vcpu.c +3 −3
  63. arch/riscv/kvm/vcpu_insn.c +22 −0
  64. arch/riscv/kvm/vcpu_sbi.c +9 −1
  65. arch/riscv/kvm/vcpu_sbi_base.c +1 −27
  66. arch/riscv/kvm/vcpu_sbi_forward.c +34 −0
  67. arch/riscv/kvm/vcpu_sbi_replace.c +0 −32
  68. arch/riscv/kvm/vcpu_sbi_system.c +1 −3
  69. arch/riscv/kvm/vcpu_sbi_v01.c +1 −2
  70. arch/riscv/kvm/vmid.c +0 −23
  71. arch/s390/include/asm/kvm_host.h +4 −4
  72. arch/s390/include/asm/stacktrace.h +1 −0
  73. arch/s390/kernel/asm-offsets.c +1 −0
  74. arch/s390/kernel/entry.S +2 −0
  75. arch/s390/kvm/Kconfig +1 −1
  76. arch/s390/kvm/gaccess.c +6 −21
  77. arch/s390/kvm/intercept.c +3 −0
  78. arch/s390/kvm/interrupt.c +17 −63
  79. arch/s390/kvm/kvm-s390.c +64 −169
  80. arch/s390/kvm/kvm-s390.h +1 −8
  81. arch/s390/kvm/vsie.c +14 −6
  82. arch/x86/include/asm/cpufeatures.h +7 −0
  83. arch/x86/include/asm/hardirq.h +2 −2
  84. arch/x86/include/asm/kvm-x86-ops.h +1 −0
  85. arch/x86/include/asm/kvm_host.h +14 −9
  86. arch/x86/include/asm/nospec-branch.h +15 −15
  87. arch/x86/include/asm/svm.h +4 −1
  88. arch/x86/include/uapi/asm/kvm.h +1 −0
  89. arch/x86/kernel/cpu/bugs.c +9 −13
  90. arch/x86/kernel/cpu/scattered.c +1 −0
  91. arch/x86/kvm/cpuid.c +1 −0
  92. arch/x86/kvm/emulate.c +223 −96
  93. arch/x86/kvm/fpu.h +66 −0
  94. arch/x86/kvm/hyperv.c +1 −1
  95. arch/x86/kvm/kvm_emulate.h +16 −4
  96. arch/x86/kvm/lapic.c +31 −13
  97. arch/x86/kvm/mmu.h +1 −4
  98. arch/x86/kvm/mmu/mmu.c +88 −6
  99. arch/x86/kvm/mmu/mmu_internal.h +0 −10
  100. arch/x86/kvm/mmu/paging_tmpl.h +1 −1
  101. arch/x86/kvm/mmu/spte.c +1 −1
  102. arch/x86/kvm/mmu/spte.h +10 −0
  103. arch/x86/kvm/mmu/tdp_mmu.c +10 −40
  104. arch/x86/kvm/svm/avic.c +70 −16
  105. arch/x86/kvm/svm/nested.c +2 −10
  106. arch/x86/kvm/svm/sev.c +29 −18
  107. arch/x86/kvm/svm/svm.c +60 −43
  108. arch/x86/kvm/svm/svm.h +1 −3
  109. arch/x86/kvm/svm/vmenter.S +41 −12
  110. arch/x86/kvm/vmx/main.c +9 −0
  111. arch/x86/kvm/vmx/nested.c +61 −112
  112. arch/x86/kvm/vmx/run_flags.h +3 −7
  113. arch/x86/kvm/vmx/tdx.c +380 −425
  114. arch/x86/kvm/vmx/tdx.h +6 −3
  115. arch/x86/kvm/vmx/vmenter.S +34 −17
  116. arch/x86/kvm/vmx/vmx.c +173 −150
  117. arch/x86/kvm/vmx/vmx.h +0 −2
  118. arch/x86/kvm/vmx/x86_ops.h +1 −1
  119. arch/x86/kvm/x86.c +156 −129
  120. arch/x86/kvm/x86.h +14 −2
  121. drivers/crypto/ccp/sev-dev.c +37 −0
  122. drivers/irqchip/irq-apple-aic.c +5 −2
  123. drivers/irqchip/irq-gic.c +3 −0
  124. fs/btrfs/compression.c +2 −2
  125. fs/btrfs/verity.c +1 −1
  126. fs/erofs/zdata.c +1 −1
  127. fs/f2fs/compress.c +1 −1
  128. include/kvm/arm_vgic.h +19 −10
  129. include/linux/irqchip/arm-gic.h +6 −0
  130. include/linux/irqchip/arm-vgic-info.h +2 −0
  131. include/linux/kvm_host.h +2 −12
  132. include/linux/pagemap.h +13 −5
  133. include/linux/psp-sev.h +37 −0
  134. include/uapi/linux/kvm.h +11 −0
  135. include/uapi/linux/magic.h +1 −0
  136. mm/filemap.c +14 −9
  137. mm/mempolicy.c +6 −0
  138. mm/readahead.c +1 −1
  139. tools/arch/arm64/include/asm/esr.h +2 −0
  140. tools/testing/selftests/kvm/Makefile +1 −1
  141. tools/testing/selftests/kvm/Makefile.kvm +9 −3
  142. tools/testing/selftests/kvm/arm64/at.c +166 −0
  143. tools/testing/selftests/kvm/arm64/sea_to_user.c +331 −0
  144. tools/testing/selftests/kvm/arm64/vgic_irq.c +264 −23
  145. tools/testing/selftests/kvm/arm64/vgic_lpi_stress.c +4 −0
  146. tools/testing/selftests/kvm/guest_memfd_test.c +98 −0
  147. tools/testing/selftests/kvm/include/arm64/gic.h +1 −0
  148. tools/testing/selftests/kvm/include/arm64/gic_v3_its.h +1 −0
  149. tools/testing/selftests/kvm/include/kvm_syscalls.h +81 −0
  150. tools/testing/selftests/kvm/include/kvm_util.h +10 −35
  151. tools/testing/selftests/kvm/include/loongarch/arch_timer.h +85 −0
  152. tools/testing/selftests/kvm/include/loongarch/processor.h +80 −1
  153. tools/testing/selftests/kvm/include/numaif.h +69 −41
  154. tools/testing/selftests/kvm/include/x86/processor.h +1 −1
  155. tools/testing/selftests/kvm/include/x86/vmx.h +1 −2
  156. tools/testing/selftests/kvm/kvm_binary_stats_test.c +2 −2
  157. tools/testing/selftests/kvm/lib/arm64/gic.c +6 −0
  158. tools/testing/selftests/kvm/lib/arm64/gic_private.h +1 −0
  159. tools/testing/selftests/kvm/lib/arm64/gic_v3.c +22 −0
  160. tools/testing/selftests/kvm/lib/arm64/gic_v3_its.c +10 −0
  161. tools/testing/selftests/kvm/lib/arm64/processor.c +1 −1
  162. tools/testing/selftests/kvm/lib/kvm_util.c +80 −65
  163. tools/testing/selftests/kvm/lib/loongarch/exception.S +6 −0
  164. tools/testing/selftests/kvm/lib/loongarch/processor.c +45 −2
  165. tools/testing/selftests/kvm/lib/x86/memstress.c +1 −1
  166. tools/testing/selftests/kvm/lib/x86/processor.c +41 −41
  167. tools/testing/selftests/kvm/lib/x86/vmx.c +4 −5
  168. tools/testing/selftests/kvm/loongarch/arch_timer.c +200 −0
  169. tools/testing/selftests/kvm/mmu_stress_test.c +5 −5
  170. tools/testing/selftests/kvm/pre_fault_memory_test.c +14 −18
  171. tools/testing/selftests/kvm/riscv/get-reg-list.c +4 −0
  172. tools/testing/selftests/kvm/s390/user_operexec.c +140 −0
  173. tools/testing/selftests/kvm/x86/hyperv_features.c +1 −1
  174. tools/testing/selftests/kvm/x86/hyperv_ipi.c +9 −9
  175. tools/testing/selftests/kvm/x86/hyperv_tlb_flush.c +1 −1
  176. tools/testing/selftests/kvm/x86/nested_close_kvm_test.c +33 −9
  177. tools/testing/selftests/kvm/x86/nested_invalid_cr3_test.c +116 −0
  178. tools/testing/selftests/kvm/x86/nested_tsc_adjust_test.c +44 −35
  179. tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +43 −5
  180. tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +3 −6
  181. tools/testing/selftests/kvm/x86/sev_smoke_test.c +1 −1
  182. tools/testing/selftests/kvm/x86/state_test.c +1 −1
  183. tools/testing/selftests/kvm/x86/userspace_io_test.c +1 −1
  184. tools/testing/selftests/kvm/x86/vmx_dirty_log_test.c +6 −6
  185. tools/testing/selftests/kvm/x86/vmx_nested_la57_state_test.c +132 −0
  186. tools/testing/selftests/kvm/x86/xapic_ipi_test.c +2 −3
  187. virt/kvm/Kconfig +0 −3
  188. virt/kvm/eventfd.c +1 −1
  189. virt/kvm/guest_memfd.c +279 −94
  190. virt/kvm/kvm_main.c +10 −5
  191. virt/kvm/kvm_mm.h +5 −4
Documentation/virt/kvm/api.rst+66−4
@@ -7286,6 +7286,41 @@ exit, even without calls to ``KVM_ENABLE_CAP`` or similar. In this case, · 7285 unmodified lines
72867286 it will enter with output fields already valid; in the common case, the
72877287 ``unknown.ret`` field of the union will be ``TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED``.
72887288 Userspace need not do anything if it does not wish to support a TDVMCALL.
7289+
7290+::
7291+
7292+ /* KVM_EXIT_ARM_SEA */
7293+ struct {
7294+ #define KVM_EXIT_ARM_SEA_FLAG_GPA_VALID (1ULL << 0)
7295+ __u64 flags;
7296+ __u64 esr;
7297+ __u64 gva;
7298+ __u64 gpa;
7299+ } arm_sea;
7300+
7301+Used on arm64 systems. When the VM capability ``KVM_CAP_ARM_SEA_TO_USER`` is
7302+enabled, a KVM exits to userspace if a guest access causes a synchronous
7303+external abort (SEA) and the host APEI fails to handle the SEA.
7304+
7305+``esr`` is set to a sanitized value of ESR_EL2 from the exception taken to KVM,
7306+consisting of the following fields:
7307+
7308+ - ``ESR_EL2.EC``
7309+ - ``ESR_EL2.IL``
7310+ - ``ESR_EL2.FnV``
7311+ - ``ESR_EL2.EA``
7312+ - ``ESR_EL2.CM``
7313+ - ``ESR_EL2.WNR``
7314+ - ``ESR_EL2.FSC``
7315+ - ``ESR_EL2.SET`` (when FEAT_RAS is implemented for the VM)
7316+
7317+``gva`` is set to the value of FAR_EL2 from the exception taken to KVM when
7318+``ESR_EL2.FnV == 0``. Otherwise, the value of ``gva`` is unknown.
7319+
7320+``gpa`` is set to the faulting IPA from the exception taken to KVM when
7321+the ``KVM_EXIT_ARM_SEA_FLAG_GPA_VALID`` flag is set. Otherwise, the value of
7322+``gpa`` is unknown.
7323+
72897324 ::
72907325
72917326 /* Fix the size of the union. */
@@ -7820,7 +7855,7 @@ where 0xff represents CPUs 0-7 in cluster 0. · 528 unmodified lines
78207855 :Architectures: s390
78217856 :Parameters: none
78227857
7823−With this capability enabled, all illegal instructions 0x0000 (2 bytes) will
7858+With this capability enabled, the illegal instruction 0x0000 (2 bytes) will
78247859 be intercepted and forwarded to user space. User space can use this
78257860 mechanism e.g. to realize 2-byte software breakpoints. The kernel will
78267861 not inject an operating exception for these instructions, user space has
@@ -8028,7 +8063,7 @@ will be initialized to 1 when created. This also improves performance because · 201 unmodified lines
80288063 dirty logging can be enabled gradually in small chunks on the first call
80298064 to KVM_CLEAR_DIRTY_LOG. KVM_DIRTY_LOG_INITIALLY_SET depends on
80308065 KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE (it is also only available on
8031−x86 and arm64 for now).
8066+x86, arm64 and riscv for now).
80328067
80338068 KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2 was previously available under the name
80348069 KVM_CAP_MANUAL_DIRTY_LOG_PROTECT, but the implementation had bugs that make
@@ -8524,7 +8559,7 @@ Therefore, the ioctl must be called *before* reading the content of · 489 unmodified lines
85248559 the dirty pages.
85258560
85268561 The dirty ring can get full. When it happens, the KVM_RUN of the
8527−vcpu will return with exit reason KVM_EXIT_DIRTY_LOG_FULL.
8562+vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL.
85288563
85298564 The dirty ring interface has a major difference comparing to the
85308565 KVM_GET_DIRTY_LOG interface in that, when reading the dirty ring from
@@ -8692,7 +8727,7 @@ given VM. · 161 unmodified lines
86928727 When this capability is enabled, KVM resets the VCPU when setting
86938728 MP_STATE_INIT_RECEIVED through IOCTL. The original MP_STATE is preserved.
86948729
8695−7.43 KVM_CAP_ARM_CACHEABLE_PFNMAP_SUPPORTED
8730+7.44 KVM_CAP_ARM_CACHEABLE_PFNMAP_SUPPORTED
86968731 -------------------------------------------
86978732
86988733 :Architectures: arm64
@@ -8703,6 +8738,33 @@ This capability indicate to the userspace whether a PFNMAP memory region · 4 unmodified lines
87038738 can be safely mapped as cacheable. This relies on the presence of
87048739 force write back (FWB) feature support on the hardware.
87058740
8741+7.45 KVM_CAP_ARM_SEA_TO_USER
8742+----------------------------
8743+
8744+:Architecture: arm64
8745+:Target: VM
8746+:Parameters: none
8747+:Returns: 0 on success, -EINVAL if unsupported.
8748+
8749+When this capability is enabled, KVM may exit to userspace for SEAs taken to
8750+EL2 resulting from a guest access. See ``KVM_EXIT_ARM_SEA`` for more
8751+information.
8752+
8753+7.46 KVM_CAP_S390_USER_OPEREXEC
8754+-------------------------------
8755+
8756+:Architectures: s390
8757+:Parameters: none
8758+
8759+When this capability is enabled KVM forwards all operation exceptions
8760+that it doesn't handle itself to user space. This also includes the
8761+0x0000 instructions managed by KVM_CAP_S390_USER_INSTR0. This is
8762+helpful if user space wants to emulate instructions which are not
8763+(yet) implemented in hardware.
8764+
8765+This capability can be enabled dynamically even if VCPUs were already
8766+created and are running.
8767+
87068768 8. Other capabilities.
87078769 ======================
87088770
Documentation/virt/kvm/x86/errata.rst+8−1
@@ -48,7 +48,14 @@ versus "has_error_code", i.e. KVM's ABI follows AMD behavior. · 47 unmodified lines
4848 Nested virtualization features
4949 ------------------------------
5050
51−TBD
51+On AMD CPUs, when GIF is cleared, #DB exceptions or traps due to a breakpoint
52+register match are ignored and discarded by the CPU. The CPU relies on the VMM
53+to fully virtualize this behavior, even when vGIF is enabled for the guest
54+(i.e. vGIF=0 does not cause the CPU to drop #DBs when the guest is running).
55+KVM does not virtualize this behavior as the complexity is unjustified given
56+the rarity of the use case. One way to handle this would be for KVM to
57+intercept the #DB, temporarily disable the breakpoint, single-step over the
58+instruction, then re-enable the breakpoint.
5259
5360 x2APIC
5461 ------
arch/arm64/include/asm/kvm_arm.h+1−0
@@ -111,6 +111,7 @@ · 110 unmodified lines
111111 #define TCR_EL2_DS (1UL << 32)
112112 #define TCR_EL2_RES1 ((1U << 31) | (1 << 23))
113113 #define TCR_EL2_HPD (1 << 24)
114+#define TCR_EL2_HA (1 << 21)
114115 #define TCR_EL2_TBI (1 << 20)
115116 #define TCR_EL2_PS_SHIFT 16
116117 #define TCR_EL2_PS_MASK (7 << TCR_EL2_PS_SHIFT)
arch/arm64/include/asm/kvm_asm.h+4−4
@@ -79,7 +79,7 @@ enum __kvm_host_smccc_func { · 78 unmodified lines
7979 __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_range,
8080 __KVM_HOST_SMCCC_FUNC___kvm_flush_cpu_context,
8181 __KVM_HOST_SMCCC_FUNC___kvm_timer_set_cntvoff,
82− __KVM_HOST_SMCCC_FUNC___vgic_v3_save_vmcr_aprs,
82+ __KVM_HOST_SMCCC_FUNC___vgic_v3_save_aprs,
8383 __KVM_HOST_SMCCC_FUNC___vgic_v3_restore_vmcr_aprs,
8484 __KVM_HOST_SMCCC_FUNC___pkvm_reserve_vm,
8585 __KVM_HOST_SMCCC_FUNC___pkvm_unreserve_vm,
@@ -246,9 +246,9 @@ extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); · 160 unmodified lines
246246 extern int __kvm_tlbi_s1e2(struct kvm_s2_mmu *mmu, u64 va, u64 sys_encoding);
247247
248248 extern void __kvm_timer_set_cntvoff(u64 cntvoff);
249−extern void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
250−extern void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
251−extern void __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
249+extern int __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
250+extern int __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
251+extern int __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
252252
253253 extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
254254
arch/arm64/include/asm/kvm_host.h+3−0
@@ -54,6 +54,7 @@ · 53 unmodified lines
5454 #define KVM_REQ_NESTED_S2_UNMAP KVM_ARCH_REQ(8)
5555 #define KVM_REQ_GUEST_HYP_IRQ_PENDING KVM_ARCH_REQ(9)
5656 #define KVM_REQ_MAP_L1_VNCR_EL2 KVM_ARCH_REQ(10)
57+#define KVM_REQ_VGIC_PROCESS_UPDATE KVM_ARCH_REQ(11)
5758
5859 #define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
5960 KVM_DIRTY_LOG_INITIALLY_SET)
@@ -350,6 +351,8 @@ struct kvm_arch { · 290 unmodified lines
350351 #define KVM_ARCH_FLAG_GUEST_HAS_SVE 9
351352 /* MIDR_EL1, REVIDR_EL1, and AIDR_EL1 are writable from userspace */
352353 #define KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS 10
354+ /* Unhandled SEAs are taken to userspace */
355+#define KVM_ARCH_FLAG_EXIT_SEA 11
353356 unsigned long flags;
354357
355358 /* VM-wide vCPU feature set */
arch/arm64/include/asm/kvm_hyp.h+2−1
@@ -77,12 +77,13 @@ DECLARE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); · 76 unmodified lines
7777 int __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu);
7878
7979 u64 __gic_v3_get_lr(unsigned int lr);
80+void __gic_v3_set_lr(u64 val, int lr);
8081
8182 void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if);
8283 void __vgic_v3_restore_state(struct vgic_v3_cpu_if *cpu_if);
8384 void __vgic_v3_activate_traps(struct vgic_v3_cpu_if *cpu_if);
8485 void __vgic_v3_deactivate_traps(struct vgic_v3_cpu_if *cpu_if);
85−void __vgic_v3_save_vmcr_aprs(struct vgic_v3_cpu_if *cpu_if);
86+void __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if);
8687 void __vgic_v3_restore_vmcr_aprs(struct vgic_v3_cpu_if *cpu_if);
8788 int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu);
8889
arch/arm64/include/asm/kvm_nested.h+38−2
@@ -120,9 +120,42 @@ static inline bool kvm_s2_trans_writable(struct kvm_s2_trans *trans) · 119 unmodified lines
120120 return trans->writable;
121121 }
122122
123−static inline bool kvm_s2_trans_executable(struct kvm_s2_trans *trans)
123+static inline bool kvm_has_xnx(struct kvm *kvm)
124124 {
125− return !(trans->desc & BIT(54));
125+ return cpus_have_final_cap(ARM64_HAS_XNX) &&
126+ kvm_has_feat(kvm, ID_AA64MMFR1_EL1, XNX, IMP);
127+}
128+
129+static inline bool kvm_s2_trans_exec_el0(struct kvm *kvm, struct kvm_s2_trans *trans)
130+{
131+ u8 xn = FIELD_GET(KVM_PTE_LEAF_ATTR_HI_S2_XN, trans->desc);
132+
133+ if (!kvm_has_xnx(kvm))
134+ xn &= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b10);
135+
136+ switch (xn) {
137+ case 0b00:
138+ case 0b01:
139+ return true;
140+ default:
141+ return false;
142+ }
143+}
144+
145+static inline bool kvm_s2_trans_exec_el1(struct kvm *kvm, struct kvm_s2_trans *trans)
146+{
147+ u8 xn = FIELD_GET(KVM_PTE_LEAF_ATTR_HI_S2_XN, trans->desc);
148+
149+ if (!kvm_has_xnx(kvm))
150+ xn &= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, 0b10);
151+
152+ switch (xn) {
153+ case 0b00:
154+ case 0b11:
155+ return true;
156+ default:
157+ return false;
158+ }
126159 }
127160
128161 extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,
@@ -320,6 +353,7 @@ struct s1_walk_info { · 191 unmodified lines
320353 bool be;
321354 bool s2;
322355 bool pa52bit;
356+ bool ha;
323357 };
324358
325359 struct s1_walk_result {
@@ -370,4 +404,6 @@ void kvm_handle_s1e2_tlbi(struct kvm_vcpu *vcpu, u32 inst, u64 val); · 44 unmodified lines
370404 (FIX_VNCR - __c); \
371405 })
372406
407+int __kvm_at_swap_desc(struct kvm *kvm, gpa_t ipa, u64 old, u64 new);
408+
373409 #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/include/asm/kvm_pgtable.h+42−7
@@ -89,7 +89,7 @@ typedef u64 kvm_pte_t; · 88 unmodified lines
8989
9090 #define KVM_PTE_LEAF_ATTR_HI_S1_XN BIT(54)
9191
92−#define KVM_PTE_LEAF_ATTR_HI_S2_XN BIT(54)
92+#define KVM_PTE_LEAF_ATTR_HI_S2_XN GENMASK(54, 53)
9393
9494 #define KVM_PTE_LEAF_ATTR_HI_S1_GP BIT(50)
9595
@@ -240,7 +240,9 @@ enum kvm_pgtable_stage2_flags { · 144 unmodified lines
240240
241241 /**
242242 * enum kvm_pgtable_prot - Page-table permissions and attributes.
243− * @KVM_PGTABLE_PROT_X: Execute permission.
243+ * @KVM_PGTABLE_PROT_UX: Unprivileged execute permission.
244+ * @KVM_PGTABLE_PROT_PX: Privileged execute permission.
245+ * @KVM_PGTABLE_PROT_X: Privileged and unprivileged execute permission.
244246 * @KVM_PGTABLE_PROT_W: Write permission.
245247 * @KVM_PGTABLE_PROT_R: Read permission.
246248 * @KVM_PGTABLE_PROT_DEVICE: Device attributes.
@@ -251,12 +253,15 @@ enum kvm_pgtable_stage2_flags { · 4 unmodified lines
251253 * @KVM_PGTABLE_PROT_SW3: Software bit 3.
252254 */
253255 enum kvm_pgtable_prot {
254− KVM_PGTABLE_PROT_X = BIT(0),
255− KVM_PGTABLE_PROT_W = BIT(1),
256− KVM_PGTABLE_PROT_R = BIT(2),
256+ KVM_PGTABLE_PROT_PX = BIT(0),
257+ KVM_PGTABLE_PROT_UX = BIT(1),
258+ KVM_PGTABLE_PROT_X = KVM_PGTABLE_PROT_PX |
259+ KVM_PGTABLE_PROT_UX,
260+ KVM_PGTABLE_PROT_W = BIT(2),
261+ KVM_PGTABLE_PROT_R = BIT(3),
257262
258− KVM_PGTABLE_PROT_DEVICE = BIT(3),
259− KVM_PGTABLE_PROT_NORMAL_NC = BIT(4),
263+ KVM_PGTABLE_PROT_DEVICE = BIT(4),
264+ KVM_PGTABLE_PROT_NORMAL_NC = BIT(5),
260265
261266 KVM_PGTABLE_PROT_SW0 = BIT(55),
262267 KVM_PGTABLE_PROT_SW1 = BIT(56),
@@ -355,6 +360,11 @@ static inline kvm_pte_t *kvm_dereference_pteref(struct kvm_pgtable_walker *walke · 92 unmodified lines
355360 return pteref;
356361 }
357362
363+static inline kvm_pte_t *kvm_dereference_pteref_raw(kvm_pteref_t pteref)
364+{
365+ return pteref;
366+}
367+
358368 static inline int kvm_pgtable_walk_begin(struct kvm_pgtable_walker *walker)
359369 {
360370 /*
@@ -384,6 +394,11 @@ static inline kvm_pte_t *kvm_dereference_pteref(struct kvm_pgtable_walker *walke · 23 unmodified lines
384394 return rcu_dereference_check(pteref, !(walker->flags & KVM_PGTABLE_WALK_SHARED));
385395 }
386396
397+static inline kvm_pte_t *kvm_dereference_pteref_raw(kvm_pteref_t pteref)
398+{
399+ return rcu_dereference_raw(pteref);
400+}
401+
387402 static inline int kvm_pgtable_walk_begin(struct kvm_pgtable_walker *walker)
388403 {
389404 if (walker->flags & KVM_PGTABLE_WALK_SHARED)
@@ -551,6 +566,26 @@ static inline int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2 · 161 unmodified lines
551566 */
552567 void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt);
553568
569+/**
570+ * kvm_pgtable_stage2_destroy_range() - Destroy the unlinked range of addresses.
571+ * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*().
572+ * @addr: Intermediate physical address at which to place the mapping.
573+ * @size: Size of the mapping.
574+ *
575+ * The page-table is assumed to be unreachable by any hardware walkers prior
576+ * to freeing and therefore no TLB invalidation is performed.
577+ */
578+void kvm_pgtable_stage2_destroy_range(struct kvm_pgtable *pgt,
579+ u64 addr, u64 size);
580+
581+/**
582+ * kvm_pgtable_stage2_destroy_pgd() - Destroy the PGD of guest stage-2 page-table.
583+ * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*().
584+ *
585+ * It is assumed that the rest of the page-table is freed before this operation.
586+ */
587+void kvm_pgtable_stage2_destroy_pgd(struct kvm_pgtable *pgt);
588+
554589 /**
555590 * kvm_pgtable_stage2_free_unlinked() - Free an unlinked stage-2 paging structure.
556591 * @mm_ops: Memory management callbacks.
arch/arm64/include/asm/kvm_pkvm.h+3−1
@@ -180,7 +180,9 @@ struct pkvm_mapping { · 179 unmodified lines
180180
181181 int pkvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
182182 struct kvm_pgtable_mm_ops *mm_ops);
183−void pkvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt);
183+void pkvm_pgtable_stage2_destroy_range(struct kvm_pgtable *pgt,
184+ u64 addr, u64 size);
185+void pkvm_pgtable_stage2_destroy_pgd(struct kvm_pgtable *pgt);
184186 int pkvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys,
185187 enum kvm_pgtable_prot prot, void *mc,
186188 enum kvm_pgtable_walk_flags flags);
arch/arm64/include/asm/virt.h+6−1
@@ -40,8 +40,13 @@ · 39 unmodified lines
4040 */
4141 #define HVC_FINALISE_EL2 3
4242
43+/*
44+ * HVC_GET_ICH_VTR_EL2 - Retrieve the ICH_VTR_EL2 value
45+ */
46+#define HVC_GET_ICH_VTR_EL2 4
47+
4348 /* Max number of HYP stub hypercalls */
44−#define HVC_STUB_HCALL_NR 4
49+#define HVC_STUB_HCALL_NR 5
4550
4651 /* Error returned when an invalid stub number is passed into x0 */
4752 #define HVC_STUB_ERR 0xbadca11
arch/arm64/kernel/cpufeature.c+59−0
@@ -2304,6 +2304,49 @@ static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry · 2303 unmodified lines
23042304 }
23052305 #endif
23062306
2307+static bool can_trap_icv_dir_el1(const struct arm64_cpu_capabilities *entry,
2308+ int scope)
2309+{
2310+ static const struct midr_range has_vgic_v3[] = {
2311+ MIDR_ALL_VERSIONS(MIDR_APPLE_M1_ICESTORM),
2312+ MIDR_ALL_VERSIONS(MIDR_APPLE_M1_FIRESTORM),
2313+ MIDR_ALL_VERSIONS(MIDR_APPLE_M1_ICESTORM_PRO),
2314+ MIDR_ALL_VERSIONS(MIDR_APPLE_M1_FIRESTORM_PRO),
2315+ MIDR_ALL_VERSIONS(MIDR_APPLE_M1_ICESTORM_MAX),
2316+ MIDR_ALL_VERSIONS(MIDR_APPLE_M1_FIRESTORM_MAX),
2317+ MIDR_ALL_VERSIONS(MIDR_APPLE_M2_BLIZZARD),
2318+ MIDR_ALL_VERSIONS(MIDR_APPLE_M2_AVALANCHE),
2319+ MIDR_ALL_VERSIONS(MIDR_APPLE_M2_BLIZZARD_PRO),
2320+ MIDR_ALL_VERSIONS(MIDR_APPLE_M2_AVALANCHE_PRO),
2321+ MIDR_ALL_VERSIONS(MIDR_APPLE_M2_BLIZZARD_MAX),
2322+ MIDR_ALL_VERSIONS(MIDR_APPLE_M2_AVALANCHE_MAX),
2323+ {},
2324+ };
2325+ struct arm_smccc_res res = {};
2326+
2327+ BUILD_BUG_ON(ARM64_HAS_ICH_HCR_EL2_TDIR <= ARM64_HAS_GICV3_CPUIF);
2328+ BUILD_BUG_ON(ARM64_HAS_ICH_HCR_EL2_TDIR <= ARM64_HAS_GICV5_LEGACY);
2329+ if (!this_cpu_has_cap(ARM64_HAS_GICV3_CPUIF) &&
2330+ !is_midr_in_range_list(has_vgic_v3))
2331+ return false;
2332+
2333+ if (!is_hyp_mode_available())
2334+ return false;
2335+
2336+ if (this_cpu_has_cap(ARM64_HAS_GICV5_LEGACY))
2337+ return true;
2338+
2339+ if (is_kernel_in_hyp_mode())
2340+ res.a1 = read_sysreg_s(SYS_ICH_VTR_EL2);
2341+ else
2342+ arm_smccc_1_1_hvc(HVC_GET_ICH_VTR_EL2, &res);
2343+
2344+ if (res.a0 == HVC_STUB_ERR)
2345+ return false;
2346+
2347+ return res.a1 & ICH_VTR_EL2_TDS;
2348+}
2349+
23072350 #ifdef CONFIG_ARM64_BTI
23082351 static void bti_enable(const struct arm64_cpu_capabilities *__unused)
23092352 {
@@ -2815,6 +2858,15 @@ static const struct arm64_cpu_capabilities arm64_features[] = { · 505 unmodified lines
28152858 .matches = has_gic_prio_relaxed_sync,
28162859 },
28172860 #endif
2861+ {
2862+ /*
2863+ * Depends on having GICv3
2864+ */
2865+ .desc = "ICV_DIR_EL1 trapping",
2866+ .capability = ARM64_HAS_ICH_HCR_EL2_TDIR,
2867+ .type = ARM64_CPUCAP_EARLY_LOCAL_CPU_FEATURE,
2868+ .matches = can_trap_icv_dir_el1,
2869+ },
28182870 #ifdef CONFIG_ARM64_E0PD
28192871 {
28202872 .desc = "E0PD",
@@ -3089,6 +3141,13 @@ static const struct arm64_cpu_capabilities arm64_features[] = { · 268 unmodified lines
30893141 .capability = ARM64_HAS_GICV5_LEGACY,
30903142 .matches = test_has_gicv5_legacy,
30913143 },
3144+ {
3145+ .desc = "XNX",
3146+ .capability = ARM64_HAS_XNX,
3147+ .type = ARM64_CPUCAP_SYSTEM_FEATURE,
3148+ .matches = has_cpuid_feature,
3149+ ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, XNX, IMP)
3150+ },
30923151 {},
30933152 };
30943153
arch/arm64/kernel/hyp-stub.S+5−0
@@ -54,6 +54,11 @@ SYM_CODE_START_LOCAL(elx_sync) · 53 unmodified lines
5454 1: cmp x0, #HVC_FINALISE_EL2
5555 b.eq __finalise_el2
5656
57+ cmp x0, #HVC_GET_ICH_VTR_EL2
58+ b.ne 2f
59+ mrs_s x1, SYS_ICH_VTR_EL2
60+ b 9f
61+
5762 2: cmp x0, #HVC_SOFT_RESTART
5863 b.ne 3f
5964 mov x0, x2
arch/arm64/kernel/image-vars.h+1−0
@@ -91,6 +91,7 @@ KVM_NVHE_ALIAS(spectre_bhb_patch_loop_mitigation_enable); · 90 unmodified lines
9191 KVM_NVHE_ALIAS(spectre_bhb_patch_wa3);
9292 KVM_NVHE_ALIAS(spectre_bhb_patch_clearbhb);
9393 KVM_NVHE_ALIAS(alt_cb_patch_nops);
94+KVM_NVHE_ALIAS(kvm_compute_ich_hcr_trap_bits);
9495
9596 /* Global kernel state accessed by nVHE hyp code. */
9697 KVM_NVHE_ALIAS(kvm_vgic_global_state);
arch/arm64/kvm/arm.c+17−3
@@ -132,6 +132,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · 131 unmodified lines
132132 }
133133 mutex_unlock(&kvm->lock);
134134 break;
135+ case KVM_CAP_ARM_SEA_TO_USER:
136+ r = 0;
137+ set_bit(KVM_ARCH_FLAG_EXIT_SEA, &kvm->arch.flags);
138+ break;
135139 default:
136140 break;
137141 }
@@ -327,6 +331,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · 189 unmodified lines
327331 case KVM_CAP_IRQFD_RESAMPLE:
328332 case KVM_CAP_COUNTER_OFFSET:
329333 case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS:
334+ case KVM_CAP_ARM_SEA_TO_USER:
330335 r = 1;
331336 break;
332337 case KVM_CAP_SET_GUEST_DEBUG2:
@@ -440,7 +445,7 @@ struct kvm *kvm_arch_alloc_vm(void) · 107 unmodified lines
440445 if (!has_vhe())
441446 return kzalloc(sz, GFP_KERNEL_ACCOUNT);
442447
443− return __vmalloc(sz, GFP_KERNEL_ACCOUNT | __GFP_HIGHMEM | __GFP_ZERO);
448+ return kvzalloc(sz, GFP_KERNEL_ACCOUNT);
444449 }
445450
446451 int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)
@@ -659,8 +664,7 @@ nommu: · 212 unmodified lines
659664 void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
660665 {
661666 if (is_protected_kvm_enabled()) {
662− kvm_call_hyp(__vgic_v3_save_vmcr_aprs,
663− &vcpu->arch.vgic_cpu.vgic_v3);
667+ kvm_call_hyp(__vgic_v3_save_aprs, &vcpu->arch.vgic_cpu.vgic_v3);
664668 kvm_call_hyp_nvhe(__pkvm_vcpu_put);
665669 }
666670
@@ -1042,6 +1046,10 @@ static int check_vcpu_requests(struct kvm_vcpu *vcpu) · 375 unmodified lines
10421046 */
10431047 kvm_check_request(KVM_REQ_IRQ_PENDING, vcpu);
10441048
1049+ /* Process interrupts deactivated through a trap */
1050+ if (kvm_check_request(KVM_REQ_VGIC_PROCESS_UPDATE, vcpu))
1051+ kvm_vgic_process_async_update(vcpu);
1052+
10451053 if (kvm_check_request(KVM_REQ_RECORD_STEAL, vcpu))
10461054 kvm_update_stolen_time(vcpu);
10471055
@@ -1835,6 +1843,12 @@ long kvm_arch_vcpu_ioctl(struct file *filp, · 787 unmodified lines
18351843 return r;
18361844 }
18371845
1846+long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
1847+ unsigned long arg)
1848+{
1849+ return -ENOIOCTLCMD;
1850+}
1851+
18381852 void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot)
18391853 {
18401854
arch/arm64/kvm/at.c+176−20
@@ -346,6 +346,11 @@ static int setup_s1_walk(struct kvm_vcpu *vcpu, struct s1_walk_info *wi, · 345 unmodified lines
346346
347347 wi->baddr &= GENMASK_ULL(wi->max_oa_bits - 1, x);
348348
349+ wi->ha = kvm_has_feat(vcpu->kvm, ID_AA64MMFR1_EL1, HAFDBS, AF);
350+ wi->ha &= (wi->regime == TR_EL2 ?
351+ FIELD_GET(TCR_EL2_HA, tcr) :
352+ FIELD_GET(TCR_HA, tcr));
353+
349354 return 0;
350355
351356 addrsz:
@@ -362,10 +367,42 @@ transfault: · 10 unmodified lines
362367 return -EFAULT;
363368 }
364369
370+static int kvm_read_s1_desc(struct kvm_vcpu *vcpu, u64 pa, u64 *desc,
371+ struct s1_walk_info *wi)
372+{
373+ u64 val;
374+ int r;
375+
376+ r = kvm_read_guest(vcpu->kvm, pa, &val, sizeof(val));
377+ if (r)
378+ return r;
379+
380+ if (wi->be)
381+ *desc = be64_to_cpu((__force __be64)val);
382+ else
383+ *desc = le64_to_cpu((__force __le64)val);
384+
385+ return 0;
386+}
387+
388+static int kvm_swap_s1_desc(struct kvm_vcpu *vcpu, u64 pa, u64 old, u64 new,
389+ struct s1_walk_info *wi)
390+{
391+ if (wi->be) {
392+ old = (__force u64)cpu_to_be64(old);
393+ new = (__force u64)cpu_to_be64(new);
394+ } else {
395+ old = (__force u64)cpu_to_le64(old);
396+ new = (__force u64)cpu_to_le64(new);
397+ }
398+
399+ return __kvm_at_swap_desc(vcpu->kvm, pa, old, new);
400+}
401+
365402 static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
366403 struct s1_walk_result *wr, u64 va)
367404 {
368− u64 va_top, va_bottom, baddr, desc;
405+ u64 va_top, va_bottom, baddr, desc, new_desc, ipa;
369406 int level, stride, ret;
370407
371408 level = wi->sl;
@@ -375,7 +412,7 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi, · 3 unmodified lines
375412 va_top = get_ia_size(wi) - 1;
376413
377414 while (1) {
378− u64 index, ipa;
415+ u64 index;
379416
380417 va_bottom = (3 - level) * stride + wi->pgshift;
381418 index = (va & GENMASK_ULL(va_top, va_bottom)) >> (va_bottom - 3);
@@ -414,16 +451,13 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi, · 32 unmodified lines
414451 return ret;
415452 }
416453
417− ret = kvm_read_guest(vcpu->kvm, ipa, &desc, sizeof(desc));
454+ ret = kvm_read_s1_desc(vcpu, ipa, &desc, wi);
418455 if (ret) {
419456 fail_s1_walk(wr, ESR_ELx_FSC_SEA_TTW(level), false);
420457 return ret;
421458 }
422459
423− if (wi->be)
424− desc = be64_to_cpu((__force __be64)desc);
425− else
426− desc = le64_to_cpu((__force __le64)desc);
460+ new_desc = desc;
427461
428462 /* Invalid descriptor */
429463 if (!(desc & BIT(0)))
@@ -477,6 +511,17 @@ static int walk_s1(struct kvm_vcpu *vcpu, struct s1_walk_info *wi, · 47 unmodified lines
477511 if (check_output_size(baddr & GENMASK(52, va_bottom), wi))
478512 goto addrsz;
479513
514+ if (wi->ha)
515+ new_desc |= PTE_AF;
516+
517+ if (new_desc != desc) {
518+ ret = kvm_swap_s1_desc(vcpu, ipa, desc, new_desc, wi);
519+ if (ret)
520+ return ret;
521+
522+ desc = new_desc;
523+ }
524+
480525 if (!(desc & PTE_AF)) {
481526 fail_s1_walk(wr, ESR_ELx_FSC_ACCESS_L(level), false);
482527 return -EACCES;
@@ -1221,7 +1266,7 @@ static void compute_s1_permissions(struct kvm_vcpu *vcpu, · 738 unmodified lines
12211266 wr->pr &= !pan;
12221267 }
12231268
1224−static u64 handle_at_slow(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)
1269+static int handle_at_slow(struct kvm_vcpu *vcpu, u32 op, u64 vaddr, u64 *par)
12251270 {
12261271 struct s1_walk_result wr = {};
12271272 struct s1_walk_info wi = {};
@@ -1246,6 +1291,11 @@ static u64 handle_at_slow(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) · 18 unmodified lines
12461291
12471292 srcu_read_unlock(&vcpu->kvm->srcu, idx);
12481293
1294+ /*
1295+ * Race to update a descriptor -- restart the walk.
1296+ */
1297+ if (ret == -EAGAIN)
1298+ return ret;
12491299 if (ret)
12501300 goto compute_par;
12511301
@@ -1279,7 +1329,8 @@ static u64 handle_at_slow(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) · 27 unmodified lines
12791329 fail_s1_walk(&wr, ESR_ELx_FSC_PERM_L(wr.level), false);
12801330
12811331 compute_par:
1282− return compute_par_s1(vcpu, &wi, &wr);
1332+ *par = compute_par_s1(vcpu, &wi, &wr);
1333+ return 0;
12831334 }
12841335
12851336 /*
@@ -1407,9 +1458,10 @@ static bool par_check_s1_access_fault(u64 par) · 121 unmodified lines
14071458 !(par & SYS_PAR_EL1_S));
14081459 }
14091460
1410−void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)
1461+int __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)
14111462 {
14121463 u64 par = __kvm_at_s1e01_fast(vcpu, op, vaddr);
1464+ int ret;
14131465
14141466 /*
14151467 * If PAR_EL1 reports that AT failed on a S1 permission or access
@@ -1421,15 +1473,20 @@ void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) · 5 unmodified lines
14211473 */
14221474 if ((par & SYS_PAR_EL1_F) &&
14231475 !par_check_s1_perm_fault(par) &&
1424− !par_check_s1_access_fault(par))
1425− par = handle_at_slow(vcpu, op, vaddr);
1476+ !par_check_s1_access_fault(par)) {
1477+ ret = handle_at_slow(vcpu, op, vaddr, &par);
1478+ if (ret)
1479+ return ret;
1480+ }
14261481
14271482 vcpu_write_sys_reg(vcpu, par, PAR_EL1);
1483+ return 0;
14281484 }
14291485
1430−void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)
1486+int __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)
14311487 {
14321488 u64 par;
1489+ int ret;
14331490
14341491 /*
14351492 * We've trapped, so everything is live on the CPU. As we will be
@@ -1476,13 +1533,17 @@ void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) · 40 unmodified lines
14761533 }
14771534
14781535 /* We failed the translation, let's replay it in slow motion */
1479− if ((par & SYS_PAR_EL1_F) && !par_check_s1_perm_fault(par))
1480− par = handle_at_slow(vcpu, op, vaddr);
1536+ if ((par & SYS_PAR_EL1_F) && !par_check_s1_perm_fault(par)) {
1537+ ret = handle_at_slow(vcpu, op, vaddr, &par);
1538+ if (ret)
1539+ return ret;
1540+ }
14811541
14821542 vcpu_write_sys_reg(vcpu, par, PAR_EL1);
1543+ return 0;
14831544 }
14841545
1485−void __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)
1546+int __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)
14861547 {
14871548 struct kvm_s2_trans out = {};
14881549 u64 ipa, par;
@@ -1509,13 +1570,13 @@ void __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) · 20 unmodified lines
15091570 break;
15101571 default:
15111572 WARN_ON_ONCE(1);
1512− return;
1573+ return 0;
15131574 }
15141575
15151576 __kvm_at_s1e01(vcpu, op, vaddr);
15161577 par = vcpu_read_sys_reg(vcpu, PAR_EL1);
15171578 if (par & SYS_PAR_EL1_F)
1518− return;
1579+ return 0;
15191580
15201581 /*
15211582 * If we only have a single stage of translation (EL2&0), exit
@@ -1523,14 +1584,14 @@ void __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) · 1 unmodified line
15231584 */
15241585 if (compute_translation_regime(vcpu, op) == TR_EL20 ||
15251586 !(vcpu_read_sys_reg(vcpu, HCR_EL2) & (HCR_VM | HCR_DC)))
1526− return;
1587+ return 0;
15271588
15281589 /* Do the stage-2 translation */
15291590 ipa = (par & GENMASK_ULL(47, 12)) | (vaddr & GENMASK_ULL(11, 0));
15301591 out.esr = 0;
15311592 ret = kvm_walk_nested_s2(vcpu, ipa, &out);
15321593 if (ret < 0)
1533− return;
1594+ return ret;
15341595
15351596 /* Check the access permission */
15361597 if (!out.esr &&
@@ -1539,6 +1600,7 @@ void __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) · 2 unmodified lines
15391600
15401601 par = compute_par_s12(vcpu, par, &out);
15411602 vcpu_write_sys_reg(vcpu, par, PAR_EL1);
1603+ return 0;
15421604 }
15431605
15441606 /*
@@ -1637,3 +1699,97 @@ int __kvm_find_s1_desc_level(struct kvm_vcpu *vcpu, u64 va, u64 ipa, int *level) · 92 unmodified lines
16371699 return ret;
16381700 }
16391701 }
1702+
1703+#ifdef CONFIG_ARM64_LSE_ATOMICS
1704+static int __lse_swap_desc(u64 __user *ptep, u64 old, u64 new)
1705+{
1706+ u64 tmp = old;
1707+ int ret = 0;
1708+
1709+ uaccess_enable_privileged();
1710+
1711+ asm volatile(__LSE_PREAMBLE
1712+ "1: cas %[old], %[new], %[addr]\n"
1713+ "2:\n"
1714+ _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %w[ret])
1715+ : [old] "+r" (old), [addr] "+Q" (*ptep), [ret] "+r" (ret)
1716+ : [new] "r" (new)
1717+ : "memory");
1718+
1719+ uaccess_disable_privileged();
1720+
1721+ if (ret)
1722+ return ret;
1723+ if (tmp != old)
1724+ return -EAGAIN;
1725+
1726+ return ret;
1727+}
1728+#else
1729+static int __lse_swap_desc(u64 __user *ptep, u64 old, u64 new)
1730+{
1731+ return -EINVAL;
1732+}
1733+#endif
1734+
1735+static int __llsc_swap_desc(u64 __user *ptep, u64 old, u64 new)
1736+{
1737+ int ret = 1;
1738+ u64 tmp;
1739+
1740+ uaccess_enable_privileged();
1741+
1742+ asm volatile("prfm pstl1strm, %[addr]\n"
1743+ "1: ldxr %[tmp], %[addr]\n"
1744+ "sub %[tmp], %[tmp], %[old]\n"
1745+ "cbnz %[tmp], 3f\n"
1746+ "2: stlxr %w[ret], %[new], %[addr]\n"
1747+ "3:\n"
1748+ _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %w[ret])
1749+ _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %w[ret])
1750+ : [ret] "+r" (ret), [addr] "+Q" (*ptep), [tmp] "=&r" (tmp)
1751+ : [old] "r" (old), [new] "r" (new)
1752+ : "memory");
1753+
1754+ uaccess_disable_privileged();
1755+
1756+ /* STLXR didn't update the descriptor, or the compare failed */
1757+ if (ret == 1)
1758+ return -EAGAIN;
1759+
1760+ return ret;
1761+}
1762+
1763+int __kvm_at_swap_desc(struct kvm *kvm, gpa_t ipa, u64 old, u64 new)
1764+{
1765+ struct kvm_memory_slot *slot;
1766+ unsigned long hva;
1767+ u64 __user *ptep;
1768+ bool writable;
1769+ int offset;
1770+ gfn_t gfn;
1771+ int r;
1772+
1773+ lockdep_assert(srcu_read_lock_held(&kvm->srcu));
1774+
1775+ gfn = ipa >> PAGE_SHIFT;
1776+ offset = offset_in_page(ipa);
1777+ slot = gfn_to_memslot(kvm, gfn);
1778+ hva = gfn_to_hva_memslot_prot(slot, gfn, &writable);
1779+ if (kvm_is_error_hva(hva))
1780+ return -EINVAL;
1781+ if (!writable)
1782+ return -EPERM;
1783+
1784+ ptep = (u64 __user *)hva + offset;
1785+ if (cpus_have_final_cap(ARM64_HAS_LSE_ATOMICS))
1786+ r = __lse_swap_desc(ptep, old, new);
1787+ else
1788+ r = __llsc_swap_desc(ptep, old, new);
1789+
1790+ if (r < 0)
1791+ return r;
1792+
1793+ mark_page_dirty_in_slot(kvm, slot, gfn);
1794+ return 0;
1795+}
arch/arm64/kvm/hyp/nvhe/hyp-main.c+4−3
@@ -157,6 +157,7 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) · 156 unmodified lines
157157 host_vcpu->arch.iflags = hyp_vcpu->vcpu.arch.iflags;
158158
159159 host_cpu_if->vgic_hcr = hyp_cpu_if->vgic_hcr;
160+ host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;
160161 for (i = 0; i < hyp_cpu_if->used_lrs; ++i)
161162 host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i];
162163 }
@@ -464,11 +465,11 @@ static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt) · 301 unmodified lines
464465 __vgic_v3_init_lrs();
465466 }
466467
467−static void handle___vgic_v3_save_vmcr_aprs(struct kvm_cpu_context *host_ctxt)
468+static void handle___vgic_v3_save_aprs(struct kvm_cpu_context *host_ctxt)
468469 {
469470 DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1);
470471
471− __vgic_v3_save_vmcr_aprs(kern_hyp_va(cpu_if));
472+ __vgic_v3_save_aprs(kern_hyp_va(cpu_if));
472473 }
473474
474475 static void handle___vgic_v3_restore_vmcr_aprs(struct kvm_cpu_context *host_ctxt)
@@ -616,7 +617,7 @@ static const hcall_t host_hcall[] = { · 141 unmodified lines
616617 HANDLE_FUNC(__kvm_tlb_flush_vmid_range),
617618 HANDLE_FUNC(__kvm_flush_cpu_context),
618619 HANDLE_FUNC(__kvm_timer_set_cntvoff),
619− HANDLE_FUNC(__vgic_v3_save_vmcr_aprs),
620+ HANDLE_FUNC(__vgic_v3_save_aprs),
620621 HANDLE_FUNC(__vgic_v3_restore_vmcr_aprs),
621622 HANDLE_FUNC(__pkvm_reserve_vm),
622623 HANDLE_FUNC(__pkvm_unreserve_vm),
arch/arm64/kvm/hyp/nvhe/pkvm.c+3−0
@@ -337,6 +337,9 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc · 336 unmodified lines
337337 /* CTR_EL0 is always under host control, even for protected VMs. */
338338 hyp_vm->kvm.arch.ctr_el0 = host_kvm->arch.ctr_el0;
339339
340+ /* Preserve the vgic model so that GICv3 emulation works */
341+ hyp_vm->kvm.arch.vgic.vgic_model = host_kvm->arch.vgic.vgic_model;
342+
340343 if (test_bit(KVM_ARCH_FLAG_MTE_ENABLED, &host_kvm->arch.flags))
341344 set_bit(KVM_ARCH_FLAG_MTE_ENABLED, &kvm->arch.flags);
342345
arch/arm64/kvm/hyp/nvhe/sys_regs.c+5−0
@@ -444,6 +444,8 @@ static const struct sys_reg_desc pvm_sys_reg_descs[] = { · 443 unmodified lines
444444
445445 /* Scalable Vector Registers are restricted. */
446446
447+ HOST_HANDLED(SYS_ICC_PMR_EL1),
448+
447449 RAZ_WI(SYS_ERRIDR_EL1),
448450 RAZ_WI(SYS_ERRSELR_EL1),
449451 RAZ_WI(SYS_ERXFR_EL1),
@@ -457,9 +459,12 @@ static const struct sys_reg_desc pvm_sys_reg_descs[] = { · 7 unmodified lines
457459
458460 /* Limited Ordering Regions Registers are restricted. */
459461
462+ HOST_HANDLED(SYS_ICC_DIR_EL1),
463+ HOST_HANDLED(SYS_ICC_RPR_EL1),
460464 HOST_HANDLED(SYS_ICC_SGI1R_EL1),
461465 HOST_HANDLED(SYS_ICC_ASGI1R_EL1),
462466 HOST_HANDLED(SYS_ICC_SGI0R_EL1),
467+ HOST_HANDLED(SYS_ICC_CTLR_EL1),
463468 { SYS_DESC(SYS_ICC_SRE_EL1), .access = pvm_gic_read_sre, },
464469
465470 HOST_HANDLED(SYS_CCSIDR_EL1),
arch/arm64/kvm/hyp/pgtable.c+104−18
@@ -661,11 +661,37 @@ void kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, · 660 unmodified lines
661661
662662 #define KVM_S2_MEMATTR(pgt, attr) PAGE_S2_MEMATTR(attr, stage2_has_fwb(pgt))
663663
664+static int stage2_set_xn_attr(enum kvm_pgtable_prot prot, kvm_pte_t *attr)
665+{
666+ bool px, ux;
667+ u8 xn;
668+
669+ px = prot & KVM_PGTABLE_PROT_PX;
670+ ux = prot & KVM_PGTABLE_PROT_UX;
671+
672+ if (!cpus_have_final_cap(ARM64_HAS_XNX) && px != ux)
673+ return -EINVAL;
674+
675+ if (px && ux)
676+ xn = 0b00;
677+ else if (!px && ux)
678+ xn = 0b01;
679+ else if (!px && !ux)
680+ xn = 0b10;
681+ else
682+ xn = 0b11;
683+
684+ *attr &= ~KVM_PTE_LEAF_ATTR_HI_S2_XN;
685+ *attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_S2_XN, xn);
686+ return 0;
687+}
688+
664689 static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot prot,
665690 kvm_pte_t *ptep)
666691 {
667692 kvm_pte_t attr;
668693 u32 sh = KVM_PTE_LEAF_ATTR_LO_S2_SH_IS;
694+ int r;
669695
670696 switch (prot & (KVM_PGTABLE_PROT_DEVICE |
671697 KVM_PGTABLE_PROT_NORMAL_NC)) {
@@ -685,8 +711,9 @@ static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot p · 13 unmodified lines
685711 attr = KVM_S2_MEMATTR(pgt, NORMAL);
686712 }
687713
688− if (!(prot & KVM_PGTABLE_PROT_X))
689− attr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;
714+ r = stage2_set_xn_attr(prot, &attr);
715+ if (r)
716+ return r;
690717
691718 if (prot & KVM_PGTABLE_PROT_R)
692719 attr |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R;
@@ -715,8 +742,20 @@ enum kvm_pgtable_prot kvm_pgtable_stage2_pte_prot(kvm_pte_t pte) · 22 unmodified lines
715742 prot |= KVM_PGTABLE_PROT_R;
716743 if (pte & KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W)
717744 prot |= KVM_PGTABLE_PROT_W;
718− if (!(pte & KVM_PTE_LEAF_ATTR_HI_S2_XN))
719− prot |= KVM_PGTABLE_PROT_X;
745+
746+ switch (FIELD_GET(KVM_PTE_LEAF_ATTR_HI_S2_XN, pte)) {
747+ case 0b00:
748+ prot |= KVM_PGTABLE_PROT_PX | KVM_PGTABLE_PROT_UX;
749+ break;
750+ case 0b01:
751+ prot |= KVM_PGTABLE_PROT_UX;
752+ break;
753+ case 0b11:
754+ prot |= KVM_PGTABLE_PROT_PX;
755+ break;
756+ default:
757+ break;
758+ }
720759
721760 return prot;
722761 }
@@ -1290,9 +1329,9 @@ bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr, · 567 unmodified lines
12901329 int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
12911330 enum kvm_pgtable_prot prot, enum kvm_pgtable_walk_flags flags)
12921331 {
1293− int ret;
1332+ kvm_pte_t xn = 0, set = 0, clr = 0;
12941333 s8 level;
1295− kvm_pte_t set = 0, clr = 0;
1334+ int ret;
12961335
12971336 if (prot & KVM_PTE_LEAF_ATTR_HI_SW)
12981337 return -EINVAL;
@@ -1303,8 +1342,12 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, · 4 unmodified lines
13031342 if (prot & KVM_PGTABLE_PROT_W)
13041343 set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W;
13051344
1306− if (prot & KVM_PGTABLE_PROT_X)
1307− clr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;
1345+ ret = stage2_set_xn_attr(prot, &xn);
1346+ if (ret)
1347+ return ret;
1348+
1349+ set |= xn & KVM_PTE_LEAF_ATTR_HI_S2_XN;
1350+ clr |= ~xn & KVM_PTE_LEAF_ATTR_HI_S2_XN;
13081351
13091352 ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level, flags);
13101353 if (!ret || ret == -EAGAIN)
@@ -1535,37 +1578,80 @@ size_t kvm_pgtable_stage2_pgd_size(u64 vtcr) · 224 unmodified lines
15351578 return kvm_pgd_pages(ia_bits, start_level) * PAGE_SIZE;
15361579 }
15371580
1538−static int stage2_free_walker(const struct kvm_pgtable_visit_ctx *ctx,
1539− enum kvm_pgtable_walk_flags visit)
1581+static int stage2_free_leaf(const struct kvm_pgtable_visit_ctx *ctx)
15401582 {
15411583 struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops;
15421584
1543− if (!stage2_pte_is_counted(ctx->old))
1585+ mm_ops->put_page(ctx->ptep);
1586+ return 0;
1587+}
1588+
1589+static int stage2_free_table_post(const struct kvm_pgtable_visit_ctx *ctx)
1590+{
1591+ struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops;
1592+ kvm_pte_t *childp = kvm_pte_follow(ctx->old, mm_ops);
1593+
1594+ if (mm_ops->page_count(childp) != 1)
15441595 return 0;
15451596
1597+ /*
1598+ * Drop references and clear the now stale PTE to avoid rewalking the
1599+ * freed page table.
1600+ */
15461601 mm_ops->put_page(ctx->ptep);
1602+ mm_ops->put_page(childp);
1603+ kvm_clear_pte(ctx->ptep);
1604+ return 0;
1605+}
15471606
1548− if (kvm_pte_table(ctx->old, ctx->level))
1549− mm_ops->put_page(kvm_pte_follow(ctx->old, mm_ops));
1607+static int stage2_free_walker(const struct kvm_pgtable_visit_ctx *ctx,
1608+ enum kvm_pgtable_walk_flags visit)
1609+{
1610+ if (!stage2_pte_is_counted(ctx->old))
1611+ return 0;
15501612
1551− return 0;
1613+ switch (visit) {
1614+ case KVM_PGTABLE_WALK_LEAF:
1615+ return stage2_free_leaf(ctx);
1616+ case KVM_PGTABLE_WALK_TABLE_POST:
1617+ return stage2_free_table_post(ctx);
1618+ default:
1619+ return -EINVAL;
1620+ }
15521621 }
15531622
1554−void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt)
1623+void kvm_pgtable_stage2_destroy_range(struct kvm_pgtable *pgt,
1624+ u64 addr, u64 size)
15551625 {
1556− size_t pgd_sz;
15571626 struct kvm_pgtable_walker walker = {
15581627 .cb = stage2_free_walker,
15591628 .flags = KVM_PGTABLE_WALK_LEAF |
15601629 KVM_PGTABLE_WALK_TABLE_POST,
15611630 };
15621631
1563− WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker));
1632+ WARN_ON(kvm_pgtable_walk(pgt, addr, size, &walker));
1633+}
1634+
1635+void kvm_pgtable_stage2_destroy_pgd(struct kvm_pgtable *pgt)
1636+{
1637+ size_t pgd_sz;
1638+
15641639 pgd_sz = kvm_pgd_pages(pgt->ia_bits, pgt->start_level) * PAGE_SIZE;
1565− pgt->mm_ops->free_pages_exact(kvm_dereference_pteref(&walker, pgt->pgd), pgd_sz);
1640+
1641+ /*
1642+ * Since the pgtable is unlinked at this point, and not shared with
1643+ * other walkers, safely deference pgd with kvm_dereference_pteref_raw()
1644+ */
1645+ pgt->mm_ops->free_pages_exact(kvm_dereference_pteref_raw(pgt->pgd), pgd_sz);
15661646 pgt->pgd = NULL;
15671647 }
15681648
1649+void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt)
1650+{
1651+ kvm_pgtable_stage2_destroy_range(pgt, 0, BIT(pgt->ia_bits));
1652+ kvm_pgtable_stage2_destroy_pgd(pgt);
1653+}
1654+
15691655 void kvm_pgtable_stage2_free_unlinked(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, s8 level)
15701656 {
15711657 kvm_pteref_t ptep = (kvm_pteref_t)pgtable;
arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c+4−0
@@ -63,6 +63,10 @@ int __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu) · 62 unmodified lines
6363 return -1;
6464 }
6565
66+ /* Handle deactivation as a normal exit */
67+ if ((fault_ipa - vgic->vgic_cpu_base) >= GIC_CPU_DEACTIVATE)
68+ return 0;
69+
6670 rd = kvm_vcpu_dabt_get_rd(vcpu);
6771 addr = kvm_vgic_global_state.vcpu_hyp_va;
6872 addr += fault_ipa - vgic->vgic_cpu_base;
arch/arm64/kvm/hyp/vgic-v3-sr.c+63−33
@@ -14,6 +14,8 @@ · 13 unmodified lines
1414 #include <asm/kvm_hyp.h>
1515 #include <asm/kvm_mmu.h>
1616
17+#include "../../vgic/vgic.h"
18+
1719 #define vtr_to_max_lr_idx(v) ((v) & 0xf)
1820 #define vtr_to_nr_pre_bits(v) ((((u32)(v) >> 26) & 7) + 1)
1921 #define vtr_to_nr_apr_regs(v) (1 << (vtr_to_nr_pre_bits(v) - 5))
@@ -58,7 +60,7 @@ u64 __gic_v3_get_lr(unsigned int lr) · 38 unmodified lines
5860 unreachable();
5961 }
6062
61−static void __gic_v3_set_lr(u64 val, int lr)
63+void __gic_v3_set_lr(u64 val, int lr)
6264 {
6365 switch (lr & 0xf) {
6466 case 0:
@@ -196,6 +198,11 @@ static u32 __vgic_v3_read_ap1rn(int n) · 131 unmodified lines
196198 return val;
197199 }
198200
201+static u64 compute_ich_hcr(struct vgic_v3_cpu_if *cpu_if)
202+{
203+ return cpu_if->vgic_hcr | vgic_ich_hcr_trap_bits();
204+}
205+
199206 void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if)
200207 {
201208 u64 used_lrs = cpu_if->used_lrs;
@@ -212,14 +219,12 @@ void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if) · 10 unmodified lines
212219 }
213220 }
214221
215− if (used_lrs || cpu_if->its_vpe.its_vm) {
222+ if (used_lrs) {
216223 int i;
217224 u32 elrsr;
218225
219226 elrsr = read_gicreg(ICH_ELRSR_EL2);
220227
221− write_gicreg(cpu_if->vgic_hcr & ~ICH_HCR_EL2_En, ICH_HCR_EL2);
222−
223228 for (i = 0; i < used_lrs; i++) {
224229 if (elrsr & (1 << i))
225230 cpu_if->vgic_lr[i] &= ~ICH_LR_STATE;
@@ -229,6 +234,23 @@ void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if) · 3 unmodified lines
229234 __gic_v3_set_lr(0, i);
230235 }
231236 }
237+
238+ cpu_if->vgic_vmcr = read_gicreg(ICH_VMCR_EL2);
239+
240+ if (cpu_if->vgic_hcr & ICH_HCR_EL2_LRENPIE) {
241+ u64 val = read_gicreg(ICH_HCR_EL2);
242+ cpu_if->vgic_hcr &= ~ICH_HCR_EL2_EOIcount;
243+ cpu_if->vgic_hcr |= val & ICH_HCR_EL2_EOIcount;
244+ }
245+
246+ write_gicreg(0, ICH_HCR_EL2);
247+
248+ /*
249+ * Hack alert: On NV, this results in a trap so that the above write
250+ * actually takes effect... No synchronisation is necessary, as we
251+ * only care about the effects when this traps.
252+ */
253+ read_gicreg(ICH_MISR_EL2);
232254 }
233255
234256 void __vgic_v3_restore_state(struct vgic_v3_cpu_if *cpu_if)
@@ -236,12 +258,10 @@ void __vgic_v3_restore_state(struct vgic_v3_cpu_if *cpu_if) · 1 unmodified line
236258 u64 used_lrs = cpu_if->used_lrs;
237259 int i;
238260
239− if (used_lrs || cpu_if->its_vpe.its_vm) {
240− write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2);
261+ write_gicreg(compute_ich_hcr(cpu_if), ICH_HCR_EL2);
241262
242− for (i = 0; i < used_lrs; i++)
243− __gic_v3_set_lr(cpu_if->vgic_lr[i], i);
244− }
263+ for (i = 0; i < used_lrs; i++)
264+ __gic_v3_set_lr(cpu_if->vgic_lr[i], i);
245265
246266 /*
247267 * Ensure that writes to the LRs, and on non-VHE systems ensure that
@@ -307,24 +327,20 @@ void __vgic_v3_activate_traps(struct vgic_v3_cpu_if *cpu_if) · 59 unmodified lines
307327 }
308328
309329 /*
310− * If we need to trap system registers, we must write
311− * ICH_HCR_EL2 anyway, even if no interrupts are being
312− * injected. Note that this also applies if we don't expect
313− * any system register access (no vgic at all).
330+ * If we need to trap system registers, we must write ICH_HCR_EL2
331+ * anyway, even if no interrupts are being injected. Note that this
332+ * also applies if we don't expect any system register access (no
333+ * vgic at all). In any case, no need to provide MI configuration.
314334 */
315335 if (static_branch_unlikely(&vgic_v3_cpuif_trap) ||
316336 cpu_if->its_vpe.its_vm || !cpu_if->vgic_sre)
317− write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2);
337+ write_gicreg(vgic_ich_hcr_trap_bits() | ICH_HCR_EL2_En, ICH_HCR_EL2);
318338 }
319339
320340 void __vgic_v3_deactivate_traps(struct vgic_v3_cpu_if *cpu_if)
321341 {
322342 u64 val;
323343
324− if (!cpu_if->vgic_sre) {
325− cpu_if->vgic_vmcr = read_gicreg(ICH_VMCR_EL2);
326− }
327−
328344 /* Only restore SRE if the host implements the GICv2 interface */
329345 if (static_branch_unlikely(&vgic_v3_has_v2_compat)) {
330346 val = read_gicreg(ICC_SRE_EL2);
@@ -346,7 +362,7 @@ void __vgic_v3_deactivate_traps(struct vgic_v3_cpu_if *cpu_if) · 15 unmodified lines
346362 write_gicreg(0, ICH_HCR_EL2);
347363 }
348364
349−static void __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if)
365+void __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if)
350366 {
351367 u64 val;
352368 u32 nr_pre_bits;
@@ -507,13 +523,6 @@ static void __vgic_v3_write_vmcr(u32 vmcr) · 154 unmodified lines
507523 write_gicreg(vmcr, ICH_VMCR_EL2);
508524 }
509525
510−void __vgic_v3_save_vmcr_aprs(struct vgic_v3_cpu_if *cpu_if)
511−{
512− __vgic_v3_save_aprs(cpu_if);
513− if (cpu_if->vgic_sre)
514− cpu_if->vgic_vmcr = __vgic_v3_read_vmcr();
515−}
516−
517526 void __vgic_v3_restore_vmcr_aprs(struct vgic_v3_cpu_if *cpu_if)
518527 {
519528 __vgic_v3_compat_mode_enable();
@@ -790,7 +799,7 @@ static void __vgic_v3_bump_eoicount(void) · 270 unmodified lines
790799 write_gicreg(hcr, ICH_HCR_EL2);
791800 }
792801
793−static void __vgic_v3_write_dir(struct kvm_vcpu *vcpu, u32 vmcr, int rt)
802+static int ___vgic_v3_write_dir(struct kvm_vcpu *vcpu, u32 vmcr, int rt)
794803 {
795804 u32 vid = vcpu_get_reg(vcpu, rt);
796805 u64 lr_val;
@@ -798,19 +807,25 @@ static void __vgic_v3_write_dir(struct kvm_vcpu *vcpu, u32 vmcr, int rt) · 1 unmodified line
798807
799808 /* EOImode == 0, nothing to be done here */
800809 if (!(vmcr & ICH_VMCR_EOIM_MASK))
801− return;
810+ return 1;
802811
803812 /* No deactivate to be performed on an LPI */
804813 if (vid >= VGIC_MIN_LPI)
805− return;
814+ return 1;
806815
807816 lr = __vgic_v3_find_active_lr(vcpu, vid, &lr_val);
808− if (lr == -1) {
809− __vgic_v3_bump_eoicount();
810− return;
817+ if (lr != -1) {
818+ __vgic_v3_clear_active_lr(lr, lr_val);
819+ return 1;
811820 }
812821
813− __vgic_v3_clear_active_lr(lr, lr_val);
822+ return 0;
823+}
824+
825+static void __vgic_v3_write_dir(struct kvm_vcpu *vcpu, u32 vmcr, int rt)
826+{
827+ if (!___vgic_v3_write_dir(vcpu, vmcr, rt))
828+ __vgic_v3_bump_eoicount();
814829 }
815830
816831 static void __vgic_v3_write_eoir(struct kvm_vcpu *vcpu, u32 vmcr, int rt)
@@ -1245,6 +1260,21 @@ int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu) · 428 unmodified lines
12451260 case SYS_ICC_DIR_EL1:
12461261 if (unlikely(is_read))
12471262 return 0;
1263+ /*
1264+ * Full exit if required to handle overflow deactivation,
1265+ * unless we can emulate it in the LRs (likely the majority
1266+ * of the cases).
1267+ */
1268+ if (vcpu->arch.vgic_cpu.vgic_v3.vgic_hcr & ICH_HCR_EL2_TDIR) {
1269+ int ret;
1270+
1271+ ret = ___vgic_v3_write_dir(vcpu, __vgic_v3_read_vmcr(),
1272+ kvm_vcpu_sys_get_rt(vcpu));
1273+ if (ret)
1274+ __kvm_skip_instr(vcpu);
1275+
1276+ return ret;
1277+ }
12481278 fn = __vgic_v3_write_dir;
12491279 break;
12501280 case SYS_ICC_RPR_EL1:
arch/arm64/kvm/mmu.c+124−8
@@ -904,6 +904,38 @@ static int kvm_init_ipa_range(struct kvm_s2_mmu *mmu, unsigned long type) · 903 unmodified lines
904904 return 0;
905905 }
906906
907+/*
908+ * Assume that @pgt is valid and unlinked from the KVM MMU to free the
909+ * page-table without taking the kvm_mmu_lock and without performing any
910+ * TLB invalidations.
911+ *
912+ * Also, the range of addresses can be large enough to cause need_resched
913+ * warnings, for instance on CONFIG_PREEMPT_NONE kernels. Hence, invoke
914+ * cond_resched() periodically to prevent hogging the CPU for a long time
915+ * and schedule something else, if required.
916+ */
917+static void stage2_destroy_range(struct kvm_pgtable *pgt, phys_addr_t addr,
918+ phys_addr_t end)
919+{
920+ u64 next;
921+
922+ do {
923+ next = stage2_range_addr_end(addr, end);
924+ KVM_PGT_FN(kvm_pgtable_stage2_destroy_range)(pgt, addr,
925+ next - addr);
926+ if (next != end)
927+ cond_resched();
928+ } while (addr = next, addr != end);
929+}
930+
931+static void kvm_stage2_destroy(struct kvm_pgtable *pgt)
932+{
933+ unsigned int ia_bits = VTCR_EL2_IPA(pgt->mmu->vtcr);
934+
935+ stage2_destroy_range(pgt, 0, BIT(ia_bits));
936+ KVM_PGT_FN(kvm_pgtable_stage2_destroy_pgd)(pgt);
937+}
938+
907939 /**
908940 * kvm_init_stage2_mmu - Initialise a S2 MMU structure
909941 * @kvm: The pointer to the KVM structure
@@ -980,7 +1012,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t · 70 unmodified lines
9801012 return 0;
9811013
9821014 out_destroy_pgtable:
983− KVM_PGT_FN(kvm_pgtable_stage2_destroy)(pgt);
1015+ kvm_stage2_destroy(pgt);
9841016 out_free_pgtable:
9851017 kfree(pgt);
9861018 return err;
@@ -1081,7 +1113,7 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) · 94 unmodified lines
10811113 write_unlock(&kvm->mmu_lock);
10821114
10831115 if (pgt) {
1084− KVM_PGT_FN(kvm_pgtable_stage2_destroy)(pgt);
1116+ kvm_stage2_destroy(pgt);
10851117 kfree(pgt);
10861118 }
10871119 }
@@ -1521,6 +1553,16 @@ static void adjust_nested_fault_perms(struct kvm_s2_trans *nested, · 433 unmodified lines
15211553 *prot |= kvm_encode_nested_level(nested);
15221554 }
15231555
1556+static void adjust_nested_exec_perms(struct kvm *kvm,
1557+ struct kvm_s2_trans *nested,
1558+ enum kvm_pgtable_prot *prot)
1559+{
1560+ if (!kvm_s2_trans_exec_el0(kvm, nested))
1561+ *prot &= ~KVM_PGTABLE_PROT_UX;
1562+ if (!kvm_s2_trans_exec_el1(kvm, nested))
1563+ *prot &= ~KVM_PGTABLE_PROT_PX;
1564+}
1565+
15241566 #define KVM_PGTABLE_WALK_MEMABORT_FLAGS (KVM_PGTABLE_WALK_HANDLE_FAULT | KVM_PGTABLE_WALK_SHARED)
15251567
15261568 static int gmem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
@@ -1572,11 +1614,12 @@ static int gmem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · 45 unmodified lines
15721614 if (writable)
15731615 prot |= KVM_PGTABLE_PROT_W;
15741616
1575− if (exec_fault ||
1576− (cpus_have_final_cap(ARM64_HAS_CACHE_DIC) &&
1577− (!nested || kvm_s2_trans_executable(nested))))
1617+ if (exec_fault || cpus_have_final_cap(ARM64_HAS_CACHE_DIC))
15781618 prot |= KVM_PGTABLE_PROT_X;
15791619
1620+ if (nested)
1621+ adjust_nested_exec_perms(kvm, nested, &prot);
1622+
15801623 kvm_fault_lock(kvm);
15811624 if (mmu_invalidate_retry(kvm, mmu_seq)) {
15821625 ret = -EAGAIN;
@@ -1851,11 +1894,13 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · 268 unmodified lines
18511894 prot |= KVM_PGTABLE_PROT_NORMAL_NC;
18521895 else
18531896 prot |= KVM_PGTABLE_PROT_DEVICE;
1854− } else if (cpus_have_final_cap(ARM64_HAS_CACHE_DIC) &&
1855− (!nested || kvm_s2_trans_executable(nested))) {
1897+ } else if (cpus_have_final_cap(ARM64_HAS_CACHE_DIC)) {
18561898 prot |= KVM_PGTABLE_PROT_X;
18571899 }
18581900
1901+ if (nested)
1902+ adjust_nested_exec_perms(kvm, nested, &prot);
1903+
18591904 /*
18601905 * Under the premise of getting a FSC_PERM fault, we just need to relax
18611906 * permissions only if vma_pagesize equals fault_granule. Otherwise,
@@ -1899,8 +1944,48 @@ static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa) · 37 unmodified lines
18991944 read_unlock(&vcpu->kvm->mmu_lock);
19001945 }
19011946
1947+/*
1948+ * Returns true if the SEA should be handled locally within KVM if the abort
1949+ * is caused by a kernel memory allocation (e.g. stage-2 table memory).
1950+ */
1951+static bool host_owns_sea(struct kvm_vcpu *vcpu, u64 esr)
1952+{
1953+ /*
1954+ * Without FEAT_RAS HCR_EL2.TEA is RES0, meaning any external abort
1955+ * taken from a guest EL to EL2 is due to a host-imposed access (e.g.
1956+ * stage-2 PTW).
1957+ */
1958+ if (!cpus_have_final_cap(ARM64_HAS_RAS_EXTN))
1959+ return true;
1960+
1961+ /* KVM owns the VNCR when the vCPU isn't in a nested context. */
1962+ if (is_hyp_ctxt(vcpu) && !kvm_vcpu_trap_is_iabt(vcpu) && (esr & ESR_ELx_VNCR))
1963+ return true;
1964+
1965+ /*
1966+ * Determining if an external abort during a table walk happened at
1967+ * stage-2 is only possible with S1PTW is set. Otherwise, since KVM
1968+ * sets HCR_EL2.TEA, SEAs due to a stage-1 walk (i.e. accessing the
1969+ * PA of the stage-1 descriptor) can reach here and are reported
1970+ * with a TTW ESR value.
1971+ */
1972+ return (esr_fsc_is_sea_ttw(esr) && (esr & ESR_ELx_S1PTW));
1973+}
1974+
19021975 int kvm_handle_guest_sea(struct kvm_vcpu *vcpu)
19031976 {
1977+ struct kvm *kvm = vcpu->kvm;
1978+ struct kvm_run *run = vcpu->run;
1979+ u64 esr = kvm_vcpu_get_esr(vcpu);
1980+ u64 esr_mask = ESR_ELx_EC_MASK |
1981+ ESR_ELx_IL |
1982+ ESR_ELx_FnV |
1983+ ESR_ELx_EA |
1984+ ESR_ELx_CM |
1985+ ESR_ELx_WNR |
1986+ ESR_ELx_FSC;
1987+ u64 ipa;
1988+
19041989 /*
19051990 * Give APEI the opportunity to claim the abort before handling it
19061991 * within KVM. apei_claim_sea() expects to be called with IRQs enabled.
@@ -1909,7 +1994,33 @@ int kvm_handle_guest_sea(struct kvm_vcpu *vcpu) · 2 unmodified lines
19091994 if (apei_claim_sea(NULL) == 0)
19101995 return 1;
19111996
1912− return kvm_inject_serror(vcpu);
1997+ if (host_owns_sea(vcpu, esr) ||
1998+ !test_bit(KVM_ARCH_FLAG_EXIT_SEA, &vcpu->kvm->arch.flags))
1999+ return kvm_inject_serror(vcpu);
2000+
2001+ /* ESR_ELx.SET is RES0 when FEAT_RAS isn't implemented. */
2002+ if (kvm_has_ras(kvm))
2003+ esr_mask |= ESR_ELx_SET_MASK;
2004+
2005+ /*
2006+ * Exit to userspace, and provide faulting guest virtual and physical
2007+ * addresses in case userspace wants to emulate SEA to guest by
2008+ * writing to FAR_ELx and HPFAR_ELx registers.
2009+ */
2010+ memset(&run->arm_sea, 0, sizeof(run->arm_sea));
2011+ run->exit_reason = KVM_EXIT_ARM_SEA;
2012+ run->arm_sea.esr = esr & esr_mask;
2013+
2014+ if (!(esr & ESR_ELx_FnV))
2015+ run->arm_sea.gva = kvm_vcpu_get_hfar(vcpu);
2016+
2017+ ipa = kvm_vcpu_get_fault_ipa(vcpu);
2018+ if (ipa != INVALID_GPA) {
2019+ run->arm_sea.flags |= KVM_EXIT_ARM_SEA_FLAG_GPA_VALID;
2020+ run->arm_sea.gpa = ipa;
2021+ }
2022+
2023+ return 0;
19132024 }
19142025
19152026 /**
@@ -1999,6 +2110,11 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) · 83 unmodified lines
19992110 u32 esr;
20002111
20012112 ret = kvm_walk_nested_s2(vcpu, fault_ipa, &nested_trans);
2113+ if (ret == -EAGAIN) {
2114+ ret = 1;
2115+ goto out_unlock;
2116+ }
2117+
20022118 if (ret) {
20032119 esr = kvm_s2_trans_esr(&nested_trans);
20042120 kvm_inject_s2_fault(vcpu, esr);
arch/arm64/kvm/nested.c+84−39
@@ -124,14 +124,13 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) · 123 unmodified lines
124124 }
125125
126126 struct s2_walk_info {
127− int (*read_desc)(phys_addr_t pa, u64 *desc, void *data);
128− void *data;
129− u64 baddr;
130− unsigned int max_oa_bits;
131− unsigned int pgshift;
132− unsigned int sl;
133− unsigned int t0sz;
134− bool be;
127+ u64 baddr;
128+ unsigned int max_oa_bits;
129+ unsigned int pgshift;
130+ unsigned int sl;
131+ unsigned int t0sz;
132+ bool be;
133+ bool ha;
135134 };
136135
137136 static u32 compute_fsc(int level, u32 fsc)
@@ -199,6 +198,42 @@ static int check_output_size(struct s2_walk_info *wi, phys_addr_t output) · 61 unmodified lines
199198 return 0;
200199 }
201200
201+static int read_guest_s2_desc(struct kvm_vcpu *vcpu, phys_addr_t pa, u64 *desc,
202+ struct s2_walk_info *wi)
203+{
204+ u64 val;
205+ int r;
206+
207+ r = kvm_read_guest(vcpu->kvm, pa, &val, sizeof(val));
208+ if (r)
209+ return r;
210+
211+ /*
212+ * Handle reversedescriptors if endianness differs between the
213+ * host and the guest hypervisor.
214+ */
215+ if (wi->be)
216+ *desc = be64_to_cpu((__force __be64)val);
217+ else
218+ *desc = le64_to_cpu((__force __le64)val);
219+
220+ return 0;
221+}
222+
223+static int swap_guest_s2_desc(struct kvm_vcpu *vcpu, phys_addr_t pa, u64 old, u64 new,
224+ struct s2_walk_info *wi)
225+{
226+ if (wi->be) {
227+ old = (__force u64)cpu_to_be64(old);
228+ new = (__force u64)cpu_to_be64(new);
229+ } else {
230+ old = (__force u64)cpu_to_le64(old);
231+ new = (__force u64)cpu_to_le64(new);
232+ }
233+
234+ return __kvm_at_swap_desc(vcpu->kvm, pa, old, new);
235+}
236+
202237 /*
203238 * This is essentially a C-version of the pseudo code from the ARM ARM
204239 * AArch64.TranslationTableWalk function. I strongly recommend looking at
@@ -206,13 +241,13 @@ static int check_output_size(struct s2_walk_info *wi, phys_addr_t output) · 1 unmodified line
206241 *
207242 * Must be called with the kvm->srcu read lock held
208243 */
209−static int walk_nested_s2_pgd(phys_addr_t ipa,
244+static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa,
210245 struct s2_walk_info *wi, struct kvm_s2_trans *out)
211246 {
212247 int first_block_level, level, stride, input_size, base_lower_bound;
213248 phys_addr_t base_addr;
214249 unsigned int addr_top, addr_bottom;
215− u64 desc; /* page table entry */
250+ u64 desc, new_desc; /* page table entry */
216251 int ret;
217252 phys_addr_t paddr;
218253
@@ -257,28 +292,30 @@ static int walk_nested_s2_pgd(phys_addr_t ipa, · 38 unmodified lines
257292 >> (addr_bottom - 3);
258293
259294 paddr = base_addr | index;
260− ret = wi->read_desc(paddr, &desc, wi->data);
295+ ret = read_guest_s2_desc(vcpu, paddr, &desc, wi);
261296 if (ret < 0)
262297 return ret;
263298
264− /*
265− * Handle reversedescriptors if endianness differs between the
266− * host and the guest hypervisor.
267− */
268− if (wi->be)
269− desc = be64_to_cpu((__force __be64)desc);
270− else
271− desc = le64_to_cpu((__force __le64)desc);
299+ new_desc = desc;
272300
273301 /* Check for valid descriptor at this point */
274− if (!(desc & 1) || ((desc & 3) == 1 && level == 3)) {
302+ if (!(desc & KVM_PTE_VALID)) {
275303 out->esr = compute_fsc(level, ESR_ELx_FSC_FAULT);
276304 out->desc = desc;
277305 return 1;
278306 }
279307
280− /* We're at the final level or block translation level */
281− if ((desc & 3) == 1 || level == 3)
308+ if (FIELD_GET(KVM_PTE_TYPE, desc) == KVM_PTE_TYPE_BLOCK) {
309+ if (level < 3)
310+ break;
311+
312+ out->esr = compute_fsc(level, ESR_ELx_FSC_FAULT);
313+ out->desc = desc;
314+ return 1;
315+ }
316+
317+ /* We're at the final level */
318+ if (level == 3)
282319 break;
283320
284321 if (check_output_size(wi, desc)) {
@@ -305,7 +342,18 @@ static int walk_nested_s2_pgd(phys_addr_t ipa, · 20 unmodified lines
305342 return 1;
306343 }
307344
308− if (!(desc & BIT(10))) {
345+ if (wi->ha)
346+ new_desc |= KVM_PTE_LEAF_ATTR_LO_S2_AF;
347+
348+ if (new_desc != desc) {
349+ ret = swap_guest_s2_desc(vcpu, paddr, desc, new_desc, wi);
350+ if (ret)
351+ return ret;
352+
353+ desc = new_desc;
354+ }
355+
356+ if (!(desc & KVM_PTE_LEAF_ATTR_LO_S2_AF)) {
309357 out->esr = compute_fsc(level, ESR_ELx_FSC_ACCESS);
310358 out->desc = desc;
311359 return 1;
@@ -318,20 +366,13 @@ static int walk_nested_s2_pgd(phys_addr_t ipa, · 6 unmodified lines
318366 (ipa & GENMASK_ULL(addr_bottom - 1, 0));
319367 out->output = paddr;
320368 out->block_size = 1UL << ((3 - level) * stride + wi->pgshift);
321− out->readable = desc & (0b01 << 6);
322− out->writable = desc & (0b10 << 6);
369+ out->readable = desc & KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R;
370+ out->writable = desc & KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W;
323371 out->level = level;
324372 out->desc = desc;
325373 return 0;
326374 }
327375
328−static int read_guest_s2_desc(phys_addr_t pa, u64 *desc, void *data)
329−{
330− struct kvm_vcpu *vcpu = data;
331−
332− return kvm_read_guest(vcpu->kvm, pa, desc, sizeof(*desc));
333−}
334−
335376 static void vtcr_to_walk_info(u64 vtcr, struct s2_walk_info *wi)
336377 {
337378 wi->t0sz = vtcr & TCR_EL2_T0SZ_MASK;
@@ -350,6 +391,8 @@ static void vtcr_to_walk_info(u64 vtcr, struct s2_walk_info *wi) · 12 unmodified lines
350391 /* Global limit for now, should eventually be per-VM */
351392 wi->max_oa_bits = min(get_kvm_ipa_limit(),
352393 ps_to_output_size(FIELD_GET(VTCR_EL2_PS_MASK, vtcr), false));
394+
395+ wi->ha = vtcr & VTCR_EL2_HA;
353396 }
354397
355398 int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,
@@ -364,15 +407,13 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, · 8 unmodified lines
364407 if (!vcpu_has_nv(vcpu))
365408 return 0;
366409
367− wi.read_desc = read_guest_s2_desc;
368− wi.data = vcpu;
369410 wi.baddr = vcpu_read_sys_reg(vcpu, VTTBR_EL2);
370411
371412 vtcr_to_walk_info(vtcr, &wi);
372413
373414 wi.be = vcpu_read_sys_reg(vcpu, SCTLR_EL2) & SCTLR_ELx_EE;
374415
375− ret = walk_nested_s2_pgd(gipa, &wi, result);
416+ ret = walk_nested_s2_pgd(vcpu, gipa, &wi, result);
376417 if (ret)
377418 result->esr |= (kvm_vcpu_get_esr(vcpu) & ~ESR_ELx_FSC);
378419
@@ -788,7 +829,10 @@ int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, struct kvm_s2_trans *trans) · 409 unmodified lines
788829 return 0;
789830
790831 if (kvm_vcpu_trap_is_iabt(vcpu)) {
791− forward_fault = !kvm_s2_trans_executable(trans);
832+ if (vcpu_mode_priv(vcpu))
833+ forward_fault = !kvm_s2_trans_exec_el1(vcpu->kvm, trans);
834+ else
835+ forward_fault = !kvm_s2_trans_exec_el0(vcpu->kvm, trans);
792836 } else {
793837 bool write_fault = kvm_is_write_fault(vcpu);
794838
@@ -1555,12 +1599,13 @@ u64 limit_nv_id_reg(struct kvm *kvm, u32 reg, u64 val) · 760 unmodified lines
15551599 case SYS_ID_AA64MMFR1_EL1:
15561600 val &= ~(ID_AA64MMFR1_EL1_CMOW |
15571601 ID_AA64MMFR1_EL1_nTLBPA |
1558− ID_AA64MMFR1_EL1_ETS |
1559− ID_AA64MMFR1_EL1_XNX |
1560− ID_AA64MMFR1_EL1_HAFDBS);
1602+ ID_AA64MMFR1_EL1_ETS);
1603+
15611604 /* FEAT_E2H0 implies no VHE */
15621605 if (test_bit(KVM_ARM_VCPU_HAS_EL2_E2H0, kvm->arch.vcpu_features))
15631606 val &= ~ID_AA64MMFR1_EL1_VH;
1607+
1608+ val = ID_REG_LIMIT_FIELD_ENUM(val, ID_AA64MMFR1_EL1, HAFDBS, AF);
15641609 break;
15651610
15661611 case SYS_ID_AA64MMFR2_EL1:
arch/arm64/kvm/pkvm.c+9−2
@@ -344,9 +344,16 @@ static int __pkvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 start, u64 e · 343 unmodified lines
344344 return 0;
345345 }
346346
347−void pkvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt)
347+void pkvm_pgtable_stage2_destroy_range(struct kvm_pgtable *pgt,
348+ u64 addr, u64 size)
348349 {
349− __pkvm_pgtable_stage2_unmap(pgt, 0, ~(0ULL));
350+ __pkvm_pgtable_stage2_unmap(pgt, addr, addr + size);
351+}
352+
353+void pkvm_pgtable_stage2_destroy_pgd(struct kvm_pgtable *pgt)
354+{
355+ /* Expected to be called after all pKVM mappings have been released. */
356+ WARN_ON_ONCE(!RB_EMPTY_ROOT(&pgt->pkvm_mappings.rb_root));
350357 }
351358
352359 int pkvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size,
arch/arm64/kvm/ptdump.c+27−8
@@ -31,27 +31,46 @@ static const struct ptdump_prot_bits stage2_pte_bits[] = { · 30 unmodified lines
3131 .val = PTE_VALID,
3232 .set = " ",
3333 .clear = "F",
34− }, {
34+ },
35+ {
3536 .mask = KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R,
3637 .val = KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R,
3738 .set = "R",
3839 .clear = " ",
39− }, {
40+ },
41+ {
4042 .mask = KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W,
4143 .val = KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W,
4244 .set = "W",
4345 .clear = " ",
44− }, {
46+ },
47+ {
4548 .mask = KVM_PTE_LEAF_ATTR_HI_S2_XN,
46− .val = KVM_PTE_LEAF_ATTR_HI_S2_XN,
47− .set = "NX",
48− .clear = "x ",
49− }, {
49+ .val = 0b00UL << __bf_shf(KVM_PTE_LEAF_ATTR_HI_S2_XN),
50+ .set = "px ux ",
51+ },
52+ {
53+ .mask = KVM_PTE_LEAF_ATTR_HI_S2_XN,
54+ .val = 0b01UL << __bf_shf(KVM_PTE_LEAF_ATTR_HI_S2_XN),
55+ .set = "PXNux ",
56+ },
57+ {
58+ .mask = KVM_PTE_LEAF_ATTR_HI_S2_XN,
59+ .val = 0b10UL << __bf_shf(KVM_PTE_LEAF_ATTR_HI_S2_XN),
60+ .set = "PXNUXN",
61+ },
62+ {
63+ .mask = KVM_PTE_LEAF_ATTR_HI_S2_XN,
64+ .val = 0b11UL << __bf_shf(KVM_PTE_LEAF_ATTR_HI_S2_XN),
65+ .set = "px UXN",
66+ },
67+ {
5068 .mask = KVM_PTE_LEAF_ATTR_LO_S2_AF,
5169 .val = KVM_PTE_LEAF_ATTR_LO_S2_AF,
5270 .set = "AF",
5371 .clear = " ",
54− }, {
72+ },
73+ {
5574 .mask = PMD_TYPE_MASK,
5675 .val = PMD_TYPE_SECT,
5776 .set = "BLK",
arch/arm64/kvm/sys_regs.c+23−5
@@ -666,6 +666,21 @@ static bool access_gic_sre(struct kvm_vcpu *vcpu, · 665 unmodified lines
666666 return true;
667667 }
668668
669+static bool access_gic_dir(struct kvm_vcpu *vcpu,
670+ struct sys_reg_params *p,
671+ const struct sys_reg_desc *r)
672+{
673+ if (!kvm_has_gicv3(vcpu->kvm))
674+ return undef_access(vcpu, p, r);
675+
676+ if (!p->is_write)
677+ return undef_access(vcpu, p, r);
678+
679+ vgic_v3_deactivate(vcpu, p->regval);
680+
681+ return true;
682+}
683+
669684 static bool trap_raz_wi(struct kvm_vcpu *vcpu,
670685 struct sys_reg_params *p,
671686 const struct sys_reg_desc *r)
@@ -3373,7 +3388,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { · 2701 unmodified lines
33733388 { SYS_DESC(SYS_ICC_AP1R1_EL1), undef_access },
33743389 { SYS_DESC(SYS_ICC_AP1R2_EL1), undef_access },
33753390 { SYS_DESC(SYS_ICC_AP1R3_EL1), undef_access },
3376− { SYS_DESC(SYS_ICC_DIR_EL1), undef_access },
3391+ { SYS_DESC(SYS_ICC_DIR_EL1), access_gic_dir },
33773392 { SYS_DESC(SYS_ICC_RPR_EL1), undef_access },
33783393 { SYS_DESC(SYS_ICC_SGI1R_EL1), access_gic_sgi },
33793394 { SYS_DESC(SYS_ICC_ASGI1R_EL1), access_gic_sgi },
@@ -3770,7 +3785,8 @@ static bool handle_at_s1e01(struct kvm_vcpu *vcpu, struct sys_reg_params *p, · 390 unmodified lines
37703785 {
37713786 u32 op = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2);
37723787
3773− __kvm_at_s1e01(vcpu, op, p->regval);
3788+ if (__kvm_at_s1e01(vcpu, op, p->regval))
3789+ return false;
37743790
37753791 return true;
37763792 }
@@ -3787,7 +3803,8 @@ static bool handle_at_s1e2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, · 10 unmodified lines
37873803 return false;
37883804 }
37893805
3790− __kvm_at_s1e2(vcpu, op, p->regval);
3806+ if (__kvm_at_s1e2(vcpu, op, p->regval))
3807+ return false;
37913808
37923809 return true;
37933810 }
@@ -3797,7 +3814,8 @@ static bool handle_at_s12(struct kvm_vcpu *vcpu, struct sys_reg_params *p, · 3 unmodified lines
37973814 {
37983815 u32 op = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2);
37993816
3800− __kvm_at_s12(vcpu, op, p->regval);
3817+ if (__kvm_at_s12(vcpu, op, p->regval))
3818+ return false;
38013819
38023820 return true;
38033821 }
@@ -4498,7 +4516,7 @@ static const struct sys_reg_desc cp15_regs[] = { · 694 unmodified lines
44984516 { CP15_SYS_DESC(SYS_ICC_AP1R1_EL1), undef_access },
44994517 { CP15_SYS_DESC(SYS_ICC_AP1R2_EL1), undef_access },
45004518 { CP15_SYS_DESC(SYS_ICC_AP1R3_EL1), undef_access },
4501− { CP15_SYS_DESC(SYS_ICC_DIR_EL1), undef_access },
4519+ { CP15_SYS_DESC(SYS_ICC_DIR_EL1), access_gic_dir },
45024520 { CP15_SYS_DESC(SYS_ICC_RPR_EL1), undef_access },
45034521 { CP15_SYS_DESC(SYS_ICC_IAR1_EL1), undef_access },
45044522 { CP15_SYS_DESC(SYS_ICC_EOIR1_EL1), undef_access },
arch/arm64/kvm/vgic/vgic-init.c+5−4
@@ -198,6 +198,7 @@ static int kvm_vgic_dist_init(struct kvm *kvm, unsigned int nr_spis) · 197 unmodified lines
198198 struct kvm_vcpu *vcpu0 = kvm_get_vcpu(kvm, 0);
199199 int i;
200200
201+ dist->active_spis = (atomic_t)ATOMIC_INIT(0);
201202 dist->spis = kcalloc(nr_spis, sizeof(struct vgic_irq), GFP_KERNEL_ACCOUNT);
202203 if (!dist->spis)
203204 return -ENOMEM;
@@ -363,12 +364,12 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu) · 159 unmodified lines
363364 return ret;
364365 }
365366
366−static void kvm_vgic_vcpu_enable(struct kvm_vcpu *vcpu)
367+static void kvm_vgic_vcpu_reset(struct kvm_vcpu *vcpu)
367368 {
368369 if (kvm_vgic_global_state.type == VGIC_V2)
369− vgic_v2_enable(vcpu);
370+ vgic_v2_reset(vcpu);
370371 else
371− vgic_v3_enable(vcpu);
372+ vgic_v3_reset(vcpu);
372373 }
373374
374375 /*
@@ -415,7 +416,7 @@ int vgic_init(struct kvm *kvm) · 40 unmodified lines
415416 }
416417
417418 kvm_for_each_vcpu(idx, vcpu, kvm)
418− kvm_vgic_vcpu_enable(vcpu);
419+ kvm_vgic_vcpu_reset(vcpu);
419420
420421 ret = kvm_vgic_setup_default_irq_routing(kvm);
421422 if (ret)
arch/arm64/kvm/vgic/vgic-mmio-v2.c+24−0
@@ -359,6 +359,16 @@ static void vgic_mmio_write_vcpuif(struct kvm_vcpu *vcpu, · 358 unmodified lines
359359 vgic_set_vmcr(vcpu, &vmcr);
360360 }
361361
362+static void vgic_mmio_write_dir(struct kvm_vcpu *vcpu,
363+ gpa_t addr, unsigned int len,
364+ unsigned long val)
365+{
366+ if (kvm_vgic_global_state.type == VGIC_V2)
367+ vgic_v2_deactivate(vcpu, val);
368+ else
369+ vgic_v3_deactivate(vcpu, val);
370+}
371+
362372 static unsigned long vgic_mmio_read_apr(struct kvm_vcpu *vcpu,
363373 gpa_t addr, unsigned int len)
364374 {
@@ -482,6 +492,10 @@ static const struct vgic_register_region vgic_v2_cpu_registers[] = { · 117 unmodified lines
482492 REGISTER_DESC_WITH_LENGTH(GIC_CPU_IDENT,
483493 vgic_mmio_read_vcpuif, vgic_mmio_write_vcpuif, 4,
484494 VGIC_ACCESS_32bit),
495+ REGISTER_DESC_WITH_LENGTH_UACCESS(GIC_CPU_DEACTIVATE,
496+ vgic_mmio_read_raz, vgic_mmio_write_dir,
497+ vgic_mmio_read_raz, vgic_mmio_uaccess_write_wi,
498+ 4, VGIC_ACCESS_32bit),
485499 };
486500
487501 unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev)
@@ -494,6 +508,16 @@ unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev) · 6 unmodified lines
494508 return SZ_4K;
495509 }
496510
511+unsigned int vgic_v2_init_cpuif_iodev(struct vgic_io_device *dev)
512+{
513+ dev->regions = vgic_v2_cpu_registers;
514+ dev->nr_regions = ARRAY_SIZE(vgic_v2_cpu_registers);
515+
516+ kvm_iodevice_init(&dev->dev, &kvm_io_gic_ops);
517+
518+ return KVM_VGIC_V2_CPU_SIZE;
519+}
520+
497521 int vgic_v2_has_attr_regs(struct kvm_device *dev, struct kvm_device_attr *attr)
498522 {
499523 const struct vgic_register_region *region;
arch/arm64/kvm/vgic/vgic-mmio.h+1−0
@@ -213,6 +213,7 @@ void vgic_write_irq_line_level_info(struct kvm_vcpu *vcpu, u32 intid, · 212 unmodified lines
213213 const u32 val);
214214
215215 unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
216+unsigned int vgic_v2_init_cpuif_iodev(struct vgic_io_device *dev);
216217
217218 unsigned int vgic_v3_init_dist_iodev(struct vgic_io_device *dev);
218219
arch/arm64/kvm/vgic/vgic-v2.c+220−71
@@ -9,6 +9,7 @@ · 8 unmodified lines
99 #include <kvm/arm_vgic.h>
1010 #include <asm/kvm_mmu.h>
1111
12+#include "vgic-mmio.h"
1213 #include "vgic.h"
1314
1415 static inline void vgic_v2_write_lr(int lr, u32 val)
@@ -26,11 +27,24 @@ void vgic_v2_init_lrs(void) · 11 unmodified lines
2627 vgic_v2_write_lr(i, 0);
2728 }
2829
29−void vgic_v2_set_underflow(struct kvm_vcpu *vcpu)
30+void vgic_v2_configure_hcr(struct kvm_vcpu *vcpu,
31+ struct ap_list_summary *als)
3032 {
3133 struct vgic_v2_cpu_if *cpuif = &vcpu->arch.vgic_cpu.vgic_v2;
3234
33− cpuif->vgic_hcr |= GICH_HCR_UIE;
35+ cpuif->vgic_hcr = GICH_HCR_EN;
36+
37+ if (irqs_pending_outside_lrs(als))
38+ cpuif->vgic_hcr |= GICH_HCR_NPIE;
39+ if (irqs_active_outside_lrs(als))
40+ cpuif->vgic_hcr |= GICH_HCR_LRENPIE;
41+ if (irqs_outside_lrs(als))
42+ cpuif->vgic_hcr |= GICH_HCR_UIE;
43+
44+ cpuif->vgic_hcr |= (cpuif->vgic_vmcr & GICH_VMCR_ENABLE_GRP0_MASK) ?
45+ GICH_HCR_VGrp0DIE : GICH_HCR_VGrp0EIE;
46+ cpuif->vgic_hcr |= (cpuif->vgic_vmcr & GICH_VMCR_ENABLE_GRP1_MASK) ?
47+ GICH_HCR_VGrp1DIE : GICH_HCR_VGrp1EIE;
3448 }
3549
3650 static bool lr_signals_eoi_mi(u32 lr_val)
@@ -39,43 +53,23 @@ static bool lr_signals_eoi_mi(u32 lr_val) · 2 unmodified lines
3953 !(lr_val & GICH_LR_HW);
4054 }
4155
42−/*
43− * transfer the content of the LRs back into the corresponding ap_list:
44− * - active bit is transferred as is
45− * - pending bit is
46− * - transferred as is in case of edge sensitive IRQs
47− * - set to the line-level (resample time) for level sensitive IRQs
48− */
49−void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu)
56+static void vgic_v2_fold_lr(struct kvm_vcpu *vcpu, u32 val)
5057 {
51− struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
52− struct vgic_v2_cpu_if *cpuif = &vgic_cpu->vgic_v2;
53− int lr;
54−
55− DEBUG_SPINLOCK_BUG_ON(!irqs_disabled());
56−
57− cpuif->vgic_hcr &= ~GICH_HCR_UIE;
58−
59− for (lr = 0; lr < vgic_cpu->vgic_v2.used_lrs; lr++) {
60− u32 val = cpuif->vgic_lr[lr];
61− u32 cpuid, intid = val & GICH_LR_VIRTUALID;
62− struct vgic_irq *irq;
63− bool deactivated;
64−
65− /* Extract the source vCPU id from the LR */
66− cpuid = val & GICH_LR_PHYSID_CPUID;
67− cpuid >>= GICH_LR_PHYSID_CPUID_SHIFT;
68− cpuid &= 7;
58+ u32 cpuid, intid = val & GICH_LR_VIRTUALID;
59+ struct vgic_irq *irq;
60+ bool deactivated;
6961
70− /* Notify fds when the guest EOI'ed a level-triggered SPI */
71− if (lr_signals_eoi_mi(val) && vgic_valid_spi(vcpu->kvm, intid))
72− kvm_notify_acked_irq(vcpu->kvm, 0,
73− intid - VGIC_NR_PRIVATE_IRQS);
62+ /* Extract the source vCPU id from the LR */
63+ cpuid = FIELD_GET(GICH_LR_PHYSID_CPUID, val) & 7;
7464
75− irq = vgic_get_vcpu_irq(vcpu, intid);
65+ /* Notify fds when the guest EOI'ed a level-triggered SPI */
66+ if (lr_signals_eoi_mi(val) && vgic_valid_spi(vcpu->kvm, intid))
67+ kvm_notify_acked_irq(vcpu->kvm, 0,
68+ intid - VGIC_NR_PRIVATE_IRQS);
7669
77− raw_spin_lock(&irq->irq_lock);
70+ irq = vgic_get_vcpu_irq(vcpu, intid);
7871
72+ scoped_guard(raw_spinlock, &irq->irq_lock) {
7973 /* Always preserve the active bit, note deactivation */
8074 deactivated = irq->active && !(val & GICH_LR_ACTIVE_BIT);
8175 irq->active = !!(val & GICH_LR_ACTIVE_BIT);
@@ -101,29 +95,139 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu) · 19 unmodified lines
10195 /* Handle resampling for mapped interrupts if required */
10296 vgic_irq_handle_resampling(irq, deactivated, val & GICH_LR_PENDING_BIT);
10397
104− raw_spin_unlock(&irq->irq_lock);
105− vgic_put_irq(vcpu->kvm, irq);
98+ irq->on_lr = false;
10699 }
107100
108− cpuif->used_lrs = 0;
101+ vgic_put_irq(vcpu->kvm, irq);
109102 }
110103
104+static u32 vgic_v2_compute_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq);
105+
111106 /*
112− * Populates the particular LR with the state of a given IRQ:
113− * - for an edge sensitive IRQ the pending state is cleared in struct vgic_irq
114− * - for a level sensitive IRQ the pending state value is unchanged;
115− * it is dictated directly by the input level
116− *
117− * If @irq describes an SGI with multiple sources, we choose the
118− * lowest-numbered source VCPU and clear that bit in the source bitmap.
119− *
120− * The irq_lock must be held by the caller.
107+ * transfer the content of the LRs back into the corresponding ap_list:
108+ * - active bit is transferred as is
109+ * - pending bit is
110+ * - transferred as is in case of edge sensitive IRQs
111+ * - set to the line-level (resample time) for level sensitive IRQs
121112 */
122−void vgic_v2_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr)
113+void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu)
114+{
115+ struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
116+ struct vgic_v2_cpu_if *cpuif = &vgic_cpu->vgic_v2;
117+ u32 eoicount = FIELD_GET(GICH_HCR_EOICOUNT, cpuif->vgic_hcr);
118+ struct vgic_irq *irq;
119+
120+ DEBUG_SPINLOCK_BUG_ON(!irqs_disabled());
121+
122+ for (int lr = 0; lr < vgic_cpu->vgic_v2.used_lrs; lr++)
123+ vgic_v2_fold_lr(vcpu, cpuif->vgic_lr[lr]);
124+
125+ /* See the GICv3 equivalent for the EOIcount handling rationale */
126+ list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list) {
127+ u32 lr;
128+
129+ if (!eoicount) {
130+ break;
131+ } else {
132+ guard(raw_spinlock)(&irq->irq_lock);
133+
134+ if (!(likely(vgic_target_oracle(irq) == vcpu) &&
135+ irq->active))
136+ continue;
137+
138+ lr = vgic_v2_compute_lr(vcpu, irq) & ~GICH_LR_ACTIVE_BIT;
139+ }
140+
141+ if (lr & GICH_LR_HW)
142+ writel_relaxed(FIELD_GET(GICH_LR_PHYSID_CPUID, lr),
143+ kvm_vgic_global_state.gicc_base + GIC_CPU_DEACTIVATE);
144+ vgic_v2_fold_lr(vcpu, lr);
145+ eoicount--;
146+ }
147+
148+ cpuif->used_lrs = 0;
149+}
150+
151+void vgic_v2_deactivate(struct kvm_vcpu *vcpu, u32 val)
152+{
153+ struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
154+ struct vgic_v2_cpu_if *cpuif = &vgic_cpu->vgic_v2;
155+ struct kvm_vcpu *target_vcpu = NULL;
156+ bool mmio = false;
157+ struct vgic_irq *irq;
158+ unsigned long flags;
159+ u64 lr = 0;
160+ u8 cpuid;
161+
162+ /* Snapshot CPUID, and remove it from the INTID */
163+ cpuid = FIELD_GET(GENMASK_ULL(12, 10), val);
164+ val &= ~GENMASK_ULL(12, 10);
165+
166+ /* We only deal with DIR when EOIMode==1 */
167+ if (!(cpuif->vgic_vmcr & GICH_VMCR_EOI_MODE_MASK))
168+ return;
169+
170+ /* Make sure we're in the same context as LR handling */
171+ local_irq_save(flags);
172+
173+ irq = vgic_get_vcpu_irq(vcpu, val);
174+ if (WARN_ON_ONCE(!irq))
175+ goto out;
176+
177+ /* See the corresponding v3 code for the rationale */
178+ scoped_guard(raw_spinlock, &irq->irq_lock) {
179+ target_vcpu = irq->vcpu;
180+
181+ /* Not on any ap_list? */
182+ if (!target_vcpu)
183+ goto put;
184+
185+ /*
186+ * Urgh. We're deactivating something that we cannot
187+ * observe yet... Big hammer time.
188+ */
189+ if (irq->on_lr) {
190+ mmio = true;
191+ goto put;
192+ }
193+
194+ /* SGI: check that the cpuid matches */
195+ if (val < VGIC_NR_SGIS && irq->active_source != cpuid) {
196+ target_vcpu = NULL;
197+ goto put;
198+ }
199+
200+ /* (with a Dalek voice) DEACTIVATE!!!! */
201+ lr = vgic_v2_compute_lr(vcpu, irq) & ~GICH_LR_ACTIVE_BIT;
202+ }
203+
204+ if (lr & GICH_LR_HW)
205+ writel_relaxed(FIELD_GET(GICH_LR_PHYSID_CPUID, lr),
206+ kvm_vgic_global_state.gicc_base + GIC_CPU_DEACTIVATE);
207+
208+ vgic_v2_fold_lr(vcpu, lr);
209+
210+put:
211+ vgic_put_irq(vcpu->kvm, irq);
212+
213+out:
214+ local_irq_restore(flags);
215+
216+ if (mmio)
217+ vgic_mmio_write_cactive(vcpu, (val / 32) * 4, 4, BIT(val % 32));
218+
219+ /* Force the ap_list to be pruned */
220+ if (target_vcpu)
221+ kvm_make_request(KVM_REQ_VGIC_PROCESS_UPDATE, target_vcpu);
222+}
223+
224+static u32 vgic_v2_compute_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq)
123225 {
124226 u32 val = irq->intid;
125227 bool allow_pending = true;
126228
229+ WARN_ON(irq->on_lr);
230+
127231 if (irq->active) {
128232 val |= GICH_LR_ACTIVE_BIT;
129233 if (vgic_irq_is_sgi(irq->intid))
@@ -163,22 +267,52 @@ void vgic_v2_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr) · 33 unmodified lines
163267 if (allow_pending && irq_is_pending(irq)) {
164268 val |= GICH_LR_PENDING_BIT;
165269
166− if (irq->config == VGIC_CONFIG_EDGE)
167− irq->pending_latch = false;
168−
169270 if (vgic_irq_is_sgi(irq->intid)) {
170271 u32 src = ffs(irq->source);
171272
172273 if (WARN_RATELIMIT(!src, "No SGI source for INTID %d\n",
173274 irq->intid))
174− return;
275+ return 0;
175276
176277 val |= (src - 1) << GICH_LR_PHYSID_CPUID_SHIFT;
177− irq->source &= ~(1 << (src - 1));
178− if (irq->source) {
179− irq->pending_latch = true;
278+ if (irq->source & ~BIT(src - 1))
180279 val |= GICH_LR_EOI;
181− }
280+ }
281+ }
282+
283+ /* The GICv2 LR only holds five bits of priority. */
284+ val |= (irq->priority >> 3) << GICH_LR_PRIORITY_SHIFT;
285+
286+ return val;
287+}
288+
289+/*
290+ * Populates the particular LR with the state of a given IRQ:
291+ * - for an edge sensitive IRQ the pending state is cleared in struct vgic_irq
292+ * - for a level sensitive IRQ the pending state value is unchanged;
293+ * it is dictated directly by the input level
294+ *
295+ * If @irq describes an SGI with multiple sources, we choose the
296+ * lowest-numbered source VCPU and clear that bit in the source bitmap.
297+ *
298+ * The irq_lock must be held by the caller.
299+ */
300+void vgic_v2_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr)
301+{
302+ u32 val = vgic_v2_compute_lr(vcpu, irq);
303+
304+ vcpu->arch.vgic_cpu.vgic_v2.vgic_lr[lr] = val;
305+
306+ if (val & GICH_LR_PENDING_BIT) {
307+ if (irq->config == VGIC_CONFIG_EDGE)
308+ irq->pending_latch = false;
309+
310+ if (vgic_irq_is_sgi(irq->intid)) {
311+ u32 src = ffs(irq->source);
312+
313+ irq->source &= ~BIT(src - 1);
314+ if (irq->source)
315+ irq->pending_latch = true;
182316 }
183317 }
184318
@@ -194,7 +328,7 @@ void vgic_v2_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr) · 9 unmodified lines
194328 /* The GICv2 LR only holds five bits of priority. */
195329 val |= (irq->priority >> 3) << GICH_LR_PRIORITY_SHIFT;
196330
197− vcpu->arch.vgic_cpu.vgic_v2.vgic_lr[lr] = val;
331+ irq->on_lr = true;
198332 }
199333
200334 void vgic_v2_clear_lr(struct kvm_vcpu *vcpu, int lr)
@@ -257,7 +391,7 @@ void vgic_v2_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcrp) · 56 unmodified lines
257391 GICH_VMCR_PRIMASK_SHIFT) << GICV_PMR_PRIORITY_SHIFT;
258392 }
259393
260−void vgic_v2_enable(struct kvm_vcpu *vcpu)
394+void vgic_v2_reset(struct kvm_vcpu *vcpu)
261395 {
262396 /*
263397 * By forcing VMCR to zero, the GIC will restore the binary
@@ -265,9 +399,6 @@ void vgic_v2_enable(struct kvm_vcpu *vcpu) · 1 unmodified line
265399 * anyway.
266400 */
267401 vcpu->arch.vgic_cpu.vgic_v2.vgic_vmcr = 0;
268−
269− /* Get the show on the road... */
270− vcpu->arch.vgic_cpu.vgic_v2.vgic_hcr = GICH_HCR_EN;
271402 }
272403
273404 /* check for overlapping regions and for regions crossing the end of memory */
@@ -289,6 +420,7 @@ static bool vgic_v2_check_base(gpa_t dist_base, gpa_t cpu_base) · 15 unmodified lines
289420 int vgic_v2_map_resources(struct kvm *kvm)
290421 {
291422 struct vgic_dist *dist = &kvm->arch.vgic;
423+ unsigned int len;
292424 int ret = 0;
293425
294426 if (IS_VGIC_ADDR_UNDEF(dist->vgic_dist_base) ||
@@ -312,10 +444,20 @@ int vgic_v2_map_resources(struct kvm *kvm) · 17 unmodified lines
312444 return ret;
313445 }
314446
447+ len = vgic_v2_init_cpuif_iodev(&dist->cpuif_iodev);
448+ dist->cpuif_iodev.base_addr = dist->vgic_cpu_base;
449+ dist->cpuif_iodev.iodev_type = IODEV_CPUIF;
450+ dist->cpuif_iodev.redist_vcpu = NULL;
451+
452+ ret = kvm_io_bus_register_dev(kvm, KVM_MMIO_BUS, dist->vgic_cpu_base,
453+ len, &dist->cpuif_iodev.dev);
454+ if (ret)
455+ return ret;
456+
315457 if (!static_branch_unlikely(&vgic_v2_cpuif_trap)) {
316458 ret = kvm_phys_addr_ioremap(kvm, dist->vgic_cpu_base,
317459 kvm_vgic_global_state.vcpu_base,
318− KVM_VGIC_V2_CPU_SIZE, true);
460+ KVM_VGIC_V2_CPU_SIZE - SZ_4K, true);
319461 if (ret) {
320462 kvm_err("Unable to remap VGIC CPU to VCPU\n");
321463 return ret;
@@ -385,6 +527,7 @@ int vgic_v2_probe(const struct gic_kvm_info *info) · 63 unmodified lines
385527
386528 kvm_vgic_global_state.can_emulate_gicv2 = true;
387529 kvm_vgic_global_state.vcpu_base = info->vcpu.start;
530+ kvm_vgic_global_state.gicc_base = info->gicc_base;
388531 kvm_vgic_global_state.type = VGIC_V2;
389532 kvm_vgic_global_state.max_gic_vcpus = VGIC_V2_MAX_CPUS;
390533
@@ -423,16 +566,26 @@ static void save_lrs(struct kvm_vcpu *vcpu, void __iomem *base) · 32 unmodified lines
423566
424567 void vgic_v2_save_state(struct kvm_vcpu *vcpu)
425568 {
569+ struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
426570 void __iomem *base = kvm_vgic_global_state.vctrl_base;
427571 u64 used_lrs = vcpu->arch.vgic_cpu.vgic_v2.used_lrs;
428572
429573 if (!base)
430574 return;
431575
432− if (used_lrs) {
576+ cpu_if->vgic_vmcr = readl_relaxed(kvm_vgic_global_state.vctrl_base + GICH_VMCR);
577+
578+ if (used_lrs)
433579 save_lrs(vcpu, base);
434− writel_relaxed(0, base + GICH_HCR);
580+
581+ if (cpu_if->vgic_hcr & GICH_HCR_LRENPIE) {
582+ u32 val = readl_relaxed(base + GICH_HCR);
583+
584+ cpu_if->vgic_hcr &= ~GICH_HCR_EOICOUNT;
585+ cpu_if->vgic_hcr |= val & GICH_HCR_EOICOUNT;
435586 }
587+
588+ writel_relaxed(0, base + GICH_HCR);
436589 }
437590
438591 void vgic_v2_restore_state(struct kvm_vcpu *vcpu)
@@ -445,13 +598,10 @@ void vgic_v2_restore_state(struct kvm_vcpu *vcpu) · 6 unmodified lines
445598 if (!base)
446599 return;
447600
448− if (used_lrs) {
449− writel_relaxed(cpu_if->vgic_hcr, base + GICH_HCR);
450− for (i = 0; i < used_lrs; i++) {
451− writel_relaxed(cpu_if->vgic_lr[i],
452− base + GICH_LR0 + (i * 4));
453− }
454− }
601+ writel_relaxed(cpu_if->vgic_hcr, base + GICH_HCR);
602+
603+ for (i = 0; i < used_lrs; i++)
604+ writel_relaxed(cpu_if->vgic_lr[i], base + GICH_LR0 + (i * 4));
455605 }
456606
457607 void vgic_v2_load(struct kvm_vcpu *vcpu)
@@ -468,6 +618,5 @@ void vgic_v2_put(struct kvm_vcpu *vcpu) · 10 unmodified lines
468618 {
469619 struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
470620
471− cpu_if->vgic_vmcr = readl_relaxed(kvm_vgic_global_state.vctrl_base + GICH_VMCR);
472621 cpu_if->vgic_apr = readl_relaxed(kvm_vgic_global_state.vctrl_base + GICH_APR);
473622 }
arch/arm64/kvm/vgic/vgic-v3-nested.c+52−52
@@ -70,13 +70,14 @@ static int lr_map_idx_to_shadow_idx(struct shadow_if *shadow_if, int idx) · 69 unmodified lines
7070 * - on L2 put: perform the inverse transformation, so that the result of L2
7171 * running becomes visible to L1 in the VNCR-accessible registers.
7272 *
73− * - there is nothing to do on L2 entry, as everything will have happened
74− * on load. However, this is the point where we detect that an interrupt
75− * targeting L1 and prepare the grand switcheroo.
73+ * - there is nothing to do on L2 entry apart from enabling the vgic, as
74+ * everything will have happened on load. However, this is the point where
75+ * we detect that an interrupt targeting L1 and prepare the grand
76+ * switcheroo.
7677 *
77− * - on L2 exit: emulate the HW bit, and deactivate corresponding the L1
78− * interrupt. The L0 active state will be cleared by the HW if the L1
79− * interrupt was itself backed by a HW interrupt.
78+ * - on L2 exit: resync the LRs and VMCR, emulate the HW bit, and deactivate
79+ * corresponding the L1 interrupt. The L0 active state will be cleared by
80+ * the HW if the L1 interrupt was itself backed by a HW interrupt.
8081 *
8182 * Maintenance Interrupt (MI) management:
8283 *
@@ -93,8 +94,10 @@ static int lr_map_idx_to_shadow_idx(struct shadow_if *shadow_if, int idx) · 10 unmodified lines
9394 *
9495 * - because most of the ICH_*_EL2 registers live in the VNCR page, the
9596 * quality of emulation is poor: L1 can setup the vgic so that an MI would
96− * immediately fire, and not observe anything until the next exit. Trying
97− * to read ICH_MISR_EL2 would do the trick, for example.
97+ * immediately fire, and not observe anything until the next exit.
98+ * Similarly, a pending MI is not immediately disabled by clearing
99+ * ICH_HCR_EL2.En. Trying to read ICH_MISR_EL2 would do the trick, for
100+ * example.
98101 *
99102 * System register emulation:
100103 *
@@ -265,16 +268,37 @@ static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu, · 164 unmodified lines
265268 s_cpu_if->used_lrs = hweight16(shadow_if->lr_map);
266269 }
267270
271+void vgic_v3_flush_nested(struct kvm_vcpu *vcpu)
272+{
273+ u64 val = __vcpu_sys_reg(vcpu, ICH_HCR_EL2);
274+
275+ write_sysreg_s(val | vgic_ich_hcr_trap_bits(), SYS_ICH_HCR_EL2);
276+}
277+
268278 void vgic_v3_sync_nested(struct kvm_vcpu *vcpu)
269279 {
270280 struct shadow_if *shadow_if = get_shadow_if();
271281 int i;
272282
273283 for_each_set_bit(i, &shadow_if->lr_map, kvm_vgic_global_state.nr_lr) {
274− u64 lr = __vcpu_sys_reg(vcpu, ICH_LRN(i));
275− struct vgic_irq *irq;
284+ u64 val, host_lr, lr;
285+
286+ host_lr = __gic_v3_get_lr(lr_map_idx_to_shadow_idx(shadow_if, i));
287+
288+ /* Propagate the new LR state */
289+ lr = __vcpu_sys_reg(vcpu, ICH_LRN(i));
290+ val = lr & ~ICH_LR_STATE;
291+ val |= host_lr & ICH_LR_STATE;
292+ __vcpu_assign_sys_reg(vcpu, ICH_LRN(i), val);
276293
277− if (!(lr & ICH_LR_HW) || !(lr & ICH_LR_STATE))
294+ /*
295+ * Deactivation of a HW interrupt: the LR must have the HW
296+ * bit set, have been in a non-invalid state before the run,
297+ * and now be in an invalid state. If any of that doesn't
298+ * hold, we're done with this LR.
299+ */
300+ if (!((lr & ICH_LR_HW) && (lr & ICH_LR_STATE) &&
301+ !(host_lr & ICH_LR_STATE)))
278302 continue;
279303
280304 /*
@@ -282,35 +306,27 @@ void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) · 1 unmodified line
282306 * need to emulate the HW effect between the guest hypervisor
283307 * and the nested guest.
284308 */
285− irq = vgic_get_vcpu_irq(vcpu, FIELD_GET(ICH_LR_PHYS_ID_MASK, lr));
286− if (WARN_ON(!irq)) /* Shouldn't happen as we check on load */
287− continue;
309+ vgic_v3_deactivate(vcpu, FIELD_GET(ICH_LR_PHYS_ID_MASK, lr));
310+ }
288311
289− lr = __gic_v3_get_lr(lr_map_idx_to_shadow_idx(shadow_if, i));
290− if (!(lr & ICH_LR_STATE))
291− irq->active = false;
312+ /* We need these to be synchronised to generate the MI */
313+ __vcpu_assign_sys_reg(vcpu, ICH_VMCR_EL2, read_sysreg_s(SYS_ICH_VMCR_EL2));
314+ __vcpu_rmw_sys_reg(vcpu, ICH_HCR_EL2, &=, ~ICH_HCR_EL2_EOIcount);
315+ __vcpu_rmw_sys_reg(vcpu, ICH_HCR_EL2, |=, read_sysreg_s(SYS_ICH_HCR_EL2) & ICH_HCR_EL2_EOIcount);
292316
293− vgic_put_irq(vcpu->kvm, irq);
294− }
317+ write_sysreg_s(0, SYS_ICH_HCR_EL2);
318+ isb();
319+
320+ vgic_v3_nested_update_mi(vcpu);
295321 }
296322
297323 static void vgic_v3_create_shadow_state(struct kvm_vcpu *vcpu,
298324 struct vgic_v3_cpu_if *s_cpu_if)
299325 {
300326 struct vgic_v3_cpu_if *host_if = &vcpu->arch.vgic_cpu.vgic_v3;
301− u64 val = 0;
302327 int i;
303328
304− /*
305− * If we're on a system with a broken vgic that requires
306− * trapping, propagate the trapping requirements.
307− *
308− * Ah, the smell of rotten fruits...
309− */
310− if (static_branch_unlikely(&vgic_v3_cpuif_trap))
311− val = host_if->vgic_hcr & (ICH_HCR_EL2_TALL0 | ICH_HCR_EL2_TALL1 |
312− ICH_HCR_EL2_TC | ICH_HCR_EL2_TDIR);
313− s_cpu_if->vgic_hcr = __vcpu_sys_reg(vcpu, ICH_HCR_EL2) | val;
329+ s_cpu_if->vgic_hcr = __vcpu_sys_reg(vcpu, ICH_HCR_EL2);
314330 s_cpu_if->vgic_vmcr = __vcpu_sys_reg(vcpu, ICH_VMCR_EL2);
315331 s_cpu_if->vgic_sre = host_if->vgic_sre;
316332
@@ -334,7 +350,8 @@ void vgic_v3_load_nested(struct kvm_vcpu *vcpu) · 17 unmodified lines
334350 __vgic_v3_restore_vmcr_aprs(cpu_if);
335351 __vgic_v3_activate_traps(cpu_if);
336352
337− __vgic_v3_restore_state(cpu_if);
353+ for (int i = 0; i < cpu_if->used_lrs; i++)
354+ __gic_v3_set_lr(cpu_if->vgic_lr[i], i);
338355
339356 /*
340357 * Propagate the number of used LRs for the benefit of the HYP
@@ -347,36 +364,19 @@ void vgic_v3_put_nested(struct kvm_vcpu *vcpu) · 6 unmodified lines
347364 {
348365 struct shadow_if *shadow_if = get_shadow_if();
349366 struct vgic_v3_cpu_if *s_cpu_if = &shadow_if->cpuif;
350− u64 val;
351367 int i;
352368
353− __vgic_v3_save_vmcr_aprs(s_cpu_if);
354− __vgic_v3_deactivate_traps(s_cpu_if);
355− __vgic_v3_save_state(s_cpu_if);
356−
357− /*
358− * Translate the shadow state HW fields back to the virtual ones
359− * before copying the shadow struct back to the nested one.
360− */
361− val = __vcpu_sys_reg(vcpu, ICH_HCR_EL2);
362− val &= ~ICH_HCR_EL2_EOIcount_MASK;
363− val |= (s_cpu_if->vgic_hcr & ICH_HCR_EL2_EOIcount_MASK);
364− __vcpu_assign_sys_reg(vcpu, ICH_HCR_EL2, val);
365− __vcpu_assign_sys_reg(vcpu, ICH_VMCR_EL2, s_cpu_if->vgic_vmcr);
369+ __vgic_v3_save_aprs(s_cpu_if);
366370
367371 for (i = 0; i < 4; i++) {
368372 __vcpu_assign_sys_reg(vcpu, ICH_AP0RN(i), s_cpu_if->vgic_ap0r[i]);
369373 __vcpu_assign_sys_reg(vcpu, ICH_AP1RN(i), s_cpu_if->vgic_ap1r[i]);
370374 }
371375
372− for_each_set_bit(i, &shadow_if->lr_map, kvm_vgic_global_state.nr_lr) {
373− val = __vcpu_sys_reg(vcpu, ICH_LRN(i));
374−
375− val &= ~ICH_LR_STATE;
376− val |= s_cpu_if->vgic_lr[lr_map_idx_to_shadow_idx(shadow_if, i)] & ICH_LR_STATE;
376+ for (i = 0; i < s_cpu_if->used_lrs; i++)
377+ __gic_v3_set_lr(0, i);
377378
378− __vcpu_assign_sys_reg(vcpu, ICH_LRN(i), val);
379− }
379+ __vgic_v3_deactivate_traps(s_cpu_if);
380380
381381 vcpu->arch.vgic_cpu.vgic_v3.used_lrs = 0;
382382 }
arch/arm64/kvm/vgic/vgic-v3.c+328−98
@@ -12,6 +12,7 @@ · 11 unmodified lines
1212 #include <asm/kvm_mmu.h>
1313 #include <asm/kvm_asm.h>
1414
15+#include "vgic-mmio.h"
1516 #include "vgic.h"
1617
1718 static bool group0_trap;
@@ -20,11 +21,48 @@ static bool common_trap; · 2 unmodified lines
2021 static bool dir_trap;
2122 static bool gicv4_enable;
2223
23−void vgic_v3_set_underflow(struct kvm_vcpu *vcpu)
24+void vgic_v3_configure_hcr(struct kvm_vcpu *vcpu,
25+ struct ap_list_summary *als)
2426 {
2527 struct vgic_v3_cpu_if *cpuif = &vcpu->arch.vgic_cpu.vgic_v3;
2628
27− cpuif->vgic_hcr |= ICH_HCR_EL2_UIE;
29+ if (!irqchip_in_kernel(vcpu->kvm))
30+ return;
31+
32+ cpuif->vgic_hcr = ICH_HCR_EL2_En;
33+
34+ if (irqs_pending_outside_lrs(als))
35+ cpuif->vgic_hcr |= ICH_HCR_EL2_NPIE;
36+ if (irqs_active_outside_lrs(als))
37+ cpuif->vgic_hcr |= ICH_HCR_EL2_LRENPIE;
38+ if (irqs_outside_lrs(als))
39+ cpuif->vgic_hcr |= ICH_HCR_EL2_UIE;
40+
41+ if (!als->nr_sgi)
42+ cpuif->vgic_hcr |= ICH_HCR_EL2_vSGIEOICount;
43+
44+ cpuif->vgic_hcr |= (cpuif->vgic_vmcr & ICH_VMCR_ENG0_MASK) ?
45+ ICH_HCR_EL2_VGrp0DIE : ICH_HCR_EL2_VGrp0EIE;
46+ cpuif->vgic_hcr |= (cpuif->vgic_vmcr & ICH_VMCR_ENG1_MASK) ?
47+ ICH_HCR_EL2_VGrp1DIE : ICH_HCR_EL2_VGrp1EIE;
48+
49+ /*
50+ * Dealing with EOImode=1 is a massive source of headache. Not
51+ * only do we need to track that we have active interrupts
52+ * outside of the LRs and force DIR to be trapped, we also
53+ * need to deal with SPIs that can be deactivated on another
54+ * CPU.
55+ *
56+ * On systems that do not implement TDIR, force the bit in the
57+ * shadow state anyway to avoid IPI-ing on these poor sods.
58+ *
59+ * Note that we set the trap irrespective of EOIMode, as that
60+ * can change behind our back without any warning...
61+ */
62+ if (!cpus_have_final_cap(ARM64_HAS_ICH_HCR_EL2_TDIR) ||
63+ irqs_active_outside_lrs(als) ||
64+ atomic_read(&vcpu->kvm->arch.vgic.active_spis))
65+ cpuif->vgic_hcr |= ICH_HCR_EL2_TDIR;
2866 }
2967
3068 static bool lr_signals_eoi_mi(u64 lr_val)
@@ -33,84 +71,238 @@ static bool lr_signals_eoi_mi(u64 lr_val) · 2 unmodified lines
3371 !(lr_val & ICH_LR_HW);
3472 }
3573
74+static void vgic_v3_fold_lr(struct kvm_vcpu *vcpu, u64 val)
75+{
76+ struct vgic_irq *irq;
77+ bool is_v2_sgi = false;
78+ bool deactivated;
79+ u32 intid;
80+
81+ if (vcpu->kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) {
82+ intid = val & ICH_LR_VIRTUAL_ID_MASK;
83+ } else {
84+ intid = val & GICH_LR_VIRTUALID;
85+ is_v2_sgi = vgic_irq_is_sgi(intid);
86+ }
87+
88+ irq = vgic_get_vcpu_irq(vcpu, intid);
89+ if (!irq) /* An LPI could have been unmapped. */
90+ return;
91+
92+ scoped_guard(raw_spinlock, &irq->irq_lock) {
93+ /* Always preserve the active bit for !LPIs, note deactivation */
94+ if (irq->intid >= VGIC_MIN_LPI)
95+ val &= ~ICH_LR_ACTIVE_BIT;
96+ deactivated = irq->active && !(val & ICH_LR_ACTIVE_BIT);
97+ irq->active = !!(val & ICH_LR_ACTIVE_BIT);
98+
99+ /* Edge is the only case where we preserve the pending bit */
100+ if (irq->config == VGIC_CONFIG_EDGE &&
101+ (val & ICH_LR_PENDING_BIT))
102+ irq->pending_latch = true;
103+
104+ /*
105+ * Clear soft pending state when level irqs have been acked.
106+ */
107+ if (irq->config == VGIC_CONFIG_LEVEL && !(val & ICH_LR_STATE))
108+ irq->pending_latch = false;
109+
110+ if (is_v2_sgi) {
111+ u8 cpuid = FIELD_GET(GICH_LR_PHYSID_CPUID, val);
112+
113+ if (irq->active)
114+ irq->active_source = cpuid;
115+
116+ if (val & ICH_LR_PENDING_BIT)
117+ irq->source |= BIT(cpuid);
118+ }
119+
120+ /* Handle resampling for mapped interrupts if required */
121+ vgic_irq_handle_resampling(irq, deactivated, val & ICH_LR_PENDING_BIT);
122+
123+ irq->on_lr = false;
124+ }
125+
126+ /* Notify fds when the guest EOI'ed a level-triggered SPI, and drop the refcount */
127+ if (deactivated && lr_signals_eoi_mi(val) && vgic_valid_spi(vcpu->kvm, intid)) {
128+ kvm_notify_acked_irq(vcpu->kvm, 0,
129+ intid - VGIC_NR_PRIVATE_IRQS);
130+ atomic_dec_if_positive(&vcpu->kvm->arch.vgic.active_spis);
131+ }
132+
133+ vgic_put_irq(vcpu->kvm, irq);
134+}
135+
136+static u64 vgic_v3_compute_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq);
137+
138+static void vgic_v3_deactivate_phys(u32 intid)
139+{
140+ if (cpus_have_final_cap(ARM64_HAS_GICV5_LEGACY))
141+ gic_insn(intid | FIELD_PREP(GICV5_GIC_CDDI_TYPE_MASK, 1), CDDI);
142+ else
143+ gic_write_dir(intid);
144+}
145+
36146 void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu)
37147 {
38148 struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
39149 struct vgic_v3_cpu_if *cpuif = &vgic_cpu->vgic_v3;
40− u32 model = vcpu->kvm->arch.vgic.vgic_model;
41− int lr;
150+ u32 eoicount = FIELD_GET(ICH_HCR_EL2_EOIcount, cpuif->vgic_hcr);
151+ struct vgic_irq *irq;
42152
43153 DEBUG_SPINLOCK_BUG_ON(!irqs_disabled());
44154
45− cpuif->vgic_hcr &= ~ICH_HCR_EL2_UIE;
46−
47− for (lr = 0; lr < cpuif->used_lrs; lr++) {
48− u64 val = cpuif->vgic_lr[lr];
49− u32 intid, cpuid;
50− struct vgic_irq *irq;
51− bool is_v2_sgi = false;
52− bool deactivated;
155+ for (int lr = 0; lr < cpuif->used_lrs; lr++)
156+ vgic_v3_fold_lr(vcpu, cpuif->vgic_lr[lr]);
53157
54− cpuid = val & GICH_LR_PHYSID_CPUID;
55− cpuid >>= GICH_LR_PHYSID_CPUID_SHIFT;
158+ /*
159+ * EOIMode=0: use EOIcount to emulate deactivation. We are
160+ * guaranteed to deactivate in reverse order of the activation, so
161+ * just pick one active interrupt after the other in the ap_list,
162+ * and replay the deactivation as if the CPU was doing it. We also
163+ * rely on priority drop to have taken place, and the list to be
164+ * sorted by priority.
165+ */
166+ list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list) {
167+ u64 lr;
56168
57− if (model == KVM_DEV_TYPE_ARM_VGIC_V3) {
58− intid = val & ICH_LR_VIRTUAL_ID_MASK;
169+ /*
170+ * I would have loved to write this using a scoped_guard(),
171+ * but using 'continue' here is a total train wreck.
172+ */
173+ if (!eoicount) {
174+ break;
59175 } else {
60− intid = val & GICH_LR_VIRTUALID;
61− is_v2_sgi = vgic_irq_is_sgi(intid);
176+ guard(raw_spinlock)(&irq->irq_lock);
177+
178+ if (!(likely(vgic_target_oracle(irq) == vcpu) &&
179+ irq->active))
180+ continue;
181+
182+ lr = vgic_v3_compute_lr(vcpu, irq) & ~ICH_LR_ACTIVE_BIT;
62183 }
63184
64− /* Notify fds when the guest EOI'ed a level-triggered IRQ */
65− if (lr_signals_eoi_mi(val) && vgic_valid_spi(vcpu->kvm, intid))
66− kvm_notify_acked_irq(vcpu->kvm, 0,
67− intid - VGIC_NR_PRIVATE_IRQS);
185+ if (lr & ICH_LR_HW)
186+ vgic_v3_deactivate_phys(FIELD_GET(ICH_LR_PHYS_ID_MASK, lr));
68187
69− irq = vgic_get_vcpu_irq(vcpu, intid);
70− if (!irq) /* An LPI could have been unmapped. */
71− continue;
188+ vgic_v3_fold_lr(vcpu, lr);
189+ eoicount--;
190+ }
72191
73− raw_spin_lock(&irq->irq_lock);
192+ cpuif->used_lrs = 0;
193+}
74194
75− /* Always preserve the active bit, note deactivation */
76− deactivated = irq->active && !(val & ICH_LR_ACTIVE_BIT);
77− irq->active = !!(val & ICH_LR_ACTIVE_BIT);
195+void vgic_v3_deactivate(struct kvm_vcpu *vcpu, u64 val)
196+{
197+ struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
198+ struct vgic_v3_cpu_if *cpuif = &vgic_cpu->vgic_v3;
199+ u32 model = vcpu->kvm->arch.vgic.vgic_model;
200+ struct kvm_vcpu *target_vcpu = NULL;
201+ bool mmio = false, is_v2_sgi;
202+ struct vgic_irq *irq;
203+ unsigned long flags;
204+ u64 lr = 0;
205+ u8 cpuid;
78206
79− if (irq->active && is_v2_sgi)
80− irq->active_source = cpuid;
207+ /* Snapshot CPUID, and remove it from the INTID */
208+ cpuid = FIELD_GET(GENMASK_ULL(12, 10), val);
209+ val &= ~GENMASK_ULL(12, 10);
81210
82− /* Edge is the only case where we preserve the pending bit */
83− if (irq->config == VGIC_CONFIG_EDGE &&
84− (val & ICH_LR_PENDING_BIT)) {
85− irq->pending_latch = true;
211+ is_v2_sgi = (model == KVM_DEV_TYPE_ARM_VGIC_V2 &&
212+ val < VGIC_NR_SGIS);
86213
87− if (is_v2_sgi)
88− irq->source |= (1 << cpuid);
89− }
214+ /*
215+ * We only deal with DIR when EOIMode==1, and only for SGI,
216+ * PPI or SPI.
217+ */
218+ if (!(cpuif->vgic_vmcr & ICH_VMCR_EOIM_MASK) ||
219+ val >= vcpu->kvm->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS)
220+ return;
221+
222+ /* Make sure we're in the same context as LR handling */
223+ local_irq_save(flags);
224+
225+ irq = vgic_get_vcpu_irq(vcpu, val);
226+ if (WARN_ON_ONCE(!irq))
227+ goto out;
228+
229+ /*
230+ * EOIMode=1: we must rely on traps to handle deactivate of
231+ * overflowing interrupts, as there is no ordering guarantee and
232+ * EOIcount isn't being incremented. Priority drop will have taken
233+ * place, as ICV_EOIxR_EL1 only affects the APRs and not the LRs.
234+ *
235+ * Three possibities:
236+ *
237+ * - The irq is not queued on any CPU, and there is nothing to
238+ * do,
239+ *
240+ * - Or the irq is in an LR, meaning that its state is not
241+ * directly observable. Treat it bluntly by making it as if
242+ * this was a write to GICD_ICACTIVER, which will force an
243+ * exit on all vcpus. If it hurts, don't do that.
244+ *
245+ * - Or the irq is active, but not in an LR, and we can
246+ * directly deactivate it by building a pseudo-LR, fold it,
247+ * and queue a request to prune the resulting ap_list,
248+ *
249+ * Special care must be taken to match the source CPUID when
250+ * deactivating a GICv2 SGI.
251+ */
252+ scoped_guard(raw_spinlock, &irq->irq_lock) {
253+ target_vcpu = irq->vcpu;
254+
255+ /* Not on any ap_list? */
256+ if (!target_vcpu)
257+ goto put;
90258
91259 /*
92− * Clear soft pending state when level irqs have been acked.
260+ * Urgh. We're deactivating something that we cannot
261+ * observe yet... Big hammer time.
93262 */
94− if (irq->config == VGIC_CONFIG_LEVEL && !(val & ICH_LR_STATE))
95− irq->pending_latch = false;
263+ if (irq->on_lr) {
264+ mmio = true;
265+ goto put;
266+ }
96267
97− /* Handle resampling for mapped interrupts if required */
98− vgic_irq_handle_resampling(irq, deactivated, val & ICH_LR_PENDING_BIT);
268+ /* GICv2 SGI: check that the cpuid matches */
269+ if (is_v2_sgi && irq->active_source != cpuid) {
270+ target_vcpu = NULL;
271+ goto put;
272+ }
99273
100− raw_spin_unlock(&irq->irq_lock);
101− vgic_put_irq(vcpu->kvm, irq);
274+ /* (with a Dalek voice) DEACTIVATE!!!! */
275+ lr = vgic_v3_compute_lr(vcpu, irq) & ~ICH_LR_ACTIVE_BIT;
102276 }
103277
104− cpuif->used_lrs = 0;
278+ if (lr & ICH_LR_HW)
279+ vgic_v3_deactivate_phys(FIELD_GET(ICH_LR_PHYS_ID_MASK, lr));
280+
281+ vgic_v3_fold_lr(vcpu, lr);
282+
283+put:
284+ vgic_put_irq(vcpu->kvm, irq);
285+
286+out:
287+ local_irq_restore(flags);
288+
289+ if (mmio)
290+ vgic_mmio_write_cactive(vcpu, (val / 32) * 4, 4, BIT(val % 32));
291+
292+ /* Force the ap_list to be pruned */
293+ if (target_vcpu)
294+ kvm_make_request(KVM_REQ_VGIC_PROCESS_UPDATE, target_vcpu);
105295 }
106296
107297 /* Requires the irq to be locked already */
108−void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr)
298+static u64 vgic_v3_compute_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq)
109299 {
110300 u32 model = vcpu->kvm->arch.vgic.vgic_model;
111301 u64 val = irq->intid;
112302 bool allow_pending = true, is_v2_sgi;
113303
304+ WARN_ON(irq->on_lr);
305+
114306 is_v2_sgi = (vgic_irq_is_sgi(irq->intid) &&
115307 model == KVM_DEV_TYPE_ARM_VGIC_V2);
116308
@@ -150,6 +342,35 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr) · 33 unmodified lines
150342 if (allow_pending && irq_is_pending(irq)) {
151343 val |= ICH_LR_PENDING_BIT;
152344
345+ if (is_v2_sgi) {
346+ u32 src = ffs(irq->source);
347+
348+ if (WARN_RATELIMIT(!src, "No SGI source for INTID %d\n",
349+ irq->intid))
350+ return 0;
351+
352+ val |= (src - 1) << GICH_LR_PHYSID_CPUID_SHIFT;
353+ if (irq->source & ~BIT(src - 1))
354+ val |= ICH_LR_EOI;
355+ }
356+ }
357+
358+ if (irq->group)
359+ val |= ICH_LR_GROUP;
360+
361+ val |= (u64)irq->priority << ICH_LR_PRIORITY_SHIFT;
362+
363+ return val;
364+}
365+
366+void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr)
367+{
368+ u32 model = vcpu->kvm->arch.vgic.vgic_model;
369+ u64 val = vgic_v3_compute_lr(vcpu, irq);
370+
371+ vcpu->arch.vgic_cpu.vgic_v3.vgic_lr[lr] = val;
372+
373+ if (val & ICH_LR_PENDING_BIT) {
153374 if (irq->config == VGIC_CONFIG_EDGE)
154375 irq->pending_latch = false;
155376
@@ -157,16 +378,9 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr) · 1 unmodified line
157378 model == KVM_DEV_TYPE_ARM_VGIC_V2) {
158379 u32 src = ffs(irq->source);
159380
160− if (WARN_RATELIMIT(!src, "No SGI source for INTID %d\n",
161− irq->intid))
162− return;
163−
164− val |= (src - 1) << GICH_LR_PHYSID_CPUID_SHIFT;
165− irq->source &= ~(1 << (src - 1));
166− if (irq->source) {
381+ irq->source &= ~BIT(src - 1);
382+ if (irq->source)
167383 irq->pending_latch = true;
168− val |= ICH_LR_EOI;
169− }
170384 }
171385 }
172386
@@ -179,12 +393,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr) · 6 unmodified lines
179393 if (vgic_irq_is_mapped_level(irq) && (val & ICH_LR_PENDING_BIT))
180394 irq->line_level = false;
181395
182− if (irq->group)
183− val |= ICH_LR_GROUP;
184−
185− val |= (u64)irq->priority << ICH_LR_PRIORITY_SHIFT;
186−
187− vcpu->arch.vgic_cpu.vgic_v3.vgic_lr[lr] = val;
396+ irq->on_lr = true;
188397 }
189398
190399 void vgic_v3_clear_lr(struct kvm_vcpu *vcpu, int lr)
@@ -258,7 +467,7 @@ void vgic_v3_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcrp) · 67 unmodified lines
258467 GIC_BASER_CACHEABILITY(GICR_PENDBASER, OUTER, SameAsInner) | \
259468 GIC_BASER_SHAREABILITY(GICR_PENDBASER, InnerShareable))
260469
261−void vgic_v3_enable(struct kvm_vcpu *vcpu)
470+void vgic_v3_reset(struct kvm_vcpu *vcpu)
262471 {
263472 struct vgic_v3_cpu_if *vgic_v3 = &vcpu->arch.vgic_cpu.vgic_v3;
264473
@@ -288,9 +497,6 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu) · 23 unmodified lines
288497 kvm_vgic_global_state.ich_vtr_el2);
289498 vcpu->arch.vgic_cpu.num_pri_bits = FIELD_GET(ICH_VTR_EL2_PRIbits,
290499 kvm_vgic_global_state.ich_vtr_el2) + 1;
291−
292− /* Get the show on the road... */
293− vgic_v3->vgic_hcr = ICH_HCR_EL2_En;
294500 }
295501
296502 void vcpu_set_ich_hcr(struct kvm_vcpu *vcpu)
@@ -302,20 +508,9 @@ void vcpu_set_ich_hcr(struct kvm_vcpu *vcpu) · 5 unmodified lines
302508
303509 /* Hide GICv3 sysreg if necessary */
304510 if (vcpu->kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V2 ||
305− !irqchip_in_kernel(vcpu->kvm)) {
511+ !irqchip_in_kernel(vcpu->kvm))
306512 vgic_v3->vgic_hcr |= (ICH_HCR_EL2_TALL0 | ICH_HCR_EL2_TALL1 |
307513 ICH_HCR_EL2_TC);
308− return;
309− }
310−
311− if (group0_trap)
312− vgic_v3->vgic_hcr |= ICH_HCR_EL2_TALL0;
313− if (group1_trap)
314− vgic_v3->vgic_hcr |= ICH_HCR_EL2_TALL1;
315− if (common_trap)
316− vgic_v3->vgic_hcr |= ICH_HCR_EL2_TC;
317− if (dir_trap)
318− vgic_v3->vgic_hcr |= ICH_HCR_EL2_TDIR;
319514 }
320515
321516 int vgic_v3_lpi_sync_pending_status(struct kvm *kvm, struct vgic_irq *irq)
@@ -636,8 +831,53 @@ static const struct midr_range broken_seis[] = { · 314 unmodified lines
636831
637832 static bool vgic_v3_broken_seis(void)
638833 {
639− return ((kvm_vgic_global_state.ich_vtr_el2 & ICH_VTR_EL2_SEIS) &&
640− is_midr_in_range_list(broken_seis));
834+ return (is_kernel_in_hyp_mode() &&
835+ is_midr_in_range_list(broken_seis) &&
836+ (read_sysreg_s(SYS_ICH_VTR_EL2) & ICH_VTR_EL2_SEIS));
837+}
838+
839+void noinstr kvm_compute_ich_hcr_trap_bits(struct alt_instr *alt,
840+ __le32 *origptr, __le32 *updptr,
841+ int nr_inst)
842+{
843+ u32 insn, oinsn, rd;
844+ u64 hcr = 0;
845+
846+ if (cpus_have_cap(ARM64_WORKAROUND_CAVIUM_30115)) {
847+ group0_trap = true;
848+ group1_trap = true;
849+ }
850+
851+ if (vgic_v3_broken_seis()) {
852+ /* We know that these machines have ICH_HCR_EL2.TDIR */
853+ group0_trap = true;
854+ group1_trap = true;
855+ dir_trap = true;
856+ }
857+
858+ if (!cpus_have_cap(ARM64_HAS_ICH_HCR_EL2_TDIR))
859+ common_trap = true;
860+
861+ if (group0_trap)
862+ hcr |= ICH_HCR_EL2_TALL0;
863+ if (group1_trap)
864+ hcr |= ICH_HCR_EL2_TALL1;
865+ if (common_trap)
866+ hcr |= ICH_HCR_EL2_TC;
867+ if (dir_trap)
868+ hcr |= ICH_HCR_EL2_TDIR;
869+
870+ /* Compute target register */
871+ oinsn = le32_to_cpu(*origptr);
872+ rd = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RD, oinsn);
873+
874+ /* movz rd, #(val & 0xffff) */
875+ insn = aarch64_insn_gen_movewide(rd,
876+ (u16)hcr,
877+ 0,
878+ AARCH64_INSN_VARIANT_64BIT,
879+ AARCH64_INSN_MOVEWIDE_ZERO);
880+ *updptr = cpu_to_le32(insn);
641881 }
642882
643883 /**
@@ -651,6 +891,7 @@ int vgic_v3_probe(const struct gic_kvm_info *info) · 7 unmodified lines
651891 {
652892 u64 ich_vtr_el2 = kvm_call_hyp_ret(__vgic_v3_get_gic_config);
653893 bool has_v2;
894+ u64 traps;
654895 int ret;
655896
656897 has_v2 = ich_vtr_el2 >> 63;
@@ -709,29 +950,18 @@ int vgic_v3_probe(const struct gic_kvm_info *info) · 52 unmodified lines
709950 if (has_v2)
710951 static_branch_enable(&vgic_v3_has_v2_compat);
711952
712− if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_30115)) {
713− group0_trap = true;
714− group1_trap = true;
715− }
716−
717953 if (vgic_v3_broken_seis()) {
718954 kvm_info("GICv3 with broken locally generated SEI\n");
719−
720955 kvm_vgic_global_state.ich_vtr_el2 &= ~ICH_VTR_EL2_SEIS;
721− group0_trap = true;
722− group1_trap = true;
723− if (ich_vtr_el2 & ICH_VTR_EL2_TDS)
724− dir_trap = true;
725− else
726− common_trap = true;
727956 }
728957
729− if (group0_trap || group1_trap || common_trap | dir_trap) {
958+ traps = vgic_ich_hcr_trap_bits();
959+ if (traps) {
730960 kvm_info("GICv3 sysreg trapping enabled ([%s%s%s%s], reduced performance)\n",
731− group0_trap ? "G0" : "",
732− group1_trap ? "G1" : "",
733− common_trap ? "C" : "",
734− dir_trap ? "D" : "");
961+ (traps & ICH_HCR_EL2_TALL0) ? "G0" : "",
962+ (traps & ICH_HCR_EL2_TALL1) ? "G1" : "",
963+ (traps & ICH_HCR_EL2_TC) ? "C" : "",
964+ (traps & ICH_HCR_EL2_TDIR) ? "D" : "");
735965 static_branch_enable(&vgic_v3_cpuif_trap);
736966 }
737967
@@ -771,7 +1001,7 @@ void vgic_v3_put(struct kvm_vcpu *vcpu) · 33 unmodified lines
7711001 }
7721002
7731003 if (likely(!is_protected_kvm_enabled()))
774− kvm_call_hyp(__vgic_v3_save_vmcr_aprs, cpu_if);
1004+ kvm_call_hyp(__vgic_v3_save_aprs, cpu_if);
7751005 WARN_ON(vgic_v4_put(vcpu));
7761006
7771007 if (has_vhe())
arch/arm64/kvm/vgic/vgic-v4.c+4−1
@@ -163,6 +163,7 @@ static void vgic_v4_disable_vsgis(struct kvm_vcpu *vcpu) · 162 unmodified lines
163163 struct vgic_irq *irq = vgic_get_vcpu_irq(vcpu, i);
164164 struct irq_desc *desc;
165165 unsigned long flags;
166+ bool pending;
166167 int ret;
167168
168169 raw_spin_lock_irqsave(&irq->irq_lock, flags);
@@ -173,9 +174,11 @@ static void vgic_v4_disable_vsgis(struct kvm_vcpu *vcpu) · 4 unmodified lines
173174 irq->hw = false;
174175 ret = irq_get_irqchip_state(irq->host_irq,
175176 IRQCHIP_STATE_PENDING,
176− &irq->pending_latch);
177+ &pending);
177178 WARN_ON(ret);
178179
180+ irq->pending_latch = pending;
181+
179182 desc = irq_to_desc(irq->host_irq);
180183 irq_domain_deactivate_irq(irq_desc_get_irq_data(desc));
181184 unlock:
arch/arm64/kvm/vgic/vgic.c+186−112
@@ -244,7 +244,7 @@ void vgic_irq_set_phys_active(struct vgic_irq *irq, bool active) · 243 unmodified lines
244244 *
245245 * Requires the IRQ lock to be held.
246246 */
247−static struct kvm_vcpu *vgic_target_oracle(struct vgic_irq *irq)
247+struct kvm_vcpu *vgic_target_oracle(struct vgic_irq *irq)
248248 {
249249 lockdep_assert_held(&irq->irq_lock);
250250
@@ -272,17 +272,20 @@ static struct kvm_vcpu *vgic_target_oracle(struct vgic_irq *irq) · 21 unmodified lines
272272 return NULL;
273273 }
274274
275+struct vgic_sort_info {
276+ struct kvm_vcpu *vcpu;
277+ struct vgic_vmcr vmcr;
278+};
279+
275280 /*
276281 * The order of items in the ap_lists defines how we'll pack things in LRs as
277282 * well, the first items in the list being the first things populated in the
278283 * LRs.
279284 *
280− * A hard rule is that active interrupts can never be pushed out of the LRs
281− * (and therefore take priority) since we cannot reliably trap on deactivation
282− * of IRQs and therefore they have to be present in the LRs.
283− *
285+ * Pending, non-active interrupts must be placed at the head of the list.
284286 * Otherwise things should be sorted by the priority field and the GIC
285287 * hardware support will take care of preemption of priority groups etc.
288+ * Interrupts that are not deliverable should be at the end of the list.
286289 *
287290 * Return negative if "a" sorts before "b", 0 to preserve order, and positive
288291 * to sort "b" before "a".
@@ -292,6 +295,8 @@ static int vgic_irq_cmp(void *priv, const struct list_head *a, · 3 unmodified lines
292295 {
293296 struct vgic_irq *irqa = container_of(a, struct vgic_irq, ap_list);
294297 struct vgic_irq *irqb = container_of(b, struct vgic_irq, ap_list);
298+ struct vgic_sort_info *info = priv;
299+ struct kvm_vcpu *vcpu = info->vcpu;
295300 bool penda, pendb;
296301 int ret;
297302
@@ -305,21 +310,32 @@ static int vgic_irq_cmp(void *priv, const struct list_head *a, · 7 unmodified lines
305310 raw_spin_lock(&irqa->irq_lock);
306311 raw_spin_lock_nested(&irqb->irq_lock, SINGLE_DEPTH_NESTING);
307312
308− if (irqa->active || irqb->active) {
309− ret = (int)irqb->active - (int)irqa->active;
313+ /* Undeliverable interrupts should be last */
314+ ret = (int)(vgic_target_oracle(irqb) == vcpu) - (int)(vgic_target_oracle(irqa) == vcpu);
315+ if (ret)
316+ goto out;
317+
318+ /* Same thing for interrupts targeting a disabled group */
319+ ret = (int)(irqb->group ? info->vmcr.grpen1 : info->vmcr.grpen0);
320+ ret -= (int)(irqa->group ? info->vmcr.grpen1 : info->vmcr.grpen0);
321+ if (ret)
310322 goto out;
311− }
312323
313− penda = irqa->enabled && irq_is_pending(irqa);
314− pendb = irqb->enabled && irq_is_pending(irqb);
324+ penda = irqa->enabled && irq_is_pending(irqa) && !irqa->active;
325+ pendb = irqb->enabled && irq_is_pending(irqb) && !irqb->active;
315326
316− if (!penda || !pendb) {
317− ret = (int)pendb - (int)penda;
327+ ret = (int)pendb - (int)penda;
328+ if (ret)
318329 goto out;
319− }
320330
321− /* Both pending and enabled, sort by priority */
322− ret = irqa->priority - irqb->priority;
331+ /* Both pending and enabled, sort by priority (lower number first) */
332+ ret = (int)irqa->priority - (int)irqb->priority;
333+ if (ret)
334+ goto out;
335+
336+ /* Finally, HW bit active interrupts have priority over non-HW ones */
337+ ret = (int)irqb->hw - (int)irqa->hw;
338+
323339 out:
324340 raw_spin_unlock(&irqb->irq_lock);
325341 raw_spin_unlock(&irqa->irq_lock);
@@ -330,10 +346,12 @@ out: · 4 unmodified lines
330346 static void vgic_sort_ap_list(struct kvm_vcpu *vcpu)
331347 {
332348 struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
349+ struct vgic_sort_info info = { .vcpu = vcpu, };
333350
334351 lockdep_assert_held(&vgic_cpu->ap_list_lock);
335352
336− list_sort(NULL, &vgic_cpu->ap_list_head, vgic_irq_cmp);
353+ vgic_get_vmcr(vcpu, &info.vmcr);
354+ list_sort(&info, &vgic_cpu->ap_list_head, vgic_irq_cmp);
337355 }
338356
339357 /*
@@ -356,6 +374,20 @@ static bool vgic_validate_injection(struct vgic_irq *irq, bool level, void *owne · 16 unmodified lines
356374 return false;
357375 }
358376
377+static bool vgic_model_needs_bcst_kick(struct kvm *kvm)
378+{
379+ /*
380+ * A GICv3 (or GICv3-like) system exposing a GICv3 to the guest
381+ * needs a broadcast kick to set TDIR globally.
382+ *
383+ * For systems that do not have TDIR (ARM's own v8.0 CPUs), the
384+ * shadow TDIR bit is always set, and so is the register's TC bit,
385+ * so no need to kick the CPUs.
386+ */
387+ return (cpus_have_final_cap(ARM64_HAS_ICH_HCR_EL2_TDIR) &&
388+ kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3);
389+}
390+
359391 /*
360392 * Check whether an IRQ needs to (and can) be queued to a VCPU's ap list.
361393 * Do the queuing if necessary, taking the right locks in the right order.
@@ -368,6 +400,7 @@ bool vgic_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq, · 6 unmodified lines
368400 unsigned long flags) __releases(&irq->irq_lock)
369401 {
370402 struct kvm_vcpu *vcpu;
403+ bool bcast;
371404
372405 lockdep_assert_held(&irq->irq_lock);
373406
@@ -442,11 +475,20 @@ retry: · 68 unmodified lines
442475 list_add_tail(&irq->ap_list, &vcpu->arch.vgic_cpu.ap_list_head);
443476 irq->vcpu = vcpu;
444477
478+ /* A new SPI may result in deactivation trapping on all vcpus */
479+ bcast = (vgic_model_needs_bcst_kick(vcpu->kvm) &&
480+ vgic_valid_spi(vcpu->kvm, irq->intid) &&
481+ atomic_fetch_inc(&vcpu->kvm->arch.vgic.active_spis) == 0);
482+
445483 raw_spin_unlock(&irq->irq_lock);
446484 raw_spin_unlock_irqrestore(&vcpu->arch.vgic_cpu.ap_list_lock, flags);
447485
448− kvm_make_request(KVM_REQ_IRQ_PENDING, vcpu);
449− kvm_vcpu_kick(vcpu);
486+ if (!bcast) {
487+ kvm_make_request(KVM_REQ_IRQ_PENDING, vcpu);
488+ kvm_vcpu_kick(vcpu);
489+ } else {
490+ kvm_make_all_cpus_request(vcpu->kvm, KVM_REQ_IRQ_PENDING);
491+ }
450492
451493 return true;
452494 }
@@ -798,98 +840,148 @@ static inline void vgic_clear_lr(struct kvm_vcpu *vcpu, int lr) · 345 unmodified lines
798840 vgic_v3_clear_lr(vcpu, lr);
799841 }
800842
801−static inline void vgic_set_underflow(struct kvm_vcpu *vcpu)
802−{
803− if (kvm_vgic_global_state.type == VGIC_V2)
804− vgic_v2_set_underflow(vcpu);
805− else
806− vgic_v3_set_underflow(vcpu);
807−}
808−
809−/* Requires the ap_list_lock to be held. */
810−static int compute_ap_list_depth(struct kvm_vcpu *vcpu,
811− bool *multi_sgi)
843+static void summarize_ap_list(struct kvm_vcpu *vcpu,
844+ struct ap_list_summary *als)
812845 {
813846 struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
814847 struct vgic_irq *irq;
815− int count = 0;
816−
817− *multi_sgi = false;
818848
819849 lockdep_assert_held(&vgic_cpu->ap_list_lock);
820850
851+ *als = (typeof(*als)){};
852+
821853 list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list) {
822− int w;
854+ guard(raw_spinlock)(&irq->irq_lock);
823855
824− raw_spin_lock(&irq->irq_lock);
825− /* GICv2 SGIs can count for more than one... */
826− w = vgic_irq_get_lr_count(irq);
827− raw_spin_unlock(&irq->irq_lock);
856+ if (unlikely(vgic_target_oracle(irq) != vcpu))
857+ continue;
858+
859+ if (!irq->active)
860+ als->nr_pend++;
861+ else
862+ als->nr_act++;
828863
829− count += w;
830− *multi_sgi |= (w > 1);
864+ if (irq->intid < VGIC_NR_SGIS)
865+ als->nr_sgi++;
831866 }
832− return count;
833867 }
834868
835−/* Requires the VCPU's ap_list_lock to be held. */
869+/*
870+ * Dealing with LR overflow is close to black magic -- dress accordingly.
871+ *
872+ * We have to present an almost infinite number of interrupts through a very
873+ * limited number of registers. Therefore crucial decisions must be made to
874+ * ensure we feed the most relevant interrupts into the LRs, and yet have
875+ * some facilities to let the guest interact with those that are not there.
876+ *
877+ * All considerations below are in the context of interrupts targeting a
878+ * single vcpu with non-idle state (either pending, active, or both),
879+ * colloquially called the ap_list:
880+ *
881+ * - Pending interrupts must have priority over active interrupts. This also
882+ * excludes pending+active interrupts. This ensures that a guest can
883+ * perform priority drops on any number of interrupts, and yet be
884+ * presented the next pending one.
885+ *
886+ * - Deactivation of interrupts outside of the LRs must be tracked by using
887+ * either the EOIcount-driven maintenance interrupt, and sometimes by
888+ * trapping the DIR register.
889+ *
890+ * - For EOImode=0, a non-zero EOIcount means walking the ap_list past the
891+ * point that made it into the LRs, and deactivate interrupts that would
892+ * have made it onto the LRs if we had the space.
893+ *
894+ * - The MI-generation bits must be used to try and force an exit when the
895+ * guest has done enough changes to the LRs that we want to reevaluate the
896+ * situation:
897+ *
898+ * - if the total number of pending interrupts exceeds the number of
899+ * LR, NPIE must be set in order to exit once no pending interrupts
900+ * are present in the LRs, allowing us to populate the next batch.
901+ *
902+ * - if there are active interrupts outside of the LRs, then LRENPIE
903+ * must be set so that we exit on deactivation of one of these, and
904+ * work out which one is to be deactivated. Note that this is not
905+ * enough to deal with EOImode=1, see below.
906+ *
907+ * - if the overall number of interrupts exceeds the number of LRs,
908+ * then UIE must be set to allow refilling of the LRs once the
909+ * majority of them has been processed.
910+ *
911+ * - as usual, MI triggers are only an optimisation, since we cannot
912+ * rely on the MI being delivered in timely manner...
913+ *
914+ * - EOImode=1 creates some additional problems:
915+ *
916+ * - deactivation can happen in any order, and we cannot rely on
917+ * EOImode=0's coupling of priority-drop and deactivation which
918+ * imposes strict reverse Ack order. This means that DIR must
919+ * trap if we have active interrupts outside of the LRs.
920+ *
921+ * - deactivation of SPIs can occur on any CPU, while the SPI is only
922+ * present in the ap_list of the CPU that actually ack-ed it. In that
923+ * case, EOIcount doesn't provide enough information, and we must
924+ * resort to trapping DIR even if we don't overflow the LRs. Bonus
925+ * point for not trapping DIR when no SPIs are pending or active in
926+ * the whole VM.
927+ *
928+ * - LPIs do not suffer the same problem as SPIs on deactivation, as we
929+ * have to essentially discard the active state, see below.
930+ *
931+ * - Virtual LPIs have an active state (surprise!), which gets removed on
932+ * priority drop (EOI). However, EOIcount doesn't get bumped when the LPI
933+ * is not present in the LR (surprise again!). Special care must therefore
934+ * be taken to remove the active state from any activated LPI when exiting
935+ * from the guest. This is in a way no different from what happens on the
936+ * physical side. We still rely on the running priority to have been
937+ * removed from the APRs, irrespective of the LPI being present in the LRs
938+ * or not.
939+ *
940+ * - Virtual SGIs directly injected via GICv4.1 must not affect EOIcount, as
941+ * they are not managed in SW and don't have a true active state. So only
942+ * set vSGIEOICount when no SGIs are in the ap_list.
943+ *
944+ * - GICv2 SGIs with multiple sources are injected one source at a time, as
945+ * if they were made pending sequentially. This may mean that we don't
946+ * always present the HPPI if other interrupts with lower priority are
947+ * pending in the LRs. Big deal.
948+ */
836949 static void vgic_flush_lr_state(struct kvm_vcpu *vcpu)
837950 {
838951 struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
952+ struct ap_list_summary als;
839953 struct vgic_irq *irq;
840− int count;
841− bool multi_sgi;
842− u8 prio = 0xff;
843− int i = 0;
954+ int count = 0;
844955
845956 lockdep_assert_held(&vgic_cpu->ap_list_lock);
846957
847− count = compute_ap_list_depth(vcpu, &multi_sgi);
848− if (count > kvm_vgic_global_state.nr_lr || multi_sgi)
849− vgic_sort_ap_list(vcpu);
958+ summarize_ap_list(vcpu, &als);
850959
851− count = 0;
960+ if (irqs_outside_lrs(&als))
961+ vgic_sort_ap_list(vcpu);
852962
853963 list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list) {
854− raw_spin_lock(&irq->irq_lock);
855−
856− /*
857− * If we have multi-SGIs in the pipeline, we need to
858− * guarantee that they are all seen before any IRQ of
859− * lower priority. In that case, we need to filter out
860− * these interrupts by exiting early. This is easy as
861− * the AP list has been sorted already.
862− */
863− if (multi_sgi && irq->priority > prio) {
864− raw_spin_unlock(&irq->irq_lock);
865− break;
866− }
867−
868− if (likely(vgic_target_oracle(irq) == vcpu)) {
869− vgic_populate_lr(vcpu, irq, count++);
870−
871− if (irq->source)
872− prio = irq->priority;
964+ scoped_guard(raw_spinlock, &irq->irq_lock) {
965+ if (likely(vgic_target_oracle(irq) == vcpu)) {
966+ vgic_populate_lr(vcpu, irq, count++);
967+ }
873968 }
874969
875− raw_spin_unlock(&irq->irq_lock);
876−
877− if (count == kvm_vgic_global_state.nr_lr) {
878− if (!list_is_last(&irq->ap_list,
879− &vgic_cpu->ap_list_head))
880− vgic_set_underflow(vcpu);
970+ if (count == kvm_vgic_global_state.nr_lr)
881971 break;
882− }
883972 }
884973
885974 /* Nuke remaining LRs */
886− for (i = count ; i < kvm_vgic_global_state.nr_lr; i++)
975+ for (int i = count ; i < kvm_vgic_global_state.nr_lr; i++)
887976 vgic_clear_lr(vcpu, i);
888977
889− if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
978+ if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) {
890979 vcpu->arch.vgic_cpu.vgic_v2.used_lrs = count;
891− else
980+ vgic_v2_configure_hcr(vcpu, &als);
981+ } else {
892982 vcpu->arch.vgic_cpu.vgic_v3.used_lrs = count;
983+ vgic_v3_configure_hcr(vcpu, &als);
984+ }
893985 }
894986
895987 static inline bool can_access_vgic_from_kernel(void)
@@ -913,8 +1005,6 @@ static inline void vgic_save_state(struct kvm_vcpu *vcpu) · 17 unmodified lines
9131005 /* Sync back the hardware VGIC state into our emulation after a guest's run. */
9141006 void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu)
9151007 {
916− int used_lrs;
917−
9181008 /* If nesting, emulate the HW effect from L0 to L1 */
9191009 if (vgic_state_is_nested(vcpu)) {
9201010 vgic_v3_sync_nested(vcpu);
@@ -924,21 +1014,22 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu) · 3 unmodified lines
9241014 if (vcpu_has_nv(vcpu))
9251015 vgic_v3_nested_update_mi(vcpu);
9261016
927− /* An empty ap_list_head implies used_lrs == 0 */
928− if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head))
929− return;
930−
9311017 if (can_access_vgic_from_kernel())
9321018 vgic_save_state(vcpu);
9331019
934− if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif))
935− used_lrs = vcpu->arch.vgic_cpu.vgic_v2.used_lrs;
936− else
937− used_lrs = vcpu->arch.vgic_cpu.vgic_v3.used_lrs;
1020+ vgic_fold_lr_state(vcpu);
1021+ vgic_prune_ap_list(vcpu);
1022+}
1023+
1024+/* Sync interrupts that were deactivated through a DIR trap */
1025+void kvm_vgic_process_async_update(struct kvm_vcpu *vcpu)
1026+{
1027+ unsigned long flags;
9381028
939− if (used_lrs)
940− vgic_fold_lr_state(vcpu);
1029+ /* Make sure we're in the same context as LR handling */
1030+ local_irq_save(flags);
9411031 vgic_prune_ap_list(vcpu);
1032+ local_irq_restore(flags);
9421033 }
9431034
9441035 static inline void vgic_restore_state(struct kvm_vcpu *vcpu)
@@ -965,8 +1056,9 @@ void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu) · 20 unmodified lines
9651056 * abort the entry procedure and inject the exception at the
9661057 * beginning of the run loop.
9671058 *
968− * - Otherwise, do exactly *NOTHING*. The guest state is
969− * already loaded, and we can carry on with running it.
1059+ * - Otherwise, do exactly *NOTHING* apart from enabling the virtual
1060+ * CPU interface. The guest state is already loaded, and we can
1061+ * carry on with running it.
9701062 *
9711063 * If we have NV, but are not in a nested state, compute the
9721064 * maintenance interrupt state, as it may fire.
@@ -975,35 +1067,17 @@ void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu) · 2 unmodified lines
9751067 if (kvm_vgic_vcpu_pending_irq(vcpu))
9761068 kvm_make_request(KVM_REQ_GUEST_HYP_IRQ_PENDING, vcpu);
9771069
1070+ vgic_v3_flush_nested(vcpu);
9781071 return;
9791072 }
9801073
9811074 if (vcpu_has_nv(vcpu))
9821075 vgic_v3_nested_update_mi(vcpu);
9831076
984− /*
985− * If there are no virtual interrupts active or pending for this
986− * VCPU, then there is no work to do and we can bail out without
987− * taking any lock. There is a potential race with someone injecting
988− * interrupts to the VCPU, but it is a benign race as the VCPU will
989− * either observe the new interrupt before or after doing this check,
990− * and introducing additional synchronization mechanism doesn't change
991− * this.
992− *
993− * Note that we still need to go through the whole thing if anything
994− * can be directly injected (GICv4).
995− */
996− if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head) &&
997− !vgic_supports_direct_irqs(vcpu->kvm))
998− return;
999−
10001077 DEBUG_SPINLOCK_BUG_ON(!irqs_disabled());
10011078
1002− if (!list_empty(&vcpu->arch.vgic_cpu.ap_list_head)) {
1003− raw_spin_lock(&vcpu->arch.vgic_cpu.ap_list_lock);
1079+ scoped_guard(raw_spinlock, &vcpu->arch.vgic_cpu.ap_list_lock)
10041080 vgic_flush_lr_state(vcpu);
1005− raw_spin_unlock(&vcpu->arch.vgic_cpu.ap_list_lock);
1006− }
10071081
10081082 if (can_access_vgic_from_kernel())
10091083 vgic_restore_state(vcpu);
arch/arm64/kvm/vgic/vgic.h+39−4
@@ -164,6 +164,22 @@ static inline int vgic_write_guest_lock(struct kvm *kvm, gpa_t gpa, · 163 unmodified lines
164164 return ret;
165165 }
166166
167+void kvm_compute_ich_hcr_trap_bits(struct alt_instr *alt,
168+ __le32 *origptr, __le32 *updptr, int nr_inst);
169+
170+static inline u64 vgic_ich_hcr_trap_bits(void)
171+{
172+ u64 hcr;
173+
174+ /* All the traps are in the bottom 16bits */
175+ asm volatile(ALTERNATIVE_CB("movz %0, #0\n",
176+ ARM64_ALWAYS_SYSTEM,
177+ kvm_compute_ich_hcr_trap_bits)
178+ : "=r" (hcr));
179+
180+ return hcr;
181+}
182+
167183 /*
168184 * This struct provides an intermediate representation of the fields contained
169185 * in the GICH_VMCR and ICH_VMCR registers, such that code exporting the GIC
@@ -220,6 +236,21 @@ struct its_ite { · 50 unmodified lines
220236 u32 event_id;
221237 };
222238
239+struct ap_list_summary {
240+ unsigned int nr_pend; /* purely pending, not active */
241+ unsigned int nr_act; /* active, or active+pending */
242+ unsigned int nr_sgi; /* any SGI */
243+};
244+
245+#define irqs_outside_lrs(s) \
246+ (((s)->nr_pend + (s)->nr_act) > kvm_vgic_global_state.nr_lr)
247+
248+#define irqs_pending_outside_lrs(s) \
249+ ((s)->nr_pend > kvm_vgic_global_state.nr_lr)
250+
251+#define irqs_active_outside_lrs(s) \
252+ ((s)->nr_act && irqs_outside_lrs(s))
253+
223254 int vgic_v3_parse_attr(struct kvm_device *dev, struct kvm_device_attr *attr,
224255 struct vgic_reg_attr *reg_attr);
225256 int vgic_v2_parse_attr(struct kvm_device *dev, struct kvm_device_attr *attr,
@@ -230,6 +261,7 @@ vgic_get_mmio_region(struct kvm_vcpu *vcpu, struct vgic_io_device *iodev, · 4 unmodified lines
230261 struct vgic_irq *vgic_get_irq(struct kvm *kvm, u32 intid);
231262 struct vgic_irq *vgic_get_vcpu_irq(struct kvm_vcpu *vcpu, u32 intid);
232263 void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq);
264+struct kvm_vcpu *vgic_target_oracle(struct vgic_irq *irq);
233265 bool vgic_get_phys_line_level(struct vgic_irq *irq);
234266 void vgic_irq_set_phys_pending(struct vgic_irq *irq, bool pending);
235267 void vgic_irq_set_phys_active(struct vgic_irq *irq, bool active);
@@ -245,8 +277,9 @@ int vgic_check_iorange(struct kvm *kvm, phys_addr_t ioaddr, · 9 unmodified lines
245277
246278 void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu);
247279 void vgic_v2_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr);
280+void vgic_v2_deactivate(struct kvm_vcpu *vcpu, u32 val);
248281 void vgic_v2_clear_lr(struct kvm_vcpu *vcpu, int lr);
249−void vgic_v2_set_underflow(struct kvm_vcpu *vcpu);
282+void vgic_v2_configure_hcr(struct kvm_vcpu *vcpu, struct ap_list_summary *als);
250283 int vgic_v2_has_attr_regs(struct kvm_device *dev, struct kvm_device_attr *attr);
251284 int vgic_v2_dist_uaccess(struct kvm_vcpu *vcpu, bool is_write,
252285 int offset, u32 *val);
@@ -254,7 +287,7 @@ int vgic_v2_cpuif_uaccess(struct kvm_vcpu *vcpu, bool is_write, · 1 unmodified line
254287 int offset, u32 *val);
255288 void vgic_v2_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
256289 void vgic_v2_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
257−void vgic_v2_enable(struct kvm_vcpu *vcpu);
290+void vgic_v2_reset(struct kvm_vcpu *vcpu);
258291 int vgic_v2_probe(const struct gic_kvm_info *info);
259292 int vgic_v2_map_resources(struct kvm *kvm);
260293 int vgic_register_dist_iodev(struct kvm *kvm, gpa_t dist_base_address,
@@ -286,10 +319,11 @@ static inline void vgic_get_irq_ref(struct vgic_irq *irq) · 25 unmodified lines
286319 void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu);
287320 void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr);
288321 void vgic_v3_clear_lr(struct kvm_vcpu *vcpu, int lr);
289−void vgic_v3_set_underflow(struct kvm_vcpu *vcpu);
322+void vgic_v3_deactivate(struct kvm_vcpu *vcpu, u64 val);
323+void vgic_v3_configure_hcr(struct kvm_vcpu *vcpu, struct ap_list_summary *als);
290324 void vgic_v3_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
291325 void vgic_v3_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
292−void vgic_v3_enable(struct kvm_vcpu *vcpu);
326+void vgic_v3_reset(struct kvm_vcpu *vcpu);
293327 int vgic_v3_probe(const struct gic_kvm_info *info);
294328 int vgic_v3_map_resources(struct kvm *kvm);
295329 int vgic_v3_lpi_sync_pending_status(struct kvm *kvm, struct vgic_irq *irq);
@@ -412,6 +446,7 @@ static inline bool kvm_has_gicv3(struct kvm *kvm) · 116 unmodified lines
412446 return kvm_has_feat(kvm, ID_AA64PFR0_EL1, GIC, IMP);
413447 }
414448
449+void vgic_v3_flush_nested(struct kvm_vcpu *vcpu);
415450 void vgic_v3_sync_nested(struct kvm_vcpu *vcpu);
416451 void vgic_v3_load_nested(struct kvm_vcpu *vcpu);
417452 void vgic_v3_put_nested(struct kvm_vcpu *vcpu);
arch/arm64/tools/cpucaps+2−0
@@ -40,6 +40,7 @@ HAS_GICV5_CPUIF · 39 unmodified lines
4040 HAS_GICV5_LEGACY
4141 HAS_GIC_PRIO_MASKING
4242 HAS_GIC_PRIO_RELAXED_SYNC
43+HAS_ICH_HCR_EL2_TDIR
4344 HAS_HCR_NV1
4445 HAS_HCX
4546 HAS_LDAPR
@@ -64,6 +65,7 @@ HAS_TLB_RANGE · 18 unmodified lines
6465 HAS_VA52
6566 HAS_VIRT_HOST_EXTN
6667 HAS_WFXT
68+HAS_XNX
6769 HAFT
6870 HW_DBM
6971 KVM_HVHE
arch/loongarch/include/asm/kvm_eiointc.h+8−47
@@ -10,10 +10,7 @@ · 9 unmodified lines
1010
1111 #define EIOINTC_IRQS 256
1212 #define EIOINTC_ROUTE_MAX_VCPUS 256
13−#define EIOINTC_IRQS_U8_NUMS (EIOINTC_IRQS / 8)
14−#define EIOINTC_IRQS_U16_NUMS (EIOINTC_IRQS_U8_NUMS / 2)
15−#define EIOINTC_IRQS_U32_NUMS (EIOINTC_IRQS_U8_NUMS / 4)
16−#define EIOINTC_IRQS_U64_NUMS (EIOINTC_IRQS_U8_NUMS / 8)
13+#define EIOINTC_IRQS_U64_NUMS (EIOINTC_IRQS / 64)
1714 /* map to ipnum per 32 irqs */
1815 #define EIOINTC_IRQS_NODETYPE_COUNT 16
1916
@@ -64,54 +61,18 @@ struct loongarch_eiointc { · 44 unmodified lines
6461 uint32_t status;
6562
6663 /* hardware state */
67− union nodetype {
68− u64 reg_u64[EIOINTC_IRQS_NODETYPE_COUNT / 4];
69− u32 reg_u32[EIOINTC_IRQS_NODETYPE_COUNT / 2];
70− u16 reg_u16[EIOINTC_IRQS_NODETYPE_COUNT];
71− u8 reg_u8[EIOINTC_IRQS_NODETYPE_COUNT * 2];
72− } nodetype;
64+ u64 nodetype[EIOINTC_IRQS_NODETYPE_COUNT / 4];
7365
7466 /* one bit shows the state of one irq */
75− union bounce {
76− u64 reg_u64[EIOINTC_IRQS_U64_NUMS];
77− u32 reg_u32[EIOINTC_IRQS_U32_NUMS];
78− u16 reg_u16[EIOINTC_IRQS_U16_NUMS];
79− u8 reg_u8[EIOINTC_IRQS_U8_NUMS];
80− } bounce;
81−
82− union isr {
83− u64 reg_u64[EIOINTC_IRQS_U64_NUMS];
84− u32 reg_u32[EIOINTC_IRQS_U32_NUMS];
85− u16 reg_u16[EIOINTC_IRQS_U16_NUMS];
86− u8 reg_u8[EIOINTC_IRQS_U8_NUMS];
87− } isr;
88− union coreisr {
89− u64 reg_u64[EIOINTC_ROUTE_MAX_VCPUS][EIOINTC_IRQS_U64_NUMS];
90− u32 reg_u32[EIOINTC_ROUTE_MAX_VCPUS][EIOINTC_IRQS_U32_NUMS];
91− u16 reg_u16[EIOINTC_ROUTE_MAX_VCPUS][EIOINTC_IRQS_U16_NUMS];
92− u8 reg_u8[EIOINTC_ROUTE_MAX_VCPUS][EIOINTC_IRQS_U8_NUMS];
93− } coreisr;
94− union enable {
95− u64 reg_u64[EIOINTC_IRQS_U64_NUMS];
96− u32 reg_u32[EIOINTC_IRQS_U32_NUMS];
97− u16 reg_u16[EIOINTC_IRQS_U16_NUMS];
98− u8 reg_u8[EIOINTC_IRQS_U8_NUMS];
99− } enable;
67+ u64 bounce[EIOINTC_IRQS_U64_NUMS];
68+ u64 isr[EIOINTC_IRQS_U64_NUMS];
69+ u64 coreisr[EIOINTC_ROUTE_MAX_VCPUS][EIOINTC_IRQS_U64_NUMS];
70+ u64 enable[EIOINTC_IRQS_U64_NUMS];
10071
10172 /* use one byte to config ipmap for 32 irqs at once */
102− union ipmap {
103− u64 reg_u64;
104− u32 reg_u32[EIOINTC_IRQS_U32_NUMS / 4];
105− u16 reg_u16[EIOINTC_IRQS_U16_NUMS / 4];
106− u8 reg_u8[EIOINTC_IRQS_U8_NUMS / 4];
107− } ipmap;
73+ u64 ipmap;
10874 /* use one byte to config coremap for one irq */
109− union coremap {
110− u64 reg_u64[EIOINTC_IRQS / 8];
111− u32 reg_u32[EIOINTC_IRQS / 4];
112− u16 reg_u16[EIOINTC_IRQS / 2];
113− u8 reg_u8[EIOINTC_IRQS];
114− } coremap;
75+ u64 coremap[EIOINTC_IRQS / 8];
11576
11677 DECLARE_BITMAP(sw_coreisr[EIOINTC_ROUTE_MAX_VCPUS][LOONGSON_IP_NUM], EIOINTC_IRQS);
11778 uint8_t sw_coremap[EIOINTC_IRQS];
arch/loongarch/include/asm/kvm_host.h+8−0
@@ -126,6 +126,8 @@ struct kvm_arch { · 125 unmodified lines
126126 struct kvm_phyid_map *phyid_map;
127127 /* Enabled PV features */
128128 unsigned long pv_features;
129+ /* Supported KVM features */
130+ unsigned long kvm_features;
129131
130132 s64 time_offset;
131133 struct kvm_context __percpu *vmcs;
@@ -293,6 +295,12 @@ static inline int kvm_get_pmu_num(struct kvm_vcpu_arch *arch) · 161 unmodified lines
293295 return (arch->cpucfg[6] & CPUCFG6_PMNUM) >> CPUCFG6_PMNUM_SHIFT;
294296 }
295297
298+/* Check whether KVM support this feature (VMM may disable it) */
299+static inline bool kvm_vm_support(struct kvm_arch *arch, int feature)
300+{
301+ return !!(arch->kvm_features & BIT_ULL(feature));
302+}
303+
296304 bool kvm_arch_pmi_in_guest(struct kvm_vcpu *vcpu);
297305
298306 /* Debug: dump vcpu state */
arch/loongarch/include/asm/kvm_vcpu.h+1−0
@@ -15,6 +15,7 @@ · 14 unmodified lines
1515 #define CPU_PMU (_ULCAST_(1) << 10)
1616 #define CPU_TIMER (_ULCAST_(1) << 11)
1717 #define CPU_IPI (_ULCAST_(1) << 12)
18+#define CPU_AVEC (_ULCAST_(1) << 14)
1819
1920 /* Controlled by 0x52 guest exception VIP aligned to estat bit 5~12 */
2021 #define CPU_IP0 (_ULCAST_(1))
arch/loongarch/include/asm/loongarch.h+2−0
@@ -511,6 +511,8 @@ · 510 unmodified lines
511511 #define CSR_GCFG_GPERF_SHIFT 24
512512 #define CSR_GCFG_GPERF_WIDTH 3
513513 #define CSR_GCFG_GPERF (_ULCAST_(0x7) << CSR_GCFG_GPERF_SHIFT)
514+#define CSR_GCFG_GPMP_SHIFT 23
515+#define CSR_GCFG_GPMP (_ULCAST_(0x1) << CSR_GCFG_GPMP_SHIFT)
514516 #define CSR_GCFG_GCI_SHIFT 20
515517 #define CSR_GCFG_GCI_WIDTH 2
516518 #define CSR_GCFG_GCI (_ULCAST_(0x3) << CSR_GCFG_GCI_SHIFT)
arch/loongarch/include/uapi/asm/kvm.h+1−0
@@ -104,6 +104,7 @@ struct kvm_fpu { · 103 unmodified lines
104104 #define KVM_LOONGARCH_VM_FEAT_PV_IPI 6
105105 #define KVM_LOONGARCH_VM_FEAT_PV_STEALTIME 7
106106 #define KVM_LOONGARCH_VM_FEAT_PTW 8
107+#define KVM_LOONGARCH_VM_FEAT_MSGINT 9
107108
108109 /* Device Control API on vcpu fd */
109110 #define KVM_LOONGARCH_VCPU_CPUCFG 0
arch/loongarch/kvm/Kconfig+0−1
@@ -25,7 +25,6 @@ config KVM · 24 unmodified lines
2525 select HAVE_KVM_IRQCHIP
2626 select HAVE_KVM_MSI
2727 select HAVE_KVM_READONLY_MEM
28− select HAVE_KVM_VCPU_ASYNC_IOCTL
2928 select KVM_COMMON
3029 select KVM_GENERIC_DIRTYLOG_READ_PROTECT
3130 select KVM_GENERIC_HARDWARE_ENABLING
arch/loongarch/kvm/intc/eiointc.c+40−40
@@ -13,19 +13,19 @@ static void eiointc_set_sw_coreisr(struct loongarch_eiointc *s) · 12 unmodified lines
1313 struct kvm_vcpu *vcpu;
1414
1515 for (irq = 0; irq < EIOINTC_IRQS; irq++) {
16− ipnum = s->ipmap.reg_u8[irq / 32];
16+ ipnum = (s->ipmap >> (irq / 32 * 8)) & 0xff;
1717 if (!(s->status & BIT(EIOINTC_ENABLE_INT_ENCODE))) {
1818 ipnum = count_trailing_zeros(ipnum);
1919 ipnum = (ipnum >= 0 && ipnum < 4) ? ipnum : 0;
2020 }
2121
22− cpuid = s->coremap.reg_u8[irq];
22+ cpuid = ((u8 *)s->coremap)[irq];
2323 vcpu = kvm_get_vcpu_by_cpuid(s->kvm, cpuid);
2424 if (!vcpu)
2525 continue;
2626
2727 cpu = vcpu->vcpu_id;
28− if (test_bit(irq, (unsigned long *)s->coreisr.reg_u32[cpu]))
28+ if (test_bit(irq, (unsigned long *)s->coreisr[cpu]))
2929 __set_bit(irq, s->sw_coreisr[cpu][ipnum]);
3030 else
3131 __clear_bit(irq, s->sw_coreisr[cpu][ipnum]);
@@ -38,7 +38,7 @@ static void eiointc_update_irq(struct loongarch_eiointc *s, int irq, int level) · 6 unmodified lines
3838 struct kvm_vcpu *vcpu;
3939 struct kvm_interrupt vcpu_irq;
4040
41− ipnum = s->ipmap.reg_u8[irq / 32];
41+ ipnum = (s->ipmap >> (irq / 32 * 8)) & 0xff;
4242 if (!(s->status & BIT(EIOINTC_ENABLE_INT_ENCODE))) {
4343 ipnum = count_trailing_zeros(ipnum);
4444 ipnum = (ipnum >= 0 && ipnum < 4) ? ipnum : 0;
@@ -53,13 +53,13 @@ static void eiointc_update_irq(struct loongarch_eiointc *s, int irq, int level) · 8 unmodified lines
5353
5454 if (level) {
5555 /* if not enable return false */
56− if (!test_bit(irq, (unsigned long *)s->enable.reg_u32))
56+ if (!test_bit(irq, (unsigned long *)s->enable))
5757 return;
58− __set_bit(irq, (unsigned long *)s->coreisr.reg_u32[cpu]);
58+ __set_bit(irq, (unsigned long *)s->coreisr[cpu]);
5959 found = find_first_bit(s->sw_coreisr[cpu][ipnum], EIOINTC_IRQS);
6060 __set_bit(irq, s->sw_coreisr[cpu][ipnum]);
6161 } else {
62− __clear_bit(irq, (unsigned long *)s->coreisr.reg_u32[cpu]);
62+ __clear_bit(irq, (unsigned long *)s->coreisr[cpu]);
6363 __clear_bit(irq, s->sw_coreisr[cpu][ipnum]);
6464 found = find_first_bit(s->sw_coreisr[cpu][ipnum], EIOINTC_IRQS);
6565 }
@@ -94,7 +94,7 @@ static inline void eiointc_update_sw_coremap(struct loongarch_eiointc *s, · 28 unmodified lines
9494 if (s->sw_coremap[irq + i] == cpu)
9595 continue;
9696
97− if (notify && test_bit(irq + i, (unsigned long *)s->isr.reg_u8)) {
97+ if (notify && test_bit(irq + i, (unsigned long *)s->isr)) {
9898 /* lower irq at old cpu and raise irq at new cpu */
9999 eiointc_update_irq(s, irq + i, 0);
100100 s->sw_coremap[irq + i] = cpu;
@@ -108,7 +108,7 @@ static inline void eiointc_update_sw_coremap(struct loongarch_eiointc *s, · 7 unmodified lines
108108 void eiointc_set_irq(struct loongarch_eiointc *s, int irq, int level)
109109 {
110110 unsigned long flags;
111− unsigned long *isr = (unsigned long *)s->isr.reg_u8;
111+ unsigned long *isr = (unsigned long *)s->isr;
112112
113113 spin_lock_irqsave(&s->lock, flags);
114114 level ? __set_bit(irq, isr) : __clear_bit(irq, isr);
@@ -127,27 +127,27 @@ static int loongarch_eiointc_read(struct kvm_vcpu *vcpu, struct loongarch_eioint · 12 unmodified lines
127127 switch (offset) {
128128 case EIOINTC_NODETYPE_START ... EIOINTC_NODETYPE_END:
129129 index = (offset - EIOINTC_NODETYPE_START) >> 3;
130− data = s->nodetype.reg_u64[index];
130+ data = s->nodetype[index];
131131 break;
132132 case EIOINTC_IPMAP_START ... EIOINTC_IPMAP_END:
133133 index = (offset - EIOINTC_IPMAP_START) >> 3;
134− data = s->ipmap.reg_u64;
134+ data = s->ipmap;
135135 break;
136136 case EIOINTC_ENABLE_START ... EIOINTC_ENABLE_END:
137137 index = (offset - EIOINTC_ENABLE_START) >> 3;
138− data = s->enable.reg_u64[index];
138+ data = s->enable[index];
139139 break;
140140 case EIOINTC_BOUNCE_START ... EIOINTC_BOUNCE_END:
141141 index = (offset - EIOINTC_BOUNCE_START) >> 3;
142− data = s->bounce.reg_u64[index];
142+ data = s->bounce[index];
143143 break;
144144 case EIOINTC_COREISR_START ... EIOINTC_COREISR_END:
145145 index = (offset - EIOINTC_COREISR_START) >> 3;
146− data = s->coreisr.reg_u64[vcpu->vcpu_id][index];
146+ data = s->coreisr[vcpu->vcpu_id][index];
147147 break;
148148 case EIOINTC_COREMAP_START ... EIOINTC_COREMAP_END:
149149 index = (offset - EIOINTC_COREMAP_START) >> 3;
150− data = s->coremap.reg_u64[index];
150+ data = s->coremap[index];
151151 break;
152152 default:
153153 ret = -EINVAL;
@@ -223,26 +223,26 @@ static int loongarch_eiointc_write(struct kvm_vcpu *vcpu, · 69 unmodified lines
223223 switch (offset) {
224224 case EIOINTC_NODETYPE_START ... EIOINTC_NODETYPE_END:
225225 index = (offset - EIOINTC_NODETYPE_START) >> 3;
226− old = s->nodetype.reg_u64[index];
227− s->nodetype.reg_u64[index] = (old & ~mask) | data;
226+ old = s->nodetype[index];
227+ s->nodetype[index] = (old & ~mask) | data;
228228 break;
229229 case EIOINTC_IPMAP_START ... EIOINTC_IPMAP_END:
230230 /*
231231 * ipmap cannot be set at runtime, can be set only at the beginning
232232 * of irqchip driver, need not update upper irq level
233233 */
234− old = s->ipmap.reg_u64;
235− s->ipmap.reg_u64 = (old & ~mask) | data;
234+ old = s->ipmap;
235+ s->ipmap = (old & ~mask) | data;
236236 break;
237237 case EIOINTC_ENABLE_START ... EIOINTC_ENABLE_END:
238238 index = (offset - EIOINTC_ENABLE_START) >> 3;
239− old = s->enable.reg_u64[index];
240− s->enable.reg_u64[index] = (old & ~mask) | data;
239+ old = s->enable[index];
240+ s->enable[index] = (old & ~mask) | data;
241241 /*
242242 * 1: enable irq.
243243 * update irq when isr is set.
244244 */
245− data = s->enable.reg_u64[index] & ~old & s->isr.reg_u64[index];
245+ data = s->enable[index] & ~old & s->isr[index];
246246 while (data) {
247247 irq = __ffs(data);
248248 eiointc_update_irq(s, irq + index * 64, 1);
@@ -252,7 +252,7 @@ static int loongarch_eiointc_write(struct kvm_vcpu *vcpu, · 3 unmodified lines
252252 * 0: disable irq.
253253 * update irq when isr is set.
254254 */
255− data = ~s->enable.reg_u64[index] & old & s->isr.reg_u64[index];
255+ data = ~s->enable[index] & old & s->isr[index];
256256 while (data) {
257257 irq = __ffs(data);
258258 eiointc_update_irq(s, irq + index * 64, 0);
@@ -262,16 +262,16 @@ static int loongarch_eiointc_write(struct kvm_vcpu *vcpu, · 3 unmodified lines
262262 case EIOINTC_BOUNCE_START ... EIOINTC_BOUNCE_END:
263263 /* do not emulate hw bounced irq routing */
264264 index = (offset - EIOINTC_BOUNCE_START) >> 3;
265− old = s->bounce.reg_u64[index];
266− s->bounce.reg_u64[index] = (old & ~mask) | data;
265+ old = s->bounce[index];
266+ s->bounce[index] = (old & ~mask) | data;
267267 break;
268268 case EIOINTC_COREISR_START ... EIOINTC_COREISR_END:
269269 index = (offset - EIOINTC_COREISR_START) >> 3;
270270 /* use attrs to get current cpu index */
271271 cpu = vcpu->vcpu_id;
272− old = s->coreisr.reg_u64[cpu][index];
272+ old = s->coreisr[cpu][index];
273273 /* write 1 to clear interrupt */
274− s->coreisr.reg_u64[cpu][index] = old & ~data;
274+ s->coreisr[cpu][index] = old & ~data;
275275 data &= old;
276276 while (data) {
277277 irq = __ffs(data);
@@ -281,9 +281,9 @@ static int loongarch_eiointc_write(struct kvm_vcpu *vcpu, · 3 unmodified lines
281281 break;
282282 case EIOINTC_COREMAP_START ... EIOINTC_COREMAP_END:
283283 index = (offset - EIOINTC_COREMAP_START) >> 3;
284− old = s->coremap.reg_u64[index];
285− s->coremap.reg_u64[index] = (old & ~mask) | data;
286− data = s->coremap.reg_u64[index];
284+ old = s->coremap[index];
285+ s->coremap[index] = (old & ~mask) | data;
286+ data = s->coremap[index];
287287 eiointc_update_sw_coremap(s, index * 8, data, sizeof(data), true);
288288 break;
289289 default:
@@ -451,10 +451,10 @@ static int kvm_eiointc_ctrl_access(struct kvm_device *dev, · 161 unmodified lines
451451 break;
452452 case KVM_DEV_LOONGARCH_EXTIOI_CTRL_LOAD_FINISHED:
453453 eiointc_set_sw_coreisr(s);
454− for (i = 0; i < (EIOINTC_IRQS / 4); i++) {
455− start_irq = i * 4;
454+ for (i = 0; i < (EIOINTC_IRQS / 8); i++) {
455+ start_irq = i * 8;
456456 eiointc_update_sw_coremap(s, start_irq,
457− s->coremap.reg_u32[i], sizeof(u32), false);
457+ s->coremap[i], sizeof(u64), false);
458458 }
459459 break;
460460 default:
@@ -481,34 +481,34 @@ static int kvm_eiointc_regs_access(struct kvm_device *dev, · 20 unmodified lines
481481 switch (addr) {
482482 case EIOINTC_NODETYPE_START ... EIOINTC_NODETYPE_END:
483483 offset = (addr - EIOINTC_NODETYPE_START) / 4;
484− p = &s->nodetype.reg_u32[offset];
484+ p = s->nodetype + offset * 4;
485485 break;
486486 case EIOINTC_IPMAP_START ... EIOINTC_IPMAP_END:
487487 offset = (addr - EIOINTC_IPMAP_START) / 4;
488− p = &s->ipmap.reg_u32[offset];
488+ p = &s->ipmap + offset * 4;
489489 break;
490490 case EIOINTC_ENABLE_START ... EIOINTC_ENABLE_END:
491491 offset = (addr - EIOINTC_ENABLE_START) / 4;
492− p = &s->enable.reg_u32[offset];
492+ p = s->enable + offset * 4;
493493 break;
494494 case EIOINTC_BOUNCE_START ... EIOINTC_BOUNCE_END:
495495 offset = (addr - EIOINTC_BOUNCE_START) / 4;
496− p = &s->bounce.reg_u32[offset];
496+ p = s->bounce + offset * 4;
497497 break;
498498 case EIOINTC_ISR_START ... EIOINTC_ISR_END:
499499 offset = (addr - EIOINTC_ISR_START) / 4;
500− p = &s->isr.reg_u32[offset];
500+ p = s->isr + offset * 4;
501501 break;
502502 case EIOINTC_COREISR_START ... EIOINTC_COREISR_END:
503503 if (cpu >= s->num_cpu)
504504 return -EINVAL;
505505
506506 offset = (addr - EIOINTC_COREISR_START) / 4;
507− p = &s->coreisr.reg_u32[cpu][offset];
507+ p = s->coreisr[cpu] + offset * 4;
508508 break;
509509 case EIOINTC_COREMAP_START ... EIOINTC_COREMAP_END:
510510 offset = (addr - EIOINTC_COREMAP_START) / 4;
511− p = &s->coremap.reg_u32[offset];
511+ p = s->coremap + offset * 4;
512512 break;
513513 default:
514514 kvm_err("%s: unknown eiointc register, addr = %d\n", __func__, addr);
arch/loongarch/kvm/interrupt.c+13−2
@@ -21,6 +21,7 @@ static unsigned int priority_to_irq[EXCCODE_INT_NUM] = { · 20 unmodified lines
2121 [INT_HWI5] = CPU_IP5,
2222 [INT_HWI6] = CPU_IP6,
2323 [INT_HWI7] = CPU_IP7,
24+ [INT_AVEC] = CPU_AVEC,
2425 };
2526
2627 static int kvm_irq_deliver(struct kvm_vcpu *vcpu, unsigned int priority)
@@ -31,6 +32,11 @@ static int kvm_irq_deliver(struct kvm_vcpu *vcpu, unsigned int priority) · 4 unmodified lines
3132 if (priority < EXCCODE_INT_NUM)
3233 irq = priority_to_irq[priority];
3334
35+ if (cpu_has_msgint && (priority == INT_AVEC)) {
36+ set_gcsr_estat(irq);
37+ return 1;
38+ }
39+
3440 switch (priority) {
3541 case INT_TI:
3642 case INT_IPI:
@@ -58,6 +64,11 @@ static int kvm_irq_clear(struct kvm_vcpu *vcpu, unsigned int priority) · 21 unmodified lines
5864 if (priority < EXCCODE_INT_NUM)
5965 irq = priority_to_irq[priority];
6066
67+ if (cpu_has_msgint && (priority == INT_AVEC)) {
68+ clear_gcsr_estat(irq);
69+ return 1;
70+ }
71+
6172 switch (priority) {
6273 case INT_TI:
6374 case INT_IPI:
@@ -83,10 +94,10 @@ void kvm_deliver_intr(struct kvm_vcpu *vcpu) · 19 unmodified lines
8394 unsigned long *pending = &vcpu->arch.irq_pending;
8495 unsigned long *pending_clr = &vcpu->arch.irq_clear;
8596
86− for_each_set_bit(priority, pending_clr, INT_IPI + 1)
97+ for_each_set_bit(priority, pending_clr, EXCCODE_INT_NUM)
8798 kvm_irq_clear(vcpu, priority);
8899
89− for_each_set_bit(priority, pending, INT_IPI + 1)
100+ for_each_set_bit(priority, pending, EXCCODE_INT_NUM)
90101 kvm_irq_deliver(vcpu, priority);
91102 }
92103
arch/loongarch/kvm/vcpu.c+19−4
@@ -659,8 +659,7 @@ static int _kvm_get_cpucfg_mask(int id, u64 *v) · 658 unmodified lines
659659 *v = GENMASK(31, 0);
660660 return 0;
661661 case LOONGARCH_CPUCFG1:
662− /* CPUCFG1_MSGINT is not supported by KVM */
663− *v = GENMASK(25, 0);
662+ *v = GENMASK(26, 0);
664663 return 0;
665664 case LOONGARCH_CPUCFG2:
666665 /* CPUCFG2 features unconditionally supported by KVM */
@@ -728,6 +727,10 @@ static int kvm_check_cpucfg(int id, u64 val) · 61 unmodified lines
728727 return -EINVAL;
729728
730729 switch (id) {
730+ case LOONGARCH_CPUCFG1:
731+ if ((val & CPUCFG1_MSGINT) && !cpu_has_msgint)
732+ return -EINVAL;
733+ return 0;
731734 case LOONGARCH_CPUCFG2:
732735 if (!(val & CPUCFG2_LLFTP))
733736 /* Guests must have a constant timer */
@@ -1473,8 +1476,8 @@ int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq) · 739 unmodified lines
14731476 return 0;
14741477 }
14751478
1476−long kvm_arch_vcpu_async_ioctl(struct file *filp,
1477− unsigned int ioctl, unsigned long arg)
1479+long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
1480+ unsigned long arg)
14781481 {
14791482 void __user *argp = (void __user *)arg;
14801483 struct kvm_vcpu *vcpu = filp->private_data;
@@ -1657,6 +1660,12 @@ static int _kvm_vcpu_load(struct kvm_vcpu *vcpu, int cpu) · 176 unmodified lines
16571660 kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_DMWIN2);
16581661 kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_DMWIN3);
16591662 kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_LLBCTL);
1663+ if (cpu_has_msgint) {
1664+ kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_ISR0);
1665+ kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_ISR1);
1666+ kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_ISR2);
1667+ kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_ISR3);
1668+ }
16601669
16611670 /* Restore Root.GINTC from unused Guest.GINTC register */
16621671 write_csr_gintc(csr->csrs[LOONGARCH_CSR_GINTC]);
@@ -1746,6 +1755,12 @@ static int _kvm_vcpu_put(struct kvm_vcpu *vcpu, int cpu) · 83 unmodified lines
17461755 kvm_save_hw_gcsr(csr, LOONGARCH_CSR_DMWIN1);
17471756 kvm_save_hw_gcsr(csr, LOONGARCH_CSR_DMWIN2);
17481757 kvm_save_hw_gcsr(csr, LOONGARCH_CSR_DMWIN3);
1758+ if (cpu_has_msgint) {
1759+ kvm_save_hw_gcsr(csr, LOONGARCH_CSR_ISR0);
1760+ kvm_save_hw_gcsr(csr, LOONGARCH_CSR_ISR1);
1761+ kvm_save_hw_gcsr(csr, LOONGARCH_CSR_ISR2);
1762+ kvm_save_hw_gcsr(csr, LOONGARCH_CSR_ISR3);
1763+ }
17491764
17501765 vcpu->arch.aux_inuse |= KVM_LARCH_SWCSR_LATEST;
17511766
arch/loongarch/kvm/vm.c+27−13
@@ -6,6 +6,7 @@ · 5 unmodified lines
66 #include <linux/kvm_host.h>
77 #include <asm/kvm_mmu.h>
88 #include <asm/kvm_vcpu.h>
9+#include <asm/kvm_csr.h>
910 #include <asm/kvm_eiointc.h>
1011 #include <asm/kvm_pch_pic.h>
1112
@@ -24,6 +25,23 @@ const struct kvm_stats_header kvm_vm_stats_header = { · 12 unmodified lines
2425 sizeof(kvm_vm_stats_desc),
2526 };
2627
28+static void kvm_vm_init_features(struct kvm *kvm)
29+{
30+ unsigned long val;
31+
32+ val = read_csr_gcfg();
33+ if (val & CSR_GCFG_GPMP)
34+ kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PMU);
35+
36+ /* Enable all PV features by default */
37+ kvm->arch.pv_features = BIT(KVM_FEATURE_IPI);
38+ kvm->arch.kvm_features = BIT(KVM_LOONGARCH_VM_FEAT_PV_IPI);
39+ if (kvm_pvtime_supported()) {
40+ kvm->arch.pv_features |= BIT(KVM_FEATURE_STEAL_TIME);
41+ kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_STEALTIME);
42+ }
43+}
44+
2745 int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
2846 {
2947 int i;
@@ -42,11 +60,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) · 12 unmodified lines
4260 spin_lock_init(&kvm->arch.phyid_map_lock);
4361
4462 kvm_init_vmcs(kvm);
45−
46− /* Enable all PV features by default */
47− kvm->arch.pv_features = BIT(KVM_FEATURE_IPI);
48− if (kvm_pvtime_supported())
49− kvm->arch.pv_features |= BIT(KVM_FEATURE_STEAL_TIME);
63+ kvm_vm_init_features(kvm);
5064
5165 /*
5266 * cpu_vabits means user address space only (a half of total).
@@ -136,18 +150,18 @@ static int kvm_vm_feature_has_attr(struct kvm *kvm, struct kvm_device_attr *attr · 83 unmodified lines
136150 if (cpu_has_lbt_mips)
137151 return 0;
138152 return -ENXIO;
139− case KVM_LOONGARCH_VM_FEAT_PMU:
140− if (cpu_has_pmp)
153+ case KVM_LOONGARCH_VM_FEAT_PTW:
154+ if (cpu_has_ptw)
141155 return 0;
142156 return -ENXIO;
143− case KVM_LOONGARCH_VM_FEAT_PV_IPI:
144− return 0;
145− case KVM_LOONGARCH_VM_FEAT_PV_STEALTIME:
146− if (kvm_pvtime_supported())
157+ case KVM_LOONGARCH_VM_FEAT_MSGINT:
158+ if (cpu_has_msgint)
147159 return 0;
148160 return -ENXIO;
149− case KVM_LOONGARCH_VM_FEAT_PTW:
150− if (cpu_has_ptw)
161+ case KVM_LOONGARCH_VM_FEAT_PMU:
162+ case KVM_LOONGARCH_VM_FEAT_PV_IPI:
163+ case KVM_LOONGARCH_VM_FEAT_PV_STEALTIME:
164+ if (kvm_vm_support(&kvm->arch, attr->attr))
151165 return 0;
152166 return -ENXIO;
153167 default:
arch/mips/kvm/Kconfig+0−1
@@ -22,7 +22,6 @@ config KVM · 21 unmodified lines
2222 select EXPORT_UASM
2323 select KVM_COMMON
2424 select KVM_GENERIC_DIRTYLOG_READ_PROTECT
25− select HAVE_KVM_VCPU_ASYNC_IOCTL
2625 select KVM_MMIO
2726 select KVM_GENERIC_MMU_NOTIFIER
2827 select KVM_GENERIC_HARDWARE_ENABLING
arch/mips/kvm/mips.c+2−2
@@ -895,8 +895,8 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu, · 894 unmodified lines
895895 return r;
896896 }
897897
898−long kvm_arch_vcpu_async_ioctl(struct file *filp, unsigned int ioctl,
899− unsigned long arg)
898+long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
899+ unsigned long arg)
900900 {
901901 struct kvm_vcpu *vcpu = filp->private_data;
902902 void __user *argp = (void __user *)arg;
arch/powerpc/kvm/Kconfig+0−1
@@ -20,7 +20,6 @@ if VIRTUALIZATION · 19 unmodified lines
2020 config KVM
2121 bool
2222 select KVM_COMMON
23− select HAVE_KVM_VCPU_ASYNC_IOCTL
2423 select KVM_VFIO
2524 select HAVE_KVM_IRQ_BYPASS
2625
arch/powerpc/kvm/powerpc.c+2−2
@@ -2028,8 +2028,8 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · 2027 unmodified lines
20282028 return -EINVAL;
20292029 }
20302030
2031−long kvm_arch_vcpu_async_ioctl(struct file *filp,
2032− unsigned int ioctl, unsigned long arg)
2031+long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
2032+ unsigned long arg)
20332033 {
20342034 struct kvm_vcpu *vcpu = filp->private_data;
20352035 void __user *argp = (void __user *)arg;
arch/riscv/include/asm/kvm_host.h+6−0
@@ -59,6 +59,9 @@ · 58 unmodified lines
5959 BIT(IRQ_VS_TIMER) | \
6060 BIT(IRQ_VS_EXT))
6161
62+#define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
63+ KVM_DIRTY_LOG_INITIALLY_SET)
64+
6265 struct kvm_vm_stat {
6366 struct kvm_vm_stat_generic generic;
6467 };
@@ -327,4 +330,7 @@ bool kvm_riscv_vcpu_stopped(struct kvm_vcpu *vcpu); · 262 unmodified lines
327330
328331 void kvm_riscv_vcpu_record_steal_time(struct kvm_vcpu *vcpu);
329332
333+/* Flags representing implementation specific details */
334+DECLARE_STATIC_KEY_FALSE(kvm_riscv_vsstage_tlb_no_gpa);
335+
330336 #endif /* __RISCV_KVM_HOST_H__ */
arch/riscv/include/asm/kvm_tlb.h+1−0
@@ -49,6 +49,7 @@ void kvm_riscv_local_hfence_vvma_gva(unsigned long vmid, · 48 unmodified lines
4949 unsigned long gva, unsigned long gvsz,
5050 unsigned long order);
5151 void kvm_riscv_local_hfence_vvma_all(unsigned long vmid);
52+void kvm_riscv_local_tlb_sanitize(struct kvm_vcpu *vcpu);
5253
5354 void kvm_riscv_tlb_flush_process(struct kvm_vcpu *vcpu);
5455
arch/riscv/include/asm/kvm_vcpu_sbi.h+4−1
@@ -69,7 +69,9 @@ struct kvm_vcpu_sbi_extension { · 68 unmodified lines
6969 unsigned long reg_size, const void *reg_val);
7070 };
7171
72−void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run);
72+int kvm_riscv_vcpu_sbi_forward_handler(struct kvm_vcpu *vcpu,
73+ struct kvm_run *run,
74+ struct kvm_vcpu_sbi_return *retdata);
7375 void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
7476 struct kvm_run *run,
7577 u32 type, u64 flags);
@@ -105,6 +107,7 @@ extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_dbcn; · 29 unmodified lines
105107 extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_susp;
106108 extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_sta;
107109 extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_fwft;
110+extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_mpxy;
108111 extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_experimental;
109112 extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_vendor;
110113
arch/riscv/include/asm/kvm_vmid.h+0−1
@@ -22,6 +22,5 @@ unsigned long kvm_riscv_gstage_vmid_bits(void); · 21 unmodified lines
2222 int kvm_riscv_gstage_vmid_init(struct kvm *kvm);
2323 bool kvm_riscv_gstage_vmid_ver_changed(struct kvm_vmid *vmid);
2424 void kvm_riscv_gstage_vmid_update(struct kvm_vcpu *vcpu);
25−void kvm_riscv_gstage_vmid_sanitize(struct kvm_vcpu *vcpu);
2625
2726 #endif
arch/riscv/include/uapi/asm/kvm.h+3−0
@@ -23,6 +23,8 @@ · 22 unmodified lines
2323 #define KVM_INTERRUPT_SET -1U
2424 #define KVM_INTERRUPT_UNSET -2U
2525
26+#define KVM_EXIT_FAIL_ENTRY_NO_VSFILE (1ULL << 0)
27+
2628 /* for KVM_GET_REGS and KVM_SET_REGS */
2729 struct kvm_regs {
2830 };
@@ -211,6 +213,7 @@ enum KVM_RISCV_SBI_EXT_ID { · 182 unmodified lines
211213 KVM_RISCV_SBI_EXT_STA,
212214 KVM_RISCV_SBI_EXT_SUSP,
213215 KVM_RISCV_SBI_EXT_FWFT,
216+ KVM_RISCV_SBI_EXT_MPXY,
214217 KVM_RISCV_SBI_EXT_MAX,
215218 };
216219
arch/riscv/kvm/Kconfig+0−1
@@ -23,7 +23,6 @@ config KVM · 22 unmodified lines
2323 select HAVE_KVM_IRQCHIP
2424 select HAVE_KVM_IRQ_ROUTING
2525 select HAVE_KVM_MSI
26− select HAVE_KVM_VCPU_ASYNC_IOCTL
2726 select HAVE_KVM_READONLY_MEM
2827 select HAVE_KVM_DIRTY_RING_ACQ_REL
2928 select KVM_COMMON
arch/riscv/kvm/Makefile+1−0
@@ -27,6 +27,7 @@ kvm-y += vcpu_onereg.o · 26 unmodified lines
2727 kvm-$(CONFIG_RISCV_PMU_SBI) += vcpu_pmu.o
2828 kvm-y += vcpu_sbi.o
2929 kvm-y += vcpu_sbi_base.o
30+kvm-y += vcpu_sbi_forward.o
3031 kvm-y += vcpu_sbi_fwft.o
3132 kvm-y += vcpu_sbi_hsm.o
3233 kvm-$(CONFIG_RISCV_PMU_SBI) += vcpu_sbi_pmu.o
arch/riscv/kvm/aia_imsic.c+1−1
@@ -814,7 +814,7 @@ int kvm_riscv_vcpu_aia_imsic_update(struct kvm_vcpu *vcpu) · 813 unmodified lines
814814 /* For HW acceleration mode, we can't continue */
815815 if (kvm->arch.aia.mode == KVM_DEV_RISCV_AIA_MODE_HWACCEL) {
816816 run->fail_entry.hardware_entry_failure_reason =
817− CSR_HSTATUS;
817+ KVM_EXIT_FAIL_ENTRY_NO_VSFILE;
818818 run->fail_entry.cpu = vcpu->cpu;
819819 run->exit_reason = KVM_EXIT_FAIL_ENTRY;
820820 return 0;
arch/riscv/kvm/main.c+14−0
@@ -15,6 +15,18 @@ · 14 unmodified lines
1515 #include <asm/kvm_nacl.h>
1616 #include <asm/sbi.h>
1717
18+DEFINE_STATIC_KEY_FALSE(kvm_riscv_vsstage_tlb_no_gpa);
19+
20+static void kvm_riscv_setup_vendor_features(void)
21+{
22+ /* Andes AX66: split two-stage TLBs */
23+ if (riscv_cached_mvendorid(0) == ANDES_VENDOR_ID &&
24+ (riscv_cached_marchid(0) & 0xFFFF) == 0x8A66) {
25+ static_branch_enable(&kvm_riscv_vsstage_tlb_no_gpa);
26+ kvm_info("VS-stage TLB does not cache guest physical address and VMID\n");
27+ }
28+}
29+
1830 long kvm_arch_dev_ioctl(struct file *filp,
1931 unsigned int ioctl, unsigned long arg)
2032 {
@@ -160,6 +172,8 @@ static int __init riscv_kvm_init(void) · 139 unmodified lines
160172 kvm_info("AIA available with %d guest external interrupts\n",
161173 kvm_riscv_aia_nr_hgei);
162174
175+ kvm_riscv_setup_vendor_features();
176+
163177 kvm_register_perf_callbacks(NULL);
164178
165179 rc = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
arch/riscv/kvm/mmu.c+4−1
@@ -161,8 +161,11 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, · 160 unmodified lines
161161 * allocated dirty_bitmap[], dirty pages will be tracked while
162162 * the memory slot is write protected.
163163 */
164− if (change != KVM_MR_DELETE && new->flags & KVM_MEM_LOG_DIRTY_PAGES)
164+ if (change != KVM_MR_DELETE && new->flags & KVM_MEM_LOG_DIRTY_PAGES) {
165+ if (kvm_dirty_log_manual_protect_and_init_set(kvm))
166+ return;
165167 mmu_wp_memory_region(kvm, new->id);
168+ }
166169 }
167170
168171 int kvm_arch_prepare_memory_region(struct kvm *kvm,
arch/riscv/kvm/tlb.c+30−0
@@ -158,6 +158,36 @@ void kvm_riscv_local_hfence_vvma_all(unsigned long vmid) · 157 unmodified lines
158158 csr_write(CSR_HGATP, hgatp);
159159 }
160160
161+void kvm_riscv_local_tlb_sanitize(struct kvm_vcpu *vcpu)
162+{
163+ unsigned long vmid;
164+
165+ if (!kvm_riscv_gstage_vmid_bits() ||
166+ vcpu->arch.last_exit_cpu == vcpu->cpu)
167+ return;
168+
169+ /*
170+ * On RISC-V platforms with hardware VMID support, we share same
171+ * VMID for all VCPUs of a particular Guest/VM. This means we might
172+ * have stale G-stage TLB entries on the current Host CPU due to
173+ * some other VCPU of the same Guest which ran previously on the
174+ * current Host CPU.
175+ *
176+ * To cleanup stale TLB entries, we simply flush all G-stage TLB
177+ * entries by VMID whenever underlying Host CPU changes for a VCPU.
178+ */
179+
180+ vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid);
181+ kvm_riscv_local_hfence_gvma_vmid_all(vmid);
182+
183+ /*
184+ * Flush VS-stage TLB entries for implementation where VS-stage
185+ * TLB does not cahce guest physical address and VMID.
186+ */
187+ if (static_branch_unlikely(&kvm_riscv_vsstage_tlb_no_gpa))
188+ kvm_riscv_local_hfence_vvma_all(vmid);
189+}
190+
161191 void kvm_riscv_fence_i_process(struct kvm_vcpu *vcpu)
162192 {
163193 kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_FENCE_I_RCVD);
arch/riscv/kvm/vcpu.c+3−3
@@ -238,8 +238,8 @@ vm_fault_t kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf) · 237 unmodified lines
238238 return VM_FAULT_SIGBUS;
239239 }
240240
241−long kvm_arch_vcpu_async_ioctl(struct file *filp,
242− unsigned int ioctl, unsigned long arg)
241+long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
242+ unsigned long arg)
243243 {
244244 struct kvm_vcpu *vcpu = filp->private_data;
245245 void __user *argp = (void __user *)arg;
@@ -968,7 +968,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) · 722 unmodified lines
968968 * Note: This should be done after G-stage VMID has been
969969 * updated using kvm_riscv_gstage_vmid_ver_changed()
970970 */
971− kvm_riscv_gstage_vmid_sanitize(vcpu);
971+ kvm_riscv_local_tlb_sanitize(vcpu);
972972
973973 trace_kvm_entry(vcpu);
974974
arch/riscv/kvm/vcpu_insn.c+22−0
@@ -298,6 +298,22 @@ static int system_opcode_insn(struct kvm_vcpu *vcpu, struct kvm_run *run, · 297 unmodified lines
298298 return (rc <= 0) ? rc : 1;
299299 }
300300
301+static bool is_load_guest_page_fault(unsigned long scause)
302+{
303+ /**
304+ * If a g-stage page fault occurs, the direct approach
305+ * is to let the g-stage page fault handler handle it
306+ * naturally, however, calling the g-stage page fault
307+ * handler here seems rather strange.
308+ * Considering this is a corner case, we can directly
309+ * return to the guest and re-execute the same PC, this
310+ * will trigger a g-stage page fault again and then the
311+ * regular g-stage page fault handler will populate
312+ * g-stage page table.
313+ */
314+ return (scause == EXC_LOAD_GUEST_PAGE_FAULT);
315+}
316+
301317 /**
302318 * kvm_riscv_vcpu_virtual_insn -- Handle virtual instruction trap
303319 *
@@ -323,6 +339,8 @@ int kvm_riscv_vcpu_virtual_insn(struct kvm_vcpu *vcpu, struct kvm_run *run, · 19 unmodified lines
323339 ct->sepc,
324340 &utrap);
325341 if (utrap.scause) {
342+ if (is_load_guest_page_fault(utrap.scause))
343+ return 1;
326344 utrap.sepc = ct->sepc;
327345 kvm_riscv_vcpu_trap_redirect(vcpu, &utrap);
328346 return 1;
@@ -378,6 +396,8 @@ int kvm_riscv_vcpu_mmio_load(struct kvm_vcpu *vcpu, struct kvm_run *run, · 49 unmodified lines
378396 insn = kvm_riscv_vcpu_unpriv_read(vcpu, true, ct->sepc,
379397 &utrap);
380398 if (utrap.scause) {
399+ if (is_load_guest_page_fault(utrap.scause))
400+ return 1;
381401 /* Redirect trap if we failed to read instruction */
382402 utrap.sepc = ct->sepc;
383403 kvm_riscv_vcpu_trap_redirect(vcpu, &utrap);
@@ -504,6 +524,8 @@ int kvm_riscv_vcpu_mmio_store(struct kvm_vcpu *vcpu, struct kvm_run *run, · 120 unmodified lines
504524 insn = kvm_riscv_vcpu_unpriv_read(vcpu, true, ct->sepc,
505525 &utrap);
506526 if (utrap.scause) {
527+ if (is_load_guest_page_fault(utrap.scause))
528+ return 1;
507529 /* Redirect trap if we failed to read instruction */
508530 utrap.sepc = ct->sepc;
509531 kvm_riscv_vcpu_trap_redirect(vcpu, &utrap);
arch/riscv/kvm/vcpu_sbi.c+9−1
@@ -82,6 +82,10 @@ static const struct kvm_riscv_sbi_extension_entry sbi_ext[] = { · 81 unmodified lines
8282 .ext_idx = KVM_RISCV_SBI_EXT_FWFT,
8383 .ext_ptr = &vcpu_sbi_ext_fwft,
8484 },
85+ {
86+ .ext_idx = KVM_RISCV_SBI_EXT_MPXY,
87+ .ext_ptr = &vcpu_sbi_ext_mpxy,
88+ },
8589 {
8690 .ext_idx = KVM_RISCV_SBI_EXT_EXPERIMENTAL,
8791 .ext_ptr = &vcpu_sbi_ext_experimental,
@@ -120,7 +124,9 @@ static bool riscv_vcpu_supports_sbi_ext(struct kvm_vcpu *vcpu, int idx) · 32 unmodified lines
120124 return sext && scontext->ext_status[sext->ext_idx] != KVM_RISCV_SBI_EXT_STATUS_UNAVAILABLE;
121125 }
122126
123−void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run)
127+int kvm_riscv_vcpu_sbi_forward_handler(struct kvm_vcpu *vcpu,
128+ struct kvm_run *run,
129+ struct kvm_vcpu_sbi_return *retdata)
124130 {
125131 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
126132
@@ -137,6 +143,8 @@ void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run) · 10 unmodified lines
137143 run->riscv_sbi.args[5] = cp->a5;
138144 run->riscv_sbi.ret[0] = SBI_ERR_NOT_SUPPORTED;
139145 run->riscv_sbi.ret[1] = 0;
146+ retdata->uexit = true;
147+ return 0;
140148 }
141149
142150 void kvm_riscv_vcpu_sbi_system_reset(struct kvm_vcpu *vcpu,
arch/riscv/kvm/vcpu_sbi_base.c+1−27
@@ -41,8 +41,7 @@ static int kvm_sbi_ext_base_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 40 unmodified lines
4141 * For experimental/vendor extensions
4242 * forward it to the userspace
4343 */
44− kvm_riscv_vcpu_sbi_forward(vcpu, run);
45− retdata->uexit = true;
44+ return kvm_riscv_vcpu_sbi_forward_handler(vcpu, run, retdata);
4645 } else {
4746 sbi_ext = kvm_vcpu_sbi_find_ext(vcpu, cp->a0);
4847 *out_val = sbi_ext && sbi_ext->probe ?
@@ -71,28 +70,3 @@ const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_base = { · 22 unmodified lines
7170 .extid_end = SBI_EXT_BASE,
7271 .handler = kvm_sbi_ext_base_handler,
7372 };
74−
75−static int kvm_sbi_ext_forward_handler(struct kvm_vcpu *vcpu,
76− struct kvm_run *run,
77− struct kvm_vcpu_sbi_return *retdata)
78−{
79− /*
80− * Both SBI experimental and vendor extensions are
81− * unconditionally forwarded to userspace.
82− */
83− kvm_riscv_vcpu_sbi_forward(vcpu, run);
84− retdata->uexit = true;
85− return 0;
86−}
87−
88−const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_experimental = {
89− .extid_start = SBI_EXT_EXPERIMENTAL_START,
90− .extid_end = SBI_EXT_EXPERIMENTAL_END,
91− .handler = kvm_sbi_ext_forward_handler,
92−};
93−
94−const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_vendor = {
95− .extid_start = SBI_EXT_VENDOR_START,
96− .extid_end = SBI_EXT_VENDOR_END,
97− .handler = kvm_sbi_ext_forward_handler,
98−};
arch/riscv/kvm/vcpu_sbi_forward.c+34−0
@@ -0,0 +1,34 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/*
3+ * Copyright (c) 2025 Ventana Micro Systems Inc.
4+ */
5+
6+#include <linux/kvm_host.h>
7+#include <asm/kvm_vcpu_sbi.h>
8+#include <asm/sbi.h>
9+
10+const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_experimental = {
11+ .extid_start = SBI_EXT_EXPERIMENTAL_START,
12+ .extid_end = SBI_EXT_EXPERIMENTAL_END,
13+ .handler = kvm_riscv_vcpu_sbi_forward_handler,
14+};
15+
16+const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_vendor = {
17+ .extid_start = SBI_EXT_VENDOR_START,
18+ .extid_end = SBI_EXT_VENDOR_END,
19+ .handler = kvm_riscv_vcpu_sbi_forward_handler,
20+};
21+
22+const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_dbcn = {
23+ .extid_start = SBI_EXT_DBCN,
24+ .extid_end = SBI_EXT_DBCN,
25+ .default_disabled = true,
26+ .handler = kvm_riscv_vcpu_sbi_forward_handler,
27+};
28+
29+const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_mpxy = {
30+ .extid_start = SBI_EXT_MPXY,
31+ .extid_end = SBI_EXT_MPXY,
32+ .default_disabled = true,
33+ .handler = kvm_riscv_vcpu_sbi_forward_handler,
34+};
arch/riscv/kvm/vcpu_sbi_replace.c+0−32
@@ -185,35 +185,3 @@ const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_srst = { · 184 unmodified lines
185185 .extid_end = SBI_EXT_SRST,
186186 .handler = kvm_sbi_ext_srst_handler,
187187 };
188−
189−static int kvm_sbi_ext_dbcn_handler(struct kvm_vcpu *vcpu,
190− struct kvm_run *run,
191− struct kvm_vcpu_sbi_return *retdata)
192−{
193− struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
194− unsigned long funcid = cp->a6;
195−
196− switch (funcid) {
197− case SBI_EXT_DBCN_CONSOLE_WRITE:
198− case SBI_EXT_DBCN_CONSOLE_READ:
199− case SBI_EXT_DBCN_CONSOLE_WRITE_BYTE:
200− /*
201− * The SBI debug console functions are unconditionally
202− * forwarded to the userspace.
203− */
204− kvm_riscv_vcpu_sbi_forward(vcpu, run);
205− retdata->uexit = true;
206− break;
207− default:
208− retdata->err_val = SBI_ERR_NOT_SUPPORTED;
209− }
210−
211− return 0;
212−}
213−
214−const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_dbcn = {
215− .extid_start = SBI_EXT_DBCN,
216− .extid_end = SBI_EXT_DBCN,
217− .default_disabled = true,
218− .handler = kvm_sbi_ext_dbcn_handler,
219−};
arch/riscv/kvm/vcpu_sbi_system.c+1−3
@@ -47,9 +47,7 @@ static int kvm_sbi_ext_susp_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 46 unmodified lines
4747 kvm_riscv_vcpu_sbi_request_reset(vcpu, cp->a1, cp->a2);
4848
4949 /* userspace provides the suspend implementation */
50− kvm_riscv_vcpu_sbi_forward(vcpu, run);
51− retdata->uexit = true;
52− break;
50+ return kvm_riscv_vcpu_sbi_forward_handler(vcpu, run, retdata);
5351 default:
5452 retdata->err_val = SBI_ERR_NOT_SUPPORTED;
5553 break;
arch/riscv/kvm/vcpu_sbi_v01.c+1−2
@@ -32,8 +32,7 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 31 unmodified lines
3232 * The CONSOLE_GETCHAR/CONSOLE_PUTCHAR SBI calls cannot be
3333 * handled in kernel so we forward these to user-space
3434 */
35− kvm_riscv_vcpu_sbi_forward(vcpu, run);
36− retdata->uexit = true;
35+ ret = kvm_riscv_vcpu_sbi_forward_handler(vcpu, run, retdata);
3736 break;
3837 case SBI_EXT_0_1_SET_TIMER:
3938 #if __riscv_xlen == 32
arch/riscv/kvm/vmid.c+0−23
@@ -122,26 +122,3 @@ void kvm_riscv_gstage_vmid_update(struct kvm_vcpu *vcpu) · 121 unmodified lines
122122 kvm_for_each_vcpu(i, v, vcpu->kvm)
123123 kvm_make_request(KVM_REQ_UPDATE_HGATP, v);
124124 }
125−
126−void kvm_riscv_gstage_vmid_sanitize(struct kvm_vcpu *vcpu)
127−{
128− unsigned long vmid;
129−
130− if (!kvm_riscv_gstage_vmid_bits() ||
131− vcpu->arch.last_exit_cpu == vcpu->cpu)
132− return;
133−
134− /*
135− * On RISC-V platforms with hardware VMID support, we share same
136− * VMID for all VCPUs of a particular Guest/VM. This means we might
137− * have stale G-stage TLB entries on the current Host CPU due to
138− * some other VCPU of the same Guest which ran previously on the
139− * current Host CPU.
140− *
141− * To cleanup stale TLB entries, we simply flush all G-stage TLB
142− * entries by VMID whenever underlying Host CPU changes for a VCPU.
143− */
144−
145− vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid);
146− kvm_riscv_local_hfence_gvma_vmid_all(vmid);
147−}
arch/s390/include/asm/kvm_host.h+4−4
@@ -146,6 +146,7 @@ struct kvm_vcpu_stat { · 145 unmodified lines
146146 u64 instruction_diagnose_500;
147147 u64 instruction_diagnose_other;
148148 u64 pfault_sync;
149+ u64 signal_exits;
149150 };
150151
151152 #define PGM_OPERATION 0x01
@@ -631,10 +632,8 @@ struct kvm_s390_pv { · 479 unmodified lines
631632 struct mmu_notifier mmu_notifier;
632633 };
633634
634−struct kvm_arch{
635− void *sca;
636− int use_esca;
637− rwlock_t sca_lock;
635+struct kvm_arch {
636+ struct esca_block *sca;
638637 debug_info_t *dbf;
639638 struct kvm_s390_float_interrupt float_int;
640639 struct kvm_device *flic;
@@ -650,6 +649,7 @@ struct kvm_arch{ · 9 unmodified lines
650649 int user_sigp;
651650 int user_stsi;
652651 int user_instr0;
652+ int user_operexec;
653653 struct s390_io_adapter *adapters[MAX_S390_IO_ADAPTERS];
654654 wait_queue_head_t ipte_wq;
655655 int ipte_lock_count;
arch/s390/include/asm/stacktrace.h+1−0
@@ -66,6 +66,7 @@ struct stack_frame { · 65 unmodified lines
6666 unsigned long sie_flags;
6767 unsigned long sie_control_block_phys;
6868 unsigned long sie_guest_asce;
69+ unsigned long sie_irq;
6970 };
7071 };
7172 unsigned long gprs[10];
arch/s390/kernel/asm-offsets.c+1−0
@@ -67,6 +67,7 @@ int main(void) · 66 unmodified lines
6767 OFFSET(__SF_SIE_FLAGS, stack_frame, sie_flags);
6868 OFFSET(__SF_SIE_CONTROL_PHYS, stack_frame, sie_control_block_phys);
6969 OFFSET(__SF_SIE_GUEST_ASCE, stack_frame, sie_guest_asce);
70+ OFFSET(__SF_SIE_IRQ, stack_frame, sie_irq);
7071 DEFINE(STACK_FRAME_OVERHEAD, sizeof(struct stack_frame));
7172 BLANK();
7273 OFFSET(__SFUSER_BACKCHAIN, stack_frame_user, back_chain);
arch/s390/kernel/entry.S+2−0
@@ -193,6 +193,7 @@ SYM_FUNC_START(__sie64a) · 192 unmodified lines
193193 mvc __SF_SIE_FLAGS(8,%r15),__TI_flags(%r14) # copy thread flags
194194 lmg %r0,%r13,0(%r4) # load guest gprs 0-13
195195 mvi __TI_sie(%r14),1
196+ stosm __SF_SIE_IRQ(%r15),0x03 # enable interrupts
196197 lctlg %c1,%c1,__SF_SIE_GUEST_ASCE(%r15) # load primary asce
197198 lg %r14,__SF_SIE_CONTROL(%r15) # get control block pointer
198199 oi __SIE_PROG0C+3(%r14),1 # we are going into SIE now
@@ -216,6 +217,7 @@ SYM_FUNC_START(__sie64a) · 17 unmodified lines
216217 lg %r14,__LC_CURRENT(%r14)
217218 mvi __TI_sie(%r14),0
218219 SYM_INNER_LABEL(sie_exit, SYM_L_GLOBAL)
220+ stnsm __SF_SIE_IRQ(%r15),0xfc # disable interrupts
219221 lg %r14,__SF_SIE_SAVEAREA(%r15) # load guest register save area
220222 stmg %r0,%r13,0(%r14) # save guest gprs 0-13
221223 xgr %r0,%r0 # clear guest registers to
arch/s390/kvm/Kconfig+1−1
@@ -20,7 +20,6 @@ config KVM · 19 unmodified lines
2020 def_tristate y
2121 prompt "Kernel-based Virtual Machine (KVM) support"
2222 select HAVE_KVM_CPU_RELAX_INTERCEPT
23− select HAVE_KVM_VCPU_ASYNC_IOCTL
2423 select KVM_ASYNC_PF
2524 select KVM_ASYNC_PF_SYNC
2625 select KVM_COMMON
@@ -30,6 +29,7 @@ config KVM · 3 unmodified lines
3029 select HAVE_KVM_NO_POLL
3130 select KVM_VFIO
3231 select MMU_NOTIFIER
32+ select VIRT_XFER_TO_GUEST_WORK
3333 help
3434 Support hosting paravirtualized guest machines using the SIE
3535 virtualization capability on the mainframe. This should work
arch/s390/kvm/gaccess.c+6−21
@@ -109,14 +109,9 @@ struct aste { · 108 unmodified lines
109109
110110 int ipte_lock_held(struct kvm *kvm)
111111 {
112− if (sclp.has_siif) {
113− int rc;
112+ if (sclp.has_siif)
113+ return kvm->arch.sca->ipte_control.kh != 0;
114114
115− read_lock(&kvm->arch.sca_lock);
116− rc = kvm_s390_get_ipte_control(kvm)->kh != 0;
117− read_unlock(&kvm->arch.sca_lock);
118− return rc;
119− }
120115 return kvm->arch.ipte_lock_count != 0;
121116 }
122117
@@ -129,19 +124,16 @@ static void ipte_lock_simple(struct kvm *kvm) · 6 unmodified lines
129124 if (kvm->arch.ipte_lock_count > 1)
130125 goto out;
131126 retry:
132− read_lock(&kvm->arch.sca_lock);
133− ic = kvm_s390_get_ipte_control(kvm);
127+ ic = &kvm->arch.sca->ipte_control;
134128 old = READ_ONCE(*ic);
135129 do {
136130 if (old.k) {
137− read_unlock(&kvm->arch.sca_lock);
138131 cond_resched();
139132 goto retry;
140133 }
141134 new = old;
142135 new.k = 1;
143136 } while (!try_cmpxchg(&ic->val, &old.val, new.val));
144− read_unlock(&kvm->arch.sca_lock);
145137 out:
146138 mutex_unlock(&kvm->arch.ipte_mutex);
147139 }
@@ -154,14 +146,12 @@ static void ipte_unlock_simple(struct kvm *kvm) · 6 unmodified lines
154146 kvm->arch.ipte_lock_count--;
155147 if (kvm->arch.ipte_lock_count)
156148 goto out;
157− read_lock(&kvm->arch.sca_lock);
158− ic = kvm_s390_get_ipte_control(kvm);
149+ ic = &kvm->arch.sca->ipte_control;
159150 old = READ_ONCE(*ic);
160151 do {
161152 new = old;
162153 new.k = 0;
163154 } while (!try_cmpxchg(&ic->val, &old.val, new.val));
164− read_unlock(&kvm->arch.sca_lock);
165155 wake_up(&kvm->arch.ipte_wq);
166156 out:
167157 mutex_unlock(&kvm->arch.ipte_mutex);
@@ -172,12 +162,10 @@ static void ipte_lock_siif(struct kvm *kvm) · 4 unmodified lines
172162 union ipte_control old, new, *ic;
173163
174164 retry:
175− read_lock(&kvm->arch.sca_lock);
176− ic = kvm_s390_get_ipte_control(kvm);
165+ ic = &kvm->arch.sca->ipte_control;
177166 old = READ_ONCE(*ic);
178167 do {
179168 if (old.kg) {
180− read_unlock(&kvm->arch.sca_lock);
181169 cond_resched();
182170 goto retry;
183171 }
@@ -185,15 +173,13 @@ retry: · 1 unmodified line
185173 new.k = 1;
186174 new.kh++;
187175 } while (!try_cmpxchg(&ic->val, &old.val, new.val));
188− read_unlock(&kvm->arch.sca_lock);
189176 }
190177
191178 static void ipte_unlock_siif(struct kvm *kvm)
192179 {
193180 union ipte_control old, new, *ic;
194181
195− read_lock(&kvm->arch.sca_lock);
196− ic = kvm_s390_get_ipte_control(kvm);
182+ ic = &kvm->arch.sca->ipte_control;
197183 old = READ_ONCE(*ic);
198184 do {
199185 new = old;
@@ -201,7 +187,6 @@ static void ipte_unlock_siif(struct kvm *kvm) · 1 unmodified line
201187 if (!new.kh)
202188 new.k = 0;
203189 } while (!try_cmpxchg(&ic->val, &old.val, new.val));
204− read_unlock(&kvm->arch.sca_lock);
205190 if (!new.kh)
206191 wake_up(&kvm->arch.ipte_wq);
207192 }
arch/s390/kvm/intercept.c+3−0
@@ -471,6 +471,9 @@ static int handle_operexc(struct kvm_vcpu *vcpu) · 470 unmodified lines
471471 if (vcpu->arch.sie_block->ipa == 0xb256)
472472 return handle_sthyi(vcpu);
473473
474+ if (vcpu->kvm->arch.user_operexec)
475+ return -EOPNOTSUPP;
476+
474477 if (vcpu->arch.sie_block->ipa == 0 && vcpu->kvm->arch.user_instr0)
475478 return -EOPNOTSUPP;
476479 rc = read_guest_lc(vcpu, __LC_PGM_NEW_PSW, &newpsw, sizeof(psw_t));
arch/s390/kvm/interrupt.c+17−63
@@ -44,70 +44,34 @@ static struct kvm_s390_gib *gib; · 43 unmodified lines
4444 /* handle external calls via sigp interpretation facility */
4545 static int sca_ext_call_pending(struct kvm_vcpu *vcpu, int *src_id)
4646 {
47− int c, scn;
47+ struct esca_block *sca = vcpu->kvm->arch.sca;
48+ union esca_sigp_ctrl sigp_ctrl = sca->cpu[vcpu->vcpu_id].sigp_ctrl;
4849
4950 if (!kvm_s390_test_cpuflags(vcpu, CPUSTAT_ECALL_PEND))
5051 return 0;
5152
5253 BUG_ON(!kvm_s390_use_sca_entries());
53− read_lock(&vcpu->kvm->arch.sca_lock);
54− if (vcpu->kvm->arch.use_esca) {
55− struct esca_block *sca = vcpu->kvm->arch.sca;
56− union esca_sigp_ctrl sigp_ctrl =
57− sca->cpu[vcpu->vcpu_id].sigp_ctrl;
58−
59− c = sigp_ctrl.c;
60− scn = sigp_ctrl.scn;
61− } else {
62− struct bsca_block *sca = vcpu->kvm->arch.sca;
63− union bsca_sigp_ctrl sigp_ctrl =
64− sca->cpu[vcpu->vcpu_id].sigp_ctrl;
65−
66− c = sigp_ctrl.c;
67− scn = sigp_ctrl.scn;
68− }
69− read_unlock(&vcpu->kvm->arch.sca_lock);
7054
7155 if (src_id)
72− *src_id = scn;
56+ *src_id = sigp_ctrl.scn;
7357
74− return c;
58+ return sigp_ctrl.c;
7559 }
7660
7761 static int sca_inject_ext_call(struct kvm_vcpu *vcpu, int src_id)
7862 {
63+ struct esca_block *sca = vcpu->kvm->arch.sca;
64+ union esca_sigp_ctrl *sigp_ctrl = &sca->cpu[vcpu->vcpu_id].sigp_ctrl;
65+ union esca_sigp_ctrl old_val, new_val = {.scn = src_id, .c = 1};
7966 int expect, rc;
8067
8168 BUG_ON(!kvm_s390_use_sca_entries());
82− read_lock(&vcpu->kvm->arch.sca_lock);
83− if (vcpu->kvm->arch.use_esca) {
84− struct esca_block *sca = vcpu->kvm->arch.sca;
85− union esca_sigp_ctrl *sigp_ctrl =
86− &(sca->cpu[vcpu->vcpu_id].sigp_ctrl);
87− union esca_sigp_ctrl new_val = {0}, old_val;
88−
89− old_val = READ_ONCE(*sigp_ctrl);
90− new_val.scn = src_id;
91− new_val.c = 1;
92− old_val.c = 0;
93−
94− expect = old_val.value;
95− rc = cmpxchg(&sigp_ctrl->value, old_val.value, new_val.value);
96− } else {
97− struct bsca_block *sca = vcpu->kvm->arch.sca;
98− union bsca_sigp_ctrl *sigp_ctrl =
99− &(sca->cpu[vcpu->vcpu_id].sigp_ctrl);
100− union bsca_sigp_ctrl new_val = {0}, old_val;
10169
102− old_val = READ_ONCE(*sigp_ctrl);
103− new_val.scn = src_id;
104− new_val.c = 1;
105− old_val.c = 0;
70+ old_val = READ_ONCE(*sigp_ctrl);
71+ old_val.c = 0;
10672
107− expect = old_val.value;
108− rc = cmpxchg(&sigp_ctrl->value, old_val.value, new_val.value);
109− }
110− read_unlock(&vcpu->kvm->arch.sca_lock);
73+ expect = old_val.value;
74+ rc = cmpxchg(&sigp_ctrl->value, old_val.value, new_val.value);
11175
11276 if (rc != expect) {
11377 /* another external call is pending */
@@ -119,24 +83,14 @@ static int sca_inject_ext_call(struct kvm_vcpu *vcpu, int src_id) · 5 unmodified lines
11983
12084 static void sca_clear_ext_call(struct kvm_vcpu *vcpu)
12185 {
86+ struct esca_block *sca = vcpu->kvm->arch.sca;
87+ union esca_sigp_ctrl *sigp_ctrl = &sca->cpu[vcpu->vcpu_id].sigp_ctrl;
88+
12289 if (!kvm_s390_use_sca_entries())
12390 return;
12491 kvm_s390_clear_cpuflags(vcpu, CPUSTAT_ECALL_PEND);
125− read_lock(&vcpu->kvm->arch.sca_lock);
126− if (vcpu->kvm->arch.use_esca) {
127− struct esca_block *sca = vcpu->kvm->arch.sca;
128− union esca_sigp_ctrl *sigp_ctrl =
129− &(sca->cpu[vcpu->vcpu_id].sigp_ctrl);
130−
131− WRITE_ONCE(sigp_ctrl->value, 0);
132− } else {
133− struct bsca_block *sca = vcpu->kvm->arch.sca;
134− union bsca_sigp_ctrl *sigp_ctrl =
135− &(sca->cpu[vcpu->vcpu_id].sigp_ctrl);
13692
137− WRITE_ONCE(sigp_ctrl->value, 0);
138− }
139− read_unlock(&vcpu->kvm->arch.sca_lock);
93+ WRITE_ONCE(sigp_ctrl->value, 0);
14094 }
14195
14296 int psw_extint_disabled(struct kvm_vcpu *vcpu)
@@ -1223,7 +1177,7 @@ int kvm_s390_ext_call_pending(struct kvm_vcpu *vcpu) · 1080 unmodified lines
12231177 {
12241178 struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int;
12251179
1226− if (!sclp.has_sigpif)
1180+ if (!kvm_s390_use_sca_entries())
12271181 return test_bit(IRQ_PEND_EXT_EXTERNAL, &li->pending_irqs);
12281182
12291183 return sca_ext_call_pending(vcpu, NULL);
@@ -1548,7 +1502,7 @@ static int __inject_extcall(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq) · 318 unmodified lines
15481502 if (kvm_get_vcpu_by_id(vcpu->kvm, src_id) == NULL)
15491503 return -EINVAL;
15501504
1551− if (sclp.has_sigpif && !kvm_s390_pv_cpu_get_handle(vcpu))
1505+ if (kvm_s390_use_sca_entries() && !kvm_s390_pv_cpu_get_handle(vcpu))
15521506 return sca_inject_ext_call(vcpu, src_id);
15531507
15541508 if (test_and_set_bit(IRQ_PEND_EXT_EXTERNAL, &li->pending_irqs))
arch/s390/kvm/kvm-s390.c+64−169
@@ -13,6 +13,7 @@ · 12 unmodified lines
1313 #define pr_fmt(fmt) "kvm-s390: " fmt
1414
1515 #include <linux/compiler.h>
16+#include <linux/entry-virt.h>
1617 #include <linux/export.h>
1718 #include <linux/err.h>
1819 #include <linux/fs.h>
@@ -184,7 +185,8 @@ const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { · 165 unmodified lines
184185 STATS_DESC_COUNTER(VCPU, instruction_diagnose_308),
185186 STATS_DESC_COUNTER(VCPU, instruction_diagnose_500),
186187 STATS_DESC_COUNTER(VCPU, instruction_diagnose_other),
187− STATS_DESC_COUNTER(VCPU, pfault_sync)
188+ STATS_DESC_COUNTER(VCPU, pfault_sync),
189+ STATS_DESC_COUNTER(VCPU, signal_exits)
188190 };
189191
190192 const struct kvm_stats_header kvm_vcpu_stats_header = {
@@ -271,7 +273,6 @@ debug_info_t *kvm_s390_dbf_uv; · 80 unmodified lines
271273 /* forward declarations */
272274 static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start,
273275 unsigned long end);
274−static int sca_switch_to_extended(struct kvm *kvm);
275276
276277 static void kvm_clock_sync_scb(struct kvm_s390_sie_block *scb, u64 delta)
277278 {
@@ -606,6 +607,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · 328 unmodified lines
606607 case KVM_CAP_SET_GUEST_DEBUG:
607608 case KVM_CAP_S390_DIAG318:
608609 case KVM_CAP_IRQFD_RESAMPLE:
610+ case KVM_CAP_S390_USER_OPEREXEC:
609611 r = 1;
610612 break;
611613 case KVM_CAP_SET_GUEST_DEBUG2:
@@ -631,11 +633,13 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · 19 unmodified lines
631633 case KVM_CAP_NR_VCPUS:
632634 case KVM_CAP_MAX_VCPUS:
633635 case KVM_CAP_MAX_VCPU_ID:
634− r = KVM_S390_BSCA_CPU_SLOTS;
636+ /*
637+ * Return the same value for KVM_CAP_MAX_VCPUS and
638+ * KVM_CAP_MAX_VCPU_ID to conform with the KVM API.
639+ */
640+ r = KVM_S390_ESCA_CPU_SLOTS;
635641 if (!kvm_s390_use_sca_entries())
636642 r = KVM_MAX_VCPUS;
637− else if (sclp.has_esca && sclp.has_64bscao)
638− r = KVM_S390_ESCA_CPU_SLOTS;
639643 if (ext == KVM_CAP_NR_VCPUS)
640644 r = min_t(unsigned int, num_online_cpus(), r);
641645 break;
@@ -919,6 +923,12 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap) · 277 unmodified lines
919923 VM_EVENT(kvm, 3, "ENABLE: CAP_S390_CPU_TOPOLOGY %s",
920924 r ? "(not available)" : "(success)");
921925 break;
926+ case KVM_CAP_S390_USER_OPEREXEC:
927+ VM_EVENT(kvm, 3, "%s", "ENABLE: CAP_S390_USER_OPEREXEC");
928+ kvm->arch.user_operexec = 1;
929+ icpt_operexc_on_all_vcpus(kvm);
930+ r = 0;
931+ break;
922932 default:
923933 r = -EINVAL;
924934 break;
@@ -1930,22 +1940,18 @@ static int kvm_s390_get_cpu_model(struct kvm *kvm, struct kvm_device_attr *attr) · 1005 unmodified lines
19301940 * Updates the Multiprocessor Topology-Change-Report bit to signal
19311941 * the guest with a topology change.
19321942 * This is only relevant if the topology facility is present.
1933− *
1934− * The SCA version, bsca or esca, doesn't matter as offset is the same.
19351943 */
19361944 static void kvm_s390_update_topology_change_report(struct kvm *kvm, bool val)
19371945 {
19381946 union sca_utility new, old;
1939− struct bsca_block *sca;
1947+ struct esca_block *sca;
19401948
1941− read_lock(&kvm->arch.sca_lock);
19421949 sca = kvm->arch.sca;
19431950 old = READ_ONCE(sca->utility);
19441951 do {
19451952 new = old;
19461953 new.mtcr = val;
19471954 } while (!try_cmpxchg(&sca->utility.val, &old.val, new.val));
1948− read_unlock(&kvm->arch.sca_lock);
19491955 }
19501956
19511957 static int kvm_s390_set_topo_change_indication(struct kvm *kvm,
@@ -1966,9 +1972,7 @@ static int kvm_s390_get_topo_change_indication(struct kvm *kvm, · 14 unmodified lines
19661972 if (!test_kvm_facility(kvm, 11))
19671973 return -ENXIO;
19681974
1969− read_lock(&kvm->arch.sca_lock);
1970− topo = ((struct bsca_block *)kvm->arch.sca)->utility.mtcr;
1971− read_unlock(&kvm->arch.sca_lock);
1975+ topo = kvm->arch.sca->utility.mtcr;
19721976
19731977 return put_user(topo, (u8 __user *)attr->addr);
19741978 }
@@ -2666,14 +2670,6 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) · 691 unmodified lines
26662670 if (kvm_s390_pv_is_protected(kvm))
26672671 break;
26682672
2669− /*
2670− * FMT 4 SIE needs esca. As we never switch back to bsca from
2671− * esca, we need no cleanup in the error cases below
2672− */
2673− r = sca_switch_to_extended(kvm);
2674− if (r)
2675− break;
2676−
26772673 mmap_write_lock(kvm->mm);
26782674 r = gmap_helper_disable_cow_sharing();
26792675 mmap_write_unlock(kvm->mm);
@@ -3316,10 +3312,7 @@ static void kvm_s390_crypto_init(struct kvm *kvm) · 636 unmodified lines
33163312
33173313 static void sca_dispose(struct kvm *kvm)
33183314 {
3319− if (kvm->arch.use_esca)
3320− free_pages_exact(kvm->arch.sca, sizeof(struct esca_block));
3321− else
3322− free_page((unsigned long)(kvm->arch.sca));
3315+ free_pages_exact(kvm->arch.sca, sizeof(*kvm->arch.sca));
33233316 kvm->arch.sca = NULL;
33243317 }
33253318
@@ -3333,10 +3326,9 @@ void kvm_arch_free_vm(struct kvm *kvm) · 7 unmodified lines
33333326
33343327 int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
33353328 {
3336− gfp_t alloc_flags = GFP_KERNEL_ACCOUNT;
3337− int i, rc;
3329+ gfp_t alloc_flags = GFP_KERNEL_ACCOUNT | __GFP_ZERO;
33383330 char debug_name[16];
3339− static unsigned long sca_offset;
3331+ int i, rc;
33403332
33413333 rc = -EINVAL;
33423334 #ifdef CONFIG_KVM_S390_UCONTROL
@@ -3357,20 +3349,14 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) · 14 unmodified lines
33573349
33583350 if (!sclp.has_64bscao)
33593351 alloc_flags |= GFP_DMA;
3360− rwlock_init(&kvm->arch.sca_lock);
3361− /* start with basic SCA */
3362− kvm->arch.sca = (struct bsca_block *) get_zeroed_page(alloc_flags);
3363− if (!kvm->arch.sca)
3364− goto out_err;
33653352 mutex_lock(&kvm_lock);
3366− sca_offset += 16;
3367− if (sca_offset + sizeof(struct bsca_block) > PAGE_SIZE)
3368− sca_offset = 0;
3369− kvm->arch.sca = (struct bsca_block *)
3370− ((char *) kvm->arch.sca + sca_offset);
3353+
3354+ kvm->arch.sca = alloc_pages_exact(sizeof(*kvm->arch.sca), alloc_flags);
33713355 mutex_unlock(&kvm_lock);
3356+ if (!kvm->arch.sca)
3357+ goto out_err;
33723358
3373− sprintf(debug_name, "kvm-%u", current->pid);
3359+ snprintf(debug_name, sizeof(debug_name), "kvm-%u", current->pid);
33743360
33753361 kvm->arch.dbf = debug_register(debug_name, 32, 1, 7 * sizeof(long));
33763362 if (!kvm->arch.dbf)
@@ -3547,133 +3533,38 @@ static int __kvm_ucontrol_vcpu_init(struct kvm_vcpu *vcpu) · 170 unmodified lines
35473533
35483534 static void sca_del_vcpu(struct kvm_vcpu *vcpu)
35493535 {
3536+ struct esca_block *sca = vcpu->kvm->arch.sca;
3537+
35503538 if (!kvm_s390_use_sca_entries())
35513539 return;
3552− read_lock(&vcpu->kvm->arch.sca_lock);
3553− if (vcpu->kvm->arch.use_esca) {
3554− struct esca_block *sca = vcpu->kvm->arch.sca;
35553540
3556− clear_bit_inv(vcpu->vcpu_id, (unsigned long *) sca->mcn);
3557− sca->cpu[vcpu->vcpu_id].sda = 0;
3558− } else {
3559− struct bsca_block *sca = vcpu->kvm->arch.sca;
3560−
3561− clear_bit_inv(vcpu->vcpu_id, (unsigned long *) &sca->mcn);
3562− sca->cpu[vcpu->vcpu_id].sda = 0;
3563− }
3564− read_unlock(&vcpu->kvm->arch.sca_lock);
3541+ clear_bit_inv(vcpu->vcpu_id, (unsigned long *)sca->mcn);
3542+ sca->cpu[vcpu->vcpu_id].sda = 0;
35653543 }
35663544
35673545 static void sca_add_vcpu(struct kvm_vcpu *vcpu)
35683546 {
3569− if (!kvm_s390_use_sca_entries()) {
3570− phys_addr_t sca_phys = virt_to_phys(vcpu->kvm->arch.sca);
3571−
3572− /* we still need the basic sca for the ipte control */
3573− vcpu->arch.sie_block->scaoh = sca_phys >> 32;
3574− vcpu->arch.sie_block->scaol = sca_phys;
3575− return;
3576− }
3577− read_lock(&vcpu->kvm->arch.sca_lock);
3578− if (vcpu->kvm->arch.use_esca) {
3579− struct esca_block *sca = vcpu->kvm->arch.sca;
3580− phys_addr_t sca_phys = virt_to_phys(sca);
3581−
3582− sca->cpu[vcpu->vcpu_id].sda = virt_to_phys(vcpu->arch.sie_block);
3583− vcpu->arch.sie_block->scaoh = sca_phys >> 32;
3584− vcpu->arch.sie_block->scaol = sca_phys & ESCA_SCAOL_MASK;
3585− vcpu->arch.sie_block->ecb2 |= ECB2_ESCA;
3586− set_bit_inv(vcpu->vcpu_id, (unsigned long *) sca->mcn);
3587− } else {
3588− struct bsca_block *sca = vcpu->kvm->arch.sca;
3589− phys_addr_t sca_phys = virt_to_phys(sca);
3590−
3591− sca->cpu[vcpu->vcpu_id].sda = virt_to_phys(vcpu->arch.sie_block);
3592− vcpu->arch.sie_block->scaoh = sca_phys >> 32;
3593− vcpu->arch.sie_block->scaol = sca_phys;
3594− set_bit_inv(vcpu->vcpu_id, (unsigned long *) &sca->mcn);
3595− }
3596− read_unlock(&vcpu->kvm->arch.sca_lock);
3597−}
3598−
3599−/* Basic SCA to Extended SCA data copy routines */
3600−static inline void sca_copy_entry(struct esca_entry *d, struct bsca_entry *s)
3601−{
3602− d->sda = s->sda;
3603− d->sigp_ctrl.c = s->sigp_ctrl.c;
3604− d->sigp_ctrl.scn = s->sigp_ctrl.scn;
3605−}
3606−
3607−static void sca_copy_b_to_e(struct esca_block *d, struct bsca_block *s)
3608−{
3609− int i;
3547+ struct esca_block *sca = vcpu->kvm->arch.sca;
3548+ phys_addr_t sca_phys = virt_to_phys(sca);
36103549
3611− d->ipte_control = s->ipte_control;
3612− d->mcn[0] = s->mcn;
3613− for (i = 0; i < KVM_S390_BSCA_CPU_SLOTS; i++)
3614− sca_copy_entry(&d->cpu[i], &s->cpu[i]);
3615−}
3550+ /* we still need the sca header for the ipte control */
3551+ vcpu->arch.sie_block->scaoh = sca_phys >> 32;
3552+ vcpu->arch.sie_block->scaol = sca_phys & ESCA_SCAOL_MASK;
3553+ vcpu->arch.sie_block->ecb2 |= ECB2_ESCA;
36163554
3617−static int sca_switch_to_extended(struct kvm *kvm)
3618−{
3619− struct bsca_block *old_sca = kvm->arch.sca;
3620− struct esca_block *new_sca;
3621− struct kvm_vcpu *vcpu;
3622− unsigned long vcpu_idx;
3623− u32 scaol, scaoh;
3624− phys_addr_t new_sca_phys;
3625−
3626− if (kvm->arch.use_esca)
3627− return 0;
3628−
3629− new_sca = alloc_pages_exact(sizeof(*new_sca), GFP_KERNEL_ACCOUNT | __GFP_ZERO);
3630− if (!new_sca)
3631− return -ENOMEM;
3632−
3633− new_sca_phys = virt_to_phys(new_sca);
3634− scaoh = new_sca_phys >> 32;
3635− scaol = new_sca_phys & ESCA_SCAOL_MASK;
3636−
3637− kvm_s390_vcpu_block_all(kvm);
3638− write_lock(&kvm->arch.sca_lock);
3639−
3640− sca_copy_b_to_e(new_sca, old_sca);
3641−
3642− kvm_for_each_vcpu(vcpu_idx, vcpu, kvm) {
3643− vcpu->arch.sie_block->scaoh = scaoh;
3644− vcpu->arch.sie_block->scaol = scaol;
3645− vcpu->arch.sie_block->ecb2 |= ECB2_ESCA;
3646− }
3647− kvm->arch.sca = new_sca;
3648− kvm->arch.use_esca = 1;
3649−
3650− write_unlock(&kvm->arch.sca_lock);
3651− kvm_s390_vcpu_unblock_all(kvm);
3652−
3653− free_page((unsigned long)old_sca);
3555+ if (!kvm_s390_use_sca_entries())
3556+ return;
36543557
3655− VM_EVENT(kvm, 2, "Switched to ESCA (0x%p -> 0x%p)",
3656− old_sca, kvm->arch.sca);
3657− return 0;
3558+ set_bit_inv(vcpu->vcpu_id, (unsigned long *)sca->mcn);
3559+ sca->cpu[vcpu->vcpu_id].sda = virt_to_phys(vcpu->arch.sie_block);
36583560 }
36593561
36603562 static int sca_can_add_vcpu(struct kvm *kvm, unsigned int id)
36613563 {
3662− int rc;
3663−
3664− if (!kvm_s390_use_sca_entries()) {
3665− if (id < KVM_MAX_VCPUS)
3666− return true;
3667− return false;
3668− }
3669− if (id < KVM_S390_BSCA_CPU_SLOTS)
3670− return true;
3671− if (!sclp.has_esca || !sclp.has_64bscao)
3672− return false;
3673−
3674− rc = kvm->arch.use_esca ? 0 : sca_switch_to_extended(kvm);
3564+ if (!kvm_s390_use_sca_entries())
3565+ return id < KVM_MAX_VCPUS;
36753566
3676− return rc == 0 && id < KVM_S390_ESCA_CPU_SLOTS;
3567+ return id < KVM_S390_ESCA_CPU_SLOTS;
36773568 }
36783569
36793570 /* needs disabled preemption to protect from TOD sync and vcpu_load/put */
@@ -3919,7 +3810,7 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu) · 239 unmodified lines
39193810 vcpu->arch.sie_block->eca |= ECA_IB;
39203811 if (sclp.has_siif)
39213812 vcpu->arch.sie_block->eca |= ECA_SII;
3922− if (sclp.has_sigpif)
3813+ if (kvm_s390_use_sca_entries())
39233814 vcpu->arch.sie_block->eca |= ECA_SIGPI;
39243815 if (test_kvm_facility(vcpu->kvm, 129)) {
39253816 vcpu->arch.sie_block->eca |= ECA_VX;
@@ -4366,8 +4257,6 @@ int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu, · 440 unmodified lines
43664257
43674258 int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
43684259 {
4369− int ret = 0;
4370−
43714260 vcpu_load(vcpu);
43724261
43734262 vcpu->run->s.regs.fpc = fpu->fpc;
@@ -4378,7 +4267,7 @@ int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu) · 4 unmodified lines
43784267 memcpy(vcpu->run->s.regs.fprs, &fpu->fprs, sizeof(fpu->fprs));
43794268
43804269 vcpu_put(vcpu);
4381− return ret;
4270+ return 0;
43824271 }
43834272
43844273 int kvm_arch_vcpu_ioctl_get_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
@@ -4786,9 +4675,6 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu) · 401 unmodified lines
47864675 vcpu->arch.sie_block->gg14 = vcpu->run->s.regs.gprs[14];
47874676 vcpu->arch.sie_block->gg15 = vcpu->run->s.regs.gprs[15];
47884677
4789− if (need_resched())
4790− schedule();
4791−
47924678 if (!kvm_is_ucontrol(vcpu->kvm)) {
47934679 rc = kvm_s390_deliver_pending_interrupts(vcpu);
47944680 if (rc || guestdbg_exit_pending(vcpu))
@@ -5073,13 +4959,8 @@ int noinstr kvm_s390_enter_exit_sie(struct kvm_s390_sie_block *scb, · 278 unmodified lines
50734959 * The guest_state_{enter,exit}_irqoff() functions inform lockdep and
50744960 * tracing that entry to the guest will enable host IRQs, and exit from
50754961 * the guest will disable host IRQs.
5076− *
5077− * We must not use lockdep/tracing/RCU in this critical section, so we
5078− * use the low-level arch_local_irq_*() helpers to enable/disable IRQs.
50794962 */
5080− arch_local_irq_enable();
50814963 ret = sie64a(scb, gprs, gasce);
5082− arch_local_irq_disable();
50834964
50844965 guest_state_exit_irqoff();
50854966
@@ -5098,12 +4979,12 @@ static int __vcpu_run(struct kvm_vcpu *vcpu) · 12 unmodified lines
50984979 */
50994980 kvm_vcpu_srcu_read_lock(vcpu);
51004981
5101− do {
4982+ while (true) {
51024983 rc = vcpu_pre_run(vcpu);
4984+ kvm_vcpu_srcu_read_unlock(vcpu);
51034985 if (rc || guestdbg_exit_pending(vcpu))
51044986 break;
51054987
5106− kvm_vcpu_srcu_read_unlock(vcpu);
51074988 /*
51084989 * As PF_VCPU will be used in fault handler, between
51094990 * guest_timing_enter_irqoff and guest_timing_exit_irqoff
@@ -5115,7 +4996,17 @@ static int __vcpu_run(struct kvm_vcpu *vcpu) · 5 unmodified lines
51154996 sizeof(sie_page->pv_grregs));
51164997 }
51174998
4999+xfer_to_guest_mode_check:
51185000 local_irq_disable();
5001+ xfer_to_guest_mode_prepare();
5002+ if (xfer_to_guest_mode_work_pending()) {
5003+ local_irq_enable();
5004+ rc = kvm_xfer_to_guest_mode_handle_work(vcpu);
5005+ if (rc)
5006+ break;
5007+ goto xfer_to_guest_mode_check;
5008+ }
5009+
51195010 guest_timing_enter_irqoff();
51205011 __disable_cpu_timer_accounting(vcpu);
51215012
@@ -5145,9 +5036,12 @@ static int __vcpu_run(struct kvm_vcpu *vcpu) · 23 unmodified lines
51455036 kvm_vcpu_srcu_read_lock(vcpu);
51465037
51475038 rc = vcpu_post_run(vcpu, exit_reason);
5148− } while (!signal_pending(current) && !guestdbg_exit_pending(vcpu) && !rc);
5039+ if (rc || guestdbg_exit_pending(vcpu)) {
5040+ kvm_vcpu_srcu_read_unlock(vcpu);
5041+ break;
5042+ }
5043+ }
51495044
5150− kvm_vcpu_srcu_read_unlock(vcpu);
51515045 return rc;
51525046 }
51535047
@@ -5363,6 +5257,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) · 209 unmodified lines
53635257
53645258 if (signal_pending(current) && !rc) {
53655259 kvm_run->exit_reason = KVM_EXIT_INTR;
5260+ vcpu->stat.signal_exits++;
53665261 rc = -EINTR;
53675262 }
53685263
@@ -5729,8 +5624,8 @@ static long kvm_s390_vcpu_memsida_op(struct kvm_vcpu *vcpu, · 360 unmodified lines
57295624 return r;
57305625 }
57315626
5732−long kvm_arch_vcpu_async_ioctl(struct file *filp,
5733− unsigned int ioctl, unsigned long arg)
5627+long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
5628+ unsigned long arg)
57345629 {
57355630 struct kvm_vcpu *vcpu = filp->private_data;
57365631 void __user *argp = (void __user *)arg;
arch/s390/kvm/kvm-s390.h+1−8
@@ -570,13 +570,6 @@ void kvm_s390_prepare_debug_exit(struct kvm_vcpu *vcpu); · 569 unmodified lines
570570 int kvm_s390_handle_per_ifetch_icpt(struct kvm_vcpu *vcpu);
571571 int kvm_s390_handle_per_event(struct kvm_vcpu *vcpu);
572572
573−/* support for Basic/Extended SCA handling */
574−static inline union ipte_control *kvm_s390_get_ipte_control(struct kvm *kvm)
575−{
576− struct bsca_block *sca = kvm->arch.sca; /* SCA version doesn't matter */
577−
578− return &sca->ipte_control;
579−}
580573 static inline int kvm_s390_use_sca_entries(void)
581574 {
582575 /*
@@ -584,7 +577,7 @@ static inline int kvm_s390_use_sca_entries(void) · 1 unmodified line
584577 * might use the entries. By not setting the entries and keeping them
585578 * invalid, hardware will not access them but intercept.
586579 */
587− return sclp.has_sigpif;
580+ return sclp.has_sigpif && sclp.has_esca;
588581 }
589582 void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu,
590583 struct mcck_volatile_info *mcck_info);
arch/s390/kvm/vsie.c+14−6
@@ -782,7 +782,7 @@ static int pin_blocks(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · 781 unmodified lines
782782 else if ((gpa & ~0x1fffUL) == kvm_s390_get_prefix(vcpu))
783783 rc = set_validity_icpt(scb_s, 0x0011U);
784784 else if ((gpa & PAGE_MASK) !=
785− ((gpa + sizeof(struct bsca_block) - 1) & PAGE_MASK))
785+ ((gpa + offsetof(struct bsca_block, cpu[0]) - 1) & PAGE_MASK))
786786 rc = set_validity_icpt(scb_s, 0x003bU);
787787 if (!rc) {
788788 rc = pin_guest_page(vcpu->kvm, gpa, &hpa);
@@ -1180,12 +1180,23 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · 391 unmodified lines
11801180 current->thread.gmap_int_code = 0;
11811181 barrier();
11821182 if (!kvm_s390_vcpu_sie_inhibited(vcpu)) {
1183+xfer_to_guest_mode_check:
11831184 local_irq_disable();
1185+ xfer_to_guest_mode_prepare();
1186+ if (xfer_to_guest_mode_work_pending()) {
1187+ local_irq_enable();
1188+ rc = kvm_xfer_to_guest_mode_handle_work(vcpu);
1189+ if (rc)
1190+ goto skip_sie;
1191+ goto xfer_to_guest_mode_check;
1192+ }
11841193 guest_timing_enter_irqoff();
11851194 rc = kvm_s390_enter_exit_sie(scb_s, vcpu->run->s.regs.gprs, vsie_page->gmap->asce);
11861195 guest_timing_exit_irqoff();
11871196 local_irq_enable();
11881197 }
1198+
1199+skip_sie:
11891200 barrier();
11901201 vcpu->arch.sie_block->prog0c &= ~PROG_IN_SIE;
11911202
@@ -1345,13 +1356,11 @@ static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · 153 unmodified lines
13451356 * but rewind the PSW to re-enter SIE once that's completed
13461357 * instead of passing a "no action" intercept to the guest.
13471358 */
1348− if (signal_pending(current) ||
1349− kvm_s390_vcpu_has_irq(vcpu, 0) ||
1359+ if (kvm_s390_vcpu_has_irq(vcpu, 0) ||
13501360 kvm_s390_vcpu_sie_inhibited(vcpu)) {
13511361 kvm_s390_rewind_psw(vcpu, 4);
13521362 break;
13531363 }
1354− cond_resched();
13551364 }
13561365
13571366 if (rc == -EFAULT) {
@@ -1483,8 +1492,7 @@ int kvm_s390_handle_vsie(struct kvm_vcpu *vcpu) · 125 unmodified lines
14831492 if (unlikely(scb_addr & 0x1ffUL))
14841493 return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
14851494
1486− if (signal_pending(current) || kvm_s390_vcpu_has_irq(vcpu, 0) ||
1487− kvm_s390_vcpu_sie_inhibited(vcpu)) {
1495+ if (kvm_s390_vcpu_has_irq(vcpu, 0) || kvm_s390_vcpu_sie_inhibited(vcpu)) {
14881496 kvm_s390_rewind_psw(vcpu, 4);
14891497 return 0;
14901498 }
arch/x86/include/asm/cpufeatures.h+7−0
@@ -339,6 +339,7 @@ · 338 unmodified lines
339339 #define X86_FEATURE_AMD_STIBP (13*32+15) /* Single Thread Indirect Branch Predictors */
340340 #define X86_FEATURE_AMD_STIBP_ALWAYS_ON (13*32+17) /* Single Thread Indirect Branch Predictors always-on preferred */
341341 #define X86_FEATURE_AMD_IBRS_SAME_MODE (13*32+19) /* Indirect Branch Restricted Speculation same mode protection*/
342+#define X86_FEATURE_EFER_LMSLE_MBZ (13*32+20) /* EFER.LMSLE must be zero */
342343 #define X86_FEATURE_AMD_PPIN (13*32+23) /* "amd_ppin" Protected Processor Inventory Number */
343344 #define X86_FEATURE_AMD_SSBD (13*32+24) /* Speculative Store Bypass Disable */
344345 #define X86_FEATURE_VIRT_SSBD (13*32+25) /* "virt_ssbd" Virtualized Speculative Store Bypass Disable */
@@ -506,6 +507,12 @@ · 161 unmodified lines
506507 #define X86_FEATURE_SGX_EUPDATESVN (21*32+17) /* Support for ENCLS[EUPDATESVN] instruction */
507508
508509 #define X86_FEATURE_SDCIAE (21*32+18) /* L3 Smart Data Cache Injection Allocation Enforcement */
510+#define X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO (21*32+19) /*
511+ * Clear CPU buffers before VM-Enter if the vCPU
512+ * can access host MMIO (ignored for all intents
513+ * and purposes if CLEAR_CPU_BUF_VM is set).
514+ */
515+#define X86_FEATURE_X2AVIC_EXT (21*32+20) /* AMD SVM x2AVIC support for 4k vCPUs */
509516
510517 /*
511518 * BUG word(s)
arch/x86/include/asm/hardirq.h+2−2
@@ -5,7 +5,7 @@ · 4 unmodified lines
55 #include <linux/threads.h>
66
77 typedef struct {
8−#if IS_ENABLED(CONFIG_KVM_INTEL)
8+#if IS_ENABLED(CONFIG_CPU_MITIGATIONS) && IS_ENABLED(CONFIG_KVM_INTEL)
99 u8 kvm_cpu_l1tf_flush_l1d;
1010 #endif
1111 unsigned int __nmi_count; /* arch dependent */
@@ -68,7 +68,7 @@ extern u64 arch_irq_stat(void); · 56 unmodified lines
6868 DECLARE_PER_CPU_CACHE_HOT(u16, __softirq_pending);
6969 #define local_softirq_pending_ref __softirq_pending
7070
71−#if IS_ENABLED(CONFIG_KVM_INTEL)
71+#if IS_ENABLED(CONFIG_CPU_MITIGATIONS) && IS_ENABLED(CONFIG_KVM_INTEL)
7272 /*
7373 * This function is called from noinstr interrupt contexts
7474 * and must be inlined to not get instrumentation.
arch/x86/include/asm/kvm-x86-ops.h+1−0
@@ -128,6 +128,7 @@ KVM_X86_OP(enable_smi_window) · 127 unmodified lines
128128 KVM_X86_OP_OPTIONAL(dev_get_attr)
129129 KVM_X86_OP_OPTIONAL(mem_enc_ioctl)
130130 KVM_X86_OP_OPTIONAL(vcpu_mem_enc_ioctl)
131+KVM_X86_OP_OPTIONAL(vcpu_mem_enc_unlocked_ioctl)
131132 KVM_X86_OP_OPTIONAL(mem_enc_register_region)
132133 KVM_X86_OP_OPTIONAL(mem_enc_unregister_region)
133134 KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from)
arch/x86/include/asm/kvm_host.h+14−9
@@ -1055,9 +1055,6 @@ struct kvm_vcpu_arch { · 1054 unmodified lines
10551055 /* be preempted when it's in kernel-mode(cpl=0) */
10561056 bool preempted_in_kernel;
10571057
1058− /* Flush the L1 Data cache for L1TF mitigation on VMENTER */
1059− bool l1tf_flush_l1d;
1060−
10611058 /* Host CPU on which VM-entry was most recently attempted */
10621059 int last_vmentry_cpu;
10631060
@@ -1456,8 +1453,6 @@ struct kvm_arch { · 392 unmodified lines
14561453 bool use_master_clock;
14571454 u64 master_kernel_ns;
14581455 u64 master_cycle_now;
1459− struct delayed_work kvmclock_update_work;
1460− struct delayed_work kvmclock_sync_work;
14611456
14621457 #ifdef CONFIG_KVM_HYPERV
14631458 struct kvm_hv hyperv;
@@ -1848,15 +1843,15 @@ struct kvm_x86_ops { · 384 unmodified lines
18481843 void *external_spt);
18491844 /* Update the external page table from spte getting set. */
18501845 int (*set_external_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
1851− kvm_pfn_t pfn_for_gfn);
1846+ u64 mirror_spte);
18521847
18531848 /* Update external page tables for page table about to be freed. */
18541849 int (*free_external_spt)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
18551850 void *external_spt);
18561851
18571852 /* Update external page table from spte getting removed, and flush TLB. */
1858− int (*remove_external_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
1859− kvm_pfn_t pfn_for_gfn);
1853+ void (*remove_external_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
1854+ u64 mirror_spte);
18601855
18611856 bool (*has_wbinvd_exit)(void);
18621857
@@ -1914,6 +1909,7 @@ struct kvm_x86_ops { · 51 unmodified lines
19141909 int (*dev_get_attr)(u32 group, u64 attr, u64 *val);
19151910 int (*mem_enc_ioctl)(struct kvm *kvm, void __user *argp);
19161911 int (*vcpu_mem_enc_ioctl)(struct kvm_vcpu *vcpu, void __user *argp);
1912+ int (*vcpu_mem_enc_unlocked_ioctl)(struct kvm_vcpu *vcpu, void __user *argp);
19171913 int (*mem_enc_register_region)(struct kvm *kvm, struct kvm_enc_region *argp);
19181914 int (*mem_enc_unregister_region)(struct kvm *kvm, struct kvm_enc_region *argp);
19191915 int (*vm_copy_enc_context_from)(struct kvm *kvm, unsigned int source_fd);
@@ -2143,6 +2139,11 @@ u64 vcpu_tsc_khz(struct kvm_vcpu *vcpu); · 223 unmodified lines
21432139 * the gfn, i.e. retrying the instruction will hit a
21442140 * !PRESENT fault, which results in a new shadow page
21452141 * and sends KVM back to square one.
2142+ *
2143+ * EMULTYPE_SKIP_SOFT_INT - Set in combination with EMULTYPE_SKIP to only skip
2144+ * an instruction if it could generate a given software
2145+ * interrupt, which must be encoded via
2146+ * EMULTYPE_SET_SOFT_INT_VECTOR().
21462147 */
21472148 #define EMULTYPE_NO_DECODE (1 << 0)
21482149 #define EMULTYPE_TRAP_UD (1 << 1)
@@ -2153,6 +2154,10 @@ u64 vcpu_tsc_khz(struct kvm_vcpu *vcpu); · 4 unmodified lines
21532154 #define EMULTYPE_PF (1 << 6)
21542155 #define EMULTYPE_COMPLETE_USER_EXIT (1 << 7)
21552156 #define EMULTYPE_WRITE_PF_TO_SP (1 << 8)
2157+#define EMULTYPE_SKIP_SOFT_INT (1 << 9)
2158+
2159+#define EMULTYPE_SET_SOFT_INT_VECTOR(v) ((u32)((v) & 0xff) << 16)
2160+#define EMULTYPE_GET_SOFT_INT_VECTOR(e) (((e) >> 16) & 0xff)
21562161
21572162 static inline bool kvm_can_emulate_event_vectoring(int emul_type)
21582163 {
@@ -2167,6 +2172,7 @@ void __kvm_prepare_emulation_failure_exit(struct kvm_vcpu *vcpu, · 8 unmodified lines
21672172 void kvm_prepare_emulation_failure_exit(struct kvm_vcpu *vcpu);
21682173
21692174 void kvm_prepare_event_vectoring_exit(struct kvm_vcpu *vcpu, gpa_t gpa);
2175+void kvm_prepare_unexpected_reason_exit(struct kvm_vcpu *vcpu, u64 exit_reason);
21702176
21712177 void kvm_enable_efer_bits(u64);
21722178 bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer);
@@ -2378,7 +2384,6 @@ int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ipi_bitmap_low, · 205 unmodified lines
23782384 int kvm_add_user_return_msr(u32 msr);
23792385 int kvm_find_user_return_msr(u32 msr);
23802386 int kvm_set_user_return_msr(unsigned index, u64 val, u64 mask);
2381−void kvm_user_return_msr_update_cache(unsigned int index, u64 val);
23822387 u64 kvm_get_user_return_msr(unsigned int slot);
23832388
23842389 static inline bool kvm_is_supported_user_return_msr(u32 msr)
arch/x86/include/asm/nospec-branch.h+15−15
@@ -308,24 +308,26 @@ · 307 unmodified lines
308308 * CFLAGS.ZF.
309309 * Note: Only the memory operand variant of VERW clears the CPU buffers.
310310 */
311−.macro __CLEAR_CPU_BUFFERS feature
312311 #ifdef CONFIG_X86_64
313− ALTERNATIVE "", "verw x86_verw_sel(%rip)", \feature
312+#define VERW verw x86_verw_sel(%rip)
314313 #else
315− /*
316− * In 32bit mode, the memory operand must be a %cs reference. The data
317− * segments may not be usable (vm86 mode), and the stack segment may not
318− * be flat (ESPFIX32).
319− */
320− ALTERNATIVE "", "verw %cs:x86_verw_sel", \feature
314+/*
315+ * In 32bit mode, the memory operand must be a %cs reference. The data segments
316+ * may not be usable (vm86 mode), and the stack segment may not be flat (ESPFIX32).
317+ */
318+#define VERW verw %cs:x86_verw_sel
321319 #endif
322−.endm
323320
324−#define CLEAR_CPU_BUFFERS \
325− __CLEAR_CPU_BUFFERS X86_FEATURE_CLEAR_CPU_BUF
321+/*
322+ * Provide a stringified VERW macro for simple usage, and a non-stringified
323+ * VERW macro for use in more elaborate sequences, e.g. to encode a conditional
324+ * VERW within an ALTERNATIVE.
325+ */
326+#define __CLEAR_CPU_BUFFERS __stringify(VERW)
326327
327−#define VM_CLEAR_CPU_BUFFERS \
328− __CLEAR_CPU_BUFFERS X86_FEATURE_CLEAR_CPU_BUF_VM
328+/* If necessary, emit VERW on exit-to-userspace to clear CPU buffers. */
329+#define CLEAR_CPU_BUFFERS \
330+ ALTERNATIVE "", __CLEAR_CPU_BUFFERS, X86_FEATURE_CLEAR_CPU_BUF
329331
330332 #ifdef CONFIG_X86_64
331333 .macro CLEAR_BRANCH_HISTORY
@@ -580,8 +582,6 @@ DECLARE_STATIC_KEY_FALSE(cpu_buf_idle_clear); · 248 unmodified lines
580582
581583 DECLARE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush);
582584
583−DECLARE_STATIC_KEY_FALSE(cpu_buf_vm_clear);
584−
585585 extern u16 x86_verw_sel;
586586
587587 #include <asm/segment.h>
arch/x86/include/asm/svm.h+4−1
@@ -279,7 +279,7 @@ enum avic_ipi_failure_cause { · 278 unmodified lines
279279 AVIC_IPI_FAILURE_INVALID_IPI_VECTOR,
280280 };
281281
282−#define AVIC_PHYSICAL_MAX_INDEX_MASK GENMASK_ULL(8, 0)
282+#define AVIC_PHYSICAL_MAX_INDEX_MASK GENMASK_ULL(11, 0)
283283
284284 /*
285285 * For AVIC, the max index allowed for physical APIC ID table is 0xfe (254), as
@@ -289,11 +289,14 @@ enum avic_ipi_failure_cause { · 3 unmodified lines
289289
290290 /*
291291 * For x2AVIC, the max index allowed for physical APIC ID table is 0x1ff (511).
292+ * With X86_FEATURE_X2AVIC_EXT, the max index is increased to 0xfff (4095).
292293 */
293294 #define X2AVIC_MAX_PHYSICAL_ID 0x1FFUL
295+#define X2AVIC_4K_MAX_PHYSICAL_ID 0xFFFUL
294296
295297 static_assert((AVIC_MAX_PHYSICAL_ID & AVIC_PHYSICAL_MAX_INDEX_MASK) == AVIC_MAX_PHYSICAL_ID);
296298 static_assert((X2AVIC_MAX_PHYSICAL_ID & AVIC_PHYSICAL_MAX_INDEX_MASK) == X2AVIC_MAX_PHYSICAL_ID);
299+static_assert((X2AVIC_4K_MAX_PHYSICAL_ID & AVIC_PHYSICAL_MAX_INDEX_MASK) == X2AVIC_4K_MAX_PHYSICAL_ID);
297300
298301 #define SVM_SEV_FEAT_SNP_ACTIVE BIT(0)
299302 #define SVM_SEV_FEAT_RESTRICTED_INJECTION BIT(3)
arch/x86/include/uapi/asm/kvm.h+1−0
@@ -502,6 +502,7 @@ struct kvm_sync_regs { · 501 unmodified lines
502502 /* vendor-specific groups and attributes for system fd */
503503 #define KVM_X86_GRP_SEV 1
504504 # define KVM_X86_SEV_VMSA_FEATURES 0
505+# define KVM_X86_SNP_POLICY_BITS 1
505506
506507 struct kvm_vmx_nested_state_data {
507508 __u8 vmcs12[KVM_STATE_NESTED_VMX_VMCS_SIZE];
arch/x86/kernel/cpu/bugs.c+9−13
@@ -145,14 +145,6 @@ EXPORT_SYMBOL_GPL(cpu_buf_idle_clear); · 144 unmodified lines
145145 */
146146 DEFINE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush);
147147
148−/*
149− * Controls CPU Fill buffer clear before VMenter. This is a subset of
150− * X86_FEATURE_CLEAR_CPU_BUF, and should only be enabled when KVM-only
151− * mitigation is required.
152− */
153−DEFINE_STATIC_KEY_FALSE(cpu_buf_vm_clear);
154−EXPORT_SYMBOL_FOR_KVM(cpu_buf_vm_clear);
155−
156148 #undef pr_fmt
157149 #define pr_fmt(fmt) "mitigations: " fmt
158150
@@ -349,8 +341,8 @@ static enum rfds_mitigations rfds_mitigation __ro_after_init = · 190 unmodified lines
349341 IS_ENABLED(CONFIG_MITIGATION_RFDS) ? RFDS_MITIGATION_AUTO : RFDS_MITIGATION_OFF;
350342
351343 /*
352− * Set if any of MDS/TAA/MMIO/RFDS are going to enable VERW clearing
353− * through X86_FEATURE_CLEAR_CPU_BUF on kernel and guest entry.
344+ * Set if any of MDS/TAA/MMIO/RFDS are going to enable VERW clearing on exit to
345+ * userspace *and* on entry to KVM guests.
354346 */
355347 static bool verw_clear_cpu_buf_mitigation_selected __ro_after_init;
356348
@@ -396,6 +388,7 @@ static void __init mds_apply_mitigation(void) · 39 unmodified lines
396388 if (mds_mitigation == MDS_MITIGATION_FULL ||
397389 mds_mitigation == MDS_MITIGATION_VMWERV) {
398390 setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF);
391+ setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM);
399392 if (!boot_cpu_has(X86_BUG_MSBDS_ONLY) &&
400393 (mds_nosmt || smt_mitigations == SMT_MITIGATIONS_ON))
401394 cpu_smt_disable(false);
@@ -507,6 +500,7 @@ static void __init taa_apply_mitigation(void) · 105 unmodified lines
507500 * present on host, enable the mitigation for UCODE_NEEDED as well.
508501 */
509502 setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF);
503+ setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM);
510504
511505 if (taa_nosmt || smt_mitigations == SMT_MITIGATIONS_ON)
512506 cpu_smt_disable(false);
@@ -608,9 +602,9 @@ static void __init mmio_apply_mitigation(void) · 95 unmodified lines
608602 */
609603 if (verw_clear_cpu_buf_mitigation_selected) {
610604 setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF);
611− static_branch_disable(&cpu_buf_vm_clear);
605+ setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM);
612606 } else {
613− static_branch_enable(&cpu_buf_vm_clear);
607+ setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO);
614608 }
615609
616610 /*
@@ -699,8 +693,10 @@ static void __init rfds_update_mitigation(void) · 82 unmodified lines
699693
700694 static void __init rfds_apply_mitigation(void)
701695 {
702− if (rfds_mitigation == RFDS_MITIGATION_VERW)
696+ if (rfds_mitigation == RFDS_MITIGATION_VERW) {
703697 setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF);
698+ setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM);
699+ }
704700 }
705701
706702 static __init int rfds_parse_cmdline(char *str)
arch/x86/kernel/cpu/scattered.c+1−0
@@ -53,6 +53,7 @@ static const struct cpuid_bit cpuid_bits[] = { · 52 unmodified lines
5353 { X86_FEATURE_PROC_FEEDBACK, CPUID_EDX, 11, 0x80000007, 0 },
5454 { X86_FEATURE_AMD_FAST_CPPC, CPUID_EDX, 15, 0x80000007, 0 },
5555 { X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 },
56+ { X86_FEATURE_X2AVIC_EXT, CPUID_ECX, 6, 0x8000000a, 0 },
5657 { X86_FEATURE_COHERENCY_SFW_NO, CPUID_EBX, 31, 0x8000001f, 0 },
5758 { X86_FEATURE_SMBA, CPUID_EBX, 2, 0x80000020, 0 },
5859 { X86_FEATURE_BMEC, CPUID_EBX, 3, 0x80000020, 0 },
arch/x86/kvm/cpuid.c+1−0
@@ -1135,6 +1135,7 @@ void kvm_set_cpu_caps(void) · 1134 unmodified lines
11351135 F(AMD_STIBP),
11361136 F(AMD_STIBP_ALWAYS_ON),
11371137 F(AMD_IBRS_SAME_MODE),
1138+ PASSTHROUGH_F(EFER_LMSLE_MBZ),
11381139 F(AMD_PSFD),
11391140 F(AMD_IBPB_RET),
11401141 );
arch/x86/kvm/emulate.c+223−96
@@ -81,9 +81,8 @@ · 80 unmodified lines
8181 */
8282
8383 /* Operand sizes: 8-bit operands or specified/overridden size. */
84−#define ByteOp (1<<0) /* 8-bit operands. */
85−/* Destination operand type. */
86−#define DstShift 1
84+#define ByteOp (1<<0) /* 8-bit operands. */
85+#define DstShift 1 /* Destination operand type at bits 1-5 */
8786 #define ImplicitOps (OpImplicit << DstShift)
8887 #define DstReg (OpReg << DstShift)
8988 #define DstMem (OpMem << DstShift)
@@ -95,8 +94,7 @@ · 5 unmodified lines
9594 #define DstDX (OpDX << DstShift)
9695 #define DstAccLo (OpAccLo << DstShift)
9796 #define DstMask (OpMask << DstShift)
98−/* Source operand type. */
99−#define SrcShift 6
97+#define SrcShift 6 /* Source operand type at bits 6-10 */
10098 #define SrcNone (OpNone << SrcShift)
10199 #define SrcReg (OpReg << SrcShift)
102100 #define SrcMem (OpMem << SrcShift)
@@ -119,10 +117,10 @@ · 16 unmodified lines
119117 #define SrcAccHi (OpAccHi << SrcShift)
120118 #define SrcMask (OpMask << SrcShift)
121119 #define BitOp (1<<11)
122−#define MemAbs (1<<12) /* Memory operand is absolute displacement */
120+#define MemAbs (1<<12) /* Memory operand is absolute displacement */
123121 #define String (1<<13) /* String instruction (rep capable) */
124122 #define Stack (1<<14) /* Stack instruction (push/pop) */
125−#define GroupMask (7<<15) /* Opcode uses one of the group mechanisms */
123+#define GroupMask (7<<15) /* Group mechanisms, at bits 15-17 */
126124 #define Group (1<<15) /* Bits 3:5 of modrm byte extend opcode */
127125 #define GroupDual (2<<15) /* Alternate decoding of mod == 3 */
128126 #define Prefix (3<<15) /* Instruction varies with 66/f2/f3 prefix */
@@ -131,11 +129,8 @@ · 2 unmodified lines
131129 #define InstrDual (6<<15) /* Alternate instruction decoding of mod == 3 */
132130 #define ModeDual (7<<15) /* Different instruction for 32/64 bit */
133131 #define Sse (1<<18) /* SSE Vector instruction */
134−/* Generic ModRM decode. */
135−#define ModRM (1<<19)
136−/* Destination is only written; never read. */
137−#define Mov (1<<20)
138−/* Misc flags */
132+#define ModRM (1<<19) /* Generic ModRM decode. */
133+#define Mov (1<<20) /* Destination is only written; never read. */
139134 #define Prot (1<<21) /* instruction generates #UD if not in prot-mode */
140135 #define EmulateOnUD (1<<22) /* Emulate if unsupported by the host */
141136 #define NoAccess (1<<23) /* Don't access memory (lea/invlpg/verr etc) */
@@ -143,11 +138,11 @@ · 1 unmodified line
143138 #define Undefined (1<<25) /* No Such Instruction */
144139 #define Lock (1<<26) /* lock prefix is allowed for the instruction */
145140 #define Priv (1<<27) /* instruction generates #GP if current CPL != 0 */
146−#define No64 (1<<28)
141+#define No64 (1<<28) /* Instruction generates #UD in 64-bit mode */
147142 #define PageTable (1 << 29) /* instruction used to write page table */
148143 #define NotImpl (1 << 30) /* instruction is not implemented */
149−/* Source 2 operand type */
150−#define Src2Shift (31)
144+#define Avx ((u64)1 << 31) /* Instruction uses VEX prefix */
145+#define Src2Shift (32) /* Source 2 operand type at bits 32-36 */
151146 #define Src2None (OpNone << Src2Shift)
152147 #define Src2Mem (OpMem << Src2Shift)
153148 #define Src2CL (OpCL << Src2Shift)
@@ -161,12 +156,13 @@ · 7 unmodified lines
161156 #define Src2FS (OpFS << Src2Shift)
162157 #define Src2GS (OpGS << Src2Shift)
163158 #define Src2Mask (OpMask << Src2Shift)
159+/* free: 37-39 */
164160 #define Mmx ((u64)1 << 40) /* MMX Vector instruction */
165−#define AlignMask ((u64)7 << 41)
161+#define AlignMask ((u64)3 << 41) /* Memory alignment requirement at bits 41-42 */
166162 #define Aligned ((u64)1 << 41) /* Explicitly aligned (e.g. MOVDQA) */
167163 #define Unaligned ((u64)2 << 41) /* Explicitly unaligned (e.g. MOVDQU) */
168−#define Avx ((u64)3 << 41) /* Advanced Vector Extensions */
169−#define Aligned16 ((u64)4 << 41) /* Aligned to 16 byte boundary (e.g. FXSAVE) */
164+#define Aligned16 ((u64)3 << 41) /* Aligned to 16 byte boundary (e.g. FXSAVE) */
165+/* free: 43-44 */
170166 #define NoWrite ((u64)1 << 45) /* No writeback */
171167 #define SrcWrite ((u64)1 << 46) /* Write back src operand */
172168 #define NoMod ((u64)1 << 47) /* Mod field is ignored */
@@ -243,6 +239,13 @@ enum x86_transfer_type { · 70 unmodified lines
243239 X86_TRANSFER_TASK_SWITCH,
244240 };
245241
242+enum rex_bits {
243+ REX_B = 1,
244+ REX_X = 2,
245+ REX_R = 4,
246+ REX_W = 8,
247+};
248+
246249 static void writeback_registers(struct x86_emulate_ctxt *ctxt)
247250 {
248251 unsigned long dirty = ctxt->regs_dirty;
@@ -622,7 +625,6 @@ static unsigned insn_alignment(struct x86_emulate_ctxt *ctxt, unsigned size) · 373 unmodified lines
622625
623626 switch (alignment) {
624627 case Unaligned:
625− case Avx:
626628 return 1;
627629 case Aligned16:
628630 return 16;
@@ -924,7 +926,7 @@ static void *decode_register(struct x86_emulate_ctxt *ctxt, u8 modrm_reg, · 295 unmodified lines
924926 int byteop)
925927 {
926928 void *p;
927− int highbyte_regs = (ctxt->rex_prefix == 0) && byteop;
929+ int highbyte_regs = (ctxt->rex_prefix == REX_NONE) && byteop;
928930
929931 if (highbyte_regs && modrm_reg >= 4 && modrm_reg < 8)
930932 p = (unsigned char *)reg_rmw(ctxt, modrm_reg & 3) + 1;
@@ -1030,6 +1032,7 @@ static void fetch_register_operand(struct operand *op) · 99 unmodified lines
10301032 op->val = *(u64 *)op->addr.reg;
10311033 break;
10321034 }
1035+ op->orig_val = op->val;
10331036 }
10341037
10351038 static int em_fninit(struct x86_emulate_ctxt *ctxt)
@@ -1075,17 +1078,17 @@ static int em_fnstsw(struct x86_emulate_ctxt *ctxt) · 39 unmodified lines
10751078 return X86EMUL_CONTINUE;
10761079 }
10771080
1078−static void decode_register_operand(struct x86_emulate_ctxt *ctxt,
1079− struct operand *op)
1081+static void __decode_register_operand(struct x86_emulate_ctxt *ctxt,
1082+ struct operand *op, int reg)
10801083 {
1081− unsigned int reg;
1082−
1083− if (ctxt->d & ModRM)
1084− reg = ctxt->modrm_reg;
1085− else
1086− reg = (ctxt->b & 7) | ((ctxt->rex_prefix & 1) << 3);
1087−
1088− if (ctxt->d & Sse) {
1084+ if ((ctxt->d & Avx) && ctxt->op_bytes == 32) {
1085+ op->type = OP_YMM;
1086+ op->bytes = 32;
1087+ op->addr.xmm = reg;
1088+ kvm_read_avx_reg(reg, &op->vec_val2);
1089+ return;
1090+ }
1091+ if (ctxt->d & (Avx|Sse)) {
10891092 op->type = OP_XMM;
10901093 op->bytes = 16;
10911094 op->addr.xmm = reg;
@@ -1103,9 +1106,20 @@ static void decode_register_operand(struct x86_emulate_ctxt *ctxt, · 11 unmodified lines
11031106 op->type = OP_REG;
11041107 op->bytes = (ctxt->d & ByteOp) ? 1 : ctxt->op_bytes;
11051108 op->addr.reg = decode_register(ctxt, reg, ctxt->d & ByteOp);
1106−
11071109 fetch_register_operand(op);
1108− op->orig_val = op->val;
1110+}
1111+
1112+static void decode_register_operand(struct x86_emulate_ctxt *ctxt,
1113+ struct operand *op)
1114+{
1115+ unsigned int reg;
1116+
1117+ if (ctxt->d & ModRM)
1118+ reg = ctxt->modrm_reg;
1119+ else
1120+ reg = (ctxt->b & 7) | (ctxt->rex_bits & REX_B ? 8 : 0);
1121+
1122+ __decode_register_operand(ctxt, op, reg);
11091123 }
11101124
11111125 static void adjust_modrm_seg(struct x86_emulate_ctxt *ctxt, int base_reg)
@@ -1122,9 +1136,9 @@ static int decode_modrm(struct x86_emulate_ctxt *ctxt, · 10 unmodified lines
11221136 int rc = X86EMUL_CONTINUE;
11231137 ulong modrm_ea = 0;
11241138
1125− ctxt->modrm_reg = ((ctxt->rex_prefix << 1) & 8); /* REX.R */
1126− index_reg = (ctxt->rex_prefix << 2) & 8; /* REX.X */
1127− base_reg = (ctxt->rex_prefix << 3) & 8; /* REX.B */
1139+ ctxt->modrm_reg = (ctxt->rex_bits & REX_R ? 8 : 0);
1140+ index_reg = (ctxt->rex_bits & REX_X ? 8 : 0);
1141+ base_reg = (ctxt->rex_bits & REX_B ? 8 : 0);
11281142
11291143 ctxt->modrm_mod = (ctxt->modrm & 0xc0) >> 6;
11301144 ctxt->modrm_reg |= (ctxt->modrm & 0x38) >> 3;
@@ -1132,24 +1146,7 @@ static int decode_modrm(struct x86_emulate_ctxt *ctxt, · 1 unmodified line
11321146 ctxt->modrm_seg = VCPU_SREG_DS;
11331147
11341148 if (ctxt->modrm_mod == 3 || (ctxt->d & NoMod)) {
1135− op->type = OP_REG;
1136− op->bytes = (ctxt->d & ByteOp) ? 1 : ctxt->op_bytes;
1137− op->addr.reg = decode_register(ctxt, ctxt->modrm_rm,
1138− ctxt->d & ByteOp);
1139− if (ctxt->d & Sse) {
1140− op->type = OP_XMM;
1141− op->bytes = 16;
1142− op->addr.xmm = ctxt->modrm_rm;
1143− kvm_read_sse_reg(ctxt->modrm_rm, &op->vec_val);
1144− return rc;
1145− }
1146− if (ctxt->d & Mmx) {
1147− op->type = OP_MM;
1148− op->bytes = 8;
1149− op->addr.mm = ctxt->modrm_rm & 7;
1150− return rc;
1151− }
1152− fetch_register_operand(op);
1149+ __decode_register_operand(ctxt, op, ctxt->modrm_rm);
11531150 return rc;
11541151 }
11551152
@@ -1783,7 +1780,15 @@ static int writeback(struct x86_emulate_ctxt *ctxt, struct operand *op) · 627 unmodified lines
17831780 op->data,
17841781 op->bytes * op->count);
17851782 case OP_XMM:
1786− kvm_write_sse_reg(op->addr.xmm, &op->vec_val);
1783+ if (!(ctxt->d & Avx)) {
1784+ kvm_write_sse_reg(op->addr.xmm, &op->vec_val);
1785+ break;
1786+ }
1787+ /* full YMM write but with high bytes cleared */
1788+ memset(op->valptr + 16, 0, 16);
1789+ fallthrough;
1790+ case OP_YMM:
1791+ kvm_write_avx_reg(op->addr.xmm, &op->vec_val2);
17871792 break;
17881793 case OP_MM:
17891794 kvm_write_mmx_reg(op->addr.mm, &op->mm_val);
@@ -2466,7 +2471,7 @@ static int em_sysexit(struct x86_emulate_ctxt *ctxt) · 676 unmodified lines
24662471
24672472 setup_syscalls_segments(&cs, &ss);
24682473
2469− if ((ctxt->rex_prefix & 0x8) != 0x0)
2474+ if (ctxt->rex_bits & REX_W)
24702475 usermode = X86EMUL_MODE_PROT64;
24712476 else
24722477 usermode = X86EMUL_MODE_PROT32;
@@ -3958,6 +3963,8 @@ static int check_perm_out(struct x86_emulate_ctxt *ctxt) · 1485 unmodified lines
39583963 I2bv(((_f) | DstReg | SrcMem | ModRM) & ~Lock, _e), \
39593964 I2bv(((_f) & ~Lock) | DstAcc | SrcImm, _e)
39603965
3966+static const struct opcode ud = I(SrcNone, emulate_ud);
3967+
39613968 static const struct opcode group7_rm0[] = {
39623969 N,
39633970 I(SrcNone | Priv | EmulateOnUD, em_hypercall),
@@ -4114,7 +4121,7 @@ static const struct group_dual group15 = { { · 150 unmodified lines
41144121 } };
41154122
41164123 static const struct gprefix pfx_0f_6f_0f_7f = {
4117− I(Mmx, em_mov), I(Sse | Aligned, em_mov), N, I(Sse | Unaligned, em_mov),
4124+ I(Mmx, em_mov), I(Sse | Avx | Aligned, em_mov), N, I(Sse | Avx | Unaligned, em_mov),
41184125 };
41194126
41204127 static const struct instr_dual instr_dual_0f_2b = {
@@ -4133,8 +4140,8 @@ static const struct gprefix pfx_0f_28_0f_29 = { · 12 unmodified lines
41334140 I(Aligned, em_mov), I(Aligned, em_mov), N, N,
41344141 };
41354142
4136−static const struct gprefix pfx_0f_e7 = {
4137− N, I(Sse, em_mov), N, N,
4143+static const struct gprefix pfx_0f_e7_0f_38_2a = {
4144+ N, I(Sse | Avx, em_mov), N, N,
41384145 };
41394146
41404147 static const struct escape escape_d9 = { {
@@ -4347,8 +4354,8 @@ static const struct opcode twobyte_table[256] = { · 206 unmodified lines
43474354 DI(ImplicitOps | Priv, invd), DI(ImplicitOps | Priv, wbinvd), N, N,
43484355 N, D(ImplicitOps | ModRM | SrcMem | NoAccess), N, N,
43494356 /* 0x10 - 0x1F */
4350− GP(ModRM | DstReg | SrcMem | Mov | Sse, &pfx_0f_10_0f_11),
4351− GP(ModRM | DstMem | SrcReg | Mov | Sse, &pfx_0f_10_0f_11),
4357+ GP(ModRM | DstReg | SrcMem | Mov | Sse | Avx, &pfx_0f_10_0f_11),
4358+ GP(ModRM | DstMem | SrcReg | Mov | Sse | Avx, &pfx_0f_10_0f_11),
43524359 N, N, N, N, N, N,
43534360 D(ImplicitOps | ModRM | SrcMem | NoAccess), /* 4 * prefetch + 4 * reserved NOP */
43544361 D(ImplicitOps | ModRM | SrcMem | NoAccess), N, N,
@@ -4364,9 +4371,9 @@ static const struct opcode twobyte_table[256] = { · 9 unmodified lines
43644371 IIP(ModRM | SrcMem | Priv | Op3264 | NoMod, em_dr_write, dr_write,
43654372 check_dr_write),
43664373 N, N, N, N,
4367− GP(ModRM | DstReg | SrcMem | Mov | Sse, &pfx_0f_28_0f_29),
4368− GP(ModRM | DstMem | SrcReg | Mov | Sse, &pfx_0f_28_0f_29),
4369− N, GP(ModRM | DstMem | SrcReg | Mov | Sse, &pfx_0f_2b),
4374+ GP(ModRM | DstReg | SrcMem | Mov | Sse | Avx, &pfx_0f_28_0f_29),
4375+ GP(ModRM | DstMem | SrcReg | Mov | Sse | Avx, &pfx_0f_28_0f_29),
4376+ N, GP(ModRM | DstMem | SrcReg | Mov | Sse | Avx, &pfx_0f_2b),
43704377 N, N, N, N,
43714378 /* 0x30 - 0x3F */
43724379 II(ImplicitOps | Priv, em_wrmsr, wrmsr),
@@ -4431,7 +4438,7 @@ static const struct opcode twobyte_table[256] = { · 58 unmodified lines
44314438 /* 0xD0 - 0xDF */
44324439 N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N,
44334440 /* 0xE0 - 0xEF */
4434− N, N, N, N, N, N, N, GP(SrcReg | DstMem | ModRM | Mov, &pfx_0f_e7),
4441+ N, N, N, N, N, N, N, GP(SrcReg | DstMem | ModRM | Mov, &pfx_0f_e7_0f_38_2a),
44354442 N, N, N, N, N, N, N, N,
44364443 /* 0xF0 - 0xFF */
44374444 N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N
@@ -4458,8 +4465,13 @@ static const struct gprefix three_byte_0f_38_f1 = { · 20 unmodified lines
44584465 * byte.
44594466 */
44604467 static const struct opcode opcode_map_0f_38[256] = {
4461− /* 0x00 - 0x7f */
4462− X16(N), X16(N), X16(N), X16(N), X16(N), X16(N), X16(N), X16(N),
4468+ /* 0x00 - 0x1f */
4469+ X16(N), X16(N),
4470+ /* 0x20 - 0x2f */
4471+ X8(N),
4472+ X2(N), GP(SrcReg | DstMem | ModRM | Mov | Aligned, &pfx_0f_e7_0f_38_2a), N, N, N, N, N,
4473+ /* 0x30 - 0x7f */
4474+ X16(N), X16(N), X16(N), X16(N), X16(N),
44634475 /* 0x80 - 0xef */
44644476 X16(N), X16(N), X16(N), X16(N), X16(N), X16(N), X16(N),
44654477 /* 0xf0 - 0xf1 */
@@ -4618,14 +4630,12 @@ static int decode_operand(struct x86_emulate_ctxt *ctxt, struct operand *op, · 152 unmodified lines
46184630 op->bytes = (ctxt->d & ByteOp) ? 1 : ctxt->op_bytes;
46194631 op->addr.reg = reg_rmw(ctxt, VCPU_REGS_RAX);
46204632 fetch_register_operand(op);
4621− op->orig_val = op->val;
46224633 break;
46234634 case OpAccLo:
46244635 op->type = OP_REG;
46254636 op->bytes = (ctxt->d & ByteOp) ? 2 : ctxt->op_bytes;
46264637 op->addr.reg = reg_rmw(ctxt, VCPU_REGS_RAX);
46274638 fetch_register_operand(op);
4628− op->orig_val = op->val;
46294639 break;
46304640 case OpAccHi:
46314641 if (ctxt->d & ByteOp) {
@@ -4636,7 +4646,6 @@ static int decode_operand(struct x86_emulate_ctxt *ctxt, struct operand *op, · 4 unmodified lines
46364646 op->bytes = ctxt->op_bytes;
46374647 op->addr.reg = reg_rmw(ctxt, VCPU_REGS_RDX);
46384648 fetch_register_operand(op);
4639− op->orig_val = op->val;
46404649 break;
46414650 case OpDI:
46424651 op->type = OP_MEM;
@@ -4755,12 +4764,87 @@ done: · 112 unmodified lines
47554764 return rc;
47564765 }
47574766
4767+static int x86_decode_avx(struct x86_emulate_ctxt *ctxt,
4768+ u8 vex_1st, u8 vex_2nd, struct opcode *opcode)
4769+{
4770+ u8 vex_3rd, map, pp, l, v;
4771+ int rc = X86EMUL_CONTINUE;
4772+
4773+ if (ctxt->rep_prefix || ctxt->op_prefix || ctxt->rex_prefix)
4774+ goto ud;
4775+
4776+ if (vex_1st == 0xc5) {
4777+ /* Expand RVVVVlpp to VEX3 format */
4778+ vex_3rd = vex_2nd & ~0x80; /* VVVVlpp from VEX2, w=0 */
4779+ vex_2nd = (vex_2nd & 0x80) | 0x61; /* R from VEX2, X=1 B=1 mmmmm=00001 */
4780+ } else {
4781+ vex_3rd = insn_fetch(u8, ctxt);
4782+ }
4783+
4784+ /* vex_2nd = RXBmmmmm, vex_3rd = wVVVVlpp. Fix polarity */
4785+ vex_2nd ^= 0xE0; /* binary 11100000 */
4786+ vex_3rd ^= 0x78; /* binary 01111000 */
4787+
4788+ ctxt->rex_prefix = REX_PREFIX;
4789+ ctxt->rex_bits = (vex_2nd & 0xE0) >> 5; /* RXB */
4790+ ctxt->rex_bits |= (vex_3rd & 0x80) >> 4; /* w */
4791+ if (ctxt->rex_bits && ctxt->mode != X86EMUL_MODE_PROT64)
4792+ goto ud;
4793+
4794+ map = vex_2nd & 0x1f;
4795+ v = (vex_3rd >> 3) & 0xf;
4796+ l = vex_3rd & 0x4;
4797+ pp = vex_3rd & 0x3;
4798+
4799+ ctxt->b = insn_fetch(u8, ctxt);
4800+ switch (map) {
4801+ case 1:
4802+ ctxt->opcode_len = 2;
4803+ *opcode = twobyte_table[ctxt->b];
4804+ break;
4805+ case 2:
4806+ ctxt->opcode_len = 3;
4807+ *opcode = opcode_map_0f_38[ctxt->b];
4808+ break;
4809+ case 3:
4810+ /* no 0f 3a instructions are supported yet */
4811+ return X86EMUL_UNHANDLEABLE;
4812+ default:
4813+ goto ud;
4814+ }
4815+
4816+ /*
4817+ * No three operand instructions are supported yet; those that
4818+ * *are* marked with the Avx flag reserve the VVVV flag.
4819+ */
4820+ if (v)
4821+ goto ud;
4822+
4823+ if (l)
4824+ ctxt->op_bytes = 32;
4825+ else
4826+ ctxt->op_bytes = 16;
4827+
4828+ switch (pp) {
4829+ case 0: break;
4830+ case 1: ctxt->op_prefix = true; break;
4831+ case 2: ctxt->rep_prefix = 0xf3; break;
4832+ case 3: ctxt->rep_prefix = 0xf2; break;
4833+ }
4834+
4835+done:
4836+ return rc;
4837+ud:
4838+ *opcode = ud;
4839+ return rc;
4840+}
4841+
47584842 int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int emulation_type)
47594843 {
47604844 int rc = X86EMUL_CONTINUE;
47614845 int mode = ctxt->mode;
47624846 int def_op_bytes, def_ad_bytes, goffset, simd_prefix;
4763− bool op_prefix = false;
4847+ bool vex_prefix = false;
47644848 bool has_seg_override = false;
47654849 struct opcode opcode;
47664850 u16 dummy;
@@ -4812,7 +4896,7 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int · 45 unmodified lines
48124896 for (;;) {
48134897 switch (ctxt->b = insn_fetch(u8, ctxt)) {
48144898 case 0x66: /* operand-size override */
4815− op_prefix = true;
4899+ ctxt->op_prefix = true;
48164900 /* switch between 2/4 bytes */
48174901 ctxt->op_bytes = def_op_bytes ^ 6;
48184902 break;
@@ -4851,7 +4935,8 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int · 32 unmodified lines
48514935 case 0x40 ... 0x4f: /* REX */
48524936 if (mode != X86EMUL_MODE_PROT64)
48534937 goto done_prefixes;
4854− ctxt->rex_prefix = ctxt->b;
4938+ ctxt->rex_prefix = REX_PREFIX;
4939+ ctxt->rex_bits = ctxt->b & 0xf;
48554940 continue;
48564941 case 0xf0: /* LOCK */
48574942 ctxt->lock_prefix = 1;
@@ -4865,20 +4950,33 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int · 7 unmodified lines
48654950 }
48664951
48674952 /* Any legacy prefix after a REX prefix nullifies its effect. */
4868−
4869− ctxt->rex_prefix = 0;
4953+ ctxt->rex_prefix = REX_NONE;
4954+ ctxt->rex_bits = 0;
48704955 }
48714956
48724957 done_prefixes:
48734958
48744959 /* REX prefix. */
4875− if (ctxt->rex_prefix & 8)
4876− ctxt->op_bytes = 8; /* REX.W */
4960+ if (ctxt->rex_bits & REX_W)
4961+ ctxt->op_bytes = 8;
48774962
48784963 /* Opcode byte(s). */
4879− opcode = opcode_table[ctxt->b];
4880− /* Two-byte opcode? */
4881− if (ctxt->b == 0x0f) {
4964+ if (ctxt->b == 0xc4 || ctxt->b == 0xc5) {
4965+ /* VEX or LDS/LES */
4966+ u8 vex_2nd = insn_fetch(u8, ctxt);
4967+ if (mode != X86EMUL_MODE_PROT64 && (vex_2nd & 0xc0) != 0xc0) {
4968+ opcode = opcode_table[ctxt->b];
4969+ ctxt->modrm = vex_2nd;
4970+ /* the Mod/RM byte has been fetched already! */
4971+ goto done_modrm;
4972+ }
4973+
4974+ vex_prefix = true;
4975+ rc = x86_decode_avx(ctxt, ctxt->b, vex_2nd, &opcode);
4976+ if (rc != X86EMUL_CONTINUE)
4977+ goto done;
4978+ } else if (ctxt->b == 0x0f) {
4979+ /* Two- or three-byte opcode */
48824980 ctxt->opcode_len = 2;
48834981 ctxt->b = insn_fetch(u8, ctxt);
48844982 opcode = twobyte_table[ctxt->b];
@@ -4889,18 +4987,16 @@ done_prefixes: · 4 unmodified lines
48894987 ctxt->b = insn_fetch(u8, ctxt);
48904988 opcode = opcode_map_0f_38[ctxt->b];
48914989 }
4990+ } else {
4991+ /* Opcode byte(s). */
4992+ opcode = opcode_table[ctxt->b];
48924993 }
4893− ctxt->d = opcode.flags;
48944994
4895− if (ctxt->d & ModRM)
4995+ if (opcode.flags & ModRM)
48964996 ctxt->modrm = insn_fetch(u8, ctxt);
48974997
4898− /* vex-prefix instructions are not implemented */
4899− if (ctxt->opcode_len == 1 && (ctxt->b == 0xc5 || ctxt->b == 0xc4) &&
4900− (mode == X86EMUL_MODE_PROT64 || (ctxt->modrm & 0xc0) == 0xc0)) {
4901− ctxt->d = NotImpl;
4902− }
4903−
4998+done_modrm:
4999+ ctxt->d = opcode.flags;
49045000 while (ctxt->d & GroupMask) {
49055001 switch (ctxt->d & GroupMask) {
49065002 case Group:
@@ -4919,9 +5015,9 @@ done_prefixes: · 12 unmodified lines
49195015 opcode = opcode.u.group[goffset];
49205016 break;
49215017 case Prefix:
4922− if (ctxt->rep_prefix && op_prefix)
5018+ if (ctxt->rep_prefix && ctxt->op_prefix)
49235019 return EMULATION_FAILED;
4924− simd_prefix = op_prefix ? 0x66 : ctxt->rep_prefix;
5020+ simd_prefix = ctxt->op_prefix ? 0x66 : ctxt->rep_prefix;
49255021 switch (simd_prefix) {
49265022 case 0x00: opcode = opcode.u.gprefix->pfx_no; break;
49275023 case 0x66: opcode = opcode.u.gprefix->pfx_66; break;
@@ -4966,6 +5062,19 @@ done_prefixes: · 38 unmodified lines
49665062 if (ctxt->d == 0)
49675063 return EMULATION_FAILED;
49685064
5065+ if (unlikely(vex_prefix)) {
5066+ /*
5067+ * Only specifically marked instructions support VEX. Since many
5068+ * instructions support it but are not annotated, return not implemented
5069+ * rather than #UD.
5070+ */
5071+ if (!(ctxt->d & Avx))
5072+ return EMULATION_FAILED;
5073+
5074+ if (!(ctxt->d & AlignMask))
5075+ ctxt->d |= Unaligned;
5076+ }
5077+
49695078 ctxt->execute = opcode.u.execute;
49705079
49715080 /*
@@ -5036,8 +5145,10 @@ done_prefixes: · 64 unmodified lines
50365145 if ((ctxt->d & No16) && ctxt->op_bytes == 2)
50375146 ctxt->op_bytes = 4;
50385147
5039− if (ctxt->d & Sse)
5040− ctxt->op_bytes = 16;
5148+ if (vex_prefix)
5149+ ;
5150+ else if (ctxt->d & Sse)
5151+ ctxt->op_bytes = 16, ctxt->d &= ~Avx;
50415152 else if (ctxt->d & Mmx)
50425153 ctxt->op_bytes = 8;
50435154 }
@@ -5137,8 +5248,10 @@ void init_decode_cache(struct x86_emulate_ctxt *ctxt) · 93 unmodified lines
51375248 {
51385249 /* Clear fields that are set conditionally but read without a guard. */
51395250 ctxt->rip_relative = false;
5140− ctxt->rex_prefix = 0;
5251+ ctxt->rex_prefix = REX_NONE;
5252+ ctxt->rex_bits = 0;
51415253 ctxt->lock_prefix = 0;
5254+ ctxt->op_prefix = false;
51425255 ctxt->rep_prefix = 0;
51435256 ctxt->regs_valid = 0;
51445257 ctxt->regs_dirty = 0;
@@ -5168,20 +5281,34 @@ int x86_emulate_insn(struct x86_emulate_ctxt *ctxt, bool check_intercepts) · 23 unmodified lines
51685281 }
51695282
51705283 if (unlikely(ctxt->d &
5171− (No64|Undefined|Sse|Mmx|Intercept|CheckPerm|Priv|Prot|String))) {
5284+ (No64|Undefined|Avx|Sse|Mmx|Intercept|CheckPerm|Priv|Prot|String))) {
51725285 if ((ctxt->mode == X86EMUL_MODE_PROT64 && (ctxt->d & No64)) ||
51735286 (ctxt->d & Undefined)) {
51745287 rc = emulate_ud(ctxt);
51755288 goto done;
51765289 }
51775290
5178− if (((ctxt->d & (Sse|Mmx)) && ((ops->get_cr(ctxt, 0) & X86_CR0_EM)))
5179− || ((ctxt->d & Sse) && !(ops->get_cr(ctxt, 4) & X86_CR4_OSFXSR))) {
5291+ if ((ctxt->d & (Avx|Sse|Mmx)) && ((ops->get_cr(ctxt, 0) & X86_CR0_EM))) {
51805292 rc = emulate_ud(ctxt);
51815293 goto done;
51825294 }
51835295
5184− if ((ctxt->d & (Sse|Mmx)) && (ops->get_cr(ctxt, 0) & X86_CR0_TS)) {
5296+ if (ctxt->d & Avx) {
5297+ u64 xcr = 0;
5298+ if (!(ops->get_cr(ctxt, 4) & X86_CR4_OSXSAVE)
5299+ || ops->get_xcr(ctxt, 0, &xcr)
5300+ || !(xcr & XFEATURE_MASK_YMM)) {
5301+ rc = emulate_ud(ctxt);
5302+ goto done;
5303+ }
5304+ } else if (ctxt->d & Sse) {
5305+ if (!(ops->get_cr(ctxt, 4) & X86_CR4_OSFXSR)) {
5306+ rc = emulate_ud(ctxt);
5307+ goto done;
5308+ }
5309+ }
5310+
5311+ if ((ctxt->d & (Avx|Sse|Mmx)) && (ops->get_cr(ctxt, 0) & X86_CR0_TS)) {
51855312 rc = emulate_nm(ctxt);
51865313 goto done;
51875314 }
arch/x86/kvm/fpu.h+66−0
@@ -15,6 +15,58 @@ typedef u32 __attribute__((vector_size(16))) sse128_t; · 14 unmodified lines
1515 #define sse128_l3(x) ({ __sse128_u t; t.vec = x; t.as_u32[3]; })
1616 #define sse128(lo, hi) ({ __sse128_u t; t.as_u64[0] = lo; t.as_u64[1] = hi; t.vec; })
1717
18+typedef u32 __attribute__((vector_size(32))) avx256_t;
19+
20+static inline void _kvm_read_avx_reg(int reg, avx256_t *data)
21+{
22+ switch (reg) {
23+ case 0: asm("vmovdqa %%ymm0, %0" : "=m"(*data)); break;
24+ case 1: asm("vmovdqa %%ymm1, %0" : "=m"(*data)); break;
25+ case 2: asm("vmovdqa %%ymm2, %0" : "=m"(*data)); break;
26+ case 3: asm("vmovdqa %%ymm3, %0" : "=m"(*data)); break;
27+ case 4: asm("vmovdqa %%ymm4, %0" : "=m"(*data)); break;
28+ case 5: asm("vmovdqa %%ymm5, %0" : "=m"(*data)); break;
29+ case 6: asm("vmovdqa %%ymm6, %0" : "=m"(*data)); break;
30+ case 7: asm("vmovdqa %%ymm7, %0" : "=m"(*data)); break;
31+#ifdef CONFIG_X86_64
32+ case 8: asm("vmovdqa %%ymm8, %0" : "=m"(*data)); break;
33+ case 9: asm("vmovdqa %%ymm9, %0" : "=m"(*data)); break;
34+ case 10: asm("vmovdqa %%ymm10, %0" : "=m"(*data)); break;
35+ case 11: asm("vmovdqa %%ymm11, %0" : "=m"(*data)); break;
36+ case 12: asm("vmovdqa %%ymm12, %0" : "=m"(*data)); break;
37+ case 13: asm("vmovdqa %%ymm13, %0" : "=m"(*data)); break;
38+ case 14: asm("vmovdqa %%ymm14, %0" : "=m"(*data)); break;
39+ case 15: asm("vmovdqa %%ymm15, %0" : "=m"(*data)); break;
40+#endif
41+ default: BUG();
42+ }
43+}
44+
45+static inline void _kvm_write_avx_reg(int reg, const avx256_t *data)
46+{
47+ switch (reg) {
48+ case 0: asm("vmovdqa %0, %%ymm0" : : "m"(*data)); break;
49+ case 1: asm("vmovdqa %0, %%ymm1" : : "m"(*data)); break;
50+ case 2: asm("vmovdqa %0, %%ymm2" : : "m"(*data)); break;
51+ case 3: asm("vmovdqa %0, %%ymm3" : : "m"(*data)); break;
52+ case 4: asm("vmovdqa %0, %%ymm4" : : "m"(*data)); break;
53+ case 5: asm("vmovdqa %0, %%ymm5" : : "m"(*data)); break;
54+ case 6: asm("vmovdqa %0, %%ymm6" : : "m"(*data)); break;
55+ case 7: asm("vmovdqa %0, %%ymm7" : : "m"(*data)); break;
56+#ifdef CONFIG_X86_64
57+ case 8: asm("vmovdqa %0, %%ymm8" : : "m"(*data)); break;
58+ case 9: asm("vmovdqa %0, %%ymm9" : : "m"(*data)); break;
59+ case 10: asm("vmovdqa %0, %%ymm10" : : "m"(*data)); break;
60+ case 11: asm("vmovdqa %0, %%ymm11" : : "m"(*data)); break;
61+ case 12: asm("vmovdqa %0, %%ymm12" : : "m"(*data)); break;
62+ case 13: asm("vmovdqa %0, %%ymm13" : : "m"(*data)); break;
63+ case 14: asm("vmovdqa %0, %%ymm14" : : "m"(*data)); break;
64+ case 15: asm("vmovdqa %0, %%ymm15" : : "m"(*data)); break;
65+#endif
66+ default: BUG();
67+ }
68+}
69+
1870 static inline void _kvm_read_sse_reg(int reg, sse128_t *data)
1971 {
2072 switch (reg) {
@@ -109,6 +161,20 @@ static inline void kvm_fpu_put(void) · 88 unmodified lines
109161 fpregs_unlock();
110162 }
111163
164+static inline void kvm_read_avx_reg(int reg, avx256_t *data)
165+{
166+ kvm_fpu_get();
167+ _kvm_read_avx_reg(reg, data);
168+ kvm_fpu_put();
169+}
170+
171+static inline void kvm_write_avx_reg(int reg, const avx256_t *data)
172+{
173+ kvm_fpu_get();
174+ _kvm_write_avx_reg(reg, data);
175+ kvm_fpu_put();
176+}
177+
112178 static inline void kvm_read_sse_reg(int reg, sse128_t *data)
113179 {
114180 kvm_fpu_get();
arch/x86/kvm/hyperv.c+1−1
@@ -1568,7 +1568,7 @@ static int kvm_hv_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool host) · 1567 unmodified lines
15681568 * only, there can be valuable data in the rest which needs
15691569 * to be preserved e.g. on migration.
15701570 */
1571− if (__put_user(0, (u32 __user *)addr))
1571+ if (put_user(0, (u32 __user *)addr))
15721572 return 1;
15731573 hv_vcpu->hv_vapic = data;
15741574 kvm_vcpu_mark_page_dirty(vcpu, gfn);
arch/x86/kvm/kvm_emulate.h+16−4
@@ -237,6 +237,7 @@ struct x86_emulate_ops { · 236 unmodified lines
237237 bool (*is_smm)(struct x86_emulate_ctxt *ctxt);
238238 int (*leave_smm)(struct x86_emulate_ctxt *ctxt);
239239 void (*triple_fault)(struct x86_emulate_ctxt *ctxt);
240+ int (*get_xcr)(struct x86_emulate_ctxt *ctxt, u32 index, u64 *xcr);
240241 int (*set_xcr)(struct x86_emulate_ctxt *ctxt, u32 index, u64 xcr);
241242
242243 gva_t (*get_untagged_addr)(struct x86_emulate_ctxt *ctxt, gva_t addr,
@@ -248,7 +249,7 @@ struct x86_emulate_ops { · 5 unmodified lines
248249
249250 /* Type, address-of, and value of an instruction's operand. */
250251 struct operand {
251− enum { OP_REG, OP_MEM, OP_MEM_STR, OP_IMM, OP_XMM, OP_MM, OP_NONE } type;
252+ enum { OP_REG, OP_MEM, OP_MEM_STR, OP_IMM, OP_XMM, OP_YMM, OP_MM, OP_NONE } type;
252253 unsigned int bytes;
253254 unsigned int count;
254255 union {
@@ -267,11 +268,12 @@ struct operand { · 12 unmodified lines
267268 union {
268269 unsigned long val;
269270 u64 val64;
270− char valptr[sizeof(sse128_t)];
271+ char valptr[sizeof(avx256_t)];
271272 sse128_t vec_val;
273+ avx256_t vec_val2;
272274 u64 mm_val;
273275 void *data;
274− };
276+ } __aligned(32);
275277 };
276278
277279 #define X86_MAX_INSTRUCTION_LENGTH 15
@@ -317,6 +319,14 @@ typedef void (*fastop_t)(struct fastop *); · 39 unmodified lines
317319 #define NR_EMULATOR_GPRS 8
318320 #endif
319321
322+/*
323+ * Distinguish between no prefix, REX, or in the future REX2.
324+ */
325+enum rex_type {
326+ REX_NONE,
327+ REX_PREFIX,
328+};
329+
320330 struct x86_emulate_ctxt {
321331 void *vcpu;
322332 const struct x86_emulate_ops *ops;
@@ -348,6 +358,7 @@ struct x86_emulate_ctxt { · 25 unmodified lines
348358 u8 opcode_len;
349359 u8 b;
350360 u8 intercept;
361+ bool op_prefix;
351362 u8 op_bytes;
352363 u8 ad_bytes;
353364 union {
@@ -357,7 +368,8 @@ struct x86_emulate_ctxt { · 3 unmodified lines
357368 int (*check_perm)(struct x86_emulate_ctxt *ctxt);
358369
359370 bool rip_relative;
360− u8 rex_prefix;
371+ enum rex_type rex_prefix;
372+ u8 rex_bits;
361373 u8 lock_prefix;
362374 u8 rep_prefix;
363375 /* bitmaps of registers in _regs[] that can be read */
arch/x86/kvm/lapic.c+31−13
@@ -2126,23 +2126,41 @@ static bool set_target_expiration(struct kvm_lapic *apic, u32 count_reg) · 2125 unmodified lines
21262126
21272127 static void advance_periodic_target_expiration(struct kvm_lapic *apic)
21282128 {
2129+ struct kvm_timer *ktimer = &apic->lapic_timer;
21292130 ktime_t now = ktime_get();
21302131 u64 tscl = rdtsc();
21312132 ktime_t delta;
21322133
21332134 /*
2134− * Synchronize both deadlines to the same time source or
2135− * differences in the periods (caused by differences in the
2136− * underlying clocks or numerical approximation errors) will
2137− * cause the two to drift apart over time as the errors
2138− * accumulate.
2135+ * Use kernel time as the time source for both the hrtimer deadline and
2136+ * TSC-based deadline so that they stay synchronized. Computing each
2137+ * deadline independently will cause the two deadlines to drift apart
2138+ * over time as differences in the periods accumulate, e.g. due to
2139+ * differences in the underlying clocks or numerical approximation errors.
21392140 */
2140− apic->lapic_timer.target_expiration =
2141− ktime_add_ns(apic->lapic_timer.target_expiration,
2142− apic->lapic_timer.period);
2143− delta = ktime_sub(apic->lapic_timer.target_expiration, now);
2144− apic->lapic_timer.tscdeadline = kvm_read_l1_tsc(apic->vcpu, tscl) +
2145− nsec_to_cycles(apic->vcpu, delta);
2141+ ktimer->target_expiration = ktime_add_ns(ktimer->target_expiration,
2142+ ktimer->period);
2143+
2144+ /*
2145+ * If the new expiration is in the past, e.g. because userspace stopped
2146+ * running the VM for an extended duration, then force the expiration
2147+ * to "now" and don't try to play catch-up with the missed events. KVM
2148+ * will only deliver a single interrupt regardless of how many events
2149+ * are pending, i.e. restarting the timer with an expiration in the
2150+ * past will do nothing more than waste host cycles, and can even lead
2151+ * to a hard lockup in extreme cases.
2152+ */
2153+ if (ktime_before(ktimer->target_expiration, now))
2154+ ktimer->target_expiration = now;
2155+
2156+ /*
2157+ * Note, ensuring the expiration isn't in the past also prevents delta
2158+ * from going negative, which could cause the TSC deadline to become
2159+ * excessively large due to it an unsigned value.
2160+ */
2161+ delta = ktime_sub(ktimer->target_expiration, now);
2162+ ktimer->tscdeadline = kvm_read_l1_tsc(apic->vcpu, tscl) +
2163+ nsec_to_cycles(apic->vcpu, delta);
21462164 }
21472165
21482166 static void start_sw_period(struct kvm_lapic *apic)
@@ -2970,9 +2988,9 @@ static enum hrtimer_restart apic_timer_fn(struct hrtimer *data) · 821 unmodified lines
29702988
29712989 apic_timer_expired(apic, true);
29722990
2973− if (lapic_is_periodic(apic)) {
2991+ if (lapic_is_periodic(apic) && !WARN_ON_ONCE(!apic->lapic_timer.period)) {
29742992 advance_periodic_target_expiration(apic);
2975− hrtimer_add_expires_ns(&ktimer->timer, ktimer->period);
2993+ hrtimer_set_expires(&ktimer->timer, ktimer->target_expiration);
29762994 return HRTIMER_RESTART;
29772995 } else
29782996 return HRTIMER_NORESTART;
arch/x86/kvm/mmu.h+1−4
@@ -235,8 +235,6 @@ static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, · 234 unmodified lines
235235 return -(u32)fault & errcode;
236236 }
237237
238−bool kvm_mmu_may_ignore_guest_pat(struct kvm *kvm);
239−
240238 int kvm_mmu_post_init_vm(struct kvm *kvm);
241239 void kvm_mmu_pre_destroy_vm(struct kvm *kvm);
242240
@@ -257,8 +255,7 @@ extern bool tdp_mmu_enabled; · 14 unmodified lines
257255 #define tdp_mmu_enabled false
258256 #endif
259257
260−bool kvm_tdp_mmu_gpa_is_mapped(struct kvm_vcpu *vcpu, u64 gpa);
261−int kvm_tdp_map_page(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code, u8 *level);
258+int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn);
262259
263260 static inline bool kvm_memslots_have_rmaps(struct kvm *kvm)
264261 {
arch/x86/kvm/mmu/mmu.c+88−6
@@ -4859,7 +4859,7 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, · 4858 unmodified lines
48594859 */
48604860 BUILD_BUG_ON(lower_32_bits(PFERR_SYNTHETIC_MASK));
48614861
4862− vcpu->arch.l1tf_flush_l1d = true;
4862+ kvm_request_l1tf_flush_l1d();
48634863 if (!flags) {
48644864 trace_kvm_page_fault(vcpu, fault_address, error_code);
48654865
@@ -4924,7 +4924,8 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 58 unmodified lines
49244924 return direct_page_fault(vcpu, fault);
49254925 }
49264926
4927−int kvm_tdp_map_page(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code, u8 *level)
4927+static int kvm_tdp_page_prefault(struct kvm_vcpu *vcpu, gpa_t gpa,
4928+ u64 error_code, u8 *level)
49284929 {
49294930 int r;
49304931
@@ -4966,7 +4967,6 @@ int kvm_tdp_map_page(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code, u8 *level · 35 unmodified lines
49664967 return -EIO;
49674968 }
49684969 }
4969−EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_tdp_map_page);
49704970
49714971 long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
49724972 struct kvm_pre_fault_memory *range)
@@ -5002,7 +5002,7 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, · 29 unmodified lines
50025002 * Shadow paging uses GVA for kvm page fault, so restrict to
50035003 * two-dimensional paging.
50045004 */
5005− r = kvm_tdp_map_page(vcpu, range->gpa | direct_bits, error_code, &level);
5005+ r = kvm_tdp_page_prefault(vcpu, range->gpa | direct_bits, error_code, &level);
50065006 if (r < 0)
50075007 return r;
50085008
@@ -5014,6 +5014,86 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, · 5 unmodified lines
50145014 return min(range->size, end - range->gpa);
50155015 }
50165016
5017+#ifdef CONFIG_KVM_GUEST_MEMFD
5018+static void kvm_assert_gmem_invalidate_lock_held(struct kvm_memory_slot *slot)
5019+{
5020+#ifdef CONFIG_PROVE_LOCKING
5021+ if (WARN_ON_ONCE(!kvm_slot_has_gmem(slot)) ||
5022+ WARN_ON_ONCE(!slot->gmem.file) ||
5023+ WARN_ON_ONCE(!file_count(slot->gmem.file)))
5024+ return;
5025+
5026+ lockdep_assert_held(&file_inode(slot->gmem.file)->i_mapping->invalidate_lock);
5027+#endif
5028+}
5029+
5030+int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)
5031+{
5032+ struct kvm_page_fault fault = {
5033+ .addr = gfn_to_gpa(gfn),
5034+ .error_code = PFERR_GUEST_FINAL_MASK | PFERR_PRIVATE_ACCESS,
5035+ .prefetch = true,
5036+ .is_tdp = true,
5037+ .nx_huge_page_workaround_enabled = is_nx_huge_page_enabled(vcpu->kvm),
5038+
5039+ .max_level = PG_LEVEL_4K,
5040+ .req_level = PG_LEVEL_4K,
5041+ .goal_level = PG_LEVEL_4K,
5042+ .is_private = true,
5043+
5044+ .gfn = gfn,
5045+ .slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn),
5046+ .pfn = pfn,
5047+ .map_writable = true,
5048+ };
5049+ struct kvm *kvm = vcpu->kvm;
5050+ int r;
5051+
5052+ lockdep_assert_held(&kvm->slots_lock);
5053+
5054+ /*
5055+ * Mapping a pre-determined private pfn is intended only for use when
5056+ * populating a guest_memfd instance. Assert that the slot is backed
5057+ * by guest_memfd and that the gmem instance's invalidate_lock is held.
5058+ */
5059+ kvm_assert_gmem_invalidate_lock_held(fault.slot);
5060+
5061+ if (KVM_BUG_ON(!tdp_mmu_enabled, kvm))
5062+ return -EIO;
5063+
5064+ if (kvm_gfn_is_write_tracked(kvm, fault.slot, fault.gfn))
5065+ return -EPERM;
5066+
5067+ r = kvm_mmu_reload(vcpu);
5068+ if (r)
5069+ return r;
5070+
5071+ r = mmu_topup_memory_caches(vcpu, false);
5072+ if (r)
5073+ return r;
5074+
5075+ do {
5076+ if (signal_pending(current))
5077+ return -EINTR;
5078+
5079+ if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
5080+ return -EIO;
5081+
5082+ cond_resched();
5083+
5084+ guard(read_lock)(&kvm->mmu_lock);
5085+
5086+ r = kvm_tdp_mmu_map(vcpu, &fault);
5087+ } while (r == RET_PF_RETRY);
5088+
5089+ if (r != RET_PF_FIXED)
5090+ return -EIO;
5091+
5092+ return 0;
5093+}
5094+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_tdp_mmu_map_private_pfn);
5095+#endif
5096+
50175097 static void nonpaging_init_context(struct kvm_mmu *context)
50185098 {
50195099 context->page_fault = nonpaging_page_fault;
@@ -5997,7 +6077,6 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) · 977 unmodified lines
59976077 out:
59986078 return r;
59996079 }
6000−EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_load);
60016080
60026081 void kvm_mmu_unload(struct kvm_vcpu *vcpu)
60036082 {
@@ -6863,6 +6942,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) · 859 unmodified lines
68636942
68646943 write_unlock(&kvm->mmu_lock);
68656944 }
6945+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_zap_gfn_range);
68666946
68676947 static bool slot_rmap_write_protect(struct kvm *kvm,
68686948 struct kvm_rmap_head *rmap_head,
@@ -7204,7 +7284,6 @@ restart: · 335 unmodified lines
72047284
72057285 return need_tlb_flush;
72067286 }
7207−EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_zap_gfn_range);
72087287
72097288 static void kvm_rmap_zap_collapsible_sptes(struct kvm *kvm,
72107289 const struct kvm_memory_slot *slot)
@@ -7364,6 +7443,9 @@ void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen) · 153 unmodified lines
73647443 {
73657444 WARN_ON_ONCE(gen & KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS);
73667445
7446+ if (!enable_mmio_caching)
7447+ return;
7448+
73677449 gen &= MMIO_SPTE_GEN_MASK;
73687450
73697451 /*
arch/x86/kvm/mmu/mmu_internal.h+0−10
@@ -39,16 +39,6 @@ · 38 unmodified lines
3939 #define INVALID_PAE_ROOT 0
4040 #define IS_VALID_PAE_ROOT(x) (!!(x))
4141
42−static inline hpa_t kvm_mmu_get_dummy_root(void)
43−{
44− return my_zero_pfn(0) << PAGE_SHIFT;
45−}
46−
47−static inline bool kvm_mmu_is_dummy_root(hpa_t shadow_page)
48−{
49− return is_zero_pfn(shadow_page >> PAGE_SHIFT);
50−}
51−
5242 typedef u64 __rcu *tdp_ptep_t;
5343
5444 struct kvm_mmu_page {
arch/x86/kvm/mmu/paging_tmpl.h+1−1
@@ -402,7 +402,7 @@ retry_walk: · 401 unmodified lines
402402 goto error;
403403
404404 ptep_user = (pt_element_t __user *)((void *)host_addr + offset);
405− if (unlikely(__get_user(pte, ptep_user)))
405+ if (unlikely(get_user(pte, ptep_user)))
406406 goto error;
407407 walker->ptep_user[walker->level - 1] = ptep_user;
408408
arch/x86/kvm/mmu/spte.c+1−1
@@ -292,7 +292,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, · 291 unmodified lines
292292 mark_page_dirty_in_slot(vcpu->kvm, slot, gfn);
293293 }
294294
295− if (static_branch_unlikely(&cpu_buf_vm_clear) &&
295+ if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO) &&
296296 !kvm_vcpu_can_access_host_mmio(vcpu) &&
297297 kvm_is_mmio_pfn(pfn, &is_host_mmio))
298298 kvm_track_host_mmio_mapping(vcpu);
arch/x86/kvm/mmu/spte.h+10−0
@@ -246,6 +246,16 @@ static inline int spte_index(u64 *sptep) · 245 unmodified lines
246246 */
247247 extern u64 __read_mostly shadow_nonpresent_or_rsvd_lower_gfn_mask;
248248
249+static inline hpa_t kvm_mmu_get_dummy_root(void)
250+{
251+ return my_zero_pfn(0) << PAGE_SHIFT;
252+}
253+
254+static inline bool kvm_mmu_is_dummy_root(hpa_t shadow_page)
255+{
256+ return is_zero_pfn(shadow_page >> PAGE_SHIFT);
257+}
258+
249259 static inline struct kvm_mmu_page *to_shadow_page(hpa_t shadow_page)
250260 {
251261 struct page *page = pfn_to_page((shadow_page) >> PAGE_SHIFT);
arch/x86/kvm/mmu/tdp_mmu.c+10−40
@@ -362,9 +362,6 @@ static void tdp_mmu_unlink_sp(struct kvm *kvm, struct kvm_mmu_page *sp) · 361 unmodified lines
362362 static void remove_external_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
363363 int level)
364364 {
365− kvm_pfn_t old_pfn = spte_to_pfn(old_spte);
366− int ret;
367−
368365 /*
369366 * External (TDX) SPTEs are limited to PG_LEVEL_4K, and external
370367 * PTs are removed in a special order, involving free_external_spt().
@@ -377,9 +374,8 @@ static void remove_external_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte, · 6 unmodified lines
377374
378375 /* Zapping leaf spte is allowed only when write lock is held. */
379376 lockdep_assert_held_write(&kvm->mmu_lock);
380− /* Because write lock is held, operation should success. */
381− ret = kvm_x86_call(remove_external_spte)(kvm, gfn, level, old_pfn);
382− KVM_BUG_ON(ret, kvm);
377+
378+ kvm_x86_call(remove_external_spte)(kvm, gfn, level, old_spte);
383379 }
384380
385381 /**
@@ -519,7 +515,6 @@ static int __must_check set_external_spte_present(struct kvm *kvm, tdp_ptep_t sp · 133 unmodified lines
519515 bool was_present = is_shadow_present_pte(old_spte);
520516 bool is_present = is_shadow_present_pte(new_spte);
521517 bool is_leaf = is_present && is_last_spte(new_spte, level);
522− kvm_pfn_t new_pfn = spte_to_pfn(new_spte);
523518 int ret = 0;
524519
525520 KVM_BUG_ON(was_present, kvm);
@@ -538,7 +533,7 @@ static int __must_check set_external_spte_present(struct kvm *kvm, tdp_ptep_t sp · 12 unmodified lines
538533 * external page table, or leaf.
539534 */
540535 if (is_leaf) {
541− ret = kvm_x86_call(set_external_spte)(kvm, gfn, level, new_pfn);
536+ ret = kvm_x86_call(set_external_spte)(kvm, gfn, level, new_spte);
542537 } else {
543538 void *external_spt = get_external_spt(gfn, new_spte, level);
544539
@@ -1273,6 +1268,8 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 728 unmodified lines
12731268 struct kvm_mmu_page *sp;
12741269 int ret = RET_PF_RETRY;
12751270
1271+ KVM_MMU_WARN_ON(!root || root->role.invalid);
1272+
12761273 kvm_mmu_hugepage_adjust(vcpu, fault);
12771274
12781275 trace_kvm_mmu_spte_requested(fault);
@@ -1939,13 +1936,16 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm, · 660 unmodified lines
19391936 *
19401937 * Must be called between kvm_tdp_mmu_walk_lockless_{begin,end}.
19411938 */
1942−static int __kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes,
1943− struct kvm_mmu_page *root)
1939+int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes,
1940+ int *root_level)
19441941 {
1942+ struct kvm_mmu_page *root = root_to_sp(vcpu->arch.mmu->root.hpa);
19451943 struct tdp_iter iter;
19461944 gfn_t gfn = addr >> PAGE_SHIFT;
19471945 int leaf = -1;
19481946
1947+ *root_level = vcpu->arch.mmu->root_role.level;
1948+
19491949 for_each_tdp_pte(iter, vcpu->kvm, root, gfn, gfn + 1) {
19501950 leaf = iter.level;
19511951 sptes[leaf] = iter.old_spte;
@@ -1954,36 +1954,6 @@ static int __kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, · 2 unmodified lines
19541954 return leaf;
19551955 }
19561956
1957−int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes,
1958− int *root_level)
1959−{
1960− struct kvm_mmu_page *root = root_to_sp(vcpu->arch.mmu->root.hpa);
1961− *root_level = vcpu->arch.mmu->root_role.level;
1962−
1963− return __kvm_tdp_mmu_get_walk(vcpu, addr, sptes, root);
1964−}
1965−
1966−bool kvm_tdp_mmu_gpa_is_mapped(struct kvm_vcpu *vcpu, u64 gpa)
1967−{
1968− struct kvm *kvm = vcpu->kvm;
1969− bool is_direct = kvm_is_addr_direct(kvm, gpa);
1970− hpa_t root = is_direct ? vcpu->arch.mmu->root.hpa :
1971− vcpu->arch.mmu->mirror_root_hpa;
1972− u64 sptes[PT64_ROOT_MAX_LEVEL + 1], spte;
1973− int leaf;
1974−
1975− lockdep_assert_held(&kvm->mmu_lock);
1976− rcu_read_lock();
1977− leaf = __kvm_tdp_mmu_get_walk(vcpu, gpa, sptes, root_to_sp(root));
1978− rcu_read_unlock();
1979− if (leaf < 0)
1980− return false;
1981−
1982− spte = sptes[leaf];
1983− return is_shadow_present_pte(spte) && is_last_spte(spte, leaf);
1984−}
1985−EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_tdp_mmu_gpa_is_mapped);
1986−
19871957 /*
19881958 * Returns the last level spte pointer of the shadow page walk for the given
19891959 * gpa, and sets *spte to the spte value. This spte may be non-preset. If no
arch/x86/kvm/svm/avic.c+70−16
@@ -106,7 +106,7 @@ static u32 next_vm_id = 0; · 105 unmodified lines
106106 static bool next_vm_id_wrapped = 0;
107107 static DEFINE_SPINLOCK(svm_vm_data_hash_lock);
108108 static bool x2avic_enabled;
109−
109+static u32 x2avic_max_physical_id;
110110
111111 static void avic_set_x2apic_msr_interception(struct vcpu_svm *svm,
112112 bool intercept)
@@ -158,12 +158,40 @@ static void avic_set_x2apic_msr_interception(struct vcpu_svm *svm, · 45 unmodified lines
158158 svm->x2avic_msrs_intercepted = intercept;
159159 }
160160
161+static u32 __avic_get_max_physical_id(struct kvm *kvm, struct kvm_vcpu *vcpu)
162+{
163+ u32 arch_max;
164+
165+ /*
166+ * Return the largest size (x2APIC) when querying without a vCPU, e.g.
167+ * to allocate the per-VM table..
168+ */
169+ if (x2avic_enabled && (!vcpu || apic_x2apic_mode(vcpu->arch.apic)))
170+ arch_max = x2avic_max_physical_id;
171+ else
172+ arch_max = AVIC_MAX_PHYSICAL_ID;
173+
174+ /*
175+ * Despite its name, KVM_CAP_MAX_VCPU_ID represents the maximum APIC ID
176+ * plus one, so the max possible APIC ID is one less than that.
177+ */
178+ return min(kvm->arch.max_vcpu_ids - 1, arch_max);
179+}
180+
181+static u32 avic_get_max_physical_id(struct kvm_vcpu *vcpu)
182+{
183+ return __avic_get_max_physical_id(vcpu->kvm, vcpu);
184+}
185+
161186 static void avic_activate_vmcb(struct vcpu_svm *svm)
162187 {
163188 struct vmcb *vmcb = svm->vmcb01.ptr;
189+ struct kvm_vcpu *vcpu = &svm->vcpu;
164190
165191 vmcb->control.int_ctl &= ~(AVIC_ENABLE_MASK | X2APIC_MODE_MASK);
192+
166193 vmcb->control.avic_physical_id &= ~AVIC_PHYSICAL_MAX_INDEX_MASK;
194+ vmcb->control.avic_physical_id |= avic_get_max_physical_id(vcpu);
167195
168196 vmcb->control.int_ctl |= AVIC_ENABLE_MASK;
169197
@@ -176,7 +204,7 @@ static void avic_activate_vmcb(struct vcpu_svm *svm) · 6 unmodified lines
176204 */
177205 if (x2avic_enabled && apic_x2apic_mode(svm->vcpu.arch.apic)) {
178206 vmcb->control.int_ctl |= X2APIC_MODE_MASK;
179− vmcb->control.avic_physical_id |= X2AVIC_MAX_PHYSICAL_ID;
207+
180208 /* Disabling MSR intercept for x2APIC registers */
181209 avic_set_x2apic_msr_interception(svm, false);
182210 } else {
@@ -186,8 +214,6 @@ static void avic_activate_vmcb(struct vcpu_svm *svm) · 3 unmodified lines
186214 */
187215 kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, &svm->vcpu);
188216
189− /* For xAVIC and hybrid-xAVIC modes */
190− vmcb->control.avic_physical_id |= AVIC_MAX_PHYSICAL_ID;
191217 /* Enabling MSR intercept for x2APIC registers */
192218 avic_set_x2apic_msr_interception(svm, true);
193219 }
@@ -247,6 +273,30 @@ static int avic_ga_log_notifier(u32 ga_tag) · 53 unmodified lines
247273 return 0;
248274 }
249275
276+static int avic_get_physical_id_table_order(struct kvm *kvm)
277+{
278+ /* Provision for the maximum physical ID supported in x2avic mode */
279+ return get_order((__avic_get_max_physical_id(kvm, NULL) + 1) * sizeof(u64));
280+}
281+
282+int avic_alloc_physical_id_table(struct kvm *kvm)
283+{
284+ struct kvm_svm *kvm_svm = to_kvm_svm(kvm);
285+
286+ if (!irqchip_in_kernel(kvm) || !enable_apicv)
287+ return 0;
288+
289+ if (kvm_svm->avic_physical_id_table)
290+ return 0;
291+
292+ kvm_svm->avic_physical_id_table = (void *)__get_free_pages(GFP_KERNEL_ACCOUNT | __GFP_ZERO,
293+ avic_get_physical_id_table_order(kvm));
294+ if (!kvm_svm->avic_physical_id_table)
295+ return -ENOMEM;
296+
297+ return 0;
298+}
299+
250300 void avic_vm_destroy(struct kvm *kvm)
251301 {
252302 unsigned long flags;
@@ -256,7 +306,8 @@ void avic_vm_destroy(struct kvm *kvm) · 3 unmodified lines
256306 return;
257307
258308 free_page((unsigned long)kvm_svm->avic_logical_id_table);
259− free_page((unsigned long)kvm_svm->avic_physical_id_table);
309+ free_pages((unsigned long)kvm_svm->avic_physical_id_table,
310+ avic_get_physical_id_table_order(kvm));
260311
261312 spin_lock_irqsave(&svm_vm_data_hash_lock, flags);
262313 hash_del(&kvm_svm->hnode);
@@ -274,10 +325,6 @@ int avic_vm_init(struct kvm *kvm) · 11 unmodified lines
274325 if (!enable_apicv)
275326 return 0;
276327
277− kvm_svm->avic_physical_id_table = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT);
278− if (!kvm_svm->avic_physical_id_table)
279− goto free_avic;
280−
281328 kvm_svm->avic_logical_id_table = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT);
282329 if (!kvm_svm->avic_logical_id_table)
283330 goto free_avic;
@@ -342,7 +389,7 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu) · 58 unmodified lines
342389 * fully initialized AVIC.
343390 */
344391 if ((!x2avic_enabled && id > AVIC_MAX_PHYSICAL_ID) ||
345− (id > X2AVIC_MAX_PHYSICAL_ID)) {
392+ (id > x2avic_max_physical_id)) {
346393 kvm_set_apicv_inhibit(vcpu->kvm, APICV_INHIBIT_REASON_PHYSICAL_ID_TOO_BIG);
347394 vcpu->arch.apic->apicv_active = false;
348395 return 0;
@@ -562,7 +609,7 @@ int avic_incomplete_ipi_interception(struct kvm_vcpu *vcpu) · 213 unmodified lines
562609 u32 icrh = svm->vmcb->control.exit_info_1 >> 32;
563610 u32 icrl = svm->vmcb->control.exit_info_1;
564611 u32 id = svm->vmcb->control.exit_info_2 >> 32;
565− u32 index = svm->vmcb->control.exit_info_2 & 0x1FF;
612+ u32 index = svm->vmcb->control.exit_info_2 & AVIC_PHYSICAL_MAX_INDEX_MASK;
566613 struct kvm_lapic *apic = vcpu->arch.apic;
567614
568615 trace_kvm_avic_incomplete_ipi(vcpu->vcpu_id, icrh, icrl, id, index);
@@ -962,7 +1009,8 @@ static void __avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu, · 393 unmodified lines
9621009 if (WARN_ON(h_physical_id & ~AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK))
9631010 return;
9641011
965− if (WARN_ON_ONCE(vcpu->vcpu_id * sizeof(entry) >= PAGE_SIZE))
1012+ if (WARN_ON_ONCE(vcpu->vcpu_id * sizeof(entry) >=
1013+ PAGE_SIZE << avic_get_physical_id_table_order(vcpu->kvm)))
9661014 return;
9671015
9681016 /*
@@ -1024,7 +1072,8 @@ static void __avic_vcpu_put(struct kvm_vcpu *vcpu, enum avic_vcpu_action action) · 55 unmodified lines
10241072
10251073 lockdep_assert_preemption_disabled();
10261074
1027− if (WARN_ON_ONCE(vcpu->vcpu_id * sizeof(entry) >= PAGE_SIZE))
1075+ if (WARN_ON_ONCE(vcpu->vcpu_id * sizeof(entry) >=
1076+ PAGE_SIZE << avic_get_physical_id_table_order(vcpu->kvm)))
10281077 return;
10291078
10301079 /*
@@ -1226,10 +1275,15 @@ bool __init avic_hardware_setup(void) · 195 unmodified lines
12261275
12271276 /* AVIC is a prerequisite for x2AVIC. */
12281277 x2avic_enabled = boot_cpu_has(X86_FEATURE_X2AVIC);
1229− if (x2avic_enabled)
1230− pr_info("x2AVIC enabled\n");
1231− else
1278+ if (x2avic_enabled) {
1279+ if (cpu_feature_enabled(X86_FEATURE_X2AVIC_EXT))
1280+ x2avic_max_physical_id = X2AVIC_4K_MAX_PHYSICAL_ID;
1281+ else
1282+ x2avic_max_physical_id = X2AVIC_MAX_PHYSICAL_ID;
1283+ pr_info("x2AVIC enabled (max %u vCPUs)\n", x2avic_max_physical_id + 1);
1284+ } else {
12321285 svm_x86_ops.allow_apicv_in_x2apic_without_x2apic_virtualization = true;
1286+ }
12331287
12341288 /*
12351289 * Disable IPI virtualization for AMD Family 17h CPUs (Zen1 and Zen2)
arch/x86/kvm/svm/nested.c+2−10
@@ -613,6 +613,7 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm, struct vmcb *vmcb12 · 612 unmodified lines
613613 struct kvm_vcpu *vcpu = &svm->vcpu;
614614
615615 nested_vmcb02_compute_g_pat(svm);
616+ vmcb_mark_dirty(vmcb02, VMCB_NPT);
616617
617618 /* Load the nested guest state */
618619 if (svm->nested.vmcb12_gpa != svm->nested.last_vmcb12_gpa) {
@@ -751,6 +752,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm, · 132 unmodified lines
751752 vmcb02->control.nested_ctl = vmcb01->control.nested_ctl;
752753 vmcb02->control.iopm_base_pa = vmcb01->control.iopm_base_pa;
753754 vmcb02->control.msrpm_base_pa = vmcb01->control.msrpm_base_pa;
755+ vmcb_mark_dirty(vmcb02, VMCB_PERM_MAP);
754756
755757 /*
756758 * Stash vmcb02's counter if the guest hasn't moved past the guilty
@@ -1430,16 +1432,6 @@ static int nested_svm_intercept(struct vcpu_svm *svm) · 673 unmodified lines
14301432 case SVM_EXIT_IOIO:
14311433 vmexit = nested_svm_intercept_ioio(svm);
14321434 break;
1433− case SVM_EXIT_READ_CR0 ... SVM_EXIT_WRITE_CR8: {
1434− if (vmcb12_is_intercept(&svm->nested.ctl, exit_code))
1435− vmexit = NESTED_EXIT_DONE;
1436− break;
1437− }
1438− case SVM_EXIT_READ_DR0 ... SVM_EXIT_WRITE_DR7: {
1439− if (vmcb12_is_intercept(&svm->nested.ctl, exit_code))
1440− vmexit = NESTED_EXIT_DONE;
1441− break;
1442− }
14431435 case SVM_EXIT_EXCP_BASE ... SVM_EXIT_EXCP_BASE + 0x1f: {
14441436 /*
14451437 * Host-intercepted exceptions have been checked already in
arch/x86/kvm/svm/sev.c+29−18
@@ -65,20 +65,24 @@ module_param_named(ciphertext_hiding_asids, nr_ciphertext_hiding_asids, uint, 04 · 64 unmodified lines
6565 #define AP_RESET_HOLD_NAE_EVENT 1
6666 #define AP_RESET_HOLD_MSR_PROTO 2
6767
68−/* As defined by SEV-SNP Firmware ABI, under "Guest Policy". */
69−#define SNP_POLICY_MASK_API_MINOR GENMASK_ULL(7, 0)
70−#define SNP_POLICY_MASK_API_MAJOR GENMASK_ULL(15, 8)
71−#define SNP_POLICY_MASK_SMT BIT_ULL(16)
72−#define SNP_POLICY_MASK_RSVD_MBO BIT_ULL(17)
73−#define SNP_POLICY_MASK_DEBUG BIT_ULL(19)
74−#define SNP_POLICY_MASK_SINGLE_SOCKET BIT_ULL(20)
75−
76−#define SNP_POLICY_MASK_VALID (SNP_POLICY_MASK_API_MINOR | \
77− SNP_POLICY_MASK_API_MAJOR | \
78− SNP_POLICY_MASK_SMT | \
79− SNP_POLICY_MASK_RSVD_MBO | \
80− SNP_POLICY_MASK_DEBUG | \
81− SNP_POLICY_MASK_SINGLE_SOCKET)
68+/*
69+ * SEV-SNP policy bits that can be supported by KVM. These include policy bits
70+ * that have implementation support within KVM or policy bits that do not
71+ * require implementation support within KVM to enforce the policy.
72+ */
73+#define KVM_SNP_POLICY_MASK_VALID (SNP_POLICY_MASK_API_MINOR | \
74+ SNP_POLICY_MASK_API_MAJOR | \
75+ SNP_POLICY_MASK_SMT | \
76+ SNP_POLICY_MASK_RSVD_MBO | \
77+ SNP_POLICY_MASK_DEBUG | \
78+ SNP_POLICY_MASK_SINGLE_SOCKET | \
79+ SNP_POLICY_MASK_CXL_ALLOW | \
80+ SNP_POLICY_MASK_MEM_AES_256_XTS | \
81+ SNP_POLICY_MASK_RAPL_DIS | \
82+ SNP_POLICY_MASK_CIPHERTEXT_HIDING_DRAM | \
83+ SNP_POLICY_MASK_PAGE_SWAP_DISABLE)
84+
85+static u64 snp_supported_policy_bits __ro_after_init;
8286
8387 #define INITIAL_VMSA_GPA 0xFFFFFFFFF000
8488
@@ -2143,6 +2147,10 @@ int sev_dev_get_attr(u32 group, u64 attr, u64 *val) · 2058 unmodified lines
21432147 *val = sev_supported_vmsa_features;
21442148 return 0;
21452149
2150+ case KVM_X86_SNP_POLICY_BITS:
2151+ *val = snp_supported_policy_bits;
2152+ return 0;
2153+
21462154 default:
21472155 return -ENXIO;
21482156 }
@@ -2207,7 +2215,7 @@ static int snp_launch_start(struct kvm *kvm, struct kvm_sev_cmd *argp) · 58 unmodified lines
22072215 if (params.flags)
22082216 return -EINVAL;
22092217
2210− if (params.policy & ~SNP_POLICY_MASK_VALID)
2218+ if (params.policy & ~snp_supported_policy_bits)
22112219 return -EINVAL;
22122220
22132221 /* Check for policy bits that must be set */
@@ -3100,8 +3108,11 @@ out: · 886 unmodified lines
31003108 else if (sev_snp_supported)
31013109 sev_snp_supported = is_sev_snp_initialized();
31023110
3103− if (sev_snp_supported)
3111+ if (sev_snp_supported) {
3112+ snp_supported_policy_bits = sev_get_snp_policy_bits() &
3113+ KVM_SNP_POLICY_MASK_VALID;
31043114 nr_ciphertext_hiding_asids = init_args.max_snp_asid;
3115+ }
31053116
31063117 /*
31073118 * If ciphertext hiding is enabled, the joint SEV-ES/SEV-SNP
@@ -5085,10 +5096,10 @@ struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu) · 1977 unmodified lines
50855096
50865097 /* Check if the SEV policy allows debugging */
50875098 if (sev_snp_guest(vcpu->kvm)) {
5088− if (!(sev->policy & SNP_POLICY_DEBUG))
5099+ if (!(sev->policy & SNP_POLICY_MASK_DEBUG))
50895100 return NULL;
50905101 } else {
5091− if (sev->policy & SEV_POLICY_NODBG)
5102+ if (sev->policy & SEV_POLICY_MASK_NODBG)
50925103 return NULL;
50935104 }
50945105
arch/x86/kvm/svm/svm.c+60−43
@@ -272,6 +272,7 @@ static void svm_set_interrupt_shadow(struct kvm_vcpu *vcpu, int mask) · 271 unmodified lines
272272 }
273273
274274 static int __svm_skip_emulated_instruction(struct kvm_vcpu *vcpu,
275+ int emul_type,
275276 bool commit_side_effects)
276277 {
277278 struct vcpu_svm *svm = to_svm(vcpu);
@@ -293,7 +294,7 @@ static int __svm_skip_emulated_instruction(struct kvm_vcpu *vcpu, · 15 unmodified lines
293294 if (unlikely(!commit_side_effects))
294295 old_rflags = svm->vmcb->save.rflags;
295296
296− if (!kvm_emulate_instruction(vcpu, EMULTYPE_SKIP))
297+ if (!kvm_emulate_instruction(vcpu, emul_type))
297298 return 0;
298299
299300 if (unlikely(!commit_side_effects))
@@ -311,11 +312,13 @@ done: · 11 unmodified lines
311312
312313 static int svm_skip_emulated_instruction(struct kvm_vcpu *vcpu)
313314 {
314− return __svm_skip_emulated_instruction(vcpu, true);
315+ return __svm_skip_emulated_instruction(vcpu, EMULTYPE_SKIP, true);
315316 }
316317
317−static int svm_update_soft_interrupt_rip(struct kvm_vcpu *vcpu)
318+static int svm_update_soft_interrupt_rip(struct kvm_vcpu *vcpu, u8 vector)
318319 {
320+ const int emul_type = EMULTYPE_SKIP | EMULTYPE_SKIP_SOFT_INT |
321+ EMULTYPE_SET_SOFT_INT_VECTOR(vector);
319322 unsigned long rip, old_rip = kvm_rip_read(vcpu);
320323 struct vcpu_svm *svm = to_svm(vcpu);
321324
@@ -331,7 +334,7 @@ static int svm_update_soft_interrupt_rip(struct kvm_vcpu *vcpu) · 9 unmodified lines
331334 * in use, the skip must not commit any side effects such as clearing
332335 * the interrupt shadow or RFLAGS.RF.
333336 */
334− if (!__svm_skip_emulated_instruction(vcpu, !nrips))
337+ if (!__svm_skip_emulated_instruction(vcpu, emul_type, !nrips))
335338 return -EIO;
336339
337340 rip = kvm_rip_read(vcpu);
@@ -367,7 +370,7 @@ static void svm_inject_exception(struct kvm_vcpu *vcpu) · 29 unmodified lines
367370 kvm_deliver_exception_payload(vcpu, ex);
368371
369372 if (kvm_exception_is_soft(ex->vector) &&
370− svm_update_soft_interrupt_rip(vcpu))
373+ svm_update_soft_interrupt_rip(vcpu, ex->vector))
371374 return;
372375
373376 svm->vmcb->control.event_inj = ex->vector
@@ -1198,6 +1201,11 @@ void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_vmcb) · 824 unmodified lines
11981201 svm->vmcb = target_vmcb->ptr;
11991202 }
12001203
1204+static int svm_vcpu_precreate(struct kvm *kvm)
1205+{
1206+ return avic_alloc_physical_id_table(kvm);
1207+}
1208+
12011209 static int svm_vcpu_create(struct kvm_vcpu *vcpu)
12021210 {
12031211 struct vcpu_svm *svm;
@@ -3442,13 +3450,8 @@ static bool svm_check_exit_valid(u64 exit_code) · 2238 unmodified lines
34423450
34433451 static int svm_handle_invalid_exit(struct kvm_vcpu *vcpu, u64 exit_code)
34443452 {
3445− vcpu_unimpl(vcpu, "svm: unexpected exit reason 0x%llx\n", exit_code);
34463453 dump_vmcb(vcpu);
3447− vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
3448− vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_UNEXPECTED_EXIT_REASON;
3449− vcpu->run->internal.ndata = 2;
3450− vcpu->run->internal.data[0] = exit_code;
3451− vcpu->run->internal.data[1] = vcpu->arch.last_vmentry_cpu;
3454+ kvm_prepare_unexpected_reason_exit(vcpu, exit_code);
34523455 return 0;
34533456 }
34543457
@@ -3633,11 +3636,12 @@ static bool svm_set_vnmi_pending(struct kvm_vcpu *vcpu) · 178 unmodified lines
36333636
36343637 static void svm_inject_irq(struct kvm_vcpu *vcpu, bool reinjected)
36353638 {
3639+ struct kvm_queued_interrupt *intr = &vcpu->arch.interrupt;
36363640 struct vcpu_svm *svm = to_svm(vcpu);
36373641 u32 type;
36383642
3639− if (vcpu->arch.interrupt.soft) {
3640− if (svm_update_soft_interrupt_rip(vcpu))
3643+ if (intr->soft) {
3644+ if (svm_update_soft_interrupt_rip(vcpu, intr->nr))
36413645 return;
36423646
36433647 type = SVM_EVTINJ_TYPE_SOFT;
@@ -3645,12 +3649,10 @@ static void svm_inject_irq(struct kvm_vcpu *vcpu, bool reinjected) · 1 unmodified line
36453649 type = SVM_EVTINJ_TYPE_INTR;
36463650 }
36473651
3648− trace_kvm_inj_virq(vcpu->arch.interrupt.nr,
3649− vcpu->arch.interrupt.soft, reinjected);
3652+ trace_kvm_inj_virq(intr->nr, intr->soft, reinjected);
36503653 ++vcpu->stat.irq_injections;
36513654
3652− svm->vmcb->control.event_inj = vcpu->arch.interrupt.nr |
3653− SVM_EVTINJ_VALID | type;
3655+ svm->vmcb->control.event_inj = intr->nr | SVM_EVTINJ_VALID | type;
36543656 }
36553657
36563658 void svm_complete_interrupt_delivery(struct kvm_vcpu *vcpu, int delivery_mode,
@@ -4251,7 +4253,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) · 594 unmodified lines
42514253 svm_set_dr6(vcpu, DR6_ACTIVE_LOW);
42524254
42534255 clgi();
4254− kvm_load_guest_xsave_state(vcpu);
42554256
42564257 /*
42574258 * Hardware only context switches DEBUGCTL if LBR virtualization is
@@ -4294,7 +4295,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) · 36 unmodified lines
42944295 vcpu->arch.host_debugctl != svm->vmcb->save.dbgctl)
42954296 update_debugctlmsr(vcpu->arch.host_debugctl);
42964297
4297− kvm_load_host_xsave_state(vcpu);
42984298 stgi();
42994299
43004300 /* Any pending NMI will happen here */
@@ -4326,14 +4326,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) · 25 unmodified lines
43264326
43274327 vcpu->arch.regs_avail &= ~SVM_REGS_LAZY_LOAD_SET;
43284328
4329− /*
4330− * We need to handle MC intercepts here before the vcpu has a chance to
4331− * change the physical cpu
4332− */
4333− if (unlikely(svm->vmcb->control.exit_code ==
4334− SVM_EXIT_EXCP_BASE + MC_VECTOR))
4335− svm_handle_mce(vcpu);
4336−
43374329 trace_kvm_exit(vcpu, KVM_ISA_SVM);
43384330
43394331 svm_complete_interrupts(vcpu);
@@ -4526,31 +4518,45 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu, · 186 unmodified lines
45264518 case SVM_EXIT_WRITE_CR0: {
45274519 unsigned long cr0, val;
45284520
4529− if (info->intercept == x86_intercept_cr_write)
4521+ /*
4522+ * Adjust the exit code accordingly if a CR other than CR0 is
4523+ * being written, and skip straight to the common handling as
4524+ * only CR0 has an additional selective intercept.
4525+ */
4526+ if (info->intercept == x86_intercept_cr_write && info->modrm_reg) {
45304527 icpt_info.exit_code += info->modrm_reg;
4531−
4532− if (icpt_info.exit_code != SVM_EXIT_WRITE_CR0 ||
4533− info->intercept == x86_intercept_clts)
45344528 break;
4529+ }
45354530
4536− if (!(vmcb12_is_intercept(&svm->nested.ctl,
4537− INTERCEPT_SELECTIVE_CR0)))
4531+ /*
4532+ * Convert the exit_code to SVM_EXIT_CR0_SEL_WRITE if a
4533+ * selective CR0 intercept is triggered (the common logic will
4534+ * treat the selective intercept as being enabled). Note, the
4535+ * unconditional intercept has higher priority, i.e. this is
4536+ * only relevant if *only* the selective intercept is enabled.
4537+ */
4538+ if (vmcb12_is_intercept(&svm->nested.ctl, INTERCEPT_CR0_WRITE) ||
4539+ !(vmcb12_is_intercept(&svm->nested.ctl, INTERCEPT_SELECTIVE_CR0)))
45384540 break;
45394541
4540− cr0 = vcpu->arch.cr0 & ~SVM_CR0_SELECTIVE_MASK;
4541− val = info->src_val & ~SVM_CR0_SELECTIVE_MASK;
4542+ /* CLTS never triggers INTERCEPT_SELECTIVE_CR0 */
4543+ if (info->intercept == x86_intercept_clts)
4544+ break;
45424545
4546+ /* LMSW always triggers INTERCEPT_SELECTIVE_CR0 */
45434547 if (info->intercept == x86_intercept_lmsw) {
4544− cr0 &= 0xfUL;
4545− val &= 0xfUL;
4546− /* lmsw can't clear PE - catch this here */
4547− if (cr0 & X86_CR0_PE)
4548− val |= X86_CR0_PE;
4548+ icpt_info.exit_code = SVM_EXIT_CR0_SEL_WRITE;
4549+ break;
45494550 }
45504551
4552+ /*
4553+ * MOV-to-CR0 only triggers INTERCEPT_SELECTIVE_CR0 if any bit
4554+ * other than SVM_CR0_SELECTIVE_MASK is changed.
4555+ */
4556+ cr0 = vcpu->arch.cr0 & ~SVM_CR0_SELECTIVE_MASK;
4557+ val = info->src_val & ~SVM_CR0_SELECTIVE_MASK;
45514558 if (cr0 ^ val)
45524559 icpt_info.exit_code = SVM_EXIT_CR0_SEL_WRITE;
4553−
45544560 break;
45554561 }
45564562 case SVM_EXIT_READ_DR0:
@@ -4622,8 +4628,16 @@ out: · 65 unmodified lines
46224628
46234629 static void svm_handle_exit_irqoff(struct kvm_vcpu *vcpu)
46244630 {
4625− if (to_svm(vcpu)->vmcb->control.exit_code == SVM_EXIT_INTR)
4631+ switch (to_svm(vcpu)->vmcb->control.exit_code) {
4632+ case SVM_EXIT_EXCP_BASE + MC_VECTOR:
4633+ svm_handle_mce(vcpu);
4634+ break;
4635+ case SVM_EXIT_INTR:
46264636 vcpu->arch.at_instruction_boundary = true;
4637+ break;
4638+ default:
4639+ break;
4640+ }
46274641 }
46284642
46294643 static void svm_setup_mce(struct kvm_vcpu *vcpu)
@@ -5012,6 +5026,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = { · 382 unmodified lines
50125026 .emergency_disable_virtualization_cpu = svm_emergency_disable_virtualization_cpu,
50135027 .has_emulated_msr = svm_has_emulated_msr,
50145028
5029+ .vcpu_precreate = svm_vcpu_precreate,
50155030 .vcpu_create = svm_vcpu_create,
50165031 .vcpu_free = svm_vcpu_free,
50175032 .vcpu_reset = svm_vcpu_reset,
@@ -5316,7 +5331,9 @@ static __init int svm_hardware_setup(void) · 298 unmodified lines
53165331
53175332 if (nested) {
53185333 pr_info("Nested Virtualization enabled\n");
5319− kvm_enable_efer_bits(EFER_SVME | EFER_LMSLE);
5334+ kvm_enable_efer_bits(EFER_SVME);
5335+ if (!boot_cpu_has(X86_FEATURE_EFER_LMSLE_MBZ))
5336+ kvm_enable_efer_bits(EFER_LMSLE);
53205337
53215338 r = nested_svm_init_msrpm_merge_offsets();
53225339 if (r)
arch/x86/kvm/svm/svm.h+1−3
@@ -117,9 +117,6 @@ struct kvm_sev_info { · 116 unmodified lines
117117 cpumask_var_t have_run_cpus; /* CPUs that have done VMRUN for this VM. */
118118 };
119119
120−#define SEV_POLICY_NODBG BIT_ULL(0)
121−#define SNP_POLICY_DEBUG BIT_ULL(19)
122−
123120 struct kvm_svm {
124121 struct kvm kvm;
125122
@@ -807,6 +804,7 @@ extern struct kvm_x86_nested_ops svm_nested_ops; · 681 unmodified lines
807804
808805 bool __init avic_hardware_setup(void);
809806 void avic_hardware_unsetup(void);
807+int avic_alloc_physical_id_table(struct kvm *kvm);
810808 void avic_vm_destroy(struct kvm *kvm);
811809 int avic_vm_init(struct kvm *kvm);
812810 void avic_init_vmcb(struct vcpu_svm *svm, struct vmcb *vmcb);
arch/x86/kvm/svm/vmenter.S+41−12
@@ -52,11 +52,23 @@ · 51 unmodified lines
5252 * there must not be any returns or indirect branches between this code
5353 * and vmentry.
5454 */
55− movl SVM_spec_ctrl(%_ASM_DI), %eax
56− cmp PER_CPU_VAR(x86_spec_ctrl_current), %eax
55+#ifdef CONFIG_X86_64
56+ mov SVM_spec_ctrl(%rdi), %rdx
57+ cmp PER_CPU_VAR(x86_spec_ctrl_current), %rdx
58+ je 801b
59+ movl %edx, %eax
60+ shr $32, %rdx
61+#else
62+ mov SVM_spec_ctrl(%edi), %eax
63+ mov PER_CPU_VAR(x86_spec_ctrl_current), %ecx
64+ xor %eax, %ecx
65+ mov SVM_spec_ctrl + 4(%edi), %edx
66+ mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %esi
67+ xor %edx, %esi
68+ or %esi, %ecx
5769 je 801b
70+#endif
5871 mov $MSR_IA32_SPEC_CTRL, %ecx
59− xor %edx, %edx
6072 wrmsr
6173 jmp 801b
6274 .endm
@@ -81,17 +93,31 @@ · 18 unmodified lines
8193 jnz 998f
8294 rdmsr
8395 movl %eax, SVM_spec_ctrl(%_ASM_DI)
96+ movl %edx, SVM_spec_ctrl + 4(%_ASM_DI)
8497 998:
85−
8698 /* Now restore the host value of the MSR if different from the guest's. */
87− movl PER_CPU_VAR(x86_spec_ctrl_current), %eax
88− cmp SVM_spec_ctrl(%_ASM_DI), %eax
99+#ifdef CONFIG_X86_64
100+ mov PER_CPU_VAR(x86_spec_ctrl_current), %rdx
101+ cmp SVM_spec_ctrl(%rdi), %rdx
89102 je 901b
90− xor %edx, %edx
103+ movl %edx, %eax
104+ shr $32, %rdx
105+#else
106+ mov PER_CPU_VAR(x86_spec_ctrl_current), %eax
107+ mov SVM_spec_ctrl(%edi), %esi
108+ xor %eax, %esi
109+ mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %edx
110+ mov SVM_spec_ctrl + 4(%edi), %edi
111+ xor %edx, %edi
112+ or %edi, %esi
113+ je 901b
114+#endif
91115 wrmsr
92116 jmp 901b
93117 .endm
94118
119+#define SVM_CLEAR_CPU_BUFFERS \
120+ ALTERNATIVE "", __CLEAR_CPU_BUFFERS, X86_FEATURE_CLEAR_CPU_BUF_VM
95121
96122 /**
97123 * __svm_vcpu_run - Run a vCPU via a transition to SVM guest mode
@@ -134,7 +160,7 @@ SYM_FUNC_START(__svm_vcpu_run) · 36 unmodified lines
134160 mov %_ASM_ARG1, %_ASM_DI
135161 .endif
136162
137− /* Clobbers RAX, RCX, RDX. */
163+ /* Clobbers RAX, RCX, RDX (and ESI on 32-bit), consumes RDI (@svm). */
138164 RESTORE_GUEST_SPEC_CTRL
139165
140166 /*
@@ -170,7 +196,7 @@ SYM_FUNC_START(__svm_vcpu_run) · 29 unmodified lines
170196 mov VCPU_RDI(%_ASM_DI), %_ASM_DI
171197
172198 /* Clobbers EFLAGS.ZF */
173− VM_CLEAR_CPU_BUFFERS
199+ SVM_CLEAR_CPU_BUFFERS
174200
175201 /* Enter guest mode */
176202 3: vmrun %_ASM_AX
@@ -211,7 +237,10 @@ SYM_FUNC_START(__svm_vcpu_run) · 34 unmodified lines
211237 /* IMPORTANT: Stuff the RSB immediately after VM-Exit, before RET! */
212238 FILL_RETURN_BUFFER %_ASM_AX, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_VMEXIT
213239
214− /* Clobbers RAX, RCX, RDX. */
240+ /*
241+ * Clobbers RAX, RCX, RDX (and ESI, EDI on 32-bit), consumes RDI (@svm)
242+ * and RSP (pointer to @spec_ctrl_intercepted).
243+ */
215244 RESTORE_HOST_SPEC_CTRL
216245
217246 /*
@@ -331,7 +360,7 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) · 113 unmodified lines
331360 mov %rdi, SEV_ES_RDI (%rdx)
332361 mov %rsi, SEV_ES_RSI (%rdx)
333362
334− /* Clobbers RAX, RCX, RDX (@hostsa). */
363+ /* Clobbers RAX, RCX, and RDX (@hostsa), consumes RDI (@svm). */
335364 RESTORE_GUEST_SPEC_CTRL
336365
337366 /* Get svm->current_vmcb->pa into RAX. */
@@ -339,7 +368,7 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) · 1 unmodified line
339368 mov KVM_VMCB_pa(%rax), %rax
340369
341370 /* Clobbers EFLAGS.ZF */
342− VM_CLEAR_CPU_BUFFERS
371+ SVM_CLEAR_CPU_BUFFERS
343372
344373 /* Enter guest mode */
345374 1: vmrun %rax
arch/x86/kvm/vmx/main.c+9−0
@@ -831,6 +831,14 @@ static int vt_vcpu_mem_enc_ioctl(struct kvm_vcpu *vcpu, void __user *argp) · 830 unmodified lines
831831 return tdx_vcpu_ioctl(vcpu, argp);
832832 }
833833
834+static int vt_vcpu_mem_enc_unlocked_ioctl(struct kvm_vcpu *vcpu, void __user *argp)
835+{
836+ if (!is_td_vcpu(vcpu))
837+ return -EINVAL;
838+
839+ return tdx_vcpu_unlocked_ioctl(vcpu, argp);
840+}
841+
834842 static int vt_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn,
835843 bool is_private)
836844 {
@@ -1005,6 +1013,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { · 168 unmodified lines
10051013
10061014 .mem_enc_ioctl = vt_op_tdx_only(mem_enc_ioctl),
10071015 .vcpu_mem_enc_ioctl = vt_op_tdx_only(vcpu_mem_enc_ioctl),
1016+ .vcpu_mem_enc_unlocked_ioctl = vt_op_tdx_only(vcpu_mem_enc_unlocked_ioctl),
10081017
10091018 .gmem_max_mapping_level = vt_op_tdx_only(gmem_max_mapping_level)
10101019 };
arch/x86/kvm/vmx/nested.c+61−112
@@ -23,8 +23,8 @@ · 22 unmodified lines
2323 static bool __read_mostly enable_shadow_vmcs = 1;
2424 module_param_named(enable_shadow_vmcs, enable_shadow_vmcs, bool, S_IRUGO);
2525
26−static bool __read_mostly nested_early_check = 0;
27−module_param(nested_early_check, bool, S_IRUGO);
26+static bool __ro_after_init warn_on_missed_cc;
27+module_param(warn_on_missed_cc, bool, 0444);
2828
2929 #define CC KVM_NESTED_VMENTER_CONSISTENCY_CHECK
3030
@@ -555,6 +555,9 @@ static int nested_vmx_check_tpr_shadow_controls(struct kvm_vcpu *vcpu, · 524 unmodified lines
555555 if (CC(!page_address_valid(vcpu, vmcs12->virtual_apic_page_addr)))
556556 return -EINVAL;
557557
558+ if (CC(!nested_cpu_has_vid(vmcs12) && vmcs12->tpr_threshold >> 4))
559+ return -EINVAL;
560+
558561 return 0;
559562 }
560563
@@ -761,7 +764,7 @@ static void nested_cache_shadow_vmcs12(struct kvm_vcpu *vcpu, · 200 unmodified lines
761764 vmcs12->vmcs_link_pointer, VMCS12_SIZE))
762765 return;
763766
764− kvm_read_guest_cached(vmx->vcpu.kvm, ghc, get_shadow_vmcs12(vcpu),
767+ kvm_read_guest_cached(vcpu->kvm, ghc, get_shadow_vmcs12(vcpu),
765768 VMCS12_SIZE);
766769 }
767770
@@ -780,7 +783,7 @@ static void nested_flush_cached_shadow_vmcs12(struct kvm_vcpu *vcpu, · 12 unmodified lines
780783 vmcs12->vmcs_link_pointer, VMCS12_SIZE))
781784 return;
782785
783− kvm_write_guest_cached(vmx->vcpu.kvm, ghc, get_shadow_vmcs12(vcpu),
786+ kvm_write_guest_cached(vcpu->kvm, ghc, get_shadow_vmcs12(vcpu),
784787 VMCS12_SIZE);
785788 }
786789
@@ -2296,15 +2299,6 @@ static void prepare_vmcs02_constant_state(struct vcpu_vmx *vmx) · 1509 unmodified lines
22962299 return;
22972300 vmx->nested.vmcs02_initialized = true;
22982301
2299− /*
2300− * We don't care what the EPTP value is we just need to guarantee
2301− * it's valid so we don't get a false positive when doing early
2302− * consistency checks.
2303− */
2304− if (enable_ept && nested_early_check)
2305− vmcs_write64(EPT_POINTER,
2306− construct_eptp(&vmx->vcpu, 0, PT64_ROOT_4LEVEL));
2307−
23082302 if (vmx->ve_info)
23092303 vmcs_write64(VE_INFORMATION_ADDRESS, __pa(vmx->ve_info));
23102304
@@ -2749,7 +2743,7 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, · 438 unmodified lines
27492743 vmcs_write64(GUEST_IA32_PAT, vmcs12->guest_ia32_pat);
27502744 vcpu->arch.pat = vmcs12->guest_ia32_pat;
27512745 } else if (vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_PAT) {
2752− vmcs_write64(GUEST_IA32_PAT, vmx->vcpu.arch.pat);
2746+ vmcs_write64(GUEST_IA32_PAT, vcpu->arch.pat);
27532747 }
27542748
27552749 vcpu->arch.tsc_offset = kvm_calc_nested_tsc_offset(
@@ -2961,6 +2955,10 @@ static int nested_check_vm_execution_controls(struct kvm_vcpu *vcpu, · 205 unmodified lines
29612955 }
29622956 }
29632957
2958+ if (nested_cpu_has2(vmcs12, SECONDARY_EXEC_TSC_SCALING) &&
2959+ CC(!vmcs12->tsc_multiplier))
2960+ return -EINVAL;
2961+
29642962 return 0;
29652963 }
29662964
@@ -3078,6 +3076,38 @@ static int nested_vmx_check_controls(struct kvm_vcpu *vcpu, · 111 unmodified lines
30783076 return 0;
30793077 }
30803078
3079+static int nested_vmx_check_controls_late(struct kvm_vcpu *vcpu,
3080+ struct vmcs12 *vmcs12)
3081+{
3082+ void *vapic = to_vmx(vcpu)->nested.virtual_apic_map.hva;
3083+ u32 vtpr = vapic ? (*(u32 *)(vapic + APIC_TASKPRI)) >> 4 : 0;
3084+
3085+ /*
3086+ * Don't bother with the consistency checks if KVM isn't configured to
3087+ * WARN on missed consistency checks, as KVM needs to rely on hardware
3088+ * to fully detect an illegal vTPR vs. TRP Threshold combination due to
3089+ * the vTPR being writable by L1 at all times (it's an in-memory value,
3090+ * not a VMCS field). I.e. even if the check passes now, it might fail
3091+ * at the actual VM-Enter.
3092+ *
3093+ * Keying off the module param also allows treating an invalid vAPIC
3094+ * mapping as a consistency check failure without increasing the risk
3095+ * of breaking a "real" VM.
3096+ */
3097+ if (!warn_on_missed_cc)
3098+ return 0;
3099+
3100+ if ((exec_controls_get(to_vmx(vcpu)) & CPU_BASED_TPR_SHADOW) &&
3101+ nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW) &&
3102+ !nested_cpu_has_vid(vmcs12) &&
3103+ !nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES) &&
3104+ (CC(!vapic) ||
3105+ CC((vmcs12->tpr_threshold & GENMASK(3, 0)) > (vtpr & GENMASK(3, 0)))))
3106+ return -EINVAL;
3107+
3108+ return 0;
3109+}
3110+
30813111 static int nested_vmx_check_address_space_size(struct kvm_vcpu *vcpu,
30823112 struct vmcs12 *vmcs12)
30833113 {
@@ -3333,84 +3363,6 @@ static int nested_vmx_check_guest_state(struct kvm_vcpu *vcpu, · 249 unmodified lines
33333363 return 0;
33343364 }
33353365
3336−static int nested_vmx_check_vmentry_hw(struct kvm_vcpu *vcpu)
3337−{
3338− struct vcpu_vmx *vmx = to_vmx(vcpu);
3339− unsigned long cr3, cr4;
3340− bool vm_fail;
3341−
3342− if (!nested_early_check)
3343− return 0;
3344−
3345− if (vmx->msr_autoload.host.nr)
3346− vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, 0);
3347− if (vmx->msr_autoload.guest.nr)
3348− vmcs_write32(VM_ENTRY_MSR_LOAD_COUNT, 0);
3349−
3350− preempt_disable();
3351−
3352− vmx_prepare_switch_to_guest(vcpu);
3353−
3354− /*
3355− * Induce a consistency check VMExit by clearing bit 1 in GUEST_RFLAGS,
3356− * which is reserved to '1' by hardware. GUEST_RFLAGS is guaranteed to
3357− * be written (by prepare_vmcs02()) before the "real" VMEnter, i.e.
3358− * there is no need to preserve other bits or save/restore the field.
3359− */
3360− vmcs_writel(GUEST_RFLAGS, 0);
3361−
3362− cr3 = __get_current_cr3_fast();
3363− if (unlikely(cr3 != vmx->loaded_vmcs->host_state.cr3)) {
3364− vmcs_writel(HOST_CR3, cr3);
3365− vmx->loaded_vmcs->host_state.cr3 = cr3;
3366− }
3367−
3368− cr4 = cr4_read_shadow();
3369− if (unlikely(cr4 != vmx->loaded_vmcs->host_state.cr4)) {
3370− vmcs_writel(HOST_CR4, cr4);
3371− vmx->loaded_vmcs->host_state.cr4 = cr4;
3372− }
3373−
3374− vm_fail = __vmx_vcpu_run(vmx, (unsigned long *)&vcpu->arch.regs,
3375− __vmx_vcpu_run_flags(vmx));
3376−
3377− if (vmx->msr_autoload.host.nr)
3378− vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr);
3379− if (vmx->msr_autoload.guest.nr)
3380− vmcs_write32(VM_ENTRY_MSR_LOAD_COUNT, vmx->msr_autoload.guest.nr);
3381−
3382− if (vm_fail) {
3383− u32 error = vmcs_read32(VM_INSTRUCTION_ERROR);
3384−
3385− preempt_enable();
3386−
3387− trace_kvm_nested_vmenter_failed(
3388− "early hardware check VM-instruction error: ", error);
3389− WARN_ON_ONCE(error != VMXERR_ENTRY_INVALID_CONTROL_FIELD);
3390− return 1;
3391− }
3392−
3393− /*
3394− * VMExit clears RFLAGS.IF and DR7, even on a consistency check.
3395− */
3396− if (hw_breakpoint_active())
3397− set_debugreg(__this_cpu_read(cpu_dr7), 7);
3398− local_irq_enable();
3399− preempt_enable();
3400−
3401− /*
3402− * A non-failing VMEntry means we somehow entered guest mode with
3403− * an illegal RIP, and that's just the tip of the iceberg. There
3404− * is no telling what memory has been modified or what state has
3405− * been exposed to unknown code. Hitting this all but guarantees
3406− * a (very critical) hardware issue.
3407− */
3408− WARN_ON(!(vmcs_read32(VM_EXIT_REASON) &
3409− VMX_EXIT_REASONS_FAILED_VMENTRY));
3410−
3411− return 0;
3412−}
3413−
34143366 #ifdef CONFIG_KVM_HYPERV
34153367 static bool nested_get_evmcs_page(struct kvm_vcpu *vcpu)
34163368 {
@@ -3667,22 +3619,18 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, · 250 unmodified lines
36673619 &vmx->nested.pre_vmenter_ssp_tbl);
36683620
36693621 /*
3670− * Overwrite vmcs01.GUEST_CR3 with L1's CR3 if EPT is disabled *and*
3671− * nested early checks are disabled. In the event of a "late" VM-Fail,
3672− * i.e. a VM-Fail detected by hardware but not KVM, KVM must unwind its
3673− * software model to the pre-VMEntry host state. When EPT is disabled,
3674− * GUEST_CR3 holds KVM's shadow CR3, not L1's "real" CR3, which causes
3675− * nested_vmx_restore_host_state() to corrupt vcpu->arch.cr3. Stuffing
3676− * vmcs01.GUEST_CR3 results in the unwind naturally setting arch.cr3 to
3677− * the correct value. Smashing vmcs01.GUEST_CR3 is safe because nested
3678− * VM-Exits, and the unwind, reset KVM's MMU, i.e. vmcs01.GUEST_CR3 is
3679− * guaranteed to be overwritten with a shadow CR3 prior to re-entering
3680− * L1. Don't stuff vmcs01.GUEST_CR3 when using nested early checks as
3681− * KVM modifies vcpu->arch.cr3 if and only if the early hardware checks
3682− * pass, and early VM-Fails do not reset KVM's MMU, i.e. the VM-Fail
3683− * path would need to manually save/restore vmcs01.GUEST_CR3.
3622+ * Overwrite vmcs01.GUEST_CR3 with L1's CR3 if EPT is disabled. In the
3623+ * event of a "late" VM-Fail, i.e. a VM-Fail detected by hardware but
3624+ * not KVM, KVM must unwind its software model to the pre-VM-Entry host
3625+ * state. When EPT is disabled, GUEST_CR3 holds KVM's shadow CR3, not
3626+ * L1's "real" CR3, which causes nested_vmx_restore_host_state() to
3627+ * corrupt vcpu->arch.cr3. Stuffing vmcs01.GUEST_CR3 results in the
3628+ * unwind naturally setting arch.cr3 to the correct value. Smashing
3629+ * vmcs01.GUEST_CR3 is safe because nested VM-Exits, and the unwind,
3630+ * reset KVM's MMU, i.e. vmcs01.GUEST_CR3 is guaranteed to be
3631+ * overwritten with a shadow CR3 prior to re-entering L1.
36843632 */
3685− if (!enable_ept && !nested_early_check)
3633+ if (!enable_ept)
36863634 vmcs_writel(GUEST_CR3, vcpu->arch.cr3);
36873635
36883636 vmx_switch_vmcs(vcpu, &vmx->nested.vmcs02);
@@ -3695,7 +3643,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, · 6 unmodified lines
36953643 return NVMX_VMENTRY_KVM_INTERNAL_ERROR;
36963644 }
36973645
3698− if (nested_vmx_check_vmentry_hw(vcpu)) {
3646+ if (nested_vmx_check_controls_late(vcpu, vmcs12)) {
36993647 vmx_switch_vmcs(vcpu, &vmx->vmcs01);
37003648 return NVMX_VMENTRY_VMFAIL;
37013649 }
@@ -3880,7 +3828,7 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch) · 178 unmodified lines
38803828 goto vmentry_failed;
38813829
38823830 /* Hide L1D cache contents from the nested guest. */
3883− vmx->vcpu.arch.l1tf_flush_l1d = true;
3831+ kvm_request_l1tf_flush_l1d();
38843832
38853833 /*
38863834 * Must happen outside of nested_vmx_enter_non_root_mode() as it will
@@ -5164,12 +5112,13 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason, · 1277 unmodified lines
51645112 /*
51655113 * The only expected VM-instruction error is "VM entry with
51665114 * invalid control field(s)." Anything else indicates a
5167− * problem with L0. And we should never get here with a
5168− * VMFail of any type if early consistency checks are enabled.
5115+ * problem with L0.
51695116 */
51705117 WARN_ON_ONCE(vmcs_read32(VM_INSTRUCTION_ERROR) !=
51715118 VMXERR_ENTRY_INVALID_CONTROL_FIELD);
5172− WARN_ON_ONCE(nested_early_check);
5119+
5120+ /* VM-Fail at VM-Entry means KVM missed a consistency check. */
5121+ WARN_ON_ONCE(warn_on_missed_cc);
51735122 }
51745123
51755124 /*
arch/x86/kvm/vmx/run_flags.h+3−7
@@ -2,12 +2,8 @@ · 1 unmodified line
22 #ifndef __KVM_X86_VMX_RUN_FLAGS_H
33 #define __KVM_X86_VMX_RUN_FLAGS_H
44
5−#define VMX_RUN_VMRESUME_SHIFT 0
6−#define VMX_RUN_SAVE_SPEC_CTRL_SHIFT 1
7−#define VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO_SHIFT 2
8−
9−#define VMX_RUN_VMRESUME BIT(VMX_RUN_VMRESUME_SHIFT)
10−#define VMX_RUN_SAVE_SPEC_CTRL BIT(VMX_RUN_SAVE_SPEC_CTRL_SHIFT)
11−#define VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO BIT(VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO_SHIFT)
5+#define VMX_RUN_VMRESUME BIT(0)
6+#define VMX_RUN_SAVE_SPEC_CTRL BIT(1)
7+#define VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO BIT(2)
128
139 #endif /* __KVM_X86_VMX_RUN_FLAGS_H */
arch/x86/kvm/vmx/tdx.c+380−425
@@ -24,20 +24,33 @@ · 23 unmodified lines
2424 #undef pr_fmt
2525 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
2626
27−#define pr_tdx_error(__fn, __err) \
28− pr_err_ratelimited("SEAMCALL %s failed: 0x%llx\n", #__fn, __err)
27+#define __TDX_BUG_ON(__err, __f, __kvm, __fmt, __args...) \
28+({ \
29+ struct kvm *_kvm = (__kvm); \
30+ bool __ret = !!(__err); \
31+ \
32+ if (WARN_ON_ONCE(__ret && (!_kvm || !_kvm->vm_bugged))) { \
33+ if (_kvm) \
34+ kvm_vm_bugged(_kvm); \
35+ pr_err_ratelimited("SEAMCALL " __f " failed: 0x%llx" __fmt "\n",\
36+ __err, __args); \
37+ } \
38+ unlikely(__ret); \
39+})
40+
41+#define TDX_BUG_ON(__err, __fn, __kvm) \
42+ __TDX_BUG_ON(__err, #__fn, __kvm, "%s", "")
43+
44+#define TDX_BUG_ON_1(__err, __fn, a1, __kvm) \
45+ __TDX_BUG_ON(__err, #__fn, __kvm, ", " #a1 " 0x%llx", a1)
46+
47+#define TDX_BUG_ON_2(__err, __fn, a1, a2, __kvm) \
48+ __TDX_BUG_ON(__err, #__fn, __kvm, ", " #a1 " 0x%llx, " #a2 " 0x%llx", a1, a2)
49+
50+#define TDX_BUG_ON_3(__err, __fn, a1, a2, a3, __kvm) \
51+ __TDX_BUG_ON(__err, #__fn, __kvm, ", " #a1 " 0x%llx, " #a2 ", 0x%llx, " #a3 " 0x%llx", \
52+ a1, a2, a3)
2953
30−#define __pr_tdx_error_N(__fn_str, __err, __fmt, ...) \
31− pr_err_ratelimited("SEAMCALL " __fn_str " failed: 0x%llx, " __fmt, __err, __VA_ARGS__)
32−
33−#define pr_tdx_error_1(__fn, __err, __rcx) \
34− __pr_tdx_error_N(#__fn, __err, "rcx 0x%llx\n", __rcx)
35−
36−#define pr_tdx_error_2(__fn, __err, __rcx, __rdx) \
37− __pr_tdx_error_N(#__fn, __err, "rcx 0x%llx, rdx 0x%llx\n", __rcx, __rdx)
38−
39−#define pr_tdx_error_3(__fn, __err, __rcx, __rdx, __r8) \
40− __pr_tdx_error_N(#__fn, __err, "rcx 0x%llx, rdx 0x%llx, r8 0x%llx\n", __rcx, __rdx, __r8)
4154
4255 bool enable_tdx __ro_after_init;
4356 module_param_named(tdx, enable_tdx, bool, 0444);
@@ -281,25 +294,34 @@ static inline void tdx_disassociate_vp(struct kvm_vcpu *vcpu) · 237 unmodified lines
281294 vcpu->cpu = -1;
282295 }
283296
284−static void tdx_no_vcpus_enter_start(struct kvm *kvm)
285−{
286− struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
287−
288− lockdep_assert_held_write(&kvm->mmu_lock);
289−
290− WRITE_ONCE(kvm_tdx->wait_for_sept_zap, true);
291−
292− kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE);
293−}
294−
295−static void tdx_no_vcpus_enter_stop(struct kvm *kvm)
296−{
297− struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
298−
299− lockdep_assert_held_write(&kvm->mmu_lock);
300−
301− WRITE_ONCE(kvm_tdx->wait_for_sept_zap, false);
302−}
297+/*
298+ * Execute a SEAMCALL related to removing/blocking S-EPT entries, with a single
299+ * retry (if necessary) after forcing vCPUs to exit and wait for the operation
300+ * to complete. All flows that remove/block S-EPT entries run with mmu_lock
301+ * held for write, i.e. are mutually exclusive with each other, but they aren't
302+ * mutually exclusive with running vCPUs, and so can fail with "operand busy"
303+ * if a vCPU acquires a relevant lock in the TDX-Module, e.g. when doing TDCALL.
304+ *
305+ * Note, the retry is guaranteed to succeed, absent KVM and/or TDX-Module bugs.
306+ */
307+#define tdh_do_no_vcpus(tdh_func, kvm, args...) \
308+({ \
309+ struct kvm_tdx *__kvm_tdx = to_kvm_tdx(kvm); \
310+ u64 __err; \
311+ \
312+ lockdep_assert_held_write(&kvm->mmu_lock); \
313+ \
314+ __err = tdh_func(args); \
315+ if (unlikely(tdx_operand_busy(__err))) { \
316+ WRITE_ONCE(__kvm_tdx->wait_for_sept_zap, true); \
317+ kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE); \
318+ \
319+ __err = tdh_func(args); \
320+ \
321+ WRITE_ONCE(__kvm_tdx->wait_for_sept_zap, false); \
322+ } \
323+ __err; \
324+})
303325
304326 /* TDH.PHYMEM.PAGE.RECLAIM is allowed only when destroying the TD. */
305327 static int __tdx_reclaim_page(struct page *page)
@@ -313,10 +335,9 @@ static int __tdx_reclaim_page(struct page *page) · 7 unmodified lines
313335 * before the HKID is released and control pages have also been
314336 * released at this point, so there is no possibility of contention.
315337 */
316− if (WARN_ON_ONCE(err)) {
317− pr_tdx_error_3(TDH_PHYMEM_PAGE_RECLAIM, err, rcx, rdx, r8);
338+ if (TDX_BUG_ON_3(err, TDH_PHYMEM_PAGE_RECLAIM, rcx, rdx, r8, NULL))
318339 return -EIO;
319− }
340+
320341 return 0;
321342 }
322343
@@ -404,8 +425,8 @@ static void tdx_flush_vp_on_cpu(struct kvm_vcpu *vcpu) · 81 unmodified lines
404425 return;
405426
406427 smp_call_function_single(cpu, tdx_flush_vp, &arg, 1);
407− if (KVM_BUG_ON(arg.err, vcpu->kvm))
408− pr_tdx_error(TDH_VP_FLUSH, arg.err);
428+
429+ TDX_BUG_ON(arg.err, TDH_VP_FLUSH, vcpu->kvm);
409430 }
410431
411432 void tdx_disable_virtualization_cpu(void)
@@ -464,8 +485,7 @@ static void smp_func_do_phymem_cache_wb(void *unused) · 52 unmodified lines
464485 }
465486
466487 out:
467− if (WARN_ON_ONCE(err))
468− pr_tdx_error(TDH_PHYMEM_CACHE_WB, err);
488+ TDX_BUG_ON(err, TDH_PHYMEM_CACHE_WB, NULL);
469489 }
470490
471491 void tdx_mmu_release_hkid(struct kvm *kvm)
@@ -504,8 +524,7 @@ void tdx_mmu_release_hkid(struct kvm *kvm) · 32 unmodified lines
504524 err = tdh_mng_vpflushdone(&kvm_tdx->td);
505525 if (err == TDX_FLUSHVP_NOT_DONE)
506526 goto out;
507− if (KVM_BUG_ON(err, kvm)) {
508− pr_tdx_error(TDH_MNG_VPFLUSHDONE, err);
527+ if (TDX_BUG_ON(err, TDH_MNG_VPFLUSHDONE, kvm)) {
509528 pr_err("tdh_mng_vpflushdone() failed. HKID %d is leaked.\n",
510529 kvm_tdx->hkid);
511530 goto out;
@@ -528,8 +547,7 @@ void tdx_mmu_release_hkid(struct kvm *kvm) · 16 unmodified lines
528547 * tdh_mng_key_freeid() will fail.
529548 */
530549 err = tdh_mng_key_freeid(&kvm_tdx->td);
531− if (KVM_BUG_ON(err, kvm)) {
532− pr_tdx_error(TDH_MNG_KEY_FREEID, err);
550+ if (TDX_BUG_ON(err, TDH_MNG_KEY_FREEID, kvm)) {
533551 pr_err("tdh_mng_key_freeid() failed. HKID %d is leaked.\n",
534552 kvm_tdx->hkid);
535553 } else {
@@ -580,10 +598,9 @@ static void tdx_reclaim_td_control_pages(struct kvm *kvm) · 44 unmodified lines
580598 * when it is reclaiming TDCS).
581599 */
582600 err = tdh_phymem_page_wbinvd_tdr(&kvm_tdx->td);
583− if (KVM_BUG_ON(err, kvm)) {
584− pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err);
601+ if (TDX_BUG_ON(err, TDH_PHYMEM_PAGE_WBINVD, kvm))
585602 return;
586− }
603+
587604 tdx_quirk_reset_page(kvm_tdx->td.tdr_page);
588605
589606 __free_page(kvm_tdx->td.tdr_page);
@@ -606,11 +623,8 @@ static int tdx_do_tdh_mng_key_config(void *param) · 16 unmodified lines
606623
607624 /* TDX_RND_NO_ENTROPY related retries are handled by sc_retry() */
608625 err = tdh_mng_key_config(&kvm_tdx->td);
609−
610− if (KVM_BUG_ON(err, &kvm_tdx->kvm)) {
611− pr_tdx_error(TDH_MNG_KEY_CONFIG, err);
626+ if (TDX_BUG_ON(err, TDH_MNG_KEY_CONFIG, &kvm_tdx->kvm))
612627 return -EIO;
613− }
614628
615629 return 0;
616630 }
@@ -763,25 +777,6 @@ static bool tdx_protected_apic_has_interrupt(struct kvm_vcpu *vcpu) · 146 unmodified lines
763777 return tdx_vcpu_state_details_intr_pending(vcpu_state_details);
764778 }
765779
766−/*
767− * Compared to vmx_prepare_switch_to_guest(), there is not much to do
768− * as SEAMCALL/SEAMRET calls take care of most of save and restore.
769− */
770−void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu)
771−{
772− struct vcpu_vt *vt = to_vt(vcpu);
773−
774− if (vt->guest_state_loaded)
775− return;
776−
777− if (likely(is_64bit_mm(current->mm)))
778− vt->msr_host_kernel_gs_base = current->thread.gsbase;
779− else
780− vt->msr_host_kernel_gs_base = read_msr(MSR_KERNEL_GS_BASE);
781−
782− vt->guest_state_loaded = true;
783−}
784−
785780 struct tdx_uret_msr {
786781 u32 msr;
787782 unsigned int slot;
@@ -795,19 +790,38 @@ static struct tdx_uret_msr tdx_uret_msrs[] = { · 7 unmodified lines
795790 {.msr = MSR_TSC_AUX,},
796791 };
797792
798−static void tdx_user_return_msr_update_cache(void)
793+void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu)
799794 {
795+ struct vcpu_vt *vt = to_vt(vcpu);
800796 int i;
801797
798+ if (vt->guest_state_loaded)
799+ return;
800+
801+ if (likely(is_64bit_mm(current->mm)))
802+ vt->msr_host_kernel_gs_base = current->thread.gsbase;
803+ else
804+ vt->msr_host_kernel_gs_base = read_msr(MSR_KERNEL_GS_BASE);
805+
806+ vt->guest_state_loaded = true;
807+
808+ /*
809+ * Explicitly set user-return MSRs that are clobbered by the TDX-Module
810+ * if VP.ENTER succeeds, i.e. on TD-Exit, with the values that would be
811+ * written by the TDX-Module. Don't rely on the TDX-Module to actually
812+ * clobber the MSRs, as the contract is poorly defined and not upheld.
813+ * E.g. the TDX-Module will synthesize an EPT Violation without doing
814+ * VM-Enter if it suspects a zero-step attack, and never "restore" VMM
815+ * state.
816+ */
802817 for (i = 0; i < ARRAY_SIZE(tdx_uret_msrs); i++)
803− kvm_user_return_msr_update_cache(tdx_uret_msrs[i].slot,
804− tdx_uret_msrs[i].defval);
818+ kvm_set_user_return_msr(tdx_uret_msrs[i].slot,
819+ tdx_uret_msrs[i].defval, -1ull);
805820 }
806821
807822 static void tdx_prepare_switch_to_host(struct kvm_vcpu *vcpu)
808823 {
809824 struct vcpu_vt *vt = to_vt(vcpu);
810− struct vcpu_tdx *tdx = to_tdx(vcpu);
811825
812826 if (!vt->guest_state_loaded)
813827 return;
@@ -815,11 +829,6 @@ static void tdx_prepare_switch_to_host(struct kvm_vcpu *vcpu) · 1 unmodified line
815829 ++vcpu->stat.host_state_reload;
816830 wrmsrl(MSR_KERNEL_GS_BASE, vt->msr_host_kernel_gs_base);
817831
818− if (tdx->guest_entered) {
819− tdx_user_return_msr_update_cache();
820− tdx->guest_entered = false;
821− }
822−
823832 vt->guest_state_loaded = false;
824833 }
825834
@@ -829,19 +838,52 @@ void tdx_vcpu_put(struct kvm_vcpu *vcpu) · 3 unmodified lines
829838 tdx_prepare_switch_to_host(vcpu);
830839 }
831840
841+/*
842+ * Life cycles for a TD and a vCPU:
843+ * 1. KVM_CREATE_VM ioctl.
844+ * TD state is TD_STATE_UNINITIALIZED.
845+ * hkid is not assigned at this stage.
846+ * 2. KVM_TDX_INIT_VM ioctl.
847+ * TD transitions to TD_STATE_INITIALIZED.
848+ * hkid is assigned after this stage.
849+ * 3. KVM_CREATE_VCPU ioctl. (only when TD is TD_STATE_INITIALIZED).
850+ * 3.1 tdx_vcpu_create() transitions vCPU state to VCPU_TD_STATE_UNINITIALIZED.
851+ * 3.2 vcpu_load() and vcpu_put() in kvm_arch_vcpu_create().
852+ * 3.3 (conditional) if any error encountered after kvm_arch_vcpu_create()
853+ * kvm_arch_vcpu_destroy() --> tdx_vcpu_free().
854+ * 4. KVM_TDX_INIT_VCPU ioctl.
855+ * tdx_vcpu_init() transitions vCPU state to VCPU_TD_STATE_INITIALIZED.
856+ * vCPU control structures are allocated at this stage.
857+ * 5. kvm_destroy_vm().
858+ * 5.1 tdx_mmu_release_hkid(): (1) tdh_vp_flush(), disassociates all vCPUs.
859+ * (2) puts hkid to !assigned state.
860+ * 5.2 kvm_destroy_vcpus() --> tdx_vcpu_free():
861+ * transitions vCPU to VCPU_TD_STATE_UNINITIALIZED state.
862+ * 5.3 tdx_vm_destroy()
863+ * transitions TD to TD_STATE_UNINITIALIZED state.
864+ *
865+ * tdx_vcpu_free() can be invoked only at 3.3 or 5.2.
866+ * - If at 3.3, hkid is still assigned, but the vCPU must be in
867+ * VCPU_TD_STATE_UNINITIALIZED state.
868+ * - if at 5.2, hkid must be !assigned and all vCPUs must be in
869+ * VCPU_TD_STATE_INITIALIZED state and have been dissociated.
870+ */
832871 void tdx_vcpu_free(struct kvm_vcpu *vcpu)
833872 {
834873 struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
835874 struct vcpu_tdx *tdx = to_tdx(vcpu);
836875 int i;
837876
877+ if (vcpu->cpu != -1) {
878+ KVM_BUG_ON(tdx->state == VCPU_TD_STATE_INITIALIZED, vcpu->kvm);
879+ tdx_flush_vp_on_cpu(vcpu);
880+ return;
881+ }
882+
838883 /*
839884 * It is not possible to reclaim pages while hkid is assigned. It might
840− * be assigned if:
841− * 1. the TD VM is being destroyed but freeing hkid failed, in which
842− * case the pages are leaked
843− * 2. TD VCPU creation failed and this on the error path, in which case
844− * there is nothing to do anyway
885+ * be assigned if the TD VM is being destroyed but freeing hkid failed,
886+ * in which case the pages are leaked.
845887 */
846888 if (is_hkid_assigned(kvm_tdx))
847889 return;
@@ -856,7 +898,7 @@ void tdx_vcpu_free(struct kvm_vcpu *vcpu) · 8 unmodified lines
856898 }
857899 if (tdx->vp.tdvpr_page) {
858900 tdx_reclaim_control_page(tdx->vp.tdvpr_page);
859− tdx->vp.tdvpr_page = 0;
901+ tdx->vp.tdvpr_page = NULL;
860902 tdx->vp.tdvpr_pa = 0;
861903 }
862904
@@ -1059,7 +1101,6 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) · 196 unmodified lines
10591101 update_debugctlmsr(vcpu->arch.host_debugctl);
10601102
10611103 tdx_load_host_xsave_state(vcpu);
1062− tdx->guest_entered = true;
10631104
10641105 vcpu->arch.regs_avail &= TDX_REGS_AVAIL_SET;
10651106
@@ -1069,9 +1110,6 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) · 3 unmodified lines
10691110 if (unlikely((tdx->vp_enter_ret & TDX_SW_ERROR) == TDX_SW_ERROR))
10701111 return EXIT_FASTPATH_NONE;
10711112
1072− if (unlikely(vmx_get_exit_reason(vcpu).basic == EXIT_REASON_MCE_DURING_VMENTRY))
1073− kvm_machine_check();
1074−
10751113 trace_kvm_exit(vcpu, KVM_ISA_VMX);
10761114
10771115 if (unlikely(tdx_failed_vmentry(vcpu)))
@@ -1583,137 +1621,79 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · 505 unmodified lines
15831621 td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa);
15841622 }
15851623
1586−static void tdx_unpin(struct kvm *kvm, struct page *page)
1624+static int tdx_mem_page_add(struct kvm *kvm, gfn_t gfn, enum pg_level level,
1625+ kvm_pfn_t pfn)
15871626 {
1588− put_page(page);
1627+ struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
1628+ u64 err, entry, level_state;
1629+ gpa_t gpa = gfn_to_gpa(gfn);
1630+
1631+ lockdep_assert_held(&kvm->slots_lock);
1632+
1633+ if (KVM_BUG_ON(kvm->arch.pre_fault_allowed, kvm) ||
1634+ KVM_BUG_ON(!kvm_tdx->page_add_src, kvm))
1635+ return -EIO;
1636+
1637+ err = tdh_mem_page_add(&kvm_tdx->td, gpa, pfn_to_page(pfn),
1638+ kvm_tdx->page_add_src, &entry, &level_state);
1639+ if (unlikely(tdx_operand_busy(err)))
1640+ return -EBUSY;
1641+
1642+ if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_ADD, entry, level_state, kvm))
1643+ return -EIO;
1644+
1645+ return 0;
15891646 }
15901647
15911648 static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,
1592− enum pg_level level, struct page *page)
1649+ enum pg_level level, kvm_pfn_t pfn)
15931650 {
15941651 int tdx_level = pg_level_to_tdx_sept_level(level);
15951652 struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
1653+ struct page *page = pfn_to_page(pfn);
15961654 gpa_t gpa = gfn_to_gpa(gfn);
15971655 u64 entry, level_state;
15981656 u64 err;
15991657
16001658 err = tdh_mem_page_aug(&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state);
1601− if (unlikely(tdx_operand_busy(err))) {
1602− tdx_unpin(kvm, page);
1659+ if (unlikely(tdx_operand_busy(err)))
16031660 return -EBUSY;
1604− }
16051661
1606− if (KVM_BUG_ON(err, kvm)) {
1607− pr_tdx_error_2(TDH_MEM_PAGE_AUG, err, entry, level_state);
1608− tdx_unpin(kvm, page);
1662+ if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_AUG, entry, level_state, kvm))
16091663 return -EIO;
1610− }
1611−
1612− return 0;
1613−}
1614−
1615−/*
1616− * KVM_TDX_INIT_MEM_REGION calls kvm_gmem_populate() to map guest pages; the
1617− * callback tdx_gmem_post_populate() then maps pages into private memory.
1618− * through the a seamcall TDH.MEM.PAGE.ADD(). The SEAMCALL also requires the
1619− * private EPT structures for the page to have been built before, which is
1620− * done via kvm_tdp_map_page(). nr_premapped counts the number of pages that
1621− * were added to the EPT structures but not added with TDH.MEM.PAGE.ADD().
1622− * The counter has to be zero on KVM_TDX_FINALIZE_VM, to ensure that there
1623− * are no half-initialized shared EPT pages.
1624− */
1625−static int tdx_mem_page_record_premap_cnt(struct kvm *kvm, gfn_t gfn,
1626− enum pg_level level, kvm_pfn_t pfn)
1627−{
1628− struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
1629−
1630− if (KVM_BUG_ON(kvm->arch.pre_fault_allowed, kvm))
1631− return -EINVAL;
16321664
1633− /* nr_premapped will be decreased when tdh_mem_page_add() is called. */
1634− atomic64_inc(&kvm_tdx->nr_premapped);
16351665 return 0;
16361666 }
16371667
16381668 static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,
1639− enum pg_level level, kvm_pfn_t pfn)
1669+ enum pg_level level, u64 mirror_spte)
16401670 {
16411671 struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
1642− struct page *page = pfn_to_page(pfn);
1672+ kvm_pfn_t pfn = spte_to_pfn(mirror_spte);
16431673
16441674 /* TODO: handle large pages. */
16451675 if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
1646− return -EINVAL;
1676+ return -EIO;
16471677
1648− /*
1649− * Because guest_memfd doesn't support page migration with
1650− * a_ops->migrate_folio (yet), no callback is triggered for KVM on page
1651− * migration. Until guest_memfd supports page migration, prevent page
1652− * migration.
1653− * TODO: Once guest_memfd introduces callback on page migration,
1654− * implement it and remove get_page/put_page().
1655− */
1656− get_page(page);
1678+ WARN_ON_ONCE(!is_shadow_present_pte(mirror_spte) ||
1679+ (mirror_spte & VMX_EPT_RWX_MASK) != VMX_EPT_RWX_MASK);
16571680
16581681 /*
1659− * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching
1660− * barrier in tdx_td_finalize().
1682+ * Ensure pre_fault_allowed is read by kvm_arch_vcpu_pre_fault_memory()
1683+ * before kvm_tdx->state. Userspace must not be allowed to pre-fault
1684+ * arbitrary memory until the initial memory image is finalized. Pairs
1685+ * with the smp_wmb() in tdx_td_finalize().
16611686 */
16621687 smp_rmb();
1663− if (likely(kvm_tdx->state == TD_STATE_RUNNABLE))
1664− return tdx_mem_page_aug(kvm, gfn, level, page);
1665−
1666− return tdx_mem_page_record_premap_cnt(kvm, gfn, level, pfn);
1667−}
1668−
1669−static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
1670− enum pg_level level, struct page *page)
1671−{
1672− int tdx_level = pg_level_to_tdx_sept_level(level);
1673− struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
1674− gpa_t gpa = gfn_to_gpa(gfn);
1675− u64 err, entry, level_state;
1676−
1677− /* TODO: handle large pages. */
1678− if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
1679− return -EINVAL;
1680−
1681− if (KVM_BUG_ON(!is_hkid_assigned(kvm_tdx), kvm))
1682− return -EINVAL;
16831688
16841689 /*
1685− * When zapping private page, write lock is held. So no race condition
1686− * with other vcpu sept operation.
1687− * Race with TDH.VP.ENTER due to (0-step mitigation) and Guest TDCALLs.
1690+ * If the TD isn't finalized/runnable, then userspace is initializing
1691+ * the VM image via KVM_TDX_INIT_MEM_REGION; ADD the page to the TD.
16881692 */
1689− err = tdh_mem_page_remove(&kvm_tdx->td, gpa, tdx_level, &entry,
1690− &level_state);
1691−
1692− if (unlikely(tdx_operand_busy(err))) {
1693− /*
1694− * The second retry is expected to succeed after kicking off all
1695− * other vCPUs and prevent them from invoking TDH.VP.ENTER.
1696− */
1697− tdx_no_vcpus_enter_start(kvm);
1698− err = tdh_mem_page_remove(&kvm_tdx->td, gpa, tdx_level, &entry,
1699− &level_state);
1700− tdx_no_vcpus_enter_stop(kvm);
1701− }
1702−
1703− if (KVM_BUG_ON(err, kvm)) {
1704− pr_tdx_error_2(TDH_MEM_PAGE_REMOVE, err, entry, level_state);
1705− return -EIO;
1706− }
1707−
1708− err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page);
1693+ if (unlikely(kvm_tdx->state != TD_STATE_RUNNABLE))
1694+ return tdx_mem_page_add(kvm, gfn, level, pfn);
17091695
1710− if (KVM_BUG_ON(err, kvm)) {
1711− pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err);
1712− return -EIO;
1713− }
1714− tdx_quirk_reset_page(page);
1715− tdx_unpin(kvm, page);
1716− return 0;
1696+ return tdx_mem_page_aug(kvm, gfn, level, pfn);
17171697 }
17181698
17191699 static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,
@@ -1729,80 +1709,12 @@ static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn, · 9 unmodified lines
17291709 if (unlikely(tdx_operand_busy(err)))
17301710 return -EBUSY;
17311711
1732− if (KVM_BUG_ON(err, kvm)) {
1733− pr_tdx_error_2(TDH_MEM_SEPT_ADD, err, entry, level_state);
1712+ if (TDX_BUG_ON_2(err, TDH_MEM_SEPT_ADD, entry, level_state, kvm))
17341713 return -EIO;
1735− }
17361714
17371715 return 0;
17381716 }
17391717
1740−/*
1741− * Check if the error returned from a SEPT zap SEAMCALL is due to that a page is
1742− * mapped by KVM_TDX_INIT_MEM_REGION without tdh_mem_page_add() being called
1743− * successfully.
1744− *
1745− * Since tdh_mem_sept_add() must have been invoked successfully before a
1746− * non-leaf entry present in the mirrored page table, the SEPT ZAP related
1747− * SEAMCALLs should not encounter err TDX_EPT_WALK_FAILED. They should instead
1748− * find TDX_EPT_ENTRY_STATE_INCORRECT due to an empty leaf entry found in the
1749− * SEPT.
1750− *
1751− * Further check if the returned entry from SEPT walking is with RWX permissions
1752− * to filter out anything unexpected.
1753− *
1754− * Note: @level is pg_level, not the tdx_level. The tdx_level extracted from
1755− * level_state returned from a SEAMCALL error is the same as that passed into
1756− * the SEAMCALL.
1757− */
1758−static int tdx_is_sept_zap_err_due_to_premap(struct kvm_tdx *kvm_tdx, u64 err,
1759− u64 entry, int level)
1760−{
1761− if (!err || kvm_tdx->state == TD_STATE_RUNNABLE)
1762− return false;
1763−
1764− if (err != (TDX_EPT_ENTRY_STATE_INCORRECT | TDX_OPERAND_ID_RCX))
1765− return false;
1766−
1767− if ((is_last_spte(entry, level) && (entry & VMX_EPT_RWX_MASK)))
1768− return false;
1769−
1770− return true;
1771−}
1772−
1773−static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,
1774− enum pg_level level, struct page *page)
1775−{
1776− int tdx_level = pg_level_to_tdx_sept_level(level);
1777− struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
1778− gpa_t gpa = gfn_to_gpa(gfn) & KVM_HPAGE_MASK(level);
1779− u64 err, entry, level_state;
1780−
1781− /* For now large page isn't supported yet. */
1782− WARN_ON_ONCE(level != PG_LEVEL_4K);
1783−
1784− err = tdh_mem_range_block(&kvm_tdx->td, gpa, tdx_level, &entry, &level_state);
1785−
1786− if (unlikely(tdx_operand_busy(err))) {
1787− /* After no vCPUs enter, the second retry is expected to succeed */
1788− tdx_no_vcpus_enter_start(kvm);
1789− err = tdh_mem_range_block(&kvm_tdx->td, gpa, tdx_level, &entry, &level_state);
1790− tdx_no_vcpus_enter_stop(kvm);
1791− }
1792− if (tdx_is_sept_zap_err_due_to_premap(kvm_tdx, err, entry, level) &&
1793− !KVM_BUG_ON(!atomic64_read(&kvm_tdx->nr_premapped), kvm)) {
1794− atomic64_dec(&kvm_tdx->nr_premapped);
1795− tdx_unpin(kvm, page);
1796− return 0;
1797− }
1798−
1799− if (KVM_BUG_ON(err, kvm)) {
1800− pr_tdx_error_2(TDH_MEM_RANGE_BLOCK, err, entry, level_state);
1801− return -EIO;
1802− }
1803− return 1;
1804−}
1805−
18061718 /*
18071719 * Ensure shared and private EPTs to be flushed on all vCPUs.
18081720 * tdh_mem_track() is the only caller that increases TD epoch. An increase in
@@ -1836,18 +1748,15 @@ static void tdx_track(struct kvm *kvm) · 27 unmodified lines
18361748 if (unlikely(kvm_tdx->state != TD_STATE_RUNNABLE))
18371749 return;
18381750
1751+ /*
1752+ * The full sequence of TDH.MEM.TRACK and forcing vCPUs out of guest
1753+ * mode must be serialized, as TDH.MEM.TRACK will fail if the previous
1754+ * tracking epoch hasn't completed.
1755+ */
18391756 lockdep_assert_held_write(&kvm->mmu_lock);
18401757
1841− err = tdh_mem_track(&kvm_tdx->td);
1842− if (unlikely(tdx_operand_busy(err))) {
1843− /* After no vCPUs enter, the second retry is expected to succeed */
1844− tdx_no_vcpus_enter_start(kvm);
1845− err = tdh_mem_track(&kvm_tdx->td);
1846− tdx_no_vcpus_enter_stop(kvm);
1847− }
1848−
1849− if (KVM_BUG_ON(err, kvm))
1850− pr_tdx_error(TDH_MEM_TRACK, err);
1758+ err = tdh_do_no_vcpus(tdh_mem_track, kvm, &kvm_tdx->td);
1759+ TDX_BUG_ON(err, TDH_MEM_TRACK, kvm);
18511760
18521761 kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE);
18531762 }
@@ -1866,7 +1775,7 @@ static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, · 12 unmodified lines
18661775 * and slot move/deletion.
18671776 */
18681777 if (KVM_BUG_ON(is_hkid_assigned(kvm_tdx), kvm))
1869− return -EINVAL;
1778+ return -EIO;
18701779
18711780 /*
18721781 * The HKID assigned to this TD was already freed and cache was
@@ -1875,11 +1784,16 @@ static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, · 2 unmodified lines
18751784 return tdx_reclaim_page(virt_to_page(private_spt));
18761785 }
18771786
1878−static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
1879− enum pg_level level, kvm_pfn_t pfn)
1787+static void tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
1788+ enum pg_level level, u64 mirror_spte)
18801789 {
1881− struct page *page = pfn_to_page(pfn);
1882− int ret;
1790+ struct page *page = pfn_to_page(spte_to_pfn(mirror_spte));
1791+ int tdx_level = pg_level_to_tdx_sept_level(level);
1792+ struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
1793+ gpa_t gpa = gfn_to_gpa(gfn);
1794+ u64 err, entry, level_state;
1795+
1796+ lockdep_assert_held_write(&kvm->mmu_lock);
18831797
18841798 /*
18851799 * HKID is released after all private pages have been removed, and set
@@ -1887,11 +1801,16 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, · 1 unmodified line
18871801 * there can't be anything populated in the private EPT.
18881802 */
18891803 if (KVM_BUG_ON(!is_hkid_assigned(to_kvm_tdx(kvm)), kvm))
1890− return -EINVAL;
1804+ return;
18911805
1892− ret = tdx_sept_zap_private_spte(kvm, gfn, level, page);
1893− if (ret <= 0)
1894− return ret;
1806+ /* TODO: handle large pages. */
1807+ if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
1808+ return;
1809+
1810+ err = tdh_do_no_vcpus(tdh_mem_range_block, kvm, &kvm_tdx->td, gpa,
1811+ tdx_level, &entry, &level_state);
1812+ if (TDX_BUG_ON_2(err, TDH_MEM_RANGE_BLOCK, entry, level_state, kvm))
1813+ return;
18951814
18961815 /*
18971816 * TDX requires TLB tracking before dropping private page. Do
@@ -1899,7 +1818,21 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, · 1 unmodified line
18991818 */
19001819 tdx_track(kvm);
19011820
1902− return tdx_sept_drop_private_spte(kvm, gfn, level, page);
1821+ /*
1822+ * When zapping private page, write lock is held. So no race condition
1823+ * with other vcpu sept operation.
1824+ * Race with TDH.VP.ENTER due to (0-step mitigation) and Guest TDCALLs.
1825+ */
1826+ err = tdh_do_no_vcpus(tdh_mem_page_remove, kvm, &kvm_tdx->td, gpa,
1827+ tdx_level, &entry, &level_state);
1828+ if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_REMOVE, entry, level_state, kvm))
1829+ return;
1830+
1831+ err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page);
1832+ if (TDX_BUG_ON(err, TDH_PHYMEM_PAGE_WBINVD, kvm))
1833+ return;
1834+
1835+ tdx_quirk_reset_page(page);
19031836 }
19041837
19051838 void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,
@@ -2145,11 +2078,7 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) · 239 unmodified lines
21452078 }
21462079
21472080 unhandled_exit:
2148− vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
2149− vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_UNEXPECTED_EXIT_REASON;
2150− vcpu->run->internal.ndata = 2;
2151− vcpu->run->internal.data[0] = vp_enter_ret;
2152− vcpu->run->internal.data[1] = vcpu->arch.last_vmentry_cpu;
2081+ kvm_prepare_unexpected_reason_exit(vcpu, vp_enter_ret);
21532082 return 0;
21542083 }
21552084
@@ -2282,37 +2211,28 @@ static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) · 126 unmodified lines
22822211 if (cmd->flags)
22832212 return -EINVAL;
22842213
2285− caps = kzalloc(sizeof(*caps) +
2286− sizeof(struct kvm_cpuid_entry2) * td_conf->num_cpuid_config,
2287− GFP_KERNEL);
2288− if (!caps)
2289− return -ENOMEM;
2290−
22912214 user_caps = u64_to_user_ptr(cmd->data);
2292− if (get_user(nr_user_entries, &user_caps->cpuid.nent)) {
2293− ret = -EFAULT;
2294− goto out;
2295− }
2215+ if (get_user(nr_user_entries, &user_caps->cpuid.nent))
2216+ return -EFAULT;
22962217
2297− if (nr_user_entries < td_conf->num_cpuid_config) {
2298− ret = -E2BIG;
2299− goto out;
2300− }
2218+ if (nr_user_entries < td_conf->num_cpuid_config)
2219+ return -E2BIG;
2220+
2221+ caps = kzalloc(struct_size(caps, cpuid.entries,
2222+ td_conf->num_cpuid_config), GFP_KERNEL);
2223+ if (!caps)
2224+ return -ENOMEM;
23012225
23022226 ret = init_kvm_tdx_caps(td_conf, caps);
23032227 if (ret)
23042228 goto out;
23052229
2306− if (copy_to_user(user_caps, caps, sizeof(*caps))) {
2230+ if (copy_to_user(user_caps, caps, struct_size(caps, cpuid.entries,
2231+ caps->cpuid.nent))) {
23072232 ret = -EFAULT;
23082233 goto out;
23092234 }
23102235
2311− if (copy_to_user(user_caps->cpuid.entries, caps->cpuid.entries,
2312− caps->cpuid.nent *
2313− sizeof(caps->cpuid.entries[0])))
2314− ret = -EFAULT;
2315−
23162236 out:
23172237 /* kfree() accepts NULL. */
23182238 kfree(caps);
@@ -2537,8 +2457,7 @@ static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params, · 218 unmodified lines
25372457 goto free_packages;
25382458 }
25392459
2540− if (WARN_ON_ONCE(err)) {
2541− pr_tdx_error(TDH_MNG_CREATE, err);
2460+ if (TDX_BUG_ON(err, TDH_MNG_CREATE, kvm)) {
25422461 ret = -EIO;
25432462 goto free_packages;
25442463 }
@@ -2579,8 +2498,7 @@ static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params, · 34 unmodified lines
25792498 ret = -EAGAIN;
25802499 goto teardown;
25812500 }
2582− if (WARN_ON_ONCE(err)) {
2583− pr_tdx_error(TDH_MNG_ADDCX, err);
2501+ if (TDX_BUG_ON(err, TDH_MNG_ADDCX, kvm)) {
25842502 ret = -EIO;
25852503 goto teardown;
25862504 }
@@ -2597,8 +2515,7 @@ static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params, · 10 unmodified lines
25972515 *seamcall_err = err;
25982516 ret = -EINVAL;
25992517 goto teardown;
2600− } else if (WARN_ON_ONCE(err)) {
2601− pr_tdx_error_1(TDH_MNG_INIT, err, rcx);
2518+ } else if (TDX_BUG_ON_1(err, TDH_MNG_INIT, rcx, kvm)) {
26022519 ret = -EIO;
26032520 goto teardown;
26042521 }
@@ -2642,7 +2559,7 @@ free_tdcs: · 37 unmodified lines
26422559 free_tdr:
26432560 if (tdr_page)
26442561 __free_page(tdr_page);
2645− kvm_tdx->td.tdr_page = 0;
2562+ kvm_tdx->td.tdr_page = NULL;
26462563
26472564 free_hkid:
26482565 tdx_hkid_free(kvm_tdx);
@@ -2747,11 +2664,53 @@ err_out: · 98 unmodified lines
27472664 return -EIO;
27482665 }
27492666
2667+typedef void *tdx_vm_state_guard_t;
2668+
2669+static tdx_vm_state_guard_t tdx_acquire_vm_state_locks(struct kvm *kvm)
2670+{
2671+ int r;
2672+
2673+ mutex_lock(&kvm->lock);
2674+
2675+ if (kvm->created_vcpus != atomic_read(&kvm->online_vcpus)) {
2676+ r = -EBUSY;
2677+ goto out_err;
2678+ }
2679+
2680+ r = kvm_lock_all_vcpus(kvm);
2681+ if (r)
2682+ goto out_err;
2683+
2684+ /*
2685+ * Note the unintuitive ordering! vcpu->mutex must be taken outside
2686+ * kvm->slots_lock!
2687+ */
2688+ mutex_lock(&kvm->slots_lock);
2689+ return kvm;
2690+
2691+out_err:
2692+ mutex_unlock(&kvm->lock);
2693+ return ERR_PTR(r);
2694+}
2695+
2696+static void tdx_release_vm_state_locks(struct kvm *kvm)
2697+{
2698+ mutex_unlock(&kvm->slots_lock);
2699+ kvm_unlock_all_vcpus(kvm);
2700+ mutex_unlock(&kvm->lock);
2701+}
2702+
2703+DEFINE_CLASS(tdx_vm_state_guard, tdx_vm_state_guard_t,
2704+ if (!IS_ERR(_T)) tdx_release_vm_state_locks(_T),
2705+ tdx_acquire_vm_state_locks(kvm), struct kvm *kvm);
2706+
27502707 static int tdx_td_init(struct kvm *kvm, struct kvm_tdx_cmd *cmd)
27512708 {
2709+ struct kvm_tdx_init_vm __user *user_data = u64_to_user_ptr(cmd->data);
27522710 struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
27532711 struct kvm_tdx_init_vm *init_vm;
27542712 struct td_params *td_params = NULL;
2713+ u32 nr_user_entries;
27552714 int ret;
27562715
27572716 BUILD_BUG_ON(sizeof(*init_vm) != 256 + sizeof_field(struct kvm_tdx_init_vm, cpuid));
@@ -2763,28 +2722,16 @@ static int tdx_td_init(struct kvm *kvm, struct kvm_tdx_cmd *cmd) · 5 unmodified lines
27632722 if (cmd->flags)
27642723 return -EINVAL;
27652724
2766− init_vm = kmalloc(sizeof(*init_vm) +
2767− sizeof(init_vm->cpuid.entries[0]) * KVM_MAX_CPUID_ENTRIES,
2768− GFP_KERNEL);
2769− if (!init_vm)
2770− return -ENOMEM;
2771−
2772− if (copy_from_user(init_vm, u64_to_user_ptr(cmd->data), sizeof(*init_vm))) {
2773− ret = -EFAULT;
2774− goto out;
2775− }
2725+ if (get_user(nr_user_entries, &user_data->cpuid.nent))
2726+ return -EFAULT;
27762727
2777− if (init_vm->cpuid.nent > KVM_MAX_CPUID_ENTRIES) {
2778− ret = -E2BIG;
2779− goto out;
2780− }
2728+ if (nr_user_entries > KVM_MAX_CPUID_ENTRIES)
2729+ return -E2BIG;
27812730
2782− if (copy_from_user(init_vm->cpuid.entries,
2783− u64_to_user_ptr(cmd->data) + sizeof(*init_vm),
2784− flex_array_size(init_vm, cpuid.entries, init_vm->cpuid.nent))) {
2785− ret = -EFAULT;
2786− goto out;
2787− }
2731+ init_vm = memdup_user(user_data,
2732+ struct_size(user_data, cpuid.entries, nr_user_entries));
2733+ if (IS_ERR(init_vm))
2734+ return PTR_ERR(init_vm);
27882735
27892736 if (memchr_inv(init_vm->reserved, 0, sizeof(init_vm->reserved))) {
27902737 ret = -EINVAL;
@@ -2868,24 +2815,14 @@ static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd) · 77 unmodified lines
28682815 {
28692816 struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
28702817
2871− guard(mutex)(&kvm->slots_lock);
2872−
28732818 if (!is_hkid_assigned(kvm_tdx) || kvm_tdx->state == TD_STATE_RUNNABLE)
28742819 return -EINVAL;
2875− /*
2876− * Pages are pending for KVM_TDX_INIT_MEM_REGION to issue
2877− * TDH.MEM.PAGE.ADD().
2878− */
2879− if (atomic64_read(&kvm_tdx->nr_premapped))
2880− return -EINVAL;
28812820
28822821 cmd->hw_error = tdh_mr_finalize(&kvm_tdx->td);
28832822 if (tdx_operand_busy(cmd->hw_error))
28842823 return -EBUSY;
2885− if (KVM_BUG_ON(cmd->hw_error, kvm)) {
2886− pr_tdx_error(TDH_MR_FINALIZE, cmd->hw_error);
2824+ if (TDX_BUG_ON(cmd->hw_error, TDH_MR_FINALIZE, kvm))
28872825 return -EIO;
2888− }
28892826
28902827 kvm_tdx->state = TD_STATE_RUNNABLE;
28912828 /* TD_STATE_RUNNABLE must be set before 'pre_fault_allowed' */
@@ -2894,27 +2831,38 @@ static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd) · 2 unmodified lines
28942831 return 0;
28952832 }
28962833
2897−int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)
2834+static int tdx_get_cmd(void __user *argp, struct kvm_tdx_cmd *cmd)
28982835 {
2899− struct kvm_tdx_cmd tdx_cmd;
2900− int r;
2901−
2902− if (copy_from_user(&tdx_cmd, argp, sizeof(struct kvm_tdx_cmd)))
2836+ if (copy_from_user(cmd, argp, sizeof(*cmd)))
29032837 return -EFAULT;
29042838
29052839 /*
2906− * Userspace should never set hw_error. It is used to fill
2907− * hardware-defined error by the kernel.
2840+ * Userspace should never set hw_error. KVM writes hw_error to report
2841+ * hardware-defined error back to userspace.
29082842 */
2909− if (tdx_cmd.hw_error)
2843+ if (cmd->hw_error)
29102844 return -EINVAL;
29112845
2912− mutex_lock(&kvm->lock);
2846+ return 0;
2847+}
2848+
2849+int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)
2850+{
2851+ struct kvm_tdx_cmd tdx_cmd;
2852+ int r;
2853+
2854+ r = tdx_get_cmd(argp, &tdx_cmd);
2855+ if (r)
2856+ return r;
2857+
2858+ if (tdx_cmd.id == KVM_TDX_CAPABILITIES)
2859+ return tdx_get_capabilities(&tdx_cmd);
2860+
2861+ CLASS(tdx_vm_state_guard, guard)(kvm);
2862+ if (IS_ERR(guard))
2863+ return PTR_ERR(guard);
29132864
29142865 switch (tdx_cmd.id) {
2915− case KVM_TDX_CAPABILITIES:
2916− r = tdx_get_capabilities(&tdx_cmd);
2917− break;
29182866 case KVM_TDX_INIT_VM:
29192867 r = tdx_td_init(kvm, &tdx_cmd);
29202868 break;
@@ -2922,15 +2870,12 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) · 1 unmodified line
29222870 r = tdx_td_finalize(kvm, &tdx_cmd);
29232871 break;
29242872 default:
2925− r = -EINVAL;
2926− goto out;
2873+ return -EINVAL;
29272874 }
29282875
29292876 if (copy_to_user(argp, &tdx_cmd, sizeof(struct kvm_tdx_cmd)))
2930− r = -EFAULT;
2877+ return -EFAULT;
29312878
2932−out:
2933− mutex_unlock(&kvm->lock);
29342879 return r;
29352880 }
29362881
@@ -2972,16 +2917,14 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u64 vcpu_rcx) · 35 unmodified lines
29722917 }
29732918
29742919 err = tdh_vp_create(&kvm_tdx->td, &tdx->vp);
2975− if (KVM_BUG_ON(err, vcpu->kvm)) {
2920+ if (TDX_BUG_ON(err, TDH_VP_CREATE, vcpu->kvm)) {
29762921 ret = -EIO;
2977− pr_tdx_error(TDH_VP_CREATE, err);
29782922 goto free_tdcx;
29792923 }
29802924
29812925 for (i = 0; i < kvm_tdx->td.tdcx_nr_pages; i++) {
29822926 err = tdh_vp_addcx(&tdx->vp, tdx->vp.tdcx_pages[i]);
2983− if (KVM_BUG_ON(err, vcpu->kvm)) {
2984− pr_tdx_error(TDH_VP_ADDCX, err);
2927+ if (TDX_BUG_ON(err, TDH_VP_ADDCX, vcpu->kvm)) {
29852928 /*
29862929 * Pages already added are reclaimed by the vcpu_free
29872930 * method, but the rest are freed here.
@@ -2994,10 +2937,19 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u64 vcpu_rcx) · 6 unmodified lines
29942937 }
29952938 }
29962939
2997− err = tdh_vp_init(&tdx->vp, vcpu_rcx, vcpu->vcpu_id);
2998− if (KVM_BUG_ON(err, vcpu->kvm)) {
2999− pr_tdx_error(TDH_VP_INIT, err);
3000− return -EIO;
2940+ /*
2941+ * tdh_vp_init() can take an exclusive lock of the TDR resource inside
2942+ * the TDX-Module. The TDR resource is also taken as shared in several
2943+ * no-fail MMU paths, which could return TDX_OPERAND_BUSY on contention
2944+ * (TDX-Module locks are try-lock implementations with no slow path).
2945+ * Take mmu_lock for write to reflect the nature of the lock taken by
2946+ * the TDX-Module, and to ensure the no-fail MMU paths succeed, e.g. if
2947+ * a concurrent PUNCH_HOLE on guest_memfd triggers removal of SPTEs.
2948+ */
2949+ scoped_guard(write_lock, &vcpu->kvm->mmu_lock) {
2950+ err = tdh_vp_init(&tdx->vp, vcpu_rcx, vcpu->vcpu_id);
2951+ if (TDX_BUG_ON(err, TDH_VP_INIT, vcpu->kvm))
2952+ return -EIO;
30012953 }
30022954
30032955 vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE;
@@ -3016,7 +2968,7 @@ free_tdcx: · 12 unmodified lines
30162968 free_tdvpr:
30172969 if (tdx->vp.tdvpr_page)
30182970 __free_page(tdx->vp.tdvpr_page);
3019− tdx->vp.tdvpr_page = 0;
2971+ tdx->vp.tdvpr_page = NULL;
30202972 tdx->vp.tdvpr_pa = 0;
30212973
30222974 return ret;
@@ -3054,7 +3006,8 @@ static int tdx_vcpu_get_cpuid_leaf(struct kvm_vcpu *vcpu, u32 leaf, int *entry_i · 31 unmodified lines
30543006
30553007 static int tdx_vcpu_get_cpuid(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd)
30563008 {
3057− struct kvm_cpuid2 __user *output, *td_cpuid;
3009+ struct kvm_cpuid2 __user *output;
3010+ struct kvm_cpuid2 *td_cpuid;
30583011 int r = 0, i = 0, leaf;
30593012 u32 level;
30603013
@@ -3167,15 +3120,15 @@ struct tdx_gmem_post_populate_arg { · 106 unmodified lines
31673120 static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
31683121 void __user *src, int order, void *_arg)
31693122 {
3170− u64 error_code = PFERR_GUEST_FINAL_MASK | PFERR_PRIVATE_ACCESS;
3171− struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
31723123 struct tdx_gmem_post_populate_arg *arg = _arg;
3173− struct kvm_vcpu *vcpu = arg->vcpu;
3124+ struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
3125+ u64 err, entry, level_state;
31743126 gpa_t gpa = gfn_to_gpa(gfn);
3175− u8 level = PG_LEVEL_4K;
31763127 struct page *src_page;
31773128 int ret, i;
3178− u64 err, entry, level_state;
3129+
3130+ if (KVM_BUG_ON(kvm_tdx->page_add_src, kvm))
3131+ return -EIO;
31793132
31803133 /*
31813134 * Get the source page if it has been faulted in. Return failure if the
@@ -3187,49 +3140,29 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, · 5 unmodified lines
31873140 if (ret != 1)
31883141 return -ENOMEM;
31893142
3190− ret = kvm_tdp_map_page(vcpu, gpa, error_code, &level);
3191− if (ret < 0)
3192− goto out;
3193−
3194− /*
3195− * The private mem cannot be zapped after kvm_tdp_map_page()
3196− * because all paths are covered by slots_lock and the
3197− * filemap invalidate lock. Check that they are indeed enough.
3198− */
3199− if (IS_ENABLED(CONFIG_KVM_PROVE_MMU)) {
3200− scoped_guard(read_lock, &kvm->mmu_lock) {
3201− if (KVM_BUG_ON(!kvm_tdp_mmu_gpa_is_mapped(vcpu, gpa), kvm)) {
3202− ret = -EIO;
3203− goto out;
3204− }
3205− }
3206− }
3143+ kvm_tdx->page_add_src = src_page;
3144+ ret = kvm_tdp_mmu_map_private_pfn(arg->vcpu, gfn, pfn);
3145+ kvm_tdx->page_add_src = NULL;
32073146
3208− ret = 0;
3209− err = tdh_mem_page_add(&kvm_tdx->td, gpa, pfn_to_page(pfn),
3210− src_page, &entry, &level_state);
3211− if (err) {
3212− ret = unlikely(tdx_operand_busy(err)) ? -EBUSY : -EIO;
3213− goto out;
3214− }
3147+ put_page(src_page);
32153148
3216− if (!KVM_BUG_ON(!atomic64_read(&kvm_tdx->nr_premapped), kvm))
3217− atomic64_dec(&kvm_tdx->nr_premapped);
3149+ if (ret || !(arg->flags & KVM_TDX_MEASURE_MEMORY_REGION))
3150+ return ret;
32183151
3219− if (arg->flags & KVM_TDX_MEASURE_MEMORY_REGION) {
3220− for (i = 0; i < PAGE_SIZE; i += TDX_EXTENDMR_CHUNKSIZE) {
3221− err = tdh_mr_extend(&kvm_tdx->td, gpa + i, &entry,
3222− &level_state);
3223− if (err) {
3224− ret = -EIO;
3225− break;
3226− }
3227− }
3152+ /*
3153+ * Note, MR.EXTEND can fail if the S-EPT mapping is somehow removed
3154+ * between mapping the pfn and now, but slots_lock prevents memslot
3155+ * updates, filemap_invalidate_lock() prevents guest_memfd updates,
3156+ * mmu_notifier events can't reach S-EPT entries, and KVM's internal
3157+ * zapping flows are mutually exclusive with S-EPT mappings.
3158+ */
3159+ for (i = 0; i < PAGE_SIZE; i += TDX_EXTENDMR_CHUNKSIZE) {
3160+ err = tdh_mr_extend(&kvm_tdx->td, gpa + i, &entry, &level_state);
3161+ if (TDX_BUG_ON_2(err, TDH_MR_EXTEND, entry, level_state, kvm))
3162+ return -EIO;
32283163 }
32293164
3230−out:
3231− put_page(src_page);
3232− return ret;
3165+ return 0;
32333166 }
32343167
32353168 static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd)
@@ -3245,8 +3178,6 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c · 9 unmodified lines
32453178 if (tdx->state != VCPU_TD_STATE_INITIALIZED)
32463179 return -EINVAL;
32473180
3248− guard(mutex)(&kvm->slots_lock);
3249−
32503181 /* Once TD is finalized, the initial guest memory is fixed. */
32513182 if (kvm_tdx->state == TD_STATE_RUNNABLE)
32523183 return -EINVAL;
@@ -3264,7 +3195,6 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c · 11 unmodified lines
32643195 !vt_is_tdx_private_gpa(kvm, region.gpa + (region.nr_pages << PAGE_SHIFT) - 1))
32653196 return -EINVAL;
32663197
3267− kvm_mmu_reload(vcpu);
32683198 ret = 0;
32693199 while (region.nr_pages) {
32703200 if (signal_pending(current)) {
@@ -3301,28 +3231,57 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c · 30 unmodified lines
33013231 return ret;
33023232 }
33033233
3304−int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)
3234+int tdx_vcpu_unlocked_ioctl(struct kvm_vcpu *vcpu, void __user *argp)
33053235 {
3306− struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
3236+ struct kvm *kvm = vcpu->kvm;
3237+ struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
33073238 struct kvm_tdx_cmd cmd;
3308− int ret;
3239+ int r;
33093240
3310− if (!is_hkid_assigned(kvm_tdx) || kvm_tdx->state == TD_STATE_RUNNABLE)
3311− return -EINVAL;
3241+ r = tdx_get_cmd(argp, &cmd);
3242+ if (r)
3243+ return r;
33123244
3313− if (copy_from_user(&cmd, argp, sizeof(cmd)))
3314− return -EFAULT;
3245+ CLASS(tdx_vm_state_guard, guard)(kvm);
3246+ if (IS_ERR(guard))
3247+ return PTR_ERR(guard);
33153248
3316− if (cmd.hw_error)
3249+ if (!is_hkid_assigned(kvm_tdx) || kvm_tdx->state == TD_STATE_RUNNABLE)
33173250 return -EINVAL;
33183251
3252+ vcpu_load(vcpu);
3253+
33193254 switch (cmd.id) {
3255+ case KVM_TDX_INIT_MEM_REGION:
3256+ r = tdx_vcpu_init_mem_region(vcpu, &cmd);
3257+ break;
33203258 case KVM_TDX_INIT_VCPU:
3321− ret = tdx_vcpu_init(vcpu, &cmd);
3259+ r = tdx_vcpu_init(vcpu, &cmd);
33223260 break;
3323− case KVM_TDX_INIT_MEM_REGION:
3324− ret = tdx_vcpu_init_mem_region(vcpu, &cmd);
3261+ default:
3262+ r = -ENOIOCTLCMD;
33253263 break;
3264+ }
3265+
3266+ vcpu_put(vcpu);
3267+
3268+ return r;
3269+}
3270+
3271+int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)
3272+{
3273+ struct kvm_tdx *kvm_tdx = to_kvm_tdx(vcpu->kvm);
3274+ struct kvm_tdx_cmd cmd;
3275+ int ret;
3276+
3277+ if (!is_hkid_assigned(kvm_tdx) || kvm_tdx->state == TD_STATE_RUNNABLE)
3278+ return -EINVAL;
3279+
3280+ ret = tdx_get_cmd(argp, &cmd);
3281+ if (ret)
3282+ return ret;
3283+
3284+ switch (cmd.id) {
33263285 case KVM_TDX_GET_CPUID:
33273286 ret = tdx_vcpu_get_cpuid(vcpu, &cmd);
33283287 break;
@@ -3447,10 +3406,6 @@ static int __init __tdx_bringup(void) · 118 unmodified lines
34473406 /*
34483407 * Check if MSRs (tdx_uret_msrs) can be saved/restored
34493408 * before returning to user space.
3450− *
3451− * this_cpu_ptr(user_return_msrs)->registered isn't checked
3452− * because the registration is done at vcpu runtime by
3453− * tdx_user_return_msr_update_cache().
34543409 */
34553410 tdx_uret_msrs[i].slot = kvm_find_user_return_msr(tdx_uret_msrs[i].msr);
34563411 if (tdx_uret_msrs[i].slot == -1) {
arch/x86/kvm/vmx/tdx.h+6−3
@@ -36,8 +36,12 @@ struct kvm_tdx { · 35 unmodified lines
3636
3737 struct tdx_td td;
3838
39− /* For KVM_TDX_INIT_MEM_REGION. */
40− atomic64_t nr_premapped;
39+ /*
40+ * Scratch pointer used to pass the source page to tdx_mem_page_add().
41+ * Protected by slots_lock, and non-NULL only when mapping a private
42+ * pfn via tdx_gmem_post_populate().
43+ */
44+ struct page *page_add_src;
4145
4246 /*
4347 * Prevent vCPUs from TD entry to ensure SEPT zap related SEAMCALLs do
@@ -67,7 +71,6 @@ struct vcpu_tdx { · 23 unmodified lines
6771 u64 vp_enter_ret;
6872
6973 enum vcpu_tdx_state state;
70− bool guest_entered;
7174
7275 u64 map_gpa_next;
7376 u64 map_gpa_end;
arch/x86/kvm/vmx/vmenter.S+34−17
@@ -71,6 +71,7 @@ · 70 unmodified lines
7171 * @regs: unsigned long * (to guest registers)
7272 * @flags: VMX_RUN_VMRESUME: use VMRESUME instead of VMLAUNCH
7373 * VMX_RUN_SAVE_SPEC_CTRL: save guest SPEC_CTRL into vmx->spec_ctrl
74+ * VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO: vCPU can access host MMIO
7475 *
7576 * Returns:
7677 * 0 on VM-Exit, 1 on VM-Fail
@@ -92,7 +93,7 @@ SYM_FUNC_START(__vmx_vcpu_run) · 15 unmodified lines
9293 /* Save @vmx for SPEC_CTRL handling */
9394 push %_ASM_ARG1
9495
95− /* Save @flags for SPEC_CTRL handling */
96+ /* Save @flags (used for VMLAUNCH vs. VMRESUME and mitigations). */
9697 push %_ASM_ARG3
9798
9899 /*
@@ -101,9 +102,6 @@ SYM_FUNC_START(__vmx_vcpu_run) · 2 unmodified lines
101102 */
102103 push %_ASM_ARG2
103104
104− /* Copy @flags to EBX, _ASM_ARG3 is volatile. */
105− mov %_ASM_ARG3L, %ebx
106−
107105 lea (%_ASM_SP), %_ASM_ARG2
108106 call vmx_update_host_rsp
109107
@@ -118,13 +116,23 @@ SYM_FUNC_START(__vmx_vcpu_run) · 8 unmodified lines
118116 * and vmentry.
119117 */
120118 mov 2*WORD_SIZE(%_ASM_SP), %_ASM_DI
121− movl VMX_spec_ctrl(%_ASM_DI), %edi
122− movl PER_CPU_VAR(x86_spec_ctrl_current), %esi
123− cmp %edi, %esi
119+#ifdef CONFIG_X86_64
120+ mov VMX_spec_ctrl(%rdi), %rdx
121+ cmp PER_CPU_VAR(x86_spec_ctrl_current), %rdx
124122 je .Lspec_ctrl_done
123+ movl %edx, %eax
124+ shr $32, %rdx
125+#else
126+ mov VMX_spec_ctrl(%edi), %eax
127+ mov PER_CPU_VAR(x86_spec_ctrl_current), %ecx
128+ xor %eax, %ecx
129+ mov VMX_spec_ctrl + 4(%edi), %edx
130+ mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %edi
131+ xor %edx, %edi
132+ or %edi, %ecx
133+ je .Lspec_ctrl_done
134+#endif
125135 mov $MSR_IA32_SPEC_CTRL, %ecx
126− xor %edx, %edx
127− mov %edi, %eax
128136 wrmsr
129137
130138 .Lspec_ctrl_done:
@@ -137,9 +145,6 @@ SYM_FUNC_START(__vmx_vcpu_run) · 6 unmodified lines
137145 /* Load @regs to RAX. */
138146 mov (%_ASM_SP), %_ASM_AX
139147
140− /* Check if vmlaunch or vmresume is needed */
141− bt $VMX_RUN_VMRESUME_SHIFT, %ebx
142−
143148 /* Load guest registers. Don't clobber flags. */
144149 mov VCPU_RCX(%_ASM_AX), %_ASM_CX
145150 mov VCPU_RDX(%_ASM_AX), %_ASM_DX
@@ -160,11 +165,23 @@ SYM_FUNC_START(__vmx_vcpu_run) · 14 unmodified lines
160165 /* Load guest RAX. This kills the @regs pointer! */
161166 mov VCPU_RAX(%_ASM_AX), %_ASM_AX
162167
163− /* Clobbers EFLAGS.ZF */
164− CLEAR_CPU_BUFFERS
165−
166− /* Check EFLAGS.CF from the VMX_RUN_VMRESUME bit test above. */
167− jnc .Lvmlaunch
168+ /*
169+ * Note, ALTERNATIVE_2 works in reverse order. If CLEAR_CPU_BUF_VM is
170+ * enabled, do VERW unconditionally. If CPU_BUF_VM_MMIO is enabled,
171+ * check @flags to see if the vCPU has access to host MMIO, and if so,
172+ * do VERW. Else, do nothing (no mitigations needed/enabled).
173+ */
174+ ALTERNATIVE_2 "", \
175+ __stringify(testl $VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO, WORD_SIZE(%_ASM_SP); \
176+ jz .Lskip_mmio_verw; \
177+ VERW; \
178+ .Lskip_mmio_verw:), \
179+ X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO, \
180+ __stringify(VERW), X86_FEATURE_CLEAR_CPU_BUF_VM
181+
182+ /* Check @flags to see if VMLAUNCH or VMRESUME is needed. */
183+ testl $VMX_RUN_VMRESUME, WORD_SIZE(%_ASM_SP)
184+ jz .Lvmlaunch
168185
169186 /*
170187 * After a successful VMRESUME/VMLAUNCH, control flow "magically"
arch/x86/kvm/vmx/vmx.c+173−150
@@ -203,6 +203,7 @@ module_param(pt_mode, int, S_IRUGO); · 202 unmodified lines
203203
204204 struct x86_pmu_lbr __ro_after_init vmx_lbr_caps;
205205
206+#ifdef CONFIG_CPU_MITIGATIONS
206207 static DEFINE_STATIC_KEY_FALSE(vmx_l1d_should_flush);
207208 static DEFINE_STATIC_KEY_FALSE(vmx_l1d_flush_cond);
208209 static DEFINE_MUTEX(vmx_l1d_flush_mutex);
@@ -225,7 +226,7 @@ static const struct { · 16 unmodified lines
225226 #define L1D_CACHE_ORDER 4
226227 static void *vmx_l1d_flush_pages;
227228
228−static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf)
229+static int __vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf)
229230 {
230231 struct page *page;
231232 unsigned int i;
@@ -302,6 +303,26 @@ static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf) · 70 unmodified lines
302303 return 0;
303304 }
304305
306+static int vmx_setup_l1d_flush(void)
307+{
308+ /*
309+ * Hand the parameter mitigation value in which was stored in the pre
310+ * module init parser. If no parameter was given, it will contain
311+ * 'auto' which will be turned into the default 'cond' mitigation mode.
312+ */
313+ return __vmx_setup_l1d_flush(vmentry_l1d_flush_param);
314+}
315+
316+static void vmx_cleanup_l1d_flush(void)
317+{
318+ if (vmx_l1d_flush_pages) {
319+ free_pages((unsigned long)vmx_l1d_flush_pages, L1D_CACHE_ORDER);
320+ vmx_l1d_flush_pages = NULL;
321+ }
322+ /* Restore state so sysfs ignores VMX */
323+ l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_AUTO;
324+}
325+
305326 static int vmentry_l1d_flush_parse(const char *s)
306327 {
307328 unsigned int i;
@@ -339,7 +360,7 @@ static int vmentry_l1d_flush_set(const char *s, const struct kernel_param *kp) · 31 unmodified lines
339360 }
340361
341362 mutex_lock(&vmx_l1d_flush_mutex);
342− ret = vmx_setup_l1d_flush(l1tf);
363+ ret = __vmx_setup_l1d_flush(l1tf);
343364 mutex_unlock(&vmx_l1d_flush_mutex);
344365 return ret;
345366 }
@@ -352,6 +373,101 @@ static int vmentry_l1d_flush_get(char *s, const struct kernel_param *kp) · 6 unmodified lines
352373 return sysfs_emit(s, "%s\n", vmentry_l1d_param[l1tf_vmx_mitigation].option);
353374 }
354375
376+/*
377+ * Software based L1D cache flush which is used when microcode providing
378+ * the cache control MSR is not loaded.
379+ *
380+ * The L1D cache is 32 KiB on Nehalem and later microarchitectures, but to
381+ * flush it is required to read in 64 KiB because the replacement algorithm
382+ * is not exactly LRU. This could be sized at runtime via topology
383+ * information but as all relevant affected CPUs have 32KiB L1D cache size
384+ * there is no point in doing so.
385+ */
386+static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)
387+{
388+ int size = PAGE_SIZE << L1D_CACHE_ORDER;
389+
390+ if (!static_branch_unlikely(&vmx_l1d_should_flush))
391+ return;
392+
393+ /*
394+ * This code is only executed when the flush mode is 'cond' or
395+ * 'always'
396+ */
397+ if (static_branch_likely(&vmx_l1d_flush_cond)) {
398+ /*
399+ * Clear the per-cpu flush bit, it gets set again if the vCPU
400+ * is reloaded, i.e. if the vCPU is scheduled out or if KVM
401+ * exits to userspace, or if KVM reaches one of the unsafe
402+ * VMEXIT handlers, e.g. if KVM calls into the emulator,
403+ * or from the interrupt handlers.
404+ */
405+ if (!kvm_get_cpu_l1tf_flush_l1d())
406+ return;
407+ kvm_clear_cpu_l1tf_flush_l1d();
408+ }
409+
410+ vcpu->stat.l1d_flush++;
411+
412+ if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) {
413+ native_wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH);
414+ return;
415+ }
416+
417+ asm volatile(
418+ /* First ensure the pages are in the TLB */
419+ "xorl %%eax, %%eax\n"
420+ ".Lpopulate_tlb:\n\t"
421+ "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t"
422+ "addl $4096, %%eax\n\t"
423+ "cmpl %%eax, %[size]\n\t"
424+ "jne .Lpopulate_tlb\n\t"
425+ "xorl %%eax, %%eax\n\t"
426+ "cpuid\n\t"
427+ /* Now fill the cache */
428+ "xorl %%eax, %%eax\n"
429+ ".Lfill_cache:\n"
430+ "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t"
431+ "addl $64, %%eax\n\t"
432+ "cmpl %%eax, %[size]\n\t"
433+ "jne .Lfill_cache\n\t"
434+ "lfence\n"
435+ :: [flush_pages] "r" (vmx_l1d_flush_pages),
436+ [size] "r" (size)
437+ : "eax", "ebx", "ecx", "edx");
438+}
439+
440+#else /* CONFIG_CPU_MITIGATIONS*/
441+static int vmx_setup_l1d_flush(void)
442+{
443+ l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_NEVER;
444+ return 0;
445+}
446+static void vmx_cleanup_l1d_flush(void)
447+{
448+ l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_AUTO;
449+}
450+static __always_inline void vmx_l1d_flush(struct kvm_vcpu *vcpu)
451+{
452+
453+}
454+static int vmentry_l1d_flush_set(const char *s, const struct kernel_param *kp)
455+{
456+ pr_warn_once("Kernel compiled without mitigations, ignoring vmentry_l1d_flush\n");
457+ return 0;
458+}
459+static int vmentry_l1d_flush_get(char *s, const struct kernel_param *kp)
460+{
461+ return sysfs_emit(s, "never\n");
462+}
463+#endif
464+
465+static const struct kernel_param_ops vmentry_l1d_flush_ops = {
466+ .set = vmentry_l1d_flush_set,
467+ .get = vmentry_l1d_flush_get,
468+};
469+module_param_cb(vmentry_l1d_flush, &vmentry_l1d_flush_ops, NULL, 0644);
470+
355471 static __always_inline void vmx_disable_fb_clear(struct vcpu_vmx *vmx)
356472 {
357473 u64 msr;
@@ -404,12 +520,6 @@ static void vmx_update_fb_clear_dis(struct kvm_vcpu *vcpu, struct vcpu_vmx *vmx) · 46 unmodified lines
404520 vmx->disable_fb_clear = false;
405521 }
406522
407−static const struct kernel_param_ops vmentry_l1d_flush_ops = {
408− .set = vmentry_l1d_flush_set,
409− .get = vmentry_l1d_flush_get,
410−};
411−module_param_cb(vmentry_l1d_flush, &vmentry_l1d_flush_ops, NULL, 0644);
412−
413523 static u32 vmx_segment_access_rights(struct kvm_segment *var);
414524
415525 void vmx_vmexit(void);
@@ -752,7 +862,7 @@ static void __loaded_vmcs_clear(void *arg) · 336 unmodified lines
752862 loaded_vmcs->launched = 0;
753863 }
754864
755−void loaded_vmcs_clear(struct loaded_vmcs *loaded_vmcs)
865+static void loaded_vmcs_clear(struct loaded_vmcs *loaded_vmcs)
756866 {
757867 int cpu = loaded_vmcs->cpu;
758868
@@ -903,7 +1013,7 @@ unsigned int __vmx_vcpu_run_flags(struct vcpu_vmx *vmx) · 144 unmodified lines
9031013 if (!msr_write_intercepted(vmx, MSR_IA32_SPEC_CTRL))
9041014 flags |= VMX_RUN_SAVE_SPEC_CTRL;
9051015
906− if (static_branch_unlikely(&cpu_buf_vm_clear) &&
1016+ if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO) &&
9071017 kvm_vcpu_can_access_host_mmio(&vmx->vcpu))
9081018 flags |= VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO;
9091019
@@ -3219,6 +3329,40 @@ static inline int vmx_get_current_vpid(struct kvm_vcpu *vcpu) · 2309 unmodified lines
32193329 return to_vmx(vcpu)->vpid;
32203330 }
32213331
3332+static u64 construct_eptp(hpa_t root_hpa)
3333+{
3334+ u64 eptp = root_hpa | VMX_EPTP_MT_WB;
3335+ struct kvm_mmu_page *root;
3336+
3337+ if (kvm_mmu_is_dummy_root(root_hpa))
3338+ return eptp | VMX_EPTP_PWL_4;
3339+
3340+ /*
3341+ * EPT roots should always have an associated MMU page. Return a "bad"
3342+ * EPTP to induce VM-Fail instead of continuing on in a unknown state.
3343+ */
3344+ root = root_to_sp(root_hpa);
3345+ if (WARN_ON_ONCE(!root))
3346+ return INVALID_PAGE;
3347+
3348+ eptp |= (root->role.level == 5) ? VMX_EPTP_PWL_5 : VMX_EPTP_PWL_4;
3349+
3350+ if (enable_ept_ad_bits && !root->role.ad_disabled)
3351+ eptp |= VMX_EPTP_AD_ENABLE_BIT;
3352+
3353+ return eptp;
3354+}
3355+
3356+static void vmx_flush_tlb_ept_root(hpa_t root_hpa)
3357+{
3358+ u64 eptp = construct_eptp(root_hpa);
3359+
3360+ if (VALID_PAGE(eptp))
3361+ ept_sync_context(eptp);
3362+ else
3363+ ept_sync_global();
3364+}
3365+
32223366 void vmx_flush_tlb_current(struct kvm_vcpu *vcpu)
32233367 {
32243368 struct kvm_mmu *mmu = vcpu->arch.mmu;
@@ -3229,8 +3373,7 @@ void vmx_flush_tlb_current(struct kvm_vcpu *vcpu) · 4 unmodified lines
32293373 return;
32303374
32313375 if (enable_ept)
3232− ept_sync_context(construct_eptp(vcpu, root_hpa,
3233− mmu->root_role.level));
3376+ vmx_flush_tlb_ept_root(root_hpa);
32343377 else
32353378 vpid_sync_context(vmx_get_current_vpid(vcpu));
32363379 }
@@ -3396,30 +3539,16 @@ static int vmx_get_max_ept_level(void) · 159 unmodified lines
33963539 return 4;
33973540 }
33983541
3399−u64 construct_eptp(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level)
3400−{
3401− u64 eptp = VMX_EPTP_MT_WB;
3402−
3403− eptp |= (root_level == 5) ? VMX_EPTP_PWL_5 : VMX_EPTP_PWL_4;
3404−
3405− if (enable_ept_ad_bits &&
3406− (!is_guest_mode(vcpu) || nested_ept_ad_enabled(vcpu)))
3407− eptp |= VMX_EPTP_AD_ENABLE_BIT;
3408− eptp |= root_hpa;
3409−
3410− return eptp;
3411−}
3412−
34133542 void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level)
34143543 {
34153544 struct kvm *kvm = vcpu->kvm;
34163545 bool update_guest_cr3 = true;
34173546 unsigned long guest_cr3;
3418− u64 eptp;
34193547
34203548 if (enable_ept) {
3421− eptp = construct_eptp(vcpu, root_hpa, root_level);
3422− vmcs_write64(EPT_POINTER, eptp);
3549+ KVM_MMU_WARN_ON(root_to_sp(root_hpa) &&
3550+ root_level != root_to_sp(root_hpa)->role.level);
3551+ vmcs_write64(EPT_POINTER, construct_eptp(root_hpa));
34233552
34243553 hv_track_root_tdp(vcpu, root_hpa);
34253554
@@ -6631,15 +6760,8 @@ static int __vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) · 3205 unmodified lines
66316760 return kvm_vmx_exit_handlers[exit_handler_index](vcpu);
66326761
66336762 unexpected_vmexit:
6634− vcpu_unimpl(vcpu, "vmx: unexpected exit reason 0x%x\n",
6635− exit_reason.full);
66366763 dump_vmcs(vcpu);
6637− vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
6638− vcpu->run->internal.suberror =
6639− KVM_INTERNAL_ERROR_UNEXPECTED_EXIT_REASON;
6640− vcpu->run->internal.ndata = 2;
6641− vcpu->run->internal.data[0] = exit_reason.full;
6642− vcpu->run->internal.data[1] = vcpu->arch.last_vmentry_cpu;
6764+ kvm_prepare_unexpected_reason_exit(vcpu, exit_reason.full);
66436765 return 0;
66446766 }
66456767
@@ -6661,77 +6783,6 @@ int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) · 15 unmodified lines
66616783 return ret;
66626784 }
66636785
6664−/*
6665− * Software based L1D cache flush which is used when microcode providing
6666− * the cache control MSR is not loaded.
6667− *
6668− * The L1D cache is 32 KiB on Nehalem and later microarchitectures, but to
6669− * flush it is required to read in 64 KiB because the replacement algorithm
6670− * is not exactly LRU. This could be sized at runtime via topology
6671− * information but as all relevant affected CPUs have 32KiB L1D cache size
6672− * there is no point in doing so.
6673− */
6674−static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu)
6675−{
6676− int size = PAGE_SIZE << L1D_CACHE_ORDER;
6677−
6678− /*
6679− * This code is only executed when the flush mode is 'cond' or
6680− * 'always'
6681− */
6682− if (static_branch_likely(&vmx_l1d_flush_cond)) {
6683− bool flush_l1d;
6684−
6685− /*
6686− * Clear the per-vcpu flush bit, it gets set again if the vCPU
6687− * is reloaded, i.e. if the vCPU is scheduled out or if KVM
6688− * exits to userspace, or if KVM reaches one of the unsafe
6689− * VMEXIT handlers, e.g. if KVM calls into the emulator.
6690− */
6691− flush_l1d = vcpu->arch.l1tf_flush_l1d;
6692− vcpu->arch.l1tf_flush_l1d = false;
6693−
6694− /*
6695− * Clear the per-cpu flush bit, it gets set again from
6696− * the interrupt handlers.
6697− */
6698− flush_l1d |= kvm_get_cpu_l1tf_flush_l1d();
6699− kvm_clear_cpu_l1tf_flush_l1d();
6700−
6701− if (!flush_l1d)
6702− return;
6703− }
6704−
6705− vcpu->stat.l1d_flush++;
6706−
6707− if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) {
6708− native_wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH);
6709− return;
6710− }
6711−
6712− asm volatile(
6713− /* First ensure the pages are in the TLB */
6714− "xorl %%eax, %%eax\n"
6715− ".Lpopulate_tlb:\n\t"
6716− "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t"
6717− "addl $4096, %%eax\n\t"
6718− "cmpl %%eax, %[size]\n\t"
6719− "jne .Lpopulate_tlb\n\t"
6720− "xorl %%eax, %%eax\n\t"
6721− "cpuid\n\t"
6722− /* Now fill the cache */
6723− "xorl %%eax, %%eax\n"
6724− ".Lfill_cache:\n"
6725− "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t"
6726− "addl $64, %%eax\n\t"
6727− "cmpl %%eax, %[size]\n\t"
6728− "jne .Lfill_cache\n\t"
6729− "lfence\n"
6730− :: [flush_pages] "r" (vmx_l1d_flush_pages),
6731− [size] "r" (size)
6732− : "eax", "ebx", "ecx", "edx");
6733−}
6734−
67356786 void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr)
67366787 {
67376788 struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
@@ -7050,10 +7101,19 @@ void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu) · 312 unmodified lines
70507101 if (to_vt(vcpu)->emulation_required)
70517102 return;
70527103
7053− if (vmx_get_exit_reason(vcpu).basic == EXIT_REASON_EXTERNAL_INTERRUPT)
7104+ switch (vmx_get_exit_reason(vcpu).basic) {
7105+ case EXIT_REASON_EXTERNAL_INTERRUPT:
70547106 handle_external_interrupt_irqoff(vcpu, vmx_get_intr_info(vcpu));
7055− else if (vmx_get_exit_reason(vcpu).basic == EXIT_REASON_EXCEPTION_NMI)
7107+ break;
7108+ case EXIT_REASON_EXCEPTION_NMI:
70567109 handle_exception_irqoff(vcpu, vmx_get_intr_info(vcpu));
7110+ break;
7111+ case EXIT_REASON_MCE_DURING_VMENTRY:
7112+ kvm_machine_check();
7113+ break;
7114+ default:
7115+ break;
7116+ }
70577117 }
70587118
70597119 /*
@@ -7328,21 +7388,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, · 268 unmodified lines
73287388
73297389 guest_state_enter_irqoff();
73307390
7331− /*
7332− * L1D Flush includes CPU buffer clear to mitigate MDS, but VERW
7333− * mitigation for MDS is done late in VMentry and is still
7334− * executed in spite of L1D Flush. This is because an extra VERW
7335− * should not matter much after the big hammer L1D Flush.
7336− *
7337− * cpu_buf_vm_clear is used when system is not vulnerable to MDS/TAA,
7338− * and is affected by MMIO Stale Data. In such cases mitigation in only
7339− * needed against an MMIO capable guest.
7340− */
7341− if (static_branch_unlikely(&vmx_l1d_should_flush))
7342− vmx_l1d_flush(vcpu);
7343− else if (static_branch_unlikely(&cpu_buf_vm_clear) &&
7344− (flags & VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO))
7345− x86_clear_cpu_buffers();
7391+ vmx_l1d_flush(vcpu);
73467392
73477393 vmx_disable_fb_clear(vmx);
73487394
@@ -7454,8 +7500,6 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) · 105 unmodified lines
74547500 if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP)
74557501 vmx_set_interrupt_shadow(vcpu, 0);
74567502
7457− kvm_load_guest_xsave_state(vcpu);
7458−
74597503 pt_guest_enter(vmx);
74607504
74617505 atomic_switch_perf_msrs(vmx);
@@ -7499,8 +7543,6 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) · 37 unmodified lines
74997543
75007544 pt_guest_exit(vmx);
75017545
7502− kvm_load_host_xsave_state(vcpu);
7503−
75047546 if (is_guest_mode(vcpu)) {
75057547 /*
75067548 * Track VMLAUNCH/VMRESUME that have made past guest state
@@ -7516,9 +7558,6 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) · 9 unmodified lines
75167558 if (unlikely(vmx->fail))
75177559 return EXIT_FASTPATH_NONE;
75187560
7519− if (unlikely((u16)vmx_get_exit_reason(vcpu).basic == EXIT_REASON_MCE_DURING_VMENTRY))
7520− kvm_machine_check();
7521−
75227561 trace_kvm_exit(vcpu, KVM_ISA_VMX);
75237562
75247563 if (unlikely(vmx_get_exit_reason(vcpu).failed_vmentry))
@@ -8679,16 +8718,6 @@ __init int vmx_hardware_setup(void) · 1154 unmodified lines
86798718 return r;
86808719 }
86818720
8682−static void vmx_cleanup_l1d_flush(void)
8683−{
8684− if (vmx_l1d_flush_pages) {
8685− free_pages((unsigned long)vmx_l1d_flush_pages, L1D_CACHE_ORDER);
8686− vmx_l1d_flush_pages = NULL;
8687− }
8688− /* Restore state so sysfs ignores VMX */
8689− l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_AUTO;
8690−}
8691−
86928721 void vmx_exit(void)
86938722 {
86948723 allow_smaller_maxphyaddr = false;
@@ -8724,14 +8753,8 @@ int __init vmx_init(void) · 29 unmodified lines
87248753 if (r)
87258754 return r;
87268755
8727− /*
8728− * Must be called after common x86 init so enable_ept is properly set
8729− * up. Hand the parameter mitigation value in which was stored in
8730− * the pre module init parser. If no parameter was given, it will
8731− * contain 'auto' which will be turned into the default 'cond'
8732− * mitigation mode.
8733− */
8734− r = vmx_setup_l1d_flush(vmentry_l1d_flush_param);
8756+ /* Must be called after common x86 init so enable_ept is setup. */
8757+ r = vmx_setup_l1d_flush();
87358758 if (r)
87368759 goto err_l1d_flush;
87378760
arch/x86/kvm/vmx/vmx.h+0−2
@@ -369,7 +369,6 @@ void set_cr4_guest_host_mask(struct vcpu_vmx *vmx); · 368 unmodified lines
369369 void ept_save_pdptrs(struct kvm_vcpu *vcpu);
370370 void vmx_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg);
371371 void __vmx_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg);
372−u64 construct_eptp(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level);
373372
374373 bool vmx_guest_inject_ac(struct kvm_vcpu *vcpu);
375374 void vmx_update_exception_bitmap(struct kvm_vcpu *vcpu);
@@ -681,7 +680,6 @@ struct vmcs *alloc_vmcs_cpu(bool shadow, int cpu, gfp_t flags); · 305 unmodified lines
681680 void free_vmcs(struct vmcs *vmcs);
682681 int alloc_loaded_vmcs(struct loaded_vmcs *loaded_vmcs);
683682 void free_loaded_vmcs(struct loaded_vmcs *loaded_vmcs);
684−void loaded_vmcs_clear(struct loaded_vmcs *loaded_vmcs);
685683
686684 static inline struct vmcs *alloc_vmcs(bool shadow)
687685 {
arch/x86/kvm/vmx/x86_ops.h+1−1
@@ -73,7 +73,6 @@ void vmx_get_idt(struct kvm_vcpu *vcpu, struct desc_ptr *dt); · 72 unmodified lines
7373 void vmx_set_idt(struct kvm_vcpu *vcpu, struct desc_ptr *dt);
7474 void vmx_get_gdt(struct kvm_vcpu *vcpu, struct desc_ptr *dt);
7575 void vmx_set_gdt(struct kvm_vcpu *vcpu, struct desc_ptr *dt);
76−void vmx_set_dr6(struct kvm_vcpu *vcpu, unsigned long val);
7776 void vmx_set_dr7(struct kvm_vcpu *vcpu, unsigned long val);
7877 void vmx_sync_dirty_debug_regs(struct kvm_vcpu *vcpu);
7978 void vmx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg);
@@ -149,6 +148,7 @@ int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr); · 69 unmodified lines
149148 int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr);
150149
151150 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp);
151+int tdx_vcpu_unlocked_ioctl(struct kvm_vcpu *vcpu, void __user *argp);
152152
153153 void tdx_flush_tlb_current(struct kvm_vcpu *vcpu);
154154 void tdx_flush_tlb_all(struct kvm_vcpu *vcpu);
arch/x86/kvm/x86.c+156−129
@@ -159,9 +159,6 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(report_ignored_msrs); · 158 unmodified lines
159159 unsigned int min_timer_period_us = 200;
160160 module_param(min_timer_period_us, uint, 0644);
161161
162−static bool __read_mostly kvmclock_periodic_sync = true;
163−module_param(kvmclock_periodic_sync, bool, 0444);
164−
165162 /* tsc tolerance in parts per million - default to 1/2 of the NTP threshold */
166163 static u32 __read_mostly tsc_tolerance_ppm = 250;
167164 module_param(tsc_tolerance_ppm, uint, 0644);
@@ -212,7 +209,7 @@ struct kvm_user_return_msrs { · 44 unmodified lines
212209 u32 __read_mostly kvm_nr_uret_msrs;
213210 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_nr_uret_msrs);
214211 static u32 __read_mostly kvm_uret_msrs_list[KVM_MAX_NR_USER_RETURN_MSRS];
215−static struct kvm_user_return_msrs __percpu *user_return_msrs;
212+static DEFINE_PER_CPU(struct kvm_user_return_msrs, user_return_msrs);
216213
217214 #define KVM_SUPPORTED_XCR0 (XFEATURE_MASK_FP | XFEATURE_MASK_SSE \
218215 | XFEATURE_MASK_YMM | XFEATURE_MASK_BNDREGS \
@@ -575,24 +572,26 @@ static inline void kvm_async_pf_hash_reset(struct kvm_vcpu *vcpu) · 356 unmodified lines
575572 vcpu->arch.apf.gfns[i] = ~0;
576573 }
577574
575+static void kvm_destroy_user_return_msrs(void)
576+{
577+ int cpu;
578+
579+ for_each_possible_cpu(cpu)
580+ WARN_ON_ONCE(per_cpu(user_return_msrs, cpu).registered);
581+
582+ kvm_nr_uret_msrs = 0;
583+}
584+
578585 static void kvm_on_user_return(struct user_return_notifier *urn)
579586 {
580587 unsigned slot;
581588 struct kvm_user_return_msrs *msrs
582589 = container_of(urn, struct kvm_user_return_msrs, urn);
583590 struct kvm_user_return_msr_values *values;
584− unsigned long flags;
585591
586− /*
587− * Disabling irqs at this point since the following code could be
588− * interrupted and executed through kvm_arch_disable_virtualization_cpu()
589− */
590− local_irq_save(flags);
591− if (msrs->registered) {
592− msrs->registered = false;
593− user_return_notifier_unregister(urn);
594− }
595− local_irq_restore(flags);
592+ msrs->registered = false;
593+ user_return_notifier_unregister(urn);
594+
596595 for (slot = 0; slot < kvm_nr_uret_msrs; ++slot) {
597596 values = &msrs->values[slot];
598597 if (values->host != values->curr) {
@@ -643,7 +642,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_find_user_return_msr); · 44 unmodified lines
643642
644643 static void kvm_user_return_msr_cpu_online(void)
645644 {
646− struct kvm_user_return_msrs *msrs = this_cpu_ptr(user_return_msrs);
645+ struct kvm_user_return_msrs *msrs = this_cpu_ptr(&user_return_msrs);
647646 u64 value;
648647 int i;
649648
@@ -665,7 +664,7 @@ static void kvm_user_return_register_notifier(struct kvm_user_return_msrs *msrs) · 15 unmodified lines
665664
666665 int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask)
667666 {
668− struct kvm_user_return_msrs *msrs = this_cpu_ptr(user_return_msrs);
667+ struct kvm_user_return_msrs *msrs = this_cpu_ptr(&user_return_msrs);
669668 int err;
670669
671670 value = (value & mask) | (msrs->values[slot].host & ~mask);
@@ -681,24 +680,15 @@ int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask) · 9 unmodified lines
681680 }
682681 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_user_return_msr);
683682
684−void kvm_user_return_msr_update_cache(unsigned int slot, u64 value)
685−{
686− struct kvm_user_return_msrs *msrs = this_cpu_ptr(user_return_msrs);
687−
688− msrs->values[slot].curr = value;
689− kvm_user_return_register_notifier(msrs);
690−}
691−EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_user_return_msr_update_cache);
692−
693683 u64 kvm_get_user_return_msr(unsigned int slot)
694684 {
695− return this_cpu_ptr(user_return_msrs)->values[slot].curr;
685+ return this_cpu_ptr(&user_return_msrs)->values[slot].curr;
696686 }
697687 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_user_return_msr);
698688
699689 static void drop_user_return_notifiers(void)
700690 {
701− struct kvm_user_return_msrs *msrs = this_cpu_ptr(user_return_msrs);
691+ struct kvm_user_return_msrs *msrs = this_cpu_ptr(&user_return_msrs);
702692
703693 if (msrs->registered)
704694 kvm_on_user_return(&msrs->urn);
@@ -1045,6 +1035,13 @@ bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr) · 340 unmodified lines
10451035 }
10461036 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_require_dr);
10471037
1038+static bool kvm_pv_async_pf_enabled(struct kvm_vcpu *vcpu)
1039+{
1040+ u64 mask = KVM_ASYNC_PF_ENABLED | KVM_ASYNC_PF_DELIVERY_AS_INT;
1041+
1042+ return (vcpu->arch.apf.msr_en_val & mask) == mask;
1043+}
1044+
10481045 static inline u64 pdptr_rsvd_bits(struct kvm_vcpu *vcpu)
10491046 {
10501047 return vcpu->arch.reserved_gpa_bits | rsvd_bits(5, 8) | rsvd_bits(1, 2);
@@ -1137,15 +1134,20 @@ void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned lon · 86 unmodified lines
11371134 }
11381135
11391136 if ((cr0 ^ old_cr0) & X86_CR0_PG) {
1140− kvm_clear_async_pf_completion_queue(vcpu);
1141− kvm_async_pf_hash_reset(vcpu);
1142−
11431137 /*
11441138 * Clearing CR0.PG is defined to flush the TLB from the guest's
11451139 * perspective.
11461140 */
11471141 if (!(cr0 & X86_CR0_PG))
11481142 kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
1143+ /*
1144+ * Check for async #PF completion events when enabling paging,
1145+ * as the vCPU may have previously encountered async #PFs (it's
1146+ * entirely legal for the guest to toggle paging on/off without
1147+ * waiting for the async #PF queue to drain).
1148+ */
1149+ else if (kvm_pv_async_pf_enabled(vcpu))
1150+ kvm_make_request(KVM_REQ_APF_READY, vcpu);
11491151 }
11501152
11511153 if ((cr0 ^ old_cr0) & KVM_MMU_CR0_ROLE_BITS)
@@ -1203,20 +1205,27 @@ void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw) · 51 unmodified lines
12031205 }
12041206 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_lmsw);
12051207
1206−void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
1208+static void kvm_load_xfeatures(struct kvm_vcpu *vcpu, bool load_guest)
12071209 {
12081210 if (vcpu->arch.guest_state_protected)
12091211 return;
12101212
1211− if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
1213+ if (!kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE))
1214+ return;
12121215
1213− if (vcpu->arch.xcr0 != kvm_host.xcr0)
1214− xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
1216+ if (vcpu->arch.xcr0 != kvm_host.xcr0)
1217+ xsetbv(XCR_XFEATURE_ENABLED_MASK,
1218+ load_guest ? vcpu->arch.xcr0 : kvm_host.xcr0);
12151219
1216− if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) &&
1217− vcpu->arch.ia32_xss != kvm_host.xss)
1218− wrmsrq(MSR_IA32_XSS, vcpu->arch.ia32_xss);
1219− }
1220+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) &&
1221+ vcpu->arch.ia32_xss != kvm_host.xss)
1222+ wrmsrq(MSR_IA32_XSS, load_guest ? vcpu->arch.ia32_xss : kvm_host.xss);
1223+}
1224+
1225+static void kvm_load_guest_pkru(struct kvm_vcpu *vcpu)
1226+{
1227+ if (vcpu->arch.guest_state_protected)
1228+ return;
12201229
12211230 if (cpu_feature_enabled(X86_FEATURE_PKU) &&
12221231 vcpu->arch.pkru != vcpu->arch.host_pkru &&
@@ -1224,9 +1233,8 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu) · 1 unmodified line
12241233 kvm_is_cr4_bit_set(vcpu, X86_CR4_PKE)))
12251234 wrpkru(vcpu->arch.pkru);
12261235 }
1227−EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_load_guest_xsave_state);
12281236
1229−void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
1237+static void kvm_load_host_pkru(struct kvm_vcpu *vcpu)
12301238 {
12311239 if (vcpu->arch.guest_state_protected)
12321240 return;
@@ -1238,19 +1246,7 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu) · 5 unmodified lines
12381246 if (vcpu->arch.pkru != vcpu->arch.host_pkru)
12391247 wrpkru(vcpu->arch.host_pkru);
12401248 }
1241−
1242− if (kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE)) {
1243−
1244− if (vcpu->arch.xcr0 != kvm_host.xcr0)
1245− xsetbv(XCR_XFEATURE_ENABLED_MASK, kvm_host.xcr0);
1246−
1247− if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) &&
1248− vcpu->arch.ia32_xss != kvm_host.xss)
1249− wrmsrq(MSR_IA32_XSS, kvm_host.xss);
1250− }
1251−
12521249 }
1253−EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_load_host_xsave_state);
12541250
12551251 #ifdef CONFIG_X86_64
12561252 static inline u64 kvm_guest_supported_xfd(struct kvm_vcpu *vcpu)
@@ -3505,27 +3501,17 @@ uint64_t kvm_get_wall_clock_epoch(struct kvm *kvm) · 2248 unmodified lines
35053501 /*
35063502 * kvmclock updates which are isolated to a given vcpu, such as
35073503 * vcpu->cpu migration, should not allow system_timestamp from
3508− * the rest of the vcpus to remain static. Otherwise ntp frequency
3509− * correction applies to one vcpu's system_timestamp but not
3510− * the others.
3504+ * the rest of the vcpus to remain static.
35113505 *
35123506 * So in those cases, request a kvmclock update for all vcpus.
3513− * We need to rate-limit these requests though, as they can
3514− * considerably slow guests that have a large number of vcpus.
3515− * The time for a remote vcpu to update its kvmclock is bound
3516− * by the delay we use to rate-limit the updates.
3507+ * The worst case for a remote vcpu to update its kvmclock
3508+ * is then bounded by maximum nohz sleep latency.
35173509 */
3518−
3519−#define KVMCLOCK_UPDATE_DELAY msecs_to_jiffies(100)
3520−
3521−static void kvmclock_update_fn(struct work_struct *work)
3510+static void kvm_gen_kvmclock_update(struct kvm_vcpu *v)
35223511 {
35233512 unsigned long i;
3524− struct delayed_work *dwork = to_delayed_work(work);
3525− struct kvm_arch *ka = container_of(dwork, struct kvm_arch,
3526− kvmclock_update_work);
3527− struct kvm *kvm = container_of(ka, struct kvm, arch);
35283513 struct kvm_vcpu *vcpu;
3514+ struct kvm *kvm = v->kvm;
35293515
35303516 kvm_for_each_vcpu(i, vcpu, kvm) {
35313517 kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu);
@@ -3533,29 +3519,6 @@ static void kvmclock_update_fn(struct work_struct *work) · 1 unmodified line
35333519 }
35343520 }
35353521
3536−static void kvm_gen_kvmclock_update(struct kvm_vcpu *v)
3537−{
3538− struct kvm *kvm = v->kvm;
3539−
3540− kvm_make_request(KVM_REQ_CLOCK_UPDATE, v);
3541− schedule_delayed_work(&kvm->arch.kvmclock_update_work,
3542− KVMCLOCK_UPDATE_DELAY);
3543−}
3544−
3545−#define KVMCLOCK_SYNC_PERIOD (300 * HZ)
3546−
3547−static void kvmclock_sync_fn(struct work_struct *work)
3548−{
3549− struct delayed_work *dwork = to_delayed_work(work);
3550− struct kvm_arch *ka = container_of(dwork, struct kvm_arch,
3551− kvmclock_sync_work);
3552− struct kvm *kvm = container_of(ka, struct kvm, arch);
3553−
3554− schedule_delayed_work(&kvm->arch.kvmclock_update_work, 0);
3555− schedule_delayed_work(&kvm->arch.kvmclock_sync_work,
3556− KVMCLOCK_SYNC_PERIOD);
3557−}
3558−
35593522 /* These helpers are safe iff @msr is known to be an MCx bank MSR. */
35603523 static bool is_mci_control_msr(u32 msr)
35613524 {
@@ -3650,13 +3613,6 @@ static int set_msr_mce(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 88 unmodified lines
36503613 return 0;
36513614 }
36523615
3653−static inline bool kvm_pv_async_pf_enabled(struct kvm_vcpu *vcpu)
3654−{
3655− u64 mask = KVM_ASYNC_PF_ENABLED | KVM_ASYNC_PF_DELIVERY_AS_INT;
3656−
3657− return (vcpu->arch.apf.msr_en_val & mask) == mask;
3658−}
3659−
36603616 static int kvm_pv_enable_async_pf(struct kvm_vcpu *vcpu, u64 data)
36613617 {
36623618 gpa_t gpa = data & ~0x3f;
@@ -4182,7 +4138,12 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 519 unmodified lines
41824138 if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT))
41834139 return 1;
41844140 if (data & 0x1) {
4185− vcpu->arch.apf.pageready_pending = false;
4141+ /*
4142+ * Pairs with the smp_mb__after_atomic() in
4143+ * kvm_arch_async_page_present_queued().
4144+ */
4145+ smp_store_mb(vcpu->arch.apf.pageready_pending, false);
4146+
41864147 kvm_check_async_pf_completion(vcpu);
41874148 }
41884149 break;
@@ -5188,7 +5149,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) · 999 unmodified lines
51885149 {
51895150 struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
51905151
5191− vcpu->arch.l1tf_flush_l1d = true;
5152+ kvm_request_l1tf_flush_l1d();
51925153
51935154 if (vcpu->scheduled_out && pmu->version && pmu->event_count) {
51945155 pmu->need_cleanup = true;
@@ -7239,6 +7200,19 @@ static int kvm_vm_ioctl_set_clock(struct kvm *kvm, void __user *argp) · 2044 unmodified lines
72397200 return 0;
72407201 }
72417202
7203+long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl,
7204+ unsigned long arg)
7205+{
7206+ struct kvm_vcpu *vcpu = filp->private_data;
7207+ void __user *argp = (void __user *)arg;
7208+
7209+ if (ioctl == KVM_MEMORY_ENCRYPT_OP &&
7210+ kvm_x86_ops.vcpu_mem_enc_unlocked_ioctl)
7211+ return kvm_x86_call(vcpu_mem_enc_unlocked_ioctl)(vcpu, argp);
7212+
7213+ return -ENOIOCTLCMD;
7214+}
7215+
72427216 int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
72437217 {
72447218 struct kvm *kvm = filp->private_data;
@@ -7998,7 +7972,7 @@ int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu, gva_t addr, void *val, · 753 unmodified lines
79987972 unsigned int bytes, struct x86_exception *exception)
79997973 {
80007974 /* kvm_write_guest_virt_system can pull in tons of pages. */
8001− vcpu->arch.l1tf_flush_l1d = true;
7975+ kvm_request_l1tf_flush_l1d();
80027976
80037977 return kvm_write_guest_virt_helper(addr, val, bytes, vcpu,
80047978 PFERR_WRITE_MASK, exception);
@@ -8842,6 +8816,14 @@ static void emulator_triple_fault(struct x86_emulate_ctxt *ctxt) · 837 unmodified lines
88428816 kvm_make_request(KVM_REQ_TRIPLE_FAULT, emul_to_vcpu(ctxt));
88438817 }
88448818
8819+static int emulator_get_xcr(struct x86_emulate_ctxt *ctxt, u32 index, u64 *xcr)
8820+{
8821+ if (index != XCR_XFEATURE_ENABLED_MASK)
8822+ return 1;
8823+ *xcr = emul_to_vcpu(ctxt)->arch.xcr0;
8824+ return 0;
8825+}
8826+
88458827 static int emulator_set_xcr(struct x86_emulate_ctxt *ctxt, u32 index, u64 xcr)
88468828 {
88478829 return __kvm_set_xcr(emul_to_vcpu(ctxt), index, xcr);
@@ -8914,6 +8896,7 @@ static const struct x86_emulate_ops emulate_ops = { · 66 unmodified lines
89148896 .is_smm = emulator_is_smm,
89158897 .leave_smm = emulator_leave_smm,
89168898 .triple_fault = emulator_triple_fault,
8899+ .get_xcr = emulator_get_xcr,
89178900 .set_xcr = emulator_set_xcr,
89188901 .get_untagged_addr = emulator_get_untagged_addr,
89198902 .is_canonical_addr = emulator_is_canonical_addr,
@@ -9109,6 +9092,18 @@ void kvm_prepare_event_vectoring_exit(struct kvm_vcpu *vcpu, gpa_t gpa) · 189 unmodified lines
91099092 }
91109093 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_prepare_event_vectoring_exit);
91119094
9095+void kvm_prepare_unexpected_reason_exit(struct kvm_vcpu *vcpu, u64 exit_reason)
9096+{
9097+ vcpu_unimpl(vcpu, "unexpected exit reason 0x%llx\n", exit_reason);
9098+
9099+ vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
9100+ vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_UNEXPECTED_EXIT_REASON;
9101+ vcpu->run->internal.ndata = 2;
9102+ vcpu->run->internal.data[0] = exit_reason;
9103+ vcpu->run->internal.data[1] = vcpu->arch.last_vmentry_cpu;
9104+}
9105+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_prepare_unexpected_reason_exit);
9106+
91129107 static int handle_emulation_failure(struct kvm_vcpu *vcpu, int emulation_type)
91139108 {
91149109 struct kvm *kvm = vcpu->kvm;
@@ -9337,6 +9332,23 @@ static bool is_vmware_backdoor_opcode(struct x86_emulate_ctxt *ctxt) · 222 unmodified lines
93379332 return false;
93389333 }
93399334
9335+static bool is_soft_int_instruction(struct x86_emulate_ctxt *ctxt,
9336+ int emulation_type)
9337+{
9338+ u8 vector = EMULTYPE_GET_SOFT_INT_VECTOR(emulation_type);
9339+
9340+ switch (ctxt->b) {
9341+ case 0xcc:
9342+ return vector == BP_VECTOR;
9343+ case 0xcd:
9344+ return vector == ctxt->src.val;
9345+ case 0xce:
9346+ return vector == OF_VECTOR;
9347+ default:
9348+ return false;
9349+ }
9350+}
9351+
93409352 /*
93419353 * Decode an instruction for emulation. The caller is responsible for handling
93429354 * code breakpoints. Note, manually detecting code breakpoints is unnecessary
@@ -9394,7 +9406,7 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, · 51 unmodified lines
93949406 return handle_emulation_failure(vcpu, emulation_type);
93959407 }
93969408
9397− vcpu->arch.l1tf_flush_l1d = true;
9409+ kvm_request_l1tf_flush_l1d();
93989410
93999411 if (!(emulation_type & EMULTYPE_NO_DECODE)) {
94009412 kvm_clear_exception_queue(vcpu);
@@ -9447,6 +9459,10 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, · 46 unmodified lines
94479459 * injecting single-step #DBs.
94489460 */
94499461 if (emulation_type & EMULTYPE_SKIP) {
9462+ if (emulation_type & EMULTYPE_SKIP_SOFT_INT &&
9463+ !is_soft_int_instruction(ctxt, emulation_type))
9464+ return 0;
9465+
94509466 if (ctxt->mode != X86EMUL_MODE_PROT64)
94519467 ctxt->eip = (u32)ctxt->_eip;
94529468 else
@@ -10031,17 +10047,9 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) · 578 unmodified lines
1003110047 return -ENOMEM;
1003210048 }
1003310049
10034− user_return_msrs = alloc_percpu(struct kvm_user_return_msrs);
10035− if (!user_return_msrs) {
10036− pr_err("failed to allocate percpu kvm_user_return_msrs\n");
10037− r = -ENOMEM;
10038− goto out_free_x86_emulator_cache;
10039− }
10040− kvm_nr_uret_msrs = 0;
10041−
1004210050 r = kvm_mmu_vendor_module_init();
1004310051 if (r)
10044− goto out_free_percpu;
10052+ goto out_free_x86_emulator_cache;
1004510053
1004610054 kvm_caps.supported_vm_types = BIT(KVM_X86_DEFAULT_VM);
1004710055 kvm_caps.supported_mce_cap = MCG_CTL_P | MCG_SER_P;
@@ -10066,6 +10074,8 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) · 18 unmodified lines
1006610074 if (boot_cpu_has(X86_FEATURE_ARCH_CAPABILITIES))
1006710075 rdmsrq(MSR_IA32_ARCH_CAPABILITIES, kvm_host.arch_capabilities);
1006810076
10077+ WARN_ON_ONCE(kvm_nr_uret_msrs);
10078+
1006910079 r = ops->hardware_setup();
1007010080 if (r != 0)
1007110081 goto out_mmu_exit;
@@ -10138,9 +10148,8 @@ out_unwind_ops: · 66 unmodified lines
1013810148 kvm_x86_ops.enable_virtualization_cpu = NULL;
1013910149 kvm_x86_call(hardware_unsetup)();
1014010150 out_mmu_exit:
10151+ kvm_destroy_user_return_msrs();
1014110152 kvm_mmu_vendor_module_exit();
10142−out_free_percpu:
10143− free_percpu(user_return_msrs);
1014410153 out_free_x86_emulator_cache:
1014510154 kmem_cache_destroy(x86_emulator_cache);
1014610155 return r;
@@ -10168,8 +10177,8 @@ void kvm_x86_vendor_exit(void) · 21 unmodified lines
1016810177 cancel_work_sync(&pvclock_gtod_work);
1016910178 #endif
1017010179 kvm_x86_call(hardware_unsetup)();
10180+ kvm_destroy_user_return_msrs();
1017110181 kvm_mmu_vendor_module_exit();
10172− free_percpu(user_return_msrs);
1017310182 kmem_cache_destroy(x86_emulator_cache);
1017410183 #ifdef CONFIG_KVM_XEN
1017510184 static_key_deferred_flush(&kvm_xen_enabled);
@@ -11291,6 +11300,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · 1115 unmodified lines
1129111300 if (vcpu->arch.guest_fpu.xfd_err)
1129211301 wrmsrq(MSR_IA32_XFD_ERR, vcpu->arch.guest_fpu.xfd_err);
1129311302
11303+ kvm_load_xfeatures(vcpu, true);
11304+
1129411305 if (unlikely(vcpu->arch.switch_db_regs &&
1129511306 !(vcpu->arch.switch_db_regs & KVM_DEBUGREG_AUTO_SWITCH))) {
1129611307 set_debugreg(DR7_FIXED_1, 7);
@@ -11319,6 +11330,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · 22 unmodified lines
1131911330
1132011331 guest_timing_enter_irqoff();
1132111332
11333+ /*
11334+ * Swap PKRU with hardware breakpoints disabled to minimize the number
11335+ * of flows where non-KVM code can run with guest state loaded.
11336+ */
11337+ kvm_load_guest_pkru(vcpu);
11338+
1132211339 for (;;) {
1132311340 /*
1132411341 * Assert that vCPU vs. VM APICv state is consistent. An APICv
@@ -11347,6 +11364,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · 22 unmodified lines
1134711364 ++vcpu->stat.exits;
1134811365 }
1134911366
11367+ kvm_load_host_pkru(vcpu);
11368+
1135011369 /*
1135111370 * Do this here before restoring debug registers on the host. And
1135211371 * since we do this before handling the vmexit, a DR access vmexit
@@ -11377,6 +11396,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · 24 unmodified lines
1137711396 vcpu->mode = OUTSIDE_GUEST_MODE;
1137811397 smp_wmb();
1137911398
11399+ kvm_load_xfeatures(vcpu, false);
11400+
1138011401 /*
1138111402 * Sync xfd before calling handle_exit_irqoff() which may
1138211403 * rely on the fact that guest_fpu::xfd is up-to-date (e.g.
@@ -12734,8 +12755,6 @@ fail_mmu_destroy: · 1351 unmodified lines
1273412755
1273512756 void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
1273612757 {
12737− struct kvm *kvm = vcpu->kvm;
12738−
1273912758 if (mutex_lock_killable(&vcpu->mutex))
1274012759 return;
1274112760 vcpu_load(vcpu);
@@ -12746,10 +12765,6 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu) · 4 unmodified lines
1274612765 vcpu->arch.msr_kvm_poll_control = 1;
1274712766
1274812767 mutex_unlock(&vcpu->mutex);
12749−
12750− if (kvmclock_periodic_sync && vcpu->vcpu_idx == 0)
12751− schedule_delayed_work(&kvm->arch.kvmclock_sync_work,
12752− KVMCLOCK_SYNC_PERIOD);
1275312768 }
1275412769
1275512770 void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
@@ -13088,7 +13103,21 @@ int kvm_arch_enable_virtualization_cpu(void) · 332 unmodified lines
1308813103 void kvm_arch_disable_virtualization_cpu(void)
1308913104 {
1309013105 kvm_x86_call(disable_virtualization_cpu)();
13091− drop_user_return_notifiers();
13106+
13107+ /*
13108+ * Leave the user-return notifiers as-is when disabling virtualization
13109+ * for reboot, i.e. when disabling via IPI function call, and instead
13110+ * pin kvm.ko (if it's a module) to defend against use-after-free (in
13111+ * the *very* unlikely scenario module unload is racing with reboot).
13112+ * On a forced reboot, tasks aren't frozen before shutdown, and so KVM
13113+ * could be actively modifying user-return MSR state when the IPI to
13114+ * disable virtualization arrives. Handle the extreme edge case here
13115+ * instead of trying to account for it in the normal flows.
13116+ */
13117+ if (in_task() || WARN_ON_ONCE(!kvm_rebooting))
13118+ drop_user_return_notifiers();
13119+ else
13120+ __module_get(THIS_MODULE);
1309213121 }
1309313122
1309413123 bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu)
@@ -13160,9 +13189,6 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) · 65 unmodified lines
1316013189 kvm->arch.hv_root_tdp = INVALID_PAGE;
1316113190 #endif
1316213191
13163− INIT_DELAYED_WORK(&kvm->arch.kvmclock_update_work, kvmclock_update_fn);
13164− INIT_DELAYED_WORK(&kvm->arch.kvmclock_sync_work, kvmclock_sync_fn);
13165−
1316613192 kvm_apicv_init(kvm);
1316713193 kvm_hv_init_vm(kvm);
1316813194 kvm_xen_init_vm(kvm);
@@ -13269,9 +13295,6 @@ void kvm_arch_pre_destroy_vm(struct kvm *kvm) · 100 unmodified lines
1326913295 * is unsafe, i.e. will lead to use-after-free. The PIT also needs to
1327013296 * be stopped before IRQ routing is freed.
1327113297 */
13272− cancel_delayed_work_sync(&kvm->arch.kvmclock_sync_work);
13273− cancel_delayed_work_sync(&kvm->arch.kvmclock_update_work);
13274−
1327513298 #ifdef CONFIG_KVM_IOAPIC
1327613299 kvm_free_pit(kvm);
1327713300 #endif
@@ -13888,7 +13911,7 @@ void kvm_arch_async_page_present(struct kvm_vcpu *vcpu, · 610 unmodified lines
1388813911 if ((work->wakeup_all || work->notpresent_injected) &&
1388913912 kvm_pv_async_pf_enabled(vcpu) &&
1389013913 !apf_put_user_ready(vcpu, work->arch.token)) {
13891− vcpu->arch.apf.pageready_pending = true;
13914+ WRITE_ONCE(vcpu->arch.apf.pageready_pending, true);
1389213915 kvm_apic_set_irq(vcpu, &irq, NULL);
1389313916 }
1389413917
@@ -13899,7 +13922,11 @@ void kvm_arch_async_page_present(struct kvm_vcpu *vcpu, · 4 unmodified lines
1389913922 void kvm_arch_async_page_present_queued(struct kvm_vcpu *vcpu)
1390013923 {
1390113924 kvm_make_request(KVM_REQ_APF_READY, vcpu);
13902− if (!vcpu->arch.apf.pageready_pending)
13925+
13926+ /* Pairs with smp_store_mb() in kvm_set_msr_common(). */
13927+ smp_mb__after_atomic();
13928+
13929+ if (!READ_ONCE(vcpu->arch.apf.pageready_pending))
1390313930 kvm_vcpu_kick(vcpu);
1390413931 }
1390513932
arch/x86/kvm/x86.h+14−2
@@ -420,6 +420,20 @@ static inline bool kvm_check_has_quirk(struct kvm *kvm, u64 quirk) · 419 unmodified lines
420420 return !(kvm->arch.disabled_quirks & quirk);
421421 }
422422
423+static __always_inline void kvm_request_l1tf_flush_l1d(void)
424+{
425+#if IS_ENABLED(CONFIG_CPU_MITIGATIONS) && IS_ENABLED(CONFIG_KVM_INTEL)
426+ /*
427+ * Use a raw write to set the per-CPU flag, as KVM will ensure a flush
428+ * even if preemption is currently enabled.. If the current vCPU task
429+ * is migrated to a different CPU (or userspace runs the vCPU on a
430+ * different task) before the next VM-Entry, then kvm_arch_vcpu_load()
431+ * will request a flush on the new CPU.
432+ */
433+ raw_cpu_write(irq_stat.kvm_cpu_l1tf_flush_l1d, 1);
434+#endif
435+}
436+
423437 void kvm_inject_realmode_interrupt(struct kvm_vcpu *vcpu, int irq, int inc_eip);
424438
425439 u64 get_kvmclock_ns(struct kvm *kvm);
@@ -622,8 +636,6 @@ static inline void kvm_machine_check(void) · 196 unmodified lines
622636 #endif
623637 }
624638
625−void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu);
626−void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu);
627639 int kvm_spec_ctrl_test_value(u64 value);
628640 int kvm_handle_memory_failure(struct kvm_vcpu *vcpu, int r,
629641 struct x86_exception *e);
drivers/crypto/ccp/sev-dev.c+37−0
@@ -2770,6 +2770,43 @@ void sev_platform_shutdown(void) · 2769 unmodified lines
27702770 }
27712771 EXPORT_SYMBOL_GPL(sev_platform_shutdown);
27722772
2773+u64 sev_get_snp_policy_bits(void)
2774+{
2775+ struct psp_device *psp = psp_master;
2776+ struct sev_device *sev;
2777+ u64 policy_bits;
2778+
2779+ if (!cc_platform_has(CC_ATTR_HOST_SEV_SNP))
2780+ return 0;
2781+
2782+ if (!psp || !psp->sev_data)
2783+ return 0;
2784+
2785+ sev = psp->sev_data;
2786+
2787+ policy_bits = SNP_POLICY_MASK_BASE;
2788+
2789+ if (sev->snp_plat_status.feature_info) {
2790+ if (sev->snp_feat_info_0.ecx & SNP_RAPL_DISABLE_SUPPORTED)
2791+ policy_bits |= SNP_POLICY_MASK_RAPL_DIS;
2792+
2793+ if (sev->snp_feat_info_0.ecx & SNP_CIPHER_TEXT_HIDING_SUPPORTED)
2794+ policy_bits |= SNP_POLICY_MASK_CIPHERTEXT_HIDING_DRAM;
2795+
2796+ if (sev->snp_feat_info_0.ecx & SNP_AES_256_XTS_POLICY_SUPPORTED)
2797+ policy_bits |= SNP_POLICY_MASK_MEM_AES_256_XTS;
2798+
2799+ if (sev->snp_feat_info_0.ecx & SNP_CXL_ALLOW_POLICY_SUPPORTED)
2800+ policy_bits |= SNP_POLICY_MASK_CXL_ALLOW;
2801+
2802+ if (sev_version_greater_or_equal(1, 58))
2803+ policy_bits |= SNP_POLICY_MASK_PAGE_SWAP_DISABLE;
2804+ }
2805+
2806+ return policy_bits;
2807+}
2808+EXPORT_SYMBOL_GPL(sev_get_snp_policy_bits);
2809+
27732810 void sev_dev_destroy(struct psp_device *psp)
27742811 {
27752812 struct sev_device *sev = psp->sev_data;
drivers/irqchip/irq-apple-aic.c+5−2
@@ -411,12 +411,15 @@ static void __exception_irq_entry aic_handle_irq(struct pt_regs *regs) · 410 unmodified lines
411411 if (is_kernel_in_hyp_mode() &&
412412 (read_sysreg_s(SYS_ICH_HCR_EL2) & ICH_HCR_EL2_En) &&
413413 read_sysreg_s(SYS_ICH_MISR_EL2) != 0) {
414+ u64 val;
415+
414416 generic_handle_domain_irq(aic_irqc->hw_domain,
415417 AIC_FIQ_HWIRQ(AIC_VGIC_MI));
416418
417419 if (unlikely((read_sysreg_s(SYS_ICH_HCR_EL2) & ICH_HCR_EL2_En) &&
418− read_sysreg_s(SYS_ICH_MISR_EL2))) {
419− pr_err_ratelimited("vGIC IRQ fired and not handled by KVM, disabling.\n");
420+ (val = read_sysreg_s(SYS_ICH_MISR_EL2)))) {
421+ pr_err_ratelimited("vGIC IRQ fired and not handled by KVM (MISR=%llx), disabling.\n",
422+ val);
420423 sysreg_clear_set_s(SYS_ICH_HCR_EL2, ICH_HCR_EL2_En, 0);
421424 }
422425 }
drivers/irqchip/irq-gic.c+3−0
@@ -1459,6 +1459,8 @@ static void __init gic_of_setup_kvm_info(struct device_node *node) · 1458 unmodified lines
14591459 if (ret)
14601460 return;
14611461
1462+ gic_v2_kvm_info.gicc_base = gic_data[0].cpu_base.common_base;
1463+
14621464 if (static_branch_likely(&supports_deactivate_key))
14631465 vgic_set_kvm_info(&gic_v2_kvm_info);
14641466 }
@@ -1620,6 +1622,7 @@ static void __init gic_acpi_setup_kvm_info(void) · 155 unmodified lines
16201622 return;
16211623
16221624 gic_v2_kvm_info.maint_irq = irq;
1625+ gic_v2_kvm_info.gicc_base = gic_data[0].cpu_base.common_base;
16231626
16241627 vgic_set_kvm_info(&gic_v2_kvm_info);
16251628 }
fs/btrfs/compression.c+2−2
@@ -475,8 +475,8 @@ static noinline int add_ra_bio_pages(struct inode *inode, · 474 unmodified lines
475475 continue;
476476 }
477477
478− folio = filemap_alloc_folio(mapping_gfp_constraint(mapping,
479− ~__GFP_FS), 0);
478+ folio = filemap_alloc_folio(mapping_gfp_constraint(mapping, ~__GFP_FS),
479+ 0, NULL);
480480 if (!folio)
481481 break;
482482
fs/btrfs/verity.c+1−1
@@ -736,7 +736,7 @@ again: · 735 unmodified lines
736736 }
737737
738738 folio = filemap_alloc_folio(mapping_gfp_constraint(inode->i_mapping, ~__GFP_FS),
739− 0);
739+ 0, NULL);
740740 if (!folio)
741741 return ERR_PTR(-ENOMEM);
742742
fs/erofs/zdata.c+1−1
@@ -562,7 +562,7 @@ static void z_erofs_bind_cache(struct z_erofs_frontend *fe) · 561 unmodified lines
562562 * Allocate a managed folio for cached I/O, or it may be
563563 * then filled with a file-backed folio for in-place I/O
564564 */
565− newfolio = filemap_alloc_folio(gfp, 0);
565+ newfolio = filemap_alloc_folio(gfp, 0, NULL);
566566 if (!newfolio)
567567 continue;
568568 newfolio->private = Z_EROFS_PREALLOCATED_FOLIO;
fs/f2fs/compress.c+1−1
@@ -1947,7 +1947,7 @@ static void f2fs_cache_compressed_page(struct f2fs_sb_info *sbi, · 1946 unmodified lines
19471947 return;
19481948 }
19491949
1950− cfolio = filemap_alloc_folio(__GFP_NOWARN | __GFP_IO, 0);
1950+ cfolio = filemap_alloc_folio(__GFP_NOWARN | __GFP_IO, 0, NULL);
19511951 if (!cfolio)
19521952 return;
19531953
include/kvm/arm_vgic.h+19−10
@@ -59,6 +59,9 @@ struct vgic_global { · 58 unmodified lines
5959 /* virtual control interface mapping, HYP VA */
6060 void __iomem *vctrl_hyp;
6161
62+ /* Physical CPU interface, kernel VA */
63+ void __iomem *gicc_base;
64+
6265 /* Number of implemented list registers */
6366 int nr_lr;
6467
@@ -120,6 +123,7 @@ struct irq_ops { · 55 unmodified lines
120123
121124 struct vgic_irq {
122125 raw_spinlock_t irq_lock; /* Protects the content of the struct */
126+ u32 intid; /* Guest visible INTID */
123127 struct rcu_head rcu;
124128 struct list_head ap_list;
125129
@@ -134,17 +138,18 @@ struct vgic_irq { · 8 unmodified lines
134138 * affinity reg (v3).
135139 */
136140
137− u32 intid; /* Guest visible INTID */
138− bool line_level; /* Level only */
139− bool pending_latch; /* The pending latch state used to calculate
140− * the pending state for both level
141− * and edge triggered IRQs. */
142− bool active;
143− bool pending_release; /* Used for LPIs only, unreferenced IRQ
141+ bool pending_release:1; /* Used for LPIs only, unreferenced IRQ
144142 * pending a release */
145143
146− bool enabled;
147− bool hw; /* Tied to HW IRQ */
144+ bool pending_latch:1; /* The pending latch state used to calculate
145+ * the pending state for both level
146+ * and edge triggered IRQs. */
147+ enum vgic_irq_config config:1; /* Level or edge */
148+ bool line_level:1; /* Level only */
149+ bool enabled:1;
150+ bool active:1;
151+ bool hw:1; /* Tied to HW IRQ */
152+ bool on_lr:1; /* Present in a CPU LR */
148153 refcount_t refcount; /* Used for LPIs */
149154 u32 hwintid; /* HW INTID number */
150155 unsigned int host_irq; /* linux irq corresponding to hwintid */
@@ -156,7 +161,6 @@ struct vgic_irq { · 5 unmodified lines
156161 u8 active_source; /* GICv2 SGIs only */
157162 u8 priority;
158163 u8 group; /* 0 == group 0, 1 == group 1 */
159− enum vgic_irq_config config; /* Level or edge */
160164
161165 struct irq_ops *ops;
162166
@@ -259,6 +263,9 @@ struct vgic_dist { · 96 unmodified lines
259263 /* The GIC maintenance IRQ for nested hypervisors. */
260264 u32 mi_intid;
261265
266+ /* Track the number of in-flight active SPIs */
267+ atomic_t active_spis;
268+
262269 /* base addresses in guest physical address space: */
263270 gpa_t vgic_dist_base; /* distributor */
264271 union {
@@ -280,6 +287,7 @@ struct vgic_dist { · 15 unmodified lines
280287 struct vgic_irq *spis;
281288
282289 struct vgic_io_device dist_iodev;
290+ struct vgic_io_device cpuif_iodev;
283291
284292 bool has_its;
285293 bool table_write_in_progress;
@@ -417,6 +425,7 @@ bool kvm_vcpu_has_pending_irqs(struct kvm_vcpu *vcpu); · 131 unmodified lines
417425 void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu);
418426 void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu);
419427 void kvm_vgic_reset_mapped_irq(struct kvm_vcpu *vcpu, u32 vintid);
428+void kvm_vgic_process_async_update(struct kvm_vcpu *vcpu);
420429
421430 void vgic_v3_dispatch_sgi(struct kvm_vcpu *vcpu, u64 reg, bool allow_group1);
422431
include/linux/irqchip/arm-gic.h+6−0
@@ -86,7 +86,13 @@ · 85 unmodified lines
8686
8787 #define GICH_HCR_EN (1 << 0)
8888 #define GICH_HCR_UIE (1 << 1)
89+#define GICH_HCR_LRENPIE (1 << 2)
8990 #define GICH_HCR_NPIE (1 << 3)
91+#define GICH_HCR_VGrp0EIE (1 << 4)
92+#define GICH_HCR_VGrp0DIE (1 << 5)
93+#define GICH_HCR_VGrp1EIE (1 << 6)
94+#define GICH_HCR_VGrp1DIE (1 << 7)
95+#define GICH_HCR_EOICOUNT GENMASK(31, 27)
9096
9197 #define GICH_LR_VIRTUALID (0x3ff << 0)
9298 #define GICH_LR_PHYSID_CPUID_SHIFT (10)
include/linux/irqchip/arm-vgic-info.h+2−0
@@ -24,6 +24,8 @@ struct gic_kvm_info { · 23 unmodified lines
2424 enum gic_type type;
2525 /* Virtual CPU interface */
2626 struct resource vcpu;
27+ /* GICv2 GICC VA */
28+ void __iomem *gicc_base;
2729 /* Interrupt number */
2830 unsigned int maint_irq;
2931 /* No interrupt mask, no need to use the above field */
include/linux/kvm_host.h+2−12
@@ -1557,6 +1557,8 @@ long kvm_arch_dev_ioctl(struct file *filp, · 1556 unmodified lines
15571557 unsigned int ioctl, unsigned long arg);
15581558 long kvm_arch_vcpu_ioctl(struct file *filp,
15591559 unsigned int ioctl, unsigned long arg);
1560+long kvm_arch_vcpu_unlocked_ioctl(struct file *filp,
1561+ unsigned int ioctl, unsigned long arg);
15601562 vm_fault_t kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf);
15611563
15621564 int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext);
@@ -2437,18 +2439,6 @@ static inline bool kvm_arch_no_poll(struct kvm_vcpu *vcpu) · 874 unmodified lines
24372439 }
24382440 #endif /* CONFIG_HAVE_KVM_NO_POLL */
24392441
2440−#ifdef CONFIG_HAVE_KVM_VCPU_ASYNC_IOCTL
2441−long kvm_arch_vcpu_async_ioctl(struct file *filp,
2442− unsigned int ioctl, unsigned long arg);
2443−#else
2444−static inline long kvm_arch_vcpu_async_ioctl(struct file *filp,
2445− unsigned int ioctl,
2446− unsigned long arg)
2447−{
2448− return -ENOIOCTLCMD;
2449−}
2450−#endif /* CONFIG_HAVE_KVM_VCPU_ASYNC_IOCTL */
2451−
24522442 void kvm_arch_guest_memory_reclaimed(struct kvm *kvm);
24532443
24542444 #ifdef CONFIG_HAVE_KVM_VCPU_RUN_PID_CHANGE
include/linux/pagemap.h+13−5
@@ -651,9 +651,11 @@ static inline void *detach_page_private(struct page *page) · 650 unmodified lines
651651 }
652652
653653 #ifdef CONFIG_NUMA
654−struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order);
654+struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order,
655+ struct mempolicy *policy);
655656 #else
656−static inline struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order)
657+static inline struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order,
658+ struct mempolicy *policy)
657659 {
658660 return folio_alloc_noprof(gfp, order);
659661 }
@@ -664,7 +666,7 @@ static inline struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int o · 4 unmodified lines
664666
665667 static inline struct page *__page_cache_alloc(gfp_t gfp)
666668 {
667− return &filemap_alloc_folio(gfp, 0)->page;
669+ return &filemap_alloc_folio(gfp, 0, NULL)->page;
668670 }
669671
670672 static inline gfp_t readahead_gfp_mask(struct address_space *x)
@@ -750,11 +752,17 @@ static inline fgf_t fgf_set_order(size_t size) · 79 unmodified lines
750752 }
751753
752754 void *filemap_get_entry(struct address_space *mapping, pgoff_t index);
753−struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,
754− fgf_t fgp_flags, gfp_t gfp);
755+struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
756+ pgoff_t index, fgf_t fgf_flags, gfp_t gfp, struct mempolicy *policy);
755757 struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
756758 fgf_t fgp_flags, gfp_t gfp);
757759
760+static inline struct folio *__filemap_get_folio(struct address_space *mapping,
761+ pgoff_t index, fgf_t fgf_flags, gfp_t gfp)
762+{
763+ return __filemap_get_folio_mpol(mapping, index, fgf_flags, gfp, NULL);
764+}
765+
758766 /**
759767 * write_begin_get_folio - Get folio for write_begin with flags.
760768 * @iocb: The kiocb passed from write_begin (may be NULL).
include/linux/psp-sev.h+37−0
@@ -14,6 +14,39 @@ · 13 unmodified lines
1414
1515 #include <uapi/linux/psp-sev.h>
1616
17+/* As defined by SEV API, under "Guest Policy". */
18+#define SEV_POLICY_MASK_NODBG BIT(0)
19+#define SEV_POLICY_MASK_NOKS BIT(1)
20+#define SEV_POLICY_MASK_ES BIT(2)
21+#define SEV_POLICY_MASK_NOSEND BIT(3)
22+#define SEV_POLICY_MASK_DOMAIN BIT(4)
23+#define SEV_POLICY_MASK_SEV BIT(5)
24+#define SEV_POLICY_MASK_API_MAJOR GENMASK(23, 16)
25+#define SEV_POLICY_MASK_API_MINOR GENMASK(31, 24)
26+
27+/* As defined by SEV-SNP Firmware ABI, under "Guest Policy". */
28+#define SNP_POLICY_MASK_API_MINOR GENMASK_ULL(7, 0)
29+#define SNP_POLICY_MASK_API_MAJOR GENMASK_ULL(15, 8)
30+#define SNP_POLICY_MASK_SMT BIT_ULL(16)
31+#define SNP_POLICY_MASK_RSVD_MBO BIT_ULL(17)
32+#define SNP_POLICY_MASK_MIGRATE_MA BIT_ULL(18)
33+#define SNP_POLICY_MASK_DEBUG BIT_ULL(19)
34+#define SNP_POLICY_MASK_SINGLE_SOCKET BIT_ULL(20)
35+#define SNP_POLICY_MASK_CXL_ALLOW BIT_ULL(21)
36+#define SNP_POLICY_MASK_MEM_AES_256_XTS BIT_ULL(22)
37+#define SNP_POLICY_MASK_RAPL_DIS BIT_ULL(23)
38+#define SNP_POLICY_MASK_CIPHERTEXT_HIDING_DRAM BIT_ULL(24)
39+#define SNP_POLICY_MASK_PAGE_SWAP_DISABLE BIT_ULL(25)
40+
41+/* Base SEV-SNP policy bitmask for minimum supported SEV firmware version */
42+#define SNP_POLICY_MASK_BASE (SNP_POLICY_MASK_API_MINOR | \
43+ SNP_POLICY_MASK_API_MAJOR | \
44+ SNP_POLICY_MASK_SMT | \
45+ SNP_POLICY_MASK_RSVD_MBO | \
46+ SNP_POLICY_MASK_MIGRATE_MA | \
47+ SNP_POLICY_MASK_DEBUG | \
48+ SNP_POLICY_MASK_SINGLE_SOCKET)
49+
1750 #define SEV_FW_BLOB_MAX_SIZE 0x4000 /* 16KB */
1851
1952 /**
@@ -849,7 +882,10 @@ struct snp_feature_info { · 829 unmodified lines
849882 u32 edx;
850883 } __packed;
851884
885+#define SNP_RAPL_DISABLE_SUPPORTED BIT(2)
852886 #define SNP_CIPHER_TEXT_HIDING_SUPPORTED BIT(3)
887+#define SNP_AES_256_XTS_POLICY_SUPPORTED BIT(4)
888+#define SNP_CXL_ALLOW_POLICY_SUPPORTED BIT(5)
853889
854890 #ifdef CONFIG_CRYPTO_DEV_SP_PSP
855891
@@ -995,6 +1031,7 @@ void *snp_alloc_firmware_page(gfp_t mask); · 139 unmodified lines
9951031 void snp_free_firmware_page(void *addr);
9961032 void sev_platform_shutdown(void);
9971033 bool sev_is_snp_ciphertext_hiding_supported(void);
1034+u64 sev_get_snp_policy_bits(void);
9981035
9991036 #else /* !CONFIG_CRYPTO_DEV_SP_PSP */
10001037
include/uapi/linux/kvm.h+11−0
@@ -179,6 +179,7 @@ struct kvm_xen_exit { · 178 unmodified lines
179179 #define KVM_EXIT_LOONGARCH_IOCSR 38
180180 #define KVM_EXIT_MEMORY_FAULT 39
181181 #define KVM_EXIT_TDX 40
182+#define KVM_EXIT_ARM_SEA 41
182183
183184 /* For KVM_EXIT_INTERNAL_ERROR */
184185 /* Emulate instruction failed. */
@@ -473,6 +474,14 @@ struct kvm_run { · 288 unmodified lines
473474 } setup_event_notify;
474475 };
475476 } tdx;
477+ /* KVM_EXIT_ARM_SEA */
478+ struct {
479+#define KVM_EXIT_ARM_SEA_FLAG_GPA_VALID (1ULL << 0)
480+ __u64 flags;
481+ __u64 esr;
482+ __u64 gva;
483+ __u64 gpa;
484+ } arm_sea;
476485 /* Fix the size of the union. */
477486 char padding[256];
478487 };
@@ -963,6 +972,8 @@ struct kvm_enable_cap { · 484 unmodified lines
963972 #define KVM_CAP_RISCV_MP_STATE_RESET 242
964973 #define KVM_CAP_ARM_CACHEABLE_PFNMAP_SUPPORTED 243
965974 #define KVM_CAP_GUEST_MEMFD_FLAGS 244
975+#define KVM_CAP_ARM_SEA_TO_USER 245
976+#define KVM_CAP_S390_USER_OPEREXEC 246
966977
967978 struct kvm_irq_routing_irqchip {
968979 __u32 irqchip;
include/uapi/linux/magic.h+1−0
@@ -103,5 +103,6 @@ · 102 unmodified lines
103103 #define DEVMEM_MAGIC 0x454d444d /* "DMEM" */
104104 #define SECRETMEM_MAGIC 0x5345434d /* "SECM" */
105105 #define PID_FS_MAGIC 0x50494446 /* "PIDF" */
106+#define GUEST_MEMFD_MAGIC 0x474d454d /* "GMEM" */
106107
107108 #endif /* __LINUX_MAGIC_H__ */
mm/filemap.c+14−9
@@ -990,11 +990,16 @@ int filemap_add_folio(struct address_space *mapping, struct folio *folio, · 989 unmodified lines
990990 EXPORT_SYMBOL_GPL(filemap_add_folio);
991991
992992 #ifdef CONFIG_NUMA
993−struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order)
993+struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order,
994+ struct mempolicy *policy)
994995 {
995996 int n;
996997 struct folio *folio;
997998
999+ if (policy)
1000+ return folio_alloc_mpol_noprof(gfp, order, policy,
1001+ NO_INTERLEAVE_INDEX, numa_node_id());
1002+
9981003 if (cpuset_do_page_mem_spread()) {
9991004 unsigned int cpuset_mems_cookie;
10001005 do {
@@ -1911,11 +1916,12 @@ out: · 910 unmodified lines
19111916 }
19121917
19131918 /**
1914− * __filemap_get_folio - Find and get a reference to a folio.
1919+ * __filemap_get_folio_mpol - Find and get a reference to a folio.
19151920 * @mapping: The address_space to search.
19161921 * @index: The page index.
19171922 * @fgp_flags: %FGP flags modify how the folio is returned.
19181923 * @gfp: Memory allocation flags to use if %FGP_CREAT is specified.
1924+ * @policy: NUMA memory allocation policy to follow.
19191925 *
19201926 * Looks up the page cache entry at @mapping & @index.
19211927 *
@@ -1926,8 +1932,8 @@ out: · 4 unmodified lines
19261932 *
19271933 * Return: The found folio or an ERR_PTR() otherwise.
19281934 */
1929−struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,
1930− fgf_t fgp_flags, gfp_t gfp)
1935+struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
1936+ pgoff_t index, fgf_t fgp_flags, gfp_t gfp, struct mempolicy *policy)
19311937 {
19321938 struct folio *folio;
19331939
@@ -1997,7 +2003,7 @@ no_page: · 63 unmodified lines
19972003 err = -ENOMEM;
19982004 if (order > min_order)
19992005 alloc_gfp |= __GFP_NORETRY | __GFP_NOWARN;
2000− folio = filemap_alloc_folio(alloc_gfp, order);
2006+ folio = filemap_alloc_folio(alloc_gfp, order, policy);
20012007 if (!folio)
20022008 continue;
20032009
@@ -2044,7 +2050,7 @@ no_page: · 40 unmodified lines
20442050 folio_clear_dropbehind(folio);
20452051 return folio;
20462052 }
2047−EXPORT_SYMBOL(__filemap_get_folio);
2053+EXPORT_SYMBOL(__filemap_get_folio_mpol);
20482054
20492055 static inline struct folio *find_get_entry(struct xa_state *xas, pgoff_t max,
20502056 xa_mark_t mark)
@@ -2597,7 +2603,7 @@ static int filemap_create_folio(struct kiocb *iocb, struct folio_batch *fbatch) · 546 unmodified lines
25972603 if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ))
25982604 return -EAGAIN;
25992605
2600− folio = filemap_alloc_folio(mapping_gfp_mask(mapping), min_order);
2606+ folio = filemap_alloc_folio(mapping_gfp_mask(mapping), min_order, NULL);
26012607 if (!folio)
26022608 return -ENOMEM;
26032609 if (iocb->ki_flags & IOCB_DONTCACHE)
@@ -4050,8 +4056,7 @@ static struct folio *do_read_cache_folio(struct address_space *mapping, · 1446 unmodified lines
40504056 repeat:
40514057 folio = filemap_get_folio(mapping, index);
40524058 if (IS_ERR(folio)) {
4053− folio = filemap_alloc_folio(gfp,
4054− mapping_min_folio_order(mapping));
4059+ folio = filemap_alloc_folio(gfp, mapping_min_folio_order(mapping), NULL);
40554060 if (!folio)
40564061 return ERR_PTR(-ENOMEM);
40574062 index = mapping_align_index(mapping, index);
mm/mempolicy.c+6−0
@@ -356,6 +356,7 @@ struct mempolicy *get_task_policy(struct task_struct *p) · 355 unmodified lines
356356
357357 return &default_policy;
358358 }
359+EXPORT_SYMBOL_FOR_MODULES(get_task_policy, "kvm");
359360
360361 static const struct mempolicy_operations {
361362 int (*create)(struct mempolicy *pol, const nodemask_t *nodes);
@@ -489,6 +490,7 @@ void __mpol_put(struct mempolicy *pol) · 127 unmodified lines
489490 return;
490491 kmem_cache_free(policy_cache, pol);
491492 }
493+EXPORT_SYMBOL_FOR_MODULES(__mpol_put, "kvm");
492494
493495 static void mpol_rebind_default(struct mempolicy *pol, const nodemask_t *nodes)
494496 {
@@ -2953,6 +2955,7 @@ struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp, · 2458 unmodified lines
29532955 read_unlock(&sp->lock);
29542956 return pol;
29552957 }
2958+EXPORT_SYMBOL_FOR_MODULES(mpol_shared_policy_lookup, "kvm");
29562959
29572960 static void sp_free(struct sp_node *n)
29582961 {
@@ -3238,6 +3241,7 @@ put_mpol: · 279 unmodified lines
32383241 mpol_put(mpol); /* drop our incoming ref on sb mpol */
32393242 }
32403243 }
3244+EXPORT_SYMBOL_FOR_MODULES(mpol_shared_policy_init, "kvm");
32413245
32423246 int mpol_set_shared_policy(struct shared_policy *sp,
32433247 struct vm_area_struct *vma, struct mempolicy *pol)
@@ -3256,6 +3260,7 @@ int mpol_set_shared_policy(struct shared_policy *sp, · 12 unmodified lines
32563260 sp_free(new);
32573261 return err;
32583262 }
3263+EXPORT_SYMBOL_FOR_MODULES(mpol_set_shared_policy, "kvm");
32593264
32603265 /* Free a backing policy store on inode delete. */
32613266 void mpol_free_shared_policy(struct shared_policy *sp)
@@ -3274,6 +3279,7 @@ void mpol_free_shared_policy(struct shared_policy *sp) · 12 unmodified lines
32743279 }
32753280 write_unlock(&sp->lock);
32763281 }
3282+EXPORT_SYMBOL_FOR_MODULES(mpol_free_shared_policy, "kvm");
32773283
32783284 #ifdef CONFIG_NUMA_BALANCING
32793285 static int __initdata numabalancing_override;
mm/readahead.c+1−1
@@ -186,7 +186,7 @@ static struct folio *ractl_alloc_folio(struct readahead_control *ractl, · 185 unmodified lines
186186 {
187187 struct folio *folio;
188188
189− folio = filemap_alloc_folio(gfp_mask, order);
189+ folio = filemap_alloc_folio(gfp_mask, order, NULL);
190190 if (folio && ractl->dropbehind)
191191 __folio_set_dropbehind(folio);
192192
tools/arch/arm64/include/asm/esr.h+2−0
@@ -141,6 +141,8 @@ · 140 unmodified lines
141141 #define ESR_ELx_SF (UL(1) << ESR_ELx_SF_SHIFT)
142142 #define ESR_ELx_AR_SHIFT (14)
143143 #define ESR_ELx_AR (UL(1) << ESR_ELx_AR_SHIFT)
144+#define ESR_ELx_VNCR_SHIFT (13)
145+#define ESR_ELx_VNCR (UL(1) << ESR_ELx_VNCR_SHIFT)
144146 #define ESR_ELx_CM_SHIFT (8)
145147 #define ESR_ELx_CM (UL(1) << ESR_ELx_CM_SHIFT)
146148
tools/testing/selftests/kvm/Makefile+1−1
@@ -6,7 +6,7 @@ ARCH ?= $(SUBARCH) · 5 unmodified lines
66 ifeq ($(ARCH),$(filter $(ARCH),arm64 s390 riscv x86 x86_64 loongarch))
77 # Top-level selftests allows ARCH=x86_64 :-(
88 ifeq ($(ARCH),x86_64)
9− ARCH := x86
9+ override ARCH := x86
1010 endif
1111 include Makefile.kvm
1212 else
tools/testing/selftests/kvm/Makefile.kvm+9−3
@@ -88,8 +88,12 @@ TEST_GEN_PROGS_x86 += x86/kvm_pv_test · 87 unmodified lines
8888 TEST_GEN_PROGS_x86 += x86/kvm_buslock_test
8989 TEST_GEN_PROGS_x86 += x86/monitor_mwait_test
9090 TEST_GEN_PROGS_x86 += x86/msrs_test
91+TEST_GEN_PROGS_x86 += x86/nested_close_kvm_test
9192 TEST_GEN_PROGS_x86 += x86/nested_emulation_test
9293 TEST_GEN_PROGS_x86 += x86/nested_exceptions_test
94+TEST_GEN_PROGS_x86 += x86/nested_invalid_cr3_test
95+TEST_GEN_PROGS_x86 += x86/nested_tsc_adjust_test
96+TEST_GEN_PROGS_x86 += x86/nested_tsc_scaling_test
9397 TEST_GEN_PROGS_x86 += x86/platform_info_test
9498 TEST_GEN_PROGS_x86 += x86/pmu_counters_test
9599 TEST_GEN_PROGS_x86 += x86/pmu_event_filter_test
@@ -111,14 +115,12 @@ TEST_GEN_PROGS_x86 += x86/ucna_injection_test · 15 unmodified lines
111115 TEST_GEN_PROGS_x86 += x86/userspace_io_test
112116 TEST_GEN_PROGS_x86 += x86/userspace_msr_exit_test
113117 TEST_GEN_PROGS_x86 += x86/vmx_apic_access_test
114−TEST_GEN_PROGS_x86 += x86/vmx_close_while_nested_test
115118 TEST_GEN_PROGS_x86 += x86/vmx_dirty_log_test
116119 TEST_GEN_PROGS_x86 += x86/vmx_exception_with_invalid_guest_state
117120 TEST_GEN_PROGS_x86 += x86/vmx_msrs_test
118121 TEST_GEN_PROGS_x86 += x86/vmx_invalid_nested_guest_state
122+TEST_GEN_PROGS_x86 += x86/vmx_nested_la57_state_test
119123 TEST_GEN_PROGS_x86 += x86/vmx_set_nested_state_test
120−TEST_GEN_PROGS_x86 += x86/vmx_tsc_adjust_test
121−TEST_GEN_PROGS_x86 += x86/vmx_nested_tsc_scaling_test
122124 TEST_GEN_PROGS_x86 += x86/apic_bus_clock_test
123125 TEST_GEN_PROGS_x86 += x86/xapic_ipi_test
124126 TEST_GEN_PROGS_x86 += x86/xapic_state_test
@@ -156,6 +158,7 @@ TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_test · 31 unmodified lines
156158 TEST_GEN_PROGS_arm64 = $(TEST_GEN_PROGS_COMMON)
157159 TEST_GEN_PROGS_arm64 += arm64/aarch32_id_regs
158160 TEST_GEN_PROGS_arm64 += arm64/arch_timer_edge_cases
161+TEST_GEN_PROGS_arm64 += arm64/at
159162 TEST_GEN_PROGS_arm64 += arm64/debug-exceptions
160163 TEST_GEN_PROGS_arm64 += arm64/hello_el2
161164 TEST_GEN_PROGS_arm64 += arm64/host_sve
@@ -163,6 +166,7 @@ TEST_GEN_PROGS_arm64 += arm64/hypercalls · 1 unmodified line
163166 TEST_GEN_PROGS_arm64 += arm64/external_aborts
164167 TEST_GEN_PROGS_arm64 += arm64/page_fault_test
165168 TEST_GEN_PROGS_arm64 += arm64/psci_test
169+TEST_GEN_PROGS_arm64 += arm64/sea_to_user
166170 TEST_GEN_PROGS_arm64 += arm64/set_id_regs
167171 TEST_GEN_PROGS_arm64 += arm64/smccc_filter
168172 TEST_GEN_PROGS_arm64 += arm64/vcpu_width_config
@@ -194,6 +198,7 @@ TEST_GEN_PROGS_s390 += s390/debug_test · 25 unmodified lines
194198 TEST_GEN_PROGS_s390 += s390/cpumodel_subfuncs_test
195199 TEST_GEN_PROGS_s390 += s390/shared_zeropage_test
196200 TEST_GEN_PROGS_s390 += s390/ucontrol_test
201+TEST_GEN_PROGS_s390 += s390/user_operexec
197202 TEST_GEN_PROGS_s390 += rseq_test
198203
199204 TEST_GEN_PROGS_riscv = $(TEST_GEN_PROGS_COMMON)
@@ -210,6 +215,7 @@ TEST_GEN_PROGS_riscv += mmu_stress_test · 10 unmodified lines
210215 TEST_GEN_PROGS_riscv += rseq_test
211216 TEST_GEN_PROGS_riscv += steal_time
212217
218+TEST_GEN_PROGS_loongarch = arch_timer
213219 TEST_GEN_PROGS_loongarch += coalesced_io_test
214220 TEST_GEN_PROGS_loongarch += demand_paging_test
215221 TEST_GEN_PROGS_loongarch += dirty_log_perf_test
tools/testing/selftests/kvm/arm64/at.c+166−0
@@ -0,0 +1,166 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/*
3+ * at - Test for KVM's AT emulation in the EL2&0 and EL1&0 translation regimes.
4+ */
5+#include "kvm_util.h"
6+#include "processor.h"
7+#include "test_util.h"
8+#include "ucall.h"
9+
10+#include <asm/sysreg.h>
11+
12+#define TEST_ADDR 0x80000000
13+
14+enum {
15+ CLEAR_ACCESS_FLAG,
16+ TEST_ACCESS_FLAG,
17+};
18+
19+static u64 *ptep_hva;
20+
21+#define copy_el2_to_el1(reg) \
22+ write_sysreg_s(read_sysreg_s(SYS_##reg##_EL1), SYS_##reg##_EL12)
23+
24+/* Yes, this is an ugly hack */
25+#define __at(op, addr) write_sysreg_s(addr, op)
26+
27+#define test_at_insn(op, expect_fault) \
28+do { \
29+ u64 par, fsc; \
30+ bool fault; \
31+ \
32+ GUEST_SYNC(CLEAR_ACCESS_FLAG); \
33+ \
34+ __at(OP_AT_##op, TEST_ADDR); \
35+ isb(); \
36+ par = read_sysreg(par_el1); \
37+ \
38+ fault = par & SYS_PAR_EL1_F; \
39+ fsc = FIELD_GET(SYS_PAR_EL1_FST, par); \
40+ \
41+ __GUEST_ASSERT((expect_fault) == fault, \
42+ "AT "#op": %sexpected fault (par: %lx)1", \
43+ (expect_fault) ? "" : "un", par); \
44+ if ((expect_fault)) { \
45+ __GUEST_ASSERT(fsc == ESR_ELx_FSC_ACCESS_L(3), \
46+ "AT "#op": expected access flag fault (par: %lx)", \
47+ par); \
48+ } else { \
49+ GUEST_ASSERT_EQ(FIELD_GET(SYS_PAR_EL1_ATTR, par), MAIR_ATTR_NORMAL); \
50+ GUEST_ASSERT_EQ(FIELD_GET(SYS_PAR_EL1_SH, par), PTE_SHARED >> 8); \
51+ GUEST_ASSERT_EQ(par & SYS_PAR_EL1_PA, TEST_ADDR); \
52+ GUEST_SYNC(TEST_ACCESS_FLAG); \
53+ } \
54+} while (0)
55+
56+static void test_at(bool expect_fault)
57+{
58+ test_at_insn(S1E2R, expect_fault);
59+ test_at_insn(S1E2W, expect_fault);
60+
61+ /* Reuse the stage-1 MMU context from EL2 at EL1 */
62+ copy_el2_to_el1(SCTLR);
63+ copy_el2_to_el1(MAIR);
64+ copy_el2_to_el1(TCR);
65+ copy_el2_to_el1(TTBR0);
66+ copy_el2_to_el1(TTBR1);
67+
68+ /* Disable stage-2 translation and enter a non-host context */
69+ write_sysreg(0, vtcr_el2);
70+ write_sysreg(0, vttbr_el2);
71+ sysreg_clear_set(hcr_el2, HCR_EL2_TGE | HCR_EL2_VM, 0);
72+ isb();
73+
74+ test_at_insn(S1E1R, expect_fault);
75+ test_at_insn(S1E1W, expect_fault);
76+}
77+
78+static void guest_code(void)
79+{
80+ sysreg_clear_set(tcr_el1, TCR_HA, 0);
81+ isb();
82+
83+ test_at(true);
84+
85+ if (!SYS_FIELD_GET(ID_AA64MMFR1_EL1, HAFDBS, read_sysreg(id_aa64mmfr1_el1)))
86+ GUEST_DONE();
87+
88+ /*
89+ * KVM's software PTW makes the implementation choice that the AT
90+ * instruction sets the access flag.
91+ */
92+ sysreg_clear_set(tcr_el1, 0, TCR_HA);
93+ isb();
94+ test_at(false);
95+
96+ GUEST_DONE();
97+}
98+
99+static void handle_sync(struct kvm_vcpu *vcpu, struct ucall *uc)
100+{
101+ switch (uc->args[1]) {
102+ case CLEAR_ACCESS_FLAG:
103+ /*
104+ * Delete + reinstall the memslot to invalidate stage-2
105+ * mappings of the stage-1 page tables, forcing KVM to
106+ * use the 'slow' AT emulation path.
107+ *
108+ * This and clearing the access flag from host userspace
109+ * ensures that the access flag cannot be set speculatively
110+ * and is reliably cleared at the time of the AT instruction.
111+ */
112+ clear_bit(__ffs(PTE_AF), ptep_hva);
113+ vm_mem_region_reload(vcpu->vm, vcpu->vm->memslots[MEM_REGION_PT]);
114+ break;
115+ case TEST_ACCESS_FLAG:
116+ TEST_ASSERT(test_bit(__ffs(PTE_AF), ptep_hva),
117+ "Expected access flag to be set (desc: %lu)", *ptep_hva);
118+ break;
119+ default:
120+ TEST_FAIL("Unexpected SYNC arg: %lu", uc->args[1]);
121+ }
122+}
123+
124+static void run_test(struct kvm_vcpu *vcpu)
125+{
126+ struct ucall uc;
127+
128+ while (true) {
129+ vcpu_run(vcpu);
130+ switch (get_ucall(vcpu, &uc)) {
131+ case UCALL_DONE:
132+ return;
133+ case UCALL_SYNC:
134+ handle_sync(vcpu, &uc);
135+ continue;
136+ case UCALL_ABORT:
137+ REPORT_GUEST_ASSERT(uc);
138+ return;
139+ default:
140+ TEST_FAIL("Unexpected ucall: %lu", uc.cmd);
141+ }
142+ }
143+}
144+
145+int main(void)
146+{
147+ struct kvm_vcpu_init init;
148+ struct kvm_vcpu *vcpu;
149+ struct kvm_vm *vm;
150+
151+ TEST_REQUIRE(kvm_check_cap(KVM_CAP_ARM_EL2));
152+
153+ vm = vm_create(1);
154+
155+ kvm_get_default_vcpu_target(vm, &init);
156+ init.features[0] |= BIT(KVM_ARM_VCPU_HAS_EL2);
157+ vcpu = aarch64_vcpu_add(vm, 0, &init, guest_code);
158+ kvm_arch_vm_finalize_vcpus(vm);
159+
160+ virt_map(vm, TEST_ADDR, TEST_ADDR, 1);
161+ ptep_hva = virt_get_pte_hva_at_level(vm, TEST_ADDR, 3);
162+ run_test(vcpu);
163+
164+ kvm_vm_free(vm);
165+ return 0;
166+}
tools/testing/selftests/kvm/arm64/sea_to_user.c+331−0
@@ -0,0 +1,331 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/*
3+ * Test KVM returns to userspace with KVM_EXIT_ARM_SEA if host APEI fails
4+ * to handle SEA and userspace has opt-ed in KVM_CAP_ARM_SEA_TO_USER.
5+ *
6+ * After reaching userspace with expected arm_sea info, also test userspace
7+ * injecting a synchronous external data abort into the guest.
8+ *
9+ * This test utilizes EINJ to generate a REAL synchronous external data
10+ * abort by consuming a recoverable uncorrectable memory error. Therefore
11+ * the device under test must support EINJ in both firmware and host kernel,
12+ * including the notrigger feature. Otherwise the test will be skipped.
13+ * The under-test platform's APEI should be unable to claim SEA. Otherwise
14+ * the test will also be skipped.
15+ */
16+
17+#include <signal.h>
18+#include <stdio.h>
19+#include <stdlib.h>
20+#include <unistd.h>
21+
22+#include "test_util.h"
23+#include "kvm_util.h"
24+#include "processor.h"
25+#include "guest_modes.h"
26+
27+#define PAGE_PRESENT (1ULL << 63)
28+#define PAGE_PHYSICAL 0x007fffffffffffffULL
29+#define PAGE_ADDR_MASK (~(0xfffULL))
30+
31+/* Group ISV and ISS[23:14]. */
32+#define ESR_ELx_INST_SYNDROME ((ESR_ELx_ISV) | (ESR_ELx_SAS) | \
33+ (ESR_ELx_SSE) | (ESR_ELx_SRT_MASK) | \
34+ (ESR_ELx_SF) | (ESR_ELx_AR))
35+
36+#define EINJ_ETYPE "/sys/kernel/debug/apei/einj/error_type"
37+#define EINJ_ADDR "/sys/kernel/debug/apei/einj/param1"
38+#define EINJ_MASK "/sys/kernel/debug/apei/einj/param2"
39+#define EINJ_FLAGS "/sys/kernel/debug/apei/einj/flags"
40+#define EINJ_NOTRIGGER "/sys/kernel/debug/apei/einj/notrigger"
41+#define EINJ_DOIT "/sys/kernel/debug/apei/einj/error_inject"
42+/* Memory Uncorrectable non-fatal. */
43+#define ERROR_TYPE_MEMORY_UER 0x10
44+/* Memory address and mask valid (param1 and param2). */
45+#define MASK_MEMORY_UER 0b10
46+
47+/* Guest virtual address region = [2G, 3G). */
48+#define START_GVA 0x80000000UL
49+#define VM_MEM_SIZE 0x40000000UL
50+/* Note: EINJ_OFFSET must < VM_MEM_SIZE. */
51+#define EINJ_OFFSET 0x01234badUL
52+#define EINJ_GVA ((START_GVA) + (EINJ_OFFSET))
53+
54+static vm_paddr_t einj_gpa;
55+static void *einj_hva;
56+static uint64_t einj_hpa;
57+static bool far_invalid;
58+
59+static uint64_t translate_to_host_paddr(unsigned long vaddr)
60+{
61+ uint64_t pinfo;
62+ int64_t offset = vaddr / getpagesize() * sizeof(pinfo);
63+ int fd;
64+ uint64_t page_addr;
65+ uint64_t paddr;
66+
67+ fd = open("/proc/self/pagemap", O_RDONLY);
68+ if (fd < 0)
69+ ksft_exit_fail_perror("Failed to open /proc/self/pagemap");
70+ if (pread(fd, &pinfo, sizeof(pinfo), offset) != sizeof(pinfo)) {
71+ close(fd);
72+ ksft_exit_fail_perror("Failed to read /proc/self/pagemap");
73+ }
74+
75+ close(fd);
76+
77+ if ((pinfo & PAGE_PRESENT) == 0)
78+ ksft_exit_fail_perror("Page not present");
79+
80+ page_addr = (pinfo & PAGE_PHYSICAL) << MIN_PAGE_SHIFT;
81+ paddr = page_addr + (vaddr & (getpagesize() - 1));
82+ return paddr;
83+}
84+
85+static void write_einj_entry(const char *einj_path, uint64_t val)
86+{
87+ char cmd[256] = {0};
88+ FILE *cmdfile = NULL;
89+
90+ sprintf(cmd, "echo %#lx > %s", val, einj_path);
91+ cmdfile = popen(cmd, "r");
92+
93+ if (pclose(cmdfile) == 0)
94+ ksft_print_msg("echo %#lx > %s - done\n", val, einj_path);
95+ else
96+ ksft_exit_fail_perror("Failed to write EINJ entry");
97+}
98+
99+static void inject_uer(uint64_t paddr)
100+{
101+ if (access("/sys/firmware/acpi/tables/EINJ", R_OK) == -1)
102+ ksft_test_result_skip("EINJ table no available in firmware");
103+
104+ if (access(EINJ_ETYPE, R_OK | W_OK) == -1)
105+ ksft_test_result_skip("EINJ module probably not loaded?");
106+
107+ write_einj_entry(EINJ_ETYPE, ERROR_TYPE_MEMORY_UER);
108+ write_einj_entry(EINJ_FLAGS, MASK_MEMORY_UER);
109+ write_einj_entry(EINJ_ADDR, paddr);
110+ write_einj_entry(EINJ_MASK, ~0x0UL);
111+ write_einj_entry(EINJ_NOTRIGGER, 1);
112+ write_einj_entry(EINJ_DOIT, 1);
113+}
114+
115+/*
116+ * When host APEI successfully claims the SEA caused by guest_code, kernel
117+ * will send SIGBUS signal with BUS_MCEERR_AR to test thread.
118+ *
119+ * We set up this SIGBUS handler to skip the test for that case.
120+ */
121+static void sigbus_signal_handler(int sig, siginfo_t *si, void *v)
122+{
123+ ksft_print_msg("SIGBUS (%d) received, dumping siginfo...\n", sig);
124+ ksft_print_msg("si_signo=%d, si_errno=%d, si_code=%d, si_addr=%p\n",
125+ si->si_signo, si->si_errno, si->si_code, si->si_addr);
126+ if (si->si_code == BUS_MCEERR_AR)
127+ ksft_test_result_skip("SEA is claimed by host APEI\n");
128+ else
129+ ksft_test_result_fail("Exit with signal unhandled\n");
130+
131+ exit(0);
132+}
133+
134+static void setup_sigbus_handler(void)
135+{
136+ struct sigaction act;
137+
138+ memset(&act, 0, sizeof(act));
139+ sigemptyset(&act.sa_mask);
140+ act.sa_sigaction = sigbus_signal_handler;
141+ act.sa_flags = SA_SIGINFO;
142+ TEST_ASSERT(sigaction(SIGBUS, &act, NULL) == 0,
143+ "Failed to setup SIGBUS handler");
144+}
145+
146+static void guest_code(void)
147+{
148+ uint64_t guest_data;
149+
150+ /* Consumes error will cause a SEA. */
151+ guest_data = *(uint64_t *)EINJ_GVA;
152+
153+ GUEST_FAIL("Poison not protected by SEA: gva=%#lx, guest_data=%#lx\n",
154+ EINJ_GVA, guest_data);
155+}
156+
157+static void expect_sea_handler(struct ex_regs *regs)
158+{
159+ u64 esr = read_sysreg(esr_el1);
160+ u64 far = read_sysreg(far_el1);
161+ bool expect_far_invalid = far_invalid;
162+
163+ GUEST_PRINTF("Handling Guest SEA\n");
164+ GUEST_PRINTF("ESR_EL1=%#lx, FAR_EL1=%#lx\n", esr, far);
165+
166+ GUEST_ASSERT_EQ(ESR_ELx_EC(esr), ESR_ELx_EC_DABT_CUR);
167+ GUEST_ASSERT_EQ(esr & ESR_ELx_FSC_TYPE, ESR_ELx_FSC_EXTABT);
168+
169+ if (expect_far_invalid) {
170+ GUEST_ASSERT_EQ(esr & ESR_ELx_FnV, ESR_ELx_FnV);
171+ GUEST_PRINTF("Guest observed garbage value in FAR\n");
172+ } else {
173+ GUEST_ASSERT_EQ(esr & ESR_ELx_FnV, 0);
174+ GUEST_ASSERT_EQ(far, EINJ_GVA);
175+ }
176+
177+ GUEST_DONE();
178+}
179+
180+static void vcpu_inject_sea(struct kvm_vcpu *vcpu)
181+{
182+ struct kvm_vcpu_events events = {};
183+
184+ events.exception.ext_dabt_pending = true;
185+ vcpu_events_set(vcpu, &events);
186+}
187+
188+static void run_vm(struct kvm_vm *vm, struct kvm_vcpu *vcpu)
189+{
190+ struct ucall uc;
191+ bool guest_done = false;
192+ struct kvm_run *run = vcpu->run;
193+ u64 esr;
194+
195+ /* Resume the vCPU after error injection to consume the error. */
196+ vcpu_run(vcpu);
197+
198+ ksft_print_msg("Dump kvm_run info about KVM_EXIT_%s\n",
199+ exit_reason_str(run->exit_reason));
200+ ksft_print_msg("kvm_run.arm_sea: esr=%#llx, flags=%#llx\n",
201+ run->arm_sea.esr, run->arm_sea.flags);
202+ ksft_print_msg("kvm_run.arm_sea: gva=%#llx, gpa=%#llx\n",
203+ run->arm_sea.gva, run->arm_sea.gpa);
204+
205+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_ARM_SEA);
206+
207+ esr = run->arm_sea.esr;
208+ TEST_ASSERT_EQ(ESR_ELx_EC(esr), ESR_ELx_EC_DABT_LOW);
209+ TEST_ASSERT_EQ(esr & ESR_ELx_FSC_TYPE, ESR_ELx_FSC_EXTABT);
210+ TEST_ASSERT_EQ(ESR_ELx_ISS2(esr), 0);
211+ TEST_ASSERT_EQ((esr & ESR_ELx_INST_SYNDROME), 0);
212+ TEST_ASSERT_EQ(esr & ESR_ELx_VNCR, 0);
213+
214+ if (!(esr & ESR_ELx_FnV)) {
215+ ksft_print_msg("Expect gva to match given FnV bit is 0\n");
216+ TEST_ASSERT_EQ(run->arm_sea.gva, EINJ_GVA);
217+ }
218+
219+ if (run->arm_sea.flags & KVM_EXIT_ARM_SEA_FLAG_GPA_VALID) {
220+ ksft_print_msg("Expect gpa to match given KVM_EXIT_ARM_SEA_FLAG_GPA_VALID is set\n");
221+ TEST_ASSERT_EQ(run->arm_sea.gpa, einj_gpa & PAGE_ADDR_MASK);
222+ }
223+
224+ far_invalid = esr & ESR_ELx_FnV;
225+
226+ /* Inject a SEA into guest and expect handled in SEA handler. */
227+ vcpu_inject_sea(vcpu);
228+
229+ /* Expect the guest to reach GUEST_DONE gracefully. */
230+ do {
231+ vcpu_run(vcpu);
232+ switch (get_ucall(vcpu, &uc)) {
233+ case UCALL_PRINTF:
234+ ksft_print_msg("From guest: %s", uc.buffer);
235+ break;
236+ case UCALL_DONE:
237+ ksft_print_msg("Guest done gracefully!\n");
238+ guest_done = 1;
239+ break;
240+ case UCALL_ABORT:
241+ ksft_print_msg("Guest aborted!\n");
242+ guest_done = 1;
243+ REPORT_GUEST_ASSERT(uc);
244+ break;
245+ default:
246+ TEST_FAIL("Unexpected ucall: %lu\n", uc.cmd);
247+ }
248+ } while (!guest_done);
249+}
250+
251+static struct kvm_vm *vm_create_with_sea_handler(struct kvm_vcpu **vcpu)
252+{
253+ size_t backing_page_size;
254+ size_t guest_page_size;
255+ size_t alignment;
256+ uint64_t num_guest_pages;
257+ vm_paddr_t start_gpa;
258+ enum vm_mem_backing_src_type src_type = VM_MEM_SRC_ANONYMOUS_HUGETLB_1GB;
259+ struct kvm_vm *vm;
260+
261+ backing_page_size = get_backing_src_pagesz(src_type);
262+ guest_page_size = vm_guest_mode_params[VM_MODE_DEFAULT].page_size;
263+ alignment = max(backing_page_size, guest_page_size);
264+ num_guest_pages = VM_MEM_SIZE / guest_page_size;
265+
266+ vm = __vm_create_with_one_vcpu(vcpu, num_guest_pages, guest_code);
267+ vm_init_descriptor_tables(vm);
268+ vcpu_init_descriptor_tables(*vcpu);
269+
270+ vm_install_sync_handler(vm,
271+ /*vector=*/VECTOR_SYNC_CURRENT,
272+ /*ec=*/ESR_ELx_EC_DABT_CUR,
273+ /*handler=*/expect_sea_handler);
274+
275+ start_gpa = (vm->max_gfn - num_guest_pages) * guest_page_size;
276+ start_gpa = align_down(start_gpa, alignment);
277+
278+ vm_userspace_mem_region_add(
279+ /*vm=*/vm,
280+ /*src_type=*/src_type,
281+ /*guest_paddr=*/start_gpa,
282+ /*slot=*/1,
283+ /*npages=*/num_guest_pages,
284+ /*flags=*/0);
285+
286+ virt_map(vm, START_GVA, start_gpa, num_guest_pages);
287+
288+ ksft_print_msg("Mapped %#lx pages: gva=%#lx to gpa=%#lx\n",
289+ num_guest_pages, START_GVA, start_gpa);
290+ return vm;
291+}
292+
293+static void vm_inject_memory_uer(struct kvm_vm *vm)
294+{
295+ uint64_t guest_data;
296+
297+ einj_gpa = addr_gva2gpa(vm, EINJ_GVA);
298+ einj_hva = addr_gva2hva(vm, EINJ_GVA);
299+
300+ /* Populate certain data before injecting UER. */
301+ *(uint64_t *)einj_hva = 0xBAADCAFE;
302+ guest_data = *(uint64_t *)einj_hva;
303+ ksft_print_msg("Before EINJect: data=%#lx\n",
304+ guest_data);
305+
306+ einj_hpa = translate_to_host_paddr((unsigned long)einj_hva);
307+
308+ ksft_print_msg("EINJ_GVA=%#lx, einj_gpa=%#lx, einj_hva=%p, einj_hpa=%#lx\n",
309+ EINJ_GVA, einj_gpa, einj_hva, einj_hpa);
310+
311+ inject_uer(einj_hpa);
312+ ksft_print_msg("Memory UER EINJected\n");
313+}
314+
315+int main(int argc, char *argv[])
316+{
317+ struct kvm_vm *vm;
318+ struct kvm_vcpu *vcpu;
319+
320+ TEST_REQUIRE(kvm_has_cap(KVM_CAP_ARM_SEA_TO_USER));
321+
322+ setup_sigbus_handler();
323+
324+ vm = vm_create_with_sea_handler(&vcpu);
325+ vm_enable_cap(vm, KVM_CAP_ARM_SEA_TO_USER, 0);
326+ vm_inject_memory_uer(vm);
327+ run_vm(vm, vcpu);
328+ kvm_vm_free(vm);
329+
330+ return 0;
331+}
tools/testing/selftests/kvm/arm64/vgic_irq.c+264−23
@@ -29,6 +29,7 @@ struct test_args { · 28 unmodified lines
2929 bool level_sensitive; /* 1 is level, 0 is edge */
3030 int kvm_max_routes; /* output of KVM_CAP_IRQ_ROUTING */
3131 bool kvm_supports_irqfd; /* output of KVM_CAP_IRQFD */
32+ uint32_t shared_data;
3233 };
3334
3435 /*
@@ -205,7 +206,7 @@ static void kvm_inject_call(kvm_inject_cmd cmd, uint32_t first_intid, · 170 unmodified lines
205206 do { \
206207 uint32_t _intid; \
207208 _intid = gic_get_and_ack_irq(); \
208− GUEST_ASSERT(_intid == 0 || _intid == IAR_SPURIOUS); \
209+ GUEST_ASSERT(_intid == IAR_SPURIOUS); \
209210 } while (0)
210211
211212 #define CAT_HELPER(a, b) a ## b
@@ -359,8 +360,9 @@ static uint32_t wait_for_and_activate_irq(void) · 147 unmodified lines
359360 * interrupts for the whole test.
360361 */
361362 static void test_inject_preemption(struct test_args *args,
362− uint32_t first_intid, int num,
363− kvm_inject_cmd cmd)
363+ uint32_t first_intid, int num,
364+ const unsigned long *exclude,
365+ kvm_inject_cmd cmd)
364366 {
365367 uint32_t intid, prio, step = KVM_PRIO_STEPS;
366368 int i;
@@ -379,6 +381,10 @@ static void test_inject_preemption(struct test_args *args, · 12 unmodified lines
379381 for (i = 0; i < num; i++) {
380382 uint32_t tmp;
381383 intid = i + first_intid;
384+
385+ if (exclude && test_bit(i, exclude))
386+ continue;
387+
382388 KVM_INJECT(cmd, intid);
383389 /* Each successive IRQ will preempt the previous one. */
384390 tmp = wait_for_and_activate_irq();
@@ -390,15 +396,33 @@ static void test_inject_preemption(struct test_args *args, · 5 unmodified lines
390396 /* finish handling the IRQs starting with the highest priority one. */
391397 for (i = 0; i < num; i++) {
392398 intid = num - i - 1 + first_intid;
399+
400+ if (exclude && test_bit(intid - first_intid, exclude))
401+ continue;
402+
393403 gic_set_eoi(intid);
394− if (args->eoi_split)
395− gic_set_dir(intid);
404+ }
405+
406+ if (args->eoi_split) {
407+ for (i = 0; i < num; i++) {
408+ intid = i + first_intid;
409+
410+ if (exclude && test_bit(i, exclude))
411+ continue;
412+
413+ if (args->eoi_split)
414+ gic_set_dir(intid);
415+ }
396416 }
397417
398418 local_irq_enable();
399419
400− for (i = 0; i < num; i++)
420+ for (i = 0; i < num; i++) {
421+ if (exclude && test_bit(i, exclude))
422+ continue;
423+
401424 GUEST_ASSERT(!gic_irq_get_active(i + first_intid));
425+ }
402426 GUEST_ASSERT_EQ(gic_read_ap1r0(), 0);
403427 GUEST_ASSERT_IAR_EMPTY();
404428
@@ -436,33 +460,32 @@ static void test_injection_failure(struct test_args *args, · 31 unmodified lines
436460
437461 static void test_preemption(struct test_args *args, struct kvm_inject_desc *f)
438462 {
439− /*
440− * Test up to 4 levels of preemption. The reason is that KVM doesn't
441− * currently implement the ability to have more than the number-of-LRs
442− * number of concurrently active IRQs. The number of LRs implemented is
443− * IMPLEMENTATION DEFINED, however, it seems that most implement 4.
444− */
463+ /* Timer PPIs cannot be injected from userspace */
464+ static const unsigned long ppi_exclude = (BIT(27 - MIN_PPI) |
465+ BIT(30 - MIN_PPI) |
466+ BIT(28 - MIN_PPI) |
467+ BIT(26 - MIN_PPI));
468+
445469 if (f->sgi)
446− test_inject_preemption(args, MIN_SGI, 4, f->cmd);
470+ test_inject_preemption(args, MIN_SGI, 16, NULL, f->cmd);
447471
448472 if (f->ppi)
449− test_inject_preemption(args, MIN_PPI, 4, f->cmd);
473+ test_inject_preemption(args, MIN_PPI, 16, &ppi_exclude, f->cmd);
450474
451475 if (f->spi)
452− test_inject_preemption(args, MIN_SPI, 4, f->cmd);
476+ test_inject_preemption(args, MIN_SPI, 31, NULL, f->cmd);
453477 }
454478
455479 static void test_restore_active(struct test_args *args, struct kvm_inject_desc *f)
456480 {
457− /* Test up to 4 active IRQs. Same reason as in test_preemption. */
458481 if (f->sgi)
459− guest_restore_active(args, MIN_SGI, 4, f->cmd);
482+ guest_restore_active(args, MIN_SGI, 16, f->cmd);
460483
461484 if (f->ppi)
462− guest_restore_active(args, MIN_PPI, 4, f->cmd);
485+ guest_restore_active(args, MIN_PPI, 16, f->cmd);
463486
464487 if (f->spi)
465− guest_restore_active(args, MIN_SPI, 4, f->cmd);
488+ guest_restore_active(args, MIN_SPI, 31, f->cmd);
466489 }
467490
468491 static void guest_code(struct test_args *args)
@@ -473,12 +496,12 @@ static void guest_code(struct test_args *args) · 4 unmodified lines
473496
474497 gic_init(GIC_V3, 1);
475498
476− for (i = 0; i < nr_irqs; i++)
477− gic_irq_enable(i);
478−
479499 for (i = MIN_SPI; i < nr_irqs; i++)
480500 gic_irq_set_config(i, !level_sensitive);
481501
502+ for (i = 0; i < nr_irqs; i++)
503+ gic_irq_enable(i);
504+
482505 gic_set_eoi_split(args->eoi_split);
483506
484507 reset_priorities(args);
@@ -636,7 +659,7 @@ static void kvm_routing_and_irqfd_check(struct kvm_vm *vm, · 151 unmodified lines
636659 }
637660
638661 for (f = 0, i = intid; i < (uint64_t)intid + num; i++, f++)
639− close(fd[f]);
662+ kvm_close(fd[f]);
640663 }
641664
642665 /* handles the valid case: intid=0xffffffff num=1 */
@@ -779,6 +802,221 @@ done: · 136 unmodified lines
779802 kvm_vm_free(vm);
780803 }
781804
805+static void guest_code_asym_dir(struct test_args *args, int cpuid)
806+{
807+ gic_init(GIC_V3, 2);
808+
809+ gic_set_eoi_split(1);
810+ gic_set_priority_mask(CPU_PRIO_MASK);
811+
812+ if (cpuid == 0) {
813+ uint32_t intid;
814+
815+ local_irq_disable();
816+
817+ gic_set_priority(MIN_PPI, IRQ_DEFAULT_PRIO);
818+ gic_irq_enable(MIN_SPI);
819+ gic_irq_set_pending(MIN_SPI);
820+
821+ intid = wait_for_and_activate_irq();
822+ GUEST_ASSERT_EQ(intid, MIN_SPI);
823+
824+ gic_set_eoi(intid);
825+ isb();
826+
827+ WRITE_ONCE(args->shared_data, MIN_SPI);
828+ dsb(ishst);
829+
830+ do {
831+ dsb(ishld);
832+ } while (READ_ONCE(args->shared_data) == MIN_SPI);
833+ GUEST_ASSERT(!gic_irq_get_active(MIN_SPI));
834+ } else {
835+ do {
836+ dsb(ishld);
837+ } while (READ_ONCE(args->shared_data) != MIN_SPI);
838+
839+ gic_set_dir(MIN_SPI);
840+ isb();
841+
842+ WRITE_ONCE(args->shared_data, 0);
843+ dsb(ishst);
844+ }
845+
846+ GUEST_DONE();
847+}
848+
849+static void guest_code_group_en(struct test_args *args, int cpuid)
850+{
851+ uint32_t intid;
852+
853+ gic_init(GIC_V3, 2);
854+
855+ gic_set_eoi_split(0);
856+ gic_set_priority_mask(CPU_PRIO_MASK);
857+ /* SGI0 is G0, which is disabled */
858+ gic_irq_set_group(0, 0);
859+
860+ /* Configure all SGIs with decreasing priority */
861+ for (intid = 0; intid < MIN_PPI; intid++) {
862+ gic_set_priority(intid, (intid + 1) * 8);
863+ gic_irq_enable(intid);
864+ gic_irq_set_pending(intid);
865+ }
866+
867+ /* Ack and EOI all G1 interrupts */
868+ for (int i = 1; i < MIN_PPI; i++) {
869+ intid = wait_for_and_activate_irq();
870+
871+ GUEST_ASSERT(intid < MIN_PPI);
872+ gic_set_eoi(intid);
873+ isb();
874+ }
875+
876+ /*
877+ * Check that SGI0 is still pending, inactive, and that we cannot
878+ * ack anything.
879+ */
880+ GUEST_ASSERT(gic_irq_get_pending(0));
881+ GUEST_ASSERT(!gic_irq_get_active(0));
882+ GUEST_ASSERT_IAR_EMPTY();
883+ GUEST_ASSERT(read_sysreg_s(SYS_ICC_IAR0_EL1) == IAR_SPURIOUS);
884+
885+ /* Open the G0 gates, and verify we can ack SGI0 */
886+ write_sysreg_s(1, SYS_ICC_IGRPEN0_EL1);
887+ isb();
888+
889+ do {
890+ intid = read_sysreg_s(SYS_ICC_IAR0_EL1);
891+ } while (intid == IAR_SPURIOUS);
892+
893+ GUEST_ASSERT(intid == 0);
894+ GUEST_DONE();
895+}
896+
897+static void guest_code_timer_spi(struct test_args *args, int cpuid)
898+{
899+ uint32_t intid;
900+ u64 val;
901+
902+ gic_init(GIC_V3, 2);
903+
904+ gic_set_eoi_split(1);
905+ gic_set_priority_mask(CPU_PRIO_MASK);
906+
907+ /* Add a pending SPI so that KVM starts trapping DIR */
908+ gic_set_priority(MIN_SPI + cpuid, IRQ_DEFAULT_PRIO);
909+ gic_irq_set_pending(MIN_SPI + cpuid);
910+
911+ /* Configure the timer with a higher priority, make it pending */
912+ gic_set_priority(27, IRQ_DEFAULT_PRIO - 8);
913+
914+ isb();
915+ val = read_sysreg(cntvct_el0);
916+ write_sysreg(val, cntv_cval_el0);
917+ write_sysreg(1, cntv_ctl_el0);
918+ isb();
919+
920+ GUEST_ASSERT(gic_irq_get_pending(27));
921+
922+ /* Enable both interrupts */
923+ gic_irq_enable(MIN_SPI + cpuid);
924+ gic_irq_enable(27);
925+
926+ /* The timer must fire */
927+ intid = wait_for_and_activate_irq();
928+ GUEST_ASSERT(intid == 27);
929+
930+ /* Check that we can deassert it */
931+ write_sysreg(0, cntv_ctl_el0);
932+ isb();
933+
934+ GUEST_ASSERT(!gic_irq_get_pending(27));
935+
936+ /*
937+ * Priority drop, deactivation -- we expect that the host
938+ * deactivation will have been effective
939+ */
940+ gic_set_eoi(27);
941+ gic_set_dir(27);
942+
943+ GUEST_ASSERT(!gic_irq_get_active(27));
944+
945+ /* Do it one more time */
946+ isb();
947+ val = read_sysreg(cntvct_el0);
948+ write_sysreg(val, cntv_cval_el0);
949+ write_sysreg(1, cntv_ctl_el0);
950+ isb();
951+
952+ GUEST_ASSERT(gic_irq_get_pending(27));
953+
954+ /* The timer must fire again */
955+ intid = wait_for_and_activate_irq();
956+ GUEST_ASSERT(intid == 27);
957+
958+ GUEST_DONE();
959+}
960+
961+static void *test_vcpu_run(void *arg)
962+{
963+ struct kvm_vcpu *vcpu = arg;
964+ struct ucall uc;
965+
966+ while (1) {
967+ vcpu_run(vcpu);
968+
969+ switch (get_ucall(vcpu, &uc)) {
970+ case UCALL_ABORT:
971+ REPORT_GUEST_ASSERT(uc);
972+ break;
973+ case UCALL_DONE:
974+ return NULL;
975+ default:
976+ TEST_FAIL("Unknown ucall %lu", uc.cmd);
977+ }
978+ }
979+
980+ return NULL;
981+}
982+
983+static void test_vgic_two_cpus(void *gcode)
984+{
985+ pthread_t thr[2];
986+ struct kvm_vcpu *vcpus[2];
987+ struct test_args args = {};
988+ struct kvm_vm *vm;
989+ vm_vaddr_t args_gva;
990+ int gic_fd, ret;
991+
992+ vm = vm_create_with_vcpus(2, gcode, vcpus);
993+
994+ vm_init_descriptor_tables(vm);
995+ vcpu_init_descriptor_tables(vcpus[0]);
996+ vcpu_init_descriptor_tables(vcpus[1]);
997+
998+ /* Setup the guest args page (so it gets the args). */
999+ args_gva = vm_vaddr_alloc_page(vm);
1000+ memcpy(addr_gva2hva(vm, args_gva), &args, sizeof(args));
1001+ vcpu_args_set(vcpus[0], 2, args_gva, 0);
1002+ vcpu_args_set(vcpus[1], 2, args_gva, 1);
1003+
1004+ gic_fd = vgic_v3_setup(vm, 2, 64);
1005+
1006+ ret = pthread_create(&thr[0], NULL, test_vcpu_run, vcpus[0]);
1007+ if (ret)
1008+ TEST_FAIL("Can't create thread for vcpu 0 (%d)\n", ret);
1009+ ret = pthread_create(&thr[1], NULL, test_vcpu_run, vcpus[1]);
1010+ if (ret)
1011+ TEST_FAIL("Can't create thread for vcpu 1 (%d)\n", ret);
1012+
1013+ pthread_join(thr[0], NULL);
1014+ pthread_join(thr[1], NULL);
1015+
1016+ close(gic_fd);
1017+ kvm_vm_free(vm);
1018+}
1019+
7821020 static void help(const char *name)
7831021 {
7841022 printf(
@@ -835,6 +1073,9 @@ int main(int argc, char **argv) · 50 unmodified lines
8351073 test_vgic(nr_irqs, false /* level */, true /* eoi_split */);
8361074 test_vgic(nr_irqs, true /* level */, false /* eoi_split */);
8371075 test_vgic(nr_irqs, true /* level */, true /* eoi_split */);
1076+ test_vgic_two_cpus(guest_code_asym_dir);
1077+ test_vgic_two_cpus(guest_code_group_en);
1078+ test_vgic_two_cpus(guest_code_timer_spi);
8381079 } else {
8391080 test_vgic(nr_irqs, level_sensitive, eoi_split);
8401081 }
tools/testing/selftests/kvm/arm64/vgic_lpi_stress.c+4−0
@@ -118,6 +118,10 @@ static void guest_setup_gic(void) · 117 unmodified lines
118118
119119 guest_setup_its_mappings();
120120 guest_invalidate_all_rdists();
121+
122+ /* SYNC to ensure ITS setup is complete */
123+ for (cpuid = 0; cpuid < test_data.nr_cpus; cpuid++)
124+ its_send_sync_cmd(test_data.cmdq_base_va, cpuid);
121125 }
122126
123127 static void guest_code(size_t nr_lpis)
tools/testing/selftests/kvm/guest_memfd_test.c+98−0
@@ -19,6 +19,7 @@ · 18 unmodified lines
1919 #include <sys/stat.h>
2020
2121 #include "kvm_util.h"
22+#include "numaif.h"
2223 #include "test_util.h"
2324 #include "ucall_common.h"
2425
@@ -75,6 +76,101 @@ static void test_mmap_supported(int fd, size_t total_size) · 50 unmodified lines
7576 kvm_munmap(mem, total_size);
7677 }
7778
79+static void test_mbind(int fd, size_t total_size)
80+{
81+ const unsigned long nodemask_0 = 1; /* nid: 0 */
82+ unsigned long nodemask = 0;
83+ unsigned long maxnode = 8;
84+ int policy;
85+ char *mem;
86+ int ret;
87+
88+ if (!is_multi_numa_node_system())
89+ return;
90+
91+ mem = kvm_mmap(total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd);
92+
93+ /* Test MPOL_INTERLEAVE policy */
94+ kvm_mbind(mem, page_size * 2, MPOL_INTERLEAVE, &nodemask_0, maxnode, 0);
95+ kvm_get_mempolicy(&policy, &nodemask, maxnode, mem, MPOL_F_ADDR);
96+ TEST_ASSERT(policy == MPOL_INTERLEAVE && nodemask == nodemask_0,
97+ "Wanted MPOL_INTERLEAVE (%u) and nodemask 0x%lx, got %u and 0x%lx",
98+ MPOL_INTERLEAVE, nodemask_0, policy, nodemask);
99+
100+ /* Test basic MPOL_BIND policy */
101+ kvm_mbind(mem + page_size * 2, page_size * 2, MPOL_BIND, &nodemask_0, maxnode, 0);
102+ kvm_get_mempolicy(&policy, &nodemask, maxnode, mem + page_size * 2, MPOL_F_ADDR);
103+ TEST_ASSERT(policy == MPOL_BIND && nodemask == nodemask_0,
104+ "Wanted MPOL_BIND (%u) and nodemask 0x%lx, got %u and 0x%lx",
105+ MPOL_BIND, nodemask_0, policy, nodemask);
106+
107+ /* Test MPOL_DEFAULT policy */
108+ kvm_mbind(mem, total_size, MPOL_DEFAULT, NULL, 0, 0);
109+ kvm_get_mempolicy(&policy, &nodemask, maxnode, mem, MPOL_F_ADDR);
110+ TEST_ASSERT(policy == MPOL_DEFAULT && !nodemask,
111+ "Wanted MPOL_DEFAULT (%u) and nodemask 0x0, got %u and 0x%lx",
112+ MPOL_DEFAULT, policy, nodemask);
113+
114+ /* Test with invalid policy */
115+ ret = mbind(mem, page_size, 999, &nodemask_0, maxnode, 0);
116+ TEST_ASSERT(ret == -1 && errno == EINVAL,
117+ "mbind with invalid policy should fail with EINVAL");
118+
119+ kvm_munmap(mem, total_size);
120+}
121+
122+static void test_numa_allocation(int fd, size_t total_size)
123+{
124+ unsigned long node0_mask = 1; /* Node 0 */
125+ unsigned long node1_mask = 2; /* Node 1 */
126+ unsigned long maxnode = 8;
127+ void *pages[4];
128+ int status[4];
129+ char *mem;
130+ int i;
131+
132+ if (!is_multi_numa_node_system())
133+ return;
134+
135+ mem = kvm_mmap(total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd);
136+
137+ for (i = 0; i < 4; i++)
138+ pages[i] = (char *)mem + page_size * i;
139+
140+ /* Set NUMA policy after allocation */
141+ memset(mem, 0xaa, page_size);
142+ kvm_mbind(pages[0], page_size, MPOL_BIND, &node0_mask, maxnode, 0);
143+ kvm_fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, page_size);
144+
145+ /* Set NUMA policy before allocation */
146+ kvm_mbind(pages[0], page_size * 2, MPOL_BIND, &node1_mask, maxnode, 0);
147+ kvm_mbind(pages[2], page_size * 2, MPOL_BIND, &node0_mask, maxnode, 0);
148+ memset(mem, 0xaa, total_size);
149+
150+ /* Validate if pages are allocated on specified NUMA nodes */
151+ kvm_move_pages(0, 4, pages, NULL, status, 0);
152+ TEST_ASSERT(status[0] == 1, "Expected page 0 on node 1, got it on node %d", status[0]);
153+ TEST_ASSERT(status[1] == 1, "Expected page 1 on node 1, got it on node %d", status[1]);
154+ TEST_ASSERT(status[2] == 0, "Expected page 2 on node 0, got it on node %d", status[2]);
155+ TEST_ASSERT(status[3] == 0, "Expected page 3 on node 0, got it on node %d", status[3]);
156+
157+ /* Punch hole for all pages */
158+ kvm_fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, total_size);
159+
160+ /* Change NUMA policy nodes and reallocate */
161+ kvm_mbind(pages[0], page_size * 2, MPOL_BIND, &node0_mask, maxnode, 0);
162+ kvm_mbind(pages[2], page_size * 2, MPOL_BIND, &node1_mask, maxnode, 0);
163+ memset(mem, 0xaa, total_size);
164+
165+ kvm_move_pages(0, 4, pages, NULL, status, 0);
166+ TEST_ASSERT(status[0] == 0, "Expected page 0 on node 0, got it on node %d", status[0]);
167+ TEST_ASSERT(status[1] == 0, "Expected page 1 on node 0, got it on node %d", status[1]);
168+ TEST_ASSERT(status[2] == 1, "Expected page 2 on node 1, got it on node %d", status[2]);
169+ TEST_ASSERT(status[3] == 1, "Expected page 3 on node 1, got it on node %d", status[3]);
170+
171+ kvm_munmap(mem, total_size);
172+}
173+
78174 static void test_fault_sigbus(int fd, size_t accessible_size, size_t map_size)
79175 {
80176 const char val = 0xaa;
@@ -273,11 +369,13 @@ static void __test_guest_memfd(struct kvm_vm *vm, uint64_t flags) · 192 unmodified lines
273369 if (flags & GUEST_MEMFD_FLAG_INIT_SHARED) {
274370 gmem_test(mmap_supported, vm, flags);
275371 gmem_test(fault_overflow, vm, flags);
372+ gmem_test(numa_allocation, vm, flags);
276373 } else {
277374 gmem_test(fault_private, vm, flags);
278375 }
279376
280377 gmem_test(mmap_cow, vm, flags);
378+ gmem_test(mbind, vm, flags);
281379 } else {
282380 gmem_test(mmap_not_supported, vm, flags);
283381 }
tools/testing/selftests/kvm/include/arm64/gic.h+1−0
@@ -57,6 +57,7 @@ void gic_irq_set_pending(unsigned int intid); · 56 unmodified lines
5757 void gic_irq_clear_pending(unsigned int intid);
5858 bool gic_irq_get_pending(unsigned int intid);
5959 void gic_irq_set_config(unsigned int intid, bool is_edge);
60+void gic_irq_set_group(unsigned int intid, bool group);
6061
6162 void gic_rdist_enable_lpis(vm_paddr_t cfg_table, size_t cfg_table_size,
6263 vm_paddr_t pend_table);
tools/testing/selftests/kvm/include/arm64/gic_v3_its.h+1−0
@@ -15,5 +15,6 @@ void its_send_mapc_cmd(void *cmdq_base, u32 vcpu_id, u32 collection_id, bool val · 14 unmodified lines
1515 void its_send_mapti_cmd(void *cmdq_base, u32 device_id, u32 event_id,
1616 u32 collection_id, u32 intid);
1717 void its_send_invall_cmd(void *cmdq_base, u32 collection_id);
18+void its_send_sync_cmd(void *cmdq_base, u32 vcpu_id);
1819
1920 #endif // __SELFTESTS_GIC_V3_ITS_H__
tools/testing/selftests/kvm/include/kvm_syscalls.h+81−0
@@ -0,0 +1,81 @@
1+/* SPDX-License-Identifier: GPL-2.0-only */
2+#ifndef SELFTEST_KVM_SYSCALLS_H
3+#define SELFTEST_KVM_SYSCALLS_H
4+
5+#include <sys/syscall.h>
6+
7+#define MAP_ARGS0(m,...)
8+#define MAP_ARGS1(m,t,a,...) m(t,a)
9+#define MAP_ARGS2(m,t,a,...) m(t,a), MAP_ARGS1(m,__VA_ARGS__)
10+#define MAP_ARGS3(m,t,a,...) m(t,a), MAP_ARGS2(m,__VA_ARGS__)
11+#define MAP_ARGS4(m,t,a,...) m(t,a), MAP_ARGS3(m,__VA_ARGS__)
12+#define MAP_ARGS5(m,t,a,...) m(t,a), MAP_ARGS4(m,__VA_ARGS__)
13+#define MAP_ARGS6(m,t,a,...) m(t,a), MAP_ARGS5(m,__VA_ARGS__)
14+#define MAP_ARGS(n,...) MAP_ARGS##n(__VA_ARGS__)
15+
16+#define __DECLARE_ARGS(t, a) t a
17+#define __UNPACK_ARGS(t, a) a
18+
19+#define DECLARE_ARGS(nr_args, args...) MAP_ARGS(nr_args, __DECLARE_ARGS, args)
20+#define UNPACK_ARGS(nr_args, args...) MAP_ARGS(nr_args, __UNPACK_ARGS, args)
21+
22+#define __KVM_SYSCALL_ERROR(_name, _ret) \
23+ "%s failed, rc: %i errno: %i (%s)", (_name), (_ret), errno, strerror(errno)
24+
25+/* Define a kvm_<syscall>() API to assert success. */
26+#define __KVM_SYSCALL_DEFINE(name, nr_args, args...) \
27+static inline void kvm_##name(DECLARE_ARGS(nr_args, args)) \
28+{ \
29+ int r; \
30+ \
31+ r = name(UNPACK_ARGS(nr_args, args)); \
32+ TEST_ASSERT(!r, __KVM_SYSCALL_ERROR(#name, r)); \
33+}
34+
35+/*
36+ * Macro to define syscall APIs, either because KVM selftests doesn't link to
37+ * the standard library, e.g. libnuma, or because there is no library that yet
38+ * provides the syscall. These
39+ */
40+#define KVM_SYSCALL_DEFINE(name, nr_args, args...) \
41+static inline long name(DECLARE_ARGS(nr_args, args)) \
42+{ \
43+ return syscall(__NR_##name, UNPACK_ARGS(nr_args, args)); \
44+} \
45+__KVM_SYSCALL_DEFINE(name, nr_args, args)
46+
47+/*
48+ * Special case mmap(), as KVM selftest rarely/never specific an address,
49+ * rarely specify an offset, and because the unique return code requires
50+ * special handling anyways.
51+ */
52+static inline void *__kvm_mmap(size_t size, int prot, int flags, int fd,
53+ off_t offset)
54+{
55+ void *mem;
56+
57+ mem = mmap(NULL, size, prot, flags, fd, offset);
58+ TEST_ASSERT(mem != MAP_FAILED, __KVM_SYSCALL_ERROR("mmap()",
59+ (int)(unsigned long)MAP_FAILED));
60+ return mem;
61+}
62+
63+static inline void *kvm_mmap(size_t size, int prot, int flags, int fd)
64+{
65+ return __kvm_mmap(size, prot, flags, fd, 0);
66+}
67+
68+static inline int kvm_dup(int fd)
69+{
70+ int new_fd = dup(fd);
71+
72+ TEST_ASSERT(new_fd >= 0, __KVM_SYSCALL_ERROR("dup()", new_fd));
73+ return new_fd;
74+}
75+
76+__KVM_SYSCALL_DEFINE(munmap, 2, void *, mem, size_t, size);
77+__KVM_SYSCALL_DEFINE(close, 1, int, fd);
78+__KVM_SYSCALL_DEFINE(fallocate, 4, int, fd, int, mode, loff_t, offset, loff_t, len);
79+__KVM_SYSCALL_DEFINE(ftruncate, 2, unsigned int, fd, off_t, length);
80+
81+#endif /* SELFTEST_KVM_SYSCALLS_H */
tools/testing/selftests/kvm/include/kvm_util.h+10−35
@@ -23,6 +23,7 @@ · 22 unmodified lines
2323
2424 #include <pthread.h>
2525
26+#include "kvm_syscalls.h"
2627 #include "kvm_util_arch.h"
2728 #include "kvm_util_types.h"
2829 #include "sparsebit.h"
@@ -177,7 +178,7 @@ enum vm_guest_mode { · 148 unmodified lines
177178 VM_MODE_P40V48_4K,
178179 VM_MODE_P40V48_16K,
179180 VM_MODE_P40V48_64K,
180− VM_MODE_PXXV48_4K, /* For 48bits VA but ANY bits PA */
181+ VM_MODE_PXXVYY_4K, /* For 48-bit or 57-bit VA, depending on host support */
181182 VM_MODE_P47V64_4K,
182183 VM_MODE_P44V64_4K,
183184 VM_MODE_P36V48_4K,
@@ -219,7 +220,7 @@ extern enum vm_guest_mode vm_mode_default; · 35 unmodified lines
219220
220221 #elif defined(__x86_64__)
221222
222−#define VM_MODE_DEFAULT VM_MODE_PXXV48_4K
223+#define VM_MODE_DEFAULT VM_MODE_PXXVYY_4K
223224 #define MIN_PAGE_SHIFT 12U
224225 #define ptes_per_page(page_size) ((page_size) / 8)
225226
@@ -283,34 +284,6 @@ static inline bool kvm_has_cap(long cap) · 57 unmodified lines
283284 return kvm_check_cap(cap);
284285 }
285286
286−#define __KVM_SYSCALL_ERROR(_name, _ret) \
287− "%s failed, rc: %i errno: %i (%s)", (_name), (_ret), errno, strerror(errno)
288−
289−static inline void *__kvm_mmap(size_t size, int prot, int flags, int fd,
290− off_t offset)
291−{
292− void *mem;
293−
294− mem = mmap(NULL, size, prot, flags, fd, offset);
295− TEST_ASSERT(mem != MAP_FAILED, __KVM_SYSCALL_ERROR("mmap()",
296− (int)(unsigned long)MAP_FAILED));
297−
298− return mem;
299−}
300−
301−static inline void *kvm_mmap(size_t size, int prot, int flags, int fd)
302−{
303− return __kvm_mmap(size, prot, flags, fd, 0);
304−}
305−
306−static inline void kvm_munmap(void *mem, size_t size)
307−{
308− int ret;
309−
310− ret = munmap(mem, size);
311− TEST_ASSERT(!ret, __KVM_SYSCALL_ERROR("munmap()", ret));
312−}
313−
314287 /*
315288 * Use the "inner", double-underscore macro when reporting errors from within
316289 * other macros so that the name of ioctl() and not its literal numeric value
@@ -700,12 +673,12 @@ int __vm_set_user_memory_region2(struct kvm_vm *vm, uint32_t slot, uint32_t flag · 383 unmodified lines
700673 uint32_t guest_memfd, uint64_t guest_memfd_offset);
701674
702675 void vm_userspace_mem_region_add(struct kvm_vm *vm,
703− enum vm_mem_backing_src_type src_type,
704− uint64_t guest_paddr, uint32_t slot, uint64_t npages,
705− uint32_t flags);
676+ enum vm_mem_backing_src_type src_type,
677+ uint64_t gpa, uint32_t slot, uint64_t npages,
678+ uint32_t flags);
706679 void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
707− uint64_t guest_paddr, uint32_t slot, uint64_t npages,
708− uint32_t flags, int guest_memfd_fd, uint64_t guest_memfd_offset);
680+ uint64_t gpa, uint32_t slot, uint64_t npages, uint32_t flags,
681+ int guest_memfd_fd, uint64_t guest_memfd_offset);
709682
710683 #ifndef vm_arch_has_protected_memory
711684 static inline bool vm_arch_has_protected_memory(struct kvm_vm *vm)
@@ -715,6 +688,7 @@ static inline bool vm_arch_has_protected_memory(struct kvm_vm *vm) · 3 unmodified lines
715688 #endif
716689
717690 void vm_mem_region_set_flags(struct kvm_vm *vm, uint32_t slot, uint32_t flags);
691+void vm_mem_region_reload(struct kvm_vm *vm, uint32_t slot);
718692 void vm_mem_region_move(struct kvm_vm *vm, uint32_t slot, uint64_t new_gpa);
719693 void vm_mem_region_delete(struct kvm_vm *vm, uint32_t slot);
720694 struct kvm_vcpu *__vm_vcpu_add(struct kvm_vm *vm, uint32_t vcpu_id);
@@ -1230,6 +1204,7 @@ void virt_arch_pg_map(struct kvm_vm *vm, uint64_t vaddr, uint64_t paddr); · 509 unmodified lines
12301204 static inline void virt_pg_map(struct kvm_vm *vm, uint64_t vaddr, uint64_t paddr)
12311205 {
12321206 virt_arch_pg_map(vm, vaddr, paddr);
1207+ sparsebit_set(vm->vpages_mapped, vaddr >> vm->page_shift);
12331208 }
12341209
12351210
tools/testing/selftests/kvm/include/loongarch/arch_timer.h+85−0
@@ -0,0 +1,85 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+/*
3+ * LoongArch Constant Timer specific interface
4+ */
5+#ifndef SELFTEST_KVM_ARCH_TIMER_H
6+#define SELFTEST_KVM_ARCH_TIMER_H
7+
8+#include "processor.h"
9+
10+/* LoongArch timer frequency is constant 100MHZ */
11+#define TIMER_FREQ (100UL << 20)
12+#define msec_to_cycles(msec) (TIMER_FREQ * (unsigned long)(msec) / 1000)
13+#define usec_to_cycles(usec) (TIMER_FREQ * (unsigned long)(usec) / 1000000)
14+#define cycles_to_usec(cycles) ((unsigned long)(cycles) * 1000000 / TIMER_FREQ)
15+
16+static inline unsigned long timer_get_cycles(void)
17+{
18+ unsigned long val = 0;
19+
20+ __asm__ __volatile__(
21+ "rdtime.d %0, $zero\n\t"
22+ : "=r"(val)
23+ :
24+ );
25+
26+ return val;
27+}
28+
29+static inline unsigned long timer_get_cfg(void)
30+{
31+ return csr_read(LOONGARCH_CSR_TCFG);
32+}
33+
34+static inline unsigned long timer_get_val(void)
35+{
36+ return csr_read(LOONGARCH_CSR_TVAL);
37+}
38+
39+static inline void disable_timer(void)
40+{
41+ csr_write(0, LOONGARCH_CSR_TCFG);
42+}
43+
44+static inline void timer_irq_enable(void)
45+{
46+ unsigned long val;
47+
48+ val = csr_read(LOONGARCH_CSR_ECFG);
49+ val |= ECFGF_TIMER;
50+ csr_write(val, LOONGARCH_CSR_ECFG);
51+}
52+
53+static inline void timer_irq_disable(void)
54+{
55+ unsigned long val;
56+
57+ val = csr_read(LOONGARCH_CSR_ECFG);
58+ val &= ~ECFGF_TIMER;
59+ csr_write(val, LOONGARCH_CSR_ECFG);
60+}
61+
62+static inline void timer_set_next_cmp_ms(unsigned int msec, bool period)
63+{
64+ unsigned long val;
65+
66+ val = msec_to_cycles(msec) & CSR_TCFG_VAL;
67+ val |= CSR_TCFG_EN;
68+ if (period)
69+ val |= CSR_TCFG_PERIOD;
70+ csr_write(val, LOONGARCH_CSR_TCFG);
71+}
72+
73+static inline void __delay(uint64_t cycles)
74+{
75+ uint64_t start = timer_get_cycles();
76+
77+ while ((timer_get_cycles() - start) < cycles)
78+ cpu_relax();
79+}
80+
81+static inline void udelay(unsigned long usec)
82+{
83+ __delay(usec_to_cycles(usec));
84+}
85+#endif /* SELFTEST_KVM_ARCH_TIMER_H */
tools/testing/selftests/kvm/include/loongarch/processor.h+80−1
@@ -83,7 +83,14 @@ · 82 unmodified lines
8383 #define LOONGARCH_CSR_PRMD 0x1
8484 #define LOONGARCH_CSR_EUEN 0x2
8585 #define LOONGARCH_CSR_ECFG 0x4
86+#define ECFGB_TIMER 11
87+#define ECFGF_TIMER (BIT_ULL(ECFGB_TIMER))
8688 #define LOONGARCH_CSR_ESTAT 0x5 /* Exception status */
89+#define CSR_ESTAT_EXC_SHIFT 16
90+#define CSR_ESTAT_EXC_WIDTH 6
91+#define CSR_ESTAT_EXC (0x3f << CSR_ESTAT_EXC_SHIFT)
92+#define EXCCODE_INT 0 /* Interrupt */
93+#define INT_TI 11 /* Timer interrupt*/
8794 #define LOONGARCH_CSR_ERA 0x6 /* ERA */
8895 #define LOONGARCH_CSR_BADV 0x7 /* Bad virtual address */
8996 #define LOONGARCH_CSR_EENTRY 0xc
@@ -106,6 +113,14 @@ · 16 unmodified lines
106113 #define LOONGARCH_CSR_KS1 0x31
107114 #define LOONGARCH_CSR_TMID 0x40
108115 #define LOONGARCH_CSR_TCFG 0x41
116+#define CSR_TCFG_VAL (BIT_ULL(48) - BIT_ULL(2))
117+#define CSR_TCFG_PERIOD_SHIFT 1
118+#define CSR_TCFG_PERIOD (0x1UL << CSR_TCFG_PERIOD_SHIFT)
119+#define CSR_TCFG_EN (0x1UL)
120+#define LOONGARCH_CSR_TVAL 0x42
121+#define LOONGARCH_CSR_TINTCLR 0x44 /* Timer interrupt clear */
122+#define CSR_TINTCLR_TI_SHIFT 0
123+#define CSR_TINTCLR_TI (1 << CSR_TINTCLR_TI_SHIFT)
109124 /* TLB refill exception entry */
110125 #define LOONGARCH_CSR_TLBRENTRY 0x88
111126 #define LOONGARCH_CSR_TLBRSAVE 0x8b
@@ -113,6 +128,28 @@ · 1 unmodified line
113128 #define CSR_TLBREHI_PS_SHIFT 0
114129 #define CSR_TLBREHI_PS (0x3fUL << CSR_TLBREHI_PS_SHIFT)
115130
131+#define csr_read(csr) \
132+({ \
133+ register unsigned long __v; \
134+ __asm__ __volatile__( \
135+ "csrrd %[val], %[reg]\n\t" \
136+ : [val] "=r" (__v) \
137+ : [reg] "i" (csr) \
138+ : "memory"); \
139+ __v; \
140+})
141+
142+#define csr_write(v, csr) \
143+({ \
144+ register unsigned long __v = v; \
145+ __asm__ __volatile__ ( \
146+ "csrwr %[val], %[reg]\n\t" \
147+ : [val] "+r" (__v) \
148+ : [reg] "i" (csr) \
149+ : "memory"); \
150+ __v; \
151+})
152+
116153 #define EXREGS_GPRS (32)
117154
118155 #ifndef __ASSEMBLER__
@@ -124,18 +161,60 @@ struct ex_regs { · 5 unmodified lines
124161 unsigned long pc;
125162 unsigned long estat;
126163 unsigned long badv;
164+ unsigned long prmd;
127165 };
128166
129167 #define PC_OFFSET_EXREGS offsetof(struct ex_regs, pc)
130168 #define ESTAT_OFFSET_EXREGS offsetof(struct ex_regs, estat)
131169 #define BADV_OFFSET_EXREGS offsetof(struct ex_regs, badv)
170+#define PRMD_OFFSET_EXREGS offsetof(struct ex_regs, prmd)
132171 #define EXREGS_SIZE sizeof(struct ex_regs)
133172
173+#define VECTOR_NUM 64
174+
175+typedef void(*handler_fn)(struct ex_regs *);
176+
177+struct handlers {
178+ handler_fn exception_handlers[VECTOR_NUM];
179+};
180+
181+void vm_init_descriptor_tables(struct kvm_vm *vm);
182+void vm_install_exception_handler(struct kvm_vm *vm, int vector, handler_fn handler);
183+
184+static inline void cpu_relax(void)
185+{
186+ asm volatile("nop" ::: "memory");
187+}
188+
189+static inline void local_irq_enable(void)
190+{
191+ unsigned int flags = CSR_CRMD_IE;
192+ register unsigned int mask asm("$t0") = CSR_CRMD_IE;
193+
194+ __asm__ __volatile__(
195+ "csrxchg %[val], %[mask], %[reg]\n\t"
196+ : [val] "+r" (flags)
197+ : [mask] "r" (mask), [reg] "i" (LOONGARCH_CSR_CRMD)
198+ : "memory");
199+}
200+
201+static inline void local_irq_disable(void)
202+{
203+ unsigned int flags = 0;
204+ register unsigned int mask asm("$t0") = CSR_CRMD_IE;
205+
206+ __asm__ __volatile__(
207+ "csrxchg %[val], %[mask], %[reg]\n\t"
208+ : [val] "+r" (flags)
209+ : [mask] "r" (mask), [reg] "i" (LOONGARCH_CSR_CRMD)
210+ : "memory");
211+}
134212 #else
135213 #define PC_OFFSET_EXREGS ((EXREGS_GPRS + 0) * 8)
136214 #define ESTAT_OFFSET_EXREGS ((EXREGS_GPRS + 1) * 8)
137215 #define BADV_OFFSET_EXREGS ((EXREGS_GPRS + 2) * 8)
138−#define EXREGS_SIZE ((EXREGS_GPRS + 3) * 8)
216+#define PRMD_OFFSET_EXREGS ((EXREGS_GPRS + 3) * 8)
217+#define EXREGS_SIZE ((EXREGS_GPRS + 4) * 8)
139218 #endif
140219
141220 #endif /* SELFTEST_KVM_PROCESSOR_H */
tools/testing/selftests/kvm/include/numaif.h+69−41
@@ -1,55 +1,83 @@
11 /* SPDX-License-Identifier: GPL-2.0-only */
2−/*
3− * tools/testing/selftests/kvm/include/numaif.h
4− *
5− * Copyright (C) 2020, Google LLC.
6− *
7− * This work is licensed under the terms of the GNU GPL, version 2.
8− *
9− * Header file that provides access to NUMA API functions not explicitly
10− * exported to user space.
11− */
2+/* Copyright (C) 2020, Google LLC. */
123
134 #ifndef SELFTEST_KVM_NUMAIF_H
145 #define SELFTEST_KVM_NUMAIF_H
156
16−#define __NR_get_mempolicy 239
17−#define __NR_migrate_pages 256
7+#include <dirent.h>
188
19−/* System calls */
20−long get_mempolicy(int *policy, const unsigned long *nmask,
21− unsigned long maxnode, void *addr, int flags)
9+#include <linux/mempolicy.h>
10+
11+#include "kvm_syscalls.h"
12+
13+KVM_SYSCALL_DEFINE(get_mempolicy, 5, int *, policy, const unsigned long *, nmask,
14+ unsigned long, maxnode, void *, addr, int, flags);
15+
16+KVM_SYSCALL_DEFINE(set_mempolicy, 3, int, mode, const unsigned long *, nmask,
17+ unsigned long, maxnode);
18+
19+KVM_SYSCALL_DEFINE(set_mempolicy_home_node, 4, unsigned long, start,
20+ unsigned long, len, unsigned long, home_node,
21+ unsigned long, flags);
22+
23+KVM_SYSCALL_DEFINE(migrate_pages, 4, int, pid, unsigned long, maxnode,
24+ const unsigned long *, frommask, const unsigned long *, tomask);
25+
26+KVM_SYSCALL_DEFINE(move_pages, 6, int, pid, unsigned long, count, void *, pages,
27+ const int *, nodes, int *, status, int, flags);
28+
29+KVM_SYSCALL_DEFINE(mbind, 6, void *, addr, unsigned long, size, int, mode,
30+ const unsigned long *, nodemask, unsigned long, maxnode,
31+ unsigned int, flags);
32+
33+static inline int get_max_numa_node(void)
2234 {
23− return syscall(__NR_get_mempolicy, policy, nmask,
24− maxnode, addr, flags);
35+ struct dirent *de;
36+ int max_node = 0;
37+ DIR *d;
38+
39+ /*
40+ * Assume there's a single node if the kernel doesn't support NUMA,
41+ * or if no nodes are found.
42+ */
43+ d = opendir("/sys/devices/system/node");
44+ if (!d)
45+ return 0;
46+
47+ while ((de = readdir(d)) != NULL) {
48+ int node_id;
49+ char *endptr;
50+
51+ if (strncmp(de->d_name, "node", 4) != 0)
52+ continue;
53+
54+ node_id = strtol(de->d_name + 4, &endptr, 10);
55+ if (*endptr != '\0')
56+ continue;
57+
58+ if (node_id > max_node)
59+ max_node = node_id;
60+ }
61+ closedir(d);
62+
63+ return max_node;
2564 }
2665
27−long migrate_pages(int pid, unsigned long maxnode,
28− const unsigned long *frommask,
29− const unsigned long *tomask)
66+static bool is_numa_available(void)
3067 {
31− return syscall(__NR_migrate_pages, pid, maxnode, frommask, tomask);
68+ /*
69+ * Probe for NUMA by doing a dummy get_mempolicy(). If the syscall
70+ * fails with ENOSYS, then the kernel was built without NUMA support.
71+ * if the syscall fails with EPERM, then the process/user lacks the
72+ * necessary capabilities (CAP_SYS_NICE).
73+ */
74+ return !get_mempolicy(NULL, NULL, 0, NULL, 0) ||
75+ (errno != ENOSYS && errno != EPERM);
3276 }
3377
34−/* Policies */
35−#define MPOL_DEFAULT 0
36−#define MPOL_PREFERRED 1
37−#define MPOL_BIND 2
38−#define MPOL_INTERLEAVE 3
39−
40−#define MPOL_MAX MPOL_INTERLEAVE
41−
42−/* Flags for get_mem_policy */
43−#define MPOL_F_NODE (1<<0) /* return next il node or node of address */
44− /* Warning: MPOL_F_NODE is unsupported and
45− * subject to change. Don't use.
46− */
47−#define MPOL_F_ADDR (1<<1) /* look up vma using address */
48−#define MPOL_F_MEMS_ALLOWED (1<<2) /* query nodes allowed in cpuset */
49−
50−/* Flags for mbind */
51−#define MPOL_MF_STRICT (1<<0) /* Verify existing pages in the mapping */
52−#define MPOL_MF_MOVE (1<<1) /* Move pages owned by this process to conform to mapping */
53−#define MPOL_MF_MOVE_ALL (1<<2) /* Move every page to conform to mapping */
78+static inline bool is_multi_numa_node_system(void)
79+{
80+ return is_numa_available() && get_max_numa_node() >= 1;
81+}
5482
5583 #endif /* SELFTEST_KVM_NUMAIF_H */
tools/testing/selftests/kvm/include/x86/processor.h+1−1
@@ -1441,7 +1441,7 @@ enum pg_level { · 1440 unmodified lines
14411441 PG_LEVEL_2M,
14421442 PG_LEVEL_1G,
14431443 PG_LEVEL_512G,
1444− PG_LEVEL_NUM
1444+ PG_LEVEL_256T
14451445 };
14461446
14471447 #define PG_LEVEL_SHIFT(_level) ((_level - 1) * 9 + 12)
tools/testing/selftests/kvm/include/x86/vmx.h+1−2
@@ -568,8 +568,7 @@ void nested_map_memslot(struct vmx_pages *vmx, struct kvm_vm *vm, · 567 unmodified lines
568568 void nested_identity_map_1g(struct vmx_pages *vmx, struct kvm_vm *vm,
569569 uint64_t addr, uint64_t size);
570570 bool kvm_cpu_has_ept(void);
571−void prepare_eptp(struct vmx_pages *vmx, struct kvm_vm *vm,
572− uint32_t eptp_memslot);
571+void prepare_eptp(struct vmx_pages *vmx, struct kvm_vm *vm);
573572 void prepare_virtualize_apic_accesses(struct vmx_pages *vmx, struct kvm_vm *vm);
574573
575574 #endif /* SELFTEST_KVM_VMX_H */
tools/testing/selftests/kvm/kvm_binary_stats_test.c+2−2
@@ -239,14 +239,14 @@ int main(int argc, char *argv[]) · 238 unmodified lines
239239 * single stats file works and doesn't cause explosions.
240240 */
241241 vm_stats_fds = vm_get_stats_fd(vms[i]);
242− stats_test(dup(vm_stats_fds));
242+ stats_test(kvm_dup(vm_stats_fds));
243243
244244 /* Verify userspace can instantiate multiple stats files. */
245245 stats_test(vm_get_stats_fd(vms[i]));
246246
247247 for (j = 0; j < max_vcpu; ++j) {
248248 vcpu_stats_fds[j] = vcpu_get_stats_fd(vcpus[i * max_vcpu + j]);
249− stats_test(dup(vcpu_stats_fds[j]));
249+ stats_test(kvm_dup(vcpu_stats_fds[j]));
250250 stats_test(vcpu_get_stats_fd(vcpus[i * max_vcpu + j]));
251251 }
252252
tools/testing/selftests/kvm/lib/arm64/gic.c+6−0
@@ -155,3 +155,9 @@ void gic_irq_set_config(unsigned int intid, bool is_edge) · 154 unmodified lines
155155 GUEST_ASSERT(gic_common_ops);
156156 gic_common_ops->gic_irq_set_config(intid, is_edge);
157157 }
158+
159+void gic_irq_set_group(unsigned int intid, bool group)
160+{
161+ GUEST_ASSERT(gic_common_ops);
162+ gic_common_ops->gic_irq_set_group(intid, group);
163+}
tools/testing/selftests/kvm/lib/arm64/gic_private.h+1−0
@@ -25,6 +25,7 @@ struct gic_common_ops { · 24 unmodified lines
2525 void (*gic_irq_clear_pending)(uint32_t intid);
2626 bool (*gic_irq_get_pending)(uint32_t intid);
2727 void (*gic_irq_set_config)(uint32_t intid, bool is_edge);
28+ void (*gic_irq_set_group)(uint32_t intid, bool group);
2829 };
2930
3031 extern const struct gic_common_ops gicv3_ops;
tools/testing/selftests/kvm/lib/arm64/gic_v3.c+22−0
@@ -293,17 +293,36 @@ static void gicv3_enable_redist(volatile void *redist_base) · 292 unmodified lines
293293 }
294294 }
295295
296+static void gicv3_set_group(uint32_t intid, bool grp)
297+{
298+ uint32_t cpu_or_dist;
299+ uint32_t val;
300+
301+ cpu_or_dist = (get_intid_range(intid) == SPI_RANGE) ? DIST_BIT : guest_get_vcpuid();
302+ val = gicv3_reg_readl(cpu_or_dist, GICD_IGROUPR + (intid / 32) * 4);
303+ if (grp)
304+ val |= BIT(intid % 32);
305+ else
306+ val &= ~BIT(intid % 32);
307+ gicv3_reg_writel(cpu_or_dist, GICD_IGROUPR + (intid / 32) * 4, val);
308+}
309+
296310 static void gicv3_cpu_init(unsigned int cpu)
297311 {
298312 volatile void *sgi_base;
299313 unsigned int i;
300314 volatile void *redist_base_cpu;
315+ u64 typer;
301316
302317 GUEST_ASSERT(cpu < gicv3_data.nr_cpus);
303318
304319 redist_base_cpu = gicr_base_cpu(cpu);
305320 sgi_base = sgi_base_from_redist(redist_base_cpu);
306321
322+ /* Verify assumption that GICR_TYPER.Processor_number == cpu */
323+ typer = readq_relaxed(redist_base_cpu + GICR_TYPER);
324+ GUEST_ASSERT_EQ(GICR_TYPER_CPU_NUMBER(typer), cpu);
325+
307326 gicv3_enable_redist(redist_base_cpu);
308327
309328 /*
@@ -328,6 +347,8 @@ static void gicv3_cpu_init(unsigned int cpu) · 18 unmodified lines
328347 /* Set a default priority threshold */
329348 write_sysreg_s(ICC_PMR_DEF_PRIO, SYS_ICC_PMR_EL1);
330349
350+ /* Disable Group-0 interrupts */
351+ write_sysreg_s(ICC_IGRPEN0_EL1_MASK, SYS_ICC_IGRPEN1_EL1);
331352 /* Enable non-secure Group-1 interrupts */
332353 write_sysreg_s(ICC_IGRPEN1_EL1_MASK, SYS_ICC_IGRPEN1_EL1);
333354 }
@@ -400,6 +421,7 @@ const struct gic_common_ops gicv3_ops = { · 66 unmodified lines
400421 .gic_irq_clear_pending = gicv3_irq_clear_pending,
401422 .gic_irq_get_pending = gicv3_irq_get_pending,
402423 .gic_irq_set_config = gicv3_irq_set_config,
424+ .gic_irq_set_group = gicv3_set_group,
403425 };
404426
405427 void gic_rdist_enable_lpis(vm_paddr_t cfg_table, size_t cfg_table_size,
tools/testing/selftests/kvm/lib/arm64/gic_v3_its.c+10−0
@@ -253,3 +253,13 @@ void its_send_invall_cmd(void *cmdq_base, u32 collection_id) · 252 unmodified lines
253253
254254 its_send_cmd(cmdq_base, &cmd);
255255 }
256+
257+void its_send_sync_cmd(void *cmdq_base, u32 vcpu_id)
258+{
259+ struct its_cmd_block cmd = {};
260+
261+ its_encode_cmd(&cmd, GITS_CMD_SYNC);
262+ its_encode_target(&cmd, procnum_to_rdbase(vcpu_id));
263+
264+ its_send_cmd(cmdq_base, &cmd);
265+}
tools/testing/selftests/kvm/lib/arm64/processor.c+1−1
@@ -324,7 +324,7 @@ void aarch64_vcpu_setup(struct kvm_vcpu *vcpu, struct kvm_vcpu_init *init) · 323 unmodified lines
324324
325325 /* Configure base granule size */
326326 switch (vm->mode) {
327− case VM_MODE_PXXV48_4K:
327+ case VM_MODE_PXXVYY_4K:
328328 TEST_FAIL("AArch64 does not support 4K sized pages "
329329 "with ANY-bit physical address ranges");
330330 case VM_MODE_P52V48_64K:
tools/testing/selftests/kvm/lib/kvm_util.c+80−65
@@ -201,7 +201,7 @@ const char *vm_guest_mode_string(uint32_t i) · 200 unmodified lines
201201 [VM_MODE_P40V48_4K] = "PA-bits:40, VA-bits:48, 4K pages",
202202 [VM_MODE_P40V48_16K] = "PA-bits:40, VA-bits:48, 16K pages",
203203 [VM_MODE_P40V48_64K] = "PA-bits:40, VA-bits:48, 64K pages",
204− [VM_MODE_PXXV48_4K] = "PA-bits:ANY, VA-bits:48, 4K pages",
204+ [VM_MODE_PXXVYY_4K] = "PA-bits:ANY, VA-bits:48 or 57, 4K pages",
205205 [VM_MODE_P47V64_4K] = "PA-bits:47, VA-bits:64, 4K pages",
206206 [VM_MODE_P44V64_4K] = "PA-bits:44, VA-bits:64, 4K pages",
207207 [VM_MODE_P36V48_4K] = "PA-bits:36, VA-bits:48, 4K pages",
@@ -228,7 +228,7 @@ const struct vm_guest_mode_params vm_guest_mode_params[] = { · 20 unmodified lines
228228 [VM_MODE_P40V48_4K] = { 40, 48, 0x1000, 12 },
229229 [VM_MODE_P40V48_16K] = { 40, 48, 0x4000, 14 },
230230 [VM_MODE_P40V48_64K] = { 40, 48, 0x10000, 16 },
231− [VM_MODE_PXXV48_4K] = { 0, 0, 0x1000, 12 },
231+ [VM_MODE_PXXVYY_4K] = { 0, 0, 0x1000, 12 },
232232 [VM_MODE_P47V64_4K] = { 47, 64, 0x1000, 12 },
233233 [VM_MODE_P44V64_4K] = { 44, 64, 0x1000, 12 },
234234 [VM_MODE_P36V48_4K] = { 36, 48, 0x1000, 12 },
@@ -310,24 +310,26 @@ struct kvm_vm *____vm_create(struct vm_shape shape) · 75 unmodified lines
310310 case VM_MODE_P36V47_16K:
311311 vm->pgtable_levels = 3;
312312 break;
313− case VM_MODE_PXXV48_4K:
313+ case VM_MODE_PXXVYY_4K:
314314 #ifdef __x86_64__
315315 kvm_get_cpu_address_width(&vm->pa_bits, &vm->va_bits);
316316 kvm_init_vm_address_properties(vm);
317− /*
318− * Ignore KVM support for 5-level paging (vm->va_bits == 57),
319− * it doesn't take effect unless a CR4.LA57 is set, which it
320− * isn't for this mode (48-bit virtual address space).
321− */
322− TEST_ASSERT(vm->va_bits == 48 || vm->va_bits == 57,
323− "Linear address width (%d bits) not supported",
324− vm->va_bits);
317+
325318 pr_debug("Guest physical address width detected: %d\n",
326319 vm->pa_bits);
327− vm->pgtable_levels = 4;
328− vm->va_bits = 48;
320+ pr_debug("Guest virtual address width detected: %d\n",
321+ vm->va_bits);
322+
323+ if (vm->va_bits == 57) {
324+ vm->pgtable_levels = 5;
325+ } else {
326+ TEST_ASSERT(vm->va_bits == 48,
327+ "Unexpected guest virtual address width: %d",
328+ vm->va_bits);
329+ vm->pgtable_levels = 4;
330+ }
329331 #else
330− TEST_FAIL("VM_MODE_PXXV48_4K not supported on non-x86 platforms");
332+ TEST_FAIL("VM_MODE_PXXVYY_4K not supported on non-x86 platforms");
331333 #endif
332334 break;
333335 case VM_MODE_P47V64_4K:
@@ -704,8 +706,6 @@ userspace_mem_region_find(struct kvm_vm *vm, uint64_t start, uint64_t end) · 370 unmodified lines
704706
705707 static void kvm_stats_release(struct kvm_binary_stats *stats)
706708 {
707− int ret;
708−
709709 if (stats->fd < 0)
710710 return;
711711
@@ -714,8 +714,7 @@ static void kvm_stats_release(struct kvm_binary_stats *stats) · 2 unmodified lines
714714 stats->desc = NULL;
715715 }
716716
717− ret = close(stats->fd);
718− TEST_ASSERT(!ret, __KVM_SYSCALL_ERROR("close()", ret));
717+ kvm_close(stats->fd);
719718 stats->fd = -1;
720719 }
721720
@@ -738,8 +737,6 @@ __weak void vcpu_arch_free(struct kvm_vcpu *vcpu) · 16 unmodified lines
738737 */
739738 static void vm_vcpu_rm(struct kvm_vm *vm, struct kvm_vcpu *vcpu)
740739 {
741− int ret;
742−
743740 if (vcpu->dirty_gfns) {
744741 kvm_munmap(vcpu->dirty_gfns, vm->dirty_ring_size);
745742 vcpu->dirty_gfns = NULL;
@@ -747,9 +744,7 @@ static void vm_vcpu_rm(struct kvm_vm *vm, struct kvm_vcpu *vcpu) · 1 unmodified line
747744
748745 kvm_munmap(vcpu->run, vcpu_mmap_sz());
749746
750− ret = close(vcpu->fd);
751− TEST_ASSERT(!ret, __KVM_SYSCALL_ERROR("close()", ret));
752−
747+ kvm_close(vcpu->fd);
753748 kvm_stats_release(&vcpu->stats);
754749
755750 list_del(&vcpu->list);
@@ -761,16 +756,12 @@ static void vm_vcpu_rm(struct kvm_vm *vm, struct kvm_vcpu *vcpu) · 5 unmodified lines
761756 void kvm_vm_release(struct kvm_vm *vmp)
762757 {
763758 struct kvm_vcpu *vcpu, *tmp;
764− int ret;
765759
766760 list_for_each_entry_safe(vcpu, tmp, &vmp->vcpus, list)
767761 vm_vcpu_rm(vmp, vcpu);
768762
769− ret = close(vmp->fd);
770− TEST_ASSERT(!ret, __KVM_SYSCALL_ERROR("close()", ret));
771−
772− ret = close(vmp->kvm_fd);
773− TEST_ASSERT(!ret, __KVM_SYSCALL_ERROR("close()", ret));
763+ kvm_close(vmp->fd);
764+ kvm_close(vmp->kvm_fd);
774765
775766 /* Free cached stats metadata and close FD */
776767 kvm_stats_release(&vmp->stats);
@@ -828,7 +819,7 @@ void kvm_vm_free(struct kvm_vm *vmp) · 51 unmodified lines
828819 int kvm_memfd_alloc(size_t size, bool hugepages)
829820 {
830821 int memfd_flags = MFD_CLOEXEC;
831− int fd, r;
822+ int fd;
832823
833824 if (hugepages)
834825 memfd_flags |= MFD_HUGETLB;
@@ -836,11 +827,8 @@ int kvm_memfd_alloc(size_t size, bool hugepages) · 1 unmodified line
836827 fd = memfd_create("kvm_selftest", memfd_flags);
837828 TEST_ASSERT(fd != -1, __KVM_SYSCALL_ERROR("memfd_create()", fd));
838829
839− r = ftruncate(fd, size);
840− TEST_ASSERT(!r, __KVM_SYSCALL_ERROR("ftruncate()", r));
841−
842− r = fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, size);
843− TEST_ASSERT(!r, __KVM_SYSCALL_ERROR("fallocate()", r));
830+ kvm_ftruncate(fd, size);
831+ kvm_fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, size);
844832
845833 return fd;
846834 }
@@ -957,8 +945,8 @@ void vm_set_user_memory_region2(struct kvm_vm *vm, uint32_t slot, uint32_t flags · 110 unmodified lines
957945
958946 /* FIXME: This thing needs to be ripped apart and rewritten. */
959947 void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type,
960− uint64_t guest_paddr, uint32_t slot, uint64_t npages,
961− uint32_t flags, int guest_memfd, uint64_t guest_memfd_offset)
948+ uint64_t gpa, uint32_t slot, uint64_t npages, uint32_t flags,
949+ int guest_memfd, uint64_t guest_memfd_offset)
962950 {
963951 int ret;
964952 struct userspace_mem_region *region;
@@ -972,30 +960,29 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, · 7 unmodified lines
972960 "Number of guest pages is not compatible with the host. "
973961 "Try npages=%d", vm_adjust_num_guest_pages(vm->mode, npages));
974962
975− TEST_ASSERT((guest_paddr % vm->page_size) == 0, "Guest physical "
963+ TEST_ASSERT((gpa % vm->page_size) == 0, "Guest physical "
976964 "address not on a page boundary.\n"
977− " guest_paddr: 0x%lx vm->page_size: 0x%x",
978− guest_paddr, vm->page_size);
979− TEST_ASSERT((((guest_paddr >> vm->page_shift) + npages) - 1)
965+ " gpa: 0x%lx vm->page_size: 0x%x",
966+ gpa, vm->page_size);
967+ TEST_ASSERT((((gpa >> vm->page_shift) + npages) - 1)
980968 <= vm->max_gfn, "Physical range beyond maximum "
981969 "supported physical address,\n"
982− " guest_paddr: 0x%lx npages: 0x%lx\n"
970+ " gpa: 0x%lx npages: 0x%lx\n"
983971 " vm->max_gfn: 0x%lx vm->page_size: 0x%x",
984− guest_paddr, npages, vm->max_gfn, vm->page_size);
972+ gpa, npages, vm->max_gfn, vm->page_size);
985973
986974 /*
987975 * Confirm a mem region with an overlapping address doesn't
988976 * already exist.
989977 */
990978 region = (struct userspace_mem_region *) userspace_mem_region_find(
991− vm, guest_paddr, (guest_paddr + npages * vm->page_size) - 1);
979+ vm, gpa, (gpa + npages * vm->page_size) - 1);
992980 if (region != NULL)
993981 TEST_FAIL("overlapping userspace_mem_region already "
994982 "exists\n"
995− " requested guest_paddr: 0x%lx npages: 0x%lx "
996− "page_size: 0x%x\n"
997− " existing guest_paddr: 0x%lx size: 0x%lx",
998− guest_paddr, npages, vm->page_size,
983+ " requested gpa: 0x%lx npages: 0x%lx page_size: 0x%x\n"
984+ " existing gpa: 0x%lx size: 0x%lx",
985+ gpa, npages, vm->page_size,
999986 (uint64_t) region->region.guest_phys_addr,
1000987 (uint64_t) region->region.memory_size);
1001988
@@ -1009,8 +996,7 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, · 7 unmodified lines
1009996 "already exists.\n"
1010997 " requested slot: %u paddr: 0x%lx npages: 0x%lx\n"
1011998 " existing slot: %u paddr: 0x%lx size: 0x%lx",
1012− slot, guest_paddr, npages,
1013− region->region.slot,
999+ slot, gpa, npages, region->region.slot,
10141000 (uint64_t) region->region.guest_phys_addr,
10151001 (uint64_t) region->region.memory_size);
10161002 }
@@ -1036,7 +1022,7 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, · 19 unmodified lines
10361022 if (src_type == VM_MEM_SRC_ANONYMOUS_THP)
10371023 alignment = max(backing_src_pagesz, alignment);
10381024
1039− TEST_ASSERT_EQ(guest_paddr, align_up(guest_paddr, backing_src_pagesz));
1025+ TEST_ASSERT_EQ(gpa, align_up(gpa, backing_src_pagesz));
10401026
10411027 /* Add enough memory to align up if necessary */
10421028 if (alignment > 1)
@@ -1084,8 +1070,7 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, · 41 unmodified lines
10841070 * needing to track if the fd is owned by the framework
10851071 * or by the caller.
10861072 */
1087− guest_memfd = dup(guest_memfd);
1088− TEST_ASSERT(guest_memfd >= 0, __KVM_SYSCALL_ERROR("dup()", guest_memfd));
1073+ guest_memfd = kvm_dup(guest_memfd);
10891074 }
10901075
10911076 region->region.guest_memfd = guest_memfd;
@@ -1097,20 +1082,18 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, · 5 unmodified lines
10971082 region->unused_phy_pages = sparsebit_alloc();
10981083 if (vm_arch_has_protected_memory(vm))
10991084 region->protected_phy_pages = sparsebit_alloc();
1100− sparsebit_set_num(region->unused_phy_pages,
1101− guest_paddr >> vm->page_shift, npages);
1085+ sparsebit_set_num(region->unused_phy_pages, gpa >> vm->page_shift, npages);
11021086 region->region.slot = slot;
11031087 region->region.flags = flags;
1104− region->region.guest_phys_addr = guest_paddr;
1088+ region->region.guest_phys_addr = gpa;
11051089 region->region.memory_size = npages * vm->page_size;
11061090 region->region.userspace_addr = (uintptr_t) region->host_mem;
11071091 ret = __vm_ioctl(vm, KVM_SET_USER_MEMORY_REGION2, &region->region);
11081092 TEST_ASSERT(ret == 0, "KVM_SET_USER_MEMORY_REGION2 IOCTL failed,\n"
11091093 " rc: %i errno: %i\n"
11101094 " slot: %u flags: 0x%x\n"
1111− " guest_phys_addr: 0x%lx size: 0x%lx guest_memfd: %d",
1112− ret, errno, slot, flags,
1113− guest_paddr, (uint64_t) region->region.memory_size,
1095+ " guest_phys_addr: 0x%lx size: 0x%llx guest_memfd: %d",
1096+ ret, errno, slot, flags, gpa, region->region.memory_size,
11141097 region->region.guest_memfd);
11151098
11161099 /* Add to quick lookup data structures */
@@ -1132,10 +1115,10 @@ void vm_mem_add(struct kvm_vm *vm, enum vm_mem_backing_src_type src_type, · 15 unmodified lines
11321115
11331116 void vm_userspace_mem_region_add(struct kvm_vm *vm,
11341117 enum vm_mem_backing_src_type src_type,
1135− uint64_t guest_paddr, uint32_t slot,
1136− uint64_t npages, uint32_t flags)
1118+ uint64_t gpa, uint32_t slot, uint64_t npages,
1119+ uint32_t flags)
11371120 {
1138− vm_mem_add(vm, src_type, guest_paddr, slot, npages, flags, -1, 0);
1121+ vm_mem_add(vm, src_type, gpa, slot, npages, flags, -1, 0);
11391122 }
11401123
11411124 /*
@@ -1201,6 +1184,16 @@ void vm_mem_region_set_flags(struct kvm_vm *vm, uint32_t slot, uint32_t flags) · 59 unmodified lines
12011184 ret, errno, slot, flags);
12021185 }
12031186
1187+void vm_mem_region_reload(struct kvm_vm *vm, uint32_t slot)
1188+{
1189+ struct userspace_mem_region *region = memslot2region(vm, slot);
1190+ struct kvm_userspace_memory_region2 tmp = region->region;
1191+
1192+ tmp.memory_size = 0;
1193+ vm_ioctl(vm, KVM_SET_USER_MEMORY_REGION2, &tmp);
1194+ vm_ioctl(vm, KVM_SET_USER_MEMORY_REGION2, &region->region);
1195+}
1196+
12041197 /*
12051198 * VM Memory Region Move
12061199 *
@@ -1456,8 +1449,6 @@ static vm_vaddr_t ____vm_vaddr_alloc(struct kvm_vm *vm, size_t sz, · 249 unmodified lines
14561449 pages--, vaddr += vm->page_size, paddr += vm->page_size) {
14571450
14581451 virt_pg_map(vm, vaddr, paddr);
1459−
1460− sparsebit_set(vm->vpages_mapped, vaddr >> vm->page_shift);
14611452 }
14621453
14631454 return vaddr_start;
@@ -1571,7 +1562,6 @@ void virt_map(struct kvm_vm *vm, uint64_t vaddr, uint64_t paddr, · 107 unmodified lines
15711562
15721563 while (npages--) {
15731564 virt_pg_map(vm, vaddr, paddr);
1574− sparsebit_set(vm->vpages_mapped, vaddr >> vm->page_shift);
15751565
15761566 vaddr += page_size;
15771567 paddr += page_size;
@@ -2025,6 +2015,7 @@ static struct exit_reason { · 447 unmodified lines
20252015 KVM_EXIT_STRING(NOTIFY),
20262016 KVM_EXIT_STRING(LOONGARCH_IOCSR),
20272017 KVM_EXIT_STRING(MEMORY_FAULT),
2018+ KVM_EXIT_STRING(ARM_SEA),
20282019 };
20292020
20302021 /*
@@ -2305,11 +2296,35 @@ __weak void kvm_selftest_arch_init(void) · 274 unmodified lines
23052296 {
23062297 }
23072298
2299+static void report_unexpected_signal(int signum)
2300+{
2301+#define KVM_CASE_SIGNUM(sig) \
2302+ case sig: TEST_FAIL("Unexpected " #sig " (%d)\n", signum)
2303+
2304+ switch (signum) {
2305+ KVM_CASE_SIGNUM(SIGBUS);
2306+ KVM_CASE_SIGNUM(SIGSEGV);
2307+ KVM_CASE_SIGNUM(SIGILL);
2308+ KVM_CASE_SIGNUM(SIGFPE);
2309+ default:
2310+ TEST_FAIL("Unexpected signal %d\n", signum);
2311+ }
2312+}
2313+
23082314 void __attribute((constructor)) kvm_selftest_init(void)
23092315 {
2316+ struct sigaction sig_sa = {
2317+ .sa_handler = report_unexpected_signal,
2318+ };
2319+
23102320 /* Tell stdout not to buffer its content. */
23112321 setbuf(stdout, NULL);
23122322
2323+ sigaction(SIGBUS, &sig_sa, NULL);
2324+ sigaction(SIGSEGV, &sig_sa, NULL);
2325+ sigaction(SIGILL, &sig_sa, NULL);
2326+ sigaction(SIGFPE, &sig_sa, NULL);
2327+
23132328 guest_random_seed = last_guest_seed = random();
23142329 pr_info("Random seed: 0x%x\n", guest_random_seed);
23152330
tools/testing/selftests/kvm/lib/loongarch/exception.S+6−0
@@ -51,9 +51,15 @@ handle_exception: · 50 unmodified lines
5151 st.d t0, sp, ESTAT_OFFSET_EXREGS
5252 csrrd t0, LOONGARCH_CSR_BADV
5353 st.d t0, sp, BADV_OFFSET_EXREGS
54+ csrrd t0, LOONGARCH_CSR_PRMD
55+ st.d t0, sp, PRMD_OFFSET_EXREGS
5456
5557 or a0, sp, zero
5658 bl route_exception
59+ ld.d t0, sp, PC_OFFSET_EXREGS
60+ csrwr t0, LOONGARCH_CSR_ERA
61+ ld.d t0, sp, PRMD_OFFSET_EXREGS
62+ csrwr t0, LOONGARCH_CSR_PRMD
5763 restore_gprs sp
5864 csrrd sp, LOONGARCH_CSR_KS0
5965 ertn
tools/testing/selftests/kvm/lib/loongarch/processor.c+45−2
@@ -3,6 +3,7 @@ · 2 unmodified lines
33 #include <assert.h>
44 #include <linux/compiler.h>
55
6+#include <asm/kvm.h>
67 #include "kvm_util.h"
78 #include "processor.h"
89 #include "ucall_common.h"
@@ -11,6 +12,7 @@ · 2 unmodified lines
1112 #define LOONGARCH_GUEST_STACK_VADDR_MIN 0x200000
1213
1314 static vm_paddr_t invalid_pgtable[4];
15+static vm_vaddr_t exception_handlers;
1416
1517 static uint64_t virt_pte_index(struct kvm_vm *vm, vm_vaddr_t gva, int level)
1618 {
@@ -183,7 +185,14 @@ void assert_on_unhandled_exception(struct kvm_vcpu *vcpu) · 166 unmodified lines
183185
184186 void route_exception(struct ex_regs *regs)
185187 {
188+ int vector;
186189 unsigned long pc, estat, badv;
190+ struct handlers *handlers;
191+
192+ handlers = (struct handlers *)exception_handlers;
193+ vector = (regs->estat & CSR_ESTAT_EXC) >> CSR_ESTAT_EXC_SHIFT;
194+ if (handlers && handlers->exception_handlers[vector])
195+ return handlers->exception_handlers[vector](regs);
187196
188197 pc = regs->pc;
189198 badv = regs->badv;
@@ -192,6 +201,32 @@ void route_exception(struct ex_regs *regs) · 2 unmodified lines
192201 while (1) ;
193202 }
194203
204+void vm_init_descriptor_tables(struct kvm_vm *vm)
205+{
206+ void *addr;
207+
208+ vm->handlers = __vm_vaddr_alloc(vm, sizeof(struct handlers),
209+ LOONGARCH_GUEST_STACK_VADDR_MIN, MEM_REGION_DATA);
210+
211+ addr = addr_gva2hva(vm, vm->handlers);
212+ memset(addr, 0, vm->page_size);
213+ exception_handlers = vm->handlers;
214+ sync_global_to_guest(vm, exception_handlers);
215+}
216+
217+void vm_install_exception_handler(struct kvm_vm *vm, int vector, handler_fn handler)
218+{
219+ struct handlers *handlers = addr_gva2hva(vm, vm->handlers);
220+
221+ assert(vector < VECTOR_NUM);
222+ handlers->exception_handlers[vector] = handler;
223+}
224+
225+uint32_t guest_get_vcpuid(void)
226+{
227+ return csr_read(LOONGARCH_CSR_CPUID);
228+}
229+
195230 void vcpu_args_set(struct kvm_vcpu *vcpu, unsigned int num, ...)
196231 {
197232 int i;
@@ -211,6 +246,11 @@ void vcpu_args_set(struct kvm_vcpu *vcpu, unsigned int num, ...) · 13 unmodified lines
211246 vcpu_regs_set(vcpu, &regs);
212247 }
213248
249+static void loongarch_set_reg(struct kvm_vcpu *vcpu, uint64_t id, uint64_t val)
250+{
251+ __vcpu_set_reg(vcpu, id, val);
252+}
253+
214254 static void loongarch_get_csr(struct kvm_vcpu *vcpu, uint64_t id, void *addr)
215255 {
216256 uint64_t csrid;
@@ -242,8 +282,8 @@ static void loongarch_vcpu_setup(struct kvm_vcpu *vcpu) · 25 unmodified lines
242282 TEST_FAIL("Unknown guest mode, mode: 0x%x", vm->mode);
243283 }
244284
245− /* user mode and page enable mode */
246− val = PLV_USER | CSR_CRMD_PG;
285+ /* kernel mode and page enable mode */
286+ val = PLV_KERN | CSR_CRMD_PG;
247287 loongarch_set_csr(vcpu, LOONGARCH_CSR_CRMD, val);
248288 loongarch_set_csr(vcpu, LOONGARCH_CSR_PRMD, val);
249289 loongarch_set_csr(vcpu, LOONGARCH_CSR_EUEN, 1);
@@ -251,7 +291,10 @@ static void loongarch_vcpu_setup(struct kvm_vcpu *vcpu) · 1 unmodified line
251291 loongarch_set_csr(vcpu, LOONGARCH_CSR_TCFG, 0);
252292 loongarch_set_csr(vcpu, LOONGARCH_CSR_ASID, 1);
253293
294+ /* time count start from 0 */
254295 val = 0;
296+ loongarch_set_reg(vcpu, KVM_REG_LOONGARCH_COUNTER, val);
297+
255298 width = vm->page_shift - 3;
256299
257300 switch (vm->pgtable_levels) {
tools/testing/selftests/kvm/lib/x86/memstress.c+1−1
@@ -63,7 +63,7 @@ void memstress_setup_ept(struct vmx_pages *vmx, struct kvm_vm *vm) · 62 unmodified lines
6363 {
6464 uint64_t start, end;
6565
66− prepare_eptp(vmx, vm, 0);
66+ prepare_eptp(vmx, vm);
6767
6868 /*
6969 * Identity map the first 4G and the test region with 1G pages so that
tools/testing/selftests/kvm/lib/x86/processor.c+41−41
@@ -158,10 +158,10 @@ bool kvm_is_tdp_enabled(void) · 157 unmodified lines
158158
159159 void virt_arch_pgd_alloc(struct kvm_vm *vm)
160160 {
161− TEST_ASSERT(vm->mode == VM_MODE_PXXV48_4K, "Attempt to use "
162− "unknown or unsupported guest mode, mode: 0x%x", vm->mode);
161+ TEST_ASSERT(vm->mode == VM_MODE_PXXVYY_4K,
162+ "Unknown or unsupported guest mode: 0x%x", vm->mode);
163163
164− /* If needed, create page map l4 table. */
164+ /* If needed, create the top-level page table. */
165165 if (!vm->pgd_created) {
166166 vm->pgd = vm_alloc_page_table(vm);
167167 vm->pgd_created = true;
@@ -218,11 +218,11 @@ static uint64_t *virt_create_upper_pte(struct kvm_vm *vm, · 50 unmodified lines
218218 void __virt_pg_map(struct kvm_vm *vm, uint64_t vaddr, uint64_t paddr, int level)
219219 {
220220 const uint64_t pg_size = PG_LEVEL_SIZE(level);
221− uint64_t *pml4e, *pdpe, *pde;
222− uint64_t *pte;
221+ uint64_t *pte = &vm->pgd;
222+ int current_level;
223223
224− TEST_ASSERT(vm->mode == VM_MODE_PXXV48_4K,
225− "Unknown or unsupported guest mode, mode: 0x%x", vm->mode);
224+ TEST_ASSERT(vm->mode == VM_MODE_PXXVYY_4K,
225+ "Unknown or unsupported guest mode: 0x%x", vm->mode);
226226
227227 TEST_ASSERT((vaddr % pg_size) == 0,
228228 "Virtual address not aligned,\n"
@@ -243,20 +243,17 @@ void __virt_pg_map(struct kvm_vm *vm, uint64_t vaddr, uint64_t paddr, int level) · 14 unmodified lines
243243 * Allocate upper level page tables, if not already present. Return
244244 * early if a hugepage was created.
245245 */
246− pml4e = virt_create_upper_pte(vm, &vm->pgd, vaddr, paddr, PG_LEVEL_512G, level);
247− if (*pml4e & PTE_LARGE_MASK)
248− return;
249−
250− pdpe = virt_create_upper_pte(vm, pml4e, vaddr, paddr, PG_LEVEL_1G, level);
251− if (*pdpe & PTE_LARGE_MASK)
252− return;
253−
254− pde = virt_create_upper_pte(vm, pdpe, vaddr, paddr, PG_LEVEL_2M, level);
255− if (*pde & PTE_LARGE_MASK)
256− return;
246+ for (current_level = vm->pgtable_levels;
247+ current_level > PG_LEVEL_4K;
248+ current_level--) {
249+ pte = virt_create_upper_pte(vm, pte, vaddr, paddr,
250+ current_level, level);
251+ if (*pte & PTE_LARGE_MASK)
252+ return;
253+ }
257254
258255 /* Fill in page table entry. */
259− pte = virt_get_pte(vm, pde, vaddr, PG_LEVEL_4K);
256+ pte = virt_get_pte(vm, pte, vaddr, PG_LEVEL_4K);
260257 TEST_ASSERT(!(*pte & PTE_PRESENT_MASK),
261258 "PTE already present for 4k page at vaddr: 0x%lx", vaddr);
262259 *pte = PTE_PRESENT_MASK | PTE_WRITABLE_MASK | (paddr & PHYSICAL_PAGE_MASK);
@@ -289,6 +286,8 @@ void virt_map_level(struct kvm_vm *vm, uint64_t vaddr, uint64_t paddr, · 26 unmodified lines
289286
290287 for (i = 0; i < nr_pages; i++) {
291288 __virt_pg_map(vm, vaddr, paddr, level);
289+ sparsebit_set_num(vm->vpages_mapped, vaddr >> vm->page_shift,
290+ nr_bytes / PAGE_SIZE);
292291
293292 vaddr += pg_size;
294293 paddr += pg_size;
@@ -310,40 +309,38 @@ static bool vm_is_target_pte(uint64_t *pte, int *level, int current_level) · 15 unmodified lines
310309 uint64_t *__vm_get_page_table_entry(struct kvm_vm *vm, uint64_t vaddr,
311310 int *level)
312311 {
313− uint64_t *pml4e, *pdpe, *pde;
312+ int va_width = 12 + (vm->pgtable_levels) * 9;
313+ uint64_t *pte = &vm->pgd;
314+ int current_level;
314315
315316 TEST_ASSERT(!vm->arch.is_pt_protected,
316317 "Walking page tables of protected guests is impossible");
317318
318− TEST_ASSERT(*level >= PG_LEVEL_NONE && *level < PG_LEVEL_NUM,
319+ TEST_ASSERT(*level >= PG_LEVEL_NONE && *level <= vm->pgtable_levels,
319320 "Invalid PG_LEVEL_* '%d'", *level);
320321
321− TEST_ASSERT(vm->mode == VM_MODE_PXXV48_4K, "Attempt to use "
322− "unknown or unsupported guest mode, mode: 0x%x", vm->mode);
322+ TEST_ASSERT(vm->mode == VM_MODE_PXXVYY_4K,
323+ "Unknown or unsupported guest mode: 0x%x", vm->mode);
323324 TEST_ASSERT(sparsebit_is_set(vm->vpages_valid,
324325 (vaddr >> vm->page_shift)),
325326 "Invalid virtual address, vaddr: 0x%lx",
326327 vaddr);
327328 /*
328− * Based on the mode check above there are 48 bits in the vaddr, so
329− * shift 16 to sign extend the last bit (bit-47),
329+ * Check that the vaddr is a sign-extended va_width value.
330330 */
331− TEST_ASSERT(vaddr == (((int64_t)vaddr << 16) >> 16),
332− "Canonical check failed. The virtual address is invalid.");
333−
334− pml4e = virt_get_pte(vm, &vm->pgd, vaddr, PG_LEVEL_512G);
335− if (vm_is_target_pte(pml4e, level, PG_LEVEL_512G))
336− return pml4e;
337−
338− pdpe = virt_get_pte(vm, pml4e, vaddr, PG_LEVEL_1G);
339− if (vm_is_target_pte(pdpe, level, PG_LEVEL_1G))
340− return pdpe;
341−
342− pde = virt_get_pte(vm, pdpe, vaddr, PG_LEVEL_2M);
343− if (vm_is_target_pte(pde, level, PG_LEVEL_2M))
344− return pde;
331+ TEST_ASSERT(vaddr ==
332+ (((int64_t)vaddr << (64 - va_width) >> (64 - va_width))),
333+ "Canonical check failed. The virtual address is invalid.");
334+
335+ for (current_level = vm->pgtable_levels;
336+ current_level > PG_LEVEL_4K;
337+ current_level--) {
338+ pte = virt_get_pte(vm, pte, vaddr, current_level);
339+ if (vm_is_target_pte(pte, level, current_level))
340+ return pte;
341+ }
345342
346− return virt_get_pte(vm, pde, vaddr, PG_LEVEL_4K);
343+ return virt_get_pte(vm, pte, vaddr, PG_LEVEL_4K);
347344 }
348345
349346 uint64_t *vm_get_page_table_entry(struct kvm_vm *vm, uint64_t vaddr)
@@ -526,7 +523,8 @@ static void vcpu_init_sregs(struct kvm_vm *vm, struct kvm_vcpu *vcpu) · 176 unmodified lines
526523 {
527524 struct kvm_sregs sregs;
528525
529− TEST_ASSERT_EQ(vm->mode, VM_MODE_PXXV48_4K);
526+ TEST_ASSERT(vm->mode == VM_MODE_PXXVYY_4K,
527+ "Unknown or unsupported guest mode: 0x%x", vm->mode);
530528
531529 /* Set mode specific system register values. */
532530 vcpu_sregs_get(vcpu, &sregs);
@@ -540,6 +538,8 @@ static void vcpu_init_sregs(struct kvm_vm *vm, struct kvm_vcpu *vcpu) · 7 unmodified lines
540538 sregs.cr4 |= X86_CR4_PAE | X86_CR4_OSFXSR;
541539 if (kvm_cpu_has(X86_FEATURE_XSAVE))
542540 sregs.cr4 |= X86_CR4_OSXSAVE;
541+ if (vm->pgtable_levels == 5)
542+ sregs.cr4 |= X86_CR4_LA57;
543543 sregs.efer |= (EFER_LME | EFER_LMA | EFER_NX);
544544
545545 kvm_seg_set_unusable(&sregs.ldt);
tools/testing/selftests/kvm/lib/x86/vmx.c+4−5
@@ -401,11 +401,11 @@ void __nested_pg_map(struct vmx_pages *vmx, struct kvm_vm *vm, · 400 unmodified lines
401401 struct eptPageTableEntry *pt = vmx->eptp_hva, *pte;
402402 uint16_t index;
403403
404− TEST_ASSERT(vm->mode == VM_MODE_PXXV48_4K, "Attempt to use "
405− "unknown or unsupported guest mode, mode: 0x%x", vm->mode);
404+ TEST_ASSERT(vm->mode == VM_MODE_PXXVYY_4K,
405+ "Unknown or unsupported guest mode: 0x%x", vm->mode);
406406
407407 TEST_ASSERT((nested_paddr >> 48) == 0,
408− "Nested physical address 0x%lx requires 5-level paging",
408+ "Nested physical address 0x%lx is > 48-bits and requires 5-level EPT",
409409 nested_paddr);
410410 TEST_ASSERT((nested_paddr % page_size) == 0,
411411 "Nested physical address not on page boundary,\n"
@@ -534,8 +534,7 @@ bool kvm_cpu_has_ept(void) · 122 unmodified lines
534534 return ctrl & SECONDARY_EXEC_ENABLE_EPT;
535535 }
536536
537−void prepare_eptp(struct vmx_pages *vmx, struct kvm_vm *vm,
538− uint32_t eptp_memslot)
537+void prepare_eptp(struct vmx_pages *vmx, struct kvm_vm *vm)
539538 {
540539 TEST_ASSERT(kvm_cpu_has_ept(), "KVM doesn't support nested EPT");
541540
tools/testing/selftests/kvm/loongarch/arch_timer.c+200−0
@@ -0,0 +1,200 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/*
3+ * The test validates periodic/one-shot constant timer IRQ using
4+ * CSR.TCFG and CSR.TVAL registers.
5+ */
6+#include "arch_timer.h"
7+#include "kvm_util.h"
8+#include "processor.h"
9+#include "timer_test.h"
10+#include "ucall_common.h"
11+
12+static void do_idle(void)
13+{
14+ unsigned int intid;
15+ unsigned long estat;
16+
17+ __asm__ __volatile__("idle 0" : : : "memory");
18+
19+ estat = csr_read(LOONGARCH_CSR_ESTAT);
20+ intid = !!(estat & BIT(INT_TI));
21+
22+ /* Make sure pending timer IRQ arrived */
23+ GUEST_ASSERT_EQ(intid, 1);
24+ csr_write(CSR_TINTCLR_TI, LOONGARCH_CSR_TINTCLR);
25+}
26+
27+static void guest_irq_handler(struct ex_regs *regs)
28+{
29+ unsigned int intid;
30+ uint32_t cpu = guest_get_vcpuid();
31+ uint64_t xcnt, val, cfg, xcnt_diff_us;
32+ struct test_vcpu_shared_data *shared_data = &vcpu_shared_data[cpu];
33+
34+ intid = !!(regs->estat & BIT(INT_TI));
35+
36+ /* Make sure we are dealing with the correct timer IRQ */
37+ GUEST_ASSERT_EQ(intid, 1);
38+
39+ cfg = timer_get_cfg();
40+ if (cfg & CSR_TCFG_PERIOD) {
41+ WRITE_ONCE(shared_data->nr_iter, shared_data->nr_iter - 1);
42+ if (shared_data->nr_iter == 0)
43+ disable_timer();
44+ csr_write(CSR_TINTCLR_TI, LOONGARCH_CSR_TINTCLR);
45+ return;
46+ }
47+
48+ /*
49+ * On real machine, value of LOONGARCH_CSR_TVAL is BIT_ULL(48) - 1
50+ * On virtual machine, its value counts down from BIT_ULL(48) - 1
51+ */
52+ val = timer_get_val();
53+ xcnt = timer_get_cycles();
54+ xcnt_diff_us = cycles_to_usec(xcnt - shared_data->xcnt);
55+
56+ /* Basic 'timer condition met' check */
57+ __GUEST_ASSERT(val > cfg,
58+ "val = 0x%lx, cfg = 0x%lx, xcnt_diff_us = 0x%lx",
59+ val, cfg, xcnt_diff_us);
60+
61+ csr_write(CSR_TINTCLR_TI, LOONGARCH_CSR_TINTCLR);
62+ WRITE_ONCE(shared_data->nr_iter, shared_data->nr_iter + 1);
63+}
64+
65+static void guest_test_period_timer(uint32_t cpu)
66+{
67+ uint32_t irq_iter, config_iter;
68+ uint64_t us;
69+ struct test_vcpu_shared_data *shared_data = &vcpu_shared_data[cpu];
70+
71+ shared_data->nr_iter = test_args.nr_iter;
72+ shared_data->xcnt = timer_get_cycles();
73+ us = msecs_to_usecs(test_args.timer_period_ms) + test_args.timer_err_margin_us;
74+ timer_set_next_cmp_ms(test_args.timer_period_ms, true);
75+
76+ for (config_iter = 0; config_iter < test_args.nr_iter; config_iter++) {
77+ /* Setup a timeout for the interrupt to arrive */
78+ udelay(us);
79+ }
80+
81+ irq_iter = READ_ONCE(shared_data->nr_iter);
82+ __GUEST_ASSERT(irq_iter == 0,
83+ "irq_iter = 0x%x.\n"
84+ " Guest period timer interrupt was not triggered within the specified\n"
85+ " interval, try to increase the error margin by [-e] option.\n",
86+ irq_iter);
87+}
88+
89+static void guest_test_oneshot_timer(uint32_t cpu)
90+{
91+ uint32_t irq_iter, config_iter;
92+ uint64_t us;
93+ struct test_vcpu_shared_data *shared_data = &vcpu_shared_data[cpu];
94+
95+ shared_data->nr_iter = 0;
96+ shared_data->guest_stage = 0;
97+ us = msecs_to_usecs(test_args.timer_period_ms) + test_args.timer_err_margin_us;
98+ for (config_iter = 0; config_iter < test_args.nr_iter; config_iter++) {
99+ shared_data->xcnt = timer_get_cycles();
100+
101+ /* Setup the next interrupt */
102+ timer_set_next_cmp_ms(test_args.timer_period_ms, false);
103+ /* Setup a timeout for the interrupt to arrive */
104+ udelay(us);
105+
106+ irq_iter = READ_ONCE(shared_data->nr_iter);
107+ __GUEST_ASSERT(config_iter + 1 == irq_iter,
108+ "config_iter + 1 = 0x%x, irq_iter = 0x%x.\n"
109+ " Guest timer interrupt was not triggered within the specified\n"
110+ " interval, try to increase the error margin by [-e] option.\n",
111+ config_iter + 1, irq_iter);
112+ }
113+}
114+
115+static void guest_test_emulate_timer(uint32_t cpu)
116+{
117+ uint32_t config_iter;
118+ uint64_t xcnt_diff_us, us;
119+ struct test_vcpu_shared_data *shared_data = &vcpu_shared_data[cpu];
120+
121+ local_irq_disable();
122+ shared_data->nr_iter = 0;
123+ us = msecs_to_usecs(test_args.timer_period_ms);
124+ for (config_iter = 0; config_iter < test_args.nr_iter; config_iter++) {
125+ shared_data->xcnt = timer_get_cycles();
126+
127+ /* Setup the next interrupt */
128+ timer_set_next_cmp_ms(test_args.timer_period_ms, false);
129+ do_idle();
130+
131+ xcnt_diff_us = cycles_to_usec(timer_get_cycles() - shared_data->xcnt);
132+ __GUEST_ASSERT(xcnt_diff_us >= us,
133+ "xcnt_diff_us = 0x%lx, us = 0x%lx.\n",
134+ xcnt_diff_us, us);
135+ }
136+ local_irq_enable();
137+}
138+
139+static void guest_time_count_test(uint32_t cpu)
140+{
141+ uint32_t config_iter;
142+ unsigned long start, end, prev, us;
143+
144+ /* Assuming that test case starts to run in 1 second */
145+ start = timer_get_cycles();
146+ us = msec_to_cycles(1000);
147+ __GUEST_ASSERT(start <= us,
148+ "start = 0x%lx, us = 0x%lx.\n",
149+ start, us);
150+
151+ us = msec_to_cycles(test_args.timer_period_ms);
152+ for (config_iter = 0; config_iter < test_args.nr_iter; config_iter++) {
153+ start = timer_get_cycles();
154+ end = start + us;
155+ /* test time count growing up always */
156+ while (start < end) {
157+ prev = start;
158+ start = timer_get_cycles();
159+ __GUEST_ASSERT(prev <= start,
160+ "prev = 0x%lx, start = 0x%lx.\n",
161+ prev, start);
162+ }
163+ }
164+}
165+
166+static void guest_code(void)
167+{
168+ uint32_t cpu = guest_get_vcpuid();
169+
170+ /* must run at first */
171+ guest_time_count_test(cpu);
172+
173+ timer_irq_enable();
174+ local_irq_enable();
175+ guest_test_period_timer(cpu);
176+ guest_test_oneshot_timer(cpu);
177+ guest_test_emulate_timer(cpu);
178+
179+ GUEST_DONE();
180+}
181+
182+struct kvm_vm *test_vm_create(void)
183+{
184+ struct kvm_vm *vm;
185+ int nr_vcpus = test_args.nr_vcpus;
186+
187+ vm = vm_create_with_vcpus(nr_vcpus, guest_code, vcpus);
188+ vm_init_descriptor_tables(vm);
189+ vm_install_exception_handler(vm, EXCCODE_INT, guest_irq_handler);
190+
191+ /* Make all the test's cmdline args visible to the guest */
192+ sync_global_to_guest(vm, test_args);
193+
194+ return vm;
195+}
196+
197+void test_vm_cleanup(struct kvm_vm *vm)
198+{
199+ kvm_vm_free(vm);
200+}
tools/testing/selftests/kvm/mmu_stress_test.c+5−5
@@ -263,8 +263,10 @@ static void calc_default_nr_vcpus(void) · 262 unmodified lines
263263 TEST_ASSERT(!r, "sched_getaffinity failed, errno = %d (%s)",
264264 errno, strerror(errno));
265265
266− nr_vcpus = CPU_COUNT(&possible_mask) * 3/4;
266+ nr_vcpus = CPU_COUNT(&possible_mask);
267267 TEST_ASSERT(nr_vcpus > 0, "Uh, no CPUs?");
268+ if (nr_vcpus >= 2)
269+ nr_vcpus = nr_vcpus * 3/4;
268270 }
269271
270272 int main(int argc, char *argv[])
@@ -360,11 +362,9 @@ int main(int argc, char *argv[]) · 89 unmodified lines
360362
361363 #ifdef __x86_64__
362364 /* Identity map memory in the guest using 1gb pages. */
363− for (i = 0; i < slot_size; i += SZ_1G)
364− __virt_pg_map(vm, gpa + i, gpa + i, PG_LEVEL_1G);
365+ virt_map_level(vm, gpa, gpa, slot_size, PG_LEVEL_1G);
365366 #else
366− for (i = 0; i < slot_size; i += vm->page_size)
367− virt_pg_map(vm, gpa + i, gpa + i);
367+ virt_map(vm, gpa, gpa, slot_size >> vm->page_shift);
368368 #endif
369369 }
370370
tools/testing/selftests/kvm/pre_fault_memory_test.c+14−18
@@ -17,13 +17,13 @@ · 16 unmodified lines
1717 #define TEST_NPAGES (TEST_SIZE / PAGE_SIZE)
1818 #define TEST_SLOT 10
1919
20−static void guest_code(uint64_t base_gpa)
20+static void guest_code(uint64_t base_gva)
2121 {
2222 volatile uint64_t val __used;
2323 int i;
2424
2525 for (i = 0; i < TEST_NPAGES; i++) {
26− uint64_t *src = (uint64_t *)(base_gpa + i * PAGE_SIZE);
26+ uint64_t *src = (uint64_t *)(base_gva + i * PAGE_SIZE);
2727
2828 val = *src;
2929 }
@@ -161,6 +161,7 @@ static void pre_fault_memory(struct kvm_vcpu *vcpu, u64 base_gpa, u64 offset, · 131 unmodified lines
161161
162162 static void __test_pre_fault_memory(unsigned long vm_type, bool private)
163163 {
164+ uint64_t gpa, gva, alignment, guest_page_size;
164165 const struct vm_shape shape = {
165166 .mode = VM_MODE_DEFAULT,
166167 .type = vm_type,
@@ -170,35 +171,30 @@ static void __test_pre_fault_memory(unsigned long vm_type, bool private) · 3 unmodified lines
170171 struct kvm_vm *vm;
171172 struct ucall uc;
172173
173− uint64_t guest_test_phys_mem;
174− uint64_t guest_test_virt_mem;
175− uint64_t alignment, guest_page_size;
176−
177174 vm = vm_create_shape_with_one_vcpu(shape, &vcpu, guest_code);
178175
179176 alignment = guest_page_size = vm_guest_mode_params[VM_MODE_DEFAULT].page_size;
180− guest_test_phys_mem = (vm->max_gfn - TEST_NPAGES) * guest_page_size;
177+ gpa = (vm->max_gfn - TEST_NPAGES) * guest_page_size;
181178 #ifdef __s390x__
182179 alignment = max(0x100000UL, guest_page_size);
183180 #else
184181 alignment = SZ_2M;
185182 #endif
186− guest_test_phys_mem = align_down(guest_test_phys_mem, alignment);
187− guest_test_virt_mem = guest_test_phys_mem & ((1ULL << (vm->va_bits - 1)) - 1);
183+ gpa = align_down(gpa, alignment);
184+ gva = gpa & ((1ULL << (vm->va_bits - 1)) - 1);
188185
189− vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS,
190− guest_test_phys_mem, TEST_SLOT, TEST_NPAGES,
191− private ? KVM_MEM_GUEST_MEMFD : 0);
192− virt_map(vm, guest_test_virt_mem, guest_test_phys_mem, TEST_NPAGES);
186+ vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, gpa, TEST_SLOT,
187+ TEST_NPAGES, private ? KVM_MEM_GUEST_MEMFD : 0);
188+ virt_map(vm, gva, gpa, TEST_NPAGES);
193189
194190 if (private)
195− vm_mem_set_private(vm, guest_test_phys_mem, TEST_SIZE);
191+ vm_mem_set_private(vm, gpa, TEST_SIZE);
196192
197− pre_fault_memory(vcpu, guest_test_phys_mem, 0, SZ_2M, 0, private);
198− pre_fault_memory(vcpu, guest_test_phys_mem, SZ_2M, PAGE_SIZE * 2, PAGE_SIZE, private);
199− pre_fault_memory(vcpu, guest_test_phys_mem, TEST_SIZE, PAGE_SIZE, PAGE_SIZE, private);
193+ pre_fault_memory(vcpu, gpa, 0, SZ_2M, 0, private);
194+ pre_fault_memory(vcpu, gpa, SZ_2M, PAGE_SIZE * 2, PAGE_SIZE, private);
195+ pre_fault_memory(vcpu, gpa, TEST_SIZE, PAGE_SIZE, PAGE_SIZE, private);
200196
201− vcpu_args_set(vcpu, 1, guest_test_virt_mem);
197+ vcpu_args_set(vcpu, 1, gva);
202198 vcpu_run(vcpu);
203199
204200 run = vcpu->run;
tools/testing/selftests/kvm/riscv/get-reg-list.c+4−0
@@ -133,6 +133,7 @@ bool filter_reg(__u64 reg) · 132 unmodified lines
133133 case KVM_REG_RISCV_SBI_EXT | KVM_REG_RISCV_SBI_SINGLE | KVM_RISCV_SBI_EXT_SUSP:
134134 case KVM_REG_RISCV_SBI_EXT | KVM_REG_RISCV_SBI_SINGLE | KVM_RISCV_SBI_EXT_STA:
135135 case KVM_REG_RISCV_SBI_EXT | KVM_REG_RISCV_SBI_SINGLE | KVM_RISCV_SBI_EXT_FWFT:
136+ case KVM_REG_RISCV_SBI_EXT | KVM_REG_RISCV_SBI_SINGLE | KVM_RISCV_SBI_EXT_MPXY:
136137 case KVM_REG_RISCV_SBI_EXT | KVM_REG_RISCV_SBI_SINGLE | KVM_RISCV_SBI_EXT_EXPERIMENTAL:
137138 case KVM_REG_RISCV_SBI_EXT | KVM_REG_RISCV_SBI_SINGLE | KVM_RISCV_SBI_EXT_VENDOR:
138139 return true;
@@ -639,6 +640,7 @@ static const char *sbi_ext_single_id_to_str(__u64 reg_off) · 500 unmodified lines
639640 KVM_SBI_EXT_ARR(KVM_RISCV_SBI_EXT_SUSP),
640641 KVM_SBI_EXT_ARR(KVM_RISCV_SBI_EXT_STA),
641642 KVM_SBI_EXT_ARR(KVM_RISCV_SBI_EXT_FWFT),
643+ KVM_SBI_EXT_ARR(KVM_RISCV_SBI_EXT_MPXY),
642644 KVM_SBI_EXT_ARR(KVM_RISCV_SBI_EXT_EXPERIMENTAL),
643645 KVM_SBI_EXT_ARR(KVM_RISCV_SBI_EXT_VENDOR),
644646 };
@@ -1142,6 +1144,7 @@ KVM_SBI_EXT_SUBLIST_CONFIG(sta, STA); · 497 unmodified lines
11421144 KVM_SBI_EXT_SIMPLE_CONFIG(pmu, PMU);
11431145 KVM_SBI_EXT_SIMPLE_CONFIG(dbcn, DBCN);
11441146 KVM_SBI_EXT_SIMPLE_CONFIG(susp, SUSP);
1147+KVM_SBI_EXT_SIMPLE_CONFIG(mpxy, MPXY);
11451148 KVM_SBI_EXT_SUBLIST_CONFIG(fwft, FWFT);
11461149
11471150 KVM_ISA_EXT_SUBLIST_CONFIG(aia, AIA);
@@ -1222,6 +1225,7 @@ struct vcpu_reg_list *vcpu_configs[] = { · 74 unmodified lines
12221225 &config_sbi_pmu,
12231226 &config_sbi_dbcn,
12241227 &config_sbi_susp,
1228+ &config_sbi_mpxy,
12251229 &config_sbi_fwft,
12261230 &config_aia,
12271231 &config_fp_f,
tools/testing/selftests/kvm/s390/user_operexec.c+140−0
@@ -0,0 +1,140 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/* Test operation exception forwarding.
3+ *
4+ * Copyright IBM Corp. 2025
5+ *
6+ * Authors:
7+ * Janosch Frank <frankja@linux.ibm.com>
8+ */
9+#include "kselftest.h"
10+#include "kvm_util.h"
11+#include "test_util.h"
12+#include "sie.h"
13+
14+#include <linux/kvm.h>
15+
16+static void guest_code_instr0(void)
17+{
18+ asm(".word 0x0000");
19+}
20+
21+static void test_user_instr0(void)
22+{
23+ struct kvm_vcpu *vcpu;
24+ struct kvm_vm *vm;
25+ int rc;
26+
27+ vm = vm_create_with_one_vcpu(&vcpu, guest_code_instr0);
28+ rc = __vm_enable_cap(vm, KVM_CAP_S390_USER_INSTR0, 0);
29+ TEST_ASSERT_EQ(0, rc);
30+
31+ vcpu_run(vcpu);
32+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_S390_SIEIC);
33+ TEST_ASSERT_EQ(vcpu->run->s390_sieic.icptcode, ICPT_OPEREXC);
34+ TEST_ASSERT_EQ(vcpu->run->s390_sieic.ipa, 0);
35+
36+ kvm_vm_free(vm);
37+}
38+
39+static void guest_code_user_operexec(void)
40+{
41+ asm(".word 0x0807");
42+}
43+
44+static void test_user_operexec(void)
45+{
46+ struct kvm_vcpu *vcpu;
47+ struct kvm_vm *vm;
48+ int rc;
49+
50+ vm = vm_create_with_one_vcpu(&vcpu, guest_code_user_operexec);
51+ rc = __vm_enable_cap(vm, KVM_CAP_S390_USER_OPEREXEC, 0);
52+ TEST_ASSERT_EQ(0, rc);
53+
54+ vcpu_run(vcpu);
55+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_S390_SIEIC);
56+ TEST_ASSERT_EQ(vcpu->run->s390_sieic.icptcode, ICPT_OPEREXC);
57+ TEST_ASSERT_EQ(vcpu->run->s390_sieic.ipa, 0x0807);
58+
59+ kvm_vm_free(vm);
60+
61+ /*
62+ * Since user_operexec is the superset it can be used for the
63+ * 0 instruction.
64+ */
65+ vm = vm_create_with_one_vcpu(&vcpu, guest_code_instr0);
66+ rc = __vm_enable_cap(vm, KVM_CAP_S390_USER_OPEREXEC, 0);
67+ TEST_ASSERT_EQ(0, rc);
68+
69+ vcpu_run(vcpu);
70+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_S390_SIEIC);
71+ TEST_ASSERT_EQ(vcpu->run->s390_sieic.icptcode, ICPT_OPEREXC);
72+ TEST_ASSERT_EQ(vcpu->run->s390_sieic.ipa, 0);
73+
74+ kvm_vm_free(vm);
75+}
76+
77+/* combine user_instr0 and user_operexec */
78+static void test_user_operexec_combined(void)
79+{
80+ struct kvm_vcpu *vcpu;
81+ struct kvm_vm *vm;
82+ int rc;
83+
84+ vm = vm_create_with_one_vcpu(&vcpu, guest_code_user_operexec);
85+ rc = __vm_enable_cap(vm, KVM_CAP_S390_USER_INSTR0, 0);
86+ TEST_ASSERT_EQ(0, rc);
87+ rc = __vm_enable_cap(vm, KVM_CAP_S390_USER_OPEREXEC, 0);
88+ TEST_ASSERT_EQ(0, rc);
89+
90+ vcpu_run(vcpu);
91+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_S390_SIEIC);
92+ TEST_ASSERT_EQ(vcpu->run->s390_sieic.icptcode, ICPT_OPEREXC);
93+ TEST_ASSERT_EQ(vcpu->run->s390_sieic.ipa, 0x0807);
94+
95+ kvm_vm_free(vm);
96+
97+ /* Reverse enablement order */
98+ vm = vm_create_with_one_vcpu(&vcpu, guest_code_user_operexec);
99+ rc = __vm_enable_cap(vm, KVM_CAP_S390_USER_OPEREXEC, 0);
100+ TEST_ASSERT_EQ(0, rc);
101+ rc = __vm_enable_cap(vm, KVM_CAP_S390_USER_INSTR0, 0);
102+ TEST_ASSERT_EQ(0, rc);
103+
104+ vcpu_run(vcpu);
105+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_S390_SIEIC);
106+ TEST_ASSERT_EQ(vcpu->run->s390_sieic.icptcode, ICPT_OPEREXC);
107+ TEST_ASSERT_EQ(vcpu->run->s390_sieic.ipa, 0x0807);
108+
109+ kvm_vm_free(vm);
110+}
111+
112+/*
113+ * Run all tests above.
114+ *
115+ * Enablement after VCPU has been added is automatically tested since
116+ * we enable the capability after VCPU creation.
117+ */
118+static struct testdef {
119+ const char *name;
120+ void (*test)(void);
121+} testlist[] = {
122+ { "instr0", test_user_instr0 },
123+ { "operexec", test_user_operexec },
124+ { "operexec_combined", test_user_operexec_combined},
125+};
126+
127+int main(int argc, char *argv[])
128+{
129+ int idx;
130+
131+ TEST_REQUIRE(kvm_has_cap(KVM_CAP_S390_USER_INSTR0));
132+
133+ ksft_print_header();
134+ ksft_set_plan(ARRAY_SIZE(testlist));
135+ for (idx = 0; idx < ARRAY_SIZE(testlist); idx++) {
136+ testlist[idx].test();
137+ ksft_test_result_pass("%s\n", testlist[idx].name);
138+ }
139+ ksft_finished();
140+}
tools/testing/selftests/kvm/x86/hyperv_features.c+1−1
@@ -94,7 +94,7 @@ static void guest_hcall(vm_vaddr_t pgs_gpa, struct hcall_data *hcall) · 93 unmodified lines
9494
9595 if (!(hcall->control & HV_HYPERCALL_FAST_BIT)) {
9696 input = pgs_gpa;
97− output = pgs_gpa + 4096;
97+ output = pgs_gpa + PAGE_SIZE;
9898 } else {
9999 input = output = 0;
100100 }
tools/testing/selftests/kvm/x86/hyperv_ipi.c+9−9
@@ -102,7 +102,7 @@ static void sender_guest_code(void *hcall_page, vm_vaddr_t pgs_gpa) · 101 unmodified lines
102102 /* 'Slow' HvCallSendSyntheticClusterIpi to RECEIVER_VCPU_ID_1 */
103103 ipi->vector = IPI_VECTOR;
104104 ipi->cpu_mask = 1 << RECEIVER_VCPU_ID_1;
105− hyperv_hypercall(HVCALL_SEND_IPI, pgs_gpa, pgs_gpa + 4096);
105+ hyperv_hypercall(HVCALL_SEND_IPI, pgs_gpa, pgs_gpa + PAGE_SIZE);
106106 nop_loop();
107107 GUEST_ASSERT(ipis_rcvd[RECEIVER_VCPU_ID_1] == ++ipis_expected[0]);
108108 GUEST_ASSERT(ipis_rcvd[RECEIVER_VCPU_ID_2] == ipis_expected[1]);
@@ -116,13 +116,13 @@ static void sender_guest_code(void *hcall_page, vm_vaddr_t pgs_gpa) · 7 unmodified lines
116116 GUEST_SYNC(stage++);
117117
118118 /* 'Slow' HvCallSendSyntheticClusterIpiEx to RECEIVER_VCPU_ID_1 */
119− memset(hcall_page, 0, 4096);
119+ memset(hcall_page, 0, PAGE_SIZE);
120120 ipi_ex->vector = IPI_VECTOR;
121121 ipi_ex->vp_set.format = HV_GENERIC_SET_SPARSE_4K;
122122 ipi_ex->vp_set.valid_bank_mask = 1 << 0;
123123 ipi_ex->vp_set.bank_contents[0] = BIT(RECEIVER_VCPU_ID_1);
124124 hyperv_hypercall(HVCALL_SEND_IPI_EX | (1 << HV_HYPERCALL_VARHEAD_OFFSET),
125− pgs_gpa, pgs_gpa + 4096);
125+ pgs_gpa, pgs_gpa + PAGE_SIZE);
126126 nop_loop();
127127 GUEST_ASSERT(ipis_rcvd[RECEIVER_VCPU_ID_1] == ++ipis_expected[0]);
128128 GUEST_ASSERT(ipis_rcvd[RECEIVER_VCPU_ID_2] == ipis_expected[1]);
@@ -138,13 +138,13 @@ static void sender_guest_code(void *hcall_page, vm_vaddr_t pgs_gpa) · 9 unmodified lines
138138 GUEST_SYNC(stage++);
139139
140140 /* 'Slow' HvCallSendSyntheticClusterIpiEx to RECEIVER_VCPU_ID_2 */
141− memset(hcall_page, 0, 4096);
141+ memset(hcall_page, 0, PAGE_SIZE);
142142 ipi_ex->vector = IPI_VECTOR;
143143 ipi_ex->vp_set.format = HV_GENERIC_SET_SPARSE_4K;
144144 ipi_ex->vp_set.valid_bank_mask = 1 << 1;
145145 ipi_ex->vp_set.bank_contents[0] = BIT(RECEIVER_VCPU_ID_2 - 64);
146146 hyperv_hypercall(HVCALL_SEND_IPI_EX | (1 << HV_HYPERCALL_VARHEAD_OFFSET),
147− pgs_gpa, pgs_gpa + 4096);
147+ pgs_gpa, pgs_gpa + PAGE_SIZE);
148148 nop_loop();
149149 GUEST_ASSERT(ipis_rcvd[RECEIVER_VCPU_ID_1] == ipis_expected[0]);
150150 GUEST_ASSERT(ipis_rcvd[RECEIVER_VCPU_ID_2] == ++ipis_expected[1]);
@@ -160,14 +160,14 @@ static void sender_guest_code(void *hcall_page, vm_vaddr_t pgs_gpa) · 9 unmodified lines
160160 GUEST_SYNC(stage++);
161161
162162 /* 'Slow' HvCallSendSyntheticClusterIpiEx to both RECEIVER_VCPU_ID_{1,2} */
163− memset(hcall_page, 0, 4096);
163+ memset(hcall_page, 0, PAGE_SIZE);
164164 ipi_ex->vector = IPI_VECTOR;
165165 ipi_ex->vp_set.format = HV_GENERIC_SET_SPARSE_4K;
166166 ipi_ex->vp_set.valid_bank_mask = 1 << 1 | 1;
167167 ipi_ex->vp_set.bank_contents[0] = BIT(RECEIVER_VCPU_ID_1);
168168 ipi_ex->vp_set.bank_contents[1] = BIT(RECEIVER_VCPU_ID_2 - 64);
169169 hyperv_hypercall(HVCALL_SEND_IPI_EX | (2 << HV_HYPERCALL_VARHEAD_OFFSET),
170− pgs_gpa, pgs_gpa + 4096);
170+ pgs_gpa, pgs_gpa + PAGE_SIZE);
171171 nop_loop();
172172 GUEST_ASSERT(ipis_rcvd[RECEIVER_VCPU_ID_1] == ++ipis_expected[0]);
173173 GUEST_ASSERT(ipis_rcvd[RECEIVER_VCPU_ID_2] == ++ipis_expected[1]);
@@ -183,10 +183,10 @@ static void sender_guest_code(void *hcall_page, vm_vaddr_t pgs_gpa) · 9 unmodified lines
183183 GUEST_SYNC(stage++);
184184
185185 /* 'Slow' HvCallSendSyntheticClusterIpiEx to HV_GENERIC_SET_ALL */
186− memset(hcall_page, 0, 4096);
186+ memset(hcall_page, 0, PAGE_SIZE);
187187 ipi_ex->vector = IPI_VECTOR;
188188 ipi_ex->vp_set.format = HV_GENERIC_SET_ALL;
189− hyperv_hypercall(HVCALL_SEND_IPI_EX, pgs_gpa, pgs_gpa + 4096);
189+ hyperv_hypercall(HVCALL_SEND_IPI_EX, pgs_gpa, pgs_gpa + PAGE_SIZE);
190190 nop_loop();
191191 GUEST_ASSERT(ipis_rcvd[RECEIVER_VCPU_ID_1] == ++ipis_expected[0]);
192192 GUEST_ASSERT(ipis_rcvd[RECEIVER_VCPU_ID_2] == ++ipis_expected[1]);
tools/testing/selftests/kvm/x86/hyperv_tlb_flush.c+1−1
@@ -621,7 +621,7 @@ int main(int argc, char *argv[]) · 620 unmodified lines
621621 for (i = 0; i < NTEST_PAGES; i++) {
622622 pte = vm_get_page_table_entry(vm, data->test_pages + i * PAGE_SIZE);
623623 gpa = addr_hva2gpa(vm, pte);
624− __virt_pg_map(vm, gva + PAGE_SIZE * i, gpa & PAGE_MASK, PG_LEVEL_4K);
624+ virt_pg_map(vm, gva + PAGE_SIZE * i, gpa & PAGE_MASK);
625625 data->test_pages_pte[i] = gva + (gpa & ~PAGE_MASK);
626626 }
627627
tools/testing/selftests/kvm/x86/nested_close_kvm_test.crenamed from tools/testing/selftests/kvm/x86/vmx_close_while_nested_test.c+33−9
@@ -1,7 +1,5 @@
11 // SPDX-License-Identifier: GPL-2.0-only
22 /*
3− * vmx_close_while_nested
4− *
53 * Copyright (C) 2019, Red Hat, Inc.
64 *
75 * Verify that nothing bad happens if a KVM user exits with open
@@ -12,6 +10,7 @@ · 4 unmodified lines
1210 #include "kvm_util.h"
1311 #include "processor.h"
1412 #include "vmx.h"
13+#include "svm_util.h"
1514
1615 #include <string.h>
1716 #include <sys/ioctl.h>
@@ -22,6 +21,8 @@ enum { · 4 unmodified lines
2221 PORT_L0_EXIT = 0x2000,
2322 };
2423
24+#define L2_GUEST_STACK_SIZE 64
25+
2526 static void l2_guest_code(void)
2627 {
2728 /* Exit to L0 */
@@ -29,9 +30,8 @@ static void l2_guest_code(void) · 1 unmodified line
2930 : : [port] "d" (PORT_L0_EXIT) : "rax");
3031 }
3132
32−static void l1_guest_code(struct vmx_pages *vmx_pages)
33+static void l1_vmx_code(struct vmx_pages *vmx_pages)
3334 {
34−#define L2_GUEST_STACK_SIZE 64
3535 unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE];
3636
3737 GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages));
@@ -45,19 +45,43 @@ static void l1_guest_code(struct vmx_pages *vmx_pages) · 7 unmodified lines
4545 GUEST_ASSERT(0);
4646 }
4747
48+static void l1_svm_code(struct svm_test_data *svm)
49+{
50+ unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE];
51+
52+ /* Prepare the VMCB for L2 execution. */
53+ generic_svm_setup(svm, l2_guest_code,
54+ &l2_guest_stack[L2_GUEST_STACK_SIZE]);
55+
56+ run_guest(svm->vmcb, svm->vmcb_gpa);
57+ GUEST_ASSERT(0);
58+}
59+
60+static void l1_guest_code(void *data)
61+{
62+ if (this_cpu_has(X86_FEATURE_VMX))
63+ l1_vmx_code(data);
64+ else
65+ l1_svm_code(data);
66+}
67+
4868 int main(int argc, char *argv[])
4969 {
50− vm_vaddr_t vmx_pages_gva;
70+ vm_vaddr_t guest_gva;
5171 struct kvm_vcpu *vcpu;
5272 struct kvm_vm *vm;
5373
54− TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX));
74+ TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX) ||
75+ kvm_cpu_has(X86_FEATURE_SVM));
5576
5677 vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code);
5778
58− /* Allocate VMX pages and shared descriptors (vmx_pages). */
59− vcpu_alloc_vmx(vm, &vmx_pages_gva);
60− vcpu_args_set(vcpu, 1, vmx_pages_gva);
79+ if (kvm_cpu_has(X86_FEATURE_VMX))
80+ vcpu_alloc_vmx(vm, &guest_gva);
81+ else
82+ vcpu_alloc_svm(vm, &guest_gva);
83+
84+ vcpu_args_set(vcpu, 1, guest_gva);
6185
6286 for (;;) {
6387 volatile struct kvm_run *run = vcpu->run;
tools/testing/selftests/kvm/x86/nested_invalid_cr3_test.c+116−0
@@ -0,0 +1,116 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/*
3+ * Copyright (C) 2025, Google LLC.
4+ *
5+ * This test verifies that L1 fails to enter L2 with an invalid CR3, and
6+ * succeeds otherwise.
7+ */
8+#include "kvm_util.h"
9+#include "vmx.h"
10+#include "svm_util.h"
11+#include "kselftest.h"
12+
13+
14+#define L2_GUEST_STACK_SIZE 64
15+
16+static void l2_guest_code(void)
17+{
18+ vmcall();
19+}
20+
21+static void l1_svm_code(struct svm_test_data *svm)
22+{
23+ unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE];
24+ uintptr_t save_cr3;
25+
26+ generic_svm_setup(svm, l2_guest_code,
27+ &l2_guest_stack[L2_GUEST_STACK_SIZE]);
28+
29+ /* Try to run L2 with invalid CR3 and make sure it fails */
30+ save_cr3 = svm->vmcb->save.cr3;
31+ svm->vmcb->save.cr3 = -1ull;
32+ run_guest(svm->vmcb, svm->vmcb_gpa);
33+ GUEST_ASSERT(svm->vmcb->control.exit_code == SVM_EXIT_ERR);
34+
35+ /* Now restore CR3 and make sure L2 runs successfully */
36+ svm->vmcb->save.cr3 = save_cr3;
37+ run_guest(svm->vmcb, svm->vmcb_gpa);
38+ GUEST_ASSERT(svm->vmcb->control.exit_code == SVM_EXIT_VMMCALL);
39+
40+ GUEST_DONE();
41+}
42+
43+static void l1_vmx_code(struct vmx_pages *vmx_pages)
44+{
45+ unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE];
46+ uintptr_t save_cr3;
47+
48+ GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages));
49+ GUEST_ASSERT(load_vmcs(vmx_pages));
50+
51+ prepare_vmcs(vmx_pages, l2_guest_code,
52+ &l2_guest_stack[L2_GUEST_STACK_SIZE]);
53+
54+ /* Try to run L2 with invalid CR3 and make sure it fails */
55+ save_cr3 = vmreadz(GUEST_CR3);
56+ vmwrite(GUEST_CR3, -1ull);
57+ GUEST_ASSERT(!vmlaunch());
58+ GUEST_ASSERT(vmreadz(VM_EXIT_REASON) ==
59+ (EXIT_REASON_FAILED_VMENTRY | EXIT_REASON_INVALID_STATE));
60+
61+ /* Now restore CR3 and make sure L2 runs successfully */
62+ vmwrite(GUEST_CR3, save_cr3);
63+ GUEST_ASSERT(!vmlaunch());
64+ GUEST_ASSERT(vmreadz(VM_EXIT_REASON) == EXIT_REASON_VMCALL);
65+
66+ GUEST_DONE();
67+}
68+
69+static void l1_guest_code(void *data)
70+{
71+ if (this_cpu_has(X86_FEATURE_VMX))
72+ l1_vmx_code(data);
73+ else
74+ l1_svm_code(data);
75+}
76+
77+int main(int argc, char *argv[])
78+{
79+ struct kvm_vcpu *vcpu;
80+ struct kvm_vm *vm;
81+ vm_vaddr_t guest_gva = 0;
82+
83+ TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX) ||
84+ kvm_cpu_has(X86_FEATURE_SVM));
85+
86+ vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code);
87+
88+ if (kvm_cpu_has(X86_FEATURE_VMX))
89+ vcpu_alloc_vmx(vm, &guest_gva);
90+ else
91+ vcpu_alloc_svm(vm, &guest_gva);
92+
93+ vcpu_args_set(vcpu, 1, guest_gva);
94+
95+ for (;;) {
96+ struct ucall uc;
97+
98+ vcpu_run(vcpu);
99+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
100+
101+ switch (get_ucall(vcpu, &uc)) {
102+ case UCALL_ABORT:
103+ REPORT_GUEST_ASSERT(uc);
104+ case UCALL_SYNC:
105+ break;
106+ case UCALL_DONE:
107+ goto done;
108+ default:
109+ TEST_FAIL("Unknown ucall %lu", uc.cmd);
110+ }
111+ }
112+
113+done:
114+ kvm_vm_free(vm);
115+ return 0;
116+}
tools/testing/selftests/kvm/x86/nested_tsc_adjust_test.crenamed from tools/testing/selftests/kvm/x86/vmx_tsc_adjust_test.c+44−35
@@ -1,7 +1,5 @@
11 // SPDX-License-Identifier: GPL-2.0-only
22 /*
3− * vmx_tsc_adjust_test
4− *
53 * Copyright (C) 2018, Google LLC.
64 *
75 * IA32_TSC_ADJUST test
@@ -22,6 +20,7 @@ · 14 unmodified lines
2220 #include "kvm_util.h"
2321 #include "processor.h"
2422 #include "vmx.h"
23+#include "svm_util.h"
2524
2625 #include <string.h>
2726 #include <sys/ioctl.h>
@@ -35,6 +34,8 @@ · 7 unmodified lines
3534 #define TSC_ADJUST_VALUE (1ll << 32)
3635 #define TSC_OFFSET_VALUE -(1ll << 48)
3736
37+#define L2_GUEST_STACK_SIZE 64
38+
3839 enum {
3940 PORT_ABORT = 0x1000,
4041 PORT_REPORT,
@@ -72,42 +73,47 @@ static void l2_guest_code(void) · 31 unmodified lines
7273 __asm__ __volatile__("vmcall");
7374 }
7475
75−static void l1_guest_code(struct vmx_pages *vmx_pages)
76+static void l1_guest_code(void *data)
7677 {
77−#define L2_GUEST_STACK_SIZE 64
7878 unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE];
79− uint32_t control;
80− uintptr_t save_cr3;
8179
80+ /* Set TSC from L1 and make sure TSC_ADJUST is updated correctly */
8281 GUEST_ASSERT(rdtsc() < TSC_ADJUST_VALUE);
8382 wrmsr(MSR_IA32_TSC, rdtsc() - TSC_ADJUST_VALUE);
8483 check_ia32_tsc_adjust(-1 * TSC_ADJUST_VALUE);
8584
86− GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages));
87− GUEST_ASSERT(load_vmcs(vmx_pages));
88−
89− /* Prepare the VMCS for L2 execution. */
90− prepare_vmcs(vmx_pages, l2_guest_code,
91− &l2_guest_stack[L2_GUEST_STACK_SIZE]);
92− control = vmreadz(CPU_BASED_VM_EXEC_CONTROL);
93− control |= CPU_BASED_USE_MSR_BITMAPS | CPU_BASED_USE_TSC_OFFSETTING;
94− vmwrite(CPU_BASED_VM_EXEC_CONTROL, control);
95− vmwrite(TSC_OFFSET, TSC_OFFSET_VALUE);
96−
97− /* Jump into L2. First, test failure to load guest CR3. */
98− save_cr3 = vmreadz(GUEST_CR3);
99− vmwrite(GUEST_CR3, -1ull);
100− GUEST_ASSERT(!vmlaunch());
101− GUEST_ASSERT(vmreadz(VM_EXIT_REASON) ==
102− (EXIT_REASON_FAILED_VMENTRY | EXIT_REASON_INVALID_STATE));
103− check_ia32_tsc_adjust(-1 * TSC_ADJUST_VALUE);
104− vmwrite(GUEST_CR3, save_cr3);
105−
106− GUEST_ASSERT(!vmlaunch());
107− GUEST_ASSERT(vmreadz(VM_EXIT_REASON) == EXIT_REASON_VMCALL);
85+ /*
86+ * Run L2 with TSC_OFFSET. L2 will write to TSC, and L1 is not
87+ * intercepting the write so it should update L1's TSC_ADJUST.
88+ */
89+ if (this_cpu_has(X86_FEATURE_VMX)) {
90+ struct vmx_pages *vmx_pages = data;
91+ uint32_t control;
92+
93+ GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages));
94+ GUEST_ASSERT(load_vmcs(vmx_pages));
95+
96+ prepare_vmcs(vmx_pages, l2_guest_code,
97+ &l2_guest_stack[L2_GUEST_STACK_SIZE]);
98+ control = vmreadz(CPU_BASED_VM_EXEC_CONTROL);
99+ control |= CPU_BASED_USE_MSR_BITMAPS | CPU_BASED_USE_TSC_OFFSETTING;
100+ vmwrite(CPU_BASED_VM_EXEC_CONTROL, control);
101+ vmwrite(TSC_OFFSET, TSC_OFFSET_VALUE);
102+
103+ GUEST_ASSERT(!vmlaunch());
104+ GUEST_ASSERT(vmreadz(VM_EXIT_REASON) == EXIT_REASON_VMCALL);
105+ } else {
106+ struct svm_test_data *svm = data;
107+
108+ generic_svm_setup(svm, l2_guest_code,
109+ &l2_guest_stack[L2_GUEST_STACK_SIZE]);
110+
111+ svm->vmcb->control.tsc_offset = TSC_OFFSET_VALUE;
112+ run_guest(svm->vmcb, svm->vmcb_gpa);
113+ GUEST_ASSERT(svm->vmcb->control.exit_code == SVM_EXIT_VMMCALL);
114+ }
108115
109116 check_ia32_tsc_adjust(-2 * TSC_ADJUST_VALUE);
110−
111117 GUEST_DONE();
112118 }
113119
@@ -119,16 +125,19 @@ static void report(int64_t val) · 5 unmodified lines
119125
120126 int main(int argc, char *argv[])
121127 {
122− vm_vaddr_t vmx_pages_gva;
128+ vm_vaddr_t nested_gva;
123129 struct kvm_vcpu *vcpu;
124130
125− TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX));
131+ TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX) ||
132+ kvm_cpu_has(X86_FEATURE_SVM));
126133
127− vm = vm_create_with_one_vcpu(&vcpu, (void *) l1_guest_code);
134+ vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code);
135+ if (kvm_cpu_has(X86_FEATURE_VMX))
136+ vcpu_alloc_vmx(vm, &nested_gva);
137+ else
138+ vcpu_alloc_svm(vm, &nested_gva);
128139
129− /* Allocate VMX pages and shared descriptors (vmx_pages). */
130− vcpu_alloc_vmx(vm, &vmx_pages_gva);
131− vcpu_args_set(vcpu, 1, vmx_pages_gva);
140+ vcpu_args_set(vcpu, 1, nested_gva);
132141
133142 for (;;) {
134143 struct ucall uc;
tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.crenamed from tools/testing/selftests/kvm/x86/vmx_nested_tsc_scaling_test.c+43−5
@@ -13,6 +13,7 @@ · 12 unmodified lines
1313
1414 #include "kvm_util.h"
1515 #include "vmx.h"
16+#include "svm_util.h"
1617 #include "kselftest.h"
1718
1819 /* L2 is scaled up (from L1's perspective) by this factor */
@@ -79,7 +80,30 @@ static void l2_guest_code(void) · 60 unmodified lines
7980 __asm__ __volatile__("vmcall");
8081 }
8182
82−static void l1_guest_code(struct vmx_pages *vmx_pages)
83+static void l1_svm_code(struct svm_test_data *svm)
84+{
85+ unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE];
86+
87+ /* check that L1's frequency looks alright before launching L2 */
88+ check_tsc_freq(UCHECK_L1);
89+
90+ generic_svm_setup(svm, l2_guest_code,
91+ &l2_guest_stack[L2_GUEST_STACK_SIZE]);
92+
93+ /* enable TSC scaling for L2 */
94+ wrmsr(MSR_AMD64_TSC_RATIO, L2_SCALE_FACTOR << 32);
95+
96+ /* launch L2 */
97+ run_guest(svm->vmcb, svm->vmcb_gpa);
98+ GUEST_ASSERT(svm->vmcb->control.exit_code == SVM_EXIT_VMMCALL);
99+
100+ /* check that L1's frequency still looks good */
101+ check_tsc_freq(UCHECK_L1);
102+
103+ GUEST_DONE();
104+}
105+
106+static void l1_vmx_code(struct vmx_pages *vmx_pages)
83107 {
84108 unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE];
85109 uint32_t control;
@@ -116,11 +140,19 @@ static void l1_guest_code(struct vmx_pages *vmx_pages) · 30 unmodified lines
116140 GUEST_DONE();
117141 }
118142
143+static void l1_guest_code(void *data)
144+{
145+ if (this_cpu_has(X86_FEATURE_VMX))
146+ l1_vmx_code(data);
147+ else
148+ l1_svm_code(data);
149+}
150+
119151 int main(int argc, char *argv[])
120152 {
121153 struct kvm_vcpu *vcpu;
122154 struct kvm_vm *vm;
123− vm_vaddr_t vmx_pages_gva;
155+ vm_vaddr_t guest_gva = 0;
124156
125157 uint64_t tsc_start, tsc_end;
126158 uint64_t tsc_khz;
@@ -129,7 +161,8 @@ int main(int argc, char *argv[]) · 2 unmodified lines
129161 uint64_t l1_tsc_freq = 0;
130162 uint64_t l2_tsc_freq = 0;
131163
132− TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX));
164+ TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX) ||
165+ kvm_cpu_has(X86_FEATURE_SVM));
133166 TEST_REQUIRE(kvm_has_cap(KVM_CAP_TSC_CONTROL));
134167 TEST_REQUIRE(sys_clocksource_is_based_on_tsc());
135168
@@ -152,8 +185,13 @@ int main(int argc, char *argv[]) · 16 unmodified lines
152185 printf("real TSC frequency is around: %"PRIu64"\n", l0_tsc_freq);
153186
154187 vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code);
155− vcpu_alloc_vmx(vm, &vmx_pages_gva);
156− vcpu_args_set(vcpu, 1, vmx_pages_gva);
188+
189+ if (kvm_cpu_has(X86_FEATURE_VMX))
190+ vcpu_alloc_vmx(vm, &guest_gva);
191+ else
192+ vcpu_alloc_svm(vm, &guest_gva);
193+
194+ vcpu_args_set(vcpu, 1, guest_gva);
157195
158196 tsc_khz = __vcpu_ioctl(vcpu, KVM_GET_TSC_KHZ, NULL);
159197 TEST_ASSERT(tsc_khz != -1, "vcpu ioctl KVM_GET_TSC_KHZ failed");
tools/testing/selftests/kvm/x86/private_mem_conversions_test.c+3−6
@@ -380,7 +380,7 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, uint32_t · 379 unmodified lines
380380 struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
381381 pthread_t threads[KVM_MAX_VCPUS];
382382 struct kvm_vm *vm;
383− int memfd, i, r;
383+ int memfd, i;
384384
385385 const struct vm_shape shape = {
386386 .mode = VM_MODE_DEFAULT,
@@ -428,11 +428,8 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, uint32_t · 41 unmodified lines
428428 * should prevent the VM from being fully destroyed until the last
429429 * reference to the guest_memfd is also put.
430430 */
431− r = fallocate(memfd, FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE, 0, memfd_size);
432− TEST_ASSERT(!r, __KVM_SYSCALL_ERROR("fallocate()", r));
433−
434− r = fallocate(memfd, FALLOC_FL_KEEP_SIZE, 0, memfd_size);
435− TEST_ASSERT(!r, __KVM_SYSCALL_ERROR("fallocate()", r));
431+ kvm_fallocate(memfd, FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE, 0, memfd_size);
432+ kvm_fallocate(memfd, FALLOC_FL_KEEP_SIZE, 0, memfd_size);
436433
437434 close(memfd);
438435 }
tools/testing/selftests/kvm/x86/sev_smoke_test.c+1−1
@@ -104,7 +104,7 @@ static void test_sync_vmsa(uint32_t type, uint64_t policy) · 103 unmodified lines
104104 vm_sev_launch(vm, policy, NULL);
105105
106106 /* This page is shared, so make it decrypted. */
107− memset(hva, 0, 4096);
107+ memset(hva, 0, PAGE_SIZE);
108108
109109 vcpu_run(vcpu);
110110
tools/testing/selftests/kvm/x86/state_test.c+1−1
@@ -141,7 +141,7 @@ static void __attribute__((__flatten__)) guest_code(void *arg) · 140 unmodified lines
141141
142142 if (this_cpu_has(X86_FEATURE_XSAVE)) {
143143 uint64_t supported_xcr0 = this_cpu_supported_xcr0();
144− uint8_t buffer[4096];
144+ uint8_t buffer[PAGE_SIZE];
145145
146146 memset(buffer, 0xcc, sizeof(buffer));
147147
tools/testing/selftests/kvm/x86/userspace_io_test.c+1−1
@@ -85,7 +85,7 @@ int main(int argc, char *argv[]) · 84 unmodified lines
8585 regs.rcx = 1;
8686 if (regs.rcx == 3)
8787 regs.rcx = 8192;
88− memset((void *)run + run->io.data_offset, 0xaa, 4096);
88+ memset((void *)run + run->io.data_offset, 0xaa, PAGE_SIZE);
8989 vcpu_regs_set(vcpu, &regs);
9090 }
9191
tools/testing/selftests/kvm/x86/vmx_dirty_log_test.c+6−6
@@ -120,17 +120,17 @@ static void test_vmx_dirty_log(bool enable_ept) · 119 unmodified lines
120120 * GPAs as the EPT enabled case.
121121 */
122122 if (enable_ept) {
123− prepare_eptp(vmx, vm, 0);
123+ prepare_eptp(vmx, vm);
124124 nested_map_memslot(vmx, vm, 0);
125− nested_map(vmx, vm, NESTED_TEST_MEM1, GUEST_TEST_MEM, 4096);
126− nested_map(vmx, vm, NESTED_TEST_MEM2, GUEST_TEST_MEM, 4096);
125+ nested_map(vmx, vm, NESTED_TEST_MEM1, GUEST_TEST_MEM, PAGE_SIZE);
126+ nested_map(vmx, vm, NESTED_TEST_MEM2, GUEST_TEST_MEM, PAGE_SIZE);
127127 }
128128
129129 bmap = bitmap_zalloc(TEST_MEM_PAGES);
130130 host_test_mem = addr_gpa2hva(vm, GUEST_TEST_MEM);
131131
132132 while (!done) {
133− memset(host_test_mem, 0xaa, TEST_MEM_PAGES * 4096);
133+ memset(host_test_mem, 0xaa, TEST_MEM_PAGES * PAGE_SIZE);
134134 vcpu_run(vcpu);
135135 TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
136136
@@ -153,9 +153,9 @@ static void test_vmx_dirty_log(bool enable_ept) · 16 unmodified lines
153153 }
154154
155155 TEST_ASSERT(!test_bit(1, bmap), "Page 1 incorrectly reported dirty");
156− TEST_ASSERT(host_test_mem[4096 / 8] == 0xaaaaaaaaaaaaaaaaULL, "Page 1 written by guest");
156+ TEST_ASSERT(host_test_mem[PAGE_SIZE / 8] == 0xaaaaaaaaaaaaaaaaULL, "Page 1 written by guest");
157157 TEST_ASSERT(!test_bit(2, bmap), "Page 2 incorrectly reported dirty");
158− TEST_ASSERT(host_test_mem[8192 / 8] == 0xaaaaaaaaaaaaaaaaULL, "Page 2 written by guest");
158+ TEST_ASSERT(host_test_mem[PAGE_SIZE*2 / 8] == 0xaaaaaaaaaaaaaaaaULL, "Page 2 written by guest");
159159 break;
160160 case UCALL_DONE:
161161 done = true;
tools/testing/selftests/kvm/x86/vmx_nested_la57_state_test.c+132−0
@@ -0,0 +1,132 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/*
3+ * Copyright (C) 2025, Google LLC.
4+ *
5+ * Test KVM's ability to save and restore nested state when the L1 guest
6+ * is using 5-level paging and the L2 guest is using 4-level paging.
7+ *
8+ * This test would have failed prior to commit 9245fd6b8531 ("KVM: x86:
9+ * model canonical checks more precisely").
10+ */
11+#include "test_util.h"
12+#include "kvm_util.h"
13+#include "processor.h"
14+#include "vmx.h"
15+
16+#define LA57_GS_BASE 0xff2bc0311fb00000ull
17+
18+static void l2_guest_code(void)
19+{
20+ /*
21+ * Sync with L0 to trigger save/restore. After
22+ * resuming, execute VMCALL to exit back to L1.
23+ */
24+ GUEST_SYNC(1);
25+ vmcall();
26+}
27+
28+static void l1_guest_code(struct vmx_pages *vmx_pages)
29+{
30+#define L2_GUEST_STACK_SIZE 64
31+ unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE];
32+ u64 guest_cr4;
33+ vm_paddr_t pml5_pa, pml4_pa;
34+ u64 *pml5;
35+ u64 exit_reason;
36+
37+ /* Set GS_BASE to a value that is only canonical with LA57. */
38+ wrmsr(MSR_GS_BASE, LA57_GS_BASE);
39+ GUEST_ASSERT(rdmsr(MSR_GS_BASE) == LA57_GS_BASE);
40+
41+ GUEST_ASSERT(vmx_pages->vmcs_gpa);
42+ GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages));
43+ GUEST_ASSERT(load_vmcs(vmx_pages));
44+
45+ prepare_vmcs(vmx_pages, l2_guest_code,
46+ &l2_guest_stack[L2_GUEST_STACK_SIZE]);
47+
48+ /*
49+ * Set up L2 with a 4-level page table by pointing its CR3 to
50+ * L1's first PML4 table and clearing CR4.LA57. This creates
51+ * the CR4.LA57 mismatch that exercises the bug.
52+ */
53+ pml5_pa = get_cr3() & PHYSICAL_PAGE_MASK;
54+ pml5 = (u64 *)pml5_pa;
55+ pml4_pa = pml5[0] & PHYSICAL_PAGE_MASK;
56+ vmwrite(GUEST_CR3, pml4_pa);
57+
58+ guest_cr4 = vmreadz(GUEST_CR4);
59+ guest_cr4 &= ~X86_CR4_LA57;
60+ vmwrite(GUEST_CR4, guest_cr4);
61+
62+ GUEST_ASSERT(!vmlaunch());
63+
64+ exit_reason = vmreadz(VM_EXIT_REASON);
65+ GUEST_ASSERT(exit_reason == EXIT_REASON_VMCALL);
66+}
67+
68+void guest_code(struct vmx_pages *vmx_pages)
69+{
70+ l1_guest_code(vmx_pages);
71+ GUEST_DONE();
72+}
73+
74+int main(int argc, char *argv[])
75+{
76+ vm_vaddr_t vmx_pages_gva = 0;
77+ struct kvm_vm *vm;
78+ struct kvm_vcpu *vcpu;
79+ struct kvm_x86_state *state;
80+ struct ucall uc;
81+ int stage;
82+
83+ TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX));
84+ TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_LA57));
85+ TEST_REQUIRE(kvm_has_cap(KVM_CAP_NESTED_STATE));
86+
87+ vm = vm_create_with_one_vcpu(&vcpu, guest_code);
88+
89+ /*
90+ * L1 needs to read its own PML5 table to set up L2. Identity map
91+ * the PML5 table to facilitate this.
92+ */
93+ virt_map(vm, vm->pgd, vm->pgd, 1);
94+
95+ vcpu_alloc_vmx(vm, &vmx_pages_gva);
96+ vcpu_args_set(vcpu, 1, vmx_pages_gva);
97+
98+ for (stage = 1;; stage++) {
99+ vcpu_run(vcpu);
100+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
101+
102+ switch (get_ucall(vcpu, &uc)) {
103+ case UCALL_ABORT:
104+ REPORT_GUEST_ASSERT(uc);
105+ /* NOT REACHED */
106+ case UCALL_SYNC:
107+ break;
108+ case UCALL_DONE:
109+ goto done;
110+ default:
111+ TEST_FAIL("Unknown ucall %lu", uc.cmd);
112+ }
113+
114+ TEST_ASSERT(uc.args[1] == stage,
115+ "Expected stage %d, got stage %lu", stage, (ulong)uc.args[1]);
116+ if (stage == 1) {
117+ pr_info("L2 is active; performing save/restore.\n");
118+ state = vcpu_save_state(vcpu);
119+
120+ kvm_vm_release(vm);
121+
122+ /* Restore state in a new VM. */
123+ vcpu = vm_recreate_with_one_vcpu(vm);
124+ vcpu_load_state(vcpu, state);
125+ kvm_x86_state_cleanup(state);
126+ }
127+ }
128+
129+done:
130+ kvm_vm_free(vm);
131+ return 0;
132+}
tools/testing/selftests/kvm/x86/xapic_ipi_test.c+2−3
@@ -256,7 +256,7 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, · 255 unmodified lines
256256 int nodes = 0;
257257 time_t start_time, last_update, now;
258258 time_t interval_secs = 1;
259− int i, r;
259+ int i;
260260 int from, to;
261261 unsigned long bit;
262262 uint64_t hlt_count;
@@ -267,9 +267,8 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, · 4 unmodified lines
267267 delay_usecs);
268268
269269 /* Get set of first 64 numa nodes available */
270− r = get_mempolicy(NULL, &nodemask, sizeof(nodemask) * 8,
270+ kvm_get_mempolicy(NULL, &nodemask, sizeof(nodemask) * 8,
271271 0, MPOL_F_MEMS_ALLOWED);
272− TEST_ASSERT(r == 0, "get_mempolicy failed errno=%d", errno);
273272
274273 fprintf(stderr, "Numa nodes found amongst first %lu possible nodes "
275274 "(each 1-bit indicates node is present): %#lx\n",
virt/kvm/Kconfig+0−3
@@ -78,9 +78,6 @@ config HAVE_KVM_IRQ_BYPASS · 77 unmodified lines
7878 tristate
7979 select IRQ_BYPASS_MANAGER
8080
81−config HAVE_KVM_VCPU_ASYNC_IOCTL
82− bool
83−
8481 config HAVE_KVM_VCPU_RUN_PID_CHANGE
8582 bool
8683
virt/kvm/eventfd.c+1−1
@@ -707,7 +707,7 @@ bool kvm_notify_irqfd_resampler(struct kvm *kvm, · 706 unmodified lines
707707 */
708708 int kvm_irqfd_init(void)
709709 {
710− irqfd_cleanup_wq = alloc_workqueue("kvm-irqfd-cleanup", 0, 0);
710+ irqfd_cleanup_wq = alloc_workqueue("kvm-irqfd-cleanup", WQ_PERCPU, 0);
711711 if (!irqfd_cleanup_wq)
712712 return -ENOMEM;
713713
virt/kvm/guest_memfd.c+279−94
@@ -1,18 +1,47 @@
11 // SPDX-License-Identifier: GPL-2.0
2+#include <linux/anon_inodes.h>
23 #include <linux/backing-dev.h>
34 #include <linux/falloc.h>
5+#include <linux/fs.h>
46 #include <linux/kvm_host.h>
7+#include <linux/mempolicy.h>
8+#include <linux/pseudo_fs.h>
59 #include <linux/pagemap.h>
6−#include <linux/anon_inodes.h>
710
811 #include "kvm_mm.h"
912
10−struct kvm_gmem {
13+static struct vfsmount *kvm_gmem_mnt;
14+
15+/*
16+ * A guest_memfd instance can be associated multiple VMs, each with its own
17+ * "view" of the underlying physical memory.
18+ *
19+ * The gmem's inode is effectively the raw underlying physical storage, and is
20+ * used to track properties of the physical memory, while each gmem file is
21+ * effectively a single VM's view of that storage, and is used to track assets
22+ * specific to its associated VM, e.g. memslots=>gmem bindings.
23+ */
24+struct gmem_file {
1125 struct kvm *kvm;
1226 struct xarray bindings;
1327 struct list_head entry;
1428 };
1529
30+struct gmem_inode {
31+ struct shared_policy policy;
32+ struct inode vfs_inode;
33+
34+ u64 flags;
35+};
36+
37+static __always_inline struct gmem_inode *GMEM_I(struct inode *inode)
38+{
39+ return container_of(inode, struct gmem_inode, vfs_inode);
40+}
41+
42+#define kvm_gmem_for_each_file(f, mapping) \
43+ list_for_each_entry(f, &(mapping)->i_private_list, entry)
44+
1645 /**
1746 * folio_file_pfn - like folio_file_page, but return a pfn.
1847 * @folio: The folio which contains this index.
@@ -25,6 +54,11 @@ static inline kvm_pfn_t folio_file_pfn(struct folio *folio, pgoff_t index) · 6 unmodified lines
2554 return folio_pfn(folio) + (index & (folio_nr_pages(folio) - 1));
2655 }
2756
57+static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
58+{
59+ return gfn - slot->base_gfn + slot->gmem.pgoff;
60+}
61+
2862 static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
2963 pgoff_t index, struct folio *folio)
3064 {
@@ -77,9 +111,9 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · 46 unmodified lines
77111 * The order will be passed when creating the guest_memfd, and
78112 * checked when creating memslots.
79113 */
80− WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, 1 << folio_order(folio)));
81− index = gfn - slot->base_gfn + slot->gmem.pgoff;
82− index = ALIGN_DOWN(index, 1 << folio_order(folio));
114+ WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
115+ index = kvm_gmem_get_index(slot, gfn);
116+ index = ALIGN_DOWN(index, folio_nr_pages(folio));
83117 r = __kvm_gmem_prepare_folio(kvm, slot, index, folio);
84118 if (!r)
85119 kvm_gmem_mark_prepared(folio);
@@ -99,27 +133,45 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, · 13 unmodified lines
99133 static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
100134 {
101135 /* TODO: Support huge pages. */
102− return filemap_grab_folio(inode->i_mapping, index);
136+ struct mempolicy *policy;
137+ struct folio *folio;
138+
139+ /*
140+ * Fast-path: See if folio is already present in mapping to avoid
141+ * policy_lookup.
142+ */
143+ folio = __filemap_get_folio(inode->i_mapping, index,
144+ FGP_LOCK | FGP_ACCESSED, 0);
145+ if (!IS_ERR(folio))
146+ return folio;
147+
148+ policy = mpol_shared_policy_lookup(&GMEM_I(inode)->policy, index);
149+ folio = __filemap_get_folio_mpol(inode->i_mapping, index,
150+ FGP_LOCK | FGP_ACCESSED | FGP_CREAT,
151+ mapping_gfp_mask(inode->i_mapping), policy);
152+ mpol_cond_put(policy);
153+
154+ return folio;
103155 }
104156
105157 static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter(struct inode *inode)
106158 {
107− if ((u64)inode->i_private & GUEST_MEMFD_FLAG_INIT_SHARED)
159+ if (GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)
108160 return KVM_FILTER_SHARED;
109161
110162 return KVM_FILTER_PRIVATE;
111163 }
112164
113−static void __kvm_gmem_invalidate_begin(struct kvm_gmem *gmem, pgoff_t start,
165+static void __kvm_gmem_invalidate_begin(struct gmem_file *f, pgoff_t start,
114166 pgoff_t end,
115167 enum kvm_gfn_range_filter attr_filter)
116168 {
117169 bool flush = false, found_memslot = false;
118170 struct kvm_memory_slot *slot;
119− struct kvm *kvm = gmem->kvm;
171+ struct kvm *kvm = f->kvm;
120172 unsigned long index;
121173
122− xa_for_each_range(&gmem->bindings, index, slot, start, end - 1) {
174+ xa_for_each_range(&f->bindings, index, slot, start, end - 1) {
123175 pgoff_t pgoff = slot->gmem.pgoff;
124176
125177 struct kvm_gfn_range gfn_range = {
@@ -150,22 +202,21 @@ static void __kvm_gmem_invalidate_begin(struct kvm_gmem *gmem, pgoff_t start, · 24 unmodified lines
150202 static void kvm_gmem_invalidate_begin(struct inode *inode, pgoff_t start,
151203 pgoff_t end)
152204 {
153− struct list_head *gmem_list = &inode->i_mapping->i_private_list;
154205 enum kvm_gfn_range_filter attr_filter;
155− struct kvm_gmem *gmem;
206+ struct gmem_file *f;
156207
157208 attr_filter = kvm_gmem_get_invalidate_filter(inode);
158209
159− list_for_each_entry(gmem, gmem_list, entry)
160− __kvm_gmem_invalidate_begin(gmem, start, end, attr_filter);
210+ kvm_gmem_for_each_file(f, inode->i_mapping)
211+ __kvm_gmem_invalidate_begin(f, start, end, attr_filter);
161212 }
162213
163−static void __kvm_gmem_invalidate_end(struct kvm_gmem *gmem, pgoff_t start,
214+static void __kvm_gmem_invalidate_end(struct gmem_file *f, pgoff_t start,
164215 pgoff_t end)
165216 {
166− struct kvm *kvm = gmem->kvm;
217+ struct kvm *kvm = f->kvm;
167218
168− if (xa_find(&gmem->bindings, &start, end - 1, XA_PRESENT)) {
219+ if (xa_find(&f->bindings, &start, end - 1, XA_PRESENT)) {
169220 KVM_MMU_LOCK(kvm);
170221 kvm_mmu_invalidate_end(kvm);
171222 KVM_MMU_UNLOCK(kvm);
@@ -175,11 +226,10 @@ static void __kvm_gmem_invalidate_end(struct kvm_gmem *gmem, pgoff_t start, · 3 unmodified lines
175226 static void kvm_gmem_invalidate_end(struct inode *inode, pgoff_t start,
176227 pgoff_t end)
177228 {
178− struct list_head *gmem_list = &inode->i_mapping->i_private_list;
179− struct kvm_gmem *gmem;
229+ struct gmem_file *f;
180230
181− list_for_each_entry(gmem, gmem_list, entry)
182− __kvm_gmem_invalidate_end(gmem, start, end);
231+ kvm_gmem_for_each_file(f, inode->i_mapping)
232+ __kvm_gmem_invalidate_end(f, start, end);
183233 }
184234
185235 static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
@@ -277,9 +327,9 @@ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset, · 91 unmodified lines
277327
278328 static int kvm_gmem_release(struct inode *inode, struct file *file)
279329 {
280− struct kvm_gmem *gmem = file->private_data;
330+ struct gmem_file *f = file->private_data;
281331 struct kvm_memory_slot *slot;
282− struct kvm *kvm = gmem->kvm;
332+ struct kvm *kvm = f->kvm;
283333 unsigned long index;
284334
285335 /*
@@ -299,7 +349,7 @@ static int kvm_gmem_release(struct inode *inode, struct file *file) · 13 unmodified lines
299349
300350 filemap_invalidate_lock(inode->i_mapping);
301351
302− xa_for_each(&gmem->bindings, index, slot)
352+ xa_for_each(&f->bindings, index, slot)
303353 WRITE_ONCE(slot->gmem.file, NULL);
304354
305355 /*
@@ -307,18 +357,18 @@ static int kvm_gmem_release(struct inode *inode, struct file *file) · 1 unmodified line
307357 * Zap all SPTEs pointed at by this file. Do not free the backing
308358 * memory, as its lifetime is associated with the inode, not the file.
309359 */
310− __kvm_gmem_invalidate_begin(gmem, 0, -1ul,
360+ __kvm_gmem_invalidate_begin(f, 0, -1ul,
311361 kvm_gmem_get_invalidate_filter(inode));
312− __kvm_gmem_invalidate_end(gmem, 0, -1ul);
362+ __kvm_gmem_invalidate_end(f, 0, -1ul);
313363
314− list_del(&gmem->entry);
364+ list_del(&f->entry);
315365
316366 filemap_invalidate_unlock(inode->i_mapping);
317367
318368 mutex_unlock(&kvm->slots_lock);
319369
320− xa_destroy(&gmem->bindings);
321− kfree(gmem);
370+ xa_destroy(&f->bindings);
371+ kfree(f);
322372
323373 kvm_put_kvm(kvm);
324374
@@ -335,16 +385,12 @@ static inline struct file *kvm_gmem_get_file(struct kvm_memory_slot *slot) · 10 unmodified lines
335385 return get_file_active(&slot->gmem.file);
336386 }
337387
338−static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
339−{
340− return gfn - slot->base_gfn + slot->gmem.pgoff;
341−}
388+DEFINE_CLASS(gmem_get_file, struct file *, if (_T) fput(_T),
389+ kvm_gmem_get_file(slot), struct kvm_memory_slot *slot);
342390
343391 static bool kvm_gmem_supports_mmap(struct inode *inode)
344392 {
345− const u64 flags = (u64)inode->i_private;
346−
347− return flags & GUEST_MEMFD_FLAG_MMAP;
393+ return GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_MMAP;
348394 }
349395
350396 static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf)
@@ -356,17 +402,15 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) · 5 unmodified lines
356402 if (((loff_t)vmf->pgoff << PAGE_SHIFT) >= i_size_read(inode))
357403 return VM_FAULT_SIGBUS;
358404
359− if (!((u64)inode->i_private & GUEST_MEMFD_FLAG_INIT_SHARED))
405+ if (!(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED))
360406 return VM_FAULT_SIGBUS;
361407
362408 folio = kvm_gmem_get_folio(inode, vmf->pgoff);
363409 if (IS_ERR(folio)) {
364− int err = PTR_ERR(folio);
365−
366− if (err == -EAGAIN)
410+ if (PTR_ERR(folio) == -EAGAIN)
367411 return VM_FAULT_RETRY;
368412
369− return vmf_error(err);
413+ return vmf_error(PTR_ERR(folio));
370414 }
371415
372416 if (WARN_ON_ONCE(folio_test_large(folio))) {
@@ -390,8 +434,40 @@ out_folio: · 17 unmodified lines
390434 return ret;
391435 }
392436
437+#ifdef CONFIG_NUMA
438+static int kvm_gmem_set_policy(struct vm_area_struct *vma, struct mempolicy *mpol)
439+{
440+ struct inode *inode = file_inode(vma->vm_file);
441+
442+ return mpol_set_shared_policy(&GMEM_I(inode)->policy, vma, mpol);
443+}
444+
445+static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma,
446+ unsigned long addr, pgoff_t *pgoff)
447+{
448+ struct inode *inode = file_inode(vma->vm_file);
449+
450+ *pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT);
451+
452+ /*
453+ * Return the memory policy for this index, or NULL if none is set.
454+ *
455+ * Returning NULL, e.g. instead of the current task's memory policy, is
456+ * important for the .get_policy kernel ABI: it indicates that no
457+ * explicit policy has been set via mbind() for this memory. The caller
458+ * can then replace NULL with the default memory policy instead of the
459+ * current task's memory policy.
460+ */
461+ return mpol_shared_policy_lookup(&GMEM_I(inode)->policy, *pgoff);
462+}
463+#endif /* CONFIG_NUMA */
464+
393465 static const struct vm_operations_struct kvm_gmem_vm_ops = {
394− .fault = kvm_gmem_fault_user_mapping,
466+ .fault = kvm_gmem_fault_user_mapping,
467+#ifdef CONFIG_NUMA
468+ .get_policy = kvm_gmem_get_policy,
469+ .set_policy = kvm_gmem_set_policy,
470+#endif
395471 };
396472
397473 static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
@@ -416,11 +492,6 @@ static struct file_operations kvm_gmem_fops = { · 18 unmodified lines
416492 .fallocate = kvm_gmem_fallocate,
417493 };
418494
419−void kvm_gmem_init(struct module *module)
420−{
421− kvm_gmem_fops.owner = module;
422−}
423−
424495 static int kvm_gmem_migrate_folio(struct address_space *mapping,
425496 struct folio *dst, struct folio *src,
426497 enum migrate_mode mode)
@@ -492,8 +563,8 @@ bool __weak kvm_arch_supports_gmem_init_shared(struct kvm *kvm) · 65 unmodified lines
492563
493564 static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
494565 {
495− const char *anon_name = "[kvm-gmem]";
496− struct kvm_gmem *gmem;
566+ static const char *name = "[kvm-gmem]";
567+ struct gmem_file *f;
497568 struct inode *inode;
498569 struct file *file;
499570 int fd, err;
@@ -502,25 +573,24 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) · 2 unmodified lines
502573 if (fd < 0)
503574 return fd;
504575
505− gmem = kzalloc(sizeof(*gmem), GFP_KERNEL);
506− if (!gmem) {
576+ f = kzalloc(sizeof(*f), GFP_KERNEL);
577+ if (!f) {
507578 err = -ENOMEM;
508579 goto err_fd;
509580 }
510581
511− file = anon_inode_create_getfile(anon_name, &kvm_gmem_fops, gmem,
512− O_RDWR, NULL);
513− if (IS_ERR(file)) {
514− err = PTR_ERR(file);
582+ /* __fput() will take care of fops_put(). */
583+ if (!fops_get(&kvm_gmem_fops)) {
584+ err = -ENOENT;
515585 goto err_gmem;
516586 }
517587
518− file->f_flags |= O_LARGEFILE;
519−
520− inode = file->f_inode;
521− WARN_ON(file->f_mapping != inode->i_mapping);
588+ inode = anon_inode_make_secure_inode(kvm_gmem_mnt->mnt_sb, name, NULL);
589+ if (IS_ERR(inode)) {
590+ err = PTR_ERR(inode);
591+ goto err_fops;
592+ }
522593
523− inode->i_private = (void *)(unsigned long)flags;
524594 inode->i_op = &kvm_gmem_iops;
525595 inode->i_mapping->a_ops = &kvm_gmem_aops;
526596 inode->i_mode |= S_IFREG;
@@ -530,16 +600,31 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags) · 3 unmodified lines
530600 /* Unmovable mappings are supposed to be marked unevictable as well. */
531601 WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));
532602
603+ GMEM_I(inode)->flags = flags;
604+
605+ file = alloc_file_pseudo(inode, kvm_gmem_mnt, name, O_RDWR, &kvm_gmem_fops);
606+ if (IS_ERR(file)) {
607+ err = PTR_ERR(file);
608+ goto err_inode;
609+ }
610+
611+ file->f_flags |= O_LARGEFILE;
612+ file->private_data = f;
613+
533614 kvm_get_kvm(kvm);
534− gmem->kvm = kvm;
535− xa_init(&gmem->bindings);
536− list_add(&gmem->entry, &inode->i_mapping->i_private_list);
615+ f->kvm = kvm;
616+ xa_init(&f->bindings);
617+ list_add(&f->entry, &inode->i_mapping->i_private_list);
537618
538619 fd_install(fd, file);
539620 return fd;
540621
622+err_inode:
623+ iput(inode);
624+err_fops:
625+ fops_put(&kvm_gmem_fops);
541626 err_gmem:
542− kfree(gmem);
627+ kfree(f);
543628 err_fd:
544629 put_unused_fd(fd);
545630 return err;
@@ -564,7 +649,7 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, · 18 unmodified lines
564649 {
565650 loff_t size = slot->npages << PAGE_SHIFT;
566651 unsigned long start, end;
567− struct kvm_gmem *gmem;
652+ struct gmem_file *f;
568653 struct inode *inode;
569654 struct file *file;
570655 int r = -EINVAL;
@@ -578,8 +663,8 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, · 7 unmodified lines
578663 if (file->f_op != &kvm_gmem_fops)
579664 goto err;
580665
581− gmem = file->private_data;
582− if (gmem->kvm != kvm)
666+ f = file->private_data;
667+ if (f->kvm != kvm)
583668 goto err;
584669
585670 inode = file_inode(file);
@@ -593,8 +678,8 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, · 7 unmodified lines
593678 start = offset >> PAGE_SHIFT;
594679 end = start + slot->npages;
595680
596− if (!xa_empty(&gmem->bindings) &&
597− xa_find(&gmem->bindings, &start, end - 1, XA_PRESENT)) {
681+ if (!xa_empty(&f->bindings) &&
682+ xa_find(&f->bindings, &start, end - 1, XA_PRESENT)) {
598683 filemap_invalidate_unlock(inode->i_mapping);
599684 goto err;
600685 }
@@ -609,7 +694,7 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, · 8 unmodified lines
609694 if (kvm_gmem_supports_mmap(inode))
610695 slot->flags |= KVM_MEMSLOT_GMEM_ONLY;
611696
612− xa_store_range(&gmem->bindings, start, end - 1, slot, GFP_KERNEL);
697+ xa_store_range(&f->bindings, start, end - 1, slot, GFP_KERNEL);
613698 filemap_invalidate_unlock(inode->i_mapping);
614699
615700 /*
@@ -623,12 +708,12 @@ err: · 7 unmodified lines
623708 return r;
624709 }
625710
626−static void __kvm_gmem_unbind(struct kvm_memory_slot *slot, struct kvm_gmem *gmem)
711+static void __kvm_gmem_unbind(struct kvm_memory_slot *slot, struct gmem_file *f)
627712 {
628713 unsigned long start = slot->gmem.pgoff;
629714 unsigned long end = start + slot->npages;
630715
631− xa_store_range(&gmem->bindings, start, end - 1, NULL, GFP_KERNEL);
716+ xa_store_range(&f->bindings, start, end - 1, NULL, GFP_KERNEL);
632717
633718 /*
634719 * synchronize_srcu(&kvm->srcu) ensured that kvm_gmem_get_pfn()
@@ -639,8 +724,6 @@ static void __kvm_gmem_unbind(struct kvm_memory_slot *slot, struct kvm_gmem *gme · 4 unmodified lines
639724
640725 void kvm_gmem_unbind(struct kvm_memory_slot *slot)
641726 {
642− struct file *file;
643−
644727 /*
645728 * Nothing to do if the underlying file was _already_ closed, as
646729 * kvm_gmem_release() invalidates and nullifies all bindings.
@@ -648,7 +731,7 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot) · 1 unmodified line
648731 if (!slot->gmem.file)
649732 return;
650733
651− file = kvm_gmem_get_file(slot);
734+ CLASS(gmem_get_file, file)(slot);
652735
653736 /*
654737 * However, if the file is _being_ closed, then the bindings need to be
@@ -668,8 +751,6 @@ void kvm_gmem_unbind(struct kvm_memory_slot *slot) · 13 unmodified lines
668751 filemap_invalidate_lock(file->f_mapping);
669752 __kvm_gmem_unbind(slot, file->private_data);
670753 filemap_invalidate_unlock(file->f_mapping);
671−
672− fput(file);
673754 }
674755
675756 /* Returns a locked folio on success. */
@@ -678,18 +759,17 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file, · 2 unmodified lines
678759 pgoff_t index, kvm_pfn_t *pfn,
679760 bool *is_prepared, int *max_order)
680761 {
681− struct file *gmem_file = READ_ONCE(slot->gmem.file);
682− struct kvm_gmem *gmem = file->private_data;
762+ struct file *slot_file = READ_ONCE(slot->gmem.file);
763+ struct gmem_file *f = file->private_data;
683764 struct folio *folio;
684765
685− if (file != gmem_file) {
686− WARN_ON_ONCE(gmem_file);
766+ if (file != slot_file) {
767+ WARN_ON_ONCE(slot_file);
687768 return ERR_PTR(-EFAULT);
688769 }
689770
690− gmem = file->private_data;
691− if (xa_load(&gmem->bindings, index) != slot) {
692− WARN_ON_ONCE(xa_load(&gmem->bindings, index));
771+ if (xa_load(&f->bindings, index) != slot) {
772+ WARN_ON_ONCE(xa_load(&f->bindings, index));
693773 return ERR_PTR(-EIO);
694774 }
695775
@@ -716,19 +796,17 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, · 20 unmodified lines
716796 int *max_order)
717797 {
718798 pgoff_t index = kvm_gmem_get_index(slot, gfn);
719− struct file *file = kvm_gmem_get_file(slot);
720799 struct folio *folio;
721800 bool is_prepared = false;
722801 int r = 0;
723802
803+ CLASS(gmem_get_file, file)(slot);
724804 if (!file)
725805 return -EFAULT;
726806
727807 folio = __kvm_gmem_get_pfn(file, slot, index, pfn, &is_prepared, max_order);
728− if (IS_ERR(folio)) {
729− r = PTR_ERR(folio);
730− goto out;
731− }
808+ if (IS_ERR(folio))
809+ return PTR_ERR(folio);
732810
733811 if (!is_prepared)
734812 r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
@@ -740,8 +818,6 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, · 5 unmodified lines
740818 else
741819 folio_put(folio);
742820
743−out:
744− fput(file);
745821 return r;
746822 }
747823 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_get_pfn);
@@ -750,7 +826,6 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_get_pfn); · 2 unmodified lines
750826 long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long npages,
751827 kvm_gmem_populate_cb post_populate, void *opaque)
752828 {
753− struct file *file;
754829 struct kvm_memory_slot *slot;
755830 void __user *p;
756831
@@ -766,7 +841,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long · 9 unmodified lines
766841 if (!kvm_slot_has_gmem(slot))
767842 return -EINVAL;
768843
769− file = kvm_gmem_get_file(slot);
844+ CLASS(gmem_get_file, file)(slot);
770845 if (!file)
771846 return -EFAULT;
772847
@@ -824,8 +899,118 @@ put_folio_and_exit: · 51 unmodified lines
824899
825900 filemap_invalidate_unlock(file->f_mapping);
826901
827− fput(file);
828902 return ret && !i ? ret : i;
829903 }
830904 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_populate);
831905 #endif
906+
907+static struct kmem_cache *kvm_gmem_inode_cachep;
908+
909+static void kvm_gmem_init_inode_once(void *__gi)
910+{
911+ struct gmem_inode *gi = __gi;
912+
913+ /*
914+ * Note! Don't initialize the inode with anything specific to the
915+ * guest_memfd instance, or that might be specific to how the inode is
916+ * used (from the VFS-layer's perspective). This hook is called only
917+ * during the initial slab allocation, i.e. only fields/state that are
918+ * idempotent across _all_ use of the inode _object_ can be initialized
919+ * at this time!
920+ */
921+ inode_init_once(&gi->vfs_inode);
922+}
923+
924+static struct inode *kvm_gmem_alloc_inode(struct super_block *sb)
925+{
926+ struct gmem_inode *gi;
927+
928+ gi = alloc_inode_sb(sb, kvm_gmem_inode_cachep, GFP_KERNEL);
929+ if (!gi)
930+ return NULL;
931+
932+ mpol_shared_policy_init(&gi->policy, NULL);
933+
934+ gi->flags = 0;
935+ return &gi->vfs_inode;
936+}
937+
938+static void kvm_gmem_destroy_inode(struct inode *inode)
939+{
940+ mpol_free_shared_policy(&GMEM_I(inode)->policy);
941+}
942+
943+static void kvm_gmem_free_inode(struct inode *inode)
944+{
945+ kmem_cache_free(kvm_gmem_inode_cachep, GMEM_I(inode));
946+}
947+
948+static const struct super_operations kvm_gmem_super_operations = {
949+ .statfs = simple_statfs,
950+ .alloc_inode = kvm_gmem_alloc_inode,
951+ .destroy_inode = kvm_gmem_destroy_inode,
952+ .free_inode = kvm_gmem_free_inode,
953+};
954+
955+static int kvm_gmem_init_fs_context(struct fs_context *fc)
956+{
957+ struct pseudo_fs_context *ctx;
958+
959+ if (!init_pseudo(fc, GUEST_MEMFD_MAGIC))
960+ return -ENOMEM;
961+
962+ fc->s_iflags |= SB_I_NOEXEC;
963+ fc->s_iflags |= SB_I_NODEV;
964+ ctx = fc->fs_private;
965+ ctx->ops = &kvm_gmem_super_operations;
966+
967+ return 0;
968+}
969+
970+static struct file_system_type kvm_gmem_fs = {
971+ .name = "guest_memfd",
972+ .init_fs_context = kvm_gmem_init_fs_context,
973+ .kill_sb = kill_anon_super,
974+};
975+
976+static int kvm_gmem_init_mount(void)
977+{
978+ kvm_gmem_mnt = kern_mount(&kvm_gmem_fs);
979+
980+ if (IS_ERR(kvm_gmem_mnt))
981+ return PTR_ERR(kvm_gmem_mnt);
982+
983+ kvm_gmem_mnt->mnt_flags |= MNT_NOEXEC;
984+ return 0;
985+}
986+
987+int kvm_gmem_init(struct module *module)
988+{
989+ struct kmem_cache_args args = {
990+ .align = 0,
991+ .ctor = kvm_gmem_init_inode_once,
992+ };
993+ int ret;
994+
995+ kvm_gmem_fops.owner = module;
996+ kvm_gmem_inode_cachep = kmem_cache_create("kvm_gmem_inode_cache",
997+ sizeof(struct gmem_inode),
998+ &args, SLAB_ACCOUNT);
999+ if (!kvm_gmem_inode_cachep)
1000+ return -ENOMEM;
1001+
1002+ ret = kvm_gmem_init_mount();
1003+ if (ret) {
1004+ kmem_cache_destroy(kvm_gmem_inode_cachep);
1005+ return ret;
1006+ }
1007+ return 0;
1008+}
1009+
1010+void kvm_gmem_exit(void)
1011+{
1012+ kern_unmount(kvm_gmem_mnt);
1013+ kvm_gmem_mnt = NULL;
1014+ rcu_barrier();
1015+ kmem_cache_destroy(kvm_gmem_inode_cachep);
1016+}
virt/kvm/kvm_main.c+10−5
@@ -4027,7 +4027,7 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode) · 4026 unmodified lines
40274027
40284028 yielded = kvm_vcpu_yield_to(vcpu);
40294029 if (yielded > 0) {
4030− WRITE_ONCE(kvm->last_boosted_vcpu, i);
4030+ WRITE_ONCE(kvm->last_boosted_vcpu, idx);
40314031 break;
40324032 } else if (yielded < 0 && !--try) {
40334033 break;
@@ -4435,10 +4435,10 @@ static long kvm_vcpu_ioctl(struct file *filp, · 401 unmodified lines
44354435 return r;
44364436
44374437 /*
4438− * Some architectures have vcpu ioctls that are asynchronous to vcpu
4439− * execution; mutex_lock() would break them.
4438+ * Let arch code handle select vCPU ioctls without holding vcpu->mutex,
4439+ * e.g. to support ioctls that can run asynchronous to vCPU execution.
44404440 */
4441− r = kvm_arch_vcpu_async_ioctl(filp, ioctl, arg);
4441+ r = kvm_arch_vcpu_unlocked_ioctl(filp, ioctl, arg);
44424442 if (r != -ENOIOCTLCMD)
44434443 return r;
44444444
@@ -6524,7 +6524,9 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module) · 2079 unmodified lines
65246524 if (WARN_ON_ONCE(r))
65256525 goto err_vfio;
65266526
6527− kvm_gmem_init(module);
6527+ r = kvm_gmem_init(module);
6528+ if (r)
6529+ goto err_gmem;
65286530
65296531 r = kvm_init_virtualization();
65306532 if (r)
@@ -6545,6 +6547,8 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module) · 14 unmodified lines
65456547 err_register:
65466548 kvm_uninit_virtualization();
65476549 err_virt:
6550+ kvm_gmem_exit();
6551+err_gmem:
65486552 kvm_vfio_ops_exit();
65496553 err_vfio:
65506554 kvm_async_pf_deinit();
@@ -6576,6 +6580,7 @@ void kvm_exit(void) · 25 unmodified lines
65766580 for_each_possible_cpu(cpu)
65776581 free_cpumask_var(per_cpu(cpu_kick_mask, cpu));
65786582 kmem_cache_destroy(kvm_vcpu_cache);
6583+ kvm_gmem_exit();
65796584 kvm_vfio_ops_exit();
65806585 kvm_async_pf_deinit();
65816586 kvm_irqfd_exit();
virt/kvm/kvm_mm.h+5−4
@@ -68,17 +68,18 @@ static inline void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, · 67 unmodified lines
6868 #endif /* HAVE_KVM_PFNCACHE */
6969
7070 #ifdef CONFIG_KVM_GUEST_MEMFD
71−void kvm_gmem_init(struct module *module);
71+int kvm_gmem_init(struct module *module);
72+void kvm_gmem_exit(void);
7273 int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args);
7374 int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
7475 unsigned int fd, loff_t offset);
7576 void kvm_gmem_unbind(struct kvm_memory_slot *slot);
7677 #else
77−static inline void kvm_gmem_init(struct module *module)
78+static inline int kvm_gmem_init(struct module *module)
7879 {
79−
80+ return 0;
8081 }
81−
82+static inline void kvm_gmem_exit(void) {};
8283 static inline int kvm_gmem_bind(struct kvm *kvm,
8384 struct kvm_memory_slot *slot,
8485 unsigned int fd, loff_t offset)