History

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm

Pull kvm updates from Paolo Bonzini: "ARM: - Provide a virtual cache topology to the guest to avoid inconsistencies with migration on heterogenous systems. Non secure software has no practical need to traverse the caches by set/way in the first place - Add support for taking stage-2 access faults in parallel. This was an accidental omission in the original parallel faults implementation, but should provide a marginal improvement to machines w/o FEAT_HAFDBS (such as hardware from the fruit company) - A preamble to adding support for nested virtualization to KVM, including vEL2 register state, rudimentary nested exception handling and masking unsupported features for nested guests - Fixes to the PSCI relay that avoid an unexpected host SVE trap when resuming a CPU when running pKVM - VGIC maintenance interrupt support for the AIC - Improvements to the arch timer emulation, primarily aimed at reducing the trap overhead of running nested - Add CONFIG_USERFAULTFD to the KVM selftests config fragment in the interest of CI systems - Avoid VM-wide stop-the-world operations when a vCPU accesses its own redistributor - Serialize when toggling CPACR_EL1.SMEN to avoid unexpected exceptions in the host - Aesthetic and comment/kerneldoc fixes - Drop the vestiges of the old Columbia mailing list and add [Oliver] as co-maintainer RISC-V: - Fix wrong usage of PGDIR_SIZE instead of PUD_SIZE - Correctly place the guest in S-mode after redirecting a trap to the guest - Redirect illegal instruction traps to guest - SBI PMU support for guest s390: - Sort out confusion between virtual and physical addresses, which currently are the same on s390 - A new ioctl that performs cmpxchg on guest memory - A few fixes x86: - Change tdp_mmu to a read-only parameter - Separate TDP and shadow MMU page fault paths - Enable Hyper-V invariant TSC control - Fix a variety of APICv and AVIC bugs, some of them real-world, some of them affecting architecurally legal but unlikely to happen in practice - Mark APIC timer as expired if its in one-shot mode and the count underflows while the vCPU task was being migrated - Advertise support for Intel's new fast REP string features - Fix a double-shootdown issue in the emergency reboot code - Ensure GIF=1 and disable SVM during an emergency reboot, i.e. give SVM similar treatment to VMX - Update Xen's TSC info CPUID sub-leaves as appropriate - Add support for Hyper-V's extended hypercalls, where "support" at this point is just forwarding the hypercalls to userspace - Clean up the kvm->lock vs. kvm->srcu sequences when updating the PMU and MSR filters - One-off fixes and cleanups - Fix and cleanup the range-based TLB flushing code, used when KVM is running on Hyper-V - Add support for filtering PMU events using a mask. If userspace wants to restrict heavily what events the guest can use, it can now do so without needing an absurd number of filter entries - Clean up KVM's handling of "PMU MSRs to save", especially when vPMU support is disabled - Add PEBS support for Intel Sapphire Rapids - Fix a mostly benign overflow bug in SEV's send|receive_update_data() - Move several SVM-specific flags into vcpu_svm x86 Intel: - Handle NMI VM-Exits before leaving the noinstr region - A few trivial cleanups in the VM-Enter flows - Stop enabling VMFUNC for L1 purely to document that KVM doesn't support EPTP switching (or any other VM function) for L1 - Fix a crash when using eVMCS's enlighted MSR bitmaps Generic: - Clean up the hardware enable and initialization flow, which was scattered around multiple arch-specific hooks. Instead, just let the arch code call into generic code. Both x86 and ARM should benefit from not having to fight common KVM code's notion of how to do initialization - Account allocations in generic kvm_arch_alloc_vm() - Fix a memory leak if coalesced MMIO unregistration fails selftests: - On x86, cache the CPU vendor (AMD vs. Intel) and use the info to emit the correct hypercall instruction instead of relying on KVM to patch in VMMCALL - Use TAP interface for kvm_binary_stats_test and tsc_msrs_test" * tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (325 commits) KVM: SVM: hyper-v: placate modpost section mismatch error KVM: x86/mmu: Make tdp_mmu_allowed static KVM: arm64: nv: Use reg_to_encoding() to get sysreg ID KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 changes KVM: arm64: nv: Filter out unsupported features from ID regs KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2 KVM: arm64: nv: Allow a sysreg to be hidden from userspace only KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor KVM: arm64: nv: Add accessors for SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2 KVM: arm64: nv: Handle SMCs taken from virtual EL2 KVM: arm64: nv: Handle trapped ERET from virtual EL2 KVM: arm64: nv: Inject HVC exceptions to the virtual EL2 KVM: arm64: nv: Support virtual EL2 exceptions KVM: arm64: nv: Handle HCR_EL2.NV system register traps KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state KVM: arm64: nv: Add EL2 system registers to vcpu context KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set KVM: arm64: nv: Introduce nested virtualization VCPU feature KVM: arm64: Use the S2 MMU context to iterate over S2 table ...

Linus Torvalds committed 3 years ago

Commit 49d57592 parents 01687e7 45dd9bc200 files · +7274 −2928Browse files

Unified · Split

200 files
  1. Documentation/admin-guide/kernel-parameters.txt +6 −1
  2. Documentation/virt/kvm/api.rst +108 −16
  3. Documentation/virt/kvm/devices/vm.rst +4 −0
  4. Documentation/virt/kvm/locking.rst +16 −9
  5. Documentation/virt/kvm/x86/errata.rst +11 −0
  6. MAINTAINERS +1 −2
  7. arch/arm64/include/asm/cache.h +9 −0
  8. arch/arm64/include/asm/el2_setup.h +99 −0
  9. arch/arm64/include/asm/esr.h +4 −0
  10. arch/arm64/include/asm/kvm_arm.h +20 −3
  11. arch/arm64/include/asm/kvm_emulate.h +66 −4
  12. arch/arm64/include/asm/kvm_host.h +53 −14
  13. arch/arm64/include/asm/kvm_hyp.h +1 −0
  14. arch/arm64/include/asm/kvm_mmu.h +12 −3
  15. arch/arm64/include/asm/kvm_nested.h +20 −0
  16. arch/arm64/include/asm/kvm_pgtable.h +8 −0
  17. arch/arm64/include/asm/sysreg.h +37 −2
  18. arch/arm64/include/uapi/asm/kvm.h +1 −0
  19. arch/arm64/kernel/cacheinfo.c +0 −5
  20. arch/arm64/kernel/cpufeature.c +25 −0
  21. arch/arm64/kernel/hyp-stub.S +2 −84
  22. arch/arm64/kvm/Kconfig +1 −0
  23. arch/arm64/kvm/Makefile +1 −1
  24. arch/arm64/kvm/arch_timer.c +60 −46
  25. arch/arm64/kvm/arm.c +60 −49
  26. arch/arm64/kvm/emulate-nested.c +203 −0
  27. arch/arm64/kvm/fpsimd.c +1 −0
  28. arch/arm64/kvm/guest.c +6 −0
  29. arch/arm64/kvm/handle_exit.c +45 −2
  30. arch/arm64/kvm/hyp/exception.c +36 −12
  31. arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +18 −3
  32. arch/arm64/kvm/hyp/nvhe/hyp-init.S +1 −0
  33. arch/arm64/kvm/hyp/nvhe/sys_regs.c +1 −0
  34. arch/arm64/kvm/hyp/pgtable.c +37 −6
  35. arch/arm64/kvm/hyp/vhe/switch.c +25 −1
  36. arch/arm64/kvm/hypercalls.c +1 −1
  37. arch/arm64/kvm/inject_fault.c +55 −6
  38. arch/arm64/kvm/mmu.c +23 −23
  39. arch/arm64/kvm/nested.c +161 −0
  40. arch/arm64/kvm/pvtime.c +4 −4
  41. arch/arm64/kvm/reset.c +21 −4
  42. arch/arm64/kvm/sys_regs.c +350 −109
  43. arch/arm64/kvm/sys_regs.h +12 −2
  44. arch/arm64/kvm/trace_arm.h +59 −0
  45. arch/arm64/kvm/vgic/vgic-init.c +3 −18
  46. arch/arm64/kvm/vgic/vgic-mmio.c +8 −5
  47. arch/arm64/kvm/vgic/vgic-v3.c +5 −4
  48. arch/arm64/kvm/vmid.c +3 −3
  49. arch/arm64/tools/cpucaps +1 −0
  50. arch/arm64/tools/gen-sysreg.awk +19 −1
  51. arch/arm64/tools/sysreg +17 −0
  52. arch/mips/include/asm/kvm_host.h +1 −2
  53. arch/mips/kvm/Kconfig +1 −0
  54. arch/mips/kvm/Makefile +1 −1
  55. arch/mips/kvm/callback.c +0 −14
  56. arch/mips/kvm/mips.c +7 −27
  57. arch/mips/kvm/vz.c +4 −3
  58. arch/powerpc/include/asm/kvm_host.h +0 −3
  59. arch/powerpc/include/asm/kvm_ppc.h +0 −1
  60. arch/powerpc/kvm/book3s.c +1 −11
  61. arch/powerpc/kvm/booke.c +1 −1
  62. arch/powerpc/kvm/e500.c +3 −3
  63. arch/powerpc/kvm/e500mc.c +6 −2
  64. arch/powerpc/kvm/powerpc.c +0 −20
  65. arch/riscv/include/asm/kvm_host.h +7 −4
  66. arch/riscv/include/asm/kvm_vcpu_pmu.h +107 −0
  67. arch/riscv/include/asm/kvm_vcpu_sbi.h +11 −2
  68. arch/riscv/include/asm/sbi.h +5 −2
  69. arch/riscv/kvm/Kconfig +1 −0
  70. arch/riscv/kvm/Makefile +1 −0
  71. arch/riscv/kvm/main.c +4 −22
  72. arch/riscv/kvm/mmu.c +10 −10
  73. arch/riscv/kvm/tlb.c +4 −0
  74. arch/riscv/kvm/vcpu.c +7 −0
  75. arch/riscv/kvm/vcpu_exit.c +9 −0
  76. arch/riscv/kvm/vcpu_insn.c +3 −1
  77. arch/riscv/kvm/vcpu_pmu.c +633 −0
  78. arch/riscv/kvm/vcpu_sbi.c +35 −37
  79. arch/riscv/kvm/vcpu_sbi_base.c +14 −13
  80. arch/riscv/kvm/vcpu_sbi_hsm.c +14 −14
  81. arch/riscv/kvm/vcpu_sbi_pmu.c +86 −0
  82. arch/riscv/kvm/vcpu_sbi_replace.c +27 −23
  83. arch/riscv/kvm/vcpu_sbi_v01.c +7 −10
  84. arch/riscv/kvm/vmid.c +2 −2
  85. arch/s390/include/asm/kvm_host.h +0 −1
  86. arch/s390/kvm/gaccess.c +109 −0
  87. arch/s390/kvm/gaccess.h +3 −0
  88. arch/s390/kvm/interrupt.c +8 −5
  89. arch/s390/kvm/kvm-s390.c +208 −140
  90. arch/s390/kvm/kvm-s390.h +1 −1
  91. arch/s390/kvm/pci.c +1 −1
  92. arch/s390/kvm/pci.h +1 −1
  93. arch/x86/events/intel/core.c +1 −0
  94. arch/x86/events/intel/ds.c +3 −1
  95. arch/x86/include/asm/cpufeatures.h +3 −0
  96. arch/x86/include/asm/hyperv-tlfs.h +3 −0
  97. arch/x86/include/asm/idtentry.h +6 −10
  98. arch/x86/include/asm/kvm-x86-ops.h +1 −1
  99. arch/x86/include/asm/kvm_host.h +71 −38
  100. arch/x86/include/asm/reboot.h +2 −0
  101. arch/x86/include/asm/virtext.h +15 −1
  102. arch/x86/include/asm/xen/hypervisor.h +3 −1
  103. arch/x86/include/uapi/asm/kvm.h +32 −2
  104. arch/x86/kernel/cpu/mshyperv.c +1 −1
  105. arch/x86/kernel/crash.c +1 −16
  106. arch/x86/kernel/nmi.c +4 −4
  107. arch/x86/kernel/reboot.c +64 −24
  108. arch/x86/kernel/smp.c +3 −3
  109. arch/x86/kvm/Kconfig +1 −0
  110. arch/x86/kvm/cpuid.c +28 −15
  111. arch/x86/kvm/debugfs.c +2 −0
  112. arch/x86/kvm/emulate.c +9 −9
  113. arch/x86/kvm/hyperv.c +63 −22
  114. arch/x86/kvm/hyperv.h +27 −0
  115. arch/x86/kvm/i8254.c +2 −2
  116. arch/x86/kvm/i8259.c +3 −1
  117. arch/x86/kvm/ioapic.c +1 −0
  118. arch/x86/kvm/irq.c +1 −0
  119. arch/x86/kvm/irq_comm.c +4 −3
  120. arch/x86/kvm/kvm_cache_regs.h +12 −0
  121. arch/x86/kvm/kvm_emulate.h +2 −5
  122. arch/x86/kvm/kvm_onhyperv.c +1 −0
  123. arch/x86/kvm/lapic.c +296 −108
  124. arch/x86/kvm/lapic.h +4 −0
  125. arch/x86/kvm/mmu.h +3 −3
  126. arch/x86/kvm/mmu/mmu.c +203 −117
  127. arch/x86/kvm/mmu/mmu_internal.h +21 −1
  128. arch/x86/kvm/mmu/page_track.c +1 −0
  129. arch/x86/kvm/mmu/paging_tmpl.h +7 −18
  130. arch/x86/kvm/mmu/spte.c +5 −5
  131. arch/x86/kvm/mmu/spte.h +10 −10
  132. arch/x86/kvm/mmu/tdp_iter.c +4 −8
  133. arch/x86/kvm/mmu/tdp_mmu.c +4 −16
  134. arch/x86/kvm/mmu/tdp_mmu.h +3 −22
  135. arch/x86/kvm/mtrr.c +1 −0
  136. arch/x86/kvm/pmu.c +238 −52
  137. arch/x86/kvm/pmu.h +6 −7
  138. arch/x86/kvm/reverse_cpuid.h +7 −0
  139. arch/x86/kvm/smm.c +1 −2
  140. arch/x86/kvm/svm/avic.c +167 −207
  141. arch/x86/kvm/svm/nested.c +2 −3
  142. arch/x86/kvm/svm/pmu.c +4 −0
  143. arch/x86/kvm/svm/sev.c +4 −3
  144. arch/x86/kvm/svm/svm.c +75 −56
  145. arch/x86/kvm/svm/svm.h +43 −15
  146. arch/x86/kvm/svm/svm_onhyperv.c +1 −0
  147. arch/x86/kvm/svm/svm_onhyperv.h +4 −4
  148. arch/x86/kvm/vmx/capabilities.h +2 −2
  149. arch/x86/kvm/vmx/hyperv.c +68 −19
  150. arch/x86/kvm/vmx/hyperv.h +89 −39
  151. arch/x86/kvm/vmx/nested.c +10 −14
  152. arch/x86/kvm/vmx/pmu_intel.c +17 −11
  153. arch/x86/kvm/vmx/posted_intr.c +2 −0
  154. arch/x86/kvm/vmx/sgx.c +3 −2
  155. arch/x86/kvm/vmx/vmcs.h +2 −2
  156. arch/x86/kvm/vmx/vmcs12.c +1 −0
  157. arch/x86/kvm/vmx/vmenter.S +44 −36
  158. arch/x86/kvm/vmx/vmx.c +328 −296
  159. arch/x86/kvm/vmx/vmx.h +8 −10
  160. arch/x86/kvm/vmx/vmx_ops.h +4 −2
  161. arch/x86/kvm/x86.c +310 −247
  162. arch/x86/kvm/x86.h +15 −3
  163. arch/x86/kvm/xen.c +27 −0
  164. arch/x86/kvm/xen.h +7 −0
  165. drivers/irqchip/irq-apple-aic.c +40 −13
  166. drivers/perf/riscv_pmu_sbi.c +57 −7
  167. drivers/s390/virtio/virtio_ccw.c +24 −22
  168. include/kvm/arm_arch_timer.h +5 −1
  169. include/kvm/arm_vgic.h +4 −0
  170. include/linux/cpuhotplug.h +1 −4
  171. include/linux/kvm_host.h +8 −11
  172. include/linux/kvm_types.h +1 −1
  173. include/linux/perf/riscv_pmu.h +5 −0
  174. include/uapi/linux/kvm.h +9 −0
  175. tools/testing/selftests/kvm/Makefile +4 −0
  176. tools/testing/selftests/kvm/aarch64/page_fault_test.c +0 −2
  177. tools/testing/selftests/kvm/config +1 −0
  178. tools/testing/selftests/kvm/include/x86_64/hyperv.h +102 −47
  179. tools/testing/selftests/kvm/include/x86_64/processor.h +27 −4
  180. tools/testing/selftests/kvm/kvm_binary_stats_test.c +10 −11
  181. tools/testing/selftests/kvm/lib/elf.c +2 −0
  182. tools/testing/selftests/kvm/lib/guest_modes.c +1 −1
  183. tools/testing/selftests/kvm/lib/kvm_util.c +1 −3
  184. tools/testing/selftests/kvm/lib/x86_64/processor.c +20 −26
  185. tools/testing/selftests/kvm/memslot_perf_test.c +1 −4
  186. tools/testing/selftests/kvm/s390x/memop.c +524 −148
  187. tools/testing/selftests/kvm/x86_64/exit_on_emulation_failure_test.c +0 −3
  188. tools/testing/selftests/kvm/x86_64/fix_hypercall_test.c +2 −2
  189. tools/testing/selftests/kvm/x86_64/hyperv_clock.c +1 −1
  190. tools/testing/selftests/kvm/x86_64/hyperv_extended_hypercalls.c +97 −0
  191. tools/testing/selftests/kvm/x86_64/hyperv_features.c +207 −132
  192. tools/testing/selftests/kvm/x86_64/mmio_warning_test.c +1 −1
  193. tools/testing/selftests/kvm/x86_64/pmu_event_filter_test.c +378 −7
  194. tools/testing/selftests/kvm/x86_64/tsc_msrs_test.c +12 −4
  195. tools/testing/selftests/kvm/x86_64/vmx_exception_with_invalid_guest_state.c +1 −1
  196. tools/testing/selftests/kvm/x86_64/xapic_state_test.c +55 −0
  197. tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c +6 −9
  198. virt/kvm/Kconfig +3 −0
  199. virt/kvm/coalesced_mmio.c +5 −3
  200. virt/kvm/kvm_main.c +156 −141
Documentation/admin-guide/kernel-parameters.txt+6−1
@@ -2536,9 +2536,14 @@ · 2535 unmodified lines
25362536 protected: nVHE-based mode with support for guests whose
25372537 state is kept private from the host.
25382538
2539+ nested: VHE-based mode with support for nested
2540+ virtualization. Requires at least ARMv8.3
2541+ hardware.
2542+
25392543 Defaults to VHE/nVHE based on hardware support. Setting
25402544 mode to "protected" will disable kexec and hibernation
2541− for the host.
2545+ for the host. "nested" is experimental and should be
2546+ used with extreme caution.
25422547
25432548 kvm-arm.vgic_v3_group0_trap=
25442549 [KVM,ARM] Trap guest accesses to GICv3 group-0
Documentation/virt/kvm/api.rst+108−16
@@ -3736,7 +3736,7 @@ The fields in each entry are defined as follows: · 3735 unmodified lines
37363736 :Parameters: struct kvm_s390_mem_op (in)
37373737 :Returns: = 0 on success,
37383738 < 0 on generic error (e.g. -EFAULT or -ENOMEM),
3739− > 0 if an exception occurred while walking the page tables
3739+ 16 bit program exception code if the access causes such an exception
37403740
37413741 Read or write data from/to the VM's memory.
37423742 The KVM_CAP_S390_MEM_OP_EXTENSION capability specifies what functionality is
@@ -3754,6 +3754,8 @@ Parameters are specified via the following structure:: · 11 unmodified lines
37543754 struct {
37553755 __u8 ar; /* the access register number */
37563756 __u8 key; /* access key, ignored if flag unset */
3757+ __u8 pad1[6]; /* ignored */
3758+ __u64 old_addr; /* ignored if flag unset */
37573759 };
37583760 __u32 sida_offset; /* offset into the sida */
37593761 __u8 reserved[32]; /* ignored */
@@ -3781,6 +3783,7 @@ Possible operations are: · 21 unmodified lines
37813783 * ``KVM_S390_MEMOP_ABSOLUTE_WRITE``
37823784 * ``KVM_S390_MEMOP_SIDA_READ``
37833785 * ``KVM_S390_MEMOP_SIDA_WRITE``
3786+ * ``KVM_S390_MEMOP_ABSOLUTE_CMPXCHG``
37843787
37853788 Logical read/write:
37863789 ^^^^^^^^^^^^^^^^^^^
@@ -3829,7 +3832,7 @@ the checks required for storage key protection as one operation (as opposed to · 42 unmodified lines
38293832 user space getting the storage keys, performing the checks, and accessing
38303833 memory thereafter, which could lead to a delay between check and access).
38313834 Absolute accesses are permitted for the VM ioctl if KVM_CAP_S390_MEM_OP_EXTENSION
3832−is > 0.
3835+has the KVM_S390_MEMOP_EXTENSION_CAP_BASE bit set.
38333836 Currently absolute accesses are not permitted for VCPU ioctls.
38343837 Absolute accesses are permitted for non-protected guests only.
38353838
@@ -3837,7 +3840,26 @@ Supported flags: · 1 unmodified line
38373840 * ``KVM_S390_MEMOP_F_CHECK_ONLY``
38383841 * ``KVM_S390_MEMOP_F_SKEY_PROTECTION``
38393842
3840−The semantics of the flags are as for logical accesses.
3843+The semantics of the flags common with logical accesses are as for logical
3844+accesses.
3845+
3846+Absolute cmpxchg:
3847+^^^^^^^^^^^^^^^^^
3848+
3849+Perform cmpxchg on absolute guest memory. Intended for use with the
3850+KVM_S390_MEMOP_F_SKEY_PROTECTION flag.
3851+Instead of doing an unconditional write, the access occurs only if the target
3852+location contains the value pointed to by "old_addr".
3853+This is performed as an atomic cmpxchg with the length specified by the "size"
3854+parameter. "size" must be a power of two up to and including 16.
3855+If the exchange did not take place because the target value doesn't match the
3856+old value, the value "old_addr" points to is replaced by the target value.
3857+User space can tell if an exchange took place by checking if this replacement
3858+occurred. The cmpxchg op is permitted for the VM ioctl if
3859+KVM_CAP_S390_MEM_OP_EXTENSION has flag KVM_S390_MEMOP_EXTENSION_CAP_CMPXCHG set.
3860+
3861+Supported flags:
3862+ * ``KVM_S390_MEMOP_F_SKEY_PROTECTION``
38413863
38423864 SIDA read/write:
38433865 ^^^^^^^^^^^^^^^^
@@ -4457,6 +4479,18 @@ not holding a previously reported uncorrected error). · 613 unmodified lines
44574479 :Parameters: struct kvm_s390_cmma_log (in, out)
44584480 :Returns: 0 on success, a negative value on error
44594481
4482+Errors:
4483+
4484+ ====== =============================================================
4485+ ENOMEM not enough memory can be allocated to complete the task
4486+ ENXIO if CMMA is not enabled
4487+ EINVAL if KVM_S390_CMMA_PEEK is not set but migration mode was not enabled
4488+ EINVAL if KVM_S390_CMMA_PEEK is not set but dirty tracking has been
4489+ disabled (and thus migration mode was automatically disabled)
4490+ EFAULT if the userspace address is invalid or if no page table is
4491+ present for the addresses (e.g. when using hugepages).
4492+ ====== =============================================================
4493+
44604494 This ioctl is used to get the values of the CMMA bits on the s390
44614495 architecture. It is meant to be used in two scenarios:
44624496
@@ -4537,12 +4571,6 @@ mask is unused. · 74 unmodified lines
45374571
45384572 values points to the userspace buffer where the result will be stored.
45394573
4540−This ioctl can fail with -ENOMEM if not enough memory can be allocated to
4541−complete the task, with -ENXIO if CMMA is not enabled, with -EINVAL if
4542−KVM_S390_CMMA_PEEK is not set but migration mode was not enabled, with
4543−-EFAULT if the userspace address is invalid or if no page table is
4544−present for the addresses (e.g. when using hugepages).
4545−
45464574 4.108 KVM_S390_SET_CMMA_BITS
45474575 ----------------------------
45484576
@@ -5005,6 +5033,15 @@ using this ioctl. · 456 unmodified lines
50055033 :Parameters: struct kvm_pmu_event_filter (in)
50065034 :Returns: 0 on success, -1 on error
50075035
5036+Errors:
5037+
5038+ ====== ============================================================
5039+ EFAULT args[0] cannot be accessed
5040+ EINVAL args[0] contains invalid data in the filter or filter events
5041+ E2BIG nevents is too large
5042+ EBUSY not enough memory to allocate the filter
5043+ ====== ============================================================
5044+
50085045 ::
50095046
50105047 struct kvm_pmu_event_filter {
@@ -5016,14 +5053,69 @@ using this ioctl. · 5 unmodified lines
50165053 __u64 events[0];
50175054 };
50185055
5019−This ioctl restricts the set of PMU events that the guest can program.
5020−The argument holds a list of events which will be allowed or denied.
5021−The eventsel+umask of each event the guest attempts to program is compared
5022−against the events field to determine whether the guest should have access.
5023−The events field only controls general purpose counters; fixed purpose
5024−counters are controlled by the fixed_counter_bitmap.
5056+This ioctl restricts the set of PMU events the guest can program by limiting
5057+which event select and unit mask combinations are permitted.
5058+
5059+The argument holds a list of filter events which will be allowed or denied.
5060+
5061+Filter events only control general purpose counters; fixed purpose counters
5062+are controlled by the fixed_counter_bitmap.
5063+
5064+Valid values for 'flags'::
5065+
5066+``0``
5067+
5068+To use this mode, clear the 'flags' field.
5069+
5070+In this mode each event will contain an event select + unit mask.
5071+
5072+When the guest attempts to program the PMU the guest's event select +
5073+unit mask is compared against the filter events to determine whether the
5074+guest should have access.
5075+
5076+``KVM_PMU_EVENT_FLAG_MASKED_EVENTS``
5077+:Capability: KVM_CAP_PMU_EVENT_MASKED_EVENTS
5078+
5079+In this mode each filter event will contain an event select, mask, match, and
5080+exclude value. To encode a masked event use::
5081+
5082+ KVM_PMU_ENCODE_MASKED_ENTRY()
5083+
5084+An encoded event will follow this layout::
5085+
5086+ Bits Description
5087+ ---- -----------
5088+ 7:0 event select (low bits)
5089+ 15:8 umask match
5090+ 31:16 unused
5091+ 35:32 event select (high bits)
5092+ 36:54 unused
5093+ 55 exclude bit
5094+ 63:56 umask mask
5095+
5096+When the guest attempts to program the PMU, these steps are followed in
5097+determining if the guest should have access:
5098+
5099+ 1. Match the event select from the guest against the filter events.
5100+ 2. If a match is found, match the guest's unit mask to the mask and match
5101+ values of the included filter events.
5102+ I.e. (unit mask & mask) == match && !exclude.
5103+ 3. If a match is found, match the guest's unit mask to the mask and match
5104+ values of the excluded filter events.
5105+ I.e. (unit mask & mask) == match && exclude.
5106+ 4.
5107+ a. If an included match is found and an excluded match is not found, filter
5108+ the event.
5109+ b. For everything else, do not filter the event.
5110+ 5.
5111+ a. If the event is filtered and it's an allow list, allow the guest to
5112+ program the event.
5113+ b. If the event is filtered and it's a deny list, do not allow the guest to
5114+ program the event.
50255115
5026−No flags are defined yet, the field must be zero.
5116+When setting a new pmu event filter, -EINVAL will be returned if any of the
5117+unused fields are set or if any of the high bits (35:32) in the event
5118+select are set when called on Intel.
50275119
50285120 Valid values for 'action'::
50295121
Documentation/virt/kvm/devices/vm.rst+4−0
@@ -302,6 +302,10 @@ Allows userspace to start migration mode, needed for PGSTE migration. · 301 unmodified lines
302302 Setting this attribute when migration mode is already active will have
303303 no effects.
304304
305+Dirty tracking must be enabled on all memslots, else -EINVAL is returned. When
306+dirty tracking is disabled on any memslot, migration mode is automatically
307+stopped.
308+
305309 :Parameters: none
306310 :Returns: -ENOMEM if there is not enough free memory to start migration mode;
307311 -EINVAL if the state of the VM is invalid (e.g. no memory defined);
Documentation/virt/kvm/locking.rst+16−9
@@ -9,6 +9,8 @@ KVM Lock Overview · 8 unmodified lines
99
1010 The acquisition orders for mutexes are as follows:
1111
12+- cpus_read_lock() is taken outside kvm_lock
13+
1214 - kvm->lock is taken outside vcpu->mutex
1315
1416 - kvm->lock is taken outside kvm->slots_lock and kvm->irq_lock
@@ -226,15 +228,10 @@ time it will be set using the Dirty tracking mechanism described above. · 211 unmodified lines
226228 :Type: mutex
227229 :Arch: any
228230 :Protects: - vm_list
229−
230−``kvm_count_lock``
231−^^^^^^^^^^^^^^^^^^
232−
233−:Type: raw_spinlock_t
234−:Arch: any
235−:Protects: - hardware virtualization enable/disable
236−:Comment: 'raw' because hardware enabling/disabling must be atomic /wrt
237− migration.
231+ - kvm_usage_count
232+ - hardware virtualization enable/disable
233+:Comment: KVM also disables CPU hotplug via cpus_read_lock() during
234+ enable/disable.
238235
239236 ``kvm->mn_invalidate_lock``
240237 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -292,3 +289,13 @@ time it will be set using the Dirty tracking mechanism described above. · 51 unmodified lines
292289 wakeup notification event since external interrupts from the
293290 assigned devices happens, we will find the vCPU on the list to
294291 wakeup.
292+
293+``vendor_module_lock``
294+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
295+:Type: mutex
296+:Arch: x86
297+:Protects: loading a vendor module (kvm_amd or kvm_intel)
298+:Comment: Exists because using kvm_lock leads to deadlock. cpu_hotplug_lock is
299+ taken outside of kvm_lock, e.g. in KVM's CPU online/offline callbacks, and
300+ many operations need to take cpu_hotplug_lock when loading a vendor module,
301+ e.g. updating static calls.
Documentation/virt/kvm/x86/errata.rst+11−0
@@ -37,3 +37,14 @@ Nested virtualization features · 36 unmodified lines
3737 ------------------------------
3838
3939 TBD
40+
41+x2APIC
42+------
43+When KVM_X2APIC_API_USE_32BIT_IDS is enabled, KVM activates a hack/quirk that
44+allows sending events to a single vCPU using its x2APIC ID even if the target
45+vCPU has legacy xAPIC enabled, e.g. to bring up hotplugged vCPUs via INIT-SIPI
46+on VMs with > 255 vCPUs. A side effect of the quirk is that, if multiple vCPUs
47+have the same physical APIC ID, KVM will deliver events targeting that APIC ID
48+only to the vCPU with the lowest vCPU ID. If KVM_X2APIC_API_USE_32BIT_IDS is
49+not enabled, KVM follows x86 architecture when processing interrupts (all vCPUs
50+matching the target APIC ID receive the interrupt).
MAINTAINERS+1−2
@@ -11269,13 +11269,12 @@ F: virt/kvm/* · 11268 unmodified lines
1126911269
1127011270 KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)
1127111271 M: Marc Zyngier <maz@kernel.org>
11272+M: Oliver Upton <oliver.upton@linux.dev>
1127211273 R: James Morse <james.morse@arm.com>
1127311274 R: Suzuki K Poulose <suzuki.poulose@arm.com>
11274−R: Oliver Upton <oliver.upton@linux.dev>
1127511275 R: Zenghui Yu <yuzenghui@huawei.com>
1127611276 L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
1127711277 L: kvmarm@lists.linux.dev
11278−L: kvmarm@lists.cs.columbia.edu (deprecated, moderated for non-subscribers)
1127911278 S: Maintained
1128011279 T: git git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git
1128111280 F: arch/arm64/include/asm/kvm*
arch/arm64/include/asm/cache.h+9−0
@@ -16,6 +16,15 @@ · 15 unmodified lines
1616 #define CLIDR_LOC(clidr) (((clidr) >> CLIDR_LOC_SHIFT) & 0x7)
1717 #define CLIDR_LOUIS(clidr) (((clidr) >> CLIDR_LOUIS_SHIFT) & 0x7)
1818
19+/* Ctypen, bits[3(n - 1) + 2 : 3(n - 1)], for n = 1 to 7 */
20+#define CLIDR_CTYPE_SHIFT(level) (3 * (level - 1))
21+#define CLIDR_CTYPE_MASK(level) (7 << CLIDR_CTYPE_SHIFT(level))
22+#define CLIDR_CTYPE(clidr, level) \
23+ (((clidr) & CLIDR_CTYPE_MASK(level)) >> CLIDR_CTYPE_SHIFT(level))
24+
25+/* Ttypen, bits [2(n - 1) + 34 : 2(n - 1) + 33], for n = 1 to 7 */
26+#define CLIDR_TTYPE_SHIFT(level) (2 * ((level) - 1) + CLIDR_EL1_Ttypen_SHIFT)
27+
1928 /*
2029 * Memory returned by kmalloc() may be used for DMA, so we must make
2130 * sure that all such allocations are cache aligned. Otherwise,
arch/arm64/include/asm/el2_setup.h+99−0
@@ -196,4 +196,103 @@ · 195 unmodified lines
196196 __init_el2_nvhe_prepare_eret
197197 .endm
198198
199+#ifndef __KVM_NVHE_HYPERVISOR__
200+// This will clobber tmp1 and tmp2, and expect tmp1 to contain
201+// the id register value as read from the HW
202+.macro __check_override idreg, fld, width, pass, fail, tmp1, tmp2
203+ ubfx \tmp1, \tmp1, #\fld, #\width
204+ cbz \tmp1, \fail
205+
206+ adr_l \tmp1, \idreg\()_override
207+ ldr \tmp2, [\tmp1, FTR_OVR_VAL_OFFSET]
208+ ldr \tmp1, [\tmp1, FTR_OVR_MASK_OFFSET]
209+ ubfx \tmp2, \tmp2, #\fld, #\width
210+ ubfx \tmp1, \tmp1, #\fld, #\width
211+ cmp \tmp1, xzr
212+ and \tmp2, \tmp2, \tmp1
213+ csinv \tmp2, \tmp2, xzr, ne
214+ cbnz \tmp2, \pass
215+ b \fail
216+.endm
217+
218+// This will clobber tmp1 and tmp2
219+.macro check_override idreg, fld, pass, fail, tmp1, tmp2
220+ mrs \tmp1, \idreg\()_el1
221+ __check_override \idreg \fld 4 \pass \fail \tmp1 \tmp2
222+.endm
223+#else
224+// This will clobber tmp
225+.macro __check_override idreg, fld, width, pass, fail, tmp, ignore
226+ ldr_l \tmp, \idreg\()_el1_sys_val
227+ ubfx \tmp, \tmp, #\fld, #\width
228+ cbnz \tmp, \pass
229+ b \fail
230+.endm
231+
232+.macro check_override idreg, fld, pass, fail, tmp, ignore
233+ __check_override \idreg \fld 4 \pass \fail \tmp \ignore
234+.endm
235+#endif
236+
237+.macro finalise_el2_state
238+ check_override id_aa64pfr0, ID_AA64PFR0_EL1_SVE_SHIFT, .Linit_sve_\@, .Lskip_sve_\@, x1, x2
239+
240+.Linit_sve_\@: /* SVE register access */
241+ mrs x0, cptr_el2 // Disable SVE traps
242+ bic x0, x0, #CPTR_EL2_TZ
243+ msr cptr_el2, x0
244+ isb
245+ mov x1, #ZCR_ELx_LEN_MASK // SVE: Enable full vector
246+ msr_s SYS_ZCR_EL2, x1 // length for EL1.
247+
248+.Lskip_sve_\@:
249+ check_override id_aa64pfr1, ID_AA64PFR1_EL1_SME_SHIFT, .Linit_sme_\@, .Lskip_sme_\@, x1, x2
250+
251+.Linit_sme_\@: /* SME register access and priority mapping */
252+ mrs x0, cptr_el2 // Disable SME traps
253+ bic x0, x0, #CPTR_EL2_TSM
254+ msr cptr_el2, x0
255+ isb
256+
257+ mrs x1, sctlr_el2
258+ orr x1, x1, #SCTLR_ELx_ENTP2 // Disable TPIDR2 traps
259+ msr sctlr_el2, x1
260+ isb
261+
262+ mov x0, #0 // SMCR controls
263+
264+ // Full FP in SM?
265+ mrs_s x1, SYS_ID_AA64SMFR0_EL1
266+ __check_override id_aa64smfr0, ID_AA64SMFR0_EL1_FA64_SHIFT, 1, .Linit_sme_fa64_\@, .Lskip_sme_fa64_\@, x1, x2
267+
268+.Linit_sme_fa64_\@:
269+ orr x0, x0, SMCR_ELx_FA64_MASK
270+.Lskip_sme_fa64_\@:
271+
272+ // ZT0 available?
273+ mrs_s x1, SYS_ID_AA64SMFR0_EL1
274+ __check_override id_aa64smfr0, ID_AA64SMFR0_EL1_SMEver_SHIFT, 4, .Linit_sme_zt0_\@, .Lskip_sme_zt0_\@, x1, x2
275+.Linit_sme_zt0_\@:
276+ orr x0, x0, SMCR_ELx_EZT0_MASK
277+.Lskip_sme_zt0_\@:
278+
279+ orr x0, x0, #SMCR_ELx_LEN_MASK // Enable full SME vector
280+ msr_s SYS_SMCR_EL2, x0 // length for EL1.
281+
282+ mrs_s x1, SYS_SMIDR_EL1 // Priority mapping supported?
283+ ubfx x1, x1, #SMIDR_EL1_SMPS_SHIFT, #1
284+ cbz x1, .Lskip_sme_\@
285+
286+ msr_s SYS_SMPRIMAP_EL2, xzr // Make all priorities equal
287+
288+ mrs x1, id_aa64mmfr1_el1 // HCRX_EL2 present?
289+ ubfx x1, x1, #ID_AA64MMFR1_EL1_HCX_SHIFT, #4
290+ cbz x1, .Lskip_sme_\@
291+
292+ mrs_s x1, SYS_HCRX_EL2
293+ orr x1, x1, #HCRX_EL2_SMPME_MASK // Enable priority mapping
294+ msr_s SYS_HCRX_EL2, x1
295+.Lskip_sme_\@:
296+.endm
297+
199298 #endif /* __ARM_KVM_INIT_H__ */
arch/arm64/include/asm/esr.h+4−0
@@ -272,6 +272,10 @@ · 271 unmodified lines
272272 (((e) & ESR_ELx_SYS64_ISS_OP2_MASK) >> \
273273 ESR_ELx_SYS64_ISS_OP2_SHIFT))
274274
275+/* ISS field definitions for ERET/ERETAA/ERETAB trapping */
276+#define ESR_ELx_ERET_ISS_ERET 0x2
277+#define ESR_ELx_ERET_ISS_ERETA 0x1
278+
275279 /*
276280 * ISS field definitions for floating-point exception traps
277281 * (FP_EXC_32/FP_EXC_64).
arch/arm64/include/asm/kvm_arm.h+20−3
@@ -81,11 +81,12 @@ · 80 unmodified lines
8181 * SWIO: Turn set/way invalidates into set/way clean+invalidate
8282 * PTW: Take a stage2 fault if a stage1 walk steps in device memory
8383 * TID3: Trap EL1 reads of group 3 ID registers
84+ * TID2: Trap CTR_EL0, CCSIDR2_EL1, CLIDR_EL1, and CSSELR_EL1
8485 */
8586 #define HCR_GUEST_FLAGS (HCR_TSC | HCR_TSW | HCR_TWE | HCR_TWI | HCR_VM | \
8687 HCR_BSU_IS | HCR_FB | HCR_TACR | \
8788 HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \
88− HCR_FMO | HCR_IMO | HCR_PTW | HCR_TID3 )
89+ HCR_FMO | HCR_IMO | HCR_PTW | HCR_TID3 | HCR_TID2)
8990 #define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF)
9091 #define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK | HCR_ATA)
9192 #define HCR_HOST_NVHE_PROTECTED_FLAGS (HCR_HOST_NVHE_FLAGS | HCR_TSC)
@@ -344,10 +345,26 @@ · 252 unmodified lines
344345 ECN(SP_ALIGN), ECN(FP_EXC32), ECN(FP_EXC64), ECN(SERROR), \
345346 ECN(BREAKPT_LOW), ECN(BREAKPT_CUR), ECN(SOFTSTP_LOW), \
346347 ECN(SOFTSTP_CUR), ECN(WATCHPT_LOW), ECN(WATCHPT_CUR), \
347− ECN(BKPT32), ECN(VECTOR32), ECN(BRK64)
348+ ECN(BKPT32), ECN(VECTOR32), ECN(BRK64), ECN(ERET)
348349
349−#define CPACR_EL1_TTA (1 << 28)
350350 #define CPACR_EL1_DEFAULT (CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN |\
351351 CPACR_EL1_ZEN_EL1EN)
352352
353+#define kvm_mode_names \
354+ { PSR_MODE_EL0t, "EL0t" }, \
355+ { PSR_MODE_EL1t, "EL1t" }, \
356+ { PSR_MODE_EL1h, "EL1h" }, \
357+ { PSR_MODE_EL2t, "EL2t" }, \
358+ { PSR_MODE_EL2h, "EL2h" }, \
359+ { PSR_MODE_EL3t, "EL3t" }, \
360+ { PSR_MODE_EL3h, "EL3h" }, \
361+ { PSR_AA32_MODE_USR, "32-bit USR" }, \
362+ { PSR_AA32_MODE_FIQ, "32-bit FIQ" }, \
363+ { PSR_AA32_MODE_IRQ, "32-bit IRQ" }, \
364+ { PSR_AA32_MODE_SVC, "32-bit SVC" }, \
365+ { PSR_AA32_MODE_ABT, "32-bit ABT" }, \
366+ { PSR_AA32_MODE_HYP, "32-bit HYP" }, \
367+ { PSR_AA32_MODE_UND, "32-bit UND" }, \
368+ { PSR_AA32_MODE_SYS, "32-bit SYS" }
369+
353370 #endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/include/asm/kvm_emulate.h+66−4
@@ -33,6 +33,12 @@ enum exception_type { · 32 unmodified lines
3333 except_type_serror = 0x180,
3434 };
3535
36+#define kvm_exception_type_names \
37+ { except_type_sync, "SYNC" }, \
38+ { except_type_irq, "IRQ" }, \
39+ { except_type_fiq, "FIQ" }, \
40+ { except_type_serror, "SERROR" }
41+
3642 bool kvm_condition_valid32(const struct kvm_vcpu *vcpu);
3743 void kvm_skip_instr32(struct kvm_vcpu *vcpu);
3844
@@ -44,6 +50,10 @@ void kvm_inject_size_fault(struct kvm_vcpu *vcpu); · 5 unmodified lines
4450
4551 void kvm_vcpu_wfi(struct kvm_vcpu *vcpu);
4652
53+void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu);
54+int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2);
55+int kvm_inject_nested_irq(struct kvm_vcpu *vcpu);
56+
4757 #if defined(__KVM_VHE_HYPERVISOR__) || defined(__KVM_NVHE_HYPERVISOR__)
4858 static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
4959 {
@@ -88,10 +98,6 @@ static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu) · 38 unmodified lines
8898 if (vcpu_el1_is_32bit(vcpu))
8999 vcpu->arch.hcr_el2 &= ~HCR_RW;
90100
91− if (cpus_have_const_cap(ARM64_MISMATCHED_CACHE_TYPE) ||
92− vcpu_el1_is_32bit(vcpu))
93− vcpu->arch.hcr_el2 |= HCR_TID2;
94−
95101 if (kvm_has_mte(vcpu->kvm))
96102 vcpu->arch.hcr_el2 |= HCR_ATA;
97103 }
@@ -183,6 +189,62 @@ static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · 85 unmodified lines
183189 vcpu_gp_regs(vcpu)->regs[reg_num] = val;
184190 }
185191
192+static inline bool vcpu_is_el2_ctxt(const struct kvm_cpu_context *ctxt)
193+{
194+ switch (ctxt->regs.pstate & (PSR_MODE32_BIT | PSR_MODE_MASK)) {
195+ case PSR_MODE_EL2h:
196+ case PSR_MODE_EL2t:
197+ return true;
198+ default:
199+ return false;
200+ }
201+}
202+
203+static inline bool vcpu_is_el2(const struct kvm_vcpu *vcpu)
204+{
205+ return vcpu_is_el2_ctxt(&vcpu->arch.ctxt);
206+}
207+
208+static inline bool __vcpu_el2_e2h_is_set(const struct kvm_cpu_context *ctxt)
209+{
210+ return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_E2H;
211+}
212+
213+static inline bool vcpu_el2_e2h_is_set(const struct kvm_vcpu *vcpu)
214+{
215+ return __vcpu_el2_e2h_is_set(&vcpu->arch.ctxt);
216+}
217+
218+static inline bool __vcpu_el2_tge_is_set(const struct kvm_cpu_context *ctxt)
219+{
220+ return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_TGE;
221+}
222+
223+static inline bool vcpu_el2_tge_is_set(const struct kvm_vcpu *vcpu)
224+{
225+ return __vcpu_el2_tge_is_set(&vcpu->arch.ctxt);
226+}
227+
228+static inline bool __is_hyp_ctxt(const struct kvm_cpu_context *ctxt)
229+{
230+ /*
231+ * We are in a hypervisor context if the vcpu mode is EL2 or
232+ * E2H and TGE bits are set. The latter means we are in the user space
233+ * of the VHE kernel. ARMv8.1 ARM describes this as 'InHost'
234+ *
235+ * Note that the HCR_EL2.{E2H,TGE}={0,1} isn't really handled in the
236+ * rest of the KVM code, and will result in a misbehaving guest.
237+ */
238+ return vcpu_is_el2_ctxt(ctxt) ||
239+ (__vcpu_el2_e2h_is_set(ctxt) && __vcpu_el2_tge_is_set(ctxt)) ||
240+ __vcpu_el2_tge_is_set(ctxt);
241+}
242+
243+static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu)
244+{
245+ return __is_hyp_ctxt(&vcpu->arch.ctxt);
246+}
247+
186248 /*
187249 * The layout of SPSR for an AArch32 state is different when observed from an
188250 * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32
arch/arm64/include/asm/kvm_host.h+53−14
@@ -60,14 +60,19 @@ · 59 unmodified lines
6060 enum kvm_mode {
6161 KVM_MODE_DEFAULT,
6262 KVM_MODE_PROTECTED,
63+ KVM_MODE_NV,
6364 KVM_MODE_NONE,
6465 };
66+#ifdef CONFIG_KVM
6567 enum kvm_mode kvm_get_mode(void);
68+#else
69+static inline enum kvm_mode kvm_get_mode(void) { return KVM_MODE_NONE; };
70+#endif
6671
6772 DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use);
6873
69−extern unsigned int kvm_sve_max_vl;
70−int kvm_arm_init_sve(void);
74+extern unsigned int __ro_after_init kvm_sve_max_vl;
75+int __init kvm_arm_init_sve(void);
7176
7277 u32 __attribute_const__ kvm_target_cpu(void);
7378 int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
@@ -252,6 +257,7 @@ struct kvm_vcpu_fault_info { · 178 unmodified lines
252257 enum vcpu_sysreg {
253258 __INVALID_SYSREG__, /* 0 is reserved as an invalid value */
254259 MPIDR_EL1, /* MultiProcessor Affinity Register */
260+ CLIDR_EL1, /* Cache Level ID Register */
255261 CSSELR_EL1, /* Cache Size Selection Register */
256262 SCTLR_EL1, /* System Control Register */
257263 ACTLR_EL1, /* Auxiliary Control Register */
@@ -320,12 +326,43 @@ enum vcpu_sysreg { · 62 unmodified lines
320326 TFSR_EL1, /* Tag Fault Status Register (EL1) */
321327 TFSRE0_EL1, /* Tag Fault Status Register (EL0) */
322328
323− /* 32bit specific registers. Keep them at the end of the range */
329+ /* 32bit specific registers. */
324330 DACR32_EL2, /* Domain Access Control Register */
325331 IFSR32_EL2, /* Instruction Fault Status Register */
326332 FPEXC32_EL2, /* Floating-Point Exception Control Register */
327333 DBGVCR32_EL2, /* Debug Vector Catch Register */
328334
335+ /* EL2 registers */
336+ VPIDR_EL2, /* Virtualization Processor ID Register */
337+ VMPIDR_EL2, /* Virtualization Multiprocessor ID Register */
338+ SCTLR_EL2, /* System Control Register (EL2) */
339+ ACTLR_EL2, /* Auxiliary Control Register (EL2) */
340+ HCR_EL2, /* Hypervisor Configuration Register */
341+ MDCR_EL2, /* Monitor Debug Configuration Register (EL2) */
342+ CPTR_EL2, /* Architectural Feature Trap Register (EL2) */
343+ HSTR_EL2, /* Hypervisor System Trap Register */
344+ HACR_EL2, /* Hypervisor Auxiliary Control Register */
345+ TTBR0_EL2, /* Translation Table Base Register 0 (EL2) */
346+ TTBR1_EL2, /* Translation Table Base Register 1 (EL2) */
347+ TCR_EL2, /* Translation Control Register (EL2) */
348+ VTTBR_EL2, /* Virtualization Translation Table Base Register */
349+ VTCR_EL2, /* Virtualization Translation Control Register */
350+ SPSR_EL2, /* EL2 saved program status register */
351+ ELR_EL2, /* EL2 exception link register */
352+ AFSR0_EL2, /* Auxiliary Fault Status Register 0 (EL2) */
353+ AFSR1_EL2, /* Auxiliary Fault Status Register 1 (EL2) */
354+ ESR_EL2, /* Exception Syndrome Register (EL2) */
355+ FAR_EL2, /* Fault Address Register (EL2) */
356+ HPFAR_EL2, /* Hypervisor IPA Fault Address Register */
357+ MAIR_EL2, /* Memory Attribute Indirection Register (EL2) */
358+ AMAIR_EL2, /* Auxiliary Memory Attribute Indirection Register (EL2) */
359+ VBAR_EL2, /* Vector Base Address Register (EL2) */
360+ RVBAR_EL2, /* Reset Vector Base Address Register */
361+ CONTEXTIDR_EL2, /* Context ID Register (EL2) */
362+ TPIDR_EL2, /* EL2 Software Thread ID Register */
363+ CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */
364+ SP_EL2, /* EL2 Stack Pointer */
365+
329366 NR_SYS_REGS /* Nothing after this line! */
330367 };
331368
@@ -501,6 +538,9 @@ struct kvm_vcpu_arch { · 169 unmodified lines
501538 u64 last_steal;
502539 gpa_t base;
503540 } steal;
541+
542+ /* Per-vcpu CCSIDR override or NULL */
543+ u32 *ccsidr;
504544 };
505545
506546 /*
@@ -598,7 +638,7 @@ struct kvm_vcpu_arch { · 91 unmodified lines
598638 #define EXCEPT_AA64_EL1_IRQ __vcpu_except_flags(1)
599639 #define EXCEPT_AA64_EL1_FIQ __vcpu_except_flags(2)
600640 #define EXCEPT_AA64_EL1_SERR __vcpu_except_flags(3)
601−/* For AArch64 with NV (one day): */
641+/* For AArch64 with NV: */
602642 #define EXCEPT_AA64_EL2_SYNC __vcpu_except_flags(4)
603643 #define EXCEPT_AA64_EL2_IRQ __vcpu_except_flags(5)
604644 #define EXCEPT_AA64_EL2_FIQ __vcpu_except_flags(6)
@@ -609,6 +649,8 @@ struct kvm_vcpu_arch { · 4 unmodified lines
609649 #define DEBUG_STATE_SAVE_SPE __vcpu_single_flag(iflags, BIT(5))
610650 /* Save TRBE context if active */
611651 #define DEBUG_STATE_SAVE_TRBE __vcpu_single_flag(iflags, BIT(6))
652+/* vcpu running in HYP context */
653+#define VCPU_HYP_CONTEXT __vcpu_single_flag(iflags, BIT(7))
612654
613655 /* SVE enabled for host EL0 */
614656 #define HOST_SVE_ENABLED __vcpu_single_flag(sflags, BIT(0))
@@ -705,7 +747,6 @@ static inline bool __vcpu_read_sys_reg_from_cpu(int reg, u64 *val) · 90 unmodified lines
705747 return false;
706748
707749 switch (reg) {
708− case CSSELR_EL1: *val = read_sysreg_s(SYS_CSSELR_EL1); break;
709750 case SCTLR_EL1: *val = read_sysreg_s(SYS_SCTLR_EL12); break;
710751 case CPACR_EL1: *val = read_sysreg_s(SYS_CPACR_EL12); break;
711752 case TTBR0_EL1: *val = read_sysreg_s(SYS_TTBR0_EL12); break;
@@ -750,7 +791,6 @@ static inline bool __vcpu_write_sys_reg_to_cpu(u64 val, int reg) · 38 unmodified lines
750791 return false;
751792
752793 switch (reg) {
753− case CSSELR_EL1: write_sysreg_s(val, SYS_CSSELR_EL1); break;
754794 case SCTLR_EL1: write_sysreg_s(val, SYS_SCTLR_EL12); break;
755795 case CPACR_EL1: write_sysreg_s(val, SYS_CPACR_EL12); break;
756796 case TTBR0_EL1: write_sysreg_s(val, SYS_TTBR0_EL12); break;
@@ -877,7 +917,7 @@ int kvm_handle_cp10_id(struct kvm_vcpu *vcpu); · 120 unmodified lines
877917
878918 void kvm_reset_sys_regs(struct kvm_vcpu *vcpu);
879919
880−int kvm_sys_reg_table_init(void);
920+int __init kvm_sys_reg_table_init(void);
881921
882922 /* MMIO helpers */
883923 void kvm_mmio_write_buf(void *buf, unsigned int len, unsigned long data);
@@ -908,20 +948,20 @@ int kvm_arm_pvtime_get_attr(struct kvm_vcpu *vcpu, · 24 unmodified lines
908948 int kvm_arm_pvtime_has_attr(struct kvm_vcpu *vcpu,
909949 struct kvm_device_attr *attr);
910950
911−extern unsigned int kvm_arm_vmid_bits;
912−int kvm_arm_vmid_alloc_init(void);
913−void kvm_arm_vmid_alloc_free(void);
951+extern unsigned int __ro_after_init kvm_arm_vmid_bits;
952+int __init kvm_arm_vmid_alloc_init(void);
953+void __init kvm_arm_vmid_alloc_free(void);
914954 void kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid);
915955 void kvm_arm_vmid_clear_active(void);
916956
917957 static inline void kvm_arm_pvtime_vcpu_init(struct kvm_vcpu_arch *vcpu_arch)
918958 {
919− vcpu_arch->steal.base = GPA_INVALID;
959+ vcpu_arch->steal.base = INVALID_GPA;
920960 }
921961
922962 static inline bool kvm_arm_is_pvtime_enabled(struct kvm_vcpu_arch *vcpu_arch)
923963 {
924− return (vcpu_arch->steal.base != GPA_INVALID);
964+ return (vcpu_arch->steal.base != INVALID_GPA);
925965 }
926966
927967 void kvm_set_sei_esr(struct kvm_vcpu *vcpu, u64 syndrome);
@@ -943,7 +983,6 @@ static inline bool kvm_system_needs_idmapped_vectors(void) · 15 unmodified lines
943983
944984 void kvm_arm_vcpu_ptrauth_trap(struct kvm_vcpu *vcpu);
945985
946−static inline void kvm_arch_hardware_unsetup(void) {}
947986 static inline void kvm_arch_sync_events(struct kvm *kvm) {}
948987 static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
949988
@@ -994,7 +1033,7 @@ static inline void kvm_clr_pmu_events(u32 clr) {} · 44 unmodified lines
9941033 void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu);
9951034 void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu);
9961035
997−int kvm_set_ipa_limit(void);
1036+int __init kvm_set_ipa_limit(void);
9981037
9991038 #define __KVM_HAVE_ARCH_VM_ALLOC
10001039 struct kvm *kvm_arch_alloc_vm(void);
arch/arm64/include/asm/kvm_hyp.h+1−0
@@ -122,6 +122,7 @@ extern u64 kvm_nvhe_sym(id_aa64isar2_el1_sys_val); · 121 unmodified lines
122122 extern u64 kvm_nvhe_sym(id_aa64mmfr0_el1_sys_val);
123123 extern u64 kvm_nvhe_sym(id_aa64mmfr1_el1_sys_val);
124124 extern u64 kvm_nvhe_sym(id_aa64mmfr2_el1_sys_val);
125+extern u64 kvm_nvhe_sym(id_aa64smfr0_el1_sys_val);
125126
126127 extern unsigned long kvm_nvhe_sym(__icache_flags);
127128 extern unsigned int kvm_nvhe_sym(kvm_arm_vmid_bits);
arch/arm64/include/asm/kvm_mmu.h+12−3
@@ -115,6 +115,7 @@ alternative_cb_end · 114 unmodified lines
115115 #include <asm/cache.h>
116116 #include <asm/cacheflush.h>
117117 #include <asm/mmu_context.h>
118+#include <asm/kvm_emulate.h>
118119 #include <asm/kvm_host.h>
119120
120121 void kvm_update_va_mask(struct alt_instr *alt,
@@ -163,7 +164,7 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size, · 42 unmodified lines
163164 void __iomem **haddr);
164165 int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
165166 void **haddr);
166−void free_hyp_pgds(void);
167+void __init free_hyp_pgds(void);
167168
168169 void stage2_unmap_vm(struct kvm *kvm);
169170 int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long type);
@@ -175,7 +176,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu); · 5 unmodified lines
175176
176177 phys_addr_t kvm_mmu_get_httbr(void);
177178 phys_addr_t kvm_get_idmap_vector(void);
178−int kvm_mmu_init(u32 *hyp_va_bits);
179+int __init kvm_mmu_init(u32 *hyp_va_bits);
179180
180181 static inline void *__kvm_vector_slot2addr(void *base,
181182 enum arm64_hyp_spectre_vector slot)
@@ -192,7 +193,15 @@ struct kvm; · 10 unmodified lines
192193
193194 static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu)
194195 {
195− return (vcpu_read_sys_reg(vcpu, SCTLR_EL1) & 0b101) == 0b101;
196+ u64 cache_bits = SCTLR_ELx_M | SCTLR_ELx_C;
197+ int reg;
198+
199+ if (vcpu_is_el2(vcpu))
200+ reg = SCTLR_EL2;
201+ else
202+ reg = SCTLR_EL1;
203+
204+ return (vcpu_read_sys_reg(vcpu, reg) & cache_bits) == cache_bits;
196205 }
197206
198207 static inline void __clean_dcache_guest_page(void *va, size_t size)
arch/arm64/include/asm/kvm_nested.h+20−0
@@ -0,0 +1,20 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+#ifndef __ARM64_KVM_NESTED_H
3+#define __ARM64_KVM_NESTED_H
4+
5+#include <linux/kvm_host.h>
6+
7+static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu)
8+{
9+ return (!__is_defined(__KVM_NVHE_HYPERVISOR__) &&
10+ cpus_have_final_cap(ARM64_HAS_NESTED_VIRT) &&
11+ test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features));
12+}
13+
14+struct sys_reg_params;
15+struct sys_reg_desc;
16+
17+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
18+ const struct sys_reg_desc *r);
19+
20+#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/include/asm/kvm_pgtable.h+8−0
@@ -71,6 +71,11 @@ static inline kvm_pte_t kvm_phys_to_pte(u64 pa) · 70 unmodified lines
7171 return pte;
7272 }
7373
74+static inline kvm_pfn_t kvm_pte_to_pfn(kvm_pte_t pte)
75+{
76+ return __phys_to_pfn(kvm_pte_to_phys(pte));
77+}
78+
7479 static inline u64 kvm_granule_shift(u32 level)
7580 {
7681 /* Assumes KVM_PGTABLE_MAX_LEVELS is 4 */
@@ -188,12 +193,15 @@ typedef bool (*kvm_pgtable_force_pte_cb_t)(u64 addr, u64 end, · 111 unmodified lines
188193 * children.
189194 * @KVM_PGTABLE_WALK_SHARED: Indicates the page-tables may be shared
190195 * with other software walkers.
196+ * @KVM_PGTABLE_WALK_HANDLE_FAULT: Indicates the page-table walk was
197+ * invoked from a fault handler.
191198 */
192199 enum kvm_pgtable_walk_flags {
193200 KVM_PGTABLE_WALK_LEAF = BIT(0),
194201 KVM_PGTABLE_WALK_TABLE_PRE = BIT(1),
195202 KVM_PGTABLE_WALK_TABLE_POST = BIT(2),
196203 KVM_PGTABLE_WALK_SHARED = BIT(3),
204+ KVM_PGTABLE_WALK_HANDLE_FAULT = BIT(4),
197205 };
198206
199207 struct kvm_pgtable_visit_ctx {
arch/arm64/include/asm/sysreg.h+37−2
@@ -325,7 +325,6 @@ · 324 unmodified lines
325325
326326 #define SYS_CNTKCTL_EL1 sys_reg(3, 0, 14, 1, 0)
327327
328−#define SYS_CCSIDR_EL1 sys_reg(3, 1, 0, 0, 0)
329328 #define SYS_AIDR_EL1 sys_reg(3, 1, 0, 0, 7)
330329
331330 #define SYS_RNDR_EL0 sys_reg(3, 3, 2, 4, 0)
@@ -411,23 +410,51 @@ · 79 unmodified lines
411410
412411 #define SYS_PMCCFILTR_EL0 sys_reg(3, 3, 14, 15, 7)
413412
413+#define SYS_VPIDR_EL2 sys_reg(3, 4, 0, 0, 0)
414+#define SYS_VMPIDR_EL2 sys_reg(3, 4, 0, 0, 5)
415+
414416 #define SYS_SCTLR_EL2 sys_reg(3, 4, 1, 0, 0)
417+#define SYS_ACTLR_EL2 sys_reg(3, 4, 1, 0, 1)
418+#define SYS_HCR_EL2 sys_reg(3, 4, 1, 1, 0)
419+#define SYS_MDCR_EL2 sys_reg(3, 4, 1, 1, 1)
420+#define SYS_CPTR_EL2 sys_reg(3, 4, 1, 1, 2)
421+#define SYS_HSTR_EL2 sys_reg(3, 4, 1, 1, 3)
415422 #define SYS_HFGRTR_EL2 sys_reg(3, 4, 1, 1, 4)
416423 #define SYS_HFGWTR_EL2 sys_reg(3, 4, 1, 1, 5)
417424 #define SYS_HFGITR_EL2 sys_reg(3, 4, 1, 1, 6)
425+#define SYS_HACR_EL2 sys_reg(3, 4, 1, 1, 7)
426+
427+#define SYS_TTBR0_EL2 sys_reg(3, 4, 2, 0, 0)
428+#define SYS_TTBR1_EL2 sys_reg(3, 4, 2, 0, 1)
429+#define SYS_TCR_EL2 sys_reg(3, 4, 2, 0, 2)
430+#define SYS_VTTBR_EL2 sys_reg(3, 4, 2, 1, 0)
431+#define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2)
432+
418433 #define SYS_TRFCR_EL2 sys_reg(3, 4, 1, 2, 1)
419434 #define SYS_HDFGRTR_EL2 sys_reg(3, 4, 3, 1, 4)
420435 #define SYS_HDFGWTR_EL2 sys_reg(3, 4, 3, 1, 5)
421436 #define SYS_HAFGRTR_EL2 sys_reg(3, 4, 3, 1, 6)
422437 #define SYS_SPSR_EL2 sys_reg(3, 4, 4, 0, 0)
423438 #define SYS_ELR_EL2 sys_reg(3, 4, 4, 0, 1)
439+#define SYS_SP_EL1 sys_reg(3, 4, 4, 1, 0)
424440 #define SYS_IFSR32_EL2 sys_reg(3, 4, 5, 0, 1)
441+#define SYS_AFSR0_EL2 sys_reg(3, 4, 5, 1, 0)
442+#define SYS_AFSR1_EL2 sys_reg(3, 4, 5, 1, 1)
425443 #define SYS_ESR_EL2 sys_reg(3, 4, 5, 2, 0)
426444 #define SYS_VSESR_EL2 sys_reg(3, 4, 5, 2, 3)
427445 #define SYS_FPEXC32_EL2 sys_reg(3, 4, 5, 3, 0)
428446 #define SYS_TFSR_EL2 sys_reg(3, 4, 5, 6, 0)
429447
430−#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1)
448+#define SYS_FAR_EL2 sys_reg(3, 4, 6, 0, 0)
449+#define SYS_HPFAR_EL2 sys_reg(3, 4, 6, 0, 4)
450+
451+#define SYS_MAIR_EL2 sys_reg(3, 4, 10, 2, 0)
452+#define SYS_AMAIR_EL2 sys_reg(3, 4, 10, 3, 0)
453+
454+#define SYS_VBAR_EL2 sys_reg(3, 4, 12, 0, 0)
455+#define SYS_RVBAR_EL2 sys_reg(3, 4, 12, 0, 1)
456+#define SYS_RMR_EL2 sys_reg(3, 4, 12, 0, 2)
457+#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1)
431458 #define __SYS__AP0Rx_EL2(x) sys_reg(3, 4, 12, 8, x)
432459 #define SYS_ICH_AP0R0_EL2 __SYS__AP0Rx_EL2(0)
433460 #define SYS_ICH_AP0R1_EL2 __SYS__AP0Rx_EL2(1)
@@ -469,6 +496,12 @@ · 35 unmodified lines
469496 #define SYS_ICH_LR14_EL2 __SYS__LR8_EL2(6)
470497 #define SYS_ICH_LR15_EL2 __SYS__LR8_EL2(7)
471498
499+#define SYS_CONTEXTIDR_EL2 sys_reg(3, 4, 13, 0, 1)
500+#define SYS_TPIDR_EL2 sys_reg(3, 4, 13, 0, 2)
501+
502+#define SYS_CNTVOFF_EL2 sys_reg(3, 4, 14, 0, 3)
503+#define SYS_CNTHCTL_EL2 sys_reg(3, 4, 14, 1, 0)
504+
472505 /* VHE encodings for architectural EL0/1 system registers */
473506 #define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0)
474507 #define SYS_TTBR0_EL12 sys_reg(3, 5, 2, 0, 0)
@@ -491,6 +524,8 @@ · 16 unmodified lines
491524 #define SYS_CNTV_CTL_EL02 sys_reg(3, 5, 14, 3, 1)
492525 #define SYS_CNTV_CVAL_EL02 sys_reg(3, 5, 14, 3, 2)
493526
527+#define SYS_SP_EL2 sys_reg(3, 6, 4, 1, 0)
528+
494529 /* Common SCTLR_ELx flags. */
495530 #define SCTLR_ELx_ENTP2 (BIT(60))
496531 #define SCTLR_ELx_DSSBS (BIT(44))
arch/arm64/include/uapi/asm/kvm.h+1−0
@@ -109,6 +109,7 @@ struct kvm_regs { · 108 unmodified lines
109109 #define KVM_ARM_VCPU_SVE 4 /* enable SVE for this CPU */
110110 #define KVM_ARM_VCPU_PTRAUTH_ADDRESS 5 /* VCPU uses address authentication */
111111 #define KVM_ARM_VCPU_PTRAUTH_GENERIC 6 /* VCPU uses generic authentication */
112+#define KVM_ARM_VCPU_HAS_EL2 7 /* Support nested virtualization */
112113
113114 struct kvm_vcpu_init {
114115 __u32 target;
arch/arm64/kernel/cacheinfo.c+0−5
@@ -11,11 +11,6 @@ · 10 unmodified lines
1111 #include <linux/of.h>
1212
1313 #define MAX_CACHE_LEVEL 7 /* Max 7 level supported */
14−/* Ctypen, bits[3(n - 1) + 2 : 3(n - 1)], for n = 1 to 7 */
15−#define CLIDR_CTYPE_SHIFT(level) (3 * (level - 1))
16−#define CLIDR_CTYPE_MASK(level) (7 << CLIDR_CTYPE_SHIFT(level))
17−#define CLIDR_CTYPE(clidr, level) \
18− (((clidr) & CLIDR_CTYPE_MASK(level)) >> CLIDR_CTYPE_SHIFT(level))
1914
2015 int cache_line_size(void)
2116 {
arch/arm64/kernel/cpufeature.c+25−0
@@ -1967,6 +1967,20 @@ static void cpu_copy_el2regs(const struct arm64_cpu_capabilities *__unused) · 1966 unmodified lines
19671967 write_sysreg(read_sysreg(tpidr_el1), tpidr_el2);
19681968 }
19691969
1970+static bool has_nested_virt_support(const struct arm64_cpu_capabilities *cap,
1971+ int scope)
1972+{
1973+ if (kvm_get_mode() != KVM_MODE_NV)
1974+ return false;
1975+
1976+ if (!has_cpuid_feature(cap, scope)) {
1977+ pr_warn("unavailable: %s\n", cap->desc);
1978+ return false;
1979+ }
1980+
1981+ return true;
1982+}
1983+
19701984 #ifdef CONFIG_ARM64_PAN
19711985 static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused)
19721986 {
@@ -2262,6 +2276,17 @@ static const struct arm64_cpu_capabilities arm64_features[] = { · 289 unmodified lines
22622276 .matches = runs_at_el2,
22632277 .cpu_enable = cpu_copy_el2regs,
22642278 },
2279+ {
2280+ .desc = "Nested Virtualization Support",
2281+ .capability = ARM64_HAS_NESTED_VIRT,
2282+ .type = ARM64_CPUCAP_SYSTEM_FEATURE,
2283+ .matches = has_nested_virt_support,
2284+ .sys_reg = SYS_ID_AA64MMFR2_EL1,
2285+ .sign = FTR_UNSIGNED,
2286+ .field_pos = ID_AA64MMFR2_EL1_NV_SHIFT,
2287+ .field_width = 4,
2288+ .min_field_value = ID_AA64MMFR2_EL1_NV_IMP,
2289+ },
22652290 {
22662291 .capability = ARM64_HAS_32BIT_EL0_DO_NOT_USE,
22672292 .type = ARM64_CPUCAP_SYSTEM_FEATURE,
arch/arm64/kernel/hyp-stub.S+2−84
@@ -16,30 +16,6 @@ · 15 unmodified lines
1616 #include <asm/ptrace.h>
1717 #include <asm/virt.h>
1818
19−// Warning, hardcoded register allocation
20−// This will clobber x1 and x2, and expect x1 to contain
21−// the id register value as read from the HW
22−.macro __check_override idreg, fld, width, pass, fail
23− ubfx x1, x1, #\fld, #\width
24− cbz x1, \fail
25−
26− adr_l x1, \idreg\()_override
27− ldr x2, [x1, FTR_OVR_VAL_OFFSET]
28− ldr x1, [x1, FTR_OVR_MASK_OFFSET]
29− ubfx x2, x2, #\fld, #\width
30− ubfx x1, x1, #\fld, #\width
31− cmp x1, xzr
32− and x2, x2, x1
33− csinv x2, x2, xzr, ne
34− cbnz x2, \pass
35− b \fail
36−.endm
37−
38−.macro check_override idreg, fld, pass, fail
39− mrs x1, \idreg\()_el1
40− __check_override \idreg \fld 4 \pass \fail
41−.endm
42−
4319 .text
4420 .pushsection .hyp.text, "ax"
4521
@@ -98,65 +74,7 @@ SYM_CODE_START_LOCAL(elx_sync) · 52 unmodified lines
9874 SYM_CODE_END(elx_sync)
9975
10076 SYM_CODE_START_LOCAL(__finalise_el2)
101− check_override id_aa64pfr0 ID_AA64PFR0_EL1_SVE_SHIFT .Linit_sve .Lskip_sve
102−
103−.Linit_sve: /* SVE register access */
104− mrs x0, cptr_el2 // Disable SVE traps
105− bic x0, x0, #CPTR_EL2_TZ
106− msr cptr_el2, x0
107− isb
108− mov x1, #ZCR_ELx_LEN_MASK // SVE: Enable full vector
109− msr_s SYS_ZCR_EL2, x1 // length for EL1.
110−
111−.Lskip_sve:
112− check_override id_aa64pfr1 ID_AA64PFR1_EL1_SME_SHIFT .Linit_sme .Lskip_sme
113−
114−.Linit_sme: /* SME register access and priority mapping */
115− mrs x0, cptr_el2 // Disable SME traps
116− bic x0, x0, #CPTR_EL2_TSM
117− msr cptr_el2, x0
118− isb
119−
120− mrs x1, sctlr_el2
121− orr x1, x1, #SCTLR_ELx_ENTP2 // Disable TPIDR2 traps
122− msr sctlr_el2, x1
123− isb
124−
125− mov x0, #0 // SMCR controls
126−
127− // Full FP in SM?
128− mrs_s x1, SYS_ID_AA64SMFR0_EL1
129− __check_override id_aa64smfr0 ID_AA64SMFR0_EL1_FA64_SHIFT 1 .Linit_sme_fa64 .Lskip_sme_fa64
130−
131−.Linit_sme_fa64:
132− orr x0, x0, SMCR_ELx_FA64_MASK
133−.Lskip_sme_fa64:
134−
135− // ZT0 available?
136− mrs_s x1, SYS_ID_AA64SMFR0_EL1
137− __check_override id_aa64smfr0 ID_AA64SMFR0_EL1_SMEver_SHIFT 4 .Linit_sme_zt0 .Lskip_sme_zt0
138−.Linit_sme_zt0:
139− orr x0, x0, SMCR_ELx_EZT0_MASK
140−.Lskip_sme_zt0:
141−
142− orr x0, x0, #SMCR_ELx_LEN_MASK // Enable full SME vector
143− msr_s SYS_SMCR_EL2, x0 // length for EL1.
144−
145− mrs_s x1, SYS_SMIDR_EL1 // Priority mapping supported?
146− ubfx x1, x1, #SMIDR_EL1_SMPS_SHIFT, #1
147− cbz x1, .Lskip_sme
148−
149− msr_s SYS_SMPRIMAP_EL2, xzr // Make all priorities equal
150−
151− mrs x1, id_aa64mmfr1_el1 // HCRX_EL2 present?
152− ubfx x1, x1, #ID_AA64MMFR1_EL1_HCX_SHIFT, #4
153− cbz x1, .Lskip_sme
154−
155− mrs_s x1, SYS_HCRX_EL2
156− orr x1, x1, #HCRX_EL2_SMPME_MASK // Enable priority mapping
157− msr_s SYS_HCRX_EL2, x1
158−
159−.Lskip_sme:
77+ finalise_el2_state
16078
16179 // nVHE? No way! Give me the real thing!
16280 // Sanity check: MMU *must* be off
@@ -164,7 +82,7 @@ SYM_CODE_START_LOCAL(__finalise_el2) · 1 unmodified line
16482 tbnz x1, #0, 1f
16583
16684 // Needs to be VHE capable, obviously
167− check_override id_aa64mmfr1 ID_AA64MMFR1_EL1_VH_SHIFT 2f 1f
85+ check_override id_aa64mmfr1 ID_AA64MMFR1_EL1_VH_SHIFT 2f 1f x1 x2
16886
16987 1: mov_q x0, HVC_STUB_ERR
17088 eret
arch/arm64/kvm/Kconfig+1−0
@@ -21,6 +21,7 @@ if VIRTUALIZATION · 20 unmodified lines
2121 menuconfig KVM
2222 bool "Kernel-based Virtual Machine (KVM) support"
2323 depends on HAVE_KVM
24+ select KVM_GENERIC_HARDWARE_ENABLING
2425 select MMU_NOTIFIER
2526 select PREEMPT_NOTIFIERS
2627 select HAVE_KVM_CPU_RELAX_INTERCEPT
arch/arm64/kvm/Makefile+1−1
@@ -14,7 +14,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ · 13 unmodified lines
1414 inject_fault.o va_layout.o handle_exit.o \
1515 guest.o debug.o reset.o sys_regs.o stacktrace.o \
1616 vgic-sys-reg-v3.o fpsimd.o pkvm.o \
17− arch_timer.o trng.o vmid.o \
17+ arch_timer.o trng.o vmid.o emulate-nested.o nested.o \
1818 vgic/vgic.o vgic/vgic-init.o \
1919 vgic/vgic-irqfd.o vgic/vgic-v2.o \
2020 vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/arch_timer.c+60−46
@@ -428,14 +428,17 @@ static void timer_emulate(struct arch_timer_context *ctx) · 427 unmodified lines
428428 * scheduled for the future. If the timer cannot fire at all,
429429 * then we also don't need a soft timer.
430430 */
431− if (!kvm_timer_irq_can_fire(ctx)) {
432− soft_timer_cancel(&ctx->hrtimer);
431+ if (should_fire || !kvm_timer_irq_can_fire(ctx))
433432 return;
434− }
435433
436434 soft_timer_start(&ctx->hrtimer, kvm_timer_compute_delta(ctx));
437435 }
438436
437+static void set_cntvoff(u64 cntvoff)
438+{
439+ kvm_call_hyp(__kvm_timer_set_cntvoff, cntvoff);
440+}
441+
439442 static void timer_save_state(struct arch_timer_context *ctx)
440443 {
441444 struct arch_timer_cpu *timer = vcpu_timer(ctx->vcpu);
@@ -459,6 +462,22 @@ static void timer_save_state(struct arch_timer_context *ctx) · 17 unmodified lines
459462 write_sysreg_el0(0, SYS_CNTV_CTL);
460463 isb();
461464
465+ /*
466+ * The kernel may decide to run userspace after
467+ * calling vcpu_put, so we reset cntvoff to 0 to
468+ * ensure a consistent read between user accesses to
469+ * the virtual counter and kernel access to the
470+ * physical counter of non-VHE case.
471+ *
472+ * For VHE, the virtual counter uses a fixed virtual
473+ * offset of zero, so no need to zero CNTVOFF_EL2
474+ * register, but this is actually useful when switching
475+ * between EL1/vEL2 with NV.
476+ *
477+ * Do it unconditionally, as this is either unavoidable
478+ * or dirt cheap.
479+ */
480+ set_cntvoff(0);
462481 break;
463482 case TIMER_PTIMER:
464483 timer_set_ctl(ctx, read_sysreg_el0(SYS_CNTP_CTL));
@@ -532,6 +551,7 @@ static void timer_restore_state(struct arch_timer_context *ctx) · 67 unmodified lines
532551
533552 switch (index) {
534553 case TIMER_VTIMER:
554+ set_cntvoff(timer_get_offset(ctx));
535555 write_sysreg_el0(timer_get_cval(ctx), SYS_CNTV_CVAL);
536556 isb();
537557 write_sysreg_el0(timer_get_ctl(ctx), SYS_CNTV_CTL);
@@ -552,11 +572,6 @@ out: · 14 unmodified lines
552572 local_irq_restore(flags);
553573 }
554574
555−static void set_cntvoff(u64 cntvoff)
556−{
557− kvm_call_hyp(__kvm_timer_set_cntvoff, cntvoff);
558−}
559−
560575 static inline void set_timer_irq_phys_active(struct arch_timer_context *ctx, bool active)
561576 {
562577 int r;
@@ -631,8 +646,6 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) · 68 unmodified lines
631646 kvm_timer_vcpu_load_nogic(vcpu);
632647 }
633648
634− set_cntvoff(timer_get_offset(map.direct_vtimer));
635−
636649 kvm_timer_unblocking(vcpu);
637650
638651 timer_restore_state(map.direct_vtimer);
@@ -688,15 +701,6 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu) · 49 unmodified lines
688701
689702 if (kvm_vcpu_is_blocking(vcpu))
690703 kvm_timer_blocking(vcpu);
691−
692− /*
693− * The kernel may decide to run userspace after calling vcpu_put, so
694− * we reset cntvoff to 0 to ensure a consistent read between user
695− * accesses to the virtual counter and kernel access to the physical
696− * counter of non-VHE case. For VHE, the virtual counter uses a fixed
697− * virtual offset of zero, so no need to zero CNTVOFF_EL2 register.
698− */
699− set_cntvoff(0);
700704 }
701705
702706 /*
@@ -811,10 +815,18 @@ void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu) · 108 unmodified lines
811815 ptimer->host_timer_irq_flags = host_ptimer_irq_flags;
812816 }
813817
814−static void kvm_timer_init_interrupt(void *info)
818+void kvm_timer_cpu_up(void)
815819 {
816820 enable_percpu_irq(host_vtimer_irq, host_vtimer_irq_flags);
817− enable_percpu_irq(host_ptimer_irq, host_ptimer_irq_flags);
821+ if (host_ptimer_irq)
822+ enable_percpu_irq(host_ptimer_irq, host_ptimer_irq_flags);
823+}
824+
825+void kvm_timer_cpu_down(void)
826+{
827+ disable_percpu_irq(host_vtimer_irq);
828+ if (host_ptimer_irq)
829+ disable_percpu_irq(host_ptimer_irq);
818830 }
819831
820832 int kvm_arm_timer_set_reg(struct kvm_vcpu *vcpu, u64 regid, u64 value)
@@ -926,14 +938,22 @@ u64 kvm_arm_timer_read_sysreg(struct kvm_vcpu *vcpu, · 105 unmodified lines
926938 enum kvm_arch_timers tmr,
927939 enum kvm_arch_timer_regs treg)
928940 {
941+ struct arch_timer_context *timer;
942+ struct timer_map map;
929943 u64 val;
930944
945+ get_timer_map(vcpu, &map);
946+ timer = vcpu_get_timer(vcpu, tmr);
947+
948+ if (timer == map.emul_ptimer)
949+ return kvm_arm_timer_read(vcpu, timer, treg);
950+
931951 preempt_disable();
932− kvm_timer_vcpu_put(vcpu);
952+ timer_save_state(timer);
933953
934− val = kvm_arm_timer_read(vcpu, vcpu_get_timer(vcpu, tmr), treg);
954+ val = kvm_arm_timer_read(vcpu, timer, treg);
935955
936− kvm_timer_vcpu_load(vcpu);
956+ timer_restore_state(timer);
937957 preempt_enable();
938958
939959 return val;
@@ -967,25 +987,22 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · 27 unmodified lines
967987 enum kvm_arch_timer_regs treg,
968988 u64 val)
969989 {
970− preempt_disable();
971− kvm_timer_vcpu_put(vcpu);
972−
973− kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);
974−
975− kvm_timer_vcpu_load(vcpu);
976− preempt_enable();
977−}
978−
979−static int kvm_timer_starting_cpu(unsigned int cpu)
980−{
981− kvm_timer_init_interrupt(NULL);
982− return 0;
983−}
990+ struct arch_timer_context *timer;
991+ struct timer_map map;
984992
985−static int kvm_timer_dying_cpu(unsigned int cpu)
986−{
987− disable_percpu_irq(host_vtimer_irq);
988− return 0;
993+ get_timer_map(vcpu, &map);
994+ timer = vcpu_get_timer(vcpu, tmr);
995+ if (timer == map.emul_ptimer) {
996+ soft_timer_cancel(&timer->hrtimer);
997+ kvm_arm_timer_write(vcpu, timer, treg, val);
998+ timer_emulate(timer);
999+ } else {
1000+ preempt_disable();
1001+ timer_save_state(timer);
1002+ kvm_arm_timer_write(vcpu, timer, treg, val);
1003+ timer_restore_state(timer);
1004+ preempt_enable();
1005+ }
9891006 }
9901007
9911008 static int timer_irq_set_vcpu_affinity(struct irq_data *d, void *vcpu)
@@ -1117,7 +1134,7 @@ static int kvm_irq_init(struct arch_timer_kvm_info *info) · 125 unmodified lines
11171134 return 0;
11181135 }
11191136
1120−int kvm_timer_hyp_init(bool has_gic)
1137+int __init kvm_timer_hyp_init(bool has_gic)
11211138 {
11221139 struct arch_timer_kvm_info *info;
11231140 int err;
@@ -1185,9 +1202,6 @@ int kvm_timer_hyp_init(bool has_gic) · 61 unmodified lines
11851202 goto out_free_irq;
11861203 }
11871204
1188− cpuhp_setup_state(CPUHP_AP_KVM_ARM_TIMER_STARTING,
1189− "kvm/arm/timer:starting", kvm_timer_starting_cpu,
1190− kvm_timer_dying_cpu);
11911205 return 0;
11921206 out_free_irq:
11931207 free_percpu_irq(host_vtimer_irq, kvm_get_running_vcpus());
arch/arm64/kvm/arm.c+60−49
@@ -63,16 +63,6 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu) · 62 unmodified lines
6363 return kvm_vcpu_exiting_guest_mode(vcpu) == IN_GUEST_MODE;
6464 }
6565
66−int kvm_arch_hardware_setup(void *opaque)
67−{
68− return 0;
69−}
70−
71−int kvm_arch_check_processor_compat(void *opaque)
72−{
73− return 0;
74−}
75−
7666 int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
7767 struct kvm_enable_cap *cap)
7868 {
@@ -146,7 +136,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) · 67 unmodified lines
146136 if (ret)
147137 goto err_unshare_kvm;
148138
149− if (!zalloc_cpumask_var(&kvm->arch.supported_cpus, GFP_KERNEL)) {
139+ if (!zalloc_cpumask_var(&kvm->arch.supported_cpus, GFP_KERNEL_ACCOUNT)) {
150140 ret = -ENOMEM;
151141 goto err_unshare_kvm;
152142 }
@@ -1539,7 +1529,7 @@ static int kvm_init_vector_slots(void) · 1386 unmodified lines
15391529 return 0;
15401530 }
15411531
1542−static void cpu_prepare_hyp_mode(int cpu, u32 hyp_va_bits)
1532+static void __init cpu_prepare_hyp_mode(int cpu, u32 hyp_va_bits)
15431533 {
15441534 struct kvm_nvhe_init_params *params = per_cpu_ptr_nvhe_sym(kvm_init_params, cpu);
15451535 unsigned long tcr;
@@ -1682,7 +1672,15 @@ static void _kvm_arch_hardware_enable(void *discard) · 136 unmodified lines
16821672
16831673 int kvm_arch_hardware_enable(void)
16841674 {
1675+ int was_enabled = __this_cpu_read(kvm_arm_hardware_enabled);
1676+
16851677 _kvm_arch_hardware_enable(NULL);
1678+
1679+ if (!was_enabled) {
1680+ kvm_vgic_cpu_up();
1681+ kvm_timer_cpu_up();
1682+ }
1683+
16861684 return 0;
16871685 }
16881686
@@ -1696,6 +1694,11 @@ static void _kvm_arch_hardware_disable(void *discard) · 7 unmodified lines
16961694
16971695 void kvm_arch_hardware_disable(void)
16981696 {
1697+ if (__this_cpu_read(kvm_arm_hardware_enabled)) {
1698+ kvm_timer_cpu_down();
1699+ kvm_vgic_cpu_down();
1700+ }
1701+
16991702 if (!is_protected_kvm_enabled())
17001703 _kvm_arch_hardware_disable(NULL);
17011704 }
@@ -1738,26 +1741,26 @@ static struct notifier_block hyp_init_cpu_pm_nb = { · 36 unmodified lines
17381741 .notifier_call = hyp_init_cpu_pm_notifier,
17391742 };
17401743
1741−static void hyp_cpu_pm_init(void)
1744+static void __init hyp_cpu_pm_init(void)
17421745 {
17431746 if (!is_protected_kvm_enabled())
17441747 cpu_pm_register_notifier(&hyp_init_cpu_pm_nb);
17451748 }
1746−static void hyp_cpu_pm_exit(void)
1749+static void __init hyp_cpu_pm_exit(void)
17471750 {
17481751 if (!is_protected_kvm_enabled())
17491752 cpu_pm_unregister_notifier(&hyp_init_cpu_pm_nb);
17501753 }
17511754 #else
1752−static inline void hyp_cpu_pm_init(void)
1755+static inline void __init hyp_cpu_pm_init(void)
17531756 {
17541757 }
1755−static inline void hyp_cpu_pm_exit(void)
1758+static inline void __init hyp_cpu_pm_exit(void)
17561759 {
17571760 }
17581761 #endif
17591762
1760−static void init_cpu_logical_map(void)
1763+static void __init init_cpu_logical_map(void)
17611764 {
17621765 unsigned int cpu;
17631766
@@ -1774,7 +1777,7 @@ static void init_cpu_logical_map(void) · 10 unmodified lines
17741777 #define init_psci_0_1_impl_state(config, what) \
17751778 config.psci_0_1_ ## what ## _implemented = psci_ops.what
17761779
1777−static bool init_psci_relay(void)
1780+static bool __init init_psci_relay(void)
17781781 {
17791782 /*
17801783 * If PSCI has not been initialized, protected KVM cannot install
@@ -1797,7 +1800,7 @@ static bool init_psci_relay(void) · 16 unmodified lines
17971800 return true;
17981801 }
17991802
1800−static int init_subsystems(void)
1803+static int __init init_subsystems(void)
18011804 {
18021805 int err = 0;
18031806
@@ -1838,13 +1841,22 @@ static int init_subsystems(void) · 34 unmodified lines
18381841 kvm_register_perf_callbacks(NULL);
18391842
18401843 out:
1844+ if (err)
1845+ hyp_cpu_pm_exit();
1846+
18411847 if (err || !is_protected_kvm_enabled())
18421848 on_each_cpu(_kvm_arch_hardware_disable, NULL, 1);
18431849
18441850 return err;
18451851 }
18461852
1847−static void teardown_hyp_mode(void)
1853+static void __init teardown_subsystems(void)
1854+{
1855+ kvm_unregister_perf_callbacks();
1856+ hyp_cpu_pm_exit();
1857+}
1858+
1859+static void __init teardown_hyp_mode(void)
18481860 {
18491861 int cpu;
18501862
@@ -1855,7 +1867,7 @@ static void teardown_hyp_mode(void) · 4 unmodified lines
18551867 }
18561868 }
18571869
1858−static int do_pkvm_init(u32 hyp_va_bits)
1870+static int __init do_pkvm_init(u32 hyp_va_bits)
18591871 {
18601872 void *per_cpu_base = kvm_ksym_ref(kvm_nvhe_sym(kvm_arm_hyp_percpu_base));
18611873 int ret;
@@ -1887,11 +1899,12 @@ static void kvm_hyp_init_symbols(void) · 25 unmodified lines
18871899 kvm_nvhe_sym(id_aa64mmfr0_el1_sys_val) = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
18881900 kvm_nvhe_sym(id_aa64mmfr1_el1_sys_val) = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
18891901 kvm_nvhe_sym(id_aa64mmfr2_el1_sys_val) = read_sanitised_ftr_reg(SYS_ID_AA64MMFR2_EL1);
1902+ kvm_nvhe_sym(id_aa64smfr0_el1_sys_val) = read_sanitised_ftr_reg(SYS_ID_AA64SMFR0_EL1);
18901903 kvm_nvhe_sym(__icache_flags) = __icache_flags;
18911904 kvm_nvhe_sym(kvm_arm_vmid_bits) = kvm_arm_vmid_bits;
18921905 }
18931906
1894−static int kvm_hyp_init_protection(u32 hyp_va_bits)
1907+static int __init kvm_hyp_init_protection(u32 hyp_va_bits)
18951908 {
18961909 void *addr = phys_to_virt(hyp_mem_base);
18971910 int ret;
@@ -1909,10 +1922,8 @@ static int kvm_hyp_init_protection(u32 hyp_va_bits) · 11 unmodified lines
19091922 return 0;
19101923 }
19111924
1912−/**
1913− * Inits Hyp-mode on all online CPUs
1914− */
1915−static int init_hyp_mode(void)
1925+/* Inits Hyp-mode on all online CPUs */
1926+static int __init init_hyp_mode(void)
19161927 {
19171928 u32 hyp_va_bits;
19181929 int cpu;
@@ -2094,7 +2105,7 @@ out_err: · 175 unmodified lines
20942105 return err;
20952106 }
20962107
2097−static void _kvm_host_prot_finalize(void *arg)
2108+static void __init _kvm_host_prot_finalize(void *arg)
20982109 {
20992110 int *err = arg;
21002111
@@ -2102,7 +2113,7 @@ static void _kvm_host_prot_finalize(void *arg) · 1 unmodified line
21022113 WRITE_ONCE(*err, -EINVAL);
21032114 }
21042115
2105−static int pkvm_drop_host_privileges(void)
2116+static int __init pkvm_drop_host_privileges(void)
21062117 {
21072118 int ret = 0;
21082119
@@ -2115,7 +2126,7 @@ static int pkvm_drop_host_privileges(void) · 6 unmodified lines
21152126 return ret;
21162127 }
21172128
2118−static int finalize_hyp_mode(void)
2129+static int __init finalize_hyp_mode(void)
21192130 {
21202131 if (!is_protected_kvm_enabled())
21212132 return 0;
@@ -2187,10 +2198,8 @@ void kvm_arch_irq_bypass_start(struct irq_bypass_consumer *cons) · 65 unmodified lines
21872198 kvm_arm_resume_guest(irqfd->kvm);
21882199 }
21892200
2190−/**
2191− * Initialize Hyp-mode and memory mappings on all CPUs.
2192− */
2193−int kvm_arch_init(void *opaque)
2201+/* Initialize Hyp-mode and memory mappings on all CPUs */
2202+static __init int kvm_arm_init(void)
21942203 {
21952204 int err;
21962205 bool in_hyp_mode;
@@ -2241,7 +2250,7 @@ int kvm_arch_init(void *opaque) · 44 unmodified lines
22412250 err = kvm_init_vector_slots();
22422251 if (err) {
22432252 kvm_err("Cannot initialise vector slots\n");
2244− goto out_err;
2253+ goto out_hyp;
22452254 }
22462255
22472256 err = init_subsystems();
@@ -2252,7 +2261,7 @@ int kvm_arch_init(void *opaque) · 4 unmodified lines
22522261 err = finalize_hyp_mode();
22532262 if (err) {
22542263 kvm_err("Failed to finalize Hyp protection\n");
2255− goto out_hyp;
2264+ goto out_subs;
22562265 }
22572266 }
22582267
@@ -2264,10 +2273,19 @@ int kvm_arch_init(void *opaque) · 5 unmodified lines
22642273 kvm_info("Hyp mode initialized successfully\n");
22652274 }
22662275
2276+ /*
2277+ * FIXME: Do something reasonable if kvm_init() fails after pKVM
2278+ * hypervisor protection is finalized.
2279+ */
2280+ err = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
2281+ if (err)
2282+ goto out_subs;
2283+
22672284 return 0;
22682285
2286+out_subs:
2287+ teardown_subsystems();
22692288 out_hyp:
2270− hyp_cpu_pm_exit();
22712289 if (!in_hyp_mode)
22722290 teardown_hyp_mode();
22732291 out_err:
@@ -2275,12 +2293,6 @@ out_err: · 1 unmodified line
22752293 return err;
22762294 }
22772295
2278−/* NOP: Compiling as a module not supported */
2279−void kvm_arch_exit(void)
2280−{
2281− kvm_unregister_perf_callbacks();
2282−}
2283−
22842296 static int __init early_kvm_mode_cfg(char *arg)
22852297 {
22862298 if (!arg)
@@ -2310,6 +2322,11 @@ static int __init early_kvm_mode_cfg(char *arg) · 23 unmodified lines
23102322 return 0;
23112323 }
23122324
2325+ if (strcmp(arg, "nested") == 0 && !WARN_ON(!is_kernel_in_hyp_mode())) {
2326+ kvm_mode = KVM_MODE_NV;
2327+ return 0;
2328+ }
2329+
23132330 return -EINVAL;
23142331 }
23152332 early_param("kvm-arm.mode", early_kvm_mode_cfg);
@@ -2319,10 +2336,4 @@ enum kvm_mode kvm_get_mode(void) · 3 unmodified lines
23192336 return kvm_mode;
23202337 }
23212338
2322−static int arm_init(void)
2323−{
2324− int rc = kvm_init(NULL, sizeof(struct kvm_vcpu), 0, THIS_MODULE);
2325− return rc;
2326−}
2327−
2328−module_init(arm_init);
2339+module_init(kvm_arm_init);
arch/arm64/kvm/emulate-nested.c+203−0
@@ -0,0 +1,203 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/*
3+ * Copyright (C) 2016 - Linaro and Columbia University
4+ * Author: Jintack Lim <jintack.lim@linaro.org>
5+ */
6+
7+#include <linux/kvm.h>
8+#include <linux/kvm_host.h>
9+
10+#include <asm/kvm_emulate.h>
11+#include <asm/kvm_nested.h>
12+
13+#include "hyp/include/hyp/adjust_pc.h"
14+
15+#include "trace.h"
16+
17+static u64 kvm_check_illegal_exception_return(struct kvm_vcpu *vcpu, u64 spsr)
18+{
19+ u64 mode = spsr & PSR_MODE_MASK;
20+
21+ /*
22+ * Possible causes for an Illegal Exception Return from EL2:
23+ * - trying to return to EL3
24+ * - trying to return to an illegal M value
25+ * - trying to return to a 32bit EL
26+ * - trying to return to EL1 with HCR_EL2.TGE set
27+ */
28+ if (mode == PSR_MODE_EL3t || mode == PSR_MODE_EL3h ||
29+ mode == 0b00001 || (mode & BIT(1)) ||
30+ (spsr & PSR_MODE32_BIT) ||
31+ (vcpu_el2_tge_is_set(vcpu) && (mode == PSR_MODE_EL1t ||
32+ mode == PSR_MODE_EL1h))) {
33+ /*
34+ * The guest is playing with our nerves. Preserve EL, SP,
35+ * masks, flags from the existing PSTATE, and set IL.
36+ * The HW will then generate an Illegal State Exception
37+ * immediately after ERET.
38+ */
39+ spsr = *vcpu_cpsr(vcpu);
40+
41+ spsr &= (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT |
42+ PSR_N_BIT | PSR_Z_BIT | PSR_C_BIT | PSR_V_BIT |
43+ PSR_MODE_MASK | PSR_MODE32_BIT);
44+ spsr |= PSR_IL_BIT;
45+ }
46+
47+ return spsr;
48+}
49+
50+void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
51+{
52+ u64 spsr, elr, mode;
53+ bool direct_eret;
54+
55+ /*
56+ * Going through the whole put/load motions is a waste of time
57+ * if this is a VHE guest hypervisor returning to its own
58+ * userspace, or the hypervisor performing a local exception
59+ * return. No need to save/restore registers, no need to
60+ * switch S2 MMU. Just do the canonical ERET.
61+ */
62+ spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2);
63+ spsr = kvm_check_illegal_exception_return(vcpu, spsr);
64+
65+ mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT);
66+
67+ direct_eret = (mode == PSR_MODE_EL0t &&
68+ vcpu_el2_e2h_is_set(vcpu) &&
69+ vcpu_el2_tge_is_set(vcpu));
70+ direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);
71+
72+ if (direct_eret) {
73+ *vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2);
74+ *vcpu_cpsr(vcpu) = spsr;
75+ trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr);
76+ return;
77+ }
78+
79+ preempt_disable();
80+ kvm_arch_vcpu_put(vcpu);
81+
82+ elr = __vcpu_sys_reg(vcpu, ELR_EL2);
83+
84+ trace_kvm_nested_eret(vcpu, elr, spsr);
85+
86+ /*
87+ * Note that the current exception level is always the virtual EL2,
88+ * since we set HCR_EL2.NV bit only when entering the virtual EL2.
89+ */
90+ *vcpu_pc(vcpu) = elr;
91+ *vcpu_cpsr(vcpu) = spsr;
92+
93+ kvm_arch_vcpu_load(vcpu, smp_processor_id());
94+ preempt_enable();
95+}
96+
97+static void kvm_inject_el2_exception(struct kvm_vcpu *vcpu, u64 esr_el2,
98+ enum exception_type type)
99+{
100+ trace_kvm_inject_nested_exception(vcpu, esr_el2, type);
101+
102+ switch (type) {
103+ case except_type_sync:
104+ kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_SYNC);
105+ vcpu_write_sys_reg(vcpu, esr_el2, ESR_EL2);
106+ break;
107+ case except_type_irq:
108+ kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_IRQ);
109+ break;
110+ default:
111+ WARN_ONCE(1, "Unsupported EL2 exception injection %d\n", type);
112+ }
113+}
114+
115+/*
116+ * Emulate taking an exception to EL2.
117+ * See ARM ARM J8.1.2 AArch64.TakeException()
118+ */
119+static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2,
120+ enum exception_type type)
121+{
122+ u64 pstate, mode;
123+ bool direct_inject;
124+
125+ if (!vcpu_has_nv(vcpu)) {
126+ kvm_err("Unexpected call to %s for the non-nesting configuration\n",
127+ __func__);
128+ return -EINVAL;
129+ }
130+
131+ /*
132+ * As for ERET, we can avoid doing too much on the injection path by
133+ * checking that we either took the exception from a VHE host
134+ * userspace or from vEL2. In these cases, there is no change in
135+ * translation regime (or anything else), so let's do as little as
136+ * possible.
137+ */
138+ pstate = *vcpu_cpsr(vcpu);
139+ mode = pstate & (PSR_MODE_MASK | PSR_MODE32_BIT);
140+
141+ direct_inject = (mode == PSR_MODE_EL0t &&
142+ vcpu_el2_e2h_is_set(vcpu) &&
143+ vcpu_el2_tge_is_set(vcpu));
144+ direct_inject |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);
145+
146+ if (direct_inject) {
147+ kvm_inject_el2_exception(vcpu, esr_el2, type);
148+ return 1;
149+ }
150+
151+ preempt_disable();
152+
153+ /*
154+ * We may have an exception or PC update in the EL0/EL1 context.
155+ * Commit it before entering EL2.
156+ */
157+ __kvm_adjust_pc(vcpu);
158+
159+ kvm_arch_vcpu_put(vcpu);
160+
161+ kvm_inject_el2_exception(vcpu, esr_el2, type);
162+
163+ /*
164+ * A hard requirement is that a switch between EL1 and EL2
165+ * contexts has to happen between a put/load, so that we can
166+ * pick the correct timer and interrupt configuration, among
167+ * other things.
168+ *
169+ * Make sure the exception actually took place before we load
170+ * the new context.
171+ */
172+ __kvm_adjust_pc(vcpu);
173+
174+ kvm_arch_vcpu_load(vcpu, smp_processor_id());
175+ preempt_enable();
176+
177+ return 1;
178+}
179+
180+int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2)
181+{
182+ return kvm_inject_nested(vcpu, esr_el2, except_type_sync);
183+}
184+
185+int kvm_inject_nested_irq(struct kvm_vcpu *vcpu)
186+{
187+ /*
188+ * Do not inject an irq if the:
189+ * - Current exception level is EL2, and
190+ * - virtual HCR_EL2.TGE == 0
191+ * - virtual HCR_EL2.IMO == 0
192+ *
193+ * See Table D1-17 "Physical interrupt target and masking when EL3 is
194+ * not implemented and EL2 is implemented" in ARM DDI 0487C.a.
195+ */
196+
197+ if (vcpu_is_el2(vcpu) && !vcpu_el2_tge_is_set(vcpu) &&
198+ !(__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_IMO))
199+ return 1;
200+
201+ /* esr_el2 value doesn't matter for exits due to irqs. */
202+ return kvm_inject_nested(vcpu, 0, except_type_irq);
203+}
arch/arm64/kvm/fpsimd.c+1−0
@@ -184,6 +184,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) · 183 unmodified lines
184184 sysreg_clear_set(CPACR_EL1,
185185 CPACR_EL1_SMEN_EL0EN,
186186 CPACR_EL1_SMEN_EL1EN);
187+ isb();
187188 }
188189
189190 if (vcpu->arch.fp_state == FP_STATE_GUEST_OWNED) {
arch/arm64/kvm/guest.c+6−0
@@ -24,6 +24,7 @@ · 23 unmodified lines
2424 #include <asm/fpsimd.h>
2525 #include <asm/kvm.h>
2626 #include <asm/kvm_emulate.h>
27+#include <asm/kvm_nested.h>
2728 #include <asm/sigcontext.h>
2829
2930 #include "trace.h"
@@ -253,6 +254,11 @@ static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) · 223 unmodified lines
253254 if (!vcpu_el1_is_32bit(vcpu))
254255 return -EINVAL;
255256 break;
257+ case PSR_MODE_EL2h:
258+ case PSR_MODE_EL2t:
259+ if (!vcpu_has_nv(vcpu))
260+ return -EINVAL;
261+ fallthrough;
256262 case PSR_MODE_EL0t:
257263 case PSR_MODE_EL1t:
258264 case PSR_MODE_EL1h:
arch/arm64/kvm/handle_exit.c+45−2
@@ -16,6 +16,7 @@ · 15 unmodified lines
1616 #include <asm/kvm_asm.h>
1717 #include <asm/kvm_emulate.h>
1818 #include <asm/kvm_mmu.h>
19+#include <asm/kvm_nested.h>
1920 #include <asm/debug-monitors.h>
2021 #include <asm/stacktrace/nvhe.h>
2122 #include <asm/traps.h>
@@ -41,6 +42,16 @@ static int handle_hvc(struct kvm_vcpu *vcpu) · 19 unmodified lines
4142 kvm_vcpu_hvc_get_imm(vcpu));
4243 vcpu->stat.hvc_exit_stat++;
4344
45+ /* Forward hvc instructions to the virtual EL2 if the guest has EL2. */
46+ if (vcpu_has_nv(vcpu)) {
47+ if (vcpu_read_sys_reg(vcpu, HCR_EL2) & HCR_HCD)
48+ kvm_inject_undefined(vcpu);
49+ else
50+ kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu));
51+
52+ return 1;
53+ }
54+
4455 ret = kvm_hvc_call_handler(vcpu);
4556 if (ret < 0) {
4657 vcpu_set_reg(vcpu, 0, ~0UL);
@@ -52,6 +63,8 @@ static int handle_hvc(struct kvm_vcpu *vcpu) · 5 unmodified lines
5263
5364 static int handle_smc(struct kvm_vcpu *vcpu)
5465 {
66+ int ret;
67+
5568 /*
5669 * "If an SMC instruction executed at Non-secure EL1 is
5770 * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a
@@ -59,10 +72,30 @@ static int handle_smc(struct kvm_vcpu *vcpu) · 1 unmodified line
5972 *
6073 * We need to advance the PC after the trap, as it would
6174 * otherwise return to the same address...
75+ *
76+ * Only handle SMCs from the virtual EL2 with an immediate of zero and
77+ * skip it otherwise.
6278 */
63− vcpu_set_reg(vcpu, 0, ~0UL);
79+ if (!vcpu_is_el2(vcpu) || kvm_vcpu_hvc_get_imm(vcpu)) {
80+ vcpu_set_reg(vcpu, 0, ~0UL);
81+ kvm_incr_pc(vcpu);
82+ return 1;
83+ }
84+
85+ /*
86+ * If imm is zero then it is likely an SMCCC call.
87+ *
88+ * Note that on ARMv8.3, even if EL3 is not implemented, SMC executed
89+ * at Non-secure EL1 is trapped to EL2 if HCR_EL2.TSC==1, rather than
90+ * being treated as UNDEFINED.
91+ */
92+ ret = kvm_hvc_call_handler(vcpu);
93+ if (ret < 0)
94+ vcpu_set_reg(vcpu, 0, ~0UL);
95+
6496 kvm_incr_pc(vcpu);
65− return 1;
97+
98+ return ret;
6699 }
67100
68101 /*
@@ -196,6 +229,15 @@ static int kvm_handle_ptrauth(struct kvm_vcpu *vcpu) · 127 unmodified lines
196229 return 1;
197230 }
198231
232+static int kvm_handle_eret(struct kvm_vcpu *vcpu)
233+{
234+ if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_ERET_ISS_ERET)
235+ return kvm_handle_ptrauth(vcpu);
236+
237+ kvm_emulate_nested_eret(vcpu);
238+ return 1;
239+}
240+
199241 static exit_handle_fn arm_exit_handlers[] = {
200242 [0 ... ESR_ELx_EC_MAX] = kvm_handle_unknown_ec,
201243 [ESR_ELx_EC_WFx] = kvm_handle_wfx,
@@ -211,6 +253,7 @@ static exit_handle_fn arm_exit_handlers[] = { · 9 unmodified lines
211253 [ESR_ELx_EC_SMC64] = handle_smc,
212254 [ESR_ELx_EC_SYS64] = kvm_handle_sys_reg,
213255 [ESR_ELx_EC_SVE] = handle_sve,
256+ [ESR_ELx_EC_ERET] = kvm_handle_eret,
214257 [ESR_ELx_EC_IABT_LOW] = kvm_handle_guest_abort,
215258 [ESR_ELx_EC_DABT_LOW] = kvm_handle_guest_abort,
216259 [ESR_ELx_EC_SOFTSTP_LOW]= kvm_handle_guest_debug,
arch/arm64/kvm/hyp/exception.c+36−12
@@ -14,6 +14,7 @@ · 13 unmodified lines
1414 #include <linux/kvm_host.h>
1515 #include <asm/kvm_emulate.h>
1616 #include <asm/kvm_mmu.h>
17+#include <asm/kvm_nested.h>
1718
1819 #if !defined (__KVM_NVHE_HYPERVISOR__) && !defined (__KVM_VHE_HYPERVISOR__)
1920 #error Hypervisor code only!
@@ -23,7 +24,9 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) · 3 unmodified lines
2324 {
2425 u64 val;
2526
26− if (__vcpu_read_sys_reg_from_cpu(reg, &val))
27+ if (unlikely(vcpu_has_nv(vcpu)))
28+ return vcpu_read_sys_reg(vcpu, reg);
29+ else if (__vcpu_read_sys_reg_from_cpu(reg, &val))
2730 return val;
2831
2932 return __vcpu_sys_reg(vcpu, reg);
@@ -31,18 +34,25 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) · 1 unmodified line
3134
3235 static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
3336 {
34− if (__vcpu_write_sys_reg_to_cpu(val, reg))
35− return;
36−
37− __vcpu_sys_reg(vcpu, reg) = val;
37+ if (unlikely(vcpu_has_nv(vcpu)))
38+ vcpu_write_sys_reg(vcpu, val, reg);
39+ else if (!__vcpu_write_sys_reg_to_cpu(val, reg))
40+ __vcpu_sys_reg(vcpu, reg) = val;
3841 }
3942
40−static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, u64 val)
43+static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long target_mode,
44+ u64 val)
4145 {
42− if (has_vhe())
46+ if (unlikely(vcpu_has_nv(vcpu))) {
47+ if (target_mode == PSR_MODE_EL1h)
48+ vcpu_write_sys_reg(vcpu, val, SPSR_EL1);
49+ else
50+ vcpu_write_sys_reg(vcpu, val, SPSR_EL2);
51+ } else if (has_vhe()) {
4352 write_sysreg_el1(val, SYS_SPSR);
44− else
53+ } else {
4554 __vcpu_sys_reg(vcpu, SPSR_EL1) = val;
55+ }
4656 }
4757
4858 static void __vcpu_write_spsr_abt(struct kvm_vcpu *vcpu, u64 val)
@@ -101,6 +111,11 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, · 52 unmodified lines
101111 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
102112 __vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
103113 break;
114+ case PSR_MODE_EL2h:
115+ vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL2);
116+ sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL2);
117+ __vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2);
118+ break;
104119 default:
105120 /* Don't do that */
106121 BUG();
@@ -153,7 +168,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, · 46 unmodified lines
153168 new |= target_mode;
154169
155170 *vcpu_cpsr(vcpu) = new;
156− __vcpu_write_spsr(vcpu, old);
171+ __vcpu_write_spsr(vcpu, target_mode, old);
157172 }
158173
159174 /*
@@ -323,11 +338,20 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu) · 163 unmodified lines
323338 case unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC):
324339 enter_exception64(vcpu, PSR_MODE_EL1h, except_type_sync);
325340 break;
341+
342+ case unpack_vcpu_flag(EXCEPT_AA64_EL2_SYNC):
343+ enter_exception64(vcpu, PSR_MODE_EL2h, except_type_sync);
344+ break;
345+
346+ case unpack_vcpu_flag(EXCEPT_AA64_EL2_IRQ):
347+ enter_exception64(vcpu, PSR_MODE_EL2h, except_type_irq);
348+ break;
349+
326350 default:
327351 /*
328− * Only EL1_SYNC makes sense so far, EL2_{SYNC,IRQ}
329− * will be implemented at some point. Everything
330− * else gets silently ignored.
352+ * Only EL1_SYNC and EL2_{SYNC,IRQ} makes
353+ * sense so far. Everything else gets silently
354+ * ignored.
331355 */
332356 break;
333357 }
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h+18−3
@@ -39,7 +39,6 @@ static inline bool ctxt_has_mte(struct kvm_cpu_context *ctxt) · 38 unmodified lines
3939
4040 static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
4141 {
42− ctxt_sys_reg(ctxt, CSSELR_EL1) = read_sysreg(csselr_el1);
4342 ctxt_sys_reg(ctxt, SCTLR_EL1) = read_sysreg_el1(SYS_SCTLR);
4443 ctxt_sys_reg(ctxt, CPACR_EL1) = read_sysreg_el1(SYS_CPACR);
4544 ctxt_sys_reg(ctxt, TTBR0_EL1) = read_sysreg_el1(SYS_TTBR0);
@@ -95,7 +94,6 @@ static inline void __sysreg_restore_user_state(struct kvm_cpu_context *ctxt) · 49 unmodified lines
9594 static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
9695 {
9796 write_sysreg(ctxt_sys_reg(ctxt, MPIDR_EL1), vmpidr_el2);
98− write_sysreg(ctxt_sys_reg(ctxt, CSSELR_EL1), csselr_el1);
9997
10098 if (has_vhe() ||
10199 !cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {
@@ -156,9 +154,26 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) · 54 unmodified lines
156154 write_sysreg_el1(ctxt_sys_reg(ctxt, SPSR_EL1), SYS_SPSR);
157155 }
158156
157+/* Read the VCPU state's PSTATE, but translate (v)EL2 to EL1. */
158+static inline u64 to_hw_pstate(const struct kvm_cpu_context *ctxt)
159+{
160+ u64 mode = ctxt->regs.pstate & (PSR_MODE_MASK | PSR_MODE32_BIT);
161+
162+ switch (mode) {
163+ case PSR_MODE_EL2t:
164+ mode = PSR_MODE_EL1t;
165+ break;
166+ case PSR_MODE_EL2h:
167+ mode = PSR_MODE_EL1h;
168+ break;
169+ }
170+
171+ return (ctxt->regs.pstate & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode;
172+}
173+
159174 static inline void __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt)
160175 {
161− u64 pstate = ctxt->regs.pstate;
176+ u64 pstate = to_hw_pstate(ctxt);
162177 u64 mode = pstate & PSR_AA32_MODE_MASK;
163178
164179 /*
arch/arm64/kvm/hyp/nvhe/hyp-init.S+1−0
@@ -183,6 +183,7 @@ SYM_CODE_START_LOCAL(__kvm_hyp_init_cpu) · 182 unmodified lines
183183
184184 /* Initialize EL2 CPU state to sane values. */
185185 init_el2_state // Clobbers x0..x2
186+ finalise_el2_state
186187
187188 /* Enable MMU, set vectors and stack. */
188189 mov x0, x28
arch/arm64/kvm/hyp/nvhe/sys_regs.c+1−0
@@ -26,6 +26,7 @@ u64 id_aa64isar2_el1_sys_val; · 25 unmodified lines
2626 u64 id_aa64mmfr0_el1_sys_val;
2727 u64 id_aa64mmfr1_el1_sys_val;
2828 u64 id_aa64mmfr2_el1_sys_val;
29+u64 id_aa64smfr0_el1_sys_val;
2930
3031 /*
3132 * Inject an unknown/undefined exception to an AArch64 guest while most of its
arch/arm64/kvm/hyp/pgtable.c+37−6
@@ -168,6 +168,25 @@ static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, · 167 unmodified lines
168168 return walker->cb(ctx, visit);
169169 }
170170
171+static bool kvm_pgtable_walk_continue(const struct kvm_pgtable_walker *walker,
172+ int r)
173+{
174+ /*
175+ * Visitor callbacks return EAGAIN when the conditions that led to a
176+ * fault are no longer reflected in the page tables due to a race to
177+ * update a PTE. In the context of a fault handler this is interpreted
178+ * as a signal to retry guest execution.
179+ *
180+ * Ignore the return code altogether for walkers outside a fault handler
181+ * (e.g. write protecting a range of memory) and chug along with the
182+ * page table walk.
183+ */
184+ if (r == -EAGAIN)
185+ return !(walker->flags & KVM_PGTABLE_WALK_HANDLE_FAULT);
186+
187+ return !r;
188+}
189+
171190 static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data,
172191 struct kvm_pgtable_mm_ops *mm_ops, kvm_pteref_t pgtable, u32 level);
173192
@@ -200,7 +219,7 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, · 26 unmodified lines
200219 table = kvm_pte_table(ctx.old, level);
201220 }
202221
203− if (ret)
222+ if (!kvm_pgtable_walk_continue(data->walker, ret))
204223 goto out;
205224
206225 if (!table) {
@@ -211,13 +230,16 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, · 4 unmodified lines
211230
212231 childp = (kvm_pteref_t)kvm_pte_follow(ctx.old, mm_ops);
213232 ret = __kvm_pgtable_walk(data, mm_ops, childp, level + 1);
214− if (ret)
233+ if (!kvm_pgtable_walk_continue(data->walker, ret))
215234 goto out;
216235
217236 if (ctx.flags & KVM_PGTABLE_WALK_TABLE_POST)
218237 ret = kvm_pgtable_visitor_cb(data, &ctx, KVM_PGTABLE_WALK_TABLE_POST);
219238
220239 out:
240+ if (kvm_pgtable_walk_continue(data->walker, ret))
241+ return 0;
242+
221243 return ret;
222244 }
223245
@@ -584,12 +606,14 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift) · 360 unmodified lines
584606 lvls = 2;
585607 vtcr |= VTCR_EL2_LVLS_TO_SL0(lvls);
586608
609+#ifdef CONFIG_ARM64_HW_AFDBM
587610 /*
588611 * Enable the Hardware Access Flag management, unconditionally
589612 * on all CPUs. The features is RES0 on CPUs without the support
590613 * and must be ignored by the CPUs.
591614 */
592615 vtcr |= VTCR_EL2_HA;
616+#endif /* CONFIG_ARM64_HW_AFDBM */
593617
594618 /* Set the vmid bits */
595619 vtcr |= (get_vmid_bits(mmfr1) == 16) ?
@@ -1026,7 +1050,7 @@ static int stage2_attr_walker(const struct kvm_pgtable_visit_ctx *ctx, · 430 unmodified lines
10261050 struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops;
10271051
10281052 if (!kvm_pte_valid(ctx->old))
1029− return 0;
1053+ return -EAGAIN;
10301054
10311055 data->level = ctx->level;
10321056 data->pte = pte;
@@ -1094,9 +1118,15 @@ int kvm_pgtable_stage2_wrprotect(struct kvm_pgtable *pgt, u64 addr, u64 size) · 61 unmodified lines
10941118 kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr)
10951119 {
10961120 kvm_pte_t pte = 0;
1097− stage2_update_leaf_attrs(pgt, addr, 1, KVM_PTE_LEAF_ATTR_LO_S2_AF, 0,
1098− &pte, NULL, 0);
1099− dsb(ishst);
1121+ int ret;
1122+
1123+ ret = stage2_update_leaf_attrs(pgt, addr, 1, KVM_PTE_LEAF_ATTR_LO_S2_AF, 0,
1124+ &pte, NULL,
1125+ KVM_PGTABLE_WALK_HANDLE_FAULT |
1126+ KVM_PGTABLE_WALK_SHARED);
1127+ if (!ret)
1128+ dsb(ishst);
1129+
11001130 return pte;
11011131 }
11021132
@@ -1141,6 +1171,7 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, · 38 unmodified lines
11411171 clr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;
11421172
11431173 ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level,
1174+ KVM_PGTABLE_WALK_HANDLE_FAULT |
11441175 KVM_PGTABLE_WALK_SHARED);
11451176 if (!ret)
11461177 kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, pgt->mmu, addr, level);
arch/arm64/kvm/hyp/vhe/switch.c+25−1
@@ -40,7 +40,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu) · 39 unmodified lines
4040 ___activate_traps(vcpu);
4141
4242 val = read_sysreg(cpacr_el1);
43− val |= CPACR_EL1_TTA;
43+ val |= CPACR_ELx_TTA;
4444 val &= ~(CPACR_EL1_ZEN_EL0EN | CPACR_EL1_ZEN_EL1EN |
4545 CPACR_EL1_SMEN_EL0EN | CPACR_EL1_SMEN_EL1EN);
4646
@@ -120,6 +120,25 @@ static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu) · 73 unmodified lines
120120
121121 static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code)
122122 {
123+ /*
124+ * If we were in HYP context on entry, adjust the PSTATE view
125+ * so that the usual helpers work correctly.
126+ */
127+ if (unlikely(vcpu_get_flag(vcpu, VCPU_HYP_CONTEXT))) {
128+ u64 mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT);
129+
130+ switch (mode) {
131+ case PSR_MODE_EL1t:
132+ mode = PSR_MODE_EL2t;
133+ break;
134+ case PSR_MODE_EL1h:
135+ mode = PSR_MODE_EL2h;
136+ break;
137+ }
138+
139+ *vcpu_cpsr(vcpu) &= ~(PSR_MODE_MASK | PSR_MODE32_BIT);
140+ *vcpu_cpsr(vcpu) |= mode;
141+ }
123142 }
124143
125144 /* Switch to the guest for VHE systems running in EL2 */
@@ -154,6 +173,11 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) · 28 unmodified lines
154173 sysreg_restore_guest_state_vhe(guest_ctxt);
155174 __debug_switch_to_guest(vcpu);
156175
176+ if (is_hyp_ctxt(vcpu))
177+ vcpu_set_flag(vcpu, VCPU_HYP_CONTEXT);
178+ else
179+ vcpu_clear_flag(vcpu, VCPU_HYP_CONTEXT);
180+
157181 do {
158182 /* Jump in the fire! */
159183 exit_code = __guest_enter(vcpu);
arch/arm64/kvm/hypercalls.c+1−1
@@ -198,7 +198,7 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu) · 197 unmodified lines
198198 break;
199199 case ARM_SMCCC_HV_PV_TIME_ST:
200200 gpa = kvm_init_stolen_time(vcpu);
201− if (gpa != GPA_INVALID)
201+ if (gpa != INVALID_GPA)
202202 val[0] = gpa;
203203 break;
204204 case ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID:
arch/arm64/kvm/inject_fault.c+55−6
@@ -12,17 +12,55 @@ · 11 unmodified lines
1212
1313 #include <linux/kvm_host.h>
1414 #include <asm/kvm_emulate.h>
15+#include <asm/kvm_nested.h>
1516 #include <asm/esr.h>
1617
18+static void pend_sync_exception(struct kvm_vcpu *vcpu)
19+{
20+ /* If not nesting, EL1 is the only possible exception target */
21+ if (likely(!vcpu_has_nv(vcpu))) {
22+ kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
23+ return;
24+ }
25+
26+ /*
27+ * With NV, we need to pick between EL1 and EL2. Note that we
28+ * never deal with a nesting exception here, hence never
29+ * changing context, and the exception itself can be delayed
30+ * until the next entry.
31+ */
32+ switch(*vcpu_cpsr(vcpu) & PSR_MODE_MASK) {
33+ case PSR_MODE_EL2h:
34+ case PSR_MODE_EL2t:
35+ kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_SYNC);
36+ break;
37+ case PSR_MODE_EL1h:
38+ case PSR_MODE_EL1t:
39+ kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
40+ break;
41+ case PSR_MODE_EL0t:
42+ if (vcpu_el2_tge_is_set(vcpu))
43+ kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_SYNC);
44+ else
45+ kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
46+ break;
47+ default:
48+ BUG();
49+ }
50+}
51+
52+static bool match_target_el(struct kvm_vcpu *vcpu, unsigned long target)
53+{
54+ return (vcpu_get_flag(vcpu, EXCEPT_MASK) == target);
55+}
56+
1757 static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr)
1858 {
1959 unsigned long cpsr = *vcpu_cpsr(vcpu);
2060 bool is_aarch32 = vcpu_mode_is_32bit(vcpu);
2161 u64 esr = 0;
2262
23− kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
24−
25− vcpu_write_sys_reg(vcpu, addr, FAR_EL1);
63+ pend_sync_exception(vcpu);
2664
2765 /*
2866 * Build an {i,d}abort, depending on the level and the
@@ -43,14 +81,22 @@ static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr · 14 unmodified lines
4381 if (!is_iabt)
4482 esr |= ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT;
4583
46− vcpu_write_sys_reg(vcpu, esr | ESR_ELx_FSC_EXTABT, ESR_EL1);
84+ esr |= ESR_ELx_FSC_EXTABT;
85+
86+ if (match_target_el(vcpu, unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC))) {
87+ vcpu_write_sys_reg(vcpu, addr, FAR_EL1);
88+ vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
89+ } else {
90+ vcpu_write_sys_reg(vcpu, addr, FAR_EL2);
91+ vcpu_write_sys_reg(vcpu, esr, ESR_EL2);
92+ }
4793 }
4894
4995 static void inject_undef64(struct kvm_vcpu *vcpu)
5096 {
5197 u64 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT);
5298
53− kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
99+ pend_sync_exception(vcpu);
54100
55101 /*
56102 * Build an unknown exception, depending on the instruction
@@ -59,7 +105,10 @@ static void inject_undef64(struct kvm_vcpu *vcpu) · 2 unmodified lines
59105 if (kvm_vcpu_trap_il_is32bit(vcpu))
60106 esr |= ESR_ELx_IL;
61107
62− vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
108+ if (match_target_el(vcpu, unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC)))
109+ vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
110+ else
111+ vcpu_write_sys_reg(vcpu, esr, ESR_EL2);
63112 }
64113
65114 #define DFSR_FSC_EXTABT_LPAE 0x10
arch/arm64/kvm/mmu.c+23−23
@@ -25,11 +25,11 @@ · 24 unmodified lines
2525 static struct kvm_pgtable *hyp_pgtable;
2626 static DEFINE_MUTEX(kvm_hyp_pgd_mutex);
2727
28−static unsigned long hyp_idmap_start;
29−static unsigned long hyp_idmap_end;
30−static phys_addr_t hyp_idmap_vector;
28+static unsigned long __ro_after_init hyp_idmap_start;
29+static unsigned long __ro_after_init hyp_idmap_end;
30+static phys_addr_t __ro_after_init hyp_idmap_vector;
3131
32−static unsigned long io_map_base;
32+static unsigned long __ro_after_init io_map_base;
3333
3434 static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end)
3535 {
@@ -46,16 +46,17 @@ static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end) · 10 unmodified lines
4646 * long will also starve other vCPUs. We have to also make sure that the page
4747 * tables are not freed while we released the lock.
4848 */
49−static int stage2_apply_range(struct kvm *kvm, phys_addr_t addr,
49+static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr,
5050 phys_addr_t end,
5151 int (*fn)(struct kvm_pgtable *, u64, u64),
5252 bool resched)
5353 {
54+ struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
5455 int ret;
5556 u64 next;
5657
5758 do {
58− struct kvm_pgtable *pgt = kvm->arch.mmu.pgt;
59+ struct kvm_pgtable *pgt = mmu->pgt;
5960 if (!pgt)
6061 return -EINVAL;
6162
@@ -71,8 +72,8 @@ static int stage2_apply_range(struct kvm *kvm, phys_addr_t addr, · 9 unmodified lines
7172 return ret;
7273 }
7374
74−#define stage2_apply_range_resched(kvm, addr, end, fn) \
75− stage2_apply_range(kvm, addr, end, fn, true)
75+#define stage2_apply_range_resched(mmu, addr, end, fn) \
76+ stage2_apply_range(mmu, addr, end, fn, true)
7677
7778 static bool memslot_is_logging(struct kvm_memory_slot *memslot)
7879 {
@@ -235,7 +236,7 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 · 156 unmodified lines
235236
236237 lockdep_assert_held_write(&kvm->mmu_lock);
237238 WARN_ON(size & ~PAGE_MASK);
238− WARN_ON(stage2_apply_range(kvm, start, end, kvm_pgtable_stage2_unmap,
239+ WARN_ON(stage2_apply_range(mmu, start, end, kvm_pgtable_stage2_unmap,
239240 may_block));
240241 }
241242
@@ -250,7 +251,7 @@ static void stage2_flush_memslot(struct kvm *kvm, · 8 unmodified lines
250251 phys_addr_t addr = memslot->base_gfn << PAGE_SHIFT;
251252 phys_addr_t end = addr + PAGE_SIZE * memslot->npages;
252253
253− stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_flush);
254+ stage2_apply_range_resched(&kvm->arch.mmu, addr, end, kvm_pgtable_stage2_flush);
254255 }
255256
256257 /**
@@ -280,7 +281,7 @@ static void stage2_flush_vm(struct kvm *kvm) · 23 unmodified lines
280281 /**
281282 * free_hyp_pgds - free Hyp-mode page tables
282283 */
283−void free_hyp_pgds(void)
284+void __init free_hyp_pgds(void)
284285 {
285286 mutex_lock(&kvm_hyp_pgd_mutex);
286287 if (hyp_pgtable) {
@@ -934,8 +935,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, · 647 unmodified lines
934935 */
935936 static void stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end)
936937 {
937− struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
938− stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_wrprotect);
938+ stage2_apply_range_resched(mmu, addr, end, kvm_pgtable_stage2_wrprotect);
939939 }
940940
941941 /**
@@ -1383,7 +1383,9 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · 441 unmodified lines
13831383 else
13841384 ret = kvm_pgtable_stage2_map(pgt, fault_ipa, vma_pagesize,
13851385 __pfn_to_phys(pfn), prot,
1386− memcache, KVM_PGTABLE_WALK_SHARED);
1386+ memcache,
1387+ KVM_PGTABLE_WALK_HANDLE_FAULT |
1388+ KVM_PGTABLE_WALK_SHARED);
13871389
13881390 /* Mark the page dirty only if the fault is handled successfully */
13891391 if (writable && !ret) {
@@ -1401,20 +1403,18 @@ out_unlock: · 11 unmodified lines
14011403 /* Resolve the access fault by making the page young again. */
14021404 static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
14031405 {
1404− pte_t pte;
1405− kvm_pte_t kpte;
1406+ kvm_pte_t pte;
14061407 struct kvm_s2_mmu *mmu;
14071408
14081409 trace_kvm_access_fault(fault_ipa);
14091410
1410− write_lock(&vcpu->kvm->mmu_lock);
1411+ read_lock(&vcpu->kvm->mmu_lock);
14111412 mmu = vcpu->arch.hw_mmu;
1412− kpte = kvm_pgtable_stage2_mkyoung(mmu->pgt, fault_ipa);
1413− write_unlock(&vcpu->kvm->mmu_lock);
1413+ pte = kvm_pgtable_stage2_mkyoung(mmu->pgt, fault_ipa);
1414+ read_unlock(&vcpu->kvm->mmu_lock);
14141415
1415− pte = __pte(kpte);
1416− if (pte_valid(pte))
1417− kvm_set_pfn_accessed(pte_pfn(pte));
1416+ if (kvm_pte_valid(pte))
1417+ kvm_set_pfn_accessed(kvm_pte_to_pfn(pte));
14181418 }
14191419
14201420 /**
@@ -1668,7 +1668,7 @@ static struct kvm_pgtable_mm_ops kvm_hyp_mm_ops = { · 247 unmodified lines
16681668 .virt_to_phys = kvm_host_pa,
16691669 };
16701670
1671−int kvm_mmu_init(u32 *hyp_va_bits)
1671+int __init kvm_mmu_init(u32 *hyp_va_bits)
16721672 {
16731673 int err;
16741674 u32 idmap_bits;
arch/arm64/kvm/nested.c+161−0
@@ -0,0 +1,161 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/*
3+ * Copyright (C) 2017 - Columbia University and Linaro Ltd.
4+ * Author: Jintack Lim <jintack.lim@linaro.org>
5+ */
6+
7+#include <linux/kvm.h>
8+#include <linux/kvm_host.h>
9+
10+#include <asm/kvm_emulate.h>
11+#include <asm/kvm_nested.h>
12+#include <asm/sysreg.h>
13+
14+#include "sys_regs.h"
15+
16+/* Protection against the sysreg repainting madness... */
17+#define NV_FTR(r, f) ID_AA64##r##_EL1_##f
18+
19+/*
20+ * Our emulated CPU doesn't support all the possible features. For the
21+ * sake of simplicity (and probably mental sanity), wipe out a number
22+ * of feature bits we don't intend to support for the time being.
23+ * This list should get updated as new features get added to the NV
24+ * support, and new extension to the architecture.
25+ */
26+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
27+ const struct sys_reg_desc *r)
28+{
29+ u32 id = reg_to_encoding(r);
30+ u64 val, tmp;
31+
32+ val = p->regval;
33+
34+ switch (id) {
35+ case SYS_ID_AA64ISAR0_EL1:
36+ /* Support everything but TME, O.S. and Range TLBIs */
37+ val &= ~(NV_FTR(ISAR0, TLB) |
38+ NV_FTR(ISAR0, TME));
39+ break;
40+
41+ case SYS_ID_AA64ISAR1_EL1:
42+ /* Support everything but PtrAuth and Spec Invalidation */
43+ val &= ~(GENMASK_ULL(63, 56) |
44+ NV_FTR(ISAR1, SPECRES) |
45+ NV_FTR(ISAR1, GPI) |
46+ NV_FTR(ISAR1, GPA) |
47+ NV_FTR(ISAR1, API) |
48+ NV_FTR(ISAR1, APA));
49+ break;
50+
51+ case SYS_ID_AA64PFR0_EL1:
52+ /* No AMU, MPAM, S-EL2, RAS or SVE */
53+ val &= ~(GENMASK_ULL(55, 52) |
54+ NV_FTR(PFR0, AMU) |
55+ NV_FTR(PFR0, MPAM) |
56+ NV_FTR(PFR0, SEL2) |
57+ NV_FTR(PFR0, RAS) |
58+ NV_FTR(PFR0, SVE) |
59+ NV_FTR(PFR0, EL3) |
60+ NV_FTR(PFR0, EL2) |
61+ NV_FTR(PFR0, EL1));
62+ /* 64bit EL1/EL2/EL3 only */
63+ val |= FIELD_PREP(NV_FTR(PFR0, EL1), 0b0001);
64+ val |= FIELD_PREP(NV_FTR(PFR0, EL2), 0b0001);
65+ val |= FIELD_PREP(NV_FTR(PFR0, EL3), 0b0001);
66+ break;
67+
68+ case SYS_ID_AA64PFR1_EL1:
69+ /* Only support SSBS */
70+ val &= NV_FTR(PFR1, SSBS);
71+ break;
72+
73+ case SYS_ID_AA64MMFR0_EL1:
74+ /* Hide ECV, FGT, ExS, Secure Memory */
75+ val &= ~(GENMASK_ULL(63, 43) |
76+ NV_FTR(MMFR0, TGRAN4_2) |
77+ NV_FTR(MMFR0, TGRAN16_2) |
78+ NV_FTR(MMFR0, TGRAN64_2) |
79+ NV_FTR(MMFR0, SNSMEM));
80+
81+ /* Disallow unsupported S2 page sizes */
82+ switch (PAGE_SIZE) {
83+ case SZ_64K:
84+ val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN16_2), 0b0001);
85+ fallthrough;
86+ case SZ_16K:
87+ val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN4_2), 0b0001);
88+ fallthrough;
89+ case SZ_4K:
90+ /* Support everything */
91+ break;
92+ }
93+ /*
94+ * Since we can't support a guest S2 page size smaller than
95+ * the host's own page size (due to KVM only populating its
96+ * own S2 using the kernel's page size), advertise the
97+ * limitation using FEAT_GTG.
98+ */
99+ switch (PAGE_SIZE) {
100+ case SZ_4K:
101+ val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN4_2), 0b0010);
102+ fallthrough;
103+ case SZ_16K:
104+ val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN16_2), 0b0010);
105+ fallthrough;
106+ case SZ_64K:
107+ val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN64_2), 0b0010);
108+ break;
109+ }
110+ /* Cap PARange to 48bits */
111+ tmp = FIELD_GET(NV_FTR(MMFR0, PARANGE), val);
112+ if (tmp > 0b0101) {
113+ val &= ~NV_FTR(MMFR0, PARANGE);
114+ val |= FIELD_PREP(NV_FTR(MMFR0, PARANGE), 0b0101);
115+ }
116+ break;
117+
118+ case SYS_ID_AA64MMFR1_EL1:
119+ val &= (NV_FTR(MMFR1, PAN) |
120+ NV_FTR(MMFR1, LO) |
121+ NV_FTR(MMFR1, HPDS) |
122+ NV_FTR(MMFR1, VH) |
123+ NV_FTR(MMFR1, VMIDBits));
124+ break;
125+
126+ case SYS_ID_AA64MMFR2_EL1:
127+ val &= ~(NV_FTR(MMFR2, EVT) |
128+ NV_FTR(MMFR2, BBM) |
129+ NV_FTR(MMFR2, TTL) |
130+ GENMASK_ULL(47, 44) |
131+ NV_FTR(MMFR2, ST) |
132+ NV_FTR(MMFR2, CCIDX) |
133+ NV_FTR(MMFR2, VARange));
134+
135+ /* Force TTL support */
136+ val |= FIELD_PREP(NV_FTR(MMFR2, TTL), 0b0001);
137+ break;
138+
139+ case SYS_ID_AA64DFR0_EL1:
140+ /* Only limited support for PMU, Debug, BPs and WPs */
141+ val &= (NV_FTR(DFR0, PMUVer) |
142+ NV_FTR(DFR0, WRPs) |
143+ NV_FTR(DFR0, BRPs) |
144+ NV_FTR(DFR0, DebugVer));
145+
146+ /* Cap Debug to ARMv8.1 */
147+ tmp = FIELD_GET(NV_FTR(DFR0, DebugVer), val);
148+ if (tmp > 0b0111) {
149+ val &= ~NV_FTR(DFR0, DebugVer);
150+ val |= FIELD_PREP(NV_FTR(DFR0, DebugVer), 0b0111);
151+ }
152+ break;
153+
154+ default:
155+ /* Unknown register, just wipe it clean */
156+ val = 0;
157+ break;
158+ }
159+
160+ p->regval = val;
161+}
arch/arm64/kvm/pvtime.c+4−4
@@ -19,7 +19,7 @@ void kvm_update_stolen_time(struct kvm_vcpu *vcpu) · 18 unmodified lines
1919 u64 steal = 0;
2020 int idx;
2121
22− if (base == GPA_INVALID)
22+ if (base == INVALID_GPA)
2323 return;
2424
2525 idx = srcu_read_lock(&kvm->srcu);
@@ -40,7 +40,7 @@ long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu) · 14 unmodified lines
4040 switch (feature) {
4141 case ARM_SMCCC_HV_PV_TIME_FEATURES:
4242 case ARM_SMCCC_HV_PV_TIME_ST:
43− if (vcpu->arch.steal.base != GPA_INVALID)
43+ if (vcpu->arch.steal.base != INVALID_GPA)
4444 val = SMCCC_RET_SUCCESS;
4545 break;
4646 }
@@ -54,7 +54,7 @@ gpa_t kvm_init_stolen_time(struct kvm_vcpu *vcpu) · 7 unmodified lines
5454 struct kvm *kvm = vcpu->kvm;
5555 u64 base = vcpu->arch.steal.base;
5656
57− if (base == GPA_INVALID)
57+ if (base == INVALID_GPA)
5858 return base;
5959
6060 /*
@@ -89,7 +89,7 @@ int kvm_arm_pvtime_set_attr(struct kvm_vcpu *vcpu, · 28 unmodified lines
8989 return -EFAULT;
9090 if (!IS_ALIGNED(ipa, 64))
9191 return -EINVAL;
92− if (vcpu->arch.steal.base != GPA_INVALID)
92+ if (vcpu->arch.steal.base != INVALID_GPA)
9393 return -EEXIST;
9494
9595 /* Check the address is in a valid memslot */
arch/arm64/kvm/reset.c+21−4
@@ -27,10 +27,11 @@ · 26 unmodified lines
2727 #include <asm/kvm_asm.h>
2828 #include <asm/kvm_emulate.h>
2929 #include <asm/kvm_mmu.h>
30+#include <asm/kvm_nested.h>
3031 #include <asm/virt.h>
3132
3233 /* Maximum phys_shift supported for any VM on this host */
33−static u32 kvm_ipa_limit;
34+static u32 __ro_after_init kvm_ipa_limit;
3435
3536 /*
3637 * ARMv8 Reset Values
@@ -38,12 +39,15 @@ static u32 kvm_ipa_limit; · 1 unmodified line
3839 #define VCPU_RESET_PSTATE_EL1 (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
3940 PSR_F_BIT | PSR_D_BIT)
4041
42+#define VCPU_RESET_PSTATE_EL2 (PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
43+ PSR_F_BIT | PSR_D_BIT)
44+
4145 #define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
4246 PSR_AA32_I_BIT | PSR_AA32_F_BIT)
4347
44−unsigned int kvm_sve_max_vl;
48+unsigned int __ro_after_init kvm_sve_max_vl;
4549
46−int kvm_arm_init_sve(void)
50+int __init kvm_arm_init_sve(void)
4751 {
4852 if (system_supports_sve()) {
4953 kvm_sve_max_vl = sve_max_virtualisable_vl();
@@ -157,6 +161,7 @@ void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu) · 107 unmodified lines
157161 if (sve_state)
158162 kvm_unshare_hyp(sve_state, sve_state + vcpu_sve_state_size(vcpu));
159163 kfree(sve_state);
164+ kfree(vcpu->arch.ccsidr);
160165 }
161166
162167 static void kvm_vcpu_reset_sve(struct kvm_vcpu *vcpu)
@@ -220,6 +225,10 @@ static int kvm_set_vm_width(struct kvm_vcpu *vcpu) · 57 unmodified lines
220225 if (kvm_has_mte(kvm) && is32bit)
221226 return -EINVAL;
222227
228+ /* NV is incompatible with AArch32 */
229+ if (vcpu_has_nv(vcpu) && is32bit)
230+ return -EINVAL;
231+
223232 if (is32bit)
224233 set_bit(KVM_ARCH_FLAG_EL1_32BIT, &kvm->arch.flags);
225234
@@ -272,6 +281,12 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) · 46 unmodified lines
272281 if (loaded)
273282 kvm_arch_vcpu_put(vcpu);
274283
284+ /* Disallow NV+SVE for the time being */
285+ if (vcpu_has_nv(vcpu) && vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE)) {
286+ ret = -EINVAL;
287+ goto out;
288+ }
289+
275290 if (!kvm_arm_vcpu_sve_finalized(vcpu)) {
276291 if (test_bit(KVM_ARM_VCPU_SVE, vcpu->arch.features)) {
277292 ret = kvm_vcpu_enable_sve(vcpu);
@@ -294,6 +309,8 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) · 16 unmodified lines
294309 default:
295310 if (vcpu_el1_is_32bit(vcpu)) {
296311 pstate = VCPU_RESET_PSTATE_SVC;
312+ } else if (vcpu_has_nv(vcpu)) {
313+ pstate = VCPU_RESET_PSTATE_EL2;
297314 } else {
298315 pstate = VCPU_RESET_PSTATE_EL1;
299316 }
@@ -352,7 +369,7 @@ u32 get_kvm_ipa_limit(void) · 52 unmodified lines
352369 return kvm_ipa_limit;
353370 }
354371
355−int kvm_set_ipa_limit(void)
372+int __init kvm_set_ipa_limit(void)
356373 {
357374 unsigned int parange;
358375 u64 mmfr0;
arch/arm64/kvm/sys_regs.c+350−109
@@ -11,6 +11,7 @@ · 10 unmodified lines
1111
1212 #include <linux/bitfield.h>
1313 #include <linux/bsearch.h>
14+#include <linux/cacheinfo.h>
1415 #include <linux/kvm_host.h>
1516 #include <linux/mm.h>
1617 #include <linux/printk.h>
@@ -24,6 +25,7 @@ · 7 unmodified lines
2425 #include <asm/kvm_emulate.h>
2526 #include <asm/kvm_hyp.h>
2627 #include <asm/kvm_mmu.h>
28+#include <asm/kvm_nested.h>
2729 #include <asm/perf_event.h>
2830 #include <asm/sysreg.h>
2931
@@ -78,28 +80,112 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) · 48 unmodified lines
7880 __vcpu_write_sys_reg_to_cpu(val, reg))
7981 return;
8082
81− __vcpu_sys_reg(vcpu, reg) = val;
83+ __vcpu_sys_reg(vcpu, reg) = val;
8284 }
8385
84−/* 3 bits per cache level, as per CLIDR, but non-existent caches always 0 */
85−static u32 cache_levels;
86−
8786 /* CSSELR values; used to index KVM_REG_ARM_DEMUX_ID_CCSIDR */
8887 #define CSSELR_MAX 14
8988
89+/*
90+ * Returns the minimum line size for the selected cache, expressed as
91+ * Log2(bytes).
92+ */
93+static u8 get_min_cache_line_size(bool icache)
94+{
95+ u64 ctr = read_sanitised_ftr_reg(SYS_CTR_EL0);
96+ u8 field;
97+
98+ if (icache)
99+ field = SYS_FIELD_GET(CTR_EL0, IminLine, ctr);
100+ else
101+ field = SYS_FIELD_GET(CTR_EL0, DminLine, ctr);
102+
103+ /*
104+ * Cache line size is represented as Log2(words) in CTR_EL0.
105+ * Log2(bytes) can be derived with the following:
106+ *
107+ * Log2(words) + 2 = Log2(bytes / 4) + 2
108+ * = Log2(bytes) - 2 + 2
109+ * = Log2(bytes)
110+ */
111+ return field + 2;
112+}
113+
90114 /* Which cache CCSIDR represents depends on CSSELR value. */
91−static u32 get_ccsidr(u32 csselr)
115+static u32 get_ccsidr(struct kvm_vcpu *vcpu, u32 csselr)
92116 {
93− u32 ccsidr;
117+ u8 line_size;
94118
95− /* Make sure noone else changes CSSELR during this! */
96− local_irq_disable();
97− write_sysreg(csselr, csselr_el1);
98− isb();
99− ccsidr = read_sysreg(ccsidr_el1);
100− local_irq_enable();
119+ if (vcpu->arch.ccsidr)
120+ return vcpu->arch.ccsidr[csselr];
101121
102− return ccsidr;
122+ line_size = get_min_cache_line_size(csselr & CSSELR_EL1_InD);
123+
124+ /*
125+ * Fabricate a CCSIDR value as the overriding value does not exist.
126+ * The real CCSIDR value will not be used as it can vary by the
127+ * physical CPU which the vcpu currently resides in.
128+ *
129+ * The line size is determined with get_min_cache_line_size(), which
130+ * should be valid for all CPUs even if they have different cache
131+ * configuration.
132+ *
133+ * The associativity bits are cleared, meaning the geometry of all data
134+ * and unified caches (which are guaranteed to be PIPT and thus
135+ * non-aliasing) are 1 set and 1 way.
136+ * Guests should not be doing cache operations by set/way at all, and
137+ * for this reason, we trap them and attempt to infer the intent, so
138+ * that we can flush the entire guest's address space at the appropriate
139+ * time. The exposed geometry minimizes the number of the traps.
140+ * [If guests should attempt to infer aliasing properties from the
141+ * geometry (which is not permitted by the architecture), they would
142+ * only do so for virtually indexed caches.]
143+ *
144+ * We don't check if the cache level exists as it is allowed to return
145+ * an UNKNOWN value if not.
146+ */
147+ return SYS_FIELD_PREP(CCSIDR_EL1, LineSize, line_size - 4);
148+}
149+
150+static int set_ccsidr(struct kvm_vcpu *vcpu, u32 csselr, u32 val)
151+{
152+ u8 line_size = FIELD_GET(CCSIDR_EL1_LineSize, val) + 4;
153+ u32 *ccsidr = vcpu->arch.ccsidr;
154+ u32 i;
155+
156+ if ((val & CCSIDR_EL1_RES0) ||
157+ line_size < get_min_cache_line_size(csselr & CSSELR_EL1_InD))
158+ return -EINVAL;
159+
160+ if (!ccsidr) {
161+ if (val == get_ccsidr(vcpu, csselr))
162+ return 0;
163+
164+ ccsidr = kmalloc_array(CSSELR_MAX, sizeof(u32), GFP_KERNEL_ACCOUNT);
165+ if (!ccsidr)
166+ return -ENOMEM;
167+
168+ for (i = 0; i < CSSELR_MAX; i++)
169+ ccsidr[i] = get_ccsidr(vcpu, i);
170+
171+ vcpu->arch.ccsidr = ccsidr;
172+ }
173+
174+ ccsidr[csselr] = val;
175+
176+ return 0;
177+}
178+
179+static bool access_rw(struct kvm_vcpu *vcpu,
180+ struct sys_reg_params *p,
181+ const struct sys_reg_desc *r)
182+{
183+ if (p->is_write)
184+ vcpu_write_sys_reg(vcpu, p->regval, r->reg);
185+ else
186+ p->regval = vcpu_read_sys_reg(vcpu, r->reg);
187+
188+ return true;
103189 }
104190
105191 /*
@@ -260,6 +346,14 @@ static bool trap_raz_wi(struct kvm_vcpu *vcpu, · 154 unmodified lines
260346 return read_zero(vcpu, p);
261347 }
262348
349+static bool trap_undef(struct kvm_vcpu *vcpu,
350+ struct sys_reg_params *p,
351+ const struct sys_reg_desc *r)
352+{
353+ kvm_inject_undefined(vcpu);
354+ return false;
355+}
356+
263357 /*
264358 * ARMv8.1 mandates at least a trivial LORegion implementation, where all the
265359 * RW registers are RES0 (which we can implement as RAZ/WI). On an ARMv8.0
@@ -370,12 +464,9 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu, · 104 unmodified lines
370464 struct sys_reg_params *p,
371465 const struct sys_reg_desc *r)
372466 {
373− if (p->is_write) {
374− vcpu_write_sys_reg(vcpu, p->regval, r->reg);
467+ access_rw(vcpu, p, r);
468+ if (p->is_write)
375469 vcpu_set_flag(vcpu, DEBUG_DIRTY);
376− } else {
377− p->regval = vcpu_read_sys_reg(vcpu, r->reg);
378− }
379470
380471 trace_trap_reg(__func__, r->reg, p->is_write, p->regval);
381472
@@ -1049,7 +1140,9 @@ static bool access_arch_timer(struct kvm_vcpu *vcpu, · 667 unmodified lines
10491140 treg = TIMER_REG_CVAL;
10501141 break;
10511142 default:
1052− BUG();
1143+ print_sys_reg_msg(p, "%s", "Unhandled trapped timer register");
1144+ kvm_inject_undefined(vcpu);
1145+ return false;
10531146 }
10541147
10551148 if (p->is_write)
@@ -1155,6 +1248,12 @@ static u64 read_id_reg(const struct kvm_vcpu *vcpu, struct sys_reg_desc const *r · 99 unmodified lines
11551248 val |= FIELD_PREP(ARM64_FEATURE_MASK(ID_DFR0_EL1_PerfMon),
11561249 pmuver_to_perfmon(vcpu_pmuver(vcpu)));
11571250 break;
1251+ case SYS_ID_AA64MMFR2_EL1:
1252+ val &= ~ID_AA64MMFR2_EL1_CCIDX_MASK;
1253+ break;
1254+ case SYS_ID_MMFR4_EL1:
1255+ val &= ~ARM64_FEATURE_MASK(ID_MMFR4_EL1_CCIDX);
1256+ break;
11581257 }
11591258
11601259 return val;
@@ -1205,6 +1304,9 @@ static bool access_id_reg(struct kvm_vcpu *vcpu, · 44 unmodified lines
12051304 return write_to_read_only(vcpu, p, r);
12061305
12071306 p->regval = read_id_reg(vcpu, r);
1307+ if (vcpu_has_nv(vcpu))
1308+ access_nested_id_reg(vcpu, p, r);
1309+
12081310 return true;
12091311 }
12101312
@@ -1385,10 +1487,78 @@ static bool access_clidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, · 174 unmodified lines
13851487 if (p->is_write)
13861488 return write_to_read_only(vcpu, p, r);
13871489
1388− p->regval = read_sysreg(clidr_el1);
1490+ p->regval = __vcpu_sys_reg(vcpu, r->reg);
13891491 return true;
13901492 }
13911493
1494+/*
1495+ * Fabricate a CLIDR_EL1 value instead of using the real value, which can vary
1496+ * by the physical CPU which the vcpu currently resides in.
1497+ */
1498+static void reset_clidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
1499+{
1500+ u64 ctr_el0 = read_sanitised_ftr_reg(SYS_CTR_EL0);
1501+ u64 clidr;
1502+ u8 loc;
1503+
1504+ if ((ctr_el0 & CTR_EL0_IDC)) {
1505+ /*
1506+ * Data cache clean to the PoU is not required so LoUU and LoUIS
1507+ * will not be set and a unified cache, which will be marked as
1508+ * LoC, will be added.
1509+ *
1510+ * If not DIC, let the unified cache L2 so that an instruction
1511+ * cache can be added as L1 later.
1512+ */
1513+ loc = (ctr_el0 & CTR_EL0_DIC) ? 1 : 2;
1514+ clidr = CACHE_TYPE_UNIFIED << CLIDR_CTYPE_SHIFT(loc);
1515+ } else {
1516+ /*
1517+ * Data cache clean to the PoU is required so let L1 have a data
1518+ * cache and mark it as LoUU and LoUIS. As L1 has a data cache,
1519+ * it can be marked as LoC too.
1520+ */
1521+ loc = 1;
1522+ clidr = 1 << CLIDR_LOUU_SHIFT;
1523+ clidr |= 1 << CLIDR_LOUIS_SHIFT;
1524+ clidr |= CACHE_TYPE_DATA << CLIDR_CTYPE_SHIFT(1);
1525+ }
1526+
1527+ /*
1528+ * Instruction cache invalidation to the PoU is required so let L1 have
1529+ * an instruction cache. If L1 already has a data cache, it will be
1530+ * CACHE_TYPE_SEPARATE.
1531+ */
1532+ if (!(ctr_el0 & CTR_EL0_DIC))
1533+ clidr |= CACHE_TYPE_INST << CLIDR_CTYPE_SHIFT(1);
1534+
1535+ clidr |= loc << CLIDR_LOC_SHIFT;
1536+
1537+ /*
1538+ * Add tag cache unified to data cache. Allocation tags and data are
1539+ * unified in a cache line so that it looks valid even if there is only
1540+ * one cache line.
1541+ */
1542+ if (kvm_has_mte(vcpu->kvm))
1543+ clidr |= 2 << CLIDR_TTYPE_SHIFT(loc);
1544+
1545+ __vcpu_sys_reg(vcpu, r->reg) = clidr;
1546+}
1547+
1548+static int set_clidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
1549+ u64 val)
1550+{
1551+ u64 ctr_el0 = read_sanitised_ftr_reg(SYS_CTR_EL0);
1552+ u64 idc = !CLIDR_LOC(val) || (!CLIDR_LOUIS(val) && !CLIDR_LOUU(val));
1553+
1554+ if ((val & CLIDR_EL1_RES0) || (!(ctr_el0 & CTR_EL0_IDC) && idc))
1555+ return -EINVAL;
1556+
1557+ __vcpu_sys_reg(vcpu, rd->reg) = val;
1558+
1559+ return 0;
1560+}
1561+
13921562 static bool access_csselr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
13931563 const struct sys_reg_desc *r)
13941564 {
@@ -1410,22 +1580,10 @@ static bool access_ccsidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, · 15 unmodified lines
14101580 return write_to_read_only(vcpu, p, r);
14111581
14121582 csselr = vcpu_read_sys_reg(vcpu, CSSELR_EL1);
1413− p->regval = get_ccsidr(csselr);
1583+ csselr &= CSSELR_EL1_Level | CSSELR_EL1_InD;
1584+ if (csselr < CSSELR_MAX)
1585+ p->regval = get_ccsidr(vcpu, csselr);
14141586
1415− /*
1416− * Guests should not be doing cache operations by set/way at all, and
1417− * for this reason, we trap them and attempt to infer the intent, so
1418− * that we can flush the entire guest's address space at the appropriate
1419− * time.
1420− * To prevent this trapping from causing performance problems, let's
1421− * expose the geometry of all data and unified caches (which are
1422− * guaranteed to be PIPT and thus non-aliasing) as 1 set and 1 way.
1423− * [If guests should attempt to infer aliasing properties from the
1424− * geometry (which is not permitted by the architecture), they would
1425− * only do so for virtually indexed caches.]
1426− */
1427− if (!(csselr & 1)) // data or unified cache
1428− p->regval &= ~GENMASK(27, 3);
14291587 return true;
14301588 }
14311589
@@ -1446,6 +1604,44 @@ static unsigned int mte_visibility(const struct kvm_vcpu *vcpu, · 14 unmodified lines
14461604 .visibility = mte_visibility, \
14471605 }
14481606
1607+static unsigned int el2_visibility(const struct kvm_vcpu *vcpu,
1608+ const struct sys_reg_desc *rd)
1609+{
1610+ if (vcpu_has_nv(vcpu))
1611+ return 0;
1612+
1613+ return REG_HIDDEN;
1614+}
1615+
1616+#define EL2_REG(name, acc, rst, v) { \
1617+ SYS_DESC(SYS_##name), \
1618+ .access = acc, \
1619+ .reset = rst, \
1620+ .reg = name, \
1621+ .visibility = el2_visibility, \
1622+ .val = v, \
1623+}
1624+
1625+/*
1626+ * EL{0,1}2 registers are the EL2 view on an EL0 or EL1 register when
1627+ * HCR_EL2.E2H==1, and only in the sysreg table for convenience of
1628+ * handling traps. Given that, they are always hidden from userspace.
1629+ */
1630+static unsigned int elx2_visibility(const struct kvm_vcpu *vcpu,
1631+ const struct sys_reg_desc *rd)
1632+{
1633+ return REG_HIDDEN_USER;
1634+}
1635+
1636+#define EL12_REG(name, acc, rst, v) { \
1637+ SYS_DESC(SYS_##name##_EL12), \
1638+ .access = acc, \
1639+ .reset = rst, \
1640+ .reg = name##_EL1, \
1641+ .val = v, \
1642+ .visibility = elx2_visibility, \
1643+}
1644+
14491645 /* sys_reg_desc initialiser for known cpufeature ID registers */
14501646 #define ID_SANITISED(name) { \
14511647 SYS_DESC(SYS_##name), \
@@ -1490,6 +1686,42 @@ static unsigned int mte_visibility(const struct kvm_vcpu *vcpu, · 38 unmodified lines
14901686 .visibility = raz_visibility, \
14911687 }
14921688
1689+static bool access_sp_el1(struct kvm_vcpu *vcpu,
1690+ struct sys_reg_params *p,
1691+ const struct sys_reg_desc *r)
1692+{
1693+ if (p->is_write)
1694+ __vcpu_sys_reg(vcpu, SP_EL1) = p->regval;
1695+ else
1696+ p->regval = __vcpu_sys_reg(vcpu, SP_EL1);
1697+
1698+ return true;
1699+}
1700+
1701+static bool access_elr(struct kvm_vcpu *vcpu,
1702+ struct sys_reg_params *p,
1703+ const struct sys_reg_desc *r)
1704+{
1705+ if (p->is_write)
1706+ vcpu_write_sys_reg(vcpu, p->regval, ELR_EL1);
1707+ else
1708+ p->regval = vcpu_read_sys_reg(vcpu, ELR_EL1);
1709+
1710+ return true;
1711+}
1712+
1713+static bool access_spsr(struct kvm_vcpu *vcpu,
1714+ struct sys_reg_params *p,
1715+ const struct sys_reg_desc *r)
1716+{
1717+ if (p->is_write)
1718+ __vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval;
1719+ else
1720+ p->regval = __vcpu_sys_reg(vcpu, SPSR_EL1);
1721+
1722+ return true;
1723+}
1724+
14931725 /*
14941726 * Architected system registers.
14951727 * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2
@@ -1646,6 +1878,9 @@ static const struct sys_reg_desc sys_reg_descs[] = { · 150 unmodified lines
16461878 PTRAUTH_KEY(APDB),
16471879 PTRAUTH_KEY(APGA),
16481880
1881+ { SYS_DESC(SYS_SPSR_EL1), access_spsr},
1882+ { SYS_DESC(SYS_ELR_EL1), access_elr},
1883+
16491884 { SYS_DESC(SYS_AFSR0_EL1), access_vm_reg, reset_unknown, AFSR0_EL1 },
16501885 { SYS_DESC(SYS_AFSR1_EL1), access_vm_reg, reset_unknown, AFSR1_EL1 },
16511886 { SYS_DESC(SYS_ESR_EL1), access_vm_reg, reset_unknown, ESR_EL1 },
@@ -1693,7 +1928,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { · 41 unmodified lines
16931928 { SYS_DESC(SYS_LORC_EL1), trap_loregion },
16941929 { SYS_DESC(SYS_LORID_EL1), trap_loregion },
16951930
1696− { SYS_DESC(SYS_VBAR_EL1), NULL, reset_val, VBAR_EL1, 0 },
1931+ { SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 },
16971932 { SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 },
16981933
16991934 { SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only },
@@ -1717,7 +1952,9 @@ static const struct sys_reg_desc sys_reg_descs[] = { · 17 unmodified lines
17171952 { SYS_DESC(SYS_CNTKCTL_EL1), NULL, reset_val, CNTKCTL_EL1, 0},
17181953
17191954 { SYS_DESC(SYS_CCSIDR_EL1), access_ccsidr },
1720− { SYS_DESC(SYS_CLIDR_EL1), access_clidr },
1955+ { SYS_DESC(SYS_CLIDR_EL1), access_clidr, reset_clidr, CLIDR_EL1,
1956+ .set_user = set_clidr },
1957+ { SYS_DESC(SYS_CCSIDR2_EL1), undef_access },
17211958 { SYS_DESC(SYS_SMIDR_EL1), undef_access },
17221959 { SYS_DESC(SYS_CSSELR_EL1), access_csselr, reset_unknown, CSSELR_EL1 },
17231960 { SYS_DESC(SYS_CTR_EL0), access_ctr },
@@ -1913,9 +2150,67 @@ static const struct sys_reg_desc sys_reg_descs[] = { · 189 unmodified lines
19132150 { PMU_SYS_REG(SYS_PMCCFILTR_EL0), .access = access_pmu_evtyper,
19142151 .reset = reset_val, .reg = PMCCFILTR_EL0, .val = 0 },
19152152
2153+ EL2_REG(VPIDR_EL2, access_rw, reset_unknown, 0),
2154+ EL2_REG(VMPIDR_EL2, access_rw, reset_unknown, 0),
2155+ EL2_REG(SCTLR_EL2, access_rw, reset_val, SCTLR_EL2_RES1),
2156+ EL2_REG(ACTLR_EL2, access_rw, reset_val, 0),
2157+ EL2_REG(HCR_EL2, access_rw, reset_val, 0),
2158+ EL2_REG(MDCR_EL2, access_rw, reset_val, 0),
2159+ EL2_REG(CPTR_EL2, access_rw, reset_val, CPTR_EL2_DEFAULT ),
2160+ EL2_REG(HSTR_EL2, access_rw, reset_val, 0),
2161+ EL2_REG(HACR_EL2, access_rw, reset_val, 0),
2162+
2163+ EL2_REG(TTBR0_EL2, access_rw, reset_val, 0),
2164+ EL2_REG(TTBR1_EL2, access_rw, reset_val, 0),
2165+ EL2_REG(TCR_EL2, access_rw, reset_val, TCR_EL2_RES1),
2166+ EL2_REG(VTTBR_EL2, access_rw, reset_val, 0),
2167+ EL2_REG(VTCR_EL2, access_rw, reset_val, 0),
2168+
19162169 { SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 },
2170+ EL2_REG(SPSR_EL2, access_rw, reset_val, 0),
2171+ EL2_REG(ELR_EL2, access_rw, reset_val, 0),
2172+ { SYS_DESC(SYS_SP_EL1), access_sp_el1},
2173+
19172174 { SYS_DESC(SYS_IFSR32_EL2), NULL, reset_unknown, IFSR32_EL2 },
2175+ EL2_REG(AFSR0_EL2, access_rw, reset_val, 0),
2176+ EL2_REG(AFSR1_EL2, access_rw, reset_val, 0),
2177+ EL2_REG(ESR_EL2, access_rw, reset_val, 0),
19182178 { SYS_DESC(SYS_FPEXC32_EL2), NULL, reset_val, FPEXC32_EL2, 0x700 },
2179+
2180+ EL2_REG(FAR_EL2, access_rw, reset_val, 0),
2181+ EL2_REG(HPFAR_EL2, access_rw, reset_val, 0),
2182+
2183+ EL2_REG(MAIR_EL2, access_rw, reset_val, 0),
2184+ EL2_REG(AMAIR_EL2, access_rw, reset_val, 0),
2185+
2186+ EL2_REG(VBAR_EL2, access_rw, reset_val, 0),
2187+ EL2_REG(RVBAR_EL2, access_rw, reset_val, 0),
2188+ { SYS_DESC(SYS_RMR_EL2), trap_undef },
2189+
2190+ EL2_REG(CONTEXTIDR_EL2, access_rw, reset_val, 0),
2191+ EL2_REG(TPIDR_EL2, access_rw, reset_val, 0),
2192+
2193+ EL2_REG(CNTVOFF_EL2, access_rw, reset_val, 0),
2194+ EL2_REG(CNTHCTL_EL2, access_rw, reset_val, 0),
2195+
2196+ EL12_REG(SCTLR, access_vm_reg, reset_val, 0x00C50078),
2197+ EL12_REG(CPACR, access_rw, reset_val, 0),
2198+ EL12_REG(TTBR0, access_vm_reg, reset_unknown, 0),
2199+ EL12_REG(TTBR1, access_vm_reg, reset_unknown, 0),
2200+ EL12_REG(TCR, access_vm_reg, reset_val, 0),
2201+ { SYS_DESC(SYS_SPSR_EL12), access_spsr},
2202+ { SYS_DESC(SYS_ELR_EL12), access_elr},
2203+ EL12_REG(AFSR0, access_vm_reg, reset_unknown, 0),
2204+ EL12_REG(AFSR1, access_vm_reg, reset_unknown, 0),
2205+ EL12_REG(ESR, access_vm_reg, reset_unknown, 0),
2206+ EL12_REG(FAR, access_vm_reg, reset_unknown, 0),
2207+ EL12_REG(MAIR, access_vm_reg, reset_unknown, 0),
2208+ EL12_REG(AMAIR, access_vm_reg, reset_amair_el1, 0),
2209+ EL12_REG(VBAR, access_rw, reset_val, 0),
2210+ EL12_REG(CONTEXTIDR, access_vm_reg, reset_val, 0),
2211+ EL12_REG(CNTKCTL, access_rw, reset_val, 0),
2212+
2213+ EL2_REG(SP_EL2, NULL, reset_unknown, 0),
19192214 };
19202215
19212216 static bool trap_dbgdidr(struct kvm_vcpu *vcpu,
@@ -2219,6 +2514,10 @@ static const struct sys_reg_desc cp15_regs[] = { · 297 unmodified lines
22192514
22202515 { Op1(1), CRn( 0), CRm( 0), Op2(0), access_ccsidr },
22212516 { Op1(1), CRn( 0), CRm( 0), Op2(1), access_clidr },
2517+
2518+ /* CCSIDR2 */
2519+ { Op1(1), CRn( 0), CRm( 0), Op2(2), undef_access },
2520+
22222521 { Op1(2), CRn( 0), CRm( 0), Op2(0), access_csselr, NULL, CSSELR_EL1 },
22232522 };
22242523
@@ -2724,7 +3023,6 @@ id_to_sys_reg_desc(struct kvm_vcpu *vcpu, u64 id, · 499 unmodified lines
27243023
27253024 FUNCTION_INVARIANT(midr_el1)
27263025 FUNCTION_INVARIANT(revidr_el1)
2727−FUNCTION_INVARIANT(clidr_el1)
27283026 FUNCTION_INVARIANT(aidr_el1)
27293027
27303028 static void get_ctr_el0(struct kvm_vcpu *v, const struct sys_reg_desc *r)
@@ -2733,10 +3031,9 @@ static void get_ctr_el0(struct kvm_vcpu *v, const struct sys_reg_desc *r) · 2 unmodified lines
27333031 }
27343032
27353033 /* ->val is filled in by kvm_sys_reg_table_init() */
2736−static struct sys_reg_desc invariant_sys_regs[] = {
3034+static struct sys_reg_desc invariant_sys_regs[] __ro_after_init = {
27373035 { SYS_DESC(SYS_MIDR_EL1), NULL, get_midr_el1 },
27383036 { SYS_DESC(SYS_REVIDR_EL1), NULL, get_revidr_el1 },
2739− { SYS_DESC(SYS_CLIDR_EL1), NULL, get_clidr_el1 },
27403037 { SYS_DESC(SYS_AIDR_EL1), NULL, get_aidr_el1 },
27413038 { SYS_DESC(SYS_CTR_EL0), NULL, get_ctr_el0 },
27423039 };
@@ -2773,33 +3070,7 @@ static int set_invariant_sys_reg(u64 id, u64 __user *uaddr) · 30 unmodified lines
27733070 return 0;
27743071 }
27753072
2776−static bool is_valid_cache(u32 val)
2777−{
2778− u32 level, ctype;
2779−
2780− if (val >= CSSELR_MAX)
2781− return false;
2782−
2783− /* Bottom bit is Instruction or Data bit. Next 3 bits are level. */
2784− level = (val >> 1);
2785− ctype = (cache_levels >> (level * 3)) & 7;
2786−
2787− switch (ctype) {
2788− case 0: /* No cache */
2789− return false;
2790− case 1: /* Instruction cache only */
2791− return (val & 1);
2792− case 2: /* Data cache only */
2793− case 4: /* Unified cache */
2794− return !(val & 1);
2795− case 3: /* Separate instruction and data caches */
2796− return true;
2797− default: /* Reserved: we can't know instruction or data. */
2798− return false;
2799− }
2800−}
2801−
2802−static int demux_c15_get(u64 id, void __user *uaddr)
3073+static int demux_c15_get(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
28033074 {
28043075 u32 val;
28053076 u32 __user *uval = uaddr;
@@ -2815,16 +3086,16 @@ static int demux_c15_get(u64 id, void __user *uaddr) · 9 unmodified lines
28153086 return -ENOENT;
28163087 val = (id & KVM_REG_ARM_DEMUX_VAL_MASK)
28173088 >> KVM_REG_ARM_DEMUX_VAL_SHIFT;
2818− if (!is_valid_cache(val))
3089+ if (val >= CSSELR_MAX)
28193090 return -ENOENT;
28203091
2821− return put_user(get_ccsidr(val), uval);
3092+ return put_user(get_ccsidr(vcpu, val), uval);
28223093 default:
28233094 return -ENOENT;
28243095 }
28253096 }
28263097
2827−static int demux_c15_set(u64 id, void __user *uaddr)
3098+static int demux_c15_set(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
28283099 {
28293100 u32 val, newval;
28303101 u32 __user *uval = uaddr;
@@ -2840,16 +3111,13 @@ static int demux_c15_set(u64 id, void __user *uaddr) · 9 unmodified lines
28403111 return -ENOENT;
28413112 val = (id & KVM_REG_ARM_DEMUX_VAL_MASK)
28423113 >> KVM_REG_ARM_DEMUX_VAL_SHIFT;
2843− if (!is_valid_cache(val))
3114+ if (val >= CSSELR_MAX)
28443115 return -ENOENT;
28453116
28463117 if (get_user(newval, uval))
28473118 return -EFAULT;
28483119
2849− /* This is also invariant: you can't change it. */
2850− if (newval != get_ccsidr(val))
2851− return -EINVAL;
2852− return 0;
3120+ return set_ccsidr(vcpu, val, newval);
28533121 default:
28543122 return -ENOENT;
28553123 }
@@ -2864,7 +3132,7 @@ int kvm_sys_reg_get_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg, · 8 unmodified lines
28643132 int ret;
28653133
28663134 r = id_to_sys_reg_desc(vcpu, reg->id, table, num);
2867− if (!r)
3135+ if (!r || sysreg_hidden_user(vcpu, r))
28683136 return -ENOENT;
28693137
28703138 if (r->get_user) {
@@ -2886,7 +3154,7 @@ int kvm_arm_sys_reg_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg · 15 unmodified lines
28863154 int err;
28873155
28883156 if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX)
2889− return demux_c15_get(reg->id, uaddr);
3157+ return demux_c15_get(vcpu, reg->id, uaddr);
28903158
28913159 err = get_invariant_sys_reg(reg->id, uaddr);
28923160 if (err != -ENOENT)
@@ -2908,7 +3176,7 @@ int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg, · 15 unmodified lines
29083176 return -EFAULT;
29093177
29103178 r = id_to_sys_reg_desc(vcpu, reg->id, table, num);
2911− if (!r)
3179+ if (!r || sysreg_hidden_user(vcpu, r))
29123180 return -ENOENT;
29133181
29143182 if (sysreg_user_write_ignore(vcpu, r))
@@ -2930,7 +3198,7 @@ int kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg · 15 unmodified lines
29303198 int err;
29313199
29323200 if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX)
2933− return demux_c15_set(reg->id, uaddr);
3201+ return demux_c15_set(vcpu, reg->id, uaddr);
29343202
29353203 err = set_invariant_sys_reg(reg->id, uaddr);
29363204 if (err != -ENOENT)
@@ -2942,13 +3210,7 @@ int kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg · 5 unmodified lines
29423210
29433211 static unsigned int num_demux_regs(void)
29443212 {
2945− unsigned int i, count = 0;
2946−
2947− for (i = 0; i < CSSELR_MAX; i++)
2948− if (is_valid_cache(i))
2949− count++;
2950−
2951− return count;
3213+ return CSSELR_MAX;
29523214 }
29533215
29543216 static int write_demux_regids(u64 __user *uindices)
@@ -2958,8 +3220,6 @@ static int write_demux_regids(u64 __user *uindices) · 3 unmodified lines
29583220
29593221 val |= KVM_REG_ARM_DEMUX_ID_CCSIDR;
29603222 for (i = 0; i < CSSELR_MAX; i++) {
2961− if (!is_valid_cache(i))
2962− continue;
29633223 if (put_user(val | i, uindices))
29643224 return -EFAULT;
29653225 uindices++;
@@ -3002,7 +3262,7 @@ static int walk_one_sys_reg(const struct kvm_vcpu *vcpu, · 36 unmodified lines
30023262 if (!(rd->reg || rd->get_user))
30033263 return 0;
30043264
3005− if (sysreg_hidden(vcpu, rd))
3265+ if (sysreg_hidden_user(vcpu, rd))
30063266 return 0;
30073267
30083268 if (!copy_reg_to_user(rd, uind))
@@ -3057,11 +3317,10 @@ int kvm_arm_copy_sys_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices) · 48 unmodified lines
30573317 return write_demux_regids(uindices);
30583318 }
30593319
3060−int kvm_sys_reg_table_init(void)
3320+int __init kvm_sys_reg_table_init(void)
30613321 {
30623322 bool valid = true;
30633323 unsigned int i;
3064− struct sys_reg_desc clidr;
30653324
30663325 /* Make sure tables are unique and in order. */
30673326 valid &= check_sysreg_table(sys_reg_descs, ARRAY_SIZE(sys_reg_descs), false);
@@ -3078,23 +3337,5 @@ int kvm_sys_reg_table_init(void) · 10 unmodified lines
30783337 for (i = 0; i < ARRAY_SIZE(invariant_sys_regs); i++)
30793338 invariant_sys_regs[i].reset(NULL, &invariant_sys_regs[i]);
30803339
3081− /*
3082− * CLIDR format is awkward, so clean it up. See ARM B4.1.20:
3083− *
3084− * If software reads the Cache Type fields from Ctype1
3085− * upwards, once it has seen a value of 0b000, no caches
3086− * exist at further-out levels of the hierarchy. So, for
3087− * example, if Ctype3 is the first Cache Type field with a
3088− * value of 0b000, the values of Ctype4 to Ctype7 must be
3089− * ignored.
3090− */
3091− get_clidr_el1(NULL, &clidr); /* Ugly... */
3092− cache_levels = clidr.val;
3093− for (i = 0; i < 7; i++)
3094− if (((cache_levels >> (i*3)) & 7) == 0)
3095− break;
3096− /* Clear all higher bits. */
3097− cache_levels &= (1 << (i*3))-1;
3098−
30993340 return 0;
31003341 }
arch/arm64/kvm/sys_regs.h+12−2
@@ -85,8 +85,9 @@ struct sys_reg_desc { · 84 unmodified lines
8585 };
8686
8787 #define REG_HIDDEN (1 << 0) /* hidden from userspace and guest */
88−#define REG_RAZ (1 << 1) /* RAZ from userspace and guest */
89−#define REG_USER_WI (1 << 2) /* WI from userspace only */
88+#define REG_HIDDEN_USER (1 << 1) /* hidden from userspace only */
89+#define REG_RAZ (1 << 2) /* RAZ from userspace and guest */
90+#define REG_USER_WI (1 << 3) /* WI from userspace only */
9091
9192 static __printf(2, 3)
9293 inline void print_sys_reg_msg(const struct sys_reg_params *p,
@@ -152,6 +153,15 @@ static inline bool sysreg_hidden(const struct kvm_vcpu *vcpu, · 59 unmodified lines
152153 return sysreg_visibility(vcpu, r) & REG_HIDDEN;
153154 }
154155
156+static inline bool sysreg_hidden_user(const struct kvm_vcpu *vcpu,
157+ const struct sys_reg_desc *r)
158+{
159+ if (likely(!r->visibility))
160+ return false;
161+
162+ return r->visibility(vcpu, r) & (REG_HIDDEN | REG_HIDDEN_USER);
163+}
164+
155165 static inline bool sysreg_visible_as_raz(const struct kvm_vcpu *vcpu,
156166 const struct sys_reg_desc *r)
157167 {
arch/arm64/kvm/trace_arm.h+59−0
@@ -2,6 +2,7 @@ · 1 unmodified line
22 #if !defined(_TRACE_ARM_ARM64_KVM_H) || defined(TRACE_HEADER_MULTI_READ)
33 #define _TRACE_ARM_ARM64_KVM_H
44
5+#include <asm/kvm_emulate.h>
56 #include <kvm/arm_arch_timer.h>
67 #include <linux/tracepoint.h>
78
@@ -301,6 +302,64 @@ TRACE_EVENT(kvm_timer_emulate, · 293 unmodified lines
301302 __entry->timer_idx, __entry->should_fire)
302303 );
303304
305+TRACE_EVENT(kvm_nested_eret,
306+ TP_PROTO(struct kvm_vcpu *vcpu, unsigned long elr_el2,
307+ unsigned long spsr_el2),
308+ TP_ARGS(vcpu, elr_el2, spsr_el2),
309+
310+ TP_STRUCT__entry(
311+ __field(struct kvm_vcpu *, vcpu)
312+ __field(unsigned long, elr_el2)
313+ __field(unsigned long, spsr_el2)
314+ __field(unsigned long, target_mode)
315+ __field(unsigned long, hcr_el2)
316+ ),
317+
318+ TP_fast_assign(
319+ __entry->vcpu = vcpu;
320+ __entry->elr_el2 = elr_el2;
321+ __entry->spsr_el2 = spsr_el2;
322+ __entry->target_mode = spsr_el2 & (PSR_MODE_MASK | PSR_MODE32_BIT);
323+ __entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
324+ ),
325+
326+ TP_printk("elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx",
327+ __entry->elr_el2, __entry->spsr_el2,
328+ __print_symbolic(__entry->target_mode, kvm_mode_names),
329+ __entry->hcr_el2)
330+);
331+
332+TRACE_EVENT(kvm_inject_nested_exception,
333+ TP_PROTO(struct kvm_vcpu *vcpu, u64 esr_el2, int type),
334+ TP_ARGS(vcpu, esr_el2, type),
335+
336+ TP_STRUCT__entry(
337+ __field(struct kvm_vcpu *, vcpu)
338+ __field(unsigned long, esr_el2)
339+ __field(int, type)
340+ __field(unsigned long, spsr_el2)
341+ __field(unsigned long, pc)
342+ __field(unsigned long, source_mode)
343+ __field(unsigned long, hcr_el2)
344+ ),
345+
346+ TP_fast_assign(
347+ __entry->vcpu = vcpu;
348+ __entry->esr_el2 = esr_el2;
349+ __entry->type = type;
350+ __entry->spsr_el2 = *vcpu_cpsr(vcpu);
351+ __entry->pc = *vcpu_pc(vcpu);
352+ __entry->source_mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT);
353+ __entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
354+ ),
355+
356+ TP_printk("%s: esr_el2 0x%lx elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx",
357+ __print_symbolic(__entry->type, kvm_exception_type_names),
358+ __entry->esr_el2, __entry->pc, __entry->spsr_el2,
359+ __print_symbolic(__entry->source_mode, kvm_mode_names),
360+ __entry->hcr_el2)
361+);
362+
304363 #endif /* _TRACE_ARM_ARM64_KVM_H */
305364
306365 #undef TRACE_INCLUDE_PATH
arch/arm64/kvm/vgic/vgic-init.c+3−18
@@ -465,17 +465,15 @@ out: · 464 unmodified lines
465465
466466 /* GENERIC PROBE */
467467
468−static int vgic_init_cpu_starting(unsigned int cpu)
468+void kvm_vgic_cpu_up(void)
469469 {
470470 enable_percpu_irq(kvm_vgic_global_state.maint_irq, 0);
471− return 0;
472471 }
473472
474473
475−static int vgic_init_cpu_dying(unsigned int cpu)
474+void kvm_vgic_cpu_down(void)
476475 {
477476 disable_percpu_irq(kvm_vgic_global_state.maint_irq);
478− return 0;
479477 }
480478
481479 static irqreturn_t vgic_maintenance_handler(int irq, void *data)
@@ -572,7 +570,7 @@ int kvm_vgic_hyp_init(void) · 90 unmodified lines
572570 if (ret)
573571 return ret;
574572
575− if (!has_mask)
573+ if (!has_mask && !kvm_vgic_global_state.maint_irq)
576574 return 0;
577575
578576 ret = request_percpu_irq(kvm_vgic_global_state.maint_irq,
@@ -584,19 +582,6 @@ int kvm_vgic_hyp_init(void) · 5 unmodified lines
584582 return ret;
585583 }
586584
587− ret = cpuhp_setup_state(CPUHP_AP_KVM_ARM_VGIC_INIT_STARTING,
588− "kvm/arm/vgic:starting",
589− vgic_init_cpu_starting, vgic_init_cpu_dying);
590− if (ret) {
591− kvm_err("Cannot register vgic CPU notifier\n");
592− goto out_free_irq;
593− }
594−
595585 kvm_info("vgic interrupt IRQ%d\n", kvm_vgic_global_state.maint_irq);
596586 return 0;
597−
598−out_free_irq:
599− free_percpu_irq(kvm_vgic_global_state.maint_irq,
600− kvm_get_running_vcpus());
601− return ret;
602587 }
arch/arm64/kvm/vgic/vgic-mmio.c+8−5
@@ -473,9 +473,10 @@ int vgic_uaccess_write_cpending(struct kvm_vcpu *vcpu, · 472 unmodified lines
473473 * active state can be overwritten when the VCPU's state is synced coming back
474474 * from the guest.
475475 *
476− * For shared interrupts as well as GICv3 private interrupts, we have to
477− * stop all the VCPUs because interrupts can be migrated while we don't hold
478− * the IRQ locks and we don't want to be chasing moving targets.
476+ * For shared interrupts as well as GICv3 private interrupts accessed from the
477+ * non-owning CPU, we have to stop all the VCPUs because interrupts can be
478+ * migrated while we don't hold the IRQ locks and we don't want to be chasing
479+ * moving targets.
479480 *
480481 * For GICv2 private interrupts we don't have to do anything because
481482 * userspace accesses to the VGIC state already require all VCPUs to be
@@ -484,7 +485,8 @@ int vgic_uaccess_write_cpending(struct kvm_vcpu *vcpu, · 2 unmodified lines
484485 */
485486 static void vgic_access_active_prepare(struct kvm_vcpu *vcpu, u32 intid)
486487 {
487− if (vcpu->kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3 ||
488+ if ((vcpu->kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3 &&
489+ vcpu != kvm_get_running_vcpu()) ||
488490 intid >= VGIC_NR_PRIVATE_IRQS)
489491 kvm_arm_halt_guest(vcpu->kvm);
490492 }
@@ -492,7 +494,8 @@ static void vgic_access_active_prepare(struct kvm_vcpu *vcpu, u32 intid) · 1 unmodified line
492494 /* See vgic_access_active_prepare */
493495 static void vgic_access_active_finish(struct kvm_vcpu *vcpu, u32 intid)
494496 {
495− if (vcpu->kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3 ||
497+ if ((vcpu->kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3 &&
498+ vcpu != kvm_get_running_vcpu()) ||
496499 intid >= VGIC_NR_PRIVATE_IRQS)
497500 kvm_arm_resume_guest(vcpu->kvm);
498501 }
arch/arm64/kvm/vgic/vgic-v3.c+5−4
@@ -3,6 +3,7 @@ · 2 unmodified lines
33 #include <linux/irqchip/arm-gic-v3.h>
44 #include <linux/irq.h>
55 #include <linux/irqdomain.h>
6+#include <linux/kstrtox.h>
67 #include <linux/kvm.h>
78 #include <linux/kvm_host.h>
89 #include <kvm/arm_vgic.h>
@@ -584,25 +585,25 @@ DEFINE_STATIC_KEY_FALSE(vgic_v3_cpuif_trap); · 575 unmodified lines
584585
585586 static int __init early_group0_trap_cfg(char *buf)
586587 {
587− return strtobool(buf, &group0_trap);
588+ return kstrtobool(buf, &group0_trap);
588589 }
589590 early_param("kvm-arm.vgic_v3_group0_trap", early_group0_trap_cfg);
590591
591592 static int __init early_group1_trap_cfg(char *buf)
592593 {
593− return strtobool(buf, &group1_trap);
594+ return kstrtobool(buf, &group1_trap);
594595 }
595596 early_param("kvm-arm.vgic_v3_group1_trap", early_group1_trap_cfg);
596597
597598 static int __init early_common_trap_cfg(char *buf)
598599 {
599− return strtobool(buf, &common_trap);
600+ return kstrtobool(buf, &common_trap);
600601 }
601602 early_param("kvm-arm.vgic_v3_common_trap", early_common_trap_cfg);
602603
603604 static int __init early_gicv4_enable(char *buf)
604605 {
605− return strtobool(buf, &gicv4_enable);
606+ return kstrtobool(buf, &gicv4_enable);
606607 }
607608 early_param("kvm-arm.vgic_v4_enable", early_gicv4_enable);
608609
arch/arm64/kvm/vmid.c+3−3
@@ -16,7 +16,7 @@ · 15 unmodified lines
1616 #include <asm/kvm_asm.h>
1717 #include <asm/kvm_mmu.h>
1818
19−unsigned int kvm_arm_vmid_bits;
19+unsigned int __ro_after_init kvm_arm_vmid_bits;
2020 static DEFINE_RAW_SPINLOCK(cpu_vmid_lock);
2121
2222 static atomic64_t vmid_generation;
@@ -172,7 +172,7 @@ void kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid) · 149 unmodified lines
172172 /*
173173 * Initialize the VMID allocator
174174 */
175−int kvm_arm_vmid_alloc_init(void)
175+int __init kvm_arm_vmid_alloc_init(void)
176176 {
177177 kvm_arm_vmid_bits = kvm_get_vmid_bits();
178178
@@ -190,7 +190,7 @@ int kvm_arm_vmid_alloc_init(void) · 11 unmodified lines
190190 return 0;
191191 }
192192
193−void kvm_arm_vmid_alloc_free(void)
193+void __init kvm_arm_vmid_alloc_free(void)
194194 {
195195 kfree(vmid_map);
196196 }
arch/arm64/tools/cpucaps+1−0
@@ -33,6 +33,7 @@ HAS_GIC_PRIO_MASKING · 32 unmodified lines
3333 HAS_GIC_PRIO_RELAXED_SYNC
3434 HAS_LDAPR
3535 HAS_LSE_ATOMICS
36+HAS_NESTED_VIRT
3637 HAS_NO_FPSIMD
3738 HAS_NO_HW_PREFETCH
3839 HAS_PAN
arch/arm64/tools/gen-sysreg.awk+19−1
@@ -103,6 +103,7 @@ END { · 102 unmodified lines
103103
104104 res0 = "UL(0)"
105105 res1 = "UL(0)"
106+ unkn = "UL(0)"
106107
107108 next_bit = 63
108109
@@ -117,11 +118,13 @@ END { · 8 unmodified lines
117118
118119 define(reg "_RES0", "(" res0 ")")
119120 define(reg "_RES1", "(" res1 ")")
121+ define(reg "_UNKN", "(" unkn ")")
120122 print ""
121123
122124 reg = null
123125 res0 = null
124126 res1 = null
127+ unkn = null
125128
126129 next
127130 }
@@ -139,6 +142,7 @@ END { · 11 unmodified lines
139142
140143 res0 = "UL(0)"
141144 res1 = "UL(0)"
145+ unkn = "UL(0)"
142146
143147 define("REG_" reg, "S" op0 "_" op1 "_C" crn "_C" crm "_" op2)
144148 define("SYS_" reg, "sys_reg(" op0 ", " op1 ", " crn ", " crm ", " op2 ")")
@@ -166,7 +170,9 @@ END { · 21 unmodified lines
166170 define(reg "_RES0", "(" res0 ")")
167171 if (res1 != null)
168172 define(reg "_RES1", "(" res1 ")")
169− if (res0 != null || res1 != null)
173+ if (unkn != null)
174+ define(reg "_UNKN", "(" unkn ")")
175+ if (res0 != null || res1 != null || unkn != null)
170176 print ""
171177
172178 reg = null
@@ -177,6 +183,7 @@ END { · 4 unmodified lines
177183 op2 = null
178184 res0 = null
179185 res1 = null
186+ unkn = null
180187
181188 next
182189 }
@@ -195,6 +202,7 @@ END { · 12 unmodified lines
195202 next_bit = 0
196203 res0 = null
197204 res1 = null
205+ unkn = null
198206
199207 next
200208 }
@@ -220,6 +228,16 @@ END { · 19 unmodified lines
220228 next
221229 }
222230
231+/^Unkn/ && (block == "Sysreg" || block == "SysregFields") {
232+ expect_fields(2)
233+ parse_bitdef(reg, "UNKN", $2)
234+ field = "UNKN_" msb "_" lsb
235+
236+ unkn = unkn " | GENMASK_ULL(" msb ", " lsb ")"
237+
238+ next
239+}
240+
223241 /^Field/ && (block == "Sysreg" || block == "SysregFields") {
224242 expect_fields(3)
225243 field = $3
arch/arm64/tools/sysreg+17−0
@@ -15,6 +15,8 @@ · 14 unmodified lines
1515
1616 # Res1 <msb>[:<lsb>]
1717
18+# Unkn <msb>[:<lsb>]
19+
1820 # Field <msb>[:<lsb>] <name>
1921
2022 # Enum <msb>[:<lsb>] <name>
@@ -1779,6 +1781,16 @@ Sysreg SCXTNUM_EL1 3 0 13 0 7 · 1758 unmodified lines
17791781 Field 63:0 SoftwareContextNumber
17801782 EndSysreg
17811783
1784+# The bit layout for CCSIDR_EL1 depends on whether FEAT_CCIDX is implemented.
1785+# The following is for case when FEAT_CCIDX is not implemented.
1786+Sysreg CCSIDR_EL1 3 1 0 0 0
1787+Res0 63:32
1788+Unkn 31:28
1789+Field 27:13 NumSets
1790+Field 12:3 Associativity
1791+Field 2:0 LineSize
1792+EndSysreg
1793+
17821794 Sysreg CLIDR_EL1 3 1 0 0 1
17831795 Res0 63:47
17841796 Field 46:33 Ttypen
@@ -1795,6 +1807,11 @@ Field 5:3 Ctype2 · 10 unmodified lines
17951807 Field 2:0 Ctype1
17961808 EndSysreg
17971809
1810+Sysreg CCSIDR2_EL1 3 1 0 0 2
1811+Res0 63:24
1812+Field 23:0 NumSets
1813+EndSysreg
1814+
17981815 Sysreg GMID_EL1 3 1 0 0 4
17991816 Res0 63:4
18001817 Field 3:0 BS
arch/mips/include/asm/kvm_host.h+1−2
@@ -758,7 +758,7 @@ struct kvm_mips_callbacks { · 757 unmodified lines
758758 void (*vcpu_reenter)(struct kvm_vcpu *vcpu);
759759 };
760760 extern struct kvm_mips_callbacks *kvm_mips_callbacks;
761−int kvm_mips_emulation_init(struct kvm_mips_callbacks **install_callbacks);
761+int kvm_mips_emulation_init(void);
762762
763763 /* Debug: dump vcpu state */
764764 int kvm_arch_vcpu_dump_regs(struct kvm_vcpu *vcpu);
@@ -888,7 +888,6 @@ extern unsigned long kvm_mips_get_ramsize(struct kvm *kvm); · 123 unmodified lines
888888 extern int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu,
889889 struct kvm_mips_interrupt *irq);
890890
891−static inline void kvm_arch_hardware_unsetup(void) {}
892891 static inline void kvm_arch_sync_events(struct kvm *kvm) {}
893892 static inline void kvm_arch_free_memslot(struct kvm *kvm,
894893 struct kvm_memory_slot *slot) {}
arch/mips/kvm/Kconfig+1−0
@@ -28,6 +28,7 @@ config KVM · 27 unmodified lines
2828 select MMU_NOTIFIER
2929 select SRCU
3030 select INTERVAL_TREE
31+ select KVM_GENERIC_HARDWARE_ENABLING
3132 help
3233 Support for hosting Guest kernels.
3334
arch/mips/kvm/Makefile+1−1
@@ -17,4 +17,4 @@ kvm-$(CONFIG_CPU_LOONGSON64) += loongson_ipi.o · 16 unmodified lines
1717
1818 kvm-y += vz.o
1919 obj-$(CONFIG_KVM) += kvm.o
20−obj-y += callback.o tlb.o
20+obj-y += tlb.o
arch/mips/kvm/callback.c+0−14
@@ -1,14 +0,0 @@
1−/*
2− * This file is subject to the terms and conditions of the GNU General Public
3− * License. See the file "COPYING" in the main directory of this archive
4− * for more details.
5− *
6− * Copyright (C) 2012 MIPS Technologies, Inc. All rights reserved.
7− * Authors: Yann Le Du <ledu@kymasys.com>
8− */
9−
10−#include <linux/export.h>
11−#include <linux/kvm_host.h>
12−
13−struct kvm_mips_callbacks *kvm_mips_callbacks;
14−EXPORT_SYMBOL_GPL(kvm_mips_callbacks);
arch/mips/kvm/mips.c+7−27
@@ -135,16 +135,6 @@ void kvm_arch_hardware_disable(void) · 134 unmodified lines
135135 kvm_mips_callbacks->hardware_disable();
136136 }
137137
138−int kvm_arch_hardware_setup(void *opaque)
139−{
140− return 0;
141−}
142−
143−int kvm_arch_check_processor_compat(void *opaque)
144−{
145− return 0;
146−}
147−
148138 extern void kvm_init_loongson_ipi(struct kvm *kvm);
149139
150140 int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
@@ -1015,21 +1005,6 @@ long kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) · 864 unmodified lines
10151005 return r;
10161006 }
10171007
1018−int kvm_arch_init(void *opaque)
1019−{
1020− if (kvm_mips_callbacks) {
1021− kvm_err("kvm: module already exists\n");
1022− return -EEXIST;
1023− }
1024−
1025− return kvm_mips_emulation_init(&kvm_mips_callbacks);
1026−}
1027−
1028−void kvm_arch_exit(void)
1029−{
1030− kvm_mips_callbacks = NULL;
1031−}
1032−
10331008 int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu,
10341009 struct kvm_sregs *sregs)
10351010 {
@@ -1646,16 +1621,21 @@ static int __init kvm_mips_init(void) · 610 unmodified lines
16461621 if (ret)
16471622 return ret;
16481623
1649− ret = kvm_init(NULL, sizeof(struct kvm_vcpu), 0, THIS_MODULE);
1650−
1624+ ret = kvm_mips_emulation_init();
16511625 if (ret)
16521626 return ret;
16531627
1628+
16541629 if (boot_cpu_type() == CPU_LOONGSON64)
16551630 kvm_priority_to_irq = kvm_loongson3_priority_to_irq;
16561631
16571632 register_die_notifier(&kvm_mips_csr_die_notifier);
16581633
1634+ ret = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
1635+ if (ret) {
1636+ unregister_die_notifier(&kvm_mips_csr_die_notifier);
1637+ return ret;
1638+ }
16591639 return 0;
16601640 }
16611641
arch/mips/kvm/vz.c+4−3
@@ -3304,7 +3304,10 @@ static struct kvm_mips_callbacks kvm_vz_callbacks = { · 3303 unmodified lines
33043304 .vcpu_reenter = kvm_vz_vcpu_reenter,
33053305 };
33063306
3307−int kvm_mips_emulation_init(struct kvm_mips_callbacks **install_callbacks)
3307+/* FIXME: Get rid of the callbacks now that trap-and-emulate is gone. */
3308+struct kvm_mips_callbacks *kvm_mips_callbacks = &kvm_vz_callbacks;
3309+
3310+int kvm_mips_emulation_init(void)
33083311 {
33093312 if (!cpu_has_vz)
33103313 return -ENODEV;
@@ -3318,7 +3321,5 @@ int kvm_mips_emulation_init(struct kvm_mips_callbacks **install_callbacks) · 7 unmodified lines
33183321 return -ENODEV;
33193322
33203323 pr_info("Starting KVM with MIPS VZ extensions\n");
3321−
3322− *install_callbacks = &kvm_vz_callbacks;
33233324 return 0;
33243325 }
arch/powerpc/include/asm/kvm_host.h+0−3
@@ -876,13 +876,10 @@ struct kvm_vcpu_arch { · 875 unmodified lines
876876 #define __KVM_HAVE_ARCH_WQP
877877 #define __KVM_HAVE_CREATE_DEVICE
878878
879−static inline void kvm_arch_hardware_disable(void) {}
880−static inline void kvm_arch_hardware_unsetup(void) {}
881879 static inline void kvm_arch_sync_events(struct kvm *kvm) {}
882880 static inline void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) {}
883881 static inline void kvm_arch_flush_shadow_all(struct kvm *kvm) {}
884882 static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
885−static inline void kvm_arch_exit(void) {}
886883 static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
887884 static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
888885
arch/powerpc/include/asm/kvm_ppc.h+0−1
@@ -118,7 +118,6 @@ extern int kvmppc_xlate(struct kvm_vcpu *vcpu, ulong eaddr, · 117 unmodified lines
118118 extern int kvmppc_core_vcpu_create(struct kvm_vcpu *vcpu);
119119 extern void kvmppc_core_vcpu_free(struct kvm_vcpu *vcpu);
120120 extern int kvmppc_core_vcpu_setup(struct kvm_vcpu *vcpu);
121−extern int kvmppc_core_check_processor_compat(void);
122121 extern int kvmppc_core_vcpu_translate(struct kvm_vcpu *vcpu,
123122 struct kvm_translation *tr);
124123
arch/powerpc/kvm/book3s.c+1−11
@@ -999,16 +999,6 @@ int kvmppc_h_logical_ci_store(struct kvm_vcpu *vcpu) · 998 unmodified lines
999999 }
10001000 EXPORT_SYMBOL_GPL(kvmppc_h_logical_ci_store);
10011001
1002−int kvmppc_core_check_processor_compat(void)
1003−{
1004− /*
1005− * We always return 0 for book3s. We check
1006− * for compatibility while loading the HV
1007− * or PR module
1008− */
1009− return 0;
1010−}
1011−
10121002 int kvmppc_book3s_hcall_implemented(struct kvm *kvm, unsigned long hcall)
10131003 {
10141004 return kvm->arch.kvm_ops->hcall_implemented(hcall);
@@ -1062,7 +1052,7 @@ static int kvmppc_book3s_init(void) · 47 unmodified lines
10621052 {
10631053 int r;
10641054
1065− r = kvm_init(NULL, sizeof(struct kvm_vcpu), 0, THIS_MODULE);
1055+ r = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
10661056 if (r)
10671057 return r;
10681058 #ifdef CONFIG_KVM_BOOK3S_32_HANDLER
arch/powerpc/kvm/booke.c+1−1
@@ -1210,7 +1210,7 @@ int kvmppc_handle_exit(struct kvm_vcpu *vcpu, unsigned int exit_nr) · 1209 unmodified lines
12101210
12111211 /*
12121212 * On cores with Vector category, KVM is loaded only if CONFIG_ALTIVEC,
1213− * see kvmppc_core_check_processor_compat().
1213+ * see kvmppc_e500mc_check_processor_compat().
12141214 */
12151215 #ifdef CONFIG_ALTIVEC
12161216 case BOOKE_INTERRUPT_ALTIVEC_UNAVAIL:
arch/powerpc/kvm/e500.c+3−3
@@ -314,7 +314,7 @@ static void kvmppc_core_vcpu_put_e500(struct kvm_vcpu *vcpu) · 313 unmodified lines
314314 kvmppc_booke_vcpu_put(vcpu);
315315 }
316316
317−int kvmppc_core_check_processor_compat(void)
317+static int kvmppc_e500_check_processor_compat(void)
318318 {
319319 int r;
320320
@@ -507,7 +507,7 @@ static int __init kvmppc_e500_init(void) · 186 unmodified lines
507507 unsigned long handler_len;
508508 unsigned long max_ivor = 0;
509509
510− r = kvmppc_core_check_processor_compat();
510+ r = kvmppc_e500_check_processor_compat();
511511 if (r)
512512 goto err_out;
513513
@@ -531,7 +531,7 @@ static int __init kvmppc_e500_init(void) · 17 unmodified lines
531531 flush_icache_range(kvmppc_booke_handlers, kvmppc_booke_handlers +
532532 ivor[max_ivor] + handler_len);
533533
534− r = kvm_init(NULL, sizeof(struct kvmppc_vcpu_e500), 0, THIS_MODULE);
534+ r = kvm_init(sizeof(struct kvmppc_vcpu_e500), 0, THIS_MODULE);
535535 if (r)
536536 goto err_out;
537537 kvm_ops_e500.owner = THIS_MODULE;
arch/powerpc/kvm/e500mc.c+6−2
@@ -168,7 +168,7 @@ static void kvmppc_core_vcpu_put_e500mc(struct kvm_vcpu *vcpu) · 167 unmodified lines
168168 kvmppc_booke_vcpu_put(vcpu);
169169 }
170170
171−int kvmppc_core_check_processor_compat(void)
171+int kvmppc_e500mc_check_processor_compat(void)
172172 {
173173 int r;
174174
@@ -388,6 +388,10 @@ static int __init kvmppc_e500mc_init(void) · 213 unmodified lines
388388 {
389389 int r;
390390
391+ r = kvmppc_e500mc_check_processor_compat();
392+ if (r)
393+ goto err_out;
394+
391395 r = kvmppc_booke_init();
392396 if (r)
393397 goto err_out;
@@ -400,7 +404,7 @@ static int __init kvmppc_e500mc_init(void) · 6 unmodified lines
400404 */
401405 kvmppc_init_lpid(KVMPPC_NR_LPIDS/threads_per_core);
402406
403− r = kvm_init(NULL, sizeof(struct kvmppc_vcpu_e500), 0, THIS_MODULE);
407+ r = kvm_init(sizeof(struct kvmppc_vcpu_e500), 0, THIS_MODULE);
404408 if (r)
405409 goto err_out;
406410 kvm_ops_e500mc.owner = THIS_MODULE;
arch/powerpc/kvm/powerpc.c+0−20
@@ -435,21 +435,6 @@ int kvmppc_ld(struct kvm_vcpu *vcpu, ulong *eaddr, int size, void *ptr, · 434 unmodified lines
435435 }
436436 EXPORT_SYMBOL_GPL(kvmppc_ld);
437437
438−int kvm_arch_hardware_enable(void)
439−{
440− return 0;
441−}
442−
443−int kvm_arch_hardware_setup(void *opaque)
444−{
445− return 0;
446−}
447−
448−int kvm_arch_check_processor_compat(void *opaque)
449−{
450− return kvmppc_core_check_processor_compat();
451−}
452−
453438 int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
454439 {
455440 struct kvmppc_ops *kvm_ops = NULL;
@@ -2544,11 +2529,6 @@ void kvmppc_init_lpid(unsigned long nr_lpids_param) · 2088 unmodified lines
25442529 }
25452530 EXPORT_SYMBOL_GPL(kvmppc_init_lpid);
25462531
2547−int kvm_arch_init(void *opaque)
2548−{
2549− return 0;
2550−}
2551−
25522532 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_ppc_instr);
25532533
25542534 void kvm_arch_create_vcpu_debugfs(struct kvm_vcpu *vcpu, struct dentry *debugfs_dentry)
arch/riscv/include/asm/kvm_host.h+7−4
@@ -18,6 +18,7 @@ · 17 unmodified lines
1818 #include <asm/kvm_vcpu_insn.h>
1919 #include <asm/kvm_vcpu_sbi.h>
2020 #include <asm/kvm_vcpu_timer.h>
21+#include <asm/kvm_vcpu_pmu.h>
2122
2223 #define KVM_MAX_VCPUS 1024
2324
@@ -228,9 +229,11 @@ struct kvm_vcpu_arch { · 204 unmodified lines
228229
229230 /* Don't run the VCPU (blocked) */
230231 bool pause;
232+
233+ /* Performance monitoring context */
234+ struct kvm_pmu pmu_context;
231235 };
232236
233−static inline void kvm_arch_hardware_unsetup(void) {}
234237 static inline void kvm_arch_sync_events(struct kvm *kvm) {}
235238 static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
236239
@@ -297,11 +300,11 @@ int kvm_riscv_gstage_map(struct kvm_vcpu *vcpu, · 60 unmodified lines
297300 int kvm_riscv_gstage_alloc_pgd(struct kvm *kvm);
298301 void kvm_riscv_gstage_free_pgd(struct kvm *kvm);
299302 void kvm_riscv_gstage_update_hgatp(struct kvm_vcpu *vcpu);
300−void kvm_riscv_gstage_mode_detect(void);
301−unsigned long kvm_riscv_gstage_mode(void);
303+void __init kvm_riscv_gstage_mode_detect(void);
304+unsigned long __init kvm_riscv_gstage_mode(void);
302305 int kvm_riscv_gstage_gpa_bits(void);
303306
304−void kvm_riscv_gstage_vmid_detect(void);
307+void __init kvm_riscv_gstage_vmid_detect(void);
305308 unsigned long kvm_riscv_gstage_vmid_bits(void);
306309 int kvm_riscv_gstage_vmid_init(struct kvm *kvm);
307310 bool kvm_riscv_gstage_vmid_ver_changed(struct kvm_vmid *vmid);
arch/riscv/include/asm/kvm_vcpu_pmu.h+107−0
@@ -0,0 +1,107 @@
1+/* SPDX-License-Identifier: GPL-2.0-only */
2+/*
3+ * Copyright (c) 2023 Rivos Inc
4+ *
5+ * Authors:
6+ * Atish Patra <atishp@rivosinc.com>
7+ */
8+
9+#ifndef __KVM_VCPU_RISCV_PMU_H
10+#define __KVM_VCPU_RISCV_PMU_H
11+
12+#include <linux/perf/riscv_pmu.h>
13+#include <asm/sbi.h>
14+
15+#ifdef CONFIG_RISCV_PMU_SBI
16+#define RISCV_KVM_MAX_FW_CTRS 32
17+#define RISCV_KVM_MAX_HW_CTRS 32
18+#define RISCV_KVM_MAX_COUNTERS (RISCV_KVM_MAX_HW_CTRS + RISCV_KVM_MAX_FW_CTRS)
19+static_assert(RISCV_KVM_MAX_COUNTERS <= 64);
20+
21+struct kvm_fw_event {
22+ /* Current value of the event */
23+ unsigned long value;
24+
25+ /* Event monitoring status */
26+ bool started;
27+};
28+
29+/* Per virtual pmu counter data */
30+struct kvm_pmc {
31+ u8 idx;
32+ struct perf_event *perf_event;
33+ u64 counter_val;
34+ union sbi_pmu_ctr_info cinfo;
35+ /* Event monitoring status */
36+ bool started;
37+ /* Monitoring event ID */
38+ unsigned long event_idx;
39+};
40+
41+/* PMU data structure per vcpu */
42+struct kvm_pmu {
43+ struct kvm_pmc pmc[RISCV_KVM_MAX_COUNTERS];
44+ struct kvm_fw_event fw_event[RISCV_KVM_MAX_FW_CTRS];
45+ /* Number of the virtual firmware counters available */
46+ int num_fw_ctrs;
47+ /* Number of the virtual hardware counters available */
48+ int num_hw_ctrs;
49+ /* A flag to indicate that pmu initialization is done */
50+ bool init_done;
51+ /* Bit map of all the virtual counter used */
52+ DECLARE_BITMAP(pmc_in_use, RISCV_KVM_MAX_COUNTERS);
53+};
54+
55+#define vcpu_to_pmu(vcpu) (&(vcpu)->arch.pmu_context)
56+#define pmu_to_vcpu(pmu) (container_of((pmu), struct kvm_vcpu, arch.pmu_context))
57+
58+#if defined(CONFIG_32BIT)
59+#define KVM_RISCV_VCPU_HPMCOUNTER_CSR_FUNCS \
60+{.base = CSR_CYCLEH, .count = 31, .func = kvm_riscv_vcpu_pmu_read_hpm }, \
61+{.base = CSR_CYCLE, .count = 31, .func = kvm_riscv_vcpu_pmu_read_hpm },
62+#else
63+#define KVM_RISCV_VCPU_HPMCOUNTER_CSR_FUNCS \
64+{.base = CSR_CYCLE, .count = 31, .func = kvm_riscv_vcpu_pmu_read_hpm },
65+#endif
66+
67+int kvm_riscv_vcpu_pmu_incr_fw(struct kvm_vcpu *vcpu, unsigned long fid);
68+int kvm_riscv_vcpu_pmu_read_hpm(struct kvm_vcpu *vcpu, unsigned int csr_num,
69+ unsigned long *val, unsigned long new_val,
70+ unsigned long wr_mask);
71+
72+int kvm_riscv_vcpu_pmu_num_ctrs(struct kvm_vcpu *vcpu, struct kvm_vcpu_sbi_return *retdata);
73+int kvm_riscv_vcpu_pmu_ctr_info(struct kvm_vcpu *vcpu, unsigned long cidx,
74+ struct kvm_vcpu_sbi_return *retdata);
75+int kvm_riscv_vcpu_pmu_ctr_start(struct kvm_vcpu *vcpu, unsigned long ctr_base,
76+ unsigned long ctr_mask, unsigned long flags, u64 ival,
77+ struct kvm_vcpu_sbi_return *retdata);
78+int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
79+ unsigned long ctr_mask, unsigned long flags,
80+ struct kvm_vcpu_sbi_return *retdata);
81+int kvm_riscv_vcpu_pmu_ctr_cfg_match(struct kvm_vcpu *vcpu, unsigned long ctr_base,
82+ unsigned long ctr_mask, unsigned long flags,
83+ unsigned long eidx, u64 evtdata,
84+ struct kvm_vcpu_sbi_return *retdata);
85+int kvm_riscv_vcpu_pmu_ctr_read(struct kvm_vcpu *vcpu, unsigned long cidx,
86+ struct kvm_vcpu_sbi_return *retdata);
87+void kvm_riscv_vcpu_pmu_init(struct kvm_vcpu *vcpu);
88+void kvm_riscv_vcpu_pmu_deinit(struct kvm_vcpu *vcpu);
89+void kvm_riscv_vcpu_pmu_reset(struct kvm_vcpu *vcpu);
90+
91+#else
92+struct kvm_pmu {
93+};
94+
95+#define KVM_RISCV_VCPU_HPMCOUNTER_CSR_FUNCS \
96+{.base = 0, .count = 0, .func = NULL },
97+
98+static inline void kvm_riscv_vcpu_pmu_init(struct kvm_vcpu *vcpu) {}
99+static inline int kvm_riscv_vcpu_pmu_incr_fw(struct kvm_vcpu *vcpu, unsigned long fid)
100+{
101+ return 0;
102+}
103+
104+static inline void kvm_riscv_vcpu_pmu_deinit(struct kvm_vcpu *vcpu) {}
105+static inline void kvm_riscv_vcpu_pmu_reset(struct kvm_vcpu *vcpu) {}
106+#endif /* CONFIG_RISCV_PMU_SBI */
107+#endif /* !__KVM_VCPU_RISCV_PMU_H */
arch/riscv/include/asm/kvm_vcpu_sbi.h+11−2
@@ -18,6 +18,13 @@ struct kvm_vcpu_sbi_context { · 17 unmodified lines
1818 int return_handled;
1919 };
2020
21+struct kvm_vcpu_sbi_return {
22+ unsigned long out_val;
23+ unsigned long err_val;
24+ struct kvm_cpu_trap *utrap;
25+ bool uexit;
26+};
27+
2128 struct kvm_vcpu_sbi_extension {
2229 unsigned long extid_start;
2330 unsigned long extid_end;
@@ -27,8 +34,10 @@ struct kvm_vcpu_sbi_extension { · 3 unmodified lines
2734 * specific error codes.
2835 */
2936 int (*handler)(struct kvm_vcpu *vcpu, struct kvm_run *run,
30− unsigned long *out_val, struct kvm_cpu_trap *utrap,
31− bool *exit);
37+ struct kvm_vcpu_sbi_return *retdata);
38+
39+ /* Extension specific probe function */
40+ unsigned long (*probe)(struct kvm_vcpu *vcpu);
3241 };
3342
3443 void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run);
arch/riscv/include/asm/sbi.h+5−2
@@ -169,9 +169,9 @@ enum sbi_pmu_fw_generic_events_t { · 168 unmodified lines
169169 SBI_PMU_FW_ILLEGAL_INSN = 4,
170170 SBI_PMU_FW_SET_TIMER = 5,
171171 SBI_PMU_FW_IPI_SENT = 6,
172− SBI_PMU_FW_IPI_RECVD = 7,
172+ SBI_PMU_FW_IPI_RCVD = 7,
173173 SBI_PMU_FW_FENCE_I_SENT = 8,
174− SBI_PMU_FW_FENCE_I_RECVD = 9,
174+ SBI_PMU_FW_FENCE_I_RCVD = 9,
175175 SBI_PMU_FW_SFENCE_VMA_SENT = 10,
176176 SBI_PMU_FW_SFENCE_VMA_RCVD = 11,
177177 SBI_PMU_FW_SFENCE_VMA_ASID_SENT = 12,
@@ -215,6 +215,9 @@ enum sbi_pmu_ctr_type { · 37 unmodified lines
215215 #define SBI_PMU_EVENT_CACHE_OP_ID_CODE_MASK 0x06
216216 #define SBI_PMU_EVENT_CACHE_RESULT_ID_CODE_MASK 0x01
217217
218+#define SBI_PMU_EVENT_CACHE_ID_SHIFT 3
219+#define SBI_PMU_EVENT_CACHE_OP_SHIFT 1
220+
218221 #define SBI_PMU_EVENT_IDX_INVALID 0xFFFFFFFF
219222
220223 /* Flags defined for config matching function */
arch/riscv/kvm/Kconfig+1−0
@@ -20,6 +20,7 @@ if VIRTUALIZATION · 19 unmodified lines
2020 config KVM
2121 tristate "Kernel-based Virtual Machine (KVM) support (EXPERIMENTAL)"
2222 depends on RISCV_SBI && MMU
23+ select KVM_GENERIC_HARDWARE_ENABLING
2324 select MMU_NOTIFIER
2425 select PREEMPT_NOTIFIERS
2526 select KVM_MMIO
arch/riscv/kvm/Makefile+1−0
@@ -25,3 +25,4 @@ kvm-y += vcpu_sbi_base.o · 24 unmodified lines
2525 kvm-y += vcpu_sbi_replace.o
2626 kvm-y += vcpu_sbi_hsm.o
2727 kvm-y += vcpu_timer.o
28+kvm-$(CONFIG_RISCV_PMU_SBI) += vcpu_pmu.o vcpu_sbi_pmu.o
arch/riscv/kvm/main.c+4−22
@@ -20,16 +20,6 @@ long kvm_arch_dev_ioctl(struct file *filp, · 19 unmodified lines
2020 return -EINVAL;
2121 }
2222
23−int kvm_arch_check_processor_compat(void *opaque)
24−{
25− return 0;
26−}
27−
28−int kvm_arch_hardware_setup(void *opaque)
29−{
30− return 0;
31−}
32−
3323 int kvm_arch_hardware_enable(void)
3424 {
3525 unsigned long hideleg, hedeleg;
@@ -49,7 +39,8 @@ int kvm_arch_hardware_enable(void) · 13 unmodified lines
4939 hideleg |= (1UL << IRQ_VS_EXT);
5040 csr_write(CSR_HIDELEG, hideleg);
5141
52− csr_write(CSR_HCOUNTEREN, -1UL);
42+ /* VS should access only the time counter directly. Everything else should trap */
43+ csr_write(CSR_HCOUNTEREN, 0x02);
5344
5445 csr_write(CSR_HVIP, 0);
5546
@@ -70,7 +61,7 @@ void kvm_arch_hardware_disable(void) · 14 unmodified lines
7061 csr_write(CSR_HIDELEG, 0);
7162 }
7263
73−int kvm_arch_init(void *opaque)
64+static int __init riscv_kvm_init(void)
7465 {
7566 const char *str;
7667
@@ -115,16 +106,7 @@ int kvm_arch_init(void *opaque) · 38 unmodified lines
115106
116107 kvm_info("VMID %ld bits available\n", kvm_riscv_gstage_vmid_bits());
117108
118− return 0;
119−}
120−
121−void kvm_arch_exit(void)
122−{
123−}
124−
125−static int __init riscv_kvm_init(void)
126−{
127− return kvm_init(NULL, sizeof(struct kvm_vcpu), 0, THIS_MODULE);
109+ return kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
128110 }
129111 module_init(riscv_kvm_init);
130112
arch/riscv/kvm/mmu.c+10−10
@@ -20,12 +20,12 @@ · 19 unmodified lines
2020 #include <asm/pgtable.h>
2121
2222 #ifdef CONFIG_64BIT
23−static unsigned long gstage_mode = (HGATP_MODE_SV39X4 << HGATP_MODE_SHIFT);
24−static unsigned long gstage_pgd_levels = 3;
23+static unsigned long gstage_mode __ro_after_init = (HGATP_MODE_SV39X4 << HGATP_MODE_SHIFT);
24+static unsigned long gstage_pgd_levels __ro_after_init = 3;
2525 #define gstage_index_bits 9
2626 #else
27−static unsigned long gstage_mode = (HGATP_MODE_SV32X4 << HGATP_MODE_SHIFT);
28−static unsigned long gstage_pgd_levels = 2;
27+static unsigned long gstage_mode __ro_after_init = (HGATP_MODE_SV32X4 << HGATP_MODE_SHIFT);
28+static unsigned long gstage_pgd_levels __ro_after_init = 2;
2929 #define gstage_index_bits 10
3030 #endif
3131
@@ -585,7 +585,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · 553 unmodified lines
585585 if (!kvm->arch.pgd)
586586 return false;
587587
588− WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PGDIR_SIZE);
588+ WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE);
589589
590590 if (!gstage_get_leaf_entry(kvm, range->start << PAGE_SHIFT,
591591 &ptep, &ptep_level))
@@ -603,7 +603,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · 11 unmodified lines
603603 if (!kvm->arch.pgd)
604604 return false;
605605
606− WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PGDIR_SIZE);
606+ WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE);
607607
608608 if (!gstage_get_leaf_entry(kvm, range->start << PAGE_SHIFT,
609609 &ptep, &ptep_level))
@@ -645,12 +645,12 @@ int kvm_riscv_gstage_map(struct kvm_vcpu *vcpu, · 35 unmodified lines
645645 if (logging || (vma->vm_flags & VM_PFNMAP))
646646 vma_pagesize = PAGE_SIZE;
647647
648− if (vma_pagesize == PMD_SIZE || vma_pagesize == PGDIR_SIZE)
648+ if (vma_pagesize == PMD_SIZE || vma_pagesize == PUD_SIZE)
649649 gfn = (gpa & huge_page_mask(hstate_vma(vma))) >> PAGE_SHIFT;
650650
651651 mmap_read_unlock(current->mm);
652652
653− if (vma_pagesize != PGDIR_SIZE &&
653+ if (vma_pagesize != PUD_SIZE &&
654654 vma_pagesize != PMD_SIZE &&
655655 vma_pagesize != PAGE_SIZE) {
656656 kvm_err("Invalid VMA page size 0x%lx\n", vma_pagesize);
@@ -758,7 +758,7 @@ void kvm_riscv_gstage_update_hgatp(struct kvm_vcpu *vcpu) · 101 unmodified lines
758758 kvm_riscv_local_hfence_gvma_all();
759759 }
760760
761−void kvm_riscv_gstage_mode_detect(void)
761+void __init kvm_riscv_gstage_mode_detect(void)
762762 {
763763 #ifdef CONFIG_64BIT
764764 /* Try Sv57x4 G-stage mode */
@@ -782,7 +782,7 @@ skip_sv48x4_test: · 17 unmodified lines
782782 #endif
783783 }
784784
785−unsigned long kvm_riscv_gstage_mode(void)
785+unsigned long __init kvm_riscv_gstage_mode(void)
786786 {
787787 return gstage_mode >> HGATP_MODE_SHIFT;
788788 }
arch/riscv/kvm/tlb.c+4−0
@@ -180,6 +180,7 @@ void kvm_riscv_local_tlb_sanitize(struct kvm_vcpu *vcpu) · 179 unmodified lines
180180
181181 void kvm_riscv_fence_i_process(struct kvm_vcpu *vcpu)
182182 {
183+ kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_FENCE_I_RCVD);
183184 local_flush_icache_all();
184185 }
185186
@@ -263,15 +264,18 @@ void kvm_riscv_hfence_process(struct kvm_vcpu *vcpu) · 77 unmodified lines
263264 d.addr, d.size, d.order);
264265 break;
265266 case KVM_RISCV_HFENCE_VVMA_ASID_GVA:
267+ kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_ASID_RCVD);
266268 kvm_riscv_local_hfence_vvma_asid_gva(
267269 READ_ONCE(v->vmid), d.asid,
268270 d.addr, d.size, d.order);
269271 break;
270272 case KVM_RISCV_HFENCE_VVMA_ASID_ALL:
273+ kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_ASID_RCVD);
271274 kvm_riscv_local_hfence_vvma_asid_all(
272275 READ_ONCE(v->vmid), d.asid);
273276 break;
274277 case KVM_RISCV_HFENCE_VVMA_GVA:
278+ kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_RCVD);
275279 kvm_riscv_local_hfence_vvma_gva(
276280 READ_ONCE(v->vmid),
277281 d.addr, d.size, d.order);
arch/riscv/kvm/vcpu.c+7−0
@@ -138,6 +138,8 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu) · 137 unmodified lines
138138 WRITE_ONCE(vcpu->arch.irqs_pending, 0);
139139 WRITE_ONCE(vcpu->arch.irqs_pending_mask, 0);
140140
141+ kvm_riscv_vcpu_pmu_reset(vcpu);
142+
141143 vcpu->arch.hfence_head = 0;
142144 vcpu->arch.hfence_tail = 0;
143145 memset(vcpu->arch.hfence_queue, 0, sizeof(vcpu->arch.hfence_queue));
@@ -194,6 +196,9 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · 50 unmodified lines
194196 /* Setup VCPU timer */
195197 kvm_riscv_vcpu_timer_init(vcpu);
196198
199+ /* setup performance monitoring */
200+ kvm_riscv_vcpu_pmu_init(vcpu);
201+
197202 /* Reset VCPU */
198203 kvm_riscv_reset_vcpu(vcpu);
199204
@@ -216,6 +221,8 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) · 16 unmodified lines
216221 /* Cleanup VCPU timer */
217222 kvm_riscv_vcpu_timer_deinit(vcpu);
218223
224+ kvm_riscv_vcpu_pmu_deinit(vcpu);
225+
219226 /* Free unused pages pre-allocated for G-stage page table mappings */
220227 kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache);
221228 }
arch/riscv/kvm/vcpu_exit.c+9−0
@@ -160,6 +160,9 @@ void kvm_riscv_vcpu_trap_redirect(struct kvm_vcpu *vcpu, · 159 unmodified lines
160160
161161 /* Set Guest PC to Guest exception vector */
162162 vcpu->arch.guest_context.sepc = csr_read(CSR_VSTVEC);
163+
164+ /* Set Guest privilege mode to supervisor */
165+ vcpu->arch.guest_context.sstatus |= SR_SPP;
163166 }
164167
165168 /*
@@ -179,6 +182,12 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run, · 13 unmodified lines
179182 ret = -EFAULT;
180183 run->exit_reason = KVM_EXIT_UNKNOWN;
181184 switch (trap->scause) {
185+ case EXC_INST_ILLEGAL:
186+ if (vcpu->arch.guest_context.hstatus & HSTATUS_SPV) {
187+ kvm_riscv_vcpu_trap_redirect(vcpu, trap);
188+ ret = 1;
189+ }
190+ break;
182191 case EXC_VIRTUAL_INST_FAULT:
183192 if (vcpu->arch.guest_context.hstatus & HSTATUS_SPV)
184193 ret = kvm_riscv_vcpu_virtual_insn(vcpu, run, trap);
arch/riscv/kvm/vcpu_insn.c+3−1
@@ -213,7 +213,9 @@ struct csr_func { · 212 unmodified lines
213213 unsigned long wr_mask);
214214 };
215215
216−static const struct csr_func csr_funcs[] = { };
216+static const struct csr_func csr_funcs[] = {
217+ KVM_RISCV_VCPU_HPMCOUNTER_CSR_FUNCS
218+};
217219
218220 /**
219221 * kvm_riscv_vcpu_csr_return -- Handle CSR read/write after user space
arch/riscv/kvm/vcpu_pmu.c+633−0
@@ -0,0 +1,633 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/*
3+ * Copyright (c) 2023 Rivos Inc
4+ *
5+ * Authors:
6+ * Atish Patra <atishp@rivosinc.com>
7+ */
8+
9+#define pr_fmt(fmt) "riscv-kvm-pmu: " fmt
10+#include <linux/errno.h>
11+#include <linux/err.h>
12+#include <linux/kvm_host.h>
13+#include <linux/perf/riscv_pmu.h>
14+#include <asm/csr.h>
15+#include <asm/kvm_vcpu_sbi.h>
16+#include <asm/kvm_vcpu_pmu.h>
17+#include <linux/bitops.h>
18+
19+#define kvm_pmu_num_counters(pmu) ((pmu)->num_hw_ctrs + (pmu)->num_fw_ctrs)
20+#define get_event_type(x) (((x) & SBI_PMU_EVENT_IDX_TYPE_MASK) >> 16)
21+#define get_event_code(x) ((x) & SBI_PMU_EVENT_IDX_CODE_MASK)
22+
23+static enum perf_hw_id hw_event_perf_map[SBI_PMU_HW_GENERAL_MAX] = {
24+ [SBI_PMU_HW_CPU_CYCLES] = PERF_COUNT_HW_CPU_CYCLES,
25+ [SBI_PMU_HW_INSTRUCTIONS] = PERF_COUNT_HW_INSTRUCTIONS,
26+ [SBI_PMU_HW_CACHE_REFERENCES] = PERF_COUNT_HW_CACHE_REFERENCES,
27+ [SBI_PMU_HW_CACHE_MISSES] = PERF_COUNT_HW_CACHE_MISSES,
28+ [SBI_PMU_HW_BRANCH_INSTRUCTIONS] = PERF_COUNT_HW_BRANCH_INSTRUCTIONS,
29+ [SBI_PMU_HW_BRANCH_MISSES] = PERF_COUNT_HW_BRANCH_MISSES,
30+ [SBI_PMU_HW_BUS_CYCLES] = PERF_COUNT_HW_BUS_CYCLES,
31+ [SBI_PMU_HW_STALLED_CYCLES_FRONTEND] = PERF_COUNT_HW_STALLED_CYCLES_FRONTEND,
32+ [SBI_PMU_HW_STALLED_CYCLES_BACKEND] = PERF_COUNT_HW_STALLED_CYCLES_BACKEND,
33+ [SBI_PMU_HW_REF_CPU_CYCLES] = PERF_COUNT_HW_REF_CPU_CYCLES,
34+};
35+
36+static u64 kvm_pmu_get_sample_period(struct kvm_pmc *pmc)
37+{
38+ u64 counter_val_mask = GENMASK(pmc->cinfo.width, 0);
39+ u64 sample_period;
40+
41+ if (!pmc->counter_val)
42+ sample_period = counter_val_mask + 1;
43+ else
44+ sample_period = (-pmc->counter_val) & counter_val_mask;
45+
46+ return sample_period;
47+}
48+
49+static u32 kvm_pmu_get_perf_event_type(unsigned long eidx)
50+{
51+ enum sbi_pmu_event_type etype = get_event_type(eidx);
52+ u32 type = PERF_TYPE_MAX;
53+
54+ switch (etype) {
55+ case SBI_PMU_EVENT_TYPE_HW:
56+ type = PERF_TYPE_HARDWARE;
57+ break;
58+ case SBI_PMU_EVENT_TYPE_CACHE:
59+ type = PERF_TYPE_HW_CACHE;
60+ break;
61+ case SBI_PMU_EVENT_TYPE_RAW:
62+ case SBI_PMU_EVENT_TYPE_FW:
63+ type = PERF_TYPE_RAW;
64+ break;
65+ default:
66+ break;
67+ }
68+
69+ return type;
70+}
71+
72+static bool kvm_pmu_is_fw_event(unsigned long eidx)
73+{
74+ return get_event_type(eidx) == SBI_PMU_EVENT_TYPE_FW;
75+}
76+
77+static void kvm_pmu_release_perf_event(struct kvm_pmc *pmc)
78+{
79+ if (pmc->perf_event) {
80+ perf_event_disable(pmc->perf_event);
81+ perf_event_release_kernel(pmc->perf_event);
82+ pmc->perf_event = NULL;
83+ }
84+}
85+
86+static u64 kvm_pmu_get_perf_event_hw_config(u32 sbi_event_code)
87+{
88+ return hw_event_perf_map[sbi_event_code];
89+}
90+
91+static u64 kvm_pmu_get_perf_event_cache_config(u32 sbi_event_code)
92+{
93+ u64 config = U64_MAX;
94+ unsigned int cache_type, cache_op, cache_result;
95+
96+ /* All the cache event masks lie within 0xFF. No separate masking is necessary */
97+ cache_type = (sbi_event_code & SBI_PMU_EVENT_CACHE_ID_CODE_MASK) >>
98+ SBI_PMU_EVENT_CACHE_ID_SHIFT;
99+ cache_op = (sbi_event_code & SBI_PMU_EVENT_CACHE_OP_ID_CODE_MASK) >>
100+ SBI_PMU_EVENT_CACHE_OP_SHIFT;
101+ cache_result = sbi_event_code & SBI_PMU_EVENT_CACHE_RESULT_ID_CODE_MASK;
102+
103+ if (cache_type >= PERF_COUNT_HW_CACHE_MAX ||
104+ cache_op >= PERF_COUNT_HW_CACHE_OP_MAX ||
105+ cache_result >= PERF_COUNT_HW_CACHE_RESULT_MAX)
106+ return config;
107+
108+ config = cache_type | (cache_op << 8) | (cache_result << 16);
109+
110+ return config;
111+}
112+
113+static u64 kvm_pmu_get_perf_event_config(unsigned long eidx, uint64_t evt_data)
114+{
115+ enum sbi_pmu_event_type etype = get_event_type(eidx);
116+ u32 ecode = get_event_code(eidx);
117+ u64 config = U64_MAX;
118+
119+ switch (etype) {
120+ case SBI_PMU_EVENT_TYPE_HW:
121+ if (ecode < SBI_PMU_HW_GENERAL_MAX)
122+ config = kvm_pmu_get_perf_event_hw_config(ecode);
123+ break;
124+ case SBI_PMU_EVENT_TYPE_CACHE:
125+ config = kvm_pmu_get_perf_event_cache_config(ecode);
126+ break;
127+ case SBI_PMU_EVENT_TYPE_RAW:
128+ config = evt_data & RISCV_PMU_RAW_EVENT_MASK;
129+ break;
130+ case SBI_PMU_EVENT_TYPE_FW:
131+ if (ecode < SBI_PMU_FW_MAX)
132+ config = (1ULL << 63) | ecode;
133+ break;
134+ default:
135+ break;
136+ }
137+
138+ return config;
139+}
140+
141+static int kvm_pmu_get_fixed_pmc_index(unsigned long eidx)
142+{
143+ u32 etype = kvm_pmu_get_perf_event_type(eidx);
144+ u32 ecode = get_event_code(eidx);
145+
146+ if (etype != SBI_PMU_EVENT_TYPE_HW)
147+ return -EINVAL;
148+
149+ if (ecode == SBI_PMU_HW_CPU_CYCLES)
150+ return 0;
151+ else if (ecode == SBI_PMU_HW_INSTRUCTIONS)
152+ return 2;
153+ else
154+ return -EINVAL;
155+}
156+
157+static int kvm_pmu_get_programmable_pmc_index(struct kvm_pmu *kvpmu, unsigned long eidx,
158+ unsigned long cbase, unsigned long cmask)
159+{
160+ int ctr_idx = -1;
161+ int i, pmc_idx;
162+ int min, max;
163+
164+ if (kvm_pmu_is_fw_event(eidx)) {
165+ /* Firmware counters are mapped 1:1 starting from num_hw_ctrs for simplicity */
166+ min = kvpmu->num_hw_ctrs;
167+ max = min + kvpmu->num_fw_ctrs;
168+ } else {
169+ /* First 3 counters are reserved for fixed counters */
170+ min = 3;
171+ max = kvpmu->num_hw_ctrs;
172+ }
173+
174+ for_each_set_bit(i, &cmask, BITS_PER_LONG) {
175+ pmc_idx = i + cbase;
176+ if ((pmc_idx >= min && pmc_idx < max) &&
177+ !test_bit(pmc_idx, kvpmu->pmc_in_use)) {
178+ ctr_idx = pmc_idx;
179+ break;
180+ }
181+ }
182+
183+ return ctr_idx;
184+}
185+
186+static int pmu_get_pmc_index(struct kvm_pmu *pmu, unsigned long eidx,
187+ unsigned long cbase, unsigned long cmask)
188+{
189+ int ret;
190+
191+ /* Fixed counters need to be have fixed mapping as they have different width */
192+ ret = kvm_pmu_get_fixed_pmc_index(eidx);
193+ if (ret >= 0)
194+ return ret;
195+
196+ return kvm_pmu_get_programmable_pmc_index(pmu, eidx, cbase, cmask);
197+}
198+
199+static int pmu_ctr_read(struct kvm_vcpu *vcpu, unsigned long cidx,
200+ unsigned long *out_val)
201+{
202+ struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
203+ struct kvm_pmc *pmc;
204+ u64 enabled, running;
205+ int fevent_code;
206+
207+ pmc = &kvpmu->pmc[cidx];
208+
209+ if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW) {
210+ fevent_code = get_event_code(pmc->event_idx);
211+ pmc->counter_val = kvpmu->fw_event[fevent_code].value;
212+ } else if (pmc->perf_event) {
213+ pmc->counter_val += perf_event_read_value(pmc->perf_event, &enabled, &running);
214+ } else {
215+ return -EINVAL;
216+ }
217+ *out_val = pmc->counter_val;
218+
219+ return 0;
220+}
221+
222+static int kvm_pmu_validate_counter_mask(struct kvm_pmu *kvpmu, unsigned long ctr_base,
223+ unsigned long ctr_mask)
224+{
225+ /* Make sure the we have a valid counter mask requested from the caller */
226+ if (!ctr_mask || (ctr_base + __fls(ctr_mask) >= kvm_pmu_num_counters(kvpmu)))
227+ return -EINVAL;
228+
229+ return 0;
230+}
231+
232+static int kvm_pmu_create_perf_event(struct kvm_pmc *pmc, struct perf_event_attr *attr,
233+ unsigned long flags, unsigned long eidx, unsigned long evtdata)
234+{
235+ struct perf_event *event;
236+
237+ kvm_pmu_release_perf_event(pmc);
238+ attr->config = kvm_pmu_get_perf_event_config(eidx, evtdata);
239+ if (flags & SBI_PMU_CFG_FLAG_CLEAR_VALUE) {
240+ //TODO: Do we really want to clear the value in hardware counter
241+ pmc->counter_val = 0;
242+ }
243+
244+ /*
245+ * Set the default sample_period for now. The guest specified value
246+ * will be updated in the start call.
247+ */
248+ attr->sample_period = kvm_pmu_get_sample_period(pmc);
249+
250+ event = perf_event_create_kernel_counter(attr, -1, current, NULL, pmc);
251+ if (IS_ERR(event)) {
252+ pr_err("kvm pmu event creation failed for eidx %lx: %ld\n", eidx, PTR_ERR(event));
253+ return PTR_ERR(event);
254+ }
255+
256+ pmc->perf_event = event;
257+ if (flags & SBI_PMU_CFG_FLAG_AUTO_START)
258+ perf_event_enable(pmc->perf_event);
259+
260+ return 0;
261+}
262+
263+int kvm_riscv_vcpu_pmu_incr_fw(struct kvm_vcpu *vcpu, unsigned long fid)
264+{
265+ struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
266+ struct kvm_fw_event *fevent;
267+
268+ if (!kvpmu || fid >= SBI_PMU_FW_MAX)
269+ return -EINVAL;
270+
271+ fevent = &kvpmu->fw_event[fid];
272+ if (fevent->started)
273+ fevent->value++;
274+
275+ return 0;
276+}
277+
278+int kvm_riscv_vcpu_pmu_read_hpm(struct kvm_vcpu *vcpu, unsigned int csr_num,
279+ unsigned long *val, unsigned long new_val,
280+ unsigned long wr_mask)
281+{
282+ struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
283+ int cidx, ret = KVM_INSN_CONTINUE_NEXT_SEPC;
284+
285+ if (!kvpmu || !kvpmu->init_done) {
286+ /*
287+ * In absence of sscofpmf in the platform, the guest OS may use
288+ * the legacy PMU driver to read cycle/instret. In that case,
289+ * just return 0 to avoid any illegal trap. However, any other
290+ * hpmcounter access should result in illegal trap as they must
291+ * be access through SBI PMU only.
292+ */
293+ if (csr_num == CSR_CYCLE || csr_num == CSR_INSTRET) {
294+ *val = 0;
295+ return ret;
296+ } else {
297+ return KVM_INSN_ILLEGAL_TRAP;
298+ }
299+ }
300+
301+ /* The counter CSR are read only. Thus, any write should result in illegal traps */
302+ if (wr_mask)
303+ return KVM_INSN_ILLEGAL_TRAP;
304+
305+ cidx = csr_num - CSR_CYCLE;
306+
307+ if (pmu_ctr_read(vcpu, cidx, val) < 0)
308+ return KVM_INSN_ILLEGAL_TRAP;
309+
310+ return ret;
311+}
312+
313+int kvm_riscv_vcpu_pmu_num_ctrs(struct kvm_vcpu *vcpu,
314+ struct kvm_vcpu_sbi_return *retdata)
315+{
316+ struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
317+
318+ retdata->out_val = kvm_pmu_num_counters(kvpmu);
319+
320+ return 0;
321+}
322+
323+int kvm_riscv_vcpu_pmu_ctr_info(struct kvm_vcpu *vcpu, unsigned long cidx,
324+ struct kvm_vcpu_sbi_return *retdata)
325+{
326+ struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
327+
328+ if (cidx > RISCV_KVM_MAX_COUNTERS || cidx == 1) {
329+ retdata->err_val = SBI_ERR_INVALID_PARAM;
330+ return 0;
331+ }
332+
333+ retdata->out_val = kvpmu->pmc[cidx].cinfo.value;
334+
335+ return 0;
336+}
337+
338+int kvm_riscv_vcpu_pmu_ctr_start(struct kvm_vcpu *vcpu, unsigned long ctr_base,
339+ unsigned long ctr_mask, unsigned long flags, u64 ival,
340+ struct kvm_vcpu_sbi_return *retdata)
341+{
342+ struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
343+ int i, pmc_index, sbiret = 0;
344+ struct kvm_pmc *pmc;
345+ int fevent_code;
346+
347+ if (kvm_pmu_validate_counter_mask(kvpmu, ctr_base, ctr_mask) < 0) {
348+ sbiret = SBI_ERR_INVALID_PARAM;
349+ goto out;
350+ }
351+
352+ /* Start the counters that have been configured and requested by the guest */
353+ for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) {
354+ pmc_index = i + ctr_base;
355+ if (!test_bit(pmc_index, kvpmu->pmc_in_use))
356+ continue;
357+ pmc = &kvpmu->pmc[pmc_index];
358+ if (flags & SBI_PMU_START_FLAG_SET_INIT_VALUE)
359+ pmc->counter_val = ival;
360+ if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW) {
361+ fevent_code = get_event_code(pmc->event_idx);
362+ if (fevent_code >= SBI_PMU_FW_MAX) {
363+ sbiret = SBI_ERR_INVALID_PARAM;
364+ goto out;
365+ }
366+
367+ /* Check if the counter was already started for some reason */
368+ if (kvpmu->fw_event[fevent_code].started) {
369+ sbiret = SBI_ERR_ALREADY_STARTED;
370+ continue;
371+ }
372+
373+ kvpmu->fw_event[fevent_code].started = true;
374+ kvpmu->fw_event[fevent_code].value = pmc->counter_val;
375+ } else if (pmc->perf_event) {
376+ if (unlikely(pmc->started)) {
377+ sbiret = SBI_ERR_ALREADY_STARTED;
378+ continue;
379+ }
380+ perf_event_period(pmc->perf_event, kvm_pmu_get_sample_period(pmc));
381+ perf_event_enable(pmc->perf_event);
382+ pmc->started = true;
383+ } else {
384+ sbiret = SBI_ERR_INVALID_PARAM;
385+ }
386+ }
387+
388+out:
389+ retdata->err_val = sbiret;
390+
391+ return 0;
392+}
393+
394+int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
395+ unsigned long ctr_mask, unsigned long flags,
396+ struct kvm_vcpu_sbi_return *retdata)
397+{
398+ struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
399+ int i, pmc_index, sbiret = 0;
400+ u64 enabled, running;
401+ struct kvm_pmc *pmc;
402+ int fevent_code;
403+
404+ if (kvm_pmu_validate_counter_mask(kvpmu, ctr_base, ctr_mask) < 0) {
405+ sbiret = SBI_ERR_INVALID_PARAM;
406+ goto out;
407+ }
408+
409+ /* Stop the counters that have been configured and requested by the guest */
410+ for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) {
411+ pmc_index = i + ctr_base;
412+ if (!test_bit(pmc_index, kvpmu->pmc_in_use))
413+ continue;
414+ pmc = &kvpmu->pmc[pmc_index];
415+ if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW) {
416+ fevent_code = get_event_code(pmc->event_idx);
417+ if (fevent_code >= SBI_PMU_FW_MAX) {
418+ sbiret = SBI_ERR_INVALID_PARAM;
419+ goto out;
420+ }
421+
422+ if (!kvpmu->fw_event[fevent_code].started)
423+ sbiret = SBI_ERR_ALREADY_STOPPED;
424+
425+ kvpmu->fw_event[fevent_code].started = false;
426+ } else if (pmc->perf_event) {
427+ if (pmc->started) {
428+ /* Stop counting the counter */
429+ perf_event_disable(pmc->perf_event);
430+ pmc->started = false;
431+ } else {
432+ sbiret = SBI_ERR_ALREADY_STOPPED;
433+ }
434+
435+ if (flags & SBI_PMU_STOP_FLAG_RESET) {
436+ /* Relase the counter if this is a reset request */
437+ pmc->counter_val += perf_event_read_value(pmc->perf_event,
438+ &enabled, &running);
439+ kvm_pmu_release_perf_event(pmc);
440+ }
441+ } else {
442+ sbiret = SBI_ERR_INVALID_PARAM;
443+ }
444+ if (flags & SBI_PMU_STOP_FLAG_RESET) {
445+ pmc->event_idx = SBI_PMU_EVENT_IDX_INVALID;
446+ clear_bit(pmc_index, kvpmu->pmc_in_use);
447+ }
448+ }
449+
450+out:
451+ retdata->err_val = sbiret;
452+
453+ return 0;
454+}
455+
456+int kvm_riscv_vcpu_pmu_ctr_cfg_match(struct kvm_vcpu *vcpu, unsigned long ctr_base,
457+ unsigned long ctr_mask, unsigned long flags,
458+ unsigned long eidx, u64 evtdata,
459+ struct kvm_vcpu_sbi_return *retdata)
460+{
461+ int ctr_idx, ret, sbiret = 0;
462+ bool is_fevent;
463+ unsigned long event_code;
464+ u32 etype = kvm_pmu_get_perf_event_type(eidx);
465+ struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
466+ struct kvm_pmc *pmc = NULL;
467+ struct perf_event_attr attr = {
468+ .type = etype,
469+ .size = sizeof(struct perf_event_attr),
470+ .pinned = true,
471+ /*
472+ * It should never reach here if the platform doesn't support the sscofpmf
473+ * extension as mode filtering won't work without it.
474+ */
475+ .exclude_host = true,
476+ .exclude_hv = true,
477+ .exclude_user = !!(flags & SBI_PMU_CFG_FLAG_SET_UINH),
478+ .exclude_kernel = !!(flags & SBI_PMU_CFG_FLAG_SET_SINH),
479+ .config1 = RISCV_PMU_CONFIG1_GUEST_EVENTS,
480+ };
481+
482+ if (kvm_pmu_validate_counter_mask(kvpmu, ctr_base, ctr_mask) < 0) {
483+ sbiret = SBI_ERR_INVALID_PARAM;
484+ goto out;
485+ }
486+
487+ event_code = get_event_code(eidx);
488+ is_fevent = kvm_pmu_is_fw_event(eidx);
489+ if (is_fevent && event_code >= SBI_PMU_FW_MAX) {
490+ sbiret = SBI_ERR_NOT_SUPPORTED;
491+ goto out;
492+ }
493+
494+ /*
495+ * SKIP_MATCH flag indicates the caller is aware of the assigned counter
496+ * for this event. Just do a sanity check if it already marked used.
497+ */
498+ if (flags & SBI_PMU_CFG_FLAG_SKIP_MATCH) {
499+ if (!test_bit(ctr_base + __ffs(ctr_mask), kvpmu->pmc_in_use)) {
500+ sbiret = SBI_ERR_FAILURE;
501+ goto out;
502+ }
503+ ctr_idx = ctr_base + __ffs(ctr_mask);
504+ } else {
505+ ctr_idx = pmu_get_pmc_index(kvpmu, eidx, ctr_base, ctr_mask);
506+ if (ctr_idx < 0) {
507+ sbiret = SBI_ERR_NOT_SUPPORTED;
508+ goto out;
509+ }
510+ }
511+
512+ pmc = &kvpmu->pmc[ctr_idx];
513+ pmc->idx = ctr_idx;
514+
515+ if (is_fevent) {
516+ if (flags & SBI_PMU_CFG_FLAG_AUTO_START)
517+ kvpmu->fw_event[event_code].started = true;
518+ } else {
519+ ret = kvm_pmu_create_perf_event(pmc, &attr, flags, eidx, evtdata);
520+ if (ret)
521+ return ret;
522+ }
523+
524+ set_bit(ctr_idx, kvpmu->pmc_in_use);
525+ pmc->event_idx = eidx;
526+ retdata->out_val = ctr_idx;
527+out:
528+ retdata->err_val = sbiret;
529+
530+ return 0;
531+}
532+
533+int kvm_riscv_vcpu_pmu_ctr_read(struct kvm_vcpu *vcpu, unsigned long cidx,
534+ struct kvm_vcpu_sbi_return *retdata)
535+{
536+ int ret;
537+
538+ ret = pmu_ctr_read(vcpu, cidx, &retdata->out_val);
539+ if (ret == -EINVAL)
540+ retdata->err_val = SBI_ERR_INVALID_PARAM;
541+
542+ return 0;
543+}
544+
545+void kvm_riscv_vcpu_pmu_init(struct kvm_vcpu *vcpu)
546+{
547+ int i = 0, ret, num_hw_ctrs = 0, hpm_width = 0;
548+ struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
549+ struct kvm_pmc *pmc;
550+
551+ /*
552+ * PMU functionality should be only available to guests if privilege mode
553+ * filtering is available in the host. Otherwise, guest will always count
554+ * events while the execution is in hypervisor mode.
555+ */
556+ if (!riscv_isa_extension_available(NULL, SSCOFPMF))
557+ return;
558+
559+ ret = riscv_pmu_get_hpm_info(&hpm_width, &num_hw_ctrs);
560+ if (ret < 0 || !hpm_width || !num_hw_ctrs)
561+ return;
562+
563+ /*
564+ * Increase the number of hardware counters to offset the time counter.
565+ */
566+ kvpmu->num_hw_ctrs = num_hw_ctrs + 1;
567+ kvpmu->num_fw_ctrs = SBI_PMU_FW_MAX;
568+ memset(&kvpmu->fw_event, 0, SBI_PMU_FW_MAX * sizeof(struct kvm_fw_event));
569+
570+ if (kvpmu->num_hw_ctrs > RISCV_KVM_MAX_HW_CTRS) {
571+ pr_warn_once("Limiting the hardware counters to 32 as specified by the ISA");
572+ kvpmu->num_hw_ctrs = RISCV_KVM_MAX_HW_CTRS;
573+ }
574+
575+ /*
576+ * There is no correlation between the logical hardware counter and virtual counters.
577+ * However, we need to encode a hpmcounter CSR in the counter info field so that
578+ * KVM can trap n emulate the read. This works well in the migration use case as
579+ * KVM doesn't care if the actual hpmcounter is available in the hardware or not.
580+ */
581+ for (i = 0; i < kvm_pmu_num_counters(kvpmu); i++) {
582+ /* TIME CSR shouldn't be read from perf interface */
583+ if (i == 1)
584+ continue;
585+ pmc = &kvpmu->pmc[i];
586+ pmc->idx = i;
587+ pmc->event_idx = SBI_PMU_EVENT_IDX_INVALID;
588+ if (i < kvpmu->num_hw_ctrs) {
589+ pmc->cinfo.type = SBI_PMU_CTR_TYPE_HW;
590+ if (i < 3)
591+ /* CY, IR counters */
592+ pmc->cinfo.width = 63;
593+ else
594+ pmc->cinfo.width = hpm_width;
595+ /*
596+ * The CSR number doesn't have any relation with the logical
597+ * hardware counters. The CSR numbers are encoded sequentially
598+ * to avoid maintaining a map between the virtual counter
599+ * and CSR number.
600+ */
601+ pmc->cinfo.csr = CSR_CYCLE + i;
602+ } else {
603+ pmc->cinfo.type = SBI_PMU_CTR_TYPE_FW;
604+ pmc->cinfo.width = BITS_PER_LONG - 1;
605+ }
606+ }
607+
608+ kvpmu->init_done = true;
609+}
610+
611+void kvm_riscv_vcpu_pmu_deinit(struct kvm_vcpu *vcpu)
612+{
613+ struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
614+ struct kvm_pmc *pmc;
615+ int i;
616+
617+ if (!kvpmu)
618+ return;
619+
620+ for_each_set_bit(i, kvpmu->pmc_in_use, RISCV_MAX_COUNTERS) {
621+ pmc = &kvpmu->pmc[i];
622+ pmc->counter_val = 0;
623+ kvm_pmu_release_perf_event(pmc);
624+ pmc->event_idx = SBI_PMU_EVENT_IDX_INVALID;
625+ }
626+ bitmap_zero(kvpmu->pmc_in_use, RISCV_MAX_COUNTERS);
627+ memset(&kvpmu->fw_event, 0, SBI_PMU_FW_MAX * sizeof(struct kvm_fw_event));
628+}
629+
630+void kvm_riscv_vcpu_pmu_reset(struct kvm_vcpu *vcpu)
631+{
632+ kvm_riscv_vcpu_pmu_deinit(vcpu);
633+}
arch/riscv/kvm/vcpu_sbi.c+35−37
@@ -12,26 +12,6 @@ · 11 unmodified lines
1212 #include <asm/sbi.h>
1313 #include <asm/kvm_vcpu_sbi.h>
1414
15−static int kvm_linux_err_map_sbi(int err)
16−{
17− switch (err) {
18− case 0:
19− return SBI_SUCCESS;
20− case -EPERM:
21− return SBI_ERR_DENIED;
22− case -EINVAL:
23− return SBI_ERR_INVALID_PARAM;
24− case -EFAULT:
25− return SBI_ERR_INVALID_ADDRESS;
26− case -EOPNOTSUPP:
27− return SBI_ERR_NOT_SUPPORTED;
28− case -EALREADY:
29− return SBI_ERR_ALREADY_AVAILABLE;
30− default:
31− return SBI_ERR_FAILURE;
32− };
33−}
34−
3515 #ifndef CONFIG_RISCV_SBI_V01
3616 static const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_v01 = {
3717 .extid_start = -1UL,
@@ -40,6 +20,16 @@ static const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_v01 = { · 2 unmodified lines
4020 };
4121 #endif
4222
23+#ifdef CONFIG_RISCV_PMU_SBI
24+extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_pmu;
25+#else
26+static const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_pmu = {
27+ .extid_start = -1UL,
28+ .extid_end = -1UL,
29+ .handler = NULL,
30+};
31+#endif
32+
4333 static const struct kvm_vcpu_sbi_extension *sbi_ext[] = {
4434 &vcpu_sbi_ext_v01,
4535 &vcpu_sbi_ext_base,
@@ -48,6 +38,7 @@ static const struct kvm_vcpu_sbi_extension *sbi_ext[] = { · 2 unmodified lines
4838 &vcpu_sbi_ext_rfence,
4939 &vcpu_sbi_ext_srst,
5040 &vcpu_sbi_ext_hsm,
41+ &vcpu_sbi_ext_pmu,
5142 &vcpu_sbi_ext_experimental,
5243 &vcpu_sbi_ext_vendor,
5344 };
@@ -125,11 +116,14 @@ int kvm_riscv_vcpu_sbi_ecall(struct kvm_vcpu *vcpu, struct kvm_run *run) · 71 unmodified lines
125116 {
126117 int ret = 1;
127118 bool next_sepc = true;
128− bool userspace_exit = false;
129119 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
130120 const struct kvm_vcpu_sbi_extension *sbi_ext;
131− struct kvm_cpu_trap utrap = { 0 };
132− unsigned long out_val = 0;
121+ struct kvm_cpu_trap utrap = {0};
122+ struct kvm_vcpu_sbi_return sbi_ret = {
123+ .out_val = 0,
124+ .err_val = 0,
125+ .utrap = &utrap,
126+ };
133127 bool ext_is_v01 = false;
134128
135129 sbi_ext = kvm_vcpu_sbi_find_ext(cp->a7);
@@ -139,42 +133,46 @@ int kvm_riscv_vcpu_sbi_ecall(struct kvm_vcpu *vcpu, struct kvm_run *run) · 3 unmodified lines
139133 cp->a7 <= SBI_EXT_0_1_SHUTDOWN)
140134 ext_is_v01 = true;
141135 #endif
142− ret = sbi_ext->handler(vcpu, run, &out_val, &utrap, &userspace_exit);
136+ ret = sbi_ext->handler(vcpu, run, &sbi_ret);
143137 } else {
144138 /* Return error for unsupported SBI calls */
145139 cp->a0 = SBI_ERR_NOT_SUPPORTED;
146140 goto ecall_done;
147141 }
148142
143+ /*
144+ * When the SBI extension returns a Linux error code, it exits the ioctl
145+ * loop and forwards the error to userspace.
146+ */
147+ if (ret < 0) {
148+ next_sepc = false;
149+ goto ecall_done;
150+ }
151+
149152 /* Handle special error cases i.e trap, exit or userspace forward */
150− if (utrap.scause) {
153+ if (sbi_ret.utrap->scause) {
151154 /* No need to increment sepc or exit ioctl loop */
152155 ret = 1;
153− utrap.sepc = cp->sepc;
154− kvm_riscv_vcpu_trap_redirect(vcpu, &utrap);
156+ sbi_ret.utrap->sepc = cp->sepc;
157+ kvm_riscv_vcpu_trap_redirect(vcpu, sbi_ret.utrap);
155158 next_sepc = false;
156159 goto ecall_done;
157160 }
158161
159162 /* Exit ioctl loop or Propagate the error code the guest */
160− if (userspace_exit) {
163+ if (sbi_ret.uexit) {
161164 next_sepc = false;
162165 ret = 0;
163166 } else {
164− /**
165− * SBI extension handler always returns an Linux error code. Convert
166− * it to the SBI specific error code that can be propagated the SBI
167− * caller.
168− */
169− ret = kvm_linux_err_map_sbi(ret);
170− cp->a0 = ret;
167+ cp->a0 = sbi_ret.err_val;
171168 ret = 1;
172169 }
173170 ecall_done:
174171 if (next_sepc)
175172 cp->sepc += 4;
176− if (!ext_is_v01)
177− cp->a1 = out_val;
173+ /* a1 should only be updated when we continue the ioctl loop */
174+ if (!ext_is_v01 && ret == 1)
175+ cp->a1 = sbi_ret.out_val;
178176
179177 return ret;
180178 }
arch/riscv/kvm/vcpu_sbi_base.c+14−13
@@ -14,11 +14,11 @@ · 13 unmodified lines
1414 #include <asm/kvm_vcpu_sbi.h>
1515
1616 static int kvm_sbi_ext_base_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
17− unsigned long *out_val,
18− struct kvm_cpu_trap *trap, bool *exit)
17+ struct kvm_vcpu_sbi_return *retdata)
1918 {
20− int ret = 0;
2119 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
20+ const struct kvm_vcpu_sbi_extension *sbi_ext;
21+ unsigned long *out_val = &retdata->out_val;
2222
2323 switch (cp->a6) {
2424 case SBI_EXT_BASE_GET_SPEC_VERSION:
@@ -42,9 +42,12 @@ static int kvm_sbi_ext_base_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 17 unmodified lines
4242 * forward it to the userspace
4343 */
4444 kvm_riscv_vcpu_sbi_forward(vcpu, run);
45− *exit = true;
46− } else
47− *out_val = kvm_vcpu_sbi_find_ext(cp->a0) ? 1 : 0;
45+ retdata->uexit = true;
46+ } else {
47+ sbi_ext = kvm_vcpu_sbi_find_ext(cp->a0);
48+ *out_val = sbi_ext && sbi_ext->probe ?
49+ sbi_ext->probe(vcpu) : !!sbi_ext;
50+ }
4851 break;
4952 case SBI_EXT_BASE_GET_MVENDORID:
5053 *out_val = vcpu->arch.mvendorid;
@@ -56,11 +59,11 @@ static int kvm_sbi_ext_base_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 5 unmodified lines
5659 *out_val = vcpu->arch.mimpid;
5760 break;
5861 default:
59− ret = -EOPNOTSUPP;
62+ retdata->err_val = SBI_ERR_NOT_SUPPORTED;
6063 break;
6164 }
6265
63− return ret;
66+ return 0;
6467 }
6568
6669 const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_base = {
@@ -70,17 +73,15 @@ const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_base = { · 3 unmodified lines
7073 };
7174
7275 static int kvm_sbi_ext_forward_handler(struct kvm_vcpu *vcpu,
73− struct kvm_run *run,
74− unsigned long *out_val,
75− struct kvm_cpu_trap *utrap,
76− bool *exit)
76+ struct kvm_run *run,
77+ struct kvm_vcpu_sbi_return *retdata)
7778 {
7879 /*
7980 * Both SBI experimental and vendor extensions are
8081 * unconditionally forwarded to userspace.
8182 */
8283 kvm_riscv_vcpu_sbi_forward(vcpu, run);
83− *exit = true;
84+ retdata->uexit = true;
8485 return 0;
8586 }
8687
arch/riscv/kvm/vcpu_sbi_hsm.c+14−14
@@ -21,9 +21,9 @@ static int kvm_sbi_hsm_vcpu_start(struct kvm_vcpu *vcpu) · 20 unmodified lines
2121
2222 target_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, target_vcpuid);
2323 if (!target_vcpu)
24− return -EINVAL;
24+ return SBI_ERR_INVALID_PARAM;
2525 if (!target_vcpu->arch.power_off)
26− return -EALREADY;
26+ return SBI_ERR_ALREADY_AVAILABLE;
2727
2828 reset_cntx = &target_vcpu->arch.guest_reset_context;
2929 /* start address */
@@ -42,7 +42,7 @@ static int kvm_sbi_hsm_vcpu_start(struct kvm_vcpu *vcpu) · 12 unmodified lines
4242 static int kvm_sbi_hsm_vcpu_stop(struct kvm_vcpu *vcpu)
4343 {
4444 if (vcpu->arch.power_off)
45− return -EINVAL;
45+ return SBI_ERR_FAILURE;
4646
4747 kvm_riscv_vcpu_power_off(vcpu);
4848
@@ -57,7 +57,7 @@ static int kvm_sbi_hsm_vcpu_get_status(struct kvm_vcpu *vcpu) · 8 unmodified lines
5757
5858 target_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, target_vcpuid);
5959 if (!target_vcpu)
60− return -EINVAL;
60+ return SBI_ERR_INVALID_PARAM;
6161 if (!target_vcpu->arch.power_off)
6262 return SBI_HSM_STATE_STARTED;
6363 else if (vcpu->stat.generic.blocking)
@@ -67,9 +67,7 @@ static int kvm_sbi_hsm_vcpu_get_status(struct kvm_vcpu *vcpu) · 3 unmodified lines
6767 }
6868
6969 static int kvm_sbi_ext_hsm_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
70− unsigned long *out_val,
71− struct kvm_cpu_trap *utrap,
72− bool *exit)
70+ struct kvm_vcpu_sbi_return *retdata)
7371 {
7472 int ret = 0;
7573 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
@@ -88,27 +86,29 @@ static int kvm_sbi_ext_hsm_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 12 unmodified lines
8886 case SBI_EXT_HSM_HART_STATUS:
8987 ret = kvm_sbi_hsm_vcpu_get_status(vcpu);
9088 if (ret >= 0) {
91− *out_val = ret;
92− ret = 0;
89+ retdata->out_val = ret;
90+ retdata->err_val = 0;
9391 }
94− break;
92+ return 0;
9593 case SBI_EXT_HSM_HART_SUSPEND:
9694 switch (cp->a0) {
9795 case SBI_HSM_SUSPEND_RET_DEFAULT:
9896 kvm_riscv_vcpu_wfi(vcpu);
9997 break;
10098 case SBI_HSM_SUSPEND_NON_RET_DEFAULT:
101− ret = -EOPNOTSUPP;
99+ ret = SBI_ERR_NOT_SUPPORTED;
102100 break;
103101 default:
104− ret = -EINVAL;
102+ ret = SBI_ERR_INVALID_PARAM;
105103 }
106104 break;
107105 default:
108− ret = -EOPNOTSUPP;
106+ ret = SBI_ERR_NOT_SUPPORTED;
109107 }
110108
111− return ret;
109+ retdata->err_val = ret;
110+
111+ return 0;
112112 }
113113
114114 const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_hsm = {
arch/riscv/kvm/vcpu_sbi_pmu.c+86−0
@@ -0,0 +1,86 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/*
3+ * Copyright (c) 2023 Rivos Inc
4+ *
5+ * Authors:
6+ * Atish Patra <atishp@rivosinc.com>
7+ */
8+
9+#include <linux/errno.h>
10+#include <linux/err.h>
11+#include <linux/kvm_host.h>
12+#include <asm/csr.h>
13+#include <asm/sbi.h>
14+#include <asm/kvm_vcpu_sbi.h>
15+
16+static int kvm_sbi_ext_pmu_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
17+ struct kvm_vcpu_sbi_return *retdata)
18+{
19+ int ret = 0;
20+ struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
21+ struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
22+ unsigned long funcid = cp->a6;
23+ u64 temp;
24+
25+ if (!kvpmu->init_done) {
26+ retdata->err_val = SBI_ERR_NOT_SUPPORTED;
27+ return 0;
28+ }
29+
30+ switch (funcid) {
31+ case SBI_EXT_PMU_NUM_COUNTERS:
32+ ret = kvm_riscv_vcpu_pmu_num_ctrs(vcpu, retdata);
33+ break;
34+ case SBI_EXT_PMU_COUNTER_GET_INFO:
35+ ret = kvm_riscv_vcpu_pmu_ctr_info(vcpu, cp->a0, retdata);
36+ break;
37+ case SBI_EXT_PMU_COUNTER_CFG_MATCH:
38+#if defined(CONFIG_32BIT)
39+ temp = ((uint64_t)cp->a5 << 32) | cp->a4;
40+#else
41+ temp = cp->a4;
42+#endif
43+ /*
44+ * This can fail if perf core framework fails to create an event.
45+ * Forward the error to userspace because it's an error which
46+ * happened within the host kernel. The other option would be
47+ * to convert to an SBI error and forward to the guest.
48+ */
49+ ret = kvm_riscv_vcpu_pmu_ctr_cfg_match(vcpu, cp->a0, cp->a1,
50+ cp->a2, cp->a3, temp, retdata);
51+ break;
52+ case SBI_EXT_PMU_COUNTER_START:
53+#if defined(CONFIG_32BIT)
54+ temp = ((uint64_t)cp->a4 << 32) | cp->a3;
55+#else
56+ temp = cp->a3;
57+#endif
58+ ret = kvm_riscv_vcpu_pmu_ctr_start(vcpu, cp->a0, cp->a1, cp->a2,
59+ temp, retdata);
60+ break;
61+ case SBI_EXT_PMU_COUNTER_STOP:
62+ ret = kvm_riscv_vcpu_pmu_ctr_stop(vcpu, cp->a0, cp->a1, cp->a2, retdata);
63+ break;
64+ case SBI_EXT_PMU_COUNTER_FW_READ:
65+ ret = kvm_riscv_vcpu_pmu_ctr_read(vcpu, cp->a0, retdata);
66+ break;
67+ default:
68+ retdata->err_val = SBI_ERR_NOT_SUPPORTED;
69+ }
70+
71+ return ret;
72+}
73+
74+static unsigned long kvm_sbi_ext_pmu_probe(struct kvm_vcpu *vcpu)
75+{
76+ struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
77+
78+ return kvpmu->init_done;
79+}
80+
81+const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_pmu = {
82+ .extid_start = SBI_EXT_PMU,
83+ .extid_end = SBI_EXT_PMU,
84+ .handler = kvm_sbi_ext_pmu_handler,
85+ .probe = kvm_sbi_ext_pmu_probe,
86+};
arch/riscv/kvm/vcpu_sbi_replace.c+27−23
@@ -11,19 +11,21 @@ · 10 unmodified lines
1111 #include <linux/kvm_host.h>
1212 #include <asm/sbi.h>
1313 #include <asm/kvm_vcpu_timer.h>
14+#include <asm/kvm_vcpu_pmu.h>
1415 #include <asm/kvm_vcpu_sbi.h>
1516
1617 static int kvm_sbi_ext_time_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
17− unsigned long *out_val,
18− struct kvm_cpu_trap *utrap, bool *exit)
18+ struct kvm_vcpu_sbi_return *retdata)
1919 {
20− int ret = 0;
2120 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
2221 u64 next_cycle;
2322
24− if (cp->a6 != SBI_EXT_TIME_SET_TIMER)
25− return -EINVAL;
23+ if (cp->a6 != SBI_EXT_TIME_SET_TIMER) {
24+ retdata->err_val = SBI_ERR_INVALID_PARAM;
25+ return 0;
26+ }
2627
28+ kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_SET_TIMER);
2729 #if __riscv_xlen == 32
2830 next_cycle = ((u64)cp->a1 << 32) | (u64)cp->a0;
2931 #else
@@ -31,7 +33,7 @@ static int kvm_sbi_ext_time_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 1 unmodified line
3133 #endif
3234 kvm_riscv_vcpu_timer_next_event(vcpu, next_cycle);
3335
34− return ret;
36+ return 0;
3537 }
3638
3739 const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_time = {
@@ -41,8 +43,7 @@ const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_time = { · 3 unmodified lines
4143 };
4244
4345 static int kvm_sbi_ext_ipi_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
44− unsigned long *out_val,
45− struct kvm_cpu_trap *utrap, bool *exit)
46+ struct kvm_vcpu_sbi_return *retdata)
4647 {
4748 int ret = 0;
4849 unsigned long i;
@@ -51,9 +52,12 @@ static int kvm_sbi_ext_ipi_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 2 unmodified lines
5152 unsigned long hmask = cp->a0;
5253 unsigned long hbase = cp->a1;
5354
54− if (cp->a6 != SBI_EXT_IPI_SEND_IPI)
55− return -EINVAL;
55+ if (cp->a6 != SBI_EXT_IPI_SEND_IPI) {
56+ retdata->err_val = SBI_ERR_INVALID_PARAM;
57+ return 0;
58+ }
5659
60+ kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_IPI_SENT);
5761 kvm_for_each_vcpu(i, tmp, vcpu->kvm) {
5862 if (hbase != -1UL) {
5963 if (tmp->vcpu_id < hbase)
@@ -64,6 +68,7 @@ static int kvm_sbi_ext_ipi_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 4 unmodified lines
6468 ret = kvm_riscv_vcpu_set_interrupt(tmp, IRQ_VS_SOFT);
6569 if (ret < 0)
6670 break;
71+ kvm_riscv_vcpu_pmu_incr_fw(tmp, SBI_PMU_FW_IPI_RCVD);
6772 }
6873
6974 return ret;
@@ -76,10 +81,8 @@ const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_ipi = { · 6 unmodified lines
7681 };
7782
7883 static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
79− unsigned long *out_val,
80− struct kvm_cpu_trap *utrap, bool *exit)
84+ struct kvm_vcpu_sbi_return *retdata)
8185 {
82− int ret = 0;
8386 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
8487 unsigned long hmask = cp->a0;
8588 unsigned long hbase = cp->a1;
@@ -88,6 +91,7 @@ static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *run · 2 unmodified lines
8891 switch (funcid) {
8992 case SBI_EXT_RFENCE_REMOTE_FENCE_I:
9093 kvm_riscv_fence_i(vcpu->kvm, hbase, hmask);
94+ kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_FENCE_I_SENT);
9195 break;
9296 case SBI_EXT_RFENCE_REMOTE_SFENCE_VMA:
9397 if (cp->a2 == 0 && cp->a3 == 0)
@@ -95,6 +99,7 @@ static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *run · 1 unmodified line
9599 else
96100 kvm_riscv_hfence_vvma_gva(vcpu->kvm, hbase, hmask,
97101 cp->a2, cp->a3, PAGE_SHIFT);
102+ kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_SENT);
98103 break;
99104 case SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID:
100105 if (cp->a2 == 0 && cp->a3 == 0)
@@ -105,6 +110,7 @@ static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *run · 4 unmodified lines
105110 hbase, hmask,
106111 cp->a2, cp->a3,
107112 PAGE_SHIFT, cp->a4);
113+ kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_ASID_SENT);
108114 break;
109115 case SBI_EXT_RFENCE_REMOTE_HFENCE_GVMA:
110116 case SBI_EXT_RFENCE_REMOTE_HFENCE_GVMA_VMID:
@@ -116,10 +122,10 @@ static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *run · 5 unmodified lines
116122 */
117123 break;
118124 default:
119− ret = -EOPNOTSUPP;
125+ retdata->err_val = SBI_ERR_NOT_SUPPORTED;
120126 }
121127
122− return ret;
128+ return 0;
123129 }
124130
125131 const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_rfence = {
@@ -130,14 +136,12 @@ const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_rfence = { · 4 unmodified lines
130136
131137 static int kvm_sbi_ext_srst_handler(struct kvm_vcpu *vcpu,
132138 struct kvm_run *run,
133− unsigned long *out_val,
134− struct kvm_cpu_trap *utrap, bool *exit)
139+ struct kvm_vcpu_sbi_return *retdata)
135140 {
136141 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
137142 unsigned long funcid = cp->a6;
138143 u32 reason = cp->a1;
139144 u32 type = cp->a0;
140− int ret = 0;
141145
142146 switch (funcid) {
143147 case SBI_EXT_SRST_RESET:
@@ -146,24 +150,24 @@ static int kvm_sbi_ext_srst_handler(struct kvm_vcpu *vcpu, · 2 unmodified lines
146150 kvm_riscv_vcpu_sbi_system_reset(vcpu, run,
147151 KVM_SYSTEM_EVENT_SHUTDOWN,
148152 reason);
149− *exit = true;
153+ retdata->uexit = true;
150154 break;
151155 case SBI_SRST_RESET_TYPE_COLD_REBOOT:
152156 case SBI_SRST_RESET_TYPE_WARM_REBOOT:
153157 kvm_riscv_vcpu_sbi_system_reset(vcpu, run,
154158 KVM_SYSTEM_EVENT_RESET,
155159 reason);
156− *exit = true;
160+ retdata->uexit = true;
157161 break;
158162 default:
159− ret = -EOPNOTSUPP;
163+ retdata->err_val = SBI_ERR_NOT_SUPPORTED;
160164 }
161165 break;
162166 default:
163− ret = -EOPNOTSUPP;
167+ retdata->err_val = SBI_ERR_NOT_SUPPORTED;
164168 }
165169
166− return ret;
170+ return 0;
167171 }
168172
169173 const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_srst = {
arch/riscv/kvm/vcpu_sbi_v01.c+7−10
@@ -14,9 +14,7 @@ · 13 unmodified lines
1414 #include <asm/kvm_vcpu_sbi.h>
1515
1616 static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
17− unsigned long *out_val,
18− struct kvm_cpu_trap *utrap,
19− bool *exit)
17+ struct kvm_vcpu_sbi_return *retdata)
2018 {
2119 ulong hmask;
2220 int i, ret = 0;
@@ -24,6 +22,7 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 1 unmodified line
2422 struct kvm_vcpu *rvcpu;
2523 struct kvm *kvm = vcpu->kvm;
2624 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
25+ struct kvm_cpu_trap *utrap = retdata->utrap;
2726
2827 switch (cp->a7) {
2928 case SBI_EXT_0_1_CONSOLE_GETCHAR:
@@ -33,7 +32,7 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 3 unmodified lines
3332 * handled in kernel so we forward these to user-space
3433 */
3534 kvm_riscv_vcpu_sbi_forward(vcpu, run);
36− *exit = true;
35+ retdata->uexit = true;
3736 break;
3837 case SBI_EXT_0_1_SET_TIMER:
3938 #if __riscv_xlen == 32
@@ -48,8 +47,7 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 8 unmodified lines
4847 break;
4948 case SBI_EXT_0_1_SEND_IPI:
5049 if (cp->a0)
51− hmask = kvm_riscv_vcpu_unpriv_read(vcpu, false, cp->a0,
52− utrap);
50+ hmask = kvm_riscv_vcpu_unpriv_read(vcpu, false, cp->a0, utrap);
5351 else
5452 hmask = (1UL << atomic_read(&kvm->online_vcpus)) - 1;
5553 if (utrap->scause)
@@ -65,14 +63,13 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 9 unmodified lines
6563 case SBI_EXT_0_1_SHUTDOWN:
6664 kvm_riscv_vcpu_sbi_system_reset(vcpu, run,
6765 KVM_SYSTEM_EVENT_SHUTDOWN, 0);
68− *exit = true;
66+ retdata->uexit = true;
6967 break;
7068 case SBI_EXT_0_1_REMOTE_FENCE_I:
7169 case SBI_EXT_0_1_REMOTE_SFENCE_VMA:
7270 case SBI_EXT_0_1_REMOTE_SFENCE_VMA_ASID:
7371 if (cp->a0)
74− hmask = kvm_riscv_vcpu_unpriv_read(vcpu, false, cp->a0,
75− utrap);
72+ hmask = kvm_riscv_vcpu_unpriv_read(vcpu, false, cp->a0, utrap);
7673 else
7774 hmask = (1UL << atomic_read(&kvm->online_vcpus)) - 1;
7875 if (utrap->scause)
@@ -103,7 +100,7 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 24 unmodified lines
103100 }
104101 break;
105102 default:
106− ret = -EINVAL;
103+ retdata->err_val = SBI_ERR_NOT_SUPPORTED;
107104 break;
108105 }
109106
arch/riscv/kvm/vmid.c+2−2
@@ -17,10 +17,10 @@ · 16 unmodified lines
1717
1818 static unsigned long vmid_version = 1;
1919 static unsigned long vmid_next;
20−static unsigned long vmid_bits;
20+static unsigned long vmid_bits __ro_after_init;
2121 static DEFINE_SPINLOCK(vmid_lock);
2222
23−void kvm_riscv_gstage_vmid_detect(void)
23+void __init kvm_riscv_gstage_vmid_detect(void)
2424 {
2525 unsigned long old;
2626
arch/s390/include/asm/kvm_host.h+0−1
@@ -1031,7 +1031,6 @@ extern char sie_exit; · 1030 unmodified lines
10311031 extern int kvm_s390_gisc_register(struct kvm *kvm, u32 gisc);
10321032 extern int kvm_s390_gisc_unregister(struct kvm *kvm, u32 gisc);
10331033
1034−static inline void kvm_arch_hardware_disable(void) {}
10351034 static inline void kvm_arch_sync_events(struct kvm *kvm) {}
10361035 static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
10371036 static inline void kvm_arch_free_memslot(struct kvm *kvm,
arch/s390/kvm/gaccess.c+109−0
@@ -1161,6 +1161,115 @@ int access_guest_real(struct kvm_vcpu *vcpu, unsigned long gra, · 1160 unmodified lines
11611161 return rc;
11621162 }
11631163
1164+/**
1165+ * cmpxchg_guest_abs_with_key() - Perform cmpxchg on guest absolute address.
1166+ * @kvm: Virtual machine instance.
1167+ * @gpa: Absolute guest address of the location to be changed.
1168+ * @len: Operand length of the cmpxchg, required: 1 <= len <= 16. Providing a
1169+ * non power of two will result in failure.
1170+ * @old_addr: Pointer to old value. If the location at @gpa contains this value,
1171+ * the exchange will succeed. After calling cmpxchg_guest_abs_with_key()
1172+ * *@old_addr contains the value at @gpa before the attempt to
1173+ * exchange the value.
1174+ * @new: The value to place at @gpa.
1175+ * @access_key: The access key to use for the guest access.
1176+ * @success: output value indicating if an exchange occurred.
1177+ *
1178+ * Atomically exchange the value at @gpa by @new, if it contains *@old.
1179+ * Honors storage keys.
1180+ *
1181+ * Return: * 0: successful exchange
1182+ * * >0: a program interruption code indicating the reason cmpxchg could
1183+ * not be attempted
1184+ * * -EINVAL: address misaligned or len not power of two
1185+ * * -EAGAIN: transient failure (len 1 or 2)
1186+ * * -EOPNOTSUPP: read-only memslot (should never occur)
1187+ */
1188+int cmpxchg_guest_abs_with_key(struct kvm *kvm, gpa_t gpa, int len,
1189+ __uint128_t *old_addr, __uint128_t new,
1190+ u8 access_key, bool *success)
1191+{
1192+ gfn_t gfn = gpa_to_gfn(gpa);
1193+ struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
1194+ bool writable;
1195+ hva_t hva;
1196+ int ret;
1197+
1198+ if (!IS_ALIGNED(gpa, len))
1199+ return -EINVAL;
1200+
1201+ hva = gfn_to_hva_memslot_prot(slot, gfn, &writable);
1202+ if (kvm_is_error_hva(hva))
1203+ return PGM_ADDRESSING;
1204+ /*
1205+ * Check if it's a read-only memslot, even though that cannot occur
1206+ * since those are unsupported.
1207+ * Don't try to actually handle that case.
1208+ */
1209+ if (!writable)
1210+ return -EOPNOTSUPP;
1211+
1212+ hva += offset_in_page(gpa);
1213+ /*
1214+ * The cmpxchg_user_key macro depends on the type of "old", so we need
1215+ * a case for each valid length and get some code duplication as long
1216+ * as we don't introduce a new macro.
1217+ */
1218+ switch (len) {
1219+ case 1: {
1220+ u8 old;
1221+
1222+ ret = cmpxchg_user_key((u8 __user *)hva, &old, *old_addr, new, access_key);
1223+ *success = !ret && old == *old_addr;
1224+ *old_addr = old;
1225+ break;
1226+ }
1227+ case 2: {
1228+ u16 old;
1229+
1230+ ret = cmpxchg_user_key((u16 __user *)hva, &old, *old_addr, new, access_key);
1231+ *success = !ret && old == *old_addr;
1232+ *old_addr = old;
1233+ break;
1234+ }
1235+ case 4: {
1236+ u32 old;
1237+
1238+ ret = cmpxchg_user_key((u32 __user *)hva, &old, *old_addr, new, access_key);
1239+ *success = !ret && old == *old_addr;
1240+ *old_addr = old;
1241+ break;
1242+ }
1243+ case 8: {
1244+ u64 old;
1245+
1246+ ret = cmpxchg_user_key((u64 __user *)hva, &old, *old_addr, new, access_key);
1247+ *success = !ret && old == *old_addr;
1248+ *old_addr = old;
1249+ break;
1250+ }
1251+ case 16: {
1252+ __uint128_t old;
1253+
1254+ ret = cmpxchg_user_key((__uint128_t __user *)hva, &old, *old_addr, new, access_key);
1255+ *success = !ret && old == *old_addr;
1256+ *old_addr = old;
1257+ break;
1258+ }
1259+ default:
1260+ return -EINVAL;
1261+ }
1262+ if (*success)
1263+ mark_page_dirty_in_slot(kvm, slot, gfn);
1264+ /*
1265+ * Assume that the fault is caused by protection, either key protection
1266+ * or user page write protection.
1267+ */
1268+ if (ret == -EFAULT)
1269+ ret = PGM_PROTECTION;
1270+ return ret;
1271+}
1272+
11641273 /**
11651274 * guest_translate_address_with_key - translate guest logical into guest absolute address
11661275 * @vcpu: virtual cpu
arch/s390/kvm/gaccess.h+3−0
@@ -206,6 +206,9 @@ int access_guest_with_key(struct kvm_vcpu *vcpu, unsigned long ga, u8 ar, · 205 unmodified lines
206206 int access_guest_real(struct kvm_vcpu *vcpu, unsigned long gra,
207207 void *data, unsigned long len, enum gacc_mode mode);
208208
209+int cmpxchg_guest_abs_with_key(struct kvm *kvm, gpa_t gpa, int len, __uint128_t *old,
210+ __uint128_t new, u8 access_key, bool *success);
211+
209212 /**
210213 * write_guest_with_key - copy data from kernel space to guest space
211214 * @vcpu: virtual cpu
arch/s390/kvm/interrupt.c+8−5
@@ -3103,9 +3103,9 @@ static enum hrtimer_restart gisa_vcpu_kicker(struct hrtimer *timer) · 3102 unmodified lines
31033103 static void process_gib_alert_list(void)
31043104 {
31053105 struct kvm_s390_gisa_interrupt *gi;
3106+ u32 final, gisa_phys, origin = 0UL;
31063107 struct kvm_s390_gisa *gisa;
31073108 struct kvm *kvm;
3108− u32 final, origin = 0UL;
31093109
31103110 do {
31113111 /*
@@ -3131,9 +3131,10 @@ static void process_gib_alert_list(void) · 19 unmodified lines
31313131 * interruptions asap.
31323132 */
31333133 while (origin & GISA_ADDR_MASK) {
3134− gisa = (struct kvm_s390_gisa *)(u64)origin;
3134+ gisa_phys = origin;
3135+ gisa = phys_to_virt(gisa_phys);
31353136 origin = gisa->next_alert;
3136− gisa->next_alert = (u32)(u64)gisa;
3137+ gisa->next_alert = gisa_phys;
31373138 kvm = container_of(gisa, struct sie_page2, gisa)->kvm;
31383139 gi = &kvm->arch.gisa_int;
31393140 if (hrtimer_active(&gi->timer))
@@ -3415,8 +3416,9 @@ void kvm_s390_gib_destroy(void) · 275 unmodified lines
34153416 gib = NULL;
34163417 }
34173418
3418−int kvm_s390_gib_init(u8 nisc)
3419+int __init kvm_s390_gib_init(u8 nisc)
34193420 {
3421+ u32 gib_origin;
34203422 int rc = 0;
34213423
34223424 if (!css_general_characteristics.aiv) {
@@ -3438,7 +3440,8 @@ int kvm_s390_gib_init(u8 nisc) · 15 unmodified lines
34383440 }
34393441
34403442 gib->nisc = nisc;
3441− if (chsc_sgib((u32)(u64)gib)) {
3443+ gib_origin = virt_to_phys(gib);
3444+ if (chsc_sgib(gib_origin)) {
34423445 pr_err("Associating the GIB with the AIV facility failed\n");
34433446 free_page((unsigned long)gib);
34443447 gib = NULL;
arch/s390/kvm/kvm-s390.c+208−140
@@ -256,17 +256,6 @@ debug_info_t *kvm_s390_dbf; · 255 unmodified lines
256256 debug_info_t *kvm_s390_dbf_uv;
257257
258258 /* Section: not file related */
259−int kvm_arch_hardware_enable(void)
260−{
261− /* every s390 is virtualization enabled ;-) */
262− return 0;
263−}
264−
265−int kvm_arch_check_processor_compat(void *opaque)
266−{
267− return 0;
268−}
269−
270259 /* forward declarations */
271260 static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start,
272261 unsigned long end);
@@ -329,25 +318,6 @@ static struct notifier_block kvm_clock_notifier = { · 56 unmodified lines
329318 .notifier_call = kvm_clock_sync,
330319 };
331320
332−int kvm_arch_hardware_setup(void *opaque)
333−{
334− gmap_notifier.notifier_call = kvm_gmap_notifier;
335− gmap_register_pte_notifier(&gmap_notifier);
336− vsie_gmap_notifier.notifier_call = kvm_s390_vsie_gmap_notifier;
337− gmap_register_pte_notifier(&vsie_gmap_notifier);
338− atomic_notifier_chain_register(&s390_epoch_delta_notifier,
339− &kvm_clock_notifier);
340− return 0;
341−}
342−
343−void kvm_arch_hardware_unsetup(void)
344−{
345− gmap_unregister_pte_notifier(&gmap_notifier);
346− gmap_unregister_pte_notifier(&vsie_gmap_notifier);
347− atomic_notifier_chain_unregister(&s390_epoch_delta_notifier,
348− &kvm_clock_notifier);
349−}
350−
351321 static void allow_cpu_feat(unsigned long nr)
352322 {
353323 set_bit_inv(nr, kvm_s390_available_cpu_feat);
@@ -385,7 +355,7 @@ static __always_inline void __insn32_query(unsigned int opcode, u8 *query) · 31 unmodified lines
385355 #define INSN_SORTL 0xb938
386356 #define INSN_DFLTCC 0xb939
387357
388−static void kvm_s390_cpu_feat_init(void)
358+static void __init kvm_s390_cpu_feat_init(void)
389359 {
390360 int i;
391361
@@ -488,7 +458,7 @@ static void kvm_s390_cpu_feat_init(void) · 96 unmodified lines
488458 */
489459 }
490460
491−int kvm_arch_init(void *opaque)
461+static int __init __kvm_s390_init(void)
492462 {
493463 int rc = -ENOMEM;
494464
@@ -498,11 +468,11 @@ int kvm_arch_init(void *opaque) · 3 unmodified lines
498468
499469 kvm_s390_dbf_uv = debug_register("kvm-uv", 32, 1, 7 * sizeof(long));
500470 if (!kvm_s390_dbf_uv)
501− goto out;
471+ goto err_kvm_uv;
502472
503473 if (debug_register_view(kvm_s390_dbf, &debug_sprintf_view) ||
504474 debug_register_view(kvm_s390_dbf_uv, &debug_sprintf_view))
505− goto out;
475+ goto err_debug_view;
506476
507477 kvm_s390_cpu_feat_init();
508478
@@ -510,30 +480,49 @@ int kvm_arch_init(void *opaque) · 1 unmodified line
510480 rc = kvm_register_device_ops(&kvm_flic_ops, KVM_DEV_TYPE_FLIC);
511481 if (rc) {
512482 pr_err("A FLIC registration call failed with rc=%d\n", rc);
513− goto out;
483+ goto err_flic;
514484 }
515485
516486 if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)) {
517487 rc = kvm_s390_pci_init();
518488 if (rc) {
519489 pr_err("Unable to allocate AIFT for PCI\n");
520− goto out;
490+ goto err_pci;
521491 }
522492 }
523493
524494 rc = kvm_s390_gib_init(GAL_ISC);
525495 if (rc)
526− goto out;
496+ goto err_gib;
497+
498+ gmap_notifier.notifier_call = kvm_gmap_notifier;
499+ gmap_register_pte_notifier(&gmap_notifier);
500+ vsie_gmap_notifier.notifier_call = kvm_s390_vsie_gmap_notifier;
501+ gmap_register_pte_notifier(&vsie_gmap_notifier);
502+ atomic_notifier_chain_register(&s390_epoch_delta_notifier,
503+ &kvm_clock_notifier);
527504
528505 return 0;
529506
530−out:
531− kvm_arch_exit();
507+err_gib:
508+ if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM))
509+ kvm_s390_pci_exit();
510+err_pci:
511+err_flic:
512+err_debug_view:
513+ debug_unregister(kvm_s390_dbf_uv);
514+err_kvm_uv:
515+ debug_unregister(kvm_s390_dbf);
532516 return rc;
533517 }
534518
535−void kvm_arch_exit(void)
519+static void __kvm_s390_exit(void)
536520 {
521+ gmap_unregister_pte_notifier(&gmap_notifier);
522+ gmap_unregister_pte_notifier(&vsie_gmap_notifier);
523+ atomic_notifier_chain_unregister(&s390_epoch_delta_notifier,
524+ &kvm_clock_notifier);
525+
537526 kvm_s390_gib_destroy();
538527 if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM))
539528 kvm_s390_pci_exit();
@@ -584,7 +573,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · 44 unmodified lines
584573 case KVM_CAP_S390_VCPU_RESETS:
585574 case KVM_CAP_SET_GUEST_DEBUG:
586575 case KVM_CAP_S390_DIAG318:
587− case KVM_CAP_S390_MEM_OP_EXTENSION:
588576 r = 1;
589577 break;
590578 case KVM_CAP_SET_GUEST_DEBUG2:
@@ -598,6 +586,15 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · 7 unmodified lines
598586 case KVM_CAP_S390_MEM_OP:
599587 r = MEM_OP_MAX_SIZE;
600588 break;
589+ case KVM_CAP_S390_MEM_OP_EXTENSION:
590+ /*
591+ * Flag bits indicating which extensions are supported.
592+ * If r > 0, the base extension must also be supported/indicated,
593+ * in order to maintain backwards compatibility.
594+ */
595+ r = KVM_S390_MEMOP_EXTENSION_CAP_BASE |
596+ KVM_S390_MEMOP_EXTENSION_CAP_CMPXCHG;
597+ break;
601598 case KVM_CAP_NR_VCPUS:
602599 case KVM_CAP_MAX_VCPUS:
603600 case KVM_CAP_MAX_VCPU_ID:
@@ -2764,41 +2761,33 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) · 2160 unmodified lines
27642761 return r;
27652762 }
27662763
2767−static bool access_key_invalid(u8 access_key)
2768−{
2769− return access_key > 0xf;
2770−}
2771−
2772−static int kvm_s390_vm_mem_op(struct kvm *kvm, struct kvm_s390_mem_op *mop)
2764+static int mem_op_validate_common(struct kvm_s390_mem_op *mop, u64 supported_flags)
27732765 {
2774− void __user *uaddr = (void __user *)mop->buf;
2775− u64 supported_flags;
2776− void *tmpbuf = NULL;
2777− int r, srcu_idx;
2778−
2779− supported_flags = KVM_S390_MEMOP_F_SKEY_PROTECTION
2780− | KVM_S390_MEMOP_F_CHECK_ONLY;
27812766 if (mop->flags & ~supported_flags || !mop->size)
27822767 return -EINVAL;
27832768 if (mop->size > MEM_OP_MAX_SIZE)
27842769 return -E2BIG;
2785− /*
2786− * This is technically a heuristic only, if the kvm->lock is not
2787− * taken, it is not guaranteed that the vm is/remains non-protected.
2788− * This is ok from a kernel perspective, wrongdoing is detected
2789− * on the access, -EFAULT is returned and the vm may crash the
2790− * next time it accesses the memory in question.
2791− * There is no sane usecase to do switching and a memop on two
2792− * different CPUs at the same time.
2793− */
2794− if (kvm_s390_pv_get_handle(kvm))
2795− return -EINVAL;
27962770 if (mop->flags & KVM_S390_MEMOP_F_SKEY_PROTECTION) {
2797− if (access_key_invalid(mop->key))
2771+ if (mop->key > 0xf)
27982772 return -EINVAL;
27992773 } else {
28002774 mop->key = 0;
28012775 }
2776+ return 0;
2777+}
2778+
2779+static int kvm_s390_vm_mem_op_abs(struct kvm *kvm, struct kvm_s390_mem_op *mop)
2780+{
2781+ void __user *uaddr = (void __user *)mop->buf;
2782+ enum gacc_mode acc_mode;
2783+ void *tmpbuf = NULL;
2784+ int r, srcu_idx;
2785+
2786+ r = mem_op_validate_common(mop, KVM_S390_MEMOP_F_SKEY_PROTECTION |
2787+ KVM_S390_MEMOP_F_CHECK_ONLY);
2788+ if (r)
2789+ return r;
2790+
28022791 if (!(mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY)) {
28032792 tmpbuf = vmalloc(mop->size);
28042793 if (!tmpbuf)
@@ -2812,35 +2801,25 @@ static int kvm_s390_vm_mem_op(struct kvm *kvm, struct kvm_s390_mem_op *mop) · 7 unmodified lines
28122801 goto out_unlock;
28132802 }
28142803
2815− switch (mop->op) {
2816− case KVM_S390_MEMOP_ABSOLUTE_READ: {
2817− if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
2818− r = check_gpa_range(kvm, mop->gaddr, mop->size, GACC_FETCH, mop->key);
2819− } else {
2820− r = access_guest_abs_with_key(kvm, mop->gaddr, tmpbuf,
2821− mop->size, GACC_FETCH, mop->key);
2822− if (r == 0) {
2823− if (copy_to_user(uaddr, tmpbuf, mop->size))
2824− r = -EFAULT;
2825− }
2826− }
2827− break;
2804+ acc_mode = mop->op == KVM_S390_MEMOP_ABSOLUTE_READ ? GACC_FETCH : GACC_STORE;
2805+ if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
2806+ r = check_gpa_range(kvm, mop->gaddr, mop->size, acc_mode, mop->key);
2807+ goto out_unlock;
28282808 }
2829− case KVM_S390_MEMOP_ABSOLUTE_WRITE: {
2830− if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
2831− r = check_gpa_range(kvm, mop->gaddr, mop->size, GACC_STORE, mop->key);
2832− } else {
2833− if (copy_from_user(tmpbuf, uaddr, mop->size)) {
2834− r = -EFAULT;
2835− break;
2836− }
2837− r = access_guest_abs_with_key(kvm, mop->gaddr, tmpbuf,
2838− mop->size, GACC_STORE, mop->key);
2809+ if (acc_mode == GACC_FETCH) {
2810+ r = access_guest_abs_with_key(kvm, mop->gaddr, tmpbuf,
2811+ mop->size, GACC_FETCH, mop->key);
2812+ if (r)
2813+ goto out_unlock;
2814+ if (copy_to_user(uaddr, tmpbuf, mop->size))
2815+ r = -EFAULT;
2816+ } else {
2817+ if (copy_from_user(tmpbuf, uaddr, mop->size)) {
2818+ r = -EFAULT;
2819+ goto out_unlock;
28392820 }
2840− break;
2841− }
2842− default:
2843− r = -EINVAL;
2821+ r = access_guest_abs_with_key(kvm, mop->gaddr, tmpbuf,
2822+ mop->size, GACC_STORE, mop->key);
28442823 }
28452824
28462825 out_unlock:
@@ -2850,6 +2829,75 @@ out_unlock: · 3 unmodified lines
28502829 return r;
28512830 }
28522831
2832+static int kvm_s390_vm_mem_op_cmpxchg(struct kvm *kvm, struct kvm_s390_mem_op *mop)
2833+{
2834+ void __user *uaddr = (void __user *)mop->buf;
2835+ void __user *old_addr = (void __user *)mop->old_addr;
2836+ union {
2837+ __uint128_t quad;
2838+ char raw[sizeof(__uint128_t)];
2839+ } old = { .quad = 0}, new = { .quad = 0 };
2840+ unsigned int off_in_quad = sizeof(new) - mop->size;
2841+ int r, srcu_idx;
2842+ bool success;
2843+
2844+ r = mem_op_validate_common(mop, KVM_S390_MEMOP_F_SKEY_PROTECTION);
2845+ if (r)
2846+ return r;
2847+ /*
2848+ * This validates off_in_quad. Checking that size is a power
2849+ * of two is not necessary, as cmpxchg_guest_abs_with_key
2850+ * takes care of that
2851+ */
2852+ if (mop->size > sizeof(new))
2853+ return -EINVAL;
2854+ if (copy_from_user(&new.raw[off_in_quad], uaddr, mop->size))
2855+ return -EFAULT;
2856+ if (copy_from_user(&old.raw[off_in_quad], old_addr, mop->size))
2857+ return -EFAULT;
2858+
2859+ srcu_idx = srcu_read_lock(&kvm->srcu);
2860+
2861+ if (kvm_is_error_gpa(kvm, mop->gaddr)) {
2862+ r = PGM_ADDRESSING;
2863+ goto out_unlock;
2864+ }
2865+
2866+ r = cmpxchg_guest_abs_with_key(kvm, mop->gaddr, mop->size, &old.quad,
2867+ new.quad, mop->key, &success);
2868+ if (!success && copy_to_user(old_addr, &old.raw[off_in_quad], mop->size))
2869+ r = -EFAULT;
2870+
2871+out_unlock:
2872+ srcu_read_unlock(&kvm->srcu, srcu_idx);
2873+ return r;
2874+}
2875+
2876+static int kvm_s390_vm_mem_op(struct kvm *kvm, struct kvm_s390_mem_op *mop)
2877+{
2878+ /*
2879+ * This is technically a heuristic only, if the kvm->lock is not
2880+ * taken, it is not guaranteed that the vm is/remains non-protected.
2881+ * This is ok from a kernel perspective, wrongdoing is detected
2882+ * on the access, -EFAULT is returned and the vm may crash the
2883+ * next time it accesses the memory in question.
2884+ * There is no sane usecase to do switching and a memop on two
2885+ * different CPUs at the same time.
2886+ */
2887+ if (kvm_s390_pv_get_handle(kvm))
2888+ return -EINVAL;
2889+
2890+ switch (mop->op) {
2891+ case KVM_S390_MEMOP_ABSOLUTE_READ:
2892+ case KVM_S390_MEMOP_ABSOLUTE_WRITE:
2893+ return kvm_s390_vm_mem_op_abs(kvm, mop);
2894+ case KVM_S390_MEMOP_ABSOLUTE_CMPXCHG:
2895+ return kvm_s390_vm_mem_op_cmpxchg(kvm, mop);
2896+ default:
2897+ return -EINVAL;
2898+ }
2899+}
2900+
28532901 long kvm_arch_vm_ioctl(struct file *filp,
28542902 unsigned int ioctl, unsigned long arg)
28552903 {
@@ -5249,62 +5297,54 @@ static long kvm_s390_vcpu_mem_op(struct kvm_vcpu *vcpu, · 2393 unmodified lines
52495297 struct kvm_s390_mem_op *mop)
52505298 {
52515299 void __user *uaddr = (void __user *)mop->buf;
5300+ enum gacc_mode acc_mode;
52525301 void *tmpbuf = NULL;
5253− int r = 0;
5254− const u64 supported_flags = KVM_S390_MEMOP_F_INJECT_EXCEPTION
5255− | KVM_S390_MEMOP_F_CHECK_ONLY
5256− | KVM_S390_MEMOP_F_SKEY_PROTECTION;
5302+ int r;
52575303
5258− if (mop->flags & ~supported_flags || mop->ar >= NUM_ACRS || !mop->size)
5304+ r = mem_op_validate_common(mop, KVM_S390_MEMOP_F_INJECT_EXCEPTION |
5305+ KVM_S390_MEMOP_F_CHECK_ONLY |
5306+ KVM_S390_MEMOP_F_SKEY_PROTECTION);
5307+ if (r)
5308+ return r;
5309+ if (mop->ar >= NUM_ACRS)
52595310 return -EINVAL;
5260− if (mop->size > MEM_OP_MAX_SIZE)
5261− return -E2BIG;
52625311 if (kvm_s390_pv_cpu_is_protected(vcpu))
52635312 return -EINVAL;
5264− if (mop->flags & KVM_S390_MEMOP_F_SKEY_PROTECTION) {
5265− if (access_key_invalid(mop->key))
5266− return -EINVAL;
5267− } else {
5268− mop->key = 0;
5269− }
52705313 if (!(mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY)) {
52715314 tmpbuf = vmalloc(mop->size);
52725315 if (!tmpbuf)
52735316 return -ENOMEM;
52745317 }
52755318
5276− switch (mop->op) {
5277− case KVM_S390_MEMOP_LOGICAL_READ:
5278− if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
5279− r = check_gva_range(vcpu, mop->gaddr, mop->ar, mop->size,
5280− GACC_FETCH, mop->key);
5281− break;
5282− }
5319+ acc_mode = mop->op == KVM_S390_MEMOP_LOGICAL_READ ? GACC_FETCH : GACC_STORE;
5320+ if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
5321+ r = check_gva_range(vcpu, mop->gaddr, mop->ar, mop->size,
5322+ acc_mode, mop->key);
5323+ goto out_inject;
5324+ }
5325+ if (acc_mode == GACC_FETCH) {
52835326 r = read_guest_with_key(vcpu, mop->gaddr, mop->ar, tmpbuf,
52845327 mop->size, mop->key);
5285− if (r == 0) {
5286− if (copy_to_user(uaddr, tmpbuf, mop->size))
5287− r = -EFAULT;
5288− }
5289− break;
5290− case KVM_S390_MEMOP_LOGICAL_WRITE:
5291− if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
5292− r = check_gva_range(vcpu, mop->gaddr, mop->ar, mop->size,
5293− GACC_STORE, mop->key);
5294− break;
5328+ if (r)
5329+ goto out_inject;
5330+ if (copy_to_user(uaddr, tmpbuf, mop->size)) {
5331+ r = -EFAULT;
5332+ goto out_free;
52955333 }
5334+ } else {
52965335 if (copy_from_user(tmpbuf, uaddr, mop->size)) {
52975336 r = -EFAULT;
5298− break;
5337+ goto out_free;
52995338 }
53005339 r = write_guest_with_key(vcpu, mop->gaddr, mop->ar, tmpbuf,
53015340 mop->size, mop->key);
5302− break;
53035341 }
53045342
5343+out_inject:
53055344 if (r > 0 && (mop->flags & KVM_S390_MEMOP_F_INJECT_EXCEPTION) != 0)
53065345 kvm_s390_inject_prog_irq(vcpu, &vcpu->arch.pgm);
53075346
5347+out_free:
53085348 vfree(tmpbuf);
53095349 return r;
53105350 }
@@ -5633,23 +5673,40 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, · 322 unmodified lines
56335673 if (kvm_s390_pv_get_handle(kvm))
56345674 return -EINVAL;
56355675
5636− if (change == KVM_MR_DELETE || change == KVM_MR_FLAGS_ONLY)
5637− return 0;
5676+ if (change != KVM_MR_DELETE && change != KVM_MR_FLAGS_ONLY) {
5677+ /*
5678+ * A few sanity checks. We can have memory slots which have to be
5679+ * located/ended at a segment boundary (1MB). The memory in userland is
5680+ * ok to be fragmented into various different vmas. It is okay to mmap()
5681+ * and munmap() stuff in this slot after doing this call at any time
5682+ */
56385683
5639− /* A few sanity checks. We can have memory slots which have to be
5640− located/ended at a segment boundary (1MB). The memory in userland is
5641− ok to be fragmented into various different vmas. It is okay to mmap()
5642− and munmap() stuff in this slot after doing this call at any time */
5684+ if (new->userspace_addr & 0xffffful)
5685+ return -EINVAL;
56435686
5644− if (new->userspace_addr & 0xffffful)
5645− return -EINVAL;
5687+ size = new->npages * PAGE_SIZE;
5688+ if (size & 0xffffful)
5689+ return -EINVAL;
56465690
5647− size = new->npages * PAGE_SIZE;
5648− if (size & 0xffffful)
5649− return -EINVAL;
5691+ if ((new->base_gfn * PAGE_SIZE) + size > kvm->arch.mem_limit)
5692+ return -EINVAL;
5693+ }
56505694
5651− if ((new->base_gfn * PAGE_SIZE) + size > kvm->arch.mem_limit)
5652− return -EINVAL;
5695+ if (!kvm->arch.migration_mode)
5696+ return 0;
5697+
5698+ /*
5699+ * Turn off migration mode when:
5700+ * - userspace creates a new memslot with dirty logging off,
5701+ * - userspace modifies an existing memslot (MOVE or FLAGS_ONLY) and
5702+ * dirty logging is turned off.
5703+ * Migration mode expects dirty page logging being enabled to store
5704+ * its dirty bitmap.
5705+ */
5706+ if (change != KVM_MR_DELETE &&
5707+ !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
5708+ WARN(kvm_s390_vm_stop_migration(kvm),
5709+ "Failed to stop migration mode");
56535710
56545711 return 0;
56555712 }
@@ -5696,7 +5753,7 @@ static inline unsigned long nonhyp_mask(int i) · 40 unmodified lines
56965753
56975754 static int __init kvm_s390_init(void)
56985755 {
5699− int i;
5756+ int i, r;
57005757
57015758 if (!sclp.has_sief2) {
57025759 pr_info("SIE is not available\n");
@@ -5712,12 +5769,23 @@ static int __init kvm_s390_init(void) · 9 unmodified lines
57125769 kvm_s390_fac_base[i] |=
57135770 stfle_fac_list[i] & nonhyp_mask(i);
57145771
5715− return kvm_init(NULL, sizeof(struct kvm_vcpu), 0, THIS_MODULE);
5772+ r = __kvm_s390_init();
5773+ if (r)
5774+ return r;
5775+
5776+ r = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
5777+ if (r) {
5778+ __kvm_s390_exit();
5779+ return r;
5780+ }
5781+ return 0;
57165782 }
57175783
57185784 static void __exit kvm_s390_exit(void)
57195785 {
57205786 kvm_exit();
5787+
5788+ __kvm_s390_exit();
57215789 }
57225790
57235791 module_init(kvm_s390_init);
arch/s390/kvm/kvm-s390.h+1−1
@@ -470,7 +470,7 @@ void kvm_s390_gisa_clear(struct kvm *kvm); · 469 unmodified lines
470470 void kvm_s390_gisa_destroy(struct kvm *kvm);
471471 void kvm_s390_gisa_disable(struct kvm *kvm);
472472 void kvm_s390_gisa_enable(struct kvm *kvm);
473−int kvm_s390_gib_init(u8 nisc);
473+int __init kvm_s390_gib_init(u8 nisc);
474474 void kvm_s390_gib_destroy(void);
475475
476476 /* implemented in guestdbg.c */
arch/s390/kvm/pci.c+1−1
@@ -672,7 +672,7 @@ out: · 671 unmodified lines
672672 return r;
673673 }
674674
675−int kvm_s390_pci_init(void)
675+int __init kvm_s390_pci_init(void)
676676 {
677677 zpci_kvm_hook.kvm_register = kvm_s390_pci_register_kvm;
678678 zpci_kvm_hook.kvm_unregister = kvm_s390_pci_unregister_kvm;
arch/s390/kvm/pci.h+1−1
@@ -60,7 +60,7 @@ void kvm_s390_pci_clear_list(struct kvm *kvm); · 59 unmodified lines
6060
6161 int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op *args);
6262
63−int kvm_s390_pci_init(void);
63+int __init kvm_s390_pci_init(void);
6464 void kvm_s390_pci_exit(void);
6565
6666 static inline bool kvm_s390_pci_interp_allowed(void)
arch/x86/events/intel/core.c+1−0
@@ -6495,6 +6495,7 @@ __init int intel_pmu_init(void) · 6494 unmodified lines
64956495 x86_pmu.pebs_constraints = intel_spr_pebs_event_constraints;
64966496 x86_pmu.extra_regs = intel_spr_extra_regs;
64976497 x86_pmu.limit_period = spr_limit_period;
6498+ x86_pmu.pebs_ept = 1;
64986499 x86_pmu.pebs_aliases = NULL;
64996500 x86_pmu.pebs_prec_dist = true;
65006501 x86_pmu.pebs_block = true;
arch/x86/events/intel/ds.c+3−1
@@ -2366,8 +2366,10 @@ void __init intel_ds_init(void) · 2365 unmodified lines
23662366 x86_pmu.large_pebs_flags |= PERF_SAMPLE_TIME;
23672367 break;
23682368
2369− case 4:
23702369 case 5:
2370+ x86_pmu.pebs_ept = 1;
2371+ fallthrough;
2372+ case 4:
23712373 x86_pmu.drain_pebs = intel_pmu_drain_pebs_icl;
23722374 x86_pmu.pebs_record_size = sizeof(struct pebs_basic);
23732375 if (x86_pmu.intel_cap.pebs_baseline) {
arch/x86/include/asm/cpufeatures.h+3−0
@@ -315,6 +315,9 @@ · 314 unmodified lines
315315 #define X86_FEATURE_AVX512_BF16 (12*32+ 5) /* AVX512 BFLOAT16 instructions */
316316 #define X86_FEATURE_CMPCCXADD (12*32+ 7) /* "" CMPccXADD instructions */
317317 #define X86_FEATURE_ARCH_PERFMON_EXT (12*32+ 8) /* "" Intel Architectural PerfMon Extension */
318+#define X86_FEATURE_FZRM (12*32+10) /* "" Fast zero-length REP MOVSB */
319+#define X86_FEATURE_FSRS (12*32+11) /* "" Fast short REP STOSB */
320+#define X86_FEATURE_FSRC (12*32+12) /* "" Fast short REP {CMPSB,SCASB} */
318321 #define X86_FEATURE_LKGS (12*32+18) /* "" Load "kernel" (userspace) GS */
319322 #define X86_FEATURE_AMX_FP16 (12*32+21) /* "" AMX fp16 Support */
320323 #define X86_FEATURE_AVX_IFMA (12*32+23) /* "" Support for VPMADD52[H,L]UQ */
arch/x86/include/asm/hyperv-tlfs.h+3−0
@@ -269,6 +269,9 @@ enum hv_isolation_type { · 268 unmodified lines
269269 /* TSC invariant control */
270270 #define HV_X64_MSR_TSC_INVARIANT_CONTROL 0x40000118
271271
272+/* HV_X64_MSR_TSC_INVARIANT_CONTROL bits */
273+#define HV_EXPOSE_INVARIANT_TSC BIT_ULL(0)
274+
272275 /* Register name aliases for temporary compatibility */
273276 #define HV_X64_MSR_STIMER0_COUNT HV_REGISTER_STIMER0_COUNT
274277 #define HV_X64_MSR_STIMER0_CONFIG HV_REGISTER_STIMER0_CONFIG
arch/x86/include/asm/idtentry.h+6−10
@@ -582,18 +582,14 @@ DECLARE_IDTENTRY_RAW(X86_TRAP_MC, xenpv_exc_machine_check); · 581 unmodified lines
582582
583583 /* NMI */
584584
585−#if defined(CONFIG_X86_64) && IS_ENABLED(CONFIG_KVM_INTEL)
585+#if IS_ENABLED(CONFIG_KVM_INTEL)
586586 /*
587− * Special NOIST entry point for VMX which invokes this on the kernel
588− * stack. asm_exc_nmi() requires an IST to work correctly vs. the NMI
589− * 'executing' marker.
590− *
591− * On 32bit this just uses the regular NMI entry point because 32-bit does
592− * not have ISTs.
587+ * Special entry point for VMX which invokes this on the kernel stack, even for
588+ * 64-bit, i.e. without using an IST. asm_exc_nmi() requires an IST to work
589+ * correctly vs. the NMI 'executing' marker. Used for 32-bit kernels as well
590+ * to avoid more ifdeffery.
593591 */
594−DECLARE_IDTENTRY(X86_TRAP_NMI, exc_nmi_noist);
595−#else
596−#define asm_exc_nmi_noist asm_exc_nmi
592+DECLARE_IDTENTRY(X86_TRAP_NMI, exc_nmi_kvm_vmx);
597593 #endif
598594
599595 DECLARE_IDTENTRY_NMI(X86_TRAP_NMI, exc_nmi);
arch/x86/include/asm/kvm-x86-ops.h+1−1
@@ -14,6 +14,7 @@ BUILD_BUG_ON(1) · 13 unmodified lines
1414 * to make a definition optional, but in this case the default will
1515 * be __static_call_return0.
1616 */
17+KVM_X86_OP(check_processor_compatibility)
1718 KVM_X86_OP(hardware_enable)
1819 KVM_X86_OP(hardware_disable)
1920 KVM_X86_OP(hardware_unsetup)
@@ -76,7 +77,6 @@ KVM_X86_OP(set_nmi_mask) · 56 unmodified lines
7677 KVM_X86_OP(enable_nmi_window)
7778 KVM_X86_OP(enable_irq_window)
7879 KVM_X86_OP_OPTIONAL(update_cr8_intercept)
79−KVM_X86_OP(check_apicv_inhibit_reasons)
8080 KVM_X86_OP(refresh_apicv_exec_ctrl)
8181 KVM_X86_OP_OPTIONAL(hwapic_irr_update)
8282 KVM_X86_OP_OPTIONAL(hwapic_isr_update)
arch/x86/include/asm/kvm_host.h+71−38
@@ -134,8 +134,6 @@ · 133 unmodified lines
134134 #define INVALID_PAGE (~(hpa_t)0)
135135 #define VALID_PAGE(x) ((x) != INVALID_PAGE)
136136
137−#define INVALID_GPA (~(gpa_t)0)
138−
139137 /* KVM Hugepage definitions for x86 */
140138 #define KVM_MAX_HUGEPAGE_LEVEL PG_LEVEL_1G
141139 #define KVM_NR_PAGE_SIZES (KVM_MAX_HUGEPAGE_LEVEL - PG_LEVEL_4K + 1)
@@ -514,6 +512,7 @@ struct kvm_pmc { · 372 unmodified lines
514512 #define MSR_ARCH_PERFMON_PERFCTR_MAX (MSR_ARCH_PERFMON_PERFCTR0 + KVM_INTEL_PMC_MAX_GENERIC - 1)
515513 #define MSR_ARCH_PERFMON_EVENTSEL_MAX (MSR_ARCH_PERFMON_EVENTSEL0 + KVM_INTEL_PMC_MAX_GENERIC - 1)
516514 #define KVM_PMC_MAX_FIXED 3
515+#define MSR_ARCH_PERFMON_FIXED_CTR_MAX (MSR_ARCH_PERFMON_FIXED_CTR0 + KVM_PMC_MAX_FIXED - 1)
517516 #define KVM_AMD_PMC_MAX_GENERIC 6
518517 struct kvm_pmu {
519518 unsigned nr_arch_gp_counters;
@@ -678,6 +677,11 @@ struct kvm_vcpu_hv { · 158 unmodified lines
678677 } nested;
679678 };
680679
680+struct kvm_hypervisor_cpuid {
681+ u32 base;
682+ u32 limit;
683+};
684+
681685 /* Xen HVM per vcpu emulation context */
682686 struct kvm_vcpu_xen {
683687 u64 hypercall_rip;
@@ -698,6 +702,7 @@ struct kvm_vcpu_xen { · 14 unmodified lines
698702 struct hrtimer timer;
699703 int poll_evtchn;
700704 struct timer_list poll_timer;
705+ struct kvm_hypervisor_cpuid cpuid;
701706 };
702707
703708 struct kvm_queued_exception {
@@ -826,7 +831,7 @@ struct kvm_vcpu_arch { · 122 unmodified lines
826831
827832 int cpuid_nent;
828833 struct kvm_cpuid_entry2 *cpuid_entries;
829− u32 kvm_cpuid_base;
834+ struct kvm_hypervisor_cpuid kvm_cpuid;
830835
831836 u64 reserved_gpa_bits;
832837 int maxphyaddr;
@@ -1022,19 +1027,30 @@ struct kvm_arch_memory_slot { · 189 unmodified lines
10221027 };
10231028
10241029 /*
1025− * We use as the mode the number of bits allocated in the LDR for the
1026− * logical processor ID. It happens that these are all powers of two.
1027− * This makes it is very easy to detect cases where the APICs are
1028− * configured for multiple modes; in that case, we cannot use the map and
1029− * hence cannot use kvm_irq_delivery_to_apic_fast either.
1030+ * Track the mode of the optimized logical map, as the rules for decoding the
1031+ * destination vary per mode. Enabling the optimized logical map requires all
1032+ * software-enabled local APIs to be in the same mode, each addressable APIC to
1033+ * be mapped to only one MDA, and each MDA to map to at most one APIC.
10301034 */
1031−#define KVM_APIC_MODE_XAPIC_CLUSTER 4
1032−#define KVM_APIC_MODE_XAPIC_FLAT 8
1033−#define KVM_APIC_MODE_X2APIC 16
1035+enum kvm_apic_logical_mode {
1036+ /* All local APICs are software disabled. */
1037+ KVM_APIC_MODE_SW_DISABLED,
1038+ /* All software enabled local APICs in xAPIC cluster addressing mode. */
1039+ KVM_APIC_MODE_XAPIC_CLUSTER,
1040+ /* All software enabled local APICs in xAPIC flat addressing mode. */
1041+ KVM_APIC_MODE_XAPIC_FLAT,
1042+ /* All software enabled local APICs in x2APIC mode. */
1043+ KVM_APIC_MODE_X2APIC,
1044+ /*
1045+ * Optimized map disabled, e.g. not all local APICs in the same logical
1046+ * mode, same logical ID assigned to multiple APICs, etc.
1047+ */
1048+ KVM_APIC_MODE_MAP_DISABLED,
1049+};
10341050
10351051 struct kvm_apic_map {
10361052 struct rcu_head rcu;
1037− u8 mode;
1053+ enum kvm_apic_logical_mode logical_mode;
10381054 u32 max_apic_id;
10391055 union {
10401056 struct kvm_lapic *xapic_flat_map[8];
@@ -1088,6 +1104,7 @@ struct kvm_hv { · 47 unmodified lines
10881104 u64 hv_reenlightenment_control;
10891105 u64 hv_tsc_emulation_control;
10901106 u64 hv_tsc_emulation_status;
1107+ u64 hv_invtsc_control;
10911108
10921109 /* How many vCPUs have VP index != vCPU index */
10931110 atomic_t num_mismatched_vp_indexes;
@@ -1133,6 +1150,18 @@ struct kvm_x86_msr_filter { · 39 unmodified lines
11331150 struct msr_bitmap_range ranges[16];
11341151 };
11351152
1153+struct kvm_x86_pmu_event_filter {
1154+ __u32 action;
1155+ __u32 nevents;
1156+ __u32 fixed_counter_bitmap;
1157+ __u32 flags;
1158+ __u32 nr_includes;
1159+ __u32 nr_excludes;
1160+ __u64 *includes;
1161+ __u64 *excludes;
1162+ __u64 events[];
1163+};
1164+
11361165 enum kvm_apicv_inhibit {
11371166
11381167 /********************************************************************/
@@ -1163,6 +1192,12 @@ enum kvm_apicv_inhibit { · 24 unmodified lines
11631192 */
11641193 APICV_INHIBIT_REASON_BLOCKIRQ,
11651194
1195+ /*
1196+ * APICv is disabled because not all vCPUs have a 1:1 mapping between
1197+ * APIC ID and vCPU, _and_ KVM is not applying its x2APIC hotplug hack.
1198+ */
1199+ APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED,
1200+
11661201 /*
11671202 * For simplicity, the APIC acceleration is inhibited
11681203 * first time either APIC ID or APIC base are changed by the guest
@@ -1201,6 +1236,12 @@ enum kvm_apicv_inhibit { · 32 unmodified lines
12011236 * AVIC is disabled because SEV doesn't support it.
12021237 */
12031238 APICV_INHIBIT_REASON_SEV,
1239+
1240+ /*
1241+ * AVIC is disabled because not all vCPUs with a valid LDR have a 1:1
1242+ * mapping between logical ID and vCPU.
1243+ */
1244+ APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED,
12041245 };
12051246
12061247 struct kvm_arch {
@@ -1249,10 +1290,11 @@ struct kvm_arch { · 42 unmodified lines
12491290 struct kvm_apic_map __rcu *apic_map;
12501291 atomic_t apic_map_dirty;
12511292
1252− /* Protects apic_access_memslot_enabled and apicv_inhibit_reasons */
1253− struct rw_semaphore apicv_update_lock;
1254−
12551293 bool apic_access_memslot_enabled;
1294+ bool apic_access_memslot_inhibited;
1295+
1296+ /* Protects apicv_inhibit_reasons */
1297+ struct rw_semaphore apicv_update_lock;
12561298 unsigned long apicv_inhibit_reasons;
12571299
12581300 gpa_t wall_clock;
@@ -1302,7 +1344,6 @@ struct kvm_arch { · 43 unmodified lines
13021344 u32 bsp_vcpu_id;
13031345
13041346 u64 disabled_quirks;
1305− int cpu_dirty_logging_count;
13061347
13071348 enum kvm_irqchip_mode irqchip_mode;
13081349 u8 nr_reserved_ioapic_pins;
@@ -1338,25 +1379,16 @@ struct kvm_arch { · 29 unmodified lines
13381379 /* Guest can access the SGX PROVISIONKEY. */
13391380 bool sgx_provisioning_allowed;
13401381
1341− struct kvm_pmu_event_filter __rcu *pmu_event_filter;
1382+ struct kvm_x86_pmu_event_filter __rcu *pmu_event_filter;
13421383 struct task_struct *nx_huge_page_recovery_thread;
13431384
13441385 #ifdef CONFIG_X86_64
1345− /*
1346− * Whether the TDP MMU is enabled for this VM. This contains a
1347− * snapshot of the TDP MMU module parameter from when the VM was
1348− * created and remains unchanged for the life of the VM. If this is
1349− * true, TDP MMU handler functions will run for various MMU
1350− * operations.
1351− */
1352− bool tdp_mmu_enabled;
1353−
13541386 /* The number of TDP MMU pages across all roots. */
13551387 atomic64_t tdp_mmu_pages;
13561388
13571389 /*
1358− * List of kvm_mmu_page structs being used as roots.
1359− * All kvm_mmu_page structs in the list should have
1390+ * List of struct kvm_mmu_pages being used as roots.
1391+ * All struct kvm_mmu_pages in the list should have
13601392 * tdp_mmu_page set.
13611393 *
13621394 * For reads, this list is protected by:
@@ -1520,6 +1552,8 @@ static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical) · 157 unmodified lines
15201552 struct kvm_x86_ops {
15211553 const char *name;
15221554
1555+ int (*check_processor_compatibility)(void);
1556+
15231557 int (*hardware_enable)(void);
15241558 void (*hardware_disable)(void);
15251559 void (*hardware_unsetup)(void);
@@ -1608,6 +1642,8 @@ struct kvm_x86_ops { · 82 unmodified lines
16081642 void (*enable_irq_window)(struct kvm_vcpu *vcpu);
16091643 void (*update_cr8_intercept)(struct kvm_vcpu *vcpu, int tpr, int irr);
16101644 bool (*check_apicv_inhibit_reasons)(enum kvm_apicv_inhibit reason);
1645+ const unsigned long required_apicv_inhibits;
1646+ bool allow_apicv_in_x2apic_without_x2apic_virtualization;
16111647 void (*refresh_apicv_exec_ctrl)(struct kvm_vcpu *vcpu);
16121648 void (*hwapic_irr_update)(struct kvm_vcpu *vcpu, int max_irr);
16131649 void (*hwapic_isr_update)(int isr);
@@ -1731,9 +1767,6 @@ struct kvm_x86_nested_ops { · 117 unmodified lines
17311767 };
17321768
17331769 struct kvm_x86_init_ops {
1734− int (*cpu_has_kvm_support)(void);
1735− int (*disabled_by_bios)(void);
1736− int (*check_processor_compatibility)(void);
17371770 int (*hardware_setup)(void);
17381771 unsigned int (*handle_intel_pt_intr)(void);
17391772
@@ -1760,6 +1793,9 @@ extern struct kvm_x86_ops kvm_x86_ops; · 20 unmodified lines
17601793 #define KVM_X86_OP_OPTIONAL_RET0 KVM_X86_OP
17611794 #include <asm/kvm-x86-ops.h>
17621795
1796+int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops);
1797+void kvm_x86_vendor_exit(void);
1798+
17631799 #define __KVM_HAVE_ARCH_VM_ALLOC
17641800 static inline struct kvm *kvm_arch_alloc_vm(void)
17651801 {
@@ -1982,7 +2018,7 @@ gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva, · 216 unmodified lines
19822018
19832019 bool kvm_apicv_activated(struct kvm *kvm);
19842020 bool kvm_vcpu_apicv_activated(struct kvm_vcpu *vcpu);
1985−void kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu);
2021+void __kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu);
19862022 void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm,
19872023 enum kvm_apicv_inhibit reason, bool set);
19882024 void kvm_set_or_clear_apicv_inhibit(struct kvm *kvm,
@@ -2054,14 +2090,11 @@ enum { · 65 unmodified lines
20542090 TASK_SWITCH_GATE = 3,
20552091 };
20562092
2057−#define HF_GIF_MASK (1 << 0)
2058−#define HF_NMI_MASK (1 << 3)
2059−#define HF_IRET_MASK (1 << 4)
2060−#define HF_GUEST_MASK (1 << 5) /* VCPU is in guest-mode */
2093+#define HF_GUEST_MASK (1 << 0) /* VCPU is in guest-mode */
20612094
20622095 #ifdef CONFIG_KVM_SMM
2063−#define HF_SMM_MASK (1 << 6)
2064−#define HF_SMM_INSIDE_NMI_MASK (1 << 7)
2096+#define HF_SMM_MASK (1 << 1)
2097+#define HF_SMM_INSIDE_NMI_MASK (1 << 2)
20652098
20662099 # define __KVM_VCPU_MULTIPLE_ADDRESS_SPACE
20672100 # define KVM_ADDRESS_SPACE_NUM 2
arch/x86/include/asm/reboot.h+2−0
@@ -25,6 +25,8 @@ void __noreturn machine_real_restart(unsigned int type); · 24 unmodified lines
2525 #define MRR_BIOS 0
2626 #define MRR_APM 1
2727
28+void cpu_emergency_disable_virtualization(void);
29+
2830 typedef void (*nmi_shootdown_cb)(int, struct pt_regs*);
2931 void nmi_panic_self_stop(struct pt_regs *regs);
3032 void nmi_shootdown_cpus(nmi_shootdown_cb callback);
arch/x86/include/asm/virtext.h+15−1
@@ -126,7 +126,21 @@ static inline void cpu_svm_disable(void) · 125 unmodified lines
126126
127127 wrmsrl(MSR_VM_HSAVE_PA, 0);
128128 rdmsrl(MSR_EFER, efer);
129− wrmsrl(MSR_EFER, efer & ~EFER_SVME);
129+ if (efer & EFER_SVME) {
130+ /*
131+ * Force GIF=1 prior to disabling SVM to ensure INIT and NMI
132+ * aren't blocked, e.g. if a fatal error occurred between CLGI
133+ * and STGI. Note, STGI may #UD if SVM is disabled from NMI
134+ * context between reading EFER and executing STGI. In that
135+ * case, GIF must already be set, otherwise the NMI would have
136+ * been blocked, so just eat the fault.
137+ */
138+ asm_volatile_goto("1: stgi\n\t"
139+ _ASM_EXTABLE(1b, %l[fault])
140+ ::: "memory" : fault);
141+fault:
142+ wrmsrl(MSR_EFER, efer & ~EFER_SVME);
143+ }
130144 }
131145
132146 /** Makes sure SVM is disabled, if it is supported on the CPU
arch/x86/include/asm/xen/hypervisor.h+3−1
@@ -38,9 +38,11 @@ extern struct start_info *xen_start_info; · 37 unmodified lines
3838
3939 #include <asm/processor.h>
4040
41+#define XEN_SIGNATURE "XenVMMXenVMM"
42+
4143 static inline uint32_t xen_cpuid_base(void)
4244 {
43− return hypervisor_cpuid_base("XenVMMXenVMM", 2);
45+ return hypervisor_cpuid_base(XEN_SIGNATURE, 2);
4446 }
4547
4648 struct pci_dev;
arch/x86/include/uapi/asm/kvm.h+32−2
@@ -9,6 +9,7 @@ · 8 unmodified lines
99
1010 #include <linux/types.h>
1111 #include <linux/ioctl.h>
12+#include <linux/stddef.h>
1213
1314 #define KVM_PIO_PAGE_OFFSET 1
1415 #define KVM_COALESCED_MMIO_PAGE_OFFSET 2
@@ -507,8 +508,8 @@ struct kvm_nested_state { · 492 unmodified lines
507508 * KVM_{GET,PUT}_NESTED_STATE ioctl values.
508509 */
509510 union {
510− struct kvm_vmx_nested_state_data vmx[0];
511− struct kvm_svm_nested_state_data svm[0];
511+ __DECLARE_FLEX_ARRAY(struct kvm_vmx_nested_state_data, vmx);
512+ __DECLARE_FLEX_ARRAY(struct kvm_svm_nested_state_data, svm);
512513 } data;
513514 };
514515
@@ -525,6 +526,35 @@ struct kvm_pmu_event_filter { · 10 unmodified lines
525526 #define KVM_PMU_EVENT_ALLOW 0
526527 #define KVM_PMU_EVENT_DENY 1
527528
529+#define KVM_PMU_EVENT_FLAG_MASKED_EVENTS BIT(0)
530+#define KVM_PMU_EVENT_FLAGS_VALID_MASK (KVM_PMU_EVENT_FLAG_MASKED_EVENTS)
531+
532+/*
533+ * Masked event layout.
534+ * Bits Description
535+ * ---- -----------
536+ * 7:0 event select (low bits)
537+ * 15:8 umask match
538+ * 31:16 unused
539+ * 35:32 event select (high bits)
540+ * 36:54 unused
541+ * 55 exclude bit
542+ * 63:56 umask mask
543+ */
544+
545+#define KVM_PMU_ENCODE_MASKED_ENTRY(event_select, mask, match, exclude) \
546+ (((event_select) & 0xFFULL) | (((event_select) & 0XF00ULL) << 24) | \
547+ (((mask) & 0xFFULL) << 56) | \
548+ (((match) & 0xFFULL) << 8) | \
549+ ((__u64)(!!(exclude)) << 55))
550+
551+#define KVM_PMU_MASKED_ENTRY_EVENT_SELECT \
552+ (GENMASK_ULL(7, 0) | GENMASK_ULL(35, 32))
553+#define KVM_PMU_MASKED_ENTRY_UMASK_MASK (GENMASK_ULL(63, 56))
554+#define KVM_PMU_MASKED_ENTRY_UMASK_MATCH (GENMASK_ULL(15, 8))
555+#define KVM_PMU_MASKED_ENTRY_EXCLUDE (BIT_ULL(55))
556+#define KVM_PMU_MASKED_ENTRY_UMASK_MASK_SHIFT (56)
557+
528558 /* for KVM_{GET,SET,HAS}_DEVICE_ATTR */
529559 #define KVM_VCPU_TSC_CTRL 0 /* control group for the timestamp counter (TSC) */
530560 #define KVM_VCPU_TSC_OFFSET 0 /* attribute for the TSC offset */
arch/x86/kernel/cpu/mshyperv.c+1−1
@@ -460,7 +460,7 @@ static void __init ms_hyperv_init_platform(void) · 459 unmodified lines
460460 * setting of this MSR bit should happen before init_intel()
461461 * is called.
462462 */
463− wrmsrl(HV_X64_MSR_TSC_INVARIANT_CONTROL, 0x1);
463+ wrmsrl(HV_X64_MSR_TSC_INVARIANT_CONTROL, HV_EXPOSE_INVARIANT_TSC);
464464 setup_force_cpu_cap(X86_FEATURE_TSC_RELIABLE);
465465 }
466466
arch/x86/kernel/crash.c+1−16
@@ -37,7 +37,6 @@ · 36 unmodified lines
3737 #include <linux/kdebug.h>
3838 #include <asm/cpu.h>
3939 #include <asm/reboot.h>
40−#include <asm/virtext.h>
4140 #include <asm/intel_pt.h>
4241 #include <asm/crash.h>
4342 #include <asm/cmdline.h>
@@ -81,15 +80,6 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs) · 37 unmodified lines
8180 */
8281 cpu_crash_vmclear_loaded_vmcss();
8382
84− /* Disable VMX or SVM if needed.
85− *
86− * We need to disable virtualization on all CPUs.
87− * Having VMX or SVM enabled on any CPU may break rebooting
88− * after the kdump kernel has finished its task.
89− */
90− cpu_emergency_vmxoff();
91− cpu_emergency_svm_disable();
92−
9383 /*
9484 * Disable Intel PT to stop its logging
9585 */
@@ -148,12 +138,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs) · 52 unmodified lines
148138 */
149139 cpu_crash_vmclear_loaded_vmcss();
150140
151− /* Booting kdump kernel with VMX or SVM enabled won't work,
152− * because (among other limitations) we can't disable paging
153− * with the virt flags.
154− */
155− cpu_emergency_vmxoff();
156− cpu_emergency_svm_disable();
141+ cpu_emergency_disable_virtualization();
157142
158143 /*
159144 * Disable Intel PT to stop its logging
arch/x86/kernel/nmi.c+4−4
@@ -560,14 +560,14 @@ nmi_restart: · 559 unmodified lines
560560 }
561561 }
562562
563−#if defined(CONFIG_X86_64) && IS_ENABLED(CONFIG_KVM_INTEL)
564−DEFINE_IDTENTRY_RAW(exc_nmi_noist)
563+#if IS_ENABLED(CONFIG_KVM_INTEL)
564+DEFINE_IDTENTRY_RAW(exc_nmi_kvm_vmx)
565565 {
566566 exc_nmi(regs);
567567 }
568−#endif
569568 #if IS_MODULE(CONFIG_KVM_INTEL)
570−EXPORT_SYMBOL_GPL(asm_exc_nmi_noist);
569+EXPORT_SYMBOL_GPL(asm_exc_nmi_kvm_vmx);
570+#endif
571571 #endif
572572
573573 #ifdef CONFIG_NMI_CHECK_CPU
arch/x86/kernel/reboot.c+64−24
@@ -528,33 +528,29 @@ static inline void kb_wait(void) · 527 unmodified lines
528528 }
529529 }
530530
531−static void vmxoff_nmi(int cpu, struct pt_regs *regs)
532−{
533− cpu_emergency_vmxoff();
534−}
531+static inline void nmi_shootdown_cpus_on_restart(void);
535532
536−/* Use NMIs as IPIs to tell all CPUs to disable virtualization */
537−static void emergency_vmx_disable_all(void)
533+static void emergency_reboot_disable_virtualization(void)
538534 {
539535 /* Just make sure we won't change CPUs while doing this */
540536 local_irq_disable();
541537
542538 /*
543− * Disable VMX on all CPUs before rebooting, otherwise we risk hanging
544− * the machine, because the CPU blocks INIT when it's in VMX root.
539+ * Disable virtualization on all CPUs before rebooting to avoid hanging
540+ * the system, as VMX and SVM block INIT when running in the host.
545541 *
546542 * We can't take any locks and we may be on an inconsistent state, so
547− * use NMIs as IPIs to tell the other CPUs to exit VMX root and halt.
543+ * use NMIs as IPIs to tell the other CPUs to disable VMX/SVM and halt.
548544 *
549− * Do the NMI shootdown even if VMX if off on _this_ CPU, as that
550− * doesn't prevent a different CPU from being in VMX root operation.
545+ * Do the NMI shootdown even if virtualization is off on _this_ CPU, as
546+ * other CPUs may have virtualization enabled.
551547 */
552− if (cpu_has_vmx()) {
553− /* Safely force _this_ CPU out of VMX root operation. */
554− __cpu_emergency_vmxoff();
548+ if (cpu_has_vmx() || cpu_has_svm(NULL)) {
549+ /* Safely force _this_ CPU out of VMX/SVM operation. */
550+ cpu_emergency_disable_virtualization();
555551
556− /* Halt and exit VMX root operation on the other CPUs. */
557− nmi_shootdown_cpus(vmxoff_nmi);
552+ /* Disable VMX/SVM and halt on other CPUs. */
553+ nmi_shootdown_cpus_on_restart();
558554 }
559555 }
560556
@@ -590,7 +586,7 @@ static void native_machine_emergency_restart(void) · 29 unmodified lines
590586 unsigned short mode;
591587
592588 if (reboot_emergency)
593− emergency_vmx_disable_all();
589+ emergency_reboot_disable_virtualization();
594590
595591 tboot_shutdown(TB_SHUTDOWN_REBOOT);
596592
@@ -795,6 +791,17 @@ void machine_crash_shutdown(struct pt_regs *regs) · 198 unmodified lines
795791 /* This is the CPU performing the emergency shutdown work. */
796792 int crashing_cpu = -1;
797793
794+/*
795+ * Disable virtualization, i.e. VMX or SVM, to ensure INIT is recognized during
796+ * reboot. VMX blocks INIT if the CPU is post-VMXON, and SVM blocks INIT if
797+ * GIF=0, i.e. if the crash occurred between CLGI and STGI.
798+ */
799+void cpu_emergency_disable_virtualization(void)
800+{
801+ cpu_emergency_vmxoff();
802+ cpu_emergency_svm_disable();
803+}
804+
798805 #if defined(CONFIG_SMP)
799806
800807 static nmi_shootdown_cb shootdown_callback;
@@ -817,7 +824,14 @@ static int crash_nmi_callback(unsigned int val, struct pt_regs *regs) · 16 unmodified lines
817824 return NMI_HANDLED;
818825 local_irq_disable();
819826
820− shootdown_callback(cpu, regs);
827+ if (shootdown_callback)
828+ shootdown_callback(cpu, regs);
829+
830+ /*
831+ * Prepare the CPU for reboot _after_ invoking the callback so that the
832+ * callback can safely use virtualization instructions, e.g. VMCLEAR.
833+ */
834+ cpu_emergency_disable_virtualization();
821835
822836 atomic_dec(&waiting_for_crash_ipi);
823837 /* Assume hlt works */
@@ -828,18 +842,32 @@ static int crash_nmi_callback(unsigned int val, struct pt_regs *regs) · 4 unmodified lines
828842 return NMI_HANDLED;
829843 }
830844
831−/*
832− * Halt all other CPUs, calling the specified function on each of them
845+/**
846+ * nmi_shootdown_cpus - Stop other CPUs via NMI
847+ * @callback: Optional callback to be invoked from the NMI handler
848+ *
849+ * The NMI handler on the remote CPUs invokes @callback, if not
850+ * NULL, first and then disables virtualization to ensure that
851+ * INIT is recognized during reboot.
833852 *
834− * This function can be used to halt all other CPUs on crash
835− * or emergency reboot time. The function passed as parameter
836− * will be called inside a NMI handler on all CPUs.
853+ * nmi_shootdown_cpus() can only be invoked once. After the first
854+ * invocation all other CPUs are stuck in crash_nmi_callback() and
855+ * cannot respond to a second NMI.
837856 */
838857 void nmi_shootdown_cpus(nmi_shootdown_cb callback)
839858 {
840859 unsigned long msecs;
860+
841861 local_irq_disable();
842862
863+ /*
864+ * Avoid certain doom if a shootdown already occurred; re-registering
865+ * the NMI handler will cause list corruption, modifying the callback
866+ * will do who knows what, etc...
867+ */
868+ if (WARN_ON_ONCE(crash_ipi_issued))
869+ return;
870+
843871 /* Make a note of crashing cpu. Will be used in NMI callback. */
844872 crashing_cpu = safe_smp_processor_id();
845873
@@ -867,7 +895,17 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback) · 21 unmodified lines
867895 msecs--;
868896 }
869897
870− /* Leave the nmi callback set */
898+ /*
899+ * Leave the nmi callback set, shootdown is a one-time thing. Clearing
900+ * the callback could result in a NULL pointer dereference if a CPU
901+ * (finally) responds after the timeout expires.
902+ */
903+}
904+
905+static inline void nmi_shootdown_cpus_on_restart(void)
906+{
907+ if (!crash_ipi_issued)
908+ nmi_shootdown_cpus(NULL);
871909 }
872910
873911 /*
@@ -897,6 +935,8 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback) · 23 unmodified lines
897935 /* No other CPUs to shoot down */
898936 }
899937
938+static inline void nmi_shootdown_cpus_on_restart(void) { }
939+
900940 void run_crash_ipi_callback(struct pt_regs *regs)
901941 {
902942 }
arch/x86/kernel/smp.c+3−3
@@ -32,7 +32,7 @@ · 31 unmodified lines
3232 #include <asm/mce.h>
3333 #include <asm/trace/irq_vectors.h>
3434 #include <asm/kexec.h>
35−#include <asm/virtext.h>
35+#include <asm/reboot.h>
3636
3737 /*
3838 * Some notes on x86 processor bugs affecting SMP operation:
@@ -122,7 +122,7 @@ static int smp_stop_nmi_callback(unsigned int val, struct pt_regs *regs) · 83 unmodified lines
122122 if (raw_smp_processor_id() == atomic_read(&stopping_cpu))
123123 return NMI_HANDLED;
124124
125− cpu_emergency_vmxoff();
125+ cpu_emergency_disable_virtualization();
126126 stop_this_cpu(NULL);
127127
128128 return NMI_HANDLED;
@@ -134,7 +134,7 @@ static int smp_stop_nmi_callback(unsigned int val, struct pt_regs *regs) · 5 unmodified lines
134134 DEFINE_IDTENTRY_SYSVEC(sysvec_reboot)
135135 {
136136 ack_APIC_irq();
137− cpu_emergency_vmxoff();
137+ cpu_emergency_disable_virtualization();
138138 stop_this_cpu(NULL);
139139 }
140140
arch/x86/kvm/Kconfig+1−0
@@ -49,6 +49,7 @@ config KVM · 48 unmodified lines
4949 select SRCU
5050 select INTERVAL_TREE
5151 select HAVE_KVM_PM_NOTIFIER if PM
52+ select KVM_GENERIC_HARDWARE_ENABLING
5253 help
5354 Support hosting fully virtualized guest machines using hardware
5455 virtualization extensions. You will need a fairly recent
arch/x86/kvm/cpuid.c+28−15
@@ -8,6 +8,7 @@ · 7 unmodified lines
88 * Copyright 2011 Red Hat, Inc. and/or its affiliates.
99 * Copyright IBM Corporation, 2008
1010 */
11+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
1112
1213 #include <linux/kvm_host.h>
1314 #include <linux/export.h>
@@ -25,6 +26,7 @@ · 11 unmodified lines
2526 #include "mmu.h"
2627 #include "trace.h"
2728 #include "pmu.h"
29+#include "xen.h"
2830
2931 /*
3032 * Unlike "struct cpuinfo_x86.x86_capability", kvm_cpu_caps doesn't need to be
@@ -180,15 +182,15 @@ static int kvm_cpuid_check_equal(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 · 149 unmodified lines
180182 return 0;
181183 }
182184
183−static void kvm_update_kvm_cpuid_base(struct kvm_vcpu *vcpu)
185+static struct kvm_hypervisor_cpuid kvm_get_hypervisor_cpuid(struct kvm_vcpu *vcpu,
186+ const char *sig)
184187 {
185− u32 function;
188+ struct kvm_hypervisor_cpuid cpuid = {};
186189 struct kvm_cpuid_entry2 *entry;
190+ u32 base;
187191
188− vcpu->arch.kvm_cpuid_base = 0;
189−
190− for_each_possible_hypervisor_cpuid_base(function) {
191− entry = kvm_find_cpuid_entry(vcpu, function);
192+ for_each_possible_hypervisor_cpuid_base(base) {
193+ entry = kvm_find_cpuid_entry(vcpu, base);
192194
193195 if (entry) {
194196 u32 signature[3];
@@ -197,19 +199,21 @@ static void kvm_update_kvm_cpuid_base(struct kvm_vcpu *vcpu) · 2 unmodified lines
197199 signature[1] = entry->ecx;
198200 signature[2] = entry->edx;
199201
200− BUILD_BUG_ON(sizeof(signature) > sizeof(KVM_SIGNATURE));
201− if (!memcmp(signature, KVM_SIGNATURE, sizeof(signature))) {
202− vcpu->arch.kvm_cpuid_base = function;
202+ if (!memcmp(signature, sig, sizeof(signature))) {
203+ cpuid.base = base;
204+ cpuid.limit = entry->eax;
203205 break;
204206 }
205207 }
206208 }
209+
210+ return cpuid;
207211 }
208212
209213 static struct kvm_cpuid_entry2 *__kvm_find_kvm_cpuid_features(struct kvm_vcpu *vcpu,
210214 struct kvm_cpuid_entry2 *entries, int nent)
211215 {
212− u32 base = vcpu->arch.kvm_cpuid_base;
216+ u32 base = vcpu->arch.kvm_cpuid.base;
213217
214218 if (!base)
215219 return NULL;
@@ -439,7 +443,8 @@ static int kvm_set_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 *e2, · 223 unmodified lines
439443 vcpu->arch.cpuid_entries = e2;
440444 vcpu->arch.cpuid_nent = nent;
441445
442− kvm_update_kvm_cpuid_base(vcpu);
446+ vcpu->arch.kvm_cpuid = kvm_get_hypervisor_cpuid(vcpu, KVM_SIGNATURE);
447+ vcpu->arch.xen.cpuid = kvm_get_hypervisor_cpuid(vcpu, XEN_SIGNATURE);
443448 kvm_vcpu_after_set_cpuid(vcpu);
444449
445450 return 0;
@@ -663,8 +668,9 @@ void kvm_set_cpu_caps(void) · 217 unmodified lines
663668 kvm_cpu_cap_set(X86_FEATURE_SPEC_CTRL_SSBD);
664669
665670 kvm_cpu_cap_mask(CPUID_7_1_EAX,
666− F(AVX_VNNI) | F(AVX512_BF16) | F(CMPCCXADD) | F(AMX_FP16) |
667− F(AVX_IFMA)
671+ F(AVX_VNNI) | F(AVX512_BF16) | F(CMPCCXADD) |
672+ F(FZRM) | F(FSRS) | F(FSRC) |
673+ F(AMX_FP16) | F(AVX_IFMA)
668674 );
669675
670676 kvm_cpu_cap_init_kvm_defined(CPUID_7_1_EDX,
@@ -701,6 +707,10 @@ void kvm_set_cpu_caps(void) · 30 unmodified lines
701707 if (!tdp_enabled && IS_ENABLED(CONFIG_X86_64))
702708 kvm_cpu_cap_set(X86_FEATURE_GBPAGES);
703709
710+ kvm_cpu_cap_init_kvm_defined(CPUID_8000_0007_EDX,
711+ SF(CONSTANT_TSC)
712+ );
713+
704714 kvm_cpu_cap_mask(CPUID_8000_0008_EBX,
705715 F(CLZERO) | F(XSAVEERPTR) |
706716 F(WBNOINVD) | F(AMD_IBPB) | F(AMD_IBRS) | F(AMD_SSBD) | F(VIRT_SSBD) |
@@ -1169,8 +1179,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · 462 unmodified lines
11691179 entry->edx &= ~GENMASK(17, 16);
11701180 break;
11711181 case 0x80000007: /* Advanced power management */
1172− /* invariant TSC is CPUID.80000007H:EDX[8] */
1173− entry->edx &= (1 << 8);
1182+ cpuid_entry_override(entry, CPUID_8000_0007_EDX);
1183+
11741184 /* mask against host */
11751185 entry->edx &= boot_cpu_data.x86_power;
11761186 entry->eax = entry->ebx = entry->ecx = 0;
@@ -1485,6 +1495,9 @@ bool kvm_cpuid(struct kvm_vcpu *vcpu, u32 *eax, u32 *ebx, · 308 unmodified lines
14851495 if (!__kvm_get_msr(vcpu, MSR_IA32_TSX_CTRL, &data, true) &&
14861496 (data & TSX_CTRL_CPUID_CLEAR))
14871497 *ebx &= ~(F(RTM) | F(HLE));
1498+ } else if (function == 0x80000007) {
1499+ if (kvm_hv_invtsc_suppressed(vcpu))
1500+ *edx &= ~SF(CONSTANT_TSC);
14881501 }
14891502 } else {
14901503 *eax = *ebx = *ecx = *edx = 0;
arch/x86/kvm/debugfs.c+2−0
@@ -4,6 +4,8 @@ · 3 unmodified lines
44 *
55 * Copyright 2016 Red Hat, Inc. and/or its affiliates.
66 */
7+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
8+
79 #include <linux/kvm_host.h>
810 #include <linux/debugfs.h>
911 #include "lapic.h"
arch/x86/kvm/emulate.c+9−9
@@ -17,6 +17,7 @@ · 16 unmodified lines
1717 *
1818 * From: xen-unstable 10676:af9809f51f81a3c43f276f00c81a52ef558afda4
1919 */
20+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
2021
2122 #include <linux/kvm_host.h>
2223 #include "kvm_cache_regs.h"
@@ -1633,7 +1634,7 @@ static int __load_segment_descriptor(struct x86_emulate_ctxt *ctxt, · 1610 unmodified lines
16331634 case VCPU_SREG_SS:
16341635 /*
16351636 * segment is not a writable data segment or segment
1636− * selector's RPL != CPL or segment selector's RPL != CPL
1637+ * selector's RPL != CPL or DPL != CPL
16371638 */
16381639 if (rpl != cpl || (seg_desc.type & 0xa) != 0x2 || dpl != cpl)
16391640 goto exception;
@@ -1695,11 +1696,11 @@ static int __load_segment_descriptor(struct x86_emulate_ctxt *ctxt, · 55 unmodified lines
16951696 /*
16961697 * segment is not a data or readable code segment or
16971698 * ((segment is a data or nonconforming code segment)
1698− * and (both RPL and CPL > DPL))
1699+ * and ((RPL > DPL) or (CPL > DPL)))
16991700 */
17001701 if ((seg_desc.type & 0xa) == 0x8 ||
17011702 (((seg_desc.type & 0xc) != 0xc) &&
1702− (rpl > dpl && cpl > dpl)))
1703+ (rpl > dpl || cpl > dpl)))
17031704 goto exception;
17041705 break;
17051706 }
@@ -2309,7 +2310,7 @@ static int em_lseg(struct x86_emulate_ctxt *ctxt) · 603 unmodified lines
23092310
23102311 static int em_rsm(struct x86_emulate_ctxt *ctxt)
23112312 {
2312− if ((ctxt->ops->get_hflags(ctxt) & X86EMUL_SMM_MASK) == 0)
2313+ if (!ctxt->ops->is_smm(ctxt))
23132314 return emulate_ud(ctxt);
23142315
23152316 if (ctxt->ops->leave_smm(ctxt))
@@ -5132,7 +5133,7 @@ int x86_emulate_insn(struct x86_emulate_ctxt *ctxt) · 2816 unmodified lines
51325133 const struct x86_emulate_ops *ops = ctxt->ops;
51335134 int rc = X86EMUL_CONTINUE;
51345135 int saved_dst_type = ctxt->dst.type;
5135− unsigned emul_flags;
5136+ bool is_guest_mode = ctxt->ops->is_guest_mode(ctxt);
51365137
51375138 ctxt->mem_read.pos = 0;
51385139
@@ -5147,7 +5148,6 @@ int x86_emulate_insn(struct x86_emulate_ctxt *ctxt) · 8 unmodified lines
51475148 goto done;
51485149 }
51495150
5150− emul_flags = ctxt->ops->get_hflags(ctxt);
51515151 if (unlikely(ctxt->d &
51525152 (No64|Undefined|Sse|Mmx|Intercept|CheckPerm|Priv|Prot|String))) {
51535153 if ((ctxt->mode == X86EMUL_MODE_PROT64 && (ctxt->d & No64)) ||
@@ -5181,7 +5181,7 @@ int x86_emulate_insn(struct x86_emulate_ctxt *ctxt) · 27 unmodified lines
51815181 fetch_possible_mmx_operand(&ctxt->dst);
51825182 }
51835183
5184− if (unlikely(emul_flags & X86EMUL_GUEST_MASK) && ctxt->intercept) {
5184+ if (unlikely(is_guest_mode) && ctxt->intercept) {
51855185 rc = emulator_check_intercept(ctxt, ctxt->intercept,
51865186 X86_ICPT_PRE_EXCEPT);
51875187 if (rc != X86EMUL_CONTINUE)
@@ -5210,7 +5210,7 @@ int x86_emulate_insn(struct x86_emulate_ctxt *ctxt) · 22 unmodified lines
52105210 goto done;
52115211 }
52125212
5213− if (unlikely(emul_flags & X86EMUL_GUEST_MASK) && (ctxt->d & Intercept)) {
5213+ if (unlikely(is_guest_mode) && (ctxt->d & Intercept)) {
52145214 rc = emulator_check_intercept(ctxt, ctxt->intercept,
52155215 X86_ICPT_POST_EXCEPT);
52165216 if (rc != X86EMUL_CONTINUE)
@@ -5264,7 +5264,7 @@ int x86_emulate_insn(struct x86_emulate_ctxt *ctxt) · 47 unmodified lines
52645264
52655265 special_insn:
52665266
5267− if (unlikely(emul_flags & X86EMUL_GUEST_MASK) && (ctxt->d & Intercept)) {
5267+ if (unlikely(is_guest_mode) && (ctxt->d & Intercept)) {
52685268 rc = emulator_check_intercept(ctxt, ctxt->intercept,
52695269 X86_ICPT_POST_MEMACCESS);
52705270 if (rc != X86EMUL_CONTINUE)
arch/x86/kvm/hyperv.c+63−22
@@ -17,6 +17,7 @@ · 16 unmodified lines
1717 * Ben-Ami Yassour <benami@il.ibm.com>
1818 * Andrey Smetanin <asmetanin@virtuozzo.com>
1919 */
20+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
2021
2122 #include "x86.h"
2223 #include "lapic.h"
@@ -43,6 +44,24 @@ · 20 unmodified lines
4344
4445 #define KVM_HV_MAX_SPARSE_VCPU_SET_BITS DIV_ROUND_UP(KVM_MAX_VCPUS, HV_VCPUS_PER_SPARSE_BANK)
4546
47+/*
48+ * As per Hyper-V TLFS, extended hypercalls start from 0x8001
49+ * (HvExtCallQueryCapabilities). Response of this hypercalls is a 64 bit value
50+ * where each bit tells which extended hypercall is available besides
51+ * HvExtCallQueryCapabilities.
52+ *
53+ * 0x8001 - First extended hypercall, HvExtCallQueryCapabilities, no bit
54+ * assigned.
55+ *
56+ * 0x8002 - Bit 0
57+ * 0x8003 - Bit 1
58+ * ..
59+ * 0x8041 - Bit 63
60+ *
61+ * Therefore, HV_EXT_CALL_MAX = 0x8001 + 64
62+ */
63+#define HV_EXT_CALL_MAX (HV_EXT_CALL_QUERY_CAPABILITIES + 64)
64+
4665 static void stimer_mark_pending(struct kvm_vcpu_hv_stimer *stimer,
4766 bool vcpu_kick);
4867
@@ -999,6 +1018,7 @@ static bool kvm_hv_msr_partition_wide(u32 msr) · 950 unmodified lines
9991018 case HV_X64_MSR_REENLIGHTENMENT_CONTROL:
10001019 case HV_X64_MSR_TSC_EMULATION_CONTROL:
10011020 case HV_X64_MSR_TSC_EMULATION_STATUS:
1021+ case HV_X64_MSR_TSC_INVARIANT_CONTROL:
10021022 case HV_X64_MSR_SYNDBG_OPTIONS:
10031023 case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER:
10041024 r = true;
@@ -1283,6 +1303,9 @@ static bool hv_check_msr_access(struct kvm_vcpu_hv *hv_vcpu, u32 msr) · 278 unmodified lines
12831303 case HV_X64_MSR_TSC_EMULATION_STATUS:
12841304 return hv_vcpu->cpuid_cache.features_eax &
12851305 HV_ACCESS_REENLIGHTENMENT;
1306+ case HV_X64_MSR_TSC_INVARIANT_CONTROL:
1307+ return hv_vcpu->cpuid_cache.features_eax &
1308+ HV_ACCESS_TSC_INVARIANT;
12861309 case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4:
12871310 case HV_X64_MSR_CRASH_CTL:
12881311 return hv_vcpu->cpuid_cache.features_edx &
@@ -1410,12 +1433,22 @@ static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data, · 121 unmodified lines
14101433 if (!host)
14111434 return 1;
14121435 break;
1436+ case HV_X64_MSR_TSC_INVARIANT_CONTROL:
1437+ /* Only bit 0 is supported */
1438+ if (data & ~HV_EXPOSE_INVARIANT_TSC)
1439+ return 1;
1440+
1441+ /* The feature can't be disabled from the guest */
1442+ if (!host && hv->hv_invtsc_control && !data)
1443+ return 1;
1444+
1445+ hv->hv_invtsc_control = data;
1446+ break;
14131447 case HV_X64_MSR_SYNDBG_OPTIONS:
14141448 case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER:
14151449 return syndbg_set_msr(vcpu, msr, data, host);
14161450 default:
1417− vcpu_unimpl(vcpu, "Hyper-V unhandled wrmsr: 0x%x data 0x%llx\n",
1418− msr, data);
1451+ kvm_pr_unimpl_wrmsr(vcpu, msr, data);
14191452 return 1;
14201453 }
14211454 return 0;
@@ -1536,8 +1569,7 @@ static int kvm_hv_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool host) · 114 unmodified lines
15361569 return 1;
15371570 break;
15381571 default:
1539− vcpu_unimpl(vcpu, "Hyper-V unhandled wrmsr: 0x%x data 0x%llx\n",
1540− msr, data);
1572+ kvm_pr_unimpl_wrmsr(vcpu, msr, data);
15411573 return 1;
15421574 }
15431575
@@ -1585,11 +1617,14 @@ static int kvm_hv_get_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata, · 41 unmodified lines
15851617 case HV_X64_MSR_TSC_EMULATION_STATUS:
15861618 data = hv->hv_tsc_emulation_status;
15871619 break;
1620+ case HV_X64_MSR_TSC_INVARIANT_CONTROL:
1621+ data = hv->hv_invtsc_control;
1622+ break;
15881623 case HV_X64_MSR_SYNDBG_OPTIONS:
15891624 case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER:
15901625 return syndbg_get_msr(vcpu, msr, pdata, host);
15911626 default:
1592− vcpu_unimpl(vcpu, "Hyper-V unhandled rdmsr: 0x%x\n", msr);
1627+ kvm_pr_unimpl_rdmsr(vcpu, msr);
15931628 return 1;
15941629 }
15951630
@@ -1654,7 +1689,7 @@ static int kvm_hv_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata, · 58 unmodified lines
16541689 data = APIC_BUS_FREQUENCY;
16551690 break;
16561691 default:
1657− vcpu_unimpl(vcpu, "Hyper-V unhandled rdmsr: 0x%x\n", msr);
1692+ kvm_pr_unimpl_rdmsr(vcpu, msr);
16581693 return 1;
16591694 }
16601695 *pdata = data;
@@ -2420,6 +2455,9 @@ static bool hv_check_hypercall_access(struct kvm_vcpu_hv *hv_vcpu, u16 code) · 759 unmodified lines
24202455 case HVCALL_SEND_IPI:
24212456 return hv_vcpu->cpuid_cache.enlightenments_eax &
24222457 HV_X64_CLUSTER_IPI_RECOMMENDED;
2458+ case HV_EXT_CALL_QUERY_CAPABILITIES ... HV_EXT_CALL_MAX:
2459+ return hv_vcpu->cpuid_cache.features_ebx &
2460+ HV_ENABLE_EXTENDED_HYPERCALLS;
24232461 default:
24242462 break;
24252463 }
@@ -2512,14 +2550,7 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) · 86 unmodified lines
25122550 ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
25132551 break;
25142552 }
2515− vcpu->run->exit_reason = KVM_EXIT_HYPERV;
2516− vcpu->run->hyperv.type = KVM_EXIT_HYPERV_HCALL;
2517− vcpu->run->hyperv.u.hcall.input = hc.param;
2518− vcpu->run->hyperv.u.hcall.params[0] = hc.ingpa;
2519− vcpu->run->hyperv.u.hcall.params[1] = hc.outgpa;
2520− vcpu->arch.complete_userspace_io =
2521− kvm_hv_hypercall_complete_userspace;
2522− return 0;
2553+ goto hypercall_userspace_exit;
25232554 case HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST:
25242555 if (unlikely(hc.var_cnt)) {
25252556 ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
@@ -2578,15 +2609,14 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) · 52 unmodified lines
25782609 ret = HV_STATUS_OPERATION_DENIED;
25792610 break;
25802611 }
2581− vcpu->run->exit_reason = KVM_EXIT_HYPERV;
2582− vcpu->run->hyperv.type = KVM_EXIT_HYPERV_HCALL;
2583− vcpu->run->hyperv.u.hcall.input = hc.param;
2584− vcpu->run->hyperv.u.hcall.params[0] = hc.ingpa;
2585− vcpu->run->hyperv.u.hcall.params[1] = hc.outgpa;
2586− vcpu->arch.complete_userspace_io =
2587− kvm_hv_hypercall_complete_userspace;
2588− return 0;
2612+ goto hypercall_userspace_exit;
25892613 }
2614+ case HV_EXT_CALL_QUERY_CAPABILITIES ... HV_EXT_CALL_MAX:
2615+ if (unlikely(hc.fast)) {
2616+ ret = HV_STATUS_INVALID_PARAMETER;
2617+ break;
2618+ }
2619+ goto hypercall_userspace_exit;
25902620 default:
25912621 ret = HV_STATUS_INVALID_HYPERCALL_CODE;
25922622 break;
@@ -2594,6 +2624,15 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) · 1 unmodified line
25942624
25952625 hypercall_complete:
25962626 return kvm_hv_hypercall_complete(vcpu, ret);
2627+
2628+hypercall_userspace_exit:
2629+ vcpu->run->exit_reason = KVM_EXIT_HYPERV;
2630+ vcpu->run->hyperv.type = KVM_EXIT_HYPERV_HCALL;
2631+ vcpu->run->hyperv.u.hcall.input = hc.param;
2632+ vcpu->run->hyperv.u.hcall.params[0] = hc.ingpa;
2633+ vcpu->run->hyperv.u.hcall.params[1] = hc.outgpa;
2634+ vcpu->arch.complete_userspace_io = kvm_hv_hypercall_complete_userspace;
2635+ return 0;
25972636 }
25982637
25992638 void kvm_hv_init_vm(struct kvm *kvm)
@@ -2733,9 +2772,11 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid, · 133 unmodified lines
27332772 ent->eax |= HV_MSR_REFERENCE_TSC_AVAILABLE;
27342773 ent->eax |= HV_ACCESS_FREQUENCY_MSRS;
27352774 ent->eax |= HV_ACCESS_REENLIGHTENMENT;
2775+ ent->eax |= HV_ACCESS_TSC_INVARIANT;
27362776
27372777 ent->ebx |= HV_POST_MESSAGES;
27382778 ent->ebx |= HV_SIGNAL_EVENTS;
2779+ ent->ebx |= HV_ENABLE_EXTENDED_HYPERCALLS;
27392780
27402781 ent->edx |= HV_X64_HYPERCALL_XMM_INPUT_AVAILABLE;
27412782 ent->edx |= HV_FEATURE_FREQUENCY_MSRS_AVAILABLE;
arch/x86/kvm/hyperv.h+27−0
@@ -136,6 +136,33 @@ static inline bool kvm_hv_has_stimer_pending(struct kvm_vcpu *vcpu) · 135 unmodified lines
136136 HV_SYNIC_STIMER_COUNT);
137137 }
138138
139+/*
140+ * With HV_ACCESS_TSC_INVARIANT feature, invariant TSC (CPUID.80000007H:EDX[8])
141+ * is only observed after HV_X64_MSR_TSC_INVARIANT_CONTROL was written to.
142+ */
143+static inline bool kvm_hv_invtsc_suppressed(struct kvm_vcpu *vcpu)
144+{
145+ struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
146+
147+ /*
148+ * If Hyper-V's invariant TSC control is not exposed to the guest,
149+ * the invariant TSC CPUID flag is not suppressed, Windows guests were
150+ * observed to be able to handle it correctly. Going forward, VMMs are
151+ * encouraged to enable Hyper-V's invariant TSC control when invariant
152+ * TSC CPUID flag is set to make KVM's behavior match genuine Hyper-V.
153+ */
154+ if (!hv_vcpu ||
155+ !(hv_vcpu->cpuid_cache.features_eax & HV_ACCESS_TSC_INVARIANT))
156+ return false;
157+
158+ /*
159+ * If Hyper-V's invariant TSC control is exposed to the guest, KVM is
160+ * responsible for suppressing the invariant TSC CPUID flag if the
161+ * Hyper-V control is not enabled.
162+ */
163+ return !(to_kvm_hv(vcpu->kvm)->hv_invtsc_control & HV_EXPOSE_INVARIANT_TSC);
164+}
165+
139166 void kvm_hv_process_stimers(struct kvm_vcpu *vcpu);
140167
141168 void kvm_hv_setup_tsc_page(struct kvm *kvm,
arch/x86/kvm/i8254.c+2−2
@@ -30,7 +30,7 @@ · 29 unmodified lines
3030 * Based on QEMU and Xen.
3131 */
3232
33−#define pr_fmt(fmt) "pit: " fmt
33+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
3434
3535 #include <linux/kvm_host.h>
3636 #include <linux/slab.h>
@@ -351,7 +351,7 @@ static void create_pit_timer(struct kvm_pit *pit, u32 val, int is_period) · 314 unmodified lines
351351
352352 if (ps->period < min_period) {
353353 pr_info_ratelimited(
354− "kvm: requested %lld ns "
354+ "requested %lld ns "
355355 "i8254 timer period limited to %lld ns\n",
356356 ps->period, min_period);
357357 ps->period = min_period;
arch/x86/kvm/i8259.c+3−1
@@ -26,6 +26,8 @@ · 25 unmodified lines
2626 * Yaozu (Eddie) Dong <Eddie.dong@intel.com>
2727 * Port from Qemu.
2828 */
29+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
30+
2931 #include <linux/mm.h>
3032 #include <linux/slab.h>
3133 #include <linux/bitops.h>
@@ -35,7 +37,7 @@ · 3 unmodified lines
3537 #include "trace.h"
3638
3739 #define pr_pic_unimpl(fmt, ...) \
38− pr_err_ratelimited("kvm: pic: " fmt, ## __VA_ARGS__)
40+ pr_err_ratelimited("pic: " fmt, ## __VA_ARGS__)
3941
4042 static void pic_irq_request(struct kvm *kvm, int level);
4143
arch/x86/kvm/ioapic.c+1−0
@@ -26,6 +26,7 @@ · 25 unmodified lines
2626 * Yaozu (Eddie) Dong <eddie.dong@intel.com>
2727 * Based on Xen 3.1 code.
2828 */
29+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
2930
3031 #include <linux/kvm_host.h>
3132 #include <linux/kvm.h>
arch/x86/kvm/irq.c+1−0
@@ -7,6 +7,7 @@ · 6 unmodified lines
77 * Authors:
88 * Yaozu (Eddie) Dong <Eddie.dong@intel.com>
99 */
10+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
1011
1112 #include <linux/export.h>
1213 #include <linux/kvm_host.h>
arch/x86/kvm/irq_comm.c+4−3
@@ -8,6 +8,7 @@ · 7 unmodified lines
88 *
99 * Copyright 2010 Red Hat, Inc. and/or its affiliates.
1010 */
11+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
1112
1213 #include <linux/kvm_host.h>
1314 #include <linux/slab.h>
@@ -56,7 +57,7 @@ int kvm_irq_delivery_to_apic(struct kvm *kvm, struct kvm_lapic *src, · 42 unmodified lines
5657
5758 if (irq->dest_mode == APIC_DEST_PHYSICAL &&
5859 irq->dest_id == 0xff && kvm_lowest_prio_delivery(irq)) {
59− printk(KERN_INFO "kvm: apic: phys broadcast and lowest prio\n");
60+ pr_info("apic: phys broadcast and lowest prio\n");
6061 irq->delivery_mode = APIC_DM_FIXED;
6162 }
6263
@@ -199,7 +200,7 @@ int kvm_request_irq_source_id(struct kvm *kvm) · 136 unmodified lines
199200 irq_source_id = find_first_zero_bit(bitmap, BITS_PER_LONG);
200201
201202 if (irq_source_id >= BITS_PER_LONG) {
202− printk(KERN_WARNING "kvm: exhaust allocatable IRQ sources!\n");
203+ pr_warn("exhausted allocatable IRQ sources!\n");
203204 irq_source_id = -EFAULT;
204205 goto unlock;
205206 }
@@ -221,7 +222,7 @@ void kvm_free_irq_source_id(struct kvm *kvm, int irq_source_id) · 15 unmodified lines
221222 mutex_lock(&kvm->irq_lock);
222223 if (irq_source_id < 0 ||
223224 irq_source_id >= BITS_PER_LONG) {
224− printk(KERN_ERR "kvm: IRQ source ID out of range!\n");
225+ pr_err("IRQ source ID out of range!\n");
225226 goto unlock;
226227 }
227228 clear_bit(irq_source_id, &kvm->arch.irq_sources_bitmap);
arch/x86/kvm/kvm_cache_regs.h+12−0
@@ -75,6 +75,18 @@ static inline void kvm_register_mark_dirty(struct kvm_vcpu *vcpu, · 74 unmodified lines
7575 __set_bit(reg, (unsigned long *)&vcpu->arch.regs_dirty);
7676 }
7777
78+/*
79+ * kvm_register_test_and_mark_available() is a special snowflake that uses an
80+ * arch bitop directly to avoid the explicit instrumentation that comes with
81+ * the generic bitops. This allows code that cannot be instrumented (noinstr
82+ * functions), e.g. the low level VM-Enter/VM-Exit paths, to cache registers.
83+ */
84+static __always_inline bool kvm_register_test_and_mark_available(struct kvm_vcpu *vcpu,
85+ enum kvm_reg reg)
86+{
87+ return arch___test_and_set_bit(reg, (unsigned long *)&vcpu->arch.regs_avail);
88+}
89+
7890 /*
7991 * The "raw" register helpers are only for cases where the full 64 bits of a
8092 * register are read/written irrespective of current vCPU mode. In other words,
arch/x86/kvm/kvm_emulate.h+2−5
@@ -220,7 +220,8 @@ struct x86_emulate_ops { · 219 unmodified lines
220220
221221 void (*set_nmi_mask)(struct x86_emulate_ctxt *ctxt, bool masked);
222222
223− unsigned (*get_hflags)(struct x86_emulate_ctxt *ctxt);
223+ bool (*is_smm)(struct x86_emulate_ctxt *ctxt);
224+ bool (*is_guest_mode)(struct x86_emulate_ctxt *ctxt);
224225 int (*leave_smm)(struct x86_emulate_ctxt *ctxt);
225226 void (*triple_fault)(struct x86_emulate_ctxt *ctxt);
226227 int (*set_xcr)(struct x86_emulate_ctxt *ctxt, u32 index, u64 xcr);
@@ -275,10 +276,6 @@ enum x86emul_mode { · 48 unmodified lines
275276 X86EMUL_MODE_PROT64, /* 64-bit (long) mode. */
276277 };
277278
278−/* These match some of the HF_* flags defined in kvm_host.h */
279−#define X86EMUL_GUEST_MASK (1 << 5) /* VCPU is in guest-mode */
280−#define X86EMUL_SMM_MASK (1 << 6)
281−
282279 /*
283280 * fastop functions are declared as taking a never-defined fastop parameter,
284281 * so they can't be called from C directly.
arch/x86/kvm/kvm_onhyperv.c+1−0
@@ -2,6 +2,7 @@ · 1 unmodified line
22 /*
33 * KVM L1 hypervisor optimizations on Hyper-V.
44 */
5+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
56
67 #include <linux/kvm_host.h>
78 #include <asm/mshyperv.h>
arch/x86/kvm/lapic.c+296−108
@@ -15,6 +15,7 @@ · 14 unmodified lines
1515 *
1616 * Based on Xen 3.1 code, Copyright (c) 2004, Intel Corporation.
1717 */
18+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
1819
1920 #include <linux/kvm_host.h>
2021 #include <linux/kvm.h>
@@ -166,9 +167,19 @@ static bool kvm_use_posted_timer_interrupt(struct kvm_vcpu *vcpu) · 145 unmodified lines
166167 return kvm_can_post_timer_interrupt(vcpu) && vcpu->mode == IN_GUEST_MODE;
167168 }
168169
170+static inline u32 kvm_apic_calc_x2apic_ldr(u32 id)
171+{
172+ return ((id >> 4) << 16) | (1 << (id & 0xf));
173+}
174+
169175 static inline bool kvm_apic_map_get_logical_dest(struct kvm_apic_map *map,
170176 u32 dest_id, struct kvm_lapic ***cluster, u16 *mask) {
171− switch (map->mode) {
177+ switch (map->logical_mode) {
178+ case KVM_APIC_MODE_SW_DISABLED:
179+ /* Arbitrarily use the flat map so that @cluster isn't NULL. */
180+ *cluster = map->xapic_flat_map;
181+ *mask = 0;
182+ return true;
172183 case KVM_APIC_MODE_X2APIC: {
173184 u32 offset = (dest_id >> 16) * 16;
174185 u32 max_apic_id = map->max_apic_id;
@@ -193,8 +204,10 @@ static inline bool kvm_apic_map_get_logical_dest(struct kvm_apic_map *map, · 18 unmodified lines
193204 *cluster = map->xapic_cluster_map[(dest_id >> 4) & 0xf];
194205 *mask = dest_id & 0xf;
195206 return true;
207+ case KVM_APIC_MODE_MAP_DISABLED:
208+ return false;
196209 default:
197− /* Not optimized. */
210+ WARN_ON_ONCE(1);
198211 return false;
199212 }
200213 }
@@ -206,6 +219,134 @@ static void kvm_apic_map_free(struct rcu_head *rcu) · 5 unmodified lines
206219 kvfree(map);
207220 }
208221
222+static int kvm_recalculate_phys_map(struct kvm_apic_map *new,
223+ struct kvm_vcpu *vcpu,
224+ bool *xapic_id_mismatch)
225+{
226+ struct kvm_lapic *apic = vcpu->arch.apic;
227+ u32 x2apic_id = kvm_x2apic_id(apic);
228+ u32 xapic_id = kvm_xapic_id(apic);
229+ u32 physical_id;
230+
231+ /*
232+ * Deliberately truncate the vCPU ID when detecting a mismatched APIC
233+ * ID to avoid false positives if the vCPU ID, i.e. x2APIC ID, is a
234+ * 32-bit value. Any unwanted aliasing due to truncation results will
235+ * be detected below.
236+ */
237+ if (!apic_x2apic_mode(apic) && xapic_id != (u8)vcpu->vcpu_id)
238+ *xapic_id_mismatch = true;
239+
240+ /*
241+ * Apply KVM's hotplug hack if userspace has enable 32-bit APIC IDs.
242+ * Allow sending events to vCPUs by their x2APIC ID even if the target
243+ * vCPU is in legacy xAPIC mode, and silently ignore aliased xAPIC IDs
244+ * (the x2APIC ID is truncated to 8 bits, causing IDs > 0xff to wrap
245+ * and collide).
246+ *
247+ * Honor the architectural (and KVM's non-optimized) behavior if
248+ * userspace has not enabled 32-bit x2APIC IDs. Each APIC is supposed
249+ * to process messages independently. If multiple vCPUs have the same
250+ * effective APIC ID, e.g. due to the x2APIC wrap or because the guest
251+ * manually modified its xAPIC IDs, events targeting that ID are
252+ * supposed to be recognized by all vCPUs with said ID.
253+ */
254+ if (vcpu->kvm->arch.x2apic_format) {
255+ /* See also kvm_apic_match_physical_addr(). */
256+ if ((apic_x2apic_mode(apic) || x2apic_id > 0xff) &&
257+ x2apic_id <= new->max_apic_id)
258+ new->phys_map[x2apic_id] = apic;
259+
260+ if (!apic_x2apic_mode(apic) && !new->phys_map[xapic_id])
261+ new->phys_map[xapic_id] = apic;
262+ } else {
263+ /*
264+ * Disable the optimized map if the physical APIC ID is already
265+ * mapped, i.e. is aliased to multiple vCPUs. The optimized
266+ * map requires a strict 1:1 mapping between IDs and vCPUs.
267+ */
268+ if (apic_x2apic_mode(apic))
269+ physical_id = x2apic_id;
270+ else
271+ physical_id = xapic_id;
272+
273+ if (new->phys_map[physical_id])
274+ return -EINVAL;
275+
276+ new->phys_map[physical_id] = apic;
277+ }
278+
279+ return 0;
280+}
281+
282+static void kvm_recalculate_logical_map(struct kvm_apic_map *new,
283+ struct kvm_vcpu *vcpu)
284+{
285+ struct kvm_lapic *apic = vcpu->arch.apic;
286+ enum kvm_apic_logical_mode logical_mode;
287+ struct kvm_lapic **cluster;
288+ u16 mask;
289+ u32 ldr;
290+
291+ if (new->logical_mode == KVM_APIC_MODE_MAP_DISABLED)
292+ return;
293+
294+ if (!kvm_apic_sw_enabled(apic))
295+ return;
296+
297+ ldr = kvm_lapic_get_reg(apic, APIC_LDR);
298+ if (!ldr)
299+ return;
300+
301+ if (apic_x2apic_mode(apic)) {
302+ logical_mode = KVM_APIC_MODE_X2APIC;
303+ } else {
304+ ldr = GET_APIC_LOGICAL_ID(ldr);
305+ if (kvm_lapic_get_reg(apic, APIC_DFR) == APIC_DFR_FLAT)
306+ logical_mode = KVM_APIC_MODE_XAPIC_FLAT;
307+ else
308+ logical_mode = KVM_APIC_MODE_XAPIC_CLUSTER;
309+ }
310+
311+ /*
312+ * To optimize logical mode delivery, all software-enabled APICs must
313+ * be configured for the same mode.
314+ */
315+ if (new->logical_mode == KVM_APIC_MODE_SW_DISABLED) {
316+ new->logical_mode = logical_mode;
317+ } else if (new->logical_mode != logical_mode) {
318+ new->logical_mode = KVM_APIC_MODE_MAP_DISABLED;
319+ return;
320+ }
321+
322+ /*
323+ * In x2APIC mode, the LDR is read-only and derived directly from the
324+ * x2APIC ID, thus is guaranteed to be addressable. KVM reuses
325+ * kvm_apic_map.phys_map to optimize logical mode x2APIC interrupts by
326+ * reversing the LDR calculation to get cluster of APICs, i.e. no
327+ * additional work is required.
328+ */
329+ if (apic_x2apic_mode(apic)) {
330+ WARN_ON_ONCE(ldr != kvm_apic_calc_x2apic_ldr(kvm_x2apic_id(apic)));
331+ return;
332+ }
333+
334+ if (WARN_ON_ONCE(!kvm_apic_map_get_logical_dest(new, ldr,
335+ &cluster, &mask))) {
336+ new->logical_mode = KVM_APIC_MODE_MAP_DISABLED;
337+ return;
338+ }
339+
340+ if (!mask)
341+ return;
342+
343+ ldr = ffs(mask) - 1;
344+ if (!is_power_of_2(mask) || cluster[ldr])
345+ new->logical_mode = KVM_APIC_MODE_MAP_DISABLED;
346+ else
347+ cluster[ldr] = apic;
348+}
349+
209350 /*
210351 * CLEAN -> DIRTY and UPDATE_IN_PROGRESS -> DIRTY changes happen without a lock.
211352 *
@@ -224,6 +365,7 @@ void kvm_recalculate_apic_map(struct kvm *kvm) · 12 unmodified lines
224365 struct kvm_vcpu *vcpu;
225366 unsigned long i;
226367 u32 max_id = 255; /* enough space for any xAPIC ID */
368+ bool xapic_id_mismatch = false;
227369
228370 /* Read kvm->arch.apic_map_dirty before kvm->arch.apic_map. */
229371 if (atomic_read_acquire(&kvm->arch.apic_map_dirty) == CLEAN)
@@ -256,54 +398,41 @@ void kvm_recalculate_apic_map(struct kvm *kvm) · 26 unmodified lines
256398 goto out;
257399
258400 new->max_apic_id = max_id;
401+ new->logical_mode = KVM_APIC_MODE_SW_DISABLED;
259402
260403 kvm_for_each_vcpu(i, vcpu, kvm) {
261− struct kvm_lapic *apic = vcpu->arch.apic;
262− struct kvm_lapic **cluster;
263− u16 mask;
264− u32 ldr;
265− u8 xapic_id;
266− u32 x2apic_id;
267−
268404 if (!kvm_apic_present(vcpu))
269405 continue;
270406
271− xapic_id = kvm_xapic_id(apic);
272− x2apic_id = kvm_x2apic_id(apic);
273−
274− /* Hotplug hack: see kvm_apic_match_physical_addr(), ... */
275− if ((apic_x2apic_mode(apic) || x2apic_id > 0xff) &&
276− x2apic_id <= new->max_apic_id)
277− new->phys_map[x2apic_id] = apic;
278− /*
279− * ... xAPIC ID of VCPUs with APIC ID > 0xff will wrap-around,
280− * prevent them from masking VCPUs with APIC ID <= 0xff.
281− */
282− if (!apic_x2apic_mode(apic) && !new->phys_map[xapic_id])
283− new->phys_map[xapic_id] = apic;
284−
285− if (!kvm_apic_sw_enabled(apic))
286− continue;
287−
288− ldr = kvm_lapic_get_reg(apic, APIC_LDR);
289−
290− if (apic_x2apic_mode(apic)) {
291− new->mode |= KVM_APIC_MODE_X2APIC;
292− } else if (ldr) {
293− ldr = GET_APIC_LOGICAL_ID(ldr);
294− if (kvm_lapic_get_reg(apic, APIC_DFR) == APIC_DFR_FLAT)
295− new->mode |= KVM_APIC_MODE_XAPIC_FLAT;
296− else
297− new->mode |= KVM_APIC_MODE_XAPIC_CLUSTER;
407+ if (kvm_recalculate_phys_map(new, vcpu, &xapic_id_mismatch)) {
408+ kvfree(new);
409+ new = NULL;
410+ goto out;
298411 }
299412
300− if (!kvm_apic_map_get_logical_dest(new, ldr, &cluster, &mask))
301− continue;
302−
303− if (mask)
304− cluster[ffs(mask) - 1] = apic;
413+ kvm_recalculate_logical_map(new, vcpu);
305414 }
306415 out:
416+ /*
417+ * The optimized map is effectively KVM's internal version of APICv,
418+ * and all unwanted aliasing that results in disabling the optimized
419+ * map also applies to APICv.
420+ */
421+ if (!new)
422+ kvm_set_apicv_inhibit(kvm, APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED);
423+ else
424+ kvm_clear_apicv_inhibit(kvm, APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED);
425+
426+ if (!new || new->logical_mode == KVM_APIC_MODE_MAP_DISABLED)
427+ kvm_set_apicv_inhibit(kvm, APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED);
428+ else
429+ kvm_clear_apicv_inhibit(kvm, APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED);
430+
431+ if (xapic_id_mismatch)
432+ kvm_set_apicv_inhibit(kvm, APICV_INHIBIT_REASON_APIC_ID_MODIFIED);
433+ else
434+ kvm_clear_apicv_inhibit(kvm, APICV_INHIBIT_REASON_APIC_ID_MODIFIED);
435+
307436 old = rcu_dereference_protected(kvm->arch.apic_map,
308437 lockdep_is_held(&kvm->arch.apic_map_lock));
309438 rcu_assign_pointer(kvm->arch.apic_map, new);
@@ -360,11 +489,6 @@ static inline void kvm_apic_set_dfr(struct kvm_lapic *apic, u32 val) · 50 unmodified lines
360489 atomic_set_release(&apic->vcpu->kvm->arch.apic_map_dirty, DIRTY);
361490 }
362491
363−static inline u32 kvm_apic_calc_x2apic_ldr(u32 id)
364−{
365− return ((id >> 4) << 16) | (1 << (id & 0xf));
366−}
367−
368492 static inline void kvm_apic_set_x2apic_id(struct kvm_lapic *apic, u32 id)
369493 {
370494 u32 ldr = kvm_apic_calc_x2apic_ldr(id);
@@ -941,8 +1065,7 @@ static void kvm_apic_disabled_lapic_found(struct kvm *kvm) · 570 unmodified lines
9411065 {
9421066 if (!kvm->arch.disabled_lapic_found) {
9431067 kvm->arch.disabled_lapic_found = true;
944− printk(KERN_INFO
945− "Disabled LAPIC found during irq injection\n");
1068+ pr_info("Disabled LAPIC found during irq injection\n");
9461069 }
9471070 }
9481071
@@ -951,7 +1074,7 @@ static bool kvm_apic_is_broadcast_dest(struct kvm *kvm, struct kvm_lapic **src, · 2 unmodified lines
9511074 {
9521075 if (kvm->arch.x2apic_broadcast_quirk_disabled) {
9531076 if ((irq->dest_id == APIC_BROADCAST &&
954− map->mode != KVM_APIC_MODE_X2APIC))
1077+ map->logical_mode != KVM_APIC_MODE_X2APIC))
9551078 return true;
9561079 if (irq->dest_id == X2APIC_BROADCAST)
9571080 return true;
@@ -1364,7 +1487,6 @@ static u32 apic_get_tmcct(struct kvm_lapic *apic) · 406 unmodified lines
13641487 {
13651488 ktime_t remaining, now;
13661489 s64 ns;
1367− u32 tmcct;
13681490
13691491 ASSERT(apic != NULL);
13701492
@@ -1379,10 +1501,7 @@ static u32 apic_get_tmcct(struct kvm_lapic *apic) · 8 unmodified lines
13791501 remaining = 0;
13801502
13811503 ns = mod_64(ktime_to_ns(remaining), apic->lapic_timer.period);
1382− tmcct = div64_u64(ns,
1383− (APIC_BUS_CYCLE_NS * apic->divide_count));
1384−
1385− return tmcct;
1504+ return div64_u64(ns, (APIC_BUS_CYCLE_NS * apic->divide_count));
13861505 }
13871506
13881507 static void __report_tpr_access(struct kvm_lapic *apic, bool write)
@@ -1442,19 +1561,15 @@ static inline struct kvm_lapic *to_lapic(struct kvm_io_device *dev) · 53 unmodified lines
14421561 #define APIC_REGS_MASK(first, count) \
14431562 (APIC_REG_MASK(first) * ((1ull << (count)) - 1))
14441563
1445−static int kvm_lapic_reg_read(struct kvm_lapic *apic, u32 offset, int len,
1446− void *data)
1564+u64 kvm_lapic_readable_reg_mask(struct kvm_lapic *apic)
14471565 {
1448− unsigned char alignment = offset & 0xf;
1449− u32 result;
1450− /* this bitmask has a bit cleared for each reserved register */
1566+ /* Leave bits '0' for reserved and write-only registers. */
14511567 u64 valid_reg_mask =
14521568 APIC_REG_MASK(APIC_ID) |
14531569 APIC_REG_MASK(APIC_LVR) |
14541570 APIC_REG_MASK(APIC_TASKPRI) |
14551571 APIC_REG_MASK(APIC_PROCPRI) |
14561572 APIC_REG_MASK(APIC_LDR) |
1457− APIC_REG_MASK(APIC_DFR) |
14581573 APIC_REG_MASK(APIC_SPIV) |
14591574 APIC_REGS_MASK(APIC_ISR, APIC_ISR_NR) |
14601575 APIC_REGS_MASK(APIC_TMR, APIC_ISR_NR) |
@@ -1474,21 +1589,33 @@ static int kvm_lapic_reg_read(struct kvm_lapic *apic, u32 offset, int len, · 13 unmodified lines
14741589 if (kvm_lapic_lvt_supported(apic, LVT_CMCI))
14751590 valid_reg_mask |= APIC_REG_MASK(APIC_LVTCMCI);
14761591
1477− /*
1478− * ARBPRI and ICR2 are not valid in x2APIC mode. WARN if KVM reads ICR
1479− * in x2APIC mode as it's an 8-byte register in x2APIC and needs to be
1480− * manually handled by the caller.
1481− */
1592+ /* ARBPRI, DFR, and ICR2 are not valid in x2APIC mode. */
14821593 if (!apic_x2apic_mode(apic))
14831594 valid_reg_mask |= APIC_REG_MASK(APIC_ARBPRI) |
1595+ APIC_REG_MASK(APIC_DFR) |
14841596 APIC_REG_MASK(APIC_ICR2);
1485− else
1486− WARN_ON_ONCE(offset == APIC_ICR);
1597+
1598+ return valid_reg_mask;
1599+}
1600+EXPORT_SYMBOL_GPL(kvm_lapic_readable_reg_mask);
1601+
1602+static int kvm_lapic_reg_read(struct kvm_lapic *apic, u32 offset, int len,
1603+ void *data)
1604+{
1605+ unsigned char alignment = offset & 0xf;
1606+ u32 result;
1607+
1608+ /*
1609+ * WARN if KVM reads ICR in x2APIC mode, as it's an 8-byte register in
1610+ * x2APIC and needs to be manually handled by the caller.
1611+ */
1612+ WARN_ON_ONCE(apic_x2apic_mode(apic) && offset == APIC_ICR);
14871613
14881614 if (alignment + len > 4)
14891615 return 1;
14901616
1491− if (offset > 0x3f0 || !(valid_reg_mask & APIC_REG_MASK(offset)))
1617+ if (offset > 0x3f0 ||
1618+ !(kvm_lapic_readable_reg_mask(apic) & APIC_REG_MASK(offset)))
14921619 return 1;
14931620
14941621 result = __apic_read(apic, offset & ~0xf);
@@ -1560,7 +1687,7 @@ static void limit_periodic_timer_frequency(struct kvm_lapic *apic) · 65 unmodified lines
15601687
15611688 if (apic->lapic_timer.period < min_period) {
15621689 pr_info_ratelimited(
1563− "kvm: vcpu %i: requested %lld ns "
1690+ "vcpu %i: requested %lld ns "
15641691 "lapic timer period limited to %lld ns\n",
15651692 apic->vcpu->vcpu_id,
15661693 apic->lapic_timer.period, min_period);
@@ -1841,11 +1968,15 @@ static bool set_target_expiration(struct kvm_lapic *apic, u32 count_reg) · 274 unmodified lines
18411968 if (unlikely(count_reg != APIC_TMICT)) {
18421969 deadline = tmict_to_ns(apic,
18431970 kvm_lapic_get_reg(apic, count_reg));
1844− if (unlikely(deadline <= 0))
1845− deadline = apic->lapic_timer.period;
1971+ if (unlikely(deadline <= 0)) {
1972+ if (apic_lvtt_period(apic))
1973+ deadline = apic->lapic_timer.period;
1974+ else
1975+ deadline = 0;
1976+ }
18461977 else if (unlikely(deadline > apic->lapic_timer.period)) {
18471978 pr_info_ratelimited(
1848− "kvm: vcpu %i: requested lapic timer restore with "
1979+ "vcpu %i: requested lapic timer restore with "
18491980 "starting count register %#x=%u (%lld ns) > initial count (%lld ns). "
18501981 "Using initial count to start timer.\n",
18511982 apic->vcpu->vcpu_id,
@@ -2068,19 +2199,6 @@ static void apic_manage_nmi_watchdog(struct kvm_lapic *apic, u32 lvt0_val) · 216 unmodified lines
20682199 }
20692200 }
20702201
2071−static void kvm_lapic_xapic_id_updated(struct kvm_lapic *apic)
2072−{
2073− struct kvm *kvm = apic->vcpu->kvm;
2074−
2075− if (KVM_BUG_ON(apic_x2apic_mode(apic), kvm))
2076− return;
2077−
2078− if (kvm_xapic_id(apic) == apic->vcpu->vcpu_id)
2079− return;
2080−
2081− kvm_set_apicv_inhibit(apic->vcpu->kvm, APICV_INHIBIT_REASON_APIC_ID_MODIFIED);
2082−}
2083−
20842202 static int get_lvt_index(u32 reg)
20852203 {
20862204 if (reg == APIC_LVTCMCI)
@@ -2101,7 +2219,6 @@ static int kvm_lapic_reg_write(struct kvm_lapic *apic, u32 reg, u32 val) · 14 unmodified lines
21012219 case APIC_ID: /* Local APIC ID */
21022220 if (!apic_x2apic_mode(apic)) {
21032221 kvm_apic_set_xapic_id(apic, val >> 24);
2104− kvm_lapic_xapic_id_updated(apic);
21052222 } else {
21062223 ret = 1;
21072224 }
@@ -2219,10 +2336,14 @@ static int kvm_lapic_reg_write(struct kvm_lapic *apic, u32 reg, u32 val) · 111 unmodified lines
22192336 break;
22202337
22212338 case APIC_SELF_IPI:
2222− if (apic_x2apic_mode(apic))
2223− kvm_apic_send_ipi(apic, APIC_DEST_SELF | (val & APIC_VECTOR_MASK), 0);
2224− else
2339+ /*
2340+ * Self-IPI exists only when x2APIC is enabled. Bits 7:0 hold
2341+ * the vector, everything else is reserved.
2342+ */
2343+ if (!apic_x2apic_mode(apic) || (val & ~APIC_VECTOR_MASK))
22252344 ret = 1;
2345+ else
2346+ kvm_apic_send_ipi(apic, APIC_DEST_SELF | val, 0);
22262347 break;
22272348 default:
22282349 ret = 1;
@@ -2284,23 +2405,18 @@ void kvm_apic_write_nodecode(struct kvm_vcpu *vcpu, u32 offset) · 55 unmodified lines
22842405 struct kvm_lapic *apic = vcpu->arch.apic;
22852406 u64 val;
22862407
2287− if (apic_x2apic_mode(apic)) {
2288− if (KVM_BUG_ON(kvm_lapic_msr_read(apic, offset, &val), vcpu->kvm))
2289− return;
2290− } else {
2291− val = kvm_lapic_get_reg(apic, offset);
2292− }
2293−
22942408 /*
22952409 * ICR is a single 64-bit register when x2APIC is enabled. For legacy
22962410 * xAPIC, ICR writes need to go down the common (slightly slower) path
22972411 * to get the upper half from ICR2.
22982412 */
22992413 if (apic_x2apic_mode(apic) && offset == APIC_ICR) {
2414+ val = kvm_lapic_get_reg64(apic, APIC_ICR);
23002415 kvm_apic_send_ipi(apic, (u32)val, (u32)(val >> 32));
23012416 trace_kvm_apic_write(APIC_ICR, val);
23022417 } else {
23032418 /* TODO: optimize to just emulate side effect w/o one more write */
2419+ val = kvm_lapic_get_reg(apic, offset);
23042420 kvm_lapic_reg_write(apic, offset, (u32)val);
23052421 }
23062422 }
@@ -2394,11 +2510,15 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64 value) · 87 unmodified lines
23942510 }
23952511 }
23962512
2397− if (((old_value ^ value) & X2APIC_ENABLE) && (value & X2APIC_ENABLE))
2398− kvm_apic_set_x2apic_id(apic, vcpu->vcpu_id);
2513+ if ((old_value ^ value) & X2APIC_ENABLE) {
2514+ if (value & X2APIC_ENABLE)
2515+ kvm_apic_set_x2apic_id(apic, vcpu->vcpu_id);
2516+ else if (value & MSR_IA32_APICBASE_ENABLE)
2517+ kvm_apic_set_xapic_id(apic, vcpu->vcpu_id);
2518+ }
23992519
24002520 if ((old_value ^ value) & (MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE)) {
2401− kvm_vcpu_update_apicv(vcpu);
2521+ kvm_make_request(KVM_REQ_APICV_UPDATE, vcpu);
24022522 static_call_cond(kvm_x86_set_virtual_apic_mode)(vcpu);
24032523 }
24042524
@@ -2429,6 +2549,78 @@ void kvm_apic_update_apicv(struct kvm_vcpu *vcpu) · 24 unmodified lines
24292549 */
24302550 apic->isr_count = count_vectors(apic->regs + APIC_ISR);
24312551 }
2552+ apic->highest_isr_cache = -1;
2553+}
2554+
2555+int kvm_alloc_apic_access_page(struct kvm *kvm)
2556+{
2557+ struct page *page;
2558+ void __user *hva;
2559+ int ret = 0;
2560+
2561+ mutex_lock(&kvm->slots_lock);
2562+ if (kvm->arch.apic_access_memslot_enabled ||
2563+ kvm->arch.apic_access_memslot_inhibited)
2564+ goto out;
2565+
2566+ hva = __x86_set_memory_region(kvm, APIC_ACCESS_PAGE_PRIVATE_MEMSLOT,
2567+ APIC_DEFAULT_PHYS_BASE, PAGE_SIZE);
2568+ if (IS_ERR(hva)) {
2569+ ret = PTR_ERR(hva);
2570+ goto out;
2571+ }
2572+
2573+ page = gfn_to_page(kvm, APIC_DEFAULT_PHYS_BASE >> PAGE_SHIFT);
2574+ if (is_error_page(page)) {
2575+ ret = -EFAULT;
2576+ goto out;
2577+ }
2578+
2579+ /*
2580+ * Do not pin the page in memory, so that memory hot-unplug
2581+ * is able to migrate it.
2582+ */
2583+ put_page(page);
2584+ kvm->arch.apic_access_memslot_enabled = true;
2585+out:
2586+ mutex_unlock(&kvm->slots_lock);
2587+ return ret;
2588+}
2589+EXPORT_SYMBOL_GPL(kvm_alloc_apic_access_page);
2590+
2591+void kvm_inhibit_apic_access_page(struct kvm_vcpu *vcpu)
2592+{
2593+ struct kvm *kvm = vcpu->kvm;
2594+
2595+ if (!kvm->arch.apic_access_memslot_enabled)
2596+ return;
2597+
2598+ kvm_vcpu_srcu_read_unlock(vcpu);
2599+
2600+ mutex_lock(&kvm->slots_lock);
2601+
2602+ if (kvm->arch.apic_access_memslot_enabled) {
2603+ __x86_set_memory_region(kvm, APIC_ACCESS_PAGE_PRIVATE_MEMSLOT, 0, 0);
2604+ /*
2605+ * Clear "enabled" after the memslot is deleted so that a
2606+ * different vCPU doesn't get a false negative when checking
2607+ * the flag out of slots_lock. No additional memory barrier is
2608+ * needed as modifying memslots requires waiting other vCPUs to
2609+ * drop SRCU (see above), and false positives are ok as the
2610+ * flag is rechecked after acquiring slots_lock.
2611+ */
2612+ kvm->arch.apic_access_memslot_enabled = false;
2613+
2614+ /*
2615+ * Mark the memslot as inhibited to prevent reallocating the
2616+ * memslot during vCPU creation, e.g. if a vCPU is hotplugged.
2617+ */
2618+ kvm->arch.apic_access_memslot_inhibited = true;
2619+ }
2620+
2621+ mutex_unlock(&kvm->slots_lock);
2622+
2623+ kvm_vcpu_srcu_read_lock(vcpu);
24322624 }
24332625
24342626 void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event)
@@ -2484,7 +2676,6 @@ void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event) · 49 unmodified lines
24842676 kvm_lapic_set_reg(apic, APIC_TMR + 0x10 * i, 0);
24852677 }
24862678 kvm_apic_update_apicv(vcpu);
2487− apic->highest_isr_cache = -1;
24882679 update_divide_count(apic);
24892680 atomic_set(&apic->lapic_timer.pending, 0);
24902681
@@ -2756,9 +2947,6 @@ int kvm_apic_set_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s) · 265 unmodified lines
27562947 }
27572948 memcpy(vcpu->arch.apic->regs, s->regs, sizeof(*s));
27582949
2759− if (!apic_x2apic_mode(apic))
2760− kvm_lapic_xapic_id_updated(apic);
2761−
27622950 atomic_set_release(&apic->vcpu->kvm->arch.apic_map_dirty, DIRTY);
27632951 kvm_recalculate_apic_map(vcpu->kvm);
27642952 kvm_apic_set_version(vcpu);
@@ -2772,7 +2960,6 @@ int kvm_apic_set_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s) · 7 unmodified lines
27722960 __start_apic_timer(apic, APIC_TMCCT);
27732961 kvm_lapic_set_reg(apic, APIC_TMCCT, 0);
27742962 kvm_apic_update_apicv(vcpu);
2775− apic->highest_isr_cache = -1;
27762963 if (apic->apicv_active) {
27772964 static_call_cond(kvm_x86_apicv_post_state_restore)(vcpu);
27782965 static_call_cond(kvm_x86_hwapic_irr_update)(vcpu, apic_find_highest_irr(apic));
@@ -2943,13 +3130,17 @@ static int kvm_lapic_msr_read(struct kvm_lapic *apic, u32 reg, u64 *data) · 164 unmodified lines
29433130 static int kvm_lapic_msr_write(struct kvm_lapic *apic, u32 reg, u64 data)
29443131 {
29453132 /*
2946− * ICR is a 64-bit register in x2APIC mode (and Hyper'v PV vAPIC) and
3133+ * ICR is a 64-bit register in x2APIC mode (and Hyper-V PV vAPIC) and
29473134 * can be written as such, all other registers remain accessible only
29483135 * through 32-bit reads/writes.
29493136 */
29503137 if (reg == APIC_ICR)
29513138 return kvm_x2apic_icr_write(apic, data);
29523139
3140+ /* Bits 63:32 are reserved in all other registers. */
3141+ if (data >> 32)
3142+ return 1;
3143+
29533144 return kvm_lapic_reg_write(apic, reg, (u32)data);
29543145 }
29553146
@@ -2972,9 +3163,6 @@ int kvm_x2apic_msr_read(struct kvm_vcpu *vcpu, u32 msr, u64 *data) · 16 unmodified lines
29723163 if (!lapic_in_kernel(vcpu) || !apic_x2apic_mode(apic))
29733164 return 1;
29743165
2975− if (reg == APIC_DFR)
2976− return 1;
2977−
29783166 return kvm_lapic_msr_read(apic, reg, data);
29793167 }
29803168
arch/x86/kvm/lapic.h+4−0
@@ -112,6 +112,8 @@ int kvm_apic_set_irq(struct kvm_vcpu *vcpu, struct kvm_lapic_irq *irq, · 111 unmodified lines
112112 struct dest_map *dest_map);
113113 int kvm_apic_local_deliver(struct kvm_lapic *apic, int lvt_type);
114114 void kvm_apic_update_apicv(struct kvm_vcpu *vcpu);
115+int kvm_alloc_apic_access_page(struct kvm *kvm);
116+void kvm_inhibit_apic_access_page(struct kvm_vcpu *vcpu);
115117
116118 bool kvm_irq_delivery_to_apic_fast(struct kvm *kvm, struct kvm_lapic *src,
117119 struct kvm_lapic_irq *irq, int *r, struct dest_map *dest_map);
@@ -144,6 +146,8 @@ int kvm_hv_vapic_msr_read(struct kvm_vcpu *vcpu, u32 msr, u64 *data); · 26 unmodified lines
144146 int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len);
145147 void kvm_lapic_exit(void);
146148
149+u64 kvm_lapic_readable_reg_mask(struct kvm_lapic *apic);
150+
147151 #define VEC_POS(v) ((v) & (32 - 1))
148152 #define REG_POS(v) (((v) >> 5) << 4)
149153
arch/x86/kvm/mmu.h+3−3
@@ -230,14 +230,14 @@ static inline bool kvm_shadow_root_allocated(struct kvm *kvm) · 229 unmodified lines
230230 }
231231
232232 #ifdef CONFIG_X86_64
233−static inline bool is_tdp_mmu_enabled(struct kvm *kvm) { return kvm->arch.tdp_mmu_enabled; }
233+extern bool tdp_mmu_enabled;
234234 #else
235−static inline bool is_tdp_mmu_enabled(struct kvm *kvm) { return false; }
235+#define tdp_mmu_enabled false
236236 #endif
237237
238238 static inline bool kvm_memslots_have_rmaps(struct kvm *kvm)
239239 {
240− return !is_tdp_mmu_enabled(kvm) || kvm_shadow_root_allocated(kvm);
240+ return !tdp_mmu_enabled || kvm_shadow_root_allocated(kvm);
241241 }
242242
243243 static inline gfn_t gfn_to_index(gfn_t gfn, gfn_t base_gfn, int level)
arch/x86/kvm/mmu/mmu.c+203−117
@@ -14,6 +14,7 @@ · 13 unmodified lines
1414 * Yaniv Kamay <yaniv@qumranet.com>
1515 * Avi Kivity <avi@qumranet.com>
1616 */
17+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
1718
1819 #include "irq.h"
1920 #include "ioapic.h"
@@ -43,6 +44,7 @@ · 23 unmodified lines
4344 #include <linux/uaccess.h>
4445 #include <linux/hash.h>
4546 #include <linux/kern_levels.h>
47+#include <linux/kstrtox.h>
4648 #include <linux/kthread.h>
4749
4850 #include <asm/page.h>
@@ -99,6 +101,13 @@ module_param_named(flush_on_reuse, force_flush_and_sync_on_reuse, bool, 0644); · 50 unmodified lines
99101 */
100102 bool tdp_enabled = false;
101103
104+static bool __ro_after_init tdp_mmu_allowed;
105+
106+#ifdef CONFIG_X86_64
107+bool __read_mostly tdp_mmu_enabled = true;
108+module_param_named(tdp_mmu, tdp_mmu_enabled, bool, 0444);
109+#endif
110+
102111 static int max_huge_page_level __read_mostly;
103112 static int tdp_root_level __read_mostly;
104113 static int max_tdp_level __read_mostly;
@@ -261,6 +270,17 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, · 156 unmodified lines
261270 kvm_flush_remote_tlbs_with_range(kvm, &range);
262271 }
263272
273+static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index);
274+
275+/* Flush the range of guest memory mapped by the given SPTE. */
276+static void kvm_flush_remote_tlbs_sptep(struct kvm *kvm, u64 *sptep)
277+{
278+ struct kvm_mmu_page *sp = sptep_to_sp(sptep);
279+ gfn_t gfn = kvm_mmu_page_get_gfn(sp, spte_index(sptep));
280+
281+ kvm_flush_remote_tlbs_gfn(kvm, gfn, sp->role.level);
282+}
283+
264284 static void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn,
265285 unsigned int access)
266286 {
@@ -609,9 +629,14 @@ static bool mmu_spte_age(u64 *sptep) · 342 unmodified lines
609629 return true;
610630 }
611631
632+static inline bool is_tdp_mmu_active(struct kvm_vcpu *vcpu)
633+{
634+ return tdp_mmu_enabled && vcpu->arch.mmu->root_role.direct;
635+}
636+
612637 static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu)
613638 {
614− if (is_tdp_mmu(vcpu->arch.mmu)) {
639+ if (is_tdp_mmu_active(vcpu)) {
615640 kvm_tdp_mmu_walk_lockless_begin();
616641 } else {
617642 /*
@@ -630,7 +655,7 @@ static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu) · 12 unmodified lines
630655
631656 static void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu)
632657 {
633− if (is_tdp_mmu(vcpu->arch.mmu)) {
658+ if (is_tdp_mmu_active(vcpu)) {
634659 kvm_tdp_mmu_walk_lockless_end();
635660 } else {
636661 /*
@@ -800,7 +825,7 @@ static void account_shadowed(struct kvm *kvm, struct kvm_mmu_page *sp) · 163 unmodified lines
800825 kvm_mmu_gfn_disallow_lpage(slot, gfn);
801826
802827 if (kvm_mmu_slot_gfn_write_protect(kvm, slot, gfn, PG_LEVEL_4K))
803− kvm_flush_remote_tlbs_with_address(kvm, gfn, 1);
828+ kvm_flush_remote_tlbs_gfn(kvm, gfn, PG_LEVEL_4K);
804829 }
805830
806831 void track_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp)
@@ -1174,8 +1199,7 @@ static void drop_large_spte(struct kvm *kvm, u64 *sptep, bool flush) · 367 unmodified lines
11741199 drop_spte(kvm, sptep);
11751200
11761201 if (flush)
1177− kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
1178− KVM_PAGES_PER_HPAGE(sp->role.level));
1202+ kvm_flush_remote_tlbs_sptep(kvm, sptep);
11791203 }
11801204
11811205 /*
@@ -1279,7 +1303,7 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, · 97 unmodified lines
12791303 {
12801304 struct kvm_rmap_head *rmap_head;
12811305
1282− if (is_tdp_mmu_enabled(kvm))
1306+ if (tdp_mmu_enabled)
12831307 kvm_tdp_mmu_clear_dirty_pt_masked(kvm, slot,
12841308 slot->base_gfn + gfn_offset, mask, true);
12851309
@@ -1312,7 +1336,7 @@ static void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm, · 26 unmodified lines
13121336 {
13131337 struct kvm_rmap_head *rmap_head;
13141338
1315− if (is_tdp_mmu_enabled(kvm))
1339+ if (tdp_mmu_enabled)
13161340 kvm_tdp_mmu_clear_dirty_pt_masked(kvm, slot,
13171341 slot->base_gfn + gfn_offset, mask, false);
13181342
@@ -1395,7 +1419,7 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, · 76 unmodified lines
13951419 }
13961420 }
13971421
1398− if (is_tdp_mmu_enabled(kvm))
1422+ if (tdp_mmu_enabled)
13991423 write_protected |=
14001424 kvm_tdp_mmu_write_protect_gfn(kvm, slot, gfn, min_level);
14011425
@@ -1456,7 +1480,7 @@ restart: · 54 unmodified lines
14561480 }
14571481
14581482 if (need_flush && kvm_available_flush_tlb_with_range()) {
1459− kvm_flush_remote_tlbs_with_address(kvm, gfn, 1);
1483+ kvm_flush_remote_tlbs_gfn(kvm, gfn, level);
14601484 return false;
14611485 }
14621486
@@ -1558,7 +1582,7 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) · 95 unmodified lines
15581582 if (kvm_memslots_have_rmaps(kvm))
15591583 flush = kvm_handle_gfn_range(kvm, range, kvm_zap_rmap);
15601584
1561− if (is_tdp_mmu_enabled(kvm))
1585+ if (tdp_mmu_enabled)
15621586 flush = kvm_tdp_mmu_unmap_gfn_range(kvm, range, flush);
15631587
15641588 return flush;
@@ -1571,7 +1595,7 @@ bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · 6 unmodified lines
15711595 if (kvm_memslots_have_rmaps(kvm))
15721596 flush = kvm_handle_gfn_range(kvm, range, kvm_set_pte_rmap);
15731597
1574− if (is_tdp_mmu_enabled(kvm))
1598+ if (tdp_mmu_enabled)
15751599 flush |= kvm_tdp_mmu_set_spte_gfn(kvm, range);
15761600
15771601 return flush;
@@ -1626,8 +1650,7 @@ static void __rmap_add(struct kvm *kvm, · 48 unmodified lines
16261650 kvm->stat.max_mmu_rmap_size = rmap_count;
16271651 if (rmap_count > RMAP_RECYCLE_THRESHOLD) {
16281652 kvm_zap_all_rmap_sptes(kvm, rmap_head);
1629− kvm_flush_remote_tlbs_with_address(
1630− kvm, sp->gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
1653+ kvm_flush_remote_tlbs_gfn(kvm, gfn, sp->role.level);
16311654 }
16321655 }
16331656
@@ -1646,7 +1669,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · 12 unmodified lines
16461669 if (kvm_memslots_have_rmaps(kvm))
16471670 young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
16481671
1649− if (is_tdp_mmu_enabled(kvm))
1672+ if (tdp_mmu_enabled)
16501673 young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
16511674
16521675 return young;
@@ -1659,7 +1682,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · 6 unmodified lines
16591682 if (kvm_memslots_have_rmaps(kvm))
16601683 young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap);
16611684
1662− if (is_tdp_mmu_enabled(kvm))
1685+ if (tdp_mmu_enabled)
16631686 young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
16641687
16651688 return young;
@@ -1921,7 +1944,7 @@ static bool is_obsolete_sp(struct kvm *kvm, struct kvm_mmu_page *sp) · 255 unmodified lines
19211944 return true;
19221945
19231946 /* TDP MMU pages do not use the MMU generation. */
1924− return !sp->tdp_mmu_page &&
1947+ return !is_tdp_mmu_page(sp) &&
19251948 unlikely(sp->mmu_valid_gen != kvm->arch.mmu_valid_gen);
19261949 }
19271950
@@ -2355,7 +2378,16 @@ static void __link_shadow_page(struct kvm *kvm, · 427 unmodified lines
23552378
23562379 mmu_page_add_parent_pte(cache, sp, sptep);
23572380
2358− if (sp->unsync_children || sp->unsync)
2381+ /*
2382+ * The non-direct sub-pagetable must be updated before linking. For
2383+ * L1 sp, the pagetable is updated via kvm_sync_page() in
2384+ * kvm_mmu_find_shadow_page() without write-protecting the gfn,
2385+ * so sp->unsync can be true or false. For higher level non-direct
2386+ * sp, the pagetable is updated/synced via mmu_sync_children() in
2387+ * FNAME(fetch)(), so sp->unsync_children can only be false.
2388+ * WARN_ON_ONCE() if anything happens unexpectedly.
2389+ */
2390+ if (WARN_ON_ONCE(sp->unsync_children) || sp->unsync)
23592391 mark_unsync(sptep);
23602392 }
23612393
@@ -2383,7 +2415,7 @@ static void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep, · 21 unmodified lines
23832415 return;
23842416
23852417 drop_parent_pte(child, sptep);
2386− kvm_flush_remote_tlbs_with_address(vcpu->kvm, child->gfn, 1);
2418+ kvm_flush_remote_tlbs_sptep(vcpu->kvm, sptep);
23872419 }
23882420 }
23892421
@@ -2867,8 +2899,7 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, · 477 unmodified lines
28672899 }
28682900
28692901 if (flush)
2870− kvm_flush_remote_tlbs_with_address(vcpu->kvm, gfn,
2871− KVM_PAGES_PER_HPAGE(level));
2902+ kvm_flush_remote_tlbs_gfn(vcpu->kvm, gfn, level);
28722903
28732904 pgprintk("%s: setting spte %llx\n", __func__, *sptep);
28742905
@@ -3116,11 +3147,11 @@ void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_ · 241 unmodified lines
31163147 !is_large_pte(spte) &&
31173148 spte_to_child_sp(spte)->nx_huge_page_disallowed) {
31183149 /*
3119− * A small SPTE exists for this pfn, but FNAME(fetch)
3120− * and __direct_map would like to create a large PTE
3121− * instead: just force them to go down another level,
3122− * patching back for them into pfn the next 9 bits of
3123− * the address.
3150+ * A small SPTE exists for this pfn, but FNAME(fetch),
3151+ * direct_map(), or kvm_tdp_mmu_map() would like to create a
3152+ * large PTE instead: just force them to go down another level,
3153+ * patching back for them into pfn the next 9 bits of the
3154+ * address.
31243155 */
31253156 u64 page_mask = KVM_PAGES_PER_HPAGE(cur_level) -
31263157 KVM_PAGES_PER_HPAGE(cur_level - 1);
@@ -3129,7 +3160,7 @@ void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_ · 2 unmodified lines
31293160 }
31303161 }
31313162
3132−static int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
3163+static int direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
31333164 {
31343165 struct kvm_shadow_walk_iterator it;
31353166 struct kvm_mmu_page *sp;
@@ -3147,7 +3178,7 @@ static int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 11 unmodified lines
31473178 if (fault->nx_huge_page_workaround_enabled)
31483179 disallowed_hugepage_adjust(fault, *it.sptep, it.level);
31493180
3150− base_gfn = fault->gfn & ~(KVM_PAGES_PER_HPAGE(it.level) - 1);
3181+ base_gfn = gfn_round_for_level(fault->gfn, it.level);
31513182 if (it.level == fault->goal_level)
31523183 break;
31533184
@@ -3173,14 +3204,16 @@ static int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 19 unmodified lines
31733204 return ret;
31743205 }
31753206
3176−static void kvm_send_hwpoison_signal(unsigned long address, struct task_struct *tsk)
3207+static void kvm_send_hwpoison_signal(struct kvm_memory_slot *slot, gfn_t gfn)
31773208 {
3178− send_sig_mceerr(BUS_MCEERR_AR, (void __user *)address, PAGE_SHIFT, tsk);
3209+ unsigned long hva = gfn_to_hva_memslot(slot, gfn);
3210+
3211+ send_sig_mceerr(BUS_MCEERR_AR, (void __user *)hva, PAGE_SHIFT, current);
31793212 }
31803213
3181−static int kvm_handle_error_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)
3214+static int kvm_handle_error_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
31823215 {
3183− if (is_sigpending_pfn(pfn)) {
3216+ if (is_sigpending_pfn(fault->pfn)) {
31843217 kvm_handle_signal_exit(vcpu);
31853218 return -EINTR;
31863219 }
@@ -3190,43 +3223,43 @@ static int kvm_handle_error_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · 3 unmodified lines
31903223 * into the spte otherwise read access on readonly gfn also can
31913224 * caused mmio page fault and treat it as mmio access.
31923225 */
3193− if (pfn == KVM_PFN_ERR_RO_FAULT)
3226+ if (fault->pfn == KVM_PFN_ERR_RO_FAULT)
31943227 return RET_PF_EMULATE;
31953228
3196− if (pfn == KVM_PFN_ERR_HWPOISON) {
3197− kvm_send_hwpoison_signal(kvm_vcpu_gfn_to_hva(vcpu, gfn), current);
3229+ if (fault->pfn == KVM_PFN_ERR_HWPOISON) {
3230+ kvm_send_hwpoison_signal(fault->slot, fault->gfn);
31983231 return RET_PF_RETRY;
31993232 }
32003233
32013234 return -EFAULT;
32023235 }
32033236
3204−static int handle_abnormal_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,
3205− unsigned int access)
3237+static int kvm_handle_noslot_fault(struct kvm_vcpu *vcpu,
3238+ struct kvm_page_fault *fault,
3239+ unsigned int access)
32063240 {
3207− /* The pfn is invalid, report the error! */
3208− if (unlikely(is_error_pfn(fault->pfn)))
3209− return kvm_handle_error_pfn(vcpu, fault->gfn, fault->pfn);
3241+ gva_t gva = fault->is_tdp ? 0 : fault->addr;
32103242
3211− if (unlikely(!fault->slot)) {
3212− gva_t gva = fault->is_tdp ? 0 : fault->addr;
3243+ vcpu_cache_mmio_info(vcpu, gva, fault->gfn,
3244+ access & shadow_mmio_access_mask);
32133245
3214− vcpu_cache_mmio_info(vcpu, gva, fault->gfn,
3215− access & shadow_mmio_access_mask);
3216− /*
3217− * If MMIO caching is disabled, emulate immediately without
3218− * touching the shadow page tables as attempting to install an
3219− * MMIO SPTE will just be an expensive nop. Do not cache MMIO
3220− * whose gfn is greater than host.MAXPHYADDR, any guest that
3221− * generates such gfns is running nested and is being tricked
3222− * by L0 userspace (you can observe gfn > L1.MAXPHYADDR if
3223− * and only if L1's MAXPHYADDR is inaccurate with respect to
3224− * the hardware's).
3225− */
3226− if (unlikely(!enable_mmio_caching) ||
3227− unlikely(fault->gfn > kvm_mmu_max_gfn()))
3228− return RET_PF_EMULATE;
3229− }
3246+ /*
3247+ * If MMIO caching is disabled, emulate immediately without
3248+ * touching the shadow page tables as attempting to install an
3249+ * MMIO SPTE will just be an expensive nop.
3250+ */
3251+ if (unlikely(!enable_mmio_caching))
3252+ return RET_PF_EMULATE;
3253+
3254+ /*
3255+ * Do not create an MMIO SPTE for a gfn greater than host.MAXPHYADDR,
3256+ * any guest that generates such gfns is running nested and is being
3257+ * tricked by L0 userspace (you can observe gfn > L1.MAXPHYADDR if and
3258+ * only if L1's MAXPHYADDR is inaccurate with respect to the
3259+ * hardware's).
3260+ */
3261+ if (unlikely(fault->gfn > kvm_mmu_max_gfn()))
3262+ return RET_PF_EMULATE;
32303263
32313264 return RET_PF_CONTINUE;
32323265 }
@@ -3350,7 +3383,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 117 unmodified lines
33503383 do {
33513384 u64 new_spte;
33523385
3353− if (is_tdp_mmu(vcpu->arch.mmu))
3386+ if (tdp_mmu_enabled)
33543387 sptep = kvm_tdp_mmu_fast_pf_get_last_sptep(vcpu, fault->addr, &spte);
33553388 else
33563389 sptep = fast_pf_get_last_sptep(vcpu, fault->addr, &spte);
@@ -3433,8 +3466,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 76 unmodified lines
34333466 }
34343467
34353468 if (++retry_count > 4) {
3436− printk_once(KERN_WARNING
3437− "kvm: Fast #PF retrying more than 4 times.\n");
3469+ pr_warn_once("Fast #PF retrying more than 4 times.\n");
34383470 break;
34393471 }
34403472
@@ -3596,7 +3628,7 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) · 155 unmodified lines
35963628 if (r < 0)
35973629 goto out_unlock;
35983630
3599− if (is_tdp_mmu_enabled(vcpu->kvm)) {
3631+ if (tdp_mmu_enabled) {
36003632 root = kvm_tdp_mmu_get_vcpu_root_hpa(vcpu);
36013633 mmu->root.hpa = root;
36023634 } else if (shadow_root_level >= PT64_ROOT_4LEVEL) {
@@ -4026,7 +4058,7 @@ static bool get_mmio_spte(struct kvm_vcpu *vcpu, u64 addr, u64 *sptep) · 423 unmodified lines
40264058
40274059 walk_shadow_page_lockless_begin(vcpu);
40284060
4029− if (is_tdp_mmu(vcpu->arch.mmu))
4061+ if (is_tdp_mmu_active(vcpu))
40304062 leaf = kvm_tdp_mmu_get_walk(vcpu, addr, sptes, &root);
40314063 else
40324064 leaf = get_walk(vcpu, addr, sptes, &root);
@@ -4174,7 +4206,7 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) · 141 unmodified lines
41744206 kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true);
41754207 }
41764208
4177−static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
4209+static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
41784210 {
41794211 struct kvm_memory_slot *slot = fault->slot;
41804212 bool async;
@@ -4235,12 +4267,33 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 54 unmodified lines
42354267 return RET_PF_CONTINUE;
42364268 }
42374269
4270+static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,
4271+ unsigned int access)
4272+{
4273+ int ret;
4274+
4275+ fault->mmu_seq = vcpu->kvm->mmu_invalidate_seq;
4276+ smp_rmb();
4277+
4278+ ret = __kvm_faultin_pfn(vcpu, fault);
4279+ if (ret != RET_PF_CONTINUE)
4280+ return ret;
4281+
4282+ if (unlikely(is_error_pfn(fault->pfn)))
4283+ return kvm_handle_error_pfn(vcpu, fault);
4284+
4285+ if (unlikely(!fault->slot))
4286+ return kvm_handle_noslot_fault(vcpu, fault, access);
4287+
4288+ return RET_PF_CONTINUE;
4289+}
4290+
42384291 /*
42394292 * Returns true if the page fault is stale and needs to be retried, i.e. if the
42404293 * root was invalidated by a memslot update or a relevant mmu_notifier fired.
42414294 */
42424295 static bool is_page_fault_stale(struct kvm_vcpu *vcpu,
4243− struct kvm_page_fault *fault, int mmu_seq)
4296+ struct kvm_page_fault *fault)
42444297 {
42454298 struct kvm_mmu_page *sp = to_shadow_page(vcpu->arch.mmu->root.hpa);
42464299
@@ -4260,19 +4313,13 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu, · 13 unmodified lines
42604313 return true;
42614314
42624315 return fault->slot &&
4263− mmu_invalidate_retry_hva(vcpu->kvm, mmu_seq, fault->hva);
4316+ mmu_invalidate_retry_hva(vcpu->kvm, fault->mmu_seq, fault->hva);
42644317 }
42654318
42664319 static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
42674320 {
4268− bool is_tdp_mmu_fault = is_tdp_mmu(vcpu->arch.mmu);
4269−
4270− unsigned long mmu_seq;
42714321 int r;
42724322
4273− fault->gfn = fault->addr >> PAGE_SHIFT;
4274− fault->slot = kvm_vcpu_gfn_to_memslot(vcpu, fault->gfn);
4275−
42764323 if (page_fault_handle_page_track(vcpu, fault))
42774324 return RET_PF_EMULATE;
42784325
@@ -4284,41 +4331,24 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault · 5 unmodified lines
42844331 if (r)
42854332 return r;
42864333
4287− mmu_seq = vcpu->kvm->mmu_invalidate_seq;
4288− smp_rmb();
4289−
4290− r = kvm_faultin_pfn(vcpu, fault);
4291− if (r != RET_PF_CONTINUE)
4292− return r;
4293−
4294− r = handle_abnormal_pfn(vcpu, fault, ACC_ALL);
4334+ r = kvm_faultin_pfn(vcpu, fault, ACC_ALL);
42954335 if (r != RET_PF_CONTINUE)
42964336 return r;
42974337
42984338 r = RET_PF_RETRY;
4339+ write_lock(&vcpu->kvm->mmu_lock);
42994340
4300− if (is_tdp_mmu_fault)
4301− read_lock(&vcpu->kvm->mmu_lock);
4302− else
4303− write_lock(&vcpu->kvm->mmu_lock);
4341+ if (is_page_fault_stale(vcpu, fault))
4342+ goto out_unlock;
43044343
4305− if (is_page_fault_stale(vcpu, fault, mmu_seq))
4344+ r = make_mmu_pages_available(vcpu);
4345+ if (r)
43064346 goto out_unlock;
43074347
4308− if (is_tdp_mmu_fault) {
4309− r = kvm_tdp_mmu_map(vcpu, fault);
4310− } else {
4311− r = make_mmu_pages_available(vcpu);
4312− if (r)
4313− goto out_unlock;
4314− r = __direct_map(vcpu, fault);
4315− }
4348+ r = direct_map(vcpu, fault);
43164349
43174350 out_unlock:
4318− if (is_tdp_mmu_fault)
4319− read_unlock(&vcpu->kvm->mmu_lock);
4320− else
4321− write_unlock(&vcpu->kvm->mmu_lock);
4351+ write_unlock(&vcpu->kvm->mmu_lock);
43224352 kvm_release_pfn_clean(fault->pfn);
43234353 return r;
43244354 }
@@ -4366,6 +4396,42 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, · 41 unmodified lines
43664396 }
43674397 EXPORT_SYMBOL_GPL(kvm_handle_page_fault);
43684398
4399+#ifdef CONFIG_X86_64
4400+static int kvm_tdp_mmu_page_fault(struct kvm_vcpu *vcpu,
4401+ struct kvm_page_fault *fault)
4402+{
4403+ int r;
4404+
4405+ if (page_fault_handle_page_track(vcpu, fault))
4406+ return RET_PF_EMULATE;
4407+
4408+ r = fast_page_fault(vcpu, fault);
4409+ if (r != RET_PF_INVALID)
4410+ return r;
4411+
4412+ r = mmu_topup_memory_caches(vcpu, false);
4413+ if (r)
4414+ return r;
4415+
4416+ r = kvm_faultin_pfn(vcpu, fault, ACC_ALL);
4417+ if (r != RET_PF_CONTINUE)
4418+ return r;
4419+
4420+ r = RET_PF_RETRY;
4421+ read_lock(&vcpu->kvm->mmu_lock);
4422+
4423+ if (is_page_fault_stale(vcpu, fault))
4424+ goto out_unlock;
4425+
4426+ r = kvm_tdp_mmu_map(vcpu, fault);
4427+
4428+out_unlock:
4429+ read_unlock(&vcpu->kvm->mmu_lock);
4430+ kvm_release_pfn_clean(fault->pfn);
4431+ return r;
4432+}
4433+#endif
4434+
43694435 int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
43704436 {
43714437 /*
@@ -4383,13 +4449,19 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 11 unmodified lines
43834449 if (shadow_memtype_mask && kvm_arch_has_noncoherent_dma(vcpu->kvm)) {
43844450 for ( ; fault->max_level > PG_LEVEL_4K; --fault->max_level) {
43854451 int page_num = KVM_PAGES_PER_HPAGE(fault->max_level);
4386− gfn_t base = (fault->addr >> PAGE_SHIFT) & ~(page_num - 1);
4452+ gfn_t base = gfn_round_for_level(fault->gfn,
4453+ fault->max_level);
43874454
43884455 if (kvm_mtrr_check_gfn_range_consistency(vcpu, base, page_num))
43894456 break;
43904457 }
43914458 }
43924459
4460+#ifdef CONFIG_X86_64
4461+ if (tdp_mmu_enabled)
4462+ return kvm_tdp_mmu_page_fault(vcpu, fault);
4463+#endif
4464+
43934465 return direct_page_fault(vcpu, fault);
43944466 }
43954467
@@ -4494,10 +4566,12 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) · 98 unmodified lines
44944566 struct kvm_mmu *mmu = vcpu->arch.mmu;
44954567 union kvm_mmu_page_role new_role = mmu->root_role;
44964568
4497− if (!fast_pgd_switch(vcpu->kvm, mmu, new_pgd, new_role)) {
4498− /* kvm_mmu_ensure_valid_pgd will set up a new root. */
4569+ /*
4570+ * Return immediately if no usable root was found, kvm_mmu_reload()
4571+ * will establish a valid root prior to the next VM-Enter.
4572+ */
4573+ if (!fast_pgd_switch(vcpu->kvm, mmu, new_pgd, new_role))
44994574 return;
4500− }
45014575
45024576 /*
45034577 * It's possible that the cached previous root page is obsolete because
@@ -5719,6 +5793,9 @@ void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level, · 1215 unmodified lines
57195793 tdp_root_level = tdp_forced_root_level;
57205794 max_tdp_level = tdp_max_root_level;
57215795
5796+#ifdef CONFIG_X86_64
5797+ tdp_mmu_enabled = tdp_mmu_allowed && tdp_enabled;
5798+#endif
57225799 /*
57235800 * max_huge_page_level reflects KVM's MMU capabilities irrespective
57245801 * of kernel support, e.g. KVM may be capable of using 1GB pages when
@@ -5966,7 +6043,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) · 241 unmodified lines
59666043 * write and in the same critical section as making the reload request,
59676044 * e.g. before kvm_zap_obsolete_pages() could drop mmu_lock and yield.
59686045 */
5969− if (is_tdp_mmu_enabled(kvm))
6046+ if (tdp_mmu_enabled)
59706047 kvm_tdp_mmu_invalidate_all_roots(kvm);
59716048
59726049 /*
@@ -5991,7 +6068,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) · 18 unmodified lines
59916068 * Deferring the zap until the final reference to the root is put would
59926069 * lead to use-after-free.
59936070 */
5994− if (is_tdp_mmu_enabled(kvm))
6071+ if (tdp_mmu_enabled)
59956072 kvm_tdp_mmu_zap_invalidated_roots(kvm);
59966073 }
59976074
@@ -6017,9 +6094,11 @@ int kvm_mmu_init_vm(struct kvm *kvm) · 19 unmodified lines
60176094 INIT_LIST_HEAD(&kvm->arch.possible_nx_huge_pages);
60186095 spin_lock_init(&kvm->arch.mmu_unsync_pages_lock);
60196096
6020− r = kvm_mmu_init_tdp_mmu(kvm);
6021− if (r < 0)
6022− return r;
6097+ if (tdp_mmu_enabled) {
6098+ r = kvm_mmu_init_tdp_mmu(kvm);
6099+ if (r < 0)
6100+ return r;
6101+ }
60236102
60246103 node->track_write = kvm_mmu_pte_write;
60256104 node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;
@@ -6049,7 +6128,8 @@ void kvm_mmu_uninit_vm(struct kvm *kvm) · 23 unmodified lines
60496128
60506129 kvm_page_track_unregister_notifier(kvm, node);
60516130
6052− kvm_mmu_uninit_tdp_mmu(kvm);
6131+ if (tdp_mmu_enabled)
6132+ kvm_mmu_uninit_tdp_mmu(kvm);
60536133
60546134 mmu_free_vm_memory_caches(kvm);
60556135 }
@@ -6103,7 +6183,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) · 47 unmodified lines
61036183
61046184 flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end);
61056185
6106− if (is_tdp_mmu_enabled(kvm)) {
6186+ if (tdp_mmu_enabled) {
61076187 for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++)
61086188 flush = kvm_tdp_mmu_zap_leafs(kvm, i, gfn_start,
61096189 gfn_end, true, flush);
@@ -6136,7 +6216,7 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, · 26 unmodified lines
61366216 write_unlock(&kvm->mmu_lock);
61376217 }
61386218
6139− if (is_tdp_mmu_enabled(kvm)) {
6219+ if (tdp_mmu_enabled) {
61406220 read_lock(&kvm->mmu_lock);
61416221 kvm_tdp_mmu_wrprot_slot(kvm, memslot, start_level);
61426222 read_unlock(&kvm->mmu_lock);
@@ -6379,7 +6459,7 @@ void kvm_mmu_try_split_huge_pages(struct kvm *kvm, · 236 unmodified lines
63796459 u64 start, u64 end,
63806460 int target_level)
63816461 {
6382− if (!is_tdp_mmu_enabled(kvm))
6462+ if (!tdp_mmu_enabled)
63836463 return;
63846464
63856465 if (kvm_memslots_have_rmaps(kvm))
@@ -6400,7 +6480,7 @@ void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, · 14 unmodified lines
64006480 u64 start = memslot->base_gfn;
64016481 u64 end = start + memslot->npages;
64026482
6403− if (!is_tdp_mmu_enabled(kvm))
6483+ if (!tdp_mmu_enabled)
64046484 return;
64056485
64066486 if (kvm_memslots_have_rmaps(kvm)) {
@@ -6450,8 +6530,7 @@ restart: · 43 unmodified lines
64506530 kvm_zap_one_rmap_spte(kvm, rmap_head, sptep);
64516531
64526532 if (kvm_available_flush_tlb_with_range())
6453− kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,
6454− KVM_PAGES_PER_HPAGE(sp->role.level));
6533+ kvm_flush_remote_tlbs_sptep(kvm, sptep);
64556534 else
64566535 need_tlb_flush = 1;
64576536
@@ -6483,7 +6562,7 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, · 25 unmodified lines
64836562 write_unlock(&kvm->mmu_lock);
64846563 }
64856564
6486− if (is_tdp_mmu_enabled(kvm)) {
6565+ if (tdp_mmu_enabled) {
64876566 read_lock(&kvm->mmu_lock);
64886567 kvm_tdp_mmu_zap_collapsible_sptes(kvm, slot);
64896568 read_unlock(&kvm->mmu_lock);
@@ -6518,7 +6597,7 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, · 28 unmodified lines
65186597 write_unlock(&kvm->mmu_lock);
65196598 }
65206599
6521− if (is_tdp_mmu_enabled(kvm)) {
6600+ if (tdp_mmu_enabled) {
65226601 read_lock(&kvm->mmu_lock);
65236602 kvm_tdp_mmu_clear_dirty_slot(kvm, memslot);
65246603 read_unlock(&kvm->mmu_lock);
@@ -6553,7 +6632,7 @@ restart: · 28 unmodified lines
65536632
65546633 kvm_mmu_commit_zap_page(kvm, &invalid_list);
65556634
6556− if (is_tdp_mmu_enabled(kvm))
6635+ if (tdp_mmu_enabled)
65576636 kvm_tdp_mmu_zap_all(kvm);
65586637
65596638 write_unlock(&kvm->mmu_lock);
@@ -6579,7 +6658,7 @@ void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen) · 19 unmodified lines
65796658 * zap all shadow pages.
65806659 */
65816660 if (unlikely(gen == 0)) {
6582− kvm_debug_ratelimited("kvm: zapping shadow pages for mmio generation wraparound\n");
6661+ kvm_debug_ratelimited("zapping shadow pages for mmio generation wraparound\n");
65836662 kvm_mmu_zap_all_fast(kvm);
65846663 }
65856664 }
@@ -6684,7 +6763,7 @@ static int set_nx_huge_pages(const char *val, const struct kernel_param *kp) · 98 unmodified lines
66846763 new_val = 1;
66856764 else if (sysfs_streq(val, "auto"))
66866765 new_val = get_nx_auto_mode();
6687− else if (strtobool(val, &new_val) < 0)
6766+ else if (kstrtobool(val, &new_val) < 0)
66886767 return -EINVAL;
66896768
66906769 __set_nx_huge_pages(new_val);
@@ -6718,6 +6797,13 @@ void __init kvm_mmu_x86_module_init(void) · 27 unmodified lines
67186797 if (nx_huge_pages == -1)
67196798 __set_nx_huge_pages(get_nx_auto_mode());
67206799
6800+ /*
6801+ * Snapshot userspace's desire to enable the TDP MMU. Whether or not the
6802+ * TDP MMU is actually enabled is determined in kvm_configure_mmu()
6803+ * when the vendor module is loaded.
6804+ */
6805+ tdp_mmu_allowed = tdp_mmu_enabled;
6806+
67216807 kvm_mmu_spte_module_init();
67226808 }
67236809
arch/x86/kvm/mmu/mmu_internal.h+21−1
@@ -156,6 +156,11 @@ static inline bool kvm_mmu_page_ad_need_write_protect(struct kvm_mmu_page *sp) · 155 unmodified lines
156156 return kvm_x86_ops.cpu_dirty_log_size && sp->role.guest_mode;
157157 }
158158
159+static inline gfn_t gfn_round_for_level(gfn_t gfn, int level)
160+{
161+ return gfn & -KVM_PAGES_PER_HPAGE(level);
162+}
163+
159164 int mmu_try_to_unsync_pages(struct kvm *kvm, const struct kvm_memory_slot *slot,
160165 gfn_t gfn, bool can_unsync, bool prefetch);
161166
@@ -164,8 +169,17 @@ void kvm_mmu_gfn_allow_lpage(const struct kvm_memory_slot *slot, gfn_t gfn); · 2 unmodified lines
164169 bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,
165170 struct kvm_memory_slot *slot, u64 gfn,
166171 int min_level);
172+
167173 void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
168174 u64 start_gfn, u64 pages);
175+
176+/* Flush the given page (huge or not) of guest memory. */
177+static inline void kvm_flush_remote_tlbs_gfn(struct kvm *kvm, gfn_t gfn, int level)
178+{
179+ kvm_flush_remote_tlbs_with_address(kvm, gfn_round_for_level(gfn, level),
180+ KVM_PAGES_PER_HPAGE(level));
181+}
182+
169183 unsigned int pte_list_count(struct kvm_rmap_head *rmap_head);
170184
171185 extern int nx_huge_pages;
@@ -199,7 +213,7 @@ struct kvm_page_fault { · 27 unmodified lines
199213
200214 /*
201215 * Maximum page size that can be created for this fault; input to
202− * FNAME(fetch), __direct_map and kvm_tdp_mmu_map.
216+ * FNAME(fetch), direct_map() and kvm_tdp_mmu_map().
203217 */
204218 u8 max_level;
205219
@@ -222,6 +236,7 @@ struct kvm_page_fault { · 16 unmodified lines
222236 struct kvm_memory_slot *slot;
223237
224238 /* Outputs of kvm_faultin_pfn. */
239+ unsigned long mmu_seq;
225240 kvm_pfn_t pfn;
226241 hva_t hva;
227242 bool map_writable;
@@ -279,6 +294,11 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, · 51 unmodified lines
279294 };
280295 int r;
281296
297+ if (vcpu->arch.mmu->root_role.direct) {
298+ fault.gfn = fault.addr >> PAGE_SHIFT;
299+ fault.slot = kvm_vcpu_gfn_to_memslot(vcpu, fault.gfn);
300+ }
301+
282302 /*
283303 * Async #PF "faults", a.k.a. prefetch faults, are not faults from the
284304 * guest perspective and have already been counted at the time of the
arch/x86/kvm/mmu/page_track.c+1−0
@@ -10,6 +10,7 @@ · 9 unmodified lines
1010 * Author:
1111 * Xiao Guangrong <guangrong.xiao@linux.intel.com>
1212 */
13+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
1314
1415 #include <linux/kvm_host.h>
1516 #include <linux/rculist.h>
arch/x86/kvm/mmu/paging_tmpl.h+7−18
@@ -642,12 +642,12 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · 641 unmodified lines
642642 if (WARN_ON(!VALID_PAGE(vcpu->arch.mmu->root.hpa)))
643643 goto out_gpte_changed;
644644
645− for (shadow_walk_init(&it, vcpu, fault->addr);
646− shadow_walk_okay(&it) && it.level > gw->level;
647− shadow_walk_next(&it)) {
645+ for_each_shadow_entry(vcpu, fault->addr, it) {
648646 gfn_t table_gfn;
649647
650648 clear_sp_write_flooding_count(it.sptep);
649+ if (it.level == gw->level)
650+ break;
651651
652652 table_gfn = gw->table_gfn[it.level - 2];
653653 access = gw->pt_access[it.level - 2];
@@ -692,8 +692,6 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · 38 unmodified lines
692692 trace_kvm_mmu_spte_requested(fault);
693693
694694 for (; shadow_walk_okay(&it); shadow_walk_next(&it)) {
695− clear_sp_write_flooding_count(it.sptep);
696−
697695 /*
698696 * We cannot overwrite existing page tables with an NX
699697 * large page, as the leaf could be executable.
@@ -701,7 +699,7 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · 1 unmodified line
701699 if (fault->nx_huge_page_workaround_enabled)
702700 disallowed_hugepage_adjust(fault, *it.sptep, it.level);
703701
704− base_gfn = fault->gfn & ~(KVM_PAGES_PER_HPAGE(it.level) - 1);
702+ base_gfn = gfn_round_for_level(fault->gfn, it.level);
705703 if (it.level == fault->goal_level)
706704 break;
707705
@@ -791,7 +789,6 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault · 83 unmodified lines
791789 {
792790 struct guest_walker walker;
793791 int r;
794− unsigned long mmu_seq;
795792 bool is_self_change_mapping;
796793
797794 pgprintk("%s: addr %lx err %x\n", __func__, fault->addr, fault->error_code);
@@ -838,14 +835,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault · 40 unmodified lines
838835 else
839836 fault->max_level = walker.level;
840837
841− mmu_seq = vcpu->kvm->mmu_invalidate_seq;
842− smp_rmb();
843−
844− r = kvm_faultin_pfn(vcpu, fault);
845− if (r != RET_PF_CONTINUE)
846− return r;
847−
848− r = handle_abnormal_pfn(vcpu, fault, walker.pte_access);
838+ r = kvm_faultin_pfn(vcpu, fault, walker.pte_access);
849839 if (r != RET_PF_CONTINUE)
850840 return r;
851841
@@ -871,7 +861,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault · 19 unmodified lines
871861 r = RET_PF_RETRY;
872862 write_lock(&vcpu->kvm->mmu_lock);
873863
874− if (is_page_fault_stale(vcpu, fault, mmu_seq))
864+ if (is_page_fault_stale(vcpu, fault))
875865 goto out_unlock;
876866
877867 r = make_mmu_pages_available(vcpu);
@@ -937,8 +927,7 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa) · 59 unmodified lines
937927
938928 mmu_page_zap_pte(vcpu->kvm, sp, sptep, NULL);
939929 if (is_shadow_present_pte(old_spte))
940− kvm_flush_remote_tlbs_with_address(vcpu->kvm,
941− sp->gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
930+ kvm_flush_remote_tlbs_sptep(vcpu->kvm, sptep);
942931
943932 if (!rmap_can_add(vcpu))
944933 break;
arch/x86/kvm/mmu/spte.c+5−5
@@ -7,7 +7,7 @@ · 6 unmodified lines
77 * Copyright (C) 2006 Qumranet, Inc.
88 * Copyright 2020 Red Hat, Inc. and/or its affiliates.
99 */
10−
10+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
1111
1212 #include <linux/kvm_host.h>
1313 #include "mmu.h"
@@ -147,9 +147,9 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, · 133 unmodified lines
147147 WARN_ON_ONCE(!pte_access && !shadow_present_mask);
148148
149149 if (sp->role.ad_disabled)
150− spte |= SPTE_TDP_AD_DISABLED_MASK;
150+ spte |= SPTE_TDP_AD_DISABLED;
151151 else if (kvm_mmu_page_ad_need_write_protect(sp))
152− spte |= SPTE_TDP_AD_WRPROT_ONLY_MASK;
152+ spte |= SPTE_TDP_AD_WRPROT_ONLY;
153153
154154 /*
155155 * For the EPT case, shadow_present_mask is 0 if hardware
@@ -317,7 +317,7 @@ u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled) · 161 unmodified lines
317317 shadow_user_mask | shadow_x_mask | shadow_me_value;
318318
319319 if (ad_disabled)
320− spte |= SPTE_TDP_AD_DISABLED_MASK;
320+ spte |= SPTE_TDP_AD_DISABLED;
321321 else
322322 spte |= shadow_accessed_mask;
323323
@@ -352,7 +352,7 @@ u64 mark_spte_for_access_track(u64 spte) · 28 unmodified lines
352352
353353 WARN_ONCE(spte & (SHADOW_ACC_TRACK_SAVED_BITS_MASK <<
354354 SHADOW_ACC_TRACK_SAVED_BITS_SHIFT),
355− "kvm: Access Tracking saved bit locations are not zero\n");
355+ "Access Tracking saved bit locations are not zero\n");
356356
357357 spte |= (spte & SHADOW_ACC_TRACK_SAVED_BITS_MASK) <<
358358 SHADOW_ACC_TRACK_SAVED_BITS_SHIFT;
arch/x86/kvm/mmu/spte.h+10−10
@@ -28,10 +28,10 @@ · 27 unmodified lines
2828 */
2929 #define SPTE_TDP_AD_SHIFT 52
3030 #define SPTE_TDP_AD_MASK (3ULL << SPTE_TDP_AD_SHIFT)
31−#define SPTE_TDP_AD_ENABLED_MASK (0ULL << SPTE_TDP_AD_SHIFT)
32−#define SPTE_TDP_AD_DISABLED_MASK (1ULL << SPTE_TDP_AD_SHIFT)
33−#define SPTE_TDP_AD_WRPROT_ONLY_MASK (2ULL << SPTE_TDP_AD_SHIFT)
34−static_assert(SPTE_TDP_AD_ENABLED_MASK == 0);
31+#define SPTE_TDP_AD_ENABLED (0ULL << SPTE_TDP_AD_SHIFT)
32+#define SPTE_TDP_AD_DISABLED (1ULL << SPTE_TDP_AD_SHIFT)
33+#define SPTE_TDP_AD_WRPROT_ONLY (2ULL << SPTE_TDP_AD_SHIFT)
34+static_assert(SPTE_TDP_AD_ENABLED == 0);
3535
3636 #ifdef CONFIG_DYNAMIC_PHYSICAL_MASK
3737 #define SPTE_BASE_ADDR_MASK (physical_mask & ~(u64)(PAGE_SIZE-1))
@@ -164,7 +164,7 @@ extern u64 __read_mostly shadow_me_value; · 126 unmodified lines
164164 extern u64 __read_mostly shadow_me_mask;
165165
166166 /*
167− * SPTEs in MMUs without A/D bits are marked with SPTE_TDP_AD_DISABLED_MASK;
167+ * SPTEs in MMUs without A/D bits are marked with SPTE_TDP_AD_DISABLED;
168168 * shadow_acc_track_mask is the set of bits to be cleared in non-accessed
169169 * pages.
170170 */
@@ -266,18 +266,18 @@ static inline bool sp_ad_disabled(struct kvm_mmu_page *sp) · 95 unmodified lines
266266 static inline bool spte_ad_enabled(u64 spte)
267267 {
268268 MMU_WARN_ON(!is_shadow_present_pte(spte));
269− return (spte & SPTE_TDP_AD_MASK) != SPTE_TDP_AD_DISABLED_MASK;
269+ return (spte & SPTE_TDP_AD_MASK) != SPTE_TDP_AD_DISABLED;
270270 }
271271
272272 static inline bool spte_ad_need_write_protect(u64 spte)
273273 {
274274 MMU_WARN_ON(!is_shadow_present_pte(spte));
275275 /*
276− * This is benign for non-TDP SPTEs as SPTE_TDP_AD_ENABLED_MASK is '0',
276+ * This is benign for non-TDP SPTEs as SPTE_TDP_AD_ENABLED is '0',
277277 * and non-TDP SPTEs will never set these bits. Optimize for 64-bit
278278 * TDP and do the A/D type check unconditionally.
279279 */
280− return (spte & SPTE_TDP_AD_MASK) != SPTE_TDP_AD_ENABLED_MASK;
280+ return (spte & SPTE_TDP_AD_MASK) != SPTE_TDP_AD_ENABLED;
281281 }
282282
283283 static inline u64 spte_shadow_accessed_mask(u64 spte)
@@ -435,11 +435,11 @@ static inline void check_spte_writable_invariants(u64 spte) · 151 unmodified lines
435435 {
436436 if (spte & shadow_mmu_writable_mask)
437437 WARN_ONCE(!(spte & shadow_host_writable_mask),
438− "kvm: MMU-writable SPTE is not Host-writable: %llx",
438+ KBUILD_MODNAME ": MMU-writable SPTE is not Host-writable: %llx",
439439 spte);
440440 else
441441 WARN_ONCE(is_writable_pte(spte),
442− "kvm: Writable SPTE is not MMU-writable: %llx", spte);
442+ KBUILD_MODNAME ": Writable SPTE is not MMU-writable: %llx", spte);
443443 }
444444
445445 static inline bool is_mmu_writable_spte(u64 spte)
arch/x86/kvm/mmu/tdp_iter.c+4−8
@@ -1,4 +1,5 @@
11 // SPDX-License-Identifier: GPL-2.0
2+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
23
34 #include "mmu_internal.h"
45 #include "tdp_iter.h"
@@ -15,11 +16,6 @@ static void tdp_iter_refresh_sptep(struct tdp_iter *iter) · 10 unmodified lines
1516 iter->old_spte = kvm_tdp_mmu_read_spte(iter->sptep);
1617 }
1718
18−static gfn_t round_gfn_for_level(gfn_t gfn, int level)
19−{
20− return gfn & -KVM_PAGES_PER_HPAGE(level);
21−}
22−
2319 /*
2420 * Return the TDP iterator to the root PT and allow it to continue its
2521 * traversal over the paging structure from there.
@@ -30,7 +26,7 @@ void tdp_iter_restart(struct tdp_iter *iter) · 4 unmodified lines
3026 iter->yielded_gfn = iter->next_last_level_gfn;
3127 iter->level = iter->root_level;
3228
33− iter->gfn = round_gfn_for_level(iter->next_last_level_gfn, iter->level);
29+ iter->gfn = gfn_round_for_level(iter->next_last_level_gfn, iter->level);
3430 tdp_iter_refresh_sptep(iter);
3531
3632 iter->valid = true;
@@ -97,7 +93,7 @@ static bool try_step_down(struct tdp_iter *iter) · 60 unmodified lines
9793
9894 iter->level--;
9995 iter->pt_path[iter->level - 1] = child_pt;
100− iter->gfn = round_gfn_for_level(iter->next_last_level_gfn, iter->level);
96+ iter->gfn = gfn_round_for_level(iter->next_last_level_gfn, iter->level);
10197 tdp_iter_refresh_sptep(iter);
10298
10399 return true;
@@ -139,7 +135,7 @@ static bool try_step_up(struct tdp_iter *iter) · 35 unmodified lines
139135 return false;
140136
141137 iter->level++;
142− iter->gfn = round_gfn_for_level(iter->gfn, iter->level);
138+ iter->gfn = gfn_round_for_level(iter->gfn, iter->level);
143139 tdp_iter_refresh_sptep(iter);
144140
145141 return true;
arch/x86/kvm/mmu/tdp_mmu.c+4−16
@@ -1,4 +1,5 @@
11 // SPDX-License-Identifier: GPL-2.0
2+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
23
34 #include "mmu.h"
45 #include "mmu_internal.h"
@@ -10,23 +11,15 @@ · 5 unmodified lines
1011 #include <asm/cmpxchg.h>
1112 #include <trace/events/kvm.h>
1213
13−static bool __read_mostly tdp_mmu_enabled = true;
14−module_param_named(tdp_mmu, tdp_mmu_enabled, bool, 0644);
15−
1614 /* Initializes the TDP MMU for the VM, if enabled. */
1715 int kvm_mmu_init_tdp_mmu(struct kvm *kvm)
1816 {
1917 struct workqueue_struct *wq;
2018
21− if (!tdp_enabled || !READ_ONCE(tdp_mmu_enabled))
22− return 0;
23−
2419 wq = alloc_workqueue("kvm", WQ_UNBOUND|WQ_MEM_RECLAIM|WQ_CPU_INTENSIVE, 0);
2520 if (!wq)
2621 return -ENOMEM;
2722
28− /* This should not be changed for the lifetime of the VM. */
29− kvm->arch.tdp_mmu_enabled = true;
3023 INIT_LIST_HEAD(&kvm->arch.tdp_mmu_roots);
3124 spin_lock_init(&kvm->arch.tdp_mmu_pages_lock);
3225 kvm->arch.tdp_mmu_zap_wq = wq;
@@ -47,9 +40,6 @@ static __always_inline bool kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm, · 14 unmodified lines
4740
4841 void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
4942 {
50− if (!kvm->arch.tdp_mmu_enabled)
51− return;
52−
5343 /* Also waits for any queued work items. */
5444 destroy_workqueue(kvm->arch.tdp_mmu_zap_wq);
5545
@@ -144,7 +134,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root, · 88 unmodified lines
144134 if (!refcount_dec_and_test(&root->tdp_mmu_root_count))
145135 return;
146136
147− WARN_ON(!root->tdp_mmu_page);
137+ WARN_ON(!is_tdp_mmu_page(root));
148138
149139 /*
150140 * The root now has refcount=0. It is valid, but readers already
@@ -690,8 +680,7 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm, · 539 unmodified lines
690680 if (ret)
691681 return ret;
692682
693− kvm_flush_remote_tlbs_with_address(kvm, iter->gfn,
694− KVM_PAGES_PER_HPAGE(iter->level));
683+ kvm_flush_remote_tlbs_gfn(kvm, iter->gfn, iter->level);
695684
696685 /*
697686 * No other thread can overwrite the removed SPTE as they must either
@@ -1090,8 +1079,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, · 392 unmodified lines
10901079 return RET_PF_RETRY;
10911080 else if (is_shadow_present_pte(iter->old_spte) &&
10921081 !is_last_spte(iter->old_spte, iter->level))
1093− kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn,
1094− KVM_PAGES_PER_HPAGE(iter->level + 1));
1082+ kvm_flush_remote_tlbs_gfn(vcpu->kvm, iter->gfn, iter->level);
10951083
10961084 /*
10971085 * If the page fault was caused by a write but the page is write
arch/x86/kvm/mmu/tdp_mmu.h+3−22
@@ -7,6 +7,9 @@ · 6 unmodified lines
77
88 #include "spte.h"
99
10+int kvm_mmu_init_tdp_mmu(struct kvm *kvm);
11+void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm);
12+
1013 hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);
1114
1215 __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm_mmu_page *root)
@@ -68,31 +71,9 @@ u64 *kvm_tdp_mmu_fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, u64 addr, · 55 unmodified lines
6871 u64 *spte);
6972
7073 #ifdef CONFIG_X86_64
71−int kvm_mmu_init_tdp_mmu(struct kvm *kvm);
72−void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm);
7374 static inline bool is_tdp_mmu_page(struct kvm_mmu_page *sp) { return sp->tdp_mmu_page; }
74−
75−static inline bool is_tdp_mmu(struct kvm_mmu *mmu)
76−{
77− struct kvm_mmu_page *sp;
78− hpa_t hpa = mmu->root.hpa;
79−
80− if (WARN_ON(!VALID_PAGE(hpa)))
81− return false;
82−
83− /*
84− * A NULL shadow page is legal when shadowing a non-paging guest with
85− * PAE paging, as the MMU will be direct with root_hpa pointing at the
86− * pae_root page, not a shadow page.
87− */
88− sp = to_shadow_page(hpa);
89− return sp && is_tdp_mmu_page(sp) && sp->root_count;
90−}
9175 #else
92−static inline int kvm_mmu_init_tdp_mmu(struct kvm *kvm) { return 0; }
93−static inline void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) {}
9476 static inline bool is_tdp_mmu_page(struct kvm_mmu_page *sp) { return false; }
95−static inline bool is_tdp_mmu(struct kvm_mmu *mmu) { return false; }
9677 #endif
9778
9879 #endif /* __KVM_X86_MMU_TDP_MMU_H */
arch/x86/kvm/mtrr.c+1−0
@@ -13,6 +13,7 @@ · 12 unmodified lines
1313 * Paolo Bonzini <pbonzini@redhat.com>
1414 * Xiao Guangrong <guangrong.xiao@linux.intel.com>
1515 */
16+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
1617
1718 #include <linux/kvm_host.h>
1819 #include <asm/mtrr.h>
arch/x86/kvm/pmu.c+238−52
@@ -9,6 +9,7 @@ · 8 unmodified lines
99 * Gleb Natapov <gleb@redhat.com>
1010 * Wei Huang <wei@redhat.com>
1111 */
12+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
1213
1314 #include <linux/types.h>
1415 #include <linux/kvm_host.h>
@@ -28,9 +29,18 @@ · 13 unmodified lines
2829 struct x86_pmu_capability __read_mostly kvm_pmu_cap;
2930 EXPORT_SYMBOL_GPL(kvm_pmu_cap);
3031
31−static const struct x86_cpu_id vmx_icl_pebs_cpu[] = {
32+/* Precise Distribution of Instructions Retired (PDIR) */
33+static const struct x86_cpu_id vmx_pebs_pdir_cpu[] = {
3234 X86_MATCH_INTEL_FAM6_MODEL(ICELAKE_D, NULL),
3335 X86_MATCH_INTEL_FAM6_MODEL(ICELAKE_X, NULL),
36+ /* Instruction-Accurate PDIR (PDIR++) */
37+ X86_MATCH_INTEL_FAM6_MODEL(SAPPHIRERAPIDS_X, NULL),
38+ {}
39+};
40+
41+/* Precise Distribution (PDist) */
42+static const struct x86_cpu_id vmx_pebs_pdist_cpu[] = {
43+ X86_MATCH_INTEL_FAM6_MODEL(SAPPHIRERAPIDS_X, NULL),
3444 {}
3545 };
3646
@@ -155,6 +165,28 @@ static void kvm_perf_overflow(struct perf_event *perf_event, · 118 unmodified lines
155165 kvm_make_request(KVM_REQ_PMU, pmc->vcpu);
156166 }
157167
168+static u64 pmc_get_pebs_precise_level(struct kvm_pmc *pmc)
169+{
170+ /*
171+ * For some model specific pebs counters with special capabilities
172+ * (PDIR, PDIR++, PDIST), KVM needs to raise the event precise
173+ * level to the maximum value (currently 3, backwards compatible)
174+ * so that the perf subsystem would assign specific hardware counter
175+ * with that capability for vPMC.
176+ */
177+ if ((pmc->idx == 0 && x86_match_cpu(vmx_pebs_pdist_cpu)) ||
178+ (pmc->idx == 32 && x86_match_cpu(vmx_pebs_pdir_cpu)))
179+ return 3;
180+
181+ /*
182+ * The non-zero precision level of guest event makes the ordinary
183+ * guest event becomes a guest PEBS event and triggers the host
184+ * PEBS PMI handler to determine whether the PEBS overflow PMI
185+ * comes from the host counters or the guest.
186+ */
187+ return 1;
188+}
189+
158190 static int pmc_reprogram_counter(struct kvm_pmc *pmc, u32 type, u64 config,
159191 bool exclude_user, bool exclude_kernel,
160192 bool intr)
@@ -186,22 +218,12 @@ static int pmc_reprogram_counter(struct kvm_pmc *pmc, u32 type, u64 config, · 25 unmodified lines
186218 }
187219 if (pebs) {
188220 /*
189− * The non-zero precision level of guest event makes the ordinary
190− * guest event becomes a guest PEBS event and triggers the host
191− * PEBS PMI handler to determine whether the PEBS overflow PMI
192− * comes from the host counters or the guest.
193− *
194221 * For most PEBS hardware events, the difference in the software
195222 * precision levels of guest and host PEBS events will not affect
196223 * the accuracy of the PEBS profiling result, because the "event IP"
197224 * in the PEBS record is calibrated on the guest side.
198− *
199− * On Icelake everything is fine. Other hardware (GLC+, TNT+) that
200− * could possibly care here is unsupported and needs changes.
201225 */
202− attr.precise_ip = 1;
203− if (x86_match_cpu(vmx_icl_pebs_cpu) && pmc->idx == 32)
204− attr.precise_ip = 3;
226+ attr.precise_ip = pmc_get_pebs_precise_level(pmc);
205227 }
206228
207229 event = perf_event_create_kernel_counter(&attr, -1, current,
@@ -254,48 +276,128 @@ static bool pmc_resume_counter(struct kvm_pmc *pmc) · 46 unmodified lines
254276 return true;
255277 }
256278
257−static int cmp_u64(const void *pa, const void *pb)
279+static int filter_cmp(const void *pa, const void *pb, u64 mask)
258280 {
259− u64 a = *(u64 *)pa;
260− u64 b = *(u64 *)pb;
281+ u64 a = *(u64 *)pa & mask;
282+ u64 b = *(u64 *)pb & mask;
261283
262284 return (a > b) - (a < b);
263285 }
264286
287+
288+static int filter_sort_cmp(const void *pa, const void *pb)
289+{
290+ return filter_cmp(pa, pb, (KVM_PMU_MASKED_ENTRY_EVENT_SELECT |
291+ KVM_PMU_MASKED_ENTRY_EXCLUDE));
292+}
293+
294+/*
295+ * For the event filter, searching is done on the 'includes' list and
296+ * 'excludes' list separately rather than on the 'events' list (which
297+ * has both). As a result the exclude bit can be ignored.
298+ */
299+static int filter_event_cmp(const void *pa, const void *pb)
300+{
301+ return filter_cmp(pa, pb, (KVM_PMU_MASKED_ENTRY_EVENT_SELECT));
302+}
303+
304+static int find_filter_index(u64 *events, u64 nevents, u64 key)
305+{
306+ u64 *fe = bsearch(&key, events, nevents, sizeof(events[0]),
307+ filter_event_cmp);
308+
309+ if (!fe)
310+ return -1;
311+
312+ return fe - events;
313+}
314+
315+static bool is_filter_entry_match(u64 filter_event, u64 umask)
316+{
317+ u64 mask = filter_event >> (KVM_PMU_MASKED_ENTRY_UMASK_MASK_SHIFT - 8);
318+ u64 match = filter_event & KVM_PMU_MASKED_ENTRY_UMASK_MATCH;
319+
320+ BUILD_BUG_ON((KVM_PMU_ENCODE_MASKED_ENTRY(0, 0xff, 0, false) >>
321+ (KVM_PMU_MASKED_ENTRY_UMASK_MASK_SHIFT - 8)) !=
322+ ARCH_PERFMON_EVENTSEL_UMASK);
323+
324+ return (umask & mask) == match;
325+}
326+
327+static bool filter_contains_match(u64 *events, u64 nevents, u64 eventsel)
328+{
329+ u64 event_select = eventsel & kvm_pmu_ops.EVENTSEL_EVENT;
330+ u64 umask = eventsel & ARCH_PERFMON_EVENTSEL_UMASK;
331+ int i, index;
332+
333+ index = find_filter_index(events, nevents, event_select);
334+ if (index < 0)
335+ return false;
336+
337+ /*
338+ * Entries are sorted by the event select. Walk the list in both
339+ * directions to process all entries with the targeted event select.
340+ */
341+ for (i = index; i < nevents; i++) {
342+ if (filter_event_cmp(&events[i], &event_select))
343+ break;
344+
345+ if (is_filter_entry_match(events[i], umask))
346+ return true;
347+ }
348+
349+ for (i = index - 1; i >= 0; i--) {
350+ if (filter_event_cmp(&events[i], &event_select))
351+ break;
352+
353+ if (is_filter_entry_match(events[i], umask))
354+ return true;
355+ }
356+
357+ return false;
358+}
359+
360+static bool is_gp_event_allowed(struct kvm_x86_pmu_event_filter *f,
361+ u64 eventsel)
362+{
363+ if (filter_contains_match(f->includes, f->nr_includes, eventsel) &&
364+ !filter_contains_match(f->excludes, f->nr_excludes, eventsel))
365+ return f->action == KVM_PMU_EVENT_ALLOW;
366+
367+ return f->action == KVM_PMU_EVENT_DENY;
368+}
369+
370+static bool is_fixed_event_allowed(struct kvm_x86_pmu_event_filter *filter,
371+ int idx)
372+{
373+ int fixed_idx = idx - INTEL_PMC_IDX_FIXED;
374+
375+ if (filter->action == KVM_PMU_EVENT_DENY &&
376+ test_bit(fixed_idx, (ulong *)&filter->fixed_counter_bitmap))
377+ return false;
378+ if (filter->action == KVM_PMU_EVENT_ALLOW &&
379+ !test_bit(fixed_idx, (ulong *)&filter->fixed_counter_bitmap))
380+ return false;
381+
382+ return true;
383+}
384+
265385 static bool check_pmu_event_filter(struct kvm_pmc *pmc)
266386 {
267− struct kvm_pmu_event_filter *filter;
387+ struct kvm_x86_pmu_event_filter *filter;
268388 struct kvm *kvm = pmc->vcpu->kvm;
269− bool allow_event = true;
270− __u64 key;
271− int idx;
272389
273390 if (!static_call(kvm_x86_pmu_hw_event_available)(pmc))
274391 return false;
275392
276393 filter = srcu_dereference(kvm->arch.pmu_event_filter, &kvm->srcu);
277394 if (!filter)
278− goto out;
395+ return true;
279396
280− if (pmc_is_gp(pmc)) {
281− key = pmc->eventsel & AMD64_RAW_EVENT_MASK_NB;
282− if (bsearch(&key, filter->events, filter->nevents,
283− sizeof(__u64), cmp_u64))
284− allow_event = filter->action == KVM_PMU_EVENT_ALLOW;
285− else
286− allow_event = filter->action == KVM_PMU_EVENT_DENY;
287− } else {
288− idx = pmc->idx - INTEL_PMC_IDX_FIXED;
289− if (filter->action == KVM_PMU_EVENT_DENY &&
290− test_bit(idx, (ulong *)&filter->fixed_counter_bitmap))
291− allow_event = false;
292− if (filter->action == KVM_PMU_EVENT_ALLOW &&
293− !test_bit(idx, (ulong *)&filter->fixed_counter_bitmap))
294− allow_event = false;
295− }
397+ if (pmc_is_gp(pmc))
398+ return is_gp_event_allowed(filter, pmc->eventsel);
296399
297−out:
298− return allow_event;
400+ return is_fixed_event_allowed(filter, pmc->idx);
299401 }
300402
301403 static void reprogram_counter(struct kvm_pmc *pmc)
@@ -592,47 +694,133 @@ void kvm_pmu_trigger_event(struct kvm_vcpu *vcpu, u64 perf_hw_id) · 290 unmodified lines
592694 }
593695 EXPORT_SYMBOL_GPL(kvm_pmu_trigger_event);
594696
697+static bool is_masked_filter_valid(const struct kvm_x86_pmu_event_filter *filter)
698+{
699+ u64 mask = kvm_pmu_ops.EVENTSEL_EVENT |
700+ KVM_PMU_MASKED_ENTRY_UMASK_MASK |
701+ KVM_PMU_MASKED_ENTRY_UMASK_MATCH |
702+ KVM_PMU_MASKED_ENTRY_EXCLUDE;
703+ int i;
704+
705+ for (i = 0; i < filter->nevents; i++) {
706+ if (filter->events[i] & ~mask)
707+ return false;
708+ }
709+
710+ return true;
711+}
712+
713+static void convert_to_masked_filter(struct kvm_x86_pmu_event_filter *filter)
714+{
715+ int i, j;
716+
717+ for (i = 0, j = 0; i < filter->nevents; i++) {
718+ /*
719+ * Skip events that are impossible to match against a guest
720+ * event. When filtering, only the event select + unit mask
721+ * of the guest event is used. To maintain backwards
722+ * compatibility, impossible filters can't be rejected :-(
723+ */
724+ if (filter->events[i] & ~(kvm_pmu_ops.EVENTSEL_EVENT |
725+ ARCH_PERFMON_EVENTSEL_UMASK))
726+ continue;
727+ /*
728+ * Convert userspace events to a common in-kernel event so
729+ * only one code path is needed to support both events. For
730+ * the in-kernel events use masked events because they are
731+ * flexible enough to handle both cases. To convert to masked
732+ * events all that's needed is to add an "all ones" umask_mask,
733+ * (unmasked filter events don't support EXCLUDE).
734+ */
735+ filter->events[j++] = filter->events[i] |
736+ (0xFFULL << KVM_PMU_MASKED_ENTRY_UMASK_MASK_SHIFT);
737+ }
738+
739+ filter->nevents = j;
740+}
741+
742+static int prepare_filter_lists(struct kvm_x86_pmu_event_filter *filter)
743+{
744+ int i;
745+
746+ if (!(filter->flags & KVM_PMU_EVENT_FLAG_MASKED_EVENTS))
747+ convert_to_masked_filter(filter);
748+ else if (!is_masked_filter_valid(filter))
749+ return -EINVAL;
750+
751+ /*
752+ * Sort entries by event select and includes vs. excludes so that all
753+ * entries for a given event select can be processed efficiently during
754+ * filtering. The EXCLUDE flag uses a more significant bit than the
755+ * event select, and so the sorted list is also effectively split into
756+ * includes and excludes sub-lists.
757+ */
758+ sort(&filter->events, filter->nevents, sizeof(filter->events[0]),
759+ filter_sort_cmp, NULL);
760+
761+ i = filter->nevents;
762+ /* Find the first EXCLUDE event (only supported for masked events). */
763+ if (filter->flags & KVM_PMU_EVENT_FLAG_MASKED_EVENTS) {
764+ for (i = 0; i < filter->nevents; i++) {
765+ if (filter->events[i] & KVM_PMU_MASKED_ENTRY_EXCLUDE)
766+ break;
767+ }
768+ }
769+
770+ filter->nr_includes = i;
771+ filter->nr_excludes = filter->nevents - filter->nr_includes;
772+ filter->includes = filter->events;
773+ filter->excludes = filter->events + filter->nr_includes;
774+
775+ return 0;
776+}
777+
595778 int kvm_vm_ioctl_set_pmu_event_filter(struct kvm *kvm, void __user *argp)
596779 {
597− struct kvm_pmu_event_filter tmp, *filter;
780+ struct kvm_pmu_event_filter __user *user_filter = argp;
781+ struct kvm_x86_pmu_event_filter *filter;
782+ struct kvm_pmu_event_filter tmp;
598783 struct kvm_vcpu *vcpu;
599784 unsigned long i;
600785 size_t size;
601786 int r;
602787
603− if (copy_from_user(&tmp, argp, sizeof(tmp)))
788+ if (copy_from_user(&tmp, user_filter, sizeof(tmp)))
604789 return -EFAULT;
605790
606791 if (tmp.action != KVM_PMU_EVENT_ALLOW &&
607792 tmp.action != KVM_PMU_EVENT_DENY)
608793 return -EINVAL;
609794
610− if (tmp.flags != 0)
795+ if (tmp.flags & ~KVM_PMU_EVENT_FLAGS_VALID_MASK)
611796 return -EINVAL;
612797
613798 if (tmp.nevents > KVM_PMU_EVENT_FILTER_MAX_EVENTS)
614799 return -E2BIG;
615800
616801 size = struct_size(filter, events, tmp.nevents);
617− filter = kmalloc(size, GFP_KERNEL_ACCOUNT);
802+ filter = kzalloc(size, GFP_KERNEL_ACCOUNT);
618803 if (!filter)
619804 return -ENOMEM;
620805
806+ filter->action = tmp.action;
807+ filter->nevents = tmp.nevents;
808+ filter->fixed_counter_bitmap = tmp.fixed_counter_bitmap;
809+ filter->flags = tmp.flags;
810+
621811 r = -EFAULT;
622− if (copy_from_user(filter, argp, size))
812+ if (copy_from_user(filter->events, user_filter->events,
813+ sizeof(filter->events[0]) * filter->nevents))
623814 goto cleanup;
624815
625− /* Ensure nevents can't be changed between the user copies. */
626− *filter = tmp;
627−
628− /*
629− * Sort the in-kernel list so that we can search it with bsearch.
630− */
631− sort(&filter->events, filter->nevents, sizeof(__u64), cmp_u64, NULL);
816+ r = prepare_filter_lists(filter);
817+ if (r)
818+ goto cleanup;
632819
633820 mutex_lock(&kvm->lock);
634821 filter = rcu_replace_pointer(kvm->arch.pmu_event_filter, filter,
635822 mutex_is_locked(&kvm->lock));
823+ mutex_unlock(&kvm->lock);
636824 synchronize_srcu_expedited(&kvm->srcu);
637825
638826 BUILD_BUG_ON(sizeof(((struct kvm_pmu *)0)->reprogram_pmi) >
@@ -643,8 +831,6 @@ int kvm_vm_ioctl_set_pmu_event_filter(struct kvm *kvm, void __user *argp) · 4 unmodified lines
643831
644832 kvm_make_all_cpus_request(kvm, KVM_REQ_PMU);
645833
646− mutex_unlock(&kvm->lock);
647−
648834 r = 0;
649835 cleanup:
650836 kfree(filter);
arch/x86/kvm/pmu.h+6−7
@@ -18,12 +18,6 @@ · 17 unmodified lines
1818 #define VMWARE_BACKDOOR_PMC_REAL_TIME 0x10001
1919 #define VMWARE_BACKDOOR_PMC_APPARENT_TIME 0x10002
2020
21−struct kvm_event_hw_type_mapping {
22− u8 eventsel;
23− u8 unit_mask;
24− unsigned event_type;
25−};
26−
2721 struct kvm_pmu_ops {
2822 bool (*hw_event_available)(struct kvm_pmc *pmc);
2923 bool (*pmc_is_enabled)(struct kvm_pmc *pmc);
@@ -40,6 +34,9 @@ struct kvm_pmu_ops { · 10 unmodified lines
4034 void (*reset)(struct kvm_vcpu *vcpu);
4135 void (*deliver_pmi)(struct kvm_vcpu *vcpu);
4236 void (*cleanup)(struct kvm_vcpu *vcpu);
37+
38+ const u64 EVENTSEL_EVENT;
39+ const int MAX_NR_GP_COUNTERS;
4340 };
4441
4542 void kvm_pmu_ops_update(const struct kvm_pmu_ops *pmu_ops);
@@ -161,7 +158,7 @@ static inline bool pmc_speculative_in_use(struct kvm_pmc *pmc) · 115 unmodified lines
161158
162159 extern struct x86_pmu_capability kvm_pmu_cap;
163160
164−static inline void kvm_init_pmu_capability(void)
161+static inline void kvm_init_pmu_capability(const struct kvm_pmu_ops *pmu_ops)
165162 {
166163 bool is_intel = boot_cpu_data.x86_vendor == X86_VENDOR_INTEL;
167164
@@ -192,6 +189,8 @@ static inline void kvm_init_pmu_capability(void) · 24 unmodified lines
192189 }
193190
194191 kvm_pmu_cap.version = min(kvm_pmu_cap.version, 2);
192+ kvm_pmu_cap.num_counters_gp = min(kvm_pmu_cap.num_counters_gp,
193+ pmu_ops->MAX_NR_GP_COUNTERS);
195194 kvm_pmu_cap.num_counters_fixed = min(kvm_pmu_cap.num_counters_fixed,
196195 KVM_PMC_MAX_FIXED);
197196 }
arch/x86/kvm/reverse_cpuid.h+7−0
@@ -14,6 +14,7 @@ · 13 unmodified lines
1414 enum kvm_only_cpuid_leafs {
1515 CPUID_12_EAX = NCAPINTS,
1616 CPUID_7_1_EDX,
17+ CPUID_8000_0007_EDX,
1718 NR_KVM_CPU_CAPS,
1819
1920 NKVMCAPINTS = NR_KVM_CPU_CAPS - NCAPINTS,
@@ -43,6 +44,9 @@ enum kvm_only_cpuid_leafs { · 23 unmodified lines
4344 #define X86_FEATURE_AVX_NE_CONVERT KVM_X86_FEATURE(CPUID_7_1_EDX, 5)
4445 #define X86_FEATURE_PREFETCHITI KVM_X86_FEATURE(CPUID_7_1_EDX, 14)
4546
47+/* CPUID level 0x80000007 (EDX). */
48+#define KVM_X86_FEATURE_CONSTANT_TSC KVM_X86_FEATURE(CPUID_8000_0007_EDX, 8)
49+
4650 struct cpuid_reg {
4751 u32 function;
4852 u32 index;
@@ -68,6 +72,7 @@ static const struct cpuid_reg reverse_cpuid[] = { · 19 unmodified lines
6872 [CPUID_12_EAX] = {0x00000012, 0, CPUID_EAX},
6973 [CPUID_8000_001F_EAX] = {0x8000001f, 0, CPUID_EAX},
7074 [CPUID_7_1_EDX] = { 7, 1, CPUID_EDX},
75+ [CPUID_8000_0007_EDX] = {0x80000007, 0, CPUID_EDX},
7176 [CPUID_8000_0021_EAX] = {0x80000021, 0, CPUID_EAX},
7277 };
7378
@@ -101,6 +106,8 @@ static __always_inline u32 __feature_translate(int x86_feature) · 27 unmodified lines
101106 return KVM_X86_FEATURE_SGX2;
102107 else if (x86_feature == X86_FEATURE_SGX_EDECCSSA)
103108 return KVM_X86_FEATURE_SGX_EDECCSSA;
109+ else if (x86_feature == X86_FEATURE_CONSTANT_TSC)
110+ return KVM_X86_FEATURE_CONSTANT_TSC;
104111
105112 return x86_feature;
106113 }
arch/x86/kvm/smm.c+1−2
@@ -1,4 +1,5 @@
11 /* SPDX-License-Identifier: GPL-2.0 */
2+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
23
34 #include <linux/kvm_host.h>
45 #include "x86.h"
@@ -110,8 +111,6 @@ static void check_smram_offsets(void) · 105 unmodified lines
110111
111112 void kvm_smm_changed(struct kvm_vcpu *vcpu, bool entering_smm)
112113 {
113− BUILD_BUG_ON(HF_SMM_MASK != X86EMUL_SMM_MASK);
114−
115114 trace_kvm_smm_transition(vcpu->vcpu_id, vcpu->arch.smbase, entering_smm);
116115
117116 if (entering_smm) {
arch/x86/kvm/svm/avic.c+167−207
@@ -12,7 +12,7 @@ · 11 unmodified lines
1212 * Avi Kivity <avi@qumranet.com>
1313 */
1414
15−#define pr_fmt(fmt) "SVM: " fmt
15+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
1616
1717 #include <linux/kvm_types.h>
1818 #include <linux/hashtable.h>
@@ -53,7 +53,7 @@ static DEFINE_HASHTABLE(svm_vm_data_hash, SVM_VM_DATA_HASH_BITS); · 34 unmodified lines
5353 static u32 next_vm_id = 0;
5454 static bool next_vm_id_wrapped = 0;
5555 static DEFINE_SPINLOCK(svm_vm_data_hash_lock);
56−enum avic_modes avic_mode;
56+bool x2avic_enabled;
5757
5858 /*
5959 * This is a wrapper of struct amd_iommu_ir_data.
@@ -72,20 +72,25 @@ static void avic_activate_vmcb(struct vcpu_svm *svm) · 12 unmodified lines
7272
7373 vmcb->control.int_ctl |= AVIC_ENABLE_MASK;
7474
75− /* Note:
76− * KVM can support hybrid-AVIC mode, where KVM emulates x2APIC
77− * MSR accesses, while interrupt injection to a running vCPU
78− * can be achieved using AVIC doorbell. The AVIC hardware still
79− * accelerate MMIO accesses, but this does not cause any harm
80− * as the guest is not supposed to access xAPIC mmio when uses x2APIC.
75+ /*
76+ * Note: KVM supports hybrid-AVIC mode, where KVM emulates x2APIC MSR
77+ * accesses, while interrupt injection to a running vCPU can be
78+ * achieved using AVIC doorbell. KVM disables the APIC access page
79+ * (deletes the memslot) if any vCPU has x2APIC enabled, thus enabling
80+ * AVIC in hybrid mode activates only the doorbell mechanism.
8181 */
82− if (apic_x2apic_mode(svm->vcpu.arch.apic) &&
83− avic_mode == AVIC_MODE_X2) {
82+ if (x2avic_enabled && apic_x2apic_mode(svm->vcpu.arch.apic)) {
8483 vmcb->control.int_ctl |= X2APIC_MODE_MASK;
8584 vmcb->control.avic_physical_id |= X2AVIC_MAX_PHYSICAL_ID;
8685 /* Disabling MSR intercept for x2APIC registers */
8786 svm_set_x2apic_msr_interception(svm, false);
8887 } else {
88+ /*
89+ * Flush the TLB, the guest may have inserted a non-APIC
90+ * mapping into the TLB while AVIC was disabled.
91+ */
92+ kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, &svm->vcpu);
93+
8994 /* For xAVIC and hybrid-xAVIC modes */
9095 vmcb->control.avic_physical_id |= AVIC_MAX_PHYSICAL_ID;
9196 /* Enabling MSR intercept for x2APIC registers */
@@ -241,8 +246,8 @@ static u64 *avic_get_physical_id_entry(struct kvm_vcpu *vcpu, · 149 unmodified lines
241246 u64 *avic_physical_id_table;
242247 struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm);
243248
244− if ((avic_mode == AVIC_MODE_X1 && index > AVIC_MAX_PHYSICAL_ID) ||
245− (avic_mode == AVIC_MODE_X2 && index > X2AVIC_MAX_PHYSICAL_ID))
249+ if ((!x2avic_enabled && index > AVIC_MAX_PHYSICAL_ID) ||
250+ (index > X2AVIC_MAX_PHYSICAL_ID))
246251 return NULL;
247252
248253 avic_physical_id_table = page_address(kvm_svm->avic_physical_id_table_page);
@@ -250,47 +255,14 @@ static u64 *avic_get_physical_id_entry(struct kvm_vcpu *vcpu, · 1 unmodified line
250255 return &avic_physical_id_table[index];
251256 }
252257
253−/*
254− * Note:
255− * AVIC hardware walks the nested page table to check permissions,
256− * but does not use the SPA address specified in the leaf page
257− * table entry since it uses address in the AVIC_BACKING_PAGE pointer
258− * field of the VMCB. Therefore, we set up the
259− * APIC_ACCESS_PAGE_PRIVATE_MEMSLOT (4KB) here.
260− */
261−static int avic_alloc_access_page(struct kvm *kvm)
262−{
263− void __user *ret;
264− int r = 0;
265−
266− mutex_lock(&kvm->slots_lock);
267−
268− if (kvm->arch.apic_access_memslot_enabled)
269− goto out;
270−
271− ret = __x86_set_memory_region(kvm,
272− APIC_ACCESS_PAGE_PRIVATE_MEMSLOT,
273− APIC_DEFAULT_PHYS_BASE,
274− PAGE_SIZE);
275− if (IS_ERR(ret)) {
276− r = PTR_ERR(ret);
277− goto out;
278− }
279−
280− kvm->arch.apic_access_memslot_enabled = true;
281−out:
282− mutex_unlock(&kvm->slots_lock);
283− return r;
284−}
285−
286258 static int avic_init_backing_page(struct kvm_vcpu *vcpu)
287259 {
288260 u64 *entry, new_entry;
289261 int id = vcpu->vcpu_id;
290262 struct vcpu_svm *svm = to_svm(vcpu);
291263
292− if ((avic_mode == AVIC_MODE_X1 && id > AVIC_MAX_PHYSICAL_ID) ||
293− (avic_mode == AVIC_MODE_X2 && id > X2AVIC_MAX_PHYSICAL_ID))
264+ if ((!x2avic_enabled && id > AVIC_MAX_PHYSICAL_ID) ||
265+ (id > X2AVIC_MAX_PHYSICAL_ID))
294266 return -EINVAL;
295267
296268 if (!vcpu->arch.apic->regs)
@@ -299,7 +271,13 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu) · 2 unmodified lines
299271 if (kvm_apicv_activated(vcpu->kvm)) {
300272 int ret;
301273
302− ret = avic_alloc_access_page(vcpu->kvm);
274+ /*
275+ * Note, AVIC hardware walks the nested page table to check
276+ * permissions, but does not use the SPA address specified in
277+ * the leaf SPTE since it uses address in the AVIC_BACKING_PAGE
278+ * pointer field of the VMCB.
279+ */
280+ ret = kvm_alloc_apic_access_page(vcpu->kvm);
303281 if (ret)
304282 return ret;
305283 }
@@ -339,6 +317,60 @@ void avic_ring_doorbell(struct kvm_vcpu *vcpu) · 33 unmodified lines
339317 put_cpu();
340318 }
341319
320+
321+static void avic_kick_vcpu(struct kvm_vcpu *vcpu, u32 icrl)
322+{
323+ vcpu->arch.apic->irr_pending = true;
324+ svm_complete_interrupt_delivery(vcpu,
325+ icrl & APIC_MODE_MASK,
326+ icrl & APIC_INT_LEVELTRIG,
327+ icrl & APIC_VECTOR_MASK);
328+}
329+
330+static void avic_kick_vcpu_by_physical_id(struct kvm *kvm, u32 physical_id,
331+ u32 icrl)
332+{
333+ /*
334+ * KVM inhibits AVIC if any vCPU ID diverges from the vCPUs APIC ID,
335+ * i.e. APIC ID == vCPU ID.
336+ */
337+ struct kvm_vcpu *target_vcpu = kvm_get_vcpu_by_id(kvm, physical_id);
338+
339+ /* Once again, nothing to do if the target vCPU doesn't exist. */
340+ if (unlikely(!target_vcpu))
341+ return;
342+
343+ avic_kick_vcpu(target_vcpu, icrl);
344+}
345+
346+static void avic_kick_vcpu_by_logical_id(struct kvm *kvm, u32 *avic_logical_id_table,
347+ u32 logid_index, u32 icrl)
348+{
349+ u32 physical_id;
350+
351+ if (avic_logical_id_table) {
352+ u32 logid_entry = avic_logical_id_table[logid_index];
353+
354+ /* Nothing to do if the logical destination is invalid. */
355+ if (unlikely(!(logid_entry & AVIC_LOGICAL_ID_ENTRY_VALID_MASK)))
356+ return;
357+
358+ physical_id = logid_entry &
359+ AVIC_LOGICAL_ID_ENTRY_GUEST_PHYSICAL_ID_MASK;
360+ } else {
361+ /*
362+ * For x2APIC, the logical APIC ID is a read-only value that is
363+ * derived from the x2APIC ID, thus the x2APIC ID can be found
364+ * by reversing the calculation (stored in logid_index). Note,
365+ * bits 31:20 of the x2APIC ID aren't propagated to the logical
366+ * ID, but KVM limits the x2APIC ID limited to KVM_MAX_VCPU_IDS.
367+ */
368+ physical_id = logid_index;
369+ }
370+
371+ avic_kick_vcpu_by_physical_id(kvm, physical_id, icrl);
372+}
373+
342374 /*
343375 * A fast-path version of avic_kick_target_vcpus(), which attempts to match
344376 * destination APIC ID to vCPU without looping through all vCPUs.
@@ -346,11 +378,10 @@ void avic_ring_doorbell(struct kvm_vcpu *vcpu) · 1 unmodified line
346378 static int avic_kick_target_vcpus_fast(struct kvm *kvm, struct kvm_lapic *source,
347379 u32 icrl, u32 icrh, u32 index)
348380 {
349− u32 l1_physical_id, dest;
350− struct kvm_vcpu *target_vcpu;
351381 int dest_mode = icrl & APIC_DEST_MASK;
352382 int shorthand = icrl & APIC_SHORT_MASK;
353383 struct kvm_svm *kvm_svm = to_kvm_svm(kvm);
384+ u32 dest;
354385
355386 if (shorthand != APIC_DEST_NOSHORT)
356387 return -EINVAL;
@@ -367,18 +398,18 @@ static int avic_kick_target_vcpus_fast(struct kvm *kvm, struct kvm_lapic *source · 10 unmodified lines
367398 if (!apic_x2apic_mode(source) && dest == APIC_BROADCAST)
368399 return -EINVAL;
369400
370− l1_physical_id = dest;
371−
372− if (WARN_ON_ONCE(l1_physical_id != index))
401+ if (WARN_ON_ONCE(dest != index))
373402 return -EINVAL;
374403
404+ avic_kick_vcpu_by_physical_id(kvm, dest, icrl);
375405 } else {
376− u32 bitmap, cluster;
377− int logid_index;
406+ u32 *avic_logical_id_table;
407+ unsigned long bitmap, i;
408+ u32 cluster;
378409
379410 if (apic_x2apic_mode(source)) {
380411 /* 16 bit dest mask, 16 bit cluster id */
381− bitmap = dest & 0xFFFF0000;
412+ bitmap = dest & 0xFFFF;
382413 cluster = (dest >> 16) << 4;
383414 } else if (kvm_lapic_get_reg(source, APIC_DFR) == APIC_DFR_FLAT) {
384415 /* 8 bit dest mask*/
@@ -390,67 +421,32 @@ static int avic_kick_target_vcpus_fast(struct kvm *kvm, struct kvm_lapic *source · 5 unmodified lines
390421 cluster = (dest >> 4) << 2;
391422 }
392423
424+ /* Nothing to do if there are no destinations in the cluster. */
393425 if (unlikely(!bitmap))
394− /* guest bug: nobody to send the logical interrupt to */
395426 return 0;
396427
397− if (!is_power_of_2(bitmap))
398− /* multiple logical destinations, use slow path */
399− return -EINVAL;
400−
401− logid_index = cluster + __ffs(bitmap);
402−
403− if (!apic_x2apic_mode(source)) {
404− u32 *avic_logical_id_table =
405− page_address(kvm_svm->avic_logical_id_table_page);
406−
407− u32 logid_entry = avic_logical_id_table[logid_index];
408−
409− if (WARN_ON_ONCE(index != logid_index))
410− return -EINVAL;
411−
412− /* guest bug: non existing/reserved logical destination */
413− if (unlikely(!(logid_entry & AVIC_LOGICAL_ID_ENTRY_VALID_MASK)))
414− return 0;
415−
416− l1_physical_id = logid_entry &
417− AVIC_LOGICAL_ID_ENTRY_GUEST_PHYSICAL_ID_MASK;
418− } else {
419− /*
420− * For x2APIC logical mode, cannot leverage the index.
421− * Instead, calculate physical ID from logical ID in ICRH.
422− */
423− int cluster = (icrh & 0xffff0000) >> 16;
424− int apic = ffs(icrh & 0xffff) - 1;
425−
426− /*
427− * If the x2APIC logical ID sub-field (i.e. icrh[15:0])
428− * contains anything but a single bit, we cannot use the
429− * fast path, because it is limited to a single vCPU.
430− */
431− if (apic < 0 || icrh != (1 << apic))
432− return -EINVAL;
428+ if (apic_x2apic_mode(source))
429+ avic_logical_id_table = NULL;
430+ else
431+ avic_logical_id_table = page_address(kvm_svm->avic_logical_id_table_page);
433432
434− l1_physical_id = (cluster << 4) + apic;
435− }
433+ /*
434+ * AVIC is inhibited if vCPUs aren't mapped 1:1 with logical
435+ * IDs, thus each bit in the destination is guaranteed to map
436+ * to at most one vCPU.
437+ */
438+ for_each_set_bit(i, &bitmap, 16)
439+ avic_kick_vcpu_by_logical_id(kvm, avic_logical_id_table,
440+ cluster + i, icrl);
436441 }
437442
438− target_vcpu = kvm_get_vcpu_by_id(kvm, l1_physical_id);
439− if (unlikely(!target_vcpu))
440− /* guest bug: non existing vCPU is a target of this IPI*/
441− return 0;
442−
443− target_vcpu->arch.apic->irr_pending = true;
444− svm_complete_interrupt_delivery(target_vcpu,
445− icrl & APIC_MODE_MASK,
446− icrl & APIC_INT_LEVELTRIG,
447− icrl & APIC_VECTOR_MASK);
448443 return 0;
449444 }
450445
451446 static void avic_kick_target_vcpus(struct kvm *kvm, struct kvm_lapic *source,
452447 u32 icrl, u32 icrh, u32 index)
453448 {
449+ u32 dest = apic_x2apic_mode(source) ? icrh : GET_XAPIC_DEST_FIELD(icrh);
454450 unsigned long i;
455451 struct kvm_vcpu *vcpu;
456452
@@ -466,21 +462,9 @@ static void avic_kick_target_vcpus(struct kvm *kvm, struct kvm_lapic *source, · 9 unmodified lines
466462 * since entered the guest will have processed pending IRQs at VMRUN.
467463 */
468464 kvm_for_each_vcpu(i, vcpu, kvm) {
469− u32 dest;
470−
471− if (apic_x2apic_mode(vcpu->arch.apic))
472− dest = icrh;
473− else
474− dest = GET_XAPIC_DEST_FIELD(icrh);
475−
476465 if (kvm_apic_match_dest(vcpu, source, icrl & APIC_SHORT_MASK,
477− dest, icrl & APIC_DEST_MASK)) {
478− vcpu->arch.apic->irr_pending = true;
479− svm_complete_interrupt_delivery(vcpu,
480− icrl & APIC_MODE_MASK,
481− icrl & APIC_INT_LEVELTRIG,
482− icrl & APIC_VECTOR_MASK);
483− }
466+ dest, icrl & APIC_DEST_MASK))
467+ avic_kick_vcpu(vcpu, icrl);
484468 }
485469 }
486470
@@ -496,14 +480,18 @@ int avic_incomplete_ipi_interception(struct kvm_vcpu *vcpu) · 9 unmodified lines
496480 trace_kvm_avic_incomplete_ipi(vcpu->vcpu_id, icrh, icrl, id, index);
497481
498482 switch (id) {
483+ case AVIC_IPI_FAILURE_INVALID_TARGET:
499484 case AVIC_IPI_FAILURE_INVALID_INT_TYPE:
500485 /*
501486 * Emulate IPIs that are not handled by AVIC hardware, which
502− * only virtualizes Fixed, Edge-Triggered INTRs. The exit is
503− * a trap, e.g. ICR holds the correct value and RIP has been
504− * advanced, KVM is responsible only for emulating the IPI.
505− * Sadly, hardware may sometimes leave the BUSY flag set, in
506− * which case KVM needs to emulate the ICR write as well in
487+ * only virtualizes Fixed, Edge-Triggered INTRs, and falls over
488+ * if _any_ targets are invalid, e.g. if the logical mode mask
489+ * is a superset of running vCPUs.
490+ *
491+ * The exit is a trap, e.g. ICR holds the correct value and RIP
492+ * has been advanced, KVM is responsible only for emulating the
493+ * IPI. Sadly, hardware may sometimes leave the BUSY flag set,
494+ * in which case KVM needs to emulate the ICR write as well in
507495 * order to clear the BUSY flag.
508496 */
509497 if (icrl & APIC_ICR_BUSY)
@@ -519,8 +507,6 @@ int avic_incomplete_ipi_interception(struct kvm_vcpu *vcpu) · 9 unmodified lines
519507 */
520508 avic_kick_target_vcpus(vcpu->kvm, apic, icrl, icrh, index);
521509 break;
522− case AVIC_IPI_FAILURE_INVALID_TARGET:
523− break;
524510 case AVIC_IPI_FAILURE_INVALID_BACKING_PAGE:
525511 WARN_ONCE(1, "Invalid backing page\n");
526512 break;
@@ -541,33 +527,33 @@ unsigned long avic_vcpu_get_apicv_inhibit_reasons(struct kvm_vcpu *vcpu) · 14 unmodified lines
541527 static u32 *avic_get_logical_id_entry(struct kvm_vcpu *vcpu, u32 ldr, bool flat)
542528 {
543529 struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm);
544− int index;
545530 u32 *logical_apic_id_table;
546− int dlid = GET_APIC_LOGICAL_ID(ldr);
547−
548− if (!dlid)
549− return NULL;
531+ u32 cluster, index;
550532
551− if (flat) { /* flat */
552− index = ffs(dlid) - 1;
553− if (index > 7)
554− return NULL;
555− } else { /* cluster */
556− int cluster = (dlid & 0xf0) >> 4;
557− int apic = ffs(dlid & 0x0f) - 1;
533+ ldr = GET_APIC_LOGICAL_ID(ldr);
558534
559− if ((apic < 0) || (apic > 7) ||
560− (cluster >= 0xf))
535+ if (flat) {
536+ cluster = 0;
537+ } else {
538+ cluster = (ldr >> 4);
539+ if (cluster >= 0xf)
561540 return NULL;
562− index = (cluster << 2) + apic;
541+ ldr &= 0xf;
563542 }
543+ if (!ldr || !is_power_of_2(ldr))
544+ return NULL;
545+
546+ index = __ffs(ldr);
547+ if (WARN_ON_ONCE(index > 7))
548+ return NULL;
549+ index += (cluster << 2);
564550
565551 logical_apic_id_table = (u32 *) page_address(kvm_svm->avic_logical_id_table_page);
566552
567553 return &logical_apic_id_table[index];
568554 }
569555
570−static int avic_ldr_write(struct kvm_vcpu *vcpu, u8 g_physical_id, u32 ldr)
556+static void avic_ldr_write(struct kvm_vcpu *vcpu, u8 g_physical_id, u32 ldr)
571557 {
572558 bool flat;
573559 u32 *entry, new_entry;
@@ -575,15 +561,13 @@ static int avic_ldr_write(struct kvm_vcpu *vcpu, u8 g_physical_id, u32 ldr) · 1 unmodified line
575561 flat = kvm_lapic_get_reg(vcpu->arch.apic, APIC_DFR) == APIC_DFR_FLAT;
576562 entry = avic_get_logical_id_entry(vcpu, ldr, flat);
577563 if (!entry)
578− return -EINVAL;
564+ return;
579565
580566 new_entry = READ_ONCE(*entry);
581567 new_entry &= ~AVIC_LOGICAL_ID_ENTRY_GUEST_PHYSICAL_ID_MASK;
582568 new_entry |= (g_physical_id & AVIC_LOGICAL_ID_ENTRY_GUEST_PHYSICAL_ID_MASK);
583569 new_entry |= AVIC_LOGICAL_ID_ENTRY_VALID_MASK;
584570 WRITE_ONCE(*entry, new_entry);
585−
586− return 0;
587571 }
588572
589573 static void avic_invalidate_logical_id_entry(struct kvm_vcpu *vcpu)
@@ -601,29 +585,23 @@ static void avic_invalidate_logical_id_entry(struct kvm_vcpu *vcpu) · 11 unmodified lines
601585 clear_bit(AVIC_LOGICAL_ID_ENTRY_VALID_BIT, (unsigned long *)entry);
602586 }
603587
604−static int avic_handle_ldr_update(struct kvm_vcpu *vcpu)
588+static void avic_handle_ldr_update(struct kvm_vcpu *vcpu)
605589 {
606− int ret = 0;
607590 struct vcpu_svm *svm = to_svm(vcpu);
608591 u32 ldr = kvm_lapic_get_reg(vcpu->arch.apic, APIC_LDR);
609592 u32 id = kvm_xapic_id(vcpu->arch.apic);
610593
611594 /* AVIC does not support LDR update for x2APIC */
612595 if (apic_x2apic_mode(vcpu->arch.apic))
613− return 0;
596+ return;
614597
615598 if (ldr == svm->ldr_reg)
616− return 0;
599+ return;
617600
618601 avic_invalidate_logical_id_entry(vcpu);
619602
620− if (ldr)
621− ret = avic_ldr_write(vcpu, id, ldr);
622−
623− if (!ret)
624− svm->ldr_reg = ldr;
625−
626− return ret;
603+ svm->ldr_reg = ldr;
604+ avic_ldr_write(vcpu, id, ldr);
627605 }
628606
629607 static void avic_handle_dfr_update(struct kvm_vcpu *vcpu)
@@ -645,12 +623,14 @@ static int avic_unaccel_trap_write(struct kvm_vcpu *vcpu) · 15 unmodified lines
645623
646624 switch (offset) {
647625 case APIC_LDR:
648− if (avic_handle_ldr_update(vcpu))
649− return 0;
626+ avic_handle_ldr_update(vcpu);
650627 break;
651628 case APIC_DFR:
652629 avic_handle_dfr_update(vcpu);
653630 break;
631+ case APIC_RRR:
632+ /* Ignore writes to Read Remote Data, it's read-only. */
633+ return 1;
654634 default:
655635 break;
656636 }
@@ -739,18 +719,6 @@ void avic_apicv_post_state_restore(struct kvm_vcpu *vcpu) · 82 unmodified lines
739719 avic_handle_ldr_update(vcpu);
740720 }
741721
742−void avic_set_virtual_apic_mode(struct kvm_vcpu *vcpu)
743−{
744− if (!lapic_in_kernel(vcpu) || avic_mode == AVIC_MODE_NONE)
745− return;
746−
747− if (kvm_get_apic_mode(vcpu) == LAPIC_MODE_INVALID) {
748− WARN_ONCE(true, "Invalid local APIC state (vcpu_id=%d)", vcpu->vcpu_id);
749− return;
750− }
751− avic_refresh_apicv_exec_ctrl(vcpu);
752−}
753−
754722 static int avic_set_pi_irte_mode(struct kvm_vcpu *vcpu, bool activate)
755723 {
756724 int ret = 0;
@@ -995,23 +963,6 @@ out: · 238 unmodified lines
995963 return ret;
996964 }
997965
998−bool avic_check_apicv_inhibit_reasons(enum kvm_apicv_inhibit reason)
999−{
1000− ulong supported = BIT(APICV_INHIBIT_REASON_DISABLE) |
1001− BIT(APICV_INHIBIT_REASON_ABSENT) |
1002− BIT(APICV_INHIBIT_REASON_HYPERV) |
1003− BIT(APICV_INHIBIT_REASON_NESTED) |
1004− BIT(APICV_INHIBIT_REASON_IRQWIN) |
1005− BIT(APICV_INHIBIT_REASON_PIT_REINJ) |
1006− BIT(APICV_INHIBIT_REASON_BLOCKIRQ) |
1007− BIT(APICV_INHIBIT_REASON_SEV) |
1008− BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) |
1009− BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED);
1010−
1011− return supported & BIT(reason);
1012−}
1013−
1014−
1015966 static inline int
1016967 avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu, bool r)
1017968 {
@@ -1064,6 +1015,7 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu) · 46 unmodified lines
10641015 return;
10651016
10661017 entry = READ_ONCE(*(svm->avic_physical_id_cache));
1018+ WARN_ON_ONCE(entry & AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK);
10671019
10681020 entry &= ~AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK;
10691021 entry |= (h_physical_id & AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK);
@@ -1092,17 +1044,15 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu) · 22 unmodified lines
10921044 WRITE_ONCE(*(svm->avic_physical_id_cache), entry);
10931045 }
10941046
1095−
1096−void avic_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu)
1047+void avic_refresh_virtual_apic_mode(struct kvm_vcpu *vcpu)
10971048 {
10981049 struct vcpu_svm *svm = to_svm(vcpu);
10991050 struct vmcb *vmcb = svm->vmcb01.ptr;
1100− bool activated = kvm_vcpu_apicv_active(vcpu);
11011051
1102− if (!enable_apicv)
1052+ if (!lapic_in_kernel(vcpu) || !enable_apicv)
11031053 return;
11041054
1105− if (activated) {
1055+ if (kvm_vcpu_apicv_active(vcpu)) {
11061056 /**
11071057 * During AVIC temporary deactivation, guest could update
11081058 * APIC ID, DFR and LDR registers, which would not be trapped
@@ -1116,6 +1066,16 @@ void avic_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu) · 7 unmodified lines
11161066 avic_deactivate_vmcb(svm);
11171067 }
11181068 vmcb_mark_dirty(vmcb, VMCB_AVIC);
1069+}
1070+
1071+void avic_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu)
1072+{
1073+ bool activated = kvm_vcpu_apicv_active(vcpu);
1074+
1075+ if (!enable_apicv)
1076+ return;
1077+
1078+ avic_refresh_virtual_apic_mode(vcpu);
11191079
11201080 if (activated)
11211081 avic_vcpu_load(vcpu, vcpu->cpu);
@@ -1160,37 +1120,37 @@ void avic_vcpu_unblocking(struct kvm_vcpu *vcpu) · 38 unmodified lines
11601120 * - Hypervisor can support both xAVIC and x2AVIC in the same guest.
11611121 * - The mode can be switched at run-time.
11621122 */
1163−bool avic_hardware_setup(struct kvm_x86_ops *x86_ops)
1123+bool avic_hardware_setup(void)
11641124 {
11651125 if (!npt_enabled)
11661126 return false;
11671127
1128+ /* AVIC is a prerequisite for x2AVIC. */
1129+ if (!boot_cpu_has(X86_FEATURE_AVIC) && !force_avic) {
1130+ if (boot_cpu_has(X86_FEATURE_X2AVIC)) {
1131+ pr_warn(FW_BUG "Cannot support x2AVIC due to AVIC is disabled");
1132+ pr_warn(FW_BUG "Try enable AVIC using force_avic option");
1133+ }
1134+ return false;
1135+ }
1136+
11681137 if (boot_cpu_has(X86_FEATURE_AVIC)) {
1169− avic_mode = AVIC_MODE_X1;
11701138 pr_info("AVIC enabled\n");
11711139 } else if (force_avic) {
11721140 /*
11731141 * Some older systems does not advertise AVIC support.
11741142 * See Revision Guide for specific AMD processor for more detail.
11751143 */
1176− avic_mode = AVIC_MODE_X1;
11771144 pr_warn("AVIC is not supported in CPUID but force enabled");
11781145 pr_warn("Your system might crash and burn");
11791146 }
11801147
11811148 /* AVIC is a prerequisite for x2AVIC. */
1182− if (boot_cpu_has(X86_FEATURE_X2AVIC)) {
1183− if (avic_mode == AVIC_MODE_X1) {
1184− avic_mode = AVIC_MODE_X2;
1185− pr_info("x2AVIC enabled\n");
1186− } else {
1187− pr_warn(FW_BUG "Cannot support x2AVIC due to AVIC is disabled");
1188− pr_warn(FW_BUG "Try enable AVIC using force_avic option");
1189− }
1190− }
1149+ x2avic_enabled = boot_cpu_has(X86_FEATURE_X2AVIC);
1150+ if (x2avic_enabled)
1151+ pr_info("x2AVIC enabled\n");
11911152
1192− if (avic_mode != AVIC_MODE_NONE)
1193− amd_iommu_register_ga_log_notifier(&avic_ga_log_notifier);
1153+ amd_iommu_register_ga_log_notifier(&avic_ga_log_notifier);
11941154
1195− return !!avic_mode;
1155+ return true;
11961156 }
arch/x86/kvm/svm/nested.c+2−3
@@ -12,7 +12,7 @@ · 11 unmodified lines
1212 * Avi Kivity <avi@qumranet.com>
1313 */
1414
15−#define pr_fmt(fmt) "SVM: " fmt
15+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
1616
1717 #include <linux/kvm_types.h>
1818 #include <linux/kvm_host.h>
@@ -1008,7 +1008,6 @@ int nested_svm_vmexit(struct vcpu_svm *svm) · 989 unmodified lines
10081008 vmcb12->control.next_rip = vmcb02->control.next_rip;
10091009
10101010 vmcb12->control.int_ctl = svm->nested.ctl.int_ctl;
1011− vmcb12->control.tlb_ctl = svm->nested.ctl.tlb_ctl;
10121011 vmcb12->control.event_inj = svm->nested.ctl.event_inj;
10131012 vmcb12->control.event_inj_err = svm->nested.ctl.event_inj_err;
10141013
@@ -1104,7 +1103,7 @@ int nested_svm_vmexit(struct vcpu_svm *svm) · 89 unmodified lines
11041103 * to benefit from it right away.
11051104 */
11061105 if (kvm_apicv_activated(vcpu->kvm))
1107− kvm_vcpu_update_apicv(vcpu);
1106+ __kvm_vcpu_update_apicv(vcpu);
11081107
11091108 return 0;
11101109 }
arch/x86/kvm/svm/pmu.c+4−0
@@ -9,6 +9,8 @@ · 8 unmodified lines
99 *
1010 * Implementation is based on pmu_intel.c file
1111 */
12+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
13+
1214 #include <linux/types.h>
1315 #include <linux/kvm_host.h>
1416 #include <linux/perf_event.h>
@@ -229,4 +231,6 @@ struct kvm_pmu_ops amd_pmu_ops __initdata = { · 214 unmodified lines
229231 .refresh = amd_pmu_refresh,
230232 .init = amd_pmu_init,
231233 .reset = amd_pmu_reset,
234+ .EVENTSEL_EVENT = AMD64_EVENTSEL_EVENT,
235+ .MAX_NR_GP_COUNTERS = KVM_AMD_PMC_MAX_GENERIC,
232236 };
arch/x86/kvm/svm/sev.c+4−3
@@ -6,6 +6,7 @@ · 5 unmodified lines
66 *
77 * Copyright 2010 Red Hat, Inc. and/or its affiliates.
88 */
9+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
910
1011 #include <linux/kvm_types.h>
1112 #include <linux/kvm_host.h>
@@ -812,7 +813,7 @@ static int __sev_dbg_decrypt_user(struct kvm *kvm, unsigned long paddr, · 800 unmodified lines
812813 if (!IS_ALIGNED(dst_paddr, 16) ||
813814 !IS_ALIGNED(paddr, 16) ||
814815 !IS_ALIGNED(size, 16)) {
815− tpage = (void *)alloc_page(GFP_KERNEL | __GFP_ZERO);
816+ tpage = (void *)alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO);
816817 if (!tpage)
817818 return -ENOMEM;
818819
@@ -1293,7 +1294,7 @@ static int sev_send_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) · 474 unmodified lines
12931294
12941295 /* Check if we are crossing the page boundary */
12951296 offset = params.guest_uaddr & (PAGE_SIZE - 1);
1296− if ((params.guest_len + offset > PAGE_SIZE))
1297+ if (params.guest_len > PAGE_SIZE || (params.guest_len + offset) > PAGE_SIZE)
12971298 return -EINVAL;
12981299
12991300 /* Pin guest memory */
@@ -1473,7 +1474,7 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) · 173 unmodified lines
14731474
14741475 /* Check if we are crossing the page boundary */
14751476 offset = params.guest_uaddr & (PAGE_SIZE - 1);
1476− if ((params.guest_len + offset > PAGE_SIZE))
1477+ if (params.guest_len > PAGE_SIZE || (params.guest_len + offset) > PAGE_SIZE)
14771478 return -EINVAL;
14781479
14791480 hdr = psp_copy_user_blob(params.hdr_uaddr, params.hdr_len);
arch/x86/kvm/svm/svm.c+75−56
@@ -1,4 +1,4 @@
1−#define pr_fmt(fmt) "SVM: " fmt
1+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
22
33 #include <linux/kvm_host.h>
44
@@ -519,21 +519,37 @@ static void svm_init_osvw(struct kvm_vcpu *vcpu) · 514 unmodified lines
519519 vcpu->arch.osvw.status |= 1;
520520 }
521521
522−static int has_svm(void)
522+static bool kvm_is_svm_supported(void)
523523 {
524+ int cpu = raw_smp_processor_id();
524525 const char *msg;
526+ u64 vm_cr;
525527
526528 if (!cpu_has_svm(&msg)) {
527− printk(KERN_INFO "has_svm: %s\n", msg);
528− return 0;
529+ pr_err("SVM not supported by CPU %d, %s\n", cpu, msg);
530+ return false;
529531 }
530532
531533 if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) {
532534 pr_info("KVM is unsupported when running as an SEV guest\n");
533− return 0;
535+ return false;
534536 }
535537
536− return 1;
538+ rdmsrl(MSR_VM_CR, vm_cr);
539+ if (vm_cr & (1 << SVM_VM_CR_SVM_DISABLE)) {
540+ pr_err("SVM disabled (by BIOS) in MSR_VM_CR on CPU %d\n", cpu);
541+ return false;
542+ }
543+
544+ return true;
545+}
546+
547+static int svm_check_processor_compat(void)
548+{
549+ if (!kvm_is_svm_supported())
550+ return -EIO;
551+
552+ return 0;
537553 }
538554
539555 void __svm_write_tsc_multiplier(u64 multiplier)
@@ -572,10 +588,6 @@ static int svm_hardware_enable(void) · 32 unmodified lines
572588 if (efer & EFER_SVME)
573589 return -EBUSY;
574590
575− if (!has_svm()) {
576− pr_err("%s: err EOPNOTSUPP on %d\n", __func__, me);
577− return -EINVAL;
578− }
579591 sd = per_cpu_ptr(&svm_data, me);
580592 sd->asid_generation = 1;
581593 sd->max_asid = cpuid_ebx(SVM_CPUID_FUNC) - 1;
@@ -813,7 +825,7 @@ void svm_set_x2apic_msr_interception(struct vcpu_svm *svm, bool intercept) · 231 unmodified lines
813825 if (intercept == svm->x2avic_msrs_intercepted)
814826 return;
815827
816− if (avic_mode != AVIC_MODE_X2 ||
828+ if (!x2avic_enabled ||
817829 !apic_x2apic_mode(svm->vcpu.arch.apic))
818830 return;
819831
@@ -1326,6 +1338,9 @@ static void __svm_vcpu_reset(struct kvm_vcpu *vcpu) · 506 unmodified lines
13261338 vcpu->arch.microcode_version = 0x01000065;
13271339 svm->tsc_ratio_msr = kvm_caps.default_tsc_scaling_ratio;
13281340
1341+ svm->nmi_masked = false;
1342+ svm->awaiting_iret_completion = false;
1343+
13291344 if (sev_es_guest(vcpu->kvm))
13301345 sev_es_vcpu_reset(svm);
13311346 }
@@ -2076,7 +2091,7 @@ static void svm_handle_mce(struct kvm_vcpu *vcpu) · 744 unmodified lines
20762091 * Erratum 383 triggered. Guest state is corrupt so kill the
20772092 * guest.
20782093 */
2079− pr_err("KVM: Guest triggered AMD Erratum 383\n");
2094+ pr_err("Guest triggered AMD Erratum 383\n");
20802095
20812096 kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
20822097
@@ -2470,7 +2485,7 @@ static int iret_interception(struct kvm_vcpu *vcpu) · 387 unmodified lines
24702485 struct vcpu_svm *svm = to_svm(vcpu);
24712486
24722487 ++vcpu->stat.nmi_window_exits;
2473− vcpu->arch.hflags |= HF_IRET_MASK;
2488+ svm->awaiting_iret_completion = true;
24742489 if (!sev_es_guest(vcpu->kvm)) {
24752490 svm_clr_intercept(svm, INTERCEPT_IRET);
24762491 svm->nmi_iret_rip = kvm_rip_read(vcpu);
@@ -3003,8 +3018,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) · 526 unmodified lines
30033018 break;
30043019 case MSR_IA32_DEBUGCTLMSR:
30053020 if (!lbrv) {
3006− vcpu_unimpl(vcpu, "%s: MSR_IA32_DEBUGCTL 0x%llx, nop\n",
3007− __func__, data);
3021+ kvm_pr_unimpl_wrmsr(vcpu, ecx, data);
30083022 break;
30093023 }
30103024 if (data & DEBUGCTL_RESERVED_BITS)
@@ -3033,7 +3047,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) · 22 unmodified lines
30333047 case MSR_VM_CR:
30343048 return svm_set_vm_cr(vcpu, data);
30353049 case MSR_VM_IGNNE:
3036− vcpu_unimpl(vcpu, "unimplemented wrmsr: 0x%x data 0x%llx\n", ecx, data);
3050+ kvm_pr_unimpl_wrmsr(vcpu, ecx, data);
30373051 break;
30383052 case MSR_AMD64_DE_CFG: {
30393053 struct kvm_msr_entry msr_entry;
@@ -3466,7 +3480,7 @@ static void svm_inject_nmi(struct kvm_vcpu *vcpu) · 426 unmodified lines
34663480 if (svm->nmi_l1_to_l2)
34673481 return;
34683482
3469− vcpu->arch.hflags |= HF_NMI_MASK;
3483+ svm->nmi_masked = true;
34703484 if (!sev_es_guest(vcpu->kvm))
34713485 svm_set_intercept(svm, INTERCEPT_IRET);
34723486 ++vcpu->stat.nmi_injections;
@@ -3571,7 +3585,6 @@ bool svm_nmi_blocked(struct kvm_vcpu *vcpu) · 98 unmodified lines
35713585 {
35723586 struct vcpu_svm *svm = to_svm(vcpu);
35733587 struct vmcb *vmcb = svm->vmcb;
3574− bool ret;
35753588
35763589 if (!gif_set(svm))
35773590 return true;
@@ -3579,10 +3592,8 @@ bool svm_nmi_blocked(struct kvm_vcpu *vcpu) · 1 unmodified line
35793592 if (is_guest_mode(vcpu) && nested_exit_on_nmi(svm))
35803593 return false;
35813594
3582− ret = (vmcb->control.int_state & SVM_INTERRUPT_SHADOW_MASK) ||
3583− (vcpu->arch.hflags & HF_NMI_MASK);
3584−
3585− return ret;
3595+ return (vmcb->control.int_state & SVM_INTERRUPT_SHADOW_MASK) ||
3596+ svm->nmi_masked;
35863597 }
35873598
35883599 static int svm_nmi_allowed(struct kvm_vcpu *vcpu, bool for_injection)
@@ -3602,7 +3613,7 @@ static int svm_nmi_allowed(struct kvm_vcpu *vcpu, bool for_injection) · 13 unmodified lines
36023613
36033614 static bool svm_get_nmi_mask(struct kvm_vcpu *vcpu)
36043615 {
3605− return !!(vcpu->arch.hflags & HF_NMI_MASK);
3616+ return to_svm(vcpu)->nmi_masked;
36063617 }
36073618
36083619 static void svm_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked)
@@ -3610,11 +3621,11 @@ static void svm_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked) · 1 unmodified line
36103621 struct vcpu_svm *svm = to_svm(vcpu);
36113622
36123623 if (masked) {
3613− vcpu->arch.hflags |= HF_NMI_MASK;
3624+ svm->nmi_masked = true;
36143625 if (!sev_es_guest(vcpu->kvm))
36153626 svm_set_intercept(svm, INTERCEPT_IRET);
36163627 } else {
3617− vcpu->arch.hflags &= ~HF_NMI_MASK;
3628+ svm->nmi_masked = false;
36183629 if (!sev_es_guest(vcpu->kvm))
36193630 svm_clr_intercept(svm, INTERCEPT_IRET);
36203631 }
@@ -3700,7 +3711,7 @@ static void svm_enable_nmi_window(struct kvm_vcpu *vcpu) · 79 unmodified lines
37003711 {
37013712 struct vcpu_svm *svm = to_svm(vcpu);
37023713
3703− if ((vcpu->arch.hflags & (HF_NMI_MASK | HF_IRET_MASK)) == HF_NMI_MASK)
3714+ if (svm->nmi_masked && !svm->awaiting_iret_completion)
37043715 return; /* IRET will cause a vm exit */
37053716
37063717 if (!gif_set(svm)) {
@@ -3824,10 +3835,11 @@ static void svm_complete_interrupts(struct kvm_vcpu *vcpu) · 117 unmodified lines
38243835 * If we've made progress since setting HF_IRET_MASK, we've
38253836 * executed an IRET and can allow NMI injection.
38263837 */
3827− if ((vcpu->arch.hflags & HF_IRET_MASK) &&
3838+ if (svm->awaiting_iret_completion &&
38283839 (sev_es_guest(vcpu->kvm) ||
38293840 kvm_rip_read(vcpu) != svm->nmi_iret_rip)) {
3830− vcpu->arch.hflags &= ~(HF_NMI_MASK | HF_IRET_MASK);
3841+ svm->awaiting_iret_completion = false;
3842+ svm->nmi_masked = false;
38313843 kvm_make_request(KVM_REQ_EVENT, vcpu);
38323844 }
38333845
@@ -4076,17 +4088,6 @@ static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, · 242 unmodified lines
40764088 vmcb_mark_dirty(svm->vmcb, VMCB_CR);
40774089 }
40784090
4079−static int is_disabled(void)
4080−{
4081− u64 vm_cr;
4082−
4083− rdmsrl(MSR_VM_CR, vm_cr);
4084− if (vm_cr & (1 << SVM_VM_CR_SVM_DISABLE))
4085− return 1;
4086−
4087− return 0;
4088−}
4089−
40904091 static void
40914092 svm_patch_hypercall(struct kvm_vcpu *vcpu, unsigned char *hypercall)
40924093 {
@@ -4098,11 +4099,6 @@ svm_patch_hypercall(struct kvm_vcpu *vcpu, unsigned char *hypercall) · 5 unmodified lines
40984099 hypercall[2] = 0xd9;
40994100 }
41004101
4101−static int __init svm_check_processor_compat(void)
4102−{
4103− return 0;
4104−}
4105−
41064102 /*
41074103 * The kvm parameter can be NULL (module initialization, or invocation before
41084104 * VM creation). Be sure to check the kvm parameter before using it.
@@ -4629,7 +4625,7 @@ static bool svm_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type, · 520 unmodified lines
46294625 smap = cr4 & X86_CR4_SMAP;
46304626 is_user = svm_get_cpl(vcpu) == 3;
46314627 if (smap && (!smep || is_user)) {
4632− pr_err_ratelimited("KVM: SEV Guest triggered AMD Erratum 1096\n");
4628+ pr_err_ratelimited("SEV Guest triggered AMD Erratum 1096\n");
46334629
46344630 /*
46354631 * If the fault occurred in userspace, arbitrarily inject #GP
@@ -4701,7 +4697,9 @@ static int svm_vm_init(struct kvm *kvm) · 65 unmodified lines
47014697 }
47024698
47034699 static struct kvm_x86_ops svm_x86_ops __initdata = {
4704− .name = "kvm_amd",
4700+ .name = KBUILD_MODNAME,
4701+
4702+ .check_processor_compatibility = svm_check_processor_compat,
47054703
47064704 .hardware_unsetup = svm_hardware_unsetup,
47074705 .hardware_enable = svm_hardware_enable,
@@ -4771,10 +4769,10 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { · 63 unmodified lines
47714769 .enable_nmi_window = svm_enable_nmi_window,
47724770 .enable_irq_window = svm_enable_irq_window,
47734771 .update_cr8_intercept = svm_update_cr8_intercept,
4774− .set_virtual_apic_mode = avic_set_virtual_apic_mode,
4772+ .set_virtual_apic_mode = avic_refresh_virtual_apic_mode,
47754773 .refresh_apicv_exec_ctrl = avic_refresh_apicv_exec_ctrl,
4776− .check_apicv_inhibit_reasons = avic_check_apicv_inhibit_reasons,
47774774 .apicv_post_state_restore = avic_apicv_post_state_restore,
4775+ .required_apicv_inhibits = AVIC_REQUIRED_APICV_INHIBITS,
47784776
47794777 .get_exit_info = svm_get_exit_info,
47804778
@@ -4981,7 +4979,7 @@ static __init int svm_hardware_setup(void) · 200 unmodified lines
49814979 }
49824980
49834981 if (nested) {
4984− printk(KERN_INFO "kvm: Nested Virtualization enabled\n");
4982+ pr_info("Nested Virtualization enabled\n");
49854983 kvm_enable_efer_bits(EFER_SVME | EFER_LMSLE);
49864984 }
49874985
@@ -4999,7 +4997,7 @@ static __init int svm_hardware_setup(void) · 11 unmodified lines
49994997 /* Force VM NPT level equal to the host's paging level */
50004998 kvm_configure_mmu(npt_enabled, get_npt_level(),
50014999 get_npt_level(), PG_LEVEL_1G);
5002− pr_info("kvm: Nested Paging %sabled\n", npt_enabled ? "en" : "dis");
5000+ pr_info("Nested Paging %sabled\n", npt_enabled ? "en" : "dis");
50035001
50045002 /* Setup shadow_me_value and shadow_me_mask */
50055003 kvm_mmu_set_me_spte_mask(sme_me_mask, sme_me_mask);
@@ -5025,12 +5023,14 @@ static __init int svm_hardware_setup(void) · 19 unmodified lines
50255023 nrips = false;
50265024 }
50275025
5028− enable_apicv = avic = avic && avic_hardware_setup(&svm_x86_ops);
5026+ enable_apicv = avic = avic && avic_hardware_setup();
50295027
50305028 if (!enable_apicv) {
50315029 svm_x86_ops.vcpu_blocking = NULL;
50325030 svm_x86_ops.vcpu_unblocking = NULL;
50335031 svm_x86_ops.vcpu_get_apicv_inhibit_reasons = NULL;
5032+ } else if (!x2avic_enabled) {
5033+ svm_x86_ops.allow_apicv_in_x2apic_without_x2apic_virtualization = true;
50345034 }
50355035
50365036 if (vls) {
@@ -5089,10 +5089,7 @@ err: · 52 unmodified lines
50895089
50905090
50915091 static struct kvm_x86_init_ops svm_init_ops __initdata = {
5092− .cpu_has_kvm_support = has_svm,
5093− .disabled_by_bios = is_disabled,
50945092 .hardware_setup = svm_hardware_setup,
5095− .check_processor_compatibility = svm_check_processor_compat,
50965093
50975094 .runtime_ops = &svm_x86_ops,
50985095 .pmu_ops = &amd_pmu_ops,
@@ -5100,15 +5097,37 @@ static struct kvm_x86_init_ops svm_init_ops __initdata = { · 1 unmodified line
51005097
51015098 static int __init svm_init(void)
51025099 {
5100+ int r;
5101+
51035102 __unused_size_checks();
51045103
5105− return kvm_init(&svm_init_ops, sizeof(struct vcpu_svm),
5106− __alignof__(struct vcpu_svm), THIS_MODULE);
5104+ if (!kvm_is_svm_supported())
5105+ return -EOPNOTSUPP;
5106+
5107+ r = kvm_x86_vendor_init(&svm_init_ops);
5108+ if (r)
5109+ return r;
5110+
5111+ /*
5112+ * Common KVM initialization _must_ come last, after this, /dev/kvm is
5113+ * exposed to userspace!
5114+ */
5115+ r = kvm_init(sizeof(struct vcpu_svm), __alignof__(struct vcpu_svm),
5116+ THIS_MODULE);
5117+ if (r)
5118+ goto err_kvm_init;
5119+
5120+ return 0;
5121+
5122+err_kvm_init:
5123+ kvm_x86_vendor_exit();
5124+ return r;
51075125 }
51085126
51095127 static void __exit svm_exit(void)
51105128 {
51115129 kvm_exit();
5130+ kvm_x86_vendor_exit();
51125131 }
51135132
51145133 module_init(svm_init)
arch/x86/kvm/svm/svm.h+43−15
@@ -35,14 +35,7 @@ extern u32 msrpm_offsets[MSRPM_OFFSETS] __read_mostly; · 34 unmodified lines
3535 extern bool npt_enabled;
3636 extern int vgif;
3737 extern bool intercept_smi;
38−
39−enum avic_modes {
40− AVIC_MODE_NONE = 0,
41− AVIC_MODE_X1,
42− AVIC_MODE_X2,
43−};
44−
45−extern enum avic_modes avic_mode;
38+extern bool x2avic_enabled;
4639
4740 /*
4841 * Clean bits in VMCB.
@@ -237,8 +230,26 @@ struct vcpu_svm { · 188 unmodified lines
237230
238231 struct svm_nested_state nested;
239232
233+ /* NMI mask value, used when vNMI is not enabled */
234+ bool nmi_masked;
235+
236+ /*
237+ * True when NMIs are still masked but guest IRET was just intercepted
238+ * and KVM is waiting for RIP to change, which will signal that the
239+ * intercepted IRET was retired and thus NMI can be unmasked.
240+ */
241+ bool awaiting_iret_completion;
242+
243+ /*
244+ * Set when KVM is awaiting IRET completion and needs to inject NMIs as
245+ * soon as the IRET completes (e.g. NMI is pending injection). KVM
246+ * temporarily steals RFLAGS.TF to single-step the guest in this case
247+ * in order to regain control as soon as the NMI-blocking condition
248+ * goes away.
249+ */
240250 bool nmi_singlestep;
241251 u64 nmi_singlestep_guest_rflags;
252+
242253 bool nmi_l1_to_l2;
243254
244255 unsigned long soft_int_csbase;
@@ -280,6 +291,9 @@ struct vcpu_svm { · 35 unmodified lines
280291 bool guest_state_loaded;
281292
282293 bool x2avic_msrs_intercepted;
294+
295+ /* Guest GIF value, used when vGIF is not enabled */
296+ bool guest_gif;
283297 };
284298
285299 struct svm_cpu_data {
@@ -497,7 +511,7 @@ static inline void enable_gif(struct vcpu_svm *svm) · 211 unmodified lines
497511 if (vmcb)
498512 vmcb->control.int_ctl |= V_GIF_MASK;
499513 else
500− svm->vcpu.arch.hflags |= HF_GIF_MASK;
514+ svm->guest_gif = true;
501515 }
502516
503517 static inline void disable_gif(struct vcpu_svm *svm)
@@ -507,7 +521,7 @@ static inline void disable_gif(struct vcpu_svm *svm) · 3 unmodified lines
507521 if (vmcb)
508522 vmcb->control.int_ctl &= ~V_GIF_MASK;
509523 else
510− svm->vcpu.arch.hflags &= ~HF_GIF_MASK;
524+ svm->guest_gif = false;
511525 }
512526
513527 static inline bool gif_set(struct vcpu_svm *svm)
@@ -517,7 +531,7 @@ static inline bool gif_set(struct vcpu_svm *svm) · 3 unmodified lines
517531 if (vmcb)
518532 return !!(vmcb->control.int_ctl & V_GIF_MASK);
519533 else
520− return !!(svm->vcpu.arch.hflags & HF_GIF_MASK);
534+ return svm->guest_gif;
521535 }
522536
523537 static inline bool nested_npt_enabled(struct vcpu_svm *svm)
@@ -628,8 +642,23 @@ void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_vmcb); · 104 unmodified lines
628642 extern struct kvm_x86_nested_ops svm_nested_ops;
629643
630644 /* avic.c */
631−
632−bool avic_hardware_setup(struct kvm_x86_ops *ops);
645+#define AVIC_REQUIRED_APICV_INHIBITS \
646+( \
647+ BIT(APICV_INHIBIT_REASON_DISABLE) | \
648+ BIT(APICV_INHIBIT_REASON_ABSENT) | \
649+ BIT(APICV_INHIBIT_REASON_HYPERV) | \
650+ BIT(APICV_INHIBIT_REASON_NESTED) | \
651+ BIT(APICV_INHIBIT_REASON_IRQWIN) | \
652+ BIT(APICV_INHIBIT_REASON_PIT_REINJ) | \
653+ BIT(APICV_INHIBIT_REASON_BLOCKIRQ) | \
654+ BIT(APICV_INHIBIT_REASON_SEV) | \
655+ BIT(APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED) | \
656+ BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) | \
657+ BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED) | \
658+ BIT(APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED) \
659+)
660+
661+bool avic_hardware_setup(void);
633662 int avic_ga_log_notifier(u32 ga_tag);
634663 void avic_vm_destroy(struct kvm *kvm);
635664 int avic_vm_init(struct kvm *kvm);
@@ -641,14 +670,13 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu); · 5 unmodified lines
641670 void avic_vcpu_put(struct kvm_vcpu *vcpu);
642671 void avic_apicv_post_state_restore(struct kvm_vcpu *vcpu);
643672 void avic_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu);
644−bool avic_check_apicv_inhibit_reasons(enum kvm_apicv_inhibit reason);
645673 int avic_pi_update_irte(struct kvm *kvm, unsigned int host_irq,
646674 uint32_t guest_irq, bool set);
647675 void avic_vcpu_blocking(struct kvm_vcpu *vcpu);
648676 void avic_vcpu_unblocking(struct kvm_vcpu *vcpu);
649677 void avic_ring_doorbell(struct kvm_vcpu *vcpu);
650678 unsigned long avic_vcpu_get_apicv_inhibit_reasons(struct kvm_vcpu *vcpu);
651−void avic_set_virtual_apic_mode(struct kvm_vcpu *vcpu);
679+void avic_refresh_virtual_apic_mode(struct kvm_vcpu *vcpu);
652680
653681
654682 /* sev.c */
arch/x86/kvm/svm/svm_onhyperv.c+1−0
@@ -2,6 +2,7 @@ · 1 unmodified line
22 /*
33 * KVM L1 hypervisor optimizations on Hyper-V for SVM.
44 */
5+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
56
67 #include <linux/kvm_host.h>
78
arch/x86/kvm/svm/svm_onhyperv.h+4−4
@@ -30,11 +30,11 @@ static inline void svm_hv_init_vmcb(struct vmcb *vmcb) · 29 unmodified lines
3030 hve->hv_enlightenments_control.msr_bitmap = 1;
3131 }
3232
33−static inline void svm_hv_hardware_setup(void)
33+static inline __init void svm_hv_hardware_setup(void)
3434 {
3535 if (npt_enabled &&
3636 ms_hyperv.nested_features & HV_X64_NESTED_ENLIGHTENED_TLB) {
37− pr_info("kvm: Hyper-V enlightened NPT TLB flush enabled\n");
37+ pr_info(KBUILD_MODNAME ": Hyper-V enlightened NPT TLB flush enabled\n");
3838 svm_x86_ops.tlb_remote_flush = hv_remote_flush_tlb;
3939 svm_x86_ops.tlb_remote_flush_with_range =
4040 hv_remote_flush_tlb_with_range;
@@ -43,7 +43,7 @@ static inline void svm_hv_hardware_setup(void) · 2 unmodified lines
4343 if (ms_hyperv.nested_features & HV_X64_NESTED_DIRECT_FLUSH) {
4444 int cpu;
4545
46− pr_info("kvm: Hyper-V Direct TLB Flush enabled\n");
46+ pr_info(KBUILD_MODNAME ": Hyper-V Direct TLB Flush enabled\n");
4747 for_each_online_cpu(cpu) {
4848 struct hv_vp_assist_page *vp_ap =
4949 hv_get_vp_assist_page(cpu);
@@ -84,7 +84,7 @@ static inline void svm_hv_init_vmcb(struct vmcb *vmcb) · 34 unmodified lines
8484 {
8585 }
8686
87−static inline void svm_hv_hardware_setup(void)
87+static inline __init void svm_hv_hardware_setup(void)
8888 {
8989 }
9090
arch/x86/kvm/vmx/capabilities.h+2−2
@@ -66,13 +66,13 @@ struct vmcs_config { · 65 unmodified lines
6666 u64 misc;
6767 struct nested_vmx_msrs nested;
6868 };
69−extern struct vmcs_config vmcs_config;
69+extern struct vmcs_config vmcs_config __ro_after_init;
7070
7171 struct vmx_capability {
7272 u32 ept;
7373 u32 vpid;
7474 };
75−extern struct vmx_capability vmx_capability;
75+extern struct vmx_capability vmx_capability __ro_after_init;
7676
7777 static inline bool cpu_has_vmx_basic_inout(void)
7878 {
arch/x86/kvm/vmx/hyperv.c+68−19
@@ -1,4 +1,5 @@
11 // SPDX-License-Identifier: GPL-2.0
2+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
23
34 #include <linux/errno.h>
45 #include <linux/smp.h>
@@ -361,35 +362,43 @@ enum evmcs_revision { · 356 unmodified lines
361362 enum evmcs_ctrl_type {
362363 EVMCS_EXIT_CTRLS,
363364 EVMCS_ENTRY_CTRLS,
365+ EVMCS_EXEC_CTRL,
364366 EVMCS_2NDEXEC,
367+ EVMCS_3RDEXEC,
365368 EVMCS_PINCTRL,
366369 EVMCS_VMFUNC,
367370 NR_EVMCS_CTRLS,
368371 };
369372
370−static const u32 evmcs_unsupported_ctrls[NR_EVMCS_CTRLS][NR_EVMCS_REVISIONS] = {
373+static const u32 evmcs_supported_ctrls[NR_EVMCS_CTRLS][NR_EVMCS_REVISIONS] = {
371374 [EVMCS_EXIT_CTRLS] = {
372− [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_VMEXIT_CTRL,
375+ [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_VMEXIT_CTRL,
373376 },
374377 [EVMCS_ENTRY_CTRLS] = {
375− [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_VMENTRY_CTRL,
378+ [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_VMENTRY_CTRL,
379+ },
380+ [EVMCS_EXEC_CTRL] = {
381+ [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_EXEC_CTRL,
376382 },
377383 [EVMCS_2NDEXEC] = {
378− [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_2NDEXEC,
384+ [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_2NDEXEC & ~SECONDARY_EXEC_TSC_SCALING,
385+ },
386+ [EVMCS_3RDEXEC] = {
387+ [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_3RDEXEC,
379388 },
380389 [EVMCS_PINCTRL] = {
381− [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_PINCTRL,
390+ [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_PINCTRL,
382391 },
383392 [EVMCS_VMFUNC] = {
384− [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_VMFUNC,
393+ [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_VMFUNC,
385394 },
386395 };
387396
388−static u32 evmcs_get_unsupported_ctls(enum evmcs_ctrl_type ctrl_type)
397+static u32 evmcs_get_supported_ctls(enum evmcs_ctrl_type ctrl_type)
389398 {
390399 enum evmcs_revision evmcs_rev = EVMCSv1_LEGACY;
391400
392− return evmcs_unsupported_ctrls[ctrl_type][evmcs_rev];
401+ return evmcs_supported_ctrls[ctrl_type][evmcs_rev];
393402 }
394403
395404 static bool evmcs_has_perf_global_ctrl(struct kvm_vcpu *vcpu)
@@ -413,7 +422,7 @@ void nested_evmcs_filter_control_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 * · 17 unmodified lines
413422 {
414423 u32 ctl_low = (u32)*pdata;
415424 u32 ctl_high = (u32)(*pdata >> 32);
416− u32 unsupported_ctrls;
425+ u32 supported_ctrls;
417426
418427 /*
419428 * Hyper-V 2016 and 2019 try using these features even when eVMCS
@@ -422,27 +431,31 @@ void nested_evmcs_filter_control_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 * · 2 unmodified lines
422431 switch (msr_index) {
423432 case MSR_IA32_VMX_EXIT_CTLS:
424433 case MSR_IA32_VMX_TRUE_EXIT_CTLS:
425− unsupported_ctrls = evmcs_get_unsupported_ctls(EVMCS_EXIT_CTRLS);
434+ supported_ctrls = evmcs_get_supported_ctls(EVMCS_EXIT_CTRLS);
426435 if (!evmcs_has_perf_global_ctrl(vcpu))
427− unsupported_ctrls |= VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL;
428− ctl_high &= ~unsupported_ctrls;
436+ supported_ctrls &= ~VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL;
437+ ctl_high &= supported_ctrls;
429438 break;
430439 case MSR_IA32_VMX_ENTRY_CTLS:
431440 case MSR_IA32_VMX_TRUE_ENTRY_CTLS:
432− unsupported_ctrls = evmcs_get_unsupported_ctls(EVMCS_ENTRY_CTRLS);
441+ supported_ctrls = evmcs_get_supported_ctls(EVMCS_ENTRY_CTRLS);
433442 if (!evmcs_has_perf_global_ctrl(vcpu))
434− unsupported_ctrls |= VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL;
435− ctl_high &= ~unsupported_ctrls;
443+ supported_ctrls &= ~VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL;
444+ ctl_high &= supported_ctrls;
445+ break;
446+ case MSR_IA32_VMX_PROCBASED_CTLS:
447+ case MSR_IA32_VMX_TRUE_PROCBASED_CTLS:
448+ ctl_high &= evmcs_get_supported_ctls(EVMCS_EXEC_CTRL);
436449 break;
437450 case MSR_IA32_VMX_PROCBASED_CTLS2:
438− ctl_high &= ~evmcs_get_unsupported_ctls(EVMCS_2NDEXEC);
451+ ctl_high &= evmcs_get_supported_ctls(EVMCS_2NDEXEC);
439452 break;
440453 case MSR_IA32_VMX_TRUE_PINBASED_CTLS:
441454 case MSR_IA32_VMX_PINBASED_CTLS:
442− ctl_high &= ~evmcs_get_unsupported_ctls(EVMCS_PINCTRL);
455+ ctl_high &= evmcs_get_supported_ctls(EVMCS_PINCTRL);
443456 break;
444457 case MSR_IA32_VMX_VMFUNC:
445− ctl_low &= ~evmcs_get_unsupported_ctls(EVMCS_VMFUNC);
458+ ctl_low &= evmcs_get_supported_ctls(EVMCS_VMFUNC);
446459 break;
447460 }
448461
@@ -452,7 +465,7 @@ void nested_evmcs_filter_control_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 * · 3 unmodified lines
452465 static bool nested_evmcs_is_valid_controls(enum evmcs_ctrl_type ctrl_type,
453466 u32 val)
454467 {
455− return !(val & evmcs_get_unsupported_ctls(ctrl_type));
468+ return !(val & ~evmcs_get_supported_ctls(ctrl_type));
456469 }
457470
458471 int nested_evmcs_check_controls(struct vmcs12 *vmcs12)
@@ -461,6 +474,10 @@ int nested_evmcs_check_controls(struct vmcs12 *vmcs12) · 2 unmodified lines
461474 vmcs12->pin_based_vm_exec_control)))
462475 return -EINVAL;
463476
477+ if (CC(!nested_evmcs_is_valid_controls(EVMCS_EXEC_CTRL,
478+ vmcs12->cpu_based_vm_exec_control)))
479+ return -EINVAL;
480+
464481 if (CC(!nested_evmcs_is_valid_controls(EVMCS_2NDEXEC,
465482 vmcs12->secondary_vm_exec_control)))
466483 return -EINVAL;
@@ -488,6 +505,38 @@ int nested_evmcs_check_controls(struct vmcs12 *vmcs12) · 21 unmodified lines
488505 return 0;
489506 }
490507
508+#if IS_ENABLED(CONFIG_HYPERV)
509+/*
510+ * KVM on Hyper-V always uses the latest known eVMCSv1 revision, the assumption
511+ * is: in case a feature has corresponding fields in eVMCS described and it was
512+ * exposed in VMX feature MSRs, KVM is free to use it. Warn if KVM meets a
513+ * feature which has no corresponding eVMCS field, this likely means that KVM
514+ * needs to be updated.
515+ */
516+#define evmcs_check_vmcs_conf(field, ctrl) \
517+ do { \
518+ typeof(vmcs_conf->field) unsupported; \
519+ \
520+ unsupported = vmcs_conf->field & ~EVMCS1_SUPPORTED_ ## ctrl; \
521+ if (unsupported) { \
522+ pr_warn_once(#field " unsupported with eVMCS: 0x%llx\n",\
523+ (u64)unsupported); \
524+ vmcs_conf->field &= EVMCS1_SUPPORTED_ ## ctrl; \
525+ } \
526+ } \
527+ while (0)
528+
529+void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf)
530+{
531+ evmcs_check_vmcs_conf(cpu_based_exec_ctrl, EXEC_CTRL);
532+ evmcs_check_vmcs_conf(pin_based_exec_ctrl, PINCTRL);
533+ evmcs_check_vmcs_conf(cpu_based_2nd_exec_ctrl, 2NDEXEC);
534+ evmcs_check_vmcs_conf(cpu_based_3rd_exec_ctrl, 3RDEXEC);
535+ evmcs_check_vmcs_conf(vmentry_ctrl, VMENTRY_CTRL);
536+ evmcs_check_vmcs_conf(vmexit_ctrl, VMEXIT_CTRL);
537+}
538+#endif
539+
491540 int nested_enable_evmcs(struct kvm_vcpu *vcpu,
492541 uint16_t *vmcs_version)
493542 {
arch/x86/kvm/vmx/hyperv.h+89−39
@@ -48,22 +48,84 @@ DECLARE_STATIC_KEY_FALSE(enable_evmcs); · 47 unmodified lines
4848 * Currently unsupported in KVM:
4949 * GUEST_IA32_RTIT_CTL = 0x00002814,
5050 */
51−#define EVMCS1_UNSUPPORTED_PINCTRL (PIN_BASED_POSTED_INTR | \
52− PIN_BASED_VMX_PREEMPTION_TIMER)
53−#define EVMCS1_UNSUPPORTED_EXEC_CTRL (CPU_BASED_ACTIVATE_TERTIARY_CONTROLS)
54−#define EVMCS1_UNSUPPORTED_2NDEXEC \
55− (SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY | \
56− SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES | \
57− SECONDARY_EXEC_APIC_REGISTER_VIRT | \
58− SECONDARY_EXEC_ENABLE_PML | \
59− SECONDARY_EXEC_ENABLE_VMFUNC | \
60− SECONDARY_EXEC_SHADOW_VMCS | \
51+#define EVMCS1_SUPPORTED_PINCTRL \
52+ (PIN_BASED_ALWAYSON_WITHOUT_TRUE_MSR | \
53+ PIN_BASED_EXT_INTR_MASK | \
54+ PIN_BASED_NMI_EXITING | \
55+ PIN_BASED_VIRTUAL_NMIS)
56+
57+#define EVMCS1_SUPPORTED_EXEC_CTRL \
58+ (CPU_BASED_ALWAYSON_WITHOUT_TRUE_MSR | \
59+ CPU_BASED_HLT_EXITING | \
60+ CPU_BASED_CR3_LOAD_EXITING | \
61+ CPU_BASED_CR3_STORE_EXITING | \
62+ CPU_BASED_UNCOND_IO_EXITING | \
63+ CPU_BASED_MOV_DR_EXITING | \
64+ CPU_BASED_USE_TSC_OFFSETTING | \
65+ CPU_BASED_MWAIT_EXITING | \
66+ CPU_BASED_MONITOR_EXITING | \
67+ CPU_BASED_INVLPG_EXITING | \
68+ CPU_BASED_RDPMC_EXITING | \
69+ CPU_BASED_INTR_WINDOW_EXITING | \
70+ CPU_BASED_CR8_LOAD_EXITING | \
71+ CPU_BASED_CR8_STORE_EXITING | \
72+ CPU_BASED_RDTSC_EXITING | \
73+ CPU_BASED_TPR_SHADOW | \
74+ CPU_BASED_USE_IO_BITMAPS | \
75+ CPU_BASED_MONITOR_TRAP_FLAG | \
76+ CPU_BASED_USE_MSR_BITMAPS | \
77+ CPU_BASED_NMI_WINDOW_EXITING | \
78+ CPU_BASED_PAUSE_EXITING | \
79+ CPU_BASED_ACTIVATE_SECONDARY_CONTROLS)
80+
81+#define EVMCS1_SUPPORTED_2NDEXEC \
82+ (SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE | \
83+ SECONDARY_EXEC_WBINVD_EXITING | \
84+ SECONDARY_EXEC_ENABLE_VPID | \
85+ SECONDARY_EXEC_ENABLE_EPT | \
86+ SECONDARY_EXEC_UNRESTRICTED_GUEST | \
87+ SECONDARY_EXEC_DESC | \
88+ SECONDARY_EXEC_ENABLE_RDTSCP | \
89+ SECONDARY_EXEC_ENABLE_INVPCID | \
90+ SECONDARY_EXEC_XSAVES | \
91+ SECONDARY_EXEC_RDSEED_EXITING | \
92+ SECONDARY_EXEC_RDRAND_EXITING | \
6193 SECONDARY_EXEC_TSC_SCALING | \
62− SECONDARY_EXEC_PAUSE_LOOP_EXITING)
63−#define EVMCS1_UNSUPPORTED_VMEXIT_CTRL \
64− (VM_EXIT_SAVE_VMX_PREEMPTION_TIMER)
65−#define EVMCS1_UNSUPPORTED_VMENTRY_CTRL (0)
66−#define EVMCS1_UNSUPPORTED_VMFUNC (VMX_VMFUNC_EPTP_SWITCHING)
94+ SECONDARY_EXEC_ENABLE_USR_WAIT_PAUSE | \
95+ SECONDARY_EXEC_PT_USE_GPA | \
96+ SECONDARY_EXEC_PT_CONCEAL_VMX | \
97+ SECONDARY_EXEC_BUS_LOCK_DETECTION | \
98+ SECONDARY_EXEC_NOTIFY_VM_EXITING | \
99+ SECONDARY_EXEC_ENCLS_EXITING)
100+
101+#define EVMCS1_SUPPORTED_3RDEXEC (0ULL)
102+
103+#define EVMCS1_SUPPORTED_VMEXIT_CTRL \
104+ (VM_EXIT_ALWAYSON_WITHOUT_TRUE_MSR | \
105+ VM_EXIT_SAVE_DEBUG_CONTROLS | \
106+ VM_EXIT_ACK_INTR_ON_EXIT | \
107+ VM_EXIT_HOST_ADDR_SPACE_SIZE | \
108+ VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL | \
109+ VM_EXIT_SAVE_IA32_PAT | \
110+ VM_EXIT_LOAD_IA32_PAT | \
111+ VM_EXIT_SAVE_IA32_EFER | \
112+ VM_EXIT_LOAD_IA32_EFER | \
113+ VM_EXIT_CLEAR_BNDCFGS | \
114+ VM_EXIT_PT_CONCEAL_PIP | \
115+ VM_EXIT_CLEAR_IA32_RTIT_CTL)
116+
117+#define EVMCS1_SUPPORTED_VMENTRY_CTRL \
118+ (VM_ENTRY_ALWAYSON_WITHOUT_TRUE_MSR | \
119+ VM_ENTRY_LOAD_DEBUG_CONTROLS | \
120+ VM_ENTRY_IA32E_MODE | \
121+ VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL | \
122+ VM_ENTRY_LOAD_IA32_PAT | \
123+ VM_ENTRY_LOAD_IA32_EFER | \
124+ VM_ENTRY_LOAD_BNDCFGS | \
125+ VM_ENTRY_PT_CONCEAL_PIP | \
126+ VM_ENTRY_LOAD_IA32_RTIT_CTL)
127+
128+#define EVMCS1_SUPPORTED_VMFUNC (0)
67129
68130 struct evmcs_field {
69131 u16 offset;
@@ -117,9 +179,7 @@ static __always_inline int get_evmcs_offset(unsigned long field, · 47 unmodified lines
117179 {
118180 int offset = evmcs_field_offset(field, clean_field);
119181
120− WARN_ONCE(offset < 0, "KVM: accessing unsupported EVMCS field %lx\n",
121− field);
122−
182+ WARN_ONCE(offset < 0, "accessing unsupported EVMCS field %lx\n", field);
123183 return offset;
124184 }
125185
@@ -136,7 +196,7 @@ static __always_inline void evmcs_write64(unsigned long field, u64 value) · 10 unmodified lines
136196 current_evmcs->hv_clean_fields &= ~clean_field;
137197 }
138198
139−static inline void evmcs_write32(unsigned long field, u32 value)
199+static __always_inline void evmcs_write32(unsigned long field, u32 value)
140200 {
141201 u16 clean_field;
142202 int offset = get_evmcs_offset(field, &clean_field);
@@ -148,7 +208,7 @@ static inline void evmcs_write32(unsigned long field, u32 value) · 5 unmodified lines
148208 current_evmcs->hv_clean_fields &= ~clean_field;
149209 }
150210
151−static inline void evmcs_write16(unsigned long field, u16 value)
211+static __always_inline void evmcs_write16(unsigned long field, u16 value)
152212 {
153213 u16 clean_field;
154214 int offset = get_evmcs_offset(field, &clean_field);
@@ -160,7 +220,7 @@ static inline void evmcs_write16(unsigned long field, u16 value) · 5 unmodified lines
160220 current_evmcs->hv_clean_fields &= ~clean_field;
161221 }
162222
163−static inline u64 evmcs_read64(unsigned long field)
223+static __always_inline u64 evmcs_read64(unsigned long field)
164224 {
165225 int offset = get_evmcs_offset(field, NULL);
166226
@@ -170,7 +230,7 @@ static inline u64 evmcs_read64(unsigned long field) · 3 unmodified lines
170230 return *(u64 *)((char *)current_evmcs + offset);
171231 }
172232
173−static inline u32 evmcs_read32(unsigned long field)
233+static __always_inline u32 evmcs_read32(unsigned long field)
174234 {
175235 int offset = get_evmcs_offset(field, NULL);
176236
@@ -180,7 +240,7 @@ static inline u32 evmcs_read32(unsigned long field) · 3 unmodified lines
180240 return *(u32 *)((char *)current_evmcs + offset);
181241 }
182242
183−static inline u16 evmcs_read16(unsigned long field)
243+static __always_inline u16 evmcs_read16(unsigned long field)
184244 {
185245 int offset = get_evmcs_offset(field, NULL);
186246
@@ -190,16 +250,6 @@ static inline u16 evmcs_read16(unsigned long field) · 3 unmodified lines
190250 return *(u16 *)((char *)current_evmcs + offset);
191251 }
192252
193−static inline void evmcs_touch_msr_bitmap(void)
194−{
195− if (unlikely(!current_evmcs))
196− return;
197−
198− if (current_evmcs->hv_enlightenments_control.msr_bitmap)
199− current_evmcs->hv_clean_fields &=
200− ~HV_VMX_ENLIGHTENED_CLEAN_FIELD_MSR_BITMAP;
201−}
202−
203253 static inline void evmcs_load(u64 phys_addr)
204254 {
205255 struct hv_vp_assist_page *vp_ap =
@@ -211,15 +261,15 @@ static inline void evmcs_load(u64 phys_addr) · 5 unmodified lines
211261 vp_ap->enlighten_vmentry = 1;
212262 }
213263
264+void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf);
214265 #else /* !IS_ENABLED(CONFIG_HYPERV) */
215266 static __always_inline void evmcs_write64(unsigned long field, u64 value) {}
216−static inline void evmcs_write32(unsigned long field, u32 value) {}
217−static inline void evmcs_write16(unsigned long field, u16 value) {}
218−static inline u64 evmcs_read64(unsigned long field) { return 0; }
219−static inline u32 evmcs_read32(unsigned long field) { return 0; }
220−static inline u16 evmcs_read16(unsigned long field) { return 0; }
267+static __always_inline void evmcs_write32(unsigned long field, u32 value) {}
268+static __always_inline void evmcs_write16(unsigned long field, u16 value) {}
269+static __always_inline u64 evmcs_read64(unsigned long field) { return 0; }
270+static __always_inline u32 evmcs_read32(unsigned long field) { return 0; }
271+static __always_inline u16 evmcs_read16(unsigned long field) { return 0; }
221272 static inline void evmcs_load(u64 phys_addr) {}
222−static inline void evmcs_touch_msr_bitmap(void) {}
223273 #endif /* IS_ENABLED(CONFIG_HYPERV) */
224274
225275 #define EVMPTR_INVALID (-1ULL)
arch/x86/kvm/vmx/nested.c+10−14
@@ -1,4 +1,5 @@
11 // SPDX-License-Identifier: GPL-2.0
2+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
23
34 #include <linux/objtool.h>
45 #include <linux/percpu.h>
@@ -203,7 +204,7 @@ static void nested_vmx_abort(struct kvm_vcpu *vcpu, u32 indicator) · 198 unmodified lines
203204 {
204205 /* TODO: not to reset guest simply here. */
205206 kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
206− pr_debug_ratelimited("kvm: nested vmx abort, indicator %d\n", indicator);
207+ pr_debug_ratelimited("nested vmx abort, indicator %d\n", indicator);
207208 }
208209
209210 static inline bool vmx_control_verify(u32 control, u32 low, u32 high)
@@ -5863,11 +5864,10 @@ static int handle_vmfunc(struct kvm_vcpu *vcpu) · 5653 unmodified lines
58635864 u32 function = kvm_rax_read(vcpu);
58645865
58655866 /*
5866− * VMFUNC is only supported for nested guests, but we always enable the
5867− * secondary control for simplicity; for non-nested mode, fake that we
5868− * didn't by injecting #UD.
5867+ * VMFUNC should never execute cleanly while L1 is active; KVM supports
5868+ * VMFUNC for nested VMs, but not for L1.
58695869 */
5870− if (!is_guest_mode(vcpu)) {
5870+ if (WARN_ON_ONCE(!is_guest_mode(vcpu))) {
58715871 kvm_queue_exception(vcpu, UD_VECTOR);
58725872 return 1;
58735873 }
@@ -6880,6 +6880,7 @@ void nested_vmx_setup_ctls_msrs(struct vmcs_config *vmcs_conf, u32 ept_caps) · 1006 unmodified lines
68806880 SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |
68816881 SECONDARY_EXEC_RDRAND_EXITING |
68826882 SECONDARY_EXEC_ENABLE_INVPCID |
6883+ SECONDARY_EXEC_ENABLE_VMFUNC |
68836884 SECONDARY_EXEC_RDSEED_EXITING |
68846885 SECONDARY_EXEC_XSAVES |
68856886 SECONDARY_EXEC_TSC_SCALING |
@@ -6912,18 +6913,13 @@ void nested_vmx_setup_ctls_msrs(struct vmcs_config *vmcs_conf, u32 ept_caps) · 26 unmodified lines
69126913 SECONDARY_EXEC_ENABLE_PML;
69136914 msrs->ept_caps |= VMX_EPT_AD_BIT;
69146915 }
6915− }
69166916
6917− if (cpu_has_vmx_vmfunc()) {
6918− msrs->secondary_ctls_high |=
6919− SECONDARY_EXEC_ENABLE_VMFUNC;
69206917 /*
6921− * Advertise EPTP switching unconditionally
6922− * since we emulate it
6918+ * Advertise EPTP switching irrespective of hardware support,
6919+ * KVM emulates it in software so long as VMFUNC is supported.
69236920 */
6924− if (enable_ept)
6925− msrs->vmfunc_controls =
6926− VMX_VMFUNC_EPTP_SWITCHING;
6921+ if (cpu_has_vmx_vmfunc())
6922+ msrs->vmfunc_controls = VMX_VMFUNC_EPTP_SWITCHING;
69276923 }
69286924
69296925 /*
arch/x86/kvm/vmx/pmu_intel.c+17−11
@@ -8,6 +8,8 @@ · 7 unmodified lines
88 * Avi Kivity <avi@redhat.com>
99 * Gleb Natapov <gleb@redhat.com>
1010 */
11+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
12+
1113 #include <linux/types.h>
1214 #include <linux/kvm_host.h>
1315 #include <linux/perf_event.h>
@@ -20,16 +22,19 @@ · 6 unmodified lines
2022
2123 #define MSR_PMC_FULL_WIDTH_BIT (MSR_IA32_PMC0 - MSR_IA32_PERFCTR0)
2224
23−static struct kvm_event_hw_type_mapping intel_arch_events[] = {
24− [0] = { 0x3c, 0x00, PERF_COUNT_HW_CPU_CYCLES },
25− [1] = { 0xc0, 0x00, PERF_COUNT_HW_INSTRUCTIONS },
26− [2] = { 0x3c, 0x01, PERF_COUNT_HW_BUS_CYCLES },
27− [3] = { 0x2e, 0x4f, PERF_COUNT_HW_CACHE_REFERENCES },
28− [4] = { 0x2e, 0x41, PERF_COUNT_HW_CACHE_MISSES },
29− [5] = { 0xc4, 0x00, PERF_COUNT_HW_BRANCH_INSTRUCTIONS },
30− [6] = { 0xc5, 0x00, PERF_COUNT_HW_BRANCH_MISSES },
25+static struct {
26+ u8 eventsel;
27+ u8 unit_mask;
28+} const intel_arch_events[] = {
29+ [0] = { 0x3c, 0x00 },
30+ [1] = { 0xc0, 0x00 },
31+ [2] = { 0x3c, 0x01 },
32+ [3] = { 0x2e, 0x4f },
33+ [4] = { 0x2e, 0x41 },
34+ [5] = { 0xc4, 0x00 },
35+ [6] = { 0xc5, 0x00 },
3136 /* The above index must match CPUID 0x0A.EBX bit vector */
32− [7] = { 0x00, 0x03, PERF_COUNT_HW_REF_CPU_CYCLES },
37+ [7] = { 0x00, 0x03 },
3338 };
3439
3540 /* mapping between fixed pmc index and intel_arch_events array */
@@ -762,8 +767,7 @@ void vmx_passthrough_lbr_msrs(struct kvm_vcpu *vcpu) · 726 unmodified lines
762767 return;
763768
764769 warn:
765− pr_warn_ratelimited("kvm: vcpu-%d: fail to passthrough LBR.\n",
766− vcpu->vcpu_id);
770+ pr_warn_ratelimited("vcpu-%d: fail to passthrough LBR.\n", vcpu->vcpu_id);
767771 }
768772
769773 static void intel_pmu_cleanup(struct kvm_vcpu *vcpu)
@@ -810,4 +814,6 @@ struct kvm_pmu_ops intel_pmu_ops __initdata = { · 40 unmodified lines
810814 .reset = intel_pmu_reset,
811815 .deliver_pmi = intel_pmu_deliver_pmi,
812816 .cleanup = intel_pmu_cleanup,
817+ .EVENTSEL_EVENT = ARCH_PERFMON_EVENTSEL_EVENT,
818+ .MAX_NR_GP_COUNTERS = KVM_INTEL_PMC_MAX_GENERIC,
813819 };
arch/x86/kvm/vmx/posted_intr.c+2−0
@@ -1,4 +1,6 @@
11 // SPDX-License-Identifier: GPL-2.0-only
2+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
3+
24 #include <linux/kvm_host.h>
35
46 #include <asm/irq_remapping.h>
arch/x86/kvm/vmx/sgx.c+3−2
@@ -1,5 +1,6 @@
11 // SPDX-License-Identifier: GPL-2.0
22 /* Copyright(c) 2021 Intel Corporation. */
3+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
34
45 #include <asm/sgx.h>
56
@@ -164,7 +165,7 @@ static int __handle_encls_ecreate(struct kvm_vcpu *vcpu, · 158 unmodified lines
164165 if (!vcpu->kvm->arch.sgx_provisioning_allowed &&
165166 (attributes & SGX_ATTR_PROVISIONKEY)) {
166167 if (sgx_12_1->eax & SGX_ATTR_PROVISIONKEY)
167− pr_warn_once("KVM: SGX PROVISIONKEY advertised but not allowed\n");
168+ pr_warn_once("SGX PROVISIONKEY advertised but not allowed\n");
168169 kvm_inject_gp(vcpu, 0);
169170 return 1;
170171 }
@@ -381,7 +382,7 @@ int handle_encls(struct kvm_vcpu *vcpu) · 210 unmodified lines
381382 return handle_encls_ecreate(vcpu);
382383 if (leaf == EINIT)
383384 return handle_encls_einit(vcpu);
384− WARN(1, "KVM: unexpected exit on ENCLS[%u]", leaf);
385+ WARN_ONCE(1, "unexpected exit on ENCLS[%u]", leaf);
385386 vcpu->run->exit_reason = KVM_EXIT_UNKNOWN;
386387 vcpu->run->hw.hardware_exit_reason = EXIT_REASON_ENCLS;
387388 return 0;
arch/x86/kvm/vmx/vmcs.h+2−2
@@ -75,7 +75,7 @@ struct loaded_vmcs { · 74 unmodified lines
7575 struct vmcs_controls_shadow controls_shadow;
7676 };
7777
78−static inline bool is_intr_type(u32 intr_info, u32 type)
78+static __always_inline bool is_intr_type(u32 intr_info, u32 type)
7979 {
8080 const u32 mask = INTR_INFO_VALID_MASK | INTR_INFO_INTR_TYPE_MASK;
8181
@@ -146,7 +146,7 @@ static inline bool is_icebp(u32 intr_info) · 64 unmodified lines
146146 return is_intr_type(intr_info, INTR_TYPE_PRIV_SW_EXCEPTION);
147147 }
148148
149−static inline bool is_nmi(u32 intr_info)
149+static __always_inline bool is_nmi(u32 intr_info)
150150 {
151151 return is_intr_type(intr_info, INTR_TYPE_NMI_INTR);
152152 }
arch/x86/kvm/vmx/vmcs12.c+1−0
@@ -1,4 +1,5 @@
11 // SPDX-License-Identifier: GPL-2.0
2+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
23
34 #include "vmcs12.h"
45
arch/x86/kvm/vmx/vmenter.S+44−36
@@ -31,6 +31,39 @@ · 30 unmodified lines
3131 #define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE
3232 #endif
3333
34+.macro VMX_DO_EVENT_IRQOFF call_insn call_target
35+ /*
36+ * Unconditionally create a stack frame, getting the correct RSP on the
37+ * stack (for x86-64) would take two instructions anyways, and RBP can
38+ * be used to restore RSP to make objtool happy (see below).
39+ */
40+ push %_ASM_BP
41+ mov %_ASM_SP, %_ASM_BP
42+
43+#ifdef CONFIG_X86_64
44+ /*
45+ * Align RSP to a 16-byte boundary (to emulate CPU behavior) before
46+ * creating the synthetic interrupt stack frame for the IRQ/NMI.
47+ */
48+ and $-16, %rsp
49+ push $__KERNEL_DS
50+ push %rbp
51+#endif
52+ pushf
53+ push $__KERNEL_CS
54+ \call_insn \call_target
55+
56+ /*
57+ * "Restore" RSP from RBP, even though IRET has already unwound RSP to
58+ * the correct value. objtool doesn't know the callee will IRET and,
59+ * without the explicit restore, thinks the stack is getting walloped.
60+ * Using an unwind hint is problematic due to x86-64's dynamic alignment.
61+ */
62+ mov %_ASM_BP, %_ASM_SP
63+ pop %_ASM_BP
64+ RET
65+.endm
66+
3467 .section .noinstr.text, "ax"
3568
3669 /**
@@ -69,8 +102,8 @@ SYM_FUNC_START(__vmx_vcpu_run) · 32 unmodified lines
69102 */
70103 push %_ASM_ARG2
71104
72− /* Copy @flags to BL, _ASM_ARG3 is volatile. */
73− mov %_ASM_ARG3B, %bl
105+ /* Copy @flags to EBX, _ASM_ARG3 is volatile. */
106+ mov %_ASM_ARG3L, %ebx
74107
75108 lea (%_ASM_SP), %_ASM_ARG2
76109 call vmx_update_host_rsp
@@ -106,7 +139,7 @@ SYM_FUNC_START(__vmx_vcpu_run) · 29 unmodified lines
106139 mov (%_ASM_SP), %_ASM_AX
107140
108141 /* Check if vmlaunch or vmresume is needed */
109− testb $VMX_RUN_VMRESUME, %bl
142+ test $VMX_RUN_VMRESUME, %ebx
110143
111144 /* Load guest registers. Don't clobber flags. */
112145 mov VCPU_RCX(%_ASM_AX), %_ASM_CX
@@ -128,7 +161,7 @@ SYM_FUNC_START(__vmx_vcpu_run) · 15 unmodified lines
128161 /* Load guest RAX. This kills the @regs pointer! */
129162 mov VCPU_RAX(%_ASM_AX), %_ASM_AX
130163
131− /* Check EFLAGS.ZF from 'testb' above */
164+ /* Check EFLAGS.ZF from 'test VMX_RUN_VMRESUME' above */
132165 jz .Lvmlaunch
133166
134167 /*
@@ -266,6 +299,10 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL) · 131 unmodified lines
266299
267300 SYM_FUNC_END(__vmx_vcpu_run)
268301
302+SYM_FUNC_START(vmx_do_nmi_irqoff)
303+ VMX_DO_EVENT_IRQOFF call asm_exc_nmi_kvm_vmx
304+SYM_FUNC_END(vmx_do_nmi_irqoff)
305+
269306
270307 .section .text, "ax"
271308
@@ -320,35 +357,6 @@ SYM_FUNC_START(vmread_error_trampoline) · 48 unmodified lines
320357 SYM_FUNC_END(vmread_error_trampoline)
321358 #endif
322359
323−SYM_FUNC_START(vmx_do_interrupt_nmi_irqoff)
324− /*
325− * Unconditionally create a stack frame, getting the correct RSP on the
326− * stack (for x86-64) would take two instructions anyways, and RBP can
327− * be used to restore RSP to make objtool happy (see below).
328− */
329− push %_ASM_BP
330− mov %_ASM_SP, %_ASM_BP
331−
332−#ifdef CONFIG_X86_64
333− /*
334− * Align RSP to a 16-byte boundary (to emulate CPU behavior) before
335− * creating the synthetic interrupt stack frame for the IRQ/NMI.
336− */
337− and $-16, %rsp
338− push $__KERNEL_DS
339− push %rbp
340−#endif
341− pushf
342− push $__KERNEL_CS
343− CALL_NOSPEC _ASM_ARG1
344−
345− /*
346− * "Restore" RSP from RBP, even though IRET has already unwound RSP to
347− * the correct value. objtool doesn't know the callee will IRET and,
348− * without the explicit restore, thinks the stack is getting walloped.
349− * Using an unwind hint is problematic due to x86-64's dynamic alignment.
350− */
351− mov %_ASM_BP, %_ASM_SP
352− pop %_ASM_BP
353− RET
354−SYM_FUNC_END(vmx_do_interrupt_nmi_irqoff)
360+SYM_FUNC_START(vmx_do_interrupt_irqoff)
361+ VMX_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1
362+SYM_FUNC_END(vmx_do_interrupt_irqoff)
arch/x86/kvm/vmx/vmx.c+328−296
@@ -12,6 +12,7 @@ · 11 unmodified lines
1212 * Avi Kivity <avi@qumranet.com>
1313 * Yaniv Kamay <yaniv@qumranet.com>
1414 */
15+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
1516
1617 #include <linux/highmem.h>
1718 #include <linux/hrtimer.h>
@@ -444,36 +445,36 @@ void vmread_error(unsigned long field, bool fault) · 426 unmodified lines
444445 if (fault)
445446 kvm_spurious_fault();
446447 else
447− vmx_insn_failed("kvm: vmread failed: field=%lx\n", field);
448+ vmx_insn_failed("vmread failed: field=%lx\n", field);
448449 }
449450
450451 noinline void vmwrite_error(unsigned long field, unsigned long value)
451452 {
452− vmx_insn_failed("kvm: vmwrite failed: field=%lx val=%lx err=%u\n",
453+ vmx_insn_failed("vmwrite failed: field=%lx val=%lx err=%u\n",
453454 field, value, vmcs_read32(VM_INSTRUCTION_ERROR));
454455 }
455456
456457 noinline void vmclear_error(struct vmcs *vmcs, u64 phys_addr)
457458 {
458− vmx_insn_failed("kvm: vmclear failed: %p/%llx err=%u\n",
459+ vmx_insn_failed("vmclear failed: %p/%llx err=%u\n",
459460 vmcs, phys_addr, vmcs_read32(VM_INSTRUCTION_ERROR));
460461 }
461462
462463 noinline void vmptrld_error(struct vmcs *vmcs, u64 phys_addr)
463464 {
464− vmx_insn_failed("kvm: vmptrld failed: %p/%llx err=%u\n",
465+ vmx_insn_failed("vmptrld failed: %p/%llx err=%u\n",
465466 vmcs, phys_addr, vmcs_read32(VM_INSTRUCTION_ERROR));
466467 }
467468
468469 noinline void invvpid_error(unsigned long ext, u16 vpid, gva_t gva)
469470 {
470− vmx_insn_failed("kvm: invvpid failed: ext=0x%lx vpid=%u gva=0x%lx\n",
471+ vmx_insn_failed("invvpid failed: ext=0x%lx vpid=%u gva=0x%lx\n",
471472 ext, vpid, gva);
472473 }
473474
474475 noinline void invept_error(unsigned long ext, u64 eptp, gpa_t gpa)
475476 {
476− vmx_insn_failed("kvm: invept failed: ext=0x%lx eptp=%llx gpa=0x%llx\n",
477+ vmx_insn_failed("invept failed: ext=0x%lx eptp=%llx gpa=0x%llx\n",
477478 ext, eptp, gpa);
478479 }
479480
@@ -488,8 +489,8 @@ static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu); · 8 unmodified lines
488489 static DECLARE_BITMAP(vmx_vpid_bitmap, VMX_NR_VPIDS);
489490 static DEFINE_SPINLOCK(vmx_vpid_lock);
490491
491−struct vmcs_config vmcs_config;
492−struct vmx_capability vmx_capability;
492+struct vmcs_config vmcs_config __ro_after_init;
493+struct vmx_capability vmx_capability __ro_after_init;
493494
494495 #define VMX_SEGMENT_FIELD(seg) \
495496 [VCPU_SREG_##seg] = { \
@@ -523,6 +524,8 @@ static inline void vmx_segment_cache_clear(struct vcpu_vmx *vmx) · 27 unmodified lines
523524 static unsigned long host_idt_base;
524525
525526 #if IS_ENABLED(CONFIG_HYPERV)
527+static struct kvm_x86_ops vmx_x86_ops __initdata;
528+
526529 static bool __read_mostly enlightened_vmcs = true;
527530 module_param(enlightened_vmcs, bool, 0444);
528531
@@ -551,6 +554,71 @@ static int hv_enable_l2_tlb_flush(struct kvm_vcpu *vcpu) · 22 unmodified lines
551554 return 0;
552555 }
553556
557+static __init void hv_init_evmcs(void)
558+{
559+ int cpu;
560+
561+ if (!enlightened_vmcs)
562+ return;
563+
564+ /*
565+ * Enlightened VMCS usage should be recommended and the host needs
566+ * to support eVMCS v1 or above.
567+ */
568+ if (ms_hyperv.hints & HV_X64_ENLIGHTENED_VMCS_RECOMMENDED &&
569+ (ms_hyperv.nested_features & HV_X64_ENLIGHTENED_VMCS_VERSION) >=
570+ KVM_EVMCS_VERSION) {
571+
572+ /* Check that we have assist pages on all online CPUs */
573+ for_each_online_cpu(cpu) {
574+ if (!hv_get_vp_assist_page(cpu)) {
575+ enlightened_vmcs = false;
576+ break;
577+ }
578+ }
579+
580+ if (enlightened_vmcs) {
581+ pr_info("Using Hyper-V Enlightened VMCS\n");
582+ static_branch_enable(&enable_evmcs);
583+ }
584+
585+ if (ms_hyperv.nested_features & HV_X64_NESTED_DIRECT_FLUSH)
586+ vmx_x86_ops.enable_l2_tlb_flush
587+ = hv_enable_l2_tlb_flush;
588+
589+ } else {
590+ enlightened_vmcs = false;
591+ }
592+}
593+
594+static void hv_reset_evmcs(void)
595+{
596+ struct hv_vp_assist_page *vp_ap;
597+
598+ if (!static_branch_unlikely(&enable_evmcs))
599+ return;
600+
601+ /*
602+ * KVM should enable eVMCS if and only if all CPUs have a VP assist
603+ * page, and should reject CPU onlining if eVMCS is enabled the CPU
604+ * doesn't have a VP assist page allocated.
605+ */
606+ vp_ap = hv_get_vp_assist_page(smp_processor_id());
607+ if (WARN_ON_ONCE(!vp_ap))
608+ return;
609+
610+ /*
611+ * Reset everything to support using non-enlightened VMCS access later
612+ * (e.g. when we reload the module with enlightened_vmcs=0)
613+ */
614+ vp_ap->nested_control.features.directhypercall = 0;
615+ vp_ap->current_nested_vmcs = 0;
616+ vp_ap->enlighten_vmentry = 0;
617+}
618+
619+#else /* IS_ENABLED(CONFIG_HYPERV) */
620+static void hv_init_evmcs(void) {}
621+static void hv_reset_evmcs(void) {}
554622 #endif /* IS_ENABLED(CONFIG_HYPERV) */
555623
556624 /*
@@ -1613,8 +1681,8 @@ static int skip_emulated_instruction(struct kvm_vcpu *vcpu) · 1056 unmodified lines
16131681 if (!instr_len)
16141682 goto rip_updated;
16151683
1616− WARN(exit_reason.enclave_mode,
1617− "KVM: skipping instruction after SGX enclave VM-Exit");
1684+ WARN_ONCE(exit_reason.enclave_mode,
1685+ "skipping instruction after SGX enclave VM-Exit");
16181686
16191687 orig_rip = kvm_rip_read(vcpu);
16201688 rip = orig_rip + instr_len;
@@ -2138,9 +2206,7 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 517 unmodified lines
21382206
21392207 invalid = data & ~vmx_get_supported_debugctl(vcpu, msr_info->host_initiated);
21402208 if (invalid & (DEBUGCTLMSR_BTF|DEBUGCTLMSR_LBR)) {
2141− if (report_ignored_msrs)
2142− vcpu_unimpl(vcpu, "%s: BTF|LBR in IA32_DEBUGCTLMSR 0x%llx, nop\n",
2143− __func__, data);
2209+ kvm_pr_unimpl_wrmsr(vcpu, msr_index, data);
21442210 data &= ~(DEBUGCTLMSR_BTF|DEBUGCTLMSR_LBR);
21452211 invalid &= ~(DEBUGCTLMSR_BTF|DEBUGCTLMSR_LBR);
21462212 }
@@ -2448,88 +2514,6 @@ static void vmx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg) · 301 unmodified lines
24482514 }
24492515 }
24502516
2451−static __init int cpu_has_kvm_support(void)
2452−{
2453− return cpu_has_vmx();
2454−}
2455−
2456−static __init int vmx_disabled_by_bios(void)
2457−{
2458− return !boot_cpu_has(X86_FEATURE_MSR_IA32_FEAT_CTL) ||
2459− !boot_cpu_has(X86_FEATURE_VMX);
2460−}
2461−
2462−static int kvm_cpu_vmxon(u64 vmxon_pointer)
2463−{
2464− u64 msr;
2465−
2466− cr4_set_bits(X86_CR4_VMXE);
2467−
2468− asm_volatile_goto("1: vmxon %[vmxon_pointer]\n\t"
2469− _ASM_EXTABLE(1b, %l[fault])
2470− : : [vmxon_pointer] "m"(vmxon_pointer)
2471− : : fault);
2472− return 0;
2473−
2474−fault:
2475− WARN_ONCE(1, "VMXON faulted, MSR_IA32_FEAT_CTL (0x3a) = 0x%llx\n",
2476− rdmsrl_safe(MSR_IA32_FEAT_CTL, &msr) ? 0xdeadbeef : msr);
2477− cr4_clear_bits(X86_CR4_VMXE);
2478−
2479− return -EFAULT;
2480−}
2481−
2482−static int vmx_hardware_enable(void)
2483−{
2484− int cpu = raw_smp_processor_id();
2485− u64 phys_addr = __pa(per_cpu(vmxarea, cpu));
2486− int r;
2487−
2488− if (cr4_read_shadow() & X86_CR4_VMXE)
2489− return -EBUSY;
2490−
2491− /*
2492− * This can happen if we hot-added a CPU but failed to allocate
2493− * VP assist page for it.
2494− */
2495− if (static_branch_unlikely(&enable_evmcs) &&
2496− !hv_get_vp_assist_page(cpu))
2497− return -EFAULT;
2498−
2499− intel_pt_handle_vmx(1);
2500−
2501− r = kvm_cpu_vmxon(phys_addr);
2502− if (r) {
2503− intel_pt_handle_vmx(0);
2504− return r;
2505− }
2506−
2507− if (enable_ept)
2508− ept_sync_global();
2509−
2510− return 0;
2511−}
2512−
2513−static void vmclear_local_loaded_vmcss(void)
2514−{
2515− int cpu = raw_smp_processor_id();
2516− struct loaded_vmcs *v, *n;
2517−
2518− list_for_each_entry_safe(v, n, &per_cpu(loaded_vmcss_on_cpu, cpu),
2519− loaded_vmcss_on_cpu_link)
2520− __loaded_vmcs_clear(v);
2521−}
2522−
2523−static void vmx_hardware_disable(void)
2524−{
2525− vmclear_local_loaded_vmcss();
2526−
2527− if (cpu_vmxoff())
2528− kvm_spurious_fault();
2529−
2530− intel_pt_handle_vmx(0);
2531−}
2532−
25332517 /*
25342518 * There is no X86_FEATURE for SGX yet, but anyway we need to query CPUID
25352519 * directly instead of going through cpu_has(), to ensure KVM is trapping
@@ -2565,8 +2549,7 @@ static bool cpu_has_perf_global_ctrl_bug(void) · 29 unmodified lines
25652549 return false;
25662550 }
25672551
2568−static __init int adjust_vmx_controls(u32 ctl_min, u32 ctl_opt,
2569− u32 msr, u32 *result)
2552+static int adjust_vmx_controls(u32 ctl_min, u32 ctl_opt, u32 msr, u32 *result)
25702553 {
25712554 u32 vmx_msr_low, vmx_msr_high;
25722555 u32 ctl = ctl_min | ctl_opt;
@@ -2584,7 +2567,7 @@ static __init int adjust_vmx_controls(u32 ctl_min, u32 ctl_opt, · 11 unmodified lines
25842567 return 0;
25852568 }
25862569
2587−static __init u64 adjust_vmx_controls64(u64 ctl_opt, u32 msr)
2570+static u64 adjust_vmx_controls64(u64 ctl_opt, u32 msr)
25882571 {
25892572 u64 allowed;
25902573
@@ -2593,8 +2576,8 @@ static __init u64 adjust_vmx_controls64(u64 ctl_opt, u32 msr) · 2 unmodified lines
25932576 return ctl_opt & allowed;
25942577 }
25952578
2596−static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
2597− struct vmx_capability *vmx_cap)
2579+static int setup_vmcs_config(struct vmcs_config *vmcs_conf,
2580+ struct vmx_capability *vmx_cap)
25982581 {
25992582 u32 vmx_msr_low, vmx_msr_high;
26002583 u32 _pin_based_exec_control = 0;
@@ -2752,9 +2735,127 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf, · 151 unmodified lines
27522735 vmcs_conf->vmentry_ctrl = _vmentry_control;
27532736 vmcs_conf->misc = misc_msr;
27542737
2738+#if IS_ENABLED(CONFIG_HYPERV)
2739+ if (enlightened_vmcs)
2740+ evmcs_sanitize_exec_ctrls(vmcs_conf);
2741+#endif
2742+
2743+ return 0;
2744+}
2745+
2746+static bool kvm_is_vmx_supported(void)
2747+{
2748+ int cpu = raw_smp_processor_id();
2749+
2750+ if (!cpu_has_vmx()) {
2751+ pr_err("VMX not supported by CPU %d\n", cpu);
2752+ return false;
2753+ }
2754+
2755+ if (!this_cpu_has(X86_FEATURE_MSR_IA32_FEAT_CTL) ||
2756+ !this_cpu_has(X86_FEATURE_VMX)) {
2757+ pr_err("VMX not enabled (by BIOS) in MSR_IA32_FEAT_CTL on CPU %d\n", cpu);
2758+ return false;
2759+ }
2760+
2761+ return true;
2762+}
2763+
2764+static int vmx_check_processor_compat(void)
2765+{
2766+ int cpu = raw_smp_processor_id();
2767+ struct vmcs_config vmcs_conf;
2768+ struct vmx_capability vmx_cap;
2769+
2770+ if (!kvm_is_vmx_supported())
2771+ return -EIO;
2772+
2773+ if (setup_vmcs_config(&vmcs_conf, &vmx_cap) < 0) {
2774+ pr_err("Failed to setup VMCS config on CPU %d\n", cpu);
2775+ return -EIO;
2776+ }
2777+ if (nested)
2778+ nested_vmx_setup_ctls_msrs(&vmcs_conf, vmx_cap.ept);
2779+ if (memcmp(&vmcs_config, &vmcs_conf, sizeof(struct vmcs_config))) {
2780+ pr_err("Inconsistent VMCS config on CPU %d\n", cpu);
2781+ return -EIO;
2782+ }
2783+ return 0;
2784+}
2785+
2786+static int kvm_cpu_vmxon(u64 vmxon_pointer)
2787+{
2788+ u64 msr;
2789+
2790+ cr4_set_bits(X86_CR4_VMXE);
2791+
2792+ asm_volatile_goto("1: vmxon %[vmxon_pointer]\n\t"
2793+ _ASM_EXTABLE(1b, %l[fault])
2794+ : : [vmxon_pointer] "m"(vmxon_pointer)
2795+ : : fault);
2796+ return 0;
2797+
2798+fault:
2799+ WARN_ONCE(1, "VMXON faulted, MSR_IA32_FEAT_CTL (0x3a) = 0x%llx\n",
2800+ rdmsrl_safe(MSR_IA32_FEAT_CTL, &msr) ? 0xdeadbeef : msr);
2801+ cr4_clear_bits(X86_CR4_VMXE);
2802+
2803+ return -EFAULT;
2804+}
2805+
2806+static int vmx_hardware_enable(void)
2807+{
2808+ int cpu = raw_smp_processor_id();
2809+ u64 phys_addr = __pa(per_cpu(vmxarea, cpu));
2810+ int r;
2811+
2812+ if (cr4_read_shadow() & X86_CR4_VMXE)
2813+ return -EBUSY;
2814+
2815+ /*
2816+ * This can happen if we hot-added a CPU but failed to allocate
2817+ * VP assist page for it.
2818+ */
2819+ if (static_branch_unlikely(&enable_evmcs) &&
2820+ !hv_get_vp_assist_page(cpu))
2821+ return -EFAULT;
2822+
2823+ intel_pt_handle_vmx(1);
2824+
2825+ r = kvm_cpu_vmxon(phys_addr);
2826+ if (r) {
2827+ intel_pt_handle_vmx(0);
2828+ return r;
2829+ }
2830+
2831+ if (enable_ept)
2832+ ept_sync_global();
2833+
27552834 return 0;
27562835 }
27572836
2837+static void vmclear_local_loaded_vmcss(void)
2838+{
2839+ int cpu = raw_smp_processor_id();
2840+ struct loaded_vmcs *v, *n;
2841+
2842+ list_for_each_entry_safe(v, n, &per_cpu(loaded_vmcss_on_cpu, cpu),
2843+ loaded_vmcss_on_cpu_link)
2844+ __loaded_vmcs_clear(v);
2845+}
2846+
2847+static void vmx_hardware_disable(void)
2848+{
2849+ vmclear_local_loaded_vmcss();
2850+
2851+ if (cpu_vmxoff())
2852+ kvm_spurious_fault();
2853+
2854+ hv_reset_evmcs();
2855+
2856+ intel_pt_handle_vmx(0);
2857+}
2858+
27582859 struct vmcs *alloc_vmcs_cpu(bool shadow, int cpu, gfp_t flags)
27592860 {
27602861 int node = cpu_to_node(cpu);
@@ -2950,9 +3051,8 @@ static void fix_rmode_seg(int seg, struct kvm_segment *save) · 189 unmodified lines
29503051 var.type = 0x3;
29513052 var.avl = 0;
29523053 if (save->base & 0xf)
2953− printk_once(KERN_WARNING "kvm: segment base is not "
2954− "paragraph aligned when entering "
2955− "protected mode (seg=%d)", seg);
3054+ pr_warn_once("segment base is not paragraph aligned "
3055+ "when entering protected mode (seg=%d)", seg);
29563056 }
29573057
29583058 vmcs_write16(sf->selector, var.selector);
@@ -2982,8 +3082,7 @@ static void enter_rmode(struct kvm_vcpu *vcpu) · 23 unmodified lines
29823082 * vcpu. Warn the user that an update is overdue.
29833083 */
29843084 if (!kvm_vmx->tss_addr)
2985− printk_once(KERN_WARNING "kvm: KVM_SET_TSS_ADDR need to be "
2986− "called before entering vcpu\n");
3085+ pr_warn_once("KVM_SET_TSS_ADDR needs to be called before running vCPU\n");
29873086
29883087 vmx_segment_cache_clear(vmx);
29893088
@@ -3800,39 +3899,6 @@ static void seg_setup(int seg) · 810 unmodified lines
38003899 vmcs_write32(sf->ar_bytes, ar);
38013900 }
38023901
3803−static int alloc_apic_access_page(struct kvm *kvm)
3804−{
3805− struct page *page;
3806− void __user *hva;
3807− int ret = 0;
3808−
3809− mutex_lock(&kvm->slots_lock);
3810− if (kvm->arch.apic_access_memslot_enabled)
3811− goto out;
3812− hva = __x86_set_memory_region(kvm, APIC_ACCESS_PAGE_PRIVATE_MEMSLOT,
3813− APIC_DEFAULT_PHYS_BASE, PAGE_SIZE);
3814− if (IS_ERR(hva)) {
3815− ret = PTR_ERR(hva);
3816− goto out;
3817− }
3818−
3819− page = gfn_to_page(kvm, APIC_DEFAULT_PHYS_BASE >> PAGE_SHIFT);
3820− if (is_error_page(page)) {
3821− ret = -EFAULT;
3822− goto out;
3823− }
3824−
3825− /*
3826− * Do not pin the page in memory, so that memory hot-unplug
3827− * is able to migrate it.
3828− */
3829− put_page(page);
3830− kvm->arch.apic_access_memslot_enabled = true;
3831−out:
3832− mutex_unlock(&kvm->slots_lock);
3833− return ret;
3834−}
3835−
38363902 int allocate_vpid(void)
38373903 {
38383904 int vpid;
@@ -3865,8 +3931,13 @@ static void vmx_msr_bitmap_l01_changed(struct vcpu_vmx *vmx) · 26 unmodified lines
38653931 * 'Enlightened MSR Bitmap' feature L0 needs to know that MSR
38663932 * bitmap has changed.
38673933 */
3868− if (static_branch_unlikely(&enable_evmcs))
3869− evmcs_touch_msr_bitmap();
3934+ if (IS_ENABLED(CONFIG_HYPERV) && static_branch_unlikely(&enable_evmcs)) {
3935+ struct hv_enlightened_vmcs *evmcs = (void *)vmx->vmcs01.vmcs;
3936+
3937+ if (evmcs->hv_enlightenments_control.msr_bitmap)
3938+ evmcs->hv_clean_fields &=
3939+ ~HV_VMX_ENLIGHTENED_CLEAN_FIELD_MSR_BITMAP;
3940+ }
38703941
38713942 vmx->nested.force_msr_bitmap_recalc = true;
38723943 }
@@ -3947,29 +4018,20 @@ void vmx_enable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type) · 74 unmodified lines
39474018 vmx_set_msr_bitmap_write(msr_bitmap, msr);
39484019 }
39494020
3950−static void vmx_reset_x2apic_msrs(struct kvm_vcpu *vcpu, u8 mode)
3951−{
3952− unsigned long *msr_bitmap = to_vmx(vcpu)->vmcs01.msr_bitmap;
3953− unsigned long read_intercept;
3954− int msr;
3955−
3956− read_intercept = (mode & MSR_BITMAP_MODE_X2APIC_APICV) ? 0 : ~0;
3957−
3958− for (msr = 0x800; msr <= 0x8ff; msr += BITS_PER_LONG) {
3959− unsigned int read_idx = msr / BITS_PER_LONG;
3960− unsigned int write_idx = read_idx + (0x800 / sizeof(long));
3961−
3962− msr_bitmap[read_idx] = read_intercept;
3963− msr_bitmap[write_idx] = ~0ul;
3964− }
3965−}
3966−
39674021 static void vmx_update_msr_bitmap_x2apic(struct kvm_vcpu *vcpu)
39684022 {
4023+ /*
4024+ * x2APIC indices for 64-bit accesses into the RDMSR and WRMSR halves
4025+ * of the MSR bitmap. KVM emulates APIC registers up through 0x3f0,
4026+ * i.e. MSR 0x83f, and so only needs to dynamically manipulate 64 bits.
4027+ */
4028+ const int read_idx = APIC_BASE_MSR / BITS_PER_LONG_LONG;
4029+ const int write_idx = read_idx + (0x800 / sizeof(u64));
39694030 struct vcpu_vmx *vmx = to_vmx(vcpu);
4031+ u64 *msr_bitmap = (u64 *)vmx->vmcs01.msr_bitmap;
39704032 u8 mode;
39714033
3972− if (!cpu_has_vmx_msr_bitmap())
4034+ if (!cpu_has_vmx_msr_bitmap() || WARN_ON_ONCE(!lapic_in_kernel(vcpu)))
39734035 return;
39744036
39754037 if (cpu_has_secondary_exec_ctrls() &&
@@ -3987,7 +4049,18 @@ static void vmx_update_msr_bitmap_x2apic(struct kvm_vcpu *vcpu) · 11 unmodified lines
39874049
39884050 vmx->x2apic_msr_bitmap_mode = mode;
39894051
3990− vmx_reset_x2apic_msrs(vcpu, mode);
4052+ /*
4053+ * Reset the bitmap for MSRs 0x800 - 0x83f. Leave AMD's uber-extended
4054+ * registers (0x840 and above) intercepted, KVM doesn't support them.
4055+ * Intercept all writes by default and poke holes as needed. Pass
4056+ * through reads for all valid registers by default in x2APIC+APICv
4057+ * mode, only the current timer count needs on-demand emulation by KVM.
4058+ */
4059+ if (mode & MSR_BITMAP_MODE_X2APIC_APICV)
4060+ msr_bitmap[read_idx] = ~kvm_lapic_readable_reg_mask(vcpu->arch.apic);
4061+ else
4062+ msr_bitmap[read_idx] = ~0ull;
4063+ msr_bitmap[write_idx] = ~0ull;
39914064
39924065 /*
39934066 * TPR reads and writes can be virtualized even if virtual interrupt
@@ -4519,6 +4592,12 @@ static u32 vmx_secondary_exec_control(struct vcpu_vmx *vmx) · 525 unmodified lines
45194592 SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY);
45204593 exec_control &= ~SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE;
45214594
4595+ /*
4596+ * KVM doesn't support VMFUNC for L1, but the control is set in KVM's
4597+ * base configuration as KVM emulates VMFUNC[EPTP_SWITCHING] for L2.
4598+ */
4599+ exec_control &= ~SECONDARY_EXEC_ENABLE_VMFUNC;
4600+
45224601 /* SECONDARY_EXEC_DESC is enabled/disabled on writes to CR4.UMIP,
45234602 * in vmx_set_cr4. */
45244603 exec_control &= ~SECONDARY_EXEC_DESC;
@@ -4535,7 +4614,7 @@ static u32 vmx_secondary_exec_control(struct vcpu_vmx *vmx) · 10 unmodified lines
45354614 * it needs to be set here when dirty logging is already active, e.g.
45364615 * if this vCPU was created after dirty logging was enabled.
45374616 */
4538− if (!vcpu->kvm->arch.cpu_dirty_logging_count)
4617+ if (!enable_pml || !atomic_read(&vcpu->kvm->nr_memslots_dirty_logging))
45394618 exec_control &= ~SECONDARY_EXEC_ENABLE_PML;
45404619
45414620 if (cpu_has_vmx_xsaves()) {
@@ -5099,8 +5178,13 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu) · 557 unmodified lines
50995178 vect_info = vmx->idt_vectoring_info;
51005179 intr_info = vmx_get_intr_info(vcpu);
51015180
5181+ /*
5182+ * Machine checks are handled by handle_exception_irqoff(), or by
5183+ * vmx_vcpu_run() if a #MC occurs on VM-Entry. NMIs are handled by
5184+ * vmx_vcpu_enter_exit().
5185+ */
51025186 if (is_machine_check(intr_info) || is_nmi(intr_info))
5103− return 1; /* handled by handle_exception_nmi_irqoff() */
5187+ return 1;
51045188
51055189 /*
51065190 * Queue the exception here instead of in handle_nm_fault_irqoff().
@@ -6790,17 +6874,8 @@ static void vmx_apicv_post_state_restore(struct kvm_vcpu *vcpu) · 1683 unmodified lines
67906874 memset(vmx->pi_desc.pir, 0, sizeof(vmx->pi_desc.pir));
67916875 }
67926876
6793−void vmx_do_interrupt_nmi_irqoff(unsigned long entry);
6794−
6795−static void handle_interrupt_nmi_irqoff(struct kvm_vcpu *vcpu,
6796− unsigned long entry)
6797−{
6798− bool is_nmi = entry == (unsigned long)asm_exc_nmi_noist;
6799−
6800− kvm_before_interrupt(vcpu, is_nmi ? KVM_HANDLING_NMI : KVM_HANDLING_IRQ);
6801− vmx_do_interrupt_nmi_irqoff(entry);
6802− kvm_after_interrupt(vcpu);
6803−}
6877+void vmx_do_interrupt_irqoff(unsigned long entry);
6878+void vmx_do_nmi_irqoff(void);
68046879
68056880 static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu)
68066881 {
@@ -6822,9 +6897,8 @@ static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu) · 15 unmodified lines
68226897 rdmsrl(MSR_IA32_XFD_ERR, vcpu->arch.guest_fpu.xfd_err);
68236898 }
68246899
6825−static void handle_exception_nmi_irqoff(struct vcpu_vmx *vmx)
6900+static void handle_exception_irqoff(struct vcpu_vmx *vmx)
68266901 {
6827− const unsigned long nmi_entry = (unsigned long)asm_exc_nmi_noist;
68286902 u32 intr_info = vmx_get_intr_info(&vmx->vcpu);
68296903
68306904 /* if exit due to PF check for async PF */
@@ -6836,9 +6910,6 @@ static void handle_exception_nmi_irqoff(struct vcpu_vmx *vmx) · 5 unmodified lines
68366910 /* Handle machine checks before interrupts are enabled */
68376911 else if (is_machine_check(intr_info))
68386912 kvm_machine_check();
6839− /* We need to handle NMIs before interrupts are enabled */
6840− else if (is_nmi(intr_info))
6841− handle_interrupt_nmi_irqoff(&vmx->vcpu, nmi_entry);
68426913 }
68436914
68446915 static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu)
@@ -6848,10 +6919,13 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu) · 3 unmodified lines
68486919 gate_desc *desc = (gate_desc *)host_idt_base + vector;
68496920
68506921 if (KVM_BUG(!is_external_intr(intr_info), vcpu->kvm,
6851− "KVM: unexpected VM-Exit interrupt info: 0x%x", intr_info))
6922+ "unexpected VM-Exit interrupt info: 0x%x", intr_info))
68526923 return;
68536924
6854− handle_interrupt_nmi_irqoff(vcpu, gate_offset(desc));
6925+ kvm_before_interrupt(vcpu, KVM_HANDLING_IRQ);
6926+ vmx_do_interrupt_irqoff(gate_offset(desc));
6927+ kvm_after_interrupt(vcpu);
6928+
68556929 vcpu->arch.at_instruction_boundary = true;
68566930 }
68576931
@@ -6865,7 +6939,7 @@ static void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu) · 7 unmodified lines
68656939 if (vmx->exit_reason.basic == EXIT_REASON_EXTERNAL_INTERRUPT)
68666940 handle_external_interrupt_irqoff(vcpu);
68676941 else if (vmx->exit_reason.basic == EXIT_REASON_EXCEPTION_NMI)
6868− handle_exception_nmi_irqoff(vmx);
6942+ handle_exception_irqoff(vmx);
68696943 }
68706944
68716945 /*
@@ -7100,9 +7174,10 @@ static fastpath_t vmx_exit_handlers_fastpath(struct kvm_vcpu *vcpu) · 228 unmodified lines
71007174 }
71017175
71027176 static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu,
7103− struct vcpu_vmx *vmx,
7104− unsigned long flags)
7177+ unsigned int flags)
71057178 {
7179+ struct vcpu_vmx *vmx = to_vmx(vcpu);
7180+
71067181 guest_state_enter_irqoff();
71077182
71087183 /* L1D Flush includes CPU buffer clear to mitigate MDS */
@@ -7126,6 +7201,18 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, · 17 unmodified lines
71267201
71277202 vmx_enable_fb_clear(vmx);
71287203
7204+ if (unlikely(vmx->fail))
7205+ vmx->exit_reason.full = 0xdead;
7206+ else
7207+ vmx->exit_reason.full = vmcs_read32(VM_EXIT_REASON);
7208+
7209+ if ((u16)vmx->exit_reason.basic == EXIT_REASON_EXCEPTION_NMI &&
7210+ is_nmi(vmx_get_intr_info(vcpu))) {
7211+ kvm_before_interrupt(vcpu, KVM_HANDLING_NMI);
7212+ vmx_do_nmi_irqoff();
7213+ kvm_after_interrupt(vcpu);
7214+ }
7215+
71297216 guest_state_exit_irqoff();
71307217 }
71317218
@@ -7220,7 +7307,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) · 88 unmodified lines
72207307 kvm_wait_lapic_expire(vcpu);
72217308
72227309 /* The actual VMENTER/EXIT is in the .noinstr.text section. */
7223− vmx_vcpu_enter_exit(vcpu, vmx, __vmx_vcpu_run_flags(vmx));
7310+ vmx_vcpu_enter_exit(vcpu, __vmx_vcpu_run_flags(vmx));
72247311
72257312 /* All fields are clean at this point */
72267313 if (static_branch_unlikely(&enable_evmcs)) {
@@ -7267,12 +7354,9 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) · 40 unmodified lines
72677354
72687355 vmx->idt_vectoring_info = 0;
72697356
7270− if (unlikely(vmx->fail)) {
7271− vmx->exit_reason.full = 0xdead;
7357+ if (unlikely(vmx->fail))
72727358 return EXIT_FASTPATH_NONE;
7273− }
72747359
7275− vmx->exit_reason.full = vmcs_read32(VM_EXIT_REASON);
72767360 if (unlikely((u16)vmx->exit_reason.basic == EXIT_REASON_MCE_DURING_VMENTRY))
72777361 kvm_machine_check();
72787362
@@ -7386,7 +7470,7 @@ static int vmx_vcpu_create(struct kvm_vcpu *vcpu) · 107 unmodified lines
73867470 vmx->loaded_vmcs = &vmx->vmcs01;
73877471
73887472 if (cpu_need_virtualize_apic_accesses(vcpu)) {
7389− err = alloc_apic_access_page(vcpu->kvm);
7473+ err = kvm_alloc_apic_access_page(vcpu->kvm);
73907474 if (err)
73917475 goto free_vmcs;
73927476 }
@@ -7446,29 +7530,6 @@ static int vmx_vm_init(struct kvm *kvm) · 53 unmodified lines
74467530 return 0;
74477531 }
74487532
7449−static int __init vmx_check_processor_compat(void)
7450−{
7451− struct vmcs_config vmcs_conf;
7452− struct vmx_capability vmx_cap;
7453−
7454− if (!this_cpu_has(X86_FEATURE_MSR_IA32_FEAT_CTL) ||
7455− !this_cpu_has(X86_FEATURE_VMX)) {
7456− pr_err("kvm: VMX is disabled on CPU %d\n", smp_processor_id());
7457− return -EIO;
7458− }
7459−
7460− if (setup_vmcs_config(&vmcs_conf, &vmx_cap) < 0)
7461− return -EIO;
7462− if (nested)
7463− nested_vmx_setup_ctls_msrs(&vmcs_conf, vmx_cap.ept);
7464− if (memcmp(&vmcs_config, &vmcs_conf, sizeof(struct vmcs_config)) != 0) {
7465− printk(KERN_ERR "kvm: CPU %d feature inconsistency!\n",
7466− smp_processor_id());
7467− return -EIO;
7468− }
7469− return 0;
7470−}
7471−
74727533 static u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)
74737534 {
74747535 u8 cache;
@@ -7940,17 +8001,20 @@ void vmx_update_cpu_dirty_logging(struct kvm_vcpu *vcpu) · 465 unmodified lines
79408001 {
79418002 struct vcpu_vmx *vmx = to_vmx(vcpu);
79428003
8004+ if (WARN_ON_ONCE(!enable_pml))
8005+ return;
8006+
79438007 if (is_guest_mode(vcpu)) {
79448008 vmx->nested.update_vmcs01_cpu_dirty_logging = true;
79458009 return;
79468010 }
79478011
79488012 /*
7949− * Note, cpu_dirty_logging_count can be changed concurrent with this
8013+ * Note, nr_memslots_dirty_logging can be changed concurrent with this
79508014 * code, but in that case another update request will be made and so
79518015 * the guest will never run with a stale PML value.
79528016 */
7953− if (vcpu->kvm->arch.cpu_dirty_logging_count)
8017+ if (atomic_read(&vcpu->kvm->nr_memslots_dirty_logging))
79548018 secondary_exec_controls_setbit(vmx, SECONDARY_EXEC_ENABLE_PML);
79558019 else
79568020 secondary_exec_controls_clearbit(vmx, SECONDARY_EXEC_ENABLE_PML);
@@ -8048,17 +8112,16 @@ static void vmx_hardware_unsetup(void) · 91 unmodified lines
80488112 free_kvm_area();
80498113 }
80508114
8051−static bool vmx_check_apicv_inhibit_reasons(enum kvm_apicv_inhibit reason)
8052−{
8053− ulong supported = BIT(APICV_INHIBIT_REASON_DISABLE) |
8054− BIT(APICV_INHIBIT_REASON_ABSENT) |
8055− BIT(APICV_INHIBIT_REASON_HYPERV) |
8056− BIT(APICV_INHIBIT_REASON_BLOCKIRQ) |
8057− BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) |
8058− BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED);
8059−
8060− return supported & BIT(reason);
8061−}
8115+#define VMX_REQUIRED_APICV_INHIBITS \
8116+( \
8117+ BIT(APICV_INHIBIT_REASON_DISABLE)| \
8118+ BIT(APICV_INHIBIT_REASON_ABSENT) | \
8119+ BIT(APICV_INHIBIT_REASON_HYPERV) | \
8120+ BIT(APICV_INHIBIT_REASON_BLOCKIRQ) | \
8121+ BIT(APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED) | \
8122+ BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) | \
8123+ BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED) \
8124+)
80628125
80638126 static void vmx_vm_destroy(struct kvm *kvm)
80648127 {
@@ -8068,7 +8131,9 @@ static void vmx_vm_destroy(struct kvm *kvm) · 3 unmodified lines
80688131 }
80698132
80708133 static struct kvm_x86_ops vmx_x86_ops __initdata = {
8071− .name = "kvm_intel",
8134+ .name = KBUILD_MODNAME,
8135+
8136+ .check_processor_compatibility = vmx_check_processor_compat,
80728137
80738138 .hardware_unsetup = vmx_hardware_unsetup,
80748139
@@ -8142,7 +8207,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { · 67 unmodified lines
81428207 .refresh_apicv_exec_ctrl = vmx_refresh_apicv_exec_ctrl,
81438208 .load_eoi_exitmap = vmx_load_eoi_exitmap,
81448209 .apicv_post_state_restore = vmx_apicv_post_state_restore,
8145− .check_apicv_inhibit_reasons = vmx_check_apicv_inhibit_reasons,
8210+ .required_apicv_inhibits = VMX_REQUIRED_APICV_INHIBITS,
81468211 .hwapic_irr_update = vmx_hwapic_irr_update,
81478212 .hwapic_isr_update = vmx_hwapic_isr_update,
81488213 .guest_apic_has_interrupt = vmx_guest_apic_has_interrupt,
@@ -8288,7 +8353,7 @@ static __init int hardware_setup(void) · 139 unmodified lines
82888353 return -EIO;
82898354
82908355 if (cpu_has_perf_global_ctrl_bug())
8291− pr_warn_once("kvm: VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL "
8356+ pr_warn_once("VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL "
82928357 "does not work properly. Using workaround\n");
82938358
82948359 if (boot_cpu_has(X86_FEATURE_NX))
@@ -8296,7 +8361,7 @@ static __init int hardware_setup(void) · 1 unmodified line
82968361
82978362 if (boot_cpu_has(X86_FEATURE_MPX)) {
82988363 rdmsrl(MSR_IA32_BNDCFGS, host_bndcfgs);
8299− WARN_ONCE(host_bndcfgs, "KVM: BNDCFGS in host will be lost");
8364+ WARN_ONCE(host_bndcfgs, "BNDCFGS in host will be lost");
83008365 }
83018366
83028367 if (!cpu_has_vmx_mpx())
@@ -8315,7 +8380,7 @@ static __init int hardware_setup(void) · 12 unmodified lines
83158380
83168381 /* NX support is required for shadow paging. */
83178382 if (!enable_ept && !boot_cpu_has(X86_FEATURE_NX)) {
8318− pr_err_ratelimited("kvm: NX (Execute Disable) not supported\n");
8383+ pr_err_ratelimited("NX (Execute Disable) not supported\n");
83198384 return -EOPNOTSUPP;
83208385 }
83218386
@@ -8467,9 +8532,6 @@ static __init int hardware_setup(void) · 145 unmodified lines
84678532 }
84688533
84698534 static struct kvm_x86_init_ops vmx_init_ops __initdata = {
8470− .cpu_has_kvm_support = cpu_has_kvm_support,
8471− .disabled_by_bios = vmx_disabled_by_bios,
8472− .check_processor_compatibility = vmx_check_processor_compat,
84738535 .hardware_setup = hardware_setup,
84748536 .handle_intel_pt_intr = NULL,
84758537
@@ -8487,41 +8549,23 @@ static void vmx_cleanup_l1d_flush(void) · 11 unmodified lines
84878549 l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_AUTO;
84888550 }
84898551
8490−static void vmx_exit(void)
8552+static void __vmx_exit(void)
84918553 {
8554+ allow_smaller_maxphyaddr = false;
8555+
84928556 #ifdef CONFIG_KEXEC_CORE
84938557 RCU_INIT_POINTER(crash_vmclear_loaded_vmcss, NULL);
84948558 synchronize_rcu();
84958559 #endif
8560+ vmx_cleanup_l1d_flush();
8561+}
84968562
8563+static void vmx_exit(void)
8564+{
84978565 kvm_exit();
8566+ kvm_x86_vendor_exit();
84988567
8499−#if IS_ENABLED(CONFIG_HYPERV)
8500− if (static_branch_unlikely(&enable_evmcs)) {
8501− int cpu;
8502− struct hv_vp_assist_page *vp_ap;
8503− /*
8504− * Reset everything to support using non-enlightened VMCS
8505− * access later (e.g. when we reload the module with
8506− * enlightened_vmcs=0)
8507− */
8508− for_each_online_cpu(cpu) {
8509− vp_ap = hv_get_vp_assist_page(cpu);
8510−
8511− if (!vp_ap)
8512− continue;
8513−
8514− vp_ap->nested_control.features.directhypercall = 0;
8515− vp_ap->current_nested_vmcs = 0;
8516− vp_ap->enlighten_vmentry = 0;
8517− }
8518−
8519− static_branch_disable(&enable_evmcs);
8520− }
8521−#endif
8522− vmx_cleanup_l1d_flush();
8523−
8524− allow_smaller_maxphyaddr = false;
8568+ __vmx_exit();
85258569 }
85268570 module_exit(vmx_exit);
85278571
@@ -8529,56 +8573,29 @@ static int __init vmx_init(void) · 1 unmodified line
85298573 {
85308574 int r, cpu;
85318575
8532−#if IS_ENABLED(CONFIG_HYPERV)
8576+ if (!kvm_is_vmx_supported())
8577+ return -EOPNOTSUPP;
8578+
85338579 /*
8534− * Enlightened VMCS usage should be recommended and the host needs
8535− * to support eVMCS v1 or above. We can also disable eVMCS support
8536− * with module parameter.
8580+ * Note, hv_init_evmcs() touches only VMX knobs, i.e. there's nothing
8581+ * to unwind if a later step fails.
85378582 */
8538− if (enlightened_vmcs &&
8539− ms_hyperv.hints & HV_X64_ENLIGHTENED_VMCS_RECOMMENDED &&
8540− (ms_hyperv.nested_features & HV_X64_ENLIGHTENED_VMCS_VERSION) >=
8541− KVM_EVMCS_VERSION) {
8542−
8543− /* Check that we have assist pages on all online CPUs */
8544− for_each_online_cpu(cpu) {
8545− if (!hv_get_vp_assist_page(cpu)) {
8546− enlightened_vmcs = false;
8547− break;
8548− }
8549− }
8583+ hv_init_evmcs();
85508584
8551− if (enlightened_vmcs) {
8552− pr_info("KVM: vmx: using Hyper-V Enlightened VMCS\n");
8553− static_branch_enable(&enable_evmcs);
8554− }
8555−
8556− if (ms_hyperv.nested_features & HV_X64_NESTED_DIRECT_FLUSH)
8557− vmx_x86_ops.enable_l2_tlb_flush
8558− = hv_enable_l2_tlb_flush;
8559−
8560− } else {
8561− enlightened_vmcs = false;
8562− }
8563−#endif
8564−
8565− r = kvm_init(&vmx_init_ops, sizeof(struct vcpu_vmx),
8566− __alignof__(struct vcpu_vmx), THIS_MODULE);
8585+ r = kvm_x86_vendor_init(&vmx_init_ops);
85678586 if (r)
85688587 return r;
85698588
85708589 /*
8571− * Must be called after kvm_init() so enable_ept is properly set
8590+ * Must be called after common x86 init so enable_ept is properly set
85728591 * up. Hand the parameter mitigation value in which was stored in
85738592 * the pre module init parser. If no parameter was given, it will
85748593 * contain 'auto' which will be turned into the default 'cond'
85758594 * mitigation mode.
85768595 */
85778596 r = vmx_setup_l1d_flush(vmentry_l1d_flush_param);
8578− if (r) {
8579− vmx_exit();
8580− return r;
8581− }
8597+ if (r)
8598+ goto err_l1d_flush;
85828599
85838600 vmx_setup_fb_clear_ctrl();
85848601
@@ -8602,6 +8619,21 @@ static int __init vmx_init(void) · 17 unmodified lines
86028619 if (!enable_ept)
86038620 allow_smaller_maxphyaddr = true;
86048621
8622+ /*
8623+ * Common KVM initialization _must_ come last, after this, /dev/kvm is
8624+ * exposed to userspace!
8625+ */
8626+ r = kvm_init(sizeof(struct vcpu_vmx), __alignof__(struct vcpu_vmx),
8627+ THIS_MODULE);
8628+ if (r)
8629+ goto err_kvm_init;
8630+
86058631 return 0;
8632+
8633+err_kvm_init:
8634+ __vmx_exit();
8635+err_l1d_flush:
8636+ kvm_x86_vendor_exit();
8637+ return r;
86068638 }
86078639 module_init(vmx_init);
arch/x86/kvm/vmx/vmx.h+8−10
@@ -640,12 +640,12 @@ BUILD_CONTROLS_SHADOW(tertiary_exec, TERTIARY_VM_EXEC_CONTROL, 64) · 639 unmodified lines
640640 (1 << VCPU_EXREG_EXIT_INFO_1) | \
641641 (1 << VCPU_EXREG_EXIT_INFO_2))
642642
643−static inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm)
643+static __always_inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm)
644644 {
645645 return container_of(kvm, struct kvm_vmx, kvm);
646646 }
647647
648−static inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu)
648+static __always_inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu)
649649 {
650650 return container_of(vcpu, struct vcpu_vmx, vcpu);
651651 }
@@ -669,25 +669,23 @@ void intel_pmu_cross_mapped_check(struct kvm_pmu *pmu); · 17 unmodified lines
669669 int intel_pmu_create_guest_lbr_event(struct kvm_vcpu *vcpu);
670670 void vmx_passthrough_lbr_msrs(struct kvm_vcpu *vcpu);
671671
672−static inline unsigned long vmx_get_exit_qual(struct kvm_vcpu *vcpu)
672+static __always_inline unsigned long vmx_get_exit_qual(struct kvm_vcpu *vcpu)
673673 {
674674 struct vcpu_vmx *vmx = to_vmx(vcpu);
675675
676− if (!kvm_register_is_available(vcpu, VCPU_EXREG_EXIT_INFO_1)) {
677− kvm_register_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_1);
676+ if (!kvm_register_test_and_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_1))
678677 vmx->exit_qualification = vmcs_readl(EXIT_QUALIFICATION);
679− }
678+
680679 return vmx->exit_qualification;
681680 }
682681
683−static inline u32 vmx_get_intr_info(struct kvm_vcpu *vcpu)
682+static __always_inline u32 vmx_get_intr_info(struct kvm_vcpu *vcpu)
684683 {
685684 struct vcpu_vmx *vmx = to_vmx(vcpu);
686685
687− if (!kvm_register_is_available(vcpu, VCPU_EXREG_EXIT_INFO_2)) {
688− kvm_register_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_2);
686+ if (!kvm_register_test_and_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_2))
689687 vmx->exit_intr_info = vmcs_read32(VM_EXIT_INTR_INFO);
690− }
688+
691689 return vmx->exit_intr_info;
692690 }
693691
arch/x86/kvm/vmx/vmx_ops.h+4−2
@@ -100,8 +100,10 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field) · 99 unmodified lines
100100 return value;
101101
102102 do_fail:
103− WARN_ONCE(1, "kvm: vmread failed: field=%lx\n", field);
104− pr_warn_ratelimited("kvm: vmread failed: field=%lx\n", field);
103+ instrumentation_begin();
104+ WARN_ONCE(1, KBUILD_MODNAME ": vmread failed: field=%lx\n", field);
105+ pr_warn_ratelimited(KBUILD_MODNAME ": vmread failed: field=%lx\n", field);
106+ instrumentation_end();
105107 return 0;
106108
107109 do_exception:
arch/x86/kvm/x86.c+310−247
@@ -15,6 +15,7 @@ · 14 unmodified lines
1515 * Amit Shah <amit.shah@qumranet.com>
1616 * Ben-Ami Yassour <benami@il.ibm.com>
1717 */
18+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
1819
1920 #include <linux/kvm_host.h>
2021 #include "irq.h"
@@ -128,6 +129,7 @@ static int kvm_vcpu_do_singlestep(struct kvm_vcpu *vcpu); · 107 unmodified lines
128129 static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2);
129130 static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2);
130131
132+static DEFINE_MUTEX(vendor_module_lock);
131133 struct kvm_x86_ops kvm_x86_ops __read_mostly;
132134
133135 #define KVM_X86_OP(func) \
@@ -1421,7 +1423,7 @@ EXPORT_SYMBOL_GPL(kvm_emulate_rdpmc); · 1287 unmodified lines
14211423 * may depend on host virtualization features rather than host cpu features.
14221424 */
14231425
1424−static const u32 msrs_to_save_all[] = {
1426+static const u32 msrs_to_save_base[] = {
14251427 MSR_IA32_SYSENTER_CS, MSR_IA32_SYSENTER_ESP, MSR_IA32_SYSENTER_EIP,
14261428 MSR_STAR,
14271429 #ifdef CONFIG_X86_64
@@ -1438,6 +1440,10 @@ static const u32 msrs_to_save_all[] = { · 10 unmodified lines
14381440 MSR_IA32_RTIT_ADDR3_A, MSR_IA32_RTIT_ADDR3_B,
14391441 MSR_IA32_UMWAIT_CONTROL,
14401442
1443+ MSR_IA32_XFD, MSR_IA32_XFD_ERR,
1444+};
1445+
1446+static const u32 msrs_to_save_pmu[] = {
14411447 MSR_ARCH_PERFMON_FIXED_CTR0, MSR_ARCH_PERFMON_FIXED_CTR1,
14421448 MSR_ARCH_PERFMON_FIXED_CTR0 + 2,
14431449 MSR_CORE_PERF_FIXED_CTR_CTRL, MSR_CORE_PERF_GLOBAL_STATUS,
@@ -1462,11 +1468,10 @@ static const u32 msrs_to_save_all[] = { · 18 unmodified lines
14621468 MSR_F15H_PERF_CTL3, MSR_F15H_PERF_CTL4, MSR_F15H_PERF_CTL5,
14631469 MSR_F15H_PERF_CTR0, MSR_F15H_PERF_CTR1, MSR_F15H_PERF_CTR2,
14641470 MSR_F15H_PERF_CTR3, MSR_F15H_PERF_CTR4, MSR_F15H_PERF_CTR5,
1465−
1466− MSR_IA32_XFD, MSR_IA32_XFD_ERR,
14671471 };
14681472
1469−static u32 msrs_to_save[ARRAY_SIZE(msrs_to_save_all)];
1473+static u32 msrs_to_save[ARRAY_SIZE(msrs_to_save_base) +
1474+ ARRAY_SIZE(msrs_to_save_pmu)];
14701475 static unsigned num_msrs_to_save;
14711476
14721477 static const u32 emulated_msrs_all[] = {
@@ -1484,7 +1489,7 @@ static const u32 emulated_msrs_all[] = { · 11 unmodified lines
14841489 HV_X64_MSR_STIMER0_CONFIG,
14851490 HV_X64_MSR_VP_ASSIST_PAGE,
14861491 HV_X64_MSR_REENLIGHTENMENT_CONTROL, HV_X64_MSR_TSC_EMULATION_CONTROL,
1487− HV_X64_MSR_TSC_EMULATION_STATUS,
1492+ HV_X64_MSR_TSC_EMULATION_STATUS, HV_X64_MSR_TSC_INVARIANT_CONTROL,
14881493 HV_X64_MSR_SYNDBG_OPTIONS,
14891494 HV_X64_MSR_SYNDBG_CONTROL, HV_X64_MSR_SYNDBG_STATUS,
14901495 HV_X64_MSR_SYNDBG_SEND_BUFFER, HV_X64_MSR_SYNDBG_RECV_BUFFER,
@@ -2093,7 +2098,7 @@ static int kvm_emulate_monitor_mwait(struct kvm_vcpu *vcpu, const char *insn) · 602 unmodified lines
20932098 !guest_cpuid_has(vcpu, X86_FEATURE_MWAIT))
20942099 return kvm_handle_invalid_op(vcpu);
20952100
2096− pr_warn_once("kvm: %s instruction emulated as NOP!\n", insn);
2101+ pr_warn_once("%s instruction emulated as NOP!\n", insn);
20972102 return kvm_emulate_as_nop(vcpu);
20982103 }
20992104 int kvm_emulate_mwait(struct kvm_vcpu *vcpu)
@@ -2440,7 +2445,8 @@ static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz) · 340 unmodified lines
24402445 thresh_lo = adjust_tsc_khz(tsc_khz, -tsc_tolerance_ppm);
24412446 thresh_hi = adjust_tsc_khz(tsc_khz, tsc_tolerance_ppm);
24422447 if (user_tsc_khz < thresh_lo || user_tsc_khz > thresh_hi) {
2443− pr_debug("kvm: requested TSC rate %u falls outside tolerance [%u,%u]\n", user_tsc_khz, thresh_lo, thresh_hi);
2448+ pr_debug("requested TSC rate %u falls outside tolerance [%u,%u]\n",
2449+ user_tsc_khz, thresh_lo, thresh_hi);
24442450 use_scaling = 1;
24452451 }
24462452 return set_tsc_khz(vcpu, user_tsc_khz, use_scaling);
@@ -3165,6 +3171,7 @@ static int kvm_guest_time_update(struct kvm_vcpu *v) · 718 unmodified lines
31653171 &vcpu->hv_clock.tsc_shift,
31663172 &vcpu->hv_clock.tsc_to_system_mul);
31673173 vcpu->hw_tsc_khz = tgt_tsc_khz;
3174+ kvm_xen_update_tsc_info(v);
31683175 }
31693176
31703177 vcpu->hv_clock.tsc_timestamp = tsc_timestamp;
@@ -3562,9 +3569,20 @@ static void record_steal_time(struct kvm_vcpu *vcpu) · 391 unmodified lines
35623569 mark_page_dirty_in_slot(vcpu->kvm, ghc->memslot, gpa_to_gfn(ghc->gpa));
35633570 }
35643571
3572+static bool kvm_is_msr_to_save(u32 msr_index)
3573+{
3574+ unsigned int i;
3575+
3576+ for (i = 0; i < num_msrs_to_save; i++) {
3577+ if (msrs_to_save[i] == msr_index)
3578+ return true;
3579+ }
3580+
3581+ return false;
3582+}
3583+
35653584 int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
35663585 {
3567− bool pr = false;
35683586 u32 msr = msr_info->index;
35693587 u64 data = msr_info->data;
35703588
@@ -3610,15 +3628,13 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 39 unmodified lines
36103628 if (data == BIT_ULL(18)) {
36113629 vcpu->arch.msr_hwcr = data;
36123630 } else if (data != 0) {
3613− vcpu_unimpl(vcpu, "unimplemented HWCR wrmsr: 0x%llx\n",
3614− data);
3631+ kvm_pr_unimpl_wrmsr(vcpu, msr, data);
36153632 return 1;
36163633 }
36173634 break;
36183635 case MSR_FAM10H_MMIO_CONF_BASE:
36193636 if (data != 0) {
3620− vcpu_unimpl(vcpu, "unimplemented MMIO_CONF_BASE wrmsr: "
3621− "0x%llx\n", data);
3637+ kvm_pr_unimpl_wrmsr(vcpu, msr, data);
36223638 return 1;
36233639 }
36243640 break;
@@ -3798,16 +3814,13 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 173 unmodified lines
37983814
37993815 case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3:
38003816 case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1:
3801− pr = true;
3802− fallthrough;
38033817 case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3:
38043818 case MSR_P6_EVNTSEL0 ... MSR_P6_EVNTSEL1:
38053819 if (kvm_pmu_is_valid_msr(vcpu, msr))
38063820 return kvm_pmu_set_msr(vcpu, msr_info);
38073821
3808− if (pr || data != 0)
3809− vcpu_unimpl(vcpu, "disabled perfctr wrmsr: "
3810− "0x%x data 0x%llx\n", msr, data);
3822+ if (data)
3823+ kvm_pr_unimpl_wrmsr(vcpu, msr, data);
38113824 break;
38123825 case MSR_K7_CLK_CTL:
38133826 /*
@@ -3828,15 +3841,14 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 14 unmodified lines
38283841 case HV_X64_MSR_REENLIGHTENMENT_CONTROL:
38293842 case HV_X64_MSR_TSC_EMULATION_CONTROL:
38303843 case HV_X64_MSR_TSC_EMULATION_STATUS:
3844+ case HV_X64_MSR_TSC_INVARIANT_CONTROL:
38313845 return kvm_hv_set_msr_common(vcpu, msr, data,
38323846 msr_info->host_initiated);
38333847 case MSR_IA32_BBL_CR_CTL3:
38343848 /* Drop writes to this legacy MSR -- see rdmsr
38353849 * counterpart for further detail.
38363850 */
3837− if (report_ignored_msrs)
3838− vcpu_unimpl(vcpu, "ignored wrmsr: 0x%x data 0x%llx\n",
3839− msr, data);
3851+ kvm_pr_unimpl_wrmsr(vcpu, msr, data);
38403852 break;
38413853 case MSR_AMD64_OSVW_ID_LENGTH:
38423854 if (!guest_cpuid_has(vcpu, X86_FEATURE_OSVW))
@@ -3884,20 +3896,18 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 41 unmodified lines
38843896 vcpu->arch.guest_fpu.xfd_err = data;
38853897 break;
38863898 #endif
3887− case MSR_IA32_PEBS_ENABLE:
3888− case MSR_IA32_DS_AREA:
3889− case MSR_PEBS_DATA_CFG:
3890− case MSR_F15H_PERF_CTL0 ... MSR_F15H_PERF_CTR5:
3899+ default:
38913900 if (kvm_pmu_is_valid_msr(vcpu, msr))
38923901 return kvm_pmu_set_msr(vcpu, msr_info);
3902+
38933903 /*
38943904 * Userspace is allowed to write '0' to MSRs that KVM reports
38953905 * as to-be-saved, even if an MSRs isn't fully supported.
38963906 */
3897− return !msr_info->host_initiated || data;
3898− default:
3899− if (kvm_pmu_is_valid_msr(vcpu, msr))
3900− return kvm_pmu_set_msr(vcpu, msr_info);
3907+ if (msr_info->host_initiated && !data &&
3908+ kvm_is_msr_to_save(msr))
3909+ break;
3910+
39013911 return KVM_MSR_RET_INVALID;
39023912 }
39033913 return 0;
@@ -3987,20 +3997,6 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 83 unmodified lines
39873997 case MSR_DRAM_ENERGY_STATUS: /* DRAM controller */
39883998 msr_info->data = 0;
39893999 break;
3990− case MSR_IA32_PEBS_ENABLE:
3991− case MSR_IA32_DS_AREA:
3992− case MSR_PEBS_DATA_CFG:
3993− case MSR_F15H_PERF_CTL0 ... MSR_F15H_PERF_CTR5:
3994− if (kvm_pmu_is_valid_msr(vcpu, msr_info->index))
3995− return kvm_pmu_get_msr(vcpu, msr_info);
3996− /*
3997− * Userspace is allowed to read MSRs that KVM reports as
3998− * to-be-saved, even if an MSR isn't fully supported.
3999− */
4000− if (!msr_info->host_initiated)
4001− return 1;
4002− msr_info->data = 0;
4003− break;
40044000 case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3:
40054001 case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3:
40064002 case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1:
@@ -4198,6 +4194,7 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 191 unmodified lines
41984194 case HV_X64_MSR_REENLIGHTENMENT_CONTROL:
41994195 case HV_X64_MSR_TSC_EMULATION_CONTROL:
42004196 case HV_X64_MSR_TSC_EMULATION_STATUS:
4197+ case HV_X64_MSR_TSC_INVARIANT_CONTROL:
42014198 return kvm_hv_get_msr_common(vcpu,
42024199 msr_info->index, &msr_info->data,
42034200 msr_info->host_initiated);
@@ -4255,6 +4252,17 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 51 unmodified lines
42554252 default:
42564253 if (kvm_pmu_is_valid_msr(vcpu, msr_info->index))
42574254 return kvm_pmu_get_msr(vcpu, msr_info);
4255+
4256+ /*
4257+ * Userspace is allowed to read MSRs that KVM reports as
4258+ * to-be-saved, even if an MSR isn't fully supported.
4259+ */
4260+ if (msr_info->host_initiated &&
4261+ kvm_is_msr_to_save(msr_info->index)) {
4262+ msr_info->data = 0;
4263+ break;
4264+ }
4265+
42584266 return KVM_MSR_RET_INVALID;
42594267 }
42604268 return 0;
@@ -4292,8 +4300,8 @@ static int msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs, · 31 unmodified lines
42924300 {
42934301 struct kvm_msrs msrs;
42944302 struct kvm_msr_entry *entries;
4295− int r, n;
42964303 unsigned size;
4304+ int r;
42974305
42984306 r = -EFAULT;
42994307 if (copy_from_user(&msrs, user_msrs, sizeof(msrs)))
@@ -4310,17 +4318,11 @@ static int msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs, · 10 unmodified lines
43104318 goto out;
43114319 }
43124320
4313− r = n = __msr_io(vcpu, &msrs, entries, do_msr);
4314− if (r < 0)
4315− goto out_free;
4321+ r = __msr_io(vcpu, &msrs, entries, do_msr);
43164322
4317− r = -EFAULT;
43184323 if (writeback && copy_to_user(user_msrs->entries, entries, size))
4319− goto out_free;
4320−
4321− r = n;
4324+ r = -EFAULT;
43224325
4323−out_free:
43244326 kfree(entries);
43254327 out:
43264328 return r;
@@ -4408,6 +4410,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · 81 unmodified lines
44084410 case KVM_CAP_SPLIT_IRQCHIP:
44094411 case KVM_CAP_IMMEDIATE_EXIT:
44104412 case KVM_CAP_PMU_EVENT_FILTER:
4413+ case KVM_CAP_PMU_EVENT_MASKED_EVENTS:
44114414 case KVM_CAP_GET_MSR_FEATURES:
44124415 case KVM_CAP_MSR_PLATFORM_INFO:
44134416 case KVM_CAP_EXCEPTION_PAYLOAD:
@@ -6482,7 +6485,7 @@ static int kvm_vm_ioctl_set_msr_filter(struct kvm *kvm, · 2068 unmodified lines
64826485 struct kvm_x86_msr_filter *new_filter, *old_filter;
64836486 bool default_allow;
64846487 bool empty = true;
6485− int r = 0;
6488+ int r;
64866489 u32 i;
64876490
64886491 if (filter->flags & ~KVM_MSR_FILTER_VALID_MASK)
@@ -6508,17 +6511,14 @@ static int kvm_vm_ioctl_set_msr_filter(struct kvm *kvm, · 19 unmodified lines
65086511 }
65096512
65106513 mutex_lock(&kvm->lock);
6511−
6512− /* The per-VM filter is protected by kvm->lock... */
6513− old_filter = srcu_dereference_check(kvm->arch.msr_filter, &kvm->srcu, 1);
6514−
6515− rcu_assign_pointer(kvm->arch.msr_filter, new_filter);
6514+ old_filter = rcu_replace_pointer(kvm->arch.msr_filter, new_filter,
6515+ mutex_is_locked(&kvm->lock));
6516+ mutex_unlock(&kvm->lock);
65166517 synchronize_srcu(&kvm->srcu);
65176518
65186519 kvm_free_msr_filter(old_filter);
65196520
65206521 kvm_make_all_cpus_request(kvm, KVM_REQ_MSR_FILTER_CHANGED);
6521− mutex_unlock(&kvm->lock);
65226522
65236523 return 0;
65246524 }
@@ -7018,83 +7018,98 @@ out: · 493 unmodified lines
70187018 return r;
70197019 }
70207020
7021−static void kvm_init_msr_list(void)
7021+static void kvm_probe_msr_to_save(u32 msr_index)
70227022 {
70237023 u32 dummy[2];
7024+
7025+ if (rdmsr_safe(msr_index, &dummy[0], &dummy[1]))
7026+ return;
7027+
7028+ /*
7029+ * Even MSRs that are valid in the host may not be exposed to guests in
7030+ * some cases.
7031+ */
7032+ switch (msr_index) {
7033+ case MSR_IA32_BNDCFGS:
7034+ if (!kvm_mpx_supported())
7035+ return;
7036+ break;
7037+ case MSR_TSC_AUX:
7038+ if (!kvm_cpu_cap_has(X86_FEATURE_RDTSCP) &&
7039+ !kvm_cpu_cap_has(X86_FEATURE_RDPID))
7040+ return;
7041+ break;
7042+ case MSR_IA32_UMWAIT_CONTROL:
7043+ if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG))
7044+ return;
7045+ break;
7046+ case MSR_IA32_RTIT_CTL:
7047+ case MSR_IA32_RTIT_STATUS:
7048+ if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT))
7049+ return;
7050+ break;
7051+ case MSR_IA32_RTIT_CR3_MATCH:
7052+ if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) ||
7053+ !intel_pt_validate_hw_cap(PT_CAP_cr3_filtering))
7054+ return;
7055+ break;
7056+ case MSR_IA32_RTIT_OUTPUT_BASE:
7057+ case MSR_IA32_RTIT_OUTPUT_MASK:
7058+ if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) ||
7059+ (!intel_pt_validate_hw_cap(PT_CAP_topa_output) &&
7060+ !intel_pt_validate_hw_cap(PT_CAP_single_range_output)))
7061+ return;
7062+ break;
7063+ case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B:
7064+ if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) ||
7065+ (msr_index - MSR_IA32_RTIT_ADDR0_A >=
7066+ intel_pt_validate_hw_cap(PT_CAP_num_address_ranges) * 2))
7067+ return;
7068+ break;
7069+ case MSR_ARCH_PERFMON_PERFCTR0 ... MSR_ARCH_PERFMON_PERFCTR_MAX:
7070+ if (msr_index - MSR_ARCH_PERFMON_PERFCTR0 >=
7071+ kvm_pmu_cap.num_counters_gp)
7072+ return;
7073+ break;
7074+ case MSR_ARCH_PERFMON_EVENTSEL0 ... MSR_ARCH_PERFMON_EVENTSEL_MAX:
7075+ if (msr_index - MSR_ARCH_PERFMON_EVENTSEL0 >=
7076+ kvm_pmu_cap.num_counters_gp)
7077+ return;
7078+ break;
7079+ case MSR_ARCH_PERFMON_FIXED_CTR0 ... MSR_ARCH_PERFMON_FIXED_CTR_MAX:
7080+ if (msr_index - MSR_ARCH_PERFMON_FIXED_CTR0 >=
7081+ kvm_pmu_cap.num_counters_fixed)
7082+ return;
7083+ break;
7084+ case MSR_IA32_XFD:
7085+ case MSR_IA32_XFD_ERR:
7086+ if (!kvm_cpu_cap_has(X86_FEATURE_XFD))
7087+ return;
7088+ break;
7089+ default:
7090+ break;
7091+ }
7092+
7093+ msrs_to_save[num_msrs_to_save++] = msr_index;
7094+}
7095+
7096+static void kvm_init_msr_list(void)
7097+{
70247098 unsigned i;
70257099
70267100 BUILD_BUG_ON_MSG(KVM_PMC_MAX_FIXED != 3,
7027− "Please update the fixed PMCs in msrs_to_saved_all[]");
7101+ "Please update the fixed PMCs in msrs_to_save_pmu[]");
70287102
70297103 num_msrs_to_save = 0;
70307104 num_emulated_msrs = 0;
70317105 num_msr_based_features = 0;
70327106
7033− for (i = 0; i < ARRAY_SIZE(msrs_to_save_all); i++) {
7034− if (rdmsr_safe(msrs_to_save_all[i], &dummy[0], &dummy[1]) < 0)
7035− continue;
7036−
7037− /*
7038− * Even MSRs that are valid in the host may not be exposed
7039− * to the guests in some cases.
7040− */
7041− switch (msrs_to_save_all[i]) {
7042− case MSR_IA32_BNDCFGS:
7043− if (!kvm_mpx_supported())
7044− continue;
7045− break;
7046− case MSR_TSC_AUX:
7047− if (!kvm_cpu_cap_has(X86_FEATURE_RDTSCP) &&
7048− !kvm_cpu_cap_has(X86_FEATURE_RDPID))
7049− continue;
7050− break;
7051− case MSR_IA32_UMWAIT_CONTROL:
7052− if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG))
7053− continue;
7054− break;
7055− case MSR_IA32_RTIT_CTL:
7056− case MSR_IA32_RTIT_STATUS:
7057− if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT))
7058− continue;
7059− break;
7060− case MSR_IA32_RTIT_CR3_MATCH:
7061− if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) ||
7062− !intel_pt_validate_hw_cap(PT_CAP_cr3_filtering))
7063− continue;
7064− break;
7065− case MSR_IA32_RTIT_OUTPUT_BASE:
7066− case MSR_IA32_RTIT_OUTPUT_MASK:
7067− if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) ||
7068− (!intel_pt_validate_hw_cap(PT_CAP_topa_output) &&
7069− !intel_pt_validate_hw_cap(PT_CAP_single_range_output)))
7070− continue;
7071− break;
7072− case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B:
7073− if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) ||
7074− msrs_to_save_all[i] - MSR_IA32_RTIT_ADDR0_A >=
7075− intel_pt_validate_hw_cap(PT_CAP_num_address_ranges) * 2)
7076− continue;
7077− break;
7078− case MSR_ARCH_PERFMON_PERFCTR0 ... MSR_ARCH_PERFMON_PERFCTR_MAX:
7079− if (msrs_to_save_all[i] - MSR_ARCH_PERFMON_PERFCTR0 >=
7080− min(KVM_INTEL_PMC_MAX_GENERIC, kvm_pmu_cap.num_counters_gp))
7081− continue;
7082− break;
7083− case MSR_ARCH_PERFMON_EVENTSEL0 ... MSR_ARCH_PERFMON_EVENTSEL_MAX:
7084− if (msrs_to_save_all[i] - MSR_ARCH_PERFMON_EVENTSEL0 >=
7085− min(KVM_INTEL_PMC_MAX_GENERIC, kvm_pmu_cap.num_counters_gp))
7086− continue;
7087− break;
7088− case MSR_IA32_XFD:
7089− case MSR_IA32_XFD_ERR:
7090− if (!kvm_cpu_cap_has(X86_FEATURE_XFD))
7091− continue;
7092− break;
7093− default:
7094− break;
7095− }
7107+ for (i = 0; i < ARRAY_SIZE(msrs_to_save_base); i++)
7108+ kvm_probe_msr_to_save(msrs_to_save_base[i]);
70967109
7097− msrs_to_save[num_msrs_to_save++] = msrs_to_save_all[i];
7110+ if (enable_pmu) {
7111+ for (i = 0; i < ARRAY_SIZE(msrs_to_save_pmu); i++)
7112+ kvm_probe_msr_to_save(msrs_to_save_pmu[i]);
70987113 }
70997114
71007115 for (i = 0; i < ARRAY_SIZE(emulated_msrs_all); i++) {
@@ -7721,7 +7736,7 @@ static int emulator_cmpxchg_emulated(struct x86_emulate_ctxt *ctxt, · 620 unmodified lines
77217736 return X86EMUL_CONTINUE;
77227737
77237738 emul_write:
7724− printk_once(KERN_WARNING "kvm: emulating exchange as write\n");
7739+ pr_warn_once("emulating exchange as write\n");
77257740
77267741 return emulator_write_emulated(ctxt, addr, new, bytes, exception);
77277742 }
@@ -8167,9 +8182,14 @@ static void emulator_set_nmi_mask(struct x86_emulate_ctxt *ctxt, bool masked) · 439 unmodified lines
81678182 static_call(kvm_x86_set_nmi_mask)(emul_to_vcpu(ctxt), masked);
81688183 }
81698184
8170−static unsigned emulator_get_hflags(struct x86_emulate_ctxt *ctxt)
8185+static bool emulator_is_smm(struct x86_emulate_ctxt *ctxt)
8186+{
8187+ return is_smm(emul_to_vcpu(ctxt));
8188+}
8189+
8190+static bool emulator_is_guest_mode(struct x86_emulate_ctxt *ctxt)
81718191 {
8172− return emul_to_vcpu(ctxt)->arch.hflags;
8192+ return is_guest_mode(emul_to_vcpu(ctxt));
81738193 }
81748194
81758195 #ifndef CONFIG_KVM_SMM
@@ -8238,7 +8258,8 @@ static const struct x86_emulate_ops emulate_ops = { · 62 unmodified lines
82388258 .guest_has_fxsr = emulator_guest_has_fxsr,
82398259 .guest_has_rdpid = emulator_guest_has_rdpid,
82408260 .set_nmi_mask = emulator_set_nmi_mask,
8241− .get_hflags = emulator_get_hflags,
8261+ .is_smm = emulator_is_smm,
8262+ .is_guest_mode = emulator_is_guest_mode,
82428263 .leave_smm = emulator_leave_smm,
82438264 .triple_fault = emulator_triple_fault,
82448265 .set_xcr = emulator_set_xcr,
@@ -8282,7 +8303,7 @@ static struct x86_emulate_ctxt *alloc_emulate_ctxt(struct kvm_vcpu *vcpu) · 37 unmodified lines
82828303
82838304 ctxt = kmem_cache_zalloc(x86_emulator_cache, GFP_KERNEL_ACCOUNT);
82848305 if (!ctxt) {
8285− pr_err("kvm: failed to allocate vcpu's emulator\n");
8306+ pr_err("failed to allocate vcpu's emulator\n");
82868307 return NULL;
82878308 }
82888309
@@ -8310,8 +8331,6 @@ static void init_emulate_ctxt(struct kvm_vcpu *vcpu) · 21 unmodified lines
83108331 (cs_l && is_long_mode(vcpu)) ? X86EMUL_MODE_PROT64 :
83118332 cs_db ? X86EMUL_MODE_PROT32 :
83128333 X86EMUL_MODE_PROT16;
8313− BUILD_BUG_ON(HF_GUEST_MASK != X86EMUL_GUEST_MASK);
8314−
83158334 ctxt->interruptibility = 0;
83168335 ctxt->have_exception = false;
83178336 ctxt->exception.vector = -1;
@@ -9293,35 +9312,66 @@ static struct notifier_block pvclock_gtod_notifier = { · 975 unmodified lines
92939312 };
92949313 #endif
92959314
9296−int kvm_arch_init(void *opaque)
9315+static inline void kvm_ops_update(struct kvm_x86_init_ops *ops)
9316+{
9317+ memcpy(&kvm_x86_ops, ops->runtime_ops, sizeof(kvm_x86_ops));
9318+
9319+#define __KVM_X86_OP(func) \
9320+ static_call_update(kvm_x86_##func, kvm_x86_ops.func);
9321+#define KVM_X86_OP(func) \
9322+ WARN_ON(!kvm_x86_ops.func); __KVM_X86_OP(func)
9323+#define KVM_X86_OP_OPTIONAL __KVM_X86_OP
9324+#define KVM_X86_OP_OPTIONAL_RET0(func) \
9325+ static_call_update(kvm_x86_##func, (void *)kvm_x86_ops.func ? : \
9326+ (void *)__static_call_return0);
9327+#include <asm/kvm-x86-ops.h>
9328+#undef __KVM_X86_OP
9329+
9330+ kvm_pmu_ops_update(ops->pmu_ops);
9331+}
9332+
9333+static int kvm_x86_check_processor_compatibility(void)
9334+{
9335+ int cpu = smp_processor_id();
9336+ struct cpuinfo_x86 *c = &cpu_data(cpu);
9337+
9338+ /*
9339+ * Compatibility checks are done when loading KVM and when enabling
9340+ * hardware, e.g. during CPU hotplug, to ensure all online CPUs are
9341+ * compatible, i.e. KVM should never perform a compatibility check on
9342+ * an offline CPU.
9343+ */
9344+ WARN_ON(!cpu_online(cpu));
9345+
9346+ if (__cr4_reserved_bits(cpu_has, c) !=
9347+ __cr4_reserved_bits(cpu_has, &boot_cpu_data))
9348+ return -EIO;
9349+
9350+ return static_call(kvm_x86_check_processor_compatibility)();
9351+}
9352+
9353+static void kvm_x86_check_cpu_compat(void *ret)
9354+{
9355+ *(int *)ret = kvm_x86_check_processor_compatibility();
9356+}
9357+
9358+static int __kvm_x86_vendor_init(struct kvm_x86_init_ops *ops)
92979359 {
9298− struct kvm_x86_init_ops *ops = opaque;
92999360 u64 host_pat;
9300− int r;
9361+ int r, cpu;
93019362
93029363 if (kvm_x86_ops.hardware_enable) {
9303− pr_err("kvm: already loaded vendor module '%s'\n", kvm_x86_ops.name);
9364+ pr_err("already loaded vendor module '%s'\n", kvm_x86_ops.name);
93049365 return -EEXIST;
93059366 }
93069367
9307− if (!ops->cpu_has_kvm_support()) {
9308− pr_err_ratelimited("kvm: no hardware support for '%s'\n",
9309− ops->runtime_ops->name);
9310− return -EOPNOTSUPP;
9311− }
9312− if (ops->disabled_by_bios()) {
9313− pr_err_ratelimited("kvm: support for '%s' disabled by bios\n",
9314− ops->runtime_ops->name);
9315− return -EOPNOTSUPP;
9316− }
9317−
93189368 /*
93199369 * KVM explicitly assumes that the guest has an FPU and
93209370 * FXSAVE/FXRSTOR. For example, the KVM_GET_FPU explicitly casts the
93219371 * vCPU's FPU state as a fxregs_state struct.
93229372 */
93239373 if (!boot_cpu_has(X86_FEATURE_FPU) || !boot_cpu_has(X86_FEATURE_FXSR)) {
9324− printk(KERN_ERR "kvm: inadequate fpu\n");
9374+ pr_err("inadequate fpu\n");
93259375 return -EOPNOTSUPP;
93269376 }
93279377
@@ -9339,19 +9389,19 @@ int kvm_arch_init(void *opaque) · 11 unmodified lines
93399389 */
93409390 if (rdmsrl_safe(MSR_IA32_CR_PAT, &host_pat) ||
93419391 (host_pat & GENMASK(2, 0)) != 6) {
9342− pr_err("kvm: host PAT[0] is not WB\n");
9392+ pr_err("host PAT[0] is not WB\n");
93439393 return -EIO;
93449394 }
93459395
93469396 x86_emulator_cache = kvm_alloc_emulator_cache();
93479397 if (!x86_emulator_cache) {
9348− pr_err("kvm: failed to allocate cache for x86 emulator\n");
9398+ pr_err("failed to allocate cache for x86 emulator\n");
93499399 return -ENOMEM;
93509400 }
93519401
93529402 user_return_msrs = alloc_percpu(struct kvm_user_return_msrs);
93539403 if (!user_return_msrs) {
9354− printk(KERN_ERR "kvm: failed to allocate percpu kvm_user_return_msrs\n");
9404+ pr_err("failed to allocate percpu kvm_user_return_msrs\n");
93559405 r = -ENOMEM;
93569406 goto out_free_x86_emulator_cache;
93579407 }
@@ -9361,13 +9411,37 @@ int kvm_arch_init(void *opaque) · 3 unmodified lines
93619411 if (r)
93629412 goto out_free_percpu;
93639413
9364− kvm_timer_init();
9365−
93669414 if (boot_cpu_has(X86_FEATURE_XSAVE)) {
93679415 host_xcr0 = xgetbv(XCR_XFEATURE_ENABLED_MASK);
93689416 kvm_caps.supported_xcr0 = host_xcr0 & KVM_SUPPORTED_XCR0;
93699417 }
93709418
9419+ rdmsrl_safe(MSR_EFER, &host_efer);
9420+
9421+ if (boot_cpu_has(X86_FEATURE_XSAVES))
9422+ rdmsrl(MSR_IA32_XSS, host_xss);
9423+
9424+ kvm_init_pmu_capability(ops->pmu_ops);
9425+
9426+ r = ops->hardware_setup();
9427+ if (r != 0)
9428+ goto out_mmu_exit;
9429+
9430+ kvm_ops_update(ops);
9431+
9432+ for_each_online_cpu(cpu) {
9433+ smp_call_function_single(cpu, kvm_x86_check_cpu_compat, &r, 1);
9434+ if (r < 0)
9435+ goto out_unwind_ops;
9436+ }
9437+
9438+ /*
9439+ * Point of no return! DO NOT add error paths below this point unless
9440+ * absolutely necessary, as most operations from this point forward
9441+ * require unwinding.
9442+ */
9443+ kvm_timer_init();
9444+
93719445 if (pi_inject_timer == -1)
93729446 pi_inject_timer = housekeeping_enabled(HK_TYPE_TIMER);
93739447 #ifdef CONFIG_X86_64
@@ -9377,8 +9451,35 @@ int kvm_arch_init(void *opaque) · 3 unmodified lines
93779451 set_hv_tscchange_cb(kvm_hyperv_tsc_notifier);
93789452 #endif
93799453
9454+ kvm_register_perf_callbacks(ops->handle_intel_pt_intr);
9455+
9456+ if (!kvm_cpu_cap_has(X86_FEATURE_XSAVES))
9457+ kvm_caps.supported_xss = 0;
9458+
9459+#define __kvm_cpu_cap_has(UNUSED_, f) kvm_cpu_cap_has(f)
9460+ cr4_reserved_bits = __cr4_reserved_bits(__kvm_cpu_cap_has, UNUSED_);
9461+#undef __kvm_cpu_cap_has
9462+
9463+ if (kvm_caps.has_tsc_control) {
9464+ /*
9465+ * Make sure the user can only configure tsc_khz values that
9466+ * fit into a signed integer.
9467+ * A min value is not calculated because it will always
9468+ * be 1 on all machines.
9469+ */
9470+ u64 max = min(0x7fffffffULL,
9471+ __scale_tsc(kvm_caps.max_tsc_scaling_ratio, tsc_khz));
9472+ kvm_caps.max_guest_tsc_khz = max;
9473+ }
9474+ kvm_caps.default_tsc_scaling_ratio = 1ULL << kvm_caps.tsc_scaling_ratio_frac_bits;
9475+ kvm_init_msr_list();
93809476 return 0;
93819477
9478+out_unwind_ops:
9479+ kvm_x86_ops.hardware_enable = NULL;
9480+ static_call(kvm_x86_hardware_unsetup)();
9481+out_mmu_exit:
9482+ kvm_mmu_vendor_module_exit();
93829483 out_free_percpu:
93839484 free_percpu(user_return_msrs);
93849485 out_free_x86_emulator_cache:
@@ -9386,8 +9487,22 @@ out_free_x86_emulator_cache: · 1 unmodified line
93869487 return r;
93879488 }
93889489
9389−void kvm_arch_exit(void)
9490+int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops)
9491+{
9492+ int r;
9493+
9494+ mutex_lock(&vendor_module_lock);
9495+ r = __kvm_x86_vendor_init(ops);
9496+ mutex_unlock(&vendor_module_lock);
9497+
9498+ return r;
9499+}
9500+EXPORT_SYMBOL_GPL(kvm_x86_vendor_init);
9501+
9502+void kvm_x86_vendor_exit(void)
93909503 {
9504+ kvm_unregister_perf_callbacks();
9505+
93919506 #ifdef CONFIG_X86_64
93929507 if (hypervisor_is_type(X86_HYPER_MS_HYPERV))
93939508 clear_hv_tscchange_cb();
@@ -9404,7 +9519,7 @@ void kvm_arch_exit(void) · 10 unmodified lines
94049519 irq_work_sync(&pvclock_irq_work);
94059520 cancel_work_sync(&pvclock_gtod_work);
94069521 #endif
9407− kvm_x86_ops.hardware_enable = NULL;
9522+ static_call(kvm_x86_hardware_unsetup)();
94089523 kvm_mmu_vendor_module_exit();
94099524 free_percpu(user_return_msrs);
94109525 kmem_cache_destroy(x86_emulator_cache);
@@ -9412,7 +9527,11 @@ void kvm_arch_exit(void) · 1 unmodified line
94129527 static_key_deferred_flush(&kvm_xen_enabled);
94139528 WARN_ON(static_branch_unlikely(&kvm_xen_enabled.key));
94149529 #endif
9530+ mutex_lock(&vendor_module_lock);
9531+ kvm_x86_ops.hardware_enable = NULL;
9532+ mutex_unlock(&vendor_module_lock);
94159533 }
9534+EXPORT_SYMBOL_GPL(kvm_x86_vendor_exit);
94169535
94179536 static int __kvm_emulate_halt(struct kvm_vcpu *vcpu, int state, int reason)
94189537 {
@@ -10065,7 +10184,7 @@ void kvm_make_scan_ioapic_request(struct kvm *kvm) · 646 unmodified lines
1006510184 kvm_make_all_cpus_request(kvm, KVM_REQ_SCAN_IOAPIC);
1006610185 }
1006710186
10068−void kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu)
10187+void __kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu)
1006910188 {
1007010189 struct kvm_lapic *apic = vcpu->arch.apic;
1007110190 bool activate;
@@ -10100,7 +10219,30 @@ out: · 28 unmodified lines
1010010219 preempt_enable();
1010110220 up_read(&vcpu->kvm->arch.apicv_update_lock);
1010210221 }
10103−EXPORT_SYMBOL_GPL(kvm_vcpu_update_apicv);
10222+EXPORT_SYMBOL_GPL(__kvm_vcpu_update_apicv);
10223+
10224+static void kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu)
10225+{
10226+ if (!lapic_in_kernel(vcpu))
10227+ return;
10228+
10229+ /*
10230+ * Due to sharing page tables across vCPUs, the xAPIC memslot must be
10231+ * deleted if any vCPU has xAPIC virtualization and x2APIC enabled, but
10232+ * and hardware doesn't support x2APIC virtualization. E.g. some AMD
10233+ * CPUs support AVIC but not x2APIC. KVM still allows enabling AVIC in
10234+ * this case so that KVM can the AVIC doorbell to inject interrupts to
10235+ * running vCPUs, but KVM must not create SPTEs for the APIC base as
10236+ * the vCPU would incorrectly be able to access the vAPIC page via MMIO
10237+ * despite being in x2APIC mode. For simplicity, inhibiting the APIC
10238+ * access page is sticky.
10239+ */
10240+ if (apic_x2apic_mode(vcpu->arch.apic) &&
10241+ kvm_x86_ops.allow_apicv_in_x2apic_without_x2apic_virtualization)
10242+ kvm_inhibit_apic_access_page(vcpu);
10243+
10244+ __kvm_vcpu_update_apicv(vcpu);
10245+}
1010410246
1010510247 void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm,
1010610248 enum kvm_apicv_inhibit reason, bool set)
@@ -10109,7 +10251,7 @@ void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm, · 2 unmodified lines
1010910251
1011010252 lockdep_assert_held_write(&kvm->arch.apicv_update_lock);
1011110253
10112− if (!static_call(kvm_x86_check_apicv_inhibit_reasons)(reason))
10254+ if (!(kvm_x86_ops.required_apicv_inhibits & BIT(reason)))
1011310255 return;
1011410256
1011510257 old = new = kvm->arch.apicv_inhibit_reasons;
@@ -11553,7 +11695,7 @@ static int sync_regs(struct kvm_vcpu *vcpu) · 1437 unmodified lines
1155311695 int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)
1155411696 {
1155511697 if (kvm_check_tsc_unstable() && kvm->created_vcpus)
11556− pr_warn_once("kvm: SMP vm created on host with unstable TSC; "
11698+ pr_warn_once("SMP vm created on host with unstable TSC; "
1155711699 "guest TSC will not be reliable\n");
1155811700
1155911701 if (!kvm->arch.max_vcpu_ids)
@@ -11630,7 +11772,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · 70 unmodified lines
1163011772 goto free_wbinvd_dirty_mask;
1163111773
1163211774 if (!fpu_alloc_guest_fpstate(&vcpu->arch.guest_fpu)) {
11633− pr_err("kvm: failed to allocate vcpu's fpu\n");
11775+ pr_err("failed to allocate vcpu's fpu\n");
1163411776 goto free_emulate_ctxt;
1163511777 }
1163611778
@@ -11904,6 +12046,11 @@ int kvm_arch_hardware_enable(void) · 267 unmodified lines
1190412046 bool stable, backwards_tsc = false;
1190512047
1190612048 kvm_user_return_msr_cpu_online();
12049+
12050+ ret = kvm_x86_check_processor_compatibility();
12051+ if (ret)
12052+ return ret;
12053+
1190712054 ret = static_call(kvm_x86_hardware_enable)();
1190812055 if (ret != 0)
1190912056 return ret;
@@ -11990,88 +12137,6 @@ void kvm_arch_hardware_disable(void) · 80 unmodified lines
1199012137 drop_user_return_notifiers();
1199112138 }
1199212139
11993−static inline void kvm_ops_update(struct kvm_x86_init_ops *ops)
11994−{
11995− memcpy(&kvm_x86_ops, ops->runtime_ops, sizeof(kvm_x86_ops));
11996−
11997−#define __KVM_X86_OP(func) \
11998− static_call_update(kvm_x86_##func, kvm_x86_ops.func);
11999−#define KVM_X86_OP(func) \
12000− WARN_ON(!kvm_x86_ops.func); __KVM_X86_OP(func)
12001−#define KVM_X86_OP_OPTIONAL __KVM_X86_OP
12002−#define KVM_X86_OP_OPTIONAL_RET0(func) \
12003− static_call_update(kvm_x86_##func, (void *)kvm_x86_ops.func ? : \
12004− (void *)__static_call_return0);
12005−#include <asm/kvm-x86-ops.h>
12006−#undef __KVM_X86_OP
12007−
12008− kvm_pmu_ops_update(ops->pmu_ops);
12009−}
12010−
12011−int kvm_arch_hardware_setup(void *opaque)
12012−{
12013− struct kvm_x86_init_ops *ops = opaque;
12014− int r;
12015−
12016− rdmsrl_safe(MSR_EFER, &host_efer);
12017−
12018− if (boot_cpu_has(X86_FEATURE_XSAVES))
12019− rdmsrl(MSR_IA32_XSS, host_xss);
12020−
12021− kvm_init_pmu_capability();
12022−
12023− r = ops->hardware_setup();
12024− if (r != 0)
12025− return r;
12026−
12027− kvm_ops_update(ops);
12028−
12029− kvm_register_perf_callbacks(ops->handle_intel_pt_intr);
12030−
12031− if (!kvm_cpu_cap_has(X86_FEATURE_XSAVES))
12032− kvm_caps.supported_xss = 0;
12033−
12034−#define __kvm_cpu_cap_has(UNUSED_, f) kvm_cpu_cap_has(f)
12035− cr4_reserved_bits = __cr4_reserved_bits(__kvm_cpu_cap_has, UNUSED_);
12036−#undef __kvm_cpu_cap_has
12037−
12038− if (kvm_caps.has_tsc_control) {
12039− /*
12040− * Make sure the user can only configure tsc_khz values that
12041− * fit into a signed integer.
12042− * A min value is not calculated because it will always
12043− * be 1 on all machines.
12044− */
12045− u64 max = min(0x7fffffffULL,
12046− __scale_tsc(kvm_caps.max_tsc_scaling_ratio, tsc_khz));
12047− kvm_caps.max_guest_tsc_khz = max;
12048− }
12049− kvm_caps.default_tsc_scaling_ratio = 1ULL << kvm_caps.tsc_scaling_ratio_frac_bits;
12050− kvm_init_msr_list();
12051− return 0;
12052−}
12053−
12054−void kvm_arch_hardware_unsetup(void)
12055−{
12056− kvm_unregister_perf_callbacks();
12057−
12058− static_call(kvm_x86_hardware_unsetup)();
12059−}
12060−
12061−int kvm_arch_check_processor_compat(void *opaque)
12062−{
12063− struct cpuinfo_x86 *c = &cpu_data(smp_processor_id());
12064− struct kvm_x86_init_ops *ops = opaque;
12065−
12066− WARN_ON(!irqs_disabled());
12067−
12068− if (__cr4_reserved_bits(cpu_has, c) !=
12069− __cr4_reserved_bits(cpu_has, &boot_cpu_data))
12070− return -EIO;
12071−
12072− return ops->check_processor_compatibility();
12073−}
12074−
1207512140 bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu)
1207612141 {
1207712142 return vcpu->kvm->arch.bsp_vcpu_id == vcpu->vcpu_id;
@@ -12244,7 +12309,7 @@ void __user * __x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, · 166 unmodified lines
1224412309 */
1224512310 hva = vm_mmap(NULL, 0, size, PROT_READ | PROT_WRITE,
1224612311 MAP_SHARED | MAP_ANONYMOUS, 0);
12247− if (IS_ERR((void *)hva))
12312+ if (IS_ERR_VALUE(hva))
1224812313 return (void __user *)hva;
1224912314 } else {
1225012315 if (!slot || !slot->npages)
@@ -12459,16 +12524,14 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, · 208 unmodified lines
1245912524
1246012525 static void kvm_mmu_update_cpu_dirty_logging(struct kvm *kvm, bool enable)
1246112526 {
12462− struct kvm_arch *ka = &kvm->arch;
12527+ int nr_slots;
1246312528
1246412529 if (!kvm_x86_ops.cpu_dirty_log_size)
1246512530 return;
1246612531
12467− if ((enable && ++ka->cpu_dirty_logging_count == 1) ||
12468− (!enable && --ka->cpu_dirty_logging_count == 0))
12532+ nr_slots = atomic_read(&kvm->nr_memslots_dirty_logging);
12533+ if ((enable && nr_slots == 1) || !nr_slots)
1246912534 kvm_make_all_cpus_request(kvm, KVM_REQ_UPDATE_CPU_DIRTY_LOGGING);
12470−
12471− WARN_ON_ONCE(ka->cpu_dirty_logging_count < 0);
1247212535 }
1247312536
1247412537 static void kvm_mmu_slot_apply_flags(struct kvm *kvm,
arch/x86/kvm/x86.h+15−3
@@ -331,6 +331,18 @@ extern bool report_ignored_msrs; · 330 unmodified lines
331331
332332 extern bool eager_page_split;
333333
334+static inline void kvm_pr_unimpl_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64 data)
335+{
336+ if (report_ignored_msrs)
337+ vcpu_unimpl(vcpu, "Unhandled WRMSR(0x%x) = 0x%llx\n", msr, data);
338+}
339+
340+static inline void kvm_pr_unimpl_rdmsr(struct kvm_vcpu *vcpu, u32 msr)
341+{
342+ if (report_ignored_msrs)
343+ vcpu_unimpl(vcpu, "Unhandled RDMSR(0x%x)\n", msr);
344+}
345+
334346 static inline u64 nsec_to_cycles(struct kvm_vcpu *vcpu, u64 nsec)
335347 {
336348 return pvclock_scale_delta(nsec, vcpu->arch.virtual_tsc_mult,
@@ -382,13 +394,13 @@ enum kvm_intr_type { · 45 unmodified lines
382394 KVM_HANDLING_NMI,
383395 };
384396
385−static inline void kvm_before_interrupt(struct kvm_vcpu *vcpu,
386− enum kvm_intr_type intr)
397+static __always_inline void kvm_before_interrupt(struct kvm_vcpu *vcpu,
398+ enum kvm_intr_type intr)
387399 {
388400 WRITE_ONCE(vcpu->arch.handling_intr_from_guest, (u8)intr);
389401 }
390402
391−static inline void kvm_after_interrupt(struct kvm_vcpu *vcpu)
403+static __always_inline void kvm_after_interrupt(struct kvm_vcpu *vcpu)
392404 {
393405 WRITE_ONCE(vcpu->arch.handling_intr_from_guest, 0);
394406 }
arch/x86/kvm/xen.c+27−0
@@ -5,6 +5,7 @@ · 4 unmodified lines
55 *
66 * KVM Xen emulation
77 */
8+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
89
910 #include "x86.h"
1011 #include "xen.h"
@@ -22,6 +23,9 @@ · 11 unmodified lines
2223 #include <xen/interface/event_channel.h>
2324 #include <xen/interface/sched.h>
2425
26+#include <asm/xen/cpuid.h>
27+
28+#include "cpuid.h"
2529 #include "trace.h"
2630
2731 static int kvm_xen_set_evtchn(struct kvm_xen_evtchn *xe, struct kvm *kvm);
@@ -2076,6 +2080,29 @@ void kvm_xen_destroy_vcpu(struct kvm_vcpu *vcpu) · 2048 unmodified lines
20762080 del_timer_sync(&vcpu->arch.xen.poll_timer);
20772081 }
20782082
2083+void kvm_xen_update_tsc_info(struct kvm_vcpu *vcpu)
2084+{
2085+ struct kvm_cpuid_entry2 *entry;
2086+ u32 function;
2087+
2088+ if (!vcpu->arch.xen.cpuid.base)
2089+ return;
2090+
2091+ function = vcpu->arch.xen.cpuid.base | XEN_CPUID_LEAF(3);
2092+ if (function > vcpu->arch.xen.cpuid.limit)
2093+ return;
2094+
2095+ entry = kvm_find_cpuid_entry_index(vcpu, function, 1);
2096+ if (entry) {
2097+ entry->ecx = vcpu->arch.hv_clock.tsc_to_system_mul;
2098+ entry->edx = vcpu->arch.hv_clock.tsc_shift;
2099+ }
2100+
2101+ entry = kvm_find_cpuid_entry_index(vcpu, function, 2);
2102+ if (entry)
2103+ entry->eax = vcpu->arch.hw_tsc_khz;
2104+}
2105+
20792106 void kvm_xen_init_vm(struct kvm *kvm)
20802107 {
20812108 mutex_init(&kvm->arch.xen.xen_lock);
arch/x86/kvm/xen.h+7−0
@@ -9,6 +9,8 @@ · 8 unmodified lines
99 #ifndef __ARCH_X86_KVM_XEN_H__
1010 #define __ARCH_X86_KVM_XEN_H__
1111
12+#include <asm/xen/hypervisor.h>
13+
1214 #ifdef CONFIG_KVM_XEN
1315 #include <linux/jump_label_ratelimit.h>
1416
@@ -32,6 +34,7 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe, · 17 unmodified lines
3234 int kvm_xen_setup_evtchn(struct kvm *kvm,
3335 struct kvm_kernel_irq_routing_entry *e,
3436 const struct kvm_irq_routing_entry *ue);
37+void kvm_xen_update_tsc_info(struct kvm_vcpu *vcpu);
3538
3639 static inline bool kvm_xen_msr_enabled(struct kvm *kvm)
3740 {
@@ -135,6 +138,10 @@ static inline bool kvm_xen_timer_enabled(struct kvm_vcpu *vcpu) · 97 unmodified lines
135138 {
136139 return false;
137140 }
141+
142+static inline void kvm_xen_update_tsc_info(struct kvm_vcpu *vcpu)
143+{
144+}
138145 #endif
139146
140147 int kvm_xen_hypercall(struct kvm_vcpu *vcpu);
drivers/irqchip/irq-apple-aic.c+40−13
@@ -210,7 +210,6 @@ · 209 unmodified lines
210210 FIELD_PREP(AIC_EVENT_NUM, x))
211211 #define AIC_HWIRQ_IRQ(x) FIELD_GET(AIC_EVENT_NUM, x)
212212 #define AIC_HWIRQ_DIE(x) FIELD_GET(AIC_EVENT_DIE, x)
213−#define AIC_NR_FIQ 6
214213 #define AIC_NR_SWIPI 32
215214
216215 /*
@@ -222,11 +221,18 @@ · 5 unmodified lines
222221 * running at EL2 (with VHE). When the kernel is running at EL1, the
223222 * mapping differs and aic_irq_domain_translate() performs the remapping.
224223 */
225−
226−#define AIC_TMR_EL0_PHYS AIC_TMR_HV_PHYS
227−#define AIC_TMR_EL0_VIRT AIC_TMR_HV_VIRT
228−#define AIC_TMR_EL02_PHYS AIC_TMR_GUEST_PHYS
229−#define AIC_TMR_EL02_VIRT AIC_TMR_GUEST_VIRT
224+enum fiq_hwirq {
225+ /* Must be ordered as in apple-aic.h */
226+ AIC_TMR_EL0_PHYS = AIC_TMR_HV_PHYS,
227+ AIC_TMR_EL0_VIRT = AIC_TMR_HV_VIRT,
228+ AIC_TMR_EL02_PHYS = AIC_TMR_GUEST_PHYS,
229+ AIC_TMR_EL02_VIRT = AIC_TMR_GUEST_VIRT,
230+ AIC_CPU_PMU_Effi = AIC_CPU_PMU_E,
231+ AIC_CPU_PMU_Perf = AIC_CPU_PMU_P,
232+ /* No need for this to be discovered from DT */
233+ AIC_VGIC_MI,
234+ AIC_NR_FIQ
235+};
230236
231237 static DEFINE_STATIC_KEY_TRUE(use_fast_ipi);
232238
@@ -380,14 +386,20 @@ static void __exception_irq_entry aic_handle_irq(struct pt_regs *regs) · 147 unmodified lines
380386
381387 /*
382388 * vGIC maintenance interrupts end up here too, so we need to check
383− * for them separately. This should never trigger if KVM is working
384− * properly, because it will have already taken care of clearing it
385− * on guest exit before this handler runs.
389+ * for them separately. It should however only trigger when NV is
390+ * in use, and be cleared when coming back from the handler.
386391 */
387− if (is_kernel_in_hyp_mode() && (read_sysreg_s(SYS_ICH_HCR_EL2) & ICH_HCR_EN) &&
388− read_sysreg_s(SYS_ICH_MISR_EL2) != 0) {
389− pr_err_ratelimited("vGIC IRQ fired and not handled by KVM, disabling.\n");
390− sysreg_clear_set_s(SYS_ICH_HCR_EL2, ICH_HCR_EN, 0);
392+ if (is_kernel_in_hyp_mode() &&
393+ (read_sysreg_s(SYS_ICH_HCR_EL2) & ICH_HCR_EN) &&
394+ read_sysreg_s(SYS_ICH_MISR_EL2) != 0) {
395+ generic_handle_domain_irq(aic_irqc->hw_domain,
396+ AIC_FIQ_HWIRQ(AIC_VGIC_MI));
397+
398+ if (unlikely((read_sysreg_s(SYS_ICH_HCR_EL2) & ICH_HCR_EN) &&
399+ read_sysreg_s(SYS_ICH_MISR_EL2))) {
400+ pr_err_ratelimited("vGIC IRQ fired and not handled by KVM, disabling.\n");
401+ sysreg_clear_set_s(SYS_ICH_HCR_EL2, ICH_HCR_EN, 0);
402+ }
391403 }
392404 }
393405
@@ -1033,6 +1045,21 @@ static int __init aic_of_ic_init(struct device_node *node, struct device_node *p · 639 unmodified lines
10331045 "irqchip/apple-aic/ipi:starting",
10341046 aic_init_cpu, NULL);
10351047
1048+ if (is_kernel_in_hyp_mode()) {
1049+ struct irq_fwspec mi = {
1050+ .fwnode = of_node_to_fwnode(node),
1051+ .param_count = 3,
1052+ .param = {
1053+ [0] = AIC_FIQ, /* This is a lie */
1054+ [1] = AIC_VGIC_MI,
1055+ [2] = IRQ_TYPE_LEVEL_HIGH,
1056+ },
1057+ };
1058+
1059+ vgic_info.maint_irq = irq_create_fwspec_mapping(&mi);
1060+ WARN_ON(!vgic_info.maint_irq);
1061+ }
1062+
10361063 vgic_set_kvm_info(&vgic_info);
10371064
10381065 pr_info("Initialized with %d/%d IRQs * %d/%d die(s), %d FIQs, %d vIPIs",
drivers/perf/riscv_pmu_sbi.c+57−7
@@ -44,7 +44,7 @@ static const struct attribute_group *riscv_pmu_attr_groups[] = { · 43 unmodified lines
4444 };
4545
4646 /*
47− * RISC-V doesn't have hetergenous harts yet. This need to be part of
47+ * RISC-V doesn't have heterogeneous harts yet. This need to be part of
4848 * per_cpu in case of harts with different pmu counters
4949 */
5050 static union sbi_pmu_ctr_info *pmu_ctr_list;
@@ -52,6 +52,9 @@ static bool riscv_pmu_use_irq; · 1 unmodified line
5252 static unsigned int riscv_pmu_irq_num;
5353 static unsigned int riscv_pmu_irq;
5454
55+/* Cache the available counters in a bitmask */
56+static unsigned long cmask;
57+
5558 struct sbi_pmu_event_data {
5659 union {
5760 union {
@@ -267,6 +270,58 @@ static bool pmu_sbi_ctr_is_fw(int cidx) · 209 unmodified lines
267270 return (info->type == SBI_PMU_CTR_TYPE_FW) ? true : false;
268271 }
269272
273+/*
274+ * Returns the counter width of a programmable counter and number of hardware
275+ * counters. As we don't support heterogeneous CPUs yet, it is okay to just
276+ * return the counter width of the first programmable counter.
277+ */
278+int riscv_pmu_get_hpm_info(u32 *hw_ctr_width, u32 *num_hw_ctr)
279+{
280+ int i;
281+ union sbi_pmu_ctr_info *info;
282+ u32 hpm_width = 0, hpm_count = 0;
283+
284+ if (!cmask)
285+ return -EINVAL;
286+
287+ for_each_set_bit(i, &cmask, RISCV_MAX_COUNTERS) {
288+ info = &pmu_ctr_list[i];
289+ if (!info)
290+ continue;
291+ if (!hpm_width && info->csr != CSR_CYCLE && info->csr != CSR_INSTRET)
292+ hpm_width = info->width;
293+ if (info->type == SBI_PMU_CTR_TYPE_HW)
294+ hpm_count++;
295+ }
296+
297+ *hw_ctr_width = hpm_width;
298+ *num_hw_ctr = hpm_count;
299+
300+ return 0;
301+}
302+EXPORT_SYMBOL_GPL(riscv_pmu_get_hpm_info);
303+
304+static unsigned long pmu_sbi_get_filter_flags(struct perf_event *event)
305+{
306+ unsigned long cflags = 0;
307+ bool guest_events = false;
308+
309+ if (event->attr.config1 & RISCV_PMU_CONFIG1_GUEST_EVENTS)
310+ guest_events = true;
311+ if (event->attr.exclude_kernel)
312+ cflags |= guest_events ? SBI_PMU_CFG_FLAG_SET_VSINH : SBI_PMU_CFG_FLAG_SET_SINH;
313+ if (event->attr.exclude_user)
314+ cflags |= guest_events ? SBI_PMU_CFG_FLAG_SET_VUINH : SBI_PMU_CFG_FLAG_SET_UINH;
315+ if (guest_events && event->attr.exclude_hv)
316+ cflags |= SBI_PMU_CFG_FLAG_SET_SINH;
317+ if (event->attr.exclude_host)
318+ cflags |= SBI_PMU_CFG_FLAG_SET_UINH | SBI_PMU_CFG_FLAG_SET_SINH;
319+ if (event->attr.exclude_guest)
320+ cflags |= SBI_PMU_CFG_FLAG_SET_VSINH | SBI_PMU_CFG_FLAG_SET_VUINH;
321+
322+ return cflags;
323+}
324+
270325 static int pmu_sbi_ctr_get_idx(struct perf_event *event)
271326 {
272327 struct hw_perf_event *hwc = &event->hw;
@@ -277,11 +332,7 @@ static int pmu_sbi_ctr_get_idx(struct perf_event *event) · 4 unmodified lines
277332 uint64_t cbase = 0;
278333 unsigned long cflags = 0;
279334
280− if (event->attr.exclude_kernel)
281− cflags |= SBI_PMU_CFG_FLAG_SET_SINH;
282− if (event->attr.exclude_user)
283− cflags |= SBI_PMU_CFG_FLAG_SET_UINH;
284−
335+ cflags = pmu_sbi_get_filter_flags(event);
285336 /* retrieve the available counter index */
286337 #if defined(CONFIG_32BIT)
287338 ret = sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_CFG_MATCH, cbase,
@@ -806,7 +857,6 @@ static void riscv_pmu_destroy(struct riscv_pmu *pmu) · 518 unmodified lines
806857 static int pmu_sbi_device_probe(struct platform_device *pdev)
807858 {
808859 struct riscv_pmu *pmu = NULL;
809− unsigned long cmask = 0;
810860 int ret = -ENODEV;
811861 int num_counters;
812862
drivers/s390/virtio/virtio_ccw.c+24−22
@@ -363,7 +363,7 @@ static void virtio_ccw_drop_indicator(struct virtio_ccw_device *vcdev, · 362 unmodified lines
363363 thinint_area->isc = VIRTIO_AIRQ_ISC;
364364 ccw->cmd_code = CCW_CMD_SET_IND_ADAPTER;
365365 ccw->count = sizeof(*thinint_area);
366− ccw->cda = (__u32)(unsigned long) thinint_area;
366+ ccw->cda = (__u32)virt_to_phys(thinint_area);
367367 } else {
368368 /* payload is the address of the indicators */
369369 indicatorp = ccw_device_dma_zalloc(vcdev->cdev,
@@ -373,7 +373,7 @@ static void virtio_ccw_drop_indicator(struct virtio_ccw_device *vcdev, · 3 unmodified lines
373373 *indicatorp = 0;
374374 ccw->cmd_code = CCW_CMD_SET_IND;
375375 ccw->count = sizeof(indicators(vcdev));
376− ccw->cda = (__u32)(unsigned long) indicatorp;
376+ ccw->cda = (__u32)virt_to_phys(indicatorp);
377377 }
378378 /* Deregister indicators from host. */
379379 *indicators(vcdev) = 0;
@@ -417,7 +417,7 @@ static int virtio_ccw_read_vq_conf(struct virtio_ccw_device *vcdev, · 37 unmodified lines
417417 ccw->cmd_code = CCW_CMD_READ_VQ_CONF;
418418 ccw->flags = 0;
419419 ccw->count = sizeof(struct vq_config_block);
420− ccw->cda = (__u32)(unsigned long)(&vcdev->dma_area->config_block);
420+ ccw->cda = (__u32)virt_to_phys(&vcdev->dma_area->config_block);
421421 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_VQ_CONF);
422422 if (ret)
423423 return ret;
@@ -454,7 +454,7 @@ static void virtio_ccw_del_vq(struct virtqueue *vq, struct ccw1 *ccw) · 30 unmodified lines
454454 }
455455 ccw->cmd_code = CCW_CMD_SET_VQ;
456456 ccw->flags = 0;
457− ccw->cda = (__u32)(unsigned long)(info->info_block);
457+ ccw->cda = (__u32)virt_to_phys(info->info_block);
458458 ret = ccw_io_helper(vcdev, ccw,
459459 VIRTIO_CCW_DOING_SET_VQ | index);
460460 /*
@@ -556,7 +556,7 @@ static struct virtqueue *virtio_ccw_setup_vq(struct virtio_device *vdev, · 95 unmodified lines
556556 }
557557 ccw->cmd_code = CCW_CMD_SET_VQ;
558558 ccw->flags = 0;
559− ccw->cda = (__u32)(unsigned long)(info->info_block);
559+ ccw->cda = (__u32)virt_to_phys(info->info_block);
560560 err = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_SET_VQ | i);
561561 if (err) {
562562 dev_warn(&vcdev->cdev->dev, "SET_VQ failed\n");
@@ -590,6 +590,7 @@ static int virtio_ccw_register_adapter_ind(struct virtio_ccw_device *vcdev, · 27 unmodified lines
590590 {
591591 int ret;
592592 struct virtio_thinint_area *thinint_area = NULL;
593+ unsigned long indicator_addr;
593594 struct airq_info *info;
594595
595596 thinint_area = ccw_device_dma_zalloc(vcdev->cdev,
@@ -599,21 +600,22 @@ static int virtio_ccw_register_adapter_ind(struct virtio_ccw_device *vcdev, · 3 unmodified lines
599600 goto out;
600601 }
601602 /* Try to get an indicator. */
602− thinint_area->indicator = get_airq_indicator(vqs, nvqs,
603− &thinint_area->bit_nr,
604− &vcdev->airq_info);
605− if (!thinint_area->indicator) {
603+ indicator_addr = get_airq_indicator(vqs, nvqs,
604+ &thinint_area->bit_nr,
605+ &vcdev->airq_info);
606+ if (!indicator_addr) {
606607 ret = -ENOSPC;
607608 goto out;
608609 }
610+ thinint_area->indicator = virt_to_phys((void *)indicator_addr);
609611 info = vcdev->airq_info;
610612 thinint_area->summary_indicator =
611− (unsigned long) get_summary_indicator(info);
613+ virt_to_phys(get_summary_indicator(info));
612614 thinint_area->isc = VIRTIO_AIRQ_ISC;
613615 ccw->cmd_code = CCW_CMD_SET_IND_ADAPTER;
614616 ccw->flags = CCW_FLAG_SLI;
615617 ccw->count = sizeof(*thinint_area);
616− ccw->cda = (__u32)(unsigned long)thinint_area;
618+ ccw->cda = (__u32)virt_to_phys(thinint_area);
617619 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_SET_IND_ADAPTER);
618620 if (ret) {
619621 if (ret == -EOPNOTSUPP) {
@@ -686,7 +688,7 @@ static int virtio_ccw_find_vqs(struct virtio_device *vdev, unsigned nvqs, · 66 unmodified lines
686688 ccw->cmd_code = CCW_CMD_SET_IND;
687689 ccw->flags = 0;
688690 ccw->count = sizeof(indicators(vcdev));
689− ccw->cda = (__u32)(unsigned long) indicatorp;
691+ ccw->cda = (__u32)virt_to_phys(indicatorp);
690692 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_SET_IND);
691693 if (ret)
692694 goto out;
@@ -697,7 +699,7 @@ static int virtio_ccw_find_vqs(struct virtio_device *vdev, unsigned nvqs, · 4 unmodified lines
697699 ccw->cmd_code = CCW_CMD_SET_CONF_IND;
698700 ccw->flags = 0;
699701 ccw->count = sizeof(indicators2(vcdev));
700− ccw->cda = (__u32)(unsigned long) indicatorp;
702+ ccw->cda = (__u32)virt_to_phys(indicatorp);
701703 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_SET_CONF_IND);
702704 if (ret)
703705 goto out;
@@ -759,7 +761,7 @@ static u64 virtio_ccw_get_features(struct virtio_device *vdev) · 55 unmodified lines
759761 ccw->cmd_code = CCW_CMD_READ_FEAT;
760762 ccw->flags = 0;
761763 ccw->count = sizeof(*features);
762− ccw->cda = (__u32)(unsigned long)features;
764+ ccw->cda = (__u32)virt_to_phys(features);
763765 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_FEAT);
764766 if (ret) {
765767 rc = 0;
@@ -776,7 +778,7 @@ static u64 virtio_ccw_get_features(struct virtio_device *vdev) · 10 unmodified lines
776778 ccw->cmd_code = CCW_CMD_READ_FEAT;
777779 ccw->flags = 0;
778780 ccw->count = sizeof(*features);
779− ccw->cda = (__u32)(unsigned long)features;
781+ ccw->cda = (__u32)virt_to_phys(features);
780782 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_FEAT);
781783 if (ret == 0)
782784 rc |= (u64)le32_to_cpu(features->features) << 32;
@@ -829,7 +831,7 @@ static int virtio_ccw_finalize_features(struct virtio_device *vdev) · 46 unmodified lines
829831 ccw->cmd_code = CCW_CMD_WRITE_FEAT;
830832 ccw->flags = 0;
831833 ccw->count = sizeof(*features);
832− ccw->cda = (__u32)(unsigned long)features;
834+ ccw->cda = (__u32)virt_to_phys(features);
833835 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_WRITE_FEAT);
834836 if (ret)
835837 goto out_free;
@@ -843,7 +845,7 @@ static int virtio_ccw_finalize_features(struct virtio_device *vdev) · 7 unmodified lines
843845 ccw->cmd_code = CCW_CMD_WRITE_FEAT;
844846 ccw->flags = 0;
845847 ccw->count = sizeof(*features);
846− ccw->cda = (__u32)(unsigned long)features;
848+ ccw->cda = (__u32)virt_to_phys(features);
847849 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_WRITE_FEAT);
848850
849851 out_free:
@@ -875,7 +877,7 @@ static void virtio_ccw_get_config(struct virtio_device *vdev, · 25 unmodified lines
875877 ccw->cmd_code = CCW_CMD_READ_CONF;
876878 ccw->flags = 0;
877879 ccw->count = offset + len;
878− ccw->cda = (__u32)(unsigned long)config_area;
880+ ccw->cda = (__u32)virt_to_phys(config_area);
879881 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_CONFIG);
880882 if (ret)
881883 goto out_free;
@@ -922,7 +924,7 @@ static void virtio_ccw_set_config(struct virtio_device *vdev, · 40 unmodified lines
922924 ccw->cmd_code = CCW_CMD_WRITE_CONF;
923925 ccw->flags = 0;
924926 ccw->count = offset + len;
925− ccw->cda = (__u32)(unsigned long)config_area;
927+ ccw->cda = (__u32)virt_to_phys(config_area);
926928 ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_WRITE_CONFIG);
927929
928930 out_free:
@@ -946,7 +948,7 @@ static u8 virtio_ccw_get_status(struct virtio_device *vdev) · 17 unmodified lines
946948 ccw->cmd_code = CCW_CMD_READ_STATUS;
947949 ccw->flags = 0;
948950 ccw->count = sizeof(vcdev->dma_area->status);
949− ccw->cda = (__u32)(unsigned long)&vcdev->dma_area->status;
951+ ccw->cda = (__u32)virt_to_phys(&vcdev->dma_area->status);
950952 ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_STATUS);
951953 /*
952954 * If the channel program failed (should only happen if the device
@@ -975,7 +977,7 @@ static void virtio_ccw_set_status(struct virtio_device *vdev, u8 status) · 22 unmodified lines
975977 ccw->cmd_code = CCW_CMD_WRITE_STATUS;
976978 ccw->flags = 0;
977979 ccw->count = sizeof(status);
978− ccw->cda = (__u32)(unsigned long)&vcdev->dma_area->status;
980+ ccw->cda = (__u32)virt_to_phys(&vcdev->dma_area->status);
979981 /* We use ssch for setting the status which is a serializing
980982 * instruction that guarantees the memory writes have
981983 * completed before ssch.
@@ -1274,7 +1276,7 @@ static int virtio_ccw_set_transport_rev(struct virtio_ccw_device *vcdev) · 292 unmodified lines
12741276 ccw->cmd_code = CCW_CMD_SET_VIRTIO_REV;
12751277 ccw->flags = 0;
12761278 ccw->count = sizeof(*rev);
1277− ccw->cda = (__u32)(unsigned long)rev;
1279+ ccw->cda = (__u32)virt_to_phys(rev);
12781280
12791281 vcdev->revision = VIRTIO_CCW_REV_MAX;
12801282 do {
include/kvm/arm_arch_timer.h+5−1
@@ -60,7 +60,7 @@ struct arch_timer_cpu { · 59 unmodified lines
6060 bool enabled;
6161 };
6262
63−int kvm_timer_hyp_init(bool);
63+int __init kvm_timer_hyp_init(bool has_gic);
6464 int kvm_timer_enable(struct kvm_vcpu *vcpu);
6565 int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu);
6666 void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu);
@@ -104,4 +104,8 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · 37 unmodified lines
104104 u32 timer_get_ctl(struct arch_timer_context *ctxt);
105105 u64 timer_get_cval(struct arch_timer_context *ctxt);
106106
107+/* CPU HP callbacks */
108+void kvm_timer_cpu_up(void);
109+void kvm_timer_cpu_down(void);
110+
107111 #endif
include/kvm/arm_vgic.h+4−0
@@ -432,4 +432,8 @@ int vgic_v4_load(struct kvm_vcpu *vcpu); · 431 unmodified lines
432432 void vgic_v4_commit(struct kvm_vcpu *vcpu);
433433 int vgic_v4_put(struct kvm_vcpu *vcpu, bool need_db);
434434
435+/* CPU HP callbacks */
436+void kvm_vgic_cpu_up(void);
437+void kvm_vgic_cpu_down(void);
438+
435439 #endif /* __KVM_ARM_VGIC_H */
include/linux/cpuhotplug.h+1−4
@@ -187,10 +187,6 @@ enum cpuhp_state { · 186 unmodified lines
187187 CPUHP_AP_CSKY_TIMER_STARTING,
188188 CPUHP_AP_TI_GP_TIMER_STARTING,
189189 CPUHP_AP_HYPERV_TIMER_STARTING,
190− CPUHP_AP_KVM_STARTING,
191− CPUHP_AP_KVM_ARM_VGIC_INIT_STARTING,
192− CPUHP_AP_KVM_ARM_VGIC_STARTING,
193− CPUHP_AP_KVM_ARM_TIMER_STARTING,
194190 /* Must be the last timer callback */
195191 CPUHP_AP_DUMMY_TIMER_STARTING,
196192 CPUHP_AP_ARM_XEN_STARTING,
@@ -205,6 +201,7 @@ enum cpuhp_state { · 8 unmodified lines
205201
206202 /* Online section invoked on the hotplugged CPU from the hotplug thread */
207203 CPUHP_AP_ONLINE_IDLE,
204+ CPUHP_AP_KVM_ONLINE,
208205 CPUHP_AP_SCHED_WAIT_EMPTY,
209206 CPUHP_AP_SMPBOOT_THREADS,
210207 CPUHP_AP_X86_VDSO_VMA_ONLINE,
include/linux/kvm_host.h+8−11
@@ -324,7 +324,7 @@ struct kvm_vcpu { · 323 unmodified lines
324324 #endif
325325 int cpu;
326326 int vcpu_id; /* id given by userspace at creation */
327− int vcpu_idx; /* index in kvm->vcpus array */
327+ int vcpu_idx; /* index into kvm->vcpu_array */
328328 int ____srcu_idx; /* Don't use this directly. You've been warned. */
329329 #ifdef CONFIG_PROVE_RCU
330330 int srcu_depth;
@@ -956,8 +956,7 @@ static inline void kvm_irqfd_exit(void) · 625 unmodified lines
956956 {
957957 }
958958 #endif
959−int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
960− struct module *module);
959+int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module);
961960 void kvm_exit(void);
962961
963962 void kvm_get_kvm(struct kvm *kvm);
@@ -1354,7 +1353,7 @@ void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu); · 390 unmodified lines
13541353 bool kvm_vcpu_wake_up(struct kvm_vcpu *vcpu);
13551354 void kvm_vcpu_kick(struct kvm_vcpu *vcpu);
13561355 int kvm_vcpu_yield_to(struct kvm_vcpu *target);
1357−void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu, bool usermode_vcpu_not_eligible);
1356+void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu, bool yield_to_kernel_mode);
13581357
13591358 void kvm_flush_remote_tlbs(struct kvm *kvm);
13601359
@@ -1423,9 +1422,6 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu, · 62 unmodified lines
14231422 struct kvm_guest_debug *dbg);
14241423 int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu);
14251424
1426−int kvm_arch_init(void *opaque);
1427−void kvm_arch_exit(void);
1428−
14291425 void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu);
14301426
14311427 void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu);
@@ -1445,11 +1441,10 @@ void kvm_arch_create_vcpu_debugfs(struct kvm_vcpu *vcpu, struct dentry *debugfs_ · 13 unmodified lines
14451441 static inline void kvm_create_vcpu_debugfs(struct kvm_vcpu *vcpu) {}
14461442 #endif
14471443
1444+#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING
14481445 int kvm_arch_hardware_enable(void);
14491446 void kvm_arch_hardware_disable(void);
1450−int kvm_arch_hardware_setup(void *opaque);
1451−void kvm_arch_hardware_unsetup(void);
1452−int kvm_arch_check_processor_compat(void *opaque);
1447+#endif
14531448 int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu);
14541449 bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu);
14551450 int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu);
@@ -1466,7 +1461,7 @@ int kvm_arch_create_vm_debugfs(struct kvm *kvm); · 10 unmodified lines
14661461 */
14671462 static inline struct kvm *kvm_arch_alloc_vm(void)
14681463 {
1469− return kzalloc(sizeof(struct kvm), GFP_KERNEL);
1464+ return kzalloc(sizeof(struct kvm), GFP_KERNEL_ACCOUNT);
14701465 }
14711466 #endif
14721467
@@ -2081,7 +2076,9 @@ static inline bool kvm_check_request(int req, struct kvm_vcpu *vcpu) · 608 unmodified lines
20812076 }
20822077 }
20832078
2079+#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING
20842080 extern bool kvm_rebooting;
2081+#endif
20852082
20862083 extern unsigned int halt_poll_ns;
20872084 extern unsigned int halt_poll_ns_grow;
include/linux/kvm_types.h+1−1
@@ -40,7 +40,7 @@ typedef unsigned long gva_t; · 39 unmodified lines
4040 typedef u64 gpa_t;
4141 typedef u64 gfn_t;
4242
43−#define GPA_INVALID (~(gpa_t)0)
43+#define INVALID_GPA (~(gpa_t)0)
4444
4545 typedef unsigned long hva_t;
4646 typedef u64 hpa_t;
include/linux/perf/riscv_pmu.h+5−0
@@ -26,6 +26,8 @@ · 25 unmodified lines
2626
2727 #define RISCV_PMU_STOP_FLAG_RESET 1
2828
29+#define RISCV_PMU_CONFIG1_GUEST_EVENTS 0x1
30+
2931 struct cpu_hw_events {
3032 /* currently enabled events */
3133 int n_events;
@@ -73,6 +75,9 @@ void riscv_pmu_legacy_skip_init(void); · 41 unmodified lines
7375 static inline void riscv_pmu_legacy_skip_init(void) {};
7476 #endif
7577 struct riscv_pmu *riscv_pmu_alloc(void);
78+#ifdef CONFIG_RISCV_PMU_SBI
79+int riscv_pmu_get_hpm_info(u32 *hw_ctr_width, u32 *num_hw_ctr);
80+#endif
7681
7782 #endif /* CONFIG_RISCV_PMU */
7883
include/uapi/linux/kvm.h+9−0
@@ -583,6 +583,8 @@ struct kvm_s390_mem_op { · 582 unmodified lines
583583 struct {
584584 __u8 ar; /* the access register number */
585585 __u8 key; /* access key, ignored if flag unset */
586+ __u8 pad1[6]; /* ignored */
587+ __u64 old_addr; /* ignored if cmpxchg flag unset */
586588 };
587589 __u32 sida_offset; /* offset into the sida */
588590 __u8 reserved[32]; /* ignored */
@@ -595,11 +597,17 @@ struct kvm_s390_mem_op { · 6 unmodified lines
595597 #define KVM_S390_MEMOP_SIDA_WRITE 3
596598 #define KVM_S390_MEMOP_ABSOLUTE_READ 4
597599 #define KVM_S390_MEMOP_ABSOLUTE_WRITE 5
600+#define KVM_S390_MEMOP_ABSOLUTE_CMPXCHG 6
601+
598602 /* flags for kvm_s390_mem_op->flags */
599603 #define KVM_S390_MEMOP_F_CHECK_ONLY (1ULL << 0)
600604 #define KVM_S390_MEMOP_F_INJECT_EXCEPTION (1ULL << 1)
601605 #define KVM_S390_MEMOP_F_SKEY_PROTECTION (1ULL << 2)
602606
607+/* flags specifying extension support via KVM_CAP_S390_MEM_OP_EXTENSION */
608+#define KVM_S390_MEMOP_EXTENSION_CAP_BASE (1 << 0)
609+#define KVM_S390_MEMOP_EXTENSION_CAP_CMPXCHG (1 << 1)
610+
603611 /* for KVM_INTERRUPT */
604612 struct kvm_interrupt {
605613 /* in */
@@ -1175,6 +1183,7 @@ struct kvm_ppc_resize_hpt { · 569 unmodified lines
11751183 #define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 223
11761184 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224
11771185 #define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 225
1186+#define KVM_CAP_PMU_EVENT_MASKED_EVENTS 226
11781187
11791188 #ifdef KVM_CAP_IRQ_ROUTING
11801189
tools/testing/selftests/kvm/Makefile+4−0
@@ -67,6 +67,7 @@ TEST_GEN_PROGS_x86_64 += x86_64/fix_hypercall_test · 66 unmodified lines
6767 TEST_GEN_PROGS_x86_64 += x86_64/hyperv_clock
6868 TEST_GEN_PROGS_x86_64 += x86_64/hyperv_cpuid
6969 TEST_GEN_PROGS_x86_64 += x86_64/hyperv_evmcs
70+TEST_GEN_PROGS_x86_64 += x86_64/hyperv_extended_hypercalls
7071 TEST_GEN_PROGS_x86_64 += x86_64/hyperv_features
7172 TEST_GEN_PROGS_x86_64 += x86_64/hyperv_ipi
7273 TEST_GEN_PROGS_x86_64 += x86_64/hyperv_svm_test
@@ -200,6 +201,9 @@ CFLAGS += -Wall -Wstrict-prototypes -Wuninitialized -O2 -g -std=gnu99 \ · 127 unmodified lines
200201 -I$(LINUX_TOOL_ARCH_INCLUDE) -I$(LINUX_HDR_PATH) -Iinclude \
201202 -I$(<D) -Iinclude/$(ARCH_DIR) -I ../rseq -I.. $(EXTRA_CFLAGS) \
202203 $(KHDR_INCLUDES)
204+ifeq ($(ARCH),s390)
205+ CFLAGS += -march=z10
206+endif
203207
204208 no-pie-option := $(call try-run, echo 'int main(void) { return 0; }' | \
205209 $(CC) -Werror $(CFLAGS) -no-pie -x c - -o "$$TMP", -no-pie)
tools/testing/selftests/kvm/aarch64/page_fault_test.c+0−2
@@ -1112,8 +1112,6 @@ int main(int argc, char *argv[]) · 1111 unmodified lines
11121112 enum vm_mem_backing_src_type src_type;
11131113 int opt;
11141114
1115− setbuf(stdout, NULL);
1116−
11171115 src_type = DEFAULT_VM_MEM_SRC;
11181116
11191117 while ((opt = getopt(argc, argv, "hm:s:")) != -1) {
tools/testing/selftests/kvm/config+1−0
@@ -1,3 +1,4 @@
11 CONFIG_KVM=y
22 CONFIG_KVM_INTEL=y
33 CONFIG_KVM_AMD=y
4+CONFIG_USERFAULTFD=y
tools/testing/selftests/kvm/include/x86_64/hyperv.h+102−47
@@ -85,61 +85,110 @@ · 84 unmodified lines
8585 #define HV_X64_MSR_SYNDBG_OPTIONS 0x400000FF
8686
8787 /* HYPERV_CPUID_FEATURES.EAX */
88−#define HV_MSR_VP_RUNTIME_AVAILABLE BIT(0)
89−#define HV_MSR_TIME_REF_COUNT_AVAILABLE BIT(1)
90−#define HV_MSR_SYNIC_AVAILABLE BIT(2)
91−#define HV_MSR_SYNTIMER_AVAILABLE BIT(3)
92−#define HV_MSR_APIC_ACCESS_AVAILABLE BIT(4)
93−#define HV_MSR_HYPERCALL_AVAILABLE BIT(5)
94−#define HV_MSR_VP_INDEX_AVAILABLE BIT(6)
95−#define HV_MSR_RESET_AVAILABLE BIT(7)
96−#define HV_MSR_STAT_PAGES_AVAILABLE BIT(8)
97−#define HV_MSR_REFERENCE_TSC_AVAILABLE BIT(9)
98−#define HV_MSR_GUEST_IDLE_AVAILABLE BIT(10)
99−#define HV_ACCESS_FREQUENCY_MSRS BIT(11)
100−#define HV_ACCESS_REENLIGHTENMENT BIT(13)
101−#define HV_ACCESS_TSC_INVARIANT BIT(15)
88+#define HV_MSR_VP_RUNTIME_AVAILABLE \
89+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 0)
90+#define HV_MSR_TIME_REF_COUNT_AVAILABLE \
91+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 1)
92+#define HV_MSR_SYNIC_AVAILABLE \
93+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 2)
94+#define HV_MSR_SYNTIMER_AVAILABLE \
95+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 3)
96+#define HV_MSR_APIC_ACCESS_AVAILABLE \
97+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 4)
98+#define HV_MSR_HYPERCALL_AVAILABLE \
99+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 5)
100+#define HV_MSR_VP_INDEX_AVAILABLE \
101+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 6)
102+#define HV_MSR_RESET_AVAILABLE \
103+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 7)
104+#define HV_MSR_STAT_PAGES_AVAILABLE \
105+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 8)
106+#define HV_MSR_REFERENCE_TSC_AVAILABLE \
107+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 9)
108+#define HV_MSR_GUEST_IDLE_AVAILABLE \
109+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 10)
110+#define HV_ACCESS_FREQUENCY_MSRS \
111+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 11)
112+#define HV_ACCESS_REENLIGHTENMENT \
113+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 13)
114+#define HV_ACCESS_TSC_INVARIANT \
115+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 15)
102116
103117 /* HYPERV_CPUID_FEATURES.EBX */
104−#define HV_CREATE_PARTITIONS BIT(0)
105−#define HV_ACCESS_PARTITION_ID BIT(1)
106−#define HV_ACCESS_MEMORY_POOL BIT(2)
107−#define HV_ADJUST_MESSAGE_BUFFERS BIT(3)
108−#define HV_POST_MESSAGES BIT(4)
109−#define HV_SIGNAL_EVENTS BIT(5)
110−#define HV_CREATE_PORT BIT(6)
111−#define HV_CONNECT_PORT BIT(7)
112−#define HV_ACCESS_STATS BIT(8)
113−#define HV_DEBUGGING BIT(11)
114−#define HV_CPU_MANAGEMENT BIT(12)
115−#define HV_ISOLATION BIT(22)
118+#define HV_CREATE_PARTITIONS \
119+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 0)
120+#define HV_ACCESS_PARTITION_ID \
121+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 1)
122+#define HV_ACCESS_MEMORY_POOL \
123+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 2)
124+#define HV_ADJUST_MESSAGE_BUFFERS \
125+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 3)
126+#define HV_POST_MESSAGES \
127+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 4)
128+#define HV_SIGNAL_EVENTS \
129+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 5)
130+#define HV_CREATE_PORT \
131+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 6)
132+#define HV_CONNECT_PORT \
133+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 7)
134+#define HV_ACCESS_STATS \
135+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 8)
136+#define HV_DEBUGGING \
137+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 11)
138+#define HV_CPU_MANAGEMENT \
139+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 12)
140+#define HV_ENABLE_EXTENDED_HYPERCALLS \
141+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 20)
142+#define HV_ISOLATION \
143+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 22)
116144
117145 /* HYPERV_CPUID_FEATURES.EDX */
118−#define HV_X64_MWAIT_AVAILABLE BIT(0)
119−#define HV_X64_GUEST_DEBUGGING_AVAILABLE BIT(1)
120−#define HV_X64_PERF_MONITOR_AVAILABLE BIT(2)
121−#define HV_X64_CPU_DYNAMIC_PARTITIONING_AVAILABLE BIT(3)
122−#define HV_X64_HYPERCALL_XMM_INPUT_AVAILABLE BIT(4)
123−#define HV_X64_GUEST_IDLE_STATE_AVAILABLE BIT(5)
124−#define HV_FEATURE_FREQUENCY_MSRS_AVAILABLE BIT(8)
125−#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE BIT(10)
126−#define HV_FEATURE_DEBUG_MSRS_AVAILABLE BIT(11)
127−#define HV_STIMER_DIRECT_MODE_AVAILABLE BIT(19)
146+#define HV_X64_MWAIT_AVAILABLE \
147+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 0)
148+#define HV_X64_GUEST_DEBUGGING_AVAILABLE \
149+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 1)
150+#define HV_X64_PERF_MONITOR_AVAILABLE \
151+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 2)
152+#define HV_X64_CPU_DYNAMIC_PARTITIONING_AVAILABLE \
153+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 3)
154+#define HV_X64_HYPERCALL_XMM_INPUT_AVAILABLE \
155+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 4)
156+#define HV_X64_GUEST_IDLE_STATE_AVAILABLE \
157+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 5)
158+#define HV_FEATURE_FREQUENCY_MSRS_AVAILABLE \
159+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 8)
160+#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE \
161+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 10)
162+#define HV_FEATURE_DEBUG_MSRS_AVAILABLE \
163+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 11)
164+#define HV_STIMER_DIRECT_MODE_AVAILABLE \
165+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 19)
128166
129167 /* HYPERV_CPUID_ENLIGHTMENT_INFO.EAX */
130−#define HV_X64_AS_SWITCH_RECOMMENDED BIT(0)
131−#define HV_X64_LOCAL_TLB_FLUSH_RECOMMENDED BIT(1)
132−#define HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED BIT(2)
133−#define HV_X64_APIC_ACCESS_RECOMMENDED BIT(3)
134−#define HV_X64_SYSTEM_RESET_RECOMMENDED BIT(4)
135−#define HV_X64_RELAXED_TIMING_RECOMMENDED BIT(5)
136−#define HV_DEPRECATING_AEOI_RECOMMENDED BIT(9)
137−#define HV_X64_CLUSTER_IPI_RECOMMENDED BIT(10)
138−#define HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED BIT(11)
139−#define HV_X64_ENLIGHTENED_VMCS_RECOMMENDED BIT(14)
168+#define HV_X64_AS_SWITCH_RECOMMENDED \
169+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 0)
170+#define HV_X64_LOCAL_TLB_FLUSH_RECOMMENDED \
171+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 1)
172+#define HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED \
173+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 2)
174+#define HV_X64_APIC_ACCESS_RECOMMENDED \
175+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 3)
176+#define HV_X64_SYSTEM_RESET_RECOMMENDED \
177+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 4)
178+#define HV_X64_RELAXED_TIMING_RECOMMENDED \
179+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 5)
180+#define HV_DEPRECATING_AEOI_RECOMMENDED \
181+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 9)
182+#define HV_X64_CLUSTER_IPI_RECOMMENDED \
183+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 10)
184+#define HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED \
185+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 11)
186+#define HV_X64_ENLIGHTENED_VMCS_RECOMMENDED \
187+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 14)
140188
141189 /* HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES.EAX */
142−#define HV_X64_SYNDBG_CAP_ALLOW_KERNEL_DEBUGGING BIT(1)
190+#define HV_X64_SYNDBG_CAP_ALLOW_KERNEL_DEBUGGING \
191+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES, 0, EAX, 1)
143192
144193 /* Hypercalls */
145194 #define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE 0x0002
@@ -166,6 +215,9 @@ · 20 unmodified lines
166215 #define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE 0x00af
167216 #define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_LIST 0x00b0
168217
218+/* Extended hypercalls */
219+#define HV_EXT_CALL_QUERY_CAPABILITIES 0x8001
220+
169221 #define HV_FLUSH_ALL_PROCESSORS BIT(0)
170222 #define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES BIT(1)
171223 #define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY BIT(2)
@@ -288,4 +340,7 @@ struct hyperv_test_pages { · 116 unmodified lines
288340 struct hyperv_test_pages *vcpu_alloc_hyperv_test_pages(struct kvm_vm *vm,
289341 vm_vaddr_t *p_hv_pages_gva);
290342
343+/* HV_X64_MSR_TSC_INVARIANT_CONTROL bits */
344+#define HV_INVARIANT_TSC_EXPOSED BIT_ULL(0)
345+
291346 #endif /* !SELFTEST_KVM_HYPERV_H */
tools/testing/selftests/kvm/include/x86_64/processor.h+27−4
@@ -19,6 +19,9 @@ · 18 unmodified lines
1919
2020 #include "../kvm_util.h"
2121
22+extern bool host_cpu_is_intel;
23+extern bool host_cpu_is_amd;
24+
2225 #define NMI_VECTOR 0x02
2326
2427 #define X86_EFLAGS_FIXED (1u << 1)
@@ -137,6 +140,7 @@ struct kvm_x86_cpu_feature { · 112 unmodified lines
137140 #define X86_FEATURE_GBPAGES KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 26)
138141 #define X86_FEATURE_RDTSCP KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 27)
139142 #define X86_FEATURE_LM KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 29)
143+#define X86_FEATURE_INVTSC KVM_X86_CPU_FEATURE(0x80000007, 0, EDX, 8)
140144 #define X86_FEATURE_RDPRU KVM_X86_CPU_FEATURE(0x80000008, 0, EBX, 4)
141145 #define X86_FEATURE_AMD_IBPB KVM_X86_CPU_FEATURE(0x80000008, 0, EBX, 12)
142146 #define X86_FEATURE_NPT KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 0)
@@ -554,6 +558,28 @@ static inline uint32_t this_cpu_model(void) · 411 unmodified lines
554558 return x86_model(this_cpu_fms());
555559 }
556560
561+static inline bool this_cpu_vendor_string_is(const char *vendor)
562+{
563+ const uint32_t *chunk = (const uint32_t *)vendor;
564+ uint32_t eax, ebx, ecx, edx;
565+
566+ cpuid(0, &eax, &ebx, &ecx, &edx);
567+ return (ebx == chunk[0] && edx == chunk[1] && ecx == chunk[2]);
568+}
569+
570+static inline bool this_cpu_is_intel(void)
571+{
572+ return this_cpu_vendor_string_is("GenuineIntel");
573+}
574+
575+/*
576+ * Exclude early K5 samples with a vendor string of "AMDisbetter!"
577+ */
578+static inline bool this_cpu_is_amd(void)
579+{
580+ return this_cpu_vendor_string_is("AuthenticAMD");
581+}
582+
557583 static inline uint32_t __this_cpu_has(uint32_t function, uint32_t index,
558584 uint8_t reg, uint8_t lo, uint8_t hi)
559585 {
@@ -690,9 +716,6 @@ static inline void cpu_relax(void) · 130 unmodified lines
690716 "hlt\n" \
691717 )
692718
693−bool is_intel_cpu(void);
694−bool is_amd_cpu(void);
695−
696719 struct kvm_x86_state *vcpu_save_state(struct kvm_vcpu *vcpu);
697720 void vcpu_load_state(struct kvm_vcpu *vcpu, struct kvm_x86_state *state);
698721 void kvm_x86_state_cleanup(struct kvm_x86_state *state);
@@ -716,7 +739,7 @@ static inline void vcpu_msrs_set(struct kvm_vcpu *vcpu, struct kvm_msrs *msrs) · 17 unmodified lines
716739 int r = __vcpu_ioctl(vcpu, KVM_SET_MSRS, msrs);
717740
718741 TEST_ASSERT(r == msrs->nmsrs,
719− "KVM_GET_MSRS failed, r: %i (failed on MSR %x)",
742+ "KVM_SET_MSRS failed, r: %i (failed on MSR %x)",
720743 r, r < 0 || r >= msrs->nmsrs ? -1 : msrs->entries[r].index);
721744 }
722745 static inline void vcpu_debugregs_get(struct kvm_vcpu *vcpu,
tools/testing/selftests/kvm/kvm_binary_stats_test.c+10−11
@@ -19,6 +19,7 @@ · 18 unmodified lines
1919 #include "kvm_util.h"
2020 #include "asm/kvm.h"
2121 #include "linux/kvm.h"
22+#include "kselftest.h"
2223
2324 static void stats_test(int stats_fd)
2425 {
@@ -51,7 +52,7 @@ static void stats_test(int stats_fd) · 26 unmodified lines
5152
5253 /* Sanity check for other fields in header */
5354 if (header.num_desc == 0) {
54− printf("No KVM stats defined!");
55+ ksft_print_msg("No KVM stats defined!\n");
5556 return;
5657 }
5758 /*
@@ -133,7 +134,7 @@ static void stats_test(int stats_fd) · 75 unmodified lines
133134 "Bucket size of stats (%s) is not zero",
134135 pdesc->name);
135136 }
136− size_data += pdesc->size * sizeof(*stats_data);
137+ size_data = max(size_data, pdesc->offset + pdesc->size * sizeof(*stats_data));
137138 }
138139
139140 /*
@@ -148,14 +149,6 @@ static void stats_test(int stats_fd) · 8 unmodified lines
148149 TEST_ASSERT(size_data >= header.num_desc * sizeof(*stats_data),
149150 "Data size is not correct");
150151
151− /* Check stats offset */
152− for (i = 0; i < header.num_desc; ++i) {
153− pdesc = get_stats_descriptor(stats_desc, i, &header);
154− TEST_ASSERT(pdesc->offset < size_data,
155− "Invalid offset (%u) for stats: %s",
156− pdesc->offset, pdesc->name);
157− }
158−
159152 /* Allocate memory for stats data */
160153 stats_data = malloc(size_data);
161154 TEST_ASSERT(stats_data, "Allocate memory for stats data");
@@ -224,9 +217,13 @@ int main(int argc, char *argv[]) · 62 unmodified lines
224217 max_vcpu = DEFAULT_NUM_VCPU;
225218 }
226219
220+ ksft_print_header();
221+
227222 /* Check the extension for binary stats */
228223 TEST_REQUIRE(kvm_has_cap(KVM_CAP_BINARY_STATS_FD));
229224
225+ ksft_set_plan(max_vm);
226+
230227 /* Create VMs and VCPUs */
231228 vms = malloc(sizeof(vms[0]) * max_vm);
232229 TEST_ASSERT(vms, "Allocate memory for storing VM pointers");
@@ -245,10 +242,12 @@ int main(int argc, char *argv[]) · 12 unmodified lines
245242 vm_stats_test(vms[i]);
246243 for (j = 0; j < max_vcpu; ++j)
247244 vcpu_stats_test(vcpus[i * max_vcpu + j]);
245+ ksft_test_result_pass("vm%i\n", i);
248246 }
249247
250248 for (i = 0; i < max_vm; ++i)
251249 kvm_vm_free(vms[i]);
252250 free(vms);
253− return 0;
251+
252+ ksft_finished(); /* Print results and exit() accordingly */
254253 }
tools/testing/selftests/kvm/lib/elf.c+2−0
@@ -90,6 +90,7 @@ static void elfhdr_get(const char *filename, Elf64_Ehdr *hdrp) · 89 unmodified lines
9090 " hdrp->e_shentsize: %x\n"
9191 " expected: %zx",
9292 hdrp->e_shentsize, sizeof(Elf64_Shdr));
93+ close(fd);
9394 }
9495
9596 /* VM ELF Load
@@ -190,4 +191,5 @@ void kvm_vm_elf_load(struct kvm_vm *vm, const char *filename) · 94 unmodified lines
190191 phdr.p_filesz);
191192 }
192193 }
194+ close(fd);
193195 }
tools/testing/selftests/kvm/lib/guest_modes.c+1−1
@@ -127,7 +127,7 @@ void guest_modes_cmdline(const char *arg) · 126 unmodified lines
127127 mode_selected = true;
128128 }
129129
130− mode = strtoul(optarg, NULL, 10);
130+ mode = atoi_non_negative("Guest mode ID", arg);
131131 TEST_ASSERT(mode < NUM_VM_MODES, "Guest mode ID %d too big", mode);
132132 guest_modes[mode].enabled = true;
133133 }
tools/testing/selftests/kvm/lib/kvm_util.c+1−3
@@ -1844,6 +1844,7 @@ static struct exit_reason { · 1843 unmodified lines
18441844 {KVM_EXIT_X86_RDMSR, "RDMSR"},
18451845 {KVM_EXIT_X86_WRMSR, "WRMSR"},
18461846 {KVM_EXIT_XEN, "XEN"},
1847+ {KVM_EXIT_HYPERV, "HYPERV"},
18471848 #ifdef KVM_EXIT_MEMORY_NOT_PRESENT
18481849 {KVM_EXIT_MEMORY_NOT_PRESENT, "MEMORY_NOT_PRESENT"},
18491850 #endif
@@ -1941,9 +1942,6 @@ vm_paddr_t vm_phy_page_alloc(struct kvm_vm *vm, vm_paddr_t paddr_min, · 91 unmodified lines
19411942 return vm_phy_pages_alloc(vm, 1, paddr_min, memslot);
19421943 }
19431944
1944−/* Arbitrary minimum physical address used for virtual translation tables. */
1945−#define KVM_GUEST_PAGE_TABLE_MIN_PADDR 0x180000
1946−
19471945 vm_paddr_t vm_alloc_page_table(struct kvm_vm *vm)
19481946 {
19491947 return vm_phy_page_alloc(vm, KVM_GUEST_PAGE_TABLE_MIN_PADDR,
tools/testing/selftests/kvm/lib/x86_64/processor.c+20−26
@@ -19,6 +19,8 @@ · 18 unmodified lines
1919 #define MAX_NR_CPUID_ENTRIES 100
2020
2121 vm_vaddr_t exception_handlers;
22+bool host_cpu_is_amd;
23+bool host_cpu_is_intel;
2224
2325 static void regs_dump(FILE *stream, struct kvm_regs *regs, uint8_t indent)
2426 {
@@ -113,7 +115,7 @@ static void sregs_dump(FILE *stream, struct kvm_sregs *sregs, uint8_t indent) · 88 unmodified lines
113115
114116 bool kvm_is_tdp_enabled(void)
115117 {
116− if (is_intel_cpu())
118+ if (host_cpu_is_intel)
117119 return get_kvm_intel_param_bool("ept");
118120 else
119121 return get_kvm_amd_param_bool("npt");
@@ -555,6 +557,8 @@ static void vcpu_setup(struct kvm_vm *vm, struct kvm_vcpu *vcpu) · 435 unmodified lines
555557 void kvm_arch_vm_post_create(struct kvm_vm *vm)
556558 {
557559 vm_create_irqchip(vm);
560+ sync_global_to_guest(vm, host_cpu_is_intel);
561+ sync_global_to_guest(vm, host_cpu_is_amd);
558562 }
559563
560564 struct kvm_vcpu *vm_arch_vcpu_add(struct kvm_vm *vm, uint32_t vcpu_id,
@@ -1006,28 +1010,6 @@ void kvm_x86_state_cleanup(struct kvm_x86_state *state) · 445 unmodified lines
10061010 free(state);
10071011 }
10081012
1009−static bool cpu_vendor_string_is(const char *vendor)
1010−{
1011− const uint32_t *chunk = (const uint32_t *)vendor;
1012− uint32_t eax, ebx, ecx, edx;
1013−
1014− cpuid(0, &eax, &ebx, &ecx, &edx);
1015− return (ebx == chunk[0] && edx == chunk[1] && ecx == chunk[2]);
1016−}
1017−
1018−bool is_intel_cpu(void)
1019−{
1020− return cpu_vendor_string_is("GenuineIntel");
1021−}
1022−
1023−/*
1024− * Exclude early K5 samples with a vendor string of "AMDisbetter!"
1025− */
1026−bool is_amd_cpu(void)
1027−{
1028− return cpu_vendor_string_is("AuthenticAMD");
1029−}
1030−
10311013 void kvm_get_cpu_address_width(unsigned int *pa_bits, unsigned int *va_bits)
10321014 {
10331015 if (!kvm_cpu_has_p(X86_PROPERTY_MAX_PHY_ADDR)) {
@@ -1162,9 +1144,15 @@ uint64_t kvm_hypercall(uint64_t nr, uint64_t a0, uint64_t a1, uint64_t a2, · 128 unmodified lines
11621144 {
11631145 uint64_t r;
11641146
1165− asm volatile("vmcall"
1147+ asm volatile("test %[use_vmmcall], %[use_vmmcall]\n\t"
1148+ "jnz 1f\n\t"
1149+ "vmcall\n\t"
1150+ "jmp 2f\n\t"
1151+ "1: vmmcall\n\t"
1152+ "2:"
11661153 : "=a"(r)
1167− : "a"(nr), "b"(a0), "c"(a1), "d"(a2), "S"(a3));
1154+ : "a"(nr), "b"(a0), "c"(a1), "d"(a2), "S"(a3),
1155+ [use_vmmcall] "r" (host_cpu_is_amd));
11681156 return r;
11691157 }
11701158
@@ -1236,7 +1224,7 @@ unsigned long vm_compute_max_gfn(struct kvm_vm *vm) · 65 unmodified lines
12361224 max_gfn = (1ULL << (vm->pa_bits - vm->page_shift)) - 1;
12371225
12381226 /* Avoid reserved HyperTransport region on AMD processors. */
1239− if (!is_amd_cpu())
1227+ if (!host_cpu_is_amd)
12401228 return max_gfn;
12411229
12421230 /* On parts with <40 physical address bits, the area is fully hidden */
@@ -1276,3 +1264,9 @@ bool vm_is_unrestricted_guest(struct kvm_vm *vm) · 33 unmodified lines
12761264
12771265 return get_kvm_intel_param_bool("unrestricted_guest");
12781266 }
1267+
1268+void kvm_selftest_arch_init(void)
1269+{
1270+ host_cpu_is_intel = this_cpu_is_intel();
1271+ host_cpu_is_amd = this_cpu_is_amd();
1272+}
tools/testing/selftests/kvm/memslot_perf_test.c+1−4
@@ -308,8 +308,6 @@ static bool prepare_vm(struct vm_data *data, int nslots, uint64_t *maxslots, · 307 unmodified lines
308308 data->hva_slots = malloc(sizeof(*data->hva_slots) * data->nslots);
309309 TEST_ASSERT(data->hva_slots, "malloc() fail");
310310
311− data->vm = __vm_create_with_one_vcpu(&data->vcpu, mempages, guest_code);
312−
313311 pr_info_v("Adding slots 1..%i, each slot with %"PRIu64" pages + %"PRIu64" extra pages last\n",
314312 data->nslots, data->pages_per_slot, rempages);
315313
@@ -349,6 +347,7 @@ static bool prepare_vm(struct vm_data *data, int nslots, uint64_t *maxslots, · 33 unmodified lines
349347 virt_map(data->vm, MEM_GPA, MEM_GPA, data->npages);
350348
351349 sync = (typeof(sync))vm_gpa2hva(data, MEM_SYNC_GPA, NULL);
350+ sync->guest_page_size = data->vm->page_size;
352351 atomic_init(&sync->start_flag, false);
353352 atomic_init(&sync->exit_flag, false);
354353 atomic_init(&sync->sync_flag, false);
@@ -810,8 +809,6 @@ static bool test_execute(int nslots, uint64_t *maxslots, · 455 unmodified lines
810809 }
811810
812811 sync = (typeof(sync))vm_gpa2hva(data, MEM_SYNC_GPA, NULL);
813−
814− sync->guest_page_size = data->vm->page_size;
815812 if (tdata->prepare &&
816813 !tdata->prepare(data, sync, maxslots)) {
817814 ret = false;
tools/testing/selftests/kvm/s390x/memop.c+524−148
@@ -9,6 +9,7 @@ · 8 unmodified lines
99 #include <stdlib.h>
1010 #include <string.h>
1111 #include <sys/ioctl.h>
12+#include <pthread.h>
1213
1314 #include <linux/bits.h>
1415
@@ -26,6 +27,7 @@ enum mop_target { · 11 unmodified lines
2627 enum mop_access_mode {
2728 READ,
2829 WRITE,
30+ CMPXCHG,
2931 };
3032
3133 struct mop_desc {
@@ -44,57 +46,67 @@ struct mop_desc { · 12 unmodified lines
4446 enum mop_access_mode mode;
4547 void *buf;
4648 uint32_t sida_offset;
49+ void *old;
50+ uint8_t old_value[16];
51+ bool *cmpxchg_success;
4752 uint8_t ar;
4853 uint8_t key;
4954 };
5055
51−static struct kvm_s390_mem_op ksmo_from_desc(struct mop_desc desc)
56+const uint8_t NO_KEY = 0xff;
57+
58+static struct kvm_s390_mem_op ksmo_from_desc(struct mop_desc *desc)
5259 {
5360 struct kvm_s390_mem_op ksmo = {
54− .gaddr = (uintptr_t)desc.gaddr,
55− .size = desc.size,
56− .buf = ((uintptr_t)desc.buf),
61+ .gaddr = (uintptr_t)desc->gaddr,
62+ .size = desc->size,
63+ .buf = ((uintptr_t)desc->buf),
5764 .reserved = "ignored_ignored_ignored_ignored"
5865 };
5966
60− switch (desc.target) {
67+ switch (desc->target) {
6168 case LOGICAL:
62− if (desc.mode == READ)
69+ if (desc->mode == READ)
6370 ksmo.op = KVM_S390_MEMOP_LOGICAL_READ;
64− if (desc.mode == WRITE)
71+ if (desc->mode == WRITE)
6572 ksmo.op = KVM_S390_MEMOP_LOGICAL_WRITE;
6673 break;
6774 case SIDA:
68− if (desc.mode == READ)
75+ if (desc->mode == READ)
6976 ksmo.op = KVM_S390_MEMOP_SIDA_READ;
70− if (desc.mode == WRITE)
77+ if (desc->mode == WRITE)
7178 ksmo.op = KVM_S390_MEMOP_SIDA_WRITE;
7279 break;
7380 case ABSOLUTE:
74− if (desc.mode == READ)
81+ if (desc->mode == READ)
7582 ksmo.op = KVM_S390_MEMOP_ABSOLUTE_READ;
76− if (desc.mode == WRITE)
83+ if (desc->mode == WRITE)
7784 ksmo.op = KVM_S390_MEMOP_ABSOLUTE_WRITE;
85+ if (desc->mode == CMPXCHG) {
86+ ksmo.op = KVM_S390_MEMOP_ABSOLUTE_CMPXCHG;
87+ ksmo.old_addr = (uint64_t)desc->old;
88+ memcpy(desc->old_value, desc->old, desc->size);
89+ }
7890 break;
7991 case INVALID:
8092 ksmo.op = -1;
8193 }
82− if (desc.f_check)
94+ if (desc->f_check)
8395 ksmo.flags |= KVM_S390_MEMOP_F_CHECK_ONLY;
84− if (desc.f_inject)
96+ if (desc->f_inject)
8597 ksmo.flags |= KVM_S390_MEMOP_F_INJECT_EXCEPTION;
86− if (desc._set_flags)
87− ksmo.flags = desc.set_flags;
88− if (desc.f_key) {
98+ if (desc->_set_flags)
99+ ksmo.flags = desc->set_flags;
100+ if (desc->f_key && desc->key != NO_KEY) {
89101 ksmo.flags |= KVM_S390_MEMOP_F_SKEY_PROTECTION;
90− ksmo.key = desc.key;
102+ ksmo.key = desc->key;
91103 }
92− if (desc._ar)
93− ksmo.ar = desc.ar;
104+ if (desc->_ar)
105+ ksmo.ar = desc->ar;
94106 else
95107 ksmo.ar = 0;
96− if (desc._sida_offset)
97− ksmo.sida_offset = desc.sida_offset;
108+ if (desc->_sida_offset)
109+ ksmo.sida_offset = desc->sida_offset;
98110
99111 return ksmo;
100112 }
@@ -133,9 +145,13 @@ static void print_memop(struct kvm_vcpu *vcpu, const struct kvm_s390_mem_op *ksm · 32 unmodified lines
133145 case KVM_S390_MEMOP_ABSOLUTE_WRITE:
134146 printf("ABSOLUTE, WRITE, ");
135147 break;
148+ case KVM_S390_MEMOP_ABSOLUTE_CMPXCHG:
149+ printf("ABSOLUTE, CMPXCHG, ");
150+ break;
136151 }
137− printf("gaddr=%llu, size=%u, buf=%llu, ar=%u, key=%u",
138− ksmo->gaddr, ksmo->size, ksmo->buf, ksmo->ar, ksmo->key);
152+ printf("gaddr=%llu, size=%u, buf=%llu, ar=%u, key=%u, old_addr=%llx",
153+ ksmo->gaddr, ksmo->size, ksmo->buf, ksmo->ar, ksmo->key,
154+ ksmo->old_addr);
139155 if (ksmo->flags & KVM_S390_MEMOP_F_CHECK_ONLY)
140156 printf(", CHECK_ONLY");
141157 if (ksmo->flags & KVM_S390_MEMOP_F_INJECT_EXCEPTION)
@@ -145,24 +161,30 @@ static void print_memop(struct kvm_vcpu *vcpu, const struct kvm_s390_mem_op *ksm · 3 unmodified lines
145161 puts(")");
146162 }
147163
148−static void memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo)
164+static int err_memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo,
165+ struct mop_desc *desc)
149166 {
150167 struct kvm_vcpu *vcpu = info.vcpu;
151168
152169 if (!vcpu)
153− vm_ioctl(info.vm, KVM_S390_MEM_OP, ksmo);
170+ return __vm_ioctl(info.vm, KVM_S390_MEM_OP, ksmo);
154171 else
155− vcpu_ioctl(vcpu, KVM_S390_MEM_OP, ksmo);
172+ return __vcpu_ioctl(vcpu, KVM_S390_MEM_OP, ksmo);
156173 }
157174
158−static int err_memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo)
175+static void memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo,
176+ struct mop_desc *desc)
159177 {
160− struct kvm_vcpu *vcpu = info.vcpu;
178+ int r;
161179
162− if (!vcpu)
163− return __vm_ioctl(info.vm, KVM_S390_MEM_OP, ksmo);
164− else
165− return __vcpu_ioctl(vcpu, KVM_S390_MEM_OP, ksmo);
180+ r = err_memop_ioctl(info, ksmo, desc);
181+ if (ksmo->op == KVM_S390_MEMOP_ABSOLUTE_CMPXCHG) {
182+ if (desc->cmpxchg_success) {
183+ int diff = memcmp(desc->old_value, desc->old, desc->size);
184+ *desc->cmpxchg_success = !diff;
185+ }
186+ }
187+ TEST_ASSERT(!r, __KVM_IOCTL_ERROR("KVM_S390_MEM_OP", r));
166188 }
167189
168190 #define MEMOP(err, info_p, mop_target_p, access_mode_p, buf_p, size_p, ...) \
@@ -183,9 +205,9 @@ static int err_memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo) · 14 unmodified lines
183205 else \
184206 __desc.gaddr = __desc.gaddr_v; \
185207 } \
186− __ksmo = ksmo_from_desc(__desc); \
208+ __ksmo = ksmo_from_desc(&__desc); \
187209 print_memop(__info.vcpu, &__ksmo); \
188− err##memop_ioctl(__info, &__ksmo); \
210+ err##memop_ioctl(__info, &__ksmo, &__desc); \
189211 })
190212
191213 #define MOP(...) MEMOP(, __VA_ARGS__)
@@ -199,6 +221,8 @@ static int err_memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo) · 7 unmodified lines
199221 #define AR(a) ._ar = 1, .ar = (a)
200222 #define KEY(a) .f_key = 1, .key = (a)
201223 #define INJECT .f_inject = 1
224+#define CMPXCHG_OLD(o) .old = (o)
225+#define CMPXCHG_SUCCESS(s) .cmpxchg_success = (s)
202226
203227 #define CHECK_N_DO(f, ...) ({ f(__VA_ARGS__, CHECK_ONLY); f(__VA_ARGS__); })
204228
@@ -208,8 +232,8 @@ static int err_memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo) · 3 unmodified lines
208232 #define CR0_FETCH_PROTECTION_OVERRIDE (1UL << (63 - 38))
209233 #define CR0_STORAGE_PROTECTION_OVERRIDE (1UL << (63 - 39))
210234
211−static uint8_t mem1[65536];
212−static uint8_t mem2[65536];
235+static uint8_t __aligned(PAGE_SIZE) mem1[65536];
236+static uint8_t __aligned(PAGE_SIZE) mem2[65536];
213237
214238 struct test_default {
215239 struct kvm_vm *kvm_vm;
@@ -241,6 +265,8 @@ enum stage { · 25 unmodified lines
241265 STAGE_SKEYS_SET,
242266 /* Guest copied memory (locations up to test case) */
243267 STAGE_COPIED,
268+ /* End of guest code reached */
269+ STAGE_DONE,
244270 };
245271
246272 #define HOST_SYNC(info_p, stage) \
@@ -252,6 +278,9 @@ enum stage { · 5 unmodified lines
252278 \
253279 vcpu_run(__vcpu); \
254280 get_ucall(__vcpu, &uc); \
281+ if (uc.cmd == UCALL_ABORT) { \
282+ REPORT_GUEST_ASSERT_2(uc, "hints: %lu, %lu"); \
283+ } \
255284 ASSERT_EQ(uc.cmd, UCALL_SYNC); \
256285 ASSERT_EQ(uc.args[1], __stage); \
257286 }) \
@@ -268,34 +297,66 @@ static void prepare_mem12(void) · 10 unmodified lines
268297 #define ASSERT_MEM_EQ(p1, p2, size) \
269298 TEST_ASSERT(!memcmp(p1, p2, size), "Memory contents do not match!")
270299
271−#define DEFAULT_WRITE_READ(copy_cpu, mop_cpu, mop_target_p, size, ...) \
272−({ \
273− struct test_info __copy_cpu = (copy_cpu), __mop_cpu = (mop_cpu); \
274− enum mop_target __target = (mop_target_p); \
275− uint32_t __size = (size); \
276− \
277− prepare_mem12(); \
278− CHECK_N_DO(MOP, __mop_cpu, __target, WRITE, mem1, __size, \
279− GADDR_V(mem1), ##__VA_ARGS__); \
280− HOST_SYNC(__copy_cpu, STAGE_COPIED); \
281− CHECK_N_DO(MOP, __mop_cpu, __target, READ, mem2, __size, \
282− GADDR_V(mem2), ##__VA_ARGS__); \
283− ASSERT_MEM_EQ(mem1, mem2, __size); \
284−})
300+static void default_write_read(struct test_info copy_cpu, struct test_info mop_cpu,
301+ enum mop_target mop_target, uint32_t size, uint8_t key)
302+{
303+ prepare_mem12();
304+ CHECK_N_DO(MOP, mop_cpu, mop_target, WRITE, mem1, size,
305+ GADDR_V(mem1), KEY(key));
306+ HOST_SYNC(copy_cpu, STAGE_COPIED);
307+ CHECK_N_DO(MOP, mop_cpu, mop_target, READ, mem2, size,
308+ GADDR_V(mem2), KEY(key));
309+ ASSERT_MEM_EQ(mem1, mem2, size);
310+}
285311
286−#define DEFAULT_READ(copy_cpu, mop_cpu, mop_target_p, size, ...) \
287−({ \
288− struct test_info __copy_cpu = (copy_cpu), __mop_cpu = (mop_cpu); \
289− enum mop_target __target = (mop_target_p); \
290− uint32_t __size = (size); \
291− \
292− prepare_mem12(); \
293− CHECK_N_DO(MOP, __mop_cpu, __target, WRITE, mem1, __size, \
294− GADDR_V(mem1)); \
295− HOST_SYNC(__copy_cpu, STAGE_COPIED); \
296− CHECK_N_DO(MOP, __mop_cpu, __target, READ, mem2, __size, ##__VA_ARGS__);\
297− ASSERT_MEM_EQ(mem1, mem2, __size); \
298−})
312+static void default_read(struct test_info copy_cpu, struct test_info mop_cpu,
313+ enum mop_target mop_target, uint32_t size, uint8_t key)
314+{
315+ prepare_mem12();
316+ CHECK_N_DO(MOP, mop_cpu, mop_target, WRITE, mem1, size, GADDR_V(mem1));
317+ HOST_SYNC(copy_cpu, STAGE_COPIED);
318+ CHECK_N_DO(MOP, mop_cpu, mop_target, READ, mem2, size,
319+ GADDR_V(mem2), KEY(key));
320+ ASSERT_MEM_EQ(mem1, mem2, size);
321+}
322+
323+static void default_cmpxchg(struct test_default *test, uint8_t key)
324+{
325+ for (int size = 1; size <= 16; size *= 2) {
326+ for (int offset = 0; offset < 16; offset += size) {
327+ uint8_t __aligned(16) new[16] = {};
328+ uint8_t __aligned(16) old[16];
329+ bool succ;
330+
331+ prepare_mem12();
332+ default_write_read(test->vcpu, test->vcpu, LOGICAL, 16, NO_KEY);
333+
334+ memcpy(&old, mem1, 16);
335+ MOP(test->vm, ABSOLUTE, CMPXCHG, new + offset,
336+ size, GADDR_V(mem1 + offset),
337+ CMPXCHG_OLD(old + offset),
338+ CMPXCHG_SUCCESS(&succ), KEY(key));
339+ HOST_SYNC(test->vcpu, STAGE_COPIED);
340+ MOP(test->vm, ABSOLUTE, READ, mem2, 16, GADDR_V(mem2));
341+ TEST_ASSERT(succ, "exchange of values should succeed");
342+ memcpy(mem1 + offset, new + offset, size);
343+ ASSERT_MEM_EQ(mem1, mem2, 16);
344+
345+ memcpy(&old, mem1, 16);
346+ new[offset]++;
347+ old[offset]++;
348+ MOP(test->vm, ABSOLUTE, CMPXCHG, new + offset,
349+ size, GADDR_V(mem1 + offset),
350+ CMPXCHG_OLD(old + offset),
351+ CMPXCHG_SUCCESS(&succ), KEY(key));
352+ HOST_SYNC(test->vcpu, STAGE_COPIED);
353+ MOP(test->vm, ABSOLUTE, READ, mem2, 16, GADDR_V(mem2));
354+ TEST_ASSERT(!succ, "exchange of values should not succeed");
355+ ASSERT_MEM_EQ(mem1, mem2, 16);
356+ ASSERT_MEM_EQ(&old, mem1, 16);
357+ }
358+ }
359+}
299360
300361 static void guest_copy(void)
301362 {
@@ -310,7 +371,7 @@ static void test_copy(void) · 8 unmodified lines
310371
311372 HOST_SYNC(t.vcpu, STAGE_INITED);
312373
313− DEFAULT_WRITE_READ(t.vcpu, t.vcpu, LOGICAL, t.size);
374+ default_write_read(t.vcpu, t.vcpu, LOGICAL, t.size, NO_KEY);
314375
315376 kvm_vm_free(t.kvm_vm);
316377 }
@@ -357,26 +418,268 @@ static void test_copy_key(void) · 40 unmodified lines
357418 HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);
358419
359420 /* vm, no key */
360− DEFAULT_WRITE_READ(t.vcpu, t.vm, ABSOLUTE, t.size);
421+ default_write_read(t.vcpu, t.vm, ABSOLUTE, t.size, NO_KEY);
361422
362423 /* vm/vcpu, machting key or key 0 */
363− DEFAULT_WRITE_READ(t.vcpu, t.vcpu, LOGICAL, t.size, KEY(0));
364− DEFAULT_WRITE_READ(t.vcpu, t.vcpu, LOGICAL, t.size, KEY(9));
365− DEFAULT_WRITE_READ(t.vcpu, t.vm, ABSOLUTE, t.size, KEY(0));
366− DEFAULT_WRITE_READ(t.vcpu, t.vm, ABSOLUTE, t.size, KEY(9));
424+ default_write_read(t.vcpu, t.vcpu, LOGICAL, t.size, 0);
425+ default_write_read(t.vcpu, t.vcpu, LOGICAL, t.size, 9);
426+ default_write_read(t.vcpu, t.vm, ABSOLUTE, t.size, 0);
427+ default_write_read(t.vcpu, t.vm, ABSOLUTE, t.size, 9);
367428 /*
368429 * There used to be different code paths for key handling depending on
369430 * if the region crossed a page boundary.
370431 * There currently are not, but the more tests the merrier.
371432 */
372− DEFAULT_WRITE_READ(t.vcpu, t.vcpu, LOGICAL, 1, KEY(0));
373− DEFAULT_WRITE_READ(t.vcpu, t.vcpu, LOGICAL, 1, KEY(9));
374− DEFAULT_WRITE_READ(t.vcpu, t.vm, ABSOLUTE, 1, KEY(0));
375− DEFAULT_WRITE_READ(t.vcpu, t.vm, ABSOLUTE, 1, KEY(9));
433+ default_write_read(t.vcpu, t.vcpu, LOGICAL, 1, 0);
434+ default_write_read(t.vcpu, t.vcpu, LOGICAL, 1, 9);
435+ default_write_read(t.vcpu, t.vm, ABSOLUTE, 1, 0);
436+ default_write_read(t.vcpu, t.vm, ABSOLUTE, 1, 9);
376437
377438 /* vm/vcpu, mismatching keys on read, but no fetch protection */
378− DEFAULT_READ(t.vcpu, t.vcpu, LOGICAL, t.size, GADDR_V(mem2), KEY(2));
379− DEFAULT_READ(t.vcpu, t.vm, ABSOLUTE, t.size, GADDR_V(mem1), KEY(2));
439+ default_read(t.vcpu, t.vcpu, LOGICAL, t.size, 2);
440+ default_read(t.vcpu, t.vm, ABSOLUTE, t.size, 2);
441+
442+ kvm_vm_free(t.kvm_vm);
443+}
444+
445+static void test_cmpxchg_key(void)
446+{
447+ struct test_default t = test_default_init(guest_copy_key);
448+
449+ HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);
450+
451+ default_cmpxchg(&t, NO_KEY);
452+ default_cmpxchg(&t, 0);
453+ default_cmpxchg(&t, 9);
454+
455+ kvm_vm_free(t.kvm_vm);
456+}
457+
458+static __uint128_t cut_to_size(int size, __uint128_t val)
459+{
460+ switch (size) {
461+ case 1:
462+ return (uint8_t)val;
463+ case 2:
464+ return (uint16_t)val;
465+ case 4:
466+ return (uint32_t)val;
467+ case 8:
468+ return (uint64_t)val;
469+ case 16:
470+ return val;
471+ }
472+ GUEST_ASSERT_1(false, "Invalid size");
473+ return 0;
474+}
475+
476+static bool popcount_eq(__uint128_t a, __uint128_t b)
477+{
478+ unsigned int count_a, count_b;
479+
480+ count_a = __builtin_popcountl((uint64_t)(a >> 64)) +
481+ __builtin_popcountl((uint64_t)a);
482+ count_b = __builtin_popcountl((uint64_t)(b >> 64)) +
483+ __builtin_popcountl((uint64_t)b);
484+ return count_a == count_b;
485+}
486+
487+static __uint128_t rotate(int size, __uint128_t val, int amount)
488+{
489+ unsigned int bits = size * 8;
490+
491+ amount = (amount + bits) % bits;
492+ val = cut_to_size(size, val);
493+ return (val << (bits - amount)) | (val >> amount);
494+}
495+
496+const unsigned int max_block = 16;
497+
498+static void choose_block(bool guest, int i, int *size, int *offset)
499+{
500+ unsigned int rand;
501+
502+ rand = i;
503+ if (guest) {
504+ rand = rand * 19 + 11;
505+ *size = 1 << ((rand % 3) + 2);
506+ rand = rand * 19 + 11;
507+ *offset = (rand % max_block) & ~(*size - 1);
508+ } else {
509+ rand = rand * 17 + 5;
510+ *size = 1 << (rand % 5);
511+ rand = rand * 17 + 5;
512+ *offset = (rand % max_block) & ~(*size - 1);
513+ }
514+}
515+
516+static __uint128_t permutate_bits(bool guest, int i, int size, __uint128_t old)
517+{
518+ unsigned int rand;
519+ int amount;
520+ bool swap;
521+
522+ rand = i;
523+ rand = rand * 3 + 1;
524+ if (guest)
525+ rand = rand * 3 + 1;
526+ swap = rand % 2 == 0;
527+ if (swap) {
528+ int i, j;
529+ __uint128_t new;
530+ uint8_t byte0, byte1;
531+
532+ rand = rand * 3 + 1;
533+ i = rand % size;
534+ rand = rand * 3 + 1;
535+ j = rand % size;
536+ if (i == j)
537+ return old;
538+ new = rotate(16, old, i * 8);
539+ byte0 = new & 0xff;
540+ new &= ~0xff;
541+ new = rotate(16, new, -i * 8);
542+ new = rotate(16, new, j * 8);
543+ byte1 = new & 0xff;
544+ new = (new & ~0xff) | byte0;
545+ new = rotate(16, new, -j * 8);
546+ new = rotate(16, new, i * 8);
547+ new = new | byte1;
548+ new = rotate(16, new, -i * 8);
549+ return new;
550+ }
551+ rand = rand * 3 + 1;
552+ amount = rand % (size * 8);
553+ return rotate(size, old, amount);
554+}
555+
556+static bool _cmpxchg(int size, void *target, __uint128_t *old_addr, __uint128_t new)
557+{
558+ bool ret;
559+
560+ switch (size) {
561+ case 4: {
562+ uint32_t old = *old_addr;
563+
564+ asm volatile ("cs %[old],%[new],%[address]"
565+ : [old] "+d" (old),
566+ [address] "+Q" (*(uint32_t *)(target))
567+ : [new] "d" ((uint32_t)new)
568+ : "cc"
569+ );
570+ ret = old == (uint32_t)*old_addr;
571+ *old_addr = old;
572+ return ret;
573+ }
574+ case 8: {
575+ uint64_t old = *old_addr;
576+
577+ asm volatile ("csg %[old],%[new],%[address]"
578+ : [old] "+d" (old),
579+ [address] "+Q" (*(uint64_t *)(target))
580+ : [new] "d" ((uint64_t)new)
581+ : "cc"
582+ );
583+ ret = old == (uint64_t)*old_addr;
584+ *old_addr = old;
585+ return ret;
586+ }
587+ case 16: {
588+ __uint128_t old = *old_addr;
589+
590+ asm volatile ("cdsg %[old],%[new],%[address]"
591+ : [old] "+d" (old),
592+ [address] "+Q" (*(__uint128_t *)(target))
593+ : [new] "d" (new)
594+ : "cc"
595+ );
596+ ret = old == *old_addr;
597+ *old_addr = old;
598+ return ret;
599+ }
600+ }
601+ GUEST_ASSERT_1(false, "Invalid size");
602+ return 0;
603+}
604+
605+const unsigned int cmpxchg_iter_outer = 100, cmpxchg_iter_inner = 10000;
606+
607+static void guest_cmpxchg_key(void)
608+{
609+ int size, offset;
610+ __uint128_t old, new;
611+
612+ set_storage_key_range(mem1, max_block, 0x10);
613+ set_storage_key_range(mem2, max_block, 0x10);
614+ GUEST_SYNC(STAGE_SKEYS_SET);
615+
616+ for (int i = 0; i < cmpxchg_iter_outer; i++) {
617+ do {
618+ old = 1;
619+ } while (!_cmpxchg(16, mem1, &old, 0));
620+ for (int j = 0; j < cmpxchg_iter_inner; j++) {
621+ choose_block(true, i + j, &size, &offset);
622+ do {
623+ new = permutate_bits(true, i + j, size, old);
624+ } while (!_cmpxchg(size, mem2 + offset, &old, new));
625+ }
626+ }
627+
628+ GUEST_SYNC(STAGE_DONE);
629+}
630+
631+static void *run_guest(void *data)
632+{
633+ struct test_info *info = data;
634+
635+ HOST_SYNC(*info, STAGE_DONE);
636+ return NULL;
637+}
638+
639+static char *quad_to_char(__uint128_t *quad, int size)
640+{
641+ return ((char *)quad) + (sizeof(*quad) - size);
642+}
643+
644+static void test_cmpxchg_key_concurrent(void)
645+{
646+ struct test_default t = test_default_init(guest_cmpxchg_key);
647+ int size, offset;
648+ __uint128_t old, new;
649+ bool success;
650+ pthread_t thread;
651+
652+ HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);
653+ prepare_mem12();
654+ MOP(t.vcpu, LOGICAL, WRITE, mem1, max_block, GADDR_V(mem2));
655+ pthread_create(&thread, NULL, run_guest, &t.vcpu);
656+
657+ for (int i = 0; i < cmpxchg_iter_outer; i++) {
658+ do {
659+ old = 0;
660+ new = 1;
661+ MOP(t.vm, ABSOLUTE, CMPXCHG, &new,
662+ sizeof(new), GADDR_V(mem1),
663+ CMPXCHG_OLD(&old),
664+ CMPXCHG_SUCCESS(&success), KEY(1));
665+ } while (!success);
666+ for (int j = 0; j < cmpxchg_iter_inner; j++) {
667+ choose_block(false, i + j, &size, &offset);
668+ do {
669+ new = permutate_bits(false, i + j, size, old);
670+ MOP(t.vm, ABSOLUTE, CMPXCHG, quad_to_char(&new, size),
671+ size, GADDR_V(mem2 + offset),
672+ CMPXCHG_OLD(quad_to_char(&old, size)),
673+ CMPXCHG_SUCCESS(&success), KEY(1));
674+ } while (!success);
675+ }
676+ }
677+
678+ pthread_join(thread, NULL);
679+
680+ MOP(t.vcpu, LOGICAL, READ, mem2, max_block, GADDR_V(mem2));
681+ TEST_ASSERT(popcount_eq(*(__uint128_t *)mem1, *(__uint128_t *)mem2),
682+ "Must retain number of set bits");
380683
381684 kvm_vm_free(t.kvm_vm);
382685 }
@@ -409,7 +712,7 @@ static void test_copy_key_storage_prot_override(void) · 26 unmodified lines
409712 HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);
410713
411714 /* vcpu, mismatching keys, storage protection override in effect */
412− DEFAULT_WRITE_READ(t.vcpu, t.vcpu, LOGICAL, t.size, KEY(2));
715+ default_write_read(t.vcpu, t.vcpu, LOGICAL, t.size, 2);
413716
414717 kvm_vm_free(t.kvm_vm);
415718 }
@@ -422,8 +725,8 @@ static void test_copy_key_fetch_prot(void) · 6 unmodified lines
422725 HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);
423726
424727 /* vm/vcpu, matching key, fetch protection in effect */
425− DEFAULT_READ(t.vcpu, t.vcpu, LOGICAL, t.size, GADDR_V(mem2), KEY(9));
426− DEFAULT_READ(t.vcpu, t.vm, ABSOLUTE, t.size, GADDR_V(mem2), KEY(9));
728+ default_read(t.vcpu, t.vcpu, LOGICAL, t.size, 9);
729+ default_read(t.vcpu, t.vm, ABSOLUTE, t.size, 9);
427730
428731 kvm_vm_free(t.kvm_vm);
429732 }
@@ -454,9 +757,27 @@ static void test_errors_key(void) · 24 unmodified lines
454757
455758 /* vm/vcpu, mismatching keys, fetch protection in effect */
456759 CHECK_N_DO(ERR_PROT_MOP, t.vcpu, LOGICAL, WRITE, mem1, t.size, GADDR_V(mem1), KEY(2));
457− CHECK_N_DO(ERR_PROT_MOP, t.vcpu, LOGICAL, READ, mem2, t.size, GADDR_V(mem2), KEY(2));
760+ CHECK_N_DO(ERR_PROT_MOP, t.vcpu, LOGICAL, READ, mem2, t.size, GADDR_V(mem1), KEY(2));
458761 CHECK_N_DO(ERR_PROT_MOP, t.vm, ABSOLUTE, WRITE, mem1, t.size, GADDR_V(mem1), KEY(2));
459− CHECK_N_DO(ERR_PROT_MOP, t.vm, ABSOLUTE, READ, mem2, t.size, GADDR_V(mem2), KEY(2));
762+ CHECK_N_DO(ERR_PROT_MOP, t.vm, ABSOLUTE, READ, mem2, t.size, GADDR_V(mem1), KEY(2));
763+
764+ kvm_vm_free(t.kvm_vm);
765+}
766+
767+static void test_errors_cmpxchg_key(void)
768+{
769+ struct test_default t = test_default_init(guest_copy_key_fetch_prot);
770+ int i;
771+
772+ HOST_SYNC(t.vcpu, STAGE_INITED);
773+ HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);
774+
775+ for (i = 1; i <= 16; i *= 2) {
776+ __uint128_t old = 0;
777+
778+ ERR_PROT_MOP(t.vm, ABSOLUTE, CMPXCHG, mem2, i, GADDR_V(mem2),
779+ CMPXCHG_OLD(&old), KEY(2));
780+ }
460781
461782 kvm_vm_free(t.kvm_vm);
462783 }
@@ -518,7 +839,7 @@ static void guest_copy_key_fetch_prot_override(void) · 55 unmodified lines
518839 GUEST_SYNC(STAGE_INITED);
519840 set_storage_key_range(0, PAGE_SIZE, 0x18);
520841 set_storage_key_range((void *)last_page_addr, PAGE_SIZE, 0x0);
521− asm volatile ("sske %[key],%[addr]\n" :: [addr] "r"(0), [key] "r"(0x18) : "cc");
842+ asm volatile ("sske %[key],%[addr]\n" :: [addr] "r"(0L), [key] "r"(0x18) : "cc");
522843 GUEST_SYNC(STAGE_SKEYS_SET);
523844
524845 for (;;) {
@@ -606,7 +927,7 @@ static void test_errors_key_fetch_prot_override_enabled(void) · 81 unmodified lines
606927
607928 /*
608929 * vcpu, mismatching keys on fetch,
609− * fetch protection override does not apply because memory range acceeded
930+ * fetch protection override does not apply because memory range exceeded
610931 */
611932 CHECK_N_DO(ERR_PROT_MOP, t.vcpu, LOGICAL, READ, mem2, 2048 + 1, GADDR_V(0), KEY(2));
612933 CHECK_N_DO(ERR_PROT_MOP, t.vcpu, LOGICAL, READ, mem2, PAGE_SIZE + 2048 + 1,
@@ -645,7 +966,9 @@ static void _test_errors_common(struct test_info info, enum mop_target target, i · 32 unmodified lines
645966
646967 /* Bad guest address: */
647968 rv = ERR_MOP(info, target, WRITE, mem1, size, GADDR((void *)~0xfffUL), CHECK_ONLY);
648− TEST_ASSERT(rv > 0, "ioctl does not report bad guest memory access");
969+ TEST_ASSERT(rv > 0, "ioctl does not report bad guest memory address with CHECK_ONLY");
970+ rv = ERR_MOP(info, target, WRITE, mem1, size, GADDR((void *)~0xfffUL));
971+ TEST_ASSERT(rv > 0, "ioctl does not report bad guest memory address on write");
649972
650973 /* Bad host address: */
651974 rv = ERR_MOP(info, target, WRITE, 0, size, GADDR_V(mem1));
@@ -694,85 +1017,138 @@ static void test_errors(void) · 42 unmodified lines
6941017 kvm_vm_free(t.kvm_vm);
6951018 }
6961019
697−struct testdef {
698− const char *name;
699− void (*test)(void);
700− int extension;
701−} testlist[] = {
702− {
703− .name = "simple copy",
704− .test = test_copy,
705− },
706− {
707− .name = "generic error checks",
708− .test = test_errors,
709− },
710− {
711− .name = "copy with storage keys",
712− .test = test_copy_key,
713− .extension = 1,
714− },
715− {
716− .name = "copy with key storage protection override",
717− .test = test_copy_key_storage_prot_override,
718− .extension = 1,
719− },
720− {
721− .name = "copy with key fetch protection",
722− .test = test_copy_key_fetch_prot,
723− .extension = 1,
724− },
725− {
726− .name = "copy with key fetch protection override",
727− .test = test_copy_key_fetch_prot_override,
728− .extension = 1,
729− },
730− {
731− .name = "error checks with key",
732− .test = test_errors_key,
733− .extension = 1,
734− },
735− {
736− .name = "termination",
737− .test = test_termination,
738− .extension = 1,
739− },
740− {
741− .name = "error checks with key storage protection override",
742− .test = test_errors_key_storage_prot_override,
743− .extension = 1,
744− },
745− {
746− .name = "error checks without key fetch prot override",
747− .test = test_errors_key_fetch_prot_override_not_enabled,
748− .extension = 1,
749− },
750− {
751− .name = "error checks with key fetch prot override",
752− .test = test_errors_key_fetch_prot_override_enabled,
753− .extension = 1,
754− },
755−};
1020+static void test_errors_cmpxchg(void)
1021+{
1022+ struct test_default t = test_default_init(guest_idle);
1023+ __uint128_t old;
1024+ int rv, i, power = 1;
1025+
1026+ HOST_SYNC(t.vcpu, STAGE_INITED);
1027+
1028+ for (i = 0; i < 32; i++) {
1029+ if (i == power) {
1030+ power *= 2;
1031+ continue;
1032+ }
1033+ rv = ERR_MOP(t.vm, ABSOLUTE, CMPXCHG, mem1, i, GADDR_V(mem1),
1034+ CMPXCHG_OLD(&old));
1035+ TEST_ASSERT(rv == -1 && errno == EINVAL,
1036+ "ioctl allows bad size for cmpxchg");
1037+ }
1038+ for (i = 1; i <= 16; i *= 2) {
1039+ rv = ERR_MOP(t.vm, ABSOLUTE, CMPXCHG, mem1, i, GADDR((void *)~0xfffUL),
1040+ CMPXCHG_OLD(&old));
1041+ TEST_ASSERT(rv > 0, "ioctl allows bad guest address for cmpxchg");
1042+ }
1043+ for (i = 2; i <= 16; i *= 2) {
1044+ rv = ERR_MOP(t.vm, ABSOLUTE, CMPXCHG, mem1, i, GADDR_V(mem1 + 1),
1045+ CMPXCHG_OLD(&old));
1046+ TEST_ASSERT(rv == -1 && errno == EINVAL,
1047+ "ioctl allows bad alignment for cmpxchg");
1048+ }
1049+
1050+ kvm_vm_free(t.kvm_vm);
1051+}
7561052
7571053 int main(int argc, char *argv[])
7581054 {
7591055 int extension_cap, idx;
7601056
7611057 TEST_REQUIRE(kvm_has_cap(KVM_CAP_S390_MEM_OP));
1058+ extension_cap = kvm_check_cap(KVM_CAP_S390_MEM_OP_EXTENSION);
7621059
763− ksft_print_header();
1060+ struct testdef {
1061+ const char *name;
1062+ void (*test)(void);
1063+ bool requirements_met;
1064+ } testlist[] = {
1065+ {
1066+ .name = "simple copy",
1067+ .test = test_copy,
1068+ .requirements_met = true,
1069+ },
1070+ {
1071+ .name = "generic error checks",
1072+ .test = test_errors,
1073+ .requirements_met = true,
1074+ },
1075+ {
1076+ .name = "copy with storage keys",
1077+ .test = test_copy_key,
1078+ .requirements_met = extension_cap > 0,
1079+ },
1080+ {
1081+ .name = "cmpxchg with storage keys",
1082+ .test = test_cmpxchg_key,
1083+ .requirements_met = extension_cap & 0x2,
1084+ },
1085+ {
1086+ .name = "concurrently cmpxchg with storage keys",
1087+ .test = test_cmpxchg_key_concurrent,
1088+ .requirements_met = extension_cap & 0x2,
1089+ },
1090+ {
1091+ .name = "copy with key storage protection override",
1092+ .test = test_copy_key_storage_prot_override,
1093+ .requirements_met = extension_cap > 0,
1094+ },
1095+ {
1096+ .name = "copy with key fetch protection",
1097+ .test = test_copy_key_fetch_prot,
1098+ .requirements_met = extension_cap > 0,
1099+ },
1100+ {
1101+ .name = "copy with key fetch protection override",
1102+ .test = test_copy_key_fetch_prot_override,
1103+ .requirements_met = extension_cap > 0,
1104+ },
1105+ {
1106+ .name = "error checks with key",
1107+ .test = test_errors_key,
1108+ .requirements_met = extension_cap > 0,
1109+ },
1110+ {
1111+ .name = "error checks for cmpxchg with key",
1112+ .test = test_errors_cmpxchg_key,
1113+ .requirements_met = extension_cap & 0x2,
1114+ },
1115+ {
1116+ .name = "error checks for cmpxchg",
1117+ .test = test_errors_cmpxchg,
1118+ .requirements_met = extension_cap & 0x2,
1119+ },
1120+ {
1121+ .name = "termination",
1122+ .test = test_termination,
1123+ .requirements_met = extension_cap > 0,
1124+ },
1125+ {
1126+ .name = "error checks with key storage protection override",
1127+ .test = test_errors_key_storage_prot_override,
1128+ .requirements_met = extension_cap > 0,
1129+ },
1130+ {
1131+ .name = "error checks without key fetch prot override",
1132+ .test = test_errors_key_fetch_prot_override_not_enabled,
1133+ .requirements_met = extension_cap > 0,
1134+ },
1135+ {
1136+ .name = "error checks with key fetch prot override",
1137+ .test = test_errors_key_fetch_prot_override_enabled,
1138+ .requirements_met = extension_cap > 0,
1139+ },
1140+ };
7641141
1142+ ksft_print_header();
7651143 ksft_set_plan(ARRAY_SIZE(testlist));
7661144
767− extension_cap = kvm_check_cap(KVM_CAP_S390_MEM_OP_EXTENSION);
7681145 for (idx = 0; idx < ARRAY_SIZE(testlist); idx++) {
769− if (extension_cap >= testlist[idx].extension) {
1146+ if (testlist[idx].requirements_met) {
7701147 testlist[idx].test();
7711148 ksft_test_result_pass("%s\n", testlist[idx].name);
7721149 } else {
773− ksft_test_result_skip("%s - extension level %d not supported\n",
774− testlist[idx].name,
775− testlist[idx].extension);
1150+ ksft_test_result_skip("%s - requirements not met (kernel has extension cap %#x)\n",
1151+ testlist[idx].name, extension_cap);
7761152 }
7771153 }
7781154
tools/testing/selftests/kvm/x86_64/exit_on_emulation_failure_test.c+0−3
@@ -26,9 +26,6 @@ int main(int argc, char *argv[]) · 25 unmodified lines
2626 struct kvm_vcpu *vcpu;
2727 struct kvm_vm *vm;
2828
29− /* Tell stdout not to buffer its content */
30− setbuf(stdout, NULL);
31−
3229 TEST_REQUIRE(kvm_has_cap(KVM_CAP_EXIT_ON_EMULATION_FAILURE));
3330
3431 vm = vm_create_with_one_vcpu(&vcpu, guest_code);
tools/testing/selftests/kvm/x86_64/fix_hypercall_test.c+2−2
@@ -48,10 +48,10 @@ static void guest_main(void) · 47 unmodified lines
4848 const uint8_t *other_hypercall_insn;
4949 uint64_t ret;
5050
51− if (is_intel_cpu()) {
51+ if (host_cpu_is_intel) {
5252 native_hypercall_insn = vmx_vmcall;
5353 other_hypercall_insn = svm_vmmcall;
54− } else if (is_amd_cpu()) {
54+ } else if (host_cpu_is_amd) {
5555 native_hypercall_insn = svm_vmmcall;
5656 other_hypercall_insn = vmx_vmcall;
5757 } else {
tools/testing/selftests/kvm/x86_64/hyperv_clock.c+1−1
@@ -104,7 +104,7 @@ static void guest_main(struct ms_hyperv_tsc_page *tsc_page, vm_paddr_t tsc_page_ · 103 unmodified lines
104104
105105 /* Set Guest OS id to enable Hyper-V emulation */
106106 GUEST_SYNC(1);
107− wrmsr(HV_X64_MSR_GUEST_OS_ID, (u64)0x8100 << 48);
107+ wrmsr(HV_X64_MSR_GUEST_OS_ID, HYPERV_LINUX_OS_ID);
108108 GUEST_SYNC(2);
109109
110110 check_tsc_msr_rdtsc();
tools/testing/selftests/kvm/x86_64/hyperv_extended_hypercalls.c+97−0
@@ -0,0 +1,97 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/*
3+ * Test Hyper-V extended hypercall, HV_EXT_CALL_QUERY_CAPABILITIES (0x8001),
4+ * exit to userspace and receive result in guest.
5+ *
6+ * Negative tests are present in hyperv_features.c
7+ *
8+ * Copyright 2022 Google LLC
9+ * Author: Vipin Sharma <vipinsh@google.com>
10+ */
11+
12+#include "kvm_util.h"
13+#include "processor.h"
14+#include "hyperv.h"
15+
16+/* Any value is fine */
17+#define EXT_CAPABILITIES 0xbull
18+
19+static void guest_code(vm_paddr_t in_pg_gpa, vm_paddr_t out_pg_gpa,
20+ vm_vaddr_t out_pg_gva)
21+{
22+ uint64_t *output_gva;
23+
24+ wrmsr(HV_X64_MSR_GUEST_OS_ID, HYPERV_LINUX_OS_ID);
25+ wrmsr(HV_X64_MSR_HYPERCALL, in_pg_gpa);
26+
27+ output_gva = (uint64_t *)out_pg_gva;
28+
29+ hyperv_hypercall(HV_EXT_CALL_QUERY_CAPABILITIES, in_pg_gpa, out_pg_gpa);
30+
31+ /* TLFS states output will be a uint64_t value */
32+ GUEST_ASSERT_EQ(*output_gva, EXT_CAPABILITIES);
33+
34+ GUEST_DONE();
35+}
36+
37+int main(void)
38+{
39+ vm_vaddr_t hcall_out_page;
40+ vm_vaddr_t hcall_in_page;
41+ struct kvm_vcpu *vcpu;
42+ struct kvm_run *run;
43+ struct kvm_vm *vm;
44+ uint64_t *outval;
45+ struct ucall uc;
46+
47+ /* Verify if extended hypercalls are supported */
48+ if (!kvm_cpuid_has(kvm_get_supported_hv_cpuid(),
49+ HV_ENABLE_EXTENDED_HYPERCALLS)) {
50+ print_skip("Extended calls not supported by the kernel");
51+ exit(KSFT_SKIP);
52+ }
53+
54+ vm = vm_create_with_one_vcpu(&vcpu, guest_code);
55+ run = vcpu->run;
56+ vcpu_set_hv_cpuid(vcpu);
57+
58+ /* Hypercall input */
59+ hcall_in_page = vm_vaddr_alloc_pages(vm, 1);
60+ memset(addr_gva2hva(vm, hcall_in_page), 0x0, vm->page_size);
61+
62+ /* Hypercall output */
63+ hcall_out_page = vm_vaddr_alloc_pages(vm, 1);
64+ memset(addr_gva2hva(vm, hcall_out_page), 0x0, vm->page_size);
65+
66+ vcpu_args_set(vcpu, 3, addr_gva2gpa(vm, hcall_in_page),
67+ addr_gva2gpa(vm, hcall_out_page), hcall_out_page);
68+
69+ vcpu_run(vcpu);
70+
71+ TEST_ASSERT(run->exit_reason == KVM_EXIT_HYPERV,
72+ "Unexpected exit reason: %u (%s)",
73+ run->exit_reason, exit_reason_str(run->exit_reason));
74+
75+ outval = addr_gpa2hva(vm, run->hyperv.u.hcall.params[1]);
76+ *outval = EXT_CAPABILITIES;
77+ run->hyperv.u.hcall.result = HV_STATUS_SUCCESS;
78+
79+ vcpu_run(vcpu);
80+
81+ TEST_ASSERT(run->exit_reason == KVM_EXIT_IO,
82+ "Unexpected exit reason: %u (%s)",
83+ run->exit_reason, exit_reason_str(run->exit_reason));
84+
85+ switch (get_ucall(vcpu, &uc)) {
86+ case UCALL_ABORT:
87+ REPORT_GUEST_ASSERT_2(uc, "arg1 = %ld, arg2 = %ld");
88+ break;
89+ case UCALL_DONE:
90+ break;
91+ default:
92+ TEST_FAIL("Unhandled ucall: %ld", uc.cmd);
93+ }
94+
95+ kvm_vm_free(vm);
96+ return 0;
97+}
tools/testing/selftests/kvm/x86_64/hyperv_features.c+207−132
@@ -13,9 +13,17 @@ · 12 unmodified lines
1313 #include "processor.h"
1414 #include "hyperv.h"
1515
16+/*
17+ * HYPERV_CPUID_ENLIGHTMENT_INFO.EBX is not a 'feature' CPUID leaf
18+ * but to activate the feature it is sufficient to set it to a non-zero
19+ * value. Use BIT(0) for that.
20+ */
21+#define HV_PV_SPINLOCKS_TEST \
22+ KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EBX, 0)
23+
1624 struct msr_data {
1725 uint32_t idx;
18− bool available;
26+ bool fault_expected;
1927 bool write;
2028 u64 write_val;
2129 };
@@ -26,22 +34,46 @@ struct hcall_data { · 4 unmodified lines
2634 bool ud_expected;
2735 };
2836
37+static bool is_write_only_msr(uint32_t msr)
38+{
39+ return msr == HV_X64_MSR_EOI;
40+}
41+
2942 static void guest_msr(struct msr_data *msr)
3043 {
31− uint64_t ignored;
32− uint8_t vector;
44+ uint8_t vector = 0;
45+ uint64_t msr_val = 0;
3346
3447 GUEST_ASSERT(msr->idx);
3548
36− if (!msr->write)
37− vector = rdmsr_safe(msr->idx, &ignored);
38− else
49+ if (msr->write)
3950 vector = wrmsr_safe(msr->idx, msr->write_val);
4051
41− if (msr->available)
42− GUEST_ASSERT_2(!vector, msr->idx, vector);
52+ if (!vector && (!msr->write || !is_write_only_msr(msr->idx)))
53+ vector = rdmsr_safe(msr->idx, &msr_val);
54+
55+ if (msr->fault_expected)
56+ GUEST_ASSERT_3(vector == GP_VECTOR, msr->idx, vector, GP_VECTOR);
4357 else
44− GUEST_ASSERT_2(vector == GP_VECTOR, msr->idx, vector);
58+ GUEST_ASSERT_3(!vector, msr->idx, vector, 0);
59+
60+ if (vector || is_write_only_msr(msr->idx))
61+ goto done;
62+
63+ if (msr->write)
64+ GUEST_ASSERT_3(msr_val == msr->write_val, msr->idx,
65+ msr_val, msr->write_val);
66+
67+ /* Invariant TSC bit appears when TSC invariant control MSR is written to */
68+ if (msr->idx == HV_X64_MSR_TSC_INVARIANT_CONTROL) {
69+ if (!this_cpu_has(HV_ACCESS_TSC_INVARIANT))
70+ GUEST_ASSERT(this_cpu_has(X86_FEATURE_INVTSC));
71+ else
72+ GUEST_ASSERT(this_cpu_has(X86_FEATURE_INVTSC) ==
73+ !!(msr_val & HV_INVARIANT_TSC_EXPOSED));
74+ }
75+
76+done:
4577 GUEST_DONE();
4678 }
4779
@@ -89,7 +121,6 @@ static void vcpu_reset_hv_cpuid(struct kvm_vcpu *vcpu) · 41 unmodified lines
89121 static void guest_test_msrs_access(void)
90122 {
91123 struct kvm_cpuid2 *prev_cpuid = NULL;
92− struct kvm_cpuid_entry2 *feat, *dbg;
93124 struct kvm_vcpu *vcpu;
94125 struct kvm_run *run;
95126 struct kvm_vm *vm;
@@ -97,6 +128,7 @@ static void guest_test_msrs_access(void) · 1 unmodified line
97128 int stage = 0;
98129 vm_vaddr_t msr_gva;
99130 struct msr_data *msr;
131+ bool has_invtsc = kvm_cpu_has(X86_FEATURE_INVTSC);
100132
101133 while (true) {
102134 vm = vm_create_with_one_vcpu(&vcpu, guest_msr);
@@ -116,9 +148,6 @@ static void guest_test_msrs_access(void) · 13 unmodified lines
116148 vcpu_init_cpuid(vcpu, prev_cpuid);
117149 }
118150
119− feat = vcpu_get_cpuid_entry(vcpu, HYPERV_CPUID_FEATURES);
120− dbg = vcpu_get_cpuid_entry(vcpu, HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES);
121−
122151 vm_init_descriptor_tables(vm);
123152 vcpu_init_descriptor_tables(vcpu);
124153
@@ -134,133 +163,139 @@ static void guest_test_msrs_access(void) · 9 unmodified lines
134163 * Only available when Hyper-V identification is set
135164 */
136165 msr->idx = HV_X64_MSR_GUEST_OS_ID;
137− msr->write = 0;
138− msr->available = 0;
166+ msr->write = false;
167+ msr->fault_expected = true;
139168 break;
140169 case 1:
141170 msr->idx = HV_X64_MSR_HYPERCALL;
142− msr->write = 0;
143− msr->available = 0;
171+ msr->write = false;
172+ msr->fault_expected = true;
144173 break;
145174 case 2:
146− feat->eax |= HV_MSR_HYPERCALL_AVAILABLE;
175+ vcpu_set_cpuid_feature(vcpu, HV_MSR_HYPERCALL_AVAILABLE);
147176 /*
148177 * HV_X64_MSR_GUEST_OS_ID has to be written first to make
149178 * HV_X64_MSR_HYPERCALL available.
150179 */
151180 msr->idx = HV_X64_MSR_GUEST_OS_ID;
152− msr->write = 1;
181+ msr->write = true;
153182 msr->write_val = HYPERV_LINUX_OS_ID;
154− msr->available = 1;
183+ msr->fault_expected = false;
155184 break;
156185 case 3:
157186 msr->idx = HV_X64_MSR_GUEST_OS_ID;
158− msr->write = 0;
159− msr->available = 1;
187+ msr->write = false;
188+ msr->fault_expected = false;
160189 break;
161190 case 4:
162191 msr->idx = HV_X64_MSR_HYPERCALL;
163− msr->write = 0;
164− msr->available = 1;
192+ msr->write = false;
193+ msr->fault_expected = false;
165194 break;
166195
167196 case 5:
168197 msr->idx = HV_X64_MSR_VP_RUNTIME;
169− msr->write = 0;
170− msr->available = 0;
198+ msr->write = false;
199+ msr->fault_expected = true;
171200 break;
172201 case 6:
173− feat->eax |= HV_MSR_VP_RUNTIME_AVAILABLE;
202+ vcpu_set_cpuid_feature(vcpu, HV_MSR_VP_RUNTIME_AVAILABLE);
174203 msr->idx = HV_X64_MSR_VP_RUNTIME;
175− msr->write = 0;
176− msr->available = 1;
204+ msr->write = false;
205+ msr->fault_expected = false;
177206 break;
178207 case 7:
179208 /* Read only */
180209 msr->idx = HV_X64_MSR_VP_RUNTIME;
181− msr->write = 1;
210+ msr->write = true;
182211 msr->write_val = 1;
183− msr->available = 0;
212+ msr->fault_expected = true;
184213 break;
185214
186215 case 8:
187216 msr->idx = HV_X64_MSR_TIME_REF_COUNT;
188− msr->write = 0;
189− msr->available = 0;
217+ msr->write = false;
218+ msr->fault_expected = true;
190219 break;
191220 case 9:
192− feat->eax |= HV_MSR_TIME_REF_COUNT_AVAILABLE;
221+ vcpu_set_cpuid_feature(vcpu, HV_MSR_TIME_REF_COUNT_AVAILABLE);
193222 msr->idx = HV_X64_MSR_TIME_REF_COUNT;
194− msr->write = 0;
195− msr->available = 1;
223+ msr->write = false;
224+ msr->fault_expected = false;
196225 break;
197226 case 10:
198227 /* Read only */
199228 msr->idx = HV_X64_MSR_TIME_REF_COUNT;
200− msr->write = 1;
229+ msr->write = true;
201230 msr->write_val = 1;
202− msr->available = 0;
231+ msr->fault_expected = true;
203232 break;
204233
205234 case 11:
206235 msr->idx = HV_X64_MSR_VP_INDEX;
207− msr->write = 0;
208− msr->available = 0;
236+ msr->write = false;
237+ msr->fault_expected = true;
209238 break;
210239 case 12:
211− feat->eax |= HV_MSR_VP_INDEX_AVAILABLE;
240+ vcpu_set_cpuid_feature(vcpu, HV_MSR_VP_INDEX_AVAILABLE);
212241 msr->idx = HV_X64_MSR_VP_INDEX;
213− msr->write = 0;
214− msr->available = 1;
242+ msr->write = false;
243+ msr->fault_expected = false;
215244 break;
216245 case 13:
217246 /* Read only */
218247 msr->idx = HV_X64_MSR_VP_INDEX;
219− msr->write = 1;
248+ msr->write = true;
220249 msr->write_val = 1;
221− msr->available = 0;
250+ msr->fault_expected = true;
222251 break;
223252
224253 case 14:
225254 msr->idx = HV_X64_MSR_RESET;
226− msr->write = 0;
227− msr->available = 0;
255+ msr->write = false;
256+ msr->fault_expected = true;
228257 break;
229258 case 15:
230− feat->eax |= HV_MSR_RESET_AVAILABLE;
259+ vcpu_set_cpuid_feature(vcpu, HV_MSR_RESET_AVAILABLE);
231260 msr->idx = HV_X64_MSR_RESET;
232− msr->write = 0;
233− msr->available = 1;
261+ msr->write = false;
262+ msr->fault_expected = false;
234263 break;
235264 case 16:
236265 msr->idx = HV_X64_MSR_RESET;
237− msr->write = 1;
266+ msr->write = true;
267+ /*
268+ * TODO: the test only writes '0' to HV_X64_MSR_RESET
269+ * at the moment, writing some other value there will
270+ * trigger real vCPU reset and the code is not prepared
271+ * to handle it yet.
272+ */
238273 msr->write_val = 0;
239− msr->available = 1;
274+ msr->fault_expected = false;
240275 break;
241276
242277 case 17:
243278 msr->idx = HV_X64_MSR_REFERENCE_TSC;
244− msr->write = 0;
245− msr->available = 0;
279+ msr->write = false;
280+ msr->fault_expected = true;
246281 break;
247282 case 18:
248− feat->eax |= HV_MSR_REFERENCE_TSC_AVAILABLE;
283+ vcpu_set_cpuid_feature(vcpu, HV_MSR_REFERENCE_TSC_AVAILABLE);
249284 msr->idx = HV_X64_MSR_REFERENCE_TSC;
250− msr->write = 0;
251− msr->available = 1;
285+ msr->write = false;
286+ msr->fault_expected = false;
252287 break;
253288 case 19:
254289 msr->idx = HV_X64_MSR_REFERENCE_TSC;
255− msr->write = 1;
290+ msr->write = true;
256291 msr->write_val = 0;
257− msr->available = 1;
292+ msr->fault_expected = false;
258293 break;
259294
260295 case 20:
261296 msr->idx = HV_X64_MSR_EOM;
262− msr->write = 0;
263− msr->available = 0;
297+ msr->write = false;
298+ msr->fault_expected = true;
264299 break;
265300 case 21:
266301 /*
@@ -268,149 +303,185 @@ static void guest_test_msrs_access(void) · 1 unmodified line
268303 * capability enabled and guest visible CPUID bit unset.
269304 */
270305 msr->idx = HV_X64_MSR_EOM;
271− msr->write = 0;
272− msr->available = 0;
306+ msr->write = false;
307+ msr->fault_expected = true;
273308 break;
274309 case 22:
275− feat->eax |= HV_MSR_SYNIC_AVAILABLE;
310+ vcpu_set_cpuid_feature(vcpu, HV_MSR_SYNIC_AVAILABLE);
276311 msr->idx = HV_X64_MSR_EOM;
277− msr->write = 0;
278− msr->available = 1;
312+ msr->write = false;
313+ msr->fault_expected = false;
279314 break;
280315 case 23:
281316 msr->idx = HV_X64_MSR_EOM;
282− msr->write = 1;
317+ msr->write = true;
283318 msr->write_val = 0;
284− msr->available = 1;
319+ msr->fault_expected = false;
285320 break;
286321
287322 case 24:
288323 msr->idx = HV_X64_MSR_STIMER0_CONFIG;
289− msr->write = 0;
290− msr->available = 0;
324+ msr->write = false;
325+ msr->fault_expected = true;
291326 break;
292327 case 25:
293− feat->eax |= HV_MSR_SYNTIMER_AVAILABLE;
328+ vcpu_set_cpuid_feature(vcpu, HV_MSR_SYNTIMER_AVAILABLE);
294329 msr->idx = HV_X64_MSR_STIMER0_CONFIG;
295− msr->write = 0;
296− msr->available = 1;
330+ msr->write = false;
331+ msr->fault_expected = false;
297332 break;
298333 case 26:
299334 msr->idx = HV_X64_MSR_STIMER0_CONFIG;
300− msr->write = 1;
335+ msr->write = true;
301336 msr->write_val = 0;
302− msr->available = 1;
337+ msr->fault_expected = false;
303338 break;
304339 case 27:
305340 /* Direct mode test */
306341 msr->idx = HV_X64_MSR_STIMER0_CONFIG;
307− msr->write = 1;
342+ msr->write = true;
308343 msr->write_val = 1 << 12;
309− msr->available = 0;
344+ msr->fault_expected = true;
310345 break;
311346 case 28:
312− feat->edx |= HV_STIMER_DIRECT_MODE_AVAILABLE;
347+ vcpu_set_cpuid_feature(vcpu, HV_STIMER_DIRECT_MODE_AVAILABLE);
313348 msr->idx = HV_X64_MSR_STIMER0_CONFIG;
314− msr->write = 1;
349+ msr->write = true;
315350 msr->write_val = 1 << 12;
316− msr->available = 1;
351+ msr->fault_expected = false;
317352 break;
318353
319354 case 29:
320355 msr->idx = HV_X64_MSR_EOI;
321− msr->write = 0;
322− msr->available = 0;
356+ msr->write = false;
357+ msr->fault_expected = true;
323358 break;
324359 case 30:
325− feat->eax |= HV_MSR_APIC_ACCESS_AVAILABLE;
360+ vcpu_set_cpuid_feature(vcpu, HV_MSR_APIC_ACCESS_AVAILABLE);
326361 msr->idx = HV_X64_MSR_EOI;
327− msr->write = 1;
362+ msr->write = true;
328363 msr->write_val = 1;
329− msr->available = 1;
364+ msr->fault_expected = false;
330365 break;
331366
332367 case 31:
333368 msr->idx = HV_X64_MSR_TSC_FREQUENCY;
334− msr->write = 0;
335− msr->available = 0;
369+ msr->write = false;
370+ msr->fault_expected = true;
336371 break;
337372 case 32:
338− feat->eax |= HV_ACCESS_FREQUENCY_MSRS;
373+ vcpu_set_cpuid_feature(vcpu, HV_ACCESS_FREQUENCY_MSRS);
339374 msr->idx = HV_X64_MSR_TSC_FREQUENCY;
340− msr->write = 0;
341− msr->available = 1;
375+ msr->write = false;
376+ msr->fault_expected = false;
342377 break;
343378 case 33:
344379 /* Read only */
345380 msr->idx = HV_X64_MSR_TSC_FREQUENCY;
346− msr->write = 1;
381+ msr->write = true;
347382 msr->write_val = 1;
348− msr->available = 0;
383+ msr->fault_expected = true;
349384 break;
350385
351386 case 34:
352387 msr->idx = HV_X64_MSR_REENLIGHTENMENT_CONTROL;
353− msr->write = 0;
354− msr->available = 0;
388+ msr->write = false;
389+ msr->fault_expected = true;
355390 break;
356391 case 35:
357− feat->eax |= HV_ACCESS_REENLIGHTENMENT;
392+ vcpu_set_cpuid_feature(vcpu, HV_ACCESS_REENLIGHTENMENT);
358393 msr->idx = HV_X64_MSR_REENLIGHTENMENT_CONTROL;
359− msr->write = 0;
360− msr->available = 1;
394+ msr->write = false;
395+ msr->fault_expected = false;
361396 break;
362397 case 36:
363398 msr->idx = HV_X64_MSR_REENLIGHTENMENT_CONTROL;
364− msr->write = 1;
399+ msr->write = true;
365400 msr->write_val = 1;
366− msr->available = 1;
401+ msr->fault_expected = false;
367402 break;
368403 case 37:
369404 /* Can only write '0' */
370405 msr->idx = HV_X64_MSR_TSC_EMULATION_STATUS;
371− msr->write = 1;
406+ msr->write = true;
372407 msr->write_val = 1;
373− msr->available = 0;
408+ msr->fault_expected = true;
374409 break;
375410
376411 case 38:
377412 msr->idx = HV_X64_MSR_CRASH_P0;
378− msr->write = 0;
379− msr->available = 0;
413+ msr->write = false;
414+ msr->fault_expected = true;
380415 break;
381416 case 39:
382− feat->edx |= HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE;
417+ vcpu_set_cpuid_feature(vcpu, HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE);
383418 msr->idx = HV_X64_MSR_CRASH_P0;
384− msr->write = 0;
385− msr->available = 1;
419+ msr->write = false;
420+ msr->fault_expected = false;
386421 break;
387422 case 40:
388423 msr->idx = HV_X64_MSR_CRASH_P0;
389− msr->write = 1;
424+ msr->write = true;
390425 msr->write_val = 1;
391− msr->available = 1;
426+ msr->fault_expected = false;
392427 break;
393428
394429 case 41:
395430 msr->idx = HV_X64_MSR_SYNDBG_STATUS;
396− msr->write = 0;
397− msr->available = 0;
431+ msr->write = false;
432+ msr->fault_expected = true;
398433 break;
399434 case 42:
400− feat->edx |= HV_FEATURE_DEBUG_MSRS_AVAILABLE;
401− dbg->eax |= HV_X64_SYNDBG_CAP_ALLOW_KERNEL_DEBUGGING;
435+ vcpu_set_cpuid_feature(vcpu, HV_FEATURE_DEBUG_MSRS_AVAILABLE);
436+ vcpu_set_cpuid_feature(vcpu, HV_X64_SYNDBG_CAP_ALLOW_KERNEL_DEBUGGING);
402437 msr->idx = HV_X64_MSR_SYNDBG_STATUS;
403− msr->write = 0;
404− msr->available = 1;
438+ msr->write = false;
439+ msr->fault_expected = false;
405440 break;
406441 case 43:
407442 msr->idx = HV_X64_MSR_SYNDBG_STATUS;
408− msr->write = 1;
443+ msr->write = true;
409444 msr->write_val = 0;
410− msr->available = 1;
445+ msr->fault_expected = false;
411446 break;
412447
413448 case 44:
449+ /* MSR is not available when CPUID feature bit is unset */
450+ if (!has_invtsc)
451+ continue;
452+ msr->idx = HV_X64_MSR_TSC_INVARIANT_CONTROL;
453+ msr->write = false;
454+ msr->fault_expected = true;
455+ break;
456+ case 45:
457+ /* MSR is vailable when CPUID feature bit is set */
458+ if (!has_invtsc)
459+ continue;
460+ vcpu_set_cpuid_feature(vcpu, HV_ACCESS_TSC_INVARIANT);
461+ msr->idx = HV_X64_MSR_TSC_INVARIANT_CONTROL;
462+ msr->write = false;
463+ msr->fault_expected = false;
464+ break;
465+ case 46:
466+ /* Writing bits other than 0 is forbidden */
467+ if (!has_invtsc)
468+ continue;
469+ msr->idx = HV_X64_MSR_TSC_INVARIANT_CONTROL;
470+ msr->write = true;
471+ msr->write_val = 0xdeadbeef;
472+ msr->fault_expected = true;
473+ break;
474+ case 47:
475+ /* Setting bit 0 enables the feature */
476+ if (!has_invtsc)
477+ continue;
478+ msr->idx = HV_X64_MSR_TSC_INVARIANT_CONTROL;
479+ msr->write = true;
480+ msr->write_val = 1;
481+ msr->fault_expected = false;
482+ break;
483+
484+ default:
414485 kvm_vm_free(vm);
415486 return;
416487 }
@@ -429,7 +500,7 @@ static void guest_test_msrs_access(void) · 12 unmodified lines
429500
430501 switch (get_ucall(vcpu, &uc)) {
431502 case UCALL_ABORT:
432− REPORT_GUEST_ASSERT_2(uc, "MSR = %lx, vector = %lx");
503+ REPORT_GUEST_ASSERT_3(uc, "MSR = %lx, arg1 = %lx, arg2 = %lx");
433504 return;
434505 case UCALL_DONE:
435506 break;
@@ -445,7 +516,6 @@ static void guest_test_msrs_access(void) · 9 unmodified lines
445516
446517 static void guest_test_hcalls_access(void)
447518 {
448− struct kvm_cpuid_entry2 *feat, *recomm, *dbg;
449519 struct kvm_cpuid2 *prev_cpuid = NULL;
450520 struct kvm_vcpu *vcpu;
451521 struct kvm_run *run;
@@ -480,15 +550,11 @@ static void guest_test_hcalls_access(void) · 28 unmodified lines
480550 vcpu_init_cpuid(vcpu, prev_cpuid);
481551 }
482552
483− feat = vcpu_get_cpuid_entry(vcpu, HYPERV_CPUID_FEATURES);
484− recomm = vcpu_get_cpuid_entry(vcpu, HYPERV_CPUID_ENLIGHTMENT_INFO);
485− dbg = vcpu_get_cpuid_entry(vcpu, HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES);
486−
487553 run = vcpu->run;
488554
489555 switch (stage) {
490556 case 0:
491− feat->eax |= HV_MSR_HYPERCALL_AVAILABLE;
557+ vcpu_set_cpuid_feature(vcpu, HV_MSR_HYPERCALL_AVAILABLE);
492558 hcall->control = 0xbeef;
493559 hcall->expect = HV_STATUS_INVALID_HYPERCALL_CODE;
494560 break;
@@ -498,7 +564,7 @@ static void guest_test_hcalls_access(void) · 3 unmodified lines
498564 hcall->expect = HV_STATUS_ACCESS_DENIED;
499565 break;
500566 case 2:
501− feat->ebx |= HV_POST_MESSAGES;
567+ vcpu_set_cpuid_feature(vcpu, HV_POST_MESSAGES);
502568 hcall->control = HVCALL_POST_MESSAGE;
503569 hcall->expect = HV_STATUS_INVALID_HYPERCALL_INPUT;
504570 break;
@@ -508,7 +574,7 @@ static void guest_test_hcalls_access(void) · 3 unmodified lines
508574 hcall->expect = HV_STATUS_ACCESS_DENIED;
509575 break;
510576 case 4:
511− feat->ebx |= HV_SIGNAL_EVENTS;
577+ vcpu_set_cpuid_feature(vcpu, HV_SIGNAL_EVENTS);
512578 hcall->control = HVCALL_SIGNAL_EVENT;
513579 hcall->expect = HV_STATUS_INVALID_HYPERCALL_INPUT;
514580 break;
@@ -518,12 +584,12 @@ static void guest_test_hcalls_access(void) · 3 unmodified lines
518584 hcall->expect = HV_STATUS_INVALID_HYPERCALL_CODE;
519585 break;
520586 case 6:
521− dbg->eax |= HV_X64_SYNDBG_CAP_ALLOW_KERNEL_DEBUGGING;
587+ vcpu_set_cpuid_feature(vcpu, HV_X64_SYNDBG_CAP_ALLOW_KERNEL_DEBUGGING);
522588 hcall->control = HVCALL_RESET_DEBUG_SESSION;
523589 hcall->expect = HV_STATUS_ACCESS_DENIED;
524590 break;
525591 case 7:
526− feat->ebx |= HV_DEBUGGING;
592+ vcpu_set_cpuid_feature(vcpu, HV_DEBUGGING);
527593 hcall->control = HVCALL_RESET_DEBUG_SESSION;
528594 hcall->expect = HV_STATUS_OPERATION_DENIED;
529595 break;
@@ -533,7 +599,7 @@ static void guest_test_hcalls_access(void) · 3 unmodified lines
533599 hcall->expect = HV_STATUS_ACCESS_DENIED;
534600 break;
535601 case 9:
536− recomm->eax |= HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED;
602+ vcpu_set_cpuid_feature(vcpu, HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED);
537603 hcall->control = HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE;
538604 hcall->expect = HV_STATUS_SUCCESS;
539605 break;
@@ -542,7 +608,7 @@ static void guest_test_hcalls_access(void) · 2 unmodified lines
542608 hcall->expect = HV_STATUS_ACCESS_DENIED;
543609 break;
544610 case 11:
545− recomm->eax |= HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED;
611+ vcpu_set_cpuid_feature(vcpu, HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED);
546612 hcall->control = HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX;
547613 hcall->expect = HV_STATUS_SUCCESS;
548614 break;
@@ -552,7 +618,7 @@ static void guest_test_hcalls_access(void) · 3 unmodified lines
552618 hcall->expect = HV_STATUS_ACCESS_DENIED;
553619 break;
554620 case 13:
555− recomm->eax |= HV_X64_CLUSTER_IPI_RECOMMENDED;
621+ vcpu_set_cpuid_feature(vcpu, HV_X64_CLUSTER_IPI_RECOMMENDED);
556622 hcall->control = HVCALL_SEND_IPI;
557623 hcall->expect = HV_STATUS_INVALID_HYPERCALL_INPUT;
558624 break;
@@ -567,7 +633,7 @@ static void guest_test_hcalls_access(void) · 8 unmodified lines
567633 hcall->expect = HV_STATUS_ACCESS_DENIED;
568634 break;
569635 case 16:
570− recomm->ebx = 0xfff;
636+ vcpu_set_cpuid_feature(vcpu, HV_PV_SPINLOCKS_TEST);
571637 hcall->control = HVCALL_NOTIFY_LONG_SPIN_WAIT;
572638 hcall->expect = HV_STATUS_SUCCESS;
573639 break;
@@ -577,12 +643,21 @@ static void guest_test_hcalls_access(void) · 3 unmodified lines
577643 hcall->ud_expected = true;
578644 break;
579645 case 18:
580− feat->edx |= HV_X64_HYPERCALL_XMM_INPUT_AVAILABLE;
646+ vcpu_set_cpuid_feature(vcpu, HV_X64_HYPERCALL_XMM_INPUT_AVAILABLE);
581647 hcall->control = HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE | HV_HYPERCALL_FAST_BIT;
582648 hcall->ud_expected = false;
583649 hcall->expect = HV_STATUS_SUCCESS;
584650 break;
585651 case 19:
652+ hcall->control = HV_EXT_CALL_QUERY_CAPABILITIES;
653+ hcall->expect = HV_STATUS_ACCESS_DENIED;
654+ break;
655+ case 20:
656+ vcpu_set_cpuid_feature(vcpu, HV_ENABLE_EXTENDED_HYPERCALLS);
657+ hcall->control = HV_EXT_CALL_QUERY_CAPABILITIES | HV_HYPERCALL_FAST_BIT;
658+ hcall->expect = HV_STATUS_INVALID_PARAMETER;
659+ break;
660+ case 21:
586661 kvm_vm_free(vm);
587662 return;
588663 }
tools/testing/selftests/kvm/x86_64/mmio_warning_test.c+1−1
@@ -93,7 +93,7 @@ int main(void) · 92 unmodified lines
9393 {
9494 int warnings_before, warnings_after;
9595
96− TEST_REQUIRE(is_intel_cpu());
96+ TEST_REQUIRE(host_cpu_is_intel);
9797
9898 TEST_REQUIRE(!vm_is_unrestricted_guest(NULL));
9999
tools/testing/selftests/kvm/x86_64/pmu_event_filter_test.c+378−7
@@ -198,14 +198,15 @@ static struct kvm_pmu_event_filter *alloc_pmu_event_filter(uint32_t nevents) · 197 unmodified lines
198198
199199
200200 static struct kvm_pmu_event_filter *
201−create_pmu_event_filter(const uint64_t event_list[],
202− int nevents, uint32_t action)
201+create_pmu_event_filter(const uint64_t event_list[], int nevents,
202+ uint32_t action, uint32_t flags)
203203 {
204204 struct kvm_pmu_event_filter *f;
205205 int i;
206206
207207 f = alloc_pmu_event_filter(nevents);
208208 f->action = action;
209+ f->flags = flags;
209210 for (i = 0; i < nevents; i++)
210211 f->events[i] = event_list[i];
211212
@@ -216,7 +217,7 @@ static struct kvm_pmu_event_filter *event_filter(uint32_t action) · 4 unmodified lines
216217 {
217218 return create_pmu_event_filter(event_list,
218219 ARRAY_SIZE(event_list),
219− action);
220+ action, 0);
220221 }
221222
222223 /*
@@ -263,7 +264,7 @@ static void test_amd_deny_list(struct kvm_vcpu *vcpu) · 40 unmodified lines
263264 struct kvm_pmu_event_filter *f;
264265 uint64_t count;
265266
266− f = create_pmu_event_filter(&event, 1, KVM_PMU_EVENT_DENY);
267+ f = create_pmu_event_filter(&event, 1, KVM_PMU_EVENT_DENY, 0);
267268 count = test_with_filter(vcpu, f);
268269
269270 free(f);
@@ -363,7 +364,7 @@ static void test_pmu_config_disable(void (*guest_code)(void)) · 93 unmodified lines
363364 */
364365 static bool use_intel_pmu(void)
365366 {
366− return is_intel_cpu() &&
367+ return host_cpu_is_intel &&
367368 kvm_cpu_property(X86_PROPERTY_PMU_VERSION) &&
368369 kvm_cpu_property(X86_PROPERTY_PMU_NR_GP_COUNTERS) &&
369370 kvm_pmu_has(X86_PMU_FEATURE_BRANCH_INSNS_RETIRED);
@@ -397,19 +398,378 @@ static bool use_amd_pmu(void) · 27 unmodified lines
397398 uint32_t family = kvm_cpu_family();
398399 uint32_t model = kvm_cpu_model();
399400
400− return is_amd_cpu() &&
401+ return host_cpu_is_amd &&
401402 (is_zen1(family, model) ||
402403 is_zen2(family, model) ||
403404 is_zen3(family, model));
404405 }
405406
407+/*
408+ * "MEM_INST_RETIRED.ALL_LOADS", "MEM_INST_RETIRED.ALL_STORES", and
409+ * "MEM_INST_RETIRED.ANY" from https://perfmon-events.intel.com/
410+ * supported on Intel Xeon processors:
411+ * - Sapphire Rapids, Ice Lake, Cascade Lake, Skylake.
412+ */
413+#define MEM_INST_RETIRED 0xD0
414+#define MEM_INST_RETIRED_LOAD EVENT(MEM_INST_RETIRED, 0x81)
415+#define MEM_INST_RETIRED_STORE EVENT(MEM_INST_RETIRED, 0x82)
416+#define MEM_INST_RETIRED_LOAD_STORE EVENT(MEM_INST_RETIRED, 0x83)
417+
418+static bool supports_event_mem_inst_retired(void)
419+{
420+ uint32_t eax, ebx, ecx, edx;
421+
422+ cpuid(1, &eax, &ebx, &ecx, &edx);
423+ if (x86_family(eax) == 0x6) {
424+ switch (x86_model(eax)) {
425+ /* Sapphire Rapids */
426+ case 0x8F:
427+ /* Ice Lake */
428+ case 0x6A:
429+ /* Skylake */
430+ /* Cascade Lake */
431+ case 0x55:
432+ return true;
433+ }
434+ }
435+
436+ return false;
437+}
438+
439+/*
440+ * "LS Dispatch", from Processor Programming Reference
441+ * (PPR) for AMD Family 17h Model 01h, Revision B1 Processors,
442+ * Preliminary Processor Programming Reference (PPR) for AMD Family
443+ * 17h Model 31h, Revision B0 Processors, and Preliminary Processor
444+ * Programming Reference (PPR) for AMD Family 19h Model 01h, Revision
445+ * B1 Processors Volume 1 of 2.
446+ */
447+#define LS_DISPATCH 0x29
448+#define LS_DISPATCH_LOAD EVENT(LS_DISPATCH, BIT(0))
449+#define LS_DISPATCH_STORE EVENT(LS_DISPATCH, BIT(1))
450+#define LS_DISPATCH_LOAD_STORE EVENT(LS_DISPATCH, BIT(2))
451+
452+#define INCLUDE_MASKED_ENTRY(event_select, mask, match) \
453+ KVM_PMU_ENCODE_MASKED_ENTRY(event_select, mask, match, false)
454+#define EXCLUDE_MASKED_ENTRY(event_select, mask, match) \
455+ KVM_PMU_ENCODE_MASKED_ENTRY(event_select, mask, match, true)
456+
457+struct perf_counter {
458+ union {
459+ uint64_t raw;
460+ struct {
461+ uint64_t loads:22;
462+ uint64_t stores:22;
463+ uint64_t loads_stores:20;
464+ };
465+ };
466+};
467+
468+static uint64_t masked_events_guest_test(uint32_t msr_base)
469+{
470+ uint64_t ld0, ld1, st0, st1, ls0, ls1;
471+ struct perf_counter c;
472+ int val;
473+
474+ /*
475+ * The acutal value of the counters don't determine the outcome of
476+ * the test. Only that they are zero or non-zero.
477+ */
478+ ld0 = rdmsr(msr_base + 0);
479+ st0 = rdmsr(msr_base + 1);
480+ ls0 = rdmsr(msr_base + 2);
481+
482+ __asm__ __volatile__("movl $0, %[v];"
483+ "movl %[v], %%eax;"
484+ "incl %[v];"
485+ : [v]"+m"(val) :: "eax");
486+
487+ ld1 = rdmsr(msr_base + 0);
488+ st1 = rdmsr(msr_base + 1);
489+ ls1 = rdmsr(msr_base + 2);
490+
491+ c.loads = ld1 - ld0;
492+ c.stores = st1 - st0;
493+ c.loads_stores = ls1 - ls0;
494+
495+ return c.raw;
496+}
497+
498+static void intel_masked_events_guest_code(void)
499+{
500+ uint64_t r;
501+
502+ for (;;) {
503+ wrmsr(MSR_CORE_PERF_GLOBAL_CTRL, 0);
504+
505+ wrmsr(MSR_P6_EVNTSEL0 + 0, ARCH_PERFMON_EVENTSEL_ENABLE |
506+ ARCH_PERFMON_EVENTSEL_OS | MEM_INST_RETIRED_LOAD);
507+ wrmsr(MSR_P6_EVNTSEL0 + 1, ARCH_PERFMON_EVENTSEL_ENABLE |
508+ ARCH_PERFMON_EVENTSEL_OS | MEM_INST_RETIRED_STORE);
509+ wrmsr(MSR_P6_EVNTSEL0 + 2, ARCH_PERFMON_EVENTSEL_ENABLE |
510+ ARCH_PERFMON_EVENTSEL_OS | MEM_INST_RETIRED_LOAD_STORE);
511+
512+ wrmsr(MSR_CORE_PERF_GLOBAL_CTRL, 0x7);
513+
514+ r = masked_events_guest_test(MSR_IA32_PMC0);
515+
516+ GUEST_SYNC(r);
517+ }
518+}
519+
520+static void amd_masked_events_guest_code(void)
521+{
522+ uint64_t r;
523+
524+ for (;;) {
525+ wrmsr(MSR_K7_EVNTSEL0, 0);
526+ wrmsr(MSR_K7_EVNTSEL1, 0);
527+ wrmsr(MSR_K7_EVNTSEL2, 0);
528+
529+ wrmsr(MSR_K7_EVNTSEL0, ARCH_PERFMON_EVENTSEL_ENABLE |
530+ ARCH_PERFMON_EVENTSEL_OS | LS_DISPATCH_LOAD);
531+ wrmsr(MSR_K7_EVNTSEL1, ARCH_PERFMON_EVENTSEL_ENABLE |
532+ ARCH_PERFMON_EVENTSEL_OS | LS_DISPATCH_STORE);
533+ wrmsr(MSR_K7_EVNTSEL2, ARCH_PERFMON_EVENTSEL_ENABLE |
534+ ARCH_PERFMON_EVENTSEL_OS | LS_DISPATCH_LOAD_STORE);
535+
536+ r = masked_events_guest_test(MSR_K7_PERFCTR0);
537+
538+ GUEST_SYNC(r);
539+ }
540+}
541+
542+static struct perf_counter run_masked_events_test(struct kvm_vcpu *vcpu,
543+ const uint64_t masked_events[],
544+ const int nmasked_events)
545+{
546+ struct kvm_pmu_event_filter *f;
547+ struct perf_counter r;
548+
549+ f = create_pmu_event_filter(masked_events, nmasked_events,
550+ KVM_PMU_EVENT_ALLOW,
551+ KVM_PMU_EVENT_FLAG_MASKED_EVENTS);
552+ r.raw = test_with_filter(vcpu, f);
553+ free(f);
554+
555+ return r;
556+}
557+
558+/* Matches KVM_PMU_EVENT_FILTER_MAX_EVENTS in pmu.c */
559+#define MAX_FILTER_EVENTS 300
560+#define MAX_TEST_EVENTS 10
561+
562+#define ALLOW_LOADS BIT(0)
563+#define ALLOW_STORES BIT(1)
564+#define ALLOW_LOADS_STORES BIT(2)
565+
566+struct masked_events_test {
567+ uint64_t intel_events[MAX_TEST_EVENTS];
568+ uint64_t intel_event_end;
569+ uint64_t amd_events[MAX_TEST_EVENTS];
570+ uint64_t amd_event_end;
571+ const char *msg;
572+ uint32_t flags;
573+};
574+
575+/*
576+ * These are the test cases for the masked events tests.
577+ *
578+ * For each test, the guest enables 3 PMU counters (loads, stores,
579+ * loads + stores). The filter is then set in KVM with the masked events
580+ * provided. The test then verifies that the counters agree with which
581+ * ones should be counting and which ones should be filtered.
582+ */
583+const struct masked_events_test test_cases[] = {
584+ {
585+ .intel_events = {
586+ INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0xFF, 0x81),
587+ },
588+ .amd_events = {
589+ INCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xFF, BIT(0)),
590+ },
591+ .msg = "Only allow loads.",
592+ .flags = ALLOW_LOADS,
593+ }, {
594+ .intel_events = {
595+ INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0xFF, 0x82),
596+ },
597+ .amd_events = {
598+ INCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xFF, BIT(1)),
599+ },
600+ .msg = "Only allow stores.",
601+ .flags = ALLOW_STORES,
602+ }, {
603+ .intel_events = {
604+ INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0xFF, 0x83),
605+ },
606+ .amd_events = {
607+ INCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xFF, BIT(2)),
608+ },
609+ .msg = "Only allow loads + stores.",
610+ .flags = ALLOW_LOADS_STORES,
611+ }, {
612+ .intel_events = {
613+ INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0x7C, 0),
614+ EXCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0xFF, 0x83),
615+ },
616+ .amd_events = {
617+ INCLUDE_MASKED_ENTRY(LS_DISPATCH, ~(BIT(0) | BIT(1)), 0),
618+ },
619+ .msg = "Only allow loads and stores.",
620+ .flags = ALLOW_LOADS | ALLOW_STORES,
621+ }, {
622+ .intel_events = {
623+ INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0x7C, 0),
624+ EXCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0xFF, 0x82),
625+ },
626+ .amd_events = {
627+ INCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xF8, 0),
628+ EXCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xFF, BIT(1)),
629+ },
630+ .msg = "Only allow loads and loads + stores.",
631+ .flags = ALLOW_LOADS | ALLOW_LOADS_STORES
632+ }, {
633+ .intel_events = {
634+ INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0xFE, 0x82),
635+ },
636+ .amd_events = {
637+ INCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xF8, 0),
638+ EXCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xFF, BIT(0)),
639+ },
640+ .msg = "Only allow stores and loads + stores.",
641+ .flags = ALLOW_STORES | ALLOW_LOADS_STORES
642+ }, {
643+ .intel_events = {
644+ INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0x7C, 0),
645+ },
646+ .amd_events = {
647+ INCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xF8, 0),
648+ },
649+ .msg = "Only allow loads, stores, and loads + stores.",
650+ .flags = ALLOW_LOADS | ALLOW_STORES | ALLOW_LOADS_STORES
651+ },
652+};
653+
654+static int append_test_events(const struct masked_events_test *test,
655+ uint64_t *events, int nevents)
656+{
657+ const uint64_t *evts;
658+ int i;
659+
660+ evts = use_intel_pmu() ? test->intel_events : test->amd_events;
661+ for (i = 0; i < MAX_TEST_EVENTS; i++) {
662+ if (evts[i] == 0)
663+ break;
664+
665+ events[nevents + i] = evts[i];
666+ }
667+
668+ return nevents + i;
669+}
670+
671+static bool bool_eq(bool a, bool b)
672+{
673+ return a == b;
674+}
675+
676+static void run_masked_events_tests(struct kvm_vcpu *vcpu, uint64_t *events,
677+ int nevents)
678+{
679+ int ntests = ARRAY_SIZE(test_cases);
680+ struct perf_counter c;
681+ int i, n;
682+
683+ for (i = 0; i < ntests; i++) {
684+ const struct masked_events_test *test = &test_cases[i];
685+
686+ /* Do any test case events overflow MAX_TEST_EVENTS? */
687+ assert(test->intel_event_end == 0);
688+ assert(test->amd_event_end == 0);
689+
690+ n = append_test_events(test, events, nevents);
691+
692+ c = run_masked_events_test(vcpu, events, n);
693+ TEST_ASSERT(bool_eq(c.loads, test->flags & ALLOW_LOADS) &&
694+ bool_eq(c.stores, test->flags & ALLOW_STORES) &&
695+ bool_eq(c.loads_stores,
696+ test->flags & ALLOW_LOADS_STORES),
697+ "%s loads: %u, stores: %u, loads + stores: %u",
698+ test->msg, c.loads, c.stores, c.loads_stores);
699+ }
700+}
701+
702+static void add_dummy_events(uint64_t *events, int nevents)
703+{
704+ int i;
705+
706+ for (i = 0; i < nevents; i++) {
707+ int event_select = i % 0xFF;
708+ bool exclude = ((i % 4) == 0);
709+
710+ if (event_select == MEM_INST_RETIRED ||
711+ event_select == LS_DISPATCH)
712+ event_select++;
713+
714+ events[i] = KVM_PMU_ENCODE_MASKED_ENTRY(event_select, 0,
715+ 0, exclude);
716+ }
717+}
718+
719+static void test_masked_events(struct kvm_vcpu *vcpu)
720+{
721+ int nevents = MAX_FILTER_EVENTS - MAX_TEST_EVENTS;
722+ uint64_t events[MAX_FILTER_EVENTS];
723+
724+ /* Run the test cases against a sparse PMU event filter. */
725+ run_masked_events_tests(vcpu, events, 0);
726+
727+ /* Run the test cases against a dense PMU event filter. */
728+ add_dummy_events(events, MAX_FILTER_EVENTS);
729+ run_masked_events_tests(vcpu, events, nevents);
730+}
731+
732+static int run_filter_test(struct kvm_vcpu *vcpu, const uint64_t *events,
733+ int nevents, uint32_t flags)
734+{
735+ struct kvm_pmu_event_filter *f;
736+ int r;
737+
738+ f = create_pmu_event_filter(events, nevents, KVM_PMU_EVENT_ALLOW, flags);
739+ r = __vm_ioctl(vcpu->vm, KVM_SET_PMU_EVENT_FILTER, f);
740+ free(f);
741+
742+ return r;
743+}
744+
745+static void test_filter_ioctl(struct kvm_vcpu *vcpu)
746+{
747+ uint64_t e = ~0ul;
748+ int r;
749+
750+ /*
751+ * Unfortunately having invalid bits set in event data is expected to
752+ * pass when flags == 0 (bits other than eventsel+umask).
753+ */
754+ r = run_filter_test(vcpu, &e, 1, 0);
755+ TEST_ASSERT(r == 0, "Valid PMU Event Filter is failing");
756+
757+ r = run_filter_test(vcpu, &e, 1, KVM_PMU_EVENT_FLAG_MASKED_EVENTS);
758+ TEST_ASSERT(r != 0, "Invalid PMU Event Filter is expected to fail");
759+
760+ e = KVM_PMU_ENCODE_MASKED_ENTRY(0xff, 0xff, 0xff, 0xf);
761+ r = run_filter_test(vcpu, &e, 1, KVM_PMU_EVENT_FLAG_MASKED_EVENTS);
762+ TEST_ASSERT(r == 0, "Valid PMU Event Filter is failing");
763+}
764+
406765 int main(int argc, char *argv[])
407766 {
408767 void (*guest_code)(void);
409− struct kvm_vcpu *vcpu;
768+ struct kvm_vcpu *vcpu, *vcpu2 = NULL;
410769 struct kvm_vm *vm;
411770
412771 TEST_REQUIRE(kvm_has_cap(KVM_CAP_PMU_EVENT_FILTER));
772+ TEST_REQUIRE(kvm_has_cap(KVM_CAP_PMU_EVENT_MASKED_EVENTS));
413773
414774 TEST_REQUIRE(use_intel_pmu() || use_amd_pmu());
415775 guest_code = use_intel_pmu() ? intel_guest_code : amd_guest_code;
@@ -430,6 +790,17 @@ int main(int argc, char *argv[]) · 14 unmodified lines
430790 test_not_member_deny_list(vcpu);
431791 test_not_member_allow_list(vcpu);
432792
793+ if (use_intel_pmu() &&
794+ supports_event_mem_inst_retired() &&
795+ kvm_cpu_property(X86_PROPERTY_PMU_NR_GP_COUNTERS) >= 3)
796+ vcpu2 = vm_vcpu_add(vm, 2, intel_masked_events_guest_code);
797+ else if (use_amd_pmu())
798+ vcpu2 = vm_vcpu_add(vm, 2, amd_masked_events_guest_code);
799+
800+ if (vcpu2)
801+ test_masked_events(vcpu2);
802+ test_filter_ioctl(vcpu);
803+
433804 kvm_vm_free(vm);
434805
435806 test_pmu_config_disable(guest_code);
tools/testing/selftests/kvm/x86_64/tsc_msrs_test.c+12−4
@@ -72,11 +72,16 @@ static void run_vcpu(struct kvm_vcpu *vcpu, int stage) · 71 unmodified lines
7272
7373 switch (get_ucall(vcpu, &uc)) {
7474 case UCALL_SYNC:
75− TEST_ASSERT(!strcmp((const char *)uc.args[0], "hello") &&
76− uc.args[1] == stage + 1, "Stage %d: Unexpected register values vmexit, got %lx",
77− stage + 1, (ulong)uc.args[1]);
75+ if (!strcmp((const char *)uc.args[0], "hello") &&
76+ uc.args[1] == stage + 1)
77+ ksft_test_result_pass("stage %d passed\n", stage + 1);
78+ else
79+ ksft_test_result_fail(
80+ "stage %d: Unexpected register values vmexit, got %lx",
81+ stage + 1, (ulong)uc.args[1]);
7882 return;
7983 case UCALL_DONE:
84+ ksft_test_result_pass("stage %d passed\n", stage + 1);
8085 return;
8186 case UCALL_ABORT:
8287 REPORT_GUEST_ASSERT_2(uc, "values: %#lx, %#lx");
@@ -92,6 +97,9 @@ int main(void) · 9 unmodified lines
9297 struct kvm_vm *vm;
9398 uint64_t val;
9499
100+ ksft_print_header();
101+ ksft_set_plan(5);
102+
95103 vm = vm_create_with_one_vcpu(&vcpu, guest_code);
96104
97105 val = 0;
@@ -149,5 +157,5 @@ int main(void) · 51 unmodified lines
149157
150158 kvm_vm_free(vm);
151159
152− return 0;
160+ ksft_finished(); /* Print results and exit() accordingly */
153161 }
tools/testing/selftests/kvm/x86_64/vmx_exception_with_invalid_guest_state.c+1−1
@@ -111,7 +111,7 @@ int main(int argc, char *argv[]) · 110 unmodified lines
111111 struct kvm_vcpu *vcpu;
112112 struct kvm_vm *vm;
113113
114− TEST_REQUIRE(is_intel_cpu());
114+ TEST_REQUIRE(host_cpu_is_intel);
115115 TEST_REQUIRE(!vm_is_unrestricted_guest(NULL));
116116
117117 vm = vm_create_with_one_vcpu(&vcpu, guest_code);
tools/testing/selftests/kvm/x86_64/xapic_state_test.c+55−0
@@ -132,6 +132,59 @@ static void test_icr(struct xapic_vcpu *x) · 131 unmodified lines
132132 __test_icr(x, -1ull & ~APIC_DM_FIXED_MASK);
133133 }
134134
135+static void __test_apic_id(struct kvm_vcpu *vcpu, uint64_t apic_base)
136+{
137+ uint32_t apic_id, expected;
138+ struct kvm_lapic_state xapic;
139+
140+ vcpu_set_msr(vcpu, MSR_IA32_APICBASE, apic_base);
141+
142+ vcpu_ioctl(vcpu, KVM_GET_LAPIC, &xapic);
143+
144+ expected = apic_base & X2APIC_ENABLE ? vcpu->id : vcpu->id << 24;
145+ apic_id = *((u32 *)&xapic.regs[APIC_ID]);
146+
147+ TEST_ASSERT(apic_id == expected,
148+ "APIC_ID not set back to %s format; wanted = %x, got = %x",
149+ (apic_base & X2APIC_ENABLE) ? "x2APIC" : "xAPIC",
150+ expected, apic_id);
151+}
152+
153+/*
154+ * Verify that KVM switches the APIC_ID between xAPIC and x2APIC when userspace
155+ * stuffs MSR_IA32_APICBASE. Setting the APIC_ID when x2APIC is enabled and
156+ * when the APIC transitions for DISABLED to ENABLED is architectural behavior
157+ * (on Intel), whereas the x2APIC => xAPIC transition behavior is KVM ABI since
158+ * attempted to transition from x2APIC to xAPIC without disabling the APIC is
159+ * architecturally disallowed.
160+ */
161+static void test_apic_id(void)
162+{
163+ const uint32_t NR_VCPUS = 3;
164+ struct kvm_vcpu *vcpus[NR_VCPUS];
165+ uint64_t apic_base;
166+ struct kvm_vm *vm;
167+ int i;
168+
169+ vm = vm_create_with_vcpus(NR_VCPUS, NULL, vcpus);
170+ vm_enable_cap(vm, KVM_CAP_X2APIC_API, KVM_X2APIC_API_USE_32BIT_IDS);
171+
172+ for (i = 0; i < NR_VCPUS; i++) {
173+ apic_base = vcpu_get_msr(vcpus[i], MSR_IA32_APICBASE);
174+
175+ TEST_ASSERT(apic_base & MSR_IA32_APICBASE_ENABLE,
176+ "APIC not in ENABLED state at vCPU RESET");
177+ TEST_ASSERT(!(apic_base & X2APIC_ENABLE),
178+ "APIC not in xAPIC mode at vCPU RESET");
179+
180+ __test_apic_id(vcpus[i], apic_base);
181+ __test_apic_id(vcpus[i], apic_base | X2APIC_ENABLE);
182+ __test_apic_id(vcpus[i], apic_base);
183+ }
184+
185+ kvm_vm_free(vm);
186+}
187+
135188 int main(int argc, char *argv[])
136189 {
137190 struct xapic_vcpu x = {
@@ -157,4 +210,6 @@ int main(int argc, char *argv[]) · 19 unmodified lines
157210 virt_pg_map(vm, APIC_DEFAULT_GPA, APIC_DEFAULT_GPA);
158211 test_icr(&x);
159212 kvm_vm_free(vm);
213+
214+ test_apic_id();
160215 }
tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c+6−9
@@ -19,9 +19,6 @@ · 18 unmodified lines
1919
2020 #include <sys/eventfd.h>
2121
22−/* Defined in include/linux/kvm_types.h */
23−#define GPA_INVALID (~(ulong)0)
24−
2522 #define SHINFO_REGION_GVA 0xc0000000ULL
2623 #define SHINFO_REGION_GPA 0xc0000000ULL
2724 #define SHINFO_REGION_SLOT 10
@@ -412,21 +409,21 @@ static void *juggle_shinfo_state(void *arg) · 384 unmodified lines
412409 {
413410 struct kvm_vm *vm = (struct kvm_vm *)arg;
414411
415− struct kvm_xen_hvm_attr cache_init = {
412+ struct kvm_xen_hvm_attr cache_activate = {
416413 .type = KVM_XEN_ATTR_TYPE_SHARED_INFO,
417414 .u.shared_info.gfn = SHINFO_REGION_GPA / PAGE_SIZE
418415 };
419416
420− struct kvm_xen_hvm_attr cache_destroy = {
417+ struct kvm_xen_hvm_attr cache_deactivate = {
421418 .type = KVM_XEN_ATTR_TYPE_SHARED_INFO,
422− .u.shared_info.gfn = GPA_INVALID
419+ .u.shared_info.gfn = KVM_XEN_INVALID_GFN
423420 };
424421
425422 for (;;) {
426− __vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &cache_init);
427− __vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &cache_destroy);
423+ __vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &cache_activate);
424+ __vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &cache_deactivate);
428425 pthread_testcancel();
429− };
426+ }
430427
431428 return NULL;
432429 }
virt/kvm/Kconfig+3−0
@@ -92,3 +92,6 @@ config KVM_XFER_TO_GUEST_WORK · 91 unmodified lines
9292
9393 config HAVE_KVM_PM_NOTIFIER
9494 bool
95+
96+config KVM_GENERIC_HARDWARE_ENABLING
97+ bool
virt/kvm/coalesced_mmio.c+5−3
@@ -187,15 +187,17 @@ int kvm_vm_ioctl_unregister_coalesced_mmio(struct kvm *kvm, · 186 unmodified lines
187187 r = kvm_io_bus_unregister_dev(kvm,
188188 zone->pio ? KVM_PIO_BUS : KVM_MMIO_BUS, &dev->dev);
189189
190+ kvm_iodevice_destructor(&dev->dev);
191+
190192 /*
191193 * On failure, unregister destroys all devices on the
192194 * bus _except_ the target device, i.e. coalesced_zones
193− * has been modified. No need to restart the walk as
194− * there aren't any zones left.
195+ * has been modified. Bail after destroying the target
196+ * device, there's no need to restart the walk as there
197+ * aren't any zones left.
195198 */
196199 if (r)
197200 break;
198− kvm_iodevice_destructor(&dev->dev);
199201 }
200202 }
201203
virt/kvm/kvm_main.c+156−141
@@ -100,13 +100,8 @@ EXPORT_SYMBOL_GPL(halt_poll_ns_shrink); · 99 unmodified lines
100100 */
101101
102102 DEFINE_MUTEX(kvm_lock);
103−static DEFINE_RAW_SPINLOCK(kvm_count_lock);
104103 LIST_HEAD(vm_list);
105104
106−static cpumask_var_t cpus_hardware_enabled;
107−static int kvm_usage_count;
108−static atomic_t hardware_enable_failed;
109−
110105 static struct kmem_cache *kvm_vcpu_cache;
111106
112107 static __read_mostly struct preempt_ops kvm_preempt_ops;
@@ -148,9 +143,6 @@ static void hardware_disable_all(void); · 35 unmodified lines
148143
149144 static void kvm_io_bus_destroy(struct kvm_io_bus *bus);
150145
151−__visible bool kvm_rebooting;
152−EXPORT_SYMBOL_GPL(kvm_rebooting);
153−
154146 #define KVM_EVENT_CREATE_VM 0
155147 #define KVM_EVENT_DESTROY_VM 1
156148 static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm);
@@ -5102,50 +5094,70 @@ static struct miscdevice kvm_dev = { · 4945 unmodified lines
51025094 &kvm_chardev_ops,
51035095 };
51045096
5105−static void hardware_enable_nolock(void *junk)
5106−{
5107− int cpu = raw_smp_processor_id();
5108− int r;
5109−
5110− if (cpumask_test_cpu(cpu, cpus_hardware_enabled))
5111− return;
5097+#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING
5098+__visible bool kvm_rebooting;
5099+EXPORT_SYMBOL_GPL(kvm_rebooting);
51125100
5113− cpumask_set_cpu(cpu, cpus_hardware_enabled);
5101+static DEFINE_PER_CPU(bool, hardware_enabled);
5102+static int kvm_usage_count;
51145103
5115− r = kvm_arch_hardware_enable();
5104+static int __hardware_enable_nolock(void)
5105+{
5106+ if (__this_cpu_read(hardware_enabled))
5107+ return 0;
51165108
5117− if (r) {
5118− cpumask_clear_cpu(cpu, cpus_hardware_enabled);
5119− atomic_inc(&hardware_enable_failed);
5120− pr_info("kvm: enabling virtualization on CPU%d failed\n", cpu);
5109+ if (kvm_arch_hardware_enable()) {
5110+ pr_info("kvm: enabling virtualization on CPU%d failed\n",
5111+ raw_smp_processor_id());
5112+ return -EIO;
51215113 }
5114+
5115+ __this_cpu_write(hardware_enabled, true);
5116+ return 0;
51225117 }
51235118
5124−static int kvm_starting_cpu(unsigned int cpu)
5119+static void hardware_enable_nolock(void *failed)
51255120 {
5126− raw_spin_lock(&kvm_count_lock);
5121+ if (__hardware_enable_nolock())
5122+ atomic_inc(failed);
5123+}
5124+
5125+static int kvm_online_cpu(unsigned int cpu)
5126+{
5127+ int ret = 0;
5128+
5129+ /*
5130+ * Abort the CPU online process if hardware virtualization cannot
5131+ * be enabled. Otherwise running VMs would encounter unrecoverable
5132+ * errors when scheduled to this CPU.
5133+ */
5134+ mutex_lock(&kvm_lock);
51275135 if (kvm_usage_count)
5128− hardware_enable_nolock(NULL);
5129− raw_spin_unlock(&kvm_count_lock);
5130− return 0;
5136+ ret = __hardware_enable_nolock();
5137+ mutex_unlock(&kvm_lock);
5138+ return ret;
51315139 }
51325140
51335141 static void hardware_disable_nolock(void *junk)
51345142 {
5135− int cpu = raw_smp_processor_id();
5136−
5137− if (!cpumask_test_cpu(cpu, cpus_hardware_enabled))
5143+ /*
5144+ * Note, hardware_disable_all_nolock() tells all online CPUs to disable
5145+ * hardware, not just CPUs that successfully enabled hardware!
5146+ */
5147+ if (!__this_cpu_read(hardware_enabled))
51385148 return;
5139− cpumask_clear_cpu(cpu, cpus_hardware_enabled);
5149+
51405150 kvm_arch_hardware_disable();
5151+
5152+ __this_cpu_write(hardware_enabled, false);
51415153 }
51425154
5143−static int kvm_dying_cpu(unsigned int cpu)
5155+static int kvm_offline_cpu(unsigned int cpu)
51445156 {
5145− raw_spin_lock(&kvm_count_lock);
5157+ mutex_lock(&kvm_lock);
51465158 if (kvm_usage_count)
51475159 hardware_disable_nolock(NULL);
5148− raw_spin_unlock(&kvm_count_lock);
5160+ mutex_unlock(&kvm_lock);
51495161 return 0;
51505162 }
51515163
@@ -5160,29 +5172,41 @@ static void hardware_disable_all_nolock(void) · 8 unmodified lines
51605172
51615173 static void hardware_disable_all(void)
51625174 {
5163− raw_spin_lock(&kvm_count_lock);
5175+ cpus_read_lock();
5176+ mutex_lock(&kvm_lock);
51645177 hardware_disable_all_nolock();
5165− raw_spin_unlock(&kvm_count_lock);
5178+ mutex_unlock(&kvm_lock);
5179+ cpus_read_unlock();
51665180 }
51675181
51685182 static int hardware_enable_all(void)
51695183 {
5184+ atomic_t failed = ATOMIC_INIT(0);
51705185 int r = 0;
51715186
5172− raw_spin_lock(&kvm_count_lock);
5187+ /*
5188+ * When onlining a CPU, cpu_online_mask is set before kvm_online_cpu()
5189+ * is called, and so on_each_cpu() between them includes the CPU that
5190+ * is being onlined. As a result, hardware_enable_nolock() may get
5191+ * invoked before kvm_online_cpu(), which also enables hardware if the
5192+ * usage count is non-zero. Disable CPU hotplug to avoid attempting to
5193+ * enable hardware multiple times.
5194+ */
5195+ cpus_read_lock();
5196+ mutex_lock(&kvm_lock);
51735197
51745198 kvm_usage_count++;
51755199 if (kvm_usage_count == 1) {
5176− atomic_set(&hardware_enable_failed, 0);
5177− on_each_cpu(hardware_enable_nolock, NULL, 1);
5200+ on_each_cpu(hardware_enable_nolock, &failed, 1);
51785201
5179− if (atomic_read(&hardware_enable_failed)) {
5202+ if (atomic_read(&failed)) {
51805203 hardware_disable_all_nolock();
51815204 r = -EBUSY;
51825205 }
51835206 }
51845207
5185− raw_spin_unlock(&kvm_count_lock);
5208+ mutex_unlock(&kvm_lock);
5209+ cpus_read_unlock();
51865210
51875211 return r;
51885212 }
@@ -5207,6 +5231,49 @@ static struct notifier_block kvm_reboot_notifier = { · 18 unmodified lines
52075231 .priority = 0,
52085232 };
52095233
5234+static int kvm_suspend(void)
5235+{
5236+ /*
5237+ * Secondary CPUs and CPU hotplug are disabled across the suspend/resume
5238+ * callbacks, i.e. no need to acquire kvm_lock to ensure the usage count
5239+ * is stable. Assert that kvm_lock is not held to ensure the system
5240+ * isn't suspended while KVM is enabling hardware. Hardware enabling
5241+ * can be preempted, but the task cannot be frozen until it has dropped
5242+ * all locks (userspace tasks are frozen via a fake signal).
5243+ */
5244+ lockdep_assert_not_held(&kvm_lock);
5245+ lockdep_assert_irqs_disabled();
5246+
5247+ if (kvm_usage_count)
5248+ hardware_disable_nolock(NULL);
5249+ return 0;
5250+}
5251+
5252+static void kvm_resume(void)
5253+{
5254+ lockdep_assert_not_held(&kvm_lock);
5255+ lockdep_assert_irqs_disabled();
5256+
5257+ if (kvm_usage_count)
5258+ WARN_ON_ONCE(__hardware_enable_nolock());
5259+}
5260+
5261+static struct syscore_ops kvm_syscore_ops = {
5262+ .suspend = kvm_suspend,
5263+ .resume = kvm_resume,
5264+};
5265+#else /* CONFIG_KVM_GENERIC_HARDWARE_ENABLING */
5266+static int hardware_enable_all(void)
5267+{
5268+ return 0;
5269+}
5270+
5271+static void hardware_disable_all(void)
5272+{
5273+
5274+}
5275+#endif /* CONFIG_KVM_GENERIC_HARDWARE_ENABLING */
5276+
52105277 static void kvm_io_bus_destroy(struct kvm_io_bus *bus)
52115278 {
52125279 int i;
@@ -5785,26 +5852,6 @@ static void kvm_init_debug(void) · 572 unmodified lines
57855852 }
57865853 }
57875854
5788−static int kvm_suspend(void)
5789−{
5790− if (kvm_usage_count)
5791− hardware_disable_nolock(NULL);
5792− return 0;
5793−}
5794−
5795−static void kvm_resume(void)
5796−{
5797− if (kvm_usage_count) {
5798− lockdep_assert_not_held(&kvm_count_lock);
5799− hardware_enable_nolock(NULL);
5800− }
5801−}
5802−
5803−static struct syscore_ops kvm_syscore_ops = {
5804− .suspend = kvm_suspend,
5805− .resume = kvm_resume,
5806−};
5807−
58085855 static inline
58095856 struct kvm_vcpu *preempt_notifier_to_vcpu(struct preempt_notifier *pn)
58105857 {
@@ -5909,62 +5956,20 @@ void kvm_unregister_perf_callbacks(void) · 98 unmodified lines
59095956 }
59105957 #endif
59115958
5912−struct kvm_cpu_compat_check {
5913− void *opaque;
5914− int *ret;
5915−};
5916−
5917−static void check_processor_compat(void *data)
5918−{
5919− struct kvm_cpu_compat_check *c = data;
5920−
5921− *c->ret = kvm_arch_check_processor_compat(c->opaque);
5922−}
5923−
5924−int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
5925− struct module *module)
5959+int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module)
59265960 {
5927− struct kvm_cpu_compat_check c;
59285961 int r;
59295962 int cpu;
59305963
5931− r = kvm_arch_init(opaque);
5932− if (r)
5933− goto out_fail;
5934−
5935− /*
5936− * kvm_arch_init makes sure there's at most one caller
5937− * for architectures that support multiple implementations,
5938− * like intel and amd on x86.
5939− * kvm_arch_init must be called before kvm_irqfd_init to avoid creating
5940− * conflicts in case kvm is already setup for another implementation.
5941− */
5942− r = kvm_irqfd_init();
5964+#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING
5965+ r = cpuhp_setup_state_nocalls(CPUHP_AP_KVM_ONLINE, "kvm/cpu:online",
5966+ kvm_online_cpu, kvm_offline_cpu);
59435967 if (r)
5944− goto out_irqfd;
5945−
5946− if (!zalloc_cpumask_var(&cpus_hardware_enabled, GFP_KERNEL)) {
5947− r = -ENOMEM;
5948− goto out_free_0;
5949− }
5950−
5951− r = kvm_arch_hardware_setup(opaque);
5952− if (r < 0)
5953− goto out_free_1;
5954−
5955− c.ret = &r;
5956− c.opaque = opaque;
5957− for_each_online_cpu(cpu) {
5958− smp_call_function_single(cpu, check_processor_compat, &c, 1);
5959− if (r < 0)
5960− goto out_free_2;
5961− }
5968+ return r;
59625969
5963− r = cpuhp_setup_state_nocalls(CPUHP_AP_KVM_STARTING, "kvm/cpu:starting",
5964− kvm_starting_cpu, kvm_dying_cpu);
5965− if (r)
5966− goto out_free_2;
59675970 register_reboot_notifier(&kvm_reboot_notifier);
5971+ register_syscore_ops(&kvm_syscore_ops);
5972+#endif
59685973
59695974 /* A kmem cache lets us meet the alignment requirements of fx_save. */
59705975 if (!vcpu_align)
@@ -5978,59 +5983,65 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align, · 7 unmodified lines
59785983 NULL);
59795984 if (!kvm_vcpu_cache) {
59805985 r = -ENOMEM;
5981− goto out_free_3;
5986+ goto err_vcpu_cache;
59825987 }
59835988
59845989 for_each_possible_cpu(cpu) {
59855990 if (!alloc_cpumask_var_node(&per_cpu(cpu_kick_mask, cpu),
59865991 GFP_KERNEL, cpu_to_node(cpu))) {
59875992 r = -ENOMEM;
5988− goto out_free_4;
5993+ goto err_cpu_kick_mask;
59895994 }
59905995 }
59915996
5997+ r = kvm_irqfd_init();
5998+ if (r)
5999+ goto err_irqfd;
6000+
59926001 r = kvm_async_pf_init();
59936002 if (r)
5994− goto out_free_4;
6003+ goto err_async_pf;
59956004
59966005 kvm_chardev_ops.owner = module;
59976006
5998− r = misc_register(&kvm_dev);
5999− if (r) {
6000− pr_err("kvm: misc device register failed\n");
6001− goto out_unreg;
6002− }
6003−
6004− register_syscore_ops(&kvm_syscore_ops);
6005−
60066007 kvm_preempt_ops.sched_in = kvm_sched_in;
60076008 kvm_preempt_ops.sched_out = kvm_sched_out;
60086009
60096010 kvm_init_debug();
60106011
60116012 r = kvm_vfio_ops_init();
6012− WARN_ON(r);
6013+ if (WARN_ON_ONCE(r))
6014+ goto err_vfio;
6015+
6016+ /*
6017+ * Registration _must_ be the very last thing done, as this exposes
6018+ * /dev/kvm to userspace, i.e. all infrastructure must be setup!
6019+ */
6020+ r = misc_register(&kvm_dev);
6021+ if (r) {
6022+ pr_err("kvm: misc device register failed\n");
6023+ goto err_register;
6024+ }
60136025
60146026 return 0;
60156027
6016−out_unreg:
6028+err_register:
6029+ kvm_vfio_ops_exit();
6030+err_vfio:
60176031 kvm_async_pf_deinit();
6018−out_free_4:
6032+err_async_pf:
6033+ kvm_irqfd_exit();
6034+err_irqfd:
6035+err_cpu_kick_mask:
60196036 for_each_possible_cpu(cpu)
60206037 free_cpumask_var(per_cpu(cpu_kick_mask, cpu));
60216038 kmem_cache_destroy(kvm_vcpu_cache);
6022−out_free_3:
6039+err_vcpu_cache:
6040+#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING
6041+ unregister_syscore_ops(&kvm_syscore_ops);
60236042 unregister_reboot_notifier(&kvm_reboot_notifier);
6024− cpuhp_remove_state_nocalls(CPUHP_AP_KVM_STARTING);
6025−out_free_2:
6026− kvm_arch_hardware_unsetup();
6027−out_free_1:
6028− free_cpumask_var(cpus_hardware_enabled);
6029−out_free_0:
6030− kvm_irqfd_exit();
6031−out_irqfd:
6032− kvm_arch_exit();
6033−out_fail:
6043+ cpuhp_remove_state_nocalls(CPUHP_AP_KVM_ONLINE);
6044+#endif
60346045 return r;
60356046 }
60366047 EXPORT_SYMBOL_GPL(kvm_init);
@@ -6039,21 +6050,25 @@ void kvm_exit(void) · 2 unmodified lines
60396050 {
60406051 int cpu;
60416052
6042− debugfs_remove_recursive(kvm_debugfs_dir);
6053+ /*
6054+ * Note, unregistering /dev/kvm doesn't strictly need to come first,
6055+ * fops_get(), a.k.a. try_module_get(), prevents acquiring references
6056+ * to KVM while the module is being stopped.
6057+ */
60436058 misc_deregister(&kvm_dev);
6059+
6060+ debugfs_remove_recursive(kvm_debugfs_dir);
60446061 for_each_possible_cpu(cpu)
60456062 free_cpumask_var(per_cpu(cpu_kick_mask, cpu));
60466063 kmem_cache_destroy(kvm_vcpu_cache);
6064+ kvm_vfio_ops_exit();
60476065 kvm_async_pf_deinit();
6066+#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING
60486067 unregister_syscore_ops(&kvm_syscore_ops);
60496068 unregister_reboot_notifier(&kvm_reboot_notifier);
6050− cpuhp_remove_state_nocalls(CPUHP_AP_KVM_STARTING);
6051− on_each_cpu(hardware_disable_nolock, NULL, 1);
6052− kvm_arch_hardware_unsetup();
6053− kvm_arch_exit();
6069+ cpuhp_remove_state_nocalls(CPUHP_AP_KVM_ONLINE);
6070+#endif
60546071 kvm_irqfd_exit();
6055− free_cpumask_var(cpus_hardware_enabled);
6056− kvm_vfio_ops_exit();
60576072 }
60586073 EXPORT_SYMBOL_GPL(kvm_exit);
60596074

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t