History

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm

Pull kvm updates from Paolo Bonzini: "ARM: - Provide a virtual cache topology to the guest to avoid inconsistencies with migration on heterogenous systems. Non secure software has no practical need to traverse the caches by set/way in the first place - Add support for taking stage-2 access faults in parallel. This was an accidental omission in the original parallel faults implementation, but should provide a marginal improvement to machines w/o FEAT_HAFDBS (such as hardware from the fruit company) - A preamble to adding support for nested virtualization to KVM, including vEL2 register state, rudimentary nested exception handling and masking unsupported features for nested guests - Fixes to the PSCI relay that avoid an unexpected host SVE trap when resuming a CPU when running pKVM - VGIC maintenance interrupt support for the AIC - Improvements to the arch timer emulation, primarily aimed at reducing the trap overhead of running nested - Add CONFIG_USERFAULTFD to the KVM selftests config fragment in the interest of CI systems - Avoid VM-wide stop-the-world operations when a vCPU accesses its own redistributor - Serialize when toggling CPACR_EL1.SMEN to avoid unexpected exceptions in the host - Aesthetic and comment/kerneldoc fixes - Drop the vestiges of the old Columbia mailing list and add [Oliver] as co-maintainer RISC-V: - Fix wrong usage of PGDIR_SIZE instead of PUD_SIZE - Correctly place the guest in S-mode after redirecting a trap to the guest - Redirect illegal instruction traps to guest - SBI PMU support for guest s390: - Sort out confusion between virtual and physical addresses, which currently are the same on s390 - A new ioctl that performs cmpxchg on guest memory - A few fixes x86: - Change tdp_mmu to a read-only parameter - Separate TDP and shadow MMU page fault paths - Enable Hyper-V invariant TSC control - Fix a variety of APICv and AVIC bugs, some of them real-world, some of them affecting architecurally legal but unlikely to happen in practice - Mark APIC timer as expired if its in one-shot mode and the count underflows while the vCPU task was being migrated - Advertise support for Intel's new fast REP string features - Fix a double-shootdown issue in the emergency reboot code - Ensure GIF=1 and disable SVM during an emergency reboot, i.e. give SVM similar treatment to VMX - Update Xen's TSC info CPUID sub-leaves as appropriate - Add support for Hyper-V's extended hypercalls, where "support" at this point is just forwarding the hypercalls to userspace - Clean up the kvm->lock vs. kvm->srcu sequences when updating the PMU and MSR filters - One-off fixes and cleanups - Fix and cleanup the range-based TLB flushing code, used when KVM is running on Hyper-V - Add support for filtering PMU events using a mask. If userspace wants to restrict heavily what events the guest can use, it can now do so without needing an absurd number of filter entries - Clean up KVM's handling of "PMU MSRs to save", especially when vPMU support is disabled - Add PEBS support for Intel Sapphire Rapids - Fix a mostly benign overflow bug in SEV's send|receive_update_data() - Move several SVM-specific flags into vcpu_svm x86 Intel: - Handle NMI VM-Exits before leaving the noinstr region - A few trivial cleanups in the VM-Enter flows - Stop enabling VMFUNC for L1 purely to document that KVM doesn't support EPTP switching (or any other VM function) for L1 - Fix a crash when using eVMCS's enlighted MSR bitmaps Generic: - Clean up the hardware enable and initialization flow, which was scattered around multiple arch-specific hooks. Instead, just let the arch code call into generic code. Both x86 and ARM should benefit from not having to fight common KVM code's notion of how to do initialization - Account allocations in generic kvm_arch_alloc_vm() - Fix a memory leak if coalesced MMIO unregistration fails selftests: - On x86, cache the CPU vendor (AMD vs. Intel) and use the info to emit the correct hypercall instruction instead of relying on KVM to patch in VMMCALL - Use TAP interface for kvm_binary_stats_test and tsc_msrs_test" * tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (325 commits) KVM: SVM: hyper-v: placate modpost section mismatch error KVM: x86/mmu: Make tdp_mmu_allowed static KVM: arm64: nv: Use reg_to_encoding() to get sysreg ID KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 changes KVM: arm64: nv: Filter out unsupported features from ID regs KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2 KVM: arm64: nv: Allow a sysreg to be hidden from userspace only KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor KVM: arm64: nv: Add accessors for SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2 KVM: arm64: nv: Handle SMCs taken from virtual EL2 KVM: arm64: nv: Handle trapped ERET from virtual EL2 KVM: arm64: nv: Inject HVC exceptions to the virtual EL2 KVM: arm64: nv: Support virtual EL2 exceptions KVM: arm64: nv: Handle HCR_EL2.NV system register traps KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state KVM: arm64: nv: Add EL2 system registers to vcpu context KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set KVM: arm64: nv: Introduce nested virtualization VCPU feature KVM: arm64: Use the S2 MMU context to iterate over S2 table ...

Linus Torvalds committed 3 years ago

Commit 49d57592 parents 01687e7 45dd9bc200 files · +7274 −2928Browse files

Unified · Split

200 files
  1. Documentation/admin-guide/kernel-parameters.txt +6 −1
  2. Documentation/virt/kvm/api.rst +108 −16
  3. Documentation/virt/kvm/devices/vm.rst +4 −0
  4. Documentation/virt/kvm/locking.rst +16 −9
  5. Documentation/virt/kvm/x86/errata.rst +11 −0
  6. MAINTAINERS +1 −2
  7. arch/arm64/include/asm/cache.h +9 −0
  8. arch/arm64/include/asm/el2_setup.h +99 −0
  9. arch/arm64/include/asm/esr.h +4 −0
  10. arch/arm64/include/asm/kvm_arm.h +20 −3
  11. arch/arm64/include/asm/kvm_emulate.h +66 −4
  12. arch/arm64/include/asm/kvm_host.h +53 −14
  13. arch/arm64/include/asm/kvm_hyp.h +1 −0
  14. arch/arm64/include/asm/kvm_mmu.h +12 −3
  15. arch/arm64/include/asm/kvm_nested.h +20 −0
  16. arch/arm64/include/asm/kvm_pgtable.h +8 −0
  17. arch/arm64/include/asm/sysreg.h +37 −2
  18. arch/arm64/include/uapi/asm/kvm.h +1 −0
  19. arch/arm64/kernel/cacheinfo.c +0 −5
  20. arch/arm64/kernel/cpufeature.c +25 −0
  21. arch/arm64/kernel/hyp-stub.S +2 −84
  22. arch/arm64/kvm/Kconfig +1 −0
  23. arch/arm64/kvm/Makefile +1 −1
  24. arch/arm64/kvm/arch_timer.c +60 −46
  25. arch/arm64/kvm/arm.c +60 −49
  26. arch/arm64/kvm/emulate-nested.c +203 −0
  27. arch/arm64/kvm/fpsimd.c +1 −0
  28. arch/arm64/kvm/guest.c +6 −0
  29. arch/arm64/kvm/handle_exit.c +45 −2
  30. arch/arm64/kvm/hyp/exception.c +36 −12
  31. arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +18 −3
  32. arch/arm64/kvm/hyp/nvhe/hyp-init.S +1 −0
  33. arch/arm64/kvm/hyp/nvhe/sys_regs.c +1 −0
  34. arch/arm64/kvm/hyp/pgtable.c +37 −6
  35. arch/arm64/kvm/hyp/vhe/switch.c +25 −1
  36. arch/arm64/kvm/hypercalls.c +1 −1
  37. arch/arm64/kvm/inject_fault.c +55 −6
  38. arch/arm64/kvm/mmu.c +23 −23
  39. arch/arm64/kvm/nested.c +161 −0
  40. arch/arm64/kvm/pvtime.c +4 −4
  41. arch/arm64/kvm/reset.c +21 −4
  42. arch/arm64/kvm/sys_regs.c +350 −109
  43. arch/arm64/kvm/sys_regs.h +12 −2
  44. arch/arm64/kvm/trace_arm.h +59 −0
  45. arch/arm64/kvm/vgic/vgic-init.c +3 −18
  46. arch/arm64/kvm/vgic/vgic-mmio.c +8 −5
  47. arch/arm64/kvm/vgic/vgic-v3.c +5 −4
  48. arch/arm64/kvm/vmid.c +3 −3
  49. arch/arm64/tools/cpucaps +1 −0
  50. arch/arm64/tools/gen-sysreg.awk +19 −1
  51. arch/arm64/tools/sysreg +17 −0
  52. arch/mips/include/asm/kvm_host.h +1 −2
  53. arch/mips/kvm/Kconfig +1 −0
  54. arch/mips/kvm/Makefile +1 −1
  55. arch/mips/kvm/callback.c +0 −14
  56. arch/mips/kvm/mips.c +7 −27
  57. arch/mips/kvm/vz.c +4 −3
  58. arch/powerpc/include/asm/kvm_host.h +0 −3
  59. arch/powerpc/include/asm/kvm_ppc.h +0 −1
  60. arch/powerpc/kvm/book3s.c +1 −11
  61. arch/powerpc/kvm/booke.c +1 −1
  62. arch/powerpc/kvm/e500.c +3 −3
  63. arch/powerpc/kvm/e500mc.c +6 −2
  64. arch/powerpc/kvm/powerpc.c +0 −20
  65. arch/riscv/include/asm/kvm_host.h +7 −4
  66. arch/riscv/include/asm/kvm_vcpu_pmu.h +107 −0
  67. arch/riscv/include/asm/kvm_vcpu_sbi.h +11 −2
  68. arch/riscv/include/asm/sbi.h +5 −2
  69. arch/riscv/kvm/Kconfig +1 −0
  70. arch/riscv/kvm/Makefile +1 −0
  71. arch/riscv/kvm/main.c +4 −22
  72. arch/riscv/kvm/mmu.c +10 −10
  73. arch/riscv/kvm/tlb.c +4 −0
  74. arch/riscv/kvm/vcpu.c +7 −0
  75. arch/riscv/kvm/vcpu_exit.c +9 −0
  76. arch/riscv/kvm/vcpu_insn.c +3 −1
  77. arch/riscv/kvm/vcpu_pmu.c +633 −0
  78. arch/riscv/kvm/vcpu_sbi.c +35 −37
  79. arch/riscv/kvm/vcpu_sbi_base.c +14 −13
  80. arch/riscv/kvm/vcpu_sbi_hsm.c +14 −14
  81. arch/riscv/kvm/vcpu_sbi_pmu.c +86 −0
  82. arch/riscv/kvm/vcpu_sbi_replace.c +27 −23
  83. arch/riscv/kvm/vcpu_sbi_v01.c +7 −10
  84. arch/riscv/kvm/vmid.c +2 −2
  85. arch/s390/include/asm/kvm_host.h +0 −1
  86. arch/s390/kvm/gaccess.c +109 −0
  87. arch/s390/kvm/gaccess.h +3 −0
  88. arch/s390/kvm/interrupt.c +8 −5
  89. arch/s390/kvm/kvm-s390.c +208 −140
  90. arch/s390/kvm/kvm-s390.h +1 −1
  91. arch/s390/kvm/pci.c +1 −1
  92. arch/s390/kvm/pci.h +1 −1
  93. arch/x86/events/intel/core.c +1 −0
  94. arch/x86/events/intel/ds.c +3 −1
  95. arch/x86/include/asm/cpufeatures.h +3 −0
  96. arch/x86/include/asm/hyperv-tlfs.h +3 −0
  97. arch/x86/include/asm/idtentry.h +6 −10
  98. arch/x86/include/asm/kvm-x86-ops.h +1 −1
  99. arch/x86/include/asm/kvm_host.h +71 −38
  100. arch/x86/include/asm/reboot.h +2 −0
  101. arch/x86/include/asm/virtext.h +15 −1
  102. arch/x86/include/asm/xen/hypervisor.h +3 −1
  103. arch/x86/include/uapi/asm/kvm.h +32 −2
  104. arch/x86/kernel/cpu/mshyperv.c +1 −1
  105. arch/x86/kernel/crash.c +1 −16
  106. arch/x86/kernel/nmi.c +4 −4
  107. arch/x86/kernel/reboot.c +64 −24
  108. arch/x86/kernel/smp.c +3 −3
  109. arch/x86/kvm/Kconfig +1 −0
  110. arch/x86/kvm/cpuid.c +28 −15
  111. arch/x86/kvm/debugfs.c +2 −0
  112. arch/x86/kvm/emulate.c +9 −9
  113. arch/x86/kvm/hyperv.c +63 −22
  114. arch/x86/kvm/hyperv.h +27 −0
  115. arch/x86/kvm/i8254.c +2 −2
  116. arch/x86/kvm/i8259.c +3 −1
  117. arch/x86/kvm/ioapic.c +1 −0
  118. arch/x86/kvm/irq.c +1 −0
  119. arch/x86/kvm/irq_comm.c +4 −3
  120. arch/x86/kvm/kvm_cache_regs.h +12 −0
  121. arch/x86/kvm/kvm_emulate.h +2 −5
  122. arch/x86/kvm/kvm_onhyperv.c +1 −0
  123. arch/x86/kvm/lapic.c +296 −108
  124. arch/x86/kvm/lapic.h +4 −0
  125. arch/x86/kvm/mmu.h +3 −3
  126. arch/x86/kvm/mmu/mmu.c +203 −117
  127. arch/x86/kvm/mmu/mmu_internal.h +21 −1
  128. arch/x86/kvm/mmu/page_track.c +1 −0
  129. arch/x86/kvm/mmu/paging_tmpl.h +7 −18
  130. arch/x86/kvm/mmu/spte.c +5 −5
  131. arch/x86/kvm/mmu/spte.h +10 −10
  132. arch/x86/kvm/mmu/tdp_iter.c +4 −8
  133. arch/x86/kvm/mmu/tdp_mmu.c +4 −16
  134. arch/x86/kvm/mmu/tdp_mmu.h +3 −22
  135. arch/x86/kvm/mtrr.c +1 −0
  136. arch/x86/kvm/pmu.c +238 −52
  137. arch/x86/kvm/pmu.h +6 −7
  138. arch/x86/kvm/reverse_cpuid.h +7 −0
  139. arch/x86/kvm/smm.c +1 −2
  140. arch/x86/kvm/svm/avic.c +167 −207
  141. arch/x86/kvm/svm/nested.c +2 −3
  142. arch/x86/kvm/svm/pmu.c +4 −0
  143. arch/x86/kvm/svm/sev.c +4 −3
  144. arch/x86/kvm/svm/svm.c +75 −56
  145. arch/x86/kvm/svm/svm.h +43 −15
  146. arch/x86/kvm/svm/svm_onhyperv.c +1 −0
  147. arch/x86/kvm/svm/svm_onhyperv.h +4 −4
  148. arch/x86/kvm/vmx/capabilities.h +2 −2
  149. arch/x86/kvm/vmx/hyperv.c +68 −19
  150. arch/x86/kvm/vmx/hyperv.h +89 −39
  151. arch/x86/kvm/vmx/nested.c +10 −14
  152. arch/x86/kvm/vmx/pmu_intel.c +17 −11
  153. arch/x86/kvm/vmx/posted_intr.c +2 −0
  154. arch/x86/kvm/vmx/sgx.c +3 −2
  155. arch/x86/kvm/vmx/vmcs.h +2 −2
  156. arch/x86/kvm/vmx/vmcs12.c +1 −0
  157. arch/x86/kvm/vmx/vmenter.S +44 −36
  158. arch/x86/kvm/vmx/vmx.c +328 −296
  159. arch/x86/kvm/vmx/vmx.h +8 −10
  160. arch/x86/kvm/vmx/vmx_ops.h +4 −2
  161. arch/x86/kvm/x86.c +310 −247
  162. arch/x86/kvm/x86.h +15 −3
  163. arch/x86/kvm/xen.c +27 −0
  164. arch/x86/kvm/xen.h +7 −0
  165. drivers/irqchip/irq-apple-aic.c +40 −13
  166. drivers/perf/riscv_pmu_sbi.c +57 −7
  167. drivers/s390/virtio/virtio_ccw.c +24 −22
  168. include/kvm/arm_arch_timer.h +5 −1
  169. include/kvm/arm_vgic.h +4 −0
  170. include/linux/cpuhotplug.h +1 −4
  171. include/linux/kvm_host.h +8 −11
  172. include/linux/kvm_types.h +1 −1
  173. include/linux/perf/riscv_pmu.h +5 −0
  174. include/uapi/linux/kvm.h +9 −0
  175. tools/testing/selftests/kvm/Makefile +4 −0
  176. tools/testing/selftests/kvm/aarch64/page_fault_test.c +0 −2
  177. tools/testing/selftests/kvm/config +1 −0
  178. tools/testing/selftests/kvm/include/x86_64/hyperv.h +102 −47
  179. tools/testing/selftests/kvm/include/x86_64/processor.h +27 −4
  180. tools/testing/selftests/kvm/kvm_binary_stats_test.c +10 −11
  181. tools/testing/selftests/kvm/lib/elf.c +2 −0
  182. tools/testing/selftests/kvm/lib/guest_modes.c +1 −1
  183. tools/testing/selftests/kvm/lib/kvm_util.c +1 −3
  184. tools/testing/selftests/kvm/lib/x86_64/processor.c +20 −26
  185. tools/testing/selftests/kvm/memslot_perf_test.c +1 −4
  186. tools/testing/selftests/kvm/s390x/memop.c +524 −148
  187. tools/testing/selftests/kvm/x86_64/exit_on_emulation_failure_test.c +0 −3
  188. tools/testing/selftests/kvm/x86_64/fix_hypercall_test.c +2 −2
  189. tools/testing/selftests/kvm/x86_64/hyperv_clock.c +1 −1
  190. tools/testing/selftests/kvm/x86_64/hyperv_extended_hypercalls.c +97 −0
  191. tools/testing/selftests/kvm/x86_64/hyperv_features.c +207 −132
  192. tools/testing/selftests/kvm/x86_64/mmio_warning_test.c +1 −1
  193. tools/testing/selftests/kvm/x86_64/pmu_event_filter_test.c +378 −7
  194. tools/testing/selftests/kvm/x86_64/tsc_msrs_test.c +12 −4
  195. tools/testing/selftests/kvm/x86_64/vmx_exception_with_invalid_guest_state.c +1 −1
  196. tools/testing/selftests/kvm/x86_64/xapic_state_test.c +55 −0
  197. tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c +6 −9
  198. virt/kvm/Kconfig +3 −0
  199. virt/kvm/coalesced_mmio.c +5 −3
  200. virt/kvm/kvm_main.c +156 −141
Documentation/admin-guide/kernel-parameters.txt+6−1
@@ -2536,9 +2536,14 @@ · 2535 unmodified lines
2536 protected: nVHE-based mode with support for guests whose2536 protected: nVHE-based mode with support for guests whose
2537 state is kept private from the host.2537 state is kept private from the host.
2538 2538
2539 nested: VHE-based mode with support for nested
2540 virtualization. Requires at least ARMv8.3
2541 hardware.
2542
2539 Defaults to VHE/nVHE based on hardware support. Setting2543 Defaults to VHE/nVHE based on hardware support. Setting
2540 mode to "protected" will disable kexec and hibernation2544 mode to "protected" will disable kexec and hibernation
2541 for the host.2545 for the host. "nested" is experimental and should be
2546 used with extreme caution.
2542 2547
2543 kvm-arm.vgic_v3_group0_trap=2548 kvm-arm.vgic_v3_group0_trap=
2544 [KVM,ARM] Trap guest accesses to GICv3 group-02549 [KVM,ARM] Trap guest accesses to GICv3 group-0
Documentation/virt/kvm/api.rst+108−16
@@ -3736,7 +3736,7 @@ The fields in each entry are defined as follows: · 3735 unmodified lines
3736:Parameters: struct kvm_s390_mem_op (in)3736:Parameters: struct kvm_s390_mem_op (in)
3737:Returns: = 0 on success,3737:Returns: = 0 on success,
3738 < 0 on generic error (e.g. -EFAULT or -ENOMEM),3738 < 0 on generic error (e.g. -EFAULT or -ENOMEM),
3739 > 0 if an exception occurred while walking the page tables3739 16 bit program exception code if the access causes such an exception
3740 3740
3741Read or write data from/to the VM's memory.3741Read or write data from/to the VM's memory.
3742The KVM_CAP_S390_MEM_OP_EXTENSION capability specifies what functionality is3742The KVM_CAP_S390_MEM_OP_EXTENSION capability specifies what functionality is
@@ -3754,6 +3754,8 @@ Parameters are specified via the following structure:: · 11 unmodified lines
3754 struct {3754 struct {
3755 __u8 ar; /* the access register number */3755 __u8 ar; /* the access register number */
3756 __u8 key; /* access key, ignored if flag unset */3756 __u8 key; /* access key, ignored if flag unset */
3757 __u8 pad1[6]; /* ignored */
3758 __u64 old_addr; /* ignored if flag unset */
3757 };3759 };
3758 __u32 sida_offset; /* offset into the sida */3760 __u32 sida_offset; /* offset into the sida */
3759 __u8 reserved[32]; /* ignored */3761 __u8 reserved[32]; /* ignored */
@@ -3781,6 +3783,7 @@ Possible operations are: · 21 unmodified lines
3781 * ``KVM_S390_MEMOP_ABSOLUTE_WRITE``3783 * ``KVM_S390_MEMOP_ABSOLUTE_WRITE``
3782 * ``KVM_S390_MEMOP_SIDA_READ``3784 * ``KVM_S390_MEMOP_SIDA_READ``
3783 * ``KVM_S390_MEMOP_SIDA_WRITE``3785 * ``KVM_S390_MEMOP_SIDA_WRITE``
3786 * ``KVM_S390_MEMOP_ABSOLUTE_CMPXCHG``
3784 3787
3785Logical read/write:3788Logical read/write:
3786^^^^^^^^^^^^^^^^^^^3789^^^^^^^^^^^^^^^^^^^
@@ -3829,7 +3832,7 @@ the checks required for storage key protection as one operation (as opposed to · 42 unmodified lines
3829user space getting the storage keys, performing the checks, and accessing3832user space getting the storage keys, performing the checks, and accessing
3830memory thereafter, which could lead to a delay between check and access).3833memory thereafter, which could lead to a delay between check and access).
3831Absolute accesses are permitted for the VM ioctl if KVM_CAP_S390_MEM_OP_EXTENSION3834Absolute accesses are permitted for the VM ioctl if KVM_CAP_S390_MEM_OP_EXTENSION
3832is > 0.3835has the KVM_S390_MEMOP_EXTENSION_CAP_BASE bit set.
3833Currently absolute accesses are not permitted for VCPU ioctls.3836Currently absolute accesses are not permitted for VCPU ioctls.
3834Absolute accesses are permitted for non-protected guests only.3837Absolute accesses are permitted for non-protected guests only.
3835 3838
@@ -3837,7 +3840,26 @@ Supported flags: · 1 unmodified line
3837 * ``KVM_S390_MEMOP_F_CHECK_ONLY``3840 * ``KVM_S390_MEMOP_F_CHECK_ONLY``
3838 * ``KVM_S390_MEMOP_F_SKEY_PROTECTION``3841 * ``KVM_S390_MEMOP_F_SKEY_PROTECTION``
3839 3842
3840The semantics of the flags are as for logical accesses.3843The semantics of the flags common with logical accesses are as for logical
3844accesses.
3845
3846Absolute cmpxchg:
3847^^^^^^^^^^^^^^^^^
3848
3849Perform cmpxchg on absolute guest memory. Intended for use with the
3850KVM_S390_MEMOP_F_SKEY_PROTECTION flag.
3851Instead of doing an unconditional write, the access occurs only if the target
3852location contains the value pointed to by "old_addr".
3853This is performed as an atomic cmpxchg with the length specified by the "size"
3854parameter. "size" must be a power of two up to and including 16.
3855If the exchange did not take place because the target value doesn't match the
3856old value, the value "old_addr" points to is replaced by the target value.
3857User space can tell if an exchange took place by checking if this replacement
3858occurred. The cmpxchg op is permitted for the VM ioctl if
3859KVM_CAP_S390_MEM_OP_EXTENSION has flag KVM_S390_MEMOP_EXTENSION_CAP_CMPXCHG set.
3860
3861Supported flags:
3862 * ``KVM_S390_MEMOP_F_SKEY_PROTECTION``
3841 3863
3842SIDA read/write:3864SIDA read/write:
3843^^^^^^^^^^^^^^^^3865^^^^^^^^^^^^^^^^
@@ -4457,6 +4479,18 @@ not holding a previously reported uncorrected error). · 613 unmodified lines
4457:Parameters: struct kvm_s390_cmma_log (in, out)4479:Parameters: struct kvm_s390_cmma_log (in, out)
4458:Returns: 0 on success, a negative value on error4480:Returns: 0 on success, a negative value on error
4459 4481
4482Errors:
4483
4484 ====== =============================================================
4485 ENOMEM not enough memory can be allocated to complete the task
4486 ENXIO if CMMA is not enabled
4487 EINVAL if KVM_S390_CMMA_PEEK is not set but migration mode was not enabled
4488 EINVAL if KVM_S390_CMMA_PEEK is not set but dirty tracking has been
4489 disabled (and thus migration mode was automatically disabled)
4490 EFAULT if the userspace address is invalid or if no page table is
4491 present for the addresses (e.g. when using hugepages).
4492 ====== =============================================================
4493
4460This ioctl is used to get the values of the CMMA bits on the s3904494This ioctl is used to get the values of the CMMA bits on the s390
4461architecture. It is meant to be used in two scenarios:4495architecture. It is meant to be used in two scenarios:
4462 4496
@@ -4537,12 +4571,6 @@ mask is unused. · 74 unmodified lines
4537 4571
4538values points to the userspace buffer where the result will be stored.4572values points to the userspace buffer where the result will be stored.
4539 4573
4540This ioctl can fail with -ENOMEM if not enough memory can be allocated to
4541complete the task, with -ENXIO if CMMA is not enabled, with -EINVAL if
4542KVM_S390_CMMA_PEEK is not set but migration mode was not enabled, with
4543-EFAULT if the userspace address is invalid or if no page table is
4544present for the addresses (e.g. when using hugepages).
4545
45464.108 KVM_S390_SET_CMMA_BITS45744.108 KVM_S390_SET_CMMA_BITS
4547----------------------------4575----------------------------
4548 4576
@@ -5005,6 +5033,15 @@ using this ioctl. · 456 unmodified lines
5005:Parameters: struct kvm_pmu_event_filter (in)5033:Parameters: struct kvm_pmu_event_filter (in)
5006:Returns: 0 on success, -1 on error5034:Returns: 0 on success, -1 on error
5007 5035
5036Errors:
5037
5038 ====== ============================================================
5039 EFAULT args[0] cannot be accessed
5040 EINVAL args[0] contains invalid data in the filter or filter events
5041 E2BIG nevents is too large
5042 EBUSY not enough memory to allocate the filter
5043 ====== ============================================================
5044
5008::5045::
5009 5046
5010 struct kvm_pmu_event_filter {5047 struct kvm_pmu_event_filter {
@@ -5016,14 +5053,69 @@ using this ioctl. · 5 unmodified lines
5016 __u64 events[0];5053 __u64 events[0];
5017 };5054 };
5018 5055
5019This ioctl restricts the set of PMU events that the guest can program.5056This ioctl restricts the set of PMU events the guest can program by limiting
5020The argument holds a list of events which will be allowed or denied.5057which event select and unit mask combinations are permitted.
5021The eventsel+umask of each event the guest attempts to program is compared5058
5022against the events field to determine whether the guest should have access.5059The argument holds a list of filter events which will be allowed or denied.
5023The events field only controls general purpose counters; fixed purpose5060
5024counters are controlled by the fixed_counter_bitmap.5061Filter events only control general purpose counters; fixed purpose counters
5062are controlled by the fixed_counter_bitmap.
5063
5064Valid values for 'flags'::
5065
5066``0``
5067
5068To use this mode, clear the 'flags' field.
5069
5070In this mode each event will contain an event select + unit mask.
5071
5072When the guest attempts to program the PMU the guest's event select +
5073unit mask is compared against the filter events to determine whether the
5074guest should have access.
5075
5076``KVM_PMU_EVENT_FLAG_MASKED_EVENTS``
5077:Capability: KVM_CAP_PMU_EVENT_MASKED_EVENTS
5078
5079In this mode each filter event will contain an event select, mask, match, and
5080exclude value. To encode a masked event use::
5081
5082 KVM_PMU_ENCODE_MASKED_ENTRY()
5083
5084An encoded event will follow this layout::
5085
5086 Bits Description
5087 ---- -----------
5088 7:0 event select (low bits)
5089 15:8 umask match
5090 31:16 unused
5091 35:32 event select (high bits)
5092 36:54 unused
5093 55 exclude bit
5094 63:56 umask mask
5095
5096When the guest attempts to program the PMU, these steps are followed in
5097determining if the guest should have access:
5098
5099 1. Match the event select from the guest against the filter events.
5100 2. If a match is found, match the guest's unit mask to the mask and match
5101 values of the included filter events.
5102 I.e. (unit mask & mask) == match && !exclude.
5103 3. If a match is found, match the guest's unit mask to the mask and match
5104 values of the excluded filter events.
5105 I.e. (unit mask & mask) == match && exclude.
5106 4.
5107 a. If an included match is found and an excluded match is not found, filter
5108 the event.
5109 b. For everything else, do not filter the event.
5110 5.
5111 a. If the event is filtered and it's an allow list, allow the guest to
5112 program the event.
5113 b. If the event is filtered and it's a deny list, do not allow the guest to
5114 program the event.
5025 5115
5026No flags are defined yet, the field must be zero.5116When setting a new pmu event filter, -EINVAL will be returned if any of the
5117unused fields are set or if any of the high bits (35:32) in the event
5118select are set when called on Intel.
5027 5119
5028Valid values for 'action'::5120Valid values for 'action'::
5029 5121
Documentation/virt/kvm/devices/vm.rst+4−0
@@ -302,6 +302,10 @@ Allows userspace to start migration mode, needed for PGSTE migration. · 301 unmodified lines
302Setting this attribute when migration mode is already active will have302Setting this attribute when migration mode is already active will have
303no effects.303no effects.
304 304
305Dirty tracking must be enabled on all memslots, else -EINVAL is returned. When
306dirty tracking is disabled on any memslot, migration mode is automatically
307stopped.
308
305:Parameters: none309:Parameters: none
306:Returns: -ENOMEM if there is not enough free memory to start migration mode;310:Returns: -ENOMEM if there is not enough free memory to start migration mode;
307 -EINVAL if the state of the VM is invalid (e.g. no memory defined);311 -EINVAL if the state of the VM is invalid (e.g. no memory defined);
Documentation/virt/kvm/locking.rst+16−9
@@ -9,6 +9,8 @@ KVM Lock Overview · 8 unmodified lines
9 9
10The acquisition orders for mutexes are as follows:10The acquisition orders for mutexes are as follows:
11 11
12- cpus_read_lock() is taken outside kvm_lock
13
12- kvm->lock is taken outside vcpu->mutex14- kvm->lock is taken outside vcpu->mutex
13 15
14- kvm->lock is taken outside kvm->slots_lock and kvm->irq_lock16- kvm->lock is taken outside kvm->slots_lock and kvm->irq_lock
@@ -226,15 +228,10 @@ time it will be set using the Dirty tracking mechanism described above. · 211 unmodified lines
226:Type: mutex228:Type: mutex
227:Arch: any229:Arch: any
228:Protects: - vm_list230:Protects: - vm_list
229 231 - kvm_usage_count
230``kvm_count_lock``232 - hardware virtualization enable/disable
231^^^^^^^^^^^^^^^^^^233:Comment: KVM also disables CPU hotplug via cpus_read_lock() during
232 234 enable/disable.
233:Type: raw_spinlock_t
234:Arch: any
235:Protects: - hardware virtualization enable/disable
236:Comment: 'raw' because hardware enabling/disabling must be atomic /wrt
237 migration.
238 235
239``kvm->mn_invalidate_lock``236``kvm->mn_invalidate_lock``
240^^^^^^^^^^^^^^^^^^^^^^^^^^^237^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -292,3 +289,13 @@ time it will be set using the Dirty tracking mechanism described above. · 51 unmodified lines
292 wakeup notification event since external interrupts from the289 wakeup notification event since external interrupts from the
293 assigned devices happens, we will find the vCPU on the list to290 assigned devices happens, we will find the vCPU on the list to
294 wakeup.291 wakeup.
292
293``vendor_module_lock``
294^^^^^^^^^^^^^^^^^^^^^^^^^^^^
295:Type: mutex
296:Arch: x86
297:Protects: loading a vendor module (kvm_amd or kvm_intel)
298:Comment: Exists because using kvm_lock leads to deadlock. cpu_hotplug_lock is
299 taken outside of kvm_lock, e.g. in KVM's CPU online/offline callbacks, and
300 many operations need to take cpu_hotplug_lock when loading a vendor module,
301 e.g. updating static calls.
Documentation/virt/kvm/x86/errata.rst+11−0
@@ -37,3 +37,14 @@ Nested virtualization features · 36 unmodified lines
37------------------------------37------------------------------
38 38
39TBD39TBD
40
41x2APIC
42------
43When KVM_X2APIC_API_USE_32BIT_IDS is enabled, KVM activates a hack/quirk that
44allows sending events to a single vCPU using its x2APIC ID even if the target
45vCPU has legacy xAPIC enabled, e.g. to bring up hotplugged vCPUs via INIT-SIPI
46on VMs with > 255 vCPUs. A side effect of the quirk is that, if multiple vCPUs
47have the same physical APIC ID, KVM will deliver events targeting that APIC ID
48only to the vCPU with the lowest vCPU ID. If KVM_X2APIC_API_USE_32BIT_IDS is
49not enabled, KVM follows x86 architecture when processing interrupts (all vCPUs
50matching the target APIC ID receive the interrupt).
MAINTAINERS+1−2
@@ -11269,13 +11269,12 @@ F: virt/kvm/* · 11268 unmodified lines
11269 11269
11270KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)11270KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)
11271M: Marc Zyngier <maz@kernel.org>11271M: Marc Zyngier <maz@kernel.org>
11272M: Oliver Upton <oliver.upton@linux.dev>
11272R: James Morse <james.morse@arm.com>11273R: James Morse <james.morse@arm.com>
11273R: Suzuki K Poulose <suzuki.poulose@arm.com>11274R: Suzuki K Poulose <suzuki.poulose@arm.com>
11274R: Oliver Upton <oliver.upton@linux.dev>
11275R: Zenghui Yu <yuzenghui@huawei.com>11275R: Zenghui Yu <yuzenghui@huawei.com>
11276L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)11276L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
11277L: kvmarm@lists.linux.dev11277L: kvmarm@lists.linux.dev
11278L: kvmarm@lists.cs.columbia.edu (deprecated, moderated for non-subscribers)
11279S: Maintained11278S: Maintained
11280T: git git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git11279T: git git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git
11281F: arch/arm64/include/asm/kvm*11280F: arch/arm64/include/asm/kvm*
arch/arm64/include/asm/cache.h+9−0
@@ -16,6 +16,15 @@ · 15 unmodified lines
16#define CLIDR_LOC(clidr) (((clidr) >> CLIDR_LOC_SHIFT) & 0x7)16#define CLIDR_LOC(clidr) (((clidr) >> CLIDR_LOC_SHIFT) & 0x7)
17#define CLIDR_LOUIS(clidr) (((clidr) >> CLIDR_LOUIS_SHIFT) & 0x7)17#define CLIDR_LOUIS(clidr) (((clidr) >> CLIDR_LOUIS_SHIFT) & 0x7)
18 18
19/* Ctypen, bits[3(n - 1) + 2 : 3(n - 1)], for n = 1 to 7 */
20#define CLIDR_CTYPE_SHIFT(level) (3 * (level - 1))
21#define CLIDR_CTYPE_MASK(level) (7 << CLIDR_CTYPE_SHIFT(level))
22#define CLIDR_CTYPE(clidr, level) \
23 (((clidr) & CLIDR_CTYPE_MASK(level)) >> CLIDR_CTYPE_SHIFT(level))
24
25/* Ttypen, bits [2(n - 1) + 34 : 2(n - 1) + 33], for n = 1 to 7 */
26#define CLIDR_TTYPE_SHIFT(level) (2 * ((level) - 1) + CLIDR_EL1_Ttypen_SHIFT)
27
19/*28/*
20 * Memory returned by kmalloc() may be used for DMA, so we must make29 * Memory returned by kmalloc() may be used for DMA, so we must make
21 * sure that all such allocations are cache aligned. Otherwise,30 * sure that all such allocations are cache aligned. Otherwise,
arch/arm64/include/asm/el2_setup.h+99−0
@@ -196,4 +196,103 @@ · 195 unmodified lines
196 __init_el2_nvhe_prepare_eret196 __init_el2_nvhe_prepare_eret
197.endm197.endm
198 198
199#ifndef __KVM_NVHE_HYPERVISOR__
200// This will clobber tmp1 and tmp2, and expect tmp1 to contain
201// the id register value as read from the HW
202.macro __check_override idreg, fld, width, pass, fail, tmp1, tmp2
203 ubfx \tmp1, \tmp1, #\fld, #\width
204 cbz \tmp1, \fail
205
206 adr_l \tmp1, \idreg\()_override
207 ldr \tmp2, [\tmp1, FTR_OVR_VAL_OFFSET]
208 ldr \tmp1, [\tmp1, FTR_OVR_MASK_OFFSET]
209 ubfx \tmp2, \tmp2, #\fld, #\width
210 ubfx \tmp1, \tmp1, #\fld, #\width
211 cmp \tmp1, xzr
212 and \tmp2, \tmp2, \tmp1
213 csinv \tmp2, \tmp2, xzr, ne
214 cbnz \tmp2, \pass
215 b \fail
216.endm
217
218// This will clobber tmp1 and tmp2
219.macro check_override idreg, fld, pass, fail, tmp1, tmp2
220 mrs \tmp1, \idreg\()_el1
221 __check_override \idreg \fld 4 \pass \fail \tmp1 \tmp2
222.endm
223#else
224// This will clobber tmp
225.macro __check_override idreg, fld, width, pass, fail, tmp, ignore
226 ldr_l \tmp, \idreg\()_el1_sys_val
227 ubfx \tmp, \tmp, #\fld, #\width
228 cbnz \tmp, \pass
229 b \fail
230.endm
231
232.macro check_override idreg, fld, pass, fail, tmp, ignore
233 __check_override \idreg \fld 4 \pass \fail \tmp \ignore
234.endm
235#endif
236
237.macro finalise_el2_state
238 check_override id_aa64pfr0, ID_AA64PFR0_EL1_SVE_SHIFT, .Linit_sve_\@, .Lskip_sve_\@, x1, x2
239
240.Linit_sve_\@: /* SVE register access */
241 mrs x0, cptr_el2 // Disable SVE traps
242 bic x0, x0, #CPTR_EL2_TZ
243 msr cptr_el2, x0
244 isb
245 mov x1, #ZCR_ELx_LEN_MASK // SVE: Enable full vector
246 msr_s SYS_ZCR_EL2, x1 // length for EL1.
247
248.Lskip_sve_\@:
249 check_override id_aa64pfr1, ID_AA64PFR1_EL1_SME_SHIFT, .Linit_sme_\@, .Lskip_sme_\@, x1, x2
250
251.Linit_sme_\@: /* SME register access and priority mapping */
252 mrs x0, cptr_el2 // Disable SME traps
253 bic x0, x0, #CPTR_EL2_TSM
254 msr cptr_el2, x0
255 isb
256
257 mrs x1, sctlr_el2
258 orr x1, x1, #SCTLR_ELx_ENTP2 // Disable TPIDR2 traps
259 msr sctlr_el2, x1
260 isb
261
262 mov x0, #0 // SMCR controls
263
264 // Full FP in SM?
265 mrs_s x1, SYS_ID_AA64SMFR0_EL1
266 __check_override id_aa64smfr0, ID_AA64SMFR0_EL1_FA64_SHIFT, 1, .Linit_sme_fa64_\@, .Lskip_sme_fa64_\@, x1, x2
267
268.Linit_sme_fa64_\@:
269 orr x0, x0, SMCR_ELx_FA64_MASK
270.Lskip_sme_fa64_\@:
271
272 // ZT0 available?
273 mrs_s x1, SYS_ID_AA64SMFR0_EL1
274 __check_override id_aa64smfr0, ID_AA64SMFR0_EL1_SMEver_SHIFT, 4, .Linit_sme_zt0_\@, .Lskip_sme_zt0_\@, x1, x2
275.Linit_sme_zt0_\@:
276 orr x0, x0, SMCR_ELx_EZT0_MASK
277.Lskip_sme_zt0_\@:
278
279 orr x0, x0, #SMCR_ELx_LEN_MASK // Enable full SME vector
280 msr_s SYS_SMCR_EL2, x0 // length for EL1.
281
282 mrs_s x1, SYS_SMIDR_EL1 // Priority mapping supported?
283 ubfx x1, x1, #SMIDR_EL1_SMPS_SHIFT, #1
284 cbz x1, .Lskip_sme_\@
285
286 msr_s SYS_SMPRIMAP_EL2, xzr // Make all priorities equal
287
288 mrs x1, id_aa64mmfr1_el1 // HCRX_EL2 present?
289 ubfx x1, x1, #ID_AA64MMFR1_EL1_HCX_SHIFT, #4
290 cbz x1, .Lskip_sme_\@
291
292 mrs_s x1, SYS_HCRX_EL2
293 orr x1, x1, #HCRX_EL2_SMPME_MASK // Enable priority mapping
294 msr_s SYS_HCRX_EL2, x1
295.Lskip_sme_\@:
296.endm
297
199#endif /* __ARM_KVM_INIT_H__ */298#endif /* __ARM_KVM_INIT_H__ */
arch/arm64/include/asm/esr.h+4−0
@@ -272,6 +272,10 @@ · 271 unmodified lines
272 (((e) & ESR_ELx_SYS64_ISS_OP2_MASK) >> \272 (((e) & ESR_ELx_SYS64_ISS_OP2_MASK) >> \
273 ESR_ELx_SYS64_ISS_OP2_SHIFT))273 ESR_ELx_SYS64_ISS_OP2_SHIFT))
274 274
275/* ISS field definitions for ERET/ERETAA/ERETAB trapping */
276#define ESR_ELx_ERET_ISS_ERET 0x2
277#define ESR_ELx_ERET_ISS_ERETA 0x1
278
275/*279/*
276 * ISS field definitions for floating-point exception traps280 * ISS field definitions for floating-point exception traps
277 * (FP_EXC_32/FP_EXC_64).281 * (FP_EXC_32/FP_EXC_64).
arch/arm64/include/asm/kvm_arm.h+20−3
@@ -81,11 +81,12 @@ · 80 unmodified lines
81 * SWIO: Turn set/way invalidates into set/way clean+invalidate81 * SWIO: Turn set/way invalidates into set/way clean+invalidate
82 * PTW: Take a stage2 fault if a stage1 walk steps in device memory82 * PTW: Take a stage2 fault if a stage1 walk steps in device memory
83 * TID3: Trap EL1 reads of group 3 ID registers83 * TID3: Trap EL1 reads of group 3 ID registers
84 * TID2: Trap CTR_EL0, CCSIDR2_EL1, CLIDR_EL1, and CSSELR_EL1
84 */85 */
85#define HCR_GUEST_FLAGS (HCR_TSC | HCR_TSW | HCR_TWE | HCR_TWI | HCR_VM | \86#define HCR_GUEST_FLAGS (HCR_TSC | HCR_TSW | HCR_TWE | HCR_TWI | HCR_VM | \
86 HCR_BSU_IS | HCR_FB | HCR_TACR | \87 HCR_BSU_IS | HCR_FB | HCR_TACR | \
87 HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \88 HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \
88 HCR_FMO | HCR_IMO | HCR_PTW | HCR_TID3 )89 HCR_FMO | HCR_IMO | HCR_PTW | HCR_TID3 | HCR_TID2)
89#define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF)90#define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF)
90#define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK | HCR_ATA)91#define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK | HCR_ATA)
91#define HCR_HOST_NVHE_PROTECTED_FLAGS (HCR_HOST_NVHE_FLAGS | HCR_TSC)92#define HCR_HOST_NVHE_PROTECTED_FLAGS (HCR_HOST_NVHE_FLAGS | HCR_TSC)
@@ -344,10 +345,26 @@ · 252 unmodified lines
344 ECN(SP_ALIGN), ECN(FP_EXC32), ECN(FP_EXC64), ECN(SERROR), \345 ECN(SP_ALIGN), ECN(FP_EXC32), ECN(FP_EXC64), ECN(SERROR), \
345 ECN(BREAKPT_LOW), ECN(BREAKPT_CUR), ECN(SOFTSTP_LOW), \346 ECN(BREAKPT_LOW), ECN(BREAKPT_CUR), ECN(SOFTSTP_LOW), \
346 ECN(SOFTSTP_CUR), ECN(WATCHPT_LOW), ECN(WATCHPT_CUR), \347 ECN(SOFTSTP_CUR), ECN(WATCHPT_LOW), ECN(WATCHPT_CUR), \
347 ECN(BKPT32), ECN(VECTOR32), ECN(BRK64)348 ECN(BKPT32), ECN(VECTOR32), ECN(BRK64), ECN(ERET)
348 349
349#define CPACR_EL1_TTA (1 << 28)
350#define CPACR_EL1_DEFAULT (CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN |\350#define CPACR_EL1_DEFAULT (CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN |\
351 CPACR_EL1_ZEN_EL1EN)351 CPACR_EL1_ZEN_EL1EN)
352 352
353#define kvm_mode_names \
354 { PSR_MODE_EL0t, "EL0t" }, \
355 { PSR_MODE_EL1t, "EL1t" }, \
356 { PSR_MODE_EL1h, "EL1h" }, \
357 { PSR_MODE_EL2t, "EL2t" }, \
358 { PSR_MODE_EL2h, "EL2h" }, \
359 { PSR_MODE_EL3t, "EL3t" }, \
360 { PSR_MODE_EL3h, "EL3h" }, \
361 { PSR_AA32_MODE_USR, "32-bit USR" }, \
362 { PSR_AA32_MODE_FIQ, "32-bit FIQ" }, \
363 { PSR_AA32_MODE_IRQ, "32-bit IRQ" }, \
364 { PSR_AA32_MODE_SVC, "32-bit SVC" }, \
365 { PSR_AA32_MODE_ABT, "32-bit ABT" }, \
366 { PSR_AA32_MODE_HYP, "32-bit HYP" }, \
367 { PSR_AA32_MODE_UND, "32-bit UND" }, \
368 { PSR_AA32_MODE_SYS, "32-bit SYS" }
369
353#endif /* __ARM64_KVM_ARM_H__ */370#endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/include/asm/kvm_emulate.h+66−4
@@ -33,6 +33,12 @@ enum exception_type { · 32 unmodified lines
33 except_type_serror = 0x180,33 except_type_serror = 0x180,
34};34};
35 35
36#define kvm_exception_type_names \
37 { except_type_sync, "SYNC" }, \
38 { except_type_irq, "IRQ" }, \
39 { except_type_fiq, "FIQ" }, \
40 { except_type_serror, "SERROR" }
41
36bool kvm_condition_valid32(const struct kvm_vcpu *vcpu);42bool kvm_condition_valid32(const struct kvm_vcpu *vcpu);
37void kvm_skip_instr32(struct kvm_vcpu *vcpu);43void kvm_skip_instr32(struct kvm_vcpu *vcpu);
38 44
@@ -44,6 +50,10 @@ void kvm_inject_size_fault(struct kvm_vcpu *vcpu); · 5 unmodified lines
44 50
45void kvm_vcpu_wfi(struct kvm_vcpu *vcpu);51void kvm_vcpu_wfi(struct kvm_vcpu *vcpu);
46 52
53void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu);
54int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2);
55int kvm_inject_nested_irq(struct kvm_vcpu *vcpu);
56
47#if defined(__KVM_VHE_HYPERVISOR__) || defined(__KVM_NVHE_HYPERVISOR__)57#if defined(__KVM_VHE_HYPERVISOR__) || defined(__KVM_NVHE_HYPERVISOR__)
48static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)58static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
49{59{
@@ -88,10 +98,6 @@ static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu) · 38 unmodified lines
88 if (vcpu_el1_is_32bit(vcpu))98 if (vcpu_el1_is_32bit(vcpu))
89 vcpu->arch.hcr_el2 &= ~HCR_RW;99 vcpu->arch.hcr_el2 &= ~HCR_RW;
90 100
91 if (cpus_have_const_cap(ARM64_MISMATCHED_CACHE_TYPE) ||
92 vcpu_el1_is_32bit(vcpu))
93 vcpu->arch.hcr_el2 |= HCR_TID2;
94
95 if (kvm_has_mte(vcpu->kvm))101 if (kvm_has_mte(vcpu->kvm))
96 vcpu->arch.hcr_el2 |= HCR_ATA;102 vcpu->arch.hcr_el2 |= HCR_ATA;
97}103}
@@ -183,6 +189,62 @@ static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · 85 unmodified lines
183 vcpu_gp_regs(vcpu)->regs[reg_num] = val;189 vcpu_gp_regs(vcpu)->regs[reg_num] = val;
184}190}
185 191
192static inline bool vcpu_is_el2_ctxt(const struct kvm_cpu_context *ctxt)
193{
194 switch (ctxt->regs.pstate & (PSR_MODE32_BIT | PSR_MODE_MASK)) {
195 case PSR_MODE_EL2h:
196 case PSR_MODE_EL2t:
197 return true;
198 default:
199 return false;
200 }
201}
202
203static inline bool vcpu_is_el2(const struct kvm_vcpu *vcpu)
204{
205 return vcpu_is_el2_ctxt(&vcpu->arch.ctxt);
206}
207
208static inline bool __vcpu_el2_e2h_is_set(const struct kvm_cpu_context *ctxt)
209{
210 return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_E2H;
211}
212
213static inline bool vcpu_el2_e2h_is_set(const struct kvm_vcpu *vcpu)
214{
215 return __vcpu_el2_e2h_is_set(&vcpu->arch.ctxt);
216}
217
218static inline bool __vcpu_el2_tge_is_set(const struct kvm_cpu_context *ctxt)
219{
220 return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_TGE;
221}
222
223static inline bool vcpu_el2_tge_is_set(const struct kvm_vcpu *vcpu)
224{
225 return __vcpu_el2_tge_is_set(&vcpu->arch.ctxt);
226}
227
228static inline bool __is_hyp_ctxt(const struct kvm_cpu_context *ctxt)
229{
230 /*
231 * We are in a hypervisor context if the vcpu mode is EL2 or
232 * E2H and TGE bits are set. The latter means we are in the user space
233 * of the VHE kernel. ARMv8.1 ARM describes this as 'InHost'
234 *
235 * Note that the HCR_EL2.{E2H,TGE}={0,1} isn't really handled in the
236 * rest of the KVM code, and will result in a misbehaving guest.
237 */
238 return vcpu_is_el2_ctxt(ctxt) ||
239 (__vcpu_el2_e2h_is_set(ctxt) && __vcpu_el2_tge_is_set(ctxt)) ||
240 __vcpu_el2_tge_is_set(ctxt);
241}
242
243static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu)
244{
245 return __is_hyp_ctxt(&vcpu->arch.ctxt);
246}
247
186/*248/*
187 * The layout of SPSR for an AArch32 state is different when observed from an249 * The layout of SPSR for an AArch32 state is different when observed from an
188 * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32250 * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32
arch/arm64/include/asm/kvm_host.h+53−14
@@ -60,14 +60,19 @@ · 59 unmodified lines
60enum kvm_mode {60enum kvm_mode {
61 KVM_MODE_DEFAULT,61 KVM_MODE_DEFAULT,
62 KVM_MODE_PROTECTED,62 KVM_MODE_PROTECTED,
63 KVM_MODE_NV,
63 KVM_MODE_NONE,64 KVM_MODE_NONE,
64};65};
66#ifdef CONFIG_KVM
65enum kvm_mode kvm_get_mode(void);67enum kvm_mode kvm_get_mode(void);
68#else
69static inline enum kvm_mode kvm_get_mode(void) { return KVM_MODE_NONE; };
70#endif
66 71
67DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use);72DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use);
68 73
69extern unsigned int kvm_sve_max_vl;74extern unsigned int __ro_after_init kvm_sve_max_vl;
70int kvm_arm_init_sve(void);75int __init kvm_arm_init_sve(void);
71 76
72u32 __attribute_const__ kvm_target_cpu(void);77u32 __attribute_const__ kvm_target_cpu(void);
73int kvm_reset_vcpu(struct kvm_vcpu *vcpu);78int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
@@ -252,6 +257,7 @@ struct kvm_vcpu_fault_info { · 178 unmodified lines
252enum vcpu_sysreg {257enum vcpu_sysreg {
253 __INVALID_SYSREG__, /* 0 is reserved as an invalid value */258 __INVALID_SYSREG__, /* 0 is reserved as an invalid value */
254 MPIDR_EL1, /* MultiProcessor Affinity Register */259 MPIDR_EL1, /* MultiProcessor Affinity Register */
260 CLIDR_EL1, /* Cache Level ID Register */
255 CSSELR_EL1, /* Cache Size Selection Register */261 CSSELR_EL1, /* Cache Size Selection Register */
256 SCTLR_EL1, /* System Control Register */262 SCTLR_EL1, /* System Control Register */
257 ACTLR_EL1, /* Auxiliary Control Register */263 ACTLR_EL1, /* Auxiliary Control Register */
@@ -320,12 +326,43 @@ enum vcpu_sysreg { · 62 unmodified lines
320 TFSR_EL1, /* Tag Fault Status Register (EL1) */326 TFSR_EL1, /* Tag Fault Status Register (EL1) */
321 TFSRE0_EL1, /* Tag Fault Status Register (EL0) */327 TFSRE0_EL1, /* Tag Fault Status Register (EL0) */
322 328
323 /* 32bit specific registers. Keep them at the end of the range */329 /* 32bit specific registers. */
324 DACR32_EL2, /* Domain Access Control Register */330 DACR32_EL2, /* Domain Access Control Register */
325 IFSR32_EL2, /* Instruction Fault Status Register */331 IFSR32_EL2, /* Instruction Fault Status Register */
326 FPEXC32_EL2, /* Floating-Point Exception Control Register */332 FPEXC32_EL2, /* Floating-Point Exception Control Register */
327 DBGVCR32_EL2, /* Debug Vector Catch Register */333 DBGVCR32_EL2, /* Debug Vector Catch Register */
328 334
335 /* EL2 registers */
336 VPIDR_EL2, /* Virtualization Processor ID Register */
337 VMPIDR_EL2, /* Virtualization Multiprocessor ID Register */
338 SCTLR_EL2, /* System Control Register (EL2) */
339 ACTLR_EL2, /* Auxiliary Control Register (EL2) */
340 HCR_EL2, /* Hypervisor Configuration Register */
341 MDCR_EL2, /* Monitor Debug Configuration Register (EL2) */
342 CPTR_EL2, /* Architectural Feature Trap Register (EL2) */
343 HSTR_EL2, /* Hypervisor System Trap Register */
344 HACR_EL2, /* Hypervisor Auxiliary Control Register */
345 TTBR0_EL2, /* Translation Table Base Register 0 (EL2) */
346 TTBR1_EL2, /* Translation Table Base Register 1 (EL2) */
347 TCR_EL2, /* Translation Control Register (EL2) */
348 VTTBR_EL2, /* Virtualization Translation Table Base Register */
349 VTCR_EL2, /* Virtualization Translation Control Register */
350 SPSR_EL2, /* EL2 saved program status register */
351 ELR_EL2, /* EL2 exception link register */
352 AFSR0_EL2, /* Auxiliary Fault Status Register 0 (EL2) */
353 AFSR1_EL2, /* Auxiliary Fault Status Register 1 (EL2) */
354 ESR_EL2, /* Exception Syndrome Register (EL2) */
355 FAR_EL2, /* Fault Address Register (EL2) */
356 HPFAR_EL2, /* Hypervisor IPA Fault Address Register */
357 MAIR_EL2, /* Memory Attribute Indirection Register (EL2) */
358 AMAIR_EL2, /* Auxiliary Memory Attribute Indirection Register (EL2) */
359 VBAR_EL2, /* Vector Base Address Register (EL2) */
360 RVBAR_EL2, /* Reset Vector Base Address Register */
361 CONTEXTIDR_EL2, /* Context ID Register (EL2) */
362 TPIDR_EL2, /* EL2 Software Thread ID Register */
363 CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */
364 SP_EL2, /* EL2 Stack Pointer */
365
329 NR_SYS_REGS /* Nothing after this line! */366 NR_SYS_REGS /* Nothing after this line! */
330};367};
331 368
@@ -501,6 +538,9 @@ struct kvm_vcpu_arch { · 169 unmodified lines
501 u64 last_steal;538 u64 last_steal;
502 gpa_t base;539 gpa_t base;
503 } steal;540 } steal;
541
542 /* Per-vcpu CCSIDR override or NULL */
543 u32 *ccsidr;
504};544};
505 545
506/*546/*
@@ -598,7 +638,7 @@ struct kvm_vcpu_arch { · 91 unmodified lines
598#define EXCEPT_AA64_EL1_IRQ __vcpu_except_flags(1)638#define EXCEPT_AA64_EL1_IRQ __vcpu_except_flags(1)
599#define EXCEPT_AA64_EL1_FIQ __vcpu_except_flags(2)639#define EXCEPT_AA64_EL1_FIQ __vcpu_except_flags(2)
600#define EXCEPT_AA64_EL1_SERR __vcpu_except_flags(3)640#define EXCEPT_AA64_EL1_SERR __vcpu_except_flags(3)
601/* For AArch64 with NV (one day): */641/* For AArch64 with NV: */
602#define EXCEPT_AA64_EL2_SYNC __vcpu_except_flags(4)642#define EXCEPT_AA64_EL2_SYNC __vcpu_except_flags(4)
603#define EXCEPT_AA64_EL2_IRQ __vcpu_except_flags(5)643#define EXCEPT_AA64_EL2_IRQ __vcpu_except_flags(5)
604#define EXCEPT_AA64_EL2_FIQ __vcpu_except_flags(6)644#define EXCEPT_AA64_EL2_FIQ __vcpu_except_flags(6)
@@ -609,6 +649,8 @@ struct kvm_vcpu_arch { · 4 unmodified lines
609#define DEBUG_STATE_SAVE_SPE __vcpu_single_flag(iflags, BIT(5))649#define DEBUG_STATE_SAVE_SPE __vcpu_single_flag(iflags, BIT(5))
610/* Save TRBE context if active */650/* Save TRBE context if active */
611#define DEBUG_STATE_SAVE_TRBE __vcpu_single_flag(iflags, BIT(6))651#define DEBUG_STATE_SAVE_TRBE __vcpu_single_flag(iflags, BIT(6))
652/* vcpu running in HYP context */
653#define VCPU_HYP_CONTEXT __vcpu_single_flag(iflags, BIT(7))
612 654
613/* SVE enabled for host EL0 */655/* SVE enabled for host EL0 */
614#define HOST_SVE_ENABLED __vcpu_single_flag(sflags, BIT(0))656#define HOST_SVE_ENABLED __vcpu_single_flag(sflags, BIT(0))
@@ -705,7 +747,6 @@ static inline bool __vcpu_read_sys_reg_from_cpu(int reg, u64 *val) · 90 unmodified lines
705 return false;747 return false;
706 748
707 switch (reg) {749 switch (reg) {
708 case CSSELR_EL1: *val = read_sysreg_s(SYS_CSSELR_EL1); break;
709 case SCTLR_EL1: *val = read_sysreg_s(SYS_SCTLR_EL12); break;750 case SCTLR_EL1: *val = read_sysreg_s(SYS_SCTLR_EL12); break;
710 case CPACR_EL1: *val = read_sysreg_s(SYS_CPACR_EL12); break;751 case CPACR_EL1: *val = read_sysreg_s(SYS_CPACR_EL12); break;
711 case TTBR0_EL1: *val = read_sysreg_s(SYS_TTBR0_EL12); break;752 case TTBR0_EL1: *val = read_sysreg_s(SYS_TTBR0_EL12); break;
@@ -750,7 +791,6 @@ static inline bool __vcpu_write_sys_reg_to_cpu(u64 val, int reg) · 38 unmodified lines
750 return false;791 return false;
751 792
752 switch (reg) {793 switch (reg) {
753 case CSSELR_EL1: write_sysreg_s(val, SYS_CSSELR_EL1); break;
754 case SCTLR_EL1: write_sysreg_s(val, SYS_SCTLR_EL12); break;794 case SCTLR_EL1: write_sysreg_s(val, SYS_SCTLR_EL12); break;
755 case CPACR_EL1: write_sysreg_s(val, SYS_CPACR_EL12); break;795 case CPACR_EL1: write_sysreg_s(val, SYS_CPACR_EL12); break;
756 case TTBR0_EL1: write_sysreg_s(val, SYS_TTBR0_EL12); break;796 case TTBR0_EL1: write_sysreg_s(val, SYS_TTBR0_EL12); break;
@@ -877,7 +917,7 @@ int kvm_handle_cp10_id(struct kvm_vcpu *vcpu); · 120 unmodified lines
877 917
878void kvm_reset_sys_regs(struct kvm_vcpu *vcpu);918void kvm_reset_sys_regs(struct kvm_vcpu *vcpu);
879 919
880int kvm_sys_reg_table_init(void);920int __init kvm_sys_reg_table_init(void);
881 921
882/* MMIO helpers */922/* MMIO helpers */
883void kvm_mmio_write_buf(void *buf, unsigned int len, unsigned long data);923void kvm_mmio_write_buf(void *buf, unsigned int len, unsigned long data);
@@ -908,20 +948,20 @@ int kvm_arm_pvtime_get_attr(struct kvm_vcpu *vcpu, · 24 unmodified lines
908int kvm_arm_pvtime_has_attr(struct kvm_vcpu *vcpu,948int kvm_arm_pvtime_has_attr(struct kvm_vcpu *vcpu,
909 struct kvm_device_attr *attr);949 struct kvm_device_attr *attr);
910 950
911extern unsigned int kvm_arm_vmid_bits;951extern unsigned int __ro_after_init kvm_arm_vmid_bits;
912int kvm_arm_vmid_alloc_init(void);952int __init kvm_arm_vmid_alloc_init(void);
913void kvm_arm_vmid_alloc_free(void);953void __init kvm_arm_vmid_alloc_free(void);
914void kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid);954void kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid);
915void kvm_arm_vmid_clear_active(void);955void kvm_arm_vmid_clear_active(void);
916 956
917static inline void kvm_arm_pvtime_vcpu_init(struct kvm_vcpu_arch *vcpu_arch)957static inline void kvm_arm_pvtime_vcpu_init(struct kvm_vcpu_arch *vcpu_arch)
918{958{
919 vcpu_arch->steal.base = GPA_INVALID;959 vcpu_arch->steal.base = INVALID_GPA;
920}960}
921 961
922static inline bool kvm_arm_is_pvtime_enabled(struct kvm_vcpu_arch *vcpu_arch)962static inline bool kvm_arm_is_pvtime_enabled(struct kvm_vcpu_arch *vcpu_arch)
923{963{
924 return (vcpu_arch->steal.base != GPA_INVALID);964 return (vcpu_arch->steal.base != INVALID_GPA);
925}965}
926 966
927void kvm_set_sei_esr(struct kvm_vcpu *vcpu, u64 syndrome);967void kvm_set_sei_esr(struct kvm_vcpu *vcpu, u64 syndrome);
@@ -943,7 +983,6 @@ static inline bool kvm_system_needs_idmapped_vectors(void) · 15 unmodified lines
943 983
944void kvm_arm_vcpu_ptrauth_trap(struct kvm_vcpu *vcpu);984void kvm_arm_vcpu_ptrauth_trap(struct kvm_vcpu *vcpu);
945 985
946static inline void kvm_arch_hardware_unsetup(void) {}
947static inline void kvm_arch_sync_events(struct kvm *kvm) {}986static inline void kvm_arch_sync_events(struct kvm *kvm) {}
948static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}987static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
949 988
@@ -994,7 +1033,7 @@ static inline void kvm_clr_pmu_events(u32 clr) {} · 44 unmodified lines
994void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu);1033void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu);
995void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu);1034void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu);
996 1035
997int kvm_set_ipa_limit(void);1036int __init kvm_set_ipa_limit(void);
998 1037
999#define __KVM_HAVE_ARCH_VM_ALLOC1038#define __KVM_HAVE_ARCH_VM_ALLOC
1000struct kvm *kvm_arch_alloc_vm(void);1039struct kvm *kvm_arch_alloc_vm(void);
arch/arm64/include/asm/kvm_hyp.h+1−0
@@ -122,6 +122,7 @@ extern u64 kvm_nvhe_sym(id_aa64isar2_el1_sys_val); · 121 unmodified lines
122extern u64 kvm_nvhe_sym(id_aa64mmfr0_el1_sys_val);122extern u64 kvm_nvhe_sym(id_aa64mmfr0_el1_sys_val);
123extern u64 kvm_nvhe_sym(id_aa64mmfr1_el1_sys_val);123extern u64 kvm_nvhe_sym(id_aa64mmfr1_el1_sys_val);
124extern u64 kvm_nvhe_sym(id_aa64mmfr2_el1_sys_val);124extern u64 kvm_nvhe_sym(id_aa64mmfr2_el1_sys_val);
125extern u64 kvm_nvhe_sym(id_aa64smfr0_el1_sys_val);
125 126
126extern unsigned long kvm_nvhe_sym(__icache_flags);127extern unsigned long kvm_nvhe_sym(__icache_flags);
127extern unsigned int kvm_nvhe_sym(kvm_arm_vmid_bits);128extern unsigned int kvm_nvhe_sym(kvm_arm_vmid_bits);
arch/arm64/include/asm/kvm_mmu.h+12−3
@@ -115,6 +115,7 @@ alternative_cb_end · 114 unmodified lines
115#include <asm/cache.h>115#include <asm/cache.h>
116#include <asm/cacheflush.h>116#include <asm/cacheflush.h>
117#include <asm/mmu_context.h>117#include <asm/mmu_context.h>
118#include <asm/kvm_emulate.h>
118#include <asm/kvm_host.h>119#include <asm/kvm_host.h>
119 120
120void kvm_update_va_mask(struct alt_instr *alt,121void kvm_update_va_mask(struct alt_instr *alt,
@@ -163,7 +164,7 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size, · 42 unmodified lines
163 void __iomem **haddr);164 void __iomem **haddr);
164int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,165int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
165 void **haddr);166 void **haddr);
166void free_hyp_pgds(void);167void __init free_hyp_pgds(void);
167 168
168void stage2_unmap_vm(struct kvm *kvm);169void stage2_unmap_vm(struct kvm *kvm);
169int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long type);170int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long type);
@@ -175,7 +176,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu); · 5 unmodified lines
175 176
176phys_addr_t kvm_mmu_get_httbr(void);177phys_addr_t kvm_mmu_get_httbr(void);
177phys_addr_t kvm_get_idmap_vector(void);178phys_addr_t kvm_get_idmap_vector(void);
178int kvm_mmu_init(u32 *hyp_va_bits);179int __init kvm_mmu_init(u32 *hyp_va_bits);
179 180
180static inline void *__kvm_vector_slot2addr(void *base,181static inline void *__kvm_vector_slot2addr(void *base,
181 enum arm64_hyp_spectre_vector slot)182 enum arm64_hyp_spectre_vector slot)
@@ -192,7 +193,15 @@ struct kvm; · 10 unmodified lines
192 193
193static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu)194static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu)
194{195{
195 return (vcpu_read_sys_reg(vcpu, SCTLR_EL1) & 0b101) == 0b101;196 u64 cache_bits = SCTLR_ELx_M | SCTLR_ELx_C;
197 int reg;
198
199 if (vcpu_is_el2(vcpu))
200 reg = SCTLR_EL2;
201 else
202 reg = SCTLR_EL1;
203
204 return (vcpu_read_sys_reg(vcpu, reg) & cache_bits) == cache_bits;
196}205}
197 206
198static inline void __clean_dcache_guest_page(void *va, size_t size)207static inline void __clean_dcache_guest_page(void *va, size_t size)
arch/arm64/include/asm/kvm_nested.h+20−0
@@ -0,0 +1,20 @@
1/* SPDX-License-Identifier: GPL-2.0 */
2#ifndef __ARM64_KVM_NESTED_H
3#define __ARM64_KVM_NESTED_H
4
5#include <linux/kvm_host.h>
6
7static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu)
8{
9 return (!__is_defined(__KVM_NVHE_HYPERVISOR__) &&
10 cpus_have_final_cap(ARM64_HAS_NESTED_VIRT) &&
11 test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features));
12}
13
14struct sys_reg_params;
15struct sys_reg_desc;
16
17void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
18 const struct sys_reg_desc *r);
19
20#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/include/asm/kvm_pgtable.h+8−0
@@ -71,6 +71,11 @@ static inline kvm_pte_t kvm_phys_to_pte(u64 pa) · 70 unmodified lines
71 return pte;71 return pte;
72}72}
73 73
74static inline kvm_pfn_t kvm_pte_to_pfn(kvm_pte_t pte)
75{
76 return __phys_to_pfn(kvm_pte_to_phys(pte));
77}
78
74static inline u64 kvm_granule_shift(u32 level)79static inline u64 kvm_granule_shift(u32 level)
75{80{
76 /* Assumes KVM_PGTABLE_MAX_LEVELS is 4 */81 /* Assumes KVM_PGTABLE_MAX_LEVELS is 4 */
@@ -188,12 +193,15 @@ typedef bool (*kvm_pgtable_force_pte_cb_t)(u64 addr, u64 end, · 111 unmodified lines
188 * children.193 * children.
189 * @KVM_PGTABLE_WALK_SHARED: Indicates the page-tables may be shared194 * @KVM_PGTABLE_WALK_SHARED: Indicates the page-tables may be shared
190 * with other software walkers.195 * with other software walkers.
196 * @KVM_PGTABLE_WALK_HANDLE_FAULT: Indicates the page-table walk was
197 * invoked from a fault handler.
191 */198 */
192enum kvm_pgtable_walk_flags {199enum kvm_pgtable_walk_flags {
193 KVM_PGTABLE_WALK_LEAF = BIT(0),200 KVM_PGTABLE_WALK_LEAF = BIT(0),
194 KVM_PGTABLE_WALK_TABLE_PRE = BIT(1),201 KVM_PGTABLE_WALK_TABLE_PRE = BIT(1),
195 KVM_PGTABLE_WALK_TABLE_POST = BIT(2),202 KVM_PGTABLE_WALK_TABLE_POST = BIT(2),
196 KVM_PGTABLE_WALK_SHARED = BIT(3),203 KVM_PGTABLE_WALK_SHARED = BIT(3),
204 KVM_PGTABLE_WALK_HANDLE_FAULT = BIT(4),
197};205};
198 206
199struct kvm_pgtable_visit_ctx {207struct kvm_pgtable_visit_ctx {
arch/arm64/include/asm/sysreg.h+37−2
@@ -325,7 +325,6 @@ · 324 unmodified lines
325 325
326#define SYS_CNTKCTL_EL1 sys_reg(3, 0, 14, 1, 0)326#define SYS_CNTKCTL_EL1 sys_reg(3, 0, 14, 1, 0)
327 327
328#define SYS_CCSIDR_EL1 sys_reg(3, 1, 0, 0, 0)
329#define SYS_AIDR_EL1 sys_reg(3, 1, 0, 0, 7)328#define SYS_AIDR_EL1 sys_reg(3, 1, 0, 0, 7)
330 329
331#define SYS_RNDR_EL0 sys_reg(3, 3, 2, 4, 0)330#define SYS_RNDR_EL0 sys_reg(3, 3, 2, 4, 0)
@@ -411,23 +410,51 @@ · 79 unmodified lines
411 410
412#define SYS_PMCCFILTR_EL0 sys_reg(3, 3, 14, 15, 7)411#define SYS_PMCCFILTR_EL0 sys_reg(3, 3, 14, 15, 7)
413 412
413#define SYS_VPIDR_EL2 sys_reg(3, 4, 0, 0, 0)
414#define SYS_VMPIDR_EL2 sys_reg(3, 4, 0, 0, 5)
415
414#define SYS_SCTLR_EL2 sys_reg(3, 4, 1, 0, 0)416#define SYS_SCTLR_EL2 sys_reg(3, 4, 1, 0, 0)
417#define SYS_ACTLR_EL2 sys_reg(3, 4, 1, 0, 1)
418#define SYS_HCR_EL2 sys_reg(3, 4, 1, 1, 0)
419#define SYS_MDCR_EL2 sys_reg(3, 4, 1, 1, 1)
420#define SYS_CPTR_EL2 sys_reg(3, 4, 1, 1, 2)
421#define SYS_HSTR_EL2 sys_reg(3, 4, 1, 1, 3)
415#define SYS_HFGRTR_EL2 sys_reg(3, 4, 1, 1, 4)422#define SYS_HFGRTR_EL2 sys_reg(3, 4, 1, 1, 4)
416#define SYS_HFGWTR_EL2 sys_reg(3, 4, 1, 1, 5)423#define SYS_HFGWTR_EL2 sys_reg(3, 4, 1, 1, 5)
417#define SYS_HFGITR_EL2 sys_reg(3, 4, 1, 1, 6)424#define SYS_HFGITR_EL2 sys_reg(3, 4, 1, 1, 6)
425#define SYS_HACR_EL2 sys_reg(3, 4, 1, 1, 7)
426
427#define SYS_TTBR0_EL2 sys_reg(3, 4, 2, 0, 0)
428#define SYS_TTBR1_EL2 sys_reg(3, 4, 2, 0, 1)
429#define SYS_TCR_EL2 sys_reg(3, 4, 2, 0, 2)
430#define SYS_VTTBR_EL2 sys_reg(3, 4, 2, 1, 0)
431#define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2)
432
418#define SYS_TRFCR_EL2 sys_reg(3, 4, 1, 2, 1)433#define SYS_TRFCR_EL2 sys_reg(3, 4, 1, 2, 1)
419#define SYS_HDFGRTR_EL2 sys_reg(3, 4, 3, 1, 4)434#define SYS_HDFGRTR_EL2 sys_reg(3, 4, 3, 1, 4)
420#define SYS_HDFGWTR_EL2 sys_reg(3, 4, 3, 1, 5)435#define SYS_HDFGWTR_EL2 sys_reg(3, 4, 3, 1, 5)
421#define SYS_HAFGRTR_EL2 sys_reg(3, 4, 3, 1, 6)436#define SYS_HAFGRTR_EL2 sys_reg(3, 4, 3, 1, 6)
422#define SYS_SPSR_EL2 sys_reg(3, 4, 4, 0, 0)437#define SYS_SPSR_EL2 sys_reg(3, 4, 4, 0, 0)
423#define SYS_ELR_EL2 sys_reg(3, 4, 4, 0, 1)438#define SYS_ELR_EL2 sys_reg(3, 4, 4, 0, 1)
439#define SYS_SP_EL1 sys_reg(3, 4, 4, 1, 0)
424#define SYS_IFSR32_EL2 sys_reg(3, 4, 5, 0, 1)440#define SYS_IFSR32_EL2 sys_reg(3, 4, 5, 0, 1)
441#define SYS_AFSR0_EL2 sys_reg(3, 4, 5, 1, 0)
442#define SYS_AFSR1_EL2 sys_reg(3, 4, 5, 1, 1)
425#define SYS_ESR_EL2 sys_reg(3, 4, 5, 2, 0)443#define SYS_ESR_EL2 sys_reg(3, 4, 5, 2, 0)
426#define SYS_VSESR_EL2 sys_reg(3, 4, 5, 2, 3)444#define SYS_VSESR_EL2 sys_reg(3, 4, 5, 2, 3)
427#define SYS_FPEXC32_EL2 sys_reg(3, 4, 5, 3, 0)445#define SYS_FPEXC32_EL2 sys_reg(3, 4, 5, 3, 0)
428#define SYS_TFSR_EL2 sys_reg(3, 4, 5, 6, 0)446#define SYS_TFSR_EL2 sys_reg(3, 4, 5, 6, 0)
429 447
430#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1)448#define SYS_FAR_EL2 sys_reg(3, 4, 6, 0, 0)
449#define SYS_HPFAR_EL2 sys_reg(3, 4, 6, 0, 4)
450
451#define SYS_MAIR_EL2 sys_reg(3, 4, 10, 2, 0)
452#define SYS_AMAIR_EL2 sys_reg(3, 4, 10, 3, 0)
453
454#define SYS_VBAR_EL2 sys_reg(3, 4, 12, 0, 0)
455#define SYS_RVBAR_EL2 sys_reg(3, 4, 12, 0, 1)
456#define SYS_RMR_EL2 sys_reg(3, 4, 12, 0, 2)
457#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1)
431#define __SYS__AP0Rx_EL2(x) sys_reg(3, 4, 12, 8, x)458#define __SYS__AP0Rx_EL2(x) sys_reg(3, 4, 12, 8, x)
432#define SYS_ICH_AP0R0_EL2 __SYS__AP0Rx_EL2(0)459#define SYS_ICH_AP0R0_EL2 __SYS__AP0Rx_EL2(0)
433#define SYS_ICH_AP0R1_EL2 __SYS__AP0Rx_EL2(1)460#define SYS_ICH_AP0R1_EL2 __SYS__AP0Rx_EL2(1)
@@ -469,6 +496,12 @@ · 35 unmodified lines
469#define SYS_ICH_LR14_EL2 __SYS__LR8_EL2(6)496#define SYS_ICH_LR14_EL2 __SYS__LR8_EL2(6)
470#define SYS_ICH_LR15_EL2 __SYS__LR8_EL2(7)497#define SYS_ICH_LR15_EL2 __SYS__LR8_EL2(7)
471 498
499#define SYS_CONTEXTIDR_EL2 sys_reg(3, 4, 13, 0, 1)
500#define SYS_TPIDR_EL2 sys_reg(3, 4, 13, 0, 2)
501
502#define SYS_CNTVOFF_EL2 sys_reg(3, 4, 14, 0, 3)
503#define SYS_CNTHCTL_EL2 sys_reg(3, 4, 14, 1, 0)
504
472/* VHE encodings for architectural EL0/1 system registers */505/* VHE encodings for architectural EL0/1 system registers */
473#define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0)506#define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0)
474#define SYS_TTBR0_EL12 sys_reg(3, 5, 2, 0, 0)507#define SYS_TTBR0_EL12 sys_reg(3, 5, 2, 0, 0)
@@ -491,6 +524,8 @@ · 16 unmodified lines
491#define SYS_CNTV_CTL_EL02 sys_reg(3, 5, 14, 3, 1)524#define SYS_CNTV_CTL_EL02 sys_reg(3, 5, 14, 3, 1)
492#define SYS_CNTV_CVAL_EL02 sys_reg(3, 5, 14, 3, 2)525#define SYS_CNTV_CVAL_EL02 sys_reg(3, 5, 14, 3, 2)
493 526
527#define SYS_SP_EL2 sys_reg(3, 6, 4, 1, 0)
528
494/* Common SCTLR_ELx flags. */529/* Common SCTLR_ELx flags. */
495#define SCTLR_ELx_ENTP2 (BIT(60))530#define SCTLR_ELx_ENTP2 (BIT(60))
496#define SCTLR_ELx_DSSBS (BIT(44))531#define SCTLR_ELx_DSSBS (BIT(44))
arch/arm64/include/uapi/asm/kvm.h+1−0
@@ -109,6 +109,7 @@ struct kvm_regs { · 108 unmodified lines
109#define KVM_ARM_VCPU_SVE 4 /* enable SVE for this CPU */109#define KVM_ARM_VCPU_SVE 4 /* enable SVE for this CPU */
110#define KVM_ARM_VCPU_PTRAUTH_ADDRESS 5 /* VCPU uses address authentication */110#define KVM_ARM_VCPU_PTRAUTH_ADDRESS 5 /* VCPU uses address authentication */
111#define KVM_ARM_VCPU_PTRAUTH_GENERIC 6 /* VCPU uses generic authentication */111#define KVM_ARM_VCPU_PTRAUTH_GENERIC 6 /* VCPU uses generic authentication */
112#define KVM_ARM_VCPU_HAS_EL2 7 /* Support nested virtualization */
112 113
113struct kvm_vcpu_init {114struct kvm_vcpu_init {
114 __u32 target;115 __u32 target;
arch/arm64/kernel/cacheinfo.c+0−5
@@ -11,11 +11,6 @@ · 10 unmodified lines
11#include <linux/of.h>11#include <linux/of.h>
12 12
13#define MAX_CACHE_LEVEL 7 /* Max 7 level supported */13#define MAX_CACHE_LEVEL 7 /* Max 7 level supported */
14/* Ctypen, bits[3(n - 1) + 2 : 3(n - 1)], for n = 1 to 7 */
15#define CLIDR_CTYPE_SHIFT(level) (3 * (level - 1))
16#define CLIDR_CTYPE_MASK(level) (7 << CLIDR_CTYPE_SHIFT(level))
17#define CLIDR_CTYPE(clidr, level) \
18 (((clidr) & CLIDR_CTYPE_MASK(level)) >> CLIDR_CTYPE_SHIFT(level))
19 14
20int cache_line_size(void)15int cache_line_size(void)
21{16{
arch/arm64/kernel/cpufeature.c+25−0
@@ -1967,6 +1967,20 @@ static void cpu_copy_el2regs(const struct arm64_cpu_capabilities *__unused) · 1966 unmodified lines
1967 write_sysreg(read_sysreg(tpidr_el1), tpidr_el2);1967 write_sysreg(read_sysreg(tpidr_el1), tpidr_el2);
1968}1968}
1969 1969
1970static bool has_nested_virt_support(const struct arm64_cpu_capabilities *cap,
1971 int scope)
1972{
1973 if (kvm_get_mode() != KVM_MODE_NV)
1974 return false;
1975
1976 if (!has_cpuid_feature(cap, scope)) {
1977 pr_warn("unavailable: %s\n", cap->desc);
1978 return false;
1979 }
1980
1981 return true;
1982}
1983
1970#ifdef CONFIG_ARM64_PAN1984#ifdef CONFIG_ARM64_PAN
1971static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused)1985static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused)
1972{1986{
@@ -2262,6 +2276,17 @@ static const struct arm64_cpu_capabilities arm64_features[] = { · 289 unmodified lines
2262 .matches = runs_at_el2,2276 .matches = runs_at_el2,
2263 .cpu_enable = cpu_copy_el2regs,2277 .cpu_enable = cpu_copy_el2regs,
2264 },2278 },
2279 {
2280 .desc = "Nested Virtualization Support",
2281 .capability = ARM64_HAS_NESTED_VIRT,
2282 .type = ARM64_CPUCAP_SYSTEM_FEATURE,
2283 .matches = has_nested_virt_support,
2284 .sys_reg = SYS_ID_AA64MMFR2_EL1,
2285 .sign = FTR_UNSIGNED,
2286 .field_pos = ID_AA64MMFR2_EL1_NV_SHIFT,
2287 .field_width = 4,
2288 .min_field_value = ID_AA64MMFR2_EL1_NV_IMP,
2289 },
2265 {2290 {
2266 .capability = ARM64_HAS_32BIT_EL0_DO_NOT_USE,2291 .capability = ARM64_HAS_32BIT_EL0_DO_NOT_USE,
2267 .type = ARM64_CPUCAP_SYSTEM_FEATURE,2292 .type = ARM64_CPUCAP_SYSTEM_FEATURE,
arch/arm64/kernel/hyp-stub.S+2−84
@@ -16,30 +16,6 @@ · 15 unmodified lines
16#include <asm/ptrace.h>16#include <asm/ptrace.h>
17#include <asm/virt.h>17#include <asm/virt.h>
18 18
19// Warning, hardcoded register allocation
20// This will clobber x1 and x2, and expect x1 to contain
21// the id register value as read from the HW
22.macro __check_override idreg, fld, width, pass, fail
23 ubfx x1, x1, #\fld, #\width
24 cbz x1, \fail
25
26 adr_l x1, \idreg\()_override
27 ldr x2, [x1, FTR_OVR_VAL_OFFSET]
28 ldr x1, [x1, FTR_OVR_MASK_OFFSET]
29 ubfx x2, x2, #\fld, #\width
30 ubfx x1, x1, #\fld, #\width
31 cmp x1, xzr
32 and x2, x2, x1
33 csinv x2, x2, xzr, ne
34 cbnz x2, \pass
35 b \fail
36.endm
37
38.macro check_override idreg, fld, pass, fail
39 mrs x1, \idreg\()_el1
40 __check_override \idreg \fld 4 \pass \fail
41.endm
42
43 .text19 .text
44 .pushsection .hyp.text, "ax"20 .pushsection .hyp.text, "ax"
45 21
@@ -98,65 +74,7 @@ SYM_CODE_START_LOCAL(elx_sync) · 52 unmodified lines
98SYM_CODE_END(elx_sync)74SYM_CODE_END(elx_sync)
99 75
100SYM_CODE_START_LOCAL(__finalise_el2)76SYM_CODE_START_LOCAL(__finalise_el2)
101 check_override id_aa64pfr0 ID_AA64PFR0_EL1_SVE_SHIFT .Linit_sve .Lskip_sve77 finalise_el2_state
102
103.Linit_sve: /* SVE register access */
104 mrs x0, cptr_el2 // Disable SVE traps
105 bic x0, x0, #CPTR_EL2_TZ
106 msr cptr_el2, x0
107 isb
108 mov x1, #ZCR_ELx_LEN_MASK // SVE: Enable full vector
109 msr_s SYS_ZCR_EL2, x1 // length for EL1.
110
111.Lskip_sve:
112 check_override id_aa64pfr1 ID_AA64PFR1_EL1_SME_SHIFT .Linit_sme .Lskip_sme
113
114.Linit_sme: /* SME register access and priority mapping */
115 mrs x0, cptr_el2 // Disable SME traps
116 bic x0, x0, #CPTR_EL2_TSM
117 msr cptr_el2, x0
118 isb
119
120 mrs x1, sctlr_el2
121 orr x1, x1, #SCTLR_ELx_ENTP2 // Disable TPIDR2 traps
122 msr sctlr_el2, x1
123 isb
124
125 mov x0, #0 // SMCR controls
126
127 // Full FP in SM?
128 mrs_s x1, SYS_ID_AA64SMFR0_EL1
129 __check_override id_aa64smfr0 ID_AA64SMFR0_EL1_FA64_SHIFT 1 .Linit_sme_fa64 .Lskip_sme_fa64
130
131.Linit_sme_fa64:
132 orr x0, x0, SMCR_ELx_FA64_MASK
133.Lskip_sme_fa64:
134
135 // ZT0 available?
136 mrs_s x1, SYS_ID_AA64SMFR0_EL1
137 __check_override id_aa64smfr0 ID_AA64SMFR0_EL1_SMEver_SHIFT 4 .Linit_sme_zt0 .Lskip_sme_zt0
138.Linit_sme_zt0:
139 orr x0, x0, SMCR_ELx_EZT0_MASK
140.Lskip_sme_zt0:
141
142 orr x0, x0, #SMCR_ELx_LEN_MASK // Enable full SME vector
143 msr_s SYS_SMCR_EL2, x0 // length for EL1.
144
145 mrs_s x1, SYS_SMIDR_EL1 // Priority mapping supported?
146 ubfx x1, x1, #SMIDR_EL1_SMPS_SHIFT, #1
147 cbz x1, .Lskip_sme
148
149 msr_s SYS_SMPRIMAP_EL2, xzr // Make all priorities equal
150
151 mrs x1, id_aa64mmfr1_el1 // HCRX_EL2 present?
152 ubfx x1, x1, #ID_AA64MMFR1_EL1_HCX_SHIFT, #4
153 cbz x1, .Lskip_sme
154
155 mrs_s x1, SYS_HCRX_EL2
156 orr x1, x1, #HCRX_EL2_SMPME_MASK // Enable priority mapping
157 msr_s SYS_HCRX_EL2, x1
158
159.Lskip_sme:
160 78
161 // nVHE? No way! Give me the real thing!79 // nVHE? No way! Give me the real thing!
162 // Sanity check: MMU *must* be off80 // Sanity check: MMU *must* be off
@@ -164,7 +82,7 @@ SYM_CODE_START_LOCAL(__finalise_el2) · 1 unmodified line
164 tbnz x1, #0, 1f82 tbnz x1, #0, 1f
165 83
166 // Needs to be VHE capable, obviously84 // Needs to be VHE capable, obviously
167 check_override id_aa64mmfr1 ID_AA64MMFR1_EL1_VH_SHIFT 2f 1f85 check_override id_aa64mmfr1 ID_AA64MMFR1_EL1_VH_SHIFT 2f 1f x1 x2
168 86
1691: mov_q x0, HVC_STUB_ERR871: mov_q x0, HVC_STUB_ERR
170 eret88 eret
arch/arm64/kvm/Kconfig+1−0
@@ -21,6 +21,7 @@ if VIRTUALIZATION · 20 unmodified lines
21menuconfig KVM21menuconfig KVM
22 bool "Kernel-based Virtual Machine (KVM) support"22 bool "Kernel-based Virtual Machine (KVM) support"
23 depends on HAVE_KVM23 depends on HAVE_KVM
24 select KVM_GENERIC_HARDWARE_ENABLING
24 select MMU_NOTIFIER25 select MMU_NOTIFIER
25 select PREEMPT_NOTIFIERS26 select PREEMPT_NOTIFIERS
26 select HAVE_KVM_CPU_RELAX_INTERCEPT27 select HAVE_KVM_CPU_RELAX_INTERCEPT
arch/arm64/kvm/Makefile+1−1
@@ -14,7 +14,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ · 13 unmodified lines
14 inject_fault.o va_layout.o handle_exit.o \14 inject_fault.o va_layout.o handle_exit.o \
15 guest.o debug.o reset.o sys_regs.o stacktrace.o \15 guest.o debug.o reset.o sys_regs.o stacktrace.o \
16 vgic-sys-reg-v3.o fpsimd.o pkvm.o \16 vgic-sys-reg-v3.o fpsimd.o pkvm.o \
17 arch_timer.o trng.o vmid.o \17 arch_timer.o trng.o vmid.o emulate-nested.o nested.o \
18 vgic/vgic.o vgic/vgic-init.o \18 vgic/vgic.o vgic/vgic-init.o \
19 vgic/vgic-irqfd.o vgic/vgic-v2.o \19 vgic/vgic-irqfd.o vgic/vgic-v2.o \
20 vgic/vgic-v3.o vgic/vgic-v4.o \20 vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/arch_timer.c+60−46
@@ -428,14 +428,17 @@ static void timer_emulate(struct arch_timer_context *ctx) · 427 unmodified lines
428 * scheduled for the future. If the timer cannot fire at all,428 * scheduled for the future. If the timer cannot fire at all,
429 * then we also don't need a soft timer.429 * then we also don't need a soft timer.
430 */430 */
431 if (!kvm_timer_irq_can_fire(ctx)) {431 if (should_fire || !kvm_timer_irq_can_fire(ctx))
432 soft_timer_cancel(&ctx->hrtimer);
433 return;432 return;
434 }
435 433
436 soft_timer_start(&ctx->hrtimer, kvm_timer_compute_delta(ctx));434 soft_timer_start(&ctx->hrtimer, kvm_timer_compute_delta(ctx));
437}435}
438 436
437static void set_cntvoff(u64 cntvoff)
438{
439 kvm_call_hyp(__kvm_timer_set_cntvoff, cntvoff);
440}
441
439static void timer_save_state(struct arch_timer_context *ctx)442static void timer_save_state(struct arch_timer_context *ctx)
440{443{
441 struct arch_timer_cpu *timer = vcpu_timer(ctx->vcpu);444 struct arch_timer_cpu *timer = vcpu_timer(ctx->vcpu);
@@ -459,6 +462,22 @@ static void timer_save_state(struct arch_timer_context *ctx) · 17 unmodified lines
459 write_sysreg_el0(0, SYS_CNTV_CTL);462 write_sysreg_el0(0, SYS_CNTV_CTL);
460 isb();463 isb();
461 464
465 /*
466 * The kernel may decide to run userspace after
467 * calling vcpu_put, so we reset cntvoff to 0 to
468 * ensure a consistent read between user accesses to
469 * the virtual counter and kernel access to the
470 * physical counter of non-VHE case.
471 *
472 * For VHE, the virtual counter uses a fixed virtual
473 * offset of zero, so no need to zero CNTVOFF_EL2
474 * register, but this is actually useful when switching
475 * between EL1/vEL2 with NV.
476 *
477 * Do it unconditionally, as this is either unavoidable
478 * or dirt cheap.
479 */
480 set_cntvoff(0);
462 break;481 break;
463 case TIMER_PTIMER:482 case TIMER_PTIMER:
464 timer_set_ctl(ctx, read_sysreg_el0(SYS_CNTP_CTL));483 timer_set_ctl(ctx, read_sysreg_el0(SYS_CNTP_CTL));
@@ -532,6 +551,7 @@ static void timer_restore_state(struct arch_timer_context *ctx) · 67 unmodified lines
532 551
533 switch (index) {552 switch (index) {
534 case TIMER_VTIMER:553 case TIMER_VTIMER:
554 set_cntvoff(timer_get_offset(ctx));
535 write_sysreg_el0(timer_get_cval(ctx), SYS_CNTV_CVAL);555 write_sysreg_el0(timer_get_cval(ctx), SYS_CNTV_CVAL);
536 isb();556 isb();
537 write_sysreg_el0(timer_get_ctl(ctx), SYS_CNTV_CTL);557 write_sysreg_el0(timer_get_ctl(ctx), SYS_CNTV_CTL);
@@ -552,11 +572,6 @@ out: · 14 unmodified lines
552 local_irq_restore(flags);572 local_irq_restore(flags);
553}573}
554 574
555static void set_cntvoff(u64 cntvoff)
556{
557 kvm_call_hyp(__kvm_timer_set_cntvoff, cntvoff);
558}
559
560static inline void set_timer_irq_phys_active(struct arch_timer_context *ctx, bool active)575static inline void set_timer_irq_phys_active(struct arch_timer_context *ctx, bool active)
561{576{
562 int r;577 int r;
@@ -631,8 +646,6 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) · 68 unmodified lines
631 kvm_timer_vcpu_load_nogic(vcpu);646 kvm_timer_vcpu_load_nogic(vcpu);
632 }647 }
633 648
634 set_cntvoff(timer_get_offset(map.direct_vtimer));
635
636 kvm_timer_unblocking(vcpu);649 kvm_timer_unblocking(vcpu);
637 650
638 timer_restore_state(map.direct_vtimer);651 timer_restore_state(map.direct_vtimer);
@@ -688,15 +701,6 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu) · 49 unmodified lines
688 701
689 if (kvm_vcpu_is_blocking(vcpu))702 if (kvm_vcpu_is_blocking(vcpu))
690 kvm_timer_blocking(vcpu);703 kvm_timer_blocking(vcpu);
691
692 /*
693 * The kernel may decide to run userspace after calling vcpu_put, so
694 * we reset cntvoff to 0 to ensure a consistent read between user
695 * accesses to the virtual counter and kernel access to the physical
696 * counter of non-VHE case. For VHE, the virtual counter uses a fixed
697 * virtual offset of zero, so no need to zero CNTVOFF_EL2 register.
698 */
699 set_cntvoff(0);
700}704}
701 705
702/*706/*
@@ -811,10 +815,18 @@ void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu) · 108 unmodified lines
811 ptimer->host_timer_irq_flags = host_ptimer_irq_flags;815 ptimer->host_timer_irq_flags = host_ptimer_irq_flags;
812}816}
813 817
814static void kvm_timer_init_interrupt(void *info)818void kvm_timer_cpu_up(void)
815{819{
816 enable_percpu_irq(host_vtimer_irq, host_vtimer_irq_flags);820 enable_percpu_irq(host_vtimer_irq, host_vtimer_irq_flags);
817 enable_percpu_irq(host_ptimer_irq, host_ptimer_irq_flags);821 if (host_ptimer_irq)
822 enable_percpu_irq(host_ptimer_irq, host_ptimer_irq_flags);
823}
824
825void kvm_timer_cpu_down(void)
826{
827 disable_percpu_irq(host_vtimer_irq);
828 if (host_ptimer_irq)
829 disable_percpu_irq(host_ptimer_irq);
818}830}
819 831
820int kvm_arm_timer_set_reg(struct kvm_vcpu *vcpu, u64 regid, u64 value)832int kvm_arm_timer_set_reg(struct kvm_vcpu *vcpu, u64 regid, u64 value)
@@ -926,14 +938,22 @@ u64 kvm_arm_timer_read_sysreg(struct kvm_vcpu *vcpu, · 105 unmodified lines
926 enum kvm_arch_timers tmr,938 enum kvm_arch_timers tmr,
927 enum kvm_arch_timer_regs treg)939 enum kvm_arch_timer_regs treg)
928{940{
941 struct arch_timer_context *timer;
942 struct timer_map map;
929 u64 val;943 u64 val;
930 944
945 get_timer_map(vcpu, &map);
946 timer = vcpu_get_timer(vcpu, tmr);
947
948 if (timer == map.emul_ptimer)
949 return kvm_arm_timer_read(vcpu, timer, treg);
950
931 preempt_disable();951 preempt_disable();
932 kvm_timer_vcpu_put(vcpu);952 timer_save_state(timer);
933 953
934 val = kvm_arm_timer_read(vcpu, vcpu_get_timer(vcpu, tmr), treg);954 val = kvm_arm_timer_read(vcpu, timer, treg);
935 955
936 kvm_timer_vcpu_load(vcpu);956 timer_restore_state(timer);
937 preempt_enable();957 preempt_enable();
938 958
939 return val;959 return val;
@@ -967,25 +987,22 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · 27 unmodified lines
967 enum kvm_arch_timer_regs treg,987 enum kvm_arch_timer_regs treg,
968 u64 val)988 u64 val)
969{989{
970 preempt_disable();990 struct arch_timer_context *timer;
971 kvm_timer_vcpu_put(vcpu);991 struct timer_map map;
972
973 kvm_arm_timer_write(vcpu, vcpu_get_timer(vcpu, tmr), treg, val);
974
975 kvm_timer_vcpu_load(vcpu);
976 preempt_enable();
977}
978
979static int kvm_timer_starting_cpu(unsigned int cpu)
980{
981 kvm_timer_init_interrupt(NULL);
982 return 0;
983}
984 992
985static int kvm_timer_dying_cpu(unsigned int cpu)993 get_timer_map(vcpu, &map);
986{994 timer = vcpu_get_timer(vcpu, tmr);
987 disable_percpu_irq(host_vtimer_irq);995 if (timer == map.emul_ptimer) {
988 return 0;996 soft_timer_cancel(&timer->hrtimer);
997 kvm_arm_timer_write(vcpu, timer, treg, val);
998 timer_emulate(timer);
999 } else {
1000 preempt_disable();
1001 timer_save_state(timer);
1002 kvm_arm_timer_write(vcpu, timer, treg, val);
1003 timer_restore_state(timer);
1004 preempt_enable();
1005 }
989}1006}
990 1007
991static int timer_irq_set_vcpu_affinity(struct irq_data *d, void *vcpu)1008static int timer_irq_set_vcpu_affinity(struct irq_data *d, void *vcpu)
@@ -1117,7 +1134,7 @@ static int kvm_irq_init(struct arch_timer_kvm_info *info) · 125 unmodified lines
1117 return 0;1134 return 0;
1118}1135}
1119 1136
1120int kvm_timer_hyp_init(bool has_gic)1137int __init kvm_timer_hyp_init(bool has_gic)
1121{1138{
1122 struct arch_timer_kvm_info *info;1139 struct arch_timer_kvm_info *info;
1123 int err;1140 int err;
@@ -1185,9 +1202,6 @@ int kvm_timer_hyp_init(bool has_gic) · 61 unmodified lines
1185 goto out_free_irq;1202 goto out_free_irq;
1186 }1203 }
1187 1204
1188 cpuhp_setup_state(CPUHP_AP_KVM_ARM_TIMER_STARTING,
1189 "kvm/arm/timer:starting", kvm_timer_starting_cpu,
1190 kvm_timer_dying_cpu);
1191 return 0;1205 return 0;
1192out_free_irq:1206out_free_irq:
1193 free_percpu_irq(host_vtimer_irq, kvm_get_running_vcpus());1207 free_percpu_irq(host_vtimer_irq, kvm_get_running_vcpus());
arch/arm64/kvm/arm.c+60−49
@@ -63,16 +63,6 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu) · 62 unmodified lines
63 return kvm_vcpu_exiting_guest_mode(vcpu) == IN_GUEST_MODE;63 return kvm_vcpu_exiting_guest_mode(vcpu) == IN_GUEST_MODE;
64}64}
65 65
66int kvm_arch_hardware_setup(void *opaque)
67{
68 return 0;
69}
70
71int kvm_arch_check_processor_compat(void *opaque)
72{
73 return 0;
74}
75
76int kvm_vm_ioctl_enable_cap(struct kvm *kvm,66int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
77 struct kvm_enable_cap *cap)67 struct kvm_enable_cap *cap)
78{68{
@@ -146,7 +136,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) · 67 unmodified lines
146 if (ret)136 if (ret)
147 goto err_unshare_kvm;137 goto err_unshare_kvm;
148 138
149 if (!zalloc_cpumask_var(&kvm->arch.supported_cpus, GFP_KERNEL)) {139 if (!zalloc_cpumask_var(&kvm->arch.supported_cpus, GFP_KERNEL_ACCOUNT)) {
150 ret = -ENOMEM;140 ret = -ENOMEM;
151 goto err_unshare_kvm;141 goto err_unshare_kvm;
152 }142 }
@@ -1539,7 +1529,7 @@ static int kvm_init_vector_slots(void) · 1386 unmodified lines
1539 return 0;1529 return 0;
1540}1530}
1541 1531
1542static void cpu_prepare_hyp_mode(int cpu, u32 hyp_va_bits)1532static void __init cpu_prepare_hyp_mode(int cpu, u32 hyp_va_bits)
1543{1533{
1544 struct kvm_nvhe_init_params *params = per_cpu_ptr_nvhe_sym(kvm_init_params, cpu);1534 struct kvm_nvhe_init_params *params = per_cpu_ptr_nvhe_sym(kvm_init_params, cpu);
1545 unsigned long tcr;1535 unsigned long tcr;
@@ -1682,7 +1672,15 @@ static void _kvm_arch_hardware_enable(void *discard) · 136 unmodified lines
1682 1672
1683int kvm_arch_hardware_enable(void)1673int kvm_arch_hardware_enable(void)
1684{1674{
1675 int was_enabled = __this_cpu_read(kvm_arm_hardware_enabled);
1676
1685 _kvm_arch_hardware_enable(NULL);1677 _kvm_arch_hardware_enable(NULL);
1678
1679 if (!was_enabled) {
1680 kvm_vgic_cpu_up();
1681 kvm_timer_cpu_up();
1682 }
1683
1686 return 0;1684 return 0;
1687}1685}
1688 1686
@@ -1696,6 +1694,11 @@ static void _kvm_arch_hardware_disable(void *discard) · 7 unmodified lines
1696 1694
1697void kvm_arch_hardware_disable(void)1695void kvm_arch_hardware_disable(void)
1698{1696{
1697 if (__this_cpu_read(kvm_arm_hardware_enabled)) {
1698 kvm_timer_cpu_down();
1699 kvm_vgic_cpu_down();
1700 }
1701
1699 if (!is_protected_kvm_enabled())1702 if (!is_protected_kvm_enabled())
1700 _kvm_arch_hardware_disable(NULL);1703 _kvm_arch_hardware_disable(NULL);
1701}1704}
@@ -1738,26 +1741,26 @@ static struct notifier_block hyp_init_cpu_pm_nb = { · 36 unmodified lines
1738 .notifier_call = hyp_init_cpu_pm_notifier,1741 .notifier_call = hyp_init_cpu_pm_notifier,
1739};1742};
1740 1743
1741static void hyp_cpu_pm_init(void)1744static void __init hyp_cpu_pm_init(void)
1742{1745{
1743 if (!is_protected_kvm_enabled())1746 if (!is_protected_kvm_enabled())
1744 cpu_pm_register_notifier(&hyp_init_cpu_pm_nb);1747 cpu_pm_register_notifier(&hyp_init_cpu_pm_nb);
1745}1748}
1746static void hyp_cpu_pm_exit(void)1749static void __init hyp_cpu_pm_exit(void)
1747{1750{
1748 if (!is_protected_kvm_enabled())1751 if (!is_protected_kvm_enabled())
1749 cpu_pm_unregister_notifier(&hyp_init_cpu_pm_nb);1752 cpu_pm_unregister_notifier(&hyp_init_cpu_pm_nb);
1750}1753}
1751#else1754#else
1752static inline void hyp_cpu_pm_init(void)1755static inline void __init hyp_cpu_pm_init(void)
1753{1756{
1754}1757}
1755static inline void hyp_cpu_pm_exit(void)1758static inline void __init hyp_cpu_pm_exit(void)
1756{1759{
1757}1760}
1758#endif1761#endif
1759 1762
1760static void init_cpu_logical_map(void)1763static void __init init_cpu_logical_map(void)
1761{1764{
1762 unsigned int cpu;1765 unsigned int cpu;
1763 1766
@@ -1774,7 +1777,7 @@ static void init_cpu_logical_map(void) · 10 unmodified lines
1774#define init_psci_0_1_impl_state(config, what) \1777#define init_psci_0_1_impl_state(config, what) \
1775 config.psci_0_1_ ## what ## _implemented = psci_ops.what1778 config.psci_0_1_ ## what ## _implemented = psci_ops.what
1776 1779
1777static bool init_psci_relay(void)1780static bool __init init_psci_relay(void)
1778{1781{
1779 /*1782 /*
1780 * If PSCI has not been initialized, protected KVM cannot install1783 * If PSCI has not been initialized, protected KVM cannot install
@@ -1797,7 +1800,7 @@ static bool init_psci_relay(void) · 16 unmodified lines
1797 return true;1800 return true;
1798}1801}
1799 1802
1800static int init_subsystems(void)1803static int __init init_subsystems(void)
1801{1804{
1802 int err = 0;1805 int err = 0;
1803 1806
@@ -1838,13 +1841,22 @@ static int init_subsystems(void) · 34 unmodified lines
1838 kvm_register_perf_callbacks(NULL);1841 kvm_register_perf_callbacks(NULL);
1839 1842
1840out:1843out:
1844 if (err)
1845 hyp_cpu_pm_exit();
1846
1841 if (err || !is_protected_kvm_enabled())1847 if (err || !is_protected_kvm_enabled())
1842 on_each_cpu(_kvm_arch_hardware_disable, NULL, 1);1848 on_each_cpu(_kvm_arch_hardware_disable, NULL, 1);
1843 1849
1844 return err;1850 return err;
1845}1851}
1846 1852
1847static void teardown_hyp_mode(void)1853static void __init teardown_subsystems(void)
1854{
1855 kvm_unregister_perf_callbacks();
1856 hyp_cpu_pm_exit();
1857}
1858
1859static void __init teardown_hyp_mode(void)
1848{1860{
1849 int cpu;1861 int cpu;
1850 1862
@@ -1855,7 +1867,7 @@ static void teardown_hyp_mode(void) · 4 unmodified lines
1855 }1867 }
1856}1868}
1857 1869
1858static int do_pkvm_init(u32 hyp_va_bits)1870static int __init do_pkvm_init(u32 hyp_va_bits)
1859{1871{
1860 void *per_cpu_base = kvm_ksym_ref(kvm_nvhe_sym(kvm_arm_hyp_percpu_base));1872 void *per_cpu_base = kvm_ksym_ref(kvm_nvhe_sym(kvm_arm_hyp_percpu_base));
1861 int ret;1873 int ret;
@@ -1887,11 +1899,12 @@ static void kvm_hyp_init_symbols(void) · 25 unmodified lines
1887 kvm_nvhe_sym(id_aa64mmfr0_el1_sys_val) = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);1899 kvm_nvhe_sym(id_aa64mmfr0_el1_sys_val) = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
1888 kvm_nvhe_sym(id_aa64mmfr1_el1_sys_val) = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);1900 kvm_nvhe_sym(id_aa64mmfr1_el1_sys_val) = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
1889 kvm_nvhe_sym(id_aa64mmfr2_el1_sys_val) = read_sanitised_ftr_reg(SYS_ID_AA64MMFR2_EL1);1901 kvm_nvhe_sym(id_aa64mmfr2_el1_sys_val) = read_sanitised_ftr_reg(SYS_ID_AA64MMFR2_EL1);
1902 kvm_nvhe_sym(id_aa64smfr0_el1_sys_val) = read_sanitised_ftr_reg(SYS_ID_AA64SMFR0_EL1);
1890 kvm_nvhe_sym(__icache_flags) = __icache_flags;1903 kvm_nvhe_sym(__icache_flags) = __icache_flags;
1891 kvm_nvhe_sym(kvm_arm_vmid_bits) = kvm_arm_vmid_bits;1904 kvm_nvhe_sym(kvm_arm_vmid_bits) = kvm_arm_vmid_bits;
1892}1905}
1893 1906
1894static int kvm_hyp_init_protection(u32 hyp_va_bits)1907static int __init kvm_hyp_init_protection(u32 hyp_va_bits)
1895{1908{
1896 void *addr = phys_to_virt(hyp_mem_base);1909 void *addr = phys_to_virt(hyp_mem_base);
1897 int ret;1910 int ret;
@@ -1909,10 +1922,8 @@ static int kvm_hyp_init_protection(u32 hyp_va_bits) · 11 unmodified lines
1909 return 0;1922 return 0;
1910}1923}
1911 1924
1912/**1925/* Inits Hyp-mode on all online CPUs */
1913 * Inits Hyp-mode on all online CPUs1926static int __init init_hyp_mode(void)
1914 */
1915static int init_hyp_mode(void)
1916{1927{
1917 u32 hyp_va_bits;1928 u32 hyp_va_bits;
1918 int cpu;1929 int cpu;
@@ -2094,7 +2105,7 @@ out_err: · 175 unmodified lines
2094 return err;2105 return err;
2095}2106}
2096 2107
2097static void _kvm_host_prot_finalize(void *arg)2108static void __init _kvm_host_prot_finalize(void *arg)
2098{2109{
2099 int *err = arg;2110 int *err = arg;
2100 2111
@@ -2102,7 +2113,7 @@ static void _kvm_host_prot_finalize(void *arg) · 1 unmodified line
2102 WRITE_ONCE(*err, -EINVAL);2113 WRITE_ONCE(*err, -EINVAL);
2103}2114}
2104 2115
2105static int pkvm_drop_host_privileges(void)2116static int __init pkvm_drop_host_privileges(void)
2106{2117{
2107 int ret = 0;2118 int ret = 0;
2108 2119
@@ -2115,7 +2126,7 @@ static int pkvm_drop_host_privileges(void) · 6 unmodified lines
2115 return ret;2126 return ret;
2116}2127}
2117 2128
2118static int finalize_hyp_mode(void)2129static int __init finalize_hyp_mode(void)
2119{2130{
2120 if (!is_protected_kvm_enabled())2131 if (!is_protected_kvm_enabled())
2121 return 0;2132 return 0;
@@ -2187,10 +2198,8 @@ void kvm_arch_irq_bypass_start(struct irq_bypass_consumer *cons) · 65 unmodified lines
2187 kvm_arm_resume_guest(irqfd->kvm);2198 kvm_arm_resume_guest(irqfd->kvm);
2188}2199}
2189 2200
2190/**2201/* Initialize Hyp-mode and memory mappings on all CPUs */
2191 * Initialize Hyp-mode and memory mappings on all CPUs.2202static __init int kvm_arm_init(void)
2192 */
2193int kvm_arch_init(void *opaque)
2194{2203{
2195 int err;2204 int err;
2196 bool in_hyp_mode;2205 bool in_hyp_mode;
@@ -2241,7 +2250,7 @@ int kvm_arch_init(void *opaque) · 44 unmodified lines
2241 err = kvm_init_vector_slots();2250 err = kvm_init_vector_slots();
2242 if (err) {2251 if (err) {
2243 kvm_err("Cannot initialise vector slots\n");2252 kvm_err("Cannot initialise vector slots\n");
2244 goto out_err;2253 goto out_hyp;
2245 }2254 }
2246 2255
2247 err = init_subsystems();2256 err = init_subsystems();
@@ -2252,7 +2261,7 @@ int kvm_arch_init(void *opaque) · 4 unmodified lines
2252 err = finalize_hyp_mode();2261 err = finalize_hyp_mode();
2253 if (err) {2262 if (err) {
2254 kvm_err("Failed to finalize Hyp protection\n");2263 kvm_err("Failed to finalize Hyp protection\n");
2255 goto out_hyp;2264 goto out_subs;
2256 }2265 }
2257 }2266 }
2258 2267
@@ -2264,10 +2273,19 @@ int kvm_arch_init(void *opaque) · 5 unmodified lines
2264 kvm_info("Hyp mode initialized successfully\n");2273 kvm_info("Hyp mode initialized successfully\n");
2265 }2274 }
2266 2275
2276 /*
2277 * FIXME: Do something reasonable if kvm_init() fails after pKVM
2278 * hypervisor protection is finalized.
2279 */
2280 err = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
2281 if (err)
2282 goto out_subs;
2283
2267 return 0;2284 return 0;
2268 2285
2286out_subs:
2287 teardown_subsystems();
2269out_hyp:2288out_hyp:
2270 hyp_cpu_pm_exit();
2271 if (!in_hyp_mode)2289 if (!in_hyp_mode)
2272 teardown_hyp_mode();2290 teardown_hyp_mode();
2273out_err:2291out_err:
@@ -2275,12 +2293,6 @@ out_err: · 1 unmodified line
2275 return err;2293 return err;
2276}2294}
2277 2295
2278/* NOP: Compiling as a module not supported */
2279void kvm_arch_exit(void)
2280{
2281 kvm_unregister_perf_callbacks();
2282}
2283
2284static int __init early_kvm_mode_cfg(char *arg)2296static int __init early_kvm_mode_cfg(char *arg)
2285{2297{
2286 if (!arg)2298 if (!arg)
@@ -2310,6 +2322,11 @@ static int __init early_kvm_mode_cfg(char *arg) · 23 unmodified lines
2310 return 0;2322 return 0;
2311 }2323 }
2312 2324
2325 if (strcmp(arg, "nested") == 0 && !WARN_ON(!is_kernel_in_hyp_mode())) {
2326 kvm_mode = KVM_MODE_NV;
2327 return 0;
2328 }
2329
2313 return -EINVAL;2330 return -EINVAL;
2314}2331}
2315early_param("kvm-arm.mode", early_kvm_mode_cfg);2332early_param("kvm-arm.mode", early_kvm_mode_cfg);
@@ -2319,10 +2336,4 @@ enum kvm_mode kvm_get_mode(void) · 3 unmodified lines
2319 return kvm_mode;2336 return kvm_mode;
2320}2337}
2321 2338
2322static int arm_init(void)2339module_init(kvm_arm_init);
2323{
2324 int rc = kvm_init(NULL, sizeof(struct kvm_vcpu), 0, THIS_MODULE);
2325 return rc;
2326}
2327
2328module_init(arm_init);
arch/arm64/kvm/emulate-nested.c+203−0
@@ -0,0 +1,203 @@
1// SPDX-License-Identifier: GPL-2.0-only
2/*
3 * Copyright (C) 2016 - Linaro and Columbia University
4 * Author: Jintack Lim <jintack.lim@linaro.org>
5 */
6
7#include <linux/kvm.h>
8#include <linux/kvm_host.h>
9
10#include <asm/kvm_emulate.h>
11#include <asm/kvm_nested.h>
12
13#include "hyp/include/hyp/adjust_pc.h"
14
15#include "trace.h"
16
17static u64 kvm_check_illegal_exception_return(struct kvm_vcpu *vcpu, u64 spsr)
18{
19 u64 mode = spsr & PSR_MODE_MASK;
20
21 /*
22 * Possible causes for an Illegal Exception Return from EL2:
23 * - trying to return to EL3
24 * - trying to return to an illegal M value
25 * - trying to return to a 32bit EL
26 * - trying to return to EL1 with HCR_EL2.TGE set
27 */
28 if (mode == PSR_MODE_EL3t || mode == PSR_MODE_EL3h ||
29 mode == 0b00001 || (mode & BIT(1)) ||
30 (spsr & PSR_MODE32_BIT) ||
31 (vcpu_el2_tge_is_set(vcpu) && (mode == PSR_MODE_EL1t ||
32 mode == PSR_MODE_EL1h))) {
33 /*
34 * The guest is playing with our nerves. Preserve EL, SP,
35 * masks, flags from the existing PSTATE, and set IL.
36 * The HW will then generate an Illegal State Exception
37 * immediately after ERET.
38 */
39 spsr = *vcpu_cpsr(vcpu);
40
41 spsr &= (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT |
42 PSR_N_BIT | PSR_Z_BIT | PSR_C_BIT | PSR_V_BIT |
43 PSR_MODE_MASK | PSR_MODE32_BIT);
44 spsr |= PSR_IL_BIT;
45 }
46
47 return spsr;
48}
49
50void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
51{
52 u64 spsr, elr, mode;
53 bool direct_eret;
54
55 /*
56 * Going through the whole put/load motions is a waste of time
57 * if this is a VHE guest hypervisor returning to its own
58 * userspace, or the hypervisor performing a local exception
59 * return. No need to save/restore registers, no need to
60 * switch S2 MMU. Just do the canonical ERET.
61 */
62 spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2);
63 spsr = kvm_check_illegal_exception_return(vcpu, spsr);
64
65 mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT);
66
67 direct_eret = (mode == PSR_MODE_EL0t &&
68 vcpu_el2_e2h_is_set(vcpu) &&
69 vcpu_el2_tge_is_set(vcpu));
70 direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);
71
72 if (direct_eret) {
73 *vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2);
74 *vcpu_cpsr(vcpu) = spsr;
75 trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr);
76 return;
77 }
78
79 preempt_disable();
80 kvm_arch_vcpu_put(vcpu);
81
82 elr = __vcpu_sys_reg(vcpu, ELR_EL2);
83
84 trace_kvm_nested_eret(vcpu, elr, spsr);
85
86 /*
87 * Note that the current exception level is always the virtual EL2,
88 * since we set HCR_EL2.NV bit only when entering the virtual EL2.
89 */
90 *vcpu_pc(vcpu) = elr;
91 *vcpu_cpsr(vcpu) = spsr;
92
93 kvm_arch_vcpu_load(vcpu, smp_processor_id());
94 preempt_enable();
95}
96
97static void kvm_inject_el2_exception(struct kvm_vcpu *vcpu, u64 esr_el2,
98 enum exception_type type)
99{
100 trace_kvm_inject_nested_exception(vcpu, esr_el2, type);
101
102 switch (type) {
103 case except_type_sync:
104 kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_SYNC);
105 vcpu_write_sys_reg(vcpu, esr_el2, ESR_EL2);
106 break;
107 case except_type_irq:
108 kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_IRQ);
109 break;
110 default:
111 WARN_ONCE(1, "Unsupported EL2 exception injection %d\n", type);
112 }
113}
114
115/*
116 * Emulate taking an exception to EL2.
117 * See ARM ARM J8.1.2 AArch64.TakeException()
118 */
119static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2,
120 enum exception_type type)
121{
122 u64 pstate, mode;
123 bool direct_inject;
124
125 if (!vcpu_has_nv(vcpu)) {
126 kvm_err("Unexpected call to %s for the non-nesting configuration\n",
127 __func__);
128 return -EINVAL;
129 }
130
131 /*
132 * As for ERET, we can avoid doing too much on the injection path by
133 * checking that we either took the exception from a VHE host
134 * userspace or from vEL2. In these cases, there is no change in
135 * translation regime (or anything else), so let's do as little as
136 * possible.
137 */
138 pstate = *vcpu_cpsr(vcpu);
139 mode = pstate & (PSR_MODE_MASK | PSR_MODE32_BIT);
140
141 direct_inject = (mode == PSR_MODE_EL0t &&
142 vcpu_el2_e2h_is_set(vcpu) &&
143 vcpu_el2_tge_is_set(vcpu));
144 direct_inject |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);
145
146 if (direct_inject) {
147 kvm_inject_el2_exception(vcpu, esr_el2, type);
148 return 1;
149 }
150
151 preempt_disable();
152
153 /*
154 * We may have an exception or PC update in the EL0/EL1 context.
155 * Commit it before entering EL2.
156 */
157 __kvm_adjust_pc(vcpu);
158
159 kvm_arch_vcpu_put(vcpu);
160
161 kvm_inject_el2_exception(vcpu, esr_el2, type);
162
163 /*
164 * A hard requirement is that a switch between EL1 and EL2
165 * contexts has to happen between a put/load, so that we can
166 * pick the correct timer and interrupt configuration, among
167 * other things.
168 *
169 * Make sure the exception actually took place before we load
170 * the new context.
171 */
172 __kvm_adjust_pc(vcpu);
173
174 kvm_arch_vcpu_load(vcpu, smp_processor_id());
175 preempt_enable();
176
177 return 1;
178}
179
180int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2)
181{
182 return kvm_inject_nested(vcpu, esr_el2, except_type_sync);
183}
184
185int kvm_inject_nested_irq(struct kvm_vcpu *vcpu)
186{
187 /*
188 * Do not inject an irq if the:
189 * - Current exception level is EL2, and
190 * - virtual HCR_EL2.TGE == 0
191 * - virtual HCR_EL2.IMO == 0
192 *
193 * See Table D1-17 "Physical interrupt target and masking when EL3 is
194 * not implemented and EL2 is implemented" in ARM DDI 0487C.a.
195 */
196
197 if (vcpu_is_el2(vcpu) && !vcpu_el2_tge_is_set(vcpu) &&
198 !(__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_IMO))
199 return 1;
200
201 /* esr_el2 value doesn't matter for exits due to irqs. */
202 return kvm_inject_nested(vcpu, 0, except_type_irq);
203}
arch/arm64/kvm/fpsimd.c+1−0
@@ -184,6 +184,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) · 183 unmodified lines
184 sysreg_clear_set(CPACR_EL1,184 sysreg_clear_set(CPACR_EL1,
185 CPACR_EL1_SMEN_EL0EN,185 CPACR_EL1_SMEN_EL0EN,
186 CPACR_EL1_SMEN_EL1EN);186 CPACR_EL1_SMEN_EL1EN);
187 isb();
187 }188 }
188 189
189 if (vcpu->arch.fp_state == FP_STATE_GUEST_OWNED) {190 if (vcpu->arch.fp_state == FP_STATE_GUEST_OWNED) {
arch/arm64/kvm/guest.c+6−0
@@ -24,6 +24,7 @@ · 23 unmodified lines
24#include <asm/fpsimd.h>24#include <asm/fpsimd.h>
25#include <asm/kvm.h>25#include <asm/kvm.h>
26#include <asm/kvm_emulate.h>26#include <asm/kvm_emulate.h>
27#include <asm/kvm_nested.h>
27#include <asm/sigcontext.h>28#include <asm/sigcontext.h>
28 29
29#include "trace.h"30#include "trace.h"
@@ -253,6 +254,11 @@ static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) · 223 unmodified lines
253 if (!vcpu_el1_is_32bit(vcpu))254 if (!vcpu_el1_is_32bit(vcpu))
254 return -EINVAL;255 return -EINVAL;
255 break;256 break;
257 case PSR_MODE_EL2h:
258 case PSR_MODE_EL2t:
259 if (!vcpu_has_nv(vcpu))
260 return -EINVAL;
261 fallthrough;
256 case PSR_MODE_EL0t:262 case PSR_MODE_EL0t:
257 case PSR_MODE_EL1t:263 case PSR_MODE_EL1t:
258 case PSR_MODE_EL1h:264 case PSR_MODE_EL1h:
arch/arm64/kvm/handle_exit.c+45−2
@@ -16,6 +16,7 @@ · 15 unmodified lines
16#include <asm/kvm_asm.h>16#include <asm/kvm_asm.h>
17#include <asm/kvm_emulate.h>17#include <asm/kvm_emulate.h>
18#include <asm/kvm_mmu.h>18#include <asm/kvm_mmu.h>
19#include <asm/kvm_nested.h>
19#include <asm/debug-monitors.h>20#include <asm/debug-monitors.h>
20#include <asm/stacktrace/nvhe.h>21#include <asm/stacktrace/nvhe.h>
21#include <asm/traps.h>22#include <asm/traps.h>
@@ -41,6 +42,16 @@ static int handle_hvc(struct kvm_vcpu *vcpu) · 19 unmodified lines
41 kvm_vcpu_hvc_get_imm(vcpu));42 kvm_vcpu_hvc_get_imm(vcpu));
42 vcpu->stat.hvc_exit_stat++;43 vcpu->stat.hvc_exit_stat++;
43 44
45 /* Forward hvc instructions to the virtual EL2 if the guest has EL2. */
46 if (vcpu_has_nv(vcpu)) {
47 if (vcpu_read_sys_reg(vcpu, HCR_EL2) & HCR_HCD)
48 kvm_inject_undefined(vcpu);
49 else
50 kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu));
51
52 return 1;
53 }
54
44 ret = kvm_hvc_call_handler(vcpu);55 ret = kvm_hvc_call_handler(vcpu);
45 if (ret < 0) {56 if (ret < 0) {
46 vcpu_set_reg(vcpu, 0, ~0UL);57 vcpu_set_reg(vcpu, 0, ~0UL);
@@ -52,6 +63,8 @@ static int handle_hvc(struct kvm_vcpu *vcpu) · 5 unmodified lines
52 63
53static int handle_smc(struct kvm_vcpu *vcpu)64static int handle_smc(struct kvm_vcpu *vcpu)
54{65{
66 int ret;
67
55 /*68 /*
56 * "If an SMC instruction executed at Non-secure EL1 is69 * "If an SMC instruction executed at Non-secure EL1 is
57 * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a70 * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a
@@ -59,10 +72,30 @@ static int handle_smc(struct kvm_vcpu *vcpu) · 1 unmodified line
59 *72 *
60 * We need to advance the PC after the trap, as it would73 * We need to advance the PC after the trap, as it would
61 * otherwise return to the same address...74 * otherwise return to the same address...
75 *
76 * Only handle SMCs from the virtual EL2 with an immediate of zero and
77 * skip it otherwise.
62 */78 */
63 vcpu_set_reg(vcpu, 0, ~0UL);79 if (!vcpu_is_el2(vcpu) || kvm_vcpu_hvc_get_imm(vcpu)) {
80 vcpu_set_reg(vcpu, 0, ~0UL);
81 kvm_incr_pc(vcpu);
82 return 1;
83 }
84
85 /*
86 * If imm is zero then it is likely an SMCCC call.
87 *
88 * Note that on ARMv8.3, even if EL3 is not implemented, SMC executed
89 * at Non-secure EL1 is trapped to EL2 if HCR_EL2.TSC==1, rather than
90 * being treated as UNDEFINED.
91 */
92 ret = kvm_hvc_call_handler(vcpu);
93 if (ret < 0)
94 vcpu_set_reg(vcpu, 0, ~0UL);
95
64 kvm_incr_pc(vcpu);96 kvm_incr_pc(vcpu);
65 return 1;97
98 return ret;
66}99}
67 100
68/*101/*
@@ -196,6 +229,15 @@ static int kvm_handle_ptrauth(struct kvm_vcpu *vcpu) · 127 unmodified lines
196 return 1;229 return 1;
197}230}
198 231
232static int kvm_handle_eret(struct kvm_vcpu *vcpu)
233{
234 if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_ERET_ISS_ERET)
235 return kvm_handle_ptrauth(vcpu);
236
237 kvm_emulate_nested_eret(vcpu);
238 return 1;
239}
240
199static exit_handle_fn arm_exit_handlers[] = {241static exit_handle_fn arm_exit_handlers[] = {
200 [0 ... ESR_ELx_EC_MAX] = kvm_handle_unknown_ec,242 [0 ... ESR_ELx_EC_MAX] = kvm_handle_unknown_ec,
201 [ESR_ELx_EC_WFx] = kvm_handle_wfx,243 [ESR_ELx_EC_WFx] = kvm_handle_wfx,
@@ -211,6 +253,7 @@ static exit_handle_fn arm_exit_handlers[] = { · 9 unmodified lines
211 [ESR_ELx_EC_SMC64] = handle_smc,253 [ESR_ELx_EC_SMC64] = handle_smc,
212 [ESR_ELx_EC_SYS64] = kvm_handle_sys_reg,254 [ESR_ELx_EC_SYS64] = kvm_handle_sys_reg,
213 [ESR_ELx_EC_SVE] = handle_sve,255 [ESR_ELx_EC_SVE] = handle_sve,
256 [ESR_ELx_EC_ERET] = kvm_handle_eret,
214 [ESR_ELx_EC_IABT_LOW] = kvm_handle_guest_abort,257 [ESR_ELx_EC_IABT_LOW] = kvm_handle_guest_abort,
215 [ESR_ELx_EC_DABT_LOW] = kvm_handle_guest_abort,258 [ESR_ELx_EC_DABT_LOW] = kvm_handle_guest_abort,
216 [ESR_ELx_EC_SOFTSTP_LOW]= kvm_handle_guest_debug,259 [ESR_ELx_EC_SOFTSTP_LOW]= kvm_handle_guest_debug,
arch/arm64/kvm/hyp/exception.c+36−12
@@ -14,6 +14,7 @@ · 13 unmodified lines
14#include <linux/kvm_host.h>14#include <linux/kvm_host.h>
15#include <asm/kvm_emulate.h>15#include <asm/kvm_emulate.h>
16#include <asm/kvm_mmu.h>16#include <asm/kvm_mmu.h>
17#include <asm/kvm_nested.h>
17 18
18#if !defined (__KVM_NVHE_HYPERVISOR__) && !defined (__KVM_VHE_HYPERVISOR__)19#if !defined (__KVM_NVHE_HYPERVISOR__) && !defined (__KVM_VHE_HYPERVISOR__)
19#error Hypervisor code only!20#error Hypervisor code only!
@@ -23,7 +24,9 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) · 3 unmodified lines
23{24{
24 u64 val;25 u64 val;
25 26
26 if (__vcpu_read_sys_reg_from_cpu(reg, &val))27 if (unlikely(vcpu_has_nv(vcpu)))
28 return vcpu_read_sys_reg(vcpu, reg);
29 else if (__vcpu_read_sys_reg_from_cpu(reg, &val))
27 return val;30 return val;
28 31
29 return __vcpu_sys_reg(vcpu, reg);32 return __vcpu_sys_reg(vcpu, reg);
@@ -31,18 +34,25 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) · 1 unmodified line
31 34
32static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)35static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
33{36{
34 if (__vcpu_write_sys_reg_to_cpu(val, reg))37 if (unlikely(vcpu_has_nv(vcpu)))
35 return;38 vcpu_write_sys_reg(vcpu, val, reg);
36 39 else if (!__vcpu_write_sys_reg_to_cpu(val, reg))
37 __vcpu_sys_reg(vcpu, reg) = val;40 __vcpu_sys_reg(vcpu, reg) = val;
38}41}
39 42
40static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, u64 val)43static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long target_mode,
44 u64 val)
41{45{
42 if (has_vhe())46 if (unlikely(vcpu_has_nv(vcpu))) {
47 if (target_mode == PSR_MODE_EL1h)
48 vcpu_write_sys_reg(vcpu, val, SPSR_EL1);
49 else
50 vcpu_write_sys_reg(vcpu, val, SPSR_EL2);
51 } else if (has_vhe()) {
43 write_sysreg_el1(val, SYS_SPSR);52 write_sysreg_el1(val, SYS_SPSR);
44 else53 } else {
45 __vcpu_sys_reg(vcpu, SPSR_EL1) = val;54 __vcpu_sys_reg(vcpu, SPSR_EL1) = val;
55 }
46}56}
47 57
48static void __vcpu_write_spsr_abt(struct kvm_vcpu *vcpu, u64 val)58static void __vcpu_write_spsr_abt(struct kvm_vcpu *vcpu, u64 val)
@@ -101,6 +111,11 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, · 52 unmodified lines
101 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);111 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
102 __vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);112 __vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
103 break;113 break;
114 case PSR_MODE_EL2h:
115 vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL2);
116 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL2);
117 __vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2);
118 break;
104 default:119 default:
105 /* Don't do that */120 /* Don't do that */
106 BUG();121 BUG();
@@ -153,7 +168,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, · 46 unmodified lines
153 new |= target_mode;168 new |= target_mode;
154 169
155 *vcpu_cpsr(vcpu) = new;170 *vcpu_cpsr(vcpu) = new;
156 __vcpu_write_spsr(vcpu, old);171 __vcpu_write_spsr(vcpu, target_mode, old);
157}172}
158 173
159/*174/*
@@ -323,11 +338,20 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu) · 163 unmodified lines
323 case unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC):338 case unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC):
324 enter_exception64(vcpu, PSR_MODE_EL1h, except_type_sync);339 enter_exception64(vcpu, PSR_MODE_EL1h, except_type_sync);
325 break;340 break;
341
342 case unpack_vcpu_flag(EXCEPT_AA64_EL2_SYNC):
343 enter_exception64(vcpu, PSR_MODE_EL2h, except_type_sync);
344 break;
345
346 case unpack_vcpu_flag(EXCEPT_AA64_EL2_IRQ):
347 enter_exception64(vcpu, PSR_MODE_EL2h, except_type_irq);
348 break;
349
326 default:350 default:
327 /*351 /*
328 * Only EL1_SYNC makes sense so far, EL2_{SYNC,IRQ}352 * Only EL1_SYNC and EL2_{SYNC,IRQ} makes
329 * will be implemented at some point. Everything353 * sense so far. Everything else gets silently
330 * else gets silently ignored.354 * ignored.
331 */355 */
332 break;356 break;
333 }357 }
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h+18−3
@@ -39,7 +39,6 @@ static inline bool ctxt_has_mte(struct kvm_cpu_context *ctxt) · 38 unmodified lines
39 39
40static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)40static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
41{41{
42 ctxt_sys_reg(ctxt, CSSELR_EL1) = read_sysreg(csselr_el1);
43 ctxt_sys_reg(ctxt, SCTLR_EL1) = read_sysreg_el1(SYS_SCTLR);42 ctxt_sys_reg(ctxt, SCTLR_EL1) = read_sysreg_el1(SYS_SCTLR);
44 ctxt_sys_reg(ctxt, CPACR_EL1) = read_sysreg_el1(SYS_CPACR);43 ctxt_sys_reg(ctxt, CPACR_EL1) = read_sysreg_el1(SYS_CPACR);
45 ctxt_sys_reg(ctxt, TTBR0_EL1) = read_sysreg_el1(SYS_TTBR0);44 ctxt_sys_reg(ctxt, TTBR0_EL1) = read_sysreg_el1(SYS_TTBR0);
@@ -95,7 +94,6 @@ static inline void __sysreg_restore_user_state(struct kvm_cpu_context *ctxt) · 49 unmodified lines
95static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)94static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
96{95{
97 write_sysreg(ctxt_sys_reg(ctxt, MPIDR_EL1), vmpidr_el2);96 write_sysreg(ctxt_sys_reg(ctxt, MPIDR_EL1), vmpidr_el2);
98 write_sysreg(ctxt_sys_reg(ctxt, CSSELR_EL1), csselr_el1);
99 97
100 if (has_vhe() ||98 if (has_vhe() ||
101 !cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {99 !cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {
@@ -156,9 +154,26 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) · 54 unmodified lines
156 write_sysreg_el1(ctxt_sys_reg(ctxt, SPSR_EL1), SYS_SPSR);154 write_sysreg_el1(ctxt_sys_reg(ctxt, SPSR_EL1), SYS_SPSR);
157}155}
158 156
157/* Read the VCPU state's PSTATE, but translate (v)EL2 to EL1. */
158static inline u64 to_hw_pstate(const struct kvm_cpu_context *ctxt)
159{
160 u64 mode = ctxt->regs.pstate & (PSR_MODE_MASK | PSR_MODE32_BIT);
161
162 switch (mode) {
163 case PSR_MODE_EL2t:
164 mode = PSR_MODE_EL1t;
165 break;
166 case PSR_MODE_EL2h:
167 mode = PSR_MODE_EL1h;
168 break;
169 }
170
171 return (ctxt->regs.pstate & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode;
172}
173
159static inline void __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt)174static inline void __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt)
160{175{
161 u64 pstate = ctxt->regs.pstate;176 u64 pstate = to_hw_pstate(ctxt);
162 u64 mode = pstate & PSR_AA32_MODE_MASK;177 u64 mode = pstate & PSR_AA32_MODE_MASK;
163 178
164 /*179 /*
arch/arm64/kvm/hyp/nvhe/hyp-init.S+1−0
@@ -183,6 +183,7 @@ SYM_CODE_START_LOCAL(__kvm_hyp_init_cpu) · 182 unmodified lines
183 183
184 /* Initialize EL2 CPU state to sane values. */184 /* Initialize EL2 CPU state to sane values. */
185 init_el2_state // Clobbers x0..x2185 init_el2_state // Clobbers x0..x2
186 finalise_el2_state
186 187
187 /* Enable MMU, set vectors and stack. */188 /* Enable MMU, set vectors and stack. */
188 mov x0, x28189 mov x0, x28
arch/arm64/kvm/hyp/nvhe/sys_regs.c+1−0
@@ -26,6 +26,7 @@ u64 id_aa64isar2_el1_sys_val; · 25 unmodified lines
26u64 id_aa64mmfr0_el1_sys_val;26u64 id_aa64mmfr0_el1_sys_val;
27u64 id_aa64mmfr1_el1_sys_val;27u64 id_aa64mmfr1_el1_sys_val;
28u64 id_aa64mmfr2_el1_sys_val;28u64 id_aa64mmfr2_el1_sys_val;
29u64 id_aa64smfr0_el1_sys_val;
29 30
30/*31/*
31 * Inject an unknown/undefined exception to an AArch64 guest while most of its32 * Inject an unknown/undefined exception to an AArch64 guest while most of its
arch/arm64/kvm/hyp/pgtable.c+37−6
@@ -168,6 +168,25 @@ static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, · 167 unmodified lines
168 return walker->cb(ctx, visit);168 return walker->cb(ctx, visit);
169}169}
170 170
171static bool kvm_pgtable_walk_continue(const struct kvm_pgtable_walker *walker,
172 int r)
173{
174 /*
175 * Visitor callbacks return EAGAIN when the conditions that led to a
176 * fault are no longer reflected in the page tables due to a race to
177 * update a PTE. In the context of a fault handler this is interpreted
178 * as a signal to retry guest execution.
179 *
180 * Ignore the return code altogether for walkers outside a fault handler
181 * (e.g. write protecting a range of memory) and chug along with the
182 * page table walk.
183 */
184 if (r == -EAGAIN)
185 return !(walker->flags & KVM_PGTABLE_WALK_HANDLE_FAULT);
186
187 return !r;
188}
189
171static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data,190static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data,
172 struct kvm_pgtable_mm_ops *mm_ops, kvm_pteref_t pgtable, u32 level);191 struct kvm_pgtable_mm_ops *mm_ops, kvm_pteref_t pgtable, u32 level);
173 192
@@ -200,7 +219,7 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, · 26 unmodified lines
200 table = kvm_pte_table(ctx.old, level);219 table = kvm_pte_table(ctx.old, level);
201 }220 }
202 221
203 if (ret)222 if (!kvm_pgtable_walk_continue(data->walker, ret))
204 goto out;223 goto out;
205 224
206 if (!table) {225 if (!table) {
@@ -211,13 +230,16 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, · 4 unmodified lines
211 230
212 childp = (kvm_pteref_t)kvm_pte_follow(ctx.old, mm_ops);231 childp = (kvm_pteref_t)kvm_pte_follow(ctx.old, mm_ops);
213 ret = __kvm_pgtable_walk(data, mm_ops, childp, level + 1);232 ret = __kvm_pgtable_walk(data, mm_ops, childp, level + 1);
214 if (ret)233 if (!kvm_pgtable_walk_continue(data->walker, ret))
215 goto out;234 goto out;
216 235
217 if (ctx.flags & KVM_PGTABLE_WALK_TABLE_POST)236 if (ctx.flags & KVM_PGTABLE_WALK_TABLE_POST)
218 ret = kvm_pgtable_visitor_cb(data, &ctx, KVM_PGTABLE_WALK_TABLE_POST);237 ret = kvm_pgtable_visitor_cb(data, &ctx, KVM_PGTABLE_WALK_TABLE_POST);
219 238
220out:239out:
240 if (kvm_pgtable_walk_continue(data->walker, ret))
241 return 0;
242
221 return ret;243 return ret;
222}244}
223 245
@@ -584,12 +606,14 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift) · 360 unmodified lines
584 lvls = 2;606 lvls = 2;
585 vtcr |= VTCR_EL2_LVLS_TO_SL0(lvls);607 vtcr |= VTCR_EL2_LVLS_TO_SL0(lvls);
586 608
609#ifdef CONFIG_ARM64_HW_AFDBM
587 /*610 /*
588 * Enable the Hardware Access Flag management, unconditionally611 * Enable the Hardware Access Flag management, unconditionally
589 * on all CPUs. The features is RES0 on CPUs without the support612 * on all CPUs. The features is RES0 on CPUs without the support
590 * and must be ignored by the CPUs.613 * and must be ignored by the CPUs.
591 */614 */
592 vtcr |= VTCR_EL2_HA;615 vtcr |= VTCR_EL2_HA;
616#endif /* CONFIG_ARM64_HW_AFDBM */
593 617
594 /* Set the vmid bits */618 /* Set the vmid bits */
595 vtcr |= (get_vmid_bits(mmfr1) == 16) ?619 vtcr |= (get_vmid_bits(mmfr1) == 16) ?
@@ -1026,7 +1050,7 @@ static int stage2_attr_walker(const struct kvm_pgtable_visit_ctx *ctx, · 430 unmodified lines
1026 struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops;1050 struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops;
1027 1051
1028 if (!kvm_pte_valid(ctx->old))1052 if (!kvm_pte_valid(ctx->old))
1029 return 0;1053 return -EAGAIN;
1030 1054
1031 data->level = ctx->level;1055 data->level = ctx->level;
1032 data->pte = pte;1056 data->pte = pte;
@@ -1094,9 +1118,15 @@ int kvm_pgtable_stage2_wrprotect(struct kvm_pgtable *pgt, u64 addr, u64 size) · 61 unmodified lines
1094kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr)1118kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr)
1095{1119{
1096 kvm_pte_t pte = 0;1120 kvm_pte_t pte = 0;
1097 stage2_update_leaf_attrs(pgt, addr, 1, KVM_PTE_LEAF_ATTR_LO_S2_AF, 0,1121 int ret;
1098 &pte, NULL, 0);1122
1099 dsb(ishst);1123 ret = stage2_update_leaf_attrs(pgt, addr, 1, KVM_PTE_LEAF_ATTR_LO_S2_AF, 0,
1124 &pte, NULL,
1125 KVM_PGTABLE_WALK_HANDLE_FAULT |
1126 KVM_PGTABLE_WALK_SHARED);
1127 if (!ret)
1128 dsb(ishst);
1129
1100 return pte;1130 return pte;
1101}1131}
1102 1132
@@ -1141,6 +1171,7 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, · 38 unmodified lines
1141 clr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;1171 clr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;
1142 1172
1143 ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level,1173 ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level,
1174 KVM_PGTABLE_WALK_HANDLE_FAULT |
1144 KVM_PGTABLE_WALK_SHARED);1175 KVM_PGTABLE_WALK_SHARED);
1145 if (!ret)1176 if (!ret)
1146 kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, pgt->mmu, addr, level);1177 kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, pgt->mmu, addr, level);
arch/arm64/kvm/hyp/vhe/switch.c+25−1
@@ -40,7 +40,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu) · 39 unmodified lines
40 ___activate_traps(vcpu);40 ___activate_traps(vcpu);
41 41
42 val = read_sysreg(cpacr_el1);42 val = read_sysreg(cpacr_el1);
43 val |= CPACR_EL1_TTA;43 val |= CPACR_ELx_TTA;
44 val &= ~(CPACR_EL1_ZEN_EL0EN | CPACR_EL1_ZEN_EL1EN |44 val &= ~(CPACR_EL1_ZEN_EL0EN | CPACR_EL1_ZEN_EL1EN |
45 CPACR_EL1_SMEN_EL0EN | CPACR_EL1_SMEN_EL1EN);45 CPACR_EL1_SMEN_EL0EN | CPACR_EL1_SMEN_EL1EN);
46 46
@@ -120,6 +120,25 @@ static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu) · 73 unmodified lines
120 120
121static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code)121static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code)
122{122{
123 /*
124 * If we were in HYP context on entry, adjust the PSTATE view
125 * so that the usual helpers work correctly.
126 */
127 if (unlikely(vcpu_get_flag(vcpu, VCPU_HYP_CONTEXT))) {
128 u64 mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT);
129
130 switch (mode) {
131 case PSR_MODE_EL1t:
132 mode = PSR_MODE_EL2t;
133 break;
134 case PSR_MODE_EL1h:
135 mode = PSR_MODE_EL2h;
136 break;
137 }
138
139 *vcpu_cpsr(vcpu) &= ~(PSR_MODE_MASK | PSR_MODE32_BIT);
140 *vcpu_cpsr(vcpu) |= mode;
141 }
123}142}
124 143
125/* Switch to the guest for VHE systems running in EL2 */144/* Switch to the guest for VHE systems running in EL2 */
@@ -154,6 +173,11 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) · 28 unmodified lines
154 sysreg_restore_guest_state_vhe(guest_ctxt);173 sysreg_restore_guest_state_vhe(guest_ctxt);
155 __debug_switch_to_guest(vcpu);174 __debug_switch_to_guest(vcpu);
156 175
176 if (is_hyp_ctxt(vcpu))
177 vcpu_set_flag(vcpu, VCPU_HYP_CONTEXT);
178 else
179 vcpu_clear_flag(vcpu, VCPU_HYP_CONTEXT);
180
157 do {181 do {
158 /* Jump in the fire! */182 /* Jump in the fire! */
159 exit_code = __guest_enter(vcpu);183 exit_code = __guest_enter(vcpu);
arch/arm64/kvm/hypercalls.c+1−1
@@ -198,7 +198,7 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu) · 197 unmodified lines
198 break;198 break;
199 case ARM_SMCCC_HV_PV_TIME_ST:199 case ARM_SMCCC_HV_PV_TIME_ST:
200 gpa = kvm_init_stolen_time(vcpu);200 gpa = kvm_init_stolen_time(vcpu);
201 if (gpa != GPA_INVALID)201 if (gpa != INVALID_GPA)
202 val[0] = gpa;202 val[0] = gpa;
203 break;203 break;
204 case ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID:204 case ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID:
arch/arm64/kvm/inject_fault.c+55−6
@@ -12,17 +12,55 @@ · 11 unmodified lines
12 12
13#include <linux/kvm_host.h>13#include <linux/kvm_host.h>
14#include <asm/kvm_emulate.h>14#include <asm/kvm_emulate.h>
15#include <asm/kvm_nested.h>
15#include <asm/esr.h>16#include <asm/esr.h>
16 17
18static void pend_sync_exception(struct kvm_vcpu *vcpu)
19{
20 /* If not nesting, EL1 is the only possible exception target */
21 if (likely(!vcpu_has_nv(vcpu))) {
22 kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
23 return;
24 }
25
26 /*
27 * With NV, we need to pick between EL1 and EL2. Note that we
28 * never deal with a nesting exception here, hence never
29 * changing context, and the exception itself can be delayed
30 * until the next entry.
31 */
32 switch(*vcpu_cpsr(vcpu) & PSR_MODE_MASK) {
33 case PSR_MODE_EL2h:
34 case PSR_MODE_EL2t:
35 kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_SYNC);
36 break;
37 case PSR_MODE_EL1h:
38 case PSR_MODE_EL1t:
39 kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
40 break;
41 case PSR_MODE_EL0t:
42 if (vcpu_el2_tge_is_set(vcpu))
43 kvm_pend_exception(vcpu, EXCEPT_AA64_EL2_SYNC);
44 else
45 kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
46 break;
47 default:
48 BUG();
49 }
50}
51
52static bool match_target_el(struct kvm_vcpu *vcpu, unsigned long target)
53{
54 return (vcpu_get_flag(vcpu, EXCEPT_MASK) == target);
55}
56
17static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr)57static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr)
18{58{
19 unsigned long cpsr = *vcpu_cpsr(vcpu);59 unsigned long cpsr = *vcpu_cpsr(vcpu);
20 bool is_aarch32 = vcpu_mode_is_32bit(vcpu);60 bool is_aarch32 = vcpu_mode_is_32bit(vcpu);
21 u64 esr = 0;61 u64 esr = 0;
22 62
23 kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);63 pend_sync_exception(vcpu);
24
25 vcpu_write_sys_reg(vcpu, addr, FAR_EL1);
26 64
27 /*65 /*
28 * Build an {i,d}abort, depending on the level and the66 * Build an {i,d}abort, depending on the level and the
@@ -43,14 +81,22 @@ static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr · 14 unmodified lines
43 if (!is_iabt)81 if (!is_iabt)
44 esr |= ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT;82 esr |= ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT;
45 83
46 vcpu_write_sys_reg(vcpu, esr | ESR_ELx_FSC_EXTABT, ESR_EL1);84 esr |= ESR_ELx_FSC_EXTABT;
85
86 if (match_target_el(vcpu, unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC))) {
87 vcpu_write_sys_reg(vcpu, addr, FAR_EL1);
88 vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
89 } else {
90 vcpu_write_sys_reg(vcpu, addr, FAR_EL2);
91 vcpu_write_sys_reg(vcpu, esr, ESR_EL2);
92 }
47}93}
48 94
49static void inject_undef64(struct kvm_vcpu *vcpu)95static void inject_undef64(struct kvm_vcpu *vcpu)
50{96{
51 u64 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT);97 u64 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT);
52 98
53 kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);99 pend_sync_exception(vcpu);
54 100
55 /*101 /*
56 * Build an unknown exception, depending on the instruction102 * Build an unknown exception, depending on the instruction
@@ -59,7 +105,10 @@ static void inject_undef64(struct kvm_vcpu *vcpu) · 2 unmodified lines
59 if (kvm_vcpu_trap_il_is32bit(vcpu))105 if (kvm_vcpu_trap_il_is32bit(vcpu))
60 esr |= ESR_ELx_IL;106 esr |= ESR_ELx_IL;
61 107
62 vcpu_write_sys_reg(vcpu, esr, ESR_EL1);108 if (match_target_el(vcpu, unpack_vcpu_flag(EXCEPT_AA64_EL1_SYNC)))
109 vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
110 else
111 vcpu_write_sys_reg(vcpu, esr, ESR_EL2);
63}112}
64 113
65#define DFSR_FSC_EXTABT_LPAE 0x10114#define DFSR_FSC_EXTABT_LPAE 0x10
arch/arm64/kvm/mmu.c+23−23
@@ -25,11 +25,11 @@ · 24 unmodified lines
25static struct kvm_pgtable *hyp_pgtable;25static struct kvm_pgtable *hyp_pgtable;
26static DEFINE_MUTEX(kvm_hyp_pgd_mutex);26static DEFINE_MUTEX(kvm_hyp_pgd_mutex);
27 27
28static unsigned long hyp_idmap_start;28static unsigned long __ro_after_init hyp_idmap_start;
29static unsigned long hyp_idmap_end;29static unsigned long __ro_after_init hyp_idmap_end;
30static phys_addr_t hyp_idmap_vector;30static phys_addr_t __ro_after_init hyp_idmap_vector;
31 31
32static unsigned long io_map_base;32static unsigned long __ro_after_init io_map_base;
33 33
34static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end)34static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end)
35{35{
@@ -46,16 +46,17 @@ static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end) · 10 unmodified lines
46 * long will also starve other vCPUs. We have to also make sure that the page46 * long will also starve other vCPUs. We have to also make sure that the page
47 * tables are not freed while we released the lock.47 * tables are not freed while we released the lock.
48 */48 */
49static int stage2_apply_range(struct kvm *kvm, phys_addr_t addr,49static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr,
50 phys_addr_t end,50 phys_addr_t end,
51 int (*fn)(struct kvm_pgtable *, u64, u64),51 int (*fn)(struct kvm_pgtable *, u64, u64),
52 bool resched)52 bool resched)
53{53{
54 struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
54 int ret;55 int ret;
55 u64 next;56 u64 next;
56 57
57 do {58 do {
58 struct kvm_pgtable *pgt = kvm->arch.mmu.pgt;59 struct kvm_pgtable *pgt = mmu->pgt;
59 if (!pgt)60 if (!pgt)
60 return -EINVAL;61 return -EINVAL;
61 62
@@ -71,8 +72,8 @@ static int stage2_apply_range(struct kvm *kvm, phys_addr_t addr, · 9 unmodified lines
71 return ret;72 return ret;
72}73}
73 74
74#define stage2_apply_range_resched(kvm, addr, end, fn) \75#define stage2_apply_range_resched(mmu, addr, end, fn) \
75 stage2_apply_range(kvm, addr, end, fn, true)76 stage2_apply_range(mmu, addr, end, fn, true)
76 77
77static bool memslot_is_logging(struct kvm_memory_slot *memslot)78static bool memslot_is_logging(struct kvm_memory_slot *memslot)
78{79{
@@ -235,7 +236,7 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 · 156 unmodified lines
235 236
236 lockdep_assert_held_write(&kvm->mmu_lock);237 lockdep_assert_held_write(&kvm->mmu_lock);
237 WARN_ON(size & ~PAGE_MASK);238 WARN_ON(size & ~PAGE_MASK);
238 WARN_ON(stage2_apply_range(kvm, start, end, kvm_pgtable_stage2_unmap,239 WARN_ON(stage2_apply_range(mmu, start, end, kvm_pgtable_stage2_unmap,
239 may_block));240 may_block));
240}241}
241 242
@@ -250,7 +251,7 @@ static void stage2_flush_memslot(struct kvm *kvm, · 8 unmodified lines
250 phys_addr_t addr = memslot->base_gfn << PAGE_SHIFT;251 phys_addr_t addr = memslot->base_gfn << PAGE_SHIFT;
251 phys_addr_t end = addr + PAGE_SIZE * memslot->npages;252 phys_addr_t end = addr + PAGE_SIZE * memslot->npages;
252 253
253 stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_flush);254 stage2_apply_range_resched(&kvm->arch.mmu, addr, end, kvm_pgtable_stage2_flush);
254}255}
255 256
256/**257/**
@@ -280,7 +281,7 @@ static void stage2_flush_vm(struct kvm *kvm) · 23 unmodified lines
280/**281/**
281 * free_hyp_pgds - free Hyp-mode page tables282 * free_hyp_pgds - free Hyp-mode page tables
282 */283 */
283void free_hyp_pgds(void)284void __init free_hyp_pgds(void)
284{285{
285 mutex_lock(&kvm_hyp_pgd_mutex);286 mutex_lock(&kvm_hyp_pgd_mutex);
286 if (hyp_pgtable) {287 if (hyp_pgtable) {
@@ -934,8 +935,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, · 647 unmodified lines
934 */935 */
935static void stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end)936static void stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end)
936{937{
937 struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);938 stage2_apply_range_resched(mmu, addr, end, kvm_pgtable_stage2_wrprotect);
938 stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_wrprotect);
939}939}
940 940
941/**941/**
@@ -1383,7 +1383,9 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · 441 unmodified lines
1383 else1383 else
1384 ret = kvm_pgtable_stage2_map(pgt, fault_ipa, vma_pagesize,1384 ret = kvm_pgtable_stage2_map(pgt, fault_ipa, vma_pagesize,
1385 __pfn_to_phys(pfn), prot,1385 __pfn_to_phys(pfn), prot,
1386 memcache, KVM_PGTABLE_WALK_SHARED);1386 memcache,
1387 KVM_PGTABLE_WALK_HANDLE_FAULT |
1388 KVM_PGTABLE_WALK_SHARED);
1387 1389
1388 /* Mark the page dirty only if the fault is handled successfully */1390 /* Mark the page dirty only if the fault is handled successfully */
1389 if (writable && !ret) {1391 if (writable && !ret) {
@@ -1401,20 +1403,18 @@ out_unlock: · 11 unmodified lines
1401/* Resolve the access fault by making the page young again. */1403/* Resolve the access fault by making the page young again. */
1402static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)1404static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
1403{1405{
1404 pte_t pte;1406 kvm_pte_t pte;
1405 kvm_pte_t kpte;
1406 struct kvm_s2_mmu *mmu;1407 struct kvm_s2_mmu *mmu;
1407 1408
1408 trace_kvm_access_fault(fault_ipa);1409 trace_kvm_access_fault(fault_ipa);
1409 1410
1410 write_lock(&vcpu->kvm->mmu_lock);1411 read_lock(&vcpu->kvm->mmu_lock);
1411 mmu = vcpu->arch.hw_mmu;1412 mmu = vcpu->arch.hw_mmu;
1412 kpte = kvm_pgtable_stage2_mkyoung(mmu->pgt, fault_ipa);1413 pte = kvm_pgtable_stage2_mkyoung(mmu->pgt, fault_ipa);
1413 write_unlock(&vcpu->kvm->mmu_lock);1414 read_unlock(&vcpu->kvm->mmu_lock);
1414 1415
1415 pte = __pte(kpte);1416 if (kvm_pte_valid(pte))
1416 if (pte_valid(pte))1417 kvm_set_pfn_accessed(kvm_pte_to_pfn(pte));
1417 kvm_set_pfn_accessed(pte_pfn(pte));
1418}1418}
1419 1419
1420/**1420/**
@@ -1668,7 +1668,7 @@ static struct kvm_pgtable_mm_ops kvm_hyp_mm_ops = { · 247 unmodified lines
1668 .virt_to_phys = kvm_host_pa,1668 .virt_to_phys = kvm_host_pa,
1669};1669};
1670 1670
1671int kvm_mmu_init(u32 *hyp_va_bits)1671int __init kvm_mmu_init(u32 *hyp_va_bits)
1672{1672{
1673 int err;1673 int err;
1674 u32 idmap_bits;1674 u32 idmap_bits;
arch/arm64/kvm/nested.c+161−0
@@ -0,0 +1,161 @@
1// SPDX-License-Identifier: GPL-2.0-only
2/*
3 * Copyright (C) 2017 - Columbia University and Linaro Ltd.
4 * Author: Jintack Lim <jintack.lim@linaro.org>
5 */
6
7#include <linux/kvm.h>
8#include <linux/kvm_host.h>
9
10#include <asm/kvm_emulate.h>
11#include <asm/kvm_nested.h>
12#include <asm/sysreg.h>
13
14#include "sys_regs.h"
15
16/* Protection against the sysreg repainting madness... */
17#define NV_FTR(r, f) ID_AA64##r##_EL1_##f
18
19/*
20 * Our emulated CPU doesn't support all the possible features. For the
21 * sake of simplicity (and probably mental sanity), wipe out a number
22 * of feature bits we don't intend to support for the time being.
23 * This list should get updated as new features get added to the NV
24 * support, and new extension to the architecture.
25 */
26void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
27 const struct sys_reg_desc *r)
28{
29 u32 id = reg_to_encoding(r);
30 u64 val, tmp;
31
32 val = p->regval;
33
34 switch (id) {
35 case SYS_ID_AA64ISAR0_EL1:
36 /* Support everything but TME, O.S. and Range TLBIs */
37 val &= ~(NV_FTR(ISAR0, TLB) |
38 NV_FTR(ISAR0, TME));
39 break;
40
41 case SYS_ID_AA64ISAR1_EL1:
42 /* Support everything but PtrAuth and Spec Invalidation */
43 val &= ~(GENMASK_ULL(63, 56) |
44 NV_FTR(ISAR1, SPECRES) |
45 NV_FTR(ISAR1, GPI) |
46 NV_FTR(ISAR1, GPA) |
47 NV_FTR(ISAR1, API) |
48 NV_FTR(ISAR1, APA));
49 break;
50
51 case SYS_ID_AA64PFR0_EL1:
52 /* No AMU, MPAM, S-EL2, RAS or SVE */
53 val &= ~(GENMASK_ULL(55, 52) |
54 NV_FTR(PFR0, AMU) |
55 NV_FTR(PFR0, MPAM) |
56 NV_FTR(PFR0, SEL2) |
57 NV_FTR(PFR0, RAS) |
58 NV_FTR(PFR0, SVE) |
59 NV_FTR(PFR0, EL3) |
60 NV_FTR(PFR0, EL2) |
61 NV_FTR(PFR0, EL1));
62 /* 64bit EL1/EL2/EL3 only */
63 val |= FIELD_PREP(NV_FTR(PFR0, EL1), 0b0001);
64 val |= FIELD_PREP(NV_FTR(PFR0, EL2), 0b0001);
65 val |= FIELD_PREP(NV_FTR(PFR0, EL3), 0b0001);
66 break;
67
68 case SYS_ID_AA64PFR1_EL1:
69 /* Only support SSBS */
70 val &= NV_FTR(PFR1, SSBS);
71 break;
72
73 case SYS_ID_AA64MMFR0_EL1:
74 /* Hide ECV, FGT, ExS, Secure Memory */
75 val &= ~(GENMASK_ULL(63, 43) |
76 NV_FTR(MMFR0, TGRAN4_2) |
77 NV_FTR(MMFR0, TGRAN16_2) |
78 NV_FTR(MMFR0, TGRAN64_2) |
79 NV_FTR(MMFR0, SNSMEM));
80
81 /* Disallow unsupported S2 page sizes */
82 switch (PAGE_SIZE) {
83 case SZ_64K:
84 val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN16_2), 0b0001);
85 fallthrough;
86 case SZ_16K:
87 val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN4_2), 0b0001);
88 fallthrough;
89 case SZ_4K:
90 /* Support everything */
91 break;
92 }
93 /*
94 * Since we can't support a guest S2 page size smaller than
95 * the host's own page size (due to KVM only populating its
96 * own S2 using the kernel's page size), advertise the
97 * limitation using FEAT_GTG.
98 */
99 switch (PAGE_SIZE) {
100 case SZ_4K:
101 val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN4_2), 0b0010);
102 fallthrough;
103 case SZ_16K:
104 val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN16_2), 0b0010);
105 fallthrough;
106 case SZ_64K:
107 val |= FIELD_PREP(NV_FTR(MMFR0, TGRAN64_2), 0b0010);
108 break;
109 }
110 /* Cap PARange to 48bits */
111 tmp = FIELD_GET(NV_FTR(MMFR0, PARANGE), val);
112 if (tmp > 0b0101) {
113 val &= ~NV_FTR(MMFR0, PARANGE);
114 val |= FIELD_PREP(NV_FTR(MMFR0, PARANGE), 0b0101);
115 }
116 break;
117
118 case SYS_ID_AA64MMFR1_EL1:
119 val &= (NV_FTR(MMFR1, PAN) |
120 NV_FTR(MMFR1, LO) |
121 NV_FTR(MMFR1, HPDS) |
122 NV_FTR(MMFR1, VH) |
123 NV_FTR(MMFR1, VMIDBits));
124 break;
125
126 case SYS_ID_AA64MMFR2_EL1:
127 val &= ~(NV_FTR(MMFR2, EVT) |
128 NV_FTR(MMFR2, BBM) |
129 NV_FTR(MMFR2, TTL) |
130 GENMASK_ULL(47, 44) |
131 NV_FTR(MMFR2, ST) |
132 NV_FTR(MMFR2, CCIDX) |
133 NV_FTR(MMFR2, VARange));
134
135 /* Force TTL support */
136 val |= FIELD_PREP(NV_FTR(MMFR2, TTL), 0b0001);
137 break;
138
139 case SYS_ID_AA64DFR0_EL1:
140 /* Only limited support for PMU, Debug, BPs and WPs */
141 val &= (NV_FTR(DFR0, PMUVer) |
142 NV_FTR(DFR0, WRPs) |
143 NV_FTR(DFR0, BRPs) |
144 NV_FTR(DFR0, DebugVer));
145
146 /* Cap Debug to ARMv8.1 */
147 tmp = FIELD_GET(NV_FTR(DFR0, DebugVer), val);
148 if (tmp > 0b0111) {
149 val &= ~NV_FTR(DFR0, DebugVer);
150 val |= FIELD_PREP(NV_FTR(DFR0, DebugVer), 0b0111);
151 }
152 break;
153
154 default:
155 /* Unknown register, just wipe it clean */
156 val = 0;
157 break;
158 }
159
160 p->regval = val;
161}
arch/arm64/kvm/pvtime.c+4−4
@@ -19,7 +19,7 @@ void kvm_update_stolen_time(struct kvm_vcpu *vcpu) · 18 unmodified lines
19 u64 steal = 0;19 u64 steal = 0;
20 int idx;20 int idx;
21 21
22 if (base == GPA_INVALID)22 if (base == INVALID_GPA)
23 return;23 return;
24 24
25 idx = srcu_read_lock(&kvm->srcu);25 idx = srcu_read_lock(&kvm->srcu);
@@ -40,7 +40,7 @@ long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu) · 14 unmodified lines
40 switch (feature) {40 switch (feature) {
41 case ARM_SMCCC_HV_PV_TIME_FEATURES:41 case ARM_SMCCC_HV_PV_TIME_FEATURES:
42 case ARM_SMCCC_HV_PV_TIME_ST:42 case ARM_SMCCC_HV_PV_TIME_ST:
43 if (vcpu->arch.steal.base != GPA_INVALID)43 if (vcpu->arch.steal.base != INVALID_GPA)
44 val = SMCCC_RET_SUCCESS;44 val = SMCCC_RET_SUCCESS;
45 break;45 break;
46 }46 }
@@ -54,7 +54,7 @@ gpa_t kvm_init_stolen_time(struct kvm_vcpu *vcpu) · 7 unmodified lines
54 struct kvm *kvm = vcpu->kvm;54 struct kvm *kvm = vcpu->kvm;
55 u64 base = vcpu->arch.steal.base;55 u64 base = vcpu->arch.steal.base;
56 56
57 if (base == GPA_INVALID)57 if (base == INVALID_GPA)
58 return base;58 return base;
59 59
60 /*60 /*
@@ -89,7 +89,7 @@ int kvm_arm_pvtime_set_attr(struct kvm_vcpu *vcpu, · 28 unmodified lines
89 return -EFAULT;89 return -EFAULT;
90 if (!IS_ALIGNED(ipa, 64))90 if (!IS_ALIGNED(ipa, 64))
91 return -EINVAL;91 return -EINVAL;
92 if (vcpu->arch.steal.base != GPA_INVALID)92 if (vcpu->arch.steal.base != INVALID_GPA)
93 return -EEXIST;93 return -EEXIST;
94 94
95 /* Check the address is in a valid memslot */95 /* Check the address is in a valid memslot */
arch/arm64/kvm/reset.c+21−4
@@ -27,10 +27,11 @@ · 26 unmodified lines
27#include <asm/kvm_asm.h>27#include <asm/kvm_asm.h>
28#include <asm/kvm_emulate.h>28#include <asm/kvm_emulate.h>
29#include <asm/kvm_mmu.h>29#include <asm/kvm_mmu.h>
30#include <asm/kvm_nested.h>
30#include <asm/virt.h>31#include <asm/virt.h>
31 32
32/* Maximum phys_shift supported for any VM on this host */33/* Maximum phys_shift supported for any VM on this host */
33static u32 kvm_ipa_limit;34static u32 __ro_after_init kvm_ipa_limit;
34 35
35/*36/*
36 * ARMv8 Reset Values37 * ARMv8 Reset Values
@@ -38,12 +39,15 @@ static u32 kvm_ipa_limit; · 1 unmodified line
38#define VCPU_RESET_PSTATE_EL1 (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \39#define VCPU_RESET_PSTATE_EL1 (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
39 PSR_F_BIT | PSR_D_BIT)40 PSR_F_BIT | PSR_D_BIT)
40 41
42#define VCPU_RESET_PSTATE_EL2 (PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
43 PSR_F_BIT | PSR_D_BIT)
44
41#define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \45#define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
42 PSR_AA32_I_BIT | PSR_AA32_F_BIT)46 PSR_AA32_I_BIT | PSR_AA32_F_BIT)
43 47
44unsigned int kvm_sve_max_vl;48unsigned int __ro_after_init kvm_sve_max_vl;
45 49
46int kvm_arm_init_sve(void)50int __init kvm_arm_init_sve(void)
47{51{
48 if (system_supports_sve()) {52 if (system_supports_sve()) {
49 kvm_sve_max_vl = sve_max_virtualisable_vl();53 kvm_sve_max_vl = sve_max_virtualisable_vl();
@@ -157,6 +161,7 @@ void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu) · 107 unmodified lines
157 if (sve_state)161 if (sve_state)
158 kvm_unshare_hyp(sve_state, sve_state + vcpu_sve_state_size(vcpu));162 kvm_unshare_hyp(sve_state, sve_state + vcpu_sve_state_size(vcpu));
159 kfree(sve_state);163 kfree(sve_state);
164 kfree(vcpu->arch.ccsidr);
160}165}
161 166
162static void kvm_vcpu_reset_sve(struct kvm_vcpu *vcpu)167static void kvm_vcpu_reset_sve(struct kvm_vcpu *vcpu)
@@ -220,6 +225,10 @@ static int kvm_set_vm_width(struct kvm_vcpu *vcpu) · 57 unmodified lines
220 if (kvm_has_mte(kvm) && is32bit)225 if (kvm_has_mte(kvm) && is32bit)
221 return -EINVAL;226 return -EINVAL;
222 227
228 /* NV is incompatible with AArch32 */
229 if (vcpu_has_nv(vcpu) && is32bit)
230 return -EINVAL;
231
223 if (is32bit)232 if (is32bit)
224 set_bit(KVM_ARCH_FLAG_EL1_32BIT, &kvm->arch.flags);233 set_bit(KVM_ARCH_FLAG_EL1_32BIT, &kvm->arch.flags);
225 234
@@ -272,6 +281,12 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) · 46 unmodified lines
272 if (loaded)281 if (loaded)
273 kvm_arch_vcpu_put(vcpu);282 kvm_arch_vcpu_put(vcpu);
274 283
284 /* Disallow NV+SVE for the time being */
285 if (vcpu_has_nv(vcpu) && vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE)) {
286 ret = -EINVAL;
287 goto out;
288 }
289
275 if (!kvm_arm_vcpu_sve_finalized(vcpu)) {290 if (!kvm_arm_vcpu_sve_finalized(vcpu)) {
276 if (test_bit(KVM_ARM_VCPU_SVE, vcpu->arch.features)) {291 if (test_bit(KVM_ARM_VCPU_SVE, vcpu->arch.features)) {
277 ret = kvm_vcpu_enable_sve(vcpu);292 ret = kvm_vcpu_enable_sve(vcpu);
@@ -294,6 +309,8 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) · 16 unmodified lines
294 default:309 default:
295 if (vcpu_el1_is_32bit(vcpu)) {310 if (vcpu_el1_is_32bit(vcpu)) {
296 pstate = VCPU_RESET_PSTATE_SVC;311 pstate = VCPU_RESET_PSTATE_SVC;
312 } else if (vcpu_has_nv(vcpu)) {
313 pstate = VCPU_RESET_PSTATE_EL2;
297 } else {314 } else {
298 pstate = VCPU_RESET_PSTATE_EL1;315 pstate = VCPU_RESET_PSTATE_EL1;
299 }316 }
@@ -352,7 +369,7 @@ u32 get_kvm_ipa_limit(void) · 52 unmodified lines
352 return kvm_ipa_limit;369 return kvm_ipa_limit;
353}370}
354 371
355int kvm_set_ipa_limit(void)372int __init kvm_set_ipa_limit(void)
356{373{
357 unsigned int parange;374 unsigned int parange;
358 u64 mmfr0;375 u64 mmfr0;
arch/arm64/kvm/sys_regs.c+350−109
@@ -11,6 +11,7 @@ · 10 unmodified lines
11 11
12#include <linux/bitfield.h>12#include <linux/bitfield.h>
13#include <linux/bsearch.h>13#include <linux/bsearch.h>
14#include <linux/cacheinfo.h>
14#include <linux/kvm_host.h>15#include <linux/kvm_host.h>
15#include <linux/mm.h>16#include <linux/mm.h>
16#include <linux/printk.h>17#include <linux/printk.h>
@@ -24,6 +25,7 @@ · 7 unmodified lines
24#include <asm/kvm_emulate.h>25#include <asm/kvm_emulate.h>
25#include <asm/kvm_hyp.h>26#include <asm/kvm_hyp.h>
26#include <asm/kvm_mmu.h>27#include <asm/kvm_mmu.h>
28#include <asm/kvm_nested.h>
27#include <asm/perf_event.h>29#include <asm/perf_event.h>
28#include <asm/sysreg.h>30#include <asm/sysreg.h>
29 31
@@ -78,28 +80,112 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) · 48 unmodified lines
78 __vcpu_write_sys_reg_to_cpu(val, reg))80 __vcpu_write_sys_reg_to_cpu(val, reg))
79 return;81 return;
80 82
81 __vcpu_sys_reg(vcpu, reg) = val;83 __vcpu_sys_reg(vcpu, reg) = val;
82}84}
83 85
84/* 3 bits per cache level, as per CLIDR, but non-existent caches always 0 */
85static u32 cache_levels;
86
87/* CSSELR values; used to index KVM_REG_ARM_DEMUX_ID_CCSIDR */86/* CSSELR values; used to index KVM_REG_ARM_DEMUX_ID_CCSIDR */
88#define CSSELR_MAX 1487#define CSSELR_MAX 14
89 88
89/*
90 * Returns the minimum line size for the selected cache, expressed as
91 * Log2(bytes).
92 */
93static u8 get_min_cache_line_size(bool icache)
94{
95 u64 ctr = read_sanitised_ftr_reg(SYS_CTR_EL0);
96 u8 field;
97
98 if (icache)
99 field = SYS_FIELD_GET(CTR_EL0, IminLine, ctr);
100 else
101 field = SYS_FIELD_GET(CTR_EL0, DminLine, ctr);
102
103 /*
104 * Cache line size is represented as Log2(words) in CTR_EL0.
105 * Log2(bytes) can be derived with the following:
106 *
107 * Log2(words) + 2 = Log2(bytes / 4) + 2
108 * = Log2(bytes) - 2 + 2
109 * = Log2(bytes)
110 */
111 return field + 2;
112}
113
90/* Which cache CCSIDR represents depends on CSSELR value. */114/* Which cache CCSIDR represents depends on CSSELR value. */
91static u32 get_ccsidr(u32 csselr)115static u32 get_ccsidr(struct kvm_vcpu *vcpu, u32 csselr)
92{116{
93 u32 ccsidr;117 u8 line_size;
94 118
95 /* Make sure noone else changes CSSELR during this! */119 if (vcpu->arch.ccsidr)
96 local_irq_disable();120 return vcpu->arch.ccsidr[csselr];
97 write_sysreg(csselr, csselr_el1);
98 isb();
99 ccsidr = read_sysreg(ccsidr_el1);
100 local_irq_enable();
101 121
102 return ccsidr;122 line_size = get_min_cache_line_size(csselr & CSSELR_EL1_InD);
123
124 /*
125 * Fabricate a CCSIDR value as the overriding value does not exist.
126 * The real CCSIDR value will not be used as it can vary by the
127 * physical CPU which the vcpu currently resides in.
128 *
129 * The line size is determined with get_min_cache_line_size(), which
130 * should be valid for all CPUs even if they have different cache
131 * configuration.
132 *
133 * The associativity bits are cleared, meaning the geometry of all data
134 * and unified caches (which are guaranteed to be PIPT and thus
135 * non-aliasing) are 1 set and 1 way.
136 * Guests should not be doing cache operations by set/way at all, and
137 * for this reason, we trap them and attempt to infer the intent, so
138 * that we can flush the entire guest's address space at the appropriate
139 * time. The exposed geometry minimizes the number of the traps.
140 * [If guests should attempt to infer aliasing properties from the
141 * geometry (which is not permitted by the architecture), they would
142 * only do so for virtually indexed caches.]
143 *
144 * We don't check if the cache level exists as it is allowed to return
145 * an UNKNOWN value if not.
146 */
147 return SYS_FIELD_PREP(CCSIDR_EL1, LineSize, line_size - 4);
148}
149
150static int set_ccsidr(struct kvm_vcpu *vcpu, u32 csselr, u32 val)
151{
152 u8 line_size = FIELD_GET(CCSIDR_EL1_LineSize, val) + 4;
153 u32 *ccsidr = vcpu->arch.ccsidr;
154 u32 i;
155
156 if ((val & CCSIDR_EL1_RES0) ||
157 line_size < get_min_cache_line_size(csselr & CSSELR_EL1_InD))
158 return -EINVAL;
159
160 if (!ccsidr) {
161 if (val == get_ccsidr(vcpu, csselr))
162 return 0;
163
164 ccsidr = kmalloc_array(CSSELR_MAX, sizeof(u32), GFP_KERNEL_ACCOUNT);
165 if (!ccsidr)
166 return -ENOMEM;
167
168 for (i = 0; i < CSSELR_MAX; i++)
169 ccsidr[i] = get_ccsidr(vcpu, i);
170
171 vcpu->arch.ccsidr = ccsidr;
172 }
173
174 ccsidr[csselr] = val;
175
176 return 0;
177}
178
179static bool access_rw(struct kvm_vcpu *vcpu,
180 struct sys_reg_params *p,
181 const struct sys_reg_desc *r)
182{
183 if (p->is_write)
184 vcpu_write_sys_reg(vcpu, p->regval, r->reg);
185 else
186 p->regval = vcpu_read_sys_reg(vcpu, r->reg);
187
188 return true;
103}189}
104 190
105/*191/*
@@ -260,6 +346,14 @@ static bool trap_raz_wi(struct kvm_vcpu *vcpu, · 154 unmodified lines
260 return read_zero(vcpu, p);346 return read_zero(vcpu, p);
261}347}
262 348
349static bool trap_undef(struct kvm_vcpu *vcpu,
350 struct sys_reg_params *p,
351 const struct sys_reg_desc *r)
352{
353 kvm_inject_undefined(vcpu);
354 return false;
355}
356
263/*357/*
264 * ARMv8.1 mandates at least a trivial LORegion implementation, where all the358 * ARMv8.1 mandates at least a trivial LORegion implementation, where all the
265 * RW registers are RES0 (which we can implement as RAZ/WI). On an ARMv8.0359 * RW registers are RES0 (which we can implement as RAZ/WI). On an ARMv8.0
@@ -370,12 +464,9 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu, · 104 unmodified lines
370 struct sys_reg_params *p,464 struct sys_reg_params *p,
371 const struct sys_reg_desc *r)465 const struct sys_reg_desc *r)
372{466{
373 if (p->is_write) {467 access_rw(vcpu, p, r);
374 vcpu_write_sys_reg(vcpu, p->regval, r->reg);468 if (p->is_write)
375 vcpu_set_flag(vcpu, DEBUG_DIRTY);469 vcpu_set_flag(vcpu, DEBUG_DIRTY);
376 } else {
377 p->regval = vcpu_read_sys_reg(vcpu, r->reg);
378 }
379 470
380 trace_trap_reg(__func__, r->reg, p->is_write, p->regval);471 trace_trap_reg(__func__, r->reg, p->is_write, p->regval);
381 472
@@ -1049,7 +1140,9 @@ static bool access_arch_timer(struct kvm_vcpu *vcpu, · 667 unmodified lines
1049 treg = TIMER_REG_CVAL;1140 treg = TIMER_REG_CVAL;
1050 break;1141 break;
1051 default:1142 default:
1052 BUG();1143 print_sys_reg_msg(p, "%s", "Unhandled trapped timer register");
1144 kvm_inject_undefined(vcpu);
1145 return false;
1053 }1146 }
1054 1147
1055 if (p->is_write)1148 if (p->is_write)
@@ -1155,6 +1248,12 @@ static u64 read_id_reg(const struct kvm_vcpu *vcpu, struct sys_reg_desc const *r · 99 unmodified lines
1155 val |= FIELD_PREP(ARM64_FEATURE_MASK(ID_DFR0_EL1_PerfMon),1248 val |= FIELD_PREP(ARM64_FEATURE_MASK(ID_DFR0_EL1_PerfMon),
1156 pmuver_to_perfmon(vcpu_pmuver(vcpu)));1249 pmuver_to_perfmon(vcpu_pmuver(vcpu)));
1157 break;1250 break;
1251 case SYS_ID_AA64MMFR2_EL1:
1252 val &= ~ID_AA64MMFR2_EL1_CCIDX_MASK;
1253 break;
1254 case SYS_ID_MMFR4_EL1:
1255 val &= ~ARM64_FEATURE_MASK(ID_MMFR4_EL1_CCIDX);
1256 break;
1158 }1257 }
1159 1258
1160 return val;1259 return val;
@@ -1205,6 +1304,9 @@ static bool access_id_reg(struct kvm_vcpu *vcpu, · 44 unmodified lines
1205 return write_to_read_only(vcpu, p, r);1304 return write_to_read_only(vcpu, p, r);
1206 1305
1207 p->regval = read_id_reg(vcpu, r);1306 p->regval = read_id_reg(vcpu, r);
1307 if (vcpu_has_nv(vcpu))
1308 access_nested_id_reg(vcpu, p, r);
1309
1208 return true;1310 return true;
1209}1311}
1210 1312
@@ -1385,10 +1487,78 @@ static bool access_clidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, · 174 unmodified lines
1385 if (p->is_write)1487 if (p->is_write)
1386 return write_to_read_only(vcpu, p, r);1488 return write_to_read_only(vcpu, p, r);
1387 1489
1388 p->regval = read_sysreg(clidr_el1);1490 p->regval = __vcpu_sys_reg(vcpu, r->reg);
1389 return true;1491 return true;
1390}1492}
1391 1493
1494/*
1495 * Fabricate a CLIDR_EL1 value instead of using the real value, which can vary
1496 * by the physical CPU which the vcpu currently resides in.
1497 */
1498static void reset_clidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
1499{
1500 u64 ctr_el0 = read_sanitised_ftr_reg(SYS_CTR_EL0);
1501 u64 clidr;
1502 u8 loc;
1503
1504 if ((ctr_el0 & CTR_EL0_IDC)) {
1505 /*
1506 * Data cache clean to the PoU is not required so LoUU and LoUIS
1507 * will not be set and a unified cache, which will be marked as
1508 * LoC, will be added.
1509 *
1510 * If not DIC, let the unified cache L2 so that an instruction
1511 * cache can be added as L1 later.
1512 */
1513 loc = (ctr_el0 & CTR_EL0_DIC) ? 1 : 2;
1514 clidr = CACHE_TYPE_UNIFIED << CLIDR_CTYPE_SHIFT(loc);
1515 } else {
1516 /*
1517 * Data cache clean to the PoU is required so let L1 have a data
1518 * cache and mark it as LoUU and LoUIS. As L1 has a data cache,
1519 * it can be marked as LoC too.
1520 */
1521 loc = 1;
1522 clidr = 1 << CLIDR_LOUU_SHIFT;
1523 clidr |= 1 << CLIDR_LOUIS_SHIFT;
1524 clidr |= CACHE_TYPE_DATA << CLIDR_CTYPE_SHIFT(1);
1525 }
1526
1527 /*
1528 * Instruction cache invalidation to the PoU is required so let L1 have
1529 * an instruction cache. If L1 already has a data cache, it will be
1530 * CACHE_TYPE_SEPARATE.
1531 */
1532 if (!(ctr_el0 & CTR_EL0_DIC))
1533 clidr |= CACHE_TYPE_INST << CLIDR_CTYPE_SHIFT(1);
1534
1535 clidr |= loc << CLIDR_LOC_SHIFT;
1536
1537 /*
1538 * Add tag cache unified to data cache. Allocation tags and data are
1539 * unified in a cache line so that it looks valid even if there is only
1540 * one cache line.
1541 */
1542 if (kvm_has_mte(vcpu->kvm))
1543 clidr |= 2 << CLIDR_TTYPE_SHIFT(loc);
1544
1545 __vcpu_sys_reg(vcpu, r->reg) = clidr;
1546}
1547
1548static int set_clidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
1549 u64 val)
1550{
1551 u64 ctr_el0 = read_sanitised_ftr_reg(SYS_CTR_EL0);
1552 u64 idc = !CLIDR_LOC(val) || (!CLIDR_LOUIS(val) && !CLIDR_LOUU(val));
1553
1554 if ((val & CLIDR_EL1_RES0) || (!(ctr_el0 & CTR_EL0_IDC) && idc))
1555 return -EINVAL;
1556
1557 __vcpu_sys_reg(vcpu, rd->reg) = val;
1558
1559 return 0;
1560}
1561
1392static bool access_csselr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,1562static bool access_csselr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
1393 const struct sys_reg_desc *r)1563 const struct sys_reg_desc *r)
1394{1564{
@@ -1410,22 +1580,10 @@ static bool access_ccsidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, · 15 unmodified lines
1410 return write_to_read_only(vcpu, p, r);1580 return write_to_read_only(vcpu, p, r);
1411 1581
1412 csselr = vcpu_read_sys_reg(vcpu, CSSELR_EL1);1582 csselr = vcpu_read_sys_reg(vcpu, CSSELR_EL1);
1413 p->regval = get_ccsidr(csselr);1583 csselr &= CSSELR_EL1_Level | CSSELR_EL1_InD;
1584 if (csselr < CSSELR_MAX)
1585 p->regval = get_ccsidr(vcpu, csselr);
1414 1586
1415 /*
1416 * Guests should not be doing cache operations by set/way at all, and
1417 * for this reason, we trap them and attempt to infer the intent, so
1418 * that we can flush the entire guest's address space at the appropriate
1419 * time.
1420 * To prevent this trapping from causing performance problems, let's
1421 * expose the geometry of all data and unified caches (which are
1422 * guaranteed to be PIPT and thus non-aliasing) as 1 set and 1 way.
1423 * [If guests should attempt to infer aliasing properties from the
1424 * geometry (which is not permitted by the architecture), they would
1425 * only do so for virtually indexed caches.]
1426 */
1427 if (!(csselr & 1)) // data or unified cache
1428 p->regval &= ~GENMASK(27, 3);
1429 return true;1587 return true;
1430}1588}
1431 1589
@@ -1446,6 +1604,44 @@ static unsigned int mte_visibility(const struct kvm_vcpu *vcpu, · 14 unmodified lines
1446 .visibility = mte_visibility, \1604 .visibility = mte_visibility, \
1447}1605}
1448 1606
1607static unsigned int el2_visibility(const struct kvm_vcpu *vcpu,
1608 const struct sys_reg_desc *rd)
1609{
1610 if (vcpu_has_nv(vcpu))
1611 return 0;
1612
1613 return REG_HIDDEN;
1614}
1615
1616#define EL2_REG(name, acc, rst, v) { \
1617 SYS_DESC(SYS_##name), \
1618 .access = acc, \
1619 .reset = rst, \
1620 .reg = name, \
1621 .visibility = el2_visibility, \
1622 .val = v, \
1623}
1624
1625/*
1626 * EL{0,1}2 registers are the EL2 view on an EL0 or EL1 register when
1627 * HCR_EL2.E2H==1, and only in the sysreg table for convenience of
1628 * handling traps. Given that, they are always hidden from userspace.
1629 */
1630static unsigned int elx2_visibility(const struct kvm_vcpu *vcpu,
1631 const struct sys_reg_desc *rd)
1632{
1633 return REG_HIDDEN_USER;
1634}
1635
1636#define EL12_REG(name, acc, rst, v) { \
1637 SYS_DESC(SYS_##name##_EL12), \
1638 .access = acc, \
1639 .reset = rst, \
1640 .reg = name##_EL1, \
1641 .val = v, \
1642 .visibility = elx2_visibility, \
1643}
1644
1449/* sys_reg_desc initialiser for known cpufeature ID registers */1645/* sys_reg_desc initialiser for known cpufeature ID registers */
1450#define ID_SANITISED(name) { \1646#define ID_SANITISED(name) { \
1451 SYS_DESC(SYS_##name), \1647 SYS_DESC(SYS_##name), \
@@ -1490,6 +1686,42 @@ static unsigned int mte_visibility(const struct kvm_vcpu *vcpu, · 38 unmodified lines
1490 .visibility = raz_visibility, \1686 .visibility = raz_visibility, \
1491}1687}
1492 1688
1689static bool access_sp_el1(struct kvm_vcpu *vcpu,
1690 struct sys_reg_params *p,
1691 const struct sys_reg_desc *r)
1692{
1693 if (p->is_write)
1694 __vcpu_sys_reg(vcpu, SP_EL1) = p->regval;
1695 else
1696 p->regval = __vcpu_sys_reg(vcpu, SP_EL1);
1697
1698 return true;
1699}
1700
1701static bool access_elr(struct kvm_vcpu *vcpu,
1702 struct sys_reg_params *p,
1703 const struct sys_reg_desc *r)
1704{
1705 if (p->is_write)
1706 vcpu_write_sys_reg(vcpu, p->regval, ELR_EL1);
1707 else
1708 p->regval = vcpu_read_sys_reg(vcpu, ELR_EL1);
1709
1710 return true;
1711}
1712
1713static bool access_spsr(struct kvm_vcpu *vcpu,
1714 struct sys_reg_params *p,
1715 const struct sys_reg_desc *r)
1716{
1717 if (p->is_write)
1718 __vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval;
1719 else
1720 p->regval = __vcpu_sys_reg(vcpu, SPSR_EL1);
1721
1722 return true;
1723}
1724
1493/*1725/*
1494 * Architected system registers.1726 * Architected system registers.
1495 * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op21727 * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2
@@ -1646,6 +1878,9 @@ static const struct sys_reg_desc sys_reg_descs[] = { · 150 unmodified lines
1646 PTRAUTH_KEY(APDB),1878 PTRAUTH_KEY(APDB),
1647 PTRAUTH_KEY(APGA),1879 PTRAUTH_KEY(APGA),
1648 1880
1881 { SYS_DESC(SYS_SPSR_EL1), access_spsr},
1882 { SYS_DESC(SYS_ELR_EL1), access_elr},
1883
1649 { SYS_DESC(SYS_AFSR0_EL1), access_vm_reg, reset_unknown, AFSR0_EL1 },1884 { SYS_DESC(SYS_AFSR0_EL1), access_vm_reg, reset_unknown, AFSR0_EL1 },
1650 { SYS_DESC(SYS_AFSR1_EL1), access_vm_reg, reset_unknown, AFSR1_EL1 },1885 { SYS_DESC(SYS_AFSR1_EL1), access_vm_reg, reset_unknown, AFSR1_EL1 },
1651 { SYS_DESC(SYS_ESR_EL1), access_vm_reg, reset_unknown, ESR_EL1 },1886 { SYS_DESC(SYS_ESR_EL1), access_vm_reg, reset_unknown, ESR_EL1 },
@@ -1693,7 +1928,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { · 41 unmodified lines
1693 { SYS_DESC(SYS_LORC_EL1), trap_loregion },1928 { SYS_DESC(SYS_LORC_EL1), trap_loregion },
1694 { SYS_DESC(SYS_LORID_EL1), trap_loregion },1929 { SYS_DESC(SYS_LORID_EL1), trap_loregion },
1695 1930
1696 { SYS_DESC(SYS_VBAR_EL1), NULL, reset_val, VBAR_EL1, 0 },1931 { SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 },
1697 { SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 },1932 { SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 },
1698 1933
1699 { SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only },1934 { SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only },
@@ -1717,7 +1952,9 @@ static const struct sys_reg_desc sys_reg_descs[] = { · 17 unmodified lines
1717 { SYS_DESC(SYS_CNTKCTL_EL1), NULL, reset_val, CNTKCTL_EL1, 0},1952 { SYS_DESC(SYS_CNTKCTL_EL1), NULL, reset_val, CNTKCTL_EL1, 0},
1718 1953
1719 { SYS_DESC(SYS_CCSIDR_EL1), access_ccsidr },1954 { SYS_DESC(SYS_CCSIDR_EL1), access_ccsidr },
1720 { SYS_DESC(SYS_CLIDR_EL1), access_clidr },1955 { SYS_DESC(SYS_CLIDR_EL1), access_clidr, reset_clidr, CLIDR_EL1,
1956 .set_user = set_clidr },
1957 { SYS_DESC(SYS_CCSIDR2_EL1), undef_access },
1721 { SYS_DESC(SYS_SMIDR_EL1), undef_access },1958 { SYS_DESC(SYS_SMIDR_EL1), undef_access },
1722 { SYS_DESC(SYS_CSSELR_EL1), access_csselr, reset_unknown, CSSELR_EL1 },1959 { SYS_DESC(SYS_CSSELR_EL1), access_csselr, reset_unknown, CSSELR_EL1 },
1723 { SYS_DESC(SYS_CTR_EL0), access_ctr },1960 { SYS_DESC(SYS_CTR_EL0), access_ctr },
@@ -1913,9 +2150,67 @@ static const struct sys_reg_desc sys_reg_descs[] = { · 189 unmodified lines
1913 { PMU_SYS_REG(SYS_PMCCFILTR_EL0), .access = access_pmu_evtyper,2150 { PMU_SYS_REG(SYS_PMCCFILTR_EL0), .access = access_pmu_evtyper,
1914 .reset = reset_val, .reg = PMCCFILTR_EL0, .val = 0 },2151 .reset = reset_val, .reg = PMCCFILTR_EL0, .val = 0 },
1915 2152
2153 EL2_REG(VPIDR_EL2, access_rw, reset_unknown, 0),
2154 EL2_REG(VMPIDR_EL2, access_rw, reset_unknown, 0),
2155 EL2_REG(SCTLR_EL2, access_rw, reset_val, SCTLR_EL2_RES1),
2156 EL2_REG(ACTLR_EL2, access_rw, reset_val, 0),
2157 EL2_REG(HCR_EL2, access_rw, reset_val, 0),
2158 EL2_REG(MDCR_EL2, access_rw, reset_val, 0),
2159 EL2_REG(CPTR_EL2, access_rw, reset_val, CPTR_EL2_DEFAULT ),
2160 EL2_REG(HSTR_EL2, access_rw, reset_val, 0),
2161 EL2_REG(HACR_EL2, access_rw, reset_val, 0),
2162
2163 EL2_REG(TTBR0_EL2, access_rw, reset_val, 0),
2164 EL2_REG(TTBR1_EL2, access_rw, reset_val, 0),
2165 EL2_REG(TCR_EL2, access_rw, reset_val, TCR_EL2_RES1),
2166 EL2_REG(VTTBR_EL2, access_rw, reset_val, 0),
2167 EL2_REG(VTCR_EL2, access_rw, reset_val, 0),
2168
1916 { SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 },2169 { SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 },
2170 EL2_REG(SPSR_EL2, access_rw, reset_val, 0),
2171 EL2_REG(ELR_EL2, access_rw, reset_val, 0),
2172 { SYS_DESC(SYS_SP_EL1), access_sp_el1},
2173
1917 { SYS_DESC(SYS_IFSR32_EL2), NULL, reset_unknown, IFSR32_EL2 },2174 { SYS_DESC(SYS_IFSR32_EL2), NULL, reset_unknown, IFSR32_EL2 },
2175 EL2_REG(AFSR0_EL2, access_rw, reset_val, 0),
2176 EL2_REG(AFSR1_EL2, access_rw, reset_val, 0),
2177 EL2_REG(ESR_EL2, access_rw, reset_val, 0),
1918 { SYS_DESC(SYS_FPEXC32_EL2), NULL, reset_val, FPEXC32_EL2, 0x700 },2178 { SYS_DESC(SYS_FPEXC32_EL2), NULL, reset_val, FPEXC32_EL2, 0x700 },
2179
2180 EL2_REG(FAR_EL2, access_rw, reset_val, 0),
2181 EL2_REG(HPFAR_EL2, access_rw, reset_val, 0),
2182
2183 EL2_REG(MAIR_EL2, access_rw, reset_val, 0),
2184 EL2_REG(AMAIR_EL2, access_rw, reset_val, 0),
2185
2186 EL2_REG(VBAR_EL2, access_rw, reset_val, 0),
2187 EL2_REG(RVBAR_EL2, access_rw, reset_val, 0),
2188 { SYS_DESC(SYS_RMR_EL2), trap_undef },
2189
2190 EL2_REG(CONTEXTIDR_EL2, access_rw, reset_val, 0),
2191 EL2_REG(TPIDR_EL2, access_rw, reset_val, 0),
2192
2193 EL2_REG(CNTVOFF_EL2, access_rw, reset_val, 0),
2194 EL2_REG(CNTHCTL_EL2, access_rw, reset_val, 0),
2195
2196 EL12_REG(SCTLR, access_vm_reg, reset_val, 0x00C50078),
2197 EL12_REG(CPACR, access_rw, reset_val, 0),
2198 EL12_REG(TTBR0, access_vm_reg, reset_unknown, 0),
2199 EL12_REG(TTBR1, access_vm_reg, reset_unknown, 0),
2200 EL12_REG(TCR, access_vm_reg, reset_val, 0),
2201 { SYS_DESC(SYS_SPSR_EL12), access_spsr},
2202 { SYS_DESC(SYS_ELR_EL12), access_elr},
2203 EL12_REG(AFSR0, access_vm_reg, reset_unknown, 0),
2204 EL12_REG(AFSR1, access_vm_reg, reset_unknown, 0),
2205 EL12_REG(ESR, access_vm_reg, reset_unknown, 0),
2206 EL12_REG(FAR, access_vm_reg, reset_unknown, 0),
2207 EL12_REG(MAIR, access_vm_reg, reset_unknown, 0),
2208 EL12_REG(AMAIR, access_vm_reg, reset_amair_el1, 0),
2209 EL12_REG(VBAR, access_rw, reset_val, 0),
2210 EL12_REG(CONTEXTIDR, access_vm_reg, reset_val, 0),
2211 EL12_REG(CNTKCTL, access_rw, reset_val, 0),
2212
2213 EL2_REG(SP_EL2, NULL, reset_unknown, 0),
1919};2214};
1920 2215
1921static bool trap_dbgdidr(struct kvm_vcpu *vcpu,2216static bool trap_dbgdidr(struct kvm_vcpu *vcpu,
@@ -2219,6 +2514,10 @@ static const struct sys_reg_desc cp15_regs[] = { · 297 unmodified lines
2219 2514
2220 { Op1(1), CRn( 0), CRm( 0), Op2(0), access_ccsidr },2515 { Op1(1), CRn( 0), CRm( 0), Op2(0), access_ccsidr },
2221 { Op1(1), CRn( 0), CRm( 0), Op2(1), access_clidr },2516 { Op1(1), CRn( 0), CRm( 0), Op2(1), access_clidr },
2517
2518 /* CCSIDR2 */
2519 { Op1(1), CRn( 0), CRm( 0), Op2(2), undef_access },
2520
2222 { Op1(2), CRn( 0), CRm( 0), Op2(0), access_csselr, NULL, CSSELR_EL1 },2521 { Op1(2), CRn( 0), CRm( 0), Op2(0), access_csselr, NULL, CSSELR_EL1 },
2223};2522};
2224 2523
@@ -2724,7 +3023,6 @@ id_to_sys_reg_desc(struct kvm_vcpu *vcpu, u64 id, · 499 unmodified lines
2724 3023
2725FUNCTION_INVARIANT(midr_el1)3024FUNCTION_INVARIANT(midr_el1)
2726FUNCTION_INVARIANT(revidr_el1)3025FUNCTION_INVARIANT(revidr_el1)
2727FUNCTION_INVARIANT(clidr_el1)
2728FUNCTION_INVARIANT(aidr_el1)3026FUNCTION_INVARIANT(aidr_el1)
2729 3027
2730static void get_ctr_el0(struct kvm_vcpu *v, const struct sys_reg_desc *r)3028static void get_ctr_el0(struct kvm_vcpu *v, const struct sys_reg_desc *r)
@@ -2733,10 +3031,9 @@ static void get_ctr_el0(struct kvm_vcpu *v, const struct sys_reg_desc *r) · 2 unmodified lines
2733}3031}
2734 3032
2735/* ->val is filled in by kvm_sys_reg_table_init() */3033/* ->val is filled in by kvm_sys_reg_table_init() */
2736static struct sys_reg_desc invariant_sys_regs[] = {3034static struct sys_reg_desc invariant_sys_regs[] __ro_after_init = {
2737 { SYS_DESC(SYS_MIDR_EL1), NULL, get_midr_el1 },3035 { SYS_DESC(SYS_MIDR_EL1), NULL, get_midr_el1 },
2738 { SYS_DESC(SYS_REVIDR_EL1), NULL, get_revidr_el1 },3036 { SYS_DESC(SYS_REVIDR_EL1), NULL, get_revidr_el1 },
2739 { SYS_DESC(SYS_CLIDR_EL1), NULL, get_clidr_el1 },
2740 { SYS_DESC(SYS_AIDR_EL1), NULL, get_aidr_el1 },3037 { SYS_DESC(SYS_AIDR_EL1), NULL, get_aidr_el1 },
2741 { SYS_DESC(SYS_CTR_EL0), NULL, get_ctr_el0 },3038 { SYS_DESC(SYS_CTR_EL0), NULL, get_ctr_el0 },
2742};3039};
@@ -2773,33 +3070,7 @@ static int set_invariant_sys_reg(u64 id, u64 __user *uaddr) · 30 unmodified lines
2773 return 0;3070 return 0;
2774}3071}
2775 3072
2776static bool is_valid_cache(u32 val)3073static int demux_c15_get(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
2777{
2778 u32 level, ctype;
2779
2780 if (val >= CSSELR_MAX)
2781 return false;
2782
2783 /* Bottom bit is Instruction or Data bit. Next 3 bits are level. */
2784 level = (val >> 1);
2785 ctype = (cache_levels >> (level * 3)) & 7;
2786
2787 switch (ctype) {
2788 case 0: /* No cache */
2789 return false;
2790 case 1: /* Instruction cache only */
2791 return (val & 1);
2792 case 2: /* Data cache only */
2793 case 4: /* Unified cache */
2794 return !(val & 1);
2795 case 3: /* Separate instruction and data caches */
2796 return true;
2797 default: /* Reserved: we can't know instruction or data. */
2798 return false;
2799 }
2800}
2801
2802static int demux_c15_get(u64 id, void __user *uaddr)
2803{3074{
2804 u32 val;3075 u32 val;
2805 u32 __user *uval = uaddr;3076 u32 __user *uval = uaddr;
@@ -2815,16 +3086,16 @@ static int demux_c15_get(u64 id, void __user *uaddr) · 9 unmodified lines
2815 return -ENOENT;3086 return -ENOENT;
2816 val = (id & KVM_REG_ARM_DEMUX_VAL_MASK)3087 val = (id & KVM_REG_ARM_DEMUX_VAL_MASK)
2817 >> KVM_REG_ARM_DEMUX_VAL_SHIFT;3088 >> KVM_REG_ARM_DEMUX_VAL_SHIFT;
2818 if (!is_valid_cache(val))3089 if (val >= CSSELR_MAX)
2819 return -ENOENT;3090 return -ENOENT;
2820 3091
2821 return put_user(get_ccsidr(val), uval);3092 return put_user(get_ccsidr(vcpu, val), uval);
2822 default:3093 default:
2823 return -ENOENT;3094 return -ENOENT;
2824 }3095 }
2825}3096}
2826 3097
2827static int demux_c15_set(u64 id, void __user *uaddr)3098static int demux_c15_set(struct kvm_vcpu *vcpu, u64 id, void __user *uaddr)
2828{3099{
2829 u32 val, newval;3100 u32 val, newval;
2830 u32 __user *uval = uaddr;3101 u32 __user *uval = uaddr;
@@ -2840,16 +3111,13 @@ static int demux_c15_set(u64 id, void __user *uaddr) · 9 unmodified lines
2840 return -ENOENT;3111 return -ENOENT;
2841 val = (id & KVM_REG_ARM_DEMUX_VAL_MASK)3112 val = (id & KVM_REG_ARM_DEMUX_VAL_MASK)
2842 >> KVM_REG_ARM_DEMUX_VAL_SHIFT;3113 >> KVM_REG_ARM_DEMUX_VAL_SHIFT;
2843 if (!is_valid_cache(val))3114 if (val >= CSSELR_MAX)
2844 return -ENOENT;3115 return -ENOENT;
2845 3116
2846 if (get_user(newval, uval))3117 if (get_user(newval, uval))
2847 return -EFAULT;3118 return -EFAULT;
2848 3119
2849 /* This is also invariant: you can't change it. */3120 return set_ccsidr(vcpu, val, newval);
2850 if (newval != get_ccsidr(val))
2851 return -EINVAL;
2852 return 0;
2853 default:3121 default:
2854 return -ENOENT;3122 return -ENOENT;
2855 }3123 }
@@ -2864,7 +3132,7 @@ int kvm_sys_reg_get_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg, · 8 unmodified lines
2864 int ret;3132 int ret;
2865 3133
2866 r = id_to_sys_reg_desc(vcpu, reg->id, table, num);3134 r = id_to_sys_reg_desc(vcpu, reg->id, table, num);
2867 if (!r)3135 if (!r || sysreg_hidden_user(vcpu, r))
2868 return -ENOENT;3136 return -ENOENT;
2869 3137
2870 if (r->get_user) {3138 if (r->get_user) {
@@ -2886,7 +3154,7 @@ int kvm_arm_sys_reg_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg · 15 unmodified lines
2886 int err;3154 int err;
2887 3155
2888 if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX)3156 if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX)
2889 return demux_c15_get(reg->id, uaddr);3157 return demux_c15_get(vcpu, reg->id, uaddr);
2890 3158
2891 err = get_invariant_sys_reg(reg->id, uaddr);3159 err = get_invariant_sys_reg(reg->id, uaddr);
2892 if (err != -ENOENT)3160 if (err != -ENOENT)
@@ -2908,7 +3176,7 @@ int kvm_sys_reg_set_user(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg, · 15 unmodified lines
2908 return -EFAULT;3176 return -EFAULT;
2909 3177
2910 r = id_to_sys_reg_desc(vcpu, reg->id, table, num);3178 r = id_to_sys_reg_desc(vcpu, reg->id, table, num);
2911 if (!r)3179 if (!r || sysreg_hidden_user(vcpu, r))
2912 return -ENOENT;3180 return -ENOENT;
2913 3181
2914 if (sysreg_user_write_ignore(vcpu, r))3182 if (sysreg_user_write_ignore(vcpu, r))
@@ -2930,7 +3198,7 @@ int kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg · 15 unmodified lines
2930 int err;3198 int err;
2931 3199
2932 if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX)3200 if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_DEMUX)
2933 return demux_c15_set(reg->id, uaddr);3201 return demux_c15_set(vcpu, reg->id, uaddr);
2934 3202
2935 err = set_invariant_sys_reg(reg->id, uaddr);3203 err = set_invariant_sys_reg(reg->id, uaddr);
2936 if (err != -ENOENT)3204 if (err != -ENOENT)
@@ -2942,13 +3210,7 @@ int kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg · 5 unmodified lines
2942 3210
2943static unsigned int num_demux_regs(void)3211static unsigned int num_demux_regs(void)
2944{3212{
2945 unsigned int i, count = 0;3213 return CSSELR_MAX;
2946
2947 for (i = 0; i < CSSELR_MAX; i++)
2948 if (is_valid_cache(i))
2949 count++;
2950
2951 return count;
2952}3214}
2953 3215
2954static int write_demux_regids(u64 __user *uindices)3216static int write_demux_regids(u64 __user *uindices)
@@ -2958,8 +3220,6 @@ static int write_demux_regids(u64 __user *uindices) · 3 unmodified lines
2958 3220
2959 val |= KVM_REG_ARM_DEMUX_ID_CCSIDR;3221 val |= KVM_REG_ARM_DEMUX_ID_CCSIDR;
2960 for (i = 0; i < CSSELR_MAX; i++) {3222 for (i = 0; i < CSSELR_MAX; i++) {
2961 if (!is_valid_cache(i))
2962 continue;
2963 if (put_user(val | i, uindices))3223 if (put_user(val | i, uindices))
2964 return -EFAULT;3224 return -EFAULT;
2965 uindices++;3225 uindices++;
@@ -3002,7 +3262,7 @@ static int walk_one_sys_reg(const struct kvm_vcpu *vcpu, · 36 unmodified lines
3002 if (!(rd->reg || rd->get_user))3262 if (!(rd->reg || rd->get_user))
3003 return 0;3263 return 0;
3004 3264
3005 if (sysreg_hidden(vcpu, rd))3265 if (sysreg_hidden_user(vcpu, rd))
3006 return 0;3266 return 0;
3007 3267
3008 if (!copy_reg_to_user(rd, uind))3268 if (!copy_reg_to_user(rd, uind))
@@ -3057,11 +3317,10 @@ int kvm_arm_copy_sys_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices) · 48 unmodified lines
3057 return write_demux_regids(uindices);3317 return write_demux_regids(uindices);
3058}3318}
3059 3319
3060int kvm_sys_reg_table_init(void)3320int __init kvm_sys_reg_table_init(void)
3061{3321{
3062 bool valid = true;3322 bool valid = true;
3063 unsigned int i;3323 unsigned int i;
3064 struct sys_reg_desc clidr;
3065 3324
3066 /* Make sure tables are unique and in order. */3325 /* Make sure tables are unique and in order. */
3067 valid &= check_sysreg_table(sys_reg_descs, ARRAY_SIZE(sys_reg_descs), false);3326 valid &= check_sysreg_table(sys_reg_descs, ARRAY_SIZE(sys_reg_descs), false);
@@ -3078,23 +3337,5 @@ int kvm_sys_reg_table_init(void) · 10 unmodified lines
3078 for (i = 0; i < ARRAY_SIZE(invariant_sys_regs); i++)3337 for (i = 0; i < ARRAY_SIZE(invariant_sys_regs); i++)
3079 invariant_sys_regs[i].reset(NULL, &invariant_sys_regs[i]);3338 invariant_sys_regs[i].reset(NULL, &invariant_sys_regs[i]);
3080 3339
3081 /*
3082 * CLIDR format is awkward, so clean it up. See ARM B4.1.20:
3083 *
3084 * If software reads the Cache Type fields from Ctype1
3085 * upwards, once it has seen a value of 0b000, no caches
3086 * exist at further-out levels of the hierarchy. So, for
3087 * example, if Ctype3 is the first Cache Type field with a
3088 * value of 0b000, the values of Ctype4 to Ctype7 must be
3089 * ignored.
3090 */
3091 get_clidr_el1(NULL, &clidr); /* Ugly... */
3092 cache_levels = clidr.val;
3093 for (i = 0; i < 7; i++)
3094 if (((cache_levels >> (i*3)) & 7) == 0)
3095 break;
3096 /* Clear all higher bits. */
3097 cache_levels &= (1 << (i*3))-1;
3098
3099 return 0;3340 return 0;
3100}3341}
arch/arm64/kvm/sys_regs.h+12−2
@@ -85,8 +85,9 @@ struct sys_reg_desc { · 84 unmodified lines
85};85};
86 86
87#define REG_HIDDEN (1 << 0) /* hidden from userspace and guest */87#define REG_HIDDEN (1 << 0) /* hidden from userspace and guest */
88#define REG_RAZ (1 << 1) /* RAZ from userspace and guest */88#define REG_HIDDEN_USER (1 << 1) /* hidden from userspace only */
89#define REG_USER_WI (1 << 2) /* WI from userspace only */89#define REG_RAZ (1 << 2) /* RAZ from userspace and guest */
90#define REG_USER_WI (1 << 3) /* WI from userspace only */
90 91
91static __printf(2, 3)92static __printf(2, 3)
92inline void print_sys_reg_msg(const struct sys_reg_params *p,93inline void print_sys_reg_msg(const struct sys_reg_params *p,
@@ -152,6 +153,15 @@ static inline bool sysreg_hidden(const struct kvm_vcpu *vcpu, · 59 unmodified lines
152 return sysreg_visibility(vcpu, r) & REG_HIDDEN;153 return sysreg_visibility(vcpu, r) & REG_HIDDEN;
153}154}
154 155
156static inline bool sysreg_hidden_user(const struct kvm_vcpu *vcpu,
157 const struct sys_reg_desc *r)
158{
159 if (likely(!r->visibility))
160 return false;
161
162 return r->visibility(vcpu, r) & (REG_HIDDEN | REG_HIDDEN_USER);
163}
164
155static inline bool sysreg_visible_as_raz(const struct kvm_vcpu *vcpu,165static inline bool sysreg_visible_as_raz(const struct kvm_vcpu *vcpu,
156 const struct sys_reg_desc *r)166 const struct sys_reg_desc *r)
157{167{
arch/arm64/kvm/trace_arm.h+59−0
@@ -2,6 +2,7 @@ · 1 unmodified line
2#if !defined(_TRACE_ARM_ARM64_KVM_H) || defined(TRACE_HEADER_MULTI_READ)2#if !defined(_TRACE_ARM_ARM64_KVM_H) || defined(TRACE_HEADER_MULTI_READ)
3#define _TRACE_ARM_ARM64_KVM_H3#define _TRACE_ARM_ARM64_KVM_H
4 4
5#include <asm/kvm_emulate.h>
5#include <kvm/arm_arch_timer.h>6#include <kvm/arm_arch_timer.h>
6#include <linux/tracepoint.h>7#include <linux/tracepoint.h>
7 8
@@ -301,6 +302,64 @@ TRACE_EVENT(kvm_timer_emulate, · 293 unmodified lines
301 __entry->timer_idx, __entry->should_fire)302 __entry->timer_idx, __entry->should_fire)
302);303);
303 304
305TRACE_EVENT(kvm_nested_eret,
306 TP_PROTO(struct kvm_vcpu *vcpu, unsigned long elr_el2,
307 unsigned long spsr_el2),
308 TP_ARGS(vcpu, elr_el2, spsr_el2),
309
310 TP_STRUCT__entry(
311 __field(struct kvm_vcpu *, vcpu)
312 __field(unsigned long, elr_el2)
313 __field(unsigned long, spsr_el2)
314 __field(unsigned long, target_mode)
315 __field(unsigned long, hcr_el2)
316 ),
317
318 TP_fast_assign(
319 __entry->vcpu = vcpu;
320 __entry->elr_el2 = elr_el2;
321 __entry->spsr_el2 = spsr_el2;
322 __entry->target_mode = spsr_el2 & (PSR_MODE_MASK | PSR_MODE32_BIT);
323 __entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
324 ),
325
326 TP_printk("elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx",
327 __entry->elr_el2, __entry->spsr_el2,
328 __print_symbolic(__entry->target_mode, kvm_mode_names),
329 __entry->hcr_el2)
330);
331
332TRACE_EVENT(kvm_inject_nested_exception,
333 TP_PROTO(struct kvm_vcpu *vcpu, u64 esr_el2, int type),
334 TP_ARGS(vcpu, esr_el2, type),
335
336 TP_STRUCT__entry(
337 __field(struct kvm_vcpu *, vcpu)
338 __field(unsigned long, esr_el2)
339 __field(int, type)
340 __field(unsigned long, spsr_el2)
341 __field(unsigned long, pc)
342 __field(unsigned long, source_mode)
343 __field(unsigned long, hcr_el2)
344 ),
345
346 TP_fast_assign(
347 __entry->vcpu = vcpu;
348 __entry->esr_el2 = esr_el2;
349 __entry->type = type;
350 __entry->spsr_el2 = *vcpu_cpsr(vcpu);
351 __entry->pc = *vcpu_pc(vcpu);
352 __entry->source_mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT);
353 __entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
354 ),
355
356 TP_printk("%s: esr_el2 0x%lx elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx",
357 __print_symbolic(__entry->type, kvm_exception_type_names),
358 __entry->esr_el2, __entry->pc, __entry->spsr_el2,
359 __print_symbolic(__entry->source_mode, kvm_mode_names),
360 __entry->hcr_el2)
361);
362
304#endif /* _TRACE_ARM_ARM64_KVM_H */363#endif /* _TRACE_ARM_ARM64_KVM_H */
305 364
306#undef TRACE_INCLUDE_PATH365#undef TRACE_INCLUDE_PATH
arch/arm64/kvm/vgic/vgic-init.c+3−18
@@ -465,17 +465,15 @@ out: · 464 unmodified lines
465 465
466/* GENERIC PROBE */466/* GENERIC PROBE */
467 467
468static int vgic_init_cpu_starting(unsigned int cpu)468void kvm_vgic_cpu_up(void)
469{469{
470 enable_percpu_irq(kvm_vgic_global_state.maint_irq, 0);470 enable_percpu_irq(kvm_vgic_global_state.maint_irq, 0);
471 return 0;
472}471}
473 472
474 473
475static int vgic_init_cpu_dying(unsigned int cpu)474void kvm_vgic_cpu_down(void)
476{475{
477 disable_percpu_irq(kvm_vgic_global_state.maint_irq);476 disable_percpu_irq(kvm_vgic_global_state.maint_irq);
478 return 0;
479}477}
480 478
481static irqreturn_t vgic_maintenance_handler(int irq, void *data)479static irqreturn_t vgic_maintenance_handler(int irq, void *data)
@@ -572,7 +570,7 @@ int kvm_vgic_hyp_init(void) · 90 unmodified lines
572 if (ret)570 if (ret)
573 return ret;571 return ret;
574 572
575 if (!has_mask)573 if (!has_mask && !kvm_vgic_global_state.maint_irq)
576 return 0;574 return 0;
577 575
578 ret = request_percpu_irq(kvm_vgic_global_state.maint_irq,576 ret = request_percpu_irq(kvm_vgic_global_state.maint_irq,
@@ -584,19 +582,6 @@ int kvm_vgic_hyp_init(void) · 5 unmodified lines
584 return ret;582 return ret;
585 }583 }
586 584
587 ret = cpuhp_setup_state(CPUHP_AP_KVM_ARM_VGIC_INIT_STARTING,
588 "kvm/arm/vgic:starting",
589 vgic_init_cpu_starting, vgic_init_cpu_dying);
590 if (ret) {
591 kvm_err("Cannot register vgic CPU notifier\n");
592 goto out_free_irq;
593 }
594
595 kvm_info("vgic interrupt IRQ%d\n", kvm_vgic_global_state.maint_irq);585 kvm_info("vgic interrupt IRQ%d\n", kvm_vgic_global_state.maint_irq);
596 return 0;586 return 0;
597
598out_free_irq:
599 free_percpu_irq(kvm_vgic_global_state.maint_irq,
600 kvm_get_running_vcpus());
601 return ret;
602}587}
arch/arm64/kvm/vgic/vgic-mmio.c+8−5
@@ -473,9 +473,10 @@ int vgic_uaccess_write_cpending(struct kvm_vcpu *vcpu, · 472 unmodified lines
473 * active state can be overwritten when the VCPU's state is synced coming back473 * active state can be overwritten when the VCPU's state is synced coming back
474 * from the guest.474 * from the guest.
475 *475 *
476 * For shared interrupts as well as GICv3 private interrupts, we have to476 * For shared interrupts as well as GICv3 private interrupts accessed from the
477 * stop all the VCPUs because interrupts can be migrated while we don't hold477 * non-owning CPU, we have to stop all the VCPUs because interrupts can be
478 * the IRQ locks and we don't want to be chasing moving targets.478 * migrated while we don't hold the IRQ locks and we don't want to be chasing
479 * moving targets.
479 *480 *
480 * For GICv2 private interrupts we don't have to do anything because481 * For GICv2 private interrupts we don't have to do anything because
481 * userspace accesses to the VGIC state already require all VCPUs to be482 * userspace accesses to the VGIC state already require all VCPUs to be
@@ -484,7 +485,8 @@ int vgic_uaccess_write_cpending(struct kvm_vcpu *vcpu, · 2 unmodified lines
484 */485 */
485static void vgic_access_active_prepare(struct kvm_vcpu *vcpu, u32 intid)486static void vgic_access_active_prepare(struct kvm_vcpu *vcpu, u32 intid)
486{487{
487 if (vcpu->kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3 ||488 if ((vcpu->kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3 &&
489 vcpu != kvm_get_running_vcpu()) ||
488 intid >= VGIC_NR_PRIVATE_IRQS)490 intid >= VGIC_NR_PRIVATE_IRQS)
489 kvm_arm_halt_guest(vcpu->kvm);491 kvm_arm_halt_guest(vcpu->kvm);
490}492}
@@ -492,7 +494,8 @@ static void vgic_access_active_prepare(struct kvm_vcpu *vcpu, u32 intid) · 1 unmodified line
492/* See vgic_access_active_prepare */494/* See vgic_access_active_prepare */
493static void vgic_access_active_finish(struct kvm_vcpu *vcpu, u32 intid)495static void vgic_access_active_finish(struct kvm_vcpu *vcpu, u32 intid)
494{496{
495 if (vcpu->kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3 ||497 if ((vcpu->kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3 &&
498 vcpu != kvm_get_running_vcpu()) ||
496 intid >= VGIC_NR_PRIVATE_IRQS)499 intid >= VGIC_NR_PRIVATE_IRQS)
497 kvm_arm_resume_guest(vcpu->kvm);500 kvm_arm_resume_guest(vcpu->kvm);
498}501}
arch/arm64/kvm/vgic/vgic-v3.c+5−4
@@ -3,6 +3,7 @@ · 2 unmodified lines
3#include <linux/irqchip/arm-gic-v3.h>3#include <linux/irqchip/arm-gic-v3.h>
4#include <linux/irq.h>4#include <linux/irq.h>
5#include <linux/irqdomain.h>5#include <linux/irqdomain.h>
6#include <linux/kstrtox.h>
6#include <linux/kvm.h>7#include <linux/kvm.h>
7#include <linux/kvm_host.h>8#include <linux/kvm_host.h>
8#include <kvm/arm_vgic.h>9#include <kvm/arm_vgic.h>
@@ -584,25 +585,25 @@ DEFINE_STATIC_KEY_FALSE(vgic_v3_cpuif_trap); · 575 unmodified lines
584 585
585static int __init early_group0_trap_cfg(char *buf)586static int __init early_group0_trap_cfg(char *buf)
586{587{
587 return strtobool(buf, &group0_trap);588 return kstrtobool(buf, &group0_trap);
588}589}
589early_param("kvm-arm.vgic_v3_group0_trap", early_group0_trap_cfg);590early_param("kvm-arm.vgic_v3_group0_trap", early_group0_trap_cfg);
590 591
591static int __init early_group1_trap_cfg(char *buf)592static int __init early_group1_trap_cfg(char *buf)
592{593{
593 return strtobool(buf, &group1_trap);594 return kstrtobool(buf, &group1_trap);
594}595}
595early_param("kvm-arm.vgic_v3_group1_trap", early_group1_trap_cfg);596early_param("kvm-arm.vgic_v3_group1_trap", early_group1_trap_cfg);
596 597
597static int __init early_common_trap_cfg(char *buf)598static int __init early_common_trap_cfg(char *buf)
598{599{
599 return strtobool(buf, &common_trap);600 return kstrtobool(buf, &common_trap);
600}601}
601early_param("kvm-arm.vgic_v3_common_trap", early_common_trap_cfg);602early_param("kvm-arm.vgic_v3_common_trap", early_common_trap_cfg);
602 603
603static int __init early_gicv4_enable(char *buf)604static int __init early_gicv4_enable(char *buf)
604{605{
605 return strtobool(buf, &gicv4_enable);606 return kstrtobool(buf, &gicv4_enable);
606}607}
607early_param("kvm-arm.vgic_v4_enable", early_gicv4_enable);608early_param("kvm-arm.vgic_v4_enable", early_gicv4_enable);
608 609
arch/arm64/kvm/vmid.c+3−3
@@ -16,7 +16,7 @@ · 15 unmodified lines
16#include <asm/kvm_asm.h>16#include <asm/kvm_asm.h>
17#include <asm/kvm_mmu.h>17#include <asm/kvm_mmu.h>
18 18
19unsigned int kvm_arm_vmid_bits;19unsigned int __ro_after_init kvm_arm_vmid_bits;
20static DEFINE_RAW_SPINLOCK(cpu_vmid_lock);20static DEFINE_RAW_SPINLOCK(cpu_vmid_lock);
21 21
22static atomic64_t vmid_generation;22static atomic64_t vmid_generation;
@@ -172,7 +172,7 @@ void kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid) · 149 unmodified lines
172/*172/*
173 * Initialize the VMID allocator173 * Initialize the VMID allocator
174 */174 */
175int kvm_arm_vmid_alloc_init(void)175int __init kvm_arm_vmid_alloc_init(void)
176{176{
177 kvm_arm_vmid_bits = kvm_get_vmid_bits();177 kvm_arm_vmid_bits = kvm_get_vmid_bits();
178 178
@@ -190,7 +190,7 @@ int kvm_arm_vmid_alloc_init(void) · 11 unmodified lines
190 return 0;190 return 0;
191}191}
192 192
193void kvm_arm_vmid_alloc_free(void)193void __init kvm_arm_vmid_alloc_free(void)
194{194{
195 kfree(vmid_map);195 kfree(vmid_map);
196}196}
arch/arm64/tools/cpucaps+1−0
@@ -33,6 +33,7 @@ HAS_GIC_PRIO_MASKING · 32 unmodified lines
33HAS_GIC_PRIO_RELAXED_SYNC33HAS_GIC_PRIO_RELAXED_SYNC
34HAS_LDAPR34HAS_LDAPR
35HAS_LSE_ATOMICS35HAS_LSE_ATOMICS
36HAS_NESTED_VIRT
36HAS_NO_FPSIMD37HAS_NO_FPSIMD
37HAS_NO_HW_PREFETCH38HAS_NO_HW_PREFETCH
38HAS_PAN39HAS_PAN
arch/arm64/tools/gen-sysreg.awk+19−1
@@ -103,6 +103,7 @@ END { · 102 unmodified lines
103 103
104 res0 = "UL(0)"104 res0 = "UL(0)"
105 res1 = "UL(0)"105 res1 = "UL(0)"
106 unkn = "UL(0)"
106 107
107 next_bit = 63108 next_bit = 63
108 109
@@ -117,11 +118,13 @@ END { · 8 unmodified lines
117 118
118 define(reg "_RES0", "(" res0 ")")119 define(reg "_RES0", "(" res0 ")")
119 define(reg "_RES1", "(" res1 ")")120 define(reg "_RES1", "(" res1 ")")
121 define(reg "_UNKN", "(" unkn ")")
120 print ""122 print ""
121 123
122 reg = null124 reg = null
123 res0 = null125 res0 = null
124 res1 = null126 res1 = null
127 unkn = null
125 128
126 next129 next
127}130}
@@ -139,6 +142,7 @@ END { · 11 unmodified lines
139 142
140 res0 = "UL(0)"143 res0 = "UL(0)"
141 res1 = "UL(0)"144 res1 = "UL(0)"
145 unkn = "UL(0)"
142 146
143 define("REG_" reg, "S" op0 "_" op1 "_C" crn "_C" crm "_" op2)147 define("REG_" reg, "S" op0 "_" op1 "_C" crn "_C" crm "_" op2)
144 define("SYS_" reg, "sys_reg(" op0 ", " op1 ", " crn ", " crm ", " op2 ")")148 define("SYS_" reg, "sys_reg(" op0 ", " op1 ", " crn ", " crm ", " op2 ")")
@@ -166,7 +170,9 @@ END { · 21 unmodified lines
166 define(reg "_RES0", "(" res0 ")")170 define(reg "_RES0", "(" res0 ")")
167 if (res1 != null)171 if (res1 != null)
168 define(reg "_RES1", "(" res1 ")")172 define(reg "_RES1", "(" res1 ")")
169 if (res0 != null || res1 != null)173 if (unkn != null)
174 define(reg "_UNKN", "(" unkn ")")
175 if (res0 != null || res1 != null || unkn != null)
170 print ""176 print ""
171 177
172 reg = null178 reg = null
@@ -177,6 +183,7 @@ END { · 4 unmodified lines
177 op2 = null183 op2 = null
178 res0 = null184 res0 = null
179 res1 = null185 res1 = null
186 unkn = null
180 187
181 next188 next
182}189}
@@ -195,6 +202,7 @@ END { · 12 unmodified lines
195 next_bit = 0202 next_bit = 0
196 res0 = null203 res0 = null
197 res1 = null204 res1 = null
205 unkn = null
198 206
199 next207 next
200}208}
@@ -220,6 +228,16 @@ END { · 19 unmodified lines
220 next228 next
221}229}
222 230
231/^Unkn/ && (block == "Sysreg" || block == "SysregFields") {
232 expect_fields(2)
233 parse_bitdef(reg, "UNKN", $2)
234 field = "UNKN_" msb "_" lsb
235
236 unkn = unkn " | GENMASK_ULL(" msb ", " lsb ")"
237
238 next
239}
240
223/^Field/ && (block == "Sysreg" || block == "SysregFields") {241/^Field/ && (block == "Sysreg" || block == "SysregFields") {
224 expect_fields(3)242 expect_fields(3)
225 field = $3243 field = $3
arch/arm64/tools/sysreg+17−0
@@ -15,6 +15,8 @@ · 14 unmodified lines
15 15
16# Res1 <msb>[:<lsb>]16# Res1 <msb>[:<lsb>]
17 17
18# Unkn <msb>[:<lsb>]
19
18# Field <msb>[:<lsb>] <name>20# Field <msb>[:<lsb>] <name>
19 21
20# Enum <msb>[:<lsb>] <name>22# Enum <msb>[:<lsb>] <name>
@@ -1779,6 +1781,16 @@ Sysreg SCXTNUM_EL1 3 0 13 0 7 · 1758 unmodified lines
1779Field 63:0 SoftwareContextNumber1781Field 63:0 SoftwareContextNumber
1780EndSysreg1782EndSysreg
1781 1783
1784# The bit layout for CCSIDR_EL1 depends on whether FEAT_CCIDX is implemented.
1785# The following is for case when FEAT_CCIDX is not implemented.
1786Sysreg CCSIDR_EL1 3 1 0 0 0
1787Res0 63:32
1788Unkn 31:28
1789Field 27:13 NumSets
1790Field 12:3 Associativity
1791Field 2:0 LineSize
1792EndSysreg
1793
1782Sysreg CLIDR_EL1 3 1 0 0 11794Sysreg CLIDR_EL1 3 1 0 0 1
1783Res0 63:471795Res0 63:47
1784Field 46:33 Ttypen1796Field 46:33 Ttypen
@@ -1795,6 +1807,11 @@ Field 5:3 Ctype2 · 10 unmodified lines
1795Field 2:0 Ctype11807Field 2:0 Ctype1
1796EndSysreg1808EndSysreg
1797 1809
1810Sysreg CCSIDR2_EL1 3 1 0 0 2
1811Res0 63:24
1812Field 23:0 NumSets
1813EndSysreg
1814
1798Sysreg GMID_EL1 3 1 0 0 41815Sysreg GMID_EL1 3 1 0 0 4
1799Res0 63:41816Res0 63:4
1800Field 3:0 BS1817Field 3:0 BS
arch/mips/include/asm/kvm_host.h+1−2
@@ -758,7 +758,7 @@ struct kvm_mips_callbacks { · 757 unmodified lines
758 void (*vcpu_reenter)(struct kvm_vcpu *vcpu);758 void (*vcpu_reenter)(struct kvm_vcpu *vcpu);
759};759};
760extern struct kvm_mips_callbacks *kvm_mips_callbacks;760extern struct kvm_mips_callbacks *kvm_mips_callbacks;
761int kvm_mips_emulation_init(struct kvm_mips_callbacks **install_callbacks);761int kvm_mips_emulation_init(void);
762 762
763/* Debug: dump vcpu state */763/* Debug: dump vcpu state */
764int kvm_arch_vcpu_dump_regs(struct kvm_vcpu *vcpu);764int kvm_arch_vcpu_dump_regs(struct kvm_vcpu *vcpu);
@@ -888,7 +888,6 @@ extern unsigned long kvm_mips_get_ramsize(struct kvm *kvm); · 123 unmodified lines
888extern int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu,888extern int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu,
889 struct kvm_mips_interrupt *irq);889 struct kvm_mips_interrupt *irq);
890 890
891static inline void kvm_arch_hardware_unsetup(void) {}
892static inline void kvm_arch_sync_events(struct kvm *kvm) {}891static inline void kvm_arch_sync_events(struct kvm *kvm) {}
893static inline void kvm_arch_free_memslot(struct kvm *kvm,892static inline void kvm_arch_free_memslot(struct kvm *kvm,
894 struct kvm_memory_slot *slot) {}893 struct kvm_memory_slot *slot) {}
arch/mips/kvm/Kconfig+1−0
@@ -28,6 +28,7 @@ config KVM · 27 unmodified lines
28 select MMU_NOTIFIER28 select MMU_NOTIFIER
29 select SRCU29 select SRCU
30 select INTERVAL_TREE30 select INTERVAL_TREE
31 select KVM_GENERIC_HARDWARE_ENABLING
31 help32 help
32 Support for hosting Guest kernels.33 Support for hosting Guest kernels.
33 34
arch/mips/kvm/Makefile+1−1
@@ -17,4 +17,4 @@ kvm-$(CONFIG_CPU_LOONGSON64) += loongson_ipi.o · 16 unmodified lines
17 17
18kvm-y += vz.o18kvm-y += vz.o
19obj-$(CONFIG_KVM) += kvm.o19obj-$(CONFIG_KVM) += kvm.o
20obj-y += callback.o tlb.o20obj-y += tlb.o
arch/mips/kvm/callback.c+0−14
@@ -1,14 +0,0 @@
1/*
2 * This file is subject to the terms and conditions of the GNU General Public
3 * License. See the file "COPYING" in the main directory of this archive
4 * for more details.
5 *
6 * Copyright (C) 2012 MIPS Technologies, Inc. All rights reserved.
7 * Authors: Yann Le Du <ledu@kymasys.com>
8 */
9
10#include <linux/export.h>
11#include <linux/kvm_host.h>
12
13struct kvm_mips_callbacks *kvm_mips_callbacks;
14EXPORT_SYMBOL_GPL(kvm_mips_callbacks);
arch/mips/kvm/mips.c+7−27
@@ -135,16 +135,6 @@ void kvm_arch_hardware_disable(void) · 134 unmodified lines
135 kvm_mips_callbacks->hardware_disable();135 kvm_mips_callbacks->hardware_disable();
136}136}
137 137
138int kvm_arch_hardware_setup(void *opaque)
139{
140 return 0;
141}
142
143int kvm_arch_check_processor_compat(void *opaque)
144{
145 return 0;
146}
147
148extern void kvm_init_loongson_ipi(struct kvm *kvm);138extern void kvm_init_loongson_ipi(struct kvm *kvm);
149 139
150int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)140int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
@@ -1015,21 +1005,6 @@ long kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) · 864 unmodified lines
1015 return r;1005 return r;
1016}1006}
1017 1007
1018int kvm_arch_init(void *opaque)
1019{
1020 if (kvm_mips_callbacks) {
1021 kvm_err("kvm: module already exists\n");
1022 return -EEXIST;
1023 }
1024
1025 return kvm_mips_emulation_init(&kvm_mips_callbacks);
1026}
1027
1028void kvm_arch_exit(void)
1029{
1030 kvm_mips_callbacks = NULL;
1031}
1032
1033int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu,1008int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu,
1034 struct kvm_sregs *sregs)1009 struct kvm_sregs *sregs)
1035{1010{
@@ -1646,16 +1621,21 @@ static int __init kvm_mips_init(void) · 610 unmodified lines
1646 if (ret)1621 if (ret)
1647 return ret;1622 return ret;
1648 1623
1649 ret = kvm_init(NULL, sizeof(struct kvm_vcpu), 0, THIS_MODULE);1624 ret = kvm_mips_emulation_init();
1650
1651 if (ret)1625 if (ret)
1652 return ret;1626 return ret;
1653 1627
1628
1654 if (boot_cpu_type() == CPU_LOONGSON64)1629 if (boot_cpu_type() == CPU_LOONGSON64)
1655 kvm_priority_to_irq = kvm_loongson3_priority_to_irq;1630 kvm_priority_to_irq = kvm_loongson3_priority_to_irq;
1656 1631
1657 register_die_notifier(&kvm_mips_csr_die_notifier);1632 register_die_notifier(&kvm_mips_csr_die_notifier);
1658 1633
1634 ret = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
1635 if (ret) {
1636 unregister_die_notifier(&kvm_mips_csr_die_notifier);
1637 return ret;
1638 }
1659 return 0;1639 return 0;
1660}1640}
1661 1641
arch/mips/kvm/vz.c+4−3
@@ -3304,7 +3304,10 @@ static struct kvm_mips_callbacks kvm_vz_callbacks = { · 3303 unmodified lines
3304 .vcpu_reenter = kvm_vz_vcpu_reenter,3304 .vcpu_reenter = kvm_vz_vcpu_reenter,
3305};3305};
3306 3306
3307int kvm_mips_emulation_init(struct kvm_mips_callbacks **install_callbacks)3307/* FIXME: Get rid of the callbacks now that trap-and-emulate is gone. */
3308struct kvm_mips_callbacks *kvm_mips_callbacks = &kvm_vz_callbacks;
3309
3310int kvm_mips_emulation_init(void)
3308{3311{
3309 if (!cpu_has_vz)3312 if (!cpu_has_vz)
3310 return -ENODEV;3313 return -ENODEV;
@@ -3318,7 +3321,5 @@ int kvm_mips_emulation_init(struct kvm_mips_callbacks **install_callbacks) · 7 unmodified lines
3318 return -ENODEV;3321 return -ENODEV;
3319 3322
3320 pr_info("Starting KVM with MIPS VZ extensions\n");3323 pr_info("Starting KVM with MIPS VZ extensions\n");
3321
3322 *install_callbacks = &kvm_vz_callbacks;
3323 return 0;3324 return 0;
3324}3325}
arch/powerpc/include/asm/kvm_host.h+0−3
@@ -876,13 +876,10 @@ struct kvm_vcpu_arch { · 875 unmodified lines
876#define __KVM_HAVE_ARCH_WQP876#define __KVM_HAVE_ARCH_WQP
877#define __KVM_HAVE_CREATE_DEVICE877#define __KVM_HAVE_CREATE_DEVICE
878 878
879static inline void kvm_arch_hardware_disable(void) {}
880static inline void kvm_arch_hardware_unsetup(void) {}
881static inline void kvm_arch_sync_events(struct kvm *kvm) {}879static inline void kvm_arch_sync_events(struct kvm *kvm) {}
882static inline void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) {}880static inline void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) {}
883static inline void kvm_arch_flush_shadow_all(struct kvm *kvm) {}881static inline void kvm_arch_flush_shadow_all(struct kvm *kvm) {}
884static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}882static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
885static inline void kvm_arch_exit(void) {}
886static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}883static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
887static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}884static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
888 885
arch/powerpc/include/asm/kvm_ppc.h+0−1
@@ -118,7 +118,6 @@ extern int kvmppc_xlate(struct kvm_vcpu *vcpu, ulong eaddr, · 117 unmodified lines
118extern int kvmppc_core_vcpu_create(struct kvm_vcpu *vcpu);118extern int kvmppc_core_vcpu_create(struct kvm_vcpu *vcpu);
119extern void kvmppc_core_vcpu_free(struct kvm_vcpu *vcpu);119extern void kvmppc_core_vcpu_free(struct kvm_vcpu *vcpu);
120extern int kvmppc_core_vcpu_setup(struct kvm_vcpu *vcpu);120extern int kvmppc_core_vcpu_setup(struct kvm_vcpu *vcpu);
121extern int kvmppc_core_check_processor_compat(void);
122extern int kvmppc_core_vcpu_translate(struct kvm_vcpu *vcpu,121extern int kvmppc_core_vcpu_translate(struct kvm_vcpu *vcpu,
123 struct kvm_translation *tr);122 struct kvm_translation *tr);
124 123
arch/powerpc/kvm/book3s.c+1−11
@@ -999,16 +999,6 @@ int kvmppc_h_logical_ci_store(struct kvm_vcpu *vcpu) · 998 unmodified lines
999}999}
1000EXPORT_SYMBOL_GPL(kvmppc_h_logical_ci_store);1000EXPORT_SYMBOL_GPL(kvmppc_h_logical_ci_store);
1001 1001
1002int kvmppc_core_check_processor_compat(void)
1003{
1004 /*
1005 * We always return 0 for book3s. We check
1006 * for compatibility while loading the HV
1007 * or PR module
1008 */
1009 return 0;
1010}
1011
1012int kvmppc_book3s_hcall_implemented(struct kvm *kvm, unsigned long hcall)1002int kvmppc_book3s_hcall_implemented(struct kvm *kvm, unsigned long hcall)
1013{1003{
1014 return kvm->arch.kvm_ops->hcall_implemented(hcall);1004 return kvm->arch.kvm_ops->hcall_implemented(hcall);
@@ -1062,7 +1052,7 @@ static int kvmppc_book3s_init(void) · 47 unmodified lines
1062{1052{
1063 int r;1053 int r;
1064 1054
1065 r = kvm_init(NULL, sizeof(struct kvm_vcpu), 0, THIS_MODULE);1055 r = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
1066 if (r)1056 if (r)
1067 return r;1057 return r;
1068#ifdef CONFIG_KVM_BOOK3S_32_HANDLER1058#ifdef CONFIG_KVM_BOOK3S_32_HANDLER
arch/powerpc/kvm/booke.c+1−1
@@ -1210,7 +1210,7 @@ int kvmppc_handle_exit(struct kvm_vcpu *vcpu, unsigned int exit_nr) · 1209 unmodified lines
1210 1210
1211/*1211/*
1212 * On cores with Vector category, KVM is loaded only if CONFIG_ALTIVEC,1212 * On cores with Vector category, KVM is loaded only if CONFIG_ALTIVEC,
1213 * see kvmppc_core_check_processor_compat().1213 * see kvmppc_e500mc_check_processor_compat().
1214 */1214 */
1215#ifdef CONFIG_ALTIVEC1215#ifdef CONFIG_ALTIVEC
1216 case BOOKE_INTERRUPT_ALTIVEC_UNAVAIL:1216 case BOOKE_INTERRUPT_ALTIVEC_UNAVAIL:
arch/powerpc/kvm/e500.c+3−3
@@ -314,7 +314,7 @@ static void kvmppc_core_vcpu_put_e500(struct kvm_vcpu *vcpu) · 313 unmodified lines
314 kvmppc_booke_vcpu_put(vcpu);314 kvmppc_booke_vcpu_put(vcpu);
315}315}
316 316
317int kvmppc_core_check_processor_compat(void)317static int kvmppc_e500_check_processor_compat(void)
318{318{
319 int r;319 int r;
320 320
@@ -507,7 +507,7 @@ static int __init kvmppc_e500_init(void) · 186 unmodified lines
507 unsigned long handler_len;507 unsigned long handler_len;
508 unsigned long max_ivor = 0;508 unsigned long max_ivor = 0;
509 509
510 r = kvmppc_core_check_processor_compat();510 r = kvmppc_e500_check_processor_compat();
511 if (r)511 if (r)
512 goto err_out;512 goto err_out;
513 513
@@ -531,7 +531,7 @@ static int __init kvmppc_e500_init(void) · 17 unmodified lines
531 flush_icache_range(kvmppc_booke_handlers, kvmppc_booke_handlers +531 flush_icache_range(kvmppc_booke_handlers, kvmppc_booke_handlers +
532 ivor[max_ivor] + handler_len);532 ivor[max_ivor] + handler_len);
533 533
534 r = kvm_init(NULL, sizeof(struct kvmppc_vcpu_e500), 0, THIS_MODULE);534 r = kvm_init(sizeof(struct kvmppc_vcpu_e500), 0, THIS_MODULE);
535 if (r)535 if (r)
536 goto err_out;536 goto err_out;
537 kvm_ops_e500.owner = THIS_MODULE;537 kvm_ops_e500.owner = THIS_MODULE;
arch/powerpc/kvm/e500mc.c+6−2
@@ -168,7 +168,7 @@ static void kvmppc_core_vcpu_put_e500mc(struct kvm_vcpu *vcpu) · 167 unmodified lines
168 kvmppc_booke_vcpu_put(vcpu);168 kvmppc_booke_vcpu_put(vcpu);
169}169}
170 170
171int kvmppc_core_check_processor_compat(void)171int kvmppc_e500mc_check_processor_compat(void)
172{172{
173 int r;173 int r;
174 174
@@ -388,6 +388,10 @@ static int __init kvmppc_e500mc_init(void) · 213 unmodified lines
388{388{
389 int r;389 int r;
390 390
391 r = kvmppc_e500mc_check_processor_compat();
392 if (r)
393 goto err_out;
394
391 r = kvmppc_booke_init();395 r = kvmppc_booke_init();
392 if (r)396 if (r)
393 goto err_out;397 goto err_out;
@@ -400,7 +404,7 @@ static int __init kvmppc_e500mc_init(void) · 6 unmodified lines
400 */404 */
401 kvmppc_init_lpid(KVMPPC_NR_LPIDS/threads_per_core);405 kvmppc_init_lpid(KVMPPC_NR_LPIDS/threads_per_core);
402 406
403 r = kvm_init(NULL, sizeof(struct kvmppc_vcpu_e500), 0, THIS_MODULE);407 r = kvm_init(sizeof(struct kvmppc_vcpu_e500), 0, THIS_MODULE);
404 if (r)408 if (r)
405 goto err_out;409 goto err_out;
406 kvm_ops_e500mc.owner = THIS_MODULE;410 kvm_ops_e500mc.owner = THIS_MODULE;
arch/powerpc/kvm/powerpc.c+0−20
@@ -435,21 +435,6 @@ int kvmppc_ld(struct kvm_vcpu *vcpu, ulong *eaddr, int size, void *ptr, · 434 unmodified lines
435}435}
436EXPORT_SYMBOL_GPL(kvmppc_ld);436EXPORT_SYMBOL_GPL(kvmppc_ld);
437 437
438int kvm_arch_hardware_enable(void)
439{
440 return 0;
441}
442
443int kvm_arch_hardware_setup(void *opaque)
444{
445 return 0;
446}
447
448int kvm_arch_check_processor_compat(void *opaque)
449{
450 return kvmppc_core_check_processor_compat();
451}
452
453int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)438int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
454{439{
455 struct kvmppc_ops *kvm_ops = NULL;440 struct kvmppc_ops *kvm_ops = NULL;
@@ -2544,11 +2529,6 @@ void kvmppc_init_lpid(unsigned long nr_lpids_param) · 2088 unmodified lines
2544}2529}
2545EXPORT_SYMBOL_GPL(kvmppc_init_lpid);2530EXPORT_SYMBOL_GPL(kvmppc_init_lpid);
2546 2531
2547int kvm_arch_init(void *opaque)
2548{
2549 return 0;
2550}
2551
2552EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_ppc_instr);2532EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_ppc_instr);
2553 2533
2554void kvm_arch_create_vcpu_debugfs(struct kvm_vcpu *vcpu, struct dentry *debugfs_dentry)2534void kvm_arch_create_vcpu_debugfs(struct kvm_vcpu *vcpu, struct dentry *debugfs_dentry)
arch/riscv/include/asm/kvm_host.h+7−4
@@ -18,6 +18,7 @@ · 17 unmodified lines
18#include <asm/kvm_vcpu_insn.h>18#include <asm/kvm_vcpu_insn.h>
19#include <asm/kvm_vcpu_sbi.h>19#include <asm/kvm_vcpu_sbi.h>
20#include <asm/kvm_vcpu_timer.h>20#include <asm/kvm_vcpu_timer.h>
21#include <asm/kvm_vcpu_pmu.h>
21 22
22#define KVM_MAX_VCPUS 102423#define KVM_MAX_VCPUS 1024
23 24
@@ -228,9 +229,11 @@ struct kvm_vcpu_arch { · 204 unmodified lines
228 229
229 /* Don't run the VCPU (blocked) */230 /* Don't run the VCPU (blocked) */
230 bool pause;231 bool pause;
232
233 /* Performance monitoring context */
234 struct kvm_pmu pmu_context;
231};235};
232 236
233static inline void kvm_arch_hardware_unsetup(void) {}
234static inline void kvm_arch_sync_events(struct kvm *kvm) {}237static inline void kvm_arch_sync_events(struct kvm *kvm) {}
235static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}238static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
236 239
@@ -297,11 +300,11 @@ int kvm_riscv_gstage_map(struct kvm_vcpu *vcpu, · 60 unmodified lines
297int kvm_riscv_gstage_alloc_pgd(struct kvm *kvm);300int kvm_riscv_gstage_alloc_pgd(struct kvm *kvm);
298void kvm_riscv_gstage_free_pgd(struct kvm *kvm);301void kvm_riscv_gstage_free_pgd(struct kvm *kvm);
299void kvm_riscv_gstage_update_hgatp(struct kvm_vcpu *vcpu);302void kvm_riscv_gstage_update_hgatp(struct kvm_vcpu *vcpu);
300void kvm_riscv_gstage_mode_detect(void);303void __init kvm_riscv_gstage_mode_detect(void);
301unsigned long kvm_riscv_gstage_mode(void);304unsigned long __init kvm_riscv_gstage_mode(void);
302int kvm_riscv_gstage_gpa_bits(void);305int kvm_riscv_gstage_gpa_bits(void);
303 306
304void kvm_riscv_gstage_vmid_detect(void);307void __init kvm_riscv_gstage_vmid_detect(void);
305unsigned long kvm_riscv_gstage_vmid_bits(void);308unsigned long kvm_riscv_gstage_vmid_bits(void);
306int kvm_riscv_gstage_vmid_init(struct kvm *kvm);309int kvm_riscv_gstage_vmid_init(struct kvm *kvm);
307bool kvm_riscv_gstage_vmid_ver_changed(struct kvm_vmid *vmid);310bool kvm_riscv_gstage_vmid_ver_changed(struct kvm_vmid *vmid);
arch/riscv/include/asm/kvm_vcpu_pmu.h+107−0
@@ -0,0 +1,107 @@
1/* SPDX-License-Identifier: GPL-2.0-only */
2/*
3 * Copyright (c) 2023 Rivos Inc
4 *
5 * Authors:
6 * Atish Patra <atishp@rivosinc.com>
7 */
8
9#ifndef __KVM_VCPU_RISCV_PMU_H
10#define __KVM_VCPU_RISCV_PMU_H
11
12#include <linux/perf/riscv_pmu.h>
13#include <asm/sbi.h>
14
15#ifdef CONFIG_RISCV_PMU_SBI
16#define RISCV_KVM_MAX_FW_CTRS 32
17#define RISCV_KVM_MAX_HW_CTRS 32
18#define RISCV_KVM_MAX_COUNTERS (RISCV_KVM_MAX_HW_CTRS + RISCV_KVM_MAX_FW_CTRS)
19static_assert(RISCV_KVM_MAX_COUNTERS <= 64);
20
21struct kvm_fw_event {
22 /* Current value of the event */
23 unsigned long value;
24
25 /* Event monitoring status */
26 bool started;
27};
28
29/* Per virtual pmu counter data */
30struct kvm_pmc {
31 u8 idx;
32 struct perf_event *perf_event;
33 u64 counter_val;
34 union sbi_pmu_ctr_info cinfo;
35 /* Event monitoring status */
36 bool started;
37 /* Monitoring event ID */
38 unsigned long event_idx;
39};
40
41/* PMU data structure per vcpu */
42struct kvm_pmu {
43 struct kvm_pmc pmc[RISCV_KVM_MAX_COUNTERS];
44 struct kvm_fw_event fw_event[RISCV_KVM_MAX_FW_CTRS];
45 /* Number of the virtual firmware counters available */
46 int num_fw_ctrs;
47 /* Number of the virtual hardware counters available */
48 int num_hw_ctrs;
49 /* A flag to indicate that pmu initialization is done */
50 bool init_done;
51 /* Bit map of all the virtual counter used */
52 DECLARE_BITMAP(pmc_in_use, RISCV_KVM_MAX_COUNTERS);
53};
54
55#define vcpu_to_pmu(vcpu) (&(vcpu)->arch.pmu_context)
56#define pmu_to_vcpu(pmu) (container_of((pmu), struct kvm_vcpu, arch.pmu_context))
57
58#if defined(CONFIG_32BIT)
59#define KVM_RISCV_VCPU_HPMCOUNTER_CSR_FUNCS \
60{.base = CSR_CYCLEH, .count = 31, .func = kvm_riscv_vcpu_pmu_read_hpm }, \
61{.base = CSR_CYCLE, .count = 31, .func = kvm_riscv_vcpu_pmu_read_hpm },
62#else
63#define KVM_RISCV_VCPU_HPMCOUNTER_CSR_FUNCS \
64{.base = CSR_CYCLE, .count = 31, .func = kvm_riscv_vcpu_pmu_read_hpm },
65#endif
66
67int kvm_riscv_vcpu_pmu_incr_fw(struct kvm_vcpu *vcpu, unsigned long fid);
68int kvm_riscv_vcpu_pmu_read_hpm(struct kvm_vcpu *vcpu, unsigned int csr_num,
69 unsigned long *val, unsigned long new_val,
70 unsigned long wr_mask);
71
72int kvm_riscv_vcpu_pmu_num_ctrs(struct kvm_vcpu *vcpu, struct kvm_vcpu_sbi_return *retdata);
73int kvm_riscv_vcpu_pmu_ctr_info(struct kvm_vcpu *vcpu, unsigned long cidx,
74 struct kvm_vcpu_sbi_return *retdata);
75int kvm_riscv_vcpu_pmu_ctr_start(struct kvm_vcpu *vcpu, unsigned long ctr_base,
76 unsigned long ctr_mask, unsigned long flags, u64 ival,
77 struct kvm_vcpu_sbi_return *retdata);
78int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
79 unsigned long ctr_mask, unsigned long flags,
80 struct kvm_vcpu_sbi_return *retdata);
81int kvm_riscv_vcpu_pmu_ctr_cfg_match(struct kvm_vcpu *vcpu, unsigned long ctr_base,
82 unsigned long ctr_mask, unsigned long flags,
83 unsigned long eidx, u64 evtdata,
84 struct kvm_vcpu_sbi_return *retdata);
85int kvm_riscv_vcpu_pmu_ctr_read(struct kvm_vcpu *vcpu, unsigned long cidx,
86 struct kvm_vcpu_sbi_return *retdata);
87void kvm_riscv_vcpu_pmu_init(struct kvm_vcpu *vcpu);
88void kvm_riscv_vcpu_pmu_deinit(struct kvm_vcpu *vcpu);
89void kvm_riscv_vcpu_pmu_reset(struct kvm_vcpu *vcpu);
90
91#else
92struct kvm_pmu {
93};
94
95#define KVM_RISCV_VCPU_HPMCOUNTER_CSR_FUNCS \
96{.base = 0, .count = 0, .func = NULL },
97
98static inline void kvm_riscv_vcpu_pmu_init(struct kvm_vcpu *vcpu) {}
99static inline int kvm_riscv_vcpu_pmu_incr_fw(struct kvm_vcpu *vcpu, unsigned long fid)
100{
101 return 0;
102}
103
104static inline void kvm_riscv_vcpu_pmu_deinit(struct kvm_vcpu *vcpu) {}
105static inline void kvm_riscv_vcpu_pmu_reset(struct kvm_vcpu *vcpu) {}
106#endif /* CONFIG_RISCV_PMU_SBI */
107#endif /* !__KVM_VCPU_RISCV_PMU_H */
arch/riscv/include/asm/kvm_vcpu_sbi.h+11−2
@@ -18,6 +18,13 @@ struct kvm_vcpu_sbi_context { · 17 unmodified lines
18 int return_handled;18 int return_handled;
19};19};
20 20
21struct kvm_vcpu_sbi_return {
22 unsigned long out_val;
23 unsigned long err_val;
24 struct kvm_cpu_trap *utrap;
25 bool uexit;
26};
27
21struct kvm_vcpu_sbi_extension {28struct kvm_vcpu_sbi_extension {
22 unsigned long extid_start;29 unsigned long extid_start;
23 unsigned long extid_end;30 unsigned long extid_end;
@@ -27,8 +34,10 @@ struct kvm_vcpu_sbi_extension { · 3 unmodified lines
27 * specific error codes.34 * specific error codes.
28 */35 */
29 int (*handler)(struct kvm_vcpu *vcpu, struct kvm_run *run,36 int (*handler)(struct kvm_vcpu *vcpu, struct kvm_run *run,
30 unsigned long *out_val, struct kvm_cpu_trap *utrap,37 struct kvm_vcpu_sbi_return *retdata);
31 bool *exit);38
39 /* Extension specific probe function */
40 unsigned long (*probe)(struct kvm_vcpu *vcpu);
32};41};
33 42
34void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run);43void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run);
arch/riscv/include/asm/sbi.h+5−2
@@ -169,9 +169,9 @@ enum sbi_pmu_fw_generic_events_t { · 168 unmodified lines
169 SBI_PMU_FW_ILLEGAL_INSN = 4,169 SBI_PMU_FW_ILLEGAL_INSN = 4,
170 SBI_PMU_FW_SET_TIMER = 5,170 SBI_PMU_FW_SET_TIMER = 5,
171 SBI_PMU_FW_IPI_SENT = 6,171 SBI_PMU_FW_IPI_SENT = 6,
172 SBI_PMU_FW_IPI_RECVD = 7,172 SBI_PMU_FW_IPI_RCVD = 7,
173 SBI_PMU_FW_FENCE_I_SENT = 8,173 SBI_PMU_FW_FENCE_I_SENT = 8,
174 SBI_PMU_FW_FENCE_I_RECVD = 9,174 SBI_PMU_FW_FENCE_I_RCVD = 9,
175 SBI_PMU_FW_SFENCE_VMA_SENT = 10,175 SBI_PMU_FW_SFENCE_VMA_SENT = 10,
176 SBI_PMU_FW_SFENCE_VMA_RCVD = 11,176 SBI_PMU_FW_SFENCE_VMA_RCVD = 11,
177 SBI_PMU_FW_SFENCE_VMA_ASID_SENT = 12,177 SBI_PMU_FW_SFENCE_VMA_ASID_SENT = 12,
@@ -215,6 +215,9 @@ enum sbi_pmu_ctr_type { · 37 unmodified lines
215#define SBI_PMU_EVENT_CACHE_OP_ID_CODE_MASK 0x06215#define SBI_PMU_EVENT_CACHE_OP_ID_CODE_MASK 0x06
216#define SBI_PMU_EVENT_CACHE_RESULT_ID_CODE_MASK 0x01216#define SBI_PMU_EVENT_CACHE_RESULT_ID_CODE_MASK 0x01
217 217
218#define SBI_PMU_EVENT_CACHE_ID_SHIFT 3
219#define SBI_PMU_EVENT_CACHE_OP_SHIFT 1
220
218#define SBI_PMU_EVENT_IDX_INVALID 0xFFFFFFFF221#define SBI_PMU_EVENT_IDX_INVALID 0xFFFFFFFF
219 222
220/* Flags defined for config matching function */223/* Flags defined for config matching function */
arch/riscv/kvm/Kconfig+1−0
@@ -20,6 +20,7 @@ if VIRTUALIZATION · 19 unmodified lines
20config KVM20config KVM
21 tristate "Kernel-based Virtual Machine (KVM) support (EXPERIMENTAL)"21 tristate "Kernel-based Virtual Machine (KVM) support (EXPERIMENTAL)"
22 depends on RISCV_SBI && MMU22 depends on RISCV_SBI && MMU
23 select KVM_GENERIC_HARDWARE_ENABLING
23 select MMU_NOTIFIER24 select MMU_NOTIFIER
24 select PREEMPT_NOTIFIERS25 select PREEMPT_NOTIFIERS
25 select KVM_MMIO26 select KVM_MMIO
arch/riscv/kvm/Makefile+1−0
@@ -25,3 +25,4 @@ kvm-y += vcpu_sbi_base.o · 24 unmodified lines
25kvm-y += vcpu_sbi_replace.o25kvm-y += vcpu_sbi_replace.o
26kvm-y += vcpu_sbi_hsm.o26kvm-y += vcpu_sbi_hsm.o
27kvm-y += vcpu_timer.o27kvm-y += vcpu_timer.o
28kvm-$(CONFIG_RISCV_PMU_SBI) += vcpu_pmu.o vcpu_sbi_pmu.o
arch/riscv/kvm/main.c+4−22
@@ -20,16 +20,6 @@ long kvm_arch_dev_ioctl(struct file *filp, · 19 unmodified lines
20 return -EINVAL;20 return -EINVAL;
21}21}
22 22
23int kvm_arch_check_processor_compat(void *opaque)
24{
25 return 0;
26}
27
28int kvm_arch_hardware_setup(void *opaque)
29{
30 return 0;
31}
32
33int kvm_arch_hardware_enable(void)23int kvm_arch_hardware_enable(void)
34{24{
35 unsigned long hideleg, hedeleg;25 unsigned long hideleg, hedeleg;
@@ -49,7 +39,8 @@ int kvm_arch_hardware_enable(void) · 13 unmodified lines
49 hideleg |= (1UL << IRQ_VS_EXT);39 hideleg |= (1UL << IRQ_VS_EXT);
50 csr_write(CSR_HIDELEG, hideleg);40 csr_write(CSR_HIDELEG, hideleg);
51 41
52 csr_write(CSR_HCOUNTEREN, -1UL);42 /* VS should access only the time counter directly. Everything else should trap */
43 csr_write(CSR_HCOUNTEREN, 0x02);
53 44
54 csr_write(CSR_HVIP, 0);45 csr_write(CSR_HVIP, 0);
55 46
@@ -70,7 +61,7 @@ void kvm_arch_hardware_disable(void) · 14 unmodified lines
70 csr_write(CSR_HIDELEG, 0);61 csr_write(CSR_HIDELEG, 0);
71}62}
72 63
73int kvm_arch_init(void *opaque)64static int __init riscv_kvm_init(void)
74{65{
75 const char *str;66 const char *str;
76 67
@@ -115,16 +106,7 @@ int kvm_arch_init(void *opaque) · 38 unmodified lines
115 106
116 kvm_info("VMID %ld bits available\n", kvm_riscv_gstage_vmid_bits());107 kvm_info("VMID %ld bits available\n", kvm_riscv_gstage_vmid_bits());
117 108
118 return 0;109 return kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
119}
120
121void kvm_arch_exit(void)
122{
123}
124
125static int __init riscv_kvm_init(void)
126{
127 return kvm_init(NULL, sizeof(struct kvm_vcpu), 0, THIS_MODULE);
128}110}
129module_init(riscv_kvm_init);111module_init(riscv_kvm_init);
130 112
arch/riscv/kvm/mmu.c+10−10
@@ -20,12 +20,12 @@ · 19 unmodified lines
20#include <asm/pgtable.h>20#include <asm/pgtable.h>
21 21
22#ifdef CONFIG_64BIT22#ifdef CONFIG_64BIT
23static unsigned long gstage_mode = (HGATP_MODE_SV39X4 << HGATP_MODE_SHIFT);23static unsigned long gstage_mode __ro_after_init = (HGATP_MODE_SV39X4 << HGATP_MODE_SHIFT);
24static unsigned long gstage_pgd_levels = 3;24static unsigned long gstage_pgd_levels __ro_after_init = 3;
25#define gstage_index_bits 925#define gstage_index_bits 9
26#else26#else
27static unsigned long gstage_mode = (HGATP_MODE_SV32X4 << HGATP_MODE_SHIFT);27static unsigned long gstage_mode __ro_after_init = (HGATP_MODE_SV32X4 << HGATP_MODE_SHIFT);
28static unsigned long gstage_pgd_levels = 2;28static unsigned long gstage_pgd_levels __ro_after_init = 2;
29#define gstage_index_bits 1029#define gstage_index_bits 10
30#endif30#endif
31 31
@@ -585,7 +585,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · 553 unmodified lines
585 if (!kvm->arch.pgd)585 if (!kvm->arch.pgd)
586 return false;586 return false;
587 587
588 WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PGDIR_SIZE);588 WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE);
589 589
590 if (!gstage_get_leaf_entry(kvm, range->start << PAGE_SHIFT,590 if (!gstage_get_leaf_entry(kvm, range->start << PAGE_SHIFT,
591 &ptep, &ptep_level))591 &ptep, &ptep_level))
@@ -603,7 +603,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · 11 unmodified lines
603 if (!kvm->arch.pgd)603 if (!kvm->arch.pgd)
604 return false;604 return false;
605 605
606 WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PGDIR_SIZE);606 WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE);
607 607
608 if (!gstage_get_leaf_entry(kvm, range->start << PAGE_SHIFT,608 if (!gstage_get_leaf_entry(kvm, range->start << PAGE_SHIFT,
609 &ptep, &ptep_level))609 &ptep, &ptep_level))
@@ -645,12 +645,12 @@ int kvm_riscv_gstage_map(struct kvm_vcpu *vcpu, · 35 unmodified lines
645 if (logging || (vma->vm_flags & VM_PFNMAP))645 if (logging || (vma->vm_flags & VM_PFNMAP))
646 vma_pagesize = PAGE_SIZE;646 vma_pagesize = PAGE_SIZE;
647 647
648 if (vma_pagesize == PMD_SIZE || vma_pagesize == PGDIR_SIZE)648 if (vma_pagesize == PMD_SIZE || vma_pagesize == PUD_SIZE)
649 gfn = (gpa & huge_page_mask(hstate_vma(vma))) >> PAGE_SHIFT;649 gfn = (gpa & huge_page_mask(hstate_vma(vma))) >> PAGE_SHIFT;
650 650
651 mmap_read_unlock(current->mm);651 mmap_read_unlock(current->mm);
652 652
653 if (vma_pagesize != PGDIR_SIZE &&653 if (vma_pagesize != PUD_SIZE &&
654 vma_pagesize != PMD_SIZE &&654 vma_pagesize != PMD_SIZE &&
655 vma_pagesize != PAGE_SIZE) {655 vma_pagesize != PAGE_SIZE) {
656 kvm_err("Invalid VMA page size 0x%lx\n", vma_pagesize);656 kvm_err("Invalid VMA page size 0x%lx\n", vma_pagesize);
@@ -758,7 +758,7 @@ void kvm_riscv_gstage_update_hgatp(struct kvm_vcpu *vcpu) · 101 unmodified lines
758 kvm_riscv_local_hfence_gvma_all();758 kvm_riscv_local_hfence_gvma_all();
759}759}
760 760
761void kvm_riscv_gstage_mode_detect(void)761void __init kvm_riscv_gstage_mode_detect(void)
762{762{
763#ifdef CONFIG_64BIT763#ifdef CONFIG_64BIT
764 /* Try Sv57x4 G-stage mode */764 /* Try Sv57x4 G-stage mode */
@@ -782,7 +782,7 @@ skip_sv48x4_test: · 17 unmodified lines
782#endif782#endif
783}783}
784 784
785unsigned long kvm_riscv_gstage_mode(void)785unsigned long __init kvm_riscv_gstage_mode(void)
786{786{
787 return gstage_mode >> HGATP_MODE_SHIFT;787 return gstage_mode >> HGATP_MODE_SHIFT;
788}788}
arch/riscv/kvm/tlb.c+4−0
@@ -180,6 +180,7 @@ void kvm_riscv_local_tlb_sanitize(struct kvm_vcpu *vcpu) · 179 unmodified lines
180 180
181void kvm_riscv_fence_i_process(struct kvm_vcpu *vcpu)181void kvm_riscv_fence_i_process(struct kvm_vcpu *vcpu)
182{182{
183 kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_FENCE_I_RCVD);
183 local_flush_icache_all();184 local_flush_icache_all();
184}185}
185 186
@@ -263,15 +264,18 @@ void kvm_riscv_hfence_process(struct kvm_vcpu *vcpu) · 77 unmodified lines
263 d.addr, d.size, d.order);264 d.addr, d.size, d.order);
264 break;265 break;
265 case KVM_RISCV_HFENCE_VVMA_ASID_GVA:266 case KVM_RISCV_HFENCE_VVMA_ASID_GVA:
267 kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_ASID_RCVD);
266 kvm_riscv_local_hfence_vvma_asid_gva(268 kvm_riscv_local_hfence_vvma_asid_gva(
267 READ_ONCE(v->vmid), d.asid,269 READ_ONCE(v->vmid), d.asid,
268 d.addr, d.size, d.order);270 d.addr, d.size, d.order);
269 break;271 break;
270 case KVM_RISCV_HFENCE_VVMA_ASID_ALL:272 case KVM_RISCV_HFENCE_VVMA_ASID_ALL:
273 kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_ASID_RCVD);
271 kvm_riscv_local_hfence_vvma_asid_all(274 kvm_riscv_local_hfence_vvma_asid_all(
272 READ_ONCE(v->vmid), d.asid);275 READ_ONCE(v->vmid), d.asid);
273 break;276 break;
274 case KVM_RISCV_HFENCE_VVMA_GVA:277 case KVM_RISCV_HFENCE_VVMA_GVA:
278 kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_RCVD);
275 kvm_riscv_local_hfence_vvma_gva(279 kvm_riscv_local_hfence_vvma_gva(
276 READ_ONCE(v->vmid),280 READ_ONCE(v->vmid),
277 d.addr, d.size, d.order);281 d.addr, d.size, d.order);
arch/riscv/kvm/vcpu.c+7−0
@@ -138,6 +138,8 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu) · 137 unmodified lines
138 WRITE_ONCE(vcpu->arch.irqs_pending, 0);138 WRITE_ONCE(vcpu->arch.irqs_pending, 0);
139 WRITE_ONCE(vcpu->arch.irqs_pending_mask, 0);139 WRITE_ONCE(vcpu->arch.irqs_pending_mask, 0);
140 140
141 kvm_riscv_vcpu_pmu_reset(vcpu);
142
141 vcpu->arch.hfence_head = 0;143 vcpu->arch.hfence_head = 0;
142 vcpu->arch.hfence_tail = 0;144 vcpu->arch.hfence_tail = 0;
143 memset(vcpu->arch.hfence_queue, 0, sizeof(vcpu->arch.hfence_queue));145 memset(vcpu->arch.hfence_queue, 0, sizeof(vcpu->arch.hfence_queue));
@@ -194,6 +196,9 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · 50 unmodified lines
194 /* Setup VCPU timer */196 /* Setup VCPU timer */
195 kvm_riscv_vcpu_timer_init(vcpu);197 kvm_riscv_vcpu_timer_init(vcpu);
196 198
199 /* setup performance monitoring */
200 kvm_riscv_vcpu_pmu_init(vcpu);
201
197 /* Reset VCPU */202 /* Reset VCPU */
198 kvm_riscv_reset_vcpu(vcpu);203 kvm_riscv_reset_vcpu(vcpu);
199 204
@@ -216,6 +221,8 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) · 16 unmodified lines
216 /* Cleanup VCPU timer */221 /* Cleanup VCPU timer */
217 kvm_riscv_vcpu_timer_deinit(vcpu);222 kvm_riscv_vcpu_timer_deinit(vcpu);
218 223
224 kvm_riscv_vcpu_pmu_deinit(vcpu);
225
219 /* Free unused pages pre-allocated for G-stage page table mappings */226 /* Free unused pages pre-allocated for G-stage page table mappings */
220 kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache);227 kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache);
221}228}
arch/riscv/kvm/vcpu_exit.c+9−0
@@ -160,6 +160,9 @@ void kvm_riscv_vcpu_trap_redirect(struct kvm_vcpu *vcpu, · 159 unmodified lines
160 160
161 /* Set Guest PC to Guest exception vector */161 /* Set Guest PC to Guest exception vector */
162 vcpu->arch.guest_context.sepc = csr_read(CSR_VSTVEC);162 vcpu->arch.guest_context.sepc = csr_read(CSR_VSTVEC);
163
164 /* Set Guest privilege mode to supervisor */
165 vcpu->arch.guest_context.sstatus |= SR_SPP;
163}166}
164 167
165/*168/*
@@ -179,6 +182,12 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run, · 13 unmodified lines
179 ret = -EFAULT;182 ret = -EFAULT;
180 run->exit_reason = KVM_EXIT_UNKNOWN;183 run->exit_reason = KVM_EXIT_UNKNOWN;
181 switch (trap->scause) {184 switch (trap->scause) {
185 case EXC_INST_ILLEGAL:
186 if (vcpu->arch.guest_context.hstatus & HSTATUS_SPV) {
187 kvm_riscv_vcpu_trap_redirect(vcpu, trap);
188 ret = 1;
189 }
190 break;
182 case EXC_VIRTUAL_INST_FAULT:191 case EXC_VIRTUAL_INST_FAULT:
183 if (vcpu->arch.guest_context.hstatus & HSTATUS_SPV)192 if (vcpu->arch.guest_context.hstatus & HSTATUS_SPV)
184 ret = kvm_riscv_vcpu_virtual_insn(vcpu, run, trap);193 ret = kvm_riscv_vcpu_virtual_insn(vcpu, run, trap);
arch/riscv/kvm/vcpu_insn.c+3−1
@@ -213,7 +213,9 @@ struct csr_func { · 212 unmodified lines
213 unsigned long wr_mask);213 unsigned long wr_mask);
214};214};
215 215
216static const struct csr_func csr_funcs[] = { };216static const struct csr_func csr_funcs[] = {
217 KVM_RISCV_VCPU_HPMCOUNTER_CSR_FUNCS
218};
217 219
218/**220/**
219 * kvm_riscv_vcpu_csr_return -- Handle CSR read/write after user space221 * kvm_riscv_vcpu_csr_return -- Handle CSR read/write after user space
arch/riscv/kvm/vcpu_pmu.c+633−0
@@ -0,0 +1,633 @@
1// SPDX-License-Identifier: GPL-2.0
2/*
3 * Copyright (c) 2023 Rivos Inc
4 *
5 * Authors:
6 * Atish Patra <atishp@rivosinc.com>
7 */
8
9#define pr_fmt(fmt) "riscv-kvm-pmu: " fmt
10#include <linux/errno.h>
11#include <linux/err.h>
12#include <linux/kvm_host.h>
13#include <linux/perf/riscv_pmu.h>
14#include <asm/csr.h>
15#include <asm/kvm_vcpu_sbi.h>
16#include <asm/kvm_vcpu_pmu.h>
17#include <linux/bitops.h>
18
19#define kvm_pmu_num_counters(pmu) ((pmu)->num_hw_ctrs + (pmu)->num_fw_ctrs)
20#define get_event_type(x) (((x) & SBI_PMU_EVENT_IDX_TYPE_MASK) >> 16)
21#define get_event_code(x) ((x) & SBI_PMU_EVENT_IDX_CODE_MASK)
22
23static enum perf_hw_id hw_event_perf_map[SBI_PMU_HW_GENERAL_MAX] = {
24 [SBI_PMU_HW_CPU_CYCLES] = PERF_COUNT_HW_CPU_CYCLES,
25 [SBI_PMU_HW_INSTRUCTIONS] = PERF_COUNT_HW_INSTRUCTIONS,
26 [SBI_PMU_HW_CACHE_REFERENCES] = PERF_COUNT_HW_CACHE_REFERENCES,
27 [SBI_PMU_HW_CACHE_MISSES] = PERF_COUNT_HW_CACHE_MISSES,
28 [SBI_PMU_HW_BRANCH_INSTRUCTIONS] = PERF_COUNT_HW_BRANCH_INSTRUCTIONS,
29 [SBI_PMU_HW_BRANCH_MISSES] = PERF_COUNT_HW_BRANCH_MISSES,
30 [SBI_PMU_HW_BUS_CYCLES] = PERF_COUNT_HW_BUS_CYCLES,
31 [SBI_PMU_HW_STALLED_CYCLES_FRONTEND] = PERF_COUNT_HW_STALLED_CYCLES_FRONTEND,
32 [SBI_PMU_HW_STALLED_CYCLES_BACKEND] = PERF_COUNT_HW_STALLED_CYCLES_BACKEND,
33 [SBI_PMU_HW_REF_CPU_CYCLES] = PERF_COUNT_HW_REF_CPU_CYCLES,
34};
35
36static u64 kvm_pmu_get_sample_period(struct kvm_pmc *pmc)
37{
38 u64 counter_val_mask = GENMASK(pmc->cinfo.width, 0);
39 u64 sample_period;
40
41 if (!pmc->counter_val)
42 sample_period = counter_val_mask + 1;
43 else
44 sample_period = (-pmc->counter_val) & counter_val_mask;
45
46 return sample_period;
47}
48
49static u32 kvm_pmu_get_perf_event_type(unsigned long eidx)
50{
51 enum sbi_pmu_event_type etype = get_event_type(eidx);
52 u32 type = PERF_TYPE_MAX;
53
54 switch (etype) {
55 case SBI_PMU_EVENT_TYPE_HW:
56 type = PERF_TYPE_HARDWARE;
57 break;
58 case SBI_PMU_EVENT_TYPE_CACHE:
59 type = PERF_TYPE_HW_CACHE;
60 break;
61 case SBI_PMU_EVENT_TYPE_RAW:
62 case SBI_PMU_EVENT_TYPE_FW:
63 type = PERF_TYPE_RAW;
64 break;
65 default:
66 break;
67 }
68
69 return type;
70}
71
72static bool kvm_pmu_is_fw_event(unsigned long eidx)
73{
74 return get_event_type(eidx) == SBI_PMU_EVENT_TYPE_FW;
75}
76
77static void kvm_pmu_release_perf_event(struct kvm_pmc *pmc)
78{
79 if (pmc->perf_event) {
80 perf_event_disable(pmc->perf_event);
81 perf_event_release_kernel(pmc->perf_event);
82 pmc->perf_event = NULL;
83 }
84}
85
86static u64 kvm_pmu_get_perf_event_hw_config(u32 sbi_event_code)
87{
88 return hw_event_perf_map[sbi_event_code];
89}
90
91static u64 kvm_pmu_get_perf_event_cache_config(u32 sbi_event_code)
92{
93 u64 config = U64_MAX;
94 unsigned int cache_type, cache_op, cache_result;
95
96 /* All the cache event masks lie within 0xFF. No separate masking is necessary */
97 cache_type = (sbi_event_code & SBI_PMU_EVENT_CACHE_ID_CODE_MASK) >>
98 SBI_PMU_EVENT_CACHE_ID_SHIFT;
99 cache_op = (sbi_event_code & SBI_PMU_EVENT_CACHE_OP_ID_CODE_MASK) >>
100 SBI_PMU_EVENT_CACHE_OP_SHIFT;
101 cache_result = sbi_event_code & SBI_PMU_EVENT_CACHE_RESULT_ID_CODE_MASK;
102
103 if (cache_type >= PERF_COUNT_HW_CACHE_MAX ||
104 cache_op >= PERF_COUNT_HW_CACHE_OP_MAX ||
105 cache_result >= PERF_COUNT_HW_CACHE_RESULT_MAX)
106 return config;
107
108 config = cache_type | (cache_op << 8) | (cache_result << 16);
109
110 return config;
111}
112
113static u64 kvm_pmu_get_perf_event_config(unsigned long eidx, uint64_t evt_data)
114{
115 enum sbi_pmu_event_type etype = get_event_type(eidx);
116 u32 ecode = get_event_code(eidx);
117 u64 config = U64_MAX;
118
119 switch (etype) {
120 case SBI_PMU_EVENT_TYPE_HW:
121 if (ecode < SBI_PMU_HW_GENERAL_MAX)
122 config = kvm_pmu_get_perf_event_hw_config(ecode);
123 break;
124 case SBI_PMU_EVENT_TYPE_CACHE:
125 config = kvm_pmu_get_perf_event_cache_config(ecode);
126 break;
127 case SBI_PMU_EVENT_TYPE_RAW:
128 config = evt_data & RISCV_PMU_RAW_EVENT_MASK;
129 break;
130 case SBI_PMU_EVENT_TYPE_FW:
131 if (ecode < SBI_PMU_FW_MAX)
132 config = (1ULL << 63) | ecode;
133 break;
134 default:
135 break;
136 }
137
138 return config;
139}
140
141static int kvm_pmu_get_fixed_pmc_index(unsigned long eidx)
142{
143 u32 etype = kvm_pmu_get_perf_event_type(eidx);
144 u32 ecode = get_event_code(eidx);
145
146 if (etype != SBI_PMU_EVENT_TYPE_HW)
147 return -EINVAL;
148
149 if (ecode == SBI_PMU_HW_CPU_CYCLES)
150 return 0;
151 else if (ecode == SBI_PMU_HW_INSTRUCTIONS)
152 return 2;
153 else
154 return -EINVAL;
155}
156
157static int kvm_pmu_get_programmable_pmc_index(struct kvm_pmu *kvpmu, unsigned long eidx,
158 unsigned long cbase, unsigned long cmask)
159{
160 int ctr_idx = -1;
161 int i, pmc_idx;
162 int min, max;
163
164 if (kvm_pmu_is_fw_event(eidx)) {
165 /* Firmware counters are mapped 1:1 starting from num_hw_ctrs for simplicity */
166 min = kvpmu->num_hw_ctrs;
167 max = min + kvpmu->num_fw_ctrs;
168 } else {
169 /* First 3 counters are reserved for fixed counters */
170 min = 3;
171 max = kvpmu->num_hw_ctrs;
172 }
173
174 for_each_set_bit(i, &cmask, BITS_PER_LONG) {
175 pmc_idx = i + cbase;
176 if ((pmc_idx >= min && pmc_idx < max) &&
177 !test_bit(pmc_idx, kvpmu->pmc_in_use)) {
178 ctr_idx = pmc_idx;
179 break;
180 }
181 }
182
183 return ctr_idx;
184}
185
186static int pmu_get_pmc_index(struct kvm_pmu *pmu, unsigned long eidx,
187 unsigned long cbase, unsigned long cmask)
188{
189 int ret;
190
191 /* Fixed counters need to be have fixed mapping as they have different width */
192 ret = kvm_pmu_get_fixed_pmc_index(eidx);
193 if (ret >= 0)
194 return ret;
195
196 return kvm_pmu_get_programmable_pmc_index(pmu, eidx, cbase, cmask);
197}
198
199static int pmu_ctr_read(struct kvm_vcpu *vcpu, unsigned long cidx,
200 unsigned long *out_val)
201{
202 struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
203 struct kvm_pmc *pmc;
204 u64 enabled, running;
205 int fevent_code;
206
207 pmc = &kvpmu->pmc[cidx];
208
209 if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW) {
210 fevent_code = get_event_code(pmc->event_idx);
211 pmc->counter_val = kvpmu->fw_event[fevent_code].value;
212 } else if (pmc->perf_event) {
213 pmc->counter_val += perf_event_read_value(pmc->perf_event, &enabled, &running);
214 } else {
215 return -EINVAL;
216 }
217 *out_val = pmc->counter_val;
218
219 return 0;
220}
221
222static int kvm_pmu_validate_counter_mask(struct kvm_pmu *kvpmu, unsigned long ctr_base,
223 unsigned long ctr_mask)
224{
225 /* Make sure the we have a valid counter mask requested from the caller */
226 if (!ctr_mask || (ctr_base + __fls(ctr_mask) >= kvm_pmu_num_counters(kvpmu)))
227 return -EINVAL;
228
229 return 0;
230}
231
232static int kvm_pmu_create_perf_event(struct kvm_pmc *pmc, struct perf_event_attr *attr,
233 unsigned long flags, unsigned long eidx, unsigned long evtdata)
234{
235 struct perf_event *event;
236
237 kvm_pmu_release_perf_event(pmc);
238 attr->config = kvm_pmu_get_perf_event_config(eidx, evtdata);
239 if (flags & SBI_PMU_CFG_FLAG_CLEAR_VALUE) {
240 //TODO: Do we really want to clear the value in hardware counter
241 pmc->counter_val = 0;
242 }
243
244 /*
245 * Set the default sample_period for now. The guest specified value
246 * will be updated in the start call.
247 */
248 attr->sample_period = kvm_pmu_get_sample_period(pmc);
249
250 event = perf_event_create_kernel_counter(attr, -1, current, NULL, pmc);
251 if (IS_ERR(event)) {
252 pr_err("kvm pmu event creation failed for eidx %lx: %ld\n", eidx, PTR_ERR(event));
253 return PTR_ERR(event);
254 }
255
256 pmc->perf_event = event;
257 if (flags & SBI_PMU_CFG_FLAG_AUTO_START)
258 perf_event_enable(pmc->perf_event);
259
260 return 0;
261}
262
263int kvm_riscv_vcpu_pmu_incr_fw(struct kvm_vcpu *vcpu, unsigned long fid)
264{
265 struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
266 struct kvm_fw_event *fevent;
267
268 if (!kvpmu || fid >= SBI_PMU_FW_MAX)
269 return -EINVAL;
270
271 fevent = &kvpmu->fw_event[fid];
272 if (fevent->started)
273 fevent->value++;
274
275 return 0;
276}
277
278int kvm_riscv_vcpu_pmu_read_hpm(struct kvm_vcpu *vcpu, unsigned int csr_num,
279 unsigned long *val, unsigned long new_val,
280 unsigned long wr_mask)
281{
282 struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
283 int cidx, ret = KVM_INSN_CONTINUE_NEXT_SEPC;
284
285 if (!kvpmu || !kvpmu->init_done) {
286 /*
287 * In absence of sscofpmf in the platform, the guest OS may use
288 * the legacy PMU driver to read cycle/instret. In that case,
289 * just return 0 to avoid any illegal trap. However, any other
290 * hpmcounter access should result in illegal trap as they must
291 * be access through SBI PMU only.
292 */
293 if (csr_num == CSR_CYCLE || csr_num == CSR_INSTRET) {
294 *val = 0;
295 return ret;
296 } else {
297 return KVM_INSN_ILLEGAL_TRAP;
298 }
299 }
300
301 /* The counter CSR are read only. Thus, any write should result in illegal traps */
302 if (wr_mask)
303 return KVM_INSN_ILLEGAL_TRAP;
304
305 cidx = csr_num - CSR_CYCLE;
306
307 if (pmu_ctr_read(vcpu, cidx, val) < 0)
308 return KVM_INSN_ILLEGAL_TRAP;
309
310 return ret;
311}
312
313int kvm_riscv_vcpu_pmu_num_ctrs(struct kvm_vcpu *vcpu,
314 struct kvm_vcpu_sbi_return *retdata)
315{
316 struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
317
318 retdata->out_val = kvm_pmu_num_counters(kvpmu);
319
320 return 0;
321}
322
323int kvm_riscv_vcpu_pmu_ctr_info(struct kvm_vcpu *vcpu, unsigned long cidx,
324 struct kvm_vcpu_sbi_return *retdata)
325{
326 struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
327
328 if (cidx > RISCV_KVM_MAX_COUNTERS || cidx == 1) {
329 retdata->err_val = SBI_ERR_INVALID_PARAM;
330 return 0;
331 }
332
333 retdata->out_val = kvpmu->pmc[cidx].cinfo.value;
334
335 return 0;
336}
337
338int kvm_riscv_vcpu_pmu_ctr_start(struct kvm_vcpu *vcpu, unsigned long ctr_base,
339 unsigned long ctr_mask, unsigned long flags, u64 ival,
340 struct kvm_vcpu_sbi_return *retdata)
341{
342 struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
343 int i, pmc_index, sbiret = 0;
344 struct kvm_pmc *pmc;
345 int fevent_code;
346
347 if (kvm_pmu_validate_counter_mask(kvpmu, ctr_base, ctr_mask) < 0) {
348 sbiret = SBI_ERR_INVALID_PARAM;
349 goto out;
350 }
351
352 /* Start the counters that have been configured and requested by the guest */
353 for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) {
354 pmc_index = i + ctr_base;
355 if (!test_bit(pmc_index, kvpmu->pmc_in_use))
356 continue;
357 pmc = &kvpmu->pmc[pmc_index];
358 if (flags & SBI_PMU_START_FLAG_SET_INIT_VALUE)
359 pmc->counter_val = ival;
360 if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW) {
361 fevent_code = get_event_code(pmc->event_idx);
362 if (fevent_code >= SBI_PMU_FW_MAX) {
363 sbiret = SBI_ERR_INVALID_PARAM;
364 goto out;
365 }
366
367 /* Check if the counter was already started for some reason */
368 if (kvpmu->fw_event[fevent_code].started) {
369 sbiret = SBI_ERR_ALREADY_STARTED;
370 continue;
371 }
372
373 kvpmu->fw_event[fevent_code].started = true;
374 kvpmu->fw_event[fevent_code].value = pmc->counter_val;
375 } else if (pmc->perf_event) {
376 if (unlikely(pmc->started)) {
377 sbiret = SBI_ERR_ALREADY_STARTED;
378 continue;
379 }
380 perf_event_period(pmc->perf_event, kvm_pmu_get_sample_period(pmc));
381 perf_event_enable(pmc->perf_event);
382 pmc->started = true;
383 } else {
384 sbiret = SBI_ERR_INVALID_PARAM;
385 }
386 }
387
388out:
389 retdata->err_val = sbiret;
390
391 return 0;
392}
393
394int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
395 unsigned long ctr_mask, unsigned long flags,
396 struct kvm_vcpu_sbi_return *retdata)
397{
398 struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
399 int i, pmc_index, sbiret = 0;
400 u64 enabled, running;
401 struct kvm_pmc *pmc;
402 int fevent_code;
403
404 if (kvm_pmu_validate_counter_mask(kvpmu, ctr_base, ctr_mask) < 0) {
405 sbiret = SBI_ERR_INVALID_PARAM;
406 goto out;
407 }
408
409 /* Stop the counters that have been configured and requested by the guest */
410 for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) {
411 pmc_index = i + ctr_base;
412 if (!test_bit(pmc_index, kvpmu->pmc_in_use))
413 continue;
414 pmc = &kvpmu->pmc[pmc_index];
415 if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW) {
416 fevent_code = get_event_code(pmc->event_idx);
417 if (fevent_code >= SBI_PMU_FW_MAX) {
418 sbiret = SBI_ERR_INVALID_PARAM;
419 goto out;
420 }
421
422 if (!kvpmu->fw_event[fevent_code].started)
423 sbiret = SBI_ERR_ALREADY_STOPPED;
424
425 kvpmu->fw_event[fevent_code].started = false;
426 } else if (pmc->perf_event) {
427 if (pmc->started) {
428 /* Stop counting the counter */
429 perf_event_disable(pmc->perf_event);
430 pmc->started = false;
431 } else {
432 sbiret = SBI_ERR_ALREADY_STOPPED;
433 }
434
435 if (flags & SBI_PMU_STOP_FLAG_RESET) {
436 /* Relase the counter if this is a reset request */
437 pmc->counter_val += perf_event_read_value(pmc->perf_event,
438 &enabled, &running);
439 kvm_pmu_release_perf_event(pmc);
440 }
441 } else {
442 sbiret = SBI_ERR_INVALID_PARAM;
443 }
444 if (flags & SBI_PMU_STOP_FLAG_RESET) {
445 pmc->event_idx = SBI_PMU_EVENT_IDX_INVALID;
446 clear_bit(pmc_index, kvpmu->pmc_in_use);
447 }
448 }
449
450out:
451 retdata->err_val = sbiret;
452
453 return 0;
454}
455
456int kvm_riscv_vcpu_pmu_ctr_cfg_match(struct kvm_vcpu *vcpu, unsigned long ctr_base,
457 unsigned long ctr_mask, unsigned long flags,
458 unsigned long eidx, u64 evtdata,
459 struct kvm_vcpu_sbi_return *retdata)
460{
461 int ctr_idx, ret, sbiret = 0;
462 bool is_fevent;
463 unsigned long event_code;
464 u32 etype = kvm_pmu_get_perf_event_type(eidx);
465 struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
466 struct kvm_pmc *pmc = NULL;
467 struct perf_event_attr attr = {
468 .type = etype,
469 .size = sizeof(struct perf_event_attr),
470 .pinned = true,
471 /*
472 * It should never reach here if the platform doesn't support the sscofpmf
473 * extension as mode filtering won't work without it.
474 */
475 .exclude_host = true,
476 .exclude_hv = true,
477 .exclude_user = !!(flags & SBI_PMU_CFG_FLAG_SET_UINH),
478 .exclude_kernel = !!(flags & SBI_PMU_CFG_FLAG_SET_SINH),
479 .config1 = RISCV_PMU_CONFIG1_GUEST_EVENTS,
480 };
481
482 if (kvm_pmu_validate_counter_mask(kvpmu, ctr_base, ctr_mask) < 0) {
483 sbiret = SBI_ERR_INVALID_PARAM;
484 goto out;
485 }
486
487 event_code = get_event_code(eidx);
488 is_fevent = kvm_pmu_is_fw_event(eidx);
489 if (is_fevent && event_code >= SBI_PMU_FW_MAX) {
490 sbiret = SBI_ERR_NOT_SUPPORTED;
491 goto out;
492 }
493
494 /*
495 * SKIP_MATCH flag indicates the caller is aware of the assigned counter
496 * for this event. Just do a sanity check if it already marked used.
497 */
498 if (flags & SBI_PMU_CFG_FLAG_SKIP_MATCH) {
499 if (!test_bit(ctr_base + __ffs(ctr_mask), kvpmu->pmc_in_use)) {
500 sbiret = SBI_ERR_FAILURE;
501 goto out;
502 }
503 ctr_idx = ctr_base + __ffs(ctr_mask);
504 } else {
505 ctr_idx = pmu_get_pmc_index(kvpmu, eidx, ctr_base, ctr_mask);
506 if (ctr_idx < 0) {
507 sbiret = SBI_ERR_NOT_SUPPORTED;
508 goto out;
509 }
510 }
511
512 pmc = &kvpmu->pmc[ctr_idx];
513 pmc->idx = ctr_idx;
514
515 if (is_fevent) {
516 if (flags & SBI_PMU_CFG_FLAG_AUTO_START)
517 kvpmu->fw_event[event_code].started = true;
518 } else {
519 ret = kvm_pmu_create_perf_event(pmc, &attr, flags, eidx, evtdata);
520 if (ret)
521 return ret;
522 }
523
524 set_bit(ctr_idx, kvpmu->pmc_in_use);
525 pmc->event_idx = eidx;
526 retdata->out_val = ctr_idx;
527out:
528 retdata->err_val = sbiret;
529
530 return 0;
531}
532
533int kvm_riscv_vcpu_pmu_ctr_read(struct kvm_vcpu *vcpu, unsigned long cidx,
534 struct kvm_vcpu_sbi_return *retdata)
535{
536 int ret;
537
538 ret = pmu_ctr_read(vcpu, cidx, &retdata->out_val);
539 if (ret == -EINVAL)
540 retdata->err_val = SBI_ERR_INVALID_PARAM;
541
542 return 0;
543}
544
545void kvm_riscv_vcpu_pmu_init(struct kvm_vcpu *vcpu)
546{
547 int i = 0, ret, num_hw_ctrs = 0, hpm_width = 0;
548 struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
549 struct kvm_pmc *pmc;
550
551 /*
552 * PMU functionality should be only available to guests if privilege mode
553 * filtering is available in the host. Otherwise, guest will always count
554 * events while the execution is in hypervisor mode.
555 */
556 if (!riscv_isa_extension_available(NULL, SSCOFPMF))
557 return;
558
559 ret = riscv_pmu_get_hpm_info(&hpm_width, &num_hw_ctrs);
560 if (ret < 0 || !hpm_width || !num_hw_ctrs)
561 return;
562
563 /*
564 * Increase the number of hardware counters to offset the time counter.
565 */
566 kvpmu->num_hw_ctrs = num_hw_ctrs + 1;
567 kvpmu->num_fw_ctrs = SBI_PMU_FW_MAX;
568 memset(&kvpmu->fw_event, 0, SBI_PMU_FW_MAX * sizeof(struct kvm_fw_event));
569
570 if (kvpmu->num_hw_ctrs > RISCV_KVM_MAX_HW_CTRS) {
571 pr_warn_once("Limiting the hardware counters to 32 as specified by the ISA");
572 kvpmu->num_hw_ctrs = RISCV_KVM_MAX_HW_CTRS;
573 }
574
575 /*
576 * There is no correlation between the logical hardware counter and virtual counters.
577 * However, we need to encode a hpmcounter CSR in the counter info field so that
578 * KVM can trap n emulate the read. This works well in the migration use case as
579 * KVM doesn't care if the actual hpmcounter is available in the hardware or not.
580 */
581 for (i = 0; i < kvm_pmu_num_counters(kvpmu); i++) {
582 /* TIME CSR shouldn't be read from perf interface */
583 if (i == 1)
584 continue;
585 pmc = &kvpmu->pmc[i];
586 pmc->idx = i;
587 pmc->event_idx = SBI_PMU_EVENT_IDX_INVALID;
588 if (i < kvpmu->num_hw_ctrs) {
589 pmc->cinfo.type = SBI_PMU_CTR_TYPE_HW;
590 if (i < 3)
591 /* CY, IR counters */
592 pmc->cinfo.width = 63;
593 else
594 pmc->cinfo.width = hpm_width;
595 /*
596 * The CSR number doesn't have any relation with the logical
597 * hardware counters. The CSR numbers are encoded sequentially
598 * to avoid maintaining a map between the virtual counter
599 * and CSR number.
600 */
601 pmc->cinfo.csr = CSR_CYCLE + i;
602 } else {
603 pmc->cinfo.type = SBI_PMU_CTR_TYPE_FW;
604 pmc->cinfo.width = BITS_PER_LONG - 1;
605 }
606 }
607
608 kvpmu->init_done = true;
609}
610
611void kvm_riscv_vcpu_pmu_deinit(struct kvm_vcpu *vcpu)
612{
613 struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
614 struct kvm_pmc *pmc;
615 int i;
616
617 if (!kvpmu)
618 return;
619
620 for_each_set_bit(i, kvpmu->pmc_in_use, RISCV_MAX_COUNTERS) {
621 pmc = &kvpmu->pmc[i];
622 pmc->counter_val = 0;
623 kvm_pmu_release_perf_event(pmc);
624 pmc->event_idx = SBI_PMU_EVENT_IDX_INVALID;
625 }
626 bitmap_zero(kvpmu->pmc_in_use, RISCV_MAX_COUNTERS);
627 memset(&kvpmu->fw_event, 0, SBI_PMU_FW_MAX * sizeof(struct kvm_fw_event));
628}
629
630void kvm_riscv_vcpu_pmu_reset(struct kvm_vcpu *vcpu)
631{
632 kvm_riscv_vcpu_pmu_deinit(vcpu);
633}
arch/riscv/kvm/vcpu_sbi.c+35−37
@@ -12,26 +12,6 @@ · 11 unmodified lines
12#include <asm/sbi.h>12#include <asm/sbi.h>
13#include <asm/kvm_vcpu_sbi.h>13#include <asm/kvm_vcpu_sbi.h>
14 14
15static int kvm_linux_err_map_sbi(int err)
16{
17 switch (err) {
18 case 0:
19 return SBI_SUCCESS;
20 case -EPERM:
21 return SBI_ERR_DENIED;
22 case -EINVAL:
23 return SBI_ERR_INVALID_PARAM;
24 case -EFAULT:
25 return SBI_ERR_INVALID_ADDRESS;
26 case -EOPNOTSUPP:
27 return SBI_ERR_NOT_SUPPORTED;
28 case -EALREADY:
29 return SBI_ERR_ALREADY_AVAILABLE;
30 default:
31 return SBI_ERR_FAILURE;
32 };
33}
34
35#ifndef CONFIG_RISCV_SBI_V0115#ifndef CONFIG_RISCV_SBI_V01
36static const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_v01 = {16static const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_v01 = {
37 .extid_start = -1UL,17 .extid_start = -1UL,
@@ -40,6 +20,16 @@ static const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_v01 = { · 2 unmodified lines
40};20};
41#endif21#endif
42 22
23#ifdef CONFIG_RISCV_PMU_SBI
24extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_pmu;
25#else
26static const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_pmu = {
27 .extid_start = -1UL,
28 .extid_end = -1UL,
29 .handler = NULL,
30};
31#endif
32
43static const struct kvm_vcpu_sbi_extension *sbi_ext[] = {33static const struct kvm_vcpu_sbi_extension *sbi_ext[] = {
44 &vcpu_sbi_ext_v01,34 &vcpu_sbi_ext_v01,
45 &vcpu_sbi_ext_base,35 &vcpu_sbi_ext_base,
@@ -48,6 +38,7 @@ static const struct kvm_vcpu_sbi_extension *sbi_ext[] = { · 2 unmodified lines
48 &vcpu_sbi_ext_rfence,38 &vcpu_sbi_ext_rfence,
49 &vcpu_sbi_ext_srst,39 &vcpu_sbi_ext_srst,
50 &vcpu_sbi_ext_hsm,40 &vcpu_sbi_ext_hsm,
41 &vcpu_sbi_ext_pmu,
51 &vcpu_sbi_ext_experimental,42 &vcpu_sbi_ext_experimental,
52 &vcpu_sbi_ext_vendor,43 &vcpu_sbi_ext_vendor,
53};44};
@@ -125,11 +116,14 @@ int kvm_riscv_vcpu_sbi_ecall(struct kvm_vcpu *vcpu, struct kvm_run *run) · 71 unmodified lines
125{116{
126 int ret = 1;117 int ret = 1;
127 bool next_sepc = true;118 bool next_sepc = true;
128 bool userspace_exit = false;
129 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;119 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
130 const struct kvm_vcpu_sbi_extension *sbi_ext;120 const struct kvm_vcpu_sbi_extension *sbi_ext;
131 struct kvm_cpu_trap utrap = { 0 };121 struct kvm_cpu_trap utrap = {0};
132 unsigned long out_val = 0;122 struct kvm_vcpu_sbi_return sbi_ret = {
123 .out_val = 0,
124 .err_val = 0,
125 .utrap = &utrap,
126 };
133 bool ext_is_v01 = false;127 bool ext_is_v01 = false;
134 128
135 sbi_ext = kvm_vcpu_sbi_find_ext(cp->a7);129 sbi_ext = kvm_vcpu_sbi_find_ext(cp->a7);
@@ -139,42 +133,46 @@ int kvm_riscv_vcpu_sbi_ecall(struct kvm_vcpu *vcpu, struct kvm_run *run) · 3 unmodified lines
139 cp->a7 <= SBI_EXT_0_1_SHUTDOWN)133 cp->a7 <= SBI_EXT_0_1_SHUTDOWN)
140 ext_is_v01 = true;134 ext_is_v01 = true;
141#endif135#endif
142 ret = sbi_ext->handler(vcpu, run, &out_val, &utrap, &userspace_exit);136 ret = sbi_ext->handler(vcpu, run, &sbi_ret);
143 } else {137 } else {
144 /* Return error for unsupported SBI calls */138 /* Return error for unsupported SBI calls */
145 cp->a0 = SBI_ERR_NOT_SUPPORTED;139 cp->a0 = SBI_ERR_NOT_SUPPORTED;
146 goto ecall_done;140 goto ecall_done;
147 }141 }
148 142
143 /*
144 * When the SBI extension returns a Linux error code, it exits the ioctl
145 * loop and forwards the error to userspace.
146 */
147 if (ret < 0) {
148 next_sepc = false;
149 goto ecall_done;
150 }
151
149 /* Handle special error cases i.e trap, exit or userspace forward */152 /* Handle special error cases i.e trap, exit or userspace forward */
150 if (utrap.scause) {153 if (sbi_ret.utrap->scause) {
151 /* No need to increment sepc or exit ioctl loop */154 /* No need to increment sepc or exit ioctl loop */
152 ret = 1;155 ret = 1;
153 utrap.sepc = cp->sepc;156 sbi_ret.utrap->sepc = cp->sepc;
154 kvm_riscv_vcpu_trap_redirect(vcpu, &utrap);157 kvm_riscv_vcpu_trap_redirect(vcpu, sbi_ret.utrap);
155 next_sepc = false;158 next_sepc = false;
156 goto ecall_done;159 goto ecall_done;
157 }160 }
158 161
159 /* Exit ioctl loop or Propagate the error code the guest */162 /* Exit ioctl loop or Propagate the error code the guest */
160 if (userspace_exit) {163 if (sbi_ret.uexit) {
161 next_sepc = false;164 next_sepc = false;
162 ret = 0;165 ret = 0;
163 } else {166 } else {
164 /**167 cp->a0 = sbi_ret.err_val;
165 * SBI extension handler always returns an Linux error code. Convert
166 * it to the SBI specific error code that can be propagated the SBI
167 * caller.
168 */
169 ret = kvm_linux_err_map_sbi(ret);
170 cp->a0 = ret;
171 ret = 1;168 ret = 1;
172 }169 }
173ecall_done:170ecall_done:
174 if (next_sepc)171 if (next_sepc)
175 cp->sepc += 4;172 cp->sepc += 4;
176 if (!ext_is_v01)173 /* a1 should only be updated when we continue the ioctl loop */
177 cp->a1 = out_val;174 if (!ext_is_v01 && ret == 1)
175 cp->a1 = sbi_ret.out_val;
178 176
179 return ret;177 return ret;
180}178}
arch/riscv/kvm/vcpu_sbi_base.c+14−13
@@ -14,11 +14,11 @@ · 13 unmodified lines
14#include <asm/kvm_vcpu_sbi.h>14#include <asm/kvm_vcpu_sbi.h>
15 15
16static int kvm_sbi_ext_base_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,16static int kvm_sbi_ext_base_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
17 unsigned long *out_val,17 struct kvm_vcpu_sbi_return *retdata)
18 struct kvm_cpu_trap *trap, bool *exit)
19{18{
20 int ret = 0;
21 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;19 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
20 const struct kvm_vcpu_sbi_extension *sbi_ext;
21 unsigned long *out_val = &retdata->out_val;
22 22
23 switch (cp->a6) {23 switch (cp->a6) {
24 case SBI_EXT_BASE_GET_SPEC_VERSION:24 case SBI_EXT_BASE_GET_SPEC_VERSION:
@@ -42,9 +42,12 @@ static int kvm_sbi_ext_base_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 17 unmodified lines
42 * forward it to the userspace42 * forward it to the userspace
43 */43 */
44 kvm_riscv_vcpu_sbi_forward(vcpu, run);44 kvm_riscv_vcpu_sbi_forward(vcpu, run);
45 *exit = true;45 retdata->uexit = true;
46 } else46 } else {
47 *out_val = kvm_vcpu_sbi_find_ext(cp->a0) ? 1 : 0;47 sbi_ext = kvm_vcpu_sbi_find_ext(cp->a0);
48 *out_val = sbi_ext && sbi_ext->probe ?
49 sbi_ext->probe(vcpu) : !!sbi_ext;
50 }
48 break;51 break;
49 case SBI_EXT_BASE_GET_MVENDORID:52 case SBI_EXT_BASE_GET_MVENDORID:
50 *out_val = vcpu->arch.mvendorid;53 *out_val = vcpu->arch.mvendorid;
@@ -56,11 +59,11 @@ static int kvm_sbi_ext_base_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 5 unmodified lines
56 *out_val = vcpu->arch.mimpid;59 *out_val = vcpu->arch.mimpid;
57 break;60 break;
58 default:61 default:
59 ret = -EOPNOTSUPP;62 retdata->err_val = SBI_ERR_NOT_SUPPORTED;
60 break;63 break;
61 }64 }
62 65
63 return ret;66 return 0;
64}67}
65 68
66const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_base = {69const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_base = {
@@ -70,17 +73,15 @@ const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_base = { · 3 unmodified lines
70};73};
71 74
72static int kvm_sbi_ext_forward_handler(struct kvm_vcpu *vcpu,75static int kvm_sbi_ext_forward_handler(struct kvm_vcpu *vcpu,
73 struct kvm_run *run,76 struct kvm_run *run,
74 unsigned long *out_val,77 struct kvm_vcpu_sbi_return *retdata)
75 struct kvm_cpu_trap *utrap,
76 bool *exit)
77{78{
78 /*79 /*
79 * Both SBI experimental and vendor extensions are80 * Both SBI experimental and vendor extensions are
80 * unconditionally forwarded to userspace.81 * unconditionally forwarded to userspace.
81 */82 */
82 kvm_riscv_vcpu_sbi_forward(vcpu, run);83 kvm_riscv_vcpu_sbi_forward(vcpu, run);
83 *exit = true;84 retdata->uexit = true;
84 return 0;85 return 0;
85}86}
86 87
arch/riscv/kvm/vcpu_sbi_hsm.c+14−14
@@ -21,9 +21,9 @@ static int kvm_sbi_hsm_vcpu_start(struct kvm_vcpu *vcpu) · 20 unmodified lines
21 21
22 target_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, target_vcpuid);22 target_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, target_vcpuid);
23 if (!target_vcpu)23 if (!target_vcpu)
24 return -EINVAL;24 return SBI_ERR_INVALID_PARAM;
25 if (!target_vcpu->arch.power_off)25 if (!target_vcpu->arch.power_off)
26 return -EALREADY;26 return SBI_ERR_ALREADY_AVAILABLE;
27 27
28 reset_cntx = &target_vcpu->arch.guest_reset_context;28 reset_cntx = &target_vcpu->arch.guest_reset_context;
29 /* start address */29 /* start address */
@@ -42,7 +42,7 @@ static int kvm_sbi_hsm_vcpu_start(struct kvm_vcpu *vcpu) · 12 unmodified lines
42static int kvm_sbi_hsm_vcpu_stop(struct kvm_vcpu *vcpu)42static int kvm_sbi_hsm_vcpu_stop(struct kvm_vcpu *vcpu)
43{43{
44 if (vcpu->arch.power_off)44 if (vcpu->arch.power_off)
45 return -EINVAL;45 return SBI_ERR_FAILURE;
46 46
47 kvm_riscv_vcpu_power_off(vcpu);47 kvm_riscv_vcpu_power_off(vcpu);
48 48
@@ -57,7 +57,7 @@ static int kvm_sbi_hsm_vcpu_get_status(struct kvm_vcpu *vcpu) · 8 unmodified lines
57 57
58 target_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, target_vcpuid);58 target_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, target_vcpuid);
59 if (!target_vcpu)59 if (!target_vcpu)
60 return -EINVAL;60 return SBI_ERR_INVALID_PARAM;
61 if (!target_vcpu->arch.power_off)61 if (!target_vcpu->arch.power_off)
62 return SBI_HSM_STATE_STARTED;62 return SBI_HSM_STATE_STARTED;
63 else if (vcpu->stat.generic.blocking)63 else if (vcpu->stat.generic.blocking)
@@ -67,9 +67,7 @@ static int kvm_sbi_hsm_vcpu_get_status(struct kvm_vcpu *vcpu) · 3 unmodified lines
67}67}
68 68
69static int kvm_sbi_ext_hsm_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,69static int kvm_sbi_ext_hsm_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
70 unsigned long *out_val,70 struct kvm_vcpu_sbi_return *retdata)
71 struct kvm_cpu_trap *utrap,
72 bool *exit)
73{71{
74 int ret = 0;72 int ret = 0;
75 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;73 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
@@ -88,27 +86,29 @@ static int kvm_sbi_ext_hsm_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 12 unmodified lines
88 case SBI_EXT_HSM_HART_STATUS:86 case SBI_EXT_HSM_HART_STATUS:
89 ret = kvm_sbi_hsm_vcpu_get_status(vcpu);87 ret = kvm_sbi_hsm_vcpu_get_status(vcpu);
90 if (ret >= 0) {88 if (ret >= 0) {
91 *out_val = ret;89 retdata->out_val = ret;
92 ret = 0;90 retdata->err_val = 0;
93 }91 }
94 break;92 return 0;
95 case SBI_EXT_HSM_HART_SUSPEND:93 case SBI_EXT_HSM_HART_SUSPEND:
96 switch (cp->a0) {94 switch (cp->a0) {
97 case SBI_HSM_SUSPEND_RET_DEFAULT:95 case SBI_HSM_SUSPEND_RET_DEFAULT:
98 kvm_riscv_vcpu_wfi(vcpu);96 kvm_riscv_vcpu_wfi(vcpu);
99 break;97 break;
100 case SBI_HSM_SUSPEND_NON_RET_DEFAULT:98 case SBI_HSM_SUSPEND_NON_RET_DEFAULT:
101 ret = -EOPNOTSUPP;99 ret = SBI_ERR_NOT_SUPPORTED;
102 break;100 break;
103 default:101 default:
104 ret = -EINVAL;102 ret = SBI_ERR_INVALID_PARAM;
105 }103 }
106 break;104 break;
107 default:105 default:
108 ret = -EOPNOTSUPP;106 ret = SBI_ERR_NOT_SUPPORTED;
109 }107 }
110 108
111 return ret;109 retdata->err_val = ret;
110
111 return 0;
112}112}
113 113
114const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_hsm = {114const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_hsm = {
arch/riscv/kvm/vcpu_sbi_pmu.c+86−0
@@ -0,0 +1,86 @@
1// SPDX-License-Identifier: GPL-2.0
2/*
3 * Copyright (c) 2023 Rivos Inc
4 *
5 * Authors:
6 * Atish Patra <atishp@rivosinc.com>
7 */
8
9#include <linux/errno.h>
10#include <linux/err.h>
11#include <linux/kvm_host.h>
12#include <asm/csr.h>
13#include <asm/sbi.h>
14#include <asm/kvm_vcpu_sbi.h>
15
16static int kvm_sbi_ext_pmu_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
17 struct kvm_vcpu_sbi_return *retdata)
18{
19 int ret = 0;
20 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
21 struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
22 unsigned long funcid = cp->a6;
23 u64 temp;
24
25 if (!kvpmu->init_done) {
26 retdata->err_val = SBI_ERR_NOT_SUPPORTED;
27 return 0;
28 }
29
30 switch (funcid) {
31 case SBI_EXT_PMU_NUM_COUNTERS:
32 ret = kvm_riscv_vcpu_pmu_num_ctrs(vcpu, retdata);
33 break;
34 case SBI_EXT_PMU_COUNTER_GET_INFO:
35 ret = kvm_riscv_vcpu_pmu_ctr_info(vcpu, cp->a0, retdata);
36 break;
37 case SBI_EXT_PMU_COUNTER_CFG_MATCH:
38#if defined(CONFIG_32BIT)
39 temp = ((uint64_t)cp->a5 << 32) | cp->a4;
40#else
41 temp = cp->a4;
42#endif
43 /*
44 * This can fail if perf core framework fails to create an event.
45 * Forward the error to userspace because it's an error which
46 * happened within the host kernel. The other option would be
47 * to convert to an SBI error and forward to the guest.
48 */
49 ret = kvm_riscv_vcpu_pmu_ctr_cfg_match(vcpu, cp->a0, cp->a1,
50 cp->a2, cp->a3, temp, retdata);
51 break;
52 case SBI_EXT_PMU_COUNTER_START:
53#if defined(CONFIG_32BIT)
54 temp = ((uint64_t)cp->a4 << 32) | cp->a3;
55#else
56 temp = cp->a3;
57#endif
58 ret = kvm_riscv_vcpu_pmu_ctr_start(vcpu, cp->a0, cp->a1, cp->a2,
59 temp, retdata);
60 break;
61 case SBI_EXT_PMU_COUNTER_STOP:
62 ret = kvm_riscv_vcpu_pmu_ctr_stop(vcpu, cp->a0, cp->a1, cp->a2, retdata);
63 break;
64 case SBI_EXT_PMU_COUNTER_FW_READ:
65 ret = kvm_riscv_vcpu_pmu_ctr_read(vcpu, cp->a0, retdata);
66 break;
67 default:
68 retdata->err_val = SBI_ERR_NOT_SUPPORTED;
69 }
70
71 return ret;
72}
73
74static unsigned long kvm_sbi_ext_pmu_probe(struct kvm_vcpu *vcpu)
75{
76 struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
77
78 return kvpmu->init_done;
79}
80
81const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_pmu = {
82 .extid_start = SBI_EXT_PMU,
83 .extid_end = SBI_EXT_PMU,
84 .handler = kvm_sbi_ext_pmu_handler,
85 .probe = kvm_sbi_ext_pmu_probe,
86};
arch/riscv/kvm/vcpu_sbi_replace.c+27−23
@@ -11,19 +11,21 @@ · 10 unmodified lines
11#include <linux/kvm_host.h>11#include <linux/kvm_host.h>
12#include <asm/sbi.h>12#include <asm/sbi.h>
13#include <asm/kvm_vcpu_timer.h>13#include <asm/kvm_vcpu_timer.h>
14#include <asm/kvm_vcpu_pmu.h>
14#include <asm/kvm_vcpu_sbi.h>15#include <asm/kvm_vcpu_sbi.h>
15 16
16static int kvm_sbi_ext_time_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,17static int kvm_sbi_ext_time_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
17 unsigned long *out_val,18 struct kvm_vcpu_sbi_return *retdata)
18 struct kvm_cpu_trap *utrap, bool *exit)
19{19{
20 int ret = 0;
21 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;20 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
22 u64 next_cycle;21 u64 next_cycle;
23 22
24 if (cp->a6 != SBI_EXT_TIME_SET_TIMER)23 if (cp->a6 != SBI_EXT_TIME_SET_TIMER) {
25 return -EINVAL;24 retdata->err_val = SBI_ERR_INVALID_PARAM;
25 return 0;
26 }
26 27
28 kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_SET_TIMER);
27#if __riscv_xlen == 3229#if __riscv_xlen == 32
28 next_cycle = ((u64)cp->a1 << 32) | (u64)cp->a0;30 next_cycle = ((u64)cp->a1 << 32) | (u64)cp->a0;
29#else31#else
@@ -31,7 +33,7 @@ static int kvm_sbi_ext_time_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 1 unmodified line
31#endif33#endif
32 kvm_riscv_vcpu_timer_next_event(vcpu, next_cycle);34 kvm_riscv_vcpu_timer_next_event(vcpu, next_cycle);
33 35
34 return ret;36 return 0;
35}37}
36 38
37const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_time = {39const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_time = {
@@ -41,8 +43,7 @@ const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_time = { · 3 unmodified lines
41};43};
42 44
43static int kvm_sbi_ext_ipi_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,45static int kvm_sbi_ext_ipi_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
44 unsigned long *out_val,46 struct kvm_vcpu_sbi_return *retdata)
45 struct kvm_cpu_trap *utrap, bool *exit)
46{47{
47 int ret = 0;48 int ret = 0;
48 unsigned long i;49 unsigned long i;
@@ -51,9 +52,12 @@ static int kvm_sbi_ext_ipi_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 2 unmodified lines
51 unsigned long hmask = cp->a0;52 unsigned long hmask = cp->a0;
52 unsigned long hbase = cp->a1;53 unsigned long hbase = cp->a1;
53 54
54 if (cp->a6 != SBI_EXT_IPI_SEND_IPI)55 if (cp->a6 != SBI_EXT_IPI_SEND_IPI) {
55 return -EINVAL;56 retdata->err_val = SBI_ERR_INVALID_PARAM;
57 return 0;
58 }
56 59
60 kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_IPI_SENT);
57 kvm_for_each_vcpu(i, tmp, vcpu->kvm) {61 kvm_for_each_vcpu(i, tmp, vcpu->kvm) {
58 if (hbase != -1UL) {62 if (hbase != -1UL) {
59 if (tmp->vcpu_id < hbase)63 if (tmp->vcpu_id < hbase)
@@ -64,6 +68,7 @@ static int kvm_sbi_ext_ipi_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 4 unmodified lines
64 ret = kvm_riscv_vcpu_set_interrupt(tmp, IRQ_VS_SOFT);68 ret = kvm_riscv_vcpu_set_interrupt(tmp, IRQ_VS_SOFT);
65 if (ret < 0)69 if (ret < 0)
66 break;70 break;
71 kvm_riscv_vcpu_pmu_incr_fw(tmp, SBI_PMU_FW_IPI_RCVD);
67 }72 }
68 73
69 return ret;74 return ret;
@@ -76,10 +81,8 @@ const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_ipi = { · 6 unmodified lines
76};81};
77 82
78static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,83static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
79 unsigned long *out_val,84 struct kvm_vcpu_sbi_return *retdata)
80 struct kvm_cpu_trap *utrap, bool *exit)
81{85{
82 int ret = 0;
83 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;86 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
84 unsigned long hmask = cp->a0;87 unsigned long hmask = cp->a0;
85 unsigned long hbase = cp->a1;88 unsigned long hbase = cp->a1;
@@ -88,6 +91,7 @@ static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *run · 2 unmodified lines
88 switch (funcid) {91 switch (funcid) {
89 case SBI_EXT_RFENCE_REMOTE_FENCE_I:92 case SBI_EXT_RFENCE_REMOTE_FENCE_I:
90 kvm_riscv_fence_i(vcpu->kvm, hbase, hmask);93 kvm_riscv_fence_i(vcpu->kvm, hbase, hmask);
94 kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_FENCE_I_SENT);
91 break;95 break;
92 case SBI_EXT_RFENCE_REMOTE_SFENCE_VMA:96 case SBI_EXT_RFENCE_REMOTE_SFENCE_VMA:
93 if (cp->a2 == 0 && cp->a3 == 0)97 if (cp->a2 == 0 && cp->a3 == 0)
@@ -95,6 +99,7 @@ static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *run · 1 unmodified line
95 else99 else
96 kvm_riscv_hfence_vvma_gva(vcpu->kvm, hbase, hmask,100 kvm_riscv_hfence_vvma_gva(vcpu->kvm, hbase, hmask,
97 cp->a2, cp->a3, PAGE_SHIFT);101 cp->a2, cp->a3, PAGE_SHIFT);
102 kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_SENT);
98 break;103 break;
99 case SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID:104 case SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID:
100 if (cp->a2 == 0 && cp->a3 == 0)105 if (cp->a2 == 0 && cp->a3 == 0)
@@ -105,6 +110,7 @@ static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *run · 4 unmodified lines
105 hbase, hmask,110 hbase, hmask,
106 cp->a2, cp->a3,111 cp->a2, cp->a3,
107 PAGE_SHIFT, cp->a4);112 PAGE_SHIFT, cp->a4);
113 kvm_riscv_vcpu_pmu_incr_fw(vcpu, SBI_PMU_FW_HFENCE_VVMA_ASID_SENT);
108 break;114 break;
109 case SBI_EXT_RFENCE_REMOTE_HFENCE_GVMA:115 case SBI_EXT_RFENCE_REMOTE_HFENCE_GVMA:
110 case SBI_EXT_RFENCE_REMOTE_HFENCE_GVMA_VMID:116 case SBI_EXT_RFENCE_REMOTE_HFENCE_GVMA_VMID:
@@ -116,10 +122,10 @@ static int kvm_sbi_ext_rfence_handler(struct kvm_vcpu *vcpu, struct kvm_run *run · 5 unmodified lines
116 */122 */
117 break;123 break;
118 default:124 default:
119 ret = -EOPNOTSUPP;125 retdata->err_val = SBI_ERR_NOT_SUPPORTED;
120 }126 }
121 127
122 return ret;128 return 0;
123}129}
124 130
125const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_rfence = {131const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_rfence = {
@@ -130,14 +136,12 @@ const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_rfence = { · 4 unmodified lines
130 136
131static int kvm_sbi_ext_srst_handler(struct kvm_vcpu *vcpu,137static int kvm_sbi_ext_srst_handler(struct kvm_vcpu *vcpu,
132 struct kvm_run *run,138 struct kvm_run *run,
133 unsigned long *out_val,139 struct kvm_vcpu_sbi_return *retdata)
134 struct kvm_cpu_trap *utrap, bool *exit)
135{140{
136 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;141 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
137 unsigned long funcid = cp->a6;142 unsigned long funcid = cp->a6;
138 u32 reason = cp->a1;143 u32 reason = cp->a1;
139 u32 type = cp->a0;144 u32 type = cp->a0;
140 int ret = 0;
141 145
142 switch (funcid) {146 switch (funcid) {
143 case SBI_EXT_SRST_RESET:147 case SBI_EXT_SRST_RESET:
@@ -146,24 +150,24 @@ static int kvm_sbi_ext_srst_handler(struct kvm_vcpu *vcpu, · 2 unmodified lines
146 kvm_riscv_vcpu_sbi_system_reset(vcpu, run,150 kvm_riscv_vcpu_sbi_system_reset(vcpu, run,
147 KVM_SYSTEM_EVENT_SHUTDOWN,151 KVM_SYSTEM_EVENT_SHUTDOWN,
148 reason);152 reason);
149 *exit = true;153 retdata->uexit = true;
150 break;154 break;
151 case SBI_SRST_RESET_TYPE_COLD_REBOOT:155 case SBI_SRST_RESET_TYPE_COLD_REBOOT:
152 case SBI_SRST_RESET_TYPE_WARM_REBOOT:156 case SBI_SRST_RESET_TYPE_WARM_REBOOT:
153 kvm_riscv_vcpu_sbi_system_reset(vcpu, run,157 kvm_riscv_vcpu_sbi_system_reset(vcpu, run,
154 KVM_SYSTEM_EVENT_RESET,158 KVM_SYSTEM_EVENT_RESET,
155 reason);159 reason);
156 *exit = true;160 retdata->uexit = true;
157 break;161 break;
158 default:162 default:
159 ret = -EOPNOTSUPP;163 retdata->err_val = SBI_ERR_NOT_SUPPORTED;
160 }164 }
161 break;165 break;
162 default:166 default:
163 ret = -EOPNOTSUPP;167 retdata->err_val = SBI_ERR_NOT_SUPPORTED;
164 }168 }
165 169
166 return ret;170 return 0;
167}171}
168 172
169const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_srst = {173const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_srst = {
arch/riscv/kvm/vcpu_sbi_v01.c+7−10
@@ -14,9 +14,7 @@ · 13 unmodified lines
14#include <asm/kvm_vcpu_sbi.h>14#include <asm/kvm_vcpu_sbi.h>
15 15
16static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,16static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
17 unsigned long *out_val,17 struct kvm_vcpu_sbi_return *retdata)
18 struct kvm_cpu_trap *utrap,
19 bool *exit)
20{18{
21 ulong hmask;19 ulong hmask;
22 int i, ret = 0;20 int i, ret = 0;
@@ -24,6 +22,7 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 1 unmodified line
24 struct kvm_vcpu *rvcpu;22 struct kvm_vcpu *rvcpu;
25 struct kvm *kvm = vcpu->kvm;23 struct kvm *kvm = vcpu->kvm;
26 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;24 struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
25 struct kvm_cpu_trap *utrap = retdata->utrap;
27 26
28 switch (cp->a7) {27 switch (cp->a7) {
29 case SBI_EXT_0_1_CONSOLE_GETCHAR:28 case SBI_EXT_0_1_CONSOLE_GETCHAR:
@@ -33,7 +32,7 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 3 unmodified lines
33 * handled in kernel so we forward these to user-space32 * handled in kernel so we forward these to user-space
34 */33 */
35 kvm_riscv_vcpu_sbi_forward(vcpu, run);34 kvm_riscv_vcpu_sbi_forward(vcpu, run);
36 *exit = true;35 retdata->uexit = true;
37 break;36 break;
38 case SBI_EXT_0_1_SET_TIMER:37 case SBI_EXT_0_1_SET_TIMER:
39#if __riscv_xlen == 3238#if __riscv_xlen == 32
@@ -48,8 +47,7 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 8 unmodified lines
48 break;47 break;
49 case SBI_EXT_0_1_SEND_IPI:48 case SBI_EXT_0_1_SEND_IPI:
50 if (cp->a0)49 if (cp->a0)
51 hmask = kvm_riscv_vcpu_unpriv_read(vcpu, false, cp->a0,50 hmask = kvm_riscv_vcpu_unpriv_read(vcpu, false, cp->a0, utrap);
52 utrap);
53 else51 else
54 hmask = (1UL << atomic_read(&kvm->online_vcpus)) - 1;52 hmask = (1UL << atomic_read(&kvm->online_vcpus)) - 1;
55 if (utrap->scause)53 if (utrap->scause)
@@ -65,14 +63,13 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 9 unmodified lines
65 case SBI_EXT_0_1_SHUTDOWN:63 case SBI_EXT_0_1_SHUTDOWN:
66 kvm_riscv_vcpu_sbi_system_reset(vcpu, run,64 kvm_riscv_vcpu_sbi_system_reset(vcpu, run,
67 KVM_SYSTEM_EVENT_SHUTDOWN, 0);65 KVM_SYSTEM_EVENT_SHUTDOWN, 0);
68 *exit = true;66 retdata->uexit = true;
69 break;67 break;
70 case SBI_EXT_0_1_REMOTE_FENCE_I:68 case SBI_EXT_0_1_REMOTE_FENCE_I:
71 case SBI_EXT_0_1_REMOTE_SFENCE_VMA:69 case SBI_EXT_0_1_REMOTE_SFENCE_VMA:
72 case SBI_EXT_0_1_REMOTE_SFENCE_VMA_ASID:70 case SBI_EXT_0_1_REMOTE_SFENCE_VMA_ASID:
73 if (cp->a0)71 if (cp->a0)
74 hmask = kvm_riscv_vcpu_unpriv_read(vcpu, false, cp->a0,72 hmask = kvm_riscv_vcpu_unpriv_read(vcpu, false, cp->a0, utrap);
75 utrap);
76 else73 else
77 hmask = (1UL << atomic_read(&kvm->online_vcpus)) - 1;74 hmask = (1UL << atomic_read(&kvm->online_vcpus)) - 1;
78 if (utrap->scause)75 if (utrap->scause)
@@ -103,7 +100,7 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, · 24 unmodified lines
103 }100 }
104 break;101 break;
105 default:102 default:
106 ret = -EINVAL;103 retdata->err_val = SBI_ERR_NOT_SUPPORTED;
107 break;104 break;
108 }105 }
109 106
arch/riscv/kvm/vmid.c+2−2
@@ -17,10 +17,10 @@ · 16 unmodified lines
17 17
18static unsigned long vmid_version = 1;18static unsigned long vmid_version = 1;
19static unsigned long vmid_next;19static unsigned long vmid_next;
20static unsigned long vmid_bits;20static unsigned long vmid_bits __ro_after_init;
21static DEFINE_SPINLOCK(vmid_lock);21static DEFINE_SPINLOCK(vmid_lock);
22 22
23void kvm_riscv_gstage_vmid_detect(void)23void __init kvm_riscv_gstage_vmid_detect(void)
24{24{
25 unsigned long old;25 unsigned long old;
26 26
arch/s390/include/asm/kvm_host.h+0−1
@@ -1031,7 +1031,6 @@ extern char sie_exit; · 1030 unmodified lines
1031extern int kvm_s390_gisc_register(struct kvm *kvm, u32 gisc);1031extern int kvm_s390_gisc_register(struct kvm *kvm, u32 gisc);
1032extern int kvm_s390_gisc_unregister(struct kvm *kvm, u32 gisc);1032extern int kvm_s390_gisc_unregister(struct kvm *kvm, u32 gisc);
1033 1033
1034static inline void kvm_arch_hardware_disable(void) {}
1035static inline void kvm_arch_sync_events(struct kvm *kvm) {}1034static inline void kvm_arch_sync_events(struct kvm *kvm) {}
1036static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}1035static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
1037static inline void kvm_arch_free_memslot(struct kvm *kvm,1036static inline void kvm_arch_free_memslot(struct kvm *kvm,
arch/s390/kvm/gaccess.c+109−0
@@ -1161,6 +1161,115 @@ int access_guest_real(struct kvm_vcpu *vcpu, unsigned long gra, · 1160 unmodified lines
1161 return rc;1161 return rc;
1162}1162}
1163 1163
1164/**
1165 * cmpxchg_guest_abs_with_key() - Perform cmpxchg on guest absolute address.
1166 * @kvm: Virtual machine instance.
1167 * @gpa: Absolute guest address of the location to be changed.
1168 * @len: Operand length of the cmpxchg, required: 1 <= len <= 16. Providing a
1169 * non power of two will result in failure.
1170 * @old_addr: Pointer to old value. If the location at @gpa contains this value,
1171 * the exchange will succeed. After calling cmpxchg_guest_abs_with_key()
1172 * *@old_addr contains the value at @gpa before the attempt to
1173 * exchange the value.
1174 * @new: The value to place at @gpa.
1175 * @access_key: The access key to use for the guest access.
1176 * @success: output value indicating if an exchange occurred.
1177 *
1178 * Atomically exchange the value at @gpa by @new, if it contains *@old.
1179 * Honors storage keys.
1180 *
1181 * Return: * 0: successful exchange
1182 * * >0: a program interruption code indicating the reason cmpxchg could
1183 * not be attempted
1184 * * -EINVAL: address misaligned or len not power of two
1185 * * -EAGAIN: transient failure (len 1 or 2)
1186 * * -EOPNOTSUPP: read-only memslot (should never occur)
1187 */
1188int cmpxchg_guest_abs_with_key(struct kvm *kvm, gpa_t gpa, int len,
1189 __uint128_t *old_addr, __uint128_t new,
1190 u8 access_key, bool *success)
1191{
1192 gfn_t gfn = gpa_to_gfn(gpa);
1193 struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
1194 bool writable;
1195 hva_t hva;
1196 int ret;
1197
1198 if (!IS_ALIGNED(gpa, len))
1199 return -EINVAL;
1200
1201 hva = gfn_to_hva_memslot_prot(slot, gfn, &writable);
1202 if (kvm_is_error_hva(hva))
1203 return PGM_ADDRESSING;
1204 /*
1205 * Check if it's a read-only memslot, even though that cannot occur
1206 * since those are unsupported.
1207 * Don't try to actually handle that case.
1208 */
1209 if (!writable)
1210 return -EOPNOTSUPP;
1211
1212 hva += offset_in_page(gpa);
1213 /*
1214 * The cmpxchg_user_key macro depends on the type of "old", so we need
1215 * a case for each valid length and get some code duplication as long
1216 * as we don't introduce a new macro.
1217 */
1218 switch (len) {
1219 case 1: {
1220 u8 old;
1221
1222 ret = cmpxchg_user_key((u8 __user *)hva, &old, *old_addr, new, access_key);
1223 *success = !ret && old == *old_addr;
1224 *old_addr = old;
1225 break;
1226 }
1227 case 2: {
1228 u16 old;
1229
1230 ret = cmpxchg_user_key((u16 __user *)hva, &old, *old_addr, new, access_key);
1231 *success = !ret && old == *old_addr;
1232 *old_addr = old;
1233 break;
1234 }
1235 case 4: {
1236 u32 old;
1237
1238 ret = cmpxchg_user_key((u32 __user *)hva, &old, *old_addr, new, access_key);
1239 *success = !ret && old == *old_addr;
1240 *old_addr = old;
1241 break;
1242 }
1243 case 8: {
1244 u64 old;
1245
1246 ret = cmpxchg_user_key((u64 __user *)hva, &old, *old_addr, new, access_key);
1247 *success = !ret && old == *old_addr;
1248 *old_addr = old;
1249 break;
1250 }
1251 case 16: {
1252 __uint128_t old;
1253
1254 ret = cmpxchg_user_key((__uint128_t __user *)hva, &old, *old_addr, new, access_key);
1255 *success = !ret && old == *old_addr;
1256 *old_addr = old;
1257 break;
1258 }
1259 default:
1260 return -EINVAL;
1261 }
1262 if (*success)
1263 mark_page_dirty_in_slot(kvm, slot, gfn);
1264 /*
1265 * Assume that the fault is caused by protection, either key protection
1266 * or user page write protection.
1267 */
1268 if (ret == -EFAULT)
1269 ret = PGM_PROTECTION;
1270 return ret;
1271}
1272
1164/**1273/**
1165 * guest_translate_address_with_key - translate guest logical into guest absolute address1274 * guest_translate_address_with_key - translate guest logical into guest absolute address
1166 * @vcpu: virtual cpu1275 * @vcpu: virtual cpu
arch/s390/kvm/gaccess.h+3−0
@@ -206,6 +206,9 @@ int access_guest_with_key(struct kvm_vcpu *vcpu, unsigned long ga, u8 ar, · 205 unmodified lines
206int access_guest_real(struct kvm_vcpu *vcpu, unsigned long gra,206int access_guest_real(struct kvm_vcpu *vcpu, unsigned long gra,
207 void *data, unsigned long len, enum gacc_mode mode);207 void *data, unsigned long len, enum gacc_mode mode);
208 208
209int cmpxchg_guest_abs_with_key(struct kvm *kvm, gpa_t gpa, int len, __uint128_t *old,
210 __uint128_t new, u8 access_key, bool *success);
211
209/**212/**
210 * write_guest_with_key - copy data from kernel space to guest space213 * write_guest_with_key - copy data from kernel space to guest space
211 * @vcpu: virtual cpu214 * @vcpu: virtual cpu
arch/s390/kvm/interrupt.c+8−5
@@ -3103,9 +3103,9 @@ static enum hrtimer_restart gisa_vcpu_kicker(struct hrtimer *timer) · 3102 unmodified lines
3103static void process_gib_alert_list(void)3103static void process_gib_alert_list(void)
3104{3104{
3105 struct kvm_s390_gisa_interrupt *gi;3105 struct kvm_s390_gisa_interrupt *gi;
3106 u32 final, gisa_phys, origin = 0UL;
3106 struct kvm_s390_gisa *gisa;3107 struct kvm_s390_gisa *gisa;
3107 struct kvm *kvm;3108 struct kvm *kvm;
3108 u32 final, origin = 0UL;
3109 3109
3110 do {3110 do {
3111 /*3111 /*
@@ -3131,9 +3131,10 @@ static void process_gib_alert_list(void) · 19 unmodified lines
3131 * interruptions asap.3131 * interruptions asap.
3132 */3132 */
3133 while (origin & GISA_ADDR_MASK) {3133 while (origin & GISA_ADDR_MASK) {
3134 gisa = (struct kvm_s390_gisa *)(u64)origin;3134 gisa_phys = origin;
3135 gisa = phys_to_virt(gisa_phys);
3135 origin = gisa->next_alert;3136 origin = gisa->next_alert;
3136 gisa->next_alert = (u32)(u64)gisa;3137 gisa->next_alert = gisa_phys;
3137 kvm = container_of(gisa, struct sie_page2, gisa)->kvm;3138 kvm = container_of(gisa, struct sie_page2, gisa)->kvm;
3138 gi = &kvm->arch.gisa_int;3139 gi = &kvm->arch.gisa_int;
3139 if (hrtimer_active(&gi->timer))3140 if (hrtimer_active(&gi->timer))
@@ -3415,8 +3416,9 @@ void kvm_s390_gib_destroy(void) · 275 unmodified lines
3415 gib = NULL;3416 gib = NULL;
3416}3417}
3417 3418
3418int kvm_s390_gib_init(u8 nisc)3419int __init kvm_s390_gib_init(u8 nisc)
3419{3420{
3421 u32 gib_origin;
3420 int rc = 0;3422 int rc = 0;
3421 3423
3422 if (!css_general_characteristics.aiv) {3424 if (!css_general_characteristics.aiv) {
@@ -3438,7 +3440,8 @@ int kvm_s390_gib_init(u8 nisc) · 15 unmodified lines
3438 }3440 }
3439 3441
3440 gib->nisc = nisc;3442 gib->nisc = nisc;
3441 if (chsc_sgib((u32)(u64)gib)) {3443 gib_origin = virt_to_phys(gib);
3444 if (chsc_sgib(gib_origin)) {
3442 pr_err("Associating the GIB with the AIV facility failed\n");3445 pr_err("Associating the GIB with the AIV facility failed\n");
3443 free_page((unsigned long)gib);3446 free_page((unsigned long)gib);
3444 gib = NULL;3447 gib = NULL;
arch/s390/kvm/kvm-s390.c+208−140
@@ -256,17 +256,6 @@ debug_info_t *kvm_s390_dbf; · 255 unmodified lines
256debug_info_t *kvm_s390_dbf_uv;256debug_info_t *kvm_s390_dbf_uv;
257 257
258/* Section: not file related */258/* Section: not file related */
259int kvm_arch_hardware_enable(void)
260{
261 /* every s390 is virtualization enabled ;-) */
262 return 0;
263}
264
265int kvm_arch_check_processor_compat(void *opaque)
266{
267 return 0;
268}
269
270/* forward declarations */259/* forward declarations */
271static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start,260static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start,
272 unsigned long end);261 unsigned long end);
@@ -329,25 +318,6 @@ static struct notifier_block kvm_clock_notifier = { · 56 unmodified lines
329 .notifier_call = kvm_clock_sync,318 .notifier_call = kvm_clock_sync,
330};319};
331 320
332int kvm_arch_hardware_setup(void *opaque)
333{
334 gmap_notifier.notifier_call = kvm_gmap_notifier;
335 gmap_register_pte_notifier(&gmap_notifier);
336 vsie_gmap_notifier.notifier_call = kvm_s390_vsie_gmap_notifier;
337 gmap_register_pte_notifier(&vsie_gmap_notifier);
338 atomic_notifier_chain_register(&s390_epoch_delta_notifier,
339 &kvm_clock_notifier);
340 return 0;
341}
342
343void kvm_arch_hardware_unsetup(void)
344{
345 gmap_unregister_pte_notifier(&gmap_notifier);
346 gmap_unregister_pte_notifier(&vsie_gmap_notifier);
347 atomic_notifier_chain_unregister(&s390_epoch_delta_notifier,
348 &kvm_clock_notifier);
349}
350
351static void allow_cpu_feat(unsigned long nr)321static void allow_cpu_feat(unsigned long nr)
352{322{
353 set_bit_inv(nr, kvm_s390_available_cpu_feat);323 set_bit_inv(nr, kvm_s390_available_cpu_feat);
@@ -385,7 +355,7 @@ static __always_inline void __insn32_query(unsigned int opcode, u8 *query) · 31 unmodified lines
385#define INSN_SORTL 0xb938355#define INSN_SORTL 0xb938
386#define INSN_DFLTCC 0xb939356#define INSN_DFLTCC 0xb939
387 357
388static void kvm_s390_cpu_feat_init(void)358static void __init kvm_s390_cpu_feat_init(void)
389{359{
390 int i;360 int i;
391 361
@@ -488,7 +458,7 @@ static void kvm_s390_cpu_feat_init(void) · 96 unmodified lines
488 */458 */
489}459}
490 460
491int kvm_arch_init(void *opaque)461static int __init __kvm_s390_init(void)
492{462{
493 int rc = -ENOMEM;463 int rc = -ENOMEM;
494 464
@@ -498,11 +468,11 @@ int kvm_arch_init(void *opaque) · 3 unmodified lines
498 468
499 kvm_s390_dbf_uv = debug_register("kvm-uv", 32, 1, 7 * sizeof(long));469 kvm_s390_dbf_uv = debug_register("kvm-uv", 32, 1, 7 * sizeof(long));
500 if (!kvm_s390_dbf_uv)470 if (!kvm_s390_dbf_uv)
501 goto out;471 goto err_kvm_uv;
502 472
503 if (debug_register_view(kvm_s390_dbf, &debug_sprintf_view) ||473 if (debug_register_view(kvm_s390_dbf, &debug_sprintf_view) ||
504 debug_register_view(kvm_s390_dbf_uv, &debug_sprintf_view))474 debug_register_view(kvm_s390_dbf_uv, &debug_sprintf_view))
505 goto out;475 goto err_debug_view;
506 476
507 kvm_s390_cpu_feat_init();477 kvm_s390_cpu_feat_init();
508 478
@@ -510,30 +480,49 @@ int kvm_arch_init(void *opaque) · 1 unmodified line
510 rc = kvm_register_device_ops(&kvm_flic_ops, KVM_DEV_TYPE_FLIC);480 rc = kvm_register_device_ops(&kvm_flic_ops, KVM_DEV_TYPE_FLIC);
511 if (rc) {481 if (rc) {
512 pr_err("A FLIC registration call failed with rc=%d\n", rc);482 pr_err("A FLIC registration call failed with rc=%d\n", rc);
513 goto out;483 goto err_flic;
514 }484 }
515 485
516 if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)) {486 if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)) {
517 rc = kvm_s390_pci_init();487 rc = kvm_s390_pci_init();
518 if (rc) {488 if (rc) {
519 pr_err("Unable to allocate AIFT for PCI\n");489 pr_err("Unable to allocate AIFT for PCI\n");
520 goto out;490 goto err_pci;
521 }491 }
522 }492 }
523 493
524 rc = kvm_s390_gib_init(GAL_ISC);494 rc = kvm_s390_gib_init(GAL_ISC);
525 if (rc)495 if (rc)
526 goto out;496 goto err_gib;
497
498 gmap_notifier.notifier_call = kvm_gmap_notifier;
499 gmap_register_pte_notifier(&gmap_notifier);
500 vsie_gmap_notifier.notifier_call = kvm_s390_vsie_gmap_notifier;
501 gmap_register_pte_notifier(&vsie_gmap_notifier);
502 atomic_notifier_chain_register(&s390_epoch_delta_notifier,
503 &kvm_clock_notifier);
527 504
528 return 0;505 return 0;
529 506
530out:507err_gib:
531 kvm_arch_exit();508 if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM))
509 kvm_s390_pci_exit();
510err_pci:
511err_flic:
512err_debug_view:
513 debug_unregister(kvm_s390_dbf_uv);
514err_kvm_uv:
515 debug_unregister(kvm_s390_dbf);
532 return rc;516 return rc;
533}517}
534 518
535void kvm_arch_exit(void)519static void __kvm_s390_exit(void)
536{520{
521 gmap_unregister_pte_notifier(&gmap_notifier);
522 gmap_unregister_pte_notifier(&vsie_gmap_notifier);
523 atomic_notifier_chain_unregister(&s390_epoch_delta_notifier,
524 &kvm_clock_notifier);
525
537 kvm_s390_gib_destroy();526 kvm_s390_gib_destroy();
538 if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM))527 if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM))
539 kvm_s390_pci_exit();528 kvm_s390_pci_exit();
@@ -584,7 +573,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · 44 unmodified lines
584 case KVM_CAP_S390_VCPU_RESETS:573 case KVM_CAP_S390_VCPU_RESETS:
585 case KVM_CAP_SET_GUEST_DEBUG:574 case KVM_CAP_SET_GUEST_DEBUG:
586 case KVM_CAP_S390_DIAG318:575 case KVM_CAP_S390_DIAG318:
587 case KVM_CAP_S390_MEM_OP_EXTENSION:
588 r = 1;576 r = 1;
589 break;577 break;
590 case KVM_CAP_SET_GUEST_DEBUG2:578 case KVM_CAP_SET_GUEST_DEBUG2:
@@ -598,6 +586,15 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · 7 unmodified lines
598 case KVM_CAP_S390_MEM_OP:586 case KVM_CAP_S390_MEM_OP:
599 r = MEM_OP_MAX_SIZE;587 r = MEM_OP_MAX_SIZE;
600 break;588 break;
589 case KVM_CAP_S390_MEM_OP_EXTENSION:
590 /*
591 * Flag bits indicating which extensions are supported.
592 * If r > 0, the base extension must also be supported/indicated,
593 * in order to maintain backwards compatibility.
594 */
595 r = KVM_S390_MEMOP_EXTENSION_CAP_BASE |
596 KVM_S390_MEMOP_EXTENSION_CAP_CMPXCHG;
597 break;
601 case KVM_CAP_NR_VCPUS:598 case KVM_CAP_NR_VCPUS:
602 case KVM_CAP_MAX_VCPUS:599 case KVM_CAP_MAX_VCPUS:
603 case KVM_CAP_MAX_VCPU_ID:600 case KVM_CAP_MAX_VCPU_ID:
@@ -2764,41 +2761,33 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) · 2160 unmodified lines
2764 return r;2761 return r;
2765}2762}
2766 2763
2767static bool access_key_invalid(u8 access_key)2764static int mem_op_validate_common(struct kvm_s390_mem_op *mop, u64 supported_flags)
2768{
2769 return access_key > 0xf;
2770}
2771
2772static int kvm_s390_vm_mem_op(struct kvm *kvm, struct kvm_s390_mem_op *mop)
2773{2765{
2774 void __user *uaddr = (void __user *)mop->buf;
2775 u64 supported_flags;
2776 void *tmpbuf = NULL;
2777 int r, srcu_idx;
2778
2779 supported_flags = KVM_S390_MEMOP_F_SKEY_PROTECTION
2780 | KVM_S390_MEMOP_F_CHECK_ONLY;
2781 if (mop->flags & ~supported_flags || !mop->size)2766 if (mop->flags & ~supported_flags || !mop->size)
2782 return -EINVAL;2767 return -EINVAL;
2783 if (mop->size > MEM_OP_MAX_SIZE)2768 if (mop->size > MEM_OP_MAX_SIZE)
2784 return -E2BIG;2769 return -E2BIG;
2785 /*
2786 * This is technically a heuristic only, if the kvm->lock is not
2787 * taken, it is not guaranteed that the vm is/remains non-protected.
2788 * This is ok from a kernel perspective, wrongdoing is detected
2789 * on the access, -EFAULT is returned and the vm may crash the
2790 * next time it accesses the memory in question.
2791 * There is no sane usecase to do switching and a memop on two
2792 * different CPUs at the same time.
2793 */
2794 if (kvm_s390_pv_get_handle(kvm))
2795 return -EINVAL;
2796 if (mop->flags & KVM_S390_MEMOP_F_SKEY_PROTECTION) {2770 if (mop->flags & KVM_S390_MEMOP_F_SKEY_PROTECTION) {
2797 if (access_key_invalid(mop->key))2771 if (mop->key > 0xf)
2798 return -EINVAL;2772 return -EINVAL;
2799 } else {2773 } else {
2800 mop->key = 0;2774 mop->key = 0;
2801 }2775 }
2776 return 0;
2777}
2778
2779static int kvm_s390_vm_mem_op_abs(struct kvm *kvm, struct kvm_s390_mem_op *mop)
2780{
2781 void __user *uaddr = (void __user *)mop->buf;
2782 enum gacc_mode acc_mode;
2783 void *tmpbuf = NULL;
2784 int r, srcu_idx;
2785
2786 r = mem_op_validate_common(mop, KVM_S390_MEMOP_F_SKEY_PROTECTION |
2787 KVM_S390_MEMOP_F_CHECK_ONLY);
2788 if (r)
2789 return r;
2790
2802 if (!(mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY)) {2791 if (!(mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY)) {
2803 tmpbuf = vmalloc(mop->size);2792 tmpbuf = vmalloc(mop->size);
2804 if (!tmpbuf)2793 if (!tmpbuf)
@@ -2812,35 +2801,25 @@ static int kvm_s390_vm_mem_op(struct kvm *kvm, struct kvm_s390_mem_op *mop) · 7 unmodified lines
2812 goto out_unlock;2801 goto out_unlock;
2813 }2802 }
2814 2803
2815 switch (mop->op) {2804 acc_mode = mop->op == KVM_S390_MEMOP_ABSOLUTE_READ ? GACC_FETCH : GACC_STORE;
2816 case KVM_S390_MEMOP_ABSOLUTE_READ: {2805 if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
2817 if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {2806 r = check_gpa_range(kvm, mop->gaddr, mop->size, acc_mode, mop->key);
2818 r = check_gpa_range(kvm, mop->gaddr, mop->size, GACC_FETCH, mop->key);2807 goto out_unlock;
2819 } else {
2820 r = access_guest_abs_with_key(kvm, mop->gaddr, tmpbuf,
2821 mop->size, GACC_FETCH, mop->key);
2822 if (r == 0) {
2823 if (copy_to_user(uaddr, tmpbuf, mop->size))
2824 r = -EFAULT;
2825 }
2826 }
2827 break;
2828 }2808 }
2829 case KVM_S390_MEMOP_ABSOLUTE_WRITE: {2809 if (acc_mode == GACC_FETCH) {
2830 if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {2810 r = access_guest_abs_with_key(kvm, mop->gaddr, tmpbuf,
2831 r = check_gpa_range(kvm, mop->gaddr, mop->size, GACC_STORE, mop->key);2811 mop->size, GACC_FETCH, mop->key);
2832 } else {2812 if (r)
2833 if (copy_from_user(tmpbuf, uaddr, mop->size)) {2813 goto out_unlock;
2834 r = -EFAULT;2814 if (copy_to_user(uaddr, tmpbuf, mop->size))
2835 break;2815 r = -EFAULT;
2836 }2816 } else {
2837 r = access_guest_abs_with_key(kvm, mop->gaddr, tmpbuf,2817 if (copy_from_user(tmpbuf, uaddr, mop->size)) {
2838 mop->size, GACC_STORE, mop->key);2818 r = -EFAULT;
2819 goto out_unlock;
2839 }2820 }
2840 break;2821 r = access_guest_abs_with_key(kvm, mop->gaddr, tmpbuf,
2841 }2822 mop->size, GACC_STORE, mop->key);
2842 default:
2843 r = -EINVAL;
2844 }2823 }
2845 2824
2846out_unlock:2825out_unlock:
@@ -2850,6 +2829,75 @@ out_unlock: · 3 unmodified lines
2850 return r;2829 return r;
2851}2830}
2852 2831
2832static int kvm_s390_vm_mem_op_cmpxchg(struct kvm *kvm, struct kvm_s390_mem_op *mop)
2833{
2834 void __user *uaddr = (void __user *)mop->buf;
2835 void __user *old_addr = (void __user *)mop->old_addr;
2836 union {
2837 __uint128_t quad;
2838 char raw[sizeof(__uint128_t)];
2839 } old = { .quad = 0}, new = { .quad = 0 };
2840 unsigned int off_in_quad = sizeof(new) - mop->size;
2841 int r, srcu_idx;
2842 bool success;
2843
2844 r = mem_op_validate_common(mop, KVM_S390_MEMOP_F_SKEY_PROTECTION);
2845 if (r)
2846 return r;
2847 /*
2848 * This validates off_in_quad. Checking that size is a power
2849 * of two is not necessary, as cmpxchg_guest_abs_with_key
2850 * takes care of that
2851 */
2852 if (mop->size > sizeof(new))
2853 return -EINVAL;
2854 if (copy_from_user(&new.raw[off_in_quad], uaddr, mop->size))
2855 return -EFAULT;
2856 if (copy_from_user(&old.raw[off_in_quad], old_addr, mop->size))
2857 return -EFAULT;
2858
2859 srcu_idx = srcu_read_lock(&kvm->srcu);
2860
2861 if (kvm_is_error_gpa(kvm, mop->gaddr)) {
2862 r = PGM_ADDRESSING;
2863 goto out_unlock;
2864 }
2865
2866 r = cmpxchg_guest_abs_with_key(kvm, mop->gaddr, mop->size, &old.quad,
2867 new.quad, mop->key, &success);
2868 if (!success && copy_to_user(old_addr, &old.raw[off_in_quad], mop->size))
2869 r = -EFAULT;
2870
2871out_unlock:
2872 srcu_read_unlock(&kvm->srcu, srcu_idx);
2873 return r;
2874}
2875
2876static int kvm_s390_vm_mem_op(struct kvm *kvm, struct kvm_s390_mem_op *mop)
2877{
2878 /*
2879 * This is technically a heuristic only, if the kvm->lock is not
2880 * taken, it is not guaranteed that the vm is/remains non-protected.
2881 * This is ok from a kernel perspective, wrongdoing is detected
2882 * on the access, -EFAULT is returned and the vm may crash the
2883 * next time it accesses the memory in question.
2884 * There is no sane usecase to do switching and a memop on two
2885 * different CPUs at the same time.
2886 */
2887 if (kvm_s390_pv_get_handle(kvm))
2888 return -EINVAL;
2889
2890 switch (mop->op) {
2891 case KVM_S390_MEMOP_ABSOLUTE_READ:
2892 case KVM_S390_MEMOP_ABSOLUTE_WRITE:
2893 return kvm_s390_vm_mem_op_abs(kvm, mop);
2894 case KVM_S390_MEMOP_ABSOLUTE_CMPXCHG:
2895 return kvm_s390_vm_mem_op_cmpxchg(kvm, mop);
2896 default:
2897 return -EINVAL;
2898 }
2899}
2900
2853long kvm_arch_vm_ioctl(struct file *filp,2901long kvm_arch_vm_ioctl(struct file *filp,
2854 unsigned int ioctl, unsigned long arg)2902 unsigned int ioctl, unsigned long arg)
2855{2903{
@@ -5249,62 +5297,54 @@ static long kvm_s390_vcpu_mem_op(struct kvm_vcpu *vcpu, · 2393 unmodified lines
5249 struct kvm_s390_mem_op *mop)5297 struct kvm_s390_mem_op *mop)
5250{5298{
5251 void __user *uaddr = (void __user *)mop->buf;5299 void __user *uaddr = (void __user *)mop->buf;
5300 enum gacc_mode acc_mode;
5252 void *tmpbuf = NULL;5301 void *tmpbuf = NULL;
5253 int r = 0;5302 int r;
5254 const u64 supported_flags = KVM_S390_MEMOP_F_INJECT_EXCEPTION
5255 | KVM_S390_MEMOP_F_CHECK_ONLY
5256 | KVM_S390_MEMOP_F_SKEY_PROTECTION;
5257 5303
5258 if (mop->flags & ~supported_flags || mop->ar >= NUM_ACRS || !mop->size)5304 r = mem_op_validate_common(mop, KVM_S390_MEMOP_F_INJECT_EXCEPTION |
5305 KVM_S390_MEMOP_F_CHECK_ONLY |
5306 KVM_S390_MEMOP_F_SKEY_PROTECTION);
5307 if (r)
5308 return r;
5309 if (mop->ar >= NUM_ACRS)
5259 return -EINVAL;5310 return -EINVAL;
5260 if (mop->size > MEM_OP_MAX_SIZE)
5261 return -E2BIG;
5262 if (kvm_s390_pv_cpu_is_protected(vcpu))5311 if (kvm_s390_pv_cpu_is_protected(vcpu))
5263 return -EINVAL;5312 return -EINVAL;
5264 if (mop->flags & KVM_S390_MEMOP_F_SKEY_PROTECTION) {
5265 if (access_key_invalid(mop->key))
5266 return -EINVAL;
5267 } else {
5268 mop->key = 0;
5269 }
5270 if (!(mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY)) {5313 if (!(mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY)) {
5271 tmpbuf = vmalloc(mop->size);5314 tmpbuf = vmalloc(mop->size);
5272 if (!tmpbuf)5315 if (!tmpbuf)
5273 return -ENOMEM;5316 return -ENOMEM;
5274 }5317 }
5275 5318
5276 switch (mop->op) {5319 acc_mode = mop->op == KVM_S390_MEMOP_LOGICAL_READ ? GACC_FETCH : GACC_STORE;
5277 case KVM_S390_MEMOP_LOGICAL_READ:5320 if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
5278 if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {5321 r = check_gva_range(vcpu, mop->gaddr, mop->ar, mop->size,
5279 r = check_gva_range(vcpu, mop->gaddr, mop->ar, mop->size,5322 acc_mode, mop->key);
5280 GACC_FETCH, mop->key);5323 goto out_inject;
5281 break;5324 }
5282 }5325 if (acc_mode == GACC_FETCH) {
5283 r = read_guest_with_key(vcpu, mop->gaddr, mop->ar, tmpbuf,5326 r = read_guest_with_key(vcpu, mop->gaddr, mop->ar, tmpbuf,
5284 mop->size, mop->key);5327 mop->size, mop->key);
5285 if (r == 0) {5328 if (r)
5286 if (copy_to_user(uaddr, tmpbuf, mop->size))5329 goto out_inject;
5287 r = -EFAULT;5330 if (copy_to_user(uaddr, tmpbuf, mop->size)) {
5288 }5331 r = -EFAULT;
5289 break;5332 goto out_free;
5290 case KVM_S390_MEMOP_LOGICAL_WRITE:
5291 if (mop->flags & KVM_S390_MEMOP_F_CHECK_ONLY) {
5292 r = check_gva_range(vcpu, mop->gaddr, mop->ar, mop->size,
5293 GACC_STORE, mop->key);
5294 break;
5295 }5333 }
5334 } else {
5296 if (copy_from_user(tmpbuf, uaddr, mop->size)) {5335 if (copy_from_user(tmpbuf, uaddr, mop->size)) {
5297 r = -EFAULT;5336 r = -EFAULT;
5298 break;5337 goto out_free;
5299 }5338 }
5300 r = write_guest_with_key(vcpu, mop->gaddr, mop->ar, tmpbuf,5339 r = write_guest_with_key(vcpu, mop->gaddr, mop->ar, tmpbuf,
5301 mop->size, mop->key);5340 mop->size, mop->key);
5302 break;
5303 }5341 }
5304 5342
5343out_inject:
5305 if (r > 0 && (mop->flags & KVM_S390_MEMOP_F_INJECT_EXCEPTION) != 0)5344 if (r > 0 && (mop->flags & KVM_S390_MEMOP_F_INJECT_EXCEPTION) != 0)
5306 kvm_s390_inject_prog_irq(vcpu, &vcpu->arch.pgm);5345 kvm_s390_inject_prog_irq(vcpu, &vcpu->arch.pgm);
5307 5346
5347out_free:
5308 vfree(tmpbuf);5348 vfree(tmpbuf);
5309 return r;5349 return r;
5310}5350}
@@ -5633,23 +5673,40 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, · 322 unmodified lines
5633 if (kvm_s390_pv_get_handle(kvm))5673 if (kvm_s390_pv_get_handle(kvm))
5634 return -EINVAL;5674 return -EINVAL;
5635 5675
5636 if (change == KVM_MR_DELETE || change == KVM_MR_FLAGS_ONLY)5676 if (change != KVM_MR_DELETE && change != KVM_MR_FLAGS_ONLY) {
5637 return 0;5677 /*
5678 * A few sanity checks. We can have memory slots which have to be
5679 * located/ended at a segment boundary (1MB). The memory in userland is
5680 * ok to be fragmented into various different vmas. It is okay to mmap()
5681 * and munmap() stuff in this slot after doing this call at any time
5682 */
5638 5683
5639 /* A few sanity checks. We can have memory slots which have to be5684 if (new->userspace_addr & 0xffffful)
5640 located/ended at a segment boundary (1MB). The memory in userland is5685 return -EINVAL;
5641 ok to be fragmented into various different vmas. It is okay to mmap()
5642 and munmap() stuff in this slot after doing this call at any time */
5643 5686
5644 if (new->userspace_addr & 0xffffful)5687 size = new->npages * PAGE_SIZE;
5645 return -EINVAL;5688 if (size & 0xffffful)
5689 return -EINVAL;
5646 5690
5647 size = new->npages * PAGE_SIZE;5691 if ((new->base_gfn * PAGE_SIZE) + size > kvm->arch.mem_limit)
5648 if (size & 0xffffful)5692 return -EINVAL;
5649 return -EINVAL;5693 }
5650 5694
5651 if ((new->base_gfn * PAGE_SIZE) + size > kvm->arch.mem_limit)5695 if (!kvm->arch.migration_mode)
5652 return -EINVAL;5696 return 0;
5697
5698 /*
5699 * Turn off migration mode when:
5700 * - userspace creates a new memslot with dirty logging off,
5701 * - userspace modifies an existing memslot (MOVE or FLAGS_ONLY) and
5702 * dirty logging is turned off.
5703 * Migration mode expects dirty page logging being enabled to store
5704 * its dirty bitmap.
5705 */
5706 if (change != KVM_MR_DELETE &&
5707 !(new->flags & KVM_MEM_LOG_DIRTY_PAGES))
5708 WARN(kvm_s390_vm_stop_migration(kvm),
5709 "Failed to stop migration mode");
5653 5710
5654 return 0;5711 return 0;
5655}5712}
@@ -5696,7 +5753,7 @@ static inline unsigned long nonhyp_mask(int i) · 40 unmodified lines
5696 5753
5697static int __init kvm_s390_init(void)5754static int __init kvm_s390_init(void)
5698{5755{
5699 int i;5756 int i, r;
5700 5757
5701 if (!sclp.has_sief2) {5758 if (!sclp.has_sief2) {
5702 pr_info("SIE is not available\n");5759 pr_info("SIE is not available\n");
@@ -5712,12 +5769,23 @@ static int __init kvm_s390_init(void) · 9 unmodified lines
5712 kvm_s390_fac_base[i] |=5769 kvm_s390_fac_base[i] |=
5713 stfle_fac_list[i] & nonhyp_mask(i);5770 stfle_fac_list[i] & nonhyp_mask(i);
5714 5771
5715 return kvm_init(NULL, sizeof(struct kvm_vcpu), 0, THIS_MODULE);5772 r = __kvm_s390_init();
5773 if (r)
5774 return r;
5775
5776 r = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
5777 if (r) {
5778 __kvm_s390_exit();
5779 return r;
5780 }
5781 return 0;
5716}5782}
5717 5783
5718static void __exit kvm_s390_exit(void)5784static void __exit kvm_s390_exit(void)
5719{5785{
5720 kvm_exit();5786 kvm_exit();
5787
5788 __kvm_s390_exit();
5721}5789}
5722 5790
5723module_init(kvm_s390_init);5791module_init(kvm_s390_init);
arch/s390/kvm/kvm-s390.h+1−1
@@ -470,7 +470,7 @@ void kvm_s390_gisa_clear(struct kvm *kvm); · 469 unmodified lines
470void kvm_s390_gisa_destroy(struct kvm *kvm);470void kvm_s390_gisa_destroy(struct kvm *kvm);
471void kvm_s390_gisa_disable(struct kvm *kvm);471void kvm_s390_gisa_disable(struct kvm *kvm);
472void kvm_s390_gisa_enable(struct kvm *kvm);472void kvm_s390_gisa_enable(struct kvm *kvm);
473int kvm_s390_gib_init(u8 nisc);473int __init kvm_s390_gib_init(u8 nisc);
474void kvm_s390_gib_destroy(void);474void kvm_s390_gib_destroy(void);
475 475
476/* implemented in guestdbg.c */476/* implemented in guestdbg.c */
arch/s390/kvm/pci.c+1−1
@@ -672,7 +672,7 @@ out: · 671 unmodified lines
672 return r;672 return r;
673}673}
674 674
675int kvm_s390_pci_init(void)675int __init kvm_s390_pci_init(void)
676{676{
677 zpci_kvm_hook.kvm_register = kvm_s390_pci_register_kvm;677 zpci_kvm_hook.kvm_register = kvm_s390_pci_register_kvm;
678 zpci_kvm_hook.kvm_unregister = kvm_s390_pci_unregister_kvm;678 zpci_kvm_hook.kvm_unregister = kvm_s390_pci_unregister_kvm;
arch/s390/kvm/pci.h+1−1
@@ -60,7 +60,7 @@ void kvm_s390_pci_clear_list(struct kvm *kvm); · 59 unmodified lines
60 60
61int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op *args);61int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op *args);
62 62
63int kvm_s390_pci_init(void);63int __init kvm_s390_pci_init(void);
64void kvm_s390_pci_exit(void);64void kvm_s390_pci_exit(void);
65 65
66static inline bool kvm_s390_pci_interp_allowed(void)66static inline bool kvm_s390_pci_interp_allowed(void)
arch/x86/events/intel/core.c+1−0
@@ -6495,6 +6495,7 @@ __init int intel_pmu_init(void) · 6494 unmodified lines
6495 x86_pmu.pebs_constraints = intel_spr_pebs_event_constraints;6495 x86_pmu.pebs_constraints = intel_spr_pebs_event_constraints;
6496 x86_pmu.extra_regs = intel_spr_extra_regs;6496 x86_pmu.extra_regs = intel_spr_extra_regs;
6497 x86_pmu.limit_period = spr_limit_period;6497 x86_pmu.limit_period = spr_limit_period;
6498 x86_pmu.pebs_ept = 1;
6498 x86_pmu.pebs_aliases = NULL;6499 x86_pmu.pebs_aliases = NULL;
6499 x86_pmu.pebs_prec_dist = true;6500 x86_pmu.pebs_prec_dist = true;
6500 x86_pmu.pebs_block = true;6501 x86_pmu.pebs_block = true;
arch/x86/events/intel/ds.c+3−1
@@ -2366,8 +2366,10 @@ void __init intel_ds_init(void) · 2365 unmodified lines
2366 x86_pmu.large_pebs_flags |= PERF_SAMPLE_TIME;2366 x86_pmu.large_pebs_flags |= PERF_SAMPLE_TIME;
2367 break;2367 break;
2368 2368
2369 case 4:
2370 case 5:2369 case 5:
2370 x86_pmu.pebs_ept = 1;
2371 fallthrough;
2372 case 4:
2371 x86_pmu.drain_pebs = intel_pmu_drain_pebs_icl;2373 x86_pmu.drain_pebs = intel_pmu_drain_pebs_icl;
2372 x86_pmu.pebs_record_size = sizeof(struct pebs_basic);2374 x86_pmu.pebs_record_size = sizeof(struct pebs_basic);
2373 if (x86_pmu.intel_cap.pebs_baseline) {2375 if (x86_pmu.intel_cap.pebs_baseline) {
arch/x86/include/asm/cpufeatures.h+3−0
@@ -315,6 +315,9 @@ · 314 unmodified lines
315#define X86_FEATURE_AVX512_BF16 (12*32+ 5) /* AVX512 BFLOAT16 instructions */315#define X86_FEATURE_AVX512_BF16 (12*32+ 5) /* AVX512 BFLOAT16 instructions */
316#define X86_FEATURE_CMPCCXADD (12*32+ 7) /* "" CMPccXADD instructions */316#define X86_FEATURE_CMPCCXADD (12*32+ 7) /* "" CMPccXADD instructions */
317#define X86_FEATURE_ARCH_PERFMON_EXT (12*32+ 8) /* "" Intel Architectural PerfMon Extension */317#define X86_FEATURE_ARCH_PERFMON_EXT (12*32+ 8) /* "" Intel Architectural PerfMon Extension */
318#define X86_FEATURE_FZRM (12*32+10) /* "" Fast zero-length REP MOVSB */
319#define X86_FEATURE_FSRS (12*32+11) /* "" Fast short REP STOSB */
320#define X86_FEATURE_FSRC (12*32+12) /* "" Fast short REP {CMPSB,SCASB} */
318#define X86_FEATURE_LKGS (12*32+18) /* "" Load "kernel" (userspace) GS */321#define X86_FEATURE_LKGS (12*32+18) /* "" Load "kernel" (userspace) GS */
319#define X86_FEATURE_AMX_FP16 (12*32+21) /* "" AMX fp16 Support */322#define X86_FEATURE_AMX_FP16 (12*32+21) /* "" AMX fp16 Support */
320#define X86_FEATURE_AVX_IFMA (12*32+23) /* "" Support for VPMADD52[H,L]UQ */323#define X86_FEATURE_AVX_IFMA (12*32+23) /* "" Support for VPMADD52[H,L]UQ */
arch/x86/include/asm/hyperv-tlfs.h+3−0
@@ -269,6 +269,9 @@ enum hv_isolation_type { · 268 unmodified lines
269/* TSC invariant control */269/* TSC invariant control */
270#define HV_X64_MSR_TSC_INVARIANT_CONTROL 0x40000118270#define HV_X64_MSR_TSC_INVARIANT_CONTROL 0x40000118
271 271
272/* HV_X64_MSR_TSC_INVARIANT_CONTROL bits */
273#define HV_EXPOSE_INVARIANT_TSC BIT_ULL(0)
274
272/* Register name aliases for temporary compatibility */275/* Register name aliases for temporary compatibility */
273#define HV_X64_MSR_STIMER0_COUNT HV_REGISTER_STIMER0_COUNT276#define HV_X64_MSR_STIMER0_COUNT HV_REGISTER_STIMER0_COUNT
274#define HV_X64_MSR_STIMER0_CONFIG HV_REGISTER_STIMER0_CONFIG277#define HV_X64_MSR_STIMER0_CONFIG HV_REGISTER_STIMER0_CONFIG
arch/x86/include/asm/idtentry.h+6−10
@@ -582,18 +582,14 @@ DECLARE_IDTENTRY_RAW(X86_TRAP_MC, xenpv_exc_machine_check); · 581 unmodified lines
582 582
583/* NMI */583/* NMI */
584 584
585#if defined(CONFIG_X86_64) && IS_ENABLED(CONFIG_KVM_INTEL)585#if IS_ENABLED(CONFIG_KVM_INTEL)
586/*586/*
587 * Special NOIST entry point for VMX which invokes this on the kernel587 * Special entry point for VMX which invokes this on the kernel stack, even for
588 * stack. asm_exc_nmi() requires an IST to work correctly vs. the NMI588 * 64-bit, i.e. without using an IST. asm_exc_nmi() requires an IST to work
589 * 'executing' marker.589 * correctly vs. the NMI 'executing' marker. Used for 32-bit kernels as well
590 *590 * to avoid more ifdeffery.
591 * On 32bit this just uses the regular NMI entry point because 32-bit does
592 * not have ISTs.
593 */591 */
594DECLARE_IDTENTRY(X86_TRAP_NMI, exc_nmi_noist);592DECLARE_IDTENTRY(X86_TRAP_NMI, exc_nmi_kvm_vmx);
595#else
596#define asm_exc_nmi_noist asm_exc_nmi
597#endif593#endif
598 594
599DECLARE_IDTENTRY_NMI(X86_TRAP_NMI, exc_nmi);595DECLARE_IDTENTRY_NMI(X86_TRAP_NMI, exc_nmi);
arch/x86/include/asm/kvm-x86-ops.h+1−1
@@ -14,6 +14,7 @@ BUILD_BUG_ON(1) · 13 unmodified lines
14 * to make a definition optional, but in this case the default will14 * to make a definition optional, but in this case the default will
15 * be __static_call_return0.15 * be __static_call_return0.
16 */16 */
17KVM_X86_OP(check_processor_compatibility)
17KVM_X86_OP(hardware_enable)18KVM_X86_OP(hardware_enable)
18KVM_X86_OP(hardware_disable)19KVM_X86_OP(hardware_disable)
19KVM_X86_OP(hardware_unsetup)20KVM_X86_OP(hardware_unsetup)
@@ -76,7 +77,6 @@ KVM_X86_OP(set_nmi_mask) · 56 unmodified lines
76KVM_X86_OP(enable_nmi_window)77KVM_X86_OP(enable_nmi_window)
77KVM_X86_OP(enable_irq_window)78KVM_X86_OP(enable_irq_window)
78KVM_X86_OP_OPTIONAL(update_cr8_intercept)79KVM_X86_OP_OPTIONAL(update_cr8_intercept)
79KVM_X86_OP(check_apicv_inhibit_reasons)
80KVM_X86_OP(refresh_apicv_exec_ctrl)80KVM_X86_OP(refresh_apicv_exec_ctrl)
81KVM_X86_OP_OPTIONAL(hwapic_irr_update)81KVM_X86_OP_OPTIONAL(hwapic_irr_update)
82KVM_X86_OP_OPTIONAL(hwapic_isr_update)82KVM_X86_OP_OPTIONAL(hwapic_isr_update)
arch/x86/include/asm/kvm_host.h+71−38
@@ -134,8 +134,6 @@ · 133 unmodified lines
134#define INVALID_PAGE (~(hpa_t)0)134#define INVALID_PAGE (~(hpa_t)0)
135#define VALID_PAGE(x) ((x) != INVALID_PAGE)135#define VALID_PAGE(x) ((x) != INVALID_PAGE)
136 136
137#define INVALID_GPA (~(gpa_t)0)
138
139/* KVM Hugepage definitions for x86 */137/* KVM Hugepage definitions for x86 */
140#define KVM_MAX_HUGEPAGE_LEVEL PG_LEVEL_1G138#define KVM_MAX_HUGEPAGE_LEVEL PG_LEVEL_1G
141#define KVM_NR_PAGE_SIZES (KVM_MAX_HUGEPAGE_LEVEL - PG_LEVEL_4K + 1)139#define KVM_NR_PAGE_SIZES (KVM_MAX_HUGEPAGE_LEVEL - PG_LEVEL_4K + 1)
@@ -514,6 +512,7 @@ struct kvm_pmc { · 372 unmodified lines
514#define MSR_ARCH_PERFMON_PERFCTR_MAX (MSR_ARCH_PERFMON_PERFCTR0 + KVM_INTEL_PMC_MAX_GENERIC - 1)512#define MSR_ARCH_PERFMON_PERFCTR_MAX (MSR_ARCH_PERFMON_PERFCTR0 + KVM_INTEL_PMC_MAX_GENERIC - 1)
515#define MSR_ARCH_PERFMON_EVENTSEL_MAX (MSR_ARCH_PERFMON_EVENTSEL0 + KVM_INTEL_PMC_MAX_GENERIC - 1)513#define MSR_ARCH_PERFMON_EVENTSEL_MAX (MSR_ARCH_PERFMON_EVENTSEL0 + KVM_INTEL_PMC_MAX_GENERIC - 1)
516#define KVM_PMC_MAX_FIXED 3514#define KVM_PMC_MAX_FIXED 3
515#define MSR_ARCH_PERFMON_FIXED_CTR_MAX (MSR_ARCH_PERFMON_FIXED_CTR0 + KVM_PMC_MAX_FIXED - 1)
517#define KVM_AMD_PMC_MAX_GENERIC 6516#define KVM_AMD_PMC_MAX_GENERIC 6
518struct kvm_pmu {517struct kvm_pmu {
519 unsigned nr_arch_gp_counters;518 unsigned nr_arch_gp_counters;
@@ -678,6 +677,11 @@ struct kvm_vcpu_hv { · 158 unmodified lines
678 } nested;677 } nested;
679};678};
680 679
680struct kvm_hypervisor_cpuid {
681 u32 base;
682 u32 limit;
683};
684
681/* Xen HVM per vcpu emulation context */685/* Xen HVM per vcpu emulation context */
682struct kvm_vcpu_xen {686struct kvm_vcpu_xen {
683 u64 hypercall_rip;687 u64 hypercall_rip;
@@ -698,6 +702,7 @@ struct kvm_vcpu_xen { · 14 unmodified lines
698 struct hrtimer timer;702 struct hrtimer timer;
699 int poll_evtchn;703 int poll_evtchn;
700 struct timer_list poll_timer;704 struct timer_list poll_timer;
705 struct kvm_hypervisor_cpuid cpuid;
701};706};
702 707
703struct kvm_queued_exception {708struct kvm_queued_exception {
@@ -826,7 +831,7 @@ struct kvm_vcpu_arch { · 122 unmodified lines
826 831
827 int cpuid_nent;832 int cpuid_nent;
828 struct kvm_cpuid_entry2 *cpuid_entries;833 struct kvm_cpuid_entry2 *cpuid_entries;
829 u32 kvm_cpuid_base;834 struct kvm_hypervisor_cpuid kvm_cpuid;
830 835
831 u64 reserved_gpa_bits;836 u64 reserved_gpa_bits;
832 int maxphyaddr;837 int maxphyaddr;
@@ -1022,19 +1027,30 @@ struct kvm_arch_memory_slot { · 189 unmodified lines
1022};1027};
1023 1028
1024/*1029/*
1025 * We use as the mode the number of bits allocated in the LDR for the1030 * Track the mode of the optimized logical map, as the rules for decoding the
1026 * logical processor ID. It happens that these are all powers of two.1031 * destination vary per mode. Enabling the optimized logical map requires all
1027 * This makes it is very easy to detect cases where the APICs are1032 * software-enabled local APIs to be in the same mode, each addressable APIC to
1028 * configured for multiple modes; in that case, we cannot use the map and1033 * be mapped to only one MDA, and each MDA to map to at most one APIC.
1029 * hence cannot use kvm_irq_delivery_to_apic_fast either.
1030 */1034 */
1031#define KVM_APIC_MODE_XAPIC_CLUSTER 41035enum kvm_apic_logical_mode {
1032#define KVM_APIC_MODE_XAPIC_FLAT 81036 /* All local APICs are software disabled. */
1033#define KVM_APIC_MODE_X2APIC 161037 KVM_APIC_MODE_SW_DISABLED,
1038 /* All software enabled local APICs in xAPIC cluster addressing mode. */
1039 KVM_APIC_MODE_XAPIC_CLUSTER,
1040 /* All software enabled local APICs in xAPIC flat addressing mode. */
1041 KVM_APIC_MODE_XAPIC_FLAT,
1042 /* All software enabled local APICs in x2APIC mode. */
1043 KVM_APIC_MODE_X2APIC,
1044 /*
1045 * Optimized map disabled, e.g. not all local APICs in the same logical
1046 * mode, same logical ID assigned to multiple APICs, etc.
1047 */
1048 KVM_APIC_MODE_MAP_DISABLED,
1049};
1034 1050
1035struct kvm_apic_map {1051struct kvm_apic_map {
1036 struct rcu_head rcu;1052 struct rcu_head rcu;
1037 u8 mode;1053 enum kvm_apic_logical_mode logical_mode;
1038 u32 max_apic_id;1054 u32 max_apic_id;
1039 union {1055 union {
1040 struct kvm_lapic *xapic_flat_map[8];1056 struct kvm_lapic *xapic_flat_map[8];
@@ -1088,6 +1104,7 @@ struct kvm_hv { · 47 unmodified lines
1088 u64 hv_reenlightenment_control;1104 u64 hv_reenlightenment_control;
1089 u64 hv_tsc_emulation_control;1105 u64 hv_tsc_emulation_control;
1090 u64 hv_tsc_emulation_status;1106 u64 hv_tsc_emulation_status;
1107 u64 hv_invtsc_control;
1091 1108
1092 /* How many vCPUs have VP index != vCPU index */1109 /* How many vCPUs have VP index != vCPU index */
1093 atomic_t num_mismatched_vp_indexes;1110 atomic_t num_mismatched_vp_indexes;
@@ -1133,6 +1150,18 @@ struct kvm_x86_msr_filter { · 39 unmodified lines
1133 struct msr_bitmap_range ranges[16];1150 struct msr_bitmap_range ranges[16];
1134};1151};
1135 1152
1153struct kvm_x86_pmu_event_filter {
1154 __u32 action;
1155 __u32 nevents;
1156 __u32 fixed_counter_bitmap;
1157 __u32 flags;
1158 __u32 nr_includes;
1159 __u32 nr_excludes;
1160 __u64 *includes;
1161 __u64 *excludes;
1162 __u64 events[];
1163};
1164
1136enum kvm_apicv_inhibit {1165enum kvm_apicv_inhibit {
1137 1166
1138 /********************************************************************/1167 /********************************************************************/
@@ -1163,6 +1192,12 @@ enum kvm_apicv_inhibit { · 24 unmodified lines
1163 */1192 */
1164 APICV_INHIBIT_REASON_BLOCKIRQ,1193 APICV_INHIBIT_REASON_BLOCKIRQ,
1165 1194
1195 /*
1196 * APICv is disabled because not all vCPUs have a 1:1 mapping between
1197 * APIC ID and vCPU, _and_ KVM is not applying its x2APIC hotplug hack.
1198 */
1199 APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED,
1200
1166 /*1201 /*
1167 * For simplicity, the APIC acceleration is inhibited1202 * For simplicity, the APIC acceleration is inhibited
1168 * first time either APIC ID or APIC base are changed by the guest1203 * first time either APIC ID or APIC base are changed by the guest
@@ -1201,6 +1236,12 @@ enum kvm_apicv_inhibit { · 32 unmodified lines
1201 * AVIC is disabled because SEV doesn't support it.1236 * AVIC is disabled because SEV doesn't support it.
1202 */1237 */
1203 APICV_INHIBIT_REASON_SEV,1238 APICV_INHIBIT_REASON_SEV,
1239
1240 /*
1241 * AVIC is disabled because not all vCPUs with a valid LDR have a 1:1
1242 * mapping between logical ID and vCPU.
1243 */
1244 APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED,
1204};1245};
1205 1246
1206struct kvm_arch {1247struct kvm_arch {
@@ -1249,10 +1290,11 @@ struct kvm_arch { · 42 unmodified lines
1249 struct kvm_apic_map __rcu *apic_map;1290 struct kvm_apic_map __rcu *apic_map;
1250 atomic_t apic_map_dirty;1291 atomic_t apic_map_dirty;
1251 1292
1252 /* Protects apic_access_memslot_enabled and apicv_inhibit_reasons */
1253 struct rw_semaphore apicv_update_lock;
1254
1255 bool apic_access_memslot_enabled;1293 bool apic_access_memslot_enabled;
1294 bool apic_access_memslot_inhibited;
1295
1296 /* Protects apicv_inhibit_reasons */
1297 struct rw_semaphore apicv_update_lock;
1256 unsigned long apicv_inhibit_reasons;1298 unsigned long apicv_inhibit_reasons;
1257 1299
1258 gpa_t wall_clock;1300 gpa_t wall_clock;
@@ -1302,7 +1344,6 @@ struct kvm_arch { · 43 unmodified lines
1302 u32 bsp_vcpu_id;1344 u32 bsp_vcpu_id;
1303 1345
1304 u64 disabled_quirks;1346 u64 disabled_quirks;
1305 int cpu_dirty_logging_count;
1306 1347
1307 enum kvm_irqchip_mode irqchip_mode;1348 enum kvm_irqchip_mode irqchip_mode;
1308 u8 nr_reserved_ioapic_pins;1349 u8 nr_reserved_ioapic_pins;
@@ -1338,25 +1379,16 @@ struct kvm_arch { · 29 unmodified lines
1338 /* Guest can access the SGX PROVISIONKEY. */1379 /* Guest can access the SGX PROVISIONKEY. */
1339 bool sgx_provisioning_allowed;1380 bool sgx_provisioning_allowed;
1340 1381
1341 struct kvm_pmu_event_filter __rcu *pmu_event_filter;1382 struct kvm_x86_pmu_event_filter __rcu *pmu_event_filter;
1342 struct task_struct *nx_huge_page_recovery_thread;1383 struct task_struct *nx_huge_page_recovery_thread;
1343 1384
1344#ifdef CONFIG_X86_641385#ifdef CONFIG_X86_64
1345 /*
1346 * Whether the TDP MMU is enabled for this VM. This contains a
1347 * snapshot of the TDP MMU module parameter from when the VM was
1348 * created and remains unchanged for the life of the VM. If this is
1349 * true, TDP MMU handler functions will run for various MMU
1350 * operations.
1351 */
1352 bool tdp_mmu_enabled;
1353
1354 /* The number of TDP MMU pages across all roots. */1386 /* The number of TDP MMU pages across all roots. */
1355 atomic64_t tdp_mmu_pages;1387 atomic64_t tdp_mmu_pages;
1356 1388
1357 /*1389 /*
1358 * List of kvm_mmu_page structs being used as roots.1390 * List of struct kvm_mmu_pages being used as roots.
1359 * All kvm_mmu_page structs in the list should have1391 * All struct kvm_mmu_pages in the list should have
1360 * tdp_mmu_page set.1392 * tdp_mmu_page set.
1361 *1393 *
1362 * For reads, this list is protected by:1394 * For reads, this list is protected by:
@@ -1520,6 +1552,8 @@ static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical) · 157 unmodified lines
1520struct kvm_x86_ops {1552struct kvm_x86_ops {
1521 const char *name;1553 const char *name;
1522 1554
1555 int (*check_processor_compatibility)(void);
1556
1523 int (*hardware_enable)(void);1557 int (*hardware_enable)(void);
1524 void (*hardware_disable)(void);1558 void (*hardware_disable)(void);
1525 void (*hardware_unsetup)(void);1559 void (*hardware_unsetup)(void);
@@ -1608,6 +1642,8 @@ struct kvm_x86_ops { · 82 unmodified lines
1608 void (*enable_irq_window)(struct kvm_vcpu *vcpu);1642 void (*enable_irq_window)(struct kvm_vcpu *vcpu);
1609 void (*update_cr8_intercept)(struct kvm_vcpu *vcpu, int tpr, int irr);1643 void (*update_cr8_intercept)(struct kvm_vcpu *vcpu, int tpr, int irr);
1610 bool (*check_apicv_inhibit_reasons)(enum kvm_apicv_inhibit reason);1644 bool (*check_apicv_inhibit_reasons)(enum kvm_apicv_inhibit reason);
1645 const unsigned long required_apicv_inhibits;
1646 bool allow_apicv_in_x2apic_without_x2apic_virtualization;
1611 void (*refresh_apicv_exec_ctrl)(struct kvm_vcpu *vcpu);1647 void (*refresh_apicv_exec_ctrl)(struct kvm_vcpu *vcpu);
1612 void (*hwapic_irr_update)(struct kvm_vcpu *vcpu, int max_irr);1648 void (*hwapic_irr_update)(struct kvm_vcpu *vcpu, int max_irr);
1613 void (*hwapic_isr_update)(int isr);1649 void (*hwapic_isr_update)(int isr);
@@ -1731,9 +1767,6 @@ struct kvm_x86_nested_ops { · 117 unmodified lines
1731};1767};
1732 1768
1733struct kvm_x86_init_ops {1769struct kvm_x86_init_ops {
1734 int (*cpu_has_kvm_support)(void);
1735 int (*disabled_by_bios)(void);
1736 int (*check_processor_compatibility)(void);
1737 int (*hardware_setup)(void);1770 int (*hardware_setup)(void);
1738 unsigned int (*handle_intel_pt_intr)(void);1771 unsigned int (*handle_intel_pt_intr)(void);
1739 1772
@@ -1760,6 +1793,9 @@ extern struct kvm_x86_ops kvm_x86_ops; · 20 unmodified lines
1760#define KVM_X86_OP_OPTIONAL_RET0 KVM_X86_OP1793#define KVM_X86_OP_OPTIONAL_RET0 KVM_X86_OP
1761#include <asm/kvm-x86-ops.h>1794#include <asm/kvm-x86-ops.h>
1762 1795
1796int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops);
1797void kvm_x86_vendor_exit(void);
1798
1763#define __KVM_HAVE_ARCH_VM_ALLOC1799#define __KVM_HAVE_ARCH_VM_ALLOC
1764static inline struct kvm *kvm_arch_alloc_vm(void)1800static inline struct kvm *kvm_arch_alloc_vm(void)
1765{1801{
@@ -1982,7 +2018,7 @@ gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva, · 216 unmodified lines
1982 2018
1983bool kvm_apicv_activated(struct kvm *kvm);2019bool kvm_apicv_activated(struct kvm *kvm);
1984bool kvm_vcpu_apicv_activated(struct kvm_vcpu *vcpu);2020bool kvm_vcpu_apicv_activated(struct kvm_vcpu *vcpu);
1985void kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu);2021void __kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu);
1986void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm,2022void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm,
1987 enum kvm_apicv_inhibit reason, bool set);2023 enum kvm_apicv_inhibit reason, bool set);
1988void kvm_set_or_clear_apicv_inhibit(struct kvm *kvm,2024void kvm_set_or_clear_apicv_inhibit(struct kvm *kvm,
@@ -2054,14 +2090,11 @@ enum { · 65 unmodified lines
2054 TASK_SWITCH_GATE = 3,2090 TASK_SWITCH_GATE = 3,
2055};2091};
2056 2092
2057#define HF_GIF_MASK (1 << 0)2093#define HF_GUEST_MASK (1 << 0) /* VCPU is in guest-mode */
2058#define HF_NMI_MASK (1 << 3)
2059#define HF_IRET_MASK (1 << 4)
2060#define HF_GUEST_MASK (1 << 5) /* VCPU is in guest-mode */
2061 2094
2062#ifdef CONFIG_KVM_SMM2095#ifdef CONFIG_KVM_SMM
2063#define HF_SMM_MASK (1 << 6)2096#define HF_SMM_MASK (1 << 1)
2064#define HF_SMM_INSIDE_NMI_MASK (1 << 7)2097#define HF_SMM_INSIDE_NMI_MASK (1 << 2)
2065 2098
2066# define __KVM_VCPU_MULTIPLE_ADDRESS_SPACE2099# define __KVM_VCPU_MULTIPLE_ADDRESS_SPACE
2067# define KVM_ADDRESS_SPACE_NUM 22100# define KVM_ADDRESS_SPACE_NUM 2
arch/x86/include/asm/reboot.h+2−0
@@ -25,6 +25,8 @@ void __noreturn machine_real_restart(unsigned int type); · 24 unmodified lines
25#define MRR_BIOS 025#define MRR_BIOS 0
26#define MRR_APM 126#define MRR_APM 1
27 27
28void cpu_emergency_disable_virtualization(void);
29
28typedef void (*nmi_shootdown_cb)(int, struct pt_regs*);30typedef void (*nmi_shootdown_cb)(int, struct pt_regs*);
29void nmi_panic_self_stop(struct pt_regs *regs);31void nmi_panic_self_stop(struct pt_regs *regs);
30void nmi_shootdown_cpus(nmi_shootdown_cb callback);32void nmi_shootdown_cpus(nmi_shootdown_cb callback);
arch/x86/include/asm/virtext.h+15−1
@@ -126,7 +126,21 @@ static inline void cpu_svm_disable(void) · 125 unmodified lines
126 126
127 wrmsrl(MSR_VM_HSAVE_PA, 0);127 wrmsrl(MSR_VM_HSAVE_PA, 0);
128 rdmsrl(MSR_EFER, efer);128 rdmsrl(MSR_EFER, efer);
129 wrmsrl(MSR_EFER, efer & ~EFER_SVME);129 if (efer & EFER_SVME) {
130 /*
131 * Force GIF=1 prior to disabling SVM to ensure INIT and NMI
132 * aren't blocked, e.g. if a fatal error occurred between CLGI
133 * and STGI. Note, STGI may #UD if SVM is disabled from NMI
134 * context between reading EFER and executing STGI. In that
135 * case, GIF must already be set, otherwise the NMI would have
136 * been blocked, so just eat the fault.
137 */
138 asm_volatile_goto("1: stgi\n\t"
139 _ASM_EXTABLE(1b, %l[fault])
140 ::: "memory" : fault);
141fault:
142 wrmsrl(MSR_EFER, efer & ~EFER_SVME);
143 }
130}144}
131 145
132/** Makes sure SVM is disabled, if it is supported on the CPU146/** Makes sure SVM is disabled, if it is supported on the CPU
arch/x86/include/asm/xen/hypervisor.h+3−1
@@ -38,9 +38,11 @@ extern struct start_info *xen_start_info; · 37 unmodified lines
38 38
39#include <asm/processor.h>39#include <asm/processor.h>
40 40
41#define XEN_SIGNATURE "XenVMMXenVMM"
42
41static inline uint32_t xen_cpuid_base(void)43static inline uint32_t xen_cpuid_base(void)
42{44{
43 return hypervisor_cpuid_base("XenVMMXenVMM", 2);45 return hypervisor_cpuid_base(XEN_SIGNATURE, 2);
44}46}
45 47
46struct pci_dev;48struct pci_dev;
arch/x86/include/uapi/asm/kvm.h+32−2
@@ -9,6 +9,7 @@ · 8 unmodified lines
9 9
10#include <linux/types.h>10#include <linux/types.h>
11#include <linux/ioctl.h>11#include <linux/ioctl.h>
12#include <linux/stddef.h>
12 13
13#define KVM_PIO_PAGE_OFFSET 114#define KVM_PIO_PAGE_OFFSET 1
14#define KVM_COALESCED_MMIO_PAGE_OFFSET 215#define KVM_COALESCED_MMIO_PAGE_OFFSET 2
@@ -507,8 +508,8 @@ struct kvm_nested_state { · 492 unmodified lines
507 * KVM_{GET,PUT}_NESTED_STATE ioctl values.508 * KVM_{GET,PUT}_NESTED_STATE ioctl values.
508 */509 */
509 union {510 union {
510 struct kvm_vmx_nested_state_data vmx[0];511 __DECLARE_FLEX_ARRAY(struct kvm_vmx_nested_state_data, vmx);
511 struct kvm_svm_nested_state_data svm[0];512 __DECLARE_FLEX_ARRAY(struct kvm_svm_nested_state_data, svm);
512 } data;513 } data;
513};514};
514 515
@@ -525,6 +526,35 @@ struct kvm_pmu_event_filter { · 10 unmodified lines
525#define KVM_PMU_EVENT_ALLOW 0526#define KVM_PMU_EVENT_ALLOW 0
526#define KVM_PMU_EVENT_DENY 1527#define KVM_PMU_EVENT_DENY 1
527 528
529#define KVM_PMU_EVENT_FLAG_MASKED_EVENTS BIT(0)
530#define KVM_PMU_EVENT_FLAGS_VALID_MASK (KVM_PMU_EVENT_FLAG_MASKED_EVENTS)
531
532/*
533 * Masked event layout.
534 * Bits Description
535 * ---- -----------
536 * 7:0 event select (low bits)
537 * 15:8 umask match
538 * 31:16 unused
539 * 35:32 event select (high bits)
540 * 36:54 unused
541 * 55 exclude bit
542 * 63:56 umask mask
543 */
544
545#define KVM_PMU_ENCODE_MASKED_ENTRY(event_select, mask, match, exclude) \
546 (((event_select) & 0xFFULL) | (((event_select) & 0XF00ULL) << 24) | \
547 (((mask) & 0xFFULL) << 56) | \
548 (((match) & 0xFFULL) << 8) | \
549 ((__u64)(!!(exclude)) << 55))
550
551#define KVM_PMU_MASKED_ENTRY_EVENT_SELECT \
552 (GENMASK_ULL(7, 0) | GENMASK_ULL(35, 32))
553#define KVM_PMU_MASKED_ENTRY_UMASK_MASK (GENMASK_ULL(63, 56))
554#define KVM_PMU_MASKED_ENTRY_UMASK_MATCH (GENMASK_ULL(15, 8))
555#define KVM_PMU_MASKED_ENTRY_EXCLUDE (BIT_ULL(55))
556#define KVM_PMU_MASKED_ENTRY_UMASK_MASK_SHIFT (56)
557
528/* for KVM_{GET,SET,HAS}_DEVICE_ATTR */558/* for KVM_{GET,SET,HAS}_DEVICE_ATTR */
529#define KVM_VCPU_TSC_CTRL 0 /* control group for the timestamp counter (TSC) */559#define KVM_VCPU_TSC_CTRL 0 /* control group for the timestamp counter (TSC) */
530#define KVM_VCPU_TSC_OFFSET 0 /* attribute for the TSC offset */560#define KVM_VCPU_TSC_OFFSET 0 /* attribute for the TSC offset */
arch/x86/kernel/cpu/mshyperv.c+1−1
@@ -460,7 +460,7 @@ static void __init ms_hyperv_init_platform(void) · 459 unmodified lines
460 * setting of this MSR bit should happen before init_intel()460 * setting of this MSR bit should happen before init_intel()
461 * is called.461 * is called.
462 */462 */
463 wrmsrl(HV_X64_MSR_TSC_INVARIANT_CONTROL, 0x1);463 wrmsrl(HV_X64_MSR_TSC_INVARIANT_CONTROL, HV_EXPOSE_INVARIANT_TSC);
464 setup_force_cpu_cap(X86_FEATURE_TSC_RELIABLE);464 setup_force_cpu_cap(X86_FEATURE_TSC_RELIABLE);
465 }465 }
466 466
arch/x86/kernel/crash.c+1−16
@@ -37,7 +37,6 @@ · 36 unmodified lines
37#include <linux/kdebug.h>37#include <linux/kdebug.h>
38#include <asm/cpu.h>38#include <asm/cpu.h>
39#include <asm/reboot.h>39#include <asm/reboot.h>
40#include <asm/virtext.h>
41#include <asm/intel_pt.h>40#include <asm/intel_pt.h>
42#include <asm/crash.h>41#include <asm/crash.h>
43#include <asm/cmdline.h>42#include <asm/cmdline.h>
@@ -81,15 +80,6 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs) · 37 unmodified lines
81 */80 */
82 cpu_crash_vmclear_loaded_vmcss();81 cpu_crash_vmclear_loaded_vmcss();
83 82
84 /* Disable VMX or SVM if needed.
85 *
86 * We need to disable virtualization on all CPUs.
87 * Having VMX or SVM enabled on any CPU may break rebooting
88 * after the kdump kernel has finished its task.
89 */
90 cpu_emergency_vmxoff();
91 cpu_emergency_svm_disable();
92
93 /*83 /*
94 * Disable Intel PT to stop its logging84 * Disable Intel PT to stop its logging
95 */85 */
@@ -148,12 +138,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs) · 52 unmodified lines
148 */138 */
149 cpu_crash_vmclear_loaded_vmcss();139 cpu_crash_vmclear_loaded_vmcss();
150 140
151 /* Booting kdump kernel with VMX or SVM enabled won't work,141 cpu_emergency_disable_virtualization();
152 * because (among other limitations) we can't disable paging
153 * with the virt flags.
154 */
155 cpu_emergency_vmxoff();
156 cpu_emergency_svm_disable();
157 142
158 /*143 /*
159 * Disable Intel PT to stop its logging144 * Disable Intel PT to stop its logging
arch/x86/kernel/nmi.c+4−4
@@ -560,14 +560,14 @@ nmi_restart: · 559 unmodified lines
560 }560 }
561}561}
562 562
563#if defined(CONFIG_X86_64) && IS_ENABLED(CONFIG_KVM_INTEL)563#if IS_ENABLED(CONFIG_KVM_INTEL)
564DEFINE_IDTENTRY_RAW(exc_nmi_noist)564DEFINE_IDTENTRY_RAW(exc_nmi_kvm_vmx)
565{565{
566 exc_nmi(regs);566 exc_nmi(regs);
567}567}
568#endif
569#if IS_MODULE(CONFIG_KVM_INTEL)568#if IS_MODULE(CONFIG_KVM_INTEL)
570EXPORT_SYMBOL_GPL(asm_exc_nmi_noist);569EXPORT_SYMBOL_GPL(asm_exc_nmi_kvm_vmx);
570#endif
571#endif571#endif
572 572
573#ifdef CONFIG_NMI_CHECK_CPU573#ifdef CONFIG_NMI_CHECK_CPU
arch/x86/kernel/reboot.c+64−24
@@ -528,33 +528,29 @@ static inline void kb_wait(void) · 527 unmodified lines
528 }528 }
529}529}
530 530
531static void vmxoff_nmi(int cpu, struct pt_regs *regs)531static inline void nmi_shootdown_cpus_on_restart(void);
532{
533 cpu_emergency_vmxoff();
534}
535 532
536/* Use NMIs as IPIs to tell all CPUs to disable virtualization */533static void emergency_reboot_disable_virtualization(void)
537static void emergency_vmx_disable_all(void)
538{534{
539 /* Just make sure we won't change CPUs while doing this */535 /* Just make sure we won't change CPUs while doing this */
540 local_irq_disable();536 local_irq_disable();
541 537
542 /*538 /*
543 * Disable VMX on all CPUs before rebooting, otherwise we risk hanging539 * Disable virtualization on all CPUs before rebooting to avoid hanging
544 * the machine, because the CPU blocks INIT when it's in VMX root.540 * the system, as VMX and SVM block INIT when running in the host.
545 *541 *
546 * We can't take any locks and we may be on an inconsistent state, so542 * We can't take any locks and we may be on an inconsistent state, so
547 * use NMIs as IPIs to tell the other CPUs to exit VMX root and halt.543 * use NMIs as IPIs to tell the other CPUs to disable VMX/SVM and halt.
548 *544 *
549 * Do the NMI shootdown even if VMX if off on _this_ CPU, as that545 * Do the NMI shootdown even if virtualization is off on _this_ CPU, as
550 * doesn't prevent a different CPU from being in VMX root operation.546 * other CPUs may have virtualization enabled.
551 */547 */
552 if (cpu_has_vmx()) {548 if (cpu_has_vmx() || cpu_has_svm(NULL)) {
553 /* Safely force _this_ CPU out of VMX root operation. */549 /* Safely force _this_ CPU out of VMX/SVM operation. */
554 __cpu_emergency_vmxoff();550 cpu_emergency_disable_virtualization();
555 551
556 /* Halt and exit VMX root operation on the other CPUs. */552 /* Disable VMX/SVM and halt on other CPUs. */
557 nmi_shootdown_cpus(vmxoff_nmi);553 nmi_shootdown_cpus_on_restart();
558 }554 }
559}555}
560 556
@@ -590,7 +586,7 @@ static void native_machine_emergency_restart(void) · 29 unmodified lines
590 unsigned short mode;586 unsigned short mode;
591 587
592 if (reboot_emergency)588 if (reboot_emergency)
593 emergency_vmx_disable_all();589 emergency_reboot_disable_virtualization();
594 590
595 tboot_shutdown(TB_SHUTDOWN_REBOOT);591 tboot_shutdown(TB_SHUTDOWN_REBOOT);
596 592
@@ -795,6 +791,17 @@ void machine_crash_shutdown(struct pt_regs *regs) · 198 unmodified lines
795/* This is the CPU performing the emergency shutdown work. */791/* This is the CPU performing the emergency shutdown work. */
796int crashing_cpu = -1;792int crashing_cpu = -1;
797 793
794/*
795 * Disable virtualization, i.e. VMX or SVM, to ensure INIT is recognized during
796 * reboot. VMX blocks INIT if the CPU is post-VMXON, and SVM blocks INIT if
797 * GIF=0, i.e. if the crash occurred between CLGI and STGI.
798 */
799void cpu_emergency_disable_virtualization(void)
800{
801 cpu_emergency_vmxoff();
802 cpu_emergency_svm_disable();
803}
804
798#if defined(CONFIG_SMP)805#if defined(CONFIG_SMP)
799 806
800static nmi_shootdown_cb shootdown_callback;807static nmi_shootdown_cb shootdown_callback;
@@ -817,7 +824,14 @@ static int crash_nmi_callback(unsigned int val, struct pt_regs *regs) · 16 unmodified lines
817 return NMI_HANDLED;824 return NMI_HANDLED;
818 local_irq_disable();825 local_irq_disable();
819 826
820 shootdown_callback(cpu, regs);827 if (shootdown_callback)
828 shootdown_callback(cpu, regs);
829
830 /*
831 * Prepare the CPU for reboot _after_ invoking the callback so that the
832 * callback can safely use virtualization instructions, e.g. VMCLEAR.
833 */
834 cpu_emergency_disable_virtualization();
821 835
822 atomic_dec(&waiting_for_crash_ipi);836 atomic_dec(&waiting_for_crash_ipi);
823 /* Assume hlt works */837 /* Assume hlt works */
@@ -828,18 +842,32 @@ static int crash_nmi_callback(unsigned int val, struct pt_regs *regs) · 4 unmodified lines
828 return NMI_HANDLED;842 return NMI_HANDLED;
829}843}
830 844
831/*845/**
832 * Halt all other CPUs, calling the specified function on each of them846 * nmi_shootdown_cpus - Stop other CPUs via NMI
847 * @callback: Optional callback to be invoked from the NMI handler
848 *
849 * The NMI handler on the remote CPUs invokes @callback, if not
850 * NULL, first and then disables virtualization to ensure that
851 * INIT is recognized during reboot.
833 *852 *
834 * This function can be used to halt all other CPUs on crash853 * nmi_shootdown_cpus() can only be invoked once. After the first
835 * or emergency reboot time. The function passed as parameter854 * invocation all other CPUs are stuck in crash_nmi_callback() and
836 * will be called inside a NMI handler on all CPUs.855 * cannot respond to a second NMI.
837 */856 */
838void nmi_shootdown_cpus(nmi_shootdown_cb callback)857void nmi_shootdown_cpus(nmi_shootdown_cb callback)
839{858{
840 unsigned long msecs;859 unsigned long msecs;
860
841 local_irq_disable();861 local_irq_disable();
842 862
863 /*
864 * Avoid certain doom if a shootdown already occurred; re-registering
865 * the NMI handler will cause list corruption, modifying the callback
866 * will do who knows what, etc...
867 */
868 if (WARN_ON_ONCE(crash_ipi_issued))
869 return;
870
843 /* Make a note of crashing cpu. Will be used in NMI callback. */871 /* Make a note of crashing cpu. Will be used in NMI callback. */
844 crashing_cpu = safe_smp_processor_id();872 crashing_cpu = safe_smp_processor_id();
845 873
@@ -867,7 +895,17 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback) · 21 unmodified lines
867 msecs--;895 msecs--;
868 }896 }
869 897
870 /* Leave the nmi callback set */898 /*
899 * Leave the nmi callback set, shootdown is a one-time thing. Clearing
900 * the callback could result in a NULL pointer dereference if a CPU
901 * (finally) responds after the timeout expires.
902 */
903}
904
905static inline void nmi_shootdown_cpus_on_restart(void)
906{
907 if (!crash_ipi_issued)
908 nmi_shootdown_cpus(NULL);
871}909}
872 910
873/*911/*
@@ -897,6 +935,8 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback) · 23 unmodified lines
897 /* No other CPUs to shoot down */935 /* No other CPUs to shoot down */
898}936}
899 937
938static inline void nmi_shootdown_cpus_on_restart(void) { }
939
900void run_crash_ipi_callback(struct pt_regs *regs)940void run_crash_ipi_callback(struct pt_regs *regs)
901{941{
902}942}
arch/x86/kernel/smp.c+3−3
@@ -32,7 +32,7 @@ · 31 unmodified lines
32#include <asm/mce.h>32#include <asm/mce.h>
33#include <asm/trace/irq_vectors.h>33#include <asm/trace/irq_vectors.h>
34#include <asm/kexec.h>34#include <asm/kexec.h>
35#include <asm/virtext.h>35#include <asm/reboot.h>
36 36
37/*37/*
38 * Some notes on x86 processor bugs affecting SMP operation:38 * Some notes on x86 processor bugs affecting SMP operation:
@@ -122,7 +122,7 @@ static int smp_stop_nmi_callback(unsigned int val, struct pt_regs *regs) · 83 unmodified lines
122 if (raw_smp_processor_id() == atomic_read(&stopping_cpu))122 if (raw_smp_processor_id() == atomic_read(&stopping_cpu))
123 return NMI_HANDLED;123 return NMI_HANDLED;
124 124
125 cpu_emergency_vmxoff();125 cpu_emergency_disable_virtualization();
126 stop_this_cpu(NULL);126 stop_this_cpu(NULL);
127 127
128 return NMI_HANDLED;128 return NMI_HANDLED;
@@ -134,7 +134,7 @@ static int smp_stop_nmi_callback(unsigned int val, struct pt_regs *regs) · 5 unmodified lines
134DEFINE_IDTENTRY_SYSVEC(sysvec_reboot)134DEFINE_IDTENTRY_SYSVEC(sysvec_reboot)
135{135{
136 ack_APIC_irq();136 ack_APIC_irq();
137 cpu_emergency_vmxoff();137 cpu_emergency_disable_virtualization();
138 stop_this_cpu(NULL);138 stop_this_cpu(NULL);
139}139}
140 140
arch/x86/kvm/Kconfig+1−0
@@ -49,6 +49,7 @@ config KVM · 48 unmodified lines
49 select SRCU49 select SRCU
50 select INTERVAL_TREE50 select INTERVAL_TREE
51 select HAVE_KVM_PM_NOTIFIER if PM51 select HAVE_KVM_PM_NOTIFIER if PM
52 select KVM_GENERIC_HARDWARE_ENABLING
52 help53 help
53 Support hosting fully virtualized guest machines using hardware54 Support hosting fully virtualized guest machines using hardware
54 virtualization extensions. You will need a fairly recent55 virtualization extensions. You will need a fairly recent
arch/x86/kvm/cpuid.c+28−15
@@ -8,6 +8,7 @@ · 7 unmodified lines
8 * Copyright 2011 Red Hat, Inc. and/or its affiliates.8 * Copyright 2011 Red Hat, Inc. and/or its affiliates.
9 * Copyright IBM Corporation, 20089 * Copyright IBM Corporation, 2008
10 */10 */
11#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
11 12
12#include <linux/kvm_host.h>13#include <linux/kvm_host.h>
13#include <linux/export.h>14#include <linux/export.h>
@@ -25,6 +26,7 @@ · 11 unmodified lines
25#include "mmu.h"26#include "mmu.h"
26#include "trace.h"27#include "trace.h"
27#include "pmu.h"28#include "pmu.h"
29#include "xen.h"
28 30
29/*31/*
30 * Unlike "struct cpuinfo_x86.x86_capability", kvm_cpu_caps doesn't need to be32 * Unlike "struct cpuinfo_x86.x86_capability", kvm_cpu_caps doesn't need to be
@@ -180,15 +182,15 @@ static int kvm_cpuid_check_equal(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 · 149 unmodified lines
180 return 0;182 return 0;
181}183}
182 184
183static void kvm_update_kvm_cpuid_base(struct kvm_vcpu *vcpu)185static struct kvm_hypervisor_cpuid kvm_get_hypervisor_cpuid(struct kvm_vcpu *vcpu,
186 const char *sig)
184{187{
185 u32 function;188 struct kvm_hypervisor_cpuid cpuid = {};
186 struct kvm_cpuid_entry2 *entry;189 struct kvm_cpuid_entry2 *entry;
190 u32 base;
187 191
188 vcpu->arch.kvm_cpuid_base = 0;192 for_each_possible_hypervisor_cpuid_base(base) {
189 193 entry = kvm_find_cpuid_entry(vcpu, base);
190 for_each_possible_hypervisor_cpuid_base(function) {
191 entry = kvm_find_cpuid_entry(vcpu, function);
192 194
193 if (entry) {195 if (entry) {
194 u32 signature[3];196 u32 signature[3];
@@ -197,19 +199,21 @@ static void kvm_update_kvm_cpuid_base(struct kvm_vcpu *vcpu) · 2 unmodified lines
197 signature[1] = entry->ecx;199 signature[1] = entry->ecx;
198 signature[2] = entry->edx;200 signature[2] = entry->edx;
199 201
200 BUILD_BUG_ON(sizeof(signature) > sizeof(KVM_SIGNATURE));202 if (!memcmp(signature, sig, sizeof(signature))) {
201 if (!memcmp(signature, KVM_SIGNATURE, sizeof(signature))) {203 cpuid.base = base;
202 vcpu->arch.kvm_cpuid_base = function;204 cpuid.limit = entry->eax;
203 break;205 break;
204 }206 }
205 }207 }
206 }208 }
209
210 return cpuid;
207}211}
208 212
209static struct kvm_cpuid_entry2 *__kvm_find_kvm_cpuid_features(struct kvm_vcpu *vcpu,213static struct kvm_cpuid_entry2 *__kvm_find_kvm_cpuid_features(struct kvm_vcpu *vcpu,
210 struct kvm_cpuid_entry2 *entries, int nent)214 struct kvm_cpuid_entry2 *entries, int nent)
211{215{
212 u32 base = vcpu->arch.kvm_cpuid_base;216 u32 base = vcpu->arch.kvm_cpuid.base;
213 217
214 if (!base)218 if (!base)
215 return NULL;219 return NULL;
@@ -439,7 +443,8 @@ static int kvm_set_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 *e2, · 223 unmodified lines
439 vcpu->arch.cpuid_entries = e2;443 vcpu->arch.cpuid_entries = e2;
440 vcpu->arch.cpuid_nent = nent;444 vcpu->arch.cpuid_nent = nent;
441 445
442 kvm_update_kvm_cpuid_base(vcpu);446 vcpu->arch.kvm_cpuid = kvm_get_hypervisor_cpuid(vcpu, KVM_SIGNATURE);
447 vcpu->arch.xen.cpuid = kvm_get_hypervisor_cpuid(vcpu, XEN_SIGNATURE);
443 kvm_vcpu_after_set_cpuid(vcpu);448 kvm_vcpu_after_set_cpuid(vcpu);
444 449
445 return 0;450 return 0;
@@ -663,8 +668,9 @@ void kvm_set_cpu_caps(void) · 217 unmodified lines
663 kvm_cpu_cap_set(X86_FEATURE_SPEC_CTRL_SSBD);668 kvm_cpu_cap_set(X86_FEATURE_SPEC_CTRL_SSBD);
664 669
665 kvm_cpu_cap_mask(CPUID_7_1_EAX,670 kvm_cpu_cap_mask(CPUID_7_1_EAX,
666 F(AVX_VNNI) | F(AVX512_BF16) | F(CMPCCXADD) | F(AMX_FP16) |671 F(AVX_VNNI) | F(AVX512_BF16) | F(CMPCCXADD) |
667 F(AVX_IFMA)672 F(FZRM) | F(FSRS) | F(FSRC) |
673 F(AMX_FP16) | F(AVX_IFMA)
668 );674 );
669 675
670 kvm_cpu_cap_init_kvm_defined(CPUID_7_1_EDX,676 kvm_cpu_cap_init_kvm_defined(CPUID_7_1_EDX,
@@ -701,6 +707,10 @@ void kvm_set_cpu_caps(void) · 30 unmodified lines
701 if (!tdp_enabled && IS_ENABLED(CONFIG_X86_64))707 if (!tdp_enabled && IS_ENABLED(CONFIG_X86_64))
702 kvm_cpu_cap_set(X86_FEATURE_GBPAGES);708 kvm_cpu_cap_set(X86_FEATURE_GBPAGES);
703 709
710 kvm_cpu_cap_init_kvm_defined(CPUID_8000_0007_EDX,
711 SF(CONSTANT_TSC)
712 );
713
704 kvm_cpu_cap_mask(CPUID_8000_0008_EBX,714 kvm_cpu_cap_mask(CPUID_8000_0008_EBX,
705 F(CLZERO) | F(XSAVEERPTR) |715 F(CLZERO) | F(XSAVEERPTR) |
706 F(WBNOINVD) | F(AMD_IBPB) | F(AMD_IBRS) | F(AMD_SSBD) | F(VIRT_SSBD) |716 F(WBNOINVD) | F(AMD_IBPB) | F(AMD_IBRS) | F(AMD_SSBD) | F(VIRT_SSBD) |
@@ -1169,8 +1179,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · 462 unmodified lines
1169 entry->edx &= ~GENMASK(17, 16);1179 entry->edx &= ~GENMASK(17, 16);
1170 break;1180 break;
1171 case 0x80000007: /* Advanced power management */1181 case 0x80000007: /* Advanced power management */
1172 /* invariant TSC is CPUID.80000007H:EDX[8] */1182 cpuid_entry_override(entry, CPUID_8000_0007_EDX);
1173 entry->edx &= (1 << 8);1183
1174 /* mask against host */1184 /* mask against host */
1175 entry->edx &= boot_cpu_data.x86_power;1185 entry->edx &= boot_cpu_data.x86_power;
1176 entry->eax = entry->ebx = entry->ecx = 0;1186 entry->eax = entry->ebx = entry->ecx = 0;
@@ -1485,6 +1495,9 @@ bool kvm_cpuid(struct kvm_vcpu *vcpu, u32 *eax, u32 *ebx, · 308 unmodified lines
1485 if (!__kvm_get_msr(vcpu, MSR_IA32_TSX_CTRL, &data, true) &&1495 if (!__kvm_get_msr(vcpu, MSR_IA32_TSX_CTRL, &data, true) &&
1486 (data & TSX_CTRL_CPUID_CLEAR))1496 (data & TSX_CTRL_CPUID_CLEAR))
1487 *ebx &= ~(F(RTM) | F(HLE));1497 *ebx &= ~(F(RTM) | F(HLE));
1498 } else if (function == 0x80000007) {
1499 if (kvm_hv_invtsc_suppressed(vcpu))
1500 *edx &= ~SF(CONSTANT_TSC);
1488 }1501 }
1489 } else {1502 } else {
1490 *eax = *ebx = *ecx = *edx = 0;1503 *eax = *ebx = *ecx = *edx = 0;
arch/x86/kvm/debugfs.c+2−0
@@ -4,6 +4,8 @@ · 3 unmodified lines
4 *4 *
5 * Copyright 2016 Red Hat, Inc. and/or its affiliates.5 * Copyright 2016 Red Hat, Inc. and/or its affiliates.
6 */6 */
7#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
8
7#include <linux/kvm_host.h>9#include <linux/kvm_host.h>
8#include <linux/debugfs.h>10#include <linux/debugfs.h>
9#include "lapic.h"11#include "lapic.h"
arch/x86/kvm/emulate.c+9−9
@@ -17,6 +17,7 @@ · 16 unmodified lines
17 *17 *
18 * From: xen-unstable 10676:af9809f51f81a3c43f276f00c81a52ef558afda418 * From: xen-unstable 10676:af9809f51f81a3c43f276f00c81a52ef558afda4
19 */19 */
20#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
20 21
21#include <linux/kvm_host.h>22#include <linux/kvm_host.h>
22#include "kvm_cache_regs.h"23#include "kvm_cache_regs.h"
@@ -1633,7 +1634,7 @@ static int __load_segment_descriptor(struct x86_emulate_ctxt *ctxt, · 1610 unmodified lines
1633 case VCPU_SREG_SS:1634 case VCPU_SREG_SS:
1634 /*1635 /*
1635 * segment is not a writable data segment or segment1636 * segment is not a writable data segment or segment
1636 * selector's RPL != CPL or segment selector's RPL != CPL1637 * selector's RPL != CPL or DPL != CPL
1637 */1638 */
1638 if (rpl != cpl || (seg_desc.type & 0xa) != 0x2 || dpl != cpl)1639 if (rpl != cpl || (seg_desc.type & 0xa) != 0x2 || dpl != cpl)
1639 goto exception;1640 goto exception;
@@ -1695,11 +1696,11 @@ static int __load_segment_descriptor(struct x86_emulate_ctxt *ctxt, · 55 unmodified lines
1695 /*1696 /*
1696 * segment is not a data or readable code segment or1697 * segment is not a data or readable code segment or
1697 * ((segment is a data or nonconforming code segment)1698 * ((segment is a data or nonconforming code segment)
1698 * and (both RPL and CPL > DPL))1699 * and ((RPL > DPL) or (CPL > DPL)))
1699 */1700 */
1700 if ((seg_desc.type & 0xa) == 0x8 ||1701 if ((seg_desc.type & 0xa) == 0x8 ||
1701 (((seg_desc.type & 0xc) != 0xc) &&1702 (((seg_desc.type & 0xc) != 0xc) &&
1702 (rpl > dpl && cpl > dpl)))1703 (rpl > dpl || cpl > dpl)))
1703 goto exception;1704 goto exception;
1704 break;1705 break;
1705 }1706 }
@@ -2309,7 +2310,7 @@ static int em_lseg(struct x86_emulate_ctxt *ctxt) · 603 unmodified lines
2309 2310
2310static int em_rsm(struct x86_emulate_ctxt *ctxt)2311static int em_rsm(struct x86_emulate_ctxt *ctxt)
2311{2312{
2312 if ((ctxt->ops->get_hflags(ctxt) & X86EMUL_SMM_MASK) == 0)2313 if (!ctxt->ops->is_smm(ctxt))
2313 return emulate_ud(ctxt);2314 return emulate_ud(ctxt);
2314 2315
2315 if (ctxt->ops->leave_smm(ctxt))2316 if (ctxt->ops->leave_smm(ctxt))
@@ -5132,7 +5133,7 @@ int x86_emulate_insn(struct x86_emulate_ctxt *ctxt) · 2816 unmodified lines
5132 const struct x86_emulate_ops *ops = ctxt->ops;5133 const struct x86_emulate_ops *ops = ctxt->ops;
5133 int rc = X86EMUL_CONTINUE;5134 int rc = X86EMUL_CONTINUE;
5134 int saved_dst_type = ctxt->dst.type;5135 int saved_dst_type = ctxt->dst.type;
5135 unsigned emul_flags;5136 bool is_guest_mode = ctxt->ops->is_guest_mode(ctxt);
5136 5137
5137 ctxt->mem_read.pos = 0;5138 ctxt->mem_read.pos = 0;
5138 5139
@@ -5147,7 +5148,6 @@ int x86_emulate_insn(struct x86_emulate_ctxt *ctxt) · 8 unmodified lines
5147 goto done;5148 goto done;
5148 }5149 }
5149 5150
5150 emul_flags = ctxt->ops->get_hflags(ctxt);
5151 if (unlikely(ctxt->d &5151 if (unlikely(ctxt->d &
5152 (No64|Undefined|Sse|Mmx|Intercept|CheckPerm|Priv|Prot|String))) {5152 (No64|Undefined|Sse|Mmx|Intercept|CheckPerm|Priv|Prot|String))) {
5153 if ((ctxt->mode == X86EMUL_MODE_PROT64 && (ctxt->d & No64)) ||5153 if ((ctxt->mode == X86EMUL_MODE_PROT64 && (ctxt->d & No64)) ||
@@ -5181,7 +5181,7 @@ int x86_emulate_insn(struct x86_emulate_ctxt *ctxt) · 27 unmodified lines
5181 fetch_possible_mmx_operand(&ctxt->dst);5181 fetch_possible_mmx_operand(&ctxt->dst);
5182 }5182 }
5183 5183
5184 if (unlikely(emul_flags & X86EMUL_GUEST_MASK) && ctxt->intercept) {5184 if (unlikely(is_guest_mode) && ctxt->intercept) {
5185 rc = emulator_check_intercept(ctxt, ctxt->intercept,5185 rc = emulator_check_intercept(ctxt, ctxt->intercept,
5186 X86_ICPT_PRE_EXCEPT);5186 X86_ICPT_PRE_EXCEPT);
5187 if (rc != X86EMUL_CONTINUE)5187 if (rc != X86EMUL_CONTINUE)
@@ -5210,7 +5210,7 @@ int x86_emulate_insn(struct x86_emulate_ctxt *ctxt) · 22 unmodified lines
5210 goto done;5210 goto done;
5211 }5211 }
5212 5212
5213 if (unlikely(emul_flags & X86EMUL_GUEST_MASK) && (ctxt->d & Intercept)) {5213 if (unlikely(is_guest_mode) && (ctxt->d & Intercept)) {
5214 rc = emulator_check_intercept(ctxt, ctxt->intercept,5214 rc = emulator_check_intercept(ctxt, ctxt->intercept,
5215 X86_ICPT_POST_EXCEPT);5215 X86_ICPT_POST_EXCEPT);
5216 if (rc != X86EMUL_CONTINUE)5216 if (rc != X86EMUL_CONTINUE)
@@ -5264,7 +5264,7 @@ int x86_emulate_insn(struct x86_emulate_ctxt *ctxt) · 47 unmodified lines
5264 5264
5265special_insn:5265special_insn:
5266 5266
5267 if (unlikely(emul_flags & X86EMUL_GUEST_MASK) && (ctxt->d & Intercept)) {5267 if (unlikely(is_guest_mode) && (ctxt->d & Intercept)) {
5268 rc = emulator_check_intercept(ctxt, ctxt->intercept,5268 rc = emulator_check_intercept(ctxt, ctxt->intercept,
5269 X86_ICPT_POST_MEMACCESS);5269 X86_ICPT_POST_MEMACCESS);
5270 if (rc != X86EMUL_CONTINUE)5270 if (rc != X86EMUL_CONTINUE)
arch/x86/kvm/hyperv.c+63−22
@@ -17,6 +17,7 @@ · 16 unmodified lines
17 * Ben-Ami Yassour <benami@il.ibm.com>17 * Ben-Ami Yassour <benami@il.ibm.com>
18 * Andrey Smetanin <asmetanin@virtuozzo.com>18 * Andrey Smetanin <asmetanin@virtuozzo.com>
19 */19 */
20#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
20 21
21#include "x86.h"22#include "x86.h"
22#include "lapic.h"23#include "lapic.h"
@@ -43,6 +44,24 @@ · 20 unmodified lines
43 44
44#define KVM_HV_MAX_SPARSE_VCPU_SET_BITS DIV_ROUND_UP(KVM_MAX_VCPUS, HV_VCPUS_PER_SPARSE_BANK)45#define KVM_HV_MAX_SPARSE_VCPU_SET_BITS DIV_ROUND_UP(KVM_MAX_VCPUS, HV_VCPUS_PER_SPARSE_BANK)
45 46
47/*
48 * As per Hyper-V TLFS, extended hypercalls start from 0x8001
49 * (HvExtCallQueryCapabilities). Response of this hypercalls is a 64 bit value
50 * where each bit tells which extended hypercall is available besides
51 * HvExtCallQueryCapabilities.
52 *
53 * 0x8001 - First extended hypercall, HvExtCallQueryCapabilities, no bit
54 * assigned.
55 *
56 * 0x8002 - Bit 0
57 * 0x8003 - Bit 1
58 * ..
59 * 0x8041 - Bit 63
60 *
61 * Therefore, HV_EXT_CALL_MAX = 0x8001 + 64
62 */
63#define HV_EXT_CALL_MAX (HV_EXT_CALL_QUERY_CAPABILITIES + 64)
64
46static void stimer_mark_pending(struct kvm_vcpu_hv_stimer *stimer,65static void stimer_mark_pending(struct kvm_vcpu_hv_stimer *stimer,
47 bool vcpu_kick);66 bool vcpu_kick);
48 67
@@ -999,6 +1018,7 @@ static bool kvm_hv_msr_partition_wide(u32 msr) · 950 unmodified lines
999 case HV_X64_MSR_REENLIGHTENMENT_CONTROL:1018 case HV_X64_MSR_REENLIGHTENMENT_CONTROL:
1000 case HV_X64_MSR_TSC_EMULATION_CONTROL:1019 case HV_X64_MSR_TSC_EMULATION_CONTROL:
1001 case HV_X64_MSR_TSC_EMULATION_STATUS:1020 case HV_X64_MSR_TSC_EMULATION_STATUS:
1021 case HV_X64_MSR_TSC_INVARIANT_CONTROL:
1002 case HV_X64_MSR_SYNDBG_OPTIONS:1022 case HV_X64_MSR_SYNDBG_OPTIONS:
1003 case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER:1023 case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER:
1004 r = true;1024 r = true;
@@ -1283,6 +1303,9 @@ static bool hv_check_msr_access(struct kvm_vcpu_hv *hv_vcpu, u32 msr) · 278 unmodified lines
1283 case HV_X64_MSR_TSC_EMULATION_STATUS:1303 case HV_X64_MSR_TSC_EMULATION_STATUS:
1284 return hv_vcpu->cpuid_cache.features_eax &1304 return hv_vcpu->cpuid_cache.features_eax &
1285 HV_ACCESS_REENLIGHTENMENT;1305 HV_ACCESS_REENLIGHTENMENT;
1306 case HV_X64_MSR_TSC_INVARIANT_CONTROL:
1307 return hv_vcpu->cpuid_cache.features_eax &
1308 HV_ACCESS_TSC_INVARIANT;
1286 case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4:1309 case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4:
1287 case HV_X64_MSR_CRASH_CTL:1310 case HV_X64_MSR_CRASH_CTL:
1288 return hv_vcpu->cpuid_cache.features_edx &1311 return hv_vcpu->cpuid_cache.features_edx &
@@ -1410,12 +1433,22 @@ static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data, · 121 unmodified lines
1410 if (!host)1433 if (!host)
1411 return 1;1434 return 1;
1412 break;1435 break;
1436 case HV_X64_MSR_TSC_INVARIANT_CONTROL:
1437 /* Only bit 0 is supported */
1438 if (data & ~HV_EXPOSE_INVARIANT_TSC)
1439 return 1;
1440
1441 /* The feature can't be disabled from the guest */
1442 if (!host && hv->hv_invtsc_control && !data)
1443 return 1;
1444
1445 hv->hv_invtsc_control = data;
1446 break;
1413 case HV_X64_MSR_SYNDBG_OPTIONS:1447 case HV_X64_MSR_SYNDBG_OPTIONS:
1414 case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER:1448 case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER:
1415 return syndbg_set_msr(vcpu, msr, data, host);1449 return syndbg_set_msr(vcpu, msr, data, host);
1416 default:1450 default:
1417 vcpu_unimpl(vcpu, "Hyper-V unhandled wrmsr: 0x%x data 0x%llx\n",1451 kvm_pr_unimpl_wrmsr(vcpu, msr, data);
1418 msr, data);
1419 return 1;1452 return 1;
1420 }1453 }
1421 return 0;1454 return 0;
@@ -1536,8 +1569,7 @@ static int kvm_hv_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool host) · 114 unmodified lines
1536 return 1;1569 return 1;
1537 break;1570 break;
1538 default:1571 default:
1539 vcpu_unimpl(vcpu, "Hyper-V unhandled wrmsr: 0x%x data 0x%llx\n",1572 kvm_pr_unimpl_wrmsr(vcpu, msr, data);
1540 msr, data);
1541 return 1;1573 return 1;
1542 }1574 }
1543 1575
@@ -1585,11 +1617,14 @@ static int kvm_hv_get_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata, · 41 unmodified lines
1585 case HV_X64_MSR_TSC_EMULATION_STATUS:1617 case HV_X64_MSR_TSC_EMULATION_STATUS:
1586 data = hv->hv_tsc_emulation_status;1618 data = hv->hv_tsc_emulation_status;
1587 break;1619 break;
1620 case HV_X64_MSR_TSC_INVARIANT_CONTROL:
1621 data = hv->hv_invtsc_control;
1622 break;
1588 case HV_X64_MSR_SYNDBG_OPTIONS:1623 case HV_X64_MSR_SYNDBG_OPTIONS:
1589 case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER:1624 case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER:
1590 return syndbg_get_msr(vcpu, msr, pdata, host);1625 return syndbg_get_msr(vcpu, msr, pdata, host);
1591 default:1626 default:
1592 vcpu_unimpl(vcpu, "Hyper-V unhandled rdmsr: 0x%x\n", msr);1627 kvm_pr_unimpl_rdmsr(vcpu, msr);
1593 return 1;1628 return 1;
1594 }1629 }
1595 1630
@@ -1654,7 +1689,7 @@ static int kvm_hv_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata, · 58 unmodified lines
1654 data = APIC_BUS_FREQUENCY;1689 data = APIC_BUS_FREQUENCY;
1655 break;1690 break;
1656 default:1691 default:
1657 vcpu_unimpl(vcpu, "Hyper-V unhandled rdmsr: 0x%x\n", msr);1692 kvm_pr_unimpl_rdmsr(vcpu, msr);
1658 return 1;1693 return 1;
1659 }1694 }
1660 *pdata = data;1695 *pdata = data;
@@ -2420,6 +2455,9 @@ static bool hv_check_hypercall_access(struct kvm_vcpu_hv *hv_vcpu, u16 code) · 759 unmodified lines
2420 case HVCALL_SEND_IPI:2455 case HVCALL_SEND_IPI:
2421 return hv_vcpu->cpuid_cache.enlightenments_eax &2456 return hv_vcpu->cpuid_cache.enlightenments_eax &
2422 HV_X64_CLUSTER_IPI_RECOMMENDED;2457 HV_X64_CLUSTER_IPI_RECOMMENDED;
2458 case HV_EXT_CALL_QUERY_CAPABILITIES ... HV_EXT_CALL_MAX:
2459 return hv_vcpu->cpuid_cache.features_ebx &
2460 HV_ENABLE_EXTENDED_HYPERCALLS;
2423 default:2461 default:
2424 break;2462 break;
2425 }2463 }
@@ -2512,14 +2550,7 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) · 86 unmodified lines
2512 ret = HV_STATUS_INVALID_HYPERCALL_INPUT;2550 ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
2513 break;2551 break;
2514 }2552 }
2515 vcpu->run->exit_reason = KVM_EXIT_HYPERV;2553 goto hypercall_userspace_exit;
2516 vcpu->run->hyperv.type = KVM_EXIT_HYPERV_HCALL;
2517 vcpu->run->hyperv.u.hcall.input = hc.param;
2518 vcpu->run->hyperv.u.hcall.params[0] = hc.ingpa;
2519 vcpu->run->hyperv.u.hcall.params[1] = hc.outgpa;
2520 vcpu->arch.complete_userspace_io =
2521 kvm_hv_hypercall_complete_userspace;
2522 return 0;
2523 case HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST:2554 case HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST:
2524 if (unlikely(hc.var_cnt)) {2555 if (unlikely(hc.var_cnt)) {
2525 ret = HV_STATUS_INVALID_HYPERCALL_INPUT;2556 ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
@@ -2578,15 +2609,14 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) · 52 unmodified lines
2578 ret = HV_STATUS_OPERATION_DENIED;2609 ret = HV_STATUS_OPERATION_DENIED;
2579 break;2610 break;
2580 }2611 }
2581 vcpu->run->exit_reason = KVM_EXIT_HYPERV;2612 goto hypercall_userspace_exit;
2582 vcpu->run->hyperv.type = KVM_EXIT_HYPERV_HCALL;
2583 vcpu->run->hyperv.u.hcall.input = hc.param;
2584 vcpu->run->hyperv.u.hcall.params[0] = hc.ingpa;
2585 vcpu->run->hyperv.u.hcall.params[1] = hc.outgpa;
2586 vcpu->arch.complete_userspace_io =
2587 kvm_hv_hypercall_complete_userspace;
2588 return 0;
2589 }2613 }
2614 case HV_EXT_CALL_QUERY_CAPABILITIES ... HV_EXT_CALL_MAX:
2615 if (unlikely(hc.fast)) {
2616 ret = HV_STATUS_INVALID_PARAMETER;
2617 break;
2618 }
2619 goto hypercall_userspace_exit;
2590 default:2620 default:
2591 ret = HV_STATUS_INVALID_HYPERCALL_CODE;2621 ret = HV_STATUS_INVALID_HYPERCALL_CODE;
2592 break;2622 break;
@@ -2594,6 +2624,15 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) · 1 unmodified line
2594 2624
2595hypercall_complete:2625hypercall_complete:
2596 return kvm_hv_hypercall_complete(vcpu, ret);2626 return kvm_hv_hypercall_complete(vcpu, ret);
2627
2628hypercall_userspace_exit:
2629 vcpu->run->exit_reason = KVM_EXIT_HYPERV;
2630 vcpu->run->hyperv.type = KVM_EXIT_HYPERV_HCALL;
2631 vcpu->run->hyperv.u.hcall.input = hc.param;
2632 vcpu->run->hyperv.u.hcall.params[0] = hc.ingpa;
2633 vcpu->run->hyperv.u.hcall.params[1] = hc.outgpa;
2634 vcpu->arch.complete_userspace_io = kvm_hv_hypercall_complete_userspace;
2635 return 0;
2597}2636}
2598 2637
2599void kvm_hv_init_vm(struct kvm *kvm)2638void kvm_hv_init_vm(struct kvm *kvm)
@@ -2733,9 +2772,11 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid, · 133 unmodified lines
2733 ent->eax |= HV_MSR_REFERENCE_TSC_AVAILABLE;2772 ent->eax |= HV_MSR_REFERENCE_TSC_AVAILABLE;
2734 ent->eax |= HV_ACCESS_FREQUENCY_MSRS;2773 ent->eax |= HV_ACCESS_FREQUENCY_MSRS;
2735 ent->eax |= HV_ACCESS_REENLIGHTENMENT;2774 ent->eax |= HV_ACCESS_REENLIGHTENMENT;
2775 ent->eax |= HV_ACCESS_TSC_INVARIANT;
2736 2776
2737 ent->ebx |= HV_POST_MESSAGES;2777 ent->ebx |= HV_POST_MESSAGES;
2738 ent->ebx |= HV_SIGNAL_EVENTS;2778 ent->ebx |= HV_SIGNAL_EVENTS;
2779 ent->ebx |= HV_ENABLE_EXTENDED_HYPERCALLS;
2739 2780
2740 ent->edx |= HV_X64_HYPERCALL_XMM_INPUT_AVAILABLE;2781 ent->edx |= HV_X64_HYPERCALL_XMM_INPUT_AVAILABLE;
2741 ent->edx |= HV_FEATURE_FREQUENCY_MSRS_AVAILABLE;2782 ent->edx |= HV_FEATURE_FREQUENCY_MSRS_AVAILABLE;
arch/x86/kvm/hyperv.h+27−0
@@ -136,6 +136,33 @@ static inline bool kvm_hv_has_stimer_pending(struct kvm_vcpu *vcpu) · 135 unmodified lines
136 HV_SYNIC_STIMER_COUNT);136 HV_SYNIC_STIMER_COUNT);
137}137}
138 138
139/*
140 * With HV_ACCESS_TSC_INVARIANT feature, invariant TSC (CPUID.80000007H:EDX[8])
141 * is only observed after HV_X64_MSR_TSC_INVARIANT_CONTROL was written to.
142 */
143static inline bool kvm_hv_invtsc_suppressed(struct kvm_vcpu *vcpu)
144{
145 struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
146
147 /*
148 * If Hyper-V's invariant TSC control is not exposed to the guest,
149 * the invariant TSC CPUID flag is not suppressed, Windows guests were
150 * observed to be able to handle it correctly. Going forward, VMMs are
151 * encouraged to enable Hyper-V's invariant TSC control when invariant
152 * TSC CPUID flag is set to make KVM's behavior match genuine Hyper-V.
153 */
154 if (!hv_vcpu ||
155 !(hv_vcpu->cpuid_cache.features_eax & HV_ACCESS_TSC_INVARIANT))
156 return false;
157
158 /*
159 * If Hyper-V's invariant TSC control is exposed to the guest, KVM is
160 * responsible for suppressing the invariant TSC CPUID flag if the
161 * Hyper-V control is not enabled.
162 */
163 return !(to_kvm_hv(vcpu->kvm)->hv_invtsc_control & HV_EXPOSE_INVARIANT_TSC);
164}
165
139void kvm_hv_process_stimers(struct kvm_vcpu *vcpu);166void kvm_hv_process_stimers(struct kvm_vcpu *vcpu);
140 167
141void kvm_hv_setup_tsc_page(struct kvm *kvm,168void kvm_hv_setup_tsc_page(struct kvm *kvm,
arch/x86/kvm/i8254.c+2−2
@@ -30,7 +30,7 @@ · 29 unmodified lines
30 * Based on QEMU and Xen.30 * Based on QEMU and Xen.
31 */31 */
32 32
33#define pr_fmt(fmt) "pit: " fmt33#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
34 34
35#include <linux/kvm_host.h>35#include <linux/kvm_host.h>
36#include <linux/slab.h>36#include <linux/slab.h>
@@ -351,7 +351,7 @@ static void create_pit_timer(struct kvm_pit *pit, u32 val, int is_period) · 314 unmodified lines
351 351
352 if (ps->period < min_period) {352 if (ps->period < min_period) {
353 pr_info_ratelimited(353 pr_info_ratelimited(
354 "kvm: requested %lld ns "354 "requested %lld ns "
355 "i8254 timer period limited to %lld ns\n",355 "i8254 timer period limited to %lld ns\n",
356 ps->period, min_period);356 ps->period, min_period);
357 ps->period = min_period;357 ps->period = min_period;
arch/x86/kvm/i8259.c+3−1
@@ -26,6 +26,8 @@ · 25 unmodified lines
26 * Yaozu (Eddie) Dong <Eddie.dong@intel.com>26 * Yaozu (Eddie) Dong <Eddie.dong@intel.com>
27 * Port from Qemu.27 * Port from Qemu.
28 */28 */
29#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
30
29#include <linux/mm.h>31#include <linux/mm.h>
30#include <linux/slab.h>32#include <linux/slab.h>
31#include <linux/bitops.h>33#include <linux/bitops.h>
@@ -35,7 +37,7 @@ · 3 unmodified lines
35#include "trace.h"37#include "trace.h"
36 38
37#define pr_pic_unimpl(fmt, ...) \39#define pr_pic_unimpl(fmt, ...) \
38 pr_err_ratelimited("kvm: pic: " fmt, ## __VA_ARGS__)40 pr_err_ratelimited("pic: " fmt, ## __VA_ARGS__)
39 41
40static void pic_irq_request(struct kvm *kvm, int level);42static void pic_irq_request(struct kvm *kvm, int level);
41 43
arch/x86/kvm/ioapic.c+1−0
@@ -26,6 +26,7 @@ · 25 unmodified lines
26 * Yaozu (Eddie) Dong <eddie.dong@intel.com>26 * Yaozu (Eddie) Dong <eddie.dong@intel.com>
27 * Based on Xen 3.1 code.27 * Based on Xen 3.1 code.
28 */28 */
29#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
29 30
30#include <linux/kvm_host.h>31#include <linux/kvm_host.h>
31#include <linux/kvm.h>32#include <linux/kvm.h>
arch/x86/kvm/irq.c+1−0
@@ -7,6 +7,7 @@ · 6 unmodified lines
7 * Authors:7 * Authors:
8 * Yaozu (Eddie) Dong <Eddie.dong@intel.com>8 * Yaozu (Eddie) Dong <Eddie.dong@intel.com>
9 */9 */
10#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
10 11
11#include <linux/export.h>12#include <linux/export.h>
12#include <linux/kvm_host.h>13#include <linux/kvm_host.h>
arch/x86/kvm/irq_comm.c+4−3
@@ -8,6 +8,7 @@ · 7 unmodified lines
8 *8 *
9 * Copyright 2010 Red Hat, Inc. and/or its affiliates.9 * Copyright 2010 Red Hat, Inc. and/or its affiliates.
10 */10 */
11#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
11 12
12#include <linux/kvm_host.h>13#include <linux/kvm_host.h>
13#include <linux/slab.h>14#include <linux/slab.h>
@@ -56,7 +57,7 @@ int kvm_irq_delivery_to_apic(struct kvm *kvm, struct kvm_lapic *src, · 42 unmodified lines
56 57
57 if (irq->dest_mode == APIC_DEST_PHYSICAL &&58 if (irq->dest_mode == APIC_DEST_PHYSICAL &&
58 irq->dest_id == 0xff && kvm_lowest_prio_delivery(irq)) {59 irq->dest_id == 0xff && kvm_lowest_prio_delivery(irq)) {
59 printk(KERN_INFO "kvm: apic: phys broadcast and lowest prio\n");60 pr_info("apic: phys broadcast and lowest prio\n");
60 irq->delivery_mode = APIC_DM_FIXED;61 irq->delivery_mode = APIC_DM_FIXED;
61 }62 }
62 63
@@ -199,7 +200,7 @@ int kvm_request_irq_source_id(struct kvm *kvm) · 136 unmodified lines
199 irq_source_id = find_first_zero_bit(bitmap, BITS_PER_LONG);200 irq_source_id = find_first_zero_bit(bitmap, BITS_PER_LONG);
200 201
201 if (irq_source_id >= BITS_PER_LONG) {202 if (irq_source_id >= BITS_PER_LONG) {
202 printk(KERN_WARNING "kvm: exhaust allocatable IRQ sources!\n");203 pr_warn("exhausted allocatable IRQ sources!\n");
203 irq_source_id = -EFAULT;204 irq_source_id = -EFAULT;
204 goto unlock;205 goto unlock;
205 }206 }
@@ -221,7 +222,7 @@ void kvm_free_irq_source_id(struct kvm *kvm, int irq_source_id) · 15 unmodified lines
221 mutex_lock(&kvm->irq_lock);222 mutex_lock(&kvm->irq_lock);
222 if (irq_source_id < 0 ||223 if (irq_source_id < 0 ||
223 irq_source_id >= BITS_PER_LONG) {224 irq_source_id >= BITS_PER_LONG) {
224 printk(KERN_ERR "kvm: IRQ source ID out of range!\n");225 pr_err("IRQ source ID out of range!\n");
225 goto unlock;226 goto unlock;
226 }227 }
227 clear_bit(irq_source_id, &kvm->arch.irq_sources_bitmap);228 clear_bit(irq_source_id, &kvm->arch.irq_sources_bitmap);
arch/x86/kvm/kvm_cache_regs.h+12−0
@@ -75,6 +75,18 @@ static inline void kvm_register_mark_dirty(struct kvm_vcpu *vcpu, · 74 unmodified lines
75 __set_bit(reg, (unsigned long *)&vcpu->arch.regs_dirty);75 __set_bit(reg, (unsigned long *)&vcpu->arch.regs_dirty);
76}76}
77 77
78/*
79 * kvm_register_test_and_mark_available() is a special snowflake that uses an
80 * arch bitop directly to avoid the explicit instrumentation that comes with
81 * the generic bitops. This allows code that cannot be instrumented (noinstr
82 * functions), e.g. the low level VM-Enter/VM-Exit paths, to cache registers.
83 */
84static __always_inline bool kvm_register_test_and_mark_available(struct kvm_vcpu *vcpu,
85 enum kvm_reg reg)
86{
87 return arch___test_and_set_bit(reg, (unsigned long *)&vcpu->arch.regs_avail);
88}
89
78/*90/*
79 * The "raw" register helpers are only for cases where the full 64 bits of a91 * The "raw" register helpers are only for cases where the full 64 bits of a
80 * register are read/written irrespective of current vCPU mode. In other words,92 * register are read/written irrespective of current vCPU mode. In other words,
arch/x86/kvm/kvm_emulate.h+2−5
@@ -220,7 +220,8 @@ struct x86_emulate_ops { · 219 unmodified lines
220 220
221 void (*set_nmi_mask)(struct x86_emulate_ctxt *ctxt, bool masked);221 void (*set_nmi_mask)(struct x86_emulate_ctxt *ctxt, bool masked);
222 222
223 unsigned (*get_hflags)(struct x86_emulate_ctxt *ctxt);223 bool (*is_smm)(struct x86_emulate_ctxt *ctxt);
224 bool (*is_guest_mode)(struct x86_emulate_ctxt *ctxt);
224 int (*leave_smm)(struct x86_emulate_ctxt *ctxt);225 int (*leave_smm)(struct x86_emulate_ctxt *ctxt);
225 void (*triple_fault)(struct x86_emulate_ctxt *ctxt);226 void (*triple_fault)(struct x86_emulate_ctxt *ctxt);
226 int (*set_xcr)(struct x86_emulate_ctxt *ctxt, u32 index, u64 xcr);227 int (*set_xcr)(struct x86_emulate_ctxt *ctxt, u32 index, u64 xcr);
@@ -275,10 +276,6 @@ enum x86emul_mode { · 48 unmodified lines
275 X86EMUL_MODE_PROT64, /* 64-bit (long) mode. */276 X86EMUL_MODE_PROT64, /* 64-bit (long) mode. */
276};277};
277 278
278/* These match some of the HF_* flags defined in kvm_host.h */
279#define X86EMUL_GUEST_MASK (1 << 5) /* VCPU is in guest-mode */
280#define X86EMUL_SMM_MASK (1 << 6)
281
282/*279/*
283 * fastop functions are declared as taking a never-defined fastop parameter,280 * fastop functions are declared as taking a never-defined fastop parameter,
284 * so they can't be called from C directly.281 * so they can't be called from C directly.
arch/x86/kvm/kvm_onhyperv.c+1−0
@@ -2,6 +2,7 @@ · 1 unmodified line
2/*2/*
3 * KVM L1 hypervisor optimizations on Hyper-V.3 * KVM L1 hypervisor optimizations on Hyper-V.
4 */4 */
5#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
5 6
6#include <linux/kvm_host.h>7#include <linux/kvm_host.h>
7#include <asm/mshyperv.h>8#include <asm/mshyperv.h>
arch/x86/kvm/lapic.c+296−108
@@ -15,6 +15,7 @@ · 14 unmodified lines
15 *15 *
16 * Based on Xen 3.1 code, Copyright (c) 2004, Intel Corporation.16 * Based on Xen 3.1 code, Copyright (c) 2004, Intel Corporation.
17 */17 */
18#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
18 19
19#include <linux/kvm_host.h>20#include <linux/kvm_host.h>
20#include <linux/kvm.h>21#include <linux/kvm.h>
@@ -166,9 +167,19 @@ static bool kvm_use_posted_timer_interrupt(struct kvm_vcpu *vcpu) · 145 unmodified lines
166 return kvm_can_post_timer_interrupt(vcpu) && vcpu->mode == IN_GUEST_MODE;167 return kvm_can_post_timer_interrupt(vcpu) && vcpu->mode == IN_GUEST_MODE;
167}168}
168 169
170static inline u32 kvm_apic_calc_x2apic_ldr(u32 id)
171{
172 return ((id >> 4) << 16) | (1 << (id & 0xf));
173}
174
169static inline bool kvm_apic_map_get_logical_dest(struct kvm_apic_map *map,175static inline bool kvm_apic_map_get_logical_dest(struct kvm_apic_map *map,
170 u32 dest_id, struct kvm_lapic ***cluster, u16 *mask) {176 u32 dest_id, struct kvm_lapic ***cluster, u16 *mask) {
171 switch (map->mode) {177 switch (map->logical_mode) {
178 case KVM_APIC_MODE_SW_DISABLED:
179 /* Arbitrarily use the flat map so that @cluster isn't NULL. */
180 *cluster = map->xapic_flat_map;
181 *mask = 0;
182 return true;
172 case KVM_APIC_MODE_X2APIC: {183 case KVM_APIC_MODE_X2APIC: {
173 u32 offset = (dest_id >> 16) * 16;184 u32 offset = (dest_id >> 16) * 16;
174 u32 max_apic_id = map->max_apic_id;185 u32 max_apic_id = map->max_apic_id;
@@ -193,8 +204,10 @@ static inline bool kvm_apic_map_get_logical_dest(struct kvm_apic_map *map, · 18 unmodified lines
193 *cluster = map->xapic_cluster_map[(dest_id >> 4) & 0xf];204 *cluster = map->xapic_cluster_map[(dest_id >> 4) & 0xf];
194 *mask = dest_id & 0xf;205 *mask = dest_id & 0xf;
195 return true;206 return true;
207 case KVM_APIC_MODE_MAP_DISABLED:
208 return false;
196 default:209 default:
197 /* Not optimized. */210 WARN_ON_ONCE(1);
198 return false;211 return false;
199 }212 }
200}213}
@@ -206,6 +219,134 @@ static void kvm_apic_map_free(struct rcu_head *rcu) · 5 unmodified lines
206 kvfree(map);219 kvfree(map);
207}220}
208 221
222static int kvm_recalculate_phys_map(struct kvm_apic_map *new,
223 struct kvm_vcpu *vcpu,
224 bool *xapic_id_mismatch)
225{
226 struct kvm_lapic *apic = vcpu->arch.apic;
227 u32 x2apic_id = kvm_x2apic_id(apic);
228 u32 xapic_id = kvm_xapic_id(apic);
229 u32 physical_id;
230
231 /*
232 * Deliberately truncate the vCPU ID when detecting a mismatched APIC
233 * ID to avoid false positives if the vCPU ID, i.e. x2APIC ID, is a
234 * 32-bit value. Any unwanted aliasing due to truncation results will
235 * be detected below.
236 */
237 if (!apic_x2apic_mode(apic) && xapic_id != (u8)vcpu->vcpu_id)
238 *xapic_id_mismatch = true;
239
240 /*
241 * Apply KVM's hotplug hack if userspace has enable 32-bit APIC IDs.
242 * Allow sending events to vCPUs by their x2APIC ID even if the target
243 * vCPU is in legacy xAPIC mode, and silently ignore aliased xAPIC IDs
244 * (the x2APIC ID is truncated to 8 bits, causing IDs > 0xff to wrap
245 * and collide).
246 *
247 * Honor the architectural (and KVM's non-optimized) behavior if
248 * userspace has not enabled 32-bit x2APIC IDs. Each APIC is supposed
249 * to process messages independently. If multiple vCPUs have the same
250 * effective APIC ID, e.g. due to the x2APIC wrap or because the guest
251 * manually modified its xAPIC IDs, events targeting that ID are
252 * supposed to be recognized by all vCPUs with said ID.
253 */
254 if (vcpu->kvm->arch.x2apic_format) {
255 /* See also kvm_apic_match_physical_addr(). */
256 if ((apic_x2apic_mode(apic) || x2apic_id > 0xff) &&
257 x2apic_id <= new->max_apic_id)
258 new->phys_map[x2apic_id] = apic;
259
260 if (!apic_x2apic_mode(apic) && !new->phys_map[xapic_id])
261 new->phys_map[xapic_id] = apic;
262 } else {
263 /*
264 * Disable the optimized map if the physical APIC ID is already
265 * mapped, i.e. is aliased to multiple vCPUs. The optimized
266 * map requires a strict 1:1 mapping between IDs and vCPUs.
267 */
268 if (apic_x2apic_mode(apic))
269 physical_id = x2apic_id;
270 else
271 physical_id = xapic_id;
272
273 if (new->phys_map[physical_id])
274 return -EINVAL;
275
276 new->phys_map[physical_id] = apic;
277 }
278
279 return 0;
280}
281
282static void kvm_recalculate_logical_map(struct kvm_apic_map *new,
283 struct kvm_vcpu *vcpu)
284{
285 struct kvm_lapic *apic = vcpu->arch.apic;
286 enum kvm_apic_logical_mode logical_mode;
287 struct kvm_lapic **cluster;
288 u16 mask;
289 u32 ldr;
290
291 if (new->logical_mode == KVM_APIC_MODE_MAP_DISABLED)
292 return;
293
294 if (!kvm_apic_sw_enabled(apic))
295 return;
296
297 ldr = kvm_lapic_get_reg(apic, APIC_LDR);
298 if (!ldr)
299 return;
300
301 if (apic_x2apic_mode(apic)) {
302 logical_mode = KVM_APIC_MODE_X2APIC;
303 } else {
304 ldr = GET_APIC_LOGICAL_ID(ldr);
305 if (kvm_lapic_get_reg(apic, APIC_DFR) == APIC_DFR_FLAT)
306 logical_mode = KVM_APIC_MODE_XAPIC_FLAT;
307 else
308 logical_mode = KVM_APIC_MODE_XAPIC_CLUSTER;
309 }
310
311 /*
312 * To optimize logical mode delivery, all software-enabled APICs must
313 * be configured for the same mode.
314 */
315 if (new->logical_mode == KVM_APIC_MODE_SW_DISABLED) {
316 new->logical_mode = logical_mode;
317 } else if (new->logical_mode != logical_mode) {
318 new->logical_mode = KVM_APIC_MODE_MAP_DISABLED;
319 return;
320 }
321
322 /*
323 * In x2APIC mode, the LDR is read-only and derived directly from the
324 * x2APIC ID, thus is guaranteed to be addressable. KVM reuses
325 * kvm_apic_map.phys_map to optimize logical mode x2APIC interrupts by
326 * reversing the LDR calculation to get cluster of APICs, i.e. no
327 * additional work is required.
328 */
329 if (apic_x2apic_mode(apic)) {
330 WARN_ON_ONCE(ldr != kvm_apic_calc_x2apic_ldr(kvm_x2apic_id(apic)));
331 return;
332 }
333
334 if (WARN_ON_ONCE(!kvm_apic_map_get_logical_dest(new, ldr,
335 &cluster, &mask))) {
336 new->logical_mode = KVM_APIC_MODE_MAP_DISABLED;
337 return;
338 }
339
340 if (!mask)
341 return;
342
343 ldr = ffs(mask) - 1;
344 if (!is_power_of_2(mask) || cluster[ldr])
345 new->logical_mode = KVM_APIC_MODE_MAP_DISABLED;
346 else
347 cluster[ldr] = apic;
348}
349
209/*350/*
210 * CLEAN -> DIRTY and UPDATE_IN_PROGRESS -> DIRTY changes happen without a lock.351 * CLEAN -> DIRTY and UPDATE_IN_PROGRESS -> DIRTY changes happen without a lock.
211 *352 *
@@ -224,6 +365,7 @@ void kvm_recalculate_apic_map(struct kvm *kvm) · 12 unmodified lines
224 struct kvm_vcpu *vcpu;365 struct kvm_vcpu *vcpu;
225 unsigned long i;366 unsigned long i;
226 u32 max_id = 255; /* enough space for any xAPIC ID */367 u32 max_id = 255; /* enough space for any xAPIC ID */
368 bool xapic_id_mismatch = false;
227 369
228 /* Read kvm->arch.apic_map_dirty before kvm->arch.apic_map. */370 /* Read kvm->arch.apic_map_dirty before kvm->arch.apic_map. */
229 if (atomic_read_acquire(&kvm->arch.apic_map_dirty) == CLEAN)371 if (atomic_read_acquire(&kvm->arch.apic_map_dirty) == CLEAN)
@@ -256,54 +398,41 @@ void kvm_recalculate_apic_map(struct kvm *kvm) · 26 unmodified lines
256 goto out;398 goto out;
257 399
258 new->max_apic_id = max_id;400 new->max_apic_id = max_id;
401 new->logical_mode = KVM_APIC_MODE_SW_DISABLED;
259 402
260 kvm_for_each_vcpu(i, vcpu, kvm) {403 kvm_for_each_vcpu(i, vcpu, kvm) {
261 struct kvm_lapic *apic = vcpu->arch.apic;
262 struct kvm_lapic **cluster;
263 u16 mask;
264 u32 ldr;
265 u8 xapic_id;
266 u32 x2apic_id;
267
268 if (!kvm_apic_present(vcpu))404 if (!kvm_apic_present(vcpu))
269 continue;405 continue;
270 406
271 xapic_id = kvm_xapic_id(apic);407 if (kvm_recalculate_phys_map(new, vcpu, &xapic_id_mismatch)) {
272 x2apic_id = kvm_x2apic_id(apic);408 kvfree(new);
273 409 new = NULL;
274 /* Hotplug hack: see kvm_apic_match_physical_addr(), ... */410 goto out;
275 if ((apic_x2apic_mode(apic) || x2apic_id > 0xff) &&
276 x2apic_id <= new->max_apic_id)
277 new->phys_map[x2apic_id] = apic;
278 /*
279 * ... xAPIC ID of VCPUs with APIC ID > 0xff will wrap-around,
280 * prevent them from masking VCPUs with APIC ID <= 0xff.
281 */
282 if (!apic_x2apic_mode(apic) && !new->phys_map[xapic_id])
283 new->phys_map[xapic_id] = apic;
284
285 if (!kvm_apic_sw_enabled(apic))
286 continue;
287
288 ldr = kvm_lapic_get_reg(apic, APIC_LDR);
289
290 if (apic_x2apic_mode(apic)) {
291 new->mode |= KVM_APIC_MODE_X2APIC;
292 } else if (ldr) {
293 ldr = GET_APIC_LOGICAL_ID(ldr);
294 if (kvm_lapic_get_reg(apic, APIC_DFR) == APIC_DFR_FLAT)
295 new->mode |= KVM_APIC_MODE_XAPIC_FLAT;
296 else
297 new->mode |= KVM_APIC_MODE_XAPIC_CLUSTER;
298 }411 }
299 412
300 if (!kvm_apic_map_get_logical_dest(new, ldr, &cluster, &mask))413 kvm_recalculate_logical_map(new, vcpu);
301 continue;
302
303 if (mask)
304 cluster[ffs(mask) - 1] = apic;
305 }414 }
306out:415out:
416 /*
417 * The optimized map is effectively KVM's internal version of APICv,
418 * and all unwanted aliasing that results in disabling the optimized
419 * map also applies to APICv.
420 */
421 if (!new)
422 kvm_set_apicv_inhibit(kvm, APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED);
423 else
424 kvm_clear_apicv_inhibit(kvm, APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED);
425
426 if (!new || new->logical_mode == KVM_APIC_MODE_MAP_DISABLED)
427 kvm_set_apicv_inhibit(kvm, APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED);
428 else
429 kvm_clear_apicv_inhibit(kvm, APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED);
430
431 if (xapic_id_mismatch)
432 kvm_set_apicv_inhibit(kvm, APICV_INHIBIT_REASON_APIC_ID_MODIFIED);
433 else
434 kvm_clear_apicv_inhibit(kvm, APICV_INHIBIT_REASON_APIC_ID_MODIFIED);
435
307 old = rcu_dereference_protected(kvm->arch.apic_map,436 old = rcu_dereference_protected(kvm->arch.apic_map,
308 lockdep_is_held(&kvm->arch.apic_map_lock));437 lockdep_is_held(&kvm->arch.apic_map_lock));
309 rcu_assign_pointer(kvm->arch.apic_map, new);438 rcu_assign_pointer(kvm->arch.apic_map, new);
@@ -360,11 +489,6 @@ static inline void kvm_apic_set_dfr(struct kvm_lapic *apic, u32 val) · 50 unmodified lines
360 atomic_set_release(&apic->vcpu->kvm->arch.apic_map_dirty, DIRTY);489 atomic_set_release(&apic->vcpu->kvm->arch.apic_map_dirty, DIRTY);
361}490}
362 491
363static inline u32 kvm_apic_calc_x2apic_ldr(u32 id)
364{
365 return ((id >> 4) << 16) | (1 << (id & 0xf));
366}
367
368static inline void kvm_apic_set_x2apic_id(struct kvm_lapic *apic, u32 id)492static inline void kvm_apic_set_x2apic_id(struct kvm_lapic *apic, u32 id)
369{493{
370 u32 ldr = kvm_apic_calc_x2apic_ldr(id);494 u32 ldr = kvm_apic_calc_x2apic_ldr(id);
@@ -941,8 +1065,7 @@ static void kvm_apic_disabled_lapic_found(struct kvm *kvm) · 570 unmodified lines
941{1065{
942 if (!kvm->arch.disabled_lapic_found) {1066 if (!kvm->arch.disabled_lapic_found) {
943 kvm->arch.disabled_lapic_found = true;1067 kvm->arch.disabled_lapic_found = true;
944 printk(KERN_INFO1068 pr_info("Disabled LAPIC found during irq injection\n");
945 "Disabled LAPIC found during irq injection\n");
946 }1069 }
947}1070}
948 1071
@@ -951,7 +1074,7 @@ static bool kvm_apic_is_broadcast_dest(struct kvm *kvm, struct kvm_lapic **src, · 2 unmodified lines
951{1074{
952 if (kvm->arch.x2apic_broadcast_quirk_disabled) {1075 if (kvm->arch.x2apic_broadcast_quirk_disabled) {
953 if ((irq->dest_id == APIC_BROADCAST &&1076 if ((irq->dest_id == APIC_BROADCAST &&
954 map->mode != KVM_APIC_MODE_X2APIC))1077 map->logical_mode != KVM_APIC_MODE_X2APIC))
955 return true;1078 return true;
956 if (irq->dest_id == X2APIC_BROADCAST)1079 if (irq->dest_id == X2APIC_BROADCAST)
957 return true;1080 return true;
@@ -1364,7 +1487,6 @@ static u32 apic_get_tmcct(struct kvm_lapic *apic) · 406 unmodified lines
1364{1487{
1365 ktime_t remaining, now;1488 ktime_t remaining, now;
1366 s64 ns;1489 s64 ns;
1367 u32 tmcct;
1368 1490
1369 ASSERT(apic != NULL);1491 ASSERT(apic != NULL);
1370 1492
@@ -1379,10 +1501,7 @@ static u32 apic_get_tmcct(struct kvm_lapic *apic) · 8 unmodified lines
1379 remaining = 0;1501 remaining = 0;
1380 1502
1381 ns = mod_64(ktime_to_ns(remaining), apic->lapic_timer.period);1503 ns = mod_64(ktime_to_ns(remaining), apic->lapic_timer.period);
1382 tmcct = div64_u64(ns,1504 return div64_u64(ns, (APIC_BUS_CYCLE_NS * apic->divide_count));
1383 (APIC_BUS_CYCLE_NS * apic->divide_count));
1384
1385 return tmcct;
1386}1505}
1387 1506
1388static void __report_tpr_access(struct kvm_lapic *apic, bool write)1507static void __report_tpr_access(struct kvm_lapic *apic, bool write)
@@ -1442,19 +1561,15 @@ static inline struct kvm_lapic *to_lapic(struct kvm_io_device *dev) · 53 unmodified lines
1442#define APIC_REGS_MASK(first, count) \1561#define APIC_REGS_MASK(first, count) \
1443 (APIC_REG_MASK(first) * ((1ull << (count)) - 1))1562 (APIC_REG_MASK(first) * ((1ull << (count)) - 1))
1444 1563
1445static int kvm_lapic_reg_read(struct kvm_lapic *apic, u32 offset, int len,1564u64 kvm_lapic_readable_reg_mask(struct kvm_lapic *apic)
1446 void *data)
1447{1565{
1448 unsigned char alignment = offset & 0xf;1566 /* Leave bits '0' for reserved and write-only registers. */
1449 u32 result;
1450 /* this bitmask has a bit cleared for each reserved register */
1451 u64 valid_reg_mask =1567 u64 valid_reg_mask =
1452 APIC_REG_MASK(APIC_ID) |1568 APIC_REG_MASK(APIC_ID) |
1453 APIC_REG_MASK(APIC_LVR) |1569 APIC_REG_MASK(APIC_LVR) |
1454 APIC_REG_MASK(APIC_TASKPRI) |1570 APIC_REG_MASK(APIC_TASKPRI) |
1455 APIC_REG_MASK(APIC_PROCPRI) |1571 APIC_REG_MASK(APIC_PROCPRI) |
1456 APIC_REG_MASK(APIC_LDR) |1572 APIC_REG_MASK(APIC_LDR) |
1457 APIC_REG_MASK(APIC_DFR) |
1458 APIC_REG_MASK(APIC_SPIV) |1573 APIC_REG_MASK(APIC_SPIV) |
1459 APIC_REGS_MASK(APIC_ISR, APIC_ISR_NR) |1574 APIC_REGS_MASK(APIC_ISR, APIC_ISR_NR) |
1460 APIC_REGS_MASK(APIC_TMR, APIC_ISR_NR) |1575 APIC_REGS_MASK(APIC_TMR, APIC_ISR_NR) |
@@ -1474,21 +1589,33 @@ static int kvm_lapic_reg_read(struct kvm_lapic *apic, u32 offset, int len, · 13 unmodified lines
1474 if (kvm_lapic_lvt_supported(apic, LVT_CMCI))1589 if (kvm_lapic_lvt_supported(apic, LVT_CMCI))
1475 valid_reg_mask |= APIC_REG_MASK(APIC_LVTCMCI);1590 valid_reg_mask |= APIC_REG_MASK(APIC_LVTCMCI);
1476 1591
1477 /*1592 /* ARBPRI, DFR, and ICR2 are not valid in x2APIC mode. */
1478 * ARBPRI and ICR2 are not valid in x2APIC mode. WARN if KVM reads ICR
1479 * in x2APIC mode as it's an 8-byte register in x2APIC and needs to be
1480 * manually handled by the caller.
1481 */
1482 if (!apic_x2apic_mode(apic))1593 if (!apic_x2apic_mode(apic))
1483 valid_reg_mask |= APIC_REG_MASK(APIC_ARBPRI) |1594 valid_reg_mask |= APIC_REG_MASK(APIC_ARBPRI) |
1595 APIC_REG_MASK(APIC_DFR) |
1484 APIC_REG_MASK(APIC_ICR2);1596 APIC_REG_MASK(APIC_ICR2);
1485 else1597
1486 WARN_ON_ONCE(offset == APIC_ICR);1598 return valid_reg_mask;
1599}
1600EXPORT_SYMBOL_GPL(kvm_lapic_readable_reg_mask);
1601
1602static int kvm_lapic_reg_read(struct kvm_lapic *apic, u32 offset, int len,
1603 void *data)
1604{
1605 unsigned char alignment = offset & 0xf;
1606 u32 result;
1607
1608 /*
1609 * WARN if KVM reads ICR in x2APIC mode, as it's an 8-byte register in
1610 * x2APIC and needs to be manually handled by the caller.
1611 */
1612 WARN_ON_ONCE(apic_x2apic_mode(apic) && offset == APIC_ICR);
1487 1613
1488 if (alignment + len > 4)1614 if (alignment + len > 4)
1489 return 1;1615 return 1;
1490 1616
1491 if (offset > 0x3f0 || !(valid_reg_mask & APIC_REG_MASK(offset)))1617 if (offset > 0x3f0 ||
1618 !(kvm_lapic_readable_reg_mask(apic) & APIC_REG_MASK(offset)))
1492 return 1;1619 return 1;
1493 1620
1494 result = __apic_read(apic, offset & ~0xf);1621 result = __apic_read(apic, offset & ~0xf);
@@ -1560,7 +1687,7 @@ static void limit_periodic_timer_frequency(struct kvm_lapic *apic) · 65 unmodified lines
1560 1687
1561 if (apic->lapic_timer.period < min_period) {1688 if (apic->lapic_timer.period < min_period) {
1562 pr_info_ratelimited(1689 pr_info_ratelimited(
1563 "kvm: vcpu %i: requested %lld ns "1690 "vcpu %i: requested %lld ns "
1564 "lapic timer period limited to %lld ns\n",1691 "lapic timer period limited to %lld ns\n",
1565 apic->vcpu->vcpu_id,1692 apic->vcpu->vcpu_id,
1566 apic->lapic_timer.period, min_period);1693 apic->lapic_timer.period, min_period);
@@ -1841,11 +1968,15 @@ static bool set_target_expiration(struct kvm_lapic *apic, u32 count_reg) · 274 unmodified lines
1841 if (unlikely(count_reg != APIC_TMICT)) {1968 if (unlikely(count_reg != APIC_TMICT)) {
1842 deadline = tmict_to_ns(apic,1969 deadline = tmict_to_ns(apic,
1843 kvm_lapic_get_reg(apic, count_reg));1970 kvm_lapic_get_reg(apic, count_reg));
1844 if (unlikely(deadline <= 0))1971 if (unlikely(deadline <= 0)) {
1845 deadline = apic->lapic_timer.period;1972 if (apic_lvtt_period(apic))
1973 deadline = apic->lapic_timer.period;
1974 else
1975 deadline = 0;
1976 }
1846 else if (unlikely(deadline > apic->lapic_timer.period)) {1977 else if (unlikely(deadline > apic->lapic_timer.period)) {
1847 pr_info_ratelimited(1978 pr_info_ratelimited(
1848 "kvm: vcpu %i: requested lapic timer restore with "1979 "vcpu %i: requested lapic timer restore with "
1849 "starting count register %#x=%u (%lld ns) > initial count (%lld ns). "1980 "starting count register %#x=%u (%lld ns) > initial count (%lld ns). "
1850 "Using initial count to start timer.\n",1981 "Using initial count to start timer.\n",
1851 apic->vcpu->vcpu_id,1982 apic->vcpu->vcpu_id,
@@ -2068,19 +2199,6 @@ static void apic_manage_nmi_watchdog(struct kvm_lapic *apic, u32 lvt0_val) · 216 unmodified lines
2068 }2199 }
2069}2200}
2070 2201
2071static void kvm_lapic_xapic_id_updated(struct kvm_lapic *apic)
2072{
2073 struct kvm *kvm = apic->vcpu->kvm;
2074
2075 if (KVM_BUG_ON(apic_x2apic_mode(apic), kvm))
2076 return;
2077
2078 if (kvm_xapic_id(apic) == apic->vcpu->vcpu_id)
2079 return;
2080
2081 kvm_set_apicv_inhibit(apic->vcpu->kvm, APICV_INHIBIT_REASON_APIC_ID_MODIFIED);
2082}
2083
2084static int get_lvt_index(u32 reg)2202static int get_lvt_index(u32 reg)
2085{2203{
2086 if (reg == APIC_LVTCMCI)2204 if (reg == APIC_LVTCMCI)
@@ -2101,7 +2219,6 @@ static int kvm_lapic_reg_write(struct kvm_lapic *apic, u32 reg, u32 val) · 14 unmodified lines
2101 case APIC_ID: /* Local APIC ID */2219 case APIC_ID: /* Local APIC ID */
2102 if (!apic_x2apic_mode(apic)) {2220 if (!apic_x2apic_mode(apic)) {
2103 kvm_apic_set_xapic_id(apic, val >> 24);2221 kvm_apic_set_xapic_id(apic, val >> 24);
2104 kvm_lapic_xapic_id_updated(apic);
2105 } else {2222 } else {
2106 ret = 1;2223 ret = 1;
2107 }2224 }
@@ -2219,10 +2336,14 @@ static int kvm_lapic_reg_write(struct kvm_lapic *apic, u32 reg, u32 val) · 111 unmodified lines
2219 break;2336 break;
2220 2337
2221 case APIC_SELF_IPI:2338 case APIC_SELF_IPI:
2222 if (apic_x2apic_mode(apic))2339 /*
2223 kvm_apic_send_ipi(apic, APIC_DEST_SELF | (val & APIC_VECTOR_MASK), 0);2340 * Self-IPI exists only when x2APIC is enabled. Bits 7:0 hold
2224 else2341 * the vector, everything else is reserved.
2342 */
2343 if (!apic_x2apic_mode(apic) || (val & ~APIC_VECTOR_MASK))
2225 ret = 1;2344 ret = 1;
2345 else
2346 kvm_apic_send_ipi(apic, APIC_DEST_SELF | val, 0);
2226 break;2347 break;
2227 default:2348 default:
2228 ret = 1;2349 ret = 1;
@@ -2284,23 +2405,18 @@ void kvm_apic_write_nodecode(struct kvm_vcpu *vcpu, u32 offset) · 55 unmodified lines
2284 struct kvm_lapic *apic = vcpu->arch.apic;2405 struct kvm_lapic *apic = vcpu->arch.apic;
2285 u64 val;2406 u64 val;
2286 2407
2287 if (apic_x2apic_mode(apic)) {
2288 if (KVM_BUG_ON(kvm_lapic_msr_read(apic, offset, &val), vcpu->kvm))
2289 return;
2290 } else {
2291 val = kvm_lapic_get_reg(apic, offset);
2292 }
2293
2294 /*2408 /*
2295 * ICR is a single 64-bit register when x2APIC is enabled. For legacy2409 * ICR is a single 64-bit register when x2APIC is enabled. For legacy
2296 * xAPIC, ICR writes need to go down the common (slightly slower) path2410 * xAPIC, ICR writes need to go down the common (slightly slower) path
2297 * to get the upper half from ICR2.2411 * to get the upper half from ICR2.
2298 */2412 */
2299 if (apic_x2apic_mode(apic) && offset == APIC_ICR) {2413 if (apic_x2apic_mode(apic) && offset == APIC_ICR) {
2414 val = kvm_lapic_get_reg64(apic, APIC_ICR);
2300 kvm_apic_send_ipi(apic, (u32)val, (u32)(val >> 32));2415 kvm_apic_send_ipi(apic, (u32)val, (u32)(val >> 32));
2301 trace_kvm_apic_write(APIC_ICR, val);2416 trace_kvm_apic_write(APIC_ICR, val);
2302 } else {2417 } else {
2303 /* TODO: optimize to just emulate side effect w/o one more write */2418 /* TODO: optimize to just emulate side effect w/o one more write */
2419 val = kvm_lapic_get_reg(apic, offset);
2304 kvm_lapic_reg_write(apic, offset, (u32)val);2420 kvm_lapic_reg_write(apic, offset, (u32)val);
2305 }2421 }
2306}2422}
@@ -2394,11 +2510,15 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64 value) · 87 unmodified lines
2394 }2510 }
2395 }2511 }
2396 2512
2397 if (((old_value ^ value) & X2APIC_ENABLE) && (value & X2APIC_ENABLE))2513 if ((old_value ^ value) & X2APIC_ENABLE) {
2398 kvm_apic_set_x2apic_id(apic, vcpu->vcpu_id);2514 if (value & X2APIC_ENABLE)
2515 kvm_apic_set_x2apic_id(apic, vcpu->vcpu_id);
2516 else if (value & MSR_IA32_APICBASE_ENABLE)
2517 kvm_apic_set_xapic_id(apic, vcpu->vcpu_id);
2518 }
2399 2519
2400 if ((old_value ^ value) & (MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE)) {2520 if ((old_value ^ value) & (MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE)) {
2401 kvm_vcpu_update_apicv(vcpu);2521 kvm_make_request(KVM_REQ_APICV_UPDATE, vcpu);
2402 static_call_cond(kvm_x86_set_virtual_apic_mode)(vcpu);2522 static_call_cond(kvm_x86_set_virtual_apic_mode)(vcpu);
2403 }2523 }
2404 2524
@@ -2429,6 +2549,78 @@ void kvm_apic_update_apicv(struct kvm_vcpu *vcpu) · 24 unmodified lines
2429 */2549 */
2430 apic->isr_count = count_vectors(apic->regs + APIC_ISR);2550 apic->isr_count = count_vectors(apic->regs + APIC_ISR);
2431 }2551 }
2552 apic->highest_isr_cache = -1;
2553}
2554
2555int kvm_alloc_apic_access_page(struct kvm *kvm)
2556{
2557 struct page *page;
2558 void __user *hva;
2559 int ret = 0;
2560
2561 mutex_lock(&kvm->slots_lock);
2562 if (kvm->arch.apic_access_memslot_enabled ||
2563 kvm->arch.apic_access_memslot_inhibited)
2564 goto out;
2565
2566 hva = __x86_set_memory_region(kvm, APIC_ACCESS_PAGE_PRIVATE_MEMSLOT,
2567 APIC_DEFAULT_PHYS_BASE, PAGE_SIZE);
2568 if (IS_ERR(hva)) {
2569 ret = PTR_ERR(hva);
2570 goto out;
2571 }
2572
2573 page = gfn_to_page(kvm, APIC_DEFAULT_PHYS_BASE >> PAGE_SHIFT);
2574 if (is_error_page(page)) {
2575 ret = -EFAULT;
2576 goto out;
2577 }
2578
2579 /*
2580 * Do not pin the page in memory, so that memory hot-unplug
2581 * is able to migrate it.
2582 */
2583 put_page(page);
2584 kvm->arch.apic_access_memslot_enabled = true;
2585out:
2586 mutex_unlock(&kvm->slots_lock);
2587 return ret;
2588}
2589EXPORT_SYMBOL_GPL(kvm_alloc_apic_access_page);
2590
2591void kvm_inhibit_apic_access_page(struct kvm_vcpu *vcpu)
2592{
2593 struct kvm *kvm = vcpu->kvm;
2594
2595 if (!kvm->arch.apic_access_memslot_enabled)
2596 return;
2597
2598 kvm_vcpu_srcu_read_unlock(vcpu);
2599
2600 mutex_lock(&kvm->slots_lock);
2601
2602 if (kvm->arch.apic_access_memslot_enabled) {
2603 __x86_set_memory_region(kvm, APIC_ACCESS_PAGE_PRIVATE_MEMSLOT, 0, 0);
2604 /*
2605 * Clear "enabled" after the memslot is deleted so that a
2606 * different vCPU doesn't get a false negative when checking
2607 * the flag out of slots_lock. No additional memory barrier is
2608 * needed as modifying memslots requires waiting other vCPUs to
2609 * drop SRCU (see above), and false positives are ok as the
2610 * flag is rechecked after acquiring slots_lock.
2611 */
2612 kvm->arch.apic_access_memslot_enabled = false;
2613
2614 /*
2615 * Mark the memslot as inhibited to prevent reallocating the
2616 * memslot during vCPU creation, e.g. if a vCPU is hotplugged.
2617 */
2618 kvm->arch.apic_access_memslot_inhibited = true;
2619 }
2620
2621 mutex_unlock(&kvm->slots_lock);
2622
2623 kvm_vcpu_srcu_read_lock(vcpu);
2432}2624}
2433 2625
2434void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event)2626void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event)
@@ -2484,7 +2676,6 @@ void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event) · 49 unmodified lines
2484 kvm_lapic_set_reg(apic, APIC_TMR + 0x10 * i, 0);2676 kvm_lapic_set_reg(apic, APIC_TMR + 0x10 * i, 0);
2485 }2677 }
2486 kvm_apic_update_apicv(vcpu);2678 kvm_apic_update_apicv(vcpu);
2487 apic->highest_isr_cache = -1;
2488 update_divide_count(apic);2679 update_divide_count(apic);
2489 atomic_set(&apic->lapic_timer.pending, 0);2680 atomic_set(&apic->lapic_timer.pending, 0);
2490 2681
@@ -2756,9 +2947,6 @@ int kvm_apic_set_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s) · 265 unmodified lines
2756 }2947 }
2757 memcpy(vcpu->arch.apic->regs, s->regs, sizeof(*s));2948 memcpy(vcpu->arch.apic->regs, s->regs, sizeof(*s));
2758 2949
2759 if (!apic_x2apic_mode(apic))
2760 kvm_lapic_xapic_id_updated(apic);
2761
2762 atomic_set_release(&apic->vcpu->kvm->arch.apic_map_dirty, DIRTY);2950 atomic_set_release(&apic->vcpu->kvm->arch.apic_map_dirty, DIRTY);
2763 kvm_recalculate_apic_map(vcpu->kvm);2951 kvm_recalculate_apic_map(vcpu->kvm);
2764 kvm_apic_set_version(vcpu);2952 kvm_apic_set_version(vcpu);
@@ -2772,7 +2960,6 @@ int kvm_apic_set_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s) · 7 unmodified lines
2772 __start_apic_timer(apic, APIC_TMCCT);2960 __start_apic_timer(apic, APIC_TMCCT);
2773 kvm_lapic_set_reg(apic, APIC_TMCCT, 0);2961 kvm_lapic_set_reg(apic, APIC_TMCCT, 0);
2774 kvm_apic_update_apicv(vcpu);2962 kvm_apic_update_apicv(vcpu);
2775 apic->highest_isr_cache = -1;
2776 if (apic->apicv_active) {2963 if (apic->apicv_active) {
2777 static_call_cond(kvm_x86_apicv_post_state_restore)(vcpu);2964 static_call_cond(kvm_x86_apicv_post_state_restore)(vcpu);
2778 static_call_cond(kvm_x86_hwapic_irr_update)(vcpu, apic_find_highest_irr(apic));2965 static_call_cond(kvm_x86_hwapic_irr_update)(vcpu, apic_find_highest_irr(apic));
@@ -2943,13 +3130,17 @@ static int kvm_lapic_msr_read(struct kvm_lapic *apic, u32 reg, u64 *data) · 164 unmodified lines
2943static int kvm_lapic_msr_write(struct kvm_lapic *apic, u32 reg, u64 data)3130static int kvm_lapic_msr_write(struct kvm_lapic *apic, u32 reg, u64 data)
2944{3131{
2945 /*3132 /*
2946 * ICR is a 64-bit register in x2APIC mode (and Hyper'v PV vAPIC) and3133 * ICR is a 64-bit register in x2APIC mode (and Hyper-V PV vAPIC) and
2947 * can be written as such, all other registers remain accessible only3134 * can be written as such, all other registers remain accessible only
2948 * through 32-bit reads/writes.3135 * through 32-bit reads/writes.
2949 */3136 */
2950 if (reg == APIC_ICR)3137 if (reg == APIC_ICR)
2951 return kvm_x2apic_icr_write(apic, data);3138 return kvm_x2apic_icr_write(apic, data);
2952 3139
3140 /* Bits 63:32 are reserved in all other registers. */
3141 if (data >> 32)
3142 return 1;
3143
2953 return kvm_lapic_reg_write(apic, reg, (u32)data);3144 return kvm_lapic_reg_write(apic, reg, (u32)data);
2954}3145}
2955 3146
@@ -2972,9 +3163,6 @@ int kvm_x2apic_msr_read(struct kvm_vcpu *vcpu, u32 msr, u64 *data) · 16 unmodified lines
2972 if (!lapic_in_kernel(vcpu) || !apic_x2apic_mode(apic))3163 if (!lapic_in_kernel(vcpu) || !apic_x2apic_mode(apic))
2973 return 1;3164 return 1;
2974 3165
2975 if (reg == APIC_DFR)
2976 return 1;
2977
2978 return kvm_lapic_msr_read(apic, reg, data);3166 return kvm_lapic_msr_read(apic, reg, data);
2979}3167}
2980 3168
arch/x86/kvm/lapic.h+4−0
@@ -112,6 +112,8 @@ int kvm_apic_set_irq(struct kvm_vcpu *vcpu, struct kvm_lapic_irq *irq, · 111 unmodified lines
112 struct dest_map *dest_map);112 struct dest_map *dest_map);
113int kvm_apic_local_deliver(struct kvm_lapic *apic, int lvt_type);113int kvm_apic_local_deliver(struct kvm_lapic *apic, int lvt_type);
114void kvm_apic_update_apicv(struct kvm_vcpu *vcpu);114void kvm_apic_update_apicv(struct kvm_vcpu *vcpu);
115int kvm_alloc_apic_access_page(struct kvm *kvm);
116void kvm_inhibit_apic_access_page(struct kvm_vcpu *vcpu);
115 117
116bool kvm_irq_delivery_to_apic_fast(struct kvm *kvm, struct kvm_lapic *src,118bool kvm_irq_delivery_to_apic_fast(struct kvm *kvm, struct kvm_lapic *src,
117 struct kvm_lapic_irq *irq, int *r, struct dest_map *dest_map);119 struct kvm_lapic_irq *irq, int *r, struct dest_map *dest_map);
@@ -144,6 +146,8 @@ int kvm_hv_vapic_msr_read(struct kvm_vcpu *vcpu, u32 msr, u64 *data); · 26 unmodified lines
144int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len);146int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len);
145void kvm_lapic_exit(void);147void kvm_lapic_exit(void);
146 148
149u64 kvm_lapic_readable_reg_mask(struct kvm_lapic *apic);
150
147#define VEC_POS(v) ((v) & (32 - 1))151#define VEC_POS(v) ((v) & (32 - 1))
148#define REG_POS(v) (((v) >> 5) << 4)152#define REG_POS(v) (((v) >> 5) << 4)
149 153
arch/x86/kvm/mmu.h+3−3
@@ -230,14 +230,14 @@ static inline bool kvm_shadow_root_allocated(struct kvm *kvm) · 229 unmodified lines
230}230}
231 231
232#ifdef CONFIG_X86_64232#ifdef CONFIG_X86_64
233static inline bool is_tdp_mmu_enabled(struct kvm *kvm) { return kvm->arch.tdp_mmu_enabled; }233extern bool tdp_mmu_enabled;
234#else234#else
235static inline bool is_tdp_mmu_enabled(struct kvm *kvm) { return false; }235#define tdp_mmu_enabled false
236#endif236#endif
237 237
238static inline bool kvm_memslots_have_rmaps(struct kvm *kvm)238static inline bool kvm_memslots_have_rmaps(struct kvm *kvm)
239{239{
240 return !is_tdp_mmu_enabled(kvm) || kvm_shadow_root_allocated(kvm);240 return !tdp_mmu_enabled || kvm_shadow_root_allocated(kvm);
241}241}
242 242
243static inline gfn_t gfn_to_index(gfn_t gfn, gfn_t base_gfn, int level)243static inline gfn_t gfn_to_index(gfn_t gfn, gfn_t base_gfn, int level)
arch/x86/kvm/mmu/mmu.c+203−117
@@ -14,6 +14,7 @@ · 13 unmodified lines
14 * Yaniv Kamay <yaniv@qumranet.com>14 * Yaniv Kamay <yaniv@qumranet.com>
15 * Avi Kivity <avi@qumranet.com>15 * Avi Kivity <avi@qumranet.com>
16 */16 */
17#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
17 18
18#include "irq.h"19#include "irq.h"
19#include "ioapic.h"20#include "ioapic.h"
@@ -43,6 +44,7 @@ · 23 unmodified lines
43#include <linux/uaccess.h>44#include <linux/uaccess.h>
44#include <linux/hash.h>45#include <linux/hash.h>
45#include <linux/kern_levels.h>46#include <linux/kern_levels.h>
47#include <linux/kstrtox.h>
46#include <linux/kthread.h>48#include <linux/kthread.h>
47 49
48#include <asm/page.h>50#include <asm/page.h>
@@ -99,6 +101,13 @@ module_param_named(flush_on_reuse, force_flush_and_sync_on_reuse, bool, 0644); · 50 unmodified lines
99 */101 */
100bool tdp_enabled = false;102bool tdp_enabled = false;
101 103
104static bool __ro_after_init tdp_mmu_allowed;
105
106#ifdef CONFIG_X86_64
107bool __read_mostly tdp_mmu_enabled = true;
108module_param_named(tdp_mmu, tdp_mmu_enabled, bool, 0444);
109#endif
110
102static int max_huge_page_level __read_mostly;111static int max_huge_page_level __read_mostly;
103static int tdp_root_level __read_mostly;112static int tdp_root_level __read_mostly;
104static int max_tdp_level __read_mostly;113static int max_tdp_level __read_mostly;
@@ -261,6 +270,17 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, · 156 unmodified lines
261 kvm_flush_remote_tlbs_with_range(kvm, &range);270 kvm_flush_remote_tlbs_with_range(kvm, &range);
262}271}
263 272
273static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index);
274
275/* Flush the range of guest memory mapped by the given SPTE. */
276static void kvm_flush_remote_tlbs_sptep(struct kvm *kvm, u64 *sptep)
277{
278 struct kvm_mmu_page *sp = sptep_to_sp(sptep);
279 gfn_t gfn = kvm_mmu_page_get_gfn(sp, spte_index(sptep));
280
281 kvm_flush_remote_tlbs_gfn(kvm, gfn, sp->role.level);
282}
283
264static void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn,284static void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn,
265 unsigned int access)285 unsigned int access)
266{286{
@@ -609,9 +629,14 @@ static bool mmu_spte_age(u64 *sptep) · 342 unmodified lines
609 return true;629 return true;
610}630}
611 631
632static inline bool is_tdp_mmu_active(struct kvm_vcpu *vcpu)
633{
634 return tdp_mmu_enabled && vcpu->arch.mmu->root_role.direct;
635}
636
612static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu)637static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu)
613{638{
614 if (is_tdp_mmu(vcpu->arch.mmu)) {639 if (is_tdp_mmu_active(vcpu)) {
615 kvm_tdp_mmu_walk_lockless_begin();640 kvm_tdp_mmu_walk_lockless_begin();
616 } else {641 } else {
617 /*642 /*
@@ -630,7 +655,7 @@ static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu) · 12 unmodified lines
630 655
631static void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu)656static void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu)
632{657{
633 if (is_tdp_mmu(vcpu->arch.mmu)) {658 if (is_tdp_mmu_active(vcpu)) {
634 kvm_tdp_mmu_walk_lockless_end();659 kvm_tdp_mmu_walk_lockless_end();
635 } else {660 } else {
636 /*661 /*
@@ -800,7 +825,7 @@ static void account_shadowed(struct kvm *kvm, struct kvm_mmu_page *sp) · 163 unmodified lines
800 kvm_mmu_gfn_disallow_lpage(slot, gfn);825 kvm_mmu_gfn_disallow_lpage(slot, gfn);
801 826
802 if (kvm_mmu_slot_gfn_write_protect(kvm, slot, gfn, PG_LEVEL_4K))827 if (kvm_mmu_slot_gfn_write_protect(kvm, slot, gfn, PG_LEVEL_4K))
803 kvm_flush_remote_tlbs_with_address(kvm, gfn, 1);828 kvm_flush_remote_tlbs_gfn(kvm, gfn, PG_LEVEL_4K);
804}829}
805 830
806void track_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp)831void track_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp)
@@ -1174,8 +1199,7 @@ static void drop_large_spte(struct kvm *kvm, u64 *sptep, bool flush) · 367 unmodified lines
1174 drop_spte(kvm, sptep);1199 drop_spte(kvm, sptep);
1175 1200
1176 if (flush)1201 if (flush)
1177 kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,1202 kvm_flush_remote_tlbs_sptep(kvm, sptep);
1178 KVM_PAGES_PER_HPAGE(sp->role.level));
1179}1203}
1180 1204
1181/*1205/*
@@ -1279,7 +1303,7 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, · 97 unmodified lines
1279{1303{
1280 struct kvm_rmap_head *rmap_head;1304 struct kvm_rmap_head *rmap_head;
1281 1305
1282 if (is_tdp_mmu_enabled(kvm))1306 if (tdp_mmu_enabled)
1283 kvm_tdp_mmu_clear_dirty_pt_masked(kvm, slot,1307 kvm_tdp_mmu_clear_dirty_pt_masked(kvm, slot,
1284 slot->base_gfn + gfn_offset, mask, true);1308 slot->base_gfn + gfn_offset, mask, true);
1285 1309
@@ -1312,7 +1336,7 @@ static void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm, · 26 unmodified lines
1312{1336{
1313 struct kvm_rmap_head *rmap_head;1337 struct kvm_rmap_head *rmap_head;
1314 1338
1315 if (is_tdp_mmu_enabled(kvm))1339 if (tdp_mmu_enabled)
1316 kvm_tdp_mmu_clear_dirty_pt_masked(kvm, slot,1340 kvm_tdp_mmu_clear_dirty_pt_masked(kvm, slot,
1317 slot->base_gfn + gfn_offset, mask, false);1341 slot->base_gfn + gfn_offset, mask, false);
1318 1342
@@ -1395,7 +1419,7 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, · 76 unmodified lines
1395 }1419 }
1396 }1420 }
1397 1421
1398 if (is_tdp_mmu_enabled(kvm))1422 if (tdp_mmu_enabled)
1399 write_protected |=1423 write_protected |=
1400 kvm_tdp_mmu_write_protect_gfn(kvm, slot, gfn, min_level);1424 kvm_tdp_mmu_write_protect_gfn(kvm, slot, gfn, min_level);
1401 1425
@@ -1456,7 +1480,7 @@ restart: · 54 unmodified lines
1456 }1480 }
1457 1481
1458 if (need_flush && kvm_available_flush_tlb_with_range()) {1482 if (need_flush && kvm_available_flush_tlb_with_range()) {
1459 kvm_flush_remote_tlbs_with_address(kvm, gfn, 1);1483 kvm_flush_remote_tlbs_gfn(kvm, gfn, level);
1460 return false;1484 return false;
1461 }1485 }
1462 1486
@@ -1558,7 +1582,7 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) · 95 unmodified lines
1558 if (kvm_memslots_have_rmaps(kvm))1582 if (kvm_memslots_have_rmaps(kvm))
1559 flush = kvm_handle_gfn_range(kvm, range, kvm_zap_rmap);1583 flush = kvm_handle_gfn_range(kvm, range, kvm_zap_rmap);
1560 1584
1561 if (is_tdp_mmu_enabled(kvm))1585 if (tdp_mmu_enabled)
1562 flush = kvm_tdp_mmu_unmap_gfn_range(kvm, range, flush);1586 flush = kvm_tdp_mmu_unmap_gfn_range(kvm, range, flush);
1563 1587
1564 return flush;1588 return flush;
@@ -1571,7 +1595,7 @@ bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · 6 unmodified lines
1571 if (kvm_memslots_have_rmaps(kvm))1595 if (kvm_memslots_have_rmaps(kvm))
1572 flush = kvm_handle_gfn_range(kvm, range, kvm_set_pte_rmap);1596 flush = kvm_handle_gfn_range(kvm, range, kvm_set_pte_rmap);
1573 1597
1574 if (is_tdp_mmu_enabled(kvm))1598 if (tdp_mmu_enabled)
1575 flush |= kvm_tdp_mmu_set_spte_gfn(kvm, range);1599 flush |= kvm_tdp_mmu_set_spte_gfn(kvm, range);
1576 1600
1577 return flush;1601 return flush;
@@ -1626,8 +1650,7 @@ static void __rmap_add(struct kvm *kvm, · 48 unmodified lines
1626 kvm->stat.max_mmu_rmap_size = rmap_count;1650 kvm->stat.max_mmu_rmap_size = rmap_count;
1627 if (rmap_count > RMAP_RECYCLE_THRESHOLD) {1651 if (rmap_count > RMAP_RECYCLE_THRESHOLD) {
1628 kvm_zap_all_rmap_sptes(kvm, rmap_head);1652 kvm_zap_all_rmap_sptes(kvm, rmap_head);
1629 kvm_flush_remote_tlbs_with_address(1653 kvm_flush_remote_tlbs_gfn(kvm, gfn, sp->role.level);
1630 kvm, sp->gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
1631 }1654 }
1632}1655}
1633 1656
@@ -1646,7 +1669,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · 12 unmodified lines
1646 if (kvm_memslots_have_rmaps(kvm))1669 if (kvm_memslots_have_rmaps(kvm))
1647 young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);1670 young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
1648 1671
1649 if (is_tdp_mmu_enabled(kvm))1672 if (tdp_mmu_enabled)
1650 young |= kvm_tdp_mmu_age_gfn_range(kvm, range);1673 young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
1651 1674
1652 return young;1675 return young;
@@ -1659,7 +1682,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) · 6 unmodified lines
1659 if (kvm_memslots_have_rmaps(kvm))1682 if (kvm_memslots_have_rmaps(kvm))
1660 young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap);1683 young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap);
1661 1684
1662 if (is_tdp_mmu_enabled(kvm))1685 if (tdp_mmu_enabled)
1663 young |= kvm_tdp_mmu_test_age_gfn(kvm, range);1686 young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
1664 1687
1665 return young;1688 return young;
@@ -1921,7 +1944,7 @@ static bool is_obsolete_sp(struct kvm *kvm, struct kvm_mmu_page *sp) · 255 unmodified lines
1921 return true;1944 return true;
1922 1945
1923 /* TDP MMU pages do not use the MMU generation. */1946 /* TDP MMU pages do not use the MMU generation. */
1924 return !sp->tdp_mmu_page &&1947 return !is_tdp_mmu_page(sp) &&
1925 unlikely(sp->mmu_valid_gen != kvm->arch.mmu_valid_gen);1948 unlikely(sp->mmu_valid_gen != kvm->arch.mmu_valid_gen);
1926}1949}
1927 1950
@@ -2355,7 +2378,16 @@ static void __link_shadow_page(struct kvm *kvm, · 427 unmodified lines
2355 2378
2356 mmu_page_add_parent_pte(cache, sp, sptep);2379 mmu_page_add_parent_pte(cache, sp, sptep);
2357 2380
2358 if (sp->unsync_children || sp->unsync)2381 /*
2382 * The non-direct sub-pagetable must be updated before linking. For
2383 * L1 sp, the pagetable is updated via kvm_sync_page() in
2384 * kvm_mmu_find_shadow_page() without write-protecting the gfn,
2385 * so sp->unsync can be true or false. For higher level non-direct
2386 * sp, the pagetable is updated/synced via mmu_sync_children() in
2387 * FNAME(fetch)(), so sp->unsync_children can only be false.
2388 * WARN_ON_ONCE() if anything happens unexpectedly.
2389 */
2390 if (WARN_ON_ONCE(sp->unsync_children) || sp->unsync)
2359 mark_unsync(sptep);2391 mark_unsync(sptep);
2360}2392}
2361 2393
@@ -2383,7 +2415,7 @@ static void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep, · 21 unmodified lines
2383 return;2415 return;
2384 2416
2385 drop_parent_pte(child, sptep);2417 drop_parent_pte(child, sptep);
2386 kvm_flush_remote_tlbs_with_address(vcpu->kvm, child->gfn, 1);2418 kvm_flush_remote_tlbs_sptep(vcpu->kvm, sptep);
2387 }2419 }
2388}2420}
2389 2421
@@ -2867,8 +2899,7 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, · 477 unmodified lines
2867 }2899 }
2868 2900
2869 if (flush)2901 if (flush)
2870 kvm_flush_remote_tlbs_with_address(vcpu->kvm, gfn,2902 kvm_flush_remote_tlbs_gfn(vcpu->kvm, gfn, level);
2871 KVM_PAGES_PER_HPAGE(level));
2872 2903
2873 pgprintk("%s: setting spte %llx\n", __func__, *sptep);2904 pgprintk("%s: setting spte %llx\n", __func__, *sptep);
2874 2905
@@ -3116,11 +3147,11 @@ void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_ · 241 unmodified lines
3116 !is_large_pte(spte) &&3147 !is_large_pte(spte) &&
3117 spte_to_child_sp(spte)->nx_huge_page_disallowed) {3148 spte_to_child_sp(spte)->nx_huge_page_disallowed) {
3118 /*3149 /*
3119 * A small SPTE exists for this pfn, but FNAME(fetch)3150 * A small SPTE exists for this pfn, but FNAME(fetch),
3120 * and __direct_map would like to create a large PTE3151 * direct_map(), or kvm_tdp_mmu_map() would like to create a
3121 * instead: just force them to go down another level,3152 * large PTE instead: just force them to go down another level,
3122 * patching back for them into pfn the next 9 bits of3153 * patching back for them into pfn the next 9 bits of the
3123 * the address.3154 * address.
3124 */3155 */
3125 u64 page_mask = KVM_PAGES_PER_HPAGE(cur_level) -3156 u64 page_mask = KVM_PAGES_PER_HPAGE(cur_level) -
3126 KVM_PAGES_PER_HPAGE(cur_level - 1);3157 KVM_PAGES_PER_HPAGE(cur_level - 1);
@@ -3129,7 +3160,7 @@ void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_ · 2 unmodified lines
3129 }3160 }
3130}3161}
3131 3162
3132static int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)3163static int direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
3133{3164{
3134 struct kvm_shadow_walk_iterator it;3165 struct kvm_shadow_walk_iterator it;
3135 struct kvm_mmu_page *sp;3166 struct kvm_mmu_page *sp;
@@ -3147,7 +3178,7 @@ static int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 11 unmodified lines
3147 if (fault->nx_huge_page_workaround_enabled)3178 if (fault->nx_huge_page_workaround_enabled)
3148 disallowed_hugepage_adjust(fault, *it.sptep, it.level);3179 disallowed_hugepage_adjust(fault, *it.sptep, it.level);
3149 3180
3150 base_gfn = fault->gfn & ~(KVM_PAGES_PER_HPAGE(it.level) - 1);3181 base_gfn = gfn_round_for_level(fault->gfn, it.level);
3151 if (it.level == fault->goal_level)3182 if (it.level == fault->goal_level)
3152 break;3183 break;
3153 3184
@@ -3173,14 +3204,16 @@ static int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 19 unmodified lines
3173 return ret;3204 return ret;
3174}3205}
3175 3206
3176static void kvm_send_hwpoison_signal(unsigned long address, struct task_struct *tsk)3207static void kvm_send_hwpoison_signal(struct kvm_memory_slot *slot, gfn_t gfn)
3177{3208{
3178 send_sig_mceerr(BUS_MCEERR_AR, (void __user *)address, PAGE_SHIFT, tsk);3209 unsigned long hva = gfn_to_hva_memslot(slot, gfn);
3210
3211 send_sig_mceerr(BUS_MCEERR_AR, (void __user *)hva, PAGE_SHIFT, current);
3179}3212}
3180 3213
3181static int kvm_handle_error_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)3214static int kvm_handle_error_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
3182{3215{
3183 if (is_sigpending_pfn(pfn)) {3216 if (is_sigpending_pfn(fault->pfn)) {
3184 kvm_handle_signal_exit(vcpu);3217 kvm_handle_signal_exit(vcpu);
3185 return -EINTR;3218 return -EINTR;
3186 }3219 }
@@ -3190,43 +3223,43 @@ static int kvm_handle_error_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · 3 unmodified lines
3190 * into the spte otherwise read access on readonly gfn also can3223 * into the spte otherwise read access on readonly gfn also can
3191 * caused mmio page fault and treat it as mmio access.3224 * caused mmio page fault and treat it as mmio access.
3192 */3225 */
3193 if (pfn == KVM_PFN_ERR_RO_FAULT)3226 if (fault->pfn == KVM_PFN_ERR_RO_FAULT)
3194 return RET_PF_EMULATE;3227 return RET_PF_EMULATE;
3195 3228
3196 if (pfn == KVM_PFN_ERR_HWPOISON) {3229 if (fault->pfn == KVM_PFN_ERR_HWPOISON) {
3197 kvm_send_hwpoison_signal(kvm_vcpu_gfn_to_hva(vcpu, gfn), current);3230 kvm_send_hwpoison_signal(fault->slot, fault->gfn);
3198 return RET_PF_RETRY;3231 return RET_PF_RETRY;
3199 }3232 }
3200 3233
3201 return -EFAULT;3234 return -EFAULT;
3202}3235}
3203 3236
3204static int handle_abnormal_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,3237static int kvm_handle_noslot_fault(struct kvm_vcpu *vcpu,
3205 unsigned int access)3238 struct kvm_page_fault *fault,
3239 unsigned int access)
3206{3240{
3207 /* The pfn is invalid, report the error! */3241 gva_t gva = fault->is_tdp ? 0 : fault->addr;
3208 if (unlikely(is_error_pfn(fault->pfn)))
3209 return kvm_handle_error_pfn(vcpu, fault->gfn, fault->pfn);
3210 3242
3211 if (unlikely(!fault->slot)) {3243 vcpu_cache_mmio_info(vcpu, gva, fault->gfn,
3212 gva_t gva = fault->is_tdp ? 0 : fault->addr;3244 access & shadow_mmio_access_mask);
3213 3245
3214 vcpu_cache_mmio_info(vcpu, gva, fault->gfn,3246 /*
3215 access & shadow_mmio_access_mask);3247 * If MMIO caching is disabled, emulate immediately without
3216 /*3248 * touching the shadow page tables as attempting to install an
3217 * If MMIO caching is disabled, emulate immediately without3249 * MMIO SPTE will just be an expensive nop.
3218 * touching the shadow page tables as attempting to install an3250 */
3219 * MMIO SPTE will just be an expensive nop. Do not cache MMIO3251 if (unlikely(!enable_mmio_caching))
3220 * whose gfn is greater than host.MAXPHYADDR, any guest that3252 return RET_PF_EMULATE;
3221 * generates such gfns is running nested and is being tricked3253
3222 * by L0 userspace (you can observe gfn > L1.MAXPHYADDR if3254 /*
3223 * and only if L1's MAXPHYADDR is inaccurate with respect to3255 * Do not create an MMIO SPTE for a gfn greater than host.MAXPHYADDR,
3224 * the hardware's).3256 * any guest that generates such gfns is running nested and is being
3225 */3257 * tricked by L0 userspace (you can observe gfn > L1.MAXPHYADDR if and
3226 if (unlikely(!enable_mmio_caching) ||3258 * only if L1's MAXPHYADDR is inaccurate with respect to the
3227 unlikely(fault->gfn > kvm_mmu_max_gfn()))3259 * hardware's).
3228 return RET_PF_EMULATE;3260 */
3229 }3261 if (unlikely(fault->gfn > kvm_mmu_max_gfn()))
3262 return RET_PF_EMULATE;
3230 3263
3231 return RET_PF_CONTINUE;3264 return RET_PF_CONTINUE;
3232}3265}
@@ -3350,7 +3383,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 117 unmodified lines
3350 do {3383 do {
3351 u64 new_spte;3384 u64 new_spte;
3352 3385
3353 if (is_tdp_mmu(vcpu->arch.mmu))3386 if (tdp_mmu_enabled)
3354 sptep = kvm_tdp_mmu_fast_pf_get_last_sptep(vcpu, fault->addr, &spte);3387 sptep = kvm_tdp_mmu_fast_pf_get_last_sptep(vcpu, fault->addr, &spte);
3355 else3388 else
3356 sptep = fast_pf_get_last_sptep(vcpu, fault->addr, &spte);3389 sptep = fast_pf_get_last_sptep(vcpu, fault->addr, &spte);
@@ -3433,8 +3466,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 76 unmodified lines
3433 }3466 }
3434 3467
3435 if (++retry_count > 4) {3468 if (++retry_count > 4) {
3436 printk_once(KERN_WARNING3469 pr_warn_once("Fast #PF retrying more than 4 times.\n");
3437 "kvm: Fast #PF retrying more than 4 times.\n");
3438 break;3470 break;
3439 }3471 }
3440 3472
@@ -3596,7 +3628,7 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) · 155 unmodified lines
3596 if (r < 0)3628 if (r < 0)
3597 goto out_unlock;3629 goto out_unlock;
3598 3630
3599 if (is_tdp_mmu_enabled(vcpu->kvm)) {3631 if (tdp_mmu_enabled) {
3600 root = kvm_tdp_mmu_get_vcpu_root_hpa(vcpu);3632 root = kvm_tdp_mmu_get_vcpu_root_hpa(vcpu);
3601 mmu->root.hpa = root;3633 mmu->root.hpa = root;
3602 } else if (shadow_root_level >= PT64_ROOT_4LEVEL) {3634 } else if (shadow_root_level >= PT64_ROOT_4LEVEL) {
@@ -4026,7 +4058,7 @@ static bool get_mmio_spte(struct kvm_vcpu *vcpu, u64 addr, u64 *sptep) · 423 unmodified lines
4026 4058
4027 walk_shadow_page_lockless_begin(vcpu);4059 walk_shadow_page_lockless_begin(vcpu);
4028 4060
4029 if (is_tdp_mmu(vcpu->arch.mmu))4061 if (is_tdp_mmu_active(vcpu))
4030 leaf = kvm_tdp_mmu_get_walk(vcpu, addr, sptes, &root);4062 leaf = kvm_tdp_mmu_get_walk(vcpu, addr, sptes, &root);
4031 else4063 else
4032 leaf = get_walk(vcpu, addr, sptes, &root);4064 leaf = get_walk(vcpu, addr, sptes, &root);
@@ -4174,7 +4206,7 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) · 141 unmodified lines
4174 kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true);4206 kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true);
4175}4207}
4176 4208
4177static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)4209static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
4178{4210{
4179 struct kvm_memory_slot *slot = fault->slot;4211 struct kvm_memory_slot *slot = fault->slot;
4180 bool async;4212 bool async;
@@ -4235,12 +4267,33 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 54 unmodified lines
4235 return RET_PF_CONTINUE;4267 return RET_PF_CONTINUE;
4236}4268}
4237 4269
4270static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,
4271 unsigned int access)
4272{
4273 int ret;
4274
4275 fault->mmu_seq = vcpu->kvm->mmu_invalidate_seq;
4276 smp_rmb();
4277
4278 ret = __kvm_faultin_pfn(vcpu, fault);
4279 if (ret != RET_PF_CONTINUE)
4280 return ret;
4281
4282 if (unlikely(is_error_pfn(fault->pfn)))
4283 return kvm_handle_error_pfn(vcpu, fault);
4284
4285 if (unlikely(!fault->slot))
4286 return kvm_handle_noslot_fault(vcpu, fault, access);
4287
4288 return RET_PF_CONTINUE;
4289}
4290
4238/*4291/*
4239 * Returns true if the page fault is stale and needs to be retried, i.e. if the4292 * Returns true if the page fault is stale and needs to be retried, i.e. if the
4240 * root was invalidated by a memslot update or a relevant mmu_notifier fired.4293 * root was invalidated by a memslot update or a relevant mmu_notifier fired.
4241 */4294 */
4242static bool is_page_fault_stale(struct kvm_vcpu *vcpu,4295static bool is_page_fault_stale(struct kvm_vcpu *vcpu,
4243 struct kvm_page_fault *fault, int mmu_seq)4296 struct kvm_page_fault *fault)
4244{4297{
4245 struct kvm_mmu_page *sp = to_shadow_page(vcpu->arch.mmu->root.hpa);4298 struct kvm_mmu_page *sp = to_shadow_page(vcpu->arch.mmu->root.hpa);
4246 4299
@@ -4260,19 +4313,13 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu, · 13 unmodified lines
4260 return true;4313 return true;
4261 4314
4262 return fault->slot &&4315 return fault->slot &&
4263 mmu_invalidate_retry_hva(vcpu->kvm, mmu_seq, fault->hva);4316 mmu_invalidate_retry_hva(vcpu->kvm, fault->mmu_seq, fault->hva);
4264}4317}
4265 4318
4266static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)4319static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
4267{4320{
4268 bool is_tdp_mmu_fault = is_tdp_mmu(vcpu->arch.mmu);
4269
4270 unsigned long mmu_seq;
4271 int r;4321 int r;
4272 4322
4273 fault->gfn = fault->addr >> PAGE_SHIFT;
4274 fault->slot = kvm_vcpu_gfn_to_memslot(vcpu, fault->gfn);
4275
4276 if (page_fault_handle_page_track(vcpu, fault))4323 if (page_fault_handle_page_track(vcpu, fault))
4277 return RET_PF_EMULATE;4324 return RET_PF_EMULATE;
4278 4325
@@ -4284,41 +4331,24 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault · 5 unmodified lines
4284 if (r)4331 if (r)
4285 return r;4332 return r;
4286 4333
4287 mmu_seq = vcpu->kvm->mmu_invalidate_seq;4334 r = kvm_faultin_pfn(vcpu, fault, ACC_ALL);
4288 smp_rmb();
4289
4290 r = kvm_faultin_pfn(vcpu, fault);
4291 if (r != RET_PF_CONTINUE)
4292 return r;
4293
4294 r = handle_abnormal_pfn(vcpu, fault, ACC_ALL);
4295 if (r != RET_PF_CONTINUE)4335 if (r != RET_PF_CONTINUE)
4296 return r;4336 return r;
4297 4337
4298 r = RET_PF_RETRY;4338 r = RET_PF_RETRY;
4339 write_lock(&vcpu->kvm->mmu_lock);
4299 4340
4300 if (is_tdp_mmu_fault)4341 if (is_page_fault_stale(vcpu, fault))
4301 read_lock(&vcpu->kvm->mmu_lock);4342 goto out_unlock;
4302 else
4303 write_lock(&vcpu->kvm->mmu_lock);
4304 4343
4305 if (is_page_fault_stale(vcpu, fault, mmu_seq))4344 r = make_mmu_pages_available(vcpu);
4345 if (r)
4306 goto out_unlock;4346 goto out_unlock;
4307 4347
4308 if (is_tdp_mmu_fault) {4348 r = direct_map(vcpu, fault);
4309 r = kvm_tdp_mmu_map(vcpu, fault);
4310 } else {
4311 r = make_mmu_pages_available(vcpu);
4312 if (r)
4313 goto out_unlock;
4314 r = __direct_map(vcpu, fault);
4315 }
4316 4349
4317out_unlock:4350out_unlock:
4318 if (is_tdp_mmu_fault)4351 write_unlock(&vcpu->kvm->mmu_lock);
4319 read_unlock(&vcpu->kvm->mmu_lock);
4320 else
4321 write_unlock(&vcpu->kvm->mmu_lock);
4322 kvm_release_pfn_clean(fault->pfn);4352 kvm_release_pfn_clean(fault->pfn);
4323 return r;4353 return r;
4324}4354}
@@ -4366,6 +4396,42 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, · 41 unmodified lines
4366}4396}
4367EXPORT_SYMBOL_GPL(kvm_handle_page_fault);4397EXPORT_SYMBOL_GPL(kvm_handle_page_fault);
4368 4398
4399#ifdef CONFIG_X86_64
4400static int kvm_tdp_mmu_page_fault(struct kvm_vcpu *vcpu,
4401 struct kvm_page_fault *fault)
4402{
4403 int r;
4404
4405 if (page_fault_handle_page_track(vcpu, fault))
4406 return RET_PF_EMULATE;
4407
4408 r = fast_page_fault(vcpu, fault);
4409 if (r != RET_PF_INVALID)
4410 return r;
4411
4412 r = mmu_topup_memory_caches(vcpu, false);
4413 if (r)
4414 return r;
4415
4416 r = kvm_faultin_pfn(vcpu, fault, ACC_ALL);
4417 if (r != RET_PF_CONTINUE)
4418 return r;
4419
4420 r = RET_PF_RETRY;
4421 read_lock(&vcpu->kvm->mmu_lock);
4422
4423 if (is_page_fault_stale(vcpu, fault))
4424 goto out_unlock;
4425
4426 r = kvm_tdp_mmu_map(vcpu, fault);
4427
4428out_unlock:
4429 read_unlock(&vcpu->kvm->mmu_lock);
4430 kvm_release_pfn_clean(fault->pfn);
4431 return r;
4432}
4433#endif
4434
4369int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)4435int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
4370{4436{
4371 /*4437 /*
@@ -4383,13 +4449,19 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) · 11 unmodified lines
4383 if (shadow_memtype_mask && kvm_arch_has_noncoherent_dma(vcpu->kvm)) {4449 if (shadow_memtype_mask && kvm_arch_has_noncoherent_dma(vcpu->kvm)) {
4384 for ( ; fault->max_level > PG_LEVEL_4K; --fault->max_level) {4450 for ( ; fault->max_level > PG_LEVEL_4K; --fault->max_level) {
4385 int page_num = KVM_PAGES_PER_HPAGE(fault->max_level);4451 int page_num = KVM_PAGES_PER_HPAGE(fault->max_level);
4386 gfn_t base = (fault->addr >> PAGE_SHIFT) & ~(page_num - 1);4452 gfn_t base = gfn_round_for_level(fault->gfn,
4453 fault->max_level);
4387 4454
4388 if (kvm_mtrr_check_gfn_range_consistency(vcpu, base, page_num))4455 if (kvm_mtrr_check_gfn_range_consistency(vcpu, base, page_num))
4389 break;4456 break;
4390 }4457 }
4391 }4458 }
4392 4459
4460#ifdef CONFIG_X86_64
4461 if (tdp_mmu_enabled)
4462 return kvm_tdp_mmu_page_fault(vcpu, fault);
4463#endif
4464
4393 return direct_page_fault(vcpu, fault);4465 return direct_page_fault(vcpu, fault);
4394}4466}
4395 4467
@@ -4494,10 +4566,12 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) · 98 unmodified lines
4494 struct kvm_mmu *mmu = vcpu->arch.mmu;4566 struct kvm_mmu *mmu = vcpu->arch.mmu;
4495 union kvm_mmu_page_role new_role = mmu->root_role;4567 union kvm_mmu_page_role new_role = mmu->root_role;
4496 4568
4497 if (!fast_pgd_switch(vcpu->kvm, mmu, new_pgd, new_role)) {4569 /*
4498 /* kvm_mmu_ensure_valid_pgd will set up a new root. */4570 * Return immediately if no usable root was found, kvm_mmu_reload()
4571 * will establish a valid root prior to the next VM-Enter.
4572 */
4573 if (!fast_pgd_switch(vcpu->kvm, mmu, new_pgd, new_role))
4499 return;4574 return;
4500 }
4501 4575
4502 /*4576 /*
4503 * It's possible that the cached previous root page is obsolete because4577 * It's possible that the cached previous root page is obsolete because
@@ -5719,6 +5793,9 @@ void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level, · 1215 unmodified lines
5719 tdp_root_level = tdp_forced_root_level;5793 tdp_root_level = tdp_forced_root_level;
5720 max_tdp_level = tdp_max_root_level;5794 max_tdp_level = tdp_max_root_level;
5721 5795
5796#ifdef CONFIG_X86_64
5797 tdp_mmu_enabled = tdp_mmu_allowed && tdp_enabled;
5798#endif
5722 /*5799 /*
5723 * max_huge_page_level reflects KVM's MMU capabilities irrespective5800 * max_huge_page_level reflects KVM's MMU capabilities irrespective
5724 * of kernel support, e.g. KVM may be capable of using 1GB pages when5801 * of kernel support, e.g. KVM may be capable of using 1GB pages when
@@ -5966,7 +6043,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) · 241 unmodified lines
5966 * write and in the same critical section as making the reload request,6043 * write and in the same critical section as making the reload request,
5967 * e.g. before kvm_zap_obsolete_pages() could drop mmu_lock and yield.6044 * e.g. before kvm_zap_obsolete_pages() could drop mmu_lock and yield.
5968 */6045 */
5969 if (is_tdp_mmu_enabled(kvm))6046 if (tdp_mmu_enabled)
5970 kvm_tdp_mmu_invalidate_all_roots(kvm);6047 kvm_tdp_mmu_invalidate_all_roots(kvm);
5971 6048
5972 /*6049 /*
@@ -5991,7 +6068,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) · 18 unmodified lines
5991 * Deferring the zap until the final reference to the root is put would6068 * Deferring the zap until the final reference to the root is put would
5992 * lead to use-after-free.6069 * lead to use-after-free.
5993 */6070 */
5994 if (is_tdp_mmu_enabled(kvm))6071 if (tdp_mmu_enabled)
5995 kvm_tdp_mmu_zap_invalidated_roots(kvm);6072 kvm_tdp_mmu_zap_invalidated_roots(kvm);
5996}6073}
5997 6074
@@ -6017,9 +6094,11 @@ int kvm_mmu_init_vm(struct kvm *kvm) · 19 unmodified lines
6017 INIT_LIST_HEAD(&kvm->arch.possible_nx_huge_pages);6094 INIT_LIST_HEAD(&kvm->arch.possible_nx_huge_pages);
6018 spin_lock_init(&kvm->arch.mmu_unsync_pages_lock);6095 spin_lock_init(&kvm->arch.mmu_unsync_pages_lock);
6019 6096
6020 r = kvm_mmu_init_tdp_mmu(kvm);6097 if (tdp_mmu_enabled) {
6021 if (r < 0)6098 r = kvm_mmu_init_tdp_mmu(kvm);
6022 return r;6099 if (r < 0)
6100 return r;
6101 }
6023 6102
6024 node->track_write = kvm_mmu_pte_write;6103 node->track_write = kvm_mmu_pte_write;
6025 node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;6104 node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot;
@@ -6049,7 +6128,8 @@ void kvm_mmu_uninit_vm(struct kvm *kvm) · 23 unmodified lines
6049 6128
6050 kvm_page_track_unregister_notifier(kvm, node);6129 kvm_page_track_unregister_notifier(kvm, node);
6051 6130
6052 kvm_mmu_uninit_tdp_mmu(kvm);6131 if (tdp_mmu_enabled)
6132 kvm_mmu_uninit_tdp_mmu(kvm);
6053 6133
6054 mmu_free_vm_memory_caches(kvm);6134 mmu_free_vm_memory_caches(kvm);
6055}6135}
@@ -6103,7 +6183,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) · 47 unmodified lines
6103 6183
6104 flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end);6184 flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end);
6105 6185
6106 if (is_tdp_mmu_enabled(kvm)) {6186 if (tdp_mmu_enabled) {
6107 for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++)6187 for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++)
6108 flush = kvm_tdp_mmu_zap_leafs(kvm, i, gfn_start,6188 flush = kvm_tdp_mmu_zap_leafs(kvm, i, gfn_start,
6109 gfn_end, true, flush);6189 gfn_end, true, flush);
@@ -6136,7 +6216,7 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, · 26 unmodified lines
6136 write_unlock(&kvm->mmu_lock);6216 write_unlock(&kvm->mmu_lock);
6137 }6217 }
6138 6218
6139 if (is_tdp_mmu_enabled(kvm)) {6219 if (tdp_mmu_enabled) {
6140 read_lock(&kvm->mmu_lock);6220 read_lock(&kvm->mmu_lock);
6141 kvm_tdp_mmu_wrprot_slot(kvm, memslot, start_level);6221 kvm_tdp_mmu_wrprot_slot(kvm, memslot, start_level);
6142 read_unlock(&kvm->mmu_lock);6222 read_unlock(&kvm->mmu_lock);
@@ -6379,7 +6459,7 @@ void kvm_mmu_try_split_huge_pages(struct kvm *kvm, · 236 unmodified lines
6379 u64 start, u64 end,6459 u64 start, u64 end,
6380 int target_level)6460 int target_level)
6381{6461{
6382 if (!is_tdp_mmu_enabled(kvm))6462 if (!tdp_mmu_enabled)
6383 return;6463 return;
6384 6464
6385 if (kvm_memslots_have_rmaps(kvm))6465 if (kvm_memslots_have_rmaps(kvm))
@@ -6400,7 +6480,7 @@ void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, · 14 unmodified lines
6400 u64 start = memslot->base_gfn;6480 u64 start = memslot->base_gfn;
6401 u64 end = start + memslot->npages;6481 u64 end = start + memslot->npages;
6402 6482
6403 if (!is_tdp_mmu_enabled(kvm))6483 if (!tdp_mmu_enabled)
6404 return;6484 return;
6405 6485
6406 if (kvm_memslots_have_rmaps(kvm)) {6486 if (kvm_memslots_have_rmaps(kvm)) {
@@ -6450,8 +6530,7 @@ restart: · 43 unmodified lines
6450 kvm_zap_one_rmap_spte(kvm, rmap_head, sptep);6530 kvm_zap_one_rmap_spte(kvm, rmap_head, sptep);
6451 6531
6452 if (kvm_available_flush_tlb_with_range())6532 if (kvm_available_flush_tlb_with_range())
6453 kvm_flush_remote_tlbs_with_address(kvm, sp->gfn,6533 kvm_flush_remote_tlbs_sptep(kvm, sptep);
6454 KVM_PAGES_PER_HPAGE(sp->role.level));
6455 else6534 else
6456 need_tlb_flush = 1;6535 need_tlb_flush = 1;
6457 6536
@@ -6483,7 +6562,7 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, · 25 unmodified lines
6483 write_unlock(&kvm->mmu_lock);6562 write_unlock(&kvm->mmu_lock);
6484 }6563 }
6485 6564
6486 if (is_tdp_mmu_enabled(kvm)) {6565 if (tdp_mmu_enabled) {
6487 read_lock(&kvm->mmu_lock);6566 read_lock(&kvm->mmu_lock);
6488 kvm_tdp_mmu_zap_collapsible_sptes(kvm, slot);6567 kvm_tdp_mmu_zap_collapsible_sptes(kvm, slot);
6489 read_unlock(&kvm->mmu_lock);6568 read_unlock(&kvm->mmu_lock);
@@ -6518,7 +6597,7 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, · 28 unmodified lines
6518 write_unlock(&kvm->mmu_lock);6597 write_unlock(&kvm->mmu_lock);
6519 }6598 }
6520 6599
6521 if (is_tdp_mmu_enabled(kvm)) {6600 if (tdp_mmu_enabled) {
6522 read_lock(&kvm->mmu_lock);6601 read_lock(&kvm->mmu_lock);
6523 kvm_tdp_mmu_clear_dirty_slot(kvm, memslot);6602 kvm_tdp_mmu_clear_dirty_slot(kvm, memslot);
6524 read_unlock(&kvm->mmu_lock);6603 read_unlock(&kvm->mmu_lock);
@@ -6553,7 +6632,7 @@ restart: · 28 unmodified lines
6553 6632
6554 kvm_mmu_commit_zap_page(kvm, &invalid_list);6633 kvm_mmu_commit_zap_page(kvm, &invalid_list);
6555 6634
6556 if (is_tdp_mmu_enabled(kvm))6635 if (tdp_mmu_enabled)
6557 kvm_tdp_mmu_zap_all(kvm);6636 kvm_tdp_mmu_zap_all(kvm);
6558 6637
6559 write_unlock(&kvm->mmu_lock);6638 write_unlock(&kvm->mmu_lock);
@@ -6579,7 +6658,7 @@ void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen) · 19 unmodified lines
6579 * zap all shadow pages.6658 * zap all shadow pages.
6580 */6659 */
6581 if (unlikely(gen == 0)) {6660 if (unlikely(gen == 0)) {
6582 kvm_debug_ratelimited("kvm: zapping shadow pages for mmio generation wraparound\n");6661 kvm_debug_ratelimited("zapping shadow pages for mmio generation wraparound\n");
6583 kvm_mmu_zap_all_fast(kvm);6662 kvm_mmu_zap_all_fast(kvm);
6584 }6663 }
6585}6664}
@@ -6684,7 +6763,7 @@ static int set_nx_huge_pages(const char *val, const struct kernel_param *kp) · 98 unmodified lines
6684 new_val = 1;6763 new_val = 1;
6685 else if (sysfs_streq(val, "auto"))6764 else if (sysfs_streq(val, "auto"))
6686 new_val = get_nx_auto_mode();6765 new_val = get_nx_auto_mode();
6687 else if (strtobool(val, &new_val) < 0)6766 else if (kstrtobool(val, &new_val) < 0)
6688 return -EINVAL;6767 return -EINVAL;
6689 6768
6690 __set_nx_huge_pages(new_val);6769 __set_nx_huge_pages(new_val);
@@ -6718,6 +6797,13 @@ void __init kvm_mmu_x86_module_init(void) · 27 unmodified lines
6718 if (nx_huge_pages == -1)6797 if (nx_huge_pages == -1)
6719 __set_nx_huge_pages(get_nx_auto_mode());6798 __set_nx_huge_pages(get_nx_auto_mode());
6720 6799
6800 /*
6801 * Snapshot userspace's desire to enable the TDP MMU. Whether or not the
6802 * TDP MMU is actually enabled is determined in kvm_configure_mmu()
6803 * when the vendor module is loaded.
6804 */
6805 tdp_mmu_allowed = tdp_mmu_enabled;
6806
6721 kvm_mmu_spte_module_init();6807 kvm_mmu_spte_module_init();
6722}6808}
6723 6809
arch/x86/kvm/mmu/mmu_internal.h+21−1
@@ -156,6 +156,11 @@ static inline bool kvm_mmu_page_ad_need_write_protect(struct kvm_mmu_page *sp) · 155 unmodified lines
156 return kvm_x86_ops.cpu_dirty_log_size && sp->role.guest_mode;156 return kvm_x86_ops.cpu_dirty_log_size && sp->role.guest_mode;
157}157}
158 158
159static inline gfn_t gfn_round_for_level(gfn_t gfn, int level)
160{
161 return gfn & -KVM_PAGES_PER_HPAGE(level);
162}
163
159int mmu_try_to_unsync_pages(struct kvm *kvm, const struct kvm_memory_slot *slot,164int mmu_try_to_unsync_pages(struct kvm *kvm, const struct kvm_memory_slot *slot,
160 gfn_t gfn, bool can_unsync, bool prefetch);165 gfn_t gfn, bool can_unsync, bool prefetch);
161 166
@@ -164,8 +169,17 @@ void kvm_mmu_gfn_allow_lpage(const struct kvm_memory_slot *slot, gfn_t gfn); · 2 unmodified lines
164bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,169bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,
165 struct kvm_memory_slot *slot, u64 gfn,170 struct kvm_memory_slot *slot, u64 gfn,
166 int min_level);171 int min_level);
172
167void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,173void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,
168 u64 start_gfn, u64 pages);174 u64 start_gfn, u64 pages);
175
176/* Flush the given page (huge or not) of guest memory. */
177static inline void kvm_flush_remote_tlbs_gfn(struct kvm *kvm, gfn_t gfn, int level)
178{
179 kvm_flush_remote_tlbs_with_address(kvm, gfn_round_for_level(gfn, level),
180 KVM_PAGES_PER_HPAGE(level));
181}
182
169unsigned int pte_list_count(struct kvm_rmap_head *rmap_head);183unsigned int pte_list_count(struct kvm_rmap_head *rmap_head);
170 184
171extern int nx_huge_pages;185extern int nx_huge_pages;
@@ -199,7 +213,7 @@ struct kvm_page_fault { · 27 unmodified lines
199 213
200 /*214 /*
201 * Maximum page size that can be created for this fault; input to215 * Maximum page size that can be created for this fault; input to
202 * FNAME(fetch), __direct_map and kvm_tdp_mmu_map.216 * FNAME(fetch), direct_map() and kvm_tdp_mmu_map().
203 */217 */
204 u8 max_level;218 u8 max_level;
205 219
@@ -222,6 +236,7 @@ struct kvm_page_fault { · 16 unmodified lines
222 struct kvm_memory_slot *slot;236 struct kvm_memory_slot *slot;
223 237
224 /* Outputs of kvm_faultin_pfn. */238 /* Outputs of kvm_faultin_pfn. */
239 unsigned long mmu_seq;
225 kvm_pfn_t pfn;240 kvm_pfn_t pfn;
226 hva_t hva;241 hva_t hva;
227 bool map_writable;242 bool map_writable;
@@ -279,6 +294,11 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, · 51 unmodified lines
279 };294 };
280 int r;295 int r;
281 296
297 if (vcpu->arch.mmu->root_role.direct) {
298 fault.gfn = fault.addr >> PAGE_SHIFT;
299 fault.slot = kvm_vcpu_gfn_to_memslot(vcpu, fault.gfn);
300 }
301
282 /*302 /*
283 * Async #PF "faults", a.k.a. prefetch faults, are not faults from the303 * Async #PF "faults", a.k.a. prefetch faults, are not faults from the
284 * guest perspective and have already been counted at the time of the304 * guest perspective and have already been counted at the time of the
arch/x86/kvm/mmu/page_track.c+1−0
@@ -10,6 +10,7 @@ · 9 unmodified lines
10 * Author:10 * Author:
11 * Xiao Guangrong <guangrong.xiao@linux.intel.com>11 * Xiao Guangrong <guangrong.xiao@linux.intel.com>
12 */12 */
13#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
13 14
14#include <linux/kvm_host.h>15#include <linux/kvm_host.h>
15#include <linux/rculist.h>16#include <linux/rculist.h>
arch/x86/kvm/mmu/paging_tmpl.h+7−18
@@ -642,12 +642,12 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · 641 unmodified lines
642 if (WARN_ON(!VALID_PAGE(vcpu->arch.mmu->root.hpa)))642 if (WARN_ON(!VALID_PAGE(vcpu->arch.mmu->root.hpa)))
643 goto out_gpte_changed;643 goto out_gpte_changed;
644 644
645 for (shadow_walk_init(&it, vcpu, fault->addr);645 for_each_shadow_entry(vcpu, fault->addr, it) {
646 shadow_walk_okay(&it) && it.level > gw->level;
647 shadow_walk_next(&it)) {
648 gfn_t table_gfn;646 gfn_t table_gfn;
649 647
650 clear_sp_write_flooding_count(it.sptep);648 clear_sp_write_flooding_count(it.sptep);
649 if (it.level == gw->level)
650 break;
651 651
652 table_gfn = gw->table_gfn[it.level - 2];652 table_gfn = gw->table_gfn[it.level - 2];
653 access = gw->pt_access[it.level - 2];653 access = gw->pt_access[it.level - 2];
@@ -692,8 +692,6 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · 38 unmodified lines
692 trace_kvm_mmu_spte_requested(fault);692 trace_kvm_mmu_spte_requested(fault);
693 693
694 for (; shadow_walk_okay(&it); shadow_walk_next(&it)) {694 for (; shadow_walk_okay(&it); shadow_walk_next(&it)) {
695 clear_sp_write_flooding_count(it.sptep);
696
697 /*695 /*
698 * We cannot overwrite existing page tables with an NX696 * We cannot overwrite existing page tables with an NX
699 * large page, as the leaf could be executable.697 * large page, as the leaf could be executable.
@@ -701,7 +699,7 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · 1 unmodified line
701 if (fault->nx_huge_page_workaround_enabled)699 if (fault->nx_huge_page_workaround_enabled)
702 disallowed_hugepage_adjust(fault, *it.sptep, it.level);700 disallowed_hugepage_adjust(fault, *it.sptep, it.level);
703 701
704 base_gfn = fault->gfn & ~(KVM_PAGES_PER_HPAGE(it.level) - 1);702 base_gfn = gfn_round_for_level(fault->gfn, it.level);
705 if (it.level == fault->goal_level)703 if (it.level == fault->goal_level)
706 break;704 break;
707 705
@@ -791,7 +789,6 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault · 83 unmodified lines
791{789{
792 struct guest_walker walker;790 struct guest_walker walker;
793 int r;791 int r;
794 unsigned long mmu_seq;
795 bool is_self_change_mapping;792 bool is_self_change_mapping;
796 793
797 pgprintk("%s: addr %lx err %x\n", __func__, fault->addr, fault->error_code);794 pgprintk("%s: addr %lx err %x\n", __func__, fault->addr, fault->error_code);
@@ -838,14 +835,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault · 40 unmodified lines
838 else835 else
839 fault->max_level = walker.level;836 fault->max_level = walker.level;
840 837
841 mmu_seq = vcpu->kvm->mmu_invalidate_seq;838 r = kvm_faultin_pfn(vcpu, fault, walker.pte_access);
842 smp_rmb();
843
844 r = kvm_faultin_pfn(vcpu, fault);
845 if (r != RET_PF_CONTINUE)
846 return r;
847
848 r = handle_abnormal_pfn(vcpu, fault, walker.pte_access);
849 if (r != RET_PF_CONTINUE)839 if (r != RET_PF_CONTINUE)
850 return r;840 return r;
851 841
@@ -871,7 +861,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault · 19 unmodified lines
871 r = RET_PF_RETRY;861 r = RET_PF_RETRY;
872 write_lock(&vcpu->kvm->mmu_lock);862 write_lock(&vcpu->kvm->mmu_lock);
873 863
874 if (is_page_fault_stale(vcpu, fault, mmu_seq))864 if (is_page_fault_stale(vcpu, fault))
875 goto out_unlock;865 goto out_unlock;
876 866
877 r = make_mmu_pages_available(vcpu);867 r = make_mmu_pages_available(vcpu);
@@ -937,8 +927,7 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa) · 59 unmodified lines
937 927
938 mmu_page_zap_pte(vcpu->kvm, sp, sptep, NULL);928 mmu_page_zap_pte(vcpu->kvm, sp, sptep, NULL);
939 if (is_shadow_present_pte(old_spte))929 if (is_shadow_present_pte(old_spte))
940 kvm_flush_remote_tlbs_with_address(vcpu->kvm,930 kvm_flush_remote_tlbs_sptep(vcpu->kvm, sptep);
941 sp->gfn, KVM_PAGES_PER_HPAGE(sp->role.level));
942 931
943 if (!rmap_can_add(vcpu))932 if (!rmap_can_add(vcpu))
944 break;933 break;
arch/x86/kvm/mmu/spte.c+5−5
@@ -7,7 +7,7 @@ · 6 unmodified lines
7 * Copyright (C) 2006 Qumranet, Inc.7 * Copyright (C) 2006 Qumranet, Inc.
8 * Copyright 2020 Red Hat, Inc. and/or its affiliates.8 * Copyright 2020 Red Hat, Inc. and/or its affiliates.
9 */9 */
10 10#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
11 11
12#include <linux/kvm_host.h>12#include <linux/kvm_host.h>
13#include "mmu.h"13#include "mmu.h"
@@ -147,9 +147,9 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, · 133 unmodified lines
147 WARN_ON_ONCE(!pte_access && !shadow_present_mask);147 WARN_ON_ONCE(!pte_access && !shadow_present_mask);
148 148
149 if (sp->role.ad_disabled)149 if (sp->role.ad_disabled)
150 spte |= SPTE_TDP_AD_DISABLED_MASK;150 spte |= SPTE_TDP_AD_DISABLED;
151 else if (kvm_mmu_page_ad_need_write_protect(sp))151 else if (kvm_mmu_page_ad_need_write_protect(sp))
152 spte |= SPTE_TDP_AD_WRPROT_ONLY_MASK;152 spte |= SPTE_TDP_AD_WRPROT_ONLY;
153 153
154 /*154 /*
155 * For the EPT case, shadow_present_mask is 0 if hardware155 * For the EPT case, shadow_present_mask is 0 if hardware
@@ -317,7 +317,7 @@ u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled) · 161 unmodified lines
317 shadow_user_mask | shadow_x_mask | shadow_me_value;317 shadow_user_mask | shadow_x_mask | shadow_me_value;
318 318
319 if (ad_disabled)319 if (ad_disabled)
320 spte |= SPTE_TDP_AD_DISABLED_MASK;320 spte |= SPTE_TDP_AD_DISABLED;
321 else321 else
322 spte |= shadow_accessed_mask;322 spte |= shadow_accessed_mask;
323 323
@@ -352,7 +352,7 @@ u64 mark_spte_for_access_track(u64 spte) · 28 unmodified lines
352 352
353 WARN_ONCE(spte & (SHADOW_ACC_TRACK_SAVED_BITS_MASK <<353 WARN_ONCE(spte & (SHADOW_ACC_TRACK_SAVED_BITS_MASK <<
354 SHADOW_ACC_TRACK_SAVED_BITS_SHIFT),354 SHADOW_ACC_TRACK_SAVED_BITS_SHIFT),
355 "kvm: Access Tracking saved bit locations are not zero\n");355 "Access Tracking saved bit locations are not zero\n");
356 356
357 spte |= (spte & SHADOW_ACC_TRACK_SAVED_BITS_MASK) <<357 spte |= (spte & SHADOW_ACC_TRACK_SAVED_BITS_MASK) <<
358 SHADOW_ACC_TRACK_SAVED_BITS_SHIFT;358 SHADOW_ACC_TRACK_SAVED_BITS_SHIFT;
arch/x86/kvm/mmu/spte.h+10−10
@@ -28,10 +28,10 @@ · 27 unmodified lines
28 */28 */
29#define SPTE_TDP_AD_SHIFT 5229#define SPTE_TDP_AD_SHIFT 52
30#define SPTE_TDP_AD_MASK (3ULL << SPTE_TDP_AD_SHIFT)30#define SPTE_TDP_AD_MASK (3ULL << SPTE_TDP_AD_SHIFT)
31#define SPTE_TDP_AD_ENABLED_MASK (0ULL << SPTE_TDP_AD_SHIFT)31#define SPTE_TDP_AD_ENABLED (0ULL << SPTE_TDP_AD_SHIFT)
32#define SPTE_TDP_AD_DISABLED_MASK (1ULL << SPTE_TDP_AD_SHIFT)32#define SPTE_TDP_AD_DISABLED (1ULL << SPTE_TDP_AD_SHIFT)
33#define SPTE_TDP_AD_WRPROT_ONLY_MASK (2ULL << SPTE_TDP_AD_SHIFT)33#define SPTE_TDP_AD_WRPROT_ONLY (2ULL << SPTE_TDP_AD_SHIFT)
34static_assert(SPTE_TDP_AD_ENABLED_MASK == 0);34static_assert(SPTE_TDP_AD_ENABLED == 0);
35 35
36#ifdef CONFIG_DYNAMIC_PHYSICAL_MASK36#ifdef CONFIG_DYNAMIC_PHYSICAL_MASK
37#define SPTE_BASE_ADDR_MASK (physical_mask & ~(u64)(PAGE_SIZE-1))37#define SPTE_BASE_ADDR_MASK (physical_mask & ~(u64)(PAGE_SIZE-1))
@@ -164,7 +164,7 @@ extern u64 __read_mostly shadow_me_value; · 126 unmodified lines
164extern u64 __read_mostly shadow_me_mask;164extern u64 __read_mostly shadow_me_mask;
165 165
166/*166/*
167 * SPTEs in MMUs without A/D bits are marked with SPTE_TDP_AD_DISABLED_MASK;167 * SPTEs in MMUs without A/D bits are marked with SPTE_TDP_AD_DISABLED;
168 * shadow_acc_track_mask is the set of bits to be cleared in non-accessed168 * shadow_acc_track_mask is the set of bits to be cleared in non-accessed
169 * pages.169 * pages.
170 */170 */
@@ -266,18 +266,18 @@ static inline bool sp_ad_disabled(struct kvm_mmu_page *sp) · 95 unmodified lines
266static inline bool spte_ad_enabled(u64 spte)266static inline bool spte_ad_enabled(u64 spte)
267{267{
268 MMU_WARN_ON(!is_shadow_present_pte(spte));268 MMU_WARN_ON(!is_shadow_present_pte(spte));
269 return (spte & SPTE_TDP_AD_MASK) != SPTE_TDP_AD_DISABLED_MASK;269 return (spte & SPTE_TDP_AD_MASK) != SPTE_TDP_AD_DISABLED;
270}270}
271 271
272static inline bool spte_ad_need_write_protect(u64 spte)272static inline bool spte_ad_need_write_protect(u64 spte)
273{273{
274 MMU_WARN_ON(!is_shadow_present_pte(spte));274 MMU_WARN_ON(!is_shadow_present_pte(spte));
275 /*275 /*
276 * This is benign for non-TDP SPTEs as SPTE_TDP_AD_ENABLED_MASK is '0',276 * This is benign for non-TDP SPTEs as SPTE_TDP_AD_ENABLED is '0',
277 * and non-TDP SPTEs will never set these bits. Optimize for 64-bit277 * and non-TDP SPTEs will never set these bits. Optimize for 64-bit
278 * TDP and do the A/D type check unconditionally.278 * TDP and do the A/D type check unconditionally.
279 */279 */
280 return (spte & SPTE_TDP_AD_MASK) != SPTE_TDP_AD_ENABLED_MASK;280 return (spte & SPTE_TDP_AD_MASK) != SPTE_TDP_AD_ENABLED;
281}281}
282 282
283static inline u64 spte_shadow_accessed_mask(u64 spte)283static inline u64 spte_shadow_accessed_mask(u64 spte)
@@ -435,11 +435,11 @@ static inline void check_spte_writable_invariants(u64 spte) · 151 unmodified lines
435{435{
436 if (spte & shadow_mmu_writable_mask)436 if (spte & shadow_mmu_writable_mask)
437 WARN_ONCE(!(spte & shadow_host_writable_mask),437 WARN_ONCE(!(spte & shadow_host_writable_mask),
438 "kvm: MMU-writable SPTE is not Host-writable: %llx",438 KBUILD_MODNAME ": MMU-writable SPTE is not Host-writable: %llx",
439 spte);439 spte);
440 else440 else
441 WARN_ONCE(is_writable_pte(spte),441 WARN_ONCE(is_writable_pte(spte),
442 "kvm: Writable SPTE is not MMU-writable: %llx", spte);442 KBUILD_MODNAME ": Writable SPTE is not MMU-writable: %llx", spte);
443}443}
444 444
445static inline bool is_mmu_writable_spte(u64 spte)445static inline bool is_mmu_writable_spte(u64 spte)
arch/x86/kvm/mmu/tdp_iter.c+4−8
@@ -1,4 +1,5 @@
1// SPDX-License-Identifier: GPL-2.01// SPDX-License-Identifier: GPL-2.0
2#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
2 3
3#include "mmu_internal.h"4#include "mmu_internal.h"
4#include "tdp_iter.h"5#include "tdp_iter.h"
@@ -15,11 +16,6 @@ static void tdp_iter_refresh_sptep(struct tdp_iter *iter) · 10 unmodified lines
15 iter->old_spte = kvm_tdp_mmu_read_spte(iter->sptep);16 iter->old_spte = kvm_tdp_mmu_read_spte(iter->sptep);
16}17}
17 18
18static gfn_t round_gfn_for_level(gfn_t gfn, int level)
19{
20 return gfn & -KVM_PAGES_PER_HPAGE(level);
21}
22
23/*19/*
24 * Return the TDP iterator to the root PT and allow it to continue its20 * Return the TDP iterator to the root PT and allow it to continue its
25 * traversal over the paging structure from there.21 * traversal over the paging structure from there.
@@ -30,7 +26,7 @@ void tdp_iter_restart(struct tdp_iter *iter) · 4 unmodified lines
30 iter->yielded_gfn = iter->next_last_level_gfn;26 iter->yielded_gfn = iter->next_last_level_gfn;
31 iter->level = iter->root_level;27 iter->level = iter->root_level;
32 28
33 iter->gfn = round_gfn_for_level(iter->next_last_level_gfn, iter->level);29 iter->gfn = gfn_round_for_level(iter->next_last_level_gfn, iter->level);
34 tdp_iter_refresh_sptep(iter);30 tdp_iter_refresh_sptep(iter);
35 31
36 iter->valid = true;32 iter->valid = true;
@@ -97,7 +93,7 @@ static bool try_step_down(struct tdp_iter *iter) · 60 unmodified lines
97 93
98 iter->level--;94 iter->level--;
99 iter->pt_path[iter->level - 1] = child_pt;95 iter->pt_path[iter->level - 1] = child_pt;
100 iter->gfn = round_gfn_for_level(iter->next_last_level_gfn, iter->level);96 iter->gfn = gfn_round_for_level(iter->next_last_level_gfn, iter->level);
101 tdp_iter_refresh_sptep(iter);97 tdp_iter_refresh_sptep(iter);
102 98
103 return true;99 return true;
@@ -139,7 +135,7 @@ static bool try_step_up(struct tdp_iter *iter) · 35 unmodified lines
139 return false;135 return false;
140 136
141 iter->level++;137 iter->level++;
142 iter->gfn = round_gfn_for_level(iter->gfn, iter->level);138 iter->gfn = gfn_round_for_level(iter->gfn, iter->level);
143 tdp_iter_refresh_sptep(iter);139 tdp_iter_refresh_sptep(iter);
144 140
145 return true;141 return true;
arch/x86/kvm/mmu/tdp_mmu.c+4−16
@@ -1,4 +1,5 @@
1// SPDX-License-Identifier: GPL-2.01// SPDX-License-Identifier: GPL-2.0
2#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
2 3
3#include "mmu.h"4#include "mmu.h"
4#include "mmu_internal.h"5#include "mmu_internal.h"
@@ -10,23 +11,15 @@ · 5 unmodified lines
10#include <asm/cmpxchg.h>11#include <asm/cmpxchg.h>
11#include <trace/events/kvm.h>12#include <trace/events/kvm.h>
12 13
13static bool __read_mostly tdp_mmu_enabled = true;
14module_param_named(tdp_mmu, tdp_mmu_enabled, bool, 0644);
15
16/* Initializes the TDP MMU for the VM, if enabled. */14/* Initializes the TDP MMU for the VM, if enabled. */
17int kvm_mmu_init_tdp_mmu(struct kvm *kvm)15int kvm_mmu_init_tdp_mmu(struct kvm *kvm)
18{16{
19 struct workqueue_struct *wq;17 struct workqueue_struct *wq;
20 18
21 if (!tdp_enabled || !READ_ONCE(tdp_mmu_enabled))
22 return 0;
23
24 wq = alloc_workqueue("kvm", WQ_UNBOUND|WQ_MEM_RECLAIM|WQ_CPU_INTENSIVE, 0);19 wq = alloc_workqueue("kvm", WQ_UNBOUND|WQ_MEM_RECLAIM|WQ_CPU_INTENSIVE, 0);
25 if (!wq)20 if (!wq)
26 return -ENOMEM;21 return -ENOMEM;
27 22
28 /* This should not be changed for the lifetime of the VM. */
29 kvm->arch.tdp_mmu_enabled = true;
30 INIT_LIST_HEAD(&kvm->arch.tdp_mmu_roots);23 INIT_LIST_HEAD(&kvm->arch.tdp_mmu_roots);
31 spin_lock_init(&kvm->arch.tdp_mmu_pages_lock);24 spin_lock_init(&kvm->arch.tdp_mmu_pages_lock);
32 kvm->arch.tdp_mmu_zap_wq = wq;25 kvm->arch.tdp_mmu_zap_wq = wq;
@@ -47,9 +40,6 @@ static __always_inline bool kvm_lockdep_assert_mmu_lock_held(struct kvm *kvm, · 14 unmodified lines
47 40
48void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)41void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
49{42{
50 if (!kvm->arch.tdp_mmu_enabled)
51 return;
52
53 /* Also waits for any queued work items. */43 /* Also waits for any queued work items. */
54 destroy_workqueue(kvm->arch.tdp_mmu_zap_wq);44 destroy_workqueue(kvm->arch.tdp_mmu_zap_wq);
55 45
@@ -144,7 +134,7 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root, · 88 unmodified lines
144 if (!refcount_dec_and_test(&root->tdp_mmu_root_count))134 if (!refcount_dec_and_test(&root->tdp_mmu_root_count))
145 return;135 return;
146 136
147 WARN_ON(!root->tdp_mmu_page);137 WARN_ON(!is_tdp_mmu_page(root));
148 138
149 /*139 /*
150 * The root now has refcount=0. It is valid, but readers already140 * The root now has refcount=0. It is valid, but readers already
@@ -690,8 +680,7 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm, · 539 unmodified lines
690 if (ret)680 if (ret)
691 return ret;681 return ret;
692 682
693 kvm_flush_remote_tlbs_with_address(kvm, iter->gfn,683 kvm_flush_remote_tlbs_gfn(kvm, iter->gfn, iter->level);
694 KVM_PAGES_PER_HPAGE(iter->level));
695 684
696 /*685 /*
697 * No other thread can overwrite the removed SPTE as they must either686 * No other thread can overwrite the removed SPTE as they must either
@@ -1090,8 +1079,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, · 392 unmodified lines
1090 return RET_PF_RETRY;1079 return RET_PF_RETRY;
1091 else if (is_shadow_present_pte(iter->old_spte) &&1080 else if (is_shadow_present_pte(iter->old_spte) &&
1092 !is_last_spte(iter->old_spte, iter->level))1081 !is_last_spte(iter->old_spte, iter->level))
1093 kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn,1082 kvm_flush_remote_tlbs_gfn(vcpu->kvm, iter->gfn, iter->level);
1094 KVM_PAGES_PER_HPAGE(iter->level + 1));
1095 1083
1096 /*1084 /*
1097 * If the page fault was caused by a write but the page is write1085 * If the page fault was caused by a write but the page is write
arch/x86/kvm/mmu/tdp_mmu.h+3−22
@@ -7,6 +7,9 @@ · 6 unmodified lines
7 7
8#include "spte.h"8#include "spte.h"
9 9
10int kvm_mmu_init_tdp_mmu(struct kvm *kvm);
11void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm);
12
10hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);13hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu);
11 14
12__must_check static inline bool kvm_tdp_mmu_get_root(struct kvm_mmu_page *root)15__must_check static inline bool kvm_tdp_mmu_get_root(struct kvm_mmu_page *root)
@@ -68,31 +71,9 @@ u64 *kvm_tdp_mmu_fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, u64 addr, · 55 unmodified lines
68 u64 *spte);71 u64 *spte);
69 72
70#ifdef CONFIG_X86_6473#ifdef CONFIG_X86_64
71int kvm_mmu_init_tdp_mmu(struct kvm *kvm);
72void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm);
73static inline bool is_tdp_mmu_page(struct kvm_mmu_page *sp) { return sp->tdp_mmu_page; }74static inline bool is_tdp_mmu_page(struct kvm_mmu_page *sp) { return sp->tdp_mmu_page; }
74
75static inline bool is_tdp_mmu(struct kvm_mmu *mmu)
76{
77 struct kvm_mmu_page *sp;
78 hpa_t hpa = mmu->root.hpa;
79
80 if (WARN_ON(!VALID_PAGE(hpa)))
81 return false;
82
83 /*
84 * A NULL shadow page is legal when shadowing a non-paging guest with
85 * PAE paging, as the MMU will be direct with root_hpa pointing at the
86 * pae_root page, not a shadow page.
87 */
88 sp = to_shadow_page(hpa);
89 return sp && is_tdp_mmu_page(sp) && sp->root_count;
90}
91#else75#else
92static inline int kvm_mmu_init_tdp_mmu(struct kvm *kvm) { return 0; }
93static inline void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) {}
94static inline bool is_tdp_mmu_page(struct kvm_mmu_page *sp) { return false; }76static inline bool is_tdp_mmu_page(struct kvm_mmu_page *sp) { return false; }
95static inline bool is_tdp_mmu(struct kvm_mmu *mmu) { return false; }
96#endif77#endif
97 78
98#endif /* __KVM_X86_MMU_TDP_MMU_H */79#endif /* __KVM_X86_MMU_TDP_MMU_H */
arch/x86/kvm/mtrr.c+1−0
@@ -13,6 +13,7 @@ · 12 unmodified lines
13 * Paolo Bonzini <pbonzini@redhat.com>13 * Paolo Bonzini <pbonzini@redhat.com>
14 * Xiao Guangrong <guangrong.xiao@linux.intel.com>14 * Xiao Guangrong <guangrong.xiao@linux.intel.com>
15 */15 */
16#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
16 17
17#include <linux/kvm_host.h>18#include <linux/kvm_host.h>
18#include <asm/mtrr.h>19#include <asm/mtrr.h>
arch/x86/kvm/pmu.c+238−52
@@ -9,6 +9,7 @@ · 8 unmodified lines
9 * Gleb Natapov <gleb@redhat.com>9 * Gleb Natapov <gleb@redhat.com>
10 * Wei Huang <wei@redhat.com>10 * Wei Huang <wei@redhat.com>
11 */11 */
12#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
12 13
13#include <linux/types.h>14#include <linux/types.h>
14#include <linux/kvm_host.h>15#include <linux/kvm_host.h>
@@ -28,9 +29,18 @@ · 13 unmodified lines
28struct x86_pmu_capability __read_mostly kvm_pmu_cap;29struct x86_pmu_capability __read_mostly kvm_pmu_cap;
29EXPORT_SYMBOL_GPL(kvm_pmu_cap);30EXPORT_SYMBOL_GPL(kvm_pmu_cap);
30 31
31static const struct x86_cpu_id vmx_icl_pebs_cpu[] = {32/* Precise Distribution of Instructions Retired (PDIR) */
33static const struct x86_cpu_id vmx_pebs_pdir_cpu[] = {
32 X86_MATCH_INTEL_FAM6_MODEL(ICELAKE_D, NULL),34 X86_MATCH_INTEL_FAM6_MODEL(ICELAKE_D, NULL),
33 X86_MATCH_INTEL_FAM6_MODEL(ICELAKE_X, NULL),35 X86_MATCH_INTEL_FAM6_MODEL(ICELAKE_X, NULL),
36 /* Instruction-Accurate PDIR (PDIR++) */
37 X86_MATCH_INTEL_FAM6_MODEL(SAPPHIRERAPIDS_X, NULL),
38 {}
39};
40
41/* Precise Distribution (PDist) */
42static const struct x86_cpu_id vmx_pebs_pdist_cpu[] = {
43 X86_MATCH_INTEL_FAM6_MODEL(SAPPHIRERAPIDS_X, NULL),
34 {}44 {}
35};45};
36 46
@@ -155,6 +165,28 @@ static void kvm_perf_overflow(struct perf_event *perf_event, · 118 unmodified lines
155 kvm_make_request(KVM_REQ_PMU, pmc->vcpu);165 kvm_make_request(KVM_REQ_PMU, pmc->vcpu);
156}166}
157 167
168static u64 pmc_get_pebs_precise_level(struct kvm_pmc *pmc)
169{
170 /*
171 * For some model specific pebs counters with special capabilities
172 * (PDIR, PDIR++, PDIST), KVM needs to raise the event precise
173 * level to the maximum value (currently 3, backwards compatible)
174 * so that the perf subsystem would assign specific hardware counter
175 * with that capability for vPMC.
176 */
177 if ((pmc->idx == 0 && x86_match_cpu(vmx_pebs_pdist_cpu)) ||
178 (pmc->idx == 32 && x86_match_cpu(vmx_pebs_pdir_cpu)))
179 return 3;
180
181 /*
182 * The non-zero precision level of guest event makes the ordinary
183 * guest event becomes a guest PEBS event and triggers the host
184 * PEBS PMI handler to determine whether the PEBS overflow PMI
185 * comes from the host counters or the guest.
186 */
187 return 1;
188}
189
158static int pmc_reprogram_counter(struct kvm_pmc *pmc, u32 type, u64 config,190static int pmc_reprogram_counter(struct kvm_pmc *pmc, u32 type, u64 config,
159 bool exclude_user, bool exclude_kernel,191 bool exclude_user, bool exclude_kernel,
160 bool intr)192 bool intr)
@@ -186,22 +218,12 @@ static int pmc_reprogram_counter(struct kvm_pmc *pmc, u32 type, u64 config, · 25 unmodified lines
186 }218 }
187 if (pebs) {219 if (pebs) {
188 /*220 /*
189 * The non-zero precision level of guest event makes the ordinary
190 * guest event becomes a guest PEBS event and triggers the host
191 * PEBS PMI handler to determine whether the PEBS overflow PMI
192 * comes from the host counters or the guest.
193 *
194 * For most PEBS hardware events, the difference in the software221 * For most PEBS hardware events, the difference in the software
195 * precision levels of guest and host PEBS events will not affect222 * precision levels of guest and host PEBS events will not affect
196 * the accuracy of the PEBS profiling result, because the "event IP"223 * the accuracy of the PEBS profiling result, because the "event IP"
197 * in the PEBS record is calibrated on the guest side.224 * in the PEBS record is calibrated on the guest side.
198 *
199 * On Icelake everything is fine. Other hardware (GLC+, TNT+) that
200 * could possibly care here is unsupported and needs changes.
201 */225 */
202 attr.precise_ip = 1;226 attr.precise_ip = pmc_get_pebs_precise_level(pmc);
203 if (x86_match_cpu(vmx_icl_pebs_cpu) && pmc->idx == 32)
204 attr.precise_ip = 3;
205 }227 }
206 228
207 event = perf_event_create_kernel_counter(&attr, -1, current,229 event = perf_event_create_kernel_counter(&attr, -1, current,
@@ -254,48 +276,128 @@ static bool pmc_resume_counter(struct kvm_pmc *pmc) · 46 unmodified lines
254 return true;276 return true;
255}277}
256 278
257static int cmp_u64(const void *pa, const void *pb)279static int filter_cmp(const void *pa, const void *pb, u64 mask)
258{280{
259 u64 a = *(u64 *)pa;281 u64 a = *(u64 *)pa & mask;
260 u64 b = *(u64 *)pb;282 u64 b = *(u64 *)pb & mask;
261 283
262 return (a > b) - (a < b);284 return (a > b) - (a < b);
263}285}
264 286
287
288static int filter_sort_cmp(const void *pa, const void *pb)
289{
290 return filter_cmp(pa, pb, (KVM_PMU_MASKED_ENTRY_EVENT_SELECT |
291 KVM_PMU_MASKED_ENTRY_EXCLUDE));
292}
293
294/*
295 * For the event filter, searching is done on the 'includes' list and
296 * 'excludes' list separately rather than on the 'events' list (which
297 * has both). As a result the exclude bit can be ignored.
298 */
299static int filter_event_cmp(const void *pa, const void *pb)
300{
301 return filter_cmp(pa, pb, (KVM_PMU_MASKED_ENTRY_EVENT_SELECT));
302}
303
304static int find_filter_index(u64 *events, u64 nevents, u64 key)
305{
306 u64 *fe = bsearch(&key, events, nevents, sizeof(events[0]),
307 filter_event_cmp);
308
309 if (!fe)
310 return -1;
311
312 return fe - events;
313}
314
315static bool is_filter_entry_match(u64 filter_event, u64 umask)
316{
317 u64 mask = filter_event >> (KVM_PMU_MASKED_ENTRY_UMASK_MASK_SHIFT - 8);
318 u64 match = filter_event & KVM_PMU_MASKED_ENTRY_UMASK_MATCH;
319
320 BUILD_BUG_ON((KVM_PMU_ENCODE_MASKED_ENTRY(0, 0xff, 0, false) >>
321 (KVM_PMU_MASKED_ENTRY_UMASK_MASK_SHIFT - 8)) !=
322 ARCH_PERFMON_EVENTSEL_UMASK);
323
324 return (umask & mask) == match;
325}
326
327static bool filter_contains_match(u64 *events, u64 nevents, u64 eventsel)
328{
329 u64 event_select = eventsel & kvm_pmu_ops.EVENTSEL_EVENT;
330 u64 umask = eventsel & ARCH_PERFMON_EVENTSEL_UMASK;
331 int i, index;
332
333 index = find_filter_index(events, nevents, event_select);
334 if (index < 0)
335 return false;
336
337 /*
338 * Entries are sorted by the event select. Walk the list in both
339 * directions to process all entries with the targeted event select.
340 */
341 for (i = index; i < nevents; i++) {
342 if (filter_event_cmp(&events[i], &event_select))
343 break;
344
345 if (is_filter_entry_match(events[i], umask))
346 return true;
347 }
348
349 for (i = index - 1; i >= 0; i--) {
350 if (filter_event_cmp(&events[i], &event_select))
351 break;
352
353 if (is_filter_entry_match(events[i], umask))
354 return true;
355 }
356
357 return false;
358}
359
360static bool is_gp_event_allowed(struct kvm_x86_pmu_event_filter *f,
361 u64 eventsel)
362{
363 if (filter_contains_match(f->includes, f->nr_includes, eventsel) &&
364 !filter_contains_match(f->excludes, f->nr_excludes, eventsel))
365 return f->action == KVM_PMU_EVENT_ALLOW;
366
367 return f->action == KVM_PMU_EVENT_DENY;
368}
369
370static bool is_fixed_event_allowed(struct kvm_x86_pmu_event_filter *filter,
371 int idx)
372{
373 int fixed_idx = idx - INTEL_PMC_IDX_FIXED;
374
375 if (filter->action == KVM_PMU_EVENT_DENY &&
376 test_bit(fixed_idx, (ulong *)&filter->fixed_counter_bitmap))
377 return false;
378 if (filter->action == KVM_PMU_EVENT_ALLOW &&
379 !test_bit(fixed_idx, (ulong *)&filter->fixed_counter_bitmap))
380 return false;
381
382 return true;
383}
384
265static bool check_pmu_event_filter(struct kvm_pmc *pmc)385static bool check_pmu_event_filter(struct kvm_pmc *pmc)
266{386{
267 struct kvm_pmu_event_filter *filter;387 struct kvm_x86_pmu_event_filter *filter;
268 struct kvm *kvm = pmc->vcpu->kvm;388 struct kvm *kvm = pmc->vcpu->kvm;
269 bool allow_event = true;
270 __u64 key;
271 int idx;
272 389
273 if (!static_call(kvm_x86_pmu_hw_event_available)(pmc))390 if (!static_call(kvm_x86_pmu_hw_event_available)(pmc))
274 return false;391 return false;
275 392
276 filter = srcu_dereference(kvm->arch.pmu_event_filter, &kvm->srcu);393 filter = srcu_dereference(kvm->arch.pmu_event_filter, &kvm->srcu);
277 if (!filter)394 if (!filter)
278 goto out;395 return true;
279 396
280 if (pmc_is_gp(pmc)) {397 if (pmc_is_gp(pmc))
281 key = pmc->eventsel & AMD64_RAW_EVENT_MASK_NB;398 return is_gp_event_allowed(filter, pmc->eventsel);
282 if (bsearch(&key, filter->events, filter->nevents,
283 sizeof(__u64), cmp_u64))
284 allow_event = filter->action == KVM_PMU_EVENT_ALLOW;
285 else
286 allow_event = filter->action == KVM_PMU_EVENT_DENY;
287 } else {
288 idx = pmc->idx - INTEL_PMC_IDX_FIXED;
289 if (filter->action == KVM_PMU_EVENT_DENY &&
290 test_bit(idx, (ulong *)&filter->fixed_counter_bitmap))
291 allow_event = false;
292 if (filter->action == KVM_PMU_EVENT_ALLOW &&
293 !test_bit(idx, (ulong *)&filter->fixed_counter_bitmap))
294 allow_event = false;
295 }
296 399
297out:400 return is_fixed_event_allowed(filter, pmc->idx);
298 return allow_event;
299}401}
300 402
301static void reprogram_counter(struct kvm_pmc *pmc)403static void reprogram_counter(struct kvm_pmc *pmc)
@@ -592,47 +694,133 @@ void kvm_pmu_trigger_event(struct kvm_vcpu *vcpu, u64 perf_hw_id) · 290 unmodified lines
592}694}
593EXPORT_SYMBOL_GPL(kvm_pmu_trigger_event);695EXPORT_SYMBOL_GPL(kvm_pmu_trigger_event);
594 696
697static bool is_masked_filter_valid(const struct kvm_x86_pmu_event_filter *filter)
698{
699 u64 mask = kvm_pmu_ops.EVENTSEL_EVENT |
700 KVM_PMU_MASKED_ENTRY_UMASK_MASK |
701 KVM_PMU_MASKED_ENTRY_UMASK_MATCH |
702 KVM_PMU_MASKED_ENTRY_EXCLUDE;
703 int i;
704
705 for (i = 0; i < filter->nevents; i++) {
706 if (filter->events[i] & ~mask)
707 return false;
708 }
709
710 return true;
711}
712
713static void convert_to_masked_filter(struct kvm_x86_pmu_event_filter *filter)
714{
715 int i, j;
716
717 for (i = 0, j = 0; i < filter->nevents; i++) {
718 /*
719 * Skip events that are impossible to match against a guest
720 * event. When filtering, only the event select + unit mask
721 * of the guest event is used. To maintain backwards
722 * compatibility, impossible filters can't be rejected :-(
723 */
724 if (filter->events[i] & ~(kvm_pmu_ops.EVENTSEL_EVENT |
725 ARCH_PERFMON_EVENTSEL_UMASK))
726 continue;
727 /*
728 * Convert userspace events to a common in-kernel event so
729 * only one code path is needed to support both events. For
730 * the in-kernel events use masked events because they are
731 * flexible enough to handle both cases. To convert to masked
732 * events all that's needed is to add an "all ones" umask_mask,
733 * (unmasked filter events don't support EXCLUDE).
734 */
735 filter->events[j++] = filter->events[i] |
736 (0xFFULL << KVM_PMU_MASKED_ENTRY_UMASK_MASK_SHIFT);
737 }
738
739 filter->nevents = j;
740}
741
742static int prepare_filter_lists(struct kvm_x86_pmu_event_filter *filter)
743{
744 int i;
745
746 if (!(filter->flags & KVM_PMU_EVENT_FLAG_MASKED_EVENTS))
747 convert_to_masked_filter(filter);
748 else if (!is_masked_filter_valid(filter))
749 return -EINVAL;
750
751 /*
752 * Sort entries by event select and includes vs. excludes so that all
753 * entries for a given event select can be processed efficiently during
754 * filtering. The EXCLUDE flag uses a more significant bit than the
755 * event select, and so the sorted list is also effectively split into
756 * includes and excludes sub-lists.
757 */
758 sort(&filter->events, filter->nevents, sizeof(filter->events[0]),
759 filter_sort_cmp, NULL);
760
761 i = filter->nevents;
762 /* Find the first EXCLUDE event (only supported for masked events). */
763 if (filter->flags & KVM_PMU_EVENT_FLAG_MASKED_EVENTS) {
764 for (i = 0; i < filter->nevents; i++) {
765 if (filter->events[i] & KVM_PMU_MASKED_ENTRY_EXCLUDE)
766 break;
767 }
768 }
769
770 filter->nr_includes = i;
771 filter->nr_excludes = filter->nevents - filter->nr_includes;
772 filter->includes = filter->events;
773 filter->excludes = filter->events + filter->nr_includes;
774
775 return 0;
776}
777
595int kvm_vm_ioctl_set_pmu_event_filter(struct kvm *kvm, void __user *argp)778int kvm_vm_ioctl_set_pmu_event_filter(struct kvm *kvm, void __user *argp)
596{779{
597 struct kvm_pmu_event_filter tmp, *filter;780 struct kvm_pmu_event_filter __user *user_filter = argp;
781 struct kvm_x86_pmu_event_filter *filter;
782 struct kvm_pmu_event_filter tmp;
598 struct kvm_vcpu *vcpu;783 struct kvm_vcpu *vcpu;
599 unsigned long i;784 unsigned long i;
600 size_t size;785 size_t size;
601 int r;786 int r;
602 787
603 if (copy_from_user(&tmp, argp, sizeof(tmp)))788 if (copy_from_user(&tmp, user_filter, sizeof(tmp)))
604 return -EFAULT;789 return -EFAULT;
605 790
606 if (tmp.action != KVM_PMU_EVENT_ALLOW &&791 if (tmp.action != KVM_PMU_EVENT_ALLOW &&
607 tmp.action != KVM_PMU_EVENT_DENY)792 tmp.action != KVM_PMU_EVENT_DENY)
608 return -EINVAL;793 return -EINVAL;
609 794
610 if (tmp.flags != 0)795 if (tmp.flags & ~KVM_PMU_EVENT_FLAGS_VALID_MASK)
611 return -EINVAL;796 return -EINVAL;
612 797
613 if (tmp.nevents > KVM_PMU_EVENT_FILTER_MAX_EVENTS)798 if (tmp.nevents > KVM_PMU_EVENT_FILTER_MAX_EVENTS)
614 return -E2BIG;799 return -E2BIG;
615 800
616 size = struct_size(filter, events, tmp.nevents);801 size = struct_size(filter, events, tmp.nevents);
617 filter = kmalloc(size, GFP_KERNEL_ACCOUNT);802 filter = kzalloc(size, GFP_KERNEL_ACCOUNT);
618 if (!filter)803 if (!filter)
619 return -ENOMEM;804 return -ENOMEM;
620 805
806 filter->action = tmp.action;
807 filter->nevents = tmp.nevents;
808 filter->fixed_counter_bitmap = tmp.fixed_counter_bitmap;
809 filter->flags = tmp.flags;
810
621 r = -EFAULT;811 r = -EFAULT;
622 if (copy_from_user(filter, argp, size))812 if (copy_from_user(filter->events, user_filter->events,
813 sizeof(filter->events[0]) * filter->nevents))
623 goto cleanup;814 goto cleanup;
624 815
625 /* Ensure nevents can't be changed between the user copies. */816 r = prepare_filter_lists(filter);
626 *filter = tmp;817 if (r)
627 818 goto cleanup;
628 /*
629 * Sort the in-kernel list so that we can search it with bsearch.
630 */
631 sort(&filter->events, filter->nevents, sizeof(__u64), cmp_u64, NULL);
632 819
633 mutex_lock(&kvm->lock);820 mutex_lock(&kvm->lock);
634 filter = rcu_replace_pointer(kvm->arch.pmu_event_filter, filter,821 filter = rcu_replace_pointer(kvm->arch.pmu_event_filter, filter,
635 mutex_is_locked(&kvm->lock));822 mutex_is_locked(&kvm->lock));
823 mutex_unlock(&kvm->lock);
636 synchronize_srcu_expedited(&kvm->srcu);824 synchronize_srcu_expedited(&kvm->srcu);
637 825
638 BUILD_BUG_ON(sizeof(((struct kvm_pmu *)0)->reprogram_pmi) >826 BUILD_BUG_ON(sizeof(((struct kvm_pmu *)0)->reprogram_pmi) >
@@ -643,8 +831,6 @@ int kvm_vm_ioctl_set_pmu_event_filter(struct kvm *kvm, void __user *argp) · 4 unmodified lines
643 831
644 kvm_make_all_cpus_request(kvm, KVM_REQ_PMU);832 kvm_make_all_cpus_request(kvm, KVM_REQ_PMU);
645 833
646 mutex_unlock(&kvm->lock);
647
648 r = 0;834 r = 0;
649cleanup:835cleanup:
650 kfree(filter);836 kfree(filter);
arch/x86/kvm/pmu.h+6−7
@@ -18,12 +18,6 @@ · 17 unmodified lines
18#define VMWARE_BACKDOOR_PMC_REAL_TIME 0x1000118#define VMWARE_BACKDOOR_PMC_REAL_TIME 0x10001
19#define VMWARE_BACKDOOR_PMC_APPARENT_TIME 0x1000219#define VMWARE_BACKDOOR_PMC_APPARENT_TIME 0x10002
20 20
21struct kvm_event_hw_type_mapping {
22 u8 eventsel;
23 u8 unit_mask;
24 unsigned event_type;
25};
26
27struct kvm_pmu_ops {21struct kvm_pmu_ops {
28 bool (*hw_event_available)(struct kvm_pmc *pmc);22 bool (*hw_event_available)(struct kvm_pmc *pmc);
29 bool (*pmc_is_enabled)(struct kvm_pmc *pmc);23 bool (*pmc_is_enabled)(struct kvm_pmc *pmc);
@@ -40,6 +34,9 @@ struct kvm_pmu_ops { · 10 unmodified lines
40 void (*reset)(struct kvm_vcpu *vcpu);34 void (*reset)(struct kvm_vcpu *vcpu);
41 void (*deliver_pmi)(struct kvm_vcpu *vcpu);35 void (*deliver_pmi)(struct kvm_vcpu *vcpu);
42 void (*cleanup)(struct kvm_vcpu *vcpu);36 void (*cleanup)(struct kvm_vcpu *vcpu);
37
38 const u64 EVENTSEL_EVENT;
39 const int MAX_NR_GP_COUNTERS;
43};40};
44 41
45void kvm_pmu_ops_update(const struct kvm_pmu_ops *pmu_ops);42void kvm_pmu_ops_update(const struct kvm_pmu_ops *pmu_ops);
@@ -161,7 +158,7 @@ static inline bool pmc_speculative_in_use(struct kvm_pmc *pmc) · 115 unmodified lines
161 158
162extern struct x86_pmu_capability kvm_pmu_cap;159extern struct x86_pmu_capability kvm_pmu_cap;
163 160
164static inline void kvm_init_pmu_capability(void)161static inline void kvm_init_pmu_capability(const struct kvm_pmu_ops *pmu_ops)
165{162{
166 bool is_intel = boot_cpu_data.x86_vendor == X86_VENDOR_INTEL;163 bool is_intel = boot_cpu_data.x86_vendor == X86_VENDOR_INTEL;
167 164
@@ -192,6 +189,8 @@ static inline void kvm_init_pmu_capability(void) · 24 unmodified lines
192 }189 }
193 190
194 kvm_pmu_cap.version = min(kvm_pmu_cap.version, 2);191 kvm_pmu_cap.version = min(kvm_pmu_cap.version, 2);
192 kvm_pmu_cap.num_counters_gp = min(kvm_pmu_cap.num_counters_gp,
193 pmu_ops->MAX_NR_GP_COUNTERS);
195 kvm_pmu_cap.num_counters_fixed = min(kvm_pmu_cap.num_counters_fixed,194 kvm_pmu_cap.num_counters_fixed = min(kvm_pmu_cap.num_counters_fixed,
196 KVM_PMC_MAX_FIXED);195 KVM_PMC_MAX_FIXED);
197}196}
arch/x86/kvm/reverse_cpuid.h+7−0
@@ -14,6 +14,7 @@ · 13 unmodified lines
14enum kvm_only_cpuid_leafs {14enum kvm_only_cpuid_leafs {
15 CPUID_12_EAX = NCAPINTS,15 CPUID_12_EAX = NCAPINTS,
16 CPUID_7_1_EDX,16 CPUID_7_1_EDX,
17 CPUID_8000_0007_EDX,
17 NR_KVM_CPU_CAPS,18 NR_KVM_CPU_CAPS,
18 19
19 NKVMCAPINTS = NR_KVM_CPU_CAPS - NCAPINTS,20 NKVMCAPINTS = NR_KVM_CPU_CAPS - NCAPINTS,
@@ -43,6 +44,9 @@ enum kvm_only_cpuid_leafs { · 23 unmodified lines
43#define X86_FEATURE_AVX_NE_CONVERT KVM_X86_FEATURE(CPUID_7_1_EDX, 5)44#define X86_FEATURE_AVX_NE_CONVERT KVM_X86_FEATURE(CPUID_7_1_EDX, 5)
44#define X86_FEATURE_PREFETCHITI KVM_X86_FEATURE(CPUID_7_1_EDX, 14)45#define X86_FEATURE_PREFETCHITI KVM_X86_FEATURE(CPUID_7_1_EDX, 14)
45 46
47/* CPUID level 0x80000007 (EDX). */
48#define KVM_X86_FEATURE_CONSTANT_TSC KVM_X86_FEATURE(CPUID_8000_0007_EDX, 8)
49
46struct cpuid_reg {50struct cpuid_reg {
47 u32 function;51 u32 function;
48 u32 index;52 u32 index;
@@ -68,6 +72,7 @@ static const struct cpuid_reg reverse_cpuid[] = { · 19 unmodified lines
68 [CPUID_12_EAX] = {0x00000012, 0, CPUID_EAX},72 [CPUID_12_EAX] = {0x00000012, 0, CPUID_EAX},
69 [CPUID_8000_001F_EAX] = {0x8000001f, 0, CPUID_EAX},73 [CPUID_8000_001F_EAX] = {0x8000001f, 0, CPUID_EAX},
70 [CPUID_7_1_EDX] = { 7, 1, CPUID_EDX},74 [CPUID_7_1_EDX] = { 7, 1, CPUID_EDX},
75 [CPUID_8000_0007_EDX] = {0x80000007, 0, CPUID_EDX},
71 [CPUID_8000_0021_EAX] = {0x80000021, 0, CPUID_EAX},76 [CPUID_8000_0021_EAX] = {0x80000021, 0, CPUID_EAX},
72};77};
73 78
@@ -101,6 +106,8 @@ static __always_inline u32 __feature_translate(int x86_feature) · 27 unmodified lines
101 return KVM_X86_FEATURE_SGX2;106 return KVM_X86_FEATURE_SGX2;
102 else if (x86_feature == X86_FEATURE_SGX_EDECCSSA)107 else if (x86_feature == X86_FEATURE_SGX_EDECCSSA)
103 return KVM_X86_FEATURE_SGX_EDECCSSA;108 return KVM_X86_FEATURE_SGX_EDECCSSA;
109 else if (x86_feature == X86_FEATURE_CONSTANT_TSC)
110 return KVM_X86_FEATURE_CONSTANT_TSC;
104 111
105 return x86_feature;112 return x86_feature;
106}113}
arch/x86/kvm/smm.c+1−2
@@ -1,4 +1,5 @@
1/* SPDX-License-Identifier: GPL-2.0 */1/* SPDX-License-Identifier: GPL-2.0 */
2#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
2 3
3#include <linux/kvm_host.h>4#include <linux/kvm_host.h>
4#include "x86.h"5#include "x86.h"
@@ -110,8 +111,6 @@ static void check_smram_offsets(void) · 105 unmodified lines
110 111
111void kvm_smm_changed(struct kvm_vcpu *vcpu, bool entering_smm)112void kvm_smm_changed(struct kvm_vcpu *vcpu, bool entering_smm)
112{113{
113 BUILD_BUG_ON(HF_SMM_MASK != X86EMUL_SMM_MASK);
114
115 trace_kvm_smm_transition(vcpu->vcpu_id, vcpu->arch.smbase, entering_smm);114 trace_kvm_smm_transition(vcpu->vcpu_id, vcpu->arch.smbase, entering_smm);
116 115
117 if (entering_smm) {116 if (entering_smm) {
arch/x86/kvm/svm/avic.c+167−207
@@ -12,7 +12,7 @@ · 11 unmodified lines
12 * Avi Kivity <avi@qumranet.com>12 * Avi Kivity <avi@qumranet.com>
13 */13 */
14 14
15#define pr_fmt(fmt) "SVM: " fmt15#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
16 16
17#include <linux/kvm_types.h>17#include <linux/kvm_types.h>
18#include <linux/hashtable.h>18#include <linux/hashtable.h>
@@ -53,7 +53,7 @@ static DEFINE_HASHTABLE(svm_vm_data_hash, SVM_VM_DATA_HASH_BITS); · 34 unmodified lines
53static u32 next_vm_id = 0;53static u32 next_vm_id = 0;
54static bool next_vm_id_wrapped = 0;54static bool next_vm_id_wrapped = 0;
55static DEFINE_SPINLOCK(svm_vm_data_hash_lock);55static DEFINE_SPINLOCK(svm_vm_data_hash_lock);
56enum avic_modes avic_mode;56bool x2avic_enabled;
57 57
58/*58/*
59 * This is a wrapper of struct amd_iommu_ir_data.59 * This is a wrapper of struct amd_iommu_ir_data.
@@ -72,20 +72,25 @@ static void avic_activate_vmcb(struct vcpu_svm *svm) · 12 unmodified lines
72 72
73 vmcb->control.int_ctl |= AVIC_ENABLE_MASK;73 vmcb->control.int_ctl |= AVIC_ENABLE_MASK;
74 74
75 /* Note:75 /*
76 * KVM can support hybrid-AVIC mode, where KVM emulates x2APIC76 * Note: KVM supports hybrid-AVIC mode, where KVM emulates x2APIC MSR
77 * MSR accesses, while interrupt injection to a running vCPU77 * accesses, while interrupt injection to a running vCPU can be
78 * can be achieved using AVIC doorbell. The AVIC hardware still78 * achieved using AVIC doorbell. KVM disables the APIC access page
79 * accelerate MMIO accesses, but this does not cause any harm79 * (deletes the memslot) if any vCPU has x2APIC enabled, thus enabling
80 * as the guest is not supposed to access xAPIC mmio when uses x2APIC.80 * AVIC in hybrid mode activates only the doorbell mechanism.
81 */81 */
82 if (apic_x2apic_mode(svm->vcpu.arch.apic) &&82 if (x2avic_enabled && apic_x2apic_mode(svm->vcpu.arch.apic)) {
83 avic_mode == AVIC_MODE_X2) {
84 vmcb->control.int_ctl |= X2APIC_MODE_MASK;83 vmcb->control.int_ctl |= X2APIC_MODE_MASK;
85 vmcb->control.avic_physical_id |= X2AVIC_MAX_PHYSICAL_ID;84 vmcb->control.avic_physical_id |= X2AVIC_MAX_PHYSICAL_ID;
86 /* Disabling MSR intercept for x2APIC registers */85 /* Disabling MSR intercept for x2APIC registers */
87 svm_set_x2apic_msr_interception(svm, false);86 svm_set_x2apic_msr_interception(svm, false);
88 } else {87 } else {
88 /*
89 * Flush the TLB, the guest may have inserted a non-APIC
90 * mapping into the TLB while AVIC was disabled.
91 */
92 kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, &svm->vcpu);
93
89 /* For xAVIC and hybrid-xAVIC modes */94 /* For xAVIC and hybrid-xAVIC modes */
90 vmcb->control.avic_physical_id |= AVIC_MAX_PHYSICAL_ID;95 vmcb->control.avic_physical_id |= AVIC_MAX_PHYSICAL_ID;
91 /* Enabling MSR intercept for x2APIC registers */96 /* Enabling MSR intercept for x2APIC registers */
@@ -241,8 +246,8 @@ static u64 *avic_get_physical_id_entry(struct kvm_vcpu *vcpu, · 149 unmodified lines
241 u64 *avic_physical_id_table;246 u64 *avic_physical_id_table;
242 struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm);247 struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm);
243 248
244 if ((avic_mode == AVIC_MODE_X1 && index > AVIC_MAX_PHYSICAL_ID) ||249 if ((!x2avic_enabled && index > AVIC_MAX_PHYSICAL_ID) ||
245 (avic_mode == AVIC_MODE_X2 && index > X2AVIC_MAX_PHYSICAL_ID))250 (index > X2AVIC_MAX_PHYSICAL_ID))
246 return NULL;251 return NULL;
247 252
248 avic_physical_id_table = page_address(kvm_svm->avic_physical_id_table_page);253 avic_physical_id_table = page_address(kvm_svm->avic_physical_id_table_page);
@@ -250,47 +255,14 @@ static u64 *avic_get_physical_id_entry(struct kvm_vcpu *vcpu, · 1 unmodified line
250 return &avic_physical_id_table[index];255 return &avic_physical_id_table[index];
251}256}
252 257
253/*
254 * Note:
255 * AVIC hardware walks the nested page table to check permissions,
256 * but does not use the SPA address specified in the leaf page
257 * table entry since it uses address in the AVIC_BACKING_PAGE pointer
258 * field of the VMCB. Therefore, we set up the
259 * APIC_ACCESS_PAGE_PRIVATE_MEMSLOT (4KB) here.
260 */
261static int avic_alloc_access_page(struct kvm *kvm)
262{
263 void __user *ret;
264 int r = 0;
265
266 mutex_lock(&kvm->slots_lock);
267
268 if (kvm->arch.apic_access_memslot_enabled)
269 goto out;
270
271 ret = __x86_set_memory_region(kvm,
272 APIC_ACCESS_PAGE_PRIVATE_MEMSLOT,
273 APIC_DEFAULT_PHYS_BASE,
274 PAGE_SIZE);
275 if (IS_ERR(ret)) {
276 r = PTR_ERR(ret);
277 goto out;
278 }
279
280 kvm->arch.apic_access_memslot_enabled = true;
281out:
282 mutex_unlock(&kvm->slots_lock);
283 return r;
284}
285
286static int avic_init_backing_page(struct kvm_vcpu *vcpu)258static int avic_init_backing_page(struct kvm_vcpu *vcpu)
287{259{
288 u64 *entry, new_entry;260 u64 *entry, new_entry;
289 int id = vcpu->vcpu_id;261 int id = vcpu->vcpu_id;
290 struct vcpu_svm *svm = to_svm(vcpu);262 struct vcpu_svm *svm = to_svm(vcpu);
291 263
292 if ((avic_mode == AVIC_MODE_X1 && id > AVIC_MAX_PHYSICAL_ID) ||264 if ((!x2avic_enabled && id > AVIC_MAX_PHYSICAL_ID) ||
293 (avic_mode == AVIC_MODE_X2 && id > X2AVIC_MAX_PHYSICAL_ID))265 (id > X2AVIC_MAX_PHYSICAL_ID))
294 return -EINVAL;266 return -EINVAL;
295 267
296 if (!vcpu->arch.apic->regs)268 if (!vcpu->arch.apic->regs)
@@ -299,7 +271,13 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu) · 2 unmodified lines
299 if (kvm_apicv_activated(vcpu->kvm)) {271 if (kvm_apicv_activated(vcpu->kvm)) {
300 int ret;272 int ret;
301 273
302 ret = avic_alloc_access_page(vcpu->kvm);274 /*
275 * Note, AVIC hardware walks the nested page table to check
276 * permissions, but does not use the SPA address specified in
277 * the leaf SPTE since it uses address in the AVIC_BACKING_PAGE
278 * pointer field of the VMCB.
279 */
280 ret = kvm_alloc_apic_access_page(vcpu->kvm);
303 if (ret)281 if (ret)
304 return ret;282 return ret;
305 }283 }
@@ -339,6 +317,60 @@ void avic_ring_doorbell(struct kvm_vcpu *vcpu) · 33 unmodified lines
339 put_cpu();317 put_cpu();
340}318}
341 319
320
321static void avic_kick_vcpu(struct kvm_vcpu *vcpu, u32 icrl)
322{
323 vcpu->arch.apic->irr_pending = true;
324 svm_complete_interrupt_delivery(vcpu,
325 icrl & APIC_MODE_MASK,
326 icrl & APIC_INT_LEVELTRIG,
327 icrl & APIC_VECTOR_MASK);
328}
329
330static void avic_kick_vcpu_by_physical_id(struct kvm *kvm, u32 physical_id,
331 u32 icrl)
332{
333 /*
334 * KVM inhibits AVIC if any vCPU ID diverges from the vCPUs APIC ID,
335 * i.e. APIC ID == vCPU ID.
336 */
337 struct kvm_vcpu *target_vcpu = kvm_get_vcpu_by_id(kvm, physical_id);
338
339 /* Once again, nothing to do if the target vCPU doesn't exist. */
340 if (unlikely(!target_vcpu))
341 return;
342
343 avic_kick_vcpu(target_vcpu, icrl);
344}
345
346static void avic_kick_vcpu_by_logical_id(struct kvm *kvm, u32 *avic_logical_id_table,
347 u32 logid_index, u32 icrl)
348{
349 u32 physical_id;
350
351 if (avic_logical_id_table) {
352 u32 logid_entry = avic_logical_id_table[logid_index];
353
354 /* Nothing to do if the logical destination is invalid. */
355 if (unlikely(!(logid_entry & AVIC_LOGICAL_ID_ENTRY_VALID_MASK)))
356 return;
357
358 physical_id = logid_entry &
359 AVIC_LOGICAL_ID_ENTRY_GUEST_PHYSICAL_ID_MASK;
360 } else {
361 /*
362 * For x2APIC, the logical APIC ID is a read-only value that is
363 * derived from the x2APIC ID, thus the x2APIC ID can be found
364 * by reversing the calculation (stored in logid_index). Note,
365 * bits 31:20 of the x2APIC ID aren't propagated to the logical
366 * ID, but KVM limits the x2APIC ID limited to KVM_MAX_VCPU_IDS.
367 */
368 physical_id = logid_index;
369 }
370
371 avic_kick_vcpu_by_physical_id(kvm, physical_id, icrl);
372}
373
342/*374/*
343 * A fast-path version of avic_kick_target_vcpus(), which attempts to match375 * A fast-path version of avic_kick_target_vcpus(), which attempts to match
344 * destination APIC ID to vCPU without looping through all vCPUs.376 * destination APIC ID to vCPU without looping through all vCPUs.
@@ -346,11 +378,10 @@ void avic_ring_doorbell(struct kvm_vcpu *vcpu) · 1 unmodified line
346static int avic_kick_target_vcpus_fast(struct kvm *kvm, struct kvm_lapic *source,378static int avic_kick_target_vcpus_fast(struct kvm *kvm, struct kvm_lapic *source,
347 u32 icrl, u32 icrh, u32 index)379 u32 icrl, u32 icrh, u32 index)
348{380{
349 u32 l1_physical_id, dest;
350 struct kvm_vcpu *target_vcpu;
351 int dest_mode = icrl & APIC_DEST_MASK;381 int dest_mode = icrl & APIC_DEST_MASK;
352 int shorthand = icrl & APIC_SHORT_MASK;382 int shorthand = icrl & APIC_SHORT_MASK;
353 struct kvm_svm *kvm_svm = to_kvm_svm(kvm);383 struct kvm_svm *kvm_svm = to_kvm_svm(kvm);
384 u32 dest;
354 385
355 if (shorthand != APIC_DEST_NOSHORT)386 if (shorthand != APIC_DEST_NOSHORT)
356 return -EINVAL;387 return -EINVAL;
@@ -367,18 +398,18 @@ static int avic_kick_target_vcpus_fast(struct kvm *kvm, struct kvm_lapic *source · 10 unmodified lines
367 if (!apic_x2apic_mode(source) && dest == APIC_BROADCAST)398 if (!apic_x2apic_mode(source) && dest == APIC_BROADCAST)
368 return -EINVAL;399 return -EINVAL;
369 400
370 l1_physical_id = dest;401 if (WARN_ON_ONCE(dest != index))
371
372 if (WARN_ON_ONCE(l1_physical_id != index))
373 return -EINVAL;402 return -EINVAL;
374 403
404 avic_kick_vcpu_by_physical_id(kvm, dest, icrl);
375 } else {405 } else {
376 u32 bitmap, cluster;406 u32 *avic_logical_id_table;
377 int logid_index;407 unsigned long bitmap, i;
408 u32 cluster;
378 409
379 if (apic_x2apic_mode(source)) {410 if (apic_x2apic_mode(source)) {
380 /* 16 bit dest mask, 16 bit cluster id */411 /* 16 bit dest mask, 16 bit cluster id */
381 bitmap = dest & 0xFFFF0000;412 bitmap = dest & 0xFFFF;
382 cluster = (dest >> 16) << 4;413 cluster = (dest >> 16) << 4;
383 } else if (kvm_lapic_get_reg(source, APIC_DFR) == APIC_DFR_FLAT) {414 } else if (kvm_lapic_get_reg(source, APIC_DFR) == APIC_DFR_FLAT) {
384 /* 8 bit dest mask*/415 /* 8 bit dest mask*/
@@ -390,67 +421,32 @@ static int avic_kick_target_vcpus_fast(struct kvm *kvm, struct kvm_lapic *source · 5 unmodified lines
390 cluster = (dest >> 4) << 2;421 cluster = (dest >> 4) << 2;
391 }422 }
392 423
424 /* Nothing to do if there are no destinations in the cluster. */
393 if (unlikely(!bitmap))425 if (unlikely(!bitmap))
394 /* guest bug: nobody to send the logical interrupt to */
395 return 0;426 return 0;
396 427
397 if (!is_power_of_2(bitmap))428 if (apic_x2apic_mode(source))
398 /* multiple logical destinations, use slow path */429 avic_logical_id_table = NULL;
399 return -EINVAL;430 else
400 431 avic_logical_id_table = page_address(kvm_svm->avic_logical_id_table_page);
401 logid_index = cluster + __ffs(bitmap);
402
403 if (!apic_x2apic_mode(source)) {
404 u32 *avic_logical_id_table =
405 page_address(kvm_svm->avic_logical_id_table_page);
406
407 u32 logid_entry = avic_logical_id_table[logid_index];
408
409 if (WARN_ON_ONCE(index != logid_index))
410 return -EINVAL;
411
412 /* guest bug: non existing/reserved logical destination */
413 if (unlikely(!(logid_entry & AVIC_LOGICAL_ID_ENTRY_VALID_MASK)))
414 return 0;
415
416 l1_physical_id = logid_entry &
417 AVIC_LOGICAL_ID_ENTRY_GUEST_PHYSICAL_ID_MASK;
418 } else {
419 /*
420 * For x2APIC logical mode, cannot leverage the index.
421 * Instead, calculate physical ID from logical ID in ICRH.
422 */
423 int cluster = (icrh & 0xffff0000) >> 16;
424 int apic = ffs(icrh & 0xffff) - 1;
425
426 /*
427 * If the x2APIC logical ID sub-field (i.e. icrh[15:0])
428 * contains anything but a single bit, we cannot use the
429 * fast path, because it is limited to a single vCPU.
430 */
431 if (apic < 0 || icrh != (1 << apic))
432 return -EINVAL;
433 432
434 l1_physical_id = (cluster << 4) + apic;433 /*
435 }434 * AVIC is inhibited if vCPUs aren't mapped 1:1 with logical
435 * IDs, thus each bit in the destination is guaranteed to map
436 * to at most one vCPU.
437 */
438 for_each_set_bit(i, &bitmap, 16)
439 avic_kick_vcpu_by_logical_id(kvm, avic_logical_id_table,
440 cluster + i, icrl);
436 }441 }
437 442
438 target_vcpu = kvm_get_vcpu_by_id(kvm, l1_physical_id);
439 if (unlikely(!target_vcpu))
440 /* guest bug: non existing vCPU is a target of this IPI*/
441 return 0;
442
443 target_vcpu->arch.apic->irr_pending = true;
444 svm_complete_interrupt_delivery(target_vcpu,
445 icrl & APIC_MODE_MASK,
446 icrl & APIC_INT_LEVELTRIG,
447 icrl & APIC_VECTOR_MASK);
448 return 0;443 return 0;
449}444}
450 445
451static void avic_kick_target_vcpus(struct kvm *kvm, struct kvm_lapic *source,446static void avic_kick_target_vcpus(struct kvm *kvm, struct kvm_lapic *source,
452 u32 icrl, u32 icrh, u32 index)447 u32 icrl, u32 icrh, u32 index)
453{448{
449 u32 dest = apic_x2apic_mode(source) ? icrh : GET_XAPIC_DEST_FIELD(icrh);
454 unsigned long i;450 unsigned long i;
455 struct kvm_vcpu *vcpu;451 struct kvm_vcpu *vcpu;
456 452
@@ -466,21 +462,9 @@ static void avic_kick_target_vcpus(struct kvm *kvm, struct kvm_lapic *source, · 9 unmodified lines
466 * since entered the guest will have processed pending IRQs at VMRUN.462 * since entered the guest will have processed pending IRQs at VMRUN.
467 */463 */
468 kvm_for_each_vcpu(i, vcpu, kvm) {464 kvm_for_each_vcpu(i, vcpu, kvm) {
469 u32 dest;
470
471 if (apic_x2apic_mode(vcpu->arch.apic))
472 dest = icrh;
473 else
474 dest = GET_XAPIC_DEST_FIELD(icrh);
475
476 if (kvm_apic_match_dest(vcpu, source, icrl & APIC_SHORT_MASK,465 if (kvm_apic_match_dest(vcpu, source, icrl & APIC_SHORT_MASK,
477 dest, icrl & APIC_DEST_MASK)) {466 dest, icrl & APIC_DEST_MASK))
478 vcpu->arch.apic->irr_pending = true;467 avic_kick_vcpu(vcpu, icrl);
479 svm_complete_interrupt_delivery(vcpu,
480 icrl & APIC_MODE_MASK,
481 icrl & APIC_INT_LEVELTRIG,
482 icrl & APIC_VECTOR_MASK);
483 }
484 }468 }
485}469}
486 470
@@ -496,14 +480,18 @@ int avic_incomplete_ipi_interception(struct kvm_vcpu *vcpu) · 9 unmodified lines
496 trace_kvm_avic_incomplete_ipi(vcpu->vcpu_id, icrh, icrl, id, index);480 trace_kvm_avic_incomplete_ipi(vcpu->vcpu_id, icrh, icrl, id, index);
497 481
498 switch (id) {482 switch (id) {
483 case AVIC_IPI_FAILURE_INVALID_TARGET:
499 case AVIC_IPI_FAILURE_INVALID_INT_TYPE:484 case AVIC_IPI_FAILURE_INVALID_INT_TYPE:
500 /*485 /*
501 * Emulate IPIs that are not handled by AVIC hardware, which486 * Emulate IPIs that are not handled by AVIC hardware, which
502 * only virtualizes Fixed, Edge-Triggered INTRs. The exit is487 * only virtualizes Fixed, Edge-Triggered INTRs, and falls over
503 * a trap, e.g. ICR holds the correct value and RIP has been488 * if _any_ targets are invalid, e.g. if the logical mode mask
504 * advanced, KVM is responsible only for emulating the IPI.489 * is a superset of running vCPUs.
505 * Sadly, hardware may sometimes leave the BUSY flag set, in490 *
506 * which case KVM needs to emulate the ICR write as well in491 * The exit is a trap, e.g. ICR holds the correct value and RIP
492 * has been advanced, KVM is responsible only for emulating the
493 * IPI. Sadly, hardware may sometimes leave the BUSY flag set,
494 * in which case KVM needs to emulate the ICR write as well in
507 * order to clear the BUSY flag.495 * order to clear the BUSY flag.
508 */496 */
509 if (icrl & APIC_ICR_BUSY)497 if (icrl & APIC_ICR_BUSY)
@@ -519,8 +507,6 @@ int avic_incomplete_ipi_interception(struct kvm_vcpu *vcpu) · 9 unmodified lines
519 */507 */
520 avic_kick_target_vcpus(vcpu->kvm, apic, icrl, icrh, index);508 avic_kick_target_vcpus(vcpu->kvm, apic, icrl, icrh, index);
521 break;509 break;
522 case AVIC_IPI_FAILURE_INVALID_TARGET:
523 break;
524 case AVIC_IPI_FAILURE_INVALID_BACKING_PAGE:510 case AVIC_IPI_FAILURE_INVALID_BACKING_PAGE:
525 WARN_ONCE(1, "Invalid backing page\n");511 WARN_ONCE(1, "Invalid backing page\n");
526 break;512 break;
@@ -541,33 +527,33 @@ unsigned long avic_vcpu_get_apicv_inhibit_reasons(struct kvm_vcpu *vcpu) · 14 unmodified lines
541static u32 *avic_get_logical_id_entry(struct kvm_vcpu *vcpu, u32 ldr, bool flat)527static u32 *avic_get_logical_id_entry(struct kvm_vcpu *vcpu, u32 ldr, bool flat)
542{528{
543 struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm);529 struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm);
544 int index;
545 u32 *logical_apic_id_table;530 u32 *logical_apic_id_table;
546 int dlid = GET_APIC_LOGICAL_ID(ldr);531 u32 cluster, index;
547
548 if (!dlid)
549 return NULL;
550 532
551 if (flat) { /* flat */533 ldr = GET_APIC_LOGICAL_ID(ldr);
552 index = ffs(dlid) - 1;
553 if (index > 7)
554 return NULL;
555 } else { /* cluster */
556 int cluster = (dlid & 0xf0) >> 4;
557 int apic = ffs(dlid & 0x0f) - 1;
558 534
559 if ((apic < 0) || (apic > 7) ||535 if (flat) {
560 (cluster >= 0xf))536 cluster = 0;
537 } else {
538 cluster = (ldr >> 4);
539 if (cluster >= 0xf)
561 return NULL;540 return NULL;
562 index = (cluster << 2) + apic;541 ldr &= 0xf;
563 }542 }
543 if (!ldr || !is_power_of_2(ldr))
544 return NULL;
545
546 index = __ffs(ldr);
547 if (WARN_ON_ONCE(index > 7))
548 return NULL;
549 index += (cluster << 2);
564 550
565 logical_apic_id_table = (u32 *) page_address(kvm_svm->avic_logical_id_table_page);551 logical_apic_id_table = (u32 *) page_address(kvm_svm->avic_logical_id_table_page);
566 552
567 return &logical_apic_id_table[index];553 return &logical_apic_id_table[index];
568}554}
569 555
570static int avic_ldr_write(struct kvm_vcpu *vcpu, u8 g_physical_id, u32 ldr)556static void avic_ldr_write(struct kvm_vcpu *vcpu, u8 g_physical_id, u32 ldr)
571{557{
572 bool flat;558 bool flat;
573 u32 *entry, new_entry;559 u32 *entry, new_entry;
@@ -575,15 +561,13 @@ static int avic_ldr_write(struct kvm_vcpu *vcpu, u8 g_physical_id, u32 ldr) · 1 unmodified line
575 flat = kvm_lapic_get_reg(vcpu->arch.apic, APIC_DFR) == APIC_DFR_FLAT;561 flat = kvm_lapic_get_reg(vcpu->arch.apic, APIC_DFR) == APIC_DFR_FLAT;
576 entry = avic_get_logical_id_entry(vcpu, ldr, flat);562 entry = avic_get_logical_id_entry(vcpu, ldr, flat);
577 if (!entry)563 if (!entry)
578 return -EINVAL;564 return;
579 565
580 new_entry = READ_ONCE(*entry);566 new_entry = READ_ONCE(*entry);
581 new_entry &= ~AVIC_LOGICAL_ID_ENTRY_GUEST_PHYSICAL_ID_MASK;567 new_entry &= ~AVIC_LOGICAL_ID_ENTRY_GUEST_PHYSICAL_ID_MASK;
582 new_entry |= (g_physical_id & AVIC_LOGICAL_ID_ENTRY_GUEST_PHYSICAL_ID_MASK);568 new_entry |= (g_physical_id & AVIC_LOGICAL_ID_ENTRY_GUEST_PHYSICAL_ID_MASK);
583 new_entry |= AVIC_LOGICAL_ID_ENTRY_VALID_MASK;569 new_entry |= AVIC_LOGICAL_ID_ENTRY_VALID_MASK;
584 WRITE_ONCE(*entry, new_entry);570 WRITE_ONCE(*entry, new_entry);
585
586 return 0;
587}571}
588 572
589static void avic_invalidate_logical_id_entry(struct kvm_vcpu *vcpu)573static void avic_invalidate_logical_id_entry(struct kvm_vcpu *vcpu)
@@ -601,29 +585,23 @@ static void avic_invalidate_logical_id_entry(struct kvm_vcpu *vcpu) · 11 unmodified lines
601 clear_bit(AVIC_LOGICAL_ID_ENTRY_VALID_BIT, (unsigned long *)entry);585 clear_bit(AVIC_LOGICAL_ID_ENTRY_VALID_BIT, (unsigned long *)entry);
602}586}
603 587
604static int avic_handle_ldr_update(struct kvm_vcpu *vcpu)588static void avic_handle_ldr_update(struct kvm_vcpu *vcpu)
605{589{
606 int ret = 0;
607 struct vcpu_svm *svm = to_svm(vcpu);590 struct vcpu_svm *svm = to_svm(vcpu);
608 u32 ldr = kvm_lapic_get_reg(vcpu->arch.apic, APIC_LDR);591 u32 ldr = kvm_lapic_get_reg(vcpu->arch.apic, APIC_LDR);
609 u32 id = kvm_xapic_id(vcpu->arch.apic);592 u32 id = kvm_xapic_id(vcpu->arch.apic);
610 593
611 /* AVIC does not support LDR update for x2APIC */594 /* AVIC does not support LDR update for x2APIC */
612 if (apic_x2apic_mode(vcpu->arch.apic))595 if (apic_x2apic_mode(vcpu->arch.apic))
613 return 0;596 return;
614 597
615 if (ldr == svm->ldr_reg)598 if (ldr == svm->ldr_reg)
616 return 0;599 return;
617 600
618 avic_invalidate_logical_id_entry(vcpu);601 avic_invalidate_logical_id_entry(vcpu);
619 602
620 if (ldr)603 svm->ldr_reg = ldr;
621 ret = avic_ldr_write(vcpu, id, ldr);604 avic_ldr_write(vcpu, id, ldr);
622
623 if (!ret)
624 svm->ldr_reg = ldr;
625
626 return ret;
627}605}
628 606
629static void avic_handle_dfr_update(struct kvm_vcpu *vcpu)607static void avic_handle_dfr_update(struct kvm_vcpu *vcpu)
@@ -645,12 +623,14 @@ static int avic_unaccel_trap_write(struct kvm_vcpu *vcpu) · 15 unmodified lines
645 623
646 switch (offset) {624 switch (offset) {
647 case APIC_LDR:625 case APIC_LDR:
648 if (avic_handle_ldr_update(vcpu))626 avic_handle_ldr_update(vcpu);
649 return 0;
650 break;627 break;
651 case APIC_DFR:628 case APIC_DFR:
652 avic_handle_dfr_update(vcpu);629 avic_handle_dfr_update(vcpu);
653 break;630 break;
631 case APIC_RRR:
632 /* Ignore writes to Read Remote Data, it's read-only. */
633 return 1;
654 default:634 default:
655 break;635 break;
656 }636 }
@@ -739,18 +719,6 @@ void avic_apicv_post_state_restore(struct kvm_vcpu *vcpu) · 82 unmodified lines
739 avic_handle_ldr_update(vcpu);719 avic_handle_ldr_update(vcpu);
740}720}
741 721
742void avic_set_virtual_apic_mode(struct kvm_vcpu *vcpu)
743{
744 if (!lapic_in_kernel(vcpu) || avic_mode == AVIC_MODE_NONE)
745 return;
746
747 if (kvm_get_apic_mode(vcpu) == LAPIC_MODE_INVALID) {
748 WARN_ONCE(true, "Invalid local APIC state (vcpu_id=%d)", vcpu->vcpu_id);
749 return;
750 }
751 avic_refresh_apicv_exec_ctrl(vcpu);
752}
753
754static int avic_set_pi_irte_mode(struct kvm_vcpu *vcpu, bool activate)722static int avic_set_pi_irte_mode(struct kvm_vcpu *vcpu, bool activate)
755{723{
756 int ret = 0;724 int ret = 0;
@@ -995,23 +963,6 @@ out: · 238 unmodified lines
995 return ret;963 return ret;
996}964}
997 965
998bool avic_check_apicv_inhibit_reasons(enum kvm_apicv_inhibit reason)
999{
1000 ulong supported = BIT(APICV_INHIBIT_REASON_DISABLE) |
1001 BIT(APICV_INHIBIT_REASON_ABSENT) |
1002 BIT(APICV_INHIBIT_REASON_HYPERV) |
1003 BIT(APICV_INHIBIT_REASON_NESTED) |
1004 BIT(APICV_INHIBIT_REASON_IRQWIN) |
1005 BIT(APICV_INHIBIT_REASON_PIT_REINJ) |
1006 BIT(APICV_INHIBIT_REASON_BLOCKIRQ) |
1007 BIT(APICV_INHIBIT_REASON_SEV) |
1008 BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) |
1009 BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED);
1010
1011 return supported & BIT(reason);
1012}
1013
1014
1015static inline int966static inline int
1016avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu, bool r)967avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu, bool r)
1017{968{
@@ -1064,6 +1015,7 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu) · 46 unmodified lines
1064 return;1015 return;
1065 1016
1066 entry = READ_ONCE(*(svm->avic_physical_id_cache));1017 entry = READ_ONCE(*(svm->avic_physical_id_cache));
1018 WARN_ON_ONCE(entry & AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK);
1067 1019
1068 entry &= ~AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK;1020 entry &= ~AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK;
1069 entry |= (h_physical_id & AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK);1021 entry |= (h_physical_id & AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK);
@@ -1092,17 +1044,15 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu) · 22 unmodified lines
1092 WRITE_ONCE(*(svm->avic_physical_id_cache), entry);1044 WRITE_ONCE(*(svm->avic_physical_id_cache), entry);
1093}1045}
1094 1046
1095 1047void avic_refresh_virtual_apic_mode(struct kvm_vcpu *vcpu)
1096void avic_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu)
1097{1048{
1098 struct vcpu_svm *svm = to_svm(vcpu);1049 struct vcpu_svm *svm = to_svm(vcpu);
1099 struct vmcb *vmcb = svm->vmcb01.ptr;1050 struct vmcb *vmcb = svm->vmcb01.ptr;
1100 bool activated = kvm_vcpu_apicv_active(vcpu);
1101 1051
1102 if (!enable_apicv)1052 if (!lapic_in_kernel(vcpu) || !enable_apicv)
1103 return;1053 return;
1104 1054
1105 if (activated) {1055 if (kvm_vcpu_apicv_active(vcpu)) {
1106 /**1056 /**
1107 * During AVIC temporary deactivation, guest could update1057 * During AVIC temporary deactivation, guest could update
1108 * APIC ID, DFR and LDR registers, which would not be trapped1058 * APIC ID, DFR and LDR registers, which would not be trapped
@@ -1116,6 +1066,16 @@ void avic_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu) · 7 unmodified lines
1116 avic_deactivate_vmcb(svm);1066 avic_deactivate_vmcb(svm);
1117 }1067 }
1118 vmcb_mark_dirty(vmcb, VMCB_AVIC);1068 vmcb_mark_dirty(vmcb, VMCB_AVIC);
1069}
1070
1071void avic_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu)
1072{
1073 bool activated = kvm_vcpu_apicv_active(vcpu);
1074
1075 if (!enable_apicv)
1076 return;
1077
1078 avic_refresh_virtual_apic_mode(vcpu);
1119 1079
1120 if (activated)1080 if (activated)
1121 avic_vcpu_load(vcpu, vcpu->cpu);1081 avic_vcpu_load(vcpu, vcpu->cpu);
@@ -1160,37 +1120,37 @@ void avic_vcpu_unblocking(struct kvm_vcpu *vcpu) · 38 unmodified lines
1160 * - Hypervisor can support both xAVIC and x2AVIC in the same guest.1120 * - Hypervisor can support both xAVIC and x2AVIC in the same guest.
1161 * - The mode can be switched at run-time.1121 * - The mode can be switched at run-time.
1162 */1122 */
1163bool avic_hardware_setup(struct kvm_x86_ops *x86_ops)1123bool avic_hardware_setup(void)
1164{1124{
1165 if (!npt_enabled)1125 if (!npt_enabled)
1166 return false;1126 return false;
1167 1127
1128 /* AVIC is a prerequisite for x2AVIC. */
1129 if (!boot_cpu_has(X86_FEATURE_AVIC) && !force_avic) {
1130 if (boot_cpu_has(X86_FEATURE_X2AVIC)) {
1131 pr_warn(FW_BUG "Cannot support x2AVIC due to AVIC is disabled");
1132 pr_warn(FW_BUG "Try enable AVIC using force_avic option");
1133 }
1134 return false;
1135 }
1136
1168 if (boot_cpu_has(X86_FEATURE_AVIC)) {1137 if (boot_cpu_has(X86_FEATURE_AVIC)) {
1169 avic_mode = AVIC_MODE_X1;
1170 pr_info("AVIC enabled\n");1138 pr_info("AVIC enabled\n");
1171 } else if (force_avic) {1139 } else if (force_avic) {
1172 /*1140 /*
1173 * Some older systems does not advertise AVIC support.1141 * Some older systems does not advertise AVIC support.
1174 * See Revision Guide for specific AMD processor for more detail.1142 * See Revision Guide for specific AMD processor for more detail.
1175 */1143 */
1176 avic_mode = AVIC_MODE_X1;
1177 pr_warn("AVIC is not supported in CPUID but force enabled");1144 pr_warn("AVIC is not supported in CPUID but force enabled");
1178 pr_warn("Your system might crash and burn");1145 pr_warn("Your system might crash and burn");
1179 }1146 }
1180 1147
1181 /* AVIC is a prerequisite for x2AVIC. */1148 /* AVIC is a prerequisite for x2AVIC. */
1182 if (boot_cpu_has(X86_FEATURE_X2AVIC)) {1149 x2avic_enabled = boot_cpu_has(X86_FEATURE_X2AVIC);
1183 if (avic_mode == AVIC_MODE_X1) {1150 if (x2avic_enabled)
1184 avic_mode = AVIC_MODE_X2;1151 pr_info("x2AVIC enabled\n");
1185 pr_info("x2AVIC enabled\n");
1186 } else {
1187 pr_warn(FW_BUG "Cannot support x2AVIC due to AVIC is disabled");
1188 pr_warn(FW_BUG "Try enable AVIC using force_avic option");
1189 }
1190 }
1191 1152
1192 if (avic_mode != AVIC_MODE_NONE)1153 amd_iommu_register_ga_log_notifier(&avic_ga_log_notifier);
1193 amd_iommu_register_ga_log_notifier(&avic_ga_log_notifier);
1194 1154
1195 return !!avic_mode;1155 return true;
1196}1156}
arch/x86/kvm/svm/nested.c+2−3
@@ -12,7 +12,7 @@ · 11 unmodified lines
12 * Avi Kivity <avi@qumranet.com>12 * Avi Kivity <avi@qumranet.com>
13 */13 */
14 14
15#define pr_fmt(fmt) "SVM: " fmt15#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
16 16
17#include <linux/kvm_types.h>17#include <linux/kvm_types.h>
18#include <linux/kvm_host.h>18#include <linux/kvm_host.h>
@@ -1008,7 +1008,6 @@ int nested_svm_vmexit(struct vcpu_svm *svm) · 989 unmodified lines
1008 vmcb12->control.next_rip = vmcb02->control.next_rip;1008 vmcb12->control.next_rip = vmcb02->control.next_rip;
1009 1009
1010 vmcb12->control.int_ctl = svm->nested.ctl.int_ctl;1010 vmcb12->control.int_ctl = svm->nested.ctl.int_ctl;
1011 vmcb12->control.tlb_ctl = svm->nested.ctl.tlb_ctl;
1012 vmcb12->control.event_inj = svm->nested.ctl.event_inj;1011 vmcb12->control.event_inj = svm->nested.ctl.event_inj;
1013 vmcb12->control.event_inj_err = svm->nested.ctl.event_inj_err;1012 vmcb12->control.event_inj_err = svm->nested.ctl.event_inj_err;
1014 1013
@@ -1104,7 +1103,7 @@ int nested_svm_vmexit(struct vcpu_svm *svm) · 89 unmodified lines
1104 * to benefit from it right away.1103 * to benefit from it right away.
1105 */1104 */
1106 if (kvm_apicv_activated(vcpu->kvm))1105 if (kvm_apicv_activated(vcpu->kvm))
1107 kvm_vcpu_update_apicv(vcpu);1106 __kvm_vcpu_update_apicv(vcpu);
1108 1107
1109 return 0;1108 return 0;
1110}1109}
arch/x86/kvm/svm/pmu.c+4−0
@@ -9,6 +9,8 @@ · 8 unmodified lines
9 *9 *
10 * Implementation is based on pmu_intel.c file10 * Implementation is based on pmu_intel.c file
11 */11 */
12#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
13
12#include <linux/types.h>14#include <linux/types.h>
13#include <linux/kvm_host.h>15#include <linux/kvm_host.h>
14#include <linux/perf_event.h>16#include <linux/perf_event.h>
@@ -229,4 +231,6 @@ struct kvm_pmu_ops amd_pmu_ops __initdata = { · 214 unmodified lines
229 .refresh = amd_pmu_refresh,231 .refresh = amd_pmu_refresh,
230 .init = amd_pmu_init,232 .init = amd_pmu_init,
231 .reset = amd_pmu_reset,233 .reset = amd_pmu_reset,
234 .EVENTSEL_EVENT = AMD64_EVENTSEL_EVENT,
235 .MAX_NR_GP_COUNTERS = KVM_AMD_PMC_MAX_GENERIC,
232};236};
arch/x86/kvm/svm/sev.c+4−3
@@ -6,6 +6,7 @@ · 5 unmodified lines
6 *6 *
7 * Copyright 2010 Red Hat, Inc. and/or its affiliates.7 * Copyright 2010 Red Hat, Inc. and/or its affiliates.
8 */8 */
9#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
9 10
10#include <linux/kvm_types.h>11#include <linux/kvm_types.h>
11#include <linux/kvm_host.h>12#include <linux/kvm_host.h>
@@ -812,7 +813,7 @@ static int __sev_dbg_decrypt_user(struct kvm *kvm, unsigned long paddr, · 800 unmodified lines
812 if (!IS_ALIGNED(dst_paddr, 16) ||813 if (!IS_ALIGNED(dst_paddr, 16) ||
813 !IS_ALIGNED(paddr, 16) ||814 !IS_ALIGNED(paddr, 16) ||
814 !IS_ALIGNED(size, 16)) {815 !IS_ALIGNED(size, 16)) {
815 tpage = (void *)alloc_page(GFP_KERNEL | __GFP_ZERO);816 tpage = (void *)alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO);
816 if (!tpage)817 if (!tpage)
817 return -ENOMEM;818 return -ENOMEM;
818 819
@@ -1293,7 +1294,7 @@ static int sev_send_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) · 474 unmodified lines
1293 1294
1294 /* Check if we are crossing the page boundary */1295 /* Check if we are crossing the page boundary */
1295 offset = params.guest_uaddr & (PAGE_SIZE - 1);1296 offset = params.guest_uaddr & (PAGE_SIZE - 1);
1296 if ((params.guest_len + offset > PAGE_SIZE))1297 if (params.guest_len > PAGE_SIZE || (params.guest_len + offset) > PAGE_SIZE)
1297 return -EINVAL;1298 return -EINVAL;
1298 1299
1299 /* Pin guest memory */1300 /* Pin guest memory */
@@ -1473,7 +1474,7 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) · 173 unmodified lines
1473 1474
1474 /* Check if we are crossing the page boundary */1475 /* Check if we are crossing the page boundary */
1475 offset = params.guest_uaddr & (PAGE_SIZE - 1);1476 offset = params.guest_uaddr & (PAGE_SIZE - 1);
1476 if ((params.guest_len + offset > PAGE_SIZE))1477 if (params.guest_len > PAGE_SIZE || (params.guest_len + offset) > PAGE_SIZE)
1477 return -EINVAL;1478 return -EINVAL;
1478 1479
1479 hdr = psp_copy_user_blob(params.hdr_uaddr, params.hdr_len);1480 hdr = psp_copy_user_blob(params.hdr_uaddr, params.hdr_len);
arch/x86/kvm/svm/svm.c+75−56
@@ -1,4 +1,4 @@
1#define pr_fmt(fmt) "SVM: " fmt1#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
2 2
3#include <linux/kvm_host.h>3#include <linux/kvm_host.h>
4 4
@@ -519,21 +519,37 @@ static void svm_init_osvw(struct kvm_vcpu *vcpu) · 514 unmodified lines
519 vcpu->arch.osvw.status |= 1;519 vcpu->arch.osvw.status |= 1;
520}520}
521 521
522static int has_svm(void)522static bool kvm_is_svm_supported(void)
523{523{
524 int cpu = raw_smp_processor_id();
524 const char *msg;525 const char *msg;
526 u64 vm_cr;
525 527
526 if (!cpu_has_svm(&msg)) {528 if (!cpu_has_svm(&msg)) {
527 printk(KERN_INFO "has_svm: %s\n", msg);529 pr_err("SVM not supported by CPU %d, %s\n", cpu, msg);
528 return 0;530 return false;
529 }531 }
530 532
531 if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) {533 if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) {
532 pr_info("KVM is unsupported when running as an SEV guest\n");534 pr_info("KVM is unsupported when running as an SEV guest\n");
533 return 0;535 return false;
534 }536 }
535 537
536 return 1;538 rdmsrl(MSR_VM_CR, vm_cr);
539 if (vm_cr & (1 << SVM_VM_CR_SVM_DISABLE)) {
540 pr_err("SVM disabled (by BIOS) in MSR_VM_CR on CPU %d\n", cpu);
541 return false;
542 }
543
544 return true;
545}
546
547static int svm_check_processor_compat(void)
548{
549 if (!kvm_is_svm_supported())
550 return -EIO;
551
552 return 0;
537}553}
538 554
539void __svm_write_tsc_multiplier(u64 multiplier)555void __svm_write_tsc_multiplier(u64 multiplier)
@@ -572,10 +588,6 @@ static int svm_hardware_enable(void) · 32 unmodified lines
572 if (efer & EFER_SVME)588 if (efer & EFER_SVME)
573 return -EBUSY;589 return -EBUSY;
574 590
575 if (!has_svm()) {
576 pr_err("%s: err EOPNOTSUPP on %d\n", __func__, me);
577 return -EINVAL;
578 }
579 sd = per_cpu_ptr(&svm_data, me);591 sd = per_cpu_ptr(&svm_data, me);
580 sd->asid_generation = 1;592 sd->asid_generation = 1;
581 sd->max_asid = cpuid_ebx(SVM_CPUID_FUNC) - 1;593 sd->max_asid = cpuid_ebx(SVM_CPUID_FUNC) - 1;
@@ -813,7 +825,7 @@ void svm_set_x2apic_msr_interception(struct vcpu_svm *svm, bool intercept) · 231 unmodified lines
813 if (intercept == svm->x2avic_msrs_intercepted)825 if (intercept == svm->x2avic_msrs_intercepted)
814 return;826 return;
815 827
816 if (avic_mode != AVIC_MODE_X2 ||828 if (!x2avic_enabled ||
817 !apic_x2apic_mode(svm->vcpu.arch.apic))829 !apic_x2apic_mode(svm->vcpu.arch.apic))
818 return;830 return;
819 831
@@ -1326,6 +1338,9 @@ static void __svm_vcpu_reset(struct kvm_vcpu *vcpu) · 506 unmodified lines
1326 vcpu->arch.microcode_version = 0x01000065;1338 vcpu->arch.microcode_version = 0x01000065;
1327 svm->tsc_ratio_msr = kvm_caps.default_tsc_scaling_ratio;1339 svm->tsc_ratio_msr = kvm_caps.default_tsc_scaling_ratio;
1328 1340
1341 svm->nmi_masked = false;
1342 svm->awaiting_iret_completion = false;
1343
1329 if (sev_es_guest(vcpu->kvm))1344 if (sev_es_guest(vcpu->kvm))
1330 sev_es_vcpu_reset(svm);1345 sev_es_vcpu_reset(svm);
1331}1346}
@@ -2076,7 +2091,7 @@ static void svm_handle_mce(struct kvm_vcpu *vcpu) · 744 unmodified lines
2076 * Erratum 383 triggered. Guest state is corrupt so kill the2091 * Erratum 383 triggered. Guest state is corrupt so kill the
2077 * guest.2092 * guest.
2078 */2093 */
2079 pr_err("KVM: Guest triggered AMD Erratum 383\n");2094 pr_err("Guest triggered AMD Erratum 383\n");
2080 2095
2081 kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);2096 kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
2082 2097
@@ -2470,7 +2485,7 @@ static int iret_interception(struct kvm_vcpu *vcpu) · 387 unmodified lines
2470 struct vcpu_svm *svm = to_svm(vcpu);2485 struct vcpu_svm *svm = to_svm(vcpu);
2471 2486
2472 ++vcpu->stat.nmi_window_exits;2487 ++vcpu->stat.nmi_window_exits;
2473 vcpu->arch.hflags |= HF_IRET_MASK;2488 svm->awaiting_iret_completion = true;
2474 if (!sev_es_guest(vcpu->kvm)) {2489 if (!sev_es_guest(vcpu->kvm)) {
2475 svm_clr_intercept(svm, INTERCEPT_IRET);2490 svm_clr_intercept(svm, INTERCEPT_IRET);
2476 svm->nmi_iret_rip = kvm_rip_read(vcpu);2491 svm->nmi_iret_rip = kvm_rip_read(vcpu);
@@ -3003,8 +3018,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) · 526 unmodified lines
3003 break;3018 break;
3004 case MSR_IA32_DEBUGCTLMSR:3019 case MSR_IA32_DEBUGCTLMSR:
3005 if (!lbrv) {3020 if (!lbrv) {
3006 vcpu_unimpl(vcpu, "%s: MSR_IA32_DEBUGCTL 0x%llx, nop\n",3021 kvm_pr_unimpl_wrmsr(vcpu, ecx, data);
3007 __func__, data);
3008 break;3022 break;
3009 }3023 }
3010 if (data & DEBUGCTL_RESERVED_BITS)3024 if (data & DEBUGCTL_RESERVED_BITS)
@@ -3033,7 +3047,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) · 22 unmodified lines
3033 case MSR_VM_CR:3047 case MSR_VM_CR:
3034 return svm_set_vm_cr(vcpu, data);3048 return svm_set_vm_cr(vcpu, data);
3035 case MSR_VM_IGNNE:3049 case MSR_VM_IGNNE:
3036 vcpu_unimpl(vcpu, "unimplemented wrmsr: 0x%x data 0x%llx\n", ecx, data);3050 kvm_pr_unimpl_wrmsr(vcpu, ecx, data);
3037 break;3051 break;
3038 case MSR_AMD64_DE_CFG: {3052 case MSR_AMD64_DE_CFG: {
3039 struct kvm_msr_entry msr_entry;3053 struct kvm_msr_entry msr_entry;
@@ -3466,7 +3480,7 @@ static void svm_inject_nmi(struct kvm_vcpu *vcpu) · 426 unmodified lines
3466 if (svm->nmi_l1_to_l2)3480 if (svm->nmi_l1_to_l2)
3467 return;3481 return;
3468 3482
3469 vcpu->arch.hflags |= HF_NMI_MASK;3483 svm->nmi_masked = true;
3470 if (!sev_es_guest(vcpu->kvm))3484 if (!sev_es_guest(vcpu->kvm))
3471 svm_set_intercept(svm, INTERCEPT_IRET);3485 svm_set_intercept(svm, INTERCEPT_IRET);
3472 ++vcpu->stat.nmi_injections;3486 ++vcpu->stat.nmi_injections;
@@ -3571,7 +3585,6 @@ bool svm_nmi_blocked(struct kvm_vcpu *vcpu) · 98 unmodified lines
3571{3585{
3572 struct vcpu_svm *svm = to_svm(vcpu);3586 struct vcpu_svm *svm = to_svm(vcpu);
3573 struct vmcb *vmcb = svm->vmcb;3587 struct vmcb *vmcb = svm->vmcb;
3574 bool ret;
3575 3588
3576 if (!gif_set(svm))3589 if (!gif_set(svm))
3577 return true;3590 return true;
@@ -3579,10 +3592,8 @@ bool svm_nmi_blocked(struct kvm_vcpu *vcpu) · 1 unmodified line
3579 if (is_guest_mode(vcpu) && nested_exit_on_nmi(svm))3592 if (is_guest_mode(vcpu) && nested_exit_on_nmi(svm))
3580 return false;3593 return false;
3581 3594
3582 ret = (vmcb->control.int_state & SVM_INTERRUPT_SHADOW_MASK) ||3595 return (vmcb->control.int_state & SVM_INTERRUPT_SHADOW_MASK) ||
3583 (vcpu->arch.hflags & HF_NMI_MASK);3596 svm->nmi_masked;
3584
3585 return ret;
3586}3597}
3587 3598
3588static int svm_nmi_allowed(struct kvm_vcpu *vcpu, bool for_injection)3599static int svm_nmi_allowed(struct kvm_vcpu *vcpu, bool for_injection)
@@ -3602,7 +3613,7 @@ static int svm_nmi_allowed(struct kvm_vcpu *vcpu, bool for_injection) · 13 unmodified lines
3602 3613
3603static bool svm_get_nmi_mask(struct kvm_vcpu *vcpu)3614static bool svm_get_nmi_mask(struct kvm_vcpu *vcpu)
3604{3615{
3605 return !!(vcpu->arch.hflags & HF_NMI_MASK);3616 return to_svm(vcpu)->nmi_masked;
3606}3617}
3607 3618
3608static void svm_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked)3619static void svm_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked)
@@ -3610,11 +3621,11 @@ static void svm_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked) · 1 unmodified line
3610 struct vcpu_svm *svm = to_svm(vcpu);3621 struct vcpu_svm *svm = to_svm(vcpu);
3611 3622
3612 if (masked) {3623 if (masked) {
3613 vcpu->arch.hflags |= HF_NMI_MASK;3624 svm->nmi_masked = true;
3614 if (!sev_es_guest(vcpu->kvm))3625 if (!sev_es_guest(vcpu->kvm))
3615 svm_set_intercept(svm, INTERCEPT_IRET);3626 svm_set_intercept(svm, INTERCEPT_IRET);
3616 } else {3627 } else {
3617 vcpu->arch.hflags &= ~HF_NMI_MASK;3628 svm->nmi_masked = false;
3618 if (!sev_es_guest(vcpu->kvm))3629 if (!sev_es_guest(vcpu->kvm))
3619 svm_clr_intercept(svm, INTERCEPT_IRET);3630 svm_clr_intercept(svm, INTERCEPT_IRET);
3620 }3631 }
@@ -3700,7 +3711,7 @@ static void svm_enable_nmi_window(struct kvm_vcpu *vcpu) · 79 unmodified lines
3700{3711{
3701 struct vcpu_svm *svm = to_svm(vcpu);3712 struct vcpu_svm *svm = to_svm(vcpu);
3702 3713
3703 if ((vcpu->arch.hflags & (HF_NMI_MASK | HF_IRET_MASK)) == HF_NMI_MASK)3714 if (svm->nmi_masked && !svm->awaiting_iret_completion)
3704 return; /* IRET will cause a vm exit */3715 return; /* IRET will cause a vm exit */
3705 3716
3706 if (!gif_set(svm)) {3717 if (!gif_set(svm)) {
@@ -3824,10 +3835,11 @@ static void svm_complete_interrupts(struct kvm_vcpu *vcpu) · 117 unmodified lines
3824 * If we've made progress since setting HF_IRET_MASK, we've3835 * If we've made progress since setting HF_IRET_MASK, we've
3825 * executed an IRET and can allow NMI injection.3836 * executed an IRET and can allow NMI injection.
3826 */3837 */
3827 if ((vcpu->arch.hflags & HF_IRET_MASK) &&3838 if (svm->awaiting_iret_completion &&
3828 (sev_es_guest(vcpu->kvm) ||3839 (sev_es_guest(vcpu->kvm) ||
3829 kvm_rip_read(vcpu) != svm->nmi_iret_rip)) {3840 kvm_rip_read(vcpu) != svm->nmi_iret_rip)) {
3830 vcpu->arch.hflags &= ~(HF_NMI_MASK | HF_IRET_MASK);3841 svm->awaiting_iret_completion = false;
3842 svm->nmi_masked = false;
3831 kvm_make_request(KVM_REQ_EVENT, vcpu);3843 kvm_make_request(KVM_REQ_EVENT, vcpu);
3832 }3844 }
3833 3845
@@ -4076,17 +4088,6 @@ static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, · 242 unmodified lines
4076 vmcb_mark_dirty(svm->vmcb, VMCB_CR);4088 vmcb_mark_dirty(svm->vmcb, VMCB_CR);
4077}4089}
4078 4090
4079static int is_disabled(void)
4080{
4081 u64 vm_cr;
4082
4083 rdmsrl(MSR_VM_CR, vm_cr);
4084 if (vm_cr & (1 << SVM_VM_CR_SVM_DISABLE))
4085 return 1;
4086
4087 return 0;
4088}
4089
4090static void4091static void
4091svm_patch_hypercall(struct kvm_vcpu *vcpu, unsigned char *hypercall)4092svm_patch_hypercall(struct kvm_vcpu *vcpu, unsigned char *hypercall)
4092{4093{
@@ -4098,11 +4099,6 @@ svm_patch_hypercall(struct kvm_vcpu *vcpu, unsigned char *hypercall) · 5 unmodified lines
4098 hypercall[2] = 0xd9;4099 hypercall[2] = 0xd9;
4099}4100}
4100 4101
4101static int __init svm_check_processor_compat(void)
4102{
4103 return 0;
4104}
4105
4106/*4102/*
4107 * The kvm parameter can be NULL (module initialization, or invocation before4103 * The kvm parameter can be NULL (module initialization, or invocation before
4108 * VM creation). Be sure to check the kvm parameter before using it.4104 * VM creation). Be sure to check the kvm parameter before using it.
@@ -4629,7 +4625,7 @@ static bool svm_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type, · 520 unmodified lines
4629 smap = cr4 & X86_CR4_SMAP;4625 smap = cr4 & X86_CR4_SMAP;
4630 is_user = svm_get_cpl(vcpu) == 3;4626 is_user = svm_get_cpl(vcpu) == 3;
4631 if (smap && (!smep || is_user)) {4627 if (smap && (!smep || is_user)) {
4632 pr_err_ratelimited("KVM: SEV Guest triggered AMD Erratum 1096\n");4628 pr_err_ratelimited("SEV Guest triggered AMD Erratum 1096\n");
4633 4629
4634 /*4630 /*
4635 * If the fault occurred in userspace, arbitrarily inject #GP4631 * If the fault occurred in userspace, arbitrarily inject #GP
@@ -4701,7 +4697,9 @@ static int svm_vm_init(struct kvm *kvm) · 65 unmodified lines
4701}4697}
4702 4698
4703static struct kvm_x86_ops svm_x86_ops __initdata = {4699static struct kvm_x86_ops svm_x86_ops __initdata = {
4704 .name = "kvm_amd",4700 .name = KBUILD_MODNAME,
4701
4702 .check_processor_compatibility = svm_check_processor_compat,
4705 4703
4706 .hardware_unsetup = svm_hardware_unsetup,4704 .hardware_unsetup = svm_hardware_unsetup,
4707 .hardware_enable = svm_hardware_enable,4705 .hardware_enable = svm_hardware_enable,
@@ -4771,10 +4769,10 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { · 63 unmodified lines
4771 .enable_nmi_window = svm_enable_nmi_window,4769 .enable_nmi_window = svm_enable_nmi_window,
4772 .enable_irq_window = svm_enable_irq_window,4770 .enable_irq_window = svm_enable_irq_window,
4773 .update_cr8_intercept = svm_update_cr8_intercept,4771 .update_cr8_intercept = svm_update_cr8_intercept,
4774 .set_virtual_apic_mode = avic_set_virtual_apic_mode,4772 .set_virtual_apic_mode = avic_refresh_virtual_apic_mode,
4775 .refresh_apicv_exec_ctrl = avic_refresh_apicv_exec_ctrl,4773 .refresh_apicv_exec_ctrl = avic_refresh_apicv_exec_ctrl,
4776 .check_apicv_inhibit_reasons = avic_check_apicv_inhibit_reasons,
4777 .apicv_post_state_restore = avic_apicv_post_state_restore,4774 .apicv_post_state_restore = avic_apicv_post_state_restore,
4775 .required_apicv_inhibits = AVIC_REQUIRED_APICV_INHIBITS,
4778 4776
4779 .get_exit_info = svm_get_exit_info,4777 .get_exit_info = svm_get_exit_info,
4780 4778
@@ -4981,7 +4979,7 @@ static __init int svm_hardware_setup(void) · 200 unmodified lines
4981 }4979 }
4982 4980
4983 if (nested) {4981 if (nested) {
4984 printk(KERN_INFO "kvm: Nested Virtualization enabled\n");4982 pr_info("Nested Virtualization enabled\n");
4985 kvm_enable_efer_bits(EFER_SVME | EFER_LMSLE);4983 kvm_enable_efer_bits(EFER_SVME | EFER_LMSLE);
4986 }4984 }
4987 4985
@@ -4999,7 +4997,7 @@ static __init int svm_hardware_setup(void) · 11 unmodified lines
4999 /* Force VM NPT level equal to the host's paging level */4997 /* Force VM NPT level equal to the host's paging level */
5000 kvm_configure_mmu(npt_enabled, get_npt_level(),4998 kvm_configure_mmu(npt_enabled, get_npt_level(),
5001 get_npt_level(), PG_LEVEL_1G);4999 get_npt_level(), PG_LEVEL_1G);
5002 pr_info("kvm: Nested Paging %sabled\n", npt_enabled ? "en" : "dis");5000 pr_info("Nested Paging %sabled\n", npt_enabled ? "en" : "dis");
5003 5001
5004 /* Setup shadow_me_value and shadow_me_mask */5002 /* Setup shadow_me_value and shadow_me_mask */
5005 kvm_mmu_set_me_spte_mask(sme_me_mask, sme_me_mask);5003 kvm_mmu_set_me_spte_mask(sme_me_mask, sme_me_mask);
@@ -5025,12 +5023,14 @@ static __init int svm_hardware_setup(void) · 19 unmodified lines
5025 nrips = false;5023 nrips = false;
5026 }5024 }
5027 5025
5028 enable_apicv = avic = avic && avic_hardware_setup(&svm_x86_ops);5026 enable_apicv = avic = avic && avic_hardware_setup();
5029 5027
5030 if (!enable_apicv) {5028 if (!enable_apicv) {
5031 svm_x86_ops.vcpu_blocking = NULL;5029 svm_x86_ops.vcpu_blocking = NULL;
5032 svm_x86_ops.vcpu_unblocking = NULL;5030 svm_x86_ops.vcpu_unblocking = NULL;
5033 svm_x86_ops.vcpu_get_apicv_inhibit_reasons = NULL;5031 svm_x86_ops.vcpu_get_apicv_inhibit_reasons = NULL;
5032 } else if (!x2avic_enabled) {
5033 svm_x86_ops.allow_apicv_in_x2apic_without_x2apic_virtualization = true;
5034 }5034 }
5035 5035
5036 if (vls) {5036 if (vls) {
@@ -5089,10 +5089,7 @@ err: · 52 unmodified lines
5089 5089
5090 5090
5091static struct kvm_x86_init_ops svm_init_ops __initdata = {5091static struct kvm_x86_init_ops svm_init_ops __initdata = {
5092 .cpu_has_kvm_support = has_svm,
5093 .disabled_by_bios = is_disabled,
5094 .hardware_setup = svm_hardware_setup,5092 .hardware_setup = svm_hardware_setup,
5095 .check_processor_compatibility = svm_check_processor_compat,
5096 5093
5097 .runtime_ops = &svm_x86_ops,5094 .runtime_ops = &svm_x86_ops,
5098 .pmu_ops = &amd_pmu_ops,5095 .pmu_ops = &amd_pmu_ops,
@@ -5100,15 +5097,37 @@ static struct kvm_x86_init_ops svm_init_ops __initdata = { · 1 unmodified line
5100 5097
5101static int __init svm_init(void)5098static int __init svm_init(void)
5102{5099{
5100 int r;
5101
5103 __unused_size_checks();5102 __unused_size_checks();
5104 5103
5105 return kvm_init(&svm_init_ops, sizeof(struct vcpu_svm),5104 if (!kvm_is_svm_supported())
5106 __alignof__(struct vcpu_svm), THIS_MODULE);5105 return -EOPNOTSUPP;
5106
5107 r = kvm_x86_vendor_init(&svm_init_ops);
5108 if (r)
5109 return r;
5110
5111 /*
5112 * Common KVM initialization _must_ come last, after this, /dev/kvm is
5113 * exposed to userspace!
5114 */
5115 r = kvm_init(sizeof(struct vcpu_svm), __alignof__(struct vcpu_svm),
5116 THIS_MODULE);
5117 if (r)
5118 goto err_kvm_init;
5119
5120 return 0;
5121
5122err_kvm_init:
5123 kvm_x86_vendor_exit();
5124 return r;
5107}5125}
5108 5126
5109static void __exit svm_exit(void)5127static void __exit svm_exit(void)
5110{5128{
5111 kvm_exit();5129 kvm_exit();
5130 kvm_x86_vendor_exit();
5112}5131}
5113 5132
5114module_init(svm_init)5133module_init(svm_init)
arch/x86/kvm/svm/svm.h+43−15
@@ -35,14 +35,7 @@ extern u32 msrpm_offsets[MSRPM_OFFSETS] __read_mostly; · 34 unmodified lines
35extern bool npt_enabled;35extern bool npt_enabled;
36extern int vgif;36extern int vgif;
37extern bool intercept_smi;37extern bool intercept_smi;
38 38extern bool x2avic_enabled;
39enum avic_modes {
40 AVIC_MODE_NONE = 0,
41 AVIC_MODE_X1,
42 AVIC_MODE_X2,
43};
44
45extern enum avic_modes avic_mode;
46 39
47/*40/*
48 * Clean bits in VMCB.41 * Clean bits in VMCB.
@@ -237,8 +230,26 @@ struct vcpu_svm { · 188 unmodified lines
237 230
238 struct svm_nested_state nested;231 struct svm_nested_state nested;
239 232
233 /* NMI mask value, used when vNMI is not enabled */
234 bool nmi_masked;
235
236 /*
237 * True when NMIs are still masked but guest IRET was just intercepted
238 * and KVM is waiting for RIP to change, which will signal that the
239 * intercepted IRET was retired and thus NMI can be unmasked.
240 */
241 bool awaiting_iret_completion;
242
243 /*
244 * Set when KVM is awaiting IRET completion and needs to inject NMIs as
245 * soon as the IRET completes (e.g. NMI is pending injection). KVM
246 * temporarily steals RFLAGS.TF to single-step the guest in this case
247 * in order to regain control as soon as the NMI-blocking condition
248 * goes away.
249 */
240 bool nmi_singlestep;250 bool nmi_singlestep;
241 u64 nmi_singlestep_guest_rflags;251 u64 nmi_singlestep_guest_rflags;
252
242 bool nmi_l1_to_l2;253 bool nmi_l1_to_l2;
243 254
244 unsigned long soft_int_csbase;255 unsigned long soft_int_csbase;
@@ -280,6 +291,9 @@ struct vcpu_svm { · 35 unmodified lines
280 bool guest_state_loaded;291 bool guest_state_loaded;
281 292
282 bool x2avic_msrs_intercepted;293 bool x2avic_msrs_intercepted;
294
295 /* Guest GIF value, used when vGIF is not enabled */
296 bool guest_gif;
283};297};
284 298
285struct svm_cpu_data {299struct svm_cpu_data {
@@ -497,7 +511,7 @@ static inline void enable_gif(struct vcpu_svm *svm) · 211 unmodified lines
497 if (vmcb)511 if (vmcb)
498 vmcb->control.int_ctl |= V_GIF_MASK;512 vmcb->control.int_ctl |= V_GIF_MASK;
499 else513 else
500 svm->vcpu.arch.hflags |= HF_GIF_MASK;514 svm->guest_gif = true;
501}515}
502 516
503static inline void disable_gif(struct vcpu_svm *svm)517static inline void disable_gif(struct vcpu_svm *svm)
@@ -507,7 +521,7 @@ static inline void disable_gif(struct vcpu_svm *svm) · 3 unmodified lines
507 if (vmcb)521 if (vmcb)
508 vmcb->control.int_ctl &= ~V_GIF_MASK;522 vmcb->control.int_ctl &= ~V_GIF_MASK;
509 else523 else
510 svm->vcpu.arch.hflags &= ~HF_GIF_MASK;524 svm->guest_gif = false;
511}525}
512 526
513static inline bool gif_set(struct vcpu_svm *svm)527static inline bool gif_set(struct vcpu_svm *svm)
@@ -517,7 +531,7 @@ static inline bool gif_set(struct vcpu_svm *svm) · 3 unmodified lines
517 if (vmcb)531 if (vmcb)
518 return !!(vmcb->control.int_ctl & V_GIF_MASK);532 return !!(vmcb->control.int_ctl & V_GIF_MASK);
519 else533 else
520 return !!(svm->vcpu.arch.hflags & HF_GIF_MASK);534 return svm->guest_gif;
521}535}
522 536
523static inline bool nested_npt_enabled(struct vcpu_svm *svm)537static inline bool nested_npt_enabled(struct vcpu_svm *svm)
@@ -628,8 +642,23 @@ void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_vmcb); · 104 unmodified lines
628extern struct kvm_x86_nested_ops svm_nested_ops;642extern struct kvm_x86_nested_ops svm_nested_ops;
629 643
630/* avic.c */644/* avic.c */
631 645#define AVIC_REQUIRED_APICV_INHIBITS \
632bool avic_hardware_setup(struct kvm_x86_ops *ops);646( \
647 BIT(APICV_INHIBIT_REASON_DISABLE) | \
648 BIT(APICV_INHIBIT_REASON_ABSENT) | \
649 BIT(APICV_INHIBIT_REASON_HYPERV) | \
650 BIT(APICV_INHIBIT_REASON_NESTED) | \
651 BIT(APICV_INHIBIT_REASON_IRQWIN) | \
652 BIT(APICV_INHIBIT_REASON_PIT_REINJ) | \
653 BIT(APICV_INHIBIT_REASON_BLOCKIRQ) | \
654 BIT(APICV_INHIBIT_REASON_SEV) | \
655 BIT(APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED) | \
656 BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) | \
657 BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED) | \
658 BIT(APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED) \
659)
660
661bool avic_hardware_setup(void);
633int avic_ga_log_notifier(u32 ga_tag);662int avic_ga_log_notifier(u32 ga_tag);
634void avic_vm_destroy(struct kvm *kvm);663void avic_vm_destroy(struct kvm *kvm);
635int avic_vm_init(struct kvm *kvm);664int avic_vm_init(struct kvm *kvm);
@@ -641,14 +670,13 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu); · 5 unmodified lines
641void avic_vcpu_put(struct kvm_vcpu *vcpu);670void avic_vcpu_put(struct kvm_vcpu *vcpu);
642void avic_apicv_post_state_restore(struct kvm_vcpu *vcpu);671void avic_apicv_post_state_restore(struct kvm_vcpu *vcpu);
643void avic_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu);672void avic_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu);
644bool avic_check_apicv_inhibit_reasons(enum kvm_apicv_inhibit reason);
645int avic_pi_update_irte(struct kvm *kvm, unsigned int host_irq,673int avic_pi_update_irte(struct kvm *kvm, unsigned int host_irq,
646 uint32_t guest_irq, bool set);674 uint32_t guest_irq, bool set);
647void avic_vcpu_blocking(struct kvm_vcpu *vcpu);675void avic_vcpu_blocking(struct kvm_vcpu *vcpu);
648void avic_vcpu_unblocking(struct kvm_vcpu *vcpu);676void avic_vcpu_unblocking(struct kvm_vcpu *vcpu);
649void avic_ring_doorbell(struct kvm_vcpu *vcpu);677void avic_ring_doorbell(struct kvm_vcpu *vcpu);
650unsigned long avic_vcpu_get_apicv_inhibit_reasons(struct kvm_vcpu *vcpu);678unsigned long avic_vcpu_get_apicv_inhibit_reasons(struct kvm_vcpu *vcpu);
651void avic_set_virtual_apic_mode(struct kvm_vcpu *vcpu);679void avic_refresh_virtual_apic_mode(struct kvm_vcpu *vcpu);
652 680
653 681
654/* sev.c */682/* sev.c */
arch/x86/kvm/svm/svm_onhyperv.c+1−0
@@ -2,6 +2,7 @@ · 1 unmodified line
2/*2/*
3 * KVM L1 hypervisor optimizations on Hyper-V for SVM.3 * KVM L1 hypervisor optimizations on Hyper-V for SVM.
4 */4 */
5#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
5 6
6#include <linux/kvm_host.h>7#include <linux/kvm_host.h>
7 8
arch/x86/kvm/svm/svm_onhyperv.h+4−4
@@ -30,11 +30,11 @@ static inline void svm_hv_init_vmcb(struct vmcb *vmcb) · 29 unmodified lines
30 hve->hv_enlightenments_control.msr_bitmap = 1;30 hve->hv_enlightenments_control.msr_bitmap = 1;
31}31}
32 32
33static inline void svm_hv_hardware_setup(void)33static inline __init void svm_hv_hardware_setup(void)
34{34{
35 if (npt_enabled &&35 if (npt_enabled &&
36 ms_hyperv.nested_features & HV_X64_NESTED_ENLIGHTENED_TLB) {36 ms_hyperv.nested_features & HV_X64_NESTED_ENLIGHTENED_TLB) {
37 pr_info("kvm: Hyper-V enlightened NPT TLB flush enabled\n");37 pr_info(KBUILD_MODNAME ": Hyper-V enlightened NPT TLB flush enabled\n");
38 svm_x86_ops.tlb_remote_flush = hv_remote_flush_tlb;38 svm_x86_ops.tlb_remote_flush = hv_remote_flush_tlb;
39 svm_x86_ops.tlb_remote_flush_with_range =39 svm_x86_ops.tlb_remote_flush_with_range =
40 hv_remote_flush_tlb_with_range;40 hv_remote_flush_tlb_with_range;
@@ -43,7 +43,7 @@ static inline void svm_hv_hardware_setup(void) · 2 unmodified lines
43 if (ms_hyperv.nested_features & HV_X64_NESTED_DIRECT_FLUSH) {43 if (ms_hyperv.nested_features & HV_X64_NESTED_DIRECT_FLUSH) {
44 int cpu;44 int cpu;
45 45
46 pr_info("kvm: Hyper-V Direct TLB Flush enabled\n");46 pr_info(KBUILD_MODNAME ": Hyper-V Direct TLB Flush enabled\n");
47 for_each_online_cpu(cpu) {47 for_each_online_cpu(cpu) {
48 struct hv_vp_assist_page *vp_ap =48 struct hv_vp_assist_page *vp_ap =
49 hv_get_vp_assist_page(cpu);49 hv_get_vp_assist_page(cpu);
@@ -84,7 +84,7 @@ static inline void svm_hv_init_vmcb(struct vmcb *vmcb) · 34 unmodified lines
84{84{
85}85}
86 86
87static inline void svm_hv_hardware_setup(void)87static inline __init void svm_hv_hardware_setup(void)
88{88{
89}89}
90 90
arch/x86/kvm/vmx/capabilities.h+2−2
@@ -66,13 +66,13 @@ struct vmcs_config { · 65 unmodified lines
66 u64 misc;66 u64 misc;
67 struct nested_vmx_msrs nested;67 struct nested_vmx_msrs nested;
68};68};
69extern struct vmcs_config vmcs_config;69extern struct vmcs_config vmcs_config __ro_after_init;
70 70
71struct vmx_capability {71struct vmx_capability {
72 u32 ept;72 u32 ept;
73 u32 vpid;73 u32 vpid;
74};74};
75extern struct vmx_capability vmx_capability;75extern struct vmx_capability vmx_capability __ro_after_init;
76 76
77static inline bool cpu_has_vmx_basic_inout(void)77static inline bool cpu_has_vmx_basic_inout(void)
78{78{
arch/x86/kvm/vmx/hyperv.c+68−19
@@ -1,4 +1,5 @@
1// SPDX-License-Identifier: GPL-2.01// SPDX-License-Identifier: GPL-2.0
2#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
2 3
3#include <linux/errno.h>4#include <linux/errno.h>
4#include <linux/smp.h>5#include <linux/smp.h>
@@ -361,35 +362,43 @@ enum evmcs_revision { · 356 unmodified lines
361enum evmcs_ctrl_type {362enum evmcs_ctrl_type {
362 EVMCS_EXIT_CTRLS,363 EVMCS_EXIT_CTRLS,
363 EVMCS_ENTRY_CTRLS,364 EVMCS_ENTRY_CTRLS,
365 EVMCS_EXEC_CTRL,
364 EVMCS_2NDEXEC,366 EVMCS_2NDEXEC,
367 EVMCS_3RDEXEC,
365 EVMCS_PINCTRL,368 EVMCS_PINCTRL,
366 EVMCS_VMFUNC,369 EVMCS_VMFUNC,
367 NR_EVMCS_CTRLS,370 NR_EVMCS_CTRLS,
368};371};
369 372
370static const u32 evmcs_unsupported_ctrls[NR_EVMCS_CTRLS][NR_EVMCS_REVISIONS] = {373static const u32 evmcs_supported_ctrls[NR_EVMCS_CTRLS][NR_EVMCS_REVISIONS] = {
371 [EVMCS_EXIT_CTRLS] = {374 [EVMCS_EXIT_CTRLS] = {
372 [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_VMEXIT_CTRL,375 [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_VMEXIT_CTRL,
373 },376 },
374 [EVMCS_ENTRY_CTRLS] = {377 [EVMCS_ENTRY_CTRLS] = {
375 [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_VMENTRY_CTRL,378 [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_VMENTRY_CTRL,
379 },
380 [EVMCS_EXEC_CTRL] = {
381 [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_EXEC_CTRL,
376 },382 },
377 [EVMCS_2NDEXEC] = {383 [EVMCS_2NDEXEC] = {
378 [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_2NDEXEC,384 [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_2NDEXEC & ~SECONDARY_EXEC_TSC_SCALING,
385 },
386 [EVMCS_3RDEXEC] = {
387 [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_3RDEXEC,
379 },388 },
380 [EVMCS_PINCTRL] = {389 [EVMCS_PINCTRL] = {
381 [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_PINCTRL,390 [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_PINCTRL,
382 },391 },
383 [EVMCS_VMFUNC] = {392 [EVMCS_VMFUNC] = {
384 [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_VMFUNC,393 [EVMCSv1_LEGACY] = EVMCS1_SUPPORTED_VMFUNC,
385 },394 },
386};395};
387 396
388static u32 evmcs_get_unsupported_ctls(enum evmcs_ctrl_type ctrl_type)397static u32 evmcs_get_supported_ctls(enum evmcs_ctrl_type ctrl_type)
389{398{
390 enum evmcs_revision evmcs_rev = EVMCSv1_LEGACY;399 enum evmcs_revision evmcs_rev = EVMCSv1_LEGACY;
391 400
392 return evmcs_unsupported_ctrls[ctrl_type][evmcs_rev];401 return evmcs_supported_ctrls[ctrl_type][evmcs_rev];
393}402}
394 403
395static bool evmcs_has_perf_global_ctrl(struct kvm_vcpu *vcpu)404static bool evmcs_has_perf_global_ctrl(struct kvm_vcpu *vcpu)
@@ -413,7 +422,7 @@ void nested_evmcs_filter_control_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 * · 17 unmodified lines
413{422{
414 u32 ctl_low = (u32)*pdata;423 u32 ctl_low = (u32)*pdata;
415 u32 ctl_high = (u32)(*pdata >> 32);424 u32 ctl_high = (u32)(*pdata >> 32);
416 u32 unsupported_ctrls;425 u32 supported_ctrls;
417 426
418 /*427 /*
419 * Hyper-V 2016 and 2019 try using these features even when eVMCS428 * Hyper-V 2016 and 2019 try using these features even when eVMCS
@@ -422,27 +431,31 @@ void nested_evmcs_filter_control_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 * · 2 unmodified lines
422 switch (msr_index) {431 switch (msr_index) {
423 case MSR_IA32_VMX_EXIT_CTLS:432 case MSR_IA32_VMX_EXIT_CTLS:
424 case MSR_IA32_VMX_TRUE_EXIT_CTLS:433 case MSR_IA32_VMX_TRUE_EXIT_CTLS:
425 unsupported_ctrls = evmcs_get_unsupported_ctls(EVMCS_EXIT_CTRLS);434 supported_ctrls = evmcs_get_supported_ctls(EVMCS_EXIT_CTRLS);
426 if (!evmcs_has_perf_global_ctrl(vcpu))435 if (!evmcs_has_perf_global_ctrl(vcpu))
427 unsupported_ctrls |= VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL;436 supported_ctrls &= ~VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL;
428 ctl_high &= ~unsupported_ctrls;437 ctl_high &= supported_ctrls;
429 break;438 break;
430 case MSR_IA32_VMX_ENTRY_CTLS:439 case MSR_IA32_VMX_ENTRY_CTLS:
431 case MSR_IA32_VMX_TRUE_ENTRY_CTLS:440 case MSR_IA32_VMX_TRUE_ENTRY_CTLS:
432 unsupported_ctrls = evmcs_get_unsupported_ctls(EVMCS_ENTRY_CTRLS);441 supported_ctrls = evmcs_get_supported_ctls(EVMCS_ENTRY_CTRLS);
433 if (!evmcs_has_perf_global_ctrl(vcpu))442 if (!evmcs_has_perf_global_ctrl(vcpu))
434 unsupported_ctrls |= VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL;443 supported_ctrls &= ~VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL;
435 ctl_high &= ~unsupported_ctrls;444 ctl_high &= supported_ctrls;
445 break;
446 case MSR_IA32_VMX_PROCBASED_CTLS:
447 case MSR_IA32_VMX_TRUE_PROCBASED_CTLS:
448 ctl_high &= evmcs_get_supported_ctls(EVMCS_EXEC_CTRL);
436 break;449 break;
437 case MSR_IA32_VMX_PROCBASED_CTLS2:450 case MSR_IA32_VMX_PROCBASED_CTLS2:
438 ctl_high &= ~evmcs_get_unsupported_ctls(EVMCS_2NDEXEC);451 ctl_high &= evmcs_get_supported_ctls(EVMCS_2NDEXEC);
439 break;452 break;
440 case MSR_IA32_VMX_TRUE_PINBASED_CTLS:453 case MSR_IA32_VMX_TRUE_PINBASED_CTLS:
441 case MSR_IA32_VMX_PINBASED_CTLS:454 case MSR_IA32_VMX_PINBASED_CTLS:
442 ctl_high &= ~evmcs_get_unsupported_ctls(EVMCS_PINCTRL);455 ctl_high &= evmcs_get_supported_ctls(EVMCS_PINCTRL);
443 break;456 break;
444 case MSR_IA32_VMX_VMFUNC:457 case MSR_IA32_VMX_VMFUNC:
445 ctl_low &= ~evmcs_get_unsupported_ctls(EVMCS_VMFUNC);458 ctl_low &= evmcs_get_supported_ctls(EVMCS_VMFUNC);
446 break;459 break;
447 }460 }
448 461
@@ -452,7 +465,7 @@ void nested_evmcs_filter_control_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 * · 3 unmodified lines
452static bool nested_evmcs_is_valid_controls(enum evmcs_ctrl_type ctrl_type,465static bool nested_evmcs_is_valid_controls(enum evmcs_ctrl_type ctrl_type,
453 u32 val)466 u32 val)
454{467{
455 return !(val & evmcs_get_unsupported_ctls(ctrl_type));468 return !(val & ~evmcs_get_supported_ctls(ctrl_type));
456}469}
457 470
458int nested_evmcs_check_controls(struct vmcs12 *vmcs12)471int nested_evmcs_check_controls(struct vmcs12 *vmcs12)
@@ -461,6 +474,10 @@ int nested_evmcs_check_controls(struct vmcs12 *vmcs12) · 2 unmodified lines
461 vmcs12->pin_based_vm_exec_control)))474 vmcs12->pin_based_vm_exec_control)))
462 return -EINVAL;475 return -EINVAL;
463 476
477 if (CC(!nested_evmcs_is_valid_controls(EVMCS_EXEC_CTRL,
478 vmcs12->cpu_based_vm_exec_control)))
479 return -EINVAL;
480
464 if (CC(!nested_evmcs_is_valid_controls(EVMCS_2NDEXEC,481 if (CC(!nested_evmcs_is_valid_controls(EVMCS_2NDEXEC,
465 vmcs12->secondary_vm_exec_control)))482 vmcs12->secondary_vm_exec_control)))
466 return -EINVAL;483 return -EINVAL;
@@ -488,6 +505,38 @@ int nested_evmcs_check_controls(struct vmcs12 *vmcs12) · 21 unmodified lines
488 return 0;505 return 0;
489}506}
490 507
508#if IS_ENABLED(CONFIG_HYPERV)
509/*
510 * KVM on Hyper-V always uses the latest known eVMCSv1 revision, the assumption
511 * is: in case a feature has corresponding fields in eVMCS described and it was
512 * exposed in VMX feature MSRs, KVM is free to use it. Warn if KVM meets a
513 * feature which has no corresponding eVMCS field, this likely means that KVM
514 * needs to be updated.
515 */
516#define evmcs_check_vmcs_conf(field, ctrl) \
517 do { \
518 typeof(vmcs_conf->field) unsupported; \
519 \
520 unsupported = vmcs_conf->field & ~EVMCS1_SUPPORTED_ ## ctrl; \
521 if (unsupported) { \
522 pr_warn_once(#field " unsupported with eVMCS: 0x%llx\n",\
523 (u64)unsupported); \
524 vmcs_conf->field &= EVMCS1_SUPPORTED_ ## ctrl; \
525 } \
526 } \
527 while (0)
528
529void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf)
530{
531 evmcs_check_vmcs_conf(cpu_based_exec_ctrl, EXEC_CTRL);
532 evmcs_check_vmcs_conf(pin_based_exec_ctrl, PINCTRL);
533 evmcs_check_vmcs_conf(cpu_based_2nd_exec_ctrl, 2NDEXEC);
534 evmcs_check_vmcs_conf(cpu_based_3rd_exec_ctrl, 3RDEXEC);
535 evmcs_check_vmcs_conf(vmentry_ctrl, VMENTRY_CTRL);
536 evmcs_check_vmcs_conf(vmexit_ctrl, VMEXIT_CTRL);
537}
538#endif
539
491int nested_enable_evmcs(struct kvm_vcpu *vcpu,540int nested_enable_evmcs(struct kvm_vcpu *vcpu,
492 uint16_t *vmcs_version)541 uint16_t *vmcs_version)
493{542{
arch/x86/kvm/vmx/hyperv.h+89−39
@@ -48,22 +48,84 @@ DECLARE_STATIC_KEY_FALSE(enable_evmcs); · 47 unmodified lines
48 * Currently unsupported in KVM:48 * Currently unsupported in KVM:
49 * GUEST_IA32_RTIT_CTL = 0x00002814,49 * GUEST_IA32_RTIT_CTL = 0x00002814,
50 */50 */
51#define EVMCS1_UNSUPPORTED_PINCTRL (PIN_BASED_POSTED_INTR | \51#define EVMCS1_SUPPORTED_PINCTRL \
52 PIN_BASED_VMX_PREEMPTION_TIMER)52 (PIN_BASED_ALWAYSON_WITHOUT_TRUE_MSR | \
53#define EVMCS1_UNSUPPORTED_EXEC_CTRL (CPU_BASED_ACTIVATE_TERTIARY_CONTROLS)53 PIN_BASED_EXT_INTR_MASK | \
54#define EVMCS1_UNSUPPORTED_2NDEXEC \54 PIN_BASED_NMI_EXITING | \
55 (SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY | \55 PIN_BASED_VIRTUAL_NMIS)
56 SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES | \56
57 SECONDARY_EXEC_APIC_REGISTER_VIRT | \57#define EVMCS1_SUPPORTED_EXEC_CTRL \
58 SECONDARY_EXEC_ENABLE_PML | \58 (CPU_BASED_ALWAYSON_WITHOUT_TRUE_MSR | \
59 SECONDARY_EXEC_ENABLE_VMFUNC | \59 CPU_BASED_HLT_EXITING | \
60 SECONDARY_EXEC_SHADOW_VMCS | \60 CPU_BASED_CR3_LOAD_EXITING | \
61 CPU_BASED_CR3_STORE_EXITING | \
62 CPU_BASED_UNCOND_IO_EXITING | \
63 CPU_BASED_MOV_DR_EXITING | \
64 CPU_BASED_USE_TSC_OFFSETTING | \
65 CPU_BASED_MWAIT_EXITING | \
66 CPU_BASED_MONITOR_EXITING | \
67 CPU_BASED_INVLPG_EXITING | \
68 CPU_BASED_RDPMC_EXITING | \
69 CPU_BASED_INTR_WINDOW_EXITING | \
70 CPU_BASED_CR8_LOAD_EXITING | \
71 CPU_BASED_CR8_STORE_EXITING | \
72 CPU_BASED_RDTSC_EXITING | \
73 CPU_BASED_TPR_SHADOW | \
74 CPU_BASED_USE_IO_BITMAPS | \
75 CPU_BASED_MONITOR_TRAP_FLAG | \
76 CPU_BASED_USE_MSR_BITMAPS | \
77 CPU_BASED_NMI_WINDOW_EXITING | \
78 CPU_BASED_PAUSE_EXITING | \
79 CPU_BASED_ACTIVATE_SECONDARY_CONTROLS)
80
81#define EVMCS1_SUPPORTED_2NDEXEC \
82 (SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE | \
83 SECONDARY_EXEC_WBINVD_EXITING | \
84 SECONDARY_EXEC_ENABLE_VPID | \
85 SECONDARY_EXEC_ENABLE_EPT | \
86 SECONDARY_EXEC_UNRESTRICTED_GUEST | \
87 SECONDARY_EXEC_DESC | \
88 SECONDARY_EXEC_ENABLE_RDTSCP | \
89 SECONDARY_EXEC_ENABLE_INVPCID | \
90 SECONDARY_EXEC_XSAVES | \
91 SECONDARY_EXEC_RDSEED_EXITING | \
92 SECONDARY_EXEC_RDRAND_EXITING | \
61 SECONDARY_EXEC_TSC_SCALING | \93 SECONDARY_EXEC_TSC_SCALING | \
62 SECONDARY_EXEC_PAUSE_LOOP_EXITING)94 SECONDARY_EXEC_ENABLE_USR_WAIT_PAUSE | \
63#define EVMCS1_UNSUPPORTED_VMEXIT_CTRL \95 SECONDARY_EXEC_PT_USE_GPA | \
64 (VM_EXIT_SAVE_VMX_PREEMPTION_TIMER)96 SECONDARY_EXEC_PT_CONCEAL_VMX | \
65#define EVMCS1_UNSUPPORTED_VMENTRY_CTRL (0)97 SECONDARY_EXEC_BUS_LOCK_DETECTION | \
66#define EVMCS1_UNSUPPORTED_VMFUNC (VMX_VMFUNC_EPTP_SWITCHING)98 SECONDARY_EXEC_NOTIFY_VM_EXITING | \
99 SECONDARY_EXEC_ENCLS_EXITING)
100
101#define EVMCS1_SUPPORTED_3RDEXEC (0ULL)
102
103#define EVMCS1_SUPPORTED_VMEXIT_CTRL \
104 (VM_EXIT_ALWAYSON_WITHOUT_TRUE_MSR | \
105 VM_EXIT_SAVE_DEBUG_CONTROLS | \
106 VM_EXIT_ACK_INTR_ON_EXIT | \
107 VM_EXIT_HOST_ADDR_SPACE_SIZE | \
108 VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL | \
109 VM_EXIT_SAVE_IA32_PAT | \
110 VM_EXIT_LOAD_IA32_PAT | \
111 VM_EXIT_SAVE_IA32_EFER | \
112 VM_EXIT_LOAD_IA32_EFER | \
113 VM_EXIT_CLEAR_BNDCFGS | \
114 VM_EXIT_PT_CONCEAL_PIP | \
115 VM_EXIT_CLEAR_IA32_RTIT_CTL)
116
117#define EVMCS1_SUPPORTED_VMENTRY_CTRL \
118 (VM_ENTRY_ALWAYSON_WITHOUT_TRUE_MSR | \
119 VM_ENTRY_LOAD_DEBUG_CONTROLS | \
120 VM_ENTRY_IA32E_MODE | \
121 VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL | \
122 VM_ENTRY_LOAD_IA32_PAT | \
123 VM_ENTRY_LOAD_IA32_EFER | \
124 VM_ENTRY_LOAD_BNDCFGS | \
125 VM_ENTRY_PT_CONCEAL_PIP | \
126 VM_ENTRY_LOAD_IA32_RTIT_CTL)
127
128#define EVMCS1_SUPPORTED_VMFUNC (0)
67 129
68struct evmcs_field {130struct evmcs_field {
69 u16 offset;131 u16 offset;
@@ -117,9 +179,7 @@ static __always_inline int get_evmcs_offset(unsigned long field, · 47 unmodified lines
117{179{
118 int offset = evmcs_field_offset(field, clean_field);180 int offset = evmcs_field_offset(field, clean_field);
119 181
120 WARN_ONCE(offset < 0, "KVM: accessing unsupported EVMCS field %lx\n",182 WARN_ONCE(offset < 0, "accessing unsupported EVMCS field %lx\n", field);
121 field);
122
123 return offset;183 return offset;
124}184}
125 185
@@ -136,7 +196,7 @@ static __always_inline void evmcs_write64(unsigned long field, u64 value) · 10 unmodified lines
136 current_evmcs->hv_clean_fields &= ~clean_field;196 current_evmcs->hv_clean_fields &= ~clean_field;
137}197}
138 198
139static inline void evmcs_write32(unsigned long field, u32 value)199static __always_inline void evmcs_write32(unsigned long field, u32 value)
140{200{
141 u16 clean_field;201 u16 clean_field;
142 int offset = get_evmcs_offset(field, &clean_field);202 int offset = get_evmcs_offset(field, &clean_field);
@@ -148,7 +208,7 @@ static inline void evmcs_write32(unsigned long field, u32 value) · 5 unmodified lines
148 current_evmcs->hv_clean_fields &= ~clean_field;208 current_evmcs->hv_clean_fields &= ~clean_field;
149}209}
150 210
151static inline void evmcs_write16(unsigned long field, u16 value)211static __always_inline void evmcs_write16(unsigned long field, u16 value)
152{212{
153 u16 clean_field;213 u16 clean_field;
154 int offset = get_evmcs_offset(field, &clean_field);214 int offset = get_evmcs_offset(field, &clean_field);
@@ -160,7 +220,7 @@ static inline void evmcs_write16(unsigned long field, u16 value) · 5 unmodified lines
160 current_evmcs->hv_clean_fields &= ~clean_field;220 current_evmcs->hv_clean_fields &= ~clean_field;
161}221}
162 222
163static inline u64 evmcs_read64(unsigned long field)223static __always_inline u64 evmcs_read64(unsigned long field)
164{224{
165 int offset = get_evmcs_offset(field, NULL);225 int offset = get_evmcs_offset(field, NULL);
166 226
@@ -170,7 +230,7 @@ static inline u64 evmcs_read64(unsigned long field) · 3 unmodified lines
170 return *(u64 *)((char *)current_evmcs + offset);230 return *(u64 *)((char *)current_evmcs + offset);
171}231}
172 232
173static inline u32 evmcs_read32(unsigned long field)233static __always_inline u32 evmcs_read32(unsigned long field)
174{234{
175 int offset = get_evmcs_offset(field, NULL);235 int offset = get_evmcs_offset(field, NULL);
176 236
@@ -180,7 +240,7 @@ static inline u32 evmcs_read32(unsigned long field) · 3 unmodified lines
180 return *(u32 *)((char *)current_evmcs + offset);240 return *(u32 *)((char *)current_evmcs + offset);
181}241}
182 242
183static inline u16 evmcs_read16(unsigned long field)243static __always_inline u16 evmcs_read16(unsigned long field)
184{244{
185 int offset = get_evmcs_offset(field, NULL);245 int offset = get_evmcs_offset(field, NULL);
186 246
@@ -190,16 +250,6 @@ static inline u16 evmcs_read16(unsigned long field) · 3 unmodified lines
190 return *(u16 *)((char *)current_evmcs + offset);250 return *(u16 *)((char *)current_evmcs + offset);
191}251}
192 252
193static inline void evmcs_touch_msr_bitmap(void)
194{
195 if (unlikely(!current_evmcs))
196 return;
197
198 if (current_evmcs->hv_enlightenments_control.msr_bitmap)
199 current_evmcs->hv_clean_fields &=
200 ~HV_VMX_ENLIGHTENED_CLEAN_FIELD_MSR_BITMAP;
201}
202
203static inline void evmcs_load(u64 phys_addr)253static inline void evmcs_load(u64 phys_addr)
204{254{
205 struct hv_vp_assist_page *vp_ap =255 struct hv_vp_assist_page *vp_ap =
@@ -211,15 +261,15 @@ static inline void evmcs_load(u64 phys_addr) · 5 unmodified lines
211 vp_ap->enlighten_vmentry = 1;261 vp_ap->enlighten_vmentry = 1;
212}262}
213 263
264void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf);
214#else /* !IS_ENABLED(CONFIG_HYPERV) */265#else /* !IS_ENABLED(CONFIG_HYPERV) */
215static __always_inline void evmcs_write64(unsigned long field, u64 value) {}266static __always_inline void evmcs_write64(unsigned long field, u64 value) {}
216static inline void evmcs_write32(unsigned long field, u32 value) {}267static __always_inline void evmcs_write32(unsigned long field, u32 value) {}
217static inline void evmcs_write16(unsigned long field, u16 value) {}268static __always_inline void evmcs_write16(unsigned long field, u16 value) {}
218static inline u64 evmcs_read64(unsigned long field) { return 0; }269static __always_inline u64 evmcs_read64(unsigned long field) { return 0; }
219static inline u32 evmcs_read32(unsigned long field) { return 0; }270static __always_inline u32 evmcs_read32(unsigned long field) { return 0; }
220static inline u16 evmcs_read16(unsigned long field) { return 0; }271static __always_inline u16 evmcs_read16(unsigned long field) { return 0; }
221static inline void evmcs_load(u64 phys_addr) {}272static inline void evmcs_load(u64 phys_addr) {}
222static inline void evmcs_touch_msr_bitmap(void) {}
223#endif /* IS_ENABLED(CONFIG_HYPERV) */273#endif /* IS_ENABLED(CONFIG_HYPERV) */
224 274
225#define EVMPTR_INVALID (-1ULL)275#define EVMPTR_INVALID (-1ULL)
arch/x86/kvm/vmx/nested.c+10−14
@@ -1,4 +1,5 @@
1// SPDX-License-Identifier: GPL-2.01// SPDX-License-Identifier: GPL-2.0
2#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
2 3
3#include <linux/objtool.h>4#include <linux/objtool.h>
4#include <linux/percpu.h>5#include <linux/percpu.h>
@@ -203,7 +204,7 @@ static void nested_vmx_abort(struct kvm_vcpu *vcpu, u32 indicator) · 198 unmodified lines
203{204{
204 /* TODO: not to reset guest simply here. */205 /* TODO: not to reset guest simply here. */
205 kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);206 kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
206 pr_debug_ratelimited("kvm: nested vmx abort, indicator %d\n", indicator);207 pr_debug_ratelimited("nested vmx abort, indicator %d\n", indicator);
207}208}
208 209
209static inline bool vmx_control_verify(u32 control, u32 low, u32 high)210static inline bool vmx_control_verify(u32 control, u32 low, u32 high)
@@ -5863,11 +5864,10 @@ static int handle_vmfunc(struct kvm_vcpu *vcpu) · 5653 unmodified lines
5863 u32 function = kvm_rax_read(vcpu);5864 u32 function = kvm_rax_read(vcpu);
5864 5865
5865 /*5866 /*
5866 * VMFUNC is only supported for nested guests, but we always enable the5867 * VMFUNC should never execute cleanly while L1 is active; KVM supports
5867 * secondary control for simplicity; for non-nested mode, fake that we5868 * VMFUNC for nested VMs, but not for L1.
5868 * didn't by injecting #UD.
5869 */5869 */
5870 if (!is_guest_mode(vcpu)) {5870 if (WARN_ON_ONCE(!is_guest_mode(vcpu))) {
5871 kvm_queue_exception(vcpu, UD_VECTOR);5871 kvm_queue_exception(vcpu, UD_VECTOR);
5872 return 1;5872 return 1;
5873 }5873 }
@@ -6880,6 +6880,7 @@ void nested_vmx_setup_ctls_msrs(struct vmcs_config *vmcs_conf, u32 ept_caps) · 1006 unmodified lines
6880 SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |6880 SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |
6881 SECONDARY_EXEC_RDRAND_EXITING |6881 SECONDARY_EXEC_RDRAND_EXITING |
6882 SECONDARY_EXEC_ENABLE_INVPCID |6882 SECONDARY_EXEC_ENABLE_INVPCID |
6883 SECONDARY_EXEC_ENABLE_VMFUNC |
6883 SECONDARY_EXEC_RDSEED_EXITING |6884 SECONDARY_EXEC_RDSEED_EXITING |
6884 SECONDARY_EXEC_XSAVES |6885 SECONDARY_EXEC_XSAVES |
6885 SECONDARY_EXEC_TSC_SCALING |6886 SECONDARY_EXEC_TSC_SCALING |
@@ -6912,18 +6913,13 @@ void nested_vmx_setup_ctls_msrs(struct vmcs_config *vmcs_conf, u32 ept_caps) · 26 unmodified lines
6912 SECONDARY_EXEC_ENABLE_PML;6913 SECONDARY_EXEC_ENABLE_PML;
6913 msrs->ept_caps |= VMX_EPT_AD_BIT;6914 msrs->ept_caps |= VMX_EPT_AD_BIT;
6914 }6915 }
6915 }
6916 6916
6917 if (cpu_has_vmx_vmfunc()) {
6918 msrs->secondary_ctls_high |=
6919 SECONDARY_EXEC_ENABLE_VMFUNC;
6920 /*6917 /*
6921 * Advertise EPTP switching unconditionally6918 * Advertise EPTP switching irrespective of hardware support,
6922 * since we emulate it6919 * KVM emulates it in software so long as VMFUNC is supported.
6923 */6920 */
6924 if (enable_ept)6921 if (cpu_has_vmx_vmfunc())
6925 msrs->vmfunc_controls =6922 msrs->vmfunc_controls = VMX_VMFUNC_EPTP_SWITCHING;
6926 VMX_VMFUNC_EPTP_SWITCHING;
6927 }6923 }
6928 6924
6929 /*6925 /*
arch/x86/kvm/vmx/pmu_intel.c+17−11
@@ -8,6 +8,8 @@ · 7 unmodified lines
8 * Avi Kivity <avi@redhat.com>8 * Avi Kivity <avi@redhat.com>
9 * Gleb Natapov <gleb@redhat.com>9 * Gleb Natapov <gleb@redhat.com>
10 */10 */
11#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
12
11#include <linux/types.h>13#include <linux/types.h>
12#include <linux/kvm_host.h>14#include <linux/kvm_host.h>
13#include <linux/perf_event.h>15#include <linux/perf_event.h>
@@ -20,16 +22,19 @@ · 6 unmodified lines
20 22
21#define MSR_PMC_FULL_WIDTH_BIT (MSR_IA32_PMC0 - MSR_IA32_PERFCTR0)23#define MSR_PMC_FULL_WIDTH_BIT (MSR_IA32_PMC0 - MSR_IA32_PERFCTR0)
22 24
23static struct kvm_event_hw_type_mapping intel_arch_events[] = {25static struct {
24 [0] = { 0x3c, 0x00, PERF_COUNT_HW_CPU_CYCLES },26 u8 eventsel;
25 [1] = { 0xc0, 0x00, PERF_COUNT_HW_INSTRUCTIONS },27 u8 unit_mask;
26 [2] = { 0x3c, 0x01, PERF_COUNT_HW_BUS_CYCLES },28} const intel_arch_events[] = {
27 [3] = { 0x2e, 0x4f, PERF_COUNT_HW_CACHE_REFERENCES },29 [0] = { 0x3c, 0x00 },
28 [4] = { 0x2e, 0x41, PERF_COUNT_HW_CACHE_MISSES },30 [1] = { 0xc0, 0x00 },
29 [5] = { 0xc4, 0x00, PERF_COUNT_HW_BRANCH_INSTRUCTIONS },31 [2] = { 0x3c, 0x01 },
30 [6] = { 0xc5, 0x00, PERF_COUNT_HW_BRANCH_MISSES },32 [3] = { 0x2e, 0x4f },
33 [4] = { 0x2e, 0x41 },
34 [5] = { 0xc4, 0x00 },
35 [6] = { 0xc5, 0x00 },
31 /* The above index must match CPUID 0x0A.EBX bit vector */36 /* The above index must match CPUID 0x0A.EBX bit vector */
32 [7] = { 0x00, 0x03, PERF_COUNT_HW_REF_CPU_CYCLES },37 [7] = { 0x00, 0x03 },
33};38};
34 39
35/* mapping between fixed pmc index and intel_arch_events array */40/* mapping between fixed pmc index and intel_arch_events array */
@@ -762,8 +767,7 @@ void vmx_passthrough_lbr_msrs(struct kvm_vcpu *vcpu) · 726 unmodified lines
762 return;767 return;
763 768
764warn:769warn:
765 pr_warn_ratelimited("kvm: vcpu-%d: fail to passthrough LBR.\n",770 pr_warn_ratelimited("vcpu-%d: fail to passthrough LBR.\n", vcpu->vcpu_id);
766 vcpu->vcpu_id);
767}771}
768 772
769static void intel_pmu_cleanup(struct kvm_vcpu *vcpu)773static void intel_pmu_cleanup(struct kvm_vcpu *vcpu)
@@ -810,4 +814,6 @@ struct kvm_pmu_ops intel_pmu_ops __initdata = { · 40 unmodified lines
810 .reset = intel_pmu_reset,814 .reset = intel_pmu_reset,
811 .deliver_pmi = intel_pmu_deliver_pmi,815 .deliver_pmi = intel_pmu_deliver_pmi,
812 .cleanup = intel_pmu_cleanup,816 .cleanup = intel_pmu_cleanup,
817 .EVENTSEL_EVENT = ARCH_PERFMON_EVENTSEL_EVENT,
818 .MAX_NR_GP_COUNTERS = KVM_INTEL_PMC_MAX_GENERIC,
813};819};
arch/x86/kvm/vmx/posted_intr.c+2−0
@@ -1,4 +1,6 @@
1// SPDX-License-Identifier: GPL-2.0-only1// SPDX-License-Identifier: GPL-2.0-only
2#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
3
2#include <linux/kvm_host.h>4#include <linux/kvm_host.h>
3 5
4#include <asm/irq_remapping.h>6#include <asm/irq_remapping.h>
arch/x86/kvm/vmx/sgx.c+3−2
@@ -1,5 +1,6 @@
1// SPDX-License-Identifier: GPL-2.01// SPDX-License-Identifier: GPL-2.0
2/* Copyright(c) 2021 Intel Corporation. */2/* Copyright(c) 2021 Intel Corporation. */
3#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
3 4
4#include <asm/sgx.h>5#include <asm/sgx.h>
5 6
@@ -164,7 +165,7 @@ static int __handle_encls_ecreate(struct kvm_vcpu *vcpu, · 158 unmodified lines
164 if (!vcpu->kvm->arch.sgx_provisioning_allowed &&165 if (!vcpu->kvm->arch.sgx_provisioning_allowed &&
165 (attributes & SGX_ATTR_PROVISIONKEY)) {166 (attributes & SGX_ATTR_PROVISIONKEY)) {
166 if (sgx_12_1->eax & SGX_ATTR_PROVISIONKEY)167 if (sgx_12_1->eax & SGX_ATTR_PROVISIONKEY)
167 pr_warn_once("KVM: SGX PROVISIONKEY advertised but not allowed\n");168 pr_warn_once("SGX PROVISIONKEY advertised but not allowed\n");
168 kvm_inject_gp(vcpu, 0);169 kvm_inject_gp(vcpu, 0);
169 return 1;170 return 1;
170 }171 }
@@ -381,7 +382,7 @@ int handle_encls(struct kvm_vcpu *vcpu) · 210 unmodified lines
381 return handle_encls_ecreate(vcpu);382 return handle_encls_ecreate(vcpu);
382 if (leaf == EINIT)383 if (leaf == EINIT)
383 return handle_encls_einit(vcpu);384 return handle_encls_einit(vcpu);
384 WARN(1, "KVM: unexpected exit on ENCLS[%u]", leaf);385 WARN_ONCE(1, "unexpected exit on ENCLS[%u]", leaf);
385 vcpu->run->exit_reason = KVM_EXIT_UNKNOWN;386 vcpu->run->exit_reason = KVM_EXIT_UNKNOWN;
386 vcpu->run->hw.hardware_exit_reason = EXIT_REASON_ENCLS;387 vcpu->run->hw.hardware_exit_reason = EXIT_REASON_ENCLS;
387 return 0;388 return 0;
arch/x86/kvm/vmx/vmcs.h+2−2
@@ -75,7 +75,7 @@ struct loaded_vmcs { · 74 unmodified lines
75 struct vmcs_controls_shadow controls_shadow;75 struct vmcs_controls_shadow controls_shadow;
76};76};
77 77
78static inline bool is_intr_type(u32 intr_info, u32 type)78static __always_inline bool is_intr_type(u32 intr_info, u32 type)
79{79{
80 const u32 mask = INTR_INFO_VALID_MASK | INTR_INFO_INTR_TYPE_MASK;80 const u32 mask = INTR_INFO_VALID_MASK | INTR_INFO_INTR_TYPE_MASK;
81 81
@@ -146,7 +146,7 @@ static inline bool is_icebp(u32 intr_info) · 64 unmodified lines
146 return is_intr_type(intr_info, INTR_TYPE_PRIV_SW_EXCEPTION);146 return is_intr_type(intr_info, INTR_TYPE_PRIV_SW_EXCEPTION);
147}147}
148 148
149static inline bool is_nmi(u32 intr_info)149static __always_inline bool is_nmi(u32 intr_info)
150{150{
151 return is_intr_type(intr_info, INTR_TYPE_NMI_INTR);151 return is_intr_type(intr_info, INTR_TYPE_NMI_INTR);
152}152}
arch/x86/kvm/vmx/vmcs12.c+1−0
@@ -1,4 +1,5 @@
1// SPDX-License-Identifier: GPL-2.01// SPDX-License-Identifier: GPL-2.0
2#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
2 3
3#include "vmcs12.h"4#include "vmcs12.h"
4 5
arch/x86/kvm/vmx/vmenter.S+44−36
@@ -31,6 +31,39 @@ · 30 unmodified lines
31#define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE31#define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE
32#endif32#endif
33 33
34.macro VMX_DO_EVENT_IRQOFF call_insn call_target
35 /*
36 * Unconditionally create a stack frame, getting the correct RSP on the
37 * stack (for x86-64) would take two instructions anyways, and RBP can
38 * be used to restore RSP to make objtool happy (see below).
39 */
40 push %_ASM_BP
41 mov %_ASM_SP, %_ASM_BP
42
43#ifdef CONFIG_X86_64
44 /*
45 * Align RSP to a 16-byte boundary (to emulate CPU behavior) before
46 * creating the synthetic interrupt stack frame for the IRQ/NMI.
47 */
48 and $-16, %rsp
49 push $__KERNEL_DS
50 push %rbp
51#endif
52 pushf
53 push $__KERNEL_CS
54 \call_insn \call_target
55
56 /*
57 * "Restore" RSP from RBP, even though IRET has already unwound RSP to
58 * the correct value. objtool doesn't know the callee will IRET and,
59 * without the explicit restore, thinks the stack is getting walloped.
60 * Using an unwind hint is problematic due to x86-64's dynamic alignment.
61 */
62 mov %_ASM_BP, %_ASM_SP
63 pop %_ASM_BP
64 RET
65.endm
66
34.section .noinstr.text, "ax"67.section .noinstr.text, "ax"
35 68
36/**69/**
@@ -69,8 +102,8 @@ SYM_FUNC_START(__vmx_vcpu_run) · 32 unmodified lines
69 */102 */
70 push %_ASM_ARG2103 push %_ASM_ARG2
71 104
72 /* Copy @flags to BL, _ASM_ARG3 is volatile. */105 /* Copy @flags to EBX, _ASM_ARG3 is volatile. */
73 mov %_ASM_ARG3B, %bl106 mov %_ASM_ARG3L, %ebx
74 107
75 lea (%_ASM_SP), %_ASM_ARG2108 lea (%_ASM_SP), %_ASM_ARG2
76 call vmx_update_host_rsp109 call vmx_update_host_rsp
@@ -106,7 +139,7 @@ SYM_FUNC_START(__vmx_vcpu_run) · 29 unmodified lines
106 mov (%_ASM_SP), %_ASM_AX139 mov (%_ASM_SP), %_ASM_AX
107 140
108 /* Check if vmlaunch or vmresume is needed */141 /* Check if vmlaunch or vmresume is needed */
109 testb $VMX_RUN_VMRESUME, %bl142 test $VMX_RUN_VMRESUME, %ebx
110 143
111 /* Load guest registers. Don't clobber flags. */144 /* Load guest registers. Don't clobber flags. */
112 mov VCPU_RCX(%_ASM_AX), %_ASM_CX145 mov VCPU_RCX(%_ASM_AX), %_ASM_CX
@@ -128,7 +161,7 @@ SYM_FUNC_START(__vmx_vcpu_run) · 15 unmodified lines
128 /* Load guest RAX. This kills the @regs pointer! */161 /* Load guest RAX. This kills the @regs pointer! */
129 mov VCPU_RAX(%_ASM_AX), %_ASM_AX162 mov VCPU_RAX(%_ASM_AX), %_ASM_AX
130 163
131 /* Check EFLAGS.ZF from 'testb' above */164 /* Check EFLAGS.ZF from 'test VMX_RUN_VMRESUME' above */
132 jz .Lvmlaunch165 jz .Lvmlaunch
133 166
134 /*167 /*
@@ -266,6 +299,10 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL) · 131 unmodified lines
266 299
267SYM_FUNC_END(__vmx_vcpu_run)300SYM_FUNC_END(__vmx_vcpu_run)
268 301
302SYM_FUNC_START(vmx_do_nmi_irqoff)
303 VMX_DO_EVENT_IRQOFF call asm_exc_nmi_kvm_vmx
304SYM_FUNC_END(vmx_do_nmi_irqoff)
305
269 306
270.section .text, "ax"307.section .text, "ax"
271 308
@@ -320,35 +357,6 @@ SYM_FUNC_START(vmread_error_trampoline) · 48 unmodified lines
320SYM_FUNC_END(vmread_error_trampoline)357SYM_FUNC_END(vmread_error_trampoline)
321#endif358#endif
322 359
323SYM_FUNC_START(vmx_do_interrupt_nmi_irqoff)360SYM_FUNC_START(vmx_do_interrupt_irqoff)
324 /*361 VMX_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1
325 * Unconditionally create a stack frame, getting the correct RSP on the362SYM_FUNC_END(vmx_do_interrupt_irqoff)
326 * stack (for x86-64) would take two instructions anyways, and RBP can
327 * be used to restore RSP to make objtool happy (see below).
328 */
329 push %_ASM_BP
330 mov %_ASM_SP, %_ASM_BP
331
332#ifdef CONFIG_X86_64
333 /*
334 * Align RSP to a 16-byte boundary (to emulate CPU behavior) before
335 * creating the synthetic interrupt stack frame for the IRQ/NMI.
336 */
337 and $-16, %rsp
338 push $__KERNEL_DS
339 push %rbp
340#endif
341 pushf
342 push $__KERNEL_CS
343 CALL_NOSPEC _ASM_ARG1
344
345 /*
346 * "Restore" RSP from RBP, even though IRET has already unwound RSP to
347 * the correct value. objtool doesn't know the callee will IRET and,
348 * without the explicit restore, thinks the stack is getting walloped.
349 * Using an unwind hint is problematic due to x86-64's dynamic alignment.
350 */
351 mov %_ASM_BP, %_ASM_SP
352 pop %_ASM_BP
353 RET
354SYM_FUNC_END(vmx_do_interrupt_nmi_irqoff)
arch/x86/kvm/vmx/vmx.c+328−296
@@ -12,6 +12,7 @@ · 11 unmodified lines
12 * Avi Kivity <avi@qumranet.com>12 * Avi Kivity <avi@qumranet.com>
13 * Yaniv Kamay <yaniv@qumranet.com>13 * Yaniv Kamay <yaniv@qumranet.com>
14 */14 */
15#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
15 16
16#include <linux/highmem.h>17#include <linux/highmem.h>
17#include <linux/hrtimer.h>18#include <linux/hrtimer.h>
@@ -444,36 +445,36 @@ void vmread_error(unsigned long field, bool fault) · 426 unmodified lines
444 if (fault)445 if (fault)
445 kvm_spurious_fault();446 kvm_spurious_fault();
446 else447 else
447 vmx_insn_failed("kvm: vmread failed: field=%lx\n", field);448 vmx_insn_failed("vmread failed: field=%lx\n", field);
448}449}
449 450
450noinline void vmwrite_error(unsigned long field, unsigned long value)451noinline void vmwrite_error(unsigned long field, unsigned long value)
451{452{
452 vmx_insn_failed("kvm: vmwrite failed: field=%lx val=%lx err=%u\n",453 vmx_insn_failed("vmwrite failed: field=%lx val=%lx err=%u\n",
453 field, value, vmcs_read32(VM_INSTRUCTION_ERROR));454 field, value, vmcs_read32(VM_INSTRUCTION_ERROR));
454}455}
455 456
456noinline void vmclear_error(struct vmcs *vmcs, u64 phys_addr)457noinline void vmclear_error(struct vmcs *vmcs, u64 phys_addr)
457{458{
458 vmx_insn_failed("kvm: vmclear failed: %p/%llx err=%u\n",459 vmx_insn_failed("vmclear failed: %p/%llx err=%u\n",
459 vmcs, phys_addr, vmcs_read32(VM_INSTRUCTION_ERROR));460 vmcs, phys_addr, vmcs_read32(VM_INSTRUCTION_ERROR));
460}461}
461 462
462noinline void vmptrld_error(struct vmcs *vmcs, u64 phys_addr)463noinline void vmptrld_error(struct vmcs *vmcs, u64 phys_addr)
463{464{
464 vmx_insn_failed("kvm: vmptrld failed: %p/%llx err=%u\n",465 vmx_insn_failed("vmptrld failed: %p/%llx err=%u\n",
465 vmcs, phys_addr, vmcs_read32(VM_INSTRUCTION_ERROR));466 vmcs, phys_addr, vmcs_read32(VM_INSTRUCTION_ERROR));
466}467}
467 468
468noinline void invvpid_error(unsigned long ext, u16 vpid, gva_t gva)469noinline void invvpid_error(unsigned long ext, u16 vpid, gva_t gva)
469{470{
470 vmx_insn_failed("kvm: invvpid failed: ext=0x%lx vpid=%u gva=0x%lx\n",471 vmx_insn_failed("invvpid failed: ext=0x%lx vpid=%u gva=0x%lx\n",
471 ext, vpid, gva);472 ext, vpid, gva);
472}473}
473 474
474noinline void invept_error(unsigned long ext, u64 eptp, gpa_t gpa)475noinline void invept_error(unsigned long ext, u64 eptp, gpa_t gpa)
475{476{
476 vmx_insn_failed("kvm: invept failed: ext=0x%lx eptp=%llx gpa=0x%llx\n",477 vmx_insn_failed("invept failed: ext=0x%lx eptp=%llx gpa=0x%llx\n",
477 ext, eptp, gpa);478 ext, eptp, gpa);
478}479}
479 480
@@ -488,8 +489,8 @@ static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu); · 8 unmodified lines
488static DECLARE_BITMAP(vmx_vpid_bitmap, VMX_NR_VPIDS);489static DECLARE_BITMAP(vmx_vpid_bitmap, VMX_NR_VPIDS);
489static DEFINE_SPINLOCK(vmx_vpid_lock);490static DEFINE_SPINLOCK(vmx_vpid_lock);
490 491
491struct vmcs_config vmcs_config;492struct vmcs_config vmcs_config __ro_after_init;
492struct vmx_capability vmx_capability;493struct vmx_capability vmx_capability __ro_after_init;
493 494
494#define VMX_SEGMENT_FIELD(seg) \495#define VMX_SEGMENT_FIELD(seg) \
495 [VCPU_SREG_##seg] = { \496 [VCPU_SREG_##seg] = { \
@@ -523,6 +524,8 @@ static inline void vmx_segment_cache_clear(struct vcpu_vmx *vmx) · 27 unmodified lines
523static unsigned long host_idt_base;524static unsigned long host_idt_base;
524 525
525#if IS_ENABLED(CONFIG_HYPERV)526#if IS_ENABLED(CONFIG_HYPERV)
527static struct kvm_x86_ops vmx_x86_ops __initdata;
528
526static bool __read_mostly enlightened_vmcs = true;529static bool __read_mostly enlightened_vmcs = true;
527module_param(enlightened_vmcs, bool, 0444);530module_param(enlightened_vmcs, bool, 0444);
528 531
@@ -551,6 +554,71 @@ static int hv_enable_l2_tlb_flush(struct kvm_vcpu *vcpu) · 22 unmodified lines
551 return 0;554 return 0;
552}555}
553 556
557static __init void hv_init_evmcs(void)
558{
559 int cpu;
560
561 if (!enlightened_vmcs)
562 return;
563
564 /*
565 * Enlightened VMCS usage should be recommended and the host needs
566 * to support eVMCS v1 or above.
567 */
568 if (ms_hyperv.hints & HV_X64_ENLIGHTENED_VMCS_RECOMMENDED &&
569 (ms_hyperv.nested_features & HV_X64_ENLIGHTENED_VMCS_VERSION) >=
570 KVM_EVMCS_VERSION) {
571
572 /* Check that we have assist pages on all online CPUs */
573 for_each_online_cpu(cpu) {
574 if (!hv_get_vp_assist_page(cpu)) {
575 enlightened_vmcs = false;
576 break;
577 }
578 }
579
580 if (enlightened_vmcs) {
581 pr_info("Using Hyper-V Enlightened VMCS\n");
582 static_branch_enable(&enable_evmcs);
583 }
584
585 if (ms_hyperv.nested_features & HV_X64_NESTED_DIRECT_FLUSH)
586 vmx_x86_ops.enable_l2_tlb_flush
587 = hv_enable_l2_tlb_flush;
588
589 } else {
590 enlightened_vmcs = false;
591 }
592}
593
594static void hv_reset_evmcs(void)
595{
596 struct hv_vp_assist_page *vp_ap;
597
598 if (!static_branch_unlikely(&enable_evmcs))
599 return;
600
601 /*
602 * KVM should enable eVMCS if and only if all CPUs have a VP assist
603 * page, and should reject CPU onlining if eVMCS is enabled the CPU
604 * doesn't have a VP assist page allocated.
605 */
606 vp_ap = hv_get_vp_assist_page(smp_processor_id());
607 if (WARN_ON_ONCE(!vp_ap))
608 return;
609
610 /*
611 * Reset everything to support using non-enlightened VMCS access later
612 * (e.g. when we reload the module with enlightened_vmcs=0)
613 */
614 vp_ap->nested_control.features.directhypercall = 0;
615 vp_ap->current_nested_vmcs = 0;
616 vp_ap->enlighten_vmentry = 0;
617}
618
619#else /* IS_ENABLED(CONFIG_HYPERV) */
620static void hv_init_evmcs(void) {}
621static void hv_reset_evmcs(void) {}
554#endif /* IS_ENABLED(CONFIG_HYPERV) */622#endif /* IS_ENABLED(CONFIG_HYPERV) */
555 623
556/*624/*
@@ -1613,8 +1681,8 @@ static int skip_emulated_instruction(struct kvm_vcpu *vcpu) · 1056 unmodified lines
1613 if (!instr_len)1681 if (!instr_len)
1614 goto rip_updated;1682 goto rip_updated;
1615 1683
1616 WARN(exit_reason.enclave_mode,1684 WARN_ONCE(exit_reason.enclave_mode,
1617 "KVM: skipping instruction after SGX enclave VM-Exit");1685 "skipping instruction after SGX enclave VM-Exit");
1618 1686
1619 orig_rip = kvm_rip_read(vcpu);1687 orig_rip = kvm_rip_read(vcpu);
1620 rip = orig_rip + instr_len;1688 rip = orig_rip + instr_len;
@@ -2138,9 +2206,7 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 517 unmodified lines
2138 2206
2139 invalid = data & ~vmx_get_supported_debugctl(vcpu, msr_info->host_initiated);2207 invalid = data & ~vmx_get_supported_debugctl(vcpu, msr_info->host_initiated);
2140 if (invalid & (DEBUGCTLMSR_BTF|DEBUGCTLMSR_LBR)) {2208 if (invalid & (DEBUGCTLMSR_BTF|DEBUGCTLMSR_LBR)) {
2141 if (report_ignored_msrs)2209 kvm_pr_unimpl_wrmsr(vcpu, msr_index, data);
2142 vcpu_unimpl(vcpu, "%s: BTF|LBR in IA32_DEBUGCTLMSR 0x%llx, nop\n",
2143 __func__, data);
2144 data &= ~(DEBUGCTLMSR_BTF|DEBUGCTLMSR_LBR);2210 data &= ~(DEBUGCTLMSR_BTF|DEBUGCTLMSR_LBR);
2145 invalid &= ~(DEBUGCTLMSR_BTF|DEBUGCTLMSR_LBR);2211 invalid &= ~(DEBUGCTLMSR_BTF|DEBUGCTLMSR_LBR);
2146 }2212 }
@@ -2448,88 +2514,6 @@ static void vmx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg) · 301 unmodified lines
2448 }2514 }
2449}2515}
2450 2516
2451static __init int cpu_has_kvm_support(void)
2452{
2453 return cpu_has_vmx();
2454}
2455
2456static __init int vmx_disabled_by_bios(void)
2457{
2458 return !boot_cpu_has(X86_FEATURE_MSR_IA32_FEAT_CTL) ||
2459 !boot_cpu_has(X86_FEATURE_VMX);
2460}
2461
2462static int kvm_cpu_vmxon(u64 vmxon_pointer)
2463{
2464 u64 msr;
2465
2466 cr4_set_bits(X86_CR4_VMXE);
2467
2468 asm_volatile_goto("1: vmxon %[vmxon_pointer]\n\t"
2469 _ASM_EXTABLE(1b, %l[fault])
2470 : : [vmxon_pointer] "m"(vmxon_pointer)
2471 : : fault);
2472 return 0;
2473
2474fault:
2475 WARN_ONCE(1, "VMXON faulted, MSR_IA32_FEAT_CTL (0x3a) = 0x%llx\n",
2476 rdmsrl_safe(MSR_IA32_FEAT_CTL, &msr) ? 0xdeadbeef : msr);
2477 cr4_clear_bits(X86_CR4_VMXE);
2478
2479 return -EFAULT;
2480}
2481
2482static int vmx_hardware_enable(void)
2483{
2484 int cpu = raw_smp_processor_id();
2485 u64 phys_addr = __pa(per_cpu(vmxarea, cpu));
2486 int r;
2487
2488 if (cr4_read_shadow() & X86_CR4_VMXE)
2489 return -EBUSY;
2490
2491 /*
2492 * This can happen if we hot-added a CPU but failed to allocate
2493 * VP assist page for it.
2494 */
2495 if (static_branch_unlikely(&enable_evmcs) &&
2496 !hv_get_vp_assist_page(cpu))
2497 return -EFAULT;
2498
2499 intel_pt_handle_vmx(1);
2500
2501 r = kvm_cpu_vmxon(phys_addr);
2502 if (r) {
2503 intel_pt_handle_vmx(0);
2504 return r;
2505 }
2506
2507 if (enable_ept)
2508 ept_sync_global();
2509
2510 return 0;
2511}
2512
2513static void vmclear_local_loaded_vmcss(void)
2514{
2515 int cpu = raw_smp_processor_id();
2516 struct loaded_vmcs *v, *n;
2517
2518 list_for_each_entry_safe(v, n, &per_cpu(loaded_vmcss_on_cpu, cpu),
2519 loaded_vmcss_on_cpu_link)
2520 __loaded_vmcs_clear(v);
2521}
2522
2523static void vmx_hardware_disable(void)
2524{
2525 vmclear_local_loaded_vmcss();
2526
2527 if (cpu_vmxoff())
2528 kvm_spurious_fault();
2529
2530 intel_pt_handle_vmx(0);
2531}
2532
2533/*2517/*
2534 * There is no X86_FEATURE for SGX yet, but anyway we need to query CPUID2518 * There is no X86_FEATURE for SGX yet, but anyway we need to query CPUID
2535 * directly instead of going through cpu_has(), to ensure KVM is trapping2519 * directly instead of going through cpu_has(), to ensure KVM is trapping
@@ -2565,8 +2549,7 @@ static bool cpu_has_perf_global_ctrl_bug(void) · 29 unmodified lines
2565 return false;2549 return false;
2566}2550}
2567 2551
2568static __init int adjust_vmx_controls(u32 ctl_min, u32 ctl_opt,2552static int adjust_vmx_controls(u32 ctl_min, u32 ctl_opt, u32 msr, u32 *result)
2569 u32 msr, u32 *result)
2570{2553{
2571 u32 vmx_msr_low, vmx_msr_high;2554 u32 vmx_msr_low, vmx_msr_high;
2572 u32 ctl = ctl_min | ctl_opt;2555 u32 ctl = ctl_min | ctl_opt;
@@ -2584,7 +2567,7 @@ static __init int adjust_vmx_controls(u32 ctl_min, u32 ctl_opt, · 11 unmodified lines
2584 return 0;2567 return 0;
2585}2568}
2586 2569
2587static __init u64 adjust_vmx_controls64(u64 ctl_opt, u32 msr)2570static u64 adjust_vmx_controls64(u64 ctl_opt, u32 msr)
2588{2571{
2589 u64 allowed;2572 u64 allowed;
2590 2573
@@ -2593,8 +2576,8 @@ static __init u64 adjust_vmx_controls64(u64 ctl_opt, u32 msr) · 2 unmodified lines
2593 return ctl_opt & allowed;2576 return ctl_opt & allowed;
2594}2577}
2595 2578
2596static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,2579static int setup_vmcs_config(struct vmcs_config *vmcs_conf,
2597 struct vmx_capability *vmx_cap)2580 struct vmx_capability *vmx_cap)
2598{2581{
2599 u32 vmx_msr_low, vmx_msr_high;2582 u32 vmx_msr_low, vmx_msr_high;
2600 u32 _pin_based_exec_control = 0;2583 u32 _pin_based_exec_control = 0;
@@ -2752,9 +2735,127 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf, · 151 unmodified lines
2752 vmcs_conf->vmentry_ctrl = _vmentry_control;2735 vmcs_conf->vmentry_ctrl = _vmentry_control;
2753 vmcs_conf->misc = misc_msr;2736 vmcs_conf->misc = misc_msr;
2754 2737
2738#if IS_ENABLED(CONFIG_HYPERV)
2739 if (enlightened_vmcs)
2740 evmcs_sanitize_exec_ctrls(vmcs_conf);
2741#endif
2742
2743 return 0;
2744}
2745
2746static bool kvm_is_vmx_supported(void)
2747{
2748 int cpu = raw_smp_processor_id();
2749
2750 if (!cpu_has_vmx()) {
2751 pr_err("VMX not supported by CPU %d\n", cpu);
2752 return false;
2753 }
2754
2755 if (!this_cpu_has(X86_FEATURE_MSR_IA32_FEAT_CTL) ||
2756 !this_cpu_has(X86_FEATURE_VMX)) {
2757 pr_err("VMX not enabled (by BIOS) in MSR_IA32_FEAT_CTL on CPU %d\n", cpu);
2758 return false;
2759 }
2760
2761 return true;
2762}
2763
2764static int vmx_check_processor_compat(void)
2765{
2766 int cpu = raw_smp_processor_id();
2767 struct vmcs_config vmcs_conf;
2768 struct vmx_capability vmx_cap;
2769
2770 if (!kvm_is_vmx_supported())
2771 return -EIO;
2772
2773 if (setup_vmcs_config(&vmcs_conf, &vmx_cap) < 0) {
2774 pr_err("Failed to setup VMCS config on CPU %d\n", cpu);
2775 return -EIO;
2776 }
2777 if (nested)
2778 nested_vmx_setup_ctls_msrs(&vmcs_conf, vmx_cap.ept);
2779 if (memcmp(&vmcs_config, &vmcs_conf, sizeof(struct vmcs_config))) {
2780 pr_err("Inconsistent VMCS config on CPU %d\n", cpu);
2781 return -EIO;
2782 }
2783 return 0;
2784}
2785
2786static int kvm_cpu_vmxon(u64 vmxon_pointer)
2787{
2788 u64 msr;
2789
2790 cr4_set_bits(X86_CR4_VMXE);
2791
2792 asm_volatile_goto("1: vmxon %[vmxon_pointer]\n\t"
2793 _ASM_EXTABLE(1b, %l[fault])
2794 : : [vmxon_pointer] "m"(vmxon_pointer)
2795 : : fault);
2796 return 0;
2797
2798fault:
2799 WARN_ONCE(1, "VMXON faulted, MSR_IA32_FEAT_CTL (0x3a) = 0x%llx\n",
2800 rdmsrl_safe(MSR_IA32_FEAT_CTL, &msr) ? 0xdeadbeef : msr);
2801 cr4_clear_bits(X86_CR4_VMXE);
2802
2803 return -EFAULT;
2804}
2805
2806static int vmx_hardware_enable(void)
2807{
2808 int cpu = raw_smp_processor_id();
2809 u64 phys_addr = __pa(per_cpu(vmxarea, cpu));
2810 int r;
2811
2812 if (cr4_read_shadow() & X86_CR4_VMXE)
2813 return -EBUSY;
2814
2815 /*
2816 * This can happen if we hot-added a CPU but failed to allocate
2817 * VP assist page for it.
2818 */
2819 if (static_branch_unlikely(&enable_evmcs) &&
2820 !hv_get_vp_assist_page(cpu))
2821 return -EFAULT;
2822
2823 intel_pt_handle_vmx(1);
2824
2825 r = kvm_cpu_vmxon(phys_addr);
2826 if (r) {
2827 intel_pt_handle_vmx(0);
2828 return r;
2829 }
2830
2831 if (enable_ept)
2832 ept_sync_global();
2833
2755 return 0;2834 return 0;
2756}2835}
2757 2836
2837static void vmclear_local_loaded_vmcss(void)
2838{
2839 int cpu = raw_smp_processor_id();
2840 struct loaded_vmcs *v, *n;
2841
2842 list_for_each_entry_safe(v, n, &per_cpu(loaded_vmcss_on_cpu, cpu),
2843 loaded_vmcss_on_cpu_link)
2844 __loaded_vmcs_clear(v);
2845}
2846
2847static void vmx_hardware_disable(void)
2848{
2849 vmclear_local_loaded_vmcss();
2850
2851 if (cpu_vmxoff())
2852 kvm_spurious_fault();
2853
2854 hv_reset_evmcs();
2855
2856 intel_pt_handle_vmx(0);
2857}
2858
2758struct vmcs *alloc_vmcs_cpu(bool shadow, int cpu, gfp_t flags)2859struct vmcs *alloc_vmcs_cpu(bool shadow, int cpu, gfp_t flags)
2759{2860{
2760 int node = cpu_to_node(cpu);2861 int node = cpu_to_node(cpu);
@@ -2950,9 +3051,8 @@ static void fix_rmode_seg(int seg, struct kvm_segment *save) · 189 unmodified lines
2950 var.type = 0x3;3051 var.type = 0x3;
2951 var.avl = 0;3052 var.avl = 0;
2952 if (save->base & 0xf)3053 if (save->base & 0xf)
2953 printk_once(KERN_WARNING "kvm: segment base is not "3054 pr_warn_once("segment base is not paragraph aligned "
2954 "paragraph aligned when entering "3055 "when entering protected mode (seg=%d)", seg);
2955 "protected mode (seg=%d)", seg);
2956 }3056 }
2957 3057
2958 vmcs_write16(sf->selector, var.selector);3058 vmcs_write16(sf->selector, var.selector);
@@ -2982,8 +3082,7 @@ static void enter_rmode(struct kvm_vcpu *vcpu) · 23 unmodified lines
2982 * vcpu. Warn the user that an update is overdue.3082 * vcpu. Warn the user that an update is overdue.
2983 */3083 */
2984 if (!kvm_vmx->tss_addr)3084 if (!kvm_vmx->tss_addr)
2985 printk_once(KERN_WARNING "kvm: KVM_SET_TSS_ADDR need to be "3085 pr_warn_once("KVM_SET_TSS_ADDR needs to be called before running vCPU\n");
2986 "called before entering vcpu\n");
2987 3086
2988 vmx_segment_cache_clear(vmx);3087 vmx_segment_cache_clear(vmx);
2989 3088
@@ -3800,39 +3899,6 @@ static void seg_setup(int seg) · 810 unmodified lines
3800 vmcs_write32(sf->ar_bytes, ar);3899 vmcs_write32(sf->ar_bytes, ar);
3801}3900}
3802 3901
3803static int alloc_apic_access_page(struct kvm *kvm)
3804{
3805 struct page *page;
3806 void __user *hva;
3807 int ret = 0;
3808
3809 mutex_lock(&kvm->slots_lock);
3810 if (kvm->arch.apic_access_memslot_enabled)
3811 goto out;
3812 hva = __x86_set_memory_region(kvm, APIC_ACCESS_PAGE_PRIVATE_MEMSLOT,
3813 APIC_DEFAULT_PHYS_BASE, PAGE_SIZE);
3814 if (IS_ERR(hva)) {
3815 ret = PTR_ERR(hva);
3816 goto out;
3817 }
3818
3819 page = gfn_to_page(kvm, APIC_DEFAULT_PHYS_BASE >> PAGE_SHIFT);
3820 if (is_error_page(page)) {
3821 ret = -EFAULT;
3822 goto out;
3823 }
3824
3825 /*
3826 * Do not pin the page in memory, so that memory hot-unplug
3827 * is able to migrate it.
3828 */
3829 put_page(page);
3830 kvm->arch.apic_access_memslot_enabled = true;
3831out:
3832 mutex_unlock(&kvm->slots_lock);
3833 return ret;
3834}
3835
3836int allocate_vpid(void)3902int allocate_vpid(void)
3837{3903{
3838 int vpid;3904 int vpid;
@@ -3865,8 +3931,13 @@ static void vmx_msr_bitmap_l01_changed(struct vcpu_vmx *vmx) · 26 unmodified lines
3865 * 'Enlightened MSR Bitmap' feature L0 needs to know that MSR3931 * 'Enlightened MSR Bitmap' feature L0 needs to know that MSR
3866 * bitmap has changed.3932 * bitmap has changed.
3867 */3933 */
3868 if (static_branch_unlikely(&enable_evmcs))3934 if (IS_ENABLED(CONFIG_HYPERV) && static_branch_unlikely(&enable_evmcs)) {
3869 evmcs_touch_msr_bitmap();3935 struct hv_enlightened_vmcs *evmcs = (void *)vmx->vmcs01.vmcs;
3936
3937 if (evmcs->hv_enlightenments_control.msr_bitmap)
3938 evmcs->hv_clean_fields &=
3939 ~HV_VMX_ENLIGHTENED_CLEAN_FIELD_MSR_BITMAP;
3940 }
3870 3941
3871 vmx->nested.force_msr_bitmap_recalc = true;3942 vmx->nested.force_msr_bitmap_recalc = true;
3872}3943}
@@ -3947,29 +4018,20 @@ void vmx_enable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type) · 74 unmodified lines
3947 vmx_set_msr_bitmap_write(msr_bitmap, msr);4018 vmx_set_msr_bitmap_write(msr_bitmap, msr);
3948}4019}
3949 4020
3950static void vmx_reset_x2apic_msrs(struct kvm_vcpu *vcpu, u8 mode)
3951{
3952 unsigned long *msr_bitmap = to_vmx(vcpu)->vmcs01.msr_bitmap;
3953 unsigned long read_intercept;
3954 int msr;
3955
3956 read_intercept = (mode & MSR_BITMAP_MODE_X2APIC_APICV) ? 0 : ~0;
3957
3958 for (msr = 0x800; msr <= 0x8ff; msr += BITS_PER_LONG) {
3959 unsigned int read_idx = msr / BITS_PER_LONG;
3960 unsigned int write_idx = read_idx + (0x800 / sizeof(long));
3961
3962 msr_bitmap[read_idx] = read_intercept;
3963 msr_bitmap[write_idx] = ~0ul;
3964 }
3965}
3966
3967static void vmx_update_msr_bitmap_x2apic(struct kvm_vcpu *vcpu)4021static void vmx_update_msr_bitmap_x2apic(struct kvm_vcpu *vcpu)
3968{4022{
4023 /*
4024 * x2APIC indices for 64-bit accesses into the RDMSR and WRMSR halves
4025 * of the MSR bitmap. KVM emulates APIC registers up through 0x3f0,
4026 * i.e. MSR 0x83f, and so only needs to dynamically manipulate 64 bits.
4027 */
4028 const int read_idx = APIC_BASE_MSR / BITS_PER_LONG_LONG;
4029 const int write_idx = read_idx + (0x800 / sizeof(u64));
3969 struct vcpu_vmx *vmx = to_vmx(vcpu);4030 struct vcpu_vmx *vmx = to_vmx(vcpu);
4031 u64 *msr_bitmap = (u64 *)vmx->vmcs01.msr_bitmap;
3970 u8 mode;4032 u8 mode;
3971 4033
3972 if (!cpu_has_vmx_msr_bitmap())4034 if (!cpu_has_vmx_msr_bitmap() || WARN_ON_ONCE(!lapic_in_kernel(vcpu)))
3973 return;4035 return;
3974 4036
3975 if (cpu_has_secondary_exec_ctrls() &&4037 if (cpu_has_secondary_exec_ctrls() &&
@@ -3987,7 +4049,18 @@ static void vmx_update_msr_bitmap_x2apic(struct kvm_vcpu *vcpu) · 11 unmodified lines
3987 4049
3988 vmx->x2apic_msr_bitmap_mode = mode;4050 vmx->x2apic_msr_bitmap_mode = mode;
3989 4051
3990 vmx_reset_x2apic_msrs(vcpu, mode);4052 /*
4053 * Reset the bitmap for MSRs 0x800 - 0x83f. Leave AMD's uber-extended
4054 * registers (0x840 and above) intercepted, KVM doesn't support them.
4055 * Intercept all writes by default and poke holes as needed. Pass
4056 * through reads for all valid registers by default in x2APIC+APICv
4057 * mode, only the current timer count needs on-demand emulation by KVM.
4058 */
4059 if (mode & MSR_BITMAP_MODE_X2APIC_APICV)
4060 msr_bitmap[read_idx] = ~kvm_lapic_readable_reg_mask(vcpu->arch.apic);
4061 else
4062 msr_bitmap[read_idx] = ~0ull;
4063 msr_bitmap[write_idx] = ~0ull;
3991 4064
3992 /*4065 /*
3993 * TPR reads and writes can be virtualized even if virtual interrupt4066 * TPR reads and writes can be virtualized even if virtual interrupt
@@ -4519,6 +4592,12 @@ static u32 vmx_secondary_exec_control(struct vcpu_vmx *vmx) · 525 unmodified lines
4519 SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY);4592 SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY);
4520 exec_control &= ~SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE;4593 exec_control &= ~SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE;
4521 4594
4595 /*
4596 * KVM doesn't support VMFUNC for L1, but the control is set in KVM's
4597 * base configuration as KVM emulates VMFUNC[EPTP_SWITCHING] for L2.
4598 */
4599 exec_control &= ~SECONDARY_EXEC_ENABLE_VMFUNC;
4600
4522 /* SECONDARY_EXEC_DESC is enabled/disabled on writes to CR4.UMIP,4601 /* SECONDARY_EXEC_DESC is enabled/disabled on writes to CR4.UMIP,
4523 * in vmx_set_cr4. */4602 * in vmx_set_cr4. */
4524 exec_control &= ~SECONDARY_EXEC_DESC;4603 exec_control &= ~SECONDARY_EXEC_DESC;
@@ -4535,7 +4614,7 @@ static u32 vmx_secondary_exec_control(struct vcpu_vmx *vmx) · 10 unmodified lines
4535 * it needs to be set here when dirty logging is already active, e.g.4614 * it needs to be set here when dirty logging is already active, e.g.
4536 * if this vCPU was created after dirty logging was enabled.4615 * if this vCPU was created after dirty logging was enabled.
4537 */4616 */
4538 if (!vcpu->kvm->arch.cpu_dirty_logging_count)4617 if (!enable_pml || !atomic_read(&vcpu->kvm->nr_memslots_dirty_logging))
4539 exec_control &= ~SECONDARY_EXEC_ENABLE_PML;4618 exec_control &= ~SECONDARY_EXEC_ENABLE_PML;
4540 4619
4541 if (cpu_has_vmx_xsaves()) {4620 if (cpu_has_vmx_xsaves()) {
@@ -5099,8 +5178,13 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu) · 557 unmodified lines
5099 vect_info = vmx->idt_vectoring_info;5178 vect_info = vmx->idt_vectoring_info;
5100 intr_info = vmx_get_intr_info(vcpu);5179 intr_info = vmx_get_intr_info(vcpu);
5101 5180
5181 /*
5182 * Machine checks are handled by handle_exception_irqoff(), or by
5183 * vmx_vcpu_run() if a #MC occurs on VM-Entry. NMIs are handled by
5184 * vmx_vcpu_enter_exit().
5185 */
5102 if (is_machine_check(intr_info) || is_nmi(intr_info))5186 if (is_machine_check(intr_info) || is_nmi(intr_info))
5103 return 1; /* handled by handle_exception_nmi_irqoff() */5187 return 1;
5104 5188
5105 /*5189 /*
5106 * Queue the exception here instead of in handle_nm_fault_irqoff().5190 * Queue the exception here instead of in handle_nm_fault_irqoff().
@@ -6790,17 +6874,8 @@ static void vmx_apicv_post_state_restore(struct kvm_vcpu *vcpu) · 1683 unmodified lines
6790 memset(vmx->pi_desc.pir, 0, sizeof(vmx->pi_desc.pir));6874 memset(vmx->pi_desc.pir, 0, sizeof(vmx->pi_desc.pir));
6791}6875}
6792 6876
6793void vmx_do_interrupt_nmi_irqoff(unsigned long entry);6877void vmx_do_interrupt_irqoff(unsigned long entry);
6794 6878void vmx_do_nmi_irqoff(void);
6795static void handle_interrupt_nmi_irqoff(struct kvm_vcpu *vcpu,
6796 unsigned long entry)
6797{
6798 bool is_nmi = entry == (unsigned long)asm_exc_nmi_noist;
6799
6800 kvm_before_interrupt(vcpu, is_nmi ? KVM_HANDLING_NMI : KVM_HANDLING_IRQ);
6801 vmx_do_interrupt_nmi_irqoff(entry);
6802 kvm_after_interrupt(vcpu);
6803}
6804 6879
6805static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu)6880static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu)
6806{6881{
@@ -6822,9 +6897,8 @@ static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu) · 15 unmodified lines
6822 rdmsrl(MSR_IA32_XFD_ERR, vcpu->arch.guest_fpu.xfd_err);6897 rdmsrl(MSR_IA32_XFD_ERR, vcpu->arch.guest_fpu.xfd_err);
6823}6898}
6824 6899
6825static void handle_exception_nmi_irqoff(struct vcpu_vmx *vmx)6900static void handle_exception_irqoff(struct vcpu_vmx *vmx)
6826{6901{
6827 const unsigned long nmi_entry = (unsigned long)asm_exc_nmi_noist;
6828 u32 intr_info = vmx_get_intr_info(&vmx->vcpu);6902 u32 intr_info = vmx_get_intr_info(&vmx->vcpu);
6829 6903
6830 /* if exit due to PF check for async PF */6904 /* if exit due to PF check for async PF */
@@ -6836,9 +6910,6 @@ static void handle_exception_nmi_irqoff(struct vcpu_vmx *vmx) · 5 unmodified lines
6836 /* Handle machine checks before interrupts are enabled */6910 /* Handle machine checks before interrupts are enabled */
6837 else if (is_machine_check(intr_info))6911 else if (is_machine_check(intr_info))
6838 kvm_machine_check();6912 kvm_machine_check();
6839 /* We need to handle NMIs before interrupts are enabled */
6840 else if (is_nmi(intr_info))
6841 handle_interrupt_nmi_irqoff(&vmx->vcpu, nmi_entry);
6842}6913}
6843 6914
6844static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu)6915static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu)
@@ -6848,10 +6919,13 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu) · 3 unmodified lines
6848 gate_desc *desc = (gate_desc *)host_idt_base + vector;6919 gate_desc *desc = (gate_desc *)host_idt_base + vector;
6849 6920
6850 if (KVM_BUG(!is_external_intr(intr_info), vcpu->kvm,6921 if (KVM_BUG(!is_external_intr(intr_info), vcpu->kvm,
6851 "KVM: unexpected VM-Exit interrupt info: 0x%x", intr_info))6922 "unexpected VM-Exit interrupt info: 0x%x", intr_info))
6852 return;6923 return;
6853 6924
6854 handle_interrupt_nmi_irqoff(vcpu, gate_offset(desc));6925 kvm_before_interrupt(vcpu, KVM_HANDLING_IRQ);
6926 vmx_do_interrupt_irqoff(gate_offset(desc));
6927 kvm_after_interrupt(vcpu);
6928
6855 vcpu->arch.at_instruction_boundary = true;6929 vcpu->arch.at_instruction_boundary = true;
6856}6930}
6857 6931
@@ -6865,7 +6939,7 @@ static void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu) · 7 unmodified lines
6865 if (vmx->exit_reason.basic == EXIT_REASON_EXTERNAL_INTERRUPT)6939 if (vmx->exit_reason.basic == EXIT_REASON_EXTERNAL_INTERRUPT)
6866 handle_external_interrupt_irqoff(vcpu);6940 handle_external_interrupt_irqoff(vcpu);
6867 else if (vmx->exit_reason.basic == EXIT_REASON_EXCEPTION_NMI)6941 else if (vmx->exit_reason.basic == EXIT_REASON_EXCEPTION_NMI)
6868 handle_exception_nmi_irqoff(vmx);6942 handle_exception_irqoff(vmx);
6869}6943}
6870 6944
6871/*6945/*
@@ -7100,9 +7174,10 @@ static fastpath_t vmx_exit_handlers_fastpath(struct kvm_vcpu *vcpu) · 228 unmodified lines
7100}7174}
7101 7175
7102static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu,7176static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu,
7103 struct vcpu_vmx *vmx,7177 unsigned int flags)
7104 unsigned long flags)
7105{7178{
7179 struct vcpu_vmx *vmx = to_vmx(vcpu);
7180
7106 guest_state_enter_irqoff();7181 guest_state_enter_irqoff();
7107 7182
7108 /* L1D Flush includes CPU buffer clear to mitigate MDS */7183 /* L1D Flush includes CPU buffer clear to mitigate MDS */
@@ -7126,6 +7201,18 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, · 17 unmodified lines
7126 7201
7127 vmx_enable_fb_clear(vmx);7202 vmx_enable_fb_clear(vmx);
7128 7203
7204 if (unlikely(vmx->fail))
7205 vmx->exit_reason.full = 0xdead;
7206 else
7207 vmx->exit_reason.full = vmcs_read32(VM_EXIT_REASON);
7208
7209 if ((u16)vmx->exit_reason.basic == EXIT_REASON_EXCEPTION_NMI &&
7210 is_nmi(vmx_get_intr_info(vcpu))) {
7211 kvm_before_interrupt(vcpu, KVM_HANDLING_NMI);
7212 vmx_do_nmi_irqoff();
7213 kvm_after_interrupt(vcpu);
7214 }
7215
7129 guest_state_exit_irqoff();7216 guest_state_exit_irqoff();
7130}7217}
7131 7218
@@ -7220,7 +7307,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) · 88 unmodified lines
7220 kvm_wait_lapic_expire(vcpu);7307 kvm_wait_lapic_expire(vcpu);
7221 7308
7222 /* The actual VMENTER/EXIT is in the .noinstr.text section. */7309 /* The actual VMENTER/EXIT is in the .noinstr.text section. */
7223 vmx_vcpu_enter_exit(vcpu, vmx, __vmx_vcpu_run_flags(vmx));7310 vmx_vcpu_enter_exit(vcpu, __vmx_vcpu_run_flags(vmx));
7224 7311
7225 /* All fields are clean at this point */7312 /* All fields are clean at this point */
7226 if (static_branch_unlikely(&enable_evmcs)) {7313 if (static_branch_unlikely(&enable_evmcs)) {
@@ -7267,12 +7354,9 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) · 40 unmodified lines
7267 7354
7268 vmx->idt_vectoring_info = 0;7355 vmx->idt_vectoring_info = 0;
7269 7356
7270 if (unlikely(vmx->fail)) {7357 if (unlikely(vmx->fail))
7271 vmx->exit_reason.full = 0xdead;
7272 return EXIT_FASTPATH_NONE;7358 return EXIT_FASTPATH_NONE;
7273 }
7274 7359
7275 vmx->exit_reason.full = vmcs_read32(VM_EXIT_REASON);
7276 if (unlikely((u16)vmx->exit_reason.basic == EXIT_REASON_MCE_DURING_VMENTRY))7360 if (unlikely((u16)vmx->exit_reason.basic == EXIT_REASON_MCE_DURING_VMENTRY))
7277 kvm_machine_check();7361 kvm_machine_check();
7278 7362
@@ -7386,7 +7470,7 @@ static int vmx_vcpu_create(struct kvm_vcpu *vcpu) · 107 unmodified lines
7386 vmx->loaded_vmcs = &vmx->vmcs01;7470 vmx->loaded_vmcs = &vmx->vmcs01;
7387 7471
7388 if (cpu_need_virtualize_apic_accesses(vcpu)) {7472 if (cpu_need_virtualize_apic_accesses(vcpu)) {
7389 err = alloc_apic_access_page(vcpu->kvm);7473 err = kvm_alloc_apic_access_page(vcpu->kvm);
7390 if (err)7474 if (err)
7391 goto free_vmcs;7475 goto free_vmcs;
7392 }7476 }
@@ -7446,29 +7530,6 @@ static int vmx_vm_init(struct kvm *kvm) · 53 unmodified lines
7446 return 0;7530 return 0;
7447}7531}
7448 7532
7449static int __init vmx_check_processor_compat(void)
7450{
7451 struct vmcs_config vmcs_conf;
7452 struct vmx_capability vmx_cap;
7453
7454 if (!this_cpu_has(X86_FEATURE_MSR_IA32_FEAT_CTL) ||
7455 !this_cpu_has(X86_FEATURE_VMX)) {
7456 pr_err("kvm: VMX is disabled on CPU %d\n", smp_processor_id());
7457 return -EIO;
7458 }
7459
7460 if (setup_vmcs_config(&vmcs_conf, &vmx_cap) < 0)
7461 return -EIO;
7462 if (nested)
7463 nested_vmx_setup_ctls_msrs(&vmcs_conf, vmx_cap.ept);
7464 if (memcmp(&vmcs_config, &vmcs_conf, sizeof(struct vmcs_config)) != 0) {
7465 printk(KERN_ERR "kvm: CPU %d feature inconsistency!\n",
7466 smp_processor_id());
7467 return -EIO;
7468 }
7469 return 0;
7470}
7471
7472static u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)7533static u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)
7473{7534{
7474 u8 cache;7535 u8 cache;
@@ -7940,17 +8001,20 @@ void vmx_update_cpu_dirty_logging(struct kvm_vcpu *vcpu) · 465 unmodified lines
7940{8001{
7941 struct vcpu_vmx *vmx = to_vmx(vcpu);8002 struct vcpu_vmx *vmx = to_vmx(vcpu);
7942 8003
8004 if (WARN_ON_ONCE(!enable_pml))
8005 return;
8006
7943 if (is_guest_mode(vcpu)) {8007 if (is_guest_mode(vcpu)) {
7944 vmx->nested.update_vmcs01_cpu_dirty_logging = true;8008 vmx->nested.update_vmcs01_cpu_dirty_logging = true;
7945 return;8009 return;
7946 }8010 }
7947 8011
7948 /*8012 /*
7949 * Note, cpu_dirty_logging_count can be changed concurrent with this8013 * Note, nr_memslots_dirty_logging can be changed concurrent with this
7950 * code, but in that case another update request will be made and so8014 * code, but in that case another update request will be made and so
7951 * the guest will never run with a stale PML value.8015 * the guest will never run with a stale PML value.
7952 */8016 */
7953 if (vcpu->kvm->arch.cpu_dirty_logging_count)8017 if (atomic_read(&vcpu->kvm->nr_memslots_dirty_logging))
7954 secondary_exec_controls_setbit(vmx, SECONDARY_EXEC_ENABLE_PML);8018 secondary_exec_controls_setbit(vmx, SECONDARY_EXEC_ENABLE_PML);
7955 else8019 else
7956 secondary_exec_controls_clearbit(vmx, SECONDARY_EXEC_ENABLE_PML);8020 secondary_exec_controls_clearbit(vmx, SECONDARY_EXEC_ENABLE_PML);
@@ -8048,17 +8112,16 @@ static void vmx_hardware_unsetup(void) · 91 unmodified lines
8048 free_kvm_area();8112 free_kvm_area();
8049}8113}
8050 8114
8051static bool vmx_check_apicv_inhibit_reasons(enum kvm_apicv_inhibit reason)8115#define VMX_REQUIRED_APICV_INHIBITS \
8052{8116( \
8053 ulong supported = BIT(APICV_INHIBIT_REASON_DISABLE) |8117 BIT(APICV_INHIBIT_REASON_DISABLE)| \
8054 BIT(APICV_INHIBIT_REASON_ABSENT) |8118 BIT(APICV_INHIBIT_REASON_ABSENT) | \
8055 BIT(APICV_INHIBIT_REASON_HYPERV) |8119 BIT(APICV_INHIBIT_REASON_HYPERV) | \
8056 BIT(APICV_INHIBIT_REASON_BLOCKIRQ) |8120 BIT(APICV_INHIBIT_REASON_BLOCKIRQ) | \
8057 BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) |8121 BIT(APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED) | \
8058 BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED);8122 BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) | \
8059 8123 BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED) \
8060 return supported & BIT(reason);8124)
8061}
8062 8125
8063static void vmx_vm_destroy(struct kvm *kvm)8126static void vmx_vm_destroy(struct kvm *kvm)
8064{8127{
@@ -8068,7 +8131,9 @@ static void vmx_vm_destroy(struct kvm *kvm) · 3 unmodified lines
8068}8131}
8069 8132
8070static struct kvm_x86_ops vmx_x86_ops __initdata = {8133static struct kvm_x86_ops vmx_x86_ops __initdata = {
8071 .name = "kvm_intel",8134 .name = KBUILD_MODNAME,
8135
8136 .check_processor_compatibility = vmx_check_processor_compat,
8072 8137
8073 .hardware_unsetup = vmx_hardware_unsetup,8138 .hardware_unsetup = vmx_hardware_unsetup,
8074 8139
@@ -8142,7 +8207,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { · 67 unmodified lines
8142 .refresh_apicv_exec_ctrl = vmx_refresh_apicv_exec_ctrl,8207 .refresh_apicv_exec_ctrl = vmx_refresh_apicv_exec_ctrl,
8143 .load_eoi_exitmap = vmx_load_eoi_exitmap,8208 .load_eoi_exitmap = vmx_load_eoi_exitmap,
8144 .apicv_post_state_restore = vmx_apicv_post_state_restore,8209 .apicv_post_state_restore = vmx_apicv_post_state_restore,
8145 .check_apicv_inhibit_reasons = vmx_check_apicv_inhibit_reasons,8210 .required_apicv_inhibits = VMX_REQUIRED_APICV_INHIBITS,
8146 .hwapic_irr_update = vmx_hwapic_irr_update,8211 .hwapic_irr_update = vmx_hwapic_irr_update,
8147 .hwapic_isr_update = vmx_hwapic_isr_update,8212 .hwapic_isr_update = vmx_hwapic_isr_update,
8148 .guest_apic_has_interrupt = vmx_guest_apic_has_interrupt,8213 .guest_apic_has_interrupt = vmx_guest_apic_has_interrupt,
@@ -8288,7 +8353,7 @@ static __init int hardware_setup(void) · 139 unmodified lines
8288 return -EIO;8353 return -EIO;
8289 8354
8290 if (cpu_has_perf_global_ctrl_bug())8355 if (cpu_has_perf_global_ctrl_bug())
8291 pr_warn_once("kvm: VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL "8356 pr_warn_once("VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL "
8292 "does not work properly. Using workaround\n");8357 "does not work properly. Using workaround\n");
8293 8358
8294 if (boot_cpu_has(X86_FEATURE_NX))8359 if (boot_cpu_has(X86_FEATURE_NX))
@@ -8296,7 +8361,7 @@ static __init int hardware_setup(void) · 1 unmodified line
8296 8361
8297 if (boot_cpu_has(X86_FEATURE_MPX)) {8362 if (boot_cpu_has(X86_FEATURE_MPX)) {
8298 rdmsrl(MSR_IA32_BNDCFGS, host_bndcfgs);8363 rdmsrl(MSR_IA32_BNDCFGS, host_bndcfgs);
8299 WARN_ONCE(host_bndcfgs, "KVM: BNDCFGS in host will be lost");8364 WARN_ONCE(host_bndcfgs, "BNDCFGS in host will be lost");
8300 }8365 }
8301 8366
8302 if (!cpu_has_vmx_mpx())8367 if (!cpu_has_vmx_mpx())
@@ -8315,7 +8380,7 @@ static __init int hardware_setup(void) · 12 unmodified lines
8315 8380
8316 /* NX support is required for shadow paging. */8381 /* NX support is required for shadow paging. */
8317 if (!enable_ept && !boot_cpu_has(X86_FEATURE_NX)) {8382 if (!enable_ept && !boot_cpu_has(X86_FEATURE_NX)) {
8318 pr_err_ratelimited("kvm: NX (Execute Disable) not supported\n");8383 pr_err_ratelimited("NX (Execute Disable) not supported\n");
8319 return -EOPNOTSUPP;8384 return -EOPNOTSUPP;
8320 }8385 }
8321 8386
@@ -8467,9 +8532,6 @@ static __init int hardware_setup(void) · 145 unmodified lines
8467}8532}
8468 8533
8469static struct kvm_x86_init_ops vmx_init_ops __initdata = {8534static struct kvm_x86_init_ops vmx_init_ops __initdata = {
8470 .cpu_has_kvm_support = cpu_has_kvm_support,
8471 .disabled_by_bios = vmx_disabled_by_bios,
8472 .check_processor_compatibility = vmx_check_processor_compat,
8473 .hardware_setup = hardware_setup,8535 .hardware_setup = hardware_setup,
8474 .handle_intel_pt_intr = NULL,8536 .handle_intel_pt_intr = NULL,
8475 8537
@@ -8487,41 +8549,23 @@ static void vmx_cleanup_l1d_flush(void) · 11 unmodified lines
8487 l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_AUTO;8549 l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_AUTO;
8488}8550}
8489 8551
8490static void vmx_exit(void)8552static void __vmx_exit(void)
8491{8553{
8554 allow_smaller_maxphyaddr = false;
8555
8492#ifdef CONFIG_KEXEC_CORE8556#ifdef CONFIG_KEXEC_CORE
8493 RCU_INIT_POINTER(crash_vmclear_loaded_vmcss, NULL);8557 RCU_INIT_POINTER(crash_vmclear_loaded_vmcss, NULL);
8494 synchronize_rcu();8558 synchronize_rcu();
8495#endif8559#endif
8560 vmx_cleanup_l1d_flush();
8561}
8496 8562
8563static void vmx_exit(void)
8564{
8497 kvm_exit();8565 kvm_exit();
8566 kvm_x86_vendor_exit();
8498 8567
8499#if IS_ENABLED(CONFIG_HYPERV)8568 __vmx_exit();
8500 if (static_branch_unlikely(&enable_evmcs)) {
8501 int cpu;
8502 struct hv_vp_assist_page *vp_ap;
8503 /*
8504 * Reset everything to support using non-enlightened VMCS
8505 * access later (e.g. when we reload the module with
8506 * enlightened_vmcs=0)
8507 */
8508 for_each_online_cpu(cpu) {
8509 vp_ap = hv_get_vp_assist_page(cpu);
8510
8511 if (!vp_ap)
8512 continue;
8513
8514 vp_ap->nested_control.features.directhypercall = 0;
8515 vp_ap->current_nested_vmcs = 0;
8516 vp_ap->enlighten_vmentry = 0;
8517 }
8518
8519 static_branch_disable(&enable_evmcs);
8520 }
8521#endif
8522 vmx_cleanup_l1d_flush();
8523
8524 allow_smaller_maxphyaddr = false;
8525}8569}
8526module_exit(vmx_exit);8570module_exit(vmx_exit);
8527 8571
@@ -8529,56 +8573,29 @@ static int __init vmx_init(void) · 1 unmodified line
8529{8573{
8530 int r, cpu;8574 int r, cpu;
8531 8575
8532#if IS_ENABLED(CONFIG_HYPERV)8576 if (!kvm_is_vmx_supported())
8577 return -EOPNOTSUPP;
8578
8533 /*8579 /*
8534 * Enlightened VMCS usage should be recommended and the host needs8580 * Note, hv_init_evmcs() touches only VMX knobs, i.e. there's nothing
8535 * to support eVMCS v1 or above. We can also disable eVMCS support8581 * to unwind if a later step fails.
8536 * with module parameter.
8537 */8582 */
8538 if (enlightened_vmcs &&8583 hv_init_evmcs();
8539 ms_hyperv.hints & HV_X64_ENLIGHTENED_VMCS_RECOMMENDED &&
8540 (ms_hyperv.nested_features & HV_X64_ENLIGHTENED_VMCS_VERSION) >=
8541 KVM_EVMCS_VERSION) {
8542
8543 /* Check that we have assist pages on all online CPUs */
8544 for_each_online_cpu(cpu) {
8545 if (!hv_get_vp_assist_page(cpu)) {
8546 enlightened_vmcs = false;
8547 break;
8548 }
8549 }
8550 8584
8551 if (enlightened_vmcs) {8585 r = kvm_x86_vendor_init(&vmx_init_ops);
8552 pr_info("KVM: vmx: using Hyper-V Enlightened VMCS\n");
8553 static_branch_enable(&enable_evmcs);
8554 }
8555
8556 if (ms_hyperv.nested_features & HV_X64_NESTED_DIRECT_FLUSH)
8557 vmx_x86_ops.enable_l2_tlb_flush
8558 = hv_enable_l2_tlb_flush;
8559
8560 } else {
8561 enlightened_vmcs = false;
8562 }
8563#endif
8564
8565 r = kvm_init(&vmx_init_ops, sizeof(struct vcpu_vmx),
8566 __alignof__(struct vcpu_vmx), THIS_MODULE);
8567 if (r)8586 if (r)
8568 return r;8587 return r;
8569 8588
8570 /*8589 /*
8571 * Must be called after kvm_init() so enable_ept is properly set8590 * Must be called after common x86 init so enable_ept is properly set
8572 * up. Hand the parameter mitigation value in which was stored in8591 * up. Hand the parameter mitigation value in which was stored in
8573 * the pre module init parser. If no parameter was given, it will8592 * the pre module init parser. If no parameter was given, it will
8574 * contain 'auto' which will be turned into the default 'cond'8593 * contain 'auto' which will be turned into the default 'cond'
8575 * mitigation mode.8594 * mitigation mode.
8576 */8595 */
8577 r = vmx_setup_l1d_flush(vmentry_l1d_flush_param);8596 r = vmx_setup_l1d_flush(vmentry_l1d_flush_param);
8578 if (r) {8597 if (r)
8579 vmx_exit();8598 goto err_l1d_flush;
8580 return r;
8581 }
8582 8599
8583 vmx_setup_fb_clear_ctrl();8600 vmx_setup_fb_clear_ctrl();
8584 8601
@@ -8602,6 +8619,21 @@ static int __init vmx_init(void) · 17 unmodified lines
8602 if (!enable_ept)8619 if (!enable_ept)
8603 allow_smaller_maxphyaddr = true;8620 allow_smaller_maxphyaddr = true;
8604 8621
8622 /*
8623 * Common KVM initialization _must_ come last, after this, /dev/kvm is
8624 * exposed to userspace!
8625 */
8626 r = kvm_init(sizeof(struct vcpu_vmx), __alignof__(struct vcpu_vmx),
8627 THIS_MODULE);
8628 if (r)
8629 goto err_kvm_init;
8630
8605 return 0;8631 return 0;
8632
8633err_kvm_init:
8634 __vmx_exit();
8635err_l1d_flush:
8636 kvm_x86_vendor_exit();
8637 return r;
8606}8638}
8607module_init(vmx_init);8639module_init(vmx_init);
arch/x86/kvm/vmx/vmx.h+8−10
@@ -640,12 +640,12 @@ BUILD_CONTROLS_SHADOW(tertiary_exec, TERTIARY_VM_EXEC_CONTROL, 64) · 639 unmodified lines
640 (1 << VCPU_EXREG_EXIT_INFO_1) | \640 (1 << VCPU_EXREG_EXIT_INFO_1) | \
641 (1 << VCPU_EXREG_EXIT_INFO_2))641 (1 << VCPU_EXREG_EXIT_INFO_2))
642 642
643static inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm)643static __always_inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm)
644{644{
645 return container_of(kvm, struct kvm_vmx, kvm);645 return container_of(kvm, struct kvm_vmx, kvm);
646}646}
647 647
648static inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu)648static __always_inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu)
649{649{
650 return container_of(vcpu, struct vcpu_vmx, vcpu);650 return container_of(vcpu, struct vcpu_vmx, vcpu);
651}651}
@@ -669,25 +669,23 @@ void intel_pmu_cross_mapped_check(struct kvm_pmu *pmu); · 17 unmodified lines
669int intel_pmu_create_guest_lbr_event(struct kvm_vcpu *vcpu);669int intel_pmu_create_guest_lbr_event(struct kvm_vcpu *vcpu);
670void vmx_passthrough_lbr_msrs(struct kvm_vcpu *vcpu);670void vmx_passthrough_lbr_msrs(struct kvm_vcpu *vcpu);
671 671
672static inline unsigned long vmx_get_exit_qual(struct kvm_vcpu *vcpu)672static __always_inline unsigned long vmx_get_exit_qual(struct kvm_vcpu *vcpu)
673{673{
674 struct vcpu_vmx *vmx = to_vmx(vcpu);674 struct vcpu_vmx *vmx = to_vmx(vcpu);
675 675
676 if (!kvm_register_is_available(vcpu, VCPU_EXREG_EXIT_INFO_1)) {676 if (!kvm_register_test_and_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_1))
677 kvm_register_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_1);
678 vmx->exit_qualification = vmcs_readl(EXIT_QUALIFICATION);677 vmx->exit_qualification = vmcs_readl(EXIT_QUALIFICATION);
679 }678
680 return vmx->exit_qualification;679 return vmx->exit_qualification;
681}680}
682 681
683static inline u32 vmx_get_intr_info(struct kvm_vcpu *vcpu)682static __always_inline u32 vmx_get_intr_info(struct kvm_vcpu *vcpu)
684{683{
685 struct vcpu_vmx *vmx = to_vmx(vcpu);684 struct vcpu_vmx *vmx = to_vmx(vcpu);
686 685
687 if (!kvm_register_is_available(vcpu, VCPU_EXREG_EXIT_INFO_2)) {686 if (!kvm_register_test_and_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_2))
688 kvm_register_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_2);
689 vmx->exit_intr_info = vmcs_read32(VM_EXIT_INTR_INFO);687 vmx->exit_intr_info = vmcs_read32(VM_EXIT_INTR_INFO);
690 }688
691 return vmx->exit_intr_info;689 return vmx->exit_intr_info;
692}690}
693 691
arch/x86/kvm/vmx/vmx_ops.h+4−2
@@ -100,8 +100,10 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field) · 99 unmodified lines
100 return value;100 return value;
101 101
102do_fail:102do_fail:
103 WARN_ONCE(1, "kvm: vmread failed: field=%lx\n", field);103 instrumentation_begin();
104 pr_warn_ratelimited("kvm: vmread failed: field=%lx\n", field);104 WARN_ONCE(1, KBUILD_MODNAME ": vmread failed: field=%lx\n", field);
105 pr_warn_ratelimited(KBUILD_MODNAME ": vmread failed: field=%lx\n", field);
106 instrumentation_end();
105 return 0;107 return 0;
106 108
107do_exception:109do_exception:
arch/x86/kvm/x86.c+310−247
@@ -15,6 +15,7 @@ · 14 unmodified lines
15 * Amit Shah <amit.shah@qumranet.com>15 * Amit Shah <amit.shah@qumranet.com>
16 * Ben-Ami Yassour <benami@il.ibm.com>16 * Ben-Ami Yassour <benami@il.ibm.com>
17 */17 */
18#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
18 19
19#include <linux/kvm_host.h>20#include <linux/kvm_host.h>
20#include "irq.h"21#include "irq.h"
@@ -128,6 +129,7 @@ static int kvm_vcpu_do_singlestep(struct kvm_vcpu *vcpu); · 107 unmodified lines
128static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2);129static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2);
129static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2);130static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2);
130 131
132static DEFINE_MUTEX(vendor_module_lock);
131struct kvm_x86_ops kvm_x86_ops __read_mostly;133struct kvm_x86_ops kvm_x86_ops __read_mostly;
132 134
133#define KVM_X86_OP(func) \135#define KVM_X86_OP(func) \
@@ -1421,7 +1423,7 @@ EXPORT_SYMBOL_GPL(kvm_emulate_rdpmc); · 1287 unmodified lines
1421 * may depend on host virtualization features rather than host cpu features.1423 * may depend on host virtualization features rather than host cpu features.
1422 */1424 */
1423 1425
1424static const u32 msrs_to_save_all[] = {1426static const u32 msrs_to_save_base[] = {
1425 MSR_IA32_SYSENTER_CS, MSR_IA32_SYSENTER_ESP, MSR_IA32_SYSENTER_EIP,1427 MSR_IA32_SYSENTER_CS, MSR_IA32_SYSENTER_ESP, MSR_IA32_SYSENTER_EIP,
1426 MSR_STAR,1428 MSR_STAR,
1427#ifdef CONFIG_X86_641429#ifdef CONFIG_X86_64
@@ -1438,6 +1440,10 @@ static const u32 msrs_to_save_all[] = { · 10 unmodified lines
1438 MSR_IA32_RTIT_ADDR3_A, MSR_IA32_RTIT_ADDR3_B,1440 MSR_IA32_RTIT_ADDR3_A, MSR_IA32_RTIT_ADDR3_B,
1439 MSR_IA32_UMWAIT_CONTROL,1441 MSR_IA32_UMWAIT_CONTROL,
1440 1442
1443 MSR_IA32_XFD, MSR_IA32_XFD_ERR,
1444};
1445
1446static const u32 msrs_to_save_pmu[] = {
1441 MSR_ARCH_PERFMON_FIXED_CTR0, MSR_ARCH_PERFMON_FIXED_CTR1,1447 MSR_ARCH_PERFMON_FIXED_CTR0, MSR_ARCH_PERFMON_FIXED_CTR1,
1442 MSR_ARCH_PERFMON_FIXED_CTR0 + 2,1448 MSR_ARCH_PERFMON_FIXED_CTR0 + 2,
1443 MSR_CORE_PERF_FIXED_CTR_CTRL, MSR_CORE_PERF_GLOBAL_STATUS,1449 MSR_CORE_PERF_FIXED_CTR_CTRL, MSR_CORE_PERF_GLOBAL_STATUS,
@@ -1462,11 +1468,10 @@ static const u32 msrs_to_save_all[] = { · 18 unmodified lines
1462 MSR_F15H_PERF_CTL3, MSR_F15H_PERF_CTL4, MSR_F15H_PERF_CTL5,1468 MSR_F15H_PERF_CTL3, MSR_F15H_PERF_CTL4, MSR_F15H_PERF_CTL5,
1463 MSR_F15H_PERF_CTR0, MSR_F15H_PERF_CTR1, MSR_F15H_PERF_CTR2,1469 MSR_F15H_PERF_CTR0, MSR_F15H_PERF_CTR1, MSR_F15H_PERF_CTR2,
1464 MSR_F15H_PERF_CTR3, MSR_F15H_PERF_CTR4, MSR_F15H_PERF_CTR5,1470 MSR_F15H_PERF_CTR3, MSR_F15H_PERF_CTR4, MSR_F15H_PERF_CTR5,
1465
1466 MSR_IA32_XFD, MSR_IA32_XFD_ERR,
1467};1471};
1468 1472
1469static u32 msrs_to_save[ARRAY_SIZE(msrs_to_save_all)];1473static u32 msrs_to_save[ARRAY_SIZE(msrs_to_save_base) +
1474 ARRAY_SIZE(msrs_to_save_pmu)];
1470static unsigned num_msrs_to_save;1475static unsigned num_msrs_to_save;
1471 1476
1472static const u32 emulated_msrs_all[] = {1477static const u32 emulated_msrs_all[] = {
@@ -1484,7 +1489,7 @@ static const u32 emulated_msrs_all[] = { · 11 unmodified lines
1484 HV_X64_MSR_STIMER0_CONFIG,1489 HV_X64_MSR_STIMER0_CONFIG,
1485 HV_X64_MSR_VP_ASSIST_PAGE,1490 HV_X64_MSR_VP_ASSIST_PAGE,
1486 HV_X64_MSR_REENLIGHTENMENT_CONTROL, HV_X64_MSR_TSC_EMULATION_CONTROL,1491 HV_X64_MSR_REENLIGHTENMENT_CONTROL, HV_X64_MSR_TSC_EMULATION_CONTROL,
1487 HV_X64_MSR_TSC_EMULATION_STATUS,1492 HV_X64_MSR_TSC_EMULATION_STATUS, HV_X64_MSR_TSC_INVARIANT_CONTROL,
1488 HV_X64_MSR_SYNDBG_OPTIONS,1493 HV_X64_MSR_SYNDBG_OPTIONS,
1489 HV_X64_MSR_SYNDBG_CONTROL, HV_X64_MSR_SYNDBG_STATUS,1494 HV_X64_MSR_SYNDBG_CONTROL, HV_X64_MSR_SYNDBG_STATUS,
1490 HV_X64_MSR_SYNDBG_SEND_BUFFER, HV_X64_MSR_SYNDBG_RECV_BUFFER,1495 HV_X64_MSR_SYNDBG_SEND_BUFFER, HV_X64_MSR_SYNDBG_RECV_BUFFER,
@@ -2093,7 +2098,7 @@ static int kvm_emulate_monitor_mwait(struct kvm_vcpu *vcpu, const char *insn) · 602 unmodified lines
2093 !guest_cpuid_has(vcpu, X86_FEATURE_MWAIT))2098 !guest_cpuid_has(vcpu, X86_FEATURE_MWAIT))
2094 return kvm_handle_invalid_op(vcpu);2099 return kvm_handle_invalid_op(vcpu);
2095 2100
2096 pr_warn_once("kvm: %s instruction emulated as NOP!\n", insn);2101 pr_warn_once("%s instruction emulated as NOP!\n", insn);
2097 return kvm_emulate_as_nop(vcpu);2102 return kvm_emulate_as_nop(vcpu);
2098}2103}
2099int kvm_emulate_mwait(struct kvm_vcpu *vcpu)2104int kvm_emulate_mwait(struct kvm_vcpu *vcpu)
@@ -2440,7 +2445,8 @@ static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz) · 340 unmodified lines
2440 thresh_lo = adjust_tsc_khz(tsc_khz, -tsc_tolerance_ppm);2445 thresh_lo = adjust_tsc_khz(tsc_khz, -tsc_tolerance_ppm);
2441 thresh_hi = adjust_tsc_khz(tsc_khz, tsc_tolerance_ppm);2446 thresh_hi = adjust_tsc_khz(tsc_khz, tsc_tolerance_ppm);
2442 if (user_tsc_khz < thresh_lo || user_tsc_khz > thresh_hi) {2447 if (user_tsc_khz < thresh_lo || user_tsc_khz > thresh_hi) {
2443 pr_debug("kvm: requested TSC rate %u falls outside tolerance [%u,%u]\n", user_tsc_khz, thresh_lo, thresh_hi);2448 pr_debug("requested TSC rate %u falls outside tolerance [%u,%u]\n",
2449 user_tsc_khz, thresh_lo, thresh_hi);
2444 use_scaling = 1;2450 use_scaling = 1;
2445 }2451 }
2446 return set_tsc_khz(vcpu, user_tsc_khz, use_scaling);2452 return set_tsc_khz(vcpu, user_tsc_khz, use_scaling);
@@ -3165,6 +3171,7 @@ static int kvm_guest_time_update(struct kvm_vcpu *v) · 718 unmodified lines
3165 &vcpu->hv_clock.tsc_shift,3171 &vcpu->hv_clock.tsc_shift,
3166 &vcpu->hv_clock.tsc_to_system_mul);3172 &vcpu->hv_clock.tsc_to_system_mul);
3167 vcpu->hw_tsc_khz = tgt_tsc_khz;3173 vcpu->hw_tsc_khz = tgt_tsc_khz;
3174 kvm_xen_update_tsc_info(v);
3168 }3175 }
3169 3176
3170 vcpu->hv_clock.tsc_timestamp = tsc_timestamp;3177 vcpu->hv_clock.tsc_timestamp = tsc_timestamp;
@@ -3562,9 +3569,20 @@ static void record_steal_time(struct kvm_vcpu *vcpu) · 391 unmodified lines
3562 mark_page_dirty_in_slot(vcpu->kvm, ghc->memslot, gpa_to_gfn(ghc->gpa));3569 mark_page_dirty_in_slot(vcpu->kvm, ghc->memslot, gpa_to_gfn(ghc->gpa));
3563}3570}
3564 3571
3572static bool kvm_is_msr_to_save(u32 msr_index)
3573{
3574 unsigned int i;
3575
3576 for (i = 0; i < num_msrs_to_save; i++) {
3577 if (msrs_to_save[i] == msr_index)
3578 return true;
3579 }
3580
3581 return false;
3582}
3583
3565int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)3584int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
3566{3585{
3567 bool pr = false;
3568 u32 msr = msr_info->index;3586 u32 msr = msr_info->index;
3569 u64 data = msr_info->data;3587 u64 data = msr_info->data;
3570 3588
@@ -3610,15 +3628,13 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 39 unmodified lines
3610 if (data == BIT_ULL(18)) {3628 if (data == BIT_ULL(18)) {
3611 vcpu->arch.msr_hwcr = data;3629 vcpu->arch.msr_hwcr = data;
3612 } else if (data != 0) {3630 } else if (data != 0) {
3613 vcpu_unimpl(vcpu, "unimplemented HWCR wrmsr: 0x%llx\n",3631 kvm_pr_unimpl_wrmsr(vcpu, msr, data);
3614 data);
3615 return 1;3632 return 1;
3616 }3633 }
3617 break;3634 break;
3618 case MSR_FAM10H_MMIO_CONF_BASE:3635 case MSR_FAM10H_MMIO_CONF_BASE:
3619 if (data != 0) {3636 if (data != 0) {
3620 vcpu_unimpl(vcpu, "unimplemented MMIO_CONF_BASE wrmsr: "3637 kvm_pr_unimpl_wrmsr(vcpu, msr, data);
3621 "0x%llx\n", data);
3622 return 1;3638 return 1;
3623 }3639 }
3624 break;3640 break;
@@ -3798,16 +3814,13 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 173 unmodified lines
3798 3814
3799 case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3:3815 case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3:
3800 case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1:3816 case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1:
3801 pr = true;
3802 fallthrough;
3803 case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3:3817 case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3:
3804 case MSR_P6_EVNTSEL0 ... MSR_P6_EVNTSEL1:3818 case MSR_P6_EVNTSEL0 ... MSR_P6_EVNTSEL1:
3805 if (kvm_pmu_is_valid_msr(vcpu, msr))3819 if (kvm_pmu_is_valid_msr(vcpu, msr))
3806 return kvm_pmu_set_msr(vcpu, msr_info);3820 return kvm_pmu_set_msr(vcpu, msr_info);
3807 3821
3808 if (pr || data != 0)3822 if (data)
3809 vcpu_unimpl(vcpu, "disabled perfctr wrmsr: "3823 kvm_pr_unimpl_wrmsr(vcpu, msr, data);
3810 "0x%x data 0x%llx\n", msr, data);
3811 break;3824 break;
3812 case MSR_K7_CLK_CTL:3825 case MSR_K7_CLK_CTL:
3813 /*3826 /*
@@ -3828,15 +3841,14 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 14 unmodified lines
3828 case HV_X64_MSR_REENLIGHTENMENT_CONTROL:3841 case HV_X64_MSR_REENLIGHTENMENT_CONTROL:
3829 case HV_X64_MSR_TSC_EMULATION_CONTROL:3842 case HV_X64_MSR_TSC_EMULATION_CONTROL:
3830 case HV_X64_MSR_TSC_EMULATION_STATUS:3843 case HV_X64_MSR_TSC_EMULATION_STATUS:
3844 case HV_X64_MSR_TSC_INVARIANT_CONTROL:
3831 return kvm_hv_set_msr_common(vcpu, msr, data,3845 return kvm_hv_set_msr_common(vcpu, msr, data,
3832 msr_info->host_initiated);3846 msr_info->host_initiated);
3833 case MSR_IA32_BBL_CR_CTL3:3847 case MSR_IA32_BBL_CR_CTL3:
3834 /* Drop writes to this legacy MSR -- see rdmsr3848 /* Drop writes to this legacy MSR -- see rdmsr
3835 * counterpart for further detail.3849 * counterpart for further detail.
3836 */3850 */
3837 if (report_ignored_msrs)3851 kvm_pr_unimpl_wrmsr(vcpu, msr, data);
3838 vcpu_unimpl(vcpu, "ignored wrmsr: 0x%x data 0x%llx\n",
3839 msr, data);
3840 break;3852 break;
3841 case MSR_AMD64_OSVW_ID_LENGTH:3853 case MSR_AMD64_OSVW_ID_LENGTH:
3842 if (!guest_cpuid_has(vcpu, X86_FEATURE_OSVW))3854 if (!guest_cpuid_has(vcpu, X86_FEATURE_OSVW))
@@ -3884,20 +3896,18 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 41 unmodified lines
3884 vcpu->arch.guest_fpu.xfd_err = data;3896 vcpu->arch.guest_fpu.xfd_err = data;
3885 break;3897 break;
3886#endif3898#endif
3887 case MSR_IA32_PEBS_ENABLE:3899 default:
3888 case MSR_IA32_DS_AREA:
3889 case MSR_PEBS_DATA_CFG:
3890 case MSR_F15H_PERF_CTL0 ... MSR_F15H_PERF_CTR5:
3891 if (kvm_pmu_is_valid_msr(vcpu, msr))3900 if (kvm_pmu_is_valid_msr(vcpu, msr))
3892 return kvm_pmu_set_msr(vcpu, msr_info);3901 return kvm_pmu_set_msr(vcpu, msr_info);
3902
3893 /*3903 /*
3894 * Userspace is allowed to write '0' to MSRs that KVM reports3904 * Userspace is allowed to write '0' to MSRs that KVM reports
3895 * as to-be-saved, even if an MSRs isn't fully supported.3905 * as to-be-saved, even if an MSRs isn't fully supported.
3896 */3906 */
3897 return !msr_info->host_initiated || data;3907 if (msr_info->host_initiated && !data &&
3898 default:3908 kvm_is_msr_to_save(msr))
3899 if (kvm_pmu_is_valid_msr(vcpu, msr))3909 break;
3900 return kvm_pmu_set_msr(vcpu, msr_info);3910
3901 return KVM_MSR_RET_INVALID;3911 return KVM_MSR_RET_INVALID;
3902 }3912 }
3903 return 0;3913 return 0;
@@ -3987,20 +3997,6 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 83 unmodified lines
3987 case MSR_DRAM_ENERGY_STATUS: /* DRAM controller */3997 case MSR_DRAM_ENERGY_STATUS: /* DRAM controller */
3988 msr_info->data = 0;3998 msr_info->data = 0;
3989 break;3999 break;
3990 case MSR_IA32_PEBS_ENABLE:
3991 case MSR_IA32_DS_AREA:
3992 case MSR_PEBS_DATA_CFG:
3993 case MSR_F15H_PERF_CTL0 ... MSR_F15H_PERF_CTR5:
3994 if (kvm_pmu_is_valid_msr(vcpu, msr_info->index))
3995 return kvm_pmu_get_msr(vcpu, msr_info);
3996 /*
3997 * Userspace is allowed to read MSRs that KVM reports as
3998 * to-be-saved, even if an MSR isn't fully supported.
3999 */
4000 if (!msr_info->host_initiated)
4001 return 1;
4002 msr_info->data = 0;
4003 break;
4004 case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3:4000 case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3:
4005 case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3:4001 case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3:
4006 case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1:4002 case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1:
@@ -4198,6 +4194,7 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 191 unmodified lines
4198 case HV_X64_MSR_REENLIGHTENMENT_CONTROL:4194 case HV_X64_MSR_REENLIGHTENMENT_CONTROL:
4199 case HV_X64_MSR_TSC_EMULATION_CONTROL:4195 case HV_X64_MSR_TSC_EMULATION_CONTROL:
4200 case HV_X64_MSR_TSC_EMULATION_STATUS:4196 case HV_X64_MSR_TSC_EMULATION_STATUS:
4197 case HV_X64_MSR_TSC_INVARIANT_CONTROL:
4201 return kvm_hv_get_msr_common(vcpu,4198 return kvm_hv_get_msr_common(vcpu,
4202 msr_info->index, &msr_info->data,4199 msr_info->index, &msr_info->data,
4203 msr_info->host_initiated);4200 msr_info->host_initiated);
@@ -4255,6 +4252,17 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 51 unmodified lines
4255 default:4252 default:
4256 if (kvm_pmu_is_valid_msr(vcpu, msr_info->index))4253 if (kvm_pmu_is_valid_msr(vcpu, msr_info->index))
4257 return kvm_pmu_get_msr(vcpu, msr_info);4254 return kvm_pmu_get_msr(vcpu, msr_info);
4255
4256 /*
4257 * Userspace is allowed to read MSRs that KVM reports as
4258 * to-be-saved, even if an MSR isn't fully supported.
4259 */
4260 if (msr_info->host_initiated &&
4261 kvm_is_msr_to_save(msr_info->index)) {
4262 msr_info->data = 0;
4263 break;
4264 }
4265
4258 return KVM_MSR_RET_INVALID;4266 return KVM_MSR_RET_INVALID;
4259 }4267 }
4260 return 0;4268 return 0;
@@ -4292,8 +4300,8 @@ static int msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs, · 31 unmodified lines
4292{4300{
4293 struct kvm_msrs msrs;4301 struct kvm_msrs msrs;
4294 struct kvm_msr_entry *entries;4302 struct kvm_msr_entry *entries;
4295 int r, n;
4296 unsigned size;4303 unsigned size;
4304 int r;
4297 4305
4298 r = -EFAULT;4306 r = -EFAULT;
4299 if (copy_from_user(&msrs, user_msrs, sizeof(msrs)))4307 if (copy_from_user(&msrs, user_msrs, sizeof(msrs)))
@@ -4310,17 +4318,11 @@ static int msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs, · 10 unmodified lines
4310 goto out;4318 goto out;
4311 }4319 }
4312 4320
4313 r = n = __msr_io(vcpu, &msrs, entries, do_msr);4321 r = __msr_io(vcpu, &msrs, entries, do_msr);
4314 if (r < 0)
4315 goto out_free;
4316 4322
4317 r = -EFAULT;
4318 if (writeback && copy_to_user(user_msrs->entries, entries, size))4323 if (writeback && copy_to_user(user_msrs->entries, entries, size))
4319 goto out_free;4324 r = -EFAULT;
4320
4321 r = n;
4322 4325
4323out_free:
4324 kfree(entries);4326 kfree(entries);
4325out:4327out:
4326 return r;4328 return r;
@@ -4408,6 +4410,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · 81 unmodified lines
4408 case KVM_CAP_SPLIT_IRQCHIP:4410 case KVM_CAP_SPLIT_IRQCHIP:
4409 case KVM_CAP_IMMEDIATE_EXIT:4411 case KVM_CAP_IMMEDIATE_EXIT:
4410 case KVM_CAP_PMU_EVENT_FILTER:4412 case KVM_CAP_PMU_EVENT_FILTER:
4413 case KVM_CAP_PMU_EVENT_MASKED_EVENTS:
4411 case KVM_CAP_GET_MSR_FEATURES:4414 case KVM_CAP_GET_MSR_FEATURES:
4412 case KVM_CAP_MSR_PLATFORM_INFO:4415 case KVM_CAP_MSR_PLATFORM_INFO:
4413 case KVM_CAP_EXCEPTION_PAYLOAD:4416 case KVM_CAP_EXCEPTION_PAYLOAD:
@@ -6482,7 +6485,7 @@ static int kvm_vm_ioctl_set_msr_filter(struct kvm *kvm, · 2068 unmodified lines
6482 struct kvm_x86_msr_filter *new_filter, *old_filter;6485 struct kvm_x86_msr_filter *new_filter, *old_filter;
6483 bool default_allow;6486 bool default_allow;
6484 bool empty = true;6487 bool empty = true;
6485 int r = 0;6488 int r;
6486 u32 i;6489 u32 i;
6487 6490
6488 if (filter->flags & ~KVM_MSR_FILTER_VALID_MASK)6491 if (filter->flags & ~KVM_MSR_FILTER_VALID_MASK)
@@ -6508,17 +6511,14 @@ static int kvm_vm_ioctl_set_msr_filter(struct kvm *kvm, · 19 unmodified lines
6508 }6511 }
6509 6512
6510 mutex_lock(&kvm->lock);6513 mutex_lock(&kvm->lock);
6511 6514 old_filter = rcu_replace_pointer(kvm->arch.msr_filter, new_filter,
6512 /* The per-VM filter is protected by kvm->lock... */6515 mutex_is_locked(&kvm->lock));
6513 old_filter = srcu_dereference_check(kvm->arch.msr_filter, &kvm->srcu, 1);6516 mutex_unlock(&kvm->lock);
6514
6515 rcu_assign_pointer(kvm->arch.msr_filter, new_filter);
6516 synchronize_srcu(&kvm->srcu);6517 synchronize_srcu(&kvm->srcu);
6517 6518
6518 kvm_free_msr_filter(old_filter);6519 kvm_free_msr_filter(old_filter);
6519 6520
6520 kvm_make_all_cpus_request(kvm, KVM_REQ_MSR_FILTER_CHANGED);6521 kvm_make_all_cpus_request(kvm, KVM_REQ_MSR_FILTER_CHANGED);
6521 mutex_unlock(&kvm->lock);
6522 6522
6523 return 0;6523 return 0;
6524}6524}
@@ -7018,83 +7018,98 @@ out: · 493 unmodified lines
7018 return r;7018 return r;
7019}7019}
7020 7020
7021static void kvm_init_msr_list(void)7021static void kvm_probe_msr_to_save(u32 msr_index)
7022{7022{
7023 u32 dummy[2];7023 u32 dummy[2];
7024
7025 if (rdmsr_safe(msr_index, &dummy[0], &dummy[1]))
7026 return;
7027
7028 /*
7029 * Even MSRs that are valid in the host may not be exposed to guests in
7030 * some cases.
7031 */
7032 switch (msr_index) {
7033 case MSR_IA32_BNDCFGS:
7034 if (!kvm_mpx_supported())
7035 return;
7036 break;
7037 case MSR_TSC_AUX:
7038 if (!kvm_cpu_cap_has(X86_FEATURE_RDTSCP) &&
7039 !kvm_cpu_cap_has(X86_FEATURE_RDPID))
7040 return;
7041 break;
7042 case MSR_IA32_UMWAIT_CONTROL:
7043 if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG))
7044 return;
7045 break;
7046 case MSR_IA32_RTIT_CTL:
7047 case MSR_IA32_RTIT_STATUS:
7048 if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT))
7049 return;
7050 break;
7051 case MSR_IA32_RTIT_CR3_MATCH:
7052 if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) ||
7053 !intel_pt_validate_hw_cap(PT_CAP_cr3_filtering))
7054 return;
7055 break;
7056 case MSR_IA32_RTIT_OUTPUT_BASE:
7057 case MSR_IA32_RTIT_OUTPUT_MASK:
7058 if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) ||
7059 (!intel_pt_validate_hw_cap(PT_CAP_topa_output) &&
7060 !intel_pt_validate_hw_cap(PT_CAP_single_range_output)))
7061 return;
7062 break;
7063 case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B:
7064 if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) ||
7065 (msr_index - MSR_IA32_RTIT_ADDR0_A >=
7066 intel_pt_validate_hw_cap(PT_CAP_num_address_ranges) * 2))
7067 return;
7068 break;
7069 case MSR_ARCH_PERFMON_PERFCTR0 ... MSR_ARCH_PERFMON_PERFCTR_MAX:
7070 if (msr_index - MSR_ARCH_PERFMON_PERFCTR0 >=
7071 kvm_pmu_cap.num_counters_gp)
7072 return;
7073 break;
7074 case MSR_ARCH_PERFMON_EVENTSEL0 ... MSR_ARCH_PERFMON_EVENTSEL_MAX:
7075 if (msr_index - MSR_ARCH_PERFMON_EVENTSEL0 >=
7076 kvm_pmu_cap.num_counters_gp)
7077 return;
7078 break;
7079 case MSR_ARCH_PERFMON_FIXED_CTR0 ... MSR_ARCH_PERFMON_FIXED_CTR_MAX:
7080 if (msr_index - MSR_ARCH_PERFMON_FIXED_CTR0 >=
7081 kvm_pmu_cap.num_counters_fixed)
7082 return;
7083 break;
7084 case MSR_IA32_XFD:
7085 case MSR_IA32_XFD_ERR:
7086 if (!kvm_cpu_cap_has(X86_FEATURE_XFD))
7087 return;
7088 break;
7089 default:
7090 break;
7091 }
7092
7093 msrs_to_save[num_msrs_to_save++] = msr_index;
7094}
7095
7096static void kvm_init_msr_list(void)
7097{
7024 unsigned i;7098 unsigned i;
7025 7099
7026 BUILD_BUG_ON_MSG(KVM_PMC_MAX_FIXED != 3,7100 BUILD_BUG_ON_MSG(KVM_PMC_MAX_FIXED != 3,
7027 "Please update the fixed PMCs in msrs_to_saved_all[]");7101 "Please update the fixed PMCs in msrs_to_save_pmu[]");
7028 7102
7029 num_msrs_to_save = 0;7103 num_msrs_to_save = 0;
7030 num_emulated_msrs = 0;7104 num_emulated_msrs = 0;
7031 num_msr_based_features = 0;7105 num_msr_based_features = 0;
7032 7106
7033 for (i = 0; i < ARRAY_SIZE(msrs_to_save_all); i++) {7107 for (i = 0; i < ARRAY_SIZE(msrs_to_save_base); i++)
7034 if (rdmsr_safe(msrs_to_save_all[i], &dummy[0], &dummy[1]) < 0)7108 kvm_probe_msr_to_save(msrs_to_save_base[i]);
7035 continue;
7036
7037 /*
7038 * Even MSRs that are valid in the host may not be exposed
7039 * to the guests in some cases.
7040 */
7041 switch (msrs_to_save_all[i]) {
7042 case MSR_IA32_BNDCFGS:
7043 if (!kvm_mpx_supported())
7044 continue;
7045 break;
7046 case MSR_TSC_AUX:
7047 if (!kvm_cpu_cap_has(X86_FEATURE_RDTSCP) &&
7048 !kvm_cpu_cap_has(X86_FEATURE_RDPID))
7049 continue;
7050 break;
7051 case MSR_IA32_UMWAIT_CONTROL:
7052 if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG))
7053 continue;
7054 break;
7055 case MSR_IA32_RTIT_CTL:
7056 case MSR_IA32_RTIT_STATUS:
7057 if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT))
7058 continue;
7059 break;
7060 case MSR_IA32_RTIT_CR3_MATCH:
7061 if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) ||
7062 !intel_pt_validate_hw_cap(PT_CAP_cr3_filtering))
7063 continue;
7064 break;
7065 case MSR_IA32_RTIT_OUTPUT_BASE:
7066 case MSR_IA32_RTIT_OUTPUT_MASK:
7067 if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) ||
7068 (!intel_pt_validate_hw_cap(PT_CAP_topa_output) &&
7069 !intel_pt_validate_hw_cap(PT_CAP_single_range_output)))
7070 continue;
7071 break;
7072 case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B:
7073 if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) ||
7074 msrs_to_save_all[i] - MSR_IA32_RTIT_ADDR0_A >=
7075 intel_pt_validate_hw_cap(PT_CAP_num_address_ranges) * 2)
7076 continue;
7077 break;
7078 case MSR_ARCH_PERFMON_PERFCTR0 ... MSR_ARCH_PERFMON_PERFCTR_MAX:
7079 if (msrs_to_save_all[i] - MSR_ARCH_PERFMON_PERFCTR0 >=
7080 min(KVM_INTEL_PMC_MAX_GENERIC, kvm_pmu_cap.num_counters_gp))
7081 continue;
7082 break;
7083 case MSR_ARCH_PERFMON_EVENTSEL0 ... MSR_ARCH_PERFMON_EVENTSEL_MAX:
7084 if (msrs_to_save_all[i] - MSR_ARCH_PERFMON_EVENTSEL0 >=
7085 min(KVM_INTEL_PMC_MAX_GENERIC, kvm_pmu_cap.num_counters_gp))
7086 continue;
7087 break;
7088 case MSR_IA32_XFD:
7089 case MSR_IA32_XFD_ERR:
7090 if (!kvm_cpu_cap_has(X86_FEATURE_XFD))
7091 continue;
7092 break;
7093 default:
7094 break;
7095 }
7096 7109
7097 msrs_to_save[num_msrs_to_save++] = msrs_to_save_all[i];7110 if (enable_pmu) {
7111 for (i = 0; i < ARRAY_SIZE(msrs_to_save_pmu); i++)
7112 kvm_probe_msr_to_save(msrs_to_save_pmu[i]);
7098 }7113 }
7099 7114
7100 for (i = 0; i < ARRAY_SIZE(emulated_msrs_all); i++) {7115 for (i = 0; i < ARRAY_SIZE(emulated_msrs_all); i++) {
@@ -7721,7 +7736,7 @@ static int emulator_cmpxchg_emulated(struct x86_emulate_ctxt *ctxt, · 620 unmodified lines
7721 return X86EMUL_CONTINUE;7736 return X86EMUL_CONTINUE;
7722 7737
7723emul_write:7738emul_write:
7724 printk_once(KERN_WARNING "kvm: emulating exchange as write\n");7739 pr_warn_once("emulating exchange as write\n");
7725 7740
7726 return emulator_write_emulated(ctxt, addr, new, bytes, exception);7741 return emulator_write_emulated(ctxt, addr, new, bytes, exception);
7727}7742}
@@ -8167,9 +8182,14 @@ static void emulator_set_nmi_mask(struct x86_emulate_ctxt *ctxt, bool masked) · 439 unmodified lines
8167 static_call(kvm_x86_set_nmi_mask)(emul_to_vcpu(ctxt), masked);8182 static_call(kvm_x86_set_nmi_mask)(emul_to_vcpu(ctxt), masked);
8168}8183}
8169 8184
8170static unsigned emulator_get_hflags(struct x86_emulate_ctxt *ctxt)8185static bool emulator_is_smm(struct x86_emulate_ctxt *ctxt)
8186{
8187 return is_smm(emul_to_vcpu(ctxt));
8188}
8189
8190static bool emulator_is_guest_mode(struct x86_emulate_ctxt *ctxt)
8171{8191{
8172 return emul_to_vcpu(ctxt)->arch.hflags;8192 return is_guest_mode(emul_to_vcpu(ctxt));
8173}8193}
8174 8194
8175#ifndef CONFIG_KVM_SMM8195#ifndef CONFIG_KVM_SMM
@@ -8238,7 +8258,8 @@ static const struct x86_emulate_ops emulate_ops = { · 62 unmodified lines
8238 .guest_has_fxsr = emulator_guest_has_fxsr,8258 .guest_has_fxsr = emulator_guest_has_fxsr,
8239 .guest_has_rdpid = emulator_guest_has_rdpid,8259 .guest_has_rdpid = emulator_guest_has_rdpid,
8240 .set_nmi_mask = emulator_set_nmi_mask,8260 .set_nmi_mask = emulator_set_nmi_mask,
8241 .get_hflags = emulator_get_hflags,8261 .is_smm = emulator_is_smm,
8262 .is_guest_mode = emulator_is_guest_mode,
8242 .leave_smm = emulator_leave_smm,8263 .leave_smm = emulator_leave_smm,
8243 .triple_fault = emulator_triple_fault,8264 .triple_fault = emulator_triple_fault,
8244 .set_xcr = emulator_set_xcr,8265 .set_xcr = emulator_set_xcr,
@@ -8282,7 +8303,7 @@ static struct x86_emulate_ctxt *alloc_emulate_ctxt(struct kvm_vcpu *vcpu) · 37 unmodified lines
8282 8303
8283 ctxt = kmem_cache_zalloc(x86_emulator_cache, GFP_KERNEL_ACCOUNT);8304 ctxt = kmem_cache_zalloc(x86_emulator_cache, GFP_KERNEL_ACCOUNT);
8284 if (!ctxt) {8305 if (!ctxt) {
8285 pr_err("kvm: failed to allocate vcpu's emulator\n");8306 pr_err("failed to allocate vcpu's emulator\n");
8286 return NULL;8307 return NULL;
8287 }8308 }
8288 8309
@@ -8310,8 +8331,6 @@ static void init_emulate_ctxt(struct kvm_vcpu *vcpu) · 21 unmodified lines
8310 (cs_l && is_long_mode(vcpu)) ? X86EMUL_MODE_PROT64 :8331 (cs_l && is_long_mode(vcpu)) ? X86EMUL_MODE_PROT64 :
8311 cs_db ? X86EMUL_MODE_PROT32 :8332 cs_db ? X86EMUL_MODE_PROT32 :
8312 X86EMUL_MODE_PROT16;8333 X86EMUL_MODE_PROT16;
8313 BUILD_BUG_ON(HF_GUEST_MASK != X86EMUL_GUEST_MASK);
8314
8315 ctxt->interruptibility = 0;8334 ctxt->interruptibility = 0;
8316 ctxt->have_exception = false;8335 ctxt->have_exception = false;
8317 ctxt->exception.vector = -1;8336 ctxt->exception.vector = -1;
@@ -9293,35 +9312,66 @@ static struct notifier_block pvclock_gtod_notifier = { · 975 unmodified lines
9293};9312};
9294#endif9313#endif
9295 9314
9296int kvm_arch_init(void *opaque)9315static inline void kvm_ops_update(struct kvm_x86_init_ops *ops)
9316{
9317 memcpy(&kvm_x86_ops, ops->runtime_ops, sizeof(kvm_x86_ops));
9318
9319#define __KVM_X86_OP(func) \
9320 static_call_update(kvm_x86_##func, kvm_x86_ops.func);
9321#define KVM_X86_OP(func) \
9322 WARN_ON(!kvm_x86_ops.func); __KVM_X86_OP(func)
9323#define KVM_X86_OP_OPTIONAL __KVM_X86_OP
9324#define KVM_X86_OP_OPTIONAL_RET0(func) \
9325 static_call_update(kvm_x86_##func, (void *)kvm_x86_ops.func ? : \
9326 (void *)__static_call_return0);
9327#include <asm/kvm-x86-ops.h>
9328#undef __KVM_X86_OP
9329
9330 kvm_pmu_ops_update(ops->pmu_ops);
9331}
9332
9333static int kvm_x86_check_processor_compatibility(void)
9334{
9335 int cpu = smp_processor_id();
9336 struct cpuinfo_x86 *c = &cpu_data(cpu);
9337
9338 /*
9339 * Compatibility checks are done when loading KVM and when enabling
9340 * hardware, e.g. during CPU hotplug, to ensure all online CPUs are
9341 * compatible, i.e. KVM should never perform a compatibility check on
9342 * an offline CPU.
9343 */
9344 WARN_ON(!cpu_online(cpu));
9345
9346 if (__cr4_reserved_bits(cpu_has, c) !=
9347 __cr4_reserved_bits(cpu_has, &boot_cpu_data))
9348 return -EIO;
9349
9350 return static_call(kvm_x86_check_processor_compatibility)();
9351}
9352
9353static void kvm_x86_check_cpu_compat(void *ret)
9354{
9355 *(int *)ret = kvm_x86_check_processor_compatibility();
9356}
9357
9358static int __kvm_x86_vendor_init(struct kvm_x86_init_ops *ops)
9297{9359{
9298 struct kvm_x86_init_ops *ops = opaque;
9299 u64 host_pat;9360 u64 host_pat;
9300 int r;9361 int r, cpu;
9301 9362
9302 if (kvm_x86_ops.hardware_enable) {9363 if (kvm_x86_ops.hardware_enable) {
9303 pr_err("kvm: already loaded vendor module '%s'\n", kvm_x86_ops.name);9364 pr_err("already loaded vendor module '%s'\n", kvm_x86_ops.name);
9304 return -EEXIST;9365 return -EEXIST;
9305 }9366 }
9306 9367
9307 if (!ops->cpu_has_kvm_support()) {
9308 pr_err_ratelimited("kvm: no hardware support for '%s'\n",
9309 ops->runtime_ops->name);
9310 return -EOPNOTSUPP;
9311 }
9312 if (ops->disabled_by_bios()) {
9313 pr_err_ratelimited("kvm: support for '%s' disabled by bios\n",
9314 ops->runtime_ops->name);
9315 return -EOPNOTSUPP;
9316 }
9317
9318 /*9368 /*
9319 * KVM explicitly assumes that the guest has an FPU and9369 * KVM explicitly assumes that the guest has an FPU and
9320 * FXSAVE/FXRSTOR. For example, the KVM_GET_FPU explicitly casts the9370 * FXSAVE/FXRSTOR. For example, the KVM_GET_FPU explicitly casts the
9321 * vCPU's FPU state as a fxregs_state struct.9371 * vCPU's FPU state as a fxregs_state struct.
9322 */9372 */
9323 if (!boot_cpu_has(X86_FEATURE_FPU) || !boot_cpu_has(X86_FEATURE_FXSR)) {9373 if (!boot_cpu_has(X86_FEATURE_FPU) || !boot_cpu_has(X86_FEATURE_FXSR)) {
9324 printk(KERN_ERR "kvm: inadequate fpu\n");9374 pr_err("inadequate fpu\n");
9325 return -EOPNOTSUPP;9375 return -EOPNOTSUPP;
9326 }9376 }
9327 9377
@@ -9339,19 +9389,19 @@ int kvm_arch_init(void *opaque) · 11 unmodified lines
9339 */9389 */
9340 if (rdmsrl_safe(MSR_IA32_CR_PAT, &host_pat) ||9390 if (rdmsrl_safe(MSR_IA32_CR_PAT, &host_pat) ||
9341 (host_pat & GENMASK(2, 0)) != 6) {9391 (host_pat & GENMASK(2, 0)) != 6) {
9342 pr_err("kvm: host PAT[0] is not WB\n");9392 pr_err("host PAT[0] is not WB\n");
9343 return -EIO;9393 return -EIO;
9344 }9394 }
9345 9395
9346 x86_emulator_cache = kvm_alloc_emulator_cache();9396 x86_emulator_cache = kvm_alloc_emulator_cache();
9347 if (!x86_emulator_cache) {9397 if (!x86_emulator_cache) {
9348 pr_err("kvm: failed to allocate cache for x86 emulator\n");9398 pr_err("failed to allocate cache for x86 emulator\n");
9349 return -ENOMEM;9399 return -ENOMEM;
9350 }9400 }
9351 9401
9352 user_return_msrs = alloc_percpu(struct kvm_user_return_msrs);9402 user_return_msrs = alloc_percpu(struct kvm_user_return_msrs);
9353 if (!user_return_msrs) {9403 if (!user_return_msrs) {
9354 printk(KERN_ERR "kvm: failed to allocate percpu kvm_user_return_msrs\n");9404 pr_err("failed to allocate percpu kvm_user_return_msrs\n");
9355 r = -ENOMEM;9405 r = -ENOMEM;
9356 goto out_free_x86_emulator_cache;9406 goto out_free_x86_emulator_cache;
9357 }9407 }
@@ -9361,13 +9411,37 @@ int kvm_arch_init(void *opaque) · 3 unmodified lines
9361 if (r)9411 if (r)
9362 goto out_free_percpu;9412 goto out_free_percpu;
9363 9413
9364 kvm_timer_init();
9365
9366 if (boot_cpu_has(X86_FEATURE_XSAVE)) {9414 if (boot_cpu_has(X86_FEATURE_XSAVE)) {
9367 host_xcr0 = xgetbv(XCR_XFEATURE_ENABLED_MASK);9415 host_xcr0 = xgetbv(XCR_XFEATURE_ENABLED_MASK);
9368 kvm_caps.supported_xcr0 = host_xcr0 & KVM_SUPPORTED_XCR0;9416 kvm_caps.supported_xcr0 = host_xcr0 & KVM_SUPPORTED_XCR0;
9369 }9417 }
9370 9418
9419 rdmsrl_safe(MSR_EFER, &host_efer);
9420
9421 if (boot_cpu_has(X86_FEATURE_XSAVES))
9422 rdmsrl(MSR_IA32_XSS, host_xss);
9423
9424 kvm_init_pmu_capability(ops->pmu_ops);
9425
9426 r = ops->hardware_setup();
9427 if (r != 0)
9428 goto out_mmu_exit;
9429
9430 kvm_ops_update(ops);
9431
9432 for_each_online_cpu(cpu) {
9433 smp_call_function_single(cpu, kvm_x86_check_cpu_compat, &r, 1);
9434 if (r < 0)
9435 goto out_unwind_ops;
9436 }
9437
9438 /*
9439 * Point of no return! DO NOT add error paths below this point unless
9440 * absolutely necessary, as most operations from this point forward
9441 * require unwinding.
9442 */
9443 kvm_timer_init();
9444
9371 if (pi_inject_timer == -1)9445 if (pi_inject_timer == -1)
9372 pi_inject_timer = housekeeping_enabled(HK_TYPE_TIMER);9446 pi_inject_timer = housekeeping_enabled(HK_TYPE_TIMER);
9373#ifdef CONFIG_X86_649447#ifdef CONFIG_X86_64
@@ -9377,8 +9451,35 @@ int kvm_arch_init(void *opaque) · 3 unmodified lines
9377 set_hv_tscchange_cb(kvm_hyperv_tsc_notifier);9451 set_hv_tscchange_cb(kvm_hyperv_tsc_notifier);
9378#endif9452#endif
9379 9453
9454 kvm_register_perf_callbacks(ops->handle_intel_pt_intr);
9455
9456 if (!kvm_cpu_cap_has(X86_FEATURE_XSAVES))
9457 kvm_caps.supported_xss = 0;
9458
9459#define __kvm_cpu_cap_has(UNUSED_, f) kvm_cpu_cap_has(f)
9460 cr4_reserved_bits = __cr4_reserved_bits(__kvm_cpu_cap_has, UNUSED_);
9461#undef __kvm_cpu_cap_has
9462
9463 if (kvm_caps.has_tsc_control) {
9464 /*
9465 * Make sure the user can only configure tsc_khz values that
9466 * fit into a signed integer.
9467 * A min value is not calculated because it will always
9468 * be 1 on all machines.
9469 */
9470 u64 max = min(0x7fffffffULL,
9471 __scale_tsc(kvm_caps.max_tsc_scaling_ratio, tsc_khz));
9472 kvm_caps.max_guest_tsc_khz = max;
9473 }
9474 kvm_caps.default_tsc_scaling_ratio = 1ULL << kvm_caps.tsc_scaling_ratio_frac_bits;
9475 kvm_init_msr_list();
9380 return 0;9476 return 0;
9381 9477
9478out_unwind_ops:
9479 kvm_x86_ops.hardware_enable = NULL;
9480 static_call(kvm_x86_hardware_unsetup)();
9481out_mmu_exit:
9482 kvm_mmu_vendor_module_exit();
9382out_free_percpu:9483out_free_percpu:
9383 free_percpu(user_return_msrs);9484 free_percpu(user_return_msrs);
9384out_free_x86_emulator_cache:9485out_free_x86_emulator_cache:
@@ -9386,8 +9487,22 @@ out_free_x86_emulator_cache: · 1 unmodified line
9386 return r;9487 return r;
9387}9488}
9388 9489
9389void kvm_arch_exit(void)9490int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops)
9491{
9492 int r;
9493
9494 mutex_lock(&vendor_module_lock);
9495 r = __kvm_x86_vendor_init(ops);
9496 mutex_unlock(&vendor_module_lock);
9497
9498 return r;
9499}
9500EXPORT_SYMBOL_GPL(kvm_x86_vendor_init);
9501
9502void kvm_x86_vendor_exit(void)
9390{9503{
9504 kvm_unregister_perf_callbacks();
9505
9391#ifdef CONFIG_X86_649506#ifdef CONFIG_X86_64
9392 if (hypervisor_is_type(X86_HYPER_MS_HYPERV))9507 if (hypervisor_is_type(X86_HYPER_MS_HYPERV))
9393 clear_hv_tscchange_cb();9508 clear_hv_tscchange_cb();
@@ -9404,7 +9519,7 @@ void kvm_arch_exit(void) · 10 unmodified lines
9404 irq_work_sync(&pvclock_irq_work);9519 irq_work_sync(&pvclock_irq_work);
9405 cancel_work_sync(&pvclock_gtod_work);9520 cancel_work_sync(&pvclock_gtod_work);
9406#endif9521#endif
9407 kvm_x86_ops.hardware_enable = NULL;9522 static_call(kvm_x86_hardware_unsetup)();
9408 kvm_mmu_vendor_module_exit();9523 kvm_mmu_vendor_module_exit();
9409 free_percpu(user_return_msrs);9524 free_percpu(user_return_msrs);
9410 kmem_cache_destroy(x86_emulator_cache);9525 kmem_cache_destroy(x86_emulator_cache);
@@ -9412,7 +9527,11 @@ void kvm_arch_exit(void) · 1 unmodified line
9412 static_key_deferred_flush(&kvm_xen_enabled);9527 static_key_deferred_flush(&kvm_xen_enabled);
9413 WARN_ON(static_branch_unlikely(&kvm_xen_enabled.key));9528 WARN_ON(static_branch_unlikely(&kvm_xen_enabled.key));
9414#endif9529#endif
9530 mutex_lock(&vendor_module_lock);
9531 kvm_x86_ops.hardware_enable = NULL;
9532 mutex_unlock(&vendor_module_lock);
9415}9533}
9534EXPORT_SYMBOL_GPL(kvm_x86_vendor_exit);
9416 9535
9417static int __kvm_emulate_halt(struct kvm_vcpu *vcpu, int state, int reason)9536static int __kvm_emulate_halt(struct kvm_vcpu *vcpu, int state, int reason)
9418{9537{
@@ -10065,7 +10184,7 @@ void kvm_make_scan_ioapic_request(struct kvm *kvm) · 646 unmodified lines
10065 kvm_make_all_cpus_request(kvm, KVM_REQ_SCAN_IOAPIC);10184 kvm_make_all_cpus_request(kvm, KVM_REQ_SCAN_IOAPIC);
10066}10185}
10067 10186
10068void kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu)10187void __kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu)
10069{10188{
10070 struct kvm_lapic *apic = vcpu->arch.apic;10189 struct kvm_lapic *apic = vcpu->arch.apic;
10071 bool activate;10190 bool activate;
@@ -10100,7 +10219,30 @@ out: · 28 unmodified lines
10100 preempt_enable();10219 preempt_enable();
10101 up_read(&vcpu->kvm->arch.apicv_update_lock);10220 up_read(&vcpu->kvm->arch.apicv_update_lock);
10102}10221}
10103EXPORT_SYMBOL_GPL(kvm_vcpu_update_apicv);10222EXPORT_SYMBOL_GPL(__kvm_vcpu_update_apicv);
10223
10224static void kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu)
10225{
10226 if (!lapic_in_kernel(vcpu))
10227 return;
10228
10229 /*
10230 * Due to sharing page tables across vCPUs, the xAPIC memslot must be
10231 * deleted if any vCPU has xAPIC virtualization and x2APIC enabled, but
10232 * and hardware doesn't support x2APIC virtualization. E.g. some AMD
10233 * CPUs support AVIC but not x2APIC. KVM still allows enabling AVIC in
10234 * this case so that KVM can the AVIC doorbell to inject interrupts to
10235 * running vCPUs, but KVM must not create SPTEs for the APIC base as
10236 * the vCPU would incorrectly be able to access the vAPIC page via MMIO
10237 * despite being in x2APIC mode. For simplicity, inhibiting the APIC
10238 * access page is sticky.
10239 */
10240 if (apic_x2apic_mode(vcpu->arch.apic) &&
10241 kvm_x86_ops.allow_apicv_in_x2apic_without_x2apic_virtualization)
10242 kvm_inhibit_apic_access_page(vcpu);
10243
10244 __kvm_vcpu_update_apicv(vcpu);
10245}
10104 10246
10105void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm,10247void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm,
10106 enum kvm_apicv_inhibit reason, bool set)10248 enum kvm_apicv_inhibit reason, bool set)
@@ -10109,7 +10251,7 @@ void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm, · 2 unmodified lines
10109 10251
10110 lockdep_assert_held_write(&kvm->arch.apicv_update_lock);10252 lockdep_assert_held_write(&kvm->arch.apicv_update_lock);
10111 10253
10112 if (!static_call(kvm_x86_check_apicv_inhibit_reasons)(reason))10254 if (!(kvm_x86_ops.required_apicv_inhibits & BIT(reason)))
10113 return;10255 return;
10114 10256
10115 old = new = kvm->arch.apicv_inhibit_reasons;10257 old = new = kvm->arch.apicv_inhibit_reasons;
@@ -11553,7 +11695,7 @@ static int sync_regs(struct kvm_vcpu *vcpu) · 1437 unmodified lines
11553int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)11695int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)
11554{11696{
11555 if (kvm_check_tsc_unstable() && kvm->created_vcpus)11697 if (kvm_check_tsc_unstable() && kvm->created_vcpus)
11556 pr_warn_once("kvm: SMP vm created on host with unstable TSC; "11698 pr_warn_once("SMP vm created on host with unstable TSC; "
11557 "guest TSC will not be reliable\n");11699 "guest TSC will not be reliable\n");
11558 11700
11559 if (!kvm->arch.max_vcpu_ids)11701 if (!kvm->arch.max_vcpu_ids)
@@ -11630,7 +11772,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) · 70 unmodified lines
11630 goto free_wbinvd_dirty_mask;11772 goto free_wbinvd_dirty_mask;
11631 11773
11632 if (!fpu_alloc_guest_fpstate(&vcpu->arch.guest_fpu)) {11774 if (!fpu_alloc_guest_fpstate(&vcpu->arch.guest_fpu)) {
11633 pr_err("kvm: failed to allocate vcpu's fpu\n");11775 pr_err("failed to allocate vcpu's fpu\n");
11634 goto free_emulate_ctxt;11776 goto free_emulate_ctxt;
11635 }11777 }
11636 11778
@@ -11904,6 +12046,11 @@ int kvm_arch_hardware_enable(void) · 267 unmodified lines
11904 bool stable, backwards_tsc = false;12046 bool stable, backwards_tsc = false;
11905 12047
11906 kvm_user_return_msr_cpu_online();12048 kvm_user_return_msr_cpu_online();
12049
12050 ret = kvm_x86_check_processor_compatibility();
12051 if (ret)
12052 return ret;
12053
11907 ret = static_call(kvm_x86_hardware_enable)();12054 ret = static_call(kvm_x86_hardware_enable)();
11908 if (ret != 0)12055 if (ret != 0)
11909 return ret;12056 return ret;
@@ -11990,88 +12137,6 @@ void kvm_arch_hardware_disable(void) · 80 unmodified lines
11990 drop_user_return_notifiers();12137 drop_user_return_notifiers();
11991}12138}
11992 12139
11993static inline void kvm_ops_update(struct kvm_x86_init_ops *ops)
11994{
11995 memcpy(&kvm_x86_ops, ops->runtime_ops, sizeof(kvm_x86_ops));
11996
11997#define __KVM_X86_OP(func) \
11998 static_call_update(kvm_x86_##func, kvm_x86_ops.func);
11999#define KVM_X86_OP(func) \
12000 WARN_ON(!kvm_x86_ops.func); __KVM_X86_OP(func)
12001#define KVM_X86_OP_OPTIONAL __KVM_X86_OP
12002#define KVM_X86_OP_OPTIONAL_RET0(func) \
12003 static_call_update(kvm_x86_##func, (void *)kvm_x86_ops.func ? : \
12004 (void *)__static_call_return0);
12005#include <asm/kvm-x86-ops.h>
12006#undef __KVM_X86_OP
12007
12008 kvm_pmu_ops_update(ops->pmu_ops);
12009}
12010
12011int kvm_arch_hardware_setup(void *opaque)
12012{
12013 struct kvm_x86_init_ops *ops = opaque;
12014 int r;
12015
12016 rdmsrl_safe(MSR_EFER, &host_efer);
12017
12018 if (boot_cpu_has(X86_FEATURE_XSAVES))
12019 rdmsrl(MSR_IA32_XSS, host_xss);
12020
12021 kvm_init_pmu_capability();
12022
12023 r = ops->hardware_setup();
12024 if (r != 0)
12025 return r;
12026
12027 kvm_ops_update(ops);
12028
12029 kvm_register_perf_callbacks(ops->handle_intel_pt_intr);
12030
12031 if (!kvm_cpu_cap_has(X86_FEATURE_XSAVES))
12032 kvm_caps.supported_xss = 0;
12033
12034#define __kvm_cpu_cap_has(UNUSED_, f) kvm_cpu_cap_has(f)
12035 cr4_reserved_bits = __cr4_reserved_bits(__kvm_cpu_cap_has, UNUSED_);
12036#undef __kvm_cpu_cap_has
12037
12038 if (kvm_caps.has_tsc_control) {
12039 /*
12040 * Make sure the user can only configure tsc_khz values that
12041 * fit into a signed integer.
12042 * A min value is not calculated because it will always
12043 * be 1 on all machines.
12044 */
12045 u64 max = min(0x7fffffffULL,
12046 __scale_tsc(kvm_caps.max_tsc_scaling_ratio, tsc_khz));
12047 kvm_caps.max_guest_tsc_khz = max;
12048 }
12049 kvm_caps.default_tsc_scaling_ratio = 1ULL << kvm_caps.tsc_scaling_ratio_frac_bits;
12050 kvm_init_msr_list();
12051 return 0;
12052}
12053
12054void kvm_arch_hardware_unsetup(void)
12055{
12056 kvm_unregister_perf_callbacks();
12057
12058 static_call(kvm_x86_hardware_unsetup)();
12059}
12060
12061int kvm_arch_check_processor_compat(void *opaque)
12062{
12063 struct cpuinfo_x86 *c = &cpu_data(smp_processor_id());
12064 struct kvm_x86_init_ops *ops = opaque;
12065
12066 WARN_ON(!irqs_disabled());
12067
12068 if (__cr4_reserved_bits(cpu_has, c) !=
12069 __cr4_reserved_bits(cpu_has, &boot_cpu_data))
12070 return -EIO;
12071
12072 return ops->check_processor_compatibility();
12073}
12074
12075bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu)12140bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu)
12076{12141{
12077 return vcpu->kvm->arch.bsp_vcpu_id == vcpu->vcpu_id;12142 return vcpu->kvm->arch.bsp_vcpu_id == vcpu->vcpu_id;
@@ -12244,7 +12309,7 @@ void __user * __x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, · 166 unmodified lines
12244 */12309 */
12245 hva = vm_mmap(NULL, 0, size, PROT_READ | PROT_WRITE,12310 hva = vm_mmap(NULL, 0, size, PROT_READ | PROT_WRITE,
12246 MAP_SHARED | MAP_ANONYMOUS, 0);12311 MAP_SHARED | MAP_ANONYMOUS, 0);
12247 if (IS_ERR((void *)hva))12312 if (IS_ERR_VALUE(hva))
12248 return (void __user *)hva;12313 return (void __user *)hva;
12249 } else {12314 } else {
12250 if (!slot || !slot->npages)12315 if (!slot || !slot->npages)
@@ -12459,16 +12524,14 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, · 208 unmodified lines
12459 12524
12460static void kvm_mmu_update_cpu_dirty_logging(struct kvm *kvm, bool enable)12525static void kvm_mmu_update_cpu_dirty_logging(struct kvm *kvm, bool enable)
12461{12526{
12462 struct kvm_arch *ka = &kvm->arch;12527 int nr_slots;
12463 12528
12464 if (!kvm_x86_ops.cpu_dirty_log_size)12529 if (!kvm_x86_ops.cpu_dirty_log_size)
12465 return;12530 return;
12466 12531
12467 if ((enable && ++ka->cpu_dirty_logging_count == 1) ||12532 nr_slots = atomic_read(&kvm->nr_memslots_dirty_logging);
12468 (!enable && --ka->cpu_dirty_logging_count == 0))12533 if ((enable && nr_slots == 1) || !nr_slots)
12469 kvm_make_all_cpus_request(kvm, KVM_REQ_UPDATE_CPU_DIRTY_LOGGING);12534 kvm_make_all_cpus_request(kvm, KVM_REQ_UPDATE_CPU_DIRTY_LOGGING);
12470
12471 WARN_ON_ONCE(ka->cpu_dirty_logging_count < 0);
12472}12535}
12473 12536
12474static void kvm_mmu_slot_apply_flags(struct kvm *kvm,12537static void kvm_mmu_slot_apply_flags(struct kvm *kvm,
arch/x86/kvm/x86.h+15−3
@@ -331,6 +331,18 @@ extern bool report_ignored_msrs; · 330 unmodified lines
331 331
332extern bool eager_page_split;332extern bool eager_page_split;
333 333
334static inline void kvm_pr_unimpl_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64 data)
335{
336 if (report_ignored_msrs)
337 vcpu_unimpl(vcpu, "Unhandled WRMSR(0x%x) = 0x%llx\n", msr, data);
338}
339
340static inline void kvm_pr_unimpl_rdmsr(struct kvm_vcpu *vcpu, u32 msr)
341{
342 if (report_ignored_msrs)
343 vcpu_unimpl(vcpu, "Unhandled RDMSR(0x%x)\n", msr);
344}
345
334static inline u64 nsec_to_cycles(struct kvm_vcpu *vcpu, u64 nsec)346static inline u64 nsec_to_cycles(struct kvm_vcpu *vcpu, u64 nsec)
335{347{
336 return pvclock_scale_delta(nsec, vcpu->arch.virtual_tsc_mult,348 return pvclock_scale_delta(nsec, vcpu->arch.virtual_tsc_mult,
@@ -382,13 +394,13 @@ enum kvm_intr_type { · 45 unmodified lines
382 KVM_HANDLING_NMI,394 KVM_HANDLING_NMI,
383};395};
384 396
385static inline void kvm_before_interrupt(struct kvm_vcpu *vcpu,397static __always_inline void kvm_before_interrupt(struct kvm_vcpu *vcpu,
386 enum kvm_intr_type intr)398 enum kvm_intr_type intr)
387{399{
388 WRITE_ONCE(vcpu->arch.handling_intr_from_guest, (u8)intr);400 WRITE_ONCE(vcpu->arch.handling_intr_from_guest, (u8)intr);
389}401}
390 402
391static inline void kvm_after_interrupt(struct kvm_vcpu *vcpu)403static __always_inline void kvm_after_interrupt(struct kvm_vcpu *vcpu)
392{404{
393 WRITE_ONCE(vcpu->arch.handling_intr_from_guest, 0);405 WRITE_ONCE(vcpu->arch.handling_intr_from_guest, 0);
394}406}
arch/x86/kvm/xen.c+27−0
@@ -5,6 +5,7 @@ · 4 unmodified lines
5 *5 *
6 * KVM Xen emulation6 * KVM Xen emulation
7 */7 */
8#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
8 9
9#include "x86.h"10#include "x86.h"
10#include "xen.h"11#include "xen.h"
@@ -22,6 +23,9 @@ · 11 unmodified lines
22#include <xen/interface/event_channel.h>23#include <xen/interface/event_channel.h>
23#include <xen/interface/sched.h>24#include <xen/interface/sched.h>
24 25
26#include <asm/xen/cpuid.h>
27
28#include "cpuid.h"
25#include "trace.h"29#include "trace.h"
26 30
27static int kvm_xen_set_evtchn(struct kvm_xen_evtchn *xe, struct kvm *kvm);31static int kvm_xen_set_evtchn(struct kvm_xen_evtchn *xe, struct kvm *kvm);
@@ -2076,6 +2080,29 @@ void kvm_xen_destroy_vcpu(struct kvm_vcpu *vcpu) · 2048 unmodified lines
2076 del_timer_sync(&vcpu->arch.xen.poll_timer);2080 del_timer_sync(&vcpu->arch.xen.poll_timer);
2077}2081}
2078 2082
2083void kvm_xen_update_tsc_info(struct kvm_vcpu *vcpu)
2084{
2085 struct kvm_cpuid_entry2 *entry;
2086 u32 function;
2087
2088 if (!vcpu->arch.xen.cpuid.base)
2089 return;
2090
2091 function = vcpu->arch.xen.cpuid.base | XEN_CPUID_LEAF(3);
2092 if (function > vcpu->arch.xen.cpuid.limit)
2093 return;
2094
2095 entry = kvm_find_cpuid_entry_index(vcpu, function, 1);
2096 if (entry) {
2097 entry->ecx = vcpu->arch.hv_clock.tsc_to_system_mul;
2098 entry->edx = vcpu->arch.hv_clock.tsc_shift;
2099 }
2100
2101 entry = kvm_find_cpuid_entry_index(vcpu, function, 2);
2102 if (entry)
2103 entry->eax = vcpu->arch.hw_tsc_khz;
2104}
2105
2079void kvm_xen_init_vm(struct kvm *kvm)2106void kvm_xen_init_vm(struct kvm *kvm)
2080{2107{
2081 mutex_init(&kvm->arch.xen.xen_lock);2108 mutex_init(&kvm->arch.xen.xen_lock);
arch/x86/kvm/xen.h+7−0
@@ -9,6 +9,8 @@ · 8 unmodified lines
9#ifndef __ARCH_X86_KVM_XEN_H__9#ifndef __ARCH_X86_KVM_XEN_H__
10#define __ARCH_X86_KVM_XEN_H__10#define __ARCH_X86_KVM_XEN_H__
11 11
12#include <asm/xen/hypervisor.h>
13
12#ifdef CONFIG_KVM_XEN14#ifdef CONFIG_KVM_XEN
13#include <linux/jump_label_ratelimit.h>15#include <linux/jump_label_ratelimit.h>
14 16
@@ -32,6 +34,7 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe, · 17 unmodified lines
32int kvm_xen_setup_evtchn(struct kvm *kvm,34int kvm_xen_setup_evtchn(struct kvm *kvm,
33 struct kvm_kernel_irq_routing_entry *e,35 struct kvm_kernel_irq_routing_entry *e,
34 const struct kvm_irq_routing_entry *ue);36 const struct kvm_irq_routing_entry *ue);
37void kvm_xen_update_tsc_info(struct kvm_vcpu *vcpu);
35 38
36static inline bool kvm_xen_msr_enabled(struct kvm *kvm)39static inline bool kvm_xen_msr_enabled(struct kvm *kvm)
37{40{
@@ -135,6 +138,10 @@ static inline bool kvm_xen_timer_enabled(struct kvm_vcpu *vcpu) · 97 unmodified lines
135{138{
136 return false;139 return false;
137}140}
141
142static inline void kvm_xen_update_tsc_info(struct kvm_vcpu *vcpu)
143{
144}
138#endif145#endif
139 146
140int kvm_xen_hypercall(struct kvm_vcpu *vcpu);147int kvm_xen_hypercall(struct kvm_vcpu *vcpu);
drivers/irqchip/irq-apple-aic.c+40−13
@@ -210,7 +210,6 @@ · 209 unmodified lines
210 FIELD_PREP(AIC_EVENT_NUM, x))210 FIELD_PREP(AIC_EVENT_NUM, x))
211#define AIC_HWIRQ_IRQ(x) FIELD_GET(AIC_EVENT_NUM, x)211#define AIC_HWIRQ_IRQ(x) FIELD_GET(AIC_EVENT_NUM, x)
212#define AIC_HWIRQ_DIE(x) FIELD_GET(AIC_EVENT_DIE, x)212#define AIC_HWIRQ_DIE(x) FIELD_GET(AIC_EVENT_DIE, x)
213#define AIC_NR_FIQ 6
214#define AIC_NR_SWIPI 32213#define AIC_NR_SWIPI 32
215 214
216/*215/*
@@ -222,11 +221,18 @@ · 5 unmodified lines
222 * running at EL2 (with VHE). When the kernel is running at EL1, the221 * running at EL2 (with VHE). When the kernel is running at EL1, the
223 * mapping differs and aic_irq_domain_translate() performs the remapping.222 * mapping differs and aic_irq_domain_translate() performs the remapping.
224 */223 */
225 224enum fiq_hwirq {
226#define AIC_TMR_EL0_PHYS AIC_TMR_HV_PHYS225 /* Must be ordered as in apple-aic.h */
227#define AIC_TMR_EL0_VIRT AIC_TMR_HV_VIRT226 AIC_TMR_EL0_PHYS = AIC_TMR_HV_PHYS,
228#define AIC_TMR_EL02_PHYS AIC_TMR_GUEST_PHYS227 AIC_TMR_EL0_VIRT = AIC_TMR_HV_VIRT,
229#define AIC_TMR_EL02_VIRT AIC_TMR_GUEST_VIRT228 AIC_TMR_EL02_PHYS = AIC_TMR_GUEST_PHYS,
229 AIC_TMR_EL02_VIRT = AIC_TMR_GUEST_VIRT,
230 AIC_CPU_PMU_Effi = AIC_CPU_PMU_E,
231 AIC_CPU_PMU_Perf = AIC_CPU_PMU_P,
232 /* No need for this to be discovered from DT */
233 AIC_VGIC_MI,
234 AIC_NR_FIQ
235};
230 236
231static DEFINE_STATIC_KEY_TRUE(use_fast_ipi);237static DEFINE_STATIC_KEY_TRUE(use_fast_ipi);
232 238
@@ -380,14 +386,20 @@ static void __exception_irq_entry aic_handle_irq(struct pt_regs *regs) · 147 unmodified lines
380 386
381 /*387 /*
382 * vGIC maintenance interrupts end up here too, so we need to check388 * vGIC maintenance interrupts end up here too, so we need to check
383 * for them separately. This should never trigger if KVM is working389 * for them separately. It should however only trigger when NV is
384 * properly, because it will have already taken care of clearing it390 * in use, and be cleared when coming back from the handler.
385 * on guest exit before this handler runs.
386 */391 */
387 if (is_kernel_in_hyp_mode() && (read_sysreg_s(SYS_ICH_HCR_EL2) & ICH_HCR_EN) &&392 if (is_kernel_in_hyp_mode() &&
388 read_sysreg_s(SYS_ICH_MISR_EL2) != 0) {393 (read_sysreg_s(SYS_ICH_HCR_EL2) & ICH_HCR_EN) &&
389 pr_err_ratelimited("vGIC IRQ fired and not handled by KVM, disabling.\n");394 read_sysreg_s(SYS_ICH_MISR_EL2) != 0) {
390 sysreg_clear_set_s(SYS_ICH_HCR_EL2, ICH_HCR_EN, 0);395 generic_handle_domain_irq(aic_irqc->hw_domain,
396 AIC_FIQ_HWIRQ(AIC_VGIC_MI));
397
398 if (unlikely((read_sysreg_s(SYS_ICH_HCR_EL2) & ICH_HCR_EN) &&
399 read_sysreg_s(SYS_ICH_MISR_EL2))) {
400 pr_err_ratelimited("vGIC IRQ fired and not handled by KVM, disabling.\n");
401 sysreg_clear_set_s(SYS_ICH_HCR_EL2, ICH_HCR_EN, 0);
402 }
391 }403 }
392}404}
393 405
@@ -1033,6 +1045,21 @@ static int __init aic_of_ic_init(struct device_node *node, struct device_node *p · 639 unmodified lines
1033 "irqchip/apple-aic/ipi:starting",1045 "irqchip/apple-aic/ipi:starting",
1034 aic_init_cpu, NULL);1046 aic_init_cpu, NULL);
1035 1047
1048 if (is_kernel_in_hyp_mode()) {
1049 struct irq_fwspec mi = {
1050 .fwnode = of_node_to_fwnode(node),
1051 .param_count = 3,
1052 .param = {
1053 [0] = AIC_FIQ, /* This is a lie */
1054 [1] = AIC_VGIC_MI,
1055 [2] = IRQ_TYPE_LEVEL_HIGH,
1056 },
1057 };
1058
1059 vgic_info.maint_irq = irq_create_fwspec_mapping(&mi);
1060 WARN_ON(!vgic_info.maint_irq);
1061 }
1062
1036 vgic_set_kvm_info(&vgic_info);1063 vgic_set_kvm_info(&vgic_info);
1037 1064
1038 pr_info("Initialized with %d/%d IRQs * %d/%d die(s), %d FIQs, %d vIPIs",1065 pr_info("Initialized with %d/%d IRQs * %d/%d die(s), %d FIQs, %d vIPIs",
drivers/perf/riscv_pmu_sbi.c+57−7
@@ -44,7 +44,7 @@ static const struct attribute_group *riscv_pmu_attr_groups[] = { · 43 unmodified lines
44};44};
45 45
46/*46/*
47 * RISC-V doesn't have hetergenous harts yet. This need to be part of47 * RISC-V doesn't have heterogeneous harts yet. This need to be part of
48 * per_cpu in case of harts with different pmu counters48 * per_cpu in case of harts with different pmu counters
49 */49 */
50static union sbi_pmu_ctr_info *pmu_ctr_list;50static union sbi_pmu_ctr_info *pmu_ctr_list;
@@ -52,6 +52,9 @@ static bool riscv_pmu_use_irq; · 1 unmodified line
52static unsigned int riscv_pmu_irq_num;52static unsigned int riscv_pmu_irq_num;
53static unsigned int riscv_pmu_irq;53static unsigned int riscv_pmu_irq;
54 54
55/* Cache the available counters in a bitmask */
56static unsigned long cmask;
57
55struct sbi_pmu_event_data {58struct sbi_pmu_event_data {
56 union {59 union {
57 union {60 union {
@@ -267,6 +270,58 @@ static bool pmu_sbi_ctr_is_fw(int cidx) · 209 unmodified lines
267 return (info->type == SBI_PMU_CTR_TYPE_FW) ? true : false;270 return (info->type == SBI_PMU_CTR_TYPE_FW) ? true : false;
268}271}
269 272
273/*
274 * Returns the counter width of a programmable counter and number of hardware
275 * counters. As we don't support heterogeneous CPUs yet, it is okay to just
276 * return the counter width of the first programmable counter.
277 */
278int riscv_pmu_get_hpm_info(u32 *hw_ctr_width, u32 *num_hw_ctr)
279{
280 int i;
281 union sbi_pmu_ctr_info *info;
282 u32 hpm_width = 0, hpm_count = 0;
283
284 if (!cmask)
285 return -EINVAL;
286
287 for_each_set_bit(i, &cmask, RISCV_MAX_COUNTERS) {
288 info = &pmu_ctr_list[i];
289 if (!info)
290 continue;
291 if (!hpm_width && info->csr != CSR_CYCLE && info->csr != CSR_INSTRET)
292 hpm_width = info->width;
293 if (info->type == SBI_PMU_CTR_TYPE_HW)
294 hpm_count++;
295 }
296
297 *hw_ctr_width = hpm_width;
298 *num_hw_ctr = hpm_count;
299
300 return 0;
301}
302EXPORT_SYMBOL_GPL(riscv_pmu_get_hpm_info);
303
304static unsigned long pmu_sbi_get_filter_flags(struct perf_event *event)
305{
306 unsigned long cflags = 0;
307 bool guest_events = false;
308
309 if (event->attr.config1 & RISCV_PMU_CONFIG1_GUEST_EVENTS)
310 guest_events = true;
311 if (event->attr.exclude_kernel)
312 cflags |= guest_events ? SBI_PMU_CFG_FLAG_SET_VSINH : SBI_PMU_CFG_FLAG_SET_SINH;
313 if (event->attr.exclude_user)
314 cflags |= guest_events ? SBI_PMU_CFG_FLAG_SET_VUINH : SBI_PMU_CFG_FLAG_SET_UINH;
315 if (guest_events && event->attr.exclude_hv)
316 cflags |= SBI_PMU_CFG_FLAG_SET_SINH;
317 if (event->attr.exclude_host)
318 cflags |= SBI_PMU_CFG_FLAG_SET_UINH | SBI_PMU_CFG_FLAG_SET_SINH;
319 if (event->attr.exclude_guest)
320 cflags |= SBI_PMU_CFG_FLAG_SET_VSINH | SBI_PMU_CFG_FLAG_SET_VUINH;
321
322 return cflags;
323}
324
270static int pmu_sbi_ctr_get_idx(struct perf_event *event)325static int pmu_sbi_ctr_get_idx(struct perf_event *event)
271{326{
272 struct hw_perf_event *hwc = &event->hw;327 struct hw_perf_event *hwc = &event->hw;
@@ -277,11 +332,7 @@ static int pmu_sbi_ctr_get_idx(struct perf_event *event) · 4 unmodified lines
277 uint64_t cbase = 0;332 uint64_t cbase = 0;
278 unsigned long cflags = 0;333 unsigned long cflags = 0;
279 334
280 if (event->attr.exclude_kernel)335 cflags = pmu_sbi_get_filter_flags(event);
281 cflags |= SBI_PMU_CFG_FLAG_SET_SINH;
282 if (event->attr.exclude_user)
283 cflags |= SBI_PMU_CFG_FLAG_SET_UINH;
284
285 /* retrieve the available counter index */336 /* retrieve the available counter index */
286#if defined(CONFIG_32BIT)337#if defined(CONFIG_32BIT)
287 ret = sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_CFG_MATCH, cbase,338 ret = sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_CFG_MATCH, cbase,
@@ -806,7 +857,6 @@ static void riscv_pmu_destroy(struct riscv_pmu *pmu) · 518 unmodified lines
806static int pmu_sbi_device_probe(struct platform_device *pdev)857static int pmu_sbi_device_probe(struct platform_device *pdev)
807{858{
808 struct riscv_pmu *pmu = NULL;859 struct riscv_pmu *pmu = NULL;
809 unsigned long cmask = 0;
810 int ret = -ENODEV;860 int ret = -ENODEV;
811 int num_counters;861 int num_counters;
812 862
drivers/s390/virtio/virtio_ccw.c+24−22
@@ -363,7 +363,7 @@ static void virtio_ccw_drop_indicator(struct virtio_ccw_device *vcdev, · 362 unmodified lines
363 thinint_area->isc = VIRTIO_AIRQ_ISC;363 thinint_area->isc = VIRTIO_AIRQ_ISC;
364 ccw->cmd_code = CCW_CMD_SET_IND_ADAPTER;364 ccw->cmd_code = CCW_CMD_SET_IND_ADAPTER;
365 ccw->count = sizeof(*thinint_area);365 ccw->count = sizeof(*thinint_area);
366 ccw->cda = (__u32)(unsigned long) thinint_area;366 ccw->cda = (__u32)virt_to_phys(thinint_area);
367 } else {367 } else {
368 /* payload is the address of the indicators */368 /* payload is the address of the indicators */
369 indicatorp = ccw_device_dma_zalloc(vcdev->cdev,369 indicatorp = ccw_device_dma_zalloc(vcdev->cdev,
@@ -373,7 +373,7 @@ static void virtio_ccw_drop_indicator(struct virtio_ccw_device *vcdev, · 3 unmodified lines
373 *indicatorp = 0;373 *indicatorp = 0;
374 ccw->cmd_code = CCW_CMD_SET_IND;374 ccw->cmd_code = CCW_CMD_SET_IND;
375 ccw->count = sizeof(indicators(vcdev));375 ccw->count = sizeof(indicators(vcdev));
376 ccw->cda = (__u32)(unsigned long) indicatorp;376 ccw->cda = (__u32)virt_to_phys(indicatorp);
377 }377 }
378 /* Deregister indicators from host. */378 /* Deregister indicators from host. */
379 *indicators(vcdev) = 0;379 *indicators(vcdev) = 0;
@@ -417,7 +417,7 @@ static int virtio_ccw_read_vq_conf(struct virtio_ccw_device *vcdev, · 37 unmodified lines
417 ccw->cmd_code = CCW_CMD_READ_VQ_CONF;417 ccw->cmd_code = CCW_CMD_READ_VQ_CONF;
418 ccw->flags = 0;418 ccw->flags = 0;
419 ccw->count = sizeof(struct vq_config_block);419 ccw->count = sizeof(struct vq_config_block);
420 ccw->cda = (__u32)(unsigned long)(&vcdev->dma_area->config_block);420 ccw->cda = (__u32)virt_to_phys(&vcdev->dma_area->config_block);
421 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_VQ_CONF);421 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_VQ_CONF);
422 if (ret)422 if (ret)
423 return ret;423 return ret;
@@ -454,7 +454,7 @@ static void virtio_ccw_del_vq(struct virtqueue *vq, struct ccw1 *ccw) · 30 unmodified lines
454 }454 }
455 ccw->cmd_code = CCW_CMD_SET_VQ;455 ccw->cmd_code = CCW_CMD_SET_VQ;
456 ccw->flags = 0;456 ccw->flags = 0;
457 ccw->cda = (__u32)(unsigned long)(info->info_block);457 ccw->cda = (__u32)virt_to_phys(info->info_block);
458 ret = ccw_io_helper(vcdev, ccw,458 ret = ccw_io_helper(vcdev, ccw,
459 VIRTIO_CCW_DOING_SET_VQ | index);459 VIRTIO_CCW_DOING_SET_VQ | index);
460 /*460 /*
@@ -556,7 +556,7 @@ static struct virtqueue *virtio_ccw_setup_vq(struct virtio_device *vdev, · 95 unmodified lines
556 }556 }
557 ccw->cmd_code = CCW_CMD_SET_VQ;557 ccw->cmd_code = CCW_CMD_SET_VQ;
558 ccw->flags = 0;558 ccw->flags = 0;
559 ccw->cda = (__u32)(unsigned long)(info->info_block);559 ccw->cda = (__u32)virt_to_phys(info->info_block);
560 err = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_SET_VQ | i);560 err = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_SET_VQ | i);
561 if (err) {561 if (err) {
562 dev_warn(&vcdev->cdev->dev, "SET_VQ failed\n");562 dev_warn(&vcdev->cdev->dev, "SET_VQ failed\n");
@@ -590,6 +590,7 @@ static int virtio_ccw_register_adapter_ind(struct virtio_ccw_device *vcdev, · 27 unmodified lines
590{590{
591 int ret;591 int ret;
592 struct virtio_thinint_area *thinint_area = NULL;592 struct virtio_thinint_area *thinint_area = NULL;
593 unsigned long indicator_addr;
593 struct airq_info *info;594 struct airq_info *info;
594 595
595 thinint_area = ccw_device_dma_zalloc(vcdev->cdev,596 thinint_area = ccw_device_dma_zalloc(vcdev->cdev,
@@ -599,21 +600,22 @@ static int virtio_ccw_register_adapter_ind(struct virtio_ccw_device *vcdev, · 3 unmodified lines
599 goto out;600 goto out;
600 }601 }
601 /* Try to get an indicator. */602 /* Try to get an indicator. */
602 thinint_area->indicator = get_airq_indicator(vqs, nvqs,603 indicator_addr = get_airq_indicator(vqs, nvqs,
603 &thinint_area->bit_nr,604 &thinint_area->bit_nr,
604 &vcdev->airq_info);605 &vcdev->airq_info);
605 if (!thinint_area->indicator) {606 if (!indicator_addr) {
606 ret = -ENOSPC;607 ret = -ENOSPC;
607 goto out;608 goto out;
608 }609 }
610 thinint_area->indicator = virt_to_phys((void *)indicator_addr);
609 info = vcdev->airq_info;611 info = vcdev->airq_info;
610 thinint_area->summary_indicator =612 thinint_area->summary_indicator =
611 (unsigned long) get_summary_indicator(info);613 virt_to_phys(get_summary_indicator(info));
612 thinint_area->isc = VIRTIO_AIRQ_ISC;614 thinint_area->isc = VIRTIO_AIRQ_ISC;
613 ccw->cmd_code = CCW_CMD_SET_IND_ADAPTER;615 ccw->cmd_code = CCW_CMD_SET_IND_ADAPTER;
614 ccw->flags = CCW_FLAG_SLI;616 ccw->flags = CCW_FLAG_SLI;
615 ccw->count = sizeof(*thinint_area);617 ccw->count = sizeof(*thinint_area);
616 ccw->cda = (__u32)(unsigned long)thinint_area;618 ccw->cda = (__u32)virt_to_phys(thinint_area);
617 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_SET_IND_ADAPTER);619 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_SET_IND_ADAPTER);
618 if (ret) {620 if (ret) {
619 if (ret == -EOPNOTSUPP) {621 if (ret == -EOPNOTSUPP) {
@@ -686,7 +688,7 @@ static int virtio_ccw_find_vqs(struct virtio_device *vdev, unsigned nvqs, · 66 unmodified lines
686 ccw->cmd_code = CCW_CMD_SET_IND;688 ccw->cmd_code = CCW_CMD_SET_IND;
687 ccw->flags = 0;689 ccw->flags = 0;
688 ccw->count = sizeof(indicators(vcdev));690 ccw->count = sizeof(indicators(vcdev));
689 ccw->cda = (__u32)(unsigned long) indicatorp;691 ccw->cda = (__u32)virt_to_phys(indicatorp);
690 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_SET_IND);692 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_SET_IND);
691 if (ret)693 if (ret)
692 goto out;694 goto out;
@@ -697,7 +699,7 @@ static int virtio_ccw_find_vqs(struct virtio_device *vdev, unsigned nvqs, · 4 unmodified lines
697 ccw->cmd_code = CCW_CMD_SET_CONF_IND;699 ccw->cmd_code = CCW_CMD_SET_CONF_IND;
698 ccw->flags = 0;700 ccw->flags = 0;
699 ccw->count = sizeof(indicators2(vcdev));701 ccw->count = sizeof(indicators2(vcdev));
700 ccw->cda = (__u32)(unsigned long) indicatorp;702 ccw->cda = (__u32)virt_to_phys(indicatorp);
701 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_SET_CONF_IND);703 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_SET_CONF_IND);
702 if (ret)704 if (ret)
703 goto out;705 goto out;
@@ -759,7 +761,7 @@ static u64 virtio_ccw_get_features(struct virtio_device *vdev) · 55 unmodified lines
759 ccw->cmd_code = CCW_CMD_READ_FEAT;761 ccw->cmd_code = CCW_CMD_READ_FEAT;
760 ccw->flags = 0;762 ccw->flags = 0;
761 ccw->count = sizeof(*features);763 ccw->count = sizeof(*features);
762 ccw->cda = (__u32)(unsigned long)features;764 ccw->cda = (__u32)virt_to_phys(features);
763 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_FEAT);765 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_FEAT);
764 if (ret) {766 if (ret) {
765 rc = 0;767 rc = 0;
@@ -776,7 +778,7 @@ static u64 virtio_ccw_get_features(struct virtio_device *vdev) · 10 unmodified lines
776 ccw->cmd_code = CCW_CMD_READ_FEAT;778 ccw->cmd_code = CCW_CMD_READ_FEAT;
777 ccw->flags = 0;779 ccw->flags = 0;
778 ccw->count = sizeof(*features);780 ccw->count = sizeof(*features);
779 ccw->cda = (__u32)(unsigned long)features;781 ccw->cda = (__u32)virt_to_phys(features);
780 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_FEAT);782 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_FEAT);
781 if (ret == 0)783 if (ret == 0)
782 rc |= (u64)le32_to_cpu(features->features) << 32;784 rc |= (u64)le32_to_cpu(features->features) << 32;
@@ -829,7 +831,7 @@ static int virtio_ccw_finalize_features(struct virtio_device *vdev) · 46 unmodified lines
829 ccw->cmd_code = CCW_CMD_WRITE_FEAT;831 ccw->cmd_code = CCW_CMD_WRITE_FEAT;
830 ccw->flags = 0;832 ccw->flags = 0;
831 ccw->count = sizeof(*features);833 ccw->count = sizeof(*features);
832 ccw->cda = (__u32)(unsigned long)features;834 ccw->cda = (__u32)virt_to_phys(features);
833 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_WRITE_FEAT);835 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_WRITE_FEAT);
834 if (ret)836 if (ret)
835 goto out_free;837 goto out_free;
@@ -843,7 +845,7 @@ static int virtio_ccw_finalize_features(struct virtio_device *vdev) · 7 unmodified lines
843 ccw->cmd_code = CCW_CMD_WRITE_FEAT;845 ccw->cmd_code = CCW_CMD_WRITE_FEAT;
844 ccw->flags = 0;846 ccw->flags = 0;
845 ccw->count = sizeof(*features);847 ccw->count = sizeof(*features);
846 ccw->cda = (__u32)(unsigned long)features;848 ccw->cda = (__u32)virt_to_phys(features);
847 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_WRITE_FEAT);849 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_WRITE_FEAT);
848 850
849out_free:851out_free:
@@ -875,7 +877,7 @@ static void virtio_ccw_get_config(struct virtio_device *vdev, · 25 unmodified lines
875 ccw->cmd_code = CCW_CMD_READ_CONF;877 ccw->cmd_code = CCW_CMD_READ_CONF;
876 ccw->flags = 0;878 ccw->flags = 0;
877 ccw->count = offset + len;879 ccw->count = offset + len;
878 ccw->cda = (__u32)(unsigned long)config_area;880 ccw->cda = (__u32)virt_to_phys(config_area);
879 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_CONFIG);881 ret = ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_CONFIG);
880 if (ret)882 if (ret)
881 goto out_free;883 goto out_free;
@@ -922,7 +924,7 @@ static void virtio_ccw_set_config(struct virtio_device *vdev, · 40 unmodified lines
922 ccw->cmd_code = CCW_CMD_WRITE_CONF;924 ccw->cmd_code = CCW_CMD_WRITE_CONF;
923 ccw->flags = 0;925 ccw->flags = 0;
924 ccw->count = offset + len;926 ccw->count = offset + len;
925 ccw->cda = (__u32)(unsigned long)config_area;927 ccw->cda = (__u32)virt_to_phys(config_area);
926 ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_WRITE_CONFIG);928 ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_WRITE_CONFIG);
927 929
928out_free:930out_free:
@@ -946,7 +948,7 @@ static u8 virtio_ccw_get_status(struct virtio_device *vdev) · 17 unmodified lines
946 ccw->cmd_code = CCW_CMD_READ_STATUS;948 ccw->cmd_code = CCW_CMD_READ_STATUS;
947 ccw->flags = 0;949 ccw->flags = 0;
948 ccw->count = sizeof(vcdev->dma_area->status);950 ccw->count = sizeof(vcdev->dma_area->status);
949 ccw->cda = (__u32)(unsigned long)&vcdev->dma_area->status;951 ccw->cda = (__u32)virt_to_phys(&vcdev->dma_area->status);
950 ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_STATUS);952 ccw_io_helper(vcdev, ccw, VIRTIO_CCW_DOING_READ_STATUS);
951/*953/*
952 * If the channel program failed (should only happen if the device954 * If the channel program failed (should only happen if the device
@@ -975,7 +977,7 @@ static void virtio_ccw_set_status(struct virtio_device *vdev, u8 status) · 22 unmodified lines
975 ccw->cmd_code = CCW_CMD_WRITE_STATUS;977 ccw->cmd_code = CCW_CMD_WRITE_STATUS;
976 ccw->flags = 0;978 ccw->flags = 0;
977 ccw->count = sizeof(status);979 ccw->count = sizeof(status);
978 ccw->cda = (__u32)(unsigned long)&vcdev->dma_area->status;980 ccw->cda = (__u32)virt_to_phys(&vcdev->dma_area->status);
979 /* We use ssch for setting the status which is a serializing981 /* We use ssch for setting the status which is a serializing
980 * instruction that guarantees the memory writes have982 * instruction that guarantees the memory writes have
981 * completed before ssch.983 * completed before ssch.
@@ -1274,7 +1276,7 @@ static int virtio_ccw_set_transport_rev(struct virtio_ccw_device *vcdev) · 292 unmodified lines
1274 ccw->cmd_code = CCW_CMD_SET_VIRTIO_REV;1276 ccw->cmd_code = CCW_CMD_SET_VIRTIO_REV;
1275 ccw->flags = 0;1277 ccw->flags = 0;
1276 ccw->count = sizeof(*rev);1278 ccw->count = sizeof(*rev);
1277 ccw->cda = (__u32)(unsigned long)rev;1279 ccw->cda = (__u32)virt_to_phys(rev);
1278 1280
1279 vcdev->revision = VIRTIO_CCW_REV_MAX;1281 vcdev->revision = VIRTIO_CCW_REV_MAX;
1280 do {1282 do {
include/kvm/arm_arch_timer.h+5−1
@@ -60,7 +60,7 @@ struct arch_timer_cpu { · 59 unmodified lines
60 bool enabled;60 bool enabled;
61};61};
62 62
63int kvm_timer_hyp_init(bool);63int __init kvm_timer_hyp_init(bool has_gic);
64int kvm_timer_enable(struct kvm_vcpu *vcpu);64int kvm_timer_enable(struct kvm_vcpu *vcpu);
65int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu);65int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu);
66void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu);66void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu);
@@ -104,4 +104,8 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu, · 37 unmodified lines
104u32 timer_get_ctl(struct arch_timer_context *ctxt);104u32 timer_get_ctl(struct arch_timer_context *ctxt);
105u64 timer_get_cval(struct arch_timer_context *ctxt);105u64 timer_get_cval(struct arch_timer_context *ctxt);
106 106
107/* CPU HP callbacks */
108void kvm_timer_cpu_up(void);
109void kvm_timer_cpu_down(void);
110
107#endif111#endif
include/kvm/arm_vgic.h+4−0
@@ -432,4 +432,8 @@ int vgic_v4_load(struct kvm_vcpu *vcpu); · 431 unmodified lines
432void vgic_v4_commit(struct kvm_vcpu *vcpu);432void vgic_v4_commit(struct kvm_vcpu *vcpu);
433int vgic_v4_put(struct kvm_vcpu *vcpu, bool need_db);433int vgic_v4_put(struct kvm_vcpu *vcpu, bool need_db);
434 434
435/* CPU HP callbacks */
436void kvm_vgic_cpu_up(void);
437void kvm_vgic_cpu_down(void);
438
435#endif /* __KVM_ARM_VGIC_H */439#endif /* __KVM_ARM_VGIC_H */
include/linux/cpuhotplug.h+1−4
@@ -187,10 +187,6 @@ enum cpuhp_state { · 186 unmodified lines
187 CPUHP_AP_CSKY_TIMER_STARTING,187 CPUHP_AP_CSKY_TIMER_STARTING,
188 CPUHP_AP_TI_GP_TIMER_STARTING,188 CPUHP_AP_TI_GP_TIMER_STARTING,
189 CPUHP_AP_HYPERV_TIMER_STARTING,189 CPUHP_AP_HYPERV_TIMER_STARTING,
190 CPUHP_AP_KVM_STARTING,
191 CPUHP_AP_KVM_ARM_VGIC_INIT_STARTING,
192 CPUHP_AP_KVM_ARM_VGIC_STARTING,
193 CPUHP_AP_KVM_ARM_TIMER_STARTING,
194 /* Must be the last timer callback */190 /* Must be the last timer callback */
195 CPUHP_AP_DUMMY_TIMER_STARTING,191 CPUHP_AP_DUMMY_TIMER_STARTING,
196 CPUHP_AP_ARM_XEN_STARTING,192 CPUHP_AP_ARM_XEN_STARTING,
@@ -205,6 +201,7 @@ enum cpuhp_state { · 8 unmodified lines
205 201
206 /* Online section invoked on the hotplugged CPU from the hotplug thread */202 /* Online section invoked on the hotplugged CPU from the hotplug thread */
207 CPUHP_AP_ONLINE_IDLE,203 CPUHP_AP_ONLINE_IDLE,
204 CPUHP_AP_KVM_ONLINE,
208 CPUHP_AP_SCHED_WAIT_EMPTY,205 CPUHP_AP_SCHED_WAIT_EMPTY,
209 CPUHP_AP_SMPBOOT_THREADS,206 CPUHP_AP_SMPBOOT_THREADS,
210 CPUHP_AP_X86_VDSO_VMA_ONLINE,207 CPUHP_AP_X86_VDSO_VMA_ONLINE,
include/linux/kvm_host.h+8−11
@@ -324,7 +324,7 @@ struct kvm_vcpu { · 323 unmodified lines
324#endif324#endif
325 int cpu;325 int cpu;
326 int vcpu_id; /* id given by userspace at creation */326 int vcpu_id; /* id given by userspace at creation */
327 int vcpu_idx; /* index in kvm->vcpus array */327 int vcpu_idx; /* index into kvm->vcpu_array */
328 int ____srcu_idx; /* Don't use this directly. You've been warned. */328 int ____srcu_idx; /* Don't use this directly. You've been warned. */
329#ifdef CONFIG_PROVE_RCU329#ifdef CONFIG_PROVE_RCU
330 int srcu_depth;330 int srcu_depth;
@@ -956,8 +956,7 @@ static inline void kvm_irqfd_exit(void) · 625 unmodified lines
956{956{
957}957}
958#endif958#endif
959int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,959int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module);
960 struct module *module);
961void kvm_exit(void);960void kvm_exit(void);
962 961
963void kvm_get_kvm(struct kvm *kvm);962void kvm_get_kvm(struct kvm *kvm);
@@ -1354,7 +1353,7 @@ void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu); · 390 unmodified lines
1354bool kvm_vcpu_wake_up(struct kvm_vcpu *vcpu);1353bool kvm_vcpu_wake_up(struct kvm_vcpu *vcpu);
1355void kvm_vcpu_kick(struct kvm_vcpu *vcpu);1354void kvm_vcpu_kick(struct kvm_vcpu *vcpu);
1356int kvm_vcpu_yield_to(struct kvm_vcpu *target);1355int kvm_vcpu_yield_to(struct kvm_vcpu *target);
1357void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu, bool usermode_vcpu_not_eligible);1356void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu, bool yield_to_kernel_mode);
1358 1357
1359void kvm_flush_remote_tlbs(struct kvm *kvm);1358void kvm_flush_remote_tlbs(struct kvm *kvm);
1360 1359
@@ -1423,9 +1422,6 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu, · 62 unmodified lines
1423 struct kvm_guest_debug *dbg);1422 struct kvm_guest_debug *dbg);
1424int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu);1423int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu);
1425 1424
1426int kvm_arch_init(void *opaque);
1427void kvm_arch_exit(void);
1428
1429void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu);1425void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu);
1430 1426
1431void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu);1427void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu);
@@ -1445,11 +1441,10 @@ void kvm_arch_create_vcpu_debugfs(struct kvm_vcpu *vcpu, struct dentry *debugfs_ · 13 unmodified lines
1445static inline void kvm_create_vcpu_debugfs(struct kvm_vcpu *vcpu) {}1441static inline void kvm_create_vcpu_debugfs(struct kvm_vcpu *vcpu) {}
1446#endif1442#endif
1447 1443
1444#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING
1448int kvm_arch_hardware_enable(void);1445int kvm_arch_hardware_enable(void);
1449void kvm_arch_hardware_disable(void);1446void kvm_arch_hardware_disable(void);
1450int kvm_arch_hardware_setup(void *opaque);1447#endif
1451void kvm_arch_hardware_unsetup(void);
1452int kvm_arch_check_processor_compat(void *opaque);
1453int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu);1448int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu);
1454bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu);1449bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu);
1455int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu);1450int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu);
@@ -1466,7 +1461,7 @@ int kvm_arch_create_vm_debugfs(struct kvm *kvm); · 10 unmodified lines
1466 */1461 */
1467static inline struct kvm *kvm_arch_alloc_vm(void)1462static inline struct kvm *kvm_arch_alloc_vm(void)
1468{1463{
1469 return kzalloc(sizeof(struct kvm), GFP_KERNEL);1464 return kzalloc(sizeof(struct kvm), GFP_KERNEL_ACCOUNT);
1470}1465}
1471#endif1466#endif
1472 1467
@@ -2081,7 +2076,9 @@ static inline bool kvm_check_request(int req, struct kvm_vcpu *vcpu) · 608 unmodified lines
2081 }2076 }
2082}2077}
2083 2078
2079#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING
2084extern bool kvm_rebooting;2080extern bool kvm_rebooting;
2081#endif
2085 2082
2086extern unsigned int halt_poll_ns;2083extern unsigned int halt_poll_ns;
2087extern unsigned int halt_poll_ns_grow;2084extern unsigned int halt_poll_ns_grow;
include/linux/kvm_types.h+1−1
@@ -40,7 +40,7 @@ typedef unsigned long gva_t; · 39 unmodified lines
40typedef u64 gpa_t;40typedef u64 gpa_t;
41typedef u64 gfn_t;41typedef u64 gfn_t;
42 42
43#define GPA_INVALID (~(gpa_t)0)43#define INVALID_GPA (~(gpa_t)0)
44 44
45typedef unsigned long hva_t;45typedef unsigned long hva_t;
46typedef u64 hpa_t;46typedef u64 hpa_t;
include/linux/perf/riscv_pmu.h+5−0
@@ -26,6 +26,8 @@ · 25 unmodified lines
26 26
27#define RISCV_PMU_STOP_FLAG_RESET 127#define RISCV_PMU_STOP_FLAG_RESET 1
28 28
29#define RISCV_PMU_CONFIG1_GUEST_EVENTS 0x1
30
29struct cpu_hw_events {31struct cpu_hw_events {
30 /* currently enabled events */32 /* currently enabled events */
31 int n_events;33 int n_events;
@@ -73,6 +75,9 @@ void riscv_pmu_legacy_skip_init(void); · 41 unmodified lines
73static inline void riscv_pmu_legacy_skip_init(void) {};75static inline void riscv_pmu_legacy_skip_init(void) {};
74#endif76#endif
75struct riscv_pmu *riscv_pmu_alloc(void);77struct riscv_pmu *riscv_pmu_alloc(void);
78#ifdef CONFIG_RISCV_PMU_SBI
79int riscv_pmu_get_hpm_info(u32 *hw_ctr_width, u32 *num_hw_ctr);
80#endif
76 81
77#endif /* CONFIG_RISCV_PMU */82#endif /* CONFIG_RISCV_PMU */
78 83
include/uapi/linux/kvm.h+9−0
@@ -583,6 +583,8 @@ struct kvm_s390_mem_op { · 582 unmodified lines
583 struct {583 struct {
584 __u8 ar; /* the access register number */584 __u8 ar; /* the access register number */
585 __u8 key; /* access key, ignored if flag unset */585 __u8 key; /* access key, ignored if flag unset */
586 __u8 pad1[6]; /* ignored */
587 __u64 old_addr; /* ignored if cmpxchg flag unset */
586 };588 };
587 __u32 sida_offset; /* offset into the sida */589 __u32 sida_offset; /* offset into the sida */
588 __u8 reserved[32]; /* ignored */590 __u8 reserved[32]; /* ignored */
@@ -595,11 +597,17 @@ struct kvm_s390_mem_op { · 6 unmodified lines
595#define KVM_S390_MEMOP_SIDA_WRITE 3597#define KVM_S390_MEMOP_SIDA_WRITE 3
596#define KVM_S390_MEMOP_ABSOLUTE_READ 4598#define KVM_S390_MEMOP_ABSOLUTE_READ 4
597#define KVM_S390_MEMOP_ABSOLUTE_WRITE 5599#define KVM_S390_MEMOP_ABSOLUTE_WRITE 5
600#define KVM_S390_MEMOP_ABSOLUTE_CMPXCHG 6
601
598/* flags for kvm_s390_mem_op->flags */602/* flags for kvm_s390_mem_op->flags */
599#define KVM_S390_MEMOP_F_CHECK_ONLY (1ULL << 0)603#define KVM_S390_MEMOP_F_CHECK_ONLY (1ULL << 0)
600#define KVM_S390_MEMOP_F_INJECT_EXCEPTION (1ULL << 1)604#define KVM_S390_MEMOP_F_INJECT_EXCEPTION (1ULL << 1)
601#define KVM_S390_MEMOP_F_SKEY_PROTECTION (1ULL << 2)605#define KVM_S390_MEMOP_F_SKEY_PROTECTION (1ULL << 2)
602 606
607/* flags specifying extension support via KVM_CAP_S390_MEM_OP_EXTENSION */
608#define KVM_S390_MEMOP_EXTENSION_CAP_BASE (1 << 0)
609#define KVM_S390_MEMOP_EXTENSION_CAP_CMPXCHG (1 << 1)
610
603/* for KVM_INTERRUPT */611/* for KVM_INTERRUPT */
604struct kvm_interrupt {612struct kvm_interrupt {
605 /* in */613 /* in */
@@ -1175,6 +1183,7 @@ struct kvm_ppc_resize_hpt { · 569 unmodified lines
1175#define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 2231183#define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 223
1176#define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 2241184#define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224
1177#define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 2251185#define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 225
1186#define KVM_CAP_PMU_EVENT_MASKED_EVENTS 226
1178 1187
1179#ifdef KVM_CAP_IRQ_ROUTING1188#ifdef KVM_CAP_IRQ_ROUTING
1180 1189
tools/testing/selftests/kvm/Makefile+4−0
@@ -67,6 +67,7 @@ TEST_GEN_PROGS_x86_64 += x86_64/fix_hypercall_test · 66 unmodified lines
67TEST_GEN_PROGS_x86_64 += x86_64/hyperv_clock67TEST_GEN_PROGS_x86_64 += x86_64/hyperv_clock
68TEST_GEN_PROGS_x86_64 += x86_64/hyperv_cpuid68TEST_GEN_PROGS_x86_64 += x86_64/hyperv_cpuid
69TEST_GEN_PROGS_x86_64 += x86_64/hyperv_evmcs69TEST_GEN_PROGS_x86_64 += x86_64/hyperv_evmcs
70TEST_GEN_PROGS_x86_64 += x86_64/hyperv_extended_hypercalls
70TEST_GEN_PROGS_x86_64 += x86_64/hyperv_features71TEST_GEN_PROGS_x86_64 += x86_64/hyperv_features
71TEST_GEN_PROGS_x86_64 += x86_64/hyperv_ipi72TEST_GEN_PROGS_x86_64 += x86_64/hyperv_ipi
72TEST_GEN_PROGS_x86_64 += x86_64/hyperv_svm_test73TEST_GEN_PROGS_x86_64 += x86_64/hyperv_svm_test
@@ -200,6 +201,9 @@ CFLAGS += -Wall -Wstrict-prototypes -Wuninitialized -O2 -g -std=gnu99 \ · 127 unmodified lines
200 -I$(LINUX_TOOL_ARCH_INCLUDE) -I$(LINUX_HDR_PATH) -Iinclude \201 -I$(LINUX_TOOL_ARCH_INCLUDE) -I$(LINUX_HDR_PATH) -Iinclude \
201 -I$(<D) -Iinclude/$(ARCH_DIR) -I ../rseq -I.. $(EXTRA_CFLAGS) \202 -I$(<D) -Iinclude/$(ARCH_DIR) -I ../rseq -I.. $(EXTRA_CFLAGS) \
202 $(KHDR_INCLUDES)203 $(KHDR_INCLUDES)
204ifeq ($(ARCH),s390)
205 CFLAGS += -march=z10
206endif
203 207
204no-pie-option := $(call try-run, echo 'int main(void) { return 0; }' | \208no-pie-option := $(call try-run, echo 'int main(void) { return 0; }' | \
205 $(CC) -Werror $(CFLAGS) -no-pie -x c - -o "$$TMP", -no-pie)209 $(CC) -Werror $(CFLAGS) -no-pie -x c - -o "$$TMP", -no-pie)
tools/testing/selftests/kvm/aarch64/page_fault_test.c+0−2
@@ -1112,8 +1112,6 @@ int main(int argc, char *argv[]) · 1111 unmodified lines
1112 enum vm_mem_backing_src_type src_type;1112 enum vm_mem_backing_src_type src_type;
1113 int opt;1113 int opt;
1114 1114
1115 setbuf(stdout, NULL);
1116
1117 src_type = DEFAULT_VM_MEM_SRC;1115 src_type = DEFAULT_VM_MEM_SRC;
1118 1116
1119 while ((opt = getopt(argc, argv, "hm:s:")) != -1) {1117 while ((opt = getopt(argc, argv, "hm:s:")) != -1) {
tools/testing/selftests/kvm/config+1−0
@@ -1,3 +1,4 @@
1CONFIG_KVM=y1CONFIG_KVM=y
2CONFIG_KVM_INTEL=y2CONFIG_KVM_INTEL=y
3CONFIG_KVM_AMD=y3CONFIG_KVM_AMD=y
4CONFIG_USERFAULTFD=y
tools/testing/selftests/kvm/include/x86_64/hyperv.h+102−47
@@ -85,61 +85,110 @@ · 84 unmodified lines
85#define HV_X64_MSR_SYNDBG_OPTIONS 0x400000FF85#define HV_X64_MSR_SYNDBG_OPTIONS 0x400000FF
86 86
87/* HYPERV_CPUID_FEATURES.EAX */87/* HYPERV_CPUID_FEATURES.EAX */
88#define HV_MSR_VP_RUNTIME_AVAILABLE BIT(0)88#define HV_MSR_VP_RUNTIME_AVAILABLE \
89#define HV_MSR_TIME_REF_COUNT_AVAILABLE BIT(1)89 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 0)
90#define HV_MSR_SYNIC_AVAILABLE BIT(2)90#define HV_MSR_TIME_REF_COUNT_AVAILABLE \
91#define HV_MSR_SYNTIMER_AVAILABLE BIT(3)91 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 1)
92#define HV_MSR_APIC_ACCESS_AVAILABLE BIT(4)92#define HV_MSR_SYNIC_AVAILABLE \
93#define HV_MSR_HYPERCALL_AVAILABLE BIT(5)93 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 2)
94#define HV_MSR_VP_INDEX_AVAILABLE BIT(6)94#define HV_MSR_SYNTIMER_AVAILABLE \
95#define HV_MSR_RESET_AVAILABLE BIT(7)95 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 3)
96#define HV_MSR_STAT_PAGES_AVAILABLE BIT(8)96#define HV_MSR_APIC_ACCESS_AVAILABLE \
97#define HV_MSR_REFERENCE_TSC_AVAILABLE BIT(9)97 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 4)
98#define HV_MSR_GUEST_IDLE_AVAILABLE BIT(10)98#define HV_MSR_HYPERCALL_AVAILABLE \
99#define HV_ACCESS_FREQUENCY_MSRS BIT(11)99 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 5)
100#define HV_ACCESS_REENLIGHTENMENT BIT(13)100#define HV_MSR_VP_INDEX_AVAILABLE \
101#define HV_ACCESS_TSC_INVARIANT BIT(15)101 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 6)
102#define HV_MSR_RESET_AVAILABLE \
103 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 7)
104#define HV_MSR_STAT_PAGES_AVAILABLE \
105 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 8)
106#define HV_MSR_REFERENCE_TSC_AVAILABLE \
107 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 9)
108#define HV_MSR_GUEST_IDLE_AVAILABLE \
109 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 10)
110#define HV_ACCESS_FREQUENCY_MSRS \
111 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 11)
112#define HV_ACCESS_REENLIGHTENMENT \
113 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 13)
114#define HV_ACCESS_TSC_INVARIANT \
115 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EAX, 15)
102 116
103/* HYPERV_CPUID_FEATURES.EBX */117/* HYPERV_CPUID_FEATURES.EBX */
104#define HV_CREATE_PARTITIONS BIT(0)118#define HV_CREATE_PARTITIONS \
105#define HV_ACCESS_PARTITION_ID BIT(1)119 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 0)
106#define HV_ACCESS_MEMORY_POOL BIT(2)120#define HV_ACCESS_PARTITION_ID \
107#define HV_ADJUST_MESSAGE_BUFFERS BIT(3)121 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 1)
108#define HV_POST_MESSAGES BIT(4)122#define HV_ACCESS_MEMORY_POOL \
109#define HV_SIGNAL_EVENTS BIT(5)123 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 2)
110#define HV_CREATE_PORT BIT(6)124#define HV_ADJUST_MESSAGE_BUFFERS \
111#define HV_CONNECT_PORT BIT(7)125 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 3)
112#define HV_ACCESS_STATS BIT(8)126#define HV_POST_MESSAGES \
113#define HV_DEBUGGING BIT(11)127 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 4)
114#define HV_CPU_MANAGEMENT BIT(12)128#define HV_SIGNAL_EVENTS \
115#define HV_ISOLATION BIT(22)129 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 5)
130#define HV_CREATE_PORT \
131 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 6)
132#define HV_CONNECT_PORT \
133 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 7)
134#define HV_ACCESS_STATS \
135 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 8)
136#define HV_DEBUGGING \
137 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 11)
138#define HV_CPU_MANAGEMENT \
139 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 12)
140#define HV_ENABLE_EXTENDED_HYPERCALLS \
141 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 20)
142#define HV_ISOLATION \
143 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EBX, 22)
116 144
117/* HYPERV_CPUID_FEATURES.EDX */145/* HYPERV_CPUID_FEATURES.EDX */
118#define HV_X64_MWAIT_AVAILABLE BIT(0)146#define HV_X64_MWAIT_AVAILABLE \
119#define HV_X64_GUEST_DEBUGGING_AVAILABLE BIT(1)147 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 0)
120#define HV_X64_PERF_MONITOR_AVAILABLE BIT(2)148#define HV_X64_GUEST_DEBUGGING_AVAILABLE \
121#define HV_X64_CPU_DYNAMIC_PARTITIONING_AVAILABLE BIT(3)149 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 1)
122#define HV_X64_HYPERCALL_XMM_INPUT_AVAILABLE BIT(4)150#define HV_X64_PERF_MONITOR_AVAILABLE \
123#define HV_X64_GUEST_IDLE_STATE_AVAILABLE BIT(5)151 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 2)
124#define HV_FEATURE_FREQUENCY_MSRS_AVAILABLE BIT(8)152#define HV_X64_CPU_DYNAMIC_PARTITIONING_AVAILABLE \
125#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE BIT(10)153 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 3)
126#define HV_FEATURE_DEBUG_MSRS_AVAILABLE BIT(11)154#define HV_X64_HYPERCALL_XMM_INPUT_AVAILABLE \
127#define HV_STIMER_DIRECT_MODE_AVAILABLE BIT(19)155 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 4)
156#define HV_X64_GUEST_IDLE_STATE_AVAILABLE \
157 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 5)
158#define HV_FEATURE_FREQUENCY_MSRS_AVAILABLE \
159 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 8)
160#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE \
161 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 10)
162#define HV_FEATURE_DEBUG_MSRS_AVAILABLE \
163 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 11)
164#define HV_STIMER_DIRECT_MODE_AVAILABLE \
165 KVM_X86_CPU_FEATURE(HYPERV_CPUID_FEATURES, 0, EDX, 19)
128 166
129/* HYPERV_CPUID_ENLIGHTMENT_INFO.EAX */167/* HYPERV_CPUID_ENLIGHTMENT_INFO.EAX */
130#define HV_X64_AS_SWITCH_RECOMMENDED BIT(0)168#define HV_X64_AS_SWITCH_RECOMMENDED \
131#define HV_X64_LOCAL_TLB_FLUSH_RECOMMENDED BIT(1)169 KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 0)
132#define HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED BIT(2)170#define HV_X64_LOCAL_TLB_FLUSH_RECOMMENDED \
133#define HV_X64_APIC_ACCESS_RECOMMENDED BIT(3)171 KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 1)
134#define HV_X64_SYSTEM_RESET_RECOMMENDED BIT(4)172#define HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED \
135#define HV_X64_RELAXED_TIMING_RECOMMENDED BIT(5)173 KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 2)
136#define HV_DEPRECATING_AEOI_RECOMMENDED BIT(9)174#define HV_X64_APIC_ACCESS_RECOMMENDED \
137#define HV_X64_CLUSTER_IPI_RECOMMENDED BIT(10)175 KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 3)
138#define HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED BIT(11)176#define HV_X64_SYSTEM_RESET_RECOMMENDED \
139#define HV_X64_ENLIGHTENED_VMCS_RECOMMENDED BIT(14)177 KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 4)
178#define HV_X64_RELAXED_TIMING_RECOMMENDED \
179 KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 5)
180#define HV_DEPRECATING_AEOI_RECOMMENDED \
181 KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 9)
182#define HV_X64_CLUSTER_IPI_RECOMMENDED \
183 KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 10)
184#define HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED \
185 KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 11)
186#define HV_X64_ENLIGHTENED_VMCS_RECOMMENDED \
187 KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EAX, 14)
140 188
141/* HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES.EAX */189/* HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES.EAX */
142#define HV_X64_SYNDBG_CAP_ALLOW_KERNEL_DEBUGGING BIT(1)190#define HV_X64_SYNDBG_CAP_ALLOW_KERNEL_DEBUGGING \
191 KVM_X86_CPU_FEATURE(HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES, 0, EAX, 1)
143 192
144/* Hypercalls */193/* Hypercalls */
145#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE 0x0002194#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE 0x0002
@@ -166,6 +215,9 @@ · 20 unmodified lines
166#define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE 0x00af215#define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE 0x00af
167#define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_LIST 0x00b0216#define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_LIST 0x00b0
168 217
218/* Extended hypercalls */
219#define HV_EXT_CALL_QUERY_CAPABILITIES 0x8001
220
169#define HV_FLUSH_ALL_PROCESSORS BIT(0)221#define HV_FLUSH_ALL_PROCESSORS BIT(0)
170#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES BIT(1)222#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES BIT(1)
171#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY BIT(2)223#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY BIT(2)
@@ -288,4 +340,7 @@ struct hyperv_test_pages { · 116 unmodified lines
288struct hyperv_test_pages *vcpu_alloc_hyperv_test_pages(struct kvm_vm *vm,340struct hyperv_test_pages *vcpu_alloc_hyperv_test_pages(struct kvm_vm *vm,
289 vm_vaddr_t *p_hv_pages_gva);341 vm_vaddr_t *p_hv_pages_gva);
290 342
343/* HV_X64_MSR_TSC_INVARIANT_CONTROL bits */
344#define HV_INVARIANT_TSC_EXPOSED BIT_ULL(0)
345
291#endif /* !SELFTEST_KVM_HYPERV_H */346#endif /* !SELFTEST_KVM_HYPERV_H */
tools/testing/selftests/kvm/include/x86_64/processor.h+27−4
@@ -19,6 +19,9 @@ · 18 unmodified lines
19 19
20#include "../kvm_util.h"20#include "../kvm_util.h"
21 21
22extern bool host_cpu_is_intel;
23extern bool host_cpu_is_amd;
24
22#define NMI_VECTOR 0x0225#define NMI_VECTOR 0x02
23 26
24#define X86_EFLAGS_FIXED (1u << 1)27#define X86_EFLAGS_FIXED (1u << 1)
@@ -137,6 +140,7 @@ struct kvm_x86_cpu_feature { · 112 unmodified lines
137#define X86_FEATURE_GBPAGES KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 26)140#define X86_FEATURE_GBPAGES KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 26)
138#define X86_FEATURE_RDTSCP KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 27)141#define X86_FEATURE_RDTSCP KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 27)
139#define X86_FEATURE_LM KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 29)142#define X86_FEATURE_LM KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 29)
143#define X86_FEATURE_INVTSC KVM_X86_CPU_FEATURE(0x80000007, 0, EDX, 8)
140#define X86_FEATURE_RDPRU KVM_X86_CPU_FEATURE(0x80000008, 0, EBX, 4)144#define X86_FEATURE_RDPRU KVM_X86_CPU_FEATURE(0x80000008, 0, EBX, 4)
141#define X86_FEATURE_AMD_IBPB KVM_X86_CPU_FEATURE(0x80000008, 0, EBX, 12)145#define X86_FEATURE_AMD_IBPB KVM_X86_CPU_FEATURE(0x80000008, 0, EBX, 12)
142#define X86_FEATURE_NPT KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 0)146#define X86_FEATURE_NPT KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 0)
@@ -554,6 +558,28 @@ static inline uint32_t this_cpu_model(void) · 411 unmodified lines
554 return x86_model(this_cpu_fms());558 return x86_model(this_cpu_fms());
555}559}
556 560
561static inline bool this_cpu_vendor_string_is(const char *vendor)
562{
563 const uint32_t *chunk = (const uint32_t *)vendor;
564 uint32_t eax, ebx, ecx, edx;
565
566 cpuid(0, &eax, &ebx, &ecx, &edx);
567 return (ebx == chunk[0] && edx == chunk[1] && ecx == chunk[2]);
568}
569
570static inline bool this_cpu_is_intel(void)
571{
572 return this_cpu_vendor_string_is("GenuineIntel");
573}
574
575/*
576 * Exclude early K5 samples with a vendor string of "AMDisbetter!"
577 */
578static inline bool this_cpu_is_amd(void)
579{
580 return this_cpu_vendor_string_is("AuthenticAMD");
581}
582
557static inline uint32_t __this_cpu_has(uint32_t function, uint32_t index,583static inline uint32_t __this_cpu_has(uint32_t function, uint32_t index,
558 uint8_t reg, uint8_t lo, uint8_t hi)584 uint8_t reg, uint8_t lo, uint8_t hi)
559{585{
@@ -690,9 +716,6 @@ static inline void cpu_relax(void) · 130 unmodified lines
690 "hlt\n" \716 "hlt\n" \
691 )717 )
692 718
693bool is_intel_cpu(void);
694bool is_amd_cpu(void);
695
696struct kvm_x86_state *vcpu_save_state(struct kvm_vcpu *vcpu);719struct kvm_x86_state *vcpu_save_state(struct kvm_vcpu *vcpu);
697void vcpu_load_state(struct kvm_vcpu *vcpu, struct kvm_x86_state *state);720void vcpu_load_state(struct kvm_vcpu *vcpu, struct kvm_x86_state *state);
698void kvm_x86_state_cleanup(struct kvm_x86_state *state);721void kvm_x86_state_cleanup(struct kvm_x86_state *state);
@@ -716,7 +739,7 @@ static inline void vcpu_msrs_set(struct kvm_vcpu *vcpu, struct kvm_msrs *msrs) · 17 unmodified lines
716 int r = __vcpu_ioctl(vcpu, KVM_SET_MSRS, msrs);739 int r = __vcpu_ioctl(vcpu, KVM_SET_MSRS, msrs);
717 740
718 TEST_ASSERT(r == msrs->nmsrs,741 TEST_ASSERT(r == msrs->nmsrs,
719 "KVM_GET_MSRS failed, r: %i (failed on MSR %x)",742 "KVM_SET_MSRS failed, r: %i (failed on MSR %x)",
720 r, r < 0 || r >= msrs->nmsrs ? -1 : msrs->entries[r].index);743 r, r < 0 || r >= msrs->nmsrs ? -1 : msrs->entries[r].index);
721}744}
722static inline void vcpu_debugregs_get(struct kvm_vcpu *vcpu,745static inline void vcpu_debugregs_get(struct kvm_vcpu *vcpu,
tools/testing/selftests/kvm/kvm_binary_stats_test.c+10−11
@@ -19,6 +19,7 @@ · 18 unmodified lines
19#include "kvm_util.h"19#include "kvm_util.h"
20#include "asm/kvm.h"20#include "asm/kvm.h"
21#include "linux/kvm.h"21#include "linux/kvm.h"
22#include "kselftest.h"
22 23
23static void stats_test(int stats_fd)24static void stats_test(int stats_fd)
24{25{
@@ -51,7 +52,7 @@ static void stats_test(int stats_fd) · 26 unmodified lines
51 52
52 /* Sanity check for other fields in header */53 /* Sanity check for other fields in header */
53 if (header.num_desc == 0) {54 if (header.num_desc == 0) {
54 printf("No KVM stats defined!");55 ksft_print_msg("No KVM stats defined!\n");
55 return;56 return;
56 }57 }
57 /*58 /*
@@ -133,7 +134,7 @@ static void stats_test(int stats_fd) · 75 unmodified lines
133 "Bucket size of stats (%s) is not zero",134 "Bucket size of stats (%s) is not zero",
134 pdesc->name);135 pdesc->name);
135 }136 }
136 size_data += pdesc->size * sizeof(*stats_data);137 size_data = max(size_data, pdesc->offset + pdesc->size * sizeof(*stats_data));
137 }138 }
138 139
139 /*140 /*
@@ -148,14 +149,6 @@ static void stats_test(int stats_fd) · 8 unmodified lines
148 TEST_ASSERT(size_data >= header.num_desc * sizeof(*stats_data),149 TEST_ASSERT(size_data >= header.num_desc * sizeof(*stats_data),
149 "Data size is not correct");150 "Data size is not correct");
150 151
151 /* Check stats offset */
152 for (i = 0; i < header.num_desc; ++i) {
153 pdesc = get_stats_descriptor(stats_desc, i, &header);
154 TEST_ASSERT(pdesc->offset < size_data,
155 "Invalid offset (%u) for stats: %s",
156 pdesc->offset, pdesc->name);
157 }
158
159 /* Allocate memory for stats data */152 /* Allocate memory for stats data */
160 stats_data = malloc(size_data);153 stats_data = malloc(size_data);
161 TEST_ASSERT(stats_data, "Allocate memory for stats data");154 TEST_ASSERT(stats_data, "Allocate memory for stats data");
@@ -224,9 +217,13 @@ int main(int argc, char *argv[]) · 62 unmodified lines
224 max_vcpu = DEFAULT_NUM_VCPU;217 max_vcpu = DEFAULT_NUM_VCPU;
225 }218 }
226 219
220 ksft_print_header();
221
227 /* Check the extension for binary stats */222 /* Check the extension for binary stats */
228 TEST_REQUIRE(kvm_has_cap(KVM_CAP_BINARY_STATS_FD));223 TEST_REQUIRE(kvm_has_cap(KVM_CAP_BINARY_STATS_FD));
229 224
225 ksft_set_plan(max_vm);
226
230 /* Create VMs and VCPUs */227 /* Create VMs and VCPUs */
231 vms = malloc(sizeof(vms[0]) * max_vm);228 vms = malloc(sizeof(vms[0]) * max_vm);
232 TEST_ASSERT(vms, "Allocate memory for storing VM pointers");229 TEST_ASSERT(vms, "Allocate memory for storing VM pointers");
@@ -245,10 +242,12 @@ int main(int argc, char *argv[]) · 12 unmodified lines
245 vm_stats_test(vms[i]);242 vm_stats_test(vms[i]);
246 for (j = 0; j < max_vcpu; ++j)243 for (j = 0; j < max_vcpu; ++j)
247 vcpu_stats_test(vcpus[i * max_vcpu + j]);244 vcpu_stats_test(vcpus[i * max_vcpu + j]);
245 ksft_test_result_pass("vm%i\n", i);
248 }246 }
249 247
250 for (i = 0; i < max_vm; ++i)248 for (i = 0; i < max_vm; ++i)
251 kvm_vm_free(vms[i]);249 kvm_vm_free(vms[i]);
252 free(vms);250 free(vms);
253 return 0;251
252 ksft_finished(); /* Print results and exit() accordingly */
254}253}
tools/testing/selftests/kvm/lib/elf.c+2−0
@@ -90,6 +90,7 @@ static void elfhdr_get(const char *filename, Elf64_Ehdr *hdrp) · 89 unmodified lines
90 " hdrp->e_shentsize: %x\n"90 " hdrp->e_shentsize: %x\n"
91 " expected: %zx",91 " expected: %zx",
92 hdrp->e_shentsize, sizeof(Elf64_Shdr));92 hdrp->e_shentsize, sizeof(Elf64_Shdr));
93 close(fd);
93}94}
94 95
95/* VM ELF Load96/* VM ELF Load
@@ -190,4 +191,5 @@ void kvm_vm_elf_load(struct kvm_vm *vm, const char *filename) · 94 unmodified lines
190 phdr.p_filesz);191 phdr.p_filesz);
191 }192 }
192 }193 }
194 close(fd);
193}195}
tools/testing/selftests/kvm/lib/guest_modes.c+1−1
@@ -127,7 +127,7 @@ void guest_modes_cmdline(const char *arg) · 126 unmodified lines
127 mode_selected = true;127 mode_selected = true;
128 }128 }
129 129
130 mode = strtoul(optarg, NULL, 10);130 mode = atoi_non_negative("Guest mode ID", arg);
131 TEST_ASSERT(mode < NUM_VM_MODES, "Guest mode ID %d too big", mode);131 TEST_ASSERT(mode < NUM_VM_MODES, "Guest mode ID %d too big", mode);
132 guest_modes[mode].enabled = true;132 guest_modes[mode].enabled = true;
133}133}
tools/testing/selftests/kvm/lib/kvm_util.c+1−3
@@ -1844,6 +1844,7 @@ static struct exit_reason { · 1843 unmodified lines
1844 {KVM_EXIT_X86_RDMSR, "RDMSR"},1844 {KVM_EXIT_X86_RDMSR, "RDMSR"},
1845 {KVM_EXIT_X86_WRMSR, "WRMSR"},1845 {KVM_EXIT_X86_WRMSR, "WRMSR"},
1846 {KVM_EXIT_XEN, "XEN"},1846 {KVM_EXIT_XEN, "XEN"},
1847 {KVM_EXIT_HYPERV, "HYPERV"},
1847#ifdef KVM_EXIT_MEMORY_NOT_PRESENT1848#ifdef KVM_EXIT_MEMORY_NOT_PRESENT
1848 {KVM_EXIT_MEMORY_NOT_PRESENT, "MEMORY_NOT_PRESENT"},1849 {KVM_EXIT_MEMORY_NOT_PRESENT, "MEMORY_NOT_PRESENT"},
1849#endif1850#endif
@@ -1941,9 +1942,6 @@ vm_paddr_t vm_phy_page_alloc(struct kvm_vm *vm, vm_paddr_t paddr_min, · 91 unmodified lines
1941 return vm_phy_pages_alloc(vm, 1, paddr_min, memslot);1942 return vm_phy_pages_alloc(vm, 1, paddr_min, memslot);
1942}1943}
1943 1944
1944/* Arbitrary minimum physical address used for virtual translation tables. */
1945#define KVM_GUEST_PAGE_TABLE_MIN_PADDR 0x180000
1946
1947vm_paddr_t vm_alloc_page_table(struct kvm_vm *vm)1945vm_paddr_t vm_alloc_page_table(struct kvm_vm *vm)
1948{1946{
1949 return vm_phy_page_alloc(vm, KVM_GUEST_PAGE_TABLE_MIN_PADDR,1947 return vm_phy_page_alloc(vm, KVM_GUEST_PAGE_TABLE_MIN_PADDR,
tools/testing/selftests/kvm/lib/x86_64/processor.c+20−26
@@ -19,6 +19,8 @@ · 18 unmodified lines
19#define MAX_NR_CPUID_ENTRIES 10019#define MAX_NR_CPUID_ENTRIES 100
20 20
21vm_vaddr_t exception_handlers;21vm_vaddr_t exception_handlers;
22bool host_cpu_is_amd;
23bool host_cpu_is_intel;
22 24
23static void regs_dump(FILE *stream, struct kvm_regs *regs, uint8_t indent)25static void regs_dump(FILE *stream, struct kvm_regs *regs, uint8_t indent)
24{26{
@@ -113,7 +115,7 @@ static void sregs_dump(FILE *stream, struct kvm_sregs *sregs, uint8_t indent) · 88 unmodified lines
113 115
114bool kvm_is_tdp_enabled(void)116bool kvm_is_tdp_enabled(void)
115{117{
116 if (is_intel_cpu())118 if (host_cpu_is_intel)
117 return get_kvm_intel_param_bool("ept");119 return get_kvm_intel_param_bool("ept");
118 else120 else
119 return get_kvm_amd_param_bool("npt");121 return get_kvm_amd_param_bool("npt");
@@ -555,6 +557,8 @@ static void vcpu_setup(struct kvm_vm *vm, struct kvm_vcpu *vcpu) · 435 unmodified lines
555void kvm_arch_vm_post_create(struct kvm_vm *vm)557void kvm_arch_vm_post_create(struct kvm_vm *vm)
556{558{
557 vm_create_irqchip(vm);559 vm_create_irqchip(vm);
560 sync_global_to_guest(vm, host_cpu_is_intel);
561 sync_global_to_guest(vm, host_cpu_is_amd);
558}562}
559 563
560struct kvm_vcpu *vm_arch_vcpu_add(struct kvm_vm *vm, uint32_t vcpu_id,564struct kvm_vcpu *vm_arch_vcpu_add(struct kvm_vm *vm, uint32_t vcpu_id,
@@ -1006,28 +1010,6 @@ void kvm_x86_state_cleanup(struct kvm_x86_state *state) · 445 unmodified lines
1006 free(state);1010 free(state);
1007}1011}
1008 1012
1009static bool cpu_vendor_string_is(const char *vendor)
1010{
1011 const uint32_t *chunk = (const uint32_t *)vendor;
1012 uint32_t eax, ebx, ecx, edx;
1013
1014 cpuid(0, &eax, &ebx, &ecx, &edx);
1015 return (ebx == chunk[0] && edx == chunk[1] && ecx == chunk[2]);
1016}
1017
1018bool is_intel_cpu(void)
1019{
1020 return cpu_vendor_string_is("GenuineIntel");
1021}
1022
1023/*
1024 * Exclude early K5 samples with a vendor string of "AMDisbetter!"
1025 */
1026bool is_amd_cpu(void)
1027{
1028 return cpu_vendor_string_is("AuthenticAMD");
1029}
1030
1031void kvm_get_cpu_address_width(unsigned int *pa_bits, unsigned int *va_bits)1013void kvm_get_cpu_address_width(unsigned int *pa_bits, unsigned int *va_bits)
1032{1014{
1033 if (!kvm_cpu_has_p(X86_PROPERTY_MAX_PHY_ADDR)) {1015 if (!kvm_cpu_has_p(X86_PROPERTY_MAX_PHY_ADDR)) {
@@ -1162,9 +1144,15 @@ uint64_t kvm_hypercall(uint64_t nr, uint64_t a0, uint64_t a1, uint64_t a2, · 128 unmodified lines
1162{1144{
1163 uint64_t r;1145 uint64_t r;
1164 1146
1165 asm volatile("vmcall"1147 asm volatile("test %[use_vmmcall], %[use_vmmcall]\n\t"
1148 "jnz 1f\n\t"
1149 "vmcall\n\t"
1150 "jmp 2f\n\t"
1151 "1: vmmcall\n\t"
1152 "2:"
1166 : "=a"(r)1153 : "=a"(r)
1167 : "a"(nr), "b"(a0), "c"(a1), "d"(a2), "S"(a3));1154 : "a"(nr), "b"(a0), "c"(a1), "d"(a2), "S"(a3),
1155 [use_vmmcall] "r" (host_cpu_is_amd));
1168 return r;1156 return r;
1169}1157}
1170 1158
@@ -1236,7 +1224,7 @@ unsigned long vm_compute_max_gfn(struct kvm_vm *vm) · 65 unmodified lines
1236 max_gfn = (1ULL << (vm->pa_bits - vm->page_shift)) - 1;1224 max_gfn = (1ULL << (vm->pa_bits - vm->page_shift)) - 1;
1237 1225
1238 /* Avoid reserved HyperTransport region on AMD processors. */1226 /* Avoid reserved HyperTransport region on AMD processors. */
1239 if (!is_amd_cpu())1227 if (!host_cpu_is_amd)
1240 return max_gfn;1228 return max_gfn;
1241 1229
1242 /* On parts with <40 physical address bits, the area is fully hidden */1230 /* On parts with <40 physical address bits, the area is fully hidden */
@@ -1276,3 +1264,9 @@ bool vm_is_unrestricted_guest(struct kvm_vm *vm) · 33 unmodified lines
1276 1264
1277 return get_kvm_intel_param_bool("unrestricted_guest");1265 return get_kvm_intel_param_bool("unrestricted_guest");
1278}1266}
1267
1268void kvm_selftest_arch_init(void)
1269{
1270 host_cpu_is_intel = this_cpu_is_intel();
1271 host_cpu_is_amd = this_cpu_is_amd();
1272}
tools/testing/selftests/kvm/memslot_perf_test.c+1−4
@@ -308,8 +308,6 @@ static bool prepare_vm(struct vm_data *data, int nslots, uint64_t *maxslots, · 307 unmodified lines
308 data->hva_slots = malloc(sizeof(*data->hva_slots) * data->nslots);308 data->hva_slots = malloc(sizeof(*data->hva_slots) * data->nslots);
309 TEST_ASSERT(data->hva_slots, "malloc() fail");309 TEST_ASSERT(data->hva_slots, "malloc() fail");
310 310
311 data->vm = __vm_create_with_one_vcpu(&data->vcpu, mempages, guest_code);
312
313 pr_info_v("Adding slots 1..%i, each slot with %"PRIu64" pages + %"PRIu64" extra pages last\n",311 pr_info_v("Adding slots 1..%i, each slot with %"PRIu64" pages + %"PRIu64" extra pages last\n",
314 data->nslots, data->pages_per_slot, rempages);312 data->nslots, data->pages_per_slot, rempages);
315 313
@@ -349,6 +347,7 @@ static bool prepare_vm(struct vm_data *data, int nslots, uint64_t *maxslots, · 33 unmodified lines
349 virt_map(data->vm, MEM_GPA, MEM_GPA, data->npages);347 virt_map(data->vm, MEM_GPA, MEM_GPA, data->npages);
350 348
351 sync = (typeof(sync))vm_gpa2hva(data, MEM_SYNC_GPA, NULL);349 sync = (typeof(sync))vm_gpa2hva(data, MEM_SYNC_GPA, NULL);
350 sync->guest_page_size = data->vm->page_size;
352 atomic_init(&sync->start_flag, false);351 atomic_init(&sync->start_flag, false);
353 atomic_init(&sync->exit_flag, false);352 atomic_init(&sync->exit_flag, false);
354 atomic_init(&sync->sync_flag, false);353 atomic_init(&sync->sync_flag, false);
@@ -810,8 +809,6 @@ static bool test_execute(int nslots, uint64_t *maxslots, · 455 unmodified lines
810 }809 }
811 810
812 sync = (typeof(sync))vm_gpa2hva(data, MEM_SYNC_GPA, NULL);811 sync = (typeof(sync))vm_gpa2hva(data, MEM_SYNC_GPA, NULL);
813
814 sync->guest_page_size = data->vm->page_size;
815 if (tdata->prepare &&812 if (tdata->prepare &&
816 !tdata->prepare(data, sync, maxslots)) {813 !tdata->prepare(data, sync, maxslots)) {
817 ret = false;814 ret = false;
tools/testing/selftests/kvm/s390x/memop.c+524−148
@@ -9,6 +9,7 @@ · 8 unmodified lines
9#include <stdlib.h>9#include <stdlib.h>
10#include <string.h>10#include <string.h>
11#include <sys/ioctl.h>11#include <sys/ioctl.h>
12#include <pthread.h>
12 13
13#include <linux/bits.h>14#include <linux/bits.h>
14 15
@@ -26,6 +27,7 @@ enum mop_target { · 11 unmodified lines
26enum mop_access_mode {27enum mop_access_mode {
27 READ,28 READ,
28 WRITE,29 WRITE,
30 CMPXCHG,
29};31};
30 32
31struct mop_desc {33struct mop_desc {
@@ -44,57 +46,67 @@ struct mop_desc { · 12 unmodified lines
44 enum mop_access_mode mode;46 enum mop_access_mode mode;
45 void *buf;47 void *buf;
46 uint32_t sida_offset;48 uint32_t sida_offset;
49 void *old;
50 uint8_t old_value[16];
51 bool *cmpxchg_success;
47 uint8_t ar;52 uint8_t ar;
48 uint8_t key;53 uint8_t key;
49};54};
50 55
51static struct kvm_s390_mem_op ksmo_from_desc(struct mop_desc desc)56const uint8_t NO_KEY = 0xff;
57
58static struct kvm_s390_mem_op ksmo_from_desc(struct mop_desc *desc)
52{59{
53 struct kvm_s390_mem_op ksmo = {60 struct kvm_s390_mem_op ksmo = {
54 .gaddr = (uintptr_t)desc.gaddr,61 .gaddr = (uintptr_t)desc->gaddr,
55 .size = desc.size,62 .size = desc->size,
56 .buf = ((uintptr_t)desc.buf),63 .buf = ((uintptr_t)desc->buf),
57 .reserved = "ignored_ignored_ignored_ignored"64 .reserved = "ignored_ignored_ignored_ignored"
58 };65 };
59 66
60 switch (desc.target) {67 switch (desc->target) {
61 case LOGICAL:68 case LOGICAL:
62 if (desc.mode == READ)69 if (desc->mode == READ)
63 ksmo.op = KVM_S390_MEMOP_LOGICAL_READ;70 ksmo.op = KVM_S390_MEMOP_LOGICAL_READ;
64 if (desc.mode == WRITE)71 if (desc->mode == WRITE)
65 ksmo.op = KVM_S390_MEMOP_LOGICAL_WRITE;72 ksmo.op = KVM_S390_MEMOP_LOGICAL_WRITE;
66 break;73 break;
67 case SIDA:74 case SIDA:
68 if (desc.mode == READ)75 if (desc->mode == READ)
69 ksmo.op = KVM_S390_MEMOP_SIDA_READ;76 ksmo.op = KVM_S390_MEMOP_SIDA_READ;
70 if (desc.mode == WRITE)77 if (desc->mode == WRITE)
71 ksmo.op = KVM_S390_MEMOP_SIDA_WRITE;78 ksmo.op = KVM_S390_MEMOP_SIDA_WRITE;
72 break;79 break;
73 case ABSOLUTE:80 case ABSOLUTE:
74 if (desc.mode == READ)81 if (desc->mode == READ)
75 ksmo.op = KVM_S390_MEMOP_ABSOLUTE_READ;82 ksmo.op = KVM_S390_MEMOP_ABSOLUTE_READ;
76 if (desc.mode == WRITE)83 if (desc->mode == WRITE)
77 ksmo.op = KVM_S390_MEMOP_ABSOLUTE_WRITE;84 ksmo.op = KVM_S390_MEMOP_ABSOLUTE_WRITE;
85 if (desc->mode == CMPXCHG) {
86 ksmo.op = KVM_S390_MEMOP_ABSOLUTE_CMPXCHG;
87 ksmo.old_addr = (uint64_t)desc->old;
88 memcpy(desc->old_value, desc->old, desc->size);
89 }
78 break;90 break;
79 case INVALID:91 case INVALID:
80 ksmo.op = -1;92 ksmo.op = -1;
81 }93 }
82 if (desc.f_check)94 if (desc->f_check)
83 ksmo.flags |= KVM_S390_MEMOP_F_CHECK_ONLY;95 ksmo.flags |= KVM_S390_MEMOP_F_CHECK_ONLY;
84 if (desc.f_inject)96 if (desc->f_inject)
85 ksmo.flags |= KVM_S390_MEMOP_F_INJECT_EXCEPTION;97 ksmo.flags |= KVM_S390_MEMOP_F_INJECT_EXCEPTION;
86 if (desc._set_flags)98 if (desc->_set_flags)
87 ksmo.flags = desc.set_flags;99 ksmo.flags = desc->set_flags;
88 if (desc.f_key) {100 if (desc->f_key && desc->key != NO_KEY) {
89 ksmo.flags |= KVM_S390_MEMOP_F_SKEY_PROTECTION;101 ksmo.flags |= KVM_S390_MEMOP_F_SKEY_PROTECTION;
90 ksmo.key = desc.key;102 ksmo.key = desc->key;
91 }103 }
92 if (desc._ar)104 if (desc->_ar)
93 ksmo.ar = desc.ar;105 ksmo.ar = desc->ar;
94 else106 else
95 ksmo.ar = 0;107 ksmo.ar = 0;
96 if (desc._sida_offset)108 if (desc->_sida_offset)
97 ksmo.sida_offset = desc.sida_offset;109 ksmo.sida_offset = desc->sida_offset;
98 110
99 return ksmo;111 return ksmo;
100}112}
@@ -133,9 +145,13 @@ static void print_memop(struct kvm_vcpu *vcpu, const struct kvm_s390_mem_op *ksm · 32 unmodified lines
133 case KVM_S390_MEMOP_ABSOLUTE_WRITE:145 case KVM_S390_MEMOP_ABSOLUTE_WRITE:
134 printf("ABSOLUTE, WRITE, ");146 printf("ABSOLUTE, WRITE, ");
135 break;147 break;
148 case KVM_S390_MEMOP_ABSOLUTE_CMPXCHG:
149 printf("ABSOLUTE, CMPXCHG, ");
150 break;
136 }151 }
137 printf("gaddr=%llu, size=%u, buf=%llu, ar=%u, key=%u",152 printf("gaddr=%llu, size=%u, buf=%llu, ar=%u, key=%u, old_addr=%llx",
138 ksmo->gaddr, ksmo->size, ksmo->buf, ksmo->ar, ksmo->key);153 ksmo->gaddr, ksmo->size, ksmo->buf, ksmo->ar, ksmo->key,
154 ksmo->old_addr);
139 if (ksmo->flags & KVM_S390_MEMOP_F_CHECK_ONLY)155 if (ksmo->flags & KVM_S390_MEMOP_F_CHECK_ONLY)
140 printf(", CHECK_ONLY");156 printf(", CHECK_ONLY");
141 if (ksmo->flags & KVM_S390_MEMOP_F_INJECT_EXCEPTION)157 if (ksmo->flags & KVM_S390_MEMOP_F_INJECT_EXCEPTION)
@@ -145,24 +161,30 @@ static void print_memop(struct kvm_vcpu *vcpu, const struct kvm_s390_mem_op *ksm · 3 unmodified lines
145 puts(")");161 puts(")");
146}162}
147 163
148static void memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo)164static int err_memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo,
165 struct mop_desc *desc)
149{166{
150 struct kvm_vcpu *vcpu = info.vcpu;167 struct kvm_vcpu *vcpu = info.vcpu;
151 168
152 if (!vcpu)169 if (!vcpu)
153 vm_ioctl(info.vm, KVM_S390_MEM_OP, ksmo);170 return __vm_ioctl(info.vm, KVM_S390_MEM_OP, ksmo);
154 else171 else
155 vcpu_ioctl(vcpu, KVM_S390_MEM_OP, ksmo);172 return __vcpu_ioctl(vcpu, KVM_S390_MEM_OP, ksmo);
156}173}
157 174
158static int err_memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo)175static void memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo,
176 struct mop_desc *desc)
159{177{
160 struct kvm_vcpu *vcpu = info.vcpu;178 int r;
161 179
162 if (!vcpu)180 r = err_memop_ioctl(info, ksmo, desc);
163 return __vm_ioctl(info.vm, KVM_S390_MEM_OP, ksmo);181 if (ksmo->op == KVM_S390_MEMOP_ABSOLUTE_CMPXCHG) {
164 else182 if (desc->cmpxchg_success) {
165 return __vcpu_ioctl(vcpu, KVM_S390_MEM_OP, ksmo);183 int diff = memcmp(desc->old_value, desc->old, desc->size);
184 *desc->cmpxchg_success = !diff;
185 }
186 }
187 TEST_ASSERT(!r, __KVM_IOCTL_ERROR("KVM_S390_MEM_OP", r));
166}188}
167 189
168#define MEMOP(err, info_p, mop_target_p, access_mode_p, buf_p, size_p, ...) \190#define MEMOP(err, info_p, mop_target_p, access_mode_p, buf_p, size_p, ...) \
@@ -183,9 +205,9 @@ static int err_memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo) · 14 unmodified lines
183 else \205 else \
184 __desc.gaddr = __desc.gaddr_v; \206 __desc.gaddr = __desc.gaddr_v; \
185 } \207 } \
186 __ksmo = ksmo_from_desc(__desc); \208 __ksmo = ksmo_from_desc(&__desc); \
187 print_memop(__info.vcpu, &__ksmo); \209 print_memop(__info.vcpu, &__ksmo); \
188 err##memop_ioctl(__info, &__ksmo); \210 err##memop_ioctl(__info, &__ksmo, &__desc); \
189})211})
190 212
191#define MOP(...) MEMOP(, __VA_ARGS__)213#define MOP(...) MEMOP(, __VA_ARGS__)
@@ -199,6 +221,8 @@ static int err_memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo) · 7 unmodified lines
199#define AR(a) ._ar = 1, .ar = (a)221#define AR(a) ._ar = 1, .ar = (a)
200#define KEY(a) .f_key = 1, .key = (a)222#define KEY(a) .f_key = 1, .key = (a)
201#define INJECT .f_inject = 1223#define INJECT .f_inject = 1
224#define CMPXCHG_OLD(o) .old = (o)
225#define CMPXCHG_SUCCESS(s) .cmpxchg_success = (s)
202 226
203#define CHECK_N_DO(f, ...) ({ f(__VA_ARGS__, CHECK_ONLY); f(__VA_ARGS__); })227#define CHECK_N_DO(f, ...) ({ f(__VA_ARGS__, CHECK_ONLY); f(__VA_ARGS__); })
204 228
@@ -208,8 +232,8 @@ static int err_memop_ioctl(struct test_info info, struct kvm_s390_mem_op *ksmo) · 3 unmodified lines
208#define CR0_FETCH_PROTECTION_OVERRIDE (1UL << (63 - 38))232#define CR0_FETCH_PROTECTION_OVERRIDE (1UL << (63 - 38))
209#define CR0_STORAGE_PROTECTION_OVERRIDE (1UL << (63 - 39))233#define CR0_STORAGE_PROTECTION_OVERRIDE (1UL << (63 - 39))
210 234
211static uint8_t mem1[65536];235static uint8_t __aligned(PAGE_SIZE) mem1[65536];
212static uint8_t mem2[65536];236static uint8_t __aligned(PAGE_SIZE) mem2[65536];
213 237
214struct test_default {238struct test_default {
215 struct kvm_vm *kvm_vm;239 struct kvm_vm *kvm_vm;
@@ -241,6 +265,8 @@ enum stage { · 25 unmodified lines
241 STAGE_SKEYS_SET,265 STAGE_SKEYS_SET,
242 /* Guest copied memory (locations up to test case) */266 /* Guest copied memory (locations up to test case) */
243 STAGE_COPIED,267 STAGE_COPIED,
268 /* End of guest code reached */
269 STAGE_DONE,
244};270};
245 271
246#define HOST_SYNC(info_p, stage) \272#define HOST_SYNC(info_p, stage) \
@@ -252,6 +278,9 @@ enum stage { · 5 unmodified lines
252 \278 \
253 vcpu_run(__vcpu); \279 vcpu_run(__vcpu); \
254 get_ucall(__vcpu, &uc); \280 get_ucall(__vcpu, &uc); \
281 if (uc.cmd == UCALL_ABORT) { \
282 REPORT_GUEST_ASSERT_2(uc, "hints: %lu, %lu"); \
283 } \
255 ASSERT_EQ(uc.cmd, UCALL_SYNC); \284 ASSERT_EQ(uc.cmd, UCALL_SYNC); \
256 ASSERT_EQ(uc.args[1], __stage); \285 ASSERT_EQ(uc.args[1], __stage); \
257}) \286}) \
@@ -268,34 +297,66 @@ static void prepare_mem12(void) · 10 unmodified lines
268#define ASSERT_MEM_EQ(p1, p2, size) \297#define ASSERT_MEM_EQ(p1, p2, size) \
269 TEST_ASSERT(!memcmp(p1, p2, size), "Memory contents do not match!")298 TEST_ASSERT(!memcmp(p1, p2, size), "Memory contents do not match!")
270 299
271#define DEFAULT_WRITE_READ(copy_cpu, mop_cpu, mop_target_p, size, ...) \300static void default_write_read(struct test_info copy_cpu, struct test_info mop_cpu,
272({ \301 enum mop_target mop_target, uint32_t size, uint8_t key)
273 struct test_info __copy_cpu = (copy_cpu), __mop_cpu = (mop_cpu); \302{
274 enum mop_target __target = (mop_target_p); \303 prepare_mem12();
275 uint32_t __size = (size); \304 CHECK_N_DO(MOP, mop_cpu, mop_target, WRITE, mem1, size,
276 \305 GADDR_V(mem1), KEY(key));
277 prepare_mem12(); \306 HOST_SYNC(copy_cpu, STAGE_COPIED);
278 CHECK_N_DO(MOP, __mop_cpu, __target, WRITE, mem1, __size, \307 CHECK_N_DO(MOP, mop_cpu, mop_target, READ, mem2, size,
279 GADDR_V(mem1), ##__VA_ARGS__); \308 GADDR_V(mem2), KEY(key));
280 HOST_SYNC(__copy_cpu, STAGE_COPIED); \309 ASSERT_MEM_EQ(mem1, mem2, size);
281 CHECK_N_DO(MOP, __mop_cpu, __target, READ, mem2, __size, \310}
282 GADDR_V(mem2), ##__VA_ARGS__); \
283 ASSERT_MEM_EQ(mem1, mem2, __size); \
284})
285 311
286#define DEFAULT_READ(copy_cpu, mop_cpu, mop_target_p, size, ...) \312static void default_read(struct test_info copy_cpu, struct test_info mop_cpu,
287({ \313 enum mop_target mop_target, uint32_t size, uint8_t key)
288 struct test_info __copy_cpu = (copy_cpu), __mop_cpu = (mop_cpu); \314{
289 enum mop_target __target = (mop_target_p); \315 prepare_mem12();
290 uint32_t __size = (size); \316 CHECK_N_DO(MOP, mop_cpu, mop_target, WRITE, mem1, size, GADDR_V(mem1));
291 \317 HOST_SYNC(copy_cpu, STAGE_COPIED);
292 prepare_mem12(); \318 CHECK_N_DO(MOP, mop_cpu, mop_target, READ, mem2, size,
293 CHECK_N_DO(MOP, __mop_cpu, __target, WRITE, mem1, __size, \319 GADDR_V(mem2), KEY(key));
294 GADDR_V(mem1)); \320 ASSERT_MEM_EQ(mem1, mem2, size);
295 HOST_SYNC(__copy_cpu, STAGE_COPIED); \321}
296 CHECK_N_DO(MOP, __mop_cpu, __target, READ, mem2, __size, ##__VA_ARGS__);\322
297 ASSERT_MEM_EQ(mem1, mem2, __size); \323static void default_cmpxchg(struct test_default *test, uint8_t key)
298})324{
325 for (int size = 1; size <= 16; size *= 2) {
326 for (int offset = 0; offset < 16; offset += size) {
327 uint8_t __aligned(16) new[16] = {};
328 uint8_t __aligned(16) old[16];
329 bool succ;
330
331 prepare_mem12();
332 default_write_read(test->vcpu, test->vcpu, LOGICAL, 16, NO_KEY);
333
334 memcpy(&old, mem1, 16);
335 MOP(test->vm, ABSOLUTE, CMPXCHG, new + offset,
336 size, GADDR_V(mem1 + offset),
337 CMPXCHG_OLD(old + offset),
338 CMPXCHG_SUCCESS(&succ), KEY(key));
339 HOST_SYNC(test->vcpu, STAGE_COPIED);
340 MOP(test->vm, ABSOLUTE, READ, mem2, 16, GADDR_V(mem2));
341 TEST_ASSERT(succ, "exchange of values should succeed");
342 memcpy(mem1 + offset, new + offset, size);
343 ASSERT_MEM_EQ(mem1, mem2, 16);
344
345 memcpy(&old, mem1, 16);
346 new[offset]++;
347 old[offset]++;
348 MOP(test->vm, ABSOLUTE, CMPXCHG, new + offset,
349 size, GADDR_V(mem1 + offset),
350 CMPXCHG_OLD(old + offset),
351 CMPXCHG_SUCCESS(&succ), KEY(key));
352 HOST_SYNC(test->vcpu, STAGE_COPIED);
353 MOP(test->vm, ABSOLUTE, READ, mem2, 16, GADDR_V(mem2));
354 TEST_ASSERT(!succ, "exchange of values should not succeed");
355 ASSERT_MEM_EQ(mem1, mem2, 16);
356 ASSERT_MEM_EQ(&old, mem1, 16);
357 }
358 }
359}
299 360
300static void guest_copy(void)361static void guest_copy(void)
301{362{
@@ -310,7 +371,7 @@ static void test_copy(void) · 8 unmodified lines
310 371
311 HOST_SYNC(t.vcpu, STAGE_INITED);372 HOST_SYNC(t.vcpu, STAGE_INITED);
312 373
313 DEFAULT_WRITE_READ(t.vcpu, t.vcpu, LOGICAL, t.size);374 default_write_read(t.vcpu, t.vcpu, LOGICAL, t.size, NO_KEY);
314 375
315 kvm_vm_free(t.kvm_vm);376 kvm_vm_free(t.kvm_vm);
316}377}
@@ -357,26 +418,268 @@ static void test_copy_key(void) · 40 unmodified lines
357 HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);418 HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);
358 419
359 /* vm, no key */420 /* vm, no key */
360 DEFAULT_WRITE_READ(t.vcpu, t.vm, ABSOLUTE, t.size);421 default_write_read(t.vcpu, t.vm, ABSOLUTE, t.size, NO_KEY);
361 422
362 /* vm/vcpu, machting key or key 0 */423 /* vm/vcpu, machting key or key 0 */
363 DEFAULT_WRITE_READ(t.vcpu, t.vcpu, LOGICAL, t.size, KEY(0));424 default_write_read(t.vcpu, t.vcpu, LOGICAL, t.size, 0);
364 DEFAULT_WRITE_READ(t.vcpu, t.vcpu, LOGICAL, t.size, KEY(9));425 default_write_read(t.vcpu, t.vcpu, LOGICAL, t.size, 9);
365 DEFAULT_WRITE_READ(t.vcpu, t.vm, ABSOLUTE, t.size, KEY(0));426 default_write_read(t.vcpu, t.vm, ABSOLUTE, t.size, 0);
366 DEFAULT_WRITE_READ(t.vcpu, t.vm, ABSOLUTE, t.size, KEY(9));427 default_write_read(t.vcpu, t.vm, ABSOLUTE, t.size, 9);
367 /*428 /*
368 * There used to be different code paths for key handling depending on429 * There used to be different code paths for key handling depending on
369 * if the region crossed a page boundary.430 * if the region crossed a page boundary.
370 * There currently are not, but the more tests the merrier.431 * There currently are not, but the more tests the merrier.
371 */432 */
372 DEFAULT_WRITE_READ(t.vcpu, t.vcpu, LOGICAL, 1, KEY(0));433 default_write_read(t.vcpu, t.vcpu, LOGICAL, 1, 0);
373 DEFAULT_WRITE_READ(t.vcpu, t.vcpu, LOGICAL, 1, KEY(9));434 default_write_read(t.vcpu, t.vcpu, LOGICAL, 1, 9);
374 DEFAULT_WRITE_READ(t.vcpu, t.vm, ABSOLUTE, 1, KEY(0));435 default_write_read(t.vcpu, t.vm, ABSOLUTE, 1, 0);
375 DEFAULT_WRITE_READ(t.vcpu, t.vm, ABSOLUTE, 1, KEY(9));436 default_write_read(t.vcpu, t.vm, ABSOLUTE, 1, 9);
376 437
377 /* vm/vcpu, mismatching keys on read, but no fetch protection */438 /* vm/vcpu, mismatching keys on read, but no fetch protection */
378 DEFAULT_READ(t.vcpu, t.vcpu, LOGICAL, t.size, GADDR_V(mem2), KEY(2));439 default_read(t.vcpu, t.vcpu, LOGICAL, t.size, 2);
379 DEFAULT_READ(t.vcpu, t.vm, ABSOLUTE, t.size, GADDR_V(mem1), KEY(2));440 default_read(t.vcpu, t.vm, ABSOLUTE, t.size, 2);
441
442 kvm_vm_free(t.kvm_vm);
443}
444
445static void test_cmpxchg_key(void)
446{
447 struct test_default t = test_default_init(guest_copy_key);
448
449 HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);
450
451 default_cmpxchg(&t, NO_KEY);
452 default_cmpxchg(&t, 0);
453 default_cmpxchg(&t, 9);
454
455 kvm_vm_free(t.kvm_vm);
456}
457
458static __uint128_t cut_to_size(int size, __uint128_t val)
459{
460 switch (size) {
461 case 1:
462 return (uint8_t)val;
463 case 2:
464 return (uint16_t)val;
465 case 4:
466 return (uint32_t)val;
467 case 8:
468 return (uint64_t)val;
469 case 16:
470 return val;
471 }
472 GUEST_ASSERT_1(false, "Invalid size");
473 return 0;
474}
475
476static bool popcount_eq(__uint128_t a, __uint128_t b)
477{
478 unsigned int count_a, count_b;
479
480 count_a = __builtin_popcountl((uint64_t)(a >> 64)) +
481 __builtin_popcountl((uint64_t)a);
482 count_b = __builtin_popcountl((uint64_t)(b >> 64)) +
483 __builtin_popcountl((uint64_t)b);
484 return count_a == count_b;
485}
486
487static __uint128_t rotate(int size, __uint128_t val, int amount)
488{
489 unsigned int bits = size * 8;
490
491 amount = (amount + bits) % bits;
492 val = cut_to_size(size, val);
493 return (val << (bits - amount)) | (val >> amount);
494}
495
496const unsigned int max_block = 16;
497
498static void choose_block(bool guest, int i, int *size, int *offset)
499{
500 unsigned int rand;
501
502 rand = i;
503 if (guest) {
504 rand = rand * 19 + 11;
505 *size = 1 << ((rand % 3) + 2);
506 rand = rand * 19 + 11;
507 *offset = (rand % max_block) & ~(*size - 1);
508 } else {
509 rand = rand * 17 + 5;
510 *size = 1 << (rand % 5);
511 rand = rand * 17 + 5;
512 *offset = (rand % max_block) & ~(*size - 1);
513 }
514}
515
516static __uint128_t permutate_bits(bool guest, int i, int size, __uint128_t old)
517{
518 unsigned int rand;
519 int amount;
520 bool swap;
521
522 rand = i;
523 rand = rand * 3 + 1;
524 if (guest)
525 rand = rand * 3 + 1;
526 swap = rand % 2 == 0;
527 if (swap) {
528 int i, j;
529 __uint128_t new;
530 uint8_t byte0, byte1;
531
532 rand = rand * 3 + 1;
533 i = rand % size;
534 rand = rand * 3 + 1;
535 j = rand % size;
536 if (i == j)
537 return old;
538 new = rotate(16, old, i * 8);
539 byte0 = new & 0xff;
540 new &= ~0xff;
541 new = rotate(16, new, -i * 8);
542 new = rotate(16, new, j * 8);
543 byte1 = new & 0xff;
544 new = (new & ~0xff) | byte0;
545 new = rotate(16, new, -j * 8);
546 new = rotate(16, new, i * 8);
547 new = new | byte1;
548 new = rotate(16, new, -i * 8);
549 return new;
550 }
551 rand = rand * 3 + 1;
552 amount = rand % (size * 8);
553 return rotate(size, old, amount);
554}
555
556static bool _cmpxchg(int size, void *target, __uint128_t *old_addr, __uint128_t new)
557{
558 bool ret;
559
560 switch (size) {
561 case 4: {
562 uint32_t old = *old_addr;
563
564 asm volatile ("cs %[old],%[new],%[address]"
565 : [old] "+d" (old),
566 [address] "+Q" (*(uint32_t *)(target))
567 : [new] "d" ((uint32_t)new)
568 : "cc"
569 );
570 ret = old == (uint32_t)*old_addr;
571 *old_addr = old;
572 return ret;
573 }
574 case 8: {
575 uint64_t old = *old_addr;
576
577 asm volatile ("csg %[old],%[new],%[address]"
578 : [old] "+d" (old),
579 [address] "+Q" (*(uint64_t *)(target))
580 : [new] "d" ((uint64_t)new)
581 : "cc"
582 );
583 ret = old == (uint64_t)*old_addr;
584 *old_addr = old;
585 return ret;
586 }
587 case 16: {
588 __uint128_t old = *old_addr;
589
590 asm volatile ("cdsg %[old],%[new],%[address]"
591 : [old] "+d" (old),
592 [address] "+Q" (*(__uint128_t *)(target))
593 : [new] "d" (new)
594 : "cc"
595 );
596 ret = old == *old_addr;
597 *old_addr = old;
598 return ret;
599 }
600 }
601 GUEST_ASSERT_1(false, "Invalid size");
602 return 0;
603}
604
605const unsigned int cmpxchg_iter_outer = 100, cmpxchg_iter_inner = 10000;
606
607static void guest_cmpxchg_key(void)
608{
609 int size, offset;
610 __uint128_t old, new;
611
612 set_storage_key_range(mem1, max_block, 0x10);
613 set_storage_key_range(mem2, max_block, 0x10);
614 GUEST_SYNC(STAGE_SKEYS_SET);
615
616 for (int i = 0; i < cmpxchg_iter_outer; i++) {
617 do {
618 old = 1;
619 } while (!_cmpxchg(16, mem1, &old, 0));
620 for (int j = 0; j < cmpxchg_iter_inner; j++) {
621 choose_block(true, i + j, &size, &offset);
622 do {
623 new = permutate_bits(true, i + j, size, old);
624 } while (!_cmpxchg(size, mem2 + offset, &old, new));
625 }
626 }
627
628 GUEST_SYNC(STAGE_DONE);
629}
630
631static void *run_guest(void *data)
632{
633 struct test_info *info = data;
634
635 HOST_SYNC(*info, STAGE_DONE);
636 return NULL;
637}
638
639static char *quad_to_char(__uint128_t *quad, int size)
640{
641 return ((char *)quad) + (sizeof(*quad) - size);
642}
643
644static void test_cmpxchg_key_concurrent(void)
645{
646 struct test_default t = test_default_init(guest_cmpxchg_key);
647 int size, offset;
648 __uint128_t old, new;
649 bool success;
650 pthread_t thread;
651
652 HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);
653 prepare_mem12();
654 MOP(t.vcpu, LOGICAL, WRITE, mem1, max_block, GADDR_V(mem2));
655 pthread_create(&thread, NULL, run_guest, &t.vcpu);
656
657 for (int i = 0; i < cmpxchg_iter_outer; i++) {
658 do {
659 old = 0;
660 new = 1;
661 MOP(t.vm, ABSOLUTE, CMPXCHG, &new,
662 sizeof(new), GADDR_V(mem1),
663 CMPXCHG_OLD(&old),
664 CMPXCHG_SUCCESS(&success), KEY(1));
665 } while (!success);
666 for (int j = 0; j < cmpxchg_iter_inner; j++) {
667 choose_block(false, i + j, &size, &offset);
668 do {
669 new = permutate_bits(false, i + j, size, old);
670 MOP(t.vm, ABSOLUTE, CMPXCHG, quad_to_char(&new, size),
671 size, GADDR_V(mem2 + offset),
672 CMPXCHG_OLD(quad_to_char(&old, size)),
673 CMPXCHG_SUCCESS(&success), KEY(1));
674 } while (!success);
675 }
676 }
677
678 pthread_join(thread, NULL);
679
680 MOP(t.vcpu, LOGICAL, READ, mem2, max_block, GADDR_V(mem2));
681 TEST_ASSERT(popcount_eq(*(__uint128_t *)mem1, *(__uint128_t *)mem2),
682 "Must retain number of set bits");
380 683
381 kvm_vm_free(t.kvm_vm);684 kvm_vm_free(t.kvm_vm);
382}685}
@@ -409,7 +712,7 @@ static void test_copy_key_storage_prot_override(void) · 26 unmodified lines
409 HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);712 HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);
410 713
411 /* vcpu, mismatching keys, storage protection override in effect */714 /* vcpu, mismatching keys, storage protection override in effect */
412 DEFAULT_WRITE_READ(t.vcpu, t.vcpu, LOGICAL, t.size, KEY(2));715 default_write_read(t.vcpu, t.vcpu, LOGICAL, t.size, 2);
413 716
414 kvm_vm_free(t.kvm_vm);717 kvm_vm_free(t.kvm_vm);
415}718}
@@ -422,8 +725,8 @@ static void test_copy_key_fetch_prot(void) · 6 unmodified lines
422 HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);725 HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);
423 726
424 /* vm/vcpu, matching key, fetch protection in effect */727 /* vm/vcpu, matching key, fetch protection in effect */
425 DEFAULT_READ(t.vcpu, t.vcpu, LOGICAL, t.size, GADDR_V(mem2), KEY(9));728 default_read(t.vcpu, t.vcpu, LOGICAL, t.size, 9);
426 DEFAULT_READ(t.vcpu, t.vm, ABSOLUTE, t.size, GADDR_V(mem2), KEY(9));729 default_read(t.vcpu, t.vm, ABSOLUTE, t.size, 9);
427 730
428 kvm_vm_free(t.kvm_vm);731 kvm_vm_free(t.kvm_vm);
429}732}
@@ -454,9 +757,27 @@ static void test_errors_key(void) · 24 unmodified lines
454 757
455 /* vm/vcpu, mismatching keys, fetch protection in effect */758 /* vm/vcpu, mismatching keys, fetch protection in effect */
456 CHECK_N_DO(ERR_PROT_MOP, t.vcpu, LOGICAL, WRITE, mem1, t.size, GADDR_V(mem1), KEY(2));759 CHECK_N_DO(ERR_PROT_MOP, t.vcpu, LOGICAL, WRITE, mem1, t.size, GADDR_V(mem1), KEY(2));
457 CHECK_N_DO(ERR_PROT_MOP, t.vcpu, LOGICAL, READ, mem2, t.size, GADDR_V(mem2), KEY(2));760 CHECK_N_DO(ERR_PROT_MOP, t.vcpu, LOGICAL, READ, mem2, t.size, GADDR_V(mem1), KEY(2));
458 CHECK_N_DO(ERR_PROT_MOP, t.vm, ABSOLUTE, WRITE, mem1, t.size, GADDR_V(mem1), KEY(2));761 CHECK_N_DO(ERR_PROT_MOP, t.vm, ABSOLUTE, WRITE, mem1, t.size, GADDR_V(mem1), KEY(2));
459 CHECK_N_DO(ERR_PROT_MOP, t.vm, ABSOLUTE, READ, mem2, t.size, GADDR_V(mem2), KEY(2));762 CHECK_N_DO(ERR_PROT_MOP, t.vm, ABSOLUTE, READ, mem2, t.size, GADDR_V(mem1), KEY(2));
763
764 kvm_vm_free(t.kvm_vm);
765}
766
767static void test_errors_cmpxchg_key(void)
768{
769 struct test_default t = test_default_init(guest_copy_key_fetch_prot);
770 int i;
771
772 HOST_SYNC(t.vcpu, STAGE_INITED);
773 HOST_SYNC(t.vcpu, STAGE_SKEYS_SET);
774
775 for (i = 1; i <= 16; i *= 2) {
776 __uint128_t old = 0;
777
778 ERR_PROT_MOP(t.vm, ABSOLUTE, CMPXCHG, mem2, i, GADDR_V(mem2),
779 CMPXCHG_OLD(&old), KEY(2));
780 }
460 781
461 kvm_vm_free(t.kvm_vm);782 kvm_vm_free(t.kvm_vm);
462}783}
@@ -518,7 +839,7 @@ static void guest_copy_key_fetch_prot_override(void) · 55 unmodified lines
518 GUEST_SYNC(STAGE_INITED);839 GUEST_SYNC(STAGE_INITED);
519 set_storage_key_range(0, PAGE_SIZE, 0x18);840 set_storage_key_range(0, PAGE_SIZE, 0x18);
520 set_storage_key_range((void *)last_page_addr, PAGE_SIZE, 0x0);841 set_storage_key_range((void *)last_page_addr, PAGE_SIZE, 0x0);
521 asm volatile ("sske %[key],%[addr]\n" :: [addr] "r"(0), [key] "r"(0x18) : "cc");842 asm volatile ("sske %[key],%[addr]\n" :: [addr] "r"(0L), [key] "r"(0x18) : "cc");
522 GUEST_SYNC(STAGE_SKEYS_SET);843 GUEST_SYNC(STAGE_SKEYS_SET);
523 844
524 for (;;) {845 for (;;) {
@@ -606,7 +927,7 @@ static void test_errors_key_fetch_prot_override_enabled(void) · 81 unmodified lines
606 927
607 /*928 /*
608 * vcpu, mismatching keys on fetch,929 * vcpu, mismatching keys on fetch,
609 * fetch protection override does not apply because memory range acceeded930 * fetch protection override does not apply because memory range exceeded
610 */931 */
611 CHECK_N_DO(ERR_PROT_MOP, t.vcpu, LOGICAL, READ, mem2, 2048 + 1, GADDR_V(0), KEY(2));932 CHECK_N_DO(ERR_PROT_MOP, t.vcpu, LOGICAL, READ, mem2, 2048 + 1, GADDR_V(0), KEY(2));
612 CHECK_N_DO(ERR_PROT_MOP, t.vcpu, LOGICAL, READ, mem2, PAGE_SIZE + 2048 + 1,933 CHECK_N_DO(ERR_PROT_MOP, t.vcpu, LOGICAL, READ, mem2, PAGE_SIZE + 2048 + 1,
@@ -645,7 +966,9 @@ static void _test_errors_common(struct test_info info, enum mop_target target, i · 32 unmodified lines
645 966
646 /* Bad guest address: */967 /* Bad guest address: */
647 rv = ERR_MOP(info, target, WRITE, mem1, size, GADDR((void *)~0xfffUL), CHECK_ONLY);968 rv = ERR_MOP(info, target, WRITE, mem1, size, GADDR((void *)~0xfffUL), CHECK_ONLY);
648 TEST_ASSERT(rv > 0, "ioctl does not report bad guest memory access");969 TEST_ASSERT(rv > 0, "ioctl does not report bad guest memory address with CHECK_ONLY");
970 rv = ERR_MOP(info, target, WRITE, mem1, size, GADDR((void *)~0xfffUL));
971 TEST_ASSERT(rv > 0, "ioctl does not report bad guest memory address on write");
649 972
650 /* Bad host address: */973 /* Bad host address: */
651 rv = ERR_MOP(info, target, WRITE, 0, size, GADDR_V(mem1));974 rv = ERR_MOP(info, target, WRITE, 0, size, GADDR_V(mem1));
@@ -694,85 +1017,138 @@ static void test_errors(void) · 42 unmodified lines
694 kvm_vm_free(t.kvm_vm);1017 kvm_vm_free(t.kvm_vm);
695}1018}
696 1019
697struct testdef {1020static void test_errors_cmpxchg(void)
698 const char *name;1021{
699 void (*test)(void);1022 struct test_default t = test_default_init(guest_idle);
700 int extension;1023 __uint128_t old;
701} testlist[] = {1024 int rv, i, power = 1;
702 {1025
703 .name = "simple copy",1026 HOST_SYNC(t.vcpu, STAGE_INITED);
704 .test = test_copy,1027
705 },1028 for (i = 0; i < 32; i++) {
706 {1029 if (i == power) {
707 .name = "generic error checks",1030 power *= 2;
708 .test = test_errors,1031 continue;
709 },1032 }
710 {1033 rv = ERR_MOP(t.vm, ABSOLUTE, CMPXCHG, mem1, i, GADDR_V(mem1),
711 .name = "copy with storage keys",1034 CMPXCHG_OLD(&old));
712 .test = test_copy_key,1035 TEST_ASSERT(rv == -1 && errno == EINVAL,
713 .extension = 1,1036 "ioctl allows bad size for cmpxchg");
714 },1037 }
715 {1038 for (i = 1; i <= 16; i *= 2) {
716 .name = "copy with key storage protection override",1039 rv = ERR_MOP(t.vm, ABSOLUTE, CMPXCHG, mem1, i, GADDR((void *)~0xfffUL),
717 .test = test_copy_key_storage_prot_override,1040 CMPXCHG_OLD(&old));
718 .extension = 1,1041 TEST_ASSERT(rv > 0, "ioctl allows bad guest address for cmpxchg");
719 },1042 }
720 {1043 for (i = 2; i <= 16; i *= 2) {
721 .name = "copy with key fetch protection",1044 rv = ERR_MOP(t.vm, ABSOLUTE, CMPXCHG, mem1, i, GADDR_V(mem1 + 1),
722 .test = test_copy_key_fetch_prot,1045 CMPXCHG_OLD(&old));
723 .extension = 1,1046 TEST_ASSERT(rv == -1 && errno == EINVAL,
724 },1047 "ioctl allows bad alignment for cmpxchg");
725 {1048 }
726 .name = "copy with key fetch protection override",1049
727 .test = test_copy_key_fetch_prot_override,1050 kvm_vm_free(t.kvm_vm);
728 .extension = 1,1051}
729 },
730 {
731 .name = "error checks with key",
732 .test = test_errors_key,
733 .extension = 1,
734 },
735 {
736 .name = "termination",
737 .test = test_termination,
738 .extension = 1,
739 },
740 {
741 .name = "error checks with key storage protection override",
742 .test = test_errors_key_storage_prot_override,
743 .extension = 1,
744 },
745 {
746 .name = "error checks without key fetch prot override",
747 .test = test_errors_key_fetch_prot_override_not_enabled,
748 .extension = 1,
749 },
750 {
751 .name = "error checks with key fetch prot override",
752 .test = test_errors_key_fetch_prot_override_enabled,
753 .extension = 1,
754 },
755};
756 1052
757int main(int argc, char *argv[])1053int main(int argc, char *argv[])
758{1054{
759 int extension_cap, idx;1055 int extension_cap, idx;
760 1056
761 TEST_REQUIRE(kvm_has_cap(KVM_CAP_S390_MEM_OP));1057 TEST_REQUIRE(kvm_has_cap(KVM_CAP_S390_MEM_OP));
1058 extension_cap = kvm_check_cap(KVM_CAP_S390_MEM_OP_EXTENSION);
762 1059
763 ksft_print_header();1060 struct testdef {
1061 const char *name;
1062 void (*test)(void);
1063 bool requirements_met;
1064 } testlist[] = {
1065 {
1066 .name = "simple copy",
1067 .test = test_copy,
1068 .requirements_met = true,
1069 },
1070 {
1071 .name = "generic error checks",
1072 .test = test_errors,
1073 .requirements_met = true,
1074 },
1075 {
1076 .name = "copy with storage keys",
1077 .test = test_copy_key,
1078 .requirements_met = extension_cap > 0,
1079 },
1080 {
1081 .name = "cmpxchg with storage keys",
1082 .test = test_cmpxchg_key,
1083 .requirements_met = extension_cap & 0x2,
1084 },
1085 {
1086 .name = "concurrently cmpxchg with storage keys",
1087 .test = test_cmpxchg_key_concurrent,
1088 .requirements_met = extension_cap & 0x2,
1089 },
1090 {
1091 .name = "copy with key storage protection override",
1092 .test = test_copy_key_storage_prot_override,
1093 .requirements_met = extension_cap > 0,
1094 },
1095 {
1096 .name = "copy with key fetch protection",
1097 .test = test_copy_key_fetch_prot,
1098 .requirements_met = extension_cap > 0,
1099 },
1100 {
1101 .name = "copy with key fetch protection override",
1102 .test = test_copy_key_fetch_prot_override,
1103 .requirements_met = extension_cap > 0,
1104 },
1105 {
1106 .name = "error checks with key",
1107 .test = test_errors_key,
1108 .requirements_met = extension_cap > 0,
1109 },
1110 {
1111 .name = "error checks for cmpxchg with key",
1112 .test = test_errors_cmpxchg_key,
1113 .requirements_met = extension_cap & 0x2,
1114 },
1115 {
1116 .name = "error checks for cmpxchg",
1117 .test = test_errors_cmpxchg,
1118 .requirements_met = extension_cap & 0x2,
1119 },
1120 {
1121 .name = "termination",
1122 .test = test_termination,
1123 .requirements_met = extension_cap > 0,
1124 },
1125 {
1126 .name = "error checks with key storage protection override",
1127 .test = test_errors_key_storage_prot_override,
1128 .requirements_met = extension_cap > 0,
1129 },
1130 {
1131 .name = "error checks without key fetch prot override",
1132 .test = test_errors_key_fetch_prot_override_not_enabled,
1133 .requirements_met = extension_cap > 0,
1134 },
1135 {
1136 .name = "error checks with key fetch prot override",
1137 .test = test_errors_key_fetch_prot_override_enabled,
1138 .requirements_met = extension_cap > 0,
1139 },
1140 };
764 1141
1142 ksft_print_header();
765 ksft_set_plan(ARRAY_SIZE(testlist));1143 ksft_set_plan(ARRAY_SIZE(testlist));
766 1144
767 extension_cap = kvm_check_cap(KVM_CAP_S390_MEM_OP_EXTENSION);
768 for (idx = 0; idx < ARRAY_SIZE(testlist); idx++) {1145 for (idx = 0; idx < ARRAY_SIZE(testlist); idx++) {
769 if (extension_cap >= testlist[idx].extension) {1146 if (testlist[idx].requirements_met) {
770 testlist[idx].test();1147 testlist[idx].test();
771 ksft_test_result_pass("%s\n", testlist[idx].name);1148 ksft_test_result_pass("%s\n", testlist[idx].name);
772 } else {1149 } else {
773 ksft_test_result_skip("%s - extension level %d not supported\n",1150 ksft_test_result_skip("%s - requirements not met (kernel has extension cap %#x)\n",
774 testlist[idx].name,1151 testlist[idx].name, extension_cap);
775 testlist[idx].extension);
776 }1152 }
777 }1153 }
778 1154
tools/testing/selftests/kvm/x86_64/exit_on_emulation_failure_test.c+0−3
@@ -26,9 +26,6 @@ int main(int argc, char *argv[]) · 25 unmodified lines
26 struct kvm_vcpu *vcpu;26 struct kvm_vcpu *vcpu;
27 struct kvm_vm *vm;27 struct kvm_vm *vm;
28 28
29 /* Tell stdout not to buffer its content */
30 setbuf(stdout, NULL);
31
32 TEST_REQUIRE(kvm_has_cap(KVM_CAP_EXIT_ON_EMULATION_FAILURE));29 TEST_REQUIRE(kvm_has_cap(KVM_CAP_EXIT_ON_EMULATION_FAILURE));
33 30
34 vm = vm_create_with_one_vcpu(&vcpu, guest_code);31 vm = vm_create_with_one_vcpu(&vcpu, guest_code);
tools/testing/selftests/kvm/x86_64/fix_hypercall_test.c+2−2
@@ -48,10 +48,10 @@ static void guest_main(void) · 47 unmodified lines
48 const uint8_t *other_hypercall_insn;48 const uint8_t *other_hypercall_insn;
49 uint64_t ret;49 uint64_t ret;
50 50
51 if (is_intel_cpu()) {51 if (host_cpu_is_intel) {
52 native_hypercall_insn = vmx_vmcall;52 native_hypercall_insn = vmx_vmcall;
53 other_hypercall_insn = svm_vmmcall;53 other_hypercall_insn = svm_vmmcall;
54 } else if (is_amd_cpu()) {54 } else if (host_cpu_is_amd) {
55 native_hypercall_insn = svm_vmmcall;55 native_hypercall_insn = svm_vmmcall;
56 other_hypercall_insn = vmx_vmcall;56 other_hypercall_insn = vmx_vmcall;
57 } else {57 } else {
tools/testing/selftests/kvm/x86_64/hyperv_clock.c+1−1
@@ -104,7 +104,7 @@ static void guest_main(struct ms_hyperv_tsc_page *tsc_page, vm_paddr_t tsc_page_ · 103 unmodified lines
104 104
105 /* Set Guest OS id to enable Hyper-V emulation */105 /* Set Guest OS id to enable Hyper-V emulation */
106 GUEST_SYNC(1);106 GUEST_SYNC(1);
107 wrmsr(HV_X64_MSR_GUEST_OS_ID, (u64)0x8100 << 48);107 wrmsr(HV_X64_MSR_GUEST_OS_ID, HYPERV_LINUX_OS_ID);
108 GUEST_SYNC(2);108 GUEST_SYNC(2);
109 109
110 check_tsc_msr_rdtsc();110 check_tsc_msr_rdtsc();
tools/testing/selftests/kvm/x86_64/hyperv_extended_hypercalls.c+97−0
@@ -0,0 +1,97 @@
1// SPDX-License-Identifier: GPL-2.0-only
2/*
3 * Test Hyper-V extended hypercall, HV_EXT_CALL_QUERY_CAPABILITIES (0x8001),
4 * exit to userspace and receive result in guest.
5 *
6 * Negative tests are present in hyperv_features.c
7 *
8 * Copyright 2022 Google LLC
9 * Author: Vipin Sharma <vipinsh@google.com>
10 */
11
12#include "kvm_util.h"
13#include "processor.h"
14#include "hyperv.h"
15
16/* Any value is fine */
17#define EXT_CAPABILITIES 0xbull
18
19static void guest_code(vm_paddr_t in_pg_gpa, vm_paddr_t out_pg_gpa,
20 vm_vaddr_t out_pg_gva)
21{
22 uint64_t *output_gva;
23
24 wrmsr(HV_X64_MSR_GUEST_OS_ID, HYPERV_LINUX_OS_ID);
25 wrmsr(HV_X64_MSR_HYPERCALL, in_pg_gpa);
26
27 output_gva = (uint64_t *)out_pg_gva;
28
29 hyperv_hypercall(HV_EXT_CALL_QUERY_CAPABILITIES, in_pg_gpa, out_pg_gpa);
30
31 /* TLFS states output will be a uint64_t value */
32 GUEST_ASSERT_EQ(*output_gva, EXT_CAPABILITIES);
33
34 GUEST_DONE();
35}
36
37int main(void)
38{
39 vm_vaddr_t hcall_out_page;
40 vm_vaddr_t hcall_in_page;
41 struct kvm_vcpu *vcpu;
42 struct kvm_run *run;
43 struct kvm_vm *vm;
44 uint64_t *outval;
45 struct ucall uc;
46
47 /* Verify if extended hypercalls are supported */
48 if (!kvm_cpuid_has(kvm_get_supported_hv_cpuid(),
49 HV_ENABLE_EXTENDED_HYPERCALLS)) {
50 print_skip("Extended calls not supported by the kernel");
51 exit(KSFT_SKIP);
52 }
53
54 vm = vm_create_with_one_vcpu(&vcpu, guest_code);
55 run = vcpu->run;
56 vcpu_set_hv_cpuid(vcpu);
57
58 /* Hypercall input */
59 hcall_in_page = vm_vaddr_alloc_pages(vm, 1);
60 memset(addr_gva2hva(vm, hcall_in_page), 0x0, vm->page_size);
61
62 /* Hypercall output */
63 hcall_out_page = vm_vaddr_alloc_pages(vm, 1);
64 memset(addr_gva2hva(vm, hcall_out_page), 0x0, vm->page_size);
65
66 vcpu_args_set(vcpu, 3, addr_gva2gpa(vm, hcall_in_page),
67 addr_gva2gpa(vm, hcall_out_page), hcall_out_page);
68
69 vcpu_run(vcpu);
70
71 TEST_ASSERT(run->exit_reason == KVM_EXIT_HYPERV,
72 "Unexpected exit reason: %u (%s)",
73 run->exit_reason, exit_reason_str(run->exit_reason));
74
75 outval = addr_gpa2hva(vm, run->hyperv.u.hcall.params[1]);
76 *outval = EXT_CAPABILITIES;
77 run->hyperv.u.hcall.result = HV_STATUS_SUCCESS;
78
79 vcpu_run(vcpu);
80
81 TEST_ASSERT(run->exit_reason == KVM_EXIT_IO,
82 "Unexpected exit reason: %u (%s)",
83 run->exit_reason, exit_reason_str(run->exit_reason));
84
85 switch (get_ucall(vcpu, &uc)) {
86 case UCALL_ABORT:
87 REPORT_GUEST_ASSERT_2(uc, "arg1 = %ld, arg2 = %ld");
88 break;
89 case UCALL_DONE:
90 break;
91 default:
92 TEST_FAIL("Unhandled ucall: %ld", uc.cmd);
93 }
94
95 kvm_vm_free(vm);
96 return 0;
97}
tools/testing/selftests/kvm/x86_64/hyperv_features.c+207−132
@@ -13,9 +13,17 @@ · 12 unmodified lines
13#include "processor.h"13#include "processor.h"
14#include "hyperv.h"14#include "hyperv.h"
15 15
16/*
17 * HYPERV_CPUID_ENLIGHTMENT_INFO.EBX is not a 'feature' CPUID leaf
18 * but to activate the feature it is sufficient to set it to a non-zero
19 * value. Use BIT(0) for that.
20 */
21#define HV_PV_SPINLOCKS_TEST \
22 KVM_X86_CPU_FEATURE(HYPERV_CPUID_ENLIGHTMENT_INFO, 0, EBX, 0)
23
16struct msr_data {24struct msr_data {
17 uint32_t idx;25 uint32_t idx;
18 bool available;26 bool fault_expected;
19 bool write;27 bool write;
20 u64 write_val;28 u64 write_val;
21};29};
@@ -26,22 +34,46 @@ struct hcall_data { · 4 unmodified lines
26 bool ud_expected;34 bool ud_expected;
27};35};
28 36
37static bool is_write_only_msr(uint32_t msr)
38{
39 return msr == HV_X64_MSR_EOI;
40}
41
29static void guest_msr(struct msr_data *msr)42static void guest_msr(struct msr_data *msr)
30{43{
31 uint64_t ignored;44 uint8_t vector = 0;
32 uint8_t vector;45 uint64_t msr_val = 0;
33 46
34 GUEST_ASSERT(msr->idx);47 GUEST_ASSERT(msr->idx);
35 48
36 if (!msr->write)49 if (msr->write)
37 vector = rdmsr_safe(msr->idx, &ignored);
38 else
39 vector = wrmsr_safe(msr->idx, msr->write_val);50 vector = wrmsr_safe(msr->idx, msr->write_val);
40 51
41 if (msr->available)52 if (!vector && (!msr->write || !is_write_only_msr(msr->idx)))
42 GUEST_ASSERT_2(!vector, msr->idx, vector);53 vector = rdmsr_safe(msr->idx, &msr_val);
54
55 if (msr->fault_expected)
56 GUEST_ASSERT_3(vector == GP_VECTOR, msr->idx, vector, GP_VECTOR);
43 else57 else
44 GUEST_ASSERT_2(vector == GP_VECTOR, msr->idx, vector);58 GUEST_ASSERT_3(!vector, msr->idx, vector, 0);
59
60 if (vector || is_write_only_msr(msr->idx))
61 goto done;
62
63 if (msr->write)
64 GUEST_ASSERT_3(msr_val == msr->write_val, msr->idx,
65 msr_val, msr->write_val);
66
67 /* Invariant TSC bit appears when TSC invariant control MSR is written to */
68 if (msr->idx == HV_X64_MSR_TSC_INVARIANT_CONTROL) {
69 if (!this_cpu_has(HV_ACCESS_TSC_INVARIANT))
70 GUEST_ASSERT(this_cpu_has(X86_FEATURE_INVTSC));
71 else
72 GUEST_ASSERT(this_cpu_has(X86_FEATURE_INVTSC) ==
73 !!(msr_val & HV_INVARIANT_TSC_EXPOSED));
74 }
75
76done:
45 GUEST_DONE();77 GUEST_DONE();
46}78}
47 79
@@ -89,7 +121,6 @@ static void vcpu_reset_hv_cpuid(struct kvm_vcpu *vcpu) · 41 unmodified lines
89static void guest_test_msrs_access(void)121static void guest_test_msrs_access(void)
90{122{
91 struct kvm_cpuid2 *prev_cpuid = NULL;123 struct kvm_cpuid2 *prev_cpuid = NULL;
92 struct kvm_cpuid_entry2 *feat, *dbg;
93 struct kvm_vcpu *vcpu;124 struct kvm_vcpu *vcpu;
94 struct kvm_run *run;125 struct kvm_run *run;
95 struct kvm_vm *vm;126 struct kvm_vm *vm;
@@ -97,6 +128,7 @@ static void guest_test_msrs_access(void) · 1 unmodified line
97 int stage = 0;128 int stage = 0;
98 vm_vaddr_t msr_gva;129 vm_vaddr_t msr_gva;
99 struct msr_data *msr;130 struct msr_data *msr;
131 bool has_invtsc = kvm_cpu_has(X86_FEATURE_INVTSC);
100 132
101 while (true) {133 while (true) {
102 vm = vm_create_with_one_vcpu(&vcpu, guest_msr);134 vm = vm_create_with_one_vcpu(&vcpu, guest_msr);
@@ -116,9 +148,6 @@ static void guest_test_msrs_access(void) · 13 unmodified lines
116 vcpu_init_cpuid(vcpu, prev_cpuid);148 vcpu_init_cpuid(vcpu, prev_cpuid);
117 }149 }
118 150
119 feat = vcpu_get_cpuid_entry(vcpu, HYPERV_CPUID_FEATURES);
120 dbg = vcpu_get_cpuid_entry(vcpu, HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES);
121
122 vm_init_descriptor_tables(vm);151 vm_init_descriptor_tables(vm);
123 vcpu_init_descriptor_tables(vcpu);152 vcpu_init_descriptor_tables(vcpu);
124 153
@@ -134,133 +163,139 @@ static void guest_test_msrs_access(void) · 9 unmodified lines
134 * Only available when Hyper-V identification is set163 * Only available when Hyper-V identification is set
135 */164 */
136 msr->idx = HV_X64_MSR_GUEST_OS_ID;165 msr->idx = HV_X64_MSR_GUEST_OS_ID;
137 msr->write = 0;166 msr->write = false;
138 msr->available = 0;167 msr->fault_expected = true;
139 break;168 break;
140 case 1:169 case 1:
141 msr->idx = HV_X64_MSR_HYPERCALL;170 msr->idx = HV_X64_MSR_HYPERCALL;
142 msr->write = 0;171 msr->write = false;
143 msr->available = 0;172 msr->fault_expected = true;
144 break;173 break;
145 case 2:174 case 2:
146 feat->eax |= HV_MSR_HYPERCALL_AVAILABLE;175 vcpu_set_cpuid_feature(vcpu, HV_MSR_HYPERCALL_AVAILABLE);
147 /*176 /*
148 * HV_X64_MSR_GUEST_OS_ID has to be written first to make177 * HV_X64_MSR_GUEST_OS_ID has to be written first to make
149 * HV_X64_MSR_HYPERCALL available.178 * HV_X64_MSR_HYPERCALL available.
150 */179 */
151 msr->idx = HV_X64_MSR_GUEST_OS_ID;180 msr->idx = HV_X64_MSR_GUEST_OS_ID;
152 msr->write = 1;181 msr->write = true;
153 msr->write_val = HYPERV_LINUX_OS_ID;182 msr->write_val = HYPERV_LINUX_OS_ID;
154 msr->available = 1;183 msr->fault_expected = false;
155 break;184 break;
156 case 3:185 case 3:
157 msr->idx = HV_X64_MSR_GUEST_OS_ID;186 msr->idx = HV_X64_MSR_GUEST_OS_ID;
158 msr->write = 0;187 msr->write = false;
159 msr->available = 1;188 msr->fault_expected = false;
160 break;189 break;
161 case 4:190 case 4:
162 msr->idx = HV_X64_MSR_HYPERCALL;191 msr->idx = HV_X64_MSR_HYPERCALL;
163 msr->write = 0;192 msr->write = false;
164 msr->available = 1;193 msr->fault_expected = false;
165 break;194 break;
166 195
167 case 5:196 case 5:
168 msr->idx = HV_X64_MSR_VP_RUNTIME;197 msr->idx = HV_X64_MSR_VP_RUNTIME;
169 msr->write = 0;198 msr->write = false;
170 msr->available = 0;199 msr->fault_expected = true;
171 break;200 break;
172 case 6:201 case 6:
173 feat->eax |= HV_MSR_VP_RUNTIME_AVAILABLE;202 vcpu_set_cpuid_feature(vcpu, HV_MSR_VP_RUNTIME_AVAILABLE);
174 msr->idx = HV_X64_MSR_VP_RUNTIME;203 msr->idx = HV_X64_MSR_VP_RUNTIME;
175 msr->write = 0;204 msr->write = false;
176 msr->available = 1;205 msr->fault_expected = false;
177 break;206 break;
178 case 7:207 case 7:
179 /* Read only */208 /* Read only */
180 msr->idx = HV_X64_MSR_VP_RUNTIME;209 msr->idx = HV_X64_MSR_VP_RUNTIME;
181 msr->write = 1;210 msr->write = true;
182 msr->write_val = 1;211 msr->write_val = 1;
183 msr->available = 0;212 msr->fault_expected = true;
184 break;213 break;
185 214
186 case 8:215 case 8:
187 msr->idx = HV_X64_MSR_TIME_REF_COUNT;216 msr->idx = HV_X64_MSR_TIME_REF_COUNT;
188 msr->write = 0;217 msr->write = false;
189 msr->available = 0;218 msr->fault_expected = true;
190 break;219 break;
191 case 9:220 case 9:
192 feat->eax |= HV_MSR_TIME_REF_COUNT_AVAILABLE;221 vcpu_set_cpuid_feature(vcpu, HV_MSR_TIME_REF_COUNT_AVAILABLE);
193 msr->idx = HV_X64_MSR_TIME_REF_COUNT;222 msr->idx = HV_X64_MSR_TIME_REF_COUNT;
194 msr->write = 0;223 msr->write = false;
195 msr->available = 1;224 msr->fault_expected = false;
196 break;225 break;
197 case 10:226 case 10:
198 /* Read only */227 /* Read only */
199 msr->idx = HV_X64_MSR_TIME_REF_COUNT;228 msr->idx = HV_X64_MSR_TIME_REF_COUNT;
200 msr->write = 1;229 msr->write = true;
201 msr->write_val = 1;230 msr->write_val = 1;
202 msr->available = 0;231 msr->fault_expected = true;
203 break;232 break;
204 233
205 case 11:234 case 11:
206 msr->idx = HV_X64_MSR_VP_INDEX;235 msr->idx = HV_X64_MSR_VP_INDEX;
207 msr->write = 0;236 msr->write = false;
208 msr->available = 0;237 msr->fault_expected = true;
209 break;238 break;
210 case 12:239 case 12:
211 feat->eax |= HV_MSR_VP_INDEX_AVAILABLE;240 vcpu_set_cpuid_feature(vcpu, HV_MSR_VP_INDEX_AVAILABLE);
212 msr->idx = HV_X64_MSR_VP_INDEX;241 msr->idx = HV_X64_MSR_VP_INDEX;
213 msr->write = 0;242 msr->write = false;
214 msr->available = 1;243 msr->fault_expected = false;
215 break;244 break;
216 case 13:245 case 13:
217 /* Read only */246 /* Read only */
218 msr->idx = HV_X64_MSR_VP_INDEX;247 msr->idx = HV_X64_MSR_VP_INDEX;
219 msr->write = 1;248 msr->write = true;
220 msr->write_val = 1;249 msr->write_val = 1;
221 msr->available = 0;250 msr->fault_expected = true;
222 break;251 break;
223 252
224 case 14:253 case 14:
225 msr->idx = HV_X64_MSR_RESET;254 msr->idx = HV_X64_MSR_RESET;
226 msr->write = 0;255 msr->write = false;
227 msr->available = 0;256 msr->fault_expected = true;
228 break;257 break;
229 case 15:258 case 15:
230 feat->eax |= HV_MSR_RESET_AVAILABLE;259 vcpu_set_cpuid_feature(vcpu, HV_MSR_RESET_AVAILABLE);
231 msr->idx = HV_X64_MSR_RESET;260 msr->idx = HV_X64_MSR_RESET;
232 msr->write = 0;261 msr->write = false;
233 msr->available = 1;262 msr->fault_expected = false;
234 break;263 break;
235 case 16:264 case 16:
236 msr->idx = HV_X64_MSR_RESET;265 msr->idx = HV_X64_MSR_RESET;
237 msr->write = 1;266 msr->write = true;
267 /*
268 * TODO: the test only writes '0' to HV_X64_MSR_RESET
269 * at the moment, writing some other value there will
270 * trigger real vCPU reset and the code is not prepared
271 * to handle it yet.
272 */
238 msr->write_val = 0;273 msr->write_val = 0;
239 msr->available = 1;274 msr->fault_expected = false;
240 break;275 break;
241 276
242 case 17:277 case 17:
243 msr->idx = HV_X64_MSR_REFERENCE_TSC;278 msr->idx = HV_X64_MSR_REFERENCE_TSC;
244 msr->write = 0;279 msr->write = false;
245 msr->available = 0;280 msr->fault_expected = true;
246 break;281 break;
247 case 18:282 case 18:
248 feat->eax |= HV_MSR_REFERENCE_TSC_AVAILABLE;283 vcpu_set_cpuid_feature(vcpu, HV_MSR_REFERENCE_TSC_AVAILABLE);
249 msr->idx = HV_X64_MSR_REFERENCE_TSC;284 msr->idx = HV_X64_MSR_REFERENCE_TSC;
250 msr->write = 0;285 msr->write = false;
251 msr->available = 1;286 msr->fault_expected = false;
252 break;287 break;
253 case 19:288 case 19:
254 msr->idx = HV_X64_MSR_REFERENCE_TSC;289 msr->idx = HV_X64_MSR_REFERENCE_TSC;
255 msr->write = 1;290 msr->write = true;
256 msr->write_val = 0;291 msr->write_val = 0;
257 msr->available = 1;292 msr->fault_expected = false;
258 break;293 break;
259 294
260 case 20:295 case 20:
261 msr->idx = HV_X64_MSR_EOM;296 msr->idx = HV_X64_MSR_EOM;
262 msr->write = 0;297 msr->write = false;
263 msr->available = 0;298 msr->fault_expected = true;
264 break;299 break;
265 case 21:300 case 21:
266 /*301 /*
@@ -268,149 +303,185 @@ static void guest_test_msrs_access(void) · 1 unmodified line
268 * capability enabled and guest visible CPUID bit unset.303 * capability enabled and guest visible CPUID bit unset.
269 */304 */
270 msr->idx = HV_X64_MSR_EOM;305 msr->idx = HV_X64_MSR_EOM;
271 msr->write = 0;306 msr->write = false;
272 msr->available = 0;307 msr->fault_expected = true;
273 break;308 break;
274 case 22:309 case 22:
275 feat->eax |= HV_MSR_SYNIC_AVAILABLE;310 vcpu_set_cpuid_feature(vcpu, HV_MSR_SYNIC_AVAILABLE);
276 msr->idx = HV_X64_MSR_EOM;311 msr->idx = HV_X64_MSR_EOM;
277 msr->write = 0;312 msr->write = false;
278 msr->available = 1;313 msr->fault_expected = false;
279 break;314 break;
280 case 23:315 case 23:
281 msr->idx = HV_X64_MSR_EOM;316 msr->idx = HV_X64_MSR_EOM;
282 msr->write = 1;317 msr->write = true;
283 msr->write_val = 0;318 msr->write_val = 0;
284 msr->available = 1;319 msr->fault_expected = false;
285 break;320 break;
286 321
287 case 24:322 case 24:
288 msr->idx = HV_X64_MSR_STIMER0_CONFIG;323 msr->idx = HV_X64_MSR_STIMER0_CONFIG;
289 msr->write = 0;324 msr->write = false;
290 msr->available = 0;325 msr->fault_expected = true;
291 break;326 break;
292 case 25:327 case 25:
293 feat->eax |= HV_MSR_SYNTIMER_AVAILABLE;328 vcpu_set_cpuid_feature(vcpu, HV_MSR_SYNTIMER_AVAILABLE);
294 msr->idx = HV_X64_MSR_STIMER0_CONFIG;329 msr->idx = HV_X64_MSR_STIMER0_CONFIG;
295 msr->write = 0;330 msr->write = false;
296 msr->available = 1;331 msr->fault_expected = false;
297 break;332 break;
298 case 26:333 case 26:
299 msr->idx = HV_X64_MSR_STIMER0_CONFIG;334 msr->idx = HV_X64_MSR_STIMER0_CONFIG;
300 msr->write = 1;335 msr->write = true;
301 msr->write_val = 0;336 msr->write_val = 0;
302 msr->available = 1;337 msr->fault_expected = false;
303 break;338 break;
304 case 27:339 case 27:
305 /* Direct mode test */340 /* Direct mode test */
306 msr->idx = HV_X64_MSR_STIMER0_CONFIG;341 msr->idx = HV_X64_MSR_STIMER0_CONFIG;
307 msr->write = 1;342 msr->write = true;
308 msr->write_val = 1 << 12;343 msr->write_val = 1 << 12;
309 msr->available = 0;344 msr->fault_expected = true;
310 break;345 break;
311 case 28:346 case 28:
312 feat->edx |= HV_STIMER_DIRECT_MODE_AVAILABLE;347 vcpu_set_cpuid_feature(vcpu, HV_STIMER_DIRECT_MODE_AVAILABLE);
313 msr->idx = HV_X64_MSR_STIMER0_CONFIG;348 msr->idx = HV_X64_MSR_STIMER0_CONFIG;
314 msr->write = 1;349 msr->write = true;
315 msr->write_val = 1 << 12;350 msr->write_val = 1 << 12;
316 msr->available = 1;351 msr->fault_expected = false;
317 break;352 break;
318 353
319 case 29:354 case 29:
320 msr->idx = HV_X64_MSR_EOI;355 msr->idx = HV_X64_MSR_EOI;
321 msr->write = 0;356 msr->write = false;
322 msr->available = 0;357 msr->fault_expected = true;
323 break;358 break;
324 case 30:359 case 30:
325 feat->eax |= HV_MSR_APIC_ACCESS_AVAILABLE;360 vcpu_set_cpuid_feature(vcpu, HV_MSR_APIC_ACCESS_AVAILABLE);
326 msr->idx = HV_X64_MSR_EOI;361 msr->idx = HV_X64_MSR_EOI;
327 msr->write = 1;362 msr->write = true;
328 msr->write_val = 1;363 msr->write_val = 1;
329 msr->available = 1;364 msr->fault_expected = false;
330 break;365 break;
331 366
332 case 31:367 case 31:
333 msr->idx = HV_X64_MSR_TSC_FREQUENCY;368 msr->idx = HV_X64_MSR_TSC_FREQUENCY;
334 msr->write = 0;369 msr->write = false;
335 msr->available = 0;370 msr->fault_expected = true;
336 break;371 break;
337 case 32:372 case 32:
338 feat->eax |= HV_ACCESS_FREQUENCY_MSRS;373 vcpu_set_cpuid_feature(vcpu, HV_ACCESS_FREQUENCY_MSRS);
339 msr->idx = HV_X64_MSR_TSC_FREQUENCY;374 msr->idx = HV_X64_MSR_TSC_FREQUENCY;
340 msr->write = 0;375 msr->write = false;
341 msr->available = 1;376 msr->fault_expected = false;
342 break;377 break;
343 case 33:378 case 33:
344 /* Read only */379 /* Read only */
345 msr->idx = HV_X64_MSR_TSC_FREQUENCY;380 msr->idx = HV_X64_MSR_TSC_FREQUENCY;
346 msr->write = 1;381 msr->write = true;
347 msr->write_val = 1;382 msr->write_val = 1;
348 msr->available = 0;383 msr->fault_expected = true;
349 break;384 break;
350 385
351 case 34:386 case 34:
352 msr->idx = HV_X64_MSR_REENLIGHTENMENT_CONTROL;387 msr->idx = HV_X64_MSR_REENLIGHTENMENT_CONTROL;
353 msr->write = 0;388 msr->write = false;
354 msr->available = 0;389 msr->fault_expected = true;
355 break;390 break;
356 case 35:391 case 35:
357 feat->eax |= HV_ACCESS_REENLIGHTENMENT;392 vcpu_set_cpuid_feature(vcpu, HV_ACCESS_REENLIGHTENMENT);
358 msr->idx = HV_X64_MSR_REENLIGHTENMENT_CONTROL;393 msr->idx = HV_X64_MSR_REENLIGHTENMENT_CONTROL;
359 msr->write = 0;394 msr->write = false;
360 msr->available = 1;395 msr->fault_expected = false;
361 break;396 break;
362 case 36:397 case 36:
363 msr->idx = HV_X64_MSR_REENLIGHTENMENT_CONTROL;398 msr->idx = HV_X64_MSR_REENLIGHTENMENT_CONTROL;
364 msr->write = 1;399 msr->write = true;
365 msr->write_val = 1;400 msr->write_val = 1;
366 msr->available = 1;401 msr->fault_expected = false;
367 break;402 break;
368 case 37:403 case 37:
369 /* Can only write '0' */404 /* Can only write '0' */
370 msr->idx = HV_X64_MSR_TSC_EMULATION_STATUS;405 msr->idx = HV_X64_MSR_TSC_EMULATION_STATUS;
371 msr->write = 1;406 msr->write = true;
372 msr->write_val = 1;407 msr->write_val = 1;
373 msr->available = 0;408 msr->fault_expected = true;
374 break;409 break;
375 410
376 case 38:411 case 38:
377 msr->idx = HV_X64_MSR_CRASH_P0;412 msr->idx = HV_X64_MSR_CRASH_P0;
378 msr->write = 0;413 msr->write = false;
379 msr->available = 0;414 msr->fault_expected = true;
380 break;415 break;
381 case 39:416 case 39:
382 feat->edx |= HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE;417 vcpu_set_cpuid_feature(vcpu, HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE);
383 msr->idx = HV_X64_MSR_CRASH_P0;418 msr->idx = HV_X64_MSR_CRASH_P0;
384 msr->write = 0;419 msr->write = false;
385 msr->available = 1;420 msr->fault_expected = false;
386 break;421 break;
387 case 40:422 case 40:
388 msr->idx = HV_X64_MSR_CRASH_P0;423 msr->idx = HV_X64_MSR_CRASH_P0;
389 msr->write = 1;424 msr->write = true;
390 msr->write_val = 1;425 msr->write_val = 1;
391 msr->available = 1;426 msr->fault_expected = false;
392 break;427 break;
393 428
394 case 41:429 case 41:
395 msr->idx = HV_X64_MSR_SYNDBG_STATUS;430 msr->idx = HV_X64_MSR_SYNDBG_STATUS;
396 msr->write = 0;431 msr->write = false;
397 msr->available = 0;432 msr->fault_expected = true;
398 break;433 break;
399 case 42:434 case 42:
400 feat->edx |= HV_FEATURE_DEBUG_MSRS_AVAILABLE;435 vcpu_set_cpuid_feature(vcpu, HV_FEATURE_DEBUG_MSRS_AVAILABLE);
401 dbg->eax |= HV_X64_SYNDBG_CAP_ALLOW_KERNEL_DEBUGGING;436 vcpu_set_cpuid_feature(vcpu, HV_X64_SYNDBG_CAP_ALLOW_KERNEL_DEBUGGING);
402 msr->idx = HV_X64_MSR_SYNDBG_STATUS;437 msr->idx = HV_X64_MSR_SYNDBG_STATUS;
403 msr->write = 0;438 msr->write = false;
404 msr->available = 1;439 msr->fault_expected = false;
405 break;440 break;
406 case 43:441 case 43:
407 msr->idx = HV_X64_MSR_SYNDBG_STATUS;442 msr->idx = HV_X64_MSR_SYNDBG_STATUS;
408 msr->write = 1;443 msr->write = true;
409 msr->write_val = 0;444 msr->write_val = 0;
410 msr->available = 1;445 msr->fault_expected = false;
411 break;446 break;
412 447
413 case 44:448 case 44:
449 /* MSR is not available when CPUID feature bit is unset */
450 if (!has_invtsc)
451 continue;
452 msr->idx = HV_X64_MSR_TSC_INVARIANT_CONTROL;
453 msr->write = false;
454 msr->fault_expected = true;
455 break;
456 case 45:
457 /* MSR is vailable when CPUID feature bit is set */
458 if (!has_invtsc)
459 continue;
460 vcpu_set_cpuid_feature(vcpu, HV_ACCESS_TSC_INVARIANT);
461 msr->idx = HV_X64_MSR_TSC_INVARIANT_CONTROL;
462 msr->write = false;
463 msr->fault_expected = false;
464 break;
465 case 46:
466 /* Writing bits other than 0 is forbidden */
467 if (!has_invtsc)
468 continue;
469 msr->idx = HV_X64_MSR_TSC_INVARIANT_CONTROL;
470 msr->write = true;
471 msr->write_val = 0xdeadbeef;
472 msr->fault_expected = true;
473 break;
474 case 47:
475 /* Setting bit 0 enables the feature */
476 if (!has_invtsc)
477 continue;
478 msr->idx = HV_X64_MSR_TSC_INVARIANT_CONTROL;
479 msr->write = true;
480 msr->write_val = 1;
481 msr->fault_expected = false;
482 break;
483
484 default:
414 kvm_vm_free(vm);485 kvm_vm_free(vm);
415 return;486 return;
416 }487 }
@@ -429,7 +500,7 @@ static void guest_test_msrs_access(void) · 12 unmodified lines
429 500
430 switch (get_ucall(vcpu, &uc)) {501 switch (get_ucall(vcpu, &uc)) {
431 case UCALL_ABORT:502 case UCALL_ABORT:
432 REPORT_GUEST_ASSERT_2(uc, "MSR = %lx, vector = %lx");503 REPORT_GUEST_ASSERT_3(uc, "MSR = %lx, arg1 = %lx, arg2 = %lx");
433 return;504 return;
434 case UCALL_DONE:505 case UCALL_DONE:
435 break;506 break;
@@ -445,7 +516,6 @@ static void guest_test_msrs_access(void) · 9 unmodified lines
445 516
446static void guest_test_hcalls_access(void)517static void guest_test_hcalls_access(void)
447{518{
448 struct kvm_cpuid_entry2 *feat, *recomm, *dbg;
449 struct kvm_cpuid2 *prev_cpuid = NULL;519 struct kvm_cpuid2 *prev_cpuid = NULL;
450 struct kvm_vcpu *vcpu;520 struct kvm_vcpu *vcpu;
451 struct kvm_run *run;521 struct kvm_run *run;
@@ -480,15 +550,11 @@ static void guest_test_hcalls_access(void) · 28 unmodified lines
480 vcpu_init_cpuid(vcpu, prev_cpuid);550 vcpu_init_cpuid(vcpu, prev_cpuid);
481 }551 }
482 552
483 feat = vcpu_get_cpuid_entry(vcpu, HYPERV_CPUID_FEATURES);
484 recomm = vcpu_get_cpuid_entry(vcpu, HYPERV_CPUID_ENLIGHTMENT_INFO);
485 dbg = vcpu_get_cpuid_entry(vcpu, HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES);
486
487 run = vcpu->run;553 run = vcpu->run;
488 554
489 switch (stage) {555 switch (stage) {
490 case 0:556 case 0:
491 feat->eax |= HV_MSR_HYPERCALL_AVAILABLE;557 vcpu_set_cpuid_feature(vcpu, HV_MSR_HYPERCALL_AVAILABLE);
492 hcall->control = 0xbeef;558 hcall->control = 0xbeef;
493 hcall->expect = HV_STATUS_INVALID_HYPERCALL_CODE;559 hcall->expect = HV_STATUS_INVALID_HYPERCALL_CODE;
494 break;560 break;
@@ -498,7 +564,7 @@ static void guest_test_hcalls_access(void) · 3 unmodified lines
498 hcall->expect = HV_STATUS_ACCESS_DENIED;564 hcall->expect = HV_STATUS_ACCESS_DENIED;
499 break;565 break;
500 case 2:566 case 2:
501 feat->ebx |= HV_POST_MESSAGES;567 vcpu_set_cpuid_feature(vcpu, HV_POST_MESSAGES);
502 hcall->control = HVCALL_POST_MESSAGE;568 hcall->control = HVCALL_POST_MESSAGE;
503 hcall->expect = HV_STATUS_INVALID_HYPERCALL_INPUT;569 hcall->expect = HV_STATUS_INVALID_HYPERCALL_INPUT;
504 break;570 break;
@@ -508,7 +574,7 @@ static void guest_test_hcalls_access(void) · 3 unmodified lines
508 hcall->expect = HV_STATUS_ACCESS_DENIED;574 hcall->expect = HV_STATUS_ACCESS_DENIED;
509 break;575 break;
510 case 4:576 case 4:
511 feat->ebx |= HV_SIGNAL_EVENTS;577 vcpu_set_cpuid_feature(vcpu, HV_SIGNAL_EVENTS);
512 hcall->control = HVCALL_SIGNAL_EVENT;578 hcall->control = HVCALL_SIGNAL_EVENT;
513 hcall->expect = HV_STATUS_INVALID_HYPERCALL_INPUT;579 hcall->expect = HV_STATUS_INVALID_HYPERCALL_INPUT;
514 break;580 break;
@@ -518,12 +584,12 @@ static void guest_test_hcalls_access(void) · 3 unmodified lines
518 hcall->expect = HV_STATUS_INVALID_HYPERCALL_CODE;584 hcall->expect = HV_STATUS_INVALID_HYPERCALL_CODE;
519 break;585 break;
520 case 6:586 case 6:
521 dbg->eax |= HV_X64_SYNDBG_CAP_ALLOW_KERNEL_DEBUGGING;587 vcpu_set_cpuid_feature(vcpu, HV_X64_SYNDBG_CAP_ALLOW_KERNEL_DEBUGGING);
522 hcall->control = HVCALL_RESET_DEBUG_SESSION;588 hcall->control = HVCALL_RESET_DEBUG_SESSION;
523 hcall->expect = HV_STATUS_ACCESS_DENIED;589 hcall->expect = HV_STATUS_ACCESS_DENIED;
524 break;590 break;
525 case 7:591 case 7:
526 feat->ebx |= HV_DEBUGGING;592 vcpu_set_cpuid_feature(vcpu, HV_DEBUGGING);
527 hcall->control = HVCALL_RESET_DEBUG_SESSION;593 hcall->control = HVCALL_RESET_DEBUG_SESSION;
528 hcall->expect = HV_STATUS_OPERATION_DENIED;594 hcall->expect = HV_STATUS_OPERATION_DENIED;
529 break;595 break;
@@ -533,7 +599,7 @@ static void guest_test_hcalls_access(void) · 3 unmodified lines
533 hcall->expect = HV_STATUS_ACCESS_DENIED;599 hcall->expect = HV_STATUS_ACCESS_DENIED;
534 break;600 break;
535 case 9:601 case 9:
536 recomm->eax |= HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED;602 vcpu_set_cpuid_feature(vcpu, HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED);
537 hcall->control = HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE;603 hcall->control = HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE;
538 hcall->expect = HV_STATUS_SUCCESS;604 hcall->expect = HV_STATUS_SUCCESS;
539 break;605 break;
@@ -542,7 +608,7 @@ static void guest_test_hcalls_access(void) · 2 unmodified lines
542 hcall->expect = HV_STATUS_ACCESS_DENIED;608 hcall->expect = HV_STATUS_ACCESS_DENIED;
543 break;609 break;
544 case 11:610 case 11:
545 recomm->eax |= HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED;611 vcpu_set_cpuid_feature(vcpu, HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED);
546 hcall->control = HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX;612 hcall->control = HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX;
547 hcall->expect = HV_STATUS_SUCCESS;613 hcall->expect = HV_STATUS_SUCCESS;
548 break;614 break;
@@ -552,7 +618,7 @@ static void guest_test_hcalls_access(void) · 3 unmodified lines
552 hcall->expect = HV_STATUS_ACCESS_DENIED;618 hcall->expect = HV_STATUS_ACCESS_DENIED;
553 break;619 break;
554 case 13:620 case 13:
555 recomm->eax |= HV_X64_CLUSTER_IPI_RECOMMENDED;621 vcpu_set_cpuid_feature(vcpu, HV_X64_CLUSTER_IPI_RECOMMENDED);
556 hcall->control = HVCALL_SEND_IPI;622 hcall->control = HVCALL_SEND_IPI;
557 hcall->expect = HV_STATUS_INVALID_HYPERCALL_INPUT;623 hcall->expect = HV_STATUS_INVALID_HYPERCALL_INPUT;
558 break;624 break;
@@ -567,7 +633,7 @@ static void guest_test_hcalls_access(void) · 8 unmodified lines
567 hcall->expect = HV_STATUS_ACCESS_DENIED;633 hcall->expect = HV_STATUS_ACCESS_DENIED;
568 break;634 break;
569 case 16:635 case 16:
570 recomm->ebx = 0xfff;636 vcpu_set_cpuid_feature(vcpu, HV_PV_SPINLOCKS_TEST);
571 hcall->control = HVCALL_NOTIFY_LONG_SPIN_WAIT;637 hcall->control = HVCALL_NOTIFY_LONG_SPIN_WAIT;
572 hcall->expect = HV_STATUS_SUCCESS;638 hcall->expect = HV_STATUS_SUCCESS;
573 break;639 break;
@@ -577,12 +643,21 @@ static void guest_test_hcalls_access(void) · 3 unmodified lines
577 hcall->ud_expected = true;643 hcall->ud_expected = true;
578 break;644 break;
579 case 18:645 case 18:
580 feat->edx |= HV_X64_HYPERCALL_XMM_INPUT_AVAILABLE;646 vcpu_set_cpuid_feature(vcpu, HV_X64_HYPERCALL_XMM_INPUT_AVAILABLE);
581 hcall->control = HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE | HV_HYPERCALL_FAST_BIT;647 hcall->control = HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE | HV_HYPERCALL_FAST_BIT;
582 hcall->ud_expected = false;648 hcall->ud_expected = false;
583 hcall->expect = HV_STATUS_SUCCESS;649 hcall->expect = HV_STATUS_SUCCESS;
584 break;650 break;
585 case 19:651 case 19:
652 hcall->control = HV_EXT_CALL_QUERY_CAPABILITIES;
653 hcall->expect = HV_STATUS_ACCESS_DENIED;
654 break;
655 case 20:
656 vcpu_set_cpuid_feature(vcpu, HV_ENABLE_EXTENDED_HYPERCALLS);
657 hcall->control = HV_EXT_CALL_QUERY_CAPABILITIES | HV_HYPERCALL_FAST_BIT;
658 hcall->expect = HV_STATUS_INVALID_PARAMETER;
659 break;
660 case 21:
586 kvm_vm_free(vm);661 kvm_vm_free(vm);
587 return;662 return;
588 }663 }
tools/testing/selftests/kvm/x86_64/mmio_warning_test.c+1−1
@@ -93,7 +93,7 @@ int main(void) · 92 unmodified lines
93{93{
94 int warnings_before, warnings_after;94 int warnings_before, warnings_after;
95 95
96 TEST_REQUIRE(is_intel_cpu());96 TEST_REQUIRE(host_cpu_is_intel);
97 97
98 TEST_REQUIRE(!vm_is_unrestricted_guest(NULL));98 TEST_REQUIRE(!vm_is_unrestricted_guest(NULL));
99 99
tools/testing/selftests/kvm/x86_64/pmu_event_filter_test.c+378−7
@@ -198,14 +198,15 @@ static struct kvm_pmu_event_filter *alloc_pmu_event_filter(uint32_t nevents) · 197 unmodified lines
198 198
199 199
200static struct kvm_pmu_event_filter *200static struct kvm_pmu_event_filter *
201create_pmu_event_filter(const uint64_t event_list[],201create_pmu_event_filter(const uint64_t event_list[], int nevents,
202 int nevents, uint32_t action)202 uint32_t action, uint32_t flags)
203{203{
204 struct kvm_pmu_event_filter *f;204 struct kvm_pmu_event_filter *f;
205 int i;205 int i;
206 206
207 f = alloc_pmu_event_filter(nevents);207 f = alloc_pmu_event_filter(nevents);
208 f->action = action;208 f->action = action;
209 f->flags = flags;
209 for (i = 0; i < nevents; i++)210 for (i = 0; i < nevents; i++)
210 f->events[i] = event_list[i];211 f->events[i] = event_list[i];
211 212
@@ -216,7 +217,7 @@ static struct kvm_pmu_event_filter *event_filter(uint32_t action) · 4 unmodified lines
216{217{
217 return create_pmu_event_filter(event_list,218 return create_pmu_event_filter(event_list,
218 ARRAY_SIZE(event_list),219 ARRAY_SIZE(event_list),
219 action);220 action, 0);
220}221}
221 222
222/*223/*
@@ -263,7 +264,7 @@ static void test_amd_deny_list(struct kvm_vcpu *vcpu) · 40 unmodified lines
263 struct kvm_pmu_event_filter *f;264 struct kvm_pmu_event_filter *f;
264 uint64_t count;265 uint64_t count;
265 266
266 f = create_pmu_event_filter(&event, 1, KVM_PMU_EVENT_DENY);267 f = create_pmu_event_filter(&event, 1, KVM_PMU_EVENT_DENY, 0);
267 count = test_with_filter(vcpu, f);268 count = test_with_filter(vcpu, f);
268 269
269 free(f);270 free(f);
@@ -363,7 +364,7 @@ static void test_pmu_config_disable(void (*guest_code)(void)) · 93 unmodified lines
363 */364 */
364static bool use_intel_pmu(void)365static bool use_intel_pmu(void)
365{366{
366 return is_intel_cpu() &&367 return host_cpu_is_intel &&
367 kvm_cpu_property(X86_PROPERTY_PMU_VERSION) &&368 kvm_cpu_property(X86_PROPERTY_PMU_VERSION) &&
368 kvm_cpu_property(X86_PROPERTY_PMU_NR_GP_COUNTERS) &&369 kvm_cpu_property(X86_PROPERTY_PMU_NR_GP_COUNTERS) &&
369 kvm_pmu_has(X86_PMU_FEATURE_BRANCH_INSNS_RETIRED);370 kvm_pmu_has(X86_PMU_FEATURE_BRANCH_INSNS_RETIRED);
@@ -397,19 +398,378 @@ static bool use_amd_pmu(void) · 27 unmodified lines
397 uint32_t family = kvm_cpu_family();398 uint32_t family = kvm_cpu_family();
398 uint32_t model = kvm_cpu_model();399 uint32_t model = kvm_cpu_model();
399 400
400 return is_amd_cpu() &&401 return host_cpu_is_amd &&
401 (is_zen1(family, model) ||402 (is_zen1(family, model) ||
402 is_zen2(family, model) ||403 is_zen2(family, model) ||
403 is_zen3(family, model));404 is_zen3(family, model));
404}405}
405 406
407/*
408 * "MEM_INST_RETIRED.ALL_LOADS", "MEM_INST_RETIRED.ALL_STORES", and
409 * "MEM_INST_RETIRED.ANY" from https://perfmon-events.intel.com/
410 * supported on Intel Xeon processors:
411 * - Sapphire Rapids, Ice Lake, Cascade Lake, Skylake.
412 */
413#define MEM_INST_RETIRED 0xD0
414#define MEM_INST_RETIRED_LOAD EVENT(MEM_INST_RETIRED, 0x81)
415#define MEM_INST_RETIRED_STORE EVENT(MEM_INST_RETIRED, 0x82)
416#define MEM_INST_RETIRED_LOAD_STORE EVENT(MEM_INST_RETIRED, 0x83)
417
418static bool supports_event_mem_inst_retired(void)
419{
420 uint32_t eax, ebx, ecx, edx;
421
422 cpuid(1, &eax, &ebx, &ecx, &edx);
423 if (x86_family(eax) == 0x6) {
424 switch (x86_model(eax)) {
425 /* Sapphire Rapids */
426 case 0x8F:
427 /* Ice Lake */
428 case 0x6A:
429 /* Skylake */
430 /* Cascade Lake */
431 case 0x55:
432 return true;
433 }
434 }
435
436 return false;
437}
438
439/*
440 * "LS Dispatch", from Processor Programming Reference
441 * (PPR) for AMD Family 17h Model 01h, Revision B1 Processors,
442 * Preliminary Processor Programming Reference (PPR) for AMD Family
443 * 17h Model 31h, Revision B0 Processors, and Preliminary Processor
444 * Programming Reference (PPR) for AMD Family 19h Model 01h, Revision
445 * B1 Processors Volume 1 of 2.
446 */
447#define LS_DISPATCH 0x29
448#define LS_DISPATCH_LOAD EVENT(LS_DISPATCH, BIT(0))
449#define LS_DISPATCH_STORE EVENT(LS_DISPATCH, BIT(1))
450#define LS_DISPATCH_LOAD_STORE EVENT(LS_DISPATCH, BIT(2))
451
452#define INCLUDE_MASKED_ENTRY(event_select, mask, match) \
453 KVM_PMU_ENCODE_MASKED_ENTRY(event_select, mask, match, false)
454#define EXCLUDE_MASKED_ENTRY(event_select, mask, match) \
455 KVM_PMU_ENCODE_MASKED_ENTRY(event_select, mask, match, true)
456
457struct perf_counter {
458 union {
459 uint64_t raw;
460 struct {
461 uint64_t loads:22;
462 uint64_t stores:22;
463 uint64_t loads_stores:20;
464 };
465 };
466};
467
468static uint64_t masked_events_guest_test(uint32_t msr_base)
469{
470 uint64_t ld0, ld1, st0, st1, ls0, ls1;
471 struct perf_counter c;
472 int val;
473
474 /*
475 * The acutal value of the counters don't determine the outcome of
476 * the test. Only that they are zero or non-zero.
477 */
478 ld0 = rdmsr(msr_base + 0);
479 st0 = rdmsr(msr_base + 1);
480 ls0 = rdmsr(msr_base + 2);
481
482 __asm__ __volatile__("movl $0, %[v];"
483 "movl %[v], %%eax;"
484 "incl %[v];"
485 : [v]"+m"(val) :: "eax");
486
487 ld1 = rdmsr(msr_base + 0);
488 st1 = rdmsr(msr_base + 1);
489 ls1 = rdmsr(msr_base + 2);
490
491 c.loads = ld1 - ld0;
492 c.stores = st1 - st0;
493 c.loads_stores = ls1 - ls0;
494
495 return c.raw;
496}
497
498static void intel_masked_events_guest_code(void)
499{
500 uint64_t r;
501
502 for (;;) {
503 wrmsr(MSR_CORE_PERF_GLOBAL_CTRL, 0);
504
505 wrmsr(MSR_P6_EVNTSEL0 + 0, ARCH_PERFMON_EVENTSEL_ENABLE |
506 ARCH_PERFMON_EVENTSEL_OS | MEM_INST_RETIRED_LOAD);
507 wrmsr(MSR_P6_EVNTSEL0 + 1, ARCH_PERFMON_EVENTSEL_ENABLE |
508 ARCH_PERFMON_EVENTSEL_OS | MEM_INST_RETIRED_STORE);
509 wrmsr(MSR_P6_EVNTSEL0 + 2, ARCH_PERFMON_EVENTSEL_ENABLE |
510 ARCH_PERFMON_EVENTSEL_OS | MEM_INST_RETIRED_LOAD_STORE);
511
512 wrmsr(MSR_CORE_PERF_GLOBAL_CTRL, 0x7);
513
514 r = masked_events_guest_test(MSR_IA32_PMC0);
515
516 GUEST_SYNC(r);
517 }
518}
519
520static void amd_masked_events_guest_code(void)
521{
522 uint64_t r;
523
524 for (;;) {
525 wrmsr(MSR_K7_EVNTSEL0, 0);
526 wrmsr(MSR_K7_EVNTSEL1, 0);
527 wrmsr(MSR_K7_EVNTSEL2, 0);
528
529 wrmsr(MSR_K7_EVNTSEL0, ARCH_PERFMON_EVENTSEL_ENABLE |
530 ARCH_PERFMON_EVENTSEL_OS | LS_DISPATCH_LOAD);
531 wrmsr(MSR_K7_EVNTSEL1, ARCH_PERFMON_EVENTSEL_ENABLE |
532 ARCH_PERFMON_EVENTSEL_OS | LS_DISPATCH_STORE);
533 wrmsr(MSR_K7_EVNTSEL2, ARCH_PERFMON_EVENTSEL_ENABLE |
534 ARCH_PERFMON_EVENTSEL_OS | LS_DISPATCH_LOAD_STORE);
535
536 r = masked_events_guest_test(MSR_K7_PERFCTR0);
537
538 GUEST_SYNC(r);
539 }
540}
541
542static struct perf_counter run_masked_events_test(struct kvm_vcpu *vcpu,
543 const uint64_t masked_events[],
544 const int nmasked_events)
545{
546 struct kvm_pmu_event_filter *f;
547 struct perf_counter r;
548
549 f = create_pmu_event_filter(masked_events, nmasked_events,
550 KVM_PMU_EVENT_ALLOW,
551 KVM_PMU_EVENT_FLAG_MASKED_EVENTS);
552 r.raw = test_with_filter(vcpu, f);
553 free(f);
554
555 return r;
556}
557
558/* Matches KVM_PMU_EVENT_FILTER_MAX_EVENTS in pmu.c */
559#define MAX_FILTER_EVENTS 300
560#define MAX_TEST_EVENTS 10
561
562#define ALLOW_LOADS BIT(0)
563#define ALLOW_STORES BIT(1)
564#define ALLOW_LOADS_STORES BIT(2)
565
566struct masked_events_test {
567 uint64_t intel_events[MAX_TEST_EVENTS];
568 uint64_t intel_event_end;
569 uint64_t amd_events[MAX_TEST_EVENTS];
570 uint64_t amd_event_end;
571 const char *msg;
572 uint32_t flags;
573};
574
575/*
576 * These are the test cases for the masked events tests.
577 *
578 * For each test, the guest enables 3 PMU counters (loads, stores,
579 * loads + stores). The filter is then set in KVM with the masked events
580 * provided. The test then verifies that the counters agree with which
581 * ones should be counting and which ones should be filtered.
582 */
583const struct masked_events_test test_cases[] = {
584 {
585 .intel_events = {
586 INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0xFF, 0x81),
587 },
588 .amd_events = {
589 INCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xFF, BIT(0)),
590 },
591 .msg = "Only allow loads.",
592 .flags = ALLOW_LOADS,
593 }, {
594 .intel_events = {
595 INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0xFF, 0x82),
596 },
597 .amd_events = {
598 INCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xFF, BIT(1)),
599 },
600 .msg = "Only allow stores.",
601 .flags = ALLOW_STORES,
602 }, {
603 .intel_events = {
604 INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0xFF, 0x83),
605 },
606 .amd_events = {
607 INCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xFF, BIT(2)),
608 },
609 .msg = "Only allow loads + stores.",
610 .flags = ALLOW_LOADS_STORES,
611 }, {
612 .intel_events = {
613 INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0x7C, 0),
614 EXCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0xFF, 0x83),
615 },
616 .amd_events = {
617 INCLUDE_MASKED_ENTRY(LS_DISPATCH, ~(BIT(0) | BIT(1)), 0),
618 },
619 .msg = "Only allow loads and stores.",
620 .flags = ALLOW_LOADS | ALLOW_STORES,
621 }, {
622 .intel_events = {
623 INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0x7C, 0),
624 EXCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0xFF, 0x82),
625 },
626 .amd_events = {
627 INCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xF8, 0),
628 EXCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xFF, BIT(1)),
629 },
630 .msg = "Only allow loads and loads + stores.",
631 .flags = ALLOW_LOADS | ALLOW_LOADS_STORES
632 }, {
633 .intel_events = {
634 INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0xFE, 0x82),
635 },
636 .amd_events = {
637 INCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xF8, 0),
638 EXCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xFF, BIT(0)),
639 },
640 .msg = "Only allow stores and loads + stores.",
641 .flags = ALLOW_STORES | ALLOW_LOADS_STORES
642 }, {
643 .intel_events = {
644 INCLUDE_MASKED_ENTRY(MEM_INST_RETIRED, 0x7C, 0),
645 },
646 .amd_events = {
647 INCLUDE_MASKED_ENTRY(LS_DISPATCH, 0xF8, 0),
648 },
649 .msg = "Only allow loads, stores, and loads + stores.",
650 .flags = ALLOW_LOADS | ALLOW_STORES | ALLOW_LOADS_STORES
651 },
652};
653
654static int append_test_events(const struct masked_events_test *test,
655 uint64_t *events, int nevents)
656{
657 const uint64_t *evts;
658 int i;
659
660 evts = use_intel_pmu() ? test->intel_events : test->amd_events;
661 for (i = 0; i < MAX_TEST_EVENTS; i++) {
662 if (evts[i] == 0)
663 break;
664
665 events[nevents + i] = evts[i];
666 }
667
668 return nevents + i;
669}
670
671static bool bool_eq(bool a, bool b)
672{
673 return a == b;
674}
675
676static void run_masked_events_tests(struct kvm_vcpu *vcpu, uint64_t *events,
677 int nevents)
678{
679 int ntests = ARRAY_SIZE(test_cases);
680 struct perf_counter c;
681 int i, n;
682
683 for (i = 0; i < ntests; i++) {
684 const struct masked_events_test *test = &test_cases[i];
685
686 /* Do any test case events overflow MAX_TEST_EVENTS? */
687 assert(test->intel_event_end == 0);
688 assert(test->amd_event_end == 0);
689
690 n = append_test_events(test, events, nevents);
691
692 c = run_masked_events_test(vcpu, events, n);
693 TEST_ASSERT(bool_eq(c.loads, test->flags & ALLOW_LOADS) &&
694 bool_eq(c.stores, test->flags & ALLOW_STORES) &&
695 bool_eq(c.loads_stores,
696 test->flags & ALLOW_LOADS_STORES),
697 "%s loads: %u, stores: %u, loads + stores: %u",
698 test->msg, c.loads, c.stores, c.loads_stores);
699 }
700}
701
702static void add_dummy_events(uint64_t *events, int nevents)
703{
704 int i;
705
706 for (i = 0; i < nevents; i++) {
707 int event_select = i % 0xFF;
708 bool exclude = ((i % 4) == 0);
709
710 if (event_select == MEM_INST_RETIRED ||
711 event_select == LS_DISPATCH)
712 event_select++;
713
714 events[i] = KVM_PMU_ENCODE_MASKED_ENTRY(event_select, 0,
715 0, exclude);
716 }
717}
718
719static void test_masked_events(struct kvm_vcpu *vcpu)
720{
721 int nevents = MAX_FILTER_EVENTS - MAX_TEST_EVENTS;
722 uint64_t events[MAX_FILTER_EVENTS];
723
724 /* Run the test cases against a sparse PMU event filter. */
725 run_masked_events_tests(vcpu, events, 0);
726
727 /* Run the test cases against a dense PMU event filter. */
728 add_dummy_events(events, MAX_FILTER_EVENTS);
729 run_masked_events_tests(vcpu, events, nevents);
730}
731
732static int run_filter_test(struct kvm_vcpu *vcpu, const uint64_t *events,
733 int nevents, uint32_t flags)
734{
735 struct kvm_pmu_event_filter *f;
736 int r;
737
738 f = create_pmu_event_filter(events, nevents, KVM_PMU_EVENT_ALLOW, flags);
739 r = __vm_ioctl(vcpu->vm, KVM_SET_PMU_EVENT_FILTER, f);
740 free(f);
741
742 return r;
743}
744
745static void test_filter_ioctl(struct kvm_vcpu *vcpu)
746{
747 uint64_t e = ~0ul;
748 int r;
749
750 /*
751 * Unfortunately having invalid bits set in event data is expected to
752 * pass when flags == 0 (bits other than eventsel+umask).
753 */
754 r = run_filter_test(vcpu, &e, 1, 0);
755 TEST_ASSERT(r == 0, "Valid PMU Event Filter is failing");
756
757 r = run_filter_test(vcpu, &e, 1, KVM_PMU_EVENT_FLAG_MASKED_EVENTS);
758 TEST_ASSERT(r != 0, "Invalid PMU Event Filter is expected to fail");
759
760 e = KVM_PMU_ENCODE_MASKED_ENTRY(0xff, 0xff, 0xff, 0xf);
761 r = run_filter_test(vcpu, &e, 1, KVM_PMU_EVENT_FLAG_MASKED_EVENTS);
762 TEST_ASSERT(r == 0, "Valid PMU Event Filter is failing");
763}
764
406int main(int argc, char *argv[])765int main(int argc, char *argv[])
407{766{
408 void (*guest_code)(void);767 void (*guest_code)(void);
409 struct kvm_vcpu *vcpu;768 struct kvm_vcpu *vcpu, *vcpu2 = NULL;
410 struct kvm_vm *vm;769 struct kvm_vm *vm;
411 770
412 TEST_REQUIRE(kvm_has_cap(KVM_CAP_PMU_EVENT_FILTER));771 TEST_REQUIRE(kvm_has_cap(KVM_CAP_PMU_EVENT_FILTER));
772 TEST_REQUIRE(kvm_has_cap(KVM_CAP_PMU_EVENT_MASKED_EVENTS));
413 773
414 TEST_REQUIRE(use_intel_pmu() || use_amd_pmu());774 TEST_REQUIRE(use_intel_pmu() || use_amd_pmu());
415 guest_code = use_intel_pmu() ? intel_guest_code : amd_guest_code;775 guest_code = use_intel_pmu() ? intel_guest_code : amd_guest_code;
@@ -430,6 +790,17 @@ int main(int argc, char *argv[]) · 14 unmodified lines
430 test_not_member_deny_list(vcpu);790 test_not_member_deny_list(vcpu);
431 test_not_member_allow_list(vcpu);791 test_not_member_allow_list(vcpu);
432 792
793 if (use_intel_pmu() &&
794 supports_event_mem_inst_retired() &&
795 kvm_cpu_property(X86_PROPERTY_PMU_NR_GP_COUNTERS) >= 3)
796 vcpu2 = vm_vcpu_add(vm, 2, intel_masked_events_guest_code);
797 else if (use_amd_pmu())
798 vcpu2 = vm_vcpu_add(vm, 2, amd_masked_events_guest_code);
799
800 if (vcpu2)
801 test_masked_events(vcpu2);
802 test_filter_ioctl(vcpu);
803
433 kvm_vm_free(vm);804 kvm_vm_free(vm);
434 805
435 test_pmu_config_disable(guest_code);806 test_pmu_config_disable(guest_code);
tools/testing/selftests/kvm/x86_64/tsc_msrs_test.c+12−4
@@ -72,11 +72,16 @@ static void run_vcpu(struct kvm_vcpu *vcpu, int stage) · 71 unmodified lines
72 72
73 switch (get_ucall(vcpu, &uc)) {73 switch (get_ucall(vcpu, &uc)) {
74 case UCALL_SYNC:74 case UCALL_SYNC:
75 TEST_ASSERT(!strcmp((const char *)uc.args[0], "hello") &&75 if (!strcmp((const char *)uc.args[0], "hello") &&
76 uc.args[1] == stage + 1, "Stage %d: Unexpected register values vmexit, got %lx",76 uc.args[1] == stage + 1)
77 stage + 1, (ulong)uc.args[1]);77 ksft_test_result_pass("stage %d passed\n", stage + 1);
78 else
79 ksft_test_result_fail(
80 "stage %d: Unexpected register values vmexit, got %lx",
81 stage + 1, (ulong)uc.args[1]);
78 return;82 return;
79 case UCALL_DONE:83 case UCALL_DONE:
84 ksft_test_result_pass("stage %d passed\n", stage + 1);
80 return;85 return;
81 case UCALL_ABORT:86 case UCALL_ABORT:
82 REPORT_GUEST_ASSERT_2(uc, "values: %#lx, %#lx");87 REPORT_GUEST_ASSERT_2(uc, "values: %#lx, %#lx");
@@ -92,6 +97,9 @@ int main(void) · 9 unmodified lines
92 struct kvm_vm *vm;97 struct kvm_vm *vm;
93 uint64_t val;98 uint64_t val;
94 99
100 ksft_print_header();
101 ksft_set_plan(5);
102
95 vm = vm_create_with_one_vcpu(&vcpu, guest_code);103 vm = vm_create_with_one_vcpu(&vcpu, guest_code);
96 104
97 val = 0;105 val = 0;
@@ -149,5 +157,5 @@ int main(void) · 51 unmodified lines
149 157
150 kvm_vm_free(vm);158 kvm_vm_free(vm);
151 159
152 return 0;160 ksft_finished(); /* Print results and exit() accordingly */
153}161}
tools/testing/selftests/kvm/x86_64/vmx_exception_with_invalid_guest_state.c+1−1
@@ -111,7 +111,7 @@ int main(int argc, char *argv[]) · 110 unmodified lines
111 struct kvm_vcpu *vcpu;111 struct kvm_vcpu *vcpu;
112 struct kvm_vm *vm;112 struct kvm_vm *vm;
113 113
114 TEST_REQUIRE(is_intel_cpu());114 TEST_REQUIRE(host_cpu_is_intel);
115 TEST_REQUIRE(!vm_is_unrestricted_guest(NULL));115 TEST_REQUIRE(!vm_is_unrestricted_guest(NULL));
116 116
117 vm = vm_create_with_one_vcpu(&vcpu, guest_code);117 vm = vm_create_with_one_vcpu(&vcpu, guest_code);
tools/testing/selftests/kvm/x86_64/xapic_state_test.c+55−0
@@ -132,6 +132,59 @@ static void test_icr(struct xapic_vcpu *x) · 131 unmodified lines
132 __test_icr(x, -1ull & ~APIC_DM_FIXED_MASK);132 __test_icr(x, -1ull & ~APIC_DM_FIXED_MASK);
133}133}
134 134
135static void __test_apic_id(struct kvm_vcpu *vcpu, uint64_t apic_base)
136{
137 uint32_t apic_id, expected;
138 struct kvm_lapic_state xapic;
139
140 vcpu_set_msr(vcpu, MSR_IA32_APICBASE, apic_base);
141
142 vcpu_ioctl(vcpu, KVM_GET_LAPIC, &xapic);
143
144 expected = apic_base & X2APIC_ENABLE ? vcpu->id : vcpu->id << 24;
145 apic_id = *((u32 *)&xapic.regs[APIC_ID]);
146
147 TEST_ASSERT(apic_id == expected,
148 "APIC_ID not set back to %s format; wanted = %x, got = %x",
149 (apic_base & X2APIC_ENABLE) ? "x2APIC" : "xAPIC",
150 expected, apic_id);
151}
152
153/*
154 * Verify that KVM switches the APIC_ID between xAPIC and x2APIC when userspace
155 * stuffs MSR_IA32_APICBASE. Setting the APIC_ID when x2APIC is enabled and
156 * when the APIC transitions for DISABLED to ENABLED is architectural behavior
157 * (on Intel), whereas the x2APIC => xAPIC transition behavior is KVM ABI since
158 * attempted to transition from x2APIC to xAPIC without disabling the APIC is
159 * architecturally disallowed.
160 */
161static void test_apic_id(void)
162{
163 const uint32_t NR_VCPUS = 3;
164 struct kvm_vcpu *vcpus[NR_VCPUS];
165 uint64_t apic_base;
166 struct kvm_vm *vm;
167 int i;
168
169 vm = vm_create_with_vcpus(NR_VCPUS, NULL, vcpus);
170 vm_enable_cap(vm, KVM_CAP_X2APIC_API, KVM_X2APIC_API_USE_32BIT_IDS);
171
172 for (i = 0; i < NR_VCPUS; i++) {
173 apic_base = vcpu_get_msr(vcpus[i], MSR_IA32_APICBASE);
174
175 TEST_ASSERT(apic_base & MSR_IA32_APICBASE_ENABLE,
176 "APIC not in ENABLED state at vCPU RESET");
177 TEST_ASSERT(!(apic_base & X2APIC_ENABLE),
178 "APIC not in xAPIC mode at vCPU RESET");
179
180 __test_apic_id(vcpus[i], apic_base);
181 __test_apic_id(vcpus[i], apic_base | X2APIC_ENABLE);
182 __test_apic_id(vcpus[i], apic_base);
183 }
184
185 kvm_vm_free(vm);
186}
187
135int main(int argc, char *argv[])188int main(int argc, char *argv[])
136{189{
137 struct xapic_vcpu x = {190 struct xapic_vcpu x = {
@@ -157,4 +210,6 @@ int main(int argc, char *argv[]) · 19 unmodified lines
157 virt_pg_map(vm, APIC_DEFAULT_GPA, APIC_DEFAULT_GPA);210 virt_pg_map(vm, APIC_DEFAULT_GPA, APIC_DEFAULT_GPA);
158 test_icr(&x);211 test_icr(&x);
159 kvm_vm_free(vm);212 kvm_vm_free(vm);
213
214 test_apic_id();
160}215}
tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c+6−9
@@ -19,9 +19,6 @@ · 18 unmodified lines
19 19
20#include <sys/eventfd.h>20#include <sys/eventfd.h>
21 21
22/* Defined in include/linux/kvm_types.h */
23#define GPA_INVALID (~(ulong)0)
24
25#define SHINFO_REGION_GVA 0xc0000000ULL22#define SHINFO_REGION_GVA 0xc0000000ULL
26#define SHINFO_REGION_GPA 0xc0000000ULL23#define SHINFO_REGION_GPA 0xc0000000ULL
27#define SHINFO_REGION_SLOT 1024#define SHINFO_REGION_SLOT 10
@@ -412,21 +409,21 @@ static void *juggle_shinfo_state(void *arg) · 384 unmodified lines
412{409{
413 struct kvm_vm *vm = (struct kvm_vm *)arg;410 struct kvm_vm *vm = (struct kvm_vm *)arg;
414 411
415 struct kvm_xen_hvm_attr cache_init = {412 struct kvm_xen_hvm_attr cache_activate = {
416 .type = KVM_XEN_ATTR_TYPE_SHARED_INFO,413 .type = KVM_XEN_ATTR_TYPE_SHARED_INFO,
417 .u.shared_info.gfn = SHINFO_REGION_GPA / PAGE_SIZE414 .u.shared_info.gfn = SHINFO_REGION_GPA / PAGE_SIZE
418 };415 };
419 416
420 struct kvm_xen_hvm_attr cache_destroy = {417 struct kvm_xen_hvm_attr cache_deactivate = {
421 .type = KVM_XEN_ATTR_TYPE_SHARED_INFO,418 .type = KVM_XEN_ATTR_TYPE_SHARED_INFO,
422 .u.shared_info.gfn = GPA_INVALID419 .u.shared_info.gfn = KVM_XEN_INVALID_GFN
423 };420 };
424 421
425 for (;;) {422 for (;;) {
426 __vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &cache_init);423 __vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &cache_activate);
427 __vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &cache_destroy);424 __vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &cache_deactivate);
428 pthread_testcancel();425 pthread_testcancel();
429 };426 }
430 427
431 return NULL;428 return NULL;
432}429}
virt/kvm/Kconfig+3−0
@@ -92,3 +92,6 @@ config KVM_XFER_TO_GUEST_WORK · 91 unmodified lines
92 92
93config HAVE_KVM_PM_NOTIFIER93config HAVE_KVM_PM_NOTIFIER
94 bool94 bool
95
96config KVM_GENERIC_HARDWARE_ENABLING
97 bool
virt/kvm/coalesced_mmio.c+5−3
@@ -187,15 +187,17 @@ int kvm_vm_ioctl_unregister_coalesced_mmio(struct kvm *kvm, · 186 unmodified lines
187 r = kvm_io_bus_unregister_dev(kvm,187 r = kvm_io_bus_unregister_dev(kvm,
188 zone->pio ? KVM_PIO_BUS : KVM_MMIO_BUS, &dev->dev);188 zone->pio ? KVM_PIO_BUS : KVM_MMIO_BUS, &dev->dev);
189 189
190 kvm_iodevice_destructor(&dev->dev);
191
190 /*192 /*
191 * On failure, unregister destroys all devices on the193 * On failure, unregister destroys all devices on the
192 * bus _except_ the target device, i.e. coalesced_zones194 * bus _except_ the target device, i.e. coalesced_zones
193 * has been modified. No need to restart the walk as195 * has been modified. Bail after destroying the target
194 * there aren't any zones left.196 * device, there's no need to restart the walk as there
197 * aren't any zones left.
195 */198 */
196 if (r)199 if (r)
197 break;200 break;
198 kvm_iodevice_destructor(&dev->dev);
199 }201 }
200 }202 }
201 203
virt/kvm/kvm_main.c+156−141
@@ -100,13 +100,8 @@ EXPORT_SYMBOL_GPL(halt_poll_ns_shrink); · 99 unmodified lines
100 */100 */
101 101
102DEFINE_MUTEX(kvm_lock);102DEFINE_MUTEX(kvm_lock);
103static DEFINE_RAW_SPINLOCK(kvm_count_lock);
104LIST_HEAD(vm_list);103LIST_HEAD(vm_list);
105 104
106static cpumask_var_t cpus_hardware_enabled;
107static int kvm_usage_count;
108static atomic_t hardware_enable_failed;
109
110static struct kmem_cache *kvm_vcpu_cache;105static struct kmem_cache *kvm_vcpu_cache;
111 106
112static __read_mostly struct preempt_ops kvm_preempt_ops;107static __read_mostly struct preempt_ops kvm_preempt_ops;
@@ -148,9 +143,6 @@ static void hardware_disable_all(void); · 35 unmodified lines
148 143
149static void kvm_io_bus_destroy(struct kvm_io_bus *bus);144static void kvm_io_bus_destroy(struct kvm_io_bus *bus);
150 145
151__visible bool kvm_rebooting;
152EXPORT_SYMBOL_GPL(kvm_rebooting);
153
154#define KVM_EVENT_CREATE_VM 0146#define KVM_EVENT_CREATE_VM 0
155#define KVM_EVENT_DESTROY_VM 1147#define KVM_EVENT_DESTROY_VM 1
156static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm);148static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm);
@@ -5102,50 +5094,70 @@ static struct miscdevice kvm_dev = { · 4945 unmodified lines
5102 &kvm_chardev_ops,5094 &kvm_chardev_ops,
5103};5095};
5104 5096
5105static void hardware_enable_nolock(void *junk)5097#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING
5106{5098__visible bool kvm_rebooting;
5107 int cpu = raw_smp_processor_id();5099EXPORT_SYMBOL_GPL(kvm_rebooting);
5108 int r;
5109
5110 if (cpumask_test_cpu(cpu, cpus_hardware_enabled))
5111 return;
5112 5100
5113 cpumask_set_cpu(cpu, cpus_hardware_enabled);5101static DEFINE_PER_CPU(bool, hardware_enabled);
5102static int kvm_usage_count;
5114 5103
5115 r = kvm_arch_hardware_enable();5104static int __hardware_enable_nolock(void)
5105{
5106 if (__this_cpu_read(hardware_enabled))
5107 return 0;
5116 5108
5117 if (r) {5109 if (kvm_arch_hardware_enable()) {
5118 cpumask_clear_cpu(cpu, cpus_hardware_enabled);5110 pr_info("kvm: enabling virtualization on CPU%d failed\n",
5119 atomic_inc(&hardware_enable_failed);5111 raw_smp_processor_id());
5120 pr_info("kvm: enabling virtualization on CPU%d failed\n", cpu);5112 return -EIO;
5121 }5113 }
5114
5115 __this_cpu_write(hardware_enabled, true);
5116 return 0;
5122}5117}
5123 5118
5124static int kvm_starting_cpu(unsigned int cpu)5119static void hardware_enable_nolock(void *failed)
5125{5120{
5126 raw_spin_lock(&kvm_count_lock);5121 if (__hardware_enable_nolock())
5122 atomic_inc(failed);
5123}
5124
5125static int kvm_online_cpu(unsigned int cpu)
5126{
5127 int ret = 0;
5128
5129 /*
5130 * Abort the CPU online process if hardware virtualization cannot
5131 * be enabled. Otherwise running VMs would encounter unrecoverable
5132 * errors when scheduled to this CPU.
5133 */
5134 mutex_lock(&kvm_lock);
5127 if (kvm_usage_count)5135 if (kvm_usage_count)
5128 hardware_enable_nolock(NULL);5136 ret = __hardware_enable_nolock();
5129 raw_spin_unlock(&kvm_count_lock);5137 mutex_unlock(&kvm_lock);
5130 return 0;5138 return ret;
5131}5139}
5132 5140
5133static void hardware_disable_nolock(void *junk)5141static void hardware_disable_nolock(void *junk)
5134{5142{
5135 int cpu = raw_smp_processor_id();5143 /*
5136 5144 * Note, hardware_disable_all_nolock() tells all online CPUs to disable
5137 if (!cpumask_test_cpu(cpu, cpus_hardware_enabled))5145 * hardware, not just CPUs that successfully enabled hardware!
5146 */
5147 if (!__this_cpu_read(hardware_enabled))
5138 return;5148 return;
5139 cpumask_clear_cpu(cpu, cpus_hardware_enabled);5149
5140 kvm_arch_hardware_disable();5150 kvm_arch_hardware_disable();
5151
5152 __this_cpu_write(hardware_enabled, false);
5141}5153}
5142 5154
5143static int kvm_dying_cpu(unsigned int cpu)5155static int kvm_offline_cpu(unsigned int cpu)
5144{5156{
5145 raw_spin_lock(&kvm_count_lock);5157 mutex_lock(&kvm_lock);
5146 if (kvm_usage_count)5158 if (kvm_usage_count)
5147 hardware_disable_nolock(NULL);5159 hardware_disable_nolock(NULL);
5148 raw_spin_unlock(&kvm_count_lock);5160 mutex_unlock(&kvm_lock);
5149 return 0;5161 return 0;
5150}5162}
5151 5163
@@ -5160,29 +5172,41 @@ static void hardware_disable_all_nolock(void) · 8 unmodified lines
5160 5172
5161static void hardware_disable_all(void)5173static void hardware_disable_all(void)
5162{5174{
5163 raw_spin_lock(&kvm_count_lock);5175 cpus_read_lock();
5176 mutex_lock(&kvm_lock);
5164 hardware_disable_all_nolock();5177 hardware_disable_all_nolock();
5165 raw_spin_unlock(&kvm_count_lock);5178 mutex_unlock(&kvm_lock);
5179 cpus_read_unlock();
5166}5180}
5167 5181
5168static int hardware_enable_all(void)5182static int hardware_enable_all(void)
5169{5183{
5184 atomic_t failed = ATOMIC_INIT(0);
5170 int r = 0;5185 int r = 0;
5171 5186
5172 raw_spin_lock(&kvm_count_lock);5187 /*
5188 * When onlining a CPU, cpu_online_mask is set before kvm_online_cpu()
5189 * is called, and so on_each_cpu() between them includes the CPU that
5190 * is being onlined. As a result, hardware_enable_nolock() may get
5191 * invoked before kvm_online_cpu(), which also enables hardware if the
5192 * usage count is non-zero. Disable CPU hotplug to avoid attempting to
5193 * enable hardware multiple times.
5194 */
5195 cpus_read_lock();
5196 mutex_lock(&kvm_lock);
5173 5197
5174 kvm_usage_count++;5198 kvm_usage_count++;
5175 if (kvm_usage_count == 1) {5199 if (kvm_usage_count == 1) {
5176 atomic_set(&hardware_enable_failed, 0);5200 on_each_cpu(hardware_enable_nolock, &failed, 1);
5177 on_each_cpu(hardware_enable_nolock, NULL, 1);
5178 5201
5179 if (atomic_read(&hardware_enable_failed)) {5202 if (atomic_read(&failed)) {
5180 hardware_disable_all_nolock();5203 hardware_disable_all_nolock();
5181 r = -EBUSY;5204 r = -EBUSY;
5182 }5205 }
5183 }5206 }
5184 5207
5185 raw_spin_unlock(&kvm_count_lock);5208 mutex_unlock(&kvm_lock);
5209 cpus_read_unlock();
5186 5210
5187 return r;5211 return r;
5188}5212}
@@ -5207,6 +5231,49 @@ static struct notifier_block kvm_reboot_notifier = { · 18 unmodified lines
5207 .priority = 0,5231 .priority = 0,
5208};5232};
5209 5233
5234static int kvm_suspend(void)
5235{
5236 /*
5237 * Secondary CPUs and CPU hotplug are disabled across the suspend/resume
5238 * callbacks, i.e. no need to acquire kvm_lock to ensure the usage count
5239 * is stable. Assert that kvm_lock is not held to ensure the system
5240 * isn't suspended while KVM is enabling hardware. Hardware enabling
5241 * can be preempted, but the task cannot be frozen until it has dropped
5242 * all locks (userspace tasks are frozen via a fake signal).
5243 */
5244 lockdep_assert_not_held(&kvm_lock);
5245 lockdep_assert_irqs_disabled();
5246
5247 if (kvm_usage_count)
5248 hardware_disable_nolock(NULL);
5249 return 0;
5250}
5251
5252static void kvm_resume(void)
5253{
5254 lockdep_assert_not_held(&kvm_lock);
5255 lockdep_assert_irqs_disabled();
5256
5257 if (kvm_usage_count)
5258 WARN_ON_ONCE(__hardware_enable_nolock());
5259}
5260
5261static struct syscore_ops kvm_syscore_ops = {
5262 .suspend = kvm_suspend,
5263 .resume = kvm_resume,
5264};
5265#else /* CONFIG_KVM_GENERIC_HARDWARE_ENABLING */
5266static int hardware_enable_all(void)
5267{
5268 return 0;
5269}
5270
5271static void hardware_disable_all(void)
5272{
5273
5274}
5275#endif /* CONFIG_KVM_GENERIC_HARDWARE_ENABLING */
5276
5210static void kvm_io_bus_destroy(struct kvm_io_bus *bus)5277static void kvm_io_bus_destroy(struct kvm_io_bus *bus)
5211{5278{
5212 int i;5279 int i;
@@ -5785,26 +5852,6 @@ static void kvm_init_debug(void) · 572 unmodified lines
5785 }5852 }
5786}5853}
5787 5854
5788static int kvm_suspend(void)
5789{
5790 if (kvm_usage_count)
5791 hardware_disable_nolock(NULL);
5792 return 0;
5793}
5794
5795static void kvm_resume(void)
5796{
5797 if (kvm_usage_count) {
5798 lockdep_assert_not_held(&kvm_count_lock);
5799 hardware_enable_nolock(NULL);
5800 }
5801}
5802
5803static struct syscore_ops kvm_syscore_ops = {
5804 .suspend = kvm_suspend,
5805 .resume = kvm_resume,
5806};
5807
5808static inline5855static inline
5809struct kvm_vcpu *preempt_notifier_to_vcpu(struct preempt_notifier *pn)5856struct kvm_vcpu *preempt_notifier_to_vcpu(struct preempt_notifier *pn)
5810{5857{
@@ -5909,62 +5956,20 @@ void kvm_unregister_perf_callbacks(void) · 98 unmodified lines
5909}5956}
5910#endif5957#endif
5911 5958
5912struct kvm_cpu_compat_check {5959int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module)
5913 void *opaque;
5914 int *ret;
5915};
5916
5917static void check_processor_compat(void *data)
5918{
5919 struct kvm_cpu_compat_check *c = data;
5920
5921 *c->ret = kvm_arch_check_processor_compat(c->opaque);
5922}
5923
5924int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
5925 struct module *module)
5926{5960{
5927 struct kvm_cpu_compat_check c;
5928 int r;5961 int r;
5929 int cpu;5962 int cpu;
5930 5963
5931 r = kvm_arch_init(opaque);5964#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING
5932 if (r)5965 r = cpuhp_setup_state_nocalls(CPUHP_AP_KVM_ONLINE, "kvm/cpu:online",
5933 goto out_fail;5966 kvm_online_cpu, kvm_offline_cpu);
5934
5935 /*
5936 * kvm_arch_init makes sure there's at most one caller
5937 * for architectures that support multiple implementations,
5938 * like intel and amd on x86.
5939 * kvm_arch_init must be called before kvm_irqfd_init to avoid creating
5940 * conflicts in case kvm is already setup for another implementation.
5941 */
5942 r = kvm_irqfd_init();
5943 if (r)5967 if (r)
5944 goto out_irqfd;5968 return r;
5945
5946 if (!zalloc_cpumask_var(&cpus_hardware_enabled, GFP_KERNEL)) {
5947 r = -ENOMEM;
5948 goto out_free_0;
5949 }
5950
5951 r = kvm_arch_hardware_setup(opaque);
5952 if (r < 0)
5953 goto out_free_1;
5954
5955 c.ret = &r;
5956 c.opaque = opaque;
5957 for_each_online_cpu(cpu) {
5958 smp_call_function_single(cpu, check_processor_compat, &c, 1);
5959 if (r < 0)
5960 goto out_free_2;
5961 }
5962 5969
5963 r = cpuhp_setup_state_nocalls(CPUHP_AP_KVM_STARTING, "kvm/cpu:starting",
5964 kvm_starting_cpu, kvm_dying_cpu);
5965 if (r)
5966 goto out_free_2;
5967 register_reboot_notifier(&kvm_reboot_notifier);5970 register_reboot_notifier(&kvm_reboot_notifier);
5971 register_syscore_ops(&kvm_syscore_ops);
5972#endif
5968 5973
5969 /* A kmem cache lets us meet the alignment requirements of fx_save. */5974 /* A kmem cache lets us meet the alignment requirements of fx_save. */
5970 if (!vcpu_align)5975 if (!vcpu_align)
@@ -5978,59 +5983,65 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align, · 7 unmodified lines
5978 NULL);5983 NULL);
5979 if (!kvm_vcpu_cache) {5984 if (!kvm_vcpu_cache) {
5980 r = -ENOMEM;5985 r = -ENOMEM;
5981 goto out_free_3;5986 goto err_vcpu_cache;
5982 }5987 }
5983 5988
5984 for_each_possible_cpu(cpu) {5989 for_each_possible_cpu(cpu) {
5985 if (!alloc_cpumask_var_node(&per_cpu(cpu_kick_mask, cpu),5990 if (!alloc_cpumask_var_node(&per_cpu(cpu_kick_mask, cpu),
5986 GFP_KERNEL, cpu_to_node(cpu))) {5991 GFP_KERNEL, cpu_to_node(cpu))) {
5987 r = -ENOMEM;5992 r = -ENOMEM;
5988 goto out_free_4;5993 goto err_cpu_kick_mask;
5989 }5994 }
5990 }5995 }
5991 5996
5997 r = kvm_irqfd_init();
5998 if (r)
5999 goto err_irqfd;
6000
5992 r = kvm_async_pf_init();6001 r = kvm_async_pf_init();
5993 if (r)6002 if (r)
5994 goto out_free_4;6003 goto err_async_pf;
5995 6004
5996 kvm_chardev_ops.owner = module;6005 kvm_chardev_ops.owner = module;
5997 6006
5998 r = misc_register(&kvm_dev);
5999 if (r) {
6000 pr_err("kvm: misc device register failed\n");
6001 goto out_unreg;
6002 }
6003
6004 register_syscore_ops(&kvm_syscore_ops);
6005
6006 kvm_preempt_ops.sched_in = kvm_sched_in;6007 kvm_preempt_ops.sched_in = kvm_sched_in;
6007 kvm_preempt_ops.sched_out = kvm_sched_out;6008 kvm_preempt_ops.sched_out = kvm_sched_out;
6008 6009
6009 kvm_init_debug();6010 kvm_init_debug();
6010 6011
6011 r = kvm_vfio_ops_init();6012 r = kvm_vfio_ops_init();
6012 WARN_ON(r);6013 if (WARN_ON_ONCE(r))
6014 goto err_vfio;
6015
6016 /*
6017 * Registration _must_ be the very last thing done, as this exposes
6018 * /dev/kvm to userspace, i.e. all infrastructure must be setup!
6019 */
6020 r = misc_register(&kvm_dev);
6021 if (r) {
6022 pr_err("kvm: misc device register failed\n");
6023 goto err_register;
6024 }
6013 6025
6014 return 0;6026 return 0;
6015 6027
6016out_unreg:6028err_register:
6029 kvm_vfio_ops_exit();
6030err_vfio:
6017 kvm_async_pf_deinit();6031 kvm_async_pf_deinit();
6018out_free_4:6032err_async_pf:
6033 kvm_irqfd_exit();
6034err_irqfd:
6035err_cpu_kick_mask:
6019 for_each_possible_cpu(cpu)6036 for_each_possible_cpu(cpu)
6020 free_cpumask_var(per_cpu(cpu_kick_mask, cpu));6037 free_cpumask_var(per_cpu(cpu_kick_mask, cpu));
6021 kmem_cache_destroy(kvm_vcpu_cache);6038 kmem_cache_destroy(kvm_vcpu_cache);
6022out_free_3:6039err_vcpu_cache:
6040#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING
6041 unregister_syscore_ops(&kvm_syscore_ops);
6023 unregister_reboot_notifier(&kvm_reboot_notifier);6042 unregister_reboot_notifier(&kvm_reboot_notifier);
6024 cpuhp_remove_state_nocalls(CPUHP_AP_KVM_STARTING);6043 cpuhp_remove_state_nocalls(CPUHP_AP_KVM_ONLINE);
6025out_free_2:6044#endif
6026 kvm_arch_hardware_unsetup();
6027out_free_1:
6028 free_cpumask_var(cpus_hardware_enabled);
6029out_free_0:
6030 kvm_irqfd_exit();
6031out_irqfd:
6032 kvm_arch_exit();
6033out_fail:
6034 return r;6045 return r;
6035}6046}
6036EXPORT_SYMBOL_GPL(kvm_init);6047EXPORT_SYMBOL_GPL(kvm_init);
@@ -6039,21 +6050,25 @@ void kvm_exit(void) · 2 unmodified lines
6039{6050{
6040 int cpu;6051 int cpu;
6041 6052
6042 debugfs_remove_recursive(kvm_debugfs_dir);6053 /*
6054 * Note, unregistering /dev/kvm doesn't strictly need to come first,
6055 * fops_get(), a.k.a. try_module_get(), prevents acquiring references
6056 * to KVM while the module is being stopped.
6057 */
6043 misc_deregister(&kvm_dev);6058 misc_deregister(&kvm_dev);
6059
6060 debugfs_remove_recursive(kvm_debugfs_dir);
6044 for_each_possible_cpu(cpu)6061 for_each_possible_cpu(cpu)
6045 free_cpumask_var(per_cpu(cpu_kick_mask, cpu));6062 free_cpumask_var(per_cpu(cpu_kick_mask, cpu));
6046 kmem_cache_destroy(kvm_vcpu_cache);6063 kmem_cache_destroy(kvm_vcpu_cache);
6064 kvm_vfio_ops_exit();
6047 kvm_async_pf_deinit();6065 kvm_async_pf_deinit();
6066#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING
6048 unregister_syscore_ops(&kvm_syscore_ops);6067 unregister_syscore_ops(&kvm_syscore_ops);
6049 unregister_reboot_notifier(&kvm_reboot_notifier);6068 unregister_reboot_notifier(&kvm_reboot_notifier);
6050 cpuhp_remove_state_nocalls(CPUHP_AP_KVM_STARTING);6069 cpuhp_remove_state_nocalls(CPUHP_AP_KVM_ONLINE);
6051 on_each_cpu(hardware_disable_nolock, NULL, 1);6070#endif
6052 kvm_arch_hardware_unsetup();
6053 kvm_arch_exit();
6054 kvm_irqfd_exit();6071 kvm_irqfd_exit();
6055 free_cpumask_var(cpus_hardware_enabled);
6056 kvm_vfio_ops_exit();
6057}6072}
6058EXPORT_SYMBOL_GPL(kvm_exit);6073EXPORT_SYMBOL_GPL(kvm_exit);
6059 6074

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t