Pull kvm fixes from Paolo Bonzini: "arm64: - Fix a tiny buglet when propagating the deactivation of an interrupt from a nested guest, which happened to trigger a gold plated CPU bug on a particular implementation - Fix a race between LPI unmapping and mapping, resulting in leaked LPIs - Make LPI mapping more robust on memory allocation failure - Fix the handling of the EL2 tracing clock being disabled - A couple of Sashiko-driven fixes for corner cases in the EL2 tracing code - Add missing sysreg tracepoint for the EL2 code - Tidy-up the mutual exclusion of guest-memfd and MTE - Update Fuad's email address to point to @linux.dev s390: - several fixes for PCI passthru in s390 kvm - fix a 7.2-rc regression in the adapter interrupt mapping code x86: - Add memory clobber to asm for VMX instructions; without one, the compiler could reorder them in troublesome ways because "asm volatile" and "asm goto" only protect against removal of the asm. - Cancel delayed I/O APIC EOI handling before destroying vCPUs - Check all address spaces (normal and SMM) for write tracking and large pages, not just the current one. - Always update x2APIC MSR intercepts for L1 when AVIC is deactivated, even if not running L1. If the deactivation is VM-wide rather than being caused by something in L2's vCPU state, after a nested vmexit L1 will be able to access the host's APIC state" * tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (21 commits) KVM: SVM: Update x2APIC MSR intercepts if AVIC is inhibited while L2 is active KVM: x86/mmu: Check all address spaces before skipping unsync KVM: x86/mmu: Check write tracking in all address spaces KVM: x86: Cancel delayed I/O APIC EOI handling before destroying vCPUs KVM: VMX: add memory clobber to asm for VMX instructions KVM: s390: Fall back to short-term pinning in MAP ioctl KVM: s390: pci: Validate AIBV and AISB before pinning guest pages KVM: s390: pci: Fix resource leak on IRQ registration failure KVM: s390: pci: Fix NULL dereference on AIBV allocation failure KVM: s390: pci: Fix missing error codes and memory unaccounting KVM: s390: pci: Fix memory accounting for pinned/unpinned pages KVM: s390: pci: Reject adapter interrupt forwarding if already enabled KVM: arm64: Reject guest_memfd memslots when the VM has MTE KVM: arm64: Add missing hyp_enter when trapping sysreg KVM: arm64: Fix hyp_trace_desc allocation size in hyp_trace_load() KVM: arm64: Fix potential leak in hyp_trace_buffer_alloc_bpages_backing KVM: arm64: Fix hyp_trace clock disabling KVM: arm64: vgic: Mitigate potential LPI registration failure KVM: arm64: vgic: Fix race between LPI release and re-registration KVM: arm64: Update Fuad Tabba's email address ...
Unified · Split
| @@ -296,6 +296,7 @@ Frank Rowand <frowand.list@gmail.com> <frank.rowand@sony.com> · 295 unmodified lines | |||
| 296 | Frank Rowand <frowand.list@gmail.com> <frank.rowand@sonymobile.com> | 296 | Frank Rowand <frowand.list@gmail.com> <frank.rowand@sonymobile.com> |
| 297 | Frank Rowand <frowand.list@gmail.com> <frowand@mvista.com> | 297 | Frank Rowand <frowand.list@gmail.com> <frowand@mvista.com> |
| 298 | Frank Zago <fzago@systemfabricworks.com> | 298 | Frank Zago <fzago@systemfabricworks.com> |
| 299 | Fuad Tabba <fuad.tabba@linux.dev> <tabba@google.com> | ||
| 299 | Gao Xiang <xiang@kernel.org> <gaoxiang25@huawei.com> | 300 | Gao Xiang <xiang@kernel.org> <gaoxiang25@huawei.com> |
| 300 | Gao Xiang <xiang@kernel.org> <hsiangkao@aol.com> | 301 | Gao Xiang <xiang@kernel.org> <hsiangkao@aol.com> |
| 301 | Gao Xiang <xiang@kernel.org> <hsiangkao@linux.alibaba.com> | 302 | Gao Xiang <xiang@kernel.org> <hsiangkao@linux.alibaba.com> |
| @@ -55,10 +55,14 @@ stable kernels. · 54 unmodified lines | |||
| 55 | +----------------+-----------------+-----------------+-----------------------------+ | 55 | +----------------+-----------------+-----------------+-----------------------------+ |
| 56 | | Ampere | AmpereOne | AC03_CPU_38 | AMPERE_ERRATUM_AC03_CPU_38 | | 56 | | Ampere | AmpereOne | AC03_CPU_38 | AMPERE_ERRATUM_AC03_CPU_38 | |
| 57 | +----------------+-----------------+-----------------+-----------------------------+ | 57 | +----------------+-----------------+-----------------+-----------------------------+ |
| 58 | | Ampere | AmpereOne | AC03_CPU_57 | N/A | | ||
| 59 | +----------------+-----------------+-----------------+-----------------------------+ | ||
| 58 | | Ampere | AmpereOne AC04 | AC04_CPU_10 | AMPERE_ERRATUM_AC03_CPU_38 | | 60 | | Ampere | AmpereOne AC04 | AC04_CPU_10 | AMPERE_ERRATUM_AC03_CPU_38 | |
| 59 | +----------------+-----------------+-----------------+-----------------------------+ | 61 | +----------------+-----------------+-----------------+-----------------------------+ |
| 60 | | Ampere | AmpereOne AC04 | AC04_CPU_23 | AMPERE_ERRATUM_AC04_CPU_23 | | 62 | | Ampere | AmpereOne AC04 | AC04_CPU_23 | AMPERE_ERRATUM_AC04_CPU_23 | |
| 61 | +----------------+-----------------+-----------------+-----------------------------+ | 63 | +----------------+-----------------+-----------------+-----------------------------+ |
| 64 | | Ampere | AmpereOne AC04 | AC04_CPU_29 | N/A | | ||
| 65 | +----------------+-----------------+-----------------+-----------------------------+ | ||
| 62 | +----------------+-----------------+-----------------+-----------------------------+ | 66 | +----------------+-----------------+-----------------+-----------------------------+ |
| 63 | | ARM | Cortex-A510 | #2457168 | ARM64_ERRATUM_2457168 | | 67 | | ARM | Cortex-A510 | #2457168 | ARM64_ERRATUM_2457168 | |
| 64 | +----------------+-----------------+-----------------+-----------------------------+ | 68 | +----------------+-----------------+-----------------+-----------------------------+ |
| @@ -8414,6 +8414,12 @@ When this capability is enabled all memory in memslots must be mapped as · 8413 unmodified lines | |||
| 8414 | attempts to create a memslot with an invalid mmap will result in an | 8414 | attempts to create a memslot with an invalid mmap will result in an |
| 8415 | -EINVAL return. | 8415 | -EINVAL return. |
| 8416 | 8416 | ||
| 8417 | ``guest_memfd``, even though it is an anonymous file, is not supported with MTE. | ||
| 8418 | Attempting to create a memslot backed by ``guest_memfd`` when the MTE capability | ||
| 8419 | is enabled, or attempting to enable the MTE capability after | ||
| 8420 | ``guest_memfd``-backed memslots have been created, will result in an -EINVAL | ||
| 8421 | return. | ||
| 8422 | |||
| 8417 | When enabled the VMM may make use of the ``KVM_ARM_MTE_COPY_TAGS`` ioctl to | 8423 | When enabled the VMM may make use of the ``KVM_ARM_MTE_COPY_TAGS`` ioctl to |
| 8418 | perform a bulk copy of tags to/from the guest. | 8424 | perform a bulk copy of tags to/from the guest. |
| 8419 | 8425 | ||
| @@ -112,9 +112,20 @@ Groups: · 111 unmodified lines | |||
| 112 | mask or unmask the adapter, as specified in mask | 112 | mask or unmask the adapter, as specified in mask |
| 113 | 113 | ||
| 114 | KVM_S390_IO_ADAPTER_MAP | 114 | KVM_S390_IO_ADAPTER_MAP |
| 115 | This is now a no-op. The mapping is purely done by the irq route. | 115 | Map an adapter indicator or summary page for long-term pinning so that |
| 116 | interrupt injection can be performed in atomic context. If long-term | ||
| 117 | pinning is not possible (e.g. file-backed memory), the page is verified | ||
| 118 | via a short-term pin and the ioctl returns success; interrupt injection | ||
| 119 | will use the non-atomic irqfd path with short-term pinning on each | ||
| 120 | interrupt. In Secure Execution mode this is a no-op and the ioctl | ||
| 121 | returns success. | ||
| 122 | |||
| 116 | KVM_S390_IO_ADAPTER_UNMAP | 123 | KVM_S390_IO_ADAPTER_UNMAP |
| 117 | This is now a no-op. The mapping is purely done by the irq route. | 124 | Unmap a previously mapped adapter indicator or summary page and release |
| 125 | the long-term pin. If the page was not long-term pinned (e.g. file-backed | ||
| 126 | memory), the map entry is removed and success is returned; if no prior | ||
| 127 | map entry exists, -ENOENT is returned. In Secure Execution mode this is | ||
| 128 | a no-op and the ioctl returns success. | ||
| 118 | 129 | ||
| 119 | KVM_DEV_FLIC_AISM | 130 | KVM_DEV_FLIC_AISM |
| 120 | modify the adapter-interruption-suppression mode for a given isc if the | 131 | modify the adapter-interruption-suppression mode for a given isc if the |
| @@ -14193,7 +14193,7 @@ F: virt/kvm/* · 14192 unmodified lines | |||
| 14193 | KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64) | 14193 | KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64) |
| 14194 | M: Marc Zyngier <maz@kernel.org> | 14194 | M: Marc Zyngier <maz@kernel.org> |
| 14195 | M: Oliver Upton <oupton@kernel.org> | 14195 | M: Oliver Upton <oupton@kernel.org> |
| 14196 | R: Fuad Tabba <tabba@google.com> | 14196 | R: Fuad Tabba <fuad.tabba@linux.dev> |
| 14197 | R: Joey Gouly <joey.gouly@arm.com> | 14197 | R: Joey Gouly <joey.gouly@arm.com> |
| 14198 | R: Steffen Eiden <seiden@linux.ibm.com> | 14198 | R: Steffen Eiden <seiden@linux.ibm.com> |
| 14199 | R: Suzuki K Poulose <suzuki.poulose@arm.com> | 14199 | R: Suzuki K Poulose <suzuki.poulose@arm.com> |
| @@ -12,6 +12,7 @@ · 11 unmodified lines | |||
| 12 | enum hyp_enter_exit_reason { | 12 | enum hyp_enter_exit_reason { |
| 13 | HYP_REASON_SMC, | 13 | HYP_REASON_SMC, |
| 14 | HYP_REASON_HVC, | 14 | HYP_REASON_HVC, |
| 15 | HYP_REASON_SYS, | ||
| 15 | HYP_REASON_PSCI, | 16 | HYP_REASON_PSCI, |
| 16 | HYP_REASON_HOST_ABORT, | 17 | HYP_REASON_HOST_ABORT, |
| 17 | HYP_REASON_GUEST_EXIT, | 18 | HYP_REASON_GUEST_EXIT, |
| @@ -149,14 +149,27 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · 148 unmodified lines | |||
| 149 | set_bit(KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER, | 149 | set_bit(KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER, |
| 150 | &kvm->arch.flags); | 150 | &kvm->arch.flags); |
| 151 | break; | 151 | break; |
| 152 | case KVM_CAP_ARM_MTE: | 152 | case KVM_CAP_ARM_MTE: { |
| 153 | mutex_lock(&kvm->lock); | 153 | struct kvm_memory_slot *memslot; |
| 154 | if (system_supports_mte() && !kvm->created_vcpus) { | 154 | int bkt; |
| 155 | r = 0; | 155 | |
| 156 | set_bit(KVM_ARCH_FLAG_MTE_ENABLED, &kvm->arch.flags); | 156 | guard(mutex)(&kvm->lock); |
| 157 | if (!system_supports_mte() || kvm->created_vcpus) | ||
| 158 | break; | ||
| 159 | |||
| 160 | r = 0; | ||
| 161 | guard(mutex)(&kvm->slots_lock); | ||
| 162 | kvm_for_each_memslot(memslot, bkt, kvm_memslots(kvm)) { | ||
| 163 | if (kvm_slot_has_gmem(memslot)) { | ||
| 164 | r = -EINVAL; | ||
| 165 | break; | ||
| 166 | } | ||
| 157 | } | 167 | } |
| 158 | mutex_unlock(&kvm->lock); | 168 | if (r == 0) |
| 169 | set_bit(KVM_ARCH_FLAG_MTE_ENABLED, &kvm->arch.flags); | ||
| 159 | break; | 170 | break; |
| 171 | |||
| 172 | } | ||
| 160 | case KVM_CAP_ARM_SYSTEM_SUSPEND: | 173 | case KVM_CAP_ARM_SYSTEM_SUSPEND: |
| 161 | r = 0; | 174 | r = 0; |
| 162 | set_bit(KVM_ARCH_FLAG_SYSTEM_SUSPEND_ENABLED, &kvm->arch.flags); | 175 | set_bit(KVM_ARCH_FLAG_SYSTEM_SUSPEND_ENABLED, &kvm->arch.flags); |
| @@ -932,6 +932,7 @@ void handle_trap(struct kvm_cpu_context *host_ctxt) · 931 unmodified lines | |||
| 932 | handle_host_mem_abort(host_ctxt); | 932 | handle_host_mem_abort(host_ctxt); |
| 933 | break; | 933 | break; |
| 934 | case ESR_ELx_EC_SYS64: | 934 | case ESR_ELx_EC_SYS64: |
| 935 | trace_hyp_enter(host_ctxt, HYP_REASON_SYS); | ||
| 935 | if (handle_host_mte(esr)) | 936 | if (handle_host_mte(esr)) |
| 936 | break; | 937 | break; |
| 937 | fallthrough; | 938 | fallthrough; |
| @@ -37,8 +37,6 @@ static struct hyp_trace_clock { · 36 unmodified lines | |||
| 37 | u32 shift; | 37 | u32 shift; |
| 38 | struct delayed_work work; | 38 | struct delayed_work work; |
| 39 | struct completion ready; | 39 | struct completion ready; |
| 40 | struct mutex lock; | ||
| 41 | bool running; | ||
| 42 | } hyp_clock; | 40 | } hyp_clock; |
| 43 | 41 | ||
| 44 | static void __hyp_clock_work(struct work_struct *work) | 42 | static void __hyp_clock_work(struct work_struct *work) |
| @@ -110,12 +108,9 @@ static void hyp_trace_clock_enable(struct hyp_trace_clock *hyp_clock, bool enabl · 65 unmodified lines | |||
| 110 | { | 108 | { |
| 111 | struct system_time_snapshot snap; | 109 | struct system_time_snapshot snap; |
| 112 | 110 | ||
| 113 | if (hyp_clock->running == enable) | ||
| 114 | return; | ||
| 115 | |||
| 116 | if (!enable) { | 111 | if (!enable) { |
| 117 | cancel_delayed_work_sync(&hyp_clock->work); | 112 | cancel_delayed_work_sync(&hyp_clock->work); |
| 118 | hyp_clock->running = false; | 113 | return; |
| 119 | } | 114 | } |
| 120 | 115 | ||
| 121 | ktime_get_snapshot_id(CLOCK_BOOTTIME, &snap); | 116 | ktime_get_snapshot_id(CLOCK_BOOTTIME, &snap); |
| @@ -128,7 +123,6 @@ static void hyp_trace_clock_enable(struct hyp_trace_clock *hyp_clock, bool enabl · 6 unmodified lines | |||
| 128 | INIT_DELAYED_WORK(&hyp_clock->work, __hyp_clock_work); | 123 | INIT_DELAYED_WORK(&hyp_clock->work, __hyp_clock_work); |
| 129 | schedule_delayed_work(&hyp_clock->work, msecs_to_jiffies(CLOCK_INIT_MS)); | 124 | schedule_delayed_work(&hyp_clock->work, msecs_to_jiffies(CLOCK_INIT_MS)); |
| 130 | wait_for_completion(&hyp_clock->ready); | 125 | wait_for_completion(&hyp_clock->ready); |
| 131 | hyp_clock->running = true; | ||
| 132 | } | 126 | } |
| 133 | 127 | ||
| 134 | /* Access to this struct within the trace_remote_callbacks are protected by the trace_remote lock */ | 128 | /* Access to this struct within the trace_remote_callbacks are protected by the trace_remote lock */ |
| @@ -160,6 +154,7 @@ static int hyp_trace_buffer_alloc_bpages_backing(struct hyp_trace_buffer *trace_ · 25 unmodified lines | |||
| 160 | int nr_bpages = (PAGE_ALIGN(size) / PAGE_SIZE) + 1; | 154 | int nr_bpages = (PAGE_ALIGN(size) / PAGE_SIZE) + 1; |
| 161 | size_t backing_size; | 155 | size_t backing_size; |
| 162 | void *start; | 156 | void *start; |
| 157 | int ret; | ||
| 163 | 158 | ||
| 164 | backing_size = PAGE_ALIGN(sizeof(struct simple_buffer_page) * nr_bpages * | 159 | backing_size = PAGE_ALIGN(sizeof(struct simple_buffer_page) * nr_bpages * |
| 165 | num_possible_cpus()); | 160 | num_possible_cpus()); |
| @@ -168,10 +163,16 @@ static int hyp_trace_buffer_alloc_bpages_backing(struct hyp_trace_buffer *trace_ · 2 unmodified lines | |||
| 168 | if (!start) | 163 | if (!start) |
| 169 | return -ENOMEM; | 164 | return -ENOMEM; |
| 170 | 165 | ||
| 166 | ret = __map_hyp(start, backing_size); | ||
| 167 | if (ret) { | ||
| 168 | free_pages_exact(start, backing_size); | ||
| 169 | return ret; | ||
| 170 | } | ||
| 171 | |||
| 171 | trace_buffer->desc->bpages_backing_start = (unsigned long)start; | 172 | trace_buffer->desc->bpages_backing_start = (unsigned long)start; |
| 172 | trace_buffer->desc->bpages_backing_size = backing_size; | 173 | trace_buffer->desc->bpages_backing_size = backing_size; |
| 173 | 174 | ||
| 174 | return __map_hyp(start, backing_size); | 175 | return ret; |
| 175 | } | 176 | } |
| 176 | 177 | ||
| 177 | static void hyp_trace_buffer_free_bpages_backing(struct hyp_trace_buffer *trace_buffer) | 178 | static void hyp_trace_buffer_free_bpages_backing(struct hyp_trace_buffer *trace_buffer) |
| @@ -228,18 +229,22 @@ static int hyp_trace_buffer_share_hyp(struct hyp_trace_buffer *trace_buffer) · 50 unmodified lines | |||
| 228 | static struct trace_buffer_desc *hyp_trace_load(unsigned long size, void *priv) | 229 | static struct trace_buffer_desc *hyp_trace_load(unsigned long size, void *priv) |
| 229 | { | 230 | { |
| 230 | struct hyp_trace_buffer *trace_buffer = priv; | 231 | struct hyp_trace_buffer *trace_buffer = priv; |
| 232 | size_t desc_size, tb_desc_size; | ||
| 231 | struct hyp_trace_desc *desc; | 233 | struct hyp_trace_desc *desc; |
| 232 | size_t desc_size; | ||
| 233 | int ret; | 234 | int ret; |
| 234 | 235 | ||
| 235 | if (WARN_ON(trace_buffer->desc)) | 236 | if (WARN_ON(trace_buffer->desc)) |
| 236 | return ERR_PTR(-EINVAL); | 237 | return ERR_PTR(-EINVAL); |
| 237 | 238 | ||
| 238 | desc_size = trace_buffer_desc_size(size, num_possible_cpus()); | 239 | tb_desc_size = trace_buffer_desc_size(size, num_possible_cpus()); |
| 240 | desc_size = size_add(tb_desc_size, offsetof(struct hyp_trace_desc, trace_buffer_desc)); | ||
| 239 | if (desc_size == SIZE_MAX) | 241 | if (desc_size == SIZE_MAX) |
| 240 | return ERR_PTR(-E2BIG); | 242 | return ERR_PTR(-E2BIG); |
| 241 | 243 | ||
| 242 | desc_size = PAGE_ALIGN(desc_size); | 244 | desc_size = PAGE_ALIGN(desc_size); |
| 245 | if (!desc_size) | ||
| 246 | return ERR_PTR(-E2BIG); | ||
| 247 | |||
| 243 | desc = (struct hyp_trace_desc *)alloc_pages_exact(desc_size, GFP_KERNEL); | 248 | desc = (struct hyp_trace_desc *)alloc_pages_exact(desc_size, GFP_KERNEL); |
| 244 | if (!desc) | 249 | if (!desc) |
| 245 | return ERR_PTR(-ENOMEM); | 250 | return ERR_PTR(-ENOMEM); |
| @@ -255,7 +260,7 @@ static struct trace_buffer_desc *hyp_trace_load(unsigned long size, void *priv) · 9 unmodified lines | |||
| 255 | if (ret) | 260 | if (ret) |
| 256 | goto err_free_desc; | 261 | goto err_free_desc; |
| 257 | 262 | ||
| 258 | ret = trace_remote_alloc_buffer(&desc->trace_buffer_desc, desc_size, size, | 263 | ret = trace_remote_alloc_buffer(&desc->trace_buffer_desc, tb_desc_size, size, |
| 259 | cpu_possible_mask); | 264 | cpu_possible_mask); |
| 260 | if (ret) | 265 | if (ret) |
| 261 | goto err_free_backing; | 266 | goto err_free_backing; |
| @@ -304,9 +309,15 @@ static void hyp_trace_unload(struct trace_buffer_desc *desc, void *priv) · 42 unmodified lines | |||
| 304 | 309 | ||
| 305 | static int hyp_trace_enable_tracing(bool enable, void *priv) | 310 | static int hyp_trace_enable_tracing(bool enable, void *priv) |
| 306 | { | 311 | { |
| 312 | int ret; | ||
| 313 | |||
| 307 | hyp_trace_clock_enable(&hyp_clock, enable); | 314 | hyp_trace_clock_enable(&hyp_clock, enable); |
| 308 | 315 | ||
| 309 | return kvm_call_hyp_nvhe(__tracing_enable, enable); | 316 | ret = kvm_call_hyp_nvhe(__tracing_enable, enable); |
| 317 | if (ret) | ||
| 318 | hyp_trace_clock_enable(&hyp_clock, !enable); | ||
| 319 | |||
| 320 | return ret; | ||
| 310 | } | 321 | } |
| 311 | 322 | ||
| 312 | static int hyp_trace_swap_reader_page(unsigned int cpu, void *priv) | 323 | static int hyp_trace_swap_reader_page(unsigned int cpu, void *priv) |
| @@ -398,6 +409,7 @@ static const char *__hyp_enter_exit_reason_str(u8 reason) · 85 unmodified lines | |||
| 398 | static const char strs[][12] = { | 409 | static const char strs[][12] = { |
| 399 | "smc", | 410 | "smc", |
| 400 | "hvc", | 411 | "hvc", |
| 412 | "sys", | ||
| 401 | "psci", | 413 | "psci", |
| 402 | "host_abort", | 414 | "host_abort", |
| 403 | "guest_exit", | 415 | "guest_exit", |
| @@ -2652,6 +2652,10 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, · 2651 unmodified lines | |||
| 2652 | if (kvm_slot_has_gmem(new) && !kvm_memslot_is_gmem_only(new)) | 2652 | if (kvm_slot_has_gmem(new) && !kvm_memslot_is_gmem_only(new)) |
| 2653 | return -EINVAL; | 2653 | return -EINVAL; |
| 2654 | 2654 | ||
| 2655 | /* guest_memfd is incompatible with MTE. */ | ||
| 2656 | if (kvm_slot_has_gmem(new) && kvm_has_mte(kvm)) | ||
| 2657 | return -EINVAL; | ||
| 2658 | |||
| 2655 | hva = new->userspace_addr; | 2659 | hva = new->userspace_addr; |
| 2656 | reg_end = hva + (new->npages << PAGE_SHIFT); | 2660 | reg_end = hva + (new->npages << PAGE_SHIFT); |
| 2657 | 2661 | ||
| @@ -116,18 +116,27 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, · 115 unmodified lines | |||
| 116 | kfree(irq); | 116 | kfree(irq); |
| 117 | irq = oldirq; | 117 | irq = oldirq; |
| 118 | } else { | 118 | } else { |
| 119 | ret = xa_err(__xa_store(&dist->lpi_xa, intid, irq, 0)); | 119 | /* |
| 120 | * The entry is either empty or contains a dead LPI (refcount=0) | ||
| 121 | * from the deferred release path, pending cleanup by | ||
| 122 | * vgic_release_deleted_lpis(). Evict and free it if present. | ||
| 123 | */ | ||
| 124 | oldirq = __xa_store(&dist->lpi_xa, intid, irq, | ||
| 125 | GFP_NOWAIT | __GFP_ACCOUNT); | ||
| 126 | ret = xa_err(oldirq); | ||
| 127 | if (ret) { | ||
| 128 | xa_unlock_irqrestore(&dist->lpi_xa, flags); | ||
| 129 | kfree(irq); | ||
| 130 | |||
| 131 | return ERR_PTR(ret); | ||
| 132 | } | ||
| 133 | |||
| 134 | if (oldirq && !WARN_ON_ONCE(refcount_read(&oldirq->refcount))) | ||
| 135 | kfree_rcu(oldirq, rcu); | ||
| 120 | } | 136 | } |
| 121 | 137 | ||
| 122 | xa_unlock_irqrestore(&dist->lpi_xa, flags); | 138 | xa_unlock_irqrestore(&dist->lpi_xa, flags); |
| 123 | 139 | ||
| 124 | if (ret) { | ||
| 125 | xa_release(&dist->lpi_xa, intid); | ||
| 126 | kfree(irq); | ||
| 127 | |||
| 128 | return ERR_PTR(ret); | ||
| 129 | } | ||
| 130 | |||
| 131 | /* | 140 | /* |
| 132 | * We "cache" the configuration table entries in our struct vgic_irq's. | 141 | * We "cache" the configuration table entries in our struct vgic_irq's. |
| 133 | * However we only have those structs for mapped IRQs, so we read in | 142 | * However we only have those structs for mapped IRQs, so we read in |
| @@ -275,7 +275,13 @@ void vgic_v3_deactivate(struct kvm_vcpu *vcpu, u64 val) · 274 unmodified lines | |||
| 275 | lr = vgic_v3_compute_lr(vcpu, irq) & ~ICH_LR_ACTIVE_BIT; | 275 | lr = vgic_v3_compute_lr(vcpu, irq) & ~ICH_LR_ACTIVE_BIT; |
| 276 | } | 276 | } |
| 277 | 277 | ||
| 278 | if (lr & ICH_LR_HW) | 278 | /* |
| 279 | * In the nested state, the irq has already been deactivated via the HW | ||
| 280 | * bit in the LR. Deactivating again would be harmless except AmpereOne | ||
| 281 | * errata AC03_CPU_57, AC04_CPU_29 could cause irq delivery to break if | ||
| 282 | * the deactivation hits the highest priority pending irq. | ||
| 283 | */ | ||
| 284 | if ((lr & ICH_LR_HW) && !vgic_state_is_nested(vcpu)) | ||
| 279 | vgic_v3_deactivate_phys(FIELD_GET(ICH_LR_PHYS_ID_MASK, lr)); | 285 | vgic_v3_deactivate_phys(FIELD_GET(ICH_LR_PHYS_ID_MASK, lr)); |
| 280 | 286 | ||
| 281 | vgic_v3_fold_lr(vcpu, lr); | 287 | vgic_v3_fold_lr(vcpu, lr); |
| @@ -146,11 +146,7 @@ static __must_check bool __vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq) · 145 unmodified lines | |||
| 146 | 146 | ||
| 147 | static __must_check bool vgic_put_irq_norelease(struct kvm *kvm, struct vgic_irq *irq) | 147 | static __must_check bool vgic_put_irq_norelease(struct kvm *kvm, struct vgic_irq *irq) |
| 148 | { | 148 | { |
| 149 | if (!__vgic_put_irq(kvm, irq)) | 149 | return __vgic_put_irq(kvm, irq); |
| 150 | return false; | ||
| 151 | |||
| 152 | irq->pending_release = true; | ||
| 153 | return true; | ||
| 154 | } | 150 | } |
| 155 | 151 | ||
| 156 | void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq) | 152 | void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq) |
| @@ -167,12 +163,14 @@ void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq) · 10 unmodified lines | |||
| 167 | guard(spinlock_irqsave)(&dist->lpi_xa.xa_lock); | 163 | guard(spinlock_irqsave)(&dist->lpi_xa.xa_lock); |
| 168 | } | 164 | } |
| 169 | 165 | ||
| 170 | if (!__vgic_put_irq(kvm, irq)) | 166 | if (!irq_is_lpi(kvm, irq->intid)) |
| 171 | return; | 167 | return; |
| 172 | 168 | ||
| 173 | xa_lock_irqsave(&dist->lpi_xa, flags); | 169 | if (refcount_dec_and_lock_irqsave(&irq->refcount, |
| 174 | vgic_release_lpi_locked(dist, irq); | 170 | &dist->lpi_xa.xa_lock, &flags)) { |
| 175 | xa_unlock_irqrestore(&dist->lpi_xa, flags); | 171 | vgic_release_lpi_locked(dist, irq); |
| 172 | xa_unlock_irqrestore(&dist->lpi_xa, flags); | ||
| 173 | } | ||
| 176 | } | 174 | } |
| 177 | 175 | ||
| 178 | static void vgic_release_deleted_lpis(struct kvm *kvm) | 176 | static void vgic_release_deleted_lpis(struct kvm *kvm) |
| @@ -184,7 +182,7 @@ static void vgic_release_deleted_lpis(struct kvm *kvm) · 5 unmodified lines | |||
| 184 | xa_lock_irqsave(&dist->lpi_xa, flags); | 182 | xa_lock_irqsave(&dist->lpi_xa, flags); |
| 185 | 183 | ||
| 186 | xa_for_each(&dist->lpi_xa, intid, irq) { | 184 | xa_for_each(&dist->lpi_xa, intid, irq) { |
| 187 | if (irq->pending_release) | 185 | if (!refcount_read(&irq->refcount)) |
| 188 | vgic_release_lpi_locked(dist, irq); | 186 | vgic_release_lpi_locked(dist, irq); |
| 189 | } | 187 | } |
| 190 | 188 | ||
| @@ -476,6 +476,11 @@ struct s390_map_info { · 475 unmodified lines | |||
| 476 | __u64 guest_addr; | 476 | __u64 guest_addr; |
| 477 | __u64 addr; | 477 | __u64 addr; |
| 478 | struct page *page; | 478 | struct page *page; |
| 479 | /* | ||
| 480 | * True if the page is long-term pinned. False if long-term pinning | ||
| 481 | * failed and this entry exists only to preserve MAP/UNMAP symmetry. | ||
| 482 | */ | ||
| 483 | bool pinned; | ||
| 479 | }; | 484 | }; |
| 480 | 485 | ||
| 481 | struct s390_io_adapter { | 486 | struct s390_io_adapter { |
| @@ -2520,8 +2520,22 @@ static int kvm_s390_adapter_map(struct kvm *kvm, unsigned int id, __u64 addr) · 2519 unmodified lines | |||
| 2520 | map->addr = host_addr; | 2520 | map->addr = host_addr; |
| 2521 | map->page = pin_map_page(kvm, host_addr, FOLL_LONGTERM); | 2521 | map->page = pin_map_page(kvm, host_addr, FOLL_LONGTERM); |
| 2522 | if (!map->page) { | 2522 | if (!map->page) { |
| 2523 | ret = -EINVAL; | 2523 | /* |
| 2524 | goto out; | 2524 | * Long-term pinning may fail for memory types such as file-backed |
| 2525 | * memory. Verify that short-term pinning succeeds so that the | ||
| 2526 | * non-atomic irqfd path can handle interrupt injection. | ||
| 2527 | */ | ||
| 2528 | map->page = pin_map_page(kvm, host_addr, 0); | ||
| 2529 | if (!map->page) { | ||
| 2530 | ret = -EINVAL; | ||
| 2531 | goto out; | ||
| 2532 | } | ||
| 2533 | unpin_user_page(map->page); | ||
| 2534 | map->page = NULL; | ||
| 2535 | map->pinned = false; | ||
| 2536 | /* Add an entry to preserve MAP/UNMAP symmetry. */ | ||
| 2537 | } else { | ||
| 2538 | map->pinned = true; | ||
| 2525 | } | 2539 | } |
| 2526 | spin_lock_irqsave(&adapter->maps_lock, flags); | 2540 | spin_lock_irqsave(&adapter->maps_lock, flags); |
| 2527 | if (adapter->nr_maps < MAX_S390_ADAPTER_MAPS) { | 2541 | if (adapter->nr_maps < MAX_S390_ADAPTER_MAPS) { |
| @@ -2532,7 +2546,7 @@ static int kvm_s390_adapter_map(struct kvm *kvm, unsigned int id, __u64 addr) · 4 unmodified lines | |||
| 2532 | ret = -EINVAL; | 2546 | ret = -EINVAL; |
| 2533 | } | 2547 | } |
| 2534 | spin_unlock_irqrestore(&adapter->maps_lock, flags); | 2548 | spin_unlock_irqrestore(&adapter->maps_lock, flags); |
| 2535 | if (ret) | 2549 | if (ret && map->page) |
| 2536 | unpin_user_page(map->page); | 2550 | unpin_user_page(map->page); |
| 2537 | out: | 2551 | out: |
| 2538 | if (ret) | 2552 | if (ret) |
| @@ -2546,6 +2560,7 @@ static int kvm_s390_adapter_unmap(struct kvm *kvm, unsigned int id, __u64 addr) · 7 unmodified lines | |||
| 2546 | struct s390_map_info *map, *tmp, *map_to_free; | 2560 | struct s390_map_info *map, *tmp, *map_to_free; |
| 2547 | struct page *map_page_to_put = NULL; | 2561 | struct page *map_page_to_put = NULL; |
| 2548 | u64 map_addr_to_mark = 0; | 2562 | u64 map_addr_to_mark = 0; |
| 2563 | bool map_pinned = false; | ||
| 2549 | unsigned long flags; | 2564 | unsigned long flags; |
| 2550 | int found = 0, idx; | 2565 | int found = 0, idx; |
| 2551 | 2566 | ||
| @@ -2560,6 +2575,7 @@ static int kvm_s390_adapter_unmap(struct kvm *kvm, unsigned int id, __u64 addr) · 8 unmodified lines | |||
| 2560 | list_del(&map->list); | 2575 | list_del(&map->list); |
| 2561 | map_page_to_put = map->page; | 2576 | map_page_to_put = map->page; |
| 2562 | map_addr_to_mark = map->guest_addr; | 2577 | map_addr_to_mark = map->guest_addr; |
| 2578 | map_pinned = map->pinned; | ||
| 2563 | map_to_free = map; | 2579 | map_to_free = map; |
| 2564 | break; | 2580 | break; |
| 2565 | } | 2581 | } |
| @@ -2568,11 +2584,18 @@ static int kvm_s390_adapter_unmap(struct kvm *kvm, unsigned int id, __u64 addr) · 2 unmodified lines | |||
| 2568 | 2584 | ||
| 2569 | if (found) { | 2585 | if (found) { |
| 2570 | kfree(map_to_free); | 2586 | kfree(map_to_free); |
| 2571 | idx = srcu_read_lock(&kvm->srcu); | 2587 | if (map_pinned) { |
| 2572 | mark_page_dirty(kvm, map_addr_to_mark >> PAGE_SHIFT); | 2588 | /* |
| 2573 | set_page_dirty_lock(map_page_to_put); | 2589 | * Only long-term pinned pages need to be marked dirty |
| 2574 | srcu_read_unlock(&kvm->srcu, idx); | 2590 | * and released. Fallback entries exist only for |
| 2575 | unpin_user_page(map_page_to_put); | 2591 | * MAP/UNMAP symmetry. |
| 2592 | */ | ||
| 2593 | idx = srcu_read_lock(&kvm->srcu); | ||
| 2594 | mark_page_dirty(kvm, map_addr_to_mark >> PAGE_SHIFT); | ||
| 2595 | set_page_dirty_lock(map_page_to_put); | ||
| 2596 | srcu_read_unlock(&kvm->srcu, idx); | ||
| 2597 | unpin_user_page(map_page_to_put); | ||
| 2598 | } | ||
| 2576 | } | 2599 | } |
| 2577 | 2600 | ||
| 2578 | return found ? 0 : -ENOENT; | 2601 | return found ? 0 : -ENOENT; |
| @@ -2598,11 +2621,13 @@ void kvm_s390_unmap_all_adapters(struct kvm *kvm) · 19 unmodified lines | |||
| 2598 | 2621 | ||
| 2599 | list_for_each_entry_safe(map, tmp, &local_list, list) { | 2622 | list_for_each_entry_safe(map, tmp, &local_list, list) { |
| 2600 | list_del(&map->list); | 2623 | list_del(&map->list); |
| 2601 | idx = srcu_read_lock(&kvm->srcu); | 2624 | if (map->pinned) { |
| 2602 | mark_page_dirty(kvm, map->guest_addr >> PAGE_SHIFT); | 2625 | idx = srcu_read_lock(&kvm->srcu); |
| 2603 | set_page_dirty_lock(map->page); | 2626 | mark_page_dirty(kvm, map->guest_addr >> PAGE_SHIFT); |
| 2604 | srcu_read_unlock(&kvm->srcu, idx); | 2627 | set_page_dirty_lock(map->page); |
| 2605 | unpin_user_page(map->page); | 2628 | srcu_read_unlock(&kvm->srcu, idx); |
| 2629 | unpin_user_page(map->page); | ||
| 2630 | } | ||
| 2606 | kfree(map); | 2631 | kfree(map); |
| 2607 | } | 2632 | } |
| 2608 | } | 2633 | } |
| @@ -2929,8 +2954,11 @@ static struct s390_map_info *get_map_info(struct s390_io_adapter *adapter, · 320 unmodified lines | |||
| 2929 | return NULL; | 2954 | return NULL; |
| 2930 | 2955 | ||
| 2931 | list_for_each_entry(map, &adapter->maps, list) { | 2956 | list_for_each_entry(map, &adapter->maps, list) { |
| 2932 | if (map->addr == addr) | 2957 | if (map->addr == addr) { |
| 2958 | if (!map->pinned) | ||
| 2959 | return NULL; | ||
| 2933 | return map; | 2960 | return map; |
| 2961 | } | ||
| 2934 | } | 2962 | } |
| 2935 | return NULL; | 2963 | return NULL; |
| 2936 | } | 2964 | } |
| @@ -190,40 +190,61 @@ static int kvm_zpci_clear_airq(struct zpci_dev *zdev) · 189 unmodified lines | |||
| 190 | return cc ? -EIO : 0; | 190 | return cc ? -EIO : 0; |
| 191 | } | 191 | } |
| 192 | 192 | ||
| 193 | static inline void unaccount_mem(unsigned long nr_pages) | 193 | static inline void unaccount_mem(struct kvm_zdev *kzdev, unsigned long nr_pages) |
| 194 | { | 194 | { |
| 195 | struct user_struct *user = get_uid(current_user()); | 195 | struct user_struct *user = kzdev->user_account; |
| 196 | struct mm_struct *mm_account = kzdev->mm_account; | ||
| 196 | 197 | ||
| 197 | if (user) | 198 | if (user) { |
| 198 | atomic_long_sub(nr_pages, &user->locked_vm); | 199 | atomic_long_sub(nr_pages, &user->locked_vm); |
| 199 | if (current->mm) | 200 | free_uid(user); |
| 200 | atomic64_sub(nr_pages, ¤t->mm->pinned_vm); | 201 | kzdev->user_account = NULL; |
| 202 | } | ||
| 203 | |||
| 204 | if (mm_account) { | ||
| 205 | atomic64_sub(nr_pages, &mm_account->pinned_vm); | ||
| 206 | mmdrop(mm_account); | ||
| 207 | kzdev->mm_account = NULL; | ||
| 208 | } | ||
| 201 | } | 209 | } |
| 202 | 210 | ||
| 203 | static inline int account_mem(unsigned long nr_pages) | 211 | static inline int account_mem(struct kvm_zdev *kzdev, unsigned long nr_pages) |
| 204 | { | 212 | { |
| 205 | struct user_struct *user = get_uid(current_user()); | 213 | struct user_struct *user = get_uid(current_user()); |
| 206 | unsigned long page_limit, cur_pages, new_pages; | 214 | unsigned long page_limit, cur_pages, new_pages; |
| 215 | int rc = 0; | ||
| 207 | 216 | ||
| 208 | page_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; | 217 | page_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; |
| 209 | 218 | ||
| 210 | cur_pages = atomic_long_read(&user->locked_vm); | 219 | cur_pages = atomic_long_read(&user->locked_vm); |
| 211 | do { | 220 | do { |
| 212 | new_pages = cur_pages + nr_pages; | 221 | new_pages = cur_pages + nr_pages; |
| 213 | if (new_pages > page_limit) | 222 | if (new_pages > page_limit) { |
| 214 | return -ENOMEM; | 223 | rc = -ENOMEM; |
| 224 | goto out; | ||
| 225 | } | ||
| 215 | } while (!atomic_long_try_cmpxchg(&user->locked_vm, &cur_pages, new_pages)); | 226 | } while (!atomic_long_try_cmpxchg(&user->locked_vm, &cur_pages, new_pages)); |
| 216 | 227 | ||
| 217 | atomic64_add(nr_pages, ¤t->mm->pinned_vm); | 228 | if (current->mm) { |
| 229 | mmgrab(current->mm); | ||
| 230 | atomic64_add(nr_pages, ¤t->mm->pinned_vm); | ||
| 231 | } | ||
| 232 | |||
| 233 | kzdev->user_account = user; | ||
| 234 | kzdev->mm_account = current->mm; | ||
| 218 | 235 | ||
| 219 | return 0; | 236 | return 0; |
| 237 | |||
| 238 | out: | ||
| 239 | free_uid(user); | ||
| 240 | return rc; | ||
| 220 | } | 241 | } |
| 221 | 242 | ||
| 222 | static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib, | 243 | static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib, |
| 223 | bool assist) | 244 | bool assist) |
| 224 | { | 245 | { |
| 225 | struct page *pages[1], *aibv_page, *aisb_page = NULL; | 246 | struct page *pages[1], *aibv_page, *aisb_page = NULL; |
| 226 | unsigned int msi_vecs, idx; | 247 | unsigned int msi_vecs, idx, size; |
| 227 | struct zpci_gaite *gaite; | 248 | struct zpci_gaite *gaite; |
| 228 | unsigned long hva, bit; | 249 | unsigned long hva, bit; |
| 229 | struct kvm *kvm; | 250 | struct kvm *kvm; |
| @@ -237,6 +258,10 @@ static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib, · 7 unmodified lines | |||
| 237 | if (zdev->gisa == 0) | 258 | if (zdev->gisa == 0) |
| 238 | return -EINVAL; | 259 | return -EINVAL; |
| 239 | 260 | ||
| 261 | /* AIF already enabled for the device */ | ||
| 262 | if (zdev->kzdev->fib.fmt0.aibv != 0) | ||
| 263 | return -EINVAL; | ||
| 264 | |||
| 240 | kvm = zdev->kzdev->kvm; | 265 | kvm = zdev->kzdev->kvm; |
| 241 | msi_vecs = min_t(unsigned int, fib->fmt0.noi, zdev->max_msi); | 266 | msi_vecs = min_t(unsigned int, fib->fmt0.noi, zdev->max_msi); |
| 242 | 267 | ||
| @@ -246,6 +271,14 @@ static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib, · 3 unmodified lines | |||
| 246 | return gisc; | 271 | return gisc; |
| 247 | 272 | ||
| 248 | /* Replace AIBV address */ | 273 | /* Replace AIBV address */ |
| 274 | size = BITS_TO_LONGS(msi_vecs + fib->fmt0.aibvo) * sizeof(unsigned long); | ||
| 275 | npages = DIV_ROUND_UP((fib->fmt0.aibv & ~PAGE_MASK) + size, PAGE_SIZE); | ||
| 276 | /* AIBV cannot span more than 1 page */ | ||
| 277 | if (npages > 1) { | ||
| 278 | rc = -EINVAL; | ||
| 279 | goto out; | ||
| 280 | } | ||
| 281 | |||
| 249 | idx = srcu_read_lock(&kvm->srcu); | 282 | idx = srcu_read_lock(&kvm->srcu); |
| 250 | hva = gfn_to_hva(kvm, gpa_to_gfn((gpa_t)fib->fmt0.aibv)); | 283 | hva = gfn_to_hva(kvm, gpa_to_gfn((gpa_t)fib->fmt0.aibv)); |
| 251 | npages = pin_user_pages_fast(hva, 1, FOLL_WRITE | FOLL_LONGTERM, pages); | 284 | npages = pin_user_pages_fast(hva, 1, FOLL_WRITE | FOLL_LONGTERM, pages); |
| @@ -261,6 +294,12 @@ static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib, · 9 unmodified lines | |||
| 261 | 294 | ||
| 262 | /* Pin the guest AISB if one was specified */ | 295 | /* Pin the guest AISB if one was specified */ |
| 263 | if (fib->fmt0.sum == 1) { | 296 | if (fib->fmt0.sum == 1) { |
| 297 | /* AISB must be dword aligned */ | ||
| 298 | if (fib->fmt0.aisb & 0x7) { | ||
| 299 | rc = -EINVAL; | ||
| 300 | goto unpin1; | ||
| 301 | } | ||
| 302 | |||
| 264 | idx = srcu_read_lock(&kvm->srcu); | 303 | idx = srcu_read_lock(&kvm->srcu); |
| 265 | hva = gfn_to_hva(kvm, gpa_to_gfn((gpa_t)fib->fmt0.aisb)); | 304 | hva = gfn_to_hva(kvm, gpa_to_gfn((gpa_t)fib->fmt0.aisb)); |
| 266 | npages = pin_user_pages_fast(hva, 1, FOLL_WRITE | FOLL_LONGTERM, | 305 | npages = pin_user_pages_fast(hva, 1, FOLL_WRITE | FOLL_LONGTERM, |
| @@ -275,20 +314,28 @@ static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib, · 8 unmodified lines | |||
| 275 | } | 314 | } |
| 276 | 315 | ||
| 277 | /* Account for pinned pages, roll back on failure */ | 316 | /* Account for pinned pages, roll back on failure */ |
| 278 | if (account_mem(pcount)) | 317 | rc = account_mem(zdev->kzdev, pcount); |
| 318 | if (rc) | ||
| 279 | goto unpin2; | 319 | goto unpin2; |
| 280 | 320 | ||
| 281 | /* AISB must be allocated before we can fill in GAITE */ | 321 | /* AISB must be allocated before we can fill in GAITE */ |
| 282 | mutex_lock(&aift->aift_lock); | 322 | mutex_lock(&aift->aift_lock); |
| 283 | bit = airq_iv_alloc_bit(aift->sbv); | 323 | bit = airq_iv_alloc_bit(aift->sbv); |
| 284 | if (bit == -1UL) | 324 | if (bit == -1UL) { |
| 325 | rc = -ENOMEM; | ||
| 285 | goto unlock; | 326 | goto unlock; |
| 327 | } | ||
| 286 | zdev->aisb = bit; /* store the summary bit number */ | 328 | zdev->aisb = bit; /* store the summary bit number */ |
| 287 | zdev->aibv = airq_iv_create(msi_vecs, AIRQ_IV_DATA | | 329 | zdev->aibv = airq_iv_create(msi_vecs, AIRQ_IV_DATA | |
| 288 | AIRQ_IV_BITLOCK | | 330 | AIRQ_IV_BITLOCK | |
| 289 | AIRQ_IV_GUESTVEC, | 331 | AIRQ_IV_GUESTVEC, |
| 290 | phys_to_virt(fib->fmt0.aibv)); | 332 | phys_to_virt(fib->fmt0.aibv)); |
| 291 | 333 | ||
| 334 | if (!zdev->aibv) { | ||
| 335 | rc = -ENOMEM; | ||
| 336 | goto free_aisb; | ||
| 337 | } | ||
| 338 | |||
| 292 | spin_lock_irq(&aift->gait_lock); | 339 | spin_lock_irq(&aift->gait_lock); |
| 293 | gaite = aift->gait + zdev->aisb; | 340 | gaite = aift->gait + zdev->aisb; |
| 294 | 341 | ||
| @@ -311,21 +358,39 @@ static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib, · 16 unmodified lines | |||
| 311 | aift->kzdev[zdev->aisb] = zdev->kzdev; | 358 | aift->kzdev[zdev->aisb] = zdev->kzdev; |
| 312 | spin_unlock_irq(&aift->gait_lock); | 359 | spin_unlock_irq(&aift->gait_lock); |
| 313 | 360 | ||
| 314 | /* Update guest FIB for re-issue */ | ||
| 315 | fib->fmt0.aisbo = zdev->aisb & 63; | ||
| 316 | fib->fmt0.aisb = virt_to_phys(aift->sbv->vector) + (zdev->aisb / 64) * 8; | ||
| 317 | fib->fmt0.isc = gisc; | ||
| 318 | |||
| 319 | /* Save some guest fib values in the host for later use */ | 361 | /* Save some guest fib values in the host for later use */ |
| 320 | zdev->kzdev->fib.fmt0.isc = fib->fmt0.isc; | 362 | zdev->kzdev->fib.fmt0.isc = gisc; |
| 321 | zdev->kzdev->fib.fmt0.aibv = fib->fmt0.aibv; | 363 | zdev->kzdev->fib.fmt0.aibv = fib->fmt0.aibv; |
| 322 | mutex_unlock(&aift->aift_lock); | ||
| 323 | 364 | ||
| 324 | /* Issue the clp to setup the irq now */ | 365 | /* Issue the clp to setup the irq now */ |
| 325 | rc = kvm_zpci_set_airq(zdev); | 366 | rc = kvm_zpci_set_airq(zdev); |
| 326 | return rc; | 367 | if (!rc) { |
| 368 | mutex_unlock(&aift->aift_lock); | ||
| 369 | return rc; | ||
| 370 | } | ||
| 371 | |||
| 372 | /* Start cleanup */ | ||
| 373 | zdev->kzdev->fib.fmt0.isc = 0; | ||
| 374 | zdev->kzdev->fib.fmt0.aibv = 0; | ||
| 327 | 375 | ||
| 376 | spin_lock_irq(&aift->gait_lock); | ||
| 377 | gaite->count--; | ||
| 378 | gaite->aisb = 0; | ||
| 379 | gaite->gisc = 0; | ||
| 380 | gaite->aisbo = 0; | ||
| 381 | gaite->gisa = 0; | ||
| 382 | aift->kzdev[zdev->aisb] = NULL; | ||
| 383 | spin_unlock_irq(&aift->gait_lock); | ||
| 384 | |||
| 385 | airq_iv_release(zdev->aibv); | ||
| 386 | zdev->aibv = NULL; | ||
| 387 | |||
| 388 | free_aisb: | ||
| 389 | airq_iv_free_bit(aift->sbv, zdev->aisb); | ||
| 390 | zdev->aisb = 0; | ||
| 328 | unlock: | 391 | unlock: |
| 392 | if (pcount > 0) | ||
| 393 | unaccount_mem(zdev->kzdev, pcount); | ||
| 329 | mutex_unlock(&aift->aift_lock); | 394 | mutex_unlock(&aift->aift_lock); |
| 330 | unpin2: | 395 | unpin2: |
| 331 | if (fib->fmt0.sum == 1) | 396 | if (fib->fmt0.sum == 1) |
| @@ -396,7 +461,7 @@ static int kvm_s390_pci_aif_disable(struct zpci_dev *zdev, bool force) · 64 unmodified lines | |||
| 396 | pcount++; | 461 | pcount++; |
| 397 | } | 462 | } |
| 398 | if (pcount > 0) | 463 | if (pcount > 0) |
| 399 | unaccount_mem(pcount); | 464 | unaccount_mem(kzdev, pcount); |
| 400 | out: | 465 | out: |
| 401 | mutex_unlock(&aift->aift_lock); | 466 | mutex_unlock(&aift->aift_lock); |
| 402 | 467 | ||
| @@ -22,6 +22,8 @@ struct kvm_zdev { · 21 unmodified lines | |||
| 22 | struct kvm *kvm; | 22 | struct kvm *kvm; |
| 23 | struct zpci_fib fib; | 23 | struct zpci_fib fib; |
| 24 | struct list_head entry; | 24 | struct list_head entry; |
| 25 | struct user_struct *user_account; | ||
| 26 | struct mm_struct *mm_account; | ||
| 25 | }; | 27 | }; |
| 26 | 28 | ||
| 27 | struct zpci_gaite { | 29 | struct zpci_gaite { |
| @@ -722,6 +722,26 @@ static struct kvm_lpage_info *lpage_info_slot(gfn_t gfn, · 721 unmodified lines | |||
| 722 | return &slot->arch.lpage_info[level - 2][idx]; | 722 | return &slot->arch.lpage_info[level - 2][idx]; |
| 723 | } | 723 | } |
| 724 | 724 | ||
| 725 | static bool kvm_gfn_is_lpage_allowed(struct kvm *kvm, | ||
| 726 | const struct kvm_memory_slot *slot, | ||
| 727 | gfn_t gfn, int level) | ||
| 728 | { | ||
| 729 | const struct kvm_memory_slot *other_slot; | ||
| 730 | |||
| 731 | BUILD_BUG_ON(KVM_MAX_NR_ADDRESS_SPACES > 2); | ||
| 732 | |||
| 733 | if (lpage_info_slot(gfn, slot, level)->disallow_lpage) | ||
| 734 | return false; | ||
| 735 | |||
| 736 | if (kvm_arch_nr_memslot_as_ids(kvm) > 1) { | ||
| 737 | other_slot = __gfn_to_memslot(__kvm_memslots(kvm, slot->as_id ^ 1), gfn); | ||
| 738 | if (other_slot && lpage_info_slot(gfn, other_slot, level)->disallow_lpage) | ||
| 739 | return false; | ||
| 740 | } | ||
| 741 | |||
| 742 | return true; | ||
| 743 | } | ||
| 744 | |||
| 725 | /* | 745 | /* |
| 726 | * The most significant bit in disallow_lpage tracks whether or not memory | 746 | * The most significant bit in disallow_lpage tracks whether or not memory |
| 727 | * attributes are mixed, i.e. not identical for all gfns at the current level. | 747 | * attributes are mixed, i.e. not identical for all gfns at the current level. |
| @@ -2968,7 +2988,7 @@ int mmu_try_to_unsync_pages(struct kvm *kvm, const struct kvm_memory_slot *slot, · 2240 unmodified lines | |||
| 2968 | * write-protected (see above), thus if the gfn can be mapped with a | 2988 | * write-protected (see above), thus if the gfn can be mapped with a |
| 2969 | * hugepage and isn't write-tracked, it can't have a shadow page. | 2989 | * hugepage and isn't write-tracked, it can't have a shadow page. |
| 2970 | */ | 2990 | */ |
| 2971 | if (!lpage_info_slot(gfn, slot, PG_LEVEL_2M)->disallow_lpage) | 2991 | if (kvm_gfn_is_lpage_allowed(kvm, slot, gfn, PG_LEVEL_2M)) |
| 2972 | return 0; | 2992 | return 0; |
| 2973 | 2993 | ||
| 2974 | /* | 2994 | /* |
| @@ -130,13 +130,23 @@ void __kvm_write_track_remove_gfn(struct kvm *kvm, · 129 unmodified lines | |||
| 130 | kvm_mmu_gfn_allow_lpage(slot, gfn); | 130 | kvm_mmu_gfn_allow_lpage(slot, gfn); |
| 131 | } | 131 | } |
| 132 | 132 | ||
| 133 | /* | 133 | static bool __kvm_gfn_is_write_tracked(const struct kvm_memory_slot *slot, |
| 134 | * check if the corresponding access on the specified guest page is tracked. | 134 | gfn_t gfn) |
| 135 | */ | 135 | { |
| 136 | int index; | ||
| 137 | |||
| 138 | if (!slot) | ||
| 139 | return false; | ||
| 140 | |||
| 141 | index = gfn_to_index(gfn, slot->base_gfn, PG_LEVEL_4K); | ||
| 142 | return !!READ_ONCE(slot->arch.gfn_write_track[index]); | ||
| 143 | } | ||
| 144 | |||
| 145 | /* check if write access is tracked on the specified guest page. */ | ||
| 136 | bool kvm_gfn_is_write_tracked(struct kvm *kvm, | 146 | bool kvm_gfn_is_write_tracked(struct kvm *kvm, |
| 137 | const struct kvm_memory_slot *slot, gfn_t gfn) | 147 | const struct kvm_memory_slot *slot, gfn_t gfn) |
| 138 | { | 148 | { |
| 139 | int index; | 149 | const struct kvm_memory_slot *other_slot; |
| 140 | 150 | ||
| 141 | if (!slot) | 151 | if (!slot) |
| 142 | return false; | 152 | return false; |
| @@ -144,8 +154,18 @@ bool kvm_gfn_is_write_tracked(struct kvm *kvm, · 1 unmodified line | |||
| 144 | if (!kvm_page_track_write_tracking_enabled(kvm)) | 154 | if (!kvm_page_track_write_tracking_enabled(kvm)) |
| 145 | return false; | 155 | return false; |
| 146 | 156 | ||
| 147 | index = gfn_to_index(gfn, slot->base_gfn, PG_LEVEL_4K); | 157 | BUILD_BUG_ON(KVM_MAX_NR_ADDRESS_SPACES > 2); |
| 148 | return !!READ_ONCE(slot->arch.gfn_write_track[index]); | 158 | |
| 159 | if (__kvm_gfn_is_write_tracked(slot, gfn)) | ||
| 160 | return true; | ||
| 161 | |||
| 162 | if (kvm_arch_nr_memslot_as_ids(kvm) > 1) { | ||
| 163 | other_slot = __gfn_to_memslot(__kvm_memslots(kvm, slot->as_id ^ 1), gfn); | ||
| 164 | if (__kvm_gfn_is_write_tracked(other_slot, gfn)) | ||
| 165 | return true; | ||
| 166 | } | ||
| 167 | |||
| 168 | return false; | ||
| 149 | } | 169 | } |
| 150 | 170 | ||
| 151 | #ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING | 171 | #ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING |
| @@ -240,14 +240,6 @@ static void avic_deactivate_vmcb(struct vcpu_svm *svm) · 239 unmodified lines | |||
| 240 | if (!is_sev_es_guest(&svm->vcpu)) | 240 | if (!is_sev_es_guest(&svm->vcpu)) |
| 241 | svm_set_intercept(svm, INTERCEPT_CR8_WRITE); | 241 | svm_set_intercept(svm, INTERCEPT_CR8_WRITE); |
| 242 | 242 | ||
| 243 | /* | ||
| 244 | * If running nested and the guest uses its own MSR bitmap, there | ||
| 245 | * is no need to update L0's msr bitmap | ||
| 246 | */ | ||
| 247 | if (is_guest_mode(&svm->vcpu) && | ||
| 248 | vmcb12_is_intercept(&svm->nested.ctl, INTERCEPT_MSR_PROT)) | ||
| 249 | return; | ||
| 250 | |||
| 251 | /* Enabling MSR intercept for x2APIC registers */ | 243 | /* Enabling MSR intercept for x2APIC registers */ |
| 252 | avic_set_x2apic_msr_interception(svm, true); | 244 | avic_set_x2apic_msr_interception(svm, true); |
| 253 | } | 245 | } |
| @@ -101,7 +101,7 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field) · 100 unmodified lines | |||
| 101 | 101 | ||
| 102 | : [output] "=r" (value) | 102 | : [output] "=r" (value) |
| 103 | : [field] "r" (field) | 103 | : [field] "r" (field) |
| 104 | : "cc" | 104 | : "cc", "memory" |
| 105 | : do_fail, do_exception); | 105 | : do_fail, do_exception); |
| 106 | 106 | ||
| 107 | return value; | 107 | return value; |
| @@ -145,7 +145,7 @@ do_exception: · 37 unmodified lines | |||
| 145 | 145 | ||
| 146 | : ASM_CALL_CONSTRAINT, [output] "=&r" (value) | 146 | : ASM_CALL_CONSTRAINT, [output] "=&r" (value) |
| 147 | : [field] "r" (field) | 147 | : [field] "r" (field) |
| 148 | : "cc"); | 148 | : "cc", "memory"); |
| 149 | return value; | 149 | return value; |
| 150 | 150 | ||
| 151 | #endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ | 151 | #endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ |
| @@ -192,7 +192,7 @@ do { \ · 40 unmodified lines | |||
| 192 | asm goto("1: " __stringify(insn) " %0\n\t" \ | 192 | asm goto("1: " __stringify(insn) " %0\n\t" \ |
| 193 | "jna %l[error]\n\t" \ | 193 | "jna %l[error]\n\t" \ |
| 194 | _ASM_EXTABLE(1b, %l[fault]) \ | 194 | _ASM_EXTABLE(1b, %l[fault]) \ |
| 195 | : : op1 : "cc" : error, fault); \ | 195 | : : op1 : "cc", "memory" : error, fault); \ |
| 196 | return; \ | 196 | return; \ |
| 197 | error: \ | 197 | error: \ |
| 198 | instrumentation_begin(); \ | 198 | instrumentation_begin(); \ |
| @@ -208,7 +208,7 @@ do { \ · 9 unmodified lines | |||
| 208 | asm goto("1: " __stringify(insn) " %1, %0\n\t" \ | 208 | asm goto("1: " __stringify(insn) " %1, %0\n\t" \ |
| 209 | "jna %l[error]\n\t" \ | 209 | "jna %l[error]\n\t" \ |
| 210 | _ASM_EXTABLE(1b, %l[fault]) \ | 210 | _ASM_EXTABLE(1b, %l[fault]) \ |
| 211 | : : op1, op2 : "cc" : error, fault); \ | 211 | : : op1, op2 : "cc", "memory" : error, fault);\ |
| 212 | return; \ | 212 | return; \ |
| 213 | error: \ | 213 | error: \ |
| 214 | instrumentation_begin(); \ | 214 | instrumentation_begin(); \ |
| @@ -13429,9 +13429,15 @@ void kvm_arch_pre_destroy_vm(struct kvm *kvm) · 13428 unmodified lines | |||
| 13429 | * iterating over vCPUs in a different task while vCPUs are being freed | 13429 | * iterating over vCPUs in a different task while vCPUs are being freed |
| 13430 | * is unsafe, i.e. will lead to use-after-free. The PIT also needs to | 13430 | * is unsafe, i.e. will lead to use-after-free. The PIT also needs to |
| 13431 | * be stopped before IRQ routing is freed. | 13431 | * be stopped before IRQ routing is freed. |
| 13432 | * | ||
| 13433 | * Do NOT free the in-kernel PIC or I/O APIC here (but as above, make | ||
| 13434 | * sure to flush any background work), as KVM expects interrupt routing | ||
| 13435 | * structures to be valid until vCPUs are destroyed. | ||
| 13432 | */ | 13436 | */ |
| 13433 | #ifdef CONFIG_KVM_IOAPIC | 13437 | #ifdef CONFIG_KVM_IOAPIC |
| 13434 | kvm_free_pit(kvm); | 13438 | kvm_free_pit(kvm); |
| 13439 | if (kvm->arch.vioapic) | ||
| 13440 | cancel_delayed_work_sync(&kvm->arch.vioapic->eoi_inject); | ||
| 13435 | #endif | 13441 | #endif |
| 13436 | 13442 | ||
| 13437 | kvm_mmu_pre_destroy_vm(kvm); | 13443 | kvm_mmu_pre_destroy_vm(kvm); |
| @@ -247,9 +247,6 @@ struct vgic_irq { · 246 unmodified lines | |||
| 247 | * affinity reg (v3). | 247 | * affinity reg (v3). |
| 248 | */ | 248 | */ |
| 249 | 249 | ||
| 250 | bool pending_release:1; /* Used for LPIs only, unreferenced IRQ | ||
| 251 | * pending a release */ | ||
| 252 | |||
| 253 | bool pending_latch:1; /* The pending latch state used to calculate | 250 | bool pending_latch:1; /* The pending latch state used to calculate |
| 254 | * the pending state for both level | 251 | * the pending state for both level |
| 255 | * and edge triggered IRQs. */ | 252 | * and edge triggered IRQs. */ |