History

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm

Pull kvm fixes from Paolo Bonzini: "arm64: - Fix a tiny buglet when propagating the deactivation of an interrupt from a nested guest, which happened to trigger a gold plated CPU bug on a particular implementation - Fix a race between LPI unmapping and mapping, resulting in leaked LPIs - Make LPI mapping more robust on memory allocation failure - Fix the handling of the EL2 tracing clock being disabled - A couple of Sashiko-driven fixes for corner cases in the EL2 tracing code - Add missing sysreg tracepoint for the EL2 code - Tidy-up the mutual exclusion of guest-memfd and MTE - Update Fuad's email address to point to @linux.dev s390: - several fixes for PCI passthru in s390 kvm - fix a 7.2-rc regression in the adapter interrupt mapping code x86: - Add memory clobber to asm for VMX instructions; without one, the compiler could reorder them in troublesome ways because "asm volatile" and "asm goto" only protect against removal of the asm. - Cancel delayed I/O APIC EOI handling before destroying vCPUs - Check all address spaces (normal and SMM) for write tracking and large pages, not just the current one. - Always update x2APIC MSR intercepts for L1 when AVIC is deactivated, even if not running L1. If the deactivation is VM-wide rather than being caused by something in L2's vCPU state, after a nested vmexit L1 will be able to access the host's APIC state" * tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (21 commits) KVM: SVM: Update x2APIC MSR intercepts if AVIC is inhibited while L2 is active KVM: x86/mmu: Check all address spaces before skipping unsync KVM: x86/mmu: Check write tracking in all address spaces KVM: x86: Cancel delayed I/O APIC EOI handling before destroying vCPUs KVM: VMX: add memory clobber to asm for VMX instructions KVM: s390: Fall back to short-term pinning in MAP ioctl KVM: s390: pci: Validate AIBV and AISB before pinning guest pages KVM: s390: pci: Fix resource leak on IRQ registration failure KVM: s390: pci: Fix NULL dereference on AIBV allocation failure KVM: s390: pci: Fix missing error codes and memory unaccounting KVM: s390: pci: Fix memory accounting for pinned/unpinned pages KVM: s390: pci: Reject adapter interrupt forwarding if already enabled KVM: arm64: Reject guest_memfd memslots when the VM has MTE KVM: arm64: Add missing hyp_enter when trapping sysreg KVM: arm64: Fix hyp_trace_desc allocation size in hyp_trace_load() KVM: arm64: Fix potential leak in hyp_trace_buffer_alloc_bpages_backing KVM: arm64: Fix hyp_trace clock disabling KVM: arm64: vgic: Mitigate potential LPI registration failure KVM: arm64: vgic: Fix race between LPI release and re-registration KVM: arm64: Update Fuad Tabba's email address ...

Linus Torvalds committed 3 months ago

Commit fc46aed2 parents fc02acf 7d3aae223 files · +298 −97Browse files

Unified · Split

23 files
  1. .mailmap +1 −0
  2. Documentation/arch/arm64/silicon-errata.rst +4 −0
  3. Documentation/virt/kvm/api.rst +6 −0
  4. Documentation/virt/kvm/devices/s390_flic.rst +13 −2
  5. MAINTAINERS +1 −1
  6. arch/arm64/include/asm/kvm_hypevents.h +1 −0
  7. arch/arm64/kvm/arm.c +19 −6
  8. arch/arm64/kvm/hyp/nvhe/hyp-main.c +1 −0
  9. arch/arm64/kvm/hyp_trace.c +24 −12
  10. arch/arm64/kvm/mmu.c +4 −0
  11. arch/arm64/kvm/vgic/vgic-its.c +17 −8
  12. arch/arm64/kvm/vgic/vgic-v3.c +7 −1
  13. arch/arm64/kvm/vgic/vgic.c +8 −10
  14. arch/s390/include/asm/kvm_host.h +5 −0
  15. arch/s390/kvm/interrupt.c +42 −14
  16. arch/s390/kvm/pci.c +86 −21
  17. arch/s390/kvm/pci.h +2 −0
  18. arch/x86/kvm/mmu/mmu.c +21 −1
  19. arch/x86/kvm/mmu/page_track.c +26 −6
  20. arch/x86/kvm/svm/avic.c +0 −8
  21. arch/x86/kvm/vmx/vmx_ops.h +4 −4
  22. arch/x86/kvm/x86.c +6 −0
  23. include/kvm/arm_vgic.h +0 −3
.mailmap+1−0
@@ -296,6 +296,7 @@ Frank Rowand <frowand.list@gmail.com> <frank.rowand@sony.com> · 295 unmodified lines
296296 Frank Rowand <frowand.list@gmail.com> <frank.rowand@sonymobile.com>
297297 Frank Rowand <frowand.list@gmail.com> <frowand@mvista.com>
298298 Frank Zago <fzago@systemfabricworks.com>
299+Fuad Tabba <fuad.tabba@linux.dev> <tabba@google.com>
299300 Gao Xiang <xiang@kernel.org> <gaoxiang25@huawei.com>
300301 Gao Xiang <xiang@kernel.org> <hsiangkao@aol.com>
301302 Gao Xiang <xiang@kernel.org> <hsiangkao@linux.alibaba.com>
Documentation/arch/arm64/silicon-errata.rst+4−0
@@ -55,10 +55,14 @@ stable kernels. · 54 unmodified lines
5555 +----------------+-----------------+-----------------+-----------------------------+
5656 | Ampere | AmpereOne | AC03_CPU_38 | AMPERE_ERRATUM_AC03_CPU_38 |
5757 +----------------+-----------------+-----------------+-----------------------------+
58+| Ampere | AmpereOne | AC03_CPU_57 | N/A |
59++----------------+-----------------+-----------------+-----------------------------+
5860 | Ampere | AmpereOne AC04 | AC04_CPU_10 | AMPERE_ERRATUM_AC03_CPU_38 |
5961 +----------------+-----------------+-----------------+-----------------------------+
6062 | Ampere | AmpereOne AC04 | AC04_CPU_23 | AMPERE_ERRATUM_AC04_CPU_23 |
6163 +----------------+-----------------+-----------------+-----------------------------+
64+| Ampere | AmpereOne AC04 | AC04_CPU_29 | N/A |
65++----------------+-----------------+-----------------+-----------------------------+
6266 +----------------+-----------------+-----------------+-----------------------------+
6367 | ARM | Cortex-A510 | #2457168 | ARM64_ERRATUM_2457168 |
6468 +----------------+-----------------+-----------------+-----------------------------+
Documentation/virt/kvm/api.rst+6−0
@@ -8414,6 +8414,12 @@ When this capability is enabled all memory in memslots must be mapped as · 8413 unmodified lines
84148414 attempts to create a memslot with an invalid mmap will result in an
84158415 -EINVAL return.
84168416
8417+``guest_memfd``, even though it is an anonymous file, is not supported with MTE.
8418+Attempting to create a memslot backed by ``guest_memfd`` when the MTE capability
8419+is enabled, or attempting to enable the MTE capability after
8420+``guest_memfd``-backed memslots have been created, will result in an -EINVAL
8421+return.
8422+
84178423 When enabled the VMM may make use of the ``KVM_ARM_MTE_COPY_TAGS`` ioctl to
84188424 perform a bulk copy of tags to/from the guest.
84198425
Documentation/virt/kvm/devices/s390_flic.rst+13−2
@@ -112,9 +112,20 @@ Groups: · 111 unmodified lines
112112 mask or unmask the adapter, as specified in mask
113113
114114 KVM_S390_IO_ADAPTER_MAP
115− This is now a no-op. The mapping is purely done by the irq route.
115+ Map an adapter indicator or summary page for long-term pinning so that
116+ interrupt injection can be performed in atomic context. If long-term
117+ pinning is not possible (e.g. file-backed memory), the page is verified
118+ via a short-term pin and the ioctl returns success; interrupt injection
119+ will use the non-atomic irqfd path with short-term pinning on each
120+ interrupt. In Secure Execution mode this is a no-op and the ioctl
121+ returns success.
122+
116123 KVM_S390_IO_ADAPTER_UNMAP
117− This is now a no-op. The mapping is purely done by the irq route.
124+ Unmap a previously mapped adapter indicator or summary page and release
125+ the long-term pin. If the page was not long-term pinned (e.g. file-backed
126+ memory), the map entry is removed and success is returned; if no prior
127+ map entry exists, -ENOENT is returned. In Secure Execution mode this is
128+ a no-op and the ioctl returns success.
118129
119130 KVM_DEV_FLIC_AISM
120131 modify the adapter-interruption-suppression mode for a given isc if the
MAINTAINERS+1−1
@@ -14193,7 +14193,7 @@ F: virt/kvm/* · 14192 unmodified lines
1419314193 KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)
1419414194 M: Marc Zyngier <maz@kernel.org>
1419514195 M: Oliver Upton <oupton@kernel.org>
14196−R: Fuad Tabba <tabba@google.com>
14196+R: Fuad Tabba <fuad.tabba@linux.dev>
1419714197 R: Joey Gouly <joey.gouly@arm.com>
1419814198 R: Steffen Eiden <seiden@linux.ibm.com>
1419914199 R: Suzuki K Poulose <suzuki.poulose@arm.com>
arch/arm64/include/asm/kvm_hypevents.h+1−0
@@ -12,6 +12,7 @@ · 11 unmodified lines
1212 enum hyp_enter_exit_reason {
1313 HYP_REASON_SMC,
1414 HYP_REASON_HVC,
15+ HYP_REASON_SYS,
1516 HYP_REASON_PSCI,
1617 HYP_REASON_HOST_ABORT,
1718 HYP_REASON_GUEST_EXIT,
arch/arm64/kvm/arm.c+19−6
@@ -149,14 +149,27 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · 148 unmodified lines
149149 set_bit(KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER,
150150 &kvm->arch.flags);
151151 break;
152− case KVM_CAP_ARM_MTE:
153− mutex_lock(&kvm->lock);
154− if (system_supports_mte() && !kvm->created_vcpus) {
155− r = 0;
156− set_bit(KVM_ARCH_FLAG_MTE_ENABLED, &kvm->arch.flags);
152+ case KVM_CAP_ARM_MTE: {
153+ struct kvm_memory_slot *memslot;
154+ int bkt;
155+
156+ guard(mutex)(&kvm->lock);
157+ if (!system_supports_mte() || kvm->created_vcpus)
158+ break;
159+
160+ r = 0;
161+ guard(mutex)(&kvm->slots_lock);
162+ kvm_for_each_memslot(memslot, bkt, kvm_memslots(kvm)) {
163+ if (kvm_slot_has_gmem(memslot)) {
164+ r = -EINVAL;
165+ break;
166+ }
157167 }
158− mutex_unlock(&kvm->lock);
168+ if (r == 0)
169+ set_bit(KVM_ARCH_FLAG_MTE_ENABLED, &kvm->arch.flags);
159170 break;
171+
172+ }
160173 case KVM_CAP_ARM_SYSTEM_SUSPEND:
161174 r = 0;
162175 set_bit(KVM_ARCH_FLAG_SYSTEM_SUSPEND_ENABLED, &kvm->arch.flags);
arch/arm64/kvm/hyp/nvhe/hyp-main.c+1−0
@@ -932,6 +932,7 @@ void handle_trap(struct kvm_cpu_context *host_ctxt) · 931 unmodified lines
932932 handle_host_mem_abort(host_ctxt);
933933 break;
934934 case ESR_ELx_EC_SYS64:
935+ trace_hyp_enter(host_ctxt, HYP_REASON_SYS);
935936 if (handle_host_mte(esr))
936937 break;
937938 fallthrough;
arch/arm64/kvm/hyp_trace.c+24−12
@@ -37,8 +37,6 @@ static struct hyp_trace_clock { · 36 unmodified lines
3737 u32 shift;
3838 struct delayed_work work;
3939 struct completion ready;
40− struct mutex lock;
41− bool running;
4240 } hyp_clock;
4341
4442 static void __hyp_clock_work(struct work_struct *work)
@@ -110,12 +108,9 @@ static void hyp_trace_clock_enable(struct hyp_trace_clock *hyp_clock, bool enabl · 65 unmodified lines
110108 {
111109 struct system_time_snapshot snap;
112110
113− if (hyp_clock->running == enable)
114− return;
115−
116111 if (!enable) {
117112 cancel_delayed_work_sync(&hyp_clock->work);
118− hyp_clock->running = false;
113+ return;
119114 }
120115
121116 ktime_get_snapshot_id(CLOCK_BOOTTIME, &snap);
@@ -128,7 +123,6 @@ static void hyp_trace_clock_enable(struct hyp_trace_clock *hyp_clock, bool enabl · 6 unmodified lines
128123 INIT_DELAYED_WORK(&hyp_clock->work, __hyp_clock_work);
129124 schedule_delayed_work(&hyp_clock->work, msecs_to_jiffies(CLOCK_INIT_MS));
130125 wait_for_completion(&hyp_clock->ready);
131− hyp_clock->running = true;
132126 }
133127
134128 /* Access to this struct within the trace_remote_callbacks are protected by the trace_remote lock */
@@ -160,6 +154,7 @@ static int hyp_trace_buffer_alloc_bpages_backing(struct hyp_trace_buffer *trace_ · 25 unmodified lines
160154 int nr_bpages = (PAGE_ALIGN(size) / PAGE_SIZE) + 1;
161155 size_t backing_size;
162156 void *start;
157+ int ret;
163158
164159 backing_size = PAGE_ALIGN(sizeof(struct simple_buffer_page) * nr_bpages *
165160 num_possible_cpus());
@@ -168,10 +163,16 @@ static int hyp_trace_buffer_alloc_bpages_backing(struct hyp_trace_buffer *trace_ · 2 unmodified lines
168163 if (!start)
169164 return -ENOMEM;
170165
166+ ret = __map_hyp(start, backing_size);
167+ if (ret) {
168+ free_pages_exact(start, backing_size);
169+ return ret;
170+ }
171+
171172 trace_buffer->desc->bpages_backing_start = (unsigned long)start;
172173 trace_buffer->desc->bpages_backing_size = backing_size;
173174
174− return __map_hyp(start, backing_size);
175+ return ret;
175176 }
176177
177178 static void hyp_trace_buffer_free_bpages_backing(struct hyp_trace_buffer *trace_buffer)
@@ -228,18 +229,22 @@ static int hyp_trace_buffer_share_hyp(struct hyp_trace_buffer *trace_buffer) · 50 unmodified lines
228229 static struct trace_buffer_desc *hyp_trace_load(unsigned long size, void *priv)
229230 {
230231 struct hyp_trace_buffer *trace_buffer = priv;
232+ size_t desc_size, tb_desc_size;
231233 struct hyp_trace_desc *desc;
232− size_t desc_size;
233234 int ret;
234235
235236 if (WARN_ON(trace_buffer->desc))
236237 return ERR_PTR(-EINVAL);
237238
238− desc_size = trace_buffer_desc_size(size, num_possible_cpus());
239+ tb_desc_size = trace_buffer_desc_size(size, num_possible_cpus());
240+ desc_size = size_add(tb_desc_size, offsetof(struct hyp_trace_desc, trace_buffer_desc));
239241 if (desc_size == SIZE_MAX)
240242 return ERR_PTR(-E2BIG);
241243
242244 desc_size = PAGE_ALIGN(desc_size);
245+ if (!desc_size)
246+ return ERR_PTR(-E2BIG);
247+
243248 desc = (struct hyp_trace_desc *)alloc_pages_exact(desc_size, GFP_KERNEL);
244249 if (!desc)
245250 return ERR_PTR(-ENOMEM);
@@ -255,7 +260,7 @@ static struct trace_buffer_desc *hyp_trace_load(unsigned long size, void *priv) · 9 unmodified lines
255260 if (ret)
256261 goto err_free_desc;
257262
258− ret = trace_remote_alloc_buffer(&desc->trace_buffer_desc, desc_size, size,
263+ ret = trace_remote_alloc_buffer(&desc->trace_buffer_desc, tb_desc_size, size,
259264 cpu_possible_mask);
260265 if (ret)
261266 goto err_free_backing;
@@ -304,9 +309,15 @@ static void hyp_trace_unload(struct trace_buffer_desc *desc, void *priv) · 42 unmodified lines
304309
305310 static int hyp_trace_enable_tracing(bool enable, void *priv)
306311 {
312+ int ret;
313+
307314 hyp_trace_clock_enable(&hyp_clock, enable);
308315
309− return kvm_call_hyp_nvhe(__tracing_enable, enable);
316+ ret = kvm_call_hyp_nvhe(__tracing_enable, enable);
317+ if (ret)
318+ hyp_trace_clock_enable(&hyp_clock, !enable);
319+
320+ return ret;
310321 }
311322
312323 static int hyp_trace_swap_reader_page(unsigned int cpu, void *priv)
@@ -398,6 +409,7 @@ static const char *__hyp_enter_exit_reason_str(u8 reason) · 85 unmodified lines
398409 static const char strs[][12] = {
399410 "smc",
400411 "hvc",
412+ "sys",
401413 "psci",
402414 "host_abort",
403415 "guest_exit",
arch/arm64/kvm/mmu.c+4−0
@@ -2652,6 +2652,10 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, · 2651 unmodified lines
26522652 if (kvm_slot_has_gmem(new) && !kvm_memslot_is_gmem_only(new))
26532653 return -EINVAL;
26542654
2655+ /* guest_memfd is incompatible with MTE. */
2656+ if (kvm_slot_has_gmem(new) && kvm_has_mte(kvm))
2657+ return -EINVAL;
2658+
26552659 hva = new->userspace_addr;
26562660 reg_end = hva + (new->npages << PAGE_SHIFT);
26572661
arch/arm64/kvm/vgic/vgic-its.c+17−8
@@ -116,18 +116,27 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, · 115 unmodified lines
116116 kfree(irq);
117117 irq = oldirq;
118118 } else {
119− ret = xa_err(__xa_store(&dist->lpi_xa, intid, irq, 0));
119+ /*
120+ * The entry is either empty or contains a dead LPI (refcount=0)
121+ * from the deferred release path, pending cleanup by
122+ * vgic_release_deleted_lpis(). Evict and free it if present.
123+ */
124+ oldirq = __xa_store(&dist->lpi_xa, intid, irq,
125+ GFP_NOWAIT | __GFP_ACCOUNT);
126+ ret = xa_err(oldirq);
127+ if (ret) {
128+ xa_unlock_irqrestore(&dist->lpi_xa, flags);
129+ kfree(irq);
130+
131+ return ERR_PTR(ret);
132+ }
133+
134+ if (oldirq && !WARN_ON_ONCE(refcount_read(&oldirq->refcount)))
135+ kfree_rcu(oldirq, rcu);
120136 }
121137
122138 xa_unlock_irqrestore(&dist->lpi_xa, flags);
123139
124− if (ret) {
125− xa_release(&dist->lpi_xa, intid);
126− kfree(irq);
127−
128− return ERR_PTR(ret);
129− }
130−
131140 /*
132141 * We "cache" the configuration table entries in our struct vgic_irq's.
133142 * However we only have those structs for mapped IRQs, so we read in
arch/arm64/kvm/vgic/vgic-v3.c+7−1
@@ -275,7 +275,13 @@ void vgic_v3_deactivate(struct kvm_vcpu *vcpu, u64 val) · 274 unmodified lines
275275 lr = vgic_v3_compute_lr(vcpu, irq) & ~ICH_LR_ACTIVE_BIT;
276276 }
277277
278− if (lr & ICH_LR_HW)
278+ /*
279+ * In the nested state, the irq has already been deactivated via the HW
280+ * bit in the LR. Deactivating again would be harmless except AmpereOne
281+ * errata AC03_CPU_57, AC04_CPU_29 could cause irq delivery to break if
282+ * the deactivation hits the highest priority pending irq.
283+ */
284+ if ((lr & ICH_LR_HW) && !vgic_state_is_nested(vcpu))
279285 vgic_v3_deactivate_phys(FIELD_GET(ICH_LR_PHYS_ID_MASK, lr));
280286
281287 vgic_v3_fold_lr(vcpu, lr);
arch/arm64/kvm/vgic/vgic.c+8−10
@@ -146,11 +146,7 @@ static __must_check bool __vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq) · 145 unmodified lines
146146
147147 static __must_check bool vgic_put_irq_norelease(struct kvm *kvm, struct vgic_irq *irq)
148148 {
149− if (!__vgic_put_irq(kvm, irq))
150− return false;
151−
152− irq->pending_release = true;
153− return true;
149+ return __vgic_put_irq(kvm, irq);
154150 }
155151
156152 void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq)
@@ -167,12 +163,14 @@ void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq) · 10 unmodified lines
167163 guard(spinlock_irqsave)(&dist->lpi_xa.xa_lock);
168164 }
169165
170− if (!__vgic_put_irq(kvm, irq))
166+ if (!irq_is_lpi(kvm, irq->intid))
171167 return;
172168
173− xa_lock_irqsave(&dist->lpi_xa, flags);
174− vgic_release_lpi_locked(dist, irq);
175− xa_unlock_irqrestore(&dist->lpi_xa, flags);
169+ if (refcount_dec_and_lock_irqsave(&irq->refcount,
170+ &dist->lpi_xa.xa_lock, &flags)) {
171+ vgic_release_lpi_locked(dist, irq);
172+ xa_unlock_irqrestore(&dist->lpi_xa, flags);
173+ }
176174 }
177175
178176 static void vgic_release_deleted_lpis(struct kvm *kvm)
@@ -184,7 +182,7 @@ static void vgic_release_deleted_lpis(struct kvm *kvm) · 5 unmodified lines
184182 xa_lock_irqsave(&dist->lpi_xa, flags);
185183
186184 xa_for_each(&dist->lpi_xa, intid, irq) {
187− if (irq->pending_release)
185+ if (!refcount_read(&irq->refcount))
188186 vgic_release_lpi_locked(dist, irq);
189187 }
190188
arch/s390/include/asm/kvm_host.h+5−0
@@ -476,6 +476,11 @@ struct s390_map_info { · 475 unmodified lines
476476 __u64 guest_addr;
477477 __u64 addr;
478478 struct page *page;
479+ /*
480+ * True if the page is long-term pinned. False if long-term pinning
481+ * failed and this entry exists only to preserve MAP/UNMAP symmetry.
482+ */
483+ bool pinned;
479484 };
480485
481486 struct s390_io_adapter {
arch/s390/kvm/interrupt.c+42−14
@@ -2520,8 +2520,22 @@ static int kvm_s390_adapter_map(struct kvm *kvm, unsigned int id, __u64 addr) · 2519 unmodified lines
25202520 map->addr = host_addr;
25212521 map->page = pin_map_page(kvm, host_addr, FOLL_LONGTERM);
25222522 if (!map->page) {
2523− ret = -EINVAL;
2524− goto out;
2523+ /*
2524+ * Long-term pinning may fail for memory types such as file-backed
2525+ * memory. Verify that short-term pinning succeeds so that the
2526+ * non-atomic irqfd path can handle interrupt injection.
2527+ */
2528+ map->page = pin_map_page(kvm, host_addr, 0);
2529+ if (!map->page) {
2530+ ret = -EINVAL;
2531+ goto out;
2532+ }
2533+ unpin_user_page(map->page);
2534+ map->page = NULL;
2535+ map->pinned = false;
2536+ /* Add an entry to preserve MAP/UNMAP symmetry. */
2537+ } else {
2538+ map->pinned = true;
25252539 }
25262540 spin_lock_irqsave(&adapter->maps_lock, flags);
25272541 if (adapter->nr_maps < MAX_S390_ADAPTER_MAPS) {
@@ -2532,7 +2546,7 @@ static int kvm_s390_adapter_map(struct kvm *kvm, unsigned int id, __u64 addr) · 4 unmodified lines
25322546 ret = -EINVAL;
25332547 }
25342548 spin_unlock_irqrestore(&adapter->maps_lock, flags);
2535− if (ret)
2549+ if (ret && map->page)
25362550 unpin_user_page(map->page);
25372551 out:
25382552 if (ret)
@@ -2546,6 +2560,7 @@ static int kvm_s390_adapter_unmap(struct kvm *kvm, unsigned int id, __u64 addr) · 7 unmodified lines
25462560 struct s390_map_info *map, *tmp, *map_to_free;
25472561 struct page *map_page_to_put = NULL;
25482562 u64 map_addr_to_mark = 0;
2563+ bool map_pinned = false;
25492564 unsigned long flags;
25502565 int found = 0, idx;
25512566
@@ -2560,6 +2575,7 @@ static int kvm_s390_adapter_unmap(struct kvm *kvm, unsigned int id, __u64 addr) · 8 unmodified lines
25602575 list_del(&map->list);
25612576 map_page_to_put = map->page;
25622577 map_addr_to_mark = map->guest_addr;
2578+ map_pinned = map->pinned;
25632579 map_to_free = map;
25642580 break;
25652581 }
@@ -2568,11 +2584,18 @@ static int kvm_s390_adapter_unmap(struct kvm *kvm, unsigned int id, __u64 addr) · 2 unmodified lines
25682584
25692585 if (found) {
25702586 kfree(map_to_free);
2571− idx = srcu_read_lock(&kvm->srcu);
2572− mark_page_dirty(kvm, map_addr_to_mark >> PAGE_SHIFT);
2573− set_page_dirty_lock(map_page_to_put);
2574− srcu_read_unlock(&kvm->srcu, idx);
2575− unpin_user_page(map_page_to_put);
2587+ if (map_pinned) {
2588+ /*
2589+ * Only long-term pinned pages need to be marked dirty
2590+ * and released. Fallback entries exist only for
2591+ * MAP/UNMAP symmetry.
2592+ */
2593+ idx = srcu_read_lock(&kvm->srcu);
2594+ mark_page_dirty(kvm, map_addr_to_mark >> PAGE_SHIFT);
2595+ set_page_dirty_lock(map_page_to_put);
2596+ srcu_read_unlock(&kvm->srcu, idx);
2597+ unpin_user_page(map_page_to_put);
2598+ }
25762599 }
25772600
25782601 return found ? 0 : -ENOENT;
@@ -2598,11 +2621,13 @@ void kvm_s390_unmap_all_adapters(struct kvm *kvm) · 19 unmodified lines
25982621
25992622 list_for_each_entry_safe(map, tmp, &local_list, list) {
26002623 list_del(&map->list);
2601− idx = srcu_read_lock(&kvm->srcu);
2602− mark_page_dirty(kvm, map->guest_addr >> PAGE_SHIFT);
2603− set_page_dirty_lock(map->page);
2604− srcu_read_unlock(&kvm->srcu, idx);
2605− unpin_user_page(map->page);
2624+ if (map->pinned) {
2625+ idx = srcu_read_lock(&kvm->srcu);
2626+ mark_page_dirty(kvm, map->guest_addr >> PAGE_SHIFT);
2627+ set_page_dirty_lock(map->page);
2628+ srcu_read_unlock(&kvm->srcu, idx);
2629+ unpin_user_page(map->page);
2630+ }
26062631 kfree(map);
26072632 }
26082633 }
@@ -2929,8 +2954,11 @@ static struct s390_map_info *get_map_info(struct s390_io_adapter *adapter, · 320 unmodified lines
29292954 return NULL;
29302955
29312956 list_for_each_entry(map, &adapter->maps, list) {
2932− if (map->addr == addr)
2957+ if (map->addr == addr) {
2958+ if (!map->pinned)
2959+ return NULL;
29332960 return map;
2961+ }
29342962 }
29352963 return NULL;
29362964 }
arch/s390/kvm/pci.c+86−21
@@ -190,40 +190,61 @@ static int kvm_zpci_clear_airq(struct zpci_dev *zdev) · 189 unmodified lines
190190 return cc ? -EIO : 0;
191191 }
192192
193−static inline void unaccount_mem(unsigned long nr_pages)
193+static inline void unaccount_mem(struct kvm_zdev *kzdev, unsigned long nr_pages)
194194 {
195− struct user_struct *user = get_uid(current_user());
195+ struct user_struct *user = kzdev->user_account;
196+ struct mm_struct *mm_account = kzdev->mm_account;
196197
197− if (user)
198+ if (user) {
198199 atomic_long_sub(nr_pages, &user->locked_vm);
199− if (current->mm)
200− atomic64_sub(nr_pages, &current->mm->pinned_vm);
200+ free_uid(user);
201+ kzdev->user_account = NULL;
202+ }
203+
204+ if (mm_account) {
205+ atomic64_sub(nr_pages, &mm_account->pinned_vm);
206+ mmdrop(mm_account);
207+ kzdev->mm_account = NULL;
208+ }
201209 }
202210
203−static inline int account_mem(unsigned long nr_pages)
211+static inline int account_mem(struct kvm_zdev *kzdev, unsigned long nr_pages)
204212 {
205213 struct user_struct *user = get_uid(current_user());
206214 unsigned long page_limit, cur_pages, new_pages;
215+ int rc = 0;
207216
208217 page_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
209218
210219 cur_pages = atomic_long_read(&user->locked_vm);
211220 do {
212221 new_pages = cur_pages + nr_pages;
213− if (new_pages > page_limit)
214− return -ENOMEM;
222+ if (new_pages > page_limit) {
223+ rc = -ENOMEM;
224+ goto out;
225+ }
215226 } while (!atomic_long_try_cmpxchg(&user->locked_vm, &cur_pages, new_pages));
216227
217− atomic64_add(nr_pages, &current->mm->pinned_vm);
228+ if (current->mm) {
229+ mmgrab(current->mm);
230+ atomic64_add(nr_pages, &current->mm->pinned_vm);
231+ }
232+
233+ kzdev->user_account = user;
234+ kzdev->mm_account = current->mm;
218235
219236 return 0;
237+
238+out:
239+ free_uid(user);
240+ return rc;
220241 }
221242
222243 static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib,
223244 bool assist)
224245 {
225246 struct page *pages[1], *aibv_page, *aisb_page = NULL;
226− unsigned int msi_vecs, idx;
247+ unsigned int msi_vecs, idx, size;
227248 struct zpci_gaite *gaite;
228249 unsigned long hva, bit;
229250 struct kvm *kvm;
@@ -237,6 +258,10 @@ static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib, · 7 unmodified lines
237258 if (zdev->gisa == 0)
238259 return -EINVAL;
239260
261+ /* AIF already enabled for the device */
262+ if (zdev->kzdev->fib.fmt0.aibv != 0)
263+ return -EINVAL;
264+
240265 kvm = zdev->kzdev->kvm;
241266 msi_vecs = min_t(unsigned int, fib->fmt0.noi, zdev->max_msi);
242267
@@ -246,6 +271,14 @@ static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib, · 3 unmodified lines
246271 return gisc;
247272
248273 /* Replace AIBV address */
274+ size = BITS_TO_LONGS(msi_vecs + fib->fmt0.aibvo) * sizeof(unsigned long);
275+ npages = DIV_ROUND_UP((fib->fmt0.aibv & ~PAGE_MASK) + size, PAGE_SIZE);
276+ /* AIBV cannot span more than 1 page */
277+ if (npages > 1) {
278+ rc = -EINVAL;
279+ goto out;
280+ }
281+
249282 idx = srcu_read_lock(&kvm->srcu);
250283 hva = gfn_to_hva(kvm, gpa_to_gfn((gpa_t)fib->fmt0.aibv));
251284 npages = pin_user_pages_fast(hva, 1, FOLL_WRITE | FOLL_LONGTERM, pages);
@@ -261,6 +294,12 @@ static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib, · 9 unmodified lines
261294
262295 /* Pin the guest AISB if one was specified */
263296 if (fib->fmt0.sum == 1) {
297+ /* AISB must be dword aligned */
298+ if (fib->fmt0.aisb & 0x7) {
299+ rc = -EINVAL;
300+ goto unpin1;
301+ }
302+
264303 idx = srcu_read_lock(&kvm->srcu);
265304 hva = gfn_to_hva(kvm, gpa_to_gfn((gpa_t)fib->fmt0.aisb));
266305 npages = pin_user_pages_fast(hva, 1, FOLL_WRITE | FOLL_LONGTERM,
@@ -275,20 +314,28 @@ static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib, · 8 unmodified lines
275314 }
276315
277316 /* Account for pinned pages, roll back on failure */
278− if (account_mem(pcount))
317+ rc = account_mem(zdev->kzdev, pcount);
318+ if (rc)
279319 goto unpin2;
280320
281321 /* AISB must be allocated before we can fill in GAITE */
282322 mutex_lock(&aift->aift_lock);
283323 bit = airq_iv_alloc_bit(aift->sbv);
284− if (bit == -1UL)
324+ if (bit == -1UL) {
325+ rc = -ENOMEM;
285326 goto unlock;
327+ }
286328 zdev->aisb = bit; /* store the summary bit number */
287329 zdev->aibv = airq_iv_create(msi_vecs, AIRQ_IV_DATA |
288330 AIRQ_IV_BITLOCK |
289331 AIRQ_IV_GUESTVEC,
290332 phys_to_virt(fib->fmt0.aibv));
291333
334+ if (!zdev->aibv) {
335+ rc = -ENOMEM;
336+ goto free_aisb;
337+ }
338+
292339 spin_lock_irq(&aift->gait_lock);
293340 gaite = aift->gait + zdev->aisb;
294341
@@ -311,21 +358,39 @@ static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib, · 16 unmodified lines
311358 aift->kzdev[zdev->aisb] = zdev->kzdev;
312359 spin_unlock_irq(&aift->gait_lock);
313360
314− /* Update guest FIB for re-issue */
315− fib->fmt0.aisbo = zdev->aisb & 63;
316− fib->fmt0.aisb = virt_to_phys(aift->sbv->vector) + (zdev->aisb / 64) * 8;
317− fib->fmt0.isc = gisc;
318−
319361 /* Save some guest fib values in the host for later use */
320− zdev->kzdev->fib.fmt0.isc = fib->fmt0.isc;
362+ zdev->kzdev->fib.fmt0.isc = gisc;
321363 zdev->kzdev->fib.fmt0.aibv = fib->fmt0.aibv;
322− mutex_unlock(&aift->aift_lock);
323364
324365 /* Issue the clp to setup the irq now */
325366 rc = kvm_zpci_set_airq(zdev);
326− return rc;
367+ if (!rc) {
368+ mutex_unlock(&aift->aift_lock);
369+ return rc;
370+ }
371+
372+ /* Start cleanup */
373+ zdev->kzdev->fib.fmt0.isc = 0;
374+ zdev->kzdev->fib.fmt0.aibv = 0;
327375
376+ spin_lock_irq(&aift->gait_lock);
377+ gaite->count--;
378+ gaite->aisb = 0;
379+ gaite->gisc = 0;
380+ gaite->aisbo = 0;
381+ gaite->gisa = 0;
382+ aift->kzdev[zdev->aisb] = NULL;
383+ spin_unlock_irq(&aift->gait_lock);
384+
385+ airq_iv_release(zdev->aibv);
386+ zdev->aibv = NULL;
387+
388+free_aisb:
389+ airq_iv_free_bit(aift->sbv, zdev->aisb);
390+ zdev->aisb = 0;
328391 unlock:
392+ if (pcount > 0)
393+ unaccount_mem(zdev->kzdev, pcount);
329394 mutex_unlock(&aift->aift_lock);
330395 unpin2:
331396 if (fib->fmt0.sum == 1)
@@ -396,7 +461,7 @@ static int kvm_s390_pci_aif_disable(struct zpci_dev *zdev, bool force) · 64 unmodified lines
396461 pcount++;
397462 }
398463 if (pcount > 0)
399− unaccount_mem(pcount);
464+ unaccount_mem(kzdev, pcount);
400465 out:
401466 mutex_unlock(&aift->aift_lock);
402467
arch/s390/kvm/pci.h+2−0
@@ -22,6 +22,8 @@ struct kvm_zdev { · 21 unmodified lines
2222 struct kvm *kvm;
2323 struct zpci_fib fib;
2424 struct list_head entry;
25+ struct user_struct *user_account;
26+ struct mm_struct *mm_account;
2527 };
2628
2729 struct zpci_gaite {
arch/x86/kvm/mmu/mmu.c+21−1
@@ -722,6 +722,26 @@ static struct kvm_lpage_info *lpage_info_slot(gfn_t gfn, · 721 unmodified lines
722722 return &slot->arch.lpage_info[level - 2][idx];
723723 }
724724
725+static bool kvm_gfn_is_lpage_allowed(struct kvm *kvm,
726+ const struct kvm_memory_slot *slot,
727+ gfn_t gfn, int level)
728+{
729+ const struct kvm_memory_slot *other_slot;
730+
731+ BUILD_BUG_ON(KVM_MAX_NR_ADDRESS_SPACES > 2);
732+
733+ if (lpage_info_slot(gfn, slot, level)->disallow_lpage)
734+ return false;
735+
736+ if (kvm_arch_nr_memslot_as_ids(kvm) > 1) {
737+ other_slot = __gfn_to_memslot(__kvm_memslots(kvm, slot->as_id ^ 1), gfn);
738+ if (other_slot && lpage_info_slot(gfn, other_slot, level)->disallow_lpage)
739+ return false;
740+ }
741+
742+ return true;
743+}
744+
725745 /*
726746 * The most significant bit in disallow_lpage tracks whether or not memory
727747 * attributes are mixed, i.e. not identical for all gfns at the current level.
@@ -2968,7 +2988,7 @@ int mmu_try_to_unsync_pages(struct kvm *kvm, const struct kvm_memory_slot *slot, · 2240 unmodified lines
29682988 * write-protected (see above), thus if the gfn can be mapped with a
29692989 * hugepage and isn't write-tracked, it can't have a shadow page.
29702990 */
2971− if (!lpage_info_slot(gfn, slot, PG_LEVEL_2M)->disallow_lpage)
2991+ if (kvm_gfn_is_lpage_allowed(kvm, slot, gfn, PG_LEVEL_2M))
29722992 return 0;
29732993
29742994 /*
arch/x86/kvm/mmu/page_track.c+26−6
@@ -130,13 +130,23 @@ void __kvm_write_track_remove_gfn(struct kvm *kvm, · 129 unmodified lines
130130 kvm_mmu_gfn_allow_lpage(slot, gfn);
131131 }
132132
133−/*
134− * check if the corresponding access on the specified guest page is tracked.
135− */
133+static bool __kvm_gfn_is_write_tracked(const struct kvm_memory_slot *slot,
134+ gfn_t gfn)
135+{
136+ int index;
137+
138+ if (!slot)
139+ return false;
140+
141+ index = gfn_to_index(gfn, slot->base_gfn, PG_LEVEL_4K);
142+ return !!READ_ONCE(slot->arch.gfn_write_track[index]);
143+}
144+
145+/* check if write access is tracked on the specified guest page. */
136146 bool kvm_gfn_is_write_tracked(struct kvm *kvm,
137147 const struct kvm_memory_slot *slot, gfn_t gfn)
138148 {
139− int index;
149+ const struct kvm_memory_slot *other_slot;
140150
141151 if (!slot)
142152 return false;
@@ -144,8 +154,18 @@ bool kvm_gfn_is_write_tracked(struct kvm *kvm, · 1 unmodified line
144154 if (!kvm_page_track_write_tracking_enabled(kvm))
145155 return false;
146156
147− index = gfn_to_index(gfn, slot->base_gfn, PG_LEVEL_4K);
148− return !!READ_ONCE(slot->arch.gfn_write_track[index]);
157+ BUILD_BUG_ON(KVM_MAX_NR_ADDRESS_SPACES > 2);
158+
159+ if (__kvm_gfn_is_write_tracked(slot, gfn))
160+ return true;
161+
162+ if (kvm_arch_nr_memslot_as_ids(kvm) > 1) {
163+ other_slot = __gfn_to_memslot(__kvm_memslots(kvm, slot->as_id ^ 1), gfn);
164+ if (__kvm_gfn_is_write_tracked(other_slot, gfn))
165+ return true;
166+ }
167+
168+ return false;
149169 }
150170
151171 #ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
arch/x86/kvm/svm/avic.c+0−8
@@ -240,14 +240,6 @@ static void avic_deactivate_vmcb(struct vcpu_svm *svm) · 239 unmodified lines
240240 if (!is_sev_es_guest(&svm->vcpu))
241241 svm_set_intercept(svm, INTERCEPT_CR8_WRITE);
242242
243− /*
244− * If running nested and the guest uses its own MSR bitmap, there
245− * is no need to update L0's msr bitmap
246− */
247− if (is_guest_mode(&svm->vcpu) &&
248− vmcb12_is_intercept(&svm->nested.ctl, INTERCEPT_MSR_PROT))
249− return;
250−
251243 /* Enabling MSR intercept for x2APIC registers */
252244 avic_set_x2apic_msr_interception(svm, true);
253245 }
arch/x86/kvm/vmx/vmx_ops.h+4−4
@@ -101,7 +101,7 @@ static __always_inline unsigned long __vmcs_readl(unsigned long field) · 100 unmodified lines
101101
102102 : [output] "=r" (value)
103103 : [field] "r" (field)
104− : "cc"
104+ : "cc", "memory"
105105 : do_fail, do_exception);
106106
107107 return value;
@@ -145,7 +145,7 @@ do_exception: · 37 unmodified lines
145145
146146 : ASM_CALL_CONSTRAINT, [output] "=&r" (value)
147147 : [field] "r" (field)
148− : "cc");
148+ : "cc", "memory");
149149 return value;
150150
151151 #endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */
@@ -192,7 +192,7 @@ do { \ · 40 unmodified lines
192192 asm goto("1: " __stringify(insn) " %0\n\t" \
193193 "jna %l[error]\n\t" \
194194 _ASM_EXTABLE(1b, %l[fault]) \
195− : : op1 : "cc" : error, fault); \
195+ : : op1 : "cc", "memory" : error, fault); \
196196 return; \
197197 error: \
198198 instrumentation_begin(); \
@@ -208,7 +208,7 @@ do { \ · 9 unmodified lines
208208 asm goto("1: " __stringify(insn) " %1, %0\n\t" \
209209 "jna %l[error]\n\t" \
210210 _ASM_EXTABLE(1b, %l[fault]) \
211− : : op1, op2 : "cc" : error, fault); \
211+ : : op1, op2 : "cc", "memory" : error, fault);\
212212 return; \
213213 error: \
214214 instrumentation_begin(); \
arch/x86/kvm/x86.c+6−0
@@ -13429,9 +13429,15 @@ void kvm_arch_pre_destroy_vm(struct kvm *kvm) · 13428 unmodified lines
1342913429 * iterating over vCPUs in a different task while vCPUs are being freed
1343013430 * is unsafe, i.e. will lead to use-after-free. The PIT also needs to
1343113431 * be stopped before IRQ routing is freed.
13432+ *
13433+ * Do NOT free the in-kernel PIC or I/O APIC here (but as above, make
13434+ * sure to flush any background work), as KVM expects interrupt routing
13435+ * structures to be valid until vCPUs are destroyed.
1343213436 */
1343313437 #ifdef CONFIG_KVM_IOAPIC
1343413438 kvm_free_pit(kvm);
13439+ if (kvm->arch.vioapic)
13440+ cancel_delayed_work_sync(&kvm->arch.vioapic->eoi_inject);
1343513441 #endif
1343613442
1343713443 kvm_mmu_pre_destroy_vm(kvm);
include/kvm/arm_vgic.h+0−3
@@ -247,9 +247,6 @@ struct vgic_irq { · 246 unmodified lines
247247 * affinity reg (v3).
248248 */
249249
250− bool pending_release:1; /* Used for LPIs only, unreferenced IRQ
251− * pending a release */
252−
253250 bool pending_latch:1; /* The pending latch state used to calculate
254251 * the pending state for both level
255252 * and edge triggered IRQs. */

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t