History

Merge tag 'for-linus-iommufd' of git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd

Pull iommufd updates from Jason Gunthorpe: "Two significant new items: - Allow reporting IOMMU HW events to userspace when the events are clearly linked to a device. This is linked to the VIOMMU object and is intended to be used by a VMM to forward HW events to the virtual machine as part of emulating a vIOMMU. ARM SMMUv3 is the first driver to use this mechanism. Like the existing fault events the data is delivered through a simple FD returning event records on read(). - PASID support in VFIO. The "Process Address Space ID" is a PCI feature that allows the device to tag all PCI DMA operations with an ID. The IOMMU will then use the ID to select a unique translation for those DMAs. This is part of Intel's vIOMMU support as VT-D HW requires the hypervisor to manage each PASID entry. The support is generic so any VFIO user could attach any translation to a PASID, and the support should work on ARM SMMUv3 as well. AMD requires additional driver work. Some minor updates, along with fixes: - Prevent using nested parents with fault's, no driver support today - Put a single "cookie_type" value in the iommu_domain to indicate what owns the various opaque owner fields" * tag 'for-linus-iommufd' of git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd: (49 commits) iommufd: Test attach before detaching pasid iommufd: Fix iommu_vevent_header tables markup iommu: Convert unreachable() to BUG() iommufd: Balance veventq->num_events inc/dec iommufd: Initialize the flags of vevent in iommufd_viommu_report_event() iommufd/selftest: Add coverage for reporting max_pasid_log2 via IOMMU_HW_INFO iommufd: Extend IOMMU_GET_HW_INFO to report PASID capability vfio: VFIO_DEVICE_[AT|DE]TACH_IOMMUFD_PT support pasid vfio-iommufd: Support pasid [at|de]tach for physical VFIO devices ida: Add ida_find_first_range() iommufd/selftest: Add coverage for iommufd pasid attach/detach iommufd/selftest: Add test ops to test pasid attach/detach iommufd/selftest: Add a helper to get test device iommufd/selftest: Add set_dev_pasid in mock iommu iommufd: Allow allocating PASID-compatible domain iommu/vt-d: Add IOMMU_HWPT_ALLOC_PASID support iommufd: Enforce PASID-compatible domain for RID iommufd: Support pasid attach/replace iommufd: Enforce PASID-compatible domain in PASID path iommufd/device: Add pasid_attach array to track per-PASID attach ...

Linus Torvalds committed 1 year ago

Commit 48552152 parents 792b830 7be11d339 files · +3147 −829Browse files

Unified · Split

39 files
  1. Documentation/userspace-api/iommufd.rst +17 −0
  2. drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +60 −0
  3. drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +52 −28
  4. drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +36 −0
  5. drivers/iommu/dma-iommu.c +104 −100
  6. drivers/iommu/dma-iommu.h +14 −0
  7. drivers/iommu/intel/iommu.c +2 −1
  8. drivers/iommu/intel/nested.c +1 −1
  9. drivers/iommu/iommu-priv.h +16 −0
  10. drivers/iommu/iommu-sva.c +1 −0
  11. drivers/iommu/iommu.c +151 −9
  12. drivers/iommu/iommufd/Kconfig +1 −1
  13. drivers/iommu/iommufd/Makefile +1 −1
  14. drivers/iommu/iommufd/device.c +294 −205
  15. drivers/iommu/iommufd/driver.c +198 −0
  16. drivers/iommu/iommufd/eventq.c +598 −0
  17. drivers/iommu/iommufd/fault.c +0 −342
  18. drivers/iommu/iommufd/hw_pagetable.c +26 −16
  19. drivers/iommu/iommufd/iommufd_private.h +114 −42
  20. drivers/iommu/iommufd/iommufd_test.h +40 −0
  21. drivers/iommu/iommufd/main.c +7 −0
  22. drivers/iommu/iommufd/selftest.c +275 −22
  23. drivers/iommu/iommufd/viommu.c +2 −0
  24. drivers/pci/ats.c +33 −0
  25. drivers/vfio/device_cdev.c +52 −8
  26. drivers/vfio/iommufd.c +56 −4
  27. drivers/vfio/pci/vfio_pci.c +2 −0
  28. include/linux/idr.h +11 −0
  29. include/linux/iommu.h +15 −20
  30. include/linux/iommufd.h +29 −3
  31. include/linux/pci-ats.h +3 −0
  32. include/linux/vfio.h +14 −0
  33. include/uapi/linux/iommufd.h +128 −1
  34. include/uapi/linux/vfio.h +19 −10
  35. lib/idr.c +67 −0
  36. lib/test_ida.c +70 −0
  37. tools/testing/selftests/iommu/iommufd.c +365 −0
  38. tools/testing/selftests/iommu/iommufd_fail_nth.c +51 −8
  39. tools/testing/selftests/iommu/iommufd_utils.h +222 −7
Documentation/userspace-api/iommufd.rst+17−0
@@ -63,6 +63,13 @@ Following IOMMUFD objects are exposed to userspace: · 62 unmodified lines
6363 space usually has mappings from guest-level I/O virtual addresses to guest-
6464 level physical addresses.
6565
66+- IOMMUFD_FAULT, representing a software queue for an HWPT reporting IO page
67+ faults using the IOMMU HW's PRI (Page Request Interface). This queue object
68+ provides user space an FD to poll the page fault events and also to respond
69+ to those events. A FAULT object must be created first to get a fault_id that
70+ could be then used to allocate a fault-enabled HWPT via the IOMMU_HWPT_ALLOC
71+ command by setting the IOMMU_HWPT_FAULT_ID_VALID bit in its flags field.
72+
6673 - IOMMUFD_OBJ_VIOMMU, representing a slice of the physical IOMMU instance,
6774 passed to or shared with a VM. It may be some HW-accelerated virtualization
6875 features and some SW resources used by the VM. For examples:
@@ -109,6 +116,14 @@ Following IOMMUFD objects are exposed to userspace: · 40 unmodified lines
109116 vIOMMU, which is a separate ioctl call from attaching the same device to an
110117 HWPT_PAGING that the vIOMMU holds.
111118
119+- IOMMUFD_OBJ_VEVENTQ, representing a software queue for a vIOMMU to report its
120+ events such as translation faults occurred to a nested stage-1 (excluding I/O
121+ page faults that should go through IOMMUFD_OBJ_FAULT) and HW-specific events.
122+ This queue object provides user space an FD to poll/read the vIOMMU events. A
123+ vIOMMU object must be created first to get its viommu_id, which could be then
124+ used to allocate a vEVENTQ. Each vIOMMU can support multiple types of vEVENTS,
125+ but is confined to one vEVENTQ per vEVENTQ type.
126+
112127 All user-visible objects are destroyed via the IOMMU_DESTROY uAPI.
113128
114129 The diagrams below show relationships between user-visible objects and kernel
@@ -251,8 +266,10 @@ User visible objects are backed by following datastructures: · 136 unmodified lines
251266 - iommufd_device for IOMMUFD_OBJ_DEVICE.
252267 - iommufd_hwpt_paging for IOMMUFD_OBJ_HWPT_PAGING.
253268 - iommufd_hwpt_nested for IOMMUFD_OBJ_HWPT_NESTED.
269+- iommufd_fault for IOMMUFD_OBJ_FAULT.
254270 - iommufd_viommu for IOMMUFD_OBJ_VIOMMU.
255271 - iommufd_vdevice for IOMMUFD_OBJ_VDEVICE.
272+- iommufd_veventq for IOMMUFD_OBJ_VEVENTQ.
256273
257274 Several terminologies when looking at these datastructures:
258275
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c+60−0
@@ -43,6 +43,8 @@ static void arm_smmu_make_nested_cd_table_ste( · 42 unmodified lines
4343 target->data[0] |= nested_domain->ste[0] &
4444 ~cpu_to_le64(STRTAB_STE_0_CFG);
4545 target->data[1] |= nested_domain->ste[1];
46+ /* Merge events for DoS mitigations on eventq */
47+ target->data[1] |= cpu_to_le64(STRTAB_STE_1_MEV);
4648 }
4749
4850 /*
@@ -85,6 +87,47 @@ static void arm_smmu_make_nested_domain_ste( · 36 unmodified lines
8587 }
8688 }
8789
90+int arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state,
91+ struct arm_smmu_nested_domain *nested_domain)
92+{
93+ struct arm_smmu_vmaster *vmaster;
94+ unsigned long vsid;
95+ int ret;
96+
97+ iommu_group_mutex_assert(state->master->dev);
98+
99+ ret = iommufd_viommu_get_vdev_id(&nested_domain->vsmmu->core,
100+ state->master->dev, &vsid);
101+ if (ret)
102+ return ret;
103+
104+ vmaster = kzalloc(sizeof(*vmaster), GFP_KERNEL);
105+ if (!vmaster)
106+ return -ENOMEM;
107+ vmaster->vsmmu = nested_domain->vsmmu;
108+ vmaster->vsid = vsid;
109+ state->vmaster = vmaster;
110+
111+ return 0;
112+}
113+
114+void arm_smmu_attach_commit_vmaster(struct arm_smmu_attach_state *state)
115+{
116+ struct arm_smmu_master *master = state->master;
117+
118+ mutex_lock(&master->smmu->streams_mutex);
119+ kfree(master->vmaster);
120+ master->vmaster = state->vmaster;
121+ mutex_unlock(&master->smmu->streams_mutex);
122+}
123+
124+void arm_smmu_master_clear_vmaster(struct arm_smmu_master *master)
125+{
126+ struct arm_smmu_attach_state state = { .master = master };
127+
128+ arm_smmu_attach_commit_vmaster(&state);
129+}
130+
88131 static int arm_smmu_attach_dev_nested(struct iommu_domain *domain,
89132 struct device *dev)
90133 {
@@ -392,4 +435,21 @@ struct iommufd_viommu *arm_vsmmu_alloc(struct device *dev, · 301 unmodified lines
392435 return &vsmmu->core;
393436 }
394437
438+int arm_vmaster_report_event(struct arm_smmu_vmaster *vmaster, u64 *evt)
439+{
440+ struct iommu_vevent_arm_smmuv3 vevt;
441+ int i;
442+
443+ lockdep_assert_held(&vmaster->vsmmu->smmu->streams_mutex);
444+
445+ vevt.evt[0] = cpu_to_le64((evt[0] & ~EVTQ_0_SID) |
446+ FIELD_PREP(EVTQ_0_SID, vmaster->vsid));
447+ for (i = 1; i < EVTQ_ENT_DWORDS; i++)
448+ vevt.evt[i] = cpu_to_le64(evt[i]);
449+
450+ return iommufd_viommu_report_event(&vmaster->vsmmu->core,
451+ IOMMU_VEVENTQ_TYPE_ARM_SMMUV3, &vevt,
452+ sizeof(vevt));
453+}
454+
395455 MODULE_IMPORT_NS("IOMMUFD");
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c+52−28
@@ -1052,7 +1052,7 @@ void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) · 1051 unmodified lines
10521052 cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR |
10531053 STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH |
10541054 STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW |
1055− STRTAB_STE_1_EATS);
1055+ STRTAB_STE_1_EATS | STRTAB_STE_1_MEV);
10561056 used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
10571057
10581058 /*
@@ -1068,7 +1068,7 @@ void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) · 9 unmodified lines
10681068 if (cfg & BIT(1)) {
10691069 used_bits[1] |=
10701070 cpu_to_le64(STRTAB_STE_1_S2FWB | STRTAB_STE_1_EATS |
1071− STRTAB_STE_1_SHCFG);
1071+ STRTAB_STE_1_SHCFG | STRTAB_STE_1_MEV);
10721072 used_bits[2] |=
10731073 cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR |
10741074 STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI |
@@ -1813,8 +1813,8 @@ static void arm_smmu_decode_event(struct arm_smmu_device *smmu, u64 *raw, · 738 unmodified lines
18131813 mutex_unlock(&smmu->streams_mutex);
18141814 }
18151815
1816−static int arm_smmu_handle_event(struct arm_smmu_device *smmu,
1817− struct arm_smmu_event *event)
1816+static int arm_smmu_handle_event(struct arm_smmu_device *smmu, u64 *evt,
1817+ struct arm_smmu_event *event)
18181818 {
18191819 int ret = 0;
18201820 u32 perm = 0;
@@ -1823,6 +1823,10 @@ static int arm_smmu_handle_event(struct arm_smmu_device *smmu, · 2 unmodified lines
18231823 struct iommu_fault *flt = &fault_evt.fault;
18241824
18251825 switch (event->id) {
1826+ case EVT_ID_BAD_STE_CONFIG:
1827+ case EVT_ID_STREAM_DISABLED_FAULT:
1828+ case EVT_ID_BAD_SUBSTREAMID_CONFIG:
1829+ case EVT_ID_BAD_CD_CONFIG:
18261830 case EVT_ID_TRANSLATION_FAULT:
18271831 case EVT_ID_ADDR_SIZE_FAULT:
18281832 case EVT_ID_ACCESS_FAULT:
@@ -1832,31 +1836,30 @@ static int arm_smmu_handle_event(struct arm_smmu_device *smmu, · 3 unmodified lines
18321836 return -EOPNOTSUPP;
18331837 }
18341838
1835− if (!event->stall)
1836− return -EOPNOTSUPP;
1837−
1838− if (event->read)
1839− perm |= IOMMU_FAULT_PERM_READ;
1840− else
1841− perm |= IOMMU_FAULT_PERM_WRITE;
1839+ if (event->stall) {
1840+ if (event->read)
1841+ perm |= IOMMU_FAULT_PERM_READ;
1842+ else
1843+ perm |= IOMMU_FAULT_PERM_WRITE;
18421844
1843− if (event->instruction)
1844− perm |= IOMMU_FAULT_PERM_EXEC;
1845+ if (event->instruction)
1846+ perm |= IOMMU_FAULT_PERM_EXEC;
18451847
1846− if (event->privileged)
1847− perm |= IOMMU_FAULT_PERM_PRIV;
1848+ if (event->privileged)
1849+ perm |= IOMMU_FAULT_PERM_PRIV;
18481850
1849− flt->type = IOMMU_FAULT_PAGE_REQ;
1850− flt->prm = (struct iommu_fault_page_request) {
1851− .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE,
1852− .grpid = event->stag,
1853− .perm = perm,
1854− .addr = event->iova,
1855− };
1851+ flt->type = IOMMU_FAULT_PAGE_REQ;
1852+ flt->prm = (struct iommu_fault_page_request){
1853+ .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE,
1854+ .grpid = event->stag,
1855+ .perm = perm,
1856+ .addr = event->iova,
1857+ };
18561858
1857− if (event->ssv) {
1858− flt->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID;
1859− flt->prm.pasid = event->ssid;
1859+ if (event->ssv) {
1860+ flt->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID;
1861+ flt->prm.pasid = event->ssid;
1862+ }
18601863 }
18611864
18621865 mutex_lock(&smmu->streams_mutex);
@@ -1866,7 +1869,12 @@ static int arm_smmu_handle_event(struct arm_smmu_device *smmu, · 3 unmodified lines
18661869 goto out_unlock;
18671870 }
18681871
1869− ret = iommu_report_device_fault(master->dev, &fault_evt);
1872+ if (event->stall)
1873+ ret = iommu_report_device_fault(master->dev, &fault_evt);
1874+ else if (master->vmaster && !event->s2)
1875+ ret = arm_vmaster_report_event(master->vmaster, evt);
1876+ else
1877+ ret = -EOPNOTSUPP; /* Unhandled events should be pinned */
18701878 out_unlock:
18711879 mutex_unlock(&smmu->streams_mutex);
18721880 return ret;
@@ -1944,7 +1952,7 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) · 71 unmodified lines
19441952 do {
19451953 while (!queue_remove_raw(q, evt)) {
19461954 arm_smmu_decode_event(smmu, evt, &event);
1947− if (arm_smmu_handle_event(smmu, &event))
1955+ if (arm_smmu_handle_event(smmu, evt, &event))
19481956 arm_smmu_dump_event(smmu, evt, &event, &rs);
19491957
19501958 put_device(event.dev);
@@ -2803,6 +2811,7 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state, · 852 unmodified lines
28032811 struct arm_smmu_domain *smmu_domain =
28042812 to_smmu_domain_devices(new_domain);
28052813 unsigned long flags;
2814+ int ret;
28062815
28072816 /*
28082817 * arm_smmu_share_asid() must not see two domains pointing to the same
@@ -2832,9 +2841,18 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state, · 23 unmodified lines
28322841 }
28332842
28342843 if (smmu_domain) {
2844+ if (new_domain->type == IOMMU_DOMAIN_NESTED) {
2845+ ret = arm_smmu_attach_prepare_vmaster(
2846+ state, to_smmu_nested_domain(new_domain));
2847+ if (ret)
2848+ return ret;
2849+ }
2850+
28352851 master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
2836− if (!master_domain)
2852+ if (!master_domain) {
2853+ kfree(state->vmaster);
28372854 return -ENOMEM;
2855+ }
28382856 master_domain->master = master;
28392857 master_domain->ssid = state->ssid;
28402858 if (new_domain->type == IOMMU_DOMAIN_NESTED)
@@ -2861,6 +2879,7 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state, · 20 unmodified lines
28612879 spin_unlock_irqrestore(&smmu_domain->devices_lock,
28622880 flags);
28632881 kfree(master_domain);
2882+ kfree(state->vmaster);
28642883 return -EINVAL;
28652884 }
28662885
@@ -2893,6 +2912,8 @@ void arm_smmu_attach_commit(struct arm_smmu_attach_state *state) · 26 unmodified lines
28932912
28942913 lockdep_assert_held(&arm_smmu_asid_lock);
28952914
2915+ arm_smmu_attach_commit_vmaster(state);
2916+
28962917 if (state->ats_enabled && !master->ats_enabled) {
28972918 arm_smmu_enable_ats(master);
28982919 } else if (state->ats_enabled && master->ats_enabled) {
@@ -3162,6 +3183,7 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain, · 263 unmodified lines
31623183 struct arm_smmu_ste ste;
31633184 struct arm_smmu_master *master = dev_iommu_priv_get(dev);
31643185
3186+ arm_smmu_master_clear_vmaster(master);
31653187 arm_smmu_make_bypass_ste(master->smmu, &ste);
31663188 arm_smmu_attach_dev_ste(domain, dev, &ste, STRTAB_STE_1_S1DSS_BYPASS);
31673189 return 0;
@@ -3180,7 +3202,9 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain, · 12 unmodified lines
31803202 struct device *dev)
31813203 {
31823204 struct arm_smmu_ste ste;
3205+ struct arm_smmu_master *master = dev_iommu_priv_get(dev);
31833206
3207+ arm_smmu_master_clear_vmaster(master);
31843208 arm_smmu_make_abort_ste(&ste);
31853209 arm_smmu_attach_dev_ste(domain, dev, &ste,
31863210 STRTAB_STE_1_S1DSS_TERMINATE);
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h+36−0
@@ -266,6 +266,7 @@ static inline u32 arm_smmu_strtab_l2_idx(u32 sid) · 265 unmodified lines
266266 #define STRTAB_STE_1_S1COR GENMASK_ULL(5, 4)
267267 #define STRTAB_STE_1_S1CSH GENMASK_ULL(7, 6)
268268
269+#define STRTAB_STE_1_MEV (1UL << 19)
269270 #define STRTAB_STE_1_S2FWB (1UL << 25)
270271 #define STRTAB_STE_1_S1STALLD (1UL << 27)
271272
@@ -799,6 +800,11 @@ struct arm_smmu_stream { · 527 unmodified lines
799800 struct rb_node node;
800801 };
801802
803+struct arm_smmu_vmaster {
804+ struct arm_vsmmu *vsmmu;
805+ unsigned long vsid;
806+};
807+
802808 struct arm_smmu_event {
803809 u8 stall : 1,
804810 ssv : 1,
@@ -824,6 +830,7 @@ struct arm_smmu_master { · 19 unmodified lines
824830 struct arm_smmu_device *smmu;
825831 struct device *dev;
826832 struct arm_smmu_stream *streams;
833+ struct arm_smmu_vmaster *vmaster; /* use smmu->streams_mutex */
827834 /* Locked by the iommu core using the group mutex */
828835 struct arm_smmu_ctx_desc_cfg cd_table;
829836 unsigned int num_streams;
@@ -972,6 +979,7 @@ struct arm_smmu_attach_state { · 142 unmodified lines
972979 bool disable_ats;
973980 ioasid_t ssid;
974981 /* Resulting state */
982+ struct arm_smmu_vmaster *vmaster;
975983 bool ats_enabled;
976984 };
977985
@@ -1055,9 +1063,37 @@ struct iommufd_viommu *arm_vsmmu_alloc(struct device *dev, · 77 unmodified lines
10551063 struct iommu_domain *parent,
10561064 struct iommufd_ctx *ictx,
10571065 unsigned int viommu_type);
1066+int arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state,
1067+ struct arm_smmu_nested_domain *nested_domain);
1068+void arm_smmu_attach_commit_vmaster(struct arm_smmu_attach_state *state);
1069+void arm_smmu_master_clear_vmaster(struct arm_smmu_master *master);
1070+int arm_vmaster_report_event(struct arm_smmu_vmaster *vmaster, u64 *evt);
10581071 #else
10591072 #define arm_smmu_hw_info NULL
10601073 #define arm_vsmmu_alloc NULL
1074+
1075+static inline int
1076+arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state,
1077+ struct arm_smmu_nested_domain *nested_domain)
1078+{
1079+ return 0;
1080+}
1081+
1082+static inline void
1083+arm_smmu_attach_commit_vmaster(struct arm_smmu_attach_state *state)
1084+{
1085+}
1086+
1087+static inline void
1088+arm_smmu_master_clear_vmaster(struct arm_smmu_master *master)
1089+{
1090+}
1091+
1092+static inline int arm_vmaster_report_event(struct arm_smmu_vmaster *vmaster,
1093+ u64 *evt)
1094+{
1095+ return -EOPNOTSUPP;
1096+}
10611097 #endif /* CONFIG_ARM_SMMU_V3_IOMMUFD */
10621098
10631099 #endif /* _ARM_SMMU_V3_H */
drivers/iommu/dma-iommu.c+104−100
@@ -42,11 +42,6 @@ struct iommu_dma_msi_page { · 41 unmodified lines
4242 phys_addr_t phys;
4343 };
4444
45−enum iommu_dma_cookie_type {
46− IOMMU_DMA_IOVA_COOKIE,
47− IOMMU_DMA_MSI_COOKIE,
48−};
49−
5045 enum iommu_dma_queue_type {
5146 IOMMU_DMA_OPTS_PER_CPU_QUEUE,
5247 IOMMU_DMA_OPTS_SINGLE_QUEUE,
@@ -59,34 +54,30 @@ struct iommu_dma_options { · 6 unmodified lines
5954 };
6055
6156 struct iommu_dma_cookie {
62− enum iommu_dma_cookie_type type;
57+ struct iova_domain iovad;
58+ struct list_head msi_page_list;
59+ /* Flush queue */
6360 union {
64− /* Full allocator for IOMMU_DMA_IOVA_COOKIE */
65− struct {
66− struct iova_domain iovad;
67− /* Flush queue */
68− union {
69− struct iova_fq *single_fq;
70− struct iova_fq __percpu *percpu_fq;
71− };
72− /* Number of TLB flushes that have been started */
73− atomic64_t fq_flush_start_cnt;
74− /* Number of TLB flushes that have been finished */
75− atomic64_t fq_flush_finish_cnt;
76− /* Timer to regularily empty the flush queues */
77− struct timer_list fq_timer;
78− /* 1 when timer is active, 0 when not */
79− atomic_t fq_timer_on;
80− };
81− /* Trivial linear page allocator for IOMMU_DMA_MSI_COOKIE */
82− dma_addr_t msi_iova;
61+ struct iova_fq *single_fq;
62+ struct iova_fq __percpu *percpu_fq;
8363 };
84− struct list_head msi_page_list;
85−
64+ /* Number of TLB flushes that have been started */
65+ atomic64_t fq_flush_start_cnt;
66+ /* Number of TLB flushes that have been finished */
67+ atomic64_t fq_flush_finish_cnt;
68+ /* Timer to regularily empty the flush queues */
69+ struct timer_list fq_timer;
70+ /* 1 when timer is active, 0 when not */
71+ atomic_t fq_timer_on;
8672 /* Domain for flush queue callback; NULL if flush queue not in use */
87− struct iommu_domain *fq_domain;
73+ struct iommu_domain *fq_domain;
8874 /* Options for dma-iommu use */
89− struct iommu_dma_options options;
75+ struct iommu_dma_options options;
76+};
77+
78+struct iommu_dma_msi_cookie {
79+ dma_addr_t msi_iova;
80+ struct list_head msi_page_list;
9081 };
9182
9283 static DEFINE_STATIC_KEY_FALSE(iommu_deferred_attach_enabled);
@@ -102,9 +93,6 @@ static int __init iommu_dma_forcedac_setup(char *str) · 9 unmodified lines
10293 }
10394 early_param("iommu.forcedac", iommu_dma_forcedac_setup);
10495
105−static int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
106− phys_addr_t msi_addr);
107−
10896 /* Number of entries per flush queue */
10997 #define IOVA_DEFAULT_FQ_SIZE 256
11098 #define IOVA_SINGLE_FQ_SIZE 32768
@@ -368,39 +356,24 @@ int iommu_dma_init_fq(struct iommu_domain *domain) · 257 unmodified lines
368356 return 0;
369357 }
370358
371−static inline size_t cookie_msi_granule(struct iommu_dma_cookie *cookie)
372−{
373− if (cookie->type == IOMMU_DMA_IOVA_COOKIE)
374− return cookie->iovad.granule;
375− return PAGE_SIZE;
376−}
377−
378−static struct iommu_dma_cookie *cookie_alloc(enum iommu_dma_cookie_type type)
379−{
380− struct iommu_dma_cookie *cookie;
381−
382− cookie = kzalloc(sizeof(*cookie), GFP_KERNEL);
383− if (cookie) {
384− INIT_LIST_HEAD(&cookie->msi_page_list);
385− cookie->type = type;
386− }
387− return cookie;
388−}
389−
390359 /**
391360 * iommu_get_dma_cookie - Acquire DMA-API resources for a domain
392361 * @domain: IOMMU domain to prepare for DMA-API usage
393362 */
394363 int iommu_get_dma_cookie(struct iommu_domain *domain)
395364 {
396− if (domain->iova_cookie)
365+ struct iommu_dma_cookie *cookie;
366+
367+ if (domain->cookie_type != IOMMU_COOKIE_NONE)
397368 return -EEXIST;
398369
399− domain->iova_cookie = cookie_alloc(IOMMU_DMA_IOVA_COOKIE);
400− if (!domain->iova_cookie)
370+ cookie = kzalloc(sizeof(*cookie), GFP_KERNEL);
371+ if (!cookie)
401372 return -ENOMEM;
402373
403− iommu_domain_set_sw_msi(domain, iommu_dma_sw_msi);
374+ INIT_LIST_HEAD(&cookie->msi_page_list);
375+ domain->cookie_type = IOMMU_COOKIE_DMA_IOVA;
376+ domain->iova_cookie = cookie;
404377 return 0;
405378 }
406379
@@ -418,54 +391,56 @@ int iommu_get_dma_cookie(struct iommu_domain *domain) · 11 unmodified lines
418391 */
419392 int iommu_get_msi_cookie(struct iommu_domain *domain, dma_addr_t base)
420393 {
421− struct iommu_dma_cookie *cookie;
394+ struct iommu_dma_msi_cookie *cookie;
422395
423396 if (domain->type != IOMMU_DOMAIN_UNMANAGED)
424397 return -EINVAL;
425398
426− if (domain->iova_cookie)
399+ if (domain->cookie_type != IOMMU_COOKIE_NONE)
427400 return -EEXIST;
428401
429− cookie = cookie_alloc(IOMMU_DMA_MSI_COOKIE);
402+ cookie = kzalloc(sizeof(*cookie), GFP_KERNEL);
430403 if (!cookie)
431404 return -ENOMEM;
432405
433406 cookie->msi_iova = base;
434− domain->iova_cookie = cookie;
435− iommu_domain_set_sw_msi(domain, iommu_dma_sw_msi);
407+ INIT_LIST_HEAD(&cookie->msi_page_list);
408+ domain->cookie_type = IOMMU_COOKIE_DMA_MSI;
409+ domain->msi_cookie = cookie;
436410 return 0;
437411 }
438412 EXPORT_SYMBOL(iommu_get_msi_cookie);
439413
440414 /**
441415 * iommu_put_dma_cookie - Release a domain's DMA mapping resources
442− * @domain: IOMMU domain previously prepared by iommu_get_dma_cookie() or
443− * iommu_get_msi_cookie()
416+ * @domain: IOMMU domain previously prepared by iommu_get_dma_cookie()
444417 */
445418 void iommu_put_dma_cookie(struct iommu_domain *domain)
446419 {
447420 struct iommu_dma_cookie *cookie = domain->iova_cookie;
448421 struct iommu_dma_msi_page *msi, *tmp;
449422
450−#if IS_ENABLED(CONFIG_IRQ_MSI_IOMMU)
451− if (domain->sw_msi != iommu_dma_sw_msi)
452− return;
453−#endif
454−
455− if (!cookie)
456− return;
457−
458− if (cookie->type == IOMMU_DMA_IOVA_COOKIE && cookie->iovad.granule) {
423+ if (cookie->iovad.granule) {
459424 iommu_dma_free_fq(cookie);
460425 put_iova_domain(&cookie->iovad);
461426 }
427+ list_for_each_entry_safe(msi, tmp, &cookie->msi_page_list, list)
428+ kfree(msi);
429+ kfree(cookie);
430+}
431+
432+/**
433+ * iommu_put_msi_cookie - Release a domain's MSI mapping resources
434+ * @domain: IOMMU domain previously prepared by iommu_get_msi_cookie()
435+ */
436+void iommu_put_msi_cookie(struct iommu_domain *domain)
437+{
438+ struct iommu_dma_msi_cookie *cookie = domain->msi_cookie;
439+ struct iommu_dma_msi_page *msi, *tmp;
462440
463− list_for_each_entry_safe(msi, tmp, &cookie->msi_page_list, list) {
464− list_del(&msi->list);
441+ list_for_each_entry_safe(msi, tmp, &cookie->msi_page_list, list)
465442 kfree(msi);
466− }
467443 kfree(cookie);
468− domain->iova_cookie = NULL;
469444 }
470445
471446 /**
@@ -685,7 +660,7 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, struct device *dev · 213 unmodified lines
685660 struct iova_domain *iovad;
686661 int ret;
687662
688− if (!cookie || cookie->type != IOMMU_DMA_IOVA_COOKIE)
663+ if (!cookie || domain->cookie_type != IOMMU_COOKIE_DMA_IOVA)
689664 return -EINVAL;
690665
691666 iovad = &cookie->iovad;
@@ -768,9 +743,9 @@ static dma_addr_t iommu_dma_alloc_iova(struct iommu_domain *domain, · 76 unmodified lines
768743 struct iova_domain *iovad = &cookie->iovad;
769744 unsigned long shift, iova_len, iova;
770745
771− if (cookie->type == IOMMU_DMA_MSI_COOKIE) {
772− cookie->msi_iova += size;
773− return cookie->msi_iova - size;
746+ if (domain->cookie_type == IOMMU_COOKIE_DMA_MSI) {
747+ domain->msi_cookie->msi_iova += size;
748+ return domain->msi_cookie->msi_iova - size;
774749 }
775750
776751 shift = iova_shift(iovad);
@@ -807,16 +782,16 @@ done: · 30 unmodified lines
807782 return (dma_addr_t)iova << shift;
808783 }
809784
810−static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie,
811− dma_addr_t iova, size_t size, struct iommu_iotlb_gather *gather)
785+static void iommu_dma_free_iova(struct iommu_domain *domain, dma_addr_t iova,
786+ size_t size, struct iommu_iotlb_gather *gather)
812787 {
813− struct iova_domain *iovad = &cookie->iovad;
788+ struct iova_domain *iovad = &domain->iova_cookie->iovad;
814789
815790 /* The MSI case is only ever cleaning up its most recent allocation */
816− if (cookie->type == IOMMU_DMA_MSI_COOKIE)
817− cookie->msi_iova -= size;
791+ if (domain->cookie_type == IOMMU_COOKIE_DMA_MSI)
792+ domain->msi_cookie->msi_iova -= size;
818793 else if (gather && gather->queued)
819− queue_iova(cookie, iova_pfn(iovad, iova),
794+ queue_iova(domain->iova_cookie, iova_pfn(iovad, iova),
820795 size >> iova_shift(iovad),
821796 &gather->freelist);
822797 else
@@ -844,7 +819,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, · 21 unmodified lines
844819
845820 if (!iotlb_gather.queued)
846821 iommu_iotlb_sync(domain, &iotlb_gather);
847− iommu_dma_free_iova(cookie, dma_addr, size, &iotlb_gather);
822+ iommu_dma_free_iova(domain, dma_addr, size, &iotlb_gather);
848823 }
849824
850825 static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys,
@@ -872,7 +847,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, · 21 unmodified lines
872847 return DMA_MAPPING_ERROR;
873848
874849 if (iommu_map(domain, iova, phys - iova_off, size, prot, GFP_ATOMIC)) {
875− iommu_dma_free_iova(cookie, iova, size, NULL);
850+ iommu_dma_free_iova(domain, iova, size, NULL);
876851 return DMA_MAPPING_ERROR;
877852 }
878853 return iova + iova_off;
@@ -1009,7 +984,7 @@ static struct page **__iommu_dma_alloc_noncontiguous(struct device *dev, · 130 unmodified lines
1009984 out_free_sg:
1010985 sg_free_table(sgt);
1011986 out_free_iova:
1012− iommu_dma_free_iova(cookie, iova, size, NULL);
987+ iommu_dma_free_iova(domain, iova, size, NULL);
1013988 out_free_pages:
1014989 __iommu_dma_free_pages(pages, count);
1015990 return NULL;
@@ -1486,7 +1461,7 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents, · 470 unmodified lines
14861461 return __finalise_sg(dev, sg, nents, iova);
14871462
14881463 out_free_iova:
1489− iommu_dma_free_iova(cookie, iova, iova_len, NULL);
1464+ iommu_dma_free_iova(domain, iova, iova_len, NULL);
14901465 out_restore_sg:
14911466 __invalidate_sg(sg, nents);
14921467 out:
@@ -1764,17 +1739,47 @@ out_err: · 271 unmodified lines
17641739 dev->dma_iommu = false;
17651740 }
17661741
1742+static bool has_msi_cookie(const struct iommu_domain *domain)
1743+{
1744+ return domain && (domain->cookie_type == IOMMU_COOKIE_DMA_IOVA ||
1745+ domain->cookie_type == IOMMU_COOKIE_DMA_MSI);
1746+}
1747+
1748+static size_t cookie_msi_granule(const struct iommu_domain *domain)
1749+{
1750+ switch (domain->cookie_type) {
1751+ case IOMMU_COOKIE_DMA_IOVA:
1752+ return domain->iova_cookie->iovad.granule;
1753+ case IOMMU_COOKIE_DMA_MSI:
1754+ return PAGE_SIZE;
1755+ default:
1756+ BUG();
1757+ };
1758+}
1759+
1760+static struct list_head *cookie_msi_pages(const struct iommu_domain *domain)
1761+{
1762+ switch (domain->cookie_type) {
1763+ case IOMMU_COOKIE_DMA_IOVA:
1764+ return &domain->iova_cookie->msi_page_list;
1765+ case IOMMU_COOKIE_DMA_MSI:
1766+ return &domain->msi_cookie->msi_page_list;
1767+ default:
1768+ BUG();
1769+ };
1770+}
1771+
17671772 static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev,
17681773 phys_addr_t msi_addr, struct iommu_domain *domain)
17691774 {
1770− struct iommu_dma_cookie *cookie = domain->iova_cookie;
1775+ struct list_head *msi_page_list = cookie_msi_pages(domain);
17711776 struct iommu_dma_msi_page *msi_page;
17721777 dma_addr_t iova;
17731778 int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
1774− size_t size = cookie_msi_granule(cookie);
1779+ size_t size = cookie_msi_granule(domain);
17751780
17761781 msi_addr &= ~(phys_addr_t)(size - 1);
1777− list_for_each_entry(msi_page, &cookie->msi_page_list, list)
1782+ list_for_each_entry(msi_page, msi_page_list, list)
17781783 if (msi_page->phys == msi_addr)
17791784 return msi_page;
17801785
@@ -1792,23 +1797,23 @@ static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev, · 11 unmodified lines
17921797 INIT_LIST_HEAD(&msi_page->list);
17931798 msi_page->phys = msi_addr;
17941799 msi_page->iova = iova;
1795− list_add(&msi_page->list, &cookie->msi_page_list);
1800+ list_add(&msi_page->list, msi_page_list);
17961801 return msi_page;
17971802
17981803 out_free_iova:
1799− iommu_dma_free_iova(cookie, iova, size, NULL);
1804+ iommu_dma_free_iova(domain, iova, size, NULL);
18001805 out_free_page:
18011806 kfree(msi_page);
18021807 return NULL;
18031808 }
18041809
1805−static int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
1806− phys_addr_t msi_addr)
1810+int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
1811+ phys_addr_t msi_addr)
18071812 {
18081813 struct device *dev = msi_desc_to_dev(desc);
18091814 const struct iommu_dma_msi_page *msi_page;
18101815
1811− if (!domain->iova_cookie) {
1816+ if (!has_msi_cookie(domain)) {
18121817 msi_desc_set_iommu_msi_iova(desc, 0, 0);
18131818 return 0;
18141819 }
@@ -1818,9 +1823,8 @@ static int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, · 3 unmodified lines
18181823 if (!msi_page)
18191824 return -ENOMEM;
18201825
1821− msi_desc_set_iommu_msi_iova(
1822− desc, msi_page->iova,
1823− ilog2(cookie_msi_granule(domain->iova_cookie)));
1826+ msi_desc_set_iommu_msi_iova(desc, msi_page->iova,
1827+ ilog2(cookie_msi_granule(domain)));
18241828 return 0;
18251829 }
18261830
drivers/iommu/dma-iommu.h+14−0
@@ -13,11 +13,15 @@ void iommu_setup_dma_ops(struct device *dev); · 12 unmodified lines
1313
1414 int iommu_get_dma_cookie(struct iommu_domain *domain);
1515 void iommu_put_dma_cookie(struct iommu_domain *domain);
16+void iommu_put_msi_cookie(struct iommu_domain *domain);
1617
1718 int iommu_dma_init_fq(struct iommu_domain *domain);
1819
1920 void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list);
2021
22+int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
23+ phys_addr_t msi_addr);
24+
2125 extern bool iommu_dma_forcedac;
2226
2327 #else /* CONFIG_IOMMU_DMA */
@@ -40,9 +44,19 @@ static inline void iommu_put_dma_cookie(struct iommu_domain *domain) · 16 unmodified lines
4044 {
4145 }
4246
47+static inline void iommu_put_msi_cookie(struct iommu_domain *domain)
48+{
49+}
50+
4351 static inline void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list)
4452 {
4553 }
4654
55+static inline int iommu_dma_sw_msi(struct iommu_domain *domain,
56+ struct msi_desc *desc, phys_addr_t msi_addr)
57+{
58+ return -ENODEV;
59+}
60+
4761 #endif /* CONFIG_IOMMU_DMA */
4862 #endif /* __DMA_IOMMU_H */
drivers/iommu/intel/iommu.c+2−1
@@ -3383,7 +3383,8 @@ intel_iommu_domain_alloc_paging_flags(struct device *dev, u32 flags, · 3382 unmodified lines
33833383 bool first_stage;
33843384
33853385 if (flags &
3386− (~(IOMMU_HWPT_ALLOC_NEST_PARENT | IOMMU_HWPT_ALLOC_DIRTY_TRACKING)))
3386+ (~(IOMMU_HWPT_ALLOC_NEST_PARENT | IOMMU_HWPT_ALLOC_DIRTY_TRACKING |
3387+ IOMMU_HWPT_ALLOC_PASID)))
33873388 return ERR_PTR(-EOPNOTSUPP);
33883389 if (nested_parent && !nested_supported(iommu))
33893390 return ERR_PTR(-EOPNOTSUPP);
drivers/iommu/intel/nested.c+1−1
@@ -198,7 +198,7 @@ intel_iommu_domain_alloc_nested(struct device *dev, struct iommu_domain *parent, · 197 unmodified lines
198198 struct dmar_domain *domain;
199199 int ret;
200200
201− if (!nested_supported(iommu) || flags)
201+ if (!nested_supported(iommu) || flags & ~IOMMU_HWPT_ALLOC_PASID)
202202 return ERR_PTR(-EOPNOTSUPP);
203203
204204 /* Must be nested domain */
drivers/iommu/iommu-priv.h+16−0
@@ -5,6 +5,7 @@ · 4 unmodified lines
55 #define __LINUX_IOMMU_PRIV_H
66
77 #include <linux/iommu.h>
8+#include <linux/msi.h>
89
910 static inline const struct iommu_ops *dev_iommu_ops(struct device *dev)
1011 {
@@ -47,4 +48,19 @@ void iommu_detach_group_handle(struct iommu_domain *domain, · 36 unmodified lines
4748 int iommu_replace_group_handle(struct iommu_group *group,
4849 struct iommu_domain *new_domain,
4950 struct iommu_attach_handle *handle);
51+
52+#if IS_ENABLED(CONFIG_IOMMUFD_DRIVER_CORE) && IS_ENABLED(CONFIG_IRQ_MSI_IOMMU)
53+int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
54+ phys_addr_t msi_addr);
55+#else /* !CONFIG_IOMMUFD_DRIVER_CORE || !CONFIG_IRQ_MSI_IOMMU */
56+static inline int iommufd_sw_msi(struct iommu_domain *domain,
57+ struct msi_desc *desc, phys_addr_t msi_addr)
58+{
59+ return -EOPNOTSUPP;
60+}
61+#endif /* CONFIG_IOMMUFD_DRIVER_CORE && CONFIG_IRQ_MSI_IOMMU */
62+
63+int iommu_replace_device_pasid(struct iommu_domain *domain,
64+ struct device *dev, ioasid_t pasid,
65+ struct iommu_attach_handle *handle);
5066 #endif /* __LINUX_IOMMU_PRIV_H */
drivers/iommu/iommu-sva.c+1−0
@@ -310,6 +310,7 @@ static struct iommu_domain *iommu_sva_domain_alloc(struct device *dev, · 309 unmodified lines
310310 }
311311
312312 domain->type = IOMMU_DOMAIN_SVA;
313+ domain->cookie_type = IOMMU_COOKIE_SVA;
313314 mmgrab(mm);
314315 domain->mm = mm;
315316 domain->owner = ops;
drivers/iommu/iommu.c+151−9
@@ -18,6 +18,7 @@ · 17 unmodified lines
1818 #include <linux/errno.h>
1919 #include <linux/host1x_context_bus.h>
2020 #include <linux/iommu.h>
21+#include <linux/iommufd.h>
2122 #include <linux/idr.h>
2223 #include <linux/err.h>
2324 #include <linux/pci.h>
@@ -539,6 +540,13 @@ static void iommu_deinit_device(struct device *dev) · 515 unmodified lines
539540 dev_iommu_free(dev);
540541 }
541542
543+static struct iommu_domain *pasid_array_entry_to_domain(void *entry)
544+{
545+ if (xa_pointer_tag(entry) == IOMMU_PASID_ARRAY_DOMAIN)
546+ return xa_untag_pointer(entry);
547+ return ((struct iommu_attach_handle *)xa_untag_pointer(entry))->domain;
548+}
549+
542550 DEFINE_MUTEX(iommu_probe_device_lock);
543551
544552 static int __iommu_probe_device(struct device *dev, struct list_head *group_list)
@@ -1973,8 +1981,10 @@ void iommu_set_fault_handler(struct iommu_domain *domain, · 1428 unmodified lines
19731981 iommu_fault_handler_t handler,
19741982 void *token)
19751983 {
1976− BUG_ON(!domain);
1984+ if (WARN_ON(!domain || domain->cookie_type != IOMMU_COOKIE_NONE))
1985+ return;
19771986
1987+ domain->cookie_type = IOMMU_COOKIE_FAULT_HANDLER;
19781988 domain->handler = handler;
19791989 domain->handler_token = token;
19801990 }
@@ -2044,9 +2054,19 @@ EXPORT_SYMBOL_GPL(iommu_paging_domain_alloc_flags); · 63 unmodified lines
20442054
20452055 void iommu_domain_free(struct iommu_domain *domain)
20462056 {
2047− if (domain->type == IOMMU_DOMAIN_SVA)
2057+ switch (domain->cookie_type) {
2058+ case IOMMU_COOKIE_DMA_IOVA:
2059+ iommu_put_dma_cookie(domain);
2060+ break;
2061+ case IOMMU_COOKIE_DMA_MSI:
2062+ iommu_put_msi_cookie(domain);
2063+ break;
2064+ case IOMMU_COOKIE_SVA:
20482065 mmdrop(domain->mm);
2049− iommu_put_dma_cookie(domain);
2066+ break;
2067+ default:
2068+ break;
2069+ }
20502070 if (domain->ops->free)
20512071 domain->ops->free(domain);
20522072 }
@@ -3335,14 +3355,15 @@ static void iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid, · 1282 unmodified lines
33353355 }
33363356
33373357 static int __iommu_set_group_pasid(struct iommu_domain *domain,
3338− struct iommu_group *group, ioasid_t pasid)
3358+ struct iommu_group *group, ioasid_t pasid,
3359+ struct iommu_domain *old)
33393360 {
33403361 struct group_device *device, *last_gdev;
33413362 int ret;
33423363
33433364 for_each_group_device(group, device) {
33443365 ret = domain->ops->set_dev_pasid(domain, device->dev,
3345− pasid, NULL);
3366+ pasid, old);
33463367 if (ret)
33473368 goto err_revert;
33483369 }
@@ -3354,7 +3375,15 @@ err_revert: · 5 unmodified lines
33543375 for_each_group_device(group, device) {
33553376 if (device == last_gdev)
33563377 break;
3357− iommu_remove_dev_pasid(device->dev, pasid, domain);
3378+ /*
3379+ * If no old domain, undo the succeeded devices/pasid.
3380+ * Otherwise, rollback the succeeded devices/pasid to the old
3381+ * domain. And it is a driver bug to fail attaching with a
3382+ * previously good domain.
3383+ */
3384+ if (!old || WARN_ON(old->ops->set_dev_pasid(old, device->dev,
3385+ pasid, domain)))
3386+ iommu_remove_dev_pasid(device->dev, pasid, domain);
33583387 }
33593388 return ret;
33603389 }
@@ -3376,6 +3405,9 @@ static void __iommu_remove_group_pasid(struct iommu_group *group, · 15 unmodified lines
33763405 * @pasid: the pasid of the device.
33773406 * @handle: the attach handle.
33783407 *
3408+ * Caller should always provide a new handle to avoid race with the paths
3409+ * that have lockless reference to handle if it intends to pass a valid handle.
3410+ *
33793411 * Return: 0 on success, or an error.
33803412 */
33813413 int iommu_attach_device_pasid(struct iommu_domain *domain,
@@ -3420,7 +3452,7 @@ int iommu_attach_device_pasid(struct iommu_domain *domain, · 38 unmodified lines
34203452 if (ret)
34213453 goto out_unlock;
34223454
3423− ret = __iommu_set_group_pasid(domain, group, pasid);
3455+ ret = __iommu_set_group_pasid(domain, group, pasid, NULL);
34243456 if (ret) {
34253457 xa_release(&group->pasid_array, pasid);
34263458 goto out_unlock;
@@ -3441,6 +3473,97 @@ out_unlock: · 14 unmodified lines
34413473 }
34423474 EXPORT_SYMBOL_GPL(iommu_attach_device_pasid);
34433475
3476+/**
3477+ * iommu_replace_device_pasid - Replace the domain that a specific pasid
3478+ * of the device is attached to
3479+ * @domain: the new iommu domain
3480+ * @dev: the attached device.
3481+ * @pasid: the pasid of the device.
3482+ * @handle: the attach handle.
3483+ *
3484+ * This API allows the pasid to switch domains. The @pasid should have been
3485+ * attached. Otherwise, this fails. The pasid will keep the old configuration
3486+ * if replacement failed.
3487+ *
3488+ * Caller should always provide a new handle to avoid race with the paths
3489+ * that have lockless reference to handle if it intends to pass a valid handle.
3490+ *
3491+ * Return 0 on success, or an error.
3492+ */
3493+int iommu_replace_device_pasid(struct iommu_domain *domain,
3494+ struct device *dev, ioasid_t pasid,
3495+ struct iommu_attach_handle *handle)
3496+{
3497+ /* Caller must be a probed driver on dev */
3498+ struct iommu_group *group = dev->iommu_group;
3499+ struct iommu_attach_handle *entry;
3500+ struct iommu_domain *curr_domain;
3501+ void *curr;
3502+ int ret;
3503+
3504+ if (!group)
3505+ return -ENODEV;
3506+
3507+ if (!domain->ops->set_dev_pasid)
3508+ return -EOPNOTSUPP;
3509+
3510+ if (dev_iommu_ops(dev) != domain->owner ||
3511+ pasid == IOMMU_NO_PASID || !handle)
3512+ return -EINVAL;
3513+
3514+ mutex_lock(&group->mutex);
3515+ entry = iommu_make_pasid_array_entry(domain, handle);
3516+ curr = xa_cmpxchg(&group->pasid_array, pasid, NULL,
3517+ XA_ZERO_ENTRY, GFP_KERNEL);
3518+ if (xa_is_err(curr)) {
3519+ ret = xa_err(curr);
3520+ goto out_unlock;
3521+ }
3522+
3523+ /*
3524+ * No domain (with or without handle) attached, hence not
3525+ * a replace case.
3526+ */
3527+ if (!curr) {
3528+ xa_release(&group->pasid_array, pasid);
3529+ ret = -EINVAL;
3530+ goto out_unlock;
3531+ }
3532+
3533+ /*
3534+ * Reusing handle is problematic as there are paths that refers
3535+ * the handle without lock. To avoid race, reject the callers that
3536+ * attempt it.
3537+ */
3538+ if (curr == entry) {
3539+ WARN_ON(1);
3540+ ret = -EINVAL;
3541+ goto out_unlock;
3542+ }
3543+
3544+ curr_domain = pasid_array_entry_to_domain(curr);
3545+ ret = 0;
3546+
3547+ if (curr_domain != domain) {
3548+ ret = __iommu_set_group_pasid(domain, group,
3549+ pasid, curr_domain);
3550+ if (ret)
3551+ goto out_unlock;
3552+ }
3553+
3554+ /*
3555+ * The above xa_cmpxchg() reserved the memory, and the
3556+ * group->mutex is held, this cannot fail.
3557+ */
3558+ WARN_ON(xa_is_err(xa_store(&group->pasid_array,
3559+ pasid, entry, GFP_KERNEL)));
3560+
3561+out_unlock:
3562+ mutex_unlock(&group->mutex);
3563+ return ret;
3564+}
3565+EXPORT_SYMBOL_NS_GPL(iommu_replace_device_pasid, "IOMMUFD_INTERNAL");
3566+
34443567 /*
34453568 * iommu_detach_device_pasid() - Detach the domain from pasid of device
34463569 * @domain: the iommu domain.
@@ -3536,6 +3659,9 @@ EXPORT_SYMBOL_NS_GPL(iommu_attach_handle_get, "IOMMUFD_INTERNAL"); · 89 unmodified lines
35363659 * This is a variant of iommu_attach_group(). It allows the caller to provide
35373660 * an attach handle and use it when the domain is attached. This is currently
35383661 * used by IOMMUFD to deliver the I/O page faults.
3662+ *
3663+ * Caller should always provide a new handle to avoid race with the paths
3664+ * that have lockless reference to handle.
35393665 */
35403666 int iommu_attach_group_handle(struct iommu_domain *domain,
35413667 struct iommu_group *group,
@@ -3605,6 +3731,9 @@ EXPORT_SYMBOL_NS_GPL(iommu_detach_group_handle, "IOMMUFD_INTERNAL"); · 63 unmodified lines
36053731 *
36063732 * If the currently attached domain is a core domain (e.g. a default_domain),
36073733 * it will act just like the iommu_attach_group_handle().
3734+ *
3735+ * Caller should always provide a new handle to avoid race with the paths
3736+ * that have lockless reference to handle.
36083737 */
36093738 int iommu_replace_group_handle(struct iommu_group *group,
36103739 struct iommu_domain *new_domain,
@@ -3662,8 +3791,21 @@ int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_addr) · 51 unmodified lines
36623791 return 0;
36633792
36643793 mutex_lock(&group->mutex);
3665− if (group->domain && group->domain->sw_msi)
3666− ret = group->domain->sw_msi(group->domain, desc, msi_addr);
3794+ /* An IDENTITY domain must pass through */
3795+ if (group->domain && group->domain->type != IOMMU_DOMAIN_IDENTITY) {
3796+ switch (group->domain->cookie_type) {
3797+ case IOMMU_COOKIE_DMA_MSI:
3798+ case IOMMU_COOKIE_DMA_IOVA:
3799+ ret = iommu_dma_sw_msi(group->domain, desc, msi_addr);
3800+ break;
3801+ case IOMMU_COOKIE_IOMMUFD:
3802+ ret = iommufd_sw_msi(group->domain, desc, msi_addr);
3803+ break;
3804+ default:
3805+ ret = -EOPNOTSUPP;
3806+ break;
3807+ }
3808+ }
36673809 mutex_unlock(&group->mutex);
36683810 return ret;
36693811 }
drivers/iommu/iommufd/Kconfig+1−1
@@ -1,6 +1,6 @@
11 # SPDX-License-Identifier: GPL-2.0-only
22 config IOMMUFD_DRIVER_CORE
3− tristate
3+ bool
44 default (IOMMUFD_DRIVER || IOMMUFD) if IOMMUFD!=n
55
66 config IOMMUFD
drivers/iommu/iommufd/Makefile+1−1
@@ -1,7 +1,7 @@
11 # SPDX-License-Identifier: GPL-2.0-only
22 iommufd-y := \
33 device.o \
4− fault.o \
4+ eventq.o \
55 hw_pagetable.o \
66 io_pagetable.o \
77 ioas.o \
drivers/iommu/iommufd/device.c+294−205
@@ -3,9 +3,9 @@ · 2 unmodified lines
33 */
44 #include <linux/iommu.h>
55 #include <linux/iommufd.h>
6+#include <linux/pci-ats.h>
67 #include <linux/slab.h>
78 #include <uapi/linux/iommufd.h>
8−#include <linux/msi.h>
99
1010 #include "../iommu-priv.h"
1111 #include "io_pagetable.h"
@@ -18,12 +18,17 @@ MODULE_PARM_DESC( · 6 unmodified lines
1818 "Allow IOMMUFD to bind to devices even if the platform cannot isolate "
1919 "the MSI interrupt window. Enabling this is a security weakness.");
2020
21+struct iommufd_attach {
22+ struct iommufd_hw_pagetable *hwpt;
23+ struct xarray device_array;
24+};
25+
2126 static void iommufd_group_release(struct kref *kref)
2227 {
2328 struct iommufd_group *igroup =
2429 container_of(kref, struct iommufd_group, ref);
2530
26− WARN_ON(igroup->hwpt || !list_empty(&igroup->device_list));
31+ WARN_ON(!xa_empty(&igroup->pasid_attach));
2732
2833 xa_cmpxchg(&igroup->ictx->groups, iommu_group_id(igroup->group), igroup,
2934 NULL, GFP_KERNEL);
@@ -90,7 +95,7 @@ static struct iommufd_group *iommufd_get_group(struct iommufd_ctx *ictx, · 60 unmodified lines
9095
9196 kref_init(&new_igroup->ref);
9297 mutex_init(&new_igroup->lock);
93− INIT_LIST_HEAD(&new_igroup->device_list);
98+ xa_init(&new_igroup->pasid_attach);
9499 new_igroup->sw_msi_start = PHYS_ADDR_MAX;
95100 /* group reference moves into new_igroup */
96101 new_igroup->group = group;
@@ -294,129 +299,24 @@ u32 iommufd_device_to_id(struct iommufd_device *idev) · 197 unmodified lines
294299 }
295300 EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, "IOMMUFD");
296301
297−/*
298− * Get a iommufd_sw_msi_map for the msi physical address requested by the irq
299− * layer. The mapping to IOVA is global to the iommufd file descriptor, every
300− * domain that is attached to a device using the same MSI parameters will use
301− * the same IOVA.
302− */
303−static __maybe_unused struct iommufd_sw_msi_map *
304−iommufd_sw_msi_get_map(struct iommufd_ctx *ictx, phys_addr_t msi_addr,
305− phys_addr_t sw_msi_start)
306−{
307− struct iommufd_sw_msi_map *cur;
308− unsigned int max_pgoff = 0;
309−
310− lockdep_assert_held(&ictx->sw_msi_lock);
311−
312− list_for_each_entry(cur, &ictx->sw_msi_list, sw_msi_item) {
313− if (cur->sw_msi_start != sw_msi_start)
314− continue;
315− max_pgoff = max(max_pgoff, cur->pgoff + 1);
316− if (cur->msi_addr == msi_addr)
317− return cur;
318− }
319−
320− if (ictx->sw_msi_id >=
321− BITS_PER_BYTE * sizeof_field(struct iommufd_sw_msi_maps, bitmap))
322− return ERR_PTR(-EOVERFLOW);
323−
324− cur = kzalloc(sizeof(*cur), GFP_KERNEL);
325− if (!cur)
326− return ERR_PTR(-ENOMEM);
327−
328− cur->sw_msi_start = sw_msi_start;
329− cur->msi_addr = msi_addr;
330− cur->pgoff = max_pgoff;
331− cur->id = ictx->sw_msi_id++;
332− list_add_tail(&cur->sw_msi_item, &ictx->sw_msi_list);
333− return cur;
334−}
335−
336−static int iommufd_sw_msi_install(struct iommufd_ctx *ictx,
337− struct iommufd_hwpt_paging *hwpt_paging,
338− struct iommufd_sw_msi_map *msi_map)
302+static unsigned int iommufd_group_device_num(struct iommufd_group *igroup,
303+ ioasid_t pasid)
339304 {
340− unsigned long iova;
341−
342− lockdep_assert_held(&ictx->sw_msi_lock);
305+ struct iommufd_attach *attach;
306+ struct iommufd_device *idev;
307+ unsigned int count = 0;
308+ unsigned long index;
343309
344− iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE;
345− if (!test_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap)) {
346− int rc;
310+ lockdep_assert_held(&igroup->lock);
347311
348− rc = iommu_map(hwpt_paging->common.domain, iova,
349− msi_map->msi_addr, PAGE_SIZE,
350− IOMMU_WRITE | IOMMU_READ | IOMMU_MMIO,
351− GFP_KERNEL_ACCOUNT);
352− if (rc)
353− return rc;
354− __set_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap);
355− }
356− return 0;
312+ attach = xa_load(&igroup->pasid_attach, pasid);
313+ if (attach)
314+ xa_for_each(&attach->device_array, index, idev)
315+ count++;
316+ return count;
357317 }
358318
359−/*
360− * Called by the irq code if the platform translates the MSI address through the
361− * IOMMU. msi_addr is the physical address of the MSI page. iommufd will
362− * allocate a fd global iova for the physical page that is the same on all
363− * domains and devices.
364− */
365319 #ifdef CONFIG_IRQ_MSI_IOMMU
366−int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
367− phys_addr_t msi_addr)
368−{
369− struct device *dev = msi_desc_to_dev(desc);
370− struct iommufd_hwpt_paging *hwpt_paging;
371− struct iommu_attach_handle *raw_handle;
372− struct iommufd_attach_handle *handle;
373− struct iommufd_sw_msi_map *msi_map;
374− struct iommufd_ctx *ictx;
375− unsigned long iova;
376− int rc;
377−
378− /*
379− * It is safe to call iommu_attach_handle_get() here because the iommu
380− * core code invokes this under the group mutex which also prevents any
381− * change of the attach handle for the duration of this function.
382− */
383− iommu_group_mutex_assert(dev);
384−
385− raw_handle =
386− iommu_attach_handle_get(dev->iommu_group, IOMMU_NO_PASID, 0);
387− if (IS_ERR(raw_handle))
388− return 0;
389− hwpt_paging = find_hwpt_paging(domain->iommufd_hwpt);
390−
391− handle = to_iommufd_handle(raw_handle);
392− /* No IOMMU_RESV_SW_MSI means no change to the msi_msg */
393− if (handle->idev->igroup->sw_msi_start == PHYS_ADDR_MAX)
394− return 0;
395−
396− ictx = handle->idev->ictx;
397− guard(mutex)(&ictx->sw_msi_lock);
398− /*
399− * The input msi_addr is the exact byte offset of the MSI doorbell, we
400− * assume the caller has checked that it is contained with a MMIO region
401− * that is secure to map at PAGE_SIZE.
402− */
403− msi_map = iommufd_sw_msi_get_map(handle->idev->ictx,
404− msi_addr & PAGE_MASK,
405− handle->idev->igroup->sw_msi_start);
406− if (IS_ERR(msi_map))
407− return PTR_ERR(msi_map);
408−
409− rc = iommufd_sw_msi_install(ictx, hwpt_paging, msi_map);
410− if (rc)
411− return rc;
412− __set_bit(msi_map->id, handle->idev->igroup->required_sw_msi.bitmap);
413−
414− iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE;
415− msi_desc_set_iommu_msi_iova(desc, iova, PAGE_SHIFT);
416− return 0;
417−}
418−#endif
419−
420320 static int iommufd_group_setup_msi(struct iommufd_group *igroup,
421321 struct iommufd_hwpt_paging *hwpt_paging)
422322 {
@@ -443,23 +343,39 @@ static int iommufd_group_setup_msi(struct iommufd_group *igroup, · 20 unmodified lines
443343 }
444344 return 0;
445345 }
346+#else
347+static inline int
348+iommufd_group_setup_msi(struct iommufd_group *igroup,
349+ struct iommufd_hwpt_paging *hwpt_paging)
350+{
351+ return 0;
352+}
353+#endif
354+
355+static bool
356+iommufd_group_first_attach(struct iommufd_group *igroup, ioasid_t pasid)
357+{
358+ lockdep_assert_held(&igroup->lock);
359+ return !xa_load(&igroup->pasid_attach, pasid);
360+}
446361
447362 static int
448363 iommufd_device_attach_reserved_iova(struct iommufd_device *idev,
449364 struct iommufd_hwpt_paging *hwpt_paging)
450365 {
366+ struct iommufd_group *igroup = idev->igroup;
451367 int rc;
452368
453− lockdep_assert_held(&idev->igroup->lock);
369+ lockdep_assert_held(&igroup->lock);
454370
455371 rc = iopt_table_enforce_dev_resv_regions(&hwpt_paging->ioas->iopt,
456372 idev->dev,
457− &idev->igroup->sw_msi_start);
373+ &igroup->sw_msi_start);
458374 if (rc)
459375 return rc;
460376
461− if (list_empty(&idev->igroup->device_list)) {
462− rc = iommufd_group_setup_msi(idev->igroup, hwpt_paging);
377+ if (iommufd_group_first_attach(igroup, IOMMU_NO_PASID)) {
378+ rc = iommufd_group_setup_msi(igroup, hwpt_paging);
463379 if (rc) {
464380 iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt,
465381 idev->dev);
@@ -471,13 +387,54 @@ iommufd_device_attach_reserved_iova(struct iommufd_device *idev, · 5 unmodified lines
471387
472388 /* The device attach/detach/replace helpers for attach_handle */
473389
390+static bool iommufd_device_is_attached(struct iommufd_device *idev,
391+ ioasid_t pasid)
392+{
393+ struct iommufd_attach *attach;
394+
395+ attach = xa_load(&idev->igroup->pasid_attach, pasid);
396+ return xa_load(&attach->device_array, idev->obj.id);
397+}
398+
399+static int iommufd_hwpt_pasid_compat(struct iommufd_hw_pagetable *hwpt,
400+ struct iommufd_device *idev,
401+ ioasid_t pasid)
402+{
403+ struct iommufd_group *igroup = idev->igroup;
404+
405+ lockdep_assert_held(&igroup->lock);
406+
407+ if (pasid == IOMMU_NO_PASID) {
408+ unsigned long start = IOMMU_NO_PASID;
409+
410+ if (!hwpt->pasid_compat &&
411+ xa_find_after(&igroup->pasid_attach,
412+ &start, UINT_MAX, XA_PRESENT))
413+ return -EINVAL;
414+ } else {
415+ struct iommufd_attach *attach;
416+
417+ if (!hwpt->pasid_compat)
418+ return -EINVAL;
419+
420+ attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID);
421+ if (attach && attach->hwpt && !attach->hwpt->pasid_compat)
422+ return -EINVAL;
423+ }
424+
425+ return 0;
426+}
427+
474428 static int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *hwpt,
475− struct iommufd_device *idev)
429+ struct iommufd_device *idev,
430+ ioasid_t pasid)
476431 {
477432 struct iommufd_attach_handle *handle;
478433 int rc;
479434
480− lockdep_assert_held(&idev->igroup->lock);
435+ rc = iommufd_hwpt_pasid_compat(hwpt, idev, pasid);
436+ if (rc)
437+ return rc;
481438
482439 handle = kzalloc(sizeof(*handle), GFP_KERNEL);
483440 if (!handle)
@@ -490,8 +447,12 @@ static int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *hwpt, · 6 unmodified lines
490447 }
491448
492449 handle->idev = idev;
493− rc = iommu_attach_group_handle(hwpt->domain, idev->igroup->group,
494− &handle->handle);
450+ if (pasid == IOMMU_NO_PASID)
451+ rc = iommu_attach_group_handle(hwpt->domain, idev->igroup->group,
452+ &handle->handle);
453+ else
454+ rc = iommu_attach_device_pasid(hwpt->domain, idev->dev, pasid,
455+ &handle->handle);
495456 if (rc)
496457 goto out_disable_iopf;
497458
@@ -506,26 +467,31 @@ out_free_handle: · 8 unmodified lines
506467 }
507468
508469 static struct iommufd_attach_handle *
509−iommufd_device_get_attach_handle(struct iommufd_device *idev)
470+iommufd_device_get_attach_handle(struct iommufd_device *idev, ioasid_t pasid)
510471 {
511472 struct iommu_attach_handle *handle;
512473
513474 lockdep_assert_held(&idev->igroup->lock);
514475
515476 handle =
516− iommu_attach_handle_get(idev->igroup->group, IOMMU_NO_PASID, 0);
477+ iommu_attach_handle_get(idev->igroup->group, pasid, 0);
517478 if (IS_ERR(handle))
518479 return NULL;
519480 return to_iommufd_handle(handle);
520481 }
521482
522483 static void iommufd_hwpt_detach_device(struct iommufd_hw_pagetable *hwpt,
523− struct iommufd_device *idev)
484+ struct iommufd_device *idev,
485+ ioasid_t pasid)
524486 {
525487 struct iommufd_attach_handle *handle;
526488
527− handle = iommufd_device_get_attach_handle(idev);
528− iommu_detach_group_handle(hwpt->domain, idev->igroup->group);
489+ handle = iommufd_device_get_attach_handle(idev, pasid);
490+ if (pasid == IOMMU_NO_PASID)
491+ iommu_detach_group_handle(hwpt->domain, idev->igroup->group);
492+ else
493+ iommu_detach_device_pasid(hwpt->domain, idev->dev, pasid);
494+
529495 if (hwpt->fault) {
530496 iommufd_auto_response_faults(hwpt, handle);
531497 iommufd_fault_iopf_disable(idev);
@@ -534,13 +500,19 @@ static void iommufd_hwpt_detach_device(struct iommufd_hw_pagetable *hwpt, · 2 unmodified lines
534500 }
535501
536502 static int iommufd_hwpt_replace_device(struct iommufd_device *idev,
503+ ioasid_t pasid,
537504 struct iommufd_hw_pagetable *hwpt,
538505 struct iommufd_hw_pagetable *old)
539506 {
540− struct iommufd_attach_handle *handle, *old_handle =
541− iommufd_device_get_attach_handle(idev);
507+ struct iommufd_attach_handle *handle, *old_handle;
542508 int rc;
543509
510+ rc = iommufd_hwpt_pasid_compat(hwpt, idev, pasid);
511+ if (rc)
512+ return rc;
513+
514+ old_handle = iommufd_device_get_attach_handle(idev, pasid);
515+
544516 handle = kzalloc(sizeof(*handle), GFP_KERNEL);
545517 if (!handle)
546518 return -ENOMEM;
@@ -552,8 +524,12 @@ static int iommufd_hwpt_replace_device(struct iommufd_device *idev, · 5 unmodified lines
552524 }
553525
554526 handle->idev = idev;
555− rc = iommu_replace_group_handle(idev->igroup->group, hwpt->domain,
556− &handle->handle);
527+ if (pasid == IOMMU_NO_PASID)
528+ rc = iommu_replace_group_handle(idev->igroup->group,
529+ hwpt->domain, &handle->handle);
530+ else
531+ rc = iommu_replace_device_pasid(hwpt->domain, idev->dev,
532+ pasid, &handle->handle);
557533 if (rc)
558534 goto out_disable_iopf;
559535
@@ -575,22 +551,51 @@ out_free_handle: · 15 unmodified lines
575551 }
576552
577553 int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,
578− struct iommufd_device *idev)
554+ struct iommufd_device *idev, ioasid_t pasid)
579555 {
580556 struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt);
557+ bool attach_resv = hwpt_paging && pasid == IOMMU_NO_PASID;
558+ struct iommufd_group *igroup = idev->igroup;
559+ struct iommufd_hw_pagetable *old_hwpt;
560+ struct iommufd_attach *attach;
581561 int rc;
582562
583− mutex_lock(&idev->igroup->lock);
563+ mutex_lock(&igroup->lock);
584564
585− if (idev->igroup->hwpt != NULL && idev->igroup->hwpt != hwpt) {
586− rc = -EINVAL;
565+ attach = xa_cmpxchg(&igroup->pasid_attach, pasid, NULL,
566+ XA_ZERO_ENTRY, GFP_KERNEL);
567+ if (xa_is_err(attach)) {
568+ rc = xa_err(attach);
587569 goto err_unlock;
588570 }
589571
590− if (hwpt_paging) {
572+ if (!attach) {
573+ attach = kzalloc(sizeof(*attach), GFP_KERNEL);
574+ if (!attach) {
575+ rc = -ENOMEM;
576+ goto err_release_pasid;
577+ }
578+ xa_init(&attach->device_array);
579+ }
580+
581+ old_hwpt = attach->hwpt;
582+
583+ rc = xa_insert(&attach->device_array, idev->obj.id, XA_ZERO_ENTRY,
584+ GFP_KERNEL);
585+ if (rc) {
586+ WARN_ON(rc == -EBUSY && !old_hwpt);
587+ goto err_free_attach;
588+ }
589+
590+ if (old_hwpt && old_hwpt != hwpt) {
591+ rc = -EINVAL;
592+ goto err_release_devid;
593+ }
594+
595+ if (attach_resv) {
591596 rc = iommufd_device_attach_reserved_iova(idev, hwpt_paging);
592597 if (rc)
593− goto err_unlock;
598+ goto err_release_devid;
594599 }
595600
596601 /*
@@ -600,51 +605,74 @@ int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt, · 3 unmodified lines
600605 * reserved regions are only updated during individual device
601606 * attachment.
602607 */
603− if (list_empty(&idev->igroup->device_list)) {
604− rc = iommufd_hwpt_attach_device(hwpt, idev);
608+ if (iommufd_group_first_attach(igroup, pasid)) {
609+ rc = iommufd_hwpt_attach_device(hwpt, idev, pasid);
605610 if (rc)
606611 goto err_unresv;
607− idev->igroup->hwpt = hwpt;
612+ attach->hwpt = hwpt;
613+ WARN_ON(xa_is_err(xa_store(&igroup->pasid_attach, pasid, attach,
614+ GFP_KERNEL)));
608615 }
609616 refcount_inc(&hwpt->obj.users);
610− list_add_tail(&idev->group_item, &idev->igroup->device_list);
611− mutex_unlock(&idev->igroup->lock);
617+ WARN_ON(xa_is_err(xa_store(&attach->device_array, idev->obj.id,
618+ idev, GFP_KERNEL)));
619+ mutex_unlock(&igroup->lock);
612620 return 0;
613621 err_unresv:
614− if (hwpt_paging)
622+ if (attach_resv)
615623 iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, idev->dev);
624+err_release_devid:
625+ xa_release(&attach->device_array, idev->obj.id);
626+err_free_attach:
627+ if (iommufd_group_first_attach(igroup, pasid))
628+ kfree(attach);
629+err_release_pasid:
630+ if (iommufd_group_first_attach(igroup, pasid))
631+ xa_release(&igroup->pasid_attach, pasid);
616632 err_unlock:
617− mutex_unlock(&idev->igroup->lock);
633+ mutex_unlock(&igroup->lock);
618634 return rc;
619635 }
620636
621637 struct iommufd_hw_pagetable *
622−iommufd_hw_pagetable_detach(struct iommufd_device *idev)
638+iommufd_hw_pagetable_detach(struct iommufd_device *idev, ioasid_t pasid)
623639 {
624− struct iommufd_hw_pagetable *hwpt = idev->igroup->hwpt;
625− struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt);
640+ struct iommufd_group *igroup = idev->igroup;
641+ struct iommufd_hwpt_paging *hwpt_paging;
642+ struct iommufd_hw_pagetable *hwpt;
643+ struct iommufd_attach *attach;
644+
645+ mutex_lock(&igroup->lock);
646+ attach = xa_load(&igroup->pasid_attach, pasid);
647+ if (!attach) {
648+ mutex_unlock(&igroup->lock);
649+ return NULL;
650+ }
626651
627− mutex_lock(&idev->igroup->lock);
628− list_del(&idev->group_item);
629− if (list_empty(&idev->igroup->device_list)) {
630− iommufd_hwpt_detach_device(hwpt, idev);
631− idev->igroup->hwpt = NULL;
652+ hwpt = attach->hwpt;
653+ hwpt_paging = find_hwpt_paging(hwpt);
654+
655+ xa_erase(&attach->device_array, idev->obj.id);
656+ if (xa_empty(&attach->device_array)) {
657+ iommufd_hwpt_detach_device(hwpt, idev, pasid);
658+ xa_erase(&igroup->pasid_attach, pasid);
659+ kfree(attach);
632660 }
633− if (hwpt_paging)
661+ if (hwpt_paging && pasid == IOMMU_NO_PASID)
634662 iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, idev->dev);
635− mutex_unlock(&idev->igroup->lock);
663+ mutex_unlock(&igroup->lock);
636664
637665 /* Caller must destroy hwpt */
638666 return hwpt;
639667 }
640668
641669 static struct iommufd_hw_pagetable *
642−iommufd_device_do_attach(struct iommufd_device *idev,
670+iommufd_device_do_attach(struct iommufd_device *idev, ioasid_t pasid,
643671 struct iommufd_hw_pagetable *hwpt)
644672 {
645673 int rc;
646674
647− rc = iommufd_hw_pagetable_attach(hwpt, idev);
675+ rc = iommufd_hw_pagetable_attach(hwpt, idev, pasid);
648676 if (rc)
649677 return ERR_PTR(rc);
650678 return NULL;
@@ -654,11 +682,14 @@ static void · 3 unmodified lines
654682 iommufd_group_remove_reserved_iova(struct iommufd_group *igroup,
655683 struct iommufd_hwpt_paging *hwpt_paging)
656684 {
685+ struct iommufd_attach *attach;
657686 struct iommufd_device *cur;
687+ unsigned long index;
658688
659689 lockdep_assert_held(&igroup->lock);
660690
661− list_for_each_entry(cur, &igroup->device_list, group_item)
691+ attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID);
692+ xa_for_each(&attach->device_array, index, cur)
662693 iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, cur->dev);
663694 }
664695
@@ -667,14 +698,17 @@ iommufd_group_do_replace_reserved_iova(struct iommufd_group *igroup, · 2 unmodified lines
667698 struct iommufd_hwpt_paging *hwpt_paging)
668699 {
669700 struct iommufd_hwpt_paging *old_hwpt_paging;
701+ struct iommufd_attach *attach;
670702 struct iommufd_device *cur;
703+ unsigned long index;
671704 int rc;
672705
673706 lockdep_assert_held(&igroup->lock);
674707
675− old_hwpt_paging = find_hwpt_paging(igroup->hwpt);
708+ attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID);
709+ old_hwpt_paging = find_hwpt_paging(attach->hwpt);
676710 if (!old_hwpt_paging || hwpt_paging->ioas != old_hwpt_paging->ioas) {
677− list_for_each_entry(cur, &igroup->device_list, group_item) {
711+ xa_for_each(&attach->device_array, index, cur) {
678712 rc = iopt_table_enforce_dev_resv_regions(
679713 &hwpt_paging->ioas->iopt, cur->dev, NULL);
680714 if (rc)
@@ -693,69 +727,81 @@ err_unresv: · 12 unmodified lines
693727 }
694728
695729 static struct iommufd_hw_pagetable *
696−iommufd_device_do_replace(struct iommufd_device *idev,
730+iommufd_device_do_replace(struct iommufd_device *idev, ioasid_t pasid,
697731 struct iommufd_hw_pagetable *hwpt)
698732 {
699733 struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt);
734+ bool attach_resv = hwpt_paging && pasid == IOMMU_NO_PASID;
700735 struct iommufd_hwpt_paging *old_hwpt_paging;
701736 struct iommufd_group *igroup = idev->igroup;
702737 struct iommufd_hw_pagetable *old_hwpt;
738+ struct iommufd_attach *attach;
703739 unsigned int num_devices;
704740 int rc;
705741
706− mutex_lock(&idev->igroup->lock);
742+ mutex_lock(&igroup->lock);
743+
744+ attach = xa_load(&igroup->pasid_attach, pasid);
745+ if (!attach) {
746+ rc = -EINVAL;
747+ goto err_unlock;
748+ }
749+
750+ old_hwpt = attach->hwpt;
707751
708− if (igroup->hwpt == NULL) {
752+ WARN_ON(!old_hwpt || xa_empty(&attach->device_array));
753+
754+ if (!iommufd_device_is_attached(idev, pasid)) {
709755 rc = -EINVAL;
710756 goto err_unlock;
711757 }
712758
713− if (hwpt == igroup->hwpt) {
714− mutex_unlock(&idev->igroup->lock);
759+ if (hwpt == old_hwpt) {
760+ mutex_unlock(&igroup->lock);
715761 return NULL;
716762 }
717763
718− old_hwpt = igroup->hwpt;
719− if (hwpt_paging) {
764+ if (attach_resv) {
720765 rc = iommufd_group_do_replace_reserved_iova(igroup, hwpt_paging);
721766 if (rc)
722767 goto err_unlock;
723768 }
724769
725− rc = iommufd_hwpt_replace_device(idev, hwpt, old_hwpt);
770+ rc = iommufd_hwpt_replace_device(idev, pasid, hwpt, old_hwpt);
726771 if (rc)
727772 goto err_unresv;
728773
729774 old_hwpt_paging = find_hwpt_paging(old_hwpt);
730− if (old_hwpt_paging &&
775+ if (old_hwpt_paging && pasid == IOMMU_NO_PASID &&
731776 (!hwpt_paging || hwpt_paging->ioas != old_hwpt_paging->ioas))
732777 iommufd_group_remove_reserved_iova(igroup, old_hwpt_paging);
733778
734− igroup->hwpt = hwpt;
779+ attach->hwpt = hwpt;
735780
736− num_devices = list_count_nodes(&igroup->device_list);
781+ num_devices = iommufd_group_device_num(igroup, pasid);
737782 /*
738− * Move the refcounts held by the device_list to the new hwpt. Retain a
783+ * Move the refcounts held by the device_array to the new hwpt. Retain a
739784 * refcount for this thread as the caller will free it.
740785 */
741786 refcount_add(num_devices, &hwpt->obj.users);
742787 if (num_devices > 1)
743788 WARN_ON(refcount_sub_and_test(num_devices - 1,
744789 &old_hwpt->obj.users));
745− mutex_unlock(&idev->igroup->lock);
790+ mutex_unlock(&igroup->lock);
746791
747792 /* Caller must destroy old_hwpt */
748793 return old_hwpt;
749794 err_unresv:
750− if (hwpt_paging)
795+ if (attach_resv)
751796 iommufd_group_remove_reserved_iova(igroup, hwpt_paging);
752797 err_unlock:
753− mutex_unlock(&idev->igroup->lock);
798+ mutex_unlock(&igroup->lock);
754799 return ERR_PTR(rc);
755800 }
756801
757802 typedef struct iommufd_hw_pagetable *(*attach_fn)(
758− struct iommufd_device *idev, struct iommufd_hw_pagetable *hwpt);
803+ struct iommufd_device *idev, ioasid_t pasid,
804+ struct iommufd_hw_pagetable *hwpt);
759805
760806 /*
761807 * When automatically managing the domains we search for a compatible domain in
@@ -763,7 +809,7 @@ typedef struct iommufd_hw_pagetable *(*attach_fn)( · 1 unmodified line
763809 * Automatic domain selection will never pick a manually created domain.
764810 */
765811 static struct iommufd_hw_pagetable *
766−iommufd_device_auto_get_domain(struct iommufd_device *idev,
812+iommufd_device_auto_get_domain(struct iommufd_device *idev, ioasid_t pasid,
767813 struct iommufd_ioas *ioas, u32 *pt_id,
768814 attach_fn do_attach)
769815 {
@@ -792,7 +838,7 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, · 22 unmodified lines
792838 hwpt = &hwpt_paging->common;
793839 if (!iommufd_lock_obj(&hwpt->obj))
794840 continue;
795− destroy_hwpt = (*do_attach)(idev, hwpt);
841+ destroy_hwpt = (*do_attach)(idev, pasid, hwpt);
796842 if (IS_ERR(destroy_hwpt)) {
797843 iommufd_put_object(idev->ictx, &hwpt->obj);
798844 /*
@@ -810,8 +856,8 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, · 11 unmodified lines
810856 goto out_unlock;
811857 }
812858
813− hwpt_paging = iommufd_hwpt_paging_alloc(idev->ictx, ioas, idev, 0,
814− immediate_attach, NULL);
859+ hwpt_paging = iommufd_hwpt_paging_alloc(idev->ictx, ioas, idev, pasid,
860+ 0, immediate_attach, NULL);
815861 if (IS_ERR(hwpt_paging)) {
816862 destroy_hwpt = ERR_CAST(hwpt_paging);
817863 goto out_unlock;
@@ -819,7 +865,7 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, · 1 unmodified line
819865 hwpt = &hwpt_paging->common;
820866
821867 if (!immediate_attach) {
822− destroy_hwpt = (*do_attach)(idev, hwpt);
868+ destroy_hwpt = (*do_attach)(idev, pasid, hwpt);
823869 if (IS_ERR(destroy_hwpt))
824870 goto out_abort;
825871 } else {
@@ -840,8 +886,9 @@ out_unlock: · 14 unmodified lines
840886 return destroy_hwpt;
841887 }
842888
843−static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id,
844− attach_fn do_attach)
889+static int iommufd_device_change_pt(struct iommufd_device *idev,
890+ ioasid_t pasid,
891+ u32 *pt_id, attach_fn do_attach)
845892 {
846893 struct iommufd_hw_pagetable *destroy_hwpt;
847894 struct iommufd_object *pt_obj;
@@ -856,7 +903,7 @@ static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id, · 8 unmodified lines
856903 struct iommufd_hw_pagetable *hwpt =
857904 container_of(pt_obj, struct iommufd_hw_pagetable, obj);
858905
859− destroy_hwpt = (*do_attach)(idev, hwpt);
906+ destroy_hwpt = (*do_attach)(idev, pasid, hwpt);
860907 if (IS_ERR(destroy_hwpt))
861908 goto out_put_pt_obj;
862909 break;
@@ -865,8 +912,8 @@ static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id, · 2 unmodified lines
865912 struct iommufd_ioas *ioas =
866913 container_of(pt_obj, struct iommufd_ioas, obj);
867914
868− destroy_hwpt = iommufd_device_auto_get_domain(idev, ioas, pt_id,
869− do_attach);
915+ destroy_hwpt = iommufd_device_auto_get_domain(idev, pasid, ioas,
916+ pt_id, do_attach);
870917 if (IS_ERR(destroy_hwpt))
871918 goto out_put_pt_obj;
872919 break;
@@ -888,22 +935,26 @@ out_put_pt_obj: · 15 unmodified lines
888935 }
889936
890937 /**
891− * iommufd_device_attach - Connect a device to an iommu_domain
938+ * iommufd_device_attach - Connect a device/pasid to an iommu_domain
892939 * @idev: device to attach
940+ * @pasid: pasid to attach
893941 * @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING
894942 * Output the IOMMUFD_OBJ_HWPT_PAGING ID
895943 *
896− * This connects the device to an iommu_domain, either automatically or manually
897− * selected. Once this completes the device could do DMA.
944+ * This connects the device/pasid to an iommu_domain, either automatically
945+ * or manually selected. Once this completes the device could do DMA with
946+ * @pasid. @pasid is IOMMU_NO_PASID if this attach is for no pasid usage.
898947 *
899948 * The caller should return the resulting pt_id back to userspace.
900949 * This function is undone by calling iommufd_device_detach().
901950 */
902−int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id)
951+int iommufd_device_attach(struct iommufd_device *idev, ioasid_t pasid,
952+ u32 *pt_id)
903953 {
904954 int rc;
905955
906− rc = iommufd_device_change_pt(idev, pt_id, &iommufd_device_do_attach);
956+ rc = iommufd_device_change_pt(idev, pasid, pt_id,
957+ &iommufd_device_do_attach);
907958 if (rc)
908959 return rc;
909960
@@ -917,8 +968,9 @@ int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id) · 7 unmodified lines
917968 EXPORT_SYMBOL_NS_GPL(iommufd_device_attach, "IOMMUFD");
918969
919970 /**
920− * iommufd_device_replace - Change the device's iommu_domain
971+ * iommufd_device_replace - Change the device/pasid's iommu_domain
921972 * @idev: device to change
973+ * @pasid: pasid to change
922974 * @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING
923975 * Output the IOMMUFD_OBJ_HWPT_PAGING ID
924976 *
@@ -929,27 +981,33 @@ EXPORT_SYMBOL_NS_GPL(iommufd_device_attach, "IOMMUFD"); · 4 unmodified lines
929981 *
930982 * If it fails then no change is made to the attachment. The iommu driver may
931983 * implement this so there is no disruption in translation. This can only be
932− * called if iommufd_device_attach() has already succeeded.
984+ * called if iommufd_device_attach() has already succeeded. @pasid is
985+ * IOMMU_NO_PASID for no pasid usage.
933986 */
934−int iommufd_device_replace(struct iommufd_device *idev, u32 *pt_id)
987+int iommufd_device_replace(struct iommufd_device *idev, ioasid_t pasid,
988+ u32 *pt_id)
935989 {
936− return iommufd_device_change_pt(idev, pt_id,
990+ return iommufd_device_change_pt(idev, pasid, pt_id,
937991 &iommufd_device_do_replace);
938992 }
939993 EXPORT_SYMBOL_NS_GPL(iommufd_device_replace, "IOMMUFD");
940994
941995 /**
942− * iommufd_device_detach - Disconnect a device to an iommu_domain
996+ * iommufd_device_detach - Disconnect a device/device to an iommu_domain
943997 * @idev: device to detach
998+ * @pasid: pasid to detach
944999 *
9451000 * Undo iommufd_device_attach(). This disconnects the idev from the previously
9461001 * attached pt_id. The device returns back to a blocked DMA translation.
1002+ * @pasid is IOMMU_NO_PASID for no pasid usage.
9471003 */
948−void iommufd_device_detach(struct iommufd_device *idev)
1004+void iommufd_device_detach(struct iommufd_device *idev, ioasid_t pasid)
9491005 {
9501006 struct iommufd_hw_pagetable *hwpt;
9511007
952− hwpt = iommufd_hw_pagetable_detach(idev);
1008+ hwpt = iommufd_hw_pagetable_detach(idev, pasid);
1009+ if (!hwpt)
1010+ return;
9531011 iommufd_hw_pagetable_put(idev->ictx, hwpt);
9541012 refcount_dec(&idev->obj.users);
9551013 }
@@ -1349,7 +1407,7 @@ int iommufd_access_rw(struct iommufd_access *access, unsigned long iova, · 393 unmodified lines
13491407 struct io_pagetable *iopt;
13501408 struct iopt_area *area;
13511409 unsigned long last_iova;
1352− int rc;
1410+ int rc = -EINVAL;
13531411
13541412 if (!length)
13551413 return -EINVAL;
@@ -1405,7 +1463,8 @@ int iommufd_get_hw_info(struct iommufd_ucmd *ucmd) · 49 unmodified lines
14051463 void *data;
14061464 int rc;
14071465
1408− if (cmd->flags || cmd->__reserved)
1466+ if (cmd->flags || cmd->__reserved[0] || cmd->__reserved[1] ||
1467+ cmd->__reserved[2])
14091468 return -EOPNOTSUPP;
14101469
14111470 idev = iommufd_get_device(ucmd, cmd->dev_id);
@@ -1462,6 +1521,36 @@ int iommufd_get_hw_info(struct iommufd_ucmd *ucmd) · 50 unmodified lines
14621521 if (device_iommu_capable(idev->dev, IOMMU_CAP_DIRTY_TRACKING))
14631522 cmd->out_capabilities |= IOMMU_HW_CAP_DIRTY_TRACKING;
14641523
1524+ cmd->out_max_pasid_log2 = 0;
1525+ /*
1526+ * Currently, all iommu drivers enable PASID in the probe_device()
1527+ * op if iommu and device supports it. So the max_pasids stored in
1528+ * dev->iommu indicates both PASID support and enable status. A
1529+ * non-zero dev->iommu->max_pasids means PASID is supported and
1530+ * enabled. The iommufd only reports PASID capability to userspace
1531+ * if it's enabled.
1532+ */
1533+ if (idev->dev->iommu->max_pasids) {
1534+ cmd->out_max_pasid_log2 = ilog2(idev->dev->iommu->max_pasids);
1535+
1536+ if (dev_is_pci(idev->dev)) {
1537+ struct pci_dev *pdev = to_pci_dev(idev->dev);
1538+ int ctrl;
1539+
1540+ ctrl = pci_pasid_status(pdev);
1541+
1542+ WARN_ON_ONCE(ctrl < 0 ||
1543+ !(ctrl & PCI_PASID_CTRL_ENABLE));
1544+
1545+ if (ctrl & PCI_PASID_CTRL_EXEC)
1546+ cmd->out_capabilities |=
1547+ IOMMU_HW_CAP_PCI_PASID_EXEC;
1548+ if (ctrl & PCI_PASID_CTRL_PRIV)
1549+ cmd->out_capabilities |=
1550+ IOMMU_HW_CAP_PCI_PASID_PRIV;
1551+ }
1552+ }
1553+
14651554 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
14661555 out_free:
14671556 kfree(data);
drivers/iommu/iommufd/driver.c+198−0
@@ -49,5 +49,203 @@ struct device *iommufd_viommu_find_dev(struct iommufd_viommu *viommu, · 48 unmodified lines
4949 }
5050 EXPORT_SYMBOL_NS_GPL(iommufd_viommu_find_dev, "IOMMUFD");
5151
52+/* Return -ENOENT if device is not associated to the vIOMMU */
53+int iommufd_viommu_get_vdev_id(struct iommufd_viommu *viommu,
54+ struct device *dev, unsigned long *vdev_id)
55+{
56+ struct iommufd_vdevice *vdev;
57+ unsigned long index;
58+ int rc = -ENOENT;
59+
60+ if (WARN_ON_ONCE(!vdev_id))
61+ return -EINVAL;
62+
63+ xa_lock(&viommu->vdevs);
64+ xa_for_each(&viommu->vdevs, index, vdev) {
65+ if (vdev->dev == dev) {
66+ *vdev_id = vdev->id;
67+ rc = 0;
68+ break;
69+ }
70+ }
71+ xa_unlock(&viommu->vdevs);
72+ return rc;
73+}
74+EXPORT_SYMBOL_NS_GPL(iommufd_viommu_get_vdev_id, "IOMMUFD");
75+
76+/*
77+ * Typically called in driver's threaded IRQ handler.
78+ * The @type and @event_data must be defined in include/uapi/linux/iommufd.h
79+ */
80+int iommufd_viommu_report_event(struct iommufd_viommu *viommu,
81+ enum iommu_veventq_type type, void *event_data,
82+ size_t data_len)
83+{
84+ struct iommufd_veventq *veventq;
85+ struct iommufd_vevent *vevent;
86+ int rc = 0;
87+
88+ if (WARN_ON_ONCE(!data_len || !event_data))
89+ return -EINVAL;
90+
91+ down_read(&viommu->veventqs_rwsem);
92+
93+ veventq = iommufd_viommu_find_veventq(viommu, type);
94+ if (!veventq) {
95+ rc = -EOPNOTSUPP;
96+ goto out_unlock_veventqs;
97+ }
98+
99+ spin_lock(&veventq->common.lock);
100+ if (veventq->num_events == veventq->depth) {
101+ vevent = &veventq->lost_events_header;
102+ goto out_set_header;
103+ }
104+
105+ vevent = kzalloc(struct_size(vevent, event_data, data_len), GFP_ATOMIC);
106+ if (!vevent) {
107+ rc = -ENOMEM;
108+ vevent = &veventq->lost_events_header;
109+ goto out_set_header;
110+ }
111+ memcpy(vevent->event_data, event_data, data_len);
112+ vevent->data_len = data_len;
113+ veventq->num_events++;
114+
115+out_set_header:
116+ iommufd_vevent_handler(veventq, vevent);
117+ spin_unlock(&veventq->common.lock);
118+out_unlock_veventqs:
119+ up_read(&viommu->veventqs_rwsem);
120+ return rc;
121+}
122+EXPORT_SYMBOL_NS_GPL(iommufd_viommu_report_event, "IOMMUFD");
123+
124+#ifdef CONFIG_IRQ_MSI_IOMMU
125+/*
126+ * Get a iommufd_sw_msi_map for the msi physical address requested by the irq
127+ * layer. The mapping to IOVA is global to the iommufd file descriptor, every
128+ * domain that is attached to a device using the same MSI parameters will use
129+ * the same IOVA.
130+ */
131+static struct iommufd_sw_msi_map *
132+iommufd_sw_msi_get_map(struct iommufd_ctx *ictx, phys_addr_t msi_addr,
133+ phys_addr_t sw_msi_start)
134+{
135+ struct iommufd_sw_msi_map *cur;
136+ unsigned int max_pgoff = 0;
137+
138+ lockdep_assert_held(&ictx->sw_msi_lock);
139+
140+ list_for_each_entry(cur, &ictx->sw_msi_list, sw_msi_item) {
141+ if (cur->sw_msi_start != sw_msi_start)
142+ continue;
143+ max_pgoff = max(max_pgoff, cur->pgoff + 1);
144+ if (cur->msi_addr == msi_addr)
145+ return cur;
146+ }
147+
148+ if (ictx->sw_msi_id >=
149+ BITS_PER_BYTE * sizeof_field(struct iommufd_sw_msi_maps, bitmap))
150+ return ERR_PTR(-EOVERFLOW);
151+
152+ cur = kzalloc(sizeof(*cur), GFP_KERNEL);
153+ if (!cur)
154+ return ERR_PTR(-ENOMEM);
155+
156+ cur->sw_msi_start = sw_msi_start;
157+ cur->msi_addr = msi_addr;
158+ cur->pgoff = max_pgoff;
159+ cur->id = ictx->sw_msi_id++;
160+ list_add_tail(&cur->sw_msi_item, &ictx->sw_msi_list);
161+ return cur;
162+}
163+
164+int iommufd_sw_msi_install(struct iommufd_ctx *ictx,
165+ struct iommufd_hwpt_paging *hwpt_paging,
166+ struct iommufd_sw_msi_map *msi_map)
167+{
168+ unsigned long iova;
169+
170+ lockdep_assert_held(&ictx->sw_msi_lock);
171+
172+ iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE;
173+ if (!test_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap)) {
174+ int rc;
175+
176+ rc = iommu_map(hwpt_paging->common.domain, iova,
177+ msi_map->msi_addr, PAGE_SIZE,
178+ IOMMU_WRITE | IOMMU_READ | IOMMU_MMIO,
179+ GFP_KERNEL_ACCOUNT);
180+ if (rc)
181+ return rc;
182+ __set_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap);
183+ }
184+ return 0;
185+}
186+EXPORT_SYMBOL_NS_GPL(iommufd_sw_msi_install, "IOMMUFD_INTERNAL");
187+
188+/*
189+ * Called by the irq code if the platform translates the MSI address through the
190+ * IOMMU. msi_addr is the physical address of the MSI page. iommufd will
191+ * allocate a fd global iova for the physical page that is the same on all
192+ * domains and devices.
193+ */
194+int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
195+ phys_addr_t msi_addr)
196+{
197+ struct device *dev = msi_desc_to_dev(desc);
198+ struct iommufd_hwpt_paging *hwpt_paging;
199+ struct iommu_attach_handle *raw_handle;
200+ struct iommufd_attach_handle *handle;
201+ struct iommufd_sw_msi_map *msi_map;
202+ struct iommufd_ctx *ictx;
203+ unsigned long iova;
204+ int rc;
205+
206+ /*
207+ * It is safe to call iommu_attach_handle_get() here because the iommu
208+ * core code invokes this under the group mutex which also prevents any
209+ * change of the attach handle for the duration of this function.
210+ */
211+ iommu_group_mutex_assert(dev);
212+
213+ raw_handle =
214+ iommu_attach_handle_get(dev->iommu_group, IOMMU_NO_PASID, 0);
215+ if (IS_ERR(raw_handle))
216+ return 0;
217+ hwpt_paging = find_hwpt_paging(domain->iommufd_hwpt);
218+
219+ handle = to_iommufd_handle(raw_handle);
220+ /* No IOMMU_RESV_SW_MSI means no change to the msi_msg */
221+ if (handle->idev->igroup->sw_msi_start == PHYS_ADDR_MAX)
222+ return 0;
223+
224+ ictx = handle->idev->ictx;
225+ guard(mutex)(&ictx->sw_msi_lock);
226+ /*
227+ * The input msi_addr is the exact byte offset of the MSI doorbell, we
228+ * assume the caller has checked that it is contained with a MMIO region
229+ * that is secure to map at PAGE_SIZE.
230+ */
231+ msi_map = iommufd_sw_msi_get_map(handle->idev->ictx,
232+ msi_addr & PAGE_MASK,
233+ handle->idev->igroup->sw_msi_start);
234+ if (IS_ERR(msi_map))
235+ return PTR_ERR(msi_map);
236+
237+ rc = iommufd_sw_msi_install(ictx, hwpt_paging, msi_map);
238+ if (rc)
239+ return rc;
240+ __set_bit(msi_map->id, handle->idev->igroup->required_sw_msi.bitmap);
241+
242+ iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE;
243+ msi_desc_set_iommu_msi_iova(desc, iova, PAGE_SHIFT);
244+ return 0;
245+}
246+EXPORT_SYMBOL_NS_GPL(iommufd_sw_msi, "IOMMUFD");
247+#endif
248+
52249 MODULE_DESCRIPTION("iommufd code shared with builtin modules");
250+MODULE_IMPORT_NS("IOMMUFD_INTERNAL");
53251 MODULE_LICENSE("GPL");
drivers/iommu/iommufd/eventq.c+598−0
@@ -0,0 +1,598 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/* Copyright (C) 2024 Intel Corporation
3+ */
4+#define pr_fmt(fmt) "iommufd: " fmt
5+
6+#include <linux/anon_inodes.h>
7+#include <linux/file.h>
8+#include <linux/fs.h>
9+#include <linux/iommufd.h>
10+#include <linux/module.h>
11+#include <linux/mutex.h>
12+#include <linux/pci.h>
13+#include <linux/pci-ats.h>
14+#include <linux/poll.h>
15+#include <uapi/linux/iommufd.h>
16+
17+#include "../iommu-priv.h"
18+#include "iommufd_private.h"
19+
20+/* IOMMUFD_OBJ_FAULT Functions */
21+
22+int iommufd_fault_iopf_enable(struct iommufd_device *idev)
23+{
24+ struct device *dev = idev->dev;
25+ int ret;
26+
27+ /*
28+ * Once we turn on PCI/PRI support for VF, the response failure code
29+ * should not be forwarded to the hardware due to PRI being a shared
30+ * resource between PF and VFs. There is no coordination for this
31+ * shared capability. This waits for a vPRI reset to recover.
32+ */
33+ if (dev_is_pci(dev)) {
34+ struct pci_dev *pdev = to_pci_dev(dev);
35+
36+ if (pdev->is_virtfn && pci_pri_supported(pdev))
37+ return -EINVAL;
38+ }
39+
40+ mutex_lock(&idev->iopf_lock);
41+ /* Device iopf has already been on. */
42+ if (++idev->iopf_enabled > 1) {
43+ mutex_unlock(&idev->iopf_lock);
44+ return 0;
45+ }
46+
47+ ret = iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_IOPF);
48+ if (ret)
49+ --idev->iopf_enabled;
50+ mutex_unlock(&idev->iopf_lock);
51+
52+ return ret;
53+}
54+
55+void iommufd_fault_iopf_disable(struct iommufd_device *idev)
56+{
57+ mutex_lock(&idev->iopf_lock);
58+ if (!WARN_ON(idev->iopf_enabled == 0)) {
59+ if (--idev->iopf_enabled == 0)
60+ iommu_dev_disable_feature(idev->dev, IOMMU_DEV_FEAT_IOPF);
61+ }
62+ mutex_unlock(&idev->iopf_lock);
63+}
64+
65+void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt,
66+ struct iommufd_attach_handle *handle)
67+{
68+ struct iommufd_fault *fault = hwpt->fault;
69+ struct iopf_group *group, *next;
70+ struct list_head free_list;
71+ unsigned long index;
72+
73+ if (!fault)
74+ return;
75+ INIT_LIST_HEAD(&free_list);
76+
77+ mutex_lock(&fault->mutex);
78+ spin_lock(&fault->common.lock);
79+ list_for_each_entry_safe(group, next, &fault->common.deliver, node) {
80+ if (group->attach_handle != &handle->handle)
81+ continue;
82+ list_move(&group->node, &free_list);
83+ }
84+ spin_unlock(&fault->common.lock);
85+
86+ list_for_each_entry_safe(group, next, &free_list, node) {
87+ list_del(&group->node);
88+ iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
89+ iopf_free_group(group);
90+ }
91+
92+ xa_for_each(&fault->response, index, group) {
93+ if (group->attach_handle != &handle->handle)
94+ continue;
95+ xa_erase(&fault->response, index);
96+ iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
97+ iopf_free_group(group);
98+ }
99+ mutex_unlock(&fault->mutex);
100+}
101+
102+void iommufd_fault_destroy(struct iommufd_object *obj)
103+{
104+ struct iommufd_eventq *eventq =
105+ container_of(obj, struct iommufd_eventq, obj);
106+ struct iommufd_fault *fault = eventq_to_fault(eventq);
107+ struct iopf_group *group, *next;
108+ unsigned long index;
109+
110+ /*
111+ * The iommufd object's reference count is zero at this point.
112+ * We can be confident that no other threads are currently
113+ * accessing this pointer. Therefore, acquiring the mutex here
114+ * is unnecessary.
115+ */
116+ list_for_each_entry_safe(group, next, &fault->common.deliver, node) {
117+ list_del(&group->node);
118+ iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
119+ iopf_free_group(group);
120+ }
121+ xa_for_each(&fault->response, index, group) {
122+ xa_erase(&fault->response, index);
123+ iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
124+ iopf_free_group(group);
125+ }
126+ xa_destroy(&fault->response);
127+ mutex_destroy(&fault->mutex);
128+}
129+
130+static void iommufd_compose_fault_message(struct iommu_fault *fault,
131+ struct iommu_hwpt_pgfault *hwpt_fault,
132+ struct iommufd_device *idev,
133+ u32 cookie)
134+{
135+ hwpt_fault->flags = fault->prm.flags;
136+ hwpt_fault->dev_id = idev->obj.id;
137+ hwpt_fault->pasid = fault->prm.pasid;
138+ hwpt_fault->grpid = fault->prm.grpid;
139+ hwpt_fault->perm = fault->prm.perm;
140+ hwpt_fault->addr = fault->prm.addr;
141+ hwpt_fault->length = 0;
142+ hwpt_fault->cookie = cookie;
143+}
144+
145+/* Fetch the first node out of the fault->deliver list */
146+static struct iopf_group *
147+iommufd_fault_deliver_fetch(struct iommufd_fault *fault)
148+{
149+ struct list_head *list = &fault->common.deliver;
150+ struct iopf_group *group = NULL;
151+
152+ spin_lock(&fault->common.lock);
153+ if (!list_empty(list)) {
154+ group = list_first_entry(list, struct iopf_group, node);
155+ list_del(&group->node);
156+ }
157+ spin_unlock(&fault->common.lock);
158+ return group;
159+}
160+
161+/* Restore a node back to the head of the fault->deliver list */
162+static void iommufd_fault_deliver_restore(struct iommufd_fault *fault,
163+ struct iopf_group *group)
164+{
165+ spin_lock(&fault->common.lock);
166+ list_add(&group->node, &fault->common.deliver);
167+ spin_unlock(&fault->common.lock);
168+}
169+
170+static ssize_t iommufd_fault_fops_read(struct file *filep, char __user *buf,
171+ size_t count, loff_t *ppos)
172+{
173+ size_t fault_size = sizeof(struct iommu_hwpt_pgfault);
174+ struct iommufd_eventq *eventq = filep->private_data;
175+ struct iommufd_fault *fault = eventq_to_fault(eventq);
176+ struct iommu_hwpt_pgfault data = {};
177+ struct iommufd_device *idev;
178+ struct iopf_group *group;
179+ struct iopf_fault *iopf;
180+ size_t done = 0;
181+ int rc = 0;
182+
183+ if (*ppos || count % fault_size)
184+ return -ESPIPE;
185+
186+ mutex_lock(&fault->mutex);
187+ while ((group = iommufd_fault_deliver_fetch(fault))) {
188+ if (done >= count ||
189+ group->fault_count * fault_size > count - done) {
190+ iommufd_fault_deliver_restore(fault, group);
191+ break;
192+ }
193+
194+ rc = xa_alloc(&fault->response, &group->cookie, group,
195+ xa_limit_32b, GFP_KERNEL);
196+ if (rc) {
197+ iommufd_fault_deliver_restore(fault, group);
198+ break;
199+ }
200+
201+ idev = to_iommufd_handle(group->attach_handle)->idev;
202+ list_for_each_entry(iopf, &group->faults, list) {
203+ iommufd_compose_fault_message(&iopf->fault,
204+ &data, idev,
205+ group->cookie);
206+ if (copy_to_user(buf + done, &data, fault_size)) {
207+ xa_erase(&fault->response, group->cookie);
208+ iommufd_fault_deliver_restore(fault, group);
209+ rc = -EFAULT;
210+ break;
211+ }
212+ done += fault_size;
213+ }
214+ }
215+ mutex_unlock(&fault->mutex);
216+
217+ return done == 0 ? rc : done;
218+}
219+
220+static ssize_t iommufd_fault_fops_write(struct file *filep, const char __user *buf,
221+ size_t count, loff_t *ppos)
222+{
223+ size_t response_size = sizeof(struct iommu_hwpt_page_response);
224+ struct iommufd_eventq *eventq = filep->private_data;
225+ struct iommufd_fault *fault = eventq_to_fault(eventq);
226+ struct iommu_hwpt_page_response response;
227+ struct iopf_group *group;
228+ size_t done = 0;
229+ int rc = 0;
230+
231+ if (*ppos || count % response_size)
232+ return -ESPIPE;
233+
234+ mutex_lock(&fault->mutex);
235+ while (count > done) {
236+ rc = copy_from_user(&response, buf + done, response_size);
237+ if (rc)
238+ break;
239+
240+ static_assert((int)IOMMUFD_PAGE_RESP_SUCCESS ==
241+ (int)IOMMU_PAGE_RESP_SUCCESS);
242+ static_assert((int)IOMMUFD_PAGE_RESP_INVALID ==
243+ (int)IOMMU_PAGE_RESP_INVALID);
244+ if (response.code != IOMMUFD_PAGE_RESP_SUCCESS &&
245+ response.code != IOMMUFD_PAGE_RESP_INVALID) {
246+ rc = -EINVAL;
247+ break;
248+ }
249+
250+ group = xa_erase(&fault->response, response.cookie);
251+ if (!group) {
252+ rc = -EINVAL;
253+ break;
254+ }
255+
256+ iopf_group_response(group, response.code);
257+ iopf_free_group(group);
258+ done += response_size;
259+ }
260+ mutex_unlock(&fault->mutex);
261+
262+ return done == 0 ? rc : done;
263+}
264+
265+/* IOMMUFD_OBJ_VEVENTQ Functions */
266+
267+void iommufd_veventq_abort(struct iommufd_object *obj)
268+{
269+ struct iommufd_eventq *eventq =
270+ container_of(obj, struct iommufd_eventq, obj);
271+ struct iommufd_veventq *veventq = eventq_to_veventq(eventq);
272+ struct iommufd_viommu *viommu = veventq->viommu;
273+ struct iommufd_vevent *cur, *next;
274+
275+ lockdep_assert_held_write(&viommu->veventqs_rwsem);
276+
277+ list_for_each_entry_safe(cur, next, &eventq->deliver, node) {
278+ list_del(&cur->node);
279+ if (cur != &veventq->lost_events_header)
280+ kfree(cur);
281+ }
282+
283+ refcount_dec(&viommu->obj.users);
284+ list_del(&veventq->node);
285+}
286+
287+void iommufd_veventq_destroy(struct iommufd_object *obj)
288+{
289+ struct iommufd_veventq *veventq = eventq_to_veventq(
290+ container_of(obj, struct iommufd_eventq, obj));
291+
292+ down_write(&veventq->viommu->veventqs_rwsem);
293+ iommufd_veventq_abort(obj);
294+ up_write(&veventq->viommu->veventqs_rwsem);
295+}
296+
297+static struct iommufd_vevent *
298+iommufd_veventq_deliver_fetch(struct iommufd_veventq *veventq)
299+{
300+ struct iommufd_eventq *eventq = &veventq->common;
301+ struct list_head *list = &eventq->deliver;
302+ struct iommufd_vevent *vevent = NULL;
303+
304+ spin_lock(&eventq->lock);
305+ if (!list_empty(list)) {
306+ struct iommufd_vevent *next;
307+
308+ next = list_first_entry(list, struct iommufd_vevent, node);
309+ /* Make a copy of the lost_events_header for copy_to_user */
310+ if (next == &veventq->lost_events_header) {
311+ vevent = kzalloc(sizeof(*vevent), GFP_ATOMIC);
312+ if (!vevent)
313+ goto out_unlock;
314+ }
315+ list_del(&next->node);
316+ if (vevent)
317+ memcpy(vevent, next, sizeof(*vevent));
318+ else
319+ vevent = next;
320+ }
321+out_unlock:
322+ spin_unlock(&eventq->lock);
323+ return vevent;
324+}
325+
326+static void iommufd_veventq_deliver_restore(struct iommufd_veventq *veventq,
327+ struct iommufd_vevent *vevent)
328+{
329+ struct iommufd_eventq *eventq = &veventq->common;
330+ struct list_head *list = &eventq->deliver;
331+
332+ spin_lock(&eventq->lock);
333+ if (vevent_for_lost_events_header(vevent)) {
334+ /* Remove the copy of the lost_events_header */
335+ kfree(vevent);
336+ vevent = NULL;
337+ /* An empty list needs the lost_events_header back */
338+ if (list_empty(list))
339+ vevent = &veventq->lost_events_header;
340+ }
341+ if (vevent)
342+ list_add(&vevent->node, list);
343+ spin_unlock(&eventq->lock);
344+}
345+
346+static ssize_t iommufd_veventq_fops_read(struct file *filep, char __user *buf,
347+ size_t count, loff_t *ppos)
348+{
349+ struct iommufd_eventq *eventq = filep->private_data;
350+ struct iommufd_veventq *veventq = eventq_to_veventq(eventq);
351+ struct iommufd_vevent_header *hdr;
352+ struct iommufd_vevent *cur;
353+ size_t done = 0;
354+ int rc = 0;
355+
356+ if (*ppos)
357+ return -ESPIPE;
358+
359+ while ((cur = iommufd_veventq_deliver_fetch(veventq))) {
360+ /* Validate the remaining bytes against the header size */
361+ if (done >= count || sizeof(*hdr) > count - done) {
362+ iommufd_veventq_deliver_restore(veventq, cur);
363+ break;
364+ }
365+ hdr = &cur->header;
366+
367+ /* If being a normal vEVENT, validate against the full size */
368+ if (!vevent_for_lost_events_header(cur) &&
369+ sizeof(hdr) + cur->data_len > count - done) {
370+ iommufd_veventq_deliver_restore(veventq, cur);
371+ break;
372+ }
373+
374+ if (copy_to_user(buf + done, hdr, sizeof(*hdr))) {
375+ iommufd_veventq_deliver_restore(veventq, cur);
376+ rc = -EFAULT;
377+ break;
378+ }
379+ done += sizeof(*hdr);
380+
381+ if (cur->data_len &&
382+ copy_to_user(buf + done, cur->event_data, cur->data_len)) {
383+ iommufd_veventq_deliver_restore(veventq, cur);
384+ rc = -EFAULT;
385+ break;
386+ }
387+ spin_lock(&eventq->lock);
388+ if (!vevent_for_lost_events_header(cur))
389+ veventq->num_events--;
390+ spin_unlock(&eventq->lock);
391+ done += cur->data_len;
392+ kfree(cur);
393+ }
394+
395+ return done == 0 ? rc : done;
396+}
397+
398+/* Common Event Queue Functions */
399+
400+static __poll_t iommufd_eventq_fops_poll(struct file *filep,
401+ struct poll_table_struct *wait)
402+{
403+ struct iommufd_eventq *eventq = filep->private_data;
404+ __poll_t pollflags = 0;
405+
406+ if (eventq->obj.type == IOMMUFD_OBJ_FAULT)
407+ pollflags |= EPOLLOUT;
408+
409+ poll_wait(filep, &eventq->wait_queue, wait);
410+ spin_lock(&eventq->lock);
411+ if (!list_empty(&eventq->deliver))
412+ pollflags |= EPOLLIN | EPOLLRDNORM;
413+ spin_unlock(&eventq->lock);
414+
415+ return pollflags;
416+}
417+
418+static int iommufd_eventq_fops_release(struct inode *inode, struct file *filep)
419+{
420+ struct iommufd_eventq *eventq = filep->private_data;
421+
422+ refcount_dec(&eventq->obj.users);
423+ iommufd_ctx_put(eventq->ictx);
424+ return 0;
425+}
426+
427+#define INIT_EVENTQ_FOPS(read_op, write_op) \
428+ ((const struct file_operations){ \
429+ .owner = THIS_MODULE, \
430+ .open = nonseekable_open, \
431+ .read = read_op, \
432+ .write = write_op, \
433+ .poll = iommufd_eventq_fops_poll, \
434+ .release = iommufd_eventq_fops_release, \
435+ })
436+
437+static int iommufd_eventq_init(struct iommufd_eventq *eventq, char *name,
438+ struct iommufd_ctx *ictx,
439+ const struct file_operations *fops)
440+{
441+ struct file *filep;
442+ int fdno;
443+
444+ spin_lock_init(&eventq->lock);
445+ INIT_LIST_HEAD(&eventq->deliver);
446+ init_waitqueue_head(&eventq->wait_queue);
447+
448+ filep = anon_inode_getfile(name, fops, eventq, O_RDWR);
449+ if (IS_ERR(filep))
450+ return PTR_ERR(filep);
451+
452+ eventq->ictx = ictx;
453+ iommufd_ctx_get(eventq->ictx);
454+ eventq->filep = filep;
455+ refcount_inc(&eventq->obj.users);
456+
457+ fdno = get_unused_fd_flags(O_CLOEXEC);
458+ if (fdno < 0)
459+ fput(filep);
460+ return fdno;
461+}
462+
463+static const struct file_operations iommufd_fault_fops =
464+ INIT_EVENTQ_FOPS(iommufd_fault_fops_read, iommufd_fault_fops_write);
465+
466+int iommufd_fault_alloc(struct iommufd_ucmd *ucmd)
467+{
468+ struct iommu_fault_alloc *cmd = ucmd->cmd;
469+ struct iommufd_fault *fault;
470+ int fdno;
471+ int rc;
472+
473+ if (cmd->flags)
474+ return -EOPNOTSUPP;
475+
476+ fault = __iommufd_object_alloc(ucmd->ictx, fault, IOMMUFD_OBJ_FAULT,
477+ common.obj);
478+ if (IS_ERR(fault))
479+ return PTR_ERR(fault);
480+
481+ xa_init_flags(&fault->response, XA_FLAGS_ALLOC1);
482+ mutex_init(&fault->mutex);
483+
484+ fdno = iommufd_eventq_init(&fault->common, "[iommufd-pgfault]",
485+ ucmd->ictx, &iommufd_fault_fops);
486+ if (fdno < 0) {
487+ rc = fdno;
488+ goto out_abort;
489+ }
490+
491+ cmd->out_fault_id = fault->common.obj.id;
492+ cmd->out_fault_fd = fdno;
493+
494+ rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
495+ if (rc)
496+ goto out_put_fdno;
497+ iommufd_object_finalize(ucmd->ictx, &fault->common.obj);
498+
499+ fd_install(fdno, fault->common.filep);
500+
501+ return 0;
502+out_put_fdno:
503+ put_unused_fd(fdno);
504+ fput(fault->common.filep);
505+out_abort:
506+ iommufd_object_abort_and_destroy(ucmd->ictx, &fault->common.obj);
507+
508+ return rc;
509+}
510+
511+int iommufd_fault_iopf_handler(struct iopf_group *group)
512+{
513+ struct iommufd_hw_pagetable *hwpt;
514+ struct iommufd_fault *fault;
515+
516+ hwpt = group->attach_handle->domain->iommufd_hwpt;
517+ fault = hwpt->fault;
518+
519+ spin_lock(&fault->common.lock);
520+ list_add_tail(&group->node, &fault->common.deliver);
521+ spin_unlock(&fault->common.lock);
522+
523+ wake_up_interruptible(&fault->common.wait_queue);
524+
525+ return 0;
526+}
527+
528+static const struct file_operations iommufd_veventq_fops =
529+ INIT_EVENTQ_FOPS(iommufd_veventq_fops_read, NULL);
530+
531+int iommufd_veventq_alloc(struct iommufd_ucmd *ucmd)
532+{
533+ struct iommu_veventq_alloc *cmd = ucmd->cmd;
534+ struct iommufd_veventq *veventq;
535+ struct iommufd_viommu *viommu;
536+ int fdno;
537+ int rc;
538+
539+ if (cmd->flags || cmd->__reserved ||
540+ cmd->type == IOMMU_VEVENTQ_TYPE_DEFAULT)
541+ return -EOPNOTSUPP;
542+ if (!cmd->veventq_depth)
543+ return -EINVAL;
544+
545+ viommu = iommufd_get_viommu(ucmd, cmd->viommu_id);
546+ if (IS_ERR(viommu))
547+ return PTR_ERR(viommu);
548+
549+ down_write(&viommu->veventqs_rwsem);
550+
551+ if (iommufd_viommu_find_veventq(viommu, cmd->type)) {
552+ rc = -EEXIST;
553+ goto out_unlock_veventqs;
554+ }
555+
556+ veventq = __iommufd_object_alloc(ucmd->ictx, veventq,
557+ IOMMUFD_OBJ_VEVENTQ, common.obj);
558+ if (IS_ERR(veventq)) {
559+ rc = PTR_ERR(veventq);
560+ goto out_unlock_veventqs;
561+ }
562+
563+ veventq->type = cmd->type;
564+ veventq->viommu = viommu;
565+ refcount_inc(&viommu->obj.users);
566+ veventq->depth = cmd->veventq_depth;
567+ list_add_tail(&veventq->node, &viommu->veventqs);
568+ veventq->lost_events_header.header.flags =
569+ IOMMU_VEVENTQ_FLAG_LOST_EVENTS;
570+
571+ fdno = iommufd_eventq_init(&veventq->common, "[iommufd-viommu-event]",
572+ ucmd->ictx, &iommufd_veventq_fops);
573+ if (fdno < 0) {
574+ rc = fdno;
575+ goto out_abort;
576+ }
577+
578+ cmd->out_veventq_id = veventq->common.obj.id;
579+ cmd->out_veventq_fd = fdno;
580+
581+ rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
582+ if (rc)
583+ goto out_put_fdno;
584+
585+ iommufd_object_finalize(ucmd->ictx, &veventq->common.obj);
586+ fd_install(fdno, veventq->common.filep);
587+ goto out_unlock_veventqs;
588+
589+out_put_fdno:
590+ put_unused_fd(fdno);
591+ fput(veventq->common.filep);
592+out_abort:
593+ iommufd_object_abort_and_destroy(ucmd->ictx, &veventq->common.obj);
594+out_unlock_veventqs:
595+ up_write(&viommu->veventqs_rwsem);
596+ iommufd_put_object(ucmd->ictx, &viommu->obj);
597+ return rc;
598+}
drivers/iommu/iommufd/fault.c+0−342
@@ -1,342 +0,0 @@
1−// SPDX-License-Identifier: GPL-2.0-only
2−/* Copyright (C) 2024 Intel Corporation
3− */
4−#define pr_fmt(fmt) "iommufd: " fmt
5−
6−#include <linux/anon_inodes.h>
7−#include <linux/file.h>
8−#include <linux/fs.h>
9−#include <linux/iommufd.h>
10−#include <linux/module.h>
11−#include <linux/mutex.h>
12−#include <linux/pci.h>
13−#include <linux/pci-ats.h>
14−#include <linux/poll.h>
15−#include <uapi/linux/iommufd.h>
16−
17−#include "../iommu-priv.h"
18−#include "iommufd_private.h"
19−
20−int iommufd_fault_iopf_enable(struct iommufd_device *idev)
21−{
22− struct device *dev = idev->dev;
23− int ret;
24−
25− /*
26− * Once we turn on PCI/PRI support for VF, the response failure code
27− * should not be forwarded to the hardware due to PRI being a shared
28− * resource between PF and VFs. There is no coordination for this
29− * shared capability. This waits for a vPRI reset to recover.
30− */
31− if (dev_is_pci(dev)) {
32− struct pci_dev *pdev = to_pci_dev(dev);
33−
34− if (pdev->is_virtfn && pci_pri_supported(pdev))
35− return -EINVAL;
36− }
37−
38− mutex_lock(&idev->iopf_lock);
39− /* Device iopf has already been on. */
40− if (++idev->iopf_enabled > 1) {
41− mutex_unlock(&idev->iopf_lock);
42− return 0;
43− }
44−
45− ret = iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_IOPF);
46− if (ret)
47− --idev->iopf_enabled;
48− mutex_unlock(&idev->iopf_lock);
49−
50− return ret;
51−}
52−
53−void iommufd_fault_iopf_disable(struct iommufd_device *idev)
54−{
55− mutex_lock(&idev->iopf_lock);
56− if (!WARN_ON(idev->iopf_enabled == 0)) {
57− if (--idev->iopf_enabled == 0)
58− iommu_dev_disable_feature(idev->dev, IOMMU_DEV_FEAT_IOPF);
59− }
60− mutex_unlock(&idev->iopf_lock);
61−}
62−
63−void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt,
64− struct iommufd_attach_handle *handle)
65−{
66− struct iommufd_fault *fault = hwpt->fault;
67− struct iopf_group *group, *next;
68− struct list_head free_list;
69− unsigned long index;
70−
71− if (!fault)
72− return;
73− INIT_LIST_HEAD(&free_list);
74−
75− mutex_lock(&fault->mutex);
76− spin_lock(&fault->lock);
77− list_for_each_entry_safe(group, next, &fault->deliver, node) {
78− if (group->attach_handle != &handle->handle)
79− continue;
80− list_move(&group->node, &free_list);
81− }
82− spin_unlock(&fault->lock);
83−
84− list_for_each_entry_safe(group, next, &free_list, node) {
85− list_del(&group->node);
86− iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
87− iopf_free_group(group);
88− }
89−
90− xa_for_each(&fault->response, index, group) {
91− if (group->attach_handle != &handle->handle)
92− continue;
93− xa_erase(&fault->response, index);
94− iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
95− iopf_free_group(group);
96− }
97− mutex_unlock(&fault->mutex);
98−}
99−
100−void iommufd_fault_destroy(struct iommufd_object *obj)
101−{
102− struct iommufd_fault *fault = container_of(obj, struct iommufd_fault, obj);
103− struct iopf_group *group, *next;
104− unsigned long index;
105−
106− /*
107− * The iommufd object's reference count is zero at this point.
108− * We can be confident that no other threads are currently
109− * accessing this pointer. Therefore, acquiring the mutex here
110− * is unnecessary.
111− */
112− list_for_each_entry_safe(group, next, &fault->deliver, node) {
113− list_del(&group->node);
114− iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
115− iopf_free_group(group);
116− }
117− xa_for_each(&fault->response, index, group) {
118− xa_erase(&fault->response, index);
119− iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
120− iopf_free_group(group);
121− }
122− xa_destroy(&fault->response);
123− mutex_destroy(&fault->mutex);
124−}
125−
126−static void iommufd_compose_fault_message(struct iommu_fault *fault,
127− struct iommu_hwpt_pgfault *hwpt_fault,
128− struct iommufd_device *idev,
129− u32 cookie)
130−{
131− hwpt_fault->flags = fault->prm.flags;
132− hwpt_fault->dev_id = idev->obj.id;
133− hwpt_fault->pasid = fault->prm.pasid;
134− hwpt_fault->grpid = fault->prm.grpid;
135− hwpt_fault->perm = fault->prm.perm;
136− hwpt_fault->addr = fault->prm.addr;
137− hwpt_fault->length = 0;
138− hwpt_fault->cookie = cookie;
139−}
140−
141−static ssize_t iommufd_fault_fops_read(struct file *filep, char __user *buf,
142− size_t count, loff_t *ppos)
143−{
144− size_t fault_size = sizeof(struct iommu_hwpt_pgfault);
145− struct iommufd_fault *fault = filep->private_data;
146− struct iommu_hwpt_pgfault data = {};
147− struct iommufd_device *idev;
148− struct iopf_group *group;
149− struct iopf_fault *iopf;
150− size_t done = 0;
151− int rc = 0;
152−
153− if (*ppos || count % fault_size)
154− return -ESPIPE;
155−
156− mutex_lock(&fault->mutex);
157− while ((group = iommufd_fault_deliver_fetch(fault))) {
158− if (done >= count ||
159− group->fault_count * fault_size > count - done) {
160− iommufd_fault_deliver_restore(fault, group);
161− break;
162− }
163−
164− rc = xa_alloc(&fault->response, &group->cookie, group,
165− xa_limit_32b, GFP_KERNEL);
166− if (rc) {
167− iommufd_fault_deliver_restore(fault, group);
168− break;
169− }
170−
171− idev = to_iommufd_handle(group->attach_handle)->idev;
172− list_for_each_entry(iopf, &group->faults, list) {
173− iommufd_compose_fault_message(&iopf->fault,
174− &data, idev,
175− group->cookie);
176− if (copy_to_user(buf + done, &data, fault_size)) {
177− xa_erase(&fault->response, group->cookie);
178− iommufd_fault_deliver_restore(fault, group);
179− rc = -EFAULT;
180− break;
181− }
182− done += fault_size;
183− }
184− }
185− mutex_unlock(&fault->mutex);
186−
187− return done == 0 ? rc : done;
188−}
189−
190−static ssize_t iommufd_fault_fops_write(struct file *filep, const char __user *buf,
191− size_t count, loff_t *ppos)
192−{
193− size_t response_size = sizeof(struct iommu_hwpt_page_response);
194− struct iommufd_fault *fault = filep->private_data;
195− struct iommu_hwpt_page_response response;
196− struct iopf_group *group;
197− size_t done = 0;
198− int rc = 0;
199−
200− if (*ppos || count % response_size)
201− return -ESPIPE;
202−
203− mutex_lock(&fault->mutex);
204− while (count > done) {
205− rc = copy_from_user(&response, buf + done, response_size);
206− if (rc)
207− break;
208−
209− static_assert((int)IOMMUFD_PAGE_RESP_SUCCESS ==
210− (int)IOMMU_PAGE_RESP_SUCCESS);
211− static_assert((int)IOMMUFD_PAGE_RESP_INVALID ==
212− (int)IOMMU_PAGE_RESP_INVALID);
213− if (response.code != IOMMUFD_PAGE_RESP_SUCCESS &&
214− response.code != IOMMUFD_PAGE_RESP_INVALID) {
215− rc = -EINVAL;
216− break;
217− }
218−
219− group = xa_erase(&fault->response, response.cookie);
220− if (!group) {
221− rc = -EINVAL;
222− break;
223− }
224−
225− iopf_group_response(group, response.code);
226− iopf_free_group(group);
227− done += response_size;
228− }
229− mutex_unlock(&fault->mutex);
230−
231− return done == 0 ? rc : done;
232−}
233−
234−static __poll_t iommufd_fault_fops_poll(struct file *filep,
235− struct poll_table_struct *wait)
236−{
237− struct iommufd_fault *fault = filep->private_data;
238− __poll_t pollflags = EPOLLOUT;
239−
240− poll_wait(filep, &fault->wait_queue, wait);
241− spin_lock(&fault->lock);
242− if (!list_empty(&fault->deliver))
243− pollflags |= EPOLLIN | EPOLLRDNORM;
244− spin_unlock(&fault->lock);
245−
246− return pollflags;
247−}
248−
249−static int iommufd_fault_fops_release(struct inode *inode, struct file *filep)
250−{
251− struct iommufd_fault *fault = filep->private_data;
252−
253− refcount_dec(&fault->obj.users);
254− iommufd_ctx_put(fault->ictx);
255− return 0;
256−}
257−
258−static const struct file_operations iommufd_fault_fops = {
259− .owner = THIS_MODULE,
260− .open = nonseekable_open,
261− .read = iommufd_fault_fops_read,
262− .write = iommufd_fault_fops_write,
263− .poll = iommufd_fault_fops_poll,
264− .release = iommufd_fault_fops_release,
265−};
266−
267−int iommufd_fault_alloc(struct iommufd_ucmd *ucmd)
268−{
269− struct iommu_fault_alloc *cmd = ucmd->cmd;
270− struct iommufd_fault *fault;
271− struct file *filep;
272− int fdno;
273− int rc;
274−
275− if (cmd->flags)
276− return -EOPNOTSUPP;
277−
278− fault = iommufd_object_alloc(ucmd->ictx, fault, IOMMUFD_OBJ_FAULT);
279− if (IS_ERR(fault))
280− return PTR_ERR(fault);
281−
282− fault->ictx = ucmd->ictx;
283− INIT_LIST_HEAD(&fault->deliver);
284− xa_init_flags(&fault->response, XA_FLAGS_ALLOC1);
285− mutex_init(&fault->mutex);
286− spin_lock_init(&fault->lock);
287− init_waitqueue_head(&fault->wait_queue);
288−
289− filep = anon_inode_getfile("[iommufd-pgfault]", &iommufd_fault_fops,
290− fault, O_RDWR);
291− if (IS_ERR(filep)) {
292− rc = PTR_ERR(filep);
293− goto out_abort;
294− }
295−
296− refcount_inc(&fault->obj.users);
297− iommufd_ctx_get(fault->ictx);
298− fault->filep = filep;
299−
300− fdno = get_unused_fd_flags(O_CLOEXEC);
301− if (fdno < 0) {
302− rc = fdno;
303− goto out_fput;
304− }
305−
306− cmd->out_fault_id = fault->obj.id;
307− cmd->out_fault_fd = fdno;
308−
309− rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
310− if (rc)
311− goto out_put_fdno;
312− iommufd_object_finalize(ucmd->ictx, &fault->obj);
313−
314− fd_install(fdno, fault->filep);
315−
316− return 0;
317−out_put_fdno:
318− put_unused_fd(fdno);
319−out_fput:
320− fput(filep);
321−out_abort:
322− iommufd_object_abort_and_destroy(ucmd->ictx, &fault->obj);
323−
324− return rc;
325−}
326−
327−int iommufd_fault_iopf_handler(struct iopf_group *group)
328−{
329− struct iommufd_hw_pagetable *hwpt;
330− struct iommufd_fault *fault;
331−
332− hwpt = group->attach_handle->domain->iommufd_hwpt;
333− fault = hwpt->fault;
334−
335− spin_lock(&fault->lock);
336− list_add_tail(&group->node, &fault->deliver);
337− spin_unlock(&fault->lock);
338−
339− wake_up_interruptible(&fault->wait_queue);
340−
341− return 0;
342−}
drivers/iommu/iommufd/hw_pagetable.c+26−16
@@ -14,7 +14,7 @@ static void __iommufd_hwpt_destroy(struct iommufd_hw_pagetable *hwpt) · 13 unmodified lines
1414 iommu_domain_free(hwpt->domain);
1515
1616 if (hwpt->fault)
17− refcount_dec(&hwpt->fault->obj.users);
17+ refcount_dec(&hwpt->fault->common.obj.users);
1818 }
1919
2020 void iommufd_hwpt_paging_destroy(struct iommufd_object *obj)
@@ -90,6 +90,7 @@ iommufd_hwpt_paging_enforce_cc(struct iommufd_hwpt_paging *hwpt_paging) · 69 unmodified lines
9090 * @ictx: iommufd context
9191 * @ioas: IOAS to associate the domain with
9292 * @idev: Device to get an iommu_domain for
93+ * @pasid: PASID to get an iommu_domain for
9394 * @flags: Flags from userspace
9495 * @immediate_attach: True if idev should be attached to the hwpt
9596 * @user_data: The user provided driver specific data describing the domain to
@@ -105,13 +106,14 @@ iommufd_hwpt_paging_enforce_cc(struct iommufd_hwpt_paging *hwpt_paging) · 9 unmodified lines
105106 */
106107 struct iommufd_hwpt_paging *
107108 iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas,
108− struct iommufd_device *idev, u32 flags,
109− bool immediate_attach,
109+ struct iommufd_device *idev, ioasid_t pasid,
110+ u32 flags, bool immediate_attach,
110111 const struct iommu_user_data *user_data)
111112 {
112113 const u32 valid_flags = IOMMU_HWPT_ALLOC_NEST_PARENT |
113114 IOMMU_HWPT_ALLOC_DIRTY_TRACKING |
114− IOMMU_HWPT_FAULT_ID_VALID;
115+ IOMMU_HWPT_FAULT_ID_VALID |
116+ IOMMU_HWPT_ALLOC_PASID;
115117 const struct iommu_ops *ops = dev_iommu_ops(idev->dev);
116118 struct iommufd_hwpt_paging *hwpt_paging;
117119 struct iommufd_hw_pagetable *hwpt;
@@ -126,12 +128,16 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 8 unmodified lines
126128 if ((flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING) &&
127129 !device_iommu_capable(idev->dev, IOMMU_CAP_DIRTY_TRACKING))
128130 return ERR_PTR(-EOPNOTSUPP);
131+ if ((flags & IOMMU_HWPT_FAULT_ID_VALID) &&
132+ (flags & IOMMU_HWPT_ALLOC_NEST_PARENT))
133+ return ERR_PTR(-EOPNOTSUPP);
129134
130135 hwpt_paging = __iommufd_object_alloc(
131136 ictx, hwpt_paging, IOMMUFD_OBJ_HWPT_PAGING, common.obj);
132137 if (IS_ERR(hwpt_paging))
133138 return ERR_CAST(hwpt_paging);
134139 hwpt = &hwpt_paging->common;
140+ hwpt->pasid_compat = flags & IOMMU_HWPT_ALLOC_PASID;
135141
136142 INIT_LIST_HEAD(&hwpt_paging->hwpt_item);
137143 /* Pairs with iommufd_hw_pagetable_destroy() */
@@ -156,7 +162,8 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 18 unmodified lines
156162 goto out_abort;
157163 }
158164 }
159− iommu_domain_set_sw_msi(hwpt->domain, iommufd_sw_msi);
165+ hwpt->domain->iommufd_hwpt = hwpt;
166+ hwpt->domain->cookie_type = IOMMU_COOKIE_IOMMUFD;
160167
161168 /*
162169 * Set the coherency mode before we do iopt_table_add_domain() as some
@@ -185,7 +192,7 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 22 unmodified lines
185192 * sequence. Once those drivers are fixed this should be removed.
186193 */
187194 if (immediate_attach) {
188− rc = iommufd_hw_pagetable_attach(hwpt, idev);
195+ rc = iommufd_hw_pagetable_attach(hwpt, idev, pasid);
189196 if (rc)
190197 goto out_abort;
191198 }
@@ -198,7 +205,7 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 6 unmodified lines
198205
199206 out_detach:
200207 if (immediate_attach)
201− iommufd_hw_pagetable_detach(idev);
208+ iommufd_hw_pagetable_detach(idev, pasid);
202209 out_abort:
203210 iommufd_object_abort_and_destroy(ictx, &hwpt->obj);
204211 return ERR_PTR(rc);
@@ -227,7 +234,7 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, · 22 unmodified lines
227234 struct iommufd_hw_pagetable *hwpt;
228235 int rc;
229236
230− if ((flags & ~IOMMU_HWPT_FAULT_ID_VALID) ||
237+ if ((flags & ~(IOMMU_HWPT_FAULT_ID_VALID | IOMMU_HWPT_ALLOC_PASID)) ||
231238 !user_data->len || !ops->domain_alloc_nested)
232239 return ERR_PTR(-EOPNOTSUPP);
233240 if (parent->auto_domain || !parent->nest_parent ||
@@ -239,6 +246,7 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, · 5 unmodified lines
239246 if (IS_ERR(hwpt_nested))
240247 return ERR_CAST(hwpt_nested);
241248 hwpt = &hwpt_nested->common;
249+ hwpt->pasid_compat = flags & IOMMU_HWPT_ALLOC_PASID;
242250
243251 refcount_inc(&parent->common.obj.users);
244252 hwpt_nested->parent = parent;
@@ -252,7 +260,8 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, · 7 unmodified lines
252260 goto out_abort;
253261 }
254262 hwpt->domain->owner = ops;
255− iommu_domain_set_sw_msi(hwpt->domain, iommufd_sw_msi);
263+ hwpt->domain->iommufd_hwpt = hwpt;
264+ hwpt->domain->cookie_type = IOMMU_COOKIE_IOMMUFD;
256265
257266 if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) {
258267 rc = -EINVAL;
@@ -282,7 +291,7 @@ iommufd_viommu_alloc_hwpt_nested(struct iommufd_viommu *viommu, u32 flags, · 23 unmodified lines
282291 struct iommufd_hw_pagetable *hwpt;
283292 int rc;
284293
285− if (flags & ~IOMMU_HWPT_FAULT_ID_VALID)
294+ if (flags & ~(IOMMU_HWPT_FAULT_ID_VALID | IOMMU_HWPT_ALLOC_PASID))
286295 return ERR_PTR(-EOPNOTSUPP);
287296 if (!user_data->len)
288297 return ERR_PTR(-EOPNOTSUPP);
@@ -294,6 +303,7 @@ iommufd_viommu_alloc_hwpt_nested(struct iommufd_viommu *viommu, u32 flags, · 5 unmodified lines
294303 if (IS_ERR(hwpt_nested))
295304 return ERR_CAST(hwpt_nested);
296305 hwpt = &hwpt_nested->common;
306+ hwpt->pasid_compat = flags & IOMMU_HWPT_ALLOC_PASID;
297307
298308 hwpt_nested->viommu = viommu;
299309 refcount_inc(&viommu->obj.users);
@@ -308,8 +318,9 @@ iommufd_viommu_alloc_hwpt_nested(struct iommufd_viommu *viommu, u32 flags, · 8 unmodified lines
308318 hwpt->domain = NULL;
309319 goto out_abort;
310320 }
321+ hwpt->domain->iommufd_hwpt = hwpt;
311322 hwpt->domain->owner = viommu->iommu_dev->ops;
312− iommu_domain_set_sw_msi(hwpt->domain, iommufd_sw_msi);
323+ hwpt->domain->cookie_type = IOMMU_COOKIE_IOMMUFD;
313324
314325 if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) {
315326 rc = -EINVAL;
@@ -358,8 +369,8 @@ int iommufd_hwpt_alloc(struct iommufd_ucmd *ucmd) · 42 unmodified lines
358369 ioas = container_of(pt_obj, struct iommufd_ioas, obj);
359370 mutex_lock(&ioas->mutex);
360371 hwpt_paging = iommufd_hwpt_paging_alloc(
361− ucmd->ictx, ioas, idev, cmd->flags, false,
362− user_data.len ? &user_data : NULL);
372+ ucmd->ictx, ioas, idev, IOMMU_NO_PASID, cmd->flags,
373+ false, user_data.len ? &user_data : NULL);
363374 if (IS_ERR(hwpt_paging)) {
364375 rc = PTR_ERR(hwpt_paging);
365376 goto out_unlock;
@@ -409,10 +420,9 @@ int iommufd_hwpt_alloc(struct iommufd_ucmd *ucmd) · 43 unmodified lines
409420 }
410421 hwpt->fault = fault;
411422 hwpt->domain->iopf_handler = iommufd_fault_iopf_handler;
412− refcount_inc(&fault->obj.users);
413− iommufd_put_object(ucmd->ictx, &fault->obj);
423+ refcount_inc(&fault->common.obj.users);
424+ iommufd_put_object(ucmd->ictx, &fault->common.obj);
414425 }
415− hwpt->domain->iommufd_hwpt = hwpt;
416426
417427 cmd->out_hwpt_id = hwpt->obj.id;
418428 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
drivers/iommu/iommufd/iommufd_private.h+114−42
@@ -32,8 +32,11 @@ struct iommufd_sw_msi_maps { · 31 unmodified lines
3232 DECLARE_BITMAP(bitmap, 64);
3333 };
3434
35−int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
36− phys_addr_t msi_addr);
35+#ifdef CONFIG_IRQ_MSI_IOMMU
36+int iommufd_sw_msi_install(struct iommufd_ctx *ictx,
37+ struct iommufd_hwpt_paging *hwpt_paging,
38+ struct iommufd_sw_msi_map *msi_map);
39+#endif
3740
3841 struct iommufd_ctx {
3942 struct file *file;
@@ -296,6 +299,7 @@ struct iommufd_hw_pagetable { · 256 unmodified lines
296299 struct iommufd_object obj;
297300 struct iommu_domain *domain;
298301 struct iommufd_fault *fault;
302+ bool pasid_compat : 1;
299303 };
300304
301305 struct iommufd_hwpt_paging {
@@ -366,13 +370,13 @@ int iommufd_hwpt_get_dirty_bitmap(struct iommufd_ucmd *ucmd); · 64 unmodified lines
366370
367371 struct iommufd_hwpt_paging *
368372 iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas,
369− struct iommufd_device *idev, u32 flags,
370− bool immediate_attach,
373+ struct iommufd_device *idev, ioasid_t pasid,
374+ u32 flags, bool immediate_attach,
371375 const struct iommu_user_data *user_data);
372376 int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,
373− struct iommufd_device *idev);
377+ struct iommufd_device *idev, ioasid_t pasid);
374378 struct iommufd_hw_pagetable *
375−iommufd_hw_pagetable_detach(struct iommufd_device *idev);
379+iommufd_hw_pagetable_detach(struct iommufd_device *idev, ioasid_t pasid);
376380 void iommufd_hwpt_paging_destroy(struct iommufd_object *obj);
377381 void iommufd_hwpt_paging_abort(struct iommufd_object *obj);
378382 void iommufd_hwpt_nested_destroy(struct iommufd_object *obj);
@@ -396,13 +400,14 @@ static inline void iommufd_hw_pagetable_put(struct iommufd_ctx *ictx, · 17 unmodified lines
396400 refcount_dec(&hwpt->obj.users);
397401 }
398402
403+struct iommufd_attach;
404+
399405 struct iommufd_group {
400406 struct kref ref;
401407 struct mutex lock;
402408 struct iommufd_ctx *ictx;
403409 struct iommu_group *group;
404− struct iommufd_hw_pagetable *hwpt;
405− struct list_head device_list;
410+ struct xarray pasid_attach;
406411 struct iommufd_sw_msi_maps required_sw_msi;
407412 phys_addr_t sw_msi_start;
408413 };
@@ -454,49 +459,17 @@ void iopt_remove_access(struct io_pagetable *iopt, · 45 unmodified lines
454459 u32 iopt_access_list_id);
455460 void iommufd_access_destroy_object(struct iommufd_object *obj);
456461
457−/*
458− * An iommufd_fault object represents an interface to deliver I/O page faults
459− * to the user space. These objects are created/destroyed by the user space and
460− * associated with hardware page table objects during page-table allocation.
461− */
462−struct iommufd_fault {
462+struct iommufd_eventq {
463463 struct iommufd_object obj;
464464 struct iommufd_ctx *ictx;
465465 struct file *filep;
466466
467467 spinlock_t lock; /* protects the deliver list */
468468 struct list_head deliver;
469− struct mutex mutex; /* serializes response flows */
470− struct xarray response;
471469
472470 struct wait_queue_head wait_queue;
473471 };
474472
475−/* Fetch the first node out of the fault->deliver list */
476−static inline struct iopf_group *
477−iommufd_fault_deliver_fetch(struct iommufd_fault *fault)
478−{
479− struct list_head *list = &fault->deliver;
480− struct iopf_group *group = NULL;
481−
482− spin_lock(&fault->lock);
483− if (!list_empty(list)) {
484− group = list_first_entry(list, struct iopf_group, node);
485− list_del(&group->node);
486− }
487− spin_unlock(&fault->lock);
488− return group;
489−}
490−
491−/* Restore a node back to the head of the fault->deliver list */
492−static inline void iommufd_fault_deliver_restore(struct iommufd_fault *fault,
493− struct iopf_group *group)
494−{
495− spin_lock(&fault->lock);
496− list_add(&group->node, &fault->deliver);
497− spin_unlock(&fault->lock);
498−}
499−
500473 struct iommufd_attach_handle {
501474 struct iommu_attach_handle handle;
502475 struct iommufd_device *idev;
@@ -505,12 +478,29 @@ struct iommufd_attach_handle { · 2 unmodified lines
505478 /* Convert an iommu attach handle to iommufd handle. */
506479 #define to_iommufd_handle(hdl) container_of(hdl, struct iommufd_attach_handle, handle)
507480
481+/*
482+ * An iommufd_fault object represents an interface to deliver I/O page faults
483+ * to the user space. These objects are created/destroyed by the user space and
484+ * associated with hardware page table objects during page-table allocation.
485+ */
486+struct iommufd_fault {
487+ struct iommufd_eventq common;
488+ struct mutex mutex; /* serializes response flows */
489+ struct xarray response;
490+};
491+
492+static inline struct iommufd_fault *
493+eventq_to_fault(struct iommufd_eventq *eventq)
494+{
495+ return container_of(eventq, struct iommufd_fault, common);
496+}
497+
508498 static inline struct iommufd_fault *
509499 iommufd_get_fault(struct iommufd_ucmd *ucmd, u32 id)
510500 {
511501 return container_of(iommufd_get_object(ucmd->ictx, id,
512502 IOMMUFD_OBJ_FAULT),
513− struct iommufd_fault, obj);
503+ struct iommufd_fault, common.obj);
514504 }
515505
516506 int iommufd_fault_alloc(struct iommufd_ucmd *ucmd);
@@ -522,6 +512,74 @@ void iommufd_fault_iopf_disable(struct iommufd_device *idev); · 5 unmodified lines
522512 void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt,
523513 struct iommufd_attach_handle *handle);
524514
515+/* An iommufd_vevent represents a vIOMMU event in an iommufd_veventq */
516+struct iommufd_vevent {
517+ struct iommufd_vevent_header header;
518+ struct list_head node; /* for iommufd_eventq::deliver */
519+ ssize_t data_len;
520+ u64 event_data[] __counted_by(data_len);
521+};
522+
523+#define vevent_for_lost_events_header(vevent) \
524+ (vevent->header.flags & IOMMU_VEVENTQ_FLAG_LOST_EVENTS)
525+
526+/*
527+ * An iommufd_veventq object represents an interface to deliver vIOMMU events to
528+ * the user space. It is created/destroyed by the user space and associated with
529+ * a vIOMMU object during the allocations.
530+ */
531+struct iommufd_veventq {
532+ struct iommufd_eventq common;
533+ struct iommufd_viommu *viommu;
534+ struct list_head node; /* for iommufd_viommu::veventqs */
535+ struct iommufd_vevent lost_events_header;
536+
537+ unsigned int type;
538+ unsigned int depth;
539+
540+ /* Use common.lock for protection */
541+ u32 num_events;
542+ u32 sequence;
543+};
544+
545+static inline struct iommufd_veventq *
546+eventq_to_veventq(struct iommufd_eventq *eventq)
547+{
548+ return container_of(eventq, struct iommufd_veventq, common);
549+}
550+
551+static inline struct iommufd_veventq *
552+iommufd_get_veventq(struct iommufd_ucmd *ucmd, u32 id)
553+{
554+ return container_of(iommufd_get_object(ucmd->ictx, id,
555+ IOMMUFD_OBJ_VEVENTQ),
556+ struct iommufd_veventq, common.obj);
557+}
558+
559+int iommufd_veventq_alloc(struct iommufd_ucmd *ucmd);
560+void iommufd_veventq_destroy(struct iommufd_object *obj);
561+void iommufd_veventq_abort(struct iommufd_object *obj);
562+
563+static inline void iommufd_vevent_handler(struct iommufd_veventq *veventq,
564+ struct iommufd_vevent *vevent)
565+{
566+ struct iommufd_eventq *eventq = &veventq->common;
567+
568+ lockdep_assert_held(&eventq->lock);
569+
570+ /*
571+ * Remove the lost_events_header and add the new node at the same time.
572+ * Note the new node can be lost_events_header, for a sequence update.
573+ */
574+ if (list_is_last(&veventq->lost_events_header.node, &eventq->deliver))
575+ list_del(&veventq->lost_events_header.node);
576+ list_add_tail(&vevent->node, &eventq->deliver);
577+ vevent->header.sequence = veventq->sequence;
578+ veventq->sequence = (veventq->sequence + 1) & INT_MAX;
579+
580+ wake_up_interruptible(&eventq->wait_queue);
581+}
582+
525583 static inline struct iommufd_viommu *
526584 iommufd_get_viommu(struct iommufd_ucmd *ucmd, u32 id)
527585 {
@@ -530,6 +588,20 @@ iommufd_get_viommu(struct iommufd_ucmd *ucmd, u32 id) · 2 unmodified lines
530588 struct iommufd_viommu, obj);
531589 }
532590
591+static inline struct iommufd_veventq *
592+iommufd_viommu_find_veventq(struct iommufd_viommu *viommu, u32 type)
593+{
594+ struct iommufd_veventq *veventq, *next;
595+
596+ lockdep_assert_held(&viommu->veventqs_rwsem);
597+
598+ list_for_each_entry_safe(veventq, next, &viommu->veventqs, node) {
599+ if (veventq->type == type)
600+ return veventq;
601+ }
602+ return NULL;
603+}
604+
533605 int iommufd_viommu_alloc_ioctl(struct iommufd_ucmd *ucmd);
534606 void iommufd_viommu_destroy(struct iommufd_object *obj);
535607 int iommufd_vdevice_alloc_ioctl(struct iommufd_ucmd *ucmd);
drivers/iommu/iommufd/iommufd_test.h+40−0
@@ -24,6 +24,11 @@ enum { · 23 unmodified lines
2424 IOMMU_TEST_OP_MD_CHECK_IOTLB,
2525 IOMMU_TEST_OP_TRIGGER_IOPF,
2626 IOMMU_TEST_OP_DEV_CHECK_CACHE,
27+ IOMMU_TEST_OP_TRIGGER_VEVENT,
28+ IOMMU_TEST_OP_PASID_ATTACH,
29+ IOMMU_TEST_OP_PASID_REPLACE,
30+ IOMMU_TEST_OP_PASID_DETACH,
31+ IOMMU_TEST_OP_PASID_CHECK_HWPT,
2732 };
2833
2934 enum {
@@ -48,6 +53,7 @@ enum { · 18 unmodified lines
4853 enum {
4954 MOCK_FLAGS_DEVICE_NO_DIRTY = 1 << 0,
5055 MOCK_FLAGS_DEVICE_HUGE_IOVA = 1 << 1,
56+ MOCK_FLAGS_DEVICE_PASID = 1 << 2,
5157 };
5258
5359 enum {
@@ -60,6 +66,9 @@ enum { · 6 unmodified lines
6066 MOCK_DEV_CACHE_NUM = 4,
6167 };
6268
69+/* Reserved for special pasid replace test */
70+#define IOMMU_TEST_PASID_RESERVED 1024
71+
6372 struct iommu_test_cmd {
6473 __u32 size;
6574 __u32 op;
@@ -145,11 +154,36 @@ struct iommu_test_cmd { · 79 unmodified lines
145154 __u32 id;
146155 __u32 cache;
147156 } check_dev_cache;
157+ struct {
158+ __u32 dev_id;
159+ } trigger_vevent;
160+ struct {
161+ __u32 pasid;
162+ __u32 pt_id;
163+ /* @id is stdev_id */
164+ } pasid_attach;
165+ struct {
166+ __u32 pasid;
167+ __u32 pt_id;
168+ /* @id is stdev_id */
169+ } pasid_replace;
170+ struct {
171+ __u32 pasid;
172+ /* @id is stdev_id */
173+ } pasid_detach;
174+ struct {
175+ __u32 pasid;
176+ __u32 hwpt_id;
177+ /* @id is stdev_id */
178+ } pasid_check;
148179 };
149180 __u32 last;
150181 };
151182 #define IOMMU_TEST_CMD _IO(IOMMUFD_TYPE, IOMMUFD_CMD_BASE + 32)
152183
184+/* Mock device/iommu PASID width */
185+#define MOCK_PASID_WIDTH 20
186+
153187 /* Mock structs for IOMMU_DEVICE_GET_HW_INFO ioctl */
154188 #define IOMMU_HW_INFO_TYPE_SELFTEST 0xfeedbeef
155189 #define IOMMU_HW_INFO_SELFTEST_REGVAL 0xdeadbeef
@@ -212,4 +246,10 @@ struct iommu_viommu_invalidate_selftest { · 56 unmodified lines
212246 __u32 cache_id;
213247 };
214248
249+#define IOMMU_VEVENTQ_TYPE_SELFTEST 0xbeefbeef
250+
251+struct iommu_viommu_event_selftest {
252+ __u32 virt_id;
253+};
254+
215255 #endif
drivers/iommu/iommufd/main.c+7−0
@@ -317,6 +317,7 @@ union ucmd_buffer { · 316 unmodified lines
317317 struct iommu_ioas_unmap unmap;
318318 struct iommu_option option;
319319 struct iommu_vdevice_alloc vdev;
320+ struct iommu_veventq_alloc veventq;
320321 struct iommu_vfio_ioas vfio_ioas;
321322 struct iommu_viommu_alloc viommu;
322323 #ifdef CONFIG_IOMMUFD_TEST
@@ -372,6 +373,8 @@ static const struct iommufd_ioctl_op iommufd_ioctl_ops[] = { · 49 unmodified lines
372373 IOCTL_OP(IOMMU_OPTION, iommufd_option, struct iommu_option, val64),
373374 IOCTL_OP(IOMMU_VDEVICE_ALLOC, iommufd_vdevice_alloc_ioctl,
374375 struct iommu_vdevice_alloc, virt_id),
376+ IOCTL_OP(IOMMU_VEVENTQ_ALLOC, iommufd_veventq_alloc,
377+ struct iommu_veventq_alloc, out_veventq_fd),
375378 IOCTL_OP(IOMMU_VFIO_IOAS, iommufd_vfio_ioas, struct iommu_vfio_ioas,
376379 __reserved),
377380 IOCTL_OP(IOMMU_VIOMMU_ALLOC, iommufd_viommu_alloc_ioctl,
@@ -514,6 +517,10 @@ static const struct iommufd_object_ops iommufd_object_ops[] = { · 136 unmodified lines
514517 [IOMMUFD_OBJ_VDEVICE] = {
515518 .destroy = iommufd_vdevice_destroy,
516519 },
520+ [IOMMUFD_OBJ_VEVENTQ] = {
521+ .destroy = iommufd_veventq_destroy,
522+ .abort = iommufd_veventq_abort,
523+ },
517524 [IOMMUFD_OBJ_VIOMMU] = {
518525 .destroy = iommufd_viommu_destroy,
519526 },
drivers/iommu/iommufd/selftest.c+275−22
@@ -161,9 +161,13 @@ enum selftest_obj_type { · 160 unmodified lines
161161
162162 struct mock_dev {
163163 struct device dev;
164+ struct mock_viommu *viommu;
165+ struct rw_semaphore viommu_rwsem;
164166 unsigned long flags;
167+ unsigned long vdev_id;
165168 int id;
166169 u32 cache[MOCK_DEV_CACHE_NUM];
170+ atomic_t pasid_1024_fake_error;
167171 };
168172
169173 static inline struct mock_dev *to_mock_dev(struct device *dev)
@@ -193,15 +197,71 @@ static int mock_domain_nop_attach(struct iommu_domain *domain, · 23 unmodified lines
193197 struct device *dev)
194198 {
195199 struct mock_dev *mdev = to_mock_dev(dev);
200+ struct mock_viommu *new_viommu = NULL;
201+ unsigned long vdev_id = 0;
202+ int rc;
196203
197204 if (domain->dirty_ops && (mdev->flags & MOCK_FLAGS_DEVICE_NO_DIRTY))
198205 return -EINVAL;
199206
207+ iommu_group_mutex_assert(dev);
208+ if (domain->type == IOMMU_DOMAIN_NESTED) {
209+ new_viommu = to_mock_nested(domain)->mock_viommu;
210+ if (new_viommu) {
211+ rc = iommufd_viommu_get_vdev_id(&new_viommu->core, dev,
212+ &vdev_id);
213+ if (rc)
214+ return rc;
215+ }
216+ }
217+ if (new_viommu != mdev->viommu) {
218+ down_write(&mdev->viommu_rwsem);
219+ mdev->viommu = new_viommu;
220+ mdev->vdev_id = vdev_id;
221+ up_write(&mdev->viommu_rwsem);
222+ }
223+
224+ return 0;
225+}
226+
227+static int mock_domain_set_dev_pasid_nop(struct iommu_domain *domain,
228+ struct device *dev, ioasid_t pasid,
229+ struct iommu_domain *old)
230+{
231+ struct mock_dev *mdev = to_mock_dev(dev);
232+
233+ /*
234+ * Per the first attach with pasid 1024, set the
235+ * mdev->pasid_1024_fake_error. Hence the second call of this op
236+ * can fake an error to validate the error path of the core. This
237+ * is helpful to test the case in which the iommu core needs to
238+ * rollback to the old domain due to driver failure. e.g. replace.
239+ * User should be careful about the third call of this op, it shall
240+ * succeed since the mdev->pasid_1024_fake_error is cleared in the
241+ * second call.
242+ */
243+ if (pasid == 1024) {
244+ if (domain->type == IOMMU_DOMAIN_BLOCKED) {
245+ atomic_set(&mdev->pasid_1024_fake_error, 0);
246+ } else if (atomic_read(&mdev->pasid_1024_fake_error)) {
247+ /*
248+ * Clear the flag, and fake an error to fail the
249+ * replacement.
250+ */
251+ atomic_set(&mdev->pasid_1024_fake_error, 0);
252+ return -ENOMEM;
253+ } else {
254+ /* Set the flag to fake an error in next call */
255+ atomic_set(&mdev->pasid_1024_fake_error, 1);
256+ }
257+ }
258+
200259 return 0;
201260 }
202261
203262 static const struct iommu_domain_ops mock_blocking_ops = {
204263 .attach_dev = mock_domain_nop_attach,
264+ .set_dev_pasid = mock_domain_set_dev_pasid_nop
205265 };
206266
207267 static struct iommu_domain mock_blocking_domain = {
@@ -343,7 +403,7 @@ mock_domain_alloc_nested(struct device *dev, struct iommu_domain *parent, · 135 unmodified lines
343403 struct mock_iommu_domain_nested *mock_nested;
344404 struct mock_iommu_domain *mock_parent;
345405
346− if (flags)
406+ if (flags & ~IOMMU_HWPT_ALLOC_PASID)
347407 return ERR_PTR(-EOPNOTSUPP);
348408 if (!parent || parent->ops != mock_ops.default_domain_ops)
349409 return ERR_PTR(-EINVAL);
@@ -365,7 +425,8 @@ mock_domain_alloc_paging_flags(struct device *dev, u32 flags, · 15 unmodified lines
365425 {
366426 bool has_dirty_flag = flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING;
367427 const u32 PAGING_FLAGS = IOMMU_HWPT_ALLOC_DIRTY_TRACKING |
368− IOMMU_HWPT_ALLOC_NEST_PARENT;
428+ IOMMU_HWPT_ALLOC_NEST_PARENT |
429+ IOMMU_HWPT_ALLOC_PASID;
369430 struct mock_dev *mdev = to_mock_dev(dev);
370431 bool no_dirty_ops = mdev->flags & MOCK_FLAGS_DEVICE_NO_DIRTY;
371432 struct mock_iommu_domain *mock;
@@ -585,7 +646,7 @@ mock_viommu_alloc_domain_nested(struct iommufd_viommu *viommu, u32 flags, · 213 unmodified lines
585646 struct mock_viommu *mock_viommu = to_mock_viommu(viommu);
586647 struct mock_iommu_domain_nested *mock_nested;
587648
588− if (flags)
649+ if (flags & ~IOMMU_HWPT_ALLOC_PASID)
589650 return ERR_PTR(-EOPNOTSUPP);
590651
591652 mock_nested = __mock_domain_alloc_nested(user_data);
@@ -720,6 +781,7 @@ static const struct iommu_ops mock_ops = { · 128 unmodified lines
720781 .map_pages = mock_domain_map_pages,
721782 .unmap_pages = mock_domain_unmap_pages,
722783 .iova_to_phys = mock_domain_iova_to_phys,
784+ .set_dev_pasid = mock_domain_set_dev_pasid_nop,
723785 },
724786 };
725787
@@ -780,6 +842,7 @@ static struct iommu_domain_ops domain_nested_ops = { · 54 unmodified lines
780842 .free = mock_domain_free_nested,
781843 .attach_dev = mock_domain_nop_attach,
782844 .cache_invalidate_user = mock_domain_cache_invalidate_user,
845+ .set_dev_pasid = mock_domain_set_dev_pasid_nop,
783846 };
784847
785848 static inline struct iommufd_hw_pagetable *
@@ -839,17 +902,24 @@ static void mock_dev_release(struct device *dev) · 53 unmodified lines
839902
840903 static struct mock_dev *mock_dev_create(unsigned long dev_flags)
841904 {
905+ struct property_entry prop[] = {
906+ PROPERTY_ENTRY_U32("pasid-num-bits", 0),
907+ {},
908+ };
909+ const u32 valid_flags = MOCK_FLAGS_DEVICE_NO_DIRTY |
910+ MOCK_FLAGS_DEVICE_HUGE_IOVA |
911+ MOCK_FLAGS_DEVICE_PASID;
842912 struct mock_dev *mdev;
843913 int rc, i;
844914
845− if (dev_flags &
846− ~(MOCK_FLAGS_DEVICE_NO_DIRTY | MOCK_FLAGS_DEVICE_HUGE_IOVA))
915+ if (dev_flags & ~valid_flags)
847916 return ERR_PTR(-EINVAL);
848917
849918 mdev = kzalloc(sizeof(*mdev), GFP_KERNEL);
850919 if (!mdev)
851920 return ERR_PTR(-ENOMEM);
852921
922+ init_rwsem(&mdev->viommu_rwsem);
853923 device_initialize(&mdev->dev);
854924 mdev->flags = dev_flags;
855925 mdev->dev.release = mock_dev_release;
@@ -866,6 +936,15 @@ static struct mock_dev *mock_dev_create(unsigned long dev_flags) · 10 unmodified lines
866936 if (rc)
867937 goto err_put;
868938
939+ if (dev_flags & MOCK_FLAGS_DEVICE_PASID)
940+ prop[0] = PROPERTY_ENTRY_U32("pasid-num-bits", MOCK_PASID_WIDTH);
941+
942+ rc = device_create_managed_software_node(&mdev->dev, prop, NULL);
943+ if (rc) {
944+ dev_err(&mdev->dev, "add pasid-num-bits property failed, rc: %d", rc);
945+ goto err_put;
946+ }
947+
869948 rc = device_add(&mdev->dev);
870949 if (rc)
871950 goto err_put;
@@ -921,7 +1000,7 @@ static int iommufd_test_mock_domain(struct iommufd_ucmd *ucmd, · 49 unmodified lines
9211000 }
9221001 sobj->idev.idev = idev;
9231002
924− rc = iommufd_device_attach(idev, &pt_id);
1003+ rc = iommufd_device_attach(idev, IOMMU_NO_PASID, &pt_id);
9251004 if (rc)
9261005 goto out_unbind;
9271006
@@ -936,7 +1015,7 @@ static int iommufd_test_mock_domain(struct iommufd_ucmd *ucmd, · 8 unmodified lines
9361015 return 0;
9371016
9381017 out_detach:
939− iommufd_device_detach(idev);
1018+ iommufd_device_detach(idev, IOMMU_NO_PASID);
9401019 out_unbind:
9411020 iommufd_device_unbind(idev);
9421021 out_mdev:
@@ -946,39 +1025,49 @@ out_sobj: · 3 unmodified lines
9461025 return rc;
9471026 }
9481027
949−/* Replace the mock domain with a manually allocated hw_pagetable */
950−static int iommufd_test_mock_domain_replace(struct iommufd_ucmd *ucmd,
951− unsigned int device_id, u32 pt_id,
952− struct iommu_test_cmd *cmd)
1028+static struct selftest_obj *
1029+iommufd_test_get_selftest_obj(struct iommufd_ctx *ictx, u32 id)
9531030 {
9541031 struct iommufd_object *dev_obj;
9551032 struct selftest_obj *sobj;
956− int rc;
9571033
9581034 /*
9591035 * Prefer to use the OBJ_SELFTEST because the destroy_rwsem will ensure
9601036 * it doesn't race with detach, which is not allowed.
9611037 */
962− dev_obj =
963− iommufd_get_object(ucmd->ictx, device_id, IOMMUFD_OBJ_SELFTEST);
1038+ dev_obj = iommufd_get_object(ictx, id, IOMMUFD_OBJ_SELFTEST);
9641039 if (IS_ERR(dev_obj))
965− return PTR_ERR(dev_obj);
1040+ return ERR_CAST(dev_obj);
9661041
9671042 sobj = to_selftest_obj(dev_obj);
9681043 if (sobj->type != TYPE_IDEV) {
969− rc = -EINVAL;
970− goto out_dev_obj;
1044+ iommufd_put_object(ictx, dev_obj);
1045+ return ERR_PTR(-EINVAL);
9711046 }
1047+ return sobj;
1048+}
9721049
973− rc = iommufd_device_replace(sobj->idev.idev, &pt_id);
1050+/* Replace the mock domain with a manually allocated hw_pagetable */
1051+static int iommufd_test_mock_domain_replace(struct iommufd_ucmd *ucmd,
1052+ unsigned int device_id, u32 pt_id,
1053+ struct iommu_test_cmd *cmd)
1054+{
1055+ struct selftest_obj *sobj;
1056+ int rc;
1057+
1058+ sobj = iommufd_test_get_selftest_obj(ucmd->ictx, device_id);
1059+ if (IS_ERR(sobj))
1060+ return PTR_ERR(sobj);
1061+
1062+ rc = iommufd_device_replace(sobj->idev.idev, IOMMU_NO_PASID, &pt_id);
9741063 if (rc)
975− goto out_dev_obj;
1064+ goto out_sobj;
9761065
9771066 cmd->mock_domain_replace.pt_id = pt_id;
9781067 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
9791068
980−out_dev_obj:
981− iommufd_put_object(ucmd->ictx, dev_obj);
1069+out_sobj:
1070+ iommufd_put_object(ucmd->ictx, &sobj->obj);
9821071 return rc;
9831072 }
9841073
@@ -1597,13 +1686,166 @@ static int iommufd_test_trigger_iopf(struct iommufd_ucmd *ucmd, · 612 unmodified lines
15971686 return 0;
15981687 }
15991688
1689+static int iommufd_test_trigger_vevent(struct iommufd_ucmd *ucmd,
1690+ struct iommu_test_cmd *cmd)
1691+{
1692+ struct iommu_viommu_event_selftest test = {};
1693+ struct iommufd_device *idev;
1694+ struct mock_dev *mdev;
1695+ int rc = -ENOENT;
1696+
1697+ idev = iommufd_get_device(ucmd, cmd->trigger_vevent.dev_id);
1698+ if (IS_ERR(idev))
1699+ return PTR_ERR(idev);
1700+ mdev = to_mock_dev(idev->dev);
1701+
1702+ down_read(&mdev->viommu_rwsem);
1703+ if (!mdev->viommu || !mdev->vdev_id)
1704+ goto out_unlock;
1705+
1706+ test.virt_id = mdev->vdev_id;
1707+ rc = iommufd_viommu_report_event(&mdev->viommu->core,
1708+ IOMMU_VEVENTQ_TYPE_SELFTEST, &test,
1709+ sizeof(test));
1710+out_unlock:
1711+ up_read(&mdev->viommu_rwsem);
1712+ iommufd_put_object(ucmd->ictx, &idev->obj);
1713+
1714+ return rc;
1715+}
1716+
1717+static inline struct iommufd_hw_pagetable *
1718+iommufd_get_hwpt(struct iommufd_ucmd *ucmd, u32 id)
1719+{
1720+ struct iommufd_object *pt_obj;
1721+
1722+ pt_obj = iommufd_get_object(ucmd->ictx, id, IOMMUFD_OBJ_ANY);
1723+ if (IS_ERR(pt_obj))
1724+ return ERR_CAST(pt_obj);
1725+
1726+ if (pt_obj->type != IOMMUFD_OBJ_HWPT_NESTED &&
1727+ pt_obj->type != IOMMUFD_OBJ_HWPT_PAGING) {
1728+ iommufd_put_object(ucmd->ictx, pt_obj);
1729+ return ERR_PTR(-EINVAL);
1730+ }
1731+
1732+ return container_of(pt_obj, struct iommufd_hw_pagetable, obj);
1733+}
1734+
1735+static int iommufd_test_pasid_check_hwpt(struct iommufd_ucmd *ucmd,
1736+ struct iommu_test_cmd *cmd)
1737+{
1738+ u32 hwpt_id = cmd->pasid_check.hwpt_id;
1739+ struct iommu_domain *attached_domain;
1740+ struct iommu_attach_handle *handle;
1741+ struct iommufd_hw_pagetable *hwpt;
1742+ struct selftest_obj *sobj;
1743+ struct mock_dev *mdev;
1744+ int rc = 0;
1745+
1746+ sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id);
1747+ if (IS_ERR(sobj))
1748+ return PTR_ERR(sobj);
1749+
1750+ mdev = sobj->idev.mock_dev;
1751+
1752+ handle = iommu_attach_handle_get(mdev->dev.iommu_group,
1753+ cmd->pasid_check.pasid, 0);
1754+ if (IS_ERR(handle))
1755+ attached_domain = NULL;
1756+ else
1757+ attached_domain = handle->domain;
1758+
1759+ /* hwpt_id == 0 means to check if pasid is detached */
1760+ if (!hwpt_id) {
1761+ if (attached_domain)
1762+ rc = -EINVAL;
1763+ goto out_sobj;
1764+ }
1765+
1766+ hwpt = iommufd_get_hwpt(ucmd, hwpt_id);
1767+ if (IS_ERR(hwpt)) {
1768+ rc = PTR_ERR(hwpt);
1769+ goto out_sobj;
1770+ }
1771+
1772+ if (attached_domain != hwpt->domain)
1773+ rc = -EINVAL;
1774+
1775+ iommufd_put_object(ucmd->ictx, &hwpt->obj);
1776+out_sobj:
1777+ iommufd_put_object(ucmd->ictx, &sobj->obj);
1778+ return rc;
1779+}
1780+
1781+static int iommufd_test_pasid_attach(struct iommufd_ucmd *ucmd,
1782+ struct iommu_test_cmd *cmd)
1783+{
1784+ struct selftest_obj *sobj;
1785+ int rc;
1786+
1787+ sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id);
1788+ if (IS_ERR(sobj))
1789+ return PTR_ERR(sobj);
1790+
1791+ rc = iommufd_device_attach(sobj->idev.idev, cmd->pasid_attach.pasid,
1792+ &cmd->pasid_attach.pt_id);
1793+ if (rc)
1794+ goto out_sobj;
1795+
1796+ rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
1797+ if (rc)
1798+ iommufd_device_detach(sobj->idev.idev,
1799+ cmd->pasid_attach.pasid);
1800+
1801+out_sobj:
1802+ iommufd_put_object(ucmd->ictx, &sobj->obj);
1803+ return rc;
1804+}
1805+
1806+static int iommufd_test_pasid_replace(struct iommufd_ucmd *ucmd,
1807+ struct iommu_test_cmd *cmd)
1808+{
1809+ struct selftest_obj *sobj;
1810+ int rc;
1811+
1812+ sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id);
1813+ if (IS_ERR(sobj))
1814+ return PTR_ERR(sobj);
1815+
1816+ rc = iommufd_device_replace(sobj->idev.idev, cmd->pasid_attach.pasid,
1817+ &cmd->pasid_attach.pt_id);
1818+ if (rc)
1819+ goto out_sobj;
1820+
1821+ rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
1822+
1823+out_sobj:
1824+ iommufd_put_object(ucmd->ictx, &sobj->obj);
1825+ return rc;
1826+}
1827+
1828+static int iommufd_test_pasid_detach(struct iommufd_ucmd *ucmd,
1829+ struct iommu_test_cmd *cmd)
1830+{
1831+ struct selftest_obj *sobj;
1832+
1833+ sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id);
1834+ if (IS_ERR(sobj))
1835+ return PTR_ERR(sobj);
1836+
1837+ iommufd_device_detach(sobj->idev.idev, cmd->pasid_detach.pasid);
1838+ iommufd_put_object(ucmd->ictx, &sobj->obj);
1839+ return 0;
1840+}
1841+
16001842 void iommufd_selftest_destroy(struct iommufd_object *obj)
16011843 {
16021844 struct selftest_obj *sobj = to_selftest_obj(obj);
16031845
16041846 switch (sobj->type) {
16051847 case TYPE_IDEV:
1606− iommufd_device_detach(sobj->idev.idev);
1848+ iommufd_device_detach(sobj->idev.idev, IOMMU_NO_PASID);
16071849 iommufd_device_unbind(sobj->idev.idev);
16081850 mock_dev_destroy(sobj->idev.mock_dev);
16091851 break;
@@ -1678,6 +1920,16 @@ int iommufd_test(struct iommufd_ucmd *ucmd) · 68 unmodified lines
16781920 cmd->dirty.flags);
16791921 case IOMMU_TEST_OP_TRIGGER_IOPF:
16801922 return iommufd_test_trigger_iopf(ucmd, cmd);
1923+ case IOMMU_TEST_OP_TRIGGER_VEVENT:
1924+ return iommufd_test_trigger_vevent(ucmd, cmd);
1925+ case IOMMU_TEST_OP_PASID_ATTACH:
1926+ return iommufd_test_pasid_attach(ucmd, cmd);
1927+ case IOMMU_TEST_OP_PASID_REPLACE:
1928+ return iommufd_test_pasid_replace(ucmd, cmd);
1929+ case IOMMU_TEST_OP_PASID_DETACH:
1930+ return iommufd_test_pasid_detach(ucmd, cmd);
1931+ case IOMMU_TEST_OP_PASID_CHECK_HWPT:
1932+ return iommufd_test_pasid_check_hwpt(ucmd, cmd);
16811933 default:
16821934 return -EOPNOTSUPP;
16831935 }
@@ -1724,6 +1976,7 @@ int __init iommufd_test_init(void) · 40 unmodified lines
17241976 init_completion(&mock_iommu.complete);
17251977
17261978 mock_iommu_iopf_queue = iopf_queue_alloc("mock-iopfq");
1979+ mock_iommu.iommu_dev.max_pasids = (1 << MOCK_PASID_WIDTH);
17271980
17281981 return 0;
17291982
drivers/iommu/iommufd/viommu.c+2−0
@@ -59,6 +59,8 @@ int iommufd_viommu_alloc_ioctl(struct iommufd_ucmd *ucmd) · 58 unmodified lines
5959 viommu->ictx = ucmd->ictx;
6060 viommu->hwpt = hwpt_paging;
6161 refcount_inc(&viommu->hwpt->common.obj.users);
62+ INIT_LIST_HEAD(&viommu->veventqs);
63+ init_rwsem(&viommu->veventqs_rwsem);
6264 /*
6365 * It is the most likely case that a physical IOMMU is unpluggable. A
6466 * pluggable IOMMU instance (if exists) is responsible for refcounting
drivers/pci/ats.c+33−0
@@ -538,4 +538,37 @@ int pci_max_pasids(struct pci_dev *pdev) · 537 unmodified lines
538538 return (1 << FIELD_GET(PCI_PASID_CAP_WIDTH, supported));
539539 }
540540 EXPORT_SYMBOL_GPL(pci_max_pasids);
541+
542+/**
543+ * pci_pasid_status - Check the PASID status
544+ * @pdev: PCI device structure
545+ *
546+ * Returns a negative value when no PASID capability is present.
547+ * Otherwise the value of the control register is returned.
548+ * Status reported are:
549+ *
550+ * PCI_PASID_CTRL_ENABLE - PASID enabled
551+ * PCI_PASID_CTRL_EXEC - Execute permission enabled
552+ * PCI_PASID_CTRL_PRIV - Privileged mode enabled
553+ */
554+int pci_pasid_status(struct pci_dev *pdev)
555+{
556+ int pasid;
557+ u16 ctrl;
558+
559+ if (pdev->is_virtfn)
560+ pdev = pci_physfn(pdev);
561+
562+ pasid = pdev->pasid_cap;
563+ if (!pasid)
564+ return -EINVAL;
565+
566+ pci_read_config_word(pdev, pasid + PCI_PASID_CTRL, &ctrl);
567+
568+ ctrl &= PCI_PASID_CTRL_ENABLE | PCI_PASID_CTRL_EXEC |
569+ PCI_PASID_CTRL_PRIV;
570+
571+ return ctrl;
572+}
573+EXPORT_SYMBOL_GPL(pci_pasid_status);
541574 #endif /* CONFIG_PCI_PASID */
drivers/vfio/device_cdev.c+52−8
@@ -162,9 +162,9 @@ void vfio_df_unbind_iommufd(struct vfio_device_file *df) · 161 unmodified lines
162162 int vfio_df_ioctl_attach_pt(struct vfio_device_file *df,
163163 struct vfio_device_attach_iommufd_pt __user *arg)
164164 {
165− struct vfio_device *device = df->device;
166165 struct vfio_device_attach_iommufd_pt attach;
167− unsigned long minsz;
166+ struct vfio_device *device = df->device;
167+ unsigned long minsz, xend = 0;
168168 int ret;
169169
170170 minsz = offsetofend(struct vfio_device_attach_iommufd_pt, pt_id);
@@ -172,11 +172,34 @@ int vfio_df_ioctl_attach_pt(struct vfio_device_file *df, · 1 unmodified line
172172 if (copy_from_user(&attach, arg, minsz))
173173 return -EFAULT;
174174
175− if (attach.argsz < minsz || attach.flags)
175+ if (attach.argsz < minsz)
176176 return -EINVAL;
177177
178+ if (attach.flags & ~VFIO_DEVICE_ATTACH_PASID)
179+ return -EINVAL;
180+
181+ if (attach.flags & VFIO_DEVICE_ATTACH_PASID) {
182+ if (!device->ops->pasid_attach_ioas)
183+ return -EOPNOTSUPP;
184+ xend = offsetofend(struct vfio_device_attach_iommufd_pt, pasid);
185+ }
186+
187+ if (xend) {
188+ if (attach.argsz < xend)
189+ return -EINVAL;
190+
191+ if (copy_from_user((void *)&attach + minsz,
192+ (void __user *)arg + minsz, xend - minsz))
193+ return -EFAULT;
194+ }
195+
178196 mutex_lock(&device->dev_set->lock);
179− ret = device->ops->attach_ioas(device, &attach.pt_id);
197+ if (attach.flags & VFIO_DEVICE_ATTACH_PASID)
198+ ret = device->ops->pasid_attach_ioas(device,
199+ attach.pasid,
200+ &attach.pt_id);
201+ else
202+ ret = device->ops->attach_ioas(device, &attach.pt_id);
180203 if (ret)
181204 goto out_unlock;
182205
@@ -198,20 +221,41 @@ out_unlock: · 15 unmodified lines
198221 int vfio_df_ioctl_detach_pt(struct vfio_device_file *df,
199222 struct vfio_device_detach_iommufd_pt __user *arg)
200223 {
201− struct vfio_device *device = df->device;
202224 struct vfio_device_detach_iommufd_pt detach;
203− unsigned long minsz;
225+ struct vfio_device *device = df->device;
226+ unsigned long minsz, xend = 0;
204227
205228 minsz = offsetofend(struct vfio_device_detach_iommufd_pt, flags);
206229
207230 if (copy_from_user(&detach, arg, minsz))
208231 return -EFAULT;
209232
210− if (detach.argsz < minsz || detach.flags)
233+ if (detach.argsz < minsz)
211234 return -EINVAL;
212235
236+ if (detach.flags & ~VFIO_DEVICE_DETACH_PASID)
237+ return -EINVAL;
238+
239+ if (detach.flags & VFIO_DEVICE_DETACH_PASID) {
240+ if (!device->ops->pasid_detach_ioas)
241+ return -EOPNOTSUPP;
242+ xend = offsetofend(struct vfio_device_detach_iommufd_pt, pasid);
243+ }
244+
245+ if (xend) {
246+ if (detach.argsz < xend)
247+ return -EINVAL;
248+
249+ if (copy_from_user((void *)&detach + minsz,
250+ (void __user *)arg + minsz, xend - minsz))
251+ return -EFAULT;
252+ }
253+
213254 mutex_lock(&device->dev_set->lock);
214− device->ops->detach_ioas(device);
255+ if (detach.flags & VFIO_DEVICE_DETACH_PASID)
256+ device->ops->pasid_detach_ioas(device, detach.pasid);
257+ else
258+ device->ops->detach_ioas(device);
215259 mutex_unlock(&device->dev_set->lock);
216260
217261 return 0;
drivers/vfio/iommufd.c+56−4
@@ -119,16 +119,24 @@ int vfio_iommufd_physical_bind(struct vfio_device *vdev, · 118 unmodified lines
119119 if (IS_ERR(idev))
120120 return PTR_ERR(idev);
121121 vdev->iommufd_device = idev;
122+ ida_init(&vdev->pasids);
122123 return 0;
123124 }
124125 EXPORT_SYMBOL_GPL(vfio_iommufd_physical_bind);
125126
126127 void vfio_iommufd_physical_unbind(struct vfio_device *vdev)
127128 {
129+ int pasid;
130+
128131 lockdep_assert_held(&vdev->dev_set->lock);
129132
133+ while ((pasid = ida_find_first(&vdev->pasids)) >= 0) {
134+ iommufd_device_detach(vdev->iommufd_device, pasid);
135+ ida_free(&vdev->pasids, pasid);
136+ }
137+
130138 if (vdev->iommufd_attached) {
131− iommufd_device_detach(vdev->iommufd_device);
139+ iommufd_device_detach(vdev->iommufd_device, IOMMU_NO_PASID);
132140 vdev->iommufd_attached = false;
133141 }
134142 iommufd_device_unbind(vdev->iommufd_device);
@@ -146,9 +154,11 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id) · 11 unmodified lines
146154 return -EINVAL;
147155
148156 if (vdev->iommufd_attached)
149− rc = iommufd_device_replace(vdev->iommufd_device, pt_id);
157+ rc = iommufd_device_replace(vdev->iommufd_device,
158+ IOMMU_NO_PASID, pt_id);
150159 else
151− rc = iommufd_device_attach(vdev->iommufd_device, pt_id);
160+ rc = iommufd_device_attach(vdev->iommufd_device,
161+ IOMMU_NO_PASID, pt_id);
152162 if (rc)
153163 return rc;
154164 vdev->iommufd_attached = true;
@@ -163,11 +173,53 @@ void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev) · 8 unmodified lines
163173 if (WARN_ON(!vdev->iommufd_device) || !vdev->iommufd_attached)
164174 return;
165175
166− iommufd_device_detach(vdev->iommufd_device);
176+ iommufd_device_detach(vdev->iommufd_device, IOMMU_NO_PASID);
167177 vdev->iommufd_attached = false;
168178 }
169179 EXPORT_SYMBOL_GPL(vfio_iommufd_physical_detach_ioas);
170180
181+int vfio_iommufd_physical_pasid_attach_ioas(struct vfio_device *vdev,
182+ u32 pasid, u32 *pt_id)
183+{
184+ int rc;
185+
186+ lockdep_assert_held(&vdev->dev_set->lock);
187+
188+ if (WARN_ON(!vdev->iommufd_device))
189+ return -EINVAL;
190+
191+ if (ida_exists(&vdev->pasids, pasid))
192+ return iommufd_device_replace(vdev->iommufd_device,
193+ pasid, pt_id);
194+
195+ rc = ida_alloc_range(&vdev->pasids, pasid, pasid, GFP_KERNEL);
196+ if (rc < 0)
197+ return rc;
198+
199+ rc = iommufd_device_attach(vdev->iommufd_device, pasid, pt_id);
200+ if (rc)
201+ ida_free(&vdev->pasids, pasid);
202+
203+ return rc;
204+}
205+EXPORT_SYMBOL_GPL(vfio_iommufd_physical_pasid_attach_ioas);
206+
207+void vfio_iommufd_physical_pasid_detach_ioas(struct vfio_device *vdev,
208+ u32 pasid)
209+{
210+ lockdep_assert_held(&vdev->dev_set->lock);
211+
212+ if (WARN_ON(!vdev->iommufd_device))
213+ return;
214+
215+ if (!ida_exists(&vdev->pasids, pasid))
216+ return;
217+
218+ iommufd_device_detach(vdev->iommufd_device, pasid);
219+ ida_free(&vdev->pasids, pasid);
220+}
221+EXPORT_SYMBOL_GPL(vfio_iommufd_physical_pasid_detach_ioas);
222+
171223 /*
172224 * The emulated standard ops mean that vfio_device is going to use the
173225 * "mdev path" and will call vfio_pin_pages()/vfio_dma_rw(). Drivers using this
drivers/vfio/pci/vfio_pci.c+2−0
@@ -144,6 +144,8 @@ static const struct vfio_device_ops vfio_pci_ops = { · 143 unmodified lines
144144 .unbind_iommufd = vfio_iommufd_physical_unbind,
145145 .attach_ioas = vfio_iommufd_physical_attach_ioas,
146146 .detach_ioas = vfio_iommufd_physical_detach_ioas,
147+ .pasid_attach_ioas = vfio_iommufd_physical_pasid_attach_ioas,
148+ .pasid_detach_ioas = vfio_iommufd_physical_pasid_detach_ioas,
147149 };
148150
149151 static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
include/linux/idr.h+11−0
@@ -274,6 +274,7 @@ struct ida { · 273 unmodified lines
274274 int ida_alloc_range(struct ida *, unsigned int min, unsigned int max, gfp_t);
275275 void ida_free(struct ida *, unsigned int id);
276276 void ida_destroy(struct ida *ida);
277+int ida_find_first_range(struct ida *ida, unsigned int min, unsigned int max);
277278
278279 /**
279280 * ida_alloc() - Allocate an unused ID.
@@ -345,4 +346,14 @@ static inline bool ida_is_empty(const struct ida *ida) · 65 unmodified lines
345346 {
346347 return xa_empty(&ida->xa);
347348 }
349+
350+static inline bool ida_exists(struct ida *ida, unsigned int id)
351+{
352+ return ida_find_first_range(ida, id, id) == id;
353+}
354+
355+static inline int ida_find_first(struct ida *ida)
356+{
357+ return ida_find_first_range(ida, 0, ~0);
358+}
348359 #endif /* __IDR_H__ */
include/linux/iommu.h+15−20
@@ -41,6 +41,7 @@ struct iommu_dirty_ops; · 40 unmodified lines
4141 struct notifier_block;
4242 struct iommu_sva;
4343 struct iommu_dma_cookie;
44+struct iommu_dma_msi_cookie;
4445 struct iommu_fault_param;
4546 struct iommufd_ctx;
4647 struct iommufd_viommu;
@@ -165,6 +166,15 @@ struct iommu_domain_geometry { · 118 unmodified lines
165166 bool force_aperture; /* DMA only allowed in mappable range? */
166167 };
167168
169+enum iommu_domain_cookie_type {
170+ IOMMU_COOKIE_NONE,
171+ IOMMU_COOKIE_DMA_IOVA,
172+ IOMMU_COOKIE_DMA_MSI,
173+ IOMMU_COOKIE_FAULT_HANDLER,
174+ IOMMU_COOKIE_SVA,
175+ IOMMU_COOKIE_IOMMUFD,
176+};
177+
168178 /* Domain feature flags */
169179 #define __IOMMU_DOMAIN_PAGING (1U << 0) /* Support for iommu_map/unmap */
170180 #define __IOMMU_DOMAIN_DMA_API (1U << 1) /* Domain for use in DMA-API
@@ -211,23 +221,18 @@ struct iommu_domain_geometry { · 40 unmodified lines
211221
212222 struct iommu_domain {
213223 unsigned type;
224+ enum iommu_domain_cookie_type cookie_type;
214225 const struct iommu_domain_ops *ops;
215226 const struct iommu_dirty_ops *dirty_ops;
216227 const struct iommu_ops *owner; /* Whose domain_alloc we came from */
217228 unsigned long pgsize_bitmap; /* Bitmap of page sizes in use */
218229 struct iommu_domain_geometry geometry;
219− struct iommu_dma_cookie *iova_cookie;
220230 int (*iopf_handler)(struct iopf_group *group);
221231
222−#if IS_ENABLED(CONFIG_IRQ_MSI_IOMMU)
223− int (*sw_msi)(struct iommu_domain *domain, struct msi_desc *desc,
224− phys_addr_t msi_addr);
225−#endif
226−
227− union { /* Pointer usable by owner of the domain */
228− struct iommufd_hw_pagetable *iommufd_hwpt; /* iommufd */
229− };
230− union { /* Fault handler */
232+ union { /* cookie */
233+ struct iommu_dma_cookie *iova_cookie;
234+ struct iommu_dma_msi_cookie *msi_cookie;
235+ struct iommufd_hw_pagetable *iommufd_hwpt;
231236 struct {
232237 iommu_fault_handler_t handler;
233238 void *handler_token;
@@ -244,16 +249,6 @@ struct iommu_domain { · 10 unmodified lines
244249 };
245250 };
246251
247−static inline void iommu_domain_set_sw_msi(
248− struct iommu_domain *domain,
249− int (*sw_msi)(struct iommu_domain *domain, struct msi_desc *desc,
250− phys_addr_t msi_addr))
251−{
252−#if IS_ENABLED(CONFIG_IRQ_MSI_IOMMU)
253− domain->sw_msi = sw_msi;
254−#endif
255−}
256−
257252 static inline bool iommu_is_dma_domain(struct iommu_domain *domain)
258253 {
259254 return domain->type & __IOMMU_DOMAIN_DMA_API;
include/linux/iommufd.h+29−3
@@ -8,9 +8,11 @@ · 7 unmodified lines
88
99 #include <linux/err.h>
1010 #include <linux/errno.h>
11+#include <linux/iommu.h>
1112 #include <linux/refcount.h>
1213 #include <linux/types.h>
1314 #include <linux/xarray.h>
15+#include <uapi/linux/iommufd.h>
1416
1517 struct device;
1618 struct file;
@@ -34,6 +36,7 @@ enum iommufd_object_type { · 17 unmodified lines
3436 IOMMUFD_OBJ_FAULT,
3537 IOMMUFD_OBJ_VIOMMU,
3638 IOMMUFD_OBJ_VDEVICE,
39+ IOMMUFD_OBJ_VEVENTQ,
3740 #ifdef CONFIG_IOMMUFD_TEST
3841 IOMMUFD_OBJ_SELFTEST,
3942 #endif
@@ -52,9 +55,11 @@ struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, · 12 unmodified lines
5255 struct device *dev, u32 *id);
5356 void iommufd_device_unbind(struct iommufd_device *idev);
5457
55−int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id);
56−int iommufd_device_replace(struct iommufd_device *idev, u32 *pt_id);
57−void iommufd_device_detach(struct iommufd_device *idev);
58+int iommufd_device_attach(struct iommufd_device *idev, ioasid_t pasid,
59+ u32 *pt_id);
60+int iommufd_device_replace(struct iommufd_device *idev, ioasid_t pasid,
61+ u32 *pt_id);
62+void iommufd_device_detach(struct iommufd_device *idev, ioasid_t pasid);
5863
5964 struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev);
6065 u32 iommufd_device_to_id(struct iommufd_device *idev);
@@ -93,6 +98,8 @@ struct iommufd_viommu { · 32 unmodified lines
9398 const struct iommufd_viommu_ops *ops;
9499
95100 struct xarray vdevs;
101+ struct list_head veventqs;
102+ struct rw_semaphore veventqs_rwsem;
96103
97104 unsigned int type;
98105 };
@@ -187,6 +194,11 @@ struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx, · 88 unmodified lines
187194 enum iommufd_object_type type);
188195 struct device *iommufd_viommu_find_dev(struct iommufd_viommu *viommu,
189196 unsigned long vdev_id);
197+int iommufd_viommu_get_vdev_id(struct iommufd_viommu *viommu,
198+ struct device *dev, unsigned long *vdev_id);
199+int iommufd_viommu_report_event(struct iommufd_viommu *viommu,
200+ enum iommu_veventq_type type, void *event_data,
201+ size_t data_len);
190202 #else /* !CONFIG_IOMMUFD_DRIVER_CORE */
191203 static inline struct iommufd_object *
192204 _iommufd_object_alloc(struct iommufd_ctx *ictx, size_t size,
@@ -200,6 +212,20 @@ iommufd_viommu_find_dev(struct iommufd_viommu *viommu, unsigned long vdev_id) · 7 unmodified lines
200212 {
201213 return NULL;
202214 }
215+
216+static inline int iommufd_viommu_get_vdev_id(struct iommufd_viommu *viommu,
217+ struct device *dev,
218+ unsigned long *vdev_id)
219+{
220+ return -ENOENT;
221+}
222+
223+static inline int iommufd_viommu_report_event(struct iommufd_viommu *viommu,
224+ enum iommu_veventq_type type,
225+ void *event_data, size_t data_len)
226+{
227+ return -EOPNOTSUPP;
228+}
203229 #endif /* CONFIG_IOMMUFD_DRIVER_CORE */
204230
205231 /*
include/linux/pci-ats.h+3−0
@@ -42,6 +42,7 @@ int pci_enable_pasid(struct pci_dev *pdev, int features); · 41 unmodified lines
4242 void pci_disable_pasid(struct pci_dev *pdev);
4343 int pci_pasid_features(struct pci_dev *pdev);
4444 int pci_max_pasids(struct pci_dev *pdev);
45+int pci_pasid_status(struct pci_dev *pdev);
4546 #else /* CONFIG_PCI_PASID */
4647 static inline int pci_enable_pasid(struct pci_dev *pdev, int features)
4748 { return -EINVAL; }
@@ -50,6 +51,8 @@ static inline int pci_pasid_features(struct pci_dev *pdev) · 2 unmodified lines
5051 { return -EINVAL; }
5152 static inline int pci_max_pasids(struct pci_dev *pdev)
5253 { return -EINVAL; }
54+static inline int pci_pasid_status(struct pci_dev *pdev)
55+{ return -EINVAL; }
5356 #endif /* CONFIG_PCI_PASID */
5457
5558 #endif /* LINUX_PCI_ATS_H */
include/linux/vfio.h+14−0
@@ -67,6 +67,7 @@ struct vfio_device { · 66 unmodified lines
6767 struct inode *inode;
6868 #if IS_ENABLED(CONFIG_IOMMUFD)
6969 struct iommufd_device *iommufd_device;
70+ struct ida pasids;
7071 u8 iommufd_attached:1;
7172 #endif
7273 u8 cdev_opened:1;
@@ -91,6 +92,8 @@ struct vfio_device { · 18 unmodified lines
9192 * bound iommufd. Undo in unbind_iommufd if @detach_ioas is not
9293 * called.
9394 * @detach_ioas: Opposite of attach_ioas
95+ * @pasid_attach_ioas: The pasid variation of attach_ioas
96+ * @pasid_detach_ioas: Opposite of pasid_attach_ioas
9497 * @open_device: Called when the first file descriptor is opened for this device
9598 * @close_device: Opposite of open_device
9699 * @read: Perform read(2) on device file descriptor
@@ -115,6 +118,9 @@ struct vfio_device_ops { · 18 unmodified lines
115118 void (*unbind_iommufd)(struct vfio_device *vdev);
116119 int (*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);
117120 void (*detach_ioas)(struct vfio_device *vdev);
121+ int (*pasid_attach_ioas)(struct vfio_device *vdev, u32 pasid,
122+ u32 *pt_id);
123+ void (*pasid_detach_ioas)(struct vfio_device *vdev, u32 pasid);
118124 int (*open_device)(struct vfio_device *vdev);
119125 void (*close_device)(struct vfio_device *vdev);
120126 ssize_t (*read)(struct vfio_device *vdev, char __user *buf,
@@ -139,6 +145,10 @@ int vfio_iommufd_physical_bind(struct vfio_device *vdev, · 18 unmodified lines
139145 void vfio_iommufd_physical_unbind(struct vfio_device *vdev);
140146 int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
141147 void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev);
148+int vfio_iommufd_physical_pasid_attach_ioas(struct vfio_device *vdev,
149+ u32 pasid, u32 *pt_id);
150+void vfio_iommufd_physical_pasid_detach_ioas(struct vfio_device *vdev,
151+ u32 pasid);
142152 int vfio_iommufd_emulated_bind(struct vfio_device *vdev,
143153 struct iommufd_ctx *ictx, u32 *out_device_id);
144154 void vfio_iommufd_emulated_unbind(struct vfio_device *vdev);
@@ -166,6 +176,10 @@ vfio_iommufd_get_dev_id(struct vfio_device *vdev, struct iommufd_ctx *ictx) · 21 unmodified lines
166176 ((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
167177 #define vfio_iommufd_physical_detach_ioas \
168178 ((void (*)(struct vfio_device *vdev)) NULL)
179+#define vfio_iommufd_physical_pasid_attach_ioas \
180+ ((int (*)(struct vfio_device *vdev, u32 pasid, u32 *pt_id)) NULL)
181+#define vfio_iommufd_physical_pasid_detach_ioas \
182+ ((void (*)(struct vfio_device *vdev, u32 pasid)) NULL)
169183 #define vfio_iommufd_emulated_bind \
170184 ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \
171185 u32 *out_device_id)) NULL)
include/uapi/linux/iommufd.h+128−1
@@ -55,6 +55,7 @@ enum { · 54 unmodified lines
5555 IOMMUFD_CMD_VIOMMU_ALLOC = 0x90,
5656 IOMMUFD_CMD_VDEVICE_ALLOC = 0x91,
5757 IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92,
58+ IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93,
5859 };
5960
6061 /**
@@ -392,6 +393,9 @@ struct iommu_vfio_ioas { · 331 unmodified lines
392393 * Any domain attached to the non-PASID part of the
393394 * device must also be flagged, otherwise attaching a
394395 * PASID will blocked.
396+ * For the user that wants to attach PASID, ioas is
397+ * not recommended for both the non-PASID part
398+ * and PASID part of the device.
395399 * If IOMMU does not support PASID it will return
396400 * error (-EOPNOTSUPP).
397401 */
@@ -608,9 +612,17 @@ enum iommu_hw_info_type { · 210 unmodified lines
608612 * IOMMU_HWPT_GET_DIRTY_BITMAP
609613 * IOMMU_HWPT_SET_DIRTY_TRACKING
610614 *
615+ * @IOMMU_HW_CAP_PCI_PASID_EXEC: Execute Permission Supported, user ignores it
616+ * when the struct
617+ * iommu_hw_info::out_max_pasid_log2 is zero.
618+ * @IOMMU_HW_CAP_PCI_PASID_PRIV: Privileged Mode Supported, user ignores it
619+ * when the struct
620+ * iommu_hw_info::out_max_pasid_log2 is zero.
611621 */
612622 enum iommufd_hw_capabilities {
613623 IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0,
624+ IOMMU_HW_CAP_PCI_PASID_EXEC = 1 << 1,
625+ IOMMU_HW_CAP_PCI_PASID_PRIV = 1 << 2,
614626 };
615627
616628 /**
@@ -626,6 +638,9 @@ enum iommufd_hw_capabilities { · 9 unmodified lines
626638 * iommu_hw_info_type.
627639 * @out_capabilities: Output the generic iommu capability info type as defined
628640 * in the enum iommu_hw_capabilities.
641+ * @out_max_pasid_log2: Output the width of PASIDs. 0 means no PASID support.
642+ * PCI devices turn to out_capabilities to check if the
643+ * specific capabilities is supported or not.
629644 * @__reserved: Must be 0
630645 *
631646 * Query an iommu type specific hardware information data from an iommu behind
@@ -649,7 +664,8 @@ struct iommu_hw_info { · 17 unmodified lines
649664 __u32 data_len;
650665 __aligned_u64 data_uptr;
651666 __u32 out_data_type;
652− __u32 __reserved;
667+ __u8 out_max_pasid_log2;
668+ __u8 __reserved[3];
653669 __aligned_u64 out_capabilities;
654670 };
655671 #define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO)
@@ -1014,4 +1030,115 @@ struct iommu_ioas_change_process { · 358 unmodified lines
10141030 #define IOMMU_IOAS_CHANGE_PROCESS \
10151031 _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS)
10161032
1033+/**
1034+ * enum iommu_veventq_flag - flag for struct iommufd_vevent_header
1035+ * @IOMMU_VEVENTQ_FLAG_LOST_EVENTS: vEVENTQ has lost vEVENTs
1036+ */
1037+enum iommu_veventq_flag {
1038+ IOMMU_VEVENTQ_FLAG_LOST_EVENTS = (1U << 0),
1039+};
1040+
1041+/**
1042+ * struct iommufd_vevent_header - Virtual Event Header for a vEVENTQ Status
1043+ * @flags: Combination of enum iommu_veventq_flag
1044+ * @sequence: The sequence index of a vEVENT in the vEVENTQ, with a range of
1045+ * [0, INT_MAX] where the following index of INT_MAX is 0
1046+ *
1047+ * Each iommufd_vevent_header reports a sequence index of the following vEVENT:
1048+ *
1049+ * +----------------------+-------+----------------------+-------+---+-------+
1050+ * | header0 {sequence=0} | data0 | header1 {sequence=1} | data1 |...| dataN |
1051+ * +----------------------+-------+----------------------+-------+---+-------+
1052+ *
1053+ * And this sequence index is expected to be monotonic to the sequence index of
1054+ * the previous vEVENT. If two adjacent sequence indexes has a delta larger than
1055+ * 1, it means that delta - 1 number of vEVENTs has lost, e.g. two lost vEVENTs:
1056+ *
1057+ * +-----+----------------------+-------+----------------------+-------+-----+
1058+ * | ... | header3 {sequence=3} | data3 | header6 {sequence=6} | data6 | ... |
1059+ * +-----+----------------------+-------+----------------------+-------+-----+
1060+ *
1061+ * If a vEVENT lost at the tail of the vEVENTQ and there is no following vEVENT
1062+ * providing the next sequence index, an IOMMU_VEVENTQ_FLAG_LOST_EVENTS header
1063+ * would be added to the tail, and no data would follow this header:
1064+ *
1065+ * +--+----------------------+-------+-----------------------------------------+
1066+ * |..| header3 {sequence=3} | data3 | header4 {flags=LOST_EVENTS, sequence=4} |
1067+ * +--+----------------------+-------+-----------------------------------------+
1068+ */
1069+struct iommufd_vevent_header {
1070+ __u32 flags;
1071+ __u32 sequence;
1072+};
1073+
1074+/**
1075+ * enum iommu_veventq_type - Virtual Event Queue Type
1076+ * @IOMMU_VEVENTQ_TYPE_DEFAULT: Reserved for future use
1077+ * @IOMMU_VEVENTQ_TYPE_ARM_SMMUV3: ARM SMMUv3 Virtual Event Queue
1078+ */
1079+enum iommu_veventq_type {
1080+ IOMMU_VEVENTQ_TYPE_DEFAULT = 0,
1081+ IOMMU_VEVENTQ_TYPE_ARM_SMMUV3 = 1,
1082+};
1083+
1084+/**
1085+ * struct iommu_vevent_arm_smmuv3 - ARM SMMUv3 Virtual Event
1086+ * (IOMMU_VEVENTQ_TYPE_ARM_SMMUV3)
1087+ * @evt: 256-bit ARM SMMUv3 Event record, little-endian.
1088+ * Reported event records: (Refer to "7.3 Event records" in SMMUv3 HW Spec)
1089+ * - 0x04 C_BAD_STE
1090+ * - 0x06 F_STREAM_DISABLED
1091+ * - 0x08 C_BAD_SUBSTREAMID
1092+ * - 0x0a C_BAD_CD
1093+ * - 0x10 F_TRANSLATION
1094+ * - 0x11 F_ADDR_SIZE
1095+ * - 0x12 F_ACCESS
1096+ * - 0x13 F_PERMISSION
1097+ *
1098+ * StreamID field reports a virtual device ID. To receive a virtual event for a
1099+ * device, a vDEVICE must be allocated via IOMMU_VDEVICE_ALLOC.
1100+ */
1101+struct iommu_vevent_arm_smmuv3 {
1102+ __aligned_le64 evt[4];
1103+};
1104+
1105+/**
1106+ * struct iommu_veventq_alloc - ioctl(IOMMU_VEVENTQ_ALLOC)
1107+ * @size: sizeof(struct iommu_veventq_alloc)
1108+ * @flags: Must be 0
1109+ * @viommu_id: virtual IOMMU ID to associate the vEVENTQ with
1110+ * @type: Type of the vEVENTQ. Must be defined in enum iommu_veventq_type
1111+ * @veventq_depth: Maximum number of events in the vEVENTQ
1112+ * @out_veventq_id: The ID of the new vEVENTQ
1113+ * @out_veventq_fd: The fd of the new vEVENTQ. User space must close the
1114+ * successfully returned fd after using it
1115+ * @__reserved: Must be 0
1116+ *
1117+ * Explicitly allocate a virtual event queue interface for a vIOMMU. A vIOMMU
1118+ * can have multiple FDs for different types, but is confined to one per @type.
1119+ * User space should open the @out_veventq_fd to read vEVENTs out of a vEVENTQ,
1120+ * if there are vEVENTs available. A vEVENTQ will lose events due to overflow,
1121+ * if the number of the vEVENTs hits @veventq_depth.
1122+ *
1123+ * Each vEVENT in a vEVENTQ encloses a struct iommufd_vevent_header followed by
1124+ * a type-specific data structure, in a normal case:
1125+ *
1126+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
1127+ * | | header0 | data0 | header1 | data1 | ... | headerN | dataN | |
1128+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
1129+ *
1130+ * unless a tailing IOMMU_VEVENTQ_FLAG_LOST_EVENTS header is logged (refer to
1131+ * struct iommufd_vevent_header).
1132+ */
1133+struct iommu_veventq_alloc {
1134+ __u32 size;
1135+ __u32 flags;
1136+ __u32 viommu_id;
1137+ __u32 type;
1138+ __u32 veventq_depth;
1139+ __u32 out_veventq_id;
1140+ __u32 out_veventq_fd;
1141+ __u32 __reserved;
1142+};
1143+#define IOMMU_VEVENTQ_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VEVENTQ_ALLOC)
10171144 #endif
include/uapi/linux/vfio.h+19−10
@@ -932,29 +932,34 @@ struct vfio_device_bind_iommufd { · 931 unmodified lines
932932 * VFIO_DEVICE_ATTACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 19,
933933 * struct vfio_device_attach_iommufd_pt)
934934 * @argsz: User filled size of this data.
935− * @flags: Must be 0.
935+ * @flags: Flags for attach.
936936 * @pt_id: Input the target id which can represent an ioas or a hwpt
937937 * allocated via iommufd subsystem.
938938 * Output the input ioas id or the attached hwpt id which could
939939 * be the specified hwpt itself or a hwpt automatically created
940940 * for the specified ioas by kernel during the attachment.
941+ * @pasid: The pasid to be attached, only meaningful when
942+ * VFIO_DEVICE_ATTACH_PASID is set in @flags
941943 *
942944 * Associate the device with an address space within the bound iommufd.
943945 * Undo by VFIO_DEVICE_DETACH_IOMMUFD_PT or device fd close. This is only
944946 * allowed on cdev fds.
945947 *
946− * If a vfio device is currently attached to a valid hw_pagetable, without doing
947− * a VFIO_DEVICE_DETACH_IOMMUFD_PT, a second VFIO_DEVICE_ATTACH_IOMMUFD_PT ioctl
948− * passing in another hw_pagetable (hwpt) id is allowed. This action, also known
949− * as a hw_pagetable replacement, will replace the device's currently attached
950− * hw_pagetable with a new hw_pagetable corresponding to the given pt_id.
948+ * If a vfio device or a pasid of this device is currently attached to a valid
949+ * hw_pagetable (hwpt), without doing a VFIO_DEVICE_DETACH_IOMMUFD_PT, a second
950+ * VFIO_DEVICE_ATTACH_IOMMUFD_PT ioctl passing in another hwpt id is allowed.
951+ * This action, also known as a hw_pagetable replacement, will replace the
952+ * currently attached hwpt of the device or the pasid of this device with a new
953+ * hwpt corresponding to the given pt_id.
951954 *
952955 * Return: 0 on success, -errno on failure.
953956 */
954957 struct vfio_device_attach_iommufd_pt {
955958 __u32 argsz;
956959 __u32 flags;
960+#define VFIO_DEVICE_ATTACH_PASID (1 << 0)
957961 __u32 pt_id;
962+ __u32 pasid;
958963 };
959964
960965 #define VFIO_DEVICE_ATTACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 19)
@@ -963,17 +968,21 @@ struct vfio_device_attach_iommufd_pt { · 2 unmodified lines
963968 * VFIO_DEVICE_DETACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 20,
964969 * struct vfio_device_detach_iommufd_pt)
965970 * @argsz: User filled size of this data.
966− * @flags: Must be 0.
971+ * @flags: Flags for detach.
972+ * @pasid: The pasid to be detached, only meaningful when
973+ * VFIO_DEVICE_DETACH_PASID is set in @flags
967974 *
968− * Remove the association of the device and its current associated address
969− * space. After it, the device should be in a blocking DMA state. This is only
970− * allowed on cdev fds.
975+ * Remove the association of the device or a pasid of the device and its current
976+ * associated address space. After it, the device or the pasid should be in a
977+ * blocking DMA state. This is only allowed on cdev fds.
971978 *
972979 * Return: 0 on success, -errno on failure.
973980 */
974981 struct vfio_device_detach_iommufd_pt {
975982 __u32 argsz;
976983 __u32 flags;
984+#define VFIO_DEVICE_DETACH_PASID (1 << 0)
985+ __u32 pasid;
977986 };
978987
979988 #define VFIO_DEVICE_DETACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 20)
lib/idr.c+67−0
@@ -476,6 +476,73 @@ nospc: · 475 unmodified lines
476476 }
477477 EXPORT_SYMBOL(ida_alloc_range);
478478
479+/**
480+ * ida_find_first_range - Get the lowest used ID.
481+ * @ida: IDA handle.
482+ * @min: Lowest ID to get.
483+ * @max: Highest ID to get.
484+ *
485+ * Get the lowest used ID between @min and @max, inclusive. The returned
486+ * ID will not exceed %INT_MAX, even if @max is larger.
487+ *
488+ * Context: Any context. Takes and releases the xa_lock.
489+ * Return: The lowest used ID, or errno if no used ID is found.
490+ */
491+int ida_find_first_range(struct ida *ida, unsigned int min, unsigned int max)
492+{
493+ unsigned long index = min / IDA_BITMAP_BITS;
494+ unsigned int offset = min % IDA_BITMAP_BITS;
495+ unsigned long *addr, size, bit;
496+ unsigned long tmp = 0;
497+ unsigned long flags;
498+ void *entry;
499+ int ret;
500+
501+ if ((int)min < 0)
502+ return -EINVAL;
503+ if ((int)max < 0)
504+ max = INT_MAX;
505+
506+ xa_lock_irqsave(&ida->xa, flags);
507+
508+ entry = xa_find(&ida->xa, &index, max / IDA_BITMAP_BITS, XA_PRESENT);
509+ if (!entry) {
510+ ret = -ENOENT;
511+ goto err_unlock;
512+ }
513+
514+ if (index > min / IDA_BITMAP_BITS)
515+ offset = 0;
516+ if (index * IDA_BITMAP_BITS + offset > max) {
517+ ret = -ENOENT;
518+ goto err_unlock;
519+ }
520+
521+ if (xa_is_value(entry)) {
522+ tmp = xa_to_value(entry);
523+ addr = &tmp;
524+ size = BITS_PER_XA_VALUE;
525+ } else {
526+ addr = ((struct ida_bitmap *)entry)->bitmap;
527+ size = IDA_BITMAP_BITS;
528+ }
529+
530+ bit = find_next_bit(addr, size, offset);
531+
532+ xa_unlock_irqrestore(&ida->xa, flags);
533+
534+ if (bit == size ||
535+ index * IDA_BITMAP_BITS + bit > max)
536+ return -ENOENT;
537+
538+ return index * IDA_BITMAP_BITS + bit;
539+
540+err_unlock:
541+ xa_unlock_irqrestore(&ida->xa, flags);
542+ return ret;
543+}
544+EXPORT_SYMBOL(ida_find_first_range);
545+
479546 /**
480547 * ida_free() - Release an allocated ID.
481548 * @ida: IDA handle.
lib/test_ida.c+70−0
@@ -189,6 +189,75 @@ static void ida_check_bad_free(struct ida *ida) · 188 unmodified lines
189189 IDA_BUG_ON(ida, !ida_is_empty(ida));
190190 }
191191
192+/*
193+ * Check ida_find_first_range() and varriants.
194+ */
195+static void ida_check_find_first(struct ida *ida)
196+{
197+ /* IDA is empty; all of the below should be not exist */
198+ IDA_BUG_ON(ida, ida_exists(ida, 0));
199+ IDA_BUG_ON(ida, ida_exists(ida, 3));
200+ IDA_BUG_ON(ida, ida_exists(ida, 63));
201+ IDA_BUG_ON(ida, ida_exists(ida, 1023));
202+ IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1));
203+
204+ /* IDA contains a single value entry */
205+ IDA_BUG_ON(ida, ida_alloc_min(ida, 3, GFP_KERNEL) != 3);
206+ IDA_BUG_ON(ida, ida_exists(ida, 0));
207+ IDA_BUG_ON(ida, !ida_exists(ida, 3));
208+ IDA_BUG_ON(ida, ida_exists(ida, 63));
209+ IDA_BUG_ON(ida, ida_exists(ida, 1023));
210+ IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1));
211+
212+ IDA_BUG_ON(ida, ida_alloc_min(ida, 63, GFP_KERNEL) != 63);
213+ IDA_BUG_ON(ida, ida_exists(ida, 0));
214+ IDA_BUG_ON(ida, !ida_exists(ida, 3));
215+ IDA_BUG_ON(ida, !ida_exists(ida, 63));
216+ IDA_BUG_ON(ida, ida_exists(ida, 1023));
217+ IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1));
218+
219+ /* IDA contains a single bitmap */
220+ IDA_BUG_ON(ida, ida_alloc_min(ida, 1023, GFP_KERNEL) != 1023);
221+ IDA_BUG_ON(ida, ida_exists(ida, 0));
222+ IDA_BUG_ON(ida, !ida_exists(ida, 3));
223+ IDA_BUG_ON(ida, !ida_exists(ida, 63));
224+ IDA_BUG_ON(ida, !ida_exists(ida, 1023));
225+ IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1));
226+
227+ /* IDA contains a tree */
228+ IDA_BUG_ON(ida, ida_alloc_min(ida, (1 << 20) - 1, GFP_KERNEL) != (1 << 20) - 1);
229+ IDA_BUG_ON(ida, ida_exists(ida, 0));
230+ IDA_BUG_ON(ida, !ida_exists(ida, 3));
231+ IDA_BUG_ON(ida, !ida_exists(ida, 63));
232+ IDA_BUG_ON(ida, !ida_exists(ida, 1023));
233+ IDA_BUG_ON(ida, !ida_exists(ida, (1 << 20) - 1));
234+
235+ /* Now try to find first */
236+ IDA_BUG_ON(ida, ida_find_first(ida) != 3);
237+ IDA_BUG_ON(ida, ida_find_first_range(ida, -1, 2) != -EINVAL);
238+ IDA_BUG_ON(ida, ida_find_first_range(ida, 0, 2) != -ENOENT); // no used ID
239+ IDA_BUG_ON(ida, ida_find_first_range(ida, 0, 3) != 3);
240+ IDA_BUG_ON(ida, ida_find_first_range(ida, 1, 3) != 3);
241+ IDA_BUG_ON(ida, ida_find_first_range(ida, 3, 3) != 3);
242+ IDA_BUG_ON(ida, ida_find_first_range(ida, 2, 4) != 3);
243+ IDA_BUG_ON(ida, ida_find_first_range(ida, 4, 3) != -ENOENT); // min > max, fail
244+ IDA_BUG_ON(ida, ida_find_first_range(ida, 4, 60) != -ENOENT); // no used ID
245+ IDA_BUG_ON(ida, ida_find_first_range(ida, 4, 64) != 63);
246+ IDA_BUG_ON(ida, ida_find_first_range(ida, 63, 63) != 63);
247+ IDA_BUG_ON(ida, ida_find_first_range(ida, 64, 1026) != 1023);
248+ IDA_BUG_ON(ida, ida_find_first_range(ida, 1023, 1023) != 1023);
249+ IDA_BUG_ON(ida, ida_find_first_range(ida, 1023, (1 << 20) - 1) != 1023);
250+ IDA_BUG_ON(ida, ida_find_first_range(ida, 1024, (1 << 20) - 1) != (1 << 20) - 1);
251+ IDA_BUG_ON(ida, ida_find_first_range(ida, (1 << 20), INT_MAX) != -ENOENT);
252+
253+ ida_free(ida, 3);
254+ ida_free(ida, 63);
255+ ida_free(ida, 1023);
256+ ida_free(ida, (1 << 20) - 1);
257+
258+ IDA_BUG_ON(ida, !ida_is_empty(ida));
259+}
260+
192261 static DEFINE_IDA(ida);
193262
194263 static int ida_checks(void)
@@ -202,6 +271,7 @@ static int ida_checks(void) · 7 unmodified lines
202271 ida_check_max(&ida);
203272 ida_check_conv(&ida);
204273 ida_check_bad_free(&ida);
274+ ida_check_find_first(&ida);
205275
206276 printk("IDA: %u of %u tests passed\n", tests_passed, tests_run);
207277 return (tests_run != tests_passed) ? 0 : -EINVAL;
tools/testing/selftests/iommu/iommufd.c+365−0
@@ -342,12 +342,14 @@ FIXTURE(iommufd_ioas) · 341 unmodified lines
342342 uint32_t hwpt_id;
343343 uint32_t device_id;
344344 uint64_t base_iova;
345+ uint32_t device_pasid_id;
345346 };
346347
347348 FIXTURE_VARIANT(iommufd_ioas)
348349 {
349350 unsigned int mock_domains;
350351 unsigned int memory_limit;
352+ bool pasid_capable;
351353 };
352354
353355 FIXTURE_SETUP(iommufd_ioas)
@@ -372,6 +374,12 @@ FIXTURE_SETUP(iommufd_ioas) · 18 unmodified lines
372374 IOMMU_TEST_DEV_CACHE_DEFAULT);
373375 self->base_iova = MOCK_APERTURE_START;
374376 }
377+
378+ if (variant->pasid_capable)
379+ test_cmd_mock_domain_flags(self->ioas_id,
380+ MOCK_FLAGS_DEVICE_PASID,
381+ NULL, NULL,
382+ &self->device_pasid_id);
375383 }
376384
377385 FIXTURE_TEARDOWN(iommufd_ioas)
@@ -387,6 +395,7 @@ FIXTURE_VARIANT_ADD(iommufd_ioas, no_domain) · 9 unmodified lines
387395 FIXTURE_VARIANT_ADD(iommufd_ioas, mock_domain)
388396 {
389397 .mock_domains = 1,
398+ .pasid_capable = true,
390399 };
391400
392401 FIXTURE_VARIANT_ADD(iommufd_ioas, two_mock_domain)
@@ -439,6 +448,10 @@ TEST_F(iommufd_ioas, alloc_hwpt_nested) · 46 unmodified lines
439448 &test_hwpt_id);
440449 test_err_hwpt_alloc(EINVAL, self->device_id, self->device_id, 0,
441450 &test_hwpt_id);
451+ test_err_hwpt_alloc(EOPNOTSUPP, self->device_id, self->ioas_id,
452+ IOMMU_HWPT_ALLOC_NEST_PARENT |
453+ IOMMU_HWPT_FAULT_ID_VALID,
454+ &test_hwpt_id);
442455
443456 test_cmd_hwpt_alloc(self->device_id, self->ioas_id,
444457 IOMMU_HWPT_ALLOC_NEST_PARENT,
@@ -748,6 +761,8 @@ TEST_F(iommufd_ioas, get_hw_info) · 303 unmodified lines
748761 } buffer_smaller;
749762
750763 if (self->device_id) {
764+ uint8_t max_pasid = 0;
765+
751766 /* Provide a zero-size user_buffer */
752767 test_cmd_get_hw_info(self->device_id, NULL, 0);
753768 /* Provide a user_buffer with exact size */
@@ -762,6 +777,13 @@ TEST_F(iommufd_ioas, get_hw_info) · 8 unmodified lines
762777 * the fields within the size range still gets updated.
763778 */
764779 test_cmd_get_hw_info(self->device_id, &buffer_smaller, sizeof(buffer_smaller));
780+ test_cmd_get_hw_info_pasid(self->device_id, &max_pasid);
781+ ASSERT_EQ(0, max_pasid);
782+ if (variant->pasid_capable) {
783+ test_cmd_get_hw_info_pasid(self->device_pasid_id,
784+ &max_pasid);
785+ ASSERT_EQ(MOCK_PASID_WIDTH, max_pasid);
786+ }
765787 } else {
766788 test_err_get_hw_info(ENOENT, self->device_id,
767789 &buffer_exact, sizeof(buffer_exact));
@@ -2736,6 +2758,7 @@ TEST_F(iommufd_viommu, viommu_alloc_nested_iopf) · 1968 unmodified lines
27362758 uint32_t iopf_hwpt_id;
27372759 uint32_t fault_id;
27382760 uint32_t fault_fd;
2761+ uint32_t vdev_id;
27392762
27402763 if (self->device_id) {
27412764 test_ioctl_fault_alloc(&fault_id, &fault_fd);
@@ -2752,6 +2775,10 @@ TEST_F(iommufd_viommu, viommu_alloc_nested_iopf) · 10 unmodified lines
27522775 &iopf_hwpt_id, IOMMU_HWPT_DATA_SELFTEST, &data,
27532776 sizeof(data));
27542777
2778+ /* Must allocate vdevice before attaching to a nested hwpt */
2779+ test_err_mock_domain_replace(ENOENT, self->stdev_id,
2780+ iopf_hwpt_id);
2781+ test_cmd_vdevice_alloc(viommu_id, dev_id, 0x99, &vdev_id);
27552782 test_cmd_mock_domain_replace(self->stdev_id, iopf_hwpt_id);
27562783 EXPECT_ERRNO(EBUSY,
27572784 _test_ioctl_destroy(self->fd, iopf_hwpt_id));
@@ -2769,15 +2796,46 @@ TEST_F(iommufd_viommu, vdevice_alloc) · 11 unmodified lines
27692796 uint32_t viommu_id = self->viommu_id;
27702797 uint32_t dev_id = self->device_id;
27712798 uint32_t vdev_id = 0;
2799+ uint32_t veventq_id;
2800+ uint32_t veventq_fd;
2801+ int prev_seq = -1;
27722802
27732803 if (dev_id) {
2804+ /* Must allocate vdevice before attaching to a nested hwpt */
2805+ test_err_mock_domain_replace(ENOENT, self->stdev_id,
2806+ self->nested_hwpt_id);
2807+
2808+ /* Allocate a vEVENTQ with veventq_depth=2 */
2809+ test_cmd_veventq_alloc(viommu_id, IOMMU_VEVENTQ_TYPE_SELFTEST,
2810+ &veventq_id, &veventq_fd);
2811+ test_err_veventq_alloc(EEXIST, viommu_id,
2812+ IOMMU_VEVENTQ_TYPE_SELFTEST, NULL, NULL);
27742813 /* Set vdev_id to 0x99, unset it, and set to 0x88 */
27752814 test_cmd_vdevice_alloc(viommu_id, dev_id, 0x99, &vdev_id);
2815+ test_cmd_mock_domain_replace(self->stdev_id,
2816+ self->nested_hwpt_id);
2817+ test_cmd_trigger_vevents(dev_id, 1);
2818+ test_cmd_read_vevents(veventq_fd, 1, 0x99, &prev_seq);
27762819 test_err_vdevice_alloc(EEXIST, viommu_id, dev_id, 0x99,
27772820 &vdev_id);
2821+ test_cmd_mock_domain_replace(self->stdev_id, self->ioas_id);
27782822 test_ioctl_destroy(vdev_id);
2823+
2824+ /* Try again with 0x88 */
27792825 test_cmd_vdevice_alloc(viommu_id, dev_id, 0x88, &vdev_id);
2826+ test_cmd_mock_domain_replace(self->stdev_id,
2827+ self->nested_hwpt_id);
2828+ /* Trigger an overflow with three events */
2829+ test_cmd_trigger_vevents(dev_id, 3);
2830+ test_err_read_vevents(EOVERFLOW, veventq_fd, 3, 0x88,
2831+ &prev_seq);
2832+ /* Overflow must be gone after the previous reads */
2833+ test_cmd_trigger_vevents(dev_id, 1);
2834+ test_cmd_read_vevents(veventq_fd, 1, 0x88, &prev_seq);
2835+ close(veventq_fd);
2836+ test_cmd_mock_domain_replace(self->stdev_id, self->ioas_id);
27802837 test_ioctl_destroy(vdev_id);
2838+ test_ioctl_destroy(veventq_id);
27812839 } else {
27822840 test_err_vdevice_alloc(ENOENT, viommu_id, dev_id, 0x99, NULL);
27832841 }
@@ -2956,4 +3014,311 @@ TEST_F(iommufd_viommu, vdevice_cache) · 172 unmodified lines
29563014 }
29573015 }
29583016
3017+FIXTURE(iommufd_device_pasid)
3018+{
3019+ int fd;
3020+ uint32_t ioas_id;
3021+ uint32_t hwpt_id;
3022+ uint32_t stdev_id;
3023+ uint32_t device_id;
3024+ uint32_t no_pasid_stdev_id;
3025+ uint32_t no_pasid_device_id;
3026+};
3027+
3028+FIXTURE_VARIANT(iommufd_device_pasid)
3029+{
3030+ bool pasid_capable;
3031+};
3032+
3033+FIXTURE_SETUP(iommufd_device_pasid)
3034+{
3035+ self->fd = open("/dev/iommu", O_RDWR);
3036+ ASSERT_NE(-1, self->fd);
3037+ test_ioctl_ioas_alloc(&self->ioas_id);
3038+
3039+ test_cmd_mock_domain_flags(self->ioas_id,
3040+ MOCK_FLAGS_DEVICE_PASID,
3041+ &self->stdev_id, &self->hwpt_id,
3042+ &self->device_id);
3043+ if (!variant->pasid_capable)
3044+ test_cmd_mock_domain_flags(self->ioas_id, 0,
3045+ &self->no_pasid_stdev_id, NULL,
3046+ &self->no_pasid_device_id);
3047+}
3048+
3049+FIXTURE_TEARDOWN(iommufd_device_pasid)
3050+{
3051+ teardown_iommufd(self->fd, _metadata);
3052+}
3053+
3054+FIXTURE_VARIANT_ADD(iommufd_device_pasid, no_pasid)
3055+{
3056+ .pasid_capable = false,
3057+};
3058+
3059+FIXTURE_VARIANT_ADD(iommufd_device_pasid, has_pasid)
3060+{
3061+ .pasid_capable = true,
3062+};
3063+
3064+TEST_F(iommufd_device_pasid, pasid_attach)
3065+{
3066+ struct iommu_hwpt_selftest data = {
3067+ .iotlb = IOMMU_TEST_IOTLB_DEFAULT,
3068+ };
3069+ uint32_t nested_hwpt_id[3] = {};
3070+ uint32_t parent_hwpt_id = 0;
3071+ uint32_t fault_id, fault_fd;
3072+ uint32_t s2_hwpt_id = 0;
3073+ uint32_t iopf_hwpt_id;
3074+ uint32_t pasid = 100;
3075+ uint32_t viommu_id;
3076+
3077+ /*
3078+ * Negative, detach pasid without attaching, this is not expected.
3079+ * But it should not result in failure anyway.
3080+ */
3081+ test_cmd_pasid_detach(pasid);
3082+
3083+ /* Allocate two nested hwpts sharing one common parent hwpt */
3084+ test_cmd_hwpt_alloc(self->device_id, self->ioas_id,
3085+ IOMMU_HWPT_ALLOC_NEST_PARENT,
3086+ &parent_hwpt_id);
3087+ test_cmd_hwpt_alloc_nested(self->device_id, parent_hwpt_id,
3088+ IOMMU_HWPT_ALLOC_PASID,
3089+ &nested_hwpt_id[0],
3090+ IOMMU_HWPT_DATA_SELFTEST,
3091+ &data, sizeof(data));
3092+ test_cmd_hwpt_alloc_nested(self->device_id, parent_hwpt_id,
3093+ IOMMU_HWPT_ALLOC_PASID,
3094+ &nested_hwpt_id[1],
3095+ IOMMU_HWPT_DATA_SELFTEST,
3096+ &data, sizeof(data));
3097+
3098+ /* Fault related preparation */
3099+ test_ioctl_fault_alloc(&fault_id, &fault_fd);
3100+ test_cmd_hwpt_alloc_iopf(self->device_id, parent_hwpt_id, fault_id,
3101+ IOMMU_HWPT_FAULT_ID_VALID | IOMMU_HWPT_ALLOC_PASID,
3102+ &iopf_hwpt_id,
3103+ IOMMU_HWPT_DATA_SELFTEST, &data,
3104+ sizeof(data));
3105+
3106+ /* Allocate a regular nested hwpt based on viommu */
3107+ test_cmd_viommu_alloc(self->device_id, parent_hwpt_id,
3108+ IOMMU_VIOMMU_TYPE_SELFTEST,
3109+ &viommu_id);
3110+ test_cmd_hwpt_alloc_nested(self->device_id, viommu_id,
3111+ IOMMU_HWPT_ALLOC_PASID,
3112+ &nested_hwpt_id[2],
3113+ IOMMU_HWPT_DATA_SELFTEST, &data,
3114+ sizeof(data));
3115+
3116+ test_cmd_hwpt_alloc(self->device_id, self->ioas_id,
3117+ IOMMU_HWPT_ALLOC_PASID,
3118+ &s2_hwpt_id);
3119+
3120+ /* Attach RID to non-pasid compat domain, */
3121+ test_cmd_mock_domain_replace(self->stdev_id, parent_hwpt_id);
3122+ /* then attach to pasid should fail */
3123+ test_err_pasid_attach(EINVAL, pasid, s2_hwpt_id);
3124+
3125+ /* Attach RID to pasid compat domain, */
3126+ test_cmd_mock_domain_replace(self->stdev_id, s2_hwpt_id);
3127+ /* then attach to pasid should succeed, */
3128+ test_cmd_pasid_attach(pasid, nested_hwpt_id[0]);
3129+ /* but attach RID to non-pasid compat domain should fail now. */
3130+ test_err_mock_domain_replace(EINVAL, self->stdev_id, parent_hwpt_id);
3131+ /*
3132+ * Detach hwpt from pasid 100, and check if the pasid 100
3133+ * has null domain.
3134+ */
3135+ test_cmd_pasid_detach(pasid);
3136+ ASSERT_EQ(0,
3137+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3138+ pasid, 0));
3139+ /* RID is attached to pasid-comapt domain, pasid path is not used */
3140+
3141+ if (!variant->pasid_capable) {
3142+ /*
3143+ * PASID-compatible domain can be used by non-PASID-capable
3144+ * device.
3145+ */
3146+ test_cmd_mock_domain_replace(self->no_pasid_stdev_id, nested_hwpt_id[0]);
3147+ test_cmd_mock_domain_replace(self->no_pasid_stdev_id, self->ioas_id);
3148+ /*
3149+ * Attach hwpt to pasid 100 of non-PASID-capable device,
3150+ * should fail, no matter domain is pasid-comapt or not.
3151+ */
3152+ EXPECT_ERRNO(EINVAL,
3153+ _test_cmd_pasid_attach(self->fd, self->no_pasid_stdev_id,
3154+ pasid, parent_hwpt_id));
3155+ EXPECT_ERRNO(EINVAL,
3156+ _test_cmd_pasid_attach(self->fd, self->no_pasid_stdev_id,
3157+ pasid, s2_hwpt_id));
3158+ }
3159+
3160+ /*
3161+ * Attach non pasid compat hwpt to pasid-capable device, should
3162+ * fail, and have null domain.
3163+ */
3164+ test_err_pasid_attach(EINVAL, pasid, parent_hwpt_id);
3165+ ASSERT_EQ(0,
3166+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3167+ pasid, 0));
3168+
3169+ /*
3170+ * Attach ioas to pasid 100, should fail, domain should
3171+ * be null.
3172+ */
3173+ test_err_pasid_attach(EINVAL, pasid, self->ioas_id);
3174+ ASSERT_EQ(0,
3175+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3176+ pasid, 0));
3177+
3178+ /*
3179+ * Attach the s2_hwpt to pasid 100, should succeed, domain should
3180+ * be valid.
3181+ */
3182+ test_cmd_pasid_attach(pasid, s2_hwpt_id);
3183+ ASSERT_EQ(0,
3184+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3185+ pasid, s2_hwpt_id));
3186+
3187+ /*
3188+ * Try attach pasid 100 with another hwpt, should FAIL
3189+ * as attach does not allow overwrite, use REPLACE instead.
3190+ */
3191+ test_err_pasid_attach(EBUSY, pasid, nested_hwpt_id[0]);
3192+
3193+ /*
3194+ * Detach hwpt from pasid 100 for next test, should succeed,
3195+ * and have null domain.
3196+ */
3197+ test_cmd_pasid_detach(pasid);
3198+ ASSERT_EQ(0,
3199+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3200+ pasid, 0));
3201+
3202+ /*
3203+ * Attach nested hwpt to pasid 100, should succeed, domain
3204+ * should be valid.
3205+ */
3206+ test_cmd_pasid_attach(pasid, nested_hwpt_id[0]);
3207+ ASSERT_EQ(0,
3208+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3209+ pasid, nested_hwpt_id[0]));
3210+
3211+ /* Attach to pasid 100 which has been attached, should fail. */
3212+ test_err_pasid_attach(EBUSY, pasid, nested_hwpt_id[0]);
3213+
3214+ /* cleanup pasid 100 */
3215+ test_cmd_pasid_detach(pasid);
3216+
3217+ /* Replace tests */
3218+
3219+ pasid = 200;
3220+ /*
3221+ * Replace pasid 200 without attaching it, should fail
3222+ * with -EINVAL.
3223+ */
3224+ test_err_pasid_replace(EINVAL, pasid, s2_hwpt_id);
3225+
3226+ /*
3227+ * Attach the s2 hwpt to pasid 200, should succeed, domain should
3228+ * be valid.
3229+ */
3230+ test_cmd_pasid_attach(pasid, s2_hwpt_id);
3231+ ASSERT_EQ(0,
3232+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3233+ pasid, s2_hwpt_id));
3234+
3235+ /*
3236+ * Replace pasid 200 with self->ioas_id, should fail
3237+ * and domain should be the prior s2 hwpt.
3238+ */
3239+ test_err_pasid_replace(EINVAL, pasid, self->ioas_id);
3240+ ASSERT_EQ(0,
3241+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3242+ pasid, s2_hwpt_id));
3243+
3244+ /*
3245+ * Replace a nested hwpt for pasid 200, should succeed,
3246+ * and have valid domain.
3247+ */
3248+ test_cmd_pasid_replace(pasid, nested_hwpt_id[0]);
3249+ ASSERT_EQ(0,
3250+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3251+ pasid, nested_hwpt_id[0]));
3252+
3253+ /*
3254+ * Replace with another nested hwpt for pasid 200, should
3255+ * succeed, and have valid domain.
3256+ */
3257+ test_cmd_pasid_replace(pasid, nested_hwpt_id[1]);
3258+ ASSERT_EQ(0,
3259+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3260+ pasid, nested_hwpt_id[1]));
3261+
3262+ /* cleanup pasid 200 */
3263+ test_cmd_pasid_detach(pasid);
3264+
3265+ /* Negative Tests for pasid replace, use pasid 1024 */
3266+
3267+ /*
3268+ * Attach the s2 hwpt to pasid 1024, should succeed, domain should
3269+ * be valid.
3270+ */
3271+ pasid = 1024;
3272+ test_cmd_pasid_attach(pasid, s2_hwpt_id);
3273+ ASSERT_EQ(0,
3274+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3275+ pasid, s2_hwpt_id));
3276+
3277+ /*
3278+ * Replace pasid 1024 with nested_hwpt_id[0], should fail,
3279+ * but have the old valid domain. This is a designed
3280+ * negative case. Normally, this shall succeed.
3281+ */
3282+ test_err_pasid_replace(ENOMEM, pasid, nested_hwpt_id[0]);
3283+ ASSERT_EQ(0,
3284+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3285+ pasid, s2_hwpt_id));
3286+
3287+ /* cleanup pasid 1024 */
3288+ test_cmd_pasid_detach(pasid);
3289+
3290+ /* Attach to iopf-capable hwpt */
3291+
3292+ /*
3293+ * Attach an iopf hwpt to pasid 2048, should succeed, domain should
3294+ * be valid.
3295+ */
3296+ pasid = 2048;
3297+ test_cmd_pasid_attach(pasid, iopf_hwpt_id);
3298+ ASSERT_EQ(0,
3299+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3300+ pasid, iopf_hwpt_id));
3301+
3302+ test_cmd_trigger_iopf_pasid(self->device_id, pasid, fault_fd);
3303+
3304+ /*
3305+ * Replace with s2_hwpt_id for pasid 2048, should
3306+ * succeed, and have valid domain.
3307+ */
3308+ test_cmd_pasid_replace(pasid, s2_hwpt_id);
3309+ ASSERT_EQ(0,
3310+ test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3311+ pasid, s2_hwpt_id));
3312+
3313+ /* cleanup pasid 2048 */
3314+ test_cmd_pasid_detach(pasid);
3315+
3316+ test_ioctl_destroy(iopf_hwpt_id);
3317+ close(fault_fd);
3318+ test_ioctl_destroy(fault_id);
3319+
3320+ /* Detach the s2_hwpt_id from RID */
3321+ test_cmd_mock_domain_replace(self->stdev_id, self->ioas_id);
3322+}
3323+
29593324 TEST_HARNESS_MAIN
tools/testing/selftests/iommu/iommufd_fail_nth.c+51−8
@@ -209,12 +209,16 @@ FIXTURE(basic_fail_nth) · 208 unmodified lines
209209 {
210210 int fd;
211211 uint32_t access_id;
212+ uint32_t stdev_id;
213+ uint32_t pasid;
212214 };
213215
214216 FIXTURE_SETUP(basic_fail_nth)
215217 {
216218 self->fd = -1;
217219 self->access_id = 0;
220+ self->stdev_id = 0;
221+ self->pasid = 0; //test should use a non-zero value
218222 }
219223
220224 FIXTURE_TEARDOWN(basic_fail_nth)
@@ -226,6 +230,8 @@ FIXTURE_TEARDOWN(basic_fail_nth) · 5 unmodified lines
226230 rc = _test_cmd_destroy_access(self->access_id);
227231 assert(rc == 0);
228232 }
233+ if (self->pasid && self->stdev_id)
234+ _test_cmd_pasid_detach(self->fd, self->stdev_id, self->pasid);
229235 teardown_iommufd(self->fd, _metadata);
230236 }
231237
@@ -620,10 +626,11 @@ TEST_FAIL_NTH(basic_fail_nth, device) · 388 unmodified lines
620626 };
621627 struct iommu_test_hw_info info;
622628 uint32_t fault_id, fault_fd;
629+ uint32_t veventq_id, veventq_fd;
623630 uint32_t fault_hwpt_id;
631+ uint32_t test_hwpt_id;
624632 uint32_t ioas_id;
625633 uint32_t ioas_id2;
626− uint32_t stdev_id;
627634 uint32_t idev_id;
628635 uint32_t hwpt_id;
629636 uint32_t viommu_id;
@@ -654,25 +661,30 @@ TEST_FAIL_NTH(basic_fail_nth, device) · 24 unmodified lines
654661
655662 fail_nth_enable();
656663
657− if (_test_cmd_mock_domain(self->fd, ioas_id, &stdev_id, NULL,
658− &idev_id))
664+ if (_test_cmd_mock_domain_flags(self->fd, ioas_id,
665+ MOCK_FLAGS_DEVICE_PASID,
666+ &self->stdev_id, NULL, &idev_id))
659667 return -1;
660668
661− if (_test_cmd_get_hw_info(self->fd, idev_id, &info, sizeof(info), NULL))
669+ if (_test_cmd_get_hw_info(self->fd, idev_id, &info,
670+ sizeof(info), NULL, NULL))
662671 return -1;
663672
664− if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0, 0, &hwpt_id,
673+ if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0,
674+ IOMMU_HWPT_ALLOC_PASID, &hwpt_id,
665675 IOMMU_HWPT_DATA_NONE, 0, 0))
666676 return -1;
667677
668− if (_test_cmd_mock_domain_replace(self->fd, stdev_id, ioas_id2, NULL))
678+ if (_test_cmd_mock_domain_replace(self->fd, self->stdev_id, ioas_id2, NULL))
669679 return -1;
670680
671− if (_test_cmd_mock_domain_replace(self->fd, stdev_id, hwpt_id, NULL))
681+ if (_test_cmd_mock_domain_replace(self->fd, self->stdev_id, hwpt_id, NULL))
672682 return -1;
673683
674684 if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0,
675− IOMMU_HWPT_ALLOC_NEST_PARENT, &hwpt_id,
685+ IOMMU_HWPT_ALLOC_NEST_PARENT |
686+ IOMMU_HWPT_ALLOC_PASID,
687+ &hwpt_id,
676688 IOMMU_HWPT_DATA_NONE, 0, 0))
677689 return -1;
678690
@@ -692,6 +704,37 @@ TEST_FAIL_NTH(basic_fail_nth, device) · 13 unmodified lines
692704 IOMMU_HWPT_DATA_SELFTEST, &data, sizeof(data)))
693705 return -1;
694706
707+ if (_test_cmd_veventq_alloc(self->fd, viommu_id,
708+ IOMMU_VEVENTQ_TYPE_SELFTEST, &veventq_id,
709+ &veventq_fd))
710+ return -1;
711+ close(veventq_fd);
712+
713+ if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0,
714+ IOMMU_HWPT_ALLOC_PASID,
715+ &test_hwpt_id,
716+ IOMMU_HWPT_DATA_NONE, 0, 0))
717+ return -1;
718+
719+ /* Tests for pasid attach/replace/detach */
720+
721+ self->pasid = 200;
722+
723+ if (_test_cmd_pasid_attach(self->fd, self->stdev_id,
724+ self->pasid, hwpt_id)) {
725+ self->pasid = 0;
726+ return -1;
727+ }
728+
729+ if (_test_cmd_pasid_replace(self->fd, self->stdev_id,
730+ self->pasid, test_hwpt_id))
731+ return -1;
732+
733+ if (_test_cmd_pasid_detach(self->fd, self->stdev_id, self->pasid))
734+ return -1;
735+
736+ self->pasid = 0;
737+
695738 return 0;
696739 }
697740
tools/testing/selftests/iommu/iommufd_utils.h+222−7
@@ -9,6 +9,7 @@ · 8 unmodified lines
99 #include <sys/ioctl.h>
1010 #include <stdint.h>
1111 #include <assert.h>
12+#include <poll.h>
1213
1314 #include "../kselftest_harness.h"
1415 #include "../../../../drivers/iommu/iommufd/iommufd_test.h"
@@ -757,7 +758,8 @@ static void teardown_iommufd(int fd, struct __test_metadata *_metadata) · 742 unmodified lines
757758
758759 /* @data can be NULL */
759760 static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data,
760− size_t data_len, uint32_t *capabilities)
761+ size_t data_len, uint32_t *capabilities,
762+ uint8_t *max_pasid)
761763 {
762764 struct iommu_test_hw_info *info = (struct iommu_test_hw_info *)data;
763765 struct iommu_hw_info cmd = {
@@ -802,6 +804,9 @@ static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data, · 38 unmodified lines
802804 assert(!info->flags);
803805 }
804806
807+ if (max_pasid)
808+ *max_pasid = cmd.out_max_pasid_log2;
809+
805810 if (capabilities)
806811 *capabilities = cmd.out_capabilities;
807812
@@ -810,14 +815,19 @@ static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data, · 2 unmodified lines
810815
811816 #define test_cmd_get_hw_info(device_id, data, data_len) \
812817 ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, data, \
813− data_len, NULL))
818+ data_len, NULL, NULL))
814819
815820 #define test_err_get_hw_info(_errno, device_id, data, data_len) \
816821 EXPECT_ERRNO(_errno, _test_cmd_get_hw_info(self->fd, device_id, data, \
817− data_len, NULL))
822+ data_len, NULL, NULL))
818823
819824 #define test_cmd_get_hw_capabilities(device_id, caps, mask) \
820− ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, NULL, 0, &caps))
825+ ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, NULL, \
826+ 0, &caps, NULL))
827+
828+#define test_cmd_get_hw_info_pasid(device_id, max_pasid) \
829+ ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, NULL, \
830+ 0, NULL, max_pasid))
821831
822832 static int _test_ioctl_fault_alloc(int fd, __u32 *fault_id, __u32 *fault_fd)
823833 {
@@ -842,14 +852,15 @@ static int _test_ioctl_fault_alloc(int fd, __u32 *fault_id, __u32 *fault_fd) · 18 unmodified lines
842852 ASSERT_NE(0, *(fault_fd)); \
843853 })
844854
845−static int _test_cmd_trigger_iopf(int fd, __u32 device_id, __u32 fault_fd)
855+static int _test_cmd_trigger_iopf(int fd, __u32 device_id, __u32 pasid,
856+ __u32 fault_fd)
846857 {
847858 struct iommu_test_cmd trigger_iopf_cmd = {
848859 .size = sizeof(trigger_iopf_cmd),
849860 .op = IOMMU_TEST_OP_TRIGGER_IOPF,
850861 .trigger_iopf = {
851862 .dev_id = device_id,
852− .pasid = 0x1,
863+ .pasid = pasid,
853864 .grpid = 0x2,
854865 .perm = IOMMU_PGFAULT_PERM_READ | IOMMU_PGFAULT_PERM_WRITE,
855866 .addr = 0xdeadbeaf,
@@ -880,7 +891,10 @@ static int _test_cmd_trigger_iopf(int fd, __u32 device_id, __u32 fault_fd) · 24 unmodified lines
880891 }
881892
882893 #define test_cmd_trigger_iopf(device_id, fault_fd) \
883− ASSERT_EQ(0, _test_cmd_trigger_iopf(self->fd, device_id, fault_fd))
894+ ASSERT_EQ(0, _test_cmd_trigger_iopf(self->fd, device_id, 0x1, fault_fd))
895+#define test_cmd_trigger_iopf_pasid(device_id, pasid, fault_fd) \
896+ ASSERT_EQ(0, _test_cmd_trigger_iopf(self->fd, device_id, \
897+ pasid, fault_fd))
884898
885899 static int _test_cmd_viommu_alloc(int fd, __u32 device_id, __u32 hwpt_id,
886900 __u32 type, __u32 flags, __u32 *viommu_id)
@@ -936,3 +950,204 @@ static int _test_cmd_vdevice_alloc(int fd, __u32 viommu_id, __u32 idev_id, · 49 unmodified lines
936950 EXPECT_ERRNO(_errno, \
937951 _test_cmd_vdevice_alloc(self->fd, viommu_id, idev_id, \
938952 virt_id, vdev_id))
953+
954+static int _test_cmd_veventq_alloc(int fd, __u32 viommu_id, __u32 type,
955+ __u32 *veventq_id, __u32 *veventq_fd)
956+{
957+ struct iommu_veventq_alloc cmd = {
958+ .size = sizeof(cmd),
959+ .type = type,
960+ .veventq_depth = 2,
961+ .viommu_id = viommu_id,
962+ };
963+ int ret;
964+
965+ ret = ioctl(fd, IOMMU_VEVENTQ_ALLOC, &cmd);
966+ if (ret)
967+ return ret;
968+ if (veventq_id)
969+ *veventq_id = cmd.out_veventq_id;
970+ if (veventq_fd)
971+ *veventq_fd = cmd.out_veventq_fd;
972+ return 0;
973+}
974+
975+#define test_cmd_veventq_alloc(viommu_id, type, veventq_id, veventq_fd) \
976+ ASSERT_EQ(0, _test_cmd_veventq_alloc(self->fd, viommu_id, type, \
977+ veventq_id, veventq_fd))
978+#define test_err_veventq_alloc(_errno, viommu_id, type, veventq_id, \
979+ veventq_fd) \
980+ EXPECT_ERRNO(_errno, \
981+ _test_cmd_veventq_alloc(self->fd, viommu_id, type, \
982+ veventq_id, veventq_fd))
983+
984+static int _test_cmd_trigger_vevents(int fd, __u32 dev_id, __u32 nvevents)
985+{
986+ struct iommu_test_cmd trigger_vevent_cmd = {
987+ .size = sizeof(trigger_vevent_cmd),
988+ .op = IOMMU_TEST_OP_TRIGGER_VEVENT,
989+ .trigger_vevent = {
990+ .dev_id = dev_id,
991+ },
992+ };
993+ int ret;
994+
995+ while (nvevents--) {
996+ ret = ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_TRIGGER_VEVENT),
997+ &trigger_vevent_cmd);
998+ if (ret < 0)
999+ return -1;
1000+ }
1001+ return ret;
1002+}
1003+
1004+#define test_cmd_trigger_vevents(dev_id, nvevents) \
1005+ ASSERT_EQ(0, _test_cmd_trigger_vevents(self->fd, dev_id, nvevents))
1006+
1007+static int _test_cmd_read_vevents(int fd, __u32 event_fd, __u32 nvevents,
1008+ __u32 virt_id, int *prev_seq)
1009+{
1010+ struct pollfd pollfd = { .fd = event_fd, .events = POLLIN };
1011+ struct iommu_viommu_event_selftest *event;
1012+ struct iommufd_vevent_header *hdr;
1013+ ssize_t bytes;
1014+ void *data;
1015+ int ret, i;
1016+
1017+ ret = poll(&pollfd, 1, 1000);
1018+ if (ret < 0)
1019+ return -1;
1020+
1021+ data = calloc(nvevents, sizeof(*hdr) + sizeof(*event));
1022+ if (!data) {
1023+ errno = ENOMEM;
1024+ return -1;
1025+ }
1026+
1027+ bytes = read(event_fd, data,
1028+ nvevents * (sizeof(*hdr) + sizeof(*event)));
1029+ if (bytes <= 0) {
1030+ errno = EFAULT;
1031+ ret = -1;
1032+ goto out_free;
1033+ }
1034+
1035+ for (i = 0; i < nvevents; i++) {
1036+ hdr = data + i * (sizeof(*hdr) + sizeof(*event));
1037+
1038+ if (hdr->flags & IOMMU_VEVENTQ_FLAG_LOST_EVENTS ||
1039+ hdr->sequence - *prev_seq > 1) {
1040+ *prev_seq = hdr->sequence;
1041+ errno = EOVERFLOW;
1042+ ret = -1;
1043+ goto out_free;
1044+ }
1045+ *prev_seq = hdr->sequence;
1046+ event = data + sizeof(*hdr);
1047+ if (event->virt_id != virt_id) {
1048+ errno = EINVAL;
1049+ ret = -1;
1050+ goto out_free;
1051+ }
1052+ }
1053+
1054+ ret = 0;
1055+out_free:
1056+ free(data);
1057+ return ret;
1058+}
1059+
1060+#define test_cmd_read_vevents(event_fd, nvevents, virt_id, prev_seq) \
1061+ ASSERT_EQ(0, _test_cmd_read_vevents(self->fd, event_fd, nvevents, \
1062+ virt_id, prev_seq))
1063+#define test_err_read_vevents(_errno, event_fd, nvevents, virt_id, prev_seq) \
1064+ EXPECT_ERRNO(_errno, \
1065+ _test_cmd_read_vevents(self->fd, event_fd, nvevents, \
1066+ virt_id, prev_seq))
1067+
1068+static int _test_cmd_pasid_attach(int fd, __u32 stdev_id, __u32 pasid,
1069+ __u32 pt_id)
1070+{
1071+ struct iommu_test_cmd test_attach = {
1072+ .size = sizeof(test_attach),
1073+ .op = IOMMU_TEST_OP_PASID_ATTACH,
1074+ .id = stdev_id,
1075+ .pasid_attach = {
1076+ .pasid = pasid,
1077+ .pt_id = pt_id,
1078+ },
1079+ };
1080+
1081+ return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_ATTACH),
1082+ &test_attach);
1083+}
1084+
1085+#define test_cmd_pasid_attach(pasid, hwpt_id) \
1086+ ASSERT_EQ(0, _test_cmd_pasid_attach(self->fd, self->stdev_id, \
1087+ pasid, hwpt_id))
1088+
1089+#define test_err_pasid_attach(_errno, pasid, hwpt_id) \
1090+ EXPECT_ERRNO(_errno, \
1091+ _test_cmd_pasid_attach(self->fd, self->stdev_id, \
1092+ pasid, hwpt_id))
1093+
1094+static int _test_cmd_pasid_replace(int fd, __u32 stdev_id, __u32 pasid,
1095+ __u32 pt_id)
1096+{
1097+ struct iommu_test_cmd test_replace = {
1098+ .size = sizeof(test_replace),
1099+ .op = IOMMU_TEST_OP_PASID_REPLACE,
1100+ .id = stdev_id,
1101+ .pasid_replace = {
1102+ .pasid = pasid,
1103+ .pt_id = pt_id,
1104+ },
1105+ };
1106+
1107+ return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_REPLACE),
1108+ &test_replace);
1109+}
1110+
1111+#define test_cmd_pasid_replace(pasid, hwpt_id) \
1112+ ASSERT_EQ(0, _test_cmd_pasid_replace(self->fd, self->stdev_id, \
1113+ pasid, hwpt_id))
1114+
1115+#define test_err_pasid_replace(_errno, pasid, hwpt_id) \
1116+ EXPECT_ERRNO(_errno, \
1117+ _test_cmd_pasid_replace(self->fd, self->stdev_id, \
1118+ pasid, hwpt_id))
1119+
1120+static int _test_cmd_pasid_detach(int fd, __u32 stdev_id, __u32 pasid)
1121+{
1122+ struct iommu_test_cmd test_detach = {
1123+ .size = sizeof(test_detach),
1124+ .op = IOMMU_TEST_OP_PASID_DETACH,
1125+ .id = stdev_id,
1126+ .pasid_detach = {
1127+ .pasid = pasid,
1128+ },
1129+ };
1130+
1131+ return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_DETACH),
1132+ &test_detach);
1133+}
1134+
1135+#define test_cmd_pasid_detach(pasid) \
1136+ ASSERT_EQ(0, _test_cmd_pasid_detach(self->fd, self->stdev_id, pasid))
1137+
1138+static int test_cmd_pasid_check_hwpt(int fd, __u32 stdev_id, __u32 pasid,
1139+ __u32 hwpt_id)
1140+{
1141+ struct iommu_test_cmd test_pasid_check = {
1142+ .size = sizeof(test_pasid_check),
1143+ .op = IOMMU_TEST_OP_PASID_CHECK_HWPT,
1144+ .id = stdev_id,
1145+ .pasid_check = {
1146+ .pasid = pasid,
1147+ .hwpt_id = hwpt_id,
1148+ },
1149+ };
1150+
1151+ return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_CHECK_HWPT),
1152+ &test_pasid_check);
1153+}

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t