History

Merge tag 'for-linus-iommufd' of git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd

Pull iommufd updates from Jason Gunthorpe: "Two significant new items: - Allow reporting IOMMU HW events to userspace when the events are clearly linked to a device. This is linked to the VIOMMU object and is intended to be used by a VMM to forward HW events to the virtual machine as part of emulating a vIOMMU. ARM SMMUv3 is the first driver to use this mechanism. Like the existing fault events the data is delivered through a simple FD returning event records on read(). - PASID support in VFIO. The "Process Address Space ID" is a PCI feature that allows the device to tag all PCI DMA operations with an ID. The IOMMU will then use the ID to select a unique translation for those DMAs. This is part of Intel's vIOMMU support as VT-D HW requires the hypervisor to manage each PASID entry. The support is generic so any VFIO user could attach any translation to a PASID, and the support should work on ARM SMMUv3 as well. AMD requires additional driver work. Some minor updates, along with fixes: - Prevent using nested parents with fault's, no driver support today - Put a single "cookie_type" value in the iommu_domain to indicate what owns the various opaque owner fields" * tag 'for-linus-iommufd' of git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd: (49 commits) iommufd: Test attach before detaching pasid iommufd: Fix iommu_vevent_header tables markup iommu: Convert unreachable() to BUG() iommufd: Balance veventq->num_events inc/dec iommufd: Initialize the flags of vevent in iommufd_viommu_report_event() iommufd/selftest: Add coverage for reporting max_pasid_log2 via IOMMU_HW_INFO iommufd: Extend IOMMU_GET_HW_INFO to report PASID capability vfio: VFIO_DEVICE_[AT|DE]TACH_IOMMUFD_PT support pasid vfio-iommufd: Support pasid [at|de]tach for physical VFIO devices ida: Add ida_find_first_range() iommufd/selftest: Add coverage for iommufd pasid attach/detach iommufd/selftest: Add test ops to test pasid attach/detach iommufd/selftest: Add a helper to get test device iommufd/selftest: Add set_dev_pasid in mock iommu iommufd: Allow allocating PASID-compatible domain iommu/vt-d: Add IOMMU_HWPT_ALLOC_PASID support iommufd: Enforce PASID-compatible domain for RID iommufd: Support pasid attach/replace iommufd: Enforce PASID-compatible domain in PASID path iommufd/device: Add pasid_attach array to track per-PASID attach ...

Linus Torvalds committed 1 year ago

Commit 48552152 parents 792b830 7be11d339 files · +3147 −829Browse files

Unified · Split

39 files
  1. Documentation/userspace-api/iommufd.rst +17 −0
  2. drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c +60 −0
  3. drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +52 −28
  4. drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +36 −0
  5. drivers/iommu/dma-iommu.c +104 −100
  6. drivers/iommu/dma-iommu.h +14 −0
  7. drivers/iommu/intel/iommu.c +2 −1
  8. drivers/iommu/intel/nested.c +1 −1
  9. drivers/iommu/iommu-priv.h +16 −0
  10. drivers/iommu/iommu-sva.c +1 −0
  11. drivers/iommu/iommu.c +151 −9
  12. drivers/iommu/iommufd/Kconfig +1 −1
  13. drivers/iommu/iommufd/Makefile +1 −1
  14. drivers/iommu/iommufd/device.c +294 −205
  15. drivers/iommu/iommufd/driver.c +198 −0
  16. drivers/iommu/iommufd/eventq.c +598 −0
  17. drivers/iommu/iommufd/fault.c +0 −342
  18. drivers/iommu/iommufd/hw_pagetable.c +26 −16
  19. drivers/iommu/iommufd/iommufd_private.h +114 −42
  20. drivers/iommu/iommufd/iommufd_test.h +40 −0
  21. drivers/iommu/iommufd/main.c +7 −0
  22. drivers/iommu/iommufd/selftest.c +275 −22
  23. drivers/iommu/iommufd/viommu.c +2 −0
  24. drivers/pci/ats.c +33 −0
  25. drivers/vfio/device_cdev.c +52 −8
  26. drivers/vfio/iommufd.c +56 −4
  27. drivers/vfio/pci/vfio_pci.c +2 −0
  28. include/linux/idr.h +11 −0
  29. include/linux/iommu.h +15 −20
  30. include/linux/iommufd.h +29 −3
  31. include/linux/pci-ats.h +3 −0
  32. include/linux/vfio.h +14 −0
  33. include/uapi/linux/iommufd.h +128 −1
  34. include/uapi/linux/vfio.h +19 −10
  35. lib/idr.c +67 −0
  36. lib/test_ida.c +70 −0
  37. tools/testing/selftests/iommu/iommufd.c +365 −0
  38. tools/testing/selftests/iommu/iommufd_fail_nth.c +51 −8
  39. tools/testing/selftests/iommu/iommufd_utils.h +222 −7
Documentation/userspace-api/iommufd.rst+17−0
@@ -63,6 +63,13 @@ Following IOMMUFD objects are exposed to userspace: · 62 unmodified lines
63 space usually has mappings from guest-level I/O virtual addresses to guest-63 space usually has mappings from guest-level I/O virtual addresses to guest-
64 level physical addresses.64 level physical addresses.
65 65
66- IOMMUFD_FAULT, representing a software queue for an HWPT reporting IO page
67 faults using the IOMMU HW's PRI (Page Request Interface). This queue object
68 provides user space an FD to poll the page fault events and also to respond
69 to those events. A FAULT object must be created first to get a fault_id that
70 could be then used to allocate a fault-enabled HWPT via the IOMMU_HWPT_ALLOC
71 command by setting the IOMMU_HWPT_FAULT_ID_VALID bit in its flags field.
72
66- IOMMUFD_OBJ_VIOMMU, representing a slice of the physical IOMMU instance,73- IOMMUFD_OBJ_VIOMMU, representing a slice of the physical IOMMU instance,
67 passed to or shared with a VM. It may be some HW-accelerated virtualization74 passed to or shared with a VM. It may be some HW-accelerated virtualization
68 features and some SW resources used by the VM. For examples:75 features and some SW resources used by the VM. For examples:
@@ -109,6 +116,14 @@ Following IOMMUFD objects are exposed to userspace: · 40 unmodified lines
109 vIOMMU, which is a separate ioctl call from attaching the same device to an116 vIOMMU, which is a separate ioctl call from attaching the same device to an
110 HWPT_PAGING that the vIOMMU holds.117 HWPT_PAGING that the vIOMMU holds.
111 118
119- IOMMUFD_OBJ_VEVENTQ, representing a software queue for a vIOMMU to report its
120 events such as translation faults occurred to a nested stage-1 (excluding I/O
121 page faults that should go through IOMMUFD_OBJ_FAULT) and HW-specific events.
122 This queue object provides user space an FD to poll/read the vIOMMU events. A
123 vIOMMU object must be created first to get its viommu_id, which could be then
124 used to allocate a vEVENTQ. Each vIOMMU can support multiple types of vEVENTS,
125 but is confined to one vEVENTQ per vEVENTQ type.
126
112All user-visible objects are destroyed via the IOMMU_DESTROY uAPI.127All user-visible objects are destroyed via the IOMMU_DESTROY uAPI.
113 128
114The diagrams below show relationships between user-visible objects and kernel129The diagrams below show relationships between user-visible objects and kernel
@@ -251,8 +266,10 @@ User visible objects are backed by following datastructures: · 136 unmodified lines
251- iommufd_device for IOMMUFD_OBJ_DEVICE.266- iommufd_device for IOMMUFD_OBJ_DEVICE.
252- iommufd_hwpt_paging for IOMMUFD_OBJ_HWPT_PAGING.267- iommufd_hwpt_paging for IOMMUFD_OBJ_HWPT_PAGING.
253- iommufd_hwpt_nested for IOMMUFD_OBJ_HWPT_NESTED.268- iommufd_hwpt_nested for IOMMUFD_OBJ_HWPT_NESTED.
269- iommufd_fault for IOMMUFD_OBJ_FAULT.
254- iommufd_viommu for IOMMUFD_OBJ_VIOMMU.270- iommufd_viommu for IOMMUFD_OBJ_VIOMMU.
255- iommufd_vdevice for IOMMUFD_OBJ_VDEVICE.271- iommufd_vdevice for IOMMUFD_OBJ_VDEVICE.
272- iommufd_veventq for IOMMUFD_OBJ_VEVENTQ.
256 273
257Several terminologies when looking at these datastructures:274Several terminologies when looking at these datastructures:
258 275
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c+60−0
@@ -43,6 +43,8 @@ static void arm_smmu_make_nested_cd_table_ste( · 42 unmodified lines
43 target->data[0] |= nested_domain->ste[0] &43 target->data[0] |= nested_domain->ste[0] &
44 ~cpu_to_le64(STRTAB_STE_0_CFG);44 ~cpu_to_le64(STRTAB_STE_0_CFG);
45 target->data[1] |= nested_domain->ste[1];45 target->data[1] |= nested_domain->ste[1];
46 /* Merge events for DoS mitigations on eventq */
47 target->data[1] |= cpu_to_le64(STRTAB_STE_1_MEV);
46}48}
47 49
48/*50/*
@@ -85,6 +87,47 @@ static void arm_smmu_make_nested_domain_ste( · 36 unmodified lines
85 }87 }
86}88}
87 89
90int arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state,
91 struct arm_smmu_nested_domain *nested_domain)
92{
93 struct arm_smmu_vmaster *vmaster;
94 unsigned long vsid;
95 int ret;
96
97 iommu_group_mutex_assert(state->master->dev);
98
99 ret = iommufd_viommu_get_vdev_id(&nested_domain->vsmmu->core,
100 state->master->dev, &vsid);
101 if (ret)
102 return ret;
103
104 vmaster = kzalloc(sizeof(*vmaster), GFP_KERNEL);
105 if (!vmaster)
106 return -ENOMEM;
107 vmaster->vsmmu = nested_domain->vsmmu;
108 vmaster->vsid = vsid;
109 state->vmaster = vmaster;
110
111 return 0;
112}
113
114void arm_smmu_attach_commit_vmaster(struct arm_smmu_attach_state *state)
115{
116 struct arm_smmu_master *master = state->master;
117
118 mutex_lock(&master->smmu->streams_mutex);
119 kfree(master->vmaster);
120 master->vmaster = state->vmaster;
121 mutex_unlock(&master->smmu->streams_mutex);
122}
123
124void arm_smmu_master_clear_vmaster(struct arm_smmu_master *master)
125{
126 struct arm_smmu_attach_state state = { .master = master };
127
128 arm_smmu_attach_commit_vmaster(&state);
129}
130
88static int arm_smmu_attach_dev_nested(struct iommu_domain *domain,131static int arm_smmu_attach_dev_nested(struct iommu_domain *domain,
89 struct device *dev)132 struct device *dev)
90{133{
@@ -392,4 +435,21 @@ struct iommufd_viommu *arm_vsmmu_alloc(struct device *dev, · 301 unmodified lines
392 return &vsmmu->core;435 return &vsmmu->core;
393}436}
394 437
438int arm_vmaster_report_event(struct arm_smmu_vmaster *vmaster, u64 *evt)
439{
440 struct iommu_vevent_arm_smmuv3 vevt;
441 int i;
442
443 lockdep_assert_held(&vmaster->vsmmu->smmu->streams_mutex);
444
445 vevt.evt[0] = cpu_to_le64((evt[0] & ~EVTQ_0_SID) |
446 FIELD_PREP(EVTQ_0_SID, vmaster->vsid));
447 for (i = 1; i < EVTQ_ENT_DWORDS; i++)
448 vevt.evt[i] = cpu_to_le64(evt[i]);
449
450 return iommufd_viommu_report_event(&vmaster->vsmmu->core,
451 IOMMU_VEVENTQ_TYPE_ARM_SMMUV3, &vevt,
452 sizeof(vevt));
453}
454
395MODULE_IMPORT_NS("IOMMUFD");455MODULE_IMPORT_NS("IOMMUFD");
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c+52−28
@@ -1052,7 +1052,7 @@ void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) · 1051 unmodified lines
1052 cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR |1052 cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR |
1053 STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH |1053 STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH |
1054 STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW |1054 STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW |
1055 STRTAB_STE_1_EATS);1055 STRTAB_STE_1_EATS | STRTAB_STE_1_MEV);
1056 used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);1056 used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID);
1057 1057
1058 /*1058 /*
@@ -1068,7 +1068,7 @@ void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) · 9 unmodified lines
1068 if (cfg & BIT(1)) {1068 if (cfg & BIT(1)) {
1069 used_bits[1] |=1069 used_bits[1] |=
1070 cpu_to_le64(STRTAB_STE_1_S2FWB | STRTAB_STE_1_EATS |1070 cpu_to_le64(STRTAB_STE_1_S2FWB | STRTAB_STE_1_EATS |
1071 STRTAB_STE_1_SHCFG);1071 STRTAB_STE_1_SHCFG | STRTAB_STE_1_MEV);
1072 used_bits[2] |=1072 used_bits[2] |=
1073 cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR |1073 cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR |
1074 STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI |1074 STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI |
@@ -1813,8 +1813,8 @@ static void arm_smmu_decode_event(struct arm_smmu_device *smmu, u64 *raw, · 738 unmodified lines
1813 mutex_unlock(&smmu->streams_mutex);1813 mutex_unlock(&smmu->streams_mutex);
1814}1814}
1815 1815
1816static int arm_smmu_handle_event(struct arm_smmu_device *smmu,1816static int arm_smmu_handle_event(struct arm_smmu_device *smmu, u64 *evt,
1817 struct arm_smmu_event *event)1817 struct arm_smmu_event *event)
1818{1818{
1819 int ret = 0;1819 int ret = 0;
1820 u32 perm = 0;1820 u32 perm = 0;
@@ -1823,6 +1823,10 @@ static int arm_smmu_handle_event(struct arm_smmu_device *smmu, · 2 unmodified lines
1823 struct iommu_fault *flt = &fault_evt.fault;1823 struct iommu_fault *flt = &fault_evt.fault;
1824 1824
1825 switch (event->id) {1825 switch (event->id) {
1826 case EVT_ID_BAD_STE_CONFIG:
1827 case EVT_ID_STREAM_DISABLED_FAULT:
1828 case EVT_ID_BAD_SUBSTREAMID_CONFIG:
1829 case EVT_ID_BAD_CD_CONFIG:
1826 case EVT_ID_TRANSLATION_FAULT:1830 case EVT_ID_TRANSLATION_FAULT:
1827 case EVT_ID_ADDR_SIZE_FAULT:1831 case EVT_ID_ADDR_SIZE_FAULT:
1828 case EVT_ID_ACCESS_FAULT:1832 case EVT_ID_ACCESS_FAULT:
@@ -1832,31 +1836,30 @@ static int arm_smmu_handle_event(struct arm_smmu_device *smmu, · 3 unmodified lines
1832 return -EOPNOTSUPP;1836 return -EOPNOTSUPP;
1833 }1837 }
1834 1838
1835 if (!event->stall)1839 if (event->stall) {
1836 return -EOPNOTSUPP;1840 if (event->read)
1837 1841 perm |= IOMMU_FAULT_PERM_READ;
1838 if (event->read)1842 else
1839 perm |= IOMMU_FAULT_PERM_READ;1843 perm |= IOMMU_FAULT_PERM_WRITE;
1840 else
1841 perm |= IOMMU_FAULT_PERM_WRITE;
1842 1844
1843 if (event->instruction)1845 if (event->instruction)
1844 perm |= IOMMU_FAULT_PERM_EXEC;1846 perm |= IOMMU_FAULT_PERM_EXEC;
1845 1847
1846 if (event->privileged)1848 if (event->privileged)
1847 perm |= IOMMU_FAULT_PERM_PRIV;1849 perm |= IOMMU_FAULT_PERM_PRIV;
1848 1850
1849 flt->type = IOMMU_FAULT_PAGE_REQ;1851 flt->type = IOMMU_FAULT_PAGE_REQ;
1850 flt->prm = (struct iommu_fault_page_request) {1852 flt->prm = (struct iommu_fault_page_request){
1851 .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE,1853 .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE,
1852 .grpid = event->stag,1854 .grpid = event->stag,
1853 .perm = perm,1855 .perm = perm,
1854 .addr = event->iova,1856 .addr = event->iova,
1855 };1857 };
1856 1858
1857 if (event->ssv) {1859 if (event->ssv) {
1858 flt->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID;1860 flt->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID;
1859 flt->prm.pasid = event->ssid;1861 flt->prm.pasid = event->ssid;
1862 }
1860 }1863 }
1861 1864
1862 mutex_lock(&smmu->streams_mutex);1865 mutex_lock(&smmu->streams_mutex);
@@ -1866,7 +1869,12 @@ static int arm_smmu_handle_event(struct arm_smmu_device *smmu, · 3 unmodified lines
1866 goto out_unlock;1869 goto out_unlock;
1867 }1870 }
1868 1871
1869 ret = iommu_report_device_fault(master->dev, &fault_evt);1872 if (event->stall)
1873 ret = iommu_report_device_fault(master->dev, &fault_evt);
1874 else if (master->vmaster && !event->s2)
1875 ret = arm_vmaster_report_event(master->vmaster, evt);
1876 else
1877 ret = -EOPNOTSUPP; /* Unhandled events should be pinned */
1870out_unlock:1878out_unlock:
1871 mutex_unlock(&smmu->streams_mutex);1879 mutex_unlock(&smmu->streams_mutex);
1872 return ret;1880 return ret;
@@ -1944,7 +1952,7 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) · 71 unmodified lines
1944 do {1952 do {
1945 while (!queue_remove_raw(q, evt)) {1953 while (!queue_remove_raw(q, evt)) {
1946 arm_smmu_decode_event(smmu, evt, &event);1954 arm_smmu_decode_event(smmu, evt, &event);
1947 if (arm_smmu_handle_event(smmu, &event))1955 if (arm_smmu_handle_event(smmu, evt, &event))
1948 arm_smmu_dump_event(smmu, evt, &event, &rs);1956 arm_smmu_dump_event(smmu, evt, &event, &rs);
1949 1957
1950 put_device(event.dev);1958 put_device(event.dev);
@@ -2803,6 +2811,7 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state, · 852 unmodified lines
2803 struct arm_smmu_domain *smmu_domain =2811 struct arm_smmu_domain *smmu_domain =
2804 to_smmu_domain_devices(new_domain);2812 to_smmu_domain_devices(new_domain);
2805 unsigned long flags;2813 unsigned long flags;
2814 int ret;
2806 2815
2807 /*2816 /*
2808 * arm_smmu_share_asid() must not see two domains pointing to the same2817 * arm_smmu_share_asid() must not see two domains pointing to the same
@@ -2832,9 +2841,18 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state, · 23 unmodified lines
2832 }2841 }
2833 2842
2834 if (smmu_domain) {2843 if (smmu_domain) {
2844 if (new_domain->type == IOMMU_DOMAIN_NESTED) {
2845 ret = arm_smmu_attach_prepare_vmaster(
2846 state, to_smmu_nested_domain(new_domain));
2847 if (ret)
2848 return ret;
2849 }
2850
2835 master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);2851 master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL);
2836 if (!master_domain)2852 if (!master_domain) {
2853 kfree(state->vmaster);
2837 return -ENOMEM;2854 return -ENOMEM;
2855 }
2838 master_domain->master = master;2856 master_domain->master = master;
2839 master_domain->ssid = state->ssid;2857 master_domain->ssid = state->ssid;
2840 if (new_domain->type == IOMMU_DOMAIN_NESTED)2858 if (new_domain->type == IOMMU_DOMAIN_NESTED)
@@ -2861,6 +2879,7 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state, · 20 unmodified lines
2861 spin_unlock_irqrestore(&smmu_domain->devices_lock,2879 spin_unlock_irqrestore(&smmu_domain->devices_lock,
2862 flags);2880 flags);
2863 kfree(master_domain);2881 kfree(master_domain);
2882 kfree(state->vmaster);
2864 return -EINVAL;2883 return -EINVAL;
2865 }2884 }
2866 2885
@@ -2893,6 +2912,8 @@ void arm_smmu_attach_commit(struct arm_smmu_attach_state *state) · 26 unmodified lines
2893 2912
2894 lockdep_assert_held(&arm_smmu_asid_lock);2913 lockdep_assert_held(&arm_smmu_asid_lock);
2895 2914
2915 arm_smmu_attach_commit_vmaster(state);
2916
2896 if (state->ats_enabled && !master->ats_enabled) {2917 if (state->ats_enabled && !master->ats_enabled) {
2897 arm_smmu_enable_ats(master);2918 arm_smmu_enable_ats(master);
2898 } else if (state->ats_enabled && master->ats_enabled) {2919 } else if (state->ats_enabled && master->ats_enabled) {
@@ -3162,6 +3183,7 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain, · 263 unmodified lines
3162 struct arm_smmu_ste ste;3183 struct arm_smmu_ste ste;
3163 struct arm_smmu_master *master = dev_iommu_priv_get(dev);3184 struct arm_smmu_master *master = dev_iommu_priv_get(dev);
3164 3185
3186 arm_smmu_master_clear_vmaster(master);
3165 arm_smmu_make_bypass_ste(master->smmu, &ste);3187 arm_smmu_make_bypass_ste(master->smmu, &ste);
3166 arm_smmu_attach_dev_ste(domain, dev, &ste, STRTAB_STE_1_S1DSS_BYPASS);3188 arm_smmu_attach_dev_ste(domain, dev, &ste, STRTAB_STE_1_S1DSS_BYPASS);
3167 return 0;3189 return 0;
@@ -3180,7 +3202,9 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain, · 12 unmodified lines
3180 struct device *dev)3202 struct device *dev)
3181{3203{
3182 struct arm_smmu_ste ste;3204 struct arm_smmu_ste ste;
3205 struct arm_smmu_master *master = dev_iommu_priv_get(dev);
3183 3206
3207 arm_smmu_master_clear_vmaster(master);
3184 arm_smmu_make_abort_ste(&ste);3208 arm_smmu_make_abort_ste(&ste);
3185 arm_smmu_attach_dev_ste(domain, dev, &ste,3209 arm_smmu_attach_dev_ste(domain, dev, &ste,
3186 STRTAB_STE_1_S1DSS_TERMINATE);3210 STRTAB_STE_1_S1DSS_TERMINATE);
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h+36−0
@@ -266,6 +266,7 @@ static inline u32 arm_smmu_strtab_l2_idx(u32 sid) · 265 unmodified lines
266#define STRTAB_STE_1_S1COR GENMASK_ULL(5, 4)266#define STRTAB_STE_1_S1COR GENMASK_ULL(5, 4)
267#define STRTAB_STE_1_S1CSH GENMASK_ULL(7, 6)267#define STRTAB_STE_1_S1CSH GENMASK_ULL(7, 6)
268 268
269#define STRTAB_STE_1_MEV (1UL << 19)
269#define STRTAB_STE_1_S2FWB (1UL << 25)270#define STRTAB_STE_1_S2FWB (1UL << 25)
270#define STRTAB_STE_1_S1STALLD (1UL << 27)271#define STRTAB_STE_1_S1STALLD (1UL << 27)
271 272
@@ -799,6 +800,11 @@ struct arm_smmu_stream { · 527 unmodified lines
799 struct rb_node node;800 struct rb_node node;
800};801};
801 802
803struct arm_smmu_vmaster {
804 struct arm_vsmmu *vsmmu;
805 unsigned long vsid;
806};
807
802struct arm_smmu_event {808struct arm_smmu_event {
803 u8 stall : 1,809 u8 stall : 1,
804 ssv : 1,810 ssv : 1,
@@ -824,6 +830,7 @@ struct arm_smmu_master { · 19 unmodified lines
824 struct arm_smmu_device *smmu;830 struct arm_smmu_device *smmu;
825 struct device *dev;831 struct device *dev;
826 struct arm_smmu_stream *streams;832 struct arm_smmu_stream *streams;
833 struct arm_smmu_vmaster *vmaster; /* use smmu->streams_mutex */
827 /* Locked by the iommu core using the group mutex */834 /* Locked by the iommu core using the group mutex */
828 struct arm_smmu_ctx_desc_cfg cd_table;835 struct arm_smmu_ctx_desc_cfg cd_table;
829 unsigned int num_streams;836 unsigned int num_streams;
@@ -972,6 +979,7 @@ struct arm_smmu_attach_state { · 142 unmodified lines
972 bool disable_ats;979 bool disable_ats;
973 ioasid_t ssid;980 ioasid_t ssid;
974 /* Resulting state */981 /* Resulting state */
982 struct arm_smmu_vmaster *vmaster;
975 bool ats_enabled;983 bool ats_enabled;
976};984};
977 985
@@ -1055,9 +1063,37 @@ struct iommufd_viommu *arm_vsmmu_alloc(struct device *dev, · 77 unmodified lines
1055 struct iommu_domain *parent,1063 struct iommu_domain *parent,
1056 struct iommufd_ctx *ictx,1064 struct iommufd_ctx *ictx,
1057 unsigned int viommu_type);1065 unsigned int viommu_type);
1066int arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state,
1067 struct arm_smmu_nested_domain *nested_domain);
1068void arm_smmu_attach_commit_vmaster(struct arm_smmu_attach_state *state);
1069void arm_smmu_master_clear_vmaster(struct arm_smmu_master *master);
1070int arm_vmaster_report_event(struct arm_smmu_vmaster *vmaster, u64 *evt);
1058#else1071#else
1059#define arm_smmu_hw_info NULL1072#define arm_smmu_hw_info NULL
1060#define arm_vsmmu_alloc NULL1073#define arm_vsmmu_alloc NULL
1074
1075static inline int
1076arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state,
1077 struct arm_smmu_nested_domain *nested_domain)
1078{
1079 return 0;
1080}
1081
1082static inline void
1083arm_smmu_attach_commit_vmaster(struct arm_smmu_attach_state *state)
1084{
1085}
1086
1087static inline void
1088arm_smmu_master_clear_vmaster(struct arm_smmu_master *master)
1089{
1090}
1091
1092static inline int arm_vmaster_report_event(struct arm_smmu_vmaster *vmaster,
1093 u64 *evt)
1094{
1095 return -EOPNOTSUPP;
1096}
1061#endif /* CONFIG_ARM_SMMU_V3_IOMMUFD */1097#endif /* CONFIG_ARM_SMMU_V3_IOMMUFD */
1062 1098
1063#endif /* _ARM_SMMU_V3_H */1099#endif /* _ARM_SMMU_V3_H */
drivers/iommu/dma-iommu.c+104−100
@@ -42,11 +42,6 @@ struct iommu_dma_msi_page { · 41 unmodified lines
42 phys_addr_t phys;42 phys_addr_t phys;
43};43};
44 44
45enum iommu_dma_cookie_type {
46 IOMMU_DMA_IOVA_COOKIE,
47 IOMMU_DMA_MSI_COOKIE,
48};
49
50enum iommu_dma_queue_type {45enum iommu_dma_queue_type {
51 IOMMU_DMA_OPTS_PER_CPU_QUEUE,46 IOMMU_DMA_OPTS_PER_CPU_QUEUE,
52 IOMMU_DMA_OPTS_SINGLE_QUEUE,47 IOMMU_DMA_OPTS_SINGLE_QUEUE,
@@ -59,34 +54,30 @@ struct iommu_dma_options { · 6 unmodified lines
59};54};
60 55
61struct iommu_dma_cookie {56struct iommu_dma_cookie {
62 enum iommu_dma_cookie_type type;57 struct iova_domain iovad;
58 struct list_head msi_page_list;
59 /* Flush queue */
63 union {60 union {
64 /* Full allocator for IOMMU_DMA_IOVA_COOKIE */61 struct iova_fq *single_fq;
65 struct {62 struct iova_fq __percpu *percpu_fq;
66 struct iova_domain iovad;
67 /* Flush queue */
68 union {
69 struct iova_fq *single_fq;
70 struct iova_fq __percpu *percpu_fq;
71 };
72 /* Number of TLB flushes that have been started */
73 atomic64_t fq_flush_start_cnt;
74 /* Number of TLB flushes that have been finished */
75 atomic64_t fq_flush_finish_cnt;
76 /* Timer to regularily empty the flush queues */
77 struct timer_list fq_timer;
78 /* 1 when timer is active, 0 when not */
79 atomic_t fq_timer_on;
80 };
81 /* Trivial linear page allocator for IOMMU_DMA_MSI_COOKIE */
82 dma_addr_t msi_iova;
83 };63 };
84 struct list_head msi_page_list;64 /* Number of TLB flushes that have been started */
85 65 atomic64_t fq_flush_start_cnt;
66 /* Number of TLB flushes that have been finished */
67 atomic64_t fq_flush_finish_cnt;
68 /* Timer to regularily empty the flush queues */
69 struct timer_list fq_timer;
70 /* 1 when timer is active, 0 when not */
71 atomic_t fq_timer_on;
86 /* Domain for flush queue callback; NULL if flush queue not in use */72 /* Domain for flush queue callback; NULL if flush queue not in use */
87 struct iommu_domain *fq_domain;73 struct iommu_domain *fq_domain;
88 /* Options for dma-iommu use */74 /* Options for dma-iommu use */
89 struct iommu_dma_options options;75 struct iommu_dma_options options;
76};
77
78struct iommu_dma_msi_cookie {
79 dma_addr_t msi_iova;
80 struct list_head msi_page_list;
90};81};
91 82
92static DEFINE_STATIC_KEY_FALSE(iommu_deferred_attach_enabled);83static DEFINE_STATIC_KEY_FALSE(iommu_deferred_attach_enabled);
@@ -102,9 +93,6 @@ static int __init iommu_dma_forcedac_setup(char *str) · 9 unmodified lines
102}93}
103early_param("iommu.forcedac", iommu_dma_forcedac_setup);94early_param("iommu.forcedac", iommu_dma_forcedac_setup);
104 95
105static int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
106 phys_addr_t msi_addr);
107
108/* Number of entries per flush queue */96/* Number of entries per flush queue */
109#define IOVA_DEFAULT_FQ_SIZE 25697#define IOVA_DEFAULT_FQ_SIZE 256
110#define IOVA_SINGLE_FQ_SIZE 3276898#define IOVA_SINGLE_FQ_SIZE 32768
@@ -368,39 +356,24 @@ int iommu_dma_init_fq(struct iommu_domain *domain) · 257 unmodified lines
368 return 0;356 return 0;
369}357}
370 358
371static inline size_t cookie_msi_granule(struct iommu_dma_cookie *cookie)
372{
373 if (cookie->type == IOMMU_DMA_IOVA_COOKIE)
374 return cookie->iovad.granule;
375 return PAGE_SIZE;
376}
377
378static struct iommu_dma_cookie *cookie_alloc(enum iommu_dma_cookie_type type)
379{
380 struct iommu_dma_cookie *cookie;
381
382 cookie = kzalloc(sizeof(*cookie), GFP_KERNEL);
383 if (cookie) {
384 INIT_LIST_HEAD(&cookie->msi_page_list);
385 cookie->type = type;
386 }
387 return cookie;
388}
389
390/**359/**
391 * iommu_get_dma_cookie - Acquire DMA-API resources for a domain360 * iommu_get_dma_cookie - Acquire DMA-API resources for a domain
392 * @domain: IOMMU domain to prepare for DMA-API usage361 * @domain: IOMMU domain to prepare for DMA-API usage
393 */362 */
394int iommu_get_dma_cookie(struct iommu_domain *domain)363int iommu_get_dma_cookie(struct iommu_domain *domain)
395{364{
396 if (domain->iova_cookie)365 struct iommu_dma_cookie *cookie;
366
367 if (domain->cookie_type != IOMMU_COOKIE_NONE)
397 return -EEXIST;368 return -EEXIST;
398 369
399 domain->iova_cookie = cookie_alloc(IOMMU_DMA_IOVA_COOKIE);370 cookie = kzalloc(sizeof(*cookie), GFP_KERNEL);
400 if (!domain->iova_cookie)371 if (!cookie)
401 return -ENOMEM;372 return -ENOMEM;
402 373
403 iommu_domain_set_sw_msi(domain, iommu_dma_sw_msi);374 INIT_LIST_HEAD(&cookie->msi_page_list);
375 domain->cookie_type = IOMMU_COOKIE_DMA_IOVA;
376 domain->iova_cookie = cookie;
404 return 0;377 return 0;
405}378}
406 379
@@ -418,54 +391,56 @@ int iommu_get_dma_cookie(struct iommu_domain *domain) · 11 unmodified lines
418 */391 */
419int iommu_get_msi_cookie(struct iommu_domain *domain, dma_addr_t base)392int iommu_get_msi_cookie(struct iommu_domain *domain, dma_addr_t base)
420{393{
421 struct iommu_dma_cookie *cookie;394 struct iommu_dma_msi_cookie *cookie;
422 395
423 if (domain->type != IOMMU_DOMAIN_UNMANAGED)396 if (domain->type != IOMMU_DOMAIN_UNMANAGED)
424 return -EINVAL;397 return -EINVAL;
425 398
426 if (domain->iova_cookie)399 if (domain->cookie_type != IOMMU_COOKIE_NONE)
427 return -EEXIST;400 return -EEXIST;
428 401
429 cookie = cookie_alloc(IOMMU_DMA_MSI_COOKIE);402 cookie = kzalloc(sizeof(*cookie), GFP_KERNEL);
430 if (!cookie)403 if (!cookie)
431 return -ENOMEM;404 return -ENOMEM;
432 405
433 cookie->msi_iova = base;406 cookie->msi_iova = base;
434 domain->iova_cookie = cookie;407 INIT_LIST_HEAD(&cookie->msi_page_list);
435 iommu_domain_set_sw_msi(domain, iommu_dma_sw_msi);408 domain->cookie_type = IOMMU_COOKIE_DMA_MSI;
409 domain->msi_cookie = cookie;
436 return 0;410 return 0;
437}411}
438EXPORT_SYMBOL(iommu_get_msi_cookie);412EXPORT_SYMBOL(iommu_get_msi_cookie);
439 413
440/**414/**
441 * iommu_put_dma_cookie - Release a domain's DMA mapping resources415 * iommu_put_dma_cookie - Release a domain's DMA mapping resources
442 * @domain: IOMMU domain previously prepared by iommu_get_dma_cookie() or416 * @domain: IOMMU domain previously prepared by iommu_get_dma_cookie()
443 * iommu_get_msi_cookie()
444 */417 */
445void iommu_put_dma_cookie(struct iommu_domain *domain)418void iommu_put_dma_cookie(struct iommu_domain *domain)
446{419{
447 struct iommu_dma_cookie *cookie = domain->iova_cookie;420 struct iommu_dma_cookie *cookie = domain->iova_cookie;
448 struct iommu_dma_msi_page *msi, *tmp;421 struct iommu_dma_msi_page *msi, *tmp;
449 422
450#if IS_ENABLED(CONFIG_IRQ_MSI_IOMMU)423 if (cookie->iovad.granule) {
451 if (domain->sw_msi != iommu_dma_sw_msi)
452 return;
453#endif
454
455 if (!cookie)
456 return;
457
458 if (cookie->type == IOMMU_DMA_IOVA_COOKIE && cookie->iovad.granule) {
459 iommu_dma_free_fq(cookie);424 iommu_dma_free_fq(cookie);
460 put_iova_domain(&cookie->iovad);425 put_iova_domain(&cookie->iovad);
461 }426 }
427 list_for_each_entry_safe(msi, tmp, &cookie->msi_page_list, list)
428 kfree(msi);
429 kfree(cookie);
430}
431
432/**
433 * iommu_put_msi_cookie - Release a domain's MSI mapping resources
434 * @domain: IOMMU domain previously prepared by iommu_get_msi_cookie()
435 */
436void iommu_put_msi_cookie(struct iommu_domain *domain)
437{
438 struct iommu_dma_msi_cookie *cookie = domain->msi_cookie;
439 struct iommu_dma_msi_page *msi, *tmp;
462 440
463 list_for_each_entry_safe(msi, tmp, &cookie->msi_page_list, list) {441 list_for_each_entry_safe(msi, tmp, &cookie->msi_page_list, list)
464 list_del(&msi->list);
465 kfree(msi);442 kfree(msi);
466 }
467 kfree(cookie);443 kfree(cookie);
468 domain->iova_cookie = NULL;
469}444}
470 445
471/**446/**
@@ -685,7 +660,7 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, struct device *dev · 213 unmodified lines
685 struct iova_domain *iovad;660 struct iova_domain *iovad;
686 int ret;661 int ret;
687 662
688 if (!cookie || cookie->type != IOMMU_DMA_IOVA_COOKIE)663 if (!cookie || domain->cookie_type != IOMMU_COOKIE_DMA_IOVA)
689 return -EINVAL;664 return -EINVAL;
690 665
691 iovad = &cookie->iovad;666 iovad = &cookie->iovad;
@@ -768,9 +743,9 @@ static dma_addr_t iommu_dma_alloc_iova(struct iommu_domain *domain, · 76 unmodified lines
768 struct iova_domain *iovad = &cookie->iovad;743 struct iova_domain *iovad = &cookie->iovad;
769 unsigned long shift, iova_len, iova;744 unsigned long shift, iova_len, iova;
770 745
771 if (cookie->type == IOMMU_DMA_MSI_COOKIE) {746 if (domain->cookie_type == IOMMU_COOKIE_DMA_MSI) {
772 cookie->msi_iova += size;747 domain->msi_cookie->msi_iova += size;
773 return cookie->msi_iova - size;748 return domain->msi_cookie->msi_iova - size;
774 }749 }
775 750
776 shift = iova_shift(iovad);751 shift = iova_shift(iovad);
@@ -807,16 +782,16 @@ done: · 30 unmodified lines
807 return (dma_addr_t)iova << shift;782 return (dma_addr_t)iova << shift;
808}783}
809 784
810static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie,785static void iommu_dma_free_iova(struct iommu_domain *domain, dma_addr_t iova,
811 dma_addr_t iova, size_t size, struct iommu_iotlb_gather *gather)786 size_t size, struct iommu_iotlb_gather *gather)
812{787{
813 struct iova_domain *iovad = &cookie->iovad;788 struct iova_domain *iovad = &domain->iova_cookie->iovad;
814 789
815 /* The MSI case is only ever cleaning up its most recent allocation */790 /* The MSI case is only ever cleaning up its most recent allocation */
816 if (cookie->type == IOMMU_DMA_MSI_COOKIE)791 if (domain->cookie_type == IOMMU_COOKIE_DMA_MSI)
817 cookie->msi_iova -= size;792 domain->msi_cookie->msi_iova -= size;
818 else if (gather && gather->queued)793 else if (gather && gather->queued)
819 queue_iova(cookie, iova_pfn(iovad, iova),794 queue_iova(domain->iova_cookie, iova_pfn(iovad, iova),
820 size >> iova_shift(iovad),795 size >> iova_shift(iovad),
821 &gather->freelist);796 &gather->freelist);
822 else797 else
@@ -844,7 +819,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, · 21 unmodified lines
844 819
845 if (!iotlb_gather.queued)820 if (!iotlb_gather.queued)
846 iommu_iotlb_sync(domain, &iotlb_gather);821 iommu_iotlb_sync(domain, &iotlb_gather);
847 iommu_dma_free_iova(cookie, dma_addr, size, &iotlb_gather);822 iommu_dma_free_iova(domain, dma_addr, size, &iotlb_gather);
848}823}
849 824
850static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys,825static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys,
@@ -872,7 +847,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, · 21 unmodified lines
872 return DMA_MAPPING_ERROR;847 return DMA_MAPPING_ERROR;
873 848
874 if (iommu_map(domain, iova, phys - iova_off, size, prot, GFP_ATOMIC)) {849 if (iommu_map(domain, iova, phys - iova_off, size, prot, GFP_ATOMIC)) {
875 iommu_dma_free_iova(cookie, iova, size, NULL);850 iommu_dma_free_iova(domain, iova, size, NULL);
876 return DMA_MAPPING_ERROR;851 return DMA_MAPPING_ERROR;
877 }852 }
878 return iova + iova_off;853 return iova + iova_off;
@@ -1009,7 +984,7 @@ static struct page **__iommu_dma_alloc_noncontiguous(struct device *dev, · 130 unmodified lines
1009out_free_sg:984out_free_sg:
1010 sg_free_table(sgt);985 sg_free_table(sgt);
1011out_free_iova:986out_free_iova:
1012 iommu_dma_free_iova(cookie, iova, size, NULL);987 iommu_dma_free_iova(domain, iova, size, NULL);
1013out_free_pages:988out_free_pages:
1014 __iommu_dma_free_pages(pages, count);989 __iommu_dma_free_pages(pages, count);
1015 return NULL;990 return NULL;
@@ -1486,7 +1461,7 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents, · 470 unmodified lines
1486 return __finalise_sg(dev, sg, nents, iova);1461 return __finalise_sg(dev, sg, nents, iova);
1487 1462
1488out_free_iova:1463out_free_iova:
1489 iommu_dma_free_iova(cookie, iova, iova_len, NULL);1464 iommu_dma_free_iova(domain, iova, iova_len, NULL);
1490out_restore_sg:1465out_restore_sg:
1491 __invalidate_sg(sg, nents);1466 __invalidate_sg(sg, nents);
1492out:1467out:
@@ -1764,17 +1739,47 @@ out_err: · 271 unmodified lines
1764 dev->dma_iommu = false;1739 dev->dma_iommu = false;
1765}1740}
1766 1741
1742static bool has_msi_cookie(const struct iommu_domain *domain)
1743{
1744 return domain && (domain->cookie_type == IOMMU_COOKIE_DMA_IOVA ||
1745 domain->cookie_type == IOMMU_COOKIE_DMA_MSI);
1746}
1747
1748static size_t cookie_msi_granule(const struct iommu_domain *domain)
1749{
1750 switch (domain->cookie_type) {
1751 case IOMMU_COOKIE_DMA_IOVA:
1752 return domain->iova_cookie->iovad.granule;
1753 case IOMMU_COOKIE_DMA_MSI:
1754 return PAGE_SIZE;
1755 default:
1756 BUG();
1757 };
1758}
1759
1760static struct list_head *cookie_msi_pages(const struct iommu_domain *domain)
1761{
1762 switch (domain->cookie_type) {
1763 case IOMMU_COOKIE_DMA_IOVA:
1764 return &domain->iova_cookie->msi_page_list;
1765 case IOMMU_COOKIE_DMA_MSI:
1766 return &domain->msi_cookie->msi_page_list;
1767 default:
1768 BUG();
1769 };
1770}
1771
1767static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev,1772static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev,
1768 phys_addr_t msi_addr, struct iommu_domain *domain)1773 phys_addr_t msi_addr, struct iommu_domain *domain)
1769{1774{
1770 struct iommu_dma_cookie *cookie = domain->iova_cookie;1775 struct list_head *msi_page_list = cookie_msi_pages(domain);
1771 struct iommu_dma_msi_page *msi_page;1776 struct iommu_dma_msi_page *msi_page;
1772 dma_addr_t iova;1777 dma_addr_t iova;
1773 int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;1778 int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
1774 size_t size = cookie_msi_granule(cookie);1779 size_t size = cookie_msi_granule(domain);
1775 1780
1776 msi_addr &= ~(phys_addr_t)(size - 1);1781 msi_addr &= ~(phys_addr_t)(size - 1);
1777 list_for_each_entry(msi_page, &cookie->msi_page_list, list)1782 list_for_each_entry(msi_page, msi_page_list, list)
1778 if (msi_page->phys == msi_addr)1783 if (msi_page->phys == msi_addr)
1779 return msi_page;1784 return msi_page;
1780 1785
@@ -1792,23 +1797,23 @@ static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev, · 11 unmodified lines
1792 INIT_LIST_HEAD(&msi_page->list);1797 INIT_LIST_HEAD(&msi_page->list);
1793 msi_page->phys = msi_addr;1798 msi_page->phys = msi_addr;
1794 msi_page->iova = iova;1799 msi_page->iova = iova;
1795 list_add(&msi_page->list, &cookie->msi_page_list);1800 list_add(&msi_page->list, msi_page_list);
1796 return msi_page;1801 return msi_page;
1797 1802
1798out_free_iova:1803out_free_iova:
1799 iommu_dma_free_iova(cookie, iova, size, NULL);1804 iommu_dma_free_iova(domain, iova, size, NULL);
1800out_free_page:1805out_free_page:
1801 kfree(msi_page);1806 kfree(msi_page);
1802 return NULL;1807 return NULL;
1803}1808}
1804 1809
1805static int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,1810int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
1806 phys_addr_t msi_addr)1811 phys_addr_t msi_addr)
1807{1812{
1808 struct device *dev = msi_desc_to_dev(desc);1813 struct device *dev = msi_desc_to_dev(desc);
1809 const struct iommu_dma_msi_page *msi_page;1814 const struct iommu_dma_msi_page *msi_page;
1810 1815
1811 if (!domain->iova_cookie) {1816 if (!has_msi_cookie(domain)) {
1812 msi_desc_set_iommu_msi_iova(desc, 0, 0);1817 msi_desc_set_iommu_msi_iova(desc, 0, 0);
1813 return 0;1818 return 0;
1814 }1819 }
@@ -1818,9 +1823,8 @@ static int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, · 3 unmodified lines
1818 if (!msi_page)1823 if (!msi_page)
1819 return -ENOMEM;1824 return -ENOMEM;
1820 1825
1821 msi_desc_set_iommu_msi_iova(1826 msi_desc_set_iommu_msi_iova(desc, msi_page->iova,
1822 desc, msi_page->iova,1827 ilog2(cookie_msi_granule(domain)));
1823 ilog2(cookie_msi_granule(domain->iova_cookie)));
1824 return 0;1828 return 0;
1825}1829}
1826 1830
drivers/iommu/dma-iommu.h+14−0
@@ -13,11 +13,15 @@ void iommu_setup_dma_ops(struct device *dev); · 12 unmodified lines
13 13
14int iommu_get_dma_cookie(struct iommu_domain *domain);14int iommu_get_dma_cookie(struct iommu_domain *domain);
15void iommu_put_dma_cookie(struct iommu_domain *domain);15void iommu_put_dma_cookie(struct iommu_domain *domain);
16void iommu_put_msi_cookie(struct iommu_domain *domain);
16 17
17int iommu_dma_init_fq(struct iommu_domain *domain);18int iommu_dma_init_fq(struct iommu_domain *domain);
18 19
19void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list);20void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list);
20 21
22int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
23 phys_addr_t msi_addr);
24
21extern bool iommu_dma_forcedac;25extern bool iommu_dma_forcedac;
22 26
23#else /* CONFIG_IOMMU_DMA */27#else /* CONFIG_IOMMU_DMA */
@@ -40,9 +44,19 @@ static inline void iommu_put_dma_cookie(struct iommu_domain *domain) · 16 unmodified lines
40{44{
41}45}
42 46
47static inline void iommu_put_msi_cookie(struct iommu_domain *domain)
48{
49}
50
43static inline void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list)51static inline void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list)
44{52{
45}53}
46 54
55static inline int iommu_dma_sw_msi(struct iommu_domain *domain,
56 struct msi_desc *desc, phys_addr_t msi_addr)
57{
58 return -ENODEV;
59}
60
47#endif /* CONFIG_IOMMU_DMA */61#endif /* CONFIG_IOMMU_DMA */
48#endif /* __DMA_IOMMU_H */62#endif /* __DMA_IOMMU_H */
drivers/iommu/intel/iommu.c+2−1
@@ -3383,7 +3383,8 @@ intel_iommu_domain_alloc_paging_flags(struct device *dev, u32 flags, · 3382 unmodified lines
3383 bool first_stage;3383 bool first_stage;
3384 3384
3385 if (flags &3385 if (flags &
3386 (~(IOMMU_HWPT_ALLOC_NEST_PARENT | IOMMU_HWPT_ALLOC_DIRTY_TRACKING)))3386 (~(IOMMU_HWPT_ALLOC_NEST_PARENT | IOMMU_HWPT_ALLOC_DIRTY_TRACKING |
3387 IOMMU_HWPT_ALLOC_PASID)))
3387 return ERR_PTR(-EOPNOTSUPP);3388 return ERR_PTR(-EOPNOTSUPP);
3388 if (nested_parent && !nested_supported(iommu))3389 if (nested_parent && !nested_supported(iommu))
3389 return ERR_PTR(-EOPNOTSUPP);3390 return ERR_PTR(-EOPNOTSUPP);
drivers/iommu/intel/nested.c+1−1
@@ -198,7 +198,7 @@ intel_iommu_domain_alloc_nested(struct device *dev, struct iommu_domain *parent, · 197 unmodified lines
198 struct dmar_domain *domain;198 struct dmar_domain *domain;
199 int ret;199 int ret;
200 200
201 if (!nested_supported(iommu) || flags)201 if (!nested_supported(iommu) || flags & ~IOMMU_HWPT_ALLOC_PASID)
202 return ERR_PTR(-EOPNOTSUPP);202 return ERR_PTR(-EOPNOTSUPP);
203 203
204 /* Must be nested domain */204 /* Must be nested domain */
drivers/iommu/iommu-priv.h+16−0
@@ -5,6 +5,7 @@ · 4 unmodified lines
5#define __LINUX_IOMMU_PRIV_H5#define __LINUX_IOMMU_PRIV_H
6 6
7#include <linux/iommu.h>7#include <linux/iommu.h>
8#include <linux/msi.h>
8 9
9static inline const struct iommu_ops *dev_iommu_ops(struct device *dev)10static inline const struct iommu_ops *dev_iommu_ops(struct device *dev)
10{11{
@@ -47,4 +48,19 @@ void iommu_detach_group_handle(struct iommu_domain *domain, · 36 unmodified lines
47int iommu_replace_group_handle(struct iommu_group *group,48int iommu_replace_group_handle(struct iommu_group *group,
48 struct iommu_domain *new_domain,49 struct iommu_domain *new_domain,
49 struct iommu_attach_handle *handle);50 struct iommu_attach_handle *handle);
51
52#if IS_ENABLED(CONFIG_IOMMUFD_DRIVER_CORE) && IS_ENABLED(CONFIG_IRQ_MSI_IOMMU)
53int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
54 phys_addr_t msi_addr);
55#else /* !CONFIG_IOMMUFD_DRIVER_CORE || !CONFIG_IRQ_MSI_IOMMU */
56static inline int iommufd_sw_msi(struct iommu_domain *domain,
57 struct msi_desc *desc, phys_addr_t msi_addr)
58{
59 return -EOPNOTSUPP;
60}
61#endif /* CONFIG_IOMMUFD_DRIVER_CORE && CONFIG_IRQ_MSI_IOMMU */
62
63int iommu_replace_device_pasid(struct iommu_domain *domain,
64 struct device *dev, ioasid_t pasid,
65 struct iommu_attach_handle *handle);
50#endif /* __LINUX_IOMMU_PRIV_H */66#endif /* __LINUX_IOMMU_PRIV_H */
drivers/iommu/iommu-sva.c+1−0
@@ -310,6 +310,7 @@ static struct iommu_domain *iommu_sva_domain_alloc(struct device *dev, · 309 unmodified lines
310 }310 }
311 311
312 domain->type = IOMMU_DOMAIN_SVA;312 domain->type = IOMMU_DOMAIN_SVA;
313 domain->cookie_type = IOMMU_COOKIE_SVA;
313 mmgrab(mm);314 mmgrab(mm);
314 domain->mm = mm;315 domain->mm = mm;
315 domain->owner = ops;316 domain->owner = ops;
drivers/iommu/iommu.c+151−9
@@ -18,6 +18,7 @@ · 17 unmodified lines
18#include <linux/errno.h>18#include <linux/errno.h>
19#include <linux/host1x_context_bus.h>19#include <linux/host1x_context_bus.h>
20#include <linux/iommu.h>20#include <linux/iommu.h>
21#include <linux/iommufd.h>
21#include <linux/idr.h>22#include <linux/idr.h>
22#include <linux/err.h>23#include <linux/err.h>
23#include <linux/pci.h>24#include <linux/pci.h>
@@ -539,6 +540,13 @@ static void iommu_deinit_device(struct device *dev) · 515 unmodified lines
539 dev_iommu_free(dev);540 dev_iommu_free(dev);
540}541}
541 542
543static struct iommu_domain *pasid_array_entry_to_domain(void *entry)
544{
545 if (xa_pointer_tag(entry) == IOMMU_PASID_ARRAY_DOMAIN)
546 return xa_untag_pointer(entry);
547 return ((struct iommu_attach_handle *)xa_untag_pointer(entry))->domain;
548}
549
542DEFINE_MUTEX(iommu_probe_device_lock);550DEFINE_MUTEX(iommu_probe_device_lock);
543 551
544static int __iommu_probe_device(struct device *dev, struct list_head *group_list)552static int __iommu_probe_device(struct device *dev, struct list_head *group_list)
@@ -1973,8 +1981,10 @@ void iommu_set_fault_handler(struct iommu_domain *domain, · 1428 unmodified lines
1973 iommu_fault_handler_t handler,1981 iommu_fault_handler_t handler,
1974 void *token)1982 void *token)
1975{1983{
1976 BUG_ON(!domain);1984 if (WARN_ON(!domain || domain->cookie_type != IOMMU_COOKIE_NONE))
1985 return;
1977 1986
1987 domain->cookie_type = IOMMU_COOKIE_FAULT_HANDLER;
1978 domain->handler = handler;1988 domain->handler = handler;
1979 domain->handler_token = token;1989 domain->handler_token = token;
1980}1990}
@@ -2044,9 +2054,19 @@ EXPORT_SYMBOL_GPL(iommu_paging_domain_alloc_flags); · 63 unmodified lines
2044 2054
2045void iommu_domain_free(struct iommu_domain *domain)2055void iommu_domain_free(struct iommu_domain *domain)
2046{2056{
2047 if (domain->type == IOMMU_DOMAIN_SVA)2057 switch (domain->cookie_type) {
2058 case IOMMU_COOKIE_DMA_IOVA:
2059 iommu_put_dma_cookie(domain);
2060 break;
2061 case IOMMU_COOKIE_DMA_MSI:
2062 iommu_put_msi_cookie(domain);
2063 break;
2064 case IOMMU_COOKIE_SVA:
2048 mmdrop(domain->mm);2065 mmdrop(domain->mm);
2049 iommu_put_dma_cookie(domain);2066 break;
2067 default:
2068 break;
2069 }
2050 if (domain->ops->free)2070 if (domain->ops->free)
2051 domain->ops->free(domain);2071 domain->ops->free(domain);
2052}2072}
@@ -3335,14 +3355,15 @@ static void iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid, · 1282 unmodified lines
3335}3355}
3336 3356
3337static int __iommu_set_group_pasid(struct iommu_domain *domain,3357static int __iommu_set_group_pasid(struct iommu_domain *domain,
3338 struct iommu_group *group, ioasid_t pasid)3358 struct iommu_group *group, ioasid_t pasid,
3359 struct iommu_domain *old)
3339{3360{
3340 struct group_device *device, *last_gdev;3361 struct group_device *device, *last_gdev;
3341 int ret;3362 int ret;
3342 3363
3343 for_each_group_device(group, device) {3364 for_each_group_device(group, device) {
3344 ret = domain->ops->set_dev_pasid(domain, device->dev,3365 ret = domain->ops->set_dev_pasid(domain, device->dev,
3345 pasid, NULL);3366 pasid, old);
3346 if (ret)3367 if (ret)
3347 goto err_revert;3368 goto err_revert;
3348 }3369 }
@@ -3354,7 +3375,15 @@ err_revert: · 5 unmodified lines
3354 for_each_group_device(group, device) {3375 for_each_group_device(group, device) {
3355 if (device == last_gdev)3376 if (device == last_gdev)
3356 break;3377 break;
3357 iommu_remove_dev_pasid(device->dev, pasid, domain);3378 /*
3379 * If no old domain, undo the succeeded devices/pasid.
3380 * Otherwise, rollback the succeeded devices/pasid to the old
3381 * domain. And it is a driver bug to fail attaching with a
3382 * previously good domain.
3383 */
3384 if (!old || WARN_ON(old->ops->set_dev_pasid(old, device->dev,
3385 pasid, domain)))
3386 iommu_remove_dev_pasid(device->dev, pasid, domain);
3358 }3387 }
3359 return ret;3388 return ret;
3360}3389}
@@ -3376,6 +3405,9 @@ static void __iommu_remove_group_pasid(struct iommu_group *group, · 15 unmodified lines
3376 * @pasid: the pasid of the device.3405 * @pasid: the pasid of the device.
3377 * @handle: the attach handle.3406 * @handle: the attach handle.
3378 *3407 *
3408 * Caller should always provide a new handle to avoid race with the paths
3409 * that have lockless reference to handle if it intends to pass a valid handle.
3410 *
3379 * Return: 0 on success, or an error.3411 * Return: 0 on success, or an error.
3380 */3412 */
3381int iommu_attach_device_pasid(struct iommu_domain *domain,3413int iommu_attach_device_pasid(struct iommu_domain *domain,
@@ -3420,7 +3452,7 @@ int iommu_attach_device_pasid(struct iommu_domain *domain, · 38 unmodified lines
3420 if (ret)3452 if (ret)
3421 goto out_unlock;3453 goto out_unlock;
3422 3454
3423 ret = __iommu_set_group_pasid(domain, group, pasid);3455 ret = __iommu_set_group_pasid(domain, group, pasid, NULL);
3424 if (ret) {3456 if (ret) {
3425 xa_release(&group->pasid_array, pasid);3457 xa_release(&group->pasid_array, pasid);
3426 goto out_unlock;3458 goto out_unlock;
@@ -3441,6 +3473,97 @@ out_unlock: · 14 unmodified lines
3441}3473}
3442EXPORT_SYMBOL_GPL(iommu_attach_device_pasid);3474EXPORT_SYMBOL_GPL(iommu_attach_device_pasid);
3443 3475
3476/**
3477 * iommu_replace_device_pasid - Replace the domain that a specific pasid
3478 * of the device is attached to
3479 * @domain: the new iommu domain
3480 * @dev: the attached device.
3481 * @pasid: the pasid of the device.
3482 * @handle: the attach handle.
3483 *
3484 * This API allows the pasid to switch domains. The @pasid should have been
3485 * attached. Otherwise, this fails. The pasid will keep the old configuration
3486 * if replacement failed.
3487 *
3488 * Caller should always provide a new handle to avoid race with the paths
3489 * that have lockless reference to handle if it intends to pass a valid handle.
3490 *
3491 * Return 0 on success, or an error.
3492 */
3493int iommu_replace_device_pasid(struct iommu_domain *domain,
3494 struct device *dev, ioasid_t pasid,
3495 struct iommu_attach_handle *handle)
3496{
3497 /* Caller must be a probed driver on dev */
3498 struct iommu_group *group = dev->iommu_group;
3499 struct iommu_attach_handle *entry;
3500 struct iommu_domain *curr_domain;
3501 void *curr;
3502 int ret;
3503
3504 if (!group)
3505 return -ENODEV;
3506
3507 if (!domain->ops->set_dev_pasid)
3508 return -EOPNOTSUPP;
3509
3510 if (dev_iommu_ops(dev) != domain->owner ||
3511 pasid == IOMMU_NO_PASID || !handle)
3512 return -EINVAL;
3513
3514 mutex_lock(&group->mutex);
3515 entry = iommu_make_pasid_array_entry(domain, handle);
3516 curr = xa_cmpxchg(&group->pasid_array, pasid, NULL,
3517 XA_ZERO_ENTRY, GFP_KERNEL);
3518 if (xa_is_err(curr)) {
3519 ret = xa_err(curr);
3520 goto out_unlock;
3521 }
3522
3523 /*
3524 * No domain (with or without handle) attached, hence not
3525 * a replace case.
3526 */
3527 if (!curr) {
3528 xa_release(&group->pasid_array, pasid);
3529 ret = -EINVAL;
3530 goto out_unlock;
3531 }
3532
3533 /*
3534 * Reusing handle is problematic as there are paths that refers
3535 * the handle without lock. To avoid race, reject the callers that
3536 * attempt it.
3537 */
3538 if (curr == entry) {
3539 WARN_ON(1);
3540 ret = -EINVAL;
3541 goto out_unlock;
3542 }
3543
3544 curr_domain = pasid_array_entry_to_domain(curr);
3545 ret = 0;
3546
3547 if (curr_domain != domain) {
3548 ret = __iommu_set_group_pasid(domain, group,
3549 pasid, curr_domain);
3550 if (ret)
3551 goto out_unlock;
3552 }
3553
3554 /*
3555 * The above xa_cmpxchg() reserved the memory, and the
3556 * group->mutex is held, this cannot fail.
3557 */
3558 WARN_ON(xa_is_err(xa_store(&group->pasid_array,
3559 pasid, entry, GFP_KERNEL)));
3560
3561out_unlock:
3562 mutex_unlock(&group->mutex);
3563 return ret;
3564}
3565EXPORT_SYMBOL_NS_GPL(iommu_replace_device_pasid, "IOMMUFD_INTERNAL");
3566
3444/*3567/*
3445 * iommu_detach_device_pasid() - Detach the domain from pasid of device3568 * iommu_detach_device_pasid() - Detach the domain from pasid of device
3446 * @domain: the iommu domain.3569 * @domain: the iommu domain.
@@ -3536,6 +3659,9 @@ EXPORT_SYMBOL_NS_GPL(iommu_attach_handle_get, "IOMMUFD_INTERNAL"); · 89 unmodified lines
3536 * This is a variant of iommu_attach_group(). It allows the caller to provide3659 * This is a variant of iommu_attach_group(). It allows the caller to provide
3537 * an attach handle and use it when the domain is attached. This is currently3660 * an attach handle and use it when the domain is attached. This is currently
3538 * used by IOMMUFD to deliver the I/O page faults.3661 * used by IOMMUFD to deliver the I/O page faults.
3662 *
3663 * Caller should always provide a new handle to avoid race with the paths
3664 * that have lockless reference to handle.
3539 */3665 */
3540int iommu_attach_group_handle(struct iommu_domain *domain,3666int iommu_attach_group_handle(struct iommu_domain *domain,
3541 struct iommu_group *group,3667 struct iommu_group *group,
@@ -3605,6 +3731,9 @@ EXPORT_SYMBOL_NS_GPL(iommu_detach_group_handle, "IOMMUFD_INTERNAL"); · 63 unmodified lines
3605 *3731 *
3606 * If the currently attached domain is a core domain (e.g. a default_domain),3732 * If the currently attached domain is a core domain (e.g. a default_domain),
3607 * it will act just like the iommu_attach_group_handle().3733 * it will act just like the iommu_attach_group_handle().
3734 *
3735 * Caller should always provide a new handle to avoid race with the paths
3736 * that have lockless reference to handle.
3608 */3737 */
3609int iommu_replace_group_handle(struct iommu_group *group,3738int iommu_replace_group_handle(struct iommu_group *group,
3610 struct iommu_domain *new_domain,3739 struct iommu_domain *new_domain,
@@ -3662,8 +3791,21 @@ int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_addr) · 51 unmodified lines
3662 return 0;3791 return 0;
3663 3792
3664 mutex_lock(&group->mutex);3793 mutex_lock(&group->mutex);
3665 if (group->domain && group->domain->sw_msi)3794 /* An IDENTITY domain must pass through */
3666 ret = group->domain->sw_msi(group->domain, desc, msi_addr);3795 if (group->domain && group->domain->type != IOMMU_DOMAIN_IDENTITY) {
3796 switch (group->domain->cookie_type) {
3797 case IOMMU_COOKIE_DMA_MSI:
3798 case IOMMU_COOKIE_DMA_IOVA:
3799 ret = iommu_dma_sw_msi(group->domain, desc, msi_addr);
3800 break;
3801 case IOMMU_COOKIE_IOMMUFD:
3802 ret = iommufd_sw_msi(group->domain, desc, msi_addr);
3803 break;
3804 default:
3805 ret = -EOPNOTSUPP;
3806 break;
3807 }
3808 }
3667 mutex_unlock(&group->mutex);3809 mutex_unlock(&group->mutex);
3668 return ret;3810 return ret;
3669}3811}
drivers/iommu/iommufd/Kconfig+1−1
@@ -1,6 +1,6 @@
1# SPDX-License-Identifier: GPL-2.0-only1# SPDX-License-Identifier: GPL-2.0-only
2config IOMMUFD_DRIVER_CORE2config IOMMUFD_DRIVER_CORE
3 tristate3 bool
4 default (IOMMUFD_DRIVER || IOMMUFD) if IOMMUFD!=n4 default (IOMMUFD_DRIVER || IOMMUFD) if IOMMUFD!=n
5 5
6config IOMMUFD6config IOMMUFD
drivers/iommu/iommufd/Makefile+1−1
@@ -1,7 +1,7 @@
1# SPDX-License-Identifier: GPL-2.0-only1# SPDX-License-Identifier: GPL-2.0-only
2iommufd-y := \2iommufd-y := \
3 device.o \3 device.o \
4 fault.o \4 eventq.o \
5 hw_pagetable.o \5 hw_pagetable.o \
6 io_pagetable.o \6 io_pagetable.o \
7 ioas.o \7 ioas.o \
drivers/iommu/iommufd/device.c+294−205
@@ -3,9 +3,9 @@ · 2 unmodified lines
3 */3 */
4#include <linux/iommu.h>4#include <linux/iommu.h>
5#include <linux/iommufd.h>5#include <linux/iommufd.h>
6#include <linux/pci-ats.h>
6#include <linux/slab.h>7#include <linux/slab.h>
7#include <uapi/linux/iommufd.h>8#include <uapi/linux/iommufd.h>
8#include <linux/msi.h>
9 9
10#include "../iommu-priv.h"10#include "../iommu-priv.h"
11#include "io_pagetable.h"11#include "io_pagetable.h"
@@ -18,12 +18,17 @@ MODULE_PARM_DESC( · 6 unmodified lines
18 "Allow IOMMUFD to bind to devices even if the platform cannot isolate "18 "Allow IOMMUFD to bind to devices even if the platform cannot isolate "
19 "the MSI interrupt window. Enabling this is a security weakness.");19 "the MSI interrupt window. Enabling this is a security weakness.");
20 20
21struct iommufd_attach {
22 struct iommufd_hw_pagetable *hwpt;
23 struct xarray device_array;
24};
25
21static void iommufd_group_release(struct kref *kref)26static void iommufd_group_release(struct kref *kref)
22{27{
23 struct iommufd_group *igroup =28 struct iommufd_group *igroup =
24 container_of(kref, struct iommufd_group, ref);29 container_of(kref, struct iommufd_group, ref);
25 30
26 WARN_ON(igroup->hwpt || !list_empty(&igroup->device_list));31 WARN_ON(!xa_empty(&igroup->pasid_attach));
27 32
28 xa_cmpxchg(&igroup->ictx->groups, iommu_group_id(igroup->group), igroup,33 xa_cmpxchg(&igroup->ictx->groups, iommu_group_id(igroup->group), igroup,
29 NULL, GFP_KERNEL);34 NULL, GFP_KERNEL);
@@ -90,7 +95,7 @@ static struct iommufd_group *iommufd_get_group(struct iommufd_ctx *ictx, · 60 unmodified lines
90 95
91 kref_init(&new_igroup->ref);96 kref_init(&new_igroup->ref);
92 mutex_init(&new_igroup->lock);97 mutex_init(&new_igroup->lock);
93 INIT_LIST_HEAD(&new_igroup->device_list);98 xa_init(&new_igroup->pasid_attach);
94 new_igroup->sw_msi_start = PHYS_ADDR_MAX;99 new_igroup->sw_msi_start = PHYS_ADDR_MAX;
95 /* group reference moves into new_igroup */100 /* group reference moves into new_igroup */
96 new_igroup->group = group;101 new_igroup->group = group;
@@ -294,129 +299,24 @@ u32 iommufd_device_to_id(struct iommufd_device *idev) · 197 unmodified lines
294}299}
295EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, "IOMMUFD");300EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, "IOMMUFD");
296 301
297/*302static unsigned int iommufd_group_device_num(struct iommufd_group *igroup,
298 * Get a iommufd_sw_msi_map for the msi physical address requested by the irq303 ioasid_t pasid)
299 * layer. The mapping to IOVA is global to the iommufd file descriptor, every
300 * domain that is attached to a device using the same MSI parameters will use
301 * the same IOVA.
302 */
303static __maybe_unused struct iommufd_sw_msi_map *
304iommufd_sw_msi_get_map(struct iommufd_ctx *ictx, phys_addr_t msi_addr,
305 phys_addr_t sw_msi_start)
306{
307 struct iommufd_sw_msi_map *cur;
308 unsigned int max_pgoff = 0;
309
310 lockdep_assert_held(&ictx->sw_msi_lock);
311
312 list_for_each_entry(cur, &ictx->sw_msi_list, sw_msi_item) {
313 if (cur->sw_msi_start != sw_msi_start)
314 continue;
315 max_pgoff = max(max_pgoff, cur->pgoff + 1);
316 if (cur->msi_addr == msi_addr)
317 return cur;
318 }
319
320 if (ictx->sw_msi_id >=
321 BITS_PER_BYTE * sizeof_field(struct iommufd_sw_msi_maps, bitmap))
322 return ERR_PTR(-EOVERFLOW);
323
324 cur = kzalloc(sizeof(*cur), GFP_KERNEL);
325 if (!cur)
326 return ERR_PTR(-ENOMEM);
327
328 cur->sw_msi_start = sw_msi_start;
329 cur->msi_addr = msi_addr;
330 cur->pgoff = max_pgoff;
331 cur->id = ictx->sw_msi_id++;
332 list_add_tail(&cur->sw_msi_item, &ictx->sw_msi_list);
333 return cur;
334}
335
336static int iommufd_sw_msi_install(struct iommufd_ctx *ictx,
337 struct iommufd_hwpt_paging *hwpt_paging,
338 struct iommufd_sw_msi_map *msi_map)
339{304{
340 unsigned long iova;305 struct iommufd_attach *attach;
341 306 struct iommufd_device *idev;
342 lockdep_assert_held(&ictx->sw_msi_lock);307 unsigned int count = 0;
308 unsigned long index;
343 309
344 iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE;310 lockdep_assert_held(&igroup->lock);
345 if (!test_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap)) {
346 int rc;
347 311
348 rc = iommu_map(hwpt_paging->common.domain, iova,312 attach = xa_load(&igroup->pasid_attach, pasid);
349 msi_map->msi_addr, PAGE_SIZE,313 if (attach)
350 IOMMU_WRITE | IOMMU_READ | IOMMU_MMIO,314 xa_for_each(&attach->device_array, index, idev)
351 GFP_KERNEL_ACCOUNT);315 count++;
352 if (rc)316 return count;
353 return rc;
354 __set_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap);
355 }
356 return 0;
357}317}
358 318
359/*
360 * Called by the irq code if the platform translates the MSI address through the
361 * IOMMU. msi_addr is the physical address of the MSI page. iommufd will
362 * allocate a fd global iova for the physical page that is the same on all
363 * domains and devices.
364 */
365#ifdef CONFIG_IRQ_MSI_IOMMU319#ifdef CONFIG_IRQ_MSI_IOMMU
366int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
367 phys_addr_t msi_addr)
368{
369 struct device *dev = msi_desc_to_dev(desc);
370 struct iommufd_hwpt_paging *hwpt_paging;
371 struct iommu_attach_handle *raw_handle;
372 struct iommufd_attach_handle *handle;
373 struct iommufd_sw_msi_map *msi_map;
374 struct iommufd_ctx *ictx;
375 unsigned long iova;
376 int rc;
377
378 /*
379 * It is safe to call iommu_attach_handle_get() here because the iommu
380 * core code invokes this under the group mutex which also prevents any
381 * change of the attach handle for the duration of this function.
382 */
383 iommu_group_mutex_assert(dev);
384
385 raw_handle =
386 iommu_attach_handle_get(dev->iommu_group, IOMMU_NO_PASID, 0);
387 if (IS_ERR(raw_handle))
388 return 0;
389 hwpt_paging = find_hwpt_paging(domain->iommufd_hwpt);
390
391 handle = to_iommufd_handle(raw_handle);
392 /* No IOMMU_RESV_SW_MSI means no change to the msi_msg */
393 if (handle->idev->igroup->sw_msi_start == PHYS_ADDR_MAX)
394 return 0;
395
396 ictx = handle->idev->ictx;
397 guard(mutex)(&ictx->sw_msi_lock);
398 /*
399 * The input msi_addr is the exact byte offset of the MSI doorbell, we
400 * assume the caller has checked that it is contained with a MMIO region
401 * that is secure to map at PAGE_SIZE.
402 */
403 msi_map = iommufd_sw_msi_get_map(handle->idev->ictx,
404 msi_addr & PAGE_MASK,
405 handle->idev->igroup->sw_msi_start);
406 if (IS_ERR(msi_map))
407 return PTR_ERR(msi_map);
408
409 rc = iommufd_sw_msi_install(ictx, hwpt_paging, msi_map);
410 if (rc)
411 return rc;
412 __set_bit(msi_map->id, handle->idev->igroup->required_sw_msi.bitmap);
413
414 iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE;
415 msi_desc_set_iommu_msi_iova(desc, iova, PAGE_SHIFT);
416 return 0;
417}
418#endif
419
420static int iommufd_group_setup_msi(struct iommufd_group *igroup,320static int iommufd_group_setup_msi(struct iommufd_group *igroup,
421 struct iommufd_hwpt_paging *hwpt_paging)321 struct iommufd_hwpt_paging *hwpt_paging)
422{322{
@@ -443,23 +343,39 @@ static int iommufd_group_setup_msi(struct iommufd_group *igroup, · 20 unmodified lines
443 }343 }
444 return 0;344 return 0;
445}345}
346#else
347static inline int
348iommufd_group_setup_msi(struct iommufd_group *igroup,
349 struct iommufd_hwpt_paging *hwpt_paging)
350{
351 return 0;
352}
353#endif
354
355static bool
356iommufd_group_first_attach(struct iommufd_group *igroup, ioasid_t pasid)
357{
358 lockdep_assert_held(&igroup->lock);
359 return !xa_load(&igroup->pasid_attach, pasid);
360}
446 361
447static int362static int
448iommufd_device_attach_reserved_iova(struct iommufd_device *idev,363iommufd_device_attach_reserved_iova(struct iommufd_device *idev,
449 struct iommufd_hwpt_paging *hwpt_paging)364 struct iommufd_hwpt_paging *hwpt_paging)
450{365{
366 struct iommufd_group *igroup = idev->igroup;
451 int rc;367 int rc;
452 368
453 lockdep_assert_held(&idev->igroup->lock);369 lockdep_assert_held(&igroup->lock);
454 370
455 rc = iopt_table_enforce_dev_resv_regions(&hwpt_paging->ioas->iopt,371 rc = iopt_table_enforce_dev_resv_regions(&hwpt_paging->ioas->iopt,
456 idev->dev,372 idev->dev,
457 &idev->igroup->sw_msi_start);373 &igroup->sw_msi_start);
458 if (rc)374 if (rc)
459 return rc;375 return rc;
460 376
461 if (list_empty(&idev->igroup->device_list)) {377 if (iommufd_group_first_attach(igroup, IOMMU_NO_PASID)) {
462 rc = iommufd_group_setup_msi(idev->igroup, hwpt_paging);378 rc = iommufd_group_setup_msi(igroup, hwpt_paging);
463 if (rc) {379 if (rc) {
464 iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt,380 iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt,
465 idev->dev);381 idev->dev);
@@ -471,13 +387,54 @@ iommufd_device_attach_reserved_iova(struct iommufd_device *idev, · 5 unmodified lines
471 387
472/* The device attach/detach/replace helpers for attach_handle */388/* The device attach/detach/replace helpers for attach_handle */
473 389
390static bool iommufd_device_is_attached(struct iommufd_device *idev,
391 ioasid_t pasid)
392{
393 struct iommufd_attach *attach;
394
395 attach = xa_load(&idev->igroup->pasid_attach, pasid);
396 return xa_load(&attach->device_array, idev->obj.id);
397}
398
399static int iommufd_hwpt_pasid_compat(struct iommufd_hw_pagetable *hwpt,
400 struct iommufd_device *idev,
401 ioasid_t pasid)
402{
403 struct iommufd_group *igroup = idev->igroup;
404
405 lockdep_assert_held(&igroup->lock);
406
407 if (pasid == IOMMU_NO_PASID) {
408 unsigned long start = IOMMU_NO_PASID;
409
410 if (!hwpt->pasid_compat &&
411 xa_find_after(&igroup->pasid_attach,
412 &start, UINT_MAX, XA_PRESENT))
413 return -EINVAL;
414 } else {
415 struct iommufd_attach *attach;
416
417 if (!hwpt->pasid_compat)
418 return -EINVAL;
419
420 attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID);
421 if (attach && attach->hwpt && !attach->hwpt->pasid_compat)
422 return -EINVAL;
423 }
424
425 return 0;
426}
427
474static int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *hwpt,428static int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *hwpt,
475 struct iommufd_device *idev)429 struct iommufd_device *idev,
430 ioasid_t pasid)
476{431{
477 struct iommufd_attach_handle *handle;432 struct iommufd_attach_handle *handle;
478 int rc;433 int rc;
479 434
480 lockdep_assert_held(&idev->igroup->lock);435 rc = iommufd_hwpt_pasid_compat(hwpt, idev, pasid);
436 if (rc)
437 return rc;
481 438
482 handle = kzalloc(sizeof(*handle), GFP_KERNEL);439 handle = kzalloc(sizeof(*handle), GFP_KERNEL);
483 if (!handle)440 if (!handle)
@@ -490,8 +447,12 @@ static int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *hwpt, · 6 unmodified lines
490 }447 }
491 448
492 handle->idev = idev;449 handle->idev = idev;
493 rc = iommu_attach_group_handle(hwpt->domain, idev->igroup->group,450 if (pasid == IOMMU_NO_PASID)
494 &handle->handle);451 rc = iommu_attach_group_handle(hwpt->domain, idev->igroup->group,
452 &handle->handle);
453 else
454 rc = iommu_attach_device_pasid(hwpt->domain, idev->dev, pasid,
455 &handle->handle);
495 if (rc)456 if (rc)
496 goto out_disable_iopf;457 goto out_disable_iopf;
497 458
@@ -506,26 +467,31 @@ out_free_handle: · 8 unmodified lines
506}467}
507 468
508static struct iommufd_attach_handle *469static struct iommufd_attach_handle *
509iommufd_device_get_attach_handle(struct iommufd_device *idev)470iommufd_device_get_attach_handle(struct iommufd_device *idev, ioasid_t pasid)
510{471{
511 struct iommu_attach_handle *handle;472 struct iommu_attach_handle *handle;
512 473
513 lockdep_assert_held(&idev->igroup->lock);474 lockdep_assert_held(&idev->igroup->lock);
514 475
515 handle =476 handle =
516 iommu_attach_handle_get(idev->igroup->group, IOMMU_NO_PASID, 0);477 iommu_attach_handle_get(idev->igroup->group, pasid, 0);
517 if (IS_ERR(handle))478 if (IS_ERR(handle))
518 return NULL;479 return NULL;
519 return to_iommufd_handle(handle);480 return to_iommufd_handle(handle);
520}481}
521 482
522static void iommufd_hwpt_detach_device(struct iommufd_hw_pagetable *hwpt,483static void iommufd_hwpt_detach_device(struct iommufd_hw_pagetable *hwpt,
523 struct iommufd_device *idev)484 struct iommufd_device *idev,
485 ioasid_t pasid)
524{486{
525 struct iommufd_attach_handle *handle;487 struct iommufd_attach_handle *handle;
526 488
527 handle = iommufd_device_get_attach_handle(idev);489 handle = iommufd_device_get_attach_handle(idev, pasid);
528 iommu_detach_group_handle(hwpt->domain, idev->igroup->group);490 if (pasid == IOMMU_NO_PASID)
491 iommu_detach_group_handle(hwpt->domain, idev->igroup->group);
492 else
493 iommu_detach_device_pasid(hwpt->domain, idev->dev, pasid);
494
529 if (hwpt->fault) {495 if (hwpt->fault) {
530 iommufd_auto_response_faults(hwpt, handle);496 iommufd_auto_response_faults(hwpt, handle);
531 iommufd_fault_iopf_disable(idev);497 iommufd_fault_iopf_disable(idev);
@@ -534,13 +500,19 @@ static void iommufd_hwpt_detach_device(struct iommufd_hw_pagetable *hwpt, · 2 unmodified lines
534}500}
535 501
536static int iommufd_hwpt_replace_device(struct iommufd_device *idev,502static int iommufd_hwpt_replace_device(struct iommufd_device *idev,
503 ioasid_t pasid,
537 struct iommufd_hw_pagetable *hwpt,504 struct iommufd_hw_pagetable *hwpt,
538 struct iommufd_hw_pagetable *old)505 struct iommufd_hw_pagetable *old)
539{506{
540 struct iommufd_attach_handle *handle, *old_handle =507 struct iommufd_attach_handle *handle, *old_handle;
541 iommufd_device_get_attach_handle(idev);
542 int rc;508 int rc;
543 509
510 rc = iommufd_hwpt_pasid_compat(hwpt, idev, pasid);
511 if (rc)
512 return rc;
513
514 old_handle = iommufd_device_get_attach_handle(idev, pasid);
515
544 handle = kzalloc(sizeof(*handle), GFP_KERNEL);516 handle = kzalloc(sizeof(*handle), GFP_KERNEL);
545 if (!handle)517 if (!handle)
546 return -ENOMEM;518 return -ENOMEM;
@@ -552,8 +524,12 @@ static int iommufd_hwpt_replace_device(struct iommufd_device *idev, · 5 unmodified lines
552 }524 }
553 525
554 handle->idev = idev;526 handle->idev = idev;
555 rc = iommu_replace_group_handle(idev->igroup->group, hwpt->domain,527 if (pasid == IOMMU_NO_PASID)
556 &handle->handle);528 rc = iommu_replace_group_handle(idev->igroup->group,
529 hwpt->domain, &handle->handle);
530 else
531 rc = iommu_replace_device_pasid(hwpt->domain, idev->dev,
532 pasid, &handle->handle);
557 if (rc)533 if (rc)
558 goto out_disable_iopf;534 goto out_disable_iopf;
559 535
@@ -575,22 +551,51 @@ out_free_handle: · 15 unmodified lines
575}551}
576 552
577int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,553int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,
578 struct iommufd_device *idev)554 struct iommufd_device *idev, ioasid_t pasid)
579{555{
580 struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt);556 struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt);
557 bool attach_resv = hwpt_paging && pasid == IOMMU_NO_PASID;
558 struct iommufd_group *igroup = idev->igroup;
559 struct iommufd_hw_pagetable *old_hwpt;
560 struct iommufd_attach *attach;
581 int rc;561 int rc;
582 562
583 mutex_lock(&idev->igroup->lock);563 mutex_lock(&igroup->lock);
584 564
585 if (idev->igroup->hwpt != NULL && idev->igroup->hwpt != hwpt) {565 attach = xa_cmpxchg(&igroup->pasid_attach, pasid, NULL,
586 rc = -EINVAL;566 XA_ZERO_ENTRY, GFP_KERNEL);
567 if (xa_is_err(attach)) {
568 rc = xa_err(attach);
587 goto err_unlock;569 goto err_unlock;
588 }570 }
589 571
590 if (hwpt_paging) {572 if (!attach) {
573 attach = kzalloc(sizeof(*attach), GFP_KERNEL);
574 if (!attach) {
575 rc = -ENOMEM;
576 goto err_release_pasid;
577 }
578 xa_init(&attach->device_array);
579 }
580
581 old_hwpt = attach->hwpt;
582
583 rc = xa_insert(&attach->device_array, idev->obj.id, XA_ZERO_ENTRY,
584 GFP_KERNEL);
585 if (rc) {
586 WARN_ON(rc == -EBUSY && !old_hwpt);
587 goto err_free_attach;
588 }
589
590 if (old_hwpt && old_hwpt != hwpt) {
591 rc = -EINVAL;
592 goto err_release_devid;
593 }
594
595 if (attach_resv) {
591 rc = iommufd_device_attach_reserved_iova(idev, hwpt_paging);596 rc = iommufd_device_attach_reserved_iova(idev, hwpt_paging);
592 if (rc)597 if (rc)
593 goto err_unlock;598 goto err_release_devid;
594 }599 }
595 600
596 /*601 /*
@@ -600,51 +605,74 @@ int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt, · 3 unmodified lines
600 * reserved regions are only updated during individual device605 * reserved regions are only updated during individual device
601 * attachment.606 * attachment.
602 */607 */
603 if (list_empty(&idev->igroup->device_list)) {608 if (iommufd_group_first_attach(igroup, pasid)) {
604 rc = iommufd_hwpt_attach_device(hwpt, idev);609 rc = iommufd_hwpt_attach_device(hwpt, idev, pasid);
605 if (rc)610 if (rc)
606 goto err_unresv;611 goto err_unresv;
607 idev->igroup->hwpt = hwpt;612 attach->hwpt = hwpt;
613 WARN_ON(xa_is_err(xa_store(&igroup->pasid_attach, pasid, attach,
614 GFP_KERNEL)));
608 }615 }
609 refcount_inc(&hwpt->obj.users);616 refcount_inc(&hwpt->obj.users);
610 list_add_tail(&idev->group_item, &idev->igroup->device_list);617 WARN_ON(xa_is_err(xa_store(&attach->device_array, idev->obj.id,
611 mutex_unlock(&idev->igroup->lock);618 idev, GFP_KERNEL)));
619 mutex_unlock(&igroup->lock);
612 return 0;620 return 0;
613err_unresv:621err_unresv:
614 if (hwpt_paging)622 if (attach_resv)
615 iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, idev->dev);623 iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, idev->dev);
624err_release_devid:
625 xa_release(&attach->device_array, idev->obj.id);
626err_free_attach:
627 if (iommufd_group_first_attach(igroup, pasid))
628 kfree(attach);
629err_release_pasid:
630 if (iommufd_group_first_attach(igroup, pasid))
631 xa_release(&igroup->pasid_attach, pasid);
616err_unlock:632err_unlock:
617 mutex_unlock(&idev->igroup->lock);633 mutex_unlock(&igroup->lock);
618 return rc;634 return rc;
619}635}
620 636
621struct iommufd_hw_pagetable *637struct iommufd_hw_pagetable *
622iommufd_hw_pagetable_detach(struct iommufd_device *idev)638iommufd_hw_pagetable_detach(struct iommufd_device *idev, ioasid_t pasid)
623{639{
624 struct iommufd_hw_pagetable *hwpt = idev->igroup->hwpt;640 struct iommufd_group *igroup = idev->igroup;
625 struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt);641 struct iommufd_hwpt_paging *hwpt_paging;
642 struct iommufd_hw_pagetable *hwpt;
643 struct iommufd_attach *attach;
644
645 mutex_lock(&igroup->lock);
646 attach = xa_load(&igroup->pasid_attach, pasid);
647 if (!attach) {
648 mutex_unlock(&igroup->lock);
649 return NULL;
650 }
626 651
627 mutex_lock(&idev->igroup->lock);652 hwpt = attach->hwpt;
628 list_del(&idev->group_item);653 hwpt_paging = find_hwpt_paging(hwpt);
629 if (list_empty(&idev->igroup->device_list)) {654
630 iommufd_hwpt_detach_device(hwpt, idev);655 xa_erase(&attach->device_array, idev->obj.id);
631 idev->igroup->hwpt = NULL;656 if (xa_empty(&attach->device_array)) {
657 iommufd_hwpt_detach_device(hwpt, idev, pasid);
658 xa_erase(&igroup->pasid_attach, pasid);
659 kfree(attach);
632 }660 }
633 if (hwpt_paging)661 if (hwpt_paging && pasid == IOMMU_NO_PASID)
634 iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, idev->dev);662 iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, idev->dev);
635 mutex_unlock(&idev->igroup->lock);663 mutex_unlock(&igroup->lock);
636 664
637 /* Caller must destroy hwpt */665 /* Caller must destroy hwpt */
638 return hwpt;666 return hwpt;
639}667}
640 668
641static struct iommufd_hw_pagetable *669static struct iommufd_hw_pagetable *
642iommufd_device_do_attach(struct iommufd_device *idev,670iommufd_device_do_attach(struct iommufd_device *idev, ioasid_t pasid,
643 struct iommufd_hw_pagetable *hwpt)671 struct iommufd_hw_pagetable *hwpt)
644{672{
645 int rc;673 int rc;
646 674
647 rc = iommufd_hw_pagetable_attach(hwpt, idev);675 rc = iommufd_hw_pagetable_attach(hwpt, idev, pasid);
648 if (rc)676 if (rc)
649 return ERR_PTR(rc);677 return ERR_PTR(rc);
650 return NULL;678 return NULL;
@@ -654,11 +682,14 @@ static void · 3 unmodified lines
654iommufd_group_remove_reserved_iova(struct iommufd_group *igroup,682iommufd_group_remove_reserved_iova(struct iommufd_group *igroup,
655 struct iommufd_hwpt_paging *hwpt_paging)683 struct iommufd_hwpt_paging *hwpt_paging)
656{684{
685 struct iommufd_attach *attach;
657 struct iommufd_device *cur;686 struct iommufd_device *cur;
687 unsigned long index;
658 688
659 lockdep_assert_held(&igroup->lock);689 lockdep_assert_held(&igroup->lock);
660 690
661 list_for_each_entry(cur, &igroup->device_list, group_item)691 attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID);
692 xa_for_each(&attach->device_array, index, cur)
662 iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, cur->dev);693 iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, cur->dev);
663}694}
664 695
@@ -667,14 +698,17 @@ iommufd_group_do_replace_reserved_iova(struct iommufd_group *igroup, · 2 unmodified lines
667 struct iommufd_hwpt_paging *hwpt_paging)698 struct iommufd_hwpt_paging *hwpt_paging)
668{699{
669 struct iommufd_hwpt_paging *old_hwpt_paging;700 struct iommufd_hwpt_paging *old_hwpt_paging;
701 struct iommufd_attach *attach;
670 struct iommufd_device *cur;702 struct iommufd_device *cur;
703 unsigned long index;
671 int rc;704 int rc;
672 705
673 lockdep_assert_held(&igroup->lock);706 lockdep_assert_held(&igroup->lock);
674 707
675 old_hwpt_paging = find_hwpt_paging(igroup->hwpt);708 attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID);
709 old_hwpt_paging = find_hwpt_paging(attach->hwpt);
676 if (!old_hwpt_paging || hwpt_paging->ioas != old_hwpt_paging->ioas) {710 if (!old_hwpt_paging || hwpt_paging->ioas != old_hwpt_paging->ioas) {
677 list_for_each_entry(cur, &igroup->device_list, group_item) {711 xa_for_each(&attach->device_array, index, cur) {
678 rc = iopt_table_enforce_dev_resv_regions(712 rc = iopt_table_enforce_dev_resv_regions(
679 &hwpt_paging->ioas->iopt, cur->dev, NULL);713 &hwpt_paging->ioas->iopt, cur->dev, NULL);
680 if (rc)714 if (rc)
@@ -693,69 +727,81 @@ err_unresv: · 12 unmodified lines
693}727}
694 728
695static struct iommufd_hw_pagetable *729static struct iommufd_hw_pagetable *
696iommufd_device_do_replace(struct iommufd_device *idev,730iommufd_device_do_replace(struct iommufd_device *idev, ioasid_t pasid,
697 struct iommufd_hw_pagetable *hwpt)731 struct iommufd_hw_pagetable *hwpt)
698{732{
699 struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt);733 struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt);
734 bool attach_resv = hwpt_paging && pasid == IOMMU_NO_PASID;
700 struct iommufd_hwpt_paging *old_hwpt_paging;735 struct iommufd_hwpt_paging *old_hwpt_paging;
701 struct iommufd_group *igroup = idev->igroup;736 struct iommufd_group *igroup = idev->igroup;
702 struct iommufd_hw_pagetable *old_hwpt;737 struct iommufd_hw_pagetable *old_hwpt;
738 struct iommufd_attach *attach;
703 unsigned int num_devices;739 unsigned int num_devices;
704 int rc;740 int rc;
705 741
706 mutex_lock(&idev->igroup->lock);742 mutex_lock(&igroup->lock);
743
744 attach = xa_load(&igroup->pasid_attach, pasid);
745 if (!attach) {
746 rc = -EINVAL;
747 goto err_unlock;
748 }
749
750 old_hwpt = attach->hwpt;
707 751
708 if (igroup->hwpt == NULL) {752 WARN_ON(!old_hwpt || xa_empty(&attach->device_array));
753
754 if (!iommufd_device_is_attached(idev, pasid)) {
709 rc = -EINVAL;755 rc = -EINVAL;
710 goto err_unlock;756 goto err_unlock;
711 }757 }
712 758
713 if (hwpt == igroup->hwpt) {759 if (hwpt == old_hwpt) {
714 mutex_unlock(&idev->igroup->lock);760 mutex_unlock(&igroup->lock);
715 return NULL;761 return NULL;
716 }762 }
717 763
718 old_hwpt = igroup->hwpt;764 if (attach_resv) {
719 if (hwpt_paging) {
720 rc = iommufd_group_do_replace_reserved_iova(igroup, hwpt_paging);765 rc = iommufd_group_do_replace_reserved_iova(igroup, hwpt_paging);
721 if (rc)766 if (rc)
722 goto err_unlock;767 goto err_unlock;
723 }768 }
724 769
725 rc = iommufd_hwpt_replace_device(idev, hwpt, old_hwpt);770 rc = iommufd_hwpt_replace_device(idev, pasid, hwpt, old_hwpt);
726 if (rc)771 if (rc)
727 goto err_unresv;772 goto err_unresv;
728 773
729 old_hwpt_paging = find_hwpt_paging(old_hwpt);774 old_hwpt_paging = find_hwpt_paging(old_hwpt);
730 if (old_hwpt_paging &&775 if (old_hwpt_paging && pasid == IOMMU_NO_PASID &&
731 (!hwpt_paging || hwpt_paging->ioas != old_hwpt_paging->ioas))776 (!hwpt_paging || hwpt_paging->ioas != old_hwpt_paging->ioas))
732 iommufd_group_remove_reserved_iova(igroup, old_hwpt_paging);777 iommufd_group_remove_reserved_iova(igroup, old_hwpt_paging);
733 778
734 igroup->hwpt = hwpt;779 attach->hwpt = hwpt;
735 780
736 num_devices = list_count_nodes(&igroup->device_list);781 num_devices = iommufd_group_device_num(igroup, pasid);
737 /*782 /*
738 * Move the refcounts held by the device_list to the new hwpt. Retain a783 * Move the refcounts held by the device_array to the new hwpt. Retain a
739 * refcount for this thread as the caller will free it.784 * refcount for this thread as the caller will free it.
740 */785 */
741 refcount_add(num_devices, &hwpt->obj.users);786 refcount_add(num_devices, &hwpt->obj.users);
742 if (num_devices > 1)787 if (num_devices > 1)
743 WARN_ON(refcount_sub_and_test(num_devices - 1,788 WARN_ON(refcount_sub_and_test(num_devices - 1,
744 &old_hwpt->obj.users));789 &old_hwpt->obj.users));
745 mutex_unlock(&idev->igroup->lock);790 mutex_unlock(&igroup->lock);
746 791
747 /* Caller must destroy old_hwpt */792 /* Caller must destroy old_hwpt */
748 return old_hwpt;793 return old_hwpt;
749err_unresv:794err_unresv:
750 if (hwpt_paging)795 if (attach_resv)
751 iommufd_group_remove_reserved_iova(igroup, hwpt_paging);796 iommufd_group_remove_reserved_iova(igroup, hwpt_paging);
752err_unlock:797err_unlock:
753 mutex_unlock(&idev->igroup->lock);798 mutex_unlock(&igroup->lock);
754 return ERR_PTR(rc);799 return ERR_PTR(rc);
755}800}
756 801
757typedef struct iommufd_hw_pagetable *(*attach_fn)(802typedef struct iommufd_hw_pagetable *(*attach_fn)(
758 struct iommufd_device *idev, struct iommufd_hw_pagetable *hwpt);803 struct iommufd_device *idev, ioasid_t pasid,
804 struct iommufd_hw_pagetable *hwpt);
759 805
760/*806/*
761 * When automatically managing the domains we search for a compatible domain in807 * When automatically managing the domains we search for a compatible domain in
@@ -763,7 +809,7 @@ typedef struct iommufd_hw_pagetable *(*attach_fn)( · 1 unmodified line
763 * Automatic domain selection will never pick a manually created domain.809 * Automatic domain selection will never pick a manually created domain.
764 */810 */
765static struct iommufd_hw_pagetable *811static struct iommufd_hw_pagetable *
766iommufd_device_auto_get_domain(struct iommufd_device *idev,812iommufd_device_auto_get_domain(struct iommufd_device *idev, ioasid_t pasid,
767 struct iommufd_ioas *ioas, u32 *pt_id,813 struct iommufd_ioas *ioas, u32 *pt_id,
768 attach_fn do_attach)814 attach_fn do_attach)
769{815{
@@ -792,7 +838,7 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, · 22 unmodified lines
792 hwpt = &hwpt_paging->common;838 hwpt = &hwpt_paging->common;
793 if (!iommufd_lock_obj(&hwpt->obj))839 if (!iommufd_lock_obj(&hwpt->obj))
794 continue;840 continue;
795 destroy_hwpt = (*do_attach)(idev, hwpt);841 destroy_hwpt = (*do_attach)(idev, pasid, hwpt);
796 if (IS_ERR(destroy_hwpt)) {842 if (IS_ERR(destroy_hwpt)) {
797 iommufd_put_object(idev->ictx, &hwpt->obj);843 iommufd_put_object(idev->ictx, &hwpt->obj);
798 /*844 /*
@@ -810,8 +856,8 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, · 11 unmodified lines
810 goto out_unlock;856 goto out_unlock;
811 }857 }
812 858
813 hwpt_paging = iommufd_hwpt_paging_alloc(idev->ictx, ioas, idev, 0,859 hwpt_paging = iommufd_hwpt_paging_alloc(idev->ictx, ioas, idev, pasid,
814 immediate_attach, NULL);860 0, immediate_attach, NULL);
815 if (IS_ERR(hwpt_paging)) {861 if (IS_ERR(hwpt_paging)) {
816 destroy_hwpt = ERR_CAST(hwpt_paging);862 destroy_hwpt = ERR_CAST(hwpt_paging);
817 goto out_unlock;863 goto out_unlock;
@@ -819,7 +865,7 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, · 1 unmodified line
819 hwpt = &hwpt_paging->common;865 hwpt = &hwpt_paging->common;
820 866
821 if (!immediate_attach) {867 if (!immediate_attach) {
822 destroy_hwpt = (*do_attach)(idev, hwpt);868 destroy_hwpt = (*do_attach)(idev, pasid, hwpt);
823 if (IS_ERR(destroy_hwpt))869 if (IS_ERR(destroy_hwpt))
824 goto out_abort;870 goto out_abort;
825 } else {871 } else {
@@ -840,8 +886,9 @@ out_unlock: · 14 unmodified lines
840 return destroy_hwpt;886 return destroy_hwpt;
841}887}
842 888
843static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id,889static int iommufd_device_change_pt(struct iommufd_device *idev,
844 attach_fn do_attach)890 ioasid_t pasid,
891 u32 *pt_id, attach_fn do_attach)
845{892{
846 struct iommufd_hw_pagetable *destroy_hwpt;893 struct iommufd_hw_pagetable *destroy_hwpt;
847 struct iommufd_object *pt_obj;894 struct iommufd_object *pt_obj;
@@ -856,7 +903,7 @@ static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id, · 8 unmodified lines
856 struct iommufd_hw_pagetable *hwpt =903 struct iommufd_hw_pagetable *hwpt =
857 container_of(pt_obj, struct iommufd_hw_pagetable, obj);904 container_of(pt_obj, struct iommufd_hw_pagetable, obj);
858 905
859 destroy_hwpt = (*do_attach)(idev, hwpt);906 destroy_hwpt = (*do_attach)(idev, pasid, hwpt);
860 if (IS_ERR(destroy_hwpt))907 if (IS_ERR(destroy_hwpt))
861 goto out_put_pt_obj;908 goto out_put_pt_obj;
862 break;909 break;
@@ -865,8 +912,8 @@ static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id, · 2 unmodified lines
865 struct iommufd_ioas *ioas =912 struct iommufd_ioas *ioas =
866 container_of(pt_obj, struct iommufd_ioas, obj);913 container_of(pt_obj, struct iommufd_ioas, obj);
867 914
868 destroy_hwpt = iommufd_device_auto_get_domain(idev, ioas, pt_id,915 destroy_hwpt = iommufd_device_auto_get_domain(idev, pasid, ioas,
869 do_attach);916 pt_id, do_attach);
870 if (IS_ERR(destroy_hwpt))917 if (IS_ERR(destroy_hwpt))
871 goto out_put_pt_obj;918 goto out_put_pt_obj;
872 break;919 break;
@@ -888,22 +935,26 @@ out_put_pt_obj: · 15 unmodified lines
888}935}
889 936
890/**937/**
891 * iommufd_device_attach - Connect a device to an iommu_domain938 * iommufd_device_attach - Connect a device/pasid to an iommu_domain
892 * @idev: device to attach939 * @idev: device to attach
940 * @pasid: pasid to attach
893 * @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING941 * @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING
894 * Output the IOMMUFD_OBJ_HWPT_PAGING ID942 * Output the IOMMUFD_OBJ_HWPT_PAGING ID
895 *943 *
896 * This connects the device to an iommu_domain, either automatically or manually944 * This connects the device/pasid to an iommu_domain, either automatically
897 * selected. Once this completes the device could do DMA.945 * or manually selected. Once this completes the device could do DMA with
946 * @pasid. @pasid is IOMMU_NO_PASID if this attach is for no pasid usage.
898 *947 *
899 * The caller should return the resulting pt_id back to userspace.948 * The caller should return the resulting pt_id back to userspace.
900 * This function is undone by calling iommufd_device_detach().949 * This function is undone by calling iommufd_device_detach().
901 */950 */
902int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id)951int iommufd_device_attach(struct iommufd_device *idev, ioasid_t pasid,
952 u32 *pt_id)
903{953{
904 int rc;954 int rc;
905 955
906 rc = iommufd_device_change_pt(idev, pt_id, &iommufd_device_do_attach);956 rc = iommufd_device_change_pt(idev, pasid, pt_id,
957 &iommufd_device_do_attach);
907 if (rc)958 if (rc)
908 return rc;959 return rc;
909 960
@@ -917,8 +968,9 @@ int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id) · 7 unmodified lines
917EXPORT_SYMBOL_NS_GPL(iommufd_device_attach, "IOMMUFD");968EXPORT_SYMBOL_NS_GPL(iommufd_device_attach, "IOMMUFD");
918 969
919/**970/**
920 * iommufd_device_replace - Change the device's iommu_domain971 * iommufd_device_replace - Change the device/pasid's iommu_domain
921 * @idev: device to change972 * @idev: device to change
973 * @pasid: pasid to change
922 * @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING974 * @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING
923 * Output the IOMMUFD_OBJ_HWPT_PAGING ID975 * Output the IOMMUFD_OBJ_HWPT_PAGING ID
924 *976 *
@@ -929,27 +981,33 @@ EXPORT_SYMBOL_NS_GPL(iommufd_device_attach, "IOMMUFD"); · 4 unmodified lines
929 *981 *
930 * If it fails then no change is made to the attachment. The iommu driver may982 * If it fails then no change is made to the attachment. The iommu driver may
931 * implement this so there is no disruption in translation. This can only be983 * implement this so there is no disruption in translation. This can only be
932 * called if iommufd_device_attach() has already succeeded.984 * called if iommufd_device_attach() has already succeeded. @pasid is
985 * IOMMU_NO_PASID for no pasid usage.
933 */986 */
934int iommufd_device_replace(struct iommufd_device *idev, u32 *pt_id)987int iommufd_device_replace(struct iommufd_device *idev, ioasid_t pasid,
988 u32 *pt_id)
935{989{
936 return iommufd_device_change_pt(idev, pt_id,990 return iommufd_device_change_pt(idev, pasid, pt_id,
937 &iommufd_device_do_replace);991 &iommufd_device_do_replace);
938}992}
939EXPORT_SYMBOL_NS_GPL(iommufd_device_replace, "IOMMUFD");993EXPORT_SYMBOL_NS_GPL(iommufd_device_replace, "IOMMUFD");
940 994
941/**995/**
942 * iommufd_device_detach - Disconnect a device to an iommu_domain996 * iommufd_device_detach - Disconnect a device/device to an iommu_domain
943 * @idev: device to detach997 * @idev: device to detach
998 * @pasid: pasid to detach
944 *999 *
945 * Undo iommufd_device_attach(). This disconnects the idev from the previously1000 * Undo iommufd_device_attach(). This disconnects the idev from the previously
946 * attached pt_id. The device returns back to a blocked DMA translation.1001 * attached pt_id. The device returns back to a blocked DMA translation.
1002 * @pasid is IOMMU_NO_PASID for no pasid usage.
947 */1003 */
948void iommufd_device_detach(struct iommufd_device *idev)1004void iommufd_device_detach(struct iommufd_device *idev, ioasid_t pasid)
949{1005{
950 struct iommufd_hw_pagetable *hwpt;1006 struct iommufd_hw_pagetable *hwpt;
951 1007
952 hwpt = iommufd_hw_pagetable_detach(idev);1008 hwpt = iommufd_hw_pagetable_detach(idev, pasid);
1009 if (!hwpt)
1010 return;
953 iommufd_hw_pagetable_put(idev->ictx, hwpt);1011 iommufd_hw_pagetable_put(idev->ictx, hwpt);
954 refcount_dec(&idev->obj.users);1012 refcount_dec(&idev->obj.users);
955}1013}
@@ -1349,7 +1407,7 @@ int iommufd_access_rw(struct iommufd_access *access, unsigned long iova, · 393 unmodified lines
1349 struct io_pagetable *iopt;1407 struct io_pagetable *iopt;
1350 struct iopt_area *area;1408 struct iopt_area *area;
1351 unsigned long last_iova;1409 unsigned long last_iova;
1352 int rc;1410 int rc = -EINVAL;
1353 1411
1354 if (!length)1412 if (!length)
1355 return -EINVAL;1413 return -EINVAL;
@@ -1405,7 +1463,8 @@ int iommufd_get_hw_info(struct iommufd_ucmd *ucmd) · 49 unmodified lines
1405 void *data;1463 void *data;
1406 int rc;1464 int rc;
1407 1465
1408 if (cmd->flags || cmd->__reserved)1466 if (cmd->flags || cmd->__reserved[0] || cmd->__reserved[1] ||
1467 cmd->__reserved[2])
1409 return -EOPNOTSUPP;1468 return -EOPNOTSUPP;
1410 1469
1411 idev = iommufd_get_device(ucmd, cmd->dev_id);1470 idev = iommufd_get_device(ucmd, cmd->dev_id);
@@ -1462,6 +1521,36 @@ int iommufd_get_hw_info(struct iommufd_ucmd *ucmd) · 50 unmodified lines
1462 if (device_iommu_capable(idev->dev, IOMMU_CAP_DIRTY_TRACKING))1521 if (device_iommu_capable(idev->dev, IOMMU_CAP_DIRTY_TRACKING))
1463 cmd->out_capabilities |= IOMMU_HW_CAP_DIRTY_TRACKING;1522 cmd->out_capabilities |= IOMMU_HW_CAP_DIRTY_TRACKING;
1464 1523
1524 cmd->out_max_pasid_log2 = 0;
1525 /*
1526 * Currently, all iommu drivers enable PASID in the probe_device()
1527 * op if iommu and device supports it. So the max_pasids stored in
1528 * dev->iommu indicates both PASID support and enable status. A
1529 * non-zero dev->iommu->max_pasids means PASID is supported and
1530 * enabled. The iommufd only reports PASID capability to userspace
1531 * if it's enabled.
1532 */
1533 if (idev->dev->iommu->max_pasids) {
1534 cmd->out_max_pasid_log2 = ilog2(idev->dev->iommu->max_pasids);
1535
1536 if (dev_is_pci(idev->dev)) {
1537 struct pci_dev *pdev = to_pci_dev(idev->dev);
1538 int ctrl;
1539
1540 ctrl = pci_pasid_status(pdev);
1541
1542 WARN_ON_ONCE(ctrl < 0 ||
1543 !(ctrl & PCI_PASID_CTRL_ENABLE));
1544
1545 if (ctrl & PCI_PASID_CTRL_EXEC)
1546 cmd->out_capabilities |=
1547 IOMMU_HW_CAP_PCI_PASID_EXEC;
1548 if (ctrl & PCI_PASID_CTRL_PRIV)
1549 cmd->out_capabilities |=
1550 IOMMU_HW_CAP_PCI_PASID_PRIV;
1551 }
1552 }
1553
1465 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));1554 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
1466out_free:1555out_free:
1467 kfree(data);1556 kfree(data);
drivers/iommu/iommufd/driver.c+198−0
@@ -49,5 +49,203 @@ struct device *iommufd_viommu_find_dev(struct iommufd_viommu *viommu, · 48 unmodified lines
49}49}
50EXPORT_SYMBOL_NS_GPL(iommufd_viommu_find_dev, "IOMMUFD");50EXPORT_SYMBOL_NS_GPL(iommufd_viommu_find_dev, "IOMMUFD");
51 51
52/* Return -ENOENT if device is not associated to the vIOMMU */
53int iommufd_viommu_get_vdev_id(struct iommufd_viommu *viommu,
54 struct device *dev, unsigned long *vdev_id)
55{
56 struct iommufd_vdevice *vdev;
57 unsigned long index;
58 int rc = -ENOENT;
59
60 if (WARN_ON_ONCE(!vdev_id))
61 return -EINVAL;
62
63 xa_lock(&viommu->vdevs);
64 xa_for_each(&viommu->vdevs, index, vdev) {
65 if (vdev->dev == dev) {
66 *vdev_id = vdev->id;
67 rc = 0;
68 break;
69 }
70 }
71 xa_unlock(&viommu->vdevs);
72 return rc;
73}
74EXPORT_SYMBOL_NS_GPL(iommufd_viommu_get_vdev_id, "IOMMUFD");
75
76/*
77 * Typically called in driver's threaded IRQ handler.
78 * The @type and @event_data must be defined in include/uapi/linux/iommufd.h
79 */
80int iommufd_viommu_report_event(struct iommufd_viommu *viommu,
81 enum iommu_veventq_type type, void *event_data,
82 size_t data_len)
83{
84 struct iommufd_veventq *veventq;
85 struct iommufd_vevent *vevent;
86 int rc = 0;
87
88 if (WARN_ON_ONCE(!data_len || !event_data))
89 return -EINVAL;
90
91 down_read(&viommu->veventqs_rwsem);
92
93 veventq = iommufd_viommu_find_veventq(viommu, type);
94 if (!veventq) {
95 rc = -EOPNOTSUPP;
96 goto out_unlock_veventqs;
97 }
98
99 spin_lock(&veventq->common.lock);
100 if (veventq->num_events == veventq->depth) {
101 vevent = &veventq->lost_events_header;
102 goto out_set_header;
103 }
104
105 vevent = kzalloc(struct_size(vevent, event_data, data_len), GFP_ATOMIC);
106 if (!vevent) {
107 rc = -ENOMEM;
108 vevent = &veventq->lost_events_header;
109 goto out_set_header;
110 }
111 memcpy(vevent->event_data, event_data, data_len);
112 vevent->data_len = data_len;
113 veventq->num_events++;
114
115out_set_header:
116 iommufd_vevent_handler(veventq, vevent);
117 spin_unlock(&veventq->common.lock);
118out_unlock_veventqs:
119 up_read(&viommu->veventqs_rwsem);
120 return rc;
121}
122EXPORT_SYMBOL_NS_GPL(iommufd_viommu_report_event, "IOMMUFD");
123
124#ifdef CONFIG_IRQ_MSI_IOMMU
125/*
126 * Get a iommufd_sw_msi_map for the msi physical address requested by the irq
127 * layer. The mapping to IOVA is global to the iommufd file descriptor, every
128 * domain that is attached to a device using the same MSI parameters will use
129 * the same IOVA.
130 */
131static struct iommufd_sw_msi_map *
132iommufd_sw_msi_get_map(struct iommufd_ctx *ictx, phys_addr_t msi_addr,
133 phys_addr_t sw_msi_start)
134{
135 struct iommufd_sw_msi_map *cur;
136 unsigned int max_pgoff = 0;
137
138 lockdep_assert_held(&ictx->sw_msi_lock);
139
140 list_for_each_entry(cur, &ictx->sw_msi_list, sw_msi_item) {
141 if (cur->sw_msi_start != sw_msi_start)
142 continue;
143 max_pgoff = max(max_pgoff, cur->pgoff + 1);
144 if (cur->msi_addr == msi_addr)
145 return cur;
146 }
147
148 if (ictx->sw_msi_id >=
149 BITS_PER_BYTE * sizeof_field(struct iommufd_sw_msi_maps, bitmap))
150 return ERR_PTR(-EOVERFLOW);
151
152 cur = kzalloc(sizeof(*cur), GFP_KERNEL);
153 if (!cur)
154 return ERR_PTR(-ENOMEM);
155
156 cur->sw_msi_start = sw_msi_start;
157 cur->msi_addr = msi_addr;
158 cur->pgoff = max_pgoff;
159 cur->id = ictx->sw_msi_id++;
160 list_add_tail(&cur->sw_msi_item, &ictx->sw_msi_list);
161 return cur;
162}
163
164int iommufd_sw_msi_install(struct iommufd_ctx *ictx,
165 struct iommufd_hwpt_paging *hwpt_paging,
166 struct iommufd_sw_msi_map *msi_map)
167{
168 unsigned long iova;
169
170 lockdep_assert_held(&ictx->sw_msi_lock);
171
172 iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE;
173 if (!test_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap)) {
174 int rc;
175
176 rc = iommu_map(hwpt_paging->common.domain, iova,
177 msi_map->msi_addr, PAGE_SIZE,
178 IOMMU_WRITE | IOMMU_READ | IOMMU_MMIO,
179 GFP_KERNEL_ACCOUNT);
180 if (rc)
181 return rc;
182 __set_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap);
183 }
184 return 0;
185}
186EXPORT_SYMBOL_NS_GPL(iommufd_sw_msi_install, "IOMMUFD_INTERNAL");
187
188/*
189 * Called by the irq code if the platform translates the MSI address through the
190 * IOMMU. msi_addr is the physical address of the MSI page. iommufd will
191 * allocate a fd global iova for the physical page that is the same on all
192 * domains and devices.
193 */
194int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
195 phys_addr_t msi_addr)
196{
197 struct device *dev = msi_desc_to_dev(desc);
198 struct iommufd_hwpt_paging *hwpt_paging;
199 struct iommu_attach_handle *raw_handle;
200 struct iommufd_attach_handle *handle;
201 struct iommufd_sw_msi_map *msi_map;
202 struct iommufd_ctx *ictx;
203 unsigned long iova;
204 int rc;
205
206 /*
207 * It is safe to call iommu_attach_handle_get() here because the iommu
208 * core code invokes this under the group mutex which also prevents any
209 * change of the attach handle for the duration of this function.
210 */
211 iommu_group_mutex_assert(dev);
212
213 raw_handle =
214 iommu_attach_handle_get(dev->iommu_group, IOMMU_NO_PASID, 0);
215 if (IS_ERR(raw_handle))
216 return 0;
217 hwpt_paging = find_hwpt_paging(domain->iommufd_hwpt);
218
219 handle = to_iommufd_handle(raw_handle);
220 /* No IOMMU_RESV_SW_MSI means no change to the msi_msg */
221 if (handle->idev->igroup->sw_msi_start == PHYS_ADDR_MAX)
222 return 0;
223
224 ictx = handle->idev->ictx;
225 guard(mutex)(&ictx->sw_msi_lock);
226 /*
227 * The input msi_addr is the exact byte offset of the MSI doorbell, we
228 * assume the caller has checked that it is contained with a MMIO region
229 * that is secure to map at PAGE_SIZE.
230 */
231 msi_map = iommufd_sw_msi_get_map(handle->idev->ictx,
232 msi_addr & PAGE_MASK,
233 handle->idev->igroup->sw_msi_start);
234 if (IS_ERR(msi_map))
235 return PTR_ERR(msi_map);
236
237 rc = iommufd_sw_msi_install(ictx, hwpt_paging, msi_map);
238 if (rc)
239 return rc;
240 __set_bit(msi_map->id, handle->idev->igroup->required_sw_msi.bitmap);
241
242 iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE;
243 msi_desc_set_iommu_msi_iova(desc, iova, PAGE_SHIFT);
244 return 0;
245}
246EXPORT_SYMBOL_NS_GPL(iommufd_sw_msi, "IOMMUFD");
247#endif
248
52MODULE_DESCRIPTION("iommufd code shared with builtin modules");249MODULE_DESCRIPTION("iommufd code shared with builtin modules");
250MODULE_IMPORT_NS("IOMMUFD_INTERNAL");
53MODULE_LICENSE("GPL");251MODULE_LICENSE("GPL");
drivers/iommu/iommufd/eventq.c+598−0
@@ -0,0 +1,598 @@
1// SPDX-License-Identifier: GPL-2.0-only
2/* Copyright (C) 2024 Intel Corporation
3 */
4#define pr_fmt(fmt) "iommufd: " fmt
5
6#include <linux/anon_inodes.h>
7#include <linux/file.h>
8#include <linux/fs.h>
9#include <linux/iommufd.h>
10#include <linux/module.h>
11#include <linux/mutex.h>
12#include <linux/pci.h>
13#include <linux/pci-ats.h>
14#include <linux/poll.h>
15#include <uapi/linux/iommufd.h>
16
17#include "../iommu-priv.h"
18#include "iommufd_private.h"
19
20/* IOMMUFD_OBJ_FAULT Functions */
21
22int iommufd_fault_iopf_enable(struct iommufd_device *idev)
23{
24 struct device *dev = idev->dev;
25 int ret;
26
27 /*
28 * Once we turn on PCI/PRI support for VF, the response failure code
29 * should not be forwarded to the hardware due to PRI being a shared
30 * resource between PF and VFs. There is no coordination for this
31 * shared capability. This waits for a vPRI reset to recover.
32 */
33 if (dev_is_pci(dev)) {
34 struct pci_dev *pdev = to_pci_dev(dev);
35
36 if (pdev->is_virtfn && pci_pri_supported(pdev))
37 return -EINVAL;
38 }
39
40 mutex_lock(&idev->iopf_lock);
41 /* Device iopf has already been on. */
42 if (++idev->iopf_enabled > 1) {
43 mutex_unlock(&idev->iopf_lock);
44 return 0;
45 }
46
47 ret = iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_IOPF);
48 if (ret)
49 --idev->iopf_enabled;
50 mutex_unlock(&idev->iopf_lock);
51
52 return ret;
53}
54
55void iommufd_fault_iopf_disable(struct iommufd_device *idev)
56{
57 mutex_lock(&idev->iopf_lock);
58 if (!WARN_ON(idev->iopf_enabled == 0)) {
59 if (--idev->iopf_enabled == 0)
60 iommu_dev_disable_feature(idev->dev, IOMMU_DEV_FEAT_IOPF);
61 }
62 mutex_unlock(&idev->iopf_lock);
63}
64
65void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt,
66 struct iommufd_attach_handle *handle)
67{
68 struct iommufd_fault *fault = hwpt->fault;
69 struct iopf_group *group, *next;
70 struct list_head free_list;
71 unsigned long index;
72
73 if (!fault)
74 return;
75 INIT_LIST_HEAD(&free_list);
76
77 mutex_lock(&fault->mutex);
78 spin_lock(&fault->common.lock);
79 list_for_each_entry_safe(group, next, &fault->common.deliver, node) {
80 if (group->attach_handle != &handle->handle)
81 continue;
82 list_move(&group->node, &free_list);
83 }
84 spin_unlock(&fault->common.lock);
85
86 list_for_each_entry_safe(group, next, &free_list, node) {
87 list_del(&group->node);
88 iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
89 iopf_free_group(group);
90 }
91
92 xa_for_each(&fault->response, index, group) {
93 if (group->attach_handle != &handle->handle)
94 continue;
95 xa_erase(&fault->response, index);
96 iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
97 iopf_free_group(group);
98 }
99 mutex_unlock(&fault->mutex);
100}
101
102void iommufd_fault_destroy(struct iommufd_object *obj)
103{
104 struct iommufd_eventq *eventq =
105 container_of(obj, struct iommufd_eventq, obj);
106 struct iommufd_fault *fault = eventq_to_fault(eventq);
107 struct iopf_group *group, *next;
108 unsigned long index;
109
110 /*
111 * The iommufd object's reference count is zero at this point.
112 * We can be confident that no other threads are currently
113 * accessing this pointer. Therefore, acquiring the mutex here
114 * is unnecessary.
115 */
116 list_for_each_entry_safe(group, next, &fault->common.deliver, node) {
117 list_del(&group->node);
118 iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
119 iopf_free_group(group);
120 }
121 xa_for_each(&fault->response, index, group) {
122 xa_erase(&fault->response, index);
123 iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
124 iopf_free_group(group);
125 }
126 xa_destroy(&fault->response);
127 mutex_destroy(&fault->mutex);
128}
129
130static void iommufd_compose_fault_message(struct iommu_fault *fault,
131 struct iommu_hwpt_pgfault *hwpt_fault,
132 struct iommufd_device *idev,
133 u32 cookie)
134{
135 hwpt_fault->flags = fault->prm.flags;
136 hwpt_fault->dev_id = idev->obj.id;
137 hwpt_fault->pasid = fault->prm.pasid;
138 hwpt_fault->grpid = fault->prm.grpid;
139 hwpt_fault->perm = fault->prm.perm;
140 hwpt_fault->addr = fault->prm.addr;
141 hwpt_fault->length = 0;
142 hwpt_fault->cookie = cookie;
143}
144
145/* Fetch the first node out of the fault->deliver list */
146static struct iopf_group *
147iommufd_fault_deliver_fetch(struct iommufd_fault *fault)
148{
149 struct list_head *list = &fault->common.deliver;
150 struct iopf_group *group = NULL;
151
152 spin_lock(&fault->common.lock);
153 if (!list_empty(list)) {
154 group = list_first_entry(list, struct iopf_group, node);
155 list_del(&group->node);
156 }
157 spin_unlock(&fault->common.lock);
158 return group;
159}
160
161/* Restore a node back to the head of the fault->deliver list */
162static void iommufd_fault_deliver_restore(struct iommufd_fault *fault,
163 struct iopf_group *group)
164{
165 spin_lock(&fault->common.lock);
166 list_add(&group->node, &fault->common.deliver);
167 spin_unlock(&fault->common.lock);
168}
169
170static ssize_t iommufd_fault_fops_read(struct file *filep, char __user *buf,
171 size_t count, loff_t *ppos)
172{
173 size_t fault_size = sizeof(struct iommu_hwpt_pgfault);
174 struct iommufd_eventq *eventq = filep->private_data;
175 struct iommufd_fault *fault = eventq_to_fault(eventq);
176 struct iommu_hwpt_pgfault data = {};
177 struct iommufd_device *idev;
178 struct iopf_group *group;
179 struct iopf_fault *iopf;
180 size_t done = 0;
181 int rc = 0;
182
183 if (*ppos || count % fault_size)
184 return -ESPIPE;
185
186 mutex_lock(&fault->mutex);
187 while ((group = iommufd_fault_deliver_fetch(fault))) {
188 if (done >= count ||
189 group->fault_count * fault_size > count - done) {
190 iommufd_fault_deliver_restore(fault, group);
191 break;
192 }
193
194 rc = xa_alloc(&fault->response, &group->cookie, group,
195 xa_limit_32b, GFP_KERNEL);
196 if (rc) {
197 iommufd_fault_deliver_restore(fault, group);
198 break;
199 }
200
201 idev = to_iommufd_handle(group->attach_handle)->idev;
202 list_for_each_entry(iopf, &group->faults, list) {
203 iommufd_compose_fault_message(&iopf->fault,
204 &data, idev,
205 group->cookie);
206 if (copy_to_user(buf + done, &data, fault_size)) {
207 xa_erase(&fault->response, group->cookie);
208 iommufd_fault_deliver_restore(fault, group);
209 rc = -EFAULT;
210 break;
211 }
212 done += fault_size;
213 }
214 }
215 mutex_unlock(&fault->mutex);
216
217 return done == 0 ? rc : done;
218}
219
220static ssize_t iommufd_fault_fops_write(struct file *filep, const char __user *buf,
221 size_t count, loff_t *ppos)
222{
223 size_t response_size = sizeof(struct iommu_hwpt_page_response);
224 struct iommufd_eventq *eventq = filep->private_data;
225 struct iommufd_fault *fault = eventq_to_fault(eventq);
226 struct iommu_hwpt_page_response response;
227 struct iopf_group *group;
228 size_t done = 0;
229 int rc = 0;
230
231 if (*ppos || count % response_size)
232 return -ESPIPE;
233
234 mutex_lock(&fault->mutex);
235 while (count > done) {
236 rc = copy_from_user(&response, buf + done, response_size);
237 if (rc)
238 break;
239
240 static_assert((int)IOMMUFD_PAGE_RESP_SUCCESS ==
241 (int)IOMMU_PAGE_RESP_SUCCESS);
242 static_assert((int)IOMMUFD_PAGE_RESP_INVALID ==
243 (int)IOMMU_PAGE_RESP_INVALID);
244 if (response.code != IOMMUFD_PAGE_RESP_SUCCESS &&
245 response.code != IOMMUFD_PAGE_RESP_INVALID) {
246 rc = -EINVAL;
247 break;
248 }
249
250 group = xa_erase(&fault->response, response.cookie);
251 if (!group) {
252 rc = -EINVAL;
253 break;
254 }
255
256 iopf_group_response(group, response.code);
257 iopf_free_group(group);
258 done += response_size;
259 }
260 mutex_unlock(&fault->mutex);
261
262 return done == 0 ? rc : done;
263}
264
265/* IOMMUFD_OBJ_VEVENTQ Functions */
266
267void iommufd_veventq_abort(struct iommufd_object *obj)
268{
269 struct iommufd_eventq *eventq =
270 container_of(obj, struct iommufd_eventq, obj);
271 struct iommufd_veventq *veventq = eventq_to_veventq(eventq);
272 struct iommufd_viommu *viommu = veventq->viommu;
273 struct iommufd_vevent *cur, *next;
274
275 lockdep_assert_held_write(&viommu->veventqs_rwsem);
276
277 list_for_each_entry_safe(cur, next, &eventq->deliver, node) {
278 list_del(&cur->node);
279 if (cur != &veventq->lost_events_header)
280 kfree(cur);
281 }
282
283 refcount_dec(&viommu->obj.users);
284 list_del(&veventq->node);
285}
286
287void iommufd_veventq_destroy(struct iommufd_object *obj)
288{
289 struct iommufd_veventq *veventq = eventq_to_veventq(
290 container_of(obj, struct iommufd_eventq, obj));
291
292 down_write(&veventq->viommu->veventqs_rwsem);
293 iommufd_veventq_abort(obj);
294 up_write(&veventq->viommu->veventqs_rwsem);
295}
296
297static struct iommufd_vevent *
298iommufd_veventq_deliver_fetch(struct iommufd_veventq *veventq)
299{
300 struct iommufd_eventq *eventq = &veventq->common;
301 struct list_head *list = &eventq->deliver;
302 struct iommufd_vevent *vevent = NULL;
303
304 spin_lock(&eventq->lock);
305 if (!list_empty(list)) {
306 struct iommufd_vevent *next;
307
308 next = list_first_entry(list, struct iommufd_vevent, node);
309 /* Make a copy of the lost_events_header for copy_to_user */
310 if (next == &veventq->lost_events_header) {
311 vevent = kzalloc(sizeof(*vevent), GFP_ATOMIC);
312 if (!vevent)
313 goto out_unlock;
314 }
315 list_del(&next->node);
316 if (vevent)
317 memcpy(vevent, next, sizeof(*vevent));
318 else
319 vevent = next;
320 }
321out_unlock:
322 spin_unlock(&eventq->lock);
323 return vevent;
324}
325
326static void iommufd_veventq_deliver_restore(struct iommufd_veventq *veventq,
327 struct iommufd_vevent *vevent)
328{
329 struct iommufd_eventq *eventq = &veventq->common;
330 struct list_head *list = &eventq->deliver;
331
332 spin_lock(&eventq->lock);
333 if (vevent_for_lost_events_header(vevent)) {
334 /* Remove the copy of the lost_events_header */
335 kfree(vevent);
336 vevent = NULL;
337 /* An empty list needs the lost_events_header back */
338 if (list_empty(list))
339 vevent = &veventq->lost_events_header;
340 }
341 if (vevent)
342 list_add(&vevent->node, list);
343 spin_unlock(&eventq->lock);
344}
345
346static ssize_t iommufd_veventq_fops_read(struct file *filep, char __user *buf,
347 size_t count, loff_t *ppos)
348{
349 struct iommufd_eventq *eventq = filep->private_data;
350 struct iommufd_veventq *veventq = eventq_to_veventq(eventq);
351 struct iommufd_vevent_header *hdr;
352 struct iommufd_vevent *cur;
353 size_t done = 0;
354 int rc = 0;
355
356 if (*ppos)
357 return -ESPIPE;
358
359 while ((cur = iommufd_veventq_deliver_fetch(veventq))) {
360 /* Validate the remaining bytes against the header size */
361 if (done >= count || sizeof(*hdr) > count - done) {
362 iommufd_veventq_deliver_restore(veventq, cur);
363 break;
364 }
365 hdr = &cur->header;
366
367 /* If being a normal vEVENT, validate against the full size */
368 if (!vevent_for_lost_events_header(cur) &&
369 sizeof(hdr) + cur->data_len > count - done) {
370 iommufd_veventq_deliver_restore(veventq, cur);
371 break;
372 }
373
374 if (copy_to_user(buf + done, hdr, sizeof(*hdr))) {
375 iommufd_veventq_deliver_restore(veventq, cur);
376 rc = -EFAULT;
377 break;
378 }
379 done += sizeof(*hdr);
380
381 if (cur->data_len &&
382 copy_to_user(buf + done, cur->event_data, cur->data_len)) {
383 iommufd_veventq_deliver_restore(veventq, cur);
384 rc = -EFAULT;
385 break;
386 }
387 spin_lock(&eventq->lock);
388 if (!vevent_for_lost_events_header(cur))
389 veventq->num_events--;
390 spin_unlock(&eventq->lock);
391 done += cur->data_len;
392 kfree(cur);
393 }
394
395 return done == 0 ? rc : done;
396}
397
398/* Common Event Queue Functions */
399
400static __poll_t iommufd_eventq_fops_poll(struct file *filep,
401 struct poll_table_struct *wait)
402{
403 struct iommufd_eventq *eventq = filep->private_data;
404 __poll_t pollflags = 0;
405
406 if (eventq->obj.type == IOMMUFD_OBJ_FAULT)
407 pollflags |= EPOLLOUT;
408
409 poll_wait(filep, &eventq->wait_queue, wait);
410 spin_lock(&eventq->lock);
411 if (!list_empty(&eventq->deliver))
412 pollflags |= EPOLLIN | EPOLLRDNORM;
413 spin_unlock(&eventq->lock);
414
415 return pollflags;
416}
417
418static int iommufd_eventq_fops_release(struct inode *inode, struct file *filep)
419{
420 struct iommufd_eventq *eventq = filep->private_data;
421
422 refcount_dec(&eventq->obj.users);
423 iommufd_ctx_put(eventq->ictx);
424 return 0;
425}
426
427#define INIT_EVENTQ_FOPS(read_op, write_op) \
428 ((const struct file_operations){ \
429 .owner = THIS_MODULE, \
430 .open = nonseekable_open, \
431 .read = read_op, \
432 .write = write_op, \
433 .poll = iommufd_eventq_fops_poll, \
434 .release = iommufd_eventq_fops_release, \
435 })
436
437static int iommufd_eventq_init(struct iommufd_eventq *eventq, char *name,
438 struct iommufd_ctx *ictx,
439 const struct file_operations *fops)
440{
441 struct file *filep;
442 int fdno;
443
444 spin_lock_init(&eventq->lock);
445 INIT_LIST_HEAD(&eventq->deliver);
446 init_waitqueue_head(&eventq->wait_queue);
447
448 filep = anon_inode_getfile(name, fops, eventq, O_RDWR);
449 if (IS_ERR(filep))
450 return PTR_ERR(filep);
451
452 eventq->ictx = ictx;
453 iommufd_ctx_get(eventq->ictx);
454 eventq->filep = filep;
455 refcount_inc(&eventq->obj.users);
456
457 fdno = get_unused_fd_flags(O_CLOEXEC);
458 if (fdno < 0)
459 fput(filep);
460 return fdno;
461}
462
463static const struct file_operations iommufd_fault_fops =
464 INIT_EVENTQ_FOPS(iommufd_fault_fops_read, iommufd_fault_fops_write);
465
466int iommufd_fault_alloc(struct iommufd_ucmd *ucmd)
467{
468 struct iommu_fault_alloc *cmd = ucmd->cmd;
469 struct iommufd_fault *fault;
470 int fdno;
471 int rc;
472
473 if (cmd->flags)
474 return -EOPNOTSUPP;
475
476 fault = __iommufd_object_alloc(ucmd->ictx, fault, IOMMUFD_OBJ_FAULT,
477 common.obj);
478 if (IS_ERR(fault))
479 return PTR_ERR(fault);
480
481 xa_init_flags(&fault->response, XA_FLAGS_ALLOC1);
482 mutex_init(&fault->mutex);
483
484 fdno = iommufd_eventq_init(&fault->common, "[iommufd-pgfault]",
485 ucmd->ictx, &iommufd_fault_fops);
486 if (fdno < 0) {
487 rc = fdno;
488 goto out_abort;
489 }
490
491 cmd->out_fault_id = fault->common.obj.id;
492 cmd->out_fault_fd = fdno;
493
494 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
495 if (rc)
496 goto out_put_fdno;
497 iommufd_object_finalize(ucmd->ictx, &fault->common.obj);
498
499 fd_install(fdno, fault->common.filep);
500
501 return 0;
502out_put_fdno:
503 put_unused_fd(fdno);
504 fput(fault->common.filep);
505out_abort:
506 iommufd_object_abort_and_destroy(ucmd->ictx, &fault->common.obj);
507
508 return rc;
509}
510
511int iommufd_fault_iopf_handler(struct iopf_group *group)
512{
513 struct iommufd_hw_pagetable *hwpt;
514 struct iommufd_fault *fault;
515
516 hwpt = group->attach_handle->domain->iommufd_hwpt;
517 fault = hwpt->fault;
518
519 spin_lock(&fault->common.lock);
520 list_add_tail(&group->node, &fault->common.deliver);
521 spin_unlock(&fault->common.lock);
522
523 wake_up_interruptible(&fault->common.wait_queue);
524
525 return 0;
526}
527
528static const struct file_operations iommufd_veventq_fops =
529 INIT_EVENTQ_FOPS(iommufd_veventq_fops_read, NULL);
530
531int iommufd_veventq_alloc(struct iommufd_ucmd *ucmd)
532{
533 struct iommu_veventq_alloc *cmd = ucmd->cmd;
534 struct iommufd_veventq *veventq;
535 struct iommufd_viommu *viommu;
536 int fdno;
537 int rc;
538
539 if (cmd->flags || cmd->__reserved ||
540 cmd->type == IOMMU_VEVENTQ_TYPE_DEFAULT)
541 return -EOPNOTSUPP;
542 if (!cmd->veventq_depth)
543 return -EINVAL;
544
545 viommu = iommufd_get_viommu(ucmd, cmd->viommu_id);
546 if (IS_ERR(viommu))
547 return PTR_ERR(viommu);
548
549 down_write(&viommu->veventqs_rwsem);
550
551 if (iommufd_viommu_find_veventq(viommu, cmd->type)) {
552 rc = -EEXIST;
553 goto out_unlock_veventqs;
554 }
555
556 veventq = __iommufd_object_alloc(ucmd->ictx, veventq,
557 IOMMUFD_OBJ_VEVENTQ, common.obj);
558 if (IS_ERR(veventq)) {
559 rc = PTR_ERR(veventq);
560 goto out_unlock_veventqs;
561 }
562
563 veventq->type = cmd->type;
564 veventq->viommu = viommu;
565 refcount_inc(&viommu->obj.users);
566 veventq->depth = cmd->veventq_depth;
567 list_add_tail(&veventq->node, &viommu->veventqs);
568 veventq->lost_events_header.header.flags =
569 IOMMU_VEVENTQ_FLAG_LOST_EVENTS;
570
571 fdno = iommufd_eventq_init(&veventq->common, "[iommufd-viommu-event]",
572 ucmd->ictx, &iommufd_veventq_fops);
573 if (fdno < 0) {
574 rc = fdno;
575 goto out_abort;
576 }
577
578 cmd->out_veventq_id = veventq->common.obj.id;
579 cmd->out_veventq_fd = fdno;
580
581 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
582 if (rc)
583 goto out_put_fdno;
584
585 iommufd_object_finalize(ucmd->ictx, &veventq->common.obj);
586 fd_install(fdno, veventq->common.filep);
587 goto out_unlock_veventqs;
588
589out_put_fdno:
590 put_unused_fd(fdno);
591 fput(veventq->common.filep);
592out_abort:
593 iommufd_object_abort_and_destroy(ucmd->ictx, &veventq->common.obj);
594out_unlock_veventqs:
595 up_write(&viommu->veventqs_rwsem);
596 iommufd_put_object(ucmd->ictx, &viommu->obj);
597 return rc;
598}
drivers/iommu/iommufd/fault.c+0−342
@@ -1,342 +0,0 @@
1// SPDX-License-Identifier: GPL-2.0-only
2/* Copyright (C) 2024 Intel Corporation
3 */
4#define pr_fmt(fmt) "iommufd: " fmt
5
6#include <linux/anon_inodes.h>
7#include <linux/file.h>
8#include <linux/fs.h>
9#include <linux/iommufd.h>
10#include <linux/module.h>
11#include <linux/mutex.h>
12#include <linux/pci.h>
13#include <linux/pci-ats.h>
14#include <linux/poll.h>
15#include <uapi/linux/iommufd.h>
16
17#include "../iommu-priv.h"
18#include "iommufd_private.h"
19
20int iommufd_fault_iopf_enable(struct iommufd_device *idev)
21{
22 struct device *dev = idev->dev;
23 int ret;
24
25 /*
26 * Once we turn on PCI/PRI support for VF, the response failure code
27 * should not be forwarded to the hardware due to PRI being a shared
28 * resource between PF and VFs. There is no coordination for this
29 * shared capability. This waits for a vPRI reset to recover.
30 */
31 if (dev_is_pci(dev)) {
32 struct pci_dev *pdev = to_pci_dev(dev);
33
34 if (pdev->is_virtfn && pci_pri_supported(pdev))
35 return -EINVAL;
36 }
37
38 mutex_lock(&idev->iopf_lock);
39 /* Device iopf has already been on. */
40 if (++idev->iopf_enabled > 1) {
41 mutex_unlock(&idev->iopf_lock);
42 return 0;
43 }
44
45 ret = iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_IOPF);
46 if (ret)
47 --idev->iopf_enabled;
48 mutex_unlock(&idev->iopf_lock);
49
50 return ret;
51}
52
53void iommufd_fault_iopf_disable(struct iommufd_device *idev)
54{
55 mutex_lock(&idev->iopf_lock);
56 if (!WARN_ON(idev->iopf_enabled == 0)) {
57 if (--idev->iopf_enabled == 0)
58 iommu_dev_disable_feature(idev->dev, IOMMU_DEV_FEAT_IOPF);
59 }
60 mutex_unlock(&idev->iopf_lock);
61}
62
63void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt,
64 struct iommufd_attach_handle *handle)
65{
66 struct iommufd_fault *fault = hwpt->fault;
67 struct iopf_group *group, *next;
68 struct list_head free_list;
69 unsigned long index;
70
71 if (!fault)
72 return;
73 INIT_LIST_HEAD(&free_list);
74
75 mutex_lock(&fault->mutex);
76 spin_lock(&fault->lock);
77 list_for_each_entry_safe(group, next, &fault->deliver, node) {
78 if (group->attach_handle != &handle->handle)
79 continue;
80 list_move(&group->node, &free_list);
81 }
82 spin_unlock(&fault->lock);
83
84 list_for_each_entry_safe(group, next, &free_list, node) {
85 list_del(&group->node);
86 iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
87 iopf_free_group(group);
88 }
89
90 xa_for_each(&fault->response, index, group) {
91 if (group->attach_handle != &handle->handle)
92 continue;
93 xa_erase(&fault->response, index);
94 iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
95 iopf_free_group(group);
96 }
97 mutex_unlock(&fault->mutex);
98}
99
100void iommufd_fault_destroy(struct iommufd_object *obj)
101{
102 struct iommufd_fault *fault = container_of(obj, struct iommufd_fault, obj);
103 struct iopf_group *group, *next;
104 unsigned long index;
105
106 /*
107 * The iommufd object's reference count is zero at this point.
108 * We can be confident that no other threads are currently
109 * accessing this pointer. Therefore, acquiring the mutex here
110 * is unnecessary.
111 */
112 list_for_each_entry_safe(group, next, &fault->deliver, node) {
113 list_del(&group->node);
114 iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
115 iopf_free_group(group);
116 }
117 xa_for_each(&fault->response, index, group) {
118 xa_erase(&fault->response, index);
119 iopf_group_response(group, IOMMU_PAGE_RESP_INVALID);
120 iopf_free_group(group);
121 }
122 xa_destroy(&fault->response);
123 mutex_destroy(&fault->mutex);
124}
125
126static void iommufd_compose_fault_message(struct iommu_fault *fault,
127 struct iommu_hwpt_pgfault *hwpt_fault,
128 struct iommufd_device *idev,
129 u32 cookie)
130{
131 hwpt_fault->flags = fault->prm.flags;
132 hwpt_fault->dev_id = idev->obj.id;
133 hwpt_fault->pasid = fault->prm.pasid;
134 hwpt_fault->grpid = fault->prm.grpid;
135 hwpt_fault->perm = fault->prm.perm;
136 hwpt_fault->addr = fault->prm.addr;
137 hwpt_fault->length = 0;
138 hwpt_fault->cookie = cookie;
139}
140
141static ssize_t iommufd_fault_fops_read(struct file *filep, char __user *buf,
142 size_t count, loff_t *ppos)
143{
144 size_t fault_size = sizeof(struct iommu_hwpt_pgfault);
145 struct iommufd_fault *fault = filep->private_data;
146 struct iommu_hwpt_pgfault data = {};
147 struct iommufd_device *idev;
148 struct iopf_group *group;
149 struct iopf_fault *iopf;
150 size_t done = 0;
151 int rc = 0;
152
153 if (*ppos || count % fault_size)
154 return -ESPIPE;
155
156 mutex_lock(&fault->mutex);
157 while ((group = iommufd_fault_deliver_fetch(fault))) {
158 if (done >= count ||
159 group->fault_count * fault_size > count - done) {
160 iommufd_fault_deliver_restore(fault, group);
161 break;
162 }
163
164 rc = xa_alloc(&fault->response, &group->cookie, group,
165 xa_limit_32b, GFP_KERNEL);
166 if (rc) {
167 iommufd_fault_deliver_restore(fault, group);
168 break;
169 }
170
171 idev = to_iommufd_handle(group->attach_handle)->idev;
172 list_for_each_entry(iopf, &group->faults, list) {
173 iommufd_compose_fault_message(&iopf->fault,
174 &data, idev,
175 group->cookie);
176 if (copy_to_user(buf + done, &data, fault_size)) {
177 xa_erase(&fault->response, group->cookie);
178 iommufd_fault_deliver_restore(fault, group);
179 rc = -EFAULT;
180 break;
181 }
182 done += fault_size;
183 }
184 }
185 mutex_unlock(&fault->mutex);
186
187 return done == 0 ? rc : done;
188}
189
190static ssize_t iommufd_fault_fops_write(struct file *filep, const char __user *buf,
191 size_t count, loff_t *ppos)
192{
193 size_t response_size = sizeof(struct iommu_hwpt_page_response);
194 struct iommufd_fault *fault = filep->private_data;
195 struct iommu_hwpt_page_response response;
196 struct iopf_group *group;
197 size_t done = 0;
198 int rc = 0;
199
200 if (*ppos || count % response_size)
201 return -ESPIPE;
202
203 mutex_lock(&fault->mutex);
204 while (count > done) {
205 rc = copy_from_user(&response, buf + done, response_size);
206 if (rc)
207 break;
208
209 static_assert((int)IOMMUFD_PAGE_RESP_SUCCESS ==
210 (int)IOMMU_PAGE_RESP_SUCCESS);
211 static_assert((int)IOMMUFD_PAGE_RESP_INVALID ==
212 (int)IOMMU_PAGE_RESP_INVALID);
213 if (response.code != IOMMUFD_PAGE_RESP_SUCCESS &&
214 response.code != IOMMUFD_PAGE_RESP_INVALID) {
215 rc = -EINVAL;
216 break;
217 }
218
219 group = xa_erase(&fault->response, response.cookie);
220 if (!group) {
221 rc = -EINVAL;
222 break;
223 }
224
225 iopf_group_response(group, response.code);
226 iopf_free_group(group);
227 done += response_size;
228 }
229 mutex_unlock(&fault->mutex);
230
231 return done == 0 ? rc : done;
232}
233
234static __poll_t iommufd_fault_fops_poll(struct file *filep,
235 struct poll_table_struct *wait)
236{
237 struct iommufd_fault *fault = filep->private_data;
238 __poll_t pollflags = EPOLLOUT;
239
240 poll_wait(filep, &fault->wait_queue, wait);
241 spin_lock(&fault->lock);
242 if (!list_empty(&fault->deliver))
243 pollflags |= EPOLLIN | EPOLLRDNORM;
244 spin_unlock(&fault->lock);
245
246 return pollflags;
247}
248
249static int iommufd_fault_fops_release(struct inode *inode, struct file *filep)
250{
251 struct iommufd_fault *fault = filep->private_data;
252
253 refcount_dec(&fault->obj.users);
254 iommufd_ctx_put(fault->ictx);
255 return 0;
256}
257
258static const struct file_operations iommufd_fault_fops = {
259 .owner = THIS_MODULE,
260 .open = nonseekable_open,
261 .read = iommufd_fault_fops_read,
262 .write = iommufd_fault_fops_write,
263 .poll = iommufd_fault_fops_poll,
264 .release = iommufd_fault_fops_release,
265};
266
267int iommufd_fault_alloc(struct iommufd_ucmd *ucmd)
268{
269 struct iommu_fault_alloc *cmd = ucmd->cmd;
270 struct iommufd_fault *fault;
271 struct file *filep;
272 int fdno;
273 int rc;
274
275 if (cmd->flags)
276 return -EOPNOTSUPP;
277
278 fault = iommufd_object_alloc(ucmd->ictx, fault, IOMMUFD_OBJ_FAULT);
279 if (IS_ERR(fault))
280 return PTR_ERR(fault);
281
282 fault->ictx = ucmd->ictx;
283 INIT_LIST_HEAD(&fault->deliver);
284 xa_init_flags(&fault->response, XA_FLAGS_ALLOC1);
285 mutex_init(&fault->mutex);
286 spin_lock_init(&fault->lock);
287 init_waitqueue_head(&fault->wait_queue);
288
289 filep = anon_inode_getfile("[iommufd-pgfault]", &iommufd_fault_fops,
290 fault, O_RDWR);
291 if (IS_ERR(filep)) {
292 rc = PTR_ERR(filep);
293 goto out_abort;
294 }
295
296 refcount_inc(&fault->obj.users);
297 iommufd_ctx_get(fault->ictx);
298 fault->filep = filep;
299
300 fdno = get_unused_fd_flags(O_CLOEXEC);
301 if (fdno < 0) {
302 rc = fdno;
303 goto out_fput;
304 }
305
306 cmd->out_fault_id = fault->obj.id;
307 cmd->out_fault_fd = fdno;
308
309 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
310 if (rc)
311 goto out_put_fdno;
312 iommufd_object_finalize(ucmd->ictx, &fault->obj);
313
314 fd_install(fdno, fault->filep);
315
316 return 0;
317out_put_fdno:
318 put_unused_fd(fdno);
319out_fput:
320 fput(filep);
321out_abort:
322 iommufd_object_abort_and_destroy(ucmd->ictx, &fault->obj);
323
324 return rc;
325}
326
327int iommufd_fault_iopf_handler(struct iopf_group *group)
328{
329 struct iommufd_hw_pagetable *hwpt;
330 struct iommufd_fault *fault;
331
332 hwpt = group->attach_handle->domain->iommufd_hwpt;
333 fault = hwpt->fault;
334
335 spin_lock(&fault->lock);
336 list_add_tail(&group->node, &fault->deliver);
337 spin_unlock(&fault->lock);
338
339 wake_up_interruptible(&fault->wait_queue);
340
341 return 0;
342}
drivers/iommu/iommufd/hw_pagetable.c+26−16
@@ -14,7 +14,7 @@ static void __iommufd_hwpt_destroy(struct iommufd_hw_pagetable *hwpt) · 13 unmodified lines
14 iommu_domain_free(hwpt->domain);14 iommu_domain_free(hwpt->domain);
15 15
16 if (hwpt->fault)16 if (hwpt->fault)
17 refcount_dec(&hwpt->fault->obj.users);17 refcount_dec(&hwpt->fault->common.obj.users);
18}18}
19 19
20void iommufd_hwpt_paging_destroy(struct iommufd_object *obj)20void iommufd_hwpt_paging_destroy(struct iommufd_object *obj)
@@ -90,6 +90,7 @@ iommufd_hwpt_paging_enforce_cc(struct iommufd_hwpt_paging *hwpt_paging) · 69 unmodified lines
90 * @ictx: iommufd context90 * @ictx: iommufd context
91 * @ioas: IOAS to associate the domain with91 * @ioas: IOAS to associate the domain with
92 * @idev: Device to get an iommu_domain for92 * @idev: Device to get an iommu_domain for
93 * @pasid: PASID to get an iommu_domain for
93 * @flags: Flags from userspace94 * @flags: Flags from userspace
94 * @immediate_attach: True if idev should be attached to the hwpt95 * @immediate_attach: True if idev should be attached to the hwpt
95 * @user_data: The user provided driver specific data describing the domain to96 * @user_data: The user provided driver specific data describing the domain to
@@ -105,13 +106,14 @@ iommufd_hwpt_paging_enforce_cc(struct iommufd_hwpt_paging *hwpt_paging) · 9 unmodified lines
105 */106 */
106struct iommufd_hwpt_paging *107struct iommufd_hwpt_paging *
107iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas,108iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas,
108 struct iommufd_device *idev, u32 flags,109 struct iommufd_device *idev, ioasid_t pasid,
109 bool immediate_attach,110 u32 flags, bool immediate_attach,
110 const struct iommu_user_data *user_data)111 const struct iommu_user_data *user_data)
111{112{
112 const u32 valid_flags = IOMMU_HWPT_ALLOC_NEST_PARENT |113 const u32 valid_flags = IOMMU_HWPT_ALLOC_NEST_PARENT |
113 IOMMU_HWPT_ALLOC_DIRTY_TRACKING |114 IOMMU_HWPT_ALLOC_DIRTY_TRACKING |
114 IOMMU_HWPT_FAULT_ID_VALID;115 IOMMU_HWPT_FAULT_ID_VALID |
116 IOMMU_HWPT_ALLOC_PASID;
115 const struct iommu_ops *ops = dev_iommu_ops(idev->dev);117 const struct iommu_ops *ops = dev_iommu_ops(idev->dev);
116 struct iommufd_hwpt_paging *hwpt_paging;118 struct iommufd_hwpt_paging *hwpt_paging;
117 struct iommufd_hw_pagetable *hwpt;119 struct iommufd_hw_pagetable *hwpt;
@@ -126,12 +128,16 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 8 unmodified lines
126 if ((flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING) &&128 if ((flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING) &&
127 !device_iommu_capable(idev->dev, IOMMU_CAP_DIRTY_TRACKING))129 !device_iommu_capable(idev->dev, IOMMU_CAP_DIRTY_TRACKING))
128 return ERR_PTR(-EOPNOTSUPP);130 return ERR_PTR(-EOPNOTSUPP);
131 if ((flags & IOMMU_HWPT_FAULT_ID_VALID) &&
132 (flags & IOMMU_HWPT_ALLOC_NEST_PARENT))
133 return ERR_PTR(-EOPNOTSUPP);
129 134
130 hwpt_paging = __iommufd_object_alloc(135 hwpt_paging = __iommufd_object_alloc(
131 ictx, hwpt_paging, IOMMUFD_OBJ_HWPT_PAGING, common.obj);136 ictx, hwpt_paging, IOMMUFD_OBJ_HWPT_PAGING, common.obj);
132 if (IS_ERR(hwpt_paging))137 if (IS_ERR(hwpt_paging))
133 return ERR_CAST(hwpt_paging);138 return ERR_CAST(hwpt_paging);
134 hwpt = &hwpt_paging->common;139 hwpt = &hwpt_paging->common;
140 hwpt->pasid_compat = flags & IOMMU_HWPT_ALLOC_PASID;
135 141
136 INIT_LIST_HEAD(&hwpt_paging->hwpt_item);142 INIT_LIST_HEAD(&hwpt_paging->hwpt_item);
137 /* Pairs with iommufd_hw_pagetable_destroy() */143 /* Pairs with iommufd_hw_pagetable_destroy() */
@@ -156,7 +162,8 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 18 unmodified lines
156 goto out_abort;162 goto out_abort;
157 }163 }
158 }164 }
159 iommu_domain_set_sw_msi(hwpt->domain, iommufd_sw_msi);165 hwpt->domain->iommufd_hwpt = hwpt;
166 hwpt->domain->cookie_type = IOMMU_COOKIE_IOMMUFD;
160 167
161 /*168 /*
162 * Set the coherency mode before we do iopt_table_add_domain() as some169 * Set the coherency mode before we do iopt_table_add_domain() as some
@@ -185,7 +192,7 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 22 unmodified lines
185 * sequence. Once those drivers are fixed this should be removed.192 * sequence. Once those drivers are fixed this should be removed.
186 */193 */
187 if (immediate_attach) {194 if (immediate_attach) {
188 rc = iommufd_hw_pagetable_attach(hwpt, idev);195 rc = iommufd_hw_pagetable_attach(hwpt, idev, pasid);
189 if (rc)196 if (rc)
190 goto out_abort;197 goto out_abort;
191 }198 }
@@ -198,7 +205,7 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 6 unmodified lines
198 205
199out_detach:206out_detach:
200 if (immediate_attach)207 if (immediate_attach)
201 iommufd_hw_pagetable_detach(idev);208 iommufd_hw_pagetable_detach(idev, pasid);
202out_abort:209out_abort:
203 iommufd_object_abort_and_destroy(ictx, &hwpt->obj);210 iommufd_object_abort_and_destroy(ictx, &hwpt->obj);
204 return ERR_PTR(rc);211 return ERR_PTR(rc);
@@ -227,7 +234,7 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, · 22 unmodified lines
227 struct iommufd_hw_pagetable *hwpt;234 struct iommufd_hw_pagetable *hwpt;
228 int rc;235 int rc;
229 236
230 if ((flags & ~IOMMU_HWPT_FAULT_ID_VALID) ||237 if ((flags & ~(IOMMU_HWPT_FAULT_ID_VALID | IOMMU_HWPT_ALLOC_PASID)) ||
231 !user_data->len || !ops->domain_alloc_nested)238 !user_data->len || !ops->domain_alloc_nested)
232 return ERR_PTR(-EOPNOTSUPP);239 return ERR_PTR(-EOPNOTSUPP);
233 if (parent->auto_domain || !parent->nest_parent ||240 if (parent->auto_domain || !parent->nest_parent ||
@@ -239,6 +246,7 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, · 5 unmodified lines
239 if (IS_ERR(hwpt_nested))246 if (IS_ERR(hwpt_nested))
240 return ERR_CAST(hwpt_nested);247 return ERR_CAST(hwpt_nested);
241 hwpt = &hwpt_nested->common;248 hwpt = &hwpt_nested->common;
249 hwpt->pasid_compat = flags & IOMMU_HWPT_ALLOC_PASID;
242 250
243 refcount_inc(&parent->common.obj.users);251 refcount_inc(&parent->common.obj.users);
244 hwpt_nested->parent = parent;252 hwpt_nested->parent = parent;
@@ -252,7 +260,8 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, · 7 unmodified lines
252 goto out_abort;260 goto out_abort;
253 }261 }
254 hwpt->domain->owner = ops;262 hwpt->domain->owner = ops;
255 iommu_domain_set_sw_msi(hwpt->domain, iommufd_sw_msi);263 hwpt->domain->iommufd_hwpt = hwpt;
264 hwpt->domain->cookie_type = IOMMU_COOKIE_IOMMUFD;
256 265
257 if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) {266 if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) {
258 rc = -EINVAL;267 rc = -EINVAL;
@@ -282,7 +291,7 @@ iommufd_viommu_alloc_hwpt_nested(struct iommufd_viommu *viommu, u32 flags, · 23 unmodified lines
282 struct iommufd_hw_pagetable *hwpt;291 struct iommufd_hw_pagetable *hwpt;
283 int rc;292 int rc;
284 293
285 if (flags & ~IOMMU_HWPT_FAULT_ID_VALID)294 if (flags & ~(IOMMU_HWPT_FAULT_ID_VALID | IOMMU_HWPT_ALLOC_PASID))
286 return ERR_PTR(-EOPNOTSUPP);295 return ERR_PTR(-EOPNOTSUPP);
287 if (!user_data->len)296 if (!user_data->len)
288 return ERR_PTR(-EOPNOTSUPP);297 return ERR_PTR(-EOPNOTSUPP);
@@ -294,6 +303,7 @@ iommufd_viommu_alloc_hwpt_nested(struct iommufd_viommu *viommu, u32 flags, · 5 unmodified lines
294 if (IS_ERR(hwpt_nested))303 if (IS_ERR(hwpt_nested))
295 return ERR_CAST(hwpt_nested);304 return ERR_CAST(hwpt_nested);
296 hwpt = &hwpt_nested->common;305 hwpt = &hwpt_nested->common;
306 hwpt->pasid_compat = flags & IOMMU_HWPT_ALLOC_PASID;
297 307
298 hwpt_nested->viommu = viommu;308 hwpt_nested->viommu = viommu;
299 refcount_inc(&viommu->obj.users);309 refcount_inc(&viommu->obj.users);
@@ -308,8 +318,9 @@ iommufd_viommu_alloc_hwpt_nested(struct iommufd_viommu *viommu, u32 flags, · 8 unmodified lines
308 hwpt->domain = NULL;318 hwpt->domain = NULL;
309 goto out_abort;319 goto out_abort;
310 }320 }
321 hwpt->domain->iommufd_hwpt = hwpt;
311 hwpt->domain->owner = viommu->iommu_dev->ops;322 hwpt->domain->owner = viommu->iommu_dev->ops;
312 iommu_domain_set_sw_msi(hwpt->domain, iommufd_sw_msi);323 hwpt->domain->cookie_type = IOMMU_COOKIE_IOMMUFD;
313 324
314 if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) {325 if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) {
315 rc = -EINVAL;326 rc = -EINVAL;
@@ -358,8 +369,8 @@ int iommufd_hwpt_alloc(struct iommufd_ucmd *ucmd) · 42 unmodified lines
358 ioas = container_of(pt_obj, struct iommufd_ioas, obj);369 ioas = container_of(pt_obj, struct iommufd_ioas, obj);
359 mutex_lock(&ioas->mutex);370 mutex_lock(&ioas->mutex);
360 hwpt_paging = iommufd_hwpt_paging_alloc(371 hwpt_paging = iommufd_hwpt_paging_alloc(
361 ucmd->ictx, ioas, idev, cmd->flags, false,372 ucmd->ictx, ioas, idev, IOMMU_NO_PASID, cmd->flags,
362 user_data.len ? &user_data : NULL);373 false, user_data.len ? &user_data : NULL);
363 if (IS_ERR(hwpt_paging)) {374 if (IS_ERR(hwpt_paging)) {
364 rc = PTR_ERR(hwpt_paging);375 rc = PTR_ERR(hwpt_paging);
365 goto out_unlock;376 goto out_unlock;
@@ -409,10 +420,9 @@ int iommufd_hwpt_alloc(struct iommufd_ucmd *ucmd) · 43 unmodified lines
409 }420 }
410 hwpt->fault = fault;421 hwpt->fault = fault;
411 hwpt->domain->iopf_handler = iommufd_fault_iopf_handler;422 hwpt->domain->iopf_handler = iommufd_fault_iopf_handler;
412 refcount_inc(&fault->obj.users);423 refcount_inc(&fault->common.obj.users);
413 iommufd_put_object(ucmd->ictx, &fault->obj);424 iommufd_put_object(ucmd->ictx, &fault->common.obj);
414 }425 }
415 hwpt->domain->iommufd_hwpt = hwpt;
416 426
417 cmd->out_hwpt_id = hwpt->obj.id;427 cmd->out_hwpt_id = hwpt->obj.id;
418 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));428 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
drivers/iommu/iommufd/iommufd_private.h+114−42
@@ -32,8 +32,11 @@ struct iommufd_sw_msi_maps { · 31 unmodified lines
32 DECLARE_BITMAP(bitmap, 64);32 DECLARE_BITMAP(bitmap, 64);
33};33};
34 34
35int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,35#ifdef CONFIG_IRQ_MSI_IOMMU
36 phys_addr_t msi_addr);36int iommufd_sw_msi_install(struct iommufd_ctx *ictx,
37 struct iommufd_hwpt_paging *hwpt_paging,
38 struct iommufd_sw_msi_map *msi_map);
39#endif
37 40
38struct iommufd_ctx {41struct iommufd_ctx {
39 struct file *file;42 struct file *file;
@@ -296,6 +299,7 @@ struct iommufd_hw_pagetable { · 256 unmodified lines
296 struct iommufd_object obj;299 struct iommufd_object obj;
297 struct iommu_domain *domain;300 struct iommu_domain *domain;
298 struct iommufd_fault *fault;301 struct iommufd_fault *fault;
302 bool pasid_compat : 1;
299};303};
300 304
301struct iommufd_hwpt_paging {305struct iommufd_hwpt_paging {
@@ -366,13 +370,13 @@ int iommufd_hwpt_get_dirty_bitmap(struct iommufd_ucmd *ucmd); · 64 unmodified lines
366 370
367struct iommufd_hwpt_paging *371struct iommufd_hwpt_paging *
368iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas,372iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas,
369 struct iommufd_device *idev, u32 flags,373 struct iommufd_device *idev, ioasid_t pasid,
370 bool immediate_attach,374 u32 flags, bool immediate_attach,
371 const struct iommu_user_data *user_data);375 const struct iommu_user_data *user_data);
372int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,376int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,
373 struct iommufd_device *idev);377 struct iommufd_device *idev, ioasid_t pasid);
374struct iommufd_hw_pagetable *378struct iommufd_hw_pagetable *
375iommufd_hw_pagetable_detach(struct iommufd_device *idev);379iommufd_hw_pagetable_detach(struct iommufd_device *idev, ioasid_t pasid);
376void iommufd_hwpt_paging_destroy(struct iommufd_object *obj);380void iommufd_hwpt_paging_destroy(struct iommufd_object *obj);
377void iommufd_hwpt_paging_abort(struct iommufd_object *obj);381void iommufd_hwpt_paging_abort(struct iommufd_object *obj);
378void iommufd_hwpt_nested_destroy(struct iommufd_object *obj);382void iommufd_hwpt_nested_destroy(struct iommufd_object *obj);
@@ -396,13 +400,14 @@ static inline void iommufd_hw_pagetable_put(struct iommufd_ctx *ictx, · 17 unmodified lines
396 refcount_dec(&hwpt->obj.users);400 refcount_dec(&hwpt->obj.users);
397}401}
398 402
403struct iommufd_attach;
404
399struct iommufd_group {405struct iommufd_group {
400 struct kref ref;406 struct kref ref;
401 struct mutex lock;407 struct mutex lock;
402 struct iommufd_ctx *ictx;408 struct iommufd_ctx *ictx;
403 struct iommu_group *group;409 struct iommu_group *group;
404 struct iommufd_hw_pagetable *hwpt;410 struct xarray pasid_attach;
405 struct list_head device_list;
406 struct iommufd_sw_msi_maps required_sw_msi;411 struct iommufd_sw_msi_maps required_sw_msi;
407 phys_addr_t sw_msi_start;412 phys_addr_t sw_msi_start;
408};413};
@@ -454,49 +459,17 @@ void iopt_remove_access(struct io_pagetable *iopt, · 45 unmodified lines
454 u32 iopt_access_list_id);459 u32 iopt_access_list_id);
455void iommufd_access_destroy_object(struct iommufd_object *obj);460void iommufd_access_destroy_object(struct iommufd_object *obj);
456 461
457/*462struct iommufd_eventq {
458 * An iommufd_fault object represents an interface to deliver I/O page faults
459 * to the user space. These objects are created/destroyed by the user space and
460 * associated with hardware page table objects during page-table allocation.
461 */
462struct iommufd_fault {
463 struct iommufd_object obj;463 struct iommufd_object obj;
464 struct iommufd_ctx *ictx;464 struct iommufd_ctx *ictx;
465 struct file *filep;465 struct file *filep;
466 466
467 spinlock_t lock; /* protects the deliver list */467 spinlock_t lock; /* protects the deliver list */
468 struct list_head deliver;468 struct list_head deliver;
469 struct mutex mutex; /* serializes response flows */
470 struct xarray response;
471 469
472 struct wait_queue_head wait_queue;470 struct wait_queue_head wait_queue;
473};471};
474 472
475/* Fetch the first node out of the fault->deliver list */
476static inline struct iopf_group *
477iommufd_fault_deliver_fetch(struct iommufd_fault *fault)
478{
479 struct list_head *list = &fault->deliver;
480 struct iopf_group *group = NULL;
481
482 spin_lock(&fault->lock);
483 if (!list_empty(list)) {
484 group = list_first_entry(list, struct iopf_group, node);
485 list_del(&group->node);
486 }
487 spin_unlock(&fault->lock);
488 return group;
489}
490
491/* Restore a node back to the head of the fault->deliver list */
492static inline void iommufd_fault_deliver_restore(struct iommufd_fault *fault,
493 struct iopf_group *group)
494{
495 spin_lock(&fault->lock);
496 list_add(&group->node, &fault->deliver);
497 spin_unlock(&fault->lock);
498}
499
500struct iommufd_attach_handle {473struct iommufd_attach_handle {
501 struct iommu_attach_handle handle;474 struct iommu_attach_handle handle;
502 struct iommufd_device *idev;475 struct iommufd_device *idev;
@@ -505,12 +478,29 @@ struct iommufd_attach_handle { · 2 unmodified lines
505/* Convert an iommu attach handle to iommufd handle. */478/* Convert an iommu attach handle to iommufd handle. */
506#define to_iommufd_handle(hdl) container_of(hdl, struct iommufd_attach_handle, handle)479#define to_iommufd_handle(hdl) container_of(hdl, struct iommufd_attach_handle, handle)
507 480
481/*
482 * An iommufd_fault object represents an interface to deliver I/O page faults
483 * to the user space. These objects are created/destroyed by the user space and
484 * associated with hardware page table objects during page-table allocation.
485 */
486struct iommufd_fault {
487 struct iommufd_eventq common;
488 struct mutex mutex; /* serializes response flows */
489 struct xarray response;
490};
491
492static inline struct iommufd_fault *
493eventq_to_fault(struct iommufd_eventq *eventq)
494{
495 return container_of(eventq, struct iommufd_fault, common);
496}
497
508static inline struct iommufd_fault *498static inline struct iommufd_fault *
509iommufd_get_fault(struct iommufd_ucmd *ucmd, u32 id)499iommufd_get_fault(struct iommufd_ucmd *ucmd, u32 id)
510{500{
511 return container_of(iommufd_get_object(ucmd->ictx, id,501 return container_of(iommufd_get_object(ucmd->ictx, id,
512 IOMMUFD_OBJ_FAULT),502 IOMMUFD_OBJ_FAULT),
513 struct iommufd_fault, obj);503 struct iommufd_fault, common.obj);
514}504}
515 505
516int iommufd_fault_alloc(struct iommufd_ucmd *ucmd);506int iommufd_fault_alloc(struct iommufd_ucmd *ucmd);
@@ -522,6 +512,74 @@ void iommufd_fault_iopf_disable(struct iommufd_device *idev); · 5 unmodified lines
522void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt,512void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt,
523 struct iommufd_attach_handle *handle);513 struct iommufd_attach_handle *handle);
524 514
515/* An iommufd_vevent represents a vIOMMU event in an iommufd_veventq */
516struct iommufd_vevent {
517 struct iommufd_vevent_header header;
518 struct list_head node; /* for iommufd_eventq::deliver */
519 ssize_t data_len;
520 u64 event_data[] __counted_by(data_len);
521};
522
523#define vevent_for_lost_events_header(vevent) \
524 (vevent->header.flags & IOMMU_VEVENTQ_FLAG_LOST_EVENTS)
525
526/*
527 * An iommufd_veventq object represents an interface to deliver vIOMMU events to
528 * the user space. It is created/destroyed by the user space and associated with
529 * a vIOMMU object during the allocations.
530 */
531struct iommufd_veventq {
532 struct iommufd_eventq common;
533 struct iommufd_viommu *viommu;
534 struct list_head node; /* for iommufd_viommu::veventqs */
535 struct iommufd_vevent lost_events_header;
536
537 unsigned int type;
538 unsigned int depth;
539
540 /* Use common.lock for protection */
541 u32 num_events;
542 u32 sequence;
543};
544
545static inline struct iommufd_veventq *
546eventq_to_veventq(struct iommufd_eventq *eventq)
547{
548 return container_of(eventq, struct iommufd_veventq, common);
549}
550
551static inline struct iommufd_veventq *
552iommufd_get_veventq(struct iommufd_ucmd *ucmd, u32 id)
553{
554 return container_of(iommufd_get_object(ucmd->ictx, id,
555 IOMMUFD_OBJ_VEVENTQ),
556 struct iommufd_veventq, common.obj);
557}
558
559int iommufd_veventq_alloc(struct iommufd_ucmd *ucmd);
560void iommufd_veventq_destroy(struct iommufd_object *obj);
561void iommufd_veventq_abort(struct iommufd_object *obj);
562
563static inline void iommufd_vevent_handler(struct iommufd_veventq *veventq,
564 struct iommufd_vevent *vevent)
565{
566 struct iommufd_eventq *eventq = &veventq->common;
567
568 lockdep_assert_held(&eventq->lock);
569
570 /*
571 * Remove the lost_events_header and add the new node at the same time.
572 * Note the new node can be lost_events_header, for a sequence update.
573 */
574 if (list_is_last(&veventq->lost_events_header.node, &eventq->deliver))
575 list_del(&veventq->lost_events_header.node);
576 list_add_tail(&vevent->node, &eventq->deliver);
577 vevent->header.sequence = veventq->sequence;
578 veventq->sequence = (veventq->sequence + 1) & INT_MAX;
579
580 wake_up_interruptible(&eventq->wait_queue);
581}
582
525static inline struct iommufd_viommu *583static inline struct iommufd_viommu *
526iommufd_get_viommu(struct iommufd_ucmd *ucmd, u32 id)584iommufd_get_viommu(struct iommufd_ucmd *ucmd, u32 id)
527{585{
@@ -530,6 +588,20 @@ iommufd_get_viommu(struct iommufd_ucmd *ucmd, u32 id) · 2 unmodified lines
530 struct iommufd_viommu, obj);588 struct iommufd_viommu, obj);
531}589}
532 590
591static inline struct iommufd_veventq *
592iommufd_viommu_find_veventq(struct iommufd_viommu *viommu, u32 type)
593{
594 struct iommufd_veventq *veventq, *next;
595
596 lockdep_assert_held(&viommu->veventqs_rwsem);
597
598 list_for_each_entry_safe(veventq, next, &viommu->veventqs, node) {
599 if (veventq->type == type)
600 return veventq;
601 }
602 return NULL;
603}
604
533int iommufd_viommu_alloc_ioctl(struct iommufd_ucmd *ucmd);605int iommufd_viommu_alloc_ioctl(struct iommufd_ucmd *ucmd);
534void iommufd_viommu_destroy(struct iommufd_object *obj);606void iommufd_viommu_destroy(struct iommufd_object *obj);
535int iommufd_vdevice_alloc_ioctl(struct iommufd_ucmd *ucmd);607int iommufd_vdevice_alloc_ioctl(struct iommufd_ucmd *ucmd);
drivers/iommu/iommufd/iommufd_test.h+40−0
@@ -24,6 +24,11 @@ enum { · 23 unmodified lines
24 IOMMU_TEST_OP_MD_CHECK_IOTLB,24 IOMMU_TEST_OP_MD_CHECK_IOTLB,
25 IOMMU_TEST_OP_TRIGGER_IOPF,25 IOMMU_TEST_OP_TRIGGER_IOPF,
26 IOMMU_TEST_OP_DEV_CHECK_CACHE,26 IOMMU_TEST_OP_DEV_CHECK_CACHE,
27 IOMMU_TEST_OP_TRIGGER_VEVENT,
28 IOMMU_TEST_OP_PASID_ATTACH,
29 IOMMU_TEST_OP_PASID_REPLACE,
30 IOMMU_TEST_OP_PASID_DETACH,
31 IOMMU_TEST_OP_PASID_CHECK_HWPT,
27};32};
28 33
29enum {34enum {
@@ -48,6 +53,7 @@ enum { · 18 unmodified lines
48enum {53enum {
49 MOCK_FLAGS_DEVICE_NO_DIRTY = 1 << 0,54 MOCK_FLAGS_DEVICE_NO_DIRTY = 1 << 0,
50 MOCK_FLAGS_DEVICE_HUGE_IOVA = 1 << 1,55 MOCK_FLAGS_DEVICE_HUGE_IOVA = 1 << 1,
56 MOCK_FLAGS_DEVICE_PASID = 1 << 2,
51};57};
52 58
53enum {59enum {
@@ -60,6 +66,9 @@ enum { · 6 unmodified lines
60 MOCK_DEV_CACHE_NUM = 4,66 MOCK_DEV_CACHE_NUM = 4,
61};67};
62 68
69/* Reserved for special pasid replace test */
70#define IOMMU_TEST_PASID_RESERVED 1024
71
63struct iommu_test_cmd {72struct iommu_test_cmd {
64 __u32 size;73 __u32 size;
65 __u32 op;74 __u32 op;
@@ -145,11 +154,36 @@ struct iommu_test_cmd { · 79 unmodified lines
145 __u32 id;154 __u32 id;
146 __u32 cache;155 __u32 cache;
147 } check_dev_cache;156 } check_dev_cache;
157 struct {
158 __u32 dev_id;
159 } trigger_vevent;
160 struct {
161 __u32 pasid;
162 __u32 pt_id;
163 /* @id is stdev_id */
164 } pasid_attach;
165 struct {
166 __u32 pasid;
167 __u32 pt_id;
168 /* @id is stdev_id */
169 } pasid_replace;
170 struct {
171 __u32 pasid;
172 /* @id is stdev_id */
173 } pasid_detach;
174 struct {
175 __u32 pasid;
176 __u32 hwpt_id;
177 /* @id is stdev_id */
178 } pasid_check;
148 };179 };
149 __u32 last;180 __u32 last;
150};181};
151#define IOMMU_TEST_CMD _IO(IOMMUFD_TYPE, IOMMUFD_CMD_BASE + 32)182#define IOMMU_TEST_CMD _IO(IOMMUFD_TYPE, IOMMUFD_CMD_BASE + 32)
152 183
184/* Mock device/iommu PASID width */
185#define MOCK_PASID_WIDTH 20
186
153/* Mock structs for IOMMU_DEVICE_GET_HW_INFO ioctl */187/* Mock structs for IOMMU_DEVICE_GET_HW_INFO ioctl */
154#define IOMMU_HW_INFO_TYPE_SELFTEST 0xfeedbeef188#define IOMMU_HW_INFO_TYPE_SELFTEST 0xfeedbeef
155#define IOMMU_HW_INFO_SELFTEST_REGVAL 0xdeadbeef189#define IOMMU_HW_INFO_SELFTEST_REGVAL 0xdeadbeef
@@ -212,4 +246,10 @@ struct iommu_viommu_invalidate_selftest { · 56 unmodified lines
212 __u32 cache_id;246 __u32 cache_id;
213};247};
214 248
249#define IOMMU_VEVENTQ_TYPE_SELFTEST 0xbeefbeef
250
251struct iommu_viommu_event_selftest {
252 __u32 virt_id;
253};
254
215#endif255#endif
drivers/iommu/iommufd/main.c+7−0
@@ -317,6 +317,7 @@ union ucmd_buffer { · 316 unmodified lines
317 struct iommu_ioas_unmap unmap;317 struct iommu_ioas_unmap unmap;
318 struct iommu_option option;318 struct iommu_option option;
319 struct iommu_vdevice_alloc vdev;319 struct iommu_vdevice_alloc vdev;
320 struct iommu_veventq_alloc veventq;
320 struct iommu_vfio_ioas vfio_ioas;321 struct iommu_vfio_ioas vfio_ioas;
321 struct iommu_viommu_alloc viommu;322 struct iommu_viommu_alloc viommu;
322#ifdef CONFIG_IOMMUFD_TEST323#ifdef CONFIG_IOMMUFD_TEST
@@ -372,6 +373,8 @@ static const struct iommufd_ioctl_op iommufd_ioctl_ops[] = { · 49 unmodified lines
372 IOCTL_OP(IOMMU_OPTION, iommufd_option, struct iommu_option, val64),373 IOCTL_OP(IOMMU_OPTION, iommufd_option, struct iommu_option, val64),
373 IOCTL_OP(IOMMU_VDEVICE_ALLOC, iommufd_vdevice_alloc_ioctl,374 IOCTL_OP(IOMMU_VDEVICE_ALLOC, iommufd_vdevice_alloc_ioctl,
374 struct iommu_vdevice_alloc, virt_id),375 struct iommu_vdevice_alloc, virt_id),
376 IOCTL_OP(IOMMU_VEVENTQ_ALLOC, iommufd_veventq_alloc,
377 struct iommu_veventq_alloc, out_veventq_fd),
375 IOCTL_OP(IOMMU_VFIO_IOAS, iommufd_vfio_ioas, struct iommu_vfio_ioas,378 IOCTL_OP(IOMMU_VFIO_IOAS, iommufd_vfio_ioas, struct iommu_vfio_ioas,
376 __reserved),379 __reserved),
377 IOCTL_OP(IOMMU_VIOMMU_ALLOC, iommufd_viommu_alloc_ioctl,380 IOCTL_OP(IOMMU_VIOMMU_ALLOC, iommufd_viommu_alloc_ioctl,
@@ -514,6 +517,10 @@ static const struct iommufd_object_ops iommufd_object_ops[] = { · 136 unmodified lines
514 [IOMMUFD_OBJ_VDEVICE] = {517 [IOMMUFD_OBJ_VDEVICE] = {
515 .destroy = iommufd_vdevice_destroy,518 .destroy = iommufd_vdevice_destroy,
516 },519 },
520 [IOMMUFD_OBJ_VEVENTQ] = {
521 .destroy = iommufd_veventq_destroy,
522 .abort = iommufd_veventq_abort,
523 },
517 [IOMMUFD_OBJ_VIOMMU] = {524 [IOMMUFD_OBJ_VIOMMU] = {
518 .destroy = iommufd_viommu_destroy,525 .destroy = iommufd_viommu_destroy,
519 },526 },
drivers/iommu/iommufd/selftest.c+275−22
@@ -161,9 +161,13 @@ enum selftest_obj_type { · 160 unmodified lines
161 161
162struct mock_dev {162struct mock_dev {
163 struct device dev;163 struct device dev;
164 struct mock_viommu *viommu;
165 struct rw_semaphore viommu_rwsem;
164 unsigned long flags;166 unsigned long flags;
167 unsigned long vdev_id;
165 int id;168 int id;
166 u32 cache[MOCK_DEV_CACHE_NUM];169 u32 cache[MOCK_DEV_CACHE_NUM];
170 atomic_t pasid_1024_fake_error;
167};171};
168 172
169static inline struct mock_dev *to_mock_dev(struct device *dev)173static inline struct mock_dev *to_mock_dev(struct device *dev)
@@ -193,15 +197,71 @@ static int mock_domain_nop_attach(struct iommu_domain *domain, · 23 unmodified lines
193 struct device *dev)197 struct device *dev)
194{198{
195 struct mock_dev *mdev = to_mock_dev(dev);199 struct mock_dev *mdev = to_mock_dev(dev);
200 struct mock_viommu *new_viommu = NULL;
201 unsigned long vdev_id = 0;
202 int rc;
196 203
197 if (domain->dirty_ops && (mdev->flags & MOCK_FLAGS_DEVICE_NO_DIRTY))204 if (domain->dirty_ops && (mdev->flags & MOCK_FLAGS_DEVICE_NO_DIRTY))
198 return -EINVAL;205 return -EINVAL;
199 206
207 iommu_group_mutex_assert(dev);
208 if (domain->type == IOMMU_DOMAIN_NESTED) {
209 new_viommu = to_mock_nested(domain)->mock_viommu;
210 if (new_viommu) {
211 rc = iommufd_viommu_get_vdev_id(&new_viommu->core, dev,
212 &vdev_id);
213 if (rc)
214 return rc;
215 }
216 }
217 if (new_viommu != mdev->viommu) {
218 down_write(&mdev->viommu_rwsem);
219 mdev->viommu = new_viommu;
220 mdev->vdev_id = vdev_id;
221 up_write(&mdev->viommu_rwsem);
222 }
223
224 return 0;
225}
226
227static int mock_domain_set_dev_pasid_nop(struct iommu_domain *domain,
228 struct device *dev, ioasid_t pasid,
229 struct iommu_domain *old)
230{
231 struct mock_dev *mdev = to_mock_dev(dev);
232
233 /*
234 * Per the first attach with pasid 1024, set the
235 * mdev->pasid_1024_fake_error. Hence the second call of this op
236 * can fake an error to validate the error path of the core. This
237 * is helpful to test the case in which the iommu core needs to
238 * rollback to the old domain due to driver failure. e.g. replace.
239 * User should be careful about the third call of this op, it shall
240 * succeed since the mdev->pasid_1024_fake_error is cleared in the
241 * second call.
242 */
243 if (pasid == 1024) {
244 if (domain->type == IOMMU_DOMAIN_BLOCKED) {
245 atomic_set(&mdev->pasid_1024_fake_error, 0);
246 } else if (atomic_read(&mdev->pasid_1024_fake_error)) {
247 /*
248 * Clear the flag, and fake an error to fail the
249 * replacement.
250 */
251 atomic_set(&mdev->pasid_1024_fake_error, 0);
252 return -ENOMEM;
253 } else {
254 /* Set the flag to fake an error in next call */
255 atomic_set(&mdev->pasid_1024_fake_error, 1);
256 }
257 }
258
200 return 0;259 return 0;
201}260}
202 261
203static const struct iommu_domain_ops mock_blocking_ops = {262static const struct iommu_domain_ops mock_blocking_ops = {
204 .attach_dev = mock_domain_nop_attach,263 .attach_dev = mock_domain_nop_attach,
264 .set_dev_pasid = mock_domain_set_dev_pasid_nop
205};265};
206 266
207static struct iommu_domain mock_blocking_domain = {267static struct iommu_domain mock_blocking_domain = {
@@ -343,7 +403,7 @@ mock_domain_alloc_nested(struct device *dev, struct iommu_domain *parent, · 135 unmodified lines
343 struct mock_iommu_domain_nested *mock_nested;403 struct mock_iommu_domain_nested *mock_nested;
344 struct mock_iommu_domain *mock_parent;404 struct mock_iommu_domain *mock_parent;
345 405
346 if (flags)406 if (flags & ~IOMMU_HWPT_ALLOC_PASID)
347 return ERR_PTR(-EOPNOTSUPP);407 return ERR_PTR(-EOPNOTSUPP);
348 if (!parent || parent->ops != mock_ops.default_domain_ops)408 if (!parent || parent->ops != mock_ops.default_domain_ops)
349 return ERR_PTR(-EINVAL);409 return ERR_PTR(-EINVAL);
@@ -365,7 +425,8 @@ mock_domain_alloc_paging_flags(struct device *dev, u32 flags, · 15 unmodified lines
365{425{
366 bool has_dirty_flag = flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING;426 bool has_dirty_flag = flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING;
367 const u32 PAGING_FLAGS = IOMMU_HWPT_ALLOC_DIRTY_TRACKING |427 const u32 PAGING_FLAGS = IOMMU_HWPT_ALLOC_DIRTY_TRACKING |
368 IOMMU_HWPT_ALLOC_NEST_PARENT;428 IOMMU_HWPT_ALLOC_NEST_PARENT |
429 IOMMU_HWPT_ALLOC_PASID;
369 struct mock_dev *mdev = to_mock_dev(dev);430 struct mock_dev *mdev = to_mock_dev(dev);
370 bool no_dirty_ops = mdev->flags & MOCK_FLAGS_DEVICE_NO_DIRTY;431 bool no_dirty_ops = mdev->flags & MOCK_FLAGS_DEVICE_NO_DIRTY;
371 struct mock_iommu_domain *mock;432 struct mock_iommu_domain *mock;
@@ -585,7 +646,7 @@ mock_viommu_alloc_domain_nested(struct iommufd_viommu *viommu, u32 flags, · 213 unmodified lines
585 struct mock_viommu *mock_viommu = to_mock_viommu(viommu);646 struct mock_viommu *mock_viommu = to_mock_viommu(viommu);
586 struct mock_iommu_domain_nested *mock_nested;647 struct mock_iommu_domain_nested *mock_nested;
587 648
588 if (flags)649 if (flags & ~IOMMU_HWPT_ALLOC_PASID)
589 return ERR_PTR(-EOPNOTSUPP);650 return ERR_PTR(-EOPNOTSUPP);
590 651
591 mock_nested = __mock_domain_alloc_nested(user_data);652 mock_nested = __mock_domain_alloc_nested(user_data);
@@ -720,6 +781,7 @@ static const struct iommu_ops mock_ops = { · 128 unmodified lines
720 .map_pages = mock_domain_map_pages,781 .map_pages = mock_domain_map_pages,
721 .unmap_pages = mock_domain_unmap_pages,782 .unmap_pages = mock_domain_unmap_pages,
722 .iova_to_phys = mock_domain_iova_to_phys,783 .iova_to_phys = mock_domain_iova_to_phys,
784 .set_dev_pasid = mock_domain_set_dev_pasid_nop,
723 },785 },
724};786};
725 787
@@ -780,6 +842,7 @@ static struct iommu_domain_ops domain_nested_ops = { · 54 unmodified lines
780 .free = mock_domain_free_nested,842 .free = mock_domain_free_nested,
781 .attach_dev = mock_domain_nop_attach,843 .attach_dev = mock_domain_nop_attach,
782 .cache_invalidate_user = mock_domain_cache_invalidate_user,844 .cache_invalidate_user = mock_domain_cache_invalidate_user,
845 .set_dev_pasid = mock_domain_set_dev_pasid_nop,
783};846};
784 847
785static inline struct iommufd_hw_pagetable *848static inline struct iommufd_hw_pagetable *
@@ -839,17 +902,24 @@ static void mock_dev_release(struct device *dev) · 53 unmodified lines
839 902
840static struct mock_dev *mock_dev_create(unsigned long dev_flags)903static struct mock_dev *mock_dev_create(unsigned long dev_flags)
841{904{
905 struct property_entry prop[] = {
906 PROPERTY_ENTRY_U32("pasid-num-bits", 0),
907 {},
908 };
909 const u32 valid_flags = MOCK_FLAGS_DEVICE_NO_DIRTY |
910 MOCK_FLAGS_DEVICE_HUGE_IOVA |
911 MOCK_FLAGS_DEVICE_PASID;
842 struct mock_dev *mdev;912 struct mock_dev *mdev;
843 int rc, i;913 int rc, i;
844 914
845 if (dev_flags &915 if (dev_flags & ~valid_flags)
846 ~(MOCK_FLAGS_DEVICE_NO_DIRTY | MOCK_FLAGS_DEVICE_HUGE_IOVA))
847 return ERR_PTR(-EINVAL);916 return ERR_PTR(-EINVAL);
848 917
849 mdev = kzalloc(sizeof(*mdev), GFP_KERNEL);918 mdev = kzalloc(sizeof(*mdev), GFP_KERNEL);
850 if (!mdev)919 if (!mdev)
851 return ERR_PTR(-ENOMEM);920 return ERR_PTR(-ENOMEM);
852 921
922 init_rwsem(&mdev->viommu_rwsem);
853 device_initialize(&mdev->dev);923 device_initialize(&mdev->dev);
854 mdev->flags = dev_flags;924 mdev->flags = dev_flags;
855 mdev->dev.release = mock_dev_release;925 mdev->dev.release = mock_dev_release;
@@ -866,6 +936,15 @@ static struct mock_dev *mock_dev_create(unsigned long dev_flags) · 10 unmodified lines
866 if (rc)936 if (rc)
867 goto err_put;937 goto err_put;
868 938
939 if (dev_flags & MOCK_FLAGS_DEVICE_PASID)
940 prop[0] = PROPERTY_ENTRY_U32("pasid-num-bits", MOCK_PASID_WIDTH);
941
942 rc = device_create_managed_software_node(&mdev->dev, prop, NULL);
943 if (rc) {
944 dev_err(&mdev->dev, "add pasid-num-bits property failed, rc: %d", rc);
945 goto err_put;
946 }
947
869 rc = device_add(&mdev->dev);948 rc = device_add(&mdev->dev);
870 if (rc)949 if (rc)
871 goto err_put;950 goto err_put;
@@ -921,7 +1000,7 @@ static int iommufd_test_mock_domain(struct iommufd_ucmd *ucmd, · 49 unmodified lines
921 }1000 }
922 sobj->idev.idev = idev;1001 sobj->idev.idev = idev;
923 1002
924 rc = iommufd_device_attach(idev, &pt_id);1003 rc = iommufd_device_attach(idev, IOMMU_NO_PASID, &pt_id);
925 if (rc)1004 if (rc)
926 goto out_unbind;1005 goto out_unbind;
927 1006
@@ -936,7 +1015,7 @@ static int iommufd_test_mock_domain(struct iommufd_ucmd *ucmd, · 8 unmodified lines
936 return 0;1015 return 0;
937 1016
938out_detach:1017out_detach:
939 iommufd_device_detach(idev);1018 iommufd_device_detach(idev, IOMMU_NO_PASID);
940out_unbind:1019out_unbind:
941 iommufd_device_unbind(idev);1020 iommufd_device_unbind(idev);
942out_mdev:1021out_mdev:
@@ -946,39 +1025,49 @@ out_sobj: · 3 unmodified lines
946 return rc;1025 return rc;
947}1026}
948 1027
949/* Replace the mock domain with a manually allocated hw_pagetable */1028static struct selftest_obj *
950static int iommufd_test_mock_domain_replace(struct iommufd_ucmd *ucmd,1029iommufd_test_get_selftest_obj(struct iommufd_ctx *ictx, u32 id)
951 unsigned int device_id, u32 pt_id,
952 struct iommu_test_cmd *cmd)
953{1030{
954 struct iommufd_object *dev_obj;1031 struct iommufd_object *dev_obj;
955 struct selftest_obj *sobj;1032 struct selftest_obj *sobj;
956 int rc;
957 1033
958 /*1034 /*
959 * Prefer to use the OBJ_SELFTEST because the destroy_rwsem will ensure1035 * Prefer to use the OBJ_SELFTEST because the destroy_rwsem will ensure
960 * it doesn't race with detach, which is not allowed.1036 * it doesn't race with detach, which is not allowed.
961 */1037 */
962 dev_obj =1038 dev_obj = iommufd_get_object(ictx, id, IOMMUFD_OBJ_SELFTEST);
963 iommufd_get_object(ucmd->ictx, device_id, IOMMUFD_OBJ_SELFTEST);
964 if (IS_ERR(dev_obj))1039 if (IS_ERR(dev_obj))
965 return PTR_ERR(dev_obj);1040 return ERR_CAST(dev_obj);
966 1041
967 sobj = to_selftest_obj(dev_obj);1042 sobj = to_selftest_obj(dev_obj);
968 if (sobj->type != TYPE_IDEV) {1043 if (sobj->type != TYPE_IDEV) {
969 rc = -EINVAL;1044 iommufd_put_object(ictx, dev_obj);
970 goto out_dev_obj;1045 return ERR_PTR(-EINVAL);
971 }1046 }
1047 return sobj;
1048}
972 1049
973 rc = iommufd_device_replace(sobj->idev.idev, &pt_id);1050/* Replace the mock domain with a manually allocated hw_pagetable */
1051static int iommufd_test_mock_domain_replace(struct iommufd_ucmd *ucmd,
1052 unsigned int device_id, u32 pt_id,
1053 struct iommu_test_cmd *cmd)
1054{
1055 struct selftest_obj *sobj;
1056 int rc;
1057
1058 sobj = iommufd_test_get_selftest_obj(ucmd->ictx, device_id);
1059 if (IS_ERR(sobj))
1060 return PTR_ERR(sobj);
1061
1062 rc = iommufd_device_replace(sobj->idev.idev, IOMMU_NO_PASID, &pt_id);
974 if (rc)1063 if (rc)
975 goto out_dev_obj;1064 goto out_sobj;
976 1065
977 cmd->mock_domain_replace.pt_id = pt_id;1066 cmd->mock_domain_replace.pt_id = pt_id;
978 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));1067 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
979 1068
980out_dev_obj:1069out_sobj:
981 iommufd_put_object(ucmd->ictx, dev_obj);1070 iommufd_put_object(ucmd->ictx, &sobj->obj);
982 return rc;1071 return rc;
983}1072}
984 1073
@@ -1597,13 +1686,166 @@ static int iommufd_test_trigger_iopf(struct iommufd_ucmd *ucmd, · 612 unmodified lines
1597 return 0;1686 return 0;
1598}1687}
1599 1688
1689static int iommufd_test_trigger_vevent(struct iommufd_ucmd *ucmd,
1690 struct iommu_test_cmd *cmd)
1691{
1692 struct iommu_viommu_event_selftest test = {};
1693 struct iommufd_device *idev;
1694 struct mock_dev *mdev;
1695 int rc = -ENOENT;
1696
1697 idev = iommufd_get_device(ucmd, cmd->trigger_vevent.dev_id);
1698 if (IS_ERR(idev))
1699 return PTR_ERR(idev);
1700 mdev = to_mock_dev(idev->dev);
1701
1702 down_read(&mdev->viommu_rwsem);
1703 if (!mdev->viommu || !mdev->vdev_id)
1704 goto out_unlock;
1705
1706 test.virt_id = mdev->vdev_id;
1707 rc = iommufd_viommu_report_event(&mdev->viommu->core,
1708 IOMMU_VEVENTQ_TYPE_SELFTEST, &test,
1709 sizeof(test));
1710out_unlock:
1711 up_read(&mdev->viommu_rwsem);
1712 iommufd_put_object(ucmd->ictx, &idev->obj);
1713
1714 return rc;
1715}
1716
1717static inline struct iommufd_hw_pagetable *
1718iommufd_get_hwpt(struct iommufd_ucmd *ucmd, u32 id)
1719{
1720 struct iommufd_object *pt_obj;
1721
1722 pt_obj = iommufd_get_object(ucmd->ictx, id, IOMMUFD_OBJ_ANY);
1723 if (IS_ERR(pt_obj))
1724 return ERR_CAST(pt_obj);
1725
1726 if (pt_obj->type != IOMMUFD_OBJ_HWPT_NESTED &&
1727 pt_obj->type != IOMMUFD_OBJ_HWPT_PAGING) {
1728 iommufd_put_object(ucmd->ictx, pt_obj);
1729 return ERR_PTR(-EINVAL);
1730 }
1731
1732 return container_of(pt_obj, struct iommufd_hw_pagetable, obj);
1733}
1734
1735static int iommufd_test_pasid_check_hwpt(struct iommufd_ucmd *ucmd,
1736 struct iommu_test_cmd *cmd)
1737{
1738 u32 hwpt_id = cmd->pasid_check.hwpt_id;
1739 struct iommu_domain *attached_domain;
1740 struct iommu_attach_handle *handle;
1741 struct iommufd_hw_pagetable *hwpt;
1742 struct selftest_obj *sobj;
1743 struct mock_dev *mdev;
1744 int rc = 0;
1745
1746 sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id);
1747 if (IS_ERR(sobj))
1748 return PTR_ERR(sobj);
1749
1750 mdev = sobj->idev.mock_dev;
1751
1752 handle = iommu_attach_handle_get(mdev->dev.iommu_group,
1753 cmd->pasid_check.pasid, 0);
1754 if (IS_ERR(handle))
1755 attached_domain = NULL;
1756 else
1757 attached_domain = handle->domain;
1758
1759 /* hwpt_id == 0 means to check if pasid is detached */
1760 if (!hwpt_id) {
1761 if (attached_domain)
1762 rc = -EINVAL;
1763 goto out_sobj;
1764 }
1765
1766 hwpt = iommufd_get_hwpt(ucmd, hwpt_id);
1767 if (IS_ERR(hwpt)) {
1768 rc = PTR_ERR(hwpt);
1769 goto out_sobj;
1770 }
1771
1772 if (attached_domain != hwpt->domain)
1773 rc = -EINVAL;
1774
1775 iommufd_put_object(ucmd->ictx, &hwpt->obj);
1776out_sobj:
1777 iommufd_put_object(ucmd->ictx, &sobj->obj);
1778 return rc;
1779}
1780
1781static int iommufd_test_pasid_attach(struct iommufd_ucmd *ucmd,
1782 struct iommu_test_cmd *cmd)
1783{
1784 struct selftest_obj *sobj;
1785 int rc;
1786
1787 sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id);
1788 if (IS_ERR(sobj))
1789 return PTR_ERR(sobj);
1790
1791 rc = iommufd_device_attach(sobj->idev.idev, cmd->pasid_attach.pasid,
1792 &cmd->pasid_attach.pt_id);
1793 if (rc)
1794 goto out_sobj;
1795
1796 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
1797 if (rc)
1798 iommufd_device_detach(sobj->idev.idev,
1799 cmd->pasid_attach.pasid);
1800
1801out_sobj:
1802 iommufd_put_object(ucmd->ictx, &sobj->obj);
1803 return rc;
1804}
1805
1806static int iommufd_test_pasid_replace(struct iommufd_ucmd *ucmd,
1807 struct iommu_test_cmd *cmd)
1808{
1809 struct selftest_obj *sobj;
1810 int rc;
1811
1812 sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id);
1813 if (IS_ERR(sobj))
1814 return PTR_ERR(sobj);
1815
1816 rc = iommufd_device_replace(sobj->idev.idev, cmd->pasid_attach.pasid,
1817 &cmd->pasid_attach.pt_id);
1818 if (rc)
1819 goto out_sobj;
1820
1821 rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd));
1822
1823out_sobj:
1824 iommufd_put_object(ucmd->ictx, &sobj->obj);
1825 return rc;
1826}
1827
1828static int iommufd_test_pasid_detach(struct iommufd_ucmd *ucmd,
1829 struct iommu_test_cmd *cmd)
1830{
1831 struct selftest_obj *sobj;
1832
1833 sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id);
1834 if (IS_ERR(sobj))
1835 return PTR_ERR(sobj);
1836
1837 iommufd_device_detach(sobj->idev.idev, cmd->pasid_detach.pasid);
1838 iommufd_put_object(ucmd->ictx, &sobj->obj);
1839 return 0;
1840}
1841
1600void iommufd_selftest_destroy(struct iommufd_object *obj)1842void iommufd_selftest_destroy(struct iommufd_object *obj)
1601{1843{
1602 struct selftest_obj *sobj = to_selftest_obj(obj);1844 struct selftest_obj *sobj = to_selftest_obj(obj);
1603 1845
1604 switch (sobj->type) {1846 switch (sobj->type) {
1605 case TYPE_IDEV:1847 case TYPE_IDEV:
1606 iommufd_device_detach(sobj->idev.idev);1848 iommufd_device_detach(sobj->idev.idev, IOMMU_NO_PASID);
1607 iommufd_device_unbind(sobj->idev.idev);1849 iommufd_device_unbind(sobj->idev.idev);
1608 mock_dev_destroy(sobj->idev.mock_dev);1850 mock_dev_destroy(sobj->idev.mock_dev);
1609 break;1851 break;
@@ -1678,6 +1920,16 @@ int iommufd_test(struct iommufd_ucmd *ucmd) · 68 unmodified lines
1678 cmd->dirty.flags);1920 cmd->dirty.flags);
1679 case IOMMU_TEST_OP_TRIGGER_IOPF:1921 case IOMMU_TEST_OP_TRIGGER_IOPF:
1680 return iommufd_test_trigger_iopf(ucmd, cmd);1922 return iommufd_test_trigger_iopf(ucmd, cmd);
1923 case IOMMU_TEST_OP_TRIGGER_VEVENT:
1924 return iommufd_test_trigger_vevent(ucmd, cmd);
1925 case IOMMU_TEST_OP_PASID_ATTACH:
1926 return iommufd_test_pasid_attach(ucmd, cmd);
1927 case IOMMU_TEST_OP_PASID_REPLACE:
1928 return iommufd_test_pasid_replace(ucmd, cmd);
1929 case IOMMU_TEST_OP_PASID_DETACH:
1930 return iommufd_test_pasid_detach(ucmd, cmd);
1931 case IOMMU_TEST_OP_PASID_CHECK_HWPT:
1932 return iommufd_test_pasid_check_hwpt(ucmd, cmd);
1681 default:1933 default:
1682 return -EOPNOTSUPP;1934 return -EOPNOTSUPP;
1683 }1935 }
@@ -1724,6 +1976,7 @@ int __init iommufd_test_init(void) · 40 unmodified lines
1724 init_completion(&mock_iommu.complete);1976 init_completion(&mock_iommu.complete);
1725 1977
1726 mock_iommu_iopf_queue = iopf_queue_alloc("mock-iopfq");1978 mock_iommu_iopf_queue = iopf_queue_alloc("mock-iopfq");
1979 mock_iommu.iommu_dev.max_pasids = (1 << MOCK_PASID_WIDTH);
1727 1980
1728 return 0;1981 return 0;
1729 1982
drivers/iommu/iommufd/viommu.c+2−0
@@ -59,6 +59,8 @@ int iommufd_viommu_alloc_ioctl(struct iommufd_ucmd *ucmd) · 58 unmodified lines
59 viommu->ictx = ucmd->ictx;59 viommu->ictx = ucmd->ictx;
60 viommu->hwpt = hwpt_paging;60 viommu->hwpt = hwpt_paging;
61 refcount_inc(&viommu->hwpt->common.obj.users);61 refcount_inc(&viommu->hwpt->common.obj.users);
62 INIT_LIST_HEAD(&viommu->veventqs);
63 init_rwsem(&viommu->veventqs_rwsem);
62 /*64 /*
63 * It is the most likely case that a physical IOMMU is unpluggable. A65 * It is the most likely case that a physical IOMMU is unpluggable. A
64 * pluggable IOMMU instance (if exists) is responsible for refcounting66 * pluggable IOMMU instance (if exists) is responsible for refcounting
drivers/pci/ats.c+33−0
@@ -538,4 +538,37 @@ int pci_max_pasids(struct pci_dev *pdev) · 537 unmodified lines
538 return (1 << FIELD_GET(PCI_PASID_CAP_WIDTH, supported));538 return (1 << FIELD_GET(PCI_PASID_CAP_WIDTH, supported));
539}539}
540EXPORT_SYMBOL_GPL(pci_max_pasids);540EXPORT_SYMBOL_GPL(pci_max_pasids);
541
542/**
543 * pci_pasid_status - Check the PASID status
544 * @pdev: PCI device structure
545 *
546 * Returns a negative value when no PASID capability is present.
547 * Otherwise the value of the control register is returned.
548 * Status reported are:
549 *
550 * PCI_PASID_CTRL_ENABLE - PASID enabled
551 * PCI_PASID_CTRL_EXEC - Execute permission enabled
552 * PCI_PASID_CTRL_PRIV - Privileged mode enabled
553 */
554int pci_pasid_status(struct pci_dev *pdev)
555{
556 int pasid;
557 u16 ctrl;
558
559 if (pdev->is_virtfn)
560 pdev = pci_physfn(pdev);
561
562 pasid = pdev->pasid_cap;
563 if (!pasid)
564 return -EINVAL;
565
566 pci_read_config_word(pdev, pasid + PCI_PASID_CTRL, &ctrl);
567
568 ctrl &= PCI_PASID_CTRL_ENABLE | PCI_PASID_CTRL_EXEC |
569 PCI_PASID_CTRL_PRIV;
570
571 return ctrl;
572}
573EXPORT_SYMBOL_GPL(pci_pasid_status);
541#endif /* CONFIG_PCI_PASID */574#endif /* CONFIG_PCI_PASID */
drivers/vfio/device_cdev.c+52−8
@@ -162,9 +162,9 @@ void vfio_df_unbind_iommufd(struct vfio_device_file *df) · 161 unmodified lines
162int vfio_df_ioctl_attach_pt(struct vfio_device_file *df,162int vfio_df_ioctl_attach_pt(struct vfio_device_file *df,
163 struct vfio_device_attach_iommufd_pt __user *arg)163 struct vfio_device_attach_iommufd_pt __user *arg)
164{164{
165 struct vfio_device *device = df->device;
166 struct vfio_device_attach_iommufd_pt attach;165 struct vfio_device_attach_iommufd_pt attach;
167 unsigned long minsz;166 struct vfio_device *device = df->device;
167 unsigned long minsz, xend = 0;
168 int ret;168 int ret;
169 169
170 minsz = offsetofend(struct vfio_device_attach_iommufd_pt, pt_id);170 minsz = offsetofend(struct vfio_device_attach_iommufd_pt, pt_id);
@@ -172,11 +172,34 @@ int vfio_df_ioctl_attach_pt(struct vfio_device_file *df, · 1 unmodified line
172 if (copy_from_user(&attach, arg, minsz))172 if (copy_from_user(&attach, arg, minsz))
173 return -EFAULT;173 return -EFAULT;
174 174
175 if (attach.argsz < minsz || attach.flags)175 if (attach.argsz < minsz)
176 return -EINVAL;176 return -EINVAL;
177 177
178 if (attach.flags & ~VFIO_DEVICE_ATTACH_PASID)
179 return -EINVAL;
180
181 if (attach.flags & VFIO_DEVICE_ATTACH_PASID) {
182 if (!device->ops->pasid_attach_ioas)
183 return -EOPNOTSUPP;
184 xend = offsetofend(struct vfio_device_attach_iommufd_pt, pasid);
185 }
186
187 if (xend) {
188 if (attach.argsz < xend)
189 return -EINVAL;
190
191 if (copy_from_user((void *)&attach + minsz,
192 (void __user *)arg + minsz, xend - minsz))
193 return -EFAULT;
194 }
195
178 mutex_lock(&device->dev_set->lock);196 mutex_lock(&device->dev_set->lock);
179 ret = device->ops->attach_ioas(device, &attach.pt_id);197 if (attach.flags & VFIO_DEVICE_ATTACH_PASID)
198 ret = device->ops->pasid_attach_ioas(device,
199 attach.pasid,
200 &attach.pt_id);
201 else
202 ret = device->ops->attach_ioas(device, &attach.pt_id);
180 if (ret)203 if (ret)
181 goto out_unlock;204 goto out_unlock;
182 205
@@ -198,20 +221,41 @@ out_unlock: · 15 unmodified lines
198int vfio_df_ioctl_detach_pt(struct vfio_device_file *df,221int vfio_df_ioctl_detach_pt(struct vfio_device_file *df,
199 struct vfio_device_detach_iommufd_pt __user *arg)222 struct vfio_device_detach_iommufd_pt __user *arg)
200{223{
201 struct vfio_device *device = df->device;
202 struct vfio_device_detach_iommufd_pt detach;224 struct vfio_device_detach_iommufd_pt detach;
203 unsigned long minsz;225 struct vfio_device *device = df->device;
226 unsigned long minsz, xend = 0;
204 227
205 minsz = offsetofend(struct vfio_device_detach_iommufd_pt, flags);228 minsz = offsetofend(struct vfio_device_detach_iommufd_pt, flags);
206 229
207 if (copy_from_user(&detach, arg, minsz))230 if (copy_from_user(&detach, arg, minsz))
208 return -EFAULT;231 return -EFAULT;
209 232
210 if (detach.argsz < minsz || detach.flags)233 if (detach.argsz < minsz)
211 return -EINVAL;234 return -EINVAL;
212 235
236 if (detach.flags & ~VFIO_DEVICE_DETACH_PASID)
237 return -EINVAL;
238
239 if (detach.flags & VFIO_DEVICE_DETACH_PASID) {
240 if (!device->ops->pasid_detach_ioas)
241 return -EOPNOTSUPP;
242 xend = offsetofend(struct vfio_device_detach_iommufd_pt, pasid);
243 }
244
245 if (xend) {
246 if (detach.argsz < xend)
247 return -EINVAL;
248
249 if (copy_from_user((void *)&detach + minsz,
250 (void __user *)arg + minsz, xend - minsz))
251 return -EFAULT;
252 }
253
213 mutex_lock(&device->dev_set->lock);254 mutex_lock(&device->dev_set->lock);
214 device->ops->detach_ioas(device);255 if (detach.flags & VFIO_DEVICE_DETACH_PASID)
256 device->ops->pasid_detach_ioas(device, detach.pasid);
257 else
258 device->ops->detach_ioas(device);
215 mutex_unlock(&device->dev_set->lock);259 mutex_unlock(&device->dev_set->lock);
216 260
217 return 0;261 return 0;
drivers/vfio/iommufd.c+56−4
@@ -119,16 +119,24 @@ int vfio_iommufd_physical_bind(struct vfio_device *vdev, · 118 unmodified lines
119 if (IS_ERR(idev))119 if (IS_ERR(idev))
120 return PTR_ERR(idev);120 return PTR_ERR(idev);
121 vdev->iommufd_device = idev;121 vdev->iommufd_device = idev;
122 ida_init(&vdev->pasids);
122 return 0;123 return 0;
123}124}
124EXPORT_SYMBOL_GPL(vfio_iommufd_physical_bind);125EXPORT_SYMBOL_GPL(vfio_iommufd_physical_bind);
125 126
126void vfio_iommufd_physical_unbind(struct vfio_device *vdev)127void vfio_iommufd_physical_unbind(struct vfio_device *vdev)
127{128{
129 int pasid;
130
128 lockdep_assert_held(&vdev->dev_set->lock);131 lockdep_assert_held(&vdev->dev_set->lock);
129 132
133 while ((pasid = ida_find_first(&vdev->pasids)) >= 0) {
134 iommufd_device_detach(vdev->iommufd_device, pasid);
135 ida_free(&vdev->pasids, pasid);
136 }
137
130 if (vdev->iommufd_attached) {138 if (vdev->iommufd_attached) {
131 iommufd_device_detach(vdev->iommufd_device);139 iommufd_device_detach(vdev->iommufd_device, IOMMU_NO_PASID);
132 vdev->iommufd_attached = false;140 vdev->iommufd_attached = false;
133 }141 }
134 iommufd_device_unbind(vdev->iommufd_device);142 iommufd_device_unbind(vdev->iommufd_device);
@@ -146,9 +154,11 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id) · 11 unmodified lines
146 return -EINVAL;154 return -EINVAL;
147 155
148 if (vdev->iommufd_attached)156 if (vdev->iommufd_attached)
149 rc = iommufd_device_replace(vdev->iommufd_device, pt_id);157 rc = iommufd_device_replace(vdev->iommufd_device,
158 IOMMU_NO_PASID, pt_id);
150 else159 else
151 rc = iommufd_device_attach(vdev->iommufd_device, pt_id);160 rc = iommufd_device_attach(vdev->iommufd_device,
161 IOMMU_NO_PASID, pt_id);
152 if (rc)162 if (rc)
153 return rc;163 return rc;
154 vdev->iommufd_attached = true;164 vdev->iommufd_attached = true;
@@ -163,11 +173,53 @@ void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev) · 8 unmodified lines
163 if (WARN_ON(!vdev->iommufd_device) || !vdev->iommufd_attached)173 if (WARN_ON(!vdev->iommufd_device) || !vdev->iommufd_attached)
164 return;174 return;
165 175
166 iommufd_device_detach(vdev->iommufd_device);176 iommufd_device_detach(vdev->iommufd_device, IOMMU_NO_PASID);
167 vdev->iommufd_attached = false;177 vdev->iommufd_attached = false;
168}178}
169EXPORT_SYMBOL_GPL(vfio_iommufd_physical_detach_ioas);179EXPORT_SYMBOL_GPL(vfio_iommufd_physical_detach_ioas);
170 180
181int vfio_iommufd_physical_pasid_attach_ioas(struct vfio_device *vdev,
182 u32 pasid, u32 *pt_id)
183{
184 int rc;
185
186 lockdep_assert_held(&vdev->dev_set->lock);
187
188 if (WARN_ON(!vdev->iommufd_device))
189 return -EINVAL;
190
191 if (ida_exists(&vdev->pasids, pasid))
192 return iommufd_device_replace(vdev->iommufd_device,
193 pasid, pt_id);
194
195 rc = ida_alloc_range(&vdev->pasids, pasid, pasid, GFP_KERNEL);
196 if (rc < 0)
197 return rc;
198
199 rc = iommufd_device_attach(vdev->iommufd_device, pasid, pt_id);
200 if (rc)
201 ida_free(&vdev->pasids, pasid);
202
203 return rc;
204}
205EXPORT_SYMBOL_GPL(vfio_iommufd_physical_pasid_attach_ioas);
206
207void vfio_iommufd_physical_pasid_detach_ioas(struct vfio_device *vdev,
208 u32 pasid)
209{
210 lockdep_assert_held(&vdev->dev_set->lock);
211
212 if (WARN_ON(!vdev->iommufd_device))
213 return;
214
215 if (!ida_exists(&vdev->pasids, pasid))
216 return;
217
218 iommufd_device_detach(vdev->iommufd_device, pasid);
219 ida_free(&vdev->pasids, pasid);
220}
221EXPORT_SYMBOL_GPL(vfio_iommufd_physical_pasid_detach_ioas);
222
171/*223/*
172 * The emulated standard ops mean that vfio_device is going to use the224 * The emulated standard ops mean that vfio_device is going to use the
173 * "mdev path" and will call vfio_pin_pages()/vfio_dma_rw(). Drivers using this225 * "mdev path" and will call vfio_pin_pages()/vfio_dma_rw(). Drivers using this
drivers/vfio/pci/vfio_pci.c+2−0
@@ -144,6 +144,8 @@ static const struct vfio_device_ops vfio_pci_ops = { · 143 unmodified lines
144 .unbind_iommufd = vfio_iommufd_physical_unbind,144 .unbind_iommufd = vfio_iommufd_physical_unbind,
145 .attach_ioas = vfio_iommufd_physical_attach_ioas,145 .attach_ioas = vfio_iommufd_physical_attach_ioas,
146 .detach_ioas = vfio_iommufd_physical_detach_ioas,146 .detach_ioas = vfio_iommufd_physical_detach_ioas,
147 .pasid_attach_ioas = vfio_iommufd_physical_pasid_attach_ioas,
148 .pasid_detach_ioas = vfio_iommufd_physical_pasid_detach_ioas,
147};149};
148 150
149static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)151static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
include/linux/idr.h+11−0
@@ -274,6 +274,7 @@ struct ida { · 273 unmodified lines
274int ida_alloc_range(struct ida *, unsigned int min, unsigned int max, gfp_t);274int ida_alloc_range(struct ida *, unsigned int min, unsigned int max, gfp_t);
275void ida_free(struct ida *, unsigned int id);275void ida_free(struct ida *, unsigned int id);
276void ida_destroy(struct ida *ida);276void ida_destroy(struct ida *ida);
277int ida_find_first_range(struct ida *ida, unsigned int min, unsigned int max);
277 278
278/**279/**
279 * ida_alloc() - Allocate an unused ID.280 * ida_alloc() - Allocate an unused ID.
@@ -345,4 +346,14 @@ static inline bool ida_is_empty(const struct ida *ida) · 65 unmodified lines
345{346{
346 return xa_empty(&ida->xa);347 return xa_empty(&ida->xa);
347}348}
349
350static inline bool ida_exists(struct ida *ida, unsigned int id)
351{
352 return ida_find_first_range(ida, id, id) == id;
353}
354
355static inline int ida_find_first(struct ida *ida)
356{
357 return ida_find_first_range(ida, 0, ~0);
358}
348#endif /* __IDR_H__ */359#endif /* __IDR_H__ */
include/linux/iommu.h+15−20
@@ -41,6 +41,7 @@ struct iommu_dirty_ops; · 40 unmodified lines
41struct notifier_block;41struct notifier_block;
42struct iommu_sva;42struct iommu_sva;
43struct iommu_dma_cookie;43struct iommu_dma_cookie;
44struct iommu_dma_msi_cookie;
44struct iommu_fault_param;45struct iommu_fault_param;
45struct iommufd_ctx;46struct iommufd_ctx;
46struct iommufd_viommu;47struct iommufd_viommu;
@@ -165,6 +166,15 @@ struct iommu_domain_geometry { · 118 unmodified lines
165 bool force_aperture; /* DMA only allowed in mappable range? */166 bool force_aperture; /* DMA only allowed in mappable range? */
166};167};
167 168
169enum iommu_domain_cookie_type {
170 IOMMU_COOKIE_NONE,
171 IOMMU_COOKIE_DMA_IOVA,
172 IOMMU_COOKIE_DMA_MSI,
173 IOMMU_COOKIE_FAULT_HANDLER,
174 IOMMU_COOKIE_SVA,
175 IOMMU_COOKIE_IOMMUFD,
176};
177
168/* Domain feature flags */178/* Domain feature flags */
169#define __IOMMU_DOMAIN_PAGING (1U << 0) /* Support for iommu_map/unmap */179#define __IOMMU_DOMAIN_PAGING (1U << 0) /* Support for iommu_map/unmap */
170#define __IOMMU_DOMAIN_DMA_API (1U << 1) /* Domain for use in DMA-API180#define __IOMMU_DOMAIN_DMA_API (1U << 1) /* Domain for use in DMA-API
@@ -211,23 +221,18 @@ struct iommu_domain_geometry { · 40 unmodified lines
211 221
212struct iommu_domain {222struct iommu_domain {
213 unsigned type;223 unsigned type;
224 enum iommu_domain_cookie_type cookie_type;
214 const struct iommu_domain_ops *ops;225 const struct iommu_domain_ops *ops;
215 const struct iommu_dirty_ops *dirty_ops;226 const struct iommu_dirty_ops *dirty_ops;
216 const struct iommu_ops *owner; /* Whose domain_alloc we came from */227 const struct iommu_ops *owner; /* Whose domain_alloc we came from */
217 unsigned long pgsize_bitmap; /* Bitmap of page sizes in use */228 unsigned long pgsize_bitmap; /* Bitmap of page sizes in use */
218 struct iommu_domain_geometry geometry;229 struct iommu_domain_geometry geometry;
219 struct iommu_dma_cookie *iova_cookie;
220 int (*iopf_handler)(struct iopf_group *group);230 int (*iopf_handler)(struct iopf_group *group);
221 231
222#if IS_ENABLED(CONFIG_IRQ_MSI_IOMMU)232 union { /* cookie */
223 int (*sw_msi)(struct iommu_domain *domain, struct msi_desc *desc,233 struct iommu_dma_cookie *iova_cookie;
224 phys_addr_t msi_addr);234 struct iommu_dma_msi_cookie *msi_cookie;
225#endif235 struct iommufd_hw_pagetable *iommufd_hwpt;
226
227 union { /* Pointer usable by owner of the domain */
228 struct iommufd_hw_pagetable *iommufd_hwpt; /* iommufd */
229 };
230 union { /* Fault handler */
231 struct {236 struct {
232 iommu_fault_handler_t handler;237 iommu_fault_handler_t handler;
233 void *handler_token;238 void *handler_token;
@@ -244,16 +249,6 @@ struct iommu_domain { · 10 unmodified lines
244 };249 };
245};250};
246 251
247static inline void iommu_domain_set_sw_msi(
248 struct iommu_domain *domain,
249 int (*sw_msi)(struct iommu_domain *domain, struct msi_desc *desc,
250 phys_addr_t msi_addr))
251{
252#if IS_ENABLED(CONFIG_IRQ_MSI_IOMMU)
253 domain->sw_msi = sw_msi;
254#endif
255}
256
257static inline bool iommu_is_dma_domain(struct iommu_domain *domain)252static inline bool iommu_is_dma_domain(struct iommu_domain *domain)
258{253{
259 return domain->type & __IOMMU_DOMAIN_DMA_API;254 return domain->type & __IOMMU_DOMAIN_DMA_API;
include/linux/iommufd.h+29−3
@@ -8,9 +8,11 @@ · 7 unmodified lines
8 8
9#include <linux/err.h>9#include <linux/err.h>
10#include <linux/errno.h>10#include <linux/errno.h>
11#include <linux/iommu.h>
11#include <linux/refcount.h>12#include <linux/refcount.h>
12#include <linux/types.h>13#include <linux/types.h>
13#include <linux/xarray.h>14#include <linux/xarray.h>
15#include <uapi/linux/iommufd.h>
14 16
15struct device;17struct device;
16struct file;18struct file;
@@ -34,6 +36,7 @@ enum iommufd_object_type { · 17 unmodified lines
34 IOMMUFD_OBJ_FAULT,36 IOMMUFD_OBJ_FAULT,
35 IOMMUFD_OBJ_VIOMMU,37 IOMMUFD_OBJ_VIOMMU,
36 IOMMUFD_OBJ_VDEVICE,38 IOMMUFD_OBJ_VDEVICE,
39 IOMMUFD_OBJ_VEVENTQ,
37#ifdef CONFIG_IOMMUFD_TEST40#ifdef CONFIG_IOMMUFD_TEST
38 IOMMUFD_OBJ_SELFTEST,41 IOMMUFD_OBJ_SELFTEST,
39#endif42#endif
@@ -52,9 +55,11 @@ struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, · 12 unmodified lines
52 struct device *dev, u32 *id);55 struct device *dev, u32 *id);
53void iommufd_device_unbind(struct iommufd_device *idev);56void iommufd_device_unbind(struct iommufd_device *idev);
54 57
55int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id);58int iommufd_device_attach(struct iommufd_device *idev, ioasid_t pasid,
56int iommufd_device_replace(struct iommufd_device *idev, u32 *pt_id);59 u32 *pt_id);
57void iommufd_device_detach(struct iommufd_device *idev);60int iommufd_device_replace(struct iommufd_device *idev, ioasid_t pasid,
61 u32 *pt_id);
62void iommufd_device_detach(struct iommufd_device *idev, ioasid_t pasid);
58 63
59struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev);64struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev);
60u32 iommufd_device_to_id(struct iommufd_device *idev);65u32 iommufd_device_to_id(struct iommufd_device *idev);
@@ -93,6 +98,8 @@ struct iommufd_viommu { · 32 unmodified lines
93 const struct iommufd_viommu_ops *ops;98 const struct iommufd_viommu_ops *ops;
94 99
95 struct xarray vdevs;100 struct xarray vdevs;
101 struct list_head veventqs;
102 struct rw_semaphore veventqs_rwsem;
96 103
97 unsigned int type;104 unsigned int type;
98};105};
@@ -187,6 +194,11 @@ struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx, · 88 unmodified lines
187 enum iommufd_object_type type);194 enum iommufd_object_type type);
188struct device *iommufd_viommu_find_dev(struct iommufd_viommu *viommu,195struct device *iommufd_viommu_find_dev(struct iommufd_viommu *viommu,
189 unsigned long vdev_id);196 unsigned long vdev_id);
197int iommufd_viommu_get_vdev_id(struct iommufd_viommu *viommu,
198 struct device *dev, unsigned long *vdev_id);
199int iommufd_viommu_report_event(struct iommufd_viommu *viommu,
200 enum iommu_veventq_type type, void *event_data,
201 size_t data_len);
190#else /* !CONFIG_IOMMUFD_DRIVER_CORE */202#else /* !CONFIG_IOMMUFD_DRIVER_CORE */
191static inline struct iommufd_object *203static inline struct iommufd_object *
192_iommufd_object_alloc(struct iommufd_ctx *ictx, size_t size,204_iommufd_object_alloc(struct iommufd_ctx *ictx, size_t size,
@@ -200,6 +212,20 @@ iommufd_viommu_find_dev(struct iommufd_viommu *viommu, unsigned long vdev_id) · 7 unmodified lines
200{212{
201 return NULL;213 return NULL;
202}214}
215
216static inline int iommufd_viommu_get_vdev_id(struct iommufd_viommu *viommu,
217 struct device *dev,
218 unsigned long *vdev_id)
219{
220 return -ENOENT;
221}
222
223static inline int iommufd_viommu_report_event(struct iommufd_viommu *viommu,
224 enum iommu_veventq_type type,
225 void *event_data, size_t data_len)
226{
227 return -EOPNOTSUPP;
228}
203#endif /* CONFIG_IOMMUFD_DRIVER_CORE */229#endif /* CONFIG_IOMMUFD_DRIVER_CORE */
204 230
205/*231/*
include/linux/pci-ats.h+3−0
@@ -42,6 +42,7 @@ int pci_enable_pasid(struct pci_dev *pdev, int features); · 41 unmodified lines
42void pci_disable_pasid(struct pci_dev *pdev);42void pci_disable_pasid(struct pci_dev *pdev);
43int pci_pasid_features(struct pci_dev *pdev);43int pci_pasid_features(struct pci_dev *pdev);
44int pci_max_pasids(struct pci_dev *pdev);44int pci_max_pasids(struct pci_dev *pdev);
45int pci_pasid_status(struct pci_dev *pdev);
45#else /* CONFIG_PCI_PASID */46#else /* CONFIG_PCI_PASID */
46static inline int pci_enable_pasid(struct pci_dev *pdev, int features)47static inline int pci_enable_pasid(struct pci_dev *pdev, int features)
47{ return -EINVAL; }48{ return -EINVAL; }
@@ -50,6 +51,8 @@ static inline int pci_pasid_features(struct pci_dev *pdev) · 2 unmodified lines
50{ return -EINVAL; }51{ return -EINVAL; }
51static inline int pci_max_pasids(struct pci_dev *pdev)52static inline int pci_max_pasids(struct pci_dev *pdev)
52{ return -EINVAL; }53{ return -EINVAL; }
54static inline int pci_pasid_status(struct pci_dev *pdev)
55{ return -EINVAL; }
53#endif /* CONFIG_PCI_PASID */56#endif /* CONFIG_PCI_PASID */
54 57
55#endif /* LINUX_PCI_ATS_H */58#endif /* LINUX_PCI_ATS_H */
include/linux/vfio.h+14−0
@@ -67,6 +67,7 @@ struct vfio_device { · 66 unmodified lines
67 struct inode *inode;67 struct inode *inode;
68#if IS_ENABLED(CONFIG_IOMMUFD)68#if IS_ENABLED(CONFIG_IOMMUFD)
69 struct iommufd_device *iommufd_device;69 struct iommufd_device *iommufd_device;
70 struct ida pasids;
70 u8 iommufd_attached:1;71 u8 iommufd_attached:1;
71#endif72#endif
72 u8 cdev_opened:1;73 u8 cdev_opened:1;
@@ -91,6 +92,8 @@ struct vfio_device { · 18 unmodified lines
91 * bound iommufd. Undo in unbind_iommufd if @detach_ioas is not92 * bound iommufd. Undo in unbind_iommufd if @detach_ioas is not
92 * called.93 * called.
93 * @detach_ioas: Opposite of attach_ioas94 * @detach_ioas: Opposite of attach_ioas
95 * @pasid_attach_ioas: The pasid variation of attach_ioas
96 * @pasid_detach_ioas: Opposite of pasid_attach_ioas
94 * @open_device: Called when the first file descriptor is opened for this device97 * @open_device: Called when the first file descriptor is opened for this device
95 * @close_device: Opposite of open_device98 * @close_device: Opposite of open_device
96 * @read: Perform read(2) on device file descriptor99 * @read: Perform read(2) on device file descriptor
@@ -115,6 +118,9 @@ struct vfio_device_ops { · 18 unmodified lines
115 void (*unbind_iommufd)(struct vfio_device *vdev);118 void (*unbind_iommufd)(struct vfio_device *vdev);
116 int (*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);119 int (*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);
117 void (*detach_ioas)(struct vfio_device *vdev);120 void (*detach_ioas)(struct vfio_device *vdev);
121 int (*pasid_attach_ioas)(struct vfio_device *vdev, u32 pasid,
122 u32 *pt_id);
123 void (*pasid_detach_ioas)(struct vfio_device *vdev, u32 pasid);
118 int (*open_device)(struct vfio_device *vdev);124 int (*open_device)(struct vfio_device *vdev);
119 void (*close_device)(struct vfio_device *vdev);125 void (*close_device)(struct vfio_device *vdev);
120 ssize_t (*read)(struct vfio_device *vdev, char __user *buf,126 ssize_t (*read)(struct vfio_device *vdev, char __user *buf,
@@ -139,6 +145,10 @@ int vfio_iommufd_physical_bind(struct vfio_device *vdev, · 18 unmodified lines
139void vfio_iommufd_physical_unbind(struct vfio_device *vdev);145void vfio_iommufd_physical_unbind(struct vfio_device *vdev);
140int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id);146int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
141void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev);147void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev);
148int vfio_iommufd_physical_pasid_attach_ioas(struct vfio_device *vdev,
149 u32 pasid, u32 *pt_id);
150void vfio_iommufd_physical_pasid_detach_ioas(struct vfio_device *vdev,
151 u32 pasid);
142int vfio_iommufd_emulated_bind(struct vfio_device *vdev,152int vfio_iommufd_emulated_bind(struct vfio_device *vdev,
143 struct iommufd_ctx *ictx, u32 *out_device_id);153 struct iommufd_ctx *ictx, u32 *out_device_id);
144void vfio_iommufd_emulated_unbind(struct vfio_device *vdev);154void vfio_iommufd_emulated_unbind(struct vfio_device *vdev);
@@ -166,6 +176,10 @@ vfio_iommufd_get_dev_id(struct vfio_device *vdev, struct iommufd_ctx *ictx) · 21 unmodified lines
166 ((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)176 ((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
167#define vfio_iommufd_physical_detach_ioas \177#define vfio_iommufd_physical_detach_ioas \
168 ((void (*)(struct vfio_device *vdev)) NULL)178 ((void (*)(struct vfio_device *vdev)) NULL)
179#define vfio_iommufd_physical_pasid_attach_ioas \
180 ((int (*)(struct vfio_device *vdev, u32 pasid, u32 *pt_id)) NULL)
181#define vfio_iommufd_physical_pasid_detach_ioas \
182 ((void (*)(struct vfio_device *vdev, u32 pasid)) NULL)
169#define vfio_iommufd_emulated_bind \183#define vfio_iommufd_emulated_bind \
170 ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \184 ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \
171 u32 *out_device_id)) NULL)185 u32 *out_device_id)) NULL)
include/uapi/linux/iommufd.h+128−1
@@ -55,6 +55,7 @@ enum { · 54 unmodified lines
55 IOMMUFD_CMD_VIOMMU_ALLOC = 0x90,55 IOMMUFD_CMD_VIOMMU_ALLOC = 0x90,
56 IOMMUFD_CMD_VDEVICE_ALLOC = 0x91,56 IOMMUFD_CMD_VDEVICE_ALLOC = 0x91,
57 IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92,57 IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92,
58 IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93,
58};59};
59 60
60/**61/**
@@ -392,6 +393,9 @@ struct iommu_vfio_ioas { · 331 unmodified lines
392 * Any domain attached to the non-PASID part of the393 * Any domain attached to the non-PASID part of the
393 * device must also be flagged, otherwise attaching a394 * device must also be flagged, otherwise attaching a
394 * PASID will blocked.395 * PASID will blocked.
396 * For the user that wants to attach PASID, ioas is
397 * not recommended for both the non-PASID part
398 * and PASID part of the device.
395 * If IOMMU does not support PASID it will return399 * If IOMMU does not support PASID it will return
396 * error (-EOPNOTSUPP).400 * error (-EOPNOTSUPP).
397 */401 */
@@ -608,9 +612,17 @@ enum iommu_hw_info_type { · 210 unmodified lines
608 * IOMMU_HWPT_GET_DIRTY_BITMAP612 * IOMMU_HWPT_GET_DIRTY_BITMAP
609 * IOMMU_HWPT_SET_DIRTY_TRACKING613 * IOMMU_HWPT_SET_DIRTY_TRACKING
610 *614 *
615 * @IOMMU_HW_CAP_PCI_PASID_EXEC: Execute Permission Supported, user ignores it
616 * when the struct
617 * iommu_hw_info::out_max_pasid_log2 is zero.
618 * @IOMMU_HW_CAP_PCI_PASID_PRIV: Privileged Mode Supported, user ignores it
619 * when the struct
620 * iommu_hw_info::out_max_pasid_log2 is zero.
611 */621 */
612enum iommufd_hw_capabilities {622enum iommufd_hw_capabilities {
613 IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0,623 IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0,
624 IOMMU_HW_CAP_PCI_PASID_EXEC = 1 << 1,
625 IOMMU_HW_CAP_PCI_PASID_PRIV = 1 << 2,
614};626};
615 627
616/**628/**
@@ -626,6 +638,9 @@ enum iommufd_hw_capabilities { · 9 unmodified lines
626 * iommu_hw_info_type.638 * iommu_hw_info_type.
627 * @out_capabilities: Output the generic iommu capability info type as defined639 * @out_capabilities: Output the generic iommu capability info type as defined
628 * in the enum iommu_hw_capabilities.640 * in the enum iommu_hw_capabilities.
641 * @out_max_pasid_log2: Output the width of PASIDs. 0 means no PASID support.
642 * PCI devices turn to out_capabilities to check if the
643 * specific capabilities is supported or not.
629 * @__reserved: Must be 0644 * @__reserved: Must be 0
630 *645 *
631 * Query an iommu type specific hardware information data from an iommu behind646 * Query an iommu type specific hardware information data from an iommu behind
@@ -649,7 +664,8 @@ struct iommu_hw_info { · 17 unmodified lines
649 __u32 data_len;664 __u32 data_len;
650 __aligned_u64 data_uptr;665 __aligned_u64 data_uptr;
651 __u32 out_data_type;666 __u32 out_data_type;
652 __u32 __reserved;667 __u8 out_max_pasid_log2;
668 __u8 __reserved[3];
653 __aligned_u64 out_capabilities;669 __aligned_u64 out_capabilities;
654};670};
655#define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO)671#define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO)
@@ -1014,4 +1030,115 @@ struct iommu_ioas_change_process { · 358 unmodified lines
1014#define IOMMU_IOAS_CHANGE_PROCESS \1030#define IOMMU_IOAS_CHANGE_PROCESS \
1015 _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS)1031 _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS)
1016 1032
1033/**
1034 * enum iommu_veventq_flag - flag for struct iommufd_vevent_header
1035 * @IOMMU_VEVENTQ_FLAG_LOST_EVENTS: vEVENTQ has lost vEVENTs
1036 */
1037enum iommu_veventq_flag {
1038 IOMMU_VEVENTQ_FLAG_LOST_EVENTS = (1U << 0),
1039};
1040
1041/**
1042 * struct iommufd_vevent_header - Virtual Event Header for a vEVENTQ Status
1043 * @flags: Combination of enum iommu_veventq_flag
1044 * @sequence: The sequence index of a vEVENT in the vEVENTQ, with a range of
1045 * [0, INT_MAX] where the following index of INT_MAX is 0
1046 *
1047 * Each iommufd_vevent_header reports a sequence index of the following vEVENT:
1048 *
1049 * +----------------------+-------+----------------------+-------+---+-------+
1050 * | header0 {sequence=0} | data0 | header1 {sequence=1} | data1 |...| dataN |
1051 * +----------------------+-------+----------------------+-------+---+-------+
1052 *
1053 * And this sequence index is expected to be monotonic to the sequence index of
1054 * the previous vEVENT. If two adjacent sequence indexes has a delta larger than
1055 * 1, it means that delta - 1 number of vEVENTs has lost, e.g. two lost vEVENTs:
1056 *
1057 * +-----+----------------------+-------+----------------------+-------+-----+
1058 * | ... | header3 {sequence=3} | data3 | header6 {sequence=6} | data6 | ... |
1059 * +-----+----------------------+-------+----------------------+-------+-----+
1060 *
1061 * If a vEVENT lost at the tail of the vEVENTQ and there is no following vEVENT
1062 * providing the next sequence index, an IOMMU_VEVENTQ_FLAG_LOST_EVENTS header
1063 * would be added to the tail, and no data would follow this header:
1064 *
1065 * +--+----------------------+-------+-----------------------------------------+
1066 * |..| header3 {sequence=3} | data3 | header4 {flags=LOST_EVENTS, sequence=4} |
1067 * +--+----------------------+-------+-----------------------------------------+
1068 */
1069struct iommufd_vevent_header {
1070 __u32 flags;
1071 __u32 sequence;
1072};
1073
1074/**
1075 * enum iommu_veventq_type - Virtual Event Queue Type
1076 * @IOMMU_VEVENTQ_TYPE_DEFAULT: Reserved for future use
1077 * @IOMMU_VEVENTQ_TYPE_ARM_SMMUV3: ARM SMMUv3 Virtual Event Queue
1078 */
1079enum iommu_veventq_type {
1080 IOMMU_VEVENTQ_TYPE_DEFAULT = 0,
1081 IOMMU_VEVENTQ_TYPE_ARM_SMMUV3 = 1,
1082};
1083
1084/**
1085 * struct iommu_vevent_arm_smmuv3 - ARM SMMUv3 Virtual Event
1086 * (IOMMU_VEVENTQ_TYPE_ARM_SMMUV3)
1087 * @evt: 256-bit ARM SMMUv3 Event record, little-endian.
1088 * Reported event records: (Refer to "7.3 Event records" in SMMUv3 HW Spec)
1089 * - 0x04 C_BAD_STE
1090 * - 0x06 F_STREAM_DISABLED
1091 * - 0x08 C_BAD_SUBSTREAMID
1092 * - 0x0a C_BAD_CD
1093 * - 0x10 F_TRANSLATION
1094 * - 0x11 F_ADDR_SIZE
1095 * - 0x12 F_ACCESS
1096 * - 0x13 F_PERMISSION
1097 *
1098 * StreamID field reports a virtual device ID. To receive a virtual event for a
1099 * device, a vDEVICE must be allocated via IOMMU_VDEVICE_ALLOC.
1100 */
1101struct iommu_vevent_arm_smmuv3 {
1102 __aligned_le64 evt[4];
1103};
1104
1105/**
1106 * struct iommu_veventq_alloc - ioctl(IOMMU_VEVENTQ_ALLOC)
1107 * @size: sizeof(struct iommu_veventq_alloc)
1108 * @flags: Must be 0
1109 * @viommu_id: virtual IOMMU ID to associate the vEVENTQ with
1110 * @type: Type of the vEVENTQ. Must be defined in enum iommu_veventq_type
1111 * @veventq_depth: Maximum number of events in the vEVENTQ
1112 * @out_veventq_id: The ID of the new vEVENTQ
1113 * @out_veventq_fd: The fd of the new vEVENTQ. User space must close the
1114 * successfully returned fd after using it
1115 * @__reserved: Must be 0
1116 *
1117 * Explicitly allocate a virtual event queue interface for a vIOMMU. A vIOMMU
1118 * can have multiple FDs for different types, but is confined to one per @type.
1119 * User space should open the @out_veventq_fd to read vEVENTs out of a vEVENTQ,
1120 * if there are vEVENTs available. A vEVENTQ will lose events due to overflow,
1121 * if the number of the vEVENTs hits @veventq_depth.
1122 *
1123 * Each vEVENT in a vEVENTQ encloses a struct iommufd_vevent_header followed by
1124 * a type-specific data structure, in a normal case:
1125 *
1126 * +-+---------+-------+---------+-------+-----+---------+-------+-+
1127 * | | header0 | data0 | header1 | data1 | ... | headerN | dataN | |
1128 * +-+---------+-------+---------+-------+-----+---------+-------+-+
1129 *
1130 * unless a tailing IOMMU_VEVENTQ_FLAG_LOST_EVENTS header is logged (refer to
1131 * struct iommufd_vevent_header).
1132 */
1133struct iommu_veventq_alloc {
1134 __u32 size;
1135 __u32 flags;
1136 __u32 viommu_id;
1137 __u32 type;
1138 __u32 veventq_depth;
1139 __u32 out_veventq_id;
1140 __u32 out_veventq_fd;
1141 __u32 __reserved;
1142};
1143#define IOMMU_VEVENTQ_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VEVENTQ_ALLOC)
1017#endif1144#endif
include/uapi/linux/vfio.h+19−10
@@ -932,29 +932,34 @@ struct vfio_device_bind_iommufd { · 931 unmodified lines
932 * VFIO_DEVICE_ATTACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 19,932 * VFIO_DEVICE_ATTACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 19,
933 * struct vfio_device_attach_iommufd_pt)933 * struct vfio_device_attach_iommufd_pt)
934 * @argsz: User filled size of this data.934 * @argsz: User filled size of this data.
935 * @flags: Must be 0.935 * @flags: Flags for attach.
936 * @pt_id: Input the target id which can represent an ioas or a hwpt936 * @pt_id: Input the target id which can represent an ioas or a hwpt
937 * allocated via iommufd subsystem.937 * allocated via iommufd subsystem.
938 * Output the input ioas id or the attached hwpt id which could938 * Output the input ioas id or the attached hwpt id which could
939 * be the specified hwpt itself or a hwpt automatically created939 * be the specified hwpt itself or a hwpt automatically created
940 * for the specified ioas by kernel during the attachment.940 * for the specified ioas by kernel during the attachment.
941 * @pasid: The pasid to be attached, only meaningful when
942 * VFIO_DEVICE_ATTACH_PASID is set in @flags
941 *943 *
942 * Associate the device with an address space within the bound iommufd.944 * Associate the device with an address space within the bound iommufd.
943 * Undo by VFIO_DEVICE_DETACH_IOMMUFD_PT or device fd close. This is only945 * Undo by VFIO_DEVICE_DETACH_IOMMUFD_PT or device fd close. This is only
944 * allowed on cdev fds.946 * allowed on cdev fds.
945 *947 *
946 * If a vfio device is currently attached to a valid hw_pagetable, without doing948 * If a vfio device or a pasid of this device is currently attached to a valid
947 * a VFIO_DEVICE_DETACH_IOMMUFD_PT, a second VFIO_DEVICE_ATTACH_IOMMUFD_PT ioctl949 * hw_pagetable (hwpt), without doing a VFIO_DEVICE_DETACH_IOMMUFD_PT, a second
948 * passing in another hw_pagetable (hwpt) id is allowed. This action, also known950 * VFIO_DEVICE_ATTACH_IOMMUFD_PT ioctl passing in another hwpt id is allowed.
949 * as a hw_pagetable replacement, will replace the device's currently attached951 * This action, also known as a hw_pagetable replacement, will replace the
950 * hw_pagetable with a new hw_pagetable corresponding to the given pt_id.952 * currently attached hwpt of the device or the pasid of this device with a new
953 * hwpt corresponding to the given pt_id.
951 *954 *
952 * Return: 0 on success, -errno on failure.955 * Return: 0 on success, -errno on failure.
953 */956 */
954struct vfio_device_attach_iommufd_pt {957struct vfio_device_attach_iommufd_pt {
955 __u32 argsz;958 __u32 argsz;
956 __u32 flags;959 __u32 flags;
960#define VFIO_DEVICE_ATTACH_PASID (1 << 0)
957 __u32 pt_id;961 __u32 pt_id;
962 __u32 pasid;
958};963};
959 964
960#define VFIO_DEVICE_ATTACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 19)965#define VFIO_DEVICE_ATTACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 19)
@@ -963,17 +968,21 @@ struct vfio_device_attach_iommufd_pt { · 2 unmodified lines
963 * VFIO_DEVICE_DETACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 20,968 * VFIO_DEVICE_DETACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 20,
964 * struct vfio_device_detach_iommufd_pt)969 * struct vfio_device_detach_iommufd_pt)
965 * @argsz: User filled size of this data.970 * @argsz: User filled size of this data.
966 * @flags: Must be 0.971 * @flags: Flags for detach.
972 * @pasid: The pasid to be detached, only meaningful when
973 * VFIO_DEVICE_DETACH_PASID is set in @flags
967 *974 *
968 * Remove the association of the device and its current associated address975 * Remove the association of the device or a pasid of the device and its current
969 * space. After it, the device should be in a blocking DMA state. This is only976 * associated address space. After it, the device or the pasid should be in a
970 * allowed on cdev fds.977 * blocking DMA state. This is only allowed on cdev fds.
971 *978 *
972 * Return: 0 on success, -errno on failure.979 * Return: 0 on success, -errno on failure.
973 */980 */
974struct vfio_device_detach_iommufd_pt {981struct vfio_device_detach_iommufd_pt {
975 __u32 argsz;982 __u32 argsz;
976 __u32 flags;983 __u32 flags;
984#define VFIO_DEVICE_DETACH_PASID (1 << 0)
985 __u32 pasid;
977};986};
978 987
979#define VFIO_DEVICE_DETACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 20)988#define VFIO_DEVICE_DETACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 20)
lib/idr.c+67−0
@@ -476,6 +476,73 @@ nospc: · 475 unmodified lines
476}476}
477EXPORT_SYMBOL(ida_alloc_range);477EXPORT_SYMBOL(ida_alloc_range);
478 478
479/**
480 * ida_find_first_range - Get the lowest used ID.
481 * @ida: IDA handle.
482 * @min: Lowest ID to get.
483 * @max: Highest ID to get.
484 *
485 * Get the lowest used ID between @min and @max, inclusive. The returned
486 * ID will not exceed %INT_MAX, even if @max is larger.
487 *
488 * Context: Any context. Takes and releases the xa_lock.
489 * Return: The lowest used ID, or errno if no used ID is found.
490 */
491int ida_find_first_range(struct ida *ida, unsigned int min, unsigned int max)
492{
493 unsigned long index = min / IDA_BITMAP_BITS;
494 unsigned int offset = min % IDA_BITMAP_BITS;
495 unsigned long *addr, size, bit;
496 unsigned long tmp = 0;
497 unsigned long flags;
498 void *entry;
499 int ret;
500
501 if ((int)min < 0)
502 return -EINVAL;
503 if ((int)max < 0)
504 max = INT_MAX;
505
506 xa_lock_irqsave(&ida->xa, flags);
507
508 entry = xa_find(&ida->xa, &index, max / IDA_BITMAP_BITS, XA_PRESENT);
509 if (!entry) {
510 ret = -ENOENT;
511 goto err_unlock;
512 }
513
514 if (index > min / IDA_BITMAP_BITS)
515 offset = 0;
516 if (index * IDA_BITMAP_BITS + offset > max) {
517 ret = -ENOENT;
518 goto err_unlock;
519 }
520
521 if (xa_is_value(entry)) {
522 tmp = xa_to_value(entry);
523 addr = &tmp;
524 size = BITS_PER_XA_VALUE;
525 } else {
526 addr = ((struct ida_bitmap *)entry)->bitmap;
527 size = IDA_BITMAP_BITS;
528 }
529
530 bit = find_next_bit(addr, size, offset);
531
532 xa_unlock_irqrestore(&ida->xa, flags);
533
534 if (bit == size ||
535 index * IDA_BITMAP_BITS + bit > max)
536 return -ENOENT;
537
538 return index * IDA_BITMAP_BITS + bit;
539
540err_unlock:
541 xa_unlock_irqrestore(&ida->xa, flags);
542 return ret;
543}
544EXPORT_SYMBOL(ida_find_first_range);
545
479/**546/**
480 * ida_free() - Release an allocated ID.547 * ida_free() - Release an allocated ID.
481 * @ida: IDA handle.548 * @ida: IDA handle.
lib/test_ida.c+70−0
@@ -189,6 +189,75 @@ static void ida_check_bad_free(struct ida *ida) · 188 unmodified lines
189 IDA_BUG_ON(ida, !ida_is_empty(ida));189 IDA_BUG_ON(ida, !ida_is_empty(ida));
190}190}
191 191
192/*
193 * Check ida_find_first_range() and varriants.
194 */
195static void ida_check_find_first(struct ida *ida)
196{
197 /* IDA is empty; all of the below should be not exist */
198 IDA_BUG_ON(ida, ida_exists(ida, 0));
199 IDA_BUG_ON(ida, ida_exists(ida, 3));
200 IDA_BUG_ON(ida, ida_exists(ida, 63));
201 IDA_BUG_ON(ida, ida_exists(ida, 1023));
202 IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1));
203
204 /* IDA contains a single value entry */
205 IDA_BUG_ON(ida, ida_alloc_min(ida, 3, GFP_KERNEL) != 3);
206 IDA_BUG_ON(ida, ida_exists(ida, 0));
207 IDA_BUG_ON(ida, !ida_exists(ida, 3));
208 IDA_BUG_ON(ida, ida_exists(ida, 63));
209 IDA_BUG_ON(ida, ida_exists(ida, 1023));
210 IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1));
211
212 IDA_BUG_ON(ida, ida_alloc_min(ida, 63, GFP_KERNEL) != 63);
213 IDA_BUG_ON(ida, ida_exists(ida, 0));
214 IDA_BUG_ON(ida, !ida_exists(ida, 3));
215 IDA_BUG_ON(ida, !ida_exists(ida, 63));
216 IDA_BUG_ON(ida, ida_exists(ida, 1023));
217 IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1));
218
219 /* IDA contains a single bitmap */
220 IDA_BUG_ON(ida, ida_alloc_min(ida, 1023, GFP_KERNEL) != 1023);
221 IDA_BUG_ON(ida, ida_exists(ida, 0));
222 IDA_BUG_ON(ida, !ida_exists(ida, 3));
223 IDA_BUG_ON(ida, !ida_exists(ida, 63));
224 IDA_BUG_ON(ida, !ida_exists(ida, 1023));
225 IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1));
226
227 /* IDA contains a tree */
228 IDA_BUG_ON(ida, ida_alloc_min(ida, (1 << 20) - 1, GFP_KERNEL) != (1 << 20) - 1);
229 IDA_BUG_ON(ida, ida_exists(ida, 0));
230 IDA_BUG_ON(ida, !ida_exists(ida, 3));
231 IDA_BUG_ON(ida, !ida_exists(ida, 63));
232 IDA_BUG_ON(ida, !ida_exists(ida, 1023));
233 IDA_BUG_ON(ida, !ida_exists(ida, (1 << 20) - 1));
234
235 /* Now try to find first */
236 IDA_BUG_ON(ida, ida_find_first(ida) != 3);
237 IDA_BUG_ON(ida, ida_find_first_range(ida, -1, 2) != -EINVAL);
238 IDA_BUG_ON(ida, ida_find_first_range(ida, 0, 2) != -ENOENT); // no used ID
239 IDA_BUG_ON(ida, ida_find_first_range(ida, 0, 3) != 3);
240 IDA_BUG_ON(ida, ida_find_first_range(ida, 1, 3) != 3);
241 IDA_BUG_ON(ida, ida_find_first_range(ida, 3, 3) != 3);
242 IDA_BUG_ON(ida, ida_find_first_range(ida, 2, 4) != 3);
243 IDA_BUG_ON(ida, ida_find_first_range(ida, 4, 3) != -ENOENT); // min > max, fail
244 IDA_BUG_ON(ida, ida_find_first_range(ida, 4, 60) != -ENOENT); // no used ID
245 IDA_BUG_ON(ida, ida_find_first_range(ida, 4, 64) != 63);
246 IDA_BUG_ON(ida, ida_find_first_range(ida, 63, 63) != 63);
247 IDA_BUG_ON(ida, ida_find_first_range(ida, 64, 1026) != 1023);
248 IDA_BUG_ON(ida, ida_find_first_range(ida, 1023, 1023) != 1023);
249 IDA_BUG_ON(ida, ida_find_first_range(ida, 1023, (1 << 20) - 1) != 1023);
250 IDA_BUG_ON(ida, ida_find_first_range(ida, 1024, (1 << 20) - 1) != (1 << 20) - 1);
251 IDA_BUG_ON(ida, ida_find_first_range(ida, (1 << 20), INT_MAX) != -ENOENT);
252
253 ida_free(ida, 3);
254 ida_free(ida, 63);
255 ida_free(ida, 1023);
256 ida_free(ida, (1 << 20) - 1);
257
258 IDA_BUG_ON(ida, !ida_is_empty(ida));
259}
260
192static DEFINE_IDA(ida);261static DEFINE_IDA(ida);
193 262
194static int ida_checks(void)263static int ida_checks(void)
@@ -202,6 +271,7 @@ static int ida_checks(void) · 7 unmodified lines
202 ida_check_max(&ida);271 ida_check_max(&ida);
203 ida_check_conv(&ida);272 ida_check_conv(&ida);
204 ida_check_bad_free(&ida);273 ida_check_bad_free(&ida);
274 ida_check_find_first(&ida);
205 275
206 printk("IDA: %u of %u tests passed\n", tests_passed, tests_run);276 printk("IDA: %u of %u tests passed\n", tests_passed, tests_run);
207 return (tests_run != tests_passed) ? 0 : -EINVAL;277 return (tests_run != tests_passed) ? 0 : -EINVAL;
tools/testing/selftests/iommu/iommufd.c+365−0
@@ -342,12 +342,14 @@ FIXTURE(iommufd_ioas) · 341 unmodified lines
342 uint32_t hwpt_id;342 uint32_t hwpt_id;
343 uint32_t device_id;343 uint32_t device_id;
344 uint64_t base_iova;344 uint64_t base_iova;
345 uint32_t device_pasid_id;
345};346};
346 347
347FIXTURE_VARIANT(iommufd_ioas)348FIXTURE_VARIANT(iommufd_ioas)
348{349{
349 unsigned int mock_domains;350 unsigned int mock_domains;
350 unsigned int memory_limit;351 unsigned int memory_limit;
352 bool pasid_capable;
351};353};
352 354
353FIXTURE_SETUP(iommufd_ioas)355FIXTURE_SETUP(iommufd_ioas)
@@ -372,6 +374,12 @@ FIXTURE_SETUP(iommufd_ioas) · 18 unmodified lines
372 IOMMU_TEST_DEV_CACHE_DEFAULT);374 IOMMU_TEST_DEV_CACHE_DEFAULT);
373 self->base_iova = MOCK_APERTURE_START;375 self->base_iova = MOCK_APERTURE_START;
374 }376 }
377
378 if (variant->pasid_capable)
379 test_cmd_mock_domain_flags(self->ioas_id,
380 MOCK_FLAGS_DEVICE_PASID,
381 NULL, NULL,
382 &self->device_pasid_id);
375}383}
376 384
377FIXTURE_TEARDOWN(iommufd_ioas)385FIXTURE_TEARDOWN(iommufd_ioas)
@@ -387,6 +395,7 @@ FIXTURE_VARIANT_ADD(iommufd_ioas, no_domain) · 9 unmodified lines
387FIXTURE_VARIANT_ADD(iommufd_ioas, mock_domain)395FIXTURE_VARIANT_ADD(iommufd_ioas, mock_domain)
388{396{
389 .mock_domains = 1,397 .mock_domains = 1,
398 .pasid_capable = true,
390};399};
391 400
392FIXTURE_VARIANT_ADD(iommufd_ioas, two_mock_domain)401FIXTURE_VARIANT_ADD(iommufd_ioas, two_mock_domain)
@@ -439,6 +448,10 @@ TEST_F(iommufd_ioas, alloc_hwpt_nested) · 46 unmodified lines
439 &test_hwpt_id);448 &test_hwpt_id);
440 test_err_hwpt_alloc(EINVAL, self->device_id, self->device_id, 0,449 test_err_hwpt_alloc(EINVAL, self->device_id, self->device_id, 0,
441 &test_hwpt_id);450 &test_hwpt_id);
451 test_err_hwpt_alloc(EOPNOTSUPP, self->device_id, self->ioas_id,
452 IOMMU_HWPT_ALLOC_NEST_PARENT |
453 IOMMU_HWPT_FAULT_ID_VALID,
454 &test_hwpt_id);
442 455
443 test_cmd_hwpt_alloc(self->device_id, self->ioas_id,456 test_cmd_hwpt_alloc(self->device_id, self->ioas_id,
444 IOMMU_HWPT_ALLOC_NEST_PARENT,457 IOMMU_HWPT_ALLOC_NEST_PARENT,
@@ -748,6 +761,8 @@ TEST_F(iommufd_ioas, get_hw_info) · 303 unmodified lines
748 } buffer_smaller;761 } buffer_smaller;
749 762
750 if (self->device_id) {763 if (self->device_id) {
764 uint8_t max_pasid = 0;
765
751 /* Provide a zero-size user_buffer */766 /* Provide a zero-size user_buffer */
752 test_cmd_get_hw_info(self->device_id, NULL, 0);767 test_cmd_get_hw_info(self->device_id, NULL, 0);
753 /* Provide a user_buffer with exact size */768 /* Provide a user_buffer with exact size */
@@ -762,6 +777,13 @@ TEST_F(iommufd_ioas, get_hw_info) · 8 unmodified lines
762 * the fields within the size range still gets updated.777 * the fields within the size range still gets updated.
763 */778 */
764 test_cmd_get_hw_info(self->device_id, &buffer_smaller, sizeof(buffer_smaller));779 test_cmd_get_hw_info(self->device_id, &buffer_smaller, sizeof(buffer_smaller));
780 test_cmd_get_hw_info_pasid(self->device_id, &max_pasid);
781 ASSERT_EQ(0, max_pasid);
782 if (variant->pasid_capable) {
783 test_cmd_get_hw_info_pasid(self->device_pasid_id,
784 &max_pasid);
785 ASSERT_EQ(MOCK_PASID_WIDTH, max_pasid);
786 }
765 } else {787 } else {
766 test_err_get_hw_info(ENOENT, self->device_id,788 test_err_get_hw_info(ENOENT, self->device_id,
767 &buffer_exact, sizeof(buffer_exact));789 &buffer_exact, sizeof(buffer_exact));
@@ -2736,6 +2758,7 @@ TEST_F(iommufd_viommu, viommu_alloc_nested_iopf) · 1968 unmodified lines
2736 uint32_t iopf_hwpt_id;2758 uint32_t iopf_hwpt_id;
2737 uint32_t fault_id;2759 uint32_t fault_id;
2738 uint32_t fault_fd;2760 uint32_t fault_fd;
2761 uint32_t vdev_id;
2739 2762
2740 if (self->device_id) {2763 if (self->device_id) {
2741 test_ioctl_fault_alloc(&fault_id, &fault_fd);2764 test_ioctl_fault_alloc(&fault_id, &fault_fd);
@@ -2752,6 +2775,10 @@ TEST_F(iommufd_viommu, viommu_alloc_nested_iopf) · 10 unmodified lines
2752 &iopf_hwpt_id, IOMMU_HWPT_DATA_SELFTEST, &data,2775 &iopf_hwpt_id, IOMMU_HWPT_DATA_SELFTEST, &data,
2753 sizeof(data));2776 sizeof(data));
2754 2777
2778 /* Must allocate vdevice before attaching to a nested hwpt */
2779 test_err_mock_domain_replace(ENOENT, self->stdev_id,
2780 iopf_hwpt_id);
2781 test_cmd_vdevice_alloc(viommu_id, dev_id, 0x99, &vdev_id);
2755 test_cmd_mock_domain_replace(self->stdev_id, iopf_hwpt_id);2782 test_cmd_mock_domain_replace(self->stdev_id, iopf_hwpt_id);
2756 EXPECT_ERRNO(EBUSY,2783 EXPECT_ERRNO(EBUSY,
2757 _test_ioctl_destroy(self->fd, iopf_hwpt_id));2784 _test_ioctl_destroy(self->fd, iopf_hwpt_id));
@@ -2769,15 +2796,46 @@ TEST_F(iommufd_viommu, vdevice_alloc) · 11 unmodified lines
2769 uint32_t viommu_id = self->viommu_id;2796 uint32_t viommu_id = self->viommu_id;
2770 uint32_t dev_id = self->device_id;2797 uint32_t dev_id = self->device_id;
2771 uint32_t vdev_id = 0;2798 uint32_t vdev_id = 0;
2799 uint32_t veventq_id;
2800 uint32_t veventq_fd;
2801 int prev_seq = -1;
2772 2802
2773 if (dev_id) {2803 if (dev_id) {
2804 /* Must allocate vdevice before attaching to a nested hwpt */
2805 test_err_mock_domain_replace(ENOENT, self->stdev_id,
2806 self->nested_hwpt_id);
2807
2808 /* Allocate a vEVENTQ with veventq_depth=2 */
2809 test_cmd_veventq_alloc(viommu_id, IOMMU_VEVENTQ_TYPE_SELFTEST,
2810 &veventq_id, &veventq_fd);
2811 test_err_veventq_alloc(EEXIST, viommu_id,
2812 IOMMU_VEVENTQ_TYPE_SELFTEST, NULL, NULL);
2774 /* Set vdev_id to 0x99, unset it, and set to 0x88 */2813 /* Set vdev_id to 0x99, unset it, and set to 0x88 */
2775 test_cmd_vdevice_alloc(viommu_id, dev_id, 0x99, &vdev_id);2814 test_cmd_vdevice_alloc(viommu_id, dev_id, 0x99, &vdev_id);
2815 test_cmd_mock_domain_replace(self->stdev_id,
2816 self->nested_hwpt_id);
2817 test_cmd_trigger_vevents(dev_id, 1);
2818 test_cmd_read_vevents(veventq_fd, 1, 0x99, &prev_seq);
2776 test_err_vdevice_alloc(EEXIST, viommu_id, dev_id, 0x99,2819 test_err_vdevice_alloc(EEXIST, viommu_id, dev_id, 0x99,
2777 &vdev_id);2820 &vdev_id);
2821 test_cmd_mock_domain_replace(self->stdev_id, self->ioas_id);
2778 test_ioctl_destroy(vdev_id);2822 test_ioctl_destroy(vdev_id);
2823
2824 /* Try again with 0x88 */
2779 test_cmd_vdevice_alloc(viommu_id, dev_id, 0x88, &vdev_id);2825 test_cmd_vdevice_alloc(viommu_id, dev_id, 0x88, &vdev_id);
2826 test_cmd_mock_domain_replace(self->stdev_id,
2827 self->nested_hwpt_id);
2828 /* Trigger an overflow with three events */
2829 test_cmd_trigger_vevents(dev_id, 3);
2830 test_err_read_vevents(EOVERFLOW, veventq_fd, 3, 0x88,
2831 &prev_seq);
2832 /* Overflow must be gone after the previous reads */
2833 test_cmd_trigger_vevents(dev_id, 1);
2834 test_cmd_read_vevents(veventq_fd, 1, 0x88, &prev_seq);
2835 close(veventq_fd);
2836 test_cmd_mock_domain_replace(self->stdev_id, self->ioas_id);
2780 test_ioctl_destroy(vdev_id);2837 test_ioctl_destroy(vdev_id);
2838 test_ioctl_destroy(veventq_id);
2781 } else {2839 } else {
2782 test_err_vdevice_alloc(ENOENT, viommu_id, dev_id, 0x99, NULL);2840 test_err_vdevice_alloc(ENOENT, viommu_id, dev_id, 0x99, NULL);
2783 }2841 }
@@ -2956,4 +3014,311 @@ TEST_F(iommufd_viommu, vdevice_cache) · 172 unmodified lines
2956 }3014 }
2957}3015}
2958 3016
3017FIXTURE(iommufd_device_pasid)
3018{
3019 int fd;
3020 uint32_t ioas_id;
3021 uint32_t hwpt_id;
3022 uint32_t stdev_id;
3023 uint32_t device_id;
3024 uint32_t no_pasid_stdev_id;
3025 uint32_t no_pasid_device_id;
3026};
3027
3028FIXTURE_VARIANT(iommufd_device_pasid)
3029{
3030 bool pasid_capable;
3031};
3032
3033FIXTURE_SETUP(iommufd_device_pasid)
3034{
3035 self->fd = open("/dev/iommu", O_RDWR);
3036 ASSERT_NE(-1, self->fd);
3037 test_ioctl_ioas_alloc(&self->ioas_id);
3038
3039 test_cmd_mock_domain_flags(self->ioas_id,
3040 MOCK_FLAGS_DEVICE_PASID,
3041 &self->stdev_id, &self->hwpt_id,
3042 &self->device_id);
3043 if (!variant->pasid_capable)
3044 test_cmd_mock_domain_flags(self->ioas_id, 0,
3045 &self->no_pasid_stdev_id, NULL,
3046 &self->no_pasid_device_id);
3047}
3048
3049FIXTURE_TEARDOWN(iommufd_device_pasid)
3050{
3051 teardown_iommufd(self->fd, _metadata);
3052}
3053
3054FIXTURE_VARIANT_ADD(iommufd_device_pasid, no_pasid)
3055{
3056 .pasid_capable = false,
3057};
3058
3059FIXTURE_VARIANT_ADD(iommufd_device_pasid, has_pasid)
3060{
3061 .pasid_capable = true,
3062};
3063
3064TEST_F(iommufd_device_pasid, pasid_attach)
3065{
3066 struct iommu_hwpt_selftest data = {
3067 .iotlb = IOMMU_TEST_IOTLB_DEFAULT,
3068 };
3069 uint32_t nested_hwpt_id[3] = {};
3070 uint32_t parent_hwpt_id = 0;
3071 uint32_t fault_id, fault_fd;
3072 uint32_t s2_hwpt_id = 0;
3073 uint32_t iopf_hwpt_id;
3074 uint32_t pasid = 100;
3075 uint32_t viommu_id;
3076
3077 /*
3078 * Negative, detach pasid without attaching, this is not expected.
3079 * But it should not result in failure anyway.
3080 */
3081 test_cmd_pasid_detach(pasid);
3082
3083 /* Allocate two nested hwpts sharing one common parent hwpt */
3084 test_cmd_hwpt_alloc(self->device_id, self->ioas_id,
3085 IOMMU_HWPT_ALLOC_NEST_PARENT,
3086 &parent_hwpt_id);
3087 test_cmd_hwpt_alloc_nested(self->device_id, parent_hwpt_id,
3088 IOMMU_HWPT_ALLOC_PASID,
3089 &nested_hwpt_id[0],
3090 IOMMU_HWPT_DATA_SELFTEST,
3091 &data, sizeof(data));
3092 test_cmd_hwpt_alloc_nested(self->device_id, parent_hwpt_id,
3093 IOMMU_HWPT_ALLOC_PASID,
3094 &nested_hwpt_id[1],
3095 IOMMU_HWPT_DATA_SELFTEST,
3096 &data, sizeof(data));
3097
3098 /* Fault related preparation */
3099 test_ioctl_fault_alloc(&fault_id, &fault_fd);
3100 test_cmd_hwpt_alloc_iopf(self->device_id, parent_hwpt_id, fault_id,
3101 IOMMU_HWPT_FAULT_ID_VALID | IOMMU_HWPT_ALLOC_PASID,
3102 &iopf_hwpt_id,
3103 IOMMU_HWPT_DATA_SELFTEST, &data,
3104 sizeof(data));
3105
3106 /* Allocate a regular nested hwpt based on viommu */
3107 test_cmd_viommu_alloc(self->device_id, parent_hwpt_id,
3108 IOMMU_VIOMMU_TYPE_SELFTEST,
3109 &viommu_id);
3110 test_cmd_hwpt_alloc_nested(self->device_id, viommu_id,
3111 IOMMU_HWPT_ALLOC_PASID,
3112 &nested_hwpt_id[2],
3113 IOMMU_HWPT_DATA_SELFTEST, &data,
3114 sizeof(data));
3115
3116 test_cmd_hwpt_alloc(self->device_id, self->ioas_id,
3117 IOMMU_HWPT_ALLOC_PASID,
3118 &s2_hwpt_id);
3119
3120 /* Attach RID to non-pasid compat domain, */
3121 test_cmd_mock_domain_replace(self->stdev_id, parent_hwpt_id);
3122 /* then attach to pasid should fail */
3123 test_err_pasid_attach(EINVAL, pasid, s2_hwpt_id);
3124
3125 /* Attach RID to pasid compat domain, */
3126 test_cmd_mock_domain_replace(self->stdev_id, s2_hwpt_id);
3127 /* then attach to pasid should succeed, */
3128 test_cmd_pasid_attach(pasid, nested_hwpt_id[0]);
3129 /* but attach RID to non-pasid compat domain should fail now. */
3130 test_err_mock_domain_replace(EINVAL, self->stdev_id, parent_hwpt_id);
3131 /*
3132 * Detach hwpt from pasid 100, and check if the pasid 100
3133 * has null domain.
3134 */
3135 test_cmd_pasid_detach(pasid);
3136 ASSERT_EQ(0,
3137 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3138 pasid, 0));
3139 /* RID is attached to pasid-comapt domain, pasid path is not used */
3140
3141 if (!variant->pasid_capable) {
3142 /*
3143 * PASID-compatible domain can be used by non-PASID-capable
3144 * device.
3145 */
3146 test_cmd_mock_domain_replace(self->no_pasid_stdev_id, nested_hwpt_id[0]);
3147 test_cmd_mock_domain_replace(self->no_pasid_stdev_id, self->ioas_id);
3148 /*
3149 * Attach hwpt to pasid 100 of non-PASID-capable device,
3150 * should fail, no matter domain is pasid-comapt or not.
3151 */
3152 EXPECT_ERRNO(EINVAL,
3153 _test_cmd_pasid_attach(self->fd, self->no_pasid_stdev_id,
3154 pasid, parent_hwpt_id));
3155 EXPECT_ERRNO(EINVAL,
3156 _test_cmd_pasid_attach(self->fd, self->no_pasid_stdev_id,
3157 pasid, s2_hwpt_id));
3158 }
3159
3160 /*
3161 * Attach non pasid compat hwpt to pasid-capable device, should
3162 * fail, and have null domain.
3163 */
3164 test_err_pasid_attach(EINVAL, pasid, parent_hwpt_id);
3165 ASSERT_EQ(0,
3166 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3167 pasid, 0));
3168
3169 /*
3170 * Attach ioas to pasid 100, should fail, domain should
3171 * be null.
3172 */
3173 test_err_pasid_attach(EINVAL, pasid, self->ioas_id);
3174 ASSERT_EQ(0,
3175 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3176 pasid, 0));
3177
3178 /*
3179 * Attach the s2_hwpt to pasid 100, should succeed, domain should
3180 * be valid.
3181 */
3182 test_cmd_pasid_attach(pasid, s2_hwpt_id);
3183 ASSERT_EQ(0,
3184 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3185 pasid, s2_hwpt_id));
3186
3187 /*
3188 * Try attach pasid 100 with another hwpt, should FAIL
3189 * as attach does not allow overwrite, use REPLACE instead.
3190 */
3191 test_err_pasid_attach(EBUSY, pasid, nested_hwpt_id[0]);
3192
3193 /*
3194 * Detach hwpt from pasid 100 for next test, should succeed,
3195 * and have null domain.
3196 */
3197 test_cmd_pasid_detach(pasid);
3198 ASSERT_EQ(0,
3199 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3200 pasid, 0));
3201
3202 /*
3203 * Attach nested hwpt to pasid 100, should succeed, domain
3204 * should be valid.
3205 */
3206 test_cmd_pasid_attach(pasid, nested_hwpt_id[0]);
3207 ASSERT_EQ(0,
3208 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3209 pasid, nested_hwpt_id[0]));
3210
3211 /* Attach to pasid 100 which has been attached, should fail. */
3212 test_err_pasid_attach(EBUSY, pasid, nested_hwpt_id[0]);
3213
3214 /* cleanup pasid 100 */
3215 test_cmd_pasid_detach(pasid);
3216
3217 /* Replace tests */
3218
3219 pasid = 200;
3220 /*
3221 * Replace pasid 200 without attaching it, should fail
3222 * with -EINVAL.
3223 */
3224 test_err_pasid_replace(EINVAL, pasid, s2_hwpt_id);
3225
3226 /*
3227 * Attach the s2 hwpt to pasid 200, should succeed, domain should
3228 * be valid.
3229 */
3230 test_cmd_pasid_attach(pasid, s2_hwpt_id);
3231 ASSERT_EQ(0,
3232 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3233 pasid, s2_hwpt_id));
3234
3235 /*
3236 * Replace pasid 200 with self->ioas_id, should fail
3237 * and domain should be the prior s2 hwpt.
3238 */
3239 test_err_pasid_replace(EINVAL, pasid, self->ioas_id);
3240 ASSERT_EQ(0,
3241 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3242 pasid, s2_hwpt_id));
3243
3244 /*
3245 * Replace a nested hwpt for pasid 200, should succeed,
3246 * and have valid domain.
3247 */
3248 test_cmd_pasid_replace(pasid, nested_hwpt_id[0]);
3249 ASSERT_EQ(0,
3250 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3251 pasid, nested_hwpt_id[0]));
3252
3253 /*
3254 * Replace with another nested hwpt for pasid 200, should
3255 * succeed, and have valid domain.
3256 */
3257 test_cmd_pasid_replace(pasid, nested_hwpt_id[1]);
3258 ASSERT_EQ(0,
3259 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3260 pasid, nested_hwpt_id[1]));
3261
3262 /* cleanup pasid 200 */
3263 test_cmd_pasid_detach(pasid);
3264
3265 /* Negative Tests for pasid replace, use pasid 1024 */
3266
3267 /*
3268 * Attach the s2 hwpt to pasid 1024, should succeed, domain should
3269 * be valid.
3270 */
3271 pasid = 1024;
3272 test_cmd_pasid_attach(pasid, s2_hwpt_id);
3273 ASSERT_EQ(0,
3274 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3275 pasid, s2_hwpt_id));
3276
3277 /*
3278 * Replace pasid 1024 with nested_hwpt_id[0], should fail,
3279 * but have the old valid domain. This is a designed
3280 * negative case. Normally, this shall succeed.
3281 */
3282 test_err_pasid_replace(ENOMEM, pasid, nested_hwpt_id[0]);
3283 ASSERT_EQ(0,
3284 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3285 pasid, s2_hwpt_id));
3286
3287 /* cleanup pasid 1024 */
3288 test_cmd_pasid_detach(pasid);
3289
3290 /* Attach to iopf-capable hwpt */
3291
3292 /*
3293 * Attach an iopf hwpt to pasid 2048, should succeed, domain should
3294 * be valid.
3295 */
3296 pasid = 2048;
3297 test_cmd_pasid_attach(pasid, iopf_hwpt_id);
3298 ASSERT_EQ(0,
3299 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3300 pasid, iopf_hwpt_id));
3301
3302 test_cmd_trigger_iopf_pasid(self->device_id, pasid, fault_fd);
3303
3304 /*
3305 * Replace with s2_hwpt_id for pasid 2048, should
3306 * succeed, and have valid domain.
3307 */
3308 test_cmd_pasid_replace(pasid, s2_hwpt_id);
3309 ASSERT_EQ(0,
3310 test_cmd_pasid_check_hwpt(self->fd, self->stdev_id,
3311 pasid, s2_hwpt_id));
3312
3313 /* cleanup pasid 2048 */
3314 test_cmd_pasid_detach(pasid);
3315
3316 test_ioctl_destroy(iopf_hwpt_id);
3317 close(fault_fd);
3318 test_ioctl_destroy(fault_id);
3319
3320 /* Detach the s2_hwpt_id from RID */
3321 test_cmd_mock_domain_replace(self->stdev_id, self->ioas_id);
3322}
3323
2959TEST_HARNESS_MAIN3324TEST_HARNESS_MAIN
tools/testing/selftests/iommu/iommufd_fail_nth.c+51−8
@@ -209,12 +209,16 @@ FIXTURE(basic_fail_nth) · 208 unmodified lines
209{209{
210 int fd;210 int fd;
211 uint32_t access_id;211 uint32_t access_id;
212 uint32_t stdev_id;
213 uint32_t pasid;
212};214};
213 215
214FIXTURE_SETUP(basic_fail_nth)216FIXTURE_SETUP(basic_fail_nth)
215{217{
216 self->fd = -1;218 self->fd = -1;
217 self->access_id = 0;219 self->access_id = 0;
220 self->stdev_id = 0;
221 self->pasid = 0; //test should use a non-zero value
218}222}
219 223
220FIXTURE_TEARDOWN(basic_fail_nth)224FIXTURE_TEARDOWN(basic_fail_nth)
@@ -226,6 +230,8 @@ FIXTURE_TEARDOWN(basic_fail_nth) · 5 unmodified lines
226 rc = _test_cmd_destroy_access(self->access_id);230 rc = _test_cmd_destroy_access(self->access_id);
227 assert(rc == 0);231 assert(rc == 0);
228 }232 }
233 if (self->pasid && self->stdev_id)
234 _test_cmd_pasid_detach(self->fd, self->stdev_id, self->pasid);
229 teardown_iommufd(self->fd, _metadata);235 teardown_iommufd(self->fd, _metadata);
230}236}
231 237
@@ -620,10 +626,11 @@ TEST_FAIL_NTH(basic_fail_nth, device) · 388 unmodified lines
620 };626 };
621 struct iommu_test_hw_info info;627 struct iommu_test_hw_info info;
622 uint32_t fault_id, fault_fd;628 uint32_t fault_id, fault_fd;
629 uint32_t veventq_id, veventq_fd;
623 uint32_t fault_hwpt_id;630 uint32_t fault_hwpt_id;
631 uint32_t test_hwpt_id;
624 uint32_t ioas_id;632 uint32_t ioas_id;
625 uint32_t ioas_id2;633 uint32_t ioas_id2;
626 uint32_t stdev_id;
627 uint32_t idev_id;634 uint32_t idev_id;
628 uint32_t hwpt_id;635 uint32_t hwpt_id;
629 uint32_t viommu_id;636 uint32_t viommu_id;
@@ -654,25 +661,30 @@ TEST_FAIL_NTH(basic_fail_nth, device) · 24 unmodified lines
654 661
655 fail_nth_enable();662 fail_nth_enable();
656 663
657 if (_test_cmd_mock_domain(self->fd, ioas_id, &stdev_id, NULL,664 if (_test_cmd_mock_domain_flags(self->fd, ioas_id,
658 &idev_id))665 MOCK_FLAGS_DEVICE_PASID,
666 &self->stdev_id, NULL, &idev_id))
659 return -1;667 return -1;
660 668
661 if (_test_cmd_get_hw_info(self->fd, idev_id, &info, sizeof(info), NULL))669 if (_test_cmd_get_hw_info(self->fd, idev_id, &info,
670 sizeof(info), NULL, NULL))
662 return -1;671 return -1;
663 672
664 if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0, 0, &hwpt_id,673 if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0,
674 IOMMU_HWPT_ALLOC_PASID, &hwpt_id,
665 IOMMU_HWPT_DATA_NONE, 0, 0))675 IOMMU_HWPT_DATA_NONE, 0, 0))
666 return -1;676 return -1;
667 677
668 if (_test_cmd_mock_domain_replace(self->fd, stdev_id, ioas_id2, NULL))678 if (_test_cmd_mock_domain_replace(self->fd, self->stdev_id, ioas_id2, NULL))
669 return -1;679 return -1;
670 680
671 if (_test_cmd_mock_domain_replace(self->fd, stdev_id, hwpt_id, NULL))681 if (_test_cmd_mock_domain_replace(self->fd, self->stdev_id, hwpt_id, NULL))
672 return -1;682 return -1;
673 683
674 if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0,684 if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0,
675 IOMMU_HWPT_ALLOC_NEST_PARENT, &hwpt_id,685 IOMMU_HWPT_ALLOC_NEST_PARENT |
686 IOMMU_HWPT_ALLOC_PASID,
687 &hwpt_id,
676 IOMMU_HWPT_DATA_NONE, 0, 0))688 IOMMU_HWPT_DATA_NONE, 0, 0))
677 return -1;689 return -1;
678 690
@@ -692,6 +704,37 @@ TEST_FAIL_NTH(basic_fail_nth, device) · 13 unmodified lines
692 IOMMU_HWPT_DATA_SELFTEST, &data, sizeof(data)))704 IOMMU_HWPT_DATA_SELFTEST, &data, sizeof(data)))
693 return -1;705 return -1;
694 706
707 if (_test_cmd_veventq_alloc(self->fd, viommu_id,
708 IOMMU_VEVENTQ_TYPE_SELFTEST, &veventq_id,
709 &veventq_fd))
710 return -1;
711 close(veventq_fd);
712
713 if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0,
714 IOMMU_HWPT_ALLOC_PASID,
715 &test_hwpt_id,
716 IOMMU_HWPT_DATA_NONE, 0, 0))
717 return -1;
718
719 /* Tests for pasid attach/replace/detach */
720
721 self->pasid = 200;
722
723 if (_test_cmd_pasid_attach(self->fd, self->stdev_id,
724 self->pasid, hwpt_id)) {
725 self->pasid = 0;
726 return -1;
727 }
728
729 if (_test_cmd_pasid_replace(self->fd, self->stdev_id,
730 self->pasid, test_hwpt_id))
731 return -1;
732
733 if (_test_cmd_pasid_detach(self->fd, self->stdev_id, self->pasid))
734 return -1;
735
736 self->pasid = 0;
737
695 return 0;738 return 0;
696}739}
697 740
tools/testing/selftests/iommu/iommufd_utils.h+222−7
@@ -9,6 +9,7 @@ · 8 unmodified lines
9#include <sys/ioctl.h>9#include <sys/ioctl.h>
10#include <stdint.h>10#include <stdint.h>
11#include <assert.h>11#include <assert.h>
12#include <poll.h>
12 13
13#include "../kselftest_harness.h"14#include "../kselftest_harness.h"
14#include "../../../../drivers/iommu/iommufd/iommufd_test.h"15#include "../../../../drivers/iommu/iommufd/iommufd_test.h"
@@ -757,7 +758,8 @@ static void teardown_iommufd(int fd, struct __test_metadata *_metadata) · 742 unmodified lines
757 758
758/* @data can be NULL */759/* @data can be NULL */
759static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data,760static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data,
760 size_t data_len, uint32_t *capabilities)761 size_t data_len, uint32_t *capabilities,
762 uint8_t *max_pasid)
761{763{
762 struct iommu_test_hw_info *info = (struct iommu_test_hw_info *)data;764 struct iommu_test_hw_info *info = (struct iommu_test_hw_info *)data;
763 struct iommu_hw_info cmd = {765 struct iommu_hw_info cmd = {
@@ -802,6 +804,9 @@ static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data, · 38 unmodified lines
802 assert(!info->flags);804 assert(!info->flags);
803 }805 }
804 806
807 if (max_pasid)
808 *max_pasid = cmd.out_max_pasid_log2;
809
805 if (capabilities)810 if (capabilities)
806 *capabilities = cmd.out_capabilities;811 *capabilities = cmd.out_capabilities;
807 812
@@ -810,14 +815,19 @@ static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data, · 2 unmodified lines
810 815
811#define test_cmd_get_hw_info(device_id, data, data_len) \816#define test_cmd_get_hw_info(device_id, data, data_len) \
812 ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, data, \817 ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, data, \
813 data_len, NULL))818 data_len, NULL, NULL))
814 819
815#define test_err_get_hw_info(_errno, device_id, data, data_len) \820#define test_err_get_hw_info(_errno, device_id, data, data_len) \
816 EXPECT_ERRNO(_errno, _test_cmd_get_hw_info(self->fd, device_id, data, \821 EXPECT_ERRNO(_errno, _test_cmd_get_hw_info(self->fd, device_id, data, \
817 data_len, NULL))822 data_len, NULL, NULL))
818 823
819#define test_cmd_get_hw_capabilities(device_id, caps, mask) \824#define test_cmd_get_hw_capabilities(device_id, caps, mask) \
820 ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, NULL, 0, &caps))825 ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, NULL, \
826 0, &caps, NULL))
827
828#define test_cmd_get_hw_info_pasid(device_id, max_pasid) \
829 ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, NULL, \
830 0, NULL, max_pasid))
821 831
822static int _test_ioctl_fault_alloc(int fd, __u32 *fault_id, __u32 *fault_fd)832static int _test_ioctl_fault_alloc(int fd, __u32 *fault_id, __u32 *fault_fd)
823{833{
@@ -842,14 +852,15 @@ static int _test_ioctl_fault_alloc(int fd, __u32 *fault_id, __u32 *fault_fd) · 18 unmodified lines
842 ASSERT_NE(0, *(fault_fd)); \852 ASSERT_NE(0, *(fault_fd)); \
843 })853 })
844 854
845static int _test_cmd_trigger_iopf(int fd, __u32 device_id, __u32 fault_fd)855static int _test_cmd_trigger_iopf(int fd, __u32 device_id, __u32 pasid,
856 __u32 fault_fd)
846{857{
847 struct iommu_test_cmd trigger_iopf_cmd = {858 struct iommu_test_cmd trigger_iopf_cmd = {
848 .size = sizeof(trigger_iopf_cmd),859 .size = sizeof(trigger_iopf_cmd),
849 .op = IOMMU_TEST_OP_TRIGGER_IOPF,860 .op = IOMMU_TEST_OP_TRIGGER_IOPF,
850 .trigger_iopf = {861 .trigger_iopf = {
851 .dev_id = device_id,862 .dev_id = device_id,
852 .pasid = 0x1,863 .pasid = pasid,
853 .grpid = 0x2,864 .grpid = 0x2,
854 .perm = IOMMU_PGFAULT_PERM_READ | IOMMU_PGFAULT_PERM_WRITE,865 .perm = IOMMU_PGFAULT_PERM_READ | IOMMU_PGFAULT_PERM_WRITE,
855 .addr = 0xdeadbeaf,866 .addr = 0xdeadbeaf,
@@ -880,7 +891,10 @@ static int _test_cmd_trigger_iopf(int fd, __u32 device_id, __u32 fault_fd) · 24 unmodified lines
880}891}
881 892
882#define test_cmd_trigger_iopf(device_id, fault_fd) \893#define test_cmd_trigger_iopf(device_id, fault_fd) \
883 ASSERT_EQ(0, _test_cmd_trigger_iopf(self->fd, device_id, fault_fd))894 ASSERT_EQ(0, _test_cmd_trigger_iopf(self->fd, device_id, 0x1, fault_fd))
895#define test_cmd_trigger_iopf_pasid(device_id, pasid, fault_fd) \
896 ASSERT_EQ(0, _test_cmd_trigger_iopf(self->fd, device_id, \
897 pasid, fault_fd))
884 898
885static int _test_cmd_viommu_alloc(int fd, __u32 device_id, __u32 hwpt_id,899static int _test_cmd_viommu_alloc(int fd, __u32 device_id, __u32 hwpt_id,
886 __u32 type, __u32 flags, __u32 *viommu_id)900 __u32 type, __u32 flags, __u32 *viommu_id)
@@ -936,3 +950,204 @@ static int _test_cmd_vdevice_alloc(int fd, __u32 viommu_id, __u32 idev_id, · 49 unmodified lines
936 EXPECT_ERRNO(_errno, \950 EXPECT_ERRNO(_errno, \
937 _test_cmd_vdevice_alloc(self->fd, viommu_id, idev_id, \951 _test_cmd_vdevice_alloc(self->fd, viommu_id, idev_id, \
938 virt_id, vdev_id))952 virt_id, vdev_id))
953
954static int _test_cmd_veventq_alloc(int fd, __u32 viommu_id, __u32 type,
955 __u32 *veventq_id, __u32 *veventq_fd)
956{
957 struct iommu_veventq_alloc cmd = {
958 .size = sizeof(cmd),
959 .type = type,
960 .veventq_depth = 2,
961 .viommu_id = viommu_id,
962 };
963 int ret;
964
965 ret = ioctl(fd, IOMMU_VEVENTQ_ALLOC, &cmd);
966 if (ret)
967 return ret;
968 if (veventq_id)
969 *veventq_id = cmd.out_veventq_id;
970 if (veventq_fd)
971 *veventq_fd = cmd.out_veventq_fd;
972 return 0;
973}
974
975#define test_cmd_veventq_alloc(viommu_id, type, veventq_id, veventq_fd) \
976 ASSERT_EQ(0, _test_cmd_veventq_alloc(self->fd, viommu_id, type, \
977 veventq_id, veventq_fd))
978#define test_err_veventq_alloc(_errno, viommu_id, type, veventq_id, \
979 veventq_fd) \
980 EXPECT_ERRNO(_errno, \
981 _test_cmd_veventq_alloc(self->fd, viommu_id, type, \
982 veventq_id, veventq_fd))
983
984static int _test_cmd_trigger_vevents(int fd, __u32 dev_id, __u32 nvevents)
985{
986 struct iommu_test_cmd trigger_vevent_cmd = {
987 .size = sizeof(trigger_vevent_cmd),
988 .op = IOMMU_TEST_OP_TRIGGER_VEVENT,
989 .trigger_vevent = {
990 .dev_id = dev_id,
991 },
992 };
993 int ret;
994
995 while (nvevents--) {
996 ret = ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_TRIGGER_VEVENT),
997 &trigger_vevent_cmd);
998 if (ret < 0)
999 return -1;
1000 }
1001 return ret;
1002}
1003
1004#define test_cmd_trigger_vevents(dev_id, nvevents) \
1005 ASSERT_EQ(0, _test_cmd_trigger_vevents(self->fd, dev_id, nvevents))
1006
1007static int _test_cmd_read_vevents(int fd, __u32 event_fd, __u32 nvevents,
1008 __u32 virt_id, int *prev_seq)
1009{
1010 struct pollfd pollfd = { .fd = event_fd, .events = POLLIN };
1011 struct iommu_viommu_event_selftest *event;
1012 struct iommufd_vevent_header *hdr;
1013 ssize_t bytes;
1014 void *data;
1015 int ret, i;
1016
1017 ret = poll(&pollfd, 1, 1000);
1018 if (ret < 0)
1019 return -1;
1020
1021 data = calloc(nvevents, sizeof(*hdr) + sizeof(*event));
1022 if (!data) {
1023 errno = ENOMEM;
1024 return -1;
1025 }
1026
1027 bytes = read(event_fd, data,
1028 nvevents * (sizeof(*hdr) + sizeof(*event)));
1029 if (bytes <= 0) {
1030 errno = EFAULT;
1031 ret = -1;
1032 goto out_free;
1033 }
1034
1035 for (i = 0; i < nvevents; i++) {
1036 hdr = data + i * (sizeof(*hdr) + sizeof(*event));
1037
1038 if (hdr->flags & IOMMU_VEVENTQ_FLAG_LOST_EVENTS ||
1039 hdr->sequence - *prev_seq > 1) {
1040 *prev_seq = hdr->sequence;
1041 errno = EOVERFLOW;
1042 ret = -1;
1043 goto out_free;
1044 }
1045 *prev_seq = hdr->sequence;
1046 event = data + sizeof(*hdr);
1047 if (event->virt_id != virt_id) {
1048 errno = EINVAL;
1049 ret = -1;
1050 goto out_free;
1051 }
1052 }
1053
1054 ret = 0;
1055out_free:
1056 free(data);
1057 return ret;
1058}
1059
1060#define test_cmd_read_vevents(event_fd, nvevents, virt_id, prev_seq) \
1061 ASSERT_EQ(0, _test_cmd_read_vevents(self->fd, event_fd, nvevents, \
1062 virt_id, prev_seq))
1063#define test_err_read_vevents(_errno, event_fd, nvevents, virt_id, prev_seq) \
1064 EXPECT_ERRNO(_errno, \
1065 _test_cmd_read_vevents(self->fd, event_fd, nvevents, \
1066 virt_id, prev_seq))
1067
1068static int _test_cmd_pasid_attach(int fd, __u32 stdev_id, __u32 pasid,
1069 __u32 pt_id)
1070{
1071 struct iommu_test_cmd test_attach = {
1072 .size = sizeof(test_attach),
1073 .op = IOMMU_TEST_OP_PASID_ATTACH,
1074 .id = stdev_id,
1075 .pasid_attach = {
1076 .pasid = pasid,
1077 .pt_id = pt_id,
1078 },
1079 };
1080
1081 return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_ATTACH),
1082 &test_attach);
1083}
1084
1085#define test_cmd_pasid_attach(pasid, hwpt_id) \
1086 ASSERT_EQ(0, _test_cmd_pasid_attach(self->fd, self->stdev_id, \
1087 pasid, hwpt_id))
1088
1089#define test_err_pasid_attach(_errno, pasid, hwpt_id) \
1090 EXPECT_ERRNO(_errno, \
1091 _test_cmd_pasid_attach(self->fd, self->stdev_id, \
1092 pasid, hwpt_id))
1093
1094static int _test_cmd_pasid_replace(int fd, __u32 stdev_id, __u32 pasid,
1095 __u32 pt_id)
1096{
1097 struct iommu_test_cmd test_replace = {
1098 .size = sizeof(test_replace),
1099 .op = IOMMU_TEST_OP_PASID_REPLACE,
1100 .id = stdev_id,
1101 .pasid_replace = {
1102 .pasid = pasid,
1103 .pt_id = pt_id,
1104 },
1105 };
1106
1107 return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_REPLACE),
1108 &test_replace);
1109}
1110
1111#define test_cmd_pasid_replace(pasid, hwpt_id) \
1112 ASSERT_EQ(0, _test_cmd_pasid_replace(self->fd, self->stdev_id, \
1113 pasid, hwpt_id))
1114
1115#define test_err_pasid_replace(_errno, pasid, hwpt_id) \
1116 EXPECT_ERRNO(_errno, \
1117 _test_cmd_pasid_replace(self->fd, self->stdev_id, \
1118 pasid, hwpt_id))
1119
1120static int _test_cmd_pasid_detach(int fd, __u32 stdev_id, __u32 pasid)
1121{
1122 struct iommu_test_cmd test_detach = {
1123 .size = sizeof(test_detach),
1124 .op = IOMMU_TEST_OP_PASID_DETACH,
1125 .id = stdev_id,
1126 .pasid_detach = {
1127 .pasid = pasid,
1128 },
1129 };
1130
1131 return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_DETACH),
1132 &test_detach);
1133}
1134
1135#define test_cmd_pasid_detach(pasid) \
1136 ASSERT_EQ(0, _test_cmd_pasid_detach(self->fd, self->stdev_id, pasid))
1137
1138static int test_cmd_pasid_check_hwpt(int fd, __u32 stdev_id, __u32 pasid,
1139 __u32 hwpt_id)
1140{
1141 struct iommu_test_cmd test_pasid_check = {
1142 .size = sizeof(test_pasid_check),
1143 .op = IOMMU_TEST_OP_PASID_CHECK_HWPT,
1144 .id = stdev_id,
1145 .pasid_check = {
1146 .pasid = pasid,
1147 .hwpt_id = hwpt_id,
1148 },
1149 };
1150
1151 return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_CHECK_HWPT),
1152 &test_pasid_check);
1153}

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t