Pull iommufd updates from Jason Gunthorpe: "Two significant new items: - Allow reporting IOMMU HW events to userspace when the events are clearly linked to a device. This is linked to the VIOMMU object and is intended to be used by a VMM to forward HW events to the virtual machine as part of emulating a vIOMMU. ARM SMMUv3 is the first driver to use this mechanism. Like the existing fault events the data is delivered through a simple FD returning event records on read(). - PASID support in VFIO. The "Process Address Space ID" is a PCI feature that allows the device to tag all PCI DMA operations with an ID. The IOMMU will then use the ID to select a unique translation for those DMAs. This is part of Intel's vIOMMU support as VT-D HW requires the hypervisor to manage each PASID entry. The support is generic so any VFIO user could attach any translation to a PASID, and the support should work on ARM SMMUv3 as well. AMD requires additional driver work. Some minor updates, along with fixes: - Prevent using nested parents with fault's, no driver support today - Put a single "cookie_type" value in the iommu_domain to indicate what owns the various opaque owner fields" * tag 'for-linus-iommufd' of git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd: (49 commits) iommufd: Test attach before detaching pasid iommufd: Fix iommu_vevent_header tables markup iommu: Convert unreachable() to BUG() iommufd: Balance veventq->num_events inc/dec iommufd: Initialize the flags of vevent in iommufd_viommu_report_event() iommufd/selftest: Add coverage for reporting max_pasid_log2 via IOMMU_HW_INFO iommufd: Extend IOMMU_GET_HW_INFO to report PASID capability vfio: VFIO_DEVICE_[AT|DE]TACH_IOMMUFD_PT support pasid vfio-iommufd: Support pasid [at|de]tach for physical VFIO devices ida: Add ida_find_first_range() iommufd/selftest: Add coverage for iommufd pasid attach/detach iommufd/selftest: Add test ops to test pasid attach/detach iommufd/selftest: Add a helper to get test device iommufd/selftest: Add set_dev_pasid in mock iommu iommufd: Allow allocating PASID-compatible domain iommu/vt-d: Add IOMMU_HWPT_ALLOC_PASID support iommufd: Enforce PASID-compatible domain for RID iommufd: Support pasid attach/replace iommufd: Enforce PASID-compatible domain in PASID path iommufd/device: Add pasid_attach array to track per-PASID attach ...
Unified · Split
| @@ -63,6 +63,13 @@ Following IOMMUFD objects are exposed to userspace: · 62 unmodified lines | |||
| 63 | space usually has mappings from guest-level I/O virtual addresses to guest- | 63 | space usually has mappings from guest-level I/O virtual addresses to guest- |
| 64 | level physical addresses. | 64 | level physical addresses. |
| 65 | 65 | ||
| 66 | - IOMMUFD_FAULT, representing a software queue for an HWPT reporting IO page | ||
| 67 | faults using the IOMMU HW's PRI (Page Request Interface). This queue object | ||
| 68 | provides user space an FD to poll the page fault events and also to respond | ||
| 69 | to those events. A FAULT object must be created first to get a fault_id that | ||
| 70 | could be then used to allocate a fault-enabled HWPT via the IOMMU_HWPT_ALLOC | ||
| 71 | command by setting the IOMMU_HWPT_FAULT_ID_VALID bit in its flags field. | ||
| 72 | |||
| 66 | - IOMMUFD_OBJ_VIOMMU, representing a slice of the physical IOMMU instance, | 73 | - IOMMUFD_OBJ_VIOMMU, representing a slice of the physical IOMMU instance, |
| 67 | passed to or shared with a VM. It may be some HW-accelerated virtualization | 74 | passed to or shared with a VM. It may be some HW-accelerated virtualization |
| 68 | features and some SW resources used by the VM. For examples: | 75 | features and some SW resources used by the VM. For examples: |
| @@ -109,6 +116,14 @@ Following IOMMUFD objects are exposed to userspace: · 40 unmodified lines | |||
| 109 | vIOMMU, which is a separate ioctl call from attaching the same device to an | 116 | vIOMMU, which is a separate ioctl call from attaching the same device to an |
| 110 | HWPT_PAGING that the vIOMMU holds. | 117 | HWPT_PAGING that the vIOMMU holds. |
| 111 | 118 | ||
| 119 | - IOMMUFD_OBJ_VEVENTQ, representing a software queue for a vIOMMU to report its | ||
| 120 | events such as translation faults occurred to a nested stage-1 (excluding I/O | ||
| 121 | page faults that should go through IOMMUFD_OBJ_FAULT) and HW-specific events. | ||
| 122 | This queue object provides user space an FD to poll/read the vIOMMU events. A | ||
| 123 | vIOMMU object must be created first to get its viommu_id, which could be then | ||
| 124 | used to allocate a vEVENTQ. Each vIOMMU can support multiple types of vEVENTS, | ||
| 125 | but is confined to one vEVENTQ per vEVENTQ type. | ||
| 126 | |||
| 112 | All user-visible objects are destroyed via the IOMMU_DESTROY uAPI. | 127 | All user-visible objects are destroyed via the IOMMU_DESTROY uAPI. |
| 113 | 128 | ||
| 114 | The diagrams below show relationships between user-visible objects and kernel | 129 | The diagrams below show relationships between user-visible objects and kernel |
| @@ -251,8 +266,10 @@ User visible objects are backed by following datastructures: · 136 unmodified lines | |||
| 251 | - iommufd_device for IOMMUFD_OBJ_DEVICE. | 266 | - iommufd_device for IOMMUFD_OBJ_DEVICE. |
| 252 | - iommufd_hwpt_paging for IOMMUFD_OBJ_HWPT_PAGING. | 267 | - iommufd_hwpt_paging for IOMMUFD_OBJ_HWPT_PAGING. |
| 253 | - iommufd_hwpt_nested for IOMMUFD_OBJ_HWPT_NESTED. | 268 | - iommufd_hwpt_nested for IOMMUFD_OBJ_HWPT_NESTED. |
| 269 | - iommufd_fault for IOMMUFD_OBJ_FAULT. | ||
| 254 | - iommufd_viommu for IOMMUFD_OBJ_VIOMMU. | 270 | - iommufd_viommu for IOMMUFD_OBJ_VIOMMU. |
| 255 | - iommufd_vdevice for IOMMUFD_OBJ_VDEVICE. | 271 | - iommufd_vdevice for IOMMUFD_OBJ_VDEVICE. |
| 272 | - iommufd_veventq for IOMMUFD_OBJ_VEVENTQ. | ||
| 256 | 273 | ||
| 257 | Several terminologies when looking at these datastructures: | 274 | Several terminologies when looking at these datastructures: |
| 258 | 275 | ||
| @@ -43,6 +43,8 @@ static void arm_smmu_make_nested_cd_table_ste( · 42 unmodified lines | |||
| 43 | target->data[0] |= nested_domain->ste[0] & | 43 | target->data[0] |= nested_domain->ste[0] & |
| 44 | ~cpu_to_le64(STRTAB_STE_0_CFG); | 44 | ~cpu_to_le64(STRTAB_STE_0_CFG); |
| 45 | target->data[1] |= nested_domain->ste[1]; | 45 | target->data[1] |= nested_domain->ste[1]; |
| 46 | /* Merge events for DoS mitigations on eventq */ | ||
| 47 | target->data[1] |= cpu_to_le64(STRTAB_STE_1_MEV); | ||
| 46 | } | 48 | } |
| 47 | 49 | ||
| 48 | /* | 50 | /* |
| @@ -85,6 +87,47 @@ static void arm_smmu_make_nested_domain_ste( · 36 unmodified lines | |||
| 85 | } | 87 | } |
| 86 | } | 88 | } |
| 87 | 89 | ||
| 90 | int arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state, | ||
| 91 | struct arm_smmu_nested_domain *nested_domain) | ||
| 92 | { | ||
| 93 | struct arm_smmu_vmaster *vmaster; | ||
| 94 | unsigned long vsid; | ||
| 95 | int ret; | ||
| 96 | |||
| 97 | iommu_group_mutex_assert(state->master->dev); | ||
| 98 | |||
| 99 | ret = iommufd_viommu_get_vdev_id(&nested_domain->vsmmu->core, | ||
| 100 | state->master->dev, &vsid); | ||
| 101 | if (ret) | ||
| 102 | return ret; | ||
| 103 | |||
| 104 | vmaster = kzalloc(sizeof(*vmaster), GFP_KERNEL); | ||
| 105 | if (!vmaster) | ||
| 106 | return -ENOMEM; | ||
| 107 | vmaster->vsmmu = nested_domain->vsmmu; | ||
| 108 | vmaster->vsid = vsid; | ||
| 109 | state->vmaster = vmaster; | ||
| 110 | |||
| 111 | return 0; | ||
| 112 | } | ||
| 113 | |||
| 114 | void arm_smmu_attach_commit_vmaster(struct arm_smmu_attach_state *state) | ||
| 115 | { | ||
| 116 | struct arm_smmu_master *master = state->master; | ||
| 117 | |||
| 118 | mutex_lock(&master->smmu->streams_mutex); | ||
| 119 | kfree(master->vmaster); | ||
| 120 | master->vmaster = state->vmaster; | ||
| 121 | mutex_unlock(&master->smmu->streams_mutex); | ||
| 122 | } | ||
| 123 | |||
| 124 | void arm_smmu_master_clear_vmaster(struct arm_smmu_master *master) | ||
| 125 | { | ||
| 126 | struct arm_smmu_attach_state state = { .master = master }; | ||
| 127 | |||
| 128 | arm_smmu_attach_commit_vmaster(&state); | ||
| 129 | } | ||
| 130 | |||
| 88 | static int arm_smmu_attach_dev_nested(struct iommu_domain *domain, | 131 | static int arm_smmu_attach_dev_nested(struct iommu_domain *domain, |
| 89 | struct device *dev) | 132 | struct device *dev) |
| 90 | { | 133 | { |
| @@ -392,4 +435,21 @@ struct iommufd_viommu *arm_vsmmu_alloc(struct device *dev, · 301 unmodified lines | |||
| 392 | return &vsmmu->core; | 435 | return &vsmmu->core; |
| 393 | } | 436 | } |
| 394 | 437 | ||
| 438 | int arm_vmaster_report_event(struct arm_smmu_vmaster *vmaster, u64 *evt) | ||
| 439 | { | ||
| 440 | struct iommu_vevent_arm_smmuv3 vevt; | ||
| 441 | int i; | ||
| 442 | |||
| 443 | lockdep_assert_held(&vmaster->vsmmu->smmu->streams_mutex); | ||
| 444 | |||
| 445 | vevt.evt[0] = cpu_to_le64((evt[0] & ~EVTQ_0_SID) | | ||
| 446 | FIELD_PREP(EVTQ_0_SID, vmaster->vsid)); | ||
| 447 | for (i = 1; i < EVTQ_ENT_DWORDS; i++) | ||
| 448 | vevt.evt[i] = cpu_to_le64(evt[i]); | ||
| 449 | |||
| 450 | return iommufd_viommu_report_event(&vmaster->vsmmu->core, | ||
| 451 | IOMMU_VEVENTQ_TYPE_ARM_SMMUV3, &vevt, | ||
| 452 | sizeof(vevt)); | ||
| 453 | } | ||
| 454 | |||
| 395 | MODULE_IMPORT_NS("IOMMUFD"); | 455 | MODULE_IMPORT_NS("IOMMUFD"); |
| @@ -1052,7 +1052,7 @@ void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) · 1051 unmodified lines | |||
| 1052 | cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR | | 1052 | cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR | |
| 1053 | STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH | | 1053 | STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH | |
| 1054 | STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW | | 1054 | STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW | |
| 1055 | STRTAB_STE_1_EATS); | 1055 | STRTAB_STE_1_EATS | STRTAB_STE_1_MEV); |
| 1056 | used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID); | 1056 | used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID); |
| 1057 | 1057 | ||
| 1058 | /* | 1058 | /* |
| @@ -1068,7 +1068,7 @@ void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) · 9 unmodified lines | |||
| 1068 | if (cfg & BIT(1)) { | 1068 | if (cfg & BIT(1)) { |
| 1069 | used_bits[1] |= | 1069 | used_bits[1] |= |
| 1070 | cpu_to_le64(STRTAB_STE_1_S2FWB | STRTAB_STE_1_EATS | | 1070 | cpu_to_le64(STRTAB_STE_1_S2FWB | STRTAB_STE_1_EATS | |
| 1071 | STRTAB_STE_1_SHCFG); | 1071 | STRTAB_STE_1_SHCFG | STRTAB_STE_1_MEV); |
| 1072 | used_bits[2] |= | 1072 | used_bits[2] |= |
| 1073 | cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR | | 1073 | cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR | |
| 1074 | STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI | | 1074 | STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI | |
| @@ -1813,8 +1813,8 @@ static void arm_smmu_decode_event(struct arm_smmu_device *smmu, u64 *raw, · 738 unmodified lines | |||
| 1813 | mutex_unlock(&smmu->streams_mutex); | 1813 | mutex_unlock(&smmu->streams_mutex); |
| 1814 | } | 1814 | } |
| 1815 | 1815 | ||
| 1816 | static int arm_smmu_handle_event(struct arm_smmu_device *smmu, | 1816 | static int arm_smmu_handle_event(struct arm_smmu_device *smmu, u64 *evt, |
| 1817 | struct arm_smmu_event *event) | 1817 | struct arm_smmu_event *event) |
| 1818 | { | 1818 | { |
| 1819 | int ret = 0; | 1819 | int ret = 0; |
| 1820 | u32 perm = 0; | 1820 | u32 perm = 0; |
| @@ -1823,6 +1823,10 @@ static int arm_smmu_handle_event(struct arm_smmu_device *smmu, · 2 unmodified lines | |||
| 1823 | struct iommu_fault *flt = &fault_evt.fault; | 1823 | struct iommu_fault *flt = &fault_evt.fault; |
| 1824 | 1824 | ||
| 1825 | switch (event->id) { | 1825 | switch (event->id) { |
| 1826 | case EVT_ID_BAD_STE_CONFIG: | ||
| 1827 | case EVT_ID_STREAM_DISABLED_FAULT: | ||
| 1828 | case EVT_ID_BAD_SUBSTREAMID_CONFIG: | ||
| 1829 | case EVT_ID_BAD_CD_CONFIG: | ||
| 1826 | case EVT_ID_TRANSLATION_FAULT: | 1830 | case EVT_ID_TRANSLATION_FAULT: |
| 1827 | case EVT_ID_ADDR_SIZE_FAULT: | 1831 | case EVT_ID_ADDR_SIZE_FAULT: |
| 1828 | case EVT_ID_ACCESS_FAULT: | 1832 | case EVT_ID_ACCESS_FAULT: |
| @@ -1832,31 +1836,30 @@ static int arm_smmu_handle_event(struct arm_smmu_device *smmu, · 3 unmodified lines | |||
| 1832 | return -EOPNOTSUPP; | 1836 | return -EOPNOTSUPP; |
| 1833 | } | 1837 | } |
| 1834 | 1838 | ||
| 1835 | if (!event->stall) | 1839 | if (event->stall) { |
| 1836 | return -EOPNOTSUPP; | 1840 | if (event->read) |
| 1837 | 1841 | perm |= IOMMU_FAULT_PERM_READ; | |
| 1838 | if (event->read) | 1842 | else |
| 1839 | perm |= IOMMU_FAULT_PERM_READ; | 1843 | perm |= IOMMU_FAULT_PERM_WRITE; |
| 1840 | else | ||
| 1841 | perm |= IOMMU_FAULT_PERM_WRITE; | ||
| 1842 | 1844 | ||
| 1843 | if (event->instruction) | 1845 | if (event->instruction) |
| 1844 | perm |= IOMMU_FAULT_PERM_EXEC; | 1846 | perm |= IOMMU_FAULT_PERM_EXEC; |
| 1845 | 1847 | ||
| 1846 | if (event->privileged) | 1848 | if (event->privileged) |
| 1847 | perm |= IOMMU_FAULT_PERM_PRIV; | 1849 | perm |= IOMMU_FAULT_PERM_PRIV; |
| 1848 | 1850 | ||
| 1849 | flt->type = IOMMU_FAULT_PAGE_REQ; | 1851 | flt->type = IOMMU_FAULT_PAGE_REQ; |
| 1850 | flt->prm = (struct iommu_fault_page_request) { | 1852 | flt->prm = (struct iommu_fault_page_request){ |
| 1851 | .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE, | 1853 | .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE, |
| 1852 | .grpid = event->stag, | 1854 | .grpid = event->stag, |
| 1853 | .perm = perm, | 1855 | .perm = perm, |
| 1854 | .addr = event->iova, | 1856 | .addr = event->iova, |
| 1855 | }; | 1857 | }; |
| 1856 | 1858 | ||
| 1857 | if (event->ssv) { | 1859 | if (event->ssv) { |
| 1858 | flt->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; | 1860 | flt->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; |
| 1859 | flt->prm.pasid = event->ssid; | 1861 | flt->prm.pasid = event->ssid; |
| 1862 | } | ||
| 1860 | } | 1863 | } |
| 1861 | 1864 | ||
| 1862 | mutex_lock(&smmu->streams_mutex); | 1865 | mutex_lock(&smmu->streams_mutex); |
| @@ -1866,7 +1869,12 @@ static int arm_smmu_handle_event(struct arm_smmu_device *smmu, · 3 unmodified lines | |||
| 1866 | goto out_unlock; | 1869 | goto out_unlock; |
| 1867 | } | 1870 | } |
| 1868 | 1871 | ||
| 1869 | ret = iommu_report_device_fault(master->dev, &fault_evt); | 1872 | if (event->stall) |
| 1873 | ret = iommu_report_device_fault(master->dev, &fault_evt); | ||
| 1874 | else if (master->vmaster && !event->s2) | ||
| 1875 | ret = arm_vmaster_report_event(master->vmaster, evt); | ||
| 1876 | else | ||
| 1877 | ret = -EOPNOTSUPP; /* Unhandled events should be pinned */ | ||
| 1870 | out_unlock: | 1878 | out_unlock: |
| 1871 | mutex_unlock(&smmu->streams_mutex); | 1879 | mutex_unlock(&smmu->streams_mutex); |
| 1872 | return ret; | 1880 | return ret; |
| @@ -1944,7 +1952,7 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) · 71 unmodified lines | |||
| 1944 | do { | 1952 | do { |
| 1945 | while (!queue_remove_raw(q, evt)) { | 1953 | while (!queue_remove_raw(q, evt)) { |
| 1946 | arm_smmu_decode_event(smmu, evt, &event); | 1954 | arm_smmu_decode_event(smmu, evt, &event); |
| 1947 | if (arm_smmu_handle_event(smmu, &event)) | 1955 | if (arm_smmu_handle_event(smmu, evt, &event)) |
| 1948 | arm_smmu_dump_event(smmu, evt, &event, &rs); | 1956 | arm_smmu_dump_event(smmu, evt, &event, &rs); |
| 1949 | 1957 | ||
| 1950 | put_device(event.dev); | 1958 | put_device(event.dev); |
| @@ -2803,6 +2811,7 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state, · 852 unmodified lines | |||
| 2803 | struct arm_smmu_domain *smmu_domain = | 2811 | struct arm_smmu_domain *smmu_domain = |
| 2804 | to_smmu_domain_devices(new_domain); | 2812 | to_smmu_domain_devices(new_domain); |
| 2805 | unsigned long flags; | 2813 | unsigned long flags; |
| 2814 | int ret; | ||
| 2806 | 2815 | ||
| 2807 | /* | 2816 | /* |
| 2808 | * arm_smmu_share_asid() must not see two domains pointing to the same | 2817 | * arm_smmu_share_asid() must not see two domains pointing to the same |
| @@ -2832,9 +2841,18 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state, · 23 unmodified lines | |||
| 2832 | } | 2841 | } |
| 2833 | 2842 | ||
| 2834 | if (smmu_domain) { | 2843 | if (smmu_domain) { |
| 2844 | if (new_domain->type == IOMMU_DOMAIN_NESTED) { | ||
| 2845 | ret = arm_smmu_attach_prepare_vmaster( | ||
| 2846 | state, to_smmu_nested_domain(new_domain)); | ||
| 2847 | if (ret) | ||
| 2848 | return ret; | ||
| 2849 | } | ||
| 2850 | |||
| 2835 | master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL); | 2851 | master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL); |
| 2836 | if (!master_domain) | 2852 | if (!master_domain) { |
| 2853 | kfree(state->vmaster); | ||
| 2837 | return -ENOMEM; | 2854 | return -ENOMEM; |
| 2855 | } | ||
| 2838 | master_domain->master = master; | 2856 | master_domain->master = master; |
| 2839 | master_domain->ssid = state->ssid; | 2857 | master_domain->ssid = state->ssid; |
| 2840 | if (new_domain->type == IOMMU_DOMAIN_NESTED) | 2858 | if (new_domain->type == IOMMU_DOMAIN_NESTED) |
| @@ -2861,6 +2879,7 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state, · 20 unmodified lines | |||
| 2861 | spin_unlock_irqrestore(&smmu_domain->devices_lock, | 2879 | spin_unlock_irqrestore(&smmu_domain->devices_lock, |
| 2862 | flags); | 2880 | flags); |
| 2863 | kfree(master_domain); | 2881 | kfree(master_domain); |
| 2882 | kfree(state->vmaster); | ||
| 2864 | return -EINVAL; | 2883 | return -EINVAL; |
| 2865 | } | 2884 | } |
| 2866 | 2885 | ||
| @@ -2893,6 +2912,8 @@ void arm_smmu_attach_commit(struct arm_smmu_attach_state *state) · 26 unmodified lines | |||
| 2893 | 2912 | ||
| 2894 | lockdep_assert_held(&arm_smmu_asid_lock); | 2913 | lockdep_assert_held(&arm_smmu_asid_lock); |
| 2895 | 2914 | ||
| 2915 | arm_smmu_attach_commit_vmaster(state); | ||
| 2916 | |||
| 2896 | if (state->ats_enabled && !master->ats_enabled) { | 2917 | if (state->ats_enabled && !master->ats_enabled) { |
| 2897 | arm_smmu_enable_ats(master); | 2918 | arm_smmu_enable_ats(master); |
| 2898 | } else if (state->ats_enabled && master->ats_enabled) { | 2919 | } else if (state->ats_enabled && master->ats_enabled) { |
| @@ -3162,6 +3183,7 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain, · 263 unmodified lines | |||
| 3162 | struct arm_smmu_ste ste; | 3183 | struct arm_smmu_ste ste; |
| 3163 | struct arm_smmu_master *master = dev_iommu_priv_get(dev); | 3184 | struct arm_smmu_master *master = dev_iommu_priv_get(dev); |
| 3164 | 3185 | ||
| 3186 | arm_smmu_master_clear_vmaster(master); | ||
| 3165 | arm_smmu_make_bypass_ste(master->smmu, &ste); | 3187 | arm_smmu_make_bypass_ste(master->smmu, &ste); |
| 3166 | arm_smmu_attach_dev_ste(domain, dev, &ste, STRTAB_STE_1_S1DSS_BYPASS); | 3188 | arm_smmu_attach_dev_ste(domain, dev, &ste, STRTAB_STE_1_S1DSS_BYPASS); |
| 3167 | return 0; | 3189 | return 0; |
| @@ -3180,7 +3202,9 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain, · 12 unmodified lines | |||
| 3180 | struct device *dev) | 3202 | struct device *dev) |
| 3181 | { | 3203 | { |
| 3182 | struct arm_smmu_ste ste; | 3204 | struct arm_smmu_ste ste; |
| 3205 | struct arm_smmu_master *master = dev_iommu_priv_get(dev); | ||
| 3183 | 3206 | ||
| 3207 | arm_smmu_master_clear_vmaster(master); | ||
| 3184 | arm_smmu_make_abort_ste(&ste); | 3208 | arm_smmu_make_abort_ste(&ste); |
| 3185 | arm_smmu_attach_dev_ste(domain, dev, &ste, | 3209 | arm_smmu_attach_dev_ste(domain, dev, &ste, |
| 3186 | STRTAB_STE_1_S1DSS_TERMINATE); | 3210 | STRTAB_STE_1_S1DSS_TERMINATE); |
| @@ -266,6 +266,7 @@ static inline u32 arm_smmu_strtab_l2_idx(u32 sid) · 265 unmodified lines | |||
| 266 | #define STRTAB_STE_1_S1COR GENMASK_ULL(5, 4) | 266 | #define STRTAB_STE_1_S1COR GENMASK_ULL(5, 4) |
| 267 | #define STRTAB_STE_1_S1CSH GENMASK_ULL(7, 6) | 267 | #define STRTAB_STE_1_S1CSH GENMASK_ULL(7, 6) |
| 268 | 268 | ||
| 269 | #define STRTAB_STE_1_MEV (1UL << 19) | ||
| 269 | #define STRTAB_STE_1_S2FWB (1UL << 25) | 270 | #define STRTAB_STE_1_S2FWB (1UL << 25) |
| 270 | #define STRTAB_STE_1_S1STALLD (1UL << 27) | 271 | #define STRTAB_STE_1_S1STALLD (1UL << 27) |
| 271 | 272 | ||
| @@ -799,6 +800,11 @@ struct arm_smmu_stream { · 527 unmodified lines | |||
| 799 | struct rb_node node; | 800 | struct rb_node node; |
| 800 | }; | 801 | }; |
| 801 | 802 | ||
| 803 | struct arm_smmu_vmaster { | ||
| 804 | struct arm_vsmmu *vsmmu; | ||
| 805 | unsigned long vsid; | ||
| 806 | }; | ||
| 807 | |||
| 802 | struct arm_smmu_event { | 808 | struct arm_smmu_event { |
| 803 | u8 stall : 1, | 809 | u8 stall : 1, |
| 804 | ssv : 1, | 810 | ssv : 1, |
| @@ -824,6 +830,7 @@ struct arm_smmu_master { · 19 unmodified lines | |||
| 824 | struct arm_smmu_device *smmu; | 830 | struct arm_smmu_device *smmu; |
| 825 | struct device *dev; | 831 | struct device *dev; |
| 826 | struct arm_smmu_stream *streams; | 832 | struct arm_smmu_stream *streams; |
| 833 | struct arm_smmu_vmaster *vmaster; /* use smmu->streams_mutex */ | ||
| 827 | /* Locked by the iommu core using the group mutex */ | 834 | /* Locked by the iommu core using the group mutex */ |
| 828 | struct arm_smmu_ctx_desc_cfg cd_table; | 835 | struct arm_smmu_ctx_desc_cfg cd_table; |
| 829 | unsigned int num_streams; | 836 | unsigned int num_streams; |
| @@ -972,6 +979,7 @@ struct arm_smmu_attach_state { · 142 unmodified lines | |||
| 972 | bool disable_ats; | 979 | bool disable_ats; |
| 973 | ioasid_t ssid; | 980 | ioasid_t ssid; |
| 974 | /* Resulting state */ | 981 | /* Resulting state */ |
| 982 | struct arm_smmu_vmaster *vmaster; | ||
| 975 | bool ats_enabled; | 983 | bool ats_enabled; |
| 976 | }; | 984 | }; |
| 977 | 985 | ||
| @@ -1055,9 +1063,37 @@ struct iommufd_viommu *arm_vsmmu_alloc(struct device *dev, · 77 unmodified lines | |||
| 1055 | struct iommu_domain *parent, | 1063 | struct iommu_domain *parent, |
| 1056 | struct iommufd_ctx *ictx, | 1064 | struct iommufd_ctx *ictx, |
| 1057 | unsigned int viommu_type); | 1065 | unsigned int viommu_type); |
| 1066 | int arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state, | ||
| 1067 | struct arm_smmu_nested_domain *nested_domain); | ||
| 1068 | void arm_smmu_attach_commit_vmaster(struct arm_smmu_attach_state *state); | ||
| 1069 | void arm_smmu_master_clear_vmaster(struct arm_smmu_master *master); | ||
| 1070 | int arm_vmaster_report_event(struct arm_smmu_vmaster *vmaster, u64 *evt); | ||
| 1058 | #else | 1071 | #else |
| 1059 | #define arm_smmu_hw_info NULL | 1072 | #define arm_smmu_hw_info NULL |
| 1060 | #define arm_vsmmu_alloc NULL | 1073 | #define arm_vsmmu_alloc NULL |
| 1074 | |||
| 1075 | static inline int | ||
| 1076 | arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state, | ||
| 1077 | struct arm_smmu_nested_domain *nested_domain) | ||
| 1078 | { | ||
| 1079 | return 0; | ||
| 1080 | } | ||
| 1081 | |||
| 1082 | static inline void | ||
| 1083 | arm_smmu_attach_commit_vmaster(struct arm_smmu_attach_state *state) | ||
| 1084 | { | ||
| 1085 | } | ||
| 1086 | |||
| 1087 | static inline void | ||
| 1088 | arm_smmu_master_clear_vmaster(struct arm_smmu_master *master) | ||
| 1089 | { | ||
| 1090 | } | ||
| 1091 | |||
| 1092 | static inline int arm_vmaster_report_event(struct arm_smmu_vmaster *vmaster, | ||
| 1093 | u64 *evt) | ||
| 1094 | { | ||
| 1095 | return -EOPNOTSUPP; | ||
| 1096 | } | ||
| 1061 | #endif /* CONFIG_ARM_SMMU_V3_IOMMUFD */ | 1097 | #endif /* CONFIG_ARM_SMMU_V3_IOMMUFD */ |
| 1062 | 1098 | ||
| 1063 | #endif /* _ARM_SMMU_V3_H */ | 1099 | #endif /* _ARM_SMMU_V3_H */ |
| @@ -42,11 +42,6 @@ struct iommu_dma_msi_page { · 41 unmodified lines | |||
| 42 | phys_addr_t phys; | 42 | phys_addr_t phys; |
| 43 | }; | 43 | }; |
| 44 | 44 | ||
| 45 | enum iommu_dma_cookie_type { | ||
| 46 | IOMMU_DMA_IOVA_COOKIE, | ||
| 47 | IOMMU_DMA_MSI_COOKIE, | ||
| 48 | }; | ||
| 49 | |||
| 50 | enum iommu_dma_queue_type { | 45 | enum iommu_dma_queue_type { |
| 51 | IOMMU_DMA_OPTS_PER_CPU_QUEUE, | 46 | IOMMU_DMA_OPTS_PER_CPU_QUEUE, |
| 52 | IOMMU_DMA_OPTS_SINGLE_QUEUE, | 47 | IOMMU_DMA_OPTS_SINGLE_QUEUE, |
| @@ -59,34 +54,30 @@ struct iommu_dma_options { · 6 unmodified lines | |||
| 59 | }; | 54 | }; |
| 60 | 55 | ||
| 61 | struct iommu_dma_cookie { | 56 | struct iommu_dma_cookie { |
| 62 | enum iommu_dma_cookie_type type; | 57 | struct iova_domain iovad; |
| 58 | struct list_head msi_page_list; | ||
| 59 | /* Flush queue */ | ||
| 63 | union { | 60 | union { |
| 64 | /* Full allocator for IOMMU_DMA_IOVA_COOKIE */ | 61 | struct iova_fq *single_fq; |
| 65 | struct { | 62 | struct iova_fq __percpu *percpu_fq; |
| 66 | struct iova_domain iovad; | ||
| 67 | /* Flush queue */ | ||
| 68 | union { | ||
| 69 | struct iova_fq *single_fq; | ||
| 70 | struct iova_fq __percpu *percpu_fq; | ||
| 71 | }; | ||
| 72 | /* Number of TLB flushes that have been started */ | ||
| 73 | atomic64_t fq_flush_start_cnt; | ||
| 74 | /* Number of TLB flushes that have been finished */ | ||
| 75 | atomic64_t fq_flush_finish_cnt; | ||
| 76 | /* Timer to regularily empty the flush queues */ | ||
| 77 | struct timer_list fq_timer; | ||
| 78 | /* 1 when timer is active, 0 when not */ | ||
| 79 | atomic_t fq_timer_on; | ||
| 80 | }; | ||
| 81 | /* Trivial linear page allocator for IOMMU_DMA_MSI_COOKIE */ | ||
| 82 | dma_addr_t msi_iova; | ||
| 83 | }; | 63 | }; |
| 84 | struct list_head msi_page_list; | 64 | /* Number of TLB flushes that have been started */ |
| 85 | 65 | atomic64_t fq_flush_start_cnt; | |
| 66 | /* Number of TLB flushes that have been finished */ | ||
| 67 | atomic64_t fq_flush_finish_cnt; | ||
| 68 | /* Timer to regularily empty the flush queues */ | ||
| 69 | struct timer_list fq_timer; | ||
| 70 | /* 1 when timer is active, 0 when not */ | ||
| 71 | atomic_t fq_timer_on; | ||
| 86 | /* Domain for flush queue callback; NULL if flush queue not in use */ | 72 | /* Domain for flush queue callback; NULL if flush queue not in use */ |
| 87 | struct iommu_domain *fq_domain; | 73 | struct iommu_domain *fq_domain; |
| 88 | /* Options for dma-iommu use */ | 74 | /* Options for dma-iommu use */ |
| 89 | struct iommu_dma_options options; | 75 | struct iommu_dma_options options; |
| 76 | }; | ||
| 77 | |||
| 78 | struct iommu_dma_msi_cookie { | ||
| 79 | dma_addr_t msi_iova; | ||
| 80 | struct list_head msi_page_list; | ||
| 90 | }; | 81 | }; |
| 91 | 82 | ||
| 92 | static DEFINE_STATIC_KEY_FALSE(iommu_deferred_attach_enabled); | 83 | static DEFINE_STATIC_KEY_FALSE(iommu_deferred_attach_enabled); |
| @@ -102,9 +93,6 @@ static int __init iommu_dma_forcedac_setup(char *str) · 9 unmodified lines | |||
| 102 | } | 93 | } |
| 103 | early_param("iommu.forcedac", iommu_dma_forcedac_setup); | 94 | early_param("iommu.forcedac", iommu_dma_forcedac_setup); |
| 104 | 95 | ||
| 105 | static int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | ||
| 106 | phys_addr_t msi_addr); | ||
| 107 | |||
| 108 | /* Number of entries per flush queue */ | 96 | /* Number of entries per flush queue */ |
| 109 | #define IOVA_DEFAULT_FQ_SIZE 256 | 97 | #define IOVA_DEFAULT_FQ_SIZE 256 |
| 110 | #define IOVA_SINGLE_FQ_SIZE 32768 | 98 | #define IOVA_SINGLE_FQ_SIZE 32768 |
| @@ -368,39 +356,24 @@ int iommu_dma_init_fq(struct iommu_domain *domain) · 257 unmodified lines | |||
| 368 | return 0; | 356 | return 0; |
| 369 | } | 357 | } |
| 370 | 358 | ||
| 371 | static inline size_t cookie_msi_granule(struct iommu_dma_cookie *cookie) | ||
| 372 | { | ||
| 373 | if (cookie->type == IOMMU_DMA_IOVA_COOKIE) | ||
| 374 | return cookie->iovad.granule; | ||
| 375 | return PAGE_SIZE; | ||
| 376 | } | ||
| 377 | |||
| 378 | static struct iommu_dma_cookie *cookie_alloc(enum iommu_dma_cookie_type type) | ||
| 379 | { | ||
| 380 | struct iommu_dma_cookie *cookie; | ||
| 381 | |||
| 382 | cookie = kzalloc(sizeof(*cookie), GFP_KERNEL); | ||
| 383 | if (cookie) { | ||
| 384 | INIT_LIST_HEAD(&cookie->msi_page_list); | ||
| 385 | cookie->type = type; | ||
| 386 | } | ||
| 387 | return cookie; | ||
| 388 | } | ||
| 389 | |||
| 390 | /** | 359 | /** |
| 391 | * iommu_get_dma_cookie - Acquire DMA-API resources for a domain | 360 | * iommu_get_dma_cookie - Acquire DMA-API resources for a domain |
| 392 | * @domain: IOMMU domain to prepare for DMA-API usage | 361 | * @domain: IOMMU domain to prepare for DMA-API usage |
| 393 | */ | 362 | */ |
| 394 | int iommu_get_dma_cookie(struct iommu_domain *domain) | 363 | int iommu_get_dma_cookie(struct iommu_domain *domain) |
| 395 | { | 364 | { |
| 396 | if (domain->iova_cookie) | 365 | struct iommu_dma_cookie *cookie; |
| 366 | |||
| 367 | if (domain->cookie_type != IOMMU_COOKIE_NONE) | ||
| 397 | return -EEXIST; | 368 | return -EEXIST; |
| 398 | 369 | ||
| 399 | domain->iova_cookie = cookie_alloc(IOMMU_DMA_IOVA_COOKIE); | 370 | cookie = kzalloc(sizeof(*cookie), GFP_KERNEL); |
| 400 | if (!domain->iova_cookie) | 371 | if (!cookie) |
| 401 | return -ENOMEM; | 372 | return -ENOMEM; |
| 402 | 373 | ||
| 403 | iommu_domain_set_sw_msi(domain, iommu_dma_sw_msi); | 374 | INIT_LIST_HEAD(&cookie->msi_page_list); |
| 375 | domain->cookie_type = IOMMU_COOKIE_DMA_IOVA; | ||
| 376 | domain->iova_cookie = cookie; | ||
| 404 | return 0; | 377 | return 0; |
| 405 | } | 378 | } |
| 406 | 379 | ||
| @@ -418,54 +391,56 @@ int iommu_get_dma_cookie(struct iommu_domain *domain) · 11 unmodified lines | |||
| 418 | */ | 391 | */ |
| 419 | int iommu_get_msi_cookie(struct iommu_domain *domain, dma_addr_t base) | 392 | int iommu_get_msi_cookie(struct iommu_domain *domain, dma_addr_t base) |
| 420 | { | 393 | { |
| 421 | struct iommu_dma_cookie *cookie; | 394 | struct iommu_dma_msi_cookie *cookie; |
| 422 | 395 | ||
| 423 | if (domain->type != IOMMU_DOMAIN_UNMANAGED) | 396 | if (domain->type != IOMMU_DOMAIN_UNMANAGED) |
| 424 | return -EINVAL; | 397 | return -EINVAL; |
| 425 | 398 | ||
| 426 | if (domain->iova_cookie) | 399 | if (domain->cookie_type != IOMMU_COOKIE_NONE) |
| 427 | return -EEXIST; | 400 | return -EEXIST; |
| 428 | 401 | ||
| 429 | cookie = cookie_alloc(IOMMU_DMA_MSI_COOKIE); | 402 | cookie = kzalloc(sizeof(*cookie), GFP_KERNEL); |
| 430 | if (!cookie) | 403 | if (!cookie) |
| 431 | return -ENOMEM; | 404 | return -ENOMEM; |
| 432 | 405 | ||
| 433 | cookie->msi_iova = base; | 406 | cookie->msi_iova = base; |
| 434 | domain->iova_cookie = cookie; | 407 | INIT_LIST_HEAD(&cookie->msi_page_list); |
| 435 | iommu_domain_set_sw_msi(domain, iommu_dma_sw_msi); | 408 | domain->cookie_type = IOMMU_COOKIE_DMA_MSI; |
| 409 | domain->msi_cookie = cookie; | ||
| 436 | return 0; | 410 | return 0; |
| 437 | } | 411 | } |
| 438 | EXPORT_SYMBOL(iommu_get_msi_cookie); | 412 | EXPORT_SYMBOL(iommu_get_msi_cookie); |
| 439 | 413 | ||
| 440 | /** | 414 | /** |
| 441 | * iommu_put_dma_cookie - Release a domain's DMA mapping resources | 415 | * iommu_put_dma_cookie - Release a domain's DMA mapping resources |
| 442 | * @domain: IOMMU domain previously prepared by iommu_get_dma_cookie() or | 416 | * @domain: IOMMU domain previously prepared by iommu_get_dma_cookie() |
| 443 | * iommu_get_msi_cookie() | ||
| 444 | */ | 417 | */ |
| 445 | void iommu_put_dma_cookie(struct iommu_domain *domain) | 418 | void iommu_put_dma_cookie(struct iommu_domain *domain) |
| 446 | { | 419 | { |
| 447 | struct iommu_dma_cookie *cookie = domain->iova_cookie; | 420 | struct iommu_dma_cookie *cookie = domain->iova_cookie; |
| 448 | struct iommu_dma_msi_page *msi, *tmp; | 421 | struct iommu_dma_msi_page *msi, *tmp; |
| 449 | 422 | ||
| 450 | #if IS_ENABLED(CONFIG_IRQ_MSI_IOMMU) | 423 | if (cookie->iovad.granule) { |
| 451 | if (domain->sw_msi != iommu_dma_sw_msi) | ||
| 452 | return; | ||
| 453 | #endif | ||
| 454 | |||
| 455 | if (!cookie) | ||
| 456 | return; | ||
| 457 | |||
| 458 | if (cookie->type == IOMMU_DMA_IOVA_COOKIE && cookie->iovad.granule) { | ||
| 459 | iommu_dma_free_fq(cookie); | 424 | iommu_dma_free_fq(cookie); |
| 460 | put_iova_domain(&cookie->iovad); | 425 | put_iova_domain(&cookie->iovad); |
| 461 | } | 426 | } |
| 427 | list_for_each_entry_safe(msi, tmp, &cookie->msi_page_list, list) | ||
| 428 | kfree(msi); | ||
| 429 | kfree(cookie); | ||
| 430 | } | ||
| 431 | |||
| 432 | /** | ||
| 433 | * iommu_put_msi_cookie - Release a domain's MSI mapping resources | ||
| 434 | * @domain: IOMMU domain previously prepared by iommu_get_msi_cookie() | ||
| 435 | */ | ||
| 436 | void iommu_put_msi_cookie(struct iommu_domain *domain) | ||
| 437 | { | ||
| 438 | struct iommu_dma_msi_cookie *cookie = domain->msi_cookie; | ||
| 439 | struct iommu_dma_msi_page *msi, *tmp; | ||
| 462 | 440 | ||
| 463 | list_for_each_entry_safe(msi, tmp, &cookie->msi_page_list, list) { | 441 | list_for_each_entry_safe(msi, tmp, &cookie->msi_page_list, list) |
| 464 | list_del(&msi->list); | ||
| 465 | kfree(msi); | 442 | kfree(msi); |
| 466 | } | ||
| 467 | kfree(cookie); | 443 | kfree(cookie); |
| 468 | domain->iova_cookie = NULL; | ||
| 469 | } | 444 | } |
| 470 | 445 | ||
| 471 | /** | 446 | /** |
| @@ -685,7 +660,7 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, struct device *dev · 213 unmodified lines | |||
| 685 | struct iova_domain *iovad; | 660 | struct iova_domain *iovad; |
| 686 | int ret; | 661 | int ret; |
| 687 | 662 | ||
| 688 | if (!cookie || cookie->type != IOMMU_DMA_IOVA_COOKIE) | 663 | if (!cookie || domain->cookie_type != IOMMU_COOKIE_DMA_IOVA) |
| 689 | return -EINVAL; | 664 | return -EINVAL; |
| 690 | 665 | ||
| 691 | iovad = &cookie->iovad; | 666 | iovad = &cookie->iovad; |
| @@ -768,9 +743,9 @@ static dma_addr_t iommu_dma_alloc_iova(struct iommu_domain *domain, · 76 unmodified lines | |||
| 768 | struct iova_domain *iovad = &cookie->iovad; | 743 | struct iova_domain *iovad = &cookie->iovad; |
| 769 | unsigned long shift, iova_len, iova; | 744 | unsigned long shift, iova_len, iova; |
| 770 | 745 | ||
| 771 | if (cookie->type == IOMMU_DMA_MSI_COOKIE) { | 746 | if (domain->cookie_type == IOMMU_COOKIE_DMA_MSI) { |
| 772 | cookie->msi_iova += size; | 747 | domain->msi_cookie->msi_iova += size; |
| 773 | return cookie->msi_iova - size; | 748 | return domain->msi_cookie->msi_iova - size; |
| 774 | } | 749 | } |
| 775 | 750 | ||
| 776 | shift = iova_shift(iovad); | 751 | shift = iova_shift(iovad); |
| @@ -807,16 +782,16 @@ done: · 30 unmodified lines | |||
| 807 | return (dma_addr_t)iova << shift; | 782 | return (dma_addr_t)iova << shift; |
| 808 | } | 783 | } |
| 809 | 784 | ||
| 810 | static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie, | 785 | static void iommu_dma_free_iova(struct iommu_domain *domain, dma_addr_t iova, |
| 811 | dma_addr_t iova, size_t size, struct iommu_iotlb_gather *gather) | 786 | size_t size, struct iommu_iotlb_gather *gather) |
| 812 | { | 787 | { |
| 813 | struct iova_domain *iovad = &cookie->iovad; | 788 | struct iova_domain *iovad = &domain->iova_cookie->iovad; |
| 814 | 789 | ||
| 815 | /* The MSI case is only ever cleaning up its most recent allocation */ | 790 | /* The MSI case is only ever cleaning up its most recent allocation */ |
| 816 | if (cookie->type == IOMMU_DMA_MSI_COOKIE) | 791 | if (domain->cookie_type == IOMMU_COOKIE_DMA_MSI) |
| 817 | cookie->msi_iova -= size; | 792 | domain->msi_cookie->msi_iova -= size; |
| 818 | else if (gather && gather->queued) | 793 | else if (gather && gather->queued) |
| 819 | queue_iova(cookie, iova_pfn(iovad, iova), | 794 | queue_iova(domain->iova_cookie, iova_pfn(iovad, iova), |
| 820 | size >> iova_shift(iovad), | 795 | size >> iova_shift(iovad), |
| 821 | &gather->freelist); | 796 | &gather->freelist); |
| 822 | else | 797 | else |
| @@ -844,7 +819,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, · 21 unmodified lines | |||
| 844 | 819 | ||
| 845 | if (!iotlb_gather.queued) | 820 | if (!iotlb_gather.queued) |
| 846 | iommu_iotlb_sync(domain, &iotlb_gather); | 821 | iommu_iotlb_sync(domain, &iotlb_gather); |
| 847 | iommu_dma_free_iova(cookie, dma_addr, size, &iotlb_gather); | 822 | iommu_dma_free_iova(domain, dma_addr, size, &iotlb_gather); |
| 848 | } | 823 | } |
| 849 | 824 | ||
| 850 | static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, | 825 | static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, |
| @@ -872,7 +847,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, · 21 unmodified lines | |||
| 872 | return DMA_MAPPING_ERROR; | 847 | return DMA_MAPPING_ERROR; |
| 873 | 848 | ||
| 874 | if (iommu_map(domain, iova, phys - iova_off, size, prot, GFP_ATOMIC)) { | 849 | if (iommu_map(domain, iova, phys - iova_off, size, prot, GFP_ATOMIC)) { |
| 875 | iommu_dma_free_iova(cookie, iova, size, NULL); | 850 | iommu_dma_free_iova(domain, iova, size, NULL); |
| 876 | return DMA_MAPPING_ERROR; | 851 | return DMA_MAPPING_ERROR; |
| 877 | } | 852 | } |
| 878 | return iova + iova_off; | 853 | return iova + iova_off; |
| @@ -1009,7 +984,7 @@ static struct page **__iommu_dma_alloc_noncontiguous(struct device *dev, · 130 unmodified lines | |||
| 1009 | out_free_sg: | 984 | out_free_sg: |
| 1010 | sg_free_table(sgt); | 985 | sg_free_table(sgt); |
| 1011 | out_free_iova: | 986 | out_free_iova: |
| 1012 | iommu_dma_free_iova(cookie, iova, size, NULL); | 987 | iommu_dma_free_iova(domain, iova, size, NULL); |
| 1013 | out_free_pages: | 988 | out_free_pages: |
| 1014 | __iommu_dma_free_pages(pages, count); | 989 | __iommu_dma_free_pages(pages, count); |
| 1015 | return NULL; | 990 | return NULL; |
| @@ -1486,7 +1461,7 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents, · 470 unmodified lines | |||
| 1486 | return __finalise_sg(dev, sg, nents, iova); | 1461 | return __finalise_sg(dev, sg, nents, iova); |
| 1487 | 1462 | ||
| 1488 | out_free_iova: | 1463 | out_free_iova: |
| 1489 | iommu_dma_free_iova(cookie, iova, iova_len, NULL); | 1464 | iommu_dma_free_iova(domain, iova, iova_len, NULL); |
| 1490 | out_restore_sg: | 1465 | out_restore_sg: |
| 1491 | __invalidate_sg(sg, nents); | 1466 | __invalidate_sg(sg, nents); |
| 1492 | out: | 1467 | out: |
| @@ -1764,17 +1739,47 @@ out_err: · 271 unmodified lines | |||
| 1764 | dev->dma_iommu = false; | 1739 | dev->dma_iommu = false; |
| 1765 | } | 1740 | } |
| 1766 | 1741 | ||
| 1742 | static bool has_msi_cookie(const struct iommu_domain *domain) | ||
| 1743 | { | ||
| 1744 | return domain && (domain->cookie_type == IOMMU_COOKIE_DMA_IOVA || | ||
| 1745 | domain->cookie_type == IOMMU_COOKIE_DMA_MSI); | ||
| 1746 | } | ||
| 1747 | |||
| 1748 | static size_t cookie_msi_granule(const struct iommu_domain *domain) | ||
| 1749 | { | ||
| 1750 | switch (domain->cookie_type) { | ||
| 1751 | case IOMMU_COOKIE_DMA_IOVA: | ||
| 1752 | return domain->iova_cookie->iovad.granule; | ||
| 1753 | case IOMMU_COOKIE_DMA_MSI: | ||
| 1754 | return PAGE_SIZE; | ||
| 1755 | default: | ||
| 1756 | BUG(); | ||
| 1757 | }; | ||
| 1758 | } | ||
| 1759 | |||
| 1760 | static struct list_head *cookie_msi_pages(const struct iommu_domain *domain) | ||
| 1761 | { | ||
| 1762 | switch (domain->cookie_type) { | ||
| 1763 | case IOMMU_COOKIE_DMA_IOVA: | ||
| 1764 | return &domain->iova_cookie->msi_page_list; | ||
| 1765 | case IOMMU_COOKIE_DMA_MSI: | ||
| 1766 | return &domain->msi_cookie->msi_page_list; | ||
| 1767 | default: | ||
| 1768 | BUG(); | ||
| 1769 | }; | ||
| 1770 | } | ||
| 1771 | |||
| 1767 | static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev, | 1772 | static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev, |
| 1768 | phys_addr_t msi_addr, struct iommu_domain *domain) | 1773 | phys_addr_t msi_addr, struct iommu_domain *domain) |
| 1769 | { | 1774 | { |
| 1770 | struct iommu_dma_cookie *cookie = domain->iova_cookie; | 1775 | struct list_head *msi_page_list = cookie_msi_pages(domain); |
| 1771 | struct iommu_dma_msi_page *msi_page; | 1776 | struct iommu_dma_msi_page *msi_page; |
| 1772 | dma_addr_t iova; | 1777 | dma_addr_t iova; |
| 1773 | int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO; | 1778 | int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO; |
| 1774 | size_t size = cookie_msi_granule(cookie); | 1779 | size_t size = cookie_msi_granule(domain); |
| 1775 | 1780 | ||
| 1776 | msi_addr &= ~(phys_addr_t)(size - 1); | 1781 | msi_addr &= ~(phys_addr_t)(size - 1); |
| 1777 | list_for_each_entry(msi_page, &cookie->msi_page_list, list) | 1782 | list_for_each_entry(msi_page, msi_page_list, list) |
| 1778 | if (msi_page->phys == msi_addr) | 1783 | if (msi_page->phys == msi_addr) |
| 1779 | return msi_page; | 1784 | return msi_page; |
| 1780 | 1785 | ||
| @@ -1792,23 +1797,23 @@ static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev, · 11 unmodified lines | |||
| 1792 | INIT_LIST_HEAD(&msi_page->list); | 1797 | INIT_LIST_HEAD(&msi_page->list); |
| 1793 | msi_page->phys = msi_addr; | 1798 | msi_page->phys = msi_addr; |
| 1794 | msi_page->iova = iova; | 1799 | msi_page->iova = iova; |
| 1795 | list_add(&msi_page->list, &cookie->msi_page_list); | 1800 | list_add(&msi_page->list, msi_page_list); |
| 1796 | return msi_page; | 1801 | return msi_page; |
| 1797 | 1802 | ||
| 1798 | out_free_iova: | 1803 | out_free_iova: |
| 1799 | iommu_dma_free_iova(cookie, iova, size, NULL); | 1804 | iommu_dma_free_iova(domain, iova, size, NULL); |
| 1800 | out_free_page: | 1805 | out_free_page: |
| 1801 | kfree(msi_page); | 1806 | kfree(msi_page); |
| 1802 | return NULL; | 1807 | return NULL; |
| 1803 | } | 1808 | } |
| 1804 | 1809 | ||
| 1805 | static int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | 1810 | int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, |
| 1806 | phys_addr_t msi_addr) | 1811 | phys_addr_t msi_addr) |
| 1807 | { | 1812 | { |
| 1808 | struct device *dev = msi_desc_to_dev(desc); | 1813 | struct device *dev = msi_desc_to_dev(desc); |
| 1809 | const struct iommu_dma_msi_page *msi_page; | 1814 | const struct iommu_dma_msi_page *msi_page; |
| 1810 | 1815 | ||
| 1811 | if (!domain->iova_cookie) { | 1816 | if (!has_msi_cookie(domain)) { |
| 1812 | msi_desc_set_iommu_msi_iova(desc, 0, 0); | 1817 | msi_desc_set_iommu_msi_iova(desc, 0, 0); |
| 1813 | return 0; | 1818 | return 0; |
| 1814 | } | 1819 | } |
| @@ -1818,9 +1823,8 @@ static int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, · 3 unmodified lines | |||
| 1818 | if (!msi_page) | 1823 | if (!msi_page) |
| 1819 | return -ENOMEM; | 1824 | return -ENOMEM; |
| 1820 | 1825 | ||
| 1821 | msi_desc_set_iommu_msi_iova( | 1826 | msi_desc_set_iommu_msi_iova(desc, msi_page->iova, |
| 1822 | desc, msi_page->iova, | 1827 | ilog2(cookie_msi_granule(domain))); |
| 1823 | ilog2(cookie_msi_granule(domain->iova_cookie))); | ||
| 1824 | return 0; | 1828 | return 0; |
| 1825 | } | 1829 | } |
| 1826 | 1830 | ||
| @@ -13,11 +13,15 @@ void iommu_setup_dma_ops(struct device *dev); · 12 unmodified lines | |||
| 13 | 13 | ||
| 14 | int iommu_get_dma_cookie(struct iommu_domain *domain); | 14 | int iommu_get_dma_cookie(struct iommu_domain *domain); |
| 15 | void iommu_put_dma_cookie(struct iommu_domain *domain); | 15 | void iommu_put_dma_cookie(struct iommu_domain *domain); |
| 16 | void iommu_put_msi_cookie(struct iommu_domain *domain); | ||
| 16 | 17 | ||
| 17 | int iommu_dma_init_fq(struct iommu_domain *domain); | 18 | int iommu_dma_init_fq(struct iommu_domain *domain); |
| 18 | 19 | ||
| 19 | void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list); | 20 | void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list); |
| 20 | 21 | ||
| 22 | int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | ||
| 23 | phys_addr_t msi_addr); | ||
| 24 | |||
| 21 | extern bool iommu_dma_forcedac; | 25 | extern bool iommu_dma_forcedac; |
| 22 | 26 | ||
| 23 | #else /* CONFIG_IOMMU_DMA */ | 27 | #else /* CONFIG_IOMMU_DMA */ |
| @@ -40,9 +44,19 @@ static inline void iommu_put_dma_cookie(struct iommu_domain *domain) · 16 unmodified lines | |||
| 40 | { | 44 | { |
| 41 | } | 45 | } |
| 42 | 46 | ||
| 47 | static inline void iommu_put_msi_cookie(struct iommu_domain *domain) | ||
| 48 | { | ||
| 49 | } | ||
| 50 | |||
| 43 | static inline void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list) | 51 | static inline void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list) |
| 44 | { | 52 | { |
| 45 | } | 53 | } |
| 46 | 54 | ||
| 55 | static inline int iommu_dma_sw_msi(struct iommu_domain *domain, | ||
| 56 | struct msi_desc *desc, phys_addr_t msi_addr) | ||
| 57 | { | ||
| 58 | return -ENODEV; | ||
| 59 | } | ||
| 60 | |||
| 47 | #endif /* CONFIG_IOMMU_DMA */ | 61 | #endif /* CONFIG_IOMMU_DMA */ |
| 48 | #endif /* __DMA_IOMMU_H */ | 62 | #endif /* __DMA_IOMMU_H */ |
| @@ -3383,7 +3383,8 @@ intel_iommu_domain_alloc_paging_flags(struct device *dev, u32 flags, · 3382 unmodified lines | |||
| 3383 | bool first_stage; | 3383 | bool first_stage; |
| 3384 | 3384 | ||
| 3385 | if (flags & | 3385 | if (flags & |
| 3386 | (~(IOMMU_HWPT_ALLOC_NEST_PARENT | IOMMU_HWPT_ALLOC_DIRTY_TRACKING))) | 3386 | (~(IOMMU_HWPT_ALLOC_NEST_PARENT | IOMMU_HWPT_ALLOC_DIRTY_TRACKING | |
| 3387 | IOMMU_HWPT_ALLOC_PASID))) | ||
| 3387 | return ERR_PTR(-EOPNOTSUPP); | 3388 | return ERR_PTR(-EOPNOTSUPP); |
| 3388 | if (nested_parent && !nested_supported(iommu)) | 3389 | if (nested_parent && !nested_supported(iommu)) |
| 3389 | return ERR_PTR(-EOPNOTSUPP); | 3390 | return ERR_PTR(-EOPNOTSUPP); |
| @@ -198,7 +198,7 @@ intel_iommu_domain_alloc_nested(struct device *dev, struct iommu_domain *parent, · 197 unmodified lines | |||
| 198 | struct dmar_domain *domain; | 198 | struct dmar_domain *domain; |
| 199 | int ret; | 199 | int ret; |
| 200 | 200 | ||
| 201 | if (!nested_supported(iommu) || flags) | 201 | if (!nested_supported(iommu) || flags & ~IOMMU_HWPT_ALLOC_PASID) |
| 202 | return ERR_PTR(-EOPNOTSUPP); | 202 | return ERR_PTR(-EOPNOTSUPP); |
| 203 | 203 | ||
| 204 | /* Must be nested domain */ | 204 | /* Must be nested domain */ |
| @@ -5,6 +5,7 @@ · 4 unmodified lines | |||
| 5 | #define __LINUX_IOMMU_PRIV_H | 5 | #define __LINUX_IOMMU_PRIV_H |
| 6 | 6 | ||
| 7 | #include <linux/iommu.h> | 7 | #include <linux/iommu.h> |
| 8 | #include <linux/msi.h> | ||
| 8 | 9 | ||
| 9 | static inline const struct iommu_ops *dev_iommu_ops(struct device *dev) | 10 | static inline const struct iommu_ops *dev_iommu_ops(struct device *dev) |
| 10 | { | 11 | { |
| @@ -47,4 +48,19 @@ void iommu_detach_group_handle(struct iommu_domain *domain, · 36 unmodified lines | |||
| 47 | int iommu_replace_group_handle(struct iommu_group *group, | 48 | int iommu_replace_group_handle(struct iommu_group *group, |
| 48 | struct iommu_domain *new_domain, | 49 | struct iommu_domain *new_domain, |
| 49 | struct iommu_attach_handle *handle); | 50 | struct iommu_attach_handle *handle); |
| 51 | |||
| 52 | #if IS_ENABLED(CONFIG_IOMMUFD_DRIVER_CORE) && IS_ENABLED(CONFIG_IRQ_MSI_IOMMU) | ||
| 53 | int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | ||
| 54 | phys_addr_t msi_addr); | ||
| 55 | #else /* !CONFIG_IOMMUFD_DRIVER_CORE || !CONFIG_IRQ_MSI_IOMMU */ | ||
| 56 | static inline int iommufd_sw_msi(struct iommu_domain *domain, | ||
| 57 | struct msi_desc *desc, phys_addr_t msi_addr) | ||
| 58 | { | ||
| 59 | return -EOPNOTSUPP; | ||
| 60 | } | ||
| 61 | #endif /* CONFIG_IOMMUFD_DRIVER_CORE && CONFIG_IRQ_MSI_IOMMU */ | ||
| 62 | |||
| 63 | int iommu_replace_device_pasid(struct iommu_domain *domain, | ||
| 64 | struct device *dev, ioasid_t pasid, | ||
| 65 | struct iommu_attach_handle *handle); | ||
| 50 | #endif /* __LINUX_IOMMU_PRIV_H */ | 66 | #endif /* __LINUX_IOMMU_PRIV_H */ |
| @@ -310,6 +310,7 @@ static struct iommu_domain *iommu_sva_domain_alloc(struct device *dev, · 309 unmodified lines | |||
| 310 | } | 310 | } |
| 311 | 311 | ||
| 312 | domain->type = IOMMU_DOMAIN_SVA; | 312 | domain->type = IOMMU_DOMAIN_SVA; |
| 313 | domain->cookie_type = IOMMU_COOKIE_SVA; | ||
| 313 | mmgrab(mm); | 314 | mmgrab(mm); |
| 314 | domain->mm = mm; | 315 | domain->mm = mm; |
| 315 | domain->owner = ops; | 316 | domain->owner = ops; |
| @@ -18,6 +18,7 @@ · 17 unmodified lines | |||
| 18 | #include <linux/errno.h> | 18 | #include <linux/errno.h> |
| 19 | #include <linux/host1x_context_bus.h> | 19 | #include <linux/host1x_context_bus.h> |
| 20 | #include <linux/iommu.h> | 20 | #include <linux/iommu.h> |
| 21 | #include <linux/iommufd.h> | ||
| 21 | #include <linux/idr.h> | 22 | #include <linux/idr.h> |
| 22 | #include <linux/err.h> | 23 | #include <linux/err.h> |
| 23 | #include <linux/pci.h> | 24 | #include <linux/pci.h> |
| @@ -539,6 +540,13 @@ static void iommu_deinit_device(struct device *dev) · 515 unmodified lines | |||
| 539 | dev_iommu_free(dev); | 540 | dev_iommu_free(dev); |
| 540 | } | 541 | } |
| 541 | 542 | ||
| 543 | static struct iommu_domain *pasid_array_entry_to_domain(void *entry) | ||
| 544 | { | ||
| 545 | if (xa_pointer_tag(entry) == IOMMU_PASID_ARRAY_DOMAIN) | ||
| 546 | return xa_untag_pointer(entry); | ||
| 547 | return ((struct iommu_attach_handle *)xa_untag_pointer(entry))->domain; | ||
| 548 | } | ||
| 549 | |||
| 542 | DEFINE_MUTEX(iommu_probe_device_lock); | 550 | DEFINE_MUTEX(iommu_probe_device_lock); |
| 543 | 551 | ||
| 544 | static int __iommu_probe_device(struct device *dev, struct list_head *group_list) | 552 | static int __iommu_probe_device(struct device *dev, struct list_head *group_list) |
| @@ -1973,8 +1981,10 @@ void iommu_set_fault_handler(struct iommu_domain *domain, · 1428 unmodified lines | |||
| 1973 | iommu_fault_handler_t handler, | 1981 | iommu_fault_handler_t handler, |
| 1974 | void *token) | 1982 | void *token) |
| 1975 | { | 1983 | { |
| 1976 | BUG_ON(!domain); | 1984 | if (WARN_ON(!domain || domain->cookie_type != IOMMU_COOKIE_NONE)) |
| 1985 | return; | ||
| 1977 | 1986 | ||
| 1987 | domain->cookie_type = IOMMU_COOKIE_FAULT_HANDLER; | ||
| 1978 | domain->handler = handler; | 1988 | domain->handler = handler; |
| 1979 | domain->handler_token = token; | 1989 | domain->handler_token = token; |
| 1980 | } | 1990 | } |
| @@ -2044,9 +2054,19 @@ EXPORT_SYMBOL_GPL(iommu_paging_domain_alloc_flags); · 63 unmodified lines | |||
| 2044 | 2054 | ||
| 2045 | void iommu_domain_free(struct iommu_domain *domain) | 2055 | void iommu_domain_free(struct iommu_domain *domain) |
| 2046 | { | 2056 | { |
| 2047 | if (domain->type == IOMMU_DOMAIN_SVA) | 2057 | switch (domain->cookie_type) { |
| 2058 | case IOMMU_COOKIE_DMA_IOVA: | ||
| 2059 | iommu_put_dma_cookie(domain); | ||
| 2060 | break; | ||
| 2061 | case IOMMU_COOKIE_DMA_MSI: | ||
| 2062 | iommu_put_msi_cookie(domain); | ||
| 2063 | break; | ||
| 2064 | case IOMMU_COOKIE_SVA: | ||
| 2048 | mmdrop(domain->mm); | 2065 | mmdrop(domain->mm); |
| 2049 | iommu_put_dma_cookie(domain); | 2066 | break; |
| 2067 | default: | ||
| 2068 | break; | ||
| 2069 | } | ||
| 2050 | if (domain->ops->free) | 2070 | if (domain->ops->free) |
| 2051 | domain->ops->free(domain); | 2071 | domain->ops->free(domain); |
| 2052 | } | 2072 | } |
| @@ -3335,14 +3355,15 @@ static void iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid, · 1282 unmodified lines | |||
| 3335 | } | 3355 | } |
| 3336 | 3356 | ||
| 3337 | static int __iommu_set_group_pasid(struct iommu_domain *domain, | 3357 | static int __iommu_set_group_pasid(struct iommu_domain *domain, |
| 3338 | struct iommu_group *group, ioasid_t pasid) | 3358 | struct iommu_group *group, ioasid_t pasid, |
| 3359 | struct iommu_domain *old) | ||
| 3339 | { | 3360 | { |
| 3340 | struct group_device *device, *last_gdev; | 3361 | struct group_device *device, *last_gdev; |
| 3341 | int ret; | 3362 | int ret; |
| 3342 | 3363 | ||
| 3343 | for_each_group_device(group, device) { | 3364 | for_each_group_device(group, device) { |
| 3344 | ret = domain->ops->set_dev_pasid(domain, device->dev, | 3365 | ret = domain->ops->set_dev_pasid(domain, device->dev, |
| 3345 | pasid, NULL); | 3366 | pasid, old); |
| 3346 | if (ret) | 3367 | if (ret) |
| 3347 | goto err_revert; | 3368 | goto err_revert; |
| 3348 | } | 3369 | } |
| @@ -3354,7 +3375,15 @@ err_revert: · 5 unmodified lines | |||
| 3354 | for_each_group_device(group, device) { | 3375 | for_each_group_device(group, device) { |
| 3355 | if (device == last_gdev) | 3376 | if (device == last_gdev) |
| 3356 | break; | 3377 | break; |
| 3357 | iommu_remove_dev_pasid(device->dev, pasid, domain); | 3378 | /* |
| 3379 | * If no old domain, undo the succeeded devices/pasid. | ||
| 3380 | * Otherwise, rollback the succeeded devices/pasid to the old | ||
| 3381 | * domain. And it is a driver bug to fail attaching with a | ||
| 3382 | * previously good domain. | ||
| 3383 | */ | ||
| 3384 | if (!old || WARN_ON(old->ops->set_dev_pasid(old, device->dev, | ||
| 3385 | pasid, domain))) | ||
| 3386 | iommu_remove_dev_pasid(device->dev, pasid, domain); | ||
| 3358 | } | 3387 | } |
| 3359 | return ret; | 3388 | return ret; |
| 3360 | } | 3389 | } |
| @@ -3376,6 +3405,9 @@ static void __iommu_remove_group_pasid(struct iommu_group *group, · 15 unmodified lines | |||
| 3376 | * @pasid: the pasid of the device. | 3405 | * @pasid: the pasid of the device. |
| 3377 | * @handle: the attach handle. | 3406 | * @handle: the attach handle. |
| 3378 | * | 3407 | * |
| 3408 | * Caller should always provide a new handle to avoid race with the paths | ||
| 3409 | * that have lockless reference to handle if it intends to pass a valid handle. | ||
| 3410 | * | ||
| 3379 | * Return: 0 on success, or an error. | 3411 | * Return: 0 on success, or an error. |
| 3380 | */ | 3412 | */ |
| 3381 | int iommu_attach_device_pasid(struct iommu_domain *domain, | 3413 | int iommu_attach_device_pasid(struct iommu_domain *domain, |
| @@ -3420,7 +3452,7 @@ int iommu_attach_device_pasid(struct iommu_domain *domain, · 38 unmodified lines | |||
| 3420 | if (ret) | 3452 | if (ret) |
| 3421 | goto out_unlock; | 3453 | goto out_unlock; |
| 3422 | 3454 | ||
| 3423 | ret = __iommu_set_group_pasid(domain, group, pasid); | 3455 | ret = __iommu_set_group_pasid(domain, group, pasid, NULL); |
| 3424 | if (ret) { | 3456 | if (ret) { |
| 3425 | xa_release(&group->pasid_array, pasid); | 3457 | xa_release(&group->pasid_array, pasid); |
| 3426 | goto out_unlock; | 3458 | goto out_unlock; |
| @@ -3441,6 +3473,97 @@ out_unlock: · 14 unmodified lines | |||
| 3441 | } | 3473 | } |
| 3442 | EXPORT_SYMBOL_GPL(iommu_attach_device_pasid); | 3474 | EXPORT_SYMBOL_GPL(iommu_attach_device_pasid); |
| 3443 | 3475 | ||
| 3476 | /** | ||
| 3477 | * iommu_replace_device_pasid - Replace the domain that a specific pasid | ||
| 3478 | * of the device is attached to | ||
| 3479 | * @domain: the new iommu domain | ||
| 3480 | * @dev: the attached device. | ||
| 3481 | * @pasid: the pasid of the device. | ||
| 3482 | * @handle: the attach handle. | ||
| 3483 | * | ||
| 3484 | * This API allows the pasid to switch domains. The @pasid should have been | ||
| 3485 | * attached. Otherwise, this fails. The pasid will keep the old configuration | ||
| 3486 | * if replacement failed. | ||
| 3487 | * | ||
| 3488 | * Caller should always provide a new handle to avoid race with the paths | ||
| 3489 | * that have lockless reference to handle if it intends to pass a valid handle. | ||
| 3490 | * | ||
| 3491 | * Return 0 on success, or an error. | ||
| 3492 | */ | ||
| 3493 | int iommu_replace_device_pasid(struct iommu_domain *domain, | ||
| 3494 | struct device *dev, ioasid_t pasid, | ||
| 3495 | struct iommu_attach_handle *handle) | ||
| 3496 | { | ||
| 3497 | /* Caller must be a probed driver on dev */ | ||
| 3498 | struct iommu_group *group = dev->iommu_group; | ||
| 3499 | struct iommu_attach_handle *entry; | ||
| 3500 | struct iommu_domain *curr_domain; | ||
| 3501 | void *curr; | ||
| 3502 | int ret; | ||
| 3503 | |||
| 3504 | if (!group) | ||
| 3505 | return -ENODEV; | ||
| 3506 | |||
| 3507 | if (!domain->ops->set_dev_pasid) | ||
| 3508 | return -EOPNOTSUPP; | ||
| 3509 | |||
| 3510 | if (dev_iommu_ops(dev) != domain->owner || | ||
| 3511 | pasid == IOMMU_NO_PASID || !handle) | ||
| 3512 | return -EINVAL; | ||
| 3513 | |||
| 3514 | mutex_lock(&group->mutex); | ||
| 3515 | entry = iommu_make_pasid_array_entry(domain, handle); | ||
| 3516 | curr = xa_cmpxchg(&group->pasid_array, pasid, NULL, | ||
| 3517 | XA_ZERO_ENTRY, GFP_KERNEL); | ||
| 3518 | if (xa_is_err(curr)) { | ||
| 3519 | ret = xa_err(curr); | ||
| 3520 | goto out_unlock; | ||
| 3521 | } | ||
| 3522 | |||
| 3523 | /* | ||
| 3524 | * No domain (with or without handle) attached, hence not | ||
| 3525 | * a replace case. | ||
| 3526 | */ | ||
| 3527 | if (!curr) { | ||
| 3528 | xa_release(&group->pasid_array, pasid); | ||
| 3529 | ret = -EINVAL; | ||
| 3530 | goto out_unlock; | ||
| 3531 | } | ||
| 3532 | |||
| 3533 | /* | ||
| 3534 | * Reusing handle is problematic as there are paths that refers | ||
| 3535 | * the handle without lock. To avoid race, reject the callers that | ||
| 3536 | * attempt it. | ||
| 3537 | */ | ||
| 3538 | if (curr == entry) { | ||
| 3539 | WARN_ON(1); | ||
| 3540 | ret = -EINVAL; | ||
| 3541 | goto out_unlock; | ||
| 3542 | } | ||
| 3543 | |||
| 3544 | curr_domain = pasid_array_entry_to_domain(curr); | ||
| 3545 | ret = 0; | ||
| 3546 | |||
| 3547 | if (curr_domain != domain) { | ||
| 3548 | ret = __iommu_set_group_pasid(domain, group, | ||
| 3549 | pasid, curr_domain); | ||
| 3550 | if (ret) | ||
| 3551 | goto out_unlock; | ||
| 3552 | } | ||
| 3553 | |||
| 3554 | /* | ||
| 3555 | * The above xa_cmpxchg() reserved the memory, and the | ||
| 3556 | * group->mutex is held, this cannot fail. | ||
| 3557 | */ | ||
| 3558 | WARN_ON(xa_is_err(xa_store(&group->pasid_array, | ||
| 3559 | pasid, entry, GFP_KERNEL))); | ||
| 3560 | |||
| 3561 | out_unlock: | ||
| 3562 | mutex_unlock(&group->mutex); | ||
| 3563 | return ret; | ||
| 3564 | } | ||
| 3565 | EXPORT_SYMBOL_NS_GPL(iommu_replace_device_pasid, "IOMMUFD_INTERNAL"); | ||
| 3566 | |||
| 3444 | /* | 3567 | /* |
| 3445 | * iommu_detach_device_pasid() - Detach the domain from pasid of device | 3568 | * iommu_detach_device_pasid() - Detach the domain from pasid of device |
| 3446 | * @domain: the iommu domain. | 3569 | * @domain: the iommu domain. |
| @@ -3536,6 +3659,9 @@ EXPORT_SYMBOL_NS_GPL(iommu_attach_handle_get, "IOMMUFD_INTERNAL"); · 89 unmodified lines | |||
| 3536 | * This is a variant of iommu_attach_group(). It allows the caller to provide | 3659 | * This is a variant of iommu_attach_group(). It allows the caller to provide |
| 3537 | * an attach handle and use it when the domain is attached. This is currently | 3660 | * an attach handle and use it when the domain is attached. This is currently |
| 3538 | * used by IOMMUFD to deliver the I/O page faults. | 3661 | * used by IOMMUFD to deliver the I/O page faults. |
| 3662 | * | ||
| 3663 | * Caller should always provide a new handle to avoid race with the paths | ||
| 3664 | * that have lockless reference to handle. | ||
| 3539 | */ | 3665 | */ |
| 3540 | int iommu_attach_group_handle(struct iommu_domain *domain, | 3666 | int iommu_attach_group_handle(struct iommu_domain *domain, |
| 3541 | struct iommu_group *group, | 3667 | struct iommu_group *group, |
| @@ -3605,6 +3731,9 @@ EXPORT_SYMBOL_NS_GPL(iommu_detach_group_handle, "IOMMUFD_INTERNAL"); · 63 unmodified lines | |||
| 3605 | * | 3731 | * |
| 3606 | * If the currently attached domain is a core domain (e.g. a default_domain), | 3732 | * If the currently attached domain is a core domain (e.g. a default_domain), |
| 3607 | * it will act just like the iommu_attach_group_handle(). | 3733 | * it will act just like the iommu_attach_group_handle(). |
| 3734 | * | ||
| 3735 | * Caller should always provide a new handle to avoid race with the paths | ||
| 3736 | * that have lockless reference to handle. | ||
| 3608 | */ | 3737 | */ |
| 3609 | int iommu_replace_group_handle(struct iommu_group *group, | 3738 | int iommu_replace_group_handle(struct iommu_group *group, |
| 3610 | struct iommu_domain *new_domain, | 3739 | struct iommu_domain *new_domain, |
| @@ -3662,8 +3791,21 @@ int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_addr) · 51 unmodified lines | |||
| 3662 | return 0; | 3791 | return 0; |
| 3663 | 3792 | ||
| 3664 | mutex_lock(&group->mutex); | 3793 | mutex_lock(&group->mutex); |
| 3665 | if (group->domain && group->domain->sw_msi) | 3794 | /* An IDENTITY domain must pass through */ |
| 3666 | ret = group->domain->sw_msi(group->domain, desc, msi_addr); | 3795 | if (group->domain && group->domain->type != IOMMU_DOMAIN_IDENTITY) { |
| 3796 | switch (group->domain->cookie_type) { | ||
| 3797 | case IOMMU_COOKIE_DMA_MSI: | ||
| 3798 | case IOMMU_COOKIE_DMA_IOVA: | ||
| 3799 | ret = iommu_dma_sw_msi(group->domain, desc, msi_addr); | ||
| 3800 | break; | ||
| 3801 | case IOMMU_COOKIE_IOMMUFD: | ||
| 3802 | ret = iommufd_sw_msi(group->domain, desc, msi_addr); | ||
| 3803 | break; | ||
| 3804 | default: | ||
| 3805 | ret = -EOPNOTSUPP; | ||
| 3806 | break; | ||
| 3807 | } | ||
| 3808 | } | ||
| 3667 | mutex_unlock(&group->mutex); | 3809 | mutex_unlock(&group->mutex); |
| 3668 | return ret; | 3810 | return ret; |
| 3669 | } | 3811 | } |
| @@ -1,6 +1,6 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only | 1 | # SPDX-License-Identifier: GPL-2.0-only |
| 2 | config IOMMUFD_DRIVER_CORE | 2 | config IOMMUFD_DRIVER_CORE |
| 3 | tristate | 3 | bool |
| 4 | default (IOMMUFD_DRIVER || IOMMUFD) if IOMMUFD!=n | 4 | default (IOMMUFD_DRIVER || IOMMUFD) if IOMMUFD!=n |
| 5 | 5 | ||
| 6 | config IOMMUFD | 6 | config IOMMUFD |
| @@ -1,7 +1,7 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only | 1 | # SPDX-License-Identifier: GPL-2.0-only |
| 2 | iommufd-y := \ | 2 | iommufd-y := \ |
| 3 | device.o \ | 3 | device.o \ |
| 4 | fault.o \ | 4 | eventq.o \ |
| 5 | hw_pagetable.o \ | 5 | hw_pagetable.o \ |
| 6 | io_pagetable.o \ | 6 | io_pagetable.o \ |
| 7 | ioas.o \ | 7 | ioas.o \ |
| @@ -3,9 +3,9 @@ · 2 unmodified lines | |||
| 3 | */ | 3 | */ |
| 4 | #include <linux/iommu.h> | 4 | #include <linux/iommu.h> |
| 5 | #include <linux/iommufd.h> | 5 | #include <linux/iommufd.h> |
| 6 | #include <linux/pci-ats.h> | ||
| 6 | #include <linux/slab.h> | 7 | #include <linux/slab.h> |
| 7 | #include <uapi/linux/iommufd.h> | 8 | #include <uapi/linux/iommufd.h> |
| 8 | #include <linux/msi.h> | ||
| 9 | 9 | ||
| 10 | #include "../iommu-priv.h" | 10 | #include "../iommu-priv.h" |
| 11 | #include "io_pagetable.h" | 11 | #include "io_pagetable.h" |
| @@ -18,12 +18,17 @@ MODULE_PARM_DESC( · 6 unmodified lines | |||
| 18 | "Allow IOMMUFD to bind to devices even if the platform cannot isolate " | 18 | "Allow IOMMUFD to bind to devices even if the platform cannot isolate " |
| 19 | "the MSI interrupt window. Enabling this is a security weakness."); | 19 | "the MSI interrupt window. Enabling this is a security weakness."); |
| 20 | 20 | ||
| 21 | struct iommufd_attach { | ||
| 22 | struct iommufd_hw_pagetable *hwpt; | ||
| 23 | struct xarray device_array; | ||
| 24 | }; | ||
| 25 | |||
| 21 | static void iommufd_group_release(struct kref *kref) | 26 | static void iommufd_group_release(struct kref *kref) |
| 22 | { | 27 | { |
| 23 | struct iommufd_group *igroup = | 28 | struct iommufd_group *igroup = |
| 24 | container_of(kref, struct iommufd_group, ref); | 29 | container_of(kref, struct iommufd_group, ref); |
| 25 | 30 | ||
| 26 | WARN_ON(igroup->hwpt || !list_empty(&igroup->device_list)); | 31 | WARN_ON(!xa_empty(&igroup->pasid_attach)); |
| 27 | 32 | ||
| 28 | xa_cmpxchg(&igroup->ictx->groups, iommu_group_id(igroup->group), igroup, | 33 | xa_cmpxchg(&igroup->ictx->groups, iommu_group_id(igroup->group), igroup, |
| 29 | NULL, GFP_KERNEL); | 34 | NULL, GFP_KERNEL); |
| @@ -90,7 +95,7 @@ static struct iommufd_group *iommufd_get_group(struct iommufd_ctx *ictx, · 60 unmodified lines | |||
| 90 | 95 | ||
| 91 | kref_init(&new_igroup->ref); | 96 | kref_init(&new_igroup->ref); |
| 92 | mutex_init(&new_igroup->lock); | 97 | mutex_init(&new_igroup->lock); |
| 93 | INIT_LIST_HEAD(&new_igroup->device_list); | 98 | xa_init(&new_igroup->pasid_attach); |
| 94 | new_igroup->sw_msi_start = PHYS_ADDR_MAX; | 99 | new_igroup->sw_msi_start = PHYS_ADDR_MAX; |
| 95 | /* group reference moves into new_igroup */ | 100 | /* group reference moves into new_igroup */ |
| 96 | new_igroup->group = group; | 101 | new_igroup->group = group; |
| @@ -294,129 +299,24 @@ u32 iommufd_device_to_id(struct iommufd_device *idev) · 197 unmodified lines | |||
| 294 | } | 299 | } |
| 295 | EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, "IOMMUFD"); | 300 | EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, "IOMMUFD"); |
| 296 | 301 | ||
| 297 | /* | 302 | static unsigned int iommufd_group_device_num(struct iommufd_group *igroup, |
| 298 | * Get a iommufd_sw_msi_map for the msi physical address requested by the irq | 303 | ioasid_t pasid) |
| 299 | * layer. The mapping to IOVA is global to the iommufd file descriptor, every | ||
| 300 | * domain that is attached to a device using the same MSI parameters will use | ||
| 301 | * the same IOVA. | ||
| 302 | */ | ||
| 303 | static __maybe_unused struct iommufd_sw_msi_map * | ||
| 304 | iommufd_sw_msi_get_map(struct iommufd_ctx *ictx, phys_addr_t msi_addr, | ||
| 305 | phys_addr_t sw_msi_start) | ||
| 306 | { | ||
| 307 | struct iommufd_sw_msi_map *cur; | ||
| 308 | unsigned int max_pgoff = 0; | ||
| 309 | |||
| 310 | lockdep_assert_held(&ictx->sw_msi_lock); | ||
| 311 | |||
| 312 | list_for_each_entry(cur, &ictx->sw_msi_list, sw_msi_item) { | ||
| 313 | if (cur->sw_msi_start != sw_msi_start) | ||
| 314 | continue; | ||
| 315 | max_pgoff = max(max_pgoff, cur->pgoff + 1); | ||
| 316 | if (cur->msi_addr == msi_addr) | ||
| 317 | return cur; | ||
| 318 | } | ||
| 319 | |||
| 320 | if (ictx->sw_msi_id >= | ||
| 321 | BITS_PER_BYTE * sizeof_field(struct iommufd_sw_msi_maps, bitmap)) | ||
| 322 | return ERR_PTR(-EOVERFLOW); | ||
| 323 | |||
| 324 | cur = kzalloc(sizeof(*cur), GFP_KERNEL); | ||
| 325 | if (!cur) | ||
| 326 | return ERR_PTR(-ENOMEM); | ||
| 327 | |||
| 328 | cur->sw_msi_start = sw_msi_start; | ||
| 329 | cur->msi_addr = msi_addr; | ||
| 330 | cur->pgoff = max_pgoff; | ||
| 331 | cur->id = ictx->sw_msi_id++; | ||
| 332 | list_add_tail(&cur->sw_msi_item, &ictx->sw_msi_list); | ||
| 333 | return cur; | ||
| 334 | } | ||
| 335 | |||
| 336 | static int iommufd_sw_msi_install(struct iommufd_ctx *ictx, | ||
| 337 | struct iommufd_hwpt_paging *hwpt_paging, | ||
| 338 | struct iommufd_sw_msi_map *msi_map) | ||
| 339 | { | 304 | { |
| 340 | unsigned long iova; | 305 | struct iommufd_attach *attach; |
| 341 | 306 | struct iommufd_device *idev; | |
| 342 | lockdep_assert_held(&ictx->sw_msi_lock); | 307 | unsigned int count = 0; |
| 308 | unsigned long index; | ||
| 343 | 309 | ||
| 344 | iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE; | 310 | lockdep_assert_held(&igroup->lock); |
| 345 | if (!test_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap)) { | ||
| 346 | int rc; | ||
| 347 | 311 | ||
| 348 | rc = iommu_map(hwpt_paging->common.domain, iova, | 312 | attach = xa_load(&igroup->pasid_attach, pasid); |
| 349 | msi_map->msi_addr, PAGE_SIZE, | 313 | if (attach) |
| 350 | IOMMU_WRITE | IOMMU_READ | IOMMU_MMIO, | 314 | xa_for_each(&attach->device_array, index, idev) |
| 351 | GFP_KERNEL_ACCOUNT); | 315 | count++; |
| 352 | if (rc) | 316 | return count; |
| 353 | return rc; | ||
| 354 | __set_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap); | ||
| 355 | } | ||
| 356 | return 0; | ||
| 357 | } | 317 | } |
| 358 | 318 | ||
| 359 | /* | ||
| 360 | * Called by the irq code if the platform translates the MSI address through the | ||
| 361 | * IOMMU. msi_addr is the physical address of the MSI page. iommufd will | ||
| 362 | * allocate a fd global iova for the physical page that is the same on all | ||
| 363 | * domains and devices. | ||
| 364 | */ | ||
| 365 | #ifdef CONFIG_IRQ_MSI_IOMMU | 319 | #ifdef CONFIG_IRQ_MSI_IOMMU |
| 366 | int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | ||
| 367 | phys_addr_t msi_addr) | ||
| 368 | { | ||
| 369 | struct device *dev = msi_desc_to_dev(desc); | ||
| 370 | struct iommufd_hwpt_paging *hwpt_paging; | ||
| 371 | struct iommu_attach_handle *raw_handle; | ||
| 372 | struct iommufd_attach_handle *handle; | ||
| 373 | struct iommufd_sw_msi_map *msi_map; | ||
| 374 | struct iommufd_ctx *ictx; | ||
| 375 | unsigned long iova; | ||
| 376 | int rc; | ||
| 377 | |||
| 378 | /* | ||
| 379 | * It is safe to call iommu_attach_handle_get() here because the iommu | ||
| 380 | * core code invokes this under the group mutex which also prevents any | ||
| 381 | * change of the attach handle for the duration of this function. | ||
| 382 | */ | ||
| 383 | iommu_group_mutex_assert(dev); | ||
| 384 | |||
| 385 | raw_handle = | ||
| 386 | iommu_attach_handle_get(dev->iommu_group, IOMMU_NO_PASID, 0); | ||
| 387 | if (IS_ERR(raw_handle)) | ||
| 388 | return 0; | ||
| 389 | hwpt_paging = find_hwpt_paging(domain->iommufd_hwpt); | ||
| 390 | |||
| 391 | handle = to_iommufd_handle(raw_handle); | ||
| 392 | /* No IOMMU_RESV_SW_MSI means no change to the msi_msg */ | ||
| 393 | if (handle->idev->igroup->sw_msi_start == PHYS_ADDR_MAX) | ||
| 394 | return 0; | ||
| 395 | |||
| 396 | ictx = handle->idev->ictx; | ||
| 397 | guard(mutex)(&ictx->sw_msi_lock); | ||
| 398 | /* | ||
| 399 | * The input msi_addr is the exact byte offset of the MSI doorbell, we | ||
| 400 | * assume the caller has checked that it is contained with a MMIO region | ||
| 401 | * that is secure to map at PAGE_SIZE. | ||
| 402 | */ | ||
| 403 | msi_map = iommufd_sw_msi_get_map(handle->idev->ictx, | ||
| 404 | msi_addr & PAGE_MASK, | ||
| 405 | handle->idev->igroup->sw_msi_start); | ||
| 406 | if (IS_ERR(msi_map)) | ||
| 407 | return PTR_ERR(msi_map); | ||
| 408 | |||
| 409 | rc = iommufd_sw_msi_install(ictx, hwpt_paging, msi_map); | ||
| 410 | if (rc) | ||
| 411 | return rc; | ||
| 412 | __set_bit(msi_map->id, handle->idev->igroup->required_sw_msi.bitmap); | ||
| 413 | |||
| 414 | iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE; | ||
| 415 | msi_desc_set_iommu_msi_iova(desc, iova, PAGE_SHIFT); | ||
| 416 | return 0; | ||
| 417 | } | ||
| 418 | #endif | ||
| 419 | |||
| 420 | static int iommufd_group_setup_msi(struct iommufd_group *igroup, | 320 | static int iommufd_group_setup_msi(struct iommufd_group *igroup, |
| 421 | struct iommufd_hwpt_paging *hwpt_paging) | 321 | struct iommufd_hwpt_paging *hwpt_paging) |
| 422 | { | 322 | { |
| @@ -443,23 +343,39 @@ static int iommufd_group_setup_msi(struct iommufd_group *igroup, · 20 unmodified lines | |||
| 443 | } | 343 | } |
| 444 | return 0; | 344 | return 0; |
| 445 | } | 345 | } |
| 346 | #else | ||
| 347 | static inline int | ||
| 348 | iommufd_group_setup_msi(struct iommufd_group *igroup, | ||
| 349 | struct iommufd_hwpt_paging *hwpt_paging) | ||
| 350 | { | ||
| 351 | return 0; | ||
| 352 | } | ||
| 353 | #endif | ||
| 354 | |||
| 355 | static bool | ||
| 356 | iommufd_group_first_attach(struct iommufd_group *igroup, ioasid_t pasid) | ||
| 357 | { | ||
| 358 | lockdep_assert_held(&igroup->lock); | ||
| 359 | return !xa_load(&igroup->pasid_attach, pasid); | ||
| 360 | } | ||
| 446 | 361 | ||
| 447 | static int | 362 | static int |
| 448 | iommufd_device_attach_reserved_iova(struct iommufd_device *idev, | 363 | iommufd_device_attach_reserved_iova(struct iommufd_device *idev, |
| 449 | struct iommufd_hwpt_paging *hwpt_paging) | 364 | struct iommufd_hwpt_paging *hwpt_paging) |
| 450 | { | 365 | { |
| 366 | struct iommufd_group *igroup = idev->igroup; | ||
| 451 | int rc; | 367 | int rc; |
| 452 | 368 | ||
| 453 | lockdep_assert_held(&idev->igroup->lock); | 369 | lockdep_assert_held(&igroup->lock); |
| 454 | 370 | ||
| 455 | rc = iopt_table_enforce_dev_resv_regions(&hwpt_paging->ioas->iopt, | 371 | rc = iopt_table_enforce_dev_resv_regions(&hwpt_paging->ioas->iopt, |
| 456 | idev->dev, | 372 | idev->dev, |
| 457 | &idev->igroup->sw_msi_start); | 373 | &igroup->sw_msi_start); |
| 458 | if (rc) | 374 | if (rc) |
| 459 | return rc; | 375 | return rc; |
| 460 | 376 | ||
| 461 | if (list_empty(&idev->igroup->device_list)) { | 377 | if (iommufd_group_first_attach(igroup, IOMMU_NO_PASID)) { |
| 462 | rc = iommufd_group_setup_msi(idev->igroup, hwpt_paging); | 378 | rc = iommufd_group_setup_msi(igroup, hwpt_paging); |
| 463 | if (rc) { | 379 | if (rc) { |
| 464 | iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, | 380 | iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, |
| 465 | idev->dev); | 381 | idev->dev); |
| @@ -471,13 +387,54 @@ iommufd_device_attach_reserved_iova(struct iommufd_device *idev, · 5 unmodified lines | |||
| 471 | 387 | ||
| 472 | /* The device attach/detach/replace helpers for attach_handle */ | 388 | /* The device attach/detach/replace helpers for attach_handle */ |
| 473 | 389 | ||
| 390 | static bool iommufd_device_is_attached(struct iommufd_device *idev, | ||
| 391 | ioasid_t pasid) | ||
| 392 | { | ||
| 393 | struct iommufd_attach *attach; | ||
| 394 | |||
| 395 | attach = xa_load(&idev->igroup->pasid_attach, pasid); | ||
| 396 | return xa_load(&attach->device_array, idev->obj.id); | ||
| 397 | } | ||
| 398 | |||
| 399 | static int iommufd_hwpt_pasid_compat(struct iommufd_hw_pagetable *hwpt, | ||
| 400 | struct iommufd_device *idev, | ||
| 401 | ioasid_t pasid) | ||
| 402 | { | ||
| 403 | struct iommufd_group *igroup = idev->igroup; | ||
| 404 | |||
| 405 | lockdep_assert_held(&igroup->lock); | ||
| 406 | |||
| 407 | if (pasid == IOMMU_NO_PASID) { | ||
| 408 | unsigned long start = IOMMU_NO_PASID; | ||
| 409 | |||
| 410 | if (!hwpt->pasid_compat && | ||
| 411 | xa_find_after(&igroup->pasid_attach, | ||
| 412 | &start, UINT_MAX, XA_PRESENT)) | ||
| 413 | return -EINVAL; | ||
| 414 | } else { | ||
| 415 | struct iommufd_attach *attach; | ||
| 416 | |||
| 417 | if (!hwpt->pasid_compat) | ||
| 418 | return -EINVAL; | ||
| 419 | |||
| 420 | attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID); | ||
| 421 | if (attach && attach->hwpt && !attach->hwpt->pasid_compat) | ||
| 422 | return -EINVAL; | ||
| 423 | } | ||
| 424 | |||
| 425 | return 0; | ||
| 426 | } | ||
| 427 | |||
| 474 | static int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *hwpt, | 428 | static int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *hwpt, |
| 475 | struct iommufd_device *idev) | 429 | struct iommufd_device *idev, |
| 430 | ioasid_t pasid) | ||
| 476 | { | 431 | { |
| 477 | struct iommufd_attach_handle *handle; | 432 | struct iommufd_attach_handle *handle; |
| 478 | int rc; | 433 | int rc; |
| 479 | 434 | ||
| 480 | lockdep_assert_held(&idev->igroup->lock); | 435 | rc = iommufd_hwpt_pasid_compat(hwpt, idev, pasid); |
| 436 | if (rc) | ||
| 437 | return rc; | ||
| 481 | 438 | ||
| 482 | handle = kzalloc(sizeof(*handle), GFP_KERNEL); | 439 | handle = kzalloc(sizeof(*handle), GFP_KERNEL); |
| 483 | if (!handle) | 440 | if (!handle) |
| @@ -490,8 +447,12 @@ static int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *hwpt, · 6 unmodified lines | |||
| 490 | } | 447 | } |
| 491 | 448 | ||
| 492 | handle->idev = idev; | 449 | handle->idev = idev; |
| 493 | rc = iommu_attach_group_handle(hwpt->domain, idev->igroup->group, | 450 | if (pasid == IOMMU_NO_PASID) |
| 494 | &handle->handle); | 451 | rc = iommu_attach_group_handle(hwpt->domain, idev->igroup->group, |
| 452 | &handle->handle); | ||
| 453 | else | ||
| 454 | rc = iommu_attach_device_pasid(hwpt->domain, idev->dev, pasid, | ||
| 455 | &handle->handle); | ||
| 495 | if (rc) | 456 | if (rc) |
| 496 | goto out_disable_iopf; | 457 | goto out_disable_iopf; |
| 497 | 458 | ||
| @@ -506,26 +467,31 @@ out_free_handle: · 8 unmodified lines | |||
| 506 | } | 467 | } |
| 507 | 468 | ||
| 508 | static struct iommufd_attach_handle * | 469 | static struct iommufd_attach_handle * |
| 509 | iommufd_device_get_attach_handle(struct iommufd_device *idev) | 470 | iommufd_device_get_attach_handle(struct iommufd_device *idev, ioasid_t pasid) |
| 510 | { | 471 | { |
| 511 | struct iommu_attach_handle *handle; | 472 | struct iommu_attach_handle *handle; |
| 512 | 473 | ||
| 513 | lockdep_assert_held(&idev->igroup->lock); | 474 | lockdep_assert_held(&idev->igroup->lock); |
| 514 | 475 | ||
| 515 | handle = | 476 | handle = |
| 516 | iommu_attach_handle_get(idev->igroup->group, IOMMU_NO_PASID, 0); | 477 | iommu_attach_handle_get(idev->igroup->group, pasid, 0); |
| 517 | if (IS_ERR(handle)) | 478 | if (IS_ERR(handle)) |
| 518 | return NULL; | 479 | return NULL; |
| 519 | return to_iommufd_handle(handle); | 480 | return to_iommufd_handle(handle); |
| 520 | } | 481 | } |
| 521 | 482 | ||
| 522 | static void iommufd_hwpt_detach_device(struct iommufd_hw_pagetable *hwpt, | 483 | static void iommufd_hwpt_detach_device(struct iommufd_hw_pagetable *hwpt, |
| 523 | struct iommufd_device *idev) | 484 | struct iommufd_device *idev, |
| 485 | ioasid_t pasid) | ||
| 524 | { | 486 | { |
| 525 | struct iommufd_attach_handle *handle; | 487 | struct iommufd_attach_handle *handle; |
| 526 | 488 | ||
| 527 | handle = iommufd_device_get_attach_handle(idev); | 489 | handle = iommufd_device_get_attach_handle(idev, pasid); |
| 528 | iommu_detach_group_handle(hwpt->domain, idev->igroup->group); | 490 | if (pasid == IOMMU_NO_PASID) |
| 491 | iommu_detach_group_handle(hwpt->domain, idev->igroup->group); | ||
| 492 | else | ||
| 493 | iommu_detach_device_pasid(hwpt->domain, idev->dev, pasid); | ||
| 494 | |||
| 529 | if (hwpt->fault) { | 495 | if (hwpt->fault) { |
| 530 | iommufd_auto_response_faults(hwpt, handle); | 496 | iommufd_auto_response_faults(hwpt, handle); |
| 531 | iommufd_fault_iopf_disable(idev); | 497 | iommufd_fault_iopf_disable(idev); |
| @@ -534,13 +500,19 @@ static void iommufd_hwpt_detach_device(struct iommufd_hw_pagetable *hwpt, · 2 unmodified lines | |||
| 534 | } | 500 | } |
| 535 | 501 | ||
| 536 | static int iommufd_hwpt_replace_device(struct iommufd_device *idev, | 502 | static int iommufd_hwpt_replace_device(struct iommufd_device *idev, |
| 503 | ioasid_t pasid, | ||
| 537 | struct iommufd_hw_pagetable *hwpt, | 504 | struct iommufd_hw_pagetable *hwpt, |
| 538 | struct iommufd_hw_pagetable *old) | 505 | struct iommufd_hw_pagetable *old) |
| 539 | { | 506 | { |
| 540 | struct iommufd_attach_handle *handle, *old_handle = | 507 | struct iommufd_attach_handle *handle, *old_handle; |
| 541 | iommufd_device_get_attach_handle(idev); | ||
| 542 | int rc; | 508 | int rc; |
| 543 | 509 | ||
| 510 | rc = iommufd_hwpt_pasid_compat(hwpt, idev, pasid); | ||
| 511 | if (rc) | ||
| 512 | return rc; | ||
| 513 | |||
| 514 | old_handle = iommufd_device_get_attach_handle(idev, pasid); | ||
| 515 | |||
| 544 | handle = kzalloc(sizeof(*handle), GFP_KERNEL); | 516 | handle = kzalloc(sizeof(*handle), GFP_KERNEL); |
| 545 | if (!handle) | 517 | if (!handle) |
| 546 | return -ENOMEM; | 518 | return -ENOMEM; |
| @@ -552,8 +524,12 @@ static int iommufd_hwpt_replace_device(struct iommufd_device *idev, · 5 unmodified lines | |||
| 552 | } | 524 | } |
| 553 | 525 | ||
| 554 | handle->idev = idev; | 526 | handle->idev = idev; |
| 555 | rc = iommu_replace_group_handle(idev->igroup->group, hwpt->domain, | 527 | if (pasid == IOMMU_NO_PASID) |
| 556 | &handle->handle); | 528 | rc = iommu_replace_group_handle(idev->igroup->group, |
| 529 | hwpt->domain, &handle->handle); | ||
| 530 | else | ||
| 531 | rc = iommu_replace_device_pasid(hwpt->domain, idev->dev, | ||
| 532 | pasid, &handle->handle); | ||
| 557 | if (rc) | 533 | if (rc) |
| 558 | goto out_disable_iopf; | 534 | goto out_disable_iopf; |
| 559 | 535 | ||
| @@ -575,22 +551,51 @@ out_free_handle: · 15 unmodified lines | |||
| 575 | } | 551 | } |
| 576 | 552 | ||
| 577 | int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt, | 553 | int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt, |
| 578 | struct iommufd_device *idev) | 554 | struct iommufd_device *idev, ioasid_t pasid) |
| 579 | { | 555 | { |
| 580 | struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt); | 556 | struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt); |
| 557 | bool attach_resv = hwpt_paging && pasid == IOMMU_NO_PASID; | ||
| 558 | struct iommufd_group *igroup = idev->igroup; | ||
| 559 | struct iommufd_hw_pagetable *old_hwpt; | ||
| 560 | struct iommufd_attach *attach; | ||
| 581 | int rc; | 561 | int rc; |
| 582 | 562 | ||
| 583 | mutex_lock(&idev->igroup->lock); | 563 | mutex_lock(&igroup->lock); |
| 584 | 564 | ||
| 585 | if (idev->igroup->hwpt != NULL && idev->igroup->hwpt != hwpt) { | 565 | attach = xa_cmpxchg(&igroup->pasid_attach, pasid, NULL, |
| 586 | rc = -EINVAL; | 566 | XA_ZERO_ENTRY, GFP_KERNEL); |
| 567 | if (xa_is_err(attach)) { | ||
| 568 | rc = xa_err(attach); | ||
| 587 | goto err_unlock; | 569 | goto err_unlock; |
| 588 | } | 570 | } |
| 589 | 571 | ||
| 590 | if (hwpt_paging) { | 572 | if (!attach) { |
| 573 | attach = kzalloc(sizeof(*attach), GFP_KERNEL); | ||
| 574 | if (!attach) { | ||
| 575 | rc = -ENOMEM; | ||
| 576 | goto err_release_pasid; | ||
| 577 | } | ||
| 578 | xa_init(&attach->device_array); | ||
| 579 | } | ||
| 580 | |||
| 581 | old_hwpt = attach->hwpt; | ||
| 582 | |||
| 583 | rc = xa_insert(&attach->device_array, idev->obj.id, XA_ZERO_ENTRY, | ||
| 584 | GFP_KERNEL); | ||
| 585 | if (rc) { | ||
| 586 | WARN_ON(rc == -EBUSY && !old_hwpt); | ||
| 587 | goto err_free_attach; | ||
| 588 | } | ||
| 589 | |||
| 590 | if (old_hwpt && old_hwpt != hwpt) { | ||
| 591 | rc = -EINVAL; | ||
| 592 | goto err_release_devid; | ||
| 593 | } | ||
| 594 | |||
| 595 | if (attach_resv) { | ||
| 591 | rc = iommufd_device_attach_reserved_iova(idev, hwpt_paging); | 596 | rc = iommufd_device_attach_reserved_iova(idev, hwpt_paging); |
| 592 | if (rc) | 597 | if (rc) |
| 593 | goto err_unlock; | 598 | goto err_release_devid; |
| 594 | } | 599 | } |
| 595 | 600 | ||
| 596 | /* | 601 | /* |
| @@ -600,51 +605,74 @@ int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt, · 3 unmodified lines | |||
| 600 | * reserved regions are only updated during individual device | 605 | * reserved regions are only updated during individual device |
| 601 | * attachment. | 606 | * attachment. |
| 602 | */ | 607 | */ |
| 603 | if (list_empty(&idev->igroup->device_list)) { | 608 | if (iommufd_group_first_attach(igroup, pasid)) { |
| 604 | rc = iommufd_hwpt_attach_device(hwpt, idev); | 609 | rc = iommufd_hwpt_attach_device(hwpt, idev, pasid); |
| 605 | if (rc) | 610 | if (rc) |
| 606 | goto err_unresv; | 611 | goto err_unresv; |
| 607 | idev->igroup->hwpt = hwpt; | 612 | attach->hwpt = hwpt; |
| 613 | WARN_ON(xa_is_err(xa_store(&igroup->pasid_attach, pasid, attach, | ||
| 614 | GFP_KERNEL))); | ||
| 608 | } | 615 | } |
| 609 | refcount_inc(&hwpt->obj.users); | 616 | refcount_inc(&hwpt->obj.users); |
| 610 | list_add_tail(&idev->group_item, &idev->igroup->device_list); | 617 | WARN_ON(xa_is_err(xa_store(&attach->device_array, idev->obj.id, |
| 611 | mutex_unlock(&idev->igroup->lock); | 618 | idev, GFP_KERNEL))); |
| 619 | mutex_unlock(&igroup->lock); | ||
| 612 | return 0; | 620 | return 0; |
| 613 | err_unresv: | 621 | err_unresv: |
| 614 | if (hwpt_paging) | 622 | if (attach_resv) |
| 615 | iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, idev->dev); | 623 | iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, idev->dev); |
| 624 | err_release_devid: | ||
| 625 | xa_release(&attach->device_array, idev->obj.id); | ||
| 626 | err_free_attach: | ||
| 627 | if (iommufd_group_first_attach(igroup, pasid)) | ||
| 628 | kfree(attach); | ||
| 629 | err_release_pasid: | ||
| 630 | if (iommufd_group_first_attach(igroup, pasid)) | ||
| 631 | xa_release(&igroup->pasid_attach, pasid); | ||
| 616 | err_unlock: | 632 | err_unlock: |
| 617 | mutex_unlock(&idev->igroup->lock); | 633 | mutex_unlock(&igroup->lock); |
| 618 | return rc; | 634 | return rc; |
| 619 | } | 635 | } |
| 620 | 636 | ||
| 621 | struct iommufd_hw_pagetable * | 637 | struct iommufd_hw_pagetable * |
| 622 | iommufd_hw_pagetable_detach(struct iommufd_device *idev) | 638 | iommufd_hw_pagetable_detach(struct iommufd_device *idev, ioasid_t pasid) |
| 623 | { | 639 | { |
| 624 | struct iommufd_hw_pagetable *hwpt = idev->igroup->hwpt; | 640 | struct iommufd_group *igroup = idev->igroup; |
| 625 | struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt); | 641 | struct iommufd_hwpt_paging *hwpt_paging; |
| 642 | struct iommufd_hw_pagetable *hwpt; | ||
| 643 | struct iommufd_attach *attach; | ||
| 644 | |||
| 645 | mutex_lock(&igroup->lock); | ||
| 646 | attach = xa_load(&igroup->pasid_attach, pasid); | ||
| 647 | if (!attach) { | ||
| 648 | mutex_unlock(&igroup->lock); | ||
| 649 | return NULL; | ||
| 650 | } | ||
| 626 | 651 | ||
| 627 | mutex_lock(&idev->igroup->lock); | 652 | hwpt = attach->hwpt; |
| 628 | list_del(&idev->group_item); | 653 | hwpt_paging = find_hwpt_paging(hwpt); |
| 629 | if (list_empty(&idev->igroup->device_list)) { | 654 | |
| 630 | iommufd_hwpt_detach_device(hwpt, idev); | 655 | xa_erase(&attach->device_array, idev->obj.id); |
| 631 | idev->igroup->hwpt = NULL; | 656 | if (xa_empty(&attach->device_array)) { |
| 657 | iommufd_hwpt_detach_device(hwpt, idev, pasid); | ||
| 658 | xa_erase(&igroup->pasid_attach, pasid); | ||
| 659 | kfree(attach); | ||
| 632 | } | 660 | } |
| 633 | if (hwpt_paging) | 661 | if (hwpt_paging && pasid == IOMMU_NO_PASID) |
| 634 | iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, idev->dev); | 662 | iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, idev->dev); |
| 635 | mutex_unlock(&idev->igroup->lock); | 663 | mutex_unlock(&igroup->lock); |
| 636 | 664 | ||
| 637 | /* Caller must destroy hwpt */ | 665 | /* Caller must destroy hwpt */ |
| 638 | return hwpt; | 666 | return hwpt; |
| 639 | } | 667 | } |
| 640 | 668 | ||
| 641 | static struct iommufd_hw_pagetable * | 669 | static struct iommufd_hw_pagetable * |
| 642 | iommufd_device_do_attach(struct iommufd_device *idev, | 670 | iommufd_device_do_attach(struct iommufd_device *idev, ioasid_t pasid, |
| 643 | struct iommufd_hw_pagetable *hwpt) | 671 | struct iommufd_hw_pagetable *hwpt) |
| 644 | { | 672 | { |
| 645 | int rc; | 673 | int rc; |
| 646 | 674 | ||
| 647 | rc = iommufd_hw_pagetable_attach(hwpt, idev); | 675 | rc = iommufd_hw_pagetable_attach(hwpt, idev, pasid); |
| 648 | if (rc) | 676 | if (rc) |
| 649 | return ERR_PTR(rc); | 677 | return ERR_PTR(rc); |
| 650 | return NULL; | 678 | return NULL; |
| @@ -654,11 +682,14 @@ static void · 3 unmodified lines | |||
| 654 | iommufd_group_remove_reserved_iova(struct iommufd_group *igroup, | 682 | iommufd_group_remove_reserved_iova(struct iommufd_group *igroup, |
| 655 | struct iommufd_hwpt_paging *hwpt_paging) | 683 | struct iommufd_hwpt_paging *hwpt_paging) |
| 656 | { | 684 | { |
| 685 | struct iommufd_attach *attach; | ||
| 657 | struct iommufd_device *cur; | 686 | struct iommufd_device *cur; |
| 687 | unsigned long index; | ||
| 658 | 688 | ||
| 659 | lockdep_assert_held(&igroup->lock); | 689 | lockdep_assert_held(&igroup->lock); |
| 660 | 690 | ||
| 661 | list_for_each_entry(cur, &igroup->device_list, group_item) | 691 | attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID); |
| 692 | xa_for_each(&attach->device_array, index, cur) | ||
| 662 | iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, cur->dev); | 693 | iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, cur->dev); |
| 663 | } | 694 | } |
| 664 | 695 | ||
| @@ -667,14 +698,17 @@ iommufd_group_do_replace_reserved_iova(struct iommufd_group *igroup, · 2 unmodified lines | |||
| 667 | struct iommufd_hwpt_paging *hwpt_paging) | 698 | struct iommufd_hwpt_paging *hwpt_paging) |
| 668 | { | 699 | { |
| 669 | struct iommufd_hwpt_paging *old_hwpt_paging; | 700 | struct iommufd_hwpt_paging *old_hwpt_paging; |
| 701 | struct iommufd_attach *attach; | ||
| 670 | struct iommufd_device *cur; | 702 | struct iommufd_device *cur; |
| 703 | unsigned long index; | ||
| 671 | int rc; | 704 | int rc; |
| 672 | 705 | ||
| 673 | lockdep_assert_held(&igroup->lock); | 706 | lockdep_assert_held(&igroup->lock); |
| 674 | 707 | ||
| 675 | old_hwpt_paging = find_hwpt_paging(igroup->hwpt); | 708 | attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID); |
| 709 | old_hwpt_paging = find_hwpt_paging(attach->hwpt); | ||
| 676 | if (!old_hwpt_paging || hwpt_paging->ioas != old_hwpt_paging->ioas) { | 710 | if (!old_hwpt_paging || hwpt_paging->ioas != old_hwpt_paging->ioas) { |
| 677 | list_for_each_entry(cur, &igroup->device_list, group_item) { | 711 | xa_for_each(&attach->device_array, index, cur) { |
| 678 | rc = iopt_table_enforce_dev_resv_regions( | 712 | rc = iopt_table_enforce_dev_resv_regions( |
| 679 | &hwpt_paging->ioas->iopt, cur->dev, NULL); | 713 | &hwpt_paging->ioas->iopt, cur->dev, NULL); |
| 680 | if (rc) | 714 | if (rc) |
| @@ -693,69 +727,81 @@ err_unresv: · 12 unmodified lines | |||
| 693 | } | 727 | } |
| 694 | 728 | ||
| 695 | static struct iommufd_hw_pagetable * | 729 | static struct iommufd_hw_pagetable * |
| 696 | iommufd_device_do_replace(struct iommufd_device *idev, | 730 | iommufd_device_do_replace(struct iommufd_device *idev, ioasid_t pasid, |
| 697 | struct iommufd_hw_pagetable *hwpt) | 731 | struct iommufd_hw_pagetable *hwpt) |
| 698 | { | 732 | { |
| 699 | struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt); | 733 | struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt); |
| 734 | bool attach_resv = hwpt_paging && pasid == IOMMU_NO_PASID; | ||
| 700 | struct iommufd_hwpt_paging *old_hwpt_paging; | 735 | struct iommufd_hwpt_paging *old_hwpt_paging; |
| 701 | struct iommufd_group *igroup = idev->igroup; | 736 | struct iommufd_group *igroup = idev->igroup; |
| 702 | struct iommufd_hw_pagetable *old_hwpt; | 737 | struct iommufd_hw_pagetable *old_hwpt; |
| 738 | struct iommufd_attach *attach; | ||
| 703 | unsigned int num_devices; | 739 | unsigned int num_devices; |
| 704 | int rc; | 740 | int rc; |
| 705 | 741 | ||
| 706 | mutex_lock(&idev->igroup->lock); | 742 | mutex_lock(&igroup->lock); |
| 743 | |||
| 744 | attach = xa_load(&igroup->pasid_attach, pasid); | ||
| 745 | if (!attach) { | ||
| 746 | rc = -EINVAL; | ||
| 747 | goto err_unlock; | ||
| 748 | } | ||
| 749 | |||
| 750 | old_hwpt = attach->hwpt; | ||
| 707 | 751 | ||
| 708 | if (igroup->hwpt == NULL) { | 752 | WARN_ON(!old_hwpt || xa_empty(&attach->device_array)); |
| 753 | |||
| 754 | if (!iommufd_device_is_attached(idev, pasid)) { | ||
| 709 | rc = -EINVAL; | 755 | rc = -EINVAL; |
| 710 | goto err_unlock; | 756 | goto err_unlock; |
| 711 | } | 757 | } |
| 712 | 758 | ||
| 713 | if (hwpt == igroup->hwpt) { | 759 | if (hwpt == old_hwpt) { |
| 714 | mutex_unlock(&idev->igroup->lock); | 760 | mutex_unlock(&igroup->lock); |
| 715 | return NULL; | 761 | return NULL; |
| 716 | } | 762 | } |
| 717 | 763 | ||
| 718 | old_hwpt = igroup->hwpt; | 764 | if (attach_resv) { |
| 719 | if (hwpt_paging) { | ||
| 720 | rc = iommufd_group_do_replace_reserved_iova(igroup, hwpt_paging); | 765 | rc = iommufd_group_do_replace_reserved_iova(igroup, hwpt_paging); |
| 721 | if (rc) | 766 | if (rc) |
| 722 | goto err_unlock; | 767 | goto err_unlock; |
| 723 | } | 768 | } |
| 724 | 769 | ||
| 725 | rc = iommufd_hwpt_replace_device(idev, hwpt, old_hwpt); | 770 | rc = iommufd_hwpt_replace_device(idev, pasid, hwpt, old_hwpt); |
| 726 | if (rc) | 771 | if (rc) |
| 727 | goto err_unresv; | 772 | goto err_unresv; |
| 728 | 773 | ||
| 729 | old_hwpt_paging = find_hwpt_paging(old_hwpt); | 774 | old_hwpt_paging = find_hwpt_paging(old_hwpt); |
| 730 | if (old_hwpt_paging && | 775 | if (old_hwpt_paging && pasid == IOMMU_NO_PASID && |
| 731 | (!hwpt_paging || hwpt_paging->ioas != old_hwpt_paging->ioas)) | 776 | (!hwpt_paging || hwpt_paging->ioas != old_hwpt_paging->ioas)) |
| 732 | iommufd_group_remove_reserved_iova(igroup, old_hwpt_paging); | 777 | iommufd_group_remove_reserved_iova(igroup, old_hwpt_paging); |
| 733 | 778 | ||
| 734 | igroup->hwpt = hwpt; | 779 | attach->hwpt = hwpt; |
| 735 | 780 | ||
| 736 | num_devices = list_count_nodes(&igroup->device_list); | 781 | num_devices = iommufd_group_device_num(igroup, pasid); |
| 737 | /* | 782 | /* |
| 738 | * Move the refcounts held by the device_list to the new hwpt. Retain a | 783 | * Move the refcounts held by the device_array to the new hwpt. Retain a |
| 739 | * refcount for this thread as the caller will free it. | 784 | * refcount for this thread as the caller will free it. |
| 740 | */ | 785 | */ |
| 741 | refcount_add(num_devices, &hwpt->obj.users); | 786 | refcount_add(num_devices, &hwpt->obj.users); |
| 742 | if (num_devices > 1) | 787 | if (num_devices > 1) |
| 743 | WARN_ON(refcount_sub_and_test(num_devices - 1, | 788 | WARN_ON(refcount_sub_and_test(num_devices - 1, |
| 744 | &old_hwpt->obj.users)); | 789 | &old_hwpt->obj.users)); |
| 745 | mutex_unlock(&idev->igroup->lock); | 790 | mutex_unlock(&igroup->lock); |
| 746 | 791 | ||
| 747 | /* Caller must destroy old_hwpt */ | 792 | /* Caller must destroy old_hwpt */ |
| 748 | return old_hwpt; | 793 | return old_hwpt; |
| 749 | err_unresv: | 794 | err_unresv: |
| 750 | if (hwpt_paging) | 795 | if (attach_resv) |
| 751 | iommufd_group_remove_reserved_iova(igroup, hwpt_paging); | 796 | iommufd_group_remove_reserved_iova(igroup, hwpt_paging); |
| 752 | err_unlock: | 797 | err_unlock: |
| 753 | mutex_unlock(&idev->igroup->lock); | 798 | mutex_unlock(&igroup->lock); |
| 754 | return ERR_PTR(rc); | 799 | return ERR_PTR(rc); |
| 755 | } | 800 | } |
| 756 | 801 | ||
| 757 | typedef struct iommufd_hw_pagetable *(*attach_fn)( | 802 | typedef struct iommufd_hw_pagetable *(*attach_fn)( |
| 758 | struct iommufd_device *idev, struct iommufd_hw_pagetable *hwpt); | 803 | struct iommufd_device *idev, ioasid_t pasid, |
| 804 | struct iommufd_hw_pagetable *hwpt); | ||
| 759 | 805 | ||
| 760 | /* | 806 | /* |
| 761 | * When automatically managing the domains we search for a compatible domain in | 807 | * When automatically managing the domains we search for a compatible domain in |
| @@ -763,7 +809,7 @@ typedef struct iommufd_hw_pagetable *(*attach_fn)( · 1 unmodified line | |||
| 763 | * Automatic domain selection will never pick a manually created domain. | 809 | * Automatic domain selection will never pick a manually created domain. |
| 764 | */ | 810 | */ |
| 765 | static struct iommufd_hw_pagetable * | 811 | static struct iommufd_hw_pagetable * |
| 766 | iommufd_device_auto_get_domain(struct iommufd_device *idev, | 812 | iommufd_device_auto_get_domain(struct iommufd_device *idev, ioasid_t pasid, |
| 767 | struct iommufd_ioas *ioas, u32 *pt_id, | 813 | struct iommufd_ioas *ioas, u32 *pt_id, |
| 768 | attach_fn do_attach) | 814 | attach_fn do_attach) |
| 769 | { | 815 | { |
| @@ -792,7 +838,7 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, · 22 unmodified lines | |||
| 792 | hwpt = &hwpt_paging->common; | 838 | hwpt = &hwpt_paging->common; |
| 793 | if (!iommufd_lock_obj(&hwpt->obj)) | 839 | if (!iommufd_lock_obj(&hwpt->obj)) |
| 794 | continue; | 840 | continue; |
| 795 | destroy_hwpt = (*do_attach)(idev, hwpt); | 841 | destroy_hwpt = (*do_attach)(idev, pasid, hwpt); |
| 796 | if (IS_ERR(destroy_hwpt)) { | 842 | if (IS_ERR(destroy_hwpt)) { |
| 797 | iommufd_put_object(idev->ictx, &hwpt->obj); | 843 | iommufd_put_object(idev->ictx, &hwpt->obj); |
| 798 | /* | 844 | /* |
| @@ -810,8 +856,8 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, · 11 unmodified lines | |||
| 810 | goto out_unlock; | 856 | goto out_unlock; |
| 811 | } | 857 | } |
| 812 | 858 | ||
| 813 | hwpt_paging = iommufd_hwpt_paging_alloc(idev->ictx, ioas, idev, 0, | 859 | hwpt_paging = iommufd_hwpt_paging_alloc(idev->ictx, ioas, idev, pasid, |
| 814 | immediate_attach, NULL); | 860 | 0, immediate_attach, NULL); |
| 815 | if (IS_ERR(hwpt_paging)) { | 861 | if (IS_ERR(hwpt_paging)) { |
| 816 | destroy_hwpt = ERR_CAST(hwpt_paging); | 862 | destroy_hwpt = ERR_CAST(hwpt_paging); |
| 817 | goto out_unlock; | 863 | goto out_unlock; |
| @@ -819,7 +865,7 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, · 1 unmodified line | |||
| 819 | hwpt = &hwpt_paging->common; | 865 | hwpt = &hwpt_paging->common; |
| 820 | 866 | ||
| 821 | if (!immediate_attach) { | 867 | if (!immediate_attach) { |
| 822 | destroy_hwpt = (*do_attach)(idev, hwpt); | 868 | destroy_hwpt = (*do_attach)(idev, pasid, hwpt); |
| 823 | if (IS_ERR(destroy_hwpt)) | 869 | if (IS_ERR(destroy_hwpt)) |
| 824 | goto out_abort; | 870 | goto out_abort; |
| 825 | } else { | 871 | } else { |
| @@ -840,8 +886,9 @@ out_unlock: · 14 unmodified lines | |||
| 840 | return destroy_hwpt; | 886 | return destroy_hwpt; |
| 841 | } | 887 | } |
| 842 | 888 | ||
| 843 | static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id, | 889 | static int iommufd_device_change_pt(struct iommufd_device *idev, |
| 844 | attach_fn do_attach) | 890 | ioasid_t pasid, |
| 891 | u32 *pt_id, attach_fn do_attach) | ||
| 845 | { | 892 | { |
| 846 | struct iommufd_hw_pagetable *destroy_hwpt; | 893 | struct iommufd_hw_pagetable *destroy_hwpt; |
| 847 | struct iommufd_object *pt_obj; | 894 | struct iommufd_object *pt_obj; |
| @@ -856,7 +903,7 @@ static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id, · 8 unmodified lines | |||
| 856 | struct iommufd_hw_pagetable *hwpt = | 903 | struct iommufd_hw_pagetable *hwpt = |
| 857 | container_of(pt_obj, struct iommufd_hw_pagetable, obj); | 904 | container_of(pt_obj, struct iommufd_hw_pagetable, obj); |
| 858 | 905 | ||
| 859 | destroy_hwpt = (*do_attach)(idev, hwpt); | 906 | destroy_hwpt = (*do_attach)(idev, pasid, hwpt); |
| 860 | if (IS_ERR(destroy_hwpt)) | 907 | if (IS_ERR(destroy_hwpt)) |
| 861 | goto out_put_pt_obj; | 908 | goto out_put_pt_obj; |
| 862 | break; | 909 | break; |
| @@ -865,8 +912,8 @@ static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id, · 2 unmodified lines | |||
| 865 | struct iommufd_ioas *ioas = | 912 | struct iommufd_ioas *ioas = |
| 866 | container_of(pt_obj, struct iommufd_ioas, obj); | 913 | container_of(pt_obj, struct iommufd_ioas, obj); |
| 867 | 914 | ||
| 868 | destroy_hwpt = iommufd_device_auto_get_domain(idev, ioas, pt_id, | 915 | destroy_hwpt = iommufd_device_auto_get_domain(idev, pasid, ioas, |
| 869 | do_attach); | 916 | pt_id, do_attach); |
| 870 | if (IS_ERR(destroy_hwpt)) | 917 | if (IS_ERR(destroy_hwpt)) |
| 871 | goto out_put_pt_obj; | 918 | goto out_put_pt_obj; |
| 872 | break; | 919 | break; |
| @@ -888,22 +935,26 @@ out_put_pt_obj: · 15 unmodified lines | |||
| 888 | } | 935 | } |
| 889 | 936 | ||
| 890 | /** | 937 | /** |
| 891 | * iommufd_device_attach - Connect a device to an iommu_domain | 938 | * iommufd_device_attach - Connect a device/pasid to an iommu_domain |
| 892 | * @idev: device to attach | 939 | * @idev: device to attach |
| 940 | * @pasid: pasid to attach | ||
| 893 | * @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING | 941 | * @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING |
| 894 | * Output the IOMMUFD_OBJ_HWPT_PAGING ID | 942 | * Output the IOMMUFD_OBJ_HWPT_PAGING ID |
| 895 | * | 943 | * |
| 896 | * This connects the device to an iommu_domain, either automatically or manually | 944 | * This connects the device/pasid to an iommu_domain, either automatically |
| 897 | * selected. Once this completes the device could do DMA. | 945 | * or manually selected. Once this completes the device could do DMA with |
| 946 | * @pasid. @pasid is IOMMU_NO_PASID if this attach is for no pasid usage. | ||
| 898 | * | 947 | * |
| 899 | * The caller should return the resulting pt_id back to userspace. | 948 | * The caller should return the resulting pt_id back to userspace. |
| 900 | * This function is undone by calling iommufd_device_detach(). | 949 | * This function is undone by calling iommufd_device_detach(). |
| 901 | */ | 950 | */ |
| 902 | int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id) | 951 | int iommufd_device_attach(struct iommufd_device *idev, ioasid_t pasid, |
| 952 | u32 *pt_id) | ||
| 903 | { | 953 | { |
| 904 | int rc; | 954 | int rc; |
| 905 | 955 | ||
| 906 | rc = iommufd_device_change_pt(idev, pt_id, &iommufd_device_do_attach); | 956 | rc = iommufd_device_change_pt(idev, pasid, pt_id, |
| 957 | &iommufd_device_do_attach); | ||
| 907 | if (rc) | 958 | if (rc) |
| 908 | return rc; | 959 | return rc; |
| 909 | 960 | ||
| @@ -917,8 +968,9 @@ int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id) · 7 unmodified lines | |||
| 917 | EXPORT_SYMBOL_NS_GPL(iommufd_device_attach, "IOMMUFD"); | 968 | EXPORT_SYMBOL_NS_GPL(iommufd_device_attach, "IOMMUFD"); |
| 918 | 969 | ||
| 919 | /** | 970 | /** |
| 920 | * iommufd_device_replace - Change the device's iommu_domain | 971 | * iommufd_device_replace - Change the device/pasid's iommu_domain |
| 921 | * @idev: device to change | 972 | * @idev: device to change |
| 973 | * @pasid: pasid to change | ||
| 922 | * @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING | 974 | * @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING |
| 923 | * Output the IOMMUFD_OBJ_HWPT_PAGING ID | 975 | * Output the IOMMUFD_OBJ_HWPT_PAGING ID |
| 924 | * | 976 | * |
| @@ -929,27 +981,33 @@ EXPORT_SYMBOL_NS_GPL(iommufd_device_attach, "IOMMUFD"); · 4 unmodified lines | |||
| 929 | * | 981 | * |
| 930 | * If it fails then no change is made to the attachment. The iommu driver may | 982 | * If it fails then no change is made to the attachment. The iommu driver may |
| 931 | * implement this so there is no disruption in translation. This can only be | 983 | * implement this so there is no disruption in translation. This can only be |
| 932 | * called if iommufd_device_attach() has already succeeded. | 984 | * called if iommufd_device_attach() has already succeeded. @pasid is |
| 985 | * IOMMU_NO_PASID for no pasid usage. | ||
| 933 | */ | 986 | */ |
| 934 | int iommufd_device_replace(struct iommufd_device *idev, u32 *pt_id) | 987 | int iommufd_device_replace(struct iommufd_device *idev, ioasid_t pasid, |
| 988 | u32 *pt_id) | ||
| 935 | { | 989 | { |
| 936 | return iommufd_device_change_pt(idev, pt_id, | 990 | return iommufd_device_change_pt(idev, pasid, pt_id, |
| 937 | &iommufd_device_do_replace); | 991 | &iommufd_device_do_replace); |
| 938 | } | 992 | } |
| 939 | EXPORT_SYMBOL_NS_GPL(iommufd_device_replace, "IOMMUFD"); | 993 | EXPORT_SYMBOL_NS_GPL(iommufd_device_replace, "IOMMUFD"); |
| 940 | 994 | ||
| 941 | /** | 995 | /** |
| 942 | * iommufd_device_detach - Disconnect a device to an iommu_domain | 996 | * iommufd_device_detach - Disconnect a device/device to an iommu_domain |
| 943 | * @idev: device to detach | 997 | * @idev: device to detach |
| 998 | * @pasid: pasid to detach | ||
| 944 | * | 999 | * |
| 945 | * Undo iommufd_device_attach(). This disconnects the idev from the previously | 1000 | * Undo iommufd_device_attach(). This disconnects the idev from the previously |
| 946 | * attached pt_id. The device returns back to a blocked DMA translation. | 1001 | * attached pt_id. The device returns back to a blocked DMA translation. |
| 1002 | * @pasid is IOMMU_NO_PASID for no pasid usage. | ||
| 947 | */ | 1003 | */ |
| 948 | void iommufd_device_detach(struct iommufd_device *idev) | 1004 | void iommufd_device_detach(struct iommufd_device *idev, ioasid_t pasid) |
| 949 | { | 1005 | { |
| 950 | struct iommufd_hw_pagetable *hwpt; | 1006 | struct iommufd_hw_pagetable *hwpt; |
| 951 | 1007 | ||
| 952 | hwpt = iommufd_hw_pagetable_detach(idev); | 1008 | hwpt = iommufd_hw_pagetable_detach(idev, pasid); |
| 1009 | if (!hwpt) | ||
| 1010 | return; | ||
| 953 | iommufd_hw_pagetable_put(idev->ictx, hwpt); | 1011 | iommufd_hw_pagetable_put(idev->ictx, hwpt); |
| 954 | refcount_dec(&idev->obj.users); | 1012 | refcount_dec(&idev->obj.users); |
| 955 | } | 1013 | } |
| @@ -1349,7 +1407,7 @@ int iommufd_access_rw(struct iommufd_access *access, unsigned long iova, · 393 unmodified lines | |||
| 1349 | struct io_pagetable *iopt; | 1407 | struct io_pagetable *iopt; |
| 1350 | struct iopt_area *area; | 1408 | struct iopt_area *area; |
| 1351 | unsigned long last_iova; | 1409 | unsigned long last_iova; |
| 1352 | int rc; | 1410 | int rc = -EINVAL; |
| 1353 | 1411 | ||
| 1354 | if (!length) | 1412 | if (!length) |
| 1355 | return -EINVAL; | 1413 | return -EINVAL; |
| @@ -1405,7 +1463,8 @@ int iommufd_get_hw_info(struct iommufd_ucmd *ucmd) · 49 unmodified lines | |||
| 1405 | void *data; | 1463 | void *data; |
| 1406 | int rc; | 1464 | int rc; |
| 1407 | 1465 | ||
| 1408 | if (cmd->flags || cmd->__reserved) | 1466 | if (cmd->flags || cmd->__reserved[0] || cmd->__reserved[1] || |
| 1467 | cmd->__reserved[2]) | ||
| 1409 | return -EOPNOTSUPP; | 1468 | return -EOPNOTSUPP; |
| 1410 | 1469 | ||
| 1411 | idev = iommufd_get_device(ucmd, cmd->dev_id); | 1470 | idev = iommufd_get_device(ucmd, cmd->dev_id); |
| @@ -1462,6 +1521,36 @@ int iommufd_get_hw_info(struct iommufd_ucmd *ucmd) · 50 unmodified lines | |||
| 1462 | if (device_iommu_capable(idev->dev, IOMMU_CAP_DIRTY_TRACKING)) | 1521 | if (device_iommu_capable(idev->dev, IOMMU_CAP_DIRTY_TRACKING)) |
| 1463 | cmd->out_capabilities |= IOMMU_HW_CAP_DIRTY_TRACKING; | 1522 | cmd->out_capabilities |= IOMMU_HW_CAP_DIRTY_TRACKING; |
| 1464 | 1523 | ||
| 1524 | cmd->out_max_pasid_log2 = 0; | ||
| 1525 | /* | ||
| 1526 | * Currently, all iommu drivers enable PASID in the probe_device() | ||
| 1527 | * op if iommu and device supports it. So the max_pasids stored in | ||
| 1528 | * dev->iommu indicates both PASID support and enable status. A | ||
| 1529 | * non-zero dev->iommu->max_pasids means PASID is supported and | ||
| 1530 | * enabled. The iommufd only reports PASID capability to userspace | ||
| 1531 | * if it's enabled. | ||
| 1532 | */ | ||
| 1533 | if (idev->dev->iommu->max_pasids) { | ||
| 1534 | cmd->out_max_pasid_log2 = ilog2(idev->dev->iommu->max_pasids); | ||
| 1535 | |||
| 1536 | if (dev_is_pci(idev->dev)) { | ||
| 1537 | struct pci_dev *pdev = to_pci_dev(idev->dev); | ||
| 1538 | int ctrl; | ||
| 1539 | |||
| 1540 | ctrl = pci_pasid_status(pdev); | ||
| 1541 | |||
| 1542 | WARN_ON_ONCE(ctrl < 0 || | ||
| 1543 | !(ctrl & PCI_PASID_CTRL_ENABLE)); | ||
| 1544 | |||
| 1545 | if (ctrl & PCI_PASID_CTRL_EXEC) | ||
| 1546 | cmd->out_capabilities |= | ||
| 1547 | IOMMU_HW_CAP_PCI_PASID_EXEC; | ||
| 1548 | if (ctrl & PCI_PASID_CTRL_PRIV) | ||
| 1549 | cmd->out_capabilities |= | ||
| 1550 | IOMMU_HW_CAP_PCI_PASID_PRIV; | ||
| 1551 | } | ||
| 1552 | } | ||
| 1553 | |||
| 1465 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); | 1554 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); |
| 1466 | out_free: | 1555 | out_free: |
| 1467 | kfree(data); | 1556 | kfree(data); |
| @@ -49,5 +49,203 @@ struct device *iommufd_viommu_find_dev(struct iommufd_viommu *viommu, · 48 unmodified lines | |||
| 49 | } | 49 | } |
| 50 | EXPORT_SYMBOL_NS_GPL(iommufd_viommu_find_dev, "IOMMUFD"); | 50 | EXPORT_SYMBOL_NS_GPL(iommufd_viommu_find_dev, "IOMMUFD"); |
| 51 | 51 | ||
| 52 | /* Return -ENOENT if device is not associated to the vIOMMU */ | ||
| 53 | int iommufd_viommu_get_vdev_id(struct iommufd_viommu *viommu, | ||
| 54 | struct device *dev, unsigned long *vdev_id) | ||
| 55 | { | ||
| 56 | struct iommufd_vdevice *vdev; | ||
| 57 | unsigned long index; | ||
| 58 | int rc = -ENOENT; | ||
| 59 | |||
| 60 | if (WARN_ON_ONCE(!vdev_id)) | ||
| 61 | return -EINVAL; | ||
| 62 | |||
| 63 | xa_lock(&viommu->vdevs); | ||
| 64 | xa_for_each(&viommu->vdevs, index, vdev) { | ||
| 65 | if (vdev->dev == dev) { | ||
| 66 | *vdev_id = vdev->id; | ||
| 67 | rc = 0; | ||
| 68 | break; | ||
| 69 | } | ||
| 70 | } | ||
| 71 | xa_unlock(&viommu->vdevs); | ||
| 72 | return rc; | ||
| 73 | } | ||
| 74 | EXPORT_SYMBOL_NS_GPL(iommufd_viommu_get_vdev_id, "IOMMUFD"); | ||
| 75 | |||
| 76 | /* | ||
| 77 | * Typically called in driver's threaded IRQ handler. | ||
| 78 | * The @type and @event_data must be defined in include/uapi/linux/iommufd.h | ||
| 79 | */ | ||
| 80 | int iommufd_viommu_report_event(struct iommufd_viommu *viommu, | ||
| 81 | enum iommu_veventq_type type, void *event_data, | ||
| 82 | size_t data_len) | ||
| 83 | { | ||
| 84 | struct iommufd_veventq *veventq; | ||
| 85 | struct iommufd_vevent *vevent; | ||
| 86 | int rc = 0; | ||
| 87 | |||
| 88 | if (WARN_ON_ONCE(!data_len || !event_data)) | ||
| 89 | return -EINVAL; | ||
| 90 | |||
| 91 | down_read(&viommu->veventqs_rwsem); | ||
| 92 | |||
| 93 | veventq = iommufd_viommu_find_veventq(viommu, type); | ||
| 94 | if (!veventq) { | ||
| 95 | rc = -EOPNOTSUPP; | ||
| 96 | goto out_unlock_veventqs; | ||
| 97 | } | ||
| 98 | |||
| 99 | spin_lock(&veventq->common.lock); | ||
| 100 | if (veventq->num_events == veventq->depth) { | ||
| 101 | vevent = &veventq->lost_events_header; | ||
| 102 | goto out_set_header; | ||
| 103 | } | ||
| 104 | |||
| 105 | vevent = kzalloc(struct_size(vevent, event_data, data_len), GFP_ATOMIC); | ||
| 106 | if (!vevent) { | ||
| 107 | rc = -ENOMEM; | ||
| 108 | vevent = &veventq->lost_events_header; | ||
| 109 | goto out_set_header; | ||
| 110 | } | ||
| 111 | memcpy(vevent->event_data, event_data, data_len); | ||
| 112 | vevent->data_len = data_len; | ||
| 113 | veventq->num_events++; | ||
| 114 | |||
| 115 | out_set_header: | ||
| 116 | iommufd_vevent_handler(veventq, vevent); | ||
| 117 | spin_unlock(&veventq->common.lock); | ||
| 118 | out_unlock_veventqs: | ||
| 119 | up_read(&viommu->veventqs_rwsem); | ||
| 120 | return rc; | ||
| 121 | } | ||
| 122 | EXPORT_SYMBOL_NS_GPL(iommufd_viommu_report_event, "IOMMUFD"); | ||
| 123 | |||
| 124 | #ifdef CONFIG_IRQ_MSI_IOMMU | ||
| 125 | /* | ||
| 126 | * Get a iommufd_sw_msi_map for the msi physical address requested by the irq | ||
| 127 | * layer. The mapping to IOVA is global to the iommufd file descriptor, every | ||
| 128 | * domain that is attached to a device using the same MSI parameters will use | ||
| 129 | * the same IOVA. | ||
| 130 | */ | ||
| 131 | static struct iommufd_sw_msi_map * | ||
| 132 | iommufd_sw_msi_get_map(struct iommufd_ctx *ictx, phys_addr_t msi_addr, | ||
| 133 | phys_addr_t sw_msi_start) | ||
| 134 | { | ||
| 135 | struct iommufd_sw_msi_map *cur; | ||
| 136 | unsigned int max_pgoff = 0; | ||
| 137 | |||
| 138 | lockdep_assert_held(&ictx->sw_msi_lock); | ||
| 139 | |||
| 140 | list_for_each_entry(cur, &ictx->sw_msi_list, sw_msi_item) { | ||
| 141 | if (cur->sw_msi_start != sw_msi_start) | ||
| 142 | continue; | ||
| 143 | max_pgoff = max(max_pgoff, cur->pgoff + 1); | ||
| 144 | if (cur->msi_addr == msi_addr) | ||
| 145 | return cur; | ||
| 146 | } | ||
| 147 | |||
| 148 | if (ictx->sw_msi_id >= | ||
| 149 | BITS_PER_BYTE * sizeof_field(struct iommufd_sw_msi_maps, bitmap)) | ||
| 150 | return ERR_PTR(-EOVERFLOW); | ||
| 151 | |||
| 152 | cur = kzalloc(sizeof(*cur), GFP_KERNEL); | ||
| 153 | if (!cur) | ||
| 154 | return ERR_PTR(-ENOMEM); | ||
| 155 | |||
| 156 | cur->sw_msi_start = sw_msi_start; | ||
| 157 | cur->msi_addr = msi_addr; | ||
| 158 | cur->pgoff = max_pgoff; | ||
| 159 | cur->id = ictx->sw_msi_id++; | ||
| 160 | list_add_tail(&cur->sw_msi_item, &ictx->sw_msi_list); | ||
| 161 | return cur; | ||
| 162 | } | ||
| 163 | |||
| 164 | int iommufd_sw_msi_install(struct iommufd_ctx *ictx, | ||
| 165 | struct iommufd_hwpt_paging *hwpt_paging, | ||
| 166 | struct iommufd_sw_msi_map *msi_map) | ||
| 167 | { | ||
| 168 | unsigned long iova; | ||
| 169 | |||
| 170 | lockdep_assert_held(&ictx->sw_msi_lock); | ||
| 171 | |||
| 172 | iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE; | ||
| 173 | if (!test_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap)) { | ||
| 174 | int rc; | ||
| 175 | |||
| 176 | rc = iommu_map(hwpt_paging->common.domain, iova, | ||
| 177 | msi_map->msi_addr, PAGE_SIZE, | ||
| 178 | IOMMU_WRITE | IOMMU_READ | IOMMU_MMIO, | ||
| 179 | GFP_KERNEL_ACCOUNT); | ||
| 180 | if (rc) | ||
| 181 | return rc; | ||
| 182 | __set_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap); | ||
| 183 | } | ||
| 184 | return 0; | ||
| 185 | } | ||
| 186 | EXPORT_SYMBOL_NS_GPL(iommufd_sw_msi_install, "IOMMUFD_INTERNAL"); | ||
| 187 | |||
| 188 | /* | ||
| 189 | * Called by the irq code if the platform translates the MSI address through the | ||
| 190 | * IOMMU. msi_addr is the physical address of the MSI page. iommufd will | ||
| 191 | * allocate a fd global iova for the physical page that is the same on all | ||
| 192 | * domains and devices. | ||
| 193 | */ | ||
| 194 | int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | ||
| 195 | phys_addr_t msi_addr) | ||
| 196 | { | ||
| 197 | struct device *dev = msi_desc_to_dev(desc); | ||
| 198 | struct iommufd_hwpt_paging *hwpt_paging; | ||
| 199 | struct iommu_attach_handle *raw_handle; | ||
| 200 | struct iommufd_attach_handle *handle; | ||
| 201 | struct iommufd_sw_msi_map *msi_map; | ||
| 202 | struct iommufd_ctx *ictx; | ||
| 203 | unsigned long iova; | ||
| 204 | int rc; | ||
| 205 | |||
| 206 | /* | ||
| 207 | * It is safe to call iommu_attach_handle_get() here because the iommu | ||
| 208 | * core code invokes this under the group mutex which also prevents any | ||
| 209 | * change of the attach handle for the duration of this function. | ||
| 210 | */ | ||
| 211 | iommu_group_mutex_assert(dev); | ||
| 212 | |||
| 213 | raw_handle = | ||
| 214 | iommu_attach_handle_get(dev->iommu_group, IOMMU_NO_PASID, 0); | ||
| 215 | if (IS_ERR(raw_handle)) | ||
| 216 | return 0; | ||
| 217 | hwpt_paging = find_hwpt_paging(domain->iommufd_hwpt); | ||
| 218 | |||
| 219 | handle = to_iommufd_handle(raw_handle); | ||
| 220 | /* No IOMMU_RESV_SW_MSI means no change to the msi_msg */ | ||
| 221 | if (handle->idev->igroup->sw_msi_start == PHYS_ADDR_MAX) | ||
| 222 | return 0; | ||
| 223 | |||
| 224 | ictx = handle->idev->ictx; | ||
| 225 | guard(mutex)(&ictx->sw_msi_lock); | ||
| 226 | /* | ||
| 227 | * The input msi_addr is the exact byte offset of the MSI doorbell, we | ||
| 228 | * assume the caller has checked that it is contained with a MMIO region | ||
| 229 | * that is secure to map at PAGE_SIZE. | ||
| 230 | */ | ||
| 231 | msi_map = iommufd_sw_msi_get_map(handle->idev->ictx, | ||
| 232 | msi_addr & PAGE_MASK, | ||
| 233 | handle->idev->igroup->sw_msi_start); | ||
| 234 | if (IS_ERR(msi_map)) | ||
| 235 | return PTR_ERR(msi_map); | ||
| 236 | |||
| 237 | rc = iommufd_sw_msi_install(ictx, hwpt_paging, msi_map); | ||
| 238 | if (rc) | ||
| 239 | return rc; | ||
| 240 | __set_bit(msi_map->id, handle->idev->igroup->required_sw_msi.bitmap); | ||
| 241 | |||
| 242 | iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE; | ||
| 243 | msi_desc_set_iommu_msi_iova(desc, iova, PAGE_SHIFT); | ||
| 244 | return 0; | ||
| 245 | } | ||
| 246 | EXPORT_SYMBOL_NS_GPL(iommufd_sw_msi, "IOMMUFD"); | ||
| 247 | #endif | ||
| 248 | |||
| 52 | MODULE_DESCRIPTION("iommufd code shared with builtin modules"); | 249 | MODULE_DESCRIPTION("iommufd code shared with builtin modules"); |
| 250 | MODULE_IMPORT_NS("IOMMUFD_INTERNAL"); | ||
| 53 | MODULE_LICENSE("GPL"); | 251 | MODULE_LICENSE("GPL"); |
| @@ -0,0 +1,598 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only | ||
| 2 | /* Copyright (C) 2024 Intel Corporation | ||
| 3 | */ | ||
| 4 | #define pr_fmt(fmt) "iommufd: " fmt | ||
| 5 | |||
| 6 | #include <linux/anon_inodes.h> | ||
| 7 | #include <linux/file.h> | ||
| 8 | #include <linux/fs.h> | ||
| 9 | #include <linux/iommufd.h> | ||
| 10 | #include <linux/module.h> | ||
| 11 | #include <linux/mutex.h> | ||
| 12 | #include <linux/pci.h> | ||
| 13 | #include <linux/pci-ats.h> | ||
| 14 | #include <linux/poll.h> | ||
| 15 | #include <uapi/linux/iommufd.h> | ||
| 16 | |||
| 17 | #include "../iommu-priv.h" | ||
| 18 | #include "iommufd_private.h" | ||
| 19 | |||
| 20 | /* IOMMUFD_OBJ_FAULT Functions */ | ||
| 21 | |||
| 22 | int iommufd_fault_iopf_enable(struct iommufd_device *idev) | ||
| 23 | { | ||
| 24 | struct device *dev = idev->dev; | ||
| 25 | int ret; | ||
| 26 | |||
| 27 | /* | ||
| 28 | * Once we turn on PCI/PRI support for VF, the response failure code | ||
| 29 | * should not be forwarded to the hardware due to PRI being a shared | ||
| 30 | * resource between PF and VFs. There is no coordination for this | ||
| 31 | * shared capability. This waits for a vPRI reset to recover. | ||
| 32 | */ | ||
| 33 | if (dev_is_pci(dev)) { | ||
| 34 | struct pci_dev *pdev = to_pci_dev(dev); | ||
| 35 | |||
| 36 | if (pdev->is_virtfn && pci_pri_supported(pdev)) | ||
| 37 | return -EINVAL; | ||
| 38 | } | ||
| 39 | |||
| 40 | mutex_lock(&idev->iopf_lock); | ||
| 41 | /* Device iopf has already been on. */ | ||
| 42 | if (++idev->iopf_enabled > 1) { | ||
| 43 | mutex_unlock(&idev->iopf_lock); | ||
| 44 | return 0; | ||
| 45 | } | ||
| 46 | |||
| 47 | ret = iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_IOPF); | ||
| 48 | if (ret) | ||
| 49 | --idev->iopf_enabled; | ||
| 50 | mutex_unlock(&idev->iopf_lock); | ||
| 51 | |||
| 52 | return ret; | ||
| 53 | } | ||
| 54 | |||
| 55 | void iommufd_fault_iopf_disable(struct iommufd_device *idev) | ||
| 56 | { | ||
| 57 | mutex_lock(&idev->iopf_lock); | ||
| 58 | if (!WARN_ON(idev->iopf_enabled == 0)) { | ||
| 59 | if (--idev->iopf_enabled == 0) | ||
| 60 | iommu_dev_disable_feature(idev->dev, IOMMU_DEV_FEAT_IOPF); | ||
| 61 | } | ||
| 62 | mutex_unlock(&idev->iopf_lock); | ||
| 63 | } | ||
| 64 | |||
| 65 | void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt, | ||
| 66 | struct iommufd_attach_handle *handle) | ||
| 67 | { | ||
| 68 | struct iommufd_fault *fault = hwpt->fault; | ||
| 69 | struct iopf_group *group, *next; | ||
| 70 | struct list_head free_list; | ||
| 71 | unsigned long index; | ||
| 72 | |||
| 73 | if (!fault) | ||
| 74 | return; | ||
| 75 | INIT_LIST_HEAD(&free_list); | ||
| 76 | |||
| 77 | mutex_lock(&fault->mutex); | ||
| 78 | spin_lock(&fault->common.lock); | ||
| 79 | list_for_each_entry_safe(group, next, &fault->common.deliver, node) { | ||
| 80 | if (group->attach_handle != &handle->handle) | ||
| 81 | continue; | ||
| 82 | list_move(&group->node, &free_list); | ||
| 83 | } | ||
| 84 | spin_unlock(&fault->common.lock); | ||
| 85 | |||
| 86 | list_for_each_entry_safe(group, next, &free_list, node) { | ||
| 87 | list_del(&group->node); | ||
| 88 | iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | ||
| 89 | iopf_free_group(group); | ||
| 90 | } | ||
| 91 | |||
| 92 | xa_for_each(&fault->response, index, group) { | ||
| 93 | if (group->attach_handle != &handle->handle) | ||
| 94 | continue; | ||
| 95 | xa_erase(&fault->response, index); | ||
| 96 | iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | ||
| 97 | iopf_free_group(group); | ||
| 98 | } | ||
| 99 | mutex_unlock(&fault->mutex); | ||
| 100 | } | ||
| 101 | |||
| 102 | void iommufd_fault_destroy(struct iommufd_object *obj) | ||
| 103 | { | ||
| 104 | struct iommufd_eventq *eventq = | ||
| 105 | container_of(obj, struct iommufd_eventq, obj); | ||
| 106 | struct iommufd_fault *fault = eventq_to_fault(eventq); | ||
| 107 | struct iopf_group *group, *next; | ||
| 108 | unsigned long index; | ||
| 109 | |||
| 110 | /* | ||
| 111 | * The iommufd object's reference count is zero at this point. | ||
| 112 | * We can be confident that no other threads are currently | ||
| 113 | * accessing this pointer. Therefore, acquiring the mutex here | ||
| 114 | * is unnecessary. | ||
| 115 | */ | ||
| 116 | list_for_each_entry_safe(group, next, &fault->common.deliver, node) { | ||
| 117 | list_del(&group->node); | ||
| 118 | iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | ||
| 119 | iopf_free_group(group); | ||
| 120 | } | ||
| 121 | xa_for_each(&fault->response, index, group) { | ||
| 122 | xa_erase(&fault->response, index); | ||
| 123 | iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | ||
| 124 | iopf_free_group(group); | ||
| 125 | } | ||
| 126 | xa_destroy(&fault->response); | ||
| 127 | mutex_destroy(&fault->mutex); | ||
| 128 | } | ||
| 129 | |||
| 130 | static void iommufd_compose_fault_message(struct iommu_fault *fault, | ||
| 131 | struct iommu_hwpt_pgfault *hwpt_fault, | ||
| 132 | struct iommufd_device *idev, | ||
| 133 | u32 cookie) | ||
| 134 | { | ||
| 135 | hwpt_fault->flags = fault->prm.flags; | ||
| 136 | hwpt_fault->dev_id = idev->obj.id; | ||
| 137 | hwpt_fault->pasid = fault->prm.pasid; | ||
| 138 | hwpt_fault->grpid = fault->prm.grpid; | ||
| 139 | hwpt_fault->perm = fault->prm.perm; | ||
| 140 | hwpt_fault->addr = fault->prm.addr; | ||
| 141 | hwpt_fault->length = 0; | ||
| 142 | hwpt_fault->cookie = cookie; | ||
| 143 | } | ||
| 144 | |||
| 145 | /* Fetch the first node out of the fault->deliver list */ | ||
| 146 | static struct iopf_group * | ||
| 147 | iommufd_fault_deliver_fetch(struct iommufd_fault *fault) | ||
| 148 | { | ||
| 149 | struct list_head *list = &fault->common.deliver; | ||
| 150 | struct iopf_group *group = NULL; | ||
| 151 | |||
| 152 | spin_lock(&fault->common.lock); | ||
| 153 | if (!list_empty(list)) { | ||
| 154 | group = list_first_entry(list, struct iopf_group, node); | ||
| 155 | list_del(&group->node); | ||
| 156 | } | ||
| 157 | spin_unlock(&fault->common.lock); | ||
| 158 | return group; | ||
| 159 | } | ||
| 160 | |||
| 161 | /* Restore a node back to the head of the fault->deliver list */ | ||
| 162 | static void iommufd_fault_deliver_restore(struct iommufd_fault *fault, | ||
| 163 | struct iopf_group *group) | ||
| 164 | { | ||
| 165 | spin_lock(&fault->common.lock); | ||
| 166 | list_add(&group->node, &fault->common.deliver); | ||
| 167 | spin_unlock(&fault->common.lock); | ||
| 168 | } | ||
| 169 | |||
| 170 | static ssize_t iommufd_fault_fops_read(struct file *filep, char __user *buf, | ||
| 171 | size_t count, loff_t *ppos) | ||
| 172 | { | ||
| 173 | size_t fault_size = sizeof(struct iommu_hwpt_pgfault); | ||
| 174 | struct iommufd_eventq *eventq = filep->private_data; | ||
| 175 | struct iommufd_fault *fault = eventq_to_fault(eventq); | ||
| 176 | struct iommu_hwpt_pgfault data = {}; | ||
| 177 | struct iommufd_device *idev; | ||
| 178 | struct iopf_group *group; | ||
| 179 | struct iopf_fault *iopf; | ||
| 180 | size_t done = 0; | ||
| 181 | int rc = 0; | ||
| 182 | |||
| 183 | if (*ppos || count % fault_size) | ||
| 184 | return -ESPIPE; | ||
| 185 | |||
| 186 | mutex_lock(&fault->mutex); | ||
| 187 | while ((group = iommufd_fault_deliver_fetch(fault))) { | ||
| 188 | if (done >= count || | ||
| 189 | group->fault_count * fault_size > count - done) { | ||
| 190 | iommufd_fault_deliver_restore(fault, group); | ||
| 191 | break; | ||
| 192 | } | ||
| 193 | |||
| 194 | rc = xa_alloc(&fault->response, &group->cookie, group, | ||
| 195 | xa_limit_32b, GFP_KERNEL); | ||
| 196 | if (rc) { | ||
| 197 | iommufd_fault_deliver_restore(fault, group); | ||
| 198 | break; | ||
| 199 | } | ||
| 200 | |||
| 201 | idev = to_iommufd_handle(group->attach_handle)->idev; | ||
| 202 | list_for_each_entry(iopf, &group->faults, list) { | ||
| 203 | iommufd_compose_fault_message(&iopf->fault, | ||
| 204 | &data, idev, | ||
| 205 | group->cookie); | ||
| 206 | if (copy_to_user(buf + done, &data, fault_size)) { | ||
| 207 | xa_erase(&fault->response, group->cookie); | ||
| 208 | iommufd_fault_deliver_restore(fault, group); | ||
| 209 | rc = -EFAULT; | ||
| 210 | break; | ||
| 211 | } | ||
| 212 | done += fault_size; | ||
| 213 | } | ||
| 214 | } | ||
| 215 | mutex_unlock(&fault->mutex); | ||
| 216 | |||
| 217 | return done == 0 ? rc : done; | ||
| 218 | } | ||
| 219 | |||
| 220 | static ssize_t iommufd_fault_fops_write(struct file *filep, const char __user *buf, | ||
| 221 | size_t count, loff_t *ppos) | ||
| 222 | { | ||
| 223 | size_t response_size = sizeof(struct iommu_hwpt_page_response); | ||
| 224 | struct iommufd_eventq *eventq = filep->private_data; | ||
| 225 | struct iommufd_fault *fault = eventq_to_fault(eventq); | ||
| 226 | struct iommu_hwpt_page_response response; | ||
| 227 | struct iopf_group *group; | ||
| 228 | size_t done = 0; | ||
| 229 | int rc = 0; | ||
| 230 | |||
| 231 | if (*ppos || count % response_size) | ||
| 232 | return -ESPIPE; | ||
| 233 | |||
| 234 | mutex_lock(&fault->mutex); | ||
| 235 | while (count > done) { | ||
| 236 | rc = copy_from_user(&response, buf + done, response_size); | ||
| 237 | if (rc) | ||
| 238 | break; | ||
| 239 | |||
| 240 | static_assert((int)IOMMUFD_PAGE_RESP_SUCCESS == | ||
| 241 | (int)IOMMU_PAGE_RESP_SUCCESS); | ||
| 242 | static_assert((int)IOMMUFD_PAGE_RESP_INVALID == | ||
| 243 | (int)IOMMU_PAGE_RESP_INVALID); | ||
| 244 | if (response.code != IOMMUFD_PAGE_RESP_SUCCESS && | ||
| 245 | response.code != IOMMUFD_PAGE_RESP_INVALID) { | ||
| 246 | rc = -EINVAL; | ||
| 247 | break; | ||
| 248 | } | ||
| 249 | |||
| 250 | group = xa_erase(&fault->response, response.cookie); | ||
| 251 | if (!group) { | ||
| 252 | rc = -EINVAL; | ||
| 253 | break; | ||
| 254 | } | ||
| 255 | |||
| 256 | iopf_group_response(group, response.code); | ||
| 257 | iopf_free_group(group); | ||
| 258 | done += response_size; | ||
| 259 | } | ||
| 260 | mutex_unlock(&fault->mutex); | ||
| 261 | |||
| 262 | return done == 0 ? rc : done; | ||
| 263 | } | ||
| 264 | |||
| 265 | /* IOMMUFD_OBJ_VEVENTQ Functions */ | ||
| 266 | |||
| 267 | void iommufd_veventq_abort(struct iommufd_object *obj) | ||
| 268 | { | ||
| 269 | struct iommufd_eventq *eventq = | ||
| 270 | container_of(obj, struct iommufd_eventq, obj); | ||
| 271 | struct iommufd_veventq *veventq = eventq_to_veventq(eventq); | ||
| 272 | struct iommufd_viommu *viommu = veventq->viommu; | ||
| 273 | struct iommufd_vevent *cur, *next; | ||
| 274 | |||
| 275 | lockdep_assert_held_write(&viommu->veventqs_rwsem); | ||
| 276 | |||
| 277 | list_for_each_entry_safe(cur, next, &eventq->deliver, node) { | ||
| 278 | list_del(&cur->node); | ||
| 279 | if (cur != &veventq->lost_events_header) | ||
| 280 | kfree(cur); | ||
| 281 | } | ||
| 282 | |||
| 283 | refcount_dec(&viommu->obj.users); | ||
| 284 | list_del(&veventq->node); | ||
| 285 | } | ||
| 286 | |||
| 287 | void iommufd_veventq_destroy(struct iommufd_object *obj) | ||
| 288 | { | ||
| 289 | struct iommufd_veventq *veventq = eventq_to_veventq( | ||
| 290 | container_of(obj, struct iommufd_eventq, obj)); | ||
| 291 | |||
| 292 | down_write(&veventq->viommu->veventqs_rwsem); | ||
| 293 | iommufd_veventq_abort(obj); | ||
| 294 | up_write(&veventq->viommu->veventqs_rwsem); | ||
| 295 | } | ||
| 296 | |||
| 297 | static struct iommufd_vevent * | ||
| 298 | iommufd_veventq_deliver_fetch(struct iommufd_veventq *veventq) | ||
| 299 | { | ||
| 300 | struct iommufd_eventq *eventq = &veventq->common; | ||
| 301 | struct list_head *list = &eventq->deliver; | ||
| 302 | struct iommufd_vevent *vevent = NULL; | ||
| 303 | |||
| 304 | spin_lock(&eventq->lock); | ||
| 305 | if (!list_empty(list)) { | ||
| 306 | struct iommufd_vevent *next; | ||
| 307 | |||
| 308 | next = list_first_entry(list, struct iommufd_vevent, node); | ||
| 309 | /* Make a copy of the lost_events_header for copy_to_user */ | ||
| 310 | if (next == &veventq->lost_events_header) { | ||
| 311 | vevent = kzalloc(sizeof(*vevent), GFP_ATOMIC); | ||
| 312 | if (!vevent) | ||
| 313 | goto out_unlock; | ||
| 314 | } | ||
| 315 | list_del(&next->node); | ||
| 316 | if (vevent) | ||
| 317 | memcpy(vevent, next, sizeof(*vevent)); | ||
| 318 | else | ||
| 319 | vevent = next; | ||
| 320 | } | ||
| 321 | out_unlock: | ||
| 322 | spin_unlock(&eventq->lock); | ||
| 323 | return vevent; | ||
| 324 | } | ||
| 325 | |||
| 326 | static void iommufd_veventq_deliver_restore(struct iommufd_veventq *veventq, | ||
| 327 | struct iommufd_vevent *vevent) | ||
| 328 | { | ||
| 329 | struct iommufd_eventq *eventq = &veventq->common; | ||
| 330 | struct list_head *list = &eventq->deliver; | ||
| 331 | |||
| 332 | spin_lock(&eventq->lock); | ||
| 333 | if (vevent_for_lost_events_header(vevent)) { | ||
| 334 | /* Remove the copy of the lost_events_header */ | ||
| 335 | kfree(vevent); | ||
| 336 | vevent = NULL; | ||
| 337 | /* An empty list needs the lost_events_header back */ | ||
| 338 | if (list_empty(list)) | ||
| 339 | vevent = &veventq->lost_events_header; | ||
| 340 | } | ||
| 341 | if (vevent) | ||
| 342 | list_add(&vevent->node, list); | ||
| 343 | spin_unlock(&eventq->lock); | ||
| 344 | } | ||
| 345 | |||
| 346 | static ssize_t iommufd_veventq_fops_read(struct file *filep, char __user *buf, | ||
| 347 | size_t count, loff_t *ppos) | ||
| 348 | { | ||
| 349 | struct iommufd_eventq *eventq = filep->private_data; | ||
| 350 | struct iommufd_veventq *veventq = eventq_to_veventq(eventq); | ||
| 351 | struct iommufd_vevent_header *hdr; | ||
| 352 | struct iommufd_vevent *cur; | ||
| 353 | size_t done = 0; | ||
| 354 | int rc = 0; | ||
| 355 | |||
| 356 | if (*ppos) | ||
| 357 | return -ESPIPE; | ||
| 358 | |||
| 359 | while ((cur = iommufd_veventq_deliver_fetch(veventq))) { | ||
| 360 | /* Validate the remaining bytes against the header size */ | ||
| 361 | if (done >= count || sizeof(*hdr) > count - done) { | ||
| 362 | iommufd_veventq_deliver_restore(veventq, cur); | ||
| 363 | break; | ||
| 364 | } | ||
| 365 | hdr = &cur->header; | ||
| 366 | |||
| 367 | /* If being a normal vEVENT, validate against the full size */ | ||
| 368 | if (!vevent_for_lost_events_header(cur) && | ||
| 369 | sizeof(hdr) + cur->data_len > count - done) { | ||
| 370 | iommufd_veventq_deliver_restore(veventq, cur); | ||
| 371 | break; | ||
| 372 | } | ||
| 373 | |||
| 374 | if (copy_to_user(buf + done, hdr, sizeof(*hdr))) { | ||
| 375 | iommufd_veventq_deliver_restore(veventq, cur); | ||
| 376 | rc = -EFAULT; | ||
| 377 | break; | ||
| 378 | } | ||
| 379 | done += sizeof(*hdr); | ||
| 380 | |||
| 381 | if (cur->data_len && | ||
| 382 | copy_to_user(buf + done, cur->event_data, cur->data_len)) { | ||
| 383 | iommufd_veventq_deliver_restore(veventq, cur); | ||
| 384 | rc = -EFAULT; | ||
| 385 | break; | ||
| 386 | } | ||
| 387 | spin_lock(&eventq->lock); | ||
| 388 | if (!vevent_for_lost_events_header(cur)) | ||
| 389 | veventq->num_events--; | ||
| 390 | spin_unlock(&eventq->lock); | ||
| 391 | done += cur->data_len; | ||
| 392 | kfree(cur); | ||
| 393 | } | ||
| 394 | |||
| 395 | return done == 0 ? rc : done; | ||
| 396 | } | ||
| 397 | |||
| 398 | /* Common Event Queue Functions */ | ||
| 399 | |||
| 400 | static __poll_t iommufd_eventq_fops_poll(struct file *filep, | ||
| 401 | struct poll_table_struct *wait) | ||
| 402 | { | ||
| 403 | struct iommufd_eventq *eventq = filep->private_data; | ||
| 404 | __poll_t pollflags = 0; | ||
| 405 | |||
| 406 | if (eventq->obj.type == IOMMUFD_OBJ_FAULT) | ||
| 407 | pollflags |= EPOLLOUT; | ||
| 408 | |||
| 409 | poll_wait(filep, &eventq->wait_queue, wait); | ||
| 410 | spin_lock(&eventq->lock); | ||
| 411 | if (!list_empty(&eventq->deliver)) | ||
| 412 | pollflags |= EPOLLIN | EPOLLRDNORM; | ||
| 413 | spin_unlock(&eventq->lock); | ||
| 414 | |||
| 415 | return pollflags; | ||
| 416 | } | ||
| 417 | |||
| 418 | static int iommufd_eventq_fops_release(struct inode *inode, struct file *filep) | ||
| 419 | { | ||
| 420 | struct iommufd_eventq *eventq = filep->private_data; | ||
| 421 | |||
| 422 | refcount_dec(&eventq->obj.users); | ||
| 423 | iommufd_ctx_put(eventq->ictx); | ||
| 424 | return 0; | ||
| 425 | } | ||
| 426 | |||
| 427 | #define INIT_EVENTQ_FOPS(read_op, write_op) \ | ||
| 428 | ((const struct file_operations){ \ | ||
| 429 | .owner = THIS_MODULE, \ | ||
| 430 | .open = nonseekable_open, \ | ||
| 431 | .read = read_op, \ | ||
| 432 | .write = write_op, \ | ||
| 433 | .poll = iommufd_eventq_fops_poll, \ | ||
| 434 | .release = iommufd_eventq_fops_release, \ | ||
| 435 | }) | ||
| 436 | |||
| 437 | static int iommufd_eventq_init(struct iommufd_eventq *eventq, char *name, | ||
| 438 | struct iommufd_ctx *ictx, | ||
| 439 | const struct file_operations *fops) | ||
| 440 | { | ||
| 441 | struct file *filep; | ||
| 442 | int fdno; | ||
| 443 | |||
| 444 | spin_lock_init(&eventq->lock); | ||
| 445 | INIT_LIST_HEAD(&eventq->deliver); | ||
| 446 | init_waitqueue_head(&eventq->wait_queue); | ||
| 447 | |||
| 448 | filep = anon_inode_getfile(name, fops, eventq, O_RDWR); | ||
| 449 | if (IS_ERR(filep)) | ||
| 450 | return PTR_ERR(filep); | ||
| 451 | |||
| 452 | eventq->ictx = ictx; | ||
| 453 | iommufd_ctx_get(eventq->ictx); | ||
| 454 | eventq->filep = filep; | ||
| 455 | refcount_inc(&eventq->obj.users); | ||
| 456 | |||
| 457 | fdno = get_unused_fd_flags(O_CLOEXEC); | ||
| 458 | if (fdno < 0) | ||
| 459 | fput(filep); | ||
| 460 | return fdno; | ||
| 461 | } | ||
| 462 | |||
| 463 | static const struct file_operations iommufd_fault_fops = | ||
| 464 | INIT_EVENTQ_FOPS(iommufd_fault_fops_read, iommufd_fault_fops_write); | ||
| 465 | |||
| 466 | int iommufd_fault_alloc(struct iommufd_ucmd *ucmd) | ||
| 467 | { | ||
| 468 | struct iommu_fault_alloc *cmd = ucmd->cmd; | ||
| 469 | struct iommufd_fault *fault; | ||
| 470 | int fdno; | ||
| 471 | int rc; | ||
| 472 | |||
| 473 | if (cmd->flags) | ||
| 474 | return -EOPNOTSUPP; | ||
| 475 | |||
| 476 | fault = __iommufd_object_alloc(ucmd->ictx, fault, IOMMUFD_OBJ_FAULT, | ||
| 477 | common.obj); | ||
| 478 | if (IS_ERR(fault)) | ||
| 479 | return PTR_ERR(fault); | ||
| 480 | |||
| 481 | xa_init_flags(&fault->response, XA_FLAGS_ALLOC1); | ||
| 482 | mutex_init(&fault->mutex); | ||
| 483 | |||
| 484 | fdno = iommufd_eventq_init(&fault->common, "[iommufd-pgfault]", | ||
| 485 | ucmd->ictx, &iommufd_fault_fops); | ||
| 486 | if (fdno < 0) { | ||
| 487 | rc = fdno; | ||
| 488 | goto out_abort; | ||
| 489 | } | ||
| 490 | |||
| 491 | cmd->out_fault_id = fault->common.obj.id; | ||
| 492 | cmd->out_fault_fd = fdno; | ||
| 493 | |||
| 494 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); | ||
| 495 | if (rc) | ||
| 496 | goto out_put_fdno; | ||
| 497 | iommufd_object_finalize(ucmd->ictx, &fault->common.obj); | ||
| 498 | |||
| 499 | fd_install(fdno, fault->common.filep); | ||
| 500 | |||
| 501 | return 0; | ||
| 502 | out_put_fdno: | ||
| 503 | put_unused_fd(fdno); | ||
| 504 | fput(fault->common.filep); | ||
| 505 | out_abort: | ||
| 506 | iommufd_object_abort_and_destroy(ucmd->ictx, &fault->common.obj); | ||
| 507 | |||
| 508 | return rc; | ||
| 509 | } | ||
| 510 | |||
| 511 | int iommufd_fault_iopf_handler(struct iopf_group *group) | ||
| 512 | { | ||
| 513 | struct iommufd_hw_pagetable *hwpt; | ||
| 514 | struct iommufd_fault *fault; | ||
| 515 | |||
| 516 | hwpt = group->attach_handle->domain->iommufd_hwpt; | ||
| 517 | fault = hwpt->fault; | ||
| 518 | |||
| 519 | spin_lock(&fault->common.lock); | ||
| 520 | list_add_tail(&group->node, &fault->common.deliver); | ||
| 521 | spin_unlock(&fault->common.lock); | ||
| 522 | |||
| 523 | wake_up_interruptible(&fault->common.wait_queue); | ||
| 524 | |||
| 525 | return 0; | ||
| 526 | } | ||
| 527 | |||
| 528 | static const struct file_operations iommufd_veventq_fops = | ||
| 529 | INIT_EVENTQ_FOPS(iommufd_veventq_fops_read, NULL); | ||
| 530 | |||
| 531 | int iommufd_veventq_alloc(struct iommufd_ucmd *ucmd) | ||
| 532 | { | ||
| 533 | struct iommu_veventq_alloc *cmd = ucmd->cmd; | ||
| 534 | struct iommufd_veventq *veventq; | ||
| 535 | struct iommufd_viommu *viommu; | ||
| 536 | int fdno; | ||
| 537 | int rc; | ||
| 538 | |||
| 539 | if (cmd->flags || cmd->__reserved || | ||
| 540 | cmd->type == IOMMU_VEVENTQ_TYPE_DEFAULT) | ||
| 541 | return -EOPNOTSUPP; | ||
| 542 | if (!cmd->veventq_depth) | ||
| 543 | return -EINVAL; | ||
| 544 | |||
| 545 | viommu = iommufd_get_viommu(ucmd, cmd->viommu_id); | ||
| 546 | if (IS_ERR(viommu)) | ||
| 547 | return PTR_ERR(viommu); | ||
| 548 | |||
| 549 | down_write(&viommu->veventqs_rwsem); | ||
| 550 | |||
| 551 | if (iommufd_viommu_find_veventq(viommu, cmd->type)) { | ||
| 552 | rc = -EEXIST; | ||
| 553 | goto out_unlock_veventqs; | ||
| 554 | } | ||
| 555 | |||
| 556 | veventq = __iommufd_object_alloc(ucmd->ictx, veventq, | ||
| 557 | IOMMUFD_OBJ_VEVENTQ, common.obj); | ||
| 558 | if (IS_ERR(veventq)) { | ||
| 559 | rc = PTR_ERR(veventq); | ||
| 560 | goto out_unlock_veventqs; | ||
| 561 | } | ||
| 562 | |||
| 563 | veventq->type = cmd->type; | ||
| 564 | veventq->viommu = viommu; | ||
| 565 | refcount_inc(&viommu->obj.users); | ||
| 566 | veventq->depth = cmd->veventq_depth; | ||
| 567 | list_add_tail(&veventq->node, &viommu->veventqs); | ||
| 568 | veventq->lost_events_header.header.flags = | ||
| 569 | IOMMU_VEVENTQ_FLAG_LOST_EVENTS; | ||
| 570 | |||
| 571 | fdno = iommufd_eventq_init(&veventq->common, "[iommufd-viommu-event]", | ||
| 572 | ucmd->ictx, &iommufd_veventq_fops); | ||
| 573 | if (fdno < 0) { | ||
| 574 | rc = fdno; | ||
| 575 | goto out_abort; | ||
| 576 | } | ||
| 577 | |||
| 578 | cmd->out_veventq_id = veventq->common.obj.id; | ||
| 579 | cmd->out_veventq_fd = fdno; | ||
| 580 | |||
| 581 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); | ||
| 582 | if (rc) | ||
| 583 | goto out_put_fdno; | ||
| 584 | |||
| 585 | iommufd_object_finalize(ucmd->ictx, &veventq->common.obj); | ||
| 586 | fd_install(fdno, veventq->common.filep); | ||
| 587 | goto out_unlock_veventqs; | ||
| 588 | |||
| 589 | out_put_fdno: | ||
| 590 | put_unused_fd(fdno); | ||
| 591 | fput(veventq->common.filep); | ||
| 592 | out_abort: | ||
| 593 | iommufd_object_abort_and_destroy(ucmd->ictx, &veventq->common.obj); | ||
| 594 | out_unlock_veventqs: | ||
| 595 | up_write(&viommu->veventqs_rwsem); | ||
| 596 | iommufd_put_object(ucmd->ictx, &viommu->obj); | ||
| 597 | return rc; | ||
| 598 | } | ||
| @@ -1,342 +0,0 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only | ||
| 2 | /* Copyright (C) 2024 Intel Corporation | ||
| 3 | */ | ||
| 4 | #define pr_fmt(fmt) "iommufd: " fmt | ||
| 5 | |||
| 6 | #include <linux/anon_inodes.h> | ||
| 7 | #include <linux/file.h> | ||
| 8 | #include <linux/fs.h> | ||
| 9 | #include <linux/iommufd.h> | ||
| 10 | #include <linux/module.h> | ||
| 11 | #include <linux/mutex.h> | ||
| 12 | #include <linux/pci.h> | ||
| 13 | #include <linux/pci-ats.h> | ||
| 14 | #include <linux/poll.h> | ||
| 15 | #include <uapi/linux/iommufd.h> | ||
| 16 | |||
| 17 | #include "../iommu-priv.h" | ||
| 18 | #include "iommufd_private.h" | ||
| 19 | |||
| 20 | int iommufd_fault_iopf_enable(struct iommufd_device *idev) | ||
| 21 | { | ||
| 22 | struct device *dev = idev->dev; | ||
| 23 | int ret; | ||
| 24 | |||
| 25 | /* | ||
| 26 | * Once we turn on PCI/PRI support for VF, the response failure code | ||
| 27 | * should not be forwarded to the hardware due to PRI being a shared | ||
| 28 | * resource between PF and VFs. There is no coordination for this | ||
| 29 | * shared capability. This waits for a vPRI reset to recover. | ||
| 30 | */ | ||
| 31 | if (dev_is_pci(dev)) { | ||
| 32 | struct pci_dev *pdev = to_pci_dev(dev); | ||
| 33 | |||
| 34 | if (pdev->is_virtfn && pci_pri_supported(pdev)) | ||
| 35 | return -EINVAL; | ||
| 36 | } | ||
| 37 | |||
| 38 | mutex_lock(&idev->iopf_lock); | ||
| 39 | /* Device iopf has already been on. */ | ||
| 40 | if (++idev->iopf_enabled > 1) { | ||
| 41 | mutex_unlock(&idev->iopf_lock); | ||
| 42 | return 0; | ||
| 43 | } | ||
| 44 | |||
| 45 | ret = iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_IOPF); | ||
| 46 | if (ret) | ||
| 47 | --idev->iopf_enabled; | ||
| 48 | mutex_unlock(&idev->iopf_lock); | ||
| 49 | |||
| 50 | return ret; | ||
| 51 | } | ||
| 52 | |||
| 53 | void iommufd_fault_iopf_disable(struct iommufd_device *idev) | ||
| 54 | { | ||
| 55 | mutex_lock(&idev->iopf_lock); | ||
| 56 | if (!WARN_ON(idev->iopf_enabled == 0)) { | ||
| 57 | if (--idev->iopf_enabled == 0) | ||
| 58 | iommu_dev_disable_feature(idev->dev, IOMMU_DEV_FEAT_IOPF); | ||
| 59 | } | ||
| 60 | mutex_unlock(&idev->iopf_lock); | ||
| 61 | } | ||
| 62 | |||
| 63 | void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt, | ||
| 64 | struct iommufd_attach_handle *handle) | ||
| 65 | { | ||
| 66 | struct iommufd_fault *fault = hwpt->fault; | ||
| 67 | struct iopf_group *group, *next; | ||
| 68 | struct list_head free_list; | ||
| 69 | unsigned long index; | ||
| 70 | |||
| 71 | if (!fault) | ||
| 72 | return; | ||
| 73 | INIT_LIST_HEAD(&free_list); | ||
| 74 | |||
| 75 | mutex_lock(&fault->mutex); | ||
| 76 | spin_lock(&fault->lock); | ||
| 77 | list_for_each_entry_safe(group, next, &fault->deliver, node) { | ||
| 78 | if (group->attach_handle != &handle->handle) | ||
| 79 | continue; | ||
| 80 | list_move(&group->node, &free_list); | ||
| 81 | } | ||
| 82 | spin_unlock(&fault->lock); | ||
| 83 | |||
| 84 | list_for_each_entry_safe(group, next, &free_list, node) { | ||
| 85 | list_del(&group->node); | ||
| 86 | iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | ||
| 87 | iopf_free_group(group); | ||
| 88 | } | ||
| 89 | |||
| 90 | xa_for_each(&fault->response, index, group) { | ||
| 91 | if (group->attach_handle != &handle->handle) | ||
| 92 | continue; | ||
| 93 | xa_erase(&fault->response, index); | ||
| 94 | iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | ||
| 95 | iopf_free_group(group); | ||
| 96 | } | ||
| 97 | mutex_unlock(&fault->mutex); | ||
| 98 | } | ||
| 99 | |||
| 100 | void iommufd_fault_destroy(struct iommufd_object *obj) | ||
| 101 | { | ||
| 102 | struct iommufd_fault *fault = container_of(obj, struct iommufd_fault, obj); | ||
| 103 | struct iopf_group *group, *next; | ||
| 104 | unsigned long index; | ||
| 105 | |||
| 106 | /* | ||
| 107 | * The iommufd object's reference count is zero at this point. | ||
| 108 | * We can be confident that no other threads are currently | ||
| 109 | * accessing this pointer. Therefore, acquiring the mutex here | ||
| 110 | * is unnecessary. | ||
| 111 | */ | ||
| 112 | list_for_each_entry_safe(group, next, &fault->deliver, node) { | ||
| 113 | list_del(&group->node); | ||
| 114 | iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | ||
| 115 | iopf_free_group(group); | ||
| 116 | } | ||
| 117 | xa_for_each(&fault->response, index, group) { | ||
| 118 | xa_erase(&fault->response, index); | ||
| 119 | iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | ||
| 120 | iopf_free_group(group); | ||
| 121 | } | ||
| 122 | xa_destroy(&fault->response); | ||
| 123 | mutex_destroy(&fault->mutex); | ||
| 124 | } | ||
| 125 | |||
| 126 | static void iommufd_compose_fault_message(struct iommu_fault *fault, | ||
| 127 | struct iommu_hwpt_pgfault *hwpt_fault, | ||
| 128 | struct iommufd_device *idev, | ||
| 129 | u32 cookie) | ||
| 130 | { | ||
| 131 | hwpt_fault->flags = fault->prm.flags; | ||
| 132 | hwpt_fault->dev_id = idev->obj.id; | ||
| 133 | hwpt_fault->pasid = fault->prm.pasid; | ||
| 134 | hwpt_fault->grpid = fault->prm.grpid; | ||
| 135 | hwpt_fault->perm = fault->prm.perm; | ||
| 136 | hwpt_fault->addr = fault->prm.addr; | ||
| 137 | hwpt_fault->length = 0; | ||
| 138 | hwpt_fault->cookie = cookie; | ||
| 139 | } | ||
| 140 | |||
| 141 | static ssize_t iommufd_fault_fops_read(struct file *filep, char __user *buf, | ||
| 142 | size_t count, loff_t *ppos) | ||
| 143 | { | ||
| 144 | size_t fault_size = sizeof(struct iommu_hwpt_pgfault); | ||
| 145 | struct iommufd_fault *fault = filep->private_data; | ||
| 146 | struct iommu_hwpt_pgfault data = {}; | ||
| 147 | struct iommufd_device *idev; | ||
| 148 | struct iopf_group *group; | ||
| 149 | struct iopf_fault *iopf; | ||
| 150 | size_t done = 0; | ||
| 151 | int rc = 0; | ||
| 152 | |||
| 153 | if (*ppos || count % fault_size) | ||
| 154 | return -ESPIPE; | ||
| 155 | |||
| 156 | mutex_lock(&fault->mutex); | ||
| 157 | while ((group = iommufd_fault_deliver_fetch(fault))) { | ||
| 158 | if (done >= count || | ||
| 159 | group->fault_count * fault_size > count - done) { | ||
| 160 | iommufd_fault_deliver_restore(fault, group); | ||
| 161 | break; | ||
| 162 | } | ||
| 163 | |||
| 164 | rc = xa_alloc(&fault->response, &group->cookie, group, | ||
| 165 | xa_limit_32b, GFP_KERNEL); | ||
| 166 | if (rc) { | ||
| 167 | iommufd_fault_deliver_restore(fault, group); | ||
| 168 | break; | ||
| 169 | } | ||
| 170 | |||
| 171 | idev = to_iommufd_handle(group->attach_handle)->idev; | ||
| 172 | list_for_each_entry(iopf, &group->faults, list) { | ||
| 173 | iommufd_compose_fault_message(&iopf->fault, | ||
| 174 | &data, idev, | ||
| 175 | group->cookie); | ||
| 176 | if (copy_to_user(buf + done, &data, fault_size)) { | ||
| 177 | xa_erase(&fault->response, group->cookie); | ||
| 178 | iommufd_fault_deliver_restore(fault, group); | ||
| 179 | rc = -EFAULT; | ||
| 180 | break; | ||
| 181 | } | ||
| 182 | done += fault_size; | ||
| 183 | } | ||
| 184 | } | ||
| 185 | mutex_unlock(&fault->mutex); | ||
| 186 | |||
| 187 | return done == 0 ? rc : done; | ||
| 188 | } | ||
| 189 | |||
| 190 | static ssize_t iommufd_fault_fops_write(struct file *filep, const char __user *buf, | ||
| 191 | size_t count, loff_t *ppos) | ||
| 192 | { | ||
| 193 | size_t response_size = sizeof(struct iommu_hwpt_page_response); | ||
| 194 | struct iommufd_fault *fault = filep->private_data; | ||
| 195 | struct iommu_hwpt_page_response response; | ||
| 196 | struct iopf_group *group; | ||
| 197 | size_t done = 0; | ||
| 198 | int rc = 0; | ||
| 199 | |||
| 200 | if (*ppos || count % response_size) | ||
| 201 | return -ESPIPE; | ||
| 202 | |||
| 203 | mutex_lock(&fault->mutex); | ||
| 204 | while (count > done) { | ||
| 205 | rc = copy_from_user(&response, buf + done, response_size); | ||
| 206 | if (rc) | ||
| 207 | break; | ||
| 208 | |||
| 209 | static_assert((int)IOMMUFD_PAGE_RESP_SUCCESS == | ||
| 210 | (int)IOMMU_PAGE_RESP_SUCCESS); | ||
| 211 | static_assert((int)IOMMUFD_PAGE_RESP_INVALID == | ||
| 212 | (int)IOMMU_PAGE_RESP_INVALID); | ||
| 213 | if (response.code != IOMMUFD_PAGE_RESP_SUCCESS && | ||
| 214 | response.code != IOMMUFD_PAGE_RESP_INVALID) { | ||
| 215 | rc = -EINVAL; | ||
| 216 | break; | ||
| 217 | } | ||
| 218 | |||
| 219 | group = xa_erase(&fault->response, response.cookie); | ||
| 220 | if (!group) { | ||
| 221 | rc = -EINVAL; | ||
| 222 | break; | ||
| 223 | } | ||
| 224 | |||
| 225 | iopf_group_response(group, response.code); | ||
| 226 | iopf_free_group(group); | ||
| 227 | done += response_size; | ||
| 228 | } | ||
| 229 | mutex_unlock(&fault->mutex); | ||
| 230 | |||
| 231 | return done == 0 ? rc : done; | ||
| 232 | } | ||
| 233 | |||
| 234 | static __poll_t iommufd_fault_fops_poll(struct file *filep, | ||
| 235 | struct poll_table_struct *wait) | ||
| 236 | { | ||
| 237 | struct iommufd_fault *fault = filep->private_data; | ||
| 238 | __poll_t pollflags = EPOLLOUT; | ||
| 239 | |||
| 240 | poll_wait(filep, &fault->wait_queue, wait); | ||
| 241 | spin_lock(&fault->lock); | ||
| 242 | if (!list_empty(&fault->deliver)) | ||
| 243 | pollflags |= EPOLLIN | EPOLLRDNORM; | ||
| 244 | spin_unlock(&fault->lock); | ||
| 245 | |||
| 246 | return pollflags; | ||
| 247 | } | ||
| 248 | |||
| 249 | static int iommufd_fault_fops_release(struct inode *inode, struct file *filep) | ||
| 250 | { | ||
| 251 | struct iommufd_fault *fault = filep->private_data; | ||
| 252 | |||
| 253 | refcount_dec(&fault->obj.users); | ||
| 254 | iommufd_ctx_put(fault->ictx); | ||
| 255 | return 0; | ||
| 256 | } | ||
| 257 | |||
| 258 | static const struct file_operations iommufd_fault_fops = { | ||
| 259 | .owner = THIS_MODULE, | ||
| 260 | .open = nonseekable_open, | ||
| 261 | .read = iommufd_fault_fops_read, | ||
| 262 | .write = iommufd_fault_fops_write, | ||
| 263 | .poll = iommufd_fault_fops_poll, | ||
| 264 | .release = iommufd_fault_fops_release, | ||
| 265 | }; | ||
| 266 | |||
| 267 | int iommufd_fault_alloc(struct iommufd_ucmd *ucmd) | ||
| 268 | { | ||
| 269 | struct iommu_fault_alloc *cmd = ucmd->cmd; | ||
| 270 | struct iommufd_fault *fault; | ||
| 271 | struct file *filep; | ||
| 272 | int fdno; | ||
| 273 | int rc; | ||
| 274 | |||
| 275 | if (cmd->flags) | ||
| 276 | return -EOPNOTSUPP; | ||
| 277 | |||
| 278 | fault = iommufd_object_alloc(ucmd->ictx, fault, IOMMUFD_OBJ_FAULT); | ||
| 279 | if (IS_ERR(fault)) | ||
| 280 | return PTR_ERR(fault); | ||
| 281 | |||
| 282 | fault->ictx = ucmd->ictx; | ||
| 283 | INIT_LIST_HEAD(&fault->deliver); | ||
| 284 | xa_init_flags(&fault->response, XA_FLAGS_ALLOC1); | ||
| 285 | mutex_init(&fault->mutex); | ||
| 286 | spin_lock_init(&fault->lock); | ||
| 287 | init_waitqueue_head(&fault->wait_queue); | ||
| 288 | |||
| 289 | filep = anon_inode_getfile("[iommufd-pgfault]", &iommufd_fault_fops, | ||
| 290 | fault, O_RDWR); | ||
| 291 | if (IS_ERR(filep)) { | ||
| 292 | rc = PTR_ERR(filep); | ||
| 293 | goto out_abort; | ||
| 294 | } | ||
| 295 | |||
| 296 | refcount_inc(&fault->obj.users); | ||
| 297 | iommufd_ctx_get(fault->ictx); | ||
| 298 | fault->filep = filep; | ||
| 299 | |||
| 300 | fdno = get_unused_fd_flags(O_CLOEXEC); | ||
| 301 | if (fdno < 0) { | ||
| 302 | rc = fdno; | ||
| 303 | goto out_fput; | ||
| 304 | } | ||
| 305 | |||
| 306 | cmd->out_fault_id = fault->obj.id; | ||
| 307 | cmd->out_fault_fd = fdno; | ||
| 308 | |||
| 309 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); | ||
| 310 | if (rc) | ||
| 311 | goto out_put_fdno; | ||
| 312 | iommufd_object_finalize(ucmd->ictx, &fault->obj); | ||
| 313 | |||
| 314 | fd_install(fdno, fault->filep); | ||
| 315 | |||
| 316 | return 0; | ||
| 317 | out_put_fdno: | ||
| 318 | put_unused_fd(fdno); | ||
| 319 | out_fput: | ||
| 320 | fput(filep); | ||
| 321 | out_abort: | ||
| 322 | iommufd_object_abort_and_destroy(ucmd->ictx, &fault->obj); | ||
| 323 | |||
| 324 | return rc; | ||
| 325 | } | ||
| 326 | |||
| 327 | int iommufd_fault_iopf_handler(struct iopf_group *group) | ||
| 328 | { | ||
| 329 | struct iommufd_hw_pagetable *hwpt; | ||
| 330 | struct iommufd_fault *fault; | ||
| 331 | |||
| 332 | hwpt = group->attach_handle->domain->iommufd_hwpt; | ||
| 333 | fault = hwpt->fault; | ||
| 334 | |||
| 335 | spin_lock(&fault->lock); | ||
| 336 | list_add_tail(&group->node, &fault->deliver); | ||
| 337 | spin_unlock(&fault->lock); | ||
| 338 | |||
| 339 | wake_up_interruptible(&fault->wait_queue); | ||
| 340 | |||
| 341 | return 0; | ||
| 342 | } | ||
| @@ -14,7 +14,7 @@ static void __iommufd_hwpt_destroy(struct iommufd_hw_pagetable *hwpt) · 13 unmodified lines | |||
| 14 | iommu_domain_free(hwpt->domain); | 14 | iommu_domain_free(hwpt->domain); |
| 15 | 15 | ||
| 16 | if (hwpt->fault) | 16 | if (hwpt->fault) |
| 17 | refcount_dec(&hwpt->fault->obj.users); | 17 | refcount_dec(&hwpt->fault->common.obj.users); |
| 18 | } | 18 | } |
| 19 | 19 | ||
| 20 | void iommufd_hwpt_paging_destroy(struct iommufd_object *obj) | 20 | void iommufd_hwpt_paging_destroy(struct iommufd_object *obj) |
| @@ -90,6 +90,7 @@ iommufd_hwpt_paging_enforce_cc(struct iommufd_hwpt_paging *hwpt_paging) · 69 unmodified lines | |||
| 90 | * @ictx: iommufd context | 90 | * @ictx: iommufd context |
| 91 | * @ioas: IOAS to associate the domain with | 91 | * @ioas: IOAS to associate the domain with |
| 92 | * @idev: Device to get an iommu_domain for | 92 | * @idev: Device to get an iommu_domain for |
| 93 | * @pasid: PASID to get an iommu_domain for | ||
| 93 | * @flags: Flags from userspace | 94 | * @flags: Flags from userspace |
| 94 | * @immediate_attach: True if idev should be attached to the hwpt | 95 | * @immediate_attach: True if idev should be attached to the hwpt |
| 95 | * @user_data: The user provided driver specific data describing the domain to | 96 | * @user_data: The user provided driver specific data describing the domain to |
| @@ -105,13 +106,14 @@ iommufd_hwpt_paging_enforce_cc(struct iommufd_hwpt_paging *hwpt_paging) · 9 unmodified lines | |||
| 105 | */ | 106 | */ |
| 106 | struct iommufd_hwpt_paging * | 107 | struct iommufd_hwpt_paging * |
| 107 | iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, | 108 | iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, |
| 108 | struct iommufd_device *idev, u32 flags, | 109 | struct iommufd_device *idev, ioasid_t pasid, |
| 109 | bool immediate_attach, | 110 | u32 flags, bool immediate_attach, |
| 110 | const struct iommu_user_data *user_data) | 111 | const struct iommu_user_data *user_data) |
| 111 | { | 112 | { |
| 112 | const u32 valid_flags = IOMMU_HWPT_ALLOC_NEST_PARENT | | 113 | const u32 valid_flags = IOMMU_HWPT_ALLOC_NEST_PARENT | |
| 113 | IOMMU_HWPT_ALLOC_DIRTY_TRACKING | | 114 | IOMMU_HWPT_ALLOC_DIRTY_TRACKING | |
| 114 | IOMMU_HWPT_FAULT_ID_VALID; | 115 | IOMMU_HWPT_FAULT_ID_VALID | |
| 116 | IOMMU_HWPT_ALLOC_PASID; | ||
| 115 | const struct iommu_ops *ops = dev_iommu_ops(idev->dev); | 117 | const struct iommu_ops *ops = dev_iommu_ops(idev->dev); |
| 116 | struct iommufd_hwpt_paging *hwpt_paging; | 118 | struct iommufd_hwpt_paging *hwpt_paging; |
| 117 | struct iommufd_hw_pagetable *hwpt; | 119 | struct iommufd_hw_pagetable *hwpt; |
| @@ -126,12 +128,16 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 8 unmodified lines | |||
| 126 | if ((flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING) && | 128 | if ((flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING) && |
| 127 | !device_iommu_capable(idev->dev, IOMMU_CAP_DIRTY_TRACKING)) | 129 | !device_iommu_capable(idev->dev, IOMMU_CAP_DIRTY_TRACKING)) |
| 128 | return ERR_PTR(-EOPNOTSUPP); | 130 | return ERR_PTR(-EOPNOTSUPP); |
| 131 | if ((flags & IOMMU_HWPT_FAULT_ID_VALID) && | ||
| 132 | (flags & IOMMU_HWPT_ALLOC_NEST_PARENT)) | ||
| 133 | return ERR_PTR(-EOPNOTSUPP); | ||
| 129 | 134 | ||
| 130 | hwpt_paging = __iommufd_object_alloc( | 135 | hwpt_paging = __iommufd_object_alloc( |
| 131 | ictx, hwpt_paging, IOMMUFD_OBJ_HWPT_PAGING, common.obj); | 136 | ictx, hwpt_paging, IOMMUFD_OBJ_HWPT_PAGING, common.obj); |
| 132 | if (IS_ERR(hwpt_paging)) | 137 | if (IS_ERR(hwpt_paging)) |
| 133 | return ERR_CAST(hwpt_paging); | 138 | return ERR_CAST(hwpt_paging); |
| 134 | hwpt = &hwpt_paging->common; | 139 | hwpt = &hwpt_paging->common; |
| 140 | hwpt->pasid_compat = flags & IOMMU_HWPT_ALLOC_PASID; | ||
| 135 | 141 | ||
| 136 | INIT_LIST_HEAD(&hwpt_paging->hwpt_item); | 142 | INIT_LIST_HEAD(&hwpt_paging->hwpt_item); |
| 137 | /* Pairs with iommufd_hw_pagetable_destroy() */ | 143 | /* Pairs with iommufd_hw_pagetable_destroy() */ |
| @@ -156,7 +162,8 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 18 unmodified lines | |||
| 156 | goto out_abort; | 162 | goto out_abort; |
| 157 | } | 163 | } |
| 158 | } | 164 | } |
| 159 | iommu_domain_set_sw_msi(hwpt->domain, iommufd_sw_msi); | 165 | hwpt->domain->iommufd_hwpt = hwpt; |
| 166 | hwpt->domain->cookie_type = IOMMU_COOKIE_IOMMUFD; | ||
| 160 | 167 | ||
| 161 | /* | 168 | /* |
| 162 | * Set the coherency mode before we do iopt_table_add_domain() as some | 169 | * Set the coherency mode before we do iopt_table_add_domain() as some |
| @@ -185,7 +192,7 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 22 unmodified lines | |||
| 185 | * sequence. Once those drivers are fixed this should be removed. | 192 | * sequence. Once those drivers are fixed this should be removed. |
| 186 | */ | 193 | */ |
| 187 | if (immediate_attach) { | 194 | if (immediate_attach) { |
| 188 | rc = iommufd_hw_pagetable_attach(hwpt, idev); | 195 | rc = iommufd_hw_pagetable_attach(hwpt, idev, pasid); |
| 189 | if (rc) | 196 | if (rc) |
| 190 | goto out_abort; | 197 | goto out_abort; |
| 191 | } | 198 | } |
| @@ -198,7 +205,7 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 6 unmodified lines | |||
| 198 | 205 | ||
| 199 | out_detach: | 206 | out_detach: |
| 200 | if (immediate_attach) | 207 | if (immediate_attach) |
| 201 | iommufd_hw_pagetable_detach(idev); | 208 | iommufd_hw_pagetable_detach(idev, pasid); |
| 202 | out_abort: | 209 | out_abort: |
| 203 | iommufd_object_abort_and_destroy(ictx, &hwpt->obj); | 210 | iommufd_object_abort_and_destroy(ictx, &hwpt->obj); |
| 204 | return ERR_PTR(rc); | 211 | return ERR_PTR(rc); |
| @@ -227,7 +234,7 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, · 22 unmodified lines | |||
| 227 | struct iommufd_hw_pagetable *hwpt; | 234 | struct iommufd_hw_pagetable *hwpt; |
| 228 | int rc; | 235 | int rc; |
| 229 | 236 | ||
| 230 | if ((flags & ~IOMMU_HWPT_FAULT_ID_VALID) || | 237 | if ((flags & ~(IOMMU_HWPT_FAULT_ID_VALID | IOMMU_HWPT_ALLOC_PASID)) || |
| 231 | !user_data->len || !ops->domain_alloc_nested) | 238 | !user_data->len || !ops->domain_alloc_nested) |
| 232 | return ERR_PTR(-EOPNOTSUPP); | 239 | return ERR_PTR(-EOPNOTSUPP); |
| 233 | if (parent->auto_domain || !parent->nest_parent || | 240 | if (parent->auto_domain || !parent->nest_parent || |
| @@ -239,6 +246,7 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, · 5 unmodified lines | |||
| 239 | if (IS_ERR(hwpt_nested)) | 246 | if (IS_ERR(hwpt_nested)) |
| 240 | return ERR_CAST(hwpt_nested); | 247 | return ERR_CAST(hwpt_nested); |
| 241 | hwpt = &hwpt_nested->common; | 248 | hwpt = &hwpt_nested->common; |
| 249 | hwpt->pasid_compat = flags & IOMMU_HWPT_ALLOC_PASID; | ||
| 242 | 250 | ||
| 243 | refcount_inc(&parent->common.obj.users); | 251 | refcount_inc(&parent->common.obj.users); |
| 244 | hwpt_nested->parent = parent; | 252 | hwpt_nested->parent = parent; |
| @@ -252,7 +260,8 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, · 7 unmodified lines | |||
| 252 | goto out_abort; | 260 | goto out_abort; |
| 253 | } | 261 | } |
| 254 | hwpt->domain->owner = ops; | 262 | hwpt->domain->owner = ops; |
| 255 | iommu_domain_set_sw_msi(hwpt->domain, iommufd_sw_msi); | 263 | hwpt->domain->iommufd_hwpt = hwpt; |
| 264 | hwpt->domain->cookie_type = IOMMU_COOKIE_IOMMUFD; | ||
| 256 | 265 | ||
| 257 | if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) { | 266 | if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) { |
| 258 | rc = -EINVAL; | 267 | rc = -EINVAL; |
| @@ -282,7 +291,7 @@ iommufd_viommu_alloc_hwpt_nested(struct iommufd_viommu *viommu, u32 flags, · 23 unmodified lines | |||
| 282 | struct iommufd_hw_pagetable *hwpt; | 291 | struct iommufd_hw_pagetable *hwpt; |
| 283 | int rc; | 292 | int rc; |
| 284 | 293 | ||
| 285 | if (flags & ~IOMMU_HWPT_FAULT_ID_VALID) | 294 | if (flags & ~(IOMMU_HWPT_FAULT_ID_VALID | IOMMU_HWPT_ALLOC_PASID)) |
| 286 | return ERR_PTR(-EOPNOTSUPP); | 295 | return ERR_PTR(-EOPNOTSUPP); |
| 287 | if (!user_data->len) | 296 | if (!user_data->len) |
| 288 | return ERR_PTR(-EOPNOTSUPP); | 297 | return ERR_PTR(-EOPNOTSUPP); |
| @@ -294,6 +303,7 @@ iommufd_viommu_alloc_hwpt_nested(struct iommufd_viommu *viommu, u32 flags, · 5 unmodified lines | |||
| 294 | if (IS_ERR(hwpt_nested)) | 303 | if (IS_ERR(hwpt_nested)) |
| 295 | return ERR_CAST(hwpt_nested); | 304 | return ERR_CAST(hwpt_nested); |
| 296 | hwpt = &hwpt_nested->common; | 305 | hwpt = &hwpt_nested->common; |
| 306 | hwpt->pasid_compat = flags & IOMMU_HWPT_ALLOC_PASID; | ||
| 297 | 307 | ||
| 298 | hwpt_nested->viommu = viommu; | 308 | hwpt_nested->viommu = viommu; |
| 299 | refcount_inc(&viommu->obj.users); | 309 | refcount_inc(&viommu->obj.users); |
| @@ -308,8 +318,9 @@ iommufd_viommu_alloc_hwpt_nested(struct iommufd_viommu *viommu, u32 flags, · 8 unmodified lines | |||
| 308 | hwpt->domain = NULL; | 318 | hwpt->domain = NULL; |
| 309 | goto out_abort; | 319 | goto out_abort; |
| 310 | } | 320 | } |
| 321 | hwpt->domain->iommufd_hwpt = hwpt; | ||
| 311 | hwpt->domain->owner = viommu->iommu_dev->ops; | 322 | hwpt->domain->owner = viommu->iommu_dev->ops; |
| 312 | iommu_domain_set_sw_msi(hwpt->domain, iommufd_sw_msi); | 323 | hwpt->domain->cookie_type = IOMMU_COOKIE_IOMMUFD; |
| 313 | 324 | ||
| 314 | if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) { | 325 | if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) { |
| 315 | rc = -EINVAL; | 326 | rc = -EINVAL; |
| @@ -358,8 +369,8 @@ int iommufd_hwpt_alloc(struct iommufd_ucmd *ucmd) · 42 unmodified lines | |||
| 358 | ioas = container_of(pt_obj, struct iommufd_ioas, obj); | 369 | ioas = container_of(pt_obj, struct iommufd_ioas, obj); |
| 359 | mutex_lock(&ioas->mutex); | 370 | mutex_lock(&ioas->mutex); |
| 360 | hwpt_paging = iommufd_hwpt_paging_alloc( | 371 | hwpt_paging = iommufd_hwpt_paging_alloc( |
| 361 | ucmd->ictx, ioas, idev, cmd->flags, false, | 372 | ucmd->ictx, ioas, idev, IOMMU_NO_PASID, cmd->flags, |
| 362 | user_data.len ? &user_data : NULL); | 373 | false, user_data.len ? &user_data : NULL); |
| 363 | if (IS_ERR(hwpt_paging)) { | 374 | if (IS_ERR(hwpt_paging)) { |
| 364 | rc = PTR_ERR(hwpt_paging); | 375 | rc = PTR_ERR(hwpt_paging); |
| 365 | goto out_unlock; | 376 | goto out_unlock; |
| @@ -409,10 +420,9 @@ int iommufd_hwpt_alloc(struct iommufd_ucmd *ucmd) · 43 unmodified lines | |||
| 409 | } | 420 | } |
| 410 | hwpt->fault = fault; | 421 | hwpt->fault = fault; |
| 411 | hwpt->domain->iopf_handler = iommufd_fault_iopf_handler; | 422 | hwpt->domain->iopf_handler = iommufd_fault_iopf_handler; |
| 412 | refcount_inc(&fault->obj.users); | 423 | refcount_inc(&fault->common.obj.users); |
| 413 | iommufd_put_object(ucmd->ictx, &fault->obj); | 424 | iommufd_put_object(ucmd->ictx, &fault->common.obj); |
| 414 | } | 425 | } |
| 415 | hwpt->domain->iommufd_hwpt = hwpt; | ||
| 416 | 426 | ||
| 417 | cmd->out_hwpt_id = hwpt->obj.id; | 427 | cmd->out_hwpt_id = hwpt->obj.id; |
| 418 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); | 428 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); |
| @@ -32,8 +32,11 @@ struct iommufd_sw_msi_maps { · 31 unmodified lines | |||
| 32 | DECLARE_BITMAP(bitmap, 64); | 32 | DECLARE_BITMAP(bitmap, 64); |
| 33 | }; | 33 | }; |
| 34 | 34 | ||
| 35 | int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | 35 | #ifdef CONFIG_IRQ_MSI_IOMMU |
| 36 | phys_addr_t msi_addr); | 36 | int iommufd_sw_msi_install(struct iommufd_ctx *ictx, |
| 37 | struct iommufd_hwpt_paging *hwpt_paging, | ||
| 38 | struct iommufd_sw_msi_map *msi_map); | ||
| 39 | #endif | ||
| 37 | 40 | ||
| 38 | struct iommufd_ctx { | 41 | struct iommufd_ctx { |
| 39 | struct file *file; | 42 | struct file *file; |
| @@ -296,6 +299,7 @@ struct iommufd_hw_pagetable { · 256 unmodified lines | |||
| 296 | struct iommufd_object obj; | 299 | struct iommufd_object obj; |
| 297 | struct iommu_domain *domain; | 300 | struct iommu_domain *domain; |
| 298 | struct iommufd_fault *fault; | 301 | struct iommufd_fault *fault; |
| 302 | bool pasid_compat : 1; | ||
| 299 | }; | 303 | }; |
| 300 | 304 | ||
| 301 | struct iommufd_hwpt_paging { | 305 | struct iommufd_hwpt_paging { |
| @@ -366,13 +370,13 @@ int iommufd_hwpt_get_dirty_bitmap(struct iommufd_ucmd *ucmd); · 64 unmodified lines | |||
| 366 | 370 | ||
| 367 | struct iommufd_hwpt_paging * | 371 | struct iommufd_hwpt_paging * |
| 368 | iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, | 372 | iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, |
| 369 | struct iommufd_device *idev, u32 flags, | 373 | struct iommufd_device *idev, ioasid_t pasid, |
| 370 | bool immediate_attach, | 374 | u32 flags, bool immediate_attach, |
| 371 | const struct iommu_user_data *user_data); | 375 | const struct iommu_user_data *user_data); |
| 372 | int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt, | 376 | int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt, |
| 373 | struct iommufd_device *idev); | 377 | struct iommufd_device *idev, ioasid_t pasid); |
| 374 | struct iommufd_hw_pagetable * | 378 | struct iommufd_hw_pagetable * |
| 375 | iommufd_hw_pagetable_detach(struct iommufd_device *idev); | 379 | iommufd_hw_pagetable_detach(struct iommufd_device *idev, ioasid_t pasid); |
| 376 | void iommufd_hwpt_paging_destroy(struct iommufd_object *obj); | 380 | void iommufd_hwpt_paging_destroy(struct iommufd_object *obj); |
| 377 | void iommufd_hwpt_paging_abort(struct iommufd_object *obj); | 381 | void iommufd_hwpt_paging_abort(struct iommufd_object *obj); |
| 378 | void iommufd_hwpt_nested_destroy(struct iommufd_object *obj); | 382 | void iommufd_hwpt_nested_destroy(struct iommufd_object *obj); |
| @@ -396,13 +400,14 @@ static inline void iommufd_hw_pagetable_put(struct iommufd_ctx *ictx, · 17 unmodified lines | |||
| 396 | refcount_dec(&hwpt->obj.users); | 400 | refcount_dec(&hwpt->obj.users); |
| 397 | } | 401 | } |
| 398 | 402 | ||
| 403 | struct iommufd_attach; | ||
| 404 | |||
| 399 | struct iommufd_group { | 405 | struct iommufd_group { |
| 400 | struct kref ref; | 406 | struct kref ref; |
| 401 | struct mutex lock; | 407 | struct mutex lock; |
| 402 | struct iommufd_ctx *ictx; | 408 | struct iommufd_ctx *ictx; |
| 403 | struct iommu_group *group; | 409 | struct iommu_group *group; |
| 404 | struct iommufd_hw_pagetable *hwpt; | 410 | struct xarray pasid_attach; |
| 405 | struct list_head device_list; | ||
| 406 | struct iommufd_sw_msi_maps required_sw_msi; | 411 | struct iommufd_sw_msi_maps required_sw_msi; |
| 407 | phys_addr_t sw_msi_start; | 412 | phys_addr_t sw_msi_start; |
| 408 | }; | 413 | }; |
| @@ -454,49 +459,17 @@ void iopt_remove_access(struct io_pagetable *iopt, · 45 unmodified lines | |||
| 454 | u32 iopt_access_list_id); | 459 | u32 iopt_access_list_id); |
| 455 | void iommufd_access_destroy_object(struct iommufd_object *obj); | 460 | void iommufd_access_destroy_object(struct iommufd_object *obj); |
| 456 | 461 | ||
| 457 | /* | 462 | struct iommufd_eventq { |
| 458 | * An iommufd_fault object represents an interface to deliver I/O page faults | ||
| 459 | * to the user space. These objects are created/destroyed by the user space and | ||
| 460 | * associated with hardware page table objects during page-table allocation. | ||
| 461 | */ | ||
| 462 | struct iommufd_fault { | ||
| 463 | struct iommufd_object obj; | 463 | struct iommufd_object obj; |
| 464 | struct iommufd_ctx *ictx; | 464 | struct iommufd_ctx *ictx; |
| 465 | struct file *filep; | 465 | struct file *filep; |
| 466 | 466 | ||
| 467 | spinlock_t lock; /* protects the deliver list */ | 467 | spinlock_t lock; /* protects the deliver list */ |
| 468 | struct list_head deliver; | 468 | struct list_head deliver; |
| 469 | struct mutex mutex; /* serializes response flows */ | ||
| 470 | struct xarray response; | ||
| 471 | 469 | ||
| 472 | struct wait_queue_head wait_queue; | 470 | struct wait_queue_head wait_queue; |
| 473 | }; | 471 | }; |
| 474 | 472 | ||
| 475 | /* Fetch the first node out of the fault->deliver list */ | ||
| 476 | static inline struct iopf_group * | ||
| 477 | iommufd_fault_deliver_fetch(struct iommufd_fault *fault) | ||
| 478 | { | ||
| 479 | struct list_head *list = &fault->deliver; | ||
| 480 | struct iopf_group *group = NULL; | ||
| 481 | |||
| 482 | spin_lock(&fault->lock); | ||
| 483 | if (!list_empty(list)) { | ||
| 484 | group = list_first_entry(list, struct iopf_group, node); | ||
| 485 | list_del(&group->node); | ||
| 486 | } | ||
| 487 | spin_unlock(&fault->lock); | ||
| 488 | return group; | ||
| 489 | } | ||
| 490 | |||
| 491 | /* Restore a node back to the head of the fault->deliver list */ | ||
| 492 | static inline void iommufd_fault_deliver_restore(struct iommufd_fault *fault, | ||
| 493 | struct iopf_group *group) | ||
| 494 | { | ||
| 495 | spin_lock(&fault->lock); | ||
| 496 | list_add(&group->node, &fault->deliver); | ||
| 497 | spin_unlock(&fault->lock); | ||
| 498 | } | ||
| 499 | |||
| 500 | struct iommufd_attach_handle { | 473 | struct iommufd_attach_handle { |
| 501 | struct iommu_attach_handle handle; | 474 | struct iommu_attach_handle handle; |
| 502 | struct iommufd_device *idev; | 475 | struct iommufd_device *idev; |
| @@ -505,12 +478,29 @@ struct iommufd_attach_handle { · 2 unmodified lines | |||
| 505 | /* Convert an iommu attach handle to iommufd handle. */ | 478 | /* Convert an iommu attach handle to iommufd handle. */ |
| 506 | #define to_iommufd_handle(hdl) container_of(hdl, struct iommufd_attach_handle, handle) | 479 | #define to_iommufd_handle(hdl) container_of(hdl, struct iommufd_attach_handle, handle) |
| 507 | 480 | ||
| 481 | /* | ||
| 482 | * An iommufd_fault object represents an interface to deliver I/O page faults | ||
| 483 | * to the user space. These objects are created/destroyed by the user space and | ||
| 484 | * associated with hardware page table objects during page-table allocation. | ||
| 485 | */ | ||
| 486 | struct iommufd_fault { | ||
| 487 | struct iommufd_eventq common; | ||
| 488 | struct mutex mutex; /* serializes response flows */ | ||
| 489 | struct xarray response; | ||
| 490 | }; | ||
| 491 | |||
| 492 | static inline struct iommufd_fault * | ||
| 493 | eventq_to_fault(struct iommufd_eventq *eventq) | ||
| 494 | { | ||
| 495 | return container_of(eventq, struct iommufd_fault, common); | ||
| 496 | } | ||
| 497 | |||
| 508 | static inline struct iommufd_fault * | 498 | static inline struct iommufd_fault * |
| 509 | iommufd_get_fault(struct iommufd_ucmd *ucmd, u32 id) | 499 | iommufd_get_fault(struct iommufd_ucmd *ucmd, u32 id) |
| 510 | { | 500 | { |
| 511 | return container_of(iommufd_get_object(ucmd->ictx, id, | 501 | return container_of(iommufd_get_object(ucmd->ictx, id, |
| 512 | IOMMUFD_OBJ_FAULT), | 502 | IOMMUFD_OBJ_FAULT), |
| 513 | struct iommufd_fault, obj); | 503 | struct iommufd_fault, common.obj); |
| 514 | } | 504 | } |
| 515 | 505 | ||
| 516 | int iommufd_fault_alloc(struct iommufd_ucmd *ucmd); | 506 | int iommufd_fault_alloc(struct iommufd_ucmd *ucmd); |
| @@ -522,6 +512,74 @@ void iommufd_fault_iopf_disable(struct iommufd_device *idev); · 5 unmodified lines | |||
| 522 | void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt, | 512 | void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt, |
| 523 | struct iommufd_attach_handle *handle); | 513 | struct iommufd_attach_handle *handle); |
| 524 | 514 | ||
| 515 | /* An iommufd_vevent represents a vIOMMU event in an iommufd_veventq */ | ||
| 516 | struct iommufd_vevent { | ||
| 517 | struct iommufd_vevent_header header; | ||
| 518 | struct list_head node; /* for iommufd_eventq::deliver */ | ||
| 519 | ssize_t data_len; | ||
| 520 | u64 event_data[] __counted_by(data_len); | ||
| 521 | }; | ||
| 522 | |||
| 523 | #define vevent_for_lost_events_header(vevent) \ | ||
| 524 | (vevent->header.flags & IOMMU_VEVENTQ_FLAG_LOST_EVENTS) | ||
| 525 | |||
| 526 | /* | ||
| 527 | * An iommufd_veventq object represents an interface to deliver vIOMMU events to | ||
| 528 | * the user space. It is created/destroyed by the user space and associated with | ||
| 529 | * a vIOMMU object during the allocations. | ||
| 530 | */ | ||
| 531 | struct iommufd_veventq { | ||
| 532 | struct iommufd_eventq common; | ||
| 533 | struct iommufd_viommu *viommu; | ||
| 534 | struct list_head node; /* for iommufd_viommu::veventqs */ | ||
| 535 | struct iommufd_vevent lost_events_header; | ||
| 536 | |||
| 537 | unsigned int type; | ||
| 538 | unsigned int depth; | ||
| 539 | |||
| 540 | /* Use common.lock for protection */ | ||
| 541 | u32 num_events; | ||
| 542 | u32 sequence; | ||
| 543 | }; | ||
| 544 | |||
| 545 | static inline struct iommufd_veventq * | ||
| 546 | eventq_to_veventq(struct iommufd_eventq *eventq) | ||
| 547 | { | ||
| 548 | return container_of(eventq, struct iommufd_veventq, common); | ||
| 549 | } | ||
| 550 | |||
| 551 | static inline struct iommufd_veventq * | ||
| 552 | iommufd_get_veventq(struct iommufd_ucmd *ucmd, u32 id) | ||
| 553 | { | ||
| 554 | return container_of(iommufd_get_object(ucmd->ictx, id, | ||
| 555 | IOMMUFD_OBJ_VEVENTQ), | ||
| 556 | struct iommufd_veventq, common.obj); | ||
| 557 | } | ||
| 558 | |||
| 559 | int iommufd_veventq_alloc(struct iommufd_ucmd *ucmd); | ||
| 560 | void iommufd_veventq_destroy(struct iommufd_object *obj); | ||
| 561 | void iommufd_veventq_abort(struct iommufd_object *obj); | ||
| 562 | |||
| 563 | static inline void iommufd_vevent_handler(struct iommufd_veventq *veventq, | ||
| 564 | struct iommufd_vevent *vevent) | ||
| 565 | { | ||
| 566 | struct iommufd_eventq *eventq = &veventq->common; | ||
| 567 | |||
| 568 | lockdep_assert_held(&eventq->lock); | ||
| 569 | |||
| 570 | /* | ||
| 571 | * Remove the lost_events_header and add the new node at the same time. | ||
| 572 | * Note the new node can be lost_events_header, for a sequence update. | ||
| 573 | */ | ||
| 574 | if (list_is_last(&veventq->lost_events_header.node, &eventq->deliver)) | ||
| 575 | list_del(&veventq->lost_events_header.node); | ||
| 576 | list_add_tail(&vevent->node, &eventq->deliver); | ||
| 577 | vevent->header.sequence = veventq->sequence; | ||
| 578 | veventq->sequence = (veventq->sequence + 1) & INT_MAX; | ||
| 579 | |||
| 580 | wake_up_interruptible(&eventq->wait_queue); | ||
| 581 | } | ||
| 582 | |||
| 525 | static inline struct iommufd_viommu * | 583 | static inline struct iommufd_viommu * |
| 526 | iommufd_get_viommu(struct iommufd_ucmd *ucmd, u32 id) | 584 | iommufd_get_viommu(struct iommufd_ucmd *ucmd, u32 id) |
| 527 | { | 585 | { |
| @@ -530,6 +588,20 @@ iommufd_get_viommu(struct iommufd_ucmd *ucmd, u32 id) · 2 unmodified lines | |||
| 530 | struct iommufd_viommu, obj); | 588 | struct iommufd_viommu, obj); |
| 531 | } | 589 | } |
| 532 | 590 | ||
| 591 | static inline struct iommufd_veventq * | ||
| 592 | iommufd_viommu_find_veventq(struct iommufd_viommu *viommu, u32 type) | ||
| 593 | { | ||
| 594 | struct iommufd_veventq *veventq, *next; | ||
| 595 | |||
| 596 | lockdep_assert_held(&viommu->veventqs_rwsem); | ||
| 597 | |||
| 598 | list_for_each_entry_safe(veventq, next, &viommu->veventqs, node) { | ||
| 599 | if (veventq->type == type) | ||
| 600 | return veventq; | ||
| 601 | } | ||
| 602 | return NULL; | ||
| 603 | } | ||
| 604 | |||
| 533 | int iommufd_viommu_alloc_ioctl(struct iommufd_ucmd *ucmd); | 605 | int iommufd_viommu_alloc_ioctl(struct iommufd_ucmd *ucmd); |
| 534 | void iommufd_viommu_destroy(struct iommufd_object *obj); | 606 | void iommufd_viommu_destroy(struct iommufd_object *obj); |
| 535 | int iommufd_vdevice_alloc_ioctl(struct iommufd_ucmd *ucmd); | 607 | int iommufd_vdevice_alloc_ioctl(struct iommufd_ucmd *ucmd); |
| @@ -24,6 +24,11 @@ enum { · 23 unmodified lines | |||
| 24 | IOMMU_TEST_OP_MD_CHECK_IOTLB, | 24 | IOMMU_TEST_OP_MD_CHECK_IOTLB, |
| 25 | IOMMU_TEST_OP_TRIGGER_IOPF, | 25 | IOMMU_TEST_OP_TRIGGER_IOPF, |
| 26 | IOMMU_TEST_OP_DEV_CHECK_CACHE, | 26 | IOMMU_TEST_OP_DEV_CHECK_CACHE, |
| 27 | IOMMU_TEST_OP_TRIGGER_VEVENT, | ||
| 28 | IOMMU_TEST_OP_PASID_ATTACH, | ||
| 29 | IOMMU_TEST_OP_PASID_REPLACE, | ||
| 30 | IOMMU_TEST_OP_PASID_DETACH, | ||
| 31 | IOMMU_TEST_OP_PASID_CHECK_HWPT, | ||
| 27 | }; | 32 | }; |
| 28 | 33 | ||
| 29 | enum { | 34 | enum { |
| @@ -48,6 +53,7 @@ enum { · 18 unmodified lines | |||
| 48 | enum { | 53 | enum { |
| 49 | MOCK_FLAGS_DEVICE_NO_DIRTY = 1 << 0, | 54 | MOCK_FLAGS_DEVICE_NO_DIRTY = 1 << 0, |
| 50 | MOCK_FLAGS_DEVICE_HUGE_IOVA = 1 << 1, | 55 | MOCK_FLAGS_DEVICE_HUGE_IOVA = 1 << 1, |
| 56 | MOCK_FLAGS_DEVICE_PASID = 1 << 2, | ||
| 51 | }; | 57 | }; |
| 52 | 58 | ||
| 53 | enum { | 59 | enum { |
| @@ -60,6 +66,9 @@ enum { · 6 unmodified lines | |||
| 60 | MOCK_DEV_CACHE_NUM = 4, | 66 | MOCK_DEV_CACHE_NUM = 4, |
| 61 | }; | 67 | }; |
| 62 | 68 | ||
| 69 | /* Reserved for special pasid replace test */ | ||
| 70 | #define IOMMU_TEST_PASID_RESERVED 1024 | ||
| 71 | |||
| 63 | struct iommu_test_cmd { | 72 | struct iommu_test_cmd { |
| 64 | __u32 size; | 73 | __u32 size; |
| 65 | __u32 op; | 74 | __u32 op; |
| @@ -145,11 +154,36 @@ struct iommu_test_cmd { · 79 unmodified lines | |||
| 145 | __u32 id; | 154 | __u32 id; |
| 146 | __u32 cache; | 155 | __u32 cache; |
| 147 | } check_dev_cache; | 156 | } check_dev_cache; |
| 157 | struct { | ||
| 158 | __u32 dev_id; | ||
| 159 | } trigger_vevent; | ||
| 160 | struct { | ||
| 161 | __u32 pasid; | ||
| 162 | __u32 pt_id; | ||
| 163 | /* @id is stdev_id */ | ||
| 164 | } pasid_attach; | ||
| 165 | struct { | ||
| 166 | __u32 pasid; | ||
| 167 | __u32 pt_id; | ||
| 168 | /* @id is stdev_id */ | ||
| 169 | } pasid_replace; | ||
| 170 | struct { | ||
| 171 | __u32 pasid; | ||
| 172 | /* @id is stdev_id */ | ||
| 173 | } pasid_detach; | ||
| 174 | struct { | ||
| 175 | __u32 pasid; | ||
| 176 | __u32 hwpt_id; | ||
| 177 | /* @id is stdev_id */ | ||
| 178 | } pasid_check; | ||
| 148 | }; | 179 | }; |
| 149 | __u32 last; | 180 | __u32 last; |
| 150 | }; | 181 | }; |
| 151 | #define IOMMU_TEST_CMD _IO(IOMMUFD_TYPE, IOMMUFD_CMD_BASE + 32) | 182 | #define IOMMU_TEST_CMD _IO(IOMMUFD_TYPE, IOMMUFD_CMD_BASE + 32) |
| 152 | 183 | ||
| 184 | /* Mock device/iommu PASID width */ | ||
| 185 | #define MOCK_PASID_WIDTH 20 | ||
| 186 | |||
| 153 | /* Mock structs for IOMMU_DEVICE_GET_HW_INFO ioctl */ | 187 | /* Mock structs for IOMMU_DEVICE_GET_HW_INFO ioctl */ |
| 154 | #define IOMMU_HW_INFO_TYPE_SELFTEST 0xfeedbeef | 188 | #define IOMMU_HW_INFO_TYPE_SELFTEST 0xfeedbeef |
| 155 | #define IOMMU_HW_INFO_SELFTEST_REGVAL 0xdeadbeef | 189 | #define IOMMU_HW_INFO_SELFTEST_REGVAL 0xdeadbeef |
| @@ -212,4 +246,10 @@ struct iommu_viommu_invalidate_selftest { · 56 unmodified lines | |||
| 212 | __u32 cache_id; | 246 | __u32 cache_id; |
| 213 | }; | 247 | }; |
| 214 | 248 | ||
| 249 | #define IOMMU_VEVENTQ_TYPE_SELFTEST 0xbeefbeef | ||
| 250 | |||
| 251 | struct iommu_viommu_event_selftest { | ||
| 252 | __u32 virt_id; | ||
| 253 | }; | ||
| 254 | |||
| 215 | #endif | 255 | #endif |
| @@ -317,6 +317,7 @@ union ucmd_buffer { · 316 unmodified lines | |||
| 317 | struct iommu_ioas_unmap unmap; | 317 | struct iommu_ioas_unmap unmap; |
| 318 | struct iommu_option option; | 318 | struct iommu_option option; |
| 319 | struct iommu_vdevice_alloc vdev; | 319 | struct iommu_vdevice_alloc vdev; |
| 320 | struct iommu_veventq_alloc veventq; | ||
| 320 | struct iommu_vfio_ioas vfio_ioas; | 321 | struct iommu_vfio_ioas vfio_ioas; |
| 321 | struct iommu_viommu_alloc viommu; | 322 | struct iommu_viommu_alloc viommu; |
| 322 | #ifdef CONFIG_IOMMUFD_TEST | 323 | #ifdef CONFIG_IOMMUFD_TEST |
| @@ -372,6 +373,8 @@ static const struct iommufd_ioctl_op iommufd_ioctl_ops[] = { · 49 unmodified lines | |||
| 372 | IOCTL_OP(IOMMU_OPTION, iommufd_option, struct iommu_option, val64), | 373 | IOCTL_OP(IOMMU_OPTION, iommufd_option, struct iommu_option, val64), |
| 373 | IOCTL_OP(IOMMU_VDEVICE_ALLOC, iommufd_vdevice_alloc_ioctl, | 374 | IOCTL_OP(IOMMU_VDEVICE_ALLOC, iommufd_vdevice_alloc_ioctl, |
| 374 | struct iommu_vdevice_alloc, virt_id), | 375 | struct iommu_vdevice_alloc, virt_id), |
| 376 | IOCTL_OP(IOMMU_VEVENTQ_ALLOC, iommufd_veventq_alloc, | ||
| 377 | struct iommu_veventq_alloc, out_veventq_fd), | ||
| 375 | IOCTL_OP(IOMMU_VFIO_IOAS, iommufd_vfio_ioas, struct iommu_vfio_ioas, | 378 | IOCTL_OP(IOMMU_VFIO_IOAS, iommufd_vfio_ioas, struct iommu_vfio_ioas, |
| 376 | __reserved), | 379 | __reserved), |
| 377 | IOCTL_OP(IOMMU_VIOMMU_ALLOC, iommufd_viommu_alloc_ioctl, | 380 | IOCTL_OP(IOMMU_VIOMMU_ALLOC, iommufd_viommu_alloc_ioctl, |
| @@ -514,6 +517,10 @@ static const struct iommufd_object_ops iommufd_object_ops[] = { · 136 unmodified lines | |||
| 514 | [IOMMUFD_OBJ_VDEVICE] = { | 517 | [IOMMUFD_OBJ_VDEVICE] = { |
| 515 | .destroy = iommufd_vdevice_destroy, | 518 | .destroy = iommufd_vdevice_destroy, |
| 516 | }, | 519 | }, |
| 520 | [IOMMUFD_OBJ_VEVENTQ] = { | ||
| 521 | .destroy = iommufd_veventq_destroy, | ||
| 522 | .abort = iommufd_veventq_abort, | ||
| 523 | }, | ||
| 517 | [IOMMUFD_OBJ_VIOMMU] = { | 524 | [IOMMUFD_OBJ_VIOMMU] = { |
| 518 | .destroy = iommufd_viommu_destroy, | 525 | .destroy = iommufd_viommu_destroy, |
| 519 | }, | 526 | }, |
| @@ -161,9 +161,13 @@ enum selftest_obj_type { · 160 unmodified lines | |||
| 161 | 161 | ||
| 162 | struct mock_dev { | 162 | struct mock_dev { |
| 163 | struct device dev; | 163 | struct device dev; |
| 164 | struct mock_viommu *viommu; | ||
| 165 | struct rw_semaphore viommu_rwsem; | ||
| 164 | unsigned long flags; | 166 | unsigned long flags; |
| 167 | unsigned long vdev_id; | ||
| 165 | int id; | 168 | int id; |
| 166 | u32 cache[MOCK_DEV_CACHE_NUM]; | 169 | u32 cache[MOCK_DEV_CACHE_NUM]; |
| 170 | atomic_t pasid_1024_fake_error; | ||
| 167 | }; | 171 | }; |
| 168 | 172 | ||
| 169 | static inline struct mock_dev *to_mock_dev(struct device *dev) | 173 | static inline struct mock_dev *to_mock_dev(struct device *dev) |
| @@ -193,15 +197,71 @@ static int mock_domain_nop_attach(struct iommu_domain *domain, · 23 unmodified lines | |||
| 193 | struct device *dev) | 197 | struct device *dev) |
| 194 | { | 198 | { |
| 195 | struct mock_dev *mdev = to_mock_dev(dev); | 199 | struct mock_dev *mdev = to_mock_dev(dev); |
| 200 | struct mock_viommu *new_viommu = NULL; | ||
| 201 | unsigned long vdev_id = 0; | ||
| 202 | int rc; | ||
| 196 | 203 | ||
| 197 | if (domain->dirty_ops && (mdev->flags & MOCK_FLAGS_DEVICE_NO_DIRTY)) | 204 | if (domain->dirty_ops && (mdev->flags & MOCK_FLAGS_DEVICE_NO_DIRTY)) |
| 198 | return -EINVAL; | 205 | return -EINVAL; |
| 199 | 206 | ||
| 207 | iommu_group_mutex_assert(dev); | ||
| 208 | if (domain->type == IOMMU_DOMAIN_NESTED) { | ||
| 209 | new_viommu = to_mock_nested(domain)->mock_viommu; | ||
| 210 | if (new_viommu) { | ||
| 211 | rc = iommufd_viommu_get_vdev_id(&new_viommu->core, dev, | ||
| 212 | &vdev_id); | ||
| 213 | if (rc) | ||
| 214 | return rc; | ||
| 215 | } | ||
| 216 | } | ||
| 217 | if (new_viommu != mdev->viommu) { | ||
| 218 | down_write(&mdev->viommu_rwsem); | ||
| 219 | mdev->viommu = new_viommu; | ||
| 220 | mdev->vdev_id = vdev_id; | ||
| 221 | up_write(&mdev->viommu_rwsem); | ||
| 222 | } | ||
| 223 | |||
| 224 | return 0; | ||
| 225 | } | ||
| 226 | |||
| 227 | static int mock_domain_set_dev_pasid_nop(struct iommu_domain *domain, | ||
| 228 | struct device *dev, ioasid_t pasid, | ||
| 229 | struct iommu_domain *old) | ||
| 230 | { | ||
| 231 | struct mock_dev *mdev = to_mock_dev(dev); | ||
| 232 | |||
| 233 | /* | ||
| 234 | * Per the first attach with pasid 1024, set the | ||
| 235 | * mdev->pasid_1024_fake_error. Hence the second call of this op | ||
| 236 | * can fake an error to validate the error path of the core. This | ||
| 237 | * is helpful to test the case in which the iommu core needs to | ||
| 238 | * rollback to the old domain due to driver failure. e.g. replace. | ||
| 239 | * User should be careful about the third call of this op, it shall | ||
| 240 | * succeed since the mdev->pasid_1024_fake_error is cleared in the | ||
| 241 | * second call. | ||
| 242 | */ | ||
| 243 | if (pasid == 1024) { | ||
| 244 | if (domain->type == IOMMU_DOMAIN_BLOCKED) { | ||
| 245 | atomic_set(&mdev->pasid_1024_fake_error, 0); | ||
| 246 | } else if (atomic_read(&mdev->pasid_1024_fake_error)) { | ||
| 247 | /* | ||
| 248 | * Clear the flag, and fake an error to fail the | ||
| 249 | * replacement. | ||
| 250 | */ | ||
| 251 | atomic_set(&mdev->pasid_1024_fake_error, 0); | ||
| 252 | return -ENOMEM; | ||
| 253 | } else { | ||
| 254 | /* Set the flag to fake an error in next call */ | ||
| 255 | atomic_set(&mdev->pasid_1024_fake_error, 1); | ||
| 256 | } | ||
| 257 | } | ||
| 258 | |||
| 200 | return 0; | 259 | return 0; |
| 201 | } | 260 | } |
| 202 | 261 | ||
| 203 | static const struct iommu_domain_ops mock_blocking_ops = { | 262 | static const struct iommu_domain_ops mock_blocking_ops = { |
| 204 | .attach_dev = mock_domain_nop_attach, | 263 | .attach_dev = mock_domain_nop_attach, |
| 264 | .set_dev_pasid = mock_domain_set_dev_pasid_nop | ||
| 205 | }; | 265 | }; |
| 206 | 266 | ||
| 207 | static struct iommu_domain mock_blocking_domain = { | 267 | static struct iommu_domain mock_blocking_domain = { |
| @@ -343,7 +403,7 @@ mock_domain_alloc_nested(struct device *dev, struct iommu_domain *parent, · 135 unmodified lines | |||
| 343 | struct mock_iommu_domain_nested *mock_nested; | 403 | struct mock_iommu_domain_nested *mock_nested; |
| 344 | struct mock_iommu_domain *mock_parent; | 404 | struct mock_iommu_domain *mock_parent; |
| 345 | 405 | ||
| 346 | if (flags) | 406 | if (flags & ~IOMMU_HWPT_ALLOC_PASID) |
| 347 | return ERR_PTR(-EOPNOTSUPP); | 407 | return ERR_PTR(-EOPNOTSUPP); |
| 348 | if (!parent || parent->ops != mock_ops.default_domain_ops) | 408 | if (!parent || parent->ops != mock_ops.default_domain_ops) |
| 349 | return ERR_PTR(-EINVAL); | 409 | return ERR_PTR(-EINVAL); |
| @@ -365,7 +425,8 @@ mock_domain_alloc_paging_flags(struct device *dev, u32 flags, · 15 unmodified lines | |||
| 365 | { | 425 | { |
| 366 | bool has_dirty_flag = flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING; | 426 | bool has_dirty_flag = flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING; |
| 367 | const u32 PAGING_FLAGS = IOMMU_HWPT_ALLOC_DIRTY_TRACKING | | 427 | const u32 PAGING_FLAGS = IOMMU_HWPT_ALLOC_DIRTY_TRACKING | |
| 368 | IOMMU_HWPT_ALLOC_NEST_PARENT; | 428 | IOMMU_HWPT_ALLOC_NEST_PARENT | |
| 429 | IOMMU_HWPT_ALLOC_PASID; | ||
| 369 | struct mock_dev *mdev = to_mock_dev(dev); | 430 | struct mock_dev *mdev = to_mock_dev(dev); |
| 370 | bool no_dirty_ops = mdev->flags & MOCK_FLAGS_DEVICE_NO_DIRTY; | 431 | bool no_dirty_ops = mdev->flags & MOCK_FLAGS_DEVICE_NO_DIRTY; |
| 371 | struct mock_iommu_domain *mock; | 432 | struct mock_iommu_domain *mock; |
| @@ -585,7 +646,7 @@ mock_viommu_alloc_domain_nested(struct iommufd_viommu *viommu, u32 flags, · 213 unmodified lines | |||
| 585 | struct mock_viommu *mock_viommu = to_mock_viommu(viommu); | 646 | struct mock_viommu *mock_viommu = to_mock_viommu(viommu); |
| 586 | struct mock_iommu_domain_nested *mock_nested; | 647 | struct mock_iommu_domain_nested *mock_nested; |
| 587 | 648 | ||
| 588 | if (flags) | 649 | if (flags & ~IOMMU_HWPT_ALLOC_PASID) |
| 589 | return ERR_PTR(-EOPNOTSUPP); | 650 | return ERR_PTR(-EOPNOTSUPP); |
| 590 | 651 | ||
| 591 | mock_nested = __mock_domain_alloc_nested(user_data); | 652 | mock_nested = __mock_domain_alloc_nested(user_data); |
| @@ -720,6 +781,7 @@ static const struct iommu_ops mock_ops = { · 128 unmodified lines | |||
| 720 | .map_pages = mock_domain_map_pages, | 781 | .map_pages = mock_domain_map_pages, |
| 721 | .unmap_pages = mock_domain_unmap_pages, | 782 | .unmap_pages = mock_domain_unmap_pages, |
| 722 | .iova_to_phys = mock_domain_iova_to_phys, | 783 | .iova_to_phys = mock_domain_iova_to_phys, |
| 784 | .set_dev_pasid = mock_domain_set_dev_pasid_nop, | ||
| 723 | }, | 785 | }, |
| 724 | }; | 786 | }; |
| 725 | 787 | ||
| @@ -780,6 +842,7 @@ static struct iommu_domain_ops domain_nested_ops = { · 54 unmodified lines | |||
| 780 | .free = mock_domain_free_nested, | 842 | .free = mock_domain_free_nested, |
| 781 | .attach_dev = mock_domain_nop_attach, | 843 | .attach_dev = mock_domain_nop_attach, |
| 782 | .cache_invalidate_user = mock_domain_cache_invalidate_user, | 844 | .cache_invalidate_user = mock_domain_cache_invalidate_user, |
| 845 | .set_dev_pasid = mock_domain_set_dev_pasid_nop, | ||
| 783 | }; | 846 | }; |
| 784 | 847 | ||
| 785 | static inline struct iommufd_hw_pagetable * | 848 | static inline struct iommufd_hw_pagetable * |
| @@ -839,17 +902,24 @@ static void mock_dev_release(struct device *dev) · 53 unmodified lines | |||
| 839 | 902 | ||
| 840 | static struct mock_dev *mock_dev_create(unsigned long dev_flags) | 903 | static struct mock_dev *mock_dev_create(unsigned long dev_flags) |
| 841 | { | 904 | { |
| 905 | struct property_entry prop[] = { | ||
| 906 | PROPERTY_ENTRY_U32("pasid-num-bits", 0), | ||
| 907 | {}, | ||
| 908 | }; | ||
| 909 | const u32 valid_flags = MOCK_FLAGS_DEVICE_NO_DIRTY | | ||
| 910 | MOCK_FLAGS_DEVICE_HUGE_IOVA | | ||
| 911 | MOCK_FLAGS_DEVICE_PASID; | ||
| 842 | struct mock_dev *mdev; | 912 | struct mock_dev *mdev; |
| 843 | int rc, i; | 913 | int rc, i; |
| 844 | 914 | ||
| 845 | if (dev_flags & | 915 | if (dev_flags & ~valid_flags) |
| 846 | ~(MOCK_FLAGS_DEVICE_NO_DIRTY | MOCK_FLAGS_DEVICE_HUGE_IOVA)) | ||
| 847 | return ERR_PTR(-EINVAL); | 916 | return ERR_PTR(-EINVAL); |
| 848 | 917 | ||
| 849 | mdev = kzalloc(sizeof(*mdev), GFP_KERNEL); | 918 | mdev = kzalloc(sizeof(*mdev), GFP_KERNEL); |
| 850 | if (!mdev) | 919 | if (!mdev) |
| 851 | return ERR_PTR(-ENOMEM); | 920 | return ERR_PTR(-ENOMEM); |
| 852 | 921 | ||
| 922 | init_rwsem(&mdev->viommu_rwsem); | ||
| 853 | device_initialize(&mdev->dev); | 923 | device_initialize(&mdev->dev); |
| 854 | mdev->flags = dev_flags; | 924 | mdev->flags = dev_flags; |
| 855 | mdev->dev.release = mock_dev_release; | 925 | mdev->dev.release = mock_dev_release; |
| @@ -866,6 +936,15 @@ static struct mock_dev *mock_dev_create(unsigned long dev_flags) · 10 unmodified lines | |||
| 866 | if (rc) | 936 | if (rc) |
| 867 | goto err_put; | 937 | goto err_put; |
| 868 | 938 | ||
| 939 | if (dev_flags & MOCK_FLAGS_DEVICE_PASID) | ||
| 940 | prop[0] = PROPERTY_ENTRY_U32("pasid-num-bits", MOCK_PASID_WIDTH); | ||
| 941 | |||
| 942 | rc = device_create_managed_software_node(&mdev->dev, prop, NULL); | ||
| 943 | if (rc) { | ||
| 944 | dev_err(&mdev->dev, "add pasid-num-bits property failed, rc: %d", rc); | ||
| 945 | goto err_put; | ||
| 946 | } | ||
| 947 | |||
| 869 | rc = device_add(&mdev->dev); | 948 | rc = device_add(&mdev->dev); |
| 870 | if (rc) | 949 | if (rc) |
| 871 | goto err_put; | 950 | goto err_put; |
| @@ -921,7 +1000,7 @@ static int iommufd_test_mock_domain(struct iommufd_ucmd *ucmd, · 49 unmodified lines | |||
| 921 | } | 1000 | } |
| 922 | sobj->idev.idev = idev; | 1001 | sobj->idev.idev = idev; |
| 923 | 1002 | ||
| 924 | rc = iommufd_device_attach(idev, &pt_id); | 1003 | rc = iommufd_device_attach(idev, IOMMU_NO_PASID, &pt_id); |
| 925 | if (rc) | 1004 | if (rc) |
| 926 | goto out_unbind; | 1005 | goto out_unbind; |
| 927 | 1006 | ||
| @@ -936,7 +1015,7 @@ static int iommufd_test_mock_domain(struct iommufd_ucmd *ucmd, · 8 unmodified lines | |||
| 936 | return 0; | 1015 | return 0; |
| 937 | 1016 | ||
| 938 | out_detach: | 1017 | out_detach: |
| 939 | iommufd_device_detach(idev); | 1018 | iommufd_device_detach(idev, IOMMU_NO_PASID); |
| 940 | out_unbind: | 1019 | out_unbind: |
| 941 | iommufd_device_unbind(idev); | 1020 | iommufd_device_unbind(idev); |
| 942 | out_mdev: | 1021 | out_mdev: |
| @@ -946,39 +1025,49 @@ out_sobj: · 3 unmodified lines | |||
| 946 | return rc; | 1025 | return rc; |
| 947 | } | 1026 | } |
| 948 | 1027 | ||
| 949 | /* Replace the mock domain with a manually allocated hw_pagetable */ | 1028 | static struct selftest_obj * |
| 950 | static int iommufd_test_mock_domain_replace(struct iommufd_ucmd *ucmd, | 1029 | iommufd_test_get_selftest_obj(struct iommufd_ctx *ictx, u32 id) |
| 951 | unsigned int device_id, u32 pt_id, | ||
| 952 | struct iommu_test_cmd *cmd) | ||
| 953 | { | 1030 | { |
| 954 | struct iommufd_object *dev_obj; | 1031 | struct iommufd_object *dev_obj; |
| 955 | struct selftest_obj *sobj; | 1032 | struct selftest_obj *sobj; |
| 956 | int rc; | ||
| 957 | 1033 | ||
| 958 | /* | 1034 | /* |
| 959 | * Prefer to use the OBJ_SELFTEST because the destroy_rwsem will ensure | 1035 | * Prefer to use the OBJ_SELFTEST because the destroy_rwsem will ensure |
| 960 | * it doesn't race with detach, which is not allowed. | 1036 | * it doesn't race with detach, which is not allowed. |
| 961 | */ | 1037 | */ |
| 962 | dev_obj = | 1038 | dev_obj = iommufd_get_object(ictx, id, IOMMUFD_OBJ_SELFTEST); |
| 963 | iommufd_get_object(ucmd->ictx, device_id, IOMMUFD_OBJ_SELFTEST); | ||
| 964 | if (IS_ERR(dev_obj)) | 1039 | if (IS_ERR(dev_obj)) |
| 965 | return PTR_ERR(dev_obj); | 1040 | return ERR_CAST(dev_obj); |
| 966 | 1041 | ||
| 967 | sobj = to_selftest_obj(dev_obj); | 1042 | sobj = to_selftest_obj(dev_obj); |
| 968 | if (sobj->type != TYPE_IDEV) { | 1043 | if (sobj->type != TYPE_IDEV) { |
| 969 | rc = -EINVAL; | 1044 | iommufd_put_object(ictx, dev_obj); |
| 970 | goto out_dev_obj; | 1045 | return ERR_PTR(-EINVAL); |
| 971 | } | 1046 | } |
| 1047 | return sobj; | ||
| 1048 | } | ||
| 972 | 1049 | ||
| 973 | rc = iommufd_device_replace(sobj->idev.idev, &pt_id); | 1050 | /* Replace the mock domain with a manually allocated hw_pagetable */ |
| 1051 | static int iommufd_test_mock_domain_replace(struct iommufd_ucmd *ucmd, | ||
| 1052 | unsigned int device_id, u32 pt_id, | ||
| 1053 | struct iommu_test_cmd *cmd) | ||
| 1054 | { | ||
| 1055 | struct selftest_obj *sobj; | ||
| 1056 | int rc; | ||
| 1057 | |||
| 1058 | sobj = iommufd_test_get_selftest_obj(ucmd->ictx, device_id); | ||
| 1059 | if (IS_ERR(sobj)) | ||
| 1060 | return PTR_ERR(sobj); | ||
| 1061 | |||
| 1062 | rc = iommufd_device_replace(sobj->idev.idev, IOMMU_NO_PASID, &pt_id); | ||
| 974 | if (rc) | 1063 | if (rc) |
| 975 | goto out_dev_obj; | 1064 | goto out_sobj; |
| 976 | 1065 | ||
| 977 | cmd->mock_domain_replace.pt_id = pt_id; | 1066 | cmd->mock_domain_replace.pt_id = pt_id; |
| 978 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); | 1067 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); |
| 979 | 1068 | ||
| 980 | out_dev_obj: | 1069 | out_sobj: |
| 981 | iommufd_put_object(ucmd->ictx, dev_obj); | 1070 | iommufd_put_object(ucmd->ictx, &sobj->obj); |
| 982 | return rc; | 1071 | return rc; |
| 983 | } | 1072 | } |
| 984 | 1073 | ||
| @@ -1597,13 +1686,166 @@ static int iommufd_test_trigger_iopf(struct iommufd_ucmd *ucmd, · 612 unmodified lines | |||
| 1597 | return 0; | 1686 | return 0; |
| 1598 | } | 1687 | } |
| 1599 | 1688 | ||
| 1689 | static int iommufd_test_trigger_vevent(struct iommufd_ucmd *ucmd, | ||
| 1690 | struct iommu_test_cmd *cmd) | ||
| 1691 | { | ||
| 1692 | struct iommu_viommu_event_selftest test = {}; | ||
| 1693 | struct iommufd_device *idev; | ||
| 1694 | struct mock_dev *mdev; | ||
| 1695 | int rc = -ENOENT; | ||
| 1696 | |||
| 1697 | idev = iommufd_get_device(ucmd, cmd->trigger_vevent.dev_id); | ||
| 1698 | if (IS_ERR(idev)) | ||
| 1699 | return PTR_ERR(idev); | ||
| 1700 | mdev = to_mock_dev(idev->dev); | ||
| 1701 | |||
| 1702 | down_read(&mdev->viommu_rwsem); | ||
| 1703 | if (!mdev->viommu || !mdev->vdev_id) | ||
| 1704 | goto out_unlock; | ||
| 1705 | |||
| 1706 | test.virt_id = mdev->vdev_id; | ||
| 1707 | rc = iommufd_viommu_report_event(&mdev->viommu->core, | ||
| 1708 | IOMMU_VEVENTQ_TYPE_SELFTEST, &test, | ||
| 1709 | sizeof(test)); | ||
| 1710 | out_unlock: | ||
| 1711 | up_read(&mdev->viommu_rwsem); | ||
| 1712 | iommufd_put_object(ucmd->ictx, &idev->obj); | ||
| 1713 | |||
| 1714 | return rc; | ||
| 1715 | } | ||
| 1716 | |||
| 1717 | static inline struct iommufd_hw_pagetable * | ||
| 1718 | iommufd_get_hwpt(struct iommufd_ucmd *ucmd, u32 id) | ||
| 1719 | { | ||
| 1720 | struct iommufd_object *pt_obj; | ||
| 1721 | |||
| 1722 | pt_obj = iommufd_get_object(ucmd->ictx, id, IOMMUFD_OBJ_ANY); | ||
| 1723 | if (IS_ERR(pt_obj)) | ||
| 1724 | return ERR_CAST(pt_obj); | ||
| 1725 | |||
| 1726 | if (pt_obj->type != IOMMUFD_OBJ_HWPT_NESTED && | ||
| 1727 | pt_obj->type != IOMMUFD_OBJ_HWPT_PAGING) { | ||
| 1728 | iommufd_put_object(ucmd->ictx, pt_obj); | ||
| 1729 | return ERR_PTR(-EINVAL); | ||
| 1730 | } | ||
| 1731 | |||
| 1732 | return container_of(pt_obj, struct iommufd_hw_pagetable, obj); | ||
| 1733 | } | ||
| 1734 | |||
| 1735 | static int iommufd_test_pasid_check_hwpt(struct iommufd_ucmd *ucmd, | ||
| 1736 | struct iommu_test_cmd *cmd) | ||
| 1737 | { | ||
| 1738 | u32 hwpt_id = cmd->pasid_check.hwpt_id; | ||
| 1739 | struct iommu_domain *attached_domain; | ||
| 1740 | struct iommu_attach_handle *handle; | ||
| 1741 | struct iommufd_hw_pagetable *hwpt; | ||
| 1742 | struct selftest_obj *sobj; | ||
| 1743 | struct mock_dev *mdev; | ||
| 1744 | int rc = 0; | ||
| 1745 | |||
| 1746 | sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id); | ||
| 1747 | if (IS_ERR(sobj)) | ||
| 1748 | return PTR_ERR(sobj); | ||
| 1749 | |||
| 1750 | mdev = sobj->idev.mock_dev; | ||
| 1751 | |||
| 1752 | handle = iommu_attach_handle_get(mdev->dev.iommu_group, | ||
| 1753 | cmd->pasid_check.pasid, 0); | ||
| 1754 | if (IS_ERR(handle)) | ||
| 1755 | attached_domain = NULL; | ||
| 1756 | else | ||
| 1757 | attached_domain = handle->domain; | ||
| 1758 | |||
| 1759 | /* hwpt_id == 0 means to check if pasid is detached */ | ||
| 1760 | if (!hwpt_id) { | ||
| 1761 | if (attached_domain) | ||
| 1762 | rc = -EINVAL; | ||
| 1763 | goto out_sobj; | ||
| 1764 | } | ||
| 1765 | |||
| 1766 | hwpt = iommufd_get_hwpt(ucmd, hwpt_id); | ||
| 1767 | if (IS_ERR(hwpt)) { | ||
| 1768 | rc = PTR_ERR(hwpt); | ||
| 1769 | goto out_sobj; | ||
| 1770 | } | ||
| 1771 | |||
| 1772 | if (attached_domain != hwpt->domain) | ||
| 1773 | rc = -EINVAL; | ||
| 1774 | |||
| 1775 | iommufd_put_object(ucmd->ictx, &hwpt->obj); | ||
| 1776 | out_sobj: | ||
| 1777 | iommufd_put_object(ucmd->ictx, &sobj->obj); | ||
| 1778 | return rc; | ||
| 1779 | } | ||
| 1780 | |||
| 1781 | static int iommufd_test_pasid_attach(struct iommufd_ucmd *ucmd, | ||
| 1782 | struct iommu_test_cmd *cmd) | ||
| 1783 | { | ||
| 1784 | struct selftest_obj *sobj; | ||
| 1785 | int rc; | ||
| 1786 | |||
| 1787 | sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id); | ||
| 1788 | if (IS_ERR(sobj)) | ||
| 1789 | return PTR_ERR(sobj); | ||
| 1790 | |||
| 1791 | rc = iommufd_device_attach(sobj->idev.idev, cmd->pasid_attach.pasid, | ||
| 1792 | &cmd->pasid_attach.pt_id); | ||
| 1793 | if (rc) | ||
| 1794 | goto out_sobj; | ||
| 1795 | |||
| 1796 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); | ||
| 1797 | if (rc) | ||
| 1798 | iommufd_device_detach(sobj->idev.idev, | ||
| 1799 | cmd->pasid_attach.pasid); | ||
| 1800 | |||
| 1801 | out_sobj: | ||
| 1802 | iommufd_put_object(ucmd->ictx, &sobj->obj); | ||
| 1803 | return rc; | ||
| 1804 | } | ||
| 1805 | |||
| 1806 | static int iommufd_test_pasid_replace(struct iommufd_ucmd *ucmd, | ||
| 1807 | struct iommu_test_cmd *cmd) | ||
| 1808 | { | ||
| 1809 | struct selftest_obj *sobj; | ||
| 1810 | int rc; | ||
| 1811 | |||
| 1812 | sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id); | ||
| 1813 | if (IS_ERR(sobj)) | ||
| 1814 | return PTR_ERR(sobj); | ||
| 1815 | |||
| 1816 | rc = iommufd_device_replace(sobj->idev.idev, cmd->pasid_attach.pasid, | ||
| 1817 | &cmd->pasid_attach.pt_id); | ||
| 1818 | if (rc) | ||
| 1819 | goto out_sobj; | ||
| 1820 | |||
| 1821 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); | ||
| 1822 | |||
| 1823 | out_sobj: | ||
| 1824 | iommufd_put_object(ucmd->ictx, &sobj->obj); | ||
| 1825 | return rc; | ||
| 1826 | } | ||
| 1827 | |||
| 1828 | static int iommufd_test_pasid_detach(struct iommufd_ucmd *ucmd, | ||
| 1829 | struct iommu_test_cmd *cmd) | ||
| 1830 | { | ||
| 1831 | struct selftest_obj *sobj; | ||
| 1832 | |||
| 1833 | sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id); | ||
| 1834 | if (IS_ERR(sobj)) | ||
| 1835 | return PTR_ERR(sobj); | ||
| 1836 | |||
| 1837 | iommufd_device_detach(sobj->idev.idev, cmd->pasid_detach.pasid); | ||
| 1838 | iommufd_put_object(ucmd->ictx, &sobj->obj); | ||
| 1839 | return 0; | ||
| 1840 | } | ||
| 1841 | |||
| 1600 | void iommufd_selftest_destroy(struct iommufd_object *obj) | 1842 | void iommufd_selftest_destroy(struct iommufd_object *obj) |
| 1601 | { | 1843 | { |
| 1602 | struct selftest_obj *sobj = to_selftest_obj(obj); | 1844 | struct selftest_obj *sobj = to_selftest_obj(obj); |
| 1603 | 1845 | ||
| 1604 | switch (sobj->type) { | 1846 | switch (sobj->type) { |
| 1605 | case TYPE_IDEV: | 1847 | case TYPE_IDEV: |
| 1606 | iommufd_device_detach(sobj->idev.idev); | 1848 | iommufd_device_detach(sobj->idev.idev, IOMMU_NO_PASID); |
| 1607 | iommufd_device_unbind(sobj->idev.idev); | 1849 | iommufd_device_unbind(sobj->idev.idev); |
| 1608 | mock_dev_destroy(sobj->idev.mock_dev); | 1850 | mock_dev_destroy(sobj->idev.mock_dev); |
| 1609 | break; | 1851 | break; |
| @@ -1678,6 +1920,16 @@ int iommufd_test(struct iommufd_ucmd *ucmd) · 68 unmodified lines | |||
| 1678 | cmd->dirty.flags); | 1920 | cmd->dirty.flags); |
| 1679 | case IOMMU_TEST_OP_TRIGGER_IOPF: | 1921 | case IOMMU_TEST_OP_TRIGGER_IOPF: |
| 1680 | return iommufd_test_trigger_iopf(ucmd, cmd); | 1922 | return iommufd_test_trigger_iopf(ucmd, cmd); |
| 1923 | case IOMMU_TEST_OP_TRIGGER_VEVENT: | ||
| 1924 | return iommufd_test_trigger_vevent(ucmd, cmd); | ||
| 1925 | case IOMMU_TEST_OP_PASID_ATTACH: | ||
| 1926 | return iommufd_test_pasid_attach(ucmd, cmd); | ||
| 1927 | case IOMMU_TEST_OP_PASID_REPLACE: | ||
| 1928 | return iommufd_test_pasid_replace(ucmd, cmd); | ||
| 1929 | case IOMMU_TEST_OP_PASID_DETACH: | ||
| 1930 | return iommufd_test_pasid_detach(ucmd, cmd); | ||
| 1931 | case IOMMU_TEST_OP_PASID_CHECK_HWPT: | ||
| 1932 | return iommufd_test_pasid_check_hwpt(ucmd, cmd); | ||
| 1681 | default: | 1933 | default: |
| 1682 | return -EOPNOTSUPP; | 1934 | return -EOPNOTSUPP; |
| 1683 | } | 1935 | } |
| @@ -1724,6 +1976,7 @@ int __init iommufd_test_init(void) · 40 unmodified lines | |||
| 1724 | init_completion(&mock_iommu.complete); | 1976 | init_completion(&mock_iommu.complete); |
| 1725 | 1977 | ||
| 1726 | mock_iommu_iopf_queue = iopf_queue_alloc("mock-iopfq"); | 1978 | mock_iommu_iopf_queue = iopf_queue_alloc("mock-iopfq"); |
| 1979 | mock_iommu.iommu_dev.max_pasids = (1 << MOCK_PASID_WIDTH); | ||
| 1727 | 1980 | ||
| 1728 | return 0; | 1981 | return 0; |
| 1729 | 1982 | ||
| @@ -59,6 +59,8 @@ int iommufd_viommu_alloc_ioctl(struct iommufd_ucmd *ucmd) · 58 unmodified lines | |||
| 59 | viommu->ictx = ucmd->ictx; | 59 | viommu->ictx = ucmd->ictx; |
| 60 | viommu->hwpt = hwpt_paging; | 60 | viommu->hwpt = hwpt_paging; |
| 61 | refcount_inc(&viommu->hwpt->common.obj.users); | 61 | refcount_inc(&viommu->hwpt->common.obj.users); |
| 62 | INIT_LIST_HEAD(&viommu->veventqs); | ||
| 63 | init_rwsem(&viommu->veventqs_rwsem); | ||
| 62 | /* | 64 | /* |
| 63 | * It is the most likely case that a physical IOMMU is unpluggable. A | 65 | * It is the most likely case that a physical IOMMU is unpluggable. A |
| 64 | * pluggable IOMMU instance (if exists) is responsible for refcounting | 66 | * pluggable IOMMU instance (if exists) is responsible for refcounting |
| @@ -538,4 +538,37 @@ int pci_max_pasids(struct pci_dev *pdev) · 537 unmodified lines | |||
| 538 | return (1 << FIELD_GET(PCI_PASID_CAP_WIDTH, supported)); | 538 | return (1 << FIELD_GET(PCI_PASID_CAP_WIDTH, supported)); |
| 539 | } | 539 | } |
| 540 | EXPORT_SYMBOL_GPL(pci_max_pasids); | 540 | EXPORT_SYMBOL_GPL(pci_max_pasids); |
| 541 | |||
| 542 | /** | ||
| 543 | * pci_pasid_status - Check the PASID status | ||
| 544 | * @pdev: PCI device structure | ||
| 545 | * | ||
| 546 | * Returns a negative value when no PASID capability is present. | ||
| 547 | * Otherwise the value of the control register is returned. | ||
| 548 | * Status reported are: | ||
| 549 | * | ||
| 550 | * PCI_PASID_CTRL_ENABLE - PASID enabled | ||
| 551 | * PCI_PASID_CTRL_EXEC - Execute permission enabled | ||
| 552 | * PCI_PASID_CTRL_PRIV - Privileged mode enabled | ||
| 553 | */ | ||
| 554 | int pci_pasid_status(struct pci_dev *pdev) | ||
| 555 | { | ||
| 556 | int pasid; | ||
| 557 | u16 ctrl; | ||
| 558 | |||
| 559 | if (pdev->is_virtfn) | ||
| 560 | pdev = pci_physfn(pdev); | ||
| 561 | |||
| 562 | pasid = pdev->pasid_cap; | ||
| 563 | if (!pasid) | ||
| 564 | return -EINVAL; | ||
| 565 | |||
| 566 | pci_read_config_word(pdev, pasid + PCI_PASID_CTRL, &ctrl); | ||
| 567 | |||
| 568 | ctrl &= PCI_PASID_CTRL_ENABLE | PCI_PASID_CTRL_EXEC | | ||
| 569 | PCI_PASID_CTRL_PRIV; | ||
| 570 | |||
| 571 | return ctrl; | ||
| 572 | } | ||
| 573 | EXPORT_SYMBOL_GPL(pci_pasid_status); | ||
| 541 | #endif /* CONFIG_PCI_PASID */ | 574 | #endif /* CONFIG_PCI_PASID */ |
| @@ -162,9 +162,9 @@ void vfio_df_unbind_iommufd(struct vfio_device_file *df) · 161 unmodified lines | |||
| 162 | int vfio_df_ioctl_attach_pt(struct vfio_device_file *df, | 162 | int vfio_df_ioctl_attach_pt(struct vfio_device_file *df, |
| 163 | struct vfio_device_attach_iommufd_pt __user *arg) | 163 | struct vfio_device_attach_iommufd_pt __user *arg) |
| 164 | { | 164 | { |
| 165 | struct vfio_device *device = df->device; | ||
| 166 | struct vfio_device_attach_iommufd_pt attach; | 165 | struct vfio_device_attach_iommufd_pt attach; |
| 167 | unsigned long minsz; | 166 | struct vfio_device *device = df->device; |
| 167 | unsigned long minsz, xend = 0; | ||
| 168 | int ret; | 168 | int ret; |
| 169 | 169 | ||
| 170 | minsz = offsetofend(struct vfio_device_attach_iommufd_pt, pt_id); | 170 | minsz = offsetofend(struct vfio_device_attach_iommufd_pt, pt_id); |
| @@ -172,11 +172,34 @@ int vfio_df_ioctl_attach_pt(struct vfio_device_file *df, · 1 unmodified line | |||
| 172 | if (copy_from_user(&attach, arg, minsz)) | 172 | if (copy_from_user(&attach, arg, minsz)) |
| 173 | return -EFAULT; | 173 | return -EFAULT; |
| 174 | 174 | ||
| 175 | if (attach.argsz < minsz || attach.flags) | 175 | if (attach.argsz < minsz) |
| 176 | return -EINVAL; | 176 | return -EINVAL; |
| 177 | 177 | ||
| 178 | if (attach.flags & ~VFIO_DEVICE_ATTACH_PASID) | ||
| 179 | return -EINVAL; | ||
| 180 | |||
| 181 | if (attach.flags & VFIO_DEVICE_ATTACH_PASID) { | ||
| 182 | if (!device->ops->pasid_attach_ioas) | ||
| 183 | return -EOPNOTSUPP; | ||
| 184 | xend = offsetofend(struct vfio_device_attach_iommufd_pt, pasid); | ||
| 185 | } | ||
| 186 | |||
| 187 | if (xend) { | ||
| 188 | if (attach.argsz < xend) | ||
| 189 | return -EINVAL; | ||
| 190 | |||
| 191 | if (copy_from_user((void *)&attach + minsz, | ||
| 192 | (void __user *)arg + minsz, xend - minsz)) | ||
| 193 | return -EFAULT; | ||
| 194 | } | ||
| 195 | |||
| 178 | mutex_lock(&device->dev_set->lock); | 196 | mutex_lock(&device->dev_set->lock); |
| 179 | ret = device->ops->attach_ioas(device, &attach.pt_id); | 197 | if (attach.flags & VFIO_DEVICE_ATTACH_PASID) |
| 198 | ret = device->ops->pasid_attach_ioas(device, | ||
| 199 | attach.pasid, | ||
| 200 | &attach.pt_id); | ||
| 201 | else | ||
| 202 | ret = device->ops->attach_ioas(device, &attach.pt_id); | ||
| 180 | if (ret) | 203 | if (ret) |
| 181 | goto out_unlock; | 204 | goto out_unlock; |
| 182 | 205 | ||
| @@ -198,20 +221,41 @@ out_unlock: · 15 unmodified lines | |||
| 198 | int vfio_df_ioctl_detach_pt(struct vfio_device_file *df, | 221 | int vfio_df_ioctl_detach_pt(struct vfio_device_file *df, |
| 199 | struct vfio_device_detach_iommufd_pt __user *arg) | 222 | struct vfio_device_detach_iommufd_pt __user *arg) |
| 200 | { | 223 | { |
| 201 | struct vfio_device *device = df->device; | ||
| 202 | struct vfio_device_detach_iommufd_pt detach; | 224 | struct vfio_device_detach_iommufd_pt detach; |
| 203 | unsigned long minsz; | 225 | struct vfio_device *device = df->device; |
| 226 | unsigned long minsz, xend = 0; | ||
| 204 | 227 | ||
| 205 | minsz = offsetofend(struct vfio_device_detach_iommufd_pt, flags); | 228 | minsz = offsetofend(struct vfio_device_detach_iommufd_pt, flags); |
| 206 | 229 | ||
| 207 | if (copy_from_user(&detach, arg, minsz)) | 230 | if (copy_from_user(&detach, arg, minsz)) |
| 208 | return -EFAULT; | 231 | return -EFAULT; |
| 209 | 232 | ||
| 210 | if (detach.argsz < minsz || detach.flags) | 233 | if (detach.argsz < minsz) |
| 211 | return -EINVAL; | 234 | return -EINVAL; |
| 212 | 235 | ||
| 236 | if (detach.flags & ~VFIO_DEVICE_DETACH_PASID) | ||
| 237 | return -EINVAL; | ||
| 238 | |||
| 239 | if (detach.flags & VFIO_DEVICE_DETACH_PASID) { | ||
| 240 | if (!device->ops->pasid_detach_ioas) | ||
| 241 | return -EOPNOTSUPP; | ||
| 242 | xend = offsetofend(struct vfio_device_detach_iommufd_pt, pasid); | ||
| 243 | } | ||
| 244 | |||
| 245 | if (xend) { | ||
| 246 | if (detach.argsz < xend) | ||
| 247 | return -EINVAL; | ||
| 248 | |||
| 249 | if (copy_from_user((void *)&detach + minsz, | ||
| 250 | (void __user *)arg + minsz, xend - minsz)) | ||
| 251 | return -EFAULT; | ||
| 252 | } | ||
| 253 | |||
| 213 | mutex_lock(&device->dev_set->lock); | 254 | mutex_lock(&device->dev_set->lock); |
| 214 | device->ops->detach_ioas(device); | 255 | if (detach.flags & VFIO_DEVICE_DETACH_PASID) |
| 256 | device->ops->pasid_detach_ioas(device, detach.pasid); | ||
| 257 | else | ||
| 258 | device->ops->detach_ioas(device); | ||
| 215 | mutex_unlock(&device->dev_set->lock); | 259 | mutex_unlock(&device->dev_set->lock); |
| 216 | 260 | ||
| 217 | return 0; | 261 | return 0; |
| @@ -119,16 +119,24 @@ int vfio_iommufd_physical_bind(struct vfio_device *vdev, · 118 unmodified lines | |||
| 119 | if (IS_ERR(idev)) | 119 | if (IS_ERR(idev)) |
| 120 | return PTR_ERR(idev); | 120 | return PTR_ERR(idev); |
| 121 | vdev->iommufd_device = idev; | 121 | vdev->iommufd_device = idev; |
| 122 | ida_init(&vdev->pasids); | ||
| 122 | return 0; | 123 | return 0; |
| 123 | } | 124 | } |
| 124 | EXPORT_SYMBOL_GPL(vfio_iommufd_physical_bind); | 125 | EXPORT_SYMBOL_GPL(vfio_iommufd_physical_bind); |
| 125 | 126 | ||
| 126 | void vfio_iommufd_physical_unbind(struct vfio_device *vdev) | 127 | void vfio_iommufd_physical_unbind(struct vfio_device *vdev) |
| 127 | { | 128 | { |
| 129 | int pasid; | ||
| 130 | |||
| 128 | lockdep_assert_held(&vdev->dev_set->lock); | 131 | lockdep_assert_held(&vdev->dev_set->lock); |
| 129 | 132 | ||
| 133 | while ((pasid = ida_find_first(&vdev->pasids)) >= 0) { | ||
| 134 | iommufd_device_detach(vdev->iommufd_device, pasid); | ||
| 135 | ida_free(&vdev->pasids, pasid); | ||
| 136 | } | ||
| 137 | |||
| 130 | if (vdev->iommufd_attached) { | 138 | if (vdev->iommufd_attached) { |
| 131 | iommufd_device_detach(vdev->iommufd_device); | 139 | iommufd_device_detach(vdev->iommufd_device, IOMMU_NO_PASID); |
| 132 | vdev->iommufd_attached = false; | 140 | vdev->iommufd_attached = false; |
| 133 | } | 141 | } |
| 134 | iommufd_device_unbind(vdev->iommufd_device); | 142 | iommufd_device_unbind(vdev->iommufd_device); |
| @@ -146,9 +154,11 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id) · 11 unmodified lines | |||
| 146 | return -EINVAL; | 154 | return -EINVAL; |
| 147 | 155 | ||
| 148 | if (vdev->iommufd_attached) | 156 | if (vdev->iommufd_attached) |
| 149 | rc = iommufd_device_replace(vdev->iommufd_device, pt_id); | 157 | rc = iommufd_device_replace(vdev->iommufd_device, |
| 158 | IOMMU_NO_PASID, pt_id); | ||
| 150 | else | 159 | else |
| 151 | rc = iommufd_device_attach(vdev->iommufd_device, pt_id); | 160 | rc = iommufd_device_attach(vdev->iommufd_device, |
| 161 | IOMMU_NO_PASID, pt_id); | ||
| 152 | if (rc) | 162 | if (rc) |
| 153 | return rc; | 163 | return rc; |
| 154 | vdev->iommufd_attached = true; | 164 | vdev->iommufd_attached = true; |
| @@ -163,11 +173,53 @@ void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev) · 8 unmodified lines | |||
| 163 | if (WARN_ON(!vdev->iommufd_device) || !vdev->iommufd_attached) | 173 | if (WARN_ON(!vdev->iommufd_device) || !vdev->iommufd_attached) |
| 164 | return; | 174 | return; |
| 165 | 175 | ||
| 166 | iommufd_device_detach(vdev->iommufd_device); | 176 | iommufd_device_detach(vdev->iommufd_device, IOMMU_NO_PASID); |
| 167 | vdev->iommufd_attached = false; | 177 | vdev->iommufd_attached = false; |
| 168 | } | 178 | } |
| 169 | EXPORT_SYMBOL_GPL(vfio_iommufd_physical_detach_ioas); | 179 | EXPORT_SYMBOL_GPL(vfio_iommufd_physical_detach_ioas); |
| 170 | 180 | ||
| 181 | int vfio_iommufd_physical_pasid_attach_ioas(struct vfio_device *vdev, | ||
| 182 | u32 pasid, u32 *pt_id) | ||
| 183 | { | ||
| 184 | int rc; | ||
| 185 | |||
| 186 | lockdep_assert_held(&vdev->dev_set->lock); | ||
| 187 | |||
| 188 | if (WARN_ON(!vdev->iommufd_device)) | ||
| 189 | return -EINVAL; | ||
| 190 | |||
| 191 | if (ida_exists(&vdev->pasids, pasid)) | ||
| 192 | return iommufd_device_replace(vdev->iommufd_device, | ||
| 193 | pasid, pt_id); | ||
| 194 | |||
| 195 | rc = ida_alloc_range(&vdev->pasids, pasid, pasid, GFP_KERNEL); | ||
| 196 | if (rc < 0) | ||
| 197 | return rc; | ||
| 198 | |||
| 199 | rc = iommufd_device_attach(vdev->iommufd_device, pasid, pt_id); | ||
| 200 | if (rc) | ||
| 201 | ida_free(&vdev->pasids, pasid); | ||
| 202 | |||
| 203 | return rc; | ||
| 204 | } | ||
| 205 | EXPORT_SYMBOL_GPL(vfio_iommufd_physical_pasid_attach_ioas); | ||
| 206 | |||
| 207 | void vfio_iommufd_physical_pasid_detach_ioas(struct vfio_device *vdev, | ||
| 208 | u32 pasid) | ||
| 209 | { | ||
| 210 | lockdep_assert_held(&vdev->dev_set->lock); | ||
| 211 | |||
| 212 | if (WARN_ON(!vdev->iommufd_device)) | ||
| 213 | return; | ||
| 214 | |||
| 215 | if (!ida_exists(&vdev->pasids, pasid)) | ||
| 216 | return; | ||
| 217 | |||
| 218 | iommufd_device_detach(vdev->iommufd_device, pasid); | ||
| 219 | ida_free(&vdev->pasids, pasid); | ||
| 220 | } | ||
| 221 | EXPORT_SYMBOL_GPL(vfio_iommufd_physical_pasid_detach_ioas); | ||
| 222 | |||
| 171 | /* | 223 | /* |
| 172 | * The emulated standard ops mean that vfio_device is going to use the | 224 | * The emulated standard ops mean that vfio_device is going to use the |
| 173 | * "mdev path" and will call vfio_pin_pages()/vfio_dma_rw(). Drivers using this | 225 | * "mdev path" and will call vfio_pin_pages()/vfio_dma_rw(). Drivers using this |
| @@ -144,6 +144,8 @@ static const struct vfio_device_ops vfio_pci_ops = { · 143 unmodified lines | |||
| 144 | .unbind_iommufd = vfio_iommufd_physical_unbind, | 144 | .unbind_iommufd = vfio_iommufd_physical_unbind, |
| 145 | .attach_ioas = vfio_iommufd_physical_attach_ioas, | 145 | .attach_ioas = vfio_iommufd_physical_attach_ioas, |
| 146 | .detach_ioas = vfio_iommufd_physical_detach_ioas, | 146 | .detach_ioas = vfio_iommufd_physical_detach_ioas, |
| 147 | .pasid_attach_ioas = vfio_iommufd_physical_pasid_attach_ioas, | ||
| 148 | .pasid_detach_ioas = vfio_iommufd_physical_pasid_detach_ioas, | ||
| 147 | }; | 149 | }; |
| 148 | 150 | ||
| 149 | static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) | 151 | static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) |
| @@ -274,6 +274,7 @@ struct ida { · 273 unmodified lines | |||
| 274 | int ida_alloc_range(struct ida *, unsigned int min, unsigned int max, gfp_t); | 274 | int ida_alloc_range(struct ida *, unsigned int min, unsigned int max, gfp_t); |
| 275 | void ida_free(struct ida *, unsigned int id); | 275 | void ida_free(struct ida *, unsigned int id); |
| 276 | void ida_destroy(struct ida *ida); | 276 | void ida_destroy(struct ida *ida); |
| 277 | int ida_find_first_range(struct ida *ida, unsigned int min, unsigned int max); | ||
| 277 | 278 | ||
| 278 | /** | 279 | /** |
| 279 | * ida_alloc() - Allocate an unused ID. | 280 | * ida_alloc() - Allocate an unused ID. |
| @@ -345,4 +346,14 @@ static inline bool ida_is_empty(const struct ida *ida) · 65 unmodified lines | |||
| 345 | { | 346 | { |
| 346 | return xa_empty(&ida->xa); | 347 | return xa_empty(&ida->xa); |
| 347 | } | 348 | } |
| 349 | |||
| 350 | static inline bool ida_exists(struct ida *ida, unsigned int id) | ||
| 351 | { | ||
| 352 | return ida_find_first_range(ida, id, id) == id; | ||
| 353 | } | ||
| 354 | |||
| 355 | static inline int ida_find_first(struct ida *ida) | ||
| 356 | { | ||
| 357 | return ida_find_first_range(ida, 0, ~0); | ||
| 358 | } | ||
| 348 | #endif /* __IDR_H__ */ | 359 | #endif /* __IDR_H__ */ |
| @@ -41,6 +41,7 @@ struct iommu_dirty_ops; · 40 unmodified lines | |||
| 41 | struct notifier_block; | 41 | struct notifier_block; |
| 42 | struct iommu_sva; | 42 | struct iommu_sva; |
| 43 | struct iommu_dma_cookie; | 43 | struct iommu_dma_cookie; |
| 44 | struct iommu_dma_msi_cookie; | ||
| 44 | struct iommu_fault_param; | 45 | struct iommu_fault_param; |
| 45 | struct iommufd_ctx; | 46 | struct iommufd_ctx; |
| 46 | struct iommufd_viommu; | 47 | struct iommufd_viommu; |
| @@ -165,6 +166,15 @@ struct iommu_domain_geometry { · 118 unmodified lines | |||
| 165 | bool force_aperture; /* DMA only allowed in mappable range? */ | 166 | bool force_aperture; /* DMA only allowed in mappable range? */ |
| 166 | }; | 167 | }; |
| 167 | 168 | ||
| 169 | enum iommu_domain_cookie_type { | ||
| 170 | IOMMU_COOKIE_NONE, | ||
| 171 | IOMMU_COOKIE_DMA_IOVA, | ||
| 172 | IOMMU_COOKIE_DMA_MSI, | ||
| 173 | IOMMU_COOKIE_FAULT_HANDLER, | ||
| 174 | IOMMU_COOKIE_SVA, | ||
| 175 | IOMMU_COOKIE_IOMMUFD, | ||
| 176 | }; | ||
| 177 | |||
| 168 | /* Domain feature flags */ | 178 | /* Domain feature flags */ |
| 169 | #define __IOMMU_DOMAIN_PAGING (1U << 0) /* Support for iommu_map/unmap */ | 179 | #define __IOMMU_DOMAIN_PAGING (1U << 0) /* Support for iommu_map/unmap */ |
| 170 | #define __IOMMU_DOMAIN_DMA_API (1U << 1) /* Domain for use in DMA-API | 180 | #define __IOMMU_DOMAIN_DMA_API (1U << 1) /* Domain for use in DMA-API |
| @@ -211,23 +221,18 @@ struct iommu_domain_geometry { · 40 unmodified lines | |||
| 211 | 221 | ||
| 212 | struct iommu_domain { | 222 | struct iommu_domain { |
| 213 | unsigned type; | 223 | unsigned type; |
| 224 | enum iommu_domain_cookie_type cookie_type; | ||
| 214 | const struct iommu_domain_ops *ops; | 225 | const struct iommu_domain_ops *ops; |
| 215 | const struct iommu_dirty_ops *dirty_ops; | 226 | const struct iommu_dirty_ops *dirty_ops; |
| 216 | const struct iommu_ops *owner; /* Whose domain_alloc we came from */ | 227 | const struct iommu_ops *owner; /* Whose domain_alloc we came from */ |
| 217 | unsigned long pgsize_bitmap; /* Bitmap of page sizes in use */ | 228 | unsigned long pgsize_bitmap; /* Bitmap of page sizes in use */ |
| 218 | struct iommu_domain_geometry geometry; | 229 | struct iommu_domain_geometry geometry; |
| 219 | struct iommu_dma_cookie *iova_cookie; | ||
| 220 | int (*iopf_handler)(struct iopf_group *group); | 230 | int (*iopf_handler)(struct iopf_group *group); |
| 221 | 231 | ||
| 222 | #if IS_ENABLED(CONFIG_IRQ_MSI_IOMMU) | 232 | union { /* cookie */ |
| 223 | int (*sw_msi)(struct iommu_domain *domain, struct msi_desc *desc, | 233 | struct iommu_dma_cookie *iova_cookie; |
| 224 | phys_addr_t msi_addr); | 234 | struct iommu_dma_msi_cookie *msi_cookie; |
| 225 | #endif | 235 | struct iommufd_hw_pagetable *iommufd_hwpt; |
| 226 | |||
| 227 | union { /* Pointer usable by owner of the domain */ | ||
| 228 | struct iommufd_hw_pagetable *iommufd_hwpt; /* iommufd */ | ||
| 229 | }; | ||
| 230 | union { /* Fault handler */ | ||
| 231 | struct { | 236 | struct { |
| 232 | iommu_fault_handler_t handler; | 237 | iommu_fault_handler_t handler; |
| 233 | void *handler_token; | 238 | void *handler_token; |
| @@ -244,16 +249,6 @@ struct iommu_domain { · 10 unmodified lines | |||
| 244 | }; | 249 | }; |
| 245 | }; | 250 | }; |
| 246 | 251 | ||
| 247 | static inline void iommu_domain_set_sw_msi( | ||
| 248 | struct iommu_domain *domain, | ||
| 249 | int (*sw_msi)(struct iommu_domain *domain, struct msi_desc *desc, | ||
| 250 | phys_addr_t msi_addr)) | ||
| 251 | { | ||
| 252 | #if IS_ENABLED(CONFIG_IRQ_MSI_IOMMU) | ||
| 253 | domain->sw_msi = sw_msi; | ||
| 254 | #endif | ||
| 255 | } | ||
| 256 | |||
| 257 | static inline bool iommu_is_dma_domain(struct iommu_domain *domain) | 252 | static inline bool iommu_is_dma_domain(struct iommu_domain *domain) |
| 258 | { | 253 | { |
| 259 | return domain->type & __IOMMU_DOMAIN_DMA_API; | 254 | return domain->type & __IOMMU_DOMAIN_DMA_API; |
| @@ -8,9 +8,11 @@ · 7 unmodified lines | |||
| 8 | 8 | ||
| 9 | #include <linux/err.h> | 9 | #include <linux/err.h> |
| 10 | #include <linux/errno.h> | 10 | #include <linux/errno.h> |
| 11 | #include <linux/iommu.h> | ||
| 11 | #include <linux/refcount.h> | 12 | #include <linux/refcount.h> |
| 12 | #include <linux/types.h> | 13 | #include <linux/types.h> |
| 13 | #include <linux/xarray.h> | 14 | #include <linux/xarray.h> |
| 15 | #include <uapi/linux/iommufd.h> | ||
| 14 | 16 | ||
| 15 | struct device; | 17 | struct device; |
| 16 | struct file; | 18 | struct file; |
| @@ -34,6 +36,7 @@ enum iommufd_object_type { · 17 unmodified lines | |||
| 34 | IOMMUFD_OBJ_FAULT, | 36 | IOMMUFD_OBJ_FAULT, |
| 35 | IOMMUFD_OBJ_VIOMMU, | 37 | IOMMUFD_OBJ_VIOMMU, |
| 36 | IOMMUFD_OBJ_VDEVICE, | 38 | IOMMUFD_OBJ_VDEVICE, |
| 39 | IOMMUFD_OBJ_VEVENTQ, | ||
| 37 | #ifdef CONFIG_IOMMUFD_TEST | 40 | #ifdef CONFIG_IOMMUFD_TEST |
| 38 | IOMMUFD_OBJ_SELFTEST, | 41 | IOMMUFD_OBJ_SELFTEST, |
| 39 | #endif | 42 | #endif |
| @@ -52,9 +55,11 @@ struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, · 12 unmodified lines | |||
| 52 | struct device *dev, u32 *id); | 55 | struct device *dev, u32 *id); |
| 53 | void iommufd_device_unbind(struct iommufd_device *idev); | 56 | void iommufd_device_unbind(struct iommufd_device *idev); |
| 54 | 57 | ||
| 55 | int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id); | 58 | int iommufd_device_attach(struct iommufd_device *idev, ioasid_t pasid, |
| 56 | int iommufd_device_replace(struct iommufd_device *idev, u32 *pt_id); | 59 | u32 *pt_id); |
| 57 | void iommufd_device_detach(struct iommufd_device *idev); | 60 | int iommufd_device_replace(struct iommufd_device *idev, ioasid_t pasid, |
| 61 | u32 *pt_id); | ||
| 62 | void iommufd_device_detach(struct iommufd_device *idev, ioasid_t pasid); | ||
| 58 | 63 | ||
| 59 | struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev); | 64 | struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev); |
| 60 | u32 iommufd_device_to_id(struct iommufd_device *idev); | 65 | u32 iommufd_device_to_id(struct iommufd_device *idev); |
| @@ -93,6 +98,8 @@ struct iommufd_viommu { · 32 unmodified lines | |||
| 93 | const struct iommufd_viommu_ops *ops; | 98 | const struct iommufd_viommu_ops *ops; |
| 94 | 99 | ||
| 95 | struct xarray vdevs; | 100 | struct xarray vdevs; |
| 101 | struct list_head veventqs; | ||
| 102 | struct rw_semaphore veventqs_rwsem; | ||
| 96 | 103 | ||
| 97 | unsigned int type; | 104 | unsigned int type; |
| 98 | }; | 105 | }; |
| @@ -187,6 +194,11 @@ struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx, · 88 unmodified lines | |||
| 187 | enum iommufd_object_type type); | 194 | enum iommufd_object_type type); |
| 188 | struct device *iommufd_viommu_find_dev(struct iommufd_viommu *viommu, | 195 | struct device *iommufd_viommu_find_dev(struct iommufd_viommu *viommu, |
| 189 | unsigned long vdev_id); | 196 | unsigned long vdev_id); |
| 197 | int iommufd_viommu_get_vdev_id(struct iommufd_viommu *viommu, | ||
| 198 | struct device *dev, unsigned long *vdev_id); | ||
| 199 | int iommufd_viommu_report_event(struct iommufd_viommu *viommu, | ||
| 200 | enum iommu_veventq_type type, void *event_data, | ||
| 201 | size_t data_len); | ||
| 190 | #else /* !CONFIG_IOMMUFD_DRIVER_CORE */ | 202 | #else /* !CONFIG_IOMMUFD_DRIVER_CORE */ |
| 191 | static inline struct iommufd_object * | 203 | static inline struct iommufd_object * |
| 192 | _iommufd_object_alloc(struct iommufd_ctx *ictx, size_t size, | 204 | _iommufd_object_alloc(struct iommufd_ctx *ictx, size_t size, |
| @@ -200,6 +212,20 @@ iommufd_viommu_find_dev(struct iommufd_viommu *viommu, unsigned long vdev_id) · 7 unmodified lines | |||
| 200 | { | 212 | { |
| 201 | return NULL; | 213 | return NULL; |
| 202 | } | 214 | } |
| 215 | |||
| 216 | static inline int iommufd_viommu_get_vdev_id(struct iommufd_viommu *viommu, | ||
| 217 | struct device *dev, | ||
| 218 | unsigned long *vdev_id) | ||
| 219 | { | ||
| 220 | return -ENOENT; | ||
| 221 | } | ||
| 222 | |||
| 223 | static inline int iommufd_viommu_report_event(struct iommufd_viommu *viommu, | ||
| 224 | enum iommu_veventq_type type, | ||
| 225 | void *event_data, size_t data_len) | ||
| 226 | { | ||
| 227 | return -EOPNOTSUPP; | ||
| 228 | } | ||
| 203 | #endif /* CONFIG_IOMMUFD_DRIVER_CORE */ | 229 | #endif /* CONFIG_IOMMUFD_DRIVER_CORE */ |
| 204 | 230 | ||
| 205 | /* | 231 | /* |
| @@ -42,6 +42,7 @@ int pci_enable_pasid(struct pci_dev *pdev, int features); · 41 unmodified lines | |||
| 42 | void pci_disable_pasid(struct pci_dev *pdev); | 42 | void pci_disable_pasid(struct pci_dev *pdev); |
| 43 | int pci_pasid_features(struct pci_dev *pdev); | 43 | int pci_pasid_features(struct pci_dev *pdev); |
| 44 | int pci_max_pasids(struct pci_dev *pdev); | 44 | int pci_max_pasids(struct pci_dev *pdev); |
| 45 | int pci_pasid_status(struct pci_dev *pdev); | ||
| 45 | #else /* CONFIG_PCI_PASID */ | 46 | #else /* CONFIG_PCI_PASID */ |
| 46 | static inline int pci_enable_pasid(struct pci_dev *pdev, int features) | 47 | static inline int pci_enable_pasid(struct pci_dev *pdev, int features) |
| 47 | { return -EINVAL; } | 48 | { return -EINVAL; } |
| @@ -50,6 +51,8 @@ static inline int pci_pasid_features(struct pci_dev *pdev) · 2 unmodified lines | |||
| 50 | { return -EINVAL; } | 51 | { return -EINVAL; } |
| 51 | static inline int pci_max_pasids(struct pci_dev *pdev) | 52 | static inline int pci_max_pasids(struct pci_dev *pdev) |
| 52 | { return -EINVAL; } | 53 | { return -EINVAL; } |
| 54 | static inline int pci_pasid_status(struct pci_dev *pdev) | ||
| 55 | { return -EINVAL; } | ||
| 53 | #endif /* CONFIG_PCI_PASID */ | 56 | #endif /* CONFIG_PCI_PASID */ |
| 54 | 57 | ||
| 55 | #endif /* LINUX_PCI_ATS_H */ | 58 | #endif /* LINUX_PCI_ATS_H */ |
| @@ -67,6 +67,7 @@ struct vfio_device { · 66 unmodified lines | |||
| 67 | struct inode *inode; | 67 | struct inode *inode; |
| 68 | #if IS_ENABLED(CONFIG_IOMMUFD) | 68 | #if IS_ENABLED(CONFIG_IOMMUFD) |
| 69 | struct iommufd_device *iommufd_device; | 69 | struct iommufd_device *iommufd_device; |
| 70 | struct ida pasids; | ||
| 70 | u8 iommufd_attached:1; | 71 | u8 iommufd_attached:1; |
| 71 | #endif | 72 | #endif |
| 72 | u8 cdev_opened:1; | 73 | u8 cdev_opened:1; |
| @@ -91,6 +92,8 @@ struct vfio_device { · 18 unmodified lines | |||
| 91 | * bound iommufd. Undo in unbind_iommufd if @detach_ioas is not | 92 | * bound iommufd. Undo in unbind_iommufd if @detach_ioas is not |
| 92 | * called. | 93 | * called. |
| 93 | * @detach_ioas: Opposite of attach_ioas | 94 | * @detach_ioas: Opposite of attach_ioas |
| 95 | * @pasid_attach_ioas: The pasid variation of attach_ioas | ||
| 96 | * @pasid_detach_ioas: Opposite of pasid_attach_ioas | ||
| 94 | * @open_device: Called when the first file descriptor is opened for this device | 97 | * @open_device: Called when the first file descriptor is opened for this device |
| 95 | * @close_device: Opposite of open_device | 98 | * @close_device: Opposite of open_device |
| 96 | * @read: Perform read(2) on device file descriptor | 99 | * @read: Perform read(2) on device file descriptor |
| @@ -115,6 +118,9 @@ struct vfio_device_ops { · 18 unmodified lines | |||
| 115 | void (*unbind_iommufd)(struct vfio_device *vdev); | 118 | void (*unbind_iommufd)(struct vfio_device *vdev); |
| 116 | int (*attach_ioas)(struct vfio_device *vdev, u32 *pt_id); | 119 | int (*attach_ioas)(struct vfio_device *vdev, u32 *pt_id); |
| 117 | void (*detach_ioas)(struct vfio_device *vdev); | 120 | void (*detach_ioas)(struct vfio_device *vdev); |
| 121 | int (*pasid_attach_ioas)(struct vfio_device *vdev, u32 pasid, | ||
| 122 | u32 *pt_id); | ||
| 123 | void (*pasid_detach_ioas)(struct vfio_device *vdev, u32 pasid); | ||
| 118 | int (*open_device)(struct vfio_device *vdev); | 124 | int (*open_device)(struct vfio_device *vdev); |
| 119 | void (*close_device)(struct vfio_device *vdev); | 125 | void (*close_device)(struct vfio_device *vdev); |
| 120 | ssize_t (*read)(struct vfio_device *vdev, char __user *buf, | 126 | ssize_t (*read)(struct vfio_device *vdev, char __user *buf, |
| @@ -139,6 +145,10 @@ int vfio_iommufd_physical_bind(struct vfio_device *vdev, · 18 unmodified lines | |||
| 139 | void vfio_iommufd_physical_unbind(struct vfio_device *vdev); | 145 | void vfio_iommufd_physical_unbind(struct vfio_device *vdev); |
| 140 | int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id); | 146 | int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id); |
| 141 | void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev); | 147 | void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev); |
| 148 | int vfio_iommufd_physical_pasid_attach_ioas(struct vfio_device *vdev, | ||
| 149 | u32 pasid, u32 *pt_id); | ||
| 150 | void vfio_iommufd_physical_pasid_detach_ioas(struct vfio_device *vdev, | ||
| 151 | u32 pasid); | ||
| 142 | int vfio_iommufd_emulated_bind(struct vfio_device *vdev, | 152 | int vfio_iommufd_emulated_bind(struct vfio_device *vdev, |
| 143 | struct iommufd_ctx *ictx, u32 *out_device_id); | 153 | struct iommufd_ctx *ictx, u32 *out_device_id); |
| 144 | void vfio_iommufd_emulated_unbind(struct vfio_device *vdev); | 154 | void vfio_iommufd_emulated_unbind(struct vfio_device *vdev); |
| @@ -166,6 +176,10 @@ vfio_iommufd_get_dev_id(struct vfio_device *vdev, struct iommufd_ctx *ictx) · 21 unmodified lines | |||
| 166 | ((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL) | 176 | ((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL) |
| 167 | #define vfio_iommufd_physical_detach_ioas \ | 177 | #define vfio_iommufd_physical_detach_ioas \ |
| 168 | ((void (*)(struct vfio_device *vdev)) NULL) | 178 | ((void (*)(struct vfio_device *vdev)) NULL) |
| 179 | #define vfio_iommufd_physical_pasid_attach_ioas \ | ||
| 180 | ((int (*)(struct vfio_device *vdev, u32 pasid, u32 *pt_id)) NULL) | ||
| 181 | #define vfio_iommufd_physical_pasid_detach_ioas \ | ||
| 182 | ((void (*)(struct vfio_device *vdev, u32 pasid)) NULL) | ||
| 169 | #define vfio_iommufd_emulated_bind \ | 183 | #define vfio_iommufd_emulated_bind \ |
| 170 | ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \ | 184 | ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \ |
| 171 | u32 *out_device_id)) NULL) | 185 | u32 *out_device_id)) NULL) |
| @@ -55,6 +55,7 @@ enum { · 54 unmodified lines | |||
| 55 | IOMMUFD_CMD_VIOMMU_ALLOC = 0x90, | 55 | IOMMUFD_CMD_VIOMMU_ALLOC = 0x90, |
| 56 | IOMMUFD_CMD_VDEVICE_ALLOC = 0x91, | 56 | IOMMUFD_CMD_VDEVICE_ALLOC = 0x91, |
| 57 | IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92, | 57 | IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92, |
| 58 | IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93, | ||
| 58 | }; | 59 | }; |
| 59 | 60 | ||
| 60 | /** | 61 | /** |
| @@ -392,6 +393,9 @@ struct iommu_vfio_ioas { · 331 unmodified lines | |||
| 392 | * Any domain attached to the non-PASID part of the | 393 | * Any domain attached to the non-PASID part of the |
| 393 | * device must also be flagged, otherwise attaching a | 394 | * device must also be flagged, otherwise attaching a |
| 394 | * PASID will blocked. | 395 | * PASID will blocked. |
| 396 | * For the user that wants to attach PASID, ioas is | ||
| 397 | * not recommended for both the non-PASID part | ||
| 398 | * and PASID part of the device. | ||
| 395 | * If IOMMU does not support PASID it will return | 399 | * If IOMMU does not support PASID it will return |
| 396 | * error (-EOPNOTSUPP). | 400 | * error (-EOPNOTSUPP). |
| 397 | */ | 401 | */ |
| @@ -608,9 +612,17 @@ enum iommu_hw_info_type { · 210 unmodified lines | |||
| 608 | * IOMMU_HWPT_GET_DIRTY_BITMAP | 612 | * IOMMU_HWPT_GET_DIRTY_BITMAP |
| 609 | * IOMMU_HWPT_SET_DIRTY_TRACKING | 613 | * IOMMU_HWPT_SET_DIRTY_TRACKING |
| 610 | * | 614 | * |
| 615 | * @IOMMU_HW_CAP_PCI_PASID_EXEC: Execute Permission Supported, user ignores it | ||
| 616 | * when the struct | ||
| 617 | * iommu_hw_info::out_max_pasid_log2 is zero. | ||
| 618 | * @IOMMU_HW_CAP_PCI_PASID_PRIV: Privileged Mode Supported, user ignores it | ||
| 619 | * when the struct | ||
| 620 | * iommu_hw_info::out_max_pasid_log2 is zero. | ||
| 611 | */ | 621 | */ |
| 612 | enum iommufd_hw_capabilities { | 622 | enum iommufd_hw_capabilities { |
| 613 | IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0, | 623 | IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0, |
| 624 | IOMMU_HW_CAP_PCI_PASID_EXEC = 1 << 1, | ||
| 625 | IOMMU_HW_CAP_PCI_PASID_PRIV = 1 << 2, | ||
| 614 | }; | 626 | }; |
| 615 | 627 | ||
| 616 | /** | 628 | /** |
| @@ -626,6 +638,9 @@ enum iommufd_hw_capabilities { · 9 unmodified lines | |||
| 626 | * iommu_hw_info_type. | 638 | * iommu_hw_info_type. |
| 627 | * @out_capabilities: Output the generic iommu capability info type as defined | 639 | * @out_capabilities: Output the generic iommu capability info type as defined |
| 628 | * in the enum iommu_hw_capabilities. | 640 | * in the enum iommu_hw_capabilities. |
| 641 | * @out_max_pasid_log2: Output the width of PASIDs. 0 means no PASID support. | ||
| 642 | * PCI devices turn to out_capabilities to check if the | ||
| 643 | * specific capabilities is supported or not. | ||
| 629 | * @__reserved: Must be 0 | 644 | * @__reserved: Must be 0 |
| 630 | * | 645 | * |
| 631 | * Query an iommu type specific hardware information data from an iommu behind | 646 | * Query an iommu type specific hardware information data from an iommu behind |
| @@ -649,7 +664,8 @@ struct iommu_hw_info { · 17 unmodified lines | |||
| 649 | __u32 data_len; | 664 | __u32 data_len; |
| 650 | __aligned_u64 data_uptr; | 665 | __aligned_u64 data_uptr; |
| 651 | __u32 out_data_type; | 666 | __u32 out_data_type; |
| 652 | __u32 __reserved; | 667 | __u8 out_max_pasid_log2; |
| 668 | __u8 __reserved[3]; | ||
| 653 | __aligned_u64 out_capabilities; | 669 | __aligned_u64 out_capabilities; |
| 654 | }; | 670 | }; |
| 655 | #define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO) | 671 | #define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO) |
| @@ -1014,4 +1030,115 @@ struct iommu_ioas_change_process { · 358 unmodified lines | |||
| 1014 | #define IOMMU_IOAS_CHANGE_PROCESS \ | 1030 | #define IOMMU_IOAS_CHANGE_PROCESS \ |
| 1015 | _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS) | 1031 | _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS) |
| 1016 | 1032 | ||
| 1033 | /** | ||
| 1034 | * enum iommu_veventq_flag - flag for struct iommufd_vevent_header | ||
| 1035 | * @IOMMU_VEVENTQ_FLAG_LOST_EVENTS: vEVENTQ has lost vEVENTs | ||
| 1036 | */ | ||
| 1037 | enum iommu_veventq_flag { | ||
| 1038 | IOMMU_VEVENTQ_FLAG_LOST_EVENTS = (1U << 0), | ||
| 1039 | }; | ||
| 1040 | |||
| 1041 | /** | ||
| 1042 | * struct iommufd_vevent_header - Virtual Event Header for a vEVENTQ Status | ||
| 1043 | * @flags: Combination of enum iommu_veventq_flag | ||
| 1044 | * @sequence: The sequence index of a vEVENT in the vEVENTQ, with a range of | ||
| 1045 | * [0, INT_MAX] where the following index of INT_MAX is 0 | ||
| 1046 | * | ||
| 1047 | * Each iommufd_vevent_header reports a sequence index of the following vEVENT: | ||
| 1048 | * | ||
| 1049 | * +----------------------+-------+----------------------+-------+---+-------+ | ||
| 1050 | * | header0 {sequence=0} | data0 | header1 {sequence=1} | data1 |...| dataN | | ||
| 1051 | * +----------------------+-------+----------------------+-------+---+-------+ | ||
| 1052 | * | ||
| 1053 | * And this sequence index is expected to be monotonic to the sequence index of | ||
| 1054 | * the previous vEVENT. If two adjacent sequence indexes has a delta larger than | ||
| 1055 | * 1, it means that delta - 1 number of vEVENTs has lost, e.g. two lost vEVENTs: | ||
| 1056 | * | ||
| 1057 | * +-----+----------------------+-------+----------------------+-------+-----+ | ||
| 1058 | * | ... | header3 {sequence=3} | data3 | header6 {sequence=6} | data6 | ... | | ||
| 1059 | * +-----+----------------------+-------+----------------------+-------+-----+ | ||
| 1060 | * | ||
| 1061 | * If a vEVENT lost at the tail of the vEVENTQ and there is no following vEVENT | ||
| 1062 | * providing the next sequence index, an IOMMU_VEVENTQ_FLAG_LOST_EVENTS header | ||
| 1063 | * would be added to the tail, and no data would follow this header: | ||
| 1064 | * | ||
| 1065 | * +--+----------------------+-------+-----------------------------------------+ | ||
| 1066 | * |..| header3 {sequence=3} | data3 | header4 {flags=LOST_EVENTS, sequence=4} | | ||
| 1067 | * +--+----------------------+-------+-----------------------------------------+ | ||
| 1068 | */ | ||
| 1069 | struct iommufd_vevent_header { | ||
| 1070 | __u32 flags; | ||
| 1071 | __u32 sequence; | ||
| 1072 | }; | ||
| 1073 | |||
| 1074 | /** | ||
| 1075 | * enum iommu_veventq_type - Virtual Event Queue Type | ||
| 1076 | * @IOMMU_VEVENTQ_TYPE_DEFAULT: Reserved for future use | ||
| 1077 | * @IOMMU_VEVENTQ_TYPE_ARM_SMMUV3: ARM SMMUv3 Virtual Event Queue | ||
| 1078 | */ | ||
| 1079 | enum iommu_veventq_type { | ||
| 1080 | IOMMU_VEVENTQ_TYPE_DEFAULT = 0, | ||
| 1081 | IOMMU_VEVENTQ_TYPE_ARM_SMMUV3 = 1, | ||
| 1082 | }; | ||
| 1083 | |||
| 1084 | /** | ||
| 1085 | * struct iommu_vevent_arm_smmuv3 - ARM SMMUv3 Virtual Event | ||
| 1086 | * (IOMMU_VEVENTQ_TYPE_ARM_SMMUV3) | ||
| 1087 | * @evt: 256-bit ARM SMMUv3 Event record, little-endian. | ||
| 1088 | * Reported event records: (Refer to "7.3 Event records" in SMMUv3 HW Spec) | ||
| 1089 | * - 0x04 C_BAD_STE | ||
| 1090 | * - 0x06 F_STREAM_DISABLED | ||
| 1091 | * - 0x08 C_BAD_SUBSTREAMID | ||
| 1092 | * - 0x0a C_BAD_CD | ||
| 1093 | * - 0x10 F_TRANSLATION | ||
| 1094 | * - 0x11 F_ADDR_SIZE | ||
| 1095 | * - 0x12 F_ACCESS | ||
| 1096 | * - 0x13 F_PERMISSION | ||
| 1097 | * | ||
| 1098 | * StreamID field reports a virtual device ID. To receive a virtual event for a | ||
| 1099 | * device, a vDEVICE must be allocated via IOMMU_VDEVICE_ALLOC. | ||
| 1100 | */ | ||
| 1101 | struct iommu_vevent_arm_smmuv3 { | ||
| 1102 | __aligned_le64 evt[4]; | ||
| 1103 | }; | ||
| 1104 | |||
| 1105 | /** | ||
| 1106 | * struct iommu_veventq_alloc - ioctl(IOMMU_VEVENTQ_ALLOC) | ||
| 1107 | * @size: sizeof(struct iommu_veventq_alloc) | ||
| 1108 | * @flags: Must be 0 | ||
| 1109 | * @viommu_id: virtual IOMMU ID to associate the vEVENTQ with | ||
| 1110 | * @type: Type of the vEVENTQ. Must be defined in enum iommu_veventq_type | ||
| 1111 | * @veventq_depth: Maximum number of events in the vEVENTQ | ||
| 1112 | * @out_veventq_id: The ID of the new vEVENTQ | ||
| 1113 | * @out_veventq_fd: The fd of the new vEVENTQ. User space must close the | ||
| 1114 | * successfully returned fd after using it | ||
| 1115 | * @__reserved: Must be 0 | ||
| 1116 | * | ||
| 1117 | * Explicitly allocate a virtual event queue interface for a vIOMMU. A vIOMMU | ||
| 1118 | * can have multiple FDs for different types, but is confined to one per @type. | ||
| 1119 | * User space should open the @out_veventq_fd to read vEVENTs out of a vEVENTQ, | ||
| 1120 | * if there are vEVENTs available. A vEVENTQ will lose events due to overflow, | ||
| 1121 | * if the number of the vEVENTs hits @veventq_depth. | ||
| 1122 | * | ||
| 1123 | * Each vEVENT in a vEVENTQ encloses a struct iommufd_vevent_header followed by | ||
| 1124 | * a type-specific data structure, in a normal case: | ||
| 1125 | * | ||
| 1126 | * +-+---------+-------+---------+-------+-----+---------+-------+-+ | ||
| 1127 | * | | header0 | data0 | header1 | data1 | ... | headerN | dataN | | | ||
| 1128 | * +-+---------+-------+---------+-------+-----+---------+-------+-+ | ||
| 1129 | * | ||
| 1130 | * unless a tailing IOMMU_VEVENTQ_FLAG_LOST_EVENTS header is logged (refer to | ||
| 1131 | * struct iommufd_vevent_header). | ||
| 1132 | */ | ||
| 1133 | struct iommu_veventq_alloc { | ||
| 1134 | __u32 size; | ||
| 1135 | __u32 flags; | ||
| 1136 | __u32 viommu_id; | ||
| 1137 | __u32 type; | ||
| 1138 | __u32 veventq_depth; | ||
| 1139 | __u32 out_veventq_id; | ||
| 1140 | __u32 out_veventq_fd; | ||
| 1141 | __u32 __reserved; | ||
| 1142 | }; | ||
| 1143 | #define IOMMU_VEVENTQ_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VEVENTQ_ALLOC) | ||
| 1017 | #endif | 1144 | #endif |
| @@ -932,29 +932,34 @@ struct vfio_device_bind_iommufd { · 931 unmodified lines | |||
| 932 | * VFIO_DEVICE_ATTACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 19, | 932 | * VFIO_DEVICE_ATTACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 19, |
| 933 | * struct vfio_device_attach_iommufd_pt) | 933 | * struct vfio_device_attach_iommufd_pt) |
| 934 | * @argsz: User filled size of this data. | 934 | * @argsz: User filled size of this data. |
| 935 | * @flags: Must be 0. | 935 | * @flags: Flags for attach. |
| 936 | * @pt_id: Input the target id which can represent an ioas or a hwpt | 936 | * @pt_id: Input the target id which can represent an ioas or a hwpt |
| 937 | * allocated via iommufd subsystem. | 937 | * allocated via iommufd subsystem. |
| 938 | * Output the input ioas id or the attached hwpt id which could | 938 | * Output the input ioas id or the attached hwpt id which could |
| 939 | * be the specified hwpt itself or a hwpt automatically created | 939 | * be the specified hwpt itself or a hwpt automatically created |
| 940 | * for the specified ioas by kernel during the attachment. | 940 | * for the specified ioas by kernel during the attachment. |
| 941 | * @pasid: The pasid to be attached, only meaningful when | ||
| 942 | * VFIO_DEVICE_ATTACH_PASID is set in @flags | ||
| 941 | * | 943 | * |
| 942 | * Associate the device with an address space within the bound iommufd. | 944 | * Associate the device with an address space within the bound iommufd. |
| 943 | * Undo by VFIO_DEVICE_DETACH_IOMMUFD_PT or device fd close. This is only | 945 | * Undo by VFIO_DEVICE_DETACH_IOMMUFD_PT or device fd close. This is only |
| 944 | * allowed on cdev fds. | 946 | * allowed on cdev fds. |
| 945 | * | 947 | * |
| 946 | * If a vfio device is currently attached to a valid hw_pagetable, without doing | 948 | * If a vfio device or a pasid of this device is currently attached to a valid |
| 947 | * a VFIO_DEVICE_DETACH_IOMMUFD_PT, a second VFIO_DEVICE_ATTACH_IOMMUFD_PT ioctl | 949 | * hw_pagetable (hwpt), without doing a VFIO_DEVICE_DETACH_IOMMUFD_PT, a second |
| 948 | * passing in another hw_pagetable (hwpt) id is allowed. This action, also known | 950 | * VFIO_DEVICE_ATTACH_IOMMUFD_PT ioctl passing in another hwpt id is allowed. |
| 949 | * as a hw_pagetable replacement, will replace the device's currently attached | 951 | * This action, also known as a hw_pagetable replacement, will replace the |
| 950 | * hw_pagetable with a new hw_pagetable corresponding to the given pt_id. | 952 | * currently attached hwpt of the device or the pasid of this device with a new |
| 953 | * hwpt corresponding to the given pt_id. | ||
| 951 | * | 954 | * |
| 952 | * Return: 0 on success, -errno on failure. | 955 | * Return: 0 on success, -errno on failure. |
| 953 | */ | 956 | */ |
| 954 | struct vfio_device_attach_iommufd_pt { | 957 | struct vfio_device_attach_iommufd_pt { |
| 955 | __u32 argsz; | 958 | __u32 argsz; |
| 956 | __u32 flags; | 959 | __u32 flags; |
| 960 | #define VFIO_DEVICE_ATTACH_PASID (1 << 0) | ||
| 957 | __u32 pt_id; | 961 | __u32 pt_id; |
| 962 | __u32 pasid; | ||
| 958 | }; | 963 | }; |
| 959 | 964 | ||
| 960 | #define VFIO_DEVICE_ATTACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 19) | 965 | #define VFIO_DEVICE_ATTACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 19) |
| @@ -963,17 +968,21 @@ struct vfio_device_attach_iommufd_pt { · 2 unmodified lines | |||
| 963 | * VFIO_DEVICE_DETACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 20, | 968 | * VFIO_DEVICE_DETACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 20, |
| 964 | * struct vfio_device_detach_iommufd_pt) | 969 | * struct vfio_device_detach_iommufd_pt) |
| 965 | * @argsz: User filled size of this data. | 970 | * @argsz: User filled size of this data. |
| 966 | * @flags: Must be 0. | 971 | * @flags: Flags for detach. |
| 972 | * @pasid: The pasid to be detached, only meaningful when | ||
| 973 | * VFIO_DEVICE_DETACH_PASID is set in @flags | ||
| 967 | * | 974 | * |
| 968 | * Remove the association of the device and its current associated address | 975 | * Remove the association of the device or a pasid of the device and its current |
| 969 | * space. After it, the device should be in a blocking DMA state. This is only | 976 | * associated address space. After it, the device or the pasid should be in a |
| 970 | * allowed on cdev fds. | 977 | * blocking DMA state. This is only allowed on cdev fds. |
| 971 | * | 978 | * |
| 972 | * Return: 0 on success, -errno on failure. | 979 | * Return: 0 on success, -errno on failure. |
| 973 | */ | 980 | */ |
| 974 | struct vfio_device_detach_iommufd_pt { | 981 | struct vfio_device_detach_iommufd_pt { |
| 975 | __u32 argsz; | 982 | __u32 argsz; |
| 976 | __u32 flags; | 983 | __u32 flags; |
| 984 | #define VFIO_DEVICE_DETACH_PASID (1 << 0) | ||
| 985 | __u32 pasid; | ||
| 977 | }; | 986 | }; |
| 978 | 987 | ||
| 979 | #define VFIO_DEVICE_DETACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 20) | 988 | #define VFIO_DEVICE_DETACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 20) |
| @@ -476,6 +476,73 @@ nospc: · 475 unmodified lines | |||
| 476 | } | 476 | } |
| 477 | EXPORT_SYMBOL(ida_alloc_range); | 477 | EXPORT_SYMBOL(ida_alloc_range); |
| 478 | 478 | ||
| 479 | /** | ||
| 480 | * ida_find_first_range - Get the lowest used ID. | ||
| 481 | * @ida: IDA handle. | ||
| 482 | * @min: Lowest ID to get. | ||
| 483 | * @max: Highest ID to get. | ||
| 484 | * | ||
| 485 | * Get the lowest used ID between @min and @max, inclusive. The returned | ||
| 486 | * ID will not exceed %INT_MAX, even if @max is larger. | ||
| 487 | * | ||
| 488 | * Context: Any context. Takes and releases the xa_lock. | ||
| 489 | * Return: The lowest used ID, or errno if no used ID is found. | ||
| 490 | */ | ||
| 491 | int ida_find_first_range(struct ida *ida, unsigned int min, unsigned int max) | ||
| 492 | { | ||
| 493 | unsigned long index = min / IDA_BITMAP_BITS; | ||
| 494 | unsigned int offset = min % IDA_BITMAP_BITS; | ||
| 495 | unsigned long *addr, size, bit; | ||
| 496 | unsigned long tmp = 0; | ||
| 497 | unsigned long flags; | ||
| 498 | void *entry; | ||
| 499 | int ret; | ||
| 500 | |||
| 501 | if ((int)min < 0) | ||
| 502 | return -EINVAL; | ||
| 503 | if ((int)max < 0) | ||
| 504 | max = INT_MAX; | ||
| 505 | |||
| 506 | xa_lock_irqsave(&ida->xa, flags); | ||
| 507 | |||
| 508 | entry = xa_find(&ida->xa, &index, max / IDA_BITMAP_BITS, XA_PRESENT); | ||
| 509 | if (!entry) { | ||
| 510 | ret = -ENOENT; | ||
| 511 | goto err_unlock; | ||
| 512 | } | ||
| 513 | |||
| 514 | if (index > min / IDA_BITMAP_BITS) | ||
| 515 | offset = 0; | ||
| 516 | if (index * IDA_BITMAP_BITS + offset > max) { | ||
| 517 | ret = -ENOENT; | ||
| 518 | goto err_unlock; | ||
| 519 | } | ||
| 520 | |||
| 521 | if (xa_is_value(entry)) { | ||
| 522 | tmp = xa_to_value(entry); | ||
| 523 | addr = &tmp; | ||
| 524 | size = BITS_PER_XA_VALUE; | ||
| 525 | } else { | ||
| 526 | addr = ((struct ida_bitmap *)entry)->bitmap; | ||
| 527 | size = IDA_BITMAP_BITS; | ||
| 528 | } | ||
| 529 | |||
| 530 | bit = find_next_bit(addr, size, offset); | ||
| 531 | |||
| 532 | xa_unlock_irqrestore(&ida->xa, flags); | ||
| 533 | |||
| 534 | if (bit == size || | ||
| 535 | index * IDA_BITMAP_BITS + bit > max) | ||
| 536 | return -ENOENT; | ||
| 537 | |||
| 538 | return index * IDA_BITMAP_BITS + bit; | ||
| 539 | |||
| 540 | err_unlock: | ||
| 541 | xa_unlock_irqrestore(&ida->xa, flags); | ||
| 542 | return ret; | ||
| 543 | } | ||
| 544 | EXPORT_SYMBOL(ida_find_first_range); | ||
| 545 | |||
| 479 | /** | 546 | /** |
| 480 | * ida_free() - Release an allocated ID. | 547 | * ida_free() - Release an allocated ID. |
| 481 | * @ida: IDA handle. | 548 | * @ida: IDA handle. |
| @@ -189,6 +189,75 @@ static void ida_check_bad_free(struct ida *ida) · 188 unmodified lines | |||
| 189 | IDA_BUG_ON(ida, !ida_is_empty(ida)); | 189 | IDA_BUG_ON(ida, !ida_is_empty(ida)); |
| 190 | } | 190 | } |
| 191 | 191 | ||
| 192 | /* | ||
| 193 | * Check ida_find_first_range() and varriants. | ||
| 194 | */ | ||
| 195 | static void ida_check_find_first(struct ida *ida) | ||
| 196 | { | ||
| 197 | /* IDA is empty; all of the below should be not exist */ | ||
| 198 | IDA_BUG_ON(ida, ida_exists(ida, 0)); | ||
| 199 | IDA_BUG_ON(ida, ida_exists(ida, 3)); | ||
| 200 | IDA_BUG_ON(ida, ida_exists(ida, 63)); | ||
| 201 | IDA_BUG_ON(ida, ida_exists(ida, 1023)); | ||
| 202 | IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1)); | ||
| 203 | |||
| 204 | /* IDA contains a single value entry */ | ||
| 205 | IDA_BUG_ON(ida, ida_alloc_min(ida, 3, GFP_KERNEL) != 3); | ||
| 206 | IDA_BUG_ON(ida, ida_exists(ida, 0)); | ||
| 207 | IDA_BUG_ON(ida, !ida_exists(ida, 3)); | ||
| 208 | IDA_BUG_ON(ida, ida_exists(ida, 63)); | ||
| 209 | IDA_BUG_ON(ida, ida_exists(ida, 1023)); | ||
| 210 | IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1)); | ||
| 211 | |||
| 212 | IDA_BUG_ON(ida, ida_alloc_min(ida, 63, GFP_KERNEL) != 63); | ||
| 213 | IDA_BUG_ON(ida, ida_exists(ida, 0)); | ||
| 214 | IDA_BUG_ON(ida, !ida_exists(ida, 3)); | ||
| 215 | IDA_BUG_ON(ida, !ida_exists(ida, 63)); | ||
| 216 | IDA_BUG_ON(ida, ida_exists(ida, 1023)); | ||
| 217 | IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1)); | ||
| 218 | |||
| 219 | /* IDA contains a single bitmap */ | ||
| 220 | IDA_BUG_ON(ida, ida_alloc_min(ida, 1023, GFP_KERNEL) != 1023); | ||
| 221 | IDA_BUG_ON(ida, ida_exists(ida, 0)); | ||
| 222 | IDA_BUG_ON(ida, !ida_exists(ida, 3)); | ||
| 223 | IDA_BUG_ON(ida, !ida_exists(ida, 63)); | ||
| 224 | IDA_BUG_ON(ida, !ida_exists(ida, 1023)); | ||
| 225 | IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1)); | ||
| 226 | |||
| 227 | /* IDA contains a tree */ | ||
| 228 | IDA_BUG_ON(ida, ida_alloc_min(ida, (1 << 20) - 1, GFP_KERNEL) != (1 << 20) - 1); | ||
| 229 | IDA_BUG_ON(ida, ida_exists(ida, 0)); | ||
| 230 | IDA_BUG_ON(ida, !ida_exists(ida, 3)); | ||
| 231 | IDA_BUG_ON(ida, !ida_exists(ida, 63)); | ||
| 232 | IDA_BUG_ON(ida, !ida_exists(ida, 1023)); | ||
| 233 | IDA_BUG_ON(ida, !ida_exists(ida, (1 << 20) - 1)); | ||
| 234 | |||
| 235 | /* Now try to find first */ | ||
| 236 | IDA_BUG_ON(ida, ida_find_first(ida) != 3); | ||
| 237 | IDA_BUG_ON(ida, ida_find_first_range(ida, -1, 2) != -EINVAL); | ||
| 238 | IDA_BUG_ON(ida, ida_find_first_range(ida, 0, 2) != -ENOENT); // no used ID | ||
| 239 | IDA_BUG_ON(ida, ida_find_first_range(ida, 0, 3) != 3); | ||
| 240 | IDA_BUG_ON(ida, ida_find_first_range(ida, 1, 3) != 3); | ||
| 241 | IDA_BUG_ON(ida, ida_find_first_range(ida, 3, 3) != 3); | ||
| 242 | IDA_BUG_ON(ida, ida_find_first_range(ida, 2, 4) != 3); | ||
| 243 | IDA_BUG_ON(ida, ida_find_first_range(ida, 4, 3) != -ENOENT); // min > max, fail | ||
| 244 | IDA_BUG_ON(ida, ida_find_first_range(ida, 4, 60) != -ENOENT); // no used ID | ||
| 245 | IDA_BUG_ON(ida, ida_find_first_range(ida, 4, 64) != 63); | ||
| 246 | IDA_BUG_ON(ida, ida_find_first_range(ida, 63, 63) != 63); | ||
| 247 | IDA_BUG_ON(ida, ida_find_first_range(ida, 64, 1026) != 1023); | ||
| 248 | IDA_BUG_ON(ida, ida_find_first_range(ida, 1023, 1023) != 1023); | ||
| 249 | IDA_BUG_ON(ida, ida_find_first_range(ida, 1023, (1 << 20) - 1) != 1023); | ||
| 250 | IDA_BUG_ON(ida, ida_find_first_range(ida, 1024, (1 << 20) - 1) != (1 << 20) - 1); | ||
| 251 | IDA_BUG_ON(ida, ida_find_first_range(ida, (1 << 20), INT_MAX) != -ENOENT); | ||
| 252 | |||
| 253 | ida_free(ida, 3); | ||
| 254 | ida_free(ida, 63); | ||
| 255 | ida_free(ida, 1023); | ||
| 256 | ida_free(ida, (1 << 20) - 1); | ||
| 257 | |||
| 258 | IDA_BUG_ON(ida, !ida_is_empty(ida)); | ||
| 259 | } | ||
| 260 | |||
| 192 | static DEFINE_IDA(ida); | 261 | static DEFINE_IDA(ida); |
| 193 | 262 | ||
| 194 | static int ida_checks(void) | 263 | static int ida_checks(void) |
| @@ -202,6 +271,7 @@ static int ida_checks(void) · 7 unmodified lines | |||
| 202 | ida_check_max(&ida); | 271 | ida_check_max(&ida); |
| 203 | ida_check_conv(&ida); | 272 | ida_check_conv(&ida); |
| 204 | ida_check_bad_free(&ida); | 273 | ida_check_bad_free(&ida); |
| 274 | ida_check_find_first(&ida); | ||
| 205 | 275 | ||
| 206 | printk("IDA: %u of %u tests passed\n", tests_passed, tests_run); | 276 | printk("IDA: %u of %u tests passed\n", tests_passed, tests_run); |
| 207 | return (tests_run != tests_passed) ? 0 : -EINVAL; | 277 | return (tests_run != tests_passed) ? 0 : -EINVAL; |
| @@ -342,12 +342,14 @@ FIXTURE(iommufd_ioas) · 341 unmodified lines | |||
| 342 | uint32_t hwpt_id; | 342 | uint32_t hwpt_id; |
| 343 | uint32_t device_id; | 343 | uint32_t device_id; |
| 344 | uint64_t base_iova; | 344 | uint64_t base_iova; |
| 345 | uint32_t device_pasid_id; | ||
| 345 | }; | 346 | }; |
| 346 | 347 | ||
| 347 | FIXTURE_VARIANT(iommufd_ioas) | 348 | FIXTURE_VARIANT(iommufd_ioas) |
| 348 | { | 349 | { |
| 349 | unsigned int mock_domains; | 350 | unsigned int mock_domains; |
| 350 | unsigned int memory_limit; | 351 | unsigned int memory_limit; |
| 352 | bool pasid_capable; | ||
| 351 | }; | 353 | }; |
| 352 | 354 | ||
| 353 | FIXTURE_SETUP(iommufd_ioas) | 355 | FIXTURE_SETUP(iommufd_ioas) |
| @@ -372,6 +374,12 @@ FIXTURE_SETUP(iommufd_ioas) · 18 unmodified lines | |||
| 372 | IOMMU_TEST_DEV_CACHE_DEFAULT); | 374 | IOMMU_TEST_DEV_CACHE_DEFAULT); |
| 373 | self->base_iova = MOCK_APERTURE_START; | 375 | self->base_iova = MOCK_APERTURE_START; |
| 374 | } | 376 | } |
| 377 | |||
| 378 | if (variant->pasid_capable) | ||
| 379 | test_cmd_mock_domain_flags(self->ioas_id, | ||
| 380 | MOCK_FLAGS_DEVICE_PASID, | ||
| 381 | NULL, NULL, | ||
| 382 | &self->device_pasid_id); | ||
| 375 | } | 383 | } |
| 376 | 384 | ||
| 377 | FIXTURE_TEARDOWN(iommufd_ioas) | 385 | FIXTURE_TEARDOWN(iommufd_ioas) |
| @@ -387,6 +395,7 @@ FIXTURE_VARIANT_ADD(iommufd_ioas, no_domain) · 9 unmodified lines | |||
| 387 | FIXTURE_VARIANT_ADD(iommufd_ioas, mock_domain) | 395 | FIXTURE_VARIANT_ADD(iommufd_ioas, mock_domain) |
| 388 | { | 396 | { |
| 389 | .mock_domains = 1, | 397 | .mock_domains = 1, |
| 398 | .pasid_capable = true, | ||
| 390 | }; | 399 | }; |
| 391 | 400 | ||
| 392 | FIXTURE_VARIANT_ADD(iommufd_ioas, two_mock_domain) | 401 | FIXTURE_VARIANT_ADD(iommufd_ioas, two_mock_domain) |
| @@ -439,6 +448,10 @@ TEST_F(iommufd_ioas, alloc_hwpt_nested) · 46 unmodified lines | |||
| 439 | &test_hwpt_id); | 448 | &test_hwpt_id); |
| 440 | test_err_hwpt_alloc(EINVAL, self->device_id, self->device_id, 0, | 449 | test_err_hwpt_alloc(EINVAL, self->device_id, self->device_id, 0, |
| 441 | &test_hwpt_id); | 450 | &test_hwpt_id); |
| 451 | test_err_hwpt_alloc(EOPNOTSUPP, self->device_id, self->ioas_id, | ||
| 452 | IOMMU_HWPT_ALLOC_NEST_PARENT | | ||
| 453 | IOMMU_HWPT_FAULT_ID_VALID, | ||
| 454 | &test_hwpt_id); | ||
| 442 | 455 | ||
| 443 | test_cmd_hwpt_alloc(self->device_id, self->ioas_id, | 456 | test_cmd_hwpt_alloc(self->device_id, self->ioas_id, |
| 444 | IOMMU_HWPT_ALLOC_NEST_PARENT, | 457 | IOMMU_HWPT_ALLOC_NEST_PARENT, |
| @@ -748,6 +761,8 @@ TEST_F(iommufd_ioas, get_hw_info) · 303 unmodified lines | |||
| 748 | } buffer_smaller; | 761 | } buffer_smaller; |
| 749 | 762 | ||
| 750 | if (self->device_id) { | 763 | if (self->device_id) { |
| 764 | uint8_t max_pasid = 0; | ||
| 765 | |||
| 751 | /* Provide a zero-size user_buffer */ | 766 | /* Provide a zero-size user_buffer */ |
| 752 | test_cmd_get_hw_info(self->device_id, NULL, 0); | 767 | test_cmd_get_hw_info(self->device_id, NULL, 0); |
| 753 | /* Provide a user_buffer with exact size */ | 768 | /* Provide a user_buffer with exact size */ |
| @@ -762,6 +777,13 @@ TEST_F(iommufd_ioas, get_hw_info) · 8 unmodified lines | |||
| 762 | * the fields within the size range still gets updated. | 777 | * the fields within the size range still gets updated. |
| 763 | */ | 778 | */ |
| 764 | test_cmd_get_hw_info(self->device_id, &buffer_smaller, sizeof(buffer_smaller)); | 779 | test_cmd_get_hw_info(self->device_id, &buffer_smaller, sizeof(buffer_smaller)); |
| 780 | test_cmd_get_hw_info_pasid(self->device_id, &max_pasid); | ||
| 781 | ASSERT_EQ(0, max_pasid); | ||
| 782 | if (variant->pasid_capable) { | ||
| 783 | test_cmd_get_hw_info_pasid(self->device_pasid_id, | ||
| 784 | &max_pasid); | ||
| 785 | ASSERT_EQ(MOCK_PASID_WIDTH, max_pasid); | ||
| 786 | } | ||
| 765 | } else { | 787 | } else { |
| 766 | test_err_get_hw_info(ENOENT, self->device_id, | 788 | test_err_get_hw_info(ENOENT, self->device_id, |
| 767 | &buffer_exact, sizeof(buffer_exact)); | 789 | &buffer_exact, sizeof(buffer_exact)); |
| @@ -2736,6 +2758,7 @@ TEST_F(iommufd_viommu, viommu_alloc_nested_iopf) · 1968 unmodified lines | |||
| 2736 | uint32_t iopf_hwpt_id; | 2758 | uint32_t iopf_hwpt_id; |
| 2737 | uint32_t fault_id; | 2759 | uint32_t fault_id; |
| 2738 | uint32_t fault_fd; | 2760 | uint32_t fault_fd; |
| 2761 | uint32_t vdev_id; | ||
| 2739 | 2762 | ||
| 2740 | if (self->device_id) { | 2763 | if (self->device_id) { |
| 2741 | test_ioctl_fault_alloc(&fault_id, &fault_fd); | 2764 | test_ioctl_fault_alloc(&fault_id, &fault_fd); |
| @@ -2752,6 +2775,10 @@ TEST_F(iommufd_viommu, viommu_alloc_nested_iopf) · 10 unmodified lines | |||
| 2752 | &iopf_hwpt_id, IOMMU_HWPT_DATA_SELFTEST, &data, | 2775 | &iopf_hwpt_id, IOMMU_HWPT_DATA_SELFTEST, &data, |
| 2753 | sizeof(data)); | 2776 | sizeof(data)); |
| 2754 | 2777 | ||
| 2778 | /* Must allocate vdevice before attaching to a nested hwpt */ | ||
| 2779 | test_err_mock_domain_replace(ENOENT, self->stdev_id, | ||
| 2780 | iopf_hwpt_id); | ||
| 2781 | test_cmd_vdevice_alloc(viommu_id, dev_id, 0x99, &vdev_id); | ||
| 2755 | test_cmd_mock_domain_replace(self->stdev_id, iopf_hwpt_id); | 2782 | test_cmd_mock_domain_replace(self->stdev_id, iopf_hwpt_id); |
| 2756 | EXPECT_ERRNO(EBUSY, | 2783 | EXPECT_ERRNO(EBUSY, |
| 2757 | _test_ioctl_destroy(self->fd, iopf_hwpt_id)); | 2784 | _test_ioctl_destroy(self->fd, iopf_hwpt_id)); |
| @@ -2769,15 +2796,46 @@ TEST_F(iommufd_viommu, vdevice_alloc) · 11 unmodified lines | |||
| 2769 | uint32_t viommu_id = self->viommu_id; | 2796 | uint32_t viommu_id = self->viommu_id; |
| 2770 | uint32_t dev_id = self->device_id; | 2797 | uint32_t dev_id = self->device_id; |
| 2771 | uint32_t vdev_id = 0; | 2798 | uint32_t vdev_id = 0; |
| 2799 | uint32_t veventq_id; | ||
| 2800 | uint32_t veventq_fd; | ||
| 2801 | int prev_seq = -1; | ||
| 2772 | 2802 | ||
| 2773 | if (dev_id) { | 2803 | if (dev_id) { |
| 2804 | /* Must allocate vdevice before attaching to a nested hwpt */ | ||
| 2805 | test_err_mock_domain_replace(ENOENT, self->stdev_id, | ||
| 2806 | self->nested_hwpt_id); | ||
| 2807 | |||
| 2808 | /* Allocate a vEVENTQ with veventq_depth=2 */ | ||
| 2809 | test_cmd_veventq_alloc(viommu_id, IOMMU_VEVENTQ_TYPE_SELFTEST, | ||
| 2810 | &veventq_id, &veventq_fd); | ||
| 2811 | test_err_veventq_alloc(EEXIST, viommu_id, | ||
| 2812 | IOMMU_VEVENTQ_TYPE_SELFTEST, NULL, NULL); | ||
| 2774 | /* Set vdev_id to 0x99, unset it, and set to 0x88 */ | 2813 | /* Set vdev_id to 0x99, unset it, and set to 0x88 */ |
| 2775 | test_cmd_vdevice_alloc(viommu_id, dev_id, 0x99, &vdev_id); | 2814 | test_cmd_vdevice_alloc(viommu_id, dev_id, 0x99, &vdev_id); |
| 2815 | test_cmd_mock_domain_replace(self->stdev_id, | ||
| 2816 | self->nested_hwpt_id); | ||
| 2817 | test_cmd_trigger_vevents(dev_id, 1); | ||
| 2818 | test_cmd_read_vevents(veventq_fd, 1, 0x99, &prev_seq); | ||
| 2776 | test_err_vdevice_alloc(EEXIST, viommu_id, dev_id, 0x99, | 2819 | test_err_vdevice_alloc(EEXIST, viommu_id, dev_id, 0x99, |
| 2777 | &vdev_id); | 2820 | &vdev_id); |
| 2821 | test_cmd_mock_domain_replace(self->stdev_id, self->ioas_id); | ||
| 2778 | test_ioctl_destroy(vdev_id); | 2822 | test_ioctl_destroy(vdev_id); |
| 2823 | |||
| 2824 | /* Try again with 0x88 */ | ||
| 2779 | test_cmd_vdevice_alloc(viommu_id, dev_id, 0x88, &vdev_id); | 2825 | test_cmd_vdevice_alloc(viommu_id, dev_id, 0x88, &vdev_id); |
| 2826 | test_cmd_mock_domain_replace(self->stdev_id, | ||
| 2827 | self->nested_hwpt_id); | ||
| 2828 | /* Trigger an overflow with three events */ | ||
| 2829 | test_cmd_trigger_vevents(dev_id, 3); | ||
| 2830 | test_err_read_vevents(EOVERFLOW, veventq_fd, 3, 0x88, | ||
| 2831 | &prev_seq); | ||
| 2832 | /* Overflow must be gone after the previous reads */ | ||
| 2833 | test_cmd_trigger_vevents(dev_id, 1); | ||
| 2834 | test_cmd_read_vevents(veventq_fd, 1, 0x88, &prev_seq); | ||
| 2835 | close(veventq_fd); | ||
| 2836 | test_cmd_mock_domain_replace(self->stdev_id, self->ioas_id); | ||
| 2780 | test_ioctl_destroy(vdev_id); | 2837 | test_ioctl_destroy(vdev_id); |
| 2838 | test_ioctl_destroy(veventq_id); | ||
| 2781 | } else { | 2839 | } else { |
| 2782 | test_err_vdevice_alloc(ENOENT, viommu_id, dev_id, 0x99, NULL); | 2840 | test_err_vdevice_alloc(ENOENT, viommu_id, dev_id, 0x99, NULL); |
| 2783 | } | 2841 | } |
| @@ -2956,4 +3014,311 @@ TEST_F(iommufd_viommu, vdevice_cache) · 172 unmodified lines | |||
| 2956 | } | 3014 | } |
| 2957 | } | 3015 | } |
| 2958 | 3016 | ||
| 3017 | FIXTURE(iommufd_device_pasid) | ||
| 3018 | { | ||
| 3019 | int fd; | ||
| 3020 | uint32_t ioas_id; | ||
| 3021 | uint32_t hwpt_id; | ||
| 3022 | uint32_t stdev_id; | ||
| 3023 | uint32_t device_id; | ||
| 3024 | uint32_t no_pasid_stdev_id; | ||
| 3025 | uint32_t no_pasid_device_id; | ||
| 3026 | }; | ||
| 3027 | |||
| 3028 | FIXTURE_VARIANT(iommufd_device_pasid) | ||
| 3029 | { | ||
| 3030 | bool pasid_capable; | ||
| 3031 | }; | ||
| 3032 | |||
| 3033 | FIXTURE_SETUP(iommufd_device_pasid) | ||
| 3034 | { | ||
| 3035 | self->fd = open("/dev/iommu", O_RDWR); | ||
| 3036 | ASSERT_NE(-1, self->fd); | ||
| 3037 | test_ioctl_ioas_alloc(&self->ioas_id); | ||
| 3038 | |||
| 3039 | test_cmd_mock_domain_flags(self->ioas_id, | ||
| 3040 | MOCK_FLAGS_DEVICE_PASID, | ||
| 3041 | &self->stdev_id, &self->hwpt_id, | ||
| 3042 | &self->device_id); | ||
| 3043 | if (!variant->pasid_capable) | ||
| 3044 | test_cmd_mock_domain_flags(self->ioas_id, 0, | ||
| 3045 | &self->no_pasid_stdev_id, NULL, | ||
| 3046 | &self->no_pasid_device_id); | ||
| 3047 | } | ||
| 3048 | |||
| 3049 | FIXTURE_TEARDOWN(iommufd_device_pasid) | ||
| 3050 | { | ||
| 3051 | teardown_iommufd(self->fd, _metadata); | ||
| 3052 | } | ||
| 3053 | |||
| 3054 | FIXTURE_VARIANT_ADD(iommufd_device_pasid, no_pasid) | ||
| 3055 | { | ||
| 3056 | .pasid_capable = false, | ||
| 3057 | }; | ||
| 3058 | |||
| 3059 | FIXTURE_VARIANT_ADD(iommufd_device_pasid, has_pasid) | ||
| 3060 | { | ||
| 3061 | .pasid_capable = true, | ||
| 3062 | }; | ||
| 3063 | |||
| 3064 | TEST_F(iommufd_device_pasid, pasid_attach) | ||
| 3065 | { | ||
| 3066 | struct iommu_hwpt_selftest data = { | ||
| 3067 | .iotlb = IOMMU_TEST_IOTLB_DEFAULT, | ||
| 3068 | }; | ||
| 3069 | uint32_t nested_hwpt_id[3] = {}; | ||
| 3070 | uint32_t parent_hwpt_id = 0; | ||
| 3071 | uint32_t fault_id, fault_fd; | ||
| 3072 | uint32_t s2_hwpt_id = 0; | ||
| 3073 | uint32_t iopf_hwpt_id; | ||
| 3074 | uint32_t pasid = 100; | ||
| 3075 | uint32_t viommu_id; | ||
| 3076 | |||
| 3077 | /* | ||
| 3078 | * Negative, detach pasid without attaching, this is not expected. | ||
| 3079 | * But it should not result in failure anyway. | ||
| 3080 | */ | ||
| 3081 | test_cmd_pasid_detach(pasid); | ||
| 3082 | |||
| 3083 | /* Allocate two nested hwpts sharing one common parent hwpt */ | ||
| 3084 | test_cmd_hwpt_alloc(self->device_id, self->ioas_id, | ||
| 3085 | IOMMU_HWPT_ALLOC_NEST_PARENT, | ||
| 3086 | &parent_hwpt_id); | ||
| 3087 | test_cmd_hwpt_alloc_nested(self->device_id, parent_hwpt_id, | ||
| 3088 | IOMMU_HWPT_ALLOC_PASID, | ||
| 3089 | &nested_hwpt_id[0], | ||
| 3090 | IOMMU_HWPT_DATA_SELFTEST, | ||
| 3091 | &data, sizeof(data)); | ||
| 3092 | test_cmd_hwpt_alloc_nested(self->device_id, parent_hwpt_id, | ||
| 3093 | IOMMU_HWPT_ALLOC_PASID, | ||
| 3094 | &nested_hwpt_id[1], | ||
| 3095 | IOMMU_HWPT_DATA_SELFTEST, | ||
| 3096 | &data, sizeof(data)); | ||
| 3097 | |||
| 3098 | /* Fault related preparation */ | ||
| 3099 | test_ioctl_fault_alloc(&fault_id, &fault_fd); | ||
| 3100 | test_cmd_hwpt_alloc_iopf(self->device_id, parent_hwpt_id, fault_id, | ||
| 3101 | IOMMU_HWPT_FAULT_ID_VALID | IOMMU_HWPT_ALLOC_PASID, | ||
| 3102 | &iopf_hwpt_id, | ||
| 3103 | IOMMU_HWPT_DATA_SELFTEST, &data, | ||
| 3104 | sizeof(data)); | ||
| 3105 | |||
| 3106 | /* Allocate a regular nested hwpt based on viommu */ | ||
| 3107 | test_cmd_viommu_alloc(self->device_id, parent_hwpt_id, | ||
| 3108 | IOMMU_VIOMMU_TYPE_SELFTEST, | ||
| 3109 | &viommu_id); | ||
| 3110 | test_cmd_hwpt_alloc_nested(self->device_id, viommu_id, | ||
| 3111 | IOMMU_HWPT_ALLOC_PASID, | ||
| 3112 | &nested_hwpt_id[2], | ||
| 3113 | IOMMU_HWPT_DATA_SELFTEST, &data, | ||
| 3114 | sizeof(data)); | ||
| 3115 | |||
| 3116 | test_cmd_hwpt_alloc(self->device_id, self->ioas_id, | ||
| 3117 | IOMMU_HWPT_ALLOC_PASID, | ||
| 3118 | &s2_hwpt_id); | ||
| 3119 | |||
| 3120 | /* Attach RID to non-pasid compat domain, */ | ||
| 3121 | test_cmd_mock_domain_replace(self->stdev_id, parent_hwpt_id); | ||
| 3122 | /* then attach to pasid should fail */ | ||
| 3123 | test_err_pasid_attach(EINVAL, pasid, s2_hwpt_id); | ||
| 3124 | |||
| 3125 | /* Attach RID to pasid compat domain, */ | ||
| 3126 | test_cmd_mock_domain_replace(self->stdev_id, s2_hwpt_id); | ||
| 3127 | /* then attach to pasid should succeed, */ | ||
| 3128 | test_cmd_pasid_attach(pasid, nested_hwpt_id[0]); | ||
| 3129 | /* but attach RID to non-pasid compat domain should fail now. */ | ||
| 3130 | test_err_mock_domain_replace(EINVAL, self->stdev_id, parent_hwpt_id); | ||
| 3131 | /* | ||
| 3132 | * Detach hwpt from pasid 100, and check if the pasid 100 | ||
| 3133 | * has null domain. | ||
| 3134 | */ | ||
| 3135 | test_cmd_pasid_detach(pasid); | ||
| 3136 | ASSERT_EQ(0, | ||
| 3137 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3138 | pasid, 0)); | ||
| 3139 | /* RID is attached to pasid-comapt domain, pasid path is not used */ | ||
| 3140 | |||
| 3141 | if (!variant->pasid_capable) { | ||
| 3142 | /* | ||
| 3143 | * PASID-compatible domain can be used by non-PASID-capable | ||
| 3144 | * device. | ||
| 3145 | */ | ||
| 3146 | test_cmd_mock_domain_replace(self->no_pasid_stdev_id, nested_hwpt_id[0]); | ||
| 3147 | test_cmd_mock_domain_replace(self->no_pasid_stdev_id, self->ioas_id); | ||
| 3148 | /* | ||
| 3149 | * Attach hwpt to pasid 100 of non-PASID-capable device, | ||
| 3150 | * should fail, no matter domain is pasid-comapt or not. | ||
| 3151 | */ | ||
| 3152 | EXPECT_ERRNO(EINVAL, | ||
| 3153 | _test_cmd_pasid_attach(self->fd, self->no_pasid_stdev_id, | ||
| 3154 | pasid, parent_hwpt_id)); | ||
| 3155 | EXPECT_ERRNO(EINVAL, | ||
| 3156 | _test_cmd_pasid_attach(self->fd, self->no_pasid_stdev_id, | ||
| 3157 | pasid, s2_hwpt_id)); | ||
| 3158 | } | ||
| 3159 | |||
| 3160 | /* | ||
| 3161 | * Attach non pasid compat hwpt to pasid-capable device, should | ||
| 3162 | * fail, and have null domain. | ||
| 3163 | */ | ||
| 3164 | test_err_pasid_attach(EINVAL, pasid, parent_hwpt_id); | ||
| 3165 | ASSERT_EQ(0, | ||
| 3166 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3167 | pasid, 0)); | ||
| 3168 | |||
| 3169 | /* | ||
| 3170 | * Attach ioas to pasid 100, should fail, domain should | ||
| 3171 | * be null. | ||
| 3172 | */ | ||
| 3173 | test_err_pasid_attach(EINVAL, pasid, self->ioas_id); | ||
| 3174 | ASSERT_EQ(0, | ||
| 3175 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3176 | pasid, 0)); | ||
| 3177 | |||
| 3178 | /* | ||
| 3179 | * Attach the s2_hwpt to pasid 100, should succeed, domain should | ||
| 3180 | * be valid. | ||
| 3181 | */ | ||
| 3182 | test_cmd_pasid_attach(pasid, s2_hwpt_id); | ||
| 3183 | ASSERT_EQ(0, | ||
| 3184 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3185 | pasid, s2_hwpt_id)); | ||
| 3186 | |||
| 3187 | /* | ||
| 3188 | * Try attach pasid 100 with another hwpt, should FAIL | ||
| 3189 | * as attach does not allow overwrite, use REPLACE instead. | ||
| 3190 | */ | ||
| 3191 | test_err_pasid_attach(EBUSY, pasid, nested_hwpt_id[0]); | ||
| 3192 | |||
| 3193 | /* | ||
| 3194 | * Detach hwpt from pasid 100 for next test, should succeed, | ||
| 3195 | * and have null domain. | ||
| 3196 | */ | ||
| 3197 | test_cmd_pasid_detach(pasid); | ||
| 3198 | ASSERT_EQ(0, | ||
| 3199 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3200 | pasid, 0)); | ||
| 3201 | |||
| 3202 | /* | ||
| 3203 | * Attach nested hwpt to pasid 100, should succeed, domain | ||
| 3204 | * should be valid. | ||
| 3205 | */ | ||
| 3206 | test_cmd_pasid_attach(pasid, nested_hwpt_id[0]); | ||
| 3207 | ASSERT_EQ(0, | ||
| 3208 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3209 | pasid, nested_hwpt_id[0])); | ||
| 3210 | |||
| 3211 | /* Attach to pasid 100 which has been attached, should fail. */ | ||
| 3212 | test_err_pasid_attach(EBUSY, pasid, nested_hwpt_id[0]); | ||
| 3213 | |||
| 3214 | /* cleanup pasid 100 */ | ||
| 3215 | test_cmd_pasid_detach(pasid); | ||
| 3216 | |||
| 3217 | /* Replace tests */ | ||
| 3218 | |||
| 3219 | pasid = 200; | ||
| 3220 | /* | ||
| 3221 | * Replace pasid 200 without attaching it, should fail | ||
| 3222 | * with -EINVAL. | ||
| 3223 | */ | ||
| 3224 | test_err_pasid_replace(EINVAL, pasid, s2_hwpt_id); | ||
| 3225 | |||
| 3226 | /* | ||
| 3227 | * Attach the s2 hwpt to pasid 200, should succeed, domain should | ||
| 3228 | * be valid. | ||
| 3229 | */ | ||
| 3230 | test_cmd_pasid_attach(pasid, s2_hwpt_id); | ||
| 3231 | ASSERT_EQ(0, | ||
| 3232 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3233 | pasid, s2_hwpt_id)); | ||
| 3234 | |||
| 3235 | /* | ||
| 3236 | * Replace pasid 200 with self->ioas_id, should fail | ||
| 3237 | * and domain should be the prior s2 hwpt. | ||
| 3238 | */ | ||
| 3239 | test_err_pasid_replace(EINVAL, pasid, self->ioas_id); | ||
| 3240 | ASSERT_EQ(0, | ||
| 3241 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3242 | pasid, s2_hwpt_id)); | ||
| 3243 | |||
| 3244 | /* | ||
| 3245 | * Replace a nested hwpt for pasid 200, should succeed, | ||
| 3246 | * and have valid domain. | ||
| 3247 | */ | ||
| 3248 | test_cmd_pasid_replace(pasid, nested_hwpt_id[0]); | ||
| 3249 | ASSERT_EQ(0, | ||
| 3250 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3251 | pasid, nested_hwpt_id[0])); | ||
| 3252 | |||
| 3253 | /* | ||
| 3254 | * Replace with another nested hwpt for pasid 200, should | ||
| 3255 | * succeed, and have valid domain. | ||
| 3256 | */ | ||
| 3257 | test_cmd_pasid_replace(pasid, nested_hwpt_id[1]); | ||
| 3258 | ASSERT_EQ(0, | ||
| 3259 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3260 | pasid, nested_hwpt_id[1])); | ||
| 3261 | |||
| 3262 | /* cleanup pasid 200 */ | ||
| 3263 | test_cmd_pasid_detach(pasid); | ||
| 3264 | |||
| 3265 | /* Negative Tests for pasid replace, use pasid 1024 */ | ||
| 3266 | |||
| 3267 | /* | ||
| 3268 | * Attach the s2 hwpt to pasid 1024, should succeed, domain should | ||
| 3269 | * be valid. | ||
| 3270 | */ | ||
| 3271 | pasid = 1024; | ||
| 3272 | test_cmd_pasid_attach(pasid, s2_hwpt_id); | ||
| 3273 | ASSERT_EQ(0, | ||
| 3274 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3275 | pasid, s2_hwpt_id)); | ||
| 3276 | |||
| 3277 | /* | ||
| 3278 | * Replace pasid 1024 with nested_hwpt_id[0], should fail, | ||
| 3279 | * but have the old valid domain. This is a designed | ||
| 3280 | * negative case. Normally, this shall succeed. | ||
| 3281 | */ | ||
| 3282 | test_err_pasid_replace(ENOMEM, pasid, nested_hwpt_id[0]); | ||
| 3283 | ASSERT_EQ(0, | ||
| 3284 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3285 | pasid, s2_hwpt_id)); | ||
| 3286 | |||
| 3287 | /* cleanup pasid 1024 */ | ||
| 3288 | test_cmd_pasid_detach(pasid); | ||
| 3289 | |||
| 3290 | /* Attach to iopf-capable hwpt */ | ||
| 3291 | |||
| 3292 | /* | ||
| 3293 | * Attach an iopf hwpt to pasid 2048, should succeed, domain should | ||
| 3294 | * be valid. | ||
| 3295 | */ | ||
| 3296 | pasid = 2048; | ||
| 3297 | test_cmd_pasid_attach(pasid, iopf_hwpt_id); | ||
| 3298 | ASSERT_EQ(0, | ||
| 3299 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3300 | pasid, iopf_hwpt_id)); | ||
| 3301 | |||
| 3302 | test_cmd_trigger_iopf_pasid(self->device_id, pasid, fault_fd); | ||
| 3303 | |||
| 3304 | /* | ||
| 3305 | * Replace with s2_hwpt_id for pasid 2048, should | ||
| 3306 | * succeed, and have valid domain. | ||
| 3307 | */ | ||
| 3308 | test_cmd_pasid_replace(pasid, s2_hwpt_id); | ||
| 3309 | ASSERT_EQ(0, | ||
| 3310 | test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | ||
| 3311 | pasid, s2_hwpt_id)); | ||
| 3312 | |||
| 3313 | /* cleanup pasid 2048 */ | ||
| 3314 | test_cmd_pasid_detach(pasid); | ||
| 3315 | |||
| 3316 | test_ioctl_destroy(iopf_hwpt_id); | ||
| 3317 | close(fault_fd); | ||
| 3318 | test_ioctl_destroy(fault_id); | ||
| 3319 | |||
| 3320 | /* Detach the s2_hwpt_id from RID */ | ||
| 3321 | test_cmd_mock_domain_replace(self->stdev_id, self->ioas_id); | ||
| 3322 | } | ||
| 3323 | |||
| 2959 | TEST_HARNESS_MAIN | 3324 | TEST_HARNESS_MAIN |
| @@ -209,12 +209,16 @@ FIXTURE(basic_fail_nth) · 208 unmodified lines | |||
| 209 | { | 209 | { |
| 210 | int fd; | 210 | int fd; |
| 211 | uint32_t access_id; | 211 | uint32_t access_id; |
| 212 | uint32_t stdev_id; | ||
| 213 | uint32_t pasid; | ||
| 212 | }; | 214 | }; |
| 213 | 215 | ||
| 214 | FIXTURE_SETUP(basic_fail_nth) | 216 | FIXTURE_SETUP(basic_fail_nth) |
| 215 | { | 217 | { |
| 216 | self->fd = -1; | 218 | self->fd = -1; |
| 217 | self->access_id = 0; | 219 | self->access_id = 0; |
| 220 | self->stdev_id = 0; | ||
| 221 | self->pasid = 0; //test should use a non-zero value | ||
| 218 | } | 222 | } |
| 219 | 223 | ||
| 220 | FIXTURE_TEARDOWN(basic_fail_nth) | 224 | FIXTURE_TEARDOWN(basic_fail_nth) |
| @@ -226,6 +230,8 @@ FIXTURE_TEARDOWN(basic_fail_nth) · 5 unmodified lines | |||
| 226 | rc = _test_cmd_destroy_access(self->access_id); | 230 | rc = _test_cmd_destroy_access(self->access_id); |
| 227 | assert(rc == 0); | 231 | assert(rc == 0); |
| 228 | } | 232 | } |
| 233 | if (self->pasid && self->stdev_id) | ||
| 234 | _test_cmd_pasid_detach(self->fd, self->stdev_id, self->pasid); | ||
| 229 | teardown_iommufd(self->fd, _metadata); | 235 | teardown_iommufd(self->fd, _metadata); |
| 230 | } | 236 | } |
| 231 | 237 | ||
| @@ -620,10 +626,11 @@ TEST_FAIL_NTH(basic_fail_nth, device) · 388 unmodified lines | |||
| 620 | }; | 626 | }; |
| 621 | struct iommu_test_hw_info info; | 627 | struct iommu_test_hw_info info; |
| 622 | uint32_t fault_id, fault_fd; | 628 | uint32_t fault_id, fault_fd; |
| 629 | uint32_t veventq_id, veventq_fd; | ||
| 623 | uint32_t fault_hwpt_id; | 630 | uint32_t fault_hwpt_id; |
| 631 | uint32_t test_hwpt_id; | ||
| 624 | uint32_t ioas_id; | 632 | uint32_t ioas_id; |
| 625 | uint32_t ioas_id2; | 633 | uint32_t ioas_id2; |
| 626 | uint32_t stdev_id; | ||
| 627 | uint32_t idev_id; | 634 | uint32_t idev_id; |
| 628 | uint32_t hwpt_id; | 635 | uint32_t hwpt_id; |
| 629 | uint32_t viommu_id; | 636 | uint32_t viommu_id; |
| @@ -654,25 +661,30 @@ TEST_FAIL_NTH(basic_fail_nth, device) · 24 unmodified lines | |||
| 654 | 661 | ||
| 655 | fail_nth_enable(); | 662 | fail_nth_enable(); |
| 656 | 663 | ||
| 657 | if (_test_cmd_mock_domain(self->fd, ioas_id, &stdev_id, NULL, | 664 | if (_test_cmd_mock_domain_flags(self->fd, ioas_id, |
| 658 | &idev_id)) | 665 | MOCK_FLAGS_DEVICE_PASID, |
| 666 | &self->stdev_id, NULL, &idev_id)) | ||
| 659 | return -1; | 667 | return -1; |
| 660 | 668 | ||
| 661 | if (_test_cmd_get_hw_info(self->fd, idev_id, &info, sizeof(info), NULL)) | 669 | if (_test_cmd_get_hw_info(self->fd, idev_id, &info, |
| 670 | sizeof(info), NULL, NULL)) | ||
| 662 | return -1; | 671 | return -1; |
| 663 | 672 | ||
| 664 | if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0, 0, &hwpt_id, | 673 | if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0, |
| 674 | IOMMU_HWPT_ALLOC_PASID, &hwpt_id, | ||
| 665 | IOMMU_HWPT_DATA_NONE, 0, 0)) | 675 | IOMMU_HWPT_DATA_NONE, 0, 0)) |
| 666 | return -1; | 676 | return -1; |
| 667 | 677 | ||
| 668 | if (_test_cmd_mock_domain_replace(self->fd, stdev_id, ioas_id2, NULL)) | 678 | if (_test_cmd_mock_domain_replace(self->fd, self->stdev_id, ioas_id2, NULL)) |
| 669 | return -1; | 679 | return -1; |
| 670 | 680 | ||
| 671 | if (_test_cmd_mock_domain_replace(self->fd, stdev_id, hwpt_id, NULL)) | 681 | if (_test_cmd_mock_domain_replace(self->fd, self->stdev_id, hwpt_id, NULL)) |
| 672 | return -1; | 682 | return -1; |
| 673 | 683 | ||
| 674 | if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0, | 684 | if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0, |
| 675 | IOMMU_HWPT_ALLOC_NEST_PARENT, &hwpt_id, | 685 | IOMMU_HWPT_ALLOC_NEST_PARENT | |
| 686 | IOMMU_HWPT_ALLOC_PASID, | ||
| 687 | &hwpt_id, | ||
| 676 | IOMMU_HWPT_DATA_NONE, 0, 0)) | 688 | IOMMU_HWPT_DATA_NONE, 0, 0)) |
| 677 | return -1; | 689 | return -1; |
| 678 | 690 | ||
| @@ -692,6 +704,37 @@ TEST_FAIL_NTH(basic_fail_nth, device) · 13 unmodified lines | |||
| 692 | IOMMU_HWPT_DATA_SELFTEST, &data, sizeof(data))) | 704 | IOMMU_HWPT_DATA_SELFTEST, &data, sizeof(data))) |
| 693 | return -1; | 705 | return -1; |
| 694 | 706 | ||
| 707 | if (_test_cmd_veventq_alloc(self->fd, viommu_id, | ||
| 708 | IOMMU_VEVENTQ_TYPE_SELFTEST, &veventq_id, | ||
| 709 | &veventq_fd)) | ||
| 710 | return -1; | ||
| 711 | close(veventq_fd); | ||
| 712 | |||
| 713 | if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0, | ||
| 714 | IOMMU_HWPT_ALLOC_PASID, | ||
| 715 | &test_hwpt_id, | ||
| 716 | IOMMU_HWPT_DATA_NONE, 0, 0)) | ||
| 717 | return -1; | ||
| 718 | |||
| 719 | /* Tests for pasid attach/replace/detach */ | ||
| 720 | |||
| 721 | self->pasid = 200; | ||
| 722 | |||
| 723 | if (_test_cmd_pasid_attach(self->fd, self->stdev_id, | ||
| 724 | self->pasid, hwpt_id)) { | ||
| 725 | self->pasid = 0; | ||
| 726 | return -1; | ||
| 727 | } | ||
| 728 | |||
| 729 | if (_test_cmd_pasid_replace(self->fd, self->stdev_id, | ||
| 730 | self->pasid, test_hwpt_id)) | ||
| 731 | return -1; | ||
| 732 | |||
| 733 | if (_test_cmd_pasid_detach(self->fd, self->stdev_id, self->pasid)) | ||
| 734 | return -1; | ||
| 735 | |||
| 736 | self->pasid = 0; | ||
| 737 | |||
| 695 | return 0; | 738 | return 0; |
| 696 | } | 739 | } |
| 697 | 740 | ||
| @@ -9,6 +9,7 @@ · 8 unmodified lines | |||
| 9 | #include <sys/ioctl.h> | 9 | #include <sys/ioctl.h> |
| 10 | #include <stdint.h> | 10 | #include <stdint.h> |
| 11 | #include <assert.h> | 11 | #include <assert.h> |
| 12 | #include <poll.h> | ||
| 12 | 13 | ||
| 13 | #include "../kselftest_harness.h" | 14 | #include "../kselftest_harness.h" |
| 14 | #include "../../../../drivers/iommu/iommufd/iommufd_test.h" | 15 | #include "../../../../drivers/iommu/iommufd/iommufd_test.h" |
| @@ -757,7 +758,8 @@ static void teardown_iommufd(int fd, struct __test_metadata *_metadata) · 742 unmodified lines | |||
| 757 | 758 | ||
| 758 | /* @data can be NULL */ | 759 | /* @data can be NULL */ |
| 759 | static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data, | 760 | static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data, |
| 760 | size_t data_len, uint32_t *capabilities) | 761 | size_t data_len, uint32_t *capabilities, |
| 762 | uint8_t *max_pasid) | ||
| 761 | { | 763 | { |
| 762 | struct iommu_test_hw_info *info = (struct iommu_test_hw_info *)data; | 764 | struct iommu_test_hw_info *info = (struct iommu_test_hw_info *)data; |
| 763 | struct iommu_hw_info cmd = { | 765 | struct iommu_hw_info cmd = { |
| @@ -802,6 +804,9 @@ static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data, · 38 unmodified lines | |||
| 802 | assert(!info->flags); | 804 | assert(!info->flags); |
| 803 | } | 805 | } |
| 804 | 806 | ||
| 807 | if (max_pasid) | ||
| 808 | *max_pasid = cmd.out_max_pasid_log2; | ||
| 809 | |||
| 805 | if (capabilities) | 810 | if (capabilities) |
| 806 | *capabilities = cmd.out_capabilities; | 811 | *capabilities = cmd.out_capabilities; |
| 807 | 812 | ||
| @@ -810,14 +815,19 @@ static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data, · 2 unmodified lines | |||
| 810 | 815 | ||
| 811 | #define test_cmd_get_hw_info(device_id, data, data_len) \ | 816 | #define test_cmd_get_hw_info(device_id, data, data_len) \ |
| 812 | ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, data, \ | 817 | ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, data, \ |
| 813 | data_len, NULL)) | 818 | data_len, NULL, NULL)) |
| 814 | 819 | ||
| 815 | #define test_err_get_hw_info(_errno, device_id, data, data_len) \ | 820 | #define test_err_get_hw_info(_errno, device_id, data, data_len) \ |
| 816 | EXPECT_ERRNO(_errno, _test_cmd_get_hw_info(self->fd, device_id, data, \ | 821 | EXPECT_ERRNO(_errno, _test_cmd_get_hw_info(self->fd, device_id, data, \ |
| 817 | data_len, NULL)) | 822 | data_len, NULL, NULL)) |
| 818 | 823 | ||
| 819 | #define test_cmd_get_hw_capabilities(device_id, caps, mask) \ | 824 | #define test_cmd_get_hw_capabilities(device_id, caps, mask) \ |
| 820 | ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, NULL, 0, &caps)) | 825 | ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, NULL, \ |
| 826 | 0, &caps, NULL)) | ||
| 827 | |||
| 828 | #define test_cmd_get_hw_info_pasid(device_id, max_pasid) \ | ||
| 829 | ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, NULL, \ | ||
| 830 | 0, NULL, max_pasid)) | ||
| 821 | 831 | ||
| 822 | static int _test_ioctl_fault_alloc(int fd, __u32 *fault_id, __u32 *fault_fd) | 832 | static int _test_ioctl_fault_alloc(int fd, __u32 *fault_id, __u32 *fault_fd) |
| 823 | { | 833 | { |
| @@ -842,14 +852,15 @@ static int _test_ioctl_fault_alloc(int fd, __u32 *fault_id, __u32 *fault_fd) · 18 unmodified lines | |||
| 842 | ASSERT_NE(0, *(fault_fd)); \ | 852 | ASSERT_NE(0, *(fault_fd)); \ |
| 843 | }) | 853 | }) |
| 844 | 854 | ||
| 845 | static int _test_cmd_trigger_iopf(int fd, __u32 device_id, __u32 fault_fd) | 855 | static int _test_cmd_trigger_iopf(int fd, __u32 device_id, __u32 pasid, |
| 856 | __u32 fault_fd) | ||
| 846 | { | 857 | { |
| 847 | struct iommu_test_cmd trigger_iopf_cmd = { | 858 | struct iommu_test_cmd trigger_iopf_cmd = { |
| 848 | .size = sizeof(trigger_iopf_cmd), | 859 | .size = sizeof(trigger_iopf_cmd), |
| 849 | .op = IOMMU_TEST_OP_TRIGGER_IOPF, | 860 | .op = IOMMU_TEST_OP_TRIGGER_IOPF, |
| 850 | .trigger_iopf = { | 861 | .trigger_iopf = { |
| 851 | .dev_id = device_id, | 862 | .dev_id = device_id, |
| 852 | .pasid = 0x1, | 863 | .pasid = pasid, |
| 853 | .grpid = 0x2, | 864 | .grpid = 0x2, |
| 854 | .perm = IOMMU_PGFAULT_PERM_READ | IOMMU_PGFAULT_PERM_WRITE, | 865 | .perm = IOMMU_PGFAULT_PERM_READ | IOMMU_PGFAULT_PERM_WRITE, |
| 855 | .addr = 0xdeadbeaf, | 866 | .addr = 0xdeadbeaf, |
| @@ -880,7 +891,10 @@ static int _test_cmd_trigger_iopf(int fd, __u32 device_id, __u32 fault_fd) · 24 unmodified lines | |||
| 880 | } | 891 | } |
| 881 | 892 | ||
| 882 | #define test_cmd_trigger_iopf(device_id, fault_fd) \ | 893 | #define test_cmd_trigger_iopf(device_id, fault_fd) \ |
| 883 | ASSERT_EQ(0, _test_cmd_trigger_iopf(self->fd, device_id, fault_fd)) | 894 | ASSERT_EQ(0, _test_cmd_trigger_iopf(self->fd, device_id, 0x1, fault_fd)) |
| 895 | #define test_cmd_trigger_iopf_pasid(device_id, pasid, fault_fd) \ | ||
| 896 | ASSERT_EQ(0, _test_cmd_trigger_iopf(self->fd, device_id, \ | ||
| 897 | pasid, fault_fd)) | ||
| 884 | 898 | ||
| 885 | static int _test_cmd_viommu_alloc(int fd, __u32 device_id, __u32 hwpt_id, | 899 | static int _test_cmd_viommu_alloc(int fd, __u32 device_id, __u32 hwpt_id, |
| 886 | __u32 type, __u32 flags, __u32 *viommu_id) | 900 | __u32 type, __u32 flags, __u32 *viommu_id) |
| @@ -936,3 +950,204 @@ static int _test_cmd_vdevice_alloc(int fd, __u32 viommu_id, __u32 idev_id, · 49 unmodified lines | |||
| 936 | EXPECT_ERRNO(_errno, \ | 950 | EXPECT_ERRNO(_errno, \ |
| 937 | _test_cmd_vdevice_alloc(self->fd, viommu_id, idev_id, \ | 951 | _test_cmd_vdevice_alloc(self->fd, viommu_id, idev_id, \ |
| 938 | virt_id, vdev_id)) | 952 | virt_id, vdev_id)) |
| 953 | |||
| 954 | static int _test_cmd_veventq_alloc(int fd, __u32 viommu_id, __u32 type, | ||
| 955 | __u32 *veventq_id, __u32 *veventq_fd) | ||
| 956 | { | ||
| 957 | struct iommu_veventq_alloc cmd = { | ||
| 958 | .size = sizeof(cmd), | ||
| 959 | .type = type, | ||
| 960 | .veventq_depth = 2, | ||
| 961 | .viommu_id = viommu_id, | ||
| 962 | }; | ||
| 963 | int ret; | ||
| 964 | |||
| 965 | ret = ioctl(fd, IOMMU_VEVENTQ_ALLOC, &cmd); | ||
| 966 | if (ret) | ||
| 967 | return ret; | ||
| 968 | if (veventq_id) | ||
| 969 | *veventq_id = cmd.out_veventq_id; | ||
| 970 | if (veventq_fd) | ||
| 971 | *veventq_fd = cmd.out_veventq_fd; | ||
| 972 | return 0; | ||
| 973 | } | ||
| 974 | |||
| 975 | #define test_cmd_veventq_alloc(viommu_id, type, veventq_id, veventq_fd) \ | ||
| 976 | ASSERT_EQ(0, _test_cmd_veventq_alloc(self->fd, viommu_id, type, \ | ||
| 977 | veventq_id, veventq_fd)) | ||
| 978 | #define test_err_veventq_alloc(_errno, viommu_id, type, veventq_id, \ | ||
| 979 | veventq_fd) \ | ||
| 980 | EXPECT_ERRNO(_errno, \ | ||
| 981 | _test_cmd_veventq_alloc(self->fd, viommu_id, type, \ | ||
| 982 | veventq_id, veventq_fd)) | ||
| 983 | |||
| 984 | static int _test_cmd_trigger_vevents(int fd, __u32 dev_id, __u32 nvevents) | ||
| 985 | { | ||
| 986 | struct iommu_test_cmd trigger_vevent_cmd = { | ||
| 987 | .size = sizeof(trigger_vevent_cmd), | ||
| 988 | .op = IOMMU_TEST_OP_TRIGGER_VEVENT, | ||
| 989 | .trigger_vevent = { | ||
| 990 | .dev_id = dev_id, | ||
| 991 | }, | ||
| 992 | }; | ||
| 993 | int ret; | ||
| 994 | |||
| 995 | while (nvevents--) { | ||
| 996 | ret = ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_TRIGGER_VEVENT), | ||
| 997 | &trigger_vevent_cmd); | ||
| 998 | if (ret < 0) | ||
| 999 | return -1; | ||
| 1000 | } | ||
| 1001 | return ret; | ||
| 1002 | } | ||
| 1003 | |||
| 1004 | #define test_cmd_trigger_vevents(dev_id, nvevents) \ | ||
| 1005 | ASSERT_EQ(0, _test_cmd_trigger_vevents(self->fd, dev_id, nvevents)) | ||
| 1006 | |||
| 1007 | static int _test_cmd_read_vevents(int fd, __u32 event_fd, __u32 nvevents, | ||
| 1008 | __u32 virt_id, int *prev_seq) | ||
| 1009 | { | ||
| 1010 | struct pollfd pollfd = { .fd = event_fd, .events = POLLIN }; | ||
| 1011 | struct iommu_viommu_event_selftest *event; | ||
| 1012 | struct iommufd_vevent_header *hdr; | ||
| 1013 | ssize_t bytes; | ||
| 1014 | void *data; | ||
| 1015 | int ret, i; | ||
| 1016 | |||
| 1017 | ret = poll(&pollfd, 1, 1000); | ||
| 1018 | if (ret < 0) | ||
| 1019 | return -1; | ||
| 1020 | |||
| 1021 | data = calloc(nvevents, sizeof(*hdr) + sizeof(*event)); | ||
| 1022 | if (!data) { | ||
| 1023 | errno = ENOMEM; | ||
| 1024 | return -1; | ||
| 1025 | } | ||
| 1026 | |||
| 1027 | bytes = read(event_fd, data, | ||
| 1028 | nvevents * (sizeof(*hdr) + sizeof(*event))); | ||
| 1029 | if (bytes <= 0) { | ||
| 1030 | errno = EFAULT; | ||
| 1031 | ret = -1; | ||
| 1032 | goto out_free; | ||
| 1033 | } | ||
| 1034 | |||
| 1035 | for (i = 0; i < nvevents; i++) { | ||
| 1036 | hdr = data + i * (sizeof(*hdr) + sizeof(*event)); | ||
| 1037 | |||
| 1038 | if (hdr->flags & IOMMU_VEVENTQ_FLAG_LOST_EVENTS || | ||
| 1039 | hdr->sequence - *prev_seq > 1) { | ||
| 1040 | *prev_seq = hdr->sequence; | ||
| 1041 | errno = EOVERFLOW; | ||
| 1042 | ret = -1; | ||
| 1043 | goto out_free; | ||
| 1044 | } | ||
| 1045 | *prev_seq = hdr->sequence; | ||
| 1046 | event = data + sizeof(*hdr); | ||
| 1047 | if (event->virt_id != virt_id) { | ||
| 1048 | errno = EINVAL; | ||
| 1049 | ret = -1; | ||
| 1050 | goto out_free; | ||
| 1051 | } | ||
| 1052 | } | ||
| 1053 | |||
| 1054 | ret = 0; | ||
| 1055 | out_free: | ||
| 1056 | free(data); | ||
| 1057 | return ret; | ||
| 1058 | } | ||
| 1059 | |||
| 1060 | #define test_cmd_read_vevents(event_fd, nvevents, virt_id, prev_seq) \ | ||
| 1061 | ASSERT_EQ(0, _test_cmd_read_vevents(self->fd, event_fd, nvevents, \ | ||
| 1062 | virt_id, prev_seq)) | ||
| 1063 | #define test_err_read_vevents(_errno, event_fd, nvevents, virt_id, prev_seq) \ | ||
| 1064 | EXPECT_ERRNO(_errno, \ | ||
| 1065 | _test_cmd_read_vevents(self->fd, event_fd, nvevents, \ | ||
| 1066 | virt_id, prev_seq)) | ||
| 1067 | |||
| 1068 | static int _test_cmd_pasid_attach(int fd, __u32 stdev_id, __u32 pasid, | ||
| 1069 | __u32 pt_id) | ||
| 1070 | { | ||
| 1071 | struct iommu_test_cmd test_attach = { | ||
| 1072 | .size = sizeof(test_attach), | ||
| 1073 | .op = IOMMU_TEST_OP_PASID_ATTACH, | ||
| 1074 | .id = stdev_id, | ||
| 1075 | .pasid_attach = { | ||
| 1076 | .pasid = pasid, | ||
| 1077 | .pt_id = pt_id, | ||
| 1078 | }, | ||
| 1079 | }; | ||
| 1080 | |||
| 1081 | return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_ATTACH), | ||
| 1082 | &test_attach); | ||
| 1083 | } | ||
| 1084 | |||
| 1085 | #define test_cmd_pasid_attach(pasid, hwpt_id) \ | ||
| 1086 | ASSERT_EQ(0, _test_cmd_pasid_attach(self->fd, self->stdev_id, \ | ||
| 1087 | pasid, hwpt_id)) | ||
| 1088 | |||
| 1089 | #define test_err_pasid_attach(_errno, pasid, hwpt_id) \ | ||
| 1090 | EXPECT_ERRNO(_errno, \ | ||
| 1091 | _test_cmd_pasid_attach(self->fd, self->stdev_id, \ | ||
| 1092 | pasid, hwpt_id)) | ||
| 1093 | |||
| 1094 | static int _test_cmd_pasid_replace(int fd, __u32 stdev_id, __u32 pasid, | ||
| 1095 | __u32 pt_id) | ||
| 1096 | { | ||
| 1097 | struct iommu_test_cmd test_replace = { | ||
| 1098 | .size = sizeof(test_replace), | ||
| 1099 | .op = IOMMU_TEST_OP_PASID_REPLACE, | ||
| 1100 | .id = stdev_id, | ||
| 1101 | .pasid_replace = { | ||
| 1102 | .pasid = pasid, | ||
| 1103 | .pt_id = pt_id, | ||
| 1104 | }, | ||
| 1105 | }; | ||
| 1106 | |||
| 1107 | return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_REPLACE), | ||
| 1108 | &test_replace); | ||
| 1109 | } | ||
| 1110 | |||
| 1111 | #define test_cmd_pasid_replace(pasid, hwpt_id) \ | ||
| 1112 | ASSERT_EQ(0, _test_cmd_pasid_replace(self->fd, self->stdev_id, \ | ||
| 1113 | pasid, hwpt_id)) | ||
| 1114 | |||
| 1115 | #define test_err_pasid_replace(_errno, pasid, hwpt_id) \ | ||
| 1116 | EXPECT_ERRNO(_errno, \ | ||
| 1117 | _test_cmd_pasid_replace(self->fd, self->stdev_id, \ | ||
| 1118 | pasid, hwpt_id)) | ||
| 1119 | |||
| 1120 | static int _test_cmd_pasid_detach(int fd, __u32 stdev_id, __u32 pasid) | ||
| 1121 | { | ||
| 1122 | struct iommu_test_cmd test_detach = { | ||
| 1123 | .size = sizeof(test_detach), | ||
| 1124 | .op = IOMMU_TEST_OP_PASID_DETACH, | ||
| 1125 | .id = stdev_id, | ||
| 1126 | .pasid_detach = { | ||
| 1127 | .pasid = pasid, | ||
| 1128 | }, | ||
| 1129 | }; | ||
| 1130 | |||
| 1131 | return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_DETACH), | ||
| 1132 | &test_detach); | ||
| 1133 | } | ||
| 1134 | |||
| 1135 | #define test_cmd_pasid_detach(pasid) \ | ||
| 1136 | ASSERT_EQ(0, _test_cmd_pasid_detach(self->fd, self->stdev_id, pasid)) | ||
| 1137 | |||
| 1138 | static int test_cmd_pasid_check_hwpt(int fd, __u32 stdev_id, __u32 pasid, | ||
| 1139 | __u32 hwpt_id) | ||
| 1140 | { | ||
| 1141 | struct iommu_test_cmd test_pasid_check = { | ||
| 1142 | .size = sizeof(test_pasid_check), | ||
| 1143 | .op = IOMMU_TEST_OP_PASID_CHECK_HWPT, | ||
| 1144 | .id = stdev_id, | ||
| 1145 | .pasid_check = { | ||
| 1146 | .pasid = pasid, | ||
| 1147 | .hwpt_id = hwpt_id, | ||
| 1148 | }, | ||
| 1149 | }; | ||
| 1150 | |||
| 1151 | return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_CHECK_HWPT), | ||
| 1152 | &test_pasid_check); | ||
| 1153 | } | ||