Pull iommufd updates from Jason Gunthorpe: "Two significant new items: - Allow reporting IOMMU HW events to userspace when the events are clearly linked to a device. This is linked to the VIOMMU object and is intended to be used by a VMM to forward HW events to the virtual machine as part of emulating a vIOMMU. ARM SMMUv3 is the first driver to use this mechanism. Like the existing fault events the data is delivered through a simple FD returning event records on read(). - PASID support in VFIO. The "Process Address Space ID" is a PCI feature that allows the device to tag all PCI DMA operations with an ID. The IOMMU will then use the ID to select a unique translation for those DMAs. This is part of Intel's vIOMMU support as VT-D HW requires the hypervisor to manage each PASID entry. The support is generic so any VFIO user could attach any translation to a PASID, and the support should work on ARM SMMUv3 as well. AMD requires additional driver work. Some minor updates, along with fixes: - Prevent using nested parents with fault's, no driver support today - Put a single "cookie_type" value in the iommu_domain to indicate what owns the various opaque owner fields" * tag 'for-linus-iommufd' of git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd: (49 commits) iommufd: Test attach before detaching pasid iommufd: Fix iommu_vevent_header tables markup iommu: Convert unreachable() to BUG() iommufd: Balance veventq->num_events inc/dec iommufd: Initialize the flags of vevent in iommufd_viommu_report_event() iommufd/selftest: Add coverage for reporting max_pasid_log2 via IOMMU_HW_INFO iommufd: Extend IOMMU_GET_HW_INFO to report PASID capability vfio: VFIO_DEVICE_[AT|DE]TACH_IOMMUFD_PT support pasid vfio-iommufd: Support pasid [at|de]tach for physical VFIO devices ida: Add ida_find_first_range() iommufd/selftest: Add coverage for iommufd pasid attach/detach iommufd/selftest: Add test ops to test pasid attach/detach iommufd/selftest: Add a helper to get test device iommufd/selftest: Add set_dev_pasid in mock iommu iommufd: Allow allocating PASID-compatible domain iommu/vt-d: Add IOMMU_HWPT_ALLOC_PASID support iommufd: Enforce PASID-compatible domain for RID iommufd: Support pasid attach/replace iommufd: Enforce PASID-compatible domain in PASID path iommufd/device: Add pasid_attach array to track per-PASID attach ...
Unified · Split
| @@ -63,6 +63,13 @@ Following IOMMUFD objects are exposed to userspace: · 62 unmodified lines | ||
| 63 | 63 | space usually has mappings from guest-level I/O virtual addresses to guest- |
| 64 | 64 | level physical addresses. |
| 65 | 65 | |
| 66 | +- IOMMUFD_FAULT, representing a software queue for an HWPT reporting IO page | |
| 67 | + faults using the IOMMU HW's PRI (Page Request Interface). This queue object | |
| 68 | + provides user space an FD to poll the page fault events and also to respond | |
| 69 | + to those events. A FAULT object must be created first to get a fault_id that | |
| 70 | + could be then used to allocate a fault-enabled HWPT via the IOMMU_HWPT_ALLOC | |
| 71 | + command by setting the IOMMU_HWPT_FAULT_ID_VALID bit in its flags field. | |
| 72 | + | |
| 66 | 73 | - IOMMUFD_OBJ_VIOMMU, representing a slice of the physical IOMMU instance, |
| 67 | 74 | passed to or shared with a VM. It may be some HW-accelerated virtualization |
| 68 | 75 | features and some SW resources used by the VM. For examples: |
| @@ -109,6 +116,14 @@ Following IOMMUFD objects are exposed to userspace: · 40 unmodified lines | ||
| 109 | 116 | vIOMMU, which is a separate ioctl call from attaching the same device to an |
| 110 | 117 | HWPT_PAGING that the vIOMMU holds. |
| 111 | 118 | |
| 119 | +- IOMMUFD_OBJ_VEVENTQ, representing a software queue for a vIOMMU to report its | |
| 120 | + events such as translation faults occurred to a nested stage-1 (excluding I/O | |
| 121 | + page faults that should go through IOMMUFD_OBJ_FAULT) and HW-specific events. | |
| 122 | + This queue object provides user space an FD to poll/read the vIOMMU events. A | |
| 123 | + vIOMMU object must be created first to get its viommu_id, which could be then | |
| 124 | + used to allocate a vEVENTQ. Each vIOMMU can support multiple types of vEVENTS, | |
| 125 | + but is confined to one vEVENTQ per vEVENTQ type. | |
| 126 | + | |
| 112 | 127 | All user-visible objects are destroyed via the IOMMU_DESTROY uAPI. |
| 113 | 128 | |
| 114 | 129 | The diagrams below show relationships between user-visible objects and kernel |
| @@ -251,8 +266,10 @@ User visible objects are backed by following datastructures: · 136 unmodified lines | ||
| 251 | 266 | - iommufd_device for IOMMUFD_OBJ_DEVICE. |
| 252 | 267 | - iommufd_hwpt_paging for IOMMUFD_OBJ_HWPT_PAGING. |
| 253 | 268 | - iommufd_hwpt_nested for IOMMUFD_OBJ_HWPT_NESTED. |
| 269 | +- iommufd_fault for IOMMUFD_OBJ_FAULT. | |
| 254 | 270 | - iommufd_viommu for IOMMUFD_OBJ_VIOMMU. |
| 255 | 271 | - iommufd_vdevice for IOMMUFD_OBJ_VDEVICE. |
| 272 | +- iommufd_veventq for IOMMUFD_OBJ_VEVENTQ. | |
| 256 | 273 | |
| 257 | 274 | Several terminologies when looking at these datastructures: |
| 258 | 275 | |
| @@ -43,6 +43,8 @@ static void arm_smmu_make_nested_cd_table_ste( · 42 unmodified lines | ||
| 43 | 43 | target->data[0] |= nested_domain->ste[0] & |
| 44 | 44 | ~cpu_to_le64(STRTAB_STE_0_CFG); |
| 45 | 45 | target->data[1] |= nested_domain->ste[1]; |
| 46 | + /* Merge events for DoS mitigations on eventq */ | |
| 47 | + target->data[1] |= cpu_to_le64(STRTAB_STE_1_MEV); | |
| 46 | 48 | } |
| 47 | 49 | |
| 48 | 50 | /* |
| @@ -85,6 +87,47 @@ static void arm_smmu_make_nested_domain_ste( · 36 unmodified lines | ||
| 85 | 87 | } |
| 86 | 88 | } |
| 87 | 89 | |
| 90 | +int arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state, | |
| 91 | + struct arm_smmu_nested_domain *nested_domain) | |
| 92 | +{ | |
| 93 | + struct arm_smmu_vmaster *vmaster; | |
| 94 | + unsigned long vsid; | |
| 95 | + int ret; | |
| 96 | + | |
| 97 | + iommu_group_mutex_assert(state->master->dev); | |
| 98 | + | |
| 99 | + ret = iommufd_viommu_get_vdev_id(&nested_domain->vsmmu->core, | |
| 100 | + state->master->dev, &vsid); | |
| 101 | + if (ret) | |
| 102 | + return ret; | |
| 103 | + | |
| 104 | + vmaster = kzalloc(sizeof(*vmaster), GFP_KERNEL); | |
| 105 | + if (!vmaster) | |
| 106 | + return -ENOMEM; | |
| 107 | + vmaster->vsmmu = nested_domain->vsmmu; | |
| 108 | + vmaster->vsid = vsid; | |
| 109 | + state->vmaster = vmaster; | |
| 110 | + | |
| 111 | + return 0; | |
| 112 | +} | |
| 113 | + | |
| 114 | +void arm_smmu_attach_commit_vmaster(struct arm_smmu_attach_state *state) | |
| 115 | +{ | |
| 116 | + struct arm_smmu_master *master = state->master; | |
| 117 | + | |
| 118 | + mutex_lock(&master->smmu->streams_mutex); | |
| 119 | + kfree(master->vmaster); | |
| 120 | + master->vmaster = state->vmaster; | |
| 121 | + mutex_unlock(&master->smmu->streams_mutex); | |
| 122 | +} | |
| 123 | + | |
| 124 | +void arm_smmu_master_clear_vmaster(struct arm_smmu_master *master) | |
| 125 | +{ | |
| 126 | + struct arm_smmu_attach_state state = { .master = master }; | |
| 127 | + | |
| 128 | + arm_smmu_attach_commit_vmaster(&state); | |
| 129 | +} | |
| 130 | + | |
| 88 | 131 | static int arm_smmu_attach_dev_nested(struct iommu_domain *domain, |
| 89 | 132 | struct device *dev) |
| 90 | 133 | { |
| @@ -392,4 +435,21 @@ struct iommufd_viommu *arm_vsmmu_alloc(struct device *dev, · 301 unmodified lines | ||
| 392 | 435 | return &vsmmu->core; |
| 393 | 436 | } |
| 394 | 437 | |
| 438 | +int arm_vmaster_report_event(struct arm_smmu_vmaster *vmaster, u64 *evt) | |
| 439 | +{ | |
| 440 | + struct iommu_vevent_arm_smmuv3 vevt; | |
| 441 | + int i; | |
| 442 | + | |
| 443 | + lockdep_assert_held(&vmaster->vsmmu->smmu->streams_mutex); | |
| 444 | + | |
| 445 | + vevt.evt[0] = cpu_to_le64((evt[0] & ~EVTQ_0_SID) | | |
| 446 | + FIELD_PREP(EVTQ_0_SID, vmaster->vsid)); | |
| 447 | + for (i = 1; i < EVTQ_ENT_DWORDS; i++) | |
| 448 | + vevt.evt[i] = cpu_to_le64(evt[i]); | |
| 449 | + | |
| 450 | + return iommufd_viommu_report_event(&vmaster->vsmmu->core, | |
| 451 | + IOMMU_VEVENTQ_TYPE_ARM_SMMUV3, &vevt, | |
| 452 | + sizeof(vevt)); | |
| 453 | +} | |
| 454 | + | |
| 395 | 455 | MODULE_IMPORT_NS("IOMMUFD"); |
| @@ -1052,7 +1052,7 @@ void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) · 1051 unmodified lines | ||
| 1052 | 1052 | cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR | |
| 1053 | 1053 | STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH | |
| 1054 | 1054 | STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW | |
| 1055 | − STRTAB_STE_1_EATS); | |
| 1055 | + STRTAB_STE_1_EATS | STRTAB_STE_1_MEV); | |
| 1056 | 1056 | used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID); |
| 1057 | 1057 | |
| 1058 | 1058 | /* |
| @@ -1068,7 +1068,7 @@ void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) · 9 unmodified lines | ||
| 1068 | 1068 | if (cfg & BIT(1)) { |
| 1069 | 1069 | used_bits[1] |= |
| 1070 | 1070 | cpu_to_le64(STRTAB_STE_1_S2FWB | STRTAB_STE_1_EATS | |
| 1071 | − STRTAB_STE_1_SHCFG); | |
| 1071 | + STRTAB_STE_1_SHCFG | STRTAB_STE_1_MEV); | |
| 1072 | 1072 | used_bits[2] |= |
| 1073 | 1073 | cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR | |
| 1074 | 1074 | STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI | |
| @@ -1813,8 +1813,8 @@ static void arm_smmu_decode_event(struct arm_smmu_device *smmu, u64 *raw, · 738 unmodified lines | ||
| 1813 | 1813 | mutex_unlock(&smmu->streams_mutex); |
| 1814 | 1814 | } |
| 1815 | 1815 | |
| 1816 | −static int arm_smmu_handle_event(struct arm_smmu_device *smmu, | |
| 1817 | − struct arm_smmu_event *event) | |
| 1816 | +static int arm_smmu_handle_event(struct arm_smmu_device *smmu, u64 *evt, | |
| 1817 | + struct arm_smmu_event *event) | |
| 1818 | 1818 | { |
| 1819 | 1819 | int ret = 0; |
| 1820 | 1820 | u32 perm = 0; |
| @@ -1823,6 +1823,10 @@ static int arm_smmu_handle_event(struct arm_smmu_device *smmu, · 2 unmodified lines | ||
| 1823 | 1823 | struct iommu_fault *flt = &fault_evt.fault; |
| 1824 | 1824 | |
| 1825 | 1825 | switch (event->id) { |
| 1826 | + case EVT_ID_BAD_STE_CONFIG: | |
| 1827 | + case EVT_ID_STREAM_DISABLED_FAULT: | |
| 1828 | + case EVT_ID_BAD_SUBSTREAMID_CONFIG: | |
| 1829 | + case EVT_ID_BAD_CD_CONFIG: | |
| 1826 | 1830 | case EVT_ID_TRANSLATION_FAULT: |
| 1827 | 1831 | case EVT_ID_ADDR_SIZE_FAULT: |
| 1828 | 1832 | case EVT_ID_ACCESS_FAULT: |
| @@ -1832,31 +1836,30 @@ static int arm_smmu_handle_event(struct arm_smmu_device *smmu, · 3 unmodified lines | ||
| 1832 | 1836 | return -EOPNOTSUPP; |
| 1833 | 1837 | } |
| 1834 | 1838 | |
| 1835 | − if (!event->stall) | |
| 1836 | − return -EOPNOTSUPP; | |
| 1837 | − | |
| 1838 | − if (event->read) | |
| 1839 | − perm |= IOMMU_FAULT_PERM_READ; | |
| 1840 | − else | |
| 1841 | − perm |= IOMMU_FAULT_PERM_WRITE; | |
| 1839 | + if (event->stall) { | |
| 1840 | + if (event->read) | |
| 1841 | + perm |= IOMMU_FAULT_PERM_READ; | |
| 1842 | + else | |
| 1843 | + perm |= IOMMU_FAULT_PERM_WRITE; | |
| 1842 | 1844 | |
| 1843 | − if (event->instruction) | |
| 1844 | − perm |= IOMMU_FAULT_PERM_EXEC; | |
| 1845 | + if (event->instruction) | |
| 1846 | + perm |= IOMMU_FAULT_PERM_EXEC; | |
| 1845 | 1847 | |
| 1846 | − if (event->privileged) | |
| 1847 | − perm |= IOMMU_FAULT_PERM_PRIV; | |
| 1848 | + if (event->privileged) | |
| 1849 | + perm |= IOMMU_FAULT_PERM_PRIV; | |
| 1848 | 1850 | |
| 1849 | − flt->type = IOMMU_FAULT_PAGE_REQ; | |
| 1850 | − flt->prm = (struct iommu_fault_page_request) { | |
| 1851 | − .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE, | |
| 1852 | − .grpid = event->stag, | |
| 1853 | − .perm = perm, | |
| 1854 | − .addr = event->iova, | |
| 1855 | − }; | |
| 1851 | + flt->type = IOMMU_FAULT_PAGE_REQ; | |
| 1852 | + flt->prm = (struct iommu_fault_page_request){ | |
| 1853 | + .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE, | |
| 1854 | + .grpid = event->stag, | |
| 1855 | + .perm = perm, | |
| 1856 | + .addr = event->iova, | |
| 1857 | + }; | |
| 1856 | 1858 | |
| 1857 | − if (event->ssv) { | |
| 1858 | − flt->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; | |
| 1859 | − flt->prm.pasid = event->ssid; | |
| 1859 | + if (event->ssv) { | |
| 1860 | + flt->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; | |
| 1861 | + flt->prm.pasid = event->ssid; | |
| 1862 | + } | |
| 1860 | 1863 | } |
| 1861 | 1864 | |
| 1862 | 1865 | mutex_lock(&smmu->streams_mutex); |
| @@ -1866,7 +1869,12 @@ static int arm_smmu_handle_event(struct arm_smmu_device *smmu, · 3 unmodified lines | ||
| 1866 | 1869 | goto out_unlock; |
| 1867 | 1870 | } |
| 1868 | 1871 | |
| 1869 | − ret = iommu_report_device_fault(master->dev, &fault_evt); | |
| 1872 | + if (event->stall) | |
| 1873 | + ret = iommu_report_device_fault(master->dev, &fault_evt); | |
| 1874 | + else if (master->vmaster && !event->s2) | |
| 1875 | + ret = arm_vmaster_report_event(master->vmaster, evt); | |
| 1876 | + else | |
| 1877 | + ret = -EOPNOTSUPP; /* Unhandled events should be pinned */ | |
| 1870 | 1878 | out_unlock: |
| 1871 | 1879 | mutex_unlock(&smmu->streams_mutex); |
| 1872 | 1880 | return ret; |
| @@ -1944,7 +1952,7 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) · 71 unmodified lines | ||
| 1944 | 1952 | do { |
| 1945 | 1953 | while (!queue_remove_raw(q, evt)) { |
| 1946 | 1954 | arm_smmu_decode_event(smmu, evt, &event); |
| 1947 | − if (arm_smmu_handle_event(smmu, &event)) | |
| 1955 | + if (arm_smmu_handle_event(smmu, evt, &event)) | |
| 1948 | 1956 | arm_smmu_dump_event(smmu, evt, &event, &rs); |
| 1949 | 1957 | |
| 1950 | 1958 | put_device(event.dev); |
| @@ -2803,6 +2811,7 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state, · 852 unmodified lines | ||
| 2803 | 2811 | struct arm_smmu_domain *smmu_domain = |
| 2804 | 2812 | to_smmu_domain_devices(new_domain); |
| 2805 | 2813 | unsigned long flags; |
| 2814 | + int ret; | |
| 2806 | 2815 | |
| 2807 | 2816 | /* |
| 2808 | 2817 | * arm_smmu_share_asid() must not see two domains pointing to the same |
| @@ -2832,9 +2841,18 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state, · 23 unmodified lines | ||
| 2832 | 2841 | } |
| 2833 | 2842 | |
| 2834 | 2843 | if (smmu_domain) { |
| 2844 | + if (new_domain->type == IOMMU_DOMAIN_NESTED) { | |
| 2845 | + ret = arm_smmu_attach_prepare_vmaster( | |
| 2846 | + state, to_smmu_nested_domain(new_domain)); | |
| 2847 | + if (ret) | |
| 2848 | + return ret; | |
| 2849 | + } | |
| 2850 | + | |
| 2835 | 2851 | master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL); |
| 2836 | − if (!master_domain) | |
| 2852 | + if (!master_domain) { | |
| 2853 | + kfree(state->vmaster); | |
| 2837 | 2854 | return -ENOMEM; |
| 2855 | + } | |
| 2838 | 2856 | master_domain->master = master; |
| 2839 | 2857 | master_domain->ssid = state->ssid; |
| 2840 | 2858 | if (new_domain->type == IOMMU_DOMAIN_NESTED) |
| @@ -2861,6 +2879,7 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state, · 20 unmodified lines | ||
| 2861 | 2879 | spin_unlock_irqrestore(&smmu_domain->devices_lock, |
| 2862 | 2880 | flags); |
| 2863 | 2881 | kfree(master_domain); |
| 2882 | + kfree(state->vmaster); | |
| 2864 | 2883 | return -EINVAL; |
| 2865 | 2884 | } |
| 2866 | 2885 | |
| @@ -2893,6 +2912,8 @@ void arm_smmu_attach_commit(struct arm_smmu_attach_state *state) · 26 unmodified lines | ||
| 2893 | 2912 | |
| 2894 | 2913 | lockdep_assert_held(&arm_smmu_asid_lock); |
| 2895 | 2914 | |
| 2915 | + arm_smmu_attach_commit_vmaster(state); | |
| 2916 | + | |
| 2896 | 2917 | if (state->ats_enabled && !master->ats_enabled) { |
| 2897 | 2918 | arm_smmu_enable_ats(master); |
| 2898 | 2919 | } else if (state->ats_enabled && master->ats_enabled) { |
| @@ -3162,6 +3183,7 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain, · 263 unmodified lines | ||
| 3162 | 3183 | struct arm_smmu_ste ste; |
| 3163 | 3184 | struct arm_smmu_master *master = dev_iommu_priv_get(dev); |
| 3164 | 3185 | |
| 3186 | + arm_smmu_master_clear_vmaster(master); | |
| 3165 | 3187 | arm_smmu_make_bypass_ste(master->smmu, &ste); |
| 3166 | 3188 | arm_smmu_attach_dev_ste(domain, dev, &ste, STRTAB_STE_1_S1DSS_BYPASS); |
| 3167 | 3189 | return 0; |
| @@ -3180,7 +3202,9 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain, · 12 unmodified lines | ||
| 3180 | 3202 | struct device *dev) |
| 3181 | 3203 | { |
| 3182 | 3204 | struct arm_smmu_ste ste; |
| 3205 | + struct arm_smmu_master *master = dev_iommu_priv_get(dev); | |
| 3183 | 3206 | |
| 3207 | + arm_smmu_master_clear_vmaster(master); | |
| 3184 | 3208 | arm_smmu_make_abort_ste(&ste); |
| 3185 | 3209 | arm_smmu_attach_dev_ste(domain, dev, &ste, |
| 3186 | 3210 | STRTAB_STE_1_S1DSS_TERMINATE); |
| @@ -266,6 +266,7 @@ static inline u32 arm_smmu_strtab_l2_idx(u32 sid) · 265 unmodified lines | ||
| 266 | 266 | #define STRTAB_STE_1_S1COR GENMASK_ULL(5, 4) |
| 267 | 267 | #define STRTAB_STE_1_S1CSH GENMASK_ULL(7, 6) |
| 268 | 268 | |
| 269 | +#define STRTAB_STE_1_MEV (1UL << 19) | |
| 269 | 270 | #define STRTAB_STE_1_S2FWB (1UL << 25) |
| 270 | 271 | #define STRTAB_STE_1_S1STALLD (1UL << 27) |
| 271 | 272 | |
| @@ -799,6 +800,11 @@ struct arm_smmu_stream { · 527 unmodified lines | ||
| 799 | 800 | struct rb_node node; |
| 800 | 801 | }; |
| 801 | 802 | |
| 803 | +struct arm_smmu_vmaster { | |
| 804 | + struct arm_vsmmu *vsmmu; | |
| 805 | + unsigned long vsid; | |
| 806 | +}; | |
| 807 | + | |
| 802 | 808 | struct arm_smmu_event { |
| 803 | 809 | u8 stall : 1, |
| 804 | 810 | ssv : 1, |
| @@ -824,6 +830,7 @@ struct arm_smmu_master { · 19 unmodified lines | ||
| 824 | 830 | struct arm_smmu_device *smmu; |
| 825 | 831 | struct device *dev; |
| 826 | 832 | struct arm_smmu_stream *streams; |
| 833 | + struct arm_smmu_vmaster *vmaster; /* use smmu->streams_mutex */ | |
| 827 | 834 | /* Locked by the iommu core using the group mutex */ |
| 828 | 835 | struct arm_smmu_ctx_desc_cfg cd_table; |
| 829 | 836 | unsigned int num_streams; |
| @@ -972,6 +979,7 @@ struct arm_smmu_attach_state { · 142 unmodified lines | ||
| 972 | 979 | bool disable_ats; |
| 973 | 980 | ioasid_t ssid; |
| 974 | 981 | /* Resulting state */ |
| 982 | + struct arm_smmu_vmaster *vmaster; | |
| 975 | 983 | bool ats_enabled; |
| 976 | 984 | }; |
| 977 | 985 | |
| @@ -1055,9 +1063,37 @@ struct iommufd_viommu *arm_vsmmu_alloc(struct device *dev, · 77 unmodified lines | ||
| 1055 | 1063 | struct iommu_domain *parent, |
| 1056 | 1064 | struct iommufd_ctx *ictx, |
| 1057 | 1065 | unsigned int viommu_type); |
| 1066 | +int arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state, | |
| 1067 | + struct arm_smmu_nested_domain *nested_domain); | |
| 1068 | +void arm_smmu_attach_commit_vmaster(struct arm_smmu_attach_state *state); | |
| 1069 | +void arm_smmu_master_clear_vmaster(struct arm_smmu_master *master); | |
| 1070 | +int arm_vmaster_report_event(struct arm_smmu_vmaster *vmaster, u64 *evt); | |
| 1058 | 1071 | #else |
| 1059 | 1072 | #define arm_smmu_hw_info NULL |
| 1060 | 1073 | #define arm_vsmmu_alloc NULL |
| 1074 | + | |
| 1075 | +static inline int | |
| 1076 | +arm_smmu_attach_prepare_vmaster(struct arm_smmu_attach_state *state, | |
| 1077 | + struct arm_smmu_nested_domain *nested_domain) | |
| 1078 | +{ | |
| 1079 | + return 0; | |
| 1080 | +} | |
| 1081 | + | |
| 1082 | +static inline void | |
| 1083 | +arm_smmu_attach_commit_vmaster(struct arm_smmu_attach_state *state) | |
| 1084 | +{ | |
| 1085 | +} | |
| 1086 | + | |
| 1087 | +static inline void | |
| 1088 | +arm_smmu_master_clear_vmaster(struct arm_smmu_master *master) | |
| 1089 | +{ | |
| 1090 | +} | |
| 1091 | + | |
| 1092 | +static inline int arm_vmaster_report_event(struct arm_smmu_vmaster *vmaster, | |
| 1093 | + u64 *evt) | |
| 1094 | +{ | |
| 1095 | + return -EOPNOTSUPP; | |
| 1096 | +} | |
| 1061 | 1097 | #endif /* CONFIG_ARM_SMMU_V3_IOMMUFD */ |
| 1062 | 1098 | |
| 1063 | 1099 | #endif /* _ARM_SMMU_V3_H */ |
| @@ -42,11 +42,6 @@ struct iommu_dma_msi_page { · 41 unmodified lines | ||
| 42 | 42 | phys_addr_t phys; |
| 43 | 43 | }; |
| 44 | 44 | |
| 45 | −enum iommu_dma_cookie_type { | |
| 46 | − IOMMU_DMA_IOVA_COOKIE, | |
| 47 | − IOMMU_DMA_MSI_COOKIE, | |
| 48 | −}; | |
| 49 | − | |
| 50 | 45 | enum iommu_dma_queue_type { |
| 51 | 46 | IOMMU_DMA_OPTS_PER_CPU_QUEUE, |
| 52 | 47 | IOMMU_DMA_OPTS_SINGLE_QUEUE, |
| @@ -59,34 +54,30 @@ struct iommu_dma_options { · 6 unmodified lines | ||
| 59 | 54 | }; |
| 60 | 55 | |
| 61 | 56 | struct iommu_dma_cookie { |
| 62 | − enum iommu_dma_cookie_type type; | |
| 57 | + struct iova_domain iovad; | |
| 58 | + struct list_head msi_page_list; | |
| 59 | + /* Flush queue */ | |
| 63 | 60 | union { |
| 64 | − /* Full allocator for IOMMU_DMA_IOVA_COOKIE */ | |
| 65 | − struct { | |
| 66 | − struct iova_domain iovad; | |
| 67 | − /* Flush queue */ | |
| 68 | − union { | |
| 69 | − struct iova_fq *single_fq; | |
| 70 | − struct iova_fq __percpu *percpu_fq; | |
| 71 | − }; | |
| 72 | − /* Number of TLB flushes that have been started */ | |
| 73 | − atomic64_t fq_flush_start_cnt; | |
| 74 | − /* Number of TLB flushes that have been finished */ | |
| 75 | − atomic64_t fq_flush_finish_cnt; | |
| 76 | − /* Timer to regularily empty the flush queues */ | |
| 77 | − struct timer_list fq_timer; | |
| 78 | − /* 1 when timer is active, 0 when not */ | |
| 79 | − atomic_t fq_timer_on; | |
| 80 | − }; | |
| 81 | − /* Trivial linear page allocator for IOMMU_DMA_MSI_COOKIE */ | |
| 82 | − dma_addr_t msi_iova; | |
| 61 | + struct iova_fq *single_fq; | |
| 62 | + struct iova_fq __percpu *percpu_fq; | |
| 83 | 63 | }; |
| 84 | − struct list_head msi_page_list; | |
| 85 | − | |
| 64 | + /* Number of TLB flushes that have been started */ | |
| 65 | + atomic64_t fq_flush_start_cnt; | |
| 66 | + /* Number of TLB flushes that have been finished */ | |
| 67 | + atomic64_t fq_flush_finish_cnt; | |
| 68 | + /* Timer to regularily empty the flush queues */ | |
| 69 | + struct timer_list fq_timer; | |
| 70 | + /* 1 when timer is active, 0 when not */ | |
| 71 | + atomic_t fq_timer_on; | |
| 86 | 72 | /* Domain for flush queue callback; NULL if flush queue not in use */ |
| 87 | − struct iommu_domain *fq_domain; | |
| 73 | + struct iommu_domain *fq_domain; | |
| 88 | 74 | /* Options for dma-iommu use */ |
| 89 | − struct iommu_dma_options options; | |
| 75 | + struct iommu_dma_options options; | |
| 76 | +}; | |
| 77 | + | |
| 78 | +struct iommu_dma_msi_cookie { | |
| 79 | + dma_addr_t msi_iova; | |
| 80 | + struct list_head msi_page_list; | |
| 90 | 81 | }; |
| 91 | 82 | |
| 92 | 83 | static DEFINE_STATIC_KEY_FALSE(iommu_deferred_attach_enabled); |
| @@ -102,9 +93,6 @@ static int __init iommu_dma_forcedac_setup(char *str) · 9 unmodified lines | ||
| 102 | 93 | } |
| 103 | 94 | early_param("iommu.forcedac", iommu_dma_forcedac_setup); |
| 104 | 95 | |
| 105 | −static int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | |
| 106 | − phys_addr_t msi_addr); | |
| 107 | − | |
| 108 | 96 | /* Number of entries per flush queue */ |
| 109 | 97 | #define IOVA_DEFAULT_FQ_SIZE 256 |
| 110 | 98 | #define IOVA_SINGLE_FQ_SIZE 32768 |
| @@ -368,39 +356,24 @@ int iommu_dma_init_fq(struct iommu_domain *domain) · 257 unmodified lines | ||
| 368 | 356 | return 0; |
| 369 | 357 | } |
| 370 | 358 | |
| 371 | −static inline size_t cookie_msi_granule(struct iommu_dma_cookie *cookie) | |
| 372 | −{ | |
| 373 | − if (cookie->type == IOMMU_DMA_IOVA_COOKIE) | |
| 374 | − return cookie->iovad.granule; | |
| 375 | − return PAGE_SIZE; | |
| 376 | −} | |
| 377 | − | |
| 378 | −static struct iommu_dma_cookie *cookie_alloc(enum iommu_dma_cookie_type type) | |
| 379 | −{ | |
| 380 | − struct iommu_dma_cookie *cookie; | |
| 381 | − | |
| 382 | − cookie = kzalloc(sizeof(*cookie), GFP_KERNEL); | |
| 383 | − if (cookie) { | |
| 384 | − INIT_LIST_HEAD(&cookie->msi_page_list); | |
| 385 | − cookie->type = type; | |
| 386 | − } | |
| 387 | − return cookie; | |
| 388 | −} | |
| 389 | − | |
| 390 | 359 | /** |
| 391 | 360 | * iommu_get_dma_cookie - Acquire DMA-API resources for a domain |
| 392 | 361 | * @domain: IOMMU domain to prepare for DMA-API usage |
| 393 | 362 | */ |
| 394 | 363 | int iommu_get_dma_cookie(struct iommu_domain *domain) |
| 395 | 364 | { |
| 396 | − if (domain->iova_cookie) | |
| 365 | + struct iommu_dma_cookie *cookie; | |
| 366 | + | |
| 367 | + if (domain->cookie_type != IOMMU_COOKIE_NONE) | |
| 397 | 368 | return -EEXIST; |
| 398 | 369 | |
| 399 | − domain->iova_cookie = cookie_alloc(IOMMU_DMA_IOVA_COOKIE); | |
| 400 | − if (!domain->iova_cookie) | |
| 370 | + cookie = kzalloc(sizeof(*cookie), GFP_KERNEL); | |
| 371 | + if (!cookie) | |
| 401 | 372 | return -ENOMEM; |
| 402 | 373 | |
| 403 | − iommu_domain_set_sw_msi(domain, iommu_dma_sw_msi); | |
| 374 | + INIT_LIST_HEAD(&cookie->msi_page_list); | |
| 375 | + domain->cookie_type = IOMMU_COOKIE_DMA_IOVA; | |
| 376 | + domain->iova_cookie = cookie; | |
| 404 | 377 | return 0; |
| 405 | 378 | } |
| 406 | 379 | |
| @@ -418,54 +391,56 @@ int iommu_get_dma_cookie(struct iommu_domain *domain) · 11 unmodified lines | ||
| 418 | 391 | */ |
| 419 | 392 | int iommu_get_msi_cookie(struct iommu_domain *domain, dma_addr_t base) |
| 420 | 393 | { |
| 421 | − struct iommu_dma_cookie *cookie; | |
| 394 | + struct iommu_dma_msi_cookie *cookie; | |
| 422 | 395 | |
| 423 | 396 | if (domain->type != IOMMU_DOMAIN_UNMANAGED) |
| 424 | 397 | return -EINVAL; |
| 425 | 398 | |
| 426 | − if (domain->iova_cookie) | |
| 399 | + if (domain->cookie_type != IOMMU_COOKIE_NONE) | |
| 427 | 400 | return -EEXIST; |
| 428 | 401 | |
| 429 | − cookie = cookie_alloc(IOMMU_DMA_MSI_COOKIE); | |
| 402 | + cookie = kzalloc(sizeof(*cookie), GFP_KERNEL); | |
| 430 | 403 | if (!cookie) |
| 431 | 404 | return -ENOMEM; |
| 432 | 405 | |
| 433 | 406 | cookie->msi_iova = base; |
| 434 | − domain->iova_cookie = cookie; | |
| 435 | − iommu_domain_set_sw_msi(domain, iommu_dma_sw_msi); | |
| 407 | + INIT_LIST_HEAD(&cookie->msi_page_list); | |
| 408 | + domain->cookie_type = IOMMU_COOKIE_DMA_MSI; | |
| 409 | + domain->msi_cookie = cookie; | |
| 436 | 410 | return 0; |
| 437 | 411 | } |
| 438 | 412 | EXPORT_SYMBOL(iommu_get_msi_cookie); |
| 439 | 413 | |
| 440 | 414 | /** |
| 441 | 415 | * iommu_put_dma_cookie - Release a domain's DMA mapping resources |
| 442 | − * @domain: IOMMU domain previously prepared by iommu_get_dma_cookie() or | |
| 443 | − * iommu_get_msi_cookie() | |
| 416 | + * @domain: IOMMU domain previously prepared by iommu_get_dma_cookie() | |
| 444 | 417 | */ |
| 445 | 418 | void iommu_put_dma_cookie(struct iommu_domain *domain) |
| 446 | 419 | { |
| 447 | 420 | struct iommu_dma_cookie *cookie = domain->iova_cookie; |
| 448 | 421 | struct iommu_dma_msi_page *msi, *tmp; |
| 449 | 422 | |
| 450 | −#if IS_ENABLED(CONFIG_IRQ_MSI_IOMMU) | |
| 451 | − if (domain->sw_msi != iommu_dma_sw_msi) | |
| 452 | − return; | |
| 453 | −#endif | |
| 454 | − | |
| 455 | − if (!cookie) | |
| 456 | − return; | |
| 457 | − | |
| 458 | − if (cookie->type == IOMMU_DMA_IOVA_COOKIE && cookie->iovad.granule) { | |
| 423 | + if (cookie->iovad.granule) { | |
| 459 | 424 | iommu_dma_free_fq(cookie); |
| 460 | 425 | put_iova_domain(&cookie->iovad); |
| 461 | 426 | } |
| 427 | + list_for_each_entry_safe(msi, tmp, &cookie->msi_page_list, list) | |
| 428 | + kfree(msi); | |
| 429 | + kfree(cookie); | |
| 430 | +} | |
| 431 | + | |
| 432 | +/** | |
| 433 | + * iommu_put_msi_cookie - Release a domain's MSI mapping resources | |
| 434 | + * @domain: IOMMU domain previously prepared by iommu_get_msi_cookie() | |
| 435 | + */ | |
| 436 | +void iommu_put_msi_cookie(struct iommu_domain *domain) | |
| 437 | +{ | |
| 438 | + struct iommu_dma_msi_cookie *cookie = domain->msi_cookie; | |
| 439 | + struct iommu_dma_msi_page *msi, *tmp; | |
| 462 | 440 | |
| 463 | − list_for_each_entry_safe(msi, tmp, &cookie->msi_page_list, list) { | |
| 464 | − list_del(&msi->list); | |
| 441 | + list_for_each_entry_safe(msi, tmp, &cookie->msi_page_list, list) | |
| 465 | 442 | kfree(msi); |
| 466 | − } | |
| 467 | 443 | kfree(cookie); |
| 468 | − domain->iova_cookie = NULL; | |
| 469 | 444 | } |
| 470 | 445 | |
| 471 | 446 | /** |
| @@ -685,7 +660,7 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, struct device *dev · 213 unmodified lines | ||
| 685 | 660 | struct iova_domain *iovad; |
| 686 | 661 | int ret; |
| 687 | 662 | |
| 688 | − if (!cookie || cookie->type != IOMMU_DMA_IOVA_COOKIE) | |
| 663 | + if (!cookie || domain->cookie_type != IOMMU_COOKIE_DMA_IOVA) | |
| 689 | 664 | return -EINVAL; |
| 690 | 665 | |
| 691 | 666 | iovad = &cookie->iovad; |
| @@ -768,9 +743,9 @@ static dma_addr_t iommu_dma_alloc_iova(struct iommu_domain *domain, · 76 unmodified lines | ||
| 768 | 743 | struct iova_domain *iovad = &cookie->iovad; |
| 769 | 744 | unsigned long shift, iova_len, iova; |
| 770 | 745 | |
| 771 | − if (cookie->type == IOMMU_DMA_MSI_COOKIE) { | |
| 772 | − cookie->msi_iova += size; | |
| 773 | − return cookie->msi_iova - size; | |
| 746 | + if (domain->cookie_type == IOMMU_COOKIE_DMA_MSI) { | |
| 747 | + domain->msi_cookie->msi_iova += size; | |
| 748 | + return domain->msi_cookie->msi_iova - size; | |
| 774 | 749 | } |
| 775 | 750 | |
| 776 | 751 | shift = iova_shift(iovad); |
| @@ -807,16 +782,16 @@ done: · 30 unmodified lines | ||
| 807 | 782 | return (dma_addr_t)iova << shift; |
| 808 | 783 | } |
| 809 | 784 | |
| 810 | −static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie, | |
| 811 | − dma_addr_t iova, size_t size, struct iommu_iotlb_gather *gather) | |
| 785 | +static void iommu_dma_free_iova(struct iommu_domain *domain, dma_addr_t iova, | |
| 786 | + size_t size, struct iommu_iotlb_gather *gather) | |
| 812 | 787 | { |
| 813 | − struct iova_domain *iovad = &cookie->iovad; | |
| 788 | + struct iova_domain *iovad = &domain->iova_cookie->iovad; | |
| 814 | 789 | |
| 815 | 790 | /* The MSI case is only ever cleaning up its most recent allocation */ |
| 816 | − if (cookie->type == IOMMU_DMA_MSI_COOKIE) | |
| 817 | − cookie->msi_iova -= size; | |
| 791 | + if (domain->cookie_type == IOMMU_COOKIE_DMA_MSI) | |
| 792 | + domain->msi_cookie->msi_iova -= size; | |
| 818 | 793 | else if (gather && gather->queued) |
| 819 | − queue_iova(cookie, iova_pfn(iovad, iova), | |
| 794 | + queue_iova(domain->iova_cookie, iova_pfn(iovad, iova), | |
| 820 | 795 | size >> iova_shift(iovad), |
| 821 | 796 | &gather->freelist); |
| 822 | 797 | else |
| @@ -844,7 +819,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, · 21 unmodified lines | ||
| 844 | 819 | |
| 845 | 820 | if (!iotlb_gather.queued) |
| 846 | 821 | iommu_iotlb_sync(domain, &iotlb_gather); |
| 847 | − iommu_dma_free_iova(cookie, dma_addr, size, &iotlb_gather); | |
| 822 | + iommu_dma_free_iova(domain, dma_addr, size, &iotlb_gather); | |
| 848 | 823 | } |
| 849 | 824 | |
| 850 | 825 | static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, |
| @@ -872,7 +847,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, · 21 unmodified lines | ||
| 872 | 847 | return DMA_MAPPING_ERROR; |
| 873 | 848 | |
| 874 | 849 | if (iommu_map(domain, iova, phys - iova_off, size, prot, GFP_ATOMIC)) { |
| 875 | − iommu_dma_free_iova(cookie, iova, size, NULL); | |
| 850 | + iommu_dma_free_iova(domain, iova, size, NULL); | |
| 876 | 851 | return DMA_MAPPING_ERROR; |
| 877 | 852 | } |
| 878 | 853 | return iova + iova_off; |
| @@ -1009,7 +984,7 @@ static struct page **__iommu_dma_alloc_noncontiguous(struct device *dev, · 130 unmodified lines | ||
| 1009 | 984 | out_free_sg: |
| 1010 | 985 | sg_free_table(sgt); |
| 1011 | 986 | out_free_iova: |
| 1012 | − iommu_dma_free_iova(cookie, iova, size, NULL); | |
| 987 | + iommu_dma_free_iova(domain, iova, size, NULL); | |
| 1013 | 988 | out_free_pages: |
| 1014 | 989 | __iommu_dma_free_pages(pages, count); |
| 1015 | 990 | return NULL; |
| @@ -1486,7 +1461,7 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents, · 470 unmodified lines | ||
| 1486 | 1461 | return __finalise_sg(dev, sg, nents, iova); |
| 1487 | 1462 | |
| 1488 | 1463 | out_free_iova: |
| 1489 | − iommu_dma_free_iova(cookie, iova, iova_len, NULL); | |
| 1464 | + iommu_dma_free_iova(domain, iova, iova_len, NULL); | |
| 1490 | 1465 | out_restore_sg: |
| 1491 | 1466 | __invalidate_sg(sg, nents); |
| 1492 | 1467 | out: |
| @@ -1764,17 +1739,47 @@ out_err: · 271 unmodified lines | ||
| 1764 | 1739 | dev->dma_iommu = false; |
| 1765 | 1740 | } |
| 1766 | 1741 | |
| 1742 | +static bool has_msi_cookie(const struct iommu_domain *domain) | |
| 1743 | +{ | |
| 1744 | + return domain && (domain->cookie_type == IOMMU_COOKIE_DMA_IOVA || | |
| 1745 | + domain->cookie_type == IOMMU_COOKIE_DMA_MSI); | |
| 1746 | +} | |
| 1747 | + | |
| 1748 | +static size_t cookie_msi_granule(const struct iommu_domain *domain) | |
| 1749 | +{ | |
| 1750 | + switch (domain->cookie_type) { | |
| 1751 | + case IOMMU_COOKIE_DMA_IOVA: | |
| 1752 | + return domain->iova_cookie->iovad.granule; | |
| 1753 | + case IOMMU_COOKIE_DMA_MSI: | |
| 1754 | + return PAGE_SIZE; | |
| 1755 | + default: | |
| 1756 | + BUG(); | |
| 1757 | + }; | |
| 1758 | +} | |
| 1759 | + | |
| 1760 | +static struct list_head *cookie_msi_pages(const struct iommu_domain *domain) | |
| 1761 | +{ | |
| 1762 | + switch (domain->cookie_type) { | |
| 1763 | + case IOMMU_COOKIE_DMA_IOVA: | |
| 1764 | + return &domain->iova_cookie->msi_page_list; | |
| 1765 | + case IOMMU_COOKIE_DMA_MSI: | |
| 1766 | + return &domain->msi_cookie->msi_page_list; | |
| 1767 | + default: | |
| 1768 | + BUG(); | |
| 1769 | + }; | |
| 1770 | +} | |
| 1771 | + | |
| 1767 | 1772 | static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev, |
| 1768 | 1773 | phys_addr_t msi_addr, struct iommu_domain *domain) |
| 1769 | 1774 | { |
| 1770 | − struct iommu_dma_cookie *cookie = domain->iova_cookie; | |
| 1775 | + struct list_head *msi_page_list = cookie_msi_pages(domain); | |
| 1771 | 1776 | struct iommu_dma_msi_page *msi_page; |
| 1772 | 1777 | dma_addr_t iova; |
| 1773 | 1778 | int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO; |
| 1774 | − size_t size = cookie_msi_granule(cookie); | |
| 1779 | + size_t size = cookie_msi_granule(domain); | |
| 1775 | 1780 | |
| 1776 | 1781 | msi_addr &= ~(phys_addr_t)(size - 1); |
| 1777 | − list_for_each_entry(msi_page, &cookie->msi_page_list, list) | |
| 1782 | + list_for_each_entry(msi_page, msi_page_list, list) | |
| 1778 | 1783 | if (msi_page->phys == msi_addr) |
| 1779 | 1784 | return msi_page; |
| 1780 | 1785 | |
| @@ -1792,23 +1797,23 @@ static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev, · 11 unmodified lines | ||
| 1792 | 1797 | INIT_LIST_HEAD(&msi_page->list); |
| 1793 | 1798 | msi_page->phys = msi_addr; |
| 1794 | 1799 | msi_page->iova = iova; |
| 1795 | − list_add(&msi_page->list, &cookie->msi_page_list); | |
| 1800 | + list_add(&msi_page->list, msi_page_list); | |
| 1796 | 1801 | return msi_page; |
| 1797 | 1802 | |
| 1798 | 1803 | out_free_iova: |
| 1799 | − iommu_dma_free_iova(cookie, iova, size, NULL); | |
| 1804 | + iommu_dma_free_iova(domain, iova, size, NULL); | |
| 1800 | 1805 | out_free_page: |
| 1801 | 1806 | kfree(msi_page); |
| 1802 | 1807 | return NULL; |
| 1803 | 1808 | } |
| 1804 | 1809 | |
| 1805 | −static int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | |
| 1806 | − phys_addr_t msi_addr) | |
| 1810 | +int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | |
| 1811 | + phys_addr_t msi_addr) | |
| 1807 | 1812 | { |
| 1808 | 1813 | struct device *dev = msi_desc_to_dev(desc); |
| 1809 | 1814 | const struct iommu_dma_msi_page *msi_page; |
| 1810 | 1815 | |
| 1811 | − if (!domain->iova_cookie) { | |
| 1816 | + if (!has_msi_cookie(domain)) { | |
| 1812 | 1817 | msi_desc_set_iommu_msi_iova(desc, 0, 0); |
| 1813 | 1818 | return 0; |
| 1814 | 1819 | } |
| @@ -1818,9 +1823,8 @@ static int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, · 3 unmodified lines | ||
| 1818 | 1823 | if (!msi_page) |
| 1819 | 1824 | return -ENOMEM; |
| 1820 | 1825 | |
| 1821 | − msi_desc_set_iommu_msi_iova( | |
| 1822 | − desc, msi_page->iova, | |
| 1823 | − ilog2(cookie_msi_granule(domain->iova_cookie))); | |
| 1826 | + msi_desc_set_iommu_msi_iova(desc, msi_page->iova, | |
| 1827 | + ilog2(cookie_msi_granule(domain))); | |
| 1824 | 1828 | return 0; |
| 1825 | 1829 | } |
| 1826 | 1830 | |
| @@ -13,11 +13,15 @@ void iommu_setup_dma_ops(struct device *dev); · 12 unmodified lines | ||
| 13 | 13 | |
| 14 | 14 | int iommu_get_dma_cookie(struct iommu_domain *domain); |
| 15 | 15 | void iommu_put_dma_cookie(struct iommu_domain *domain); |
| 16 | +void iommu_put_msi_cookie(struct iommu_domain *domain); | |
| 16 | 17 | |
| 17 | 18 | int iommu_dma_init_fq(struct iommu_domain *domain); |
| 18 | 19 | |
| 19 | 20 | void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list); |
| 20 | 21 | |
| 22 | +int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | |
| 23 | + phys_addr_t msi_addr); | |
| 24 | + | |
| 21 | 25 | extern bool iommu_dma_forcedac; |
| 22 | 26 | |
| 23 | 27 | #else /* CONFIG_IOMMU_DMA */ |
| @@ -40,9 +44,19 @@ static inline void iommu_put_dma_cookie(struct iommu_domain *domain) · 16 unmodified lines | ||
| 40 | 44 | { |
| 41 | 45 | } |
| 42 | 46 | |
| 47 | +static inline void iommu_put_msi_cookie(struct iommu_domain *domain) | |
| 48 | +{ | |
| 49 | +} | |
| 50 | + | |
| 43 | 51 | static inline void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list) |
| 44 | 52 | { |
| 45 | 53 | } |
| 46 | 54 | |
| 55 | +static inline int iommu_dma_sw_msi(struct iommu_domain *domain, | |
| 56 | + struct msi_desc *desc, phys_addr_t msi_addr) | |
| 57 | +{ | |
| 58 | + return -ENODEV; | |
| 59 | +} | |
| 60 | + | |
| 47 | 61 | #endif /* CONFIG_IOMMU_DMA */ |
| 48 | 62 | #endif /* __DMA_IOMMU_H */ |
| @@ -3383,7 +3383,8 @@ intel_iommu_domain_alloc_paging_flags(struct device *dev, u32 flags, · 3382 unmodified lines | ||
| 3383 | 3383 | bool first_stage; |
| 3384 | 3384 | |
| 3385 | 3385 | if (flags & |
| 3386 | − (~(IOMMU_HWPT_ALLOC_NEST_PARENT | IOMMU_HWPT_ALLOC_DIRTY_TRACKING))) | |
| 3386 | + (~(IOMMU_HWPT_ALLOC_NEST_PARENT | IOMMU_HWPT_ALLOC_DIRTY_TRACKING | | |
| 3387 | + IOMMU_HWPT_ALLOC_PASID))) | |
| 3387 | 3388 | return ERR_PTR(-EOPNOTSUPP); |
| 3388 | 3389 | if (nested_parent && !nested_supported(iommu)) |
| 3389 | 3390 | return ERR_PTR(-EOPNOTSUPP); |
| @@ -198,7 +198,7 @@ intel_iommu_domain_alloc_nested(struct device *dev, struct iommu_domain *parent, · 197 unmodified lines | ||
| 198 | 198 | struct dmar_domain *domain; |
| 199 | 199 | int ret; |
| 200 | 200 | |
| 201 | − if (!nested_supported(iommu) || flags) | |
| 201 | + if (!nested_supported(iommu) || flags & ~IOMMU_HWPT_ALLOC_PASID) | |
| 202 | 202 | return ERR_PTR(-EOPNOTSUPP); |
| 203 | 203 | |
| 204 | 204 | /* Must be nested domain */ |
| @@ -5,6 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | #define __LINUX_IOMMU_PRIV_H |
| 6 | 6 | |
| 7 | 7 | #include <linux/iommu.h> |
| 8 | +#include <linux/msi.h> | |
| 8 | 9 | |
| 9 | 10 | static inline const struct iommu_ops *dev_iommu_ops(struct device *dev) |
| 10 | 11 | { |
| @@ -47,4 +48,19 @@ void iommu_detach_group_handle(struct iommu_domain *domain, · 36 unmodified lines | ||
| 47 | 48 | int iommu_replace_group_handle(struct iommu_group *group, |
| 48 | 49 | struct iommu_domain *new_domain, |
| 49 | 50 | struct iommu_attach_handle *handle); |
| 51 | + | |
| 52 | +#if IS_ENABLED(CONFIG_IOMMUFD_DRIVER_CORE) && IS_ENABLED(CONFIG_IRQ_MSI_IOMMU) | |
| 53 | +int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | |
| 54 | + phys_addr_t msi_addr); | |
| 55 | +#else /* !CONFIG_IOMMUFD_DRIVER_CORE || !CONFIG_IRQ_MSI_IOMMU */ | |
| 56 | +static inline int iommufd_sw_msi(struct iommu_domain *domain, | |
| 57 | + struct msi_desc *desc, phys_addr_t msi_addr) | |
| 58 | +{ | |
| 59 | + return -EOPNOTSUPP; | |
| 60 | +} | |
| 61 | +#endif /* CONFIG_IOMMUFD_DRIVER_CORE && CONFIG_IRQ_MSI_IOMMU */ | |
| 62 | + | |
| 63 | +int iommu_replace_device_pasid(struct iommu_domain *domain, | |
| 64 | + struct device *dev, ioasid_t pasid, | |
| 65 | + struct iommu_attach_handle *handle); | |
| 50 | 66 | #endif /* __LINUX_IOMMU_PRIV_H */ |
| @@ -310,6 +310,7 @@ static struct iommu_domain *iommu_sva_domain_alloc(struct device *dev, · 309 unmodified lines | ||
| 310 | 310 | } |
| 311 | 311 | |
| 312 | 312 | domain->type = IOMMU_DOMAIN_SVA; |
| 313 | + domain->cookie_type = IOMMU_COOKIE_SVA; | |
| 313 | 314 | mmgrab(mm); |
| 314 | 315 | domain->mm = mm; |
| 315 | 316 | domain->owner = ops; |
| @@ -18,6 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <linux/errno.h> |
| 19 | 19 | #include <linux/host1x_context_bus.h> |
| 20 | 20 | #include <linux/iommu.h> |
| 21 | +#include <linux/iommufd.h> | |
| 21 | 22 | #include <linux/idr.h> |
| 22 | 23 | #include <linux/err.h> |
| 23 | 24 | #include <linux/pci.h> |
| @@ -539,6 +540,13 @@ static void iommu_deinit_device(struct device *dev) · 515 unmodified lines | ||
| 539 | 540 | dev_iommu_free(dev); |
| 540 | 541 | } |
| 541 | 542 | |
| 543 | +static struct iommu_domain *pasid_array_entry_to_domain(void *entry) | |
| 544 | +{ | |
| 545 | + if (xa_pointer_tag(entry) == IOMMU_PASID_ARRAY_DOMAIN) | |
| 546 | + return xa_untag_pointer(entry); | |
| 547 | + return ((struct iommu_attach_handle *)xa_untag_pointer(entry))->domain; | |
| 548 | +} | |
| 549 | + | |
| 542 | 550 | DEFINE_MUTEX(iommu_probe_device_lock); |
| 543 | 551 | |
| 544 | 552 | static int __iommu_probe_device(struct device *dev, struct list_head *group_list) |
| @@ -1973,8 +1981,10 @@ void iommu_set_fault_handler(struct iommu_domain *domain, · 1428 unmodified lines | ||
| 1973 | 1981 | iommu_fault_handler_t handler, |
| 1974 | 1982 | void *token) |
| 1975 | 1983 | { |
| 1976 | − BUG_ON(!domain); | |
| 1984 | + if (WARN_ON(!domain || domain->cookie_type != IOMMU_COOKIE_NONE)) | |
| 1985 | + return; | |
| 1977 | 1986 | |
| 1987 | + domain->cookie_type = IOMMU_COOKIE_FAULT_HANDLER; | |
| 1978 | 1988 | domain->handler = handler; |
| 1979 | 1989 | domain->handler_token = token; |
| 1980 | 1990 | } |
| @@ -2044,9 +2054,19 @@ EXPORT_SYMBOL_GPL(iommu_paging_domain_alloc_flags); · 63 unmodified lines | ||
| 2044 | 2054 | |
| 2045 | 2055 | void iommu_domain_free(struct iommu_domain *domain) |
| 2046 | 2056 | { |
| 2047 | − if (domain->type == IOMMU_DOMAIN_SVA) | |
| 2057 | + switch (domain->cookie_type) { | |
| 2058 | + case IOMMU_COOKIE_DMA_IOVA: | |
| 2059 | + iommu_put_dma_cookie(domain); | |
| 2060 | + break; | |
| 2061 | + case IOMMU_COOKIE_DMA_MSI: | |
| 2062 | + iommu_put_msi_cookie(domain); | |
| 2063 | + break; | |
| 2064 | + case IOMMU_COOKIE_SVA: | |
| 2048 | 2065 | mmdrop(domain->mm); |
| 2049 | − iommu_put_dma_cookie(domain); | |
| 2066 | + break; | |
| 2067 | + default: | |
| 2068 | + break; | |
| 2069 | + } | |
| 2050 | 2070 | if (domain->ops->free) |
| 2051 | 2071 | domain->ops->free(domain); |
| 2052 | 2072 | } |
| @@ -3335,14 +3355,15 @@ static void iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid, · 1282 unmodified lines | ||
| 3335 | 3355 | } |
| 3336 | 3356 | |
| 3337 | 3357 | static int __iommu_set_group_pasid(struct iommu_domain *domain, |
| 3338 | − struct iommu_group *group, ioasid_t pasid) | |
| 3358 | + struct iommu_group *group, ioasid_t pasid, | |
| 3359 | + struct iommu_domain *old) | |
| 3339 | 3360 | { |
| 3340 | 3361 | struct group_device *device, *last_gdev; |
| 3341 | 3362 | int ret; |
| 3342 | 3363 | |
| 3343 | 3364 | for_each_group_device(group, device) { |
| 3344 | 3365 | ret = domain->ops->set_dev_pasid(domain, device->dev, |
| 3345 | − pasid, NULL); | |
| 3366 | + pasid, old); | |
| 3346 | 3367 | if (ret) |
| 3347 | 3368 | goto err_revert; |
| 3348 | 3369 | } |
| @@ -3354,7 +3375,15 @@ err_revert: · 5 unmodified lines | ||
| 3354 | 3375 | for_each_group_device(group, device) { |
| 3355 | 3376 | if (device == last_gdev) |
| 3356 | 3377 | break; |
| 3357 | − iommu_remove_dev_pasid(device->dev, pasid, domain); | |
| 3378 | + /* | |
| 3379 | + * If no old domain, undo the succeeded devices/pasid. | |
| 3380 | + * Otherwise, rollback the succeeded devices/pasid to the old | |
| 3381 | + * domain. And it is a driver bug to fail attaching with a | |
| 3382 | + * previously good domain. | |
| 3383 | + */ | |
| 3384 | + if (!old || WARN_ON(old->ops->set_dev_pasid(old, device->dev, | |
| 3385 | + pasid, domain))) | |
| 3386 | + iommu_remove_dev_pasid(device->dev, pasid, domain); | |
| 3358 | 3387 | } |
| 3359 | 3388 | return ret; |
| 3360 | 3389 | } |
| @@ -3376,6 +3405,9 @@ static void __iommu_remove_group_pasid(struct iommu_group *group, · 15 unmodified lines | ||
| 3376 | 3405 | * @pasid: the pasid of the device. |
| 3377 | 3406 | * @handle: the attach handle. |
| 3378 | 3407 | * |
| 3408 | + * Caller should always provide a new handle to avoid race with the paths | |
| 3409 | + * that have lockless reference to handle if it intends to pass a valid handle. | |
| 3410 | + * | |
| 3379 | 3411 | * Return: 0 on success, or an error. |
| 3380 | 3412 | */ |
| 3381 | 3413 | int iommu_attach_device_pasid(struct iommu_domain *domain, |
| @@ -3420,7 +3452,7 @@ int iommu_attach_device_pasid(struct iommu_domain *domain, · 38 unmodified lines | ||
| 3420 | 3452 | if (ret) |
| 3421 | 3453 | goto out_unlock; |
| 3422 | 3454 | |
| 3423 | − ret = __iommu_set_group_pasid(domain, group, pasid); | |
| 3455 | + ret = __iommu_set_group_pasid(domain, group, pasid, NULL); | |
| 3424 | 3456 | if (ret) { |
| 3425 | 3457 | xa_release(&group->pasid_array, pasid); |
| 3426 | 3458 | goto out_unlock; |
| @@ -3441,6 +3473,97 @@ out_unlock: · 14 unmodified lines | ||
| 3441 | 3473 | } |
| 3442 | 3474 | EXPORT_SYMBOL_GPL(iommu_attach_device_pasid); |
| 3443 | 3475 | |
| 3476 | +/** | |
| 3477 | + * iommu_replace_device_pasid - Replace the domain that a specific pasid | |
| 3478 | + * of the device is attached to | |
| 3479 | + * @domain: the new iommu domain | |
| 3480 | + * @dev: the attached device. | |
| 3481 | + * @pasid: the pasid of the device. | |
| 3482 | + * @handle: the attach handle. | |
| 3483 | + * | |
| 3484 | + * This API allows the pasid to switch domains. The @pasid should have been | |
| 3485 | + * attached. Otherwise, this fails. The pasid will keep the old configuration | |
| 3486 | + * if replacement failed. | |
| 3487 | + * | |
| 3488 | + * Caller should always provide a new handle to avoid race with the paths | |
| 3489 | + * that have lockless reference to handle if it intends to pass a valid handle. | |
| 3490 | + * | |
| 3491 | + * Return 0 on success, or an error. | |
| 3492 | + */ | |
| 3493 | +int iommu_replace_device_pasid(struct iommu_domain *domain, | |
| 3494 | + struct device *dev, ioasid_t pasid, | |
| 3495 | + struct iommu_attach_handle *handle) | |
| 3496 | +{ | |
| 3497 | + /* Caller must be a probed driver on dev */ | |
| 3498 | + struct iommu_group *group = dev->iommu_group; | |
| 3499 | + struct iommu_attach_handle *entry; | |
| 3500 | + struct iommu_domain *curr_domain; | |
| 3501 | + void *curr; | |
| 3502 | + int ret; | |
| 3503 | + | |
| 3504 | + if (!group) | |
| 3505 | + return -ENODEV; | |
| 3506 | + | |
| 3507 | + if (!domain->ops->set_dev_pasid) | |
| 3508 | + return -EOPNOTSUPP; | |
| 3509 | + | |
| 3510 | + if (dev_iommu_ops(dev) != domain->owner || | |
| 3511 | + pasid == IOMMU_NO_PASID || !handle) | |
| 3512 | + return -EINVAL; | |
| 3513 | + | |
| 3514 | + mutex_lock(&group->mutex); | |
| 3515 | + entry = iommu_make_pasid_array_entry(domain, handle); | |
| 3516 | + curr = xa_cmpxchg(&group->pasid_array, pasid, NULL, | |
| 3517 | + XA_ZERO_ENTRY, GFP_KERNEL); | |
| 3518 | + if (xa_is_err(curr)) { | |
| 3519 | + ret = xa_err(curr); | |
| 3520 | + goto out_unlock; | |
| 3521 | + } | |
| 3522 | + | |
| 3523 | + /* | |
| 3524 | + * No domain (with or without handle) attached, hence not | |
| 3525 | + * a replace case. | |
| 3526 | + */ | |
| 3527 | + if (!curr) { | |
| 3528 | + xa_release(&group->pasid_array, pasid); | |
| 3529 | + ret = -EINVAL; | |
| 3530 | + goto out_unlock; | |
| 3531 | + } | |
| 3532 | + | |
| 3533 | + /* | |
| 3534 | + * Reusing handle is problematic as there are paths that refers | |
| 3535 | + * the handle without lock. To avoid race, reject the callers that | |
| 3536 | + * attempt it. | |
| 3537 | + */ | |
| 3538 | + if (curr == entry) { | |
| 3539 | + WARN_ON(1); | |
| 3540 | + ret = -EINVAL; | |
| 3541 | + goto out_unlock; | |
| 3542 | + } | |
| 3543 | + | |
| 3544 | + curr_domain = pasid_array_entry_to_domain(curr); | |
| 3545 | + ret = 0; | |
| 3546 | + | |
| 3547 | + if (curr_domain != domain) { | |
| 3548 | + ret = __iommu_set_group_pasid(domain, group, | |
| 3549 | + pasid, curr_domain); | |
| 3550 | + if (ret) | |
| 3551 | + goto out_unlock; | |
| 3552 | + } | |
| 3553 | + | |
| 3554 | + /* | |
| 3555 | + * The above xa_cmpxchg() reserved the memory, and the | |
| 3556 | + * group->mutex is held, this cannot fail. | |
| 3557 | + */ | |
| 3558 | + WARN_ON(xa_is_err(xa_store(&group->pasid_array, | |
| 3559 | + pasid, entry, GFP_KERNEL))); | |
| 3560 | + | |
| 3561 | +out_unlock: | |
| 3562 | + mutex_unlock(&group->mutex); | |
| 3563 | + return ret; | |
| 3564 | +} | |
| 3565 | +EXPORT_SYMBOL_NS_GPL(iommu_replace_device_pasid, "IOMMUFD_INTERNAL"); | |
| 3566 | + | |
| 3444 | 3567 | /* |
| 3445 | 3568 | * iommu_detach_device_pasid() - Detach the domain from pasid of device |
| 3446 | 3569 | * @domain: the iommu domain. |
| @@ -3536,6 +3659,9 @@ EXPORT_SYMBOL_NS_GPL(iommu_attach_handle_get, "IOMMUFD_INTERNAL"); · 89 unmodified lines | ||
| 3536 | 3659 | * This is a variant of iommu_attach_group(). It allows the caller to provide |
| 3537 | 3660 | * an attach handle and use it when the domain is attached. This is currently |
| 3538 | 3661 | * used by IOMMUFD to deliver the I/O page faults. |
| 3662 | + * | |
| 3663 | + * Caller should always provide a new handle to avoid race with the paths | |
| 3664 | + * that have lockless reference to handle. | |
| 3539 | 3665 | */ |
| 3540 | 3666 | int iommu_attach_group_handle(struct iommu_domain *domain, |
| 3541 | 3667 | struct iommu_group *group, |
| @@ -3605,6 +3731,9 @@ EXPORT_SYMBOL_NS_GPL(iommu_detach_group_handle, "IOMMUFD_INTERNAL"); · 63 unmodified lines | ||
| 3605 | 3731 | * |
| 3606 | 3732 | * If the currently attached domain is a core domain (e.g. a default_domain), |
| 3607 | 3733 | * it will act just like the iommu_attach_group_handle(). |
| 3734 | + * | |
| 3735 | + * Caller should always provide a new handle to avoid race with the paths | |
| 3736 | + * that have lockless reference to handle. | |
| 3608 | 3737 | */ |
| 3609 | 3738 | int iommu_replace_group_handle(struct iommu_group *group, |
| 3610 | 3739 | struct iommu_domain *new_domain, |
| @@ -3662,8 +3791,21 @@ int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_addr) · 51 unmodified lines | ||
| 3662 | 3791 | return 0; |
| 3663 | 3792 | |
| 3664 | 3793 | mutex_lock(&group->mutex); |
| 3665 | − if (group->domain && group->domain->sw_msi) | |
| 3666 | − ret = group->domain->sw_msi(group->domain, desc, msi_addr); | |
| 3794 | + /* An IDENTITY domain must pass through */ | |
| 3795 | + if (group->domain && group->domain->type != IOMMU_DOMAIN_IDENTITY) { | |
| 3796 | + switch (group->domain->cookie_type) { | |
| 3797 | + case IOMMU_COOKIE_DMA_MSI: | |
| 3798 | + case IOMMU_COOKIE_DMA_IOVA: | |
| 3799 | + ret = iommu_dma_sw_msi(group->domain, desc, msi_addr); | |
| 3800 | + break; | |
| 3801 | + case IOMMU_COOKIE_IOMMUFD: | |
| 3802 | + ret = iommufd_sw_msi(group->domain, desc, msi_addr); | |
| 3803 | + break; | |
| 3804 | + default: | |
| 3805 | + ret = -EOPNOTSUPP; | |
| 3806 | + break; | |
| 3807 | + } | |
| 3808 | + } | |
| 3667 | 3809 | mutex_unlock(&group->mutex); |
| 3668 | 3810 | return ret; |
| 3669 | 3811 | } |
| @@ -1,6 +1,6 @@ | ||
| 1 | 1 | # SPDX-License-Identifier: GPL-2.0-only |
| 2 | 2 | config IOMMUFD_DRIVER_CORE |
| 3 | − tristate | |
| 3 | + bool | |
| 4 | 4 | default (IOMMUFD_DRIVER || IOMMUFD) if IOMMUFD!=n |
| 5 | 5 | |
| 6 | 6 | config IOMMUFD |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | # SPDX-License-Identifier: GPL-2.0-only |
| 2 | 2 | iommufd-y := \ |
| 3 | 3 | device.o \ |
| 4 | − fault.o \ | |
| 4 | + eventq.o \ | |
| 5 | 5 | hw_pagetable.o \ |
| 6 | 6 | io_pagetable.o \ |
| 7 | 7 | ioas.o \ |
| @@ -3,9 +3,9 @@ · 2 unmodified lines | ||
| 3 | 3 | */ |
| 4 | 4 | #include <linux/iommu.h> |
| 5 | 5 | #include <linux/iommufd.h> |
| 6 | +#include <linux/pci-ats.h> | |
| 6 | 7 | #include <linux/slab.h> |
| 7 | 8 | #include <uapi/linux/iommufd.h> |
| 8 | −#include <linux/msi.h> | |
| 9 | 9 | |
| 10 | 10 | #include "../iommu-priv.h" |
| 11 | 11 | #include "io_pagetable.h" |
| @@ -18,12 +18,17 @@ MODULE_PARM_DESC( · 6 unmodified lines | ||
| 18 | 18 | "Allow IOMMUFD to bind to devices even if the platform cannot isolate " |
| 19 | 19 | "the MSI interrupt window. Enabling this is a security weakness."); |
| 20 | 20 | |
| 21 | +struct iommufd_attach { | |
| 22 | + struct iommufd_hw_pagetable *hwpt; | |
| 23 | + struct xarray device_array; | |
| 24 | +}; | |
| 25 | + | |
| 21 | 26 | static void iommufd_group_release(struct kref *kref) |
| 22 | 27 | { |
| 23 | 28 | struct iommufd_group *igroup = |
| 24 | 29 | container_of(kref, struct iommufd_group, ref); |
| 25 | 30 | |
| 26 | − WARN_ON(igroup->hwpt || !list_empty(&igroup->device_list)); | |
| 31 | + WARN_ON(!xa_empty(&igroup->pasid_attach)); | |
| 27 | 32 | |
| 28 | 33 | xa_cmpxchg(&igroup->ictx->groups, iommu_group_id(igroup->group), igroup, |
| 29 | 34 | NULL, GFP_KERNEL); |
| @@ -90,7 +95,7 @@ static struct iommufd_group *iommufd_get_group(struct iommufd_ctx *ictx, · 60 unmodified lines | ||
| 90 | 95 | |
| 91 | 96 | kref_init(&new_igroup->ref); |
| 92 | 97 | mutex_init(&new_igroup->lock); |
| 93 | − INIT_LIST_HEAD(&new_igroup->device_list); | |
| 98 | + xa_init(&new_igroup->pasid_attach); | |
| 94 | 99 | new_igroup->sw_msi_start = PHYS_ADDR_MAX; |
| 95 | 100 | /* group reference moves into new_igroup */ |
| 96 | 101 | new_igroup->group = group; |
| @@ -294,129 +299,24 @@ u32 iommufd_device_to_id(struct iommufd_device *idev) · 197 unmodified lines | ||
| 294 | 299 | } |
| 295 | 300 | EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, "IOMMUFD"); |
| 296 | 301 | |
| 297 | −/* | |
| 298 | − * Get a iommufd_sw_msi_map for the msi physical address requested by the irq | |
| 299 | − * layer. The mapping to IOVA is global to the iommufd file descriptor, every | |
| 300 | − * domain that is attached to a device using the same MSI parameters will use | |
| 301 | − * the same IOVA. | |
| 302 | − */ | |
| 303 | −static __maybe_unused struct iommufd_sw_msi_map * | |
| 304 | −iommufd_sw_msi_get_map(struct iommufd_ctx *ictx, phys_addr_t msi_addr, | |
| 305 | − phys_addr_t sw_msi_start) | |
| 306 | −{ | |
| 307 | − struct iommufd_sw_msi_map *cur; | |
| 308 | − unsigned int max_pgoff = 0; | |
| 309 | − | |
| 310 | − lockdep_assert_held(&ictx->sw_msi_lock); | |
| 311 | − | |
| 312 | − list_for_each_entry(cur, &ictx->sw_msi_list, sw_msi_item) { | |
| 313 | − if (cur->sw_msi_start != sw_msi_start) | |
| 314 | − continue; | |
| 315 | − max_pgoff = max(max_pgoff, cur->pgoff + 1); | |
| 316 | − if (cur->msi_addr == msi_addr) | |
| 317 | − return cur; | |
| 318 | − } | |
| 319 | − | |
| 320 | − if (ictx->sw_msi_id >= | |
| 321 | − BITS_PER_BYTE * sizeof_field(struct iommufd_sw_msi_maps, bitmap)) | |
| 322 | − return ERR_PTR(-EOVERFLOW); | |
| 323 | − | |
| 324 | − cur = kzalloc(sizeof(*cur), GFP_KERNEL); | |
| 325 | − if (!cur) | |
| 326 | − return ERR_PTR(-ENOMEM); | |
| 327 | − | |
| 328 | − cur->sw_msi_start = sw_msi_start; | |
| 329 | − cur->msi_addr = msi_addr; | |
| 330 | − cur->pgoff = max_pgoff; | |
| 331 | − cur->id = ictx->sw_msi_id++; | |
| 332 | − list_add_tail(&cur->sw_msi_item, &ictx->sw_msi_list); | |
| 333 | − return cur; | |
| 334 | −} | |
| 335 | − | |
| 336 | −static int iommufd_sw_msi_install(struct iommufd_ctx *ictx, | |
| 337 | − struct iommufd_hwpt_paging *hwpt_paging, | |
| 338 | − struct iommufd_sw_msi_map *msi_map) | |
| 302 | +static unsigned int iommufd_group_device_num(struct iommufd_group *igroup, | |
| 303 | + ioasid_t pasid) | |
| 339 | 304 | { |
| 340 | − unsigned long iova; | |
| 341 | − | |
| 342 | − lockdep_assert_held(&ictx->sw_msi_lock); | |
| 305 | + struct iommufd_attach *attach; | |
| 306 | + struct iommufd_device *idev; | |
| 307 | + unsigned int count = 0; | |
| 308 | + unsigned long index; | |
| 343 | 309 | |
| 344 | − iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE; | |
| 345 | − if (!test_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap)) { | |
| 346 | − int rc; | |
| 310 | + lockdep_assert_held(&igroup->lock); | |
| 347 | 311 | |
| 348 | − rc = iommu_map(hwpt_paging->common.domain, iova, | |
| 349 | − msi_map->msi_addr, PAGE_SIZE, | |
| 350 | − IOMMU_WRITE | IOMMU_READ | IOMMU_MMIO, | |
| 351 | − GFP_KERNEL_ACCOUNT); | |
| 352 | − if (rc) | |
| 353 | − return rc; | |
| 354 | − __set_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap); | |
| 355 | − } | |
| 356 | − return 0; | |
| 312 | + attach = xa_load(&igroup->pasid_attach, pasid); | |
| 313 | + if (attach) | |
| 314 | + xa_for_each(&attach->device_array, index, idev) | |
| 315 | + count++; | |
| 316 | + return count; | |
| 357 | 317 | } |
| 358 | 318 | |
| 359 | −/* | |
| 360 | − * Called by the irq code if the platform translates the MSI address through the | |
| 361 | − * IOMMU. msi_addr is the physical address of the MSI page. iommufd will | |
| 362 | − * allocate a fd global iova for the physical page that is the same on all | |
| 363 | − * domains and devices. | |
| 364 | − */ | |
| 365 | 319 | #ifdef CONFIG_IRQ_MSI_IOMMU |
| 366 | −int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | |
| 367 | − phys_addr_t msi_addr) | |
| 368 | −{ | |
| 369 | − struct device *dev = msi_desc_to_dev(desc); | |
| 370 | − struct iommufd_hwpt_paging *hwpt_paging; | |
| 371 | − struct iommu_attach_handle *raw_handle; | |
| 372 | − struct iommufd_attach_handle *handle; | |
| 373 | − struct iommufd_sw_msi_map *msi_map; | |
| 374 | − struct iommufd_ctx *ictx; | |
| 375 | − unsigned long iova; | |
| 376 | − int rc; | |
| 377 | − | |
| 378 | − /* | |
| 379 | − * It is safe to call iommu_attach_handle_get() here because the iommu | |
| 380 | − * core code invokes this under the group mutex which also prevents any | |
| 381 | − * change of the attach handle for the duration of this function. | |
| 382 | − */ | |
| 383 | − iommu_group_mutex_assert(dev); | |
| 384 | − | |
| 385 | − raw_handle = | |
| 386 | − iommu_attach_handle_get(dev->iommu_group, IOMMU_NO_PASID, 0); | |
| 387 | − if (IS_ERR(raw_handle)) | |
| 388 | − return 0; | |
| 389 | − hwpt_paging = find_hwpt_paging(domain->iommufd_hwpt); | |
| 390 | − | |
| 391 | − handle = to_iommufd_handle(raw_handle); | |
| 392 | − /* No IOMMU_RESV_SW_MSI means no change to the msi_msg */ | |
| 393 | − if (handle->idev->igroup->sw_msi_start == PHYS_ADDR_MAX) | |
| 394 | − return 0; | |
| 395 | − | |
| 396 | − ictx = handle->idev->ictx; | |
| 397 | − guard(mutex)(&ictx->sw_msi_lock); | |
| 398 | − /* | |
| 399 | − * The input msi_addr is the exact byte offset of the MSI doorbell, we | |
| 400 | − * assume the caller has checked that it is contained with a MMIO region | |
| 401 | − * that is secure to map at PAGE_SIZE. | |
| 402 | − */ | |
| 403 | − msi_map = iommufd_sw_msi_get_map(handle->idev->ictx, | |
| 404 | − msi_addr & PAGE_MASK, | |
| 405 | − handle->idev->igroup->sw_msi_start); | |
| 406 | − if (IS_ERR(msi_map)) | |
| 407 | − return PTR_ERR(msi_map); | |
| 408 | − | |
| 409 | − rc = iommufd_sw_msi_install(ictx, hwpt_paging, msi_map); | |
| 410 | − if (rc) | |
| 411 | − return rc; | |
| 412 | − __set_bit(msi_map->id, handle->idev->igroup->required_sw_msi.bitmap); | |
| 413 | − | |
| 414 | − iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE; | |
| 415 | − msi_desc_set_iommu_msi_iova(desc, iova, PAGE_SHIFT); | |
| 416 | − return 0; | |
| 417 | −} | |
| 418 | −#endif | |
| 419 | − | |
| 420 | 320 | static int iommufd_group_setup_msi(struct iommufd_group *igroup, |
| 421 | 321 | struct iommufd_hwpt_paging *hwpt_paging) |
| 422 | 322 | { |
| @@ -443,23 +343,39 @@ static int iommufd_group_setup_msi(struct iommufd_group *igroup, · 20 unmodified lines | ||
| 443 | 343 | } |
| 444 | 344 | return 0; |
| 445 | 345 | } |
| 346 | +#else | |
| 347 | +static inline int | |
| 348 | +iommufd_group_setup_msi(struct iommufd_group *igroup, | |
| 349 | + struct iommufd_hwpt_paging *hwpt_paging) | |
| 350 | +{ | |
| 351 | + return 0; | |
| 352 | +} | |
| 353 | +#endif | |
| 354 | + | |
| 355 | +static bool | |
| 356 | +iommufd_group_first_attach(struct iommufd_group *igroup, ioasid_t pasid) | |
| 357 | +{ | |
| 358 | + lockdep_assert_held(&igroup->lock); | |
| 359 | + return !xa_load(&igroup->pasid_attach, pasid); | |
| 360 | +} | |
| 446 | 361 | |
| 447 | 362 | static int |
| 448 | 363 | iommufd_device_attach_reserved_iova(struct iommufd_device *idev, |
| 449 | 364 | struct iommufd_hwpt_paging *hwpt_paging) |
| 450 | 365 | { |
| 366 | + struct iommufd_group *igroup = idev->igroup; | |
| 451 | 367 | int rc; |
| 452 | 368 | |
| 453 | − lockdep_assert_held(&idev->igroup->lock); | |
| 369 | + lockdep_assert_held(&igroup->lock); | |
| 454 | 370 | |
| 455 | 371 | rc = iopt_table_enforce_dev_resv_regions(&hwpt_paging->ioas->iopt, |
| 456 | 372 | idev->dev, |
| 457 | − &idev->igroup->sw_msi_start); | |
| 373 | + &igroup->sw_msi_start); | |
| 458 | 374 | if (rc) |
| 459 | 375 | return rc; |
| 460 | 376 | |
| 461 | − if (list_empty(&idev->igroup->device_list)) { | |
| 462 | − rc = iommufd_group_setup_msi(idev->igroup, hwpt_paging); | |
| 377 | + if (iommufd_group_first_attach(igroup, IOMMU_NO_PASID)) { | |
| 378 | + rc = iommufd_group_setup_msi(igroup, hwpt_paging); | |
| 463 | 379 | if (rc) { |
| 464 | 380 | iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, |
| 465 | 381 | idev->dev); |
| @@ -471,13 +387,54 @@ iommufd_device_attach_reserved_iova(struct iommufd_device *idev, · 5 unmodified lines | ||
| 471 | 387 | |
| 472 | 388 | /* The device attach/detach/replace helpers for attach_handle */ |
| 473 | 389 | |
| 390 | +static bool iommufd_device_is_attached(struct iommufd_device *idev, | |
| 391 | + ioasid_t pasid) | |
| 392 | +{ | |
| 393 | + struct iommufd_attach *attach; | |
| 394 | + | |
| 395 | + attach = xa_load(&idev->igroup->pasid_attach, pasid); | |
| 396 | + return xa_load(&attach->device_array, idev->obj.id); | |
| 397 | +} | |
| 398 | + | |
| 399 | +static int iommufd_hwpt_pasid_compat(struct iommufd_hw_pagetable *hwpt, | |
| 400 | + struct iommufd_device *idev, | |
| 401 | + ioasid_t pasid) | |
| 402 | +{ | |
| 403 | + struct iommufd_group *igroup = idev->igroup; | |
| 404 | + | |
| 405 | + lockdep_assert_held(&igroup->lock); | |
| 406 | + | |
| 407 | + if (pasid == IOMMU_NO_PASID) { | |
| 408 | + unsigned long start = IOMMU_NO_PASID; | |
| 409 | + | |
| 410 | + if (!hwpt->pasid_compat && | |
| 411 | + xa_find_after(&igroup->pasid_attach, | |
| 412 | + &start, UINT_MAX, XA_PRESENT)) | |
| 413 | + return -EINVAL; | |
| 414 | + } else { | |
| 415 | + struct iommufd_attach *attach; | |
| 416 | + | |
| 417 | + if (!hwpt->pasid_compat) | |
| 418 | + return -EINVAL; | |
| 419 | + | |
| 420 | + attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID); | |
| 421 | + if (attach && attach->hwpt && !attach->hwpt->pasid_compat) | |
| 422 | + return -EINVAL; | |
| 423 | + } | |
| 424 | + | |
| 425 | + return 0; | |
| 426 | +} | |
| 427 | + | |
| 474 | 428 | static int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *hwpt, |
| 475 | − struct iommufd_device *idev) | |
| 429 | + struct iommufd_device *idev, | |
| 430 | + ioasid_t pasid) | |
| 476 | 431 | { |
| 477 | 432 | struct iommufd_attach_handle *handle; |
| 478 | 433 | int rc; |
| 479 | 434 | |
| 480 | − lockdep_assert_held(&idev->igroup->lock); | |
| 435 | + rc = iommufd_hwpt_pasid_compat(hwpt, idev, pasid); | |
| 436 | + if (rc) | |
| 437 | + return rc; | |
| 481 | 438 | |
| 482 | 439 | handle = kzalloc(sizeof(*handle), GFP_KERNEL); |
| 483 | 440 | if (!handle) |
| @@ -490,8 +447,12 @@ static int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *hwpt, · 6 unmodified lines | ||
| 490 | 447 | } |
| 491 | 448 | |
| 492 | 449 | handle->idev = idev; |
| 493 | − rc = iommu_attach_group_handle(hwpt->domain, idev->igroup->group, | |
| 494 | − &handle->handle); | |
| 450 | + if (pasid == IOMMU_NO_PASID) | |
| 451 | + rc = iommu_attach_group_handle(hwpt->domain, idev->igroup->group, | |
| 452 | + &handle->handle); | |
| 453 | + else | |
| 454 | + rc = iommu_attach_device_pasid(hwpt->domain, idev->dev, pasid, | |
| 455 | + &handle->handle); | |
| 495 | 456 | if (rc) |
| 496 | 457 | goto out_disable_iopf; |
| 497 | 458 | |
| @@ -506,26 +467,31 @@ out_free_handle: · 8 unmodified lines | ||
| 506 | 467 | } |
| 507 | 468 | |
| 508 | 469 | static struct iommufd_attach_handle * |
| 509 | −iommufd_device_get_attach_handle(struct iommufd_device *idev) | |
| 470 | +iommufd_device_get_attach_handle(struct iommufd_device *idev, ioasid_t pasid) | |
| 510 | 471 | { |
| 511 | 472 | struct iommu_attach_handle *handle; |
| 512 | 473 | |
| 513 | 474 | lockdep_assert_held(&idev->igroup->lock); |
| 514 | 475 | |
| 515 | 476 | handle = |
| 516 | − iommu_attach_handle_get(idev->igroup->group, IOMMU_NO_PASID, 0); | |
| 477 | + iommu_attach_handle_get(idev->igroup->group, pasid, 0); | |
| 517 | 478 | if (IS_ERR(handle)) |
| 518 | 479 | return NULL; |
| 519 | 480 | return to_iommufd_handle(handle); |
| 520 | 481 | } |
| 521 | 482 | |
| 522 | 483 | static void iommufd_hwpt_detach_device(struct iommufd_hw_pagetable *hwpt, |
| 523 | − struct iommufd_device *idev) | |
| 484 | + struct iommufd_device *idev, | |
| 485 | + ioasid_t pasid) | |
| 524 | 486 | { |
| 525 | 487 | struct iommufd_attach_handle *handle; |
| 526 | 488 | |
| 527 | − handle = iommufd_device_get_attach_handle(idev); | |
| 528 | − iommu_detach_group_handle(hwpt->domain, idev->igroup->group); | |
| 489 | + handle = iommufd_device_get_attach_handle(idev, pasid); | |
| 490 | + if (pasid == IOMMU_NO_PASID) | |
| 491 | + iommu_detach_group_handle(hwpt->domain, idev->igroup->group); | |
| 492 | + else | |
| 493 | + iommu_detach_device_pasid(hwpt->domain, idev->dev, pasid); | |
| 494 | + | |
| 529 | 495 | if (hwpt->fault) { |
| 530 | 496 | iommufd_auto_response_faults(hwpt, handle); |
| 531 | 497 | iommufd_fault_iopf_disable(idev); |
| @@ -534,13 +500,19 @@ static void iommufd_hwpt_detach_device(struct iommufd_hw_pagetable *hwpt, · 2 unmodified lines | ||
| 534 | 500 | } |
| 535 | 501 | |
| 536 | 502 | static int iommufd_hwpt_replace_device(struct iommufd_device *idev, |
| 503 | + ioasid_t pasid, | |
| 537 | 504 | struct iommufd_hw_pagetable *hwpt, |
| 538 | 505 | struct iommufd_hw_pagetable *old) |
| 539 | 506 | { |
| 540 | − struct iommufd_attach_handle *handle, *old_handle = | |
| 541 | − iommufd_device_get_attach_handle(idev); | |
| 507 | + struct iommufd_attach_handle *handle, *old_handle; | |
| 542 | 508 | int rc; |
| 543 | 509 | |
| 510 | + rc = iommufd_hwpt_pasid_compat(hwpt, idev, pasid); | |
| 511 | + if (rc) | |
| 512 | + return rc; | |
| 513 | + | |
| 514 | + old_handle = iommufd_device_get_attach_handle(idev, pasid); | |
| 515 | + | |
| 544 | 516 | handle = kzalloc(sizeof(*handle), GFP_KERNEL); |
| 545 | 517 | if (!handle) |
| 546 | 518 | return -ENOMEM; |
| @@ -552,8 +524,12 @@ static int iommufd_hwpt_replace_device(struct iommufd_device *idev, · 5 unmodified lines | ||
| 552 | 524 | } |
| 553 | 525 | |
| 554 | 526 | handle->idev = idev; |
| 555 | − rc = iommu_replace_group_handle(idev->igroup->group, hwpt->domain, | |
| 556 | − &handle->handle); | |
| 527 | + if (pasid == IOMMU_NO_PASID) | |
| 528 | + rc = iommu_replace_group_handle(idev->igroup->group, | |
| 529 | + hwpt->domain, &handle->handle); | |
| 530 | + else | |
| 531 | + rc = iommu_replace_device_pasid(hwpt->domain, idev->dev, | |
| 532 | + pasid, &handle->handle); | |
| 557 | 533 | if (rc) |
| 558 | 534 | goto out_disable_iopf; |
| 559 | 535 | |
| @@ -575,22 +551,51 @@ out_free_handle: · 15 unmodified lines | ||
| 575 | 551 | } |
| 576 | 552 | |
| 577 | 553 | int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt, |
| 578 | − struct iommufd_device *idev) | |
| 554 | + struct iommufd_device *idev, ioasid_t pasid) | |
| 579 | 555 | { |
| 580 | 556 | struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt); |
| 557 | + bool attach_resv = hwpt_paging && pasid == IOMMU_NO_PASID; | |
| 558 | + struct iommufd_group *igroup = idev->igroup; | |
| 559 | + struct iommufd_hw_pagetable *old_hwpt; | |
| 560 | + struct iommufd_attach *attach; | |
| 581 | 561 | int rc; |
| 582 | 562 | |
| 583 | − mutex_lock(&idev->igroup->lock); | |
| 563 | + mutex_lock(&igroup->lock); | |
| 584 | 564 | |
| 585 | − if (idev->igroup->hwpt != NULL && idev->igroup->hwpt != hwpt) { | |
| 586 | − rc = -EINVAL; | |
| 565 | + attach = xa_cmpxchg(&igroup->pasid_attach, pasid, NULL, | |
| 566 | + XA_ZERO_ENTRY, GFP_KERNEL); | |
| 567 | + if (xa_is_err(attach)) { | |
| 568 | + rc = xa_err(attach); | |
| 587 | 569 | goto err_unlock; |
| 588 | 570 | } |
| 589 | 571 | |
| 590 | − if (hwpt_paging) { | |
| 572 | + if (!attach) { | |
| 573 | + attach = kzalloc(sizeof(*attach), GFP_KERNEL); | |
| 574 | + if (!attach) { | |
| 575 | + rc = -ENOMEM; | |
| 576 | + goto err_release_pasid; | |
| 577 | + } | |
| 578 | + xa_init(&attach->device_array); | |
| 579 | + } | |
| 580 | + | |
| 581 | + old_hwpt = attach->hwpt; | |
| 582 | + | |
| 583 | + rc = xa_insert(&attach->device_array, idev->obj.id, XA_ZERO_ENTRY, | |
| 584 | + GFP_KERNEL); | |
| 585 | + if (rc) { | |
| 586 | + WARN_ON(rc == -EBUSY && !old_hwpt); | |
| 587 | + goto err_free_attach; | |
| 588 | + } | |
| 589 | + | |
| 590 | + if (old_hwpt && old_hwpt != hwpt) { | |
| 591 | + rc = -EINVAL; | |
| 592 | + goto err_release_devid; | |
| 593 | + } | |
| 594 | + | |
| 595 | + if (attach_resv) { | |
| 591 | 596 | rc = iommufd_device_attach_reserved_iova(idev, hwpt_paging); |
| 592 | 597 | if (rc) |
| 593 | − goto err_unlock; | |
| 598 | + goto err_release_devid; | |
| 594 | 599 | } |
| 595 | 600 | |
| 596 | 601 | /* |
| @@ -600,51 +605,74 @@ int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt, · 3 unmodified lines | ||
| 600 | 605 | * reserved regions are only updated during individual device |
| 601 | 606 | * attachment. |
| 602 | 607 | */ |
| 603 | − if (list_empty(&idev->igroup->device_list)) { | |
| 604 | − rc = iommufd_hwpt_attach_device(hwpt, idev); | |
| 608 | + if (iommufd_group_first_attach(igroup, pasid)) { | |
| 609 | + rc = iommufd_hwpt_attach_device(hwpt, idev, pasid); | |
| 605 | 610 | if (rc) |
| 606 | 611 | goto err_unresv; |
| 607 | − idev->igroup->hwpt = hwpt; | |
| 612 | + attach->hwpt = hwpt; | |
| 613 | + WARN_ON(xa_is_err(xa_store(&igroup->pasid_attach, pasid, attach, | |
| 614 | + GFP_KERNEL))); | |
| 608 | 615 | } |
| 609 | 616 | refcount_inc(&hwpt->obj.users); |
| 610 | − list_add_tail(&idev->group_item, &idev->igroup->device_list); | |
| 611 | − mutex_unlock(&idev->igroup->lock); | |
| 617 | + WARN_ON(xa_is_err(xa_store(&attach->device_array, idev->obj.id, | |
| 618 | + idev, GFP_KERNEL))); | |
| 619 | + mutex_unlock(&igroup->lock); | |
| 612 | 620 | return 0; |
| 613 | 621 | err_unresv: |
| 614 | − if (hwpt_paging) | |
| 622 | + if (attach_resv) | |
| 615 | 623 | iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, idev->dev); |
| 624 | +err_release_devid: | |
| 625 | + xa_release(&attach->device_array, idev->obj.id); | |
| 626 | +err_free_attach: | |
| 627 | + if (iommufd_group_first_attach(igroup, pasid)) | |
| 628 | + kfree(attach); | |
| 629 | +err_release_pasid: | |
| 630 | + if (iommufd_group_first_attach(igroup, pasid)) | |
| 631 | + xa_release(&igroup->pasid_attach, pasid); | |
| 616 | 632 | err_unlock: |
| 617 | − mutex_unlock(&idev->igroup->lock); | |
| 633 | + mutex_unlock(&igroup->lock); | |
| 618 | 634 | return rc; |
| 619 | 635 | } |
| 620 | 636 | |
| 621 | 637 | struct iommufd_hw_pagetable * |
| 622 | −iommufd_hw_pagetable_detach(struct iommufd_device *idev) | |
| 638 | +iommufd_hw_pagetable_detach(struct iommufd_device *idev, ioasid_t pasid) | |
| 623 | 639 | { |
| 624 | − struct iommufd_hw_pagetable *hwpt = idev->igroup->hwpt; | |
| 625 | − struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt); | |
| 640 | + struct iommufd_group *igroup = idev->igroup; | |
| 641 | + struct iommufd_hwpt_paging *hwpt_paging; | |
| 642 | + struct iommufd_hw_pagetable *hwpt; | |
| 643 | + struct iommufd_attach *attach; | |
| 644 | + | |
| 645 | + mutex_lock(&igroup->lock); | |
| 646 | + attach = xa_load(&igroup->pasid_attach, pasid); | |
| 647 | + if (!attach) { | |
| 648 | + mutex_unlock(&igroup->lock); | |
| 649 | + return NULL; | |
| 650 | + } | |
| 626 | 651 | |
| 627 | − mutex_lock(&idev->igroup->lock); | |
| 628 | − list_del(&idev->group_item); | |
| 629 | − if (list_empty(&idev->igroup->device_list)) { | |
| 630 | − iommufd_hwpt_detach_device(hwpt, idev); | |
| 631 | − idev->igroup->hwpt = NULL; | |
| 652 | + hwpt = attach->hwpt; | |
| 653 | + hwpt_paging = find_hwpt_paging(hwpt); | |
| 654 | + | |
| 655 | + xa_erase(&attach->device_array, idev->obj.id); | |
| 656 | + if (xa_empty(&attach->device_array)) { | |
| 657 | + iommufd_hwpt_detach_device(hwpt, idev, pasid); | |
| 658 | + xa_erase(&igroup->pasid_attach, pasid); | |
| 659 | + kfree(attach); | |
| 632 | 660 | } |
| 633 | − if (hwpt_paging) | |
| 661 | + if (hwpt_paging && pasid == IOMMU_NO_PASID) | |
| 634 | 662 | iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, idev->dev); |
| 635 | − mutex_unlock(&idev->igroup->lock); | |
| 663 | + mutex_unlock(&igroup->lock); | |
| 636 | 664 | |
| 637 | 665 | /* Caller must destroy hwpt */ |
| 638 | 666 | return hwpt; |
| 639 | 667 | } |
| 640 | 668 | |
| 641 | 669 | static struct iommufd_hw_pagetable * |
| 642 | −iommufd_device_do_attach(struct iommufd_device *idev, | |
| 670 | +iommufd_device_do_attach(struct iommufd_device *idev, ioasid_t pasid, | |
| 643 | 671 | struct iommufd_hw_pagetable *hwpt) |
| 644 | 672 | { |
| 645 | 673 | int rc; |
| 646 | 674 | |
| 647 | − rc = iommufd_hw_pagetable_attach(hwpt, idev); | |
| 675 | + rc = iommufd_hw_pagetable_attach(hwpt, idev, pasid); | |
| 648 | 676 | if (rc) |
| 649 | 677 | return ERR_PTR(rc); |
| 650 | 678 | return NULL; |
| @@ -654,11 +682,14 @@ static void · 3 unmodified lines | ||
| 654 | 682 | iommufd_group_remove_reserved_iova(struct iommufd_group *igroup, |
| 655 | 683 | struct iommufd_hwpt_paging *hwpt_paging) |
| 656 | 684 | { |
| 685 | + struct iommufd_attach *attach; | |
| 657 | 686 | struct iommufd_device *cur; |
| 687 | + unsigned long index; | |
| 658 | 688 | |
| 659 | 689 | lockdep_assert_held(&igroup->lock); |
| 660 | 690 | |
| 661 | − list_for_each_entry(cur, &igroup->device_list, group_item) | |
| 691 | + attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID); | |
| 692 | + xa_for_each(&attach->device_array, index, cur) | |
| 662 | 693 | iopt_remove_reserved_iova(&hwpt_paging->ioas->iopt, cur->dev); |
| 663 | 694 | } |
| 664 | 695 | |
| @@ -667,14 +698,17 @@ iommufd_group_do_replace_reserved_iova(struct iommufd_group *igroup, · 2 unmodified lines | ||
| 667 | 698 | struct iommufd_hwpt_paging *hwpt_paging) |
| 668 | 699 | { |
| 669 | 700 | struct iommufd_hwpt_paging *old_hwpt_paging; |
| 701 | + struct iommufd_attach *attach; | |
| 670 | 702 | struct iommufd_device *cur; |
| 703 | + unsigned long index; | |
| 671 | 704 | int rc; |
| 672 | 705 | |
| 673 | 706 | lockdep_assert_held(&igroup->lock); |
| 674 | 707 | |
| 675 | − old_hwpt_paging = find_hwpt_paging(igroup->hwpt); | |
| 708 | + attach = xa_load(&igroup->pasid_attach, IOMMU_NO_PASID); | |
| 709 | + old_hwpt_paging = find_hwpt_paging(attach->hwpt); | |
| 676 | 710 | if (!old_hwpt_paging || hwpt_paging->ioas != old_hwpt_paging->ioas) { |
| 677 | − list_for_each_entry(cur, &igroup->device_list, group_item) { | |
| 711 | + xa_for_each(&attach->device_array, index, cur) { | |
| 678 | 712 | rc = iopt_table_enforce_dev_resv_regions( |
| 679 | 713 | &hwpt_paging->ioas->iopt, cur->dev, NULL); |
| 680 | 714 | if (rc) |
| @@ -693,69 +727,81 @@ err_unresv: · 12 unmodified lines | ||
| 693 | 727 | } |
| 694 | 728 | |
| 695 | 729 | static struct iommufd_hw_pagetable * |
| 696 | −iommufd_device_do_replace(struct iommufd_device *idev, | |
| 730 | +iommufd_device_do_replace(struct iommufd_device *idev, ioasid_t pasid, | |
| 697 | 731 | struct iommufd_hw_pagetable *hwpt) |
| 698 | 732 | { |
| 699 | 733 | struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt); |
| 734 | + bool attach_resv = hwpt_paging && pasid == IOMMU_NO_PASID; | |
| 700 | 735 | struct iommufd_hwpt_paging *old_hwpt_paging; |
| 701 | 736 | struct iommufd_group *igroup = idev->igroup; |
| 702 | 737 | struct iommufd_hw_pagetable *old_hwpt; |
| 738 | + struct iommufd_attach *attach; | |
| 703 | 739 | unsigned int num_devices; |
| 704 | 740 | int rc; |
| 705 | 741 | |
| 706 | − mutex_lock(&idev->igroup->lock); | |
| 742 | + mutex_lock(&igroup->lock); | |
| 743 | + | |
| 744 | + attach = xa_load(&igroup->pasid_attach, pasid); | |
| 745 | + if (!attach) { | |
| 746 | + rc = -EINVAL; | |
| 747 | + goto err_unlock; | |
| 748 | + } | |
| 749 | + | |
| 750 | + old_hwpt = attach->hwpt; | |
| 707 | 751 | |
| 708 | − if (igroup->hwpt == NULL) { | |
| 752 | + WARN_ON(!old_hwpt || xa_empty(&attach->device_array)); | |
| 753 | + | |
| 754 | + if (!iommufd_device_is_attached(idev, pasid)) { | |
| 709 | 755 | rc = -EINVAL; |
| 710 | 756 | goto err_unlock; |
| 711 | 757 | } |
| 712 | 758 | |
| 713 | − if (hwpt == igroup->hwpt) { | |
| 714 | − mutex_unlock(&idev->igroup->lock); | |
| 759 | + if (hwpt == old_hwpt) { | |
| 760 | + mutex_unlock(&igroup->lock); | |
| 715 | 761 | return NULL; |
| 716 | 762 | } |
| 717 | 763 | |
| 718 | − old_hwpt = igroup->hwpt; | |
| 719 | − if (hwpt_paging) { | |
| 764 | + if (attach_resv) { | |
| 720 | 765 | rc = iommufd_group_do_replace_reserved_iova(igroup, hwpt_paging); |
| 721 | 766 | if (rc) |
| 722 | 767 | goto err_unlock; |
| 723 | 768 | } |
| 724 | 769 | |
| 725 | − rc = iommufd_hwpt_replace_device(idev, hwpt, old_hwpt); | |
| 770 | + rc = iommufd_hwpt_replace_device(idev, pasid, hwpt, old_hwpt); | |
| 726 | 771 | if (rc) |
| 727 | 772 | goto err_unresv; |
| 728 | 773 | |
| 729 | 774 | old_hwpt_paging = find_hwpt_paging(old_hwpt); |
| 730 | − if (old_hwpt_paging && | |
| 775 | + if (old_hwpt_paging && pasid == IOMMU_NO_PASID && | |
| 731 | 776 | (!hwpt_paging || hwpt_paging->ioas != old_hwpt_paging->ioas)) |
| 732 | 777 | iommufd_group_remove_reserved_iova(igroup, old_hwpt_paging); |
| 733 | 778 | |
| 734 | − igroup->hwpt = hwpt; | |
| 779 | + attach->hwpt = hwpt; | |
| 735 | 780 | |
| 736 | − num_devices = list_count_nodes(&igroup->device_list); | |
| 781 | + num_devices = iommufd_group_device_num(igroup, pasid); | |
| 737 | 782 | /* |
| 738 | − * Move the refcounts held by the device_list to the new hwpt. Retain a | |
| 783 | + * Move the refcounts held by the device_array to the new hwpt. Retain a | |
| 739 | 784 | * refcount for this thread as the caller will free it. |
| 740 | 785 | */ |
| 741 | 786 | refcount_add(num_devices, &hwpt->obj.users); |
| 742 | 787 | if (num_devices > 1) |
| 743 | 788 | WARN_ON(refcount_sub_and_test(num_devices - 1, |
| 744 | 789 | &old_hwpt->obj.users)); |
| 745 | − mutex_unlock(&idev->igroup->lock); | |
| 790 | + mutex_unlock(&igroup->lock); | |
| 746 | 791 | |
| 747 | 792 | /* Caller must destroy old_hwpt */ |
| 748 | 793 | return old_hwpt; |
| 749 | 794 | err_unresv: |
| 750 | − if (hwpt_paging) | |
| 795 | + if (attach_resv) | |
| 751 | 796 | iommufd_group_remove_reserved_iova(igroup, hwpt_paging); |
| 752 | 797 | err_unlock: |
| 753 | − mutex_unlock(&idev->igroup->lock); | |
| 798 | + mutex_unlock(&igroup->lock); | |
| 754 | 799 | return ERR_PTR(rc); |
| 755 | 800 | } |
| 756 | 801 | |
| 757 | 802 | typedef struct iommufd_hw_pagetable *(*attach_fn)( |
| 758 | − struct iommufd_device *idev, struct iommufd_hw_pagetable *hwpt); | |
| 803 | + struct iommufd_device *idev, ioasid_t pasid, | |
| 804 | + struct iommufd_hw_pagetable *hwpt); | |
| 759 | 805 | |
| 760 | 806 | /* |
| 761 | 807 | * When automatically managing the domains we search for a compatible domain in |
| @@ -763,7 +809,7 @@ typedef struct iommufd_hw_pagetable *(*attach_fn)( · 1 unmodified line | ||
| 763 | 809 | * Automatic domain selection will never pick a manually created domain. |
| 764 | 810 | */ |
| 765 | 811 | static struct iommufd_hw_pagetable * |
| 766 | −iommufd_device_auto_get_domain(struct iommufd_device *idev, | |
| 812 | +iommufd_device_auto_get_domain(struct iommufd_device *idev, ioasid_t pasid, | |
| 767 | 813 | struct iommufd_ioas *ioas, u32 *pt_id, |
| 768 | 814 | attach_fn do_attach) |
| 769 | 815 | { |
| @@ -792,7 +838,7 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, · 22 unmodified lines | ||
| 792 | 838 | hwpt = &hwpt_paging->common; |
| 793 | 839 | if (!iommufd_lock_obj(&hwpt->obj)) |
| 794 | 840 | continue; |
| 795 | − destroy_hwpt = (*do_attach)(idev, hwpt); | |
| 841 | + destroy_hwpt = (*do_attach)(idev, pasid, hwpt); | |
| 796 | 842 | if (IS_ERR(destroy_hwpt)) { |
| 797 | 843 | iommufd_put_object(idev->ictx, &hwpt->obj); |
| 798 | 844 | /* |
| @@ -810,8 +856,8 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, · 11 unmodified lines | ||
| 810 | 856 | goto out_unlock; |
| 811 | 857 | } |
| 812 | 858 | |
| 813 | − hwpt_paging = iommufd_hwpt_paging_alloc(idev->ictx, ioas, idev, 0, | |
| 814 | − immediate_attach, NULL); | |
| 859 | + hwpt_paging = iommufd_hwpt_paging_alloc(idev->ictx, ioas, idev, pasid, | |
| 860 | + 0, immediate_attach, NULL); | |
| 815 | 861 | if (IS_ERR(hwpt_paging)) { |
| 816 | 862 | destroy_hwpt = ERR_CAST(hwpt_paging); |
| 817 | 863 | goto out_unlock; |
| @@ -819,7 +865,7 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev, · 1 unmodified line | ||
| 819 | 865 | hwpt = &hwpt_paging->common; |
| 820 | 866 | |
| 821 | 867 | if (!immediate_attach) { |
| 822 | − destroy_hwpt = (*do_attach)(idev, hwpt); | |
| 868 | + destroy_hwpt = (*do_attach)(idev, pasid, hwpt); | |
| 823 | 869 | if (IS_ERR(destroy_hwpt)) |
| 824 | 870 | goto out_abort; |
| 825 | 871 | } else { |
| @@ -840,8 +886,9 @@ out_unlock: · 14 unmodified lines | ||
| 840 | 886 | return destroy_hwpt; |
| 841 | 887 | } |
| 842 | 888 | |
| 843 | −static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id, | |
| 844 | − attach_fn do_attach) | |
| 889 | +static int iommufd_device_change_pt(struct iommufd_device *idev, | |
| 890 | + ioasid_t pasid, | |
| 891 | + u32 *pt_id, attach_fn do_attach) | |
| 845 | 892 | { |
| 846 | 893 | struct iommufd_hw_pagetable *destroy_hwpt; |
| 847 | 894 | struct iommufd_object *pt_obj; |
| @@ -856,7 +903,7 @@ static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id, · 8 unmodified lines | ||
| 856 | 903 | struct iommufd_hw_pagetable *hwpt = |
| 857 | 904 | container_of(pt_obj, struct iommufd_hw_pagetable, obj); |
| 858 | 905 | |
| 859 | − destroy_hwpt = (*do_attach)(idev, hwpt); | |
| 906 | + destroy_hwpt = (*do_attach)(idev, pasid, hwpt); | |
| 860 | 907 | if (IS_ERR(destroy_hwpt)) |
| 861 | 908 | goto out_put_pt_obj; |
| 862 | 909 | break; |
| @@ -865,8 +912,8 @@ static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id, · 2 unmodified lines | ||
| 865 | 912 | struct iommufd_ioas *ioas = |
| 866 | 913 | container_of(pt_obj, struct iommufd_ioas, obj); |
| 867 | 914 | |
| 868 | − destroy_hwpt = iommufd_device_auto_get_domain(idev, ioas, pt_id, | |
| 869 | − do_attach); | |
| 915 | + destroy_hwpt = iommufd_device_auto_get_domain(idev, pasid, ioas, | |
| 916 | + pt_id, do_attach); | |
| 870 | 917 | if (IS_ERR(destroy_hwpt)) |
| 871 | 918 | goto out_put_pt_obj; |
| 872 | 919 | break; |
| @@ -888,22 +935,26 @@ out_put_pt_obj: · 15 unmodified lines | ||
| 888 | 935 | } |
| 889 | 936 | |
| 890 | 937 | /** |
| 891 | − * iommufd_device_attach - Connect a device to an iommu_domain | |
| 938 | + * iommufd_device_attach - Connect a device/pasid to an iommu_domain | |
| 892 | 939 | * @idev: device to attach |
| 940 | + * @pasid: pasid to attach | |
| 893 | 941 | * @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING |
| 894 | 942 | * Output the IOMMUFD_OBJ_HWPT_PAGING ID |
| 895 | 943 | * |
| 896 | − * This connects the device to an iommu_domain, either automatically or manually | |
| 897 | − * selected. Once this completes the device could do DMA. | |
| 944 | + * This connects the device/pasid to an iommu_domain, either automatically | |
| 945 | + * or manually selected. Once this completes the device could do DMA with | |
| 946 | + * @pasid. @pasid is IOMMU_NO_PASID if this attach is for no pasid usage. | |
| 898 | 947 | * |
| 899 | 948 | * The caller should return the resulting pt_id back to userspace. |
| 900 | 949 | * This function is undone by calling iommufd_device_detach(). |
| 901 | 950 | */ |
| 902 | −int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id) | |
| 951 | +int iommufd_device_attach(struct iommufd_device *idev, ioasid_t pasid, | |
| 952 | + u32 *pt_id) | |
| 903 | 953 | { |
| 904 | 954 | int rc; |
| 905 | 955 | |
| 906 | − rc = iommufd_device_change_pt(idev, pt_id, &iommufd_device_do_attach); | |
| 956 | + rc = iommufd_device_change_pt(idev, pasid, pt_id, | |
| 957 | + &iommufd_device_do_attach); | |
| 907 | 958 | if (rc) |
| 908 | 959 | return rc; |
| 909 | 960 | |
| @@ -917,8 +968,9 @@ int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id) · 7 unmodified lines | ||
| 917 | 968 | EXPORT_SYMBOL_NS_GPL(iommufd_device_attach, "IOMMUFD"); |
| 918 | 969 | |
| 919 | 970 | /** |
| 920 | − * iommufd_device_replace - Change the device's iommu_domain | |
| 971 | + * iommufd_device_replace - Change the device/pasid's iommu_domain | |
| 921 | 972 | * @idev: device to change |
| 973 | + * @pasid: pasid to change | |
| 922 | 974 | * @pt_id: Input a IOMMUFD_OBJ_IOAS, or IOMMUFD_OBJ_HWPT_PAGING |
| 923 | 975 | * Output the IOMMUFD_OBJ_HWPT_PAGING ID |
| 924 | 976 | * |
| @@ -929,27 +981,33 @@ EXPORT_SYMBOL_NS_GPL(iommufd_device_attach, "IOMMUFD"); · 4 unmodified lines | ||
| 929 | 981 | * |
| 930 | 982 | * If it fails then no change is made to the attachment. The iommu driver may |
| 931 | 983 | * implement this so there is no disruption in translation. This can only be |
| 932 | − * called if iommufd_device_attach() has already succeeded. | |
| 984 | + * called if iommufd_device_attach() has already succeeded. @pasid is | |
| 985 | + * IOMMU_NO_PASID for no pasid usage. | |
| 933 | 986 | */ |
| 934 | −int iommufd_device_replace(struct iommufd_device *idev, u32 *pt_id) | |
| 987 | +int iommufd_device_replace(struct iommufd_device *idev, ioasid_t pasid, | |
| 988 | + u32 *pt_id) | |
| 935 | 989 | { |
| 936 | − return iommufd_device_change_pt(idev, pt_id, | |
| 990 | + return iommufd_device_change_pt(idev, pasid, pt_id, | |
| 937 | 991 | &iommufd_device_do_replace); |
| 938 | 992 | } |
| 939 | 993 | EXPORT_SYMBOL_NS_GPL(iommufd_device_replace, "IOMMUFD"); |
| 940 | 994 | |
| 941 | 995 | /** |
| 942 | − * iommufd_device_detach - Disconnect a device to an iommu_domain | |
| 996 | + * iommufd_device_detach - Disconnect a device/device to an iommu_domain | |
| 943 | 997 | * @idev: device to detach |
| 998 | + * @pasid: pasid to detach | |
| 944 | 999 | * |
| 945 | 1000 | * Undo iommufd_device_attach(). This disconnects the idev from the previously |
| 946 | 1001 | * attached pt_id. The device returns back to a blocked DMA translation. |
| 1002 | + * @pasid is IOMMU_NO_PASID for no pasid usage. | |
| 947 | 1003 | */ |
| 948 | −void iommufd_device_detach(struct iommufd_device *idev) | |
| 1004 | +void iommufd_device_detach(struct iommufd_device *idev, ioasid_t pasid) | |
| 949 | 1005 | { |
| 950 | 1006 | struct iommufd_hw_pagetable *hwpt; |
| 951 | 1007 | |
| 952 | − hwpt = iommufd_hw_pagetable_detach(idev); | |
| 1008 | + hwpt = iommufd_hw_pagetable_detach(idev, pasid); | |
| 1009 | + if (!hwpt) | |
| 1010 | + return; | |
| 953 | 1011 | iommufd_hw_pagetable_put(idev->ictx, hwpt); |
| 954 | 1012 | refcount_dec(&idev->obj.users); |
| 955 | 1013 | } |
| @@ -1349,7 +1407,7 @@ int iommufd_access_rw(struct iommufd_access *access, unsigned long iova, · 393 unmodified lines | ||
| 1349 | 1407 | struct io_pagetable *iopt; |
| 1350 | 1408 | struct iopt_area *area; |
| 1351 | 1409 | unsigned long last_iova; |
| 1352 | − int rc; | |
| 1410 | + int rc = -EINVAL; | |
| 1353 | 1411 | |
| 1354 | 1412 | if (!length) |
| 1355 | 1413 | return -EINVAL; |
| @@ -1405,7 +1463,8 @@ int iommufd_get_hw_info(struct iommufd_ucmd *ucmd) · 49 unmodified lines | ||
| 1405 | 1463 | void *data; |
| 1406 | 1464 | int rc; |
| 1407 | 1465 | |
| 1408 | − if (cmd->flags || cmd->__reserved) | |
| 1466 | + if (cmd->flags || cmd->__reserved[0] || cmd->__reserved[1] || | |
| 1467 | + cmd->__reserved[2]) | |
| 1409 | 1468 | return -EOPNOTSUPP; |
| 1410 | 1469 | |
| 1411 | 1470 | idev = iommufd_get_device(ucmd, cmd->dev_id); |
| @@ -1462,6 +1521,36 @@ int iommufd_get_hw_info(struct iommufd_ucmd *ucmd) · 50 unmodified lines | ||
| 1462 | 1521 | if (device_iommu_capable(idev->dev, IOMMU_CAP_DIRTY_TRACKING)) |
| 1463 | 1522 | cmd->out_capabilities |= IOMMU_HW_CAP_DIRTY_TRACKING; |
| 1464 | 1523 | |
| 1524 | + cmd->out_max_pasid_log2 = 0; | |
| 1525 | + /* | |
| 1526 | + * Currently, all iommu drivers enable PASID in the probe_device() | |
| 1527 | + * op if iommu and device supports it. So the max_pasids stored in | |
| 1528 | + * dev->iommu indicates both PASID support and enable status. A | |
| 1529 | + * non-zero dev->iommu->max_pasids means PASID is supported and | |
| 1530 | + * enabled. The iommufd only reports PASID capability to userspace | |
| 1531 | + * if it's enabled. | |
| 1532 | + */ | |
| 1533 | + if (idev->dev->iommu->max_pasids) { | |
| 1534 | + cmd->out_max_pasid_log2 = ilog2(idev->dev->iommu->max_pasids); | |
| 1535 | + | |
| 1536 | + if (dev_is_pci(idev->dev)) { | |
| 1537 | + struct pci_dev *pdev = to_pci_dev(idev->dev); | |
| 1538 | + int ctrl; | |
| 1539 | + | |
| 1540 | + ctrl = pci_pasid_status(pdev); | |
| 1541 | + | |
| 1542 | + WARN_ON_ONCE(ctrl < 0 || | |
| 1543 | + !(ctrl & PCI_PASID_CTRL_ENABLE)); | |
| 1544 | + | |
| 1545 | + if (ctrl & PCI_PASID_CTRL_EXEC) | |
| 1546 | + cmd->out_capabilities |= | |
| 1547 | + IOMMU_HW_CAP_PCI_PASID_EXEC; | |
| 1548 | + if (ctrl & PCI_PASID_CTRL_PRIV) | |
| 1549 | + cmd->out_capabilities |= | |
| 1550 | + IOMMU_HW_CAP_PCI_PASID_PRIV; | |
| 1551 | + } | |
| 1552 | + } | |
| 1553 | + | |
| 1465 | 1554 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); |
| 1466 | 1555 | out_free: |
| 1467 | 1556 | kfree(data); |
| @@ -49,5 +49,203 @@ struct device *iommufd_viommu_find_dev(struct iommufd_viommu *viommu, · 48 unmodified lines | ||
| 49 | 49 | } |
| 50 | 50 | EXPORT_SYMBOL_NS_GPL(iommufd_viommu_find_dev, "IOMMUFD"); |
| 51 | 51 | |
| 52 | +/* Return -ENOENT if device is not associated to the vIOMMU */ | |
| 53 | +int iommufd_viommu_get_vdev_id(struct iommufd_viommu *viommu, | |
| 54 | + struct device *dev, unsigned long *vdev_id) | |
| 55 | +{ | |
| 56 | + struct iommufd_vdevice *vdev; | |
| 57 | + unsigned long index; | |
| 58 | + int rc = -ENOENT; | |
| 59 | + | |
| 60 | + if (WARN_ON_ONCE(!vdev_id)) | |
| 61 | + return -EINVAL; | |
| 62 | + | |
| 63 | + xa_lock(&viommu->vdevs); | |
| 64 | + xa_for_each(&viommu->vdevs, index, vdev) { | |
| 65 | + if (vdev->dev == dev) { | |
| 66 | + *vdev_id = vdev->id; | |
| 67 | + rc = 0; | |
| 68 | + break; | |
| 69 | + } | |
| 70 | + } | |
| 71 | + xa_unlock(&viommu->vdevs); | |
| 72 | + return rc; | |
| 73 | +} | |
| 74 | +EXPORT_SYMBOL_NS_GPL(iommufd_viommu_get_vdev_id, "IOMMUFD"); | |
| 75 | + | |
| 76 | +/* | |
| 77 | + * Typically called in driver's threaded IRQ handler. | |
| 78 | + * The @type and @event_data must be defined in include/uapi/linux/iommufd.h | |
| 79 | + */ | |
| 80 | +int iommufd_viommu_report_event(struct iommufd_viommu *viommu, | |
| 81 | + enum iommu_veventq_type type, void *event_data, | |
| 82 | + size_t data_len) | |
| 83 | +{ | |
| 84 | + struct iommufd_veventq *veventq; | |
| 85 | + struct iommufd_vevent *vevent; | |
| 86 | + int rc = 0; | |
| 87 | + | |
| 88 | + if (WARN_ON_ONCE(!data_len || !event_data)) | |
| 89 | + return -EINVAL; | |
| 90 | + | |
| 91 | + down_read(&viommu->veventqs_rwsem); | |
| 92 | + | |
| 93 | + veventq = iommufd_viommu_find_veventq(viommu, type); | |
| 94 | + if (!veventq) { | |
| 95 | + rc = -EOPNOTSUPP; | |
| 96 | + goto out_unlock_veventqs; | |
| 97 | + } | |
| 98 | + | |
| 99 | + spin_lock(&veventq->common.lock); | |
| 100 | + if (veventq->num_events == veventq->depth) { | |
| 101 | + vevent = &veventq->lost_events_header; | |
| 102 | + goto out_set_header; | |
| 103 | + } | |
| 104 | + | |
| 105 | + vevent = kzalloc(struct_size(vevent, event_data, data_len), GFP_ATOMIC); | |
| 106 | + if (!vevent) { | |
| 107 | + rc = -ENOMEM; | |
| 108 | + vevent = &veventq->lost_events_header; | |
| 109 | + goto out_set_header; | |
| 110 | + } | |
| 111 | + memcpy(vevent->event_data, event_data, data_len); | |
| 112 | + vevent->data_len = data_len; | |
| 113 | + veventq->num_events++; | |
| 114 | + | |
| 115 | +out_set_header: | |
| 116 | + iommufd_vevent_handler(veventq, vevent); | |
| 117 | + spin_unlock(&veventq->common.lock); | |
| 118 | +out_unlock_veventqs: | |
| 119 | + up_read(&viommu->veventqs_rwsem); | |
| 120 | + return rc; | |
| 121 | +} | |
| 122 | +EXPORT_SYMBOL_NS_GPL(iommufd_viommu_report_event, "IOMMUFD"); | |
| 123 | + | |
| 124 | +#ifdef CONFIG_IRQ_MSI_IOMMU | |
| 125 | +/* | |
| 126 | + * Get a iommufd_sw_msi_map for the msi physical address requested by the irq | |
| 127 | + * layer. The mapping to IOVA is global to the iommufd file descriptor, every | |
| 128 | + * domain that is attached to a device using the same MSI parameters will use | |
| 129 | + * the same IOVA. | |
| 130 | + */ | |
| 131 | +static struct iommufd_sw_msi_map * | |
| 132 | +iommufd_sw_msi_get_map(struct iommufd_ctx *ictx, phys_addr_t msi_addr, | |
| 133 | + phys_addr_t sw_msi_start) | |
| 134 | +{ | |
| 135 | + struct iommufd_sw_msi_map *cur; | |
| 136 | + unsigned int max_pgoff = 0; | |
| 137 | + | |
| 138 | + lockdep_assert_held(&ictx->sw_msi_lock); | |
| 139 | + | |
| 140 | + list_for_each_entry(cur, &ictx->sw_msi_list, sw_msi_item) { | |
| 141 | + if (cur->sw_msi_start != sw_msi_start) | |
| 142 | + continue; | |
| 143 | + max_pgoff = max(max_pgoff, cur->pgoff + 1); | |
| 144 | + if (cur->msi_addr == msi_addr) | |
| 145 | + return cur; | |
| 146 | + } | |
| 147 | + | |
| 148 | + if (ictx->sw_msi_id >= | |
| 149 | + BITS_PER_BYTE * sizeof_field(struct iommufd_sw_msi_maps, bitmap)) | |
| 150 | + return ERR_PTR(-EOVERFLOW); | |
| 151 | + | |
| 152 | + cur = kzalloc(sizeof(*cur), GFP_KERNEL); | |
| 153 | + if (!cur) | |
| 154 | + return ERR_PTR(-ENOMEM); | |
| 155 | + | |
| 156 | + cur->sw_msi_start = sw_msi_start; | |
| 157 | + cur->msi_addr = msi_addr; | |
| 158 | + cur->pgoff = max_pgoff; | |
| 159 | + cur->id = ictx->sw_msi_id++; | |
| 160 | + list_add_tail(&cur->sw_msi_item, &ictx->sw_msi_list); | |
| 161 | + return cur; | |
| 162 | +} | |
| 163 | + | |
| 164 | +int iommufd_sw_msi_install(struct iommufd_ctx *ictx, | |
| 165 | + struct iommufd_hwpt_paging *hwpt_paging, | |
| 166 | + struct iommufd_sw_msi_map *msi_map) | |
| 167 | +{ | |
| 168 | + unsigned long iova; | |
| 169 | + | |
| 170 | + lockdep_assert_held(&ictx->sw_msi_lock); | |
| 171 | + | |
| 172 | + iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE; | |
| 173 | + if (!test_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap)) { | |
| 174 | + int rc; | |
| 175 | + | |
| 176 | + rc = iommu_map(hwpt_paging->common.domain, iova, | |
| 177 | + msi_map->msi_addr, PAGE_SIZE, | |
| 178 | + IOMMU_WRITE | IOMMU_READ | IOMMU_MMIO, | |
| 179 | + GFP_KERNEL_ACCOUNT); | |
| 180 | + if (rc) | |
| 181 | + return rc; | |
| 182 | + __set_bit(msi_map->id, hwpt_paging->present_sw_msi.bitmap); | |
| 183 | + } | |
| 184 | + return 0; | |
| 185 | +} | |
| 186 | +EXPORT_SYMBOL_NS_GPL(iommufd_sw_msi_install, "IOMMUFD_INTERNAL"); | |
| 187 | + | |
| 188 | +/* | |
| 189 | + * Called by the irq code if the platform translates the MSI address through the | |
| 190 | + * IOMMU. msi_addr is the physical address of the MSI page. iommufd will | |
| 191 | + * allocate a fd global iova for the physical page that is the same on all | |
| 192 | + * domains and devices. | |
| 193 | + */ | |
| 194 | +int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | |
| 195 | + phys_addr_t msi_addr) | |
| 196 | +{ | |
| 197 | + struct device *dev = msi_desc_to_dev(desc); | |
| 198 | + struct iommufd_hwpt_paging *hwpt_paging; | |
| 199 | + struct iommu_attach_handle *raw_handle; | |
| 200 | + struct iommufd_attach_handle *handle; | |
| 201 | + struct iommufd_sw_msi_map *msi_map; | |
| 202 | + struct iommufd_ctx *ictx; | |
| 203 | + unsigned long iova; | |
| 204 | + int rc; | |
| 205 | + | |
| 206 | + /* | |
| 207 | + * It is safe to call iommu_attach_handle_get() here because the iommu | |
| 208 | + * core code invokes this under the group mutex which also prevents any | |
| 209 | + * change of the attach handle for the duration of this function. | |
| 210 | + */ | |
| 211 | + iommu_group_mutex_assert(dev); | |
| 212 | + | |
| 213 | + raw_handle = | |
| 214 | + iommu_attach_handle_get(dev->iommu_group, IOMMU_NO_PASID, 0); | |
| 215 | + if (IS_ERR(raw_handle)) | |
| 216 | + return 0; | |
| 217 | + hwpt_paging = find_hwpt_paging(domain->iommufd_hwpt); | |
| 218 | + | |
| 219 | + handle = to_iommufd_handle(raw_handle); | |
| 220 | + /* No IOMMU_RESV_SW_MSI means no change to the msi_msg */ | |
| 221 | + if (handle->idev->igroup->sw_msi_start == PHYS_ADDR_MAX) | |
| 222 | + return 0; | |
| 223 | + | |
| 224 | + ictx = handle->idev->ictx; | |
| 225 | + guard(mutex)(&ictx->sw_msi_lock); | |
| 226 | + /* | |
| 227 | + * The input msi_addr is the exact byte offset of the MSI doorbell, we | |
| 228 | + * assume the caller has checked that it is contained with a MMIO region | |
| 229 | + * that is secure to map at PAGE_SIZE. | |
| 230 | + */ | |
| 231 | + msi_map = iommufd_sw_msi_get_map(handle->idev->ictx, | |
| 232 | + msi_addr & PAGE_MASK, | |
| 233 | + handle->idev->igroup->sw_msi_start); | |
| 234 | + if (IS_ERR(msi_map)) | |
| 235 | + return PTR_ERR(msi_map); | |
| 236 | + | |
| 237 | + rc = iommufd_sw_msi_install(ictx, hwpt_paging, msi_map); | |
| 238 | + if (rc) | |
| 239 | + return rc; | |
| 240 | + __set_bit(msi_map->id, handle->idev->igroup->required_sw_msi.bitmap); | |
| 241 | + | |
| 242 | + iova = msi_map->sw_msi_start + msi_map->pgoff * PAGE_SIZE; | |
| 243 | + msi_desc_set_iommu_msi_iova(desc, iova, PAGE_SHIFT); | |
| 244 | + return 0; | |
| 245 | +} | |
| 246 | +EXPORT_SYMBOL_NS_GPL(iommufd_sw_msi, "IOMMUFD"); | |
| 247 | +#endif | |
| 248 | + | |
| 52 | 249 | MODULE_DESCRIPTION("iommufd code shared with builtin modules"); |
| 250 | +MODULE_IMPORT_NS("IOMMUFD_INTERNAL"); | |
| 53 | 251 | MODULE_LICENSE("GPL"); |
| @@ -0,0 +1,598 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0-only | |
| 2 | +/* Copyright (C) 2024 Intel Corporation | |
| 3 | + */ | |
| 4 | +#define pr_fmt(fmt) "iommufd: " fmt | |
| 5 | + | |
| 6 | +#include <linux/anon_inodes.h> | |
| 7 | +#include <linux/file.h> | |
| 8 | +#include <linux/fs.h> | |
| 9 | +#include <linux/iommufd.h> | |
| 10 | +#include <linux/module.h> | |
| 11 | +#include <linux/mutex.h> | |
| 12 | +#include <linux/pci.h> | |
| 13 | +#include <linux/pci-ats.h> | |
| 14 | +#include <linux/poll.h> | |
| 15 | +#include <uapi/linux/iommufd.h> | |
| 16 | + | |
| 17 | +#include "../iommu-priv.h" | |
| 18 | +#include "iommufd_private.h" | |
| 19 | + | |
| 20 | +/* IOMMUFD_OBJ_FAULT Functions */ | |
| 21 | + | |
| 22 | +int iommufd_fault_iopf_enable(struct iommufd_device *idev) | |
| 23 | +{ | |
| 24 | + struct device *dev = idev->dev; | |
| 25 | + int ret; | |
| 26 | + | |
| 27 | + /* | |
| 28 | + * Once we turn on PCI/PRI support for VF, the response failure code | |
| 29 | + * should not be forwarded to the hardware due to PRI being a shared | |
| 30 | + * resource between PF and VFs. There is no coordination for this | |
| 31 | + * shared capability. This waits for a vPRI reset to recover. | |
| 32 | + */ | |
| 33 | + if (dev_is_pci(dev)) { | |
| 34 | + struct pci_dev *pdev = to_pci_dev(dev); | |
| 35 | + | |
| 36 | + if (pdev->is_virtfn && pci_pri_supported(pdev)) | |
| 37 | + return -EINVAL; | |
| 38 | + } | |
| 39 | + | |
| 40 | + mutex_lock(&idev->iopf_lock); | |
| 41 | + /* Device iopf has already been on. */ | |
| 42 | + if (++idev->iopf_enabled > 1) { | |
| 43 | + mutex_unlock(&idev->iopf_lock); | |
| 44 | + return 0; | |
| 45 | + } | |
| 46 | + | |
| 47 | + ret = iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_IOPF); | |
| 48 | + if (ret) | |
| 49 | + --idev->iopf_enabled; | |
| 50 | + mutex_unlock(&idev->iopf_lock); | |
| 51 | + | |
| 52 | + return ret; | |
| 53 | +} | |
| 54 | + | |
| 55 | +void iommufd_fault_iopf_disable(struct iommufd_device *idev) | |
| 56 | +{ | |
| 57 | + mutex_lock(&idev->iopf_lock); | |
| 58 | + if (!WARN_ON(idev->iopf_enabled == 0)) { | |
| 59 | + if (--idev->iopf_enabled == 0) | |
| 60 | + iommu_dev_disable_feature(idev->dev, IOMMU_DEV_FEAT_IOPF); | |
| 61 | + } | |
| 62 | + mutex_unlock(&idev->iopf_lock); | |
| 63 | +} | |
| 64 | + | |
| 65 | +void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt, | |
| 66 | + struct iommufd_attach_handle *handle) | |
| 67 | +{ | |
| 68 | + struct iommufd_fault *fault = hwpt->fault; | |
| 69 | + struct iopf_group *group, *next; | |
| 70 | + struct list_head free_list; | |
| 71 | + unsigned long index; | |
| 72 | + | |
| 73 | + if (!fault) | |
| 74 | + return; | |
| 75 | + INIT_LIST_HEAD(&free_list); | |
| 76 | + | |
| 77 | + mutex_lock(&fault->mutex); | |
| 78 | + spin_lock(&fault->common.lock); | |
| 79 | + list_for_each_entry_safe(group, next, &fault->common.deliver, node) { | |
| 80 | + if (group->attach_handle != &handle->handle) | |
| 81 | + continue; | |
| 82 | + list_move(&group->node, &free_list); | |
| 83 | + } | |
| 84 | + spin_unlock(&fault->common.lock); | |
| 85 | + | |
| 86 | + list_for_each_entry_safe(group, next, &free_list, node) { | |
| 87 | + list_del(&group->node); | |
| 88 | + iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | |
| 89 | + iopf_free_group(group); | |
| 90 | + } | |
| 91 | + | |
| 92 | + xa_for_each(&fault->response, index, group) { | |
| 93 | + if (group->attach_handle != &handle->handle) | |
| 94 | + continue; | |
| 95 | + xa_erase(&fault->response, index); | |
| 96 | + iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | |
| 97 | + iopf_free_group(group); | |
| 98 | + } | |
| 99 | + mutex_unlock(&fault->mutex); | |
| 100 | +} | |
| 101 | + | |
| 102 | +void iommufd_fault_destroy(struct iommufd_object *obj) | |
| 103 | +{ | |
| 104 | + struct iommufd_eventq *eventq = | |
| 105 | + container_of(obj, struct iommufd_eventq, obj); | |
| 106 | + struct iommufd_fault *fault = eventq_to_fault(eventq); | |
| 107 | + struct iopf_group *group, *next; | |
| 108 | + unsigned long index; | |
| 109 | + | |
| 110 | + /* | |
| 111 | + * The iommufd object's reference count is zero at this point. | |
| 112 | + * We can be confident that no other threads are currently | |
| 113 | + * accessing this pointer. Therefore, acquiring the mutex here | |
| 114 | + * is unnecessary. | |
| 115 | + */ | |
| 116 | + list_for_each_entry_safe(group, next, &fault->common.deliver, node) { | |
| 117 | + list_del(&group->node); | |
| 118 | + iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | |
| 119 | + iopf_free_group(group); | |
| 120 | + } | |
| 121 | + xa_for_each(&fault->response, index, group) { | |
| 122 | + xa_erase(&fault->response, index); | |
| 123 | + iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | |
| 124 | + iopf_free_group(group); | |
| 125 | + } | |
| 126 | + xa_destroy(&fault->response); | |
| 127 | + mutex_destroy(&fault->mutex); | |
| 128 | +} | |
| 129 | + | |
| 130 | +static void iommufd_compose_fault_message(struct iommu_fault *fault, | |
| 131 | + struct iommu_hwpt_pgfault *hwpt_fault, | |
| 132 | + struct iommufd_device *idev, | |
| 133 | + u32 cookie) | |
| 134 | +{ | |
| 135 | + hwpt_fault->flags = fault->prm.flags; | |
| 136 | + hwpt_fault->dev_id = idev->obj.id; | |
| 137 | + hwpt_fault->pasid = fault->prm.pasid; | |
| 138 | + hwpt_fault->grpid = fault->prm.grpid; | |
| 139 | + hwpt_fault->perm = fault->prm.perm; | |
| 140 | + hwpt_fault->addr = fault->prm.addr; | |
| 141 | + hwpt_fault->length = 0; | |
| 142 | + hwpt_fault->cookie = cookie; | |
| 143 | +} | |
| 144 | + | |
| 145 | +/* Fetch the first node out of the fault->deliver list */ | |
| 146 | +static struct iopf_group * | |
| 147 | +iommufd_fault_deliver_fetch(struct iommufd_fault *fault) | |
| 148 | +{ | |
| 149 | + struct list_head *list = &fault->common.deliver; | |
| 150 | + struct iopf_group *group = NULL; | |
| 151 | + | |
| 152 | + spin_lock(&fault->common.lock); | |
| 153 | + if (!list_empty(list)) { | |
| 154 | + group = list_first_entry(list, struct iopf_group, node); | |
| 155 | + list_del(&group->node); | |
| 156 | + } | |
| 157 | + spin_unlock(&fault->common.lock); | |
| 158 | + return group; | |
| 159 | +} | |
| 160 | + | |
| 161 | +/* Restore a node back to the head of the fault->deliver list */ | |
| 162 | +static void iommufd_fault_deliver_restore(struct iommufd_fault *fault, | |
| 163 | + struct iopf_group *group) | |
| 164 | +{ | |
| 165 | + spin_lock(&fault->common.lock); | |
| 166 | + list_add(&group->node, &fault->common.deliver); | |
| 167 | + spin_unlock(&fault->common.lock); | |
| 168 | +} | |
| 169 | + | |
| 170 | +static ssize_t iommufd_fault_fops_read(struct file *filep, char __user *buf, | |
| 171 | + size_t count, loff_t *ppos) | |
| 172 | +{ | |
| 173 | + size_t fault_size = sizeof(struct iommu_hwpt_pgfault); | |
| 174 | + struct iommufd_eventq *eventq = filep->private_data; | |
| 175 | + struct iommufd_fault *fault = eventq_to_fault(eventq); | |
| 176 | + struct iommu_hwpt_pgfault data = {}; | |
| 177 | + struct iommufd_device *idev; | |
| 178 | + struct iopf_group *group; | |
| 179 | + struct iopf_fault *iopf; | |
| 180 | + size_t done = 0; | |
| 181 | + int rc = 0; | |
| 182 | + | |
| 183 | + if (*ppos || count % fault_size) | |
| 184 | + return -ESPIPE; | |
| 185 | + | |
| 186 | + mutex_lock(&fault->mutex); | |
| 187 | + while ((group = iommufd_fault_deliver_fetch(fault))) { | |
| 188 | + if (done >= count || | |
| 189 | + group->fault_count * fault_size > count - done) { | |
| 190 | + iommufd_fault_deliver_restore(fault, group); | |
| 191 | + break; | |
| 192 | + } | |
| 193 | + | |
| 194 | + rc = xa_alloc(&fault->response, &group->cookie, group, | |
| 195 | + xa_limit_32b, GFP_KERNEL); | |
| 196 | + if (rc) { | |
| 197 | + iommufd_fault_deliver_restore(fault, group); | |
| 198 | + break; | |
| 199 | + } | |
| 200 | + | |
| 201 | + idev = to_iommufd_handle(group->attach_handle)->idev; | |
| 202 | + list_for_each_entry(iopf, &group->faults, list) { | |
| 203 | + iommufd_compose_fault_message(&iopf->fault, | |
| 204 | + &data, idev, | |
| 205 | + group->cookie); | |
| 206 | + if (copy_to_user(buf + done, &data, fault_size)) { | |
| 207 | + xa_erase(&fault->response, group->cookie); | |
| 208 | + iommufd_fault_deliver_restore(fault, group); | |
| 209 | + rc = -EFAULT; | |
| 210 | + break; | |
| 211 | + } | |
| 212 | + done += fault_size; | |
| 213 | + } | |
| 214 | + } | |
| 215 | + mutex_unlock(&fault->mutex); | |
| 216 | + | |
| 217 | + return done == 0 ? rc : done; | |
| 218 | +} | |
| 219 | + | |
| 220 | +static ssize_t iommufd_fault_fops_write(struct file *filep, const char __user *buf, | |
| 221 | + size_t count, loff_t *ppos) | |
| 222 | +{ | |
| 223 | + size_t response_size = sizeof(struct iommu_hwpt_page_response); | |
| 224 | + struct iommufd_eventq *eventq = filep->private_data; | |
| 225 | + struct iommufd_fault *fault = eventq_to_fault(eventq); | |
| 226 | + struct iommu_hwpt_page_response response; | |
| 227 | + struct iopf_group *group; | |
| 228 | + size_t done = 0; | |
| 229 | + int rc = 0; | |
| 230 | + | |
| 231 | + if (*ppos || count % response_size) | |
| 232 | + return -ESPIPE; | |
| 233 | + | |
| 234 | + mutex_lock(&fault->mutex); | |
| 235 | + while (count > done) { | |
| 236 | + rc = copy_from_user(&response, buf + done, response_size); | |
| 237 | + if (rc) | |
| 238 | + break; | |
| 239 | + | |
| 240 | + static_assert((int)IOMMUFD_PAGE_RESP_SUCCESS == | |
| 241 | + (int)IOMMU_PAGE_RESP_SUCCESS); | |
| 242 | + static_assert((int)IOMMUFD_PAGE_RESP_INVALID == | |
| 243 | + (int)IOMMU_PAGE_RESP_INVALID); | |
| 244 | + if (response.code != IOMMUFD_PAGE_RESP_SUCCESS && | |
| 245 | + response.code != IOMMUFD_PAGE_RESP_INVALID) { | |
| 246 | + rc = -EINVAL; | |
| 247 | + break; | |
| 248 | + } | |
| 249 | + | |
| 250 | + group = xa_erase(&fault->response, response.cookie); | |
| 251 | + if (!group) { | |
| 252 | + rc = -EINVAL; | |
| 253 | + break; | |
| 254 | + } | |
| 255 | + | |
| 256 | + iopf_group_response(group, response.code); | |
| 257 | + iopf_free_group(group); | |
| 258 | + done += response_size; | |
| 259 | + } | |
| 260 | + mutex_unlock(&fault->mutex); | |
| 261 | + | |
| 262 | + return done == 0 ? rc : done; | |
| 263 | +} | |
| 264 | + | |
| 265 | +/* IOMMUFD_OBJ_VEVENTQ Functions */ | |
| 266 | + | |
| 267 | +void iommufd_veventq_abort(struct iommufd_object *obj) | |
| 268 | +{ | |
| 269 | + struct iommufd_eventq *eventq = | |
| 270 | + container_of(obj, struct iommufd_eventq, obj); | |
| 271 | + struct iommufd_veventq *veventq = eventq_to_veventq(eventq); | |
| 272 | + struct iommufd_viommu *viommu = veventq->viommu; | |
| 273 | + struct iommufd_vevent *cur, *next; | |
| 274 | + | |
| 275 | + lockdep_assert_held_write(&viommu->veventqs_rwsem); | |
| 276 | + | |
| 277 | + list_for_each_entry_safe(cur, next, &eventq->deliver, node) { | |
| 278 | + list_del(&cur->node); | |
| 279 | + if (cur != &veventq->lost_events_header) | |
| 280 | + kfree(cur); | |
| 281 | + } | |
| 282 | + | |
| 283 | + refcount_dec(&viommu->obj.users); | |
| 284 | + list_del(&veventq->node); | |
| 285 | +} | |
| 286 | + | |
| 287 | +void iommufd_veventq_destroy(struct iommufd_object *obj) | |
| 288 | +{ | |
| 289 | + struct iommufd_veventq *veventq = eventq_to_veventq( | |
| 290 | + container_of(obj, struct iommufd_eventq, obj)); | |
| 291 | + | |
| 292 | + down_write(&veventq->viommu->veventqs_rwsem); | |
| 293 | + iommufd_veventq_abort(obj); | |
| 294 | + up_write(&veventq->viommu->veventqs_rwsem); | |
| 295 | +} | |
| 296 | + | |
| 297 | +static struct iommufd_vevent * | |
| 298 | +iommufd_veventq_deliver_fetch(struct iommufd_veventq *veventq) | |
| 299 | +{ | |
| 300 | + struct iommufd_eventq *eventq = &veventq->common; | |
| 301 | + struct list_head *list = &eventq->deliver; | |
| 302 | + struct iommufd_vevent *vevent = NULL; | |
| 303 | + | |
| 304 | + spin_lock(&eventq->lock); | |
| 305 | + if (!list_empty(list)) { | |
| 306 | + struct iommufd_vevent *next; | |
| 307 | + | |
| 308 | + next = list_first_entry(list, struct iommufd_vevent, node); | |
| 309 | + /* Make a copy of the lost_events_header for copy_to_user */ | |
| 310 | + if (next == &veventq->lost_events_header) { | |
| 311 | + vevent = kzalloc(sizeof(*vevent), GFP_ATOMIC); | |
| 312 | + if (!vevent) | |
| 313 | + goto out_unlock; | |
| 314 | + } | |
| 315 | + list_del(&next->node); | |
| 316 | + if (vevent) | |
| 317 | + memcpy(vevent, next, sizeof(*vevent)); | |
| 318 | + else | |
| 319 | + vevent = next; | |
| 320 | + } | |
| 321 | +out_unlock: | |
| 322 | + spin_unlock(&eventq->lock); | |
| 323 | + return vevent; | |
| 324 | +} | |
| 325 | + | |
| 326 | +static void iommufd_veventq_deliver_restore(struct iommufd_veventq *veventq, | |
| 327 | + struct iommufd_vevent *vevent) | |
| 328 | +{ | |
| 329 | + struct iommufd_eventq *eventq = &veventq->common; | |
| 330 | + struct list_head *list = &eventq->deliver; | |
| 331 | + | |
| 332 | + spin_lock(&eventq->lock); | |
| 333 | + if (vevent_for_lost_events_header(vevent)) { | |
| 334 | + /* Remove the copy of the lost_events_header */ | |
| 335 | + kfree(vevent); | |
| 336 | + vevent = NULL; | |
| 337 | + /* An empty list needs the lost_events_header back */ | |
| 338 | + if (list_empty(list)) | |
| 339 | + vevent = &veventq->lost_events_header; | |
| 340 | + } | |
| 341 | + if (vevent) | |
| 342 | + list_add(&vevent->node, list); | |
| 343 | + spin_unlock(&eventq->lock); | |
| 344 | +} | |
| 345 | + | |
| 346 | +static ssize_t iommufd_veventq_fops_read(struct file *filep, char __user *buf, | |
| 347 | + size_t count, loff_t *ppos) | |
| 348 | +{ | |
| 349 | + struct iommufd_eventq *eventq = filep->private_data; | |
| 350 | + struct iommufd_veventq *veventq = eventq_to_veventq(eventq); | |
| 351 | + struct iommufd_vevent_header *hdr; | |
| 352 | + struct iommufd_vevent *cur; | |
| 353 | + size_t done = 0; | |
| 354 | + int rc = 0; | |
| 355 | + | |
| 356 | + if (*ppos) | |
| 357 | + return -ESPIPE; | |
| 358 | + | |
| 359 | + while ((cur = iommufd_veventq_deliver_fetch(veventq))) { | |
| 360 | + /* Validate the remaining bytes against the header size */ | |
| 361 | + if (done >= count || sizeof(*hdr) > count - done) { | |
| 362 | + iommufd_veventq_deliver_restore(veventq, cur); | |
| 363 | + break; | |
| 364 | + } | |
| 365 | + hdr = &cur->header; | |
| 366 | + | |
| 367 | + /* If being a normal vEVENT, validate against the full size */ | |
| 368 | + if (!vevent_for_lost_events_header(cur) && | |
| 369 | + sizeof(hdr) + cur->data_len > count - done) { | |
| 370 | + iommufd_veventq_deliver_restore(veventq, cur); | |
| 371 | + break; | |
| 372 | + } | |
| 373 | + | |
| 374 | + if (copy_to_user(buf + done, hdr, sizeof(*hdr))) { | |
| 375 | + iommufd_veventq_deliver_restore(veventq, cur); | |
| 376 | + rc = -EFAULT; | |
| 377 | + break; | |
| 378 | + } | |
| 379 | + done += sizeof(*hdr); | |
| 380 | + | |
| 381 | + if (cur->data_len && | |
| 382 | + copy_to_user(buf + done, cur->event_data, cur->data_len)) { | |
| 383 | + iommufd_veventq_deliver_restore(veventq, cur); | |
| 384 | + rc = -EFAULT; | |
| 385 | + break; | |
| 386 | + } | |
| 387 | + spin_lock(&eventq->lock); | |
| 388 | + if (!vevent_for_lost_events_header(cur)) | |
| 389 | + veventq->num_events--; | |
| 390 | + spin_unlock(&eventq->lock); | |
| 391 | + done += cur->data_len; | |
| 392 | + kfree(cur); | |
| 393 | + } | |
| 394 | + | |
| 395 | + return done == 0 ? rc : done; | |
| 396 | +} | |
| 397 | + | |
| 398 | +/* Common Event Queue Functions */ | |
| 399 | + | |
| 400 | +static __poll_t iommufd_eventq_fops_poll(struct file *filep, | |
| 401 | + struct poll_table_struct *wait) | |
| 402 | +{ | |
| 403 | + struct iommufd_eventq *eventq = filep->private_data; | |
| 404 | + __poll_t pollflags = 0; | |
| 405 | + | |
| 406 | + if (eventq->obj.type == IOMMUFD_OBJ_FAULT) | |
| 407 | + pollflags |= EPOLLOUT; | |
| 408 | + | |
| 409 | + poll_wait(filep, &eventq->wait_queue, wait); | |
| 410 | + spin_lock(&eventq->lock); | |
| 411 | + if (!list_empty(&eventq->deliver)) | |
| 412 | + pollflags |= EPOLLIN | EPOLLRDNORM; | |
| 413 | + spin_unlock(&eventq->lock); | |
| 414 | + | |
| 415 | + return pollflags; | |
| 416 | +} | |
| 417 | + | |
| 418 | +static int iommufd_eventq_fops_release(struct inode *inode, struct file *filep) | |
| 419 | +{ | |
| 420 | + struct iommufd_eventq *eventq = filep->private_data; | |
| 421 | + | |
| 422 | + refcount_dec(&eventq->obj.users); | |
| 423 | + iommufd_ctx_put(eventq->ictx); | |
| 424 | + return 0; | |
| 425 | +} | |
| 426 | + | |
| 427 | +#define INIT_EVENTQ_FOPS(read_op, write_op) \ | |
| 428 | + ((const struct file_operations){ \ | |
| 429 | + .owner = THIS_MODULE, \ | |
| 430 | + .open = nonseekable_open, \ | |
| 431 | + .read = read_op, \ | |
| 432 | + .write = write_op, \ | |
| 433 | + .poll = iommufd_eventq_fops_poll, \ | |
| 434 | + .release = iommufd_eventq_fops_release, \ | |
| 435 | + }) | |
| 436 | + | |
| 437 | +static int iommufd_eventq_init(struct iommufd_eventq *eventq, char *name, | |
| 438 | + struct iommufd_ctx *ictx, | |
| 439 | + const struct file_operations *fops) | |
| 440 | +{ | |
| 441 | + struct file *filep; | |
| 442 | + int fdno; | |
| 443 | + | |
| 444 | + spin_lock_init(&eventq->lock); | |
| 445 | + INIT_LIST_HEAD(&eventq->deliver); | |
| 446 | + init_waitqueue_head(&eventq->wait_queue); | |
| 447 | + | |
| 448 | + filep = anon_inode_getfile(name, fops, eventq, O_RDWR); | |
| 449 | + if (IS_ERR(filep)) | |
| 450 | + return PTR_ERR(filep); | |
| 451 | + | |
| 452 | + eventq->ictx = ictx; | |
| 453 | + iommufd_ctx_get(eventq->ictx); | |
| 454 | + eventq->filep = filep; | |
| 455 | + refcount_inc(&eventq->obj.users); | |
| 456 | + | |
| 457 | + fdno = get_unused_fd_flags(O_CLOEXEC); | |
| 458 | + if (fdno < 0) | |
| 459 | + fput(filep); | |
| 460 | + return fdno; | |
| 461 | +} | |
| 462 | + | |
| 463 | +static const struct file_operations iommufd_fault_fops = | |
| 464 | + INIT_EVENTQ_FOPS(iommufd_fault_fops_read, iommufd_fault_fops_write); | |
| 465 | + | |
| 466 | +int iommufd_fault_alloc(struct iommufd_ucmd *ucmd) | |
| 467 | +{ | |
| 468 | + struct iommu_fault_alloc *cmd = ucmd->cmd; | |
| 469 | + struct iommufd_fault *fault; | |
| 470 | + int fdno; | |
| 471 | + int rc; | |
| 472 | + | |
| 473 | + if (cmd->flags) | |
| 474 | + return -EOPNOTSUPP; | |
| 475 | + | |
| 476 | + fault = __iommufd_object_alloc(ucmd->ictx, fault, IOMMUFD_OBJ_FAULT, | |
| 477 | + common.obj); | |
| 478 | + if (IS_ERR(fault)) | |
| 479 | + return PTR_ERR(fault); | |
| 480 | + | |
| 481 | + xa_init_flags(&fault->response, XA_FLAGS_ALLOC1); | |
| 482 | + mutex_init(&fault->mutex); | |
| 483 | + | |
| 484 | + fdno = iommufd_eventq_init(&fault->common, "[iommufd-pgfault]", | |
| 485 | + ucmd->ictx, &iommufd_fault_fops); | |
| 486 | + if (fdno < 0) { | |
| 487 | + rc = fdno; | |
| 488 | + goto out_abort; | |
| 489 | + } | |
| 490 | + | |
| 491 | + cmd->out_fault_id = fault->common.obj.id; | |
| 492 | + cmd->out_fault_fd = fdno; | |
| 493 | + | |
| 494 | + rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); | |
| 495 | + if (rc) | |
| 496 | + goto out_put_fdno; | |
| 497 | + iommufd_object_finalize(ucmd->ictx, &fault->common.obj); | |
| 498 | + | |
| 499 | + fd_install(fdno, fault->common.filep); | |
| 500 | + | |
| 501 | + return 0; | |
| 502 | +out_put_fdno: | |
| 503 | + put_unused_fd(fdno); | |
| 504 | + fput(fault->common.filep); | |
| 505 | +out_abort: | |
| 506 | + iommufd_object_abort_and_destroy(ucmd->ictx, &fault->common.obj); | |
| 507 | + | |
| 508 | + return rc; | |
| 509 | +} | |
| 510 | + | |
| 511 | +int iommufd_fault_iopf_handler(struct iopf_group *group) | |
| 512 | +{ | |
| 513 | + struct iommufd_hw_pagetable *hwpt; | |
| 514 | + struct iommufd_fault *fault; | |
| 515 | + | |
| 516 | + hwpt = group->attach_handle->domain->iommufd_hwpt; | |
| 517 | + fault = hwpt->fault; | |
| 518 | + | |
| 519 | + spin_lock(&fault->common.lock); | |
| 520 | + list_add_tail(&group->node, &fault->common.deliver); | |
| 521 | + spin_unlock(&fault->common.lock); | |
| 522 | + | |
| 523 | + wake_up_interruptible(&fault->common.wait_queue); | |
| 524 | + | |
| 525 | + return 0; | |
| 526 | +} | |
| 527 | + | |
| 528 | +static const struct file_operations iommufd_veventq_fops = | |
| 529 | + INIT_EVENTQ_FOPS(iommufd_veventq_fops_read, NULL); | |
| 530 | + | |
| 531 | +int iommufd_veventq_alloc(struct iommufd_ucmd *ucmd) | |
| 532 | +{ | |
| 533 | + struct iommu_veventq_alloc *cmd = ucmd->cmd; | |
| 534 | + struct iommufd_veventq *veventq; | |
| 535 | + struct iommufd_viommu *viommu; | |
| 536 | + int fdno; | |
| 537 | + int rc; | |
| 538 | + | |
| 539 | + if (cmd->flags || cmd->__reserved || | |
| 540 | + cmd->type == IOMMU_VEVENTQ_TYPE_DEFAULT) | |
| 541 | + return -EOPNOTSUPP; | |
| 542 | + if (!cmd->veventq_depth) | |
| 543 | + return -EINVAL; | |
| 544 | + | |
| 545 | + viommu = iommufd_get_viommu(ucmd, cmd->viommu_id); | |
| 546 | + if (IS_ERR(viommu)) | |
| 547 | + return PTR_ERR(viommu); | |
| 548 | + | |
| 549 | + down_write(&viommu->veventqs_rwsem); | |
| 550 | + | |
| 551 | + if (iommufd_viommu_find_veventq(viommu, cmd->type)) { | |
| 552 | + rc = -EEXIST; | |
| 553 | + goto out_unlock_veventqs; | |
| 554 | + } | |
| 555 | + | |
| 556 | + veventq = __iommufd_object_alloc(ucmd->ictx, veventq, | |
| 557 | + IOMMUFD_OBJ_VEVENTQ, common.obj); | |
| 558 | + if (IS_ERR(veventq)) { | |
| 559 | + rc = PTR_ERR(veventq); | |
| 560 | + goto out_unlock_veventqs; | |
| 561 | + } | |
| 562 | + | |
| 563 | + veventq->type = cmd->type; | |
| 564 | + veventq->viommu = viommu; | |
| 565 | + refcount_inc(&viommu->obj.users); | |
| 566 | + veventq->depth = cmd->veventq_depth; | |
| 567 | + list_add_tail(&veventq->node, &viommu->veventqs); | |
| 568 | + veventq->lost_events_header.header.flags = | |
| 569 | + IOMMU_VEVENTQ_FLAG_LOST_EVENTS; | |
| 570 | + | |
| 571 | + fdno = iommufd_eventq_init(&veventq->common, "[iommufd-viommu-event]", | |
| 572 | + ucmd->ictx, &iommufd_veventq_fops); | |
| 573 | + if (fdno < 0) { | |
| 574 | + rc = fdno; | |
| 575 | + goto out_abort; | |
| 576 | + } | |
| 577 | + | |
| 578 | + cmd->out_veventq_id = veventq->common.obj.id; | |
| 579 | + cmd->out_veventq_fd = fdno; | |
| 580 | + | |
| 581 | + rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); | |
| 582 | + if (rc) | |
| 583 | + goto out_put_fdno; | |
| 584 | + | |
| 585 | + iommufd_object_finalize(ucmd->ictx, &veventq->common.obj); | |
| 586 | + fd_install(fdno, veventq->common.filep); | |
| 587 | + goto out_unlock_veventqs; | |
| 588 | + | |
| 589 | +out_put_fdno: | |
| 590 | + put_unused_fd(fdno); | |
| 591 | + fput(veventq->common.filep); | |
| 592 | +out_abort: | |
| 593 | + iommufd_object_abort_and_destroy(ucmd->ictx, &veventq->common.obj); | |
| 594 | +out_unlock_veventqs: | |
| 595 | + up_write(&viommu->veventqs_rwsem); | |
| 596 | + iommufd_put_object(ucmd->ictx, &viommu->obj); | |
| 597 | + return rc; | |
| 598 | +} | |
| @@ -1,342 +0,0 @@ | ||
| 1 | −// SPDX-License-Identifier: GPL-2.0-only | |
| 2 | −/* Copyright (C) 2024 Intel Corporation | |
| 3 | − */ | |
| 4 | −#define pr_fmt(fmt) "iommufd: " fmt | |
| 5 | − | |
| 6 | −#include <linux/anon_inodes.h> | |
| 7 | −#include <linux/file.h> | |
| 8 | −#include <linux/fs.h> | |
| 9 | −#include <linux/iommufd.h> | |
| 10 | −#include <linux/module.h> | |
| 11 | −#include <linux/mutex.h> | |
| 12 | −#include <linux/pci.h> | |
| 13 | −#include <linux/pci-ats.h> | |
| 14 | −#include <linux/poll.h> | |
| 15 | −#include <uapi/linux/iommufd.h> | |
| 16 | − | |
| 17 | −#include "../iommu-priv.h" | |
| 18 | −#include "iommufd_private.h" | |
| 19 | − | |
| 20 | −int iommufd_fault_iopf_enable(struct iommufd_device *idev) | |
| 21 | −{ | |
| 22 | − struct device *dev = idev->dev; | |
| 23 | − int ret; | |
| 24 | − | |
| 25 | − /* | |
| 26 | − * Once we turn on PCI/PRI support for VF, the response failure code | |
| 27 | − * should not be forwarded to the hardware due to PRI being a shared | |
| 28 | − * resource between PF and VFs. There is no coordination for this | |
| 29 | − * shared capability. This waits for a vPRI reset to recover. | |
| 30 | − */ | |
| 31 | − if (dev_is_pci(dev)) { | |
| 32 | − struct pci_dev *pdev = to_pci_dev(dev); | |
| 33 | − | |
| 34 | − if (pdev->is_virtfn && pci_pri_supported(pdev)) | |
| 35 | − return -EINVAL; | |
| 36 | − } | |
| 37 | − | |
| 38 | − mutex_lock(&idev->iopf_lock); | |
| 39 | − /* Device iopf has already been on. */ | |
| 40 | − if (++idev->iopf_enabled > 1) { | |
| 41 | − mutex_unlock(&idev->iopf_lock); | |
| 42 | − return 0; | |
| 43 | − } | |
| 44 | − | |
| 45 | − ret = iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_IOPF); | |
| 46 | − if (ret) | |
| 47 | − --idev->iopf_enabled; | |
| 48 | − mutex_unlock(&idev->iopf_lock); | |
| 49 | − | |
| 50 | − return ret; | |
| 51 | −} | |
| 52 | − | |
| 53 | −void iommufd_fault_iopf_disable(struct iommufd_device *idev) | |
| 54 | −{ | |
| 55 | − mutex_lock(&idev->iopf_lock); | |
| 56 | − if (!WARN_ON(idev->iopf_enabled == 0)) { | |
| 57 | − if (--idev->iopf_enabled == 0) | |
| 58 | − iommu_dev_disable_feature(idev->dev, IOMMU_DEV_FEAT_IOPF); | |
| 59 | − } | |
| 60 | − mutex_unlock(&idev->iopf_lock); | |
| 61 | −} | |
| 62 | − | |
| 63 | −void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt, | |
| 64 | − struct iommufd_attach_handle *handle) | |
| 65 | −{ | |
| 66 | − struct iommufd_fault *fault = hwpt->fault; | |
| 67 | − struct iopf_group *group, *next; | |
| 68 | − struct list_head free_list; | |
| 69 | − unsigned long index; | |
| 70 | − | |
| 71 | − if (!fault) | |
| 72 | − return; | |
| 73 | − INIT_LIST_HEAD(&free_list); | |
| 74 | − | |
| 75 | − mutex_lock(&fault->mutex); | |
| 76 | − spin_lock(&fault->lock); | |
| 77 | − list_for_each_entry_safe(group, next, &fault->deliver, node) { | |
| 78 | − if (group->attach_handle != &handle->handle) | |
| 79 | − continue; | |
| 80 | − list_move(&group->node, &free_list); | |
| 81 | − } | |
| 82 | − spin_unlock(&fault->lock); | |
| 83 | − | |
| 84 | − list_for_each_entry_safe(group, next, &free_list, node) { | |
| 85 | − list_del(&group->node); | |
| 86 | − iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | |
| 87 | − iopf_free_group(group); | |
| 88 | − } | |
| 89 | − | |
| 90 | − xa_for_each(&fault->response, index, group) { | |
| 91 | − if (group->attach_handle != &handle->handle) | |
| 92 | − continue; | |
| 93 | − xa_erase(&fault->response, index); | |
| 94 | − iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | |
| 95 | − iopf_free_group(group); | |
| 96 | − } | |
| 97 | − mutex_unlock(&fault->mutex); | |
| 98 | −} | |
| 99 | − | |
| 100 | −void iommufd_fault_destroy(struct iommufd_object *obj) | |
| 101 | −{ | |
| 102 | − struct iommufd_fault *fault = container_of(obj, struct iommufd_fault, obj); | |
| 103 | − struct iopf_group *group, *next; | |
| 104 | − unsigned long index; | |
| 105 | − | |
| 106 | − /* | |
| 107 | − * The iommufd object's reference count is zero at this point. | |
| 108 | − * We can be confident that no other threads are currently | |
| 109 | − * accessing this pointer. Therefore, acquiring the mutex here | |
| 110 | − * is unnecessary. | |
| 111 | − */ | |
| 112 | − list_for_each_entry_safe(group, next, &fault->deliver, node) { | |
| 113 | − list_del(&group->node); | |
| 114 | − iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | |
| 115 | − iopf_free_group(group); | |
| 116 | − } | |
| 117 | − xa_for_each(&fault->response, index, group) { | |
| 118 | − xa_erase(&fault->response, index); | |
| 119 | − iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); | |
| 120 | − iopf_free_group(group); | |
| 121 | − } | |
| 122 | − xa_destroy(&fault->response); | |
| 123 | − mutex_destroy(&fault->mutex); | |
| 124 | −} | |
| 125 | − | |
| 126 | −static void iommufd_compose_fault_message(struct iommu_fault *fault, | |
| 127 | − struct iommu_hwpt_pgfault *hwpt_fault, | |
| 128 | − struct iommufd_device *idev, | |
| 129 | − u32 cookie) | |
| 130 | −{ | |
| 131 | − hwpt_fault->flags = fault->prm.flags; | |
| 132 | − hwpt_fault->dev_id = idev->obj.id; | |
| 133 | − hwpt_fault->pasid = fault->prm.pasid; | |
| 134 | − hwpt_fault->grpid = fault->prm.grpid; | |
| 135 | − hwpt_fault->perm = fault->prm.perm; | |
| 136 | − hwpt_fault->addr = fault->prm.addr; | |
| 137 | − hwpt_fault->length = 0; | |
| 138 | − hwpt_fault->cookie = cookie; | |
| 139 | −} | |
| 140 | − | |
| 141 | −static ssize_t iommufd_fault_fops_read(struct file *filep, char __user *buf, | |
| 142 | − size_t count, loff_t *ppos) | |
| 143 | −{ | |
| 144 | − size_t fault_size = sizeof(struct iommu_hwpt_pgfault); | |
| 145 | − struct iommufd_fault *fault = filep->private_data; | |
| 146 | − struct iommu_hwpt_pgfault data = {}; | |
| 147 | − struct iommufd_device *idev; | |
| 148 | − struct iopf_group *group; | |
| 149 | − struct iopf_fault *iopf; | |
| 150 | − size_t done = 0; | |
| 151 | − int rc = 0; | |
| 152 | − | |
| 153 | − if (*ppos || count % fault_size) | |
| 154 | − return -ESPIPE; | |
| 155 | − | |
| 156 | − mutex_lock(&fault->mutex); | |
| 157 | − while ((group = iommufd_fault_deliver_fetch(fault))) { | |
| 158 | − if (done >= count || | |
| 159 | − group->fault_count * fault_size > count - done) { | |
| 160 | − iommufd_fault_deliver_restore(fault, group); | |
| 161 | − break; | |
| 162 | − } | |
| 163 | − | |
| 164 | − rc = xa_alloc(&fault->response, &group->cookie, group, | |
| 165 | − xa_limit_32b, GFP_KERNEL); | |
| 166 | − if (rc) { | |
| 167 | − iommufd_fault_deliver_restore(fault, group); | |
| 168 | − break; | |
| 169 | − } | |
| 170 | − | |
| 171 | − idev = to_iommufd_handle(group->attach_handle)->idev; | |
| 172 | − list_for_each_entry(iopf, &group->faults, list) { | |
| 173 | − iommufd_compose_fault_message(&iopf->fault, | |
| 174 | − &data, idev, | |
| 175 | − group->cookie); | |
| 176 | − if (copy_to_user(buf + done, &data, fault_size)) { | |
| 177 | − xa_erase(&fault->response, group->cookie); | |
| 178 | − iommufd_fault_deliver_restore(fault, group); | |
| 179 | − rc = -EFAULT; | |
| 180 | − break; | |
| 181 | − } | |
| 182 | − done += fault_size; | |
| 183 | − } | |
| 184 | − } | |
| 185 | − mutex_unlock(&fault->mutex); | |
| 186 | − | |
| 187 | − return done == 0 ? rc : done; | |
| 188 | −} | |
| 189 | − | |
| 190 | −static ssize_t iommufd_fault_fops_write(struct file *filep, const char __user *buf, | |
| 191 | − size_t count, loff_t *ppos) | |
| 192 | −{ | |
| 193 | − size_t response_size = sizeof(struct iommu_hwpt_page_response); | |
| 194 | − struct iommufd_fault *fault = filep->private_data; | |
| 195 | − struct iommu_hwpt_page_response response; | |
| 196 | − struct iopf_group *group; | |
| 197 | − size_t done = 0; | |
| 198 | − int rc = 0; | |
| 199 | − | |
| 200 | − if (*ppos || count % response_size) | |
| 201 | − return -ESPIPE; | |
| 202 | − | |
| 203 | − mutex_lock(&fault->mutex); | |
| 204 | − while (count > done) { | |
| 205 | − rc = copy_from_user(&response, buf + done, response_size); | |
| 206 | − if (rc) | |
| 207 | − break; | |
| 208 | − | |
| 209 | − static_assert((int)IOMMUFD_PAGE_RESP_SUCCESS == | |
| 210 | − (int)IOMMU_PAGE_RESP_SUCCESS); | |
| 211 | − static_assert((int)IOMMUFD_PAGE_RESP_INVALID == | |
| 212 | − (int)IOMMU_PAGE_RESP_INVALID); | |
| 213 | − if (response.code != IOMMUFD_PAGE_RESP_SUCCESS && | |
| 214 | − response.code != IOMMUFD_PAGE_RESP_INVALID) { | |
| 215 | − rc = -EINVAL; | |
| 216 | − break; | |
| 217 | − } | |
| 218 | − | |
| 219 | − group = xa_erase(&fault->response, response.cookie); | |
| 220 | − if (!group) { | |
| 221 | − rc = -EINVAL; | |
| 222 | − break; | |
| 223 | − } | |
| 224 | − | |
| 225 | − iopf_group_response(group, response.code); | |
| 226 | − iopf_free_group(group); | |
| 227 | − done += response_size; | |
| 228 | − } | |
| 229 | − mutex_unlock(&fault->mutex); | |
| 230 | − | |
| 231 | − return done == 0 ? rc : done; | |
| 232 | −} | |
| 233 | − | |
| 234 | −static __poll_t iommufd_fault_fops_poll(struct file *filep, | |
| 235 | − struct poll_table_struct *wait) | |
| 236 | −{ | |
| 237 | − struct iommufd_fault *fault = filep->private_data; | |
| 238 | − __poll_t pollflags = EPOLLOUT; | |
| 239 | − | |
| 240 | − poll_wait(filep, &fault->wait_queue, wait); | |
| 241 | − spin_lock(&fault->lock); | |
| 242 | − if (!list_empty(&fault->deliver)) | |
| 243 | − pollflags |= EPOLLIN | EPOLLRDNORM; | |
| 244 | − spin_unlock(&fault->lock); | |
| 245 | − | |
| 246 | − return pollflags; | |
| 247 | −} | |
| 248 | − | |
| 249 | −static int iommufd_fault_fops_release(struct inode *inode, struct file *filep) | |
| 250 | −{ | |
| 251 | − struct iommufd_fault *fault = filep->private_data; | |
| 252 | − | |
| 253 | − refcount_dec(&fault->obj.users); | |
| 254 | − iommufd_ctx_put(fault->ictx); | |
| 255 | − return 0; | |
| 256 | −} | |
| 257 | − | |
| 258 | −static const struct file_operations iommufd_fault_fops = { | |
| 259 | − .owner = THIS_MODULE, | |
| 260 | − .open = nonseekable_open, | |
| 261 | − .read = iommufd_fault_fops_read, | |
| 262 | − .write = iommufd_fault_fops_write, | |
| 263 | − .poll = iommufd_fault_fops_poll, | |
| 264 | − .release = iommufd_fault_fops_release, | |
| 265 | −}; | |
| 266 | − | |
| 267 | −int iommufd_fault_alloc(struct iommufd_ucmd *ucmd) | |
| 268 | −{ | |
| 269 | − struct iommu_fault_alloc *cmd = ucmd->cmd; | |
| 270 | − struct iommufd_fault *fault; | |
| 271 | − struct file *filep; | |
| 272 | − int fdno; | |
| 273 | − int rc; | |
| 274 | − | |
| 275 | − if (cmd->flags) | |
| 276 | − return -EOPNOTSUPP; | |
| 277 | − | |
| 278 | − fault = iommufd_object_alloc(ucmd->ictx, fault, IOMMUFD_OBJ_FAULT); | |
| 279 | − if (IS_ERR(fault)) | |
| 280 | − return PTR_ERR(fault); | |
| 281 | − | |
| 282 | − fault->ictx = ucmd->ictx; | |
| 283 | − INIT_LIST_HEAD(&fault->deliver); | |
| 284 | − xa_init_flags(&fault->response, XA_FLAGS_ALLOC1); | |
| 285 | − mutex_init(&fault->mutex); | |
| 286 | − spin_lock_init(&fault->lock); | |
| 287 | − init_waitqueue_head(&fault->wait_queue); | |
| 288 | − | |
| 289 | − filep = anon_inode_getfile("[iommufd-pgfault]", &iommufd_fault_fops, | |
| 290 | − fault, O_RDWR); | |
| 291 | − if (IS_ERR(filep)) { | |
| 292 | − rc = PTR_ERR(filep); | |
| 293 | − goto out_abort; | |
| 294 | − } | |
| 295 | − | |
| 296 | − refcount_inc(&fault->obj.users); | |
| 297 | − iommufd_ctx_get(fault->ictx); | |
| 298 | − fault->filep = filep; | |
| 299 | − | |
| 300 | − fdno = get_unused_fd_flags(O_CLOEXEC); | |
| 301 | − if (fdno < 0) { | |
| 302 | − rc = fdno; | |
| 303 | − goto out_fput; | |
| 304 | − } | |
| 305 | − | |
| 306 | − cmd->out_fault_id = fault->obj.id; | |
| 307 | − cmd->out_fault_fd = fdno; | |
| 308 | − | |
| 309 | − rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); | |
| 310 | − if (rc) | |
| 311 | − goto out_put_fdno; | |
| 312 | − iommufd_object_finalize(ucmd->ictx, &fault->obj); | |
| 313 | − | |
| 314 | − fd_install(fdno, fault->filep); | |
| 315 | − | |
| 316 | − return 0; | |
| 317 | −out_put_fdno: | |
| 318 | − put_unused_fd(fdno); | |
| 319 | −out_fput: | |
| 320 | − fput(filep); | |
| 321 | −out_abort: | |
| 322 | − iommufd_object_abort_and_destroy(ucmd->ictx, &fault->obj); | |
| 323 | − | |
| 324 | − return rc; | |
| 325 | −} | |
| 326 | − | |
| 327 | −int iommufd_fault_iopf_handler(struct iopf_group *group) | |
| 328 | −{ | |
| 329 | − struct iommufd_hw_pagetable *hwpt; | |
| 330 | − struct iommufd_fault *fault; | |
| 331 | − | |
| 332 | − hwpt = group->attach_handle->domain->iommufd_hwpt; | |
| 333 | − fault = hwpt->fault; | |
| 334 | − | |
| 335 | − spin_lock(&fault->lock); | |
| 336 | − list_add_tail(&group->node, &fault->deliver); | |
| 337 | − spin_unlock(&fault->lock); | |
| 338 | − | |
| 339 | − wake_up_interruptible(&fault->wait_queue); | |
| 340 | − | |
| 341 | − return 0; | |
| 342 | −} | |
| @@ -14,7 +14,7 @@ static void __iommufd_hwpt_destroy(struct iommufd_hw_pagetable *hwpt) · 13 unmodified lines | ||
| 14 | 14 | iommu_domain_free(hwpt->domain); |
| 15 | 15 | |
| 16 | 16 | if (hwpt->fault) |
| 17 | − refcount_dec(&hwpt->fault->obj.users); | |
| 17 | + refcount_dec(&hwpt->fault->common.obj.users); | |
| 18 | 18 | } |
| 19 | 19 | |
| 20 | 20 | void iommufd_hwpt_paging_destroy(struct iommufd_object *obj) |
| @@ -90,6 +90,7 @@ iommufd_hwpt_paging_enforce_cc(struct iommufd_hwpt_paging *hwpt_paging) · 69 unmodified lines | ||
| 90 | 90 | * @ictx: iommufd context |
| 91 | 91 | * @ioas: IOAS to associate the domain with |
| 92 | 92 | * @idev: Device to get an iommu_domain for |
| 93 | + * @pasid: PASID to get an iommu_domain for | |
| 93 | 94 | * @flags: Flags from userspace |
| 94 | 95 | * @immediate_attach: True if idev should be attached to the hwpt |
| 95 | 96 | * @user_data: The user provided driver specific data describing the domain to |
| @@ -105,13 +106,14 @@ iommufd_hwpt_paging_enforce_cc(struct iommufd_hwpt_paging *hwpt_paging) · 9 unmodified lines | ||
| 105 | 106 | */ |
| 106 | 107 | struct iommufd_hwpt_paging * |
| 107 | 108 | iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, |
| 108 | − struct iommufd_device *idev, u32 flags, | |
| 109 | − bool immediate_attach, | |
| 109 | + struct iommufd_device *idev, ioasid_t pasid, | |
| 110 | + u32 flags, bool immediate_attach, | |
| 110 | 111 | const struct iommu_user_data *user_data) |
| 111 | 112 | { |
| 112 | 113 | const u32 valid_flags = IOMMU_HWPT_ALLOC_NEST_PARENT | |
| 113 | 114 | IOMMU_HWPT_ALLOC_DIRTY_TRACKING | |
| 114 | − IOMMU_HWPT_FAULT_ID_VALID; | |
| 115 | + IOMMU_HWPT_FAULT_ID_VALID | | |
| 116 | + IOMMU_HWPT_ALLOC_PASID; | |
| 115 | 117 | const struct iommu_ops *ops = dev_iommu_ops(idev->dev); |
| 116 | 118 | struct iommufd_hwpt_paging *hwpt_paging; |
| 117 | 119 | struct iommufd_hw_pagetable *hwpt; |
| @@ -126,12 +128,16 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 8 unmodified lines | ||
| 126 | 128 | if ((flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING) && |
| 127 | 129 | !device_iommu_capable(idev->dev, IOMMU_CAP_DIRTY_TRACKING)) |
| 128 | 130 | return ERR_PTR(-EOPNOTSUPP); |
| 131 | + if ((flags & IOMMU_HWPT_FAULT_ID_VALID) && | |
| 132 | + (flags & IOMMU_HWPT_ALLOC_NEST_PARENT)) | |
| 133 | + return ERR_PTR(-EOPNOTSUPP); | |
| 129 | 134 | |
| 130 | 135 | hwpt_paging = __iommufd_object_alloc( |
| 131 | 136 | ictx, hwpt_paging, IOMMUFD_OBJ_HWPT_PAGING, common.obj); |
| 132 | 137 | if (IS_ERR(hwpt_paging)) |
| 133 | 138 | return ERR_CAST(hwpt_paging); |
| 134 | 139 | hwpt = &hwpt_paging->common; |
| 140 | + hwpt->pasid_compat = flags & IOMMU_HWPT_ALLOC_PASID; | |
| 135 | 141 | |
| 136 | 142 | INIT_LIST_HEAD(&hwpt_paging->hwpt_item); |
| 137 | 143 | /* Pairs with iommufd_hw_pagetable_destroy() */ |
| @@ -156,7 +162,8 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 18 unmodified lines | ||
| 156 | 162 | goto out_abort; |
| 157 | 163 | } |
| 158 | 164 | } |
| 159 | − iommu_domain_set_sw_msi(hwpt->domain, iommufd_sw_msi); | |
| 165 | + hwpt->domain->iommufd_hwpt = hwpt; | |
| 166 | + hwpt->domain->cookie_type = IOMMU_COOKIE_IOMMUFD; | |
| 160 | 167 | |
| 161 | 168 | /* |
| 162 | 169 | * Set the coherency mode before we do iopt_table_add_domain() as some |
| @@ -185,7 +192,7 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 22 unmodified lines | ||
| 185 | 192 | * sequence. Once those drivers are fixed this should be removed. |
| 186 | 193 | */ |
| 187 | 194 | if (immediate_attach) { |
| 188 | − rc = iommufd_hw_pagetable_attach(hwpt, idev); | |
| 195 | + rc = iommufd_hw_pagetable_attach(hwpt, idev, pasid); | |
| 189 | 196 | if (rc) |
| 190 | 197 | goto out_abort; |
| 191 | 198 | } |
| @@ -198,7 +205,7 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, · 6 unmodified lines | ||
| 198 | 205 | |
| 199 | 206 | out_detach: |
| 200 | 207 | if (immediate_attach) |
| 201 | − iommufd_hw_pagetable_detach(idev); | |
| 208 | + iommufd_hw_pagetable_detach(idev, pasid); | |
| 202 | 209 | out_abort: |
| 203 | 210 | iommufd_object_abort_and_destroy(ictx, &hwpt->obj); |
| 204 | 211 | return ERR_PTR(rc); |
| @@ -227,7 +234,7 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, · 22 unmodified lines | ||
| 227 | 234 | struct iommufd_hw_pagetable *hwpt; |
| 228 | 235 | int rc; |
| 229 | 236 | |
| 230 | − if ((flags & ~IOMMU_HWPT_FAULT_ID_VALID) || | |
| 237 | + if ((flags & ~(IOMMU_HWPT_FAULT_ID_VALID | IOMMU_HWPT_ALLOC_PASID)) || | |
| 231 | 238 | !user_data->len || !ops->domain_alloc_nested) |
| 232 | 239 | return ERR_PTR(-EOPNOTSUPP); |
| 233 | 240 | if (parent->auto_domain || !parent->nest_parent || |
| @@ -239,6 +246,7 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, · 5 unmodified lines | ||
| 239 | 246 | if (IS_ERR(hwpt_nested)) |
| 240 | 247 | return ERR_CAST(hwpt_nested); |
| 241 | 248 | hwpt = &hwpt_nested->common; |
| 249 | + hwpt->pasid_compat = flags & IOMMU_HWPT_ALLOC_PASID; | |
| 242 | 250 | |
| 243 | 251 | refcount_inc(&parent->common.obj.users); |
| 244 | 252 | hwpt_nested->parent = parent; |
| @@ -252,7 +260,8 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, · 7 unmodified lines | ||
| 252 | 260 | goto out_abort; |
| 253 | 261 | } |
| 254 | 262 | hwpt->domain->owner = ops; |
| 255 | − iommu_domain_set_sw_msi(hwpt->domain, iommufd_sw_msi); | |
| 263 | + hwpt->domain->iommufd_hwpt = hwpt; | |
| 264 | + hwpt->domain->cookie_type = IOMMU_COOKIE_IOMMUFD; | |
| 256 | 265 | |
| 257 | 266 | if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) { |
| 258 | 267 | rc = -EINVAL; |
| @@ -282,7 +291,7 @@ iommufd_viommu_alloc_hwpt_nested(struct iommufd_viommu *viommu, u32 flags, · 23 unmodified lines | ||
| 282 | 291 | struct iommufd_hw_pagetable *hwpt; |
| 283 | 292 | int rc; |
| 284 | 293 | |
| 285 | − if (flags & ~IOMMU_HWPT_FAULT_ID_VALID) | |
| 294 | + if (flags & ~(IOMMU_HWPT_FAULT_ID_VALID | IOMMU_HWPT_ALLOC_PASID)) | |
| 286 | 295 | return ERR_PTR(-EOPNOTSUPP); |
| 287 | 296 | if (!user_data->len) |
| 288 | 297 | return ERR_PTR(-EOPNOTSUPP); |
| @@ -294,6 +303,7 @@ iommufd_viommu_alloc_hwpt_nested(struct iommufd_viommu *viommu, u32 flags, · 5 unmodified lines | ||
| 294 | 303 | if (IS_ERR(hwpt_nested)) |
| 295 | 304 | return ERR_CAST(hwpt_nested); |
| 296 | 305 | hwpt = &hwpt_nested->common; |
| 306 | + hwpt->pasid_compat = flags & IOMMU_HWPT_ALLOC_PASID; | |
| 297 | 307 | |
| 298 | 308 | hwpt_nested->viommu = viommu; |
| 299 | 309 | refcount_inc(&viommu->obj.users); |
| @@ -308,8 +318,9 @@ iommufd_viommu_alloc_hwpt_nested(struct iommufd_viommu *viommu, u32 flags, · 8 unmodified lines | ||
| 308 | 318 | hwpt->domain = NULL; |
| 309 | 319 | goto out_abort; |
| 310 | 320 | } |
| 321 | + hwpt->domain->iommufd_hwpt = hwpt; | |
| 311 | 322 | hwpt->domain->owner = viommu->iommu_dev->ops; |
| 312 | − iommu_domain_set_sw_msi(hwpt->domain, iommufd_sw_msi); | |
| 323 | + hwpt->domain->cookie_type = IOMMU_COOKIE_IOMMUFD; | |
| 313 | 324 | |
| 314 | 325 | if (WARN_ON_ONCE(hwpt->domain->type != IOMMU_DOMAIN_NESTED)) { |
| 315 | 326 | rc = -EINVAL; |
| @@ -358,8 +369,8 @@ int iommufd_hwpt_alloc(struct iommufd_ucmd *ucmd) · 42 unmodified lines | ||
| 358 | 369 | ioas = container_of(pt_obj, struct iommufd_ioas, obj); |
| 359 | 370 | mutex_lock(&ioas->mutex); |
| 360 | 371 | hwpt_paging = iommufd_hwpt_paging_alloc( |
| 361 | − ucmd->ictx, ioas, idev, cmd->flags, false, | |
| 362 | − user_data.len ? &user_data : NULL); | |
| 372 | + ucmd->ictx, ioas, idev, IOMMU_NO_PASID, cmd->flags, | |
| 373 | + false, user_data.len ? &user_data : NULL); | |
| 363 | 374 | if (IS_ERR(hwpt_paging)) { |
| 364 | 375 | rc = PTR_ERR(hwpt_paging); |
| 365 | 376 | goto out_unlock; |
| @@ -409,10 +420,9 @@ int iommufd_hwpt_alloc(struct iommufd_ucmd *ucmd) · 43 unmodified lines | ||
| 409 | 420 | } |
| 410 | 421 | hwpt->fault = fault; |
| 411 | 422 | hwpt->domain->iopf_handler = iommufd_fault_iopf_handler; |
| 412 | − refcount_inc(&fault->obj.users); | |
| 413 | − iommufd_put_object(ucmd->ictx, &fault->obj); | |
| 423 | + refcount_inc(&fault->common.obj.users); | |
| 424 | + iommufd_put_object(ucmd->ictx, &fault->common.obj); | |
| 414 | 425 | } |
| 415 | − hwpt->domain->iommufd_hwpt = hwpt; | |
| 416 | 426 | |
| 417 | 427 | cmd->out_hwpt_id = hwpt->obj.id; |
| 418 | 428 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); |
| @@ -32,8 +32,11 @@ struct iommufd_sw_msi_maps { · 31 unmodified lines | ||
| 32 | 32 | DECLARE_BITMAP(bitmap, 64); |
| 33 | 33 | }; |
| 34 | 34 | |
| 35 | −int iommufd_sw_msi(struct iommu_domain *domain, struct msi_desc *desc, | |
| 36 | − phys_addr_t msi_addr); | |
| 35 | +#ifdef CONFIG_IRQ_MSI_IOMMU | |
| 36 | +int iommufd_sw_msi_install(struct iommufd_ctx *ictx, | |
| 37 | + struct iommufd_hwpt_paging *hwpt_paging, | |
| 38 | + struct iommufd_sw_msi_map *msi_map); | |
| 39 | +#endif | |
| 37 | 40 | |
| 38 | 41 | struct iommufd_ctx { |
| 39 | 42 | struct file *file; |
| @@ -296,6 +299,7 @@ struct iommufd_hw_pagetable { · 256 unmodified lines | ||
| 296 | 299 | struct iommufd_object obj; |
| 297 | 300 | struct iommu_domain *domain; |
| 298 | 301 | struct iommufd_fault *fault; |
| 302 | + bool pasid_compat : 1; | |
| 299 | 303 | }; |
| 300 | 304 | |
| 301 | 305 | struct iommufd_hwpt_paging { |
| @@ -366,13 +370,13 @@ int iommufd_hwpt_get_dirty_bitmap(struct iommufd_ucmd *ucmd); · 64 unmodified lines | ||
| 366 | 370 | |
| 367 | 371 | struct iommufd_hwpt_paging * |
| 368 | 372 | iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas, |
| 369 | − struct iommufd_device *idev, u32 flags, | |
| 370 | − bool immediate_attach, | |
| 373 | + struct iommufd_device *idev, ioasid_t pasid, | |
| 374 | + u32 flags, bool immediate_attach, | |
| 371 | 375 | const struct iommu_user_data *user_data); |
| 372 | 376 | int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt, |
| 373 | − struct iommufd_device *idev); | |
| 377 | + struct iommufd_device *idev, ioasid_t pasid); | |
| 374 | 378 | struct iommufd_hw_pagetable * |
| 375 | −iommufd_hw_pagetable_detach(struct iommufd_device *idev); | |
| 379 | +iommufd_hw_pagetable_detach(struct iommufd_device *idev, ioasid_t pasid); | |
| 376 | 380 | void iommufd_hwpt_paging_destroy(struct iommufd_object *obj); |
| 377 | 381 | void iommufd_hwpt_paging_abort(struct iommufd_object *obj); |
| 378 | 382 | void iommufd_hwpt_nested_destroy(struct iommufd_object *obj); |
| @@ -396,13 +400,14 @@ static inline void iommufd_hw_pagetable_put(struct iommufd_ctx *ictx, · 17 unmodified lines | ||
| 396 | 400 | refcount_dec(&hwpt->obj.users); |
| 397 | 401 | } |
| 398 | 402 | |
| 403 | +struct iommufd_attach; | |
| 404 | + | |
| 399 | 405 | struct iommufd_group { |
| 400 | 406 | struct kref ref; |
| 401 | 407 | struct mutex lock; |
| 402 | 408 | struct iommufd_ctx *ictx; |
| 403 | 409 | struct iommu_group *group; |
| 404 | − struct iommufd_hw_pagetable *hwpt; | |
| 405 | − struct list_head device_list; | |
| 410 | + struct xarray pasid_attach; | |
| 406 | 411 | struct iommufd_sw_msi_maps required_sw_msi; |
| 407 | 412 | phys_addr_t sw_msi_start; |
| 408 | 413 | }; |
| @@ -454,49 +459,17 @@ void iopt_remove_access(struct io_pagetable *iopt, · 45 unmodified lines | ||
| 454 | 459 | u32 iopt_access_list_id); |
| 455 | 460 | void iommufd_access_destroy_object(struct iommufd_object *obj); |
| 456 | 461 | |
| 457 | −/* | |
| 458 | − * An iommufd_fault object represents an interface to deliver I/O page faults | |
| 459 | − * to the user space. These objects are created/destroyed by the user space and | |
| 460 | − * associated with hardware page table objects during page-table allocation. | |
| 461 | − */ | |
| 462 | −struct iommufd_fault { | |
| 462 | +struct iommufd_eventq { | |
| 463 | 463 | struct iommufd_object obj; |
| 464 | 464 | struct iommufd_ctx *ictx; |
| 465 | 465 | struct file *filep; |
| 466 | 466 | |
| 467 | 467 | spinlock_t lock; /* protects the deliver list */ |
| 468 | 468 | struct list_head deliver; |
| 469 | − struct mutex mutex; /* serializes response flows */ | |
| 470 | − struct xarray response; | |
| 471 | 469 | |
| 472 | 470 | struct wait_queue_head wait_queue; |
| 473 | 471 | }; |
| 474 | 472 | |
| 475 | −/* Fetch the first node out of the fault->deliver list */ | |
| 476 | −static inline struct iopf_group * | |
| 477 | −iommufd_fault_deliver_fetch(struct iommufd_fault *fault) | |
| 478 | −{ | |
| 479 | − struct list_head *list = &fault->deliver; | |
| 480 | − struct iopf_group *group = NULL; | |
| 481 | − | |
| 482 | − spin_lock(&fault->lock); | |
| 483 | − if (!list_empty(list)) { | |
| 484 | − group = list_first_entry(list, struct iopf_group, node); | |
| 485 | − list_del(&group->node); | |
| 486 | − } | |
| 487 | − spin_unlock(&fault->lock); | |
| 488 | − return group; | |
| 489 | −} | |
| 490 | − | |
| 491 | −/* Restore a node back to the head of the fault->deliver list */ | |
| 492 | −static inline void iommufd_fault_deliver_restore(struct iommufd_fault *fault, | |
| 493 | − struct iopf_group *group) | |
| 494 | −{ | |
| 495 | − spin_lock(&fault->lock); | |
| 496 | − list_add(&group->node, &fault->deliver); | |
| 497 | − spin_unlock(&fault->lock); | |
| 498 | −} | |
| 499 | − | |
| 500 | 473 | struct iommufd_attach_handle { |
| 501 | 474 | struct iommu_attach_handle handle; |
| 502 | 475 | struct iommufd_device *idev; |
| @@ -505,12 +478,29 @@ struct iommufd_attach_handle { · 2 unmodified lines | ||
| 505 | 478 | /* Convert an iommu attach handle to iommufd handle. */ |
| 506 | 479 | #define to_iommufd_handle(hdl) container_of(hdl, struct iommufd_attach_handle, handle) |
| 507 | 480 | |
| 481 | +/* | |
| 482 | + * An iommufd_fault object represents an interface to deliver I/O page faults | |
| 483 | + * to the user space. These objects are created/destroyed by the user space and | |
| 484 | + * associated with hardware page table objects during page-table allocation. | |
| 485 | + */ | |
| 486 | +struct iommufd_fault { | |
| 487 | + struct iommufd_eventq common; | |
| 488 | + struct mutex mutex; /* serializes response flows */ | |
| 489 | + struct xarray response; | |
| 490 | +}; | |
| 491 | + | |
| 492 | +static inline struct iommufd_fault * | |
| 493 | +eventq_to_fault(struct iommufd_eventq *eventq) | |
| 494 | +{ | |
| 495 | + return container_of(eventq, struct iommufd_fault, common); | |
| 496 | +} | |
| 497 | + | |
| 508 | 498 | static inline struct iommufd_fault * |
| 509 | 499 | iommufd_get_fault(struct iommufd_ucmd *ucmd, u32 id) |
| 510 | 500 | { |
| 511 | 501 | return container_of(iommufd_get_object(ucmd->ictx, id, |
| 512 | 502 | IOMMUFD_OBJ_FAULT), |
| 513 | − struct iommufd_fault, obj); | |
| 503 | + struct iommufd_fault, common.obj); | |
| 514 | 504 | } |
| 515 | 505 | |
| 516 | 506 | int iommufd_fault_alloc(struct iommufd_ucmd *ucmd); |
| @@ -522,6 +512,74 @@ void iommufd_fault_iopf_disable(struct iommufd_device *idev); · 5 unmodified lines | ||
| 522 | 512 | void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt, |
| 523 | 513 | struct iommufd_attach_handle *handle); |
| 524 | 514 | |
| 515 | +/* An iommufd_vevent represents a vIOMMU event in an iommufd_veventq */ | |
| 516 | +struct iommufd_vevent { | |
| 517 | + struct iommufd_vevent_header header; | |
| 518 | + struct list_head node; /* for iommufd_eventq::deliver */ | |
| 519 | + ssize_t data_len; | |
| 520 | + u64 event_data[] __counted_by(data_len); | |
| 521 | +}; | |
| 522 | + | |
| 523 | +#define vevent_for_lost_events_header(vevent) \ | |
| 524 | + (vevent->header.flags & IOMMU_VEVENTQ_FLAG_LOST_EVENTS) | |
| 525 | + | |
| 526 | +/* | |
| 527 | + * An iommufd_veventq object represents an interface to deliver vIOMMU events to | |
| 528 | + * the user space. It is created/destroyed by the user space and associated with | |
| 529 | + * a vIOMMU object during the allocations. | |
| 530 | + */ | |
| 531 | +struct iommufd_veventq { | |
| 532 | + struct iommufd_eventq common; | |
| 533 | + struct iommufd_viommu *viommu; | |
| 534 | + struct list_head node; /* for iommufd_viommu::veventqs */ | |
| 535 | + struct iommufd_vevent lost_events_header; | |
| 536 | + | |
| 537 | + unsigned int type; | |
| 538 | + unsigned int depth; | |
| 539 | + | |
| 540 | + /* Use common.lock for protection */ | |
| 541 | + u32 num_events; | |
| 542 | + u32 sequence; | |
| 543 | +}; | |
| 544 | + | |
| 545 | +static inline struct iommufd_veventq * | |
| 546 | +eventq_to_veventq(struct iommufd_eventq *eventq) | |
| 547 | +{ | |
| 548 | + return container_of(eventq, struct iommufd_veventq, common); | |
| 549 | +} | |
| 550 | + | |
| 551 | +static inline struct iommufd_veventq * | |
| 552 | +iommufd_get_veventq(struct iommufd_ucmd *ucmd, u32 id) | |
| 553 | +{ | |
| 554 | + return container_of(iommufd_get_object(ucmd->ictx, id, | |
| 555 | + IOMMUFD_OBJ_VEVENTQ), | |
| 556 | + struct iommufd_veventq, common.obj); | |
| 557 | +} | |
| 558 | + | |
| 559 | +int iommufd_veventq_alloc(struct iommufd_ucmd *ucmd); | |
| 560 | +void iommufd_veventq_destroy(struct iommufd_object *obj); | |
| 561 | +void iommufd_veventq_abort(struct iommufd_object *obj); | |
| 562 | + | |
| 563 | +static inline void iommufd_vevent_handler(struct iommufd_veventq *veventq, | |
| 564 | + struct iommufd_vevent *vevent) | |
| 565 | +{ | |
| 566 | + struct iommufd_eventq *eventq = &veventq->common; | |
| 567 | + | |
| 568 | + lockdep_assert_held(&eventq->lock); | |
| 569 | + | |
| 570 | + /* | |
| 571 | + * Remove the lost_events_header and add the new node at the same time. | |
| 572 | + * Note the new node can be lost_events_header, for a sequence update. | |
| 573 | + */ | |
| 574 | + if (list_is_last(&veventq->lost_events_header.node, &eventq->deliver)) | |
| 575 | + list_del(&veventq->lost_events_header.node); | |
| 576 | + list_add_tail(&vevent->node, &eventq->deliver); | |
| 577 | + vevent->header.sequence = veventq->sequence; | |
| 578 | + veventq->sequence = (veventq->sequence + 1) & INT_MAX; | |
| 579 | + | |
| 580 | + wake_up_interruptible(&eventq->wait_queue); | |
| 581 | +} | |
| 582 | + | |
| 525 | 583 | static inline struct iommufd_viommu * |
| 526 | 584 | iommufd_get_viommu(struct iommufd_ucmd *ucmd, u32 id) |
| 527 | 585 | { |
| @@ -530,6 +588,20 @@ iommufd_get_viommu(struct iommufd_ucmd *ucmd, u32 id) · 2 unmodified lines | ||
| 530 | 588 | struct iommufd_viommu, obj); |
| 531 | 589 | } |
| 532 | 590 | |
| 591 | +static inline struct iommufd_veventq * | |
| 592 | +iommufd_viommu_find_veventq(struct iommufd_viommu *viommu, u32 type) | |
| 593 | +{ | |
| 594 | + struct iommufd_veventq *veventq, *next; | |
| 595 | + | |
| 596 | + lockdep_assert_held(&viommu->veventqs_rwsem); | |
| 597 | + | |
| 598 | + list_for_each_entry_safe(veventq, next, &viommu->veventqs, node) { | |
| 599 | + if (veventq->type == type) | |
| 600 | + return veventq; | |
| 601 | + } | |
| 602 | + return NULL; | |
| 603 | +} | |
| 604 | + | |
| 533 | 605 | int iommufd_viommu_alloc_ioctl(struct iommufd_ucmd *ucmd); |
| 534 | 606 | void iommufd_viommu_destroy(struct iommufd_object *obj); |
| 535 | 607 | int iommufd_vdevice_alloc_ioctl(struct iommufd_ucmd *ucmd); |
| @@ -24,6 +24,11 @@ enum { · 23 unmodified lines | ||
| 24 | 24 | IOMMU_TEST_OP_MD_CHECK_IOTLB, |
| 25 | 25 | IOMMU_TEST_OP_TRIGGER_IOPF, |
| 26 | 26 | IOMMU_TEST_OP_DEV_CHECK_CACHE, |
| 27 | + IOMMU_TEST_OP_TRIGGER_VEVENT, | |
| 28 | + IOMMU_TEST_OP_PASID_ATTACH, | |
| 29 | + IOMMU_TEST_OP_PASID_REPLACE, | |
| 30 | + IOMMU_TEST_OP_PASID_DETACH, | |
| 31 | + IOMMU_TEST_OP_PASID_CHECK_HWPT, | |
| 27 | 32 | }; |
| 28 | 33 | |
| 29 | 34 | enum { |
| @@ -48,6 +53,7 @@ enum { · 18 unmodified lines | ||
| 48 | 53 | enum { |
| 49 | 54 | MOCK_FLAGS_DEVICE_NO_DIRTY = 1 << 0, |
| 50 | 55 | MOCK_FLAGS_DEVICE_HUGE_IOVA = 1 << 1, |
| 56 | + MOCK_FLAGS_DEVICE_PASID = 1 << 2, | |
| 51 | 57 | }; |
| 52 | 58 | |
| 53 | 59 | enum { |
| @@ -60,6 +66,9 @@ enum { · 6 unmodified lines | ||
| 60 | 66 | MOCK_DEV_CACHE_NUM = 4, |
| 61 | 67 | }; |
| 62 | 68 | |
| 69 | +/* Reserved for special pasid replace test */ | |
| 70 | +#define IOMMU_TEST_PASID_RESERVED 1024 | |
| 71 | + | |
| 63 | 72 | struct iommu_test_cmd { |
| 64 | 73 | __u32 size; |
| 65 | 74 | __u32 op; |
| @@ -145,11 +154,36 @@ struct iommu_test_cmd { · 79 unmodified lines | ||
| 145 | 154 | __u32 id; |
| 146 | 155 | __u32 cache; |
| 147 | 156 | } check_dev_cache; |
| 157 | + struct { | |
| 158 | + __u32 dev_id; | |
| 159 | + } trigger_vevent; | |
| 160 | + struct { | |
| 161 | + __u32 pasid; | |
| 162 | + __u32 pt_id; | |
| 163 | + /* @id is stdev_id */ | |
| 164 | + } pasid_attach; | |
| 165 | + struct { | |
| 166 | + __u32 pasid; | |
| 167 | + __u32 pt_id; | |
| 168 | + /* @id is stdev_id */ | |
| 169 | + } pasid_replace; | |
| 170 | + struct { | |
| 171 | + __u32 pasid; | |
| 172 | + /* @id is stdev_id */ | |
| 173 | + } pasid_detach; | |
| 174 | + struct { | |
| 175 | + __u32 pasid; | |
| 176 | + __u32 hwpt_id; | |
| 177 | + /* @id is stdev_id */ | |
| 178 | + } pasid_check; | |
| 148 | 179 | }; |
| 149 | 180 | __u32 last; |
| 150 | 181 | }; |
| 151 | 182 | #define IOMMU_TEST_CMD _IO(IOMMUFD_TYPE, IOMMUFD_CMD_BASE + 32) |
| 152 | 183 | |
| 184 | +/* Mock device/iommu PASID width */ | |
| 185 | +#define MOCK_PASID_WIDTH 20 | |
| 186 | + | |
| 153 | 187 | /* Mock structs for IOMMU_DEVICE_GET_HW_INFO ioctl */ |
| 154 | 188 | #define IOMMU_HW_INFO_TYPE_SELFTEST 0xfeedbeef |
| 155 | 189 | #define IOMMU_HW_INFO_SELFTEST_REGVAL 0xdeadbeef |
| @@ -212,4 +246,10 @@ struct iommu_viommu_invalidate_selftest { · 56 unmodified lines | ||
| 212 | 246 | __u32 cache_id; |
| 213 | 247 | }; |
| 214 | 248 | |
| 249 | +#define IOMMU_VEVENTQ_TYPE_SELFTEST 0xbeefbeef | |
| 250 | + | |
| 251 | +struct iommu_viommu_event_selftest { | |
| 252 | + __u32 virt_id; | |
| 253 | +}; | |
| 254 | + | |
| 215 | 255 | #endif |
| @@ -317,6 +317,7 @@ union ucmd_buffer { · 316 unmodified lines | ||
| 317 | 317 | struct iommu_ioas_unmap unmap; |
| 318 | 318 | struct iommu_option option; |
| 319 | 319 | struct iommu_vdevice_alloc vdev; |
| 320 | + struct iommu_veventq_alloc veventq; | |
| 320 | 321 | struct iommu_vfio_ioas vfio_ioas; |
| 321 | 322 | struct iommu_viommu_alloc viommu; |
| 322 | 323 | #ifdef CONFIG_IOMMUFD_TEST |
| @@ -372,6 +373,8 @@ static const struct iommufd_ioctl_op iommufd_ioctl_ops[] = { · 49 unmodified lines | ||
| 372 | 373 | IOCTL_OP(IOMMU_OPTION, iommufd_option, struct iommu_option, val64), |
| 373 | 374 | IOCTL_OP(IOMMU_VDEVICE_ALLOC, iommufd_vdevice_alloc_ioctl, |
| 374 | 375 | struct iommu_vdevice_alloc, virt_id), |
| 376 | + IOCTL_OP(IOMMU_VEVENTQ_ALLOC, iommufd_veventq_alloc, | |
| 377 | + struct iommu_veventq_alloc, out_veventq_fd), | |
| 375 | 378 | IOCTL_OP(IOMMU_VFIO_IOAS, iommufd_vfio_ioas, struct iommu_vfio_ioas, |
| 376 | 379 | __reserved), |
| 377 | 380 | IOCTL_OP(IOMMU_VIOMMU_ALLOC, iommufd_viommu_alloc_ioctl, |
| @@ -514,6 +517,10 @@ static const struct iommufd_object_ops iommufd_object_ops[] = { · 136 unmodified lines | ||
| 514 | 517 | [IOMMUFD_OBJ_VDEVICE] = { |
| 515 | 518 | .destroy = iommufd_vdevice_destroy, |
| 516 | 519 | }, |
| 520 | + [IOMMUFD_OBJ_VEVENTQ] = { | |
| 521 | + .destroy = iommufd_veventq_destroy, | |
| 522 | + .abort = iommufd_veventq_abort, | |
| 523 | + }, | |
| 517 | 524 | [IOMMUFD_OBJ_VIOMMU] = { |
| 518 | 525 | .destroy = iommufd_viommu_destroy, |
| 519 | 526 | }, |
| @@ -161,9 +161,13 @@ enum selftest_obj_type { · 160 unmodified lines | ||
| 161 | 161 | |
| 162 | 162 | struct mock_dev { |
| 163 | 163 | struct device dev; |
| 164 | + struct mock_viommu *viommu; | |
| 165 | + struct rw_semaphore viommu_rwsem; | |
| 164 | 166 | unsigned long flags; |
| 167 | + unsigned long vdev_id; | |
| 165 | 168 | int id; |
| 166 | 169 | u32 cache[MOCK_DEV_CACHE_NUM]; |
| 170 | + atomic_t pasid_1024_fake_error; | |
| 167 | 171 | }; |
| 168 | 172 | |
| 169 | 173 | static inline struct mock_dev *to_mock_dev(struct device *dev) |
| @@ -193,15 +197,71 @@ static int mock_domain_nop_attach(struct iommu_domain *domain, · 23 unmodified lines | ||
| 193 | 197 | struct device *dev) |
| 194 | 198 | { |
| 195 | 199 | struct mock_dev *mdev = to_mock_dev(dev); |
| 200 | + struct mock_viommu *new_viommu = NULL; | |
| 201 | + unsigned long vdev_id = 0; | |
| 202 | + int rc; | |
| 196 | 203 | |
| 197 | 204 | if (domain->dirty_ops && (mdev->flags & MOCK_FLAGS_DEVICE_NO_DIRTY)) |
| 198 | 205 | return -EINVAL; |
| 199 | 206 | |
| 207 | + iommu_group_mutex_assert(dev); | |
| 208 | + if (domain->type == IOMMU_DOMAIN_NESTED) { | |
| 209 | + new_viommu = to_mock_nested(domain)->mock_viommu; | |
| 210 | + if (new_viommu) { | |
| 211 | + rc = iommufd_viommu_get_vdev_id(&new_viommu->core, dev, | |
| 212 | + &vdev_id); | |
| 213 | + if (rc) | |
| 214 | + return rc; | |
| 215 | + } | |
| 216 | + } | |
| 217 | + if (new_viommu != mdev->viommu) { | |
| 218 | + down_write(&mdev->viommu_rwsem); | |
| 219 | + mdev->viommu = new_viommu; | |
| 220 | + mdev->vdev_id = vdev_id; | |
| 221 | + up_write(&mdev->viommu_rwsem); | |
| 222 | + } | |
| 223 | + | |
| 224 | + return 0; | |
| 225 | +} | |
| 226 | + | |
| 227 | +static int mock_domain_set_dev_pasid_nop(struct iommu_domain *domain, | |
| 228 | + struct device *dev, ioasid_t pasid, | |
| 229 | + struct iommu_domain *old) | |
| 230 | +{ | |
| 231 | + struct mock_dev *mdev = to_mock_dev(dev); | |
| 232 | + | |
| 233 | + /* | |
| 234 | + * Per the first attach with pasid 1024, set the | |
| 235 | + * mdev->pasid_1024_fake_error. Hence the second call of this op | |
| 236 | + * can fake an error to validate the error path of the core. This | |
| 237 | + * is helpful to test the case in which the iommu core needs to | |
| 238 | + * rollback to the old domain due to driver failure. e.g. replace. | |
| 239 | + * User should be careful about the third call of this op, it shall | |
| 240 | + * succeed since the mdev->pasid_1024_fake_error is cleared in the | |
| 241 | + * second call. | |
| 242 | + */ | |
| 243 | + if (pasid == 1024) { | |
| 244 | + if (domain->type == IOMMU_DOMAIN_BLOCKED) { | |
| 245 | + atomic_set(&mdev->pasid_1024_fake_error, 0); | |
| 246 | + } else if (atomic_read(&mdev->pasid_1024_fake_error)) { | |
| 247 | + /* | |
| 248 | + * Clear the flag, and fake an error to fail the | |
| 249 | + * replacement. | |
| 250 | + */ | |
| 251 | + atomic_set(&mdev->pasid_1024_fake_error, 0); | |
| 252 | + return -ENOMEM; | |
| 253 | + } else { | |
| 254 | + /* Set the flag to fake an error in next call */ | |
| 255 | + atomic_set(&mdev->pasid_1024_fake_error, 1); | |
| 256 | + } | |
| 257 | + } | |
| 258 | + | |
| 200 | 259 | return 0; |
| 201 | 260 | } |
| 202 | 261 | |
| 203 | 262 | static const struct iommu_domain_ops mock_blocking_ops = { |
| 204 | 263 | .attach_dev = mock_domain_nop_attach, |
| 264 | + .set_dev_pasid = mock_domain_set_dev_pasid_nop | |
| 205 | 265 | }; |
| 206 | 266 | |
| 207 | 267 | static struct iommu_domain mock_blocking_domain = { |
| @@ -343,7 +403,7 @@ mock_domain_alloc_nested(struct device *dev, struct iommu_domain *parent, · 135 unmodified lines | ||
| 343 | 403 | struct mock_iommu_domain_nested *mock_nested; |
| 344 | 404 | struct mock_iommu_domain *mock_parent; |
| 345 | 405 | |
| 346 | − if (flags) | |
| 406 | + if (flags & ~IOMMU_HWPT_ALLOC_PASID) | |
| 347 | 407 | return ERR_PTR(-EOPNOTSUPP); |
| 348 | 408 | if (!parent || parent->ops != mock_ops.default_domain_ops) |
| 349 | 409 | return ERR_PTR(-EINVAL); |
| @@ -365,7 +425,8 @@ mock_domain_alloc_paging_flags(struct device *dev, u32 flags, · 15 unmodified lines | ||
| 365 | 425 | { |
| 366 | 426 | bool has_dirty_flag = flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING; |
| 367 | 427 | const u32 PAGING_FLAGS = IOMMU_HWPT_ALLOC_DIRTY_TRACKING | |
| 368 | − IOMMU_HWPT_ALLOC_NEST_PARENT; | |
| 428 | + IOMMU_HWPT_ALLOC_NEST_PARENT | | |
| 429 | + IOMMU_HWPT_ALLOC_PASID; | |
| 369 | 430 | struct mock_dev *mdev = to_mock_dev(dev); |
| 370 | 431 | bool no_dirty_ops = mdev->flags & MOCK_FLAGS_DEVICE_NO_DIRTY; |
| 371 | 432 | struct mock_iommu_domain *mock; |
| @@ -585,7 +646,7 @@ mock_viommu_alloc_domain_nested(struct iommufd_viommu *viommu, u32 flags, · 213 unmodified lines | ||
| 585 | 646 | struct mock_viommu *mock_viommu = to_mock_viommu(viommu); |
| 586 | 647 | struct mock_iommu_domain_nested *mock_nested; |
| 587 | 648 | |
| 588 | − if (flags) | |
| 649 | + if (flags & ~IOMMU_HWPT_ALLOC_PASID) | |
| 589 | 650 | return ERR_PTR(-EOPNOTSUPP); |
| 590 | 651 | |
| 591 | 652 | mock_nested = __mock_domain_alloc_nested(user_data); |
| @@ -720,6 +781,7 @@ static const struct iommu_ops mock_ops = { · 128 unmodified lines | ||
| 720 | 781 | .map_pages = mock_domain_map_pages, |
| 721 | 782 | .unmap_pages = mock_domain_unmap_pages, |
| 722 | 783 | .iova_to_phys = mock_domain_iova_to_phys, |
| 784 | + .set_dev_pasid = mock_domain_set_dev_pasid_nop, | |
| 723 | 785 | }, |
| 724 | 786 | }; |
| 725 | 787 | |
| @@ -780,6 +842,7 @@ static struct iommu_domain_ops domain_nested_ops = { · 54 unmodified lines | ||
| 780 | 842 | .free = mock_domain_free_nested, |
| 781 | 843 | .attach_dev = mock_domain_nop_attach, |
| 782 | 844 | .cache_invalidate_user = mock_domain_cache_invalidate_user, |
| 845 | + .set_dev_pasid = mock_domain_set_dev_pasid_nop, | |
| 783 | 846 | }; |
| 784 | 847 | |
| 785 | 848 | static inline struct iommufd_hw_pagetable * |
| @@ -839,17 +902,24 @@ static void mock_dev_release(struct device *dev) · 53 unmodified lines | ||
| 839 | 902 | |
| 840 | 903 | static struct mock_dev *mock_dev_create(unsigned long dev_flags) |
| 841 | 904 | { |
| 905 | + struct property_entry prop[] = { | |
| 906 | + PROPERTY_ENTRY_U32("pasid-num-bits", 0), | |
| 907 | + {}, | |
| 908 | + }; | |
| 909 | + const u32 valid_flags = MOCK_FLAGS_DEVICE_NO_DIRTY | | |
| 910 | + MOCK_FLAGS_DEVICE_HUGE_IOVA | | |
| 911 | + MOCK_FLAGS_DEVICE_PASID; | |
| 842 | 912 | struct mock_dev *mdev; |
| 843 | 913 | int rc, i; |
| 844 | 914 | |
| 845 | − if (dev_flags & | |
| 846 | − ~(MOCK_FLAGS_DEVICE_NO_DIRTY | MOCK_FLAGS_DEVICE_HUGE_IOVA)) | |
| 915 | + if (dev_flags & ~valid_flags) | |
| 847 | 916 | return ERR_PTR(-EINVAL); |
| 848 | 917 | |
| 849 | 918 | mdev = kzalloc(sizeof(*mdev), GFP_KERNEL); |
| 850 | 919 | if (!mdev) |
| 851 | 920 | return ERR_PTR(-ENOMEM); |
| 852 | 921 | |
| 922 | + init_rwsem(&mdev->viommu_rwsem); | |
| 853 | 923 | device_initialize(&mdev->dev); |
| 854 | 924 | mdev->flags = dev_flags; |
| 855 | 925 | mdev->dev.release = mock_dev_release; |
| @@ -866,6 +936,15 @@ static struct mock_dev *mock_dev_create(unsigned long dev_flags) · 10 unmodified lines | ||
| 866 | 936 | if (rc) |
| 867 | 937 | goto err_put; |
| 868 | 938 | |
| 939 | + if (dev_flags & MOCK_FLAGS_DEVICE_PASID) | |
| 940 | + prop[0] = PROPERTY_ENTRY_U32("pasid-num-bits", MOCK_PASID_WIDTH); | |
| 941 | + | |
| 942 | + rc = device_create_managed_software_node(&mdev->dev, prop, NULL); | |
| 943 | + if (rc) { | |
| 944 | + dev_err(&mdev->dev, "add pasid-num-bits property failed, rc: %d", rc); | |
| 945 | + goto err_put; | |
| 946 | + } | |
| 947 | + | |
| 869 | 948 | rc = device_add(&mdev->dev); |
| 870 | 949 | if (rc) |
| 871 | 950 | goto err_put; |
| @@ -921,7 +1000,7 @@ static int iommufd_test_mock_domain(struct iommufd_ucmd *ucmd, · 49 unmodified lines | ||
| 921 | 1000 | } |
| 922 | 1001 | sobj->idev.idev = idev; |
| 923 | 1002 | |
| 924 | − rc = iommufd_device_attach(idev, &pt_id); | |
| 1003 | + rc = iommufd_device_attach(idev, IOMMU_NO_PASID, &pt_id); | |
| 925 | 1004 | if (rc) |
| 926 | 1005 | goto out_unbind; |
| 927 | 1006 | |
| @@ -936,7 +1015,7 @@ static int iommufd_test_mock_domain(struct iommufd_ucmd *ucmd, · 8 unmodified lines | ||
| 936 | 1015 | return 0; |
| 937 | 1016 | |
| 938 | 1017 | out_detach: |
| 939 | − iommufd_device_detach(idev); | |
| 1018 | + iommufd_device_detach(idev, IOMMU_NO_PASID); | |
| 940 | 1019 | out_unbind: |
| 941 | 1020 | iommufd_device_unbind(idev); |
| 942 | 1021 | out_mdev: |
| @@ -946,39 +1025,49 @@ out_sobj: · 3 unmodified lines | ||
| 946 | 1025 | return rc; |
| 947 | 1026 | } |
| 948 | 1027 | |
| 949 | −/* Replace the mock domain with a manually allocated hw_pagetable */ | |
| 950 | −static int iommufd_test_mock_domain_replace(struct iommufd_ucmd *ucmd, | |
| 951 | − unsigned int device_id, u32 pt_id, | |
| 952 | − struct iommu_test_cmd *cmd) | |
| 1028 | +static struct selftest_obj * | |
| 1029 | +iommufd_test_get_selftest_obj(struct iommufd_ctx *ictx, u32 id) | |
| 953 | 1030 | { |
| 954 | 1031 | struct iommufd_object *dev_obj; |
| 955 | 1032 | struct selftest_obj *sobj; |
| 956 | − int rc; | |
| 957 | 1033 | |
| 958 | 1034 | /* |
| 959 | 1035 | * Prefer to use the OBJ_SELFTEST because the destroy_rwsem will ensure |
| 960 | 1036 | * it doesn't race with detach, which is not allowed. |
| 961 | 1037 | */ |
| 962 | − dev_obj = | |
| 963 | − iommufd_get_object(ucmd->ictx, device_id, IOMMUFD_OBJ_SELFTEST); | |
| 1038 | + dev_obj = iommufd_get_object(ictx, id, IOMMUFD_OBJ_SELFTEST); | |
| 964 | 1039 | if (IS_ERR(dev_obj)) |
| 965 | − return PTR_ERR(dev_obj); | |
| 1040 | + return ERR_CAST(dev_obj); | |
| 966 | 1041 | |
| 967 | 1042 | sobj = to_selftest_obj(dev_obj); |
| 968 | 1043 | if (sobj->type != TYPE_IDEV) { |
| 969 | − rc = -EINVAL; | |
| 970 | − goto out_dev_obj; | |
| 1044 | + iommufd_put_object(ictx, dev_obj); | |
| 1045 | + return ERR_PTR(-EINVAL); | |
| 971 | 1046 | } |
| 1047 | + return sobj; | |
| 1048 | +} | |
| 972 | 1049 | |
| 973 | − rc = iommufd_device_replace(sobj->idev.idev, &pt_id); | |
| 1050 | +/* Replace the mock domain with a manually allocated hw_pagetable */ | |
| 1051 | +static int iommufd_test_mock_domain_replace(struct iommufd_ucmd *ucmd, | |
| 1052 | + unsigned int device_id, u32 pt_id, | |
| 1053 | + struct iommu_test_cmd *cmd) | |
| 1054 | +{ | |
| 1055 | + struct selftest_obj *sobj; | |
| 1056 | + int rc; | |
| 1057 | + | |
| 1058 | + sobj = iommufd_test_get_selftest_obj(ucmd->ictx, device_id); | |
| 1059 | + if (IS_ERR(sobj)) | |
| 1060 | + return PTR_ERR(sobj); | |
| 1061 | + | |
| 1062 | + rc = iommufd_device_replace(sobj->idev.idev, IOMMU_NO_PASID, &pt_id); | |
| 974 | 1063 | if (rc) |
| 975 | − goto out_dev_obj; | |
| 1064 | + goto out_sobj; | |
| 976 | 1065 | |
| 977 | 1066 | cmd->mock_domain_replace.pt_id = pt_id; |
| 978 | 1067 | rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); |
| 979 | 1068 | |
| 980 | −out_dev_obj: | |
| 981 | − iommufd_put_object(ucmd->ictx, dev_obj); | |
| 1069 | +out_sobj: | |
| 1070 | + iommufd_put_object(ucmd->ictx, &sobj->obj); | |
| 982 | 1071 | return rc; |
| 983 | 1072 | } |
| 984 | 1073 | |
| @@ -1597,13 +1686,166 @@ static int iommufd_test_trigger_iopf(struct iommufd_ucmd *ucmd, · 612 unmodified lines | ||
| 1597 | 1686 | return 0; |
| 1598 | 1687 | } |
| 1599 | 1688 | |
| 1689 | +static int iommufd_test_trigger_vevent(struct iommufd_ucmd *ucmd, | |
| 1690 | + struct iommu_test_cmd *cmd) | |
| 1691 | +{ | |
| 1692 | + struct iommu_viommu_event_selftest test = {}; | |
| 1693 | + struct iommufd_device *idev; | |
| 1694 | + struct mock_dev *mdev; | |
| 1695 | + int rc = -ENOENT; | |
| 1696 | + | |
| 1697 | + idev = iommufd_get_device(ucmd, cmd->trigger_vevent.dev_id); | |
| 1698 | + if (IS_ERR(idev)) | |
| 1699 | + return PTR_ERR(idev); | |
| 1700 | + mdev = to_mock_dev(idev->dev); | |
| 1701 | + | |
| 1702 | + down_read(&mdev->viommu_rwsem); | |
| 1703 | + if (!mdev->viommu || !mdev->vdev_id) | |
| 1704 | + goto out_unlock; | |
| 1705 | + | |
| 1706 | + test.virt_id = mdev->vdev_id; | |
| 1707 | + rc = iommufd_viommu_report_event(&mdev->viommu->core, | |
| 1708 | + IOMMU_VEVENTQ_TYPE_SELFTEST, &test, | |
| 1709 | + sizeof(test)); | |
| 1710 | +out_unlock: | |
| 1711 | + up_read(&mdev->viommu_rwsem); | |
| 1712 | + iommufd_put_object(ucmd->ictx, &idev->obj); | |
| 1713 | + | |
| 1714 | + return rc; | |
| 1715 | +} | |
| 1716 | + | |
| 1717 | +static inline struct iommufd_hw_pagetable * | |
| 1718 | +iommufd_get_hwpt(struct iommufd_ucmd *ucmd, u32 id) | |
| 1719 | +{ | |
| 1720 | + struct iommufd_object *pt_obj; | |
| 1721 | + | |
| 1722 | + pt_obj = iommufd_get_object(ucmd->ictx, id, IOMMUFD_OBJ_ANY); | |
| 1723 | + if (IS_ERR(pt_obj)) | |
| 1724 | + return ERR_CAST(pt_obj); | |
| 1725 | + | |
| 1726 | + if (pt_obj->type != IOMMUFD_OBJ_HWPT_NESTED && | |
| 1727 | + pt_obj->type != IOMMUFD_OBJ_HWPT_PAGING) { | |
| 1728 | + iommufd_put_object(ucmd->ictx, pt_obj); | |
| 1729 | + return ERR_PTR(-EINVAL); | |
| 1730 | + } | |
| 1731 | + | |
| 1732 | + return container_of(pt_obj, struct iommufd_hw_pagetable, obj); | |
| 1733 | +} | |
| 1734 | + | |
| 1735 | +static int iommufd_test_pasid_check_hwpt(struct iommufd_ucmd *ucmd, | |
| 1736 | + struct iommu_test_cmd *cmd) | |
| 1737 | +{ | |
| 1738 | + u32 hwpt_id = cmd->pasid_check.hwpt_id; | |
| 1739 | + struct iommu_domain *attached_domain; | |
| 1740 | + struct iommu_attach_handle *handle; | |
| 1741 | + struct iommufd_hw_pagetable *hwpt; | |
| 1742 | + struct selftest_obj *sobj; | |
| 1743 | + struct mock_dev *mdev; | |
| 1744 | + int rc = 0; | |
| 1745 | + | |
| 1746 | + sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id); | |
| 1747 | + if (IS_ERR(sobj)) | |
| 1748 | + return PTR_ERR(sobj); | |
| 1749 | + | |
| 1750 | + mdev = sobj->idev.mock_dev; | |
| 1751 | + | |
| 1752 | + handle = iommu_attach_handle_get(mdev->dev.iommu_group, | |
| 1753 | + cmd->pasid_check.pasid, 0); | |
| 1754 | + if (IS_ERR(handle)) | |
| 1755 | + attached_domain = NULL; | |
| 1756 | + else | |
| 1757 | + attached_domain = handle->domain; | |
| 1758 | + | |
| 1759 | + /* hwpt_id == 0 means to check if pasid is detached */ | |
| 1760 | + if (!hwpt_id) { | |
| 1761 | + if (attached_domain) | |
| 1762 | + rc = -EINVAL; | |
| 1763 | + goto out_sobj; | |
| 1764 | + } | |
| 1765 | + | |
| 1766 | + hwpt = iommufd_get_hwpt(ucmd, hwpt_id); | |
| 1767 | + if (IS_ERR(hwpt)) { | |
| 1768 | + rc = PTR_ERR(hwpt); | |
| 1769 | + goto out_sobj; | |
| 1770 | + } | |
| 1771 | + | |
| 1772 | + if (attached_domain != hwpt->domain) | |
| 1773 | + rc = -EINVAL; | |
| 1774 | + | |
| 1775 | + iommufd_put_object(ucmd->ictx, &hwpt->obj); | |
| 1776 | +out_sobj: | |
| 1777 | + iommufd_put_object(ucmd->ictx, &sobj->obj); | |
| 1778 | + return rc; | |
| 1779 | +} | |
| 1780 | + | |
| 1781 | +static int iommufd_test_pasid_attach(struct iommufd_ucmd *ucmd, | |
| 1782 | + struct iommu_test_cmd *cmd) | |
| 1783 | +{ | |
| 1784 | + struct selftest_obj *sobj; | |
| 1785 | + int rc; | |
| 1786 | + | |
| 1787 | + sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id); | |
| 1788 | + if (IS_ERR(sobj)) | |
| 1789 | + return PTR_ERR(sobj); | |
| 1790 | + | |
| 1791 | + rc = iommufd_device_attach(sobj->idev.idev, cmd->pasid_attach.pasid, | |
| 1792 | + &cmd->pasid_attach.pt_id); | |
| 1793 | + if (rc) | |
| 1794 | + goto out_sobj; | |
| 1795 | + | |
| 1796 | + rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); | |
| 1797 | + if (rc) | |
| 1798 | + iommufd_device_detach(sobj->idev.idev, | |
| 1799 | + cmd->pasid_attach.pasid); | |
| 1800 | + | |
| 1801 | +out_sobj: | |
| 1802 | + iommufd_put_object(ucmd->ictx, &sobj->obj); | |
| 1803 | + return rc; | |
| 1804 | +} | |
| 1805 | + | |
| 1806 | +static int iommufd_test_pasid_replace(struct iommufd_ucmd *ucmd, | |
| 1807 | + struct iommu_test_cmd *cmd) | |
| 1808 | +{ | |
| 1809 | + struct selftest_obj *sobj; | |
| 1810 | + int rc; | |
| 1811 | + | |
| 1812 | + sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id); | |
| 1813 | + if (IS_ERR(sobj)) | |
| 1814 | + return PTR_ERR(sobj); | |
| 1815 | + | |
| 1816 | + rc = iommufd_device_replace(sobj->idev.idev, cmd->pasid_attach.pasid, | |
| 1817 | + &cmd->pasid_attach.pt_id); | |
| 1818 | + if (rc) | |
| 1819 | + goto out_sobj; | |
| 1820 | + | |
| 1821 | + rc = iommufd_ucmd_respond(ucmd, sizeof(*cmd)); | |
| 1822 | + | |
| 1823 | +out_sobj: | |
| 1824 | + iommufd_put_object(ucmd->ictx, &sobj->obj); | |
| 1825 | + return rc; | |
| 1826 | +} | |
| 1827 | + | |
| 1828 | +static int iommufd_test_pasid_detach(struct iommufd_ucmd *ucmd, | |
| 1829 | + struct iommu_test_cmd *cmd) | |
| 1830 | +{ | |
| 1831 | + struct selftest_obj *sobj; | |
| 1832 | + | |
| 1833 | + sobj = iommufd_test_get_selftest_obj(ucmd->ictx, cmd->id); | |
| 1834 | + if (IS_ERR(sobj)) | |
| 1835 | + return PTR_ERR(sobj); | |
| 1836 | + | |
| 1837 | + iommufd_device_detach(sobj->idev.idev, cmd->pasid_detach.pasid); | |
| 1838 | + iommufd_put_object(ucmd->ictx, &sobj->obj); | |
| 1839 | + return 0; | |
| 1840 | +} | |
| 1841 | + | |
| 1600 | 1842 | void iommufd_selftest_destroy(struct iommufd_object *obj) |
| 1601 | 1843 | { |
| 1602 | 1844 | struct selftest_obj *sobj = to_selftest_obj(obj); |
| 1603 | 1845 | |
| 1604 | 1846 | switch (sobj->type) { |
| 1605 | 1847 | case TYPE_IDEV: |
| 1606 | − iommufd_device_detach(sobj->idev.idev); | |
| 1848 | + iommufd_device_detach(sobj->idev.idev, IOMMU_NO_PASID); | |
| 1607 | 1849 | iommufd_device_unbind(sobj->idev.idev); |
| 1608 | 1850 | mock_dev_destroy(sobj->idev.mock_dev); |
| 1609 | 1851 | break; |
| @@ -1678,6 +1920,16 @@ int iommufd_test(struct iommufd_ucmd *ucmd) · 68 unmodified lines | ||
| 1678 | 1920 | cmd->dirty.flags); |
| 1679 | 1921 | case IOMMU_TEST_OP_TRIGGER_IOPF: |
| 1680 | 1922 | return iommufd_test_trigger_iopf(ucmd, cmd); |
| 1923 | + case IOMMU_TEST_OP_TRIGGER_VEVENT: | |
| 1924 | + return iommufd_test_trigger_vevent(ucmd, cmd); | |
| 1925 | + case IOMMU_TEST_OP_PASID_ATTACH: | |
| 1926 | + return iommufd_test_pasid_attach(ucmd, cmd); | |
| 1927 | + case IOMMU_TEST_OP_PASID_REPLACE: | |
| 1928 | + return iommufd_test_pasid_replace(ucmd, cmd); | |
| 1929 | + case IOMMU_TEST_OP_PASID_DETACH: | |
| 1930 | + return iommufd_test_pasid_detach(ucmd, cmd); | |
| 1931 | + case IOMMU_TEST_OP_PASID_CHECK_HWPT: | |
| 1932 | + return iommufd_test_pasid_check_hwpt(ucmd, cmd); | |
| 1681 | 1933 | default: |
| 1682 | 1934 | return -EOPNOTSUPP; |
| 1683 | 1935 | } |
| @@ -1724,6 +1976,7 @@ int __init iommufd_test_init(void) · 40 unmodified lines | ||
| 1724 | 1976 | init_completion(&mock_iommu.complete); |
| 1725 | 1977 | |
| 1726 | 1978 | mock_iommu_iopf_queue = iopf_queue_alloc("mock-iopfq"); |
| 1979 | + mock_iommu.iommu_dev.max_pasids = (1 << MOCK_PASID_WIDTH); | |
| 1727 | 1980 | |
| 1728 | 1981 | return 0; |
| 1729 | 1982 | |
| @@ -59,6 +59,8 @@ int iommufd_viommu_alloc_ioctl(struct iommufd_ucmd *ucmd) · 58 unmodified lines | ||
| 59 | 59 | viommu->ictx = ucmd->ictx; |
| 60 | 60 | viommu->hwpt = hwpt_paging; |
| 61 | 61 | refcount_inc(&viommu->hwpt->common.obj.users); |
| 62 | + INIT_LIST_HEAD(&viommu->veventqs); | |
| 63 | + init_rwsem(&viommu->veventqs_rwsem); | |
| 62 | 64 | /* |
| 63 | 65 | * It is the most likely case that a physical IOMMU is unpluggable. A |
| 64 | 66 | * pluggable IOMMU instance (if exists) is responsible for refcounting |
| @@ -538,4 +538,37 @@ int pci_max_pasids(struct pci_dev *pdev) · 537 unmodified lines | ||
| 538 | 538 | return (1 << FIELD_GET(PCI_PASID_CAP_WIDTH, supported)); |
| 539 | 539 | } |
| 540 | 540 | EXPORT_SYMBOL_GPL(pci_max_pasids); |
| 541 | + | |
| 542 | +/** | |
| 543 | + * pci_pasid_status - Check the PASID status | |
| 544 | + * @pdev: PCI device structure | |
| 545 | + * | |
| 546 | + * Returns a negative value when no PASID capability is present. | |
| 547 | + * Otherwise the value of the control register is returned. | |
| 548 | + * Status reported are: | |
| 549 | + * | |
| 550 | + * PCI_PASID_CTRL_ENABLE - PASID enabled | |
| 551 | + * PCI_PASID_CTRL_EXEC - Execute permission enabled | |
| 552 | + * PCI_PASID_CTRL_PRIV - Privileged mode enabled | |
| 553 | + */ | |
| 554 | +int pci_pasid_status(struct pci_dev *pdev) | |
| 555 | +{ | |
| 556 | + int pasid; | |
| 557 | + u16 ctrl; | |
| 558 | + | |
| 559 | + if (pdev->is_virtfn) | |
| 560 | + pdev = pci_physfn(pdev); | |
| 561 | + | |
| 562 | + pasid = pdev->pasid_cap; | |
| 563 | + if (!pasid) | |
| 564 | + return -EINVAL; | |
| 565 | + | |
| 566 | + pci_read_config_word(pdev, pasid + PCI_PASID_CTRL, &ctrl); | |
| 567 | + | |
| 568 | + ctrl &= PCI_PASID_CTRL_ENABLE | PCI_PASID_CTRL_EXEC | | |
| 569 | + PCI_PASID_CTRL_PRIV; | |
| 570 | + | |
| 571 | + return ctrl; | |
| 572 | +} | |
| 573 | +EXPORT_SYMBOL_GPL(pci_pasid_status); | |
| 541 | 574 | #endif /* CONFIG_PCI_PASID */ |
| @@ -162,9 +162,9 @@ void vfio_df_unbind_iommufd(struct vfio_device_file *df) · 161 unmodified lines | ||
| 162 | 162 | int vfio_df_ioctl_attach_pt(struct vfio_device_file *df, |
| 163 | 163 | struct vfio_device_attach_iommufd_pt __user *arg) |
| 164 | 164 | { |
| 165 | − struct vfio_device *device = df->device; | |
| 166 | 165 | struct vfio_device_attach_iommufd_pt attach; |
| 167 | − unsigned long minsz; | |
| 166 | + struct vfio_device *device = df->device; | |
| 167 | + unsigned long minsz, xend = 0; | |
| 168 | 168 | int ret; |
| 169 | 169 | |
| 170 | 170 | minsz = offsetofend(struct vfio_device_attach_iommufd_pt, pt_id); |
| @@ -172,11 +172,34 @@ int vfio_df_ioctl_attach_pt(struct vfio_device_file *df, · 1 unmodified line | ||
| 172 | 172 | if (copy_from_user(&attach, arg, minsz)) |
| 173 | 173 | return -EFAULT; |
| 174 | 174 | |
| 175 | − if (attach.argsz < minsz || attach.flags) | |
| 175 | + if (attach.argsz < minsz) | |
| 176 | 176 | return -EINVAL; |
| 177 | 177 | |
| 178 | + if (attach.flags & ~VFIO_DEVICE_ATTACH_PASID) | |
| 179 | + return -EINVAL; | |
| 180 | + | |
| 181 | + if (attach.flags & VFIO_DEVICE_ATTACH_PASID) { | |
| 182 | + if (!device->ops->pasid_attach_ioas) | |
| 183 | + return -EOPNOTSUPP; | |
| 184 | + xend = offsetofend(struct vfio_device_attach_iommufd_pt, pasid); | |
| 185 | + } | |
| 186 | + | |
| 187 | + if (xend) { | |
| 188 | + if (attach.argsz < xend) | |
| 189 | + return -EINVAL; | |
| 190 | + | |
| 191 | + if (copy_from_user((void *)&attach + minsz, | |
| 192 | + (void __user *)arg + minsz, xend - minsz)) | |
| 193 | + return -EFAULT; | |
| 194 | + } | |
| 195 | + | |
| 178 | 196 | mutex_lock(&device->dev_set->lock); |
| 179 | − ret = device->ops->attach_ioas(device, &attach.pt_id); | |
| 197 | + if (attach.flags & VFIO_DEVICE_ATTACH_PASID) | |
| 198 | + ret = device->ops->pasid_attach_ioas(device, | |
| 199 | + attach.pasid, | |
| 200 | + &attach.pt_id); | |
| 201 | + else | |
| 202 | + ret = device->ops->attach_ioas(device, &attach.pt_id); | |
| 180 | 203 | if (ret) |
| 181 | 204 | goto out_unlock; |
| 182 | 205 | |
| @@ -198,20 +221,41 @@ out_unlock: · 15 unmodified lines | ||
| 198 | 221 | int vfio_df_ioctl_detach_pt(struct vfio_device_file *df, |
| 199 | 222 | struct vfio_device_detach_iommufd_pt __user *arg) |
| 200 | 223 | { |
| 201 | − struct vfio_device *device = df->device; | |
| 202 | 224 | struct vfio_device_detach_iommufd_pt detach; |
| 203 | − unsigned long minsz; | |
| 225 | + struct vfio_device *device = df->device; | |
| 226 | + unsigned long minsz, xend = 0; | |
| 204 | 227 | |
| 205 | 228 | minsz = offsetofend(struct vfio_device_detach_iommufd_pt, flags); |
| 206 | 229 | |
| 207 | 230 | if (copy_from_user(&detach, arg, minsz)) |
| 208 | 231 | return -EFAULT; |
| 209 | 232 | |
| 210 | − if (detach.argsz < minsz || detach.flags) | |
| 233 | + if (detach.argsz < minsz) | |
| 211 | 234 | return -EINVAL; |
| 212 | 235 | |
| 236 | + if (detach.flags & ~VFIO_DEVICE_DETACH_PASID) | |
| 237 | + return -EINVAL; | |
| 238 | + | |
| 239 | + if (detach.flags & VFIO_DEVICE_DETACH_PASID) { | |
| 240 | + if (!device->ops->pasid_detach_ioas) | |
| 241 | + return -EOPNOTSUPP; | |
| 242 | + xend = offsetofend(struct vfio_device_detach_iommufd_pt, pasid); | |
| 243 | + } | |
| 244 | + | |
| 245 | + if (xend) { | |
| 246 | + if (detach.argsz < xend) | |
| 247 | + return -EINVAL; | |
| 248 | + | |
| 249 | + if (copy_from_user((void *)&detach + minsz, | |
| 250 | + (void __user *)arg + minsz, xend - minsz)) | |
| 251 | + return -EFAULT; | |
| 252 | + } | |
| 253 | + | |
| 213 | 254 | mutex_lock(&device->dev_set->lock); |
| 214 | − device->ops->detach_ioas(device); | |
| 255 | + if (detach.flags & VFIO_DEVICE_DETACH_PASID) | |
| 256 | + device->ops->pasid_detach_ioas(device, detach.pasid); | |
| 257 | + else | |
| 258 | + device->ops->detach_ioas(device); | |
| 215 | 259 | mutex_unlock(&device->dev_set->lock); |
| 216 | 260 | |
| 217 | 261 | return 0; |
| @@ -119,16 +119,24 @@ int vfio_iommufd_physical_bind(struct vfio_device *vdev, · 118 unmodified lines | ||
| 119 | 119 | if (IS_ERR(idev)) |
| 120 | 120 | return PTR_ERR(idev); |
| 121 | 121 | vdev->iommufd_device = idev; |
| 122 | + ida_init(&vdev->pasids); | |
| 122 | 123 | return 0; |
| 123 | 124 | } |
| 124 | 125 | EXPORT_SYMBOL_GPL(vfio_iommufd_physical_bind); |
| 125 | 126 | |
| 126 | 127 | void vfio_iommufd_physical_unbind(struct vfio_device *vdev) |
| 127 | 128 | { |
| 129 | + int pasid; | |
| 130 | + | |
| 128 | 131 | lockdep_assert_held(&vdev->dev_set->lock); |
| 129 | 132 | |
| 133 | + while ((pasid = ida_find_first(&vdev->pasids)) >= 0) { | |
| 134 | + iommufd_device_detach(vdev->iommufd_device, pasid); | |
| 135 | + ida_free(&vdev->pasids, pasid); | |
| 136 | + } | |
| 137 | + | |
| 130 | 138 | if (vdev->iommufd_attached) { |
| 131 | − iommufd_device_detach(vdev->iommufd_device); | |
| 139 | + iommufd_device_detach(vdev->iommufd_device, IOMMU_NO_PASID); | |
| 132 | 140 | vdev->iommufd_attached = false; |
| 133 | 141 | } |
| 134 | 142 | iommufd_device_unbind(vdev->iommufd_device); |
| @@ -146,9 +154,11 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id) · 11 unmodified lines | ||
| 146 | 154 | return -EINVAL; |
| 147 | 155 | |
| 148 | 156 | if (vdev->iommufd_attached) |
| 149 | − rc = iommufd_device_replace(vdev->iommufd_device, pt_id); | |
| 157 | + rc = iommufd_device_replace(vdev->iommufd_device, | |
| 158 | + IOMMU_NO_PASID, pt_id); | |
| 150 | 159 | else |
| 151 | − rc = iommufd_device_attach(vdev->iommufd_device, pt_id); | |
| 160 | + rc = iommufd_device_attach(vdev->iommufd_device, | |
| 161 | + IOMMU_NO_PASID, pt_id); | |
| 152 | 162 | if (rc) |
| 153 | 163 | return rc; |
| 154 | 164 | vdev->iommufd_attached = true; |
| @@ -163,11 +173,53 @@ void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev) · 8 unmodified lines | ||
| 163 | 173 | if (WARN_ON(!vdev->iommufd_device) || !vdev->iommufd_attached) |
| 164 | 174 | return; |
| 165 | 175 | |
| 166 | − iommufd_device_detach(vdev->iommufd_device); | |
| 176 | + iommufd_device_detach(vdev->iommufd_device, IOMMU_NO_PASID); | |
| 167 | 177 | vdev->iommufd_attached = false; |
| 168 | 178 | } |
| 169 | 179 | EXPORT_SYMBOL_GPL(vfio_iommufd_physical_detach_ioas); |
| 170 | 180 | |
| 181 | +int vfio_iommufd_physical_pasid_attach_ioas(struct vfio_device *vdev, | |
| 182 | + u32 pasid, u32 *pt_id) | |
| 183 | +{ | |
| 184 | + int rc; | |
| 185 | + | |
| 186 | + lockdep_assert_held(&vdev->dev_set->lock); | |
| 187 | + | |
| 188 | + if (WARN_ON(!vdev->iommufd_device)) | |
| 189 | + return -EINVAL; | |
| 190 | + | |
| 191 | + if (ida_exists(&vdev->pasids, pasid)) | |
| 192 | + return iommufd_device_replace(vdev->iommufd_device, | |
| 193 | + pasid, pt_id); | |
| 194 | + | |
| 195 | + rc = ida_alloc_range(&vdev->pasids, pasid, pasid, GFP_KERNEL); | |
| 196 | + if (rc < 0) | |
| 197 | + return rc; | |
| 198 | + | |
| 199 | + rc = iommufd_device_attach(vdev->iommufd_device, pasid, pt_id); | |
| 200 | + if (rc) | |
| 201 | + ida_free(&vdev->pasids, pasid); | |
| 202 | + | |
| 203 | + return rc; | |
| 204 | +} | |
| 205 | +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_pasid_attach_ioas); | |
| 206 | + | |
| 207 | +void vfio_iommufd_physical_pasid_detach_ioas(struct vfio_device *vdev, | |
| 208 | + u32 pasid) | |
| 209 | +{ | |
| 210 | + lockdep_assert_held(&vdev->dev_set->lock); | |
| 211 | + | |
| 212 | + if (WARN_ON(!vdev->iommufd_device)) | |
| 213 | + return; | |
| 214 | + | |
| 215 | + if (!ida_exists(&vdev->pasids, pasid)) | |
| 216 | + return; | |
| 217 | + | |
| 218 | + iommufd_device_detach(vdev->iommufd_device, pasid); | |
| 219 | + ida_free(&vdev->pasids, pasid); | |
| 220 | +} | |
| 221 | +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_pasid_detach_ioas); | |
| 222 | + | |
| 171 | 223 | /* |
| 172 | 224 | * The emulated standard ops mean that vfio_device is going to use the |
| 173 | 225 | * "mdev path" and will call vfio_pin_pages()/vfio_dma_rw(). Drivers using this |
| @@ -144,6 +144,8 @@ static const struct vfio_device_ops vfio_pci_ops = { · 143 unmodified lines | ||
| 144 | 144 | .unbind_iommufd = vfio_iommufd_physical_unbind, |
| 145 | 145 | .attach_ioas = vfio_iommufd_physical_attach_ioas, |
| 146 | 146 | .detach_ioas = vfio_iommufd_physical_detach_ioas, |
| 147 | + .pasid_attach_ioas = vfio_iommufd_physical_pasid_attach_ioas, | |
| 148 | + .pasid_detach_ioas = vfio_iommufd_physical_pasid_detach_ioas, | |
| 147 | 149 | }; |
| 148 | 150 | |
| 149 | 151 | static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) |
| @@ -274,6 +274,7 @@ struct ida { · 273 unmodified lines | ||
| 274 | 274 | int ida_alloc_range(struct ida *, unsigned int min, unsigned int max, gfp_t); |
| 275 | 275 | void ida_free(struct ida *, unsigned int id); |
| 276 | 276 | void ida_destroy(struct ida *ida); |
| 277 | +int ida_find_first_range(struct ida *ida, unsigned int min, unsigned int max); | |
| 277 | 278 | |
| 278 | 279 | /** |
| 279 | 280 | * ida_alloc() - Allocate an unused ID. |
| @@ -345,4 +346,14 @@ static inline bool ida_is_empty(const struct ida *ida) · 65 unmodified lines | ||
| 345 | 346 | { |
| 346 | 347 | return xa_empty(&ida->xa); |
| 347 | 348 | } |
| 349 | + | |
| 350 | +static inline bool ida_exists(struct ida *ida, unsigned int id) | |
| 351 | +{ | |
| 352 | + return ida_find_first_range(ida, id, id) == id; | |
| 353 | +} | |
| 354 | + | |
| 355 | +static inline int ida_find_first(struct ida *ida) | |
| 356 | +{ | |
| 357 | + return ida_find_first_range(ida, 0, ~0); | |
| 358 | +} | |
| 348 | 359 | #endif /* __IDR_H__ */ |
| @@ -41,6 +41,7 @@ struct iommu_dirty_ops; · 40 unmodified lines | ||
| 41 | 41 | struct notifier_block; |
| 42 | 42 | struct iommu_sva; |
| 43 | 43 | struct iommu_dma_cookie; |
| 44 | +struct iommu_dma_msi_cookie; | |
| 44 | 45 | struct iommu_fault_param; |
| 45 | 46 | struct iommufd_ctx; |
| 46 | 47 | struct iommufd_viommu; |
| @@ -165,6 +166,15 @@ struct iommu_domain_geometry { · 118 unmodified lines | ||
| 165 | 166 | bool force_aperture; /* DMA only allowed in mappable range? */ |
| 166 | 167 | }; |
| 167 | 168 | |
| 169 | +enum iommu_domain_cookie_type { | |
| 170 | + IOMMU_COOKIE_NONE, | |
| 171 | + IOMMU_COOKIE_DMA_IOVA, | |
| 172 | + IOMMU_COOKIE_DMA_MSI, | |
| 173 | + IOMMU_COOKIE_FAULT_HANDLER, | |
| 174 | + IOMMU_COOKIE_SVA, | |
| 175 | + IOMMU_COOKIE_IOMMUFD, | |
| 176 | +}; | |
| 177 | + | |
| 168 | 178 | /* Domain feature flags */ |
| 169 | 179 | #define __IOMMU_DOMAIN_PAGING (1U << 0) /* Support for iommu_map/unmap */ |
| 170 | 180 | #define __IOMMU_DOMAIN_DMA_API (1U << 1) /* Domain for use in DMA-API |
| @@ -211,23 +221,18 @@ struct iommu_domain_geometry { · 40 unmodified lines | ||
| 211 | 221 | |
| 212 | 222 | struct iommu_domain { |
| 213 | 223 | unsigned type; |
| 224 | + enum iommu_domain_cookie_type cookie_type; | |
| 214 | 225 | const struct iommu_domain_ops *ops; |
| 215 | 226 | const struct iommu_dirty_ops *dirty_ops; |
| 216 | 227 | const struct iommu_ops *owner; /* Whose domain_alloc we came from */ |
| 217 | 228 | unsigned long pgsize_bitmap; /* Bitmap of page sizes in use */ |
| 218 | 229 | struct iommu_domain_geometry geometry; |
| 219 | − struct iommu_dma_cookie *iova_cookie; | |
| 220 | 230 | int (*iopf_handler)(struct iopf_group *group); |
| 221 | 231 | |
| 222 | −#if IS_ENABLED(CONFIG_IRQ_MSI_IOMMU) | |
| 223 | − int (*sw_msi)(struct iommu_domain *domain, struct msi_desc *desc, | |
| 224 | − phys_addr_t msi_addr); | |
| 225 | −#endif | |
| 226 | − | |
| 227 | − union { /* Pointer usable by owner of the domain */ | |
| 228 | − struct iommufd_hw_pagetable *iommufd_hwpt; /* iommufd */ | |
| 229 | − }; | |
| 230 | − union { /* Fault handler */ | |
| 232 | + union { /* cookie */ | |
| 233 | + struct iommu_dma_cookie *iova_cookie; | |
| 234 | + struct iommu_dma_msi_cookie *msi_cookie; | |
| 235 | + struct iommufd_hw_pagetable *iommufd_hwpt; | |
| 231 | 236 | struct { |
| 232 | 237 | iommu_fault_handler_t handler; |
| 233 | 238 | void *handler_token; |
| @@ -244,16 +249,6 @@ struct iommu_domain { · 10 unmodified lines | ||
| 244 | 249 | }; |
| 245 | 250 | }; |
| 246 | 251 | |
| 247 | −static inline void iommu_domain_set_sw_msi( | |
| 248 | − struct iommu_domain *domain, | |
| 249 | − int (*sw_msi)(struct iommu_domain *domain, struct msi_desc *desc, | |
| 250 | − phys_addr_t msi_addr)) | |
| 251 | −{ | |
| 252 | −#if IS_ENABLED(CONFIG_IRQ_MSI_IOMMU) | |
| 253 | − domain->sw_msi = sw_msi; | |
| 254 | −#endif | |
| 255 | −} | |
| 256 | − | |
| 257 | 252 | static inline bool iommu_is_dma_domain(struct iommu_domain *domain) |
| 258 | 253 | { |
| 259 | 254 | return domain->type & __IOMMU_DOMAIN_DMA_API; |
| @@ -8,9 +8,11 @@ · 7 unmodified lines | ||
| 8 | 8 | |
| 9 | 9 | #include <linux/err.h> |
| 10 | 10 | #include <linux/errno.h> |
| 11 | +#include <linux/iommu.h> | |
| 11 | 12 | #include <linux/refcount.h> |
| 12 | 13 | #include <linux/types.h> |
| 13 | 14 | #include <linux/xarray.h> |
| 15 | +#include <uapi/linux/iommufd.h> | |
| 14 | 16 | |
| 15 | 17 | struct device; |
| 16 | 18 | struct file; |
| @@ -34,6 +36,7 @@ enum iommufd_object_type { · 17 unmodified lines | ||
| 34 | 36 | IOMMUFD_OBJ_FAULT, |
| 35 | 37 | IOMMUFD_OBJ_VIOMMU, |
| 36 | 38 | IOMMUFD_OBJ_VDEVICE, |
| 39 | + IOMMUFD_OBJ_VEVENTQ, | |
| 37 | 40 | #ifdef CONFIG_IOMMUFD_TEST |
| 38 | 41 | IOMMUFD_OBJ_SELFTEST, |
| 39 | 42 | #endif |
| @@ -52,9 +55,11 @@ struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, · 12 unmodified lines | ||
| 52 | 55 | struct device *dev, u32 *id); |
| 53 | 56 | void iommufd_device_unbind(struct iommufd_device *idev); |
| 54 | 57 | |
| 55 | −int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id); | |
| 56 | −int iommufd_device_replace(struct iommufd_device *idev, u32 *pt_id); | |
| 57 | −void iommufd_device_detach(struct iommufd_device *idev); | |
| 58 | +int iommufd_device_attach(struct iommufd_device *idev, ioasid_t pasid, | |
| 59 | + u32 *pt_id); | |
| 60 | +int iommufd_device_replace(struct iommufd_device *idev, ioasid_t pasid, | |
| 61 | + u32 *pt_id); | |
| 62 | +void iommufd_device_detach(struct iommufd_device *idev, ioasid_t pasid); | |
| 58 | 63 | |
| 59 | 64 | struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev); |
| 60 | 65 | u32 iommufd_device_to_id(struct iommufd_device *idev); |
| @@ -93,6 +98,8 @@ struct iommufd_viommu { · 32 unmodified lines | ||
| 93 | 98 | const struct iommufd_viommu_ops *ops; |
| 94 | 99 | |
| 95 | 100 | struct xarray vdevs; |
| 101 | + struct list_head veventqs; | |
| 102 | + struct rw_semaphore veventqs_rwsem; | |
| 96 | 103 | |
| 97 | 104 | unsigned int type; |
| 98 | 105 | }; |
| @@ -187,6 +194,11 @@ struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx, · 88 unmodified lines | ||
| 187 | 194 | enum iommufd_object_type type); |
| 188 | 195 | struct device *iommufd_viommu_find_dev(struct iommufd_viommu *viommu, |
| 189 | 196 | unsigned long vdev_id); |
| 197 | +int iommufd_viommu_get_vdev_id(struct iommufd_viommu *viommu, | |
| 198 | + struct device *dev, unsigned long *vdev_id); | |
| 199 | +int iommufd_viommu_report_event(struct iommufd_viommu *viommu, | |
| 200 | + enum iommu_veventq_type type, void *event_data, | |
| 201 | + size_t data_len); | |
| 190 | 202 | #else /* !CONFIG_IOMMUFD_DRIVER_CORE */ |
| 191 | 203 | static inline struct iommufd_object * |
| 192 | 204 | _iommufd_object_alloc(struct iommufd_ctx *ictx, size_t size, |
| @@ -200,6 +212,20 @@ iommufd_viommu_find_dev(struct iommufd_viommu *viommu, unsigned long vdev_id) · 7 unmodified lines | ||
| 200 | 212 | { |
| 201 | 213 | return NULL; |
| 202 | 214 | } |
| 215 | + | |
| 216 | +static inline int iommufd_viommu_get_vdev_id(struct iommufd_viommu *viommu, | |
| 217 | + struct device *dev, | |
| 218 | + unsigned long *vdev_id) | |
| 219 | +{ | |
| 220 | + return -ENOENT; | |
| 221 | +} | |
| 222 | + | |
| 223 | +static inline int iommufd_viommu_report_event(struct iommufd_viommu *viommu, | |
| 224 | + enum iommu_veventq_type type, | |
| 225 | + void *event_data, size_t data_len) | |
| 226 | +{ | |
| 227 | + return -EOPNOTSUPP; | |
| 228 | +} | |
| 203 | 229 | #endif /* CONFIG_IOMMUFD_DRIVER_CORE */ |
| 204 | 230 | |
| 205 | 231 | /* |
| @@ -42,6 +42,7 @@ int pci_enable_pasid(struct pci_dev *pdev, int features); · 41 unmodified lines | ||
| 42 | 42 | void pci_disable_pasid(struct pci_dev *pdev); |
| 43 | 43 | int pci_pasid_features(struct pci_dev *pdev); |
| 44 | 44 | int pci_max_pasids(struct pci_dev *pdev); |
| 45 | +int pci_pasid_status(struct pci_dev *pdev); | |
| 45 | 46 | #else /* CONFIG_PCI_PASID */ |
| 46 | 47 | static inline int pci_enable_pasid(struct pci_dev *pdev, int features) |
| 47 | 48 | { return -EINVAL; } |
| @@ -50,6 +51,8 @@ static inline int pci_pasid_features(struct pci_dev *pdev) · 2 unmodified lines | ||
| 50 | 51 | { return -EINVAL; } |
| 51 | 52 | static inline int pci_max_pasids(struct pci_dev *pdev) |
| 52 | 53 | { return -EINVAL; } |
| 54 | +static inline int pci_pasid_status(struct pci_dev *pdev) | |
| 55 | +{ return -EINVAL; } | |
| 53 | 56 | #endif /* CONFIG_PCI_PASID */ |
| 54 | 57 | |
| 55 | 58 | #endif /* LINUX_PCI_ATS_H */ |
| @@ -67,6 +67,7 @@ struct vfio_device { · 66 unmodified lines | ||
| 67 | 67 | struct inode *inode; |
| 68 | 68 | #if IS_ENABLED(CONFIG_IOMMUFD) |
| 69 | 69 | struct iommufd_device *iommufd_device; |
| 70 | + struct ida pasids; | |
| 70 | 71 | u8 iommufd_attached:1; |
| 71 | 72 | #endif |
| 72 | 73 | u8 cdev_opened:1; |
| @@ -91,6 +92,8 @@ struct vfio_device { · 18 unmodified lines | ||
| 91 | 92 | * bound iommufd. Undo in unbind_iommufd if @detach_ioas is not |
| 92 | 93 | * called. |
| 93 | 94 | * @detach_ioas: Opposite of attach_ioas |
| 95 | + * @pasid_attach_ioas: The pasid variation of attach_ioas | |
| 96 | + * @pasid_detach_ioas: Opposite of pasid_attach_ioas | |
| 94 | 97 | * @open_device: Called when the first file descriptor is opened for this device |
| 95 | 98 | * @close_device: Opposite of open_device |
| 96 | 99 | * @read: Perform read(2) on device file descriptor |
| @@ -115,6 +118,9 @@ struct vfio_device_ops { · 18 unmodified lines | ||
| 115 | 118 | void (*unbind_iommufd)(struct vfio_device *vdev); |
| 116 | 119 | int (*attach_ioas)(struct vfio_device *vdev, u32 *pt_id); |
| 117 | 120 | void (*detach_ioas)(struct vfio_device *vdev); |
| 121 | + int (*pasid_attach_ioas)(struct vfio_device *vdev, u32 pasid, | |
| 122 | + u32 *pt_id); | |
| 123 | + void (*pasid_detach_ioas)(struct vfio_device *vdev, u32 pasid); | |
| 118 | 124 | int (*open_device)(struct vfio_device *vdev); |
| 119 | 125 | void (*close_device)(struct vfio_device *vdev); |
| 120 | 126 | ssize_t (*read)(struct vfio_device *vdev, char __user *buf, |
| @@ -139,6 +145,10 @@ int vfio_iommufd_physical_bind(struct vfio_device *vdev, · 18 unmodified lines | ||
| 139 | 145 | void vfio_iommufd_physical_unbind(struct vfio_device *vdev); |
| 140 | 146 | int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id); |
| 141 | 147 | void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev); |
| 148 | +int vfio_iommufd_physical_pasid_attach_ioas(struct vfio_device *vdev, | |
| 149 | + u32 pasid, u32 *pt_id); | |
| 150 | +void vfio_iommufd_physical_pasid_detach_ioas(struct vfio_device *vdev, | |
| 151 | + u32 pasid); | |
| 142 | 152 | int vfio_iommufd_emulated_bind(struct vfio_device *vdev, |
| 143 | 153 | struct iommufd_ctx *ictx, u32 *out_device_id); |
| 144 | 154 | void vfio_iommufd_emulated_unbind(struct vfio_device *vdev); |
| @@ -166,6 +176,10 @@ vfio_iommufd_get_dev_id(struct vfio_device *vdev, struct iommufd_ctx *ictx) · 21 unmodified lines | ||
| 166 | 176 | ((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL) |
| 167 | 177 | #define vfio_iommufd_physical_detach_ioas \ |
| 168 | 178 | ((void (*)(struct vfio_device *vdev)) NULL) |
| 179 | +#define vfio_iommufd_physical_pasid_attach_ioas \ | |
| 180 | + ((int (*)(struct vfio_device *vdev, u32 pasid, u32 *pt_id)) NULL) | |
| 181 | +#define vfio_iommufd_physical_pasid_detach_ioas \ | |
| 182 | + ((void (*)(struct vfio_device *vdev, u32 pasid)) NULL) | |
| 169 | 183 | #define vfio_iommufd_emulated_bind \ |
| 170 | 184 | ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \ |
| 171 | 185 | u32 *out_device_id)) NULL) |
| @@ -55,6 +55,7 @@ enum { · 54 unmodified lines | ||
| 55 | 55 | IOMMUFD_CMD_VIOMMU_ALLOC = 0x90, |
| 56 | 56 | IOMMUFD_CMD_VDEVICE_ALLOC = 0x91, |
| 57 | 57 | IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92, |
| 58 | + IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93, | |
| 58 | 59 | }; |
| 59 | 60 | |
| 60 | 61 | /** |
| @@ -392,6 +393,9 @@ struct iommu_vfio_ioas { · 331 unmodified lines | ||
| 392 | 393 | * Any domain attached to the non-PASID part of the |
| 393 | 394 | * device must also be flagged, otherwise attaching a |
| 394 | 395 | * PASID will blocked. |
| 396 | + * For the user that wants to attach PASID, ioas is | |
| 397 | + * not recommended for both the non-PASID part | |
| 398 | + * and PASID part of the device. | |
| 395 | 399 | * If IOMMU does not support PASID it will return |
| 396 | 400 | * error (-EOPNOTSUPP). |
| 397 | 401 | */ |
| @@ -608,9 +612,17 @@ enum iommu_hw_info_type { · 210 unmodified lines | ||
| 608 | 612 | * IOMMU_HWPT_GET_DIRTY_BITMAP |
| 609 | 613 | * IOMMU_HWPT_SET_DIRTY_TRACKING |
| 610 | 614 | * |
| 615 | + * @IOMMU_HW_CAP_PCI_PASID_EXEC: Execute Permission Supported, user ignores it | |
| 616 | + * when the struct | |
| 617 | + * iommu_hw_info::out_max_pasid_log2 is zero. | |
| 618 | + * @IOMMU_HW_CAP_PCI_PASID_PRIV: Privileged Mode Supported, user ignores it | |
| 619 | + * when the struct | |
| 620 | + * iommu_hw_info::out_max_pasid_log2 is zero. | |
| 611 | 621 | */ |
| 612 | 622 | enum iommufd_hw_capabilities { |
| 613 | 623 | IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0, |
| 624 | + IOMMU_HW_CAP_PCI_PASID_EXEC = 1 << 1, | |
| 625 | + IOMMU_HW_CAP_PCI_PASID_PRIV = 1 << 2, | |
| 614 | 626 | }; |
| 615 | 627 | |
| 616 | 628 | /** |
| @@ -626,6 +638,9 @@ enum iommufd_hw_capabilities { · 9 unmodified lines | ||
| 626 | 638 | * iommu_hw_info_type. |
| 627 | 639 | * @out_capabilities: Output the generic iommu capability info type as defined |
| 628 | 640 | * in the enum iommu_hw_capabilities. |
| 641 | + * @out_max_pasid_log2: Output the width of PASIDs. 0 means no PASID support. | |
| 642 | + * PCI devices turn to out_capabilities to check if the | |
| 643 | + * specific capabilities is supported or not. | |
| 629 | 644 | * @__reserved: Must be 0 |
| 630 | 645 | * |
| 631 | 646 | * Query an iommu type specific hardware information data from an iommu behind |
| @@ -649,7 +664,8 @@ struct iommu_hw_info { · 17 unmodified lines | ||
| 649 | 664 | __u32 data_len; |
| 650 | 665 | __aligned_u64 data_uptr; |
| 651 | 666 | __u32 out_data_type; |
| 652 | − __u32 __reserved; | |
| 667 | + __u8 out_max_pasid_log2; | |
| 668 | + __u8 __reserved[3]; | |
| 653 | 669 | __aligned_u64 out_capabilities; |
| 654 | 670 | }; |
| 655 | 671 | #define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO) |
| @@ -1014,4 +1030,115 @@ struct iommu_ioas_change_process { · 358 unmodified lines | ||
| 1014 | 1030 | #define IOMMU_IOAS_CHANGE_PROCESS \ |
| 1015 | 1031 | _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS) |
| 1016 | 1032 | |
| 1033 | +/** | |
| 1034 | + * enum iommu_veventq_flag - flag for struct iommufd_vevent_header | |
| 1035 | + * @IOMMU_VEVENTQ_FLAG_LOST_EVENTS: vEVENTQ has lost vEVENTs | |
| 1036 | + */ | |
| 1037 | +enum iommu_veventq_flag { | |
| 1038 | + IOMMU_VEVENTQ_FLAG_LOST_EVENTS = (1U << 0), | |
| 1039 | +}; | |
| 1040 | + | |
| 1041 | +/** | |
| 1042 | + * struct iommufd_vevent_header - Virtual Event Header for a vEVENTQ Status | |
| 1043 | + * @flags: Combination of enum iommu_veventq_flag | |
| 1044 | + * @sequence: The sequence index of a vEVENT in the vEVENTQ, with a range of | |
| 1045 | + * [0, INT_MAX] where the following index of INT_MAX is 0 | |
| 1046 | + * | |
| 1047 | + * Each iommufd_vevent_header reports a sequence index of the following vEVENT: | |
| 1048 | + * | |
| 1049 | + * +----------------------+-------+----------------------+-------+---+-------+ | |
| 1050 | + * | header0 {sequence=0} | data0 | header1 {sequence=1} | data1 |...| dataN | | |
| 1051 | + * +----------------------+-------+----------------------+-------+---+-------+ | |
| 1052 | + * | |
| 1053 | + * And this sequence index is expected to be monotonic to the sequence index of | |
| 1054 | + * the previous vEVENT. If two adjacent sequence indexes has a delta larger than | |
| 1055 | + * 1, it means that delta - 1 number of vEVENTs has lost, e.g. two lost vEVENTs: | |
| 1056 | + * | |
| 1057 | + * +-----+----------------------+-------+----------------------+-------+-----+ | |
| 1058 | + * | ... | header3 {sequence=3} | data3 | header6 {sequence=6} | data6 | ... | | |
| 1059 | + * +-----+----------------------+-------+----------------------+-------+-----+ | |
| 1060 | + * | |
| 1061 | + * If a vEVENT lost at the tail of the vEVENTQ and there is no following vEVENT | |
| 1062 | + * providing the next sequence index, an IOMMU_VEVENTQ_FLAG_LOST_EVENTS header | |
| 1063 | + * would be added to the tail, and no data would follow this header: | |
| 1064 | + * | |
| 1065 | + * +--+----------------------+-------+-----------------------------------------+ | |
| 1066 | + * |..| header3 {sequence=3} | data3 | header4 {flags=LOST_EVENTS, sequence=4} | | |
| 1067 | + * +--+----------------------+-------+-----------------------------------------+ | |
| 1068 | + */ | |
| 1069 | +struct iommufd_vevent_header { | |
| 1070 | + __u32 flags; | |
| 1071 | + __u32 sequence; | |
| 1072 | +}; | |
| 1073 | + | |
| 1074 | +/** | |
| 1075 | + * enum iommu_veventq_type - Virtual Event Queue Type | |
| 1076 | + * @IOMMU_VEVENTQ_TYPE_DEFAULT: Reserved for future use | |
| 1077 | + * @IOMMU_VEVENTQ_TYPE_ARM_SMMUV3: ARM SMMUv3 Virtual Event Queue | |
| 1078 | + */ | |
| 1079 | +enum iommu_veventq_type { | |
| 1080 | + IOMMU_VEVENTQ_TYPE_DEFAULT = 0, | |
| 1081 | + IOMMU_VEVENTQ_TYPE_ARM_SMMUV3 = 1, | |
| 1082 | +}; | |
| 1083 | + | |
| 1084 | +/** | |
| 1085 | + * struct iommu_vevent_arm_smmuv3 - ARM SMMUv3 Virtual Event | |
| 1086 | + * (IOMMU_VEVENTQ_TYPE_ARM_SMMUV3) | |
| 1087 | + * @evt: 256-bit ARM SMMUv3 Event record, little-endian. | |
| 1088 | + * Reported event records: (Refer to "7.3 Event records" in SMMUv3 HW Spec) | |
| 1089 | + * - 0x04 C_BAD_STE | |
| 1090 | + * - 0x06 F_STREAM_DISABLED | |
| 1091 | + * - 0x08 C_BAD_SUBSTREAMID | |
| 1092 | + * - 0x0a C_BAD_CD | |
| 1093 | + * - 0x10 F_TRANSLATION | |
| 1094 | + * - 0x11 F_ADDR_SIZE | |
| 1095 | + * - 0x12 F_ACCESS | |
| 1096 | + * - 0x13 F_PERMISSION | |
| 1097 | + * | |
| 1098 | + * StreamID field reports a virtual device ID. To receive a virtual event for a | |
| 1099 | + * device, a vDEVICE must be allocated via IOMMU_VDEVICE_ALLOC. | |
| 1100 | + */ | |
| 1101 | +struct iommu_vevent_arm_smmuv3 { | |
| 1102 | + __aligned_le64 evt[4]; | |
| 1103 | +}; | |
| 1104 | + | |
| 1105 | +/** | |
| 1106 | + * struct iommu_veventq_alloc - ioctl(IOMMU_VEVENTQ_ALLOC) | |
| 1107 | + * @size: sizeof(struct iommu_veventq_alloc) | |
| 1108 | + * @flags: Must be 0 | |
| 1109 | + * @viommu_id: virtual IOMMU ID to associate the vEVENTQ with | |
| 1110 | + * @type: Type of the vEVENTQ. Must be defined in enum iommu_veventq_type | |
| 1111 | + * @veventq_depth: Maximum number of events in the vEVENTQ | |
| 1112 | + * @out_veventq_id: The ID of the new vEVENTQ | |
| 1113 | + * @out_veventq_fd: The fd of the new vEVENTQ. User space must close the | |
| 1114 | + * successfully returned fd after using it | |
| 1115 | + * @__reserved: Must be 0 | |
| 1116 | + * | |
| 1117 | + * Explicitly allocate a virtual event queue interface for a vIOMMU. A vIOMMU | |
| 1118 | + * can have multiple FDs for different types, but is confined to one per @type. | |
| 1119 | + * User space should open the @out_veventq_fd to read vEVENTs out of a vEVENTQ, | |
| 1120 | + * if there are vEVENTs available. A vEVENTQ will lose events due to overflow, | |
| 1121 | + * if the number of the vEVENTs hits @veventq_depth. | |
| 1122 | + * | |
| 1123 | + * Each vEVENT in a vEVENTQ encloses a struct iommufd_vevent_header followed by | |
| 1124 | + * a type-specific data structure, in a normal case: | |
| 1125 | + * | |
| 1126 | + * +-+---------+-------+---------+-------+-----+---------+-------+-+ | |
| 1127 | + * | | header0 | data0 | header1 | data1 | ... | headerN | dataN | | | |
| 1128 | + * +-+---------+-------+---------+-------+-----+---------+-------+-+ | |
| 1129 | + * | |
| 1130 | + * unless a tailing IOMMU_VEVENTQ_FLAG_LOST_EVENTS header is logged (refer to | |
| 1131 | + * struct iommufd_vevent_header). | |
| 1132 | + */ | |
| 1133 | +struct iommu_veventq_alloc { | |
| 1134 | + __u32 size; | |
| 1135 | + __u32 flags; | |
| 1136 | + __u32 viommu_id; | |
| 1137 | + __u32 type; | |
| 1138 | + __u32 veventq_depth; | |
| 1139 | + __u32 out_veventq_id; | |
| 1140 | + __u32 out_veventq_fd; | |
| 1141 | + __u32 __reserved; | |
| 1142 | +}; | |
| 1143 | +#define IOMMU_VEVENTQ_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VEVENTQ_ALLOC) | |
| 1017 | 1144 | #endif |
| @@ -932,29 +932,34 @@ struct vfio_device_bind_iommufd { · 931 unmodified lines | ||
| 932 | 932 | * VFIO_DEVICE_ATTACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 19, |
| 933 | 933 | * struct vfio_device_attach_iommufd_pt) |
| 934 | 934 | * @argsz: User filled size of this data. |
| 935 | − * @flags: Must be 0. | |
| 935 | + * @flags: Flags for attach. | |
| 936 | 936 | * @pt_id: Input the target id which can represent an ioas or a hwpt |
| 937 | 937 | * allocated via iommufd subsystem. |
| 938 | 938 | * Output the input ioas id or the attached hwpt id which could |
| 939 | 939 | * be the specified hwpt itself or a hwpt automatically created |
| 940 | 940 | * for the specified ioas by kernel during the attachment. |
| 941 | + * @pasid: The pasid to be attached, only meaningful when | |
| 942 | + * VFIO_DEVICE_ATTACH_PASID is set in @flags | |
| 941 | 943 | * |
| 942 | 944 | * Associate the device with an address space within the bound iommufd. |
| 943 | 945 | * Undo by VFIO_DEVICE_DETACH_IOMMUFD_PT or device fd close. This is only |
| 944 | 946 | * allowed on cdev fds. |
| 945 | 947 | * |
| 946 | − * If a vfio device is currently attached to a valid hw_pagetable, without doing | |
| 947 | − * a VFIO_DEVICE_DETACH_IOMMUFD_PT, a second VFIO_DEVICE_ATTACH_IOMMUFD_PT ioctl | |
| 948 | − * passing in another hw_pagetable (hwpt) id is allowed. This action, also known | |
| 949 | − * as a hw_pagetable replacement, will replace the device's currently attached | |
| 950 | − * hw_pagetable with a new hw_pagetable corresponding to the given pt_id. | |
| 948 | + * If a vfio device or a pasid of this device is currently attached to a valid | |
| 949 | + * hw_pagetable (hwpt), without doing a VFIO_DEVICE_DETACH_IOMMUFD_PT, a second | |
| 950 | + * VFIO_DEVICE_ATTACH_IOMMUFD_PT ioctl passing in another hwpt id is allowed. | |
| 951 | + * This action, also known as a hw_pagetable replacement, will replace the | |
| 952 | + * currently attached hwpt of the device or the pasid of this device with a new | |
| 953 | + * hwpt corresponding to the given pt_id. | |
| 951 | 954 | * |
| 952 | 955 | * Return: 0 on success, -errno on failure. |
| 953 | 956 | */ |
| 954 | 957 | struct vfio_device_attach_iommufd_pt { |
| 955 | 958 | __u32 argsz; |
| 956 | 959 | __u32 flags; |
| 960 | +#define VFIO_DEVICE_ATTACH_PASID (1 << 0) | |
| 957 | 961 | __u32 pt_id; |
| 962 | + __u32 pasid; | |
| 958 | 963 | }; |
| 959 | 964 | |
| 960 | 965 | #define VFIO_DEVICE_ATTACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 19) |
| @@ -963,17 +968,21 @@ struct vfio_device_attach_iommufd_pt { · 2 unmodified lines | ||
| 963 | 968 | * VFIO_DEVICE_DETACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 20, |
| 964 | 969 | * struct vfio_device_detach_iommufd_pt) |
| 965 | 970 | * @argsz: User filled size of this data. |
| 966 | − * @flags: Must be 0. | |
| 971 | + * @flags: Flags for detach. | |
| 972 | + * @pasid: The pasid to be detached, only meaningful when | |
| 973 | + * VFIO_DEVICE_DETACH_PASID is set in @flags | |
| 967 | 974 | * |
| 968 | − * Remove the association of the device and its current associated address | |
| 969 | − * space. After it, the device should be in a blocking DMA state. This is only | |
| 970 | − * allowed on cdev fds. | |
| 975 | + * Remove the association of the device or a pasid of the device and its current | |
| 976 | + * associated address space. After it, the device or the pasid should be in a | |
| 977 | + * blocking DMA state. This is only allowed on cdev fds. | |
| 971 | 978 | * |
| 972 | 979 | * Return: 0 on success, -errno on failure. |
| 973 | 980 | */ |
| 974 | 981 | struct vfio_device_detach_iommufd_pt { |
| 975 | 982 | __u32 argsz; |
| 976 | 983 | __u32 flags; |
| 984 | +#define VFIO_DEVICE_DETACH_PASID (1 << 0) | |
| 985 | + __u32 pasid; | |
| 977 | 986 | }; |
| 978 | 987 | |
| 979 | 988 | #define VFIO_DEVICE_DETACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 20) |
| @@ -476,6 +476,73 @@ nospc: · 475 unmodified lines | ||
| 476 | 476 | } |
| 477 | 477 | EXPORT_SYMBOL(ida_alloc_range); |
| 478 | 478 | |
| 479 | +/** | |
| 480 | + * ida_find_first_range - Get the lowest used ID. | |
| 481 | + * @ida: IDA handle. | |
| 482 | + * @min: Lowest ID to get. | |
| 483 | + * @max: Highest ID to get. | |
| 484 | + * | |
| 485 | + * Get the lowest used ID between @min and @max, inclusive. The returned | |
| 486 | + * ID will not exceed %INT_MAX, even if @max is larger. | |
| 487 | + * | |
| 488 | + * Context: Any context. Takes and releases the xa_lock. | |
| 489 | + * Return: The lowest used ID, or errno if no used ID is found. | |
| 490 | + */ | |
| 491 | +int ida_find_first_range(struct ida *ida, unsigned int min, unsigned int max) | |
| 492 | +{ | |
| 493 | + unsigned long index = min / IDA_BITMAP_BITS; | |
| 494 | + unsigned int offset = min % IDA_BITMAP_BITS; | |
| 495 | + unsigned long *addr, size, bit; | |
| 496 | + unsigned long tmp = 0; | |
| 497 | + unsigned long flags; | |
| 498 | + void *entry; | |
| 499 | + int ret; | |
| 500 | + | |
| 501 | + if ((int)min < 0) | |
| 502 | + return -EINVAL; | |
| 503 | + if ((int)max < 0) | |
| 504 | + max = INT_MAX; | |
| 505 | + | |
| 506 | + xa_lock_irqsave(&ida->xa, flags); | |
| 507 | + | |
| 508 | + entry = xa_find(&ida->xa, &index, max / IDA_BITMAP_BITS, XA_PRESENT); | |
| 509 | + if (!entry) { | |
| 510 | + ret = -ENOENT; | |
| 511 | + goto err_unlock; | |
| 512 | + } | |
| 513 | + | |
| 514 | + if (index > min / IDA_BITMAP_BITS) | |
| 515 | + offset = 0; | |
| 516 | + if (index * IDA_BITMAP_BITS + offset > max) { | |
| 517 | + ret = -ENOENT; | |
| 518 | + goto err_unlock; | |
| 519 | + } | |
| 520 | + | |
| 521 | + if (xa_is_value(entry)) { | |
| 522 | + tmp = xa_to_value(entry); | |
| 523 | + addr = &tmp; | |
| 524 | + size = BITS_PER_XA_VALUE; | |
| 525 | + } else { | |
| 526 | + addr = ((struct ida_bitmap *)entry)->bitmap; | |
| 527 | + size = IDA_BITMAP_BITS; | |
| 528 | + } | |
| 529 | + | |
| 530 | + bit = find_next_bit(addr, size, offset); | |
| 531 | + | |
| 532 | + xa_unlock_irqrestore(&ida->xa, flags); | |
| 533 | + | |
| 534 | + if (bit == size || | |
| 535 | + index * IDA_BITMAP_BITS + bit > max) | |
| 536 | + return -ENOENT; | |
| 537 | + | |
| 538 | + return index * IDA_BITMAP_BITS + bit; | |
| 539 | + | |
| 540 | +err_unlock: | |
| 541 | + xa_unlock_irqrestore(&ida->xa, flags); | |
| 542 | + return ret; | |
| 543 | +} | |
| 544 | +EXPORT_SYMBOL(ida_find_first_range); | |
| 545 | + | |
| 479 | 546 | /** |
| 480 | 547 | * ida_free() - Release an allocated ID. |
| 481 | 548 | * @ida: IDA handle. |
| @@ -189,6 +189,75 @@ static void ida_check_bad_free(struct ida *ida) · 188 unmodified lines | ||
| 189 | 189 | IDA_BUG_ON(ida, !ida_is_empty(ida)); |
| 190 | 190 | } |
| 191 | 191 | |
| 192 | +/* | |
| 193 | + * Check ida_find_first_range() and varriants. | |
| 194 | + */ | |
| 195 | +static void ida_check_find_first(struct ida *ida) | |
| 196 | +{ | |
| 197 | + /* IDA is empty; all of the below should be not exist */ | |
| 198 | + IDA_BUG_ON(ida, ida_exists(ida, 0)); | |
| 199 | + IDA_BUG_ON(ida, ida_exists(ida, 3)); | |
| 200 | + IDA_BUG_ON(ida, ida_exists(ida, 63)); | |
| 201 | + IDA_BUG_ON(ida, ida_exists(ida, 1023)); | |
| 202 | + IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1)); | |
| 203 | + | |
| 204 | + /* IDA contains a single value entry */ | |
| 205 | + IDA_BUG_ON(ida, ida_alloc_min(ida, 3, GFP_KERNEL) != 3); | |
| 206 | + IDA_BUG_ON(ida, ida_exists(ida, 0)); | |
| 207 | + IDA_BUG_ON(ida, !ida_exists(ida, 3)); | |
| 208 | + IDA_BUG_ON(ida, ida_exists(ida, 63)); | |
| 209 | + IDA_BUG_ON(ida, ida_exists(ida, 1023)); | |
| 210 | + IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1)); | |
| 211 | + | |
| 212 | + IDA_BUG_ON(ida, ida_alloc_min(ida, 63, GFP_KERNEL) != 63); | |
| 213 | + IDA_BUG_ON(ida, ida_exists(ida, 0)); | |
| 214 | + IDA_BUG_ON(ida, !ida_exists(ida, 3)); | |
| 215 | + IDA_BUG_ON(ida, !ida_exists(ida, 63)); | |
| 216 | + IDA_BUG_ON(ida, ida_exists(ida, 1023)); | |
| 217 | + IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1)); | |
| 218 | + | |
| 219 | + /* IDA contains a single bitmap */ | |
| 220 | + IDA_BUG_ON(ida, ida_alloc_min(ida, 1023, GFP_KERNEL) != 1023); | |
| 221 | + IDA_BUG_ON(ida, ida_exists(ida, 0)); | |
| 222 | + IDA_BUG_ON(ida, !ida_exists(ida, 3)); | |
| 223 | + IDA_BUG_ON(ida, !ida_exists(ida, 63)); | |
| 224 | + IDA_BUG_ON(ida, !ida_exists(ida, 1023)); | |
| 225 | + IDA_BUG_ON(ida, ida_exists(ida, (1 << 20) - 1)); | |
| 226 | + | |
| 227 | + /* IDA contains a tree */ | |
| 228 | + IDA_BUG_ON(ida, ida_alloc_min(ida, (1 << 20) - 1, GFP_KERNEL) != (1 << 20) - 1); | |
| 229 | + IDA_BUG_ON(ida, ida_exists(ida, 0)); | |
| 230 | + IDA_BUG_ON(ida, !ida_exists(ida, 3)); | |
| 231 | + IDA_BUG_ON(ida, !ida_exists(ida, 63)); | |
| 232 | + IDA_BUG_ON(ida, !ida_exists(ida, 1023)); | |
| 233 | + IDA_BUG_ON(ida, !ida_exists(ida, (1 << 20) - 1)); | |
| 234 | + | |
| 235 | + /* Now try to find first */ | |
| 236 | + IDA_BUG_ON(ida, ida_find_first(ida) != 3); | |
| 237 | + IDA_BUG_ON(ida, ida_find_first_range(ida, -1, 2) != -EINVAL); | |
| 238 | + IDA_BUG_ON(ida, ida_find_first_range(ida, 0, 2) != -ENOENT); // no used ID | |
| 239 | + IDA_BUG_ON(ida, ida_find_first_range(ida, 0, 3) != 3); | |
| 240 | + IDA_BUG_ON(ida, ida_find_first_range(ida, 1, 3) != 3); | |
| 241 | + IDA_BUG_ON(ida, ida_find_first_range(ida, 3, 3) != 3); | |
| 242 | + IDA_BUG_ON(ida, ida_find_first_range(ida, 2, 4) != 3); | |
| 243 | + IDA_BUG_ON(ida, ida_find_first_range(ida, 4, 3) != -ENOENT); // min > max, fail | |
| 244 | + IDA_BUG_ON(ida, ida_find_first_range(ida, 4, 60) != -ENOENT); // no used ID | |
| 245 | + IDA_BUG_ON(ida, ida_find_first_range(ida, 4, 64) != 63); | |
| 246 | + IDA_BUG_ON(ida, ida_find_first_range(ida, 63, 63) != 63); | |
| 247 | + IDA_BUG_ON(ida, ida_find_first_range(ida, 64, 1026) != 1023); | |
| 248 | + IDA_BUG_ON(ida, ida_find_first_range(ida, 1023, 1023) != 1023); | |
| 249 | + IDA_BUG_ON(ida, ida_find_first_range(ida, 1023, (1 << 20) - 1) != 1023); | |
| 250 | + IDA_BUG_ON(ida, ida_find_first_range(ida, 1024, (1 << 20) - 1) != (1 << 20) - 1); | |
| 251 | + IDA_BUG_ON(ida, ida_find_first_range(ida, (1 << 20), INT_MAX) != -ENOENT); | |
| 252 | + | |
| 253 | + ida_free(ida, 3); | |
| 254 | + ida_free(ida, 63); | |
| 255 | + ida_free(ida, 1023); | |
| 256 | + ida_free(ida, (1 << 20) - 1); | |
| 257 | + | |
| 258 | + IDA_BUG_ON(ida, !ida_is_empty(ida)); | |
| 259 | +} | |
| 260 | + | |
| 192 | 261 | static DEFINE_IDA(ida); |
| 193 | 262 | |
| 194 | 263 | static int ida_checks(void) |
| @@ -202,6 +271,7 @@ static int ida_checks(void) · 7 unmodified lines | ||
| 202 | 271 | ida_check_max(&ida); |
| 203 | 272 | ida_check_conv(&ida); |
| 204 | 273 | ida_check_bad_free(&ida); |
| 274 | + ida_check_find_first(&ida); | |
| 205 | 275 | |
| 206 | 276 | printk("IDA: %u of %u tests passed\n", tests_passed, tests_run); |
| 207 | 277 | return (tests_run != tests_passed) ? 0 : -EINVAL; |
| @@ -342,12 +342,14 @@ FIXTURE(iommufd_ioas) · 341 unmodified lines | ||
| 342 | 342 | uint32_t hwpt_id; |
| 343 | 343 | uint32_t device_id; |
| 344 | 344 | uint64_t base_iova; |
| 345 | + uint32_t device_pasid_id; | |
| 345 | 346 | }; |
| 346 | 347 | |
| 347 | 348 | FIXTURE_VARIANT(iommufd_ioas) |
| 348 | 349 | { |
| 349 | 350 | unsigned int mock_domains; |
| 350 | 351 | unsigned int memory_limit; |
| 352 | + bool pasid_capable; | |
| 351 | 353 | }; |
| 352 | 354 | |
| 353 | 355 | FIXTURE_SETUP(iommufd_ioas) |
| @@ -372,6 +374,12 @@ FIXTURE_SETUP(iommufd_ioas) · 18 unmodified lines | ||
| 372 | 374 | IOMMU_TEST_DEV_CACHE_DEFAULT); |
| 373 | 375 | self->base_iova = MOCK_APERTURE_START; |
| 374 | 376 | } |
| 377 | + | |
| 378 | + if (variant->pasid_capable) | |
| 379 | + test_cmd_mock_domain_flags(self->ioas_id, | |
| 380 | + MOCK_FLAGS_DEVICE_PASID, | |
| 381 | + NULL, NULL, | |
| 382 | + &self->device_pasid_id); | |
| 375 | 383 | } |
| 376 | 384 | |
| 377 | 385 | FIXTURE_TEARDOWN(iommufd_ioas) |
| @@ -387,6 +395,7 @@ FIXTURE_VARIANT_ADD(iommufd_ioas, no_domain) · 9 unmodified lines | ||
| 387 | 395 | FIXTURE_VARIANT_ADD(iommufd_ioas, mock_domain) |
| 388 | 396 | { |
| 389 | 397 | .mock_domains = 1, |
| 398 | + .pasid_capable = true, | |
| 390 | 399 | }; |
| 391 | 400 | |
| 392 | 401 | FIXTURE_VARIANT_ADD(iommufd_ioas, two_mock_domain) |
| @@ -439,6 +448,10 @@ TEST_F(iommufd_ioas, alloc_hwpt_nested) · 46 unmodified lines | ||
| 439 | 448 | &test_hwpt_id); |
| 440 | 449 | test_err_hwpt_alloc(EINVAL, self->device_id, self->device_id, 0, |
| 441 | 450 | &test_hwpt_id); |
| 451 | + test_err_hwpt_alloc(EOPNOTSUPP, self->device_id, self->ioas_id, | |
| 452 | + IOMMU_HWPT_ALLOC_NEST_PARENT | | |
| 453 | + IOMMU_HWPT_FAULT_ID_VALID, | |
| 454 | + &test_hwpt_id); | |
| 442 | 455 | |
| 443 | 456 | test_cmd_hwpt_alloc(self->device_id, self->ioas_id, |
| 444 | 457 | IOMMU_HWPT_ALLOC_NEST_PARENT, |
| @@ -748,6 +761,8 @@ TEST_F(iommufd_ioas, get_hw_info) · 303 unmodified lines | ||
| 748 | 761 | } buffer_smaller; |
| 749 | 762 | |
| 750 | 763 | if (self->device_id) { |
| 764 | + uint8_t max_pasid = 0; | |
| 765 | + | |
| 751 | 766 | /* Provide a zero-size user_buffer */ |
| 752 | 767 | test_cmd_get_hw_info(self->device_id, NULL, 0); |
| 753 | 768 | /* Provide a user_buffer with exact size */ |
| @@ -762,6 +777,13 @@ TEST_F(iommufd_ioas, get_hw_info) · 8 unmodified lines | ||
| 762 | 777 | * the fields within the size range still gets updated. |
| 763 | 778 | */ |
| 764 | 779 | test_cmd_get_hw_info(self->device_id, &buffer_smaller, sizeof(buffer_smaller)); |
| 780 | + test_cmd_get_hw_info_pasid(self->device_id, &max_pasid); | |
| 781 | + ASSERT_EQ(0, max_pasid); | |
| 782 | + if (variant->pasid_capable) { | |
| 783 | + test_cmd_get_hw_info_pasid(self->device_pasid_id, | |
| 784 | + &max_pasid); | |
| 785 | + ASSERT_EQ(MOCK_PASID_WIDTH, max_pasid); | |
| 786 | + } | |
| 765 | 787 | } else { |
| 766 | 788 | test_err_get_hw_info(ENOENT, self->device_id, |
| 767 | 789 | &buffer_exact, sizeof(buffer_exact)); |
| @@ -2736,6 +2758,7 @@ TEST_F(iommufd_viommu, viommu_alloc_nested_iopf) · 1968 unmodified lines | ||
| 2736 | 2758 | uint32_t iopf_hwpt_id; |
| 2737 | 2759 | uint32_t fault_id; |
| 2738 | 2760 | uint32_t fault_fd; |
| 2761 | + uint32_t vdev_id; | |
| 2739 | 2762 | |
| 2740 | 2763 | if (self->device_id) { |
| 2741 | 2764 | test_ioctl_fault_alloc(&fault_id, &fault_fd); |
| @@ -2752,6 +2775,10 @@ TEST_F(iommufd_viommu, viommu_alloc_nested_iopf) · 10 unmodified lines | ||
| 2752 | 2775 | &iopf_hwpt_id, IOMMU_HWPT_DATA_SELFTEST, &data, |
| 2753 | 2776 | sizeof(data)); |
| 2754 | 2777 | |
| 2778 | + /* Must allocate vdevice before attaching to a nested hwpt */ | |
| 2779 | + test_err_mock_domain_replace(ENOENT, self->stdev_id, | |
| 2780 | + iopf_hwpt_id); | |
| 2781 | + test_cmd_vdevice_alloc(viommu_id, dev_id, 0x99, &vdev_id); | |
| 2755 | 2782 | test_cmd_mock_domain_replace(self->stdev_id, iopf_hwpt_id); |
| 2756 | 2783 | EXPECT_ERRNO(EBUSY, |
| 2757 | 2784 | _test_ioctl_destroy(self->fd, iopf_hwpt_id)); |
| @@ -2769,15 +2796,46 @@ TEST_F(iommufd_viommu, vdevice_alloc) · 11 unmodified lines | ||
| 2769 | 2796 | uint32_t viommu_id = self->viommu_id; |
| 2770 | 2797 | uint32_t dev_id = self->device_id; |
| 2771 | 2798 | uint32_t vdev_id = 0; |
| 2799 | + uint32_t veventq_id; | |
| 2800 | + uint32_t veventq_fd; | |
| 2801 | + int prev_seq = -1; | |
| 2772 | 2802 | |
| 2773 | 2803 | if (dev_id) { |
| 2804 | + /* Must allocate vdevice before attaching to a nested hwpt */ | |
| 2805 | + test_err_mock_domain_replace(ENOENT, self->stdev_id, | |
| 2806 | + self->nested_hwpt_id); | |
| 2807 | + | |
| 2808 | + /* Allocate a vEVENTQ with veventq_depth=2 */ | |
| 2809 | + test_cmd_veventq_alloc(viommu_id, IOMMU_VEVENTQ_TYPE_SELFTEST, | |
| 2810 | + &veventq_id, &veventq_fd); | |
| 2811 | + test_err_veventq_alloc(EEXIST, viommu_id, | |
| 2812 | + IOMMU_VEVENTQ_TYPE_SELFTEST, NULL, NULL); | |
| 2774 | 2813 | /* Set vdev_id to 0x99, unset it, and set to 0x88 */ |
| 2775 | 2814 | test_cmd_vdevice_alloc(viommu_id, dev_id, 0x99, &vdev_id); |
| 2815 | + test_cmd_mock_domain_replace(self->stdev_id, | |
| 2816 | + self->nested_hwpt_id); | |
| 2817 | + test_cmd_trigger_vevents(dev_id, 1); | |
| 2818 | + test_cmd_read_vevents(veventq_fd, 1, 0x99, &prev_seq); | |
| 2776 | 2819 | test_err_vdevice_alloc(EEXIST, viommu_id, dev_id, 0x99, |
| 2777 | 2820 | &vdev_id); |
| 2821 | + test_cmd_mock_domain_replace(self->stdev_id, self->ioas_id); | |
| 2778 | 2822 | test_ioctl_destroy(vdev_id); |
| 2823 | + | |
| 2824 | + /* Try again with 0x88 */ | |
| 2779 | 2825 | test_cmd_vdevice_alloc(viommu_id, dev_id, 0x88, &vdev_id); |
| 2826 | + test_cmd_mock_domain_replace(self->stdev_id, | |
| 2827 | + self->nested_hwpt_id); | |
| 2828 | + /* Trigger an overflow with three events */ | |
| 2829 | + test_cmd_trigger_vevents(dev_id, 3); | |
| 2830 | + test_err_read_vevents(EOVERFLOW, veventq_fd, 3, 0x88, | |
| 2831 | + &prev_seq); | |
| 2832 | + /* Overflow must be gone after the previous reads */ | |
| 2833 | + test_cmd_trigger_vevents(dev_id, 1); | |
| 2834 | + test_cmd_read_vevents(veventq_fd, 1, 0x88, &prev_seq); | |
| 2835 | + close(veventq_fd); | |
| 2836 | + test_cmd_mock_domain_replace(self->stdev_id, self->ioas_id); | |
| 2780 | 2837 | test_ioctl_destroy(vdev_id); |
| 2838 | + test_ioctl_destroy(veventq_id); | |
| 2781 | 2839 | } else { |
| 2782 | 2840 | test_err_vdevice_alloc(ENOENT, viommu_id, dev_id, 0x99, NULL); |
| 2783 | 2841 | } |
| @@ -2956,4 +3014,311 @@ TEST_F(iommufd_viommu, vdevice_cache) · 172 unmodified lines | ||
| 2956 | 3014 | } |
| 2957 | 3015 | } |
| 2958 | 3016 | |
| 3017 | +FIXTURE(iommufd_device_pasid) | |
| 3018 | +{ | |
| 3019 | + int fd; | |
| 3020 | + uint32_t ioas_id; | |
| 3021 | + uint32_t hwpt_id; | |
| 3022 | + uint32_t stdev_id; | |
| 3023 | + uint32_t device_id; | |
| 3024 | + uint32_t no_pasid_stdev_id; | |
| 3025 | + uint32_t no_pasid_device_id; | |
| 3026 | +}; | |
| 3027 | + | |
| 3028 | +FIXTURE_VARIANT(iommufd_device_pasid) | |
| 3029 | +{ | |
| 3030 | + bool pasid_capable; | |
| 3031 | +}; | |
| 3032 | + | |
| 3033 | +FIXTURE_SETUP(iommufd_device_pasid) | |
| 3034 | +{ | |
| 3035 | + self->fd = open("/dev/iommu", O_RDWR); | |
| 3036 | + ASSERT_NE(-1, self->fd); | |
| 3037 | + test_ioctl_ioas_alloc(&self->ioas_id); | |
| 3038 | + | |
| 3039 | + test_cmd_mock_domain_flags(self->ioas_id, | |
| 3040 | + MOCK_FLAGS_DEVICE_PASID, | |
| 3041 | + &self->stdev_id, &self->hwpt_id, | |
| 3042 | + &self->device_id); | |
| 3043 | + if (!variant->pasid_capable) | |
| 3044 | + test_cmd_mock_domain_flags(self->ioas_id, 0, | |
| 3045 | + &self->no_pasid_stdev_id, NULL, | |
| 3046 | + &self->no_pasid_device_id); | |
| 3047 | +} | |
| 3048 | + | |
| 3049 | +FIXTURE_TEARDOWN(iommufd_device_pasid) | |
| 3050 | +{ | |
| 3051 | + teardown_iommufd(self->fd, _metadata); | |
| 3052 | +} | |
| 3053 | + | |
| 3054 | +FIXTURE_VARIANT_ADD(iommufd_device_pasid, no_pasid) | |
| 3055 | +{ | |
| 3056 | + .pasid_capable = false, | |
| 3057 | +}; | |
| 3058 | + | |
| 3059 | +FIXTURE_VARIANT_ADD(iommufd_device_pasid, has_pasid) | |
| 3060 | +{ | |
| 3061 | + .pasid_capable = true, | |
| 3062 | +}; | |
| 3063 | + | |
| 3064 | +TEST_F(iommufd_device_pasid, pasid_attach) | |
| 3065 | +{ | |
| 3066 | + struct iommu_hwpt_selftest data = { | |
| 3067 | + .iotlb = IOMMU_TEST_IOTLB_DEFAULT, | |
| 3068 | + }; | |
| 3069 | + uint32_t nested_hwpt_id[3] = {}; | |
| 3070 | + uint32_t parent_hwpt_id = 0; | |
| 3071 | + uint32_t fault_id, fault_fd; | |
| 3072 | + uint32_t s2_hwpt_id = 0; | |
| 3073 | + uint32_t iopf_hwpt_id; | |
| 3074 | + uint32_t pasid = 100; | |
| 3075 | + uint32_t viommu_id; | |
| 3076 | + | |
| 3077 | + /* | |
| 3078 | + * Negative, detach pasid without attaching, this is not expected. | |
| 3079 | + * But it should not result in failure anyway. | |
| 3080 | + */ | |
| 3081 | + test_cmd_pasid_detach(pasid); | |
| 3082 | + | |
| 3083 | + /* Allocate two nested hwpts sharing one common parent hwpt */ | |
| 3084 | + test_cmd_hwpt_alloc(self->device_id, self->ioas_id, | |
| 3085 | + IOMMU_HWPT_ALLOC_NEST_PARENT, | |
| 3086 | + &parent_hwpt_id); | |
| 3087 | + test_cmd_hwpt_alloc_nested(self->device_id, parent_hwpt_id, | |
| 3088 | + IOMMU_HWPT_ALLOC_PASID, | |
| 3089 | + &nested_hwpt_id[0], | |
| 3090 | + IOMMU_HWPT_DATA_SELFTEST, | |
| 3091 | + &data, sizeof(data)); | |
| 3092 | + test_cmd_hwpt_alloc_nested(self->device_id, parent_hwpt_id, | |
| 3093 | + IOMMU_HWPT_ALLOC_PASID, | |
| 3094 | + &nested_hwpt_id[1], | |
| 3095 | + IOMMU_HWPT_DATA_SELFTEST, | |
| 3096 | + &data, sizeof(data)); | |
| 3097 | + | |
| 3098 | + /* Fault related preparation */ | |
| 3099 | + test_ioctl_fault_alloc(&fault_id, &fault_fd); | |
| 3100 | + test_cmd_hwpt_alloc_iopf(self->device_id, parent_hwpt_id, fault_id, | |
| 3101 | + IOMMU_HWPT_FAULT_ID_VALID | IOMMU_HWPT_ALLOC_PASID, | |
| 3102 | + &iopf_hwpt_id, | |
| 3103 | + IOMMU_HWPT_DATA_SELFTEST, &data, | |
| 3104 | + sizeof(data)); | |
| 3105 | + | |
| 3106 | + /* Allocate a regular nested hwpt based on viommu */ | |
| 3107 | + test_cmd_viommu_alloc(self->device_id, parent_hwpt_id, | |
| 3108 | + IOMMU_VIOMMU_TYPE_SELFTEST, | |
| 3109 | + &viommu_id); | |
| 3110 | + test_cmd_hwpt_alloc_nested(self->device_id, viommu_id, | |
| 3111 | + IOMMU_HWPT_ALLOC_PASID, | |
| 3112 | + &nested_hwpt_id[2], | |
| 3113 | + IOMMU_HWPT_DATA_SELFTEST, &data, | |
| 3114 | + sizeof(data)); | |
| 3115 | + | |
| 3116 | + test_cmd_hwpt_alloc(self->device_id, self->ioas_id, | |
| 3117 | + IOMMU_HWPT_ALLOC_PASID, | |
| 3118 | + &s2_hwpt_id); | |
| 3119 | + | |
| 3120 | + /* Attach RID to non-pasid compat domain, */ | |
| 3121 | + test_cmd_mock_domain_replace(self->stdev_id, parent_hwpt_id); | |
| 3122 | + /* then attach to pasid should fail */ | |
| 3123 | + test_err_pasid_attach(EINVAL, pasid, s2_hwpt_id); | |
| 3124 | + | |
| 3125 | + /* Attach RID to pasid compat domain, */ | |
| 3126 | + test_cmd_mock_domain_replace(self->stdev_id, s2_hwpt_id); | |
| 3127 | + /* then attach to pasid should succeed, */ | |
| 3128 | + test_cmd_pasid_attach(pasid, nested_hwpt_id[0]); | |
| 3129 | + /* but attach RID to non-pasid compat domain should fail now. */ | |
| 3130 | + test_err_mock_domain_replace(EINVAL, self->stdev_id, parent_hwpt_id); | |
| 3131 | + /* | |
| 3132 | + * Detach hwpt from pasid 100, and check if the pasid 100 | |
| 3133 | + * has null domain. | |
| 3134 | + */ | |
| 3135 | + test_cmd_pasid_detach(pasid); | |
| 3136 | + ASSERT_EQ(0, | |
| 3137 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3138 | + pasid, 0)); | |
| 3139 | + /* RID is attached to pasid-comapt domain, pasid path is not used */ | |
| 3140 | + | |
| 3141 | + if (!variant->pasid_capable) { | |
| 3142 | + /* | |
| 3143 | + * PASID-compatible domain can be used by non-PASID-capable | |
| 3144 | + * device. | |
| 3145 | + */ | |
| 3146 | + test_cmd_mock_domain_replace(self->no_pasid_stdev_id, nested_hwpt_id[0]); | |
| 3147 | + test_cmd_mock_domain_replace(self->no_pasid_stdev_id, self->ioas_id); | |
| 3148 | + /* | |
| 3149 | + * Attach hwpt to pasid 100 of non-PASID-capable device, | |
| 3150 | + * should fail, no matter domain is pasid-comapt or not. | |
| 3151 | + */ | |
| 3152 | + EXPECT_ERRNO(EINVAL, | |
| 3153 | + _test_cmd_pasid_attach(self->fd, self->no_pasid_stdev_id, | |
| 3154 | + pasid, parent_hwpt_id)); | |
| 3155 | + EXPECT_ERRNO(EINVAL, | |
| 3156 | + _test_cmd_pasid_attach(self->fd, self->no_pasid_stdev_id, | |
| 3157 | + pasid, s2_hwpt_id)); | |
| 3158 | + } | |
| 3159 | + | |
| 3160 | + /* | |
| 3161 | + * Attach non pasid compat hwpt to pasid-capable device, should | |
| 3162 | + * fail, and have null domain. | |
| 3163 | + */ | |
| 3164 | + test_err_pasid_attach(EINVAL, pasid, parent_hwpt_id); | |
| 3165 | + ASSERT_EQ(0, | |
| 3166 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3167 | + pasid, 0)); | |
| 3168 | + | |
| 3169 | + /* | |
| 3170 | + * Attach ioas to pasid 100, should fail, domain should | |
| 3171 | + * be null. | |
| 3172 | + */ | |
| 3173 | + test_err_pasid_attach(EINVAL, pasid, self->ioas_id); | |
| 3174 | + ASSERT_EQ(0, | |
| 3175 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3176 | + pasid, 0)); | |
| 3177 | + | |
| 3178 | + /* | |
| 3179 | + * Attach the s2_hwpt to pasid 100, should succeed, domain should | |
| 3180 | + * be valid. | |
| 3181 | + */ | |
| 3182 | + test_cmd_pasid_attach(pasid, s2_hwpt_id); | |
| 3183 | + ASSERT_EQ(0, | |
| 3184 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3185 | + pasid, s2_hwpt_id)); | |
| 3186 | + | |
| 3187 | + /* | |
| 3188 | + * Try attach pasid 100 with another hwpt, should FAIL | |
| 3189 | + * as attach does not allow overwrite, use REPLACE instead. | |
| 3190 | + */ | |
| 3191 | + test_err_pasid_attach(EBUSY, pasid, nested_hwpt_id[0]); | |
| 3192 | + | |
| 3193 | + /* | |
| 3194 | + * Detach hwpt from pasid 100 for next test, should succeed, | |
| 3195 | + * and have null domain. | |
| 3196 | + */ | |
| 3197 | + test_cmd_pasid_detach(pasid); | |
| 3198 | + ASSERT_EQ(0, | |
| 3199 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3200 | + pasid, 0)); | |
| 3201 | + | |
| 3202 | + /* | |
| 3203 | + * Attach nested hwpt to pasid 100, should succeed, domain | |
| 3204 | + * should be valid. | |
| 3205 | + */ | |
| 3206 | + test_cmd_pasid_attach(pasid, nested_hwpt_id[0]); | |
| 3207 | + ASSERT_EQ(0, | |
| 3208 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3209 | + pasid, nested_hwpt_id[0])); | |
| 3210 | + | |
| 3211 | + /* Attach to pasid 100 which has been attached, should fail. */ | |
| 3212 | + test_err_pasid_attach(EBUSY, pasid, nested_hwpt_id[0]); | |
| 3213 | + | |
| 3214 | + /* cleanup pasid 100 */ | |
| 3215 | + test_cmd_pasid_detach(pasid); | |
| 3216 | + | |
| 3217 | + /* Replace tests */ | |
| 3218 | + | |
| 3219 | + pasid = 200; | |
| 3220 | + /* | |
| 3221 | + * Replace pasid 200 without attaching it, should fail | |
| 3222 | + * with -EINVAL. | |
| 3223 | + */ | |
| 3224 | + test_err_pasid_replace(EINVAL, pasid, s2_hwpt_id); | |
| 3225 | + | |
| 3226 | + /* | |
| 3227 | + * Attach the s2 hwpt to pasid 200, should succeed, domain should | |
| 3228 | + * be valid. | |
| 3229 | + */ | |
| 3230 | + test_cmd_pasid_attach(pasid, s2_hwpt_id); | |
| 3231 | + ASSERT_EQ(0, | |
| 3232 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3233 | + pasid, s2_hwpt_id)); | |
| 3234 | + | |
| 3235 | + /* | |
| 3236 | + * Replace pasid 200 with self->ioas_id, should fail | |
| 3237 | + * and domain should be the prior s2 hwpt. | |
| 3238 | + */ | |
| 3239 | + test_err_pasid_replace(EINVAL, pasid, self->ioas_id); | |
| 3240 | + ASSERT_EQ(0, | |
| 3241 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3242 | + pasid, s2_hwpt_id)); | |
| 3243 | + | |
| 3244 | + /* | |
| 3245 | + * Replace a nested hwpt for pasid 200, should succeed, | |
| 3246 | + * and have valid domain. | |
| 3247 | + */ | |
| 3248 | + test_cmd_pasid_replace(pasid, nested_hwpt_id[0]); | |
| 3249 | + ASSERT_EQ(0, | |
| 3250 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3251 | + pasid, nested_hwpt_id[0])); | |
| 3252 | + | |
| 3253 | + /* | |
| 3254 | + * Replace with another nested hwpt for pasid 200, should | |
| 3255 | + * succeed, and have valid domain. | |
| 3256 | + */ | |
| 3257 | + test_cmd_pasid_replace(pasid, nested_hwpt_id[1]); | |
| 3258 | + ASSERT_EQ(0, | |
| 3259 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3260 | + pasid, nested_hwpt_id[1])); | |
| 3261 | + | |
| 3262 | + /* cleanup pasid 200 */ | |
| 3263 | + test_cmd_pasid_detach(pasid); | |
| 3264 | + | |
| 3265 | + /* Negative Tests for pasid replace, use pasid 1024 */ | |
| 3266 | + | |
| 3267 | + /* | |
| 3268 | + * Attach the s2 hwpt to pasid 1024, should succeed, domain should | |
| 3269 | + * be valid. | |
| 3270 | + */ | |
| 3271 | + pasid = 1024; | |
| 3272 | + test_cmd_pasid_attach(pasid, s2_hwpt_id); | |
| 3273 | + ASSERT_EQ(0, | |
| 3274 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3275 | + pasid, s2_hwpt_id)); | |
| 3276 | + | |
| 3277 | + /* | |
| 3278 | + * Replace pasid 1024 with nested_hwpt_id[0], should fail, | |
| 3279 | + * but have the old valid domain. This is a designed | |
| 3280 | + * negative case. Normally, this shall succeed. | |
| 3281 | + */ | |
| 3282 | + test_err_pasid_replace(ENOMEM, pasid, nested_hwpt_id[0]); | |
| 3283 | + ASSERT_EQ(0, | |
| 3284 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3285 | + pasid, s2_hwpt_id)); | |
| 3286 | + | |
| 3287 | + /* cleanup pasid 1024 */ | |
| 3288 | + test_cmd_pasid_detach(pasid); | |
| 3289 | + | |
| 3290 | + /* Attach to iopf-capable hwpt */ | |
| 3291 | + | |
| 3292 | + /* | |
| 3293 | + * Attach an iopf hwpt to pasid 2048, should succeed, domain should | |
| 3294 | + * be valid. | |
| 3295 | + */ | |
| 3296 | + pasid = 2048; | |
| 3297 | + test_cmd_pasid_attach(pasid, iopf_hwpt_id); | |
| 3298 | + ASSERT_EQ(0, | |
| 3299 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3300 | + pasid, iopf_hwpt_id)); | |
| 3301 | + | |
| 3302 | + test_cmd_trigger_iopf_pasid(self->device_id, pasid, fault_fd); | |
| 3303 | + | |
| 3304 | + /* | |
| 3305 | + * Replace with s2_hwpt_id for pasid 2048, should | |
| 3306 | + * succeed, and have valid domain. | |
| 3307 | + */ | |
| 3308 | + test_cmd_pasid_replace(pasid, s2_hwpt_id); | |
| 3309 | + ASSERT_EQ(0, | |
| 3310 | + test_cmd_pasid_check_hwpt(self->fd, self->stdev_id, | |
| 3311 | + pasid, s2_hwpt_id)); | |
| 3312 | + | |
| 3313 | + /* cleanup pasid 2048 */ | |
| 3314 | + test_cmd_pasid_detach(pasid); | |
| 3315 | + | |
| 3316 | + test_ioctl_destroy(iopf_hwpt_id); | |
| 3317 | + close(fault_fd); | |
| 3318 | + test_ioctl_destroy(fault_id); | |
| 3319 | + | |
| 3320 | + /* Detach the s2_hwpt_id from RID */ | |
| 3321 | + test_cmd_mock_domain_replace(self->stdev_id, self->ioas_id); | |
| 3322 | +} | |
| 3323 | + | |
| 2959 | 3324 | TEST_HARNESS_MAIN |
| @@ -209,12 +209,16 @@ FIXTURE(basic_fail_nth) · 208 unmodified lines | ||
| 209 | 209 | { |
| 210 | 210 | int fd; |
| 211 | 211 | uint32_t access_id; |
| 212 | + uint32_t stdev_id; | |
| 213 | + uint32_t pasid; | |
| 212 | 214 | }; |
| 213 | 215 | |
| 214 | 216 | FIXTURE_SETUP(basic_fail_nth) |
| 215 | 217 | { |
| 216 | 218 | self->fd = -1; |
| 217 | 219 | self->access_id = 0; |
| 220 | + self->stdev_id = 0; | |
| 221 | + self->pasid = 0; //test should use a non-zero value | |
| 218 | 222 | } |
| 219 | 223 | |
| 220 | 224 | FIXTURE_TEARDOWN(basic_fail_nth) |
| @@ -226,6 +230,8 @@ FIXTURE_TEARDOWN(basic_fail_nth) · 5 unmodified lines | ||
| 226 | 230 | rc = _test_cmd_destroy_access(self->access_id); |
| 227 | 231 | assert(rc == 0); |
| 228 | 232 | } |
| 233 | + if (self->pasid && self->stdev_id) | |
| 234 | + _test_cmd_pasid_detach(self->fd, self->stdev_id, self->pasid); | |
| 229 | 235 | teardown_iommufd(self->fd, _metadata); |
| 230 | 236 | } |
| 231 | 237 | |
| @@ -620,10 +626,11 @@ TEST_FAIL_NTH(basic_fail_nth, device) · 388 unmodified lines | ||
| 620 | 626 | }; |
| 621 | 627 | struct iommu_test_hw_info info; |
| 622 | 628 | uint32_t fault_id, fault_fd; |
| 629 | + uint32_t veventq_id, veventq_fd; | |
| 623 | 630 | uint32_t fault_hwpt_id; |
| 631 | + uint32_t test_hwpt_id; | |
| 624 | 632 | uint32_t ioas_id; |
| 625 | 633 | uint32_t ioas_id2; |
| 626 | − uint32_t stdev_id; | |
| 627 | 634 | uint32_t idev_id; |
| 628 | 635 | uint32_t hwpt_id; |
| 629 | 636 | uint32_t viommu_id; |
| @@ -654,25 +661,30 @@ TEST_FAIL_NTH(basic_fail_nth, device) · 24 unmodified lines | ||
| 654 | 661 | |
| 655 | 662 | fail_nth_enable(); |
| 656 | 663 | |
| 657 | − if (_test_cmd_mock_domain(self->fd, ioas_id, &stdev_id, NULL, | |
| 658 | − &idev_id)) | |
| 664 | + if (_test_cmd_mock_domain_flags(self->fd, ioas_id, | |
| 665 | + MOCK_FLAGS_DEVICE_PASID, | |
| 666 | + &self->stdev_id, NULL, &idev_id)) | |
| 659 | 667 | return -1; |
| 660 | 668 | |
| 661 | − if (_test_cmd_get_hw_info(self->fd, idev_id, &info, sizeof(info), NULL)) | |
| 669 | + if (_test_cmd_get_hw_info(self->fd, idev_id, &info, | |
| 670 | + sizeof(info), NULL, NULL)) | |
| 662 | 671 | return -1; |
| 663 | 672 | |
| 664 | − if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0, 0, &hwpt_id, | |
| 673 | + if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0, | |
| 674 | + IOMMU_HWPT_ALLOC_PASID, &hwpt_id, | |
| 665 | 675 | IOMMU_HWPT_DATA_NONE, 0, 0)) |
| 666 | 676 | return -1; |
| 667 | 677 | |
| 668 | − if (_test_cmd_mock_domain_replace(self->fd, stdev_id, ioas_id2, NULL)) | |
| 678 | + if (_test_cmd_mock_domain_replace(self->fd, self->stdev_id, ioas_id2, NULL)) | |
| 669 | 679 | return -1; |
| 670 | 680 | |
| 671 | − if (_test_cmd_mock_domain_replace(self->fd, stdev_id, hwpt_id, NULL)) | |
| 681 | + if (_test_cmd_mock_domain_replace(self->fd, self->stdev_id, hwpt_id, NULL)) | |
| 672 | 682 | return -1; |
| 673 | 683 | |
| 674 | 684 | if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0, |
| 675 | − IOMMU_HWPT_ALLOC_NEST_PARENT, &hwpt_id, | |
| 685 | + IOMMU_HWPT_ALLOC_NEST_PARENT | | |
| 686 | + IOMMU_HWPT_ALLOC_PASID, | |
| 687 | + &hwpt_id, | |
| 676 | 688 | IOMMU_HWPT_DATA_NONE, 0, 0)) |
| 677 | 689 | return -1; |
| 678 | 690 | |
| @@ -692,6 +704,37 @@ TEST_FAIL_NTH(basic_fail_nth, device) · 13 unmodified lines | ||
| 692 | 704 | IOMMU_HWPT_DATA_SELFTEST, &data, sizeof(data))) |
| 693 | 705 | return -1; |
| 694 | 706 | |
| 707 | + if (_test_cmd_veventq_alloc(self->fd, viommu_id, | |
| 708 | + IOMMU_VEVENTQ_TYPE_SELFTEST, &veventq_id, | |
| 709 | + &veventq_fd)) | |
| 710 | + return -1; | |
| 711 | + close(veventq_fd); | |
| 712 | + | |
| 713 | + if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0, | |
| 714 | + IOMMU_HWPT_ALLOC_PASID, | |
| 715 | + &test_hwpt_id, | |
| 716 | + IOMMU_HWPT_DATA_NONE, 0, 0)) | |
| 717 | + return -1; | |
| 718 | + | |
| 719 | + /* Tests for pasid attach/replace/detach */ | |
| 720 | + | |
| 721 | + self->pasid = 200; | |
| 722 | + | |
| 723 | + if (_test_cmd_pasid_attach(self->fd, self->stdev_id, | |
| 724 | + self->pasid, hwpt_id)) { | |
| 725 | + self->pasid = 0; | |
| 726 | + return -1; | |
| 727 | + } | |
| 728 | + | |
| 729 | + if (_test_cmd_pasid_replace(self->fd, self->stdev_id, | |
| 730 | + self->pasid, test_hwpt_id)) | |
| 731 | + return -1; | |
| 732 | + | |
| 733 | + if (_test_cmd_pasid_detach(self->fd, self->stdev_id, self->pasid)) | |
| 734 | + return -1; | |
| 735 | + | |
| 736 | + self->pasid = 0; | |
| 737 | + | |
| 695 | 738 | return 0; |
| 696 | 739 | } |
| 697 | 740 | |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <sys/ioctl.h> |
| 10 | 10 | #include <stdint.h> |
| 11 | 11 | #include <assert.h> |
| 12 | +#include <poll.h> | |
| 12 | 13 | |
| 13 | 14 | #include "../kselftest_harness.h" |
| 14 | 15 | #include "../../../../drivers/iommu/iommufd/iommufd_test.h" |
| @@ -757,7 +758,8 @@ static void teardown_iommufd(int fd, struct __test_metadata *_metadata) · 742 unmodified lines | ||
| 757 | 758 | |
| 758 | 759 | /* @data can be NULL */ |
| 759 | 760 | static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data, |
| 760 | − size_t data_len, uint32_t *capabilities) | |
| 761 | + size_t data_len, uint32_t *capabilities, | |
| 762 | + uint8_t *max_pasid) | |
| 761 | 763 | { |
| 762 | 764 | struct iommu_test_hw_info *info = (struct iommu_test_hw_info *)data; |
| 763 | 765 | struct iommu_hw_info cmd = { |
| @@ -802,6 +804,9 @@ static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data, · 38 unmodified lines | ||
| 802 | 804 | assert(!info->flags); |
| 803 | 805 | } |
| 804 | 806 | |
| 807 | + if (max_pasid) | |
| 808 | + *max_pasid = cmd.out_max_pasid_log2; | |
| 809 | + | |
| 805 | 810 | if (capabilities) |
| 806 | 811 | *capabilities = cmd.out_capabilities; |
| 807 | 812 | |
| @@ -810,14 +815,19 @@ static int _test_cmd_get_hw_info(int fd, __u32 device_id, void *data, · 2 unmodified lines | ||
| 810 | 815 | |
| 811 | 816 | #define test_cmd_get_hw_info(device_id, data, data_len) \ |
| 812 | 817 | ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, data, \ |
| 813 | − data_len, NULL)) | |
| 818 | + data_len, NULL, NULL)) | |
| 814 | 819 | |
| 815 | 820 | #define test_err_get_hw_info(_errno, device_id, data, data_len) \ |
| 816 | 821 | EXPECT_ERRNO(_errno, _test_cmd_get_hw_info(self->fd, device_id, data, \ |
| 817 | − data_len, NULL)) | |
| 822 | + data_len, NULL, NULL)) | |
| 818 | 823 | |
| 819 | 824 | #define test_cmd_get_hw_capabilities(device_id, caps, mask) \ |
| 820 | − ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, NULL, 0, &caps)) | |
| 825 | + ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, NULL, \ | |
| 826 | + 0, &caps, NULL)) | |
| 827 | + | |
| 828 | +#define test_cmd_get_hw_info_pasid(device_id, max_pasid) \ | |
| 829 | + ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, NULL, \ | |
| 830 | + 0, NULL, max_pasid)) | |
| 821 | 831 | |
| 822 | 832 | static int _test_ioctl_fault_alloc(int fd, __u32 *fault_id, __u32 *fault_fd) |
| 823 | 833 | { |
| @@ -842,14 +852,15 @@ static int _test_ioctl_fault_alloc(int fd, __u32 *fault_id, __u32 *fault_fd) · 18 unmodified lines | ||
| 842 | 852 | ASSERT_NE(0, *(fault_fd)); \ |
| 843 | 853 | }) |
| 844 | 854 | |
| 845 | −static int _test_cmd_trigger_iopf(int fd, __u32 device_id, __u32 fault_fd) | |
| 855 | +static int _test_cmd_trigger_iopf(int fd, __u32 device_id, __u32 pasid, | |
| 856 | + __u32 fault_fd) | |
| 846 | 857 | { |
| 847 | 858 | struct iommu_test_cmd trigger_iopf_cmd = { |
| 848 | 859 | .size = sizeof(trigger_iopf_cmd), |
| 849 | 860 | .op = IOMMU_TEST_OP_TRIGGER_IOPF, |
| 850 | 861 | .trigger_iopf = { |
| 851 | 862 | .dev_id = device_id, |
| 852 | − .pasid = 0x1, | |
| 863 | + .pasid = pasid, | |
| 853 | 864 | .grpid = 0x2, |
| 854 | 865 | .perm = IOMMU_PGFAULT_PERM_READ | IOMMU_PGFAULT_PERM_WRITE, |
| 855 | 866 | .addr = 0xdeadbeaf, |
| @@ -880,7 +891,10 @@ static int _test_cmd_trigger_iopf(int fd, __u32 device_id, __u32 fault_fd) · 24 unmodified lines | ||
| 880 | 891 | } |
| 881 | 892 | |
| 882 | 893 | #define test_cmd_trigger_iopf(device_id, fault_fd) \ |
| 883 | − ASSERT_EQ(0, _test_cmd_trigger_iopf(self->fd, device_id, fault_fd)) | |
| 894 | + ASSERT_EQ(0, _test_cmd_trigger_iopf(self->fd, device_id, 0x1, fault_fd)) | |
| 895 | +#define test_cmd_trigger_iopf_pasid(device_id, pasid, fault_fd) \ | |
| 896 | + ASSERT_EQ(0, _test_cmd_trigger_iopf(self->fd, device_id, \ | |
| 897 | + pasid, fault_fd)) | |
| 884 | 898 | |
| 885 | 899 | static int _test_cmd_viommu_alloc(int fd, __u32 device_id, __u32 hwpt_id, |
| 886 | 900 | __u32 type, __u32 flags, __u32 *viommu_id) |
| @@ -936,3 +950,204 @@ static int _test_cmd_vdevice_alloc(int fd, __u32 viommu_id, __u32 idev_id, · 49 unmodified lines | ||
| 936 | 950 | EXPECT_ERRNO(_errno, \ |
| 937 | 951 | _test_cmd_vdevice_alloc(self->fd, viommu_id, idev_id, \ |
| 938 | 952 | virt_id, vdev_id)) |
| 953 | + | |
| 954 | +static int _test_cmd_veventq_alloc(int fd, __u32 viommu_id, __u32 type, | |
| 955 | + __u32 *veventq_id, __u32 *veventq_fd) | |
| 956 | +{ | |
| 957 | + struct iommu_veventq_alloc cmd = { | |
| 958 | + .size = sizeof(cmd), | |
| 959 | + .type = type, | |
| 960 | + .veventq_depth = 2, | |
| 961 | + .viommu_id = viommu_id, | |
| 962 | + }; | |
| 963 | + int ret; | |
| 964 | + | |
| 965 | + ret = ioctl(fd, IOMMU_VEVENTQ_ALLOC, &cmd); | |
| 966 | + if (ret) | |
| 967 | + return ret; | |
| 968 | + if (veventq_id) | |
| 969 | + *veventq_id = cmd.out_veventq_id; | |
| 970 | + if (veventq_fd) | |
| 971 | + *veventq_fd = cmd.out_veventq_fd; | |
| 972 | + return 0; | |
| 973 | +} | |
| 974 | + | |
| 975 | +#define test_cmd_veventq_alloc(viommu_id, type, veventq_id, veventq_fd) \ | |
| 976 | + ASSERT_EQ(0, _test_cmd_veventq_alloc(self->fd, viommu_id, type, \ | |
| 977 | + veventq_id, veventq_fd)) | |
| 978 | +#define test_err_veventq_alloc(_errno, viommu_id, type, veventq_id, \ | |
| 979 | + veventq_fd) \ | |
| 980 | + EXPECT_ERRNO(_errno, \ | |
| 981 | + _test_cmd_veventq_alloc(self->fd, viommu_id, type, \ | |
| 982 | + veventq_id, veventq_fd)) | |
| 983 | + | |
| 984 | +static int _test_cmd_trigger_vevents(int fd, __u32 dev_id, __u32 nvevents) | |
| 985 | +{ | |
| 986 | + struct iommu_test_cmd trigger_vevent_cmd = { | |
| 987 | + .size = sizeof(trigger_vevent_cmd), | |
| 988 | + .op = IOMMU_TEST_OP_TRIGGER_VEVENT, | |
| 989 | + .trigger_vevent = { | |
| 990 | + .dev_id = dev_id, | |
| 991 | + }, | |
| 992 | + }; | |
| 993 | + int ret; | |
| 994 | + | |
| 995 | + while (nvevents--) { | |
| 996 | + ret = ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_TRIGGER_VEVENT), | |
| 997 | + &trigger_vevent_cmd); | |
| 998 | + if (ret < 0) | |
| 999 | + return -1; | |
| 1000 | + } | |
| 1001 | + return ret; | |
| 1002 | +} | |
| 1003 | + | |
| 1004 | +#define test_cmd_trigger_vevents(dev_id, nvevents) \ | |
| 1005 | + ASSERT_EQ(0, _test_cmd_trigger_vevents(self->fd, dev_id, nvevents)) | |
| 1006 | + | |
| 1007 | +static int _test_cmd_read_vevents(int fd, __u32 event_fd, __u32 nvevents, | |
| 1008 | + __u32 virt_id, int *prev_seq) | |
| 1009 | +{ | |
| 1010 | + struct pollfd pollfd = { .fd = event_fd, .events = POLLIN }; | |
| 1011 | + struct iommu_viommu_event_selftest *event; | |
| 1012 | + struct iommufd_vevent_header *hdr; | |
| 1013 | + ssize_t bytes; | |
| 1014 | + void *data; | |
| 1015 | + int ret, i; | |
| 1016 | + | |
| 1017 | + ret = poll(&pollfd, 1, 1000); | |
| 1018 | + if (ret < 0) | |
| 1019 | + return -1; | |
| 1020 | + | |
| 1021 | + data = calloc(nvevents, sizeof(*hdr) + sizeof(*event)); | |
| 1022 | + if (!data) { | |
| 1023 | + errno = ENOMEM; | |
| 1024 | + return -1; | |
| 1025 | + } | |
| 1026 | + | |
| 1027 | + bytes = read(event_fd, data, | |
| 1028 | + nvevents * (sizeof(*hdr) + sizeof(*event))); | |
| 1029 | + if (bytes <= 0) { | |
| 1030 | + errno = EFAULT; | |
| 1031 | + ret = -1; | |
| 1032 | + goto out_free; | |
| 1033 | + } | |
| 1034 | + | |
| 1035 | + for (i = 0; i < nvevents; i++) { | |
| 1036 | + hdr = data + i * (sizeof(*hdr) + sizeof(*event)); | |
| 1037 | + | |
| 1038 | + if (hdr->flags & IOMMU_VEVENTQ_FLAG_LOST_EVENTS || | |
| 1039 | + hdr->sequence - *prev_seq > 1) { | |
| 1040 | + *prev_seq = hdr->sequence; | |
| 1041 | + errno = EOVERFLOW; | |
| 1042 | + ret = -1; | |
| 1043 | + goto out_free; | |
| 1044 | + } | |
| 1045 | + *prev_seq = hdr->sequence; | |
| 1046 | + event = data + sizeof(*hdr); | |
| 1047 | + if (event->virt_id != virt_id) { | |
| 1048 | + errno = EINVAL; | |
| 1049 | + ret = -1; | |
| 1050 | + goto out_free; | |
| 1051 | + } | |
| 1052 | + } | |
| 1053 | + | |
| 1054 | + ret = 0; | |
| 1055 | +out_free: | |
| 1056 | + free(data); | |
| 1057 | + return ret; | |
| 1058 | +} | |
| 1059 | + | |
| 1060 | +#define test_cmd_read_vevents(event_fd, nvevents, virt_id, prev_seq) \ | |
| 1061 | + ASSERT_EQ(0, _test_cmd_read_vevents(self->fd, event_fd, nvevents, \ | |
| 1062 | + virt_id, prev_seq)) | |
| 1063 | +#define test_err_read_vevents(_errno, event_fd, nvevents, virt_id, prev_seq) \ | |
| 1064 | + EXPECT_ERRNO(_errno, \ | |
| 1065 | + _test_cmd_read_vevents(self->fd, event_fd, nvevents, \ | |
| 1066 | + virt_id, prev_seq)) | |
| 1067 | + | |
| 1068 | +static int _test_cmd_pasid_attach(int fd, __u32 stdev_id, __u32 pasid, | |
| 1069 | + __u32 pt_id) | |
| 1070 | +{ | |
| 1071 | + struct iommu_test_cmd test_attach = { | |
| 1072 | + .size = sizeof(test_attach), | |
| 1073 | + .op = IOMMU_TEST_OP_PASID_ATTACH, | |
| 1074 | + .id = stdev_id, | |
| 1075 | + .pasid_attach = { | |
| 1076 | + .pasid = pasid, | |
| 1077 | + .pt_id = pt_id, | |
| 1078 | + }, | |
| 1079 | + }; | |
| 1080 | + | |
| 1081 | + return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_ATTACH), | |
| 1082 | + &test_attach); | |
| 1083 | +} | |
| 1084 | + | |
| 1085 | +#define test_cmd_pasid_attach(pasid, hwpt_id) \ | |
| 1086 | + ASSERT_EQ(0, _test_cmd_pasid_attach(self->fd, self->stdev_id, \ | |
| 1087 | + pasid, hwpt_id)) | |
| 1088 | + | |
| 1089 | +#define test_err_pasid_attach(_errno, pasid, hwpt_id) \ | |
| 1090 | + EXPECT_ERRNO(_errno, \ | |
| 1091 | + _test_cmd_pasid_attach(self->fd, self->stdev_id, \ | |
| 1092 | + pasid, hwpt_id)) | |
| 1093 | + | |
| 1094 | +static int _test_cmd_pasid_replace(int fd, __u32 stdev_id, __u32 pasid, | |
| 1095 | + __u32 pt_id) | |
| 1096 | +{ | |
| 1097 | + struct iommu_test_cmd test_replace = { | |
| 1098 | + .size = sizeof(test_replace), | |
| 1099 | + .op = IOMMU_TEST_OP_PASID_REPLACE, | |
| 1100 | + .id = stdev_id, | |
| 1101 | + .pasid_replace = { | |
| 1102 | + .pasid = pasid, | |
| 1103 | + .pt_id = pt_id, | |
| 1104 | + }, | |
| 1105 | + }; | |
| 1106 | + | |
| 1107 | + return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_REPLACE), | |
| 1108 | + &test_replace); | |
| 1109 | +} | |
| 1110 | + | |
| 1111 | +#define test_cmd_pasid_replace(pasid, hwpt_id) \ | |
| 1112 | + ASSERT_EQ(0, _test_cmd_pasid_replace(self->fd, self->stdev_id, \ | |
| 1113 | + pasid, hwpt_id)) | |
| 1114 | + | |
| 1115 | +#define test_err_pasid_replace(_errno, pasid, hwpt_id) \ | |
| 1116 | + EXPECT_ERRNO(_errno, \ | |
| 1117 | + _test_cmd_pasid_replace(self->fd, self->stdev_id, \ | |
| 1118 | + pasid, hwpt_id)) | |
| 1119 | + | |
| 1120 | +static int _test_cmd_pasid_detach(int fd, __u32 stdev_id, __u32 pasid) | |
| 1121 | +{ | |
| 1122 | + struct iommu_test_cmd test_detach = { | |
| 1123 | + .size = sizeof(test_detach), | |
| 1124 | + .op = IOMMU_TEST_OP_PASID_DETACH, | |
| 1125 | + .id = stdev_id, | |
| 1126 | + .pasid_detach = { | |
| 1127 | + .pasid = pasid, | |
| 1128 | + }, | |
| 1129 | + }; | |
| 1130 | + | |
| 1131 | + return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_DETACH), | |
| 1132 | + &test_detach); | |
| 1133 | +} | |
| 1134 | + | |
| 1135 | +#define test_cmd_pasid_detach(pasid) \ | |
| 1136 | + ASSERT_EQ(0, _test_cmd_pasid_detach(self->fd, self->stdev_id, pasid)) | |
| 1137 | + | |
| 1138 | +static int test_cmd_pasid_check_hwpt(int fd, __u32 stdev_id, __u32 pasid, | |
| 1139 | + __u32 hwpt_id) | |
| 1140 | +{ | |
| 1141 | + struct iommu_test_cmd test_pasid_check = { | |
| 1142 | + .size = sizeof(test_pasid_check), | |
| 1143 | + .op = IOMMU_TEST_OP_PASID_CHECK_HWPT, | |
| 1144 | + .id = stdev_id, | |
| 1145 | + .pasid_check = { | |
| 1146 | + .pasid = pasid, | |
| 1147 | + .hwpt_id = hwpt_id, | |
| 1148 | + }, | |
| 1149 | + }; | |
| 1150 | + | |
| 1151 | + return ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_PASID_CHECK_HWPT), | |
| 1152 | + &test_pasid_check); | |
| 1153 | +} | |