History

Merge tag 'vfio-v6.19-rc1' of https://github.com/awilliam/linux-vfio

Pull VFIO updates from Alex Williamson: - Move libvfio selftest artifacts in preparation of more tightly coupled integration with KVM selftests (David Matlack) - Fix comment typo in mtty driver (Chu Guangqing) - Support for new hardware revision in the hisi_acc vfio-pci variant driver where the migration registers can now be accessed via the PF. When enabled for this support, the full BAR can be exposed to the user (Longfang Liu) - Fix vfio cdev support for VF token passing, using the correct size for the kernel structure, thereby actually allowing userspace to provide a non-zero UUID token. Also set the match token callback for the hisi_acc, fixing VF token support for this this vfio-pci variant driver (Raghavendra Rao Ananta) - Introduce internal callbacks on vfio devices to simplify and consolidate duplicate code for generating VFIO_DEVICE_GET_REGION_INFO data, removing various ioctl intercepts with a more structured solution (Jason Gunthorpe) - Introduce dma-buf support for vfio-pci devices, allowing MMIO regions to be exposed through dma-buf objects with lifecycle managed through move operations. This enables low-level interactions such as a vfio-pci based SPDK drivers interacting directly with dma-buf capable RDMA devices to enable peer-to-peer operations. IOMMUFD is also now able to build upon this support to fill a long standing feature gap versus the legacy vfio type1 IOMMU backend with an implementation of P2P support for VM use cases that better manages the lifecycle of the P2P mapping (Leon Romanovsky, Jason Gunthorpe, Vivek Kasireddy) - Convert eventfd triggering for error and request signals to use RCU mechanisms in order to avoid a 3-way lockdep reported deadlock issue (Alex Williamson) - Fix a 32-bit overflow introduced via dma-buf support manifesting with large DMA buffers (Alex Mastro) - Convert nvgrace-gpu vfio-pci variant driver to insert mappings on fault rather than at mmap time. This conversion serves both to make use of huge PFNMAPs but also to both avoid corrected RAS events during reset by now being subject to vfio-pci-core's use of unmap_mapping_range(), and to enable a device readiness test after reset (Ankit Agrawal) - Refactoring of vfio selftests to support multi-device tests and split code to provide better separation between IOMMU and device objects. This work also enables a new test suite addition to measure parallel device initialization latency (David Matlack) * tag 'vfio-v6.19-rc1' of https://github.com/awilliam/linux-vfio: (65 commits) vfio: selftests: Add vfio_pci_device_init_perf_test vfio: selftests: Eliminate INVALID_IOVA vfio: selftests: Split libvfio.h into separate header files vfio: selftests: Move vfio_selftests_*() helpers into libvfio.c vfio: selftests: Rename vfio_util.h to libvfio.h vfio: selftests: Stop passing device for IOMMU operations vfio: selftests: Move IOVA allocator into iova_allocator.c vfio: selftests: Move IOMMU library code into iommu.c vfio: selftests: Rename struct vfio_dma_region to dma_region vfio: selftests: Upgrade driver logging to dev_err() vfio: selftests: Prefix logs with device BDF where relevant vfio: selftests: Eliminate overly chatty logging vfio: selftests: Support multiple devices in the same container/iommufd vfio: selftests: Introduce struct iommu vfio: selftests: Rename struct vfio_iommu_mode to iommu_mode vfio: selftests: Allow passing multiple BDFs on the command line vfio: selftests: Split run.sh into separate scripts vfio: selftests: Move run.sh into scripts directory vfio/nvgrace-gpu: wait for the GPU mem to be ready vfio/nvgrace-gpu: Inform devmem unmapped after reset ...

Linus Torvalds committed 11 months ago

Commit a3ebb592 parents ce5cfb0 d721f5272 files · +3426 −1899Browse files

Unified · Split

72 files
  1. Documentation/driver-api/pci/p2pdma.rst +74 −23
  2. block/blk-mq-dma.c +1 −1
  3. drivers/crypto/hisilicon/qm.c +27 −0
  4. drivers/dma-buf/Makefile +1 −1
  5. drivers/dma-buf/dma-buf-mapping.c +248 −0
  6. drivers/gpu/drm/i915/gvt/kvmgt.c +117 −146
  7. drivers/iommu/dma-iommu.c +2 −2
  8. drivers/pci/p2pdma.c +144 −42
  9. drivers/s390/cio/vfio_ccw_ops.c +7 −40
  10. drivers/vfio/cdx/main.c +11 −18
  11. drivers/vfio/device_cdev.c +1 −1
  12. drivers/vfio/fsl-mc/vfio_fsl_mc.c +19 −24
  13. drivers/vfio/pci/Kconfig +3 −0
  14. drivers/vfio/pci/Makefile +1 −0
  15. drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +108 −63
  16. drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h +21 −2
  17. drivers/vfio/pci/mlx5/main.c +1 −0
  18. drivers/vfio/pci/nvgrace-gpu/main.c +255 −87
  19. drivers/vfio/pci/pds/vfio_dev.c +1 −0
  20. drivers/vfio/pci/qat/main.c +1 −0
  21. drivers/vfio/pci/vfio_pci.c +6 −0
  22. drivers/vfio/pci/vfio_pci_config.c +19 −4
  23. drivers/vfio/pci/vfio_pci_core.c +156 −144
  24. drivers/vfio/pci/vfio_pci_dmabuf.c +316 −0
  25. drivers/vfio/pci/vfio_pci_intrs.c +33 −19
  26. drivers/vfio/pci/vfio_pci_priv.h +27 −1
  27. drivers/vfio/pci/virtio/common.h +2 −3
  28. drivers/vfio/pci/virtio/legacy_io.c +8 −30
  29. drivers/vfio/pci/virtio/main.c +4 −1
  30. drivers/vfio/platform/vfio_amba.c +1 −0
  31. drivers/vfio/platform/vfio_platform.c +1 −0
  32. drivers/vfio/platform/vfio_platform_common.c +18 −22
  33. drivers/vfio/platform/vfio_platform_private.h +3 −0
  34. drivers/vfio/vfio_main.c +51 −0
  35. include/linux/dma-buf-mapping.h +17 −0
  36. include/linux/dma-buf.h +11 −0
  37. include/linux/hisi_acc_qm.h +3 −0
  38. include/linux/pci-p2pdma.h +73 −47
  39. include/linux/vfio.h +6 −0
  40. include/linux/vfio_pci_core.h +67 −2
  41. include/uapi/linux/vfio.h +28 −0
  42. kernel/dma/direct.c +2 −2
  43. mm/hmm.c +1 −1
  44. samples/vfio-mdev/mbochs.c +24 −47
  45. samples/vfio-mdev/mdpy.c +7 −27
  46. samples/vfio-mdev/mtty.c +7 −28
  47. tools/testing/selftests/vfio/Makefile +9 −1
  48. tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c +18 −18
  49. tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c +9 −9
  50. tools/testing/selftests/vfio/lib/include/libvfio.h +26 −0
  51. tools/testing/selftests/vfio/lib/include/libvfio/assert.h +54 −0
  52. tools/testing/selftests/vfio/lib/include/libvfio/iommu.h +76 −0
  53. tools/testing/selftests/vfio/lib/include/libvfio/iova_allocator.h +23 −0
  54. tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h +125 −0
  55. tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h +97 −0
  56. tools/testing/selftests/vfio/lib/include/vfio_util.h +0 −331
  57. tools/testing/selftests/vfio/lib/iommu.c +465 −0
  58. tools/testing/selftests/vfio/lib/iova_allocator.c +94 −0
  59. tools/testing/selftests/vfio/lib/libvfio.c +78 −0
  60. tools/testing/selftests/vfio/lib/libvfio.mk +14 −9
  61. tools/testing/selftests/vfio/lib/vfio_pci_device.c +19 −537
  62. tools/testing/selftests/vfio/lib/vfio_pci_driver.c +1 −15
  63. tools/testing/selftests/vfio/run.sh +0 −109
  64. tools/testing/selftests/vfio/scripts/cleanup.sh +41 −0
  65. tools/testing/selftests/vfio/scripts/lib.sh +42 −0
  66. tools/testing/selftests/vfio/scripts/run.sh +16 −0
  67. tools/testing/selftests/vfio/scripts/setup.sh +48 −0
  68. tools/testing/selftests/vfio/vfio_dma_mapping_test.c +26 −20
  69. tools/testing/selftests/vfio/vfio_iommufd_setup_test.c +1 −1
  70. tools/testing/selftests/vfio/vfio_pci_device_init_perf_test.c +168 −0
  71. tools/testing/selftests/vfio/vfio_pci_device_test.c +9 −3
  72. tools/testing/selftests/vfio/vfio_pci_driver_test.c +33 −18
Documentation/driver-api/pci/p2pdma.rst+74−23
@@ -9,22 +9,48 @@ between two devices on the bus. This type of transaction is henceforth · 8 unmodified lines
99 called Peer-to-Peer (or P2P). However, there are a number of issues that
1010 make P2P transactions tricky to do in a perfectly safe way.
1111
12−One of the biggest issues is that PCI doesn't require forwarding
13−transactions between hierarchy domains, and in PCIe, each Root Port
14−defines a separate hierarchy domain. To make things worse, there is no
15−simple way to determine if a given Root Complex supports this or not.
16−(See PCIe r4.0, sec 1.3.1). Therefore, as of this writing, the kernel
17−only supports doing P2P when the endpoints involved are all behind the
18−same PCI bridge, as such devices are all in the same PCI hierarchy
19−domain, and the spec guarantees that all transactions within the
20−hierarchy will be routable, but it does not require routing
21−between hierarchies.
22−
23−The second issue is that to make use of existing interfaces in Linux,
24−memory that is used for P2P transactions needs to be backed by struct
25−pages. However, PCI BARs are not typically cache coherent so there are
26−a few corner case gotchas with these pages so developers need to
27−be careful about what they do with them.
12+For PCIe the routing of Transaction Layer Packets (TLPs) is well-defined up
13+until they reach a host bridge or root port. If the path includes PCIe switches
14+then based on the ACS settings the transaction can route entirely within
15+the PCIe hierarchy and never reach the root port. The kernel will evaluate
16+the PCIe topology and always permit P2P in these well-defined cases.
17+
18+However, if the P2P transaction reaches the host bridge then it might have to
19+hairpin back out the same root port, be routed inside the CPU SOC to another
20+PCIe root port, or routed internally to the SOC.
21+
22+The PCIe specification doesn't define the forwarding of transactions between
23+hierarchy domains and kernel defaults to blocking such routing. There is an
24+allow list to allow detecting known-good HW, in which case P2P between any
25+two PCIe devices will be permitted.
26+
27+Since P2P inherently is doing transactions between two devices it requires two
28+drivers to be co-operating inside the kernel. The providing driver has to convey
29+its MMIO to the consuming driver. To meet the driver model lifecycle rules the
30+MMIO must have all DMA mapping removed, all CPU accesses prevented, all page
31+table mappings undone before the providing driver completes remove().
32+
33+This requires the providing and consuming driver to actively work together to
34+guarantee that the consuming driver has stopped using the MMIO during a removal
35+cycle. This is done by either a synchronous invalidation shutdown or waiting
36+for all usage refcounts to reach zero.
37+
38+At the lowest level the P2P subsystem offers a naked struct p2p_provider that
39+delegates lifecycle management to the providing driver. It is expected that
40+drivers using this option will wrap their MMIO memory in DMABUF and use DMABUF
41+to provide an invalidation shutdown. These MMIO addresess have no struct page, and
42+if used with mmap() must create special PTEs. As such there are very few
43+kernel uAPIs that can accept pointers to them; in particular they cannot be used
44+with read()/write(), including O_DIRECT.
45+
46+Building on this, the subsystem offers a layer to wrap the MMIO in a ZONE_DEVICE
47+pgmap of MEMORY_DEVICE_PCI_P2PDMA to create struct pages. The lifecycle of
48+pgmap ensures that when the pgmap is destroyed all other drivers have stopped
49+using the MMIO. This option works with O_DIRECT flows, in some cases, if the
50+underlying subsystem supports handling MEMORY_DEVICE_PCI_P2PDMA through
51+FOLL_PCI_P2PDMA. The use of FOLL_LONGTERM is prevented. As this relies on pgmap
52+it also relies on architecture support along with alignment and minimum size
53+limitations.
2854
2955
3056 Driver Writer's Guide
@@ -114,14 +140,39 @@ allocating scatter-gather lists with P2P memory. · 83 unmodified lines
114140 Struct Page Caveats
115141 -------------------
116142
117−Driver writers should be very careful about not passing these special
118−struct pages to code that isn't prepared for it. At this time, the kernel
119−interfaces do not have any checks for ensuring this. This obviously
120−precludes passing these pages to userspace.
143+While the MEMORY_DEVICE_PCI_P2PDMA pages can be installed in VMAs,
144+pin_user_pages() and related will not return them unless FOLL_PCI_P2PDMA is set.
121145
122−P2P memory is also technically IO memory but should never have any side
123−effects behind it. Thus, the order of loads and stores should not be important
124−and ioreadX(), iowriteX() and friends should not be necessary.
146+The MEMORY_DEVICE_PCI_P2PDMA pages require care to support in the kernel. The
147+KVA is still MMIO and must still be accessed through the normal
148+readX()/writeX()/etc helpers. Direct CPU access (e.g. memcpy) is forbidden, just
149+like any other MMIO mapping. While this will actually work on some
150+architectures, others will experience corruption or just crash in the kernel.
151+Supporting FOLL_PCI_P2PDMA in a subsystem requires scrubbing it to ensure no CPU
152+access happens.
153+
154+
155+Usage With DMABUF
156+=================
157+
158+DMABUF provides an alternative to the above struct page-based
159+client/provider/orchestrator system and should be used when struct page
160+doesn't exist. In this mode the exporting driver will wrap
161+some of its MMIO in a DMABUF and give the DMABUF FD to userspace.
162+
163+Userspace can then pass the FD to an importing driver which will ask the
164+exporting driver to map it to the importer.
165+
166+In this case the initiator and target pci_devices are known and the P2P subsystem
167+is used to determine the mapping type. The phys_addr_t-based DMA API is used to
168+establish the dma_addr_t.
169+
170+Lifecycle is controlled by DMABUF move_notify(). When the exporting driver wants
171+to remove() it must deliver an invalidation shutdown to all DMABUF importing
172+drivers through move_notify() and synchronously DMA unmap all the MMIO.
173+
174+No importing driver can continue to have a DMA map to the MMIO after the
175+exporting driver has destroyed its p2p_provider.
125176
126177
127178 P2P DMA Support Library
block/blk-mq-dma.c+1−1
@@ -84,7 +84,7 @@ static inline bool blk_can_dma_map_iova(struct request *req, · 83 unmodified lines
8484
8585 static bool blk_dma_map_bus(struct blk_dma_iter *iter, struct phys_vec *vec)
8686 {
87− iter->addr = pci_p2pdma_bus_addr_map(&iter->p2pdma, vec->paddr);
87+ iter->addr = pci_p2pdma_bus_addr_map(iter->p2pdma.mem, vec->paddr);
8888 iter->len = vec->len;
8989 return true;
9090 }
drivers/crypto/hisilicon/qm.c+27−0
@@ -3032,11 +3032,36 @@ static void qm_put_pci_res(struct hisi_qm *qm) · 3031 unmodified lines
30323032 pci_release_mem_regions(pdev);
30333033 }
30343034
3035+static void hisi_mig_region_clear(struct hisi_qm *qm)
3036+{
3037+ u32 val;
3038+
3039+ /* Clear migration region set of PF */
3040+ if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) {
3041+ val = readl(qm->io_base + QM_MIG_REGION_SEL);
3042+ val &= ~QM_MIG_REGION_EN;
3043+ writel(val, qm->io_base + QM_MIG_REGION_SEL);
3044+ }
3045+}
3046+
3047+static void hisi_mig_region_enable(struct hisi_qm *qm)
3048+{
3049+ u32 val;
3050+
3051+ /* Select migration region of PF */
3052+ if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) {
3053+ val = readl(qm->io_base + QM_MIG_REGION_SEL);
3054+ val |= QM_MIG_REGION_EN;
3055+ writel(val, qm->io_base + QM_MIG_REGION_SEL);
3056+ }
3057+}
3058+
30353059 static void hisi_qm_pci_uninit(struct hisi_qm *qm)
30363060 {
30373061 struct pci_dev *pdev = qm->pdev;
30383062
30393063 pci_free_irq_vectors(pdev);
3064+ hisi_mig_region_clear(qm);
30403065 qm_put_pci_res(qm);
30413066 pci_disable_device(pdev);
30423067 }
@@ -5752,6 +5777,7 @@ int hisi_qm_init(struct hisi_qm *qm) · 2709 unmodified lines
57525777 goto err_free_qm_memory;
57535778
57545779 qm_cmd_init(qm);
5780+ hisi_mig_region_enable(qm);
57555781
57565782 return 0;
57575783
@@ -5890,6 +5916,7 @@ static int qm_rebuild_for_resume(struct hisi_qm *qm) · 132 unmodified lines
58905916 }
58915917
58925918 qm_cmd_init(qm);
5919+ hisi_mig_region_enable(qm);
58935920 hisi_qm_dev_err_init(qm);
58945921 /* Set the doorbell timeout to QM_DB_TIMEOUT_CFG ns. */
58955922 writel(QM_DB_TIMEOUT_SET, qm->io_base + QM_DB_TIMEOUT_CFG);
drivers/dma-buf/Makefile+1−1
@@ -1,6 +1,6 @@
11 # SPDX-License-Identifier: GPL-2.0-only
22 obj-y := dma-buf.o dma-fence.o dma-fence-array.o dma-fence-chain.o \
3− dma-fence-unwrap.o dma-resv.o
3+ dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o
44 obj-$(CONFIG_DMABUF_HEAPS) += dma-heap.o
55 obj-$(CONFIG_DMABUF_HEAPS) += heaps/
66 obj-$(CONFIG_SYNC_FILE) += sync_file.o
drivers/dma-buf/dma-buf-mapping.c+248−0
@@ -0,0 +1,248 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/*
3+ * DMA BUF Mapping Helpers
4+ *
5+ */
6+#include <linux/dma-buf-mapping.h>
7+#include <linux/dma-resv.h>
8+
9+static struct scatterlist *fill_sg_entry(struct scatterlist *sgl, size_t length,
10+ dma_addr_t addr)
11+{
12+ unsigned int len, nents;
13+ int i;
14+
15+ nents = DIV_ROUND_UP(length, UINT_MAX);
16+ for (i = 0; i < nents; i++) {
17+ len = min_t(size_t, length, UINT_MAX);
18+ length -= len;
19+ /*
20+ * DMABUF abuses scatterlist to create a scatterlist
21+ * that does not have any CPU list, only the DMA list.
22+ * Always set the page related values to NULL to ensure
23+ * importers can't use it. The phys_addr based DMA API
24+ * does not require the CPU list for mapping or unmapping.
25+ */
26+ sg_set_page(sgl, NULL, 0, 0);
27+ sg_dma_address(sgl) = addr + (dma_addr_t)i * UINT_MAX;
28+ sg_dma_len(sgl) = len;
29+ sgl = sg_next(sgl);
30+ }
31+
32+ return sgl;
33+}
34+
35+static unsigned int calc_sg_nents(struct dma_iova_state *state,
36+ struct dma_buf_phys_vec *phys_vec,
37+ size_t nr_ranges, size_t size)
38+{
39+ unsigned int nents = 0;
40+ size_t i;
41+
42+ if (!state || !dma_use_iova(state)) {
43+ for (i = 0; i < nr_ranges; i++)
44+ nents += DIV_ROUND_UP(phys_vec[i].len, UINT_MAX);
45+ } else {
46+ /*
47+ * In IOVA case, there is only one SG entry which spans
48+ * for whole IOVA address space, but we need to make sure
49+ * that it fits sg->length, maybe we need more.
50+ */
51+ nents = DIV_ROUND_UP(size, UINT_MAX);
52+ }
53+
54+ return nents;
55+}
56+
57+/**
58+ * struct dma_buf_dma - holds DMA mapping information
59+ * @sgt: Scatter-gather table
60+ * @state: DMA IOVA state relevant in IOMMU-based DMA
61+ * @size: Total size of DMA transfer
62+ */
63+struct dma_buf_dma {
64+ struct sg_table sgt;
65+ struct dma_iova_state *state;
66+ size_t size;
67+};
68+
69+/**
70+ * dma_buf_phys_vec_to_sgt - Returns the scatterlist table of the attachment
71+ * from arrays of physical vectors. This funciton is intended for MMIO memory
72+ * only.
73+ * @attach: [in] attachment whose scatterlist is to be returned
74+ * @provider: [in] p2pdma provider
75+ * @phys_vec: [in] array of physical vectors
76+ * @nr_ranges: [in] number of entries in phys_vec array
77+ * @size: [in] total size of phys_vec
78+ * @dir: [in] direction of DMA transfer
79+ *
80+ * Returns sg_table containing the scatterlist to be returned; returns ERR_PTR
81+ * on error. May return -EINTR if it is interrupted by a signal.
82+ *
83+ * On success, the DMA addresses and lengths in the returned scatterlist are
84+ * PAGE_SIZE aligned.
85+ *
86+ * A mapping must be unmapped by using dma_buf_free_sgt().
87+ *
88+ * NOTE: This function is intended for exporters. If direct traffic routing is
89+ * mandatory exporter should call routing pci_p2pdma_map_type() before calling
90+ * this function.
91+ */
92+struct sg_table *dma_buf_phys_vec_to_sgt(struct dma_buf_attachment *attach,
93+ struct p2pdma_provider *provider,
94+ struct dma_buf_phys_vec *phys_vec,
95+ size_t nr_ranges, size_t size,
96+ enum dma_data_direction dir)
97+{
98+ unsigned int nents, mapped_len = 0;
99+ struct dma_buf_dma *dma;
100+ struct scatterlist *sgl;
101+ dma_addr_t addr;
102+ size_t i;
103+ int ret;
104+
105+ dma_resv_assert_held(attach->dmabuf->resv);
106+
107+ if (WARN_ON(!attach || !attach->dmabuf || !provider))
108+ /* This function is supposed to work on MMIO memory only */
109+ return ERR_PTR(-EINVAL);
110+
111+ dma = kzalloc(sizeof(*dma), GFP_KERNEL);
112+ if (!dma)
113+ return ERR_PTR(-ENOMEM);
114+
115+ switch (pci_p2pdma_map_type(provider, attach->dev)) {
116+ case PCI_P2PDMA_MAP_BUS_ADDR:
117+ /*
118+ * There is no need in IOVA at all for this flow.
119+ */
120+ break;
121+ case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE:
122+ dma->state = kzalloc(sizeof(*dma->state), GFP_KERNEL);
123+ if (!dma->state) {
124+ ret = -ENOMEM;
125+ goto err_free_dma;
126+ }
127+
128+ dma_iova_try_alloc(attach->dev, dma->state, 0, size);
129+ break;
130+ default:
131+ ret = -EINVAL;
132+ goto err_free_dma;
133+ }
134+
135+ nents = calc_sg_nents(dma->state, phys_vec, nr_ranges, size);
136+ ret = sg_alloc_table(&dma->sgt, nents, GFP_KERNEL | __GFP_ZERO);
137+ if (ret)
138+ goto err_free_state;
139+
140+ sgl = dma->sgt.sgl;
141+
142+ for (i = 0; i < nr_ranges; i++) {
143+ if (!dma->state) {
144+ addr = pci_p2pdma_bus_addr_map(provider,
145+ phys_vec[i].paddr);
146+ } else if (dma_use_iova(dma->state)) {
147+ ret = dma_iova_link(attach->dev, dma->state,
148+ phys_vec[i].paddr, 0,
149+ phys_vec[i].len, dir,
150+ DMA_ATTR_MMIO);
151+ if (ret)
152+ goto err_unmap_dma;
153+
154+ mapped_len += phys_vec[i].len;
155+ } else {
156+ addr = dma_map_phys(attach->dev, phys_vec[i].paddr,
157+ phys_vec[i].len, dir,
158+ DMA_ATTR_MMIO);
159+ ret = dma_mapping_error(attach->dev, addr);
160+ if (ret)
161+ goto err_unmap_dma;
162+ }
163+
164+ if (!dma->state || !dma_use_iova(dma->state))
165+ sgl = fill_sg_entry(sgl, phys_vec[i].len, addr);
166+ }
167+
168+ if (dma->state && dma_use_iova(dma->state)) {
169+ WARN_ON_ONCE(mapped_len != size);
170+ ret = dma_iova_sync(attach->dev, dma->state, 0, mapped_len);
171+ if (ret)
172+ goto err_unmap_dma;
173+
174+ sgl = fill_sg_entry(sgl, mapped_len, dma->state->addr);
175+ }
176+
177+ dma->size = size;
178+
179+ /*
180+ * No CPU list included — set orig_nents = 0 so others can detect
181+ * this via SG table (use nents only).
182+ */
183+ dma->sgt.orig_nents = 0;
184+
185+
186+ /*
187+ * SGL must be NULL to indicate that SGL is the last one
188+ * and we allocated correct number of entries in sg_alloc_table()
189+ */
190+ WARN_ON_ONCE(sgl);
191+ return &dma->sgt;
192+
193+err_unmap_dma:
194+ if (!i || !dma->state) {
195+ ; /* Do nothing */
196+ } else if (dma_use_iova(dma->state)) {
197+ dma_iova_destroy(attach->dev, dma->state, mapped_len, dir,
198+ DMA_ATTR_MMIO);
199+ } else {
200+ for_each_sgtable_dma_sg(&dma->sgt, sgl, i)
201+ dma_unmap_phys(attach->dev, sg_dma_address(sgl),
202+ sg_dma_len(sgl), dir, DMA_ATTR_MMIO);
203+ }
204+ sg_free_table(&dma->sgt);
205+err_free_state:
206+ kfree(dma->state);
207+err_free_dma:
208+ kfree(dma);
209+ return ERR_PTR(ret);
210+}
211+EXPORT_SYMBOL_NS_GPL(dma_buf_phys_vec_to_sgt, "DMA_BUF");
212+
213+/**
214+ * dma_buf_free_sgt- unmaps the buffer
215+ * @attach: [in] attachment to unmap buffer from
216+ * @sgt: [in] scatterlist info of the buffer to unmap
217+ * @dir: [in] direction of DMA transfer
218+ *
219+ * This unmaps a DMA mapping for @attached obtained
220+ * by dma_buf_phys_vec_to_sgt().
221+ */
222+void dma_buf_free_sgt(struct dma_buf_attachment *attach, struct sg_table *sgt,
223+ enum dma_data_direction dir)
224+{
225+ struct dma_buf_dma *dma = container_of(sgt, struct dma_buf_dma, sgt);
226+ int i;
227+
228+ dma_resv_assert_held(attach->dmabuf->resv);
229+
230+ if (!dma->state) {
231+ ; /* Do nothing */
232+ } else if (dma_use_iova(dma->state)) {
233+ dma_iova_destroy(attach->dev, dma->state, dma->size, dir,
234+ DMA_ATTR_MMIO);
235+ } else {
236+ struct scatterlist *sgl;
237+
238+ for_each_sgtable_dma_sg(sgt, sgl, i)
239+ dma_unmap_phys(attach->dev, sg_dma_address(sgl),
240+ sg_dma_len(sgl), dir, DMA_ATTR_MMIO);
241+ }
242+
243+ sg_free_table(sgt);
244+ kfree(dma->state);
245+ kfree(dma);
246+
247+}
248+EXPORT_SYMBOL_NS_GPL(dma_buf_free_sgt, "DMA_BUF");
drivers/gpu/drm/i915/gvt/kvmgt.c+117−146
@@ -1141,6 +1141,122 @@ static int intel_vgpu_set_irqs(struct intel_vgpu *vgpu, u32 flags, · 1140 unmodified lines
11411141 return func(vgpu, index, start, count, flags, data);
11421142 }
11431143
1144+static int intel_vgpu_ioctl_get_region_info(struct vfio_device *vfio_dev,
1145+ struct vfio_region_info *info,
1146+ struct vfio_info_cap *caps)
1147+{
1148+ struct vfio_region_info_cap_sparse_mmap *sparse = NULL;
1149+ struct intel_vgpu *vgpu = vfio_dev_to_vgpu(vfio_dev);
1150+ int nr_areas = 1;
1151+ int cap_type_id;
1152+ unsigned int i;
1153+ int ret;
1154+
1155+ switch (info->index) {
1156+ case VFIO_PCI_CONFIG_REGION_INDEX:
1157+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1158+ info->size = vgpu->gvt->device_info.cfg_space_size;
1159+ info->flags = VFIO_REGION_INFO_FLAG_READ |
1160+ VFIO_REGION_INFO_FLAG_WRITE;
1161+ break;
1162+ case VFIO_PCI_BAR0_REGION_INDEX:
1163+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1164+ info->size = vgpu->cfg_space.bar[info->index].size;
1165+ if (!info->size) {
1166+ info->flags = 0;
1167+ break;
1168+ }
1169+
1170+ info->flags = VFIO_REGION_INFO_FLAG_READ |
1171+ VFIO_REGION_INFO_FLAG_WRITE;
1172+ break;
1173+ case VFIO_PCI_BAR1_REGION_INDEX:
1174+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1175+ info->size = 0;
1176+ info->flags = 0;
1177+ break;
1178+ case VFIO_PCI_BAR2_REGION_INDEX:
1179+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1180+ info->flags = VFIO_REGION_INFO_FLAG_CAPS |
1181+ VFIO_REGION_INFO_FLAG_MMAP |
1182+ VFIO_REGION_INFO_FLAG_READ |
1183+ VFIO_REGION_INFO_FLAG_WRITE;
1184+ info->size = gvt_aperture_sz(vgpu->gvt);
1185+
1186+ sparse = kzalloc(struct_size(sparse, areas, nr_areas),
1187+ GFP_KERNEL);
1188+ if (!sparse)
1189+ return -ENOMEM;
1190+
1191+ sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
1192+ sparse->header.version = 1;
1193+ sparse->nr_areas = nr_areas;
1194+ cap_type_id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
1195+ sparse->areas[0].offset =
1196+ PAGE_ALIGN(vgpu_aperture_offset(vgpu));
1197+ sparse->areas[0].size = vgpu_aperture_sz(vgpu);
1198+ break;
1199+
1200+ case VFIO_PCI_BAR3_REGION_INDEX ... VFIO_PCI_BAR5_REGION_INDEX:
1201+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1202+ info->size = 0;
1203+ info->flags = 0;
1204+
1205+ gvt_dbg_core("get region info bar:%d\n", info->index);
1206+ break;
1207+
1208+ case VFIO_PCI_ROM_REGION_INDEX:
1209+ case VFIO_PCI_VGA_REGION_INDEX:
1210+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1211+ info->size = 0;
1212+ info->flags = 0;
1213+
1214+ gvt_dbg_core("get region info index:%d\n", info->index);
1215+ break;
1216+ default: {
1217+ struct vfio_region_info_cap_type cap_type = {
1218+ .header.id = VFIO_REGION_INFO_CAP_TYPE,
1219+ .header.version = 1
1220+ };
1221+
1222+ if (info->index >= VFIO_PCI_NUM_REGIONS + vgpu->num_regions)
1223+ return -EINVAL;
1224+ info->index = array_index_nospec(
1225+ info->index, VFIO_PCI_NUM_REGIONS + vgpu->num_regions);
1226+
1227+ i = info->index - VFIO_PCI_NUM_REGIONS;
1228+
1229+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1230+ info->size = vgpu->region[i].size;
1231+ info->flags = vgpu->region[i].flags;
1232+
1233+ cap_type.type = vgpu->region[i].type;
1234+ cap_type.subtype = vgpu->region[i].subtype;
1235+
1236+ ret = vfio_info_add_capability(caps, &cap_type.header,
1237+ sizeof(cap_type));
1238+ if (ret)
1239+ return ret;
1240+ }
1241+ }
1242+
1243+ if ((info->flags & VFIO_REGION_INFO_FLAG_CAPS) && sparse) {
1244+ ret = -EINVAL;
1245+ if (cap_type_id == VFIO_REGION_INFO_CAP_SPARSE_MMAP) {
1246+ ret = vfio_info_add_capability(
1247+ caps, &sparse->header,
1248+ struct_size(sparse, areas, sparse->nr_areas));
1249+ }
1250+ if (ret) {
1251+ kfree(sparse);
1252+ return ret;
1253+ }
1254+ }
1255+
1256+ kfree(sparse);
1257+ return 0;
1258+}
1259+
11441260 static long intel_vgpu_ioctl(struct vfio_device *vfio_dev, unsigned int cmd,
11451261 unsigned long arg)
11461262 {
@@ -1169,152 +1285,6 @@ static long intel_vgpu_ioctl(struct vfio_device *vfio_dev, unsigned int cmd, · 22 unmodified lines
11691285 return copy_to_user((void __user *)arg, &info, minsz) ?
11701286 -EFAULT : 0;
11711287
1172− } else if (cmd == VFIO_DEVICE_GET_REGION_INFO) {
1173− struct vfio_region_info info;
1174− struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
1175− unsigned int i;
1176− int ret;
1177− struct vfio_region_info_cap_sparse_mmap *sparse = NULL;
1178− int nr_areas = 1;
1179− int cap_type_id;
1180−
1181− minsz = offsetofend(struct vfio_region_info, offset);
1182−
1183− if (copy_from_user(&info, (void __user *)arg, minsz))
1184− return -EFAULT;
1185−
1186− if (info.argsz < minsz)
1187− return -EINVAL;
1188−
1189− switch (info.index) {
1190− case VFIO_PCI_CONFIG_REGION_INDEX:
1191− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1192− info.size = vgpu->gvt->device_info.cfg_space_size;
1193− info.flags = VFIO_REGION_INFO_FLAG_READ |
1194− VFIO_REGION_INFO_FLAG_WRITE;
1195− break;
1196− case VFIO_PCI_BAR0_REGION_INDEX:
1197− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1198− info.size = vgpu->cfg_space.bar[info.index].size;
1199− if (!info.size) {
1200− info.flags = 0;
1201− break;
1202− }
1203−
1204− info.flags = VFIO_REGION_INFO_FLAG_READ |
1205− VFIO_REGION_INFO_FLAG_WRITE;
1206− break;
1207− case VFIO_PCI_BAR1_REGION_INDEX:
1208− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1209− info.size = 0;
1210− info.flags = 0;
1211− break;
1212− case VFIO_PCI_BAR2_REGION_INDEX:
1213− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1214− info.flags = VFIO_REGION_INFO_FLAG_CAPS |
1215− VFIO_REGION_INFO_FLAG_MMAP |
1216− VFIO_REGION_INFO_FLAG_READ |
1217− VFIO_REGION_INFO_FLAG_WRITE;
1218− info.size = gvt_aperture_sz(vgpu->gvt);
1219−
1220− sparse = kzalloc(struct_size(sparse, areas, nr_areas),
1221− GFP_KERNEL);
1222− if (!sparse)
1223− return -ENOMEM;
1224−
1225− sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
1226− sparse->header.version = 1;
1227− sparse->nr_areas = nr_areas;
1228− cap_type_id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
1229− sparse->areas[0].offset =
1230− PAGE_ALIGN(vgpu_aperture_offset(vgpu));
1231− sparse->areas[0].size = vgpu_aperture_sz(vgpu);
1232− break;
1233−
1234− case VFIO_PCI_BAR3_REGION_INDEX ... VFIO_PCI_BAR5_REGION_INDEX:
1235− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1236− info.size = 0;
1237− info.flags = 0;
1238−
1239− gvt_dbg_core("get region info bar:%d\n", info.index);
1240− break;
1241−
1242− case VFIO_PCI_ROM_REGION_INDEX:
1243− case VFIO_PCI_VGA_REGION_INDEX:
1244− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1245− info.size = 0;
1246− info.flags = 0;
1247−
1248− gvt_dbg_core("get region info index:%d\n", info.index);
1249− break;
1250− default:
1251− {
1252− struct vfio_region_info_cap_type cap_type = {
1253− .header.id = VFIO_REGION_INFO_CAP_TYPE,
1254− .header.version = 1 };
1255−
1256− if (info.index >= VFIO_PCI_NUM_REGIONS +
1257− vgpu->num_regions)
1258− return -EINVAL;
1259− info.index =
1260− array_index_nospec(info.index,
1261− VFIO_PCI_NUM_REGIONS +
1262− vgpu->num_regions);
1263−
1264− i = info.index - VFIO_PCI_NUM_REGIONS;
1265−
1266− info.offset =
1267− VFIO_PCI_INDEX_TO_OFFSET(info.index);
1268− info.size = vgpu->region[i].size;
1269− info.flags = vgpu->region[i].flags;
1270−
1271− cap_type.type = vgpu->region[i].type;
1272− cap_type.subtype = vgpu->region[i].subtype;
1273−
1274− ret = vfio_info_add_capability(&caps,
1275− &cap_type.header,
1276− sizeof(cap_type));
1277− if (ret)
1278− return ret;
1279− }
1280− }
1281−
1282− if ((info.flags & VFIO_REGION_INFO_FLAG_CAPS) && sparse) {
1283− ret = -EINVAL;
1284− if (cap_type_id == VFIO_REGION_INFO_CAP_SPARSE_MMAP)
1285− ret = vfio_info_add_capability(&caps,
1286− &sparse->header,
1287− struct_size(sparse, areas,
1288− sparse->nr_areas));
1289− if (ret) {
1290− kfree(sparse);
1291− return ret;
1292− }
1293− }
1294−
1295− if (caps.size) {
1296− info.flags |= VFIO_REGION_INFO_FLAG_CAPS;
1297− if (info.argsz < sizeof(info) + caps.size) {
1298− info.argsz = sizeof(info) + caps.size;
1299− info.cap_offset = 0;
1300− } else {
1301− vfio_info_cap_shift(&caps, sizeof(info));
1302− if (copy_to_user((void __user *)arg +
1303− sizeof(info), caps.buf,
1304− caps.size)) {
1305− kfree(caps.buf);
1306− kfree(sparse);
1307− return -EFAULT;
1308− }
1309− info.cap_offset = sizeof(info);
1310− }
1311−
1312− kfree(caps.buf);
1313− }
1314−
1315− kfree(sparse);
1316− return copy_to_user((void __user *)arg, &info, minsz) ?
1317− -EFAULT : 0;
13181288 } else if (cmd == VFIO_DEVICE_GET_IRQ_INFO) {
13191289 struct vfio_irq_info info;
13201290
@@ -1477,6 +1447,7 @@ static const struct vfio_device_ops intel_vgpu_dev_ops = { · 156 unmodified lines
14771447 .write = intel_vgpu_write,
14781448 .mmap = intel_vgpu_mmap,
14791449 .ioctl = intel_vgpu_ioctl,
1450+ .get_region_info_caps = intel_vgpu_ioctl_get_region_info,
14801451 .dma_unmap = intel_vgpu_dma_unmap,
14811452 .bind_iommufd = vfio_iommufd_emulated_bind,
14821453 .unbind_iommufd = vfio_iommufd_emulated_unbind,
drivers/iommu/dma-iommu.c+2−2
@@ -1439,8 +1439,8 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents, · 1438 unmodified lines
14391439 * as a bus address, __finalise_sg() will copy the dma
14401440 * address into the output segment.
14411441 */
1442− s->dma_address = pci_p2pdma_bus_addr_map(&p2pdma_state,
1443− sg_phys(s));
1442+ s->dma_address = pci_p2pdma_bus_addr_map(
1443+ p2pdma_state.mem, sg_phys(s));
14441444 sg_dma_len(s) = sg->length;
14451445 sg_dma_mark_bus_address(s);
14461446 continue;
drivers/pci/p2pdma.c+144−42
@@ -25,12 +25,12 @@ struct pci_p2pdma { · 24 unmodified lines
2525 struct gen_pool *pool;
2626 bool p2pmem_published;
2727 struct xarray map_types;
28+ struct p2pdma_provider mem[PCI_STD_NUM_BARS];
2829 };
2930
3031 struct pci_p2pdma_pagemap {
31− struct pci_dev *provider;
32− u64 bus_offset;
3332 struct dev_pagemap pgmap;
33+ struct p2pdma_provider *mem;
3434 };
3535
3636 static struct pci_p2pdma_pagemap *to_p2p_pgmap(struct dev_pagemap *pgmap)
@@ -204,8 +204,8 @@ static void p2pdma_page_free(struct page *page) · 167 unmodified lines
204204 {
205205 struct pci_p2pdma_pagemap *pgmap = to_p2p_pgmap(page_pgmap(page));
206206 /* safe to dereference while a reference is held to the percpu ref */
207− struct pci_p2pdma *p2pdma =
208− rcu_dereference_protected(pgmap->provider->p2pdma, 1);
207+ struct pci_p2pdma *p2pdma = rcu_dereference_protected(
208+ to_pci_dev(pgmap->mem->owner)->p2pdma, 1);
209209 struct percpu_ref *ref;
210210
211211 gen_pool_free_owner(p2pdma->pool, (uintptr_t)page_to_virt(page),
@@ -228,56 +228,136 @@ static void pci_p2pdma_release(void *data) · 16 unmodified lines
228228
229229 /* Flush and disable pci_alloc_p2p_mem() */
230230 pdev->p2pdma = NULL;
231− synchronize_rcu();
231+ if (p2pdma->pool)
232+ synchronize_rcu();
233+ xa_destroy(&p2pdma->map_types);
234+
235+ if (!p2pdma->pool)
236+ return;
232237
233238 gen_pool_destroy(p2pdma->pool);
234239 sysfs_remove_group(&pdev->dev.kobj, &p2pmem_group);
235− xa_destroy(&p2pdma->map_types);
236240 }
237241
238−static int pci_p2pdma_setup(struct pci_dev *pdev)
242+/**
243+ * pcim_p2pdma_init - Initialise peer-to-peer DMA providers
244+ * @pdev: The PCI device to enable P2PDMA for
245+ *
246+ * This function initializes the peer-to-peer DMA infrastructure
247+ * for a PCI device. It allocates and sets up the necessary data
248+ * structures to support P2PDMA operations, including mapping type
249+ * tracking.
250+ */
251+int pcim_p2pdma_init(struct pci_dev *pdev)
239252 {
240− int error = -ENOMEM;
241253 struct pci_p2pdma *p2p;
254+ int i, ret;
255+
256+ p2p = rcu_dereference_protected(pdev->p2pdma, 1);
257+ if (p2p)
258+ return 0;
242259
243260 p2p = devm_kzalloc(&pdev->dev, sizeof(*p2p), GFP_KERNEL);
244261 if (!p2p)
245262 return -ENOMEM;
246263
247264 xa_init(&p2p->map_types);
265+ /*
266+ * Iterate over all standard PCI BARs and record only those that
267+ * correspond to MMIO regions. Skip non-memory resources (e.g. I/O
268+ * port BARs) since they cannot be used for peer-to-peer (P2P)
269+ * transactions.
270+ */
271+ for (i = 0; i < PCI_STD_NUM_BARS; i++) {
272+ if (!(pci_resource_flags(pdev, i) & IORESOURCE_MEM))
273+ continue;
248274
249− p2p->pool = gen_pool_create(PAGE_SHIFT, dev_to_node(&pdev->dev));
250− if (!p2p->pool)
251− goto out;
275+ p2p->mem[i].owner = &pdev->dev;
276+ p2p->mem[i].bus_offset =
277+ pci_bus_address(pdev, i) - pci_resource_start(pdev, i);
278+ }
252279
253− error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_release, pdev);
254− if (error)
255− goto out_pool_destroy;
280+ ret = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_release, pdev);
281+ if (ret)
282+ goto out_p2p;
256283
257− error = sysfs_create_group(&pdev->dev.kobj, &p2pmem_group);
258− if (error)
284+ rcu_assign_pointer(pdev->p2pdma, p2p);
285+ return 0;
286+
287+out_p2p:
288+ devm_kfree(&pdev->dev, p2p);
289+ return ret;
290+}
291+EXPORT_SYMBOL_GPL(pcim_p2pdma_init);
292+
293+/**
294+ * pcim_p2pdma_provider - Get peer-to-peer DMA provider
295+ * @pdev: The PCI device to enable P2PDMA for
296+ * @bar: BAR index to get provider
297+ *
298+ * This function gets peer-to-peer DMA provider for a PCI device. The lifetime
299+ * of the provider (and of course the MMIO) is bound to the lifetime of the
300+ * driver. A driver calling this function must ensure that all references to the
301+ * provider, and any DMA mappings created for any MMIO, are all cleaned up
302+ * before the driver remove() completes.
303+ *
304+ * Since P2P is almost always shared with a second driver this means some system
305+ * to notify, invalidate and revoke the MMIO's DMA must be in place to use this
306+ * function. For example a revoke can be built using DMABUF.
307+ */
308+struct p2pdma_provider *pcim_p2pdma_provider(struct pci_dev *pdev, int bar)
309+{
310+ struct pci_p2pdma *p2p;
311+
312+ if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM))
313+ return NULL;
314+
315+ p2p = rcu_dereference_protected(pdev->p2pdma, 1);
316+ if (WARN_ON(!p2p))
317+ /* Someone forgot to call to pcim_p2pdma_init() before */
318+ return NULL;
319+
320+ return &p2p->mem[bar];
321+}
322+EXPORT_SYMBOL_GPL(pcim_p2pdma_provider);
323+
324+static int pci_p2pdma_setup_pool(struct pci_dev *pdev)
325+{
326+ struct pci_p2pdma *p2pdma;
327+ int ret;
328+
329+ p2pdma = rcu_dereference_protected(pdev->p2pdma, 1);
330+ if (p2pdma->pool)
331+ /* We already setup pools, do nothing, */
332+ return 0;
333+
334+ p2pdma->pool = gen_pool_create(PAGE_SHIFT, dev_to_node(&pdev->dev));
335+ if (!p2pdma->pool)
336+ return -ENOMEM;
337+
338+ ret = sysfs_create_group(&pdev->dev.kobj, &p2pmem_group);
339+ if (ret)
259340 goto out_pool_destroy;
260341
261− rcu_assign_pointer(pdev->p2pdma, p2p);
262342 return 0;
263343
264344 out_pool_destroy:
265− gen_pool_destroy(p2p->pool);
266−out:
267− devm_kfree(&pdev->dev, p2p);
268− return error;
345+ gen_pool_destroy(p2pdma->pool);
346+ p2pdma->pool = NULL;
347+ return ret;
269348 }
270349
271350 static void pci_p2pdma_unmap_mappings(void *data)
272351 {
273− struct pci_dev *pdev = data;
352+ struct pci_p2pdma_pagemap *p2p_pgmap = data;
274353
275354 /*
276355 * Removing the alloc attribute from sysfs will call
277356 * unmap_mapping_range() on the inode, teardown any existing userspace
278357 * mappings and prevent new ones from being created.
279358 */
280− sysfs_remove_file_from_group(&pdev->dev.kobj, &p2pmem_alloc_attr.attr,
359+ sysfs_remove_file_from_group(&p2p_pgmap->mem->owner->kobj,
360+ &p2pmem_alloc_attr.attr,
281361 p2pmem_group.name);
282362 }
283363
@@ -295,6 +375,7 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, · 11 unmodified lines
295375 u64 offset)
296376 {
297377 struct pci_p2pdma_pagemap *p2p_pgmap;
378+ struct p2pdma_provider *mem;
298379 struct dev_pagemap *pgmap;
299380 struct pci_p2pdma *p2pdma;
300381 void *addr;
@@ -312,11 +393,21 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, · 11 unmodified lines
312393 if (size + offset > pci_resource_len(pdev, bar))
313394 return -EINVAL;
314395
315− if (!pdev->p2pdma) {
316− error = pci_p2pdma_setup(pdev);
317− if (error)
318− return error;
319− }
396+ error = pcim_p2pdma_init(pdev);
397+ if (error)
398+ return error;
399+
400+ error = pci_p2pdma_setup_pool(pdev);
401+ if (error)
402+ return error;
403+
404+ mem = pcim_p2pdma_provider(pdev, bar);
405+ /*
406+ * We checked validity of BAR prior to call
407+ * to pcim_p2pdma_provider. It should never return NULL.
408+ */
409+ if (WARN_ON(!mem))
410+ return -EINVAL;
320411
321412 p2p_pgmap = devm_kzalloc(&pdev->dev, sizeof(*p2p_pgmap), GFP_KERNEL);
322413 if (!p2p_pgmap)
@@ -328,10 +419,7 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, · 5 unmodified lines
328419 pgmap->nr_range = 1;
329420 pgmap->type = MEMORY_DEVICE_PCI_P2PDMA;
330421 pgmap->ops = &p2pdma_pgmap_ops;
331−
332− p2p_pgmap->provider = pdev;
333− p2p_pgmap->bus_offset = pci_bus_address(pdev, bar) -
334− pci_resource_start(pdev, bar);
422+ p2p_pgmap->mem = mem;
335423
336424 addr = devm_memremap_pages(&pdev->dev, pgmap);
337425 if (IS_ERR(addr)) {
@@ -340,7 +428,7 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, · 2 unmodified lines
340428 }
341429
342430 error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_unmap_mappings,
343− pdev);
431+ p2p_pgmap);
344432 if (error)
345433 goto pages_free;
346434
@@ -972,16 +1060,26 @@ void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) · 625 unmodified lines
9721060 }
9731061 EXPORT_SYMBOL_GPL(pci_p2pmem_publish);
9741062
975−static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap,
976− struct device *dev)
1063+/**
1064+ * pci_p2pdma_map_type - Determine the mapping type for P2PDMA transfers
1065+ * @provider: P2PDMA provider structure
1066+ * @dev: Target device for the transfer
1067+ *
1068+ * Determines how peer-to-peer DMA transfers should be mapped between
1069+ * the provider and the target device. The mapping type indicates whether
1070+ * the transfer can be done directly through PCI switches or must go
1071+ * through the host bridge.
1072+ */
1073+enum pci_p2pdma_map_type pci_p2pdma_map_type(struct p2pdma_provider *provider,
1074+ struct device *dev)
9771075 {
9781076 enum pci_p2pdma_map_type type = PCI_P2PDMA_MAP_NOT_SUPPORTED;
979− struct pci_dev *provider = to_p2p_pgmap(pgmap)->provider;
1077+ struct pci_dev *pdev = to_pci_dev(provider->owner);
9801078 struct pci_dev *client;
9811079 struct pci_p2pdma *p2pdma;
9821080 int dist;
9831081
984− if (!provider->p2pdma)
1082+ if (!pdev->p2pdma)
9851083 return PCI_P2PDMA_MAP_NOT_SUPPORTED;
9861084
9871085 if (!dev_is_pci(dev))
@@ -990,7 +1088,7 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, · 2 unmodified lines
9901088 client = to_pci_dev(dev);
9911089
9921090 rcu_read_lock();
993− p2pdma = rcu_dereference(provider->p2pdma);
1091+ p2pdma = rcu_dereference(pdev->p2pdma);
9941092
9951093 if (p2pdma)
9961094 type = xa_to_value(xa_load(&p2pdma->map_types,
@@ -998,7 +1096,7 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, · 1 unmodified line
9981096 rcu_read_unlock();
9991097
10001098 if (type == PCI_P2PDMA_MAP_UNKNOWN)
1001− return calc_map_type_and_dist(provider, client, &dist, true);
1099+ return calc_map_type_and_dist(pdev, client, &dist, true);
10021100
10031101 return type;
10041102 }
@@ -1006,9 +1104,13 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, · 1 unmodified line
10061104 void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state,
10071105 struct device *dev, struct page *page)
10081106 {
1009− state->pgmap = page_pgmap(page);
1010− state->map = pci_p2pdma_map_type(state->pgmap, dev);
1011− state->bus_off = to_p2p_pgmap(state->pgmap)->bus_offset;
1107+ struct pci_p2pdma_pagemap *p2p_pgmap = to_p2p_pgmap(page_pgmap(page));
1108+
1109+ if (state->mem == p2p_pgmap->mem)
1110+ return;
1111+
1112+ state->mem = p2p_pgmap->mem;
1113+ state->map = pci_p2pdma_map_type(p2p_pgmap->mem, dev);
10121114 }
10131115
10141116 /**
drivers/s390/cio/vfio_ccw_ops.c+7−40
@@ -313,10 +313,12 @@ static int vfio_ccw_mdev_get_device_info(struct vfio_ccw_private *private, · 312 unmodified lines
313313 return 0;
314314 }
315315
316−static int vfio_ccw_mdev_get_region_info(struct vfio_ccw_private *private,
317− struct vfio_region_info *info,
318− unsigned long arg)
316+static int vfio_ccw_mdev_ioctl_get_region_info(struct vfio_device *vdev,
317+ struct vfio_region_info *info,
318+ struct vfio_info_cap *caps)
319319 {
320+ struct vfio_ccw_private *private =
321+ container_of(vdev, struct vfio_ccw_private, vdev);
320322 int i;
321323
322324 switch (info->index) {
@@ -328,7 +330,6 @@ static int vfio_ccw_mdev_get_region_info(struct vfio_ccw_private *private, · 5 unmodified lines
328330 return 0;
329331 default: /* all other regions are handled via capability chain */
330332 {
331− struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
332333 struct vfio_region_info_cap_type cap_type = {
333334 .header.id = VFIO_REGION_INFO_CAP_TYPE,
334335 .header.version = 1 };
@@ -351,27 +352,10 @@ static int vfio_ccw_mdev_get_region_info(struct vfio_ccw_private *private, · 16 unmodified lines
351352 cap_type.type = private->region[i].type;
352353 cap_type.subtype = private->region[i].subtype;
353354
354− ret = vfio_info_add_capability(&caps, &cap_type.header,
355+ ret = vfio_info_add_capability(caps, &cap_type.header,
355356 sizeof(cap_type));
356357 if (ret)
357358 return ret;
358−
359− info->flags |= VFIO_REGION_INFO_FLAG_CAPS;
360− if (info->argsz < sizeof(*info) + caps.size) {
361− info->argsz = sizeof(*info) + caps.size;
362− info->cap_offset = 0;
363− } else {
364− vfio_info_cap_shift(&caps, sizeof(*info));
365− if (copy_to_user((void __user *)arg + sizeof(*info),
366− caps.buf, caps.size)) {
367− kfree(caps.buf);
368− return -EFAULT;
369− }
370− info->cap_offset = sizeof(*info);
371− }
372−
373− kfree(caps.buf);
374−
375359 }
376360 }
377361 return 0;
@@ -532,24 +516,6 @@ static ssize_t vfio_ccw_mdev_ioctl(struct vfio_device *vdev, · 154 unmodified lines
532516
533517 return copy_to_user((void __user *)arg, &info, minsz) ? -EFAULT : 0;
534518 }
535− case VFIO_DEVICE_GET_REGION_INFO:
536− {
537− struct vfio_region_info info;
538−
539− minsz = offsetofend(struct vfio_region_info, offset);
540−
541− if (copy_from_user(&info, (void __user *)arg, minsz))
542− return -EFAULT;
543−
544− if (info.argsz < minsz)
545− return -EINVAL;
546−
547− ret = vfio_ccw_mdev_get_region_info(private, &info, arg);
548− if (ret)
549− return ret;
550−
551− return copy_to_user((void __user *)arg, &info, minsz) ? -EFAULT : 0;
552− }
553519 case VFIO_DEVICE_GET_IRQ_INFO:
554520 {
555521 struct vfio_irq_info info;
@@ -627,6 +593,7 @@ static const struct vfio_device_ops vfio_ccw_dev_ops = { · 71 unmodified lines
627593 .read = vfio_ccw_mdev_read,
628594 .write = vfio_ccw_mdev_write,
629595 .ioctl = vfio_ccw_mdev_ioctl,
596+ .get_region_info_caps = vfio_ccw_mdev_ioctl_get_region_info,
630597 .request = vfio_ccw_mdev_request,
631598 .dma_unmap = vfio_ccw_dma_unmap,
632599 .bind_iommufd = vfio_iommufd_emulated_bind,
drivers/vfio/cdx/main.c+11−18
@@ -129,28 +129,22 @@ static int vfio_cdx_ioctl_get_info(struct vfio_cdx_device *vdev, · 128 unmodified lines
129129 return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;
130130 }
131131
132−static int vfio_cdx_ioctl_get_region_info(struct vfio_cdx_device *vdev,
133− struct vfio_region_info __user *arg)
132+static int vfio_cdx_ioctl_get_region_info(struct vfio_device *core_vdev,
133+ struct vfio_region_info *info,
134+ struct vfio_info_cap *caps)
134135 {
135− unsigned long minsz = offsetofend(struct vfio_region_info, offset);
136+ struct vfio_cdx_device *vdev =
137+ container_of(core_vdev, struct vfio_cdx_device, vdev);
136138 struct cdx_device *cdx_dev = to_cdx_device(vdev->vdev.dev);
137− struct vfio_region_info info;
138−
139− if (copy_from_user(&info, arg, minsz))
140− return -EFAULT;
141139
142− if (info.argsz < minsz)
143− return -EINVAL;
144−
145− if (info.index >= cdx_dev->res_count)
140+ if (info->index >= cdx_dev->res_count)
146141 return -EINVAL;
147142
148143 /* map offset to the physical address */
149− info.offset = vfio_cdx_index_to_offset(info.index);
150− info.size = vdev->regions[info.index].size;
151− info.flags = vdev->regions[info.index].flags;
152−
153− return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;
144+ info->offset = vfio_cdx_index_to_offset(info->index);
145+ info->size = vdev->regions[info->index].size;
146+ info->flags = vdev->regions[info->index].flags;
147+ return 0;
154148 }
155149
156150 static int vfio_cdx_ioctl_get_irq_info(struct vfio_cdx_device *vdev,
@@ -219,8 +213,6 @@ static long vfio_cdx_ioctl(struct vfio_device *core_vdev, · 62 unmodified lines
219213 switch (cmd) {
220214 case VFIO_DEVICE_GET_INFO:
221215 return vfio_cdx_ioctl_get_info(vdev, uarg);
222− case VFIO_DEVICE_GET_REGION_INFO:
223− return vfio_cdx_ioctl_get_region_info(vdev, uarg);
224216 case VFIO_DEVICE_GET_IRQ_INFO:
225217 return vfio_cdx_ioctl_get_irq_info(vdev, uarg);
226218 case VFIO_DEVICE_SET_IRQS:
@@ -284,6 +276,7 @@ static const struct vfio_device_ops vfio_cdx_ops = { · 57 unmodified lines
284276 .open_device = vfio_cdx_open_device,
285277 .close_device = vfio_cdx_close_device,
286278 .ioctl = vfio_cdx_ioctl,
279+ .get_region_info_caps = vfio_cdx_ioctl_get_region_info,
287280 .device_feature = vfio_cdx_ioctl_feature,
288281 .mmap = vfio_cdx_mmap,
289282 .bind_iommufd = vfio_iommufd_physical_bind,
drivers/vfio/device_cdev.c+1−1
@@ -99,7 +99,7 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df, · 98 unmodified lines
9999 return ret;
100100 if (user_size < minsz)
101101 return -EINVAL;
102− ret = copy_struct_from_user(&bind, minsz, arg, user_size);
102+ ret = copy_struct_from_user(&bind, sizeof(bind), arg, user_size);
103103 if (ret)
104104 return ret;
105105
drivers/vfio/fsl-mc/vfio_fsl_mc.c+19−24
@@ -117,6 +117,24 @@ static void vfio_fsl_mc_close_device(struct vfio_device *core_vdev) · 116 unmodified lines
117117 fsl_mc_cleanup_irq_pool(mc_cont);
118118 }
119119
120+static int vfio_fsl_mc_ioctl_get_region_info(struct vfio_device *core_vdev,
121+ struct vfio_region_info *info,
122+ struct vfio_info_cap *caps)
123+{
124+ struct vfio_fsl_mc_device *vdev =
125+ container_of(core_vdev, struct vfio_fsl_mc_device, vdev);
126+ struct fsl_mc_device *mc_dev = vdev->mc_dev;
127+
128+ if (info->index >= mc_dev->obj_desc.region_count)
129+ return -EINVAL;
130+
131+ /* map offset to the physical address */
132+ info->offset = VFIO_FSL_MC_INDEX_TO_OFFSET(info->index);
133+ info->size = vdev->regions[info->index].size;
134+ info->flags = vdev->regions[info->index].flags;
135+ return 0;
136+}
137+
120138 static long vfio_fsl_mc_ioctl(struct vfio_device *core_vdev,
121139 unsigned int cmd, unsigned long arg)
122140 {
@@ -149,30 +167,6 @@ static long vfio_fsl_mc_ioctl(struct vfio_device *core_vdev, · 26 unmodified lines
149167 return copy_to_user((void __user *)arg, &info, minsz) ?
150168 -EFAULT : 0;
151169 }
152− case VFIO_DEVICE_GET_REGION_INFO:
153− {
154− struct vfio_region_info info;
155−
156− minsz = offsetofend(struct vfio_region_info, offset);
157−
158− if (copy_from_user(&info, (void __user *)arg, minsz))
159− return -EFAULT;
160−
161− if (info.argsz < minsz)
162− return -EINVAL;
163−
164− if (info.index >= mc_dev->obj_desc.region_count)
165− return -EINVAL;
166−
167− /* map offset to the physical address */
168− info.offset = VFIO_FSL_MC_INDEX_TO_OFFSET(info.index);
169− info.size = vdev->regions[info.index].size;
170− info.flags = vdev->regions[info.index].flags;
171−
172− if (copy_to_user((void __user *)arg, &info, minsz))
173− return -EFAULT;
174− return 0;
175− }
176170 case VFIO_DEVICE_GET_IRQ_INFO:
177171 {
178172 struct vfio_irq_info info;
@@ -589,6 +583,7 @@ static const struct vfio_device_ops vfio_fsl_mc_ops = { · 410 unmodified lines
589583 .open_device = vfio_fsl_mc_open_device,
590584 .close_device = vfio_fsl_mc_close_device,
591585 .ioctl = vfio_fsl_mc_ioctl,
586+ .get_region_info_caps = vfio_fsl_mc_ioctl_get_region_info,
592587 .read = vfio_fsl_mc_read,
593588 .write = vfio_fsl_mc_write,
594589 .mmap = vfio_fsl_mc_mmap,
drivers/vfio/pci/Kconfig+3−0
@@ -55,6 +55,9 @@ config VFIO_PCI_ZDEV_KVM · 54 unmodified lines
5555
5656 To enable s390x KVM vfio-pci extensions, say Y.
5757
58+config VFIO_PCI_DMABUF
59+ def_bool y if VFIO_PCI_CORE && PCI_P2PDMA && DMA_SHARED_BUFFER
60+
5861 source "drivers/vfio/pci/mlx5/Kconfig"
5962
6063 source "drivers/vfio/pci/hisilicon/Kconfig"
drivers/vfio/pci/Makefile+1−0
@@ -2,6 +2,7 @@ · 1 unmodified line
22
33 vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
44 vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
5+vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o
56 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
67
78 vfio-pci-y := vfio_pci.o
drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c+108−63
@@ -125,9 +125,25 @@ static int qm_get_cqc(struct hisi_qm *qm, u64 *addr) · 124 unmodified lines
125125 return 0;
126126 }
127127
128+static void qm_xqc_reg_offsets(struct hisi_qm *qm,
129+ u32 *eqc_addr, u32 *aeqc_addr)
130+{
131+ struct hisi_acc_vf_core_device *hisi_acc_vdev =
132+ container_of(qm, struct hisi_acc_vf_core_device, vf_qm);
133+
134+ if (hisi_acc_vdev->drv_mode == HW_ACC_MIG_VF_CTRL) {
135+ *eqc_addr = QM_EQC_VF_DW0;
136+ *aeqc_addr = QM_AEQC_VF_DW0;
137+ } else {
138+ *eqc_addr = QM_EQC_PF_DW0;
139+ *aeqc_addr = QM_AEQC_PF_DW0;
140+ }
141+}
142+
128143 static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data)
129144 {
130145 struct device *dev = &qm->pdev->dev;
146+ u32 eqc_addr, aeqc_addr;
131147 int ret;
132148
133149 ret = qm_read_regs(qm, QM_VF_AEQ_INT_MASK, &vf_data->aeq_int_mask, 1);
@@ -167,15 +183,16 @@ static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data) · 33 unmodified lines
167183 return ret;
168184 }
169185
186+ qm_xqc_reg_offsets(qm, &eqc_addr, &aeqc_addr);
170187 /* QM_EQC_DW has 7 regs */
171− ret = qm_read_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
188+ ret = qm_read_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
172189 if (ret) {
173190 dev_err(dev, "failed to read QM_EQC_DW\n");
174191 return ret;
175192 }
176193
177194 /* QM_AEQC_DW has 7 regs */
178− ret = qm_read_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, 7);
195+ ret = qm_read_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
179196 if (ret) {
180197 dev_err(dev, "failed to read QM_AEQC_DW\n");
181198 return ret;
@@ -187,6 +204,7 @@ static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data) · 5 unmodified lines
187204 static int qm_set_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data)
188205 {
189206 struct device *dev = &qm->pdev->dev;
207+ u32 eqc_addr, aeqc_addr;
190208 int ret;
191209
192210 /* Check VF state */
@@ -239,15 +257,16 @@ static int qm_set_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data) · 46 unmodified lines
239257 return ret;
240258 }
241259
260+ qm_xqc_reg_offsets(qm, &eqc_addr, &aeqc_addr);
242261 /* QM_EQC_DW has 7 regs */
243− ret = qm_write_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
262+ ret = qm_write_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
244263 if (ret) {
245264 dev_err(dev, "failed to write QM_EQC_DW\n");
246265 return ret;
247266 }
248267
249268 /* QM_AEQC_DW has 7 regs */
250− ret = qm_write_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, 7);
269+ ret = qm_write_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
251270 if (ret) {
252271 dev_err(dev, "failed to write QM_AEQC_DW\n");
253272 return ret;
@@ -1186,34 +1205,52 @@ static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device *hisi_acc_vdev) · 932 unmodified lines
11861205 {
11871206 struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
11881207 struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
1208+ struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
11891209 struct pci_dev *vf_dev = vdev->pdev;
1210+ u32 val;
11901211
1191− /*
1192− * ACC VF dev BAR2 region consists of both functional register space
1193− * and migration control register space. For migration to work, we
1194− * need access to both. Hence, we map the entire BAR2 region here.
1195− * But unnecessarily exposing the migration BAR region to the Guest
1196− * has the potential to prevent/corrupt the Guest migration. Hence,
1197− * we restrict access to the migration control space from
1198− * Guest(Please see mmap/ioctl/read/write override functions).
1199− *
1200− * Please note that it is OK to expose the entire VF BAR if migration
1201− * is not supported or required as this cannot affect the ACC PF
1202− * configurations.
1203− *
1204− * Also the HiSilicon ACC VF devices supported by this driver on
1205− * HiSilicon hardware platforms are integrated end point devices
1206− * and the platform lacks the capability to perform any PCIe P2P
1207− * between these devices.
1208− */
1212+ val = readl(pf_qm->io_base + QM_MIG_REGION_SEL);
1213+ if (pf_qm->ver > QM_HW_V3 && (val & QM_MIG_REGION_EN))
1214+ hisi_acc_vdev->drv_mode = HW_ACC_MIG_PF_CTRL;
1215+ else
1216+ hisi_acc_vdev->drv_mode = HW_ACC_MIG_VF_CTRL;
12091217
1210− vf_qm->io_base =
1211− ioremap(pci_resource_start(vf_dev, VFIO_PCI_BAR2_REGION_INDEX),
1212− pci_resource_len(vf_dev, VFIO_PCI_BAR2_REGION_INDEX));
1213− if (!vf_qm->io_base)
1214− return -EIO;
1218+ if (hisi_acc_vdev->drv_mode == HW_ACC_MIG_PF_CTRL) {
1219+ /*
1220+ * On hardware platforms greater than QM_HW_V3, the migration function
1221+ * register is placed in the BAR2 configuration region of the PF,
1222+ * and each VF device occupies 8KB of configuration space.
1223+ */
1224+ vf_qm->io_base = pf_qm->io_base + QM_MIG_REGION_OFFSET +
1225+ hisi_acc_vdev->vf_id * QM_MIG_REGION_SIZE;
1226+ } else {
1227+ /*
1228+ * ACC VF dev BAR2 region consists of both functional register space
1229+ * and migration control register space. For migration to work, we
1230+ * need access to both. Hence, we map the entire BAR2 region here.
1231+ * But unnecessarily exposing the migration BAR region to the Guest
1232+ * has the potential to prevent/corrupt the Guest migration. Hence,
1233+ * we restrict access to the migration control space from
1234+ * Guest(Please see mmap/ioctl/read/write override functions).
1235+ *
1236+ * Please note that it is OK to expose the entire VF BAR if migration
1237+ * is not supported or required as this cannot affect the ACC PF
1238+ * configurations.
1239+ *
1240+ * Also the HiSilicon ACC VF devices supported by this driver on
1241+ * HiSilicon hardware platforms are integrated end point devices
1242+ * and the platform lacks the capability to perform any PCIe P2P
1243+ * between these devices.
1244+ */
12151245
1246+ vf_qm->io_base =
1247+ ioremap(pci_resource_start(vf_dev, VFIO_PCI_BAR2_REGION_INDEX),
1248+ pci_resource_len(vf_dev, VFIO_PCI_BAR2_REGION_INDEX));
1249+ if (!vf_qm->io_base)
1250+ return -EIO;
1251+ }
12161252 vf_qm->fun_type = QM_HW_VF;
1253+ vf_qm->ver = pf_qm->ver;
12171254 vf_qm->pdev = vf_dev;
12181255 mutex_init(&vf_qm->mailbox_lock);
12191256
@@ -1250,6 +1287,28 @@ static struct hisi_qm *hisi_acc_get_pf_qm(struct pci_dev *pdev) · 30 unmodified lines
12501287 return !IS_ERR(pf_qm) ? pf_qm : NULL;
12511288 }
12521289
1290+static size_t hisi_acc_get_resource_len(struct vfio_pci_core_device *vdev,
1291+ unsigned int index)
1292+{
1293+ struct hisi_acc_vf_core_device *hisi_acc_vdev =
1294+ hisi_acc_drvdata(vdev->pdev);
1295+
1296+ /*
1297+ * On the old HW_ACC_MIG_VF_CTRL mode device, the ACC VF device
1298+ * BAR2 region encompasses both functional register space
1299+ * and migration control register space.
1300+ * only the functional region should be report to Guest.
1301+ */
1302+ if (hisi_acc_vdev->drv_mode == HW_ACC_MIG_VF_CTRL)
1303+ return (pci_resource_len(vdev->pdev, index) >> 1);
1304+ /*
1305+ * On the new HW device, the migration control register
1306+ * has been moved to the PF device BAR2 region.
1307+ * The VF device BAR2 is entirely functional register space.
1308+ */
1309+ return pci_resource_len(vdev->pdev, index);
1310+}
1311+
12531312 static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev,
12541313 size_t count, loff_t *ppos,
12551314 size_t *new_count)
@@ -1260,8 +1319,9 @@ static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev, · 4 unmodified lines
12601319
12611320 if (index == VFIO_PCI_BAR2_REGION_INDEX) {
12621321 loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
1263− resource_size_t end = pci_resource_len(vdev->pdev, index) / 2;
1322+ resource_size_t end;
12641323
1324+ end = hisi_acc_get_resource_len(vdev, index);
12651325 /* Check if access is for migration control region */
12661326 if (pos >= end)
12671327 return -EINVAL;
@@ -1282,8 +1342,9 @@ static int hisi_acc_vfio_pci_mmap(struct vfio_device *core_vdev, · 14 unmodified lines
12821342 index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
12831343 if (index == VFIO_PCI_BAR2_REGION_INDEX) {
12841344 u64 req_len, pgoff, req_start;
1285− resource_size_t end = pci_resource_len(vdev->pdev, index) / 2;
1345+ resource_size_t end;
12861346
1347+ end = hisi_acc_get_resource_len(vdev, index);
12871348 req_len = vma->vm_end - vma->vm_start;
12881349 pgoff = vma->vm_pgoff &
12891350 ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
@@ -1324,43 +1385,23 @@ static ssize_t hisi_acc_vfio_pci_read(struct vfio_device *core_vdev, · 34 unmodified lines
13241385 return vfio_pci_core_read(core_vdev, buf, new_count, ppos);
13251386 }
13261387
1327−static long hisi_acc_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
1328− unsigned long arg)
1388+static int hisi_acc_vfio_ioctl_get_region(struct vfio_device *core_vdev,
1389+ struct vfio_region_info *info,
1390+ struct vfio_info_cap *caps)
13291391 {
1330− if (cmd == VFIO_DEVICE_GET_REGION_INFO) {
1331− struct vfio_pci_core_device *vdev =
1332− container_of(core_vdev, struct vfio_pci_core_device, vdev);
1333− struct pci_dev *pdev = vdev->pdev;
1334− struct vfio_region_info info;
1335− unsigned long minsz;
1336−
1337− minsz = offsetofend(struct vfio_region_info, offset);
1338−
1339− if (copy_from_user(&info, (void __user *)arg, minsz))
1340− return -EFAULT;
1341−
1342− if (info.argsz < minsz)
1343− return -EINVAL;
1392+ struct vfio_pci_core_device *vdev =
1393+ container_of(core_vdev, struct vfio_pci_core_device, vdev);
13441394
1345− if (info.index == VFIO_PCI_BAR2_REGION_INDEX) {
1346− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1395+ if (info->index != VFIO_PCI_BAR2_REGION_INDEX)
1396+ return vfio_pci_ioctl_get_region_info(core_vdev, info, caps);
13471397
1348− /*
1349− * ACC VF dev BAR2 region consists of both functional
1350− * register space and migration control register space.
1351− * Report only the functional region to Guest.
1352− */
1353− info.size = pci_resource_len(pdev, info.index) / 2;
1398+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
13541399
1355− info.flags = VFIO_REGION_INFO_FLAG_READ |
1356− VFIO_REGION_INFO_FLAG_WRITE |
1357− VFIO_REGION_INFO_FLAG_MMAP;
1400+ info->size = hisi_acc_get_resource_len(vdev, info->index);
13581401
1359− return copy_to_user((void __user *)arg, &info, minsz) ?
1360− -EFAULT : 0;
1361− }
1362− }
1363− return vfio_pci_core_ioctl(core_vdev, cmd, arg);
1402+ info->flags = VFIO_REGION_INFO_FLAG_READ | VFIO_REGION_INFO_FLAG_WRITE |
1403+ VFIO_REGION_INFO_FLAG_MMAP;
1404+ return 0;
13641405 }
13651406
13661407 static int hisi_acc_vf_debug_check(struct seq_file *seq, struct vfio_device *vdev)
@@ -1521,7 +1562,8 @@ static void hisi_acc_vfio_pci_close_device(struct vfio_device *core_vdev) · 154 unmodified lines
15211562 hisi_acc_vf_disable_fds(hisi_acc_vdev);
15221563 mutex_lock(&hisi_acc_vdev->open_mutex);
15231564 hisi_acc_vdev->dev_opened = false;
1524− iounmap(vf_qm->io_base);
1565+ if (hisi_acc_vdev->drv_mode == HW_ACC_MIG_VF_CTRL)
1566+ iounmap(vf_qm->io_base);
15251567 mutex_unlock(&hisi_acc_vdev->open_mutex);
15261568 vfio_pci_core_close_device(core_vdev);
15271569 }
@@ -1557,13 +1599,15 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_migrn_ops = { · 29 unmodified lines
15571599 .release = vfio_pci_core_release_dev,
15581600 .open_device = hisi_acc_vfio_pci_open_device,
15591601 .close_device = hisi_acc_vfio_pci_close_device,
1560− .ioctl = hisi_acc_vfio_pci_ioctl,
1602+ .ioctl = vfio_pci_core_ioctl,
1603+ .get_region_info_caps = hisi_acc_vfio_ioctl_get_region,
15611604 .device_feature = vfio_pci_core_ioctl_feature,
15621605 .read = hisi_acc_vfio_pci_read,
15631606 .write = hisi_acc_vfio_pci_write,
15641607 .mmap = hisi_acc_vfio_pci_mmap,
15651608 .request = vfio_pci_core_request,
15661609 .match = vfio_pci_core_match,
1610+ .match_token_uuid = vfio_pci_core_match_token_uuid,
15671611 .bind_iommufd = vfio_iommufd_physical_bind,
15681612 .unbind_iommufd = vfio_iommufd_physical_unbind,
15691613 .attach_ioas = vfio_iommufd_physical_attach_ioas,
@@ -1577,6 +1621,7 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_ops = { · 7 unmodified lines
15771621 .open_device = hisi_acc_vfio_pci_open_device,
15781622 .close_device = vfio_pci_core_close_device,
15791623 .ioctl = vfio_pci_core_ioctl,
1624+ .get_region_info_caps = vfio_pci_ioctl_get_region_info,
15801625 .device_feature = vfio_pci_core_ioctl_feature,
15811626 .read = vfio_pci_core_read,
15821627 .write = vfio_pci_core_write,
drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h+21−2
@@ -50,8 +50,10 @@ · 49 unmodified lines
5050 #define QM_QUE_ISO_CFG_V 0x0030
5151 #define QM_PAGE_SIZE 0x0034
5252
53−#define QM_EQC_DW0 0X8000
54−#define QM_AEQC_DW0 0X8020
53+#define QM_EQC_VF_DW0 0X8000
54+#define QM_AEQC_VF_DW0 0X8020
55+#define QM_EQC_PF_DW0 0x1c00
56+#define QM_AEQC_PF_DW0 0x1c20
5557
5658 #define ACC_DRV_MAJOR_VER 1
5759 #define ACC_DRV_MINOR_VER 0
@@ -59,6 +61,22 @@ · 1 unmodified line
5961 #define ACC_DEV_MAGIC_V1 0XCDCDCDCDFEEDAACC
6062 #define ACC_DEV_MAGIC_V2 0xAACCFEEDDECADEDE
6163
64+#define QM_MIG_REGION_OFFSET 0x180000
65+#define QM_MIG_REGION_SIZE 0x2000
66+
67+/**
68+ * On HW_ACC_MIG_VF_CTRL mode, the configuration domain supporting live
69+ * migration functionality is located in the latter 32KB of the VF's BAR2.
70+ * The Guest is only provided with the first 32KB of the VF's BAR2.
71+ * On HW_ACC_MIG_PF_CTRL mode, the configuration domain supporting live
72+ * migration functionality is located in the PF's BAR2, and the entire 64KB
73+ * of the VF's BAR2 is allocated to the Guest.
74+ */
75+enum hw_drv_mode {
76+ HW_ACC_MIG_VF_CTRL = 0,
77+ HW_ACC_MIG_PF_CTRL,
78+};
79+
6280 struct acc_vf_data {
6381 #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state)
6482 /* QM match information */
@@ -125,6 +143,7 @@ struct hisi_acc_vf_core_device { · 60 unmodified lines
125143 struct pci_dev *vf_dev;
126144 struct hisi_qm *pf_qm;
127145 struct hisi_qm vf_qm;
146+ enum hw_drv_mode drv_mode;
128147 /*
129148 * vf_qm_state represents the QM_VF_STATE register value.
130149 * It is set by Guest driver for the ACC VF dev indicating
drivers/vfio/pci/mlx5/main.c+1−0
@@ -1366,6 +1366,7 @@ static const struct vfio_device_ops mlx5vf_pci_ops = { · 1365 unmodified lines
13661366 .open_device = mlx5vf_pci_open_device,
13671367 .close_device = mlx5vf_pci_close_device,
13681368 .ioctl = vfio_pci_core_ioctl,
1369+ .get_region_info_caps = vfio_pci_ioctl_get_region_info,
13691370 .device_feature = vfio_pci_core_ioctl_feature,
13701371 .read = vfio_pci_core_read,
13711372 .write = vfio_pci_core_write,
drivers/vfio/pci/nvgrace-gpu/main.c+255−87
@@ -7,6 +7,8 @@ · 6 unmodified lines
77 #include <linux/vfio_pci_core.h>
88 #include <linux/delay.h>
99 #include <linux/jiffies.h>
10+#include <linux/pci-p2pdma.h>
11+#include <linux/pm_runtime.h>
1012
1113 /*
1214 * The device memory usable to the workloads running in the VM is cached
@@ -58,6 +60,8 @@ struct nvgrace_gpu_pci_core_device { · 45 unmodified lines
5860 /* Lock to control device memory kernel mapping */
5961 struct mutex remap_lock;
6062 bool has_mig_hw_bug;
63+ /* GPU has just been reset */
64+ bool reset_done;
6165 };
6266
6367 static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device *core_vdev)
@@ -102,6 +106,19 @@ static int nvgrace_gpu_open_device(struct vfio_device *core_vdev) · 38 unmodified lines
102106 mutex_init(&nvdev->remap_lock);
103107 }
104108
109+ /*
110+ * GPU readiness is checked by reading the BAR0 registers.
111+ *
112+ * ioremap BAR0 to ensure that the BAR0 mapping is present before
113+ * register reads on first fault before establishing any GPU
114+ * memory mapping.
115+ */
116+ ret = vfio_pci_core_setup_barmap(vdev, 0);
117+ if (ret) {
118+ vfio_pci_core_disable(vdev);
119+ return ret;
120+ }
121+
105122 vfio_pci_core_finish_enable(vdev);
106123
107124 return 0;
@@ -130,6 +147,106 @@ static void nvgrace_gpu_close_device(struct vfio_device *core_vdev) · 22 unmodified lines
130147 vfio_pci_core_close_device(core_vdev);
131148 }
132149
150+static int nvgrace_gpu_wait_device_ready(void __iomem *io)
151+{
152+ unsigned long timeout = jiffies + msecs_to_jiffies(POLL_TIMEOUT_MS);
153+
154+ do {
155+ if ((ioread32(io + C2C_LINK_BAR0_OFFSET) == STATUS_READY) &&
156+ (ioread32(io + HBM_TRAINING_BAR0_OFFSET) == STATUS_READY))
157+ return 0;
158+ msleep(POLL_QUANTUM_MS);
159+ } while (!time_after(jiffies, timeout));
160+
161+ return -ETIME;
162+}
163+
164+/*
165+ * If the GPU memory is accessed by the CPU while the GPU is not ready
166+ * after reset, it can cause harmless corrected RAS events to be logged.
167+ * Make sure the GPU is ready before establishing the mappings.
168+ */
169+static int
170+nvgrace_gpu_check_device_ready(struct nvgrace_gpu_pci_core_device *nvdev)
171+{
172+ struct vfio_pci_core_device *vdev = &nvdev->core_device;
173+ int ret;
174+
175+ lockdep_assert_held_read(&vdev->memory_lock);
176+
177+ if (!nvdev->reset_done)
178+ return 0;
179+
180+ if (!__vfio_pci_memory_enabled(vdev))
181+ return -EIO;
182+
183+ ret = nvgrace_gpu_wait_device_ready(vdev->barmap[0]);
184+ if (ret)
185+ return ret;
186+
187+ nvdev->reset_done = false;
188+
189+ return 0;
190+}
191+
192+static unsigned long addr_to_pgoff(struct vm_area_struct *vma,
193+ unsigned long addr)
194+{
195+ u64 pgoff = vma->vm_pgoff &
196+ ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
197+
198+ return ((addr - vma->vm_start) >> PAGE_SHIFT) + pgoff;
199+}
200+
201+static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf,
202+ unsigned int order)
203+{
204+ struct vm_area_struct *vma = vmf->vma;
205+ struct nvgrace_gpu_pci_core_device *nvdev = vma->vm_private_data;
206+ struct vfio_pci_core_device *vdev = &nvdev->core_device;
207+ unsigned int index =
208+ vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
209+ vm_fault_t ret = VM_FAULT_FALLBACK;
210+ struct mem_region *memregion;
211+ unsigned long pfn, addr;
212+
213+ memregion = nvgrace_gpu_memregion(index, nvdev);
214+ if (!memregion)
215+ return VM_FAULT_SIGBUS;
216+
217+ addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
218+ pfn = PHYS_PFN(memregion->memphys) + addr_to_pgoff(vma, addr);
219+
220+ if (is_aligned_for_order(vma, addr, pfn, order)) {
221+ scoped_guard(rwsem_read, &vdev->memory_lock) {
222+ if (vdev->pm_runtime_engaged ||
223+ nvgrace_gpu_check_device_ready(nvdev))
224+ return VM_FAULT_SIGBUS;
225+
226+ ret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order);
227+ }
228+ }
229+
230+ dev_dbg_ratelimited(&vdev->pdev->dev,
231+ "%s order = %d pfn 0x%lx: 0x%x\n",
232+ __func__, order, pfn,
233+ (unsigned int)ret);
234+
235+ return ret;
236+}
237+
238+static vm_fault_t nvgrace_gpu_vfio_pci_fault(struct vm_fault *vmf)
239+{
240+ return nvgrace_gpu_vfio_pci_huge_fault(vmf, 0);
241+}
242+
243+static const struct vm_operations_struct nvgrace_gpu_vfio_pci_mmap_ops = {
244+ .fault = nvgrace_gpu_vfio_pci_fault,
245+#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP
246+ .huge_fault = nvgrace_gpu_vfio_pci_huge_fault,
247+#endif
248+};
249+
133250 static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
134251 struct vm_area_struct *vma)
135252 {
@@ -137,10 +254,8 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev, · 1 unmodified line
137254 container_of(core_vdev, struct nvgrace_gpu_pci_core_device,
138255 core_device.vdev);
139256 struct mem_region *memregion;
140− unsigned long start_pfn;
141257 u64 req_len, pgoff, end;
142258 unsigned int index;
143− int ret = 0;
144259
145260 index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
146261
@@ -157,17 +272,18 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev, · 10 unmodified lines
157272 ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
158273
159274 if (check_sub_overflow(vma->vm_end, vma->vm_start, &req_len) ||
160− check_add_overflow(PHYS_PFN(memregion->memphys), pgoff, &start_pfn) ||
161275 check_add_overflow(PFN_PHYS(pgoff), req_len, &end))
162276 return -EOVERFLOW;
163277
164278 /*
165− * Check that the mapping request does not go beyond available device
166− * memory size
279+ * Check that the mapping request does not go beyond the exposed
280+ * device memory size.
167281 */
168282 if (end > memregion->memlength)
169283 return -EINVAL;
170284
285+ vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
286+
171287 /*
172288 * The carved out region of the device memory needs the NORMAL_NC
173289 * property. Communicate as such to the hypervisor.
@@ -184,56 +300,31 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev, · 10 unmodified lines
184300 vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
185301 }
186302
187− /*
188− * Perform a PFN map to the memory and back the device BAR by the
189− * GPU memory.
190− *
191− * The available GPU memory size may not be power-of-2 aligned. The
192− * remainder is only backed by vfio_device_ops read/write handlers.
193− *
194− * During device reset, the GPU is safely disconnected to the CPU
195− * and access to the BAR will be immediately returned preventing
196− * machine check.
197− */
198− ret = remap_pfn_range(vma, vma->vm_start, start_pfn,
199− req_len, vma->vm_page_prot);
200− if (ret)
201− return ret;
202−
203− vma->vm_pgoff = start_pfn;
303+ vma->vm_ops = &nvgrace_gpu_vfio_pci_mmap_ops;
304+ vma->vm_private_data = nvdev;
204305
205306 return 0;
206307 }
207308
208−static long
209−nvgrace_gpu_ioctl_get_region_info(struct vfio_device *core_vdev,
210− unsigned long arg)
309+static int nvgrace_gpu_ioctl_get_region_info(struct vfio_device *core_vdev,
310+ struct vfio_region_info *info,
311+ struct vfio_info_cap *caps)
211312 {
212313 struct nvgrace_gpu_pci_core_device *nvdev =
213314 container_of(core_vdev, struct nvgrace_gpu_pci_core_device,
214315 core_device.vdev);
215− unsigned long minsz = offsetofend(struct vfio_region_info, offset);
216− struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
217316 struct vfio_region_info_cap_sparse_mmap *sparse;
218− struct vfio_region_info info;
219317 struct mem_region *memregion;
220318 u32 size;
221319 int ret;
222320
223− if (copy_from_user(&info, (void __user *)arg, minsz))
224− return -EFAULT;
225−
226− if (info.argsz < minsz)
227− return -EINVAL;
228−
229321 /*
230322 * Request to determine the BAR region information. Send the
231323 * GPU memory information.
232324 */
233− memregion = nvgrace_gpu_memregion(info.index, nvdev);
325+ memregion = nvgrace_gpu_memregion(info->index, nvdev);
234326 if (!memregion)
235− return vfio_pci_core_ioctl(core_vdev,
236− VFIO_DEVICE_GET_REGION_INFO, arg);
327+ return vfio_pci_ioctl_get_region_info(core_vdev, info, caps);
237328
238329 size = struct_size(sparse, areas, 1);
239330
@@ -252,49 +343,28 @@ nvgrace_gpu_ioctl_get_region_info(struct vfio_device *core_vdev, · 12 unmodified lines
252343 sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
253344 sparse->header.version = 1;
254345
255− ret = vfio_info_add_capability(&caps, &sparse->header, size);
346+ ret = vfio_info_add_capability(caps, &sparse->header, size);
256347 kfree(sparse);
257348 if (ret)
258349 return ret;
259350
260− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
351+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
261352 /*
262353 * The region memory size may not be power-of-2 aligned.
263354 * Given that the memory is a BAR and may not be
264355 * aligned, roundup to the next power-of-2.
265356 */
266− info.size = memregion->bar_size;
267− info.flags = VFIO_REGION_INFO_FLAG_READ |
357+ info->size = memregion->bar_size;
358+ info->flags = VFIO_REGION_INFO_FLAG_READ |
268359 VFIO_REGION_INFO_FLAG_WRITE |
269360 VFIO_REGION_INFO_FLAG_MMAP;
270−
271− if (caps.size) {
272− info.flags |= VFIO_REGION_INFO_FLAG_CAPS;
273− if (info.argsz < sizeof(info) + caps.size) {
274− info.argsz = sizeof(info) + caps.size;
275− info.cap_offset = 0;
276− } else {
277− vfio_info_cap_shift(&caps, sizeof(info));
278− if (copy_to_user((void __user *)arg +
279− sizeof(info), caps.buf,
280− caps.size)) {
281− kfree(caps.buf);
282− return -EFAULT;
283− }
284− info.cap_offset = sizeof(info);
285− }
286− kfree(caps.buf);
287− }
288− return copy_to_user((void __user *)arg, &info, minsz) ?
289− -EFAULT : 0;
361+ return 0;
290362 }
291363
292364 static long nvgrace_gpu_ioctl(struct vfio_device *core_vdev,
293365 unsigned int cmd, unsigned long arg)
294366 {
295367 switch (cmd) {
296− case VFIO_DEVICE_GET_REGION_INFO:
297− return nvgrace_gpu_ioctl_get_region_info(core_vdev, arg);
298368 case VFIO_DEVICE_IOEVENTFD:
299369 return -ENOTTY;
300370 case VFIO_DEVICE_RESET:
@@ -510,6 +580,7 @@ static ssize_t · 209 unmodified lines
510580 nvgrace_gpu_read_mem(struct nvgrace_gpu_pci_core_device *nvdev,
511581 char __user *buf, size_t count, loff_t *ppos)
512582 {
583+ struct vfio_pci_core_device *vdev = &nvdev->core_device;
513584 u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
514585 unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
515586 struct mem_region *memregion;
@@ -536,9 +607,15 @@ nvgrace_gpu_read_mem(struct nvgrace_gpu_pci_core_device *nvdev, · 20 unmodified lines
536607 else
537608 mem_count = min(count, memregion->memlength - (size_t)offset);
538609
539− ret = nvgrace_gpu_map_and_read(nvdev, buf, mem_count, ppos);
540− if (ret)
541− return ret;
610+ scoped_guard(rwsem_read, &vdev->memory_lock) {
611+ ret = nvgrace_gpu_check_device_ready(nvdev);
612+ if (ret)
613+ return ret;
614+
615+ ret = nvgrace_gpu_map_and_read(nvdev, buf, mem_count, ppos);
616+ if (ret)
617+ return ret;
618+ }
542619
543620 /*
544621 * Only the device memory present on the hardware is mapped, which may
@@ -563,9 +640,16 @@ nvgrace_gpu_read(struct vfio_device *core_vdev, · 18 unmodified lines
563640 struct nvgrace_gpu_pci_core_device *nvdev =
564641 container_of(core_vdev, struct nvgrace_gpu_pci_core_device,
565642 core_device.vdev);
643+ struct vfio_pci_core_device *vdev = &nvdev->core_device;
644+ int ret;
566645
567− if (nvgrace_gpu_memregion(index, nvdev))
568− return nvgrace_gpu_read_mem(nvdev, buf, count, ppos);
646+ if (nvgrace_gpu_memregion(index, nvdev)) {
647+ if (pm_runtime_resume_and_get(&vdev->pdev->dev))
648+ return -EIO;
649+ ret = nvgrace_gpu_read_mem(nvdev, buf, count, ppos);
650+ pm_runtime_put(&vdev->pdev->dev);
651+ return ret;
652+ }
569653
570654 if (index == VFIO_PCI_CONFIG_REGION_INDEX)
571655 return nvgrace_gpu_read_config_emu(core_vdev, buf, count, ppos);
@@ -627,6 +711,7 @@ static ssize_t · 55 unmodified lines
627711 nvgrace_gpu_write_mem(struct nvgrace_gpu_pci_core_device *nvdev,
628712 size_t count, loff_t *ppos, const char __user *buf)
629713 {
714+ struct vfio_pci_core_device *vdev = &nvdev->core_device;
630715 unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
631716 u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
632717 struct mem_region *memregion;
@@ -656,9 +741,15 @@ nvgrace_gpu_write_mem(struct nvgrace_gpu_pci_core_device *nvdev, · 23 unmodified lines
656741 */
657742 mem_count = min(count, memregion->memlength - (size_t)offset);
658743
659− ret = nvgrace_gpu_map_and_write(nvdev, buf, mem_count, ppos);
660− if (ret)
661− return ret;
744+ scoped_guard(rwsem_read, &vdev->memory_lock) {
745+ ret = nvgrace_gpu_check_device_ready(nvdev);
746+ if (ret)
747+ return ret;
748+
749+ ret = nvgrace_gpu_map_and_write(nvdev, buf, mem_count, ppos);
750+ if (ret)
751+ return ret;
752+ }
662753
663754 exitfn:
664755 *ppos += count;
@@ -672,10 +763,17 @@ nvgrace_gpu_write(struct vfio_device *core_vdev, · 7 unmodified lines
672763 struct nvgrace_gpu_pci_core_device *nvdev =
673764 container_of(core_vdev, struct nvgrace_gpu_pci_core_device,
674765 core_device.vdev);
766+ struct vfio_pci_core_device *vdev = &nvdev->core_device;
675767 unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
768+ int ret;
676769
677− if (nvgrace_gpu_memregion(index, nvdev))
678− return nvgrace_gpu_write_mem(nvdev, count, ppos, buf);
770+ if (nvgrace_gpu_memregion(index, nvdev)) {
771+ if (pm_runtime_resume_and_get(&vdev->pdev->dev))
772+ return -EIO;
773+ ret = nvgrace_gpu_write_mem(nvdev, count, ppos, buf);
774+ pm_runtime_put(&vdev->pdev->dev);
775+ return ret;
776+ }
679777
680778 if (index == VFIO_PCI_CONFIG_REGION_INDEX)
681779 return nvgrace_gpu_write_config_emu(core_vdev, buf, count, ppos);
@@ -683,6 +781,50 @@ nvgrace_gpu_write(struct vfio_device *core_vdev, · 1 unmodified line
683781 return vfio_pci_core_write(core_vdev, buf, count, ppos);
684782 }
685783
784+static int nvgrace_get_dmabuf_phys(struct vfio_pci_core_device *core_vdev,
785+ struct p2pdma_provider **provider,
786+ unsigned int region_index,
787+ struct dma_buf_phys_vec *phys_vec,
788+ struct vfio_region_dma_range *dma_ranges,
789+ size_t nr_ranges)
790+{
791+ struct nvgrace_gpu_pci_core_device *nvdev = container_of(
792+ core_vdev, struct nvgrace_gpu_pci_core_device, core_device);
793+ struct pci_dev *pdev = core_vdev->pdev;
794+ struct mem_region *mem_region;
795+
796+ /*
797+ * if (nvdev->resmem.memlength && region_index == RESMEM_REGION_INDEX) {
798+ * The P2P properties of the non-BAR memory is the same as the
799+ * BAR memory, so just use the provider for index 0. Someday
800+ * when CXL gets P2P support we could create CXLish providers
801+ * for the non-BAR memory.
802+ * } else if (region_index == USEMEM_REGION_INDEX) {
803+ * This is actually cachable memory and isn't treated as P2P in
804+ * the chip. For now we have no way to push cachable memory
805+ * through everything and the Grace HW doesn't care what caching
806+ * attribute is programmed into the SMMU. So use BAR 0.
807+ * }
808+ */
809+ mem_region = nvgrace_gpu_memregion(region_index, nvdev);
810+ if (mem_region) {
811+ *provider = pcim_p2pdma_provider(pdev, 0);
812+ if (!*provider)
813+ return -EINVAL;
814+ return vfio_pci_core_fill_phys_vec(phys_vec, dma_ranges,
815+ nr_ranges,
816+ mem_region->memphys,
817+ mem_region->memlength);
818+ }
819+
820+ return vfio_pci_core_get_dmabuf_phys(core_vdev, provider, region_index,
821+ phys_vec, dma_ranges, nr_ranges);
822+}
823+
824+static const struct vfio_pci_device_ops nvgrace_gpu_pci_dev_ops = {
825+ .get_dmabuf_phys = nvgrace_get_dmabuf_phys,
826+};
827+
686828 static const struct vfio_device_ops nvgrace_gpu_pci_ops = {
687829 .name = "nvgrace-gpu-vfio-pci",
688830 .init = vfio_pci_core_init_dev,
@@ -690,6 +832,7 @@ static const struct vfio_device_ops nvgrace_gpu_pci_ops = { · 1 unmodified line
690832 .open_device = nvgrace_gpu_open_device,
691833 .close_device = nvgrace_gpu_close_device,
692834 .ioctl = nvgrace_gpu_ioctl,
835+ .get_region_info_caps = nvgrace_gpu_ioctl_get_region_info,
693836 .device_feature = vfio_pci_core_ioctl_feature,
694837 .read = nvgrace_gpu_read,
695838 .write = nvgrace_gpu_write,
@@ -703,6 +846,10 @@ static const struct vfio_device_ops nvgrace_gpu_pci_ops = { · 7 unmodified lines
703846 .detach_ioas = vfio_iommufd_physical_detach_ioas,
704847 };
705848
849+static const struct vfio_pci_device_ops nvgrace_gpu_pci_dev_core_ops = {
850+ .get_dmabuf_phys = vfio_pci_core_get_dmabuf_phys,
851+};
852+
706853 static const struct vfio_device_ops nvgrace_gpu_pci_core_ops = {
707854 .name = "nvgrace-gpu-vfio-pci-core",
708855 .init = vfio_pci_core_init_dev,
@@ -710,6 +857,7 @@ static const struct vfio_device_ops nvgrace_gpu_pci_core_ops = { · 1 unmodified line
710857 .open_device = nvgrace_gpu_open_device,
711858 .close_device = vfio_pci_core_close_device,
712859 .ioctl = vfio_pci_core_ioctl,
860+ .get_region_info_caps = vfio_pci_ioctl_get_region_info,
713861 .device_feature = vfio_pci_core_ioctl_feature,
714862 .read = vfio_pci_core_read,
715863 .write = vfio_pci_core_write,
@@ -893,11 +1041,10 @@ static bool nvgrace_gpu_has_mig_hw_bug(struct pci_dev *pdev) · 177 unmodified lines
8931041 * Ensure that the BAR0 region is enabled before accessing the
8941042 * registers.
8951043 */
896−static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev)
1044+static int nvgrace_gpu_probe_check_device_ready(struct pci_dev *pdev)
8971045 {
898− unsigned long timeout = jiffies + msecs_to_jiffies(POLL_TIMEOUT_MS);
8991046 void __iomem *io;
900− int ret = -ETIME;
1047+ int ret;
9011048
9021049 ret = pci_enable_device(pdev);
9031050 if (ret)
@@ -913,16 +1060,8 @@ static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev) · 9 unmodified lines
9131060 goto iomap_exit;
9141061 }
9151062
916− do {
917− if ((ioread32(io + C2C_LINK_BAR0_OFFSET) == STATUS_READY) &&
918− (ioread32(io + HBM_TRAINING_BAR0_OFFSET) == STATUS_READY)) {
919− ret = 0;
920− goto reg_check_exit;
921− }
922− msleep(POLL_QUANTUM_MS);
923− } while (!time_after(jiffies, timeout));
1063+ ret = nvgrace_gpu_wait_device_ready(io);
9241064
925−reg_check_exit:
9261065 pci_iounmap(pdev, io);
9271066 iomap_exit:
9281067 pci_release_selected_regions(pdev, 1 << 0);
@@ -939,7 +1078,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, · 10 unmodified lines
9391078 u64 memphys, memlength;
9401079 int ret;
9411080
942− ret = nvgrace_gpu_wait_device_ready(pdev);
1081+ ret = nvgrace_gpu_probe_check_device_ready(pdev);
9431082 if (ret)
9441083 return ret;
9451084
@@ -965,6 +1104,9 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, · 19 unmodified lines
9651104 memphys, memlength);
9661105 if (ret)
9671106 goto out_put_vdev;
1107+ nvdev->core_device.pci_ops = &nvgrace_gpu_pci_dev_ops;
1108+ } else {
1109+ nvdev->core_device.pci_ops = &nvgrace_gpu_pci_dev_core_ops;
9681110 }
9691111
9701112 ret = vfio_pci_core_register_device(&nvdev->core_device);
@@ -1002,12 +1144,38 @@ static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = { · 31 unmodified lines
10021144
10031145 MODULE_DEVICE_TABLE(pci, nvgrace_gpu_vfio_pci_table);
10041146
1147+/*
1148+ * The GPU reset is required to be serialized against the *first* mapping
1149+ * faults and read/writes accesses to prevent potential RAS events logging.
1150+ *
1151+ * First fault or access after a reset needs to poll device readiness,
1152+ * flag that a reset has occurred. The readiness test is done by holding
1153+ * the memory_lock read lock and we expect all vfio-pci initiated resets to
1154+ * hold the memory_lock write lock to avoid races. However, .reset_done
1155+ * extends beyond the scope of vfio-pci initiated resets therefore we
1156+ * cannot assert this behavior and use lockdep_assert_held_write.
1157+ */
1158+static void nvgrace_gpu_vfio_pci_reset_done(struct pci_dev *pdev)
1159+{
1160+ struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev);
1161+ struct nvgrace_gpu_pci_core_device *nvdev =
1162+ container_of(core_device, struct nvgrace_gpu_pci_core_device,
1163+ core_device);
1164+
1165+ nvdev->reset_done = true;
1166+}
1167+
1168+static const struct pci_error_handlers nvgrace_gpu_vfio_pci_err_handlers = {
1169+ .reset_done = nvgrace_gpu_vfio_pci_reset_done,
1170+ .error_detected = vfio_pci_core_aer_err_detected,
1171+};
1172+
10051173 static struct pci_driver nvgrace_gpu_vfio_pci_driver = {
10061174 .name = KBUILD_MODNAME,
10071175 .id_table = nvgrace_gpu_vfio_pci_table,
10081176 .probe = nvgrace_gpu_probe,
10091177 .remove = nvgrace_gpu_remove,
1010− .err_handler = &vfio_pci_core_err_handlers,
1178+ .err_handler = &nvgrace_gpu_vfio_pci_err_handlers,
10111179 .driver_managed_dma = true,
10121180 };
10131181
drivers/vfio/pci/pds/vfio_dev.c+1−0
@@ -195,6 +195,7 @@ static const struct vfio_device_ops pds_vfio_ops = { · 194 unmodified lines
195195 .open_device = pds_vfio_open_device,
196196 .close_device = pds_vfio_close_device,
197197 .ioctl = vfio_pci_core_ioctl,
198+ .get_region_info_caps = vfio_pci_ioctl_get_region_info,
198199 .device_feature = vfio_pci_core_ioctl_feature,
199200 .read = vfio_pci_core_read,
200201 .write = vfio_pci_core_write,
drivers/vfio/pci/qat/main.c+1−0
@@ -609,6 +609,7 @@ static const struct vfio_device_ops qat_vf_pci_ops = { · 608 unmodified lines
609609 .open_device = qat_vf_pci_open_device,
610610 .close_device = qat_vf_pci_close_device,
611611 .ioctl = vfio_pci_core_ioctl,
612+ .get_region_info_caps = vfio_pci_ioctl_get_region_info,
612613 .read = vfio_pci_core_read,
613614 .write = vfio_pci_core_write,
614615 .mmap = vfio_pci_core_mmap,
drivers/vfio/pci/vfio_pci.c+6−0
@@ -132,6 +132,7 @@ static const struct vfio_device_ops vfio_pci_ops = { · 131 unmodified lines
132132 .open_device = vfio_pci_open_device,
133133 .close_device = vfio_pci_core_close_device,
134134 .ioctl = vfio_pci_core_ioctl,
135+ .get_region_info_caps = vfio_pci_ioctl_get_region_info,
135136 .device_feature = vfio_pci_core_ioctl_feature,
136137 .read = vfio_pci_core_read,
137138 .write = vfio_pci_core_write,
@@ -147,6 +148,10 @@ static const struct vfio_device_ops vfio_pci_ops = { · 9 unmodified lines
147148 .pasid_detach_ioas = vfio_iommufd_physical_pasid_detach_ioas,
148149 };
149150
151+static const struct vfio_pci_device_ops vfio_pci_dev_ops = {
152+ .get_dmabuf_phys = vfio_pci_core_get_dmabuf_phys,
153+};
154+
150155 static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
151156 {
152157 struct vfio_pci_core_device *vdev;
@@ -161,6 +166,7 @@ static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) · 8 unmodified lines
161166 return PTR_ERR(vdev);
162167
163168 dev_set_drvdata(&pdev->dev, vdev);
169+ vdev->pci_ops = &vfio_pci_dev_ops;
164170 ret = vfio_pci_core_register_device(vdev);
165171 if (ret)
166172 goto out_put_vdev;
drivers/vfio/pci/vfio_pci_config.c+19−4
@@ -416,6 +416,7 @@ bool __vfio_pci_memory_enabled(struct vfio_pci_core_device *vdev) · 415 unmodified lines
416416 return pdev->current_state < PCI_D3hot &&
417417 (pdev->no_command_memory || (cmd & PCI_COMMAND_MEMORY));
418418 }
419+EXPORT_SYMBOL_GPL(__vfio_pci_memory_enabled);
419420
420421 /*
421422 * Restore the *real* BARs after we detect a FLR or backdoor reset.
@@ -589,10 +590,12 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos, · 167 unmodified lines
589590 virt_mem = !!(le16_to_cpu(*virt_cmd) & PCI_COMMAND_MEMORY);
590591 new_mem = !!(new_cmd & PCI_COMMAND_MEMORY);
591592
592− if (!new_mem)
593+ if (!new_mem) {
593594 vfio_pci_zap_and_down_write_memory_lock(vdev);
594− else
595+ vfio_pci_dma_buf_move(vdev, true);
596+ } else {
595597 down_write(&vdev->memory_lock);
598+ }
596599
597600 /*
598601 * If the user is writing mem/io enable (new_mem/io) and we
@@ -627,6 +630,8 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos, · 28 unmodified lines
627630 *virt_cmd &= cpu_to_le16(~mask);
628631 *virt_cmd |= cpu_to_le16(new_cmd & mask);
629632
633+ if (__vfio_pci_memory_enabled(vdev))
634+ vfio_pci_dma_buf_move(vdev, false);
630635 up_write(&vdev->memory_lock);
631636 }
632637
@@ -707,12 +712,16 @@ static int __init init_pci_cap_basic_perm(struct perm_bits *perm) · 74 unmodified lines
707712 static void vfio_lock_and_set_power_state(struct vfio_pci_core_device *vdev,
708713 pci_power_t state)
709714 {
710− if (state >= PCI_D3hot)
715+ if (state >= PCI_D3hot) {
711716 vfio_pci_zap_and_down_write_memory_lock(vdev);
712− else
717+ vfio_pci_dma_buf_move(vdev, true);
718+ } else {
713719 down_write(&vdev->memory_lock);
720+ }
714721
715722 vfio_pci_set_power_state(vdev, state);
723+ if (__vfio_pci_memory_enabled(vdev))
724+ vfio_pci_dma_buf_move(vdev, false);
716725 up_write(&vdev->memory_lock);
717726 }
718727
@@ -900,7 +909,10 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos, · 181 unmodified lines
900909
901910 if (!ret && (cap & PCI_EXP_DEVCAP_FLR)) {
902911 vfio_pci_zap_and_down_write_memory_lock(vdev);
912+ vfio_pci_dma_buf_move(vdev, true);
903913 pci_try_reset_function(vdev->pdev);
914+ if (__vfio_pci_memory_enabled(vdev))
915+ vfio_pci_dma_buf_move(vdev, false);
904916 up_write(&vdev->memory_lock);
905917 }
906918 }
@@ -982,7 +994,10 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos, · 75 unmodified lines
982994
983995 if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP)) {
984996 vfio_pci_zap_and_down_write_memory_lock(vdev);
997+ vfio_pci_dma_buf_move(vdev, true);
985998 pci_try_reset_function(vdev->pdev);
999+ if (__vfio_pci_memory_enabled(vdev))
1000+ vfio_pci_dma_buf_move(vdev, false);
9861001 up_write(&vdev->memory_lock);
9871002 }
9881003 }
drivers/vfio/pci/vfio_pci_core.c+156−144
@@ -28,6 +28,7 @@ · 27 unmodified lines
2828 #include <linux/nospec.h>
2929 #include <linux/sched/mm.h>
3030 #include <linux/iommufd.h>
31+#include <linux/pci-p2pdma.h>
3132 #if IS_ENABLED(CONFIG_EEH)
3233 #include <asm/eeh.h>
3334 #endif
@@ -41,6 +42,40 @@ static bool nointxmask; · 7 unmodified lines
4142 static bool disable_vga;
4243 static bool disable_idle_d3;
4344
45+static void vfio_pci_eventfd_rcu_free(struct rcu_head *rcu)
46+{
47+ struct vfio_pci_eventfd *eventfd =
48+ container_of(rcu, struct vfio_pci_eventfd, rcu);
49+
50+ eventfd_ctx_put(eventfd->ctx);
51+ kfree(eventfd);
52+}
53+
54+int vfio_pci_eventfd_replace_locked(struct vfio_pci_core_device *vdev,
55+ struct vfio_pci_eventfd __rcu **peventfd,
56+ struct eventfd_ctx *ctx)
57+{
58+ struct vfio_pci_eventfd *new = NULL;
59+ struct vfio_pci_eventfd *old;
60+
61+ lockdep_assert_held(&vdev->igate);
62+
63+ if (ctx) {
64+ new = kzalloc(sizeof(*new), GFP_KERNEL_ACCOUNT);
65+ if (!new)
66+ return -ENOMEM;
67+
68+ new->ctx = ctx;
69+ }
70+
71+ old = rcu_replace_pointer(*peventfd, new,
72+ lockdep_is_held(&vdev->igate));
73+ if (old)
74+ call_rcu(&old->rcu, vfio_pci_eventfd_rcu_free);
75+
76+ return 0;
77+}
78+
4479 /* List of PF's that vfio_pci_core_sriov_configure() has been called on */
4580 static DEFINE_MUTEX(vfio_pci_sriov_pfs_mutex);
4681 static LIST_HEAD(vfio_pci_sriov_pfs);
@@ -286,6 +321,8 @@ static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev, · 239 unmodified lines
286321 * semaphore.
287322 */
288323 vfio_pci_zap_and_down_write_memory_lock(vdev);
324+ vfio_pci_dma_buf_move(vdev, true);
325+
289326 if (vdev->pm_runtime_engaged) {
290327 up_write(&vdev->memory_lock);
291328 return -EINVAL;
@@ -299,11 +336,9 @@ static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev, · 7 unmodified lines
299336 return 0;
300337 }
301338
302−static int vfio_pci_core_pm_entry(struct vfio_device *device, u32 flags,
339+static int vfio_pci_core_pm_entry(struct vfio_pci_core_device *vdev, u32 flags,
303340 void __user *arg, size_t argsz)
304341 {
305− struct vfio_pci_core_device *vdev =
306− container_of(device, struct vfio_pci_core_device, vdev);
307342 int ret;
308343
309344 ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET, 0);
@@ -320,12 +355,10 @@ static int vfio_pci_core_pm_entry(struct vfio_device *device, u32 flags, · 10 unmodified lines
320355 }
321356
322357 static int vfio_pci_core_pm_entry_with_wakeup(
323− struct vfio_device *device, u32 flags,
358+ struct vfio_pci_core_device *vdev, u32 flags,
324359 struct vfio_device_low_power_entry_with_wakeup __user *arg,
325360 size_t argsz)
326361 {
327− struct vfio_pci_core_device *vdev =
328− container_of(device, struct vfio_pci_core_device, vdev);
329362 struct vfio_device_low_power_entry_with_wakeup entry;
330363 struct eventfd_ctx *efdctx;
331364 int ret;
@@ -373,14 +406,14 @@ static void vfio_pci_runtime_pm_exit(struct vfio_pci_core_device *vdev) · 41 unmodified lines
373406 */
374407 down_write(&vdev->memory_lock);
375408 __vfio_pci_runtime_pm_exit(vdev);
409+ if (__vfio_pci_memory_enabled(vdev))
410+ vfio_pci_dma_buf_move(vdev, false);
376411 up_write(&vdev->memory_lock);
377412 }
378413
379−static int vfio_pci_core_pm_exit(struct vfio_device *device, u32 flags,
414+static int vfio_pci_core_pm_exit(struct vfio_pci_core_device *vdev, u32 flags,
380415 void __user *arg, size_t argsz)
381416 {
382− struct vfio_pci_core_device *vdev =
383− container_of(device, struct vfio_pci_core_device, vdev);
384417 int ret;
385418
386419 ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET, 0);
@@ -695,15 +728,11 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev) · 308 unmodified lines
695728 #endif
696729 vfio_pci_core_disable(vdev);
697730
731+ vfio_pci_dma_buf_cleanup(vdev);
732+
698733 mutex_lock(&vdev->igate);
699− if (vdev->err_trigger) {
700− eventfd_ctx_put(vdev->err_trigger);
701− vdev->err_trigger = NULL;
702− }
703− if (vdev->req_trigger) {
704− eventfd_ctx_put(vdev->req_trigger);
705− vdev->req_trigger = NULL;
706− }
734+ vfio_pci_eventfd_replace_locked(vdev, &vdev->err_trigger, NULL);
735+ vfio_pci_eventfd_replace_locked(vdev, &vdev->req_trigger, NULL);
707736 mutex_unlock(&vdev->igate);
708737 }
709738 EXPORT_SYMBOL_GPL(vfio_pci_core_close_device);
@@ -996,42 +1025,36 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev, · 286 unmodified lines
9961025 return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;
9971026 }
9981027
999−static int vfio_pci_ioctl_get_region_info(struct vfio_pci_core_device *vdev,
1000− struct vfio_region_info __user *arg)
1028+int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
1029+ struct vfio_region_info *info,
1030+ struct vfio_info_cap *caps)
10011031 {
1002− unsigned long minsz = offsetofend(struct vfio_region_info, offset);
1032+ struct vfio_pci_core_device *vdev =
1033+ container_of(core_vdev, struct vfio_pci_core_device, vdev);
10031034 struct pci_dev *pdev = vdev->pdev;
1004− struct vfio_region_info info;
1005− struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
10061035 int i, ret;
10071036
1008− if (copy_from_user(&info, arg, minsz))
1009− return -EFAULT;
1010−
1011− if (info.argsz < minsz)
1012− return -EINVAL;
1013−
1014− switch (info.index) {
1037+ switch (info->index) {
10151038 case VFIO_PCI_CONFIG_REGION_INDEX:
1016− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1017− info.size = pdev->cfg_size;
1018− info.flags = VFIO_REGION_INFO_FLAG_READ |
1019− VFIO_REGION_INFO_FLAG_WRITE;
1039+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1040+ info->size = pdev->cfg_size;
1041+ info->flags = VFIO_REGION_INFO_FLAG_READ |
1042+ VFIO_REGION_INFO_FLAG_WRITE;
10201043 break;
10211044 case VFIO_PCI_BAR0_REGION_INDEX ... VFIO_PCI_BAR5_REGION_INDEX:
1022− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1023− info.size = pci_resource_len(pdev, info.index);
1024− if (!info.size) {
1025− info.flags = 0;
1045+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1046+ info->size = pci_resource_len(pdev, info->index);
1047+ if (!info->size) {
1048+ info->flags = 0;
10261049 break;
10271050 }
10281051
1029− info.flags = VFIO_REGION_INFO_FLAG_READ |
1030− VFIO_REGION_INFO_FLAG_WRITE;
1031− if (vdev->bar_mmap_supported[info.index]) {
1032− info.flags |= VFIO_REGION_INFO_FLAG_MMAP;
1033− if (info.index == vdev->msix_bar) {
1034− ret = msix_mmappable_cap(vdev, &caps);
1052+ info->flags = VFIO_REGION_INFO_FLAG_READ |
1053+ VFIO_REGION_INFO_FLAG_WRITE;
1054+ if (vdev->bar_mmap_supported[info->index]) {
1055+ info->flags |= VFIO_REGION_INFO_FLAG_MMAP;
1056+ if (info->index == vdev->msix_bar) {
1057+ ret = msix_mmappable_cap(vdev, caps);
10351058 if (ret)
10361059 return ret;
10371060 }
@@ -1043,9 +1066,9 @@ static int vfio_pci_ioctl_get_region_info(struct vfio_pci_core_device *vdev, · 5 unmodified lines
10431066 size_t size;
10441067 u16 cmd;
10451068
1046− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1047− info.flags = 0;
1048− info.size = 0;
1069+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1070+ info->flags = 0;
1071+ info->size = 0;
10491072
10501073 if (pci_resource_start(pdev, PCI_ROM_RESOURCE)) {
10511074 /*
@@ -1055,16 +1078,17 @@ static int vfio_pci_ioctl_get_region_info(struct vfio_pci_core_device *vdev, · 3 unmodified lines
10551078 cmd = vfio_pci_memory_lock_and_enable(vdev);
10561079 io = pci_map_rom(pdev, &size);
10571080 if (io) {
1058− info.flags = VFIO_REGION_INFO_FLAG_READ;
1081+ info->flags = VFIO_REGION_INFO_FLAG_READ;
10591082 /* Report the BAR size, not the ROM size. */
1060− info.size = pci_resource_len(pdev, PCI_ROM_RESOURCE);
1083+ info->size = pci_resource_len(pdev,
1084+ PCI_ROM_RESOURCE);
10611085 pci_unmap_rom(pdev, io);
10621086 }
10631087 vfio_pci_memory_unlock_and_restore(vdev, cmd);
10641088 } else if (pdev->rom && pdev->romlen) {
1065− info.flags = VFIO_REGION_INFO_FLAG_READ;
1089+ info->flags = VFIO_REGION_INFO_FLAG_READ;
10661090 /* Report BAR size as power of two. */
1067− info.size = roundup_pow_of_two(pdev->romlen);
1091+ info->size = roundup_pow_of_two(pdev->romlen);
10681092 }
10691093
10701094 break;
@@ -1073,10 +1097,10 @@ static int vfio_pci_ioctl_get_region_info(struct vfio_pci_core_device *vdev, · 2 unmodified lines
10731097 if (!vdev->has_vga)
10741098 return -EINVAL;
10751099
1076− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1077− info.size = 0xc0000;
1078− info.flags = VFIO_REGION_INFO_FLAG_READ |
1079− VFIO_REGION_INFO_FLAG_WRITE;
1100+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1101+ info->size = 0xc0000;
1102+ info->flags = VFIO_REGION_INFO_FLAG_READ |
1103+ VFIO_REGION_INFO_FLAG_WRITE;
10801104
10811105 break;
10821106 default: {
@@ -1085,53 +1109,36 @@ static int vfio_pci_ioctl_get_region_info(struct vfio_pci_core_device *vdev, · 2 unmodified lines
10851109 .header.version = 1
10861110 };
10871111
1088− if (info.index >= VFIO_PCI_NUM_REGIONS + vdev->num_regions)
1112+ if (info->index >= VFIO_PCI_NUM_REGIONS + vdev->num_regions)
10891113 return -EINVAL;
1090− info.index = array_index_nospec(
1091− info.index, VFIO_PCI_NUM_REGIONS + vdev->num_regions);
1114+ info->index = array_index_nospec(
1115+ info->index, VFIO_PCI_NUM_REGIONS + vdev->num_regions);
10921116
1093− i = info.index - VFIO_PCI_NUM_REGIONS;
1117+ i = info->index - VFIO_PCI_NUM_REGIONS;
10941118
1095− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1096− info.size = vdev->region[i].size;
1097− info.flags = vdev->region[i].flags;
1119+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1120+ info->size = vdev->region[i].size;
1121+ info->flags = vdev->region[i].flags;
10981122
10991123 cap_type.type = vdev->region[i].type;
11001124 cap_type.subtype = vdev->region[i].subtype;
11011125
1102− ret = vfio_info_add_capability(&caps, &cap_type.header,
1126+ ret = vfio_info_add_capability(caps, &cap_type.header,
11031127 sizeof(cap_type));
11041128 if (ret)
11051129 return ret;
11061130
11071131 if (vdev->region[i].ops->add_capability) {
11081132 ret = vdev->region[i].ops->add_capability(
1109− vdev, &vdev->region[i], &caps);
1133+ vdev, &vdev->region[i], caps);
11101134 if (ret)
11111135 return ret;
11121136 }
11131137 }
11141138 }
1115−
1116− if (caps.size) {
1117− info.flags |= VFIO_REGION_INFO_FLAG_CAPS;
1118− if (info.argsz < sizeof(info) + caps.size) {
1119− info.argsz = sizeof(info) + caps.size;
1120− info.cap_offset = 0;
1121− } else {
1122− vfio_info_cap_shift(&caps, sizeof(info));
1123− if (copy_to_user(arg + 1, caps.buf, caps.size)) {
1124− kfree(caps.buf);
1125− return -EFAULT;
1126− }
1127− info.cap_offset = sizeof(*arg);
1128− }
1129−
1130− kfree(caps.buf);
1131− }
1132−
1133− return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;
1139+ return 0;
11341140 }
1141+EXPORT_SYMBOL_GPL(vfio_pci_ioctl_get_region_info);
11351142
11361143 static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
11371144 struct vfio_irq_info __user *arg)
@@ -1227,7 +1234,10 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · 89 unmodified lines
12271234 */
12281235 vfio_pci_set_power_state(vdev, PCI_D0);
12291236
1237+ vfio_pci_dma_buf_move(vdev, true);
12301238 ret = pci_try_reset_function(vdev->pdev);
1239+ if (__vfio_pci_memory_enabled(vdev))
1240+ vfio_pci_dma_buf_move(vdev, false);
12311241 up_write(&vdev->memory_lock);
12321242
12331243 return ret;
@@ -1457,8 +1467,6 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, · 223 unmodified lines
14571467 return vfio_pci_ioctl_get_irq_info(vdev, uarg);
14581468 case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO:
14591469 return vfio_pci_ioctl_get_pci_hot_reset_info(vdev, uarg);
1460− case VFIO_DEVICE_GET_REGION_INFO:
1461− return vfio_pci_ioctl_get_region_info(vdev, uarg);
14621470 case VFIO_DEVICE_IOEVENTFD:
14631471 return vfio_pci_ioctl_ioeventfd(vdev, uarg);
14641472 case VFIO_DEVICE_PCI_HOT_RESET:
@@ -1473,11 +1481,10 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, · 8 unmodified lines
14731481 }
14741482 EXPORT_SYMBOL_GPL(vfio_pci_core_ioctl);
14751483
1476−static int vfio_pci_core_feature_token(struct vfio_device *device, u32 flags,
1477− uuid_t __user *arg, size_t argsz)
1484+static int vfio_pci_core_feature_token(struct vfio_pci_core_device *vdev,
1485+ u32 flags, uuid_t __user *arg,
1486+ size_t argsz)
14781487 {
1479− struct vfio_pci_core_device *vdev =
1480− container_of(device, struct vfio_pci_core_device, vdev);
14811488 uuid_t uuid;
14821489 int ret;
14831490
@@ -1504,16 +1511,21 @@ static int vfio_pci_core_feature_token(struct vfio_device *device, u32 flags, · 20 unmodified lines
15041511 int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
15051512 void __user *arg, size_t argsz)
15061513 {
1514+ struct vfio_pci_core_device *vdev =
1515+ container_of(device, struct vfio_pci_core_device, vdev);
1516+
15071517 switch (flags & VFIO_DEVICE_FEATURE_MASK) {
15081518 case VFIO_DEVICE_FEATURE_LOW_POWER_ENTRY:
1509− return vfio_pci_core_pm_entry(device, flags, arg, argsz);
1519+ return vfio_pci_core_pm_entry(vdev, flags, arg, argsz);
15101520 case VFIO_DEVICE_FEATURE_LOW_POWER_ENTRY_WITH_WAKEUP:
1511− return vfio_pci_core_pm_entry_with_wakeup(device, flags,
1521+ return vfio_pci_core_pm_entry_with_wakeup(vdev, flags,
15121522 arg, argsz);
15131523 case VFIO_DEVICE_FEATURE_LOW_POWER_EXIT:
1514− return vfio_pci_core_pm_exit(device, flags, arg, argsz);
1524+ return vfio_pci_core_pm_exit(vdev, flags, arg, argsz);
15151525 case VFIO_DEVICE_FEATURE_PCI_VF_TOKEN:
1516− return vfio_pci_core_feature_token(device, flags, arg, argsz);
1526+ return vfio_pci_core_feature_token(vdev, flags, arg, argsz);
1527+ case VFIO_DEVICE_FEATURE_DMA_BUF:
1528+ return vfio_pci_core_feature_dma_buf(vdev, flags, arg, argsz);
15171529 default:
15181530 return -ENOTTY;
15191531 }
@@ -1640,49 +1652,49 @@ static unsigned long vma_to_pfn(struct vm_area_struct *vma) · 120 unmodified lines
16401652 return (pci_resource_start(vdev->pdev, index) >> PAGE_SHIFT) + pgoff;
16411653 }
16421654
1643−static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
1644− unsigned int order)
1655+vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
1656+ struct vm_fault *vmf,
1657+ unsigned long pfn,
1658+ unsigned int order)
16451659 {
1646− struct vm_area_struct *vma = vmf->vma;
1647− struct vfio_pci_core_device *vdev = vma->vm_private_data;
1648− unsigned long addr = vmf->address & ~((PAGE_SIZE << order) - 1);
1649− unsigned long pgoff = (addr - vma->vm_start) >> PAGE_SHIFT;
1650− unsigned long pfn = vma_to_pfn(vma) + pgoff;
1651− vm_fault_t ret = VM_FAULT_SIGBUS;
1652−
1653− if (order && (addr < vma->vm_start ||
1654− addr + (PAGE_SIZE << order) > vma->vm_end ||
1655− pfn & ((1 << order) - 1))) {
1656− ret = VM_FAULT_FALLBACK;
1657− goto out;
1658− }
1659−
1660− down_read(&vdev->memory_lock);
1660+ lockdep_assert_held_read(&vdev->memory_lock);
16611661
16621662 if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev))
1663− goto out_unlock;
1663+ return VM_FAULT_SIGBUS;
16641664
16651665 switch (order) {
16661666 case 0:
1667− ret = vmf_insert_pfn(vma, vmf->address, pfn);
1668− break;
1667+ return vmf_insert_pfn(vmf->vma, vmf->address, pfn);
16691668 #ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP
16701669 case PMD_ORDER:
1671− ret = vmf_insert_pfn_pmd(vmf, pfn, false);
1672− break;
1670+ return vmf_insert_pfn_pmd(vmf, pfn, false);
16731671 #endif
16741672 #ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP
16751673 case PUD_ORDER:
1676− ret = vmf_insert_pfn_pud(vmf, pfn, false);
1674+ return vmf_insert_pfn_pud(vmf, pfn, false);
16771675 break;
16781676 #endif
16791677 default:
1680− ret = VM_FAULT_FALLBACK;
1678+ return VM_FAULT_FALLBACK;
1679+ }
1680+}
1681+EXPORT_SYMBOL_GPL(vfio_pci_vmf_insert_pfn);
1682+
1683+static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
1684+ unsigned int order)
1685+{
1686+ struct vm_area_struct *vma = vmf->vma;
1687+ struct vfio_pci_core_device *vdev = vma->vm_private_data;
1688+ unsigned long addr = vmf->address & ~((PAGE_SIZE << order) - 1);
1689+ unsigned long pgoff = (addr - vma->vm_start) >> PAGE_SHIFT;
1690+ unsigned long pfn = vma_to_pfn(vma) + pgoff;
1691+ vm_fault_t ret = VM_FAULT_FALLBACK;
1692+
1693+ if (is_aligned_for_order(vma, addr, pfn, order)) {
1694+ scoped_guard(rwsem_read, &vdev->memory_lock)
1695+ ret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order);
16811696 }
16821697
1683−out_unlock:
1684− up_read(&vdev->memory_lock);
1685−out:
16861698 dev_dbg_ratelimited(&vdev->pdev->dev,
16871699 "%s(,order = %d) BAR %ld page offset 0x%lx: 0x%x\n",
16881700 __func__, order,
@@ -1749,18 +1761,9 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma · 60 unmodified lines
17491761 * Even though we don't make use of the barmap for the mmap,
17501762 * we need to request the region and the barmap tracks that.
17511763 */
1752− if (!vdev->barmap[index]) {
1753− ret = pci_request_selected_regions(pdev,
1754− 1 << index, "vfio-pci");
1755− if (ret)
1756− return ret;
1757−
1758− vdev->barmap[index] = pci_iomap(pdev, index, 0);
1759− if (!vdev->barmap[index]) {
1760− pci_release_selected_regions(pdev, 1 << index);
1761− return -ENOMEM;
1762− }
1763− }
1764+ ret = vfio_pci_core_setup_barmap(vdev, index);
1765+ if (ret)
1766+ return ret;
17641767
17651768 vma->vm_private_data = vdev;
17661769 vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
@@ -1800,21 +1803,21 @@ void vfio_pci_core_request(struct vfio_device *core_vdev, unsigned int count) · 33 unmodified lines
18001803 struct vfio_pci_core_device *vdev =
18011804 container_of(core_vdev, struct vfio_pci_core_device, vdev);
18021805 struct pci_dev *pdev = vdev->pdev;
1806+ struct vfio_pci_eventfd *eventfd;
18031807
1804− mutex_lock(&vdev->igate);
1805−
1806− if (vdev->req_trigger) {
1808+ rcu_read_lock();
1809+ eventfd = rcu_dereference(vdev->req_trigger);
1810+ if (eventfd) {
18071811 if (!(count % 10))
18081812 pci_notice_ratelimited(pdev,
18091813 "Relaying device request to user (#%u)\n",
18101814 count);
1811− eventfd_signal(vdev->req_trigger);
1815+ eventfd_signal(eventfd->ctx);
18121816 } else if (count == 0) {
18131817 pci_warn(pdev,
18141818 "No device request channel registered, blocked until released by user\n");
18151819 }
1816−
1817− mutex_unlock(&vdev->igate);
1820+ rcu_read_unlock();
18181821 }
18191822 EXPORT_SYMBOL_GPL(vfio_pci_core_request);
18201823
@@ -2085,6 +2088,7 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev) · 264 unmodified lines
20852088 {
20862089 struct vfio_pci_core_device *vdev =
20872090 container_of(core_vdev, struct vfio_pci_core_device, vdev);
2091+ int ret;
20882092
20892093 vdev->pdev = to_pci_dev(core_vdev->dev);
20902094 vdev->irq_type = VFIO_PCI_NUM_IRQS;
@@ -2094,6 +2098,10 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev) · 3 unmodified lines
20942098 INIT_LIST_HEAD(&vdev->dummy_resources_list);
20952099 INIT_LIST_HEAD(&vdev->ioeventfds_list);
20962100 INIT_LIST_HEAD(&vdev->sriov_pfs_item);
2101+ ret = pcim_p2pdma_init(vdev->pdev);
2102+ if (ret && ret != -EOPNOTSUPP)
2103+ return ret;
2104+ INIT_LIST_HEAD(&vdev->dmabufs);
20972105 init_rwsem(&vdev->memory_lock);
20982106 xa_init(&vdev->ctx);
20992107
@@ -2227,13 +2235,13 @@ pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev, · 127 unmodified lines
22272235 pci_channel_state_t state)
22282236 {
22292237 struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
2238+ struct vfio_pci_eventfd *eventfd;
22302239
2231− mutex_lock(&vdev->igate);
2232−
2233− if (vdev->err_trigger)
2234− eventfd_signal(vdev->err_trigger);
2235−
2236− mutex_unlock(&vdev->igate);
2240+ rcu_read_lock();
2241+ eventfd = rcu_dereference(vdev->err_trigger);
2242+ if (eventfd)
2243+ eventfd_signal(eventfd->ctx);
2244+ rcu_read_unlock();
22372245
22382246 return PCI_ERS_RESULT_CAN_RECOVER;
22392247 }
@@ -2458,6 +2466,7 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · 218 unmodified lines
24582466 break;
24592467 }
24602468
2469+ vfio_pci_dma_buf_move(vdev, true);
24612470 vfio_pci_zap_bars(vdev);
24622471 }
24632472
@@ -2486,8 +2495,11 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · 22 unmodified lines
24862495
24872496 err_undo:
24882497 list_for_each_entry_from_reverse(vdev, &dev_set->device_list,
2489− vdev.dev_set_list)
2498+ vdev.dev_set_list) {
2499+ if (vdev->vdev.open_count && __vfio_pci_memory_enabled(vdev))
2500+ vfio_pci_dma_buf_move(vdev, false);
24902501 up_write(&vdev->memory_lock);
2502+ }
24912503
24922504 list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list)
24932505 pm_runtime_put(&vdev->pdev->dev);
drivers/vfio/pci/vfio_pci_dmabuf.c+316−0
@@ -0,0 +1,316 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+/* Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES.
3+ */
4+#include <linux/dma-buf-mapping.h>
5+#include <linux/pci-p2pdma.h>
6+#include <linux/dma-resv.h>
7+
8+#include "vfio_pci_priv.h"
9+
10+MODULE_IMPORT_NS("DMA_BUF");
11+
12+struct vfio_pci_dma_buf {
13+ struct dma_buf *dmabuf;
14+ struct vfio_pci_core_device *vdev;
15+ struct list_head dmabufs_elm;
16+ size_t size;
17+ struct dma_buf_phys_vec *phys_vec;
18+ struct p2pdma_provider *provider;
19+ u32 nr_ranges;
20+ u8 revoked : 1;
21+};
22+
23+static int vfio_pci_dma_buf_attach(struct dma_buf *dmabuf,
24+ struct dma_buf_attachment *attachment)
25+{
26+ struct vfio_pci_dma_buf *priv = dmabuf->priv;
27+
28+ if (!attachment->peer2peer)
29+ return -EOPNOTSUPP;
30+
31+ if (priv->revoked)
32+ return -ENODEV;
33+
34+ return 0;
35+}
36+
37+static struct sg_table *
38+vfio_pci_dma_buf_map(struct dma_buf_attachment *attachment,
39+ enum dma_data_direction dir)
40+{
41+ struct vfio_pci_dma_buf *priv = attachment->dmabuf->priv;
42+
43+ dma_resv_assert_held(priv->dmabuf->resv);
44+
45+ if (priv->revoked)
46+ return ERR_PTR(-ENODEV);
47+
48+ return dma_buf_phys_vec_to_sgt(attachment, priv->provider,
49+ priv->phys_vec, priv->nr_ranges,
50+ priv->size, dir);
51+}
52+
53+static void vfio_pci_dma_buf_unmap(struct dma_buf_attachment *attachment,
54+ struct sg_table *sgt,
55+ enum dma_data_direction dir)
56+{
57+ dma_buf_free_sgt(attachment, sgt, dir);
58+}
59+
60+static void vfio_pci_dma_buf_release(struct dma_buf *dmabuf)
61+{
62+ struct vfio_pci_dma_buf *priv = dmabuf->priv;
63+
64+ /*
65+ * Either this or vfio_pci_dma_buf_cleanup() will remove from the list.
66+ * The refcount prevents both.
67+ */
68+ if (priv->vdev) {
69+ down_write(&priv->vdev->memory_lock);
70+ list_del_init(&priv->dmabufs_elm);
71+ up_write(&priv->vdev->memory_lock);
72+ vfio_device_put_registration(&priv->vdev->vdev);
73+ }
74+ kfree(priv->phys_vec);
75+ kfree(priv);
76+}
77+
78+static const struct dma_buf_ops vfio_pci_dmabuf_ops = {
79+ .attach = vfio_pci_dma_buf_attach,
80+ .map_dma_buf = vfio_pci_dma_buf_map,
81+ .unmap_dma_buf = vfio_pci_dma_buf_unmap,
82+ .release = vfio_pci_dma_buf_release,
83+};
84+
85+int vfio_pci_core_fill_phys_vec(struct dma_buf_phys_vec *phys_vec,
86+ struct vfio_region_dma_range *dma_ranges,
87+ size_t nr_ranges, phys_addr_t start,
88+ phys_addr_t len)
89+{
90+ phys_addr_t max_addr;
91+ unsigned int i;
92+
93+ max_addr = start + len;
94+ for (i = 0; i < nr_ranges; i++) {
95+ phys_addr_t end;
96+
97+ if (!dma_ranges[i].length)
98+ return -EINVAL;
99+
100+ if (check_add_overflow(start, dma_ranges[i].offset,
101+ &phys_vec[i].paddr) ||
102+ check_add_overflow(phys_vec[i].paddr,
103+ dma_ranges[i].length, &end))
104+ return -EOVERFLOW;
105+ if (end > max_addr)
106+ return -EINVAL;
107+
108+ phys_vec[i].len = dma_ranges[i].length;
109+ }
110+ return 0;
111+}
112+EXPORT_SYMBOL_GPL(vfio_pci_core_fill_phys_vec);
113+
114+int vfio_pci_core_get_dmabuf_phys(struct vfio_pci_core_device *vdev,
115+ struct p2pdma_provider **provider,
116+ unsigned int region_index,
117+ struct dma_buf_phys_vec *phys_vec,
118+ struct vfio_region_dma_range *dma_ranges,
119+ size_t nr_ranges)
120+{
121+ struct pci_dev *pdev = vdev->pdev;
122+
123+ *provider = pcim_p2pdma_provider(pdev, region_index);
124+ if (!*provider)
125+ return -EINVAL;
126+
127+ return vfio_pci_core_fill_phys_vec(
128+ phys_vec, dma_ranges, nr_ranges,
129+ pci_resource_start(pdev, region_index),
130+ pci_resource_len(pdev, region_index));
131+}
132+EXPORT_SYMBOL_GPL(vfio_pci_core_get_dmabuf_phys);
133+
134+static int validate_dmabuf_input(struct vfio_device_feature_dma_buf *dma_buf,
135+ struct vfio_region_dma_range *dma_ranges,
136+ size_t *lengthp)
137+{
138+ size_t length = 0;
139+ u32 i;
140+
141+ for (i = 0; i < dma_buf->nr_ranges; i++) {
142+ u64 offset = dma_ranges[i].offset;
143+ u64 len = dma_ranges[i].length;
144+
145+ if (!len || !PAGE_ALIGNED(offset) || !PAGE_ALIGNED(len))
146+ return -EINVAL;
147+
148+ if (check_add_overflow(length, len, &length))
149+ return -EINVAL;
150+ }
151+
152+ /*
153+ * dma_iova_try_alloc() will WARN on if userspace proposes a size that
154+ * is too big, eg with lots of ranges.
155+ */
156+ if ((u64)(length) & DMA_IOVA_USE_SWIOTLB)
157+ return -EINVAL;
158+
159+ *lengthp = length;
160+ return 0;
161+}
162+
163+int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
164+ struct vfio_device_feature_dma_buf __user *arg,
165+ size_t argsz)
166+{
167+ struct vfio_device_feature_dma_buf get_dma_buf = {};
168+ struct vfio_region_dma_range *dma_ranges;
169+ DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
170+ struct vfio_pci_dma_buf *priv;
171+ size_t length;
172+ int ret;
173+
174+ if (!vdev->pci_ops || !vdev->pci_ops->get_dmabuf_phys)
175+ return -EOPNOTSUPP;
176+
177+ ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_GET,
178+ sizeof(get_dma_buf));
179+ if (ret != 1)
180+ return ret;
181+
182+ if (copy_from_user(&get_dma_buf, arg, sizeof(get_dma_buf)))
183+ return -EFAULT;
184+
185+ if (!get_dma_buf.nr_ranges || get_dma_buf.flags)
186+ return -EINVAL;
187+
188+ /*
189+ * For PCI the region_index is the BAR number like everything else.
190+ */
191+ if (get_dma_buf.region_index >= VFIO_PCI_ROM_REGION_INDEX)
192+ return -ENODEV;
193+
194+ dma_ranges = memdup_array_user(&arg->dma_ranges, get_dma_buf.nr_ranges,
195+ sizeof(*dma_ranges));
196+ if (IS_ERR(dma_ranges))
197+ return PTR_ERR(dma_ranges);
198+
199+ ret = validate_dmabuf_input(&get_dma_buf, dma_ranges, &length);
200+ if (ret)
201+ goto err_free_ranges;
202+
203+ priv = kzalloc(sizeof(*priv), GFP_KERNEL);
204+ if (!priv) {
205+ ret = -ENOMEM;
206+ goto err_free_ranges;
207+ }
208+ priv->phys_vec = kcalloc(get_dma_buf.nr_ranges, sizeof(*priv->phys_vec),
209+ GFP_KERNEL);
210+ if (!priv->phys_vec) {
211+ ret = -ENOMEM;
212+ goto err_free_priv;
213+ }
214+
215+ priv->vdev = vdev;
216+ priv->nr_ranges = get_dma_buf.nr_ranges;
217+ priv->size = length;
218+ ret = vdev->pci_ops->get_dmabuf_phys(vdev, &priv->provider,
219+ get_dma_buf.region_index,
220+ priv->phys_vec, dma_ranges,
221+ priv->nr_ranges);
222+ if (ret)
223+ goto err_free_phys;
224+
225+ kfree(dma_ranges);
226+ dma_ranges = NULL;
227+
228+ if (!vfio_device_try_get_registration(&vdev->vdev)) {
229+ ret = -ENODEV;
230+ goto err_free_phys;
231+ }
232+
233+ exp_info.ops = &vfio_pci_dmabuf_ops;
234+ exp_info.size = priv->size;
235+ exp_info.flags = get_dma_buf.open_flags;
236+ exp_info.priv = priv;
237+
238+ priv->dmabuf = dma_buf_export(&exp_info);
239+ if (IS_ERR(priv->dmabuf)) {
240+ ret = PTR_ERR(priv->dmabuf);
241+ goto err_dev_put;
242+ }
243+
244+ /* dma_buf_put() now frees priv */
245+ INIT_LIST_HEAD(&priv->dmabufs_elm);
246+ down_write(&vdev->memory_lock);
247+ dma_resv_lock(priv->dmabuf->resv, NULL);
248+ priv->revoked = !__vfio_pci_memory_enabled(vdev);
249+ list_add_tail(&priv->dmabufs_elm, &vdev->dmabufs);
250+ dma_resv_unlock(priv->dmabuf->resv);
251+ up_write(&vdev->memory_lock);
252+
253+ /*
254+ * dma_buf_fd() consumes the reference, when the file closes the dmabuf
255+ * will be released.
256+ */
257+ ret = dma_buf_fd(priv->dmabuf, get_dma_buf.open_flags);
258+ if (ret < 0)
259+ goto err_dma_buf;
260+ return ret;
261+
262+err_dma_buf:
263+ dma_buf_put(priv->dmabuf);
264+err_dev_put:
265+ vfio_device_put_registration(&vdev->vdev);
266+err_free_phys:
267+ kfree(priv->phys_vec);
268+err_free_priv:
269+ kfree(priv);
270+err_free_ranges:
271+ kfree(dma_ranges);
272+ return ret;
273+}
274+
275+void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, bool revoked)
276+{
277+ struct vfio_pci_dma_buf *priv;
278+ struct vfio_pci_dma_buf *tmp;
279+
280+ lockdep_assert_held_write(&vdev->memory_lock);
281+
282+ list_for_each_entry_safe(priv, tmp, &vdev->dmabufs, dmabufs_elm) {
283+ if (!get_file_active(&priv->dmabuf->file))
284+ continue;
285+
286+ if (priv->revoked != revoked) {
287+ dma_resv_lock(priv->dmabuf->resv, NULL);
288+ priv->revoked = revoked;
289+ dma_buf_move_notify(priv->dmabuf);
290+ dma_resv_unlock(priv->dmabuf->resv);
291+ }
292+ fput(priv->dmabuf->file);
293+ }
294+}
295+
296+void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev)
297+{
298+ struct vfio_pci_dma_buf *priv;
299+ struct vfio_pci_dma_buf *tmp;
300+
301+ down_write(&vdev->memory_lock);
302+ list_for_each_entry_safe(priv, tmp, &vdev->dmabufs, dmabufs_elm) {
303+ if (!get_file_active(&priv->dmabuf->file))
304+ continue;
305+
306+ dma_resv_lock(priv->dmabuf->resv, NULL);
307+ list_del_init(&priv->dmabufs_elm);
308+ priv->vdev = NULL;
309+ priv->revoked = true;
310+ dma_buf_move_notify(priv->dmabuf);
311+ dma_resv_unlock(priv->dmabuf->resv);
312+ vfio_device_put_registration(&vdev->vdev);
313+ fput(priv->dmabuf->file);
314+ }
315+ up_write(&vdev->memory_lock);
316+}
drivers/vfio/pci/vfio_pci_intrs.c+33−19
@@ -731,21 +731,27 @@ static int vfio_pci_set_msi_trigger(struct vfio_pci_core_device *vdev, · 730 unmodified lines
731731 return 0;
732732 }
733733
734−static int vfio_pci_set_ctx_trigger_single(struct eventfd_ctx **ctx,
734+static int vfio_pci_set_ctx_trigger_single(struct vfio_pci_core_device *vdev,
735+ struct vfio_pci_eventfd __rcu **peventfd,
735736 unsigned int count, uint32_t flags,
736737 void *data)
737738 {
738739 /* DATA_NONE/DATA_BOOL enables loopback testing */
739740 if (flags & VFIO_IRQ_SET_DATA_NONE) {
740− if (*ctx) {
741− if (count) {
742− eventfd_signal(*ctx);
743− } else {
744− eventfd_ctx_put(*ctx);
745− *ctx = NULL;
746− }
741+ struct vfio_pci_eventfd *eventfd;
742+
743+ eventfd = rcu_dereference_protected(*peventfd,
744+ lockdep_is_held(&vdev->igate));
745+
746+ if (!eventfd)
747+ return -EINVAL;
748+
749+ if (count) {
750+ eventfd_signal(eventfd->ctx);
747751 return 0;
748752 }
753+
754+ return vfio_pci_eventfd_replace_locked(vdev, peventfd, NULL);
749755 } else if (flags & VFIO_IRQ_SET_DATA_BOOL) {
750756 uint8_t trigger;
751757
@@ -753,8 +759,15 @@ static int vfio_pci_set_ctx_trigger_single(struct eventfd_ctx **ctx, · 1 unmodified line
753759 return -EINVAL;
754760
755761 trigger = *(uint8_t *)data;
756− if (trigger && *ctx)
757− eventfd_signal(*ctx);
762+
763+ if (trigger) {
764+ struct vfio_pci_eventfd *eventfd =
765+ rcu_dereference_protected(*peventfd,
766+ lockdep_is_held(&vdev->igate));
767+
768+ if (eventfd)
769+ eventfd_signal(eventfd->ctx);
770+ }
758771
759772 return 0;
760773 } else if (flags & VFIO_IRQ_SET_DATA_EVENTFD) {
@@ -765,22 +778,23 @@ static int vfio_pci_set_ctx_trigger_single(struct eventfd_ctx **ctx, · 4 unmodified lines
765778
766779 fd = *(int32_t *)data;
767780 if (fd == -1) {
768− if (*ctx)
769− eventfd_ctx_put(*ctx);
770− *ctx = NULL;
781+ return vfio_pci_eventfd_replace_locked(vdev,
782+ peventfd, NULL);
771783 } else if (fd >= 0) {
772784 struct eventfd_ctx *efdctx;
785+ int ret;
773786
774787 efdctx = eventfd_ctx_fdget(fd);
775788 if (IS_ERR(efdctx))
776789 return PTR_ERR(efdctx);
777790
778− if (*ctx)
779− eventfd_ctx_put(*ctx);
791+ ret = vfio_pci_eventfd_replace_locked(vdev,
792+ peventfd, efdctx);
793+ if (ret)
794+ eventfd_ctx_put(efdctx);
780795
781− *ctx = efdctx;
796+ return ret;
782797 }
783− return 0;
784798 }
785799
786800 return -EINVAL;
@@ -793,7 +807,7 @@ static int vfio_pci_set_err_trigger(struct vfio_pci_core_device *vdev, · 6 unmodified lines
793807 if (index != VFIO_PCI_ERR_IRQ_INDEX || start != 0 || count > 1)
794808 return -EINVAL;
795809
796− return vfio_pci_set_ctx_trigger_single(&vdev->err_trigger,
810+ return vfio_pci_set_ctx_trigger_single(vdev, &vdev->err_trigger,
797811 count, flags, data);
798812 }
799813
@@ -804,7 +818,7 @@ static int vfio_pci_set_req_trigger(struct vfio_pci_core_device *vdev, · 4 unmodified lines
804818 if (index != VFIO_PCI_REQ_IRQ_INDEX || start != 0 || count > 1)
805819 return -EINVAL;
806820
807− return vfio_pci_set_ctx_trigger_single(&vdev->req_trigger,
821+ return vfio_pci_set_ctx_trigger_single(vdev, &vdev->req_trigger,
808822 count, flags, data);
809823 }
810824
drivers/vfio/pci/vfio_pci_priv.h+27−1
@@ -26,6 +26,10 @@ struct vfio_pci_ioeventfd { · 25 unmodified lines
2626 bool vfio_pci_intx_mask(struct vfio_pci_core_device *vdev);
2727 void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev);
2828
29+int vfio_pci_eventfd_replace_locked(struct vfio_pci_core_device *vdev,
30+ struct vfio_pci_eventfd __rcu **peventfd,
31+ struct eventfd_ctx *ctx);
32+
2933 int vfio_pci_set_irqs_ioctl(struct vfio_pci_core_device *vdev, uint32_t flags,
3034 unsigned index, unsigned start, unsigned count,
3135 void *data);
@@ -60,7 +64,6 @@ void vfio_config_free(struct vfio_pci_core_device *vdev); · 28 unmodified lines
6064 int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev,
6165 pci_power_t state);
6266
63−bool __vfio_pci_memory_enabled(struct vfio_pci_core_device *vdev);
6467 void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev);
6568 u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);
6669 void vfio_pci_memory_unlock_and_restore(struct vfio_pci_core_device *vdev,
@@ -107,4 +110,27 @@ static inline bool vfio_pci_is_vga(struct pci_dev *pdev) · 40 unmodified lines
107110 return (pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA;
108111 }
109112
113+#ifdef CONFIG_VFIO_PCI_DMABUF
114+int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
115+ struct vfio_device_feature_dma_buf __user *arg,
116+ size_t argsz);
117+void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev);
118+void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, bool revoked);
119+#else
120+static inline int
121+vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
122+ struct vfio_device_feature_dma_buf __user *arg,
123+ size_t argsz)
124+{
125+ return -ENOTTY;
126+}
127+static inline void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev)
128+{
129+}
130+static inline void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev,
131+ bool revoked)
132+{
133+}
134+#endif
135+
110136 #endif
drivers/vfio/pci/virtio/common.h+2−3
@@ -109,10 +109,9 @@ void virtiovf_migration_reset_done(struct pci_dev *pdev); · 108 unmodified lines
109109
110110 #ifdef CONFIG_VIRTIO_VFIO_PCI_ADMIN_LEGACY
111111 int virtiovf_open_legacy_io(struct virtiovf_pci_core_device *virtvdev);
112−long virtiovf_vfio_pci_core_ioctl(struct vfio_device *core_vdev,
113− unsigned int cmd, unsigned long arg);
114112 int virtiovf_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
115− unsigned int cmd, unsigned long arg);
113+ struct vfio_region_info *info,
114+ struct vfio_info_cap *caps);
116115 ssize_t virtiovf_pci_core_write(struct vfio_device *core_vdev,
117116 const char __user *buf, size_t count,
118117 loff_t *ppos);
drivers/vfio/pci/virtio/legacy_io.c+8−30
@@ -281,41 +281,19 @@ ssize_t virtiovf_pci_core_write(struct vfio_device *core_vdev, const char __user · 280 unmodified lines
281281 }
282282
283283 int virtiovf_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
284− unsigned int cmd, unsigned long arg)
284+ struct vfio_region_info *info,
285+ struct vfio_info_cap *caps)
285286 {
286287 struct virtiovf_pci_core_device *virtvdev = container_of(
287288 core_vdev, struct virtiovf_pci_core_device, core_device.vdev);
288− unsigned long minsz = offsetofend(struct vfio_region_info, offset);
289− void __user *uarg = (void __user *)arg;
290− struct vfio_region_info info = {};
291289
292− if (copy_from_user(&info, uarg, minsz))
293− return -EFAULT;
290+ if (info->index != VFIO_PCI_BAR0_REGION_INDEX)
291+ return vfio_pci_ioctl_get_region_info(core_vdev, info, caps);
294292
295− if (info.argsz < minsz)
296− return -EINVAL;
297−
298− switch (info.index) {
299− case VFIO_PCI_BAR0_REGION_INDEX:
300− info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
301− info.size = virtvdev->bar0_virtual_buf_size;
302− info.flags = VFIO_REGION_INFO_FLAG_READ |
303− VFIO_REGION_INFO_FLAG_WRITE;
304− return copy_to_user(uarg, &info, minsz) ? -EFAULT : 0;
305− default:
306− return vfio_pci_core_ioctl(core_vdev, cmd, arg);
307− }
308−}
309−
310−long virtiovf_vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
311− unsigned long arg)
312−{
313− switch (cmd) {
314− case VFIO_DEVICE_GET_REGION_INFO:
315− return virtiovf_pci_ioctl_get_region_info(core_vdev, cmd, arg);
316− default:
317− return vfio_pci_core_ioctl(core_vdev, cmd, arg);
318− }
293+ info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
294+ info->size = virtvdev->bar0_virtual_buf_size;
295+ info->flags = VFIO_REGION_INFO_FLAG_READ | VFIO_REGION_INFO_FLAG_WRITE;
296+ return 0;
319297 }
320298
321299 static int virtiovf_set_notify_addr(struct virtiovf_pci_core_device *virtvdev)
drivers/vfio/pci/virtio/main.c+4−1
@@ -88,6 +88,7 @@ static const struct vfio_device_ops virtiovf_vfio_pci_lm_ops = { · 87 unmodified lines
8888 .open_device = virtiovf_pci_open_device,
8989 .close_device = virtiovf_pci_close_device,
9090 .ioctl = vfio_pci_core_ioctl,
91+ .get_region_info_caps = vfio_pci_ioctl_get_region_info,
9192 .device_feature = vfio_pci_core_ioctl_feature,
9293 .read = vfio_pci_core_read,
9394 .write = vfio_pci_core_write,
@@ -108,7 +109,8 @@ static const struct vfio_device_ops virtiovf_vfio_pci_tran_lm_ops = { · 14 unmodified lines
108109 .release = virtiovf_pci_core_release_dev,
109110 .open_device = virtiovf_pci_open_device,
110111 .close_device = virtiovf_pci_close_device,
111− .ioctl = virtiovf_vfio_pci_core_ioctl,
112+ .ioctl = vfio_pci_core_ioctl,
113+ .get_region_info_caps = virtiovf_pci_ioctl_get_region_info,
112114 .device_feature = vfio_pci_core_ioctl_feature,
113115 .read = virtiovf_pci_core_read,
114116 .write = virtiovf_pci_core_write,
@@ -130,6 +132,7 @@ static const struct vfio_device_ops virtiovf_vfio_pci_ops = { · 15 unmodified lines
130132 .open_device = virtiovf_pci_open_device,
131133 .close_device = vfio_pci_core_close_device,
132134 .ioctl = vfio_pci_core_ioctl,
135+ .get_region_info_caps = vfio_pci_ioctl_get_region_info,
133136 .device_feature = vfio_pci_core_ioctl_feature,
134137 .read = vfio_pci_core_read,
135138 .write = vfio_pci_core_write,
drivers/vfio/platform/vfio_amba.c+1−0
@@ -115,6 +115,7 @@ static const struct vfio_device_ops vfio_amba_ops = { · 114 unmodified lines
115115 .open_device = vfio_platform_open_device,
116116 .close_device = vfio_platform_close_device,
117117 .ioctl = vfio_platform_ioctl,
118+ .get_region_info_caps = vfio_platform_ioctl_get_region_info,
118119 .read = vfio_platform_read,
119120 .write = vfio_platform_write,
120121 .mmap = vfio_platform_mmap,
drivers/vfio/platform/vfio_platform.c+1−0
@@ -101,6 +101,7 @@ static const struct vfio_device_ops vfio_platform_ops = { · 100 unmodified lines
101101 .open_device = vfio_platform_open_device,
102102 .close_device = vfio_platform_close_device,
103103 .ioctl = vfio_platform_ioctl,
104+ .get_region_info_caps = vfio_platform_ioctl_get_region_info,
104105 .read = vfio_platform_read,
105106 .write = vfio_platform_write,
106107 .mmap = vfio_platform_mmap,
drivers/vfio/platform/vfio_platform_common.c+18−22
@@ -272,6 +272,24 @@ err_irq: · 271 unmodified lines
272272 }
273273 EXPORT_SYMBOL_GPL(vfio_platform_open_device);
274274
275+int vfio_platform_ioctl_get_region_info(struct vfio_device *core_vdev,
276+ struct vfio_region_info *info,
277+ struct vfio_info_cap *caps)
278+{
279+ struct vfio_platform_device *vdev =
280+ container_of(core_vdev, struct vfio_platform_device, vdev);
281+
282+ if (info->index >= vdev->num_regions)
283+ return -EINVAL;
284+
285+ /* map offset to the physical address */
286+ info->offset = VFIO_PLATFORM_INDEX_TO_OFFSET(info->index);
287+ info->size = vdev->regions[info->index].size;
288+ info->flags = vdev->regions[info->index].flags;
289+ return 0;
290+}
291+EXPORT_SYMBOL_GPL(vfio_platform_ioctl_get_region_info);
292+
275293 long vfio_platform_ioctl(struct vfio_device *core_vdev,
276294 unsigned int cmd, unsigned long arg)
277295 {
@@ -300,28 +318,6 @@ long vfio_platform_ioctl(struct vfio_device *core_vdev, · 22 unmodified lines
300318 return copy_to_user((void __user *)arg, &info, minsz) ?
301319 -EFAULT : 0;
302320
303− } else if (cmd == VFIO_DEVICE_GET_REGION_INFO) {
304− struct vfio_region_info info;
305−
306− minsz = offsetofend(struct vfio_region_info, offset);
307−
308− if (copy_from_user(&info, (void __user *)arg, minsz))
309− return -EFAULT;
310−
311− if (info.argsz < minsz)
312− return -EINVAL;
313−
314− if (info.index >= vdev->num_regions)
315− return -EINVAL;
316−
317− /* map offset to the physical address */
318− info.offset = VFIO_PLATFORM_INDEX_TO_OFFSET(info.index);
319− info.size = vdev->regions[info.index].size;
320− info.flags = vdev->regions[info.index].flags;
321−
322− return copy_to_user((void __user *)arg, &info, minsz) ?
323− -EFAULT : 0;
324−
325321 } else if (cmd == VFIO_DEVICE_GET_IRQ_INFO) {
326322 struct vfio_irq_info info;
327323
drivers/vfio/platform/vfio_platform_private.h+3−0
@@ -85,6 +85,9 @@ int vfio_platform_open_device(struct vfio_device *core_vdev); · 84 unmodified lines
8585 void vfio_platform_close_device(struct vfio_device *core_vdev);
8686 long vfio_platform_ioctl(struct vfio_device *core_vdev,
8787 unsigned int cmd, unsigned long arg);
88+int vfio_platform_ioctl_get_region_info(struct vfio_device *core_vdev,
89+ struct vfio_region_info *info,
90+ struct vfio_info_cap *caps);
8891 ssize_t vfio_platform_read(struct vfio_device *core_vdev,
8992 char __user *buf, size_t count,
9093 loff_t *ppos);
drivers/vfio/vfio_main.c+51−0
@@ -172,11 +172,13 @@ void vfio_device_put_registration(struct vfio_device *device) · 171 unmodified lines
172172 if (refcount_dec_and_test(&device->refcount))
173173 complete(&device->comp);
174174 }
175+EXPORT_SYMBOL_GPL(vfio_device_put_registration);
175176
176177 bool vfio_device_try_get_registration(struct vfio_device *device)
177178 {
178179 return refcount_inc_not_zero(&device->refcount);
179180 }
181+EXPORT_SYMBOL_GPL(vfio_device_try_get_registration);
180182
181183 /*
182184 * VFIO driver API
@@ -1259,6 +1261,51 @@ static int vfio_ioctl_device_feature(struct vfio_device *device, · 1076 unmodified lines
12591261 }
12601262 }
12611263
1264+static long vfio_get_region_info(struct vfio_device *device,
1265+ struct vfio_region_info __user *arg)
1266+{
1267+ unsigned long minsz = offsetofend(struct vfio_region_info, offset);
1268+ struct vfio_region_info info = {};
1269+ struct vfio_info_cap caps = {};
1270+ int ret;
1271+
1272+ if (unlikely(!device->ops->get_region_info_caps))
1273+ return -EINVAL;
1274+
1275+ if (copy_from_user(&info, arg, minsz))
1276+ return -EFAULT;
1277+ if (info.argsz < minsz)
1278+ return -EINVAL;
1279+
1280+ ret = device->ops->get_region_info_caps(device, &info, &caps);
1281+ if (ret)
1282+ goto out_free;
1283+
1284+ if (caps.size) {
1285+ info.flags |= VFIO_REGION_INFO_FLAG_CAPS;
1286+ if (info.argsz < sizeof(info) + caps.size) {
1287+ info.argsz = sizeof(info) + caps.size;
1288+ info.cap_offset = 0;
1289+ } else {
1290+ vfio_info_cap_shift(&caps, sizeof(info));
1291+ if (copy_to_user(arg + 1, caps.buf, caps.size)) {
1292+ ret = -EFAULT;
1293+ goto out_free;
1294+ }
1295+ info.cap_offset = sizeof(info);
1296+ }
1297+ }
1298+
1299+ if (copy_to_user(arg, &info, minsz)){
1300+ ret = -EFAULT;
1301+ goto out_free;
1302+ }
1303+
1304+out_free:
1305+ kfree(caps.buf);
1306+ return ret;
1307+}
1308+
12621309 static long vfio_device_fops_unl_ioctl(struct file *filep,
12631310 unsigned int cmd, unsigned long arg)
12641311 {
@@ -1296,6 +1343,10 @@ static long vfio_device_fops_unl_ioctl(struct file *filep, · 31 unmodified lines
12961343 ret = vfio_ioctl_device_feature(device, uptr);
12971344 break;
12981345
1346+ case VFIO_DEVICE_GET_REGION_INFO:
1347+ ret = vfio_get_region_info(device, uptr);
1348+ break;
1349+
12991350 default:
13001351 if (unlikely(!device->ops->ioctl))
13011352 ret = -EINVAL;
include/linux/dma-buf-mapping.h+17−0
@@ -0,0 +1,17 @@
1+/* SPDX-License-Identifier: GPL-2.0-only */
2+/*
3+ * DMA BUF Mapping Helpers
4+ *
5+ */
6+#ifndef __DMA_BUF_MAPPING_H__
7+#define __DMA_BUF_MAPPING_H__
8+#include <linux/dma-buf.h>
9+
10+struct sg_table *dma_buf_phys_vec_to_sgt(struct dma_buf_attachment *attach,
11+ struct p2pdma_provider *provider,
12+ struct dma_buf_phys_vec *phys_vec,
13+ size_t nr_ranges, size_t size,
14+ enum dma_data_direction dir);
15+void dma_buf_free_sgt(struct dma_buf_attachment *attach, struct sg_table *sgt,
16+ enum dma_data_direction dir);
17+#endif
include/linux/dma-buf.h+11−0
@@ -22,6 +22,7 @@ · 21 unmodified lines
2222 #include <linux/fs.h>
2323 #include <linux/dma-fence.h>
2424 #include <linux/wait.h>
25+#include <linux/pci-p2pdma.h>
2526
2627 struct device;
2728 struct dma_buf;
@@ -530,6 +531,16 @@ struct dma_buf_export_info { · 502 unmodified lines
530531 void *priv;
531532 };
532533
534+/**
535+ * struct dma_buf_phys_vec - describe continuous chunk of memory
536+ * @paddr: physical address of that chunk
537+ * @len: Length of this chunk
538+ */
539+struct dma_buf_phys_vec {
540+ phys_addr_t paddr;
541+ size_t len;
542+};
543+
533544 /**
534545 * DEFINE_DMA_BUF_EXPORT_INFO - helper macro for exporters
535546 * @name: export-info name
include/linux/hisi_acc_qm.h+3−0
@@ -99,6 +99,9 @@ · 98 unmodified lines
9999
100100 #define QM_DEV_ALG_MAX_LEN 256
101101
102+#define QM_MIG_REGION_SEL 0x100198
103+#define QM_MIG_REGION_EN BIT(0)
104+
102105 /* uacce mode of the driver */
103106 #define UACCE_MODE_NOUACCE 0 /* don't use uacce */
104107 #define UACCE_MODE_SVA 1 /* use uacce sva mode */
include/linux/pci-p2pdma.h+73−47
@@ -16,7 +16,58 @@ · 15 unmodified lines
1616 struct block_device;
1717 struct scatterlist;
1818
19+/**
20+ * struct p2pdma_provider
21+ *
22+ * A p2pdma provider is a range of MMIO address space available to the CPU.
23+ */
24+struct p2pdma_provider {
25+ struct device *owner;
26+ u64 bus_offset;
27+};
28+
29+enum pci_p2pdma_map_type {
30+ /*
31+ * PCI_P2PDMA_MAP_UNKNOWN: Used internally as an initial state before
32+ * the mapping type has been calculated. Exported routines for the API
33+ * will never return this value.
34+ */
35+ PCI_P2PDMA_MAP_UNKNOWN = 0,
36+
37+ /*
38+ * Not a PCI P2PDMA transfer.
39+ */
40+ PCI_P2PDMA_MAP_NONE,
41+
42+ /*
43+ * PCI_P2PDMA_MAP_NOT_SUPPORTED: Indicates the transaction will
44+ * traverse the host bridge and the host bridge is not in the
45+ * allowlist. DMA Mapping routines should return an error when
46+ * this is returned.
47+ */
48+ PCI_P2PDMA_MAP_NOT_SUPPORTED,
49+
50+ /*
51+ * PCI_P2PDMA_MAP_BUS_ADDR: Indicates that two devices can talk to
52+ * each other directly through a PCI switch and the transaction will
53+ * not traverse the host bridge. Such a mapping should program
54+ * the DMA engine with PCI bus addresses.
55+ */
56+ PCI_P2PDMA_MAP_BUS_ADDR,
57+
58+ /*
59+ * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: Indicates two devices can talk
60+ * to each other, but the transaction traverses a host bridge on the
61+ * allowlist. In this case, a normal mapping either with CPU physical
62+ * addresses (in the case of dma-direct) or IOVA addresses (in the
63+ * case of IOMMUs) should be used to program the DMA engine.
64+ */
65+ PCI_P2PDMA_MAP_THRU_HOST_BRIDGE,
66+};
67+
1968 #ifdef CONFIG_PCI_P2PDMA
69+int pcim_p2pdma_init(struct pci_dev *pdev);
70+struct p2pdma_provider *pcim_p2pdma_provider(struct pci_dev *pdev, int bar);
2071 int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size,
2172 u64 offset);
2273 int pci_p2pdma_distance_many(struct pci_dev *provider, struct device **clients,
@@ -33,7 +84,18 @@ int pci_p2pdma_enable_store(const char *page, struct pci_dev **p2p_dev, · 10 unmodified lines
3384 bool *use_p2pdma);
3485 ssize_t pci_p2pdma_enable_show(char *page, struct pci_dev *p2p_dev,
3586 bool use_p2pdma);
87+enum pci_p2pdma_map_type pci_p2pdma_map_type(struct p2pdma_provider *provider,
88+ struct device *dev);
3689 #else /* CONFIG_PCI_P2PDMA */
90+static inline int pcim_p2pdma_init(struct pci_dev *pdev)
91+{
92+ return -EOPNOTSUPP;
93+}
94+static inline struct p2pdma_provider *pcim_p2pdma_provider(struct pci_dev *pdev,
95+ int bar)
96+{
97+ return NULL;
98+}
3799 static inline int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar,
38100 size_t size, u64 offset)
39101 {
@@ -85,6 +147,11 @@ static inline ssize_t pci_p2pdma_enable_show(char *page, · 45 unmodified lines
85147 {
86148 return sprintf(page, "none\n");
87149 }
150+static inline enum pci_p2pdma_map_type
151+pci_p2pdma_map_type(struct p2pdma_provider *provider, struct device *dev)
152+{
153+ return PCI_P2PDMA_MAP_NOT_SUPPORTED;
154+}
88155 #endif /* CONFIG_PCI_P2PDMA */
89156
90157
@@ -99,51 +166,12 @@ static inline struct pci_dev *pci_p2pmem_find(struct device *client) · 8 unmodified lines
99166 return pci_p2pmem_find_many(&client, 1);
100167 }
101168
102−enum pci_p2pdma_map_type {
103− /*
104− * PCI_P2PDMA_MAP_UNKNOWN: Used internally as an initial state before
105− * the mapping type has been calculated. Exported routines for the API
106− * will never return this value.
107− */
108− PCI_P2PDMA_MAP_UNKNOWN = 0,
109−
110− /*
111− * Not a PCI P2PDMA transfer.
112− */
113− PCI_P2PDMA_MAP_NONE,
114−
115− /*
116− * PCI_P2PDMA_MAP_NOT_SUPPORTED: Indicates the transaction will
117− * traverse the host bridge and the host bridge is not in the
118− * allowlist. DMA Mapping routines should return an error when
119− * this is returned.
120− */
121− PCI_P2PDMA_MAP_NOT_SUPPORTED,
122−
123− /*
124− * PCI_P2PDMA_MAP_BUS_ADDR: Indicates that two devices can talk to
125− * each other directly through a PCI switch and the transaction will
126− * not traverse the host bridge. Such a mapping should program
127− * the DMA engine with PCI bus addresses.
128− */
129− PCI_P2PDMA_MAP_BUS_ADDR,
130−
131− /*
132− * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: Indicates two devices can talk
133− * to each other, but the transaction traverses a host bridge on the
134− * allowlist. In this case, a normal mapping either with CPU physical
135− * addresses (in the case of dma-direct) or IOVA addresses (in the
136− * case of IOMMUs) should be used to program the DMA engine.
137− */
138− PCI_P2PDMA_MAP_THRU_HOST_BRIDGE,
139−};
140−
141169 struct pci_p2pdma_map_state {
142− struct dev_pagemap *pgmap;
170+ struct p2pdma_provider *mem;
143171 enum pci_p2pdma_map_type map;
144− u64 bus_off;
145172 };
146173
174+
147175 /* helper for pci_p2pdma_state(), do not use directly */
148176 void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state,
149177 struct device *dev, struct page *page);
@@ -162,8 +190,7 @@ pci_p2pdma_state(struct pci_p2pdma_map_state *state, struct device *dev, · 12 unmodified lines
162190 struct page *page)
163191 {
164192 if (IS_ENABLED(CONFIG_PCI_P2PDMA) && is_pci_p2pdma_page(page)) {
165− if (state->pgmap != page_pgmap(page))
166− __pci_p2pdma_update_state(state, dev, page);
193+ __pci_p2pdma_update_state(state, dev, page);
167194 return state->map;
168195 }
169196 return PCI_P2PDMA_MAP_NONE;
@@ -172,16 +199,15 @@ pci_p2pdma_state(struct pci_p2pdma_map_state *state, struct device *dev, · 2 unmodified lines
172199 /**
173200 * pci_p2pdma_bus_addr_map - Translate a physical address to a bus address
174201 * for a PCI_P2PDMA_MAP_BUS_ADDR transfer.
175− * @state: P2P state structure
202+ * @provider: P2P provider structure
176203 * @paddr: physical address to map
177204 *
178205 * Map a physically contiguous PCI_P2PDMA_MAP_BUS_ADDR transfer.
179206 */
180207 static inline dma_addr_t
181−pci_p2pdma_bus_addr_map(struct pci_p2pdma_map_state *state, phys_addr_t paddr)
208+pci_p2pdma_bus_addr_map(struct p2pdma_provider *provider, phys_addr_t paddr)
182209 {
183− WARN_ON_ONCE(state->map != PCI_P2PDMA_MAP_BUS_ADDR);
184− return paddr + state->bus_off;
210+ return paddr + provider->bus_offset;
185211 }
186212
187213 #endif /* _LINUX_PCI_P2P_H */
include/linux/vfio.h+6−0
@@ -21,6 +21,7 @@ struct kvm; · 20 unmodified lines
2121 struct iommufd_ctx;
2222 struct iommufd_device;
2323 struct iommufd_access;
24+struct vfio_info_cap;
2425
2526 /*
2627 * VFIO devices can be placed in a set, this allows all devices to share this
@@ -132,6 +133,9 @@ struct vfio_device_ops { · 105 unmodified lines
132133 size_t count, loff_t *size);
133134 long (*ioctl)(struct vfio_device *vdev, unsigned int cmd,
134135 unsigned long arg);
136+ int (*get_region_info_caps)(struct vfio_device *vdev,
137+ struct vfio_region_info *info,
138+ struct vfio_info_cap *caps);
135139 int (*mmap)(struct vfio_device *vdev, struct vm_area_struct *vma);
136140 void (*request)(struct vfio_device *vdev, unsigned int count);
137141 int (*match)(struct vfio_device *vdev, char *buf);
@@ -297,6 +301,8 @@ static inline void vfio_put_device(struct vfio_device *device) · 159 unmodified lines
297301 int vfio_register_group_dev(struct vfio_device *device);
298302 int vfio_register_emulated_iommu_dev(struct vfio_device *device);
299303 void vfio_unregister_group_dev(struct vfio_device *device);
304+bool vfio_device_try_get_registration(struct vfio_device *device);
305+void vfio_device_put_registration(struct vfio_device *device);
300306
301307 int vfio_assign_device_set(struct vfio_device *device, void *set_id);
302308 unsigned int vfio_device_set_open_count(struct vfio_device_set *dev_set);
include/linux/vfio_pci_core.h+67−2
@@ -12,6 +12,7 @@ · 11 unmodified lines
1212 #include <linux/pci.h>
1313 #include <linux/vfio.h>
1414 #include <linux/irqbypass.h>
15+#include <linux/rcupdate.h>
1516 #include <linux/types.h>
1617 #include <linux/uuid.h>
1718 #include <linux/notifier.h>
@@ -26,6 +27,13 @@ · 8 unmodified lines
2627
2728 struct vfio_pci_core_device;
2829 struct vfio_pci_region;
30+struct p2pdma_provider;
31+struct dma_buf_phys_vec;
32+
33+struct vfio_pci_eventfd {
34+ struct eventfd_ctx *ctx;
35+ struct rcu_head rcu;
36+};
2937
3038 struct vfio_pci_regops {
3139 ssize_t (*rw)(struct vfio_pci_core_device *vdev, char __user *buf,
@@ -49,9 +57,48 @@ struct vfio_pci_region { · 17 unmodified lines
4957 u32 flags;
5058 };
5159
60+struct vfio_pci_device_ops {
61+ int (*get_dmabuf_phys)(struct vfio_pci_core_device *vdev,
62+ struct p2pdma_provider **provider,
63+ unsigned int region_index,
64+ struct dma_buf_phys_vec *phys_vec,
65+ struct vfio_region_dma_range *dma_ranges,
66+ size_t nr_ranges);
67+};
68+
69+#if IS_ENABLED(CONFIG_VFIO_PCI_DMABUF)
70+int vfio_pci_core_fill_phys_vec(struct dma_buf_phys_vec *phys_vec,
71+ struct vfio_region_dma_range *dma_ranges,
72+ size_t nr_ranges, phys_addr_t start,
73+ phys_addr_t len);
74+int vfio_pci_core_get_dmabuf_phys(struct vfio_pci_core_device *vdev,
75+ struct p2pdma_provider **provider,
76+ unsigned int region_index,
77+ struct dma_buf_phys_vec *phys_vec,
78+ struct vfio_region_dma_range *dma_ranges,
79+ size_t nr_ranges);
80+#else
81+static inline int
82+vfio_pci_core_fill_phys_vec(struct dma_buf_phys_vec *phys_vec,
83+ struct vfio_region_dma_range *dma_ranges,
84+ size_t nr_ranges, phys_addr_t start,
85+ phys_addr_t len)
86+{
87+ return -EINVAL;
88+}
89+static inline int vfio_pci_core_get_dmabuf_phys(
90+ struct vfio_pci_core_device *vdev, struct p2pdma_provider **provider,
91+ unsigned int region_index, struct dma_buf_phys_vec *phys_vec,
92+ struct vfio_region_dma_range *dma_ranges, size_t nr_ranges)
93+{
94+ return -EOPNOTSUPP;
95+}
96+#endif
97+
5298 struct vfio_pci_core_device {
5399 struct vfio_device vdev;
54100 struct pci_dev *pdev;
101+ const struct vfio_pci_device_ops *pci_ops;
55102 void __iomem *barmap[PCI_STD_NUM_BARS];
56103 bool bar_mmap_supported[PCI_STD_NUM_BARS];
57104 u8 *pci_config_map;
@@ -83,8 +130,8 @@ struct vfio_pci_core_device { · 25 unmodified lines
83130 struct pci_saved_state *pci_saved_state;
84131 struct pci_saved_state *pm_save;
85132 int ioeventfds_nr;
86− struct eventfd_ctx *err_trigger;
87− struct eventfd_ctx *req_trigger;
133+ struct vfio_pci_eventfd __rcu *err_trigger;
134+ struct vfio_pci_eventfd __rcu *req_trigger;
88135 struct eventfd_ctx *pm_wake_eventfd_ctx;
89136 struct list_head dummy_resources_list;
90137 struct mutex ioeventfds_lock;
@@ -94,6 +141,7 @@ struct vfio_pci_core_device { · 3 unmodified lines
94141 struct vfio_pci_core_device *sriov_pf_core_dev;
95142 struct notifier_block nb;
96143 struct rw_semaphore memory_lock;
144+ struct list_head dmabufs;
97145 };
98146
99147 /* Will be exported for vfio pci drivers usage */
@@ -115,10 +163,16 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, · 15 unmodified lines
115163 unsigned long arg);
116164 int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
117165 void __user *arg, size_t argsz);
166+int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
167+ struct vfio_region_info *info,
168+ struct vfio_info_cap *caps);
118169 ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf,
119170 size_t count, loff_t *ppos);
120171 ssize_t vfio_pci_core_write(struct vfio_device *core_vdev, const char __user *buf,
121172 size_t count, loff_t *ppos);
173+vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
174+ struct vm_fault *vmf, unsigned long pfn,
175+ unsigned int order);
122176 int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma);
123177 void vfio_pci_core_request(struct vfio_device *core_vdev, unsigned int count);
124178 int vfio_pci_core_match(struct vfio_device *core_vdev, char *buf);
@@ -134,6 +188,7 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem, · 9 unmodified lines
134188 void __iomem *io, char __user *buf,
135189 loff_t off, size_t count, size_t x_start,
136190 size_t x_end, bool iswrite);
191+bool __vfio_pci_memory_enabled(struct vfio_pci_core_device *vdev);
137192 bool vfio_pci_core_range_intersect_range(loff_t buf_start, size_t buf_cnt,
138193 loff_t reg_start, size_t reg_cnt,
139194 loff_t *buf_offset,
@@ -161,4 +216,14 @@ VFIO_IOREAD_DECLARATION(32) · 21 unmodified lines
161216 VFIO_IOREAD_DECLARATION(64)
162217 #endif
163218
219+static inline bool is_aligned_for_order(struct vm_area_struct *vma,
220+ unsigned long addr,
221+ unsigned long pfn,
222+ unsigned int order)
223+{
224+ return !(order && (addr < vma->vm_start ||
225+ addr + (PAGE_SIZE << order) > vma->vm_end ||
226+ !IS_ALIGNED(pfn, 1 << order)));
227+}
228+
164229 #endif /* VFIO_PCI_CORE_H */
include/uapi/linux/vfio.h+28−0
@@ -14,6 +14,7 @@ · 13 unmodified lines
1414
1515 #include <linux/types.h>
1616 #include <linux/ioctl.h>
17+#include <linux/stddef.h>
1718
1819 #define VFIO_API_VERSION 0
1920
@@ -1478,6 +1479,33 @@ struct vfio_device_feature_bus_master { · 1458 unmodified lines
14781479 };
14791480 #define VFIO_DEVICE_FEATURE_BUS_MASTER 10
14801481
1482+/**
1483+ * Upon VFIO_DEVICE_FEATURE_GET create a dma_buf fd for the
1484+ * regions selected.
1485+ *
1486+ * open_flags are the typical flags passed to open(2), eg O_RDWR, O_CLOEXEC,
1487+ * etc. offset/length specify a slice of the region to create the dmabuf from.
1488+ * nr_ranges is the total number of (P2P DMA) ranges that comprise the dmabuf.
1489+ *
1490+ * flags should be 0.
1491+ *
1492+ * Return: The fd number on success, -1 and errno is set on failure.
1493+ */
1494+#define VFIO_DEVICE_FEATURE_DMA_BUF 11
1495+
1496+struct vfio_region_dma_range {
1497+ __u64 offset;
1498+ __u64 length;
1499+};
1500+
1501+struct vfio_device_feature_dma_buf {
1502+ __u32 region_index;
1503+ __u32 open_flags;
1504+ __u32 flags;
1505+ __u32 nr_ranges;
1506+ struct vfio_region_dma_range dma_ranges[] __counted_by(nr_ranges);
1507+};
1508+
14811509 /* -------- API for Type1 VFIO IOMMU -------- */
14821510
14831511 /**
kernel/dma/direct.c+2−2
@@ -479,8 +479,8 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, · 478 unmodified lines
479479 }
480480 break;
481481 case PCI_P2PDMA_MAP_BUS_ADDR:
482− sg->dma_address = pci_p2pdma_bus_addr_map(&p2pdma_state,
483− sg_phys(sg));
482+ sg->dma_address = pci_p2pdma_bus_addr_map(
483+ p2pdma_state.mem, sg_phys(sg));
484484 sg_dma_len(sg) = sg->length;
485485 sg_dma_mark_bus_address(sg);
486486 continue;
mm/hmm.c+1−1
@@ -811,7 +811,7 @@ dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map, · 810 unmodified lines
811811 break;
812812 case PCI_P2PDMA_MAP_BUS_ADDR:
813813 pfns[idx] |= HMM_PFN_P2PDMA_BUS | HMM_PFN_DMA_MAPPED;
814− return pci_p2pdma_bus_addr_map(p2pdma_state, paddr);
814+ return pci_p2pdma_bus_addr_map(p2pdma_state->mem, paddr);
815815 default:
816816 return DMA_MAPPING_ERROR;
817817 }
samples/vfio-mdev/mbochs.c+24−47
@@ -143,11 +143,6 @@ static struct mdev_parent mbochs_parent; · 142 unmodified lines
143143 static atomic_t mbochs_avail_mbytes;
144144 static const struct vfio_device_ops mbochs_dev_ops;
145145
146−struct vfio_region_info_ext {
147− struct vfio_region_info base;
148− struct vfio_region_info_cap_type type;
149−};
150−
151146 struct mbochs_mode {
152147 u32 drm_format;
153148 u32 bytepp;
@@ -1033,10 +1028,12 @@ static int mbochs_dmabuf_export(struct mbochs_dmabuf *dmabuf) · 879 unmodified lines
10331028 return 0;
10341029 }
10351030
1036−static int mbochs_get_region_info(struct mdev_state *mdev_state,
1037− struct vfio_region_info_ext *ext)
1031+static int mbochs_ioctl_get_region_info(struct vfio_device *vdev,
1032+ struct vfio_region_info *region_info,
1033+ struct vfio_info_cap *caps)
10381034 {
1039− struct vfio_region_info *region_info = &ext->base;
1035+ struct mdev_state *mdev_state =
1036+ container_of(vdev, struct mdev_state, vdev);
10401037
10411038 if (region_info->index >= MBOCHS_NUM_REGIONS)
10421039 return -EINVAL;
@@ -1061,20 +1058,23 @@ static int mbochs_get_region_info(struct mdev_state *mdev_state, · 18 unmodified lines
10611058 region_info->flags = (VFIO_REGION_INFO_FLAG_READ |
10621059 VFIO_REGION_INFO_FLAG_WRITE);
10631060 break;
1064− case MBOCHS_EDID_REGION_INDEX:
1065− ext->base.argsz = sizeof(*ext);
1066− ext->base.offset = MBOCHS_EDID_OFFSET;
1067− ext->base.size = MBOCHS_EDID_SIZE;
1068− ext->base.flags = (VFIO_REGION_INFO_FLAG_READ |
1069− VFIO_REGION_INFO_FLAG_WRITE |
1070− VFIO_REGION_INFO_FLAG_CAPS);
1071− ext->base.cap_offset = offsetof(typeof(*ext), type);
1072− ext->type.header.id = VFIO_REGION_INFO_CAP_TYPE;
1073− ext->type.header.version = 1;
1074− ext->type.header.next = 0;
1075− ext->type.type = VFIO_REGION_TYPE_GFX;
1076− ext->type.subtype = VFIO_REGION_SUBTYPE_GFX_EDID;
1077− break;
1061+ case MBOCHS_EDID_REGION_INDEX: {
1062+ struct vfio_region_info_cap_type cap_type = {
1063+ .header.id = VFIO_REGION_INFO_CAP_TYPE,
1064+ .header.version = 1,
1065+ .type = VFIO_REGION_TYPE_GFX,
1066+ .subtype = VFIO_REGION_SUBTYPE_GFX_EDID,
1067+ };
1068+
1069+ region_info->offset = MBOCHS_EDID_OFFSET;
1070+ region_info->size = MBOCHS_EDID_SIZE;
1071+ region_info->flags = (VFIO_REGION_INFO_FLAG_READ |
1072+ VFIO_REGION_INFO_FLAG_WRITE |
1073+ VFIO_REGION_INFO_FLAG_CAPS);
1074+
1075+ return vfio_info_add_capability(caps, &cap_type.header,
1076+ sizeof(cap_type));
1077+ }
10781078 default:
10791079 region_info->size = 0;
10801080 region_info->offset = 0;
@@ -1191,7 +1191,7 @@ static long mbochs_ioctl(struct vfio_device *vdev, unsigned int cmd, · 110 unmodified lines
11911191 struct mdev_state *mdev_state =
11921192 container_of(vdev, struct mdev_state, vdev);
11931193 int ret = 0;
1194− unsigned long minsz, outsz;
1194+ unsigned long minsz;
11951195
11961196 switch (cmd) {
11971197 case VFIO_DEVICE_GET_INFO:
@@ -1215,30 +1215,6 @@ static long mbochs_ioctl(struct vfio_device *vdev, unsigned int cmd, · 17 unmodified lines
12151215
12161216 return 0;
12171217 }
1218− case VFIO_DEVICE_GET_REGION_INFO:
1219− {
1220− struct vfio_region_info_ext info;
1221−
1222− minsz = offsetofend(typeof(info), base.offset);
1223−
1224− if (copy_from_user(&info, (void __user *)arg, minsz))
1225− return -EFAULT;
1226−
1227− outsz = info.base.argsz;
1228− if (outsz < minsz)
1229− return -EINVAL;
1230− if (outsz > sizeof(info))
1231− return -EINVAL;
1232−
1233− ret = mbochs_get_region_info(mdev_state, &info);
1234− if (ret)
1235− return ret;
1236−
1237− if (copy_to_user((void __user *)arg, &info, outsz))
1238− return -EFAULT;
1239−
1240− return 0;
1241− }
12421218
12431219 case VFIO_DEVICE_GET_IRQ_INFO:
12441220 {
@@ -1376,6 +1352,7 @@ static const struct vfio_device_ops mbochs_dev_ops = { · 131 unmodified lines
13761352 .read = mbochs_read,
13771353 .write = mbochs_write,
13781354 .ioctl = mbochs_ioctl,
1355+ .get_region_info_caps = mbochs_ioctl_get_region_info,
13791356 .mmap = mbochs_mmap,
13801357 .bind_iommufd = vfio_iommufd_emulated_bind,
13811358 .unbind_iommufd = vfio_iommufd_emulated_unbind,
samples/vfio-mdev/mdpy.c+7−27
@@ -435,10 +435,13 @@ static int mdpy_mmap(struct vfio_device *vdev, struct vm_area_struct *vma) · 434 unmodified lines
435435 return remap_vmalloc_range(vma, mdev_state->memblk, 0);
436436 }
437437
438−static int mdpy_get_region_info(struct mdev_state *mdev_state,
439− struct vfio_region_info *region_info,
440− u16 *cap_type_id, void **cap_type)
438+static int mdpy_ioctl_get_region_info(struct vfio_device *vdev,
439+ struct vfio_region_info *region_info,
440+ struct vfio_info_cap *caps)
441441 {
442+ struct mdev_state *mdev_state =
443+ container_of(vdev, struct mdev_state, vdev);
444+
442445 if (region_info->index >= VFIO_PCI_NUM_REGIONS &&
443446 region_info->index != MDPY_DISPLAY_REGION)
444447 return -EINVAL;
@@ -544,30 +547,6 @@ static long mdpy_ioctl(struct vfio_device *vdev, unsigned int cmd, · 99 unmodified lines
544547
545548 return 0;
546549 }
547− case VFIO_DEVICE_GET_REGION_INFO:
548− {
549− struct vfio_region_info info;
550− u16 cap_type_id = 0;
551− void *cap_type = NULL;
552−
553− minsz = offsetofend(struct vfio_region_info, offset);
554−
555− if (copy_from_user(&info, (void __user *)arg, minsz))
556− return -EFAULT;
557−
558− if (info.argsz < minsz)
559− return -EINVAL;
560−
561− ret = mdpy_get_region_info(mdev_state, &info, &cap_type_id,
562− &cap_type);
563− if (ret)
564− return ret;
565−
566− if (copy_to_user((void __user *)arg, &info, minsz))
567− return -EFAULT;
568−
569− return 0;
570− }
571550
572551 case VFIO_DEVICE_GET_IRQ_INFO:
573552 {
@@ -665,6 +644,7 @@ static const struct vfio_device_ops mdpy_dev_ops = { · 91 unmodified lines
665644 .read = mdpy_read,
666645 .write = mdpy_write,
667646 .ioctl = mdpy_ioctl,
647+ .get_region_info_caps = mdpy_ioctl_get_region_info,
668648 .mmap = mdpy_mmap,
669649 .bind_iommufd = vfio_iommufd_emulated_bind,
670650 .unbind_iommufd = vfio_iommufd_emulated_unbind,
samples/vfio-mdev/mtty.c+7−28
@@ -624,7 +624,7 @@ static void handle_bar_read(unsigned int index, struct mdev_state *mdev_state, · 623 unmodified lines
624624 u8 lsr = 0;
625625
626626 mutex_lock(&mdev_state->rxtx_lock);
627− /* atleast one char in FIFO */
627+ /* at least one char in FIFO */
628628 if (mdev_state->s[index].rxtx.head !=
629629 mdev_state->s[index].rxtx.tail)
630630 lsr |= UART_LSR_DR;
@@ -1717,10 +1717,12 @@ static int mtty_set_irqs(struct mdev_state *mdev_state, uint32_t flags, · 1086 unmodified lines
17171717 return ret;
17181718 }
17191719
1720−static int mtty_get_region_info(struct mdev_state *mdev_state,
1721− struct vfio_region_info *region_info,
1722− u16 *cap_type_id, void **cap_type)
1720+static int mtty_ioctl_get_region_info(struct vfio_device *vdev,
1721+ struct vfio_region_info *region_info,
1722+ struct vfio_info_cap *caps)
17231723 {
1724+ struct mdev_state *mdev_state =
1725+ container_of(vdev, struct mdev_state, vdev);
17241726 unsigned int size = 0;
17251727 u32 bar_index;
17261728
@@ -1817,30 +1819,6 @@ static long mtty_ioctl(struct vfio_device *vdev, unsigned int cmd, · 90 unmodified lines
18171819
18181820 return 0;
18191821 }
1820− case VFIO_DEVICE_GET_REGION_INFO:
1821− {
1822− struct vfio_region_info info;
1823− u16 cap_type_id = 0;
1824− void *cap_type = NULL;
1825−
1826− minsz = offsetofend(struct vfio_region_info, offset);
1827−
1828− if (copy_from_user(&info, (void __user *)arg, minsz))
1829− return -EFAULT;
1830−
1831− if (info.argsz < minsz)
1832− return -EINVAL;
1833−
1834− ret = mtty_get_region_info(mdev_state, &info, &cap_type_id,
1835− &cap_type);
1836− if (ret)
1837− return ret;
1838−
1839− if (copy_to_user((void __user *)arg, &info, minsz))
1840− return -EFAULT;
1841−
1842− return 0;
1843− }
18441822
18451823 case VFIO_DEVICE_GET_IRQ_INFO:
18461824 {
@@ -1949,6 +1927,7 @@ static const struct vfio_device_ops mtty_dev_ops = { · 102 unmodified lines
19491927 .read = mtty_read,
19501928 .write = mtty_write,
19511929 .ioctl = mtty_ioctl,
1930+ .get_region_info_caps = mtty_ioctl_get_region_info,
19521931 .bind_iommufd = vfio_iommufd_emulated_bind,
19531932 .unbind_iommufd = vfio_iommufd_emulated_unbind,
19541933 .attach_ioas = vfio_iommufd_emulated_attach_ioas,
tools/testing/selftests/vfio/Makefile+9−1
@@ -2,8 +2,14 @@ CFLAGS = $(KHDR_INCLUDES) · 1 unmodified line
22 TEST_GEN_PROGS += vfio_dma_mapping_test
33 TEST_GEN_PROGS += vfio_iommufd_setup_test
44 TEST_GEN_PROGS += vfio_pci_device_test
5+TEST_GEN_PROGS += vfio_pci_device_init_perf_test
56 TEST_GEN_PROGS += vfio_pci_driver_test
6−TEST_PROGS_EXTENDED := run.sh
7+
8+TEST_FILES += scripts/cleanup.sh
9+TEST_FILES += scripts/lib.sh
10+TEST_FILES += scripts/run.sh
11+TEST_FILES += scripts/setup.sh
12+
713 include ../lib.mk
814 include lib/libvfio.mk
915
@@ -11,6 +17,8 @@ CFLAGS += -I$(top_srcdir)/tools/include · 1 unmodified line
1117 CFLAGS += -MD
1218 CFLAGS += $(EXTRA_CFLAGS)
1319
20+LDFLAGS += -pthread
21+
1422 $(TEST_GEN_PROGS): %: %.o $(LIBVFIO_O)
1523 $(CC) $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $< $(LIBVFIO_O) $(LDLIBS) -o $@
1624
tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c+18−18
@@ -9,7 +9,7 @@ · 8 unmodified lines
99 #include <linux/pci_ids.h>
1010 #include <linux/sizes.h>
1111
12−#include <vfio_util.h>
12+#include <libvfio.h>
1313
1414 #include "registers.h"
1515
@@ -70,7 +70,7 @@ static int dsa_probe(struct vfio_pci_device *device) · 54 unmodified lines
7070 return -EINVAL;
7171
7272 if (dsa_int_handle_request_required(device)) {
73− printf("Device requires requesting interrupt handles\n");
73+ dev_err(device, "Device requires requesting interrupt handles\n");
7474 return -EINVAL;
7575 }
7676
@@ -91,23 +91,23 @@ static void dsa_check_sw_err(struct vfio_pci_device *device) · 14 unmodified lines
9191 return;
9292 }
9393
94− fprintf(stderr, "SWERR: 0x%016lx 0x%016lx 0x%016lx 0x%016lx\n",
94+ dev_err(device, "SWERR: 0x%016lx 0x%016lx 0x%016lx 0x%016lx\n",
9595 err.bits[0], err.bits[1], err.bits[2], err.bits[3]);
9696
97− fprintf(stderr, " valid: 0x%x\n", err.valid);
98− fprintf(stderr, " overflow: 0x%x\n", err.overflow);
99− fprintf(stderr, " desc_valid: 0x%x\n", err.desc_valid);
100− fprintf(stderr, " wq_idx_valid: 0x%x\n", err.wq_idx_valid);
101− fprintf(stderr, " batch: 0x%x\n", err.batch);
102− fprintf(stderr, " fault_rw: 0x%x\n", err.fault_rw);
103− fprintf(stderr, " priv: 0x%x\n", err.priv);
104− fprintf(stderr, " error: 0x%x\n", err.error);
105− fprintf(stderr, " wq_idx: 0x%x\n", err.wq_idx);
106− fprintf(stderr, " operation: 0x%x\n", err.operation);
107− fprintf(stderr, " pasid: 0x%x\n", err.pasid);
108− fprintf(stderr, " batch_idx: 0x%x\n", err.batch_idx);
109− fprintf(stderr, " invalid_flags: 0x%x\n", err.invalid_flags);
110− fprintf(stderr, " fault_addr: 0x%lx\n", err.fault_addr);
97+ dev_err(device, " valid: 0x%x\n", err.valid);
98+ dev_err(device, " overflow: 0x%x\n", err.overflow);
99+ dev_err(device, " desc_valid: 0x%x\n", err.desc_valid);
100+ dev_err(device, " wq_idx_valid: 0x%x\n", err.wq_idx_valid);
101+ dev_err(device, " batch: 0x%x\n", err.batch);
102+ dev_err(device, " fault_rw: 0x%x\n", err.fault_rw);
103+ dev_err(device, " priv: 0x%x\n", err.priv);
104+ dev_err(device, " error: 0x%x\n", err.error);
105+ dev_err(device, " wq_idx: 0x%x\n", err.wq_idx);
106+ dev_err(device, " operation: 0x%x\n", err.operation);
107+ dev_err(device, " pasid: 0x%x\n", err.pasid);
108+ dev_err(device, " batch_idx: 0x%x\n", err.batch_idx);
109+ dev_err(device, " invalid_flags: 0x%x\n", err.invalid_flags);
110+ dev_err(device, " fault_addr: 0x%lx\n", err.fault_addr);
111111
112112 VFIO_FAIL("Software Error Detected!\n");
113113 }
@@ -256,7 +256,7 @@ static int dsa_completion_wait(struct vfio_pci_device *device, · 142 unmodified lines
256256 if (status == DSA_COMP_SUCCESS)
257257 return 0;
258258
259− printf("Error detected during memcpy operation: 0x%x\n", status);
259+ dev_err(device, "Error detected during memcpy operation: 0x%x\n", status);
260260 return -1;
261261 }
262262
tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c+9−9
@@ -7,7 +7,7 @@ · 6 unmodified lines
77 #include <linux/pci_ids.h>
88 #include <linux/sizes.h>
99
10−#include <vfio_util.h>
10+#include <libvfio.h>
1111
1212 #include "hw.h"
1313 #include "registers.h"
@@ -51,7 +51,7 @@ static int ioat_probe(struct vfio_pci_device *device) · 37 unmodified lines
5151 r = 0;
5252 break;
5353 default:
54− printf("ioat: Unsupported version: 0x%x\n", version);
54+ dev_err(device, "ioat: Unsupported version: 0x%x\n", version);
5555 r = -EINVAL;
5656 }
5757 return r;
@@ -135,13 +135,13 @@ static void ioat_handle_error(struct vfio_pci_device *device) · 77 unmodified lines
135135 {
136136 void *registers = ioat_channel_registers(device);
137137
138− printf("Error detected during memcpy operation!\n"
139− " CHANERR: 0x%x\n"
140− " CHANERR_INT: 0x%x\n"
141− " DMAUNCERRSTS: 0x%x\n",
142− readl(registers + IOAT_CHANERR_OFFSET),
143− vfio_pci_config_readl(device, IOAT_PCI_CHANERR_INT_OFFSET),
144− vfio_pci_config_readl(device, IOAT_PCI_DMAUNCERRSTS_OFFSET));
138+ dev_err(device, "Error detected during memcpy operation!\n"
139+ " CHANERR: 0x%x\n"
140+ " CHANERR_INT: 0x%x\n"
141+ " DMAUNCERRSTS: 0x%x\n",
142+ readl(registers + IOAT_CHANERR_OFFSET),
143+ vfio_pci_config_readl(device, IOAT_PCI_CHANERR_INT_OFFSET),
144+ vfio_pci_config_readl(device, IOAT_PCI_DMAUNCERRSTS_OFFSET));
145145
146146 ioat_reset(device);
147147 }
tools/testing/selftests/vfio/lib/include/libvfio.h+26−0
@@ -0,0 +1,26 @@
1+/* SPDX-License-Identifier: GPL-2.0-only */
2+#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_H
3+#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_H
4+
5+#include <libvfio/assert.h>
6+#include <libvfio/iommu.h>
7+#include <libvfio/iova_allocator.h>
8+#include <libvfio/vfio_pci_device.h>
9+#include <libvfio/vfio_pci_driver.h>
10+
11+/*
12+ * Return the BDF string of the device that the test should use.
13+ *
14+ * If a BDF string is provided by the user on the command line (as the last
15+ * element of argv[]), then this function will return that and decrement argc
16+ * by 1.
17+ *
18+ * Otherwise this function will attempt to use the environment variable
19+ * $VFIO_SELFTESTS_BDF.
20+ *
21+ * If BDF cannot be determined then the test will exit with KSFT_SKIP.
22+ */
23+const char *vfio_selftests_get_bdf(int *argc, char *argv[]);
24+char **vfio_selftests_get_bdfs(int *argc, char *argv[], int *nr_bdfs);
25+
26+#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_H */
tools/testing/selftests/vfio/lib/include/libvfio/assert.h+54−0
@@ -0,0 +1,54 @@
1+/* SPDX-License-Identifier: GPL-2.0-only */
2+#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_ASSERT_H
3+#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_ASSERT_H
4+
5+#include <stdio.h>
6+#include <string.h>
7+#include <sys/ioctl.h>
8+
9+#include "../../../../kselftest.h"
10+
11+#define VFIO_LOG_AND_EXIT(...) do { \
12+ fprintf(stderr, " " __VA_ARGS__); \
13+ fprintf(stderr, "\n"); \
14+ exit(KSFT_FAIL); \
15+} while (0)
16+
17+#define VFIO_ASSERT_OP(_lhs, _rhs, _op, ...) do { \
18+ typeof(_lhs) __lhs = (_lhs); \
19+ typeof(_rhs) __rhs = (_rhs); \
20+ \
21+ if (__lhs _op __rhs) \
22+ break; \
23+ \
24+ fprintf(stderr, "%s:%u: Assertion Failure\n\n", __FILE__, __LINE__); \
25+ fprintf(stderr, " Expression: " #_lhs " " #_op " " #_rhs "\n"); \
26+ fprintf(stderr, " Observed: %#lx %s %#lx\n", \
27+ (u64)__lhs, #_op, (u64)__rhs); \
28+ fprintf(stderr, " [errno: %d - %s]\n", errno, strerror(errno)); \
29+ VFIO_LOG_AND_EXIT(__VA_ARGS__); \
30+} while (0)
31+
32+#define VFIO_ASSERT_EQ(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, ==, ##__VA_ARGS__)
33+#define VFIO_ASSERT_NE(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, !=, ##__VA_ARGS__)
34+#define VFIO_ASSERT_LT(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, <, ##__VA_ARGS__)
35+#define VFIO_ASSERT_LE(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, <=, ##__VA_ARGS__)
36+#define VFIO_ASSERT_GT(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, >, ##__VA_ARGS__)
37+#define VFIO_ASSERT_GE(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, >=, ##__VA_ARGS__)
38+#define VFIO_ASSERT_TRUE(_a, ...) VFIO_ASSERT_NE(false, (_a), ##__VA_ARGS__)
39+#define VFIO_ASSERT_FALSE(_a, ...) VFIO_ASSERT_EQ(false, (_a), ##__VA_ARGS__)
40+#define VFIO_ASSERT_NULL(_a, ...) VFIO_ASSERT_EQ(NULL, _a, ##__VA_ARGS__)
41+#define VFIO_ASSERT_NOT_NULL(_a, ...) VFIO_ASSERT_NE(NULL, _a, ##__VA_ARGS__)
42+
43+#define VFIO_FAIL(_fmt, ...) do { \
44+ fprintf(stderr, "%s:%u: FAIL\n\n", __FILE__, __LINE__); \
45+ VFIO_LOG_AND_EXIT(_fmt, ##__VA_ARGS__); \
46+} while (0)
47+
48+#define ioctl_assert(_fd, _op, _arg) do { \
49+ void *__arg = (_arg); \
50+ int __ret = ioctl((_fd), (_op), (__arg)); \
51+ VFIO_ASSERT_EQ(__ret, 0, "ioctl(%s, %s, %s) returned %d\n", #_fd, #_op, #_arg, __ret); \
52+} while (0)
53+
54+#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_ASSERT_H */
tools/testing/selftests/vfio/lib/include/libvfio/iommu.h+76−0
@@ -0,0 +1,76 @@
1+/* SPDX-License-Identifier: GPL-2.0-only */
2+#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_IOMMU_H
3+#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_IOMMU_H
4+
5+#include <linux/list.h>
6+#include <linux/types.h>
7+
8+#include <libvfio/assert.h>
9+
10+typedef u64 iova_t;
11+
12+struct iommu_mode {
13+ const char *name;
14+ const char *container_path;
15+ unsigned long iommu_type;
16+};
17+
18+extern const char *default_iommu_mode;
19+
20+struct dma_region {
21+ struct list_head link;
22+ void *vaddr;
23+ iova_t iova;
24+ u64 size;
25+};
26+
27+struct iommu {
28+ const struct iommu_mode *mode;
29+ int container_fd;
30+ int iommufd;
31+ u32 ioas_id;
32+ struct list_head dma_regions;
33+};
34+
35+struct iommu *iommu_init(const char *iommu_mode);
36+void iommu_cleanup(struct iommu *iommu);
37+
38+int __iommu_map(struct iommu *iommu, struct dma_region *region);
39+
40+static inline void iommu_map(struct iommu *iommu, struct dma_region *region)
41+{
42+ VFIO_ASSERT_EQ(__iommu_map(iommu, region), 0);
43+}
44+
45+int __iommu_unmap(struct iommu *iommu, struct dma_region *region, u64 *unmapped);
46+
47+static inline void iommu_unmap(struct iommu *iommu, struct dma_region *region)
48+{
49+ VFIO_ASSERT_EQ(__iommu_unmap(iommu, region, NULL), 0);
50+}
51+
52+int __iommu_unmap_all(struct iommu *iommu, u64 *unmapped);
53+
54+static inline void iommu_unmap_all(struct iommu *iommu)
55+{
56+ VFIO_ASSERT_EQ(__iommu_unmap_all(iommu, NULL), 0);
57+}
58+
59+int __iommu_hva2iova(struct iommu *iommu, void *vaddr, iova_t *iova);
60+iova_t iommu_hva2iova(struct iommu *iommu, void *vaddr);
61+
62+struct iommu_iova_range *iommu_iova_ranges(struct iommu *iommu, u32 *nranges);
63+
64+/*
65+ * Generator for VFIO selftests fixture variants that replicate across all
66+ * possible IOMMU modes. Tests must define FIXTURE_VARIANT_ADD_IOMMU_MODE()
67+ * which should then use FIXTURE_VARIANT_ADD() to create the variant.
68+ */
69+#define FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES(...) \
70+FIXTURE_VARIANT_ADD_IOMMU_MODE(vfio_type1_iommu, ##__VA_ARGS__); \
71+FIXTURE_VARIANT_ADD_IOMMU_MODE(vfio_type1v2_iommu, ##__VA_ARGS__); \
72+FIXTURE_VARIANT_ADD_IOMMU_MODE(iommufd_compat_type1, ##__VA_ARGS__); \
73+FIXTURE_VARIANT_ADD_IOMMU_MODE(iommufd_compat_type1v2, ##__VA_ARGS__); \
74+FIXTURE_VARIANT_ADD_IOMMU_MODE(iommufd, ##__VA_ARGS__)
75+
76+#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_IOMMU_H */
tools/testing/selftests/vfio/lib/include/libvfio/iova_allocator.h+23−0
@@ -0,0 +1,23 @@
1+/* SPDX-License-Identifier: GPL-2.0-only */
2+#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_IOVA_ALLOCATOR_H
3+#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_IOVA_ALLOCATOR_H
4+
5+#include <uapi/linux/types.h>
6+#include <linux/list.h>
7+#include <linux/types.h>
8+#include <linux/iommufd.h>
9+
10+#include <libvfio/iommu.h>
11+
12+struct iova_allocator {
13+ struct iommu_iova_range *ranges;
14+ u32 nranges;
15+ u32 range_idx;
16+ u64 range_offset;
17+};
18+
19+struct iova_allocator *iova_allocator_init(struct iommu *iommu);
20+void iova_allocator_cleanup(struct iova_allocator *allocator);
21+iova_t iova_allocator_alloc(struct iova_allocator *allocator, size_t size);
22+
23+#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_IOVA_ALLOCATOR_H */
tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h+125−0
@@ -0,0 +1,125 @@
1+/* SPDX-License-Identifier: GPL-2.0-only */
2+#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DEVICE_H
3+#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DEVICE_H
4+
5+#include <fcntl.h>
6+#include <linux/vfio.h>
7+#include <linux/pci_regs.h>
8+
9+#include <libvfio/assert.h>
10+#include <libvfio/iommu.h>
11+#include <libvfio/vfio_pci_driver.h>
12+
13+struct vfio_pci_bar {
14+ struct vfio_region_info info;
15+ void *vaddr;
16+};
17+
18+struct vfio_pci_device {
19+ const char *bdf;
20+ int fd;
21+ int group_fd;
22+
23+ struct iommu *iommu;
24+
25+ struct vfio_device_info info;
26+ struct vfio_region_info config_space;
27+ struct vfio_pci_bar bars[PCI_STD_NUM_BARS];
28+
29+ struct vfio_irq_info msi_info;
30+ struct vfio_irq_info msix_info;
31+
32+ /* eventfds for MSI and MSI-x interrupts */
33+ int msi_eventfds[PCI_MSIX_FLAGS_QSIZE + 1];
34+
35+ struct vfio_pci_driver driver;
36+};
37+
38+#define dev_info(_dev, _fmt, ...) printf("%s: " _fmt, (_dev)->bdf, ##__VA_ARGS__)
39+#define dev_err(_dev, _fmt, ...) fprintf(stderr, "%s: " _fmt, (_dev)->bdf, ##__VA_ARGS__)
40+
41+struct vfio_pci_device *vfio_pci_device_init(const char *bdf, struct iommu *iommu);
42+void vfio_pci_device_cleanup(struct vfio_pci_device *device);
43+
44+void vfio_pci_device_reset(struct vfio_pci_device *device);
45+
46+void vfio_pci_config_access(struct vfio_pci_device *device, bool write,
47+ size_t config, size_t size, void *data);
48+
49+#define vfio_pci_config_read(_device, _offset, _type) ({ \
50+ _type __data; \
51+ vfio_pci_config_access((_device), false, _offset, sizeof(__data), &__data); \
52+ __data; \
53+})
54+
55+#define vfio_pci_config_readb(_d, _o) vfio_pci_config_read(_d, _o, u8)
56+#define vfio_pci_config_readw(_d, _o) vfio_pci_config_read(_d, _o, u16)
57+#define vfio_pci_config_readl(_d, _o) vfio_pci_config_read(_d, _o, u32)
58+
59+#define vfio_pci_config_write(_device, _offset, _value, _type) do { \
60+ _type __data = (_value); \
61+ vfio_pci_config_access((_device), true, _offset, sizeof(_type), &__data); \
62+} while (0)
63+
64+#define vfio_pci_config_writeb(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u8)
65+#define vfio_pci_config_writew(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u16)
66+#define vfio_pci_config_writel(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u32)
67+
68+void vfio_pci_irq_enable(struct vfio_pci_device *device, u32 index,
69+ u32 vector, int count);
70+void vfio_pci_irq_disable(struct vfio_pci_device *device, u32 index);
71+void vfio_pci_irq_trigger(struct vfio_pci_device *device, u32 index, u32 vector);
72+
73+static inline void fcntl_set_nonblock(int fd)
74+{
75+ int r;
76+
77+ r = fcntl(fd, F_GETFL, 0);
78+ VFIO_ASSERT_NE(r, -1, "F_GETFL failed for fd %d\n", fd);
79+
80+ r = fcntl(fd, F_SETFL, r | O_NONBLOCK);
81+ VFIO_ASSERT_NE(r, -1, "F_SETFL O_NONBLOCK failed for fd %d\n", fd);
82+}
83+
84+static inline void vfio_pci_msi_enable(struct vfio_pci_device *device,
85+ u32 vector, int count)
86+{
87+ vfio_pci_irq_enable(device, VFIO_PCI_MSI_IRQ_INDEX, vector, count);
88+}
89+
90+static inline void vfio_pci_msi_disable(struct vfio_pci_device *device)
91+{
92+ vfio_pci_irq_disable(device, VFIO_PCI_MSI_IRQ_INDEX);
93+}
94+
95+static inline void vfio_pci_msix_enable(struct vfio_pci_device *device,
96+ u32 vector, int count)
97+{
98+ vfio_pci_irq_enable(device, VFIO_PCI_MSIX_IRQ_INDEX, vector, count);
99+}
100+
101+static inline void vfio_pci_msix_disable(struct vfio_pci_device *device)
102+{
103+ vfio_pci_irq_disable(device, VFIO_PCI_MSIX_IRQ_INDEX);
104+}
105+
106+static inline int __to_iova(struct vfio_pci_device *device, void *vaddr, iova_t *iova)
107+{
108+ return __iommu_hva2iova(device->iommu, vaddr, iova);
109+}
110+
111+static inline iova_t to_iova(struct vfio_pci_device *device, void *vaddr)
112+{
113+ return iommu_hva2iova(device->iommu, vaddr);
114+}
115+
116+static inline bool vfio_pci_device_match(struct vfio_pci_device *device,
117+ u16 vendor_id, u16 device_id)
118+{
119+ return (vendor_id == vfio_pci_config_readw(device, PCI_VENDOR_ID)) &&
120+ (device_id == vfio_pci_config_readw(device, PCI_DEVICE_ID));
121+}
122+
123+const char *vfio_pci_get_cdev_path(const char *bdf);
124+
125+#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DEVICE_H */
tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h+97−0
@@ -0,0 +1,97 @@
1+/* SPDX-License-Identifier: GPL-2.0-only */
2+#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DRIVER_H
3+#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DRIVER_H
4+
5+#include <libvfio/iommu.h>
6+
7+struct vfio_pci_device;
8+
9+struct vfio_pci_driver_ops {
10+ const char *name;
11+
12+ /**
13+ * @probe() - Check if the driver supports the given device.
14+ *
15+ * Return: 0 on success, non-0 on failure.
16+ */
17+ int (*probe)(struct vfio_pci_device *device);
18+
19+ /**
20+ * @init() - Initialize the driver for @device.
21+ *
22+ * Must be called after device->driver.region has been initialized.
23+ */
24+ void (*init)(struct vfio_pci_device *device);
25+
26+ /**
27+ * remove() - Deinitialize the driver for @device.
28+ */
29+ void (*remove)(struct vfio_pci_device *device);
30+
31+ /**
32+ * memcpy_start() - Kick off @count repeated memcpy operations from
33+ * [@src, @src + @size) to [@dst, @dst + @size).
34+ *
35+ * Guarantees:
36+ * - The device will attempt DMA reads on [src, src + size).
37+ * - The device will attempt DMA writes on [dst, dst + size).
38+ * - The device will not generate any interrupts.
39+ *
40+ * memcpy_start() returns immediately, it does not wait for the
41+ * copies to complete.
42+ */
43+ void (*memcpy_start)(struct vfio_pci_device *device,
44+ iova_t src, iova_t dst, u64 size, u64 count);
45+
46+ /**
47+ * memcpy_wait() - Wait until the memcpy operations started by
48+ * memcpy_start() have finished.
49+ *
50+ * Guarantees:
51+ * - All in-flight DMAs initiated by memcpy_start() are fully complete
52+ * before memcpy_wait() returns.
53+ *
54+ * Returns non-0 if the driver detects that an error occurred during the
55+ * memcpy, 0 otherwise.
56+ */
57+ int (*memcpy_wait)(struct vfio_pci_device *device);
58+
59+ /**
60+ * send_msi() - Make the device send the MSI device->driver.msi.
61+ *
62+ * Guarantees:
63+ * - The device will send the MSI once.
64+ */
65+ void (*send_msi)(struct vfio_pci_device *device);
66+};
67+
68+struct vfio_pci_driver {
69+ const struct vfio_pci_driver_ops *ops;
70+ bool initialized;
71+ bool memcpy_in_progress;
72+
73+ /* Region to be used by the driver (e.g. for in-memory descriptors) */
74+ struct dma_region region;
75+
76+ /* The maximum size that can be passed to memcpy_start(). */
77+ u64 max_memcpy_size;
78+
79+ /* The maximum count that can be passed to memcpy_start(). */
80+ u64 max_memcpy_count;
81+
82+ /* The MSI vector the device will signal in ops->send_msi(). */
83+ int msi;
84+};
85+
86+void vfio_pci_driver_probe(struct vfio_pci_device *device);
87+void vfio_pci_driver_init(struct vfio_pci_device *device);
88+void vfio_pci_driver_remove(struct vfio_pci_device *device);
89+int vfio_pci_driver_memcpy(struct vfio_pci_device *device,
90+ iova_t src, iova_t dst, u64 size);
91+void vfio_pci_driver_memcpy_start(struct vfio_pci_device *device,
92+ iova_t src, iova_t dst, u64 size,
93+ u64 count);
94+int vfio_pci_driver_memcpy_wait(struct vfio_pci_device *device);
95+void vfio_pci_driver_send_msi(struct vfio_pci_device *device);
96+
97+#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DRIVER_H */
tools/testing/selftests/vfio/lib/include/vfio_util.h+0−331
@@ -1,331 +0,0 @@
1−/* SPDX-License-Identifier: GPL-2.0-only */
2−#ifndef SELFTESTS_VFIO_LIB_INCLUDE_VFIO_UTIL_H
3−#define SELFTESTS_VFIO_LIB_INCLUDE_VFIO_UTIL_H
4−
5−#include <fcntl.h>
6−#include <string.h>
7−
8−#include <uapi/linux/types.h>
9−#include <linux/iommufd.h>
10−#include <linux/list.h>
11−#include <linux/pci_regs.h>
12−#include <linux/vfio.h>
13−
14−#include "../../../kselftest.h"
15−
16−#define VFIO_LOG_AND_EXIT(...) do { \
17− fprintf(stderr, " " __VA_ARGS__); \
18− fprintf(stderr, "\n"); \
19− exit(KSFT_FAIL); \
20−} while (0)
21−
22−#define VFIO_ASSERT_OP(_lhs, _rhs, _op, ...) do { \
23− typeof(_lhs) __lhs = (_lhs); \
24− typeof(_rhs) __rhs = (_rhs); \
25− \
26− if (__lhs _op __rhs) \
27− break; \
28− \
29− fprintf(stderr, "%s:%u: Assertion Failure\n\n", __FILE__, __LINE__); \
30− fprintf(stderr, " Expression: " #_lhs " " #_op " " #_rhs "\n"); \
31− fprintf(stderr, " Observed: %#lx %s %#lx\n", \
32− (u64)__lhs, #_op, (u64)__rhs); \
33− fprintf(stderr, " [errno: %d - %s]\n", errno, strerror(errno)); \
34− VFIO_LOG_AND_EXIT(__VA_ARGS__); \
35−} while (0)
36−
37−#define VFIO_ASSERT_EQ(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, ==, ##__VA_ARGS__)
38−#define VFIO_ASSERT_NE(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, !=, ##__VA_ARGS__)
39−#define VFIO_ASSERT_LT(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, <, ##__VA_ARGS__)
40−#define VFIO_ASSERT_LE(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, <=, ##__VA_ARGS__)
41−#define VFIO_ASSERT_GT(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, >, ##__VA_ARGS__)
42−#define VFIO_ASSERT_GE(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, >=, ##__VA_ARGS__)
43−#define VFIO_ASSERT_TRUE(_a, ...) VFIO_ASSERT_NE(false, (_a), ##__VA_ARGS__)
44−#define VFIO_ASSERT_FALSE(_a, ...) VFIO_ASSERT_EQ(false, (_a), ##__VA_ARGS__)
45−#define VFIO_ASSERT_NULL(_a, ...) VFIO_ASSERT_EQ(NULL, _a, ##__VA_ARGS__)
46−#define VFIO_ASSERT_NOT_NULL(_a, ...) VFIO_ASSERT_NE(NULL, _a, ##__VA_ARGS__)
47−
48−#define VFIO_FAIL(_fmt, ...) do { \
49− fprintf(stderr, "%s:%u: FAIL\n\n", __FILE__, __LINE__); \
50− VFIO_LOG_AND_EXIT(_fmt, ##__VA_ARGS__); \
51−} while (0)
52−
53−struct vfio_iommu_mode {
54− const char *name;
55− const char *container_path;
56− unsigned long iommu_type;
57−};
58−
59−/*
60− * Generator for VFIO selftests fixture variants that replicate across all
61− * possible IOMMU modes. Tests must define FIXTURE_VARIANT_ADD_IOMMU_MODE()
62− * which should then use FIXTURE_VARIANT_ADD() to create the variant.
63− */
64−#define FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES(...) \
65−FIXTURE_VARIANT_ADD_IOMMU_MODE(vfio_type1_iommu, ##__VA_ARGS__); \
66−FIXTURE_VARIANT_ADD_IOMMU_MODE(vfio_type1v2_iommu, ##__VA_ARGS__); \
67−FIXTURE_VARIANT_ADD_IOMMU_MODE(iommufd_compat_type1, ##__VA_ARGS__); \
68−FIXTURE_VARIANT_ADD_IOMMU_MODE(iommufd_compat_type1v2, ##__VA_ARGS__); \
69−FIXTURE_VARIANT_ADD_IOMMU_MODE(iommufd, ##__VA_ARGS__)
70−
71−struct vfio_pci_bar {
72− struct vfio_region_info info;
73− void *vaddr;
74−};
75−
76−typedef u64 iova_t;
77−
78−#define INVALID_IOVA UINT64_MAX
79−
80−struct vfio_dma_region {
81− struct list_head link;
82− void *vaddr;
83− iova_t iova;
84− u64 size;
85−};
86−
87−struct vfio_pci_device;
88−
89−struct vfio_pci_driver_ops {
90− const char *name;
91−
92− /**
93− * @probe() - Check if the driver supports the given device.
94− *
95− * Return: 0 on success, non-0 on failure.
96− */
97− int (*probe)(struct vfio_pci_device *device);
98−
99− /**
100− * @init() - Initialize the driver for @device.
101− *
102− * Must be called after device->driver.region has been initialized.
103− */
104− void (*init)(struct vfio_pci_device *device);
105−
106− /**
107− * remove() - Deinitialize the driver for @device.
108− */
109− void (*remove)(struct vfio_pci_device *device);
110−
111− /**
112− * memcpy_start() - Kick off @count repeated memcpy operations from
113− * [@src, @src + @size) to [@dst, @dst + @size).
114− *
115− * Guarantees:
116− * - The device will attempt DMA reads on [src, src + size).
117− * - The device will attempt DMA writes on [dst, dst + size).
118− * - The device will not generate any interrupts.
119− *
120− * memcpy_start() returns immediately, it does not wait for the
121− * copies to complete.
122− */
123− void (*memcpy_start)(struct vfio_pci_device *device,
124− iova_t src, iova_t dst, u64 size, u64 count);
125−
126− /**
127− * memcpy_wait() - Wait until the memcpy operations started by
128− * memcpy_start() have finished.
129− *
130− * Guarantees:
131− * - All in-flight DMAs initiated by memcpy_start() are fully complete
132− * before memcpy_wait() returns.
133− *
134− * Returns non-0 if the driver detects that an error occurred during the
135− * memcpy, 0 otherwise.
136− */
137− int (*memcpy_wait)(struct vfio_pci_device *device);
138−
139− /**
140− * send_msi() - Make the device send the MSI device->driver.msi.
141− *
142− * Guarantees:
143− * - The device will send the MSI once.
144− */
145− void (*send_msi)(struct vfio_pci_device *device);
146−};
147−
148−struct vfio_pci_driver {
149− const struct vfio_pci_driver_ops *ops;
150− bool initialized;
151− bool memcpy_in_progress;
152−
153− /* Region to be used by the driver (e.g. for in-memory descriptors) */
154− struct vfio_dma_region region;
155−
156− /* The maximum size that can be passed to memcpy_start(). */
157− u64 max_memcpy_size;
158−
159− /* The maximum count that can be passed to memcpy_start(). */
160− u64 max_memcpy_count;
161−
162− /* The MSI vector the device will signal in ops->send_msi(). */
163− int msi;
164−};
165−
166−struct vfio_pci_device {
167− int fd;
168−
169− const struct vfio_iommu_mode *iommu_mode;
170− int group_fd;
171− int container_fd;
172−
173− int iommufd;
174− u32 ioas_id;
175−
176− struct vfio_device_info info;
177− struct vfio_region_info config_space;
178− struct vfio_pci_bar bars[PCI_STD_NUM_BARS];
179−
180− struct vfio_irq_info msi_info;
181− struct vfio_irq_info msix_info;
182−
183− struct list_head dma_regions;
184−
185− /* eventfds for MSI and MSI-x interrupts */
186− int msi_eventfds[PCI_MSIX_FLAGS_QSIZE + 1];
187−
188− struct vfio_pci_driver driver;
189−};
190−
191−struct iova_allocator {
192− struct iommu_iova_range *ranges;
193− u32 nranges;
194− u32 range_idx;
195− u64 range_offset;
196−};
197−
198−/*
199− * Return the BDF string of the device that the test should use.
200− *
201− * If a BDF string is provided by the user on the command line (as the last
202− * element of argv[]), then this function will return that and decrement argc
203− * by 1.
204− *
205− * Otherwise this function will attempt to use the environment variable
206− * $VFIO_SELFTESTS_BDF.
207− *
208− * If BDF cannot be determined then the test will exit with KSFT_SKIP.
209− */
210−const char *vfio_selftests_get_bdf(int *argc, char *argv[]);
211−const char *vfio_pci_get_cdev_path(const char *bdf);
212−
213−extern const char *default_iommu_mode;
214−
215−struct vfio_pci_device *vfio_pci_device_init(const char *bdf, const char *iommu_mode);
216−void vfio_pci_device_cleanup(struct vfio_pci_device *device);
217−void vfio_pci_device_reset(struct vfio_pci_device *device);
218−
219−struct iommu_iova_range *vfio_pci_iova_ranges(struct vfio_pci_device *device,
220− u32 *nranges);
221−
222−struct iova_allocator *iova_allocator_init(struct vfio_pci_device *device);
223−void iova_allocator_cleanup(struct iova_allocator *allocator);
224−iova_t iova_allocator_alloc(struct iova_allocator *allocator, size_t size);
225−
226−int __vfio_pci_dma_map(struct vfio_pci_device *device,
227− struct vfio_dma_region *region);
228−int __vfio_pci_dma_unmap(struct vfio_pci_device *device,
229− struct vfio_dma_region *region,
230− u64 *unmapped);
231−int __vfio_pci_dma_unmap_all(struct vfio_pci_device *device, u64 *unmapped);
232−
233−static inline void vfio_pci_dma_map(struct vfio_pci_device *device,
234− struct vfio_dma_region *region)
235−{
236− VFIO_ASSERT_EQ(__vfio_pci_dma_map(device, region), 0);
237−}
238−
239−static inline void vfio_pci_dma_unmap(struct vfio_pci_device *device,
240− struct vfio_dma_region *region)
241−{
242− VFIO_ASSERT_EQ(__vfio_pci_dma_unmap(device, region, NULL), 0);
243−}
244−
245−static inline void vfio_pci_dma_unmap_all(struct vfio_pci_device *device)
246−{
247− VFIO_ASSERT_EQ(__vfio_pci_dma_unmap_all(device, NULL), 0);
248−}
249−
250−void vfio_pci_config_access(struct vfio_pci_device *device, bool write,
251− size_t config, size_t size, void *data);
252−
253−#define vfio_pci_config_read(_device, _offset, _type) ({ \
254− _type __data; \
255− vfio_pci_config_access((_device), false, _offset, sizeof(__data), &__data); \
256− __data; \
257−})
258−
259−#define vfio_pci_config_readb(_d, _o) vfio_pci_config_read(_d, _o, u8)
260−#define vfio_pci_config_readw(_d, _o) vfio_pci_config_read(_d, _o, u16)
261−#define vfio_pci_config_readl(_d, _o) vfio_pci_config_read(_d, _o, u32)
262−
263−#define vfio_pci_config_write(_device, _offset, _value, _type) do { \
264− _type __data = (_value); \
265− vfio_pci_config_access((_device), true, _offset, sizeof(_type), &__data); \
266−} while (0)
267−
268−#define vfio_pci_config_writeb(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u8)
269−#define vfio_pci_config_writew(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u16)
270−#define vfio_pci_config_writel(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u32)
271−
272−void vfio_pci_irq_enable(struct vfio_pci_device *device, u32 index,
273− u32 vector, int count);
274−void vfio_pci_irq_disable(struct vfio_pci_device *device, u32 index);
275−void vfio_pci_irq_trigger(struct vfio_pci_device *device, u32 index, u32 vector);
276−
277−static inline void fcntl_set_nonblock(int fd)
278−{
279− int r;
280−
281− r = fcntl(fd, F_GETFL, 0);
282− VFIO_ASSERT_NE(r, -1, "F_GETFL failed for fd %d\n", fd);
283−
284− r = fcntl(fd, F_SETFL, r | O_NONBLOCK);
285− VFIO_ASSERT_NE(r, -1, "F_SETFL O_NONBLOCK failed for fd %d\n", fd);
286−}
287−
288−static inline void vfio_pci_msi_enable(struct vfio_pci_device *device,
289− u32 vector, int count)
290−{
291− vfio_pci_irq_enable(device, VFIO_PCI_MSI_IRQ_INDEX, vector, count);
292−}
293−
294−static inline void vfio_pci_msi_disable(struct vfio_pci_device *device)
295−{
296− vfio_pci_irq_disable(device, VFIO_PCI_MSI_IRQ_INDEX);
297−}
298−
299−static inline void vfio_pci_msix_enable(struct vfio_pci_device *device,
300− u32 vector, int count)
301−{
302− vfio_pci_irq_enable(device, VFIO_PCI_MSIX_IRQ_INDEX, vector, count);
303−}
304−
305−static inline void vfio_pci_msix_disable(struct vfio_pci_device *device)
306−{
307− vfio_pci_irq_disable(device, VFIO_PCI_MSIX_IRQ_INDEX);
308−}
309−
310−iova_t __to_iova(struct vfio_pci_device *device, void *vaddr);
311−iova_t to_iova(struct vfio_pci_device *device, void *vaddr);
312−
313−static inline bool vfio_pci_device_match(struct vfio_pci_device *device,
314− u16 vendor_id, u16 device_id)
315−{
316− return (vendor_id == vfio_pci_config_readw(device, PCI_VENDOR_ID)) &&
317− (device_id == vfio_pci_config_readw(device, PCI_DEVICE_ID));
318−}
319−
320−void vfio_pci_driver_probe(struct vfio_pci_device *device);
321−void vfio_pci_driver_init(struct vfio_pci_device *device);
322−void vfio_pci_driver_remove(struct vfio_pci_device *device);
323−int vfio_pci_driver_memcpy(struct vfio_pci_device *device,
324− iova_t src, iova_t dst, u64 size);
325−void vfio_pci_driver_memcpy_start(struct vfio_pci_device *device,
326− iova_t src, iova_t dst, u64 size,
327− u64 count);
328−int vfio_pci_driver_memcpy_wait(struct vfio_pci_device *device);
329−void vfio_pci_driver_send_msi(struct vfio_pci_device *device);
330−
331−#endif /* SELFTESTS_VFIO_LIB_INCLUDE_VFIO_UTIL_H */
tools/testing/selftests/vfio/lib/iommu.c+465−0
@@ -0,0 +1,465 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+#include <dirent.h>
3+#include <fcntl.h>
4+#include <libgen.h>
5+#include <stdint.h>
6+#include <stdlib.h>
7+#include <string.h>
8+#include <unistd.h>
9+
10+#include <sys/eventfd.h>
11+#include <sys/ioctl.h>
12+#include <sys/mman.h>
13+
14+#include <uapi/linux/types.h>
15+#include <linux/limits.h>
16+#include <linux/mman.h>
17+#include <linux/types.h>
18+#include <linux/vfio.h>
19+#include <linux/iommufd.h>
20+
21+#include "../../../kselftest.h"
22+#include <libvfio.h>
23+
24+const char *default_iommu_mode = "iommufd";
25+
26+/* Reminder: Keep in sync with FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES(). */
27+static const struct iommu_mode iommu_modes[] = {
28+ {
29+ .name = "vfio_type1_iommu",
30+ .container_path = "/dev/vfio/vfio",
31+ .iommu_type = VFIO_TYPE1_IOMMU,
32+ },
33+ {
34+ .name = "vfio_type1v2_iommu",
35+ .container_path = "/dev/vfio/vfio",
36+ .iommu_type = VFIO_TYPE1v2_IOMMU,
37+ },
38+ {
39+ .name = "iommufd_compat_type1",
40+ .container_path = "/dev/iommu",
41+ .iommu_type = VFIO_TYPE1_IOMMU,
42+ },
43+ {
44+ .name = "iommufd_compat_type1v2",
45+ .container_path = "/dev/iommu",
46+ .iommu_type = VFIO_TYPE1v2_IOMMU,
47+ },
48+ {
49+ .name = "iommufd",
50+ },
51+};
52+
53+static const struct iommu_mode *lookup_iommu_mode(const char *iommu_mode)
54+{
55+ int i;
56+
57+ if (!iommu_mode)
58+ iommu_mode = default_iommu_mode;
59+
60+ for (i = 0; i < ARRAY_SIZE(iommu_modes); i++) {
61+ if (strcmp(iommu_mode, iommu_modes[i].name))
62+ continue;
63+
64+ return &iommu_modes[i];
65+ }
66+
67+ VFIO_FAIL("Unrecognized IOMMU mode: %s\n", iommu_mode);
68+}
69+
70+int __iommu_hva2iova(struct iommu *iommu, void *vaddr, iova_t *iova)
71+{
72+ struct dma_region *region;
73+
74+ list_for_each_entry(region, &iommu->dma_regions, link) {
75+ if (vaddr < region->vaddr)
76+ continue;
77+
78+ if (vaddr >= region->vaddr + region->size)
79+ continue;
80+
81+ if (iova)
82+ *iova = region->iova + (vaddr - region->vaddr);
83+
84+ return 0;
85+ }
86+
87+ return -ENOENT;
88+}
89+
90+iova_t iommu_hva2iova(struct iommu *iommu, void *vaddr)
91+{
92+ iova_t iova;
93+ int ret;
94+
95+ ret = __iommu_hva2iova(iommu, vaddr, &iova);
96+ VFIO_ASSERT_EQ(ret, 0, "%p is not mapped into the iommu\n", vaddr);
97+
98+ return iova;
99+}
100+
101+static int vfio_iommu_map(struct iommu *iommu, struct dma_region *region)
102+{
103+ struct vfio_iommu_type1_dma_map args = {
104+ .argsz = sizeof(args),
105+ .flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE,
106+ .vaddr = (u64)region->vaddr,
107+ .iova = region->iova,
108+ .size = region->size,
109+ };
110+
111+ if (ioctl(iommu->container_fd, VFIO_IOMMU_MAP_DMA, &args))
112+ return -errno;
113+
114+ return 0;
115+}
116+
117+static int iommufd_map(struct iommu *iommu, struct dma_region *region)
118+{
119+ struct iommu_ioas_map args = {
120+ .size = sizeof(args),
121+ .flags = IOMMU_IOAS_MAP_READABLE |
122+ IOMMU_IOAS_MAP_WRITEABLE |
123+ IOMMU_IOAS_MAP_FIXED_IOVA,
124+ .user_va = (u64)region->vaddr,
125+ .iova = region->iova,
126+ .length = region->size,
127+ .ioas_id = iommu->ioas_id,
128+ };
129+
130+ if (ioctl(iommu->iommufd, IOMMU_IOAS_MAP, &args))
131+ return -errno;
132+
133+ return 0;
134+}
135+
136+int __iommu_map(struct iommu *iommu, struct dma_region *region)
137+{
138+ int ret;
139+
140+ if (iommu->iommufd)
141+ ret = iommufd_map(iommu, region);
142+ else
143+ ret = vfio_iommu_map(iommu, region);
144+
145+ if (ret)
146+ return ret;
147+
148+ list_add(&region->link, &iommu->dma_regions);
149+
150+ return 0;
151+}
152+
153+static int __vfio_iommu_unmap(int fd, u64 iova, u64 size, u32 flags, u64 *unmapped)
154+{
155+ struct vfio_iommu_type1_dma_unmap args = {
156+ .argsz = sizeof(args),
157+ .iova = iova,
158+ .size = size,
159+ .flags = flags,
160+ };
161+
162+ if (ioctl(fd, VFIO_IOMMU_UNMAP_DMA, &args))
163+ return -errno;
164+
165+ if (unmapped)
166+ *unmapped = args.size;
167+
168+ return 0;
169+}
170+
171+static int vfio_iommu_unmap(struct iommu *iommu, struct dma_region *region,
172+ u64 *unmapped)
173+{
174+ return __vfio_iommu_unmap(iommu->container_fd, region->iova,
175+ region->size, 0, unmapped);
176+}
177+
178+static int __iommufd_unmap(int fd, u64 iova, u64 length, u32 ioas_id, u64 *unmapped)
179+{
180+ struct iommu_ioas_unmap args = {
181+ .size = sizeof(args),
182+ .iova = iova,
183+ .length = length,
184+ .ioas_id = ioas_id,
185+ };
186+
187+ if (ioctl(fd, IOMMU_IOAS_UNMAP, &args))
188+ return -errno;
189+
190+ if (unmapped)
191+ *unmapped = args.length;
192+
193+ return 0;
194+}
195+
196+static int iommufd_unmap(struct iommu *iommu, struct dma_region *region,
197+ u64 *unmapped)
198+{
199+ return __iommufd_unmap(iommu->iommufd, region->iova, region->size,
200+ iommu->ioas_id, unmapped);
201+}
202+
203+int __iommu_unmap(struct iommu *iommu, struct dma_region *region, u64 *unmapped)
204+{
205+ int ret;
206+
207+ if (iommu->iommufd)
208+ ret = iommufd_unmap(iommu, region, unmapped);
209+ else
210+ ret = vfio_iommu_unmap(iommu, region, unmapped);
211+
212+ if (ret)
213+ return ret;
214+
215+ list_del_init(&region->link);
216+
217+ return 0;
218+}
219+
220+int __iommu_unmap_all(struct iommu *iommu, u64 *unmapped)
221+{
222+ int ret;
223+ struct dma_region *curr, *next;
224+
225+ if (iommu->iommufd)
226+ ret = __iommufd_unmap(iommu->iommufd, 0, UINT64_MAX,
227+ iommu->ioas_id, unmapped);
228+ else
229+ ret = __vfio_iommu_unmap(iommu->container_fd, 0, 0,
230+ VFIO_DMA_UNMAP_FLAG_ALL, unmapped);
231+
232+ if (ret)
233+ return ret;
234+
235+ list_for_each_entry_safe(curr, next, &iommu->dma_regions, link)
236+ list_del_init(&curr->link);
237+
238+ return 0;
239+}
240+
241+static struct vfio_info_cap_header *next_cap_hdr(void *buf, u32 bufsz,
242+ u32 *cap_offset)
243+{
244+ struct vfio_info_cap_header *hdr;
245+
246+ if (!*cap_offset)
247+ return NULL;
248+
249+ VFIO_ASSERT_LT(*cap_offset, bufsz);
250+ VFIO_ASSERT_GE(bufsz - *cap_offset, sizeof(*hdr));
251+
252+ hdr = (struct vfio_info_cap_header *)((u8 *)buf + *cap_offset);
253+ *cap_offset = hdr->next;
254+
255+ return hdr;
256+}
257+
258+static struct vfio_info_cap_header *vfio_iommu_info_cap_hdr(struct vfio_iommu_type1_info *info,
259+ u16 cap_id)
260+{
261+ struct vfio_info_cap_header *hdr;
262+ u32 cap_offset = info->cap_offset;
263+ u32 max_depth;
264+ u32 depth = 0;
265+
266+ if (!(info->flags & VFIO_IOMMU_INFO_CAPS))
267+ return NULL;
268+
269+ if (cap_offset)
270+ VFIO_ASSERT_GE(cap_offset, sizeof(*info));
271+
272+ max_depth = (info->argsz - sizeof(*info)) / sizeof(*hdr);
273+
274+ while ((hdr = next_cap_hdr(info, info->argsz, &cap_offset))) {
275+ depth++;
276+ VFIO_ASSERT_LE(depth, max_depth, "Capability chain contains a cycle\n");
277+
278+ if (hdr->id == cap_id)
279+ return hdr;
280+ }
281+
282+ return NULL;
283+}
284+
285+/* Return buffer including capability chain, if present. Free with free() */
286+static struct vfio_iommu_type1_info *vfio_iommu_get_info(int container_fd)
287+{
288+ struct vfio_iommu_type1_info *info;
289+
290+ info = malloc(sizeof(*info));
291+ VFIO_ASSERT_NOT_NULL(info);
292+
293+ *info = (struct vfio_iommu_type1_info) {
294+ .argsz = sizeof(*info),
295+ };
296+
297+ ioctl_assert(container_fd, VFIO_IOMMU_GET_INFO, info);
298+ VFIO_ASSERT_GE(info->argsz, sizeof(*info));
299+
300+ info = realloc(info, info->argsz);
301+ VFIO_ASSERT_NOT_NULL(info);
302+
303+ ioctl_assert(container_fd, VFIO_IOMMU_GET_INFO, info);
304+ VFIO_ASSERT_GE(info->argsz, sizeof(*info));
305+
306+ return info;
307+}
308+
309+/*
310+ * Return iova ranges for the device's container. Normalize vfio_iommu_type1 to
311+ * report iommufd's iommu_iova_range. Free with free().
312+ */
313+static struct iommu_iova_range *vfio_iommu_iova_ranges(struct iommu *iommu,
314+ u32 *nranges)
315+{
316+ struct vfio_iommu_type1_info_cap_iova_range *cap_range;
317+ struct vfio_iommu_type1_info *info;
318+ struct vfio_info_cap_header *hdr;
319+ struct iommu_iova_range *ranges = NULL;
320+
321+ info = vfio_iommu_get_info(iommu->container_fd);
322+ hdr = vfio_iommu_info_cap_hdr(info, VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE);
323+ VFIO_ASSERT_NOT_NULL(hdr);
324+
325+ cap_range = container_of(hdr, struct vfio_iommu_type1_info_cap_iova_range, header);
326+ VFIO_ASSERT_GT(cap_range->nr_iovas, 0);
327+
328+ ranges = calloc(cap_range->nr_iovas, sizeof(*ranges));
329+ VFIO_ASSERT_NOT_NULL(ranges);
330+
331+ for (u32 i = 0; i < cap_range->nr_iovas; i++) {
332+ ranges[i] = (struct iommu_iova_range){
333+ .start = cap_range->iova_ranges[i].start,
334+ .last = cap_range->iova_ranges[i].end,
335+ };
336+ }
337+
338+ *nranges = cap_range->nr_iovas;
339+
340+ free(info);
341+ return ranges;
342+}
343+
344+/* Return iova ranges of the device's IOAS. Free with free() */
345+static struct iommu_iova_range *iommufd_iova_ranges(struct iommu *iommu,
346+ u32 *nranges)
347+{
348+ struct iommu_iova_range *ranges;
349+ int ret;
350+
351+ struct iommu_ioas_iova_ranges query = {
352+ .size = sizeof(query),
353+ .ioas_id = iommu->ioas_id,
354+ };
355+
356+ ret = ioctl(iommu->iommufd, IOMMU_IOAS_IOVA_RANGES, &query);
357+ VFIO_ASSERT_EQ(ret, -1);
358+ VFIO_ASSERT_EQ(errno, EMSGSIZE);
359+ VFIO_ASSERT_GT(query.num_iovas, 0);
360+
361+ ranges = calloc(query.num_iovas, sizeof(*ranges));
362+ VFIO_ASSERT_NOT_NULL(ranges);
363+
364+ query.allowed_iovas = (uintptr_t)ranges;
365+
366+ ioctl_assert(iommu->iommufd, IOMMU_IOAS_IOVA_RANGES, &query);
367+ *nranges = query.num_iovas;
368+
369+ return ranges;
370+}
371+
372+static int iova_range_comp(const void *a, const void *b)
373+{
374+ const struct iommu_iova_range *ra = a, *rb = b;
375+
376+ if (ra->start < rb->start)
377+ return -1;
378+
379+ if (ra->start > rb->start)
380+ return 1;
381+
382+ return 0;
383+}
384+
385+/* Return sorted IOVA ranges of the device. Free with free(). */
386+struct iommu_iova_range *iommu_iova_ranges(struct iommu *iommu, u32 *nranges)
387+{
388+ struct iommu_iova_range *ranges;
389+
390+ if (iommu->iommufd)
391+ ranges = iommufd_iova_ranges(iommu, nranges);
392+ else
393+ ranges = vfio_iommu_iova_ranges(iommu, nranges);
394+
395+ if (!ranges)
396+ return NULL;
397+
398+ VFIO_ASSERT_GT(*nranges, 0);
399+
400+ /* Sort and check that ranges are sane and non-overlapping */
401+ qsort(ranges, *nranges, sizeof(*ranges), iova_range_comp);
402+ VFIO_ASSERT_LT(ranges[0].start, ranges[0].last);
403+
404+ for (u32 i = 1; i < *nranges; i++) {
405+ VFIO_ASSERT_LT(ranges[i].start, ranges[i].last);
406+ VFIO_ASSERT_LT(ranges[i - 1].last, ranges[i].start);
407+ }
408+
409+ return ranges;
410+}
411+
412+static u32 iommufd_ioas_alloc(int iommufd)
413+{
414+ struct iommu_ioas_alloc args = {
415+ .size = sizeof(args),
416+ };
417+
418+ ioctl_assert(iommufd, IOMMU_IOAS_ALLOC, &args);
419+ return args.out_ioas_id;
420+}
421+
422+struct iommu *iommu_init(const char *iommu_mode)
423+{
424+ const char *container_path;
425+ struct iommu *iommu;
426+ int version;
427+
428+ iommu = calloc(1, sizeof(*iommu));
429+ VFIO_ASSERT_NOT_NULL(iommu);
430+
431+ INIT_LIST_HEAD(&iommu->dma_regions);
432+
433+ iommu->mode = lookup_iommu_mode(iommu_mode);
434+
435+ container_path = iommu->mode->container_path;
436+ if (container_path) {
437+ iommu->container_fd = open(container_path, O_RDWR);
438+ VFIO_ASSERT_GE(iommu->container_fd, 0, "open(%s) failed\n", container_path);
439+
440+ version = ioctl(iommu->container_fd, VFIO_GET_API_VERSION);
441+ VFIO_ASSERT_EQ(version, VFIO_API_VERSION, "Unsupported version: %d\n", version);
442+ } else {
443+ /*
444+ * Require device->iommufd to be >0 so that a simple non-0 check can be
445+ * used to check if iommufd is enabled. In practice open() will never
446+ * return 0 unless stdin is closed.
447+ */
448+ iommu->iommufd = open("/dev/iommu", O_RDWR);
449+ VFIO_ASSERT_GT(iommu->iommufd, 0);
450+
451+ iommu->ioas_id = iommufd_ioas_alloc(iommu->iommufd);
452+ }
453+
454+ return iommu;
455+}
456+
457+void iommu_cleanup(struct iommu *iommu)
458+{
459+ if (iommu->iommufd)
460+ VFIO_ASSERT_EQ(close(iommu->iommufd), 0);
461+ else
462+ VFIO_ASSERT_EQ(close(iommu->container_fd), 0);
463+
464+ free(iommu);
465+}
tools/testing/selftests/vfio/lib/iova_allocator.c+94−0
@@ -0,0 +1,94 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+#include <dirent.h>
3+#include <fcntl.h>
4+#include <libgen.h>
5+#include <stdint.h>
6+#include <stdlib.h>
7+#include <string.h>
8+#include <unistd.h>
9+
10+#include <sys/eventfd.h>
11+#include <sys/ioctl.h>
12+#include <sys/mman.h>
13+
14+#include <uapi/linux/types.h>
15+#include <linux/iommufd.h>
16+#include <linux/limits.h>
17+#include <linux/mman.h>
18+#include <linux/overflow.h>
19+#include <linux/types.h>
20+#include <linux/vfio.h>
21+
22+#include <libvfio.h>
23+
24+struct iova_allocator *iova_allocator_init(struct iommu *iommu)
25+{
26+ struct iova_allocator *allocator;
27+ struct iommu_iova_range *ranges;
28+ u32 nranges;
29+
30+ ranges = iommu_iova_ranges(iommu, &nranges);
31+ VFIO_ASSERT_NOT_NULL(ranges);
32+
33+ allocator = malloc(sizeof(*allocator));
34+ VFIO_ASSERT_NOT_NULL(allocator);
35+
36+ *allocator = (struct iova_allocator){
37+ .ranges = ranges,
38+ .nranges = nranges,
39+ .range_idx = 0,
40+ .range_offset = 0,
41+ };
42+
43+ return allocator;
44+}
45+
46+void iova_allocator_cleanup(struct iova_allocator *allocator)
47+{
48+ free(allocator->ranges);
49+ free(allocator);
50+}
51+
52+iova_t iova_allocator_alloc(struct iova_allocator *allocator, size_t size)
53+{
54+ VFIO_ASSERT_GT(size, 0, "Invalid size arg, zero\n");
55+ VFIO_ASSERT_EQ(size & (size - 1), 0, "Invalid size arg, non-power-of-2\n");
56+
57+ for (;;) {
58+ struct iommu_iova_range *range;
59+ iova_t iova, last;
60+
61+ VFIO_ASSERT_LT(allocator->range_idx, allocator->nranges,
62+ "IOVA allocator out of space\n");
63+
64+ range = &allocator->ranges[allocator->range_idx];
65+ iova = range->start + allocator->range_offset;
66+
67+ /* Check for sufficient space at the current offset */
68+ if (check_add_overflow(iova, size - 1, &last) ||
69+ last > range->last)
70+ goto next_range;
71+
72+ /* Align iova to size */
73+ iova = last & ~(size - 1);
74+
75+ /* Check for sufficient space at the aligned iova */
76+ if (check_add_overflow(iova, size - 1, &last) ||
77+ last > range->last)
78+ goto next_range;
79+
80+ if (last == range->last) {
81+ allocator->range_idx++;
82+ allocator->range_offset = 0;
83+ } else {
84+ allocator->range_offset = last - range->start + 1;
85+ }
86+
87+ return iova;
88+
89+next_range:
90+ allocator->range_idx++;
91+ allocator->range_offset = 0;
92+ }
93+}
94+
tools/testing/selftests/vfio/lib/libvfio.c+78−0
@@ -0,0 +1,78 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+
3+#include <stdio.h>
4+#include <stdlib.h>
5+
6+#include "../../../kselftest.h"
7+#include <libvfio.h>
8+
9+static bool is_bdf(const char *str)
10+{
11+ unsigned int s, b, d, f;
12+ int length, count;
13+
14+ count = sscanf(str, "%4x:%2x:%2x.%2x%n", &s, &b, &d, &f, &length);
15+ return count == 4 && length == strlen(str);
16+}
17+
18+static char **get_bdfs_cmdline(int *argc, char *argv[], int *nr_bdfs)
19+{
20+ int i;
21+
22+ for (i = *argc - 1; i > 0 && is_bdf(argv[i]); i--)
23+ continue;
24+
25+ i++;
26+ *nr_bdfs = *argc - i;
27+ *argc -= *nr_bdfs;
28+
29+ return *nr_bdfs ? &argv[i] : NULL;
30+}
31+
32+static char *get_bdf_env(void)
33+{
34+ char *bdf;
35+
36+ bdf = getenv("VFIO_SELFTESTS_BDF");
37+ if (!bdf)
38+ return NULL;
39+
40+ VFIO_ASSERT_TRUE(is_bdf(bdf), "Invalid BDF: %s\n", bdf);
41+ return bdf;
42+}
43+
44+char **vfio_selftests_get_bdfs(int *argc, char *argv[], int *nr_bdfs)
45+{
46+ static char *env_bdf;
47+ char **bdfs;
48+
49+ bdfs = get_bdfs_cmdline(argc, argv, nr_bdfs);
50+ if (bdfs)
51+ return bdfs;
52+
53+ env_bdf = get_bdf_env();
54+ if (env_bdf) {
55+ *nr_bdfs = 1;
56+ return &env_bdf;
57+ }
58+
59+ fprintf(stderr, "Unable to determine which device(s) to use, skipping test.\n");
60+ fprintf(stderr, "\n");
61+ fprintf(stderr, "To pass the device address via environment variable:\n");
62+ fprintf(stderr, "\n");
63+ fprintf(stderr, " export VFIO_SELFTESTS_BDF=\"segment:bus:device.function\"\n");
64+ fprintf(stderr, " %s [options]\n", argv[0]);
65+ fprintf(stderr, "\n");
66+ fprintf(stderr, "To pass the device address(es) via argv:\n");
67+ fprintf(stderr, "\n");
68+ fprintf(stderr, " %s [options] segment:bus:device.function ...\n", argv[0]);
69+ fprintf(stderr, "\n");
70+ exit(KSFT_SKIP);
71+}
72+
73+const char *vfio_selftests_get_bdf(int *argc, char *argv[])
74+{
75+ int nr_bdfs;
76+
77+ return vfio_selftests_get_bdfs(argc, argv, &nr_bdfs)[0];
78+}
tools/testing/selftests/vfio/lib/libvfio.mk+14−9
@@ -1,24 +1,29 @@
11 include $(top_srcdir)/scripts/subarch.include
22 ARCH ?= $(SUBARCH)
33
4−VFIO_DIR := $(selfdir)/vfio
4+LIBVFIO_SRCDIR := $(selfdir)/vfio/lib
55
6−LIBVFIO_C := lib/vfio_pci_device.c
7−LIBVFIO_C += lib/vfio_pci_driver.c
6+LIBVFIO_C := iommu.c
7+LIBVFIO_C += iova_allocator.c
8+LIBVFIO_C += libvfio.c
9+LIBVFIO_C += vfio_pci_device.c
10+LIBVFIO_C += vfio_pci_driver.c
811
912 ifeq ($(ARCH:x86_64=x86),x86)
10−LIBVFIO_C += lib/drivers/ioat/ioat.c
11−LIBVFIO_C += lib/drivers/dsa/dsa.c
13+LIBVFIO_C += drivers/ioat/ioat.c
14+LIBVFIO_C += drivers/dsa/dsa.c
1215 endif
1316
14−LIBVFIO_O := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBVFIO_C))
17+LIBVFIO_OUTPUT := $(OUTPUT)/libvfio
18+
19+LIBVFIO_O := $(patsubst %.c, $(LIBVFIO_OUTPUT)/%.o, $(LIBVFIO_C))
1520
1621 LIBVFIO_O_DIRS := $(shell dirname $(LIBVFIO_O) | uniq)
1722 $(shell mkdir -p $(LIBVFIO_O_DIRS))
1823
19−CFLAGS += -I$(VFIO_DIR)/lib/include
24+CFLAGS += -I$(LIBVFIO_SRCDIR)/include
2025
21−$(LIBVFIO_O): $(OUTPUT)/%.o : $(VFIO_DIR)/%.c
26+$(LIBVFIO_O): $(LIBVFIO_OUTPUT)/%.o : $(LIBVFIO_SRCDIR)/%.c
2227 $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $< -o $@
2328
24−EXTRA_CLEAN += $(LIBVFIO_O)
29+EXTRA_CLEAN += $(LIBVFIO_OUTPUT)
tools/testing/selftests/vfio/lib/vfio_pci_device.c+19−537
@@ -20,286 +20,10 @@ · 19 unmodified lines
2020 #include <linux/vfio.h>
2121
2222 #include "../../../kselftest.h"
23−#include <vfio_util.h>
23+#include <libvfio.h>
2424
2525 #define PCI_SYSFS_PATH "/sys/bus/pci/devices"
2626
27−#define ioctl_assert(_fd, _op, _arg) do { \
28− void *__arg = (_arg); \
29− int __ret = ioctl((_fd), (_op), (__arg)); \
30− VFIO_ASSERT_EQ(__ret, 0, "ioctl(%s, %s, %s) returned %d\n", #_fd, #_op, #_arg, __ret); \
31−} while (0)
32−
33−static struct vfio_info_cap_header *next_cap_hdr(void *buf, u32 bufsz,
34− u32 *cap_offset)
35−{
36− struct vfio_info_cap_header *hdr;
37−
38− if (!*cap_offset)
39− return NULL;
40−
41− VFIO_ASSERT_LT(*cap_offset, bufsz);
42− VFIO_ASSERT_GE(bufsz - *cap_offset, sizeof(*hdr));
43−
44− hdr = (struct vfio_info_cap_header *)((u8 *)buf + *cap_offset);
45− *cap_offset = hdr->next;
46−
47− return hdr;
48−}
49−
50−static struct vfio_info_cap_header *vfio_iommu_info_cap_hdr(struct vfio_iommu_type1_info *info,
51− u16 cap_id)
52−{
53− struct vfio_info_cap_header *hdr;
54− u32 cap_offset = info->cap_offset;
55− u32 max_depth;
56− u32 depth = 0;
57−
58− if (!(info->flags & VFIO_IOMMU_INFO_CAPS))
59− return NULL;
60−
61− if (cap_offset)
62− VFIO_ASSERT_GE(cap_offset, sizeof(*info));
63−
64− max_depth = (info->argsz - sizeof(*info)) / sizeof(*hdr);
65−
66− while ((hdr = next_cap_hdr(info, info->argsz, &cap_offset))) {
67− depth++;
68− VFIO_ASSERT_LE(depth, max_depth, "Capability chain contains a cycle\n");
69−
70− if (hdr->id == cap_id)
71− return hdr;
72− }
73−
74− return NULL;
75−}
76−
77−/* Return buffer including capability chain, if present. Free with free() */
78−static struct vfio_iommu_type1_info *vfio_iommu_get_info(struct vfio_pci_device *device)
79−{
80− struct vfio_iommu_type1_info *info;
81−
82− info = malloc(sizeof(*info));
83− VFIO_ASSERT_NOT_NULL(info);
84−
85− *info = (struct vfio_iommu_type1_info) {
86− .argsz = sizeof(*info),
87− };
88−
89− ioctl_assert(device->container_fd, VFIO_IOMMU_GET_INFO, info);
90− VFIO_ASSERT_GE(info->argsz, sizeof(*info));
91−
92− info = realloc(info, info->argsz);
93− VFIO_ASSERT_NOT_NULL(info);
94−
95− ioctl_assert(device->container_fd, VFIO_IOMMU_GET_INFO, info);
96− VFIO_ASSERT_GE(info->argsz, sizeof(*info));
97−
98− return info;
99−}
100−
101−/*
102− * Return iova ranges for the device's container. Normalize vfio_iommu_type1 to
103− * report iommufd's iommu_iova_range. Free with free().
104− */
105−static struct iommu_iova_range *vfio_iommu_iova_ranges(struct vfio_pci_device *device,
106− u32 *nranges)
107−{
108− struct vfio_iommu_type1_info_cap_iova_range *cap_range;
109− struct vfio_iommu_type1_info *info;
110− struct vfio_info_cap_header *hdr;
111− struct iommu_iova_range *ranges = NULL;
112−
113− info = vfio_iommu_get_info(device);
114− hdr = vfio_iommu_info_cap_hdr(info, VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE);
115− VFIO_ASSERT_NOT_NULL(hdr);
116−
117− cap_range = container_of(hdr, struct vfio_iommu_type1_info_cap_iova_range, header);
118− VFIO_ASSERT_GT(cap_range->nr_iovas, 0);
119−
120− ranges = calloc(cap_range->nr_iovas, sizeof(*ranges));
121− VFIO_ASSERT_NOT_NULL(ranges);
122−
123− for (u32 i = 0; i < cap_range->nr_iovas; i++) {
124− ranges[i] = (struct iommu_iova_range){
125− .start = cap_range->iova_ranges[i].start,
126− .last = cap_range->iova_ranges[i].end,
127− };
128− }
129−
130− *nranges = cap_range->nr_iovas;
131−
132− free(info);
133− return ranges;
134−}
135−
136−/* Return iova ranges of the device's IOAS. Free with free() */
137−static struct iommu_iova_range *iommufd_iova_ranges(struct vfio_pci_device *device,
138− u32 *nranges)
139−{
140− struct iommu_iova_range *ranges;
141− int ret;
142−
143− struct iommu_ioas_iova_ranges query = {
144− .size = sizeof(query),
145− .ioas_id = device->ioas_id,
146− };
147−
148− ret = ioctl(device->iommufd, IOMMU_IOAS_IOVA_RANGES, &query);
149− VFIO_ASSERT_EQ(ret, -1);
150− VFIO_ASSERT_EQ(errno, EMSGSIZE);
151− VFIO_ASSERT_GT(query.num_iovas, 0);
152−
153− ranges = calloc(query.num_iovas, sizeof(*ranges));
154− VFIO_ASSERT_NOT_NULL(ranges);
155−
156− query.allowed_iovas = (uintptr_t)ranges;
157−
158− ioctl_assert(device->iommufd, IOMMU_IOAS_IOVA_RANGES, &query);
159− *nranges = query.num_iovas;
160−
161− return ranges;
162−}
163−
164−static int iova_range_comp(const void *a, const void *b)
165−{
166− const struct iommu_iova_range *ra = a, *rb = b;
167−
168− if (ra->start < rb->start)
169− return -1;
170−
171− if (ra->start > rb->start)
172− return 1;
173−
174− return 0;
175−}
176−
177−/* Return sorted IOVA ranges of the device. Free with free(). */
178−struct iommu_iova_range *vfio_pci_iova_ranges(struct vfio_pci_device *device,
179− u32 *nranges)
180−{
181− struct iommu_iova_range *ranges;
182−
183− if (device->iommufd)
184− ranges = iommufd_iova_ranges(device, nranges);
185− else
186− ranges = vfio_iommu_iova_ranges(device, nranges);
187−
188− if (!ranges)
189− return NULL;
190−
191− VFIO_ASSERT_GT(*nranges, 0);
192−
193− /* Sort and check that ranges are sane and non-overlapping */
194− qsort(ranges, *nranges, sizeof(*ranges), iova_range_comp);
195− VFIO_ASSERT_LT(ranges[0].start, ranges[0].last);
196−
197− for (u32 i = 1; i < *nranges; i++) {
198− VFIO_ASSERT_LT(ranges[i].start, ranges[i].last);
199− VFIO_ASSERT_LT(ranges[i - 1].last, ranges[i].start);
200− }
201−
202− return ranges;
203−}
204−
205−struct iova_allocator *iova_allocator_init(struct vfio_pci_device *device)
206−{
207− struct iova_allocator *allocator;
208− struct iommu_iova_range *ranges;
209− u32 nranges;
210−
211− ranges = vfio_pci_iova_ranges(device, &nranges);
212− VFIO_ASSERT_NOT_NULL(ranges);
213−
214− allocator = malloc(sizeof(*allocator));
215− VFIO_ASSERT_NOT_NULL(allocator);
216−
217− *allocator = (struct iova_allocator){
218− .ranges = ranges,
219− .nranges = nranges,
220− .range_idx = 0,
221− .range_offset = 0,
222− };
223−
224− return allocator;
225−}
226−
227−void iova_allocator_cleanup(struct iova_allocator *allocator)
228−{
229− free(allocator->ranges);
230− free(allocator);
231−}
232−
233−iova_t iova_allocator_alloc(struct iova_allocator *allocator, size_t size)
234−{
235− VFIO_ASSERT_GT(size, 0, "Invalid size arg, zero\n");
236− VFIO_ASSERT_EQ(size & (size - 1), 0, "Invalid size arg, non-power-of-2\n");
237−
238− for (;;) {
239− struct iommu_iova_range *range;
240− iova_t iova, last;
241−
242− VFIO_ASSERT_LT(allocator->range_idx, allocator->nranges,
243− "IOVA allocator out of space\n");
244−
245− range = &allocator->ranges[allocator->range_idx];
246− iova = range->start + allocator->range_offset;
247−
248− /* Check for sufficient space at the current offset */
249− if (check_add_overflow(iova, size - 1, &last) ||
250− last > range->last)
251− goto next_range;
252−
253− /* Align iova to size */
254− iova = last & ~(size - 1);
255−
256− /* Check for sufficient space at the aligned iova */
257− if (check_add_overflow(iova, size - 1, &last) ||
258− last > range->last)
259− goto next_range;
260−
261− if (last == range->last) {
262− allocator->range_idx++;
263− allocator->range_offset = 0;
264− } else {
265− allocator->range_offset = last - range->start + 1;
266− }
267−
268− return iova;
269−
270−next_range:
271− allocator->range_idx++;
272− allocator->range_offset = 0;
273− }
274−}
275−
276−iova_t __to_iova(struct vfio_pci_device *device, void *vaddr)
277−{
278− struct vfio_dma_region *region;
279−
280− list_for_each_entry(region, &device->dma_regions, link) {
281− if (vaddr < region->vaddr)
282− continue;
283−
284− if (vaddr >= region->vaddr + region->size)
285− continue;
286−
287− return region->iova + (vaddr - region->vaddr);
288− }
289−
290− return INVALID_IOVA;
291−}
292−
293−iova_t to_iova(struct vfio_pci_device *device, void *vaddr)
294−{
295− iova_t iova;
296−
297− iova = __to_iova(device, vaddr);
298− VFIO_ASSERT_NE(iova, INVALID_IOVA, "%p is not mapped into device.\n", vaddr);
299−
300− return iova;
301−}
302−
30327 static void vfio_pci_irq_set(struct vfio_pci_device *device,
30428 u32 index, u32 vector, u32 count, int *fds)
30529 {
@@ -386,141 +110,6 @@ static void vfio_pci_irq_get(struct vfio_pci_device *device, u32 index, · 80 unmodified lines
386110 ioctl_assert(device->fd, VFIO_DEVICE_GET_IRQ_INFO, irq_info);
387111 }
388112
389−static int vfio_iommu_dma_map(struct vfio_pci_device *device,
390− struct vfio_dma_region *region)
391−{
392− struct vfio_iommu_type1_dma_map args = {
393− .argsz = sizeof(args),
394− .flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE,
395− .vaddr = (u64)region->vaddr,
396− .iova = region->iova,
397− .size = region->size,
398− };
399−
400− if (ioctl(device->container_fd, VFIO_IOMMU_MAP_DMA, &args))
401− return -errno;
402−
403− return 0;
404−}
405−
406−static int iommufd_dma_map(struct vfio_pci_device *device,
407− struct vfio_dma_region *region)
408−{
409− struct iommu_ioas_map args = {
410− .size = sizeof(args),
411− .flags = IOMMU_IOAS_MAP_READABLE |
412− IOMMU_IOAS_MAP_WRITEABLE |
413− IOMMU_IOAS_MAP_FIXED_IOVA,
414− .user_va = (u64)region->vaddr,
415− .iova = region->iova,
416− .length = region->size,
417− .ioas_id = device->ioas_id,
418− };
419−
420− if (ioctl(device->iommufd, IOMMU_IOAS_MAP, &args))
421− return -errno;
422−
423− return 0;
424−}
425−
426−int __vfio_pci_dma_map(struct vfio_pci_device *device,
427− struct vfio_dma_region *region)
428−{
429− int ret;
430−
431− if (device->iommufd)
432− ret = iommufd_dma_map(device, region);
433− else
434− ret = vfio_iommu_dma_map(device, region);
435−
436− if (ret)
437− return ret;
438−
439− list_add(&region->link, &device->dma_regions);
440−
441− return 0;
442−}
443−
444−static int vfio_iommu_dma_unmap(int fd, u64 iova, u64 size, u32 flags,
445− u64 *unmapped)
446−{
447− struct vfio_iommu_type1_dma_unmap args = {
448− .argsz = sizeof(args),
449− .iova = iova,
450− .size = size,
451− .flags = flags,
452− };
453−
454− if (ioctl(fd, VFIO_IOMMU_UNMAP_DMA, &args))
455− return -errno;
456−
457− if (unmapped)
458− *unmapped = args.size;
459−
460− return 0;
461−}
462−
463−static int iommufd_dma_unmap(int fd, u64 iova, u64 length, u32 ioas_id,
464− u64 *unmapped)
465−{
466− struct iommu_ioas_unmap args = {
467− .size = sizeof(args),
468− .iova = iova,
469− .length = length,
470− .ioas_id = ioas_id,
471− };
472−
473− if (ioctl(fd, IOMMU_IOAS_UNMAP, &args))
474− return -errno;
475−
476− if (unmapped)
477− *unmapped = args.length;
478−
479− return 0;
480−}
481−
482−int __vfio_pci_dma_unmap(struct vfio_pci_device *device,
483− struct vfio_dma_region *region, u64 *unmapped)
484−{
485− int ret;
486−
487− if (device->iommufd)
488− ret = iommufd_dma_unmap(device->iommufd, region->iova,
489− region->size, device->ioas_id,
490− unmapped);
491− else
492− ret = vfio_iommu_dma_unmap(device->container_fd, region->iova,
493− region->size, 0, unmapped);
494−
495− if (ret)
496− return ret;
497−
498− list_del_init(&region->link);
499−
500− return 0;
501−}
502−
503−int __vfio_pci_dma_unmap_all(struct vfio_pci_device *device, u64 *unmapped)
504−{
505− int ret;
506− struct vfio_dma_region *curr, *next;
507−
508− if (device->iommufd)
509− ret = iommufd_dma_unmap(device->iommufd, 0, UINT64_MAX,
510− device->ioas_id, unmapped);
511− else
512− ret = vfio_iommu_dma_unmap(device->container_fd, 0, 0,
513− VFIO_DMA_UNMAP_FLAG_ALL, unmapped);
514−
515− if (ret)
516− return ret;
517−
518− list_for_each_entry_safe(curr, next, &device->dma_regions, link)
519− list_del_init(&curr->link);
520−
521− return 0;
522−}
523−
524113 static void vfio_pci_region_get(struct vfio_pci_device *device, int index,
525114 struct vfio_region_info *info)
526115 {
@@ -627,28 +216,26 @@ static void vfio_pci_group_setup(struct vfio_pci_device *device, const char *bdf · 100 unmodified lines
627216 ioctl_assert(device->group_fd, VFIO_GROUP_GET_STATUS, &group_status);
628217 VFIO_ASSERT_TRUE(group_status.flags & VFIO_GROUP_FLAGS_VIABLE);
629218
630− ioctl_assert(device->group_fd, VFIO_GROUP_SET_CONTAINER, &device->container_fd);
219+ ioctl_assert(device->group_fd, VFIO_GROUP_SET_CONTAINER, &device->iommu->container_fd);
631220 }
632221
633222 static void vfio_pci_container_setup(struct vfio_pci_device *device, const char *bdf)
634223 {
635− unsigned long iommu_type = device->iommu_mode->iommu_type;
636− const char *path = device->iommu_mode->container_path;
637− int version;
224+ struct iommu *iommu = device->iommu;
225+ unsigned long iommu_type = iommu->mode->iommu_type;
638226 int ret;
639227
640− device->container_fd = open(path, O_RDWR);
641− VFIO_ASSERT_GE(device->container_fd, 0, "open(%s) failed\n", path);
642−
643− version = ioctl(device->container_fd, VFIO_GET_API_VERSION);
644− VFIO_ASSERT_EQ(version, VFIO_API_VERSION, "Unsupported version: %d\n", version);
645−
646228 vfio_pci_group_setup(device, bdf);
647229
648− ret = ioctl(device->container_fd, VFIO_CHECK_EXTENSION, iommu_type);
230+ ret = ioctl(iommu->container_fd, VFIO_CHECK_EXTENSION, iommu_type);
649231 VFIO_ASSERT_GT(ret, 0, "VFIO IOMMU type %lu not supported\n", iommu_type);
650232
651− ioctl_assert(device->container_fd, VFIO_SET_IOMMU, (void *)iommu_type);
233+ /*
234+ * Allow multiple threads to race to set the IOMMU type on the
235+ * container. The first will succeed and the rest should fail
236+ * because the IOMMU type is already set.
237+ */
238+ (void)ioctl(iommu->container_fd, VFIO_SET_IOMMU, (void *)iommu_type);
652239
653240 device->fd = ioctl(device->group_fd, VFIO_GROUP_GET_DEVICE_FD, bdf);
654241 VFIO_ASSERT_GE(device->fd, 0);
@@ -712,52 +299,6 @@ const char *vfio_pci_get_cdev_path(const char *bdf) · 57 unmodified lines
712299 return cdev_path;
713300 }
714301
715−/* Reminder: Keep in sync with FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES(). */
716−static const struct vfio_iommu_mode iommu_modes[] = {
717− {
718− .name = "vfio_type1_iommu",
719− .container_path = "/dev/vfio/vfio",
720− .iommu_type = VFIO_TYPE1_IOMMU,
721− },
722− {
723− .name = "vfio_type1v2_iommu",
724− .container_path = "/dev/vfio/vfio",
725− .iommu_type = VFIO_TYPE1v2_IOMMU,
726− },
727− {
728− .name = "iommufd_compat_type1",
729− .container_path = "/dev/iommu",
730− .iommu_type = VFIO_TYPE1_IOMMU,
731− },
732− {
733− .name = "iommufd_compat_type1v2",
734− .container_path = "/dev/iommu",
735− .iommu_type = VFIO_TYPE1v2_IOMMU,
736− },
737− {
738− .name = "iommufd",
739− },
740−};
741−
742−const char *default_iommu_mode = "iommufd";
743−
744−static const struct vfio_iommu_mode *lookup_iommu_mode(const char *iommu_mode)
745−{
746− int i;
747−
748− if (!iommu_mode)
749− iommu_mode = default_iommu_mode;
750−
751− for (i = 0; i < ARRAY_SIZE(iommu_modes); i++) {
752− if (strcmp(iommu_mode, iommu_modes[i].name))
753− continue;
754−
755− return &iommu_modes[i];
756− }
757−
758− VFIO_FAIL("Unrecognized IOMMU mode: %s\n", iommu_mode);
759−}
760−
761302 static void vfio_device_bind_iommufd(int device_fd, int iommufd)
762303 {
763304 struct vfio_device_bind_iommufd args = {
@@ -768,16 +309,6 @@ static void vfio_device_bind_iommufd(int device_fd, int iommufd) · 4 unmodified lines
768309 ioctl_assert(device_fd, VFIO_DEVICE_BIND_IOMMUFD, &args);
769310 }
770311
771−static u32 iommufd_ioas_alloc(int iommufd)
772−{
773− struct iommu_ioas_alloc args = {
774− .size = sizeof(args),
775− };
776−
777− ioctl_assert(iommufd, IOMMU_IOAS_ALLOC, &args);
778− return args.out_ioas_id;
779−}
780−
781312 static void vfio_device_attach_iommufd_pt(int device_fd, u32 pt_id)
782313 {
783314 struct vfio_device_attach_iommufd_pt args = {
@@ -796,31 +327,22 @@ static void vfio_pci_iommufd_setup(struct vfio_pci_device *device, const char *b · 12 unmodified lines
796327 VFIO_ASSERT_GE(device->fd, 0);
797328 free((void *)cdev_path);
798329
799− /*
800− * Require device->iommufd to be >0 so that a simple non-0 check can be
801− * used to check if iommufd is enabled. In practice open() will never
802− * return 0 unless stdin is closed.
803− */
804− device->iommufd = open("/dev/iommu", O_RDWR);
805− VFIO_ASSERT_GT(device->iommufd, 0);
806−
807− vfio_device_bind_iommufd(device->fd, device->iommufd);
808− device->ioas_id = iommufd_ioas_alloc(device->iommufd);
809− vfio_device_attach_iommufd_pt(device->fd, device->ioas_id);
330+ vfio_device_bind_iommufd(device->fd, device->iommu->iommufd);
331+ vfio_device_attach_iommufd_pt(device->fd, device->iommu->ioas_id);
810332 }
811333
812−struct vfio_pci_device *vfio_pci_device_init(const char *bdf, const char *iommu_mode)
334+struct vfio_pci_device *vfio_pci_device_init(const char *bdf, struct iommu *iommu)
813335 {
814336 struct vfio_pci_device *device;
815337
816338 device = calloc(1, sizeof(*device));
817339 VFIO_ASSERT_NOT_NULL(device);
818340
819− INIT_LIST_HEAD(&device->dma_regions);
820−
821− device->iommu_mode = lookup_iommu_mode(iommu_mode);
341+ VFIO_ASSERT_NOT_NULL(iommu);
342+ device->iommu = iommu;
343+ device->bdf = bdf;
822344
823− if (device->iommu_mode->container_path)
345+ if (iommu->mode->container_path)
824346 vfio_pci_container_setup(device, bdf);
825347 else
826348 vfio_pci_iommufd_setup(device, bdf);
@@ -849,48 +371,8 @@ void vfio_pci_device_cleanup(struct vfio_pci_device *device) · 22 unmodified lines
849371 VFIO_ASSERT_EQ(close(device->msi_eventfds[i]), 0);
850372 }
851373
852− if (device->iommufd) {
853− VFIO_ASSERT_EQ(close(device->iommufd), 0);
854− } else {
374+ if (device->group_fd)
855375 VFIO_ASSERT_EQ(close(device->group_fd), 0);
856− VFIO_ASSERT_EQ(close(device->container_fd), 0);
857− }
858376
859377 free(device);
860378 }
861−
862−static bool is_bdf(const char *str)
863−{
864− unsigned int s, b, d, f;
865− int length, count;
866−
867− count = sscanf(str, "%4x:%2x:%2x.%2x%n", &s, &b, &d, &f, &length);
868− return count == 4 && length == strlen(str);
869−}
870−
871−const char *vfio_selftests_get_bdf(int *argc, char *argv[])
872−{
873− char *bdf;
874−
875− if (*argc > 1 && is_bdf(argv[*argc - 1]))
876− return argv[--(*argc)];
877−
878− bdf = getenv("VFIO_SELFTESTS_BDF");
879− if (bdf) {
880− VFIO_ASSERT_TRUE(is_bdf(bdf), "Invalid BDF: %s\n", bdf);
881− return bdf;
882− }
883−
884− fprintf(stderr, "Unable to determine which device to use, skipping test.\n");
885− fprintf(stderr, "\n");
886− fprintf(stderr, "To pass the device address via environment variable:\n");
887− fprintf(stderr, "\n");
888− fprintf(stderr, " export VFIO_SELFTESTS_BDF=segment:bus:device.function\n");
889− fprintf(stderr, " %s [options]\n", argv[0]);
890− fprintf(stderr, "\n");
891− fprintf(stderr, "To pass the device address via argv:\n");
892− fprintf(stderr, "\n");
893− fprintf(stderr, " %s [options] segment:bus:device.function\n", argv[0]);
894− fprintf(stderr, "\n");
895− exit(KSFT_SKIP);
896−}
tools/testing/selftests/vfio/lib/vfio_pci_driver.c+1−15
@@ -1,8 +1,6 @@
11 // SPDX-License-Identifier: GPL-2.0-only
2−#include <stdio.h>
3−
42 #include "../../../kselftest.h"
5−#include <vfio_util.h>
3+#include <libvfio.h>
64
75 #ifdef __x86_64__
86 extern struct vfio_pci_driver_ops dsa_ops;
@@ -29,7 +27,6 @@ void vfio_pci_driver_probe(struct vfio_pci_device *device) · 20 unmodified lines
2927 if (ops->probe(device))
3028 continue;
3129
32− printf("Driver found: %s\n", ops->name);
3330 device->driver.ops = ops;
3431 }
3532 }
@@ -58,17 +55,6 @@ void vfio_pci_driver_init(struct vfio_pci_device *device) · 22 unmodified lines
5855 driver->ops->init(device);
5956
6057 driver->initialized = true;
61−
62− printf("%s: region: vaddr %p, iova 0x%lx, size 0x%lx\n",
63− driver->ops->name,
64− driver->region.vaddr,
65− driver->region.iova,
66− driver->region.size);
67−
68− printf("%s: max_memcpy_size 0x%lx, max_memcpy_count 0x%lx\n",
69− driver->ops->name,
70− driver->max_memcpy_size,
71− driver->max_memcpy_count);
7258 }
7359
7460 void vfio_pci_driver_remove(struct vfio_pci_device *device)
tools/testing/selftests/vfio/run.sh+0−109
@@ -1,109 +0,0 @@
1−# SPDX-License-Identifier: GPL-2.0-or-later
2−
3−# Global variables initialized in main() and then used during cleanup() when
4−# the script exits.
5−declare DEVICE_BDF
6−declare NEW_DRIVER
7−declare OLD_DRIVER
8−declare OLD_NUMVFS
9−declare DRIVER_OVERRIDE
10−
11−function write_to() {
12− # Unfortunately set -x does not show redirects so use echo to manually
13− # tell the user what commands are being run.
14− echo "+ echo \"${2}\" > ${1}"
15− echo "${2}" > ${1}
16−}
17−
18−function bind() {
19− write_to /sys/bus/pci/drivers/${2}/bind ${1}
20−}
21−
22−function unbind() {
23− write_to /sys/bus/pci/drivers/${2}/unbind ${1}
24−}
25−
26−function set_sriov_numvfs() {
27− write_to /sys/bus/pci/devices/${1}/sriov_numvfs ${2}
28−}
29−
30−function set_driver_override() {
31− write_to /sys/bus/pci/devices/${1}/driver_override ${2}
32−}
33−
34−function clear_driver_override() {
35− set_driver_override ${1} ""
36−}
37−
38−function cleanup() {
39− if [ "${NEW_DRIVER}" ]; then unbind ${DEVICE_BDF} ${NEW_DRIVER} ; fi
40− if [ "${DRIVER_OVERRIDE}" ]; then clear_driver_override ${DEVICE_BDF} ; fi
41− if [ "${OLD_DRIVER}" ]; then bind ${DEVICE_BDF} ${OLD_DRIVER} ; fi
42− if [ "${OLD_NUMVFS}" ]; then set_sriov_numvfs ${DEVICE_BDF} ${OLD_NUMVFS} ; fi
43−}
44−
45−function usage() {
46− echo "usage: $0 [-d segment:bus:device.function] [-s] [-h] [cmd ...]" >&2
47− echo >&2
48− echo " -d: The BDF of the device to use for the test (required)" >&2
49− echo " -h: Show this help message" >&2
50− echo " -s: Drop into a shell rather than running a command" >&2
51− echo >&2
52− echo " cmd: The command to run and arguments to pass to it." >&2
53− echo " Required when not using -s. The SBDF will be " >&2
54− echo " appended to the argument list." >&2
55− exit 1
56−}
57−
58−function main() {
59− local shell
60−
61− while getopts "d:hs" opt; do
62− case $opt in
63− d) DEVICE_BDF="$OPTARG" ;;
64− s) shell=true ;;
65− *) usage ;;
66− esac
67− done
68−
69− # Shift past all optional arguments.
70− shift $((OPTIND - 1))
71−
72− # Check that the user passed in the command to run.
73− [ ! "${shell}" ] && [ $# = 0 ] && usage
74−
75− # Check that the user passed in a BDF.
76− [ "${DEVICE_BDF}" ] || usage
77−
78− trap cleanup EXIT
79− set -e
80−
81− test -d /sys/bus/pci/devices/${DEVICE_BDF}
82−
83− if [ -f /sys/bus/pci/devices/${DEVICE_BDF}/sriov_numvfs ]; then
84− OLD_NUMVFS=$(cat /sys/bus/pci/devices/${DEVICE_BDF}/sriov_numvfs)
85− set_sriov_numvfs ${DEVICE_BDF} 0
86− fi
87−
88− if [ -L /sys/bus/pci/devices/${DEVICE_BDF}/driver ]; then
89− OLD_DRIVER=$(basename $(readlink -m /sys/bus/pci/devices/${DEVICE_BDF}/driver))
90− unbind ${DEVICE_BDF} ${OLD_DRIVER}
91− fi
92−
93− set_driver_override ${DEVICE_BDF} vfio-pci
94− DRIVER_OVERRIDE=true
95−
96− bind ${DEVICE_BDF} vfio-pci
97− NEW_DRIVER=vfio-pci
98−
99− echo
100− if [ "${shell}" ]; then
101− echo "Dropping into ${SHELL} with VFIO_SELFTESTS_BDF=${DEVICE_BDF}"
102− VFIO_SELFTESTS_BDF=${DEVICE_BDF} ${SHELL}
103− else
104− "$@" ${DEVICE_BDF}
105− fi
106− echo
107−}
108−
109−main "$@"
tools/testing/selftests/vfio/scripts/cleanup.sh+41−0
@@ -0,0 +1,41 @@
1+# SPDX-License-Identifier: GPL-2.0-or-later
2+
3+source $(dirname -- "${BASH_SOURCE[0]}")/lib.sh
4+
5+function cleanup_devices() {
6+ local device_bdf
7+ local device_dir
8+
9+ for device_bdf in "$@"; do
10+ device_dir=${DEVICES_DIR}/${device_bdf}
11+
12+ if [ -f ${device_dir}/vfio-pci ]; then
13+ unbind ${device_bdf} vfio-pci
14+ fi
15+
16+ if [ -f ${device_dir}/driver_override ]; then
17+ clear_driver_override ${device_bdf}
18+ fi
19+
20+ if [ -f ${device_dir}/driver ]; then
21+ bind ${device_bdf} $(cat ${device_dir}/driver)
22+ fi
23+
24+ if [ -f ${device_dir}/sriov_numvfs ]; then
25+ set_sriov_numvfs ${device_bdf} $(cat ${device_dir}/sriov_numvfs)
26+ fi
27+
28+ rm -rf ${device_dir}
29+ done
30+}
31+
32+function main() {
33+ if [ $# = 0 ]; then
34+ cleanup_devices $(ls ${DEVICES_DIR})
35+ rmdir ${DEVICES_DIR}
36+ else
37+ cleanup_devices "$@"
38+ fi
39+}
40+
41+main "$@"
tools/testing/selftests/vfio/scripts/lib.sh+42−0
@@ -0,0 +1,42 @@
1+# SPDX-License-Identifier: GPL-2.0-or-later
2+
3+readonly DEVICES_DIR="${TMPDIR:-/tmp}/vfio-selftests-devices"
4+
5+function write_to() {
6+ # Unfortunately set -x does not show redirects so use echo to manually
7+ # tell the user what commands are being run.
8+ echo "+ echo \"${2}\" > ${1}"
9+ echo "${2}" > ${1}
10+}
11+
12+function get_driver() {
13+ if [ -L /sys/bus/pci/devices/${1}/driver ]; then
14+ basename $(readlink -m /sys/bus/pci/devices/${1}/driver)
15+ fi
16+}
17+
18+function bind() {
19+ write_to /sys/bus/pci/drivers/${2}/bind ${1}
20+}
21+
22+function unbind() {
23+ write_to /sys/bus/pci/drivers/${2}/unbind ${1}
24+}
25+
26+function set_sriov_numvfs() {
27+ write_to /sys/bus/pci/devices/${1}/sriov_numvfs ${2}
28+}
29+
30+function get_sriov_numvfs() {
31+ if [ -f /sys/bus/pci/devices/${1}/sriov_numvfs ]; then
32+ cat /sys/bus/pci/devices/${1}/sriov_numvfs
33+ fi
34+}
35+
36+function set_driver_override() {
37+ write_to /sys/bus/pci/devices/${1}/driver_override ${2}
38+}
39+
40+function clear_driver_override() {
41+ set_driver_override ${1} ""
42+}
tools/testing/selftests/vfio/scripts/run.sh+16−0
@@ -0,0 +1,16 @@
1+# SPDX-License-Identifier: GPL-2.0-or-later
2+
3+source $(dirname -- "${BASH_SOURCE[0]}")/lib.sh
4+
5+function main() {
6+ local device_bdfs=$(ls ${DEVICES_DIR})
7+
8+ if [ -z "${device_bdfs}" ]; then
9+ echo "No devices found, skipping."
10+ exit 4
11+ fi
12+
13+ "$@" ${device_bdfs}
14+}
15+
16+main "$@"
tools/testing/selftests/vfio/scripts/setup.sh+48−0
@@ -0,0 +1,48 @@
1+# SPDX-License-Identifier: GPL-2.0-or-later
2+set -e
3+
4+source $(dirname -- "${BASH_SOURCE[0]}")/lib.sh
5+
6+function main() {
7+ local device_bdf
8+ local device_dir
9+ local numvfs
10+ local driver
11+
12+ if [ $# = 0 ]; then
13+ echo "usage: $0 segment:bus:device.function ..." >&2
14+ exit 1
15+ fi
16+
17+ for device_bdf in "$@"; do
18+ test -d /sys/bus/pci/devices/${device_bdf}
19+
20+ device_dir=${DEVICES_DIR}/${device_bdf}
21+ if [ -d "${device_dir}" ]; then
22+ echo "${device_bdf} has already been set up, exiting."
23+ exit 0
24+ fi
25+
26+ mkdir -p ${device_dir}
27+
28+ numvfs=$(get_sriov_numvfs ${device_bdf})
29+ if [ "${numvfs}" ]; then
30+ set_sriov_numvfs ${device_bdf} 0
31+ echo ${numvfs} > ${device_dir}/sriov_numvfs
32+ fi
33+
34+ driver=$(get_driver ${device_bdf})
35+ if [ "${driver}" ]; then
36+ unbind ${device_bdf} ${driver}
37+ echo ${driver} > ${device_dir}/driver
38+ fi
39+
40+ set_driver_override ${device_bdf} vfio-pci
41+ touch ${device_dir}/driver_override
42+
43+ bind ${device_bdf} vfio-pci
44+ touch ${device_dir}/vfio-pci
45+ done
46+}
47+
48+main "$@"
tools/testing/selftests/vfio/vfio_dma_mapping_test.c+26−20
@@ -10,7 +10,7 @@ · 9 unmodified lines
1010 #include <linux/sizes.h>
1111 #include <linux/vfio.h>
1212
13−#include <vfio_util.h>
13+#include <libvfio.h>
1414
1515 #include "../kselftest_harness.h"
1616
@@ -94,6 +94,7 @@ static int iommu_mapping_get(const char *bdf, u64 iova, · 77 unmodified lines
9494 }
9595
9696 FIXTURE(vfio_dma_mapping_test) {
97+ struct iommu *iommu;
9798 struct vfio_pci_device *device;
9899 struct iova_allocator *iova_allocator;
99100 };
@@ -119,21 +120,23 @@ FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES(anonymous_hugetlb_1gb, SZ_1G, MAP_HUGETLB | · 19 unmodified lines
119120
120121 FIXTURE_SETUP(vfio_dma_mapping_test)
121122 {
122− self->device = vfio_pci_device_init(device_bdf, variant->iommu_mode);
123− self->iova_allocator = iova_allocator_init(self->device);
123+ self->iommu = iommu_init(variant->iommu_mode);
124+ self->device = vfio_pci_device_init(device_bdf, self->iommu);
125+ self->iova_allocator = iova_allocator_init(self->iommu);
124126 }
125127
126128 FIXTURE_TEARDOWN(vfio_dma_mapping_test)
127129 {
128130 iova_allocator_cleanup(self->iova_allocator);
129131 vfio_pci_device_cleanup(self->device);
132+ iommu_cleanup(self->iommu);
130133 }
131134
132135 TEST_F(vfio_dma_mapping_test, dma_map_unmap)
133136 {
134137 const u64 size = variant->size ?: getpagesize();
135138 const int flags = variant->mmap_flags;
136− struct vfio_dma_region region;
139+ struct dma_region region;
137140 struct iommu_mapping mapping;
138141 u64 mapping_size = size;
139142 u64 unmapped;
@@ -150,7 +153,7 @@ TEST_F(vfio_dma_mapping_test, dma_map_unmap) · 10 unmodified lines
150153 region.iova = iova_allocator_alloc(self->iova_allocator, size);
151154 region.size = size;
152155
153− vfio_pci_dma_map(self->device, &region);
156+ iommu_map(self->iommu, &region);
154157 printf("Mapped HVA %p (size 0x%lx) at IOVA 0x%lx\n", region.vaddr, size, region.iova);
155158
156159 ASSERT_EQ(region.iova, to_iova(self->device, region.vaddr));
@@ -192,19 +195,20 @@ TEST_F(vfio_dma_mapping_test, dma_map_unmap) · 35 unmodified lines
192195 }
193196
194197 unmap:
195− rc = __vfio_pci_dma_unmap(self->device, &region, &unmapped);
198+ rc = __iommu_unmap(self->iommu, &region, &unmapped);
196199 ASSERT_EQ(rc, 0);
197200 ASSERT_EQ(unmapped, region.size);
198201 printf("Unmapped IOVA 0x%lx\n", region.iova);
199− ASSERT_EQ(INVALID_IOVA, __to_iova(self->device, region.vaddr));
202+ ASSERT_NE(0, __to_iova(self->device, region.vaddr, NULL));
200203 ASSERT_NE(0, iommu_mapping_get(device_bdf, region.iova, &mapping));
201204
202205 ASSERT_TRUE(!munmap(region.vaddr, size));
203206 }
204207
205208 FIXTURE(vfio_dma_map_limit_test) {
209+ struct iommu *iommu;
206210 struct vfio_pci_device *device;
207− struct vfio_dma_region region;
211+ struct dma_region region;
208212 size_t mmap_size;
209213 };
210214
@@ -223,7 +227,7 @@ FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES(); · 12 unmodified lines
223227
224228 FIXTURE_SETUP(vfio_dma_map_limit_test)
225229 {
226− struct vfio_dma_region *region = &self->region;
230+ struct dma_region *region = &self->region;
227231 struct iommu_iova_range *ranges;
228232 u64 region_size = getpagesize();
229233 iova_t last_iova;
@@ -235,12 +239,13 @@ FIXTURE_SETUP(vfio_dma_map_limit_test) · 5 unmodified lines
235239 */
236240 self->mmap_size = 2 * region_size;
237241
238− self->device = vfio_pci_device_init(device_bdf, variant->iommu_mode);
242+ self->iommu = iommu_init(variant->iommu_mode);
243+ self->device = vfio_pci_device_init(device_bdf, self->iommu);
239244 region->vaddr = mmap(NULL, self->mmap_size, PROT_READ | PROT_WRITE,
240245 MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
241246 ASSERT_NE(region->vaddr, MAP_FAILED);
242247
243− ranges = vfio_pci_iova_ranges(self->device, &nranges);
248+ ranges = iommu_iova_ranges(self->iommu, &nranges);
244249 VFIO_ASSERT_NOT_NULL(ranges);
245250 last_iova = ranges[nranges - 1].last;
246251 free(ranges);
@@ -253,49 +258,50 @@ FIXTURE_SETUP(vfio_dma_map_limit_test) · 6 unmodified lines
253258 FIXTURE_TEARDOWN(vfio_dma_map_limit_test)
254259 {
255260 vfio_pci_device_cleanup(self->device);
261+ iommu_cleanup(self->iommu);
256262 ASSERT_EQ(munmap(self->region.vaddr, self->mmap_size), 0);
257263 }
258264
259265 TEST_F(vfio_dma_map_limit_test, unmap_range)
260266 {
261− struct vfio_dma_region *region = &self->region;
267+ struct dma_region *region = &self->region;
262268 u64 unmapped;
263269 int rc;
264270
265− vfio_pci_dma_map(self->device, region);
271+ iommu_map(self->iommu, region);
266272 ASSERT_EQ(region->iova, to_iova(self->device, region->vaddr));
267273
268− rc = __vfio_pci_dma_unmap(self->device, region, &unmapped);
274+ rc = __iommu_unmap(self->iommu, region, &unmapped);
269275 ASSERT_EQ(rc, 0);
270276 ASSERT_EQ(unmapped, region->size);
271277 }
272278
273279 TEST_F(vfio_dma_map_limit_test, unmap_all)
274280 {
275− struct vfio_dma_region *region = &self->region;
281+ struct dma_region *region = &self->region;
276282 u64 unmapped;
277283 int rc;
278284
279− vfio_pci_dma_map(self->device, region);
285+ iommu_map(self->iommu, region);
280286 ASSERT_EQ(region->iova, to_iova(self->device, region->vaddr));
281287
282− rc = __vfio_pci_dma_unmap_all(self->device, &unmapped);
288+ rc = __iommu_unmap_all(self->iommu, &unmapped);
283289 ASSERT_EQ(rc, 0);
284290 ASSERT_EQ(unmapped, region->size);
285291 }
286292
287293 TEST_F(vfio_dma_map_limit_test, overflow)
288294 {
289− struct vfio_dma_region *region = &self->region;
295+ struct dma_region *region = &self->region;
290296 int rc;
291297
292298 region->iova = ~(iova_t)0 & ~(region->size - 1);
293299 region->size = self->mmap_size;
294300
295− rc = __vfio_pci_dma_map(self->device, region);
301+ rc = __iommu_map(self->iommu, region);
296302 ASSERT_EQ(rc, -EOVERFLOW);
297303
298− rc = __vfio_pci_dma_unmap(self->device, region, NULL);
304+ rc = __iommu_unmap(self->iommu, region, NULL);
299305 ASSERT_EQ(rc, -EOVERFLOW);
300306 }
301307
tools/testing/selftests/vfio/vfio_iommufd_setup_test.c+1−1
@@ -10,7 +10,7 @@ · 9 unmodified lines
1010 #include <sys/ioctl.h>
1111 #include <unistd.h>
1212
13−#include <vfio_util.h>
13+#include <libvfio.h>
1414 #include "../kselftest_harness.h"
1515
1616 static const char iommu_dev_path[] = "/dev/iommu";
tools/testing/selftests/vfio/vfio_pci_device_init_perf_test.c+168−0
@@ -0,0 +1,168 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+#include <pthread.h>
3+#include <sys/ioctl.h>
4+#include <sys/mman.h>
5+
6+#include <linux/sizes.h>
7+#include <linux/time64.h>
8+#include <linux/vfio.h>
9+
10+#include <libvfio.h>
11+
12+#include "../kselftest_harness.h"
13+
14+static char **device_bdfs;
15+static int nr_devices;
16+
17+struct thread_args {
18+ struct iommu *iommu;
19+ int device_index;
20+ struct timespec start;
21+ struct timespec end;
22+ pthread_barrier_t *barrier;
23+};
24+
25+FIXTURE(vfio_pci_device_init_perf_test) {
26+ pthread_t *threads;
27+ pthread_barrier_t barrier;
28+ struct thread_args *thread_args;
29+ struct iommu *iommu;
30+};
31+
32+FIXTURE_VARIANT(vfio_pci_device_init_perf_test) {
33+ const char *iommu_mode;
34+};
35+
36+#define FIXTURE_VARIANT_ADD_IOMMU_MODE(_iommu_mode) \
37+FIXTURE_VARIANT_ADD(vfio_pci_device_init_perf_test, _iommu_mode) { \
38+ .iommu_mode = #_iommu_mode, \
39+}
40+
41+FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES();
42+
43+FIXTURE_SETUP(vfio_pci_device_init_perf_test)
44+{
45+ int i;
46+
47+ self->iommu = iommu_init(variant->iommu_mode);
48+ self->threads = calloc(nr_devices, sizeof(self->threads[0]));
49+ self->thread_args = calloc(nr_devices, sizeof(self->thread_args[0]));
50+
51+ pthread_barrier_init(&self->barrier, NULL, nr_devices);
52+
53+ for (i = 0; i < nr_devices; i++) {
54+ self->thread_args[i].iommu = self->iommu;
55+ self->thread_args[i].barrier = &self->barrier;
56+ self->thread_args[i].device_index = i;
57+ }
58+}
59+
60+FIXTURE_TEARDOWN(vfio_pci_device_init_perf_test)
61+{
62+ iommu_cleanup(self->iommu);
63+ free(self->threads);
64+ free(self->thread_args);
65+}
66+
67+static s64 to_ns(struct timespec ts)
68+{
69+ return (s64)ts.tv_nsec + NSEC_PER_SEC * (s64)ts.tv_sec;
70+}
71+
72+static struct timespec to_timespec(s64 ns)
73+{
74+ struct timespec ts = {
75+ .tv_nsec = ns % NSEC_PER_SEC,
76+ .tv_sec = ns / NSEC_PER_SEC,
77+ };
78+
79+ return ts;
80+}
81+
82+static struct timespec timespec_sub(struct timespec a, struct timespec b)
83+{
84+ return to_timespec(to_ns(a) - to_ns(b));
85+}
86+
87+static struct timespec timespec_min(struct timespec a, struct timespec b)
88+{
89+ return to_ns(a) < to_ns(b) ? a : b;
90+}
91+
92+static struct timespec timespec_max(struct timespec a, struct timespec b)
93+{
94+ return to_ns(a) > to_ns(b) ? a : b;
95+}
96+
97+static void *thread_main(void *__args)
98+{
99+ struct thread_args *args = __args;
100+ struct vfio_pci_device *device;
101+
102+ pthread_barrier_wait(args->barrier);
103+
104+ clock_gettime(CLOCK_MONOTONIC, &args->start);
105+ device = vfio_pci_device_init(device_bdfs[args->device_index], args->iommu);
106+ clock_gettime(CLOCK_MONOTONIC, &args->end);
107+
108+ pthread_barrier_wait(args->barrier);
109+
110+ vfio_pci_device_cleanup(device);
111+ return NULL;
112+}
113+
114+TEST_F(vfio_pci_device_init_perf_test, init)
115+{
116+ struct timespec start = to_timespec(INT64_MAX), end = {};
117+ struct timespec min = to_timespec(INT64_MAX);
118+ struct timespec max = {};
119+ struct timespec avg = {};
120+ struct timespec wall_time;
121+ s64 thread_ns = 0;
122+ int i;
123+
124+ for (i = 0; i < nr_devices; i++) {
125+ pthread_create(&self->threads[i], NULL, thread_main,
126+ &self->thread_args[i]);
127+ }
128+
129+ for (i = 0; i < nr_devices; i++) {
130+ struct thread_args *args = &self->thread_args[i];
131+ struct timespec init_time;
132+
133+ pthread_join(self->threads[i], NULL);
134+
135+ start = timespec_min(start, args->start);
136+ end = timespec_max(end, args->end);
137+
138+ init_time = timespec_sub(args->end, args->start);
139+ min = timespec_min(min, init_time);
140+ max = timespec_max(max, init_time);
141+ thread_ns += to_ns(init_time);
142+ }
143+
144+ avg = to_timespec(thread_ns / nr_devices);
145+ wall_time = timespec_sub(end, start);
146+
147+ printf("Wall time: %lu.%09lus\n",
148+ wall_time.tv_sec, wall_time.tv_nsec);
149+ printf("Min init time (per device): %lu.%09lus\n",
150+ min.tv_sec, min.tv_nsec);
151+ printf("Max init time (per device): %lu.%09lus\n",
152+ max.tv_sec, max.tv_nsec);
153+ printf("Avg init time (per device): %lu.%09lus\n",
154+ avg.tv_sec, avg.tv_nsec);
155+}
156+
157+int main(int argc, char *argv[])
158+{
159+ int i;
160+
161+ device_bdfs = vfio_selftests_get_bdfs(&argc, argv, &nr_devices);
162+
163+ printf("Testing parallel initialization of %d devices:\n", nr_devices);
164+ for (i = 0; i < nr_devices; i++)
165+ printf(" %s\n", device_bdfs[i]);
166+
167+ return test_harness_run(argc, argv);
168+}
tools/testing/selftests/vfio/vfio_pci_device_test.c+9−3
@@ -10,7 +10,7 @@ · 9 unmodified lines
1010 #include <linux/sizes.h>
1111 #include <linux/vfio.h>
1212
13−#include <vfio_util.h>
13+#include <libvfio.h>
1414
1515 #include "../kselftest_harness.h"
1616
@@ -23,17 +23,20 @@ static const char *device_bdf; · 6 unmodified lines
2323 #define MAX_TEST_MSI 16U
2424
2525 FIXTURE(vfio_pci_device_test) {
26+ struct iommu *iommu;
2627 struct vfio_pci_device *device;
2728 };
2829
2930 FIXTURE_SETUP(vfio_pci_device_test)
3031 {
31− self->device = vfio_pci_device_init(device_bdf, default_iommu_mode);
32+ self->iommu = iommu_init(default_iommu_mode);
33+ self->device = vfio_pci_device_init(device_bdf, self->iommu);
3234 }
3335
3436 FIXTURE_TEARDOWN(vfio_pci_device_test)
3537 {
3638 vfio_pci_device_cleanup(self->device);
39+ iommu_cleanup(self->iommu);
3740 }
3841
3942 #define read_pci_id_from_sysfs(_file) ({ \
@@ -99,6 +102,7 @@ TEST_F(vfio_pci_device_test, validate_bars) · 59 unmodified lines
99102 }
100103
101104 FIXTURE(vfio_pci_irq_test) {
105+ struct iommu *iommu;
102106 struct vfio_pci_device *device;
103107 };
104108
@@ -116,12 +120,14 @@ FIXTURE_VARIANT_ADD(vfio_pci_irq_test, msix) { · 11 unmodified lines
116120
117121 FIXTURE_SETUP(vfio_pci_irq_test)
118122 {
119− self->device = vfio_pci_device_init(device_bdf, default_iommu_mode);
123+ self->iommu = iommu_init(default_iommu_mode);
124+ self->device = vfio_pci_device_init(device_bdf, self->iommu);
120125 }
121126
122127 FIXTURE_TEARDOWN(vfio_pci_irq_test)
123128 {
124129 vfio_pci_device_cleanup(self->device);
130+ iommu_cleanup(self->iommu);
125131 }
126132
127133 TEST_F(vfio_pci_irq_test, enable_trigger_disable)
tools/testing/selftests/vfio/vfio_pci_driver_test.c+33−18
@@ -5,7 +5,7 @@ · 4 unmodified lines
55 #include <linux/sizes.h>
66 #include <linux/vfio.h>
77
8−#include <vfio_util.h>
8+#include <libvfio.h>
99
1010 #include "../kselftest_harness.h"
1111
@@ -18,9 +18,9 @@ static const char *device_bdf; · 6 unmodified lines
1818 ASSERT_EQ(EAGAIN, errno); \
1919 } while (0)
2020
21−static void region_setup(struct vfio_pci_device *device,
21+static void region_setup(struct iommu *iommu,
2222 struct iova_allocator *iova_allocator,
23− struct vfio_dma_region *region, u64 size)
23+ struct dma_region *region, u64 size)
2424 {
2525 const int flags = MAP_SHARED | MAP_ANONYMOUS;
2626 const int prot = PROT_READ | PROT_WRITE;
@@ -33,20 +33,20 @@ static void region_setup(struct vfio_pci_device *device, · 6 unmodified lines
3333 region->iova = iova_allocator_alloc(iova_allocator, size);
3434 region->size = size;
3535
36− vfio_pci_dma_map(device, region);
36+ iommu_map(iommu, region);
3737 }
3838
39−static void region_teardown(struct vfio_pci_device *device,
40− struct vfio_dma_region *region)
39+static void region_teardown(struct iommu *iommu, struct dma_region *region)
4140 {
42− vfio_pci_dma_unmap(device, region);
41+ iommu_unmap(iommu, region);
4342 VFIO_ASSERT_EQ(munmap(region->vaddr, region->size), 0);
4443 }
4544
4645 FIXTURE(vfio_pci_driver_test) {
46+ struct iommu *iommu;
4747 struct vfio_pci_device *device;
4848 struct iova_allocator *iova_allocator;
49− struct vfio_dma_region memcpy_region;
49+ struct dma_region memcpy_region;
5050 void *vaddr;
5151 int msi_fd;
5252
@@ -73,13 +73,14 @@ FIXTURE_SETUP(vfio_pci_driver_test) · 20 unmodified lines
7373 {
7474 struct vfio_pci_driver *driver;
7575
76− self->device = vfio_pci_device_init(device_bdf, variant->iommu_mode);
77− self->iova_allocator = iova_allocator_init(self->device);
76+ self->iommu = iommu_init(variant->iommu_mode);
77+ self->device = vfio_pci_device_init(device_bdf, self->iommu);
78+ self->iova_allocator = iova_allocator_init(self->iommu);
7879
7980 driver = &self->device->driver;
8081
81− region_setup(self->device, self->iova_allocator, &self->memcpy_region, SZ_1G);
82− region_setup(self->device, self->iova_allocator, &driver->region, SZ_2M);
82+ region_setup(self->iommu, self->iova_allocator, &self->memcpy_region, SZ_1G);
83+ region_setup(self->iommu, self->iova_allocator, &driver->region, SZ_2M);
8384
8485 /* Any IOVA that doesn't overlap memcpy_region and driver->region. */
8586 self->unmapped_iova = iova_allocator_alloc(self->iova_allocator, SZ_1G);
@@ -108,11 +109,12 @@ FIXTURE_TEARDOWN(vfio_pci_driver_test) · 22 unmodified lines
108109
109110 vfio_pci_driver_remove(self->device);
110111
111− region_teardown(self->device, &self->memcpy_region);
112− region_teardown(self->device, &driver->region);
112+ region_teardown(self->iommu, &self->memcpy_region);
113+ region_teardown(self->iommu, &driver->region);
113114
114115 iova_allocator_cleanup(self->iova_allocator);
115116 vfio_pci_device_cleanup(self->device);
117+ iommu_cleanup(self->iommu);
116118 }
117119
118120 TEST_F(vfio_pci_driver_test, init_remove)
@@ -231,18 +233,31 @@ TEST_F_TIMEOUT(vfio_pci_driver_test, memcpy_storm, 60) · 112 unmodified lines
231233 ASSERT_NO_MSI(self->msi_fd);
232234 }
233235
234−int main(int argc, char *argv[])
236+static bool device_has_selftests_driver(const char *bdf)
235237 {
236238 struct vfio_pci_device *device;
239+ struct iommu *iommu;
240+ bool has_driver;
241+
242+ iommu = iommu_init(default_iommu_mode);
243+ device = vfio_pci_device_init(device_bdf, iommu);
244+
245+ has_driver = !!device->driver.ops;
246+
247+ vfio_pci_device_cleanup(device);
248+ iommu_cleanup(iommu);
237249
250+ return has_driver;
251+}
252+
253+int main(int argc, char *argv[])
254+{
238255 device_bdf = vfio_selftests_get_bdf(&argc, argv);
239256
240− device = vfio_pci_device_init(device_bdf, default_iommu_mode);
241− if (!device->driver.ops) {
257+ if (!device_has_selftests_driver(device_bdf)) {
242258 fprintf(stderr, "No driver found for device %s\n", device_bdf);
243259 return KSFT_SKIP;
244260 }
245− vfio_pci_device_cleanup(device);
246261
247262 return test_harness_run(argc, argv);
248263 }