lael-labs/linux

Documentation/driver-api/pci/p2pdma.rst+74−23

@@ -9,22 +9,48 @@ between two devices on the bus. This type of transaction is henceforth · 8 unmodified lines
9	called Peer-to-Peer (or P2P). However, there are a number of issues that	9	called Peer-to-Peer (or P2P). However, there are a number of issues that
10	make P2P transactions tricky to do in a perfectly safe way.	10	make P2P transactions tricky to do in a perfectly safe way.
11		11
12	One of the biggest issues is that PCI doesn't require forwarding	12	For PCIe the routing of Transaction Layer Packets (TLPs) is well-defined up
13	transactions between hierarchy domains, and in PCIe, each Root Port	13	until they reach a host bridge or root port. If the path includes PCIe switches
14	defines a separate hierarchy domain. To make things worse, there is no	14	then based on the ACS settings the transaction can route entirely within
15	simple way to determine if a given Root Complex supports this or not.	15	the PCIe hierarchy and never reach the root port. The kernel will evaluate
16	(See PCIe r4.0, sec 1.3.1). Therefore, as of this writing, the kernel	16	the PCIe topology and always permit P2P in these well-defined cases.
17	only supports doing P2P when the endpoints involved are all behind the	17
18	same PCI bridge, as such devices are all in the same PCI hierarchy	18	However, if the P2P transaction reaches the host bridge then it might have to
19	domain, and the spec guarantees that all transactions within the	19	hairpin back out the same root port, be routed inside the CPU SOC to another
20	hierarchy will be routable, but it does not require routing	20	PCIe root port, or routed internally to the SOC.
21	between hierarchies.	21
22		22	The PCIe specification doesn't define the forwarding of transactions between
23	The second issue is that to make use of existing interfaces in Linux,	23	hierarchy domains and kernel defaults to blocking such routing. There is an
24	memory that is used for P2P transactions needs to be backed by struct	24	allow list to allow detecting known-good HW, in which case P2P between any
25	pages. However, PCI BARs are not typically cache coherent so there are	25	two PCIe devices will be permitted.
26	a few corner case gotchas with these pages so developers need to	26
27	be careful about what they do with them.	27	Since P2P inherently is doing transactions between two devices it requires two
		28	drivers to be co-operating inside the kernel. The providing driver has to convey
		29	its MMIO to the consuming driver. To meet the driver model lifecycle rules the
		30	MMIO must have all DMA mapping removed, all CPU accesses prevented, all page
		31	table mappings undone before the providing driver completes remove().
		32
		33	This requires the providing and consuming driver to actively work together to
		34	guarantee that the consuming driver has stopped using the MMIO during a removal
		35	cycle. This is done by either a synchronous invalidation shutdown or waiting
		36	for all usage refcounts to reach zero.
		37
		38	At the lowest level the P2P subsystem offers a naked struct p2p_provider that
		39	delegates lifecycle management to the providing driver. It is expected that
		40	drivers using this option will wrap their MMIO memory in DMABUF and use DMABUF
		41	to provide an invalidation shutdown. These MMIO addresess have no struct page, and
		42	if used with mmap() must create special PTEs. As such there are very few
		43	kernel uAPIs that can accept pointers to them; in particular they cannot be used
		44	with read()/write(), including O_DIRECT.
		45
		46	Building on this, the subsystem offers a layer to wrap the MMIO in a ZONE_DEVICE
		47	pgmap of MEMORY_DEVICE_PCI_P2PDMA to create struct pages. The lifecycle of
		48	pgmap ensures that when the pgmap is destroyed all other drivers have stopped
		49	using the MMIO. This option works with O_DIRECT flows, in some cases, if the
		50	underlying subsystem supports handling MEMORY_DEVICE_PCI_P2PDMA through
		51	FOLL_PCI_P2PDMA. The use of FOLL_LONGTERM is prevented. As this relies on pgmap
		52	it also relies on architecture support along with alignment and minimum size
		53	limitations.
28		54
29		55
30	Driver Writer's Guide	56	Driver Writer's Guide
@@ -114,14 +140,39 @@ allocating scatter-gather lists with P2P memory. · 83 unmodified lines
114	Struct Page Caveats	140	Struct Page Caveats
115	-------------------	141	-------------------
116		142
117	Driver writers should be very careful about not passing these special	143	While the MEMORY_DEVICE_PCI_P2PDMA pages can be installed in VMAs,
118	struct pages to code that isn't prepared for it. At this time, the kernel	144	pin_user_pages() and related will not return them unless FOLL_PCI_P2PDMA is set.
119	interfaces do not have any checks for ensuring this. This obviously
120	precludes passing these pages to userspace.
121		145
122	P2P memory is also technically IO memory but should never have any side	146	The MEMORY_DEVICE_PCI_P2PDMA pages require care to support in the kernel. The
123	effects behind it. Thus, the order of loads and stores should not be important	147	KVA is still MMIO and must still be accessed through the normal
124	and ioreadX(), iowriteX() and friends should not be necessary.	148	readX()/writeX()/etc helpers. Direct CPU access (e.g. memcpy) is forbidden, just
		149	like any other MMIO mapping. While this will actually work on some
		150	architectures, others will experience corruption or just crash in the kernel.
		151	Supporting FOLL_PCI_P2PDMA in a subsystem requires scrubbing it to ensure no CPU
		152	access happens.
		153
		154
		155	Usage With DMABUF
		156	=================
		157
		158	DMABUF provides an alternative to the above struct page-based
		159	client/provider/orchestrator system and should be used when struct page
		160	doesn't exist. In this mode the exporting driver will wrap
		161	some of its MMIO in a DMABUF and give the DMABUF FD to userspace.
		162
		163	Userspace can then pass the FD to an importing driver which will ask the
		164	exporting driver to map it to the importer.
		165
		166	In this case the initiator and target pci_devices are known and the P2P subsystem
		167	is used to determine the mapping type. The phys_addr_t-based DMA API is used to
		168	establish the dma_addr_t.
		169
		170	Lifecycle is controlled by DMABUF move_notify(). When the exporting driver wants
		171	to remove() it must deliver an invalidation shutdown to all DMABUF importing
		172	drivers through move_notify() and synchronously DMA unmap all the MMIO.
		173
		174	No importing driver can continue to have a DMA map to the MMIO after the
		175	exporting driver has destroyed its p2p_provider.
125		176
126		177
127	P2P DMA Support Library	178	P2P DMA Support Library

block/blk-mq-dma.c+1−1

@@ -84,7 +84,7 @@ static inline bool blk_can_dma_map_iova(struct request *req, · 83 unmodified lines
84		84
85	static bool blk_dma_map_bus(struct blk_dma_iter iter, struct phys_vec vec)	85	static bool blk_dma_map_bus(struct blk_dma_iter iter, struct phys_vec vec)
86	{	86	{
87	iter->addr = pci_p2pdma_bus_addr_map(&iter->p2pdma, vec->paddr);	87	iter->addr = pci_p2pdma_bus_addr_map(iter->p2pdma.mem, vec->paddr);
88	iter->len = vec->len;	88	iter->len = vec->len;
89	return true;	89	return true;
90	}	90	}

drivers/crypto/hisilicon/qm.c+27−0

@@ -3032,11 +3032,36 @@ static void qm_put_pci_res(struct hisi_qm *qm) · 3031 unmodified lines
3032	pci_release_mem_regions(pdev);	3032	pci_release_mem_regions(pdev);
3033	}	3033	}
3034		3034
		3035	static void hisi_mig_region_clear(struct hisi_qm *qm)
		3036	{
		3037	u32 val;
		3038
		3039	/* Clear migration region set of PF */
		3040	if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) {
		3041	val = readl(qm->io_base + QM_MIG_REGION_SEL);
		3042	val &= ~QM_MIG_REGION_EN;
		3043	writel(val, qm->io_base + QM_MIG_REGION_SEL);
		3044	}
		3045	}
		3046
		3047	static void hisi_mig_region_enable(struct hisi_qm *qm)
		3048	{
		3049	u32 val;
		3050
		3051	/* Select migration region of PF */
		3052	if (qm->fun_type == QM_HW_PF && qm->ver > QM_HW_V3) {
		3053	val = readl(qm->io_base + QM_MIG_REGION_SEL);
		3054	val \|= QM_MIG_REGION_EN;
		3055	writel(val, qm->io_base + QM_MIG_REGION_SEL);
		3056	}
		3057	}
		3058
3035	static void hisi_qm_pci_uninit(struct hisi_qm *qm)	3059	static void hisi_qm_pci_uninit(struct hisi_qm *qm)
3036	{	3060	{
3037	struct pci_dev *pdev = qm->pdev;	3061	struct pci_dev *pdev = qm->pdev;
3038		3062
3039	pci_free_irq_vectors(pdev);	3063	pci_free_irq_vectors(pdev);
		3064	hisi_mig_region_clear(qm);
3040	qm_put_pci_res(qm);	3065	qm_put_pci_res(qm);
3041	pci_disable_device(pdev);	3066	pci_disable_device(pdev);
3042	}	3067	}
@@ -5752,6 +5777,7 @@ int hisi_qm_init(struct hisi_qm *qm) · 2709 unmodified lines
5752	goto err_free_qm_memory;	5777	goto err_free_qm_memory;
5753		5778
5754	qm_cmd_init(qm);	5779	qm_cmd_init(qm);
		5780	hisi_mig_region_enable(qm);
5755		5781
5756	return 0;	5782	return 0;
5757		5783
@@ -5890,6 +5916,7 @@ static int qm_rebuild_for_resume(struct hisi_qm *qm) · 132 unmodified lines
5890	}	5916	}
5891		5917
5892	qm_cmd_init(qm);	5918	qm_cmd_init(qm);
		5919	hisi_mig_region_enable(qm);
5893	hisi_qm_dev_err_init(qm);	5920	hisi_qm_dev_err_init(qm);
5894	/* Set the doorbell timeout to QM_DB_TIMEOUT_CFG ns. */	5921	/* Set the doorbell timeout to QM_DB_TIMEOUT_CFG ns. */
5895	writel(QM_DB_TIMEOUT_SET, qm->io_base + QM_DB_TIMEOUT_CFG);	5922	writel(QM_DB_TIMEOUT_SET, qm->io_base + QM_DB_TIMEOUT_CFG);

drivers/dma-buf/Makefile+1−1

@@ -1,6 +1,6 @@
1	# SPDX-License-Identifier: GPL-2.0-only	1	# SPDX-License-Identifier: GPL-2.0-only
2	obj-y := dma-buf.o dma-fence.o dma-fence-array.o dma-fence-chain.o \	2	obj-y := dma-buf.o dma-fence.o dma-fence-array.o dma-fence-chain.o \
3	dma-fence-unwrap.o dma-resv.o	3	dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o
4	obj-$(CONFIG_DMABUF_HEAPS) += dma-heap.o	4	obj-$(CONFIG_DMABUF_HEAPS) += dma-heap.o
5	obj-$(CONFIG_DMABUF_HEAPS) += heaps/	5	obj-$(CONFIG_DMABUF_HEAPS) += heaps/
6	obj-$(CONFIG_SYNC_FILE) += sync_file.o	6	obj-$(CONFIG_SYNC_FILE) += sync_file.o

drivers/dma-buf/dma-buf-mapping.c+248−0

@@ -0,0 +1,248 @@
	1	// SPDX-License-Identifier: GPL-2.0-only
	2	/*
	3	* DMA BUF Mapping Helpers
	4	*
	5	*/
	6	#include <linux/dma-buf-mapping.h>
	7	#include <linux/dma-resv.h>
	8
	9	static struct scatterlist fill_sg_entry(struct scatterlist sgl, size_t length,
	10	dma_addr_t addr)
	11	{
	12	unsigned int len, nents;
	13	int i;
	14
	15	nents = DIV_ROUND_UP(length, UINT_MAX);
	16	for (i = 0; i < nents; i++) {
	17	len = min_t(size_t, length, UINT_MAX);
	18	length -= len;
	19	/*
	20	* DMABUF abuses scatterlist to create a scatterlist
	21	* that does not have any CPU list, only the DMA list.
	22	* Always set the page related values to NULL to ensure
	23	* importers can't use it. The phys_addr based DMA API
	24	* does not require the CPU list for mapping or unmapping.
	25	*/
	26	sg_set_page(sgl, NULL, 0, 0);
	27	sg_dma_address(sgl) = addr + (dma_addr_t)i * UINT_MAX;
	28	sg_dma_len(sgl) = len;
	29	sgl = sg_next(sgl);
	30	}
	31
	32	return sgl;
	33	}
	34
	35	static unsigned int calc_sg_nents(struct dma_iova_state *state,
	36	struct dma_buf_phys_vec *phys_vec,
	37	size_t nr_ranges, size_t size)
	38	{
	39	unsigned int nents = 0;
	40	size_t i;
	41
	42	if (!state \|\| !dma_use_iova(state)) {
	43	for (i = 0; i < nr_ranges; i++)
	44	nents += DIV_ROUND_UP(phys_vec[i].len, UINT_MAX);
	45	} else {
	46	/*
	47	* In IOVA case, there is only one SG entry which spans
	48	* for whole IOVA address space, but we need to make sure
	49	* that it fits sg->length, maybe we need more.
	50	*/
	51	nents = DIV_ROUND_UP(size, UINT_MAX);
	52	}
	53
	54	return nents;
	55	}
	56
	57	/**
	58	* struct dma_buf_dma - holds DMA mapping information
	59	* @sgt: Scatter-gather table
	60	* @state: DMA IOVA state relevant in IOMMU-based DMA
	61	* @size: Total size of DMA transfer
	62	*/
	63	struct dma_buf_dma {
	64	struct sg_table sgt;
	65	struct dma_iova_state *state;
	66	size_t size;
	67	};
	68
	69	/**
	70	* dma_buf_phys_vec_to_sgt - Returns the scatterlist table of the attachment
	71	* from arrays of physical vectors. This funciton is intended for MMIO memory
	72	* only.
	73	* @attach: [in] attachment whose scatterlist is to be returned
	74	* @provider: [in] p2pdma provider
	75	* @phys_vec: [in] array of physical vectors
	76	* @nr_ranges: [in] number of entries in phys_vec array
	77	* @size: [in] total size of phys_vec
	78	* @dir: [in] direction of DMA transfer
	79	*
	80	* Returns sg_table containing the scatterlist to be returned; returns ERR_PTR
	81	* on error. May return -EINTR if it is interrupted by a signal.
	82	*
	83	* On success, the DMA addresses and lengths in the returned scatterlist are
	84	* PAGE_SIZE aligned.
	85	*
	86	* A mapping must be unmapped by using dma_buf_free_sgt().
	87	*
	88	* NOTE: This function is intended for exporters. If direct traffic routing is
	89	* mandatory exporter should call routing pci_p2pdma_map_type() before calling
	90	* this function.
	91	*/
	92	struct sg_table dma_buf_phys_vec_to_sgt(struct dma_buf_attachment attach,
	93	struct p2pdma_provider *provider,
	94	struct dma_buf_phys_vec *phys_vec,
	95	size_t nr_ranges, size_t size,
	96	enum dma_data_direction dir)
	97	{
	98	unsigned int nents, mapped_len = 0;
	99	struct dma_buf_dma *dma;
	100	struct scatterlist *sgl;
	101	dma_addr_t addr;
	102	size_t i;
	103	int ret;
	104
	105	dma_resv_assert_held(attach->dmabuf->resv);
	106
	107	if (WARN_ON(!attach \|\| !attach->dmabuf \|\| !provider))
	108	/* This function is supposed to work on MMIO memory only */
	109	return ERR_PTR(-EINVAL);
	110
	111	dma = kzalloc(sizeof(*dma), GFP_KERNEL);
	112	if (!dma)
	113	return ERR_PTR(-ENOMEM);
	114
	115	switch (pci_p2pdma_map_type(provider, attach->dev)) {
	116	case PCI_P2PDMA_MAP_BUS_ADDR:
	117	/*
	118	* There is no need in IOVA at all for this flow.
	119	*/
	120	break;
	121	case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE:
	122	dma->state = kzalloc(sizeof(*dma->state), GFP_KERNEL);
	123	if (!dma->state) {
	124	ret = -ENOMEM;
	125	goto err_free_dma;
	126	}
	127
	128	dma_iova_try_alloc(attach->dev, dma->state, 0, size);
	129	break;
	130	default:
	131	ret = -EINVAL;
	132	goto err_free_dma;
	133	}
	134
	135	nents = calc_sg_nents(dma->state, phys_vec, nr_ranges, size);
	136	ret = sg_alloc_table(&dma->sgt, nents, GFP_KERNEL \| __GFP_ZERO);
	137	if (ret)
	138	goto err_free_state;
	139
	140	sgl = dma->sgt.sgl;
	141
	142	for (i = 0; i < nr_ranges; i++) {
	143	if (!dma->state) {
	144	addr = pci_p2pdma_bus_addr_map(provider,
	145	phys_vec[i].paddr);
	146	} else if (dma_use_iova(dma->state)) {
	147	ret = dma_iova_link(attach->dev, dma->state,
	148	phys_vec[i].paddr, 0,
	149	phys_vec[i].len, dir,
	150	DMA_ATTR_MMIO);
	151	if (ret)
	152	goto err_unmap_dma;
	153
	154	mapped_len += phys_vec[i].len;
	155	} else {
	156	addr = dma_map_phys(attach->dev, phys_vec[i].paddr,
	157	phys_vec[i].len, dir,
	158	DMA_ATTR_MMIO);
	159	ret = dma_mapping_error(attach->dev, addr);
	160	if (ret)
	161	goto err_unmap_dma;
	162	}
	163
	164	if (!dma->state \|\| !dma_use_iova(dma->state))
	165	sgl = fill_sg_entry(sgl, phys_vec[i].len, addr);
	166	}
	167
	168	if (dma->state && dma_use_iova(dma->state)) {
	169	WARN_ON_ONCE(mapped_len != size);
	170	ret = dma_iova_sync(attach->dev, dma->state, 0, mapped_len);
	171	if (ret)
	172	goto err_unmap_dma;
	173
	174	sgl = fill_sg_entry(sgl, mapped_len, dma->state->addr);
	175	}
	176
	177	dma->size = size;
	178
	179	/*
	180	* No CPU list included — set orig_nents = 0 so others can detect
	181	* this via SG table (use nents only).
	182	*/
	183	dma->sgt.orig_nents = 0;
	184
	185
	186	/*
	187	* SGL must be NULL to indicate that SGL is the last one
	188	* and we allocated correct number of entries in sg_alloc_table()
	189	*/
	190	WARN_ON_ONCE(sgl);
	191	return &dma->sgt;
	192
	193	err_unmap_dma:
	194	if (!i \|\| !dma->state) {
	195	; /* Do nothing */
	196	} else if (dma_use_iova(dma->state)) {
	197	dma_iova_destroy(attach->dev, dma->state, mapped_len, dir,
	198	DMA_ATTR_MMIO);
	199	} else {
	200	for_each_sgtable_dma_sg(&dma->sgt, sgl, i)
	201	dma_unmap_phys(attach->dev, sg_dma_address(sgl),
	202	sg_dma_len(sgl), dir, DMA_ATTR_MMIO);
	203	}
	204	sg_free_table(&dma->sgt);
	205	err_free_state:
	206	kfree(dma->state);
	207	err_free_dma:
	208	kfree(dma);
	209	return ERR_PTR(ret);
	210	}
	211	EXPORT_SYMBOL_NS_GPL(dma_buf_phys_vec_to_sgt, "DMA_BUF");
	212
	213	/**
	214	* dma_buf_free_sgt- unmaps the buffer
	215	* @attach: [in] attachment to unmap buffer from
	216	* @sgt: [in] scatterlist info of the buffer to unmap
	217	* @dir: [in] direction of DMA transfer
	218	*
	219	* This unmaps a DMA mapping for @attached obtained
	220	* by dma_buf_phys_vec_to_sgt().
	221	*/
	222	void dma_buf_free_sgt(struct dma_buf_attachment attach, struct sg_table sgt,
	223	enum dma_data_direction dir)
	224	{
	225	struct dma_buf_dma *dma = container_of(sgt, struct dma_buf_dma, sgt);
	226	int i;
	227
	228	dma_resv_assert_held(attach->dmabuf->resv);
	229
	230	if (!dma->state) {
	231	; /* Do nothing */
	232	} else if (dma_use_iova(dma->state)) {
	233	dma_iova_destroy(attach->dev, dma->state, dma->size, dir,
	234	DMA_ATTR_MMIO);
	235	} else {
	236	struct scatterlist *sgl;
	237
	238	for_each_sgtable_dma_sg(sgt, sgl, i)
	239	dma_unmap_phys(attach->dev, sg_dma_address(sgl),
	240	sg_dma_len(sgl), dir, DMA_ATTR_MMIO);
	241	}
	242
	243	sg_free_table(sgt);
	244	kfree(dma->state);
	245	kfree(dma);
	246
	247	}
	248	EXPORT_SYMBOL_NS_GPL(dma_buf_free_sgt, "DMA_BUF");

drivers/gpu/drm/i915/gvt/kvmgt.c+117−146

@@ -1141,6 +1141,122 @@ static int intel_vgpu_set_irqs(struct intel_vgpu *vgpu, u32 flags, · 1140 unmodified lines
1141	return func(vgpu, index, start, count, flags, data);	1141	return func(vgpu, index, start, count, flags, data);
1142	}	1142	}
1143		1143
		1144	static int intel_vgpu_ioctl_get_region_info(struct vfio_device *vfio_dev,
		1145	struct vfio_region_info *info,
		1146	struct vfio_info_cap *caps)
		1147	{
		1148	struct vfio_region_info_cap_sparse_mmap *sparse = NULL;
		1149	struct intel_vgpu *vgpu = vfio_dev_to_vgpu(vfio_dev);
		1150	int nr_areas = 1;
		1151	int cap_type_id;
		1152	unsigned int i;
		1153	int ret;
		1154
		1155	switch (info->index) {
		1156	case VFIO_PCI_CONFIG_REGION_INDEX:
		1157	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
		1158	info->size = vgpu->gvt->device_info.cfg_space_size;
		1159	info->flags = VFIO_REGION_INFO_FLAG_READ \|
		1160	VFIO_REGION_INFO_FLAG_WRITE;
		1161	break;
		1162	case VFIO_PCI_BAR0_REGION_INDEX:
		1163	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
		1164	info->size = vgpu->cfg_space.bar[info->index].size;
		1165	if (!info->size) {
		1166	info->flags = 0;
		1167	break;
		1168	}
		1169
		1170	info->flags = VFIO_REGION_INFO_FLAG_READ \|
		1171	VFIO_REGION_INFO_FLAG_WRITE;
		1172	break;
		1173	case VFIO_PCI_BAR1_REGION_INDEX:
		1174	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
		1175	info->size = 0;
		1176	info->flags = 0;
		1177	break;
		1178	case VFIO_PCI_BAR2_REGION_INDEX:
		1179	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
		1180	info->flags = VFIO_REGION_INFO_FLAG_CAPS \|
		1181	VFIO_REGION_INFO_FLAG_MMAP \|
		1182	VFIO_REGION_INFO_FLAG_READ \|
		1183	VFIO_REGION_INFO_FLAG_WRITE;
		1184	info->size = gvt_aperture_sz(vgpu->gvt);
		1185
		1186	sparse = kzalloc(struct_size(sparse, areas, nr_areas),
		1187	GFP_KERNEL);
		1188	if (!sparse)
		1189	return -ENOMEM;
		1190
		1191	sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
		1192	sparse->header.version = 1;
		1193	sparse->nr_areas = nr_areas;
		1194	cap_type_id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
		1195	sparse->areas[0].offset =
		1196	PAGE_ALIGN(vgpu_aperture_offset(vgpu));
		1197	sparse->areas[0].size = vgpu_aperture_sz(vgpu);
		1198	break;
		1199
		1200	case VFIO_PCI_BAR3_REGION_INDEX ... VFIO_PCI_BAR5_REGION_INDEX:
		1201	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
		1202	info->size = 0;
		1203	info->flags = 0;
		1204
		1205	gvt_dbg_core("get region info bar:%d\n", info->index);
		1206	break;
		1207
		1208	case VFIO_PCI_ROM_REGION_INDEX:
		1209	case VFIO_PCI_VGA_REGION_INDEX:
		1210	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
		1211	info->size = 0;
		1212	info->flags = 0;
		1213
		1214	gvt_dbg_core("get region info index:%d\n", info->index);
		1215	break;
		1216	default: {
		1217	struct vfio_region_info_cap_type cap_type = {
		1218	.header.id = VFIO_REGION_INFO_CAP_TYPE,
		1219	.header.version = 1
		1220	};
		1221
		1222	if (info->index >= VFIO_PCI_NUM_REGIONS + vgpu->num_regions)
		1223	return -EINVAL;
		1224	info->index = array_index_nospec(
		1225	info->index, VFIO_PCI_NUM_REGIONS + vgpu->num_regions);
		1226
		1227	i = info->index - VFIO_PCI_NUM_REGIONS;
		1228
		1229	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
		1230	info->size = vgpu->region[i].size;
		1231	info->flags = vgpu->region[i].flags;
		1232
		1233	cap_type.type = vgpu->region[i].type;
		1234	cap_type.subtype = vgpu->region[i].subtype;
		1235
		1236	ret = vfio_info_add_capability(caps, &cap_type.header,
		1237	sizeof(cap_type));
		1238	if (ret)
		1239	return ret;
		1240	}
		1241	}
		1242
		1243	if ((info->flags & VFIO_REGION_INFO_FLAG_CAPS) && sparse) {
		1244	ret = -EINVAL;
		1245	if (cap_type_id == VFIO_REGION_INFO_CAP_SPARSE_MMAP) {
		1246	ret = vfio_info_add_capability(
		1247	caps, &sparse->header,
		1248	struct_size(sparse, areas, sparse->nr_areas));
		1249	}
		1250	if (ret) {
		1251	kfree(sparse);
		1252	return ret;
		1253	}
		1254	}
		1255
		1256	kfree(sparse);
		1257	return 0;
		1258	}
		1259
1144	static long intel_vgpu_ioctl(struct vfio_device *vfio_dev, unsigned int cmd,	1260	static long intel_vgpu_ioctl(struct vfio_device *vfio_dev, unsigned int cmd,
1145	unsigned long arg)	1261	unsigned long arg)
1146	{	1262	{
@@ -1169,152 +1285,6 @@ static long intel_vgpu_ioctl(struct vfio_device *vfio_dev, unsigned int cmd, · 22 unmodified lines
1169	return copy_to_user((void __user *)arg, &info, minsz) ?	1285	return copy_to_user((void __user *)arg, &info, minsz) ?
1170	-EFAULT : 0;	1286	-EFAULT : 0;
1171		1287
1172	} else if (cmd == VFIO_DEVICE_GET_REGION_INFO) {
1173	struct vfio_region_info info;
1174	struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
1175	unsigned int i;
1176	int ret;
1177	struct vfio_region_info_cap_sparse_mmap *sparse = NULL;
1178	int nr_areas = 1;
1179	int cap_type_id;
1180
1181	minsz = offsetofend(struct vfio_region_info, offset);
1182
1183	if (copy_from_user(&info, (void __user *)arg, minsz))
1184	return -EFAULT;
1185
1186	if (info.argsz < minsz)
1187	return -EINVAL;
1188
1189	switch (info.index) {
1190	case VFIO_PCI_CONFIG_REGION_INDEX:
1191	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1192	info.size = vgpu->gvt->device_info.cfg_space_size;
1193	info.flags = VFIO_REGION_INFO_FLAG_READ \|
1194	VFIO_REGION_INFO_FLAG_WRITE;
1195	break;
1196	case VFIO_PCI_BAR0_REGION_INDEX:
1197	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1198	info.size = vgpu->cfg_space.bar[info.index].size;
1199	if (!info.size) {
1200	info.flags = 0;
1201	break;
1202	}
1203
1204	info.flags = VFIO_REGION_INFO_FLAG_READ \|
1205	VFIO_REGION_INFO_FLAG_WRITE;
1206	break;
1207	case VFIO_PCI_BAR1_REGION_INDEX:
1208	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1209	info.size = 0;
1210	info.flags = 0;
1211	break;
1212	case VFIO_PCI_BAR2_REGION_INDEX:
1213	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1214	info.flags = VFIO_REGION_INFO_FLAG_CAPS \|
1215	VFIO_REGION_INFO_FLAG_MMAP \|
1216	VFIO_REGION_INFO_FLAG_READ \|
1217	VFIO_REGION_INFO_FLAG_WRITE;
1218	info.size = gvt_aperture_sz(vgpu->gvt);
1219
1220	sparse = kzalloc(struct_size(sparse, areas, nr_areas),
1221	GFP_KERNEL);
1222	if (!sparse)
1223	return -ENOMEM;
1224
1225	sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
1226	sparse->header.version = 1;
1227	sparse->nr_areas = nr_areas;
1228	cap_type_id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
1229	sparse->areas[0].offset =
1230	PAGE_ALIGN(vgpu_aperture_offset(vgpu));
1231	sparse->areas[0].size = vgpu_aperture_sz(vgpu);
1232	break;
1233
1234	case VFIO_PCI_BAR3_REGION_INDEX ... VFIO_PCI_BAR5_REGION_INDEX:
1235	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1236	info.size = 0;
1237	info.flags = 0;
1238
1239	gvt_dbg_core("get region info bar:%d\n", info.index);
1240	break;
1241
1242	case VFIO_PCI_ROM_REGION_INDEX:
1243	case VFIO_PCI_VGA_REGION_INDEX:
1244	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
1245	info.size = 0;
1246	info.flags = 0;
1247
1248	gvt_dbg_core("get region info index:%d\n", info.index);
1249	break;
1250	default:
1251	{
1252	struct vfio_region_info_cap_type cap_type = {
1253	.header.id = VFIO_REGION_INFO_CAP_TYPE,
1254	.header.version = 1 };
1255
1256	if (info.index >= VFIO_PCI_NUM_REGIONS +
1257	vgpu->num_regions)
1258	return -EINVAL;
1259	info.index =
1260	array_index_nospec(info.index,
1261	VFIO_PCI_NUM_REGIONS +
1262	vgpu->num_regions);
1263
1264	i = info.index - VFIO_PCI_NUM_REGIONS;
1265
1266	info.offset =
1267	VFIO_PCI_INDEX_TO_OFFSET(info.index);
1268	info.size = vgpu->region[i].size;
1269	info.flags = vgpu->region[i].flags;
1270
1271	cap_type.type = vgpu->region[i].type;
1272	cap_type.subtype = vgpu->region[i].subtype;
1273
1274	ret = vfio_info_add_capability(&caps,
1275	&cap_type.header,
1276	sizeof(cap_type));
1277	if (ret)
1278	return ret;
1279	}
1280	}
1281
1282	if ((info.flags & VFIO_REGION_INFO_FLAG_CAPS) && sparse) {
1283	ret = -EINVAL;
1284	if (cap_type_id == VFIO_REGION_INFO_CAP_SPARSE_MMAP)
1285	ret = vfio_info_add_capability(&caps,
1286	&sparse->header,
1287	struct_size(sparse, areas,
1288	sparse->nr_areas));
1289	if (ret) {
1290	kfree(sparse);
1291	return ret;
1292	}
1293	}
1294
1295	if (caps.size) {
1296	info.flags \|= VFIO_REGION_INFO_FLAG_CAPS;
1297	if (info.argsz < sizeof(info) + caps.size) {
1298	info.argsz = sizeof(info) + caps.size;
1299	info.cap_offset = 0;
1300	} else {
1301	vfio_info_cap_shift(&caps, sizeof(info));
1302	if (copy_to_user((void __user *)arg +
1303	sizeof(info), caps.buf,
1304	caps.size)) {
1305	kfree(caps.buf);
1306	kfree(sparse);
1307	return -EFAULT;
1308	}
1309	info.cap_offset = sizeof(info);
1310	}
1311
1312	kfree(caps.buf);
1313	}
1314
1315	kfree(sparse);
1316	return copy_to_user((void __user *)arg, &info, minsz) ?
1317	-EFAULT : 0;
1318	} else if (cmd == VFIO_DEVICE_GET_IRQ_INFO) {	1288	} else if (cmd == VFIO_DEVICE_GET_IRQ_INFO) {
1319	struct vfio_irq_info info;	1289	struct vfio_irq_info info;
1320		1290
@@ -1477,6 +1447,7 @@ static const struct vfio_device_ops intel_vgpu_dev_ops = { · 156 unmodified lines
1477	.write = intel_vgpu_write,	1447	.write = intel_vgpu_write,
1478	.mmap = intel_vgpu_mmap,	1448	.mmap = intel_vgpu_mmap,
1479	.ioctl = intel_vgpu_ioctl,	1449	.ioctl = intel_vgpu_ioctl,
		1450	.get_region_info_caps = intel_vgpu_ioctl_get_region_info,
1480	.dma_unmap = intel_vgpu_dma_unmap,	1451	.dma_unmap = intel_vgpu_dma_unmap,
1481	.bind_iommufd = vfio_iommufd_emulated_bind,	1452	.bind_iommufd = vfio_iommufd_emulated_bind,
1482	.unbind_iommufd = vfio_iommufd_emulated_unbind,	1453	.unbind_iommufd = vfio_iommufd_emulated_unbind,

drivers/iommu/dma-iommu.c+2−2

@@ -1439,8 +1439,8 @@ int iommu_dma_map_sg(struct device dev, struct scatterlist sg, int nents, · 1438 unmodified lines
1439	* as a bus address, __finalise_sg() will copy the dma	1439	* as a bus address, __finalise_sg() will copy the dma
1440	* address into the output segment.	1440	* address into the output segment.
1441	*/	1441	*/
1442	s->dma_address = pci_p2pdma_bus_addr_map(&p2pdma_state,	1442	s->dma_address = pci_p2pdma_bus_addr_map(
1443	sg_phys(s));	1443	p2pdma_state.mem, sg_phys(s));
1444	sg_dma_len(s) = sg->length;	1444	sg_dma_len(s) = sg->length;
1445	sg_dma_mark_bus_address(s);	1445	sg_dma_mark_bus_address(s);
1446	continue;	1446	continue;

drivers/pci/p2pdma.c+144−42

@@ -25,12 +25,12 @@ struct pci_p2pdma { · 24 unmodified lines
25	struct gen_pool *pool;	25	struct gen_pool *pool;
26	bool p2pmem_published;	26	bool p2pmem_published;
27	struct xarray map_types;	27	struct xarray map_types;
		28	struct p2pdma_provider mem[PCI_STD_NUM_BARS];
28	};	29	};
29		30
30	struct pci_p2pdma_pagemap {	31	struct pci_p2pdma_pagemap {
31	struct pci_dev *provider;
32	u64 bus_offset;
33	struct dev_pagemap pgmap;	32	struct dev_pagemap pgmap;
		33	struct p2pdma_provider *mem;
34	};	34	};
35		35
36	static struct pci_p2pdma_pagemap to_p2p_pgmap(struct dev_pagemap pgmap)	36	static struct pci_p2pdma_pagemap to_p2p_pgmap(struct dev_pagemap pgmap)
@@ -204,8 +204,8 @@ static void p2pdma_page_free(struct page *page) · 167 unmodified lines
204	{	204	{
205	struct pci_p2pdma_pagemap *pgmap = to_p2p_pgmap(page_pgmap(page));	205	struct pci_p2pdma_pagemap *pgmap = to_p2p_pgmap(page_pgmap(page));
206	/* safe to dereference while a reference is held to the percpu ref */	206	/* safe to dereference while a reference is held to the percpu ref */
207	struct pci_p2pdma *p2pdma =	207	struct pci_p2pdma *p2pdma = rcu_dereference_protected(
208	rcu_dereference_protected(pgmap->provider->p2pdma, 1);	208	to_pci_dev(pgmap->mem->owner)->p2pdma, 1);
209	struct percpu_ref *ref;	209	struct percpu_ref *ref;
210		210
211	gen_pool_free_owner(p2pdma->pool, (uintptr_t)page_to_virt(page),	211	gen_pool_free_owner(p2pdma->pool, (uintptr_t)page_to_virt(page),
@@ -228,56 +228,136 @@ static void pci_p2pdma_release(void *data) · 16 unmodified lines
228		228
229	/* Flush and disable pci_alloc_p2p_mem() */	229	/* Flush and disable pci_alloc_p2p_mem() */
230	pdev->p2pdma = NULL;	230	pdev->p2pdma = NULL;
231	synchronize_rcu();	231	if (p2pdma->pool)
		232	synchronize_rcu();
		233	xa_destroy(&p2pdma->map_types);
		234
		235	if (!p2pdma->pool)
		236	return;
232		237
233	gen_pool_destroy(p2pdma->pool);	238	gen_pool_destroy(p2pdma->pool);
234	sysfs_remove_group(&pdev->dev.kobj, &p2pmem_group);	239	sysfs_remove_group(&pdev->dev.kobj, &p2pmem_group);
235	xa_destroy(&p2pdma->map_types);
236	}	240	}
237		241
238	static int pci_p2pdma_setup(struct pci_dev *pdev)	242	/**
		243	* pcim_p2pdma_init - Initialise peer-to-peer DMA providers
		244	* @pdev: The PCI device to enable P2PDMA for
		245	*
		246	* This function initializes the peer-to-peer DMA infrastructure
		247	* for a PCI device. It allocates and sets up the necessary data
		248	* structures to support P2PDMA operations, including mapping type
		249	* tracking.
		250	*/
		251	int pcim_p2pdma_init(struct pci_dev *pdev)
239	{	252	{
240	int error = -ENOMEM;
241	struct pci_p2pdma *p2p;	253	struct pci_p2pdma *p2p;
		254	int i, ret;
		255
		256	p2p = rcu_dereference_protected(pdev->p2pdma, 1);
		257	if (p2p)
		258	return 0;
242		259
243	p2p = devm_kzalloc(&pdev->dev, sizeof(*p2p), GFP_KERNEL);	260	p2p = devm_kzalloc(&pdev->dev, sizeof(*p2p), GFP_KERNEL);
244	if (!p2p)	261	if (!p2p)
245	return -ENOMEM;	262	return -ENOMEM;
246		263
247	xa_init(&p2p->map_types);	264	xa_init(&p2p->map_types);
		265	/*
		266	* Iterate over all standard PCI BARs and record only those that
		267	* correspond to MMIO regions. Skip non-memory resources (e.g. I/O
		268	* port BARs) since they cannot be used for peer-to-peer (P2P)
		269	* transactions.
		270	*/
		271	for (i = 0; i < PCI_STD_NUM_BARS; i++) {
		272	if (!(pci_resource_flags(pdev, i) & IORESOURCE_MEM))
		273	continue;
248		274
249	p2p->pool = gen_pool_create(PAGE_SHIFT, dev_to_node(&pdev->dev));	275	p2p->mem[i].owner = &pdev->dev;
250	if (!p2p->pool)	276	p2p->mem[i].bus_offset =
251	goto out;	277	pci_bus_address(pdev, i) - pci_resource_start(pdev, i);
		278	}
252		279
253	error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_release, pdev);	280	ret = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_release, pdev);
254	if (error)	281	if (ret)
255	goto out_pool_destroy;	282	goto out_p2p;
256		283
257	error = sysfs_create_group(&pdev->dev.kobj, &p2pmem_group);	284	rcu_assign_pointer(pdev->p2pdma, p2p);
258	if (error)	285	return 0;
		286
		287	out_p2p:
		288	devm_kfree(&pdev->dev, p2p);
		289	return ret;
		290	}
		291	EXPORT_SYMBOL_GPL(pcim_p2pdma_init);
		292
		293	/**
		294	* pcim_p2pdma_provider - Get peer-to-peer DMA provider
		295	* @pdev: The PCI device to enable P2PDMA for
		296	* @bar: BAR index to get provider
		297	*
		298	* This function gets peer-to-peer DMA provider for a PCI device. The lifetime
		299	* of the provider (and of course the MMIO) is bound to the lifetime of the
		300	* driver. A driver calling this function must ensure that all references to the
		301	* provider, and any DMA mappings created for any MMIO, are all cleaned up
		302	* before the driver remove() completes.
		303	*
		304	* Since P2P is almost always shared with a second driver this means some system
		305	* to notify, invalidate and revoke the MMIO's DMA must be in place to use this
		306	* function. For example a revoke can be built using DMABUF.
		307	*/
		308	struct p2pdma_provider pcim_p2pdma_provider(struct pci_dev pdev, int bar)
		309	{
		310	struct pci_p2pdma *p2p;
		311
		312	if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM))
		313	return NULL;
		314
		315	p2p = rcu_dereference_protected(pdev->p2pdma, 1);
		316	if (WARN_ON(!p2p))
		317	/* Someone forgot to call to pcim_p2pdma_init() before */
		318	return NULL;
		319
		320	return &p2p->mem[bar];
		321	}
		322	EXPORT_SYMBOL_GPL(pcim_p2pdma_provider);
		323
		324	static int pci_p2pdma_setup_pool(struct pci_dev *pdev)
		325	{
		326	struct pci_p2pdma *p2pdma;
		327	int ret;
		328
		329	p2pdma = rcu_dereference_protected(pdev->p2pdma, 1);
		330	if (p2pdma->pool)
		331	/* We already setup pools, do nothing, */
		332	return 0;
		333
		334	p2pdma->pool = gen_pool_create(PAGE_SHIFT, dev_to_node(&pdev->dev));
		335	if (!p2pdma->pool)
		336	return -ENOMEM;
		337
		338	ret = sysfs_create_group(&pdev->dev.kobj, &p2pmem_group);
		339	if (ret)
259	goto out_pool_destroy;	340	goto out_pool_destroy;
260		341
261	rcu_assign_pointer(pdev->p2pdma, p2p);
262	return 0;	342	return 0;
263		343
264	out_pool_destroy:	344	out_pool_destroy:
265	gen_pool_destroy(p2p->pool);	345	gen_pool_destroy(p2pdma->pool);
266	out:	346	p2pdma->pool = NULL;
267	devm_kfree(&pdev->dev, p2p);	347	return ret;
268	return error;
269	}	348	}
270		349
271	static void pci_p2pdma_unmap_mappings(void *data)	350	static void pci_p2pdma_unmap_mappings(void *data)
272	{	351	{
273	struct pci_dev *pdev = data;	352	struct pci_p2pdma_pagemap *p2p_pgmap = data;
274		353
275	/*	354	/*
276	* Removing the alloc attribute from sysfs will call	355	* Removing the alloc attribute from sysfs will call
277	* unmap_mapping_range() on the inode, teardown any existing userspace	356	* unmap_mapping_range() on the inode, teardown any existing userspace
278	* mappings and prevent new ones from being created.	357	* mappings and prevent new ones from being created.
279	*/	358	*/
280	sysfs_remove_file_from_group(&pdev->dev.kobj, &p2pmem_alloc_attr.attr,	359	sysfs_remove_file_from_group(&p2p_pgmap->mem->owner->kobj,
		360	&p2pmem_alloc_attr.attr,
281	p2pmem_group.name);	361	p2pmem_group.name);
282	}	362	}
283		363
@@ -295,6 +375,7 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, · 11 unmodified lines
295	u64 offset)	375	u64 offset)
296	{	376	{
297	struct pci_p2pdma_pagemap *p2p_pgmap;	377	struct pci_p2pdma_pagemap *p2p_pgmap;
		378	struct p2pdma_provider *mem;
298	struct dev_pagemap *pgmap;	379	struct dev_pagemap *pgmap;
299	struct pci_p2pdma *p2pdma;	380	struct pci_p2pdma *p2pdma;
300	void *addr;	381	void *addr;
@@ -312,11 +393,21 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, · 11 unmodified lines
312	if (size + offset > pci_resource_len(pdev, bar))	393	if (size + offset > pci_resource_len(pdev, bar))
313	return -EINVAL;	394	return -EINVAL;
314		395
315	if (!pdev->p2pdma) {	396	error = pcim_p2pdma_init(pdev);
316	error = pci_p2pdma_setup(pdev);	397	if (error)
317	if (error)	398	return error;
318	return error;	399
319	}	400	error = pci_p2pdma_setup_pool(pdev);
		401	if (error)
		402	return error;
		403
		404	mem = pcim_p2pdma_provider(pdev, bar);
		405	/*
		406	* We checked validity of BAR prior to call
		407	* to pcim_p2pdma_provider. It should never return NULL.
		408	*/
		409	if (WARN_ON(!mem))
		410	return -EINVAL;
320		411
321	p2p_pgmap = devm_kzalloc(&pdev->dev, sizeof(*p2p_pgmap), GFP_KERNEL);	412	p2p_pgmap = devm_kzalloc(&pdev->dev, sizeof(*p2p_pgmap), GFP_KERNEL);
322	if (!p2p_pgmap)	413	if (!p2p_pgmap)
@@ -328,10 +419,7 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, · 5 unmodified lines
328	pgmap->nr_range = 1;	419	pgmap->nr_range = 1;
329	pgmap->type = MEMORY_DEVICE_PCI_P2PDMA;	420	pgmap->type = MEMORY_DEVICE_PCI_P2PDMA;
330	pgmap->ops = &p2pdma_pgmap_ops;	421	pgmap->ops = &p2pdma_pgmap_ops;
331		422	p2p_pgmap->mem = mem;
332	p2p_pgmap->provider = pdev;
333	p2p_pgmap->bus_offset = pci_bus_address(pdev, bar) -
334	pci_resource_start(pdev, bar);
335		423
336	addr = devm_memremap_pages(&pdev->dev, pgmap);	424	addr = devm_memremap_pages(&pdev->dev, pgmap);
337	if (IS_ERR(addr)) {	425	if (IS_ERR(addr)) {
@@ -340,7 +428,7 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, · 2 unmodified lines
340	}	428	}
341		429
342	error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_unmap_mappings,	430	error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_unmap_mappings,
343	pdev);	431	p2p_pgmap);
344	if (error)	432	if (error)
345	goto pages_free;	433	goto pages_free;
346		434
@@ -972,16 +1060,26 @@ void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) · 625 unmodified lines
972	}	1060	}
973	EXPORT_SYMBOL_GPL(pci_p2pmem_publish);	1061	EXPORT_SYMBOL_GPL(pci_p2pmem_publish);
974		1062
975	static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap,	1063	/**
976	struct device *dev)	1064	* pci_p2pdma_map_type - Determine the mapping type for P2PDMA transfers
		1065	* @provider: P2PDMA provider structure
		1066	* @dev: Target device for the transfer
		1067	*
		1068	* Determines how peer-to-peer DMA transfers should be mapped between
		1069	* the provider and the target device. The mapping type indicates whether
		1070	* the transfer can be done directly through PCI switches or must go
		1071	* through the host bridge.
		1072	*/
		1073	enum pci_p2pdma_map_type pci_p2pdma_map_type(struct p2pdma_provider *provider,
		1074	struct device *dev)
977	{	1075	{
978	enum pci_p2pdma_map_type type = PCI_P2PDMA_MAP_NOT_SUPPORTED;	1076	enum pci_p2pdma_map_type type = PCI_P2PDMA_MAP_NOT_SUPPORTED;
979	struct pci_dev *provider = to_p2p_pgmap(pgmap)->provider;	1077	struct pci_dev *pdev = to_pci_dev(provider->owner);
980	struct pci_dev *client;	1078	struct pci_dev *client;
981	struct pci_p2pdma *p2pdma;	1079	struct pci_p2pdma *p2pdma;
982	int dist;	1080	int dist;
983		1081
984	if (!provider->p2pdma)	1082	if (!pdev->p2pdma)
985	return PCI_P2PDMA_MAP_NOT_SUPPORTED;	1083	return PCI_P2PDMA_MAP_NOT_SUPPORTED;
986		1084
987	if (!dev_is_pci(dev))	1085	if (!dev_is_pci(dev))
@@ -990,7 +1088,7 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, · 2 unmodified lines
990	client = to_pci_dev(dev);	1088	client = to_pci_dev(dev);
991		1089
992	rcu_read_lock();	1090	rcu_read_lock();
993	p2pdma = rcu_dereference(provider->p2pdma);	1091	p2pdma = rcu_dereference(pdev->p2pdma);
994		1092
995	if (p2pdma)	1093	if (p2pdma)
996	type = xa_to_value(xa_load(&p2pdma->map_types,	1094	type = xa_to_value(xa_load(&p2pdma->map_types,
@@ -998,7 +1096,7 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, · 1 unmodified line
998	rcu_read_unlock();	1096	rcu_read_unlock();
999		1097
1000	if (type == PCI_P2PDMA_MAP_UNKNOWN)	1098	if (type == PCI_P2PDMA_MAP_UNKNOWN)
1001	return calc_map_type_and_dist(provider, client, &dist, true);	1099	return calc_map_type_and_dist(pdev, client, &dist, true);
1002		1100
1003	return type;	1101	return type;
1004	}	1102	}
@@ -1006,9 +1104,13 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, · 1 unmodified line
1006	void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state,	1104	void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state,
1007	struct device dev, struct page page)	1105	struct device dev, struct page page)
1008	{	1106	{
1009	state->pgmap = page_pgmap(page);	1107	struct pci_p2pdma_pagemap *p2p_pgmap = to_p2p_pgmap(page_pgmap(page));
1010	state->map = pci_p2pdma_map_type(state->pgmap, dev);	1108
1011	state->bus_off = to_p2p_pgmap(state->pgmap)->bus_offset;	1109	if (state->mem == p2p_pgmap->mem)
		1110	return;
		1111
		1112	state->mem = p2p_pgmap->mem;
		1113	state->map = pci_p2pdma_map_type(p2p_pgmap->mem, dev);
1012	}	1114	}
1013		1115
1014	/**	1116	/**

drivers/s390/cio/vfio_ccw_ops.c+7−40

 	return 0;
 }
-static int vfio_ccw_mdev_get_region_info(struct vfio_ccw_private *private,
+static int vfio_ccw_mdev_ioctl_get_region_info(struct vfio_device *vdev,
-					 struct vfio_region_info *info,
+					       struct vfio_region_info *info,
-					 unsigned long arg)
+					       struct vfio_info_cap *caps)
 {
+	struct vfio_ccw_private *private =
+		container_of(vdev, struct vfio_ccw_private, vdev);
 	int i;
 	switch (info->index) {
 		return 0;
 	default: /* all other regions are handled via capability chain */
 	{
-		struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
 		struct vfio_region_info_cap_type cap_type = {
 			.header.id = VFIO_REGION_INFO_CAP_TYPE,
 			.header.version = 1 };
 		cap_type.type = private->region[i].type;
 		cap_type.subtype = private->region[i].subtype;
-		ret = vfio_info_add_capability(&caps, &cap_type.header,
+		ret = vfio_info_add_capability(caps, &cap_type.header,
 					       sizeof(cap_type));
 		if (ret)
 			return ret;
-		info->flags |= VFIO_REGION_INFO_FLAG_CAPS;
-		if (info->argsz < sizeof(*info) + caps.size) {
-			info->argsz = sizeof(*info) + caps.size;
-			info->cap_offset = 0;
-		} else {
-			vfio_info_cap_shift(&caps, sizeof(*info));
-			if (copy_to_user((void __user *)arg + sizeof(*info),
-					 caps.buf, caps.size)) {
-				kfree(caps.buf);
-				return -EFAULT;
-			}
-			info->cap_offset = sizeof(*info);
-		}
-		kfree(caps.buf);
 	}
 	}
 	return 0;
 		return copy_to_user((void __user *)arg, &info, minsz) ? -EFAULT : 0;
 	}
-	case VFIO_DEVICE_GET_REGION_INFO:
-	{
-		struct vfio_region_info info;
-		minsz = offsetofend(struct vfio_region_info, offset);
-		if (copy_from_user(&info, (void __user *)arg, minsz))
-			return -EFAULT;
-		if (info.argsz < minsz)
-			return -EINVAL;
-		ret = vfio_ccw_mdev_get_region_info(private, &info, arg);
-		if (ret)
-			return ret;
-		return copy_to_user((void __user *)arg, &info, minsz) ? -EFAULT : 0;
-	}
 	case VFIO_DEVICE_GET_IRQ_INFO:
 	{
 		struct vfio_irq_info info;
 	.read = vfio_ccw_mdev_read,
 	.write = vfio_ccw_mdev_write,
 	.ioctl = vfio_ccw_mdev_ioctl,
+	.get_region_info_caps = vfio_ccw_mdev_ioctl_get_region_info,
 	.request = vfio_ccw_mdev_request,
 	.dma_unmap = vfio_ccw_dma_unmap,
 	.bind_iommufd = vfio_iommufd_emulated_bind,

drivers/vfio/cdx/main.c+11−18

@@ -129,28 +129,22 @@ static int vfio_cdx_ioctl_get_info(struct vfio_cdx_device *vdev, · 128 unmodified lines
129	return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;	129	return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;
130	}	130	}
131		131
132	static int vfio_cdx_ioctl_get_region_info(struct vfio_cdx_device *vdev,	132	static int vfio_cdx_ioctl_get_region_info(struct vfio_device *core_vdev,
133	struct vfio_region_info __user *arg)	133	struct vfio_region_info *info,
		134	struct vfio_info_cap *caps)
134	{	135	{
135	unsigned long minsz = offsetofend(struct vfio_region_info, offset);	136	struct vfio_cdx_device *vdev =
		137	container_of(core_vdev, struct vfio_cdx_device, vdev);
136	struct cdx_device *cdx_dev = to_cdx_device(vdev->vdev.dev);	138	struct cdx_device *cdx_dev = to_cdx_device(vdev->vdev.dev);
137	struct vfio_region_info info;
138
139	if (copy_from_user(&info, arg, minsz))
140	return -EFAULT;
141		139
142	if (info.argsz < minsz)	140	if (info->index >= cdx_dev->res_count)
143	return -EINVAL;
144
145	if (info.index >= cdx_dev->res_count)
146	return -EINVAL;	141	return -EINVAL;
147		142
148	/* map offset to the physical address */	143	/* map offset to the physical address */
149	info.offset = vfio_cdx_index_to_offset(info.index);	144	info->offset = vfio_cdx_index_to_offset(info->index);
150	info.size = vdev->regions[info.index].size;	145	info->size = vdev->regions[info->index].size;
151	info.flags = vdev->regions[info.index].flags;	146	info->flags = vdev->regions[info->index].flags;
152		147	return 0;
153	return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;
154	}	148	}
155		149
156	static int vfio_cdx_ioctl_get_irq_info(struct vfio_cdx_device *vdev,	150	static int vfio_cdx_ioctl_get_irq_info(struct vfio_cdx_device *vdev,
@@ -219,8 +213,6 @@ static long vfio_cdx_ioctl(struct vfio_device *core_vdev, · 62 unmodified lines
219	switch (cmd) {	213	switch (cmd) {
220	case VFIO_DEVICE_GET_INFO:	214	case VFIO_DEVICE_GET_INFO:
221	return vfio_cdx_ioctl_get_info(vdev, uarg);	215	return vfio_cdx_ioctl_get_info(vdev, uarg);
222	case VFIO_DEVICE_GET_REGION_INFO:
223	return vfio_cdx_ioctl_get_region_info(vdev, uarg);
224	case VFIO_DEVICE_GET_IRQ_INFO:	216	case VFIO_DEVICE_GET_IRQ_INFO:
225	return vfio_cdx_ioctl_get_irq_info(vdev, uarg);	217	return vfio_cdx_ioctl_get_irq_info(vdev, uarg);
226	case VFIO_DEVICE_SET_IRQS:	218	case VFIO_DEVICE_SET_IRQS:
@@ -284,6 +276,7 @@ static const struct vfio_device_ops vfio_cdx_ops = { · 57 unmodified lines
284	.open_device = vfio_cdx_open_device,	276	.open_device = vfio_cdx_open_device,
285	.close_device = vfio_cdx_close_device,	277	.close_device = vfio_cdx_close_device,
286	.ioctl = vfio_cdx_ioctl,	278	.ioctl = vfio_cdx_ioctl,
		279	.get_region_info_caps = vfio_cdx_ioctl_get_region_info,
287	.device_feature = vfio_cdx_ioctl_feature,	280	.device_feature = vfio_cdx_ioctl_feature,
288	.mmap = vfio_cdx_mmap,	281	.mmap = vfio_cdx_mmap,
289	.bind_iommufd = vfio_iommufd_physical_bind,	282	.bind_iommufd = vfio_iommufd_physical_bind,

drivers/vfio/device_cdev.c+1−1

@@ -99,7 +99,7 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df, · 98 unmodified lines
99	return ret;	99	return ret;
100	if (user_size < minsz)	100	if (user_size < minsz)
101	return -EINVAL;	101	return -EINVAL;
102	ret = copy_struct_from_user(&bind, minsz, arg, user_size);	102	ret = copy_struct_from_user(&bind, sizeof(bind), arg, user_size);
103	if (ret)	103	if (ret)
104	return ret;	104	return ret;
105		105

drivers/vfio/fsl-mc/vfio_fsl_mc.c+19−24

 	fsl_mc_cleanup_irq_pool(mc_cont);
 }
+static int vfio_fsl_mc_ioctl_get_region_info(struct vfio_device *core_vdev,
+					     struct vfio_region_info *info,
+					     struct vfio_info_cap *caps)
+{
+	struct vfio_fsl_mc_device *vdev =
+		container_of(core_vdev, struct vfio_fsl_mc_device, vdev);
+	struct fsl_mc_device *mc_dev = vdev->mc_dev;
+	if (info->index >= mc_dev->obj_desc.region_count)
+		return -EINVAL;
+	/* map offset to the physical address  */
+	info->offset = VFIO_FSL_MC_INDEX_TO_OFFSET(info->index);
+	info->size = vdev->regions[info->index].size;
+	info->flags = vdev->regions[info->index].flags;
+	return 0;
+}
 static long vfio_fsl_mc_ioctl(struct vfio_device *core_vdev,
 			      unsigned int cmd, unsigned long arg)
 {
 		return copy_to_user((void __user *)arg, &info, minsz) ?
 			-EFAULT : 0;
 	}
-	case VFIO_DEVICE_GET_REGION_INFO:
-	{
-		struct vfio_region_info info;
-		minsz = offsetofend(struct vfio_region_info, offset);
-		if (copy_from_user(&info, (void __user *)arg, minsz))
-			return -EFAULT;
-		if (info.argsz < minsz)
-			return -EINVAL;
-		if (info.index >= mc_dev->obj_desc.region_count)
-			return -EINVAL;
-		/* map offset to the physical address  */
-		info.offset = VFIO_FSL_MC_INDEX_TO_OFFSET(info.index);
-		info.size = vdev->regions[info.index].size;
-		info.flags = vdev->regions[info.index].flags;
-		if (copy_to_user((void __user *)arg, &info, minsz))
-			return -EFAULT;
-		return 0;
-	}
 	case VFIO_DEVICE_GET_IRQ_INFO:
 	{
 		struct vfio_irq_info info;
 	.open_device	= vfio_fsl_mc_open_device,
 	.close_device	= vfio_fsl_mc_close_device,
 	.ioctl		= vfio_fsl_mc_ioctl,
+	.get_region_info_caps = vfio_fsl_mc_ioctl_get_region_info,
 	.read		= vfio_fsl_mc_read,
 	.write		= vfio_fsl_mc_write,
 	.mmap		= vfio_fsl_mc_mmap,

drivers/vfio/pci/Kconfig+3−0

@@ -55,6 +55,9 @@ config VFIO_PCI_ZDEV_KVM · 54 unmodified lines
55		55
56	To enable s390x KVM vfio-pci extensions, say Y.	56	To enable s390x KVM vfio-pci extensions, say Y.
57		57
		58	config VFIO_PCI_DMABUF
		59	def_bool y if VFIO_PCI_CORE && PCI_P2PDMA && DMA_SHARED_BUFFER
		60
58	source "drivers/vfio/pci/mlx5/Kconfig"	61	source "drivers/vfio/pci/mlx5/Kconfig"
59		62
60	source "drivers/vfio/pci/hisilicon/Kconfig"	63	source "drivers/vfio/pci/hisilicon/Kconfig"

drivers/vfio/pci/Makefile+1−0

@@ -2,6 +2,7 @@ · 1 unmodified line
2		2
3	vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o	3	vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
4	vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o	4	vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
		5	vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o
5	obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o	6	obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
6		7
7	vfio-pci-y := vfio_pci.o	8	vfio-pci-y := vfio_pci.o

drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c+108−63

 	return 0;
 }
+static void qm_xqc_reg_offsets(struct hisi_qm *qm,
+			       u32 *eqc_addr, u32 *aeqc_addr)
+{
+	struct hisi_acc_vf_core_device *hisi_acc_vdev =
+		container_of(qm, struct hisi_acc_vf_core_device, vf_qm);
+	if (hisi_acc_vdev->drv_mode == HW_ACC_MIG_VF_CTRL) {
+		*eqc_addr = QM_EQC_VF_DW0;
+		*aeqc_addr = QM_AEQC_VF_DW0;
+	} else {
+		*eqc_addr = QM_EQC_PF_DW0;
+		*aeqc_addr = QM_AEQC_PF_DW0;
+	}
+}
 static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data)
 {
 	struct device *dev = &qm->pdev->dev;
+	u32 eqc_addr, aeqc_addr;
 	int ret;
 	ret = qm_read_regs(qm, QM_VF_AEQ_INT_MASK, &vf_data->aeq_int_mask, 1);
 		return ret;
 	}
+	qm_xqc_reg_offsets(qm, &eqc_addr, &aeqc_addr);
 	/* QM_EQC_DW has 7 regs */
-	ret = qm_read_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
+	ret = qm_read_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
 	if (ret) {
 		dev_err(dev, "failed to read QM_EQC_DW\n");
 		return ret;
 	}
 	/* QM_AEQC_DW has 7 regs */
-	ret = qm_read_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, 7);
+	ret = qm_read_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
 	if (ret) {
 		dev_err(dev, "failed to read QM_AEQC_DW\n");
 		return ret;
 static int qm_set_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data)
 {
 	struct device *dev = &qm->pdev->dev;
+	u32 eqc_addr, aeqc_addr;
 	int ret;
 	/* Check VF state */
 		return ret;
 	}
+	qm_xqc_reg_offsets(qm, &eqc_addr, &aeqc_addr);
 	/* QM_EQC_DW has 7 regs */
-	ret = qm_write_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7);
+	ret = qm_write_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7);
 	if (ret) {
 		dev_err(dev, "failed to write QM_EQC_DW\n");
 		return ret;
 	}
 	/* QM_AEQC_DW has 7 regs */
-	ret = qm_write_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, 7);
+	ret = qm_write_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7);
 	if (ret) {
 		dev_err(dev, "failed to write QM_AEQC_DW\n");
 		return ret;
 {
 	struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device;
 	struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm;
+	struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm;
 	struct pci_dev *vf_dev = vdev->pdev;
+	u32 val;
-	/*
+	val = readl(pf_qm->io_base + QM_MIG_REGION_SEL);
-	 * ACC VF dev BAR2 region consists of both functional register space
+	if (pf_qm->ver > QM_HW_V3 && (val & QM_MIG_REGION_EN))
-	 * and migration control register space. For migration to work, we
+		hisi_acc_vdev->drv_mode = HW_ACC_MIG_PF_CTRL;
-	 * need access to both. Hence, we map the entire BAR2 region here.
+	else
-	 * But unnecessarily exposing the migration BAR region to the Guest
+		hisi_acc_vdev->drv_mode = HW_ACC_MIG_VF_CTRL;
-	 * has the potential to prevent/corrupt the Guest migration. Hence,
-	 * we restrict access to the migration control space from
-	 * Guest(Please see mmap/ioctl/read/write override functions).
-	 *
-	 * Please note that it is OK to expose the entire VF BAR if migration
-	 * is not supported or required as this cannot affect the ACC PF
-	 * configurations.
-	 *
-	 * Also the HiSilicon ACC VF devices supported by this driver on
-	 * HiSilicon hardware platforms are integrated end point devices
-	 * and the platform lacks the capability to perform any PCIe P2P
-	 * between these devices.
-	 */
-	vf_qm->io_base =
+	if (hisi_acc_vdev->drv_mode == HW_ACC_MIG_PF_CTRL) {
-		ioremap(pci_resource_start(vf_dev, VFIO_PCI_BAR2_REGION_INDEX),
+		/*
-			pci_resource_len(vf_dev, VFIO_PCI_BAR2_REGION_INDEX));
+		 * On hardware platforms greater than QM_HW_V3, the migration function
-	if (!vf_qm->io_base)
+		 * register is placed in the BAR2 configuration region of the PF,
-		return -EIO;
+		 * and each VF device occupies 8KB of configuration space.
+		 */
+		vf_qm->io_base = pf_qm->io_base + QM_MIG_REGION_OFFSET +
+				 hisi_acc_vdev->vf_id * QM_MIG_REGION_SIZE;
+	} else {
+		/*
+		 * ACC VF dev BAR2 region consists of both functional register space
+		 * and migration control register space. For migration to work, we
+		 * need access to both. Hence, we map the entire BAR2 region here.
+		 * But unnecessarily exposing the migration BAR region to the Guest
+		 * has the potential to prevent/corrupt the Guest migration. Hence,
+		 * we restrict access to the migration control space from
+		 * Guest(Please see mmap/ioctl/read/write override functions).
+		 *
+		 * Please note that it is OK to expose the entire VF BAR if migration
+		 * is not supported or required as this cannot affect the ACC PF
+		 * configurations.
+		 *
+		 * Also the HiSilicon ACC VF devices supported by this driver on
+		 * HiSilicon hardware platforms are integrated end point devices
+		 * and the platform lacks the capability to perform any PCIe P2P
+		 * between these devices.
+		 */
+		vf_qm->io_base =
+			ioremap(pci_resource_start(vf_dev, VFIO_PCI_BAR2_REGION_INDEX),
+				pci_resource_len(vf_dev, VFIO_PCI_BAR2_REGION_INDEX));
+		if (!vf_qm->io_base)
+			return -EIO;
+	}
 	vf_qm->fun_type = QM_HW_VF;
+	vf_qm->ver = pf_qm->ver;
 	vf_qm->pdev = vf_dev;
 	mutex_init(&vf_qm->mailbox_lock);
 	return !IS_ERR(pf_qm) ? pf_qm : NULL;
 }
+static size_t hisi_acc_get_resource_len(struct vfio_pci_core_device *vdev,
+					unsigned int index)
+{
+	struct hisi_acc_vf_core_device *hisi_acc_vdev =
+			hisi_acc_drvdata(vdev->pdev);
+	/*
+	 * On the old HW_ACC_MIG_VF_CTRL mode device, the ACC VF device
+	 * BAR2 region encompasses both functional register space
+	 * and migration control register space.
+	 * only the functional region should be report to Guest.
+	 */
+	if (hisi_acc_vdev->drv_mode == HW_ACC_MIG_VF_CTRL)
+		return (pci_resource_len(vdev->pdev, index) >> 1);
+	/*
+	 * On the new HW device, the migration control register
+	 * has been moved to the PF device BAR2 region.
+	 * The VF device BAR2 is entirely functional register space.
+	 */
+	return pci_resource_len(vdev->pdev, index);
+}
 static int hisi_acc_pci_rw_access_check(struct vfio_device *core_vdev,
 					size_t count, loff_t *ppos,
 					size_t *new_count)
 	if (index == VFIO_PCI_BAR2_REGION_INDEX) {
 		loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
-		resource_size_t end = pci_resource_len(vdev->pdev, index) / 2;
+		resource_size_t end;
+		end = hisi_acc_get_resource_len(vdev, index);
 		/* Check if access is for migration control region */
 		if (pos >= end)
 			return -EINVAL;
 	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
 	if (index == VFIO_PCI_BAR2_REGION_INDEX) {
 		u64 req_len, pgoff, req_start;
-		resource_size_t end = pci_resource_len(vdev->pdev, index) / 2;
+		resource_size_t end;
+		end = hisi_acc_get_resource_len(vdev, index);
 		req_len = vma->vm_end - vma->vm_start;
 		pgoff = vma->vm_pgoff &
 			((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
 	return vfio_pci_core_read(core_vdev, buf, new_count, ppos);
 }
-static long hisi_acc_vfio_pci_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
+static int hisi_acc_vfio_ioctl_get_region(struct vfio_device *core_vdev,
-				    unsigned long arg)
+					  struct vfio_region_info *info,
+					  struct vfio_info_cap *caps)
 {
-	if (cmd == VFIO_DEVICE_GET_REGION_INFO) {
+	struct vfio_pci_core_device *vdev =
-		struct vfio_pci_core_device *vdev =
+		container_of(core_vdev, struct vfio_pci_core_device, vdev);
-			container_of(core_vdev, struct vfio_pci_core_device, vdev);
-		struct pci_dev *pdev = vdev->pdev;
-		struct vfio_region_info info;
-		unsigned long minsz;
-		minsz = offsetofend(struct vfio_region_info, offset);
-		if (copy_from_user(&info, (void __user *)arg, minsz))
-			return -EFAULT;
-		if (info.argsz < minsz)
-			return -EINVAL;
-		if (info.index == VFIO_PCI_BAR2_REGION_INDEX) {
+	if (info->index != VFIO_PCI_BAR2_REGION_INDEX)
-			info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
+		return vfio_pci_ioctl_get_region_info(core_vdev, info, caps);
-			/*
+	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
-			 * ACC VF dev BAR2 region consists of both functional
-			 * register space and migration control register space.
-			 * Report only the functional region to Guest.
-			 */
-			info.size = pci_resource_len(pdev, info.index) / 2;
-			info.flags = VFIO_REGION_INFO_FLAG_READ |
+	info->size = hisi_acc_get_resource_len(vdev, info->index);
-					VFIO_REGION_INFO_FLAG_WRITE |
-					VFIO_REGION_INFO_FLAG_MMAP;
-			return copy_to_user((void __user *)arg, &info, minsz) ?
+	info->flags = VFIO_REGION_INFO_FLAG_READ | VFIO_REGION_INFO_FLAG_WRITE |
-					    -EFAULT : 0;
+		     VFIO_REGION_INFO_FLAG_MMAP;
-		}
+	return 0;
-	}
-	return vfio_pci_core_ioctl(core_vdev, cmd, arg);
 }
 static int hisi_acc_vf_debug_check(struct seq_file *seq, struct vfio_device *vdev)
 	hisi_acc_vf_disable_fds(hisi_acc_vdev);
 	mutex_lock(&hisi_acc_vdev->open_mutex);
 	hisi_acc_vdev->dev_opened = false;
-	iounmap(vf_qm->io_base);
+	if (hisi_acc_vdev->drv_mode == HW_ACC_MIG_VF_CTRL)
+		iounmap(vf_qm->io_base);
 	mutex_unlock(&hisi_acc_vdev->open_mutex);
 	vfio_pci_core_close_device(core_vdev);
 }
 	.release = vfio_pci_core_release_dev,
 	.open_device = hisi_acc_vfio_pci_open_device,
 	.close_device = hisi_acc_vfio_pci_close_device,
-	.ioctl = hisi_acc_vfio_pci_ioctl,
+	.ioctl = vfio_pci_core_ioctl,
+	.get_region_info_caps = hisi_acc_vfio_ioctl_get_region,
 	.device_feature = vfio_pci_core_ioctl_feature,
 	.read = hisi_acc_vfio_pci_read,
 	.write = hisi_acc_vfio_pci_write,
 	.mmap = hisi_acc_vfio_pci_mmap,
 	.request = vfio_pci_core_request,
 	.match = vfio_pci_core_match,
+	.match_token_uuid = vfio_pci_core_match_token_uuid,
 	.bind_iommufd = vfio_iommufd_physical_bind,
 	.unbind_iommufd = vfio_iommufd_physical_unbind,
 	.attach_ioas = vfio_iommufd_physical_attach_ioas,
 	.open_device = hisi_acc_vfio_pci_open_device,
 	.close_device = vfio_pci_core_close_device,
 	.ioctl = vfio_pci_core_ioctl,
+	.get_region_info_caps = vfio_pci_ioctl_get_region_info,
 	.device_feature = vfio_pci_core_ioctl_feature,
 	.read = vfio_pci_core_read,
 	.write = vfio_pci_core_write,

drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h+21−2

 #define QM_QUE_ISO_CFG_V	0x0030
 #define QM_PAGE_SIZE		0x0034
-#define QM_EQC_DW0		0X8000
+#define QM_EQC_VF_DW0		0X8000
-#define QM_AEQC_DW0		0X8020
+#define QM_AEQC_VF_DW0		0X8020
+#define QM_EQC_PF_DW0		0x1c00
+#define QM_AEQC_PF_DW0		0x1c20
 #define ACC_DRV_MAJOR_VER 1
 #define ACC_DRV_MINOR_VER 0
 #define ACC_DEV_MAGIC_V1	0XCDCDCDCDFEEDAACC
 #define ACC_DEV_MAGIC_V2	0xAACCFEEDDECADEDE
+#define QM_MIG_REGION_OFFSET		0x180000
+#define QM_MIG_REGION_SIZE		0x2000
+/**
+ * On HW_ACC_MIG_VF_CTRL mode, the configuration domain supporting live
+ * migration functionality is located in the latter 32KB of the VF's BAR2.
+ * The Guest is only provided with the first 32KB of the VF's BAR2.
+ * On HW_ACC_MIG_PF_CTRL mode, the configuration domain supporting live
+ * migration functionality is located in the PF's BAR2, and the entire 64KB
+ * of the VF's BAR2 is allocated to the Guest.
+ */
+enum hw_drv_mode {
+	HW_ACC_MIG_VF_CTRL = 0,
+	HW_ACC_MIG_PF_CTRL,
+};
 struct acc_vf_data {
 #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state)
 	/* QM match information */
 	struct pci_dev *vf_dev;
 	struct hisi_qm *pf_qm;
 	struct hisi_qm vf_qm;
+	enum hw_drv_mode drv_mode;
 	/*
 	 * vf_qm_state represents the QM_VF_STATE register value.
 	 * It is set by Guest driver for the ACC VF dev indicating

drivers/vfio/pci/mlx5/main.c+1−0

@@ -1366,6 +1366,7 @@ static const struct vfio_device_ops mlx5vf_pci_ops = { · 1365 unmodified lines
1366	.open_device = mlx5vf_pci_open_device,	1366	.open_device = mlx5vf_pci_open_device,
1367	.close_device = mlx5vf_pci_close_device,	1367	.close_device = mlx5vf_pci_close_device,
1368	.ioctl = vfio_pci_core_ioctl,	1368	.ioctl = vfio_pci_core_ioctl,
		1369	.get_region_info_caps = vfio_pci_ioctl_get_region_info,
1369	.device_feature = vfio_pci_core_ioctl_feature,	1370	.device_feature = vfio_pci_core_ioctl_feature,
1370	.read = vfio_pci_core_read,	1371	.read = vfio_pci_core_read,
1371	.write = vfio_pci_core_write,	1372	.write = vfio_pci_core_write,

drivers/vfio/pci/nvgrace-gpu/main.c+255−87

@@ -7,6 +7,8 @@ · 6 unmodified lines
7	#include <linux/vfio_pci_core.h>	7	#include <linux/vfio_pci_core.h>
8	#include <linux/delay.h>	8	#include <linux/delay.h>
9	#include <linux/jiffies.h>	9	#include <linux/jiffies.h>
		10	#include <linux/pci-p2pdma.h>
		11	#include <linux/pm_runtime.h>
10		12
11	/*	13	/*
12	* The device memory usable to the workloads running in the VM is cached	14	* The device memory usable to the workloads running in the VM is cached
@@ -58,6 +60,8 @@ struct nvgrace_gpu_pci_core_device { · 45 unmodified lines
58	/* Lock to control device memory kernel mapping */	60	/* Lock to control device memory kernel mapping */
59	struct mutex remap_lock;	61	struct mutex remap_lock;
60	bool has_mig_hw_bug;	62	bool has_mig_hw_bug;
		63	/* GPU has just been reset */
		64	bool reset_done;
61	};	65	};
62		66
63	static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device *core_vdev)	67	static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device *core_vdev)
@@ -102,6 +106,19 @@ static int nvgrace_gpu_open_device(struct vfio_device *core_vdev) · 38 unmodified lines
102	mutex_init(&nvdev->remap_lock);	106	mutex_init(&nvdev->remap_lock);
103	}	107	}
104		108
		109	/*
		110	* GPU readiness is checked by reading the BAR0 registers.
		111	*
		112	* ioremap BAR0 to ensure that the BAR0 mapping is present before
		113	* register reads on first fault before establishing any GPU
		114	* memory mapping.
		115	*/
		116	ret = vfio_pci_core_setup_barmap(vdev, 0);
		117	if (ret) {
		118	vfio_pci_core_disable(vdev);
		119	return ret;
		120	}
		121
105	vfio_pci_core_finish_enable(vdev);	122	vfio_pci_core_finish_enable(vdev);
106		123
107	return 0;	124	return 0;
@@ -130,6 +147,106 @@ static void nvgrace_gpu_close_device(struct vfio_device *core_vdev) · 22 unmodified lines
130	vfio_pci_core_close_device(core_vdev);	147	vfio_pci_core_close_device(core_vdev);
131	}	148	}
132		149
		150	static int nvgrace_gpu_wait_device_ready(void __iomem *io)
		151	{
		152	unsigned long timeout = jiffies + msecs_to_jiffies(POLL_TIMEOUT_MS);
		153
		154	do {
		155	if ((ioread32(io + C2C_LINK_BAR0_OFFSET) == STATUS_READY) &&
		156	(ioread32(io + HBM_TRAINING_BAR0_OFFSET) == STATUS_READY))
		157	return 0;
		158	msleep(POLL_QUANTUM_MS);
		159	} while (!time_after(jiffies, timeout));
		160
		161	return -ETIME;
		162	}
		163
		164	/*
		165	* If the GPU memory is accessed by the CPU while the GPU is not ready
		166	* after reset, it can cause harmless corrected RAS events to be logged.
		167	* Make sure the GPU is ready before establishing the mappings.
		168	*/
		169	static int
		170	nvgrace_gpu_check_device_ready(struct nvgrace_gpu_pci_core_device *nvdev)
		171	{
		172	struct vfio_pci_core_device *vdev = &nvdev->core_device;
		173	int ret;
		174
		175	lockdep_assert_held_read(&vdev->memory_lock);
		176
		177	if (!nvdev->reset_done)
		178	return 0;
		179
		180	if (!__vfio_pci_memory_enabled(vdev))
		181	return -EIO;
		182
		183	ret = nvgrace_gpu_wait_device_ready(vdev->barmap[0]);
		184	if (ret)
		185	return ret;
		186
		187	nvdev->reset_done = false;
		188
		189	return 0;
		190	}
		191
		192	static unsigned long addr_to_pgoff(struct vm_area_struct *vma,
		193	unsigned long addr)
		194	{
		195	u64 pgoff = vma->vm_pgoff &
		196	((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
		197
		198	return ((addr - vma->vm_start) >> PAGE_SHIFT) + pgoff;
		199	}
		200
		201	static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf,
		202	unsigned int order)
		203	{
		204	struct vm_area_struct *vma = vmf->vma;
		205	struct nvgrace_gpu_pci_core_device *nvdev = vma->vm_private_data;
		206	struct vfio_pci_core_device *vdev = &nvdev->core_device;
		207	unsigned int index =
		208	vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
		209	vm_fault_t ret = VM_FAULT_FALLBACK;
		210	struct mem_region *memregion;
		211	unsigned long pfn, addr;
		212
		213	memregion = nvgrace_gpu_memregion(index, nvdev);
		214	if (!memregion)
		215	return VM_FAULT_SIGBUS;
		216
		217	addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
		218	pfn = PHYS_PFN(memregion->memphys) + addr_to_pgoff(vma, addr);
		219
		220	if (is_aligned_for_order(vma, addr, pfn, order)) {
		221	scoped_guard(rwsem_read, &vdev->memory_lock) {
		222	if (vdev->pm_runtime_engaged \|\|
		223	nvgrace_gpu_check_device_ready(nvdev))
		224	return VM_FAULT_SIGBUS;
		225
		226	ret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order);
		227	}
		228	}
		229
		230	dev_dbg_ratelimited(&vdev->pdev->dev,
		231	"%s order = %d pfn 0x%lx: 0x%x\n",
		232	__func__, order, pfn,
		233	(unsigned int)ret);
		234
		235	return ret;
		236	}
		237
		238	static vm_fault_t nvgrace_gpu_vfio_pci_fault(struct vm_fault *vmf)
		239	{
		240	return nvgrace_gpu_vfio_pci_huge_fault(vmf, 0);
		241	}
		242
		243	static const struct vm_operations_struct nvgrace_gpu_vfio_pci_mmap_ops = {
		244	.fault = nvgrace_gpu_vfio_pci_fault,
		245	#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP
		246	.huge_fault = nvgrace_gpu_vfio_pci_huge_fault,
		247	#endif
		248	};
		249
133	static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,	250	static int nvgrace_gpu_mmap(struct vfio_device *core_vdev,
134	struct vm_area_struct *vma)	251	struct vm_area_struct *vma)
135	{	252	{
@@ -137,10 +254,8 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev, · 1 unmodified line
137	container_of(core_vdev, struct nvgrace_gpu_pci_core_device,	254	container_of(core_vdev, struct nvgrace_gpu_pci_core_device,
138	core_device.vdev);	255	core_device.vdev);
139	struct mem_region *memregion;	256	struct mem_region *memregion;
140	unsigned long start_pfn;
141	u64 req_len, pgoff, end;	257	u64 req_len, pgoff, end;
142	unsigned int index;	258	unsigned int index;
143	int ret = 0;
144		259
145	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);	260	index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT);
146		261
@@ -157,17 +272,18 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev, · 10 unmodified lines
157	((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);	272	((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
158		273
159	if (check_sub_overflow(vma->vm_end, vma->vm_start, &req_len) \|\|	274	if (check_sub_overflow(vma->vm_end, vma->vm_start, &req_len) \|\|
160	check_add_overflow(PHYS_PFN(memregion->memphys), pgoff, &start_pfn) \|\|
161	check_add_overflow(PFN_PHYS(pgoff), req_len, &end))	275	check_add_overflow(PFN_PHYS(pgoff), req_len, &end))
162	return -EOVERFLOW;	276	return -EOVERFLOW;
163		277
164	/*	278	/*
165	* Check that the mapping request does not go beyond available device	279	* Check that the mapping request does not go beyond the exposed
166	* memory size	280	* device memory size.
167	*/	281	*/
168	if (end > memregion->memlength)	282	if (end > memregion->memlength)
169	return -EINVAL;	283	return -EINVAL;
170		284
		285	vm_flags_set(vma, VM_IO \| VM_PFNMAP \| VM_DONTEXPAND \| VM_DONTDUMP);
		286
171	/*	287	/*
172	* The carved out region of the device memory needs the NORMAL_NC	288	* The carved out region of the device memory needs the NORMAL_NC
173	* property. Communicate as such to the hypervisor.	289	* property. Communicate as such to the hypervisor.
@@ -184,56 +300,31 @@ static int nvgrace_gpu_mmap(struct vfio_device *core_vdev, · 10 unmodified lines
184	vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);	300	vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
185	}	301	}
186		302
187	/*	303	vma->vm_ops = &nvgrace_gpu_vfio_pci_mmap_ops;
188	* Perform a PFN map to the memory and back the device BAR by the	304	vma->vm_private_data = nvdev;
189	* GPU memory.
190	*
191	* The available GPU memory size may not be power-of-2 aligned. The
192	* remainder is only backed by vfio_device_ops read/write handlers.
193	*
194	* During device reset, the GPU is safely disconnected to the CPU
195	* and access to the BAR will be immediately returned preventing
196	* machine check.
197	*/
198	ret = remap_pfn_range(vma, vma->vm_start, start_pfn,
199	req_len, vma->vm_page_prot);
200	if (ret)
201	return ret;
202
203	vma->vm_pgoff = start_pfn;
204		305
205	return 0;	306	return 0;
206	}	307	}
207		308
208	static long	309	static int nvgrace_gpu_ioctl_get_region_info(struct vfio_device *core_vdev,
209	nvgrace_gpu_ioctl_get_region_info(struct vfio_device *core_vdev,	310	struct vfio_region_info *info,
210	unsigned long arg)	311	struct vfio_info_cap *caps)
211	{	312	{
212	struct nvgrace_gpu_pci_core_device *nvdev =	313	struct nvgrace_gpu_pci_core_device *nvdev =
213	container_of(core_vdev, struct nvgrace_gpu_pci_core_device,	314	container_of(core_vdev, struct nvgrace_gpu_pci_core_device,
214	core_device.vdev);	315	core_device.vdev);
215	unsigned long minsz = offsetofend(struct vfio_region_info, offset);
216	struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
217	struct vfio_region_info_cap_sparse_mmap *sparse;	316	struct vfio_region_info_cap_sparse_mmap *sparse;
218	struct vfio_region_info info;
219	struct mem_region *memregion;	317	struct mem_region *memregion;
220	u32 size;	318	u32 size;
221	int ret;	319	int ret;
222		320
223	if (copy_from_user(&info, (void __user *)arg, minsz))
224	return -EFAULT;
225
226	if (info.argsz < minsz)
227	return -EINVAL;
228
229	/*	321	/*
230	* Request to determine the BAR region information. Send the	322	* Request to determine the BAR region information. Send the
231	* GPU memory information.	323	* GPU memory information.
232	*/	324	*/
233	memregion = nvgrace_gpu_memregion(info.index, nvdev);	325	memregion = nvgrace_gpu_memregion(info->index, nvdev);
234	if (!memregion)	326	if (!memregion)
235	return vfio_pci_core_ioctl(core_vdev,	327	return vfio_pci_ioctl_get_region_info(core_vdev, info, caps);
236	VFIO_DEVICE_GET_REGION_INFO, arg);
237		328
238	size = struct_size(sparse, areas, 1);	329	size = struct_size(sparse, areas, 1);
239		330
@@ -252,49 +343,28 @@ nvgrace_gpu_ioctl_get_region_info(struct vfio_device *core_vdev, · 12 unmodified lines
252	sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;	343	sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
253	sparse->header.version = 1;	344	sparse->header.version = 1;
254		345
255	ret = vfio_info_add_capability(&caps, &sparse->header, size);	346	ret = vfio_info_add_capability(caps, &sparse->header, size);
256	kfree(sparse);	347	kfree(sparse);
257	if (ret)	348	if (ret)
258	return ret;	349	return ret;
259		350
260	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);	351	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
261	/*	352	/*
262	* The region memory size may not be power-of-2 aligned.	353	* The region memory size may not be power-of-2 aligned.
263	* Given that the memory is a BAR and may not be	354	* Given that the memory is a BAR and may not be
264	* aligned, roundup to the next power-of-2.	355	* aligned, roundup to the next power-of-2.
265	*/	356	*/
266	info.size = memregion->bar_size;	357	info->size = memregion->bar_size;
267	info.flags = VFIO_REGION_INFO_FLAG_READ \|	358	info->flags = VFIO_REGION_INFO_FLAG_READ \|
268	VFIO_REGION_INFO_FLAG_WRITE \|	359	VFIO_REGION_INFO_FLAG_WRITE \|
269	VFIO_REGION_INFO_FLAG_MMAP;	360	VFIO_REGION_INFO_FLAG_MMAP;
270		361	return 0;
271	if (caps.size) {
272	info.flags \|= VFIO_REGION_INFO_FLAG_CAPS;
273	if (info.argsz < sizeof(info) + caps.size) {
274	info.argsz = sizeof(info) + caps.size;
275	info.cap_offset = 0;
276	} else {
277	vfio_info_cap_shift(&caps, sizeof(info));
278	if (copy_to_user((void __user *)arg +
279	sizeof(info), caps.buf,
280	caps.size)) {
281	kfree(caps.buf);
282	return -EFAULT;
283	}
284	info.cap_offset = sizeof(info);
285	}
286	kfree(caps.buf);
287	}
288	return copy_to_user((void __user *)arg, &info, minsz) ?
289	-EFAULT : 0;
290	}	362	}
291		363
292	static long nvgrace_gpu_ioctl(struct vfio_device *core_vdev,	364	static long nvgrace_gpu_ioctl(struct vfio_device *core_vdev,
293	unsigned int cmd, unsigned long arg)	365	unsigned int cmd, unsigned long arg)
294	{	366	{
295	switch (cmd) {	367	switch (cmd) {
296	case VFIO_DEVICE_GET_REGION_INFO:
297	return nvgrace_gpu_ioctl_get_region_info(core_vdev, arg);
298	case VFIO_DEVICE_IOEVENTFD:	368	case VFIO_DEVICE_IOEVENTFD:
299	return -ENOTTY;	369	return -ENOTTY;
300	case VFIO_DEVICE_RESET:	370	case VFIO_DEVICE_RESET:
@@ -510,6 +580,7 @@ static ssize_t · 209 unmodified lines
510	nvgrace_gpu_read_mem(struct nvgrace_gpu_pci_core_device *nvdev,	580	nvgrace_gpu_read_mem(struct nvgrace_gpu_pci_core_device *nvdev,
511	char __user buf, size_t count, loff_t ppos)	581	char __user buf, size_t count, loff_t ppos)
512	{	582	{
		583	struct vfio_pci_core_device *vdev = &nvdev->core_device;
513	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;	584	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
514	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);	585	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
515	struct mem_region *memregion;	586	struct mem_region *memregion;
@@ -536,9 +607,15 @@ nvgrace_gpu_read_mem(struct nvgrace_gpu_pci_core_device *nvdev, · 20 unmodified lines
536	else	607	else
537	mem_count = min(count, memregion->memlength - (size_t)offset);	608	mem_count = min(count, memregion->memlength - (size_t)offset);
538		609
539	ret = nvgrace_gpu_map_and_read(nvdev, buf, mem_count, ppos);	610	scoped_guard(rwsem_read, &vdev->memory_lock) {
540	if (ret)	611	ret = nvgrace_gpu_check_device_ready(nvdev);
541	return ret;	612	if (ret)
		613	return ret;
		614
		615	ret = nvgrace_gpu_map_and_read(nvdev, buf, mem_count, ppos);
		616	if (ret)
		617	return ret;
		618	}
542		619
543	/*	620	/*
544	* Only the device memory present on the hardware is mapped, which may	621	* Only the device memory present on the hardware is mapped, which may
@@ -563,9 +640,16 @@ nvgrace_gpu_read(struct vfio_device *core_vdev, · 18 unmodified lines
563	struct nvgrace_gpu_pci_core_device *nvdev =	640	struct nvgrace_gpu_pci_core_device *nvdev =
564	container_of(core_vdev, struct nvgrace_gpu_pci_core_device,	641	container_of(core_vdev, struct nvgrace_gpu_pci_core_device,
565	core_device.vdev);	642	core_device.vdev);
		643	struct vfio_pci_core_device *vdev = &nvdev->core_device;
		644	int ret;
566		645
567	if (nvgrace_gpu_memregion(index, nvdev))	646	if (nvgrace_gpu_memregion(index, nvdev)) {
568	return nvgrace_gpu_read_mem(nvdev, buf, count, ppos);	647	if (pm_runtime_resume_and_get(&vdev->pdev->dev))
		648	return -EIO;
		649	ret = nvgrace_gpu_read_mem(nvdev, buf, count, ppos);
		650	pm_runtime_put(&vdev->pdev->dev);
		651	return ret;
		652	}
569		653
570	if (index == VFIO_PCI_CONFIG_REGION_INDEX)	654	if (index == VFIO_PCI_CONFIG_REGION_INDEX)
571	return nvgrace_gpu_read_config_emu(core_vdev, buf, count, ppos);	655	return nvgrace_gpu_read_config_emu(core_vdev, buf, count, ppos);
@@ -627,6 +711,7 @@ static ssize_t · 55 unmodified lines
627	nvgrace_gpu_write_mem(struct nvgrace_gpu_pci_core_device *nvdev,	711	nvgrace_gpu_write_mem(struct nvgrace_gpu_pci_core_device *nvdev,
628	size_t count, loff_t ppos, const char __user buf)	712	size_t count, loff_t ppos, const char __user buf)
629	{	713	{
		714	struct vfio_pci_core_device *vdev = &nvdev->core_device;
630	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);	715	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
631	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;	716	u64 offset = *ppos & VFIO_PCI_OFFSET_MASK;
632	struct mem_region *memregion;	717	struct mem_region *memregion;
@@ -656,9 +741,15 @@ nvgrace_gpu_write_mem(struct nvgrace_gpu_pci_core_device *nvdev, · 23 unmodified lines
656	*/	741	*/
657	mem_count = min(count, memregion->memlength - (size_t)offset);	742	mem_count = min(count, memregion->memlength - (size_t)offset);
658		743
659	ret = nvgrace_gpu_map_and_write(nvdev, buf, mem_count, ppos);	744	scoped_guard(rwsem_read, &vdev->memory_lock) {
660	if (ret)	745	ret = nvgrace_gpu_check_device_ready(nvdev);
661	return ret;	746	if (ret)
		747	return ret;
		748
		749	ret = nvgrace_gpu_map_and_write(nvdev, buf, mem_count, ppos);
		750	if (ret)
		751	return ret;
		752	}
662		753
663	exitfn:	754	exitfn:
664	*ppos += count;	755	*ppos += count;
@@ -672,10 +763,17 @@ nvgrace_gpu_write(struct vfio_device *core_vdev, · 7 unmodified lines
672	struct nvgrace_gpu_pci_core_device *nvdev =	763	struct nvgrace_gpu_pci_core_device *nvdev =
673	container_of(core_vdev, struct nvgrace_gpu_pci_core_device,	764	container_of(core_vdev, struct nvgrace_gpu_pci_core_device,
674	core_device.vdev);	765	core_device.vdev);
		766	struct vfio_pci_core_device *vdev = &nvdev->core_device;
675	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);	767	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
		768	int ret;
676		769
677	if (nvgrace_gpu_memregion(index, nvdev))	770	if (nvgrace_gpu_memregion(index, nvdev)) {
678	return nvgrace_gpu_write_mem(nvdev, count, ppos, buf);	771	if (pm_runtime_resume_and_get(&vdev->pdev->dev))
		772	return -EIO;
		773	ret = nvgrace_gpu_write_mem(nvdev, count, ppos, buf);
		774	pm_runtime_put(&vdev->pdev->dev);
		775	return ret;
		776	}
679		777
680	if (index == VFIO_PCI_CONFIG_REGION_INDEX)	778	if (index == VFIO_PCI_CONFIG_REGION_INDEX)
681	return nvgrace_gpu_write_config_emu(core_vdev, buf, count, ppos);	779	return nvgrace_gpu_write_config_emu(core_vdev, buf, count, ppos);
@@ -683,6 +781,50 @@ nvgrace_gpu_write(struct vfio_device *core_vdev, · 1 unmodified line
683	return vfio_pci_core_write(core_vdev, buf, count, ppos);	781	return vfio_pci_core_write(core_vdev, buf, count, ppos);
684	}	782	}
685		783
		784	static int nvgrace_get_dmabuf_phys(struct vfio_pci_core_device *core_vdev,
		785	struct p2pdma_provider **provider,
		786	unsigned int region_index,
		787	struct dma_buf_phys_vec *phys_vec,
		788	struct vfio_region_dma_range *dma_ranges,
		789	size_t nr_ranges)
		790	{
		791	struct nvgrace_gpu_pci_core_device *nvdev = container_of(
		792	core_vdev, struct nvgrace_gpu_pci_core_device, core_device);
		793	struct pci_dev *pdev = core_vdev->pdev;
		794	struct mem_region *mem_region;
		795
		796	/*
		797	* if (nvdev->resmem.memlength && region_index == RESMEM_REGION_INDEX) {
		798	* The P2P properties of the non-BAR memory is the same as the
		799	* BAR memory, so just use the provider for index 0. Someday
		800	* when CXL gets P2P support we could create CXLish providers
		801	* for the non-BAR memory.
		802	* } else if (region_index == USEMEM_REGION_INDEX) {
		803	* This is actually cachable memory and isn't treated as P2P in
		804	* the chip. For now we have no way to push cachable memory
		805	* through everything and the Grace HW doesn't care what caching
		806	* attribute is programmed into the SMMU. So use BAR 0.
		807	* }
		808	*/
		809	mem_region = nvgrace_gpu_memregion(region_index, nvdev);
		810	if (mem_region) {
		811	*provider = pcim_p2pdma_provider(pdev, 0);
		812	if (!*provider)
		813	return -EINVAL;
		814	return vfio_pci_core_fill_phys_vec(phys_vec, dma_ranges,
		815	nr_ranges,
		816	mem_region->memphys,
		817	mem_region->memlength);
		818	}
		819
		820	return vfio_pci_core_get_dmabuf_phys(core_vdev, provider, region_index,
		821	phys_vec, dma_ranges, nr_ranges);
		822	}
		823
		824	static const struct vfio_pci_device_ops nvgrace_gpu_pci_dev_ops = {
		825	.get_dmabuf_phys = nvgrace_get_dmabuf_phys,
		826	};
		827
686	static const struct vfio_device_ops nvgrace_gpu_pci_ops = {	828	static const struct vfio_device_ops nvgrace_gpu_pci_ops = {
687	.name = "nvgrace-gpu-vfio-pci",	829	.name = "nvgrace-gpu-vfio-pci",
688	.init = vfio_pci_core_init_dev,	830	.init = vfio_pci_core_init_dev,
@@ -690,6 +832,7 @@ static const struct vfio_device_ops nvgrace_gpu_pci_ops = { · 1 unmodified line
690	.open_device = nvgrace_gpu_open_device,	832	.open_device = nvgrace_gpu_open_device,
691	.close_device = nvgrace_gpu_close_device,	833	.close_device = nvgrace_gpu_close_device,
692	.ioctl = nvgrace_gpu_ioctl,	834	.ioctl = nvgrace_gpu_ioctl,
		835	.get_region_info_caps = nvgrace_gpu_ioctl_get_region_info,
693	.device_feature = vfio_pci_core_ioctl_feature,	836	.device_feature = vfio_pci_core_ioctl_feature,
694	.read = nvgrace_gpu_read,	837	.read = nvgrace_gpu_read,
695	.write = nvgrace_gpu_write,	838	.write = nvgrace_gpu_write,
@@ -703,6 +846,10 @@ static const struct vfio_device_ops nvgrace_gpu_pci_ops = { · 7 unmodified lines
703	.detach_ioas = vfio_iommufd_physical_detach_ioas,	846	.detach_ioas = vfio_iommufd_physical_detach_ioas,
704	};	847	};
705		848
		849	static const struct vfio_pci_device_ops nvgrace_gpu_pci_dev_core_ops = {
		850	.get_dmabuf_phys = vfio_pci_core_get_dmabuf_phys,
		851	};
		852
706	static const struct vfio_device_ops nvgrace_gpu_pci_core_ops = {	853	static const struct vfio_device_ops nvgrace_gpu_pci_core_ops = {
707	.name = "nvgrace-gpu-vfio-pci-core",	854	.name = "nvgrace-gpu-vfio-pci-core",
708	.init = vfio_pci_core_init_dev,	855	.init = vfio_pci_core_init_dev,
@@ -710,6 +857,7 @@ static const struct vfio_device_ops nvgrace_gpu_pci_core_ops = { · 1 unmodified line
710	.open_device = nvgrace_gpu_open_device,	857	.open_device = nvgrace_gpu_open_device,
711	.close_device = vfio_pci_core_close_device,	858	.close_device = vfio_pci_core_close_device,
712	.ioctl = vfio_pci_core_ioctl,	859	.ioctl = vfio_pci_core_ioctl,
		860	.get_region_info_caps = vfio_pci_ioctl_get_region_info,
713	.device_feature = vfio_pci_core_ioctl_feature,	861	.device_feature = vfio_pci_core_ioctl_feature,
714	.read = vfio_pci_core_read,	862	.read = vfio_pci_core_read,
715	.write = vfio_pci_core_write,	863	.write = vfio_pci_core_write,
@@ -893,11 +1041,10 @@ static bool nvgrace_gpu_has_mig_hw_bug(struct pci_dev *pdev) · 177 unmodified lines
893	* Ensure that the BAR0 region is enabled before accessing the	1041	* Ensure that the BAR0 region is enabled before accessing the
894	* registers.	1042	* registers.
895	*/	1043	*/
896	static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev)	1044	static int nvgrace_gpu_probe_check_device_ready(struct pci_dev *pdev)
897	{	1045	{
898	unsigned long timeout = jiffies + msecs_to_jiffies(POLL_TIMEOUT_MS);
899	void __iomem *io;	1046	void __iomem *io;
900	int ret = -ETIME;	1047	int ret;
901		1048
902	ret = pci_enable_device(pdev);	1049	ret = pci_enable_device(pdev);
903	if (ret)	1050	if (ret)
@@ -913,16 +1060,8 @@ static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev) · 9 unmodified lines
913	goto iomap_exit;	1060	goto iomap_exit;
914	}	1061	}
915		1062
916	do {	1063	ret = nvgrace_gpu_wait_device_ready(io);
917	if ((ioread32(io + C2C_LINK_BAR0_OFFSET) == STATUS_READY) &&
918	(ioread32(io + HBM_TRAINING_BAR0_OFFSET) == STATUS_READY)) {
919	ret = 0;
920	goto reg_check_exit;
921	}
922	msleep(POLL_QUANTUM_MS);
923	} while (!time_after(jiffies, timeout));
924		1064
925	reg_check_exit:
926	pci_iounmap(pdev, io);	1065	pci_iounmap(pdev, io);
927	iomap_exit:	1066	iomap_exit:
928	pci_release_selected_regions(pdev, 1 << 0);	1067	pci_release_selected_regions(pdev, 1 << 0);
@@ -939,7 +1078,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, · 10 unmodified lines
939	u64 memphys, memlength;	1078	u64 memphys, memlength;
940	int ret;	1079	int ret;
941		1080
942	ret = nvgrace_gpu_wait_device_ready(pdev);	1081	ret = nvgrace_gpu_probe_check_device_ready(pdev);
943	if (ret)	1082	if (ret)
944	return ret;	1083	return ret;
945		1084
@@ -965,6 +1104,9 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, · 19 unmodified lines
965	memphys, memlength);	1104	memphys, memlength);
966	if (ret)	1105	if (ret)
967	goto out_put_vdev;	1106	goto out_put_vdev;
		1107	nvdev->core_device.pci_ops = &nvgrace_gpu_pci_dev_ops;
		1108	} else {
		1109	nvdev->core_device.pci_ops = &nvgrace_gpu_pci_dev_core_ops;
968	}	1110	}
969		1111
970	ret = vfio_pci_core_register_device(&nvdev->core_device);	1112	ret = vfio_pci_core_register_device(&nvdev->core_device);
@@ -1002,12 +1144,38 @@ static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = { · 31 unmodified lines
1002		1144
1003	MODULE_DEVICE_TABLE(pci, nvgrace_gpu_vfio_pci_table);	1145	MODULE_DEVICE_TABLE(pci, nvgrace_gpu_vfio_pci_table);
1004		1146
		1147	/*
		1148	* The GPU reset is required to be serialized against the first mapping
		1149	* faults and read/writes accesses to prevent potential RAS events logging.
		1150	*
		1151	* First fault or access after a reset needs to poll device readiness,
		1152	* flag that a reset has occurred. The readiness test is done by holding
		1153	* the memory_lock read lock and we expect all vfio-pci initiated resets to
		1154	* hold the memory_lock write lock to avoid races. However, .reset_done
		1155	* extends beyond the scope of vfio-pci initiated resets therefore we
		1156	* cannot assert this behavior and use lockdep_assert_held_write.
		1157	*/
		1158	static void nvgrace_gpu_vfio_pci_reset_done(struct pci_dev *pdev)
		1159	{
		1160	struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev);
		1161	struct nvgrace_gpu_pci_core_device *nvdev =
		1162	container_of(core_device, struct nvgrace_gpu_pci_core_device,
		1163	core_device);
		1164
		1165	nvdev->reset_done = true;
		1166	}
		1167
		1168	static const struct pci_error_handlers nvgrace_gpu_vfio_pci_err_handlers = {
		1169	.reset_done = nvgrace_gpu_vfio_pci_reset_done,
		1170	.error_detected = vfio_pci_core_aer_err_detected,
		1171	};
		1172
1005	static struct pci_driver nvgrace_gpu_vfio_pci_driver = {	1173	static struct pci_driver nvgrace_gpu_vfio_pci_driver = {
1006	.name = KBUILD_MODNAME,	1174	.name = KBUILD_MODNAME,
1007	.id_table = nvgrace_gpu_vfio_pci_table,	1175	.id_table = nvgrace_gpu_vfio_pci_table,
1008	.probe = nvgrace_gpu_probe,	1176	.probe = nvgrace_gpu_probe,
1009	.remove = nvgrace_gpu_remove,	1177	.remove = nvgrace_gpu_remove,
1010	.err_handler = &vfio_pci_core_err_handlers,	1178	.err_handler = &nvgrace_gpu_vfio_pci_err_handlers,
1011	.driver_managed_dma = true,	1179	.driver_managed_dma = true,
1012	};	1180	};
1013		1181

drivers/vfio/pci/pds/vfio_dev.c+1−0

@@ -195,6 +195,7 @@ static const struct vfio_device_ops pds_vfio_ops = { · 194 unmodified lines
195	.open_device = pds_vfio_open_device,	195	.open_device = pds_vfio_open_device,
196	.close_device = pds_vfio_close_device,	196	.close_device = pds_vfio_close_device,
197	.ioctl = vfio_pci_core_ioctl,	197	.ioctl = vfio_pci_core_ioctl,
		198	.get_region_info_caps = vfio_pci_ioctl_get_region_info,
198	.device_feature = vfio_pci_core_ioctl_feature,	199	.device_feature = vfio_pci_core_ioctl_feature,
199	.read = vfio_pci_core_read,	200	.read = vfio_pci_core_read,
200	.write = vfio_pci_core_write,	201	.write = vfio_pci_core_write,

drivers/vfio/pci/qat/main.c+1−0

@@ -609,6 +609,7 @@ static const struct vfio_device_ops qat_vf_pci_ops = { · 608 unmodified lines
609	.open_device = qat_vf_pci_open_device,	609	.open_device = qat_vf_pci_open_device,
610	.close_device = qat_vf_pci_close_device,	610	.close_device = qat_vf_pci_close_device,
611	.ioctl = vfio_pci_core_ioctl,	611	.ioctl = vfio_pci_core_ioctl,
		612	.get_region_info_caps = vfio_pci_ioctl_get_region_info,
612	.read = vfio_pci_core_read,	613	.read = vfio_pci_core_read,
613	.write = vfio_pci_core_write,	614	.write = vfio_pci_core_write,
614	.mmap = vfio_pci_core_mmap,	615	.mmap = vfio_pci_core_mmap,

drivers/vfio/pci/vfio_pci.c+6−0

 	.open_device	= vfio_pci_open_device,
 	.close_device	= vfio_pci_core_close_device,
 	.ioctl		= vfio_pci_core_ioctl,
+	.get_region_info_caps = vfio_pci_ioctl_get_region_info,
 	.device_feature = vfio_pci_core_ioctl_feature,
 	.read		= vfio_pci_core_read,
 	.write		= vfio_pci_core_write,
 	.pasid_detach_ioas	= vfio_iommufd_physical_pasid_detach_ioas,
 };
+static const struct vfio_pci_device_ops vfio_pci_dev_ops = {
+	.get_dmabuf_phys = vfio_pci_core_get_dmabuf_phys,
+};
 static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
 	struct vfio_pci_core_device *vdev;
 		return PTR_ERR(vdev);
 	dev_set_drvdata(&pdev->dev, vdev);
+	vdev->pci_ops = &vfio_pci_dev_ops;
 	ret = vfio_pci_core_register_device(vdev);
 	if (ret)
 		goto out_put_vdev;

drivers/vfio/pci/vfio_pci_config.c+19−4

 	return pdev->current_state < PCI_D3hot &&
 	       (pdev->no_command_memory || (cmd & PCI_COMMAND_MEMORY));
 }
+EXPORT_SYMBOL_GPL(__vfio_pci_memory_enabled);
 /*
  * Restore the *real* BARs after we detect a FLR or backdoor reset.
 		virt_mem = !!(le16_to_cpu(*virt_cmd) & PCI_COMMAND_MEMORY);
 		new_mem = !!(new_cmd & PCI_COMMAND_MEMORY);
-		if (!new_mem)
+		if (!new_mem) {
 			vfio_pci_zap_and_down_write_memory_lock(vdev);
-		else
+			vfio_pci_dma_buf_move(vdev, true);
+		} else {
 			down_write(&vdev->memory_lock);
+		}
 		/*
 		 * If the user is writing mem/io enable (new_mem/io) and we
 		*virt_cmd &= cpu_to_le16(~mask);
 		*virt_cmd |= cpu_to_le16(new_cmd & mask);
+		if (__vfio_pci_memory_enabled(vdev))
+			vfio_pci_dma_buf_move(vdev, false);
 		up_write(&vdev->memory_lock);
 	}
 static void vfio_lock_and_set_power_state(struct vfio_pci_core_device *vdev,
 					  pci_power_t state)
 {
-	if (state >= PCI_D3hot)
+	if (state >= PCI_D3hot) {
 		vfio_pci_zap_and_down_write_memory_lock(vdev);
-	else
+		vfio_pci_dma_buf_move(vdev, true);
+	} else {
 		down_write(&vdev->memory_lock);
+	}
 	vfio_pci_set_power_state(vdev, state);
+	if (__vfio_pci_memory_enabled(vdev))
+		vfio_pci_dma_buf_move(vdev, false);
 	up_write(&vdev->memory_lock);
 }
 		if (!ret && (cap & PCI_EXP_DEVCAP_FLR)) {
 			vfio_pci_zap_and_down_write_memory_lock(vdev);
+			vfio_pci_dma_buf_move(vdev, true);
 			pci_try_reset_function(vdev->pdev);
+			if (__vfio_pci_memory_enabled(vdev))
+				vfio_pci_dma_buf_move(vdev, false);
 			up_write(&vdev->memory_lock);
 		}
 	}
 		if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP)) {
 			vfio_pci_zap_and_down_write_memory_lock(vdev);
+			vfio_pci_dma_buf_move(vdev, true);
 			pci_try_reset_function(vdev->pdev);
+			if (__vfio_pci_memory_enabled(vdev))
+				vfio_pci_dma_buf_move(vdev, false);
 			up_write(&vdev->memory_lock);
 		}
 	}

drivers/vfio/pci/vfio_pci_core.c+156−144

@@ -28,6 +28,7 @@ · 27 unmodified lines
28	#include <linux/nospec.h>	28	#include <linux/nospec.h>
29	#include <linux/sched/mm.h>	29	#include <linux/sched/mm.h>
30	#include <linux/iommufd.h>	30	#include <linux/iommufd.h>
		31	#include <linux/pci-p2pdma.h>
31	#if IS_ENABLED(CONFIG_EEH)	32	#if IS_ENABLED(CONFIG_EEH)
32	#include <asm/eeh.h>	33	#include <asm/eeh.h>
33	#endif	34	#endif
@@ -41,6 +42,40 @@ static bool nointxmask; · 7 unmodified lines
41	static bool disable_vga;	42	static bool disable_vga;
42	static bool disable_idle_d3;	43	static bool disable_idle_d3;
43		44
		45	static void vfio_pci_eventfd_rcu_free(struct rcu_head *rcu)
		46	{
		47	struct vfio_pci_eventfd *eventfd =
		48	container_of(rcu, struct vfio_pci_eventfd, rcu);
		49
		50	eventfd_ctx_put(eventfd->ctx);
		51	kfree(eventfd);
		52	}
		53
		54	int vfio_pci_eventfd_replace_locked(struct vfio_pci_core_device *vdev,
		55	struct vfio_pci_eventfd __rcu **peventfd,
		56	struct eventfd_ctx *ctx)
		57	{
		58	struct vfio_pci_eventfd *new = NULL;
		59	struct vfio_pci_eventfd *old;
		60
		61	lockdep_assert_held(&vdev->igate);
		62
		63	if (ctx) {
		64	new = kzalloc(sizeof(*new), GFP_KERNEL_ACCOUNT);
		65	if (!new)
		66	return -ENOMEM;
		67
		68	new->ctx = ctx;
		69	}
		70
		71	old = rcu_replace_pointer(*peventfd, new,
		72	lockdep_is_held(&vdev->igate));
		73	if (old)
		74	call_rcu(&old->rcu, vfio_pci_eventfd_rcu_free);
		75
		76	return 0;
		77	}
		78
44	/* List of PF's that vfio_pci_core_sriov_configure() has been called on */	79	/* List of PF's that vfio_pci_core_sriov_configure() has been called on */
45	static DEFINE_MUTEX(vfio_pci_sriov_pfs_mutex);	80	static DEFINE_MUTEX(vfio_pci_sriov_pfs_mutex);
46	static LIST_HEAD(vfio_pci_sriov_pfs);	81	static LIST_HEAD(vfio_pci_sriov_pfs);
@@ -286,6 +321,8 @@ static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev, · 239 unmodified lines
286	* semaphore.	321	* semaphore.
287	*/	322	*/
288	vfio_pci_zap_and_down_write_memory_lock(vdev);	323	vfio_pci_zap_and_down_write_memory_lock(vdev);
		324	vfio_pci_dma_buf_move(vdev, true);
		325
289	if (vdev->pm_runtime_engaged) {	326	if (vdev->pm_runtime_engaged) {
290	up_write(&vdev->memory_lock);	327	up_write(&vdev->memory_lock);
291	return -EINVAL;	328	return -EINVAL;
@@ -299,11 +336,9 @@ static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev, · 7 unmodified lines
299	return 0;	336	return 0;
300	}	337	}
301		338
302	static int vfio_pci_core_pm_entry(struct vfio_device *device, u32 flags,	339	static int vfio_pci_core_pm_entry(struct vfio_pci_core_device *vdev, u32 flags,
303	void __user *arg, size_t argsz)	340	void __user *arg, size_t argsz)
304	{	341	{
305	struct vfio_pci_core_device *vdev =
306	container_of(device, struct vfio_pci_core_device, vdev);
307	int ret;	342	int ret;
308		343
309	ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET, 0);	344	ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET, 0);
@@ -320,12 +355,10 @@ static int vfio_pci_core_pm_entry(struct vfio_device *device, u32 flags, · 10 unmodified lines
320	}	355	}
321		356
322	static int vfio_pci_core_pm_entry_with_wakeup(	357	static int vfio_pci_core_pm_entry_with_wakeup(
323	struct vfio_device *device, u32 flags,	358	struct vfio_pci_core_device *vdev, u32 flags,
324	struct vfio_device_low_power_entry_with_wakeup __user *arg,	359	struct vfio_device_low_power_entry_with_wakeup __user *arg,
325	size_t argsz)	360	size_t argsz)
326	{	361	{
327	struct vfio_pci_core_device *vdev =
328	container_of(device, struct vfio_pci_core_device, vdev);
329	struct vfio_device_low_power_entry_with_wakeup entry;	362	struct vfio_device_low_power_entry_with_wakeup entry;
330	struct eventfd_ctx *efdctx;	363	struct eventfd_ctx *efdctx;
331	int ret;	364	int ret;
@@ -373,14 +406,14 @@ static void vfio_pci_runtime_pm_exit(struct vfio_pci_core_device *vdev) · 41 unmodified lines
373	*/	406	*/
374	down_write(&vdev->memory_lock);	407	down_write(&vdev->memory_lock);
375	__vfio_pci_runtime_pm_exit(vdev);	408	__vfio_pci_runtime_pm_exit(vdev);
		409	if (__vfio_pci_memory_enabled(vdev))
		410	vfio_pci_dma_buf_move(vdev, false);
376	up_write(&vdev->memory_lock);	411	up_write(&vdev->memory_lock);
377	}	412	}
378		413
379	static int vfio_pci_core_pm_exit(struct vfio_device *device, u32 flags,	414	static int vfio_pci_core_pm_exit(struct vfio_pci_core_device *vdev, u32 flags,
380	void __user *arg, size_t argsz)	415	void __user *arg, size_t argsz)
381	{	416	{
382	struct vfio_pci_core_device *vdev =
383	container_of(device, struct vfio_pci_core_device, vdev);
384	int ret;	417	int ret;
385		418
386	ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET, 0);	419	ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_SET, 0);
@@ -695,15 +728,11 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev) · 308 unmodified lines
695	#endif	728	#endif
696	vfio_pci_core_disable(vdev);	729	vfio_pci_core_disable(vdev);
697		730
		731	vfio_pci_dma_buf_cleanup(vdev);
		732
698	mutex_lock(&vdev->igate);	733	mutex_lock(&vdev->igate);
699	if (vdev->err_trigger) {	734	vfio_pci_eventfd_replace_locked(vdev, &vdev->err_trigger, NULL);
700	eventfd_ctx_put(vdev->err_trigger);	735	vfio_pci_eventfd_replace_locked(vdev, &vdev->req_trigger, NULL);
701	vdev->err_trigger = NULL;
702	}
703	if (vdev->req_trigger) {
704	eventfd_ctx_put(vdev->req_trigger);
705	vdev->req_trigger = NULL;
706	}
707	mutex_unlock(&vdev->igate);	736	mutex_unlock(&vdev->igate);
708	}	737	}
709	EXPORT_SYMBOL_GPL(vfio_pci_core_close_device);	738	EXPORT_SYMBOL_GPL(vfio_pci_core_close_device);
@@ -996,42 +1025,36 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev, · 286 unmodified lines
996	return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;	1025	return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;
997	}	1026	}
998		1027
999	static int vfio_pci_ioctl_get_region_info(struct vfio_pci_core_device *vdev,	1028	int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
1000	struct vfio_region_info __user *arg)	1029	struct vfio_region_info *info,
		1030	struct vfio_info_cap *caps)
1001	{	1031	{
1002	unsigned long minsz = offsetofend(struct vfio_region_info, offset);	1032	struct vfio_pci_core_device *vdev =
		1033	container_of(core_vdev, struct vfio_pci_core_device, vdev);
1003	struct pci_dev *pdev = vdev->pdev;	1034	struct pci_dev *pdev = vdev->pdev;
1004	struct vfio_region_info info;
1005	struct vfio_info_cap caps = { .buf = NULL, .size = 0 };
1006	int i, ret;	1035	int i, ret;
1007		1036
1008	if (copy_from_user(&info, arg, minsz))	1037	switch (info->index) {
1009	return -EFAULT;
1010
1011	if (info.argsz < minsz)
1012	return -EINVAL;
1013
1014	switch (info.index) {
1015	case VFIO_PCI_CONFIG_REGION_INDEX:	1038	case VFIO_PCI_CONFIG_REGION_INDEX:
1016	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);	1039	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1017	info.size = pdev->cfg_size;	1040	info->size = pdev->cfg_size;
1018	info.flags = VFIO_REGION_INFO_FLAG_READ \|	1041	info->flags = VFIO_REGION_INFO_FLAG_READ \|
1019	VFIO_REGION_INFO_FLAG_WRITE;	1042	VFIO_REGION_INFO_FLAG_WRITE;
1020	break;	1043	break;
1021	case VFIO_PCI_BAR0_REGION_INDEX ... VFIO_PCI_BAR5_REGION_INDEX:	1044	case VFIO_PCI_BAR0_REGION_INDEX ... VFIO_PCI_BAR5_REGION_INDEX:
1022	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);	1045	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1023	info.size = pci_resource_len(pdev, info.index);	1046	info->size = pci_resource_len(pdev, info->index);
1024	if (!info.size) {	1047	if (!info->size) {
1025	info.flags = 0;	1048	info->flags = 0;
1026	break;	1049	break;
1027	}	1050	}
1028		1051
1029	info.flags = VFIO_REGION_INFO_FLAG_READ \|	1052	info->flags = VFIO_REGION_INFO_FLAG_READ \|
1030	VFIO_REGION_INFO_FLAG_WRITE;	1053	VFIO_REGION_INFO_FLAG_WRITE;
1031	if (vdev->bar_mmap_supported[info.index]) {	1054	if (vdev->bar_mmap_supported[info->index]) {
1032	info.flags \|= VFIO_REGION_INFO_FLAG_MMAP;	1055	info->flags \|= VFIO_REGION_INFO_FLAG_MMAP;
1033	if (info.index == vdev->msix_bar) {	1056	if (info->index == vdev->msix_bar) {
1034	ret = msix_mmappable_cap(vdev, &caps);	1057	ret = msix_mmappable_cap(vdev, caps);
1035	if (ret)	1058	if (ret)
1036	return ret;	1059	return ret;
1037	}	1060	}
@@ -1043,9 +1066,9 @@ static int vfio_pci_ioctl_get_region_info(struct vfio_pci_core_device *vdev, · 5 unmodified lines
1043	size_t size;	1066	size_t size;
1044	u16 cmd;	1067	u16 cmd;
1045		1068
1046	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);	1069	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1047	info.flags = 0;	1070	info->flags = 0;
1048	info.size = 0;	1071	info->size = 0;
1049		1072
1050	if (pci_resource_start(pdev, PCI_ROM_RESOURCE)) {	1073	if (pci_resource_start(pdev, PCI_ROM_RESOURCE)) {
1051	/*	1074	/*
@@ -1055,16 +1078,17 @@ static int vfio_pci_ioctl_get_region_info(struct vfio_pci_core_device *vdev, · 3 unmodified lines
1055	cmd = vfio_pci_memory_lock_and_enable(vdev);	1078	cmd = vfio_pci_memory_lock_and_enable(vdev);
1056	io = pci_map_rom(pdev, &size);	1079	io = pci_map_rom(pdev, &size);
1057	if (io) {	1080	if (io) {
1058	info.flags = VFIO_REGION_INFO_FLAG_READ;	1081	info->flags = VFIO_REGION_INFO_FLAG_READ;
1059	/* Report the BAR size, not the ROM size. */	1082	/* Report the BAR size, not the ROM size. */
1060	info.size = pci_resource_len(pdev, PCI_ROM_RESOURCE);	1083	info->size = pci_resource_len(pdev,
		1084	PCI_ROM_RESOURCE);
1061	pci_unmap_rom(pdev, io);	1085	pci_unmap_rom(pdev, io);
1062	}	1086	}
1063	vfio_pci_memory_unlock_and_restore(vdev, cmd);	1087	vfio_pci_memory_unlock_and_restore(vdev, cmd);
1064	} else if (pdev->rom && pdev->romlen) {	1088	} else if (pdev->rom && pdev->romlen) {
1065	info.flags = VFIO_REGION_INFO_FLAG_READ;	1089	info->flags = VFIO_REGION_INFO_FLAG_READ;
1066	/* Report BAR size as power of two. */	1090	/* Report BAR size as power of two. */
1067	info.size = roundup_pow_of_two(pdev->romlen);	1091	info->size = roundup_pow_of_two(pdev->romlen);
1068	}	1092	}
1069		1093
1070	break;	1094	break;
@@ -1073,10 +1097,10 @@ static int vfio_pci_ioctl_get_region_info(struct vfio_pci_core_device *vdev, · 2 unmodified lines
1073	if (!vdev->has_vga)	1097	if (!vdev->has_vga)
1074	return -EINVAL;	1098	return -EINVAL;
1075		1099
1076	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);	1100	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1077	info.size = 0xc0000;	1101	info->size = 0xc0000;
1078	info.flags = VFIO_REGION_INFO_FLAG_READ \|	1102	info->flags = VFIO_REGION_INFO_FLAG_READ \|
1079	VFIO_REGION_INFO_FLAG_WRITE;	1103	VFIO_REGION_INFO_FLAG_WRITE;
1080		1104
1081	break;	1105	break;
1082	default: {	1106	default: {
@@ -1085,53 +1109,36 @@ static int vfio_pci_ioctl_get_region_info(struct vfio_pci_core_device *vdev, · 2 unmodified lines
1085	.header.version = 1	1109	.header.version = 1
1086	};	1110	};
1087		1111
1088	if (info.index >= VFIO_PCI_NUM_REGIONS + vdev->num_regions)	1112	if (info->index >= VFIO_PCI_NUM_REGIONS + vdev->num_regions)
1089	return -EINVAL;	1113	return -EINVAL;
1090	info.index = array_index_nospec(	1114	info->index = array_index_nospec(
1091	info.index, VFIO_PCI_NUM_REGIONS + vdev->num_regions);	1115	info->index, VFIO_PCI_NUM_REGIONS + vdev->num_regions);
1092		1116
1093	i = info.index - VFIO_PCI_NUM_REGIONS;	1117	i = info->index - VFIO_PCI_NUM_REGIONS;
1094		1118
1095	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);	1119	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
1096	info.size = vdev->region[i].size;	1120	info->size = vdev->region[i].size;
1097	info.flags = vdev->region[i].flags;	1121	info->flags = vdev->region[i].flags;
1098		1122
1099	cap_type.type = vdev->region[i].type;	1123	cap_type.type = vdev->region[i].type;
1100	cap_type.subtype = vdev->region[i].subtype;	1124	cap_type.subtype = vdev->region[i].subtype;
1101		1125
1102	ret = vfio_info_add_capability(&caps, &cap_type.header,	1126	ret = vfio_info_add_capability(caps, &cap_type.header,
1103	sizeof(cap_type));	1127	sizeof(cap_type));
1104	if (ret)	1128	if (ret)
1105	return ret;	1129	return ret;
1106		1130
1107	if (vdev->region[i].ops->add_capability) {	1131	if (vdev->region[i].ops->add_capability) {
1108	ret = vdev->region[i].ops->add_capability(	1132	ret = vdev->region[i].ops->add_capability(
1109	vdev, &vdev->region[i], &caps);	1133	vdev, &vdev->region[i], caps);
1110	if (ret)	1134	if (ret)
1111	return ret;	1135	return ret;
1112	}	1136	}
1113	}	1137	}
1114	}	1138	}
1115		1139	return 0;
1116	if (caps.size) {
1117	info.flags \|= VFIO_REGION_INFO_FLAG_CAPS;
1118	if (info.argsz < sizeof(info) + caps.size) {
1119	info.argsz = sizeof(info) + caps.size;
1120	info.cap_offset = 0;
1121	} else {
1122	vfio_info_cap_shift(&caps, sizeof(info));
1123	if (copy_to_user(arg + 1, caps.buf, caps.size)) {
1124	kfree(caps.buf);
1125	return -EFAULT;
1126	}
1127	info.cap_offset = sizeof(*arg);
1128	}
1129
1130	kfree(caps.buf);
1131	}
1132
1133	return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;
1134	}	1140	}
		1141	EXPORT_SYMBOL_GPL(vfio_pci_ioctl_get_region_info);
1135		1142
1136	static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,	1143	static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
1137	struct vfio_irq_info __user *arg)	1144	struct vfio_irq_info __user *arg)
@@ -1227,7 +1234,10 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, · 89 unmodified lines
1227	*/	1234	*/
1228	vfio_pci_set_power_state(vdev, PCI_D0);	1235	vfio_pci_set_power_state(vdev, PCI_D0);
1229		1236
		1237	vfio_pci_dma_buf_move(vdev, true);
1230	ret = pci_try_reset_function(vdev->pdev);	1238	ret = pci_try_reset_function(vdev->pdev);
		1239	if (__vfio_pci_memory_enabled(vdev))
		1240	vfio_pci_dma_buf_move(vdev, false);
1231	up_write(&vdev->memory_lock);	1241	up_write(&vdev->memory_lock);
1232		1242
1233	return ret;	1243	return ret;
@@ -1457,8 +1467,6 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, · 223 unmodified lines
1457	return vfio_pci_ioctl_get_irq_info(vdev, uarg);	1467	return vfio_pci_ioctl_get_irq_info(vdev, uarg);
1458	case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO:	1468	case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO:
1459	return vfio_pci_ioctl_get_pci_hot_reset_info(vdev, uarg);	1469	return vfio_pci_ioctl_get_pci_hot_reset_info(vdev, uarg);
1460	case VFIO_DEVICE_GET_REGION_INFO:
1461	return vfio_pci_ioctl_get_region_info(vdev, uarg);
1462	case VFIO_DEVICE_IOEVENTFD:	1470	case VFIO_DEVICE_IOEVENTFD:
1463	return vfio_pci_ioctl_ioeventfd(vdev, uarg);	1471	return vfio_pci_ioctl_ioeventfd(vdev, uarg);
1464	case VFIO_DEVICE_PCI_HOT_RESET:	1472	case VFIO_DEVICE_PCI_HOT_RESET:
@@ -1473,11 +1481,10 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, · 8 unmodified lines
1473	}	1481	}
1474	EXPORT_SYMBOL_GPL(vfio_pci_core_ioctl);	1482	EXPORT_SYMBOL_GPL(vfio_pci_core_ioctl);
1475		1483
1476	static int vfio_pci_core_feature_token(struct vfio_device *device, u32 flags,	1484	static int vfio_pci_core_feature_token(struct vfio_pci_core_device *vdev,
1477	uuid_t __user *arg, size_t argsz)	1485	u32 flags, uuid_t __user *arg,
		1486	size_t argsz)
1478	{	1487	{
1479	struct vfio_pci_core_device *vdev =
1480	container_of(device, struct vfio_pci_core_device, vdev);
1481	uuid_t uuid;	1488	uuid_t uuid;
1482	int ret;	1489	int ret;
1483		1490
@@ -1504,16 +1511,21 @@ static int vfio_pci_core_feature_token(struct vfio_device *device, u32 flags, · 20 unmodified lines
1504	int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,	1511	int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
1505	void __user *arg, size_t argsz)	1512	void __user *arg, size_t argsz)
1506	{	1513	{
		1514	struct vfio_pci_core_device *vdev =
		1515	container_of(device, struct vfio_pci_core_device, vdev);
		1516
1507	switch (flags & VFIO_DEVICE_FEATURE_MASK) {	1517	switch (flags & VFIO_DEVICE_FEATURE_MASK) {
1508	case VFIO_DEVICE_FEATURE_LOW_POWER_ENTRY:	1518	case VFIO_DEVICE_FEATURE_LOW_POWER_ENTRY:
1509	return vfio_pci_core_pm_entry(device, flags, arg, argsz);	1519	return vfio_pci_core_pm_entry(vdev, flags, arg, argsz);
1510	case VFIO_DEVICE_FEATURE_LOW_POWER_ENTRY_WITH_WAKEUP:	1520	case VFIO_DEVICE_FEATURE_LOW_POWER_ENTRY_WITH_WAKEUP:
1511	return vfio_pci_core_pm_entry_with_wakeup(device, flags,	1521	return vfio_pci_core_pm_entry_with_wakeup(vdev, flags,
1512	arg, argsz);	1522	arg, argsz);
1513	case VFIO_DEVICE_FEATURE_LOW_POWER_EXIT:	1523	case VFIO_DEVICE_FEATURE_LOW_POWER_EXIT:
1514	return vfio_pci_core_pm_exit(device, flags, arg, argsz);	1524	return vfio_pci_core_pm_exit(vdev, flags, arg, argsz);
1515	case VFIO_DEVICE_FEATURE_PCI_VF_TOKEN:	1525	case VFIO_DEVICE_FEATURE_PCI_VF_TOKEN:
1516	return vfio_pci_core_feature_token(device, flags, arg, argsz);	1526	return vfio_pci_core_feature_token(vdev, flags, arg, argsz);
		1527	case VFIO_DEVICE_FEATURE_DMA_BUF:
		1528	return vfio_pci_core_feature_dma_buf(vdev, flags, arg, argsz);
1517	default:	1529	default:
1518	return -ENOTTY;	1530	return -ENOTTY;
1519	}	1531	}
@@ -1640,49 +1652,49 @@ static unsigned long vma_to_pfn(struct vm_area_struct *vma) · 120 unmodified lines
1640	return (pci_resource_start(vdev->pdev, index) >> PAGE_SHIFT) + pgoff;	1652	return (pci_resource_start(vdev->pdev, index) >> PAGE_SHIFT) + pgoff;
1641	}	1653	}
1642		1654
1643	static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,	1655	vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
1644	unsigned int order)	1656	struct vm_fault *vmf,
		1657	unsigned long pfn,
		1658	unsigned int order)
1645	{	1659	{
1646	struct vm_area_struct *vma = vmf->vma;	1660	lockdep_assert_held_read(&vdev->memory_lock);
1647	struct vfio_pci_core_device *vdev = vma->vm_private_data;
1648	unsigned long addr = vmf->address & ~((PAGE_SIZE << order) - 1);
1649	unsigned long pgoff = (addr - vma->vm_start) >> PAGE_SHIFT;
1650	unsigned long pfn = vma_to_pfn(vma) + pgoff;
1651	vm_fault_t ret = VM_FAULT_SIGBUS;
1652
1653	if (order && (addr < vma->vm_start \|\|
1654	addr + (PAGE_SIZE << order) > vma->vm_end \|\|
1655	pfn & ((1 << order) - 1))) {
1656	ret = VM_FAULT_FALLBACK;
1657	goto out;
1658	}
1659
1660	down_read(&vdev->memory_lock);
1661		1661
1662	if (vdev->pm_runtime_engaged \|\| !__vfio_pci_memory_enabled(vdev))	1662	if (vdev->pm_runtime_engaged \|\| !__vfio_pci_memory_enabled(vdev))
1663	goto out_unlock;	1663	return VM_FAULT_SIGBUS;
1664		1664
1665	switch (order) {	1665	switch (order) {
1666	case 0:	1666	case 0:
1667	ret = vmf_insert_pfn(vma, vmf->address, pfn);	1667	return vmf_insert_pfn(vmf->vma, vmf->address, pfn);
1668	break;
1669	#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP	1668	#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP
1670	case PMD_ORDER:	1669	case PMD_ORDER:
1671	ret = vmf_insert_pfn_pmd(vmf, pfn, false);	1670	return vmf_insert_pfn_pmd(vmf, pfn, false);
1672	break;
1673	#endif	1671	#endif
1674	#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP	1672	#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP
1675	case PUD_ORDER:	1673	case PUD_ORDER:
1676	ret = vmf_insert_pfn_pud(vmf, pfn, false);	1674	return vmf_insert_pfn_pud(vmf, pfn, false);
1677	break;	1675	break;
1678	#endif	1676	#endif
1679	default:	1677	default:
1680	ret = VM_FAULT_FALLBACK;	1678	return VM_FAULT_FALLBACK;
		1679	}
		1680	}
		1681	EXPORT_SYMBOL_GPL(vfio_pci_vmf_insert_pfn);
		1682
		1683	static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
		1684	unsigned int order)
		1685	{
		1686	struct vm_area_struct *vma = vmf->vma;
		1687	struct vfio_pci_core_device *vdev = vma->vm_private_data;
		1688	unsigned long addr = vmf->address & ~((PAGE_SIZE << order) - 1);
		1689	unsigned long pgoff = (addr - vma->vm_start) >> PAGE_SHIFT;
		1690	unsigned long pfn = vma_to_pfn(vma) + pgoff;
		1691	vm_fault_t ret = VM_FAULT_FALLBACK;
		1692
		1693	if (is_aligned_for_order(vma, addr, pfn, order)) {
		1694	scoped_guard(rwsem_read, &vdev->memory_lock)
		1695	ret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order);
1681	}	1696	}
1682		1697
1683	out_unlock:
1684	up_read(&vdev->memory_lock);
1685	out:
1686	dev_dbg_ratelimited(&vdev->pdev->dev,	1698	dev_dbg_ratelimited(&vdev->pdev->dev,
1687	"%s(,order = %d) BAR %ld page offset 0x%lx: 0x%x\n",	1699	"%s(,order = %d) BAR %ld page offset 0x%lx: 0x%x\n",
1688	__func__, order,	1700	__func__, order,
@@ -1749,18 +1761,9 @@ int vfio_pci_core_mmap(struct vfio_device core_vdev, struct vm_area_struct vma · 60 unmodified lines
1749	* Even though we don't make use of the barmap for the mmap,	1761	* Even though we don't make use of the barmap for the mmap,
1750	* we need to request the region and the barmap tracks that.	1762	* we need to request the region and the barmap tracks that.
1751	*/	1763	*/
1752	if (!vdev->barmap[index]) {	1764	ret = vfio_pci_core_setup_barmap(vdev, index);
1753	ret = pci_request_selected_regions(pdev,	1765	if (ret)
1754	1 << index, "vfio-pci");	1766	return ret;
1755	if (ret)
1756	return ret;
1757
1758	vdev->barmap[index] = pci_iomap(pdev, index, 0);
1759	if (!vdev->barmap[index]) {
1760	pci_release_selected_regions(pdev, 1 << index);
1761	return -ENOMEM;
1762	}
1763	}
1764		1767
1765	vma->vm_private_data = vdev;	1768	vma->vm_private_data = vdev;
1766	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);	1769	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
@@ -1800,21 +1803,21 @@ void vfio_pci_core_request(struct vfio_device *core_vdev, unsigned int count) · 33 unmodified lines
1800	struct vfio_pci_core_device *vdev =	1803	struct vfio_pci_core_device *vdev =
1801	container_of(core_vdev, struct vfio_pci_core_device, vdev);	1804	container_of(core_vdev, struct vfio_pci_core_device, vdev);
1802	struct pci_dev *pdev = vdev->pdev;	1805	struct pci_dev *pdev = vdev->pdev;
		1806	struct vfio_pci_eventfd *eventfd;
1803		1807
1804	mutex_lock(&vdev->igate);	1808	rcu_read_lock();
1805		1809	eventfd = rcu_dereference(vdev->req_trigger);
1806	if (vdev->req_trigger) {	1810	if (eventfd) {
1807	if (!(count % 10))	1811	if (!(count % 10))
1808	pci_notice_ratelimited(pdev,	1812	pci_notice_ratelimited(pdev,
1809	"Relaying device request to user (#%u)\n",	1813	"Relaying device request to user (#%u)\n",
1810	count);	1814	count);
1811	eventfd_signal(vdev->req_trigger);	1815	eventfd_signal(eventfd->ctx);
1812	} else if (count == 0) {	1816	} else if (count == 0) {
1813	pci_warn(pdev,	1817	pci_warn(pdev,
1814	"No device request channel registered, blocked until released by user\n");	1818	"No device request channel registered, blocked until released by user\n");
1815	}	1819	}
1816		1820	rcu_read_unlock();
1817	mutex_unlock(&vdev->igate);
1818	}	1821	}
1819	EXPORT_SYMBOL_GPL(vfio_pci_core_request);	1822	EXPORT_SYMBOL_GPL(vfio_pci_core_request);
1820		1823
@@ -2085,6 +2088,7 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev) · 264 unmodified lines
2085	{	2088	{
2086	struct vfio_pci_core_device *vdev =	2089	struct vfio_pci_core_device *vdev =
2087	container_of(core_vdev, struct vfio_pci_core_device, vdev);	2090	container_of(core_vdev, struct vfio_pci_core_device, vdev);
		2091	int ret;
2088		2092
2089	vdev->pdev = to_pci_dev(core_vdev->dev);	2093	vdev->pdev = to_pci_dev(core_vdev->dev);
2090	vdev->irq_type = VFIO_PCI_NUM_IRQS;	2094	vdev->irq_type = VFIO_PCI_NUM_IRQS;
@@ -2094,6 +2098,10 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev) · 3 unmodified lines
2094	INIT_LIST_HEAD(&vdev->dummy_resources_list);	2098	INIT_LIST_HEAD(&vdev->dummy_resources_list);
2095	INIT_LIST_HEAD(&vdev->ioeventfds_list);	2099	INIT_LIST_HEAD(&vdev->ioeventfds_list);
2096	INIT_LIST_HEAD(&vdev->sriov_pfs_item);	2100	INIT_LIST_HEAD(&vdev->sriov_pfs_item);
		2101	ret = pcim_p2pdma_init(vdev->pdev);
		2102	if (ret && ret != -EOPNOTSUPP)
		2103	return ret;
		2104	INIT_LIST_HEAD(&vdev->dmabufs);
2097	init_rwsem(&vdev->memory_lock);	2105	init_rwsem(&vdev->memory_lock);
2098	xa_init(&vdev->ctx);	2106	xa_init(&vdev->ctx);
2099		2107
@@ -2227,13 +2235,13 @@ pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev, · 127 unmodified lines
2227	pci_channel_state_t state)	2235	pci_channel_state_t state)
2228	{	2236	{
2229	struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);	2237	struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
		2238	struct vfio_pci_eventfd *eventfd;
2230		2239
2231	mutex_lock(&vdev->igate);	2240	rcu_read_lock();
2232		2241	eventfd = rcu_dereference(vdev->err_trigger);
2233	if (vdev->err_trigger)	2242	if (eventfd)
2234	eventfd_signal(vdev->err_trigger);	2243	eventfd_signal(eventfd->ctx);
2235		2244	rcu_read_unlock();
2236	mutex_unlock(&vdev->igate);
2237		2245
2238	return PCI_ERS_RESULT_CAN_RECOVER;	2246	return PCI_ERS_RESULT_CAN_RECOVER;
2239	}	2247	}
@@ -2458,6 +2466,7 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · 218 unmodified lines
2458	break;	2466	break;
2459	}	2467	}
2460		2468
		2469	vfio_pci_dma_buf_move(vdev, true);
2461	vfio_pci_zap_bars(vdev);	2470	vfio_pci_zap_bars(vdev);
2462	}	2471	}
2463		2472
@@ -2486,8 +2495,11 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · 22 unmodified lines
2486		2495
2487	err_undo:	2496	err_undo:
2488	list_for_each_entry_from_reverse(vdev, &dev_set->device_list,	2497	list_for_each_entry_from_reverse(vdev, &dev_set->device_list,
2489	vdev.dev_set_list)	2498	vdev.dev_set_list) {
		2499	if (vdev->vdev.open_count && __vfio_pci_memory_enabled(vdev))
		2500	vfio_pci_dma_buf_move(vdev, false);
2490	up_write(&vdev->memory_lock);	2501	up_write(&vdev->memory_lock);
		2502	}
2491		2503
2492	list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list)	2504	list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list)
2493	pm_runtime_put(&vdev->pdev->dev);	2505	pm_runtime_put(&vdev->pdev->dev);

drivers/vfio/pci/vfio_pci_dmabuf.c+316−0

@@ -0,0 +1,316 @@
	1	// SPDX-License-Identifier: GPL-2.0-only
	2	/* Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES.
	3	*/
	4	#include <linux/dma-buf-mapping.h>
	5	#include <linux/pci-p2pdma.h>
	6	#include <linux/dma-resv.h>
	7
	8	#include "vfio_pci_priv.h"
	9
	10	MODULE_IMPORT_NS("DMA_BUF");
	11
	12	struct vfio_pci_dma_buf {
	13	struct dma_buf *dmabuf;
	14	struct vfio_pci_core_device *vdev;
	15	struct list_head dmabufs_elm;
	16	size_t size;
	17	struct dma_buf_phys_vec *phys_vec;
	18	struct p2pdma_provider *provider;
	19	u32 nr_ranges;
	20	u8 revoked : 1;
	21	};
	22
	23	static int vfio_pci_dma_buf_attach(struct dma_buf *dmabuf,
	24	struct dma_buf_attachment *attachment)
	25	{
	26	struct vfio_pci_dma_buf *priv = dmabuf->priv;
	27
	28	if (!attachment->peer2peer)
	29	return -EOPNOTSUPP;
	30
	31	if (priv->revoked)
	32	return -ENODEV;
	33
	34	return 0;
	35	}
	36
	37	static struct sg_table *
	38	vfio_pci_dma_buf_map(struct dma_buf_attachment *attachment,
	39	enum dma_data_direction dir)
	40	{
	41	struct vfio_pci_dma_buf *priv = attachment->dmabuf->priv;
	42
	43	dma_resv_assert_held(priv->dmabuf->resv);
	44
	45	if (priv->revoked)
	46	return ERR_PTR(-ENODEV);
	47
	48	return dma_buf_phys_vec_to_sgt(attachment, priv->provider,
	49	priv->phys_vec, priv->nr_ranges,
	50	priv->size, dir);
	51	}
	52
	53	static void vfio_pci_dma_buf_unmap(struct dma_buf_attachment *attachment,
	54	struct sg_table *sgt,
	55	enum dma_data_direction dir)
	56	{
	57	dma_buf_free_sgt(attachment, sgt, dir);
	58	}
	59
	60	static void vfio_pci_dma_buf_release(struct dma_buf *dmabuf)
	61	{
	62	struct vfio_pci_dma_buf *priv = dmabuf->priv;
	63
	64	/*
	65	* Either this or vfio_pci_dma_buf_cleanup() will remove from the list.
	66	* The refcount prevents both.
	67	*/
	68	if (priv->vdev) {
	69	down_write(&priv->vdev->memory_lock);
	70	list_del_init(&priv->dmabufs_elm);
	71	up_write(&priv->vdev->memory_lock);
	72	vfio_device_put_registration(&priv->vdev->vdev);
	73	}
	74	kfree(priv->phys_vec);
	75	kfree(priv);
	76	}
	77
	78	static const struct dma_buf_ops vfio_pci_dmabuf_ops = {
	79	.attach = vfio_pci_dma_buf_attach,
	80	.map_dma_buf = vfio_pci_dma_buf_map,
	81	.unmap_dma_buf = vfio_pci_dma_buf_unmap,
	82	.release = vfio_pci_dma_buf_release,
	83	};
	84
	85	int vfio_pci_core_fill_phys_vec(struct dma_buf_phys_vec *phys_vec,
	86	struct vfio_region_dma_range *dma_ranges,
	87	size_t nr_ranges, phys_addr_t start,
	88	phys_addr_t len)
	89	{
	90	phys_addr_t max_addr;
	91	unsigned int i;
	92
	93	max_addr = start + len;
	94	for (i = 0; i < nr_ranges; i++) {
	95	phys_addr_t end;
	96
	97	if (!dma_ranges[i].length)
	98	return -EINVAL;
	99
	100	if (check_add_overflow(start, dma_ranges[i].offset,
	101	&phys_vec[i].paddr) \|\|
	102	check_add_overflow(phys_vec[i].paddr,
	103	dma_ranges[i].length, &end))
	104	return -EOVERFLOW;
	105	if (end > max_addr)
	106	return -EINVAL;
	107
	108	phys_vec[i].len = dma_ranges[i].length;
	109	}
	110	return 0;
	111	}
	112	EXPORT_SYMBOL_GPL(vfio_pci_core_fill_phys_vec);
	113
	114	int vfio_pci_core_get_dmabuf_phys(struct vfio_pci_core_device *vdev,
	115	struct p2pdma_provider **provider,
	116	unsigned int region_index,
	117	struct dma_buf_phys_vec *phys_vec,
	118	struct vfio_region_dma_range *dma_ranges,
	119	size_t nr_ranges)
	120	{
	121	struct pci_dev *pdev = vdev->pdev;
	122
	123	*provider = pcim_p2pdma_provider(pdev, region_index);
	124	if (!*provider)
	125	return -EINVAL;
	126
	127	return vfio_pci_core_fill_phys_vec(
	128	phys_vec, dma_ranges, nr_ranges,
	129	pci_resource_start(pdev, region_index),
	130	pci_resource_len(pdev, region_index));
	131	}
	132	EXPORT_SYMBOL_GPL(vfio_pci_core_get_dmabuf_phys);
	133
	134	static int validate_dmabuf_input(struct vfio_device_feature_dma_buf *dma_buf,
	135	struct vfio_region_dma_range *dma_ranges,
	136	size_t *lengthp)
	137	{
	138	size_t length = 0;
	139	u32 i;
	140
	141	for (i = 0; i < dma_buf->nr_ranges; i++) {
	142	u64 offset = dma_ranges[i].offset;
	143	u64 len = dma_ranges[i].length;
	144
	145	if (!len \|\| !PAGE_ALIGNED(offset) \|\| !PAGE_ALIGNED(len))
	146	return -EINVAL;
	147
	148	if (check_add_overflow(length, len, &length))
	149	return -EINVAL;
	150	}
	151
	152	/*
	153	* dma_iova_try_alloc() will WARN on if userspace proposes a size that
	154	* is too big, eg with lots of ranges.
	155	*/
	156	if ((u64)(length) & DMA_IOVA_USE_SWIOTLB)
	157	return -EINVAL;
	158
	159	*lengthp = length;
	160	return 0;
	161	}
	162
	163	int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
	164	struct vfio_device_feature_dma_buf __user *arg,
	165	size_t argsz)
	166	{
	167	struct vfio_device_feature_dma_buf get_dma_buf = {};
	168	struct vfio_region_dma_range *dma_ranges;
	169	DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
	170	struct vfio_pci_dma_buf *priv;
	171	size_t length;
	172	int ret;
	173
	174	if (!vdev->pci_ops \|\| !vdev->pci_ops->get_dmabuf_phys)
	175	return -EOPNOTSUPP;
	176
	177	ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_GET,
	178	sizeof(get_dma_buf));
	179	if (ret != 1)
	180	return ret;
	181
	182	if (copy_from_user(&get_dma_buf, arg, sizeof(get_dma_buf)))
	183	return -EFAULT;
	184
	185	if (!get_dma_buf.nr_ranges \|\| get_dma_buf.flags)
	186	return -EINVAL;
	187
	188	/*
	189	* For PCI the region_index is the BAR number like everything else.
	190	*/
	191	if (get_dma_buf.region_index >= VFIO_PCI_ROM_REGION_INDEX)
	192	return -ENODEV;
	193
	194	dma_ranges = memdup_array_user(&arg->dma_ranges, get_dma_buf.nr_ranges,
	195	sizeof(*dma_ranges));
	196	if (IS_ERR(dma_ranges))
	197	return PTR_ERR(dma_ranges);
	198
	199	ret = validate_dmabuf_input(&get_dma_buf, dma_ranges, &length);
	200	if (ret)
	201	goto err_free_ranges;
	202
	203	priv = kzalloc(sizeof(*priv), GFP_KERNEL);
	204	if (!priv) {
	205	ret = -ENOMEM;
	206	goto err_free_ranges;
	207	}
	208	priv->phys_vec = kcalloc(get_dma_buf.nr_ranges, sizeof(*priv->phys_vec),
	209	GFP_KERNEL);
	210	if (!priv->phys_vec) {
	211	ret = -ENOMEM;
	212	goto err_free_priv;
	213	}
	214
	215	priv->vdev = vdev;
	216	priv->nr_ranges = get_dma_buf.nr_ranges;
	217	priv->size = length;
	218	ret = vdev->pci_ops->get_dmabuf_phys(vdev, &priv->provider,
	219	get_dma_buf.region_index,
	220	priv->phys_vec, dma_ranges,
	221	priv->nr_ranges);
	222	if (ret)
	223	goto err_free_phys;
	224
	225	kfree(dma_ranges);
	226	dma_ranges = NULL;
	227
	228	if (!vfio_device_try_get_registration(&vdev->vdev)) {
	229	ret = -ENODEV;
	230	goto err_free_phys;
	231	}
	232
	233	exp_info.ops = &vfio_pci_dmabuf_ops;
	234	exp_info.size = priv->size;
	235	exp_info.flags = get_dma_buf.open_flags;
	236	exp_info.priv = priv;
	237
	238	priv->dmabuf = dma_buf_export(&exp_info);
	239	if (IS_ERR(priv->dmabuf)) {
	240	ret = PTR_ERR(priv->dmabuf);
	241	goto err_dev_put;
	242	}
	243
	244	/* dma_buf_put() now frees priv */
	245	INIT_LIST_HEAD(&priv->dmabufs_elm);
	246	down_write(&vdev->memory_lock);
	247	dma_resv_lock(priv->dmabuf->resv, NULL);
	248	priv->revoked = !__vfio_pci_memory_enabled(vdev);
	249	list_add_tail(&priv->dmabufs_elm, &vdev->dmabufs);
	250	dma_resv_unlock(priv->dmabuf->resv);
	251	up_write(&vdev->memory_lock);
	252
	253	/*
	254	* dma_buf_fd() consumes the reference, when the file closes the dmabuf
	255	* will be released.
	256	*/
	257	ret = dma_buf_fd(priv->dmabuf, get_dma_buf.open_flags);
	258	if (ret < 0)
	259	goto err_dma_buf;
	260	return ret;
	261
	262	err_dma_buf:
	263	dma_buf_put(priv->dmabuf);
	264	err_dev_put:
	265	vfio_device_put_registration(&vdev->vdev);
	266	err_free_phys:
	267	kfree(priv->phys_vec);
	268	err_free_priv:
	269	kfree(priv);
	270	err_free_ranges:
	271	kfree(dma_ranges);
	272	return ret;
	273	}
	274
	275	void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, bool revoked)
	276	{
	277	struct vfio_pci_dma_buf *priv;
	278	struct vfio_pci_dma_buf *tmp;
	279
	280	lockdep_assert_held_write(&vdev->memory_lock);
	281
	282	list_for_each_entry_safe(priv, tmp, &vdev->dmabufs, dmabufs_elm) {
	283	if (!get_file_active(&priv->dmabuf->file))
	284	continue;
	285
	286	if (priv->revoked != revoked) {
	287	dma_resv_lock(priv->dmabuf->resv, NULL);
	288	priv->revoked = revoked;
	289	dma_buf_move_notify(priv->dmabuf);
	290	dma_resv_unlock(priv->dmabuf->resv);
	291	}
	292	fput(priv->dmabuf->file);
	293	}
	294	}
	295
	296	void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev)
	297	{
	298	struct vfio_pci_dma_buf *priv;
	299	struct vfio_pci_dma_buf *tmp;
	300
	301	down_write(&vdev->memory_lock);
	302	list_for_each_entry_safe(priv, tmp, &vdev->dmabufs, dmabufs_elm) {
	303	if (!get_file_active(&priv->dmabuf->file))
	304	continue;
	305
	306	dma_resv_lock(priv->dmabuf->resv, NULL);
	307	list_del_init(&priv->dmabufs_elm);
	308	priv->vdev = NULL;
	309	priv->revoked = true;
	310	dma_buf_move_notify(priv->dmabuf);
	311	dma_resv_unlock(priv->dmabuf->resv);
	312	vfio_device_put_registration(&vdev->vdev);
	313	fput(priv->dmabuf->file);
	314	}
	315	up_write(&vdev->memory_lock);
	316	}

drivers/vfio/pci/vfio_pci_intrs.c+33−19

@@ -731,21 +731,27 @@ static int vfio_pci_set_msi_trigger(struct vfio_pci_core_device *vdev, · 730 unmodified lines
731	return 0;	731	return 0;
732	}	732	}
733		733
734	static int vfio_pci_set_ctx_trigger_single(struct eventfd_ctx **ctx,	734	static int vfio_pci_set_ctx_trigger_single(struct vfio_pci_core_device *vdev,
		735	struct vfio_pci_eventfd __rcu **peventfd,
735	unsigned int count, uint32_t flags,	736	unsigned int count, uint32_t flags,
736	void *data)	737	void *data)
737	{	738	{
738	/* DATA_NONE/DATA_BOOL enables loopback testing */	739	/* DATA_NONE/DATA_BOOL enables loopback testing */
739	if (flags & VFIO_IRQ_SET_DATA_NONE) {	740	if (flags & VFIO_IRQ_SET_DATA_NONE) {
740	if (*ctx) {	741	struct vfio_pci_eventfd *eventfd;
741	if (count) {	742
742	eventfd_signal(*ctx);	743	eventfd = rcu_dereference_protected(*peventfd,
743	} else {	744	lockdep_is_held(&vdev->igate));
744	eventfd_ctx_put(*ctx);	745
745	*ctx = NULL;	746	if (!eventfd)
746	}	747	return -EINVAL;
		748
		749	if (count) {
		750	eventfd_signal(eventfd->ctx);
747	return 0;	751	return 0;
748	}	752	}
		753
		754	return vfio_pci_eventfd_replace_locked(vdev, peventfd, NULL);
749	} else if (flags & VFIO_IRQ_SET_DATA_BOOL) {	755	} else if (flags & VFIO_IRQ_SET_DATA_BOOL) {
750	uint8_t trigger;	756	uint8_t trigger;
751		757
@@ -753,8 +759,15 @@ static int vfio_pci_set_ctx_trigger_single(struct eventfd_ctx **ctx, · 1 unmodified line
753	return -EINVAL;	759	return -EINVAL;
754		760
755	trigger = (uint8_t )data;	761	trigger = (uint8_t )data;
756	if (trigger && *ctx)	762
757	eventfd_signal(*ctx);	763	if (trigger) {
		764	struct vfio_pci_eventfd *eventfd =
		765	rcu_dereference_protected(*peventfd,
		766	lockdep_is_held(&vdev->igate));
		767
		768	if (eventfd)
		769	eventfd_signal(eventfd->ctx);
		770	}
758		771
759	return 0;	772	return 0;
760	} else if (flags & VFIO_IRQ_SET_DATA_EVENTFD) {	773	} else if (flags & VFIO_IRQ_SET_DATA_EVENTFD) {
@@ -765,22 +778,23 @@ static int vfio_pci_set_ctx_trigger_single(struct eventfd_ctx **ctx, · 4 unmodified lines
765		778
766	fd = (int32_t )data;	779	fd = (int32_t )data;
767	if (fd == -1) {	780	if (fd == -1) {
768	if (*ctx)	781	return vfio_pci_eventfd_replace_locked(vdev,
769	eventfd_ctx_put(*ctx);	782	peventfd, NULL);
770	*ctx = NULL;
771	} else if (fd >= 0) {	783	} else if (fd >= 0) {
772	struct eventfd_ctx *efdctx;	784	struct eventfd_ctx *efdctx;
		785	int ret;
773		786
774	efdctx = eventfd_ctx_fdget(fd);	787	efdctx = eventfd_ctx_fdget(fd);
775	if (IS_ERR(efdctx))	788	if (IS_ERR(efdctx))
776	return PTR_ERR(efdctx);	789	return PTR_ERR(efdctx);
777		790
778	if (*ctx)	791	ret = vfio_pci_eventfd_replace_locked(vdev,
779	eventfd_ctx_put(*ctx);	792	peventfd, efdctx);
		793	if (ret)
		794	eventfd_ctx_put(efdctx);
780		795
781	*ctx = efdctx;	796	return ret;
782	}	797	}
783	return 0;
784	}	798	}
785		799
786	return -EINVAL;	800	return -EINVAL;
@@ -793,7 +807,7 @@ static int vfio_pci_set_err_trigger(struct vfio_pci_core_device *vdev, · 6 unmodified lines
793	if (index != VFIO_PCI_ERR_IRQ_INDEX \|\| start != 0 \|\| count > 1)	807	if (index != VFIO_PCI_ERR_IRQ_INDEX \|\| start != 0 \|\| count > 1)
794	return -EINVAL;	808	return -EINVAL;
795		809
796	return vfio_pci_set_ctx_trigger_single(&vdev->err_trigger,	810	return vfio_pci_set_ctx_trigger_single(vdev, &vdev->err_trigger,
797	count, flags, data);	811	count, flags, data);
798	}	812	}
799		813
@@ -804,7 +818,7 @@ static int vfio_pci_set_req_trigger(struct vfio_pci_core_device *vdev, · 4 unmodified lines
804	if (index != VFIO_PCI_REQ_IRQ_INDEX \|\| start != 0 \|\| count > 1)	818	if (index != VFIO_PCI_REQ_IRQ_INDEX \|\| start != 0 \|\| count > 1)
805	return -EINVAL;	819	return -EINVAL;
806		820
807	return vfio_pci_set_ctx_trigger_single(&vdev->req_trigger,	821	return vfio_pci_set_ctx_trigger_single(vdev, &vdev->req_trigger,
808	count, flags, data);	822	count, flags, data);
809	}	823	}
810		824

drivers/vfio/pci/vfio_pci_priv.h+27−1

@@ -26,6 +26,10 @@ struct vfio_pci_ioeventfd { · 25 unmodified lines
26	bool vfio_pci_intx_mask(struct vfio_pci_core_device *vdev);	26	bool vfio_pci_intx_mask(struct vfio_pci_core_device *vdev);
27	void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev);	27	void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev);
28		28
		29	int vfio_pci_eventfd_replace_locked(struct vfio_pci_core_device *vdev,
		30	struct vfio_pci_eventfd __rcu **peventfd,
		31	struct eventfd_ctx *ctx);
		32
29	int vfio_pci_set_irqs_ioctl(struct vfio_pci_core_device *vdev, uint32_t flags,	33	int vfio_pci_set_irqs_ioctl(struct vfio_pci_core_device *vdev, uint32_t flags,
30	unsigned index, unsigned start, unsigned count,	34	unsigned index, unsigned start, unsigned count,
31	void *data);	35	void *data);
@@ -60,7 +64,6 @@ void vfio_config_free(struct vfio_pci_core_device *vdev); · 28 unmodified lines
60	int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev,	64	int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev,
61	pci_power_t state);	65	pci_power_t state);
62		66
63	bool __vfio_pci_memory_enabled(struct vfio_pci_core_device *vdev);
64	void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev);	67	void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev);
65	u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);	68	u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);
66	void vfio_pci_memory_unlock_and_restore(struct vfio_pci_core_device *vdev,	69	void vfio_pci_memory_unlock_and_restore(struct vfio_pci_core_device *vdev,
@@ -107,4 +110,27 @@ static inline bool vfio_pci_is_vga(struct pci_dev *pdev) · 40 unmodified lines
107	return (pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA;	110	return (pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA;
108	}	111	}
109		112
		113	#ifdef CONFIG_VFIO_PCI_DMABUF
		114	int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
		115	struct vfio_device_feature_dma_buf __user *arg,
		116	size_t argsz);
		117	void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev);
		118	void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, bool revoked);
		119	#else
		120	static inline int
		121	vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
		122	struct vfio_device_feature_dma_buf __user *arg,
		123	size_t argsz)
		124	{
		125	return -ENOTTY;
		126	}
		127	static inline void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev)
		128	{
		129	}
		130	static inline void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev,
		131	bool revoked)
		132	{
		133	}
		134	#endif
		135
110	#endif	136	#endif

drivers/vfio/pci/virtio/common.h+2−3

 #ifdef CONFIG_VIRTIO_VFIO_PCI_ADMIN_LEGACY
 int virtiovf_open_legacy_io(struct virtiovf_pci_core_device *virtvdev);
-long virtiovf_vfio_pci_core_ioctl(struct vfio_device *core_vdev,
-				  unsigned int cmd, unsigned long arg);
 int virtiovf_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
-				       unsigned int cmd, unsigned long arg);
+				       struct vfio_region_info *info,
+				       struct vfio_info_cap *caps);
 ssize_t virtiovf_pci_core_write(struct vfio_device *core_vdev,
 				const char __user *buf, size_t count,
 				loff_t *ppos);

drivers/vfio/pci/virtio/legacy_io.c+8−30

@@ -281,41 +281,19 @@ ssize_t virtiovf_pci_core_write(struct vfio_device *core_vdev, const char __user · 280 unmodified lines
281	}	281	}
282		282
283	int virtiovf_pci_ioctl_get_region_info(struct vfio_device *core_vdev,	283	int virtiovf_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
284	unsigned int cmd, unsigned long arg)	284	struct vfio_region_info *info,
		285	struct vfio_info_cap *caps)
285	{	286	{
286	struct virtiovf_pci_core_device *virtvdev = container_of(	287	struct virtiovf_pci_core_device *virtvdev = container_of(
287	core_vdev, struct virtiovf_pci_core_device, core_device.vdev);	288	core_vdev, struct virtiovf_pci_core_device, core_device.vdev);
288	unsigned long minsz = offsetofend(struct vfio_region_info, offset);
289	void __user uarg = (void __user )arg;
290	struct vfio_region_info info = {};
291		289
292	if (copy_from_user(&info, uarg, minsz))	290	if (info->index != VFIO_PCI_BAR0_REGION_INDEX)
293	return -EFAULT;	291	return vfio_pci_ioctl_get_region_info(core_vdev, info, caps);
294		292
295	if (info.argsz < minsz)	293	info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
296	return -EINVAL;	294	info->size = virtvdev->bar0_virtual_buf_size;
297		295	info->flags = VFIO_REGION_INFO_FLAG_READ \| VFIO_REGION_INFO_FLAG_WRITE;
298	switch (info.index) {	296	return 0;
299	case VFIO_PCI_BAR0_REGION_INDEX:
300	info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index);
301	info.size = virtvdev->bar0_virtual_buf_size;
302	info.flags = VFIO_REGION_INFO_FLAG_READ \|
303	VFIO_REGION_INFO_FLAG_WRITE;
304	return copy_to_user(uarg, &info, minsz) ? -EFAULT : 0;
305	default:
306	return vfio_pci_core_ioctl(core_vdev, cmd, arg);
307	}
308	}
309
310	long virtiovf_vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
311	unsigned long arg)
312	{
313	switch (cmd) {
314	case VFIO_DEVICE_GET_REGION_INFO:
315	return virtiovf_pci_ioctl_get_region_info(core_vdev, cmd, arg);
316	default:
317	return vfio_pci_core_ioctl(core_vdev, cmd, arg);
318	}
319	}	297	}
320		298
321	static int virtiovf_set_notify_addr(struct virtiovf_pci_core_device *virtvdev)	299	static int virtiovf_set_notify_addr(struct virtiovf_pci_core_device *virtvdev)

drivers/vfio/pci/virtio/main.c+4−1

 	.open_device = virtiovf_pci_open_device,
 	.close_device = virtiovf_pci_close_device,
 	.ioctl = vfio_pci_core_ioctl,
+	.get_region_info_caps = vfio_pci_ioctl_get_region_info,
 	.device_feature = vfio_pci_core_ioctl_feature,
 	.read = vfio_pci_core_read,
 	.write = vfio_pci_core_write,
 	.release = virtiovf_pci_core_release_dev,
 	.open_device = virtiovf_pci_open_device,
 	.close_device = virtiovf_pci_close_device,
-	.ioctl = virtiovf_vfio_pci_core_ioctl,
+	.ioctl = vfio_pci_core_ioctl,
+	.get_region_info_caps = virtiovf_pci_ioctl_get_region_info,
 	.device_feature = vfio_pci_core_ioctl_feature,
 	.read = virtiovf_pci_core_read,
 	.write = virtiovf_pci_core_write,
 	.open_device = virtiovf_pci_open_device,
 	.close_device = vfio_pci_core_close_device,
 	.ioctl = vfio_pci_core_ioctl,
+	.get_region_info_caps = vfio_pci_ioctl_get_region_info,
 	.device_feature = vfio_pci_core_ioctl_feature,
 	.read = vfio_pci_core_read,
 	.write = vfio_pci_core_write,

drivers/vfio/platform/vfio_amba.c+1−0

@@ -115,6 +115,7 @@ static const struct vfio_device_ops vfio_amba_ops = { · 114 unmodified lines
115	.open_device = vfio_platform_open_device,	115	.open_device = vfio_platform_open_device,
116	.close_device = vfio_platform_close_device,	116	.close_device = vfio_platform_close_device,
117	.ioctl = vfio_platform_ioctl,	117	.ioctl = vfio_platform_ioctl,
		118	.get_region_info_caps = vfio_platform_ioctl_get_region_info,
118	.read = vfio_platform_read,	119	.read = vfio_platform_read,
119	.write = vfio_platform_write,	120	.write = vfio_platform_write,
120	.mmap = vfio_platform_mmap,	121	.mmap = vfio_platform_mmap,

drivers/vfio/platform/vfio_platform.c+1−0

@@ -101,6 +101,7 @@ static const struct vfio_device_ops vfio_platform_ops = { · 100 unmodified lines
101	.open_device = vfio_platform_open_device,	101	.open_device = vfio_platform_open_device,
102	.close_device = vfio_platform_close_device,	102	.close_device = vfio_platform_close_device,
103	.ioctl = vfio_platform_ioctl,	103	.ioctl = vfio_platform_ioctl,
		104	.get_region_info_caps = vfio_platform_ioctl_get_region_info,
104	.read = vfio_platform_read,	105	.read = vfio_platform_read,
105	.write = vfio_platform_write,	106	.write = vfio_platform_write,
106	.mmap = vfio_platform_mmap,	107	.mmap = vfio_platform_mmap,

drivers/vfio/platform/vfio_platform_common.c+18−22

 }
 EXPORT_SYMBOL_GPL(vfio_platform_open_device);
+int vfio_platform_ioctl_get_region_info(struct vfio_device *core_vdev,
+					struct vfio_region_info *info,
+					struct vfio_info_cap *caps)
+{
+	struct vfio_platform_device *vdev =
+		container_of(core_vdev, struct vfio_platform_device, vdev);
+	if (info->index >= vdev->num_regions)
+		return -EINVAL;
+	/* map offset to the physical address  */
+	info->offset = VFIO_PLATFORM_INDEX_TO_OFFSET(info->index);
+	info->size = vdev->regions[info->index].size;
+	info->flags = vdev->regions[info->index].flags;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_platform_ioctl_get_region_info);
 long vfio_platform_ioctl(struct vfio_device *core_vdev,
 			 unsigned int cmd, unsigned long arg)
 {
 		return copy_to_user((void __user *)arg, &info, minsz) ?
 			-EFAULT : 0;
-	} else if (cmd == VFIO_DEVICE_GET_REGION_INFO) {
-		struct vfio_region_info info;
-		minsz = offsetofend(struct vfio_region_info, offset);
-		if (copy_from_user(&info, (void __user *)arg, minsz))
-			return -EFAULT;
-		if (info.argsz < minsz)
-			return -EINVAL;
-		if (info.index >= vdev->num_regions)
-			return -EINVAL;
-		/* map offset to the physical address  */
-		info.offset = VFIO_PLATFORM_INDEX_TO_OFFSET(info.index);
-		info.size = vdev->regions[info.index].size;
-		info.flags = vdev->regions[info.index].flags;
-		return copy_to_user((void __user *)arg, &info, minsz) ?
-			-EFAULT : 0;
 	} else if (cmd == VFIO_DEVICE_GET_IRQ_INFO) {
 		struct vfio_irq_info info;

drivers/vfio/platform/vfio_platform_private.h+3−0

 void vfio_platform_close_device(struct vfio_device *core_vdev);
 long vfio_platform_ioctl(struct vfio_device *core_vdev,
 			 unsigned int cmd, unsigned long arg);
+int vfio_platform_ioctl_get_region_info(struct vfio_device *core_vdev,
+					struct vfio_region_info *info,
+					struct vfio_info_cap *caps);
 ssize_t vfio_platform_read(struct vfio_device *core_vdev,
 			   char __user *buf, size_t count,
 			   loff_t *ppos);

drivers/vfio/vfio_main.c+51−0

@@ -172,11 +172,13 @@ void vfio_device_put_registration(struct vfio_device *device) · 171 unmodified lines
172	if (refcount_dec_and_test(&device->refcount))	172	if (refcount_dec_and_test(&device->refcount))
173	complete(&device->comp);	173	complete(&device->comp);
174	}	174	}
		175	EXPORT_SYMBOL_GPL(vfio_device_put_registration);
175		176
176	bool vfio_device_try_get_registration(struct vfio_device *device)	177	bool vfio_device_try_get_registration(struct vfio_device *device)
177	{	178	{
178	return refcount_inc_not_zero(&device->refcount);	179	return refcount_inc_not_zero(&device->refcount);
179	}	180	}
		181	EXPORT_SYMBOL_GPL(vfio_device_try_get_registration);
180		182
181	/*	183	/*
182	* VFIO driver API	184	* VFIO driver API
@@ -1259,6 +1261,51 @@ static int vfio_ioctl_device_feature(struct vfio_device *device, · 1076 unmodified lines
1259	}	1261	}
1260	}	1262	}
1261		1263
		1264	static long vfio_get_region_info(struct vfio_device *device,
		1265	struct vfio_region_info __user *arg)
		1266	{
		1267	unsigned long minsz = offsetofend(struct vfio_region_info, offset);
		1268	struct vfio_region_info info = {};
		1269	struct vfio_info_cap caps = {};
		1270	int ret;
		1271
		1272	if (unlikely(!device->ops->get_region_info_caps))
		1273	return -EINVAL;
		1274
		1275	if (copy_from_user(&info, arg, minsz))
		1276	return -EFAULT;
		1277	if (info.argsz < minsz)
		1278	return -EINVAL;
		1279
		1280	ret = device->ops->get_region_info_caps(device, &info, &caps);
		1281	if (ret)
		1282	goto out_free;
		1283
		1284	if (caps.size) {
		1285	info.flags \|= VFIO_REGION_INFO_FLAG_CAPS;
		1286	if (info.argsz < sizeof(info) + caps.size) {
		1287	info.argsz = sizeof(info) + caps.size;
		1288	info.cap_offset = 0;
		1289	} else {
		1290	vfio_info_cap_shift(&caps, sizeof(info));
		1291	if (copy_to_user(arg + 1, caps.buf, caps.size)) {
		1292	ret = -EFAULT;
		1293	goto out_free;
		1294	}
		1295	info.cap_offset = sizeof(info);
		1296	}
		1297	}
		1298
		1299	if (copy_to_user(arg, &info, minsz)){
		1300	ret = -EFAULT;
		1301	goto out_free;
		1302	}
		1303
		1304	out_free:
		1305	kfree(caps.buf);
		1306	return ret;
		1307	}
		1308
1262	static long vfio_device_fops_unl_ioctl(struct file *filep,	1309	static long vfio_device_fops_unl_ioctl(struct file *filep,
1263	unsigned int cmd, unsigned long arg)	1310	unsigned int cmd, unsigned long arg)
1264	{	1311	{
@@ -1296,6 +1343,10 @@ static long vfio_device_fops_unl_ioctl(struct file *filep, · 31 unmodified lines
1296	ret = vfio_ioctl_device_feature(device, uptr);	1343	ret = vfio_ioctl_device_feature(device, uptr);
1297	break;	1344	break;
1298		1345
		1346	case VFIO_DEVICE_GET_REGION_INFO:
		1347	ret = vfio_get_region_info(device, uptr);
		1348	break;
		1349
1299	default:	1350	default:
1300	if (unlikely(!device->ops->ioctl))	1351	if (unlikely(!device->ops->ioctl))
1301	ret = -EINVAL;	1352	ret = -EINVAL;

include/linux/dma-buf-mapping.h+17−0

@@ -0,0 +1,17 @@
	1	/* SPDX-License-Identifier: GPL-2.0-only */
	2	/*
	3	* DMA BUF Mapping Helpers
	4	*
	5	*/
	6	#ifndef __DMA_BUF_MAPPING_H__
	7	#define __DMA_BUF_MAPPING_H__
	8	#include <linux/dma-buf.h>
	9
	10	struct sg_table dma_buf_phys_vec_to_sgt(struct dma_buf_attachment attach,
	11	struct p2pdma_provider *provider,
	12	struct dma_buf_phys_vec *phys_vec,
	13	size_t nr_ranges, size_t size,
	14	enum dma_data_direction dir);
	15	void dma_buf_free_sgt(struct dma_buf_attachment attach, struct sg_table sgt,
	16	enum dma_data_direction dir);
	17	#endif

include/linux/dma-buf.h+11−0

 #include <linux/fs.h>
 #include <linux/dma-fence.h>
 #include <linux/wait.h>
+#include <linux/pci-p2pdma.h>
 struct device;
 struct dma_buf;
 	void *priv;
 };
+/**
+ * struct dma_buf_phys_vec - describe continuous chunk of memory
+ * @paddr:   physical address of that chunk
+ * @len:     Length of this chunk
+ */
+struct dma_buf_phys_vec {
+	phys_addr_t paddr;
+	size_t len;
+};
 /**
  * DEFINE_DMA_BUF_EXPORT_INFO - helper macro for exporters
  * @name: export-info name

include/linux/hisi_acc_qm.h+3−0

@@ -99,6 +99,9 @@ · 98 unmodified lines
99		99
100	#define QM_DEV_ALG_MAX_LEN 256	100	#define QM_DEV_ALG_MAX_LEN 256
101		101
		102	#define QM_MIG_REGION_SEL 0x100198
		103	#define QM_MIG_REGION_EN BIT(0)
		104
102	/* uacce mode of the driver */	105	/* uacce mode of the driver */
103	#define UACCE_MODE_NOUACCE 0 /* don't use uacce */	106	#define UACCE_MODE_NOUACCE 0 /* don't use uacce */
104	#define UACCE_MODE_SVA 1 /* use uacce sva mode */	107	#define UACCE_MODE_SVA 1 /* use uacce sva mode */

include/linux/pci-p2pdma.h+73−47

 struct block_device;
 struct scatterlist;
+/**
+ * struct p2pdma_provider
+ *
+ * A p2pdma provider is a range of MMIO address space available to the CPU.
+ */
+struct p2pdma_provider {
+	struct device *owner;
+	u64 bus_offset;
+};
+enum pci_p2pdma_map_type {
+	/*
+	 * PCI_P2PDMA_MAP_UNKNOWN: Used internally as an initial state before
+	 * the mapping type has been calculated. Exported routines for the API
+	 * will never return this value.
+	 */
+	PCI_P2PDMA_MAP_UNKNOWN = 0,
+	/*
+	 * Not a PCI P2PDMA transfer.
+	 */
+	PCI_P2PDMA_MAP_NONE,
+	/*
+	 * PCI_P2PDMA_MAP_NOT_SUPPORTED: Indicates the transaction will
+	 * traverse the host bridge and the host bridge is not in the
+	 * allowlist. DMA Mapping routines should return an error when
+	 * this is returned.
+	 */
+	PCI_P2PDMA_MAP_NOT_SUPPORTED,
+	/*
+	 * PCI_P2PDMA_MAP_BUS_ADDR: Indicates that two devices can talk to
+	 * each other directly through a PCI switch and the transaction will
+	 * not traverse the host bridge. Such a mapping should program
+	 * the DMA engine with PCI bus addresses.
+	 */
+	PCI_P2PDMA_MAP_BUS_ADDR,
+	/*
+	 * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: Indicates two devices can talk
+	 * to each other, but the transaction traverses a host bridge on the
+	 * allowlist. In this case, a normal mapping either with CPU physical
+	 * addresses (in the case of dma-direct) or IOVA addresses (in the
+	 * case of IOMMUs) should be used to program the DMA engine.
+	 */
+	PCI_P2PDMA_MAP_THRU_HOST_BRIDGE,
+};
 #ifdef CONFIG_PCI_P2PDMA
+int pcim_p2pdma_init(struct pci_dev *pdev);
+struct p2pdma_provider *pcim_p2pdma_provider(struct pci_dev *pdev, int bar);
 int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size,
 		u64 offset);
 int pci_p2pdma_distance_many(struct pci_dev *provider, struct device **clients,
 			    bool *use_p2pdma);
 ssize_t pci_p2pdma_enable_show(char *page, struct pci_dev *p2p_dev,
 			       bool use_p2pdma);
+enum pci_p2pdma_map_type pci_p2pdma_map_type(struct p2pdma_provider *provider,
+					     struct device *dev);
 #else /* CONFIG_PCI_P2PDMA */
+static inline int pcim_p2pdma_init(struct pci_dev *pdev)
+{
+	return -EOPNOTSUPP;
+}
+static inline struct p2pdma_provider *pcim_p2pdma_provider(struct pci_dev *pdev,
+							   int bar)
+{
+	return NULL;
+}
 static inline int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar,
 		size_t size, u64 offset)
 {
 {
 	return sprintf(page, "none\n");
 }
+static inline enum pci_p2pdma_map_type
+pci_p2pdma_map_type(struct p2pdma_provider *provider, struct device *dev)
+{
+	return PCI_P2PDMA_MAP_NOT_SUPPORTED;
+}
 #endif /* CONFIG_PCI_P2PDMA */
 	return pci_p2pmem_find_many(&client, 1);
 }
-enum pci_p2pdma_map_type {
-	/*
-	 * PCI_P2PDMA_MAP_UNKNOWN: Used internally as an initial state before
-	 * the mapping type has been calculated. Exported routines for the API
-	 * will never return this value.
-	 */
-	PCI_P2PDMA_MAP_UNKNOWN = 0,
-	/*
-	 * Not a PCI P2PDMA transfer.
-	 */
-	PCI_P2PDMA_MAP_NONE,
-	/*
-	 * PCI_P2PDMA_MAP_NOT_SUPPORTED: Indicates the transaction will
-	 * traverse the host bridge and the host bridge is not in the
-	 * allowlist. DMA Mapping routines should return an error when
-	 * this is returned.
-	 */
-	PCI_P2PDMA_MAP_NOT_SUPPORTED,
-	/*
-	 * PCI_P2PDMA_MAP_BUS_ADDR: Indicates that two devices can talk to
-	 * each other directly through a PCI switch and the transaction will
-	 * not traverse the host bridge. Such a mapping should program
-	 * the DMA engine with PCI bus addresses.
-	 */
-	PCI_P2PDMA_MAP_BUS_ADDR,
-	/*
-	 * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: Indicates two devices can talk
-	 * to each other, but the transaction traverses a host bridge on the
-	 * allowlist. In this case, a normal mapping either with CPU physical
-	 * addresses (in the case of dma-direct) or IOVA addresses (in the
-	 * case of IOMMUs) should be used to program the DMA engine.
-	 */
-	PCI_P2PDMA_MAP_THRU_HOST_BRIDGE,
-};
 struct pci_p2pdma_map_state {
-	struct dev_pagemap *pgmap;
+	struct p2pdma_provider *mem;
 	enum pci_p2pdma_map_type map;
-	u64 bus_off;
 };
 /* helper for pci_p2pdma_state(), do not use directly */
 void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state,
 		struct device *dev, struct page *page);
 		struct page *page)
 {
 	if (IS_ENABLED(CONFIG_PCI_P2PDMA) && is_pci_p2pdma_page(page)) {
-		if (state->pgmap != page_pgmap(page))
+		__pci_p2pdma_update_state(state, dev, page);
-			__pci_p2pdma_update_state(state, dev, page);
 		return state->map;
 	}
 	return PCI_P2PDMA_MAP_NONE;
 /**
  * pci_p2pdma_bus_addr_map - Translate a physical address to a bus address
  *			     for a PCI_P2PDMA_MAP_BUS_ADDR transfer.
- * @state:	P2P state structure
+ * @provider:	P2P provider structure
  * @paddr:	physical address to map
  *
  * Map a physically contiguous PCI_P2PDMA_MAP_BUS_ADDR transfer.
  */
 static inline dma_addr_t
-pci_p2pdma_bus_addr_map(struct pci_p2pdma_map_state *state, phys_addr_t paddr)
+pci_p2pdma_bus_addr_map(struct p2pdma_provider *provider, phys_addr_t paddr)
 {
-	WARN_ON_ONCE(state->map != PCI_P2PDMA_MAP_BUS_ADDR);
+	return paddr + provider->bus_offset;
-	return paddr + state->bus_off;
 }
 #endif /* _LINUX_PCI_P2P_H */

include/linux/vfio.h+6−0

 struct iommufd_ctx;
 struct iommufd_device;
 struct iommufd_access;
+struct vfio_info_cap;
 /*
  * VFIO devices can be placed in a set, this allows all devices to share this
 			 size_t count, loff_t *size);
 	long	(*ioctl)(struct vfio_device *vdev, unsigned int cmd,
 			 unsigned long arg);
+	int	(*get_region_info_caps)(struct vfio_device *vdev,
+					struct vfio_region_info *info,
+					struct vfio_info_cap *caps);
 	int	(*mmap)(struct vfio_device *vdev, struct vm_area_struct *vma);
 	void	(*request)(struct vfio_device *vdev, unsigned int count);
 	int	(*match)(struct vfio_device *vdev, char *buf);
 int vfio_register_group_dev(struct vfio_device *device);
 int vfio_register_emulated_iommu_dev(struct vfio_device *device);
 void vfio_unregister_group_dev(struct vfio_device *device);
+bool vfio_device_try_get_registration(struct vfio_device *device);
+void vfio_device_put_registration(struct vfio_device *device);
 int vfio_assign_device_set(struct vfio_device *device, void *set_id);
 unsigned int vfio_device_set_open_count(struct vfio_device_set *dev_set);

include/linux/vfio_pci_core.h+67−2

@@ -12,6 +12,7 @@ · 11 unmodified lines
12	#include <linux/pci.h>	12	#include <linux/pci.h>
13	#include <linux/vfio.h>	13	#include <linux/vfio.h>
14	#include <linux/irqbypass.h>	14	#include <linux/irqbypass.h>
		15	#include <linux/rcupdate.h>
15	#include <linux/types.h>	16	#include <linux/types.h>
16	#include <linux/uuid.h>	17	#include <linux/uuid.h>
17	#include <linux/notifier.h>	18	#include <linux/notifier.h>
@@ -26,6 +27,13 @@ · 8 unmodified lines
26		27
27	struct vfio_pci_core_device;	28	struct vfio_pci_core_device;
28	struct vfio_pci_region;	29	struct vfio_pci_region;
		30	struct p2pdma_provider;
		31	struct dma_buf_phys_vec;
		32
		33	struct vfio_pci_eventfd {
		34	struct eventfd_ctx *ctx;
		35	struct rcu_head rcu;
		36	};
29		37
30	struct vfio_pci_regops {	38	struct vfio_pci_regops {
31	ssize_t (rw)(struct vfio_pci_core_device vdev, char __user *buf,	39	ssize_t (rw)(struct vfio_pci_core_device vdev, char __user *buf,
@@ -49,9 +57,48 @@ struct vfio_pci_region { · 17 unmodified lines
49	u32 flags;	57	u32 flags;
50	};	58	};
51		59
		60	struct vfio_pci_device_ops {
		61	int (get_dmabuf_phys)(struct vfio_pci_core_device vdev,
		62	struct p2pdma_provider **provider,
		63	unsigned int region_index,
		64	struct dma_buf_phys_vec *phys_vec,
		65	struct vfio_region_dma_range *dma_ranges,
		66	size_t nr_ranges);
		67	};
		68
		69	#if IS_ENABLED(CONFIG_VFIO_PCI_DMABUF)
		70	int vfio_pci_core_fill_phys_vec(struct dma_buf_phys_vec *phys_vec,
		71	struct vfio_region_dma_range *dma_ranges,
		72	size_t nr_ranges, phys_addr_t start,
		73	phys_addr_t len);
		74	int vfio_pci_core_get_dmabuf_phys(struct vfio_pci_core_device *vdev,
		75	struct p2pdma_provider **provider,
		76	unsigned int region_index,
		77	struct dma_buf_phys_vec *phys_vec,
		78	struct vfio_region_dma_range *dma_ranges,
		79	size_t nr_ranges);
		80	#else
		81	static inline int
		82	vfio_pci_core_fill_phys_vec(struct dma_buf_phys_vec *phys_vec,
		83	struct vfio_region_dma_range *dma_ranges,
		84	size_t nr_ranges, phys_addr_t start,
		85	phys_addr_t len)
		86	{
		87	return -EINVAL;
		88	}
		89	static inline int vfio_pci_core_get_dmabuf_phys(
		90	struct vfio_pci_core_device vdev, struct p2pdma_provider *provider,
		91	unsigned int region_index, struct dma_buf_phys_vec *phys_vec,
		92	struct vfio_region_dma_range *dma_ranges, size_t nr_ranges)
		93	{
		94	return -EOPNOTSUPP;
		95	}
		96	#endif
		97
52	struct vfio_pci_core_device {	98	struct vfio_pci_core_device {
53	struct vfio_device vdev;	99	struct vfio_device vdev;
54	struct pci_dev *pdev;	100	struct pci_dev *pdev;
		101	const struct vfio_pci_device_ops *pci_ops;
55	void __iomem *barmap[PCI_STD_NUM_BARS];	102	void __iomem *barmap[PCI_STD_NUM_BARS];
56	bool bar_mmap_supported[PCI_STD_NUM_BARS];	103	bool bar_mmap_supported[PCI_STD_NUM_BARS];
57	u8 *pci_config_map;	104	u8 *pci_config_map;
@@ -83,8 +130,8 @@ struct vfio_pci_core_device { · 25 unmodified lines
83	struct pci_saved_state *pci_saved_state;	130	struct pci_saved_state *pci_saved_state;
84	struct pci_saved_state *pm_save;	131	struct pci_saved_state *pm_save;
85	int ioeventfds_nr;	132	int ioeventfds_nr;
86	struct eventfd_ctx *err_trigger;	133	struct vfio_pci_eventfd __rcu *err_trigger;
87	struct eventfd_ctx *req_trigger;	134	struct vfio_pci_eventfd __rcu *req_trigger;
88	struct eventfd_ctx *pm_wake_eventfd_ctx;	135	struct eventfd_ctx *pm_wake_eventfd_ctx;
89	struct list_head dummy_resources_list;	136	struct list_head dummy_resources_list;
90	struct mutex ioeventfds_lock;	137	struct mutex ioeventfds_lock;
@@ -94,6 +141,7 @@ struct vfio_pci_core_device { · 3 unmodified lines
94	struct vfio_pci_core_device *sriov_pf_core_dev;	141	struct vfio_pci_core_device *sriov_pf_core_dev;
95	struct notifier_block nb;	142	struct notifier_block nb;
96	struct rw_semaphore memory_lock;	143	struct rw_semaphore memory_lock;
		144	struct list_head dmabufs;
97	};	145	};
98		146
99	/* Will be exported for vfio pci drivers usage */	147	/* Will be exported for vfio pci drivers usage */
@@ -115,10 +163,16 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, · 15 unmodified lines
115	unsigned long arg);	163	unsigned long arg);
116	int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,	164	int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
117	void __user *arg, size_t argsz);	165	void __user *arg, size_t argsz);
		166	int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
		167	struct vfio_region_info *info,
		168	struct vfio_info_cap *caps);
118	ssize_t vfio_pci_core_read(struct vfio_device core_vdev, char __user buf,	169	ssize_t vfio_pci_core_read(struct vfio_device core_vdev, char __user buf,
119	size_t count, loff_t *ppos);	170	size_t count, loff_t *ppos);
120	ssize_t vfio_pci_core_write(struct vfio_device core_vdev, const char __user buf,	171	ssize_t vfio_pci_core_write(struct vfio_device core_vdev, const char __user buf,
121	size_t count, loff_t *ppos);	172	size_t count, loff_t *ppos);
		173	vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
		174	struct vm_fault *vmf, unsigned long pfn,
		175	unsigned int order);
122	int vfio_pci_core_mmap(struct vfio_device core_vdev, struct vm_area_struct vma);	176	int vfio_pci_core_mmap(struct vfio_device core_vdev, struct vm_area_struct vma);
123	void vfio_pci_core_request(struct vfio_device *core_vdev, unsigned int count);	177	void vfio_pci_core_request(struct vfio_device *core_vdev, unsigned int count);
124	int vfio_pci_core_match(struct vfio_device core_vdev, char buf);	178	int vfio_pci_core_match(struct vfio_device core_vdev, char buf);
@@ -134,6 +188,7 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem, · 9 unmodified lines
134	void __iomem io, char __user buf,	188	void __iomem io, char __user buf,
135	loff_t off, size_t count, size_t x_start,	189	loff_t off, size_t count, size_t x_start,
136	size_t x_end, bool iswrite);	190	size_t x_end, bool iswrite);
		191	bool __vfio_pci_memory_enabled(struct vfio_pci_core_device *vdev);
137	bool vfio_pci_core_range_intersect_range(loff_t buf_start, size_t buf_cnt,	192	bool vfio_pci_core_range_intersect_range(loff_t buf_start, size_t buf_cnt,
138	loff_t reg_start, size_t reg_cnt,	193	loff_t reg_start, size_t reg_cnt,
139	loff_t *buf_offset,	194	loff_t *buf_offset,
@@ -161,4 +216,14 @@ VFIO_IOREAD_DECLARATION(32) · 21 unmodified lines
161	VFIO_IOREAD_DECLARATION(64)	216	VFIO_IOREAD_DECLARATION(64)
162	#endif	217	#endif
163		218
		219	static inline bool is_aligned_for_order(struct vm_area_struct *vma,
		220	unsigned long addr,
		221	unsigned long pfn,
		222	unsigned int order)
		223	{
		224	return !(order && (addr < vma->vm_start \|\|
		225	addr + (PAGE_SIZE << order) > vma->vm_end \|\|
		226	!IS_ALIGNED(pfn, 1 << order)));
		227	}
		228
164	#endif /* VFIO_PCI_CORE_H */	229	#endif /* VFIO_PCI_CORE_H */

include/uapi/linux/vfio.h+28−0

@@ -14,6 +14,7 @@ · 13 unmodified lines
14		14
15	#include <linux/types.h>	15	#include <linux/types.h>
16	#include <linux/ioctl.h>	16	#include <linux/ioctl.h>
		17	#include <linux/stddef.h>
17		18
18	#define VFIO_API_VERSION 0	19	#define VFIO_API_VERSION 0
19		20
@@ -1478,6 +1479,33 @@ struct vfio_device_feature_bus_master { · 1458 unmodified lines
1478	};	1479	};
1479	#define VFIO_DEVICE_FEATURE_BUS_MASTER 10	1480	#define VFIO_DEVICE_FEATURE_BUS_MASTER 10
1480		1481
		1482	/**
		1483	* Upon VFIO_DEVICE_FEATURE_GET create a dma_buf fd for the
		1484	* regions selected.
		1485	*
		1486	* open_flags are the typical flags passed to open(2), eg O_RDWR, O_CLOEXEC,
		1487	* etc. offset/length specify a slice of the region to create the dmabuf from.
		1488	* nr_ranges is the total number of (P2P DMA) ranges that comprise the dmabuf.
		1489	*
		1490	* flags should be 0.
		1491	*
		1492	* Return: The fd number on success, -1 and errno is set on failure.
		1493	*/
		1494	#define VFIO_DEVICE_FEATURE_DMA_BUF 11
		1495
		1496	struct vfio_region_dma_range {
		1497	__u64 offset;
		1498	__u64 length;
		1499	};
		1500
		1501	struct vfio_device_feature_dma_buf {
		1502	__u32 region_index;
		1503	__u32 open_flags;
		1504	__u32 flags;
		1505	__u32 nr_ranges;
		1506	struct vfio_region_dma_range dma_ranges[] __counted_by(nr_ranges);
		1507	};
		1508
1481	/* -------- API for Type1 VFIO IOMMU -------- */	1509	/* -------- API for Type1 VFIO IOMMU -------- */
1482		1510
1483	/**	1511	/**

kernel/dma/direct.c+2−2

@@ -479,8 +479,8 @@ int dma_direct_map_sg(struct device dev, struct scatterlist sgl, int nents, · 478 unmodified lines
479	}	479	}
480	break;	480	break;
481	case PCI_P2PDMA_MAP_BUS_ADDR:	481	case PCI_P2PDMA_MAP_BUS_ADDR:
482	sg->dma_address = pci_p2pdma_bus_addr_map(&p2pdma_state,	482	sg->dma_address = pci_p2pdma_bus_addr_map(
483	sg_phys(sg));	483	p2pdma_state.mem, sg_phys(sg));
484	sg_dma_len(sg) = sg->length;	484	sg_dma_len(sg) = sg->length;
485	sg_dma_mark_bus_address(sg);	485	sg_dma_mark_bus_address(sg);
486	continue;	486	continue;

mm/hmm.c+1−1

@@ -811,7 +811,7 @@ dma_addr_t hmm_dma_map_pfn(struct device dev, struct hmm_dma_map map, · 810 unmodified lines
811	break;	811	break;
812	case PCI_P2PDMA_MAP_BUS_ADDR:	812	case PCI_P2PDMA_MAP_BUS_ADDR:
813	pfns[idx] \|= HMM_PFN_P2PDMA_BUS \| HMM_PFN_DMA_MAPPED;	813	pfns[idx] \|= HMM_PFN_P2PDMA_BUS \| HMM_PFN_DMA_MAPPED;
814	return pci_p2pdma_bus_addr_map(p2pdma_state, paddr);	814	return pci_p2pdma_bus_addr_map(p2pdma_state->mem, paddr);
815	default:	815	default:
816	return DMA_MAPPING_ERROR;	816	return DMA_MAPPING_ERROR;
817	}	817	}

samples/vfio-mdev/mbochs.c+24−47

@@ -143,11 +143,6 @@ static struct mdev_parent mbochs_parent; · 142 unmodified lines
143	static atomic_t mbochs_avail_mbytes;	143	static atomic_t mbochs_avail_mbytes;
144	static const struct vfio_device_ops mbochs_dev_ops;	144	static const struct vfio_device_ops mbochs_dev_ops;
145		145
146	struct vfio_region_info_ext {
147	struct vfio_region_info base;
148	struct vfio_region_info_cap_type type;
149	};
150
151	struct mbochs_mode {	146	struct mbochs_mode {
152	u32 drm_format;	147	u32 drm_format;
153	u32 bytepp;	148	u32 bytepp;
@@ -1033,10 +1028,12 @@ static int mbochs_dmabuf_export(struct mbochs_dmabuf *dmabuf) · 879 unmodified lines
1033	return 0;	1028	return 0;
1034	}	1029	}
1035		1030
1036	static int mbochs_get_region_info(struct mdev_state *mdev_state,	1031	static int mbochs_ioctl_get_region_info(struct vfio_device *vdev,
1037	struct vfio_region_info_ext *ext)	1032	struct vfio_region_info *region_info,
		1033	struct vfio_info_cap *caps)
1038	{	1034	{
1039	struct vfio_region_info *region_info = &ext->base;	1035	struct mdev_state *mdev_state =
		1036	container_of(vdev, struct mdev_state, vdev);
1040		1037
1041	if (region_info->index >= MBOCHS_NUM_REGIONS)	1038	if (region_info->index >= MBOCHS_NUM_REGIONS)
1042	return -EINVAL;	1039	return -EINVAL;
@@ -1061,20 +1058,23 @@ static int mbochs_get_region_info(struct mdev_state *mdev_state, · 18 unmodified lines
1061	region_info->flags = (VFIO_REGION_INFO_FLAG_READ \|	1058	region_info->flags = (VFIO_REGION_INFO_FLAG_READ \|
1062	VFIO_REGION_INFO_FLAG_WRITE);	1059	VFIO_REGION_INFO_FLAG_WRITE);
1063	break;	1060	break;
1064	case MBOCHS_EDID_REGION_INDEX:	1061	case MBOCHS_EDID_REGION_INDEX: {
1065	ext->base.argsz = sizeof(*ext);	1062	struct vfio_region_info_cap_type cap_type = {
1066	ext->base.offset = MBOCHS_EDID_OFFSET;	1063	.header.id = VFIO_REGION_INFO_CAP_TYPE,
1067	ext->base.size = MBOCHS_EDID_SIZE;	1064	.header.version = 1,
1068	ext->base.flags = (VFIO_REGION_INFO_FLAG_READ \|	1065	.type = VFIO_REGION_TYPE_GFX,
1069	VFIO_REGION_INFO_FLAG_WRITE \|	1066	.subtype = VFIO_REGION_SUBTYPE_GFX_EDID,
1070	VFIO_REGION_INFO_FLAG_CAPS);	1067	};
1071	ext->base.cap_offset = offsetof(typeof(*ext), type);	1068
1072	ext->type.header.id = VFIO_REGION_INFO_CAP_TYPE;	1069	region_info->offset = MBOCHS_EDID_OFFSET;
1073	ext->type.header.version = 1;	1070	region_info->size = MBOCHS_EDID_SIZE;
1074	ext->type.header.next = 0;	1071	region_info->flags = (VFIO_REGION_INFO_FLAG_READ \|
1075	ext->type.type = VFIO_REGION_TYPE_GFX;	1072	VFIO_REGION_INFO_FLAG_WRITE \|
1076	ext->type.subtype = VFIO_REGION_SUBTYPE_GFX_EDID;	1073	VFIO_REGION_INFO_FLAG_CAPS);
1077	break;	1074
		1075	return vfio_info_add_capability(caps, &cap_type.header,
		1076	sizeof(cap_type));
		1077	}
1078	default:	1078	default:
1079	region_info->size = 0;	1079	region_info->size = 0;
1080	region_info->offset = 0;	1080	region_info->offset = 0;
@@ -1191,7 +1191,7 @@ static long mbochs_ioctl(struct vfio_device *vdev, unsigned int cmd, · 110 unmodified lines
1191	struct mdev_state *mdev_state =	1191	struct mdev_state *mdev_state =
1192	container_of(vdev, struct mdev_state, vdev);	1192	container_of(vdev, struct mdev_state, vdev);
1193	int ret = 0;	1193	int ret = 0;
1194	unsigned long minsz, outsz;	1194	unsigned long minsz;
1195		1195
1196	switch (cmd) {	1196	switch (cmd) {
1197	case VFIO_DEVICE_GET_INFO:	1197	case VFIO_DEVICE_GET_INFO:
@@ -1215,30 +1215,6 @@ static long mbochs_ioctl(struct vfio_device *vdev, unsigned int cmd, · 17 unmodified lines
1215		1215
1216	return 0;	1216	return 0;
1217	}	1217	}
1218	case VFIO_DEVICE_GET_REGION_INFO:
1219	{
1220	struct vfio_region_info_ext info;
1221
1222	minsz = offsetofend(typeof(info), base.offset);
1223
1224	if (copy_from_user(&info, (void __user *)arg, minsz))
1225	return -EFAULT;
1226
1227	outsz = info.base.argsz;
1228	if (outsz < minsz)
1229	return -EINVAL;
1230	if (outsz > sizeof(info))
1231	return -EINVAL;
1232
1233	ret = mbochs_get_region_info(mdev_state, &info);
1234	if (ret)
1235	return ret;
1236
1237	if (copy_to_user((void __user *)arg, &info, outsz))
1238	return -EFAULT;
1239
1240	return 0;
1241	}
1242		1218
1243	case VFIO_DEVICE_GET_IRQ_INFO:	1219	case VFIO_DEVICE_GET_IRQ_INFO:
1244	{	1220	{
@@ -1376,6 +1352,7 @@ static const struct vfio_device_ops mbochs_dev_ops = { · 131 unmodified lines
1376	.read = mbochs_read,	1352	.read = mbochs_read,
1377	.write = mbochs_write,	1353	.write = mbochs_write,
1378	.ioctl = mbochs_ioctl,	1354	.ioctl = mbochs_ioctl,
		1355	.get_region_info_caps = mbochs_ioctl_get_region_info,
1379	.mmap = mbochs_mmap,	1356	.mmap = mbochs_mmap,
1380	.bind_iommufd = vfio_iommufd_emulated_bind,	1357	.bind_iommufd = vfio_iommufd_emulated_bind,
1381	.unbind_iommufd = vfio_iommufd_emulated_unbind,	1358	.unbind_iommufd = vfio_iommufd_emulated_unbind,

samples/vfio-mdev/mdpy.c+7−27

 	return remap_vmalloc_range(vma, mdev_state->memblk, 0);
 }
-static int mdpy_get_region_info(struct mdev_state *mdev_state,
+static int mdpy_ioctl_get_region_info(struct vfio_device *vdev,
-				struct vfio_region_info *region_info,
+				      struct vfio_region_info *region_info,
-				u16 *cap_type_id, void **cap_type)
+				      struct vfio_info_cap *caps)
 {
+	struct mdev_state *mdev_state =
+		container_of(vdev, struct mdev_state, vdev);
 	if (region_info->index >= VFIO_PCI_NUM_REGIONS &&
 	    region_info->index != MDPY_DISPLAY_REGION)
 		return -EINVAL;
 		return 0;
 	}
-	case VFIO_DEVICE_GET_REGION_INFO:
-	{
-		struct vfio_region_info info;
-		u16 cap_type_id = 0;
-		void *cap_type = NULL;
-		minsz = offsetofend(struct vfio_region_info, offset);
-		if (copy_from_user(&info, (void __user *)arg, minsz))
-			return -EFAULT;
-		if (info.argsz < minsz)
-			return -EINVAL;
-		ret = mdpy_get_region_info(mdev_state, &info, &cap_type_id,
-					   &cap_type);
-		if (ret)
-			return ret;
-		if (copy_to_user((void __user *)arg, &info, minsz))
-			return -EFAULT;
-		return 0;
-	}
 	case VFIO_DEVICE_GET_IRQ_INFO:
 	{
 	.read = mdpy_read,
 	.write = mdpy_write,
 	.ioctl = mdpy_ioctl,
+	.get_region_info_caps = mdpy_ioctl_get_region_info,
 	.mmap = mdpy_mmap,
 	.bind_iommufd	= vfio_iommufd_emulated_bind,
 	.unbind_iommufd	= vfio_iommufd_emulated_unbind,

samples/vfio-mdev/mtty.c+7−28

 		u8 lsr = 0;
 		mutex_lock(&mdev_state->rxtx_lock);
-		/* atleast one char in FIFO */
+		/* at least one char in FIFO */
 		if (mdev_state->s[index].rxtx.head !=
 				 mdev_state->s[index].rxtx.tail)
 			lsr |= UART_LSR_DR;
 	return ret;
 }
-static int mtty_get_region_info(struct mdev_state *mdev_state,
+static int mtty_ioctl_get_region_info(struct vfio_device *vdev,
-			 struct vfio_region_info *region_info,
+				      struct vfio_region_info *region_info,
-			 u16 *cap_type_id, void **cap_type)
+				      struct vfio_info_cap *caps)
 {
+	struct mdev_state *mdev_state =
+		container_of(vdev, struct mdev_state, vdev);
 	unsigned int size = 0;
 	u32 bar_index;
 		return 0;
 	}
-	case VFIO_DEVICE_GET_REGION_INFO:
-	{
-		struct vfio_region_info info;
-		u16 cap_type_id = 0;
-		void *cap_type = NULL;
-		minsz = offsetofend(struct vfio_region_info, offset);
-		if (copy_from_user(&info, (void __user *)arg, minsz))
-			return -EFAULT;
-		if (info.argsz < minsz)
-			return -EINVAL;
-		ret = mtty_get_region_info(mdev_state, &info, &cap_type_id,
-					   &cap_type);
-		if (ret)
-			return ret;
-		if (copy_to_user((void __user *)arg, &info, minsz))
-			return -EFAULT;
-		return 0;
-	}
 	case VFIO_DEVICE_GET_IRQ_INFO:
 	{
 	.read = mtty_read,
 	.write = mtty_write,
 	.ioctl = mtty_ioctl,
+	.get_region_info_caps = mtty_ioctl_get_region_info,
 	.bind_iommufd	= vfio_iommufd_emulated_bind,
 	.unbind_iommufd	= vfio_iommufd_emulated_unbind,
 	.attach_ioas	= vfio_iommufd_emulated_attach_ioas,

tools/testing/selftests/vfio/Makefile+9−1

@@ -2,8 +2,14 @@ CFLAGS = $(KHDR_INCLUDES) · 1 unmodified line
2	TEST_GEN_PROGS += vfio_dma_mapping_test	2	TEST_GEN_PROGS += vfio_dma_mapping_test
3	TEST_GEN_PROGS += vfio_iommufd_setup_test	3	TEST_GEN_PROGS += vfio_iommufd_setup_test
4	TEST_GEN_PROGS += vfio_pci_device_test	4	TEST_GEN_PROGS += vfio_pci_device_test
		5	TEST_GEN_PROGS += vfio_pci_device_init_perf_test
5	TEST_GEN_PROGS += vfio_pci_driver_test	6	TEST_GEN_PROGS += vfio_pci_driver_test
6	TEST_PROGS_EXTENDED := run.sh	7
		8	TEST_FILES += scripts/cleanup.sh
		9	TEST_FILES += scripts/lib.sh
		10	TEST_FILES += scripts/run.sh
		11	TEST_FILES += scripts/setup.sh
		12
7	include ../lib.mk	13	include ../lib.mk
8	include lib/libvfio.mk	14	include lib/libvfio.mk
9		15
@@ -11,6 +17,8 @@ CFLAGS += -I$(top_srcdir)/tools/include · 1 unmodified line
11	CFLAGS += -MD	17	CFLAGS += -MD
12	CFLAGS += $(EXTRA_CFLAGS)	18	CFLAGS += $(EXTRA_CFLAGS)
13		19
		20	LDFLAGS += -pthread
		21
14	$(TEST_GEN_PROGS): %: %.o $(LIBVFIO_O)	22	$(TEST_GEN_PROGS): %: %.o $(LIBVFIO_O)
15	$(CC) $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $< $(LIBVFIO_O) $(LDLIBS) -o $@	23	$(CC) $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $< $(LIBVFIO_O) $(LDLIBS) -o $@
16		24

tools/testing/selftests/vfio/lib/drivers/dsa/dsa.c+18−18

@@ -9,7 +9,7 @@ · 8 unmodified lines
9	#include <linux/pci_ids.h>	9	#include <linux/pci_ids.h>
10	#include <linux/sizes.h>	10	#include <linux/sizes.h>
11		11
12	#include <vfio_util.h>	12	#include <libvfio.h>
13		13
14	#include "registers.h"	14	#include "registers.h"
15		15
@@ -70,7 +70,7 @@ static int dsa_probe(struct vfio_pci_device *device) · 54 unmodified lines
70	return -EINVAL;	70	return -EINVAL;
71		71
72	if (dsa_int_handle_request_required(device)) {	72	if (dsa_int_handle_request_required(device)) {
73	printf("Device requires requesting interrupt handles\n");	73	dev_err(device, "Device requires requesting interrupt handles\n");
74	return -EINVAL;	74	return -EINVAL;
75	}	75	}
76		76
@@ -91,23 +91,23 @@ static void dsa_check_sw_err(struct vfio_pci_device *device) · 14 unmodified lines
91	return;	91	return;
92	}	92	}
93		93
94	fprintf(stderr, "SWERR: 0x%016lx 0x%016lx 0x%016lx 0x%016lx\n",	94	dev_err(device, "SWERR: 0x%016lx 0x%016lx 0x%016lx 0x%016lx\n",
95	err.bits[0], err.bits[1], err.bits[2], err.bits[3]);	95	err.bits[0], err.bits[1], err.bits[2], err.bits[3]);
96		96
97	fprintf(stderr, " valid: 0x%x\n", err.valid);	97	dev_err(device, " valid: 0x%x\n", err.valid);
98	fprintf(stderr, " overflow: 0x%x\n", err.overflow);	98	dev_err(device, " overflow: 0x%x\n", err.overflow);
99	fprintf(stderr, " desc_valid: 0x%x\n", err.desc_valid);	99	dev_err(device, " desc_valid: 0x%x\n", err.desc_valid);
100	fprintf(stderr, " wq_idx_valid: 0x%x\n", err.wq_idx_valid);	100	dev_err(device, " wq_idx_valid: 0x%x\n", err.wq_idx_valid);
101	fprintf(stderr, " batch: 0x%x\n", err.batch);	101	dev_err(device, " batch: 0x%x\n", err.batch);
102	fprintf(stderr, " fault_rw: 0x%x\n", err.fault_rw);	102	dev_err(device, " fault_rw: 0x%x\n", err.fault_rw);
103	fprintf(stderr, " priv: 0x%x\n", err.priv);	103	dev_err(device, " priv: 0x%x\n", err.priv);
104	fprintf(stderr, " error: 0x%x\n", err.error);	104	dev_err(device, " error: 0x%x\n", err.error);
105	fprintf(stderr, " wq_idx: 0x%x\n", err.wq_idx);	105	dev_err(device, " wq_idx: 0x%x\n", err.wq_idx);
106	fprintf(stderr, " operation: 0x%x\n", err.operation);	106	dev_err(device, " operation: 0x%x\n", err.operation);
107	fprintf(stderr, " pasid: 0x%x\n", err.pasid);	107	dev_err(device, " pasid: 0x%x\n", err.pasid);
108	fprintf(stderr, " batch_idx: 0x%x\n", err.batch_idx);	108	dev_err(device, " batch_idx: 0x%x\n", err.batch_idx);
109	fprintf(stderr, " invalid_flags: 0x%x\n", err.invalid_flags);	109	dev_err(device, " invalid_flags: 0x%x\n", err.invalid_flags);
110	fprintf(stderr, " fault_addr: 0x%lx\n", err.fault_addr);	110	dev_err(device, " fault_addr: 0x%lx\n", err.fault_addr);
111		111
112	VFIO_FAIL("Software Error Detected!\n");	112	VFIO_FAIL("Software Error Detected!\n");
113	}	113	}
@@ -256,7 +256,7 @@ static int dsa_completion_wait(struct vfio_pci_device *device, · 142 unmodified lines
256	if (status == DSA_COMP_SUCCESS)	256	if (status == DSA_COMP_SUCCESS)
257	return 0;	257	return 0;
258		258
259	printf("Error detected during memcpy operation: 0x%x\n", status);	259	dev_err(device, "Error detected during memcpy operation: 0x%x\n", status);
260	return -1;	260	return -1;
261	}	261	}
262		262

tools/testing/selftests/vfio/lib/drivers/ioat/ioat.c+9−9

@@ -7,7 +7,7 @@ · 6 unmodified lines
7	#include <linux/pci_ids.h>	7	#include <linux/pci_ids.h>
8	#include <linux/sizes.h>	8	#include <linux/sizes.h>
9		9
10	#include <vfio_util.h>	10	#include <libvfio.h>
11		11
12	#include "hw.h"	12	#include "hw.h"
13	#include "registers.h"	13	#include "registers.h"
@@ -51,7 +51,7 @@ static int ioat_probe(struct vfio_pci_device *device) · 37 unmodified lines
51	r = 0;	51	r = 0;
52	break;	52	break;
53	default:	53	default:
54	printf("ioat: Unsupported version: 0x%x\n", version);	54	dev_err(device, "ioat: Unsupported version: 0x%x\n", version);
55	r = -EINVAL;	55	r = -EINVAL;
56	}	56	}
57	return r;	57	return r;
@@ -135,13 +135,13 @@ static void ioat_handle_error(struct vfio_pci_device *device) · 77 unmodified lines
135	{	135	{
136	void *registers = ioat_channel_registers(device);	136	void *registers = ioat_channel_registers(device);
137		137
138	printf("Error detected during memcpy operation!\n"	138	dev_err(device, "Error detected during memcpy operation!\n"
139	" CHANERR: 0x%x\n"	139	" CHANERR: 0x%x\n"
140	" CHANERR_INT: 0x%x\n"	140	" CHANERR_INT: 0x%x\n"
141	" DMAUNCERRSTS: 0x%x\n",	141	" DMAUNCERRSTS: 0x%x\n",
142	readl(registers + IOAT_CHANERR_OFFSET),	142	readl(registers + IOAT_CHANERR_OFFSET),
143	vfio_pci_config_readl(device, IOAT_PCI_CHANERR_INT_OFFSET),	143	vfio_pci_config_readl(device, IOAT_PCI_CHANERR_INT_OFFSET),
144	vfio_pci_config_readl(device, IOAT_PCI_DMAUNCERRSTS_OFFSET));	144	vfio_pci_config_readl(device, IOAT_PCI_DMAUNCERRSTS_OFFSET));
145		145
146	ioat_reset(device);	146	ioat_reset(device);
147	}	147	}

tools/testing/selftests/vfio/lib/include/libvfio.h+26−0

@@ -0,0 +1,26 @@
	1	/* SPDX-License-Identifier: GPL-2.0-only */
	2	#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_H
	3	#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_H
	4
	5	#include <libvfio/assert.h>
	6	#include <libvfio/iommu.h>
	7	#include <libvfio/iova_allocator.h>
	8	#include <libvfio/vfio_pci_device.h>
	9	#include <libvfio/vfio_pci_driver.h>
	10
	11	/*
	12	* Return the BDF string of the device that the test should use.
	13	*
	14	* If a BDF string is provided by the user on the command line (as the last
	15	* element of argv[]), then this function will return that and decrement argc
	16	* by 1.
	17	*
	18	* Otherwise this function will attempt to use the environment variable
	19	* $VFIO_SELFTESTS_BDF.
	20	*
	21	* If BDF cannot be determined then the test will exit with KSFT_SKIP.
	22	*/
	23	const char vfio_selftests_get_bdf(int argc, char *argv[]);
	24	char *vfio_selftests_get_bdfs(int argc, char argv[], int nr_bdfs);
	25
	26	#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_H */

tools/testing/selftests/vfio/lib/include/libvfio/assert.h+54−0

@@ -0,0 +1,54 @@
	1	/* SPDX-License-Identifier: GPL-2.0-only */
	2	#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_ASSERT_H
	3	#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_ASSERT_H
	4
	5	#include <stdio.h>
	6	#include <string.h>
	7	#include <sys/ioctl.h>
	8
	9	#include "../../../../kselftest.h"
	10
	11	#define VFIO_LOG_AND_EXIT(...) do { \
	12	fprintf(stderr, " " __VA_ARGS__); \
	13	fprintf(stderr, "\n"); \
	14	exit(KSFT_FAIL); \
	15	} while (0)
	16
	17	#define VFIO_ASSERT_OP(_lhs, _rhs, _op, ...) do { \
	18	typeof(_lhs) __lhs = (_lhs); \
	19	typeof(_rhs) __rhs = (_rhs); \
	20	\
	21	if (__lhs _op __rhs) \
	22	break; \
	23	\
	24	fprintf(stderr, "%s:%u: Assertion Failure\n\n", __FILE__, __LINE__); \
	25	fprintf(stderr, " Expression: " #_lhs " " #_op " " #_rhs "\n"); \
	26	fprintf(stderr, " Observed: %#lx %s %#lx\n", \
	27	(u64)__lhs, #_op, (u64)__rhs); \
	28	fprintf(stderr, " [errno: %d - %s]\n", errno, strerror(errno)); \
	29	VFIO_LOG_AND_EXIT(__VA_ARGS__); \
	30	} while (0)
	31
	32	#define VFIO_ASSERT_EQ(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, ==, ##__VA_ARGS__)
	33	#define VFIO_ASSERT_NE(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, !=, ##__VA_ARGS__)
	34	#define VFIO_ASSERT_LT(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, <, ##__VA_ARGS__)
	35	#define VFIO_ASSERT_LE(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, <=, ##__VA_ARGS__)
	36	#define VFIO_ASSERT_GT(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, >, ##__VA_ARGS__)
	37	#define VFIO_ASSERT_GE(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, >=, ##__VA_ARGS__)
	38	#define VFIO_ASSERT_TRUE(_a, ...) VFIO_ASSERT_NE(false, (_a), ##__VA_ARGS__)
	39	#define VFIO_ASSERT_FALSE(_a, ...) VFIO_ASSERT_EQ(false, (_a), ##__VA_ARGS__)
	40	#define VFIO_ASSERT_NULL(_a, ...) VFIO_ASSERT_EQ(NULL, _a, ##__VA_ARGS__)
	41	#define VFIO_ASSERT_NOT_NULL(_a, ...) VFIO_ASSERT_NE(NULL, _a, ##__VA_ARGS__)
	42
	43	#define VFIO_FAIL(_fmt, ...) do { \
	44	fprintf(stderr, "%s:%u: FAIL\n\n", __FILE__, __LINE__); \
	45	VFIO_LOG_AND_EXIT(_fmt, ##__VA_ARGS__); \
	46	} while (0)
	47
	48	#define ioctl_assert(_fd, _op, _arg) do { \
	49	void *__arg = (_arg); \
	50	int __ret = ioctl((_fd), (_op), (__arg)); \
	51	VFIO_ASSERT_EQ(__ret, 0, "ioctl(%s, %s, %s) returned %d\n", #_fd, #_op, #_arg, __ret); \
	52	} while (0)
	53
	54	#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_ASSERT_H */

tools/testing/selftests/vfio/lib/include/libvfio/iommu.h+76−0

@@ -0,0 +1,76 @@
	1	/* SPDX-License-Identifier: GPL-2.0-only */
	2	#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_IOMMU_H
	3	#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_IOMMU_H
	4
	5	#include <linux/list.h>
	6	#include <linux/types.h>
	7
	8	#include <libvfio/assert.h>
	9
	10	typedef u64 iova_t;
	11
	12	struct iommu_mode {
	13	const char *name;
	14	const char *container_path;
	15	unsigned long iommu_type;
	16	};
	17
	18	extern const char *default_iommu_mode;
	19
	20	struct dma_region {
	21	struct list_head link;
	22	void *vaddr;
	23	iova_t iova;
	24	u64 size;
	25	};
	26
	27	struct iommu {
	28	const struct iommu_mode *mode;
	29	int container_fd;
	30	int iommufd;
	31	u32 ioas_id;
	32	struct list_head dma_regions;
	33	};
	34
	35	struct iommu iommu_init(const char iommu_mode);
	36	void iommu_cleanup(struct iommu *iommu);
	37
	38	int __iommu_map(struct iommu iommu, struct dma_region region);
	39
	40	static inline void iommu_map(struct iommu iommu, struct dma_region region)
	41	{
	42	VFIO_ASSERT_EQ(__iommu_map(iommu, region), 0);
	43	}
	44
	45	int __iommu_unmap(struct iommu iommu, struct dma_region region, u64 *unmapped);
	46
	47	static inline void iommu_unmap(struct iommu iommu, struct dma_region region)
	48	{
	49	VFIO_ASSERT_EQ(__iommu_unmap(iommu, region, NULL), 0);
	50	}
	51
	52	int __iommu_unmap_all(struct iommu iommu, u64 unmapped);
	53
	54	static inline void iommu_unmap_all(struct iommu *iommu)
	55	{
	56	VFIO_ASSERT_EQ(__iommu_unmap_all(iommu, NULL), 0);
	57	}
	58
	59	int __iommu_hva2iova(struct iommu iommu, void vaddr, iova_t *iova);
	60	iova_t iommu_hva2iova(struct iommu iommu, void vaddr);
	61
	62	struct iommu_iova_range iommu_iova_ranges(struct iommu iommu, u32 *nranges);
	63
	64	/*
	65	* Generator for VFIO selftests fixture variants that replicate across all
	66	* possible IOMMU modes. Tests must define FIXTURE_VARIANT_ADD_IOMMU_MODE()
	67	* which should then use FIXTURE_VARIANT_ADD() to create the variant.
	68	*/
	69	#define FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES(...) \
	70	FIXTURE_VARIANT_ADD_IOMMU_MODE(vfio_type1_iommu, ##__VA_ARGS__); \
	71	FIXTURE_VARIANT_ADD_IOMMU_MODE(vfio_type1v2_iommu, ##__VA_ARGS__); \
	72	FIXTURE_VARIANT_ADD_IOMMU_MODE(iommufd_compat_type1, ##__VA_ARGS__); \
	73	FIXTURE_VARIANT_ADD_IOMMU_MODE(iommufd_compat_type1v2, ##__VA_ARGS__); \
	74	FIXTURE_VARIANT_ADD_IOMMU_MODE(iommufd, ##__VA_ARGS__)
	75
	76	#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_IOMMU_H */

tools/testing/selftests/vfio/lib/include/libvfio/iova_allocator.h+23−0

@@ -0,0 +1,23 @@
	1	/* SPDX-License-Identifier: GPL-2.0-only */
	2	#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_IOVA_ALLOCATOR_H
	3	#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_IOVA_ALLOCATOR_H
	4
	5	#include <uapi/linux/types.h>
	6	#include <linux/list.h>
	7	#include <linux/types.h>
	8	#include <linux/iommufd.h>
	9
	10	#include <libvfio/iommu.h>
	11
	12	struct iova_allocator {
	13	struct iommu_iova_range *ranges;
	14	u32 nranges;
	15	u32 range_idx;
	16	u64 range_offset;
	17	};
	18
	19	struct iova_allocator iova_allocator_init(struct iommu iommu);
	20	void iova_allocator_cleanup(struct iova_allocator *allocator);
	21	iova_t iova_allocator_alloc(struct iova_allocator *allocator, size_t size);
	22
	23	#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_IOVA_ALLOCATOR_H */

tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h+125−0

@@ -0,0 +1,125 @@
	1	/* SPDX-License-Identifier: GPL-2.0-only */
	2	#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DEVICE_H
	3	#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DEVICE_H
	4
	5	#include <fcntl.h>
	6	#include <linux/vfio.h>
	7	#include <linux/pci_regs.h>
	8
	9	#include <libvfio/assert.h>
	10	#include <libvfio/iommu.h>
	11	#include <libvfio/vfio_pci_driver.h>
	12
	13	struct vfio_pci_bar {
	14	struct vfio_region_info info;
	15	void *vaddr;
	16	};
	17
	18	struct vfio_pci_device {
	19	const char *bdf;
	20	int fd;
	21	int group_fd;
	22
	23	struct iommu *iommu;
	24
	25	struct vfio_device_info info;
	26	struct vfio_region_info config_space;
	27	struct vfio_pci_bar bars[PCI_STD_NUM_BARS];
	28
	29	struct vfio_irq_info msi_info;
	30	struct vfio_irq_info msix_info;
	31
	32	/* eventfds for MSI and MSI-x interrupts */
	33	int msi_eventfds[PCI_MSIX_FLAGS_QSIZE + 1];
	34
	35	struct vfio_pci_driver driver;
	36	};
	37
	38	#define dev_info(_dev, _fmt, ...) printf("%s: " _fmt, (_dev)->bdf, ##__VA_ARGS__)
	39	#define dev_err(_dev, _fmt, ...) fprintf(stderr, "%s: " _fmt, (_dev)->bdf, ##__VA_ARGS__)
	40
	41	struct vfio_pci_device vfio_pci_device_init(const char bdf, struct iommu *iommu);
	42	void vfio_pci_device_cleanup(struct vfio_pci_device *device);
	43
	44	void vfio_pci_device_reset(struct vfio_pci_device *device);
	45
	46	void vfio_pci_config_access(struct vfio_pci_device *device, bool write,
	47	size_t config, size_t size, void *data);
	48
	49	#define vfio_pci_config_read(_device, _offset, _type) ({ \
	50	_type __data; \
	51	vfio_pci_config_access((_device), false, _offset, sizeof(__data), &__data); \
	52	__data; \
	53	})
	54
	55	#define vfio_pci_config_readb(_d, _o) vfio_pci_config_read(_d, _o, u8)
	56	#define vfio_pci_config_readw(_d, _o) vfio_pci_config_read(_d, _o, u16)
	57	#define vfio_pci_config_readl(_d, _o) vfio_pci_config_read(_d, _o, u32)
	58
	59	#define vfio_pci_config_write(_device, _offset, _value, _type) do { \
	60	_type __data = (_value); \
	61	vfio_pci_config_access((_device), true, _offset, sizeof(_type), &__data); \
	62	} while (0)
	63
	64	#define vfio_pci_config_writeb(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u8)
	65	#define vfio_pci_config_writew(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u16)
	66	#define vfio_pci_config_writel(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u32)
	67
	68	void vfio_pci_irq_enable(struct vfio_pci_device *device, u32 index,
	69	u32 vector, int count);
	70	void vfio_pci_irq_disable(struct vfio_pci_device *device, u32 index);
	71	void vfio_pci_irq_trigger(struct vfio_pci_device *device, u32 index, u32 vector);
	72
	73	static inline void fcntl_set_nonblock(int fd)
	74	{
	75	int r;
	76
	77	r = fcntl(fd, F_GETFL, 0);
	78	VFIO_ASSERT_NE(r, -1, "F_GETFL failed for fd %d\n", fd);
	79
	80	r = fcntl(fd, F_SETFL, r \| O_NONBLOCK);
	81	VFIO_ASSERT_NE(r, -1, "F_SETFL O_NONBLOCK failed for fd %d\n", fd);
	82	}
	83
	84	static inline void vfio_pci_msi_enable(struct vfio_pci_device *device,
	85	u32 vector, int count)
	86	{
	87	vfio_pci_irq_enable(device, VFIO_PCI_MSI_IRQ_INDEX, vector, count);
	88	}
	89
	90	static inline void vfio_pci_msi_disable(struct vfio_pci_device *device)
	91	{
	92	vfio_pci_irq_disable(device, VFIO_PCI_MSI_IRQ_INDEX);
	93	}
	94
	95	static inline void vfio_pci_msix_enable(struct vfio_pci_device *device,
	96	u32 vector, int count)
	97	{
	98	vfio_pci_irq_enable(device, VFIO_PCI_MSIX_IRQ_INDEX, vector, count);
	99	}
	100
	101	static inline void vfio_pci_msix_disable(struct vfio_pci_device *device)
	102	{
	103	vfio_pci_irq_disable(device, VFIO_PCI_MSIX_IRQ_INDEX);
	104	}
	105
	106	static inline int __to_iova(struct vfio_pci_device device, void vaddr, iova_t *iova)
	107	{
	108	return __iommu_hva2iova(device->iommu, vaddr, iova);
	109	}
	110
	111	static inline iova_t to_iova(struct vfio_pci_device device, void vaddr)
	112	{
	113	return iommu_hva2iova(device->iommu, vaddr);
	114	}
	115
	116	static inline bool vfio_pci_device_match(struct vfio_pci_device *device,
	117	u16 vendor_id, u16 device_id)
	118	{
	119	return (vendor_id == vfio_pci_config_readw(device, PCI_VENDOR_ID)) &&
	120	(device_id == vfio_pci_config_readw(device, PCI_DEVICE_ID));
	121	}
	122
	123	const char vfio_pci_get_cdev_path(const char bdf);
	124
	125	#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DEVICE_H */

tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_driver.h+97−0

@@ -0,0 +1,97 @@
	1	/* SPDX-License-Identifier: GPL-2.0-only */
	2	#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DRIVER_H
	3	#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DRIVER_H
	4
	5	#include <libvfio/iommu.h>
	6
	7	struct vfio_pci_device;
	8
	9	struct vfio_pci_driver_ops {
	10	const char *name;
	11
	12	/**
	13	* @probe() - Check if the driver supports the given device.
	14	*
	15	* Return: 0 on success, non-0 on failure.
	16	*/
	17	int (probe)(struct vfio_pci_device device);
	18
	19	/**
	20	* @init() - Initialize the driver for @device.
	21	*
	22	* Must be called after device->driver.region has been initialized.
	23	*/
	24	void (init)(struct vfio_pci_device device);
	25
	26	/**
	27	* remove() - Deinitialize the driver for @device.
	28	*/
	29	void (remove)(struct vfio_pci_device device);
	30
	31	/**
	32	* memcpy_start() - Kick off @count repeated memcpy operations from
	33	* [@src, @src + @size) to [@dst, @dst + @size).
	34	*
	35	* Guarantees:
	36	* - The device will attempt DMA reads on [src, src + size).
	37	* - The device will attempt DMA writes on [dst, dst + size).
	38	* - The device will not generate any interrupts.
	39	*
	40	* memcpy_start() returns immediately, it does not wait for the
	41	* copies to complete.
	42	*/
	43	void (memcpy_start)(struct vfio_pci_device device,
	44	iova_t src, iova_t dst, u64 size, u64 count);
	45
	46	/**
	47	* memcpy_wait() - Wait until the memcpy operations started by
	48	* memcpy_start() have finished.
	49	*
	50	* Guarantees:
	51	* - All in-flight DMAs initiated by memcpy_start() are fully complete
	52	* before memcpy_wait() returns.
	53	*
	54	* Returns non-0 if the driver detects that an error occurred during the
	55	* memcpy, 0 otherwise.
	56	*/
	57	int (memcpy_wait)(struct vfio_pci_device device);
	58
	59	/**
	60	* send_msi() - Make the device send the MSI device->driver.msi.
	61	*
	62	* Guarantees:
	63	* - The device will send the MSI once.
	64	*/
	65	void (send_msi)(struct vfio_pci_device device);
	66	};
	67
	68	struct vfio_pci_driver {
	69	const struct vfio_pci_driver_ops *ops;
	70	bool initialized;
	71	bool memcpy_in_progress;
	72
	73	/* Region to be used by the driver (e.g. for in-memory descriptors) */
	74	struct dma_region region;
	75
	76	/* The maximum size that can be passed to memcpy_start(). */
	77	u64 max_memcpy_size;
	78
	79	/* The maximum count that can be passed to memcpy_start(). */
	80	u64 max_memcpy_count;
	81
	82	/* The MSI vector the device will signal in ops->send_msi(). */
	83	int msi;
	84	};
	85
	86	void vfio_pci_driver_probe(struct vfio_pci_device *device);
	87	void vfio_pci_driver_init(struct vfio_pci_device *device);
	88	void vfio_pci_driver_remove(struct vfio_pci_device *device);
	89	int vfio_pci_driver_memcpy(struct vfio_pci_device *device,
	90	iova_t src, iova_t dst, u64 size);
	91	void vfio_pci_driver_memcpy_start(struct vfio_pci_device *device,
	92	iova_t src, iova_t dst, u64 size,
	93	u64 count);
	94	int vfio_pci_driver_memcpy_wait(struct vfio_pci_device *device);
	95	void vfio_pci_driver_send_msi(struct vfio_pci_device *device);
	96
	97	#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_VFIO_PCI_DRIVER_H */

tools/testing/selftests/vfio/lib/include/vfio_util.h+0−331

@@ -1,331 +0,0 @@
1	/* SPDX-License-Identifier: GPL-2.0-only */
2	#ifndef SELFTESTS_VFIO_LIB_INCLUDE_VFIO_UTIL_H
3	#define SELFTESTS_VFIO_LIB_INCLUDE_VFIO_UTIL_H
4
5	#include <fcntl.h>
6	#include <string.h>
7
8	#include <uapi/linux/types.h>
9	#include <linux/iommufd.h>
10	#include <linux/list.h>
11	#include <linux/pci_regs.h>
12	#include <linux/vfio.h>
13
14	#include "../../../kselftest.h"
15
16	#define VFIO_LOG_AND_EXIT(...) do { \
17	fprintf(stderr, " " __VA_ARGS__); \
18	fprintf(stderr, "\n"); \
19	exit(KSFT_FAIL); \
20	} while (0)
21
22	#define VFIO_ASSERT_OP(_lhs, _rhs, _op, ...) do { \
23	typeof(_lhs) __lhs = (_lhs); \
24	typeof(_rhs) __rhs = (_rhs); \
25	\
26	if (__lhs _op __rhs) \
27	break; \
28	\
29	fprintf(stderr, "%s:%u: Assertion Failure\n\n", __FILE__, __LINE__); \
30	fprintf(stderr, " Expression: " #_lhs " " #_op " " #_rhs "\n"); \
31	fprintf(stderr, " Observed: %#lx %s %#lx\n", \
32	(u64)__lhs, #_op, (u64)__rhs); \
33	fprintf(stderr, " [errno: %d - %s]\n", errno, strerror(errno)); \
34	VFIO_LOG_AND_EXIT(__VA_ARGS__); \
35	} while (0)
36
37	#define VFIO_ASSERT_EQ(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, ==, ##__VA_ARGS__)
38	#define VFIO_ASSERT_NE(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, !=, ##__VA_ARGS__)
39	#define VFIO_ASSERT_LT(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, <, ##__VA_ARGS__)
40	#define VFIO_ASSERT_LE(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, <=, ##__VA_ARGS__)
41	#define VFIO_ASSERT_GT(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, >, ##__VA_ARGS__)
42	#define VFIO_ASSERT_GE(_a, _b, ...) VFIO_ASSERT_OP(_a, _b, >=, ##__VA_ARGS__)
43	#define VFIO_ASSERT_TRUE(_a, ...) VFIO_ASSERT_NE(false, (_a), ##__VA_ARGS__)
44	#define VFIO_ASSERT_FALSE(_a, ...) VFIO_ASSERT_EQ(false, (_a), ##__VA_ARGS__)
45	#define VFIO_ASSERT_NULL(_a, ...) VFIO_ASSERT_EQ(NULL, _a, ##__VA_ARGS__)
46	#define VFIO_ASSERT_NOT_NULL(_a, ...) VFIO_ASSERT_NE(NULL, _a, ##__VA_ARGS__)
47
48	#define VFIO_FAIL(_fmt, ...) do { \
49	fprintf(stderr, "%s:%u: FAIL\n\n", __FILE__, __LINE__); \
50	VFIO_LOG_AND_EXIT(_fmt, ##__VA_ARGS__); \
51	} while (0)
52
53	struct vfio_iommu_mode {
54	const char *name;
55	const char *container_path;
56	unsigned long iommu_type;
57	};
58
59	/*
60	* Generator for VFIO selftests fixture variants that replicate across all
61	* possible IOMMU modes. Tests must define FIXTURE_VARIANT_ADD_IOMMU_MODE()
62	* which should then use FIXTURE_VARIANT_ADD() to create the variant.
63	*/
64	#define FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES(...) \
65	FIXTURE_VARIANT_ADD_IOMMU_MODE(vfio_type1_iommu, ##__VA_ARGS__); \
66	FIXTURE_VARIANT_ADD_IOMMU_MODE(vfio_type1v2_iommu, ##__VA_ARGS__); \
67	FIXTURE_VARIANT_ADD_IOMMU_MODE(iommufd_compat_type1, ##__VA_ARGS__); \
68	FIXTURE_VARIANT_ADD_IOMMU_MODE(iommufd_compat_type1v2, ##__VA_ARGS__); \
69	FIXTURE_VARIANT_ADD_IOMMU_MODE(iommufd, ##__VA_ARGS__)
70
71	struct vfio_pci_bar {
72	struct vfio_region_info info;
73	void *vaddr;
74	};
75
76	typedef u64 iova_t;
77
78	#define INVALID_IOVA UINT64_MAX
79
80	struct vfio_dma_region {
81	struct list_head link;
82	void *vaddr;
83	iova_t iova;
84	u64 size;
85	};
86
87	struct vfio_pci_device;
88
89	struct vfio_pci_driver_ops {
90	const char *name;
91
92	/**
93	* @probe() - Check if the driver supports the given device.
94	*
95	* Return: 0 on success, non-0 on failure.
96	*/
97	int (probe)(struct vfio_pci_device device);
98
99	/**
100	* @init() - Initialize the driver for @device.
101	*
102	* Must be called after device->driver.region has been initialized.
103	*/
104	void (init)(struct vfio_pci_device device);
105
106	/**
107	* remove() - Deinitialize the driver for @device.
108	*/
109	void (remove)(struct vfio_pci_device device);
110
111	/**
112	* memcpy_start() - Kick off @count repeated memcpy operations from
113	* [@src, @src + @size) to [@dst, @dst + @size).
114	*
115	* Guarantees:
116	* - The device will attempt DMA reads on [src, src + size).
117	* - The device will attempt DMA writes on [dst, dst + size).
118	* - The device will not generate any interrupts.
119	*
120	* memcpy_start() returns immediately, it does not wait for the
121	* copies to complete.
122	*/
123	void (memcpy_start)(struct vfio_pci_device device,
124	iova_t src, iova_t dst, u64 size, u64 count);
125
126	/**
127	* memcpy_wait() - Wait until the memcpy operations started by
128	* memcpy_start() have finished.
129	*
130	* Guarantees:
131	* - All in-flight DMAs initiated by memcpy_start() are fully complete
132	* before memcpy_wait() returns.
133	*
134	* Returns non-0 if the driver detects that an error occurred during the
135	* memcpy, 0 otherwise.
136	*/
137	int (memcpy_wait)(struct vfio_pci_device device);
138
139	/**
140	* send_msi() - Make the device send the MSI device->driver.msi.
141	*
142	* Guarantees:
143	* - The device will send the MSI once.
144	*/
145	void (send_msi)(struct vfio_pci_device device);
146	};
147
148	struct vfio_pci_driver {
149	const struct vfio_pci_driver_ops *ops;
150	bool initialized;
151	bool memcpy_in_progress;
152
153	/* Region to be used by the driver (e.g. for in-memory descriptors) */
154	struct vfio_dma_region region;
155
156	/* The maximum size that can be passed to memcpy_start(). */
157	u64 max_memcpy_size;
158
159	/* The maximum count that can be passed to memcpy_start(). */
160	u64 max_memcpy_count;
161
162	/* The MSI vector the device will signal in ops->send_msi(). */
163	int msi;
164	};
165
166	struct vfio_pci_device {
167	int fd;
168
169	const struct vfio_iommu_mode *iommu_mode;
170	int group_fd;
171	int container_fd;
172
173	int iommufd;
174	u32 ioas_id;
175
176	struct vfio_device_info info;
177	struct vfio_region_info config_space;
178	struct vfio_pci_bar bars[PCI_STD_NUM_BARS];
179
180	struct vfio_irq_info msi_info;
181	struct vfio_irq_info msix_info;
182
183	struct list_head dma_regions;
184
185	/* eventfds for MSI and MSI-x interrupts */
186	int msi_eventfds[PCI_MSIX_FLAGS_QSIZE + 1];
187
188	struct vfio_pci_driver driver;
189	};
190
191	struct iova_allocator {
192	struct iommu_iova_range *ranges;
193	u32 nranges;
194	u32 range_idx;
195	u64 range_offset;
196	};
197
198	/*
199	* Return the BDF string of the device that the test should use.
200	*
201	* If a BDF string is provided by the user on the command line (as the last
202	* element of argv[]), then this function will return that and decrement argc
203	* by 1.
204	*
205	* Otherwise this function will attempt to use the environment variable
206	* $VFIO_SELFTESTS_BDF.
207	*
208	* If BDF cannot be determined then the test will exit with KSFT_SKIP.
209	*/
210	const char vfio_selftests_get_bdf(int argc, char *argv[]);
211	const char vfio_pci_get_cdev_path(const char bdf);
212
213	extern const char *default_iommu_mode;
214
215	struct vfio_pci_device vfio_pci_device_init(const char bdf, const char *iommu_mode);
216	void vfio_pci_device_cleanup(struct vfio_pci_device *device);
217	void vfio_pci_device_reset(struct vfio_pci_device *device);
218
219	struct iommu_iova_range vfio_pci_iova_ranges(struct vfio_pci_device device,
220	u32 *nranges);
221
222	struct iova_allocator iova_allocator_init(struct vfio_pci_device device);
223	void iova_allocator_cleanup(struct iova_allocator *allocator);
224	iova_t iova_allocator_alloc(struct iova_allocator *allocator, size_t size);
225
226	int __vfio_pci_dma_map(struct vfio_pci_device *device,
227	struct vfio_dma_region *region);
228	int __vfio_pci_dma_unmap(struct vfio_pci_device *device,
229	struct vfio_dma_region *region,
230	u64 *unmapped);
231	int __vfio_pci_dma_unmap_all(struct vfio_pci_device device, u64 unmapped);
232
233	static inline void vfio_pci_dma_map(struct vfio_pci_device *device,
234	struct vfio_dma_region *region)
235	{
236	VFIO_ASSERT_EQ(__vfio_pci_dma_map(device, region), 0);
237	}
238
239	static inline void vfio_pci_dma_unmap(struct vfio_pci_device *device,
240	struct vfio_dma_region *region)
241	{
242	VFIO_ASSERT_EQ(__vfio_pci_dma_unmap(device, region, NULL), 0);
243	}
244
245	static inline void vfio_pci_dma_unmap_all(struct vfio_pci_device *device)
246	{
247	VFIO_ASSERT_EQ(__vfio_pci_dma_unmap_all(device, NULL), 0);
248	}
249
250	void vfio_pci_config_access(struct vfio_pci_device *device, bool write,
251	size_t config, size_t size, void *data);
252
253	#define vfio_pci_config_read(_device, _offset, _type) ({ \
254	_type __data; \
255	vfio_pci_config_access((_device), false, _offset, sizeof(__data), &__data); \
256	__data; \
257	})
258
259	#define vfio_pci_config_readb(_d, _o) vfio_pci_config_read(_d, _o, u8)
260	#define vfio_pci_config_readw(_d, _o) vfio_pci_config_read(_d, _o, u16)
261	#define vfio_pci_config_readl(_d, _o) vfio_pci_config_read(_d, _o, u32)
262
263	#define vfio_pci_config_write(_device, _offset, _value, _type) do { \
264	_type __data = (_value); \
265	vfio_pci_config_access((_device), true, _offset, sizeof(_type), &__data); \
266	} while (0)
267
268	#define vfio_pci_config_writeb(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u8)
269	#define vfio_pci_config_writew(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u16)
270	#define vfio_pci_config_writel(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u32)
271
272	void vfio_pci_irq_enable(struct vfio_pci_device *device, u32 index,
273	u32 vector, int count);
274	void vfio_pci_irq_disable(struct vfio_pci_device *device, u32 index);
275	void vfio_pci_irq_trigger(struct vfio_pci_device *device, u32 index, u32 vector);
276
277	static inline void fcntl_set_nonblock(int fd)
278	{
279	int r;
280
281	r = fcntl(fd, F_GETFL, 0);
282	VFIO_ASSERT_NE(r, -1, "F_GETFL failed for fd %d\n", fd);
283
284	r = fcntl(fd, F_SETFL, r \| O_NONBLOCK);
285	VFIO_ASSERT_NE(r, -1, "F_SETFL O_NONBLOCK failed for fd %d\n", fd);
286	}
287
288	static inline void vfio_pci_msi_enable(struct vfio_pci_device *device,
289	u32 vector, int count)
290	{
291	vfio_pci_irq_enable(device, VFIO_PCI_MSI_IRQ_INDEX, vector, count);
292	}
293
294	static inline void vfio_pci_msi_disable(struct vfio_pci_device *device)
295	{
296	vfio_pci_irq_disable(device, VFIO_PCI_MSI_IRQ_INDEX);
297	}
298
299	static inline void vfio_pci_msix_enable(struct vfio_pci_device *device,
300	u32 vector, int count)
301	{
302	vfio_pci_irq_enable(device, VFIO_PCI_MSIX_IRQ_INDEX, vector, count);
303	}
304
305	static inline void vfio_pci_msix_disable(struct vfio_pci_device *device)
306	{
307	vfio_pci_irq_disable(device, VFIO_PCI_MSIX_IRQ_INDEX);
308	}
309
310	iova_t __to_iova(struct vfio_pci_device device, void vaddr);
311	iova_t to_iova(struct vfio_pci_device device, void vaddr);
312
313	static inline bool vfio_pci_device_match(struct vfio_pci_device *device,
314	u16 vendor_id, u16 device_id)
315	{
316	return (vendor_id == vfio_pci_config_readw(device, PCI_VENDOR_ID)) &&
317	(device_id == vfio_pci_config_readw(device, PCI_DEVICE_ID));
318	}
319
320	void vfio_pci_driver_probe(struct vfio_pci_device *device);
321	void vfio_pci_driver_init(struct vfio_pci_device *device);
322	void vfio_pci_driver_remove(struct vfio_pci_device *device);
323	int vfio_pci_driver_memcpy(struct vfio_pci_device *device,
324	iova_t src, iova_t dst, u64 size);
325	void vfio_pci_driver_memcpy_start(struct vfio_pci_device *device,
326	iova_t src, iova_t dst, u64 size,
327	u64 count);
328	int vfio_pci_driver_memcpy_wait(struct vfio_pci_device *device);
329	void vfio_pci_driver_send_msi(struct vfio_pci_device *device);
330
331	#endif /* SELFTESTS_VFIO_LIB_INCLUDE_VFIO_UTIL_H */

tools/testing/selftests/vfio/lib/iommu.c+465−0

@@ -0,0 +1,465 @@
	1	// SPDX-License-Identifier: GPL-2.0-only
	2	#include <dirent.h>
	3	#include <fcntl.h>
	4	#include <libgen.h>
	5	#include <stdint.h>
	6	#include <stdlib.h>
	7	#include <string.h>
	8	#include <unistd.h>
	9
	10	#include <sys/eventfd.h>
	11	#include <sys/ioctl.h>
	12	#include <sys/mman.h>
	13
	14	#include <uapi/linux/types.h>
	15	#include <linux/limits.h>
	16	#include <linux/mman.h>
	17	#include <linux/types.h>
	18	#include <linux/vfio.h>
	19	#include <linux/iommufd.h>
	20
	21	#include "../../../kselftest.h"
	22	#include <libvfio.h>
	23
	24	const char *default_iommu_mode = "iommufd";
	25
	26	/* Reminder: Keep in sync with FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES(). */
	27	static const struct iommu_mode iommu_modes[] = {
	28	{
	29	.name = "vfio_type1_iommu",
	30	.container_path = "/dev/vfio/vfio",
	31	.iommu_type = VFIO_TYPE1_IOMMU,
	32	},
	33	{
	34	.name = "vfio_type1v2_iommu",
	35	.container_path = "/dev/vfio/vfio",
	36	.iommu_type = VFIO_TYPE1v2_IOMMU,
	37	},
	38	{
	39	.name = "iommufd_compat_type1",
	40	.container_path = "/dev/iommu",
	41	.iommu_type = VFIO_TYPE1_IOMMU,
	42	},
	43	{
	44	.name = "iommufd_compat_type1v2",
	45	.container_path = "/dev/iommu",
	46	.iommu_type = VFIO_TYPE1v2_IOMMU,
	47	},
	48	{
	49	.name = "iommufd",
	50	},
	51	};
	52
	53	static const struct iommu_mode lookup_iommu_mode(const char iommu_mode)
	54	{
	55	int i;
	56
	57	if (!iommu_mode)
	58	iommu_mode = default_iommu_mode;
	59
	60	for (i = 0; i < ARRAY_SIZE(iommu_modes); i++) {
	61	if (strcmp(iommu_mode, iommu_modes[i].name))
	62	continue;
	63
	64	return &iommu_modes[i];
	65	}
	66
	67	VFIO_FAIL("Unrecognized IOMMU mode: %s\n", iommu_mode);
	68	}
	69
	70	int __iommu_hva2iova(struct iommu iommu, void vaddr, iova_t *iova)
	71	{
	72	struct dma_region *region;
	73
	74	list_for_each_entry(region, &iommu->dma_regions, link) {
	75	if (vaddr < region->vaddr)
	76	continue;
	77
	78	if (vaddr >= region->vaddr + region->size)
	79	continue;
	80
	81	if (iova)
	82	*iova = region->iova + (vaddr - region->vaddr);
	83
	84	return 0;
	85	}
	86
	87	return -ENOENT;
	88	}
	89
	90	iova_t iommu_hva2iova(struct iommu iommu, void vaddr)
	91	{
	92	iova_t iova;
	93	int ret;
	94
	95	ret = __iommu_hva2iova(iommu, vaddr, &iova);
	96	VFIO_ASSERT_EQ(ret, 0, "%p is not mapped into the iommu\n", vaddr);
	97
	98	return iova;
	99	}
	100
	101	static int vfio_iommu_map(struct iommu iommu, struct dma_region region)
	102	{
	103	struct vfio_iommu_type1_dma_map args = {
	104	.argsz = sizeof(args),
	105	.flags = VFIO_DMA_MAP_FLAG_READ \| VFIO_DMA_MAP_FLAG_WRITE,
	106	.vaddr = (u64)region->vaddr,
	107	.iova = region->iova,
	108	.size = region->size,
	109	};
	110
	111	if (ioctl(iommu->container_fd, VFIO_IOMMU_MAP_DMA, &args))
	112	return -errno;
	113
	114	return 0;
	115	}
	116
	117	static int iommufd_map(struct iommu iommu, struct dma_region region)
	118	{
	119	struct iommu_ioas_map args = {
	120	.size = sizeof(args),
	121	.flags = IOMMU_IOAS_MAP_READABLE \|
	122	IOMMU_IOAS_MAP_WRITEABLE \|
	123	IOMMU_IOAS_MAP_FIXED_IOVA,
	124	.user_va = (u64)region->vaddr,
	125	.iova = region->iova,
	126	.length = region->size,
	127	.ioas_id = iommu->ioas_id,
	128	};
	129
	130	if (ioctl(iommu->iommufd, IOMMU_IOAS_MAP, &args))
	131	return -errno;
	132
	133	return 0;
	134	}
	135
	136	int __iommu_map(struct iommu iommu, struct dma_region region)
	137	{
	138	int ret;
	139
	140	if (iommu->iommufd)
	141	ret = iommufd_map(iommu, region);
	142	else
	143	ret = vfio_iommu_map(iommu, region);
	144
	145	if (ret)
	146	return ret;
	147
	148	list_add(&region->link, &iommu->dma_regions);
	149
	150	return 0;
	151	}
	152
	153	static int __vfio_iommu_unmap(int fd, u64 iova, u64 size, u32 flags, u64 *unmapped)
	154	{
	155	struct vfio_iommu_type1_dma_unmap args = {
	156	.argsz = sizeof(args),
	157	.iova = iova,
	158	.size = size,
	159	.flags = flags,
	160	};
	161
	162	if (ioctl(fd, VFIO_IOMMU_UNMAP_DMA, &args))
	163	return -errno;
	164
	165	if (unmapped)
	166	*unmapped = args.size;
	167
	168	return 0;
	169	}
	170
	171	static int vfio_iommu_unmap(struct iommu iommu, struct dma_region region,
	172	u64 *unmapped)
	173	{
	174	return __vfio_iommu_unmap(iommu->container_fd, region->iova,
	175	region->size, 0, unmapped);
	176	}
	177
	178	static int __iommufd_unmap(int fd, u64 iova, u64 length, u32 ioas_id, u64 *unmapped)
	179	{
	180	struct iommu_ioas_unmap args = {
	181	.size = sizeof(args),
	182	.iova = iova,
	183	.length = length,
	184	.ioas_id = ioas_id,
	185	};
	186
	187	if (ioctl(fd, IOMMU_IOAS_UNMAP, &args))
	188	return -errno;
	189
	190	if (unmapped)
	191	*unmapped = args.length;
	192
	193	return 0;
	194	}
	195
	196	static int iommufd_unmap(struct iommu iommu, struct dma_region region,
	197	u64 *unmapped)
	198	{
	199	return __iommufd_unmap(iommu->iommufd, region->iova, region->size,
	200	iommu->ioas_id, unmapped);
	201	}
	202
	203	int __iommu_unmap(struct iommu iommu, struct dma_region region, u64 *unmapped)
	204	{
	205	int ret;
	206
	207	if (iommu->iommufd)
	208	ret = iommufd_unmap(iommu, region, unmapped);
	209	else
	210	ret = vfio_iommu_unmap(iommu, region, unmapped);
	211
	212	if (ret)
	213	return ret;
	214
	215	list_del_init(&region->link);
	216
	217	return 0;
	218	}
	219
	220	int __iommu_unmap_all(struct iommu iommu, u64 unmapped)
	221	{
	222	int ret;
	223	struct dma_region curr, next;
	224
	225	if (iommu->iommufd)
	226	ret = __iommufd_unmap(iommu->iommufd, 0, UINT64_MAX,
	227	iommu->ioas_id, unmapped);
	228	else
	229	ret = __vfio_iommu_unmap(iommu->container_fd, 0, 0,
	230	VFIO_DMA_UNMAP_FLAG_ALL, unmapped);
	231
	232	if (ret)
	233	return ret;
	234
	235	list_for_each_entry_safe(curr, next, &iommu->dma_regions, link)
	236	list_del_init(&curr->link);
	237
	238	return 0;
	239	}
	240
	241	static struct vfio_info_cap_header next_cap_hdr(void buf, u32 bufsz,
	242	u32 *cap_offset)
	243	{
	244	struct vfio_info_cap_header *hdr;
	245
	246	if (!*cap_offset)
	247	return NULL;
	248
	249	VFIO_ASSERT_LT(*cap_offset, bufsz);
	250	VFIO_ASSERT_GE(bufsz - cap_offset, sizeof(hdr));
	251
	252	hdr = (struct vfio_info_cap_header )((u8 )buf + *cap_offset);
	253	*cap_offset = hdr->next;
	254
	255	return hdr;
	256	}
	257
	258	static struct vfio_info_cap_header vfio_iommu_info_cap_hdr(struct vfio_iommu_type1_info info,
	259	u16 cap_id)
	260	{
	261	struct vfio_info_cap_header *hdr;
	262	u32 cap_offset = info->cap_offset;
	263	u32 max_depth;
	264	u32 depth = 0;
	265
	266	if (!(info->flags & VFIO_IOMMU_INFO_CAPS))
	267	return NULL;
	268
	269	if (cap_offset)
	270	VFIO_ASSERT_GE(cap_offset, sizeof(*info));
	271
	272	max_depth = (info->argsz - sizeof(info)) / sizeof(hdr);
	273
	274	while ((hdr = next_cap_hdr(info, info->argsz, &cap_offset))) {
	275	depth++;
	276	VFIO_ASSERT_LE(depth, max_depth, "Capability chain contains a cycle\n");
	277
	278	if (hdr->id == cap_id)
	279	return hdr;
	280	}
	281
	282	return NULL;
	283	}
	284
	285	/* Return buffer including capability chain, if present. Free with free() */
	286	static struct vfio_iommu_type1_info *vfio_iommu_get_info(int container_fd)
	287	{
	288	struct vfio_iommu_type1_info *info;
	289
	290	info = malloc(sizeof(*info));
	291	VFIO_ASSERT_NOT_NULL(info);
	292
	293	*info = (struct vfio_iommu_type1_info) {
	294	.argsz = sizeof(*info),
	295	};
	296
	297	ioctl_assert(container_fd, VFIO_IOMMU_GET_INFO, info);
	298	VFIO_ASSERT_GE(info->argsz, sizeof(*info));
	299
	300	info = realloc(info, info->argsz);
	301	VFIO_ASSERT_NOT_NULL(info);
	302
	303	ioctl_assert(container_fd, VFIO_IOMMU_GET_INFO, info);
	304	VFIO_ASSERT_GE(info->argsz, sizeof(*info));
	305
	306	return info;
	307	}
	308
	309	/*
	310	* Return iova ranges for the device's container. Normalize vfio_iommu_type1 to
	311	* report iommufd's iommu_iova_range. Free with free().
	312	*/
	313	static struct iommu_iova_range vfio_iommu_iova_ranges(struct iommu iommu,
	314	u32 *nranges)
	315	{
	316	struct vfio_iommu_type1_info_cap_iova_range *cap_range;
	317	struct vfio_iommu_type1_info *info;
	318	struct vfio_info_cap_header *hdr;
	319	struct iommu_iova_range *ranges = NULL;
	320
	321	info = vfio_iommu_get_info(iommu->container_fd);
	322	hdr = vfio_iommu_info_cap_hdr(info, VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE);
	323	VFIO_ASSERT_NOT_NULL(hdr);
	324
	325	cap_range = container_of(hdr, struct vfio_iommu_type1_info_cap_iova_range, header);
	326	VFIO_ASSERT_GT(cap_range->nr_iovas, 0);
	327
	328	ranges = calloc(cap_range->nr_iovas, sizeof(*ranges));
	329	VFIO_ASSERT_NOT_NULL(ranges);
	330
	331	for (u32 i = 0; i < cap_range->nr_iovas; i++) {
	332	ranges[i] = (struct iommu_iova_range){
	333	.start = cap_range->iova_ranges[i].start,
	334	.last = cap_range->iova_ranges[i].end,
	335	};
	336	}
	337
	338	*nranges = cap_range->nr_iovas;
	339
	340	free(info);
	341	return ranges;
	342	}
	343
	344	/* Return iova ranges of the device's IOAS. Free with free() */
	345	static struct iommu_iova_range iommufd_iova_ranges(struct iommu iommu,
	346	u32 *nranges)
	347	{
	348	struct iommu_iova_range *ranges;
	349	int ret;
	350
	351	struct iommu_ioas_iova_ranges query = {
	352	.size = sizeof(query),
	353	.ioas_id = iommu->ioas_id,
	354	};
	355
	356	ret = ioctl(iommu->iommufd, IOMMU_IOAS_IOVA_RANGES, &query);
	357	VFIO_ASSERT_EQ(ret, -1);
	358	VFIO_ASSERT_EQ(errno, EMSGSIZE);
	359	VFIO_ASSERT_GT(query.num_iovas, 0);
	360
	361	ranges = calloc(query.num_iovas, sizeof(*ranges));
	362	VFIO_ASSERT_NOT_NULL(ranges);
	363
	364	query.allowed_iovas = (uintptr_t)ranges;
	365
	366	ioctl_assert(iommu->iommufd, IOMMU_IOAS_IOVA_RANGES, &query);
	367	*nranges = query.num_iovas;
	368
	369	return ranges;
	370	}
	371
	372	static int iova_range_comp(const void a, const void b)
	373	{
	374	const struct iommu_iova_range ra = a, rb = b;
	375
	376	if (ra->start < rb->start)
	377	return -1;
	378
	379	if (ra->start > rb->start)
	380	return 1;
	381
	382	return 0;
	383	}
	384
	385	/* Return sorted IOVA ranges of the device. Free with free(). */
	386	struct iommu_iova_range iommu_iova_ranges(struct iommu iommu, u32 *nranges)
	387	{
	388	struct iommu_iova_range *ranges;
	389
	390	if (iommu->iommufd)
	391	ranges = iommufd_iova_ranges(iommu, nranges);
	392	else
	393	ranges = vfio_iommu_iova_ranges(iommu, nranges);
	394
	395	if (!ranges)
	396	return NULL;
	397
	398	VFIO_ASSERT_GT(*nranges, 0);
	399
	400	/* Sort and check that ranges are sane and non-overlapping */
	401	qsort(ranges, nranges, sizeof(ranges), iova_range_comp);
	402	VFIO_ASSERT_LT(ranges[0].start, ranges[0].last);
	403
	404	for (u32 i = 1; i < *nranges; i++) {
	405	VFIO_ASSERT_LT(ranges[i].start, ranges[i].last);
	406	VFIO_ASSERT_LT(ranges[i - 1].last, ranges[i].start);
	407	}
	408
	409	return ranges;
	410	}
	411
	412	static u32 iommufd_ioas_alloc(int iommufd)
	413	{
	414	struct iommu_ioas_alloc args = {
	415	.size = sizeof(args),
	416	};
	417
	418	ioctl_assert(iommufd, IOMMU_IOAS_ALLOC, &args);
	419	return args.out_ioas_id;
	420	}
	421
	422	struct iommu iommu_init(const char iommu_mode)
	423	{
	424	const char *container_path;
	425	struct iommu *iommu;
	426	int version;
	427
	428	iommu = calloc(1, sizeof(*iommu));
	429	VFIO_ASSERT_NOT_NULL(iommu);
	430
	431	INIT_LIST_HEAD(&iommu->dma_regions);
	432
	433	iommu->mode = lookup_iommu_mode(iommu_mode);
	434
	435	container_path = iommu->mode->container_path;
	436	if (container_path) {
	437	iommu->container_fd = open(container_path, O_RDWR);
	438	VFIO_ASSERT_GE(iommu->container_fd, 0, "open(%s) failed\n", container_path);
	439
	440	version = ioctl(iommu->container_fd, VFIO_GET_API_VERSION);
	441	VFIO_ASSERT_EQ(version, VFIO_API_VERSION, "Unsupported version: %d\n", version);
	442	} else {
	443	/*
	444	* Require device->iommufd to be >0 so that a simple non-0 check can be
	445	* used to check if iommufd is enabled. In practice open() will never
	446	* return 0 unless stdin is closed.
	447	*/
	448	iommu->iommufd = open("/dev/iommu", O_RDWR);
	449	VFIO_ASSERT_GT(iommu->iommufd, 0);
	450
	451	iommu->ioas_id = iommufd_ioas_alloc(iommu->iommufd);
	452	}
	453
	454	return iommu;
	455	}
	456
	457	void iommu_cleanup(struct iommu *iommu)
	458	{
	459	if (iommu->iommufd)
	460	VFIO_ASSERT_EQ(close(iommu->iommufd), 0);
	461	else
	462	VFIO_ASSERT_EQ(close(iommu->container_fd), 0);
	463
	464	free(iommu);
	465	}

tools/testing/selftests/vfio/lib/iova_allocator.c+94−0

@@ -0,0 +1,94 @@
	1	// SPDX-License-Identifier: GPL-2.0-only
	2	#include <dirent.h>
	3	#include <fcntl.h>
	4	#include <libgen.h>
	5	#include <stdint.h>
	6	#include <stdlib.h>
	7	#include <string.h>
	8	#include <unistd.h>
	9
	10	#include <sys/eventfd.h>
	11	#include <sys/ioctl.h>
	12	#include <sys/mman.h>
	13
	14	#include <uapi/linux/types.h>
	15	#include <linux/iommufd.h>
	16	#include <linux/limits.h>
	17	#include <linux/mman.h>
	18	#include <linux/overflow.h>
	19	#include <linux/types.h>
	20	#include <linux/vfio.h>
	21
	22	#include <libvfio.h>
	23
	24	struct iova_allocator iova_allocator_init(struct iommu iommu)
	25	{
	26	struct iova_allocator *allocator;
	27	struct iommu_iova_range *ranges;
	28	u32 nranges;
	29
	30	ranges = iommu_iova_ranges(iommu, &nranges);
	31	VFIO_ASSERT_NOT_NULL(ranges);
	32
	33	allocator = malloc(sizeof(*allocator));
	34	VFIO_ASSERT_NOT_NULL(allocator);
	35
	36	*allocator = (struct iova_allocator){
	37	.ranges = ranges,
	38	.nranges = nranges,
	39	.range_idx = 0,
	40	.range_offset = 0,
	41	};
	42
	43	return allocator;
	44	}
	45
	46	void iova_allocator_cleanup(struct iova_allocator *allocator)
	47	{
	48	free(allocator->ranges);
	49	free(allocator);
	50	}
	51
	52	iova_t iova_allocator_alloc(struct iova_allocator *allocator, size_t size)
	53	{
	54	VFIO_ASSERT_GT(size, 0, "Invalid size arg, zero\n");
	55	VFIO_ASSERT_EQ(size & (size - 1), 0, "Invalid size arg, non-power-of-2\n");
	56
	57	for (;;) {
	58	struct iommu_iova_range *range;
	59	iova_t iova, last;
	60
	61	VFIO_ASSERT_LT(allocator->range_idx, allocator->nranges,
	62	"IOVA allocator out of space\n");
	63
	64	range = &allocator->ranges[allocator->range_idx];
	65	iova = range->start + allocator->range_offset;
	66
	67	/* Check for sufficient space at the current offset */
	68	if (check_add_overflow(iova, size - 1, &last) \|\|
	69	last > range->last)
	70	goto next_range;
	71
	72	/* Align iova to size */
	73	iova = last & ~(size - 1);
	74
	75	/* Check for sufficient space at the aligned iova */
	76	if (check_add_overflow(iova, size - 1, &last) \|\|
	77	last > range->last)
	78	goto next_range;
	79
	80	if (last == range->last) {
	81	allocator->range_idx++;
	82	allocator->range_offset = 0;
	83	} else {
	84	allocator->range_offset = last - range->start + 1;
	85	}
	86
	87	return iova;
	88
	89	next_range:
	90	allocator->range_idx++;
	91	allocator->range_offset = 0;
	92	}
	93	}
	94

tools/testing/selftests/vfio/lib/libvfio.c+78−0

@@ -0,0 +1,78 @@
	1	// SPDX-License-Identifier: GPL-2.0-only
	2
	3	#include <stdio.h>
	4	#include <stdlib.h>
	5
	6	#include "../../../kselftest.h"
	7	#include <libvfio.h>
	8
	9	static bool is_bdf(const char *str)
	10	{
	11	unsigned int s, b, d, f;
	12	int length, count;
	13
	14	count = sscanf(str, "%4x:%2x:%2x.%2x%n", &s, &b, &d, &f, &length);
	15	return count == 4 && length == strlen(str);
	16	}
	17
	18	static char *get_bdfs_cmdline(int argc, char argv[], int nr_bdfs)
	19	{
	20	int i;
	21
	22	for (i = *argc - 1; i > 0 && is_bdf(argv[i]); i--)
	23	continue;
	24
	25	i++;
	26	nr_bdfs = argc - i;
	27	argc -= nr_bdfs;
	28
	29	return *nr_bdfs ? &argv[i] : NULL;
	30	}
	31
	32	static char *get_bdf_env(void)
	33	{
	34	char *bdf;
	35
	36	bdf = getenv("VFIO_SELFTESTS_BDF");
	37	if (!bdf)
	38	return NULL;
	39
	40	VFIO_ASSERT_TRUE(is_bdf(bdf), "Invalid BDF: %s\n", bdf);
	41	return bdf;
	42	}
	43
	44	char *vfio_selftests_get_bdfs(int argc, char argv[], int nr_bdfs)
	45	{
	46	static char *env_bdf;
	47	char **bdfs;
	48
	49	bdfs = get_bdfs_cmdline(argc, argv, nr_bdfs);
	50	if (bdfs)
	51	return bdfs;
	52
	53	env_bdf = get_bdf_env();
	54	if (env_bdf) {
	55	*nr_bdfs = 1;
	56	return &env_bdf;
	57	}
	58
	59	fprintf(stderr, "Unable to determine which device(s) to use, skipping test.\n");
	60	fprintf(stderr, "\n");
	61	fprintf(stderr, "To pass the device address via environment variable:\n");
	62	fprintf(stderr, "\n");
	63	fprintf(stderr, " export VFIO_SELFTESTS_BDF=\"segment:bus:device.function\"\n");
	64	fprintf(stderr, " %s [options]\n", argv[0]);
	65	fprintf(stderr, "\n");
	66	fprintf(stderr, "To pass the device address(es) via argv:\n");
	67	fprintf(stderr, "\n");
	68	fprintf(stderr, " %s [options] segment:bus:device.function ...\n", argv[0]);
	69	fprintf(stderr, "\n");
	70	exit(KSFT_SKIP);
	71	}
	72
	73	const char vfio_selftests_get_bdf(int argc, char *argv[])
	74	{
	75	int nr_bdfs;
	76
	77	return vfio_selftests_get_bdfs(argc, argv, &nr_bdfs)[0];
	78	}

tools/testing/selftests/vfio/lib/libvfio.mk+14−9

@@ -1,24 +1,29 @@
1	include $(top_srcdir)/scripts/subarch.include	1	include $(top_srcdir)/scripts/subarch.include
2	ARCH ?= $(SUBARCH)	2	ARCH ?= $(SUBARCH)
3		3
4	VFIO_DIR := $(selfdir)/vfio	4	LIBVFIO_SRCDIR := $(selfdir)/vfio/lib
5		5
6	LIBVFIO_C := lib/vfio_pci_device.c	6	LIBVFIO_C := iommu.c
7	LIBVFIO_C += lib/vfio_pci_driver.c	7	LIBVFIO_C += iova_allocator.c
		8	LIBVFIO_C += libvfio.c
		9	LIBVFIO_C += vfio_pci_device.c
		10	LIBVFIO_C += vfio_pci_driver.c
8		11
9	ifeq ($(ARCH:x86_64=x86),x86)	12	ifeq ($(ARCH:x86_64=x86),x86)
10	LIBVFIO_C += lib/drivers/ioat/ioat.c	13	LIBVFIO_C += drivers/ioat/ioat.c
11	LIBVFIO_C += lib/drivers/dsa/dsa.c	14	LIBVFIO_C += drivers/dsa/dsa.c
12	endif	15	endif
13		16
14	LIBVFIO_O := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBVFIO_C))	17	LIBVFIO_OUTPUT := $(OUTPUT)/libvfio
		18
		19	LIBVFIO_O := $(patsubst %.c, $(LIBVFIO_OUTPUT)/%.o, $(LIBVFIO_C))
15		20
16	LIBVFIO_O_DIRS := $(shell dirname $(LIBVFIO_O) \| uniq)	21	LIBVFIO_O_DIRS := $(shell dirname $(LIBVFIO_O) \| uniq)
17	$(shell mkdir -p $(LIBVFIO_O_DIRS))	22	$(shell mkdir -p $(LIBVFIO_O_DIRS))
18		23
19	CFLAGS += -I$(VFIO_DIR)/lib/include	24	CFLAGS += -I$(LIBVFIO_SRCDIR)/include
20		25
21	$(LIBVFIO_O): $(OUTPUT)/%.o : $(VFIO_DIR)/%.c	26	$(LIBVFIO_O): $(LIBVFIO_OUTPUT)/%.o : $(LIBVFIO_SRCDIR)/%.c
22	$(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $< -o $@	27	$(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $< -o $@
23		28
24	EXTRA_CLEAN += $(LIBVFIO_O)	29	EXTRA_CLEAN += $(LIBVFIO_OUTPUT)

tools/testing/selftests/vfio/lib/vfio_pci_device.c+19−537

@@ -20,286 +20,10 @@ · 19 unmodified lines
20	#include <linux/vfio.h>	20	#include <linux/vfio.h>
21		21
22	#include "../../../kselftest.h"	22	#include "../../../kselftest.h"
23	#include <vfio_util.h>	23	#include <libvfio.h>
24		24
25	#define PCI_SYSFS_PATH "/sys/bus/pci/devices"	25	#define PCI_SYSFS_PATH "/sys/bus/pci/devices"
26		26
27	#define ioctl_assert(_fd, _op, _arg) do { \
28	void *__arg = (_arg); \
29	int __ret = ioctl((_fd), (_op), (__arg)); \
30	VFIO_ASSERT_EQ(__ret, 0, "ioctl(%s, %s, %s) returned %d\n", #_fd, #_op, #_arg, __ret); \
31	} while (0)
32
33	static struct vfio_info_cap_header next_cap_hdr(void buf, u32 bufsz,
34	u32 *cap_offset)
35	{
36	struct vfio_info_cap_header *hdr;
37
38	if (!*cap_offset)
39	return NULL;
40
41	VFIO_ASSERT_LT(*cap_offset, bufsz);
42	VFIO_ASSERT_GE(bufsz - cap_offset, sizeof(hdr));
43
44	hdr = (struct vfio_info_cap_header )((u8 )buf + *cap_offset);
45	*cap_offset = hdr->next;
46
47	return hdr;
48	}
49
50	static struct vfio_info_cap_header vfio_iommu_info_cap_hdr(struct vfio_iommu_type1_info info,
51	u16 cap_id)
52	{
53	struct vfio_info_cap_header *hdr;
54	u32 cap_offset = info->cap_offset;
55	u32 max_depth;
56	u32 depth = 0;
57
58	if (!(info->flags & VFIO_IOMMU_INFO_CAPS))
59	return NULL;
60
61	if (cap_offset)
62	VFIO_ASSERT_GE(cap_offset, sizeof(*info));
63
64	max_depth = (info->argsz - sizeof(info)) / sizeof(hdr);
65
66	while ((hdr = next_cap_hdr(info, info->argsz, &cap_offset))) {
67	depth++;
68	VFIO_ASSERT_LE(depth, max_depth, "Capability chain contains a cycle\n");
69
70	if (hdr->id == cap_id)
71	return hdr;
72	}
73
74	return NULL;
75	}
76
77	/* Return buffer including capability chain, if present. Free with free() */
78	static struct vfio_iommu_type1_info vfio_iommu_get_info(struct vfio_pci_device device)
79	{
80	struct vfio_iommu_type1_info *info;
81
82	info = malloc(sizeof(*info));
83	VFIO_ASSERT_NOT_NULL(info);
84
85	*info = (struct vfio_iommu_type1_info) {
86	.argsz = sizeof(*info),
87	};
88
89	ioctl_assert(device->container_fd, VFIO_IOMMU_GET_INFO, info);
90	VFIO_ASSERT_GE(info->argsz, sizeof(*info));
91
92	info = realloc(info, info->argsz);
93	VFIO_ASSERT_NOT_NULL(info);
94
95	ioctl_assert(device->container_fd, VFIO_IOMMU_GET_INFO, info);
96	VFIO_ASSERT_GE(info->argsz, sizeof(*info));
97
98	return info;
99	}
100
101	/*
102	* Return iova ranges for the device's container. Normalize vfio_iommu_type1 to
103	* report iommufd's iommu_iova_range. Free with free().
104	*/
105	static struct iommu_iova_range vfio_iommu_iova_ranges(struct vfio_pci_device device,
106	u32 *nranges)
107	{
108	struct vfio_iommu_type1_info_cap_iova_range *cap_range;
109	struct vfio_iommu_type1_info *info;
110	struct vfio_info_cap_header *hdr;
111	struct iommu_iova_range *ranges = NULL;
112
113	info = vfio_iommu_get_info(device);
114	hdr = vfio_iommu_info_cap_hdr(info, VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE);
115	VFIO_ASSERT_NOT_NULL(hdr);
116
117	cap_range = container_of(hdr, struct vfio_iommu_type1_info_cap_iova_range, header);
118	VFIO_ASSERT_GT(cap_range->nr_iovas, 0);
119
120	ranges = calloc(cap_range->nr_iovas, sizeof(*ranges));
121	VFIO_ASSERT_NOT_NULL(ranges);
122
123	for (u32 i = 0; i < cap_range->nr_iovas; i++) {
124	ranges[i] = (struct iommu_iova_range){
125	.start = cap_range->iova_ranges[i].start,
126	.last = cap_range->iova_ranges[i].end,
127	};
128	}
129
130	*nranges = cap_range->nr_iovas;
131
132	free(info);
133	return ranges;
134	}
135
136	/* Return iova ranges of the device's IOAS. Free with free() */
137	static struct iommu_iova_range iommufd_iova_ranges(struct vfio_pci_device device,
138	u32 *nranges)
139	{
140	struct iommu_iova_range *ranges;
141	int ret;
142
143	struct iommu_ioas_iova_ranges query = {
144	.size = sizeof(query),
145	.ioas_id = device->ioas_id,
146	};
147
148	ret = ioctl(device->iommufd, IOMMU_IOAS_IOVA_RANGES, &query);
149	VFIO_ASSERT_EQ(ret, -1);
150	VFIO_ASSERT_EQ(errno, EMSGSIZE);
151	VFIO_ASSERT_GT(query.num_iovas, 0);
152
153	ranges = calloc(query.num_iovas, sizeof(*ranges));
154	VFIO_ASSERT_NOT_NULL(ranges);
155
156	query.allowed_iovas = (uintptr_t)ranges;
157
158	ioctl_assert(device->iommufd, IOMMU_IOAS_IOVA_RANGES, &query);
159	*nranges = query.num_iovas;
160
161	return ranges;
162	}
163
164	static int iova_range_comp(const void a, const void b)
165	{
166	const struct iommu_iova_range ra = a, rb = b;
167
168	if (ra->start < rb->start)
169	return -1;
170
171	if (ra->start > rb->start)
172	return 1;
173
174	return 0;
175	}
176
177	/* Return sorted IOVA ranges of the device. Free with free(). */
178	struct iommu_iova_range vfio_pci_iova_ranges(struct vfio_pci_device device,
179	u32 *nranges)
180	{
181	struct iommu_iova_range *ranges;
182
183	if (device->iommufd)
184	ranges = iommufd_iova_ranges(device, nranges);
185	else
186	ranges = vfio_iommu_iova_ranges(device, nranges);
187
188	if (!ranges)
189	return NULL;
190
191	VFIO_ASSERT_GT(*nranges, 0);
192
193	/* Sort and check that ranges are sane and non-overlapping */
194	qsort(ranges, nranges, sizeof(ranges), iova_range_comp);
195	VFIO_ASSERT_LT(ranges[0].start, ranges[0].last);
196
197	for (u32 i = 1; i < *nranges; i++) {
198	VFIO_ASSERT_LT(ranges[i].start, ranges[i].last);
199	VFIO_ASSERT_LT(ranges[i - 1].last, ranges[i].start);
200	}
201
202	return ranges;
203	}
204
205	struct iova_allocator iova_allocator_init(struct vfio_pci_device device)
206	{
207	struct iova_allocator *allocator;
208	struct iommu_iova_range *ranges;
209	u32 nranges;
210
211	ranges = vfio_pci_iova_ranges(device, &nranges);
212	VFIO_ASSERT_NOT_NULL(ranges);
213
214	allocator = malloc(sizeof(*allocator));
215	VFIO_ASSERT_NOT_NULL(allocator);
216
217	*allocator = (struct iova_allocator){
218	.ranges = ranges,
219	.nranges = nranges,
220	.range_idx = 0,
221	.range_offset = 0,
222	};
223
224	return allocator;
225	}
226
227	void iova_allocator_cleanup(struct iova_allocator *allocator)
228	{
229	free(allocator->ranges);
230	free(allocator);
231	}
232
233	iova_t iova_allocator_alloc(struct iova_allocator *allocator, size_t size)
234	{
235	VFIO_ASSERT_GT(size, 0, "Invalid size arg, zero\n");
236	VFIO_ASSERT_EQ(size & (size - 1), 0, "Invalid size arg, non-power-of-2\n");
237
238	for (;;) {
239	struct iommu_iova_range *range;
240	iova_t iova, last;
241
242	VFIO_ASSERT_LT(allocator->range_idx, allocator->nranges,
243	"IOVA allocator out of space\n");
244
245	range = &allocator->ranges[allocator->range_idx];
246	iova = range->start + allocator->range_offset;
247
248	/* Check for sufficient space at the current offset */
249	if (check_add_overflow(iova, size - 1, &last) \|\|
250	last > range->last)
251	goto next_range;
252
253	/* Align iova to size */
254	iova = last & ~(size - 1);
255
256	/* Check for sufficient space at the aligned iova */
257	if (check_add_overflow(iova, size - 1, &last) \|\|
258	last > range->last)
259	goto next_range;
260
261	if (last == range->last) {
262	allocator->range_idx++;
263	allocator->range_offset = 0;
264	} else {
265	allocator->range_offset = last - range->start + 1;
266	}
267
268	return iova;
269
270	next_range:
271	allocator->range_idx++;
272	allocator->range_offset = 0;
273	}
274	}
275
276	iova_t __to_iova(struct vfio_pci_device device, void vaddr)
277	{
278	struct vfio_dma_region *region;
279
280	list_for_each_entry(region, &device->dma_regions, link) {
281	if (vaddr < region->vaddr)
282	continue;
283
284	if (vaddr >= region->vaddr + region->size)
285	continue;
286
287	return region->iova + (vaddr - region->vaddr);
288	}
289
290	return INVALID_IOVA;
291	}
292
293	iova_t to_iova(struct vfio_pci_device device, void vaddr)
294	{
295	iova_t iova;
296
297	iova = __to_iova(device, vaddr);
298	VFIO_ASSERT_NE(iova, INVALID_IOVA, "%p is not mapped into device.\n", vaddr);
299
300	return iova;
301	}
302
303	static void vfio_pci_irq_set(struct vfio_pci_device *device,	27	static void vfio_pci_irq_set(struct vfio_pci_device *device,
304	u32 index, u32 vector, u32 count, int *fds)	28	u32 index, u32 vector, u32 count, int *fds)
305	{	29	{
@@ -386,141 +110,6 @@ static void vfio_pci_irq_get(struct vfio_pci_device *device, u32 index, · 80 unmodified lines
386	ioctl_assert(device->fd, VFIO_DEVICE_GET_IRQ_INFO, irq_info);	110	ioctl_assert(device->fd, VFIO_DEVICE_GET_IRQ_INFO, irq_info);
387	}	111	}
388		112
389	static int vfio_iommu_dma_map(struct vfio_pci_device *device,
390	struct vfio_dma_region *region)
391	{
392	struct vfio_iommu_type1_dma_map args = {
393	.argsz = sizeof(args),
394	.flags = VFIO_DMA_MAP_FLAG_READ \| VFIO_DMA_MAP_FLAG_WRITE,
395	.vaddr = (u64)region->vaddr,
396	.iova = region->iova,
397	.size = region->size,
398	};
399
400	if (ioctl(device->container_fd, VFIO_IOMMU_MAP_DMA, &args))
401	return -errno;
402
403	return 0;
404	}
405
406	static int iommufd_dma_map(struct vfio_pci_device *device,
407	struct vfio_dma_region *region)
408	{
409	struct iommu_ioas_map args = {
410	.size = sizeof(args),
411	.flags = IOMMU_IOAS_MAP_READABLE \|
412	IOMMU_IOAS_MAP_WRITEABLE \|
413	IOMMU_IOAS_MAP_FIXED_IOVA,
414	.user_va = (u64)region->vaddr,
415	.iova = region->iova,
416	.length = region->size,
417	.ioas_id = device->ioas_id,
418	};
419
420	if (ioctl(device->iommufd, IOMMU_IOAS_MAP, &args))
421	return -errno;
422
423	return 0;
424	}
425
426	int __vfio_pci_dma_map(struct vfio_pci_device *device,
427	struct vfio_dma_region *region)
428	{
429	int ret;
430
431	if (device->iommufd)
432	ret = iommufd_dma_map(device, region);
433	else
434	ret = vfio_iommu_dma_map(device, region);
435
436	if (ret)
437	return ret;
438
439	list_add(&region->link, &device->dma_regions);
440
441	return 0;
442	}
443
444	static int vfio_iommu_dma_unmap(int fd, u64 iova, u64 size, u32 flags,
445	u64 *unmapped)
446	{
447	struct vfio_iommu_type1_dma_unmap args = {
448	.argsz = sizeof(args),
449	.iova = iova,
450	.size = size,
451	.flags = flags,
452	};
453
454	if (ioctl(fd, VFIO_IOMMU_UNMAP_DMA, &args))
455	return -errno;
456
457	if (unmapped)
458	*unmapped = args.size;
459
460	return 0;
461	}
462
463	static int iommufd_dma_unmap(int fd, u64 iova, u64 length, u32 ioas_id,
464	u64 *unmapped)
465	{
466	struct iommu_ioas_unmap args = {
467	.size = sizeof(args),
468	.iova = iova,
469	.length = length,
470	.ioas_id = ioas_id,
471	};
472
473	if (ioctl(fd, IOMMU_IOAS_UNMAP, &args))
474	return -errno;
475
476	if (unmapped)
477	*unmapped = args.length;
478
479	return 0;
480	}
481
482	int __vfio_pci_dma_unmap(struct vfio_pci_device *device,
483	struct vfio_dma_region region, u64 unmapped)
484	{
485	int ret;
486
487	if (device->iommufd)
488	ret = iommufd_dma_unmap(device->iommufd, region->iova,
489	region->size, device->ioas_id,
490	unmapped);
491	else
492	ret = vfio_iommu_dma_unmap(device->container_fd, region->iova,
493	region->size, 0, unmapped);
494
495	if (ret)
496	return ret;
497
498	list_del_init(&region->link);
499
500	return 0;
501	}
502
503	int __vfio_pci_dma_unmap_all(struct vfio_pci_device device, u64 unmapped)
504	{
505	int ret;
506	struct vfio_dma_region curr, next;
507
508	if (device->iommufd)
509	ret = iommufd_dma_unmap(device->iommufd, 0, UINT64_MAX,
510	device->ioas_id, unmapped);
511	else
512	ret = vfio_iommu_dma_unmap(device->container_fd, 0, 0,
513	VFIO_DMA_UNMAP_FLAG_ALL, unmapped);
514
515	if (ret)
516	return ret;
517
518	list_for_each_entry_safe(curr, next, &device->dma_regions, link)
519	list_del_init(&curr->link);
520
521	return 0;
522	}
523
524	static void vfio_pci_region_get(struct vfio_pci_device *device, int index,	113	static void vfio_pci_region_get(struct vfio_pci_device *device, int index,
525	struct vfio_region_info *info)	114	struct vfio_region_info *info)
526	{	115	{
@@ -627,28 +216,26 @@ static void vfio_pci_group_setup(struct vfio_pci_device device, const char bdf · 100 unmodified lines
627	ioctl_assert(device->group_fd, VFIO_GROUP_GET_STATUS, &group_status);	216	ioctl_assert(device->group_fd, VFIO_GROUP_GET_STATUS, &group_status);
628	VFIO_ASSERT_TRUE(group_status.flags & VFIO_GROUP_FLAGS_VIABLE);	217	VFIO_ASSERT_TRUE(group_status.flags & VFIO_GROUP_FLAGS_VIABLE);
629		218
630	ioctl_assert(device->group_fd, VFIO_GROUP_SET_CONTAINER, &device->container_fd);	219	ioctl_assert(device->group_fd, VFIO_GROUP_SET_CONTAINER, &device->iommu->container_fd);
631	}	220	}
632		221
633	static void vfio_pci_container_setup(struct vfio_pci_device device, const char bdf)	222	static void vfio_pci_container_setup(struct vfio_pci_device device, const char bdf)
634	{	223	{
635	unsigned long iommu_type = device->iommu_mode->iommu_type;	224	struct iommu *iommu = device->iommu;
636	const char *path = device->iommu_mode->container_path;	225	unsigned long iommu_type = iommu->mode->iommu_type;
637	int version;
638	int ret;	226	int ret;
639		227
640	device->container_fd = open(path, O_RDWR);
641	VFIO_ASSERT_GE(device->container_fd, 0, "open(%s) failed\n", path);
642
643	version = ioctl(device->container_fd, VFIO_GET_API_VERSION);
644	VFIO_ASSERT_EQ(version, VFIO_API_VERSION, "Unsupported version: %d\n", version);
645
646	vfio_pci_group_setup(device, bdf);	228	vfio_pci_group_setup(device, bdf);
647		229
648	ret = ioctl(device->container_fd, VFIO_CHECK_EXTENSION, iommu_type);	230	ret = ioctl(iommu->container_fd, VFIO_CHECK_EXTENSION, iommu_type);
649	VFIO_ASSERT_GT(ret, 0, "VFIO IOMMU type %lu not supported\n", iommu_type);	231	VFIO_ASSERT_GT(ret, 0, "VFIO IOMMU type %lu not supported\n", iommu_type);
650		232
651	ioctl_assert(device->container_fd, VFIO_SET_IOMMU, (void *)iommu_type);	233	/*
		234	* Allow multiple threads to race to set the IOMMU type on the
		235	* container. The first will succeed and the rest should fail
		236	* because the IOMMU type is already set.
		237	*/
		238	(void)ioctl(iommu->container_fd, VFIO_SET_IOMMU, (void *)iommu_type);
652		239
653	device->fd = ioctl(device->group_fd, VFIO_GROUP_GET_DEVICE_FD, bdf);	240	device->fd = ioctl(device->group_fd, VFIO_GROUP_GET_DEVICE_FD, bdf);
654	VFIO_ASSERT_GE(device->fd, 0);	241	VFIO_ASSERT_GE(device->fd, 0);
@@ -712,52 +299,6 @@ const char vfio_pci_get_cdev_path(const char bdf) · 57 unmodified lines
712	return cdev_path;	299	return cdev_path;
713	}	300	}
714		301
715	/* Reminder: Keep in sync with FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES(). */
716	static const struct vfio_iommu_mode iommu_modes[] = {
717	{
718	.name = "vfio_type1_iommu",
719	.container_path = "/dev/vfio/vfio",
720	.iommu_type = VFIO_TYPE1_IOMMU,
721	},
722	{
723	.name = "vfio_type1v2_iommu",
724	.container_path = "/dev/vfio/vfio",
725	.iommu_type = VFIO_TYPE1v2_IOMMU,
726	},
727	{
728	.name = "iommufd_compat_type1",
729	.container_path = "/dev/iommu",
730	.iommu_type = VFIO_TYPE1_IOMMU,
731	},
732	{
733	.name = "iommufd_compat_type1v2",
734	.container_path = "/dev/iommu",
735	.iommu_type = VFIO_TYPE1v2_IOMMU,
736	},
737	{
738	.name = "iommufd",
739	},
740	};
741
742	const char *default_iommu_mode = "iommufd";
743
744	static const struct vfio_iommu_mode lookup_iommu_mode(const char iommu_mode)
745	{
746	int i;
747
748	if (!iommu_mode)
749	iommu_mode = default_iommu_mode;
750
751	for (i = 0; i < ARRAY_SIZE(iommu_modes); i++) {
752	if (strcmp(iommu_mode, iommu_modes[i].name))
753	continue;
754
755	return &iommu_modes[i];
756	}
757
758	VFIO_FAIL("Unrecognized IOMMU mode: %s\n", iommu_mode);
759	}
760
761	static void vfio_device_bind_iommufd(int device_fd, int iommufd)	302	static void vfio_device_bind_iommufd(int device_fd, int iommufd)
762	{	303	{
763	struct vfio_device_bind_iommufd args = {	304	struct vfio_device_bind_iommufd args = {
@@ -768,16 +309,6 @@ static void vfio_device_bind_iommufd(int device_fd, int iommufd) · 4 unmodified lines
768	ioctl_assert(device_fd, VFIO_DEVICE_BIND_IOMMUFD, &args);	309	ioctl_assert(device_fd, VFIO_DEVICE_BIND_IOMMUFD, &args);
769	}	310	}
770		311
771	static u32 iommufd_ioas_alloc(int iommufd)
772	{
773	struct iommu_ioas_alloc args = {
774	.size = sizeof(args),
775	};
776
777	ioctl_assert(iommufd, IOMMU_IOAS_ALLOC, &args);
778	return args.out_ioas_id;
779	}
780
781	static void vfio_device_attach_iommufd_pt(int device_fd, u32 pt_id)	312	static void vfio_device_attach_iommufd_pt(int device_fd, u32 pt_id)
782	{	313	{
783	struct vfio_device_attach_iommufd_pt args = {	314	struct vfio_device_attach_iommufd_pt args = {
@@ -796,31 +327,22 @@ static void vfio_pci_iommufd_setup(struct vfio_pci_device device, const char b · 12 unmodified lines
796	VFIO_ASSERT_GE(device->fd, 0);	327	VFIO_ASSERT_GE(device->fd, 0);
797	free((void *)cdev_path);	328	free((void *)cdev_path);
798		329
799	/*	330	vfio_device_bind_iommufd(device->fd, device->iommu->iommufd);
800	* Require device->iommufd to be >0 so that a simple non-0 check can be	331	vfio_device_attach_iommufd_pt(device->fd, device->iommu->ioas_id);
801	* used to check if iommufd is enabled. In practice open() will never
802	* return 0 unless stdin is closed.
803	*/
804	device->iommufd = open("/dev/iommu", O_RDWR);
805	VFIO_ASSERT_GT(device->iommufd, 0);
806
807	vfio_device_bind_iommufd(device->fd, device->iommufd);
808	device->ioas_id = iommufd_ioas_alloc(device->iommufd);
809	vfio_device_attach_iommufd_pt(device->fd, device->ioas_id);
810	}	332	}
811		333
812	struct vfio_pci_device vfio_pci_device_init(const char bdf, const char *iommu_mode)	334	struct vfio_pci_device vfio_pci_device_init(const char bdf, struct iommu *iommu)
813	{	335	{
814	struct vfio_pci_device *device;	336	struct vfio_pci_device *device;
815		337
816	device = calloc(1, sizeof(*device));	338	device = calloc(1, sizeof(*device));
817	VFIO_ASSERT_NOT_NULL(device);	339	VFIO_ASSERT_NOT_NULL(device);
818		340
819	INIT_LIST_HEAD(&device->dma_regions);	341	VFIO_ASSERT_NOT_NULL(iommu);
820		342	device->iommu = iommu;
821	device->iommu_mode = lookup_iommu_mode(iommu_mode);	343	device->bdf = bdf;
822		344
823	if (device->iommu_mode->container_path)	345	if (iommu->mode->container_path)
824	vfio_pci_container_setup(device, bdf);	346	vfio_pci_container_setup(device, bdf);
825	else	347	else
826	vfio_pci_iommufd_setup(device, bdf);	348	vfio_pci_iommufd_setup(device, bdf);
@@ -849,48 +371,8 @@ void vfio_pci_device_cleanup(struct vfio_pci_device *device) · 22 unmodified lines
849	VFIO_ASSERT_EQ(close(device->msi_eventfds[i]), 0);	371	VFIO_ASSERT_EQ(close(device->msi_eventfds[i]), 0);
850	}	372	}
851		373
852	if (device->iommufd) {	374	if (device->group_fd)
853	VFIO_ASSERT_EQ(close(device->iommufd), 0);
854	} else {
855	VFIO_ASSERT_EQ(close(device->group_fd), 0);	375	VFIO_ASSERT_EQ(close(device->group_fd), 0);
856	VFIO_ASSERT_EQ(close(device->container_fd), 0);
857	}
858		376
859	free(device);	377	free(device);
860	}	378	}
861
862	static bool is_bdf(const char *str)
863	{
864	unsigned int s, b, d, f;
865	int length, count;
866
867	count = sscanf(str, "%4x:%2x:%2x.%2x%n", &s, &b, &d, &f, &length);
868	return count == 4 && length == strlen(str);
869	}
870
871	const char vfio_selftests_get_bdf(int argc, char *argv[])
872	{
873	char *bdf;
874
875	if (argc > 1 && is_bdf(argv[argc - 1]))
876	return argv[--(*argc)];
877
878	bdf = getenv("VFIO_SELFTESTS_BDF");
879	if (bdf) {
880	VFIO_ASSERT_TRUE(is_bdf(bdf), "Invalid BDF: %s\n", bdf);
881	return bdf;
882	}
883
884	fprintf(stderr, "Unable to determine which device to use, skipping test.\n");
885	fprintf(stderr, "\n");
886	fprintf(stderr, "To pass the device address via environment variable:\n");
887	fprintf(stderr, "\n");
888	fprintf(stderr, " export VFIO_SELFTESTS_BDF=segment:bus:device.function\n");
889	fprintf(stderr, " %s [options]\n", argv[0]);
890	fprintf(stderr, "\n");
891	fprintf(stderr, "To pass the device address via argv:\n");
892	fprintf(stderr, "\n");
893	fprintf(stderr, " %s [options] segment:bus:device.function\n", argv[0]);
894	fprintf(stderr, "\n");
895	exit(KSFT_SKIP);
896	}

tools/testing/selftests/vfio/lib/vfio_pci_driver.c+1−15

 // SPDX-License-Identifier: GPL-2.0-only
-#include <stdio.h>
 #include "../../../kselftest.h"
-#include <vfio_util.h>
+#include <libvfio.h>
 #ifdef __x86_64__
 extern struct vfio_pci_driver_ops dsa_ops;
 		if (ops->probe(device))
 			continue;
-		printf("Driver found: %s\n", ops->name);
 		device->driver.ops = ops;
 	}
 }
 	driver->ops->init(device);
 	driver->initialized = true;
-	printf("%s: region: vaddr %p, iova 0x%lx, size 0x%lx\n",
-	       driver->ops->name,
-	       driver->region.vaddr,
-	       driver->region.iova,
-	       driver->region.size);
-	printf("%s: max_memcpy_size 0x%lx, max_memcpy_count 0x%lx\n",
-	       driver->ops->name,
-	       driver->max_memcpy_size,
-	       driver->max_memcpy_count);
 }
 void vfio_pci_driver_remove(struct vfio_pci_device *device)

tools/testing/selftests/vfio/run.sh+0−109

@@ -1,109 +0,0 @@
1	# SPDX-License-Identifier: GPL-2.0-or-later
2
3	# Global variables initialized in main() and then used during cleanup() when
4	# the script exits.
5	declare DEVICE_BDF
6	declare NEW_DRIVER
7	declare OLD_DRIVER
8	declare OLD_NUMVFS
9	declare DRIVER_OVERRIDE
10
11	function write_to() {
12	# Unfortunately set -x does not show redirects so use echo to manually
13	# tell the user what commands are being run.
14	echo "+ echo \"${2}\" > ${1}"
15	echo "${2}" > ${1}
16	}
17
18	function bind() {
19	write_to /sys/bus/pci/drivers/${2}/bind ${1}
20	}
21
22	function unbind() {
23	write_to /sys/bus/pci/drivers/${2}/unbind ${1}
24	}
25
26	function set_sriov_numvfs() {
27	write_to /sys/bus/pci/devices/${1}/sriov_numvfs ${2}
28	}
29
30	function set_driver_override() {
31	write_to /sys/bus/pci/devices/${1}/driver_override ${2}
32	}
33
34	function clear_driver_override() {
35	set_driver_override ${1} ""
36	}
37
38	function cleanup() {
39	if [ "${NEW_DRIVER}" ]; then unbind ${DEVICE_BDF} ${NEW_DRIVER} ; fi
40	if [ "${DRIVER_OVERRIDE}" ]; then clear_driver_override ${DEVICE_BDF} ; fi
41	if [ "${OLD_DRIVER}" ]; then bind ${DEVICE_BDF} ${OLD_DRIVER} ; fi
42	if [ "${OLD_NUMVFS}" ]; then set_sriov_numvfs ${DEVICE_BDF} ${OLD_NUMVFS} ; fi
43	}
44
45	function usage() {
46	echo "usage: $0 [-d segment:bus:device.function] [-s] [-h] [cmd ...]" >&2
47	echo >&2
48	echo " -d: The BDF of the device to use for the test (required)" >&2
49	echo " -h: Show this help message" >&2
50	echo " -s: Drop into a shell rather than running a command" >&2
51	echo >&2
52	echo " cmd: The command to run and arguments to pass to it." >&2
53	echo " Required when not using -s. The SBDF will be " >&2
54	echo " appended to the argument list." >&2
55	exit 1
56	}
57
58	function main() {
59	local shell
60
61	while getopts "d:hs" opt; do
62	case $opt in
63	d) DEVICE_BDF="$OPTARG" ;;
64	s) shell=true ;;
65	*) usage ;;
66	esac
67	done
68
69	# Shift past all optional arguments.
70	shift $((OPTIND - 1))
71
72	# Check that the user passed in the command to run.
73	[ ! "${shell}" ] && [ $# = 0 ] && usage
74
75	# Check that the user passed in a BDF.
76	[ "${DEVICE_BDF}" ] \|\| usage
77
78	trap cleanup EXIT
79	set -e
80
81	test -d /sys/bus/pci/devices/${DEVICE_BDF}
82
83	if [ -f /sys/bus/pci/devices/${DEVICE_BDF}/sriov_numvfs ]; then
84	OLD_NUMVFS=$(cat /sys/bus/pci/devices/${DEVICE_BDF}/sriov_numvfs)
85	set_sriov_numvfs ${DEVICE_BDF} 0
86	fi
87
88	if [ -L /sys/bus/pci/devices/${DEVICE_BDF}/driver ]; then
89	OLD_DRIVER=$(basename $(readlink -m /sys/bus/pci/devices/${DEVICE_BDF}/driver))
90	unbind ${DEVICE_BDF} ${OLD_DRIVER}
91	fi
92
93	set_driver_override ${DEVICE_BDF} vfio-pci
94	DRIVER_OVERRIDE=true
95
96	bind ${DEVICE_BDF} vfio-pci
97	NEW_DRIVER=vfio-pci
98
99	echo
100	if [ "${shell}" ]; then
101	echo "Dropping into ${SHELL} with VFIO_SELFTESTS_BDF=${DEVICE_BDF}"
102	VFIO_SELFTESTS_BDF=${DEVICE_BDF} ${SHELL}
103	else
104	"$@" ${DEVICE_BDF}
105	fi
106	echo
107	}
108
109	main "$@"

tools/testing/selftests/vfio/scripts/cleanup.sh+41−0

@@ -0,0 +1,41 @@
	1	# SPDX-License-Identifier: GPL-2.0-or-later
	2
	3	source $(dirname -- "${BASH_SOURCE[0]}")/lib.sh
	4
	5	function cleanup_devices() {
	6	local device_bdf
	7	local device_dir
	8
	9	for device_bdf in "$@"; do
	10	device_dir=${DEVICES_DIR}/${device_bdf}
	11
	12	if [ -f ${device_dir}/vfio-pci ]; then
	13	unbind ${device_bdf} vfio-pci
	14	fi
	15
	16	if [ -f ${device_dir}/driver_override ]; then
	17	clear_driver_override ${device_bdf}
	18	fi
	19
	20	if [ -f ${device_dir}/driver ]; then
	21	bind ${device_bdf} $(cat ${device_dir}/driver)
	22	fi
	23
	24	if [ -f ${device_dir}/sriov_numvfs ]; then
	25	set_sriov_numvfs ${device_bdf} $(cat ${device_dir}/sriov_numvfs)
	26	fi
	27
	28	rm -rf ${device_dir}
	29	done
	30	}
	31
	32	function main() {
	33	if [ $# = 0 ]; then
	34	cleanup_devices $(ls ${DEVICES_DIR})
	35	rmdir ${DEVICES_DIR}
	36	else
	37	cleanup_devices "$@"
	38	fi
	39	}
	40
	41	main "$@"

tools/testing/selftests/vfio/scripts/lib.sh+42−0

@@ -0,0 +1,42 @@
	1	# SPDX-License-Identifier: GPL-2.0-or-later
	2
	3	readonly DEVICES_DIR="${TMPDIR:-/tmp}/vfio-selftests-devices"
	4
	5	function write_to() {
	6	# Unfortunately set -x does not show redirects so use echo to manually
	7	# tell the user what commands are being run.
	8	echo "+ echo \"${2}\" > ${1}"
	9	echo "${2}" > ${1}
	10	}
	11
	12	function get_driver() {
	13	if [ -L /sys/bus/pci/devices/${1}/driver ]; then
	14	basename $(readlink -m /sys/bus/pci/devices/${1}/driver)
	15	fi
	16	}
	17
	18	function bind() {
	19	write_to /sys/bus/pci/drivers/${2}/bind ${1}
	20	}
	21
	22	function unbind() {
	23	write_to /sys/bus/pci/drivers/${2}/unbind ${1}
	24	}
	25
	26	function set_sriov_numvfs() {
	27	write_to /sys/bus/pci/devices/${1}/sriov_numvfs ${2}
	28	}
	29
	30	function get_sriov_numvfs() {
	31	if [ -f /sys/bus/pci/devices/${1}/sriov_numvfs ]; then
	32	cat /sys/bus/pci/devices/${1}/sriov_numvfs
	33	fi
	34	}
	35
	36	function set_driver_override() {
	37	write_to /sys/bus/pci/devices/${1}/driver_override ${2}
	38	}
	39
	40	function clear_driver_override() {
	41	set_driver_override ${1} ""
	42	}

tools/testing/selftests/vfio/scripts/run.sh+16−0

@@ -0,0 +1,16 @@
	1	# SPDX-License-Identifier: GPL-2.0-or-later
	2
	3	source $(dirname -- "${BASH_SOURCE[0]}")/lib.sh
	4
	5	function main() {
	6	local device_bdfs=$(ls ${DEVICES_DIR})
	7
	8	if [ -z "${device_bdfs}" ]; then
	9	echo "No devices found, skipping."
	10	exit 4
	11	fi
	12
	13	"$@" ${device_bdfs}
	14	}
	15
	16	main "$@"

tools/testing/selftests/vfio/scripts/setup.sh+48−0

@@ -0,0 +1,48 @@
	1	# SPDX-License-Identifier: GPL-2.0-or-later
	2	set -e
	3
	4	source $(dirname -- "${BASH_SOURCE[0]}")/lib.sh
	5
	6	function main() {
	7	local device_bdf
	8	local device_dir
	9	local numvfs
	10	local driver
	11
	12	if [ $# = 0 ]; then
	13	echo "usage: $0 segment:bus:device.function ..." >&2
	14	exit 1
	15	fi
	16
	17	for device_bdf in "$@"; do
	18	test -d /sys/bus/pci/devices/${device_bdf}
	19
	20	device_dir=${DEVICES_DIR}/${device_bdf}
	21	if [ -d "${device_dir}" ]; then
	22	echo "${device_bdf} has already been set up, exiting."
	23	exit 0
	24	fi
	25
	26	mkdir -p ${device_dir}
	27
	28	numvfs=$(get_sriov_numvfs ${device_bdf})
	29	if [ "${numvfs}" ]; then
	30	set_sriov_numvfs ${device_bdf} 0
	31	echo ${numvfs} > ${device_dir}/sriov_numvfs
	32	fi
	33
	34	driver=$(get_driver ${device_bdf})
	35	if [ "${driver}" ]; then
	36	unbind ${device_bdf} ${driver}
	37	echo ${driver} > ${device_dir}/driver
	38	fi
	39
	40	set_driver_override ${device_bdf} vfio-pci
	41	touch ${device_dir}/driver_override
	42
	43	bind ${device_bdf} vfio-pci
	44	touch ${device_dir}/vfio-pci
	45	done
	46	}
	47
	48	main "$@"

tools/testing/selftests/vfio/vfio_dma_mapping_test.c+26−20

 #include <linux/sizes.h>
 #include <linux/vfio.h>
-#include <vfio_util.h>
+#include <libvfio.h>
 #include "../kselftest_harness.h"
 }
 FIXTURE(vfio_dma_mapping_test) {
+	struct iommu *iommu;
 	struct vfio_pci_device *device;
 	struct iova_allocator *iova_allocator;
 };
 FIXTURE_SETUP(vfio_dma_mapping_test)
 {
-	self->device = vfio_pci_device_init(device_bdf, variant->iommu_mode);
+	self->iommu = iommu_init(variant->iommu_mode);
-	self->iova_allocator = iova_allocator_init(self->device);
+	self->device = vfio_pci_device_init(device_bdf, self->iommu);
+	self->iova_allocator = iova_allocator_init(self->iommu);
 }
 FIXTURE_TEARDOWN(vfio_dma_mapping_test)
 {
 	iova_allocator_cleanup(self->iova_allocator);
 	vfio_pci_device_cleanup(self->device);
+	iommu_cleanup(self->iommu);
 }
 TEST_F(vfio_dma_mapping_test, dma_map_unmap)
 {
 	const u64 size = variant->size ?: getpagesize();
 	const int flags = variant->mmap_flags;
-	struct vfio_dma_region region;
+	struct dma_region region;
 	struct iommu_mapping mapping;
 	u64 mapping_size = size;
 	u64 unmapped;
 	region.iova = iova_allocator_alloc(self->iova_allocator, size);
 	region.size = size;
-	vfio_pci_dma_map(self->device, &region);
+	iommu_map(self->iommu, &region);
 	printf("Mapped HVA %p (size 0x%lx) at IOVA 0x%lx\n", region.vaddr, size, region.iova);
 	ASSERT_EQ(region.iova, to_iova(self->device, region.vaddr));
 	}
 unmap:
-	rc = __vfio_pci_dma_unmap(self->device, &region, &unmapped);
+	rc = __iommu_unmap(self->iommu, &region, &unmapped);
 	ASSERT_EQ(rc, 0);
 	ASSERT_EQ(unmapped, region.size);
 	printf("Unmapped IOVA 0x%lx\n", region.iova);
-	ASSERT_EQ(INVALID_IOVA, __to_iova(self->device, region.vaddr));
+	ASSERT_NE(0, __to_iova(self->device, region.vaddr, NULL));
 	ASSERT_NE(0, iommu_mapping_get(device_bdf, region.iova, &mapping));
 	ASSERT_TRUE(!munmap(region.vaddr, size));
 }
 FIXTURE(vfio_dma_map_limit_test) {
+	struct iommu *iommu;
 	struct vfio_pci_device *device;
-	struct vfio_dma_region region;
+	struct dma_region region;
 	size_t mmap_size;
 };
 FIXTURE_SETUP(vfio_dma_map_limit_test)
 {
-	struct vfio_dma_region *region = &self->region;
+	struct dma_region *region = &self->region;
 	struct iommu_iova_range *ranges;
 	u64 region_size = getpagesize();
 	iova_t last_iova;
 	 */
 	self->mmap_size = 2 * region_size;
-	self->device = vfio_pci_device_init(device_bdf, variant->iommu_mode);
+	self->iommu = iommu_init(variant->iommu_mode);
+	self->device = vfio_pci_device_init(device_bdf, self->iommu);
 	region->vaddr = mmap(NULL, self->mmap_size, PROT_READ | PROT_WRITE,
 			     MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
 	ASSERT_NE(region->vaddr, MAP_FAILED);
-	ranges = vfio_pci_iova_ranges(self->device, &nranges);
+	ranges = iommu_iova_ranges(self->iommu, &nranges);
 	VFIO_ASSERT_NOT_NULL(ranges);
 	last_iova = ranges[nranges - 1].last;
 	free(ranges);
 FIXTURE_TEARDOWN(vfio_dma_map_limit_test)
 {
 	vfio_pci_device_cleanup(self->device);
+	iommu_cleanup(self->iommu);
 	ASSERT_EQ(munmap(self->region.vaddr, self->mmap_size), 0);
 }
 TEST_F(vfio_dma_map_limit_test, unmap_range)
 {
-	struct vfio_dma_region *region = &self->region;
+	struct dma_region *region = &self->region;
 	u64 unmapped;
 	int rc;
-	vfio_pci_dma_map(self->device, region);
+	iommu_map(self->iommu, region);
 	ASSERT_EQ(region->iova, to_iova(self->device, region->vaddr));
-	rc = __vfio_pci_dma_unmap(self->device, region, &unmapped);
+	rc = __iommu_unmap(self->iommu, region, &unmapped);
 	ASSERT_EQ(rc, 0);
 	ASSERT_EQ(unmapped, region->size);
 }
 TEST_F(vfio_dma_map_limit_test, unmap_all)
 {
-	struct vfio_dma_region *region = &self->region;
+	struct dma_region *region = &self->region;
 	u64 unmapped;
 	int rc;
-	vfio_pci_dma_map(self->device, region);
+	iommu_map(self->iommu, region);
 	ASSERT_EQ(region->iova, to_iova(self->device, region->vaddr));
-	rc = __vfio_pci_dma_unmap_all(self->device, &unmapped);
+	rc = __iommu_unmap_all(self->iommu, &unmapped);
 	ASSERT_EQ(rc, 0);
 	ASSERT_EQ(unmapped, region->size);
 }
 TEST_F(vfio_dma_map_limit_test, overflow)
 {
-	struct vfio_dma_region *region = &self->region;
+	struct dma_region *region = &self->region;
 	int rc;
 	region->iova = ~(iova_t)0 & ~(region->size - 1);
 	region->size = self->mmap_size;
-	rc = __vfio_pci_dma_map(self->device, region);
+	rc = __iommu_map(self->iommu, region);
 	ASSERT_EQ(rc, -EOVERFLOW);
-	rc = __vfio_pci_dma_unmap(self->device, region, NULL);
+	rc = __iommu_unmap(self->iommu, region, NULL);
 	ASSERT_EQ(rc, -EOVERFLOW);
 }

tools/testing/selftests/vfio/vfio_iommufd_setup_test.c+1−1

@@ -10,7 +10,7 @@ · 9 unmodified lines
10	#include <sys/ioctl.h>	10	#include <sys/ioctl.h>
11	#include <unistd.h>	11	#include <unistd.h>
12		12
13	#include <vfio_util.h>	13	#include <libvfio.h>
14	#include "../kselftest_harness.h"	14	#include "../kselftest_harness.h"
15		15
16	static const char iommu_dev_path[] = "/dev/iommu";	16	static const char iommu_dev_path[] = "/dev/iommu";

tools/testing/selftests/vfio/vfio_pci_device_init_perf_test.c+168−0

@@ -0,0 +1,168 @@
	1	// SPDX-License-Identifier: GPL-2.0-only
	2	#include <pthread.h>
	3	#include <sys/ioctl.h>
	4	#include <sys/mman.h>
	5
	6	#include <linux/sizes.h>
	7	#include <linux/time64.h>
	8	#include <linux/vfio.h>
	9
	10	#include <libvfio.h>
	11
	12	#include "../kselftest_harness.h"
	13
	14	static char **device_bdfs;
	15	static int nr_devices;
	16
	17	struct thread_args {
	18	struct iommu *iommu;
	19	int device_index;
	20	struct timespec start;
	21	struct timespec end;
	22	pthread_barrier_t *barrier;
	23	};
	24
	25	FIXTURE(vfio_pci_device_init_perf_test) {
	26	pthread_t *threads;
	27	pthread_barrier_t barrier;
	28	struct thread_args *thread_args;
	29	struct iommu *iommu;
	30	};
	31
	32	FIXTURE_VARIANT(vfio_pci_device_init_perf_test) {
	33	const char *iommu_mode;
	34	};
	35
	36	#define FIXTURE_VARIANT_ADD_IOMMU_MODE(_iommu_mode) \
	37	FIXTURE_VARIANT_ADD(vfio_pci_device_init_perf_test, _iommu_mode) { \
	38	.iommu_mode = #_iommu_mode, \
	39	}
	40
	41	FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES();
	42
	43	FIXTURE_SETUP(vfio_pci_device_init_perf_test)
	44	{
	45	int i;
	46
	47	self->iommu = iommu_init(variant->iommu_mode);
	48	self->threads = calloc(nr_devices, sizeof(self->threads[0]));
	49	self->thread_args = calloc(nr_devices, sizeof(self->thread_args[0]));
	50
	51	pthread_barrier_init(&self->barrier, NULL, nr_devices);
	52
	53	for (i = 0; i < nr_devices; i++) {
	54	self->thread_args[i].iommu = self->iommu;
	55	self->thread_args[i].barrier = &self->barrier;
	56	self->thread_args[i].device_index = i;
	57	}
	58	}
	59
	60	FIXTURE_TEARDOWN(vfio_pci_device_init_perf_test)
	61	{
	62	iommu_cleanup(self->iommu);
	63	free(self->threads);
	64	free(self->thread_args);
	65	}
	66
	67	static s64 to_ns(struct timespec ts)
	68	{
	69	return (s64)ts.tv_nsec + NSEC_PER_SEC * (s64)ts.tv_sec;
	70	}
	71
	72	static struct timespec to_timespec(s64 ns)
	73	{
	74	struct timespec ts = {
	75	.tv_nsec = ns % NSEC_PER_SEC,
	76	.tv_sec = ns / NSEC_PER_SEC,
	77	};
	78
	79	return ts;
	80	}
	81
	82	static struct timespec timespec_sub(struct timespec a, struct timespec b)
	83	{
	84	return to_timespec(to_ns(a) - to_ns(b));
	85	}
	86
	87	static struct timespec timespec_min(struct timespec a, struct timespec b)
	88	{
	89	return to_ns(a) < to_ns(b) ? a : b;
	90	}
	91
	92	static struct timespec timespec_max(struct timespec a, struct timespec b)
	93	{
	94	return to_ns(a) > to_ns(b) ? a : b;
	95	}
	96
	97	static void thread_main(void __args)
	98	{
	99	struct thread_args *args = __args;
	100	struct vfio_pci_device *device;
	101
	102	pthread_barrier_wait(args->barrier);
	103
	104	clock_gettime(CLOCK_MONOTONIC, &args->start);
	105	device = vfio_pci_device_init(device_bdfs[args->device_index], args->iommu);
	106	clock_gettime(CLOCK_MONOTONIC, &args->end);
	107
	108	pthread_barrier_wait(args->barrier);
	109
	110	vfio_pci_device_cleanup(device);
	111	return NULL;
	112	}
	113
	114	TEST_F(vfio_pci_device_init_perf_test, init)
	115	{
	116	struct timespec start = to_timespec(INT64_MAX), end = {};
	117	struct timespec min = to_timespec(INT64_MAX);
	118	struct timespec max = {};
	119	struct timespec avg = {};
	120	struct timespec wall_time;
	121	s64 thread_ns = 0;
	122	int i;
	123
	124	for (i = 0; i < nr_devices; i++) {
	125	pthread_create(&self->threads[i], NULL, thread_main,
	126	&self->thread_args[i]);
	127	}
	128
	129	for (i = 0; i < nr_devices; i++) {
	130	struct thread_args *args = &self->thread_args[i];
	131	struct timespec init_time;
	132
	133	pthread_join(self->threads[i], NULL);
	134
	135	start = timespec_min(start, args->start);
	136	end = timespec_max(end, args->end);
	137
	138	init_time = timespec_sub(args->end, args->start);
	139	min = timespec_min(min, init_time);
	140	max = timespec_max(max, init_time);
	141	thread_ns += to_ns(init_time);
	142	}
	143
	144	avg = to_timespec(thread_ns / nr_devices);
	145	wall_time = timespec_sub(end, start);
	146
	147	printf("Wall time: %lu.%09lus\n",
	148	wall_time.tv_sec, wall_time.tv_nsec);
	149	printf("Min init time (per device): %lu.%09lus\n",
	150	min.tv_sec, min.tv_nsec);
	151	printf("Max init time (per device): %lu.%09lus\n",
	152	max.tv_sec, max.tv_nsec);
	153	printf("Avg init time (per device): %lu.%09lus\n",
	154	avg.tv_sec, avg.tv_nsec);
	155	}
	156
	157	int main(int argc, char *argv[])
	158	{
	159	int i;
	160
	161	device_bdfs = vfio_selftests_get_bdfs(&argc, argv, &nr_devices);
	162
	163	printf("Testing parallel initialization of %d devices:\n", nr_devices);
	164	for (i = 0; i < nr_devices; i++)
	165	printf(" %s\n", device_bdfs[i]);
	166
	167	return test_harness_run(argc, argv);
	168	}

tools/testing/selftests/vfio/vfio_pci_device_test.c+9−3

 #include <linux/sizes.h>
 #include <linux/vfio.h>
-#include <vfio_util.h>
+#include <libvfio.h>
 #include "../kselftest_harness.h"
 #define MAX_TEST_MSI 16U
 FIXTURE(vfio_pci_device_test) {
+	struct iommu *iommu;
 	struct vfio_pci_device *device;
 };
 FIXTURE_SETUP(vfio_pci_device_test)
 {
-	self->device = vfio_pci_device_init(device_bdf, default_iommu_mode);
+	self->iommu = iommu_init(default_iommu_mode);
+	self->device = vfio_pci_device_init(device_bdf, self->iommu);
 }
 FIXTURE_TEARDOWN(vfio_pci_device_test)
 {
 	vfio_pci_device_cleanup(self->device);
+	iommu_cleanup(self->iommu);
 }
 #define read_pci_id_from_sysfs(_file) ({							\
 }
 FIXTURE(vfio_pci_irq_test) {
+	struct iommu *iommu;
 	struct vfio_pci_device *device;
 };
 FIXTURE_SETUP(vfio_pci_irq_test)
 {
-	self->device = vfio_pci_device_init(device_bdf, default_iommu_mode);
+	self->iommu = iommu_init(default_iommu_mode);
+	self->device = vfio_pci_device_init(device_bdf, self->iommu);
 }
 FIXTURE_TEARDOWN(vfio_pci_irq_test)
 {
 	vfio_pci_device_cleanup(self->device);
+	iommu_cleanup(self->iommu);
 }
 TEST_F(vfio_pci_irq_test, enable_trigger_disable)

tools/testing/selftests/vfio/vfio_pci_driver_test.c+33−18

 #include <linux/sizes.h>
 #include <linux/vfio.h>
-#include <vfio_util.h>
+#include <libvfio.h>
 #include "../kselftest_harness.h"
 	ASSERT_EQ(EAGAIN, errno);			\
 } while (0)
-static void region_setup(struct vfio_pci_device *device,
+static void region_setup(struct iommu *iommu,
 			 struct iova_allocator *iova_allocator,
-			 struct vfio_dma_region *region, u64 size)
+			 struct dma_region *region, u64 size)
 {
 	const int flags = MAP_SHARED | MAP_ANONYMOUS;
 	const int prot = PROT_READ | PROT_WRITE;
 	region->iova = iova_allocator_alloc(iova_allocator, size);
 	region->size = size;
-	vfio_pci_dma_map(device, region);
+	iommu_map(iommu, region);
 }
-static void region_teardown(struct vfio_pci_device *device,
+static void region_teardown(struct iommu *iommu, struct dma_region *region)
-			    struct vfio_dma_region *region)
 {
-	vfio_pci_dma_unmap(device, region);
+	iommu_unmap(iommu, region);
 	VFIO_ASSERT_EQ(munmap(region->vaddr, region->size), 0);
 }
 FIXTURE(vfio_pci_driver_test) {
+	struct iommu *iommu;
 	struct vfio_pci_device *device;
 	struct iova_allocator *iova_allocator;
-	struct vfio_dma_region memcpy_region;
+	struct dma_region memcpy_region;
 	void *vaddr;
 	int msi_fd;
 {
 	struct vfio_pci_driver *driver;
-	self->device = vfio_pci_device_init(device_bdf, variant->iommu_mode);
+	self->iommu = iommu_init(variant->iommu_mode);
-	self->iova_allocator = iova_allocator_init(self->device);
+	self->device = vfio_pci_device_init(device_bdf, self->iommu);
+	self->iova_allocator = iova_allocator_init(self->iommu);
 	driver = &self->device->driver;
-	region_setup(self->device, self->iova_allocator, &self->memcpy_region, SZ_1G);
+	region_setup(self->iommu, self->iova_allocator, &self->memcpy_region, SZ_1G);
-	region_setup(self->device, self->iova_allocator, &driver->region, SZ_2M);
+	region_setup(self->iommu, self->iova_allocator, &driver->region, SZ_2M);
 	/* Any IOVA that doesn't overlap memcpy_region and driver->region. */
 	self->unmapped_iova = iova_allocator_alloc(self->iova_allocator, SZ_1G);
 	vfio_pci_driver_remove(self->device);
-	region_teardown(self->device, &self->memcpy_region);
+	region_teardown(self->iommu, &self->memcpy_region);
-	region_teardown(self->device, &driver->region);
+	region_teardown(self->iommu, &driver->region);
 	iova_allocator_cleanup(self->iova_allocator);
 	vfio_pci_device_cleanup(self->device);
+	iommu_cleanup(self->iommu);
 }
 TEST_F(vfio_pci_driver_test, init_remove)
 	ASSERT_NO_MSI(self->msi_fd);
 }
-int main(int argc, char *argv[])
+static bool device_has_selftests_driver(const char *bdf)
 {
 	struct vfio_pci_device *device;
+	struct iommu *iommu;
+	bool has_driver;
+	iommu = iommu_init(default_iommu_mode);
+	device = vfio_pci_device_init(device_bdf, iommu);
+	has_driver = !!device->driver.ops;
+	vfio_pci_device_cleanup(device);
+	iommu_cleanup(iommu);
+	return has_driver;
+}
+int main(int argc, char *argv[])
+{
 	device_bdf = vfio_selftests_get_bdf(&argc, argv);
-	device = vfio_pci_device_init(device_bdf, default_iommu_mode);
+	if (!device_has_selftests_driver(device_bdf)) {
-	if (!device->driver.ops) {
 		fprintf(stderr, "No driver found for device %s\n", device_bdf);
 		return KSFT_SKIP;
 	}
-	vfio_pci_device_cleanup(device);
 	return test_harness_run(argc, argv);
 }