Merge branch 'remotes/lorenzo/pci/endpoint'

- Use notification chain instead of EPF linkup ops for EPC events (Kishon Vijay Abraham I) - Protect concurrent allocation in endpoint outbound address region (Kishon Vijay Abraham I) - Protect concurrent access to pci_epf_ops (Kishon Vijay Abraham I) - Assign function number for each PF in endpoint core (Kishon Vijay Abraham I) - Refactor endpoint mode core initialization (Vidya Sagar) - Add API to notify when core initialization completes (Vidya Sagar) - Add test framework support to defer core initialization (Vidya Sagar) - Update Tegra SoC ABI header to support uninitialization of UPHY PLL when in endpoint mode without reference clock (Vidya Sagar) - Add DT and driver support for Tegra194 PCIe endpoint nodes (Vidya Sagar) - Add endpoint test support for DMA data transfer (Kishon Vijay Abraham I) - Print throughput information in endpoint test (Kishon Vijay Abraham I) - Use streaming DMA APIs for endpoint test buffer allocation (Kishon Vijay Abraham I) - Add endpoint test command line option for DMA (Kishon Vijay Abraham I) - When stopping a controller via configfs, clear endpoint "start" entry to prevent WARN_ON (Kunihiko Hayashi) - Update endpoint ->set_msix() to pay attention to MSI-X BAR Indicator and offset when finding MSI-X tables (Kishon Vijay Abraham I) - MSI-X tables are in local memory, not in the PCI address space. Update pcie-designware-ep to account for this (Kishon Vijay Abraham I) - Allow AM654 PCIe Endpoint to raise MSI-X interrupts (Kishon Vijay Abraham I) - Avoid using module parameter to determine irqtype for endpoint test (Kishon Vijay Abraham I) - Add ioctl to clear IRQ for endpoint test (Kishon Vijay Abraham I) - Add endpoint test 'e' option to clear IRQ (Kishon Vijay Abraham I) - Bump limit on number of endpoint test devices from 10 to 10,000 (Kishon Vijay Abraham I) - Use full pci-endpoint-test name in request_irq() for easier profiling (Kishon Vijay Abraham I) - Reduce log level of -EPROBE_DEFER error messages to debug (Thierry Reding) * remotes/lorenzo/pci/endpoint: misc: pci_endpoint_test: remove duplicate macro PCI_ENDPOINT_TEST_STATUS PCI: tegra: Print -EPROBE_DEFER error message at debug level misc: pci_endpoint_test: Use full pci-endpoint-test name in request_irq() misc: pci_endpoint_test: Fix to support > 10 pci-endpoint-test devices tools: PCI: Add 'e' to clear IRQ misc: pci_endpoint_test: Add ioctl to clear IRQ misc: pci_endpoint_test: Avoid using module parameter to determine irqtype PCI: keystone: Allow AM654 PCIe Endpoint to raise MSI-X interrupt PCI: dwc: Fix dw_pcie_ep_raise_msix_irq() to get correct MSI-X table address PCI: endpoint: Fix ->set_msix() to take BIR and offset as arguments misc: pci_endpoint_test: Add support to get DMA option from userspace tools: PCI: Add 'd' command line option to support DMA misc: pci_endpoint_test: Use streaming DMA APIs for buffer allocation PCI: endpoint: functions/pci-epf-test: Print throughput information PCI: endpoint: functions/pci-epf-test: Add DMA support to transfer data PCI: endpoint: Fix clearing start entry in configfs PCI: tegra: Add support for PCIe endpoint mode in Tegra194 dt-bindings: PCI: tegra: Add DT support for PCIe EP nodes in Tegra194 soc/tegra: bpmp: Update ABI header PCI: pci-epf-test: Add support to defer core initialization PCI: dwc: Add API to notify core initialization completion PCI: endpoint: Add notification for core init completion PCI: dwc: Refactor core initialization code for EP mode PCI: endpoint: Add core init notifying feature PCI: endpoint: Assign function number for each PF in EPC core PCI: endpoint: Protect concurrent access to pci_epf_ops with mutex PCI: endpoint: Fix for concurrent memory allocation in OB address region PCI: endpoint: Replace spinlock with mutex PCI: endpoint: Use notification chain mechanism to notify EPC events to EPF

Bjorn Helgaas committed 6 years ago

Commit b16f2ab2 parents cc36a45 e48ba3e17 files · +1655 −308Browse files

Unified · Split

17 files

Documentation/devicetree/bindings/pci/nvidia,tegra194-pcie.txt +99 −26
drivers/misc/pci_endpoint_test.c +179 −34
drivers/pci/controller/dwc/Kconfig +26 −3
drivers/pci/controller/dwc/pci-keystone.c +4 −1
drivers/pci/controller/dwc/pcie-designware-ep.c +85 −59
drivers/pci/controller/dwc/pcie-designware.h +12 −0
drivers/pci/controller/dwc/pcie-tegra194.c +695 −17
drivers/pci/endpoint/functions/pci-epf-test.c +356 −46
drivers/pci/endpoint/pci-ep-cfs.c +6 −22
drivers/pci/endpoint/pci-epc-core.c +75 −62
drivers/pci/endpoint/pci-epc-mem.c +8 −2
drivers/pci/endpoint/pci-epf-core.c +13 −22
include/linux/pci-epc.h +22 −5
include/linux/pci-epf.h +25 −4
include/soc/tegra/bpmp-abi.h +9 −1
include/uapi/linux/pcitest.h +8 −0
tools/pci/pcitest.c +33 −4

Documentation/devicetree/bindings/pci/nvidia,tegra194-pcie.txt+99−26

@@ -1,11 +1,11 @@
1	NVIDIA Tegra PCIe controller (Synopsys DesignWare Core based)	1	NVIDIA Tegra PCIe controller (Synopsys DesignWare Core based)
2		2
3	This PCIe host controller is based on the Synopsis Designware PCIe IP	3	This PCIe controller is based on the Synopsis Designware PCIe IP
4	and thus inherits all the common properties defined in designware-pcie.txt.	4	and thus inherits all the common properties defined in designware-pcie.txt.
		5	Some of the controller instances are dual mode where in they can work either
		6	in root port mode or endpoint mode but one at a time.
5		7
6	Required properties:	8	Required properties:
7	- compatible: For Tegra19x, must contain "nvidia,tegra194-pcie".
8	- device_type: Must be "pci"
9	- power-domains: A phandle to the node that controls power to the respective	9	- power-domains: A phandle to the node that controls power to the respective
10	PCIe controller and a specifier name for the PCIe controller. Following are	10	PCIe controller and a specifier name for the PCIe controller. Following are
11	the specifiers for the different PCIe controllers	11	the specifiers for the different PCIe controllers
@@ -32,6 +32,32 @@ Required properties: · 20 unmodified lines
32	entry for each entry in the interrupt-names property.	32	entry for each entry in the interrupt-names property.
33	- interrupt-names: Must include the following entries:	33	- interrupt-names: Must include the following entries:
34	"intr": The Tegra interrupt that is asserted for controller interrupts	34	"intr": The Tegra interrupt that is asserted for controller interrupts
		35	- clocks: Must contain an entry for each entry in clock-names.
		36	See ../clocks/clock-bindings.txt for details.
		37	- clock-names: Must include the following entries:
		38	- core
		39	- resets: Must contain an entry for each entry in reset-names.
		40	See ../reset/reset.txt for details.
		41	- reset-names: Must include the following entries:
		42	- apb
		43	- core
		44	- phys: Must contain a phandle to P2U PHY for each entry in phy-names.
		45	- phy-names: Must include an entry for each active lane.
		46	"p2u-N": where N ranges from 0 to one less than the total number of lanes
		47	- nvidia,bpmp: Must contain a pair of phandle to BPMP controller node followed
		48	by controller-id. Following are the controller ids for each controller.
		49	0: C0
		50	1: C1
		51	2: C2
		52	3: C3
		53	4: C4
		54	5: C5
		55	- vddio-pex-ctl-supply: Regulator supply for PCIe side band signals
		56
		57	RC mode:
		58	- compatible: Tegra19x must contain "nvidia,tegra194-pcie"
		59	- device_type: Must be "pci" for RC mode
		60	- interrupt-names: Must include the following entries:
35	"msi": The Tegra interrupt that is asserted when an MSI is received	61	"msi": The Tegra interrupt that is asserted when an MSI is received
36	- bus-range: Range of bus numbers associated with this controller	62	- bus-range: Range of bus numbers associated with this controller
37	- #address-cells: Address representation for root ports (must be 3)	63	- #address-cells: Address representation for root ports (must be 3)
@@ -60,27 +86,15 @@ Required properties: · 22 unmodified lines
60	- interrupt-map-mask and interrupt-map: Standard PCI IRQ mapping properties	86	- interrupt-map-mask and interrupt-map: Standard PCI IRQ mapping properties
61	Please refer to the standard PCI bus binding document for a more detailed	87	Please refer to the standard PCI bus binding document for a more detailed
62	explanation.	88	explanation.
63	- clocks: Must contain an entry for each entry in clock-names.	89
64	See ../clocks/clock-bindings.txt for details.	90	EP mode:
65	- clock-names: Must include the following entries:	91	In Tegra194, Only controllers C0, C4 & C5 support EP mode.
66	- core	92	- compatible: Tegra19x must contain "nvidia,tegra194-pcie-ep"
67	- resets: Must contain an entry for each entry in reset-names.	93	- reg-names: Must include the following entries:
68	See ../reset/reset.txt for details.	94	"addr_space": Used to map remote RC address space
69	- reset-names: Must include the following entries:	95	- reset-gpios: Must contain a phandle to a GPIO controller followed by
70	- apb	96	GPIO that is being used as PERST input signal. Please refer to pci.txt
71	- core	97	document.
72	- phys: Must contain a phandle to P2U PHY for each entry in phy-names.
73	- phy-names: Must include an entry for each active lane.
74	"p2u-N": where N ranges from 0 to one less than the total number of lanes
75	- nvidia,bpmp: Must contain a pair of phandle to BPMP controller node followed
76	by controller-id. Following are the controller ids for each controller.
77	0: C0
78	1: C1
79	2: C2
80	3: C3
81	4: C4
82	5: C5
83	- vddio-pex-ctl-supply: Regulator supply for PCIe side band signals
84		98
85	Optional properties:	99	Optional properties:
86	- pinctrl-names: A list of pinctrl state names.	100	- pinctrl-names: A list of pinctrl state names.
@@ -104,6 +118,8 @@ Optional properties: · 17 unmodified lines
104	specified in microseconds	118	specified in microseconds
105	- nvidia,aspm-l0s-entrance-latency-us: ASPM L0s entrance latency to be	119	- nvidia,aspm-l0s-entrance-latency-us: ASPM L0s entrance latency to be
106	specified in microseconds	120	specified in microseconds
		121
		122	RC mode:
107	- vpcie3v3-supply: A phandle to the regulator node that supplies 3.3V to the slot	123	- vpcie3v3-supply: A phandle to the regulator node that supplies 3.3V to the slot
108	if the platform has one such slot. (Ex:- x16 slot owned by C5 controller	124	if the platform has one such slot. (Ex:- x16 slot owned by C5 controller
109	in p2972-0000 platform).	125	in p2972-0000 platform).
@@ -111,11 +127,18 @@ Optional properties: · 1 unmodified line
111	if the platform has one such slot. (Ex:- x16 slot owned by C5 controller	127	if the platform has one such slot. (Ex:- x16 slot owned by C5 controller
112	in p2972-0000 platform).	128	in p2972-0000 platform).
113		129
		130	EP mode:
		131	- nvidia,refclk-select-gpios: Must contain a phandle to a GPIO controller
		132	followed by GPIO that is being used to enable REFCLK to controller from host
		133
		134	NOTE:- On Tegra194's P2972-0000 platform, only C5 controller can be enabled to
		135	operate in the endpoint mode because of the way the platform is designed.
		136
114	Examples:	137	Examples:
115	=========	138	=========
116		139
117	Tegra194:	140	Tegra194 RC mode:
118	--------	141	-----------------
119		142
120	pcie@14180000 {	143	pcie@14180000 {
121	compatible = "nvidia,tegra194-pcie", "snps,dw-pcie";	144	compatible = "nvidia,tegra194-pcie", "snps,dw-pcie";
@@ -169,3 +192,53 @@ Tegra194: · 47 unmodified lines
169	<&p2u_hsio_5>;	192	<&p2u_hsio_5>;
170	phy-names = "p2u-0", "p2u-1", "p2u-2", "p2u-3";	193	phy-names = "p2u-0", "p2u-1", "p2u-2", "p2u-3";
171	};	194	};
		195
		196	Tegra194 EP mode:
		197	-----------------
		198
		199	pcie_ep@141a0000 {
		200	compatible = "nvidia,tegra194-pcie-ep", "snps,dw-pcie-ep";
		201	power-domains = <&bpmp TEGRA194_POWER_DOMAIN_PCIEX8A>;
		202	reg = <0x00 0x141a0000 0x0 0x00020000 /* appl registers (128K) */
		203	0x00 0x3a040000 0x0 0x00040000 /* iATU_DMA reg space (256K) */
		204	0x00 0x3a080000 0x0 0x00040000 /* DBI reg space (256K) */
		205	0x1c 0x00000000 0x4 0x00000000>; /* Address Space (16G) */
		206	reg-names = "appl", "atu_dma", "dbi", "addr_space";
		207
		208	num-lanes = <8>;
		209	num-ib-windows = <2>;
		210	num-ob-windows = <8>;
		211
		212	pinctrl-names = "default";
		213	pinctrl-0 = <&clkreq_c5_bi_dir_state>;
		214
		215	clocks = <&bpmp TEGRA194_CLK_PEX1_CORE_5>;
		216	clock-names = "core";
		217
		218	resets = <&bpmp TEGRA194_RESET_PEX1_CORE_5_APB>,
		219	<&bpmp TEGRA194_RESET_PEX1_CORE_5>;
		220	reset-names = "apb", "core";
		221
		222	interrupts = <GIC_SPI 53 IRQ_TYPE_LEVEL_HIGH>; /* controller interrupt */
		223	interrupt-names = "intr";
		224
		225	nvidia,bpmp = <&bpmp 5>;
		226
		227	nvidia,aspm-cmrt-us = <60>;
		228	nvidia,aspm-pwr-on-t-us = <20>;
		229	nvidia,aspm-l0s-entrance-latency-us = <3>;
		230
		231	vddio-pex-ctl-supply = <&vdd_1v8ao>;
		232
		233	reset-gpios = <&gpio TEGRA194_MAIN_GPIO(GG, 1) GPIO_ACTIVE_LOW>;
		234
		235	nvidia,refclk-select-gpios = <&gpio_aon TEGRA194_AON_GPIO(AA, 5)
		236	GPIO_ACTIVE_HIGH>;
		237
		238	phys = <&p2u_nvhs_0>, <&p2u_nvhs_1>, <&p2u_nvhs_2>,
		239	<&p2u_nvhs_3>, <&p2u_nvhs_4>, <&p2u_nvhs_5>,
		240	<&p2u_nvhs_6>, <&p2u_nvhs_7>;
		241
		242	phy-names = "p2u-0", "p2u-1", "p2u-2", "p2u-3", "p2u-4",
		243	"p2u-5", "p2u-6", "p2u-7";
		244	};

drivers/misc/pci_endpoint_test.c+179−34

 #include <linux/mutex.h>
 #include <linux/random.h>
 #include <linux/slab.h>
+#include <linux/uaccess.h>
 #include <linux/pci.h>
 #include <linux/pci_ids.h>
 #define PCI_ENDPOINT_TEST_IRQ_TYPE		0x24
 #define PCI_ENDPOINT_TEST_IRQ_NUMBER		0x28
+#define PCI_ENDPOINT_TEST_FLAGS			0x2c
+#define FLAG_USE_DMA				BIT(0)
 #define PCI_DEVICE_ID_TI_AM654			0xb00c
 #define is_am654_pci_dev(pdev)		\
 	struct completion irq_raised;
 	int		last_irq;
 	int		num_irqs;
+	int		irq_type;
 	/* mutex to protect the ioctls */
 	struct mutex	mutex;
 	struct miscdevice miscdev;
 	enum pci_barno test_reg_bar;
 	size_t alignment;
+	const char *name;
 };
 struct pci_endpoint_test_data {
 	struct pci_dev *pdev = test->pdev;
 	pci_free_irq_vectors(pdev);
+	test->irq_type = IRQ_TYPE_UNDEFINED;
 }
 static bool pci_endpoint_test_alloc_irq_vectors(struct pci_endpoint_test *test,
 		irq = 0;
 		res = false;
 	}
+	test->irq_type = type;
 	test->num_irqs = irq;
 	return res;
 	for (i = 0; i < test->num_irqs; i++) {
 		err = devm_request_irq(dev, pci_irq_vector(pdev, i),
 				       pci_endpoint_test_irqhandler,
-				       IRQF_SHARED, DRV_MODULE_NAME, test);
+				       IRQF_SHARED, test->name, test);
 		if (err)
 			goto fail;
 	}
 	return false;
 }
-static bool pci_endpoint_test_copy(struct pci_endpoint_test *test, size_t size)
+static bool pci_endpoint_test_copy(struct pci_endpoint_test *test,
+				   unsigned long arg)
 {
+	struct pci_endpoint_test_xfer_param param;
 	bool ret = false;
 	void *src_addr;
 	void *dst_addr;
+	u32 flags = 0;
+	bool use_dma;
+	size_t size;
 	dma_addr_t src_phys_addr;
 	dma_addr_t dst_phys_addr;
 	struct pci_dev *pdev = test->pdev;
 	dma_addr_t orig_dst_phys_addr;
 	size_t offset;
 	size_t alignment = test->alignment;
+	int irq_type = test->irq_type;
 	u32 src_crc32;
 	u32 dst_crc32;
+	int err;
+	err = copy_from_user(&param, (void __user *)arg, sizeof(param));
+	if (err) {
+		dev_err(dev, "Failed to get transfer param\n");
+		return false;
+	}
+	size = param.size;
 	if (size > SIZE_MAX - alignment)
 		goto err;
+	use_dma = !!(param.flags & PCITEST_FLAGS_USE_DMA);
+	if (use_dma)
+		flags |= FLAG_USE_DMA;
 	if (irq_type < IRQ_TYPE_LEGACY || irq_type > IRQ_TYPE_MSIX) {
 		dev_err(dev, "Invalid IRQ type option\n");
 		goto err;
 	}
-	orig_src_addr = dma_alloc_coherent(dev, size + alignment,
+	orig_src_addr = kzalloc(size + alignment, GFP_KERNEL);
-					   &orig_src_phys_addr, GFP_KERNEL);
 	if (!orig_src_addr) {
 		dev_err(dev, "Failed to allocate source buffer\n");
 		ret = false;
 		goto err;
 	}
+	get_random_bytes(orig_src_addr, size + alignment);
+	orig_src_phys_addr = dma_map_single(dev, orig_src_addr,
+					    size + alignment, DMA_TO_DEVICE);
+	if (dma_mapping_error(dev, orig_src_phys_addr)) {
+		dev_err(dev, "failed to map source buffer address\n");
+		ret = false;
+		goto err_src_phys_addr;
+	}
 	if (alignment && !IS_ALIGNED(orig_src_phys_addr, alignment)) {
 		src_phys_addr = PTR_ALIGN(orig_src_phys_addr, alignment);
 		offset = src_phys_addr - orig_src_phys_addr;
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_UPPER_SRC_ADDR,
 				 upper_32_bits(src_phys_addr));
-	get_random_bytes(src_addr, size);
 	src_crc32 = crc32_le(~0, src_addr, size);
-	orig_dst_addr = dma_alloc_coherent(dev, size + alignment,
+	orig_dst_addr = kzalloc(size + alignment, GFP_KERNEL);
-					   &orig_dst_phys_addr, GFP_KERNEL);
 	if (!orig_dst_addr) {
 		dev_err(dev, "Failed to allocate destination address\n");
 		ret = false;
-		goto err_orig_src_addr;
+		goto err_dst_addr;
+	}
+	orig_dst_phys_addr = dma_map_single(dev, orig_dst_addr,
+					    size + alignment, DMA_FROM_DEVICE);
+	if (dma_mapping_error(dev, orig_dst_phys_addr)) {
+		dev_err(dev, "failed to map destination buffer address\n");
+		ret = false;
+		goto err_dst_phys_addr;
 	}
 	if (alignment && !IS_ALIGNED(orig_dst_phys_addr, alignment)) {
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_SIZE,
 				 size);
+	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_FLAGS, flags);
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_TYPE, irq_type);
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_NUMBER, 1);
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_COMMAND,
 	wait_for_completion(&test->irq_raised);
+	dma_unmap_single(dev, orig_dst_phys_addr, size + alignment,
+			 DMA_FROM_DEVICE);
 	dst_crc32 = crc32_le(~0, dst_addr, size);
 	if (dst_crc32 == src_crc32)
 		ret = true;
-	dma_free_coherent(dev, size + alignment, orig_dst_addr,
+err_dst_phys_addr:
-			  orig_dst_phys_addr);
+	kfree(orig_dst_addr);
-err_orig_src_addr:
+err_dst_addr:
-	dma_free_coherent(dev, size + alignment, orig_src_addr,
+	dma_unmap_single(dev, orig_src_phys_addr, size + alignment,
-			  orig_src_phys_addr);
+			 DMA_TO_DEVICE);
+err_src_phys_addr:
+	kfree(orig_src_addr);
 err:
 	return ret;
 }
-static bool pci_endpoint_test_write(struct pci_endpoint_test *test, size_t size)
+static bool pci_endpoint_test_write(struct pci_endpoint_test *test,
+				    unsigned long arg)
 {
+	struct pci_endpoint_test_xfer_param param;
 	bool ret = false;
+	u32 flags = 0;
+	bool use_dma;
 	u32 reg;
 	void *addr;
 	dma_addr_t phys_addr;
 	dma_addr_t orig_phys_addr;
 	size_t offset;
 	size_t alignment = test->alignment;
+	int irq_type = test->irq_type;
+	size_t size;
 	u32 crc32;
+	int err;
+	err = copy_from_user(&param, (void __user *)arg, sizeof(param));
+	if (err != 0) {
+		dev_err(dev, "Failed to get transfer param\n");
+		return false;
+	}
+	size = param.size;
 	if (size > SIZE_MAX - alignment)
 		goto err;
+	use_dma = !!(param.flags & PCITEST_FLAGS_USE_DMA);
+	if (use_dma)
+		flags |= FLAG_USE_DMA;
 	if (irq_type < IRQ_TYPE_LEGACY || irq_type > IRQ_TYPE_MSIX) {
 		dev_err(dev, "Invalid IRQ type option\n");
 		goto err;
 	}
-	orig_addr = dma_alloc_coherent(dev, size + alignment, &orig_phys_addr,
+	orig_addr = kzalloc(size + alignment, GFP_KERNEL);
-				       GFP_KERNEL);
 	if (!orig_addr) {
 		dev_err(dev, "Failed to allocate address\n");
 		ret = false;
 		goto err;
 	}
+	get_random_bytes(orig_addr, size + alignment);
+	orig_phys_addr = dma_map_single(dev, orig_addr, size + alignment,
+					DMA_TO_DEVICE);
+	if (dma_mapping_error(dev, orig_phys_addr)) {
+		dev_err(dev, "failed to map source buffer address\n");
+		ret = false;
+		goto err_phys_addr;
+	}
 	if (alignment && !IS_ALIGNED(orig_phys_addr, alignment)) {
 		phys_addr =  PTR_ALIGN(orig_phys_addr, alignment);
 		offset = phys_addr - orig_phys_addr;
 		addr = orig_addr;
 	}
-	get_random_bytes(addr, size);
 	crc32 = crc32_le(~0, addr, size);
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_CHECKSUM,
 				 crc32);
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_SIZE, size);
+	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_FLAGS, flags);
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_TYPE, irq_type);
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_NUMBER, 1);
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_COMMAND,
 	if (reg & STATUS_READ_SUCCESS)
 		ret = true;
-	dma_free_coherent(dev, size + alignment, orig_addr, orig_phys_addr);
+	dma_unmap_single(dev, orig_phys_addr, size + alignment,
+			 DMA_TO_DEVICE);
+err_phys_addr:
+	kfree(orig_addr);
 err:
 	return ret;
 }
-static bool pci_endpoint_test_read(struct pci_endpoint_test *test, size_t size)
+static bool pci_endpoint_test_read(struct pci_endpoint_test *test,
+				   unsigned long arg)
 {
+	struct pci_endpoint_test_xfer_param param;
 	bool ret = false;
+	u32 flags = 0;
+	bool use_dma;
+	size_t size;
 	void *addr;
 	dma_addr_t phys_addr;
 	struct pci_dev *pdev = test->pdev;
 	dma_addr_t orig_phys_addr;
 	size_t offset;
 	size_t alignment = test->alignment;
+	int irq_type = test->irq_type;
 	u32 crc32;
+	int err;
+	err = copy_from_user(&param, (void __user *)arg, sizeof(param));
+	if (err) {
+		dev_err(dev, "Failed to get transfer param\n");
+		return false;
+	}
+	size = param.size;
 	if (size > SIZE_MAX - alignment)
 		goto err;
+	use_dma = !!(param.flags & PCITEST_FLAGS_USE_DMA);
+	if (use_dma)
+		flags |= FLAG_USE_DMA;
 	if (irq_type < IRQ_TYPE_LEGACY || irq_type > IRQ_TYPE_MSIX) {
 		dev_err(dev, "Invalid IRQ type option\n");
 		goto err;
 	}
-	orig_addr = dma_alloc_coherent(dev, size + alignment, &orig_phys_addr,
+	orig_addr = kzalloc(size + alignment, GFP_KERNEL);
-				       GFP_KERNEL);
 	if (!orig_addr) {
 		dev_err(dev, "Failed to allocate destination address\n");
 		ret = false;
 		goto err;
 	}
+	orig_phys_addr = dma_map_single(dev, orig_addr, size + alignment,
+					DMA_FROM_DEVICE);
+	if (dma_mapping_error(dev, orig_phys_addr)) {
+		dev_err(dev, "failed to map source buffer address\n");
+		ret = false;
+		goto err_phys_addr;
+	}
 	if (alignment && !IS_ALIGNED(orig_phys_addr, alignment)) {
 		phys_addr = PTR_ALIGN(orig_phys_addr, alignment);
 		offset = phys_addr - orig_phys_addr;
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_SIZE, size);
+	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_FLAGS, flags);
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_TYPE, irq_type);
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_NUMBER, 1);
 	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_COMMAND,
 	wait_for_completion(&test->irq_raised);
+	dma_unmap_single(dev, orig_phys_addr, size + alignment,
+			 DMA_FROM_DEVICE);
 	crc32 = crc32_le(~0, addr, size);
 	if (crc32 == pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_CHECKSUM))
 		ret = true;
-	dma_free_coherent(dev, size + alignment, orig_addr, orig_phys_addr);
+err_phys_addr:
+	kfree(orig_addr);
 err:
 	return ret;
 }
+static bool pci_endpoint_test_clear_irq(struct pci_endpoint_test *test)
+{
+	pci_endpoint_test_release_irq(test);
+	pci_endpoint_test_free_irq_vectors(test);
+	return true;
+}
 static bool pci_endpoint_test_set_irq(struct pci_endpoint_test *test,
 				      int req_irq_type)
 {
 		return false;
 	}
-	if (irq_type == req_irq_type)
+	if (test->irq_type == req_irq_type)
 		return true;
 	pci_endpoint_test_release_irq(test);
 	if (!pci_endpoint_test_request_irq(test))
 		goto err;
-	irq_type = req_irq_type;
 	return true;
 err:
 	pci_endpoint_test_free_irq_vectors(test);
-	irq_type = IRQ_TYPE_UNDEFINED;
 	return false;
 }
 	case PCITEST_GET_IRQTYPE:
 		ret = irq_type;
 		break;
+	case PCITEST_CLEAR_IRQ:
+		ret = pci_endpoint_test_clear_irq(test);
+		break;
 	}
 ret:
 {
 	int err;
 	int id;
-	char name[20];
+	char name[24];
 	enum pci_barno bar;
 	void __iomem *base;
 	struct device *dev = &pdev->dev;
 	test->test_reg_bar = 0;
 	test->alignment = 0;
 	test->pdev = pdev;
+	test->irq_type = IRQ_TYPE_UNDEFINED;
 	if (no_msi)
 		irq_type = IRQ_TYPE_LEGACY;
 	init_completion(&test->irq_raised);
 	mutex_init(&test->mutex);
+	if ((dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(48)) != 0) &&
+	    dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32)) != 0) {
+		dev_err(dev, "Cannot set DMA mask\n");
+		return -EINVAL;
+	}
 	err = pci_enable_device(pdev);
 	if (err) {
 		dev_err(dev, "Cannot enable PCI device\n");
 	if (!pci_endpoint_test_alloc_irq_vectors(test, irq_type))
 		goto err_disable_irq;
-	if (!pci_endpoint_test_request_irq(test))
-		goto err_disable_irq;
 	for (bar = 0; bar < PCI_STD_NUM_BARS; bar++) {
 		if (pci_resource_flags(pdev, bar) & IORESOURCE_MEM) {
 			base = pci_ioremap_bar(pdev, bar);
 	}
 	snprintf(name, sizeof(name), DRV_MODULE_NAME ".%d", id);
+	test->name = kstrdup(name, GFP_KERNEL);
+	if (!test->name) {
+		err = -ENOMEM;
+		goto err_ida_remove;
+	}
+	if (!pci_endpoint_test_request_irq(test))
+		goto err_kfree_test_name;
 	misc_device = &test->miscdev;
 	misc_device->minor = MISC_DYNAMIC_MINOR;
 	misc_device->name = kstrdup(name, GFP_KERNEL);
 	if (!misc_device->name) {
 		err = -ENOMEM;
-		goto err_ida_remove;
+		goto err_release_irq;
 	}
 	misc_device->fops = &pci_endpoint_test_fops,
 err_kfree_name:
 	kfree(misc_device->name);
+err_release_irq:
+	pci_endpoint_test_release_irq(test);
+err_kfree_test_name:
+	kfree(test->name);
 err_ida_remove:
 	ida_simple_remove(&pci_endpoint_test_ida, id);
 		if (test->bar[bar])
 			pci_iounmap(pdev, test->bar[bar]);
 	}
-	pci_endpoint_test_release_irq(test);
 err_disable_irq:
 	pci_endpoint_test_free_irq_vectors(test);
 	misc_deregister(&test->miscdev);
 	kfree(misc_device->name);
+	kfree(test->name);
 	ida_simple_remove(&pci_endpoint_test_ida, id);
 	for (bar = 0; bar < PCI_STD_NUM_BARS; bar++) {
 		if (test->bar[bar])
 	pci_disable_device(pdev);
 }
+static const struct pci_endpoint_test_data default_data = {
+	.test_reg_bar = BAR_0,
+	.alignment = SZ_4K,
+	.irq_type = IRQ_TYPE_MSI,
+};
 static const struct pci_endpoint_test_data am654_data = {
 	.test_reg_bar = BAR_2,
 	.alignment = SZ_64K,
 };
 static const struct pci_device_id pci_endpoint_test_tbl[] = {
-	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA74x) },
+	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA74x),
-	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA72x) },
+	  .driver_data = (kernel_ulong_t)&default_data,
+	},
+	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA72x),
+	  .driver_data = (kernel_ulong_t)&default_data,
+	},
 	{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, 0x81c0) },
 	{ PCI_DEVICE_DATA(SYNOPSYS, EDDA, NULL) },
 	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_AM654),

drivers/pci/controller/dwc/Kconfig+26−3

@@ -248,14 +248,37 @@ config PCI_MESON · 247 unmodified lines
248	implement the driver.	248	implement the driver.
249		249
250	config PCIE_TEGRA194	250	config PCIE_TEGRA194
251	tristate "NVIDIA Tegra194 (and later) PCIe controller"	251	tristate
		252
		253	config PCIE_TEGRA194_HOST
		254	tristate "NVIDIA Tegra194 (and later) PCIe controller - Host Mode"
252	depends on ARCH_TEGRA_194_SOC \|\| COMPILE_TEST	255	depends on ARCH_TEGRA_194_SOC \|\| COMPILE_TEST
253	depends on PCI_MSI_IRQ_DOMAIN	256	depends on PCI_MSI_IRQ_DOMAIN
254	select PCIE_DW_HOST	257	select PCIE_DW_HOST
255	select PHY_TEGRA194_P2U	258	select PHY_TEGRA194_P2U
		259	select PCIE_TEGRA194
		260	help
		261	Enables support for the PCIe controller in the NVIDIA Tegra194 SoC to
		262	work in host mode. There are two instances of PCIe controllers in
		263	Tegra194. This controller can work either as EP or RC. In order to
		264	enable host-specific features PCIE_TEGRA194_HOST must be selected and
		265	in order to enable device-specific features PCIE_TEGRA194_EP must be
		266	selected. This uses the DesignWare core.
		267
		268	config PCIE_TEGRA194_EP
		269	tristate "NVIDIA Tegra194 (and later) PCIe controller - Endpoint Mode"
		270	depends on ARCH_TEGRA_194_SOC \|\| COMPILE_TEST
		271	depends on PCI_ENDPOINT
		272	select PCIE_DW_EP
		273	select PHY_TEGRA194_P2U
		274	select PCIE_TEGRA194
256	help	275	help
257	Say Y here if you want support for DesignWare core based PCIe host	276	Enables support for the PCIe controller in the NVIDIA Tegra194 SoC to
258	controller found in NVIDIA Tegra194 SoC.	277	work in host mode. There are two instances of PCIe controllers in
		278	Tegra194. This controller can work either as EP or RC. In order to
		279	enable host-specific features PCIE_TEGRA194_HOST must be selected and
		280	in order to enable device-specific features PCIE_TEGRA194_EP must be
		281	selected. This uses the DesignWare core.
259		282
260	config PCIE_UNIPHIER	283	config PCIE_UNIPHIER
261	bool "Socionext UniPhier PCIe controllers"	284	bool "Socionext UniPhier PCIe controllers"

drivers/pci/controller/dwc/pci-keystone.c+4−1

 	case PCI_EPC_IRQ_MSI:
 		dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
 		break;
+	case PCI_EPC_IRQ_MSIX:
+		dw_pcie_ep_raise_msix_irq(ep, func_no, interrupt_num);
+		break;
 	default:
 		dev_err(pci->dev, "UNKNOWN IRQ type\n");
 		return -EINVAL;
 static const struct pci_epc_features ks_pcie_am654_epc_features = {
 	.linkup_notifier = false,
 	.msi_capable = true,
-	.msix_capable = false,
+	.msix_capable = true,
 	.reserved_bar = 1 << BAR_0 | 1 << BAR_1,
 	.bar_fixed_64bit = 1 << BAR_0,
 	.bar_fixed_size[2] = SZ_1M,

drivers/pci/controller/dwc/pcie-designware-ep.c+85−59

 	pci_epc_linkup(epc);
 }
+EXPORT_SYMBOL_GPL(dw_pcie_ep_linkup);
+void dw_pcie_ep_init_notify(struct dw_pcie_ep *ep)
+{
+	struct pci_epc *epc = ep->epc;
+	pci_epc_init_notify(epc);
+}
+EXPORT_SYMBOL_GPL(dw_pcie_ep_init_notify);
 static void __dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar,
 				   int flags)
 	dw_pcie_disable_atu(pci, atu_index, DW_PCIE_REGION_INBOUND);
 	clear_bit(atu_index, ep->ib_window_map);
+	ep->epf_bar[bar] = NULL;
 }
 static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no,
 		dw_pcie_writel_dbi(pci, reg + 4, 0);
 	}
+	ep->epf_bar[bar] = epf_bar;
 	dw_pcie_dbi_ro_wr_dis(pci);
 	return 0;
 	return val;
 }
-static int dw_pcie_ep_set_msix(struct pci_epc *epc, u8 func_no, u16 interrupts)
+static int dw_pcie_ep_set_msix(struct pci_epc *epc, u8 func_no, u16 interrupts,
+			       enum pci_barno bir, u32 offset)
 {
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	if (!ep->msix_cap)
 		return -EINVAL;
+	dw_pcie_dbi_ro_wr_en(pci);
 	reg = ep->msix_cap + PCI_MSIX_FLAGS;
 	val = dw_pcie_readw_dbi(pci, reg);
 	val &= ~PCI_MSIX_FLAGS_QSIZE;
 	val |= interrupts;
-	dw_pcie_dbi_ro_wr_en(pci);
 	dw_pcie_writew_dbi(pci, reg, val);
+	reg = ep->msix_cap + PCI_MSIX_TABLE;
+	val = offset | bir;
+	dw_pcie_writel_dbi(pci, reg, val);
+	reg = ep->msix_cap + PCI_MSIX_PBA;
+	val = (offset + (interrupts * PCI_MSIX_ENTRY_SIZE)) | bir;
+	dw_pcie_writel_dbi(pci, reg, val);
 	dw_pcie_dbi_ro_wr_dis(pci);
 	return 0;
 			     u16 interrupt_num)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+	struct pci_epf_msix_tbl *msix_tbl;
 	struct pci_epc *epc = ep->epc;
-	u16 tbl_offset, bir;
+	struct pci_epf_bar *epf_bar;
-	u32 bar_addr_upper, bar_addr_lower;
-	u32 msg_addr_upper, msg_addr_lower;
 	u32 reg, msg_data, vec_ctrl;
-	u64 tbl_addr, msg_addr, reg_u64;
+	unsigned int aligned_offset;
-	void __iomem *msix_tbl;
+	u32 tbl_offset;
+	u64 msg_addr;
 	int ret;
+	u8 bir;
 	reg = ep->msix_cap + PCI_MSIX_TABLE;
 	tbl_offset = dw_pcie_readl_dbi(pci, reg);
 	bir = (tbl_offset & PCI_MSIX_TABLE_BIR);
 	tbl_offset &= PCI_MSIX_TABLE_OFFSET;
-	reg = PCI_BASE_ADDRESS_0 + (4 * bir);
+	epf_bar = ep->epf_bar[bir];
-	bar_addr_upper = 0;
+	msix_tbl = epf_bar->addr;
-	bar_addr_lower = dw_pcie_readl_dbi(pci, reg);
+	msix_tbl = (struct pci_epf_msix_tbl *)((char *)msix_tbl + tbl_offset);
-	reg_u64 = (bar_addr_lower & PCI_BASE_ADDRESS_MEM_TYPE_MASK);
-	if (reg_u64 == PCI_BASE_ADDRESS_MEM_TYPE_64)
-		bar_addr_upper = dw_pcie_readl_dbi(pci, reg + 4);
-	tbl_addr = ((u64) bar_addr_upper) << 32 | bar_addr_lower;
-	tbl_addr += (tbl_offset + ((interrupt_num - 1) * PCI_MSIX_ENTRY_SIZE));
-	tbl_addr &= PCI_BASE_ADDRESS_MEM_MASK;
-	msix_tbl = ioremap(ep->phys_base + tbl_addr,
-				   PCI_MSIX_ENTRY_SIZE);
-	if (!msix_tbl)
-		return -EINVAL;
-	msg_addr_lower = readl(msix_tbl + PCI_MSIX_ENTRY_LOWER_ADDR);
-	msg_addr_upper = readl(msix_tbl + PCI_MSIX_ENTRY_UPPER_ADDR);
-	msg_addr = ((u64) msg_addr_upper) << 32 | msg_addr_lower;
-	msg_data = readl(msix_tbl + PCI_MSIX_ENTRY_DATA);
-	vec_ctrl = readl(msix_tbl + PCI_MSIX_ENTRY_VECTOR_CTRL);
-	iounmap(msix_tbl);
+	msg_addr = msix_tbl[(interrupt_num - 1)].msg_addr;
+	msg_data = msix_tbl[(interrupt_num - 1)].msg_data;
+	vec_ctrl = msix_tbl[(interrupt_num - 1)].vector_ctrl;
 	if (vec_ctrl & PCI_MSIX_ENTRY_CTRL_MASKBIT) {
 		dev_dbg(pci->dev, "MSI-X entry ctrl set\n");
 		return -EPERM;
 	}
-	ret = dw_pcie_ep_map_addr(epc, func_no, ep->msi_mem_phys, msg_addr,
+	aligned_offset = msg_addr & (epc->mem->page_size - 1);
+	ret = dw_pcie_ep_map_addr(epc, func_no, ep->msi_mem_phys,  msg_addr,
 				  epc->mem->page_size);
 	if (ret)
 		return ret;
-	writel(msg_data, ep->msi_mem);
+	writel(msg_data, ep->msi_mem + aligned_offset);
 	dw_pcie_ep_unmap_addr(epc, func_no, ep->msi_mem_phys);
 	return 0;
 }
-int dw_pcie_ep_init(struct dw_pcie_ep *ep)
+int dw_pcie_ep_init_complete(struct dw_pcie_ep *ep)
 {
+	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+	unsigned int offset;
+	unsigned int nbars;
+	u8 hdr_type;
+	u32 reg;
 	int i;
+	hdr_type = dw_pcie_readb_dbi(pci, PCI_HEADER_TYPE);
+	if (hdr_type != PCI_HEADER_TYPE_NORMAL) {
+		dev_err(pci->dev,
+			"PCIe controller is not set to EP mode (hdr_type:0x%x)!\n",
+			hdr_type);
+		return -EIO;
+	}
+	ep->msi_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSI);
+	ep->msix_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSIX);
+	offset = dw_pcie_ep_find_ext_capability(pci, PCI_EXT_CAP_ID_REBAR);
+	if (offset) {
+		reg = dw_pcie_readl_dbi(pci, offset + PCI_REBAR_CTRL);
+		nbars = (reg & PCI_REBAR_CTRL_NBAR_MASK) >>
+			PCI_REBAR_CTRL_NBAR_SHIFT;
+		dw_pcie_dbi_ro_wr_en(pci);
+		for (i = 0; i < nbars; i++, offset += PCI_REBAR_CTRL)
+			dw_pcie_writel_dbi(pci, offset + PCI_REBAR_CAP, 0x0);
+		dw_pcie_dbi_ro_wr_dis(pci);
+	}
+	dw_pcie_setup(pci);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(dw_pcie_ep_init_complete);
+int dw_pcie_ep_init(struct dw_pcie_ep *ep)
+{
 	int ret;
-	u32 reg;
 	void *addr;
-	u8 hdr_type;
-	unsigned int nbars;
-	unsigned int offset;
 	struct pci_epc *epc;
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	struct device *dev = pci->dev;
 	struct device_node *np = dev->of_node;
+	const struct pci_epc_features *epc_features;
 	if (!pci->dbi_base || !pci->dbi_base2) {
 		dev_err(dev, "dbi_base/dbi_base2 is not populated\n");
 	if (ep->ops->ep_init)
 		ep->ops->ep_init(ep);
-	hdr_type = dw_pcie_readb_dbi(pci, PCI_HEADER_TYPE);
-	if (hdr_type != PCI_HEADER_TYPE_NORMAL) {
-		dev_err(pci->dev, "PCIe controller is not set to EP mode (hdr_type:0x%x)!\n",
-			hdr_type);
-		return -EIO;
-	}
 	ret = of_property_read_u8(np, "max-functions", &epc->max_functions);
 	if (ret < 0)
 		epc->max_functions = 1;
 		dev_err(dev, "Failed to reserve memory for MSI/MSI-X\n");
 		return -ENOMEM;
 	}
-	ep->msi_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSI);
-	ep->msix_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSIX);
-	offset = dw_pcie_ep_find_ext_capability(pci, PCI_EXT_CAP_ID_REBAR);
-	if (offset) {
-		reg = dw_pcie_readl_dbi(pci, offset + PCI_REBAR_CTRL);
-		nbars = (reg & PCI_REBAR_CTRL_NBAR_MASK) >>
-			PCI_REBAR_CTRL_NBAR_SHIFT;
-		dw_pcie_dbi_ro_wr_en(pci);
+	if (ep->ops->get_features) {
-		for (i = 0; i < nbars; i++, offset += PCI_REBAR_CTRL)
+		epc_features = ep->ops->get_features(ep);
-			dw_pcie_writel_dbi(pci, offset + PCI_REBAR_CAP, 0x0);
+		if (epc_features->core_init_notifier)
-		dw_pcie_dbi_ro_wr_dis(pci);
+			return 0;
 	}
-	dw_pcie_setup(pci);
+	return dw_pcie_ep_init_complete(ep);
-	return 0;
 }
+EXPORT_SYMBOL_GPL(dw_pcie_ep_init);

drivers/pci/controller/dwc/pcie-designware.h+12−0

 	phys_addr_t		msi_mem_phys;
 	u8			msi_cap;	/* MSI capability offset */
 	u8			msix_cap;	/* MSI-X capability offset */
+	struct pci_epf_bar	*epf_bar[PCI_STD_NUM_BARS];
 };
 struct dw_pcie_ops {
 #ifdef CONFIG_PCIE_DW_EP
 void dw_pcie_ep_linkup(struct dw_pcie_ep *ep);
 int dw_pcie_ep_init(struct dw_pcie_ep *ep);
+int dw_pcie_ep_init_complete(struct dw_pcie_ep *ep);
+void dw_pcie_ep_init_notify(struct dw_pcie_ep *ep);
 void dw_pcie_ep_exit(struct dw_pcie_ep *ep);
 int dw_pcie_ep_raise_legacy_irq(struct dw_pcie_ep *ep, u8 func_no);
 int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
 	return 0;
 }
+static inline int dw_pcie_ep_init_complete(struct dw_pcie_ep *ep)
+{
+	return 0;
+}
+static inline void dw_pcie_ep_init_notify(struct dw_pcie_ep *ep)
+{
+}
 static inline void dw_pcie_ep_exit(struct dw_pcie_ep *ep)
 {
 }

drivers/pci/controller/dwc/pcie-tegra194.c+695−17

@@ -11,6 +11,7 @@ · 10 unmodified lines
11	#include <linux/debugfs.h>	11	#include <linux/debugfs.h>
12	#include <linux/delay.h>	12	#include <linux/delay.h>
13	#include <linux/gpio.h>	13	#include <linux/gpio.h>
		14	#include <linux/gpio/consumer.h>
14	#include <linux/interrupt.h>	15	#include <linux/interrupt.h>
15	#include <linux/iopoll.h>	16	#include <linux/iopoll.h>
16	#include <linux/kernel.h>	17	#include <linux/kernel.h>
@@ -53,6 +54,7 @@ · 36 unmodified lines
53	#define APPL_INTR_EN_L0_0_LINK_STATE_INT_EN BIT(0)	54	#define APPL_INTR_EN_L0_0_LINK_STATE_INT_EN BIT(0)
54	#define APPL_INTR_EN_L0_0_MSI_RCV_INT_EN BIT(4)	55	#define APPL_INTR_EN_L0_0_MSI_RCV_INT_EN BIT(4)
55	#define APPL_INTR_EN_L0_0_INT_INT_EN BIT(8)	56	#define APPL_INTR_EN_L0_0_INT_INT_EN BIT(8)
		57	#define APPL_INTR_EN_L0_0_PCI_CMD_EN_INT_EN BIT(15)
56	#define APPL_INTR_EN_L0_0_CDM_REG_CHK_INT_EN BIT(19)	58	#define APPL_INTR_EN_L0_0_CDM_REG_CHK_INT_EN BIT(19)
57	#define APPL_INTR_EN_L0_0_SYS_INTR_EN BIT(30)	59	#define APPL_INTR_EN_L0_0_SYS_INTR_EN BIT(30)
58	#define APPL_INTR_EN_L0_0_SYS_MSI_INTR_EN BIT(31)	60	#define APPL_INTR_EN_L0_0_SYS_MSI_INTR_EN BIT(31)
@@ -60,19 +62,26 @@ · 1 unmodified line
60	#define APPL_INTR_STATUS_L0 0xC	62	#define APPL_INTR_STATUS_L0 0xC
61	#define APPL_INTR_STATUS_L0_LINK_STATE_INT BIT(0)	63	#define APPL_INTR_STATUS_L0_LINK_STATE_INT BIT(0)
62	#define APPL_INTR_STATUS_L0_INT_INT BIT(8)	64	#define APPL_INTR_STATUS_L0_INT_INT BIT(8)
		65	#define APPL_INTR_STATUS_L0_PCI_CMD_EN_INT BIT(15)
		66	#define APPL_INTR_STATUS_L0_PEX_RST_INT BIT(16)
63	#define APPL_INTR_STATUS_L0_CDM_REG_CHK_INT BIT(18)	67	#define APPL_INTR_STATUS_L0_CDM_REG_CHK_INT BIT(18)
64		68
65	#define APPL_INTR_EN_L1_0_0 0x1C	69	#define APPL_INTR_EN_L1_0_0 0x1C
66	#define APPL_INTR_EN_L1_0_0_LINK_REQ_RST_NOT_INT_EN BIT(1)	70	#define APPL_INTR_EN_L1_0_0_LINK_REQ_RST_NOT_INT_EN BIT(1)
		71	#define APPL_INTR_EN_L1_0_0_RDLH_LINK_UP_INT_EN BIT(3)
		72	#define APPL_INTR_EN_L1_0_0_HOT_RESET_DONE_INT_EN BIT(30)
67		73
68	#define APPL_INTR_STATUS_L1_0_0 0x20	74	#define APPL_INTR_STATUS_L1_0_0 0x20
69	#define APPL_INTR_STATUS_L1_0_0_LINK_REQ_RST_NOT_CHGED BIT(1)	75	#define APPL_INTR_STATUS_L1_0_0_LINK_REQ_RST_NOT_CHGED BIT(1)
		76	#define APPL_INTR_STATUS_L1_0_0_RDLH_LINK_UP_CHGED BIT(3)
		77	#define APPL_INTR_STATUS_L1_0_0_HOT_RESET_DONE BIT(30)
70		78
71	#define APPL_INTR_STATUS_L1_1 0x2C	79	#define APPL_INTR_STATUS_L1_1 0x2C
72	#define APPL_INTR_STATUS_L1_2 0x30	80	#define APPL_INTR_STATUS_L1_2 0x30
73	#define APPL_INTR_STATUS_L1_3 0x34	81	#define APPL_INTR_STATUS_L1_3 0x34
74	#define APPL_INTR_STATUS_L1_6 0x3C	82	#define APPL_INTR_STATUS_L1_6 0x3C
75	#define APPL_INTR_STATUS_L1_7 0x40	83	#define APPL_INTR_STATUS_L1_7 0x40
		84	#define APPL_INTR_STATUS_L1_15_CFG_BME_CHGED BIT(1)
76		85
77	#define APPL_INTR_EN_L1_8_0 0x44	86	#define APPL_INTR_EN_L1_8_0 0x44
78	#define APPL_INTR_EN_L1_8_BW_MGT_INT_EN BIT(2)	87	#define APPL_INTR_EN_L1_8_BW_MGT_INT_EN BIT(2)
@@ -103,8 +112,12 @@ · 24 unmodified lines
103	#define APPL_INTR_STATUS_L1_18_CDM_REG_CHK_CMP_ERR BIT(1)	112	#define APPL_INTR_STATUS_L1_18_CDM_REG_CHK_CMP_ERR BIT(1)
104	#define APPL_INTR_STATUS_L1_18_CDM_REG_CHK_LOGIC_ERR BIT(0)	113	#define APPL_INTR_STATUS_L1_18_CDM_REG_CHK_LOGIC_ERR BIT(0)
105		114
		115	#define APPL_MSI_CTRL_1 0xAC
		116
106	#define APPL_MSI_CTRL_2 0xB0	117	#define APPL_MSI_CTRL_2 0xB0
107		118
		119	#define APPL_LEGACY_INTX 0xB8
		120
108	#define APPL_LTR_MSG_1 0xC4	121	#define APPL_LTR_MSG_1 0xC4
109	#define LTR_MSG_REQ BIT(15)	122	#define LTR_MSG_REQ BIT(15)
110	#define LTR_MST_NO_SNOOP_SHIFT 16	123	#define LTR_MST_NO_SNOOP_SHIFT 16
@@ -205,6 +218,13 @@ · 94 unmodified lines
205	#define AMBA_ERROR_RESPONSE_CRS_OKAY_FFFFFFFF 1	218	#define AMBA_ERROR_RESPONSE_CRS_OKAY_FFFFFFFF 1
206	#define AMBA_ERROR_RESPONSE_CRS_OKAY_FFFF0001 2	219	#define AMBA_ERROR_RESPONSE_CRS_OKAY_FFFF0001 2
207		220
		221	#define MSIX_ADDR_MATCH_LOW_OFF 0x940
		222	#define MSIX_ADDR_MATCH_LOW_OFF_EN BIT(0)
		223	#define MSIX_ADDR_MATCH_LOW_OFF_MASK GENMASK(31, 2)
		224
		225	#define MSIX_ADDR_MATCH_HIGH_OFF 0x944
		226	#define MSIX_ADDR_MATCH_HIGH_OFF_MASK GENMASK(31, 0)
		227
208	#define PORT_LOGIC_MSIX_DOORBELL 0x948	228	#define PORT_LOGIC_MSIX_DOORBELL 0x948
209		229
210	#define CAP_SPCIE_CAP_OFF 0x154	230	#define CAP_SPCIE_CAP_OFF 0x154
@@ -223,6 +243,13 @@ · 12 unmodified lines
223	#define GEN3_CORE_CLK_FREQ 250000000	243	#define GEN3_CORE_CLK_FREQ 250000000
224	#define GEN4_CORE_CLK_FREQ 500000000	244	#define GEN4_CORE_CLK_FREQ 500000000
225		245
		246	#define LTR_MSG_TIMEOUT (100 * 1000)
		247
		248	#define PERST_DEBOUNCE_TIME (5 * 1000)
		249
		250	#define EP_STATE_DISABLED 0
		251	#define EP_STATE_ENABLED 1
		252
226	static const unsigned int pcie_gen_freq[] = {	253	static const unsigned int pcie_gen_freq[] = {
227	GEN1_CORE_CLK_FREQ,	254	GEN1_CORE_CLK_FREQ,
228	GEN2_CORE_CLK_FREQ,	255	GEN2_CORE_CLK_FREQ,
@@ -260,6 +287,8 @@ struct tegra_pcie_dw { · 31 unmodified lines
260	struct dw_pcie pci;	287	struct dw_pcie pci;
261	struct tegra_bpmp *bpmp;	288	struct tegra_bpmp *bpmp;
262		289
		290	enum dw_pcie_device_mode mode;
		291
263	bool supports_clkreq;	292	bool supports_clkreq;
264	bool enable_cdm_check;	293	bool enable_cdm_check;
265	bool link_state;	294	bool link_state;
@@ -283,6 +312,16 @@ struct tegra_pcie_dw { · 17 unmodified lines
283	struct phy **phys;	312	struct phy **phys;
284		313
285	struct dentry *debugfs;	314	struct dentry *debugfs;
		315
		316	/* Endpoint mode specific */
		317	struct gpio_desc *pex_rst_gpiod;
		318	struct gpio_desc *pex_refclk_sel_gpiod;
		319	unsigned int pex_rst_irq;
		320	int ep_state;
		321	};
		322
		323	struct tegra_pcie_dw_of_data {
		324	enum dw_pcie_device_mode mode;
286	};	325	};
287		326
288	static inline struct tegra_pcie_dw to_tegra_pcie(struct dw_pcie pci)	327	static inline struct tegra_pcie_dw to_tegra_pcie(struct dw_pcie pci)
@@ -339,8 +378,9 @@ static void apply_bad_link_workaround(struct pcie_port *pp) · 50 unmodified lines
339	}	378	}
340	}	379	}
341		380
342	static irqreturn_t tegra_pcie_rp_irq_handler(struct tegra_pcie_dw *pcie)	381	static irqreturn_t tegra_pcie_rp_irq_handler(int irq, void *arg)
343	{	382	{
		383	struct tegra_pcie_dw *pcie = arg;
344	struct dw_pcie *pci = &pcie->pci;	384	struct dw_pcie *pci = &pcie->pci;
345	struct pcie_port *pp = &pci->pp;	385	struct pcie_port *pp = &pci->pp;
346	u32 val, tmp;	386	u32 val, tmp;
@@ -411,11 +451,121 @@ static irqreturn_t tegra_pcie_rp_irq_handler(struct tegra_pcie_dw *pcie) · 64 unmodified lines
411	return IRQ_HANDLED;	451	return IRQ_HANDLED;
412	}	452	}
413		453
414	static irqreturn_t tegra_pcie_irq_handler(int irq, void *arg)	454	static void pex_ep_event_hot_rst_done(struct tegra_pcie_dw *pcie)
		455	{
		456	u32 val;
		457
		458	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L0);
		459	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_0_0);
		460	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_1);
		461	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_2);
		462	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_3);
		463	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_6);
		464	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_7);
		465	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_8_0);
		466	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_9);
		467	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_10);
		468	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_11);
		469	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_13);
		470	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_14);
		471	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_15);
		472	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_17);
		473	appl_writel(pcie, 0xFFFFFFFF, APPL_MSI_CTRL_2);
		474
		475	val = appl_readl(pcie, APPL_CTRL);
		476	val \|= APPL_CTRL_LTSSM_EN;
		477	appl_writel(pcie, val, APPL_CTRL);
		478	}
		479
		480	static irqreturn_t tegra_pcie_ep_irq_thread(int irq, void *arg)
415	{	481	{
416	struct tegra_pcie_dw *pcie = arg;	482	struct tegra_pcie_dw *pcie = arg;
		483	struct dw_pcie *pci = &pcie->pci;
		484	u32 val, speed;
		485
		486	speed = dw_pcie_readw_dbi(pci, pcie->pcie_cap_base + PCI_EXP_LNKSTA) &
		487	PCI_EXP_LNKSTA_CLS;
		488	clk_set_rate(pcie->core_clk, pcie_gen_freq[speed - 1]);
417		489
418	return tegra_pcie_rp_irq_handler(pcie);	490	/* If EP doesn't advertise L1SS, just return */
		491	val = dw_pcie_readl_dbi(pci, pcie->cfg_link_cap_l1sub);
		492	if (!(val & (PCI_L1SS_CAP_ASPM_L1_1 \| PCI_L1SS_CAP_ASPM_L1_2)))
		493	return IRQ_HANDLED;
		494
		495	/* Check if BME is set to '1' */
		496	val = dw_pcie_readl_dbi(pci, PCI_COMMAND);
		497	if (val & PCI_COMMAND_MASTER) {
		498	ktime_t timeout;
		499
		500	/* 110us for both snoop and no-snoop */
		501	val = 110 \| (2 << PCI_LTR_SCALE_SHIFT) \| LTR_MSG_REQ;
		502	val \|= (val << LTR_MST_NO_SNOOP_SHIFT);
		503	appl_writel(pcie, val, APPL_LTR_MSG_1);
		504
		505	/* Send LTR upstream */
		506	val = appl_readl(pcie, APPL_LTR_MSG_2);
		507	val \|= APPL_LTR_MSG_2_LTR_MSG_REQ_STATE;
		508	appl_writel(pcie, val, APPL_LTR_MSG_2);
		509
		510	timeout = ktime_add_us(ktime_get(), LTR_MSG_TIMEOUT);
		511	for (;;) {
		512	val = appl_readl(pcie, APPL_LTR_MSG_2);
		513	if (!(val & APPL_LTR_MSG_2_LTR_MSG_REQ_STATE))
		514	break;
		515	if (ktime_after(ktime_get(), timeout))
		516	break;
		517	usleep_range(1000, 1100);
		518	}
		519	if (val & APPL_LTR_MSG_2_LTR_MSG_REQ_STATE)
		520	dev_err(pcie->dev, "Failed to send LTR message\n");
		521	}
		522
		523	return IRQ_HANDLED;
		524	}
		525
		526	static irqreturn_t tegra_pcie_ep_hard_irq(int irq, void *arg)
		527	{
		528	struct tegra_pcie_dw *pcie = arg;
		529	struct dw_pcie_ep *ep = &pcie->pci.ep;
		530	int spurious = 1;
		531	u32 val, tmp;
		532
		533	val = appl_readl(pcie, APPL_INTR_STATUS_L0);
		534	if (val & APPL_INTR_STATUS_L0_LINK_STATE_INT) {
		535	val = appl_readl(pcie, APPL_INTR_STATUS_L1_0_0);
		536	appl_writel(pcie, val, APPL_INTR_STATUS_L1_0_0);
		537
		538	if (val & APPL_INTR_STATUS_L1_0_0_HOT_RESET_DONE)
		539	pex_ep_event_hot_rst_done(pcie);
		540
		541	if (val & APPL_INTR_STATUS_L1_0_0_RDLH_LINK_UP_CHGED) {
		542	tmp = appl_readl(pcie, APPL_LINK_STATUS);
		543	if (tmp & APPL_LINK_STATUS_RDLH_LINK_UP) {
		544	dev_dbg(pcie->dev, "Link is up with Host\n");
		545	dw_pcie_ep_linkup(ep);
		546	}
		547	}
		548
		549	spurious = 0;
		550	}
		551
		552	if (val & APPL_INTR_STATUS_L0_PCI_CMD_EN_INT) {
		553	val = appl_readl(pcie, APPL_INTR_STATUS_L1_15);
		554	appl_writel(pcie, val, APPL_INTR_STATUS_L1_15);
		555
		556	if (val & APPL_INTR_STATUS_L1_15_CFG_BME_CHGED)
		557	return IRQ_WAKE_THREAD;
		558
		559	spurious = 0;
		560	}
		561
		562	if (spurious) {
		563	dev_warn(pcie->dev, "Random interrupt (STATUS = 0x%08X)\n",
		564	val);
		565	appl_writel(pcie, val, APPL_INTR_STATUS_L0);
		566	}
		567
		568	return IRQ_HANDLED;
419	}	569	}
420		570
421	static int tegra_pcie_dw_rd_own_conf(struct pcie_port *pp, int where, int size,	571	static int tegra_pcie_dw_rd_own_conf(struct pcie_port *pp, int where, int size,
@@ -884,8 +1034,26 @@ static void tegra_pcie_set_msi_vec_num(struct pcie_port *pp) · 462 unmodified lines
884	pp->num_vectors = MAX_MSI_IRQS;	1034	pp->num_vectors = MAX_MSI_IRQS;
885	}	1035	}
886		1036
		1037	static int tegra_pcie_dw_start_link(struct dw_pcie *pci)
		1038	{
		1039	struct tegra_pcie_dw *pcie = to_tegra_pcie(pci);
		1040
		1041	enable_irq(pcie->pex_rst_irq);
		1042
		1043	return 0;
		1044	}
		1045
		1046	static void tegra_pcie_dw_stop_link(struct dw_pcie *pci)
		1047	{
		1048	struct tegra_pcie_dw *pcie = to_tegra_pcie(pci);
		1049
		1050	disable_irq(pcie->pex_rst_irq);
		1051	}
		1052
887	static const struct dw_pcie_ops tegra_dw_pcie_ops = {	1053	static const struct dw_pcie_ops tegra_dw_pcie_ops = {
888	.link_up = tegra_pcie_dw_link_up,	1054	.link_up = tegra_pcie_dw_link_up,
		1055	.start_link = tegra_pcie_dw_start_link,
		1056	.stop_link = tegra_pcie_dw_stop_link,
889	};	1057	};
890		1058
891	static struct dw_pcie_host_ops tegra_pcie_dw_host_ops = {	1059	static struct dw_pcie_host_ops tegra_pcie_dw_host_ops = {
@@ -986,6 +1154,40 @@ static int tegra_pcie_dw_parse_dt(struct tegra_pcie_dw *pcie) · 94 unmodified lines
986	pcie->enable_cdm_check =	1154	pcie->enable_cdm_check =
987	of_property_read_bool(np, "snps,enable-cdm-check");	1155	of_property_read_bool(np, "snps,enable-cdm-check");
988		1156
		1157	if (pcie->mode == DW_PCIE_RC_TYPE)
		1158	return 0;
		1159
		1160	/* Endpoint mode specific DT entries */
		1161	pcie->pex_rst_gpiod = devm_gpiod_get(pcie->dev, "reset", GPIOD_IN);
		1162	if (IS_ERR(pcie->pex_rst_gpiod)) {
		1163	int err = PTR_ERR(pcie->pex_rst_gpiod);
		1164	const char *level = KERN_ERR;
		1165
		1166	if (err == -EPROBE_DEFER)
		1167	level = KERN_DEBUG;
		1168
		1169	dev_printk(level, pcie->dev,
		1170	dev_fmt("Failed to get PERST GPIO: %d\n"),
		1171	err);
		1172	return err;
		1173	}
		1174
		1175	pcie->pex_refclk_sel_gpiod = devm_gpiod_get(pcie->dev,
		1176	"nvidia,refclk-select",
		1177	GPIOD_OUT_HIGH);
		1178	if (IS_ERR(pcie->pex_refclk_sel_gpiod)) {
		1179	int err = PTR_ERR(pcie->pex_refclk_sel_gpiod);
		1180	const char *level = KERN_ERR;
		1181
		1182	if (err == -EPROBE_DEFER)
		1183	level = KERN_DEBUG;
		1184
		1185	dev_printk(level, pcie->dev,
		1186	dev_fmt("Failed to get REFCLK select GPIOs: %d\n"),
		1187	err);
		1188	pcie->pex_refclk_sel_gpiod = NULL;
		1189	}
		1190
989	return 0;	1191	return 0;
990	}	1192	}
991		1193
@@ -1017,6 +1219,34 @@ static int tegra_pcie_bpmp_set_ctrl_state(struct tegra_pcie_dw *pcie, · 25 unmodified lines
1017	return tegra_bpmp_transfer(pcie->bpmp, &msg);	1219	return tegra_bpmp_transfer(pcie->bpmp, &msg);
1018	}	1220	}
1019		1221
		1222	static int tegra_pcie_bpmp_set_pll_state(struct tegra_pcie_dw *pcie,
		1223	bool enable)
		1224	{
		1225	struct mrq_uphy_response resp;
		1226	struct tegra_bpmp_message msg;
		1227	struct mrq_uphy_request req;
		1228
		1229	memset(&req, 0, sizeof(req));
		1230	memset(&resp, 0, sizeof(resp));
		1231
		1232	if (enable) {
		1233	req.cmd = CMD_UPHY_PCIE_EP_CONTROLLER_PLL_INIT;
		1234	req.ep_ctrlr_pll_init.ep_controller = pcie->cid;
		1235	} else {
		1236	req.cmd = CMD_UPHY_PCIE_EP_CONTROLLER_PLL_OFF;
		1237	req.ep_ctrlr_pll_off.ep_controller = pcie->cid;
		1238	}
		1239
		1240	memset(&msg, 0, sizeof(msg));
		1241	msg.mrq = MRQ_UPHY;
		1242	msg.tx.data = &req;
		1243	msg.tx.size = sizeof(req);
		1244	msg.rx.data = &resp;
		1245	msg.rx.size = sizeof(resp);
		1246
		1247	return tegra_bpmp_transfer(pcie->bpmp, &msg);
		1248	}
		1249
1020	static void tegra_pcie_downstream_dev_to_D0(struct tegra_pcie_dw *pcie)	1250	static void tegra_pcie_downstream_dev_to_D0(struct tegra_pcie_dw *pcie)
1021	{	1251	{
1022	struct pcie_port *pp = &pcie->pci.pp;	1252	struct pcie_port *pp = &pcie->pci.pp;
@@ -1427,8 +1657,396 @@ fail_pm_get_sync: · 404 unmodified lines
1427	return ret;	1657	return ret;
1428	}	1658	}
1429		1659
		1660	static void pex_ep_event_pex_rst_assert(struct tegra_pcie_dw *pcie)
		1661	{
		1662	u32 val;
		1663	int ret;
		1664
		1665	if (pcie->ep_state == EP_STATE_DISABLED)
		1666	return;
		1667
		1668	/* Disable LTSSM */
		1669	val = appl_readl(pcie, APPL_CTRL);
		1670	val &= ~APPL_CTRL_LTSSM_EN;
		1671	appl_writel(pcie, val, APPL_CTRL);
		1672
		1673	ret = readl_poll_timeout(pcie->appl_base + APPL_DEBUG, val,
		1674	((val & APPL_DEBUG_LTSSM_STATE_MASK) >>
		1675	APPL_DEBUG_LTSSM_STATE_SHIFT) ==
		1676	LTSSM_STATE_PRE_DETECT,
		1677	1, LTSSM_TIMEOUT);
		1678	if (ret)
		1679	dev_err(pcie->dev, "Failed to go Detect state: %d\n", ret);
		1680
		1681	reset_control_assert(pcie->core_rst);
		1682
		1683	tegra_pcie_disable_phy(pcie);
		1684
		1685	reset_control_assert(pcie->core_apb_rst);
		1686
		1687	clk_disable_unprepare(pcie->core_clk);
		1688
		1689	pm_runtime_put_sync(pcie->dev);
		1690
		1691	ret = tegra_pcie_bpmp_set_pll_state(pcie, false);
		1692	if (ret)
		1693	dev_err(pcie->dev, "Failed to turn off UPHY: %d\n", ret);
		1694
		1695	pcie->ep_state = EP_STATE_DISABLED;
		1696	dev_dbg(pcie->dev, "Uninitialization of endpoint is completed\n");
		1697	}
		1698
		1699	static void pex_ep_event_pex_rst_deassert(struct tegra_pcie_dw *pcie)
		1700	{
		1701	struct dw_pcie *pci = &pcie->pci;
		1702	struct dw_pcie_ep *ep = &pci->ep;
		1703	struct device *dev = pcie->dev;
		1704	u32 val;
		1705	int ret;
		1706
		1707	if (pcie->ep_state == EP_STATE_ENABLED)
		1708	return;
		1709
		1710	ret = pm_runtime_get_sync(dev);
		1711	if (ret < 0) {
		1712	dev_err(dev, "Failed to get runtime sync for PCIe dev: %d\n",
		1713	ret);
		1714	return;
		1715	}
		1716
		1717	ret = tegra_pcie_bpmp_set_pll_state(pcie, true);
		1718	if (ret) {
		1719	dev_err(dev, "Failed to init UPHY for PCIe EP: %d\n", ret);
		1720	goto fail_pll_init;
		1721	}
		1722
		1723	ret = clk_prepare_enable(pcie->core_clk);
		1724	if (ret) {
		1725	dev_err(dev, "Failed to enable core clock: %d\n", ret);
		1726	goto fail_core_clk_enable;
		1727	}
		1728
		1729	ret = reset_control_deassert(pcie->core_apb_rst);
		1730	if (ret) {
		1731	dev_err(dev, "Failed to deassert core APB reset: %d\n", ret);
		1732	goto fail_core_apb_rst;
		1733	}
		1734
		1735	ret = tegra_pcie_enable_phy(pcie);
		1736	if (ret) {
		1737	dev_err(dev, "Failed to enable PHY: %d\n", ret);
		1738	goto fail_phy;
		1739	}
		1740
		1741	/* Clear any stale interrupt statuses */
		1742	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L0);
		1743	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_0_0);
		1744	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_1);
		1745	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_2);
		1746	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_3);
		1747	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_6);
		1748	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_7);
		1749	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_8_0);
		1750	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_9);
		1751	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_10);
		1752	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_11);
		1753	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_13);
		1754	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_14);
		1755	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_15);
		1756	appl_writel(pcie, 0xFFFFFFFF, APPL_INTR_STATUS_L1_17);
		1757
		1758	/* configure this core for EP mode operation */
		1759	val = appl_readl(pcie, APPL_DM_TYPE);
		1760	val &= ~APPL_DM_TYPE_MASK;
		1761	val \|= APPL_DM_TYPE_EP;
		1762	appl_writel(pcie, val, APPL_DM_TYPE);
		1763
		1764	appl_writel(pcie, 0x0, APPL_CFG_SLCG_OVERRIDE);
		1765
		1766	val = appl_readl(pcie, APPL_CTRL);
		1767	val \|= APPL_CTRL_SYS_PRE_DET_STATE;
		1768	val \|= APPL_CTRL_HW_HOT_RST_EN;
		1769	appl_writel(pcie, val, APPL_CTRL);
		1770
		1771	val = appl_readl(pcie, APPL_CFG_MISC);
		1772	val \|= APPL_CFG_MISC_SLV_EP_MODE;
		1773	val \|= (APPL_CFG_MISC_ARCACHE_VAL << APPL_CFG_MISC_ARCACHE_SHIFT);
		1774	appl_writel(pcie, val, APPL_CFG_MISC);
		1775
		1776	val = appl_readl(pcie, APPL_PINMUX);
		1777	val \|= APPL_PINMUX_CLK_OUTPUT_IN_OVERRIDE_EN;
		1778	val \|= APPL_PINMUX_CLK_OUTPUT_IN_OVERRIDE;
		1779	appl_writel(pcie, val, APPL_PINMUX);
		1780
		1781	appl_writel(pcie, pcie->dbi_res->start & APPL_CFG_BASE_ADDR_MASK,
		1782	APPL_CFG_BASE_ADDR);
		1783
		1784	appl_writel(pcie, pcie->atu_dma_res->start &
		1785	APPL_CFG_IATU_DMA_BASE_ADDR_MASK,
		1786	APPL_CFG_IATU_DMA_BASE_ADDR);
		1787
		1788	val = appl_readl(pcie, APPL_INTR_EN_L0_0);
		1789	val \|= APPL_INTR_EN_L0_0_SYS_INTR_EN;
		1790	val \|= APPL_INTR_EN_L0_0_LINK_STATE_INT_EN;
		1791	val \|= APPL_INTR_EN_L0_0_PCI_CMD_EN_INT_EN;
		1792	appl_writel(pcie, val, APPL_INTR_EN_L0_0);
		1793
		1794	val = appl_readl(pcie, APPL_INTR_EN_L1_0_0);
		1795	val \|= APPL_INTR_EN_L1_0_0_HOT_RESET_DONE_INT_EN;
		1796	val \|= APPL_INTR_EN_L1_0_0_RDLH_LINK_UP_INT_EN;
		1797	appl_writel(pcie, val, APPL_INTR_EN_L1_0_0);
		1798
		1799	reset_control_deassert(pcie->core_rst);
		1800
		1801	if (pcie->update_fc_fixup) {
		1802	val = dw_pcie_readl_dbi(pci, CFG_TIMER_CTRL_MAX_FUNC_NUM_OFF);
		1803	val \|= 0x1 << CFG_TIMER_CTRL_ACK_NAK_SHIFT;
		1804	dw_pcie_writel_dbi(pci, CFG_TIMER_CTRL_MAX_FUNC_NUM_OFF, val);
		1805	}
		1806
		1807	config_gen3_gen4_eq_presets(pcie);
		1808
		1809	init_host_aspm(pcie);
		1810
		1811	/* Disable ASPM-L1SS advertisement if there is no CLKREQ routing */
		1812	if (!pcie->supports_clkreq) {
		1813	disable_aspm_l11(pcie);
		1814	disable_aspm_l12(pcie);
		1815	}
		1816
		1817	val = dw_pcie_readl_dbi(pci, GEN3_RELATED_OFF);
		1818	val &= ~GEN3_RELATED_OFF_GEN3_ZRXDC_NONCOMPL;
		1819	dw_pcie_writel_dbi(pci, GEN3_RELATED_OFF, val);
		1820
		1821	/* Configure N_FTS & FTS */
		1822	val = dw_pcie_readl_dbi(pci, PORT_LOGIC_ACK_F_ASPM_CTRL);
		1823	val &= ~(N_FTS_MASK << N_FTS_SHIFT);
		1824	val \|= N_FTS_VAL << N_FTS_SHIFT;
		1825	dw_pcie_writel_dbi(pci, PORT_LOGIC_ACK_F_ASPM_CTRL, val);
		1826
		1827	val = dw_pcie_readl_dbi(pci, PORT_LOGIC_GEN2_CTRL);
		1828	val &= ~FTS_MASK;
		1829	val \|= FTS_VAL;
		1830	dw_pcie_writel_dbi(pci, PORT_LOGIC_GEN2_CTRL, val);
		1831
		1832	/* Configure Max Speed from DT */
		1833	if (pcie->max_speed && pcie->max_speed != -EINVAL) {
		1834	val = dw_pcie_readl_dbi(pci, pcie->pcie_cap_base +
		1835	PCI_EXP_LNKCAP);
		1836	val &= ~PCI_EXP_LNKCAP_SLS;
		1837	val \|= pcie->max_speed;
		1838	dw_pcie_writel_dbi(pci, pcie->pcie_cap_base + PCI_EXP_LNKCAP,
		1839	val);
		1840	}
		1841
		1842	pcie->pcie_cap_base = dw_pcie_find_capability(&pcie->pci,
		1843	PCI_CAP_ID_EXP);
		1844	clk_set_rate(pcie->core_clk, GEN4_CORE_CLK_FREQ);
		1845
		1846	val = (ep->msi_mem_phys & MSIX_ADDR_MATCH_LOW_OFF_MASK);
		1847	val \|= MSIX_ADDR_MATCH_LOW_OFF_EN;
		1848	dw_pcie_writel_dbi(pci, MSIX_ADDR_MATCH_LOW_OFF, val);
		1849	val = (lower_32_bits(ep->msi_mem_phys) & MSIX_ADDR_MATCH_HIGH_OFF_MASK);
		1850	dw_pcie_writel_dbi(pci, MSIX_ADDR_MATCH_HIGH_OFF, val);
		1851
		1852	ret = dw_pcie_ep_init_complete(ep);
		1853	if (ret) {
		1854	dev_err(dev, "Failed to complete initialization: %d\n", ret);
		1855	goto fail_init_complete;
		1856	}
		1857
		1858	dw_pcie_ep_init_notify(ep);
		1859
		1860	/* Enable LTSSM */
		1861	val = appl_readl(pcie, APPL_CTRL);
		1862	val \|= APPL_CTRL_LTSSM_EN;
		1863	appl_writel(pcie, val, APPL_CTRL);
		1864
		1865	pcie->ep_state = EP_STATE_ENABLED;
		1866	dev_dbg(dev, "Initialization of endpoint is completed\n");
		1867
		1868	return;
		1869
		1870	fail_init_complete:
		1871	reset_control_assert(pcie->core_rst);
		1872	tegra_pcie_disable_phy(pcie);
		1873	fail_phy:
		1874	reset_control_assert(pcie->core_apb_rst);
		1875	fail_core_apb_rst:
		1876	clk_disable_unprepare(pcie->core_clk);
		1877	fail_core_clk_enable:
		1878	tegra_pcie_bpmp_set_pll_state(pcie, false);
		1879	fail_pll_init:
		1880	pm_runtime_put_sync(dev);
		1881	}
		1882
		1883	static irqreturn_t tegra_pcie_ep_pex_rst_irq(int irq, void *arg)
		1884	{
		1885	struct tegra_pcie_dw *pcie = arg;
		1886
		1887	if (gpiod_get_value(pcie->pex_rst_gpiod))
		1888	pex_ep_event_pex_rst_assert(pcie);
		1889	else
		1890	pex_ep_event_pex_rst_deassert(pcie);
		1891
		1892	return IRQ_HANDLED;
		1893	}
		1894
		1895	static int tegra_pcie_ep_raise_legacy_irq(struct tegra_pcie_dw *pcie, u16 irq)
		1896	{
		1897	/* Tegra194 supports only INTA */
		1898	if (irq > 1)
		1899	return -EINVAL;
		1900
		1901	appl_writel(pcie, 1, APPL_LEGACY_INTX);
		1902	usleep_range(1000, 2000);
		1903	appl_writel(pcie, 0, APPL_LEGACY_INTX);
		1904	return 0;
		1905	}
		1906
		1907	static int tegra_pcie_ep_raise_msi_irq(struct tegra_pcie_dw *pcie, u16 irq)
		1908	{
		1909	if (unlikely(irq > 31))
		1910	return -EINVAL;
		1911
		1912	appl_writel(pcie, (1 << irq), APPL_MSI_CTRL_1);
		1913
		1914	return 0;
		1915	}
		1916
		1917	static int tegra_pcie_ep_raise_msix_irq(struct tegra_pcie_dw *pcie, u16 irq)
		1918	{
		1919	struct dw_pcie_ep *ep = &pcie->pci.ep;
		1920
		1921	writel(irq, ep->msi_mem);
		1922
		1923	return 0;
		1924	}
		1925
		1926	static int tegra_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
		1927	enum pci_epc_irq_type type,
		1928	u16 interrupt_num)
		1929	{
		1930	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
		1931	struct tegra_pcie_dw *pcie = to_tegra_pcie(pci);
		1932
		1933	switch (type) {
		1934	case PCI_EPC_IRQ_LEGACY:
		1935	return tegra_pcie_ep_raise_legacy_irq(pcie, interrupt_num);
		1936
		1937	case PCI_EPC_IRQ_MSI:
		1938	return tegra_pcie_ep_raise_msi_irq(pcie, interrupt_num);
		1939
		1940	case PCI_EPC_IRQ_MSIX:
		1941	return tegra_pcie_ep_raise_msix_irq(pcie, interrupt_num);
		1942
		1943	default:
		1944	dev_err(pci->dev, "Unknown IRQ type\n");
		1945	return -EPERM;
		1946	}
		1947
		1948	return 0;
		1949	}
		1950
		1951	static const struct pci_epc_features tegra_pcie_epc_features = {
		1952	.linkup_notifier = true,
		1953	.core_init_notifier = true,
		1954	.msi_capable = false,
		1955	.msix_capable = false,
		1956	.reserved_bar = 1 << BAR_2 \| 1 << BAR_3 \| 1 << BAR_4 \| 1 << BAR_5,
		1957	.bar_fixed_64bit = 1 << BAR_0,
		1958	.bar_fixed_size[0] = SZ_1M,
		1959	};
		1960
		1961	static const struct pci_epc_features*
		1962	tegra_pcie_ep_get_features(struct dw_pcie_ep *ep)
		1963	{
		1964	return &tegra_pcie_epc_features;
		1965	}
		1966
		1967	static struct dw_pcie_ep_ops pcie_ep_ops = {
		1968	.raise_irq = tegra_pcie_ep_raise_irq,
		1969	.get_features = tegra_pcie_ep_get_features,
		1970	};
		1971
		1972	static int tegra_pcie_config_ep(struct tegra_pcie_dw *pcie,
		1973	struct platform_device *pdev)
		1974	{
		1975	struct dw_pcie *pci = &pcie->pci;
		1976	struct device *dev = pcie->dev;
		1977	struct dw_pcie_ep *ep;
		1978	struct resource *res;
		1979	char *name;
		1980	int ret;
		1981
		1982	ep = &pci->ep;
		1983	ep->ops = &pcie_ep_ops;
		1984
		1985	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "addr_space");
		1986	if (!res)
		1987	return -EINVAL;
		1988
		1989	ep->phys_base = res->start;
		1990	ep->addr_size = resource_size(res);
		1991	ep->page_size = SZ_64K;
		1992
		1993	ret = gpiod_set_debounce(pcie->pex_rst_gpiod, PERST_DEBOUNCE_TIME);
		1994	if (ret < 0) {
		1995	dev_err(dev, "Failed to set PERST GPIO debounce time: %d\n",
		1996	ret);
		1997	return ret;
		1998	}
		1999
		2000	ret = gpiod_to_irq(pcie->pex_rst_gpiod);
		2001	if (ret < 0) {
		2002	dev_err(dev, "Failed to get IRQ for PERST GPIO: %d\n", ret);
		2003	return ret;
		2004	}
		2005	pcie->pex_rst_irq = (unsigned int)ret;
		2006
		2007	name = devm_kasprintf(dev, GFP_KERNEL, "tegra_pcie_%u_pex_rst_irq",
		2008	pcie->cid);
		2009	if (!name) {
		2010	dev_err(dev, "Failed to create PERST IRQ string\n");
		2011	return -ENOMEM;
		2012	}
		2013
		2014	irq_set_status_flags(pcie->pex_rst_irq, IRQ_NOAUTOEN);
		2015
		2016	pcie->ep_state = EP_STATE_DISABLED;
		2017
		2018	ret = devm_request_threaded_irq(dev, pcie->pex_rst_irq, NULL,
		2019	tegra_pcie_ep_pex_rst_irq,
		2020	IRQF_TRIGGER_RISING \|
		2021	IRQF_TRIGGER_FALLING \| IRQF_ONESHOT,
		2022	name, (void *)pcie);
		2023	if (ret < 0) {
		2024	dev_err(dev, "Failed to request IRQ for PERST: %d\n", ret);
		2025	return ret;
		2026	}
		2027
		2028	name = devm_kasprintf(dev, GFP_KERNEL, "tegra_pcie_%u_ep_work",
		2029	pcie->cid);
		2030	if (!name) {
		2031	dev_err(dev, "Failed to create PCIe EP work thread string\n");
		2032	return -ENOMEM;
		2033	}
		2034
		2035	pm_runtime_enable(dev);
		2036
		2037	ret = dw_pcie_ep_init(ep);
		2038	if (ret) {
		2039	dev_err(dev, "Failed to initialize DWC Endpoint subsystem: %d\n",
		2040	ret);
		2041	return ret;
		2042	}
		2043
		2044	return 0;
		2045	}
		2046
1430	static int tegra_pcie_dw_probe(struct platform_device *pdev)	2047	static int tegra_pcie_dw_probe(struct platform_device *pdev)
1431	{	2048	{
		2049	const struct tegra_pcie_dw_of_data *data;
1432	struct device *dev = &pdev->dev;	2050	struct device *dev = &pdev->dev;
1433	struct resource *atu_dma_res;	2051	struct resource *atu_dma_res;
1434	struct tegra_pcie_dw *pcie;	2052	struct tegra_pcie_dw *pcie;
@@ -1440,6 +2058,8 @@ static int tegra_pcie_dw_probe(struct platform_device *pdev) · 5 unmodified lines
1440	int ret;	2058	int ret;
1441	u32 i;	2059	u32 i;
1442		2060
		2061	data = of_device_get_match_data(dev);
		2062
1443	pcie = devm_kzalloc(dev, sizeof(*pcie), GFP_KERNEL);	2063	pcie = devm_kzalloc(dev, sizeof(*pcie), GFP_KERNEL);
1444	if (!pcie)	2064	if (!pcie)
1445	return -ENOMEM;	2065	return -ENOMEM;
@@ -1449,19 +2069,37 @@ static int tegra_pcie_dw_probe(struct platform_device *pdev) · 3 unmodified lines
1449	pci->ops = &tegra_dw_pcie_ops;	2069	pci->ops = &tegra_dw_pcie_ops;
1450	pp = &pci->pp;	2070	pp = &pci->pp;
1451	pcie->dev = &pdev->dev;	2071	pcie->dev = &pdev->dev;
		2072	pcie->mode = (enum dw_pcie_device_mode)data->mode;
1452		2073
1453	ret = tegra_pcie_dw_parse_dt(pcie);	2074	ret = tegra_pcie_dw_parse_dt(pcie);
1454	if (ret < 0) {	2075	if (ret < 0) {
1455	dev_err(dev, "Failed to parse device tree: %d\n", ret);	2076	const char *level = KERN_ERR;
		2077
		2078	if (ret == -EPROBE_DEFER)
		2079	level = KERN_DEBUG;
		2080
		2081	dev_printk(level, dev,
		2082	dev_fmt("Failed to parse device tree: %d\n"),
		2083	ret);
1456	return ret;	2084	return ret;
1457	}	2085	}
1458		2086
1459	ret = tegra_pcie_get_slot_regulators(pcie);	2087	ret = tegra_pcie_get_slot_regulators(pcie);
1460	if (ret < 0) {	2088	if (ret < 0) {
1461	dev_err(dev, "Failed to get slot regulators: %d\n", ret);	2089	const char *level = KERN_ERR;
		2090
		2091	if (ret == -EPROBE_DEFER)
		2092	level = KERN_DEBUG;
		2093
		2094	dev_printk(level, dev,
		2095	dev_fmt("Failed to get slot regulators: %d\n"),
		2096	ret);
1462	return ret;	2097	return ret;
1463	}	2098	}
1464		2099
		2100	if (pcie->pex_refclk_sel_gpiod)
		2101	gpiod_set_value(pcie->pex_refclk_sel_gpiod, 1);
		2102
1465	pcie->pex_ctl_supply = devm_regulator_get(dev, "vddio-pex-ctl");	2103	pcie->pex_ctl_supply = devm_regulator_get(dev, "vddio-pex-ctl");
1466	if (IS_ERR(pcie->pex_ctl_supply)) {	2104	if (IS_ERR(pcie->pex_ctl_supply)) {
1467	ret = PTR_ERR(pcie->pex_ctl_supply);	2105	ret = PTR_ERR(pcie->pex_ctl_supply);
@@ -1557,24 +2195,49 @@ static int tegra_pcie_dw_probe(struct platform_device *pdev) · 89 unmodified lines
1557	return -ENODEV;	2195	return -ENODEV;
1558	}	2196	}
1559		2197
1560	ret = devm_request_irq(dev, pp->irq, tegra_pcie_irq_handler,
1561	IRQF_SHARED, "tegra-pcie-intr", pcie);
1562	if (ret) {
1563	dev_err(dev, "Failed to request IRQ %d: %d\n", pp->irq, ret);
1564	return ret;
1565	}
1566
1567	pcie->bpmp = tegra_bpmp_get(dev);	2198	pcie->bpmp = tegra_bpmp_get(dev);
1568	if (IS_ERR(pcie->bpmp))	2199	if (IS_ERR(pcie->bpmp))
1569	return PTR_ERR(pcie->bpmp);	2200	return PTR_ERR(pcie->bpmp);
1570		2201
1571	platform_set_drvdata(pdev, pcie);	2202	platform_set_drvdata(pdev, pcie);
1572		2203
1573	ret = tegra_pcie_config_rp(pcie);	2204	switch (pcie->mode) {
1574	if (ret && ret != -ENOMEDIUM)	2205	case DW_PCIE_RC_TYPE:
1575	goto fail;	2206	ret = devm_request_irq(dev, pp->irq, tegra_pcie_rp_irq_handler,
1576	else	2207	IRQF_SHARED, "tegra-pcie-intr", pcie);
1577	return 0;	2208	if (ret) {
		2209	dev_err(dev, "Failed to request IRQ %d: %d\n", pp->irq,
		2210	ret);
		2211	goto fail;
		2212	}
		2213
		2214	ret = tegra_pcie_config_rp(pcie);
		2215	if (ret && ret != -ENOMEDIUM)
		2216	goto fail;
		2217	else
		2218	return 0;
		2219	break;
		2220
		2221	case DW_PCIE_EP_TYPE:
		2222	ret = devm_request_threaded_irq(dev, pp->irq,
		2223	tegra_pcie_ep_hard_irq,
		2224	tegra_pcie_ep_irq_thread,
		2225	IRQF_SHARED \| IRQF_ONESHOT,
		2226	"tegra-pcie-ep-intr", pcie);
		2227	if (ret) {
		2228	dev_err(dev, "Failed to request IRQ %d: %d\n", pp->irq,
		2229	ret);
		2230	goto fail;
		2231	}
		2232
		2233	ret = tegra_pcie_config_ep(pcie, pdev);
		2234	if (ret < 0)
		2235	goto fail;
		2236	break;
		2237
		2238	default:
		2239	dev_err(dev, "Invalid PCIe device type %d\n", pcie->mode);
		2240	}
1578		2241
1579	fail:	2242	fail:
1580	tegra_bpmp_put(pcie->bpmp);	2243	tegra_bpmp_put(pcie->bpmp);
@@ -1593,6 +2256,8 @@ static int tegra_pcie_dw_remove(struct platform_device *pdev) · 12 unmodified lines
1593	pm_runtime_put_sync(pcie->dev);	2256	pm_runtime_put_sync(pcie->dev);
1594	pm_runtime_disable(pcie->dev);	2257	pm_runtime_disable(pcie->dev);
1595	tegra_bpmp_put(pcie->bpmp);	2258	tegra_bpmp_put(pcie->bpmp);
		2259	if (pcie->pex_refclk_sel_gpiod)
		2260	gpiod_set_value(pcie->pex_refclk_sel_gpiod, 0);
1596		2261
1597	return 0;	2262	return 0;
1598	}	2263	}
@@ -1697,9 +2362,22 @@ static void tegra_pcie_dw_shutdown(struct platform_device *pdev) · 98 unmodified lines
1697	__deinit_controller(pcie);	2362	__deinit_controller(pcie);
1698	}	2363	}
1699		2364
		2365	static const struct tegra_pcie_dw_of_data tegra_pcie_dw_rc_of_data = {
		2366	.mode = DW_PCIE_RC_TYPE,
		2367	};
		2368
		2369	static const struct tegra_pcie_dw_of_data tegra_pcie_dw_ep_of_data = {
		2370	.mode = DW_PCIE_EP_TYPE,
		2371	};
		2372
1700	static const struct of_device_id tegra_pcie_dw_of_match[] = {	2373	static const struct of_device_id tegra_pcie_dw_of_match[] = {
1701	{	2374	{
1702	.compatible = "nvidia,tegra194-pcie",	2375	.compatible = "nvidia,tegra194-pcie",
		2376	.data = &tegra_pcie_dw_rc_of_data,
		2377	},
		2378	{
		2379	.compatible = "nvidia,tegra194-pcie-ep",
		2380	.data = &tegra_pcie_dw_ep_of_data,
1703	},	2381	},
1704	{},	2382	{},
1705	};	2383	};

drivers/pci/endpoint/functions/pci-epf-test.c+356−46

@@ -8,6 +8,7 @@ · 7 unmodified lines
8		8
9	#include <linux/crc32.h>	9	#include <linux/crc32.h>
10	#include <linux/delay.h>	10	#include <linux/delay.h>
		11	#include <linux/dmaengine.h>
11	#include <linux/io.h>	12	#include <linux/io.h>
12	#include <linux/module.h>	13	#include <linux/module.h>
13	#include <linux/slab.h>	14	#include <linux/slab.h>
@@ -39,6 +40,8 @@ · 25 unmodified lines
39	#define STATUS_SRC_ADDR_INVALID BIT(7)	40	#define STATUS_SRC_ADDR_INVALID BIT(7)
40	#define STATUS_DST_ADDR_INVALID BIT(8)	41	#define STATUS_DST_ADDR_INVALID BIT(8)
41		42
		43	#define FLAG_USE_DMA BIT(0)
		44
42	#define TIMER_RESOLUTION 1	45	#define TIMER_RESOLUTION 1
43		46
44	static struct workqueue_struct *kpcitest_workqueue;	47	static struct workqueue_struct *kpcitest_workqueue;
@@ -47,7 +50,11 @@ struct pci_epf_test { · 2 unmodified lines
47	void *reg[PCI_STD_NUM_BARS];	50	void *reg[PCI_STD_NUM_BARS];
48	struct pci_epf *epf;	51	struct pci_epf *epf;
49	enum pci_barno test_reg_bar;	52	enum pci_barno test_reg_bar;
		53	size_t msix_table_offset;
50	struct delayed_work cmd_handler;	54	struct delayed_work cmd_handler;
		55	struct dma_chan *dma_chan;
		56	struct completion transfer_complete;
		57	bool dma_supported;
51	const struct pci_epc_features *epc_features;	58	const struct pci_epc_features *epc_features;
52	};	59	};
53		60
@@ -61,6 +68,7 @@ struct pci_epf_test_reg { · 7 unmodified lines
61	u32 checksum;	68	u32 checksum;
62	u32 irq_type;	69	u32 irq_type;
63	u32 irq_number;	70	u32 irq_number;
		71	u32 flags;
64	} __packed;	72	} __packed;
65		73
66	static struct pci_epf_header test_header = {	74	static struct pci_epf_header test_header = {
@@ -72,13 +80,156 @@ static struct pci_epf_header test_header = { · 5 unmodified lines
72		80
73	static size_t bar_size[] = { 512, 512, 1024, 16384, 131072, 1048576 };	81	static size_t bar_size[] = { 512, 512, 1024, 16384, 131072, 1048576 };
74		82
		83	static void pci_epf_test_dma_callback(void *param)
		84	{
		85	struct pci_epf_test *epf_test = param;
		86
		87	complete(&epf_test->transfer_complete);
		88	}
		89
		90	/**
		91	* pci_epf_test_data_transfer() - Function that uses dmaengine API to transfer
		92	* data between PCIe EP and remote PCIe RC
		93	* @epf_test: the EPF test device that performs the data transfer operation
		94	* @dma_dst: The destination address of the data transfer. It can be a physical
		95	* address given by pci_epc_mem_alloc_addr or DMA mapping APIs.
		96	* @dma_src: The source address of the data transfer. It can be a physical
		97	* address given by pci_epc_mem_alloc_addr or DMA mapping APIs.
		98	* @len: The size of the data transfer
		99	*
		100	* Function that uses dmaengine API to transfer data between PCIe EP and remote
		101	* PCIe RC. The source and destination address can be a physical address given
		102	* by pci_epc_mem_alloc_addr or the one obtained using DMA mapping APIs.
		103	*
		104	* The function returns '0' on success and negative value on failure.
		105	*/
		106	static int pci_epf_test_data_transfer(struct pci_epf_test *epf_test,
		107	dma_addr_t dma_dst, dma_addr_t dma_src,
		108	size_t len)
		109	{
		110	enum dma_ctrl_flags flags = DMA_CTRL_ACK \| DMA_PREP_INTERRUPT;
		111	struct dma_chan *chan = epf_test->dma_chan;
		112	struct pci_epf *epf = epf_test->epf;
		113	struct dma_async_tx_descriptor *tx;
		114	struct device *dev = &epf->dev;
		115	dma_cookie_t cookie;
		116	int ret;
		117
		118	if (IS_ERR_OR_NULL(chan)) {
		119	dev_err(dev, "Invalid DMA memcpy channel\n");
		120	return -EINVAL;
		121	}
		122
		123	tx = dmaengine_prep_dma_memcpy(chan, dma_dst, dma_src, len, flags);
		124	if (!tx) {
		125	dev_err(dev, "Failed to prepare DMA memcpy\n");
		126	return -EIO;
		127	}
		128
		129	tx->callback = pci_epf_test_dma_callback;
		130	tx->callback_param = epf_test;
		131	cookie = tx->tx_submit(tx);
		132	reinit_completion(&epf_test->transfer_complete);
		133
		134	ret = dma_submit_error(cookie);
		135	if (ret) {
		136	dev_err(dev, "Failed to do DMA tx_submit %d\n", cookie);
		137	return -EIO;
		138	}
		139
		140	dma_async_issue_pending(chan);
		141	ret = wait_for_completion_interruptible(&epf_test->transfer_complete);
		142	if (ret < 0) {
		143	dmaengine_terminate_sync(chan);
		144	dev_err(dev, "DMA wait_for_completion_timeout\n");
		145	return -ETIMEDOUT;
		146	}
		147
		148	return 0;
		149	}
		150
		151	/**
		152	* pci_epf_test_init_dma_chan() - Function to initialize EPF test DMA channel
		153	* @epf_test: the EPF test device that performs data transfer operation
		154	*
		155	* Function to initialize EPF test DMA channel.
		156	*/
		157	static int pci_epf_test_init_dma_chan(struct pci_epf_test *epf_test)
		158	{
		159	struct pci_epf *epf = epf_test->epf;
		160	struct device *dev = &epf->dev;
		161	struct dma_chan *dma_chan;
		162	dma_cap_mask_t mask;
		163	int ret;
		164
		165	dma_cap_zero(mask);
		166	dma_cap_set(DMA_MEMCPY, mask);
		167
		168	dma_chan = dma_request_chan_by_mask(&mask);
		169	if (IS_ERR(dma_chan)) {
		170	ret = PTR_ERR(dma_chan);
		171	if (ret != -EPROBE_DEFER)
		172	dev_err(dev, "Failed to get DMA channel\n");
		173	return ret;
		174	}
		175	init_completion(&epf_test->transfer_complete);
		176
		177	epf_test->dma_chan = dma_chan;
		178
		179	return 0;
		180	}
		181
		182	/**
		183	* pci_epf_test_clean_dma_chan() - Function to cleanup EPF test DMA channel
		184	* @epf: the EPF test device that performs data transfer operation
		185	*
		186	* Helper to cleanup EPF test DMA channel.
		187	*/
		188	static void pci_epf_test_clean_dma_chan(struct pci_epf_test *epf_test)
		189	{
		190	dma_release_channel(epf_test->dma_chan);
		191	epf_test->dma_chan = NULL;
		192	}
		193
		194	static void pci_epf_test_print_rate(const char *ops, u64 size,
		195	struct timespec64 *start,
		196	struct timespec64 *end, bool dma)
		197	{
		198	struct timespec64 ts;
		199	u64 rate, ns;
		200
		201	ts = timespec64_sub(end, start);
		202
		203	/* convert both size (stored in 'rate') and time in terms of 'ns' */
		204	ns = timespec64_to_ns(&ts);
		205	rate = size * NSEC_PER_SEC;
		206
		207	/* Divide both size (stored in 'rate') and ns by a common factor */
		208	while (ns > UINT_MAX) {
		209	rate >>= 1;
		210	ns >>= 1;
		211	}
		212
		213	if (!ns)
		214	return;
		215
		216	/* calculate the rate */
		217	do_div(rate, (uint32_t)ns);
		218
		219	pr_info("\n%s => Size: %llu bytes\t DMA: %s\t Time: %llu.%09u seconds\t"
		220	"Rate: %llu KB/s\n", ops, size, dma ? "YES" : "NO",
		221	(u64)ts.tv_sec, (u32)ts.tv_nsec, rate / 1024);
		222	}
		223
75	static int pci_epf_test_copy(struct pci_epf_test *epf_test)	224	static int pci_epf_test_copy(struct pci_epf_test *epf_test)
76	{	225	{
77	int ret;	226	int ret;
		227	bool use_dma;
78	void __iomem *src_addr;	228	void __iomem *src_addr;
79	void __iomem *dst_addr;	229	void __iomem *dst_addr;
80	phys_addr_t src_phys_addr;	230	phys_addr_t src_phys_addr;
81	phys_addr_t dst_phys_addr;	231	phys_addr_t dst_phys_addr;
		232	struct timespec64 start, end;
82	struct pci_epf *epf = epf_test->epf;	233	struct pci_epf *epf = epf_test->epf;
83	struct device *dev = &epf->dev;	234	struct device *dev = &epf->dev;
84	struct pci_epc *epc = epf->epc;	235	struct pci_epc *epc = epf->epc;
@@ -117,8 +268,26 @@ static int pci_epf_test_copy(struct pci_epf_test *epf_test) · 32 unmodified lines
117	goto err_dst_addr;	268	goto err_dst_addr;
118	}	269	}
119		270
120	memcpy(dst_addr, src_addr, reg->size);	271	ktime_get_ts64(&start);
		272	use_dma = !!(reg->flags & FLAG_USE_DMA);
		273	if (use_dma) {
		274	if (!epf_test->dma_supported) {
		275	dev_err(dev, "Cannot transfer data using DMA\n");
		276	ret = -EINVAL;
		277	goto err_map_addr;
		278	}
		279
		280	ret = pci_epf_test_data_transfer(epf_test, dst_phys_addr,
		281	src_phys_addr, reg->size);
		282	if (ret)
		283	dev_err(dev, "Data transfer failed\n");
		284	} else {
		285	memcpy(dst_addr, src_addr, reg->size);
		286	}
		287	ktime_get_ts64(&end);
		288	pci_epf_test_print_rate("COPY", reg->size, &start, &end, use_dma);
121		289
		290	err_map_addr:
122	pci_epc_unmap_addr(epc, epf->func_no, dst_phys_addr);	291	pci_epc_unmap_addr(epc, epf->func_no, dst_phys_addr);
123		292
124	err_dst_addr:	293	err_dst_addr:
@@ -140,10 +309,14 @@ static int pci_epf_test_read(struct pci_epf_test *epf_test) · 15 unmodified lines
140	void __iomem *src_addr;	309	void __iomem *src_addr;
141	void *buf;	310	void *buf;
142	u32 crc32;	311	u32 crc32;
		312	bool use_dma;
143	phys_addr_t phys_addr;	313	phys_addr_t phys_addr;
		314	phys_addr_t dst_phys_addr;
		315	struct timespec64 start, end;
144	struct pci_epf *epf = epf_test->epf;	316	struct pci_epf *epf = epf_test->epf;
145	struct device *dev = &epf->dev;	317	struct device *dev = &epf->dev;
146	struct pci_epc *epc = epf->epc;	318	struct pci_epc *epc = epf->epc;
		319	struct device *dma_dev = epf->epc->dev.parent;
147	enum pci_barno test_reg_bar = epf_test->test_reg_bar;	320	enum pci_barno test_reg_bar = epf_test->test_reg_bar;
148	struct pci_epf_test_reg *reg = epf_test->reg[test_reg_bar];	321	struct pci_epf_test_reg *reg = epf_test->reg[test_reg_bar];
149		322
@@ -169,12 +342,44 @@ static int pci_epf_test_read(struct pci_epf_test *epf_test) · 19 unmodified lines
169	goto err_map_addr;	342	goto err_map_addr;
170	}	343	}
171		344
172	memcpy_fromio(buf, src_addr, reg->size);	345	use_dma = !!(reg->flags & FLAG_USE_DMA);
		346	if (use_dma) {
		347	if (!epf_test->dma_supported) {
		348	dev_err(dev, "Cannot transfer data using DMA\n");
		349	ret = -EINVAL;
		350	goto err_dma_map;
		351	}
		352
		353	dst_phys_addr = dma_map_single(dma_dev, buf, reg->size,
		354	DMA_FROM_DEVICE);
		355	if (dma_mapping_error(dma_dev, dst_phys_addr)) {
		356	dev_err(dev, "Failed to map destination buffer addr\n");
		357	ret = -ENOMEM;
		358	goto err_dma_map;
		359	}
		360
		361	ktime_get_ts64(&start);
		362	ret = pci_epf_test_data_transfer(epf_test, dst_phys_addr,
		363	phys_addr, reg->size);
		364	if (ret)
		365	dev_err(dev, "Data transfer failed\n");
		366	ktime_get_ts64(&end);
		367
		368	dma_unmap_single(dma_dev, dst_phys_addr, reg->size,
		369	DMA_FROM_DEVICE);
		370	} else {
		371	ktime_get_ts64(&start);
		372	memcpy_fromio(buf, src_addr, reg->size);
		373	ktime_get_ts64(&end);
		374	}
		375
		376	pci_epf_test_print_rate("READ", reg->size, &start, &end, use_dma);
173		377
174	crc32 = crc32_le(~0, buf, reg->size);	378	crc32 = crc32_le(~0, buf, reg->size);
175	if (crc32 != reg->checksum)	379	if (crc32 != reg->checksum)
176	ret = -EIO;	380	ret = -EIO;
177		381
		382	err_dma_map:
178	kfree(buf);	383	kfree(buf);
179		384
180	err_map_addr:	385	err_map_addr:
@@ -192,10 +397,14 @@ static int pci_epf_test_write(struct pci_epf_test *epf_test) · 11 unmodified lines
192	int ret;	397	int ret;
193	void __iomem *dst_addr;	398	void __iomem *dst_addr;
194	void *buf;	399	void *buf;
		400	bool use_dma;
195	phys_addr_t phys_addr;	401	phys_addr_t phys_addr;
		402	phys_addr_t src_phys_addr;
		403	struct timespec64 start, end;
196	struct pci_epf *epf = epf_test->epf;	404	struct pci_epf *epf = epf_test->epf;
197	struct device *dev = &epf->dev;	405	struct device *dev = &epf->dev;
198	struct pci_epc *epc = epf->epc;	406	struct pci_epc *epc = epf->epc;
		407	struct device *dma_dev = epf->epc->dev.parent;
199	enum pci_barno test_reg_bar = epf_test->test_reg_bar;	408	enum pci_barno test_reg_bar = epf_test->test_reg_bar;
200	struct pci_epf_test_reg *reg = epf_test->reg[test_reg_bar];	409	struct pci_epf_test_reg *reg = epf_test->reg[test_reg_bar];
201		410
@@ -224,7 +433,38 @@ static int pci_epf_test_write(struct pci_epf_test *epf_test) · 22 unmodified lines
224	get_random_bytes(buf, reg->size);	433	get_random_bytes(buf, reg->size);
225	reg->checksum = crc32_le(~0, buf, reg->size);	434	reg->checksum = crc32_le(~0, buf, reg->size);
226		435
227	memcpy_toio(dst_addr, buf, reg->size);	436	use_dma = !!(reg->flags & FLAG_USE_DMA);
		437	if (use_dma) {
		438	if (!epf_test->dma_supported) {
		439	dev_err(dev, "Cannot transfer data using DMA\n");
		440	ret = -EINVAL;
		441	goto err_map_addr;
		442	}
		443
		444	src_phys_addr = dma_map_single(dma_dev, buf, reg->size,
		445	DMA_TO_DEVICE);
		446	if (dma_mapping_error(dma_dev, src_phys_addr)) {
		447	dev_err(dev, "Failed to map source buffer addr\n");
		448	ret = -ENOMEM;
		449	goto err_dma_map;
		450	}
		451
		452	ktime_get_ts64(&start);
		453	ret = pci_epf_test_data_transfer(epf_test, phys_addr,
		454	src_phys_addr, reg->size);
		455	if (ret)
		456	dev_err(dev, "Data transfer failed\n");
		457	ktime_get_ts64(&end);
		458
		459	dma_unmap_single(dma_dev, src_phys_addr, reg->size,
		460	DMA_TO_DEVICE);
		461	} else {
		462	ktime_get_ts64(&start);
		463	memcpy_toio(dst_addr, buf, reg->size);
		464	ktime_get_ts64(&end);
		465	}
		466
		467	pci_epf_test_print_rate("WRITE", reg->size, &start, &end, use_dma);
228		468
229	/*	469	/*
230	* wait 1ms inorder for the write to complete. Without this delay L3	470	* wait 1ms inorder for the write to complete. Without this delay L3
@@ -232,6 +472,7 @@ static int pci_epf_test_write(struct pci_epf_test *epf_test) · 1 unmodified line
232	*/	472	*/
233	usleep_range(1000, 2000);	473	usleep_range(1000, 2000);
234		474
		475	err_dma_map:
235	kfree(buf);	476	kfree(buf);
236		477
237	err_map_addr:	478	err_map_addr:
@@ -360,14 +601,6 @@ reset_handler: · 122 unmodified lines
360	msecs_to_jiffies(1));	601	msecs_to_jiffies(1));
361	}	602	}
362		603
363	static void pci_epf_test_linkup(struct pci_epf *epf)
364	{
365	struct pci_epf_test *epf_test = epf_get_drvdata(epf);
366
367	queue_delayed_work(kpcitest_workqueue, &epf_test->cmd_handler,
368	msecs_to_jiffies(1));
369	}
370
371	static void pci_epf_test_unbind(struct pci_epf *epf)	604	static void pci_epf_test_unbind(struct pci_epf *epf)
372	{	605	{
373	struct pci_epf_test *epf_test = epf_get_drvdata(epf);	606	struct pci_epf_test *epf_test = epf_get_drvdata(epf);
@@ -376,6 +609,7 @@ static void pci_epf_test_unbind(struct pci_epf *epf) · 2 unmodified lines
376	int bar;	609	int bar;
377		610
378	cancel_delayed_work(&epf_test->cmd_handler);	611	cancel_delayed_work(&epf_test->cmd_handler);
		612	pci_epf_test_clean_dma_chan(epf_test);
379	pci_epc_stop(epc);	613	pci_epc_stop(epc);
380	for (bar = 0; bar < PCI_STD_NUM_BARS; bar++) {	614	for (bar = 0; bar < PCI_STD_NUM_BARS; bar++) {
381	epf_bar = &epf->bar[bar];	615	epf_bar = &epf->bar[bar];
@@ -424,11 +658,90 @@ static int pci_epf_test_set_bar(struct pci_epf *epf) · 42 unmodified lines
424	return 0;	658	return 0;
425	}	659	}
426		660
		661	static int pci_epf_test_core_init(struct pci_epf *epf)
		662	{
		663	struct pci_epf_test *epf_test = epf_get_drvdata(epf);
		664	struct pci_epf_header *header = epf->header;
		665	const struct pci_epc_features *epc_features;
		666	struct pci_epc *epc = epf->epc;
		667	struct device *dev = &epf->dev;
		668	bool msix_capable = false;
		669	bool msi_capable = true;
		670	int ret;
		671
		672	epc_features = pci_epc_get_features(epc, epf->func_no);
		673	if (epc_features) {
		674	msix_capable = epc_features->msix_capable;
		675	msi_capable = epc_features->msi_capable;
		676	}
		677
		678	ret = pci_epc_write_header(epc, epf->func_no, header);
		679	if (ret) {
		680	dev_err(dev, "Configuration header write failed\n");
		681	return ret;
		682	}
		683
		684	ret = pci_epf_test_set_bar(epf);
		685	if (ret)
		686	return ret;
		687
		688	if (msi_capable) {
		689	ret = pci_epc_set_msi(epc, epf->func_no, epf->msi_interrupts);
		690	if (ret) {
		691	dev_err(dev, "MSI configuration failed\n");
		692	return ret;
		693	}
		694	}
		695
		696	if (msix_capable) {
		697	ret = pci_epc_set_msix(epc, epf->func_no, epf->msix_interrupts,
		698	epf_test->test_reg_bar,
		699	epf_test->msix_table_offset);
		700	if (ret) {
		701	dev_err(dev, "MSI-X configuration failed\n");
		702	return ret;
		703	}
		704	}
		705
		706	return 0;
		707	}
		708
		709	static int pci_epf_test_notifier(struct notifier_block *nb, unsigned long val,
		710	void *data)
		711	{
		712	struct pci_epf *epf = container_of(nb, struct pci_epf, nb);
		713	struct pci_epf_test *epf_test = epf_get_drvdata(epf);
		714	int ret;
		715
		716	switch (val) {
		717	case CORE_INIT:
		718	ret = pci_epf_test_core_init(epf);
		719	if (ret)
		720	return NOTIFY_BAD;
		721	break;
		722
		723	case LINK_UP:
		724	queue_delayed_work(kpcitest_workqueue, &epf_test->cmd_handler,
		725	msecs_to_jiffies(1));
		726	break;
		727
		728	default:
		729	dev_err(&epf->dev, "Invalid EPF test notifier event\n");
		730	return NOTIFY_BAD;
		731	}
		732
		733	return NOTIFY_OK;
		734	}
		735
427	static int pci_epf_test_alloc_space(struct pci_epf *epf)	736	static int pci_epf_test_alloc_space(struct pci_epf *epf)
428	{	737	{
429	struct pci_epf_test *epf_test = epf_get_drvdata(epf);	738	struct pci_epf_test *epf_test = epf_get_drvdata(epf);
430	struct device *dev = &epf->dev;	739	struct device *dev = &epf->dev;
431	struct pci_epf_bar *epf_bar;	740	struct pci_epf_bar *epf_bar;
		741	size_t msix_table_size = 0;
		742	size_t test_reg_bar_size;
		743	size_t pba_size = 0;
		744	bool msix_capable;
432	void *base;	745	void *base;
433	int bar, add;	746	int bar, add;
434	enum pci_barno test_reg_bar = epf_test->test_reg_bar;	747	enum pci_barno test_reg_bar = epf_test->test_reg_bar;
@@ -437,13 +750,25 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf) · 2 unmodified lines
437		750
438	epc_features = epf_test->epc_features;	751	epc_features = epf_test->epc_features;
439		752
440	if (epc_features->bar_fixed_size[test_reg_bar])	753	test_reg_bar_size = ALIGN(sizeof(struct pci_epf_test_reg), 128);
		754
		755	msix_capable = epc_features->msix_capable;
		756	if (msix_capable) {
		757	msix_table_size = PCI_MSIX_ENTRY_SIZE * epf->msix_interrupts;
		758	epf_test->msix_table_offset = test_reg_bar_size;
		759	/* Align to QWORD or 8 Bytes */
		760	pba_size = ALIGN(DIV_ROUND_UP(epf->msix_interrupts, 8), 8);
		761	}
		762	test_reg_size = test_reg_bar_size + msix_table_size + pba_size;
		763
		764	if (epc_features->bar_fixed_size[test_reg_bar]) {
		765	if (test_reg_size > bar_size[test_reg_bar])
		766	return -ENOMEM;
441	test_reg_size = bar_size[test_reg_bar];	767	test_reg_size = bar_size[test_reg_bar];
442	else	768	}
443	test_reg_size = sizeof(struct pci_epf_test_reg);
444		769
445	base = pci_epf_alloc_space(epf, test_reg_size,	770	base = pci_epf_alloc_space(epf, test_reg_size, test_reg_bar,
446	test_reg_bar, epc_features->align);	771	epc_features->align);
447	if (!base) {	772	if (!base) {
448	dev_err(dev, "Failed to allocated register space\n");	773	dev_err(dev, "Failed to allocated register space\n");
449	return -ENOMEM;	774	return -ENOMEM;
@@ -492,14 +817,11 @@ static int pci_epf_test_bind(struct pci_epf *epf) · 42 unmodified lines
492	{	817	{
493	int ret;	818	int ret;
494	struct pci_epf_test *epf_test = epf_get_drvdata(epf);	819	struct pci_epf_test *epf_test = epf_get_drvdata(epf);
495	struct pci_epf_header *header = epf->header;
496	const struct pci_epc_features *epc_features;	820	const struct pci_epc_features *epc_features;
497	enum pci_barno test_reg_bar = BAR_0;	821	enum pci_barno test_reg_bar = BAR_0;
498	struct pci_epc *epc = epf->epc;	822	struct pci_epc *epc = epf->epc;
499	struct device *dev = &epf->dev;
500	bool linkup_notifier = false;	823	bool linkup_notifier = false;
501	bool msix_capable = false;	824	bool core_init_notifier = false;
502	bool msi_capable = true;
503		825
504	if (WARN_ON_ONCE(!epc))	826	if (WARN_ON_ONCE(!epc))
505	return -EINVAL;	827	return -EINVAL;
@@ -507,8 +829,7 @@ static int pci_epf_test_bind(struct pci_epf *epf) · 1 unmodified line
507	epc_features = pci_epc_get_features(epc, epf->func_no);	829	epc_features = pci_epc_get_features(epc, epf->func_no);
508	if (epc_features) {	830	if (epc_features) {
509	linkup_notifier = epc_features->linkup_notifier;	831	linkup_notifier = epc_features->linkup_notifier;
510	msix_capable = epc_features->msix_capable;	832	core_init_notifier = epc_features->core_init_notifier;
511	msi_capable = epc_features->msi_capable;
512	test_reg_bar = pci_epc_get_first_free_bar(epc_features);	833	test_reg_bar = pci_epc_get_first_free_bar(epc_features);
513	pci_epf_configure_bar(epf, epc_features);	834	pci_epf_configure_bar(epf, epc_features);
514	}	835	}
@@ -516,38 +837,28 @@ static int pci_epf_test_bind(struct pci_epf *epf) · 1 unmodified line
516	epf_test->test_reg_bar = test_reg_bar;	837	epf_test->test_reg_bar = test_reg_bar;
517	epf_test->epc_features = epc_features;	838	epf_test->epc_features = epc_features;
518		839
519	ret = pci_epc_write_header(epc, epf->func_no, header);
520	if (ret) {
521	dev_err(dev, "Configuration header write failed\n");
522	return ret;
523	}
524
525	ret = pci_epf_test_alloc_space(epf);	840	ret = pci_epf_test_alloc_space(epf);
526	if (ret)	841	if (ret)
527	return ret;	842	return ret;
528		843
529	ret = pci_epf_test_set_bar(epf);	844	if (!core_init_notifier) {
530	if (ret)	845	ret = pci_epf_test_core_init(epf);
531	return ret;	846	if (ret)
532
533	if (msi_capable) {
534	ret = pci_epc_set_msi(epc, epf->func_no, epf->msi_interrupts);
535	if (ret) {
536	dev_err(dev, "MSI configuration failed\n");
537	return ret;	847	return ret;
538	}
539	}	848	}
540		849
541	if (msix_capable) {	850	epf_test->dma_supported = true;
542	ret = pci_epc_set_msix(epc, epf->func_no, epf->msix_interrupts);
543	if (ret) {
544	dev_err(dev, "MSI-X configuration failed\n");
545	return ret;
546	}
547	}
548		851
549	if (!linkup_notifier)	852	ret = pci_epf_test_init_dma_chan(epf_test);
		853	if (ret)
		854	epf_test->dma_supported = false;
		855
		856	if (linkup_notifier) {
		857	epf->nb.notifier_call = pci_epf_test_notifier;
		858	pci_epc_register_notifier(epc, &epf->nb);
		859	} else {
550	queue_work(kpcitest_workqueue, &epf_test->cmd_handler.work);	860	queue_work(kpcitest_workqueue, &epf_test->cmd_handler.work);
		861	}
551		862
552	return 0;	863	return 0;
553	}	864	}
@@ -580,7 +891,6 @@ static int pci_epf_test_probe(struct pci_epf *epf) · 26 unmodified lines
580	static struct pci_epf_ops ops = {	891	static struct pci_epf_ops ops = {
581	.unbind = pci_epf_test_unbind,	892	.unbind = pci_epf_test_unbind,
582	.bind = pci_epf_test_bind,	893	.bind = pci_epf_test_bind,
583	.linkup = pci_epf_test_linkup,
584	};	894	};
585		895
586	static struct pci_epf_driver test_driver = {	896	static struct pci_epf_driver test_driver = {

drivers/pci/endpoint/pci-ep-cfs.c+6−22

 	struct config_group group;
 	struct pci_epc *epc;
 	bool start;
-	unsigned long function_num_map;
 };
 static inline struct pci_epf_group *to_pci_epf_group(struct config_item *item)
 	if (!start) {
 		pci_epc_stop(epc);
+		epc_group->start = 0;
 		return len;
 	}
 			    struct config_item *epf_item)
 {
 	int ret;
-	u32 func_no = 0;
 	struct pci_epf_group *epf_group = to_pci_epf_group(epf_item);
 	struct pci_epc_group *epc_group = to_pci_epc_group(epc_item);
 	struct pci_epc *epc = epc_group->epc;
 	struct pci_epf *epf = epf_group->epf;
-	func_no = find_first_zero_bit(&epc_group->function_num_map,
-				      BITS_PER_LONG);
-	if (func_no >= BITS_PER_LONG)
-		return -EINVAL;
-	set_bit(func_no, &epc_group->function_num_map);
-	epf->func_no = func_no;
 	ret = pci_epc_add_epf(epc, epf);
 	if (ret)
-		goto err_add_epf;
+		return ret;
 	ret = pci_epf_bind(epf);
-	if (ret)
+	if (ret) {
-		goto err_epf_bind;
+		pci_epc_remove_epf(epc, epf);
+		return ret;
+	}
 	return 0;
-err_epf_bind:
-	pci_epc_remove_epf(epc, epf);
-err_add_epf:
-	clear_bit(func_no, &epc_group->function_num_map);
-	return ret;
 }
 static void pci_epc_epf_unlink(struct config_item *epc_item,
 	epc = epc_group->epc;
 	epf = epf_group->epf;
-	clear_bit(epf->func_no, &epc_group->function_num_map);
 	pci_epf_unbind(epf);
 	pci_epc_remove_epf(epc, epf);
 }

drivers/pci/endpoint/pci-epc-core.c+75−62

 						    u8 func_no)
 {
 	const struct pci_epc_features *epc_features;
-	unsigned long flags;
 	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
 		return NULL;
 	if (!epc->ops->get_features)
 		return NULL;
-	spin_lock_irqsave(&epc->lock, flags);
+	mutex_lock(&epc->lock);
 	epc_features = epc->ops->get_features(epc, func_no);
-	spin_unlock_irqrestore(&epc->lock, flags);
+	mutex_unlock(&epc->lock);
 	return epc_features;
 }
  */
 void pci_epc_stop(struct pci_epc *epc)
 {
-	unsigned long flags;
 	if (IS_ERR(epc) || !epc->ops->stop)
 		return;
-	spin_lock_irqsave(&epc->lock, flags);
+	mutex_lock(&epc->lock);
 	epc->ops->stop(epc);
-	spin_unlock_irqrestore(&epc->lock, flags);
+	mutex_unlock(&epc->lock);
 }
 EXPORT_SYMBOL_GPL(pci_epc_stop);
 int pci_epc_start(struct pci_epc *epc)
 {
 	int ret;
-	unsigned long flags;
 	if (IS_ERR(epc))
 		return -EINVAL;
 	if (!epc->ops->start)
 		return 0;
-	spin_lock_irqsave(&epc->lock, flags);
+	mutex_lock(&epc->lock);
 	ret = epc->ops->start(epc);
-	spin_unlock_irqrestore(&epc->lock, flags);
+	mutex_unlock(&epc->lock);
 	return ret;
 }
 		      enum pci_epc_irq_type type, u16 interrupt_num)
 {
 	int ret;
-	unsigned long flags;
 	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
 		return -EINVAL;
 	if (!epc->ops->raise_irq)
 		return 0;
-	spin_lock_irqsave(&epc->lock, flags);
+	mutex_lock(&epc->lock);
 	ret = epc->ops->raise_irq(epc, func_no, type, interrupt_num);
-	spin_unlock_irqrestore(&epc->lock, flags);
+	mutex_unlock(&epc->lock);
 	return ret;
 }
 int pci_epc_get_msi(struct pci_epc *epc, u8 func_no)
 {
 	int interrupt;
-	unsigned long flags;
 	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
 		return 0;
 	if (!epc->ops->get_msi)
 		return 0;
-	spin_lock_irqsave(&epc->lock, flags);
+	mutex_lock(&epc->lock);
 	interrupt = epc->ops->get_msi(epc, func_no);
-	spin_unlock_irqrestore(&epc->lock, flags);
+	mutex_unlock(&epc->lock);
 	if (interrupt < 0)
 		return 0;
 {
 	int ret;
 	u8 encode_int;
-	unsigned long flags;
 	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions ||
 	    interrupts > 32)
 	encode_int = order_base_2(interrupts);
-	spin_lock_irqsave(&epc->lock, flags);
+	mutex_lock(&epc->lock);
 	ret = epc->ops->set_msi(epc, func_no, encode_int);
-	spin_unlock_irqrestore(&epc->lock, flags);
+	mutex_unlock(&epc->lock);
 	return ret;
 }
 int pci_epc_get_msix(struct pci_epc *epc, u8 func_no)
 {
 	int interrupt;
-	unsigned long flags;
 	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
 		return 0;
 	if (!epc->ops->get_msix)
 		return 0;
-	spin_lock_irqsave(&epc->lock, flags);
+	mutex_lock(&epc->lock);
 	interrupt = epc->ops->get_msix(epc, func_no);
-	spin_unlock_irqrestore(&epc->lock, flags);
+	mutex_unlock(&epc->lock);
 	if (interrupt < 0)
 		return 0;
  * @epc: the EPC device on which MSI-X has to be configured
  * @func_no: the endpoint function number in the EPC device
  * @interrupts: number of MSI-X interrupts required by the EPF
+ * @bir: BAR where the MSI-X table resides
+ * @offset: Offset pointing to the start of MSI-X table
  *
  * Invoke to set the required number of MSI-X interrupts.
  */
-int pci_epc_set_msix(struct pci_epc *epc, u8 func_no, u16 interrupts)
+int pci_epc_set_msix(struct pci_epc *epc, u8 func_no, u16 interrupts,
+		     enum pci_barno bir, u32 offset)
 {
 	int ret;
-	unsigned long flags;
 	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions ||
 	    interrupts < 1 || interrupts > 2048)
 	if (!epc->ops->set_msix)
 		return 0;
-	spin_lock_irqsave(&epc->lock, flags);
+	mutex_lock(&epc->lock);
-	ret = epc->ops->set_msix(epc, func_no, interrupts - 1);
+	ret = epc->ops->set_msix(epc, func_no, interrupts - 1, bir, offset);
-	spin_unlock_irqrestore(&epc->lock, flags);
+	mutex_unlock(&epc->lock);
 	return ret;
 }
 void pci_epc_unmap_addr(struct pci_epc *epc, u8 func_no,
 			phys_addr_t phys_addr)
 {
-	unsigned long flags;
 	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
 		return;
 	if (!epc->ops->unmap_addr)
 		return;
-	spin_lock_irqsave(&epc->lock, flags);
+	mutex_lock(&epc->lock);
 	epc->ops->unmap_addr(epc, func_no, phys_addr);
-	spin_unlock_irqrestore(&epc->lock, flags);
+	mutex_unlock(&epc->lock);
 }
 EXPORT_SYMBOL_GPL(pci_epc_unmap_addr);
 		     phys_addr_t phys_addr, u64 pci_addr, size_t size)
 {
 	int ret;
-	unsigned long flags;
 	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
 		return -EINVAL;
 	if (!epc->ops->map_addr)
 		return 0;
-	spin_lock_irqsave(&epc->lock, flags);
+	mutex_lock(&epc->lock);
 	ret = epc->ops->map_addr(epc, func_no, phys_addr, pci_addr, size);
-	spin_unlock_irqrestore(&epc->lock, flags);
+	mutex_unlock(&epc->lock);
 	return ret;
 }
 void pci_epc_clear_bar(struct pci_epc *epc, u8 func_no,
 		       struct pci_epf_bar *epf_bar)
 {
-	unsigned long flags;
 	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions ||
 	    (epf_bar->barno == BAR_5 &&
 	     epf_bar->flags & PCI_BASE_ADDRESS_MEM_TYPE_64))
 	if (!epc->ops->clear_bar)
 		return;
-	spin_lock_irqsave(&epc->lock, flags);
+	mutex_lock(&epc->lock);
 	epc->ops->clear_bar(epc, func_no, epf_bar);
-	spin_unlock_irqrestore(&epc->lock, flags);
+	mutex_unlock(&epc->lock);
 }
 EXPORT_SYMBOL_GPL(pci_epc_clear_bar);
 		    struct pci_epf_bar *epf_bar)
 {
 	int ret;
-	unsigned long irq_flags;
 	int flags = epf_bar->flags;
 	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions ||
 	if (!epc->ops->set_bar)
 		return 0;
-	spin_lock_irqsave(&epc->lock, irq_flags);
+	mutex_lock(&epc->lock);
 	ret = epc->ops->set_bar(epc, func_no, epf_bar);
-	spin_unlock_irqrestore(&epc->lock, irq_flags);
+	mutex_unlock(&epc->lock);
 	return ret;
 }
 			 struct pci_epf_header *header)
 {
 	int ret;
-	unsigned long flags;
 	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
 		return -EINVAL;
 	if (!epc->ops->write_header)
 		return 0;
-	spin_lock_irqsave(&epc->lock, flags);
+	mutex_lock(&epc->lock);
 	ret = epc->ops->write_header(epc, func_no, header);
-	spin_unlock_irqrestore(&epc->lock, flags);
+	mutex_unlock(&epc->lock);
 	return ret;
 }
  */
 int pci_epc_add_epf(struct pci_epc *epc, struct pci_epf *epf)
 {
-	unsigned long flags;
+	u32 func_no;
+	int ret = 0;
 	if (epf->epc)
 		return -EBUSY;
 	if (IS_ERR(epc))
 		return -EINVAL;
-	if (epf->func_no > epc->max_functions - 1)
+	mutex_lock(&epc->lock);
-		return -EINVAL;
+	func_no = find_first_zero_bit(&epc->function_num_map,
+				      BITS_PER_LONG);
+	if (func_no >= BITS_PER_LONG) {
+		ret = -EINVAL;
+		goto ret;
+	}
+	if (func_no > epc->max_functions - 1) {
+		dev_err(&epc->dev, "Exceeding max supported Function Number\n");
+		ret = -EINVAL;
+		goto ret;
+	}
+	set_bit(func_no, &epc->function_num_map);
+	epf->func_no = func_no;
 	epf->epc = epc;
-	spin_lock_irqsave(&epc->lock, flags);
 	list_add_tail(&epf->list, &epc->pci_epf);
-	spin_unlock_irqrestore(&epc->lock, flags);
-	return 0;
+ret:
+	mutex_unlock(&epc->lock);
+	return ret;
 }
 EXPORT_SYMBOL_GPL(pci_epc_add_epf);
  */
 void pci_epc_remove_epf(struct pci_epc *epc, struct pci_epf *epf)
 {
-	unsigned long flags;
 	if (!epc || IS_ERR(epc) || !epf)
 		return;
-	spin_lock_irqsave(&epc->lock, flags);
+	mutex_lock(&epc->lock);
+	clear_bit(epf->func_no, &epc->function_num_map);
 	list_del(&epf->list);
 	epf->epc = NULL;
-	spin_unlock_irqrestore(&epc->lock, flags);
+	mutex_unlock(&epc->lock);
 }
 EXPORT_SYMBOL_GPL(pci_epc_remove_epf);
  */
 void pci_epc_linkup(struct pci_epc *epc)
 {
-	unsigned long flags;
-	struct pci_epf *epf;
 	if (!epc || IS_ERR(epc))
 		return;
-	spin_lock_irqsave(&epc->lock, flags);
+	atomic_notifier_call_chain(&epc->notifier, LINK_UP, NULL);
-	list_for_each_entry(epf, &epc->pci_epf, list)
-		pci_epf_linkup(epf);
-	spin_unlock_irqrestore(&epc->lock, flags);
 }
 EXPORT_SYMBOL_GPL(pci_epc_linkup);
+/**
+ * pci_epc_init_notify() - Notify the EPF device that EPC device's core
+ *			   initialization is completed.
+ * @epc: the EPC device whose core initialization is completeds
+ *
+ * Invoke to Notify the EPF device that the EPC device's initialization
+ * is completed.
+ */
+void pci_epc_init_notify(struct pci_epc *epc)
+{
+	if (!epc || IS_ERR(epc))
+		return;
+	atomic_notifier_call_chain(&epc->notifier, CORE_INIT, NULL);
+}
+EXPORT_SYMBOL_GPL(pci_epc_init_notify);
 /**
  * pci_epc_destroy() - destroy the EPC device
  * @epc: the EPC device that has to be destroyed
 		goto err_ret;
 	}
-	spin_lock_init(&epc->lock);
+	mutex_init(&epc->lock);
 	INIT_LIST_HEAD(&epc->pci_epf);
+	ATOMIC_INIT_NOTIFIER_HEAD(&epc->notifier);
 	device_initialize(&epc->dev);
 	epc->dev.class = pci_epc_class;

drivers/pci/endpoint/pci-epc-mem.c+8−2

 	mem->page_size = page_size;
 	mem->pages = pages;
 	mem->size = size;
+	mutex_init(&mem->lock);
 	epc->mem = mem;
 				     phys_addr_t *phys_addr, size_t size)
 {
 	int pageno;
-	void __iomem *virt_addr;
+	void __iomem *virt_addr = NULL;
 	struct pci_epc_mem *mem = epc->mem;
 	unsigned int page_shift = ilog2(mem->page_size);
 	int order;
 	size = ALIGN(size, mem->page_size);
 	order = pci_epc_mem_get_order(mem, size);
+	mutex_lock(&mem->lock);
 	pageno = bitmap_find_free_region(mem->bitmap, mem->pages, order);
 	if (pageno < 0)
-		return NULL;
+		goto ret;
 	*phys_addr = mem->phys_base + ((phys_addr_t)pageno << page_shift);
 	virt_addr = ioremap(*phys_addr, size);
 	if (!virt_addr)
 		bitmap_release_region(mem->bitmap, pageno, order);
+ret:
+	mutex_unlock(&mem->lock);
 	return virt_addr;
 }
 EXPORT_SYMBOL_GPL(pci_epc_mem_alloc_addr);
 	pageno = (phys_addr - mem->phys_base) >> page_shift;
 	size = ALIGN(size, mem->page_size);
 	order = pci_epc_mem_get_order(mem, size);
+	mutex_lock(&mem->lock);
 	bitmap_release_region(mem->bitmap, pageno, order);
+	mutex_unlock(&mem->lock);
 }
 EXPORT_SYMBOL_GPL(pci_epc_mem_free_addr);

drivers/pci/endpoint/pci-epf-core.c+13−22

 static struct bus_type pci_epf_bus_type;
 static const struct device_type pci_epf_type;
-/**
- * pci_epf_linkup() - Notify the function driver that EPC device has
- *		      established a connection with the Root Complex.
- * @epf: the EPF device bound to the EPC device which has established
- *	 the connection with the host
- *
- * Invoke to notify the function driver that EPC device has established
- * a connection with the Root Complex.
- */
-void pci_epf_linkup(struct pci_epf *epf)
-{
-	if (!epf->driver) {
-		dev_WARN(&epf->dev, "epf device not bound to driver\n");
-		return;
-	}
-	epf->driver->ops->linkup(epf);
-}
-EXPORT_SYMBOL_GPL(pci_epf_linkup);
 /**
  * pci_epf_unbind() - Notify the function driver that the binding between the
  *		      EPF device and EPC device has been lost
 		return;
 	}
+	mutex_lock(&epf->lock);
 	epf->driver->ops->unbind(epf);
+	mutex_unlock(&epf->lock);
 	module_put(epf->driver->owner);
 }
 EXPORT_SYMBOL_GPL(pci_epf_unbind);
  */
 int pci_epf_bind(struct pci_epf *epf)
 {
+	int ret;
 	if (!epf->driver) {
 		dev_WARN(&epf->dev, "epf device not bound to driver\n");
 		return -EINVAL;
 	if (!try_module_get(epf->driver->owner))
 		return -EAGAIN;
-	return epf->driver->ops->bind(epf);
+	mutex_lock(&epf->lock);
+	ret = epf->driver->ops->bind(epf);
+	mutex_unlock(&epf->lock);
+	return ret;
 }
 EXPORT_SYMBOL_GPL(pci_epf_bind);
 			  epf->bar[bar].phys_addr);
 	epf->bar[bar].phys_addr = 0;
+	epf->bar[bar].addr = NULL;
 	epf->bar[bar].size = 0;
 	epf->bar[bar].barno = 0;
 	epf->bar[bar].flags = 0;
 	}
 	epf->bar[bar].phys_addr = phys_addr;
+	epf->bar[bar].addr = space;
 	epf->bar[bar].size = size;
 	epf->bar[bar].barno = bar;
 	epf->bar[bar].flags |= upper_32_bits(size) ?
 	if (!driver->ops)
 		return -EINVAL;
-	if (!driver->ops->bind || !driver->ops->unbind || !driver->ops->linkup)
+	if (!driver->ops->bind || !driver->ops->unbind)
 		return -EINVAL;
 	driver->driver.bus = &pci_epf_bus_type;
 	device_initialize(dev);
 	dev->bus = &pci_epf_bus_type;
 	dev->type = &pci_epf_type;
+	mutex_init(&epf->lock);
 	ret = dev_set_name(dev, "%s", name);
 	if (ret) {

include/linux/pci-epc.h+22−5

 			      phys_addr_t addr);
 	int	(*set_msi)(struct pci_epc *epc, u8 func_no, u8 interrupts);
 	int	(*get_msi)(struct pci_epc *epc, u8 func_no);
-	int	(*set_msix)(struct pci_epc *epc, u8 func_no, u16 interrupts);
+	int	(*set_msix)(struct pci_epc *epc, u8 func_no, u16 interrupts,
+			    enum pci_barno, u32 offset);
 	int	(*get_msix)(struct pci_epc *epc, u8 func_no);
 	int	(*raise_irq)(struct pci_epc *epc, u8 func_no,
 			     enum pci_epc_irq_type type, u16 interrupt_num);
  * @bitmap: bitmap to manage the PCI address space
  * @pages: number of bits representing the address region
  * @page_size: size of each page
+ * @lock: mutex to protect bitmap
  */
 struct pci_epc_mem {
 	phys_addr_t	phys_base;
 	unsigned long	*bitmap;
 	size_t		page_size;
 	int		pages;
+	/* mutex to protect against concurrent access for memory allocation*/
+	struct mutex	lock;
 };
 /**
  * @mem: address space of the endpoint controller
  * @max_functions: max number of functions that can be configured in this EPC
  * @group: configfs group representing the PCI EPC device
- * @lock: spinlock to protect pci_epc ops
+ * @lock: mutex to protect pci_epc ops
+ * @function_num_map: bitmap to manage physical function number
+ * @notifier: used to notify EPF of any EPC events (like linkup)
  */
 struct pci_epc {
 	struct device			dev;
 	struct pci_epc_mem		*mem;
 	u8				max_functions;
 	struct config_group		*group;
-	/* spinlock to protect against concurrent access of EP controller */
+	/* mutex to protect against concurrent access of EP controller */
-	spinlock_t			lock;
+	struct mutex			lock;
+	unsigned long			function_num_map;
+	struct atomic_notifier_head	notifier;
 };
 /**
  */
 struct pci_epc_features {
 	unsigned int	linkup_notifier : 1;
+	unsigned int	core_init_notifier : 1;
 	unsigned int	msi_capable : 1;
 	unsigned int	msix_capable : 1;
 	u8	reserved_bar;
 	return dev_get_drvdata(&epc->dev);
 }
+static inline int
+pci_epc_register_notifier(struct pci_epc *epc, struct notifier_block *nb)
+{
+	return atomic_notifier_chain_register(&epc->notifier, nb);
+}
 struct pci_epc *
 __devm_pci_epc_create(struct device *dev, const struct pci_epc_ops *ops,
 		      struct module *owner);
 void pci_epc_destroy(struct pci_epc *epc);
 int pci_epc_add_epf(struct pci_epc *epc, struct pci_epf *epf);
 void pci_epc_linkup(struct pci_epc *epc);
+void pci_epc_init_notify(struct pci_epc *epc);
 void pci_epc_remove_epf(struct pci_epc *epc, struct pci_epf *epf);
 int pci_epc_write_header(struct pci_epc *epc, u8 func_no,
 			 struct pci_epf_header *hdr);
 			phys_addr_t phys_addr);
 int pci_epc_set_msi(struct pci_epc *epc, u8 func_no, u8 interrupts);
 int pci_epc_get_msi(struct pci_epc *epc, u8 func_no);
-int pci_epc_set_msix(struct pci_epc *epc, u8 func_no, u16 interrupts);
+int pci_epc_set_msix(struct pci_epc *epc, u8 func_no, u16 interrupts,
+		     enum pci_barno, u32 offset);
 int pci_epc_get_msix(struct pci_epc *epc, u8 func_no);
 int pci_epc_raise_irq(struct pci_epc *epc, u8 func_no,
 		      enum pci_epc_irq_type type, u16 interrupt_num);

include/linux/pci-epf.h+25−4

 struct pci_epf;
+enum pci_notify_event {
+	CORE_INIT,
+	LINK_UP,
+};
 enum pci_barno {
 	BAR_0,
 	BAR_1,
  * @bind: ops to perform when a EPC device has been bound to EPF device
  * @unbind: ops to perform when a binding has been lost between a EPC device
  *	    and EPF device
- * @linkup: ops to perform when the EPC device has established a connection with
- *	    a host system
  */
 struct pci_epf_ops {
 	int	(*bind)(struct pci_epf *epf);
 	void	(*unbind)(struct pci_epf *epf);
-	void	(*linkup)(struct pci_epf *epf);
 };
 /**
 /**
  * struct pci_epf_bar - represents the BAR of EPF device
  * @phys_addr: physical address that should be mapped to the BAR
+ * @addr: virtual address corresponding to the @phys_addr
  * @size: the size of the address space present in BAR
  */
 struct pci_epf_bar {
 	dma_addr_t	phys_addr;
+	void		*addr;
 	size_t		size;
 	enum pci_barno	barno;
 	int		flags;
  * @epc: the EPC device to which this EPF device is bound
  * @driver: the EPF driver to which this EPF device is bound
  * @list: to add pci_epf as a list of PCI endpoint functions to pci_epc
+ * @nb: notifier block to notify EPF of any EPC events (like linkup)
+ * @lock: mutex to protect pci_epf_ops
  */
 struct pci_epf {
 	struct device		dev;
 	struct pci_epc		*epc;
 	struct pci_epf_driver	*driver;
 	struct list_head	list;
+	struct notifier_block   nb;
+	/* mutex to protect against concurrent access of pci_epf_ops */
+	struct mutex		lock;
+};
+/**
+ * struct pci_epf_msix_tbl - represents the MSIX table entry structure
+ * @msg_addr: Writes to this address will trigger MSIX interrupt in host
+ * @msg_data: Data that should be written to @msg_addr to trigger MSIX interrupt
+ * @vector_ctrl: Identifies if the function is prohibited from sending a message
+ * using this MSIX table entry
+ */
+struct pci_epf_msix_tbl {
+	u64 msg_addr;
+	u32 msg_data;
+	u32 vector_ctrl;
 };
 #define to_pci_epf(epf_dev) container_of((epf_dev), struct pci_epf, dev)
 void pci_epf_free_space(struct pci_epf *epf, void *addr, enum pci_barno bar);
 int pci_epf_bind(struct pci_epf *epf);
 void pci_epf_unbind(struct pci_epf *epf);
-void pci_epf_linkup(struct pci_epf *epf);
 #endif /* __LINUX_PCI_EPF_H */

include/soc/tegra/bpmp-abi.h+9−1

 /* SPDX-License-Identifier: GPL-2.0-only */
 /*
- * Copyright (c) 2014-2018, NVIDIA CORPORATION.  All rights reserved.
+ * Copyright (c) 2014-2020, NVIDIA CORPORATION.  All rights reserved.
  */
 #ifndef _ABI_BPMP_ABI_H_
 	CMD_UPHY_PCIE_LANE_MARGIN_STATUS = 2,
 	CMD_UPHY_PCIE_EP_CONTROLLER_PLL_INIT = 3,
 	CMD_UPHY_PCIE_CONTROLLER_STATE = 4,
+	CMD_UPHY_PCIE_EP_CONTROLLER_PLL_OFF = 5,
 	CMD_UPHY_MAX,
 };
 	uint8_t enable;
 } __ABI_PACKED;
+struct cmd_uphy_ep_controller_pll_off_request {
+	/** @brief EP controller number, valid: 0, 4, 5 */
+	uint8_t ep_controller;
+} __ABI_PACKED;
 /**
  * @ingroup UPHY
  * @brief Request with #MRQ_UPHY
  * |CMD_UPHY_PCIE_LANE_MARGIN_STATUS     |                                        |
  * |CMD_UPHY_PCIE_EP_CONTROLLER_PLL_INIT |cmd_uphy_ep_controller_pll_init_request |
  * |CMD_UPHY_PCIE_CONTROLLER_STATE       |cmd_uphy_pcie_controller_state_request  |
+ * |CMD_UPHY_PCIE_EP_CONTROLLER_PLL_OFF  |cmd_uphy_ep_controller_pll_off_request  |
  *
  */
 		struct cmd_uphy_margin_control_request uphy_set_margin_control;
 		struct cmd_uphy_ep_controller_pll_init_request ep_ctrlr_pll_init;
 		struct cmd_uphy_pcie_controller_state_request controller_state;
+		struct cmd_uphy_ep_controller_pll_off_request ep_ctrlr_pll_off;
 	} __UNION_ANON;
 } __ABI_PACKED;

include/uapi/linux/pcitest.h+8−0

@@ -19,5 +19,13 @@ · 18 unmodified lines
19	#define PCITEST_MSIX _IOW('P', 0x7, int)	19	#define PCITEST_MSIX _IOW('P', 0x7, int)
20	#define PCITEST_SET_IRQTYPE _IOW('P', 0x8, int)	20	#define PCITEST_SET_IRQTYPE _IOW('P', 0x8, int)
21	#define PCITEST_GET_IRQTYPE _IO('P', 0x9)	21	#define PCITEST_GET_IRQTYPE _IO('P', 0x9)
		22	#define PCITEST_CLEAR_IRQ _IO('P', 0x10)
		23
		24	#define PCITEST_FLAGS_USE_DMA 0x00000001
		25
		26	struct pci_endpoint_test_xfer_param {
		27	unsigned long size;
		28	unsigned char flags;
		29	};
22		30
23	#endif /* __UAPI_LINUX_PCITEST_H */	31	#endif /* __UAPI_LINUX_PCITEST_H */

tools/pci/pcitest.c+33−4

 	int		irqtype;
 	bool		set_irqtype;
 	bool		get_irqtype;
+	bool		clear_irq;
 	bool		read;
 	bool		write;
 	bool		copy;
 	unsigned long	size;
+	bool		use_dma;
 };
 static int run_test(struct pci_test *test)
 {
+	struct pci_endpoint_test_xfer_param param;
 	int ret = -EINVAL;
 	int fd;
 			fprintf(stdout, "%s\n", irq[ret]);
 	}
+	if (test->clear_irq) {
+		ret = ioctl(fd, PCITEST_CLEAR_IRQ);
+		fprintf(stdout, "CLEAR IRQ:\t\t");
+		if (ret < 0)
+			fprintf(stdout, "FAILED\n");
+		else
+			fprintf(stdout, "%s\n", result[ret]);
+	}
 	if (test->legacyirq) {
 		ret = ioctl(fd, PCITEST_LEGACY_IRQ, 0);
 		fprintf(stdout, "LEGACY IRQ:\t");
 	}
 	if (test->write) {
-		ret = ioctl(fd, PCITEST_WRITE, test->size);
+		param.size = test->size;
+		if (test->use_dma)
+			param.flags = PCITEST_FLAGS_USE_DMA;
+		ret = ioctl(fd, PCITEST_WRITE, &param);
 		fprintf(stdout, "WRITE (%7ld bytes):\t\t", test->size);
 		if (ret < 0)
 			fprintf(stdout, "TEST FAILED\n");
 	}
 	if (test->read) {
-		ret = ioctl(fd, PCITEST_READ, test->size);
+		param.size = test->size;
+		if (test->use_dma)
+			param.flags = PCITEST_FLAGS_USE_DMA;
+		ret = ioctl(fd, PCITEST_READ, &param);
 		fprintf(stdout, "READ (%7ld bytes):\t\t", test->size);
 		if (ret < 0)
 			fprintf(stdout, "TEST FAILED\n");
 	}
 	if (test->copy) {
-		ret = ioctl(fd, PCITEST_COPY, test->size);
+		param.size = test->size;
+		if (test->use_dma)
+			param.flags = PCITEST_FLAGS_USE_DMA;
+		ret = ioctl(fd, PCITEST_COPY, &param);
 		fprintf(stdout, "COPY (%7ld bytes):\t\t", test->size);
 		if (ret < 0)
 			fprintf(stdout, "TEST FAILED\n");
 	/* set default endpoint device */
 	test->device = "/dev/pci-endpoint-test.0";
-	while ((c = getopt(argc, argv, "D:b:m:x:i:Ilhrwcs:")) != EOF)
+	while ((c = getopt(argc, argv, "D:b:m:x:i:deIlhrwcs:")) != EOF)
 	switch (c) {
 	case 'D':
 		test->device = optarg;
 	case 'c':
 		test->copy = true;
 		continue;
+	case 'e':
+		test->clear_irq = true;
+		continue;
 	case 's':
 		test->size = strtoul(optarg, NULL, 0);
 		continue;
+	case 'd':
+		test->use_dma = true;
+		continue;
 	case 'h':
 	default:
 usage:
 			"\t-m <msi num>		MSI test (msi number between 1..32)\n"
 			"\t-x <msix num>	\tMSI-X test (msix number between 1..2048)\n"
 			"\t-i <irq type>	\tSet IRQ type (0 - Legacy, 1 - MSI, 2 - MSI-X)\n"
+			"\t-e			Clear IRQ\n"
 			"\t-I			Get current IRQ type configured\n"
+			"\t-d			Use DMA\n"
 			"\t-l			Legacy IRQ test\n"
 			"\t-r			Read buffer test\n"
 			"\t-w			Write buffer test\n"

Merge branch 'remotes/lorenzo/pci/endpoint'

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code