History

Merge tag 'drm-misc-next-2023-11-23' of git://anongit.freedesktop.org/drm/drm-misc into drm-next

drm-misc-next for 6.8: UAPI Changes: Cross-subsystem Changes: Core Changes: - Drop deprecated drm_kms_helper.edid_firmware module parameter Driver Changes: - Convert platform drivers remove callback to return void - imagination: Introduction of the Imagination GPU Support - rockchip: - rk3066_hdmi: Convert to atomic - vop2: Support NV20 and NV30 - panel: - elida-kd35t133: PM reworks - New panels: Powkiddy RK2023

Daniel Vetter committed 2 years ago

Commit b26ca732 parents c79b972 815d8b0115 files · +34629 −201Browse files

Unified · Split

115 files
  1. Documentation/devicetree/bindings/display/panel/newvision,nv3051d.yaml +1 −1
  2. Documentation/devicetree/bindings/gpu/img,powervr.yaml +73 −0
  3. Documentation/gpu/drivers.rst +2 −0
  4. Documentation/gpu/imagination/index.rst +13 −0
  5. Documentation/gpu/imagination/uapi.rst +174 −0
  6. MAINTAINERS +10 −0
  7. drivers/accel/qaic/qaic_data.c +1 −1
  8. drivers/dma-buf/dma-buf.c +2 −2
  9. drivers/gpu/drm/Kconfig +2 −0
  10. drivers/gpu/drm/Makefile +1 −0
  11. drivers/gpu/drm/armada/armada_crtc.c +2 −3
  12. drivers/gpu/drm/armada/armada_drv.c +2 −3
  13. drivers/gpu/drm/bridge/cadence/cdns-mhdp8546-core.c +10 −12
  14. drivers/gpu/drm/bridge/ti-tpd12s015.c +2 −4
  15. drivers/gpu/drm/drm_edid_load.c +0 −16
  16. drivers/gpu/drm/drm_kms_helper_common.c +0 −32
  17. drivers/gpu/drm/etnaviv/etnaviv_drv.c +2 −4
  18. drivers/gpu/drm/etnaviv/etnaviv_gpu.c +2 −3
  19. drivers/gpu/drm/imagination/Kconfig +18 −0
  20. drivers/gpu/drm/imagination/Makefile +35 −0
  21. drivers/gpu/drm/imagination/pvr_ccb.c +645 −0
  22. drivers/gpu/drm/imagination/pvr_ccb.h +71 −0
  23. drivers/gpu/drm/imagination/pvr_cccb.c +267 −0
  24. drivers/gpu/drm/imagination/pvr_cccb.h +109 −0
  25. drivers/gpu/drm/imagination/pvr_context.c +464 −0
  26. drivers/gpu/drm/imagination/pvr_context.h +205 −0
  27. drivers/gpu/drm/imagination/pvr_debugfs.c +53 −0
  28. drivers/gpu/drm/imagination/pvr_debugfs.h +29 −0
  29. drivers/gpu/drm/imagination/pvr_device.c +658 −0
  30. drivers/gpu/drm/imagination/pvr_device.h +710 −0
  31. drivers/gpu/drm/imagination/pvr_device_info.c +254 −0
  32. drivers/gpu/drm/imagination/pvr_device_info.h +186 −0
  33. drivers/gpu/drm/imagination/pvr_drv.c +1501 −0
  34. drivers/gpu/drm/imagination/pvr_drv.h +129 −0
  35. drivers/gpu/drm/imagination/pvr_free_list.c +625 −0
  36. drivers/gpu/drm/imagination/pvr_free_list.h +195 −0
  37. drivers/gpu/drm/imagination/pvr_fw.c +1489 −0
  38. drivers/gpu/drm/imagination/pvr_fw.h +508 −0
  39. drivers/gpu/drm/imagination/pvr_fw_info.h +135 −0
  40. drivers/gpu/drm/imagination/pvr_fw_meta.c +554 −0
  41. drivers/gpu/drm/imagination/pvr_fw_meta.h +14 −0
  42. drivers/gpu/drm/imagination/pvr_fw_mips.c +252 −0
  43. drivers/gpu/drm/imagination/pvr_fw_mips.h +48 −0
  44. drivers/gpu/drm/imagination/pvr_fw_startstop.c +306 −0
  45. drivers/gpu/drm/imagination/pvr_fw_startstop.h +13 −0
  46. drivers/gpu/drm/imagination/pvr_fw_trace.c +515 −0
  47. drivers/gpu/drm/imagination/pvr_fw_trace.h +78 −0
  48. drivers/gpu/drm/imagination/pvr_gem.c +414 −0
  49. drivers/gpu/drm/imagination/pvr_gem.h +170 −0
  50. drivers/gpu/drm/imagination/pvr_hwrt.c +549 −0
  51. drivers/gpu/drm/imagination/pvr_hwrt.h +165 −0
  52. drivers/gpu/drm/imagination/pvr_job.c +788 −0
  53. drivers/gpu/drm/imagination/pvr_job.h +161 −0
  54. drivers/gpu/drm/imagination/pvr_mmu.c +2637 −0
  55. drivers/gpu/drm/imagination/pvr_mmu.h +108 −0
  56. drivers/gpu/drm/imagination/pvr_params.c +147 −0
  57. drivers/gpu/drm/imagination/pvr_params.h +72 −0
  58. drivers/gpu/drm/imagination/pvr_power.c +433 −0
  59. drivers/gpu/drm/imagination/pvr_power.h +41 −0
  60. drivers/gpu/drm/imagination/pvr_queue.c +1432 −0
  61. drivers/gpu/drm/imagination/pvr_queue.h +169 −0
  62. drivers/gpu/drm/imagination/pvr_rogue_cr_defs.h +6193 −0
  63. drivers/gpu/drm/imagination/pvr_rogue_cr_defs_client.h +159 −0
  64. drivers/gpu/drm/imagination/pvr_rogue_defs.h +179 −0
  65. drivers/gpu/drm/imagination/pvr_rogue_fwif.h +2188 −0
  66. drivers/gpu/drm/imagination/pvr_rogue_fwif_check.h +493 −0
  67. drivers/gpu/drm/imagination/pvr_rogue_fwif_client.h +373 −0
  68. drivers/gpu/drm/imagination/pvr_rogue_fwif_client_check.h +133 −0
  69. drivers/gpu/drm/imagination/pvr_rogue_fwif_common.h +60 −0
  70. drivers/gpu/drm/imagination/pvr_rogue_fwif_dev_info.h +113 −0
  71. drivers/gpu/drm/imagination/pvr_rogue_fwif_resetframework.h +28 −0
  72. drivers/gpu/drm/imagination/pvr_rogue_fwif_sf.h +1648 −0
  73. drivers/gpu/drm/imagination/pvr_rogue_fwif_shared.h +258 −0
  74. drivers/gpu/drm/imagination/pvr_rogue_fwif_shared_check.h +108 −0
  75. drivers/gpu/drm/imagination/pvr_rogue_fwif_stream.h +78 −0
  76. drivers/gpu/drm/imagination/pvr_rogue_heap_config.h +113 −0
  77. drivers/gpu/drm/imagination/pvr_rogue_meta.h +356 −0
  78. drivers/gpu/drm/imagination/pvr_rogue_mips.h +335 −0
  79. drivers/gpu/drm/imagination/pvr_rogue_mips_check.h +58 −0
  80. drivers/gpu/drm/imagination/pvr_rogue_mmu_defs.h +136 −0
  81. drivers/gpu/drm/imagination/pvr_stream.c +285 −0
  82. drivers/gpu/drm/imagination/pvr_stream.h +75 −0
  83. drivers/gpu/drm/imagination/pvr_stream_defs.c +351 −0
  84. drivers/gpu/drm/imagination/pvr_stream_defs.h +16 −0
  85. drivers/gpu/drm/imagination/pvr_sync.c +289 −0
  86. drivers/gpu/drm/imagination/pvr_sync.h +84 −0
  87. drivers/gpu/drm/imagination/pvr_vm.c +1107 −0
  88. drivers/gpu/drm/imagination/pvr_vm.h +65 −0
  89. drivers/gpu/drm/imagination/pvr_vm_mips.c +238 −0
  90. drivers/gpu/drm/imagination/pvr_vm_mips.h +22 −0
  91. drivers/gpu/drm/imx/dcss/dcss-drv.c +2 −4
  92. drivers/gpu/drm/imx/lcdc/imx-lcdc.c +2 −4
  93. drivers/gpu/drm/kmb/kmb_drv.c +2 −3
  94. drivers/gpu/drm/mediatek/mtk_disp_ovl_adaptor.c +2 −3
  95. drivers/gpu/drm/mediatek/mtk_ethdr.c +2 −3
  96. drivers/gpu/drm/meson/meson_dw_mipi_dsi.c +2 −4
  97. drivers/gpu/drm/nouveau/nouveau_platform.c +2 −3
  98. drivers/gpu/drm/nouveau/nv04_fence.c +1 −1
  99. drivers/gpu/drm/nouveau/nvkm/engine/fifo/chan.c +0 −1
  100. drivers/gpu/drm/panel/panel-elida-kd35t133.c +3 −34
  101. drivers/gpu/drm/panel/panel-newvision-nv3051d.c +46 −11
  102. drivers/gpu/drm/rockchip/rk3066_hdmi.c +18 −27
  103. drivers/gpu/drm/rockchip/rockchip_drm_vop2.c +5 −0
  104. drivers/gpu/drm/rockchip/rockchip_vop2_reg.c +2 −0
  105. drivers/gpu/drm/sprd/sprd_dpu.c +2 −4
  106. drivers/gpu/drm/sprd/sprd_drm.c +2 −3
  107. drivers/gpu/drm/sprd/sprd_dsi.c +2 −4
  108. drivers/gpu/drm/tilcdc/tilcdc_drv.c +1 −1
  109. drivers/gpu/drm/tiny/arcpgu.c +2 −4
  110. drivers/video/fbdev/simplefb.c +4 −0
  111. include/drm/drm_edid.h +0 −5
  112. include/drm/drm_gpuvm.h +28 −0
  113. include/drm/drm_print.h +1 −1
  114. include/linux/sizes.h +9 −0
  115. include/uapi/drm/pvr_drm.h +1297 −0
Documentation/devicetree/bindings/display/panel/newvision,nv3051d.yaml+1−1
@@ -21,7 +21,7 @@ properties: · 20 unmodified lines
21 - enum:21 - enum:
22 - anbernic,rg351v-panel22 - anbernic,rg351v-panel
23 - anbernic,rg353p-panel23 - anbernic,rg353p-panel
24 - anbernic,rg353v-panel24 - powkiddy,rk2023-panel
25 - const: newvision,nv3051d25 - const: newvision,nv3051d
26 26
27 reg: true27 reg: true
Documentation/devicetree/bindings/gpu/img,powervr.yaml+73−0
@@ -0,0 +1,73 @@
1# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
2# Copyright (c) 2023 Imagination Technologies Ltd.
3%YAML 1.2
4---
5$id: http://devicetree.org/schemas/gpu/img,powervr.yaml#
6$schema: http://devicetree.org/meta-schemas/core.yaml#
7
8title: Imagination Technologies PowerVR and IMG GPU
9
10maintainers:
11 - Frank Binns <frank.binns@imgtec.com>
12
13properties:
14 compatible:
15 items:
16 - enum:
17 - ti,am62-gpu
18 - const: img,img-axe # IMG AXE GPU model/revision is fully discoverable
19
20 reg:
21 maxItems: 1
22
23 clocks:
24 minItems: 1
25 maxItems: 3
26
27 clock-names:
28 items:
29 - const: core
30 - const: mem
31 - const: sys
32 minItems: 1
33
34 interrupts:
35 maxItems: 1
36
37 power-domains:
38 maxItems: 1
39
40required:
41 - compatible
42 - reg
43 - clocks
44 - clock-names
45 - interrupts
46
47additionalProperties: false
48
49allOf:
50 - if:
51 properties:
52 compatible:
53 contains:
54 const: ti,am62-gpu
55 then:
56 properties:
57 clocks:
58 maxItems: 1
59
60examples:
61 - |
62 #include <dt-bindings/interrupt-controller/irq.h>
63 #include <dt-bindings/interrupt-controller/arm-gic.h>
64 #include <dt-bindings/soc/ti,sci_pm_domain.h>
65
66 gpu@fd00000 {
67 compatible = "ti,am62-gpu", "img,img-axe";
68 reg = <0x0fd00000 0x20000>;
69 clocks = <&k3_clks 187 0>;
70 clock-names = "core";
71 interrupts = <GIC_SPI 86 IRQ_TYPE_LEVEL_HIGH>;
72 power-domains = <&k3_pds 187 TI_SCI_PD_EXCLUSIVE>;
73 };
Documentation/gpu/drivers.rst+2−0
@@ -3,9 +3,11 @@ GPU Driver Documentation · 2 unmodified lines
3========================3========================
4 4
5.. toctree::5.. toctree::
6 :maxdepth: 3
6 7
7 amdgpu/index8 amdgpu/index
8 i9159 i915
10 imagination/index
9 mcde11 mcde
10 meson12 meson
11 pl11113 pl111
Documentation/gpu/imagination/index.rst+13−0
@@ -0,0 +1,13 @@
1=======================================
2drm/imagination PowerVR Graphics Driver
3=======================================
4
5.. kernel-doc:: drivers/gpu/drm/imagination/pvr_drv.c
6 :doc: PowerVR Graphics Driver
7
8Contents
9========
10.. toctree::
11 :maxdepth: 2
12
13 uapi
Documentation/gpu/imagination/uapi.rst+174−0
@@ -0,0 +1,174 @@
1====
2UAPI
3====
4The sources associated with this section can be found in ``pvr_drm.h``.
5
6.. kernel-doc:: include/uapi/drm/pvr_drm.h
7 :doc: PowerVR UAPI
8
9OBJECT ARRAYS
10=============
11.. kernel-doc:: include/uapi/drm/pvr_drm.h
12 :identifiers: drm_pvr_obj_array
13
14.. kernel-doc:: include/uapi/drm/pvr_drm.h
15 :identifiers: DRM_PVR_OBJ_ARRAY
16
17IOCTLS
18======
19.. kernel-doc:: include/uapi/drm/pvr_drm.h
20 :doc: PowerVR IOCTL interface
21
22.. kernel-doc:: include/uapi/drm/pvr_drm.h
23 :identifiers: PVR_IOCTL
24
25DEV_QUERY
26---------
27.. kernel-doc:: include/uapi/drm/pvr_drm.h
28 :doc: PowerVR IOCTL DEV_QUERY interface
29
30.. kernel-doc:: include/uapi/drm/pvr_drm.h
31 :identifiers: drm_pvr_dev_query
32
33.. kernel-doc:: include/uapi/drm/pvr_drm.h
34 :identifiers: drm_pvr_ioctl_dev_query_args
35
36.. kernel-doc:: include/uapi/drm/pvr_drm.h
37 :identifiers: drm_pvr_dev_query_gpu_info
38 drm_pvr_dev_query_runtime_info
39 drm_pvr_dev_query_hwrt_info
40 drm_pvr_dev_query_quirks
41 drm_pvr_dev_query_enhancements
42
43.. kernel-doc:: include/uapi/drm/pvr_drm.h
44 :identifiers: drm_pvr_heap_id
45 drm_pvr_heap
46 drm_pvr_dev_query_heap_info
47
48.. kernel-doc:: include/uapi/drm/pvr_drm.h
49 :doc: Flags for DRM_PVR_DEV_QUERY_HEAP_INFO_GET.
50
51.. kernel-doc:: include/uapi/drm/pvr_drm.h
52 :identifiers: drm_pvr_static_data_area_usage
53 drm_pvr_static_data_area
54 drm_pvr_dev_query_static_data_areas
55
56CREATE_BO
57---------
58.. kernel-doc:: include/uapi/drm/pvr_drm.h
59 :doc: PowerVR IOCTL CREATE_BO interface
60
61.. kernel-doc:: include/uapi/drm/pvr_drm.h
62 :identifiers: drm_pvr_ioctl_create_bo_args
63
64.. kernel-doc:: include/uapi/drm/pvr_drm.h
65 :doc: Flags for CREATE_BO
66
67GET_BO_MMAP_OFFSET
68------------------
69.. kernel-doc:: include/uapi/drm/pvr_drm.h
70 :doc: PowerVR IOCTL GET_BO_MMAP_OFFSET interface
71
72.. kernel-doc:: include/uapi/drm/pvr_drm.h
73 :identifiers: drm_pvr_ioctl_get_bo_mmap_offset_args
74
75CREATE_VM_CONTEXT and DESTROY_VM_CONTEXT
76----------------------------------------
77.. kernel-doc:: include/uapi/drm/pvr_drm.h
78 :doc: PowerVR IOCTL CREATE_VM_CONTEXT and DESTROY_VM_CONTEXT interfaces
79
80.. kernel-doc:: include/uapi/drm/pvr_drm.h
81 :identifiers: drm_pvr_ioctl_create_vm_context_args
82 drm_pvr_ioctl_destroy_vm_context_args
83
84VM_MAP and VM_UNMAP
85-------------------
86.. kernel-doc:: include/uapi/drm/pvr_drm.h
87 :doc: PowerVR IOCTL VM_MAP and VM_UNMAP interfaces
88
89.. kernel-doc:: include/uapi/drm/pvr_drm.h
90 :identifiers: drm_pvr_ioctl_vm_map_args
91 drm_pvr_ioctl_vm_unmap_args
92
93CREATE_CONTEXT and DESTROY_CONTEXT
94----------------------------------
95.. kernel-doc:: include/uapi/drm/pvr_drm.h
96 :doc: PowerVR IOCTL CREATE_CONTEXT and DESTROY_CONTEXT interfaces
97
98.. kernel-doc:: include/uapi/drm/pvr_drm.h
99 :identifiers: drm_pvr_ioctl_create_context_args
100
101.. kernel-doc:: include/uapi/drm/pvr_drm.h
102 :identifiers: drm_pvr_ctx_priority
103 drm_pvr_ctx_type
104 drm_pvr_static_render_context_state
105 drm_pvr_static_render_context_state_format
106 drm_pvr_reset_framework
107 drm_pvr_reset_framework_format
108
109.. kernel-doc:: include/uapi/drm/pvr_drm.h
110 :identifiers: drm_pvr_ioctl_destroy_context_args
111
112CREATE_FREE_LIST and DESTROY_FREE_LIST
113--------------------------------------
114.. kernel-doc:: include/uapi/drm/pvr_drm.h
115 :doc: PowerVR IOCTL CREATE_FREE_LIST and DESTROY_FREE_LIST interfaces
116
117.. kernel-doc:: include/uapi/drm/pvr_drm.h
118 :identifiers: drm_pvr_ioctl_create_free_list_args
119
120.. kernel-doc:: include/uapi/drm/pvr_drm.h
121 :identifiers: drm_pvr_ioctl_destroy_free_list_args
122
123CREATE_HWRT_DATASET and DESTROY_HWRT_DATASET
124--------------------------------------
125.. kernel-doc:: include/uapi/drm/pvr_drm.h
126 :doc: PowerVR IOCTL CREATE_HWRT_DATASET and DESTROY_HWRT_DATASET interfaces
127
128.. kernel-doc:: include/uapi/drm/pvr_drm.h
129 :identifiers: drm_pvr_ioctl_create_hwrt_dataset_args
130
131.. kernel-doc:: include/uapi/drm/pvr_drm.h
132 :identifiers: drm_pvr_create_hwrt_geom_data_args
133 drm_pvr_create_hwrt_rt_data_args
134
135.. kernel-doc:: include/uapi/drm/pvr_drm.h
136 :identifiers: drm_pvr_ioctl_destroy_hwrt_dataset_args
137
138SUBMIT_JOBS
139-----------
140.. kernel-doc:: include/uapi/drm/pvr_drm.h
141 :doc: PowerVR IOCTL SUBMIT_JOBS interface
142
143.. kernel-doc:: include/uapi/drm/pvr_drm.h
144 :doc: Flags for the drm_pvr_sync_op object.
145
146.. kernel-doc:: include/uapi/drm/pvr_drm.h
147 :identifiers: drm_pvr_ioctl_submit_jobs_args
148
149.. kernel-doc:: include/uapi/drm/pvr_drm.h
150 :doc: Flags for SUBMIT_JOB ioctl geometry command.
151
152.. kernel-doc:: include/uapi/drm/pvr_drm.h
153 :doc: Flags for SUBMIT_JOB ioctl fragment command.
154
155.. kernel-doc:: include/uapi/drm/pvr_drm.h
156 :doc: Flags for SUBMIT_JOB ioctl compute command.
157
158.. kernel-doc:: include/uapi/drm/pvr_drm.h
159 :doc: Flags for SUBMIT_JOB ioctl transfer command.
160
161.. kernel-doc:: include/uapi/drm/pvr_drm.h
162 :identifiers: drm_pvr_sync_op
163 drm_pvr_job_type
164 drm_pvr_hwrt_data_ref
165 drm_pvr_job
166
167Internal notes
168==============
169.. kernel-doc:: drivers/gpu/drm/imagination/pvr_device.h
170 :doc: IOCTL validation helpers
171
172.. kernel-doc:: drivers/gpu/drm/imagination/pvr_device.h
173 :identifiers: PVR_STATIC_ASSERT_64BIT_ALIGNED PVR_IOCTL_UNION_PADDING_CHECK
174 pvr_ioctl_union_padding_check
MAINTAINERS+10−0
@@ -10391,6 +10391,16 @@ IMGTEC IR DECODER DRIVER · 10390 unmodified lines
10391S: Orphan10391S: Orphan
10392F: drivers/media/rc/img-ir/10392F: drivers/media/rc/img-ir/
10393 10393
10394IMGTEC POWERVR DRM DRIVER
10395M: Frank Binns <frank.binns@imgtec.com>
10396M: Donald Robson <donald.robson@imgtec.com>
10397M: Matt Coster <matt.coster@imgtec.com>
10398S: Supported
10399F: Documentation/devicetree/bindings/gpu/img,powervr.yaml
10400F: Documentation/gpu/imagination/
10401F: drivers/gpu/drm/imagination/
10402F: include/uapi/drm/pvr_drm.h
10403
10394IMON SOUNDGRAPH USB IR RECEIVER10404IMON SOUNDGRAPH USB IR RECEIVER
10395M: Sean Young <sean@mess.org>10405M: Sean Young <sean@mess.org>
10396L: linux-media@vger.kernel.org10406L: linux-media@vger.kernel.org
drivers/accel/qaic/qaic_data.c+1−1
@@ -452,7 +452,7 @@ static int create_sgt(struct qaic_device *qdev, struct sg_table **sgt_out, u64 s · 451 unmodified lines
452 * later452 * later
453 */453 */
454 buf_extra = (PAGE_SIZE - size % PAGE_SIZE) % PAGE_SIZE;454 buf_extra = (PAGE_SIZE - size % PAGE_SIZE) % PAGE_SIZE;
455 max_order = min(MAX_ORDER - 1, get_order(size));455 max_order = min(MAX_ORDER, get_order(size));
456 } else {456 } else {
457 /* allocate a single page for book keeping */457 /* allocate a single page for book keeping */
458 nr_pages = 1;458 nr_pages = 1;
drivers/dma-buf/dma-buf.c+2−2
@@ -46,12 +46,12 @@ static char *dmabuffs_dname(struct dentry *dentry, char *buffer, int buflen) · 45 unmodified lines
46{46{
47 struct dma_buf *dmabuf;47 struct dma_buf *dmabuf;
48 char name[DMA_BUF_NAME_LEN];48 char name[DMA_BUF_NAME_LEN];
49 size_t ret = 0;49 ssize_t ret = 0;
50 50
51 dmabuf = dentry->d_fsdata;51 dmabuf = dentry->d_fsdata;
52 spin_lock(&dmabuf->name_lock);52 spin_lock(&dmabuf->name_lock);
53 if (dmabuf->name)53 if (dmabuf->name)
54 ret = strlcpy(name, dmabuf->name, DMA_BUF_NAME_LEN);54 ret = strscpy(name, dmabuf->name, sizeof(name));
55 spin_unlock(&dmabuf->name_lock);55 spin_unlock(&dmabuf->name_lock);
56 56
57 return dynamic_dname(buffer, buflen, "/%s:%s",57 return dynamic_dname(buffer, buflen, "/%s:%s",
drivers/gpu/drm/Kconfig+2−0
@@ -394,6 +394,8 @@ source "drivers/gpu/drm/solomon/Kconfig" · 393 unmodified lines
394 394
395source "drivers/gpu/drm/sprd/Kconfig"395source "drivers/gpu/drm/sprd/Kconfig"
396 396
397source "drivers/gpu/drm/imagination/Kconfig"
398
397config DRM_HYPERV399config DRM_HYPERV
398 tristate "DRM Support for Hyper-V synthetic video device"400 tristate "DRM Support for Hyper-V synthetic video device"
399 depends on DRM && PCI && MMU && HYPERV401 depends on DRM && PCI && MMU && HYPERV
drivers/gpu/drm/Makefile+1−0
@@ -199,3 +199,4 @@ obj-$(CONFIG_DRM_HYPERV) += hyperv/ · 198 unmodified lines
199obj-y += solomon/199obj-y += solomon/
200obj-$(CONFIG_DRM_SPRD) += sprd/200obj-$(CONFIG_DRM_SPRD) += sprd/
201obj-$(CONFIG_DRM_LOONGSON) += loongson/201obj-$(CONFIG_DRM_LOONGSON) += loongson/
202obj-$(CONFIG_DRM_POWERVR) += imagination/
drivers/gpu/drm/armada/armada_crtc.c+2−3
@@ -1066,10 +1066,9 @@ static int armada_lcd_probe(struct platform_device *pdev) · 1065 unmodified lines
1066 return component_add(&pdev->dev, &armada_lcd_ops);1066 return component_add(&pdev->dev, &armada_lcd_ops);
1067}1067}
1068 1068
1069static int armada_lcd_remove(struct platform_device *pdev)1069static void armada_lcd_remove(struct platform_device *pdev)
1070{1070{
1071 component_del(&pdev->dev, &armada_lcd_ops);1071 component_del(&pdev->dev, &armada_lcd_ops);
1072 return 0;
1073}1072}
1074 1073
1075static const struct of_device_id armada_lcd_of_match[] = {1074static const struct of_device_id armada_lcd_of_match[] = {
@@ -1095,7 +1094,7 @@ MODULE_DEVICE_TABLE(platform, armada_lcd_platform_ids); · 19 unmodified lines
1095 1094
1096struct platform_driver armada_lcd_platform_driver = {1095struct platform_driver armada_lcd_platform_driver = {
1097 .probe = armada_lcd_probe,1096 .probe = armada_lcd_probe,
1098 .remove = armada_lcd_remove,1097 .remove_new = armada_lcd_remove,
1099 .driver = {1098 .driver = {
1100 .name = "armada-lcd",1099 .name = "armada-lcd",
1101 .owner = THIS_MODULE,1100 .owner = THIS_MODULE,
drivers/gpu/drm/armada/armada_drv.c+2−3
@@ -226,10 +226,9 @@ static int armada_drm_probe(struct platform_device *pdev) · 225 unmodified lines
226 match);226 match);
227}227}
228 228
229static int armada_drm_remove(struct platform_device *pdev)229static void armada_drm_remove(struct platform_device *pdev)
230{230{
231 component_master_del(&pdev->dev, &armada_master_ops);231 component_master_del(&pdev->dev, &armada_master_ops);
232 return 0;
233}232}
234 233
235static void armada_drm_shutdown(struct platform_device *pdev)234static void armada_drm_shutdown(struct platform_device *pdev)
@@ -249,7 +248,7 @@ MODULE_DEVICE_TABLE(platform, armada_drm_platform_ids); · 13 unmodified lines
249 248
250static struct platform_driver armada_drm_platform_driver = {249static struct platform_driver armada_drm_platform_driver = {
251 .probe = armada_drm_probe,250 .probe = armada_drm_probe,
252 .remove = armada_drm_remove,251 .remove_new = armada_drm_remove,
253 .shutdown = armada_drm_shutdown,252 .shutdown = armada_drm_shutdown,
254 .driver = {253 .driver = {
255 .name = "armada-drm",254 .name = "armada-drm",
drivers/gpu/drm/bridge/cadence/cdns-mhdp8546-core.c+10−12
@@ -2596,11 +2596,10 @@ clk_disable: · 2595 unmodified lines
2596 return ret;2596 return ret;
2597}2597}
2598 2598
2599static int cdns_mhdp_remove(struct platform_device *pdev)2599static void cdns_mhdp_remove(struct platform_device *pdev)
2600{2600{
2601 struct cdns_mhdp_device *mhdp = platform_get_drvdata(pdev);2601 struct cdns_mhdp_device *mhdp = platform_get_drvdata(pdev);
2602 unsigned long timeout = msecs_to_jiffies(100);2602 unsigned long timeout = msecs_to_jiffies(100);
2603 bool stop_fw = false;
2604 int ret;2603 int ret;
2605 2604
2606 drm_bridge_remove(&mhdp->bridge);2605 drm_bridge_remove(&mhdp->bridge);
@@ -2608,18 +2607,19 @@ static int cdns_mhdp_remove(struct platform_device *pdev) · 1 unmodified line
2608 ret = wait_event_timeout(mhdp->fw_load_wq,2607 ret = wait_event_timeout(mhdp->fw_load_wq,
2609 mhdp->hw_state == MHDP_HW_READY,2608 mhdp->hw_state == MHDP_HW_READY,
2610 timeout);2609 timeout);
2611 if (ret == 0)
2612 dev_err(mhdp->dev, "%s: Timeout waiting for fw loading\n",
2613 __func__);
2614 else
2615 stop_fw = true;
2616
2617 spin_lock(&mhdp->start_lock);2610 spin_lock(&mhdp->start_lock);
2618 mhdp->hw_state = MHDP_HW_STOPPED;2611 mhdp->hw_state = MHDP_HW_STOPPED;
2619 spin_unlock(&mhdp->start_lock);2612 spin_unlock(&mhdp->start_lock);
2620 2613
2621 if (stop_fw)2614 if (ret == 0) {
2615 dev_err(mhdp->dev, "%s: Timeout waiting for fw loading\n",
2616 __func__);
2617 } else {
2622 ret = cdns_mhdp_set_firmware_active(mhdp, false);2618 ret = cdns_mhdp_set_firmware_active(mhdp, false);
2619 if (ret)
2620 dev_err(mhdp->dev, "Failed to stop firmware (%pe)\n",
2621 ERR_PTR(ret));
2622 }
2623 2623
2624 phy_exit(mhdp->phy);2624 phy_exit(mhdp->phy);
2625 2625
@@ -2634,8 +2634,6 @@ static int cdns_mhdp_remove(struct platform_device *pdev) · 8 unmodified lines
2634 /* Ignoring mhdp->hdcp.check_work and mhdp->hdcp.prop_work here. */2634 /* Ignoring mhdp->hdcp.check_work and mhdp->hdcp.prop_work here. */
2635 2635
2636 clk_disable_unprepare(mhdp->clk);2636 clk_disable_unprepare(mhdp->clk);
2637
2638 return ret;
2639}2637}
2640 2638
2641static const struct of_device_id mhdp_ids[] = {2639static const struct of_device_id mhdp_ids[] = {
@@ -2658,7 +2656,7 @@ static struct platform_driver mhdp_driver = { · 16 unmodified lines
2658 .of_match_table = mhdp_ids,2656 .of_match_table = mhdp_ids,
2659 },2657 },
2660 .probe = cdns_mhdp_probe,2658 .probe = cdns_mhdp_probe,
2661 .remove = cdns_mhdp_remove,2659 .remove_new = cdns_mhdp_remove,
2662};2660};
2663module_platform_driver(mhdp_driver);2661module_platform_driver(mhdp_driver);
2664 2662
drivers/gpu/drm/bridge/ti-tpd12s015.c+2−4
@@ -179,13 +179,11 @@ static int tpd12s015_probe(struct platform_device *pdev) · 178 unmodified lines
179 return 0;179 return 0;
180}180}
181 181
182static int __exit tpd12s015_remove(struct platform_device *pdev)182static void tpd12s015_remove(struct platform_device *pdev)
183{183{
184 struct tpd12s015_device *tpd = platform_get_drvdata(pdev);184 struct tpd12s015_device *tpd = platform_get_drvdata(pdev);
185 185
186 drm_bridge_remove(&tpd->bridge);186 drm_bridge_remove(&tpd->bridge);
187
188 return 0;
189}187}
190 188
191static const struct of_device_id tpd12s015_of_match[] = {189static const struct of_device_id tpd12s015_of_match[] = {
@@ -197,7 +195,7 @@ MODULE_DEVICE_TABLE(of, tpd12s015_of_match); · 5 unmodified lines
197 195
198static struct platform_driver tpd12s015_driver = {196static struct platform_driver tpd12s015_driver = {
199 .probe = tpd12s015_probe,197 .probe = tpd12s015_probe,
200 .remove = __exit_p(tpd12s015_remove),198 .remove_new = tpd12s015_remove,
201 .driver = {199 .driver = {
202 .name = "tpd12s015",200 .name = "tpd12s015",
203 .of_match_table = tpd12s015_of_match,201 .of_match_table = tpd12s015_of_match,
drivers/gpu/drm/drm_edid_load.c+0−16
@@ -23,22 +23,6 @@ module_param_string(edid_firmware, edid_firmware, sizeof(edid_firmware), 0644); · 22 unmodified lines
23MODULE_PARM_DESC(edid_firmware, "Do not probe monitor, use specified EDID blob "23MODULE_PARM_DESC(edid_firmware, "Do not probe monitor, use specified EDID blob "
24 "from built-in data or /lib/firmware instead. ");24 "from built-in data or /lib/firmware instead. ");
25 25
26/* Use only for backward compatibility with drm_kms_helper.edid_firmware */
27int __drm_set_edid_firmware_path(const char *path)
28{
29 scnprintf(edid_firmware, sizeof(edid_firmware), "%s", path);
30
31 return 0;
32}
33EXPORT_SYMBOL(__drm_set_edid_firmware_path);
34
35/* Use only for backward compatibility with drm_kms_helper.edid_firmware */
36int __drm_get_edid_firmware_path(char *buf, size_t bufsize)
37{
38 return scnprintf(buf, bufsize, "%s", edid_firmware);
39}
40EXPORT_SYMBOL(__drm_get_edid_firmware_path);
41
42#define GENERIC_EDIDS 626#define GENERIC_EDIDS 6
43static const char * const generic_edid_name[GENERIC_EDIDS] = {27static const char * const generic_edid_name[GENERIC_EDIDS] = {
44 "edid/800x600.bin",28 "edid/800x600.bin",
drivers/gpu/drm/drm_kms_helper_common.c+0−32
@@ -27,38 +27,6 @@ · 26 unmodified lines
27 27
28#include <linux/module.h>28#include <linux/module.h>
29 29
30#include <drm/drm_edid.h>
31#include <drm/drm_print.h>
32
33#include "drm_crtc_helper_internal.h"
34
35MODULE_AUTHOR("David Airlie, Jesse Barnes");30MODULE_AUTHOR("David Airlie, Jesse Barnes");
36MODULE_DESCRIPTION("DRM KMS helper");31MODULE_DESCRIPTION("DRM KMS helper");
37MODULE_LICENSE("GPL and additional rights");32MODULE_LICENSE("GPL and additional rights");
38
39#if IS_ENABLED(CONFIG_DRM_LOAD_EDID_FIRMWARE)
40
41/* Backward compatibility for drm_kms_helper.edid_firmware */
42static int edid_firmware_set(const char *val, const struct kernel_param *kp)
43{
44 DRM_NOTE("drm_kms_helper.edid_firmware is deprecated, please use drm.edid_firmware instead.\n");
45
46 return __drm_set_edid_firmware_path(val);
47}
48
49static int edid_firmware_get(char *buffer, const struct kernel_param *kp)
50{
51 return __drm_get_edid_firmware_path(buffer, PAGE_SIZE);
52}
53
54static const struct kernel_param_ops edid_firmware_ops = {
55 .set = edid_firmware_set,
56 .get = edid_firmware_get,
57};
58
59module_param_cb(edid_firmware, &edid_firmware_ops, NULL, 0644);
60__MODULE_PARM_TYPE(edid_firmware, "charp");
61MODULE_PARM_DESC(edid_firmware,
62 "DEPRECATED. Use drm.edid_firmware module parameter instead.");
63
64#endif
drivers/gpu/drm/etnaviv/etnaviv_drv.c+2−4
@@ -640,16 +640,14 @@ static int etnaviv_pdev_probe(struct platform_device *pdev) · 639 unmodified lines
640 return component_master_add_with_match(dev, &etnaviv_master_ops, match);640 return component_master_add_with_match(dev, &etnaviv_master_ops, match);
641}641}
642 642
643static int etnaviv_pdev_remove(struct platform_device *pdev)643static void etnaviv_pdev_remove(struct platform_device *pdev)
644{644{
645 component_master_del(&pdev->dev, &etnaviv_master_ops);645 component_master_del(&pdev->dev, &etnaviv_master_ops);
646
647 return 0;
648}646}
649 647
650static struct platform_driver etnaviv_platform_driver = {648static struct platform_driver etnaviv_platform_driver = {
651 .probe = etnaviv_pdev_probe,649 .probe = etnaviv_pdev_probe,
652 .remove = etnaviv_pdev_remove,650 .remove_new = etnaviv_pdev_remove,
653 .driver = {651 .driver = {
654 .name = "etnaviv",652 .name = "etnaviv",
655 },653 },
drivers/gpu/drm/etnaviv/etnaviv_gpu.c+2−3
@@ -1904,11 +1904,10 @@ static int etnaviv_gpu_platform_probe(struct platform_device *pdev) · 1903 unmodified lines
1904 return 0;1904 return 0;
1905}1905}
1906 1906
1907static int etnaviv_gpu_platform_remove(struct platform_device *pdev)1907static void etnaviv_gpu_platform_remove(struct platform_device *pdev)
1908{1908{
1909 component_del(&pdev->dev, &gpu_ops);1909 component_del(&pdev->dev, &gpu_ops);
1910 pm_runtime_disable(&pdev->dev);1910 pm_runtime_disable(&pdev->dev);
1911 return 0;
1912}1911}
1913 1912
1914static int etnaviv_gpu_rpm_suspend(struct device *dev)1913static int etnaviv_gpu_rpm_suspend(struct device *dev)
@@ -1970,6 +1969,6 @@ struct platform_driver etnaviv_gpu_driver = { · 55 unmodified lines
1970 .of_match_table = etnaviv_gpu_match,1969 .of_match_table = etnaviv_gpu_match,
1971 },1970 },
1972 .probe = etnaviv_gpu_platform_probe,1971 .probe = etnaviv_gpu_platform_probe,
1973 .remove = etnaviv_gpu_platform_remove,1972 .remove_new = etnaviv_gpu_platform_remove,
1974 .id_table = gpu_ids,1973 .id_table = gpu_ids,
1975};1974};
drivers/gpu/drm/imagination/Kconfig+18−0
@@ -0,0 +1,18 @@
1# SPDX-License-Identifier: GPL-2.0-only OR MIT
2# Copyright (c) 2023 Imagination Technologies Ltd.
3
4config DRM_POWERVR
5 tristate "Imagination Technologies PowerVR (Series 6 and later) & IMG Graphics"
6 depends on ARM64
7 depends on DRM
8 depends on PM
9 select DRM_EXEC
10 select DRM_GEM_SHMEM_HELPER
11 select DRM_SCHED
12 select DRM_GPUVM
13 select FW_LOADER
14 help
15 Choose this option if you have a system that has an Imagination
16 Technologies PowerVR (Series 6 or later) or IMG GPU.
17
18 If "M" is selected, the module will be called powervr.
drivers/gpu/drm/imagination/Makefile+35−0
@@ -0,0 +1,35 @@
1# SPDX-License-Identifier: GPL-2.0-only OR MIT
2# Copyright (c) 2023 Imagination Technologies Ltd.
3
4subdir-ccflags-y := -I$(srctree)/$(src)
5
6powervr-y := \
7 pvr_ccb.o \
8 pvr_cccb.o \
9 pvr_context.o \
10 pvr_device.o \
11 pvr_device_info.o \
12 pvr_drv.o \
13 pvr_free_list.o \
14 pvr_fw.o \
15 pvr_fw_meta.o \
16 pvr_fw_mips.o \
17 pvr_fw_startstop.o \
18 pvr_fw_trace.o \
19 pvr_gem.o \
20 pvr_hwrt.o \
21 pvr_job.o \
22 pvr_mmu.o \
23 pvr_params.o \
24 pvr_power.o \
25 pvr_queue.o \
26 pvr_stream.o \
27 pvr_stream_defs.o \
28 pvr_sync.o \
29 pvr_vm.o \
30 pvr_vm_mips.o
31
32powervr-$(CONFIG_DEBUG_FS) += \
33 pvr_debugfs.o
34
35obj-$(CONFIG_DRM_POWERVR) += powervr.o
drivers/gpu/drm/imagination/pvr_ccb.c+645−0
@@ -0,0 +1,645 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_ccb.h"
5#include "pvr_device.h"
6#include "pvr_drv.h"
7#include "pvr_free_list.h"
8#include "pvr_fw.h"
9#include "pvr_gem.h"
10#include "pvr_power.h"
11
12#include <drm/drm_managed.h>
13#include <linux/compiler.h>
14#include <linux/delay.h>
15#include <linux/jiffies.h>
16#include <linux/kernel.h>
17#include <linux/mutex.h>
18#include <linux/types.h>
19#include <linux/workqueue.h>
20
21#define RESERVE_SLOT_TIMEOUT (1 * HZ) /* 1s */
22#define RESERVE_SLOT_MIN_RETRIES 10
23
24static void
25ccb_ctrl_init(void *cpu_ptr, void *priv)
26{
27 struct rogue_fwif_ccb_ctl *ctrl = cpu_ptr;
28 struct pvr_ccb *pvr_ccb = priv;
29
30 ctrl->write_offset = 0;
31 ctrl->read_offset = 0;
32 ctrl->wrap_mask = pvr_ccb->num_cmds - 1;
33 ctrl->cmd_size = pvr_ccb->cmd_size;
34}
35
36/**
37 * pvr_ccb_init() - Initialise a CCB
38 * @pvr_dev: Device pointer.
39 * @pvr_ccb: Pointer to CCB structure to initialise.
40 * @num_cmds_log2: Log2 of number of commands in this CCB.
41 * @cmd_size: Command size for this CCB.
42 *
43 * Return:
44 * * Zero on success, or
45 * * Any error code returned by pvr_fw_object_create_and_map().
46 */
47static int
48pvr_ccb_init(struct pvr_device *pvr_dev, struct pvr_ccb *pvr_ccb,
49 u32 num_cmds_log2, size_t cmd_size)
50{
51 u32 num_cmds = 1 << num_cmds_log2;
52 u32 ccb_size = num_cmds * cmd_size;
53 int err;
54
55 pvr_ccb->num_cmds = num_cmds;
56 pvr_ccb->cmd_size = cmd_size;
57
58 err = drmm_mutex_init(from_pvr_device(pvr_dev), &pvr_ccb->lock);
59 if (err)
60 return err;
61
62 /*
63 * Map CCB and control structure as uncached, so we don't have to flush
64 * CPU cache repeatedly when polling for space.
65 */
66 pvr_ccb->ctrl = pvr_fw_object_create_and_map(pvr_dev, sizeof(*pvr_ccb->ctrl),
67 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
68 ccb_ctrl_init, pvr_ccb, &pvr_ccb->ctrl_obj);
69 if (IS_ERR(pvr_ccb->ctrl))
70 return PTR_ERR(pvr_ccb->ctrl);
71
72 pvr_ccb->ccb = pvr_fw_object_create_and_map(pvr_dev, ccb_size,
73 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
74 NULL, NULL, &pvr_ccb->ccb_obj);
75 if (IS_ERR(pvr_ccb->ccb)) {
76 err = PTR_ERR(pvr_ccb->ccb);
77 goto err_free_ctrl;
78 }
79
80 pvr_fw_object_get_fw_addr(pvr_ccb->ctrl_obj, &pvr_ccb->ctrl_fw_addr);
81 pvr_fw_object_get_fw_addr(pvr_ccb->ccb_obj, &pvr_ccb->ccb_fw_addr);
82
83 WRITE_ONCE(pvr_ccb->ctrl->write_offset, 0);
84 WRITE_ONCE(pvr_ccb->ctrl->read_offset, 0);
85 WRITE_ONCE(pvr_ccb->ctrl->wrap_mask, num_cmds - 1);
86 WRITE_ONCE(pvr_ccb->ctrl->cmd_size, cmd_size);
87
88 return 0;
89
90err_free_ctrl:
91 pvr_fw_object_unmap_and_destroy(pvr_ccb->ctrl_obj);
92
93 return err;
94}
95
96/**
97 * pvr_ccb_fini() - Release CCB structure
98 * @pvr_ccb: CCB to release.
99 */
100void
101pvr_ccb_fini(struct pvr_ccb *pvr_ccb)
102{
103 pvr_fw_object_unmap_and_destroy(pvr_ccb->ccb_obj);
104 pvr_fw_object_unmap_and_destroy(pvr_ccb->ctrl_obj);
105}
106
107/**
108 * pvr_ccb_slot_available_locked() - Test whether any slots are available in CCB
109 * @pvr_ccb: CCB to test.
110 * @write_offset: Address to store number of next available slot. May be %NULL.
111 *
112 * Caller must hold @pvr_ccb->lock.
113 *
114 * Return:
115 * * %true if a slot is available, or
116 * * %false if no slot is available.
117 */
118static __always_inline bool
119pvr_ccb_slot_available_locked(struct pvr_ccb *pvr_ccb, u32 *write_offset)
120{
121 struct rogue_fwif_ccb_ctl *ctrl = pvr_ccb->ctrl;
122 u32 next_write_offset = (READ_ONCE(ctrl->write_offset) + 1) & READ_ONCE(ctrl->wrap_mask);
123
124 lockdep_assert_held(&pvr_ccb->lock);
125
126 if (READ_ONCE(ctrl->read_offset) != next_write_offset) {
127 if (write_offset)
128 *write_offset = next_write_offset;
129 return true;
130 }
131
132 return false;
133}
134
135static void
136process_fwccb_command(struct pvr_device *pvr_dev, struct rogue_fwif_fwccb_cmd *cmd)
137{
138 switch (cmd->cmd_type) {
139 case ROGUE_FWIF_FWCCB_CMD_REQUEST_GPU_RESTART:
140 pvr_power_reset(pvr_dev, false);
141 break;
142
143 case ROGUE_FWIF_FWCCB_CMD_FREELISTS_RECONSTRUCTION:
144 pvr_free_list_process_reconstruct_req(pvr_dev,
145 &cmd->cmd_data.cmd_freelists_reconstruction);
146 break;
147
148 case ROGUE_FWIF_FWCCB_CMD_FREELIST_GROW:
149 pvr_free_list_process_grow_req(pvr_dev, &cmd->cmd_data.cmd_free_list_gs);
150 break;
151
152 default:
153 drm_info(from_pvr_device(pvr_dev), "Received unknown FWCCB command %x\n",
154 cmd->cmd_type);
155 break;
156 }
157}
158
159/**
160 * pvr_fwccb_process() - Process any pending FWCCB commands
161 * @pvr_dev: Target PowerVR device
162 */
163void pvr_fwccb_process(struct pvr_device *pvr_dev)
164{
165 struct rogue_fwif_fwccb_cmd *fwccb = pvr_dev->fwccb.ccb;
166 struct rogue_fwif_ccb_ctl *ctrl = pvr_dev->fwccb.ctrl;
167 u32 read_offset;
168
169 mutex_lock(&pvr_dev->fwccb.lock);
170
171 while ((read_offset = READ_ONCE(ctrl->read_offset)) != READ_ONCE(ctrl->write_offset)) {
172 struct rogue_fwif_fwccb_cmd cmd = fwccb[read_offset];
173
174 WRITE_ONCE(ctrl->read_offset, (read_offset + 1) & READ_ONCE(ctrl->wrap_mask));
175
176 /* Drop FWCCB lock while we process command. */
177 mutex_unlock(&pvr_dev->fwccb.lock);
178
179 process_fwccb_command(pvr_dev, &cmd);
180
181 mutex_lock(&pvr_dev->fwccb.lock);
182 }
183
184 mutex_unlock(&pvr_dev->fwccb.lock);
185}
186
187/**
188 * pvr_kccb_capacity() - Returns the maximum number of usable KCCB slots.
189 * @pvr_dev: Target PowerVR device
190 *
191 * Return:
192 * * The maximum number of active slots.
193 */
194static u32 pvr_kccb_capacity(struct pvr_device *pvr_dev)
195{
196 /* Capacity is the number of slot minus one to cope with the wrapping
197 * mechanisms. If we were to use all slots, we might end up with
198 * read_offset == write_offset, which the FW considers as a KCCB-is-empty
199 * condition.
200 */
201 return pvr_dev->kccb.slot_count - 1;
202}
203
204/**
205 * pvr_kccb_used_slot_count_locked() - Get the number of used slots
206 * @pvr_dev: Device pointer.
207 *
208 * KCCB lock must be held.
209 *
210 * Return:
211 * * The number of slots currently used.
212 */
213static u32
214pvr_kccb_used_slot_count_locked(struct pvr_device *pvr_dev)
215{
216 struct pvr_ccb *pvr_ccb = &pvr_dev->kccb.ccb;
217 struct rogue_fwif_ccb_ctl *ctrl = pvr_ccb->ctrl;
218 u32 wr_offset = READ_ONCE(ctrl->write_offset);
219 u32 rd_offset = READ_ONCE(ctrl->read_offset);
220 u32 used_count;
221
222 lockdep_assert_held(&pvr_ccb->lock);
223
224 if (wr_offset >= rd_offset)
225 used_count = wr_offset - rd_offset;
226 else
227 used_count = wr_offset + pvr_dev->kccb.slot_count - rd_offset;
228
229 return used_count;
230}
231
232/**
233 * pvr_kccb_send_cmd_reserved_powered() - Send command to the KCCB, with the PM ref
234 * held and a slot pre-reserved
235 * @pvr_dev: Device pointer.
236 * @cmd: Command to sent.
237 * @kccb_slot: Address to store the KCCB slot for this command. May be %NULL.
238 */
239void
240pvr_kccb_send_cmd_reserved_powered(struct pvr_device *pvr_dev,
241 struct rogue_fwif_kccb_cmd *cmd,
242 u32 *kccb_slot)
243{
244 struct pvr_ccb *pvr_ccb = &pvr_dev->kccb.ccb;
245 struct rogue_fwif_kccb_cmd *kccb = pvr_ccb->ccb;
246 struct rogue_fwif_ccb_ctl *ctrl = pvr_ccb->ctrl;
247 u32 old_write_offset;
248 u32 new_write_offset;
249
250 WARN_ON(pvr_dev->lost);
251
252 mutex_lock(&pvr_ccb->lock);
253
254 if (WARN_ON(!pvr_dev->kccb.reserved_count))
255 goto out_unlock;
256
257 old_write_offset = READ_ONCE(ctrl->write_offset);
258
259 /* We reserved the slot, we should have one available. */
260 if (WARN_ON(!pvr_ccb_slot_available_locked(pvr_ccb, &new_write_offset)))
261 goto out_unlock;
262
263 memcpy(&kccb[old_write_offset], cmd,
264 sizeof(struct rogue_fwif_kccb_cmd));
265 if (kccb_slot) {
266 *kccb_slot = old_write_offset;
267 /* Clear return status for this slot. */
268 WRITE_ONCE(pvr_dev->kccb.rtn[old_write_offset],
269 ROGUE_FWIF_KCCB_RTN_SLOT_NO_RESPONSE);
270 }
271 mb(); /* memory barrier */
272 WRITE_ONCE(ctrl->write_offset, new_write_offset);
273 pvr_dev->kccb.reserved_count--;
274
275 /* Kick MTS */
276 pvr_fw_mts_schedule(pvr_dev,
277 PVR_FWIF_DM_GP & ~ROGUE_CR_MTS_SCHEDULE_DM_CLRMSK);
278
279out_unlock:
280 mutex_unlock(&pvr_ccb->lock);
281}
282
283/**
284 * pvr_kccb_try_reserve_slot() - Try to reserve a KCCB slot
285 * @pvr_dev: Device pointer.
286 *
287 * Return:
288 * * true if a KCCB slot was reserved, or
289 * * false otherwise.
290 */
291static bool pvr_kccb_try_reserve_slot(struct pvr_device *pvr_dev)
292{
293 bool reserved = false;
294 u32 used_count;
295
296 mutex_lock(&pvr_dev->kccb.ccb.lock);
297
298 used_count = pvr_kccb_used_slot_count_locked(pvr_dev);
299 if (pvr_dev->kccb.reserved_count < pvr_kccb_capacity(pvr_dev) - used_count) {
300 pvr_dev->kccb.reserved_count++;
301 reserved = true;
302 }
303
304 mutex_unlock(&pvr_dev->kccb.ccb.lock);
305
306 return reserved;
307}
308
309/**
310 * pvr_kccb_reserve_slot_sync() - Try to reserve a slot synchronously
311 * @pvr_dev: Device pointer.
312 *
313 * Return:
314 * * 0 on success, or
315 * * -EBUSY if no slots were reserved after %RESERVE_SLOT_TIMEOUT, with a minimum of
316 * %RESERVE_SLOT_MIN_RETRIES retries.
317 */
318static int pvr_kccb_reserve_slot_sync(struct pvr_device *pvr_dev)
319{
320 unsigned long start_timestamp = jiffies;
321 bool reserved = false;
322 u32 retries = 0;
323
324 while ((jiffies - start_timestamp) < (u32)RESERVE_SLOT_TIMEOUT ||
325 retries < RESERVE_SLOT_MIN_RETRIES) {
326 reserved = pvr_kccb_try_reserve_slot(pvr_dev);
327 if (reserved)
328 break;
329
330 usleep_range(1, 50);
331
332 if (retries < U32_MAX)
333 retries++;
334 }
335
336 return reserved ? 0 : -EBUSY;
337}
338
339/**
340 * pvr_kccb_send_cmd_powered() - Send command to the KCCB, with a PM ref held
341 * @pvr_dev: Device pointer.
342 * @cmd: Command to sent.
343 * @kccb_slot: Address to store the KCCB slot for this command. May be %NULL.
344 *
345 * Returns:
346 * * Zero on success, or
347 * * -EBUSY if timeout while waiting for a free KCCB slot.
348 */
349int
350pvr_kccb_send_cmd_powered(struct pvr_device *pvr_dev, struct rogue_fwif_kccb_cmd *cmd,
351 u32 *kccb_slot)
352{
353 int err;
354
355 err = pvr_kccb_reserve_slot_sync(pvr_dev);
356 if (err)
357 return err;
358
359 pvr_kccb_send_cmd_reserved_powered(pvr_dev, cmd, kccb_slot);
360 return 0;
361}
362
363/**
364 * pvr_kccb_send_cmd() - Send command to the KCCB
365 * @pvr_dev: Device pointer.
366 * @cmd: Command to sent.
367 * @kccb_slot: Address to store the KCCB slot for this command. May be %NULL.
368 *
369 * Returns:
370 * * Zero on success, or
371 * * -EBUSY if timeout while waiting for a free KCCB slot.
372 */
373int
374pvr_kccb_send_cmd(struct pvr_device *pvr_dev, struct rogue_fwif_kccb_cmd *cmd,
375 u32 *kccb_slot)
376{
377 int err;
378
379 err = pvr_power_get(pvr_dev);
380 if (err)
381 return err;
382
383 err = pvr_kccb_send_cmd_powered(pvr_dev, cmd, kccb_slot);
384
385 pvr_power_put(pvr_dev);
386
387 return err;
388}
389
390/**
391 * pvr_kccb_wait_for_completion() - Wait for a KCCB command to complete
392 * @pvr_dev: Device pointer.
393 * @slot_nr: KCCB slot to wait on.
394 * @timeout: Timeout length (in jiffies).
395 * @rtn_out: Location to store KCCB command result. May be %NULL.
396 *
397 * Returns:
398 * * Zero on success, or
399 * * -ETIMEDOUT on timeout.
400 */
401int
402pvr_kccb_wait_for_completion(struct pvr_device *pvr_dev, u32 slot_nr,
403 u32 timeout, u32 *rtn_out)
404{
405 int ret = wait_event_timeout(pvr_dev->kccb.rtn_q, READ_ONCE(pvr_dev->kccb.rtn[slot_nr]) &
406 ROGUE_FWIF_KCCB_RTN_SLOT_CMD_EXECUTED, timeout);
407
408 if (ret && rtn_out)
409 *rtn_out = READ_ONCE(pvr_dev->kccb.rtn[slot_nr]);
410
411 return ret ? 0 : -ETIMEDOUT;
412}
413
414/**
415 * pvr_kccb_is_idle() - Returns whether the device's KCCB is idle
416 * @pvr_dev: Device pointer
417 *
418 * Returns:
419 * * %true if the KCCB is idle (contains no commands), or
420 * * %false if the KCCB contains pending commands.
421 */
422bool
423pvr_kccb_is_idle(struct pvr_device *pvr_dev)
424{
425 struct rogue_fwif_ccb_ctl *ctrl = pvr_dev->kccb.ccb.ctrl;
426 bool idle;
427
428 mutex_lock(&pvr_dev->kccb.ccb.lock);
429
430 idle = (READ_ONCE(ctrl->write_offset) == READ_ONCE(ctrl->read_offset));
431
432 mutex_unlock(&pvr_dev->kccb.ccb.lock);
433
434 return idle;
435}
436
437static const char *
438pvr_kccb_fence_get_driver_name(struct dma_fence *f)
439{
440 return PVR_DRIVER_NAME;
441}
442
443static const char *
444pvr_kccb_fence_get_timeline_name(struct dma_fence *f)
445{
446 return "kccb";
447}
448
449static const struct dma_fence_ops pvr_kccb_fence_ops = {
450 .get_driver_name = pvr_kccb_fence_get_driver_name,
451 .get_timeline_name = pvr_kccb_fence_get_timeline_name,
452};
453
454/**
455 * struct pvr_kccb_fence - Fence object used to wait for a KCCB slot
456 */
457struct pvr_kccb_fence {
458 /** @base: Base dma_fence object. */
459 struct dma_fence base;
460
461 /** @node: Node used to insert the fence in the pvr_device::kccb::waiters list. */
462 struct list_head node;
463};
464
465/**
466 * pvr_kccb_wake_up_waiters() - Check the KCCB waiters
467 * @pvr_dev: Target PowerVR device
468 *
469 * Signal as many KCCB fences as we have slots available.
470 */
471void pvr_kccb_wake_up_waiters(struct pvr_device *pvr_dev)
472{
473 struct pvr_kccb_fence *fence, *tmp_fence;
474 u32 used_count, available_count;
475
476 /* Wake up those waiting for KCCB slot execution. */
477 wake_up_all(&pvr_dev->kccb.rtn_q);
478
479 /* Then iterate over all KCCB fences and signal as many as we can. */
480 mutex_lock(&pvr_dev->kccb.ccb.lock);
481 used_count = pvr_kccb_used_slot_count_locked(pvr_dev);
482
483 if (WARN_ON(used_count + pvr_dev->kccb.reserved_count > pvr_kccb_capacity(pvr_dev)))
484 goto out_unlock;
485
486 available_count = pvr_kccb_capacity(pvr_dev) - used_count - pvr_dev->kccb.reserved_count;
487 list_for_each_entry_safe(fence, tmp_fence, &pvr_dev->kccb.waiters, node) {
488 if (!available_count)
489 break;
490
491 list_del(&fence->node);
492 pvr_dev->kccb.reserved_count++;
493 available_count--;
494 dma_fence_signal(&fence->base);
495 dma_fence_put(&fence->base);
496 }
497
498out_unlock:
499 mutex_unlock(&pvr_dev->kccb.ccb.lock);
500}
501
502/**
503 * pvr_kccb_fini() - Cleanup device KCCB
504 * @pvr_dev: Target PowerVR device
505 */
506void pvr_kccb_fini(struct pvr_device *pvr_dev)
507{
508 pvr_ccb_fini(&pvr_dev->kccb.ccb);
509 WARN_ON(!list_empty(&pvr_dev->kccb.waiters));
510 WARN_ON(pvr_dev->kccb.reserved_count);
511}
512
513/**
514 * pvr_kccb_init() - Initialise device KCCB
515 * @pvr_dev: Target PowerVR device
516 *
517 * Returns:
518 * * 0 on success, or
519 * * Any error returned by pvr_ccb_init().
520 */
521int
522pvr_kccb_init(struct pvr_device *pvr_dev)
523{
524 pvr_dev->kccb.slot_count = 1 << ROGUE_FWIF_KCCB_NUMCMDS_LOG2_DEFAULT;
525 INIT_LIST_HEAD(&pvr_dev->kccb.waiters);
526 pvr_dev->kccb.fence_ctx.id = dma_fence_context_alloc(1);
527 spin_lock_init(&pvr_dev->kccb.fence_ctx.lock);
528
529 return pvr_ccb_init(pvr_dev, &pvr_dev->kccb.ccb,
530 ROGUE_FWIF_KCCB_NUMCMDS_LOG2_DEFAULT,
531 sizeof(struct rogue_fwif_kccb_cmd));
532}
533
534/**
535 * pvr_kccb_fence_alloc() - Allocate a pvr_kccb_fence object
536 *
537 * Return:
538 * * NULL if the allocation fails, or
539 * * A valid dma_fence pointer otherwise.
540 */
541struct dma_fence *pvr_kccb_fence_alloc(void)
542{
543 struct pvr_kccb_fence *kccb_fence;
544
545 kccb_fence = kzalloc(sizeof(*kccb_fence), GFP_KERNEL);
546 if (!kccb_fence)
547 return NULL;
548
549 return &kccb_fence->base;
550}
551
552/**
553 * pvr_kccb_fence_put() - Drop a KCCB fence reference
554 * @fence: The fence to drop the reference on.
555 *
556 * If the fence hasn't been initialized yet, dma_fence_free() is called. This
557 * way we have a single function taking care of both cases.
558 */
559void pvr_kccb_fence_put(struct dma_fence *fence)
560{
561 if (!fence)
562 return;
563
564 if (!fence->ops) {
565 dma_fence_free(fence);
566 } else {
567 WARN_ON(fence->ops != &pvr_kccb_fence_ops);
568 dma_fence_put(fence);
569 }
570}
571
572/**
573 * pvr_kccb_reserve_slot() - Reserve a KCCB slot for later use
574 * @pvr_dev: Target PowerVR device
575 * @f: KCCB fence object previously allocated with pvr_kccb_fence_alloc()
576 *
577 * Try to reserve a KCCB slot, and if there's no slot available,
578 * initializes the fence object and queue it to the waiters list.
579 *
580 * If NULL is returned, that means the slot is reserved. In that case,
581 * the @f is freed and shouldn't be accessed after that point.
582 *
583 * Return:
584 * * NULL if a slot was available directly, or
585 * * A valid dma_fence object to wait on if no slot was available.
586 */
587struct dma_fence *
588pvr_kccb_reserve_slot(struct pvr_device *pvr_dev, struct dma_fence *f)
589{
590 struct pvr_kccb_fence *fence = container_of(f, struct pvr_kccb_fence, base);
591 struct dma_fence *out_fence = NULL;
592 u32 used_count;
593
594 mutex_lock(&pvr_dev->kccb.ccb.lock);
595
596 used_count = pvr_kccb_used_slot_count_locked(pvr_dev);
597 if (pvr_dev->kccb.reserved_count >= pvr_kccb_capacity(pvr_dev) - used_count) {
598 dma_fence_init(&fence->base, &pvr_kccb_fence_ops,
599 &pvr_dev->kccb.fence_ctx.lock,
600 pvr_dev->kccb.fence_ctx.id,
601 atomic_inc_return(&pvr_dev->kccb.fence_ctx.seqno));
602 out_fence = dma_fence_get(&fence->base);
603 list_add_tail(&fence->node, &pvr_dev->kccb.waiters);
604 } else {
605 pvr_kccb_fence_put(f);
606 pvr_dev->kccb.reserved_count++;
607 }
608
609 mutex_unlock(&pvr_dev->kccb.ccb.lock);
610
611 return out_fence;
612}
613
614/**
615 * pvr_kccb_release_slot() - Release a KCCB slot reserved with
616 * pvr_kccb_reserve_slot()
617 * @pvr_dev: Target PowerVR device
618 *
619 * Should only be called if something failed after the
620 * pvr_kccb_reserve_slot() call and you know you won't call
621 * pvr_kccb_send_cmd_reserved().
622 */
623void pvr_kccb_release_slot(struct pvr_device *pvr_dev)
624{
625 mutex_lock(&pvr_dev->kccb.ccb.lock);
626 if (!WARN_ON(!pvr_dev->kccb.reserved_count))
627 pvr_dev->kccb.reserved_count--;
628 mutex_unlock(&pvr_dev->kccb.ccb.lock);
629}
630
631/**
632 * pvr_fwccb_init() - Initialise device FWCCB
633 * @pvr_dev: Target PowerVR device
634 *
635 * Returns:
636 * * 0 on success, or
637 * * Any error returned by pvr_ccb_init().
638 */
639int
640pvr_fwccb_init(struct pvr_device *pvr_dev)
641{
642 return pvr_ccb_init(pvr_dev, &pvr_dev->fwccb,
643 ROGUE_FWIF_FWCCB_NUMCMDS_LOG2,
644 sizeof(struct rogue_fwif_fwccb_cmd));
645}
drivers/gpu/drm/imagination/pvr_ccb.h+71−0
@@ -0,0 +1,71 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_CCB_H
5#define PVR_CCB_H
6
7#include "pvr_rogue_fwif.h"
8
9#include <linux/mutex.h>
10#include <linux/types.h>
11
12/* Forward declaration from pvr_device.h. */
13struct pvr_device;
14
15/* Forward declaration from pvr_gem.h. */
16struct pvr_fw_object;
17
18struct pvr_ccb {
19 /** @ctrl_obj: FW object representing CCB control structure. */
20 struct pvr_fw_object *ctrl_obj;
21 /** @ccb_obj: FW object representing CCB. */
22 struct pvr_fw_object *ccb_obj;
23
24 /** @ctrl_fw_addr: FW virtual address of CCB control structure. */
25 u32 ctrl_fw_addr;
26 /** @ccb_fw_addr: FW virtual address of CCB. */
27 u32 ccb_fw_addr;
28
29 /** @num_cmds: Number of commands in this CCB. */
30 u32 num_cmds;
31
32 /** @cmd_size: Size of each command in this CCB, in bytes. */
33 u32 cmd_size;
34
35 /** @lock: Mutex protecting @ctrl and @ccb. */
36 struct mutex lock;
37 /**
38 * @ctrl: Kernel mapping of CCB control structure. @lock must be held
39 * when accessing.
40 */
41 struct rogue_fwif_ccb_ctl *ctrl;
42 /** @ccb: Kernel mapping of CCB. @lock must be held when accessing. */
43 void *ccb;
44};
45
46int pvr_kccb_init(struct pvr_device *pvr_dev);
47void pvr_kccb_fini(struct pvr_device *pvr_dev);
48int pvr_fwccb_init(struct pvr_device *pvr_dev);
49void pvr_ccb_fini(struct pvr_ccb *ccb);
50
51void pvr_fwccb_process(struct pvr_device *pvr_dev);
52
53struct dma_fence *pvr_kccb_fence_alloc(void);
54void pvr_kccb_fence_put(struct dma_fence *fence);
55struct dma_fence *
56pvr_kccb_reserve_slot(struct pvr_device *pvr_dev, struct dma_fence *f);
57void pvr_kccb_release_slot(struct pvr_device *pvr_dev);
58int pvr_kccb_send_cmd(struct pvr_device *pvr_dev,
59 struct rogue_fwif_kccb_cmd *cmd, u32 *kccb_slot);
60int pvr_kccb_send_cmd_powered(struct pvr_device *pvr_dev,
61 struct rogue_fwif_kccb_cmd *cmd,
62 u32 *kccb_slot);
63void pvr_kccb_send_cmd_reserved_powered(struct pvr_device *pvr_dev,
64 struct rogue_fwif_kccb_cmd *cmd,
65 u32 *kccb_slot);
66int pvr_kccb_wait_for_completion(struct pvr_device *pvr_dev, u32 slot_nr, u32 timeout,
67 u32 *rtn_out);
68bool pvr_kccb_is_idle(struct pvr_device *pvr_dev);
69void pvr_kccb_wake_up_waiters(struct pvr_device *pvr_dev);
70
71#endif /* PVR_CCB_H */
drivers/gpu/drm/imagination/pvr_cccb.c+267−0
@@ -0,0 +1,267 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_ccb.h"
5#include "pvr_cccb.h"
6#include "pvr_device.h"
7#include "pvr_gem.h"
8#include "pvr_hwrt.h"
9
10#include <linux/compiler.h>
11#include <linux/delay.h>
12#include <linux/jiffies.h>
13#include <linux/mutex.h>
14#include <linux/types.h>
15
16static __always_inline u32
17get_ccb_space(u32 w_off, u32 r_off, u32 ccb_size)
18{
19 return (((r_off) - (w_off)) + ((ccb_size) - 1)) & ((ccb_size) - 1);
20}
21
22static void
23cccb_ctrl_init(void *cpu_ptr, void *priv)
24{
25 struct rogue_fwif_cccb_ctl *ctrl = cpu_ptr;
26 struct pvr_cccb *pvr_cccb = priv;
27
28 WRITE_ONCE(ctrl->write_offset, 0);
29 WRITE_ONCE(ctrl->read_offset, 0);
30 WRITE_ONCE(ctrl->dep_offset, 0);
31 WRITE_ONCE(ctrl->wrap_mask, pvr_cccb->wrap_mask);
32}
33
34/**
35 * pvr_cccb_init() - Initialise a Client CCB
36 * @pvr_dev: Device pointer.
37 * @pvr_cccb: Pointer to Client CCB structure to initialise.
38 * @size_log2: Log2 size of Client CCB in bytes.
39 * @name: Name of owner of Client CCB. Used for fence context.
40 *
41 * Return:
42 * * Zero on success, or
43 * * Any error code returned by pvr_fw_object_create_and_map().
44 */
45int
46pvr_cccb_init(struct pvr_device *pvr_dev, struct pvr_cccb *pvr_cccb,
47 u32 size_log2, const char *name)
48{
49 size_t size = 1 << size_log2;
50 int err;
51
52 pvr_cccb->size = size;
53 pvr_cccb->write_offset = 0;
54 pvr_cccb->wrap_mask = size - 1;
55
56 /*
57 * Map CCCB and control structure as uncached, so we don't have to flush
58 * CPU cache repeatedly when polling for space.
59 */
60 pvr_cccb->ctrl = pvr_fw_object_create_and_map(pvr_dev, sizeof(*pvr_cccb->ctrl),
61 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
62 cccb_ctrl_init, pvr_cccb,
63 &pvr_cccb->ctrl_obj);
64 if (IS_ERR(pvr_cccb->ctrl))
65 return PTR_ERR(pvr_cccb->ctrl);
66
67 pvr_cccb->cccb = pvr_fw_object_create_and_map(pvr_dev, size,
68 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
69 NULL, NULL, &pvr_cccb->cccb_obj);
70 if (IS_ERR(pvr_cccb->cccb)) {
71 err = PTR_ERR(pvr_cccb->cccb);
72 goto err_free_ctrl;
73 }
74
75 pvr_fw_object_get_fw_addr(pvr_cccb->ctrl_obj, &pvr_cccb->ctrl_fw_addr);
76 pvr_fw_object_get_fw_addr(pvr_cccb->cccb_obj, &pvr_cccb->cccb_fw_addr);
77
78 return 0;
79
80err_free_ctrl:
81 pvr_fw_object_unmap_and_destroy(pvr_cccb->ctrl_obj);
82
83 return err;
84}
85
86/**
87 * pvr_cccb_fini() - Release Client CCB structure
88 * @pvr_cccb: Client CCB to release.
89 */
90void
91pvr_cccb_fini(struct pvr_cccb *pvr_cccb)
92{
93 pvr_fw_object_unmap_and_destroy(pvr_cccb->cccb_obj);
94 pvr_fw_object_unmap_and_destroy(pvr_cccb->ctrl_obj);
95}
96
97/**
98 * pvr_cccb_cmdseq_fits() - Check if a command sequence fits in the CCCB
99 * @pvr_cccb: Target Client CCB.
100 * @size: Size of the command sequence.
101 *
102 * Check if a command sequence fits in the CCCB we have at hand.
103 *
104 * Return:
105 * * true if the command sequence fits in the CCCB, or
106 * * false otherwise.
107 */
108bool pvr_cccb_cmdseq_fits(struct pvr_cccb *pvr_cccb, size_t size)
109{
110 struct rogue_fwif_cccb_ctl *ctrl = pvr_cccb->ctrl;
111 u32 read_offset, remaining;
112 bool fits = false;
113
114 read_offset = READ_ONCE(ctrl->read_offset);
115 remaining = pvr_cccb->size - pvr_cccb->write_offset;
116
117 /* Always ensure we have enough room for a padding command at the end of the CCCB.
118 * If our command sequence does not fit, reserve the remaining space for a padding
119 * command.
120 */
121 if (size + PADDING_COMMAND_SIZE > remaining)
122 size += remaining;
123
124 if (get_ccb_space(pvr_cccb->write_offset, read_offset, pvr_cccb->size) >= size)
125 fits = true;
126
127 return fits;
128}
129
130/**
131 * pvr_cccb_write_command_with_header() - Write a command + command header to a
132 * Client CCB
133 * @pvr_cccb: Target Client CCB.
134 * @cmd_type: Client CCB command type. Must be one of %ROGUE_FWIF_CCB_CMD_TYPE_*.
135 * @cmd_size: Size of command in bytes.
136 * @cmd_data: Pointer to command to write.
137 * @ext_job_ref: External job reference.
138 * @int_job_ref: Internal job reference.
139 *
140 * Caller must make sure there's enough space in CCCB to queue this command. This
141 * can be done by calling pvr_cccb_cmdseq_fits().
142 *
143 * This function is not protected by any lock. The caller must ensure there's
144 * no concurrent caller, which should be guaranteed by the drm_sched model (job
145 * submission is serialized in drm_sched_main()).
146 */
147void
148pvr_cccb_write_command_with_header(struct pvr_cccb *pvr_cccb, u32 cmd_type, u32 cmd_size,
149 void *cmd_data, u32 ext_job_ref, u32 int_job_ref)
150{
151 u32 sz_with_hdr = pvr_cccb_get_size_of_cmd_with_hdr(cmd_size);
152 struct rogue_fwif_ccb_cmd_header cmd_header = {
153 .cmd_type = cmd_type,
154 .cmd_size = ALIGN(cmd_size, 8),
155 .ext_job_ref = ext_job_ref,
156 .int_job_ref = int_job_ref,
157 };
158 struct rogue_fwif_cccb_ctl *ctrl = pvr_cccb->ctrl;
159 u32 remaining = pvr_cccb->size - pvr_cccb->write_offset;
160 u32 required_size, cccb_space, read_offset;
161
162 /*
163 * Always ensure we have enough room for a padding command at the end of
164 * the CCCB.
165 */
166 if (remaining < sz_with_hdr + PADDING_COMMAND_SIZE) {
167 /*
168 * Command would need to wrap, so we need to pad the remainder
169 * of the CCCB.
170 */
171 required_size = sz_with_hdr + remaining;
172 } else {
173 required_size = sz_with_hdr;
174 }
175
176 read_offset = READ_ONCE(ctrl->read_offset);
177 cccb_space = get_ccb_space(pvr_cccb->write_offset, read_offset, pvr_cccb->size);
178 if (WARN_ON(cccb_space < required_size))
179 return;
180
181 if (required_size != sz_with_hdr) {
182 /* Add padding command */
183 struct rogue_fwif_ccb_cmd_header pad_cmd = {
184 .cmd_type = ROGUE_FWIF_CCB_CMD_TYPE_PADDING,
185 .cmd_size = remaining - sizeof(pad_cmd),
186 };
187
188 memcpy(&pvr_cccb->cccb[pvr_cccb->write_offset], &pad_cmd, sizeof(pad_cmd));
189 pvr_cccb->write_offset = 0;
190 }
191
192 memcpy(&pvr_cccb->cccb[pvr_cccb->write_offset], &cmd_header, sizeof(cmd_header));
193 memcpy(&pvr_cccb->cccb[pvr_cccb->write_offset + sizeof(cmd_header)], cmd_data, cmd_size);
194 pvr_cccb->write_offset += sz_with_hdr;
195}
196
197static void fill_cmd_kick_data(struct pvr_cccb *cccb, u32 ctx_fw_addr,
198 struct pvr_hwrt_data *hwrt,
199 struct rogue_fwif_kccb_cmd_kick_data *k)
200{
201 k->context_fw_addr = ctx_fw_addr;
202 k->client_woff_update = cccb->write_offset;
203 k->client_wrap_mask_update = cccb->wrap_mask;
204
205 if (hwrt) {
206 u32 cleanup_state_offset = offsetof(struct rogue_fwif_hwrtdata, cleanup_state);
207
208 pvr_fw_object_get_fw_addr_offset(hwrt->fw_obj, cleanup_state_offset,
209 &k->cleanup_ctl_fw_addr[k->num_cleanup_ctl++]);
210 }
211}
212
213/**
214 * pvr_cccb_send_kccb_kick: Send KCCB kick to trigger command processing
215 * @pvr_dev: Device pointer.
216 * @pvr_cccb: Pointer to CCCB to process.
217 * @cctx_fw_addr: FW virtual address for context owning this Client CCB.
218 * @hwrt: HWRT data set associated with this kick. May be %NULL.
219 *
220 * You must call pvr_kccb_reserve_slot() and wait for the returned fence to
221 * signal (if this function didn't return NULL) before calling
222 * pvr_cccb_send_kccb_kick().
223 */
224void
225pvr_cccb_send_kccb_kick(struct pvr_device *pvr_dev,
226 struct pvr_cccb *pvr_cccb, u32 cctx_fw_addr,
227 struct pvr_hwrt_data *hwrt)
228{
229 struct rogue_fwif_kccb_cmd cmd_kick = {
230 .cmd_type = ROGUE_FWIF_KCCB_CMD_KICK,
231 };
232
233 fill_cmd_kick_data(pvr_cccb, cctx_fw_addr, hwrt, &cmd_kick.cmd_data.cmd_kick_data);
234
235 /* Make sure the writes to the CCCB are flushed before sending the KICK. */
236 wmb();
237
238 pvr_kccb_send_cmd_reserved_powered(pvr_dev, &cmd_kick, NULL);
239}
240
241void
242pvr_cccb_send_kccb_combined_kick(struct pvr_device *pvr_dev,
243 struct pvr_cccb *geom_cccb,
244 struct pvr_cccb *frag_cccb,
245 u32 geom_ctx_fw_addr,
246 u32 frag_ctx_fw_addr,
247 struct pvr_hwrt_data *hwrt,
248 bool frag_is_pr)
249{
250 struct rogue_fwif_kccb_cmd cmd_kick = {
251 .cmd_type = ROGUE_FWIF_KCCB_CMD_COMBINED_GEOM_FRAG_KICK,
252 };
253
254 fill_cmd_kick_data(geom_cccb, geom_ctx_fw_addr, hwrt,
255 &cmd_kick.cmd_data.combined_geom_frag_cmd_kick_data.geom_cmd_kick_data);
256
257 /* If this is a partial-render job, we don't attach resources to cleanup-ctl array,
258 * because the resources are already retained by the geometry job.
259 */
260 fill_cmd_kick_data(frag_cccb, frag_ctx_fw_addr, frag_is_pr ? NULL : hwrt,
261 &cmd_kick.cmd_data.combined_geom_frag_cmd_kick_data.frag_cmd_kick_data);
262
263 /* Make sure the writes to the CCCB are flushed before sending the KICK. */
264 wmb();
265
266 pvr_kccb_send_cmd_reserved_powered(pvr_dev, &cmd_kick, NULL);
267}
drivers/gpu/drm/imagination/pvr_cccb.h+109−0
@@ -0,0 +1,109 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_CCCB_H
5#define PVR_CCCB_H
6
7#include "pvr_rogue_fwif.h"
8#include "pvr_rogue_fwif_shared.h"
9
10#include <linux/mutex.h>
11#include <linux/types.h>
12
13#define PADDING_COMMAND_SIZE sizeof(struct rogue_fwif_ccb_cmd_header)
14
15/* Forward declaration from pvr_device.h. */
16struct pvr_device;
17
18/* Forward declaration from pvr_gem.h. */
19struct pvr_fw_object;
20
21/* Forward declaration from pvr_hwrt.h. */
22struct pvr_hwrt_data;
23
24struct pvr_cccb {
25 /** @ctrl_obj: FW object representing CCCB control structure. */
26 struct pvr_fw_object *ctrl_obj;
27
28 /** @ccb_obj: FW object representing CCCB. */
29 struct pvr_fw_object *cccb_obj;
30
31 /**
32 * @ctrl: Kernel mapping of CCCB control structure. @lock must be held
33 * when accessing.
34 */
35 struct rogue_fwif_cccb_ctl *ctrl;
36
37 /** @cccb: Kernel mapping of CCCB. @lock must be held when accessing.*/
38 u8 *cccb;
39
40 /** @ctrl_fw_addr: FW virtual address of CCCB control structure. */
41 u32 ctrl_fw_addr;
42 /** @ccb_fw_addr: FW virtual address of CCCB. */
43 u32 cccb_fw_addr;
44
45 /** @size: Size of CCCB in bytes. */
46 size_t size;
47
48 /** @write_offset: CCCB write offset. */
49 u32 write_offset;
50
51 /** @wrap_mask: CCCB wrap mask. */
52 u32 wrap_mask;
53};
54
55int pvr_cccb_init(struct pvr_device *pvr_dev, struct pvr_cccb *cccb,
56 u32 size_log2, const char *name);
57void pvr_cccb_fini(struct pvr_cccb *cccb);
58
59void pvr_cccb_write_command_with_header(struct pvr_cccb *pvr_cccb,
60 u32 cmd_type, u32 cmd_size, void *cmd_data,
61 u32 ext_job_ref, u32 int_job_ref);
62void pvr_cccb_send_kccb_kick(struct pvr_device *pvr_dev,
63 struct pvr_cccb *pvr_cccb, u32 cctx_fw_addr,
64 struct pvr_hwrt_data *hwrt);
65void pvr_cccb_send_kccb_combined_kick(struct pvr_device *pvr_dev,
66 struct pvr_cccb *geom_cccb,
67 struct pvr_cccb *frag_cccb,
68 u32 geom_ctx_fw_addr,
69 u32 frag_ctx_fw_addr,
70 struct pvr_hwrt_data *hwrt,
71 bool frag_is_pr);
72bool pvr_cccb_cmdseq_fits(struct pvr_cccb *pvr_cccb, size_t size);
73
74/**
75 * pvr_cccb_get_size_of_cmd_with_hdr() - Get the size of a command and its header.
76 * @cmd_size: Command size.
77 *
78 * Returns the size of the command and its header.
79 */
80static __always_inline u32
81pvr_cccb_get_size_of_cmd_with_hdr(u32 cmd_size)
82{
83 WARN_ON(!IS_ALIGNED(cmd_size, 8));
84 return sizeof(struct rogue_fwif_ccb_cmd_header) + ALIGN(cmd_size, 8);
85}
86
87/**
88 * pvr_cccb_cmdseq_can_fit() - Check if a command sequence can fit in the CCCB.
89 * @size: Command sequence size.
90 *
91 * Returns:
92 * * true it the CCCB is big enough to contain a command sequence, or
93 * * false otherwise.
94 */
95static __always_inline bool
96pvr_cccb_cmdseq_can_fit(struct pvr_cccb *pvr_cccb, size_t size)
97{
98 /* We divide the capacity by two to simplify our CCCB fencing logic:
99 * we want to be sure that, no matter what we had queued before, we
100 * are able to either queue our command sequence at the end or add a
101 * padding command and queue the command sequence at the beginning
102 * of the CCCB. If the command sequence size is bigger than half the
103 * CCCB capacity, we'd have to queue the padding command and make sure
104 * the FW is done processing it before queueing our command sequence.
105 */
106 return size + PADDING_COMMAND_SIZE <= pvr_cccb->size / 2;
107}
108
109#endif /* PVR_CCCB_H */
drivers/gpu/drm/imagination/pvr_context.c+464−0
@@ -0,0 +1,464 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_cccb.h"
5#include "pvr_context.h"
6#include "pvr_device.h"
7#include "pvr_drv.h"
8#include "pvr_gem.h"
9#include "pvr_job.h"
10#include "pvr_power.h"
11#include "pvr_rogue_fwif.h"
12#include "pvr_rogue_fwif_common.h"
13#include "pvr_rogue_fwif_resetframework.h"
14#include "pvr_stream.h"
15#include "pvr_stream_defs.h"
16#include "pvr_vm.h"
17
18#include <drm/drm_auth.h>
19#include <drm/drm_managed.h>
20#include <linux/errno.h>
21#include <linux/kernel.h>
22#include <linux/sched.h>
23#include <linux/slab.h>
24#include <linux/string.h>
25#include <linux/types.h>
26#include <linux/xarray.h>
27
28static int
29remap_priority(struct pvr_file *pvr_file, s32 uapi_priority,
30 enum pvr_context_priority *priority_out)
31{
32 switch (uapi_priority) {
33 case DRM_PVR_CTX_PRIORITY_LOW:
34 *priority_out = PVR_CTX_PRIORITY_LOW;
35 break;
36 case DRM_PVR_CTX_PRIORITY_NORMAL:
37 *priority_out = PVR_CTX_PRIORITY_MEDIUM;
38 break;
39 case DRM_PVR_CTX_PRIORITY_HIGH:
40 if (!capable(CAP_SYS_NICE) && !drm_is_current_master(from_pvr_file(pvr_file)))
41 return -EACCES;
42 *priority_out = PVR_CTX_PRIORITY_HIGH;
43 break;
44 default:
45 return -EINVAL;
46 }
47
48 return 0;
49}
50
51static int get_fw_obj_size(enum drm_pvr_ctx_type type)
52{
53 switch (type) {
54 case DRM_PVR_CTX_TYPE_RENDER:
55 return sizeof(struct rogue_fwif_fwrendercontext);
56 case DRM_PVR_CTX_TYPE_COMPUTE:
57 return sizeof(struct rogue_fwif_fwcomputecontext);
58 case DRM_PVR_CTX_TYPE_TRANSFER_FRAG:
59 return sizeof(struct rogue_fwif_fwtransfercontext);
60 }
61
62 return -EINVAL;
63}
64
65static int
66process_static_context_state(struct pvr_device *pvr_dev, const struct pvr_stream_cmd_defs *cmd_defs,
67 u64 stream_user_ptr, u32 stream_size, void *dest)
68{
69 void *stream;
70 int err;
71
72 stream = kzalloc(stream_size, GFP_KERNEL);
73 if (!stream)
74 return -ENOMEM;
75
76 if (copy_from_user(stream, u64_to_user_ptr(stream_user_ptr), stream_size)) {
77 err = -EFAULT;
78 goto err_free;
79 }
80
81 err = pvr_stream_process(pvr_dev, cmd_defs, stream, stream_size, dest);
82 if (err)
83 goto err_free;
84
85 kfree(stream);
86
87 return 0;
88
89err_free:
90 kfree(stream);
91
92 return err;
93}
94
95static int init_render_fw_objs(struct pvr_context *ctx,
96 struct drm_pvr_ioctl_create_context_args *args,
97 void *fw_ctx_map)
98{
99 struct rogue_fwif_static_rendercontext_state *static_rendercontext_state;
100 struct rogue_fwif_fwrendercontext *fw_render_context = fw_ctx_map;
101
102 if (!args->static_context_state_len)
103 return -EINVAL;
104
105 static_rendercontext_state = &fw_render_context->static_render_context_state;
106
107 /* Copy static render context state from userspace. */
108 return process_static_context_state(ctx->pvr_dev,
109 &pvr_static_render_context_state_stream,
110 args->static_context_state,
111 args->static_context_state_len,
112 &static_rendercontext_state->ctxswitch_regs[0]);
113}
114
115static int init_compute_fw_objs(struct pvr_context *ctx,
116 struct drm_pvr_ioctl_create_context_args *args,
117 void *fw_ctx_map)
118{
119 struct rogue_fwif_fwcomputecontext *fw_compute_context = fw_ctx_map;
120 struct rogue_fwif_cdm_registers_cswitch *ctxswitch_regs;
121
122 if (!args->static_context_state_len)
123 return -EINVAL;
124
125 ctxswitch_regs = &fw_compute_context->static_compute_context_state.ctxswitch_regs;
126
127 /* Copy static render context state from userspace. */
128 return process_static_context_state(ctx->pvr_dev,
129 &pvr_static_compute_context_state_stream,
130 args->static_context_state,
131 args->static_context_state_len,
132 ctxswitch_regs);
133}
134
135static int init_transfer_fw_objs(struct pvr_context *ctx,
136 struct drm_pvr_ioctl_create_context_args *args,
137 void *fw_ctx_map)
138{
139 if (args->static_context_state_len)
140 return -EINVAL;
141
142 return 0;
143}
144
145static int init_fw_objs(struct pvr_context *ctx,
146 struct drm_pvr_ioctl_create_context_args *args,
147 void *fw_ctx_map)
148{
149 switch (ctx->type) {
150 case DRM_PVR_CTX_TYPE_RENDER:
151 return init_render_fw_objs(ctx, args, fw_ctx_map);
152 case DRM_PVR_CTX_TYPE_COMPUTE:
153 return init_compute_fw_objs(ctx, args, fw_ctx_map);
154 case DRM_PVR_CTX_TYPE_TRANSFER_FRAG:
155 return init_transfer_fw_objs(ctx, args, fw_ctx_map);
156 }
157
158 return -EINVAL;
159}
160
161static void
162ctx_fw_data_init(void *cpu_ptr, void *priv)
163{
164 struct pvr_context *ctx = priv;
165
166 memcpy(cpu_ptr, ctx->data, ctx->data_size);
167}
168
169/**
170 * pvr_context_destroy_queues() - Destroy all queues attached to a context.
171 * @ctx: Context to destroy queues on.
172 *
173 * Should be called when the last reference to a context object is dropped.
174 * It releases all resources attached to the queues bound to this context.
175 */
176static void pvr_context_destroy_queues(struct pvr_context *ctx)
177{
178 switch (ctx->type) {
179 case DRM_PVR_CTX_TYPE_RENDER:
180 pvr_queue_destroy(ctx->queues.fragment);
181 pvr_queue_destroy(ctx->queues.geometry);
182 break;
183 case DRM_PVR_CTX_TYPE_COMPUTE:
184 pvr_queue_destroy(ctx->queues.compute);
185 break;
186 case DRM_PVR_CTX_TYPE_TRANSFER_FRAG:
187 pvr_queue_destroy(ctx->queues.transfer);
188 break;
189 }
190}
191
192/**
193 * pvr_context_create_queues() - Create all queues attached to a context.
194 * @ctx: Context to create queues on.
195 * @args: Context creation arguments passed by userspace.
196 * @fw_ctx_map: CPU mapping of the FW context object.
197 *
198 * Return:
199 * * 0 on success, or
200 * * A negative error code otherwise.
201 */
202static int pvr_context_create_queues(struct pvr_context *ctx,
203 struct drm_pvr_ioctl_create_context_args *args,
204 void *fw_ctx_map)
205{
206 int err;
207
208 switch (ctx->type) {
209 case DRM_PVR_CTX_TYPE_RENDER:
210 ctx->queues.geometry = pvr_queue_create(ctx, DRM_PVR_JOB_TYPE_GEOMETRY,
211 args, fw_ctx_map);
212 if (IS_ERR(ctx->queues.geometry)) {
213 err = PTR_ERR(ctx->queues.geometry);
214 ctx->queues.geometry = NULL;
215 goto err_destroy_queues;
216 }
217
218 ctx->queues.fragment = pvr_queue_create(ctx, DRM_PVR_JOB_TYPE_FRAGMENT,
219 args, fw_ctx_map);
220 if (IS_ERR(ctx->queues.fragment)) {
221 err = PTR_ERR(ctx->queues.fragment);
222 ctx->queues.fragment = NULL;
223 goto err_destroy_queues;
224 }
225 return 0;
226
227 case DRM_PVR_CTX_TYPE_COMPUTE:
228 ctx->queues.compute = pvr_queue_create(ctx, DRM_PVR_JOB_TYPE_COMPUTE,
229 args, fw_ctx_map);
230 if (IS_ERR(ctx->queues.compute)) {
231 err = PTR_ERR(ctx->queues.compute);
232 ctx->queues.compute = NULL;
233 goto err_destroy_queues;
234 }
235 return 0;
236
237 case DRM_PVR_CTX_TYPE_TRANSFER_FRAG:
238 ctx->queues.transfer = pvr_queue_create(ctx, DRM_PVR_JOB_TYPE_TRANSFER_FRAG,
239 args, fw_ctx_map);
240 if (IS_ERR(ctx->queues.transfer)) {
241 err = PTR_ERR(ctx->queues.transfer);
242 ctx->queues.transfer = NULL;
243 goto err_destroy_queues;
244 }
245 return 0;
246 }
247
248 return -EINVAL;
249
250err_destroy_queues:
251 pvr_context_destroy_queues(ctx);
252 return err;
253}
254
255/**
256 * pvr_context_kill_queues() - Kill queues attached to context.
257 * @ctx: Context to kill queues on.
258 *
259 * Killing the queues implies making them unusable for future jobs, while still
260 * letting the currently submitted jobs a chance to finish. Queue resources will
261 * stay around until pvr_context_destroy_queues() is called.
262 */
263static void pvr_context_kill_queues(struct pvr_context *ctx)
264{
265 switch (ctx->type) {
266 case DRM_PVR_CTX_TYPE_RENDER:
267 pvr_queue_kill(ctx->queues.fragment);
268 pvr_queue_kill(ctx->queues.geometry);
269 break;
270 case DRM_PVR_CTX_TYPE_COMPUTE:
271 pvr_queue_kill(ctx->queues.compute);
272 break;
273 case DRM_PVR_CTX_TYPE_TRANSFER_FRAG:
274 pvr_queue_kill(ctx->queues.transfer);
275 break;
276 }
277}
278
279/**
280 * pvr_context_create() - Create a context.
281 * @pvr_file: File to attach the created context to.
282 * @args: Context creation arguments.
283 *
284 * Return:
285 * * 0 on success, or
286 * * A negative error code on failure.
287 */
288int pvr_context_create(struct pvr_file *pvr_file, struct drm_pvr_ioctl_create_context_args *args)
289{
290 struct pvr_device *pvr_dev = pvr_file->pvr_dev;
291 struct pvr_context *ctx;
292 int ctx_size;
293 int err;
294
295 /* Context creation flags are currently unused and must be zero. */
296 if (args->flags)
297 return -EINVAL;
298
299 ctx_size = get_fw_obj_size(args->type);
300 if (ctx_size < 0)
301 return ctx_size;
302
303 ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
304 if (!ctx)
305 return -ENOMEM;
306
307 ctx->data_size = ctx_size;
308 ctx->type = args->type;
309 ctx->flags = args->flags;
310 ctx->pvr_dev = pvr_dev;
311 kref_init(&ctx->ref_count);
312
313 err = remap_priority(pvr_file, args->priority, &ctx->priority);
314 if (err)
315 goto err_free_ctx;
316
317 ctx->vm_ctx = pvr_vm_context_lookup(pvr_file, args->vm_context_handle);
318 if (IS_ERR(ctx->vm_ctx)) {
319 err = PTR_ERR(ctx->vm_ctx);
320 goto err_free_ctx;
321 }
322
323 ctx->data = kzalloc(ctx_size, GFP_KERNEL);
324 if (!ctx->data) {
325 err = -ENOMEM;
326 goto err_put_vm;
327 }
328
329 err = pvr_context_create_queues(ctx, args, ctx->data);
330 if (err)
331 goto err_free_ctx_data;
332
333 err = init_fw_objs(ctx, args, ctx->data);
334 if (err)
335 goto err_destroy_queues;
336
337 err = pvr_fw_object_create(pvr_dev, ctx_size, PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
338 ctx_fw_data_init, ctx, &ctx->fw_obj);
339 if (err)
340 goto err_free_ctx_data;
341
342 err = xa_alloc(&pvr_dev->ctx_ids, &ctx->ctx_id, ctx, xa_limit_32b, GFP_KERNEL);
343 if (err)
344 goto err_destroy_fw_obj;
345
346 err = xa_alloc(&pvr_file->ctx_handles, &args->handle, ctx, xa_limit_32b, GFP_KERNEL);
347 if (err) {
348 /*
349 * It's possible that another thread could have taken a reference on the context at
350 * this point as it is in the ctx_ids xarray. Therefore instead of directly
351 * destroying the context, drop a reference instead.
352 */
353 pvr_context_put(ctx);
354 return err;
355 }
356
357 return 0;
358
359err_destroy_fw_obj:
360 pvr_fw_object_destroy(ctx->fw_obj);
361
362err_destroy_queues:
363 pvr_context_destroy_queues(ctx);
364
365err_free_ctx_data:
366 kfree(ctx->data);
367
368err_put_vm:
369 pvr_vm_context_put(ctx->vm_ctx);
370
371err_free_ctx:
372 kfree(ctx);
373 return err;
374}
375
376static void
377pvr_context_release(struct kref *ref_count)
378{
379 struct pvr_context *ctx =
380 container_of(ref_count, struct pvr_context, ref_count);
381 struct pvr_device *pvr_dev = ctx->pvr_dev;
382
383 xa_erase(&pvr_dev->ctx_ids, ctx->ctx_id);
384 pvr_context_destroy_queues(ctx);
385 pvr_fw_object_destroy(ctx->fw_obj);
386 kfree(ctx->data);
387 pvr_vm_context_put(ctx->vm_ctx);
388 kfree(ctx);
389}
390
391/**
392 * pvr_context_put() - Release reference on context
393 * @ctx: Target context.
394 */
395void
396pvr_context_put(struct pvr_context *ctx)
397{
398 if (ctx)
399 kref_put(&ctx->ref_count, pvr_context_release);
400}
401
402/**
403 * pvr_context_destroy() - Destroy context
404 * @pvr_file: Pointer to pvr_file structure.
405 * @handle: Userspace context handle.
406 *
407 * Removes context from context list and drops initial reference. Context will
408 * then be destroyed once all outstanding references are dropped.
409 *
410 * Return:
411 * * 0 on success, or
412 * * -%EINVAL if context not in context list.
413 */
414int
415pvr_context_destroy(struct pvr_file *pvr_file, u32 handle)
416{
417 struct pvr_context *ctx = xa_erase(&pvr_file->ctx_handles, handle);
418
419 if (!ctx)
420 return -EINVAL;
421
422 /* Make sure nothing can be queued to the queues after that point. */
423 pvr_context_kill_queues(ctx);
424
425 /* Release the reference held by the handle set. */
426 pvr_context_put(ctx);
427
428 return 0;
429}
430
431/**
432 * pvr_destroy_contexts_for_file: Destroy any contexts associated with the given file
433 * @pvr_file: Pointer to pvr_file structure.
434 *
435 * Removes all contexts associated with @pvr_file from the device context list and drops initial
436 * references. Contexts will then be destroyed once all outstanding references are dropped.
437 */
438void pvr_destroy_contexts_for_file(struct pvr_file *pvr_file)
439{
440 struct pvr_context *ctx;
441 unsigned long handle;
442
443 xa_for_each(&pvr_file->ctx_handles, handle, ctx)
444 pvr_context_destroy(pvr_file, handle);
445}
446
447/**
448 * pvr_context_device_init() - Device level initialization for queue related resources.
449 * @pvr_dev: The device to initialize.
450 */
451void pvr_context_device_init(struct pvr_device *pvr_dev)
452{
453 xa_init_flags(&pvr_dev->ctx_ids, XA_FLAGS_ALLOC1);
454}
455
456/**
457 * pvr_context_device_fini() - Device level cleanup for queue related resources.
458 * @pvr_dev: The device to cleanup.
459 */
460void pvr_context_device_fini(struct pvr_device *pvr_dev)
461{
462 WARN_ON(!xa_empty(&pvr_dev->ctx_ids));
463 xa_destroy(&pvr_dev->ctx_ids);
464}
drivers/gpu/drm/imagination/pvr_context.h+205−0
@@ -0,0 +1,205 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_CONTEXT_H
5#define PVR_CONTEXT_H
6
7#include <drm/gpu_scheduler.h>
8
9#include <linux/compiler_attributes.h>
10#include <linux/dma-fence.h>
11#include <linux/kref.h>
12#include <linux/types.h>
13#include <linux/xarray.h>
14#include <uapi/drm/pvr_drm.h>
15
16#include "pvr_cccb.h"
17#include "pvr_device.h"
18#include "pvr_queue.h"
19
20/* Forward declaration from pvr_gem.h. */
21struct pvr_fw_object;
22
23enum pvr_context_priority {
24 PVR_CTX_PRIORITY_LOW = 0,
25 PVR_CTX_PRIORITY_MEDIUM,
26 PVR_CTX_PRIORITY_HIGH,
27};
28
29/**
30 * struct pvr_context - Context data
31 */
32struct pvr_context {
33 /** @ref_count: Refcount for context. */
34 struct kref ref_count;
35
36 /** @pvr_dev: Pointer to owning device. */
37 struct pvr_device *pvr_dev;
38
39 /** @vm_ctx: Pointer to associated VM context. */
40 struct pvr_vm_context *vm_ctx;
41
42 /** @type: Type of context. */
43 enum drm_pvr_ctx_type type;
44
45 /** @flags: Context flags. */
46 u32 flags;
47
48 /** @priority: Context priority*/
49 enum pvr_context_priority priority;
50
51 /** @fw_obj: FW object representing FW-side context data. */
52 struct pvr_fw_object *fw_obj;
53
54 /** @data: Pointer to local copy of FW context data. */
55 void *data;
56
57 /** @data_size: Size of FW context data, in bytes. */
58 u32 data_size;
59
60 /** @ctx_id: FW context ID. */
61 u32 ctx_id;
62
63 /**
64 * @faulty: Set to 1 when the context queues had unfinished job when
65 * a GPU reset happened.
66 *
67 * In that case, the context is in an inconsistent state and can't be
68 * used anymore.
69 */
70 atomic_t faulty;
71
72 /** @queues: Union containing all kind of queues. */
73 union {
74 struct {
75 /** @geometry: Geometry queue. */
76 struct pvr_queue *geometry;
77
78 /** @fragment: Fragment queue. */
79 struct pvr_queue *fragment;
80 };
81
82 /** @compute: Compute queue. */
83 struct pvr_queue *compute;
84
85 /** @compute: Transfer queue. */
86 struct pvr_queue *transfer;
87 } queues;
88};
89
90static __always_inline struct pvr_queue *
91pvr_context_get_queue_for_job(struct pvr_context *ctx, enum drm_pvr_job_type type)
92{
93 switch (type) {
94 case DRM_PVR_JOB_TYPE_GEOMETRY:
95 return ctx->type == DRM_PVR_CTX_TYPE_RENDER ? ctx->queues.geometry : NULL;
96 case DRM_PVR_JOB_TYPE_FRAGMENT:
97 return ctx->type == DRM_PVR_CTX_TYPE_RENDER ? ctx->queues.fragment : NULL;
98 case DRM_PVR_JOB_TYPE_COMPUTE:
99 return ctx->type == DRM_PVR_CTX_TYPE_COMPUTE ? ctx->queues.compute : NULL;
100 case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
101 return ctx->type == DRM_PVR_CTX_TYPE_TRANSFER_FRAG ? ctx->queues.transfer : NULL;
102 }
103
104 return NULL;
105}
106
107/**
108 * pvr_context_get() - Take additional reference on context.
109 * @ctx: Context pointer.
110 *
111 * Call pvr_context_put() to release.
112 *
113 * Returns:
114 * * The requested context on success, or
115 * * %NULL if no context pointer passed.
116 */
117static __always_inline struct pvr_context *
118pvr_context_get(struct pvr_context *ctx)
119{
120 if (ctx)
121 kref_get(&ctx->ref_count);
122
123 return ctx;
124}
125
126/**
127 * pvr_context_lookup() - Lookup context pointer from handle and file.
128 * @pvr_file: Pointer to pvr_file structure.
129 * @handle: Context handle.
130 *
131 * Takes reference on context. Call pvr_context_put() to release.
132 *
133 * Return:
134 * * The requested context on success, or
135 * * %NULL on failure (context does not exist, or does not belong to @pvr_file).
136 */
137static __always_inline struct pvr_context *
138pvr_context_lookup(struct pvr_file *pvr_file, u32 handle)
139{
140 struct pvr_context *ctx;
141
142 /* Take the array lock to protect against context removal. */
143 xa_lock(&pvr_file->ctx_handles);
144 ctx = pvr_context_get(xa_load(&pvr_file->ctx_handles, handle));
145 xa_unlock(&pvr_file->ctx_handles);
146
147 return ctx;
148}
149
150/**
151 * pvr_context_lookup_id() - Lookup context pointer from ID.
152 * @pvr_dev: Device pointer.
153 * @id: FW context ID.
154 *
155 * Takes reference on context. Call pvr_context_put() to release.
156 *
157 * Return:
158 * * The requested context on success, or
159 * * %NULL on failure (context does not exist).
160 */
161static __always_inline struct pvr_context *
162pvr_context_lookup_id(struct pvr_device *pvr_dev, u32 id)
163{
164 struct pvr_context *ctx;
165
166 /* Take the array lock to protect against context removal. */
167 xa_lock(&pvr_dev->ctx_ids);
168
169 /* Contexts are removed from the ctx_ids set in the context release path,
170 * meaning the ref_count reached zero before they get removed. We need
171 * to make sure we're not trying to acquire a context that's being
172 * destroyed.
173 */
174 ctx = xa_load(&pvr_dev->ctx_ids, id);
175 if (!kref_get_unless_zero(&ctx->ref_count))
176 ctx = NULL;
177
178 xa_unlock(&pvr_dev->ctx_ids);
179
180 return ctx;
181}
182
183static __always_inline u32
184pvr_context_get_fw_addr(struct pvr_context *ctx)
185{
186 u32 ctx_fw_addr = 0;
187
188 pvr_fw_object_get_fw_addr(ctx->fw_obj, &ctx_fw_addr);
189
190 return ctx_fw_addr;
191}
192
193void pvr_context_put(struct pvr_context *ctx);
194
195int pvr_context_create(struct pvr_file *pvr_file, struct drm_pvr_ioctl_create_context_args *args);
196
197int pvr_context_destroy(struct pvr_file *pvr_file, u32 handle);
198
199void pvr_destroy_contexts_for_file(struct pvr_file *pvr_file);
200
201void pvr_context_device_init(struct pvr_device *pvr_dev);
202
203void pvr_context_device_fini(struct pvr_device *pvr_dev);
204
205#endif /* PVR_CONTEXT_H */
drivers/gpu/drm/imagination/pvr_debugfs.c+53−0
@@ -0,0 +1,53 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_debugfs.h"
5
6#include "pvr_device.h"
7#include "pvr_fw_trace.h"
8#include "pvr_params.h"
9
10#include <linux/dcache.h>
11#include <linux/debugfs.h>
12#include <linux/err.h>
13#include <linux/kernel.h>
14#include <linux/types.h>
15
16#include <drm/drm_device.h>
17#include <drm/drm_file.h>
18#include <drm/drm_print.h>
19
20static const struct pvr_debugfs_entry pvr_debugfs_entries[] = {
21 {"pvr_params", pvr_params_debugfs_init},
22 {"pvr_fw", pvr_fw_trace_debugfs_init},
23};
24
25void
26pvr_debugfs_init(struct drm_minor *minor)
27{
28 struct drm_device *drm_dev = minor->dev;
29 struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
30 struct dentry *root = minor->debugfs_root;
31 size_t i;
32
33 for (i = 0; i < ARRAY_SIZE(pvr_debugfs_entries); ++i) {
34 const struct pvr_debugfs_entry *entry = &pvr_debugfs_entries[i];
35 struct dentry *dir;
36
37 dir = debugfs_create_dir(entry->name, root);
38 if (IS_ERR(dir)) {
39 drm_warn(drm_dev,
40 "failed to create debugfs dir '%s' (err=%d)",
41 entry->name, (int)PTR_ERR(dir));
42 continue;
43 }
44
45 entry->init(pvr_dev, dir);
46 }
47}
48
49/*
50 * Since all entries are created under &drm_minor->debugfs_root, there's no
51 * need for a pvr_debugfs_fini() as DRM will clean up everything under its root
52 * automatically.
53 */
drivers/gpu/drm/imagination/pvr_debugfs.h+29−0
@@ -0,0 +1,29 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_DEBUGFS_H
5#define PVR_DEBUGFS_H
6
7/* Forward declaration from <drm/drm_drv.h>. */
8struct drm_minor;
9
10#if defined(CONFIG_DEBUG_FS)
11/* Forward declaration from "pvr_device.h". */
12struct pvr_device;
13
14/* Forward declaration from <linux/dcache.h>. */
15struct dentry;
16
17struct pvr_debugfs_entry {
18 const char *name;
19 void (*init)(struct pvr_device *pvr_dev, struct dentry *dir);
20};
21
22void pvr_debugfs_init(struct drm_minor *minor);
23#else /* defined(CONFIG_DEBUG_FS) */
24#include <linux/compiler_attributes.h>
25
26static __always_inline void pvr_debugfs_init(struct drm_minor *minor) {}
27#endif /* defined(CONFIG_DEBUG_FS) */
28
29#endif /* PVR_DEBUGFS_H */
drivers/gpu/drm/imagination/pvr_device.c+658−0
@@ -0,0 +1,658 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_device.h"
5#include "pvr_device_info.h"
6
7#include "pvr_fw.h"
8#include "pvr_params.h"
9#include "pvr_power.h"
10#include "pvr_queue.h"
11#include "pvr_rogue_cr_defs.h"
12#include "pvr_stream.h"
13#include "pvr_vm.h"
14
15#include <drm/drm_print.h>
16
17#include <linux/bitfield.h>
18#include <linux/clk.h>
19#include <linux/compiler_attributes.h>
20#include <linux/compiler_types.h>
21#include <linux/dma-mapping.h>
22#include <linux/err.h>
23#include <linux/firmware.h>
24#include <linux/gfp.h>
25#include <linux/interrupt.h>
26#include <linux/platform_device.h>
27#include <linux/pm_runtime.h>
28#include <linux/slab.h>
29#include <linux/stddef.h>
30#include <linux/types.h>
31#include <linux/workqueue.h>
32
33/* Major number for the supported version of the firmware. */
34#define PVR_FW_VERSION_MAJOR 1
35
36/**
37 * pvr_device_reg_init() - Initialize kernel access to a PowerVR device's
38 * control registers.
39 * @pvr_dev: Target PowerVR device.
40 *
41 * Sets struct pvr_device->regs.
42 *
43 * This method of mapping the device control registers into memory ensures that
44 * they are unmapped when the driver is detached (i.e. no explicit cleanup is
45 * required).
46 *
47 * Return:
48 * * 0 on success, or
49 * * Any error returned by devm_platform_ioremap_resource().
50 */
51static int
52pvr_device_reg_init(struct pvr_device *pvr_dev)
53{
54 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
55 struct platform_device *plat_dev = to_platform_device(drm_dev->dev);
56 struct resource *regs_resource;
57 void __iomem *regs;
58
59 pvr_dev->regs_resource = NULL;
60 pvr_dev->regs = NULL;
61
62 regs = devm_platform_get_and_ioremap_resource(plat_dev, 0, &regs_resource);
63 if (IS_ERR(regs))
64 return dev_err_probe(drm_dev->dev, PTR_ERR(regs),
65 "failed to ioremap gpu registers\n");
66
67 pvr_dev->regs = regs;
68 pvr_dev->regs_resource = regs_resource;
69
70 return 0;
71}
72
73/**
74 * pvr_device_clk_init() - Initialize clocks required by a PowerVR device
75 * @pvr_dev: Target PowerVR device.
76 *
77 * Sets struct pvr_device->core_clk, struct pvr_device->sys_clk and
78 * struct pvr_device->mem_clk.
79 *
80 * Three clocks are required by the PowerVR device: core, sys and mem. On
81 * return, this function guarantees that the clocks are in one of the following
82 * states:
83 *
84 * * All successfully initialized,
85 * * Core errored, sys and mem uninitialized,
86 * * Core deinitialized, sys errored, mem uninitialized, or
87 * * Core and sys deinitialized, mem errored.
88 *
89 * Return:
90 * * 0 on success,
91 * * Any error returned by devm_clk_get(), or
92 * * Any error returned by devm_clk_get_optional().
93 */
94static int pvr_device_clk_init(struct pvr_device *pvr_dev)
95{
96 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
97 struct clk *core_clk;
98 struct clk *sys_clk;
99 struct clk *mem_clk;
100
101 core_clk = devm_clk_get(drm_dev->dev, "core");
102 if (IS_ERR(core_clk))
103 return dev_err_probe(drm_dev->dev, PTR_ERR(core_clk),
104 "failed to get core clock\n");
105
106 sys_clk = devm_clk_get_optional(drm_dev->dev, "sys");
107 if (IS_ERR(sys_clk))
108 return dev_err_probe(drm_dev->dev, PTR_ERR(core_clk),
109 "failed to get sys clock\n");
110
111 mem_clk = devm_clk_get_optional(drm_dev->dev, "mem");
112 if (IS_ERR(mem_clk))
113 return dev_err_probe(drm_dev->dev, PTR_ERR(core_clk),
114 "failed to get mem clock\n");
115
116 pvr_dev->core_clk = core_clk;
117 pvr_dev->sys_clk = sys_clk;
118 pvr_dev->mem_clk = mem_clk;
119
120 return 0;
121}
122
123/**
124 * pvr_device_process_active_queues() - Process all queue related events.
125 * @pvr_dev: PowerVR device to check
126 *
127 * This is called any time we receive a FW event. It iterates over all
128 * active queues and calls pvr_queue_process() on them.
129 */
130void pvr_device_process_active_queues(struct pvr_device *pvr_dev)
131{
132 struct pvr_queue *queue, *tmp_queue;
133 LIST_HEAD(active_queues);
134
135 mutex_lock(&pvr_dev->queues.lock);
136
137 /* Move all active queues to a temporary list. Queues that remain
138 * active after we're done processing them are re-inserted to
139 * the queues.active list by pvr_queue_process().
140 */
141 list_splice_init(&pvr_dev->queues.active, &active_queues);
142
143 list_for_each_entry_safe(queue, tmp_queue, &active_queues, node)
144 pvr_queue_process(queue);
145
146 mutex_unlock(&pvr_dev->queues.lock);
147}
148
149static irqreturn_t pvr_device_irq_thread_handler(int irq, void *data)
150{
151 struct pvr_device *pvr_dev = data;
152 irqreturn_t ret = IRQ_NONE;
153
154 /* We are in the threaded handler, we can keep dequeuing events until we
155 * don't see any. This should allow us to reduce the number of interrupts
156 * when the GPU is receiving a massive amount of short jobs.
157 */
158 while (pvr_fw_irq_pending(pvr_dev)) {
159 pvr_fw_irq_clear(pvr_dev);
160
161 if (pvr_dev->fw_dev.booted) {
162 pvr_fwccb_process(pvr_dev);
163 pvr_kccb_wake_up_waiters(pvr_dev);
164 pvr_device_process_active_queues(pvr_dev);
165 }
166
167 pm_runtime_mark_last_busy(from_pvr_device(pvr_dev)->dev);
168
169 ret = IRQ_HANDLED;
170 }
171
172 /* Unmask FW irqs before returning, so new interrupts can be received. */
173 pvr_fw_irq_enable(pvr_dev);
174 return ret;
175}
176
177static irqreturn_t pvr_device_irq_handler(int irq, void *data)
178{
179 struct pvr_device *pvr_dev = data;
180
181 if (!pvr_fw_irq_pending(pvr_dev))
182 return IRQ_NONE; /* Spurious IRQ - ignore. */
183
184 /* Mask the FW interrupts before waking up the thread. Will be unmasked
185 * when the thread handler is done processing events.
186 */
187 pvr_fw_irq_disable(pvr_dev);
188 return IRQ_WAKE_THREAD;
189}
190
191/**
192 * pvr_device_irq_init() - Initialise IRQ required by a PowerVR device
193 * @pvr_dev: Target PowerVR device.
194 *
195 * Returns:
196 * * 0 on success,
197 * * Any error returned by platform_get_irq_byname(), or
198 * * Any error returned by request_irq().
199 */
200static int
201pvr_device_irq_init(struct pvr_device *pvr_dev)
202{
203 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
204 struct platform_device *plat_dev = to_platform_device(drm_dev->dev);
205
206 init_waitqueue_head(&pvr_dev->kccb.rtn_q);
207
208 pvr_dev->irq = platform_get_irq(plat_dev, 0);
209 if (pvr_dev->irq < 0)
210 return pvr_dev->irq;
211
212 /* Clear any pending events before requesting the IRQ line. */
213 pvr_fw_irq_clear(pvr_dev);
214 pvr_fw_irq_enable(pvr_dev);
215
216 return request_threaded_irq(pvr_dev->irq, pvr_device_irq_handler,
217 pvr_device_irq_thread_handler,
218 IRQF_SHARED, "gpu", pvr_dev);
219}
220
221/**
222 * pvr_device_irq_fini() - Deinitialise IRQ required by a PowerVR device
223 * @pvr_dev: Target PowerVR device.
224 */
225static void
226pvr_device_irq_fini(struct pvr_device *pvr_dev)
227{
228 free_irq(pvr_dev->irq, pvr_dev);
229}
230
231/**
232 * pvr_build_firmware_filename() - Construct a PowerVR firmware filename
233 * @pvr_dev: Target PowerVR device.
234 * @base: First part of the filename.
235 * @major: Major version number.
236 *
237 * A PowerVR firmware filename consists of three parts separated by underscores
238 * (``'_'``) along with a '.fw' file suffix. The first part is the exact value
239 * of @base, the second part is the hardware version string derived from @pvr_fw
240 * and the final part is the firmware version number constructed from @major with
241 * a 'v' prefix, e.g. powervr/rogue_4.40.2.51_v1.fw.
242 *
243 * The returned string will have been slab allocated and must be freed with
244 * kfree().
245 *
246 * Return:
247 * * The constructed filename on success, or
248 * * Any error returned by kasprintf().
249 */
250static char *
251pvr_build_firmware_filename(struct pvr_device *pvr_dev, const char *base,
252 u8 major)
253{
254 struct pvr_gpu_id *gpu_id = &pvr_dev->gpu_id;
255
256 return kasprintf(GFP_KERNEL, "%s_%d.%d.%d.%d_v%d.fw", base, gpu_id->b,
257 gpu_id->v, gpu_id->n, gpu_id->c, major);
258}
259
260static void
261pvr_release_firmware(void *data)
262{
263 struct pvr_device *pvr_dev = data;
264
265 release_firmware(pvr_dev->fw_dev.firmware);
266}
267
268/**
269 * pvr_request_firmware() - Load firmware for a PowerVR device
270 * @pvr_dev: Target PowerVR device.
271 *
272 * See pvr_build_firmware_filename() for details on firmware file naming.
273 *
274 * Return:
275 * * 0 on success,
276 * * Any error returned by pvr_build_firmware_filename(), or
277 * * Any error returned by request_firmware().
278 */
279static int
280pvr_request_firmware(struct pvr_device *pvr_dev)
281{
282 struct drm_device *drm_dev = &pvr_dev->base;
283 char *filename;
284 const struct firmware *fw;
285 int err;
286
287 filename = pvr_build_firmware_filename(pvr_dev, "powervr/rogue",
288 PVR_FW_VERSION_MAJOR);
289 if (IS_ERR(filename))
290 return PTR_ERR(filename);
291
292 /*
293 * This function takes a copy of &filename, meaning we can free our
294 * instance before returning.
295 */
296 err = request_firmware(&fw, filename, pvr_dev->base.dev);
297 if (err) {
298 drm_err(drm_dev, "failed to load firmware %s (err=%d)\n",
299 filename, err);
300 goto err_free_filename;
301 }
302
303 drm_info(drm_dev, "loaded firmware %s\n", filename);
304 kfree(filename);
305
306 pvr_dev->fw_dev.firmware = fw;
307
308 return devm_add_action_or_reset(drm_dev->dev, pvr_release_firmware, pvr_dev);
309
310err_free_filename:
311 kfree(filename);
312
313 return err;
314}
315
316/**
317 * pvr_load_gpu_id() - Load a PowerVR device's GPU ID (BVNC) from control registers.
318 *
319 * Sets struct pvr_dev.gpu_id.
320 *
321 * @pvr_dev: Target PowerVR device.
322 */
323static void
324pvr_load_gpu_id(struct pvr_device *pvr_dev)
325{
326 struct pvr_gpu_id *gpu_id = &pvr_dev->gpu_id;
327 u64 bvnc;
328
329 /*
330 * Try reading the BVNC using the newer (cleaner) method first. If the
331 * B value is zero, fall back to the older method.
332 */
333 bvnc = pvr_cr_read64(pvr_dev, ROGUE_CR_CORE_ID__PBVNC);
334
335 gpu_id->b = PVR_CR_FIELD_GET(bvnc, CORE_ID__PBVNC__BRANCH_ID);
336 if (gpu_id->b != 0) {
337 gpu_id->v = PVR_CR_FIELD_GET(bvnc, CORE_ID__PBVNC__VERSION_ID);
338 gpu_id->n = PVR_CR_FIELD_GET(bvnc, CORE_ID__PBVNC__NUMBER_OF_SCALABLE_UNITS);
339 gpu_id->c = PVR_CR_FIELD_GET(bvnc, CORE_ID__PBVNC__CONFIG_ID);
340 } else {
341 u32 core_rev = pvr_cr_read32(pvr_dev, ROGUE_CR_CORE_REVISION);
342 u32 core_id = pvr_cr_read32(pvr_dev, ROGUE_CR_CORE_ID);
343 u16 core_id_config = PVR_CR_FIELD_GET(core_id, CORE_ID_CONFIG);
344
345 gpu_id->b = PVR_CR_FIELD_GET(core_rev, CORE_REVISION_MAJOR);
346 gpu_id->v = PVR_CR_FIELD_GET(core_rev, CORE_REVISION_MINOR);
347 gpu_id->n = FIELD_GET(0xFF00, core_id_config);
348 gpu_id->c = FIELD_GET(0x00FF, core_id_config);
349 }
350}
351
352/**
353 * pvr_set_dma_info() - Set PowerVR device DMA information
354 * @pvr_dev: Target PowerVR device.
355 *
356 * Sets the DMA mask and max segment size for the PowerVR device.
357 *
358 * Return:
359 * * 0 on success,
360 * * Any error returned by PVR_FEATURE_VALUE(), or
361 * * Any error returned by dma_set_mask().
362 */
363
364static int
365pvr_set_dma_info(struct pvr_device *pvr_dev)
366{
367 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
368 u16 phys_bus_width;
369 int err;
370
371 err = PVR_FEATURE_VALUE(pvr_dev, phys_bus_width, &phys_bus_width);
372 if (err) {
373 drm_err(drm_dev, "Failed to get device physical bus width\n");
374 return err;
375 }
376
377 err = dma_set_mask(drm_dev->dev, DMA_BIT_MASK(phys_bus_width));
378 if (err) {
379 drm_err(drm_dev, "Failed to set DMA mask (err=%d)\n", err);
380 return err;
381 }
382
383 dma_set_max_seg_size(drm_dev->dev, UINT_MAX);
384
385 return 0;
386}
387
388/**
389 * pvr_device_gpu_init() - GPU-specific initialization for a PowerVR device
390 * @pvr_dev: Target PowerVR device.
391 *
392 * The following steps are taken to ensure the device is ready:
393 *
394 * 1. Read the hardware version information from control registers,
395 * 2. Initialise the hardware feature information,
396 * 3. Setup the device DMA information,
397 * 4. Setup the device-scoped memory context, and
398 * 5. Load firmware into the device.
399 *
400 * Return:
401 * * 0 on success,
402 * * -%ENODEV if the GPU is not supported,
403 * * Any error returned by pvr_set_dma_info(),
404 * * Any error returned by pvr_memory_context_init(), or
405 * * Any error returned by pvr_request_firmware().
406 */
407static int
408pvr_device_gpu_init(struct pvr_device *pvr_dev)
409{
410 int err;
411
412 pvr_load_gpu_id(pvr_dev);
413
414 err = pvr_request_firmware(pvr_dev);
415 if (err)
416 return err;
417
418 err = pvr_fw_validate_init_device_info(pvr_dev);
419 if (err)
420 return err;
421
422 if (PVR_HAS_FEATURE(pvr_dev, meta))
423 pvr_dev->fw_dev.processor_type = PVR_FW_PROCESSOR_TYPE_META;
424 else if (PVR_HAS_FEATURE(pvr_dev, mips))
425 pvr_dev->fw_dev.processor_type = PVR_FW_PROCESSOR_TYPE_MIPS;
426 else if (PVR_HAS_FEATURE(pvr_dev, riscv_fw_processor))
427 pvr_dev->fw_dev.processor_type = PVR_FW_PROCESSOR_TYPE_RISCV;
428 else
429 return -EINVAL;
430
431 pvr_stream_create_musthave_masks(pvr_dev);
432
433 err = pvr_set_dma_info(pvr_dev);
434 if (err)
435 return err;
436
437 if (pvr_dev->fw_dev.processor_type != PVR_FW_PROCESSOR_TYPE_MIPS) {
438 pvr_dev->kernel_vm_ctx = pvr_vm_create_context(pvr_dev, false);
439 if (IS_ERR(pvr_dev->kernel_vm_ctx))
440 return PTR_ERR(pvr_dev->kernel_vm_ctx);
441 }
442
443 err = pvr_fw_init(pvr_dev);
444 if (err)
445 goto err_vm_ctx_put;
446
447 return 0;
448
449err_vm_ctx_put:
450 if (pvr_dev->fw_dev.processor_type != PVR_FW_PROCESSOR_TYPE_MIPS) {
451 pvr_vm_context_put(pvr_dev->kernel_vm_ctx);
452 pvr_dev->kernel_vm_ctx = NULL;
453 }
454
455 return err;
456}
457
458/**
459 * pvr_device_gpu_fini() - GPU-specific deinitialization for a PowerVR device
460 * @pvr_dev: Target PowerVR device.
461 */
462static void
463pvr_device_gpu_fini(struct pvr_device *pvr_dev)
464{
465 pvr_fw_fini(pvr_dev);
466
467 if (pvr_dev->fw_dev.processor_type != PVR_FW_PROCESSOR_TYPE_MIPS) {
468 WARN_ON(!pvr_vm_context_put(pvr_dev->kernel_vm_ctx));
469 pvr_dev->kernel_vm_ctx = NULL;
470 }
471}
472
473/**
474 * pvr_device_init() - Initialize a PowerVR device
475 * @pvr_dev: Target PowerVR device.
476 *
477 * If this function returns successfully, the device will have been fully
478 * initialized. Otherwise, any parts of the device initialized before an error
479 * occurs will be de-initialized before returning.
480 *
481 * NOTE: The initialization steps currently taken are the bare minimum required
482 * to read from the control registers. The device is unlikely to function
483 * until further initialization steps are added. [This note should be
484 * removed when that happens.]
485 *
486 * Return:
487 * * 0 on success,
488 * * Any error returned by pvr_device_reg_init(),
489 * * Any error returned by pvr_device_clk_init(), or
490 * * Any error returned by pvr_device_gpu_init().
491 */
492int
493pvr_device_init(struct pvr_device *pvr_dev)
494{
495 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
496 struct device *dev = drm_dev->dev;
497 int err;
498
499 /*
500 * Setup device parameters. We do this first in case other steps
501 * depend on them.
502 */
503 err = pvr_device_params_init(&pvr_dev->params);
504 if (err)
505 return err;
506
507 /* Enable and initialize clocks required for the device to operate. */
508 err = pvr_device_clk_init(pvr_dev);
509 if (err)
510 return err;
511
512 /* Explicitly power the GPU so we can access control registers before the FW is booted. */
513 err = pm_runtime_resume_and_get(dev);
514 if (err)
515 return err;
516
517 /* Map the control registers into memory. */
518 err = pvr_device_reg_init(pvr_dev);
519 if (err)
520 goto err_pm_runtime_put;
521
522 /* Perform GPU-specific initialization steps. */
523 err = pvr_device_gpu_init(pvr_dev);
524 if (err)
525 goto err_pm_runtime_put;
526
527 err = pvr_device_irq_init(pvr_dev);
528 if (err)
529 goto err_device_gpu_fini;
530
531 pm_runtime_put(dev);
532
533 return 0;
534
535err_device_gpu_fini:
536 pvr_device_gpu_fini(pvr_dev);
537
538err_pm_runtime_put:
539 pm_runtime_put_sync_suspend(dev);
540
541 return err;
542}
543
544/**
545 * pvr_device_fini() - Deinitialize a PowerVR device
546 * @pvr_dev: Target PowerVR device.
547 */
548void
549pvr_device_fini(struct pvr_device *pvr_dev)
550{
551 /*
552 * Deinitialization stages are performed in reverse order compared to
553 * the initialization stages in pvr_device_init().
554 */
555 pvr_device_irq_fini(pvr_dev);
556 pvr_device_gpu_fini(pvr_dev);
557}
558
559bool
560pvr_device_has_uapi_quirk(struct pvr_device *pvr_dev, u32 quirk)
561{
562 switch (quirk) {
563 case 47217:
564 return PVR_HAS_QUIRK(pvr_dev, 47217);
565 case 48545:
566 return PVR_HAS_QUIRK(pvr_dev, 48545);
567 case 49927:
568 return PVR_HAS_QUIRK(pvr_dev, 49927);
569 case 51764:
570 return PVR_HAS_QUIRK(pvr_dev, 51764);
571 case 62269:
572 return PVR_HAS_QUIRK(pvr_dev, 62269);
573 default:
574 return false;
575 };
576}
577
578bool
579pvr_device_has_uapi_enhancement(struct pvr_device *pvr_dev, u32 enhancement)
580{
581 switch (enhancement) {
582 case 35421:
583 return PVR_HAS_ENHANCEMENT(pvr_dev, 35421);
584 case 42064:
585 return PVR_HAS_ENHANCEMENT(pvr_dev, 42064);
586 default:
587 return false;
588 };
589}
590
591/**
592 * pvr_device_has_feature() - Look up device feature based on feature definition
593 * @pvr_dev: Device pointer.
594 * @feature: Feature to look up. Should be one of %PVR_FEATURE_*.
595 *
596 * Returns:
597 * * %true if feature is present on device, or
598 * * %false if feature is not present on device.
599 */
600bool
601pvr_device_has_feature(struct pvr_device *pvr_dev, u32 feature)
602{
603 switch (feature) {
604 case PVR_FEATURE_CLUSTER_GROUPING:
605 return PVR_HAS_FEATURE(pvr_dev, cluster_grouping);
606
607 case PVR_FEATURE_COMPUTE_MORTON_CAPABLE:
608 return PVR_HAS_FEATURE(pvr_dev, compute_morton_capable);
609
610 case PVR_FEATURE_FB_CDC_V4:
611 return PVR_HAS_FEATURE(pvr_dev, fb_cdc_v4);
612
613 case PVR_FEATURE_GPU_MULTICORE_SUPPORT:
614 return PVR_HAS_FEATURE(pvr_dev, gpu_multicore_support);
615
616 case PVR_FEATURE_ISP_ZLS_D24_S8_PACKING_OGL_MODE:
617 return PVR_HAS_FEATURE(pvr_dev, isp_zls_d24_s8_packing_ogl_mode);
618
619 case PVR_FEATURE_S7_TOP_INFRASTRUCTURE:
620 return PVR_HAS_FEATURE(pvr_dev, s7_top_infrastructure);
621
622 case PVR_FEATURE_TESSELLATION:
623 return PVR_HAS_FEATURE(pvr_dev, tessellation);
624
625 case PVR_FEATURE_TPU_DM_GLOBAL_REGISTERS:
626 return PVR_HAS_FEATURE(pvr_dev, tpu_dm_global_registers);
627
628 case PVR_FEATURE_VDM_DRAWINDIRECT:
629 return PVR_HAS_FEATURE(pvr_dev, vdm_drawindirect);
630
631 case PVR_FEATURE_VDM_OBJECT_LEVEL_LLS:
632 return PVR_HAS_FEATURE(pvr_dev, vdm_object_level_lls);
633
634 case PVR_FEATURE_ZLS_SUBTILE:
635 return PVR_HAS_FEATURE(pvr_dev, zls_subtile);
636
637 /* Derived features. */
638 case PVR_FEATURE_CDM_USER_MODE_QUEUE: {
639 u8 cdm_control_stream_format = 0;
640
641 PVR_FEATURE_VALUE(pvr_dev, cdm_control_stream_format, &cdm_control_stream_format);
642 return (cdm_control_stream_format >= 2 && cdm_control_stream_format <= 4);
643 }
644
645 case PVR_FEATURE_REQUIRES_FB_CDC_ZLS_SETUP:
646 if (PVR_HAS_FEATURE(pvr_dev, fbcdc_algorithm)) {
647 u8 fbcdc_algorithm = 0;
648
649 PVR_FEATURE_VALUE(pvr_dev, fbcdc_algorithm, &fbcdc_algorithm);
650 return (fbcdc_algorithm < 3 || PVR_HAS_FEATURE(pvr_dev, fb_cdc_v4));
651 }
652 return false;
653
654 default:
655 WARN(true, "Looking up undefined feature %u\n", feature);
656 return false;
657 }
658}
drivers/gpu/drm/imagination/pvr_device.h+710−0
@@ -0,0 +1,710 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_DEVICE_H
5#define PVR_DEVICE_H
6
7#include "pvr_ccb.h"
8#include "pvr_device_info.h"
9#include "pvr_fw.h"
10#include "pvr_params.h"
11#include "pvr_rogue_fwif_stream.h"
12#include "pvr_stream.h"
13
14#include <drm/drm_device.h>
15#include <drm/drm_file.h>
16#include <drm/drm_mm.h>
17
18#include <linux/bits.h>
19#include <linux/compiler_attributes.h>
20#include <linux/compiler_types.h>
21#include <linux/io.h>
22#include <linux/iopoll.h>
23#include <linux/kernel.h>
24#include <linux/math.h>
25#include <linux/mutex.h>
26#include <linux/timer.h>
27#include <linux/types.h>
28#include <linux/wait.h>
29#include <linux/workqueue.h>
30#include <linux/xarray.h>
31
32/* Forward declaration from <linux/clk.h>. */
33struct clk;
34
35/* Forward declaration from <linux/firmware.h>. */
36struct firmware;
37
38/**
39 * struct pvr_gpu_id - Hardware GPU ID information for a PowerVR device
40 * @b: Branch ID.
41 * @v: Version ID.
42 * @n: Number of scalable units.
43 * @c: Config ID.
44 */
45struct pvr_gpu_id {
46 u16 b, v, n, c;
47};
48
49/**
50 * struct pvr_fw_version - Firmware version information
51 * @major: Major version number.
52 * @minor: Minor version number.
53 */
54struct pvr_fw_version {
55 u16 major, minor;
56};
57
58/**
59 * struct pvr_device - powervr-specific wrapper for &struct drm_device
60 */
61struct pvr_device {
62 /**
63 * @base: The underlying &struct drm_device.
64 *
65 * Do not access this member directly, instead call
66 * from_pvr_device().
67 */
68 struct drm_device base;
69
70 /** @gpu_id: GPU ID detected at runtime. */
71 struct pvr_gpu_id gpu_id;
72
73 /**
74 * @features: Hardware feature information.
75 *
76 * Do not access this member directly, instead use PVR_HAS_FEATURE()
77 * or PVR_FEATURE_VALUE() macros.
78 */
79 struct pvr_device_features features;
80
81 /**
82 * @quirks: Hardware quirk information.
83 *
84 * Do not access this member directly, instead use PVR_HAS_QUIRK().
85 */
86 struct pvr_device_quirks quirks;
87
88 /**
89 * @enhancements: Hardware enhancement information.
90 *
91 * Do not access this member directly, instead use
92 * PVR_HAS_ENHANCEMENT().
93 */
94 struct pvr_device_enhancements enhancements;
95
96 /** @fw_version: Firmware version detected at runtime. */
97 struct pvr_fw_version fw_version;
98
99 /** @regs_resource: Resource representing device control registers. */
100 struct resource *regs_resource;
101
102 /**
103 * @regs: Device control registers.
104 *
105 * These are mapped into memory when the device is initialized; that
106 * location is where this pointer points.
107 */
108 void __iomem *regs;
109
110 /**
111 * @core_clk: General core clock.
112 *
113 * This is the primary clock used by the entire GPU core.
114 */
115 struct clk *core_clk;
116
117 /**
118 * @sys_clk: Optional system bus clock.
119 *
120 * This may be used on some platforms to provide an independent clock to the SoC Interface
121 * (SOCIF). If present, this needs to be enabled/disabled together with @core_clk.
122 */
123 struct clk *sys_clk;
124
125 /**
126 * @mem_clk: Optional memory clock.
127 *
128 * This may be used on some platforms to provide an independent clock to the Memory
129 * Interface (MEMIF). If present, this needs to be enabled/disabled together with @core_clk.
130 */
131 struct clk *mem_clk;
132
133 /** @irq: IRQ number. */
134 int irq;
135
136 /** @fwccb: Firmware CCB. */
137 struct pvr_ccb fwccb;
138
139 /**
140 * @kernel_vm_ctx: Virtual memory context used for kernel mappings.
141 *
142 * This is used for mappings in the firmware address region when a META firmware processor
143 * is in use.
144 *
145 * When a MIPS firmware processor is in use, this will be %NULL.
146 */
147 struct pvr_vm_context *kernel_vm_ctx;
148
149 /** @fw_dev: Firmware related data. */
150 struct pvr_fw_device fw_dev;
151
152 /**
153 * @params: Device-specific parameters.
154 *
155 * The values of these parameters are initialized from the
156 * defaults specified as module parameters. They may be
157 * modified at runtime via debugfs (if enabled).
158 */
159 struct pvr_device_params params;
160
161 /** @stream_musthave_quirks: Bit array of "must-have" quirks for stream commands. */
162 u32 stream_musthave_quirks[PVR_STREAM_TYPE_MAX][PVR_STREAM_EXTHDR_TYPE_MAX];
163
164 /**
165 * @mmu_flush_cache_flags: Records which MMU caches require flushing
166 * before submitting the next job.
167 */
168 atomic_t mmu_flush_cache_flags;
169
170 /**
171 * @ctx_ids: Array of contexts belonging to this device. Array members
172 * are of type "struct pvr_context *".
173 *
174 * This array is used to allocate IDs used by the firmware.
175 */
176 struct xarray ctx_ids;
177
178 /**
179 * @free_list_ids: Array of free lists belonging to this device. Array members
180 * are of type "struct pvr_free_list *".
181 *
182 * This array is used to allocate IDs used by the firmware.
183 */
184 struct xarray free_list_ids;
185
186 /**
187 * @job_ids: Array of jobs belonging to this device. Array members
188 * are of type "struct pvr_job *".
189 */
190 struct xarray job_ids;
191
192 /**
193 * @queues: Queue-related fields.
194 */
195 struct {
196 /** @active: Active queue list. */
197 struct list_head active;
198
199 /** @idle: Idle queue list. */
200 struct list_head idle;
201
202 /** @lock: Lock protecting access to the active/idle lists. */
203 struct mutex lock;
204 } queues;
205
206 struct {
207 /** @work: Work item for watchdog callback. */
208 struct delayed_work work;
209
210 /** @old_kccb_cmds_executed: KCCB command execution count at last watchdog poll. */
211 u32 old_kccb_cmds_executed;
212
213 /** @kccb_stall_count: Number of watchdog polls KCCB has been stalled for. */
214 u32 kccb_stall_count;
215 } watchdog;
216
217 struct {
218 /** @ccb: Kernel CCB. */
219 struct pvr_ccb ccb;
220
221 /** @rtn_q: Waitqueue for KCCB command return waiters. */
222 wait_queue_head_t rtn_q;
223
224 /** @rtn_obj: Object representing KCCB return slots. */
225 struct pvr_fw_object *rtn_obj;
226
227 /**
228 * @rtn: Pointer to CPU mapping of KCCB return slots. Must be accessed by
229 * READ_ONCE()/WRITE_ONCE().
230 */
231 u32 *rtn;
232
233 /** @slot_count: Total number of KCCB slots available. */
234 u32 slot_count;
235
236 /** @reserved_count: Number of KCCB slots reserved for future use. */
237 u32 reserved_count;
238
239 /**
240 * @waiters: List of KCCB slot waiters.
241 */
242 struct list_head waiters;
243
244 /** @fence_ctx: KCCB fence context. */
245 struct {
246 /** @id: KCCB fence context ID allocated with dma_fence_context_alloc(). */
247 u64 id;
248
249 /** @seqno: Sequence number incremented each time a fence is created. */
250 atomic_t seqno;
251
252 /**
253 * @lock: Lock used to synchronize access to fences allocated by this
254 * context.
255 */
256 spinlock_t lock;
257 } fence_ctx;
258 } kccb;
259
260 /**
261 * @lost: %true if the device has been lost.
262 *
263 * This variable is set if the device has become irretrievably unavailable, e.g. if the
264 * firmware processor has stopped responding and can not be revived via a hard reset.
265 */
266 bool lost;
267
268 /**
269 * @reset_sem: Reset semaphore.
270 *
271 * GPU reset code will lock this for writing. Any code that submits commands to the firmware
272 * that isn't in an IRQ handler or on the scheduler workqueue must lock this for reading.
273 * Once this has been successfully locked, &pvr_dev->lost _must_ be checked, and -%EIO must
274 * be returned if it is set.
275 */
276 struct rw_semaphore reset_sem;
277
278 /** @sched_wq: Workqueue for schedulers. */
279 struct workqueue_struct *sched_wq;
280};
281
282/**
283 * struct pvr_file - powervr-specific data to be assigned to &struct
284 * drm_file.driver_priv
285 */
286struct pvr_file {
287 /**
288 * @file: A reference to the parent &struct drm_file.
289 *
290 * Do not access this member directly, instead call from_pvr_file().
291 */
292 struct drm_file *file;
293
294 /**
295 * @pvr_dev: A reference to the powervr-specific wrapper for the
296 * associated device. Saves on repeated calls to
297 * to_pvr_device().
298 */
299 struct pvr_device *pvr_dev;
300
301 /**
302 * @ctx_handles: Array of contexts belonging to this file. Array members
303 * are of type "struct pvr_context *".
304 *
305 * This array is used to allocate handles returned to userspace.
306 */
307 struct xarray ctx_handles;
308
309 /**
310 * @free_list_handles: Array of free lists belonging to this file. Array
311 * members are of type "struct pvr_free_list *".
312 *
313 * This array is used to allocate handles returned to userspace.
314 */
315 struct xarray free_list_handles;
316
317 /**
318 * @hwrt_handles: Array of HWRT datasets belonging to this file. Array
319 * members are of type "struct pvr_hwrt_dataset *".
320 *
321 * This array is used to allocate handles returned to userspace.
322 */
323 struct xarray hwrt_handles;
324
325 /**
326 * @vm_ctx_handles: Array of VM contexts belonging to this file. Array
327 * members are of type "struct pvr_vm_context *".
328 *
329 * This array is used to allocate handles returned to userspace.
330 */
331 struct xarray vm_ctx_handles;
332};
333
334/**
335 * PVR_HAS_FEATURE() - Tests whether a PowerVR device has a given feature
336 * @pvr_dev: [IN] Target PowerVR device.
337 * @feature: [IN] Hardware feature name.
338 *
339 * Feature names are derived from those found in &struct pvr_device_features by
340 * dropping the 'has_' prefix, which is applied by this macro.
341 *
342 * Return:
343 * * true if the named feature is present in the hardware
344 * * false if the named feature is not present in the hardware
345 */
346#define PVR_HAS_FEATURE(pvr_dev, feature) ((pvr_dev)->features.has_##feature)
347
348/**
349 * PVR_FEATURE_VALUE() - Gets a PowerVR device feature value
350 * @pvr_dev: [IN] Target PowerVR device.
351 * @feature: [IN] Feature name.
352 * @value_out: [OUT] Feature value.
353 *
354 * This macro will get a feature value for those features that have values.
355 * If the feature is not present, nothing will be stored to @value_out.
356 *
357 * Feature names are derived from those found in &struct pvr_device_features by
358 * dropping the 'has_' prefix.
359 *
360 * Return:
361 * * 0 on success, or
362 * * -%EINVAL if the named feature is not present in the hardware
363 */
364#define PVR_FEATURE_VALUE(pvr_dev, feature, value_out) \
365 ({ \
366 struct pvr_device *_pvr_dev = pvr_dev; \
367 int _ret = -EINVAL; \
368 if (_pvr_dev->features.has_##feature) { \
369 *(value_out) = _pvr_dev->features.feature; \
370 _ret = 0; \
371 } \
372 _ret; \
373 })
374
375/**
376 * PVR_HAS_QUIRK() - Tests whether a physical device has a given quirk
377 * @pvr_dev: [IN] Target PowerVR device.
378 * @quirk: [IN] Hardware quirk name.
379 *
380 * Quirk numbers are derived from those found in #pvr_device_quirks by
381 * dropping the 'has_brn' prefix, which is applied by this macro.
382 *
383 * Returns
384 * * true if the quirk is present in the hardware, or
385 * * false if the quirk is not present in the hardware.
386 */
387#define PVR_HAS_QUIRK(pvr_dev, quirk) ((pvr_dev)->quirks.has_brn##quirk)
388
389/**
390 * PVR_HAS_ENHANCEMENT() - Tests whether a physical device has a given
391 * enhancement
392 * @pvr_dev: [IN] Target PowerVR device.
393 * @enhancement: [IN] Hardware enhancement name.
394 *
395 * Enhancement numbers are derived from those found in #pvr_device_enhancements
396 * by dropping the 'has_ern' prefix, which is applied by this macro.
397 *
398 * Returns
399 * * true if the enhancement is present in the hardware, or
400 * * false if the enhancement is not present in the hardware.
401 */
402#define PVR_HAS_ENHANCEMENT(pvr_dev, enhancement) ((pvr_dev)->enhancements.has_ern##enhancement)
403
404#define from_pvr_device(pvr_dev) (&(pvr_dev)->base)
405
406#define to_pvr_device(drm_dev) container_of_const(drm_dev, struct pvr_device, base)
407
408#define from_pvr_file(pvr_file) ((pvr_file)->file)
409
410#define to_pvr_file(file) ((file)->driver_priv)
411
412/**
413 * PVR_PACKED_BVNC() - Packs B, V, N and C values into a 64-bit unsigned integer
414 * @b: Branch ID.
415 * @v: Version ID.
416 * @n: Number of scalable units.
417 * @c: Config ID.
418 *
419 * The packed layout is as follows:
420 *
421 * +--------+--------+--------+-------+
422 * | 63..48 | 47..32 | 31..16 | 15..0 |
423 * +========+========+========+=======+
424 * | B | V | N | C |
425 * +--------+--------+--------+-------+
426 *
427 * pvr_gpu_id_to_packed_bvnc() should be used instead of this macro when a
428 * &struct pvr_gpu_id is available in order to ensure proper type checking.
429 *
430 * Return: Packed BVNC.
431 */
432/* clang-format off */
433#define PVR_PACKED_BVNC(b, v, n, c) \
434 ((((u64)(b) & GENMASK_ULL(15, 0)) << 48) | \
435 (((u64)(v) & GENMASK_ULL(15, 0)) << 32) | \
436 (((u64)(n) & GENMASK_ULL(15, 0)) << 16) | \
437 (((u64)(c) & GENMASK_ULL(15, 0)) << 0))
438/* clang-format on */
439
440/**
441 * pvr_gpu_id_to_packed_bvnc() - Packs B, V, N and C values into a 64-bit
442 * unsigned integer
443 * @gpu_id: GPU ID.
444 *
445 * The packed layout is as follows:
446 *
447 * +--------+--------+--------+-------+
448 * | 63..48 | 47..32 | 31..16 | 15..0 |
449 * +========+========+========+=======+
450 * | B | V | N | C |
451 * +--------+--------+--------+-------+
452 *
453 * This should be used in preference to PVR_PACKED_BVNC() when a &struct
454 * pvr_gpu_id is available in order to ensure proper type checking.
455 *
456 * Return: Packed BVNC.
457 */
458static __always_inline u64
459pvr_gpu_id_to_packed_bvnc(struct pvr_gpu_id *gpu_id)
460{
461 return PVR_PACKED_BVNC(gpu_id->b, gpu_id->v, gpu_id->n, gpu_id->c);
462}
463
464static __always_inline void
465packed_bvnc_to_pvr_gpu_id(u64 bvnc, struct pvr_gpu_id *gpu_id)
466{
467 gpu_id->b = (bvnc & GENMASK_ULL(63, 48)) >> 48;
468 gpu_id->v = (bvnc & GENMASK_ULL(47, 32)) >> 32;
469 gpu_id->n = (bvnc & GENMASK_ULL(31, 16)) >> 16;
470 gpu_id->c = bvnc & GENMASK_ULL(15, 0);
471}
472
473int pvr_device_init(struct pvr_device *pvr_dev);
474void pvr_device_fini(struct pvr_device *pvr_dev);
475void pvr_device_reset(struct pvr_device *pvr_dev);
476
477bool
478pvr_device_has_uapi_quirk(struct pvr_device *pvr_dev, u32 quirk);
479bool
480pvr_device_has_uapi_enhancement(struct pvr_device *pvr_dev, u32 enhancement);
481bool
482pvr_device_has_feature(struct pvr_device *pvr_dev, u32 feature);
483
484/**
485 * PVR_CR_FIELD_GET() - Extract a single field from a PowerVR control register
486 * @val: Value of the target register.
487 * @field: Field specifier, as defined in "pvr_rogue_cr_defs.h".
488 *
489 * Return: The extracted field.
490 */
491#define PVR_CR_FIELD_GET(val, field) FIELD_GET(~ROGUE_CR_##field##_CLRMSK, val)
492
493/**
494 * pvr_cr_read32() - Read a 32-bit register from a PowerVR device
495 * @pvr_dev: Target PowerVR device.
496 * @reg: Target register.
497 *
498 * Return: The value of the requested register.
499 */
500static __always_inline u32
501pvr_cr_read32(struct pvr_device *pvr_dev, u32 reg)
502{
503 return ioread32(pvr_dev->regs + reg);
504}
505
506/**
507 * pvr_cr_read64() - Read a 64-bit register from a PowerVR device
508 * @pvr_dev: Target PowerVR device.
509 * @reg: Target register.
510 *
511 * Return: The value of the requested register.
512 */
513static __always_inline u64
514pvr_cr_read64(struct pvr_device *pvr_dev, u32 reg)
515{
516 return ioread64(pvr_dev->regs + reg);
517}
518
519/**
520 * pvr_cr_write32() - Write to a 32-bit register in a PowerVR device
521 * @pvr_dev: Target PowerVR device.
522 * @reg: Target register.
523 * @val: Value to write.
524 */
525static __always_inline void
526pvr_cr_write32(struct pvr_device *pvr_dev, u32 reg, u32 val)
527{
528 iowrite32(val, pvr_dev->regs + reg);
529}
530
531/**
532 * pvr_cr_write64() - Write to a 64-bit register in a PowerVR device
533 * @pvr_dev: Target PowerVR device.
534 * @reg: Target register.
535 * @val: Value to write.
536 */
537static __always_inline void
538pvr_cr_write64(struct pvr_device *pvr_dev, u32 reg, u64 val)
539{
540 iowrite64(val, pvr_dev->regs + reg);
541}
542
543/**
544 * pvr_cr_poll_reg32() - Wait for a 32-bit register to match a given value by
545 * polling
546 * @pvr_dev: Target PowerVR device.
547 * @reg_addr: Address of register.
548 * @reg_value: Expected register value (after masking).
549 * @reg_mask: Mask of bits valid for comparison with @reg_value.
550 * @timeout_usec: Timeout length, in us.
551 *
552 * Returns:
553 * * 0 on success, or
554 * * -%ETIMEDOUT on timeout.
555 */
556static __always_inline int
557pvr_cr_poll_reg32(struct pvr_device *pvr_dev, u32 reg_addr, u32 reg_value,
558 u32 reg_mask, u64 timeout_usec)
559{
560 u32 value;
561
562 return readl_poll_timeout(pvr_dev->regs + reg_addr, value,
563 (value & reg_mask) == reg_value, 0, timeout_usec);
564}
565
566/**
567 * pvr_cr_poll_reg64() - Wait for a 64-bit register to match a given value by
568 * polling
569 * @pvr_dev: Target PowerVR device.
570 * @reg_addr: Address of register.
571 * @reg_value: Expected register value (after masking).
572 * @reg_mask: Mask of bits valid for comparison with @reg_value.
573 * @timeout_usec: Timeout length, in us.
574 *
575 * Returns:
576 * * 0 on success, or
577 * * -%ETIMEDOUT on timeout.
578 */
579static __always_inline int
580pvr_cr_poll_reg64(struct pvr_device *pvr_dev, u32 reg_addr, u64 reg_value,
581 u64 reg_mask, u64 timeout_usec)
582{
583 u64 value;
584
585 return readq_poll_timeout(pvr_dev->regs + reg_addr, value,
586 (value & reg_mask) == reg_value, 0, timeout_usec);
587}
588
589/**
590 * pvr_round_up_to_cacheline_size() - Round up a provided size to be cacheline
591 * aligned
592 * @pvr_dev: Target PowerVR device.
593 * @size: Initial size, in bytes.
594 *
595 * Returns:
596 * * Size aligned to cacheline size.
597 */
598static __always_inline size_t
599pvr_round_up_to_cacheline_size(struct pvr_device *pvr_dev, size_t size)
600{
601 u16 slc_cacheline_size_bits = 0;
602 u16 slc_cacheline_size_bytes;
603
604 WARN_ON(!PVR_HAS_FEATURE(pvr_dev, slc_cache_line_size_bits));
605 PVR_FEATURE_VALUE(pvr_dev, slc_cache_line_size_bits,
606 &slc_cacheline_size_bits);
607 slc_cacheline_size_bytes = slc_cacheline_size_bits / 8;
608
609 return round_up(size, slc_cacheline_size_bytes);
610}
611
612/**
613 * DOC: IOCTL validation helpers
614 *
615 * To validate the constraints imposed on IOCTL argument structs, a collection
616 * of macros and helper functions exist in ``pvr_device.h``.
617 *
618 * Of the current helpers, it should only be necessary to call
619 * PVR_IOCTL_UNION_PADDING_CHECK() directly. This macro should be used once in
620 * every code path which extracts a union member from a struct passed from
621 * userspace.
622 */
623
624/**
625 * pvr_ioctl_union_padding_check() - Validate that the implicit padding between
626 * the end of a union member and the end of the union itself is zeroed.
627 * @instance: Pointer to the instance of the struct to validate.
628 * @union_offset: Offset into the type of @instance of the target union. Must
629 * be 64-bit aligned.
630 * @union_size: Size of the target union in the type of @instance. Must be
631 * 64-bit aligned.
632 * @member_size: Size of the target member in the target union specified by
633 * @union_offset and @union_size. It is assumed that the offset of the target
634 * member is zero relative to @union_offset. Must be 64-bit aligned.
635 *
636 * You probably want to use PVR_IOCTL_UNION_PADDING_CHECK() instead of calling
637 * this function directly, since that macro abstracts away much of the setup,
638 * and also provides some static validation. See its docs for details.
639 *
640 * Return:
641 * * %true if every byte between the end of the used member of the union and
642 * the end of that union is zeroed, or
643 * * %false otherwise.
644 */
645static __always_inline bool
646pvr_ioctl_union_padding_check(void *instance, size_t union_offset,
647 size_t union_size, size_t member_size)
648{
649 /*
650 * void pointer arithmetic is technically illegal - cast to a byte
651 * pointer so this addition works safely.
652 */
653 void *padding_start = ((u8 *)instance) + union_offset + member_size;
654 size_t padding_size = union_size - member_size;
655
656 return !memchr_inv(padding_start, 0, padding_size);
657}
658
659/**
660 * PVR_STATIC_ASSERT_64BIT_ALIGNED() - Inline assertion for 64-bit alignment.
661 * @static_expr_: Target expression to evaluate.
662 *
663 * If @static_expr_ does not evaluate to a constant integer which would be a
664 * 64-bit aligned address (i.e. a multiple of 8), compilation will fail.
665 *
666 * Return:
667 * The value of @static_expr_.
668 */
669#define PVR_STATIC_ASSERT_64BIT_ALIGNED(static_expr_) \
670 ({ \
671 static_assert(((static_expr_) & (sizeof(u64) - 1)) == 0); \
672 (static_expr_); \
673 })
674
675/**
676 * PVR_IOCTL_UNION_PADDING_CHECK() - Validate that the implicit padding between
677 * the end of a union member and the end of the union itself is zeroed.
678 * @struct_instance_: An expression which evaluates to a pointer to a UAPI data
679 * struct.
680 * @union_: The name of the union member of @struct_instance_ to check. If the
681 * union member is nested within the type of @struct_instance_, this may
682 * contain the member access operator (".").
683 * @member_: The name of the member of @union_ to assess.
684 *
685 * This is a wrapper around pvr_ioctl_union_padding_check() which performs
686 * alignment checks and simplifies things for the caller.
687 *
688 * Return:
689 * * %true if every byte in @struct_instance_ between the end of @member_ and
690 * the end of @union_ is zeroed, or
691 * * %false otherwise.
692 */
693#define PVR_IOCTL_UNION_PADDING_CHECK(struct_instance_, union_, member_) \
694 ({ \
695 typeof(struct_instance_) __instance = (struct_instance_); \
696 size_t __union_offset = PVR_STATIC_ASSERT_64BIT_ALIGNED( \
697 offsetof(typeof(*__instance), union_)); \
698 size_t __union_size = PVR_STATIC_ASSERT_64BIT_ALIGNED( \
699 sizeof(__instance->union_)); \
700 size_t __member_size = PVR_STATIC_ASSERT_64BIT_ALIGNED( \
701 sizeof(__instance->union_.member_)); \
702 pvr_ioctl_union_padding_check(__instance, __union_offset, \
703 __union_size, __member_size); \
704 })
705
706#define PVR_FW_PROCESSOR_TYPE_META 0
707#define PVR_FW_PROCESSOR_TYPE_MIPS 1
708#define PVR_FW_PROCESSOR_TYPE_RISCV 2
709
710#endif /* PVR_DEVICE_H */
drivers/gpu/drm/imagination/pvr_device_info.c+254−0
@@ -0,0 +1,254 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_device.h"
5#include "pvr_device_info.h"
6#include "pvr_rogue_fwif_dev_info.h"
7
8#include <drm/drm_print.h>
9
10#include <linux/bits.h>
11#include <linux/minmax.h>
12#include <linux/stddef.h>
13#include <linux/types.h>
14
15#define QUIRK_MAPPING(quirk) \
16 [PVR_FW_HAS_BRN_##quirk] = offsetof(struct pvr_device, quirks.has_brn##quirk)
17
18static const uintptr_t quirks_mapping[] = {
19 QUIRK_MAPPING(44079),
20 QUIRK_MAPPING(47217),
21 QUIRK_MAPPING(48492),
22 QUIRK_MAPPING(48545),
23 QUIRK_MAPPING(49927),
24 QUIRK_MAPPING(50767),
25 QUIRK_MAPPING(51764),
26 QUIRK_MAPPING(62269),
27 QUIRK_MAPPING(63142),
28 QUIRK_MAPPING(63553),
29 QUIRK_MAPPING(66011),
30 QUIRK_MAPPING(71242),
31};
32
33#undef QUIRK_MAPPING
34
35#define ENHANCEMENT_MAPPING(enhancement) \
36 [PVR_FW_HAS_ERN_##enhancement] = offsetof(struct pvr_device, \
37 enhancements.has_ern##enhancement)
38
39static const uintptr_t enhancements_mapping[] = {
40 ENHANCEMENT_MAPPING(35421),
41 ENHANCEMENT_MAPPING(38020),
42 ENHANCEMENT_MAPPING(38748),
43 ENHANCEMENT_MAPPING(42064),
44 ENHANCEMENT_MAPPING(42290),
45 ENHANCEMENT_MAPPING(42606),
46 ENHANCEMENT_MAPPING(47025),
47 ENHANCEMENT_MAPPING(57596),
48};
49
50#undef ENHANCEMENT_MAPPING
51
52static void pvr_device_info_set_common(struct pvr_device *pvr_dev, const u64 *bitmask,
53 u32 bitmask_size, const uintptr_t *mapping, u32 mapping_max)
54{
55 const u32 mapping_max_size = (mapping_max + 63) >> 6;
56 const u32 nr_bits = min(bitmask_size * 64, mapping_max);
57
58 /* Warn if any unsupported values in the bitmask. */
59 if (bitmask_size > mapping_max_size) {
60 if (mapping == quirks_mapping)
61 drm_warn(from_pvr_device(pvr_dev), "Unsupported quirks in firmware image");
62 else
63 drm_warn(from_pvr_device(pvr_dev),
64 "Unsupported enhancements in firmware image");
65 } else if (bitmask_size == mapping_max_size && (mapping_max & 63)) {
66 u64 invalid_mask = ~0ull << (mapping_max & 63);
67
68 if (bitmask[bitmask_size - 1] & invalid_mask) {
69 if (mapping == quirks_mapping)
70 drm_warn(from_pvr_device(pvr_dev),
71 "Unsupported quirks in firmware image");
72 else
73 drm_warn(from_pvr_device(pvr_dev),
74 "Unsupported enhancements in firmware image");
75 }
76 }
77
78 for (u32 i = 0; i < nr_bits; i++) {
79 if (bitmask[i >> 6] & BIT_ULL(i & 63))
80 *(bool *)((u8 *)pvr_dev + mapping[i]) = true;
81 }
82}
83
84/**
85 * pvr_device_info_set_quirks() - Set device quirks from device information in firmware
86 * @pvr_dev: Device pointer.
87 * @quirks: Pointer to quirks mask in device information.
88 * @quirks_size: Size of quirks mask, in u64s.
89 */
90void pvr_device_info_set_quirks(struct pvr_device *pvr_dev, const u64 *quirks, u32 quirks_size)
91{
92 BUILD_BUG_ON(ARRAY_SIZE(quirks_mapping) != PVR_FW_HAS_BRN_MAX);
93
94 pvr_device_info_set_common(pvr_dev, quirks, quirks_size, quirks_mapping,
95 ARRAY_SIZE(quirks_mapping));
96}
97
98/**
99 * pvr_device_info_set_enhancements() - Set device enhancements from device information in firmware
100 * @pvr_dev: Device pointer.
101 * @enhancements: Pointer to enhancements mask in device information.
102 * @enhancements_size: Size of enhancements mask, in u64s.
103 */
104void pvr_device_info_set_enhancements(struct pvr_device *pvr_dev, const u64 *enhancements,
105 u32 enhancements_size)
106{
107 BUILD_BUG_ON(ARRAY_SIZE(enhancements_mapping) != PVR_FW_HAS_ERN_MAX);
108
109 pvr_device_info_set_common(pvr_dev, enhancements, enhancements_size,
110 enhancements_mapping, ARRAY_SIZE(enhancements_mapping));
111}
112
113#define FEATURE_MAPPING(fw_feature, feature) \
114 [PVR_FW_HAS_FEATURE_##fw_feature] = { \
115 .flag_offset = offsetof(struct pvr_device, features.has_##feature), \
116 .value_offset = 0 \
117 }
118
119#define FEATURE_MAPPING_VALUE(fw_feature, feature) \
120 [PVR_FW_HAS_FEATURE_##fw_feature] = { \
121 .flag_offset = offsetof(struct pvr_device, features.has_##feature), \
122 .value_offset = offsetof(struct pvr_device, features.feature) \
123 }
124
125static const struct {
126 uintptr_t flag_offset;
127 uintptr_t value_offset;
128} features_mapping[] = {
129 FEATURE_MAPPING(AXI_ACELITE, axi_acelite),
130 FEATURE_MAPPING_VALUE(CDM_CONTROL_STREAM_FORMAT, cdm_control_stream_format),
131 FEATURE_MAPPING(CLUSTER_GROUPING, cluster_grouping),
132 FEATURE_MAPPING_VALUE(COMMON_STORE_SIZE_IN_DWORDS, common_store_size_in_dwords),
133 FEATURE_MAPPING(COMPUTE, compute),
134 FEATURE_MAPPING(COMPUTE_MORTON_CAPABLE, compute_morton_capable),
135 FEATURE_MAPPING(COMPUTE_OVERLAP, compute_overlap),
136 FEATURE_MAPPING(COREID_PER_OS, coreid_per_os),
137 FEATURE_MAPPING(DYNAMIC_DUST_POWER, dynamic_dust_power),
138 FEATURE_MAPPING_VALUE(ECC_RAMS, ecc_rams),
139 FEATURE_MAPPING_VALUE(FBCDC, fbcdc),
140 FEATURE_MAPPING_VALUE(FBCDC_ALGORITHM, fbcdc_algorithm),
141 FEATURE_MAPPING_VALUE(FBCDC_ARCHITECTURE, fbcdc_architecture),
142 FEATURE_MAPPING_VALUE(FBC_MAX_DEFAULT_DESCRIPTORS, fbc_max_default_descriptors),
143 FEATURE_MAPPING_VALUE(FBC_MAX_LARGE_DESCRIPTORS, fbc_max_large_descriptors),
144 FEATURE_MAPPING(FB_CDC_V4, fb_cdc_v4),
145 FEATURE_MAPPING(GPU_MULTICORE_SUPPORT, gpu_multicore_support),
146 FEATURE_MAPPING(GPU_VIRTUALISATION, gpu_virtualisation),
147 FEATURE_MAPPING(GS_RTA_SUPPORT, gs_rta_support),
148 FEATURE_MAPPING(IRQ_PER_OS, irq_per_os),
149 FEATURE_MAPPING_VALUE(ISP_MAX_TILES_IN_FLIGHT, isp_max_tiles_in_flight),
150 FEATURE_MAPPING_VALUE(ISP_SAMPLES_PER_PIXEL, isp_samples_per_pixel),
151 FEATURE_MAPPING(ISP_ZLS_D24_S8_PACKING_OGL_MODE, isp_zls_d24_s8_packing_ogl_mode),
152 FEATURE_MAPPING_VALUE(LAYOUT_MARS, layout_mars),
153 FEATURE_MAPPING_VALUE(MAX_PARTITIONS, max_partitions),
154 FEATURE_MAPPING_VALUE(META, meta),
155 FEATURE_MAPPING_VALUE(META_COREMEM_SIZE, meta_coremem_size),
156 FEATURE_MAPPING(MIPS, mips),
157 FEATURE_MAPPING_VALUE(NUM_CLUSTERS, num_clusters),
158 FEATURE_MAPPING_VALUE(NUM_ISP_IPP_PIPES, num_isp_ipp_pipes),
159 FEATURE_MAPPING_VALUE(NUM_OSIDS, num_osids),
160 FEATURE_MAPPING_VALUE(NUM_RASTER_PIPES, num_raster_pipes),
161 FEATURE_MAPPING(PBE2_IN_XE, pbe2_in_xe),
162 FEATURE_MAPPING(PBVNC_COREID_REG, pbvnc_coreid_reg),
163 FEATURE_MAPPING(PERFBUS, perfbus),
164 FEATURE_MAPPING(PERF_COUNTER_BATCH, perf_counter_batch),
165 FEATURE_MAPPING_VALUE(PHYS_BUS_WIDTH, phys_bus_width),
166 FEATURE_MAPPING(RISCV_FW_PROCESSOR, riscv_fw_processor),
167 FEATURE_MAPPING(ROGUEXE, roguexe),
168 FEATURE_MAPPING(S7_TOP_INFRASTRUCTURE, s7_top_infrastructure),
169 FEATURE_MAPPING(SIMPLE_INTERNAL_PARAMETER_FORMAT, simple_internal_parameter_format),
170 FEATURE_MAPPING(SIMPLE_INTERNAL_PARAMETER_FORMAT_V2, simple_internal_parameter_format_v2),
171 FEATURE_MAPPING_VALUE(SIMPLE_PARAMETER_FORMAT_VERSION, simple_parameter_format_version),
172 FEATURE_MAPPING_VALUE(SLC_BANKS, slc_banks),
173 FEATURE_MAPPING_VALUE(SLC_CACHE_LINE_SIZE_BITS, slc_cache_line_size_bits),
174 FEATURE_MAPPING(SLC_SIZE_CONFIGURABLE, slc_size_configurable),
175 FEATURE_MAPPING_VALUE(SLC_SIZE_IN_KILOBYTES, slc_size_in_kilobytes),
176 FEATURE_MAPPING(SOC_TIMER, soc_timer),
177 FEATURE_MAPPING(SYS_BUS_SECURE_RESET, sys_bus_secure_reset),
178 FEATURE_MAPPING(TESSELLATION, tessellation),
179 FEATURE_MAPPING(TILE_REGION_PROTECTION, tile_region_protection),
180 FEATURE_MAPPING_VALUE(TILE_SIZE_X, tile_size_x),
181 FEATURE_MAPPING_VALUE(TILE_SIZE_Y, tile_size_y),
182 FEATURE_MAPPING(TLA, tla),
183 FEATURE_MAPPING(TPU_CEM_DATAMASTER_GLOBAL_REGISTERS, tpu_cem_datamaster_global_registers),
184 FEATURE_MAPPING(TPU_DM_GLOBAL_REGISTERS, tpu_dm_global_registers),
185 FEATURE_MAPPING(TPU_FILTERING_MODE_CONTROL, tpu_filtering_mode_control),
186 FEATURE_MAPPING_VALUE(USC_MIN_OUTPUT_REGISTERS_PER_PIX, usc_min_output_registers_per_pix),
187 FEATURE_MAPPING(VDM_DRAWINDIRECT, vdm_drawindirect),
188 FEATURE_MAPPING(VDM_OBJECT_LEVEL_LLS, vdm_object_level_lls),
189 FEATURE_MAPPING_VALUE(VIRTUAL_ADDRESS_SPACE_BITS, virtual_address_space_bits),
190 FEATURE_MAPPING(WATCHDOG_TIMER, watchdog_timer),
191 FEATURE_MAPPING(WORKGROUP_PROTECTION, workgroup_protection),
192 FEATURE_MAPPING_VALUE(XE_ARCHITECTURE, xe_architecture),
193 FEATURE_MAPPING(XE_MEMORY_HIERARCHY, xe_memory_hierarchy),
194 FEATURE_MAPPING(XE_TPU2, xe_tpu2),
195 FEATURE_MAPPING_VALUE(XPU_MAX_REGBANKS_ADDR_WIDTH, xpu_max_regbanks_addr_width),
196 FEATURE_MAPPING_VALUE(XPU_MAX_SLAVES, xpu_max_slaves),
197 FEATURE_MAPPING_VALUE(XPU_REGISTER_BROADCAST, xpu_register_broadcast),
198 FEATURE_MAPPING(XT_TOP_INFRASTRUCTURE, xt_top_infrastructure),
199 FEATURE_MAPPING(ZLS_SUBTILE, zls_subtile),
200};
201
202#undef FEATURE_MAPPING_VALUE
203#undef FEATURE_MAPPING
204
205/**
206 * pvr_device_info_set_features() - Set device features from device information in firmware
207 * @pvr_dev: Device pointer.
208 * @features: Pointer to features mask in device information.
209 * @features_size: Size of features mask, in u64s.
210 * @feature_param_size: Size of feature parameters, in u64s.
211 *
212 * Returns:
213 * * 0 on success, or
214 * * -%EINVAL on malformed stream.
215 */
216int pvr_device_info_set_features(struct pvr_device *pvr_dev, const u64 *features, u32 features_size,
217 u32 feature_param_size)
218{
219 const u32 mapping_max = ARRAY_SIZE(features_mapping);
220 const u32 mapping_max_size = (mapping_max + 63) >> 6;
221 const u32 nr_bits = min(features_size * 64, mapping_max);
222 const u64 *feature_params = features + features_size;
223 u32 param_idx = 0;
224
225 BUILD_BUG_ON(ARRAY_SIZE(features_mapping) != PVR_FW_HAS_FEATURE_MAX);
226
227 /* Verify no unsupported values in the bitmask. */
228 if (features_size > mapping_max_size) {
229 drm_warn(from_pvr_device(pvr_dev), "Unsupported features in firmware image");
230 } else if (features_size == mapping_max_size && (mapping_max & 63)) {
231 u64 invalid_mask = ~0ull << (mapping_max & 63);
232
233 if (features[features_size - 1] & invalid_mask)
234 drm_warn(from_pvr_device(pvr_dev),
235 "Unsupported features in firmware image");
236 }
237
238 for (u32 i = 0; i < nr_bits; i++) {
239 if (features[i >> 6] & BIT_ULL(i & 63)) {
240 *(bool *)((u8 *)pvr_dev + features_mapping[i].flag_offset) = true;
241
242 if (features_mapping[i].value_offset) {
243 if (param_idx >= feature_param_size)
244 return -EINVAL;
245
246 *(u64 *)((u8 *)pvr_dev + features_mapping[i].value_offset) =
247 feature_params[param_idx];
248 param_idx++;
249 }
250 }
251 }
252
253 return 0;
254}
drivers/gpu/drm/imagination/pvr_device_info.h+186−0
@@ -0,0 +1,186 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_DEVICE_INFO_H
5#define PVR_DEVICE_INFO_H
6
7#include <linux/types.h>
8
9struct pvr_device;
10
11/*
12 * struct pvr_device_features - Hardware feature information
13 */
14struct pvr_device_features {
15 bool has_axi_acelite;
16 bool has_cdm_control_stream_format;
17 bool has_cluster_grouping;
18 bool has_common_store_size_in_dwords;
19 bool has_compute;
20 bool has_compute_morton_capable;
21 bool has_compute_overlap;
22 bool has_coreid_per_os;
23 bool has_dynamic_dust_power;
24 bool has_ecc_rams;
25 bool has_fb_cdc_v4;
26 bool has_fbc_max_default_descriptors;
27 bool has_fbc_max_large_descriptors;
28 bool has_fbcdc;
29 bool has_fbcdc_algorithm;
30 bool has_fbcdc_architecture;
31 bool has_gpu_multicore_support;
32 bool has_gpu_virtualisation;
33 bool has_gs_rta_support;
34 bool has_irq_per_os;
35 bool has_isp_max_tiles_in_flight;
36 bool has_isp_samples_per_pixel;
37 bool has_isp_zls_d24_s8_packing_ogl_mode;
38 bool has_layout_mars;
39 bool has_max_partitions;
40 bool has_meta;
41 bool has_meta_coremem_size;
42 bool has_mips;
43 bool has_num_clusters;
44 bool has_num_isp_ipp_pipes;
45 bool has_num_osids;
46 bool has_num_raster_pipes;
47 bool has_pbe2_in_xe;
48 bool has_pbvnc_coreid_reg;
49 bool has_perfbus;
50 bool has_perf_counter_batch;
51 bool has_phys_bus_width;
52 bool has_riscv_fw_processor;
53 bool has_roguexe;
54 bool has_s7_top_infrastructure;
55 bool has_simple_internal_parameter_format;
56 bool has_simple_internal_parameter_format_v2;
57 bool has_simple_parameter_format_version;
58 bool has_slc_banks;
59 bool has_slc_cache_line_size_bits;
60 bool has_slc_size_configurable;
61 bool has_slc_size_in_kilobytes;
62 bool has_soc_timer;
63 bool has_sys_bus_secure_reset;
64 bool has_tessellation;
65 bool has_tile_region_protection;
66 bool has_tile_size_x;
67 bool has_tile_size_y;
68 bool has_tla;
69 bool has_tpu_cem_datamaster_global_registers;
70 bool has_tpu_dm_global_registers;
71 bool has_tpu_filtering_mode_control;
72 bool has_usc_min_output_registers_per_pix;
73 bool has_vdm_drawindirect;
74 bool has_vdm_object_level_lls;
75 bool has_virtual_address_space_bits;
76 bool has_watchdog_timer;
77 bool has_workgroup_protection;
78 bool has_xe_architecture;
79 bool has_xe_memory_hierarchy;
80 bool has_xe_tpu2;
81 bool has_xpu_max_regbanks_addr_width;
82 bool has_xpu_max_slaves;
83 bool has_xpu_register_broadcast;
84 bool has_xt_top_infrastructure;
85 bool has_zls_subtile;
86
87 u64 cdm_control_stream_format;
88 u64 common_store_size_in_dwords;
89 u64 ecc_rams;
90 u64 fbc_max_default_descriptors;
91 u64 fbc_max_large_descriptors;
92 u64 fbcdc;
93 u64 fbcdc_algorithm;
94 u64 fbcdc_architecture;
95 u64 isp_max_tiles_in_flight;
96 u64 isp_samples_per_pixel;
97 u64 layout_mars;
98 u64 max_partitions;
99 u64 meta;
100 u64 meta_coremem_size;
101 u64 num_clusters;
102 u64 num_isp_ipp_pipes;
103 u64 num_osids;
104 u64 num_raster_pipes;
105 u64 phys_bus_width;
106 u64 simple_parameter_format_version;
107 u64 slc_banks;
108 u64 slc_cache_line_size_bits;
109 u64 slc_size_in_kilobytes;
110 u64 tile_size_x;
111 u64 tile_size_y;
112 u64 usc_min_output_registers_per_pix;
113 u64 virtual_address_space_bits;
114 u64 xe_architecture;
115 u64 xpu_max_regbanks_addr_width;
116 u64 xpu_max_slaves;
117 u64 xpu_register_broadcast;
118};
119
120/*
121 * struct pvr_device_quirks - Hardware quirk information
122 */
123struct pvr_device_quirks {
124 bool has_brn44079;
125 bool has_brn47217;
126 bool has_brn48492;
127 bool has_brn48545;
128 bool has_brn49927;
129 bool has_brn50767;
130 bool has_brn51764;
131 bool has_brn62269;
132 bool has_brn63142;
133 bool has_brn63553;
134 bool has_brn66011;
135 bool has_brn71242;
136};
137
138/*
139 * struct pvr_device_enhancements - Hardware enhancement information
140 */
141struct pvr_device_enhancements {
142 bool has_ern35421;
143 bool has_ern38020;
144 bool has_ern38748;
145 bool has_ern42064;
146 bool has_ern42290;
147 bool has_ern42606;
148 bool has_ern47025;
149 bool has_ern57596;
150};
151
152void pvr_device_info_set_quirks(struct pvr_device *pvr_dev, const u64 *bitmask,
153 u32 bitmask_len);
154void pvr_device_info_set_enhancements(struct pvr_device *pvr_dev, const u64 *bitmask,
155 u32 bitmask_len);
156int pvr_device_info_set_features(struct pvr_device *pvr_dev, const u64 *features, u32 features_size,
157 u32 feature_param_size);
158
159/*
160 * Meta cores
161 *
162 * These are the values for the 'meta' feature when the feature is present
163 * (as per &struct pvr_device_features)/
164 */
165#define PVR_META_MTP218 (1)
166#define PVR_META_MTP219 (2)
167#define PVR_META_LTP218 (3)
168#define PVR_META_LTP217 (4)
169
170enum {
171 PVR_FEATURE_CDM_USER_MODE_QUEUE,
172 PVR_FEATURE_CLUSTER_GROUPING,
173 PVR_FEATURE_COMPUTE_MORTON_CAPABLE,
174 PVR_FEATURE_FB_CDC_V4,
175 PVR_FEATURE_GPU_MULTICORE_SUPPORT,
176 PVR_FEATURE_ISP_ZLS_D24_S8_PACKING_OGL_MODE,
177 PVR_FEATURE_REQUIRES_FB_CDC_ZLS_SETUP,
178 PVR_FEATURE_S7_TOP_INFRASTRUCTURE,
179 PVR_FEATURE_TESSELLATION,
180 PVR_FEATURE_TPU_DM_GLOBAL_REGISTERS,
181 PVR_FEATURE_VDM_DRAWINDIRECT,
182 PVR_FEATURE_VDM_OBJECT_LEVEL_LLS,
183 PVR_FEATURE_ZLS_SUBTILE,
184};
185
186#endif /* PVR_DEVICE_INFO_H */
drivers/gpu/drm/imagination/pvr_drv.c+1501−0
@@ -0,0 +1,1501 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_context.h"
5#include "pvr_debugfs.h"
6#include "pvr_device.h"
7#include "pvr_drv.h"
8#include "pvr_free_list.h"
9#include "pvr_gem.h"
10#include "pvr_hwrt.h"
11#include "pvr_job.h"
12#include "pvr_mmu.h"
13#include "pvr_power.h"
14#include "pvr_rogue_defs.h"
15#include "pvr_rogue_fwif_client.h"
16#include "pvr_rogue_fwif_shared.h"
17#include "pvr_vm.h"
18
19#include <uapi/drm/pvr_drm.h>
20
21#include <drm/drm_device.h>
22#include <drm/drm_drv.h>
23#include <drm/drm_file.h>
24#include <drm/drm_gem.h>
25#include <drm/drm_ioctl.h>
26
27#include <linux/err.h>
28#include <linux/export.h>
29#include <linux/fs.h>
30#include <linux/kernel.h>
31#include <linux/mod_devicetable.h>
32#include <linux/module.h>
33#include <linux/moduleparam.h>
34#include <linux/of_device.h>
35#include <linux/of_platform.h>
36#include <linux/platform_device.h>
37#include <linux/pm_runtime.h>
38#include <linux/xarray.h>
39
40/**
41 * DOC: PowerVR (Series 6 and later) and IMG Graphics Driver
42 *
43 * This driver supports the following PowerVR/IMG graphics cores from Imagination Technologies:
44 *
45 * * AXE-1-16M (found in Texas Instruments AM62)
46 */
47
48/**
49 * pvr_ioctl_create_bo() - IOCTL to create a GEM buffer object.
50 * @drm_dev: [IN] Target DRM device.
51 * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
52 * &struct drm_pvr_ioctl_create_bo_args.
53 * @file: [IN] DRM file-private data.
54 *
55 * Called from userspace with %DRM_IOCTL_PVR_CREATE_BO.
56 *
57 * Return:
58 * * 0 on success,
59 * * -%EINVAL if the value of &drm_pvr_ioctl_create_bo_args.size is zero
60 * or wider than &typedef size_t,
61 * * -%EINVAL if any bits in &drm_pvr_ioctl_create_bo_args.flags that are
62 * reserved or undefined are set,
63 * * -%EINVAL if any padding fields in &drm_pvr_ioctl_create_bo_args are not
64 * zero,
65 * * Any error encountered while creating the object (see
66 * pvr_gem_object_create()), or
67 * * Any error encountered while transferring ownership of the object into a
68 * userspace-accessible handle (see pvr_gem_object_into_handle()).
69 */
70static int
71pvr_ioctl_create_bo(struct drm_device *drm_dev, void *raw_args,
72 struct drm_file *file)
73{
74 struct drm_pvr_ioctl_create_bo_args *args = raw_args;
75 struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
76 struct pvr_file *pvr_file = to_pvr_file(file);
77
78 struct pvr_gem_object *pvr_obj;
79 size_t sanitized_size;
80
81 int idx;
82 int err;
83
84 if (!drm_dev_enter(drm_dev, &idx))
85 return -EIO;
86
87 /* All padding fields must be zeroed. */
88 if (args->_padding_c != 0) {
89 err = -EINVAL;
90 goto err_drm_dev_exit;
91 }
92
93 /*
94 * On 64-bit platforms (our primary target), size_t is a u64. However,
95 * on other architectures we have to check for overflow when casting
96 * down to size_t from u64.
97 *
98 * We also disallow zero-sized allocations, and reserved (kernel-only)
99 * flags.
100 */
101 if (args->size > SIZE_MAX || args->size == 0 || args->flags &
102 ~DRM_PVR_BO_FLAGS_MASK || args->size & (PVR_DEVICE_PAGE_SIZE - 1)) {
103 err = -EINVAL;
104 goto err_drm_dev_exit;
105 }
106
107 sanitized_size = (size_t)args->size;
108
109 /*
110 * Create a buffer object and transfer ownership to a userspace-
111 * accessible handle.
112 */
113 pvr_obj = pvr_gem_object_create(pvr_dev, sanitized_size, args->flags);
114 if (IS_ERR(pvr_obj)) {
115 err = PTR_ERR(pvr_obj);
116 goto err_drm_dev_exit;
117 }
118
119 /* This function will not modify &args->handle unless it succeeds. */
120 err = pvr_gem_object_into_handle(pvr_obj, pvr_file, &args->handle);
121 if (err)
122 goto err_destroy_obj;
123
124 drm_dev_exit(idx);
125
126 return 0;
127
128err_destroy_obj:
129 /*
130 * GEM objects are refcounted, so there is no explicit destructor
131 * function. Instead, we release the singular reference we currently
132 * hold on the object and let GEM take care of the rest.
133 */
134 pvr_gem_object_put(pvr_obj);
135
136err_drm_dev_exit:
137 drm_dev_exit(idx);
138
139 return err;
140}
141
142/**
143 * pvr_ioctl_get_bo_mmap_offset() - IOCTL to generate a "fake" offset to be
144 * used when calling mmap() from userspace to map the given GEM buffer object
145 * @drm_dev: [IN] DRM device (unused).
146 * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
147 * &struct drm_pvr_ioctl_get_bo_mmap_offset_args.
148 * @file: [IN] DRM file private data.
149 *
150 * Called from userspace with %DRM_IOCTL_PVR_GET_BO_MMAP_OFFSET.
151 *
152 * This IOCTL does *not* perform an mmap. See the docs on
153 * &struct drm_pvr_ioctl_get_bo_mmap_offset_args for details.
154 *
155 * Return:
156 * * 0 on success,
157 * * -%ENOENT if the handle does not reference a valid GEM buffer object,
158 * * -%EINVAL if any padding fields in &struct
159 * drm_pvr_ioctl_get_bo_mmap_offset_args are not zero, or
160 * * Any error returned by drm_gem_create_mmap_offset().
161 */
162static int
163pvr_ioctl_get_bo_mmap_offset(struct drm_device *drm_dev, void *raw_args,
164 struct drm_file *file)
165{
166 struct drm_pvr_ioctl_get_bo_mmap_offset_args *args = raw_args;
167 struct pvr_file *pvr_file = to_pvr_file(file);
168 struct pvr_gem_object *pvr_obj;
169 struct drm_gem_object *gem_obj;
170 int idx;
171 int ret;
172
173 if (!drm_dev_enter(drm_dev, &idx))
174 return -EIO;
175
176 /* All padding fields must be zeroed. */
177 if (args->_padding_4 != 0) {
178 ret = -EINVAL;
179 goto err_drm_dev_exit;
180 }
181
182 /*
183 * Obtain a kernel reference to the buffer object. This reference is
184 * counted and must be manually dropped before returning. If a buffer
185 * object cannot be found for the specified handle, return -%ENOENT (No
186 * such file or directory).
187 */
188 pvr_obj = pvr_gem_object_from_handle(pvr_file, args->handle);
189 if (!pvr_obj) {
190 ret = -ENOENT;
191 goto err_drm_dev_exit;
192 }
193
194 gem_obj = gem_from_pvr_gem(pvr_obj);
195
196 /*
197 * Allocate a fake offset which can be used in userspace calls to mmap
198 * on the DRM device file. If this fails, return the error code. This
199 * operation is idempotent.
200 */
201 ret = drm_gem_create_mmap_offset(gem_obj);
202 if (ret != 0) {
203 /* Drop our reference to the buffer object. */
204 drm_gem_object_put(gem_obj);
205 goto err_drm_dev_exit;
206 }
207
208 /*
209 * Read out the fake offset allocated by the earlier call to
210 * drm_gem_create_mmap_offset.
211 */
212 args->offset = drm_vma_node_offset_addr(&gem_obj->vma_node);
213
214 /* Drop our reference to the buffer object. */
215 pvr_gem_object_put(pvr_obj);
216
217err_drm_dev_exit:
218 drm_dev_exit(idx);
219
220 return ret;
221}
222
223static __always_inline u64
224pvr_fw_version_packed(u32 major, u32 minor)
225{
226 return ((u64)major << 32) | minor;
227}
228
229static u32
230rogue_get_common_store_partition_space_size(struct pvr_device *pvr_dev)
231{
232 u32 max_partitions = 0;
233 u32 tile_size_x = 0;
234 u32 tile_size_y = 0;
235
236 PVR_FEATURE_VALUE(pvr_dev, tile_size_x, &tile_size_x);
237 PVR_FEATURE_VALUE(pvr_dev, tile_size_y, &tile_size_y);
238 PVR_FEATURE_VALUE(pvr_dev, max_partitions, &max_partitions);
239
240 if (tile_size_x == 16 && tile_size_y == 16) {
241 u32 usc_min_output_registers_per_pix = 0;
242
243 PVR_FEATURE_VALUE(pvr_dev, usc_min_output_registers_per_pix,
244 &usc_min_output_registers_per_pix);
245
246 return tile_size_x * tile_size_y * max_partitions *
247 usc_min_output_registers_per_pix;
248 }
249
250 return max_partitions * 1024;
251}
252
253static u32
254rogue_get_common_store_alloc_region_size(struct pvr_device *pvr_dev)
255{
256 u32 common_store_size_in_dwords = 512 * 4 * 4;
257 u32 alloc_region_size;
258
259 PVR_FEATURE_VALUE(pvr_dev, common_store_size_in_dwords, &common_store_size_in_dwords);
260
261 alloc_region_size = common_store_size_in_dwords - (256U * 4U) -
262 rogue_get_common_store_partition_space_size(pvr_dev);
263
264 if (PVR_HAS_QUIRK(pvr_dev, 44079)) {
265 u32 common_store_split_point = (768U * 4U * 4U);
266
267 return min(common_store_split_point - (256U * 4U), alloc_region_size);
268 }
269
270 return alloc_region_size;
271}
272
273static inline u32
274rogue_get_num_phantoms(struct pvr_device *pvr_dev)
275{
276 u32 num_clusters = 1;
277
278 PVR_FEATURE_VALUE(pvr_dev, num_clusters, &num_clusters);
279
280 return ROGUE_REQ_NUM_PHANTOMS(num_clusters);
281}
282
283static inline u32
284rogue_get_max_coeffs(struct pvr_device *pvr_dev)
285{
286 u32 max_coeff_additional_portion = ROGUE_MAX_VERTEX_SHARED_REGISTERS;
287 u32 pending_allocation_shared_regs = 2U * 1024U;
288 u32 pending_allocation_coeff_regs = 0U;
289 u32 num_phantoms = rogue_get_num_phantoms(pvr_dev);
290 u32 tiles_in_flight = 0;
291 u32 max_coeff_pixel_portion;
292
293 PVR_FEATURE_VALUE(pvr_dev, isp_max_tiles_in_flight, &tiles_in_flight);
294 max_coeff_pixel_portion = DIV_ROUND_UP(tiles_in_flight, num_phantoms);
295 max_coeff_pixel_portion *= ROGUE_MAX_PIXEL_SHARED_REGISTERS;
296
297 /*
298 * Compute tasks on cores with BRN48492 and without compute overlap may lock
299 * up without two additional lines of coeffs.
300 */
301 if (PVR_HAS_QUIRK(pvr_dev, 48492) && !PVR_HAS_FEATURE(pvr_dev, compute_overlap))
302 pending_allocation_coeff_regs = 2U * 1024U;
303
304 if (PVR_HAS_ENHANCEMENT(pvr_dev, 38748))
305 pending_allocation_shared_regs = 0;
306
307 if (PVR_HAS_ENHANCEMENT(pvr_dev, 38020))
308 max_coeff_additional_portion += ROGUE_MAX_COMPUTE_SHARED_REGISTERS;
309
310 return rogue_get_common_store_alloc_region_size(pvr_dev) + pending_allocation_coeff_regs -
311 (max_coeff_pixel_portion + max_coeff_additional_portion +
312 pending_allocation_shared_regs);
313}
314
315static inline u32
316rogue_get_cdm_max_local_mem_size_regs(struct pvr_device *pvr_dev)
317{
318 u32 available_coeffs_in_dwords = rogue_get_max_coeffs(pvr_dev);
319
320 if (PVR_HAS_QUIRK(pvr_dev, 48492) && PVR_HAS_FEATURE(pvr_dev, roguexe) &&
321 !PVR_HAS_FEATURE(pvr_dev, compute_overlap)) {
322 /* Driver must not use the 2 reserved lines. */
323 available_coeffs_in_dwords -= ROGUE_CSRM_LINE_SIZE_IN_DWORDS * 2;
324 }
325
326 /*
327 * The maximum amount of local memory available to a kernel is the minimum
328 * of the total number of coefficient registers available and the max common
329 * store allocation size which can be made by the CDM.
330 *
331 * If any coeff lines are reserved for tessellation or pixel then we need to
332 * subtract those too.
333 */
334 return min(available_coeffs_in_dwords, (u32)ROGUE_MAX_PER_KERNEL_LOCAL_MEM_SIZE_REGS);
335}
336
337/**
338 * pvr_dev_query_gpu_info_get()
339 * @pvr_dev: Device pointer.
340 * @args: [IN] Device query arguments containing a pointer to a userspace
341 * struct drm_pvr_dev_query_gpu_info.
342 *
343 * If the query object pointer is NULL, the size field is updated with the
344 * expected size of the query object.
345 *
346 * Returns:
347 * * 0 on success, or if size is requested using a NULL pointer, or
348 * * -%E2BIG if the indicated length of the allocation is less than is
349 * required to contain the copied data, or
350 * * -%EFAULT if local memory could not be copied to userspace.
351 */
352static int
353pvr_dev_query_gpu_info_get(struct pvr_device *pvr_dev,
354 struct drm_pvr_ioctl_dev_query_args *args)
355{
356 struct drm_pvr_dev_query_gpu_info gpu_info = {0};
357 int err;
358
359 if (!args->pointer) {
360 args->size = sizeof(struct drm_pvr_dev_query_gpu_info);
361 return 0;
362 }
363
364 gpu_info.gpu_id =
365 pvr_gpu_id_to_packed_bvnc(&pvr_dev->gpu_id);
366 gpu_info.num_phantoms = rogue_get_num_phantoms(pvr_dev);
367
368 err = PVR_UOBJ_SET(args->pointer, args->size, gpu_info);
369 if (err < 0)
370 return err;
371
372 if (args->size > sizeof(gpu_info))
373 args->size = sizeof(gpu_info);
374 return 0;
375}
376
377/**
378 * pvr_dev_query_runtime_info_get()
379 * @pvr_dev: Device pointer.
380 * @args: [IN] Device query arguments containing a pointer to a userspace
381 * struct drm_pvr_dev_query_runtime_info.
382 *
383 * If the query object pointer is NULL, the size field is updated with the
384 * expected size of the query object.
385 *
386 * Returns:
387 * * 0 on success, or if size is requested using a NULL pointer, or
388 * * -%E2BIG if the indicated length of the allocation is less than is
389 * required to contain the copied data, or
390 * * -%EFAULT if local memory could not be copied to userspace.
391 */
392static int
393pvr_dev_query_runtime_info_get(struct pvr_device *pvr_dev,
394 struct drm_pvr_ioctl_dev_query_args *args)
395{
396 struct drm_pvr_dev_query_runtime_info runtime_info = {0};
397 int err;
398
399 if (!args->pointer) {
400 args->size = sizeof(struct drm_pvr_dev_query_runtime_info);
401 return 0;
402 }
403
404 runtime_info.free_list_min_pages =
405 pvr_get_free_list_min_pages(pvr_dev);
406 runtime_info.free_list_max_pages =
407 ROGUE_PM_MAX_FREELIST_SIZE / ROGUE_PM_PAGE_SIZE;
408 runtime_info.common_store_alloc_region_size =
409 rogue_get_common_store_alloc_region_size(pvr_dev);
410 runtime_info.common_store_partition_space_size =
411 rogue_get_common_store_partition_space_size(pvr_dev);
412 runtime_info.max_coeffs = rogue_get_max_coeffs(pvr_dev);
413 runtime_info.cdm_max_local_mem_size_regs =
414 rogue_get_cdm_max_local_mem_size_regs(pvr_dev);
415
416 err = PVR_UOBJ_SET(args->pointer, args->size, runtime_info);
417 if (err < 0)
418 return err;
419
420 if (args->size > sizeof(runtime_info))
421 args->size = sizeof(runtime_info);
422 return 0;
423}
424
425/**
426 * pvr_dev_query_quirks_get() - Unpack array of quirks at the address given
427 * in a struct drm_pvr_dev_query_quirks, or gets the amount of space required
428 * for it.
429 * @pvr_dev: Device pointer.
430 * @args: [IN] Device query arguments containing a pointer to a userspace
431 * struct drm_pvr_dev_query_query_quirks.
432 *
433 * If the query object pointer is NULL, the size field is updated with the
434 * expected size of the query object.
435 * If the userspace pointer in the query object is NULL, or the count is
436 * short, no data is copied.
437 * The count field will be updated to that copied, or if either pointer is
438 * NULL, that which would have been copied.
439 * The size field in the query object will be updated to the size copied.
440 *
441 * Returns:
442 * * 0 on success, or if size/count is requested using a NULL pointer, or
443 * * -%EINVAL if args contained non-zero reserved fields, or
444 * * -%E2BIG if the indicated length of the allocation is less than is
445 * required to contain the copied data, or
446 * * -%EFAULT if local memory could not be copied to userspace.
447 */
448static int
449pvr_dev_query_quirks_get(struct pvr_device *pvr_dev,
450 struct drm_pvr_ioctl_dev_query_args *args)
451{
452 /*
453 * @FIXME - hardcoding of numbers here is intended as an
454 * intermediate step so the UAPI can be fixed, but requires a
455 * a refactor in the future to store them in a more appropriate
456 * location
457 */
458 static const u32 umd_quirks_musthave[] = {
459 47217,
460 49927,
461 62269,
462 };
463 static const u32 umd_quirks[] = {
464 48545,
465 51764,
466 };
467 struct drm_pvr_dev_query_quirks query;
468 u32 out[ARRAY_SIZE(umd_quirks_musthave) + ARRAY_SIZE(umd_quirks)];
469 size_t out_musthave_count = 0;
470 size_t out_count = 0;
471 int err;
472
473 if (!args->pointer) {
474 args->size = sizeof(struct drm_pvr_dev_query_quirks);
475 return 0;
476 }
477
478 err = PVR_UOBJ_GET(query, args->size, args->pointer);
479
480 if (err < 0)
481 return err;
482 if (query._padding_c)
483 return -EINVAL;
484
485 for (int i = 0; i < ARRAY_SIZE(umd_quirks_musthave); i++) {
486 if (pvr_device_has_uapi_quirk(pvr_dev, umd_quirks_musthave[i])) {
487 out[out_count++] = umd_quirks_musthave[i];
488 out_musthave_count++;
489 }
490 }
491
492 for (int i = 0; i < ARRAY_SIZE(umd_quirks); i++) {
493 if (pvr_device_has_uapi_quirk(pvr_dev, umd_quirks[i]))
494 out[out_count++] = umd_quirks[i];
495 }
496
497 if (!query.quirks)
498 goto copy_out;
499 if (query.count < out_count)
500 return -E2BIG;
501
502 if (copy_to_user(u64_to_user_ptr(query.quirks), out,
503 out_count * sizeof(u32))) {
504 return -EFAULT;
505 }
506
507 query.musthave_count = out_musthave_count;
508
509copy_out:
510 query.count = out_count;
511 err = PVR_UOBJ_SET(args->pointer, args->size, query);
512 if (err < 0)
513 return err;
514
515 args->size = sizeof(query);
516 return 0;
517}
518
519/**
520 * pvr_dev_query_enhancements_get() - Unpack array of enhancements at the
521 * address given in a struct drm_pvr_dev_query_enhancements, or gets the amount
522 * of space required for it.
523 * @pvr_dev: Device pointer.
524 * @args: [IN] Device query arguments containing a pointer to a userspace
525 * struct drm_pvr_dev_query_enhancements.
526 *
527 * If the query object pointer is NULL, the size field is updated with the
528 * expected size of the query object.
529 * If the userspace pointer in the query object is NULL, or the count is
530 * short, no data is copied.
531 * The count field will be updated to that copied, or if either pointer is
532 * NULL, that which would have been copied.
533 * The size field in the query object will be updated to the size copied.
534 *
535 * Returns:
536 * * 0 on success, or if size/count is requested using a NULL pointer, or
537 * * -%EINVAL if args contained non-zero reserved fields, or
538 * * -%E2BIG if the indicated length of the allocation is less than is
539 * required to contain the copied data, or
540 * * -%EFAULT if local memory could not be copied to userspace.
541 */
542static int
543pvr_dev_query_enhancements_get(struct pvr_device *pvr_dev,
544 struct drm_pvr_ioctl_dev_query_args *args)
545{
546 /*
547 * @FIXME - hardcoding of numbers here is intended as an
548 * intermediate step so the UAPI can be fixed, but requires a
549 * a refactor in the future to store them in a more appropriate
550 * location
551 */
552 const u32 umd_enhancements[] = {
553 35421,
554 42064,
555 };
556 struct drm_pvr_dev_query_enhancements query;
557 u32 out[ARRAY_SIZE(umd_enhancements)];
558 size_t out_idx = 0;
559 int err;
560
561 if (!args->pointer) {
562 args->size = sizeof(struct drm_pvr_dev_query_enhancements);
563 return 0;
564 }
565
566 err = PVR_UOBJ_GET(query, args->size, args->pointer);
567
568 if (err < 0)
569 return err;
570 if (query._padding_a)
571 return -EINVAL;
572 if (query._padding_c)
573 return -EINVAL;
574
575 for (int i = 0; i < ARRAY_SIZE(umd_enhancements); i++) {
576 if (pvr_device_has_uapi_enhancement(pvr_dev, umd_enhancements[i]))
577 out[out_idx++] = umd_enhancements[i];
578 }
579
580 if (!query.enhancements)
581 goto copy_out;
582 if (query.count < out_idx)
583 return -E2BIG;
584
585 if (copy_to_user(u64_to_user_ptr(query.enhancements), out,
586 out_idx * sizeof(u32))) {
587 return -EFAULT;
588 }
589
590copy_out:
591 query.count = out_idx;
592 err = PVR_UOBJ_SET(args->pointer, args->size, query);
593 if (err < 0)
594 return err;
595
596 args->size = sizeof(query);
597 return 0;
598}
599
600/**
601 * pvr_ioctl_dev_query() - IOCTL to copy information about a device
602 * @drm_dev: [IN] DRM device.
603 * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
604 * &struct drm_pvr_ioctl_dev_query_args.
605 * @file: [IN] DRM file private data.
606 *
607 * Called from userspace with %DRM_IOCTL_PVR_DEV_QUERY.
608 * If the given receiving struct pointer is NULL, or the indicated size is too
609 * small, the expected size of the struct type will be returned in the size
610 * argument field.
611 *
612 * Return:
613 * * 0 on success or when fetching the size with args->pointer == NULL, or
614 * * -%E2BIG if the indicated size of the receiving struct is less than is
615 * required to contain the copied data, or
616 * * -%EINVAL if the indicated struct type is unknown, or
617 * * -%ENOMEM if local memory could not be allocated, or
618 * * -%EFAULT if local memory could not be copied to userspace.
619 */
620static int
621pvr_ioctl_dev_query(struct drm_device *drm_dev, void *raw_args,
622 struct drm_file *file)
623{
624 struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
625 struct drm_pvr_ioctl_dev_query_args *args = raw_args;
626 int idx;
627 int ret = -EINVAL;
628
629 if (!drm_dev_enter(drm_dev, &idx))
630 return -EIO;
631
632 switch ((enum drm_pvr_dev_query)args->type) {
633 case DRM_PVR_DEV_QUERY_GPU_INFO_GET:
634 ret = pvr_dev_query_gpu_info_get(pvr_dev, args);
635 break;
636
637 case DRM_PVR_DEV_QUERY_RUNTIME_INFO_GET:
638 ret = pvr_dev_query_runtime_info_get(pvr_dev, args);
639 break;
640
641 case DRM_PVR_DEV_QUERY_QUIRKS_GET:
642 ret = pvr_dev_query_quirks_get(pvr_dev, args);
643 break;
644
645 case DRM_PVR_DEV_QUERY_ENHANCEMENTS_GET:
646 ret = pvr_dev_query_enhancements_get(pvr_dev, args);
647 break;
648
649 case DRM_PVR_DEV_QUERY_HEAP_INFO_GET:
650 ret = pvr_heap_info_get(pvr_dev, args);
651 break;
652
653 case DRM_PVR_DEV_QUERY_STATIC_DATA_AREAS_GET:
654 ret = pvr_static_data_areas_get(pvr_dev, args);
655 break;
656 }
657
658 drm_dev_exit(idx);
659
660 return ret;
661}
662
663/**
664 * pvr_ioctl_create_context() - IOCTL to create a context
665 * @drm_dev: [IN] DRM device.
666 * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
667 * &struct drm_pvr_ioctl_create_context_args.
668 * @file: [IN] DRM file private data.
669 *
670 * Called from userspace with %DRM_IOCTL_PVR_CREATE_CONTEXT.
671 *
672 * Return:
673 * * 0 on success, or
674 * * -%EINVAL if provided arguments are invalid, or
675 * * -%EFAULT if arguments can't be copied from userspace, or
676 * * Any error returned by pvr_create_render_context().
677 */
678static int
679pvr_ioctl_create_context(struct drm_device *drm_dev, void *raw_args,
680 struct drm_file *file)
681{
682 struct drm_pvr_ioctl_create_context_args *args = raw_args;
683 struct pvr_file *pvr_file = file->driver_priv;
684 int idx;
685 int ret;
686
687 if (!drm_dev_enter(drm_dev, &idx))
688 return -EIO;
689
690 ret = pvr_context_create(pvr_file, args);
691
692 drm_dev_exit(idx);
693
694 return ret;
695}
696
697/**
698 * pvr_ioctl_destroy_context() - IOCTL to destroy a context
699 * @drm_dev: [IN] DRM device.
700 * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
701 * &struct drm_pvr_ioctl_destroy_context_args.
702 * @file: [IN] DRM file private data.
703 *
704 * Called from userspace with %DRM_IOCTL_PVR_DESTROY_CONTEXT.
705 *
706 * Return:
707 * * 0 on success, or
708 * * -%EINVAL if context not in context list.
709 */
710static int
711pvr_ioctl_destroy_context(struct drm_device *drm_dev, void *raw_args,
712 struct drm_file *file)
713{
714 struct drm_pvr_ioctl_destroy_context_args *args = raw_args;
715 struct pvr_file *pvr_file = file->driver_priv;
716
717 if (args->_padding_4)
718 return -EINVAL;
719
720 return pvr_context_destroy(pvr_file, args->handle);
721}
722
723/**
724 * pvr_ioctl_create_free_list() - IOCTL to create a free list
725 * @drm_dev: [IN] DRM device.
726 * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
727 * &struct drm_pvr_ioctl_create_free_list_args.
728 * @file: [IN] DRM file private data.
729 *
730 * Called from userspace with %DRM_IOCTL_PVR_CREATE_FREE_LIST.
731 *
732 * Return:
733 * * 0 on success, or
734 * * Any error returned by pvr_free_list_create().
735 */
736static int
737pvr_ioctl_create_free_list(struct drm_device *drm_dev, void *raw_args,
738 struct drm_file *file)
739{
740 struct drm_pvr_ioctl_create_free_list_args *args = raw_args;
741 struct pvr_file *pvr_file = to_pvr_file(file);
742 struct pvr_free_list *free_list;
743 int idx;
744 int err;
745
746 if (!drm_dev_enter(drm_dev, &idx))
747 return -EIO;
748
749 free_list = pvr_free_list_create(pvr_file, args);
750 if (IS_ERR(free_list)) {
751 err = PTR_ERR(free_list);
752 goto err_drm_dev_exit;
753 }
754
755 /* Allocate object handle for userspace. */
756 err = xa_alloc(&pvr_file->free_list_handles,
757 &args->handle,
758 free_list,
759 xa_limit_32b,
760 GFP_KERNEL);
761 if (err < 0)
762 goto err_cleanup;
763
764 drm_dev_exit(idx);
765
766 return 0;
767
768err_cleanup:
769 pvr_free_list_put(free_list);
770
771err_drm_dev_exit:
772 drm_dev_exit(idx);
773
774 return err;
775}
776
777/**
778 * pvr_ioctl_destroy_free_list() - IOCTL to destroy a free list
779 * @drm_dev: [IN] DRM device.
780 * @raw_args: [IN] Arguments passed to this IOCTL. This must be of type
781 * &struct drm_pvr_ioctl_destroy_free_list_args.
782 * @file: [IN] DRM file private data.
783 *
784 * Called from userspace with %DRM_IOCTL_PVR_DESTROY_FREE_LIST.
785 *
786 * Return:
787 * * 0 on success, or
788 * * -%EINVAL if free list not in object list.
789 */
790static int
791pvr_ioctl_destroy_free_list(struct drm_device *drm_dev, void *raw_args,
792 struct drm_file *file)
793{
794 struct drm_pvr_ioctl_destroy_free_list_args *args = raw_args;
795 struct pvr_file *pvr_file = to_pvr_file(file);
796 struct pvr_free_list *free_list;
797
798 if (args->_padding_4)
799 return -EINVAL;
800
801 free_list = xa_erase(&pvr_file->free_list_handles, args->handle);
802 if (!free_list)
803 return -EINVAL;
804
805 pvr_free_list_put(free_list);
806 return 0;
807}
808
809/**
810 * pvr_ioctl_create_hwrt_dataset() - IOCTL to create a HWRT dataset
811 * @drm_dev: [IN] DRM device.
812 * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
813 * &struct drm_pvr_ioctl_create_hwrt_dataset_args.
814 * @file: [IN] DRM file private data.
815 *
816 * Called from userspace with %DRM_IOCTL_PVR_CREATE_HWRT_DATASET.
817 *
818 * Return:
819 * * 0 on success, or
820 * * Any error returned by pvr_hwrt_dataset_create().
821 */
822static int
823pvr_ioctl_create_hwrt_dataset(struct drm_device *drm_dev, void *raw_args,
824 struct drm_file *file)
825{
826 struct drm_pvr_ioctl_create_hwrt_dataset_args *args = raw_args;
827 struct pvr_file *pvr_file = to_pvr_file(file);
828 struct pvr_hwrt_dataset *hwrt;
829 int idx;
830 int err;
831
832 if (!drm_dev_enter(drm_dev, &idx))
833 return -EIO;
834
835 hwrt = pvr_hwrt_dataset_create(pvr_file, args);
836 if (IS_ERR(hwrt)) {
837 err = PTR_ERR(hwrt);
838 goto err_drm_dev_exit;
839 }
840
841 /* Allocate object handle for userspace. */
842 err = xa_alloc(&pvr_file->hwrt_handles,
843 &args->handle,
844 hwrt,
845 xa_limit_32b,
846 GFP_KERNEL);
847 if (err < 0)
848 goto err_cleanup;
849
850 drm_dev_exit(idx);
851
852 return 0;
853
854err_cleanup:
855 pvr_hwrt_dataset_put(hwrt);
856
857err_drm_dev_exit:
858 drm_dev_exit(idx);
859
860 return err;
861}
862
863/**
864 * pvr_ioctl_destroy_hwrt_dataset() - IOCTL to destroy a HWRT dataset
865 * @drm_dev: [IN] DRM device.
866 * @raw_args: [IN] Arguments passed to this IOCTL. This must be of type
867 * &struct drm_pvr_ioctl_destroy_hwrt_dataset_args.
868 * @file: [IN] DRM file private data.
869 *
870 * Called from userspace with %DRM_IOCTL_PVR_DESTROY_HWRT_DATASET.
871 *
872 * Return:
873 * * 0 on success, or
874 * * -%EINVAL if HWRT dataset not in object list.
875 */
876static int
877pvr_ioctl_destroy_hwrt_dataset(struct drm_device *drm_dev, void *raw_args,
878 struct drm_file *file)
879{
880 struct drm_pvr_ioctl_destroy_hwrt_dataset_args *args = raw_args;
881 struct pvr_file *pvr_file = to_pvr_file(file);
882 struct pvr_hwrt_dataset *hwrt;
883
884 if (args->_padding_4)
885 return -EINVAL;
886
887 hwrt = xa_erase(&pvr_file->hwrt_handles, args->handle);
888 if (!hwrt)
889 return -EINVAL;
890
891 pvr_hwrt_dataset_put(hwrt);
892 return 0;
893}
894
895/**
896 * pvr_ioctl_create_vm_context() - IOCTL to create a VM context
897 * @drm_dev: [IN] DRM device.
898 * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
899 * &struct drm_pvr_ioctl_create_vm_context_args.
900 * @file: [IN] DRM file private data.
901 *
902 * Called from userspace with %DRM_IOCTL_PVR_CREATE_VM_CONTEXT.
903 *
904 * Return:
905 * * 0 on success, or
906 * * Any error returned by pvr_vm_create_context().
907 */
908static int
909pvr_ioctl_create_vm_context(struct drm_device *drm_dev, void *raw_args,
910 struct drm_file *file)
911{
912 struct drm_pvr_ioctl_create_vm_context_args *args = raw_args;
913 struct pvr_file *pvr_file = to_pvr_file(file);
914 struct pvr_vm_context *vm_ctx;
915 int idx;
916 int err;
917
918 if (!drm_dev_enter(drm_dev, &idx))
919 return -EIO;
920
921 if (args->_padding_4) {
922 err = -EINVAL;
923 goto err_drm_dev_exit;
924 }
925
926 vm_ctx = pvr_vm_create_context(pvr_file->pvr_dev, true);
927 if (IS_ERR(vm_ctx)) {
928 err = PTR_ERR(vm_ctx);
929 goto err_drm_dev_exit;
930 }
931
932 /* Allocate object handle for userspace. */
933 err = xa_alloc(&pvr_file->vm_ctx_handles,
934 &args->handle,
935 vm_ctx,
936 xa_limit_32b,
937 GFP_KERNEL);
938 if (err < 0)
939 goto err_cleanup;
940
941 drm_dev_exit(idx);
942
943 return 0;
944
945err_cleanup:
946 pvr_vm_context_put(vm_ctx);
947
948err_drm_dev_exit:
949 drm_dev_exit(idx);
950
951 return err;
952}
953
954/**
955 * pvr_ioctl_destroy_vm_context() - IOCTL to destroy a VM context
956* @drm_dev: [IN] DRM device.
957* @raw_args: [IN] Arguments passed to this IOCTL. This must be of type
958* &struct drm_pvr_ioctl_destroy_vm_context_args.
959* @file: [IN] DRM file private data.
960*
961* Called from userspace with %DRM_IOCTL_PVR_DESTROY_VM_CONTEXT.
962*
963* Return:
964* * 0 on success, or
965* * -%EINVAL if object not in object list.
966 */
967static int
968pvr_ioctl_destroy_vm_context(struct drm_device *drm_dev, void *raw_args,
969 struct drm_file *file)
970{
971 struct drm_pvr_ioctl_destroy_vm_context_args *args = raw_args;
972 struct pvr_file *pvr_file = to_pvr_file(file);
973 struct pvr_vm_context *vm_ctx;
974
975 if (args->_padding_4)
976 return -EINVAL;
977
978 vm_ctx = xa_erase(&pvr_file->vm_ctx_handles, args->handle);
979 if (!vm_ctx)
980 return -EINVAL;
981
982 pvr_vm_context_put(vm_ctx);
983 return 0;
984}
985
986/**
987 * pvr_ioctl_vm_map() - IOCTL to map buffer to GPU address space.
988 * @drm_dev: [IN] DRM device.
989 * @raw_args: [IN] Arguments passed to this IOCTL. This must be of type
990 * &struct drm_pvr_ioctl_vm_map_args.
991 * @file: [IN] DRM file private data.
992 *
993 * Called from userspace with %DRM_IOCTL_PVR_VM_MAP.
994 *
995 * Return:
996 * * 0 on success,
997 * * -%EINVAL if &drm_pvr_ioctl_vm_op_map_args.flags is not zero,
998 * * -%EINVAL if the bounds specified by &drm_pvr_ioctl_vm_op_map_args.offset
999 * and &drm_pvr_ioctl_vm_op_map_args.size are not valid or do not fall
1000 * within the buffer object specified by
1001 * &drm_pvr_ioctl_vm_op_map_args.handle,
1002 * * -%EINVAL if the bounds specified by
1003 * &drm_pvr_ioctl_vm_op_map_args.device_addr and
1004 * &drm_pvr_ioctl_vm_op_map_args.size do not form a valid device-virtual
1005 * address range which falls entirely within a single heap, or
1006 * * -%ENOENT if &drm_pvr_ioctl_vm_op_map_args.handle does not refer to a
1007 * valid PowerVR buffer object.
1008 */
1009static int
1010pvr_ioctl_vm_map(struct drm_device *drm_dev, void *raw_args,
1011 struct drm_file *file)
1012{
1013 struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
1014 struct drm_pvr_ioctl_vm_map_args *args = raw_args;
1015 struct pvr_file *pvr_file = to_pvr_file(file);
1016 struct pvr_vm_context *vm_ctx;
1017
1018 struct pvr_gem_object *pvr_obj;
1019 size_t pvr_obj_size;
1020
1021 u64 offset_plus_size;
1022 int idx;
1023 int err;
1024
1025 if (!drm_dev_enter(drm_dev, &idx))
1026 return -EIO;
1027
1028 /* Initial validation of args. */
1029 if (args->_padding_14) {
1030 err = -EINVAL;
1031 goto err_drm_dev_exit;
1032 }
1033
1034 if (args->flags != 0 ||
1035 check_add_overflow(args->offset, args->size, &offset_plus_size) ||
1036 !pvr_find_heap_containing(pvr_dev, args->device_addr, args->size)) {
1037 err = -EINVAL;
1038 goto err_drm_dev_exit;
1039 }
1040
1041 vm_ctx = pvr_vm_context_lookup(pvr_file, args->vm_context_handle);
1042 if (!vm_ctx) {
1043 err = -EINVAL;
1044 goto err_drm_dev_exit;
1045 }
1046
1047 pvr_obj = pvr_gem_object_from_handle(pvr_file, args->handle);
1048 if (!pvr_obj) {
1049 err = -ENOENT;
1050 goto err_put_vm_context;
1051 }
1052
1053 pvr_obj_size = pvr_gem_object_size(pvr_obj);
1054
1055 /*
1056 * Validate offset and size args. The alignment of these will be
1057 * checked when mapping; for now just check that they're within valid
1058 * bounds
1059 */
1060 if (args->offset >= pvr_obj_size || offset_plus_size > pvr_obj_size) {
1061 err = -EINVAL;
1062 goto err_put_pvr_object;
1063 }
1064
1065 err = pvr_vm_map(vm_ctx, pvr_obj, args->offset,
1066 args->device_addr, args->size);
1067 if (err)
1068 goto err_put_pvr_object;
1069
1070 /*
1071 * In order to set up the mapping, we needed a reference to &pvr_obj.
1072 * However, pvr_vm_map() obtains and stores its own reference, so we
1073 * must release ours before returning.
1074 */
1075
1076err_put_pvr_object:
1077 pvr_gem_object_put(pvr_obj);
1078
1079err_put_vm_context:
1080 pvr_vm_context_put(vm_ctx);
1081
1082err_drm_dev_exit:
1083 drm_dev_exit(idx);
1084
1085 return err;
1086}
1087
1088/**
1089 * pvr_ioctl_vm_unmap() - IOCTL to unmap buffer from GPU address space.
1090 * @drm_dev: [IN] DRM device.
1091 * @raw_args: [IN] Arguments passed to this IOCTL. This must be of type
1092 * &struct drm_pvr_ioctl_vm_unmap_args.
1093 * @file: [IN] DRM file private data.
1094 *
1095 * Called from userspace with %DRM_IOCTL_PVR_VM_UNMAP.
1096 *
1097 * Return:
1098 * * 0 on success,
1099 * * -%EINVAL if &drm_pvr_ioctl_vm_op_unmap_args.device_addr is not a valid
1100 * device page-aligned device-virtual address, or
1101 * * -%ENOENT if there is currently no PowerVR buffer object mapped at
1102 * &drm_pvr_ioctl_vm_op_unmap_args.device_addr.
1103 */
1104static int
1105pvr_ioctl_vm_unmap(struct drm_device *drm_dev, void *raw_args,
1106 struct drm_file *file)
1107{
1108 struct drm_pvr_ioctl_vm_unmap_args *args = raw_args;
1109 struct pvr_file *pvr_file = to_pvr_file(file);
1110 struct pvr_vm_context *vm_ctx;
1111 int err;
1112
1113 /* Initial validation of args. */
1114 if (args->_padding_4)
1115 return -EINVAL;
1116
1117 vm_ctx = pvr_vm_context_lookup(pvr_file, args->vm_context_handle);
1118 if (!vm_ctx)
1119 return -EINVAL;
1120
1121 err = pvr_vm_unmap(vm_ctx, args->device_addr, args->size);
1122
1123 pvr_vm_context_put(vm_ctx);
1124
1125 return err;
1126}
1127
1128/*
1129 * pvr_ioctl_submit_job() - IOCTL to submit a job to the GPU
1130 * @drm_dev: [IN] DRM device.
1131 * @raw_args: [IN] Arguments passed to this IOCTL. This must be of type
1132 * &struct drm_pvr_ioctl_submit_job_args.
1133 * @file: [IN] DRM file private data.
1134 *
1135 * Called from userspace with %DRM_IOCTL_PVR_SUBMIT_JOB.
1136 *
1137 * Return:
1138 * * 0 on success, or
1139 * * -%EINVAL if arguments are invalid.
1140 */
1141static int
1142pvr_ioctl_submit_jobs(struct drm_device *drm_dev, void *raw_args,
1143 struct drm_file *file)
1144{
1145 struct drm_pvr_ioctl_submit_jobs_args *args = raw_args;
1146 struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
1147 struct pvr_file *pvr_file = to_pvr_file(file);
1148 int idx;
1149 int err;
1150
1151 if (!drm_dev_enter(drm_dev, &idx))
1152 return -EIO;
1153
1154 err = pvr_submit_jobs(pvr_dev, pvr_file, args);
1155
1156 drm_dev_exit(idx);
1157
1158 return err;
1159}
1160
1161int
1162pvr_get_uobj(u64 usr_ptr, u32 usr_stride, u32 min_stride, u32 obj_size, void *out)
1163{
1164 if (usr_stride < min_stride)
1165 return -EINVAL;
1166
1167 return copy_struct_from_user(out, obj_size, u64_to_user_ptr(usr_ptr), usr_stride);
1168}
1169
1170int
1171pvr_set_uobj(u64 usr_ptr, u32 usr_stride, u32 min_stride, u32 obj_size, const void *in)
1172{
1173 if (usr_stride < min_stride)
1174 return -EINVAL;
1175
1176 if (copy_to_user(u64_to_user_ptr(usr_ptr), in, min_t(u32, usr_stride, obj_size)))
1177 return -EFAULT;
1178
1179 if (usr_stride > obj_size &&
1180 clear_user(u64_to_user_ptr(usr_ptr + obj_size), usr_stride - obj_size)) {
1181 return -EFAULT;
1182 }
1183
1184 return 0;
1185}
1186
1187int
1188pvr_get_uobj_array(const struct drm_pvr_obj_array *in, u32 min_stride, u32 obj_size, void **out)
1189{
1190 int ret = 0;
1191 void *out_alloc;
1192
1193 if (in->stride < min_stride)
1194 return -EINVAL;
1195
1196 if (!in->count)
1197 return 0;
1198
1199 out_alloc = kvmalloc_array(in->count, obj_size, GFP_KERNEL);
1200 if (!out_alloc)
1201 return -ENOMEM;
1202
1203 if (obj_size == in->stride) {
1204 if (copy_from_user(out_alloc, u64_to_user_ptr(in->array),
1205 (unsigned long)obj_size * in->count))
1206 ret = -EFAULT;
1207 } else {
1208 void __user *in_ptr = u64_to_user_ptr(in->array);
1209 void *out_ptr = out_alloc;
1210
1211 for (u32 i = 0; i < in->count; i++) {
1212 ret = copy_struct_from_user(out_ptr, obj_size, in_ptr, in->stride);
1213 if (ret)
1214 break;
1215
1216 out_ptr += obj_size;
1217 in_ptr += in->stride;
1218 }
1219 }
1220
1221 if (ret) {
1222 kvfree(out_alloc);
1223 return ret;
1224 }
1225
1226 *out = out_alloc;
1227 return 0;
1228}
1229
1230int
1231pvr_set_uobj_array(const struct drm_pvr_obj_array *out, u32 min_stride, u32 obj_size,
1232 const void *in)
1233{
1234 if (out->stride < min_stride)
1235 return -EINVAL;
1236
1237 if (!out->count)
1238 return 0;
1239
1240 if (obj_size == out->stride) {
1241 if (copy_to_user(u64_to_user_ptr(out->array), in,
1242 (unsigned long)obj_size * out->count))
1243 return -EFAULT;
1244 } else {
1245 u32 cpy_elem_size = min_t(u32, out->stride, obj_size);
1246 void __user *out_ptr = u64_to_user_ptr(out->array);
1247 const void *in_ptr = in;
1248
1249 for (u32 i = 0; i < out->count; i++) {
1250 if (copy_to_user(out_ptr, in_ptr, cpy_elem_size))
1251 return -EFAULT;
1252
1253 out_ptr += obj_size;
1254 in_ptr += out->stride;
1255 }
1256
1257 if (out->stride > obj_size &&
1258 clear_user(u64_to_user_ptr(out->array + obj_size),
1259 out->stride - obj_size)) {
1260 return -EFAULT;
1261 }
1262 }
1263
1264 return 0;
1265}
1266
1267#define DRM_PVR_IOCTL(_name, _func, _flags) \
1268 DRM_IOCTL_DEF_DRV(PVR_##_name, pvr_ioctl_##_func, _flags)
1269
1270/* clang-format off */
1271
1272static const struct drm_ioctl_desc pvr_drm_driver_ioctls[] = {
1273 DRM_PVR_IOCTL(DEV_QUERY, dev_query, DRM_RENDER_ALLOW),
1274 DRM_PVR_IOCTL(CREATE_BO, create_bo, DRM_RENDER_ALLOW),
1275 DRM_PVR_IOCTL(GET_BO_MMAP_OFFSET, get_bo_mmap_offset, DRM_RENDER_ALLOW),
1276 DRM_PVR_IOCTL(CREATE_VM_CONTEXT, create_vm_context, DRM_RENDER_ALLOW),
1277 DRM_PVR_IOCTL(DESTROY_VM_CONTEXT, destroy_vm_context, DRM_RENDER_ALLOW),
1278 DRM_PVR_IOCTL(VM_MAP, vm_map, DRM_RENDER_ALLOW),
1279 DRM_PVR_IOCTL(VM_UNMAP, vm_unmap, DRM_RENDER_ALLOW),
1280 DRM_PVR_IOCTL(CREATE_CONTEXT, create_context, DRM_RENDER_ALLOW),
1281 DRM_PVR_IOCTL(DESTROY_CONTEXT, destroy_context, DRM_RENDER_ALLOW),
1282 DRM_PVR_IOCTL(CREATE_FREE_LIST, create_free_list, DRM_RENDER_ALLOW),
1283 DRM_PVR_IOCTL(DESTROY_FREE_LIST, destroy_free_list, DRM_RENDER_ALLOW),
1284 DRM_PVR_IOCTL(CREATE_HWRT_DATASET, create_hwrt_dataset, DRM_RENDER_ALLOW),
1285 DRM_PVR_IOCTL(DESTROY_HWRT_DATASET, destroy_hwrt_dataset, DRM_RENDER_ALLOW),
1286 DRM_PVR_IOCTL(SUBMIT_JOBS, submit_jobs, DRM_RENDER_ALLOW),
1287};
1288
1289/* clang-format on */
1290
1291#undef DRM_PVR_IOCTL
1292
1293/**
1294 * pvr_drm_driver_open() - Driver callback when a new &struct drm_file is opened
1295 * @drm_dev: [IN] DRM device.
1296 * @file: [IN] DRM file private data.
1297 *
1298 * Allocates powervr-specific file private data (&struct pvr_file).
1299 *
1300 * Registered in &pvr_drm_driver.
1301 *
1302 * Return:
1303 * * 0 on success,
1304 * * -%ENOMEM if the allocation of a &struct ipvr_file fails, or
1305 * * Any error returned by pvr_memory_context_init().
1306 */
1307static int
1308pvr_drm_driver_open(struct drm_device *drm_dev, struct drm_file *file)
1309{
1310 struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
1311 struct pvr_file *pvr_file;
1312
1313 pvr_file = kzalloc(sizeof(*pvr_file), GFP_KERNEL);
1314 if (!pvr_file)
1315 return -ENOMEM;
1316
1317 /*
1318 * Store reference to base DRM file private data for use by
1319 * from_pvr_file.
1320 */
1321 pvr_file->file = file;
1322
1323 /*
1324 * Store reference to powervr-specific outer device struct in file
1325 * private data for convenient access.
1326 */
1327 pvr_file->pvr_dev = pvr_dev;
1328
1329 xa_init_flags(&pvr_file->ctx_handles, XA_FLAGS_ALLOC1);
1330 xa_init_flags(&pvr_file->free_list_handles, XA_FLAGS_ALLOC1);
1331 xa_init_flags(&pvr_file->hwrt_handles, XA_FLAGS_ALLOC1);
1332 xa_init_flags(&pvr_file->vm_ctx_handles, XA_FLAGS_ALLOC1);
1333
1334 /*
1335 * Store reference to powervr-specific file private data in DRM file
1336 * private data.
1337 */
1338 file->driver_priv = pvr_file;
1339
1340 return 0;
1341}
1342
1343/**
1344 * pvr_drm_driver_postclose() - One of the driver callbacks when a &struct
1345 * drm_file is closed.
1346 * @drm_dev: [IN] DRM device (unused).
1347 * @file: [IN] DRM file private data.
1348 *
1349 * Frees powervr-specific file private data (&struct pvr_file).
1350 *
1351 * Registered in &pvr_drm_driver.
1352 */
1353static void
1354pvr_drm_driver_postclose(__always_unused struct drm_device *drm_dev,
1355 struct drm_file *file)
1356{
1357 struct pvr_file *pvr_file = to_pvr_file(file);
1358
1359 /* Kill remaining contexts. */
1360 pvr_destroy_contexts_for_file(pvr_file);
1361
1362 /* Drop references on any remaining objects. */
1363 pvr_destroy_free_lists_for_file(pvr_file);
1364 pvr_destroy_hwrt_datasets_for_file(pvr_file);
1365 pvr_destroy_vm_contexts_for_file(pvr_file);
1366
1367 kfree(pvr_file);
1368 file->driver_priv = NULL;
1369}
1370
1371DEFINE_DRM_GEM_FOPS(pvr_drm_driver_fops);
1372
1373static struct drm_driver pvr_drm_driver = {
1374 .driver_features = DRIVER_GEM | DRIVER_GEM_GPUVA | DRIVER_RENDER |
1375 DRIVER_SYNCOBJ | DRIVER_SYNCOBJ_TIMELINE,
1376 .open = pvr_drm_driver_open,
1377 .postclose = pvr_drm_driver_postclose,
1378 .ioctls = pvr_drm_driver_ioctls,
1379 .num_ioctls = ARRAY_SIZE(pvr_drm_driver_ioctls),
1380 .fops = &pvr_drm_driver_fops,
1381#if defined(CONFIG_DEBUG_FS)
1382 .debugfs_init = pvr_debugfs_init,
1383#endif
1384
1385 .name = PVR_DRIVER_NAME,
1386 .desc = PVR_DRIVER_DESC,
1387 .date = PVR_DRIVER_DATE,
1388 .major = PVR_DRIVER_MAJOR,
1389 .minor = PVR_DRIVER_MINOR,
1390 .patchlevel = PVR_DRIVER_PATCHLEVEL,
1391
1392 .gem_prime_import_sg_table = drm_gem_shmem_prime_import_sg_table,
1393 .gem_create_object = pvr_gem_create_object,
1394};
1395
1396static int
1397pvr_probe(struct platform_device *plat_dev)
1398{
1399 struct pvr_device *pvr_dev;
1400 struct drm_device *drm_dev;
1401 int err;
1402
1403 pvr_dev = devm_drm_dev_alloc(&plat_dev->dev, &pvr_drm_driver,
1404 struct pvr_device, base);
1405 if (IS_ERR(pvr_dev))
1406 return PTR_ERR(pvr_dev);
1407
1408 drm_dev = &pvr_dev->base;
1409
1410 platform_set_drvdata(plat_dev, drm_dev);
1411
1412 init_rwsem(&pvr_dev->reset_sem);
1413
1414 pvr_context_device_init(pvr_dev);
1415
1416 err = pvr_queue_device_init(pvr_dev);
1417 if (err)
1418 goto err_context_fini;
1419
1420 devm_pm_runtime_enable(&plat_dev->dev);
1421 pm_runtime_mark_last_busy(&plat_dev->dev);
1422
1423 pm_runtime_set_autosuspend_delay(&plat_dev->dev, 50);
1424 pm_runtime_use_autosuspend(&plat_dev->dev);
1425 pvr_watchdog_init(pvr_dev);
1426
1427 err = pvr_device_init(pvr_dev);
1428 if (err)
1429 goto err_watchdog_fini;
1430
1431 err = drm_dev_register(drm_dev, 0);
1432 if (err)
1433 goto err_device_fini;
1434
1435 xa_init_flags(&pvr_dev->free_list_ids, XA_FLAGS_ALLOC1);
1436 xa_init_flags(&pvr_dev->job_ids, XA_FLAGS_ALLOC1);
1437
1438 return 0;
1439
1440err_device_fini:
1441 pvr_device_fini(pvr_dev);
1442
1443err_watchdog_fini:
1444 pvr_watchdog_fini(pvr_dev);
1445
1446 pvr_queue_device_fini(pvr_dev);
1447
1448err_context_fini:
1449 pvr_context_device_fini(pvr_dev);
1450
1451 return err;
1452}
1453
1454static int
1455pvr_remove(struct platform_device *plat_dev)
1456{
1457 struct drm_device *drm_dev = platform_get_drvdata(plat_dev);
1458 struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
1459
1460 WARN_ON(!xa_empty(&pvr_dev->job_ids));
1461 WARN_ON(!xa_empty(&pvr_dev->free_list_ids));
1462
1463 xa_destroy(&pvr_dev->job_ids);
1464 xa_destroy(&pvr_dev->free_list_ids);
1465
1466 pm_runtime_suspend(drm_dev->dev);
1467 pvr_device_fini(pvr_dev);
1468 drm_dev_unplug(drm_dev);
1469 pvr_watchdog_fini(pvr_dev);
1470 pvr_queue_device_fini(pvr_dev);
1471 pvr_context_device_fini(pvr_dev);
1472
1473 return 0;
1474}
1475
1476static const struct of_device_id dt_match[] = {
1477 { .compatible = "img,img-axe", .data = NULL },
1478 {}
1479};
1480MODULE_DEVICE_TABLE(of, dt_match);
1481
1482static const struct dev_pm_ops pvr_pm_ops = {
1483 RUNTIME_PM_OPS(pvr_power_device_suspend, pvr_power_device_resume, pvr_power_device_idle)
1484};
1485
1486static struct platform_driver pvr_driver = {
1487 .probe = pvr_probe,
1488 .remove = pvr_remove,
1489 .driver = {
1490 .name = PVR_DRIVER_NAME,
1491 .pm = &pvr_pm_ops,
1492 .of_match_table = dt_match,
1493 },
1494};
1495module_platform_driver(pvr_driver);
1496
1497MODULE_AUTHOR("Imagination Technologies Ltd.");
1498MODULE_DESCRIPTION(PVR_DRIVER_DESC);
1499MODULE_LICENSE("Dual MIT/GPL");
1500MODULE_IMPORT_NS(DMA_BUF);
1501MODULE_FIRMWARE("powervr/rogue_33.15.11.3_v1.fw");
drivers/gpu/drm/imagination/pvr_drv.h+129−0
@@ -0,0 +1,129 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_DRV_H
5#define PVR_DRV_H
6
7#include "linux/compiler_attributes.h"
8#include <uapi/drm/pvr_drm.h>
9
10#define PVR_DRIVER_NAME "powervr"
11#define PVR_DRIVER_DESC "Imagination PowerVR (Series 6 and later) & IMG Graphics"
12#define PVR_DRIVER_DATE "20230904"
13
14/*
15 * Driver interface version:
16 * - 1.0: Initial interface
17 */
18#define PVR_DRIVER_MAJOR 1
19#define PVR_DRIVER_MINOR 0
20#define PVR_DRIVER_PATCHLEVEL 0
21
22int pvr_get_uobj(u64 usr_ptr, u32 usr_size, u32 min_size, u32 obj_size, void *out);
23int pvr_set_uobj(u64 usr_ptr, u32 usr_size, u32 min_size, u32 obj_size, const void *in);
24int pvr_get_uobj_array(const struct drm_pvr_obj_array *in, u32 min_stride, u32 obj_size,
25 void **out);
26int pvr_set_uobj_array(const struct drm_pvr_obj_array *out, u32 min_stride, u32 obj_size,
27 const void *in);
28
29#define PVR_UOBJ_MIN_SIZE_INTERNAL(_typename, _last_mandatory_field) \
30 (offsetof(_typename, _last_mandatory_field) + \
31 sizeof(((_typename *)NULL)->_last_mandatory_field))
32
33/* NOLINTBEGIN(bugprone-macro-parentheses) */
34#define PVR_UOBJ_DECL(_typename, _last_mandatory_field) \
35 , _typename : PVR_UOBJ_MIN_SIZE_INTERNAL(_typename, _last_mandatory_field)
36/* NOLINTEND(bugprone-macro-parentheses) */
37
38/**
39 * DOC: PVR user objects.
40 *
41 * Macros used to aid copying structured and array data to and from
42 * userspace. Objects can differ in size, provided the minimum size
43 * allowed is specified (using the last mandatory field in the struct).
44 * All types used with PVR_UOBJ_GET/SET macros must be listed here under
45 * PVR_UOBJ_MIN_SIZE, with the last mandatory struct field specified.
46 */
47
48/**
49 * PVR_UOBJ_MIN_SIZE() - Fetch the minimum copy size of a compatible type object.
50 * @_obj_name: The name of the object. Cannot be a typename - this is deduced.
51 *
52 * This cannot fail. Using the macro with an incompatible type will result in a
53 * compiler error.
54 *
55 * To add compatibility for a type, list it within the macro in an orderly
56 * fashion. The second argument is the name of the last mandatory field of the
57 * struct type, which is used to calculate the size. See also PVR_UOBJ_DECL().
58 *
59 * Return: The minimum copy size.
60 */
61#define PVR_UOBJ_MIN_SIZE(_obj_name) _Generic(_obj_name \
62 PVR_UOBJ_DECL(struct drm_pvr_job, hwrt) \
63 PVR_UOBJ_DECL(struct drm_pvr_sync_op, value) \
64 PVR_UOBJ_DECL(struct drm_pvr_dev_query_gpu_info, num_phantoms) \
65 PVR_UOBJ_DECL(struct drm_pvr_dev_query_runtime_info, cdm_max_local_mem_size_regs) \
66 PVR_UOBJ_DECL(struct drm_pvr_dev_query_quirks, _padding_c) \
67 PVR_UOBJ_DECL(struct drm_pvr_dev_query_enhancements, _padding_c) \
68 PVR_UOBJ_DECL(struct drm_pvr_heap, page_size_log2) \
69 PVR_UOBJ_DECL(struct drm_pvr_dev_query_heap_info, heaps) \
70 PVR_UOBJ_DECL(struct drm_pvr_static_data_area, offset) \
71 PVR_UOBJ_DECL(struct drm_pvr_dev_query_static_data_areas, static_data_areas) \
72 )
73
74/**
75 * PVR_UOBJ_GET() - Copies from _src_usr_ptr to &_dest_obj.
76 * @_dest_obj: The destination container object in kernel space.
77 * @_usr_size: The size of the source container in user space.
78 * @_src_usr_ptr: __u64 raw pointer to the source container in user space.
79 *
80 * Return: Error code. See pvr_get_uobj().
81 */
82#define PVR_UOBJ_GET(_dest_obj, _usr_size, _src_usr_ptr) \
83 pvr_get_uobj(_src_usr_ptr, _usr_size, \
84 PVR_UOBJ_MIN_SIZE(_dest_obj), \
85 sizeof(_dest_obj), &(_dest_obj))
86
87/**
88 * PVR_UOBJ_SET() - Copies from &_src_obj to _dest_usr_ptr.
89 * @_dest_usr_ptr: __u64 raw pointer to the destination container in user space.
90 * @_usr_size: The size of the destination container in user space.
91 * @_src_obj: The source container object in kernel space.
92 *
93 * Return: Error code. See pvr_set_uobj().
94 */
95#define PVR_UOBJ_SET(_dest_usr_ptr, _usr_size, _src_obj) \
96 pvr_set_uobj(_dest_usr_ptr, _usr_size, \
97 PVR_UOBJ_MIN_SIZE(_src_obj), \
98 sizeof(_src_obj), &(_src_obj))
99
100/**
101 * PVR_UOBJ_GET_ARRAY() - Copies from @_src_drm_pvr_obj_array.array to
102 * alloced memory and returns a pointer in _dest_array.
103 * @_dest_array: The destination C array object in kernel space.
104 * @_src_drm_pvr_obj_array: The &struct drm_pvr_obj_array containing a __u64 raw
105 * pointer to the source C array in user space and the size of each array
106 * element in user space (the 'stride').
107 *
108 * Return: Error code. See pvr_get_uobj_array().
109 */
110#define PVR_UOBJ_GET_ARRAY(_dest_array, _src_drm_pvr_obj_array) \
111 pvr_get_uobj_array(_src_drm_pvr_obj_array, \
112 PVR_UOBJ_MIN_SIZE((_dest_array)[0]), \
113 sizeof((_dest_array)[0]), (void **)&(_dest_array))
114
115/**
116 * PVR_UOBJ_SET_ARRAY() - Copies from _src_array to @_dest_drm_pvr_obj_array.array.
117 * @_dest_drm_pvr_obj_array: The &struct drm_pvr_obj_array containing a __u64 raw
118 * pointer to the destination C array in user space and the size of each array
119 * element in user space (the 'stride').
120 * @_src_array: The source C array object in kernel space.
121 *
122 * Return: Error code. See pvr_set_uobj_array().
123 */
124#define PVR_UOBJ_SET_ARRAY(_dest_drm_pvr_obj_array, _src_array) \
125 pvr_set_uobj_array(_dest_drm_pvr_obj_array, \
126 PVR_UOBJ_MIN_SIZE((_src_array)[0]), \
127 sizeof((_src_array)[0]), _src_array)
128
129#endif /* PVR_DRV_H */
drivers/gpu/drm/imagination/pvr_free_list.c+625−0
@@ -0,0 +1,625 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_free_list.h"
5#include "pvr_gem.h"
6#include "pvr_hwrt.h"
7#include "pvr_rogue_fwif.h"
8#include "pvr_vm.h"
9
10#include <drm/drm_gem.h>
11#include <linux/slab.h>
12#include <linux/xarray.h>
13#include <uapi/drm/pvr_drm.h>
14
15#define FREE_LIST_ENTRY_SIZE sizeof(u32)
16
17#define FREE_LIST_ALIGNMENT \
18 ((ROGUE_BIF_PM_FREELIST_BASE_ADDR_ALIGNSIZE / FREE_LIST_ENTRY_SIZE) - 1)
19
20#define FREE_LIST_MIN_PAGES 50
21#define FREE_LIST_MIN_PAGES_BRN66011 40
22#define FREE_LIST_MIN_PAGES_ROGUEXE 25
23
24/**
25 * pvr_get_free_list_min_pages() - Get minimum free list size for this device
26 * @pvr_dev: Device pointer.
27 *
28 * Returns:
29 * * Minimum free list size, in PM physical pages.
30 */
31u32
32pvr_get_free_list_min_pages(struct pvr_device *pvr_dev)
33{
34 u32 value;
35
36 if (PVR_HAS_FEATURE(pvr_dev, roguexe)) {
37 if (PVR_HAS_QUIRK(pvr_dev, 66011))
38 value = FREE_LIST_MIN_PAGES_BRN66011;
39 else
40 value = FREE_LIST_MIN_PAGES_ROGUEXE;
41 } else {
42 value = FREE_LIST_MIN_PAGES;
43 }
44
45 return value;
46}
47
48static int
49free_list_create_kernel_structure(struct pvr_file *pvr_file,
50 struct drm_pvr_ioctl_create_free_list_args *args,
51 struct pvr_free_list *free_list)
52{
53 struct pvr_gem_object *free_list_obj;
54 struct pvr_vm_context *vm_ctx;
55 u64 free_list_size;
56 int err;
57
58 if (args->grow_threshold > 100 ||
59 args->initial_num_pages > args->max_num_pages ||
60 args->grow_num_pages > args->max_num_pages ||
61 args->max_num_pages == 0 ||
62 (args->initial_num_pages < args->max_num_pages && !args->grow_num_pages) ||
63 (args->initial_num_pages == args->max_num_pages && args->grow_num_pages))
64 return -EINVAL;
65
66 if ((args->initial_num_pages & FREE_LIST_ALIGNMENT) ||
67 (args->max_num_pages & FREE_LIST_ALIGNMENT) ||
68 (args->grow_num_pages & FREE_LIST_ALIGNMENT))
69 return -EINVAL;
70
71 vm_ctx = pvr_vm_context_lookup(pvr_file, args->vm_context_handle);
72 if (!vm_ctx)
73 return -EINVAL;
74
75 free_list_obj = pvr_vm_find_gem_object(vm_ctx, args->free_list_gpu_addr,
76 NULL, &free_list_size);
77 if (!free_list_obj) {
78 err = -EINVAL;
79 goto err_put_vm_context;
80 }
81
82 if ((free_list_obj->flags & DRM_PVR_BO_ALLOW_CPU_USERSPACE_ACCESS) ||
83 !(free_list_obj->flags & DRM_PVR_BO_PM_FW_PROTECT) ||
84 free_list_size < (args->max_num_pages * FREE_LIST_ENTRY_SIZE)) {
85 err = -EINVAL;
86 goto err_put_free_list_obj;
87 }
88
89 free_list->pvr_dev = pvr_file->pvr_dev;
90 free_list->current_pages = 0;
91 free_list->max_pages = args->max_num_pages;
92 free_list->grow_pages = args->grow_num_pages;
93 free_list->grow_threshold = args->grow_threshold;
94 free_list->obj = free_list_obj;
95 free_list->free_list_gpu_addr = args->free_list_gpu_addr;
96 free_list->initial_num_pages = args->initial_num_pages;
97
98 pvr_vm_context_put(vm_ctx);
99
100 return 0;
101
102err_put_free_list_obj:
103 pvr_gem_object_put(free_list_obj);
104
105err_put_vm_context:
106 pvr_vm_context_put(vm_ctx);
107
108 return err;
109}
110
111static void
112free_list_destroy_kernel_structure(struct pvr_free_list *free_list)
113{
114 WARN_ON(!list_empty(&free_list->hwrt_list));
115
116 pvr_gem_object_put(free_list->obj);
117}
118
119/**
120 * calculate_free_list_ready_pages_locked() - Function to work out the number of free
121 * list pages to reserve for growing within
122 * the FW without having to wait for the
123 * host to progress a grow request
124 * @free_list: Pointer to free list.
125 * @pages: Total pages currently in free list.
126 *
127 * If the threshold or grow size means less than the alignment size (4 pages on
128 * Rogue), then the feature is not used.
129 *
130 * Caller must hold &free_list->lock.
131 *
132 * Return: number of pages to reserve.
133 */
134static u32
135calculate_free_list_ready_pages_locked(struct pvr_free_list *free_list, u32 pages)
136{
137 u32 ready_pages;
138
139 lockdep_assert_held(&free_list->lock);
140
141 ready_pages = ((pages * free_list->grow_threshold) / 100);
142
143 /* The number of pages must be less than the grow size. */
144 ready_pages = min(ready_pages, free_list->grow_pages);
145
146 /*
147 * The number of pages must be a multiple of the free list align size.
148 */
149 ready_pages &= ~FREE_LIST_ALIGNMENT;
150
151 return ready_pages;
152}
153
154static u32
155calculate_free_list_ready_pages(struct pvr_free_list *free_list, u32 pages)
156{
157 u32 ret;
158
159 mutex_lock(&free_list->lock);
160
161 ret = calculate_free_list_ready_pages_locked(free_list, pages);
162
163 mutex_unlock(&free_list->lock);
164
165 return ret;
166}
167
168static void
169free_list_fw_init(void *cpu_ptr, void *priv)
170{
171 struct rogue_fwif_freelist *fw_data = cpu_ptr;
172 struct pvr_free_list *free_list = priv;
173 u32 ready_pages;
174
175 /* Fill out FW structure */
176 ready_pages = calculate_free_list_ready_pages(free_list,
177 free_list->initial_num_pages);
178
179 fw_data->max_pages = free_list->max_pages;
180 fw_data->current_pages = free_list->initial_num_pages - ready_pages;
181 fw_data->grow_pages = free_list->grow_pages;
182 fw_data->ready_pages = ready_pages;
183 fw_data->freelist_id = free_list->fw_id;
184 fw_data->grow_pending = false;
185 fw_data->current_stack_top = fw_data->current_pages - 1;
186 fw_data->freelist_dev_addr = free_list->free_list_gpu_addr;
187 fw_data->current_dev_addr = (fw_data->freelist_dev_addr +
188 ((fw_data->max_pages - fw_data->current_pages) *
189 FREE_LIST_ENTRY_SIZE)) &
190 ~((u64)ROGUE_BIF_PM_FREELIST_BASE_ADDR_ALIGNSIZE - 1);
191}
192
193static int
194free_list_create_fw_structure(struct pvr_file *pvr_file,
195 struct drm_pvr_ioctl_create_free_list_args *args,
196 struct pvr_free_list *free_list)
197{
198 struct pvr_device *pvr_dev = pvr_file->pvr_dev;
199
200 /*
201 * Create and map the FW structure so we can initialise it. This is not
202 * accessed on the CPU side post-initialisation so the mapping lifetime
203 * is only for this function.
204 */
205 free_list->fw_data = pvr_fw_object_create_and_map(pvr_dev, sizeof(*free_list->fw_data),
206 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
207 free_list_fw_init, free_list,
208 &free_list->fw_obj);
209 if (IS_ERR(free_list->fw_data))
210 return PTR_ERR(free_list->fw_data);
211
212 return 0;
213}
214
215static void
216free_list_destroy_fw_structure(struct pvr_free_list *free_list)
217{
218 pvr_fw_object_unmap_and_destroy(free_list->fw_obj);
219}
220
221static int
222pvr_free_list_insert_pages_locked(struct pvr_free_list *free_list,
223 struct sg_table *sgt, u32 offset, u32 num_pages)
224{
225 struct sg_dma_page_iter dma_iter;
226 u32 *page_list;
227
228 lockdep_assert_held(&free_list->lock);
229
230 page_list = pvr_gem_object_vmap(free_list->obj);
231 if (IS_ERR(page_list))
232 return PTR_ERR(page_list);
233
234 offset /= FREE_LIST_ENTRY_SIZE;
235 /* clang-format off */
236 for_each_sgtable_dma_page(sgt, &dma_iter, 0) {
237 dma_addr_t dma_addr = sg_page_iter_dma_address(&dma_iter);
238 u64 dma_pfn = dma_addr >>
239 ROGUE_BIF_PM_PHYSICAL_PAGE_ALIGNSHIFT;
240 u32 dma_addr_offset;
241
242 BUILD_BUG_ON(ROGUE_BIF_PM_PHYSICAL_PAGE_SIZE > PAGE_SIZE);
243
244 for (dma_addr_offset = 0; dma_addr_offset < PAGE_SIZE;
245 dma_addr_offset += ROGUE_BIF_PM_PHYSICAL_PAGE_SIZE) {
246 WARN_ON_ONCE(dma_pfn >> 32);
247
248 page_list[offset++] = (u32)dma_pfn;
249 dma_pfn++;
250
251 num_pages--;
252 if (!num_pages)
253 break;
254 }
255
256 if (!num_pages)
257 break;
258 };
259 /* clang-format on */
260
261 /* Make sure our free_list update is flushed. */
262 wmb();
263
264 pvr_gem_object_vunmap(free_list->obj);
265
266 return 0;
267}
268
269static int
270pvr_free_list_insert_node_locked(struct pvr_free_list_node *free_list_node)
271{
272 struct pvr_free_list *free_list = free_list_node->free_list;
273 struct sg_table *sgt;
274 u32 start_page;
275 u32 offset;
276 int err;
277
278 lockdep_assert_held(&free_list->lock);
279
280 start_page = free_list->max_pages - free_list->current_pages -
281 free_list_node->num_pages;
282 offset = (start_page * FREE_LIST_ENTRY_SIZE) &
283 ~((u64)ROGUE_BIF_PM_FREELIST_BASE_ADDR_ALIGNSIZE - 1);
284
285 sgt = drm_gem_shmem_get_pages_sgt(&free_list_node->mem_obj->base);
286 if (WARN_ON(IS_ERR(sgt)))
287 return PTR_ERR(sgt);
288
289 err = pvr_free_list_insert_pages_locked(free_list, sgt,
290 offset, free_list_node->num_pages);
291 if (!err)
292 free_list->current_pages += free_list_node->num_pages;
293
294 return err;
295}
296
297static int
298pvr_free_list_grow(struct pvr_free_list *free_list, u32 num_pages)
299{
300 struct pvr_device *pvr_dev = free_list->pvr_dev;
301 struct pvr_free_list_node *free_list_node;
302 int err;
303
304 mutex_lock(&free_list->lock);
305
306 if (num_pages & FREE_LIST_ALIGNMENT) {
307 err = -EINVAL;
308 goto err_unlock;
309 }
310
311 free_list_node = kzalloc(sizeof(*free_list_node), GFP_KERNEL);
312 if (!free_list_node) {
313 err = -ENOMEM;
314 goto err_unlock;
315 }
316
317 free_list_node->num_pages = num_pages;
318 free_list_node->free_list = free_list;
319
320 free_list_node->mem_obj = pvr_gem_object_create(pvr_dev,
321 num_pages <<
322 ROGUE_BIF_PM_PHYSICAL_PAGE_ALIGNSHIFT,
323 PVR_BO_FW_FLAGS_DEVICE_CACHED);
324 if (IS_ERR(free_list_node->mem_obj)) {
325 err = PTR_ERR(free_list_node->mem_obj);
326 goto err_free;
327 }
328
329 err = pvr_free_list_insert_node_locked(free_list_node);
330 if (err)
331 goto err_destroy_gem_object;
332
333 list_add_tail(&free_list_node->node, &free_list->mem_block_list);
334
335 /*
336 * Reserve a number ready pages to allow the FW to process OOM quickly
337 * and asynchronously request a grow.
338 */
339 free_list->ready_pages =
340 calculate_free_list_ready_pages_locked(free_list,
341 free_list->current_pages);
342 free_list->current_pages -= free_list->ready_pages;
343
344 mutex_unlock(&free_list->lock);
345
346 return 0;
347
348err_destroy_gem_object:
349 pvr_gem_object_put(free_list_node->mem_obj);
350
351err_free:
352 kfree(free_list_node);
353
354err_unlock:
355 mutex_unlock(&free_list->lock);
356
357 return err;
358}
359
360void pvr_free_list_process_grow_req(struct pvr_device *pvr_dev,
361 struct rogue_fwif_fwccb_cmd_freelist_gs_data *req)
362{
363 struct pvr_free_list *free_list = pvr_free_list_lookup_id(pvr_dev, req->freelist_id);
364 struct rogue_fwif_kccb_cmd resp_cmd = {
365 .cmd_type = ROGUE_FWIF_KCCB_CMD_FREELIST_GROW_UPDATE,
366 };
367 struct rogue_fwif_freelist_gs_data *resp = &resp_cmd.cmd_data.free_list_gs_data;
368 u32 grow_pages = 0;
369
370 /* If we don't have a freelist registered for this ID, we can't do much. */
371 if (WARN_ON(!free_list))
372 return;
373
374 /* Since the FW made the request, it has already consumed the ready pages,
375 * update the host struct.
376 */
377 free_list->current_pages += free_list->ready_pages;
378 free_list->ready_pages = 0;
379
380 /* If the grow succeeds, update the grow_pages argument. */
381 if (!pvr_free_list_grow(free_list, free_list->grow_pages))
382 grow_pages = free_list->grow_pages;
383
384 /* Now prepare the response and send it back to the FW. */
385 pvr_fw_object_get_fw_addr(free_list->fw_obj, &resp->freelist_fw_addr);
386 resp->delta_pages = grow_pages;
387 resp->new_pages = free_list->current_pages + free_list->ready_pages;
388 resp->ready_pages = free_list->ready_pages;
389 pvr_free_list_put(free_list);
390
391 WARN_ON(pvr_kccb_send_cmd(pvr_dev, &resp_cmd, NULL));
392}
393
394static void
395pvr_free_list_free_node(struct pvr_free_list_node *free_list_node)
396{
397 pvr_gem_object_put(free_list_node->mem_obj);
398
399 kfree(free_list_node);
400}
401
402/**
403 * pvr_free_list_create() - Create a new free list and return an object pointer
404 * @pvr_file: Pointer to pvr_file structure.
405 * @args: Creation arguments from userspace.
406 *
407 * Return:
408 * * Pointer to new free_list, or
409 * * ERR_PTR(-%ENOMEM) on out of memory.
410 */
411struct pvr_free_list *
412pvr_free_list_create(struct pvr_file *pvr_file,
413 struct drm_pvr_ioctl_create_free_list_args *args)
414{
415 struct pvr_free_list *free_list;
416 int err;
417
418 /* Create and fill out the kernel structure */
419 free_list = kzalloc(sizeof(*free_list), GFP_KERNEL);
420
421 if (!free_list)
422 return ERR_PTR(-ENOMEM);
423
424 kref_init(&free_list->ref_count);
425 INIT_LIST_HEAD(&free_list->mem_block_list);
426 INIT_LIST_HEAD(&free_list->hwrt_list);
427 mutex_init(&free_list->lock);
428
429 err = free_list_create_kernel_structure(pvr_file, args, free_list);
430 if (err < 0)
431 goto err_free;
432
433 /* Allocate global object ID for firmware. */
434 err = xa_alloc(&pvr_file->pvr_dev->free_list_ids,
435 &free_list->fw_id,
436 free_list,
437 xa_limit_32b,
438 GFP_KERNEL);
439 if (err)
440 goto err_destroy_kernel_structure;
441
442 err = free_list_create_fw_structure(pvr_file, args, free_list);
443 if (err < 0)
444 goto err_free_fw_id;
445
446 err = pvr_free_list_grow(free_list, args->initial_num_pages);
447 if (err < 0)
448 goto err_fw_struct_cleanup;
449
450 return free_list;
451
452err_fw_struct_cleanup:
453 WARN_ON(pvr_fw_structure_cleanup(free_list->pvr_dev,
454 ROGUE_FWIF_CLEANUP_FREELIST,
455 free_list->fw_obj, 0));
456
457err_free_fw_id:
458 xa_erase(&free_list->pvr_dev->free_list_ids, free_list->fw_id);
459
460err_destroy_kernel_structure:
461 free_list_destroy_kernel_structure(free_list);
462
463err_free:
464 mutex_destroy(&free_list->lock);
465 kfree(free_list);
466
467 return ERR_PTR(err);
468}
469
470static void
471pvr_free_list_release(struct kref *ref_count)
472{
473 struct pvr_free_list *free_list =
474 container_of(ref_count, struct pvr_free_list, ref_count);
475 struct list_head *pos, *n;
476 int err;
477
478 xa_erase(&free_list->pvr_dev->free_list_ids, free_list->fw_id);
479
480 err = pvr_fw_structure_cleanup(free_list->pvr_dev,
481 ROGUE_FWIF_CLEANUP_FREELIST,
482 free_list->fw_obj, 0);
483 if (err == -EBUSY) {
484 /* Flush the FWCCB to process any HWR or freelist reconstruction
485 * request that might keep the freelist busy, and try again.
486 */
487 pvr_fwccb_process(free_list->pvr_dev);
488 err = pvr_fw_structure_cleanup(free_list->pvr_dev,
489 ROGUE_FWIF_CLEANUP_FREELIST,
490 free_list->fw_obj, 0);
491 }
492
493 WARN_ON(err);
494
495 /* clang-format off */
496 list_for_each_safe(pos, n, &free_list->mem_block_list) {
497 struct pvr_free_list_node *free_list_node =
498 container_of(pos, struct pvr_free_list_node, node);
499
500 list_del(pos);
501 pvr_free_list_free_node(free_list_node);
502 }
503 /* clang-format on */
504
505 free_list_destroy_kernel_structure(free_list);
506 free_list_destroy_fw_structure(free_list);
507 mutex_destroy(&free_list->lock);
508 kfree(free_list);
509}
510
511/**
512 * pvr_destroy_free_lists_for_file: Destroy any free lists associated with the
513 * given file.
514 * @pvr_file: Pointer to pvr_file structure.
515 *
516 * Removes all free lists associated with @pvr_file from the device free_list
517 * list and drops initial references. Free lists will then be destroyed once
518 * all outstanding references are dropped.
519 */
520void pvr_destroy_free_lists_for_file(struct pvr_file *pvr_file)
521{
522 struct pvr_free_list *free_list;
523 unsigned long handle;
524
525 xa_for_each(&pvr_file->free_list_handles, handle, free_list) {
526 (void)free_list;
527 pvr_free_list_put(xa_erase(&pvr_file->free_list_handles, handle));
528 }
529}
530
531/**
532 * pvr_free_list_put() - Release reference on free list
533 * @free_list: Pointer to list to release reference on
534 */
535void
536pvr_free_list_put(struct pvr_free_list *free_list)
537{
538 if (free_list)
539 kref_put(&free_list->ref_count, pvr_free_list_release);
540}
541
542void pvr_free_list_add_hwrt(struct pvr_free_list *free_list, struct pvr_hwrt_data *hwrt_data)
543{
544 mutex_lock(&free_list->lock);
545
546 list_add_tail(&hwrt_data->freelist_node, &free_list->hwrt_list);
547
548 mutex_unlock(&free_list->lock);
549}
550
551void pvr_free_list_remove_hwrt(struct pvr_free_list *free_list, struct pvr_hwrt_data *hwrt_data)
552{
553 mutex_lock(&free_list->lock);
554
555 list_del(&hwrt_data->freelist_node);
556
557 mutex_unlock(&free_list->lock);
558}
559
560static void
561pvr_free_list_reconstruct(struct pvr_device *pvr_dev, u32 freelist_id)
562{
563 struct pvr_free_list *free_list = pvr_free_list_lookup_id(pvr_dev, freelist_id);
564 struct pvr_free_list_node *free_list_node;
565 struct rogue_fwif_freelist *fw_data;
566 struct pvr_hwrt_data *hwrt_data;
567
568 if (!free_list)
569 return;
570
571 mutex_lock(&free_list->lock);
572
573 /* Rebuild the free list based on the memory block list. */
574 free_list->current_pages = 0;
575
576 list_for_each_entry(free_list_node, &free_list->mem_block_list, node)
577 WARN_ON(pvr_free_list_insert_node_locked(free_list_node));
578
579 /*
580 * Remove the ready pages, which are reserved to allow the FW to process OOM quickly and
581 * asynchronously request a grow.
582 */
583 free_list->current_pages -= free_list->ready_pages;
584
585 fw_data = free_list->fw_data;
586 fw_data->current_stack_top = fw_data->current_pages - 1;
587 fw_data->allocated_page_count = 0;
588 fw_data->allocated_mmu_page_count = 0;
589
590 /* Reset the state of any associated HWRTs. */
591 list_for_each_entry(hwrt_data, &free_list->hwrt_list, freelist_node) {
592 struct rogue_fwif_hwrtdata *hwrt_fw_data = pvr_fw_object_vmap(hwrt_data->fw_obj);
593
594 if (!WARN_ON(IS_ERR(hwrt_fw_data))) {
595 hwrt_fw_data->state = ROGUE_FWIF_RTDATA_STATE_HWR;
596 hwrt_fw_data->hwrt_data_flags &= ~HWRTDATA_HAS_LAST_GEOM;
597 }
598
599 pvr_fw_object_vunmap(hwrt_data->fw_obj);
600 }
601
602 mutex_unlock(&free_list->lock);
603
604 pvr_free_list_put(free_list);
605}
606
607void
608pvr_free_list_process_reconstruct_req(struct pvr_device *pvr_dev,
609 struct rogue_fwif_fwccb_cmd_freelists_reconstruction_data *req)
610{
611 struct rogue_fwif_kccb_cmd resp_cmd = {
612 .cmd_type = ROGUE_FWIF_KCCB_CMD_FREELISTS_RECONSTRUCTION_UPDATE,
613 };
614 struct rogue_fwif_freelists_reconstruction_data *resp =
615 &resp_cmd.cmd_data.free_lists_reconstruction_data;
616
617 for (u32 i = 0; i < req->freelist_count; i++)
618 pvr_free_list_reconstruct(pvr_dev, req->freelist_ids[i]);
619
620 resp->freelist_count = req->freelist_count;
621 memcpy(resp->freelist_ids, req->freelist_ids,
622 req->freelist_count * sizeof(resp->freelist_ids[0]));
623
624 WARN_ON(pvr_kccb_send_cmd(pvr_dev, &resp_cmd, NULL));
625}
drivers/gpu/drm/imagination/pvr_free_list.h+195−0
@@ -0,0 +1,195 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_FREE_LIST_H
5#define PVR_FREE_LIST_H
6
7#include <linux/compiler_attributes.h>
8#include <linux/kref.h>
9#include <linux/list.h>
10#include <linux/mutex.h>
11#include <linux/types.h>
12#include <linux/xarray.h>
13#include <uapi/drm/pvr_drm.h>
14
15#include "pvr_device.h"
16
17/* Forward declaration from pvr_gem.h. */
18struct pvr_fw_object;
19
20/* Forward declaration from pvr_gem.h. */
21struct pvr_gem_object;
22
23/* Forward declaration from pvr_hwrt.h. */
24struct pvr_hwrt_data;
25
26/**
27 * struct pvr_free_list_node - structure representing an allocation in the free
28 * list
29 */
30struct pvr_free_list_node {
31 /** @node: List node for &pvr_free_list.mem_block_list. */
32 struct list_head node;
33
34 /** @free_list: Pointer to owning free list. */
35 struct pvr_free_list *free_list;
36
37 /** @num_pages: Number of pages in this node. */
38 u32 num_pages;
39
40 /** @mem_obj: GEM object representing the pages in this node. */
41 struct pvr_gem_object *mem_obj;
42};
43
44/**
45 * struct pvr_free_list - structure representing a free list
46 */
47struct pvr_free_list {
48 /** @ref_count: Reference count of object. */
49 struct kref ref_count;
50
51 /** @pvr_dev: Pointer to device that owns this object. */
52 struct pvr_device *pvr_dev;
53
54 /** @obj: GEM object representing the free list. */
55 struct pvr_gem_object *obj;
56
57 /** @fw_obj: FW object representing the FW-side structure. */
58 struct pvr_fw_object *fw_obj;
59
60 /** @fw_data: Pointer to CPU mapping of the FW-side structure. */
61 struct rogue_fwif_freelist *fw_data;
62
63 /**
64 * @lock: Mutex protecting modification of the free list. Must be held when accessing any
65 * of the members below.
66 */
67 struct mutex lock;
68
69 /** @fw_id: Firmware ID for this object. */
70 u32 fw_id;
71
72 /** @current_pages: Current number of pages in free list. */
73 u32 current_pages;
74
75 /** @max_pages: Maximum number of pages in free list. */
76 u32 max_pages;
77
78 /** @grow_pages: Pages to grow free list by per request. */
79 u32 grow_pages;
80
81 /**
82 * @grow_threshold: Percentage of FL memory used that should trigger a
83 * new grow request.
84 */
85 u32 grow_threshold;
86
87 /**
88 * @ready_pages: Number of pages reserved for FW to use while a grow
89 * request is being processed.
90 */
91 u32 ready_pages;
92
93 /** @mem_block_list: List of memory blocks in this free list. */
94 struct list_head mem_block_list;
95
96 /** @hwrt_list: List of HWRTs using this free list. */
97 struct list_head hwrt_list;
98
99 /** @initial_num_pages: Initial number of pages in free list. */
100 u32 initial_num_pages;
101
102 /** @free_list_gpu_addr: Address of free list in GPU address space. */
103 u64 free_list_gpu_addr;
104};
105
106struct pvr_free_list *
107pvr_free_list_create(struct pvr_file *pvr_file,
108 struct drm_pvr_ioctl_create_free_list_args *args);
109
110void
111pvr_destroy_free_lists_for_file(struct pvr_file *pvr_file);
112
113u32
114pvr_get_free_list_min_pages(struct pvr_device *pvr_dev);
115
116static __always_inline struct pvr_free_list *
117pvr_free_list_get(struct pvr_free_list *free_list)
118{
119 if (free_list)
120 kref_get(&free_list->ref_count);
121
122 return free_list;
123}
124
125/**
126 * pvr_free_list_lookup() - Lookup free list pointer from handle and file
127 * @pvr_file: Pointer to pvr_file structure.
128 * @handle: Object handle.
129 *
130 * Takes reference on free list object. Call pvr_free_list_put() to release.
131 *
132 * Returns:
133 * * The requested object on success, or
134 * * %NULL on failure (object does not exist in list, is not a free list, or
135 * does not belong to @pvr_file)
136 */
137static __always_inline struct pvr_free_list *
138pvr_free_list_lookup(struct pvr_file *pvr_file, u32 handle)
139{
140 struct pvr_free_list *free_list;
141
142 xa_lock(&pvr_file->free_list_handles);
143 free_list = pvr_free_list_get(xa_load(&pvr_file->free_list_handles, handle));
144 xa_unlock(&pvr_file->free_list_handles);
145
146 return free_list;
147}
148
149/**
150 * pvr_free_list_lookup_id() - Lookup free list pointer from FW ID
151 * @pvr_dev: Device pointer.
152 * @id: FW object ID.
153 *
154 * Takes reference on free list object. Call pvr_free_list_put() to release.
155 *
156 * Returns:
157 * * The requested object on success, or
158 * * %NULL on failure (object does not exist in list, or is not a free list)
159 */
160static __always_inline struct pvr_free_list *
161pvr_free_list_lookup_id(struct pvr_device *pvr_dev, u32 id)
162{
163 struct pvr_free_list *free_list;
164
165 xa_lock(&pvr_dev->free_list_ids);
166
167 /* Contexts are removed from the ctx_ids set in the context release path,
168 * meaning the ref_count reached zero before they get removed. We need
169 * to make sure we're not trying to acquire a context that's being
170 * destroyed.
171 */
172 free_list = xa_load(&pvr_dev->free_list_ids, id);
173 if (free_list && !kref_get_unless_zero(&free_list->ref_count))
174 free_list = NULL;
175 xa_unlock(&pvr_dev->free_list_ids);
176
177 return free_list;
178}
179
180void
181pvr_free_list_put(struct pvr_free_list *free_list);
182
183void
184pvr_free_list_add_hwrt(struct pvr_free_list *free_list, struct pvr_hwrt_data *hwrt_data);
185void
186pvr_free_list_remove_hwrt(struct pvr_free_list *free_list, struct pvr_hwrt_data *hwrt_data);
187
188void pvr_free_list_process_grow_req(struct pvr_device *pvr_dev,
189 struct rogue_fwif_fwccb_cmd_freelist_gs_data *req);
190
191void
192pvr_free_list_process_reconstruct_req(struct pvr_device *pvr_dev,
193 struct rogue_fwif_fwccb_cmd_freelists_reconstruction_data *req);
194
195#endif /* PVR_FREE_LIST_H */
drivers/gpu/drm/imagination/pvr_fw.c+1489−0
@@ -0,0 +1,1489 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_ccb.h"
5#include "pvr_device.h"
6#include "pvr_device_info.h"
7#include "pvr_fw.h"
8#include "pvr_fw_info.h"
9#include "pvr_fw_startstop.h"
10#include "pvr_fw_trace.h"
11#include "pvr_gem.h"
12#include "pvr_power.h"
13#include "pvr_rogue_fwif_dev_info.h"
14#include "pvr_rogue_heap_config.h"
15#include "pvr_vm.h"
16
17#include <drm/drm_drv.h>
18#include <drm/drm_managed.h>
19#include <drm/drm_mm.h>
20#include <linux/clk.h>
21#include <linux/firmware.h>
22#include <linux/math.h>
23#include <linux/minmax.h>
24#include <linux/sizes.h>
25
26#define FW_MAX_SUPPORTED_MAJOR_VERSION 1
27
28#define FW_BOOT_TIMEOUT_USEC 5000000
29
30/* Config heap occupies top 192k of the firmware heap. */
31#define PVR_ROGUE_FW_CONFIG_HEAP_GRANULARITY SZ_64K
32#define PVR_ROGUE_FW_CONFIG_HEAP_SIZE (3 * PVR_ROGUE_FW_CONFIG_HEAP_GRANULARITY)
33
34/* Main firmware allocations should come from the remainder of the heap. */
35#define PVR_ROGUE_FW_MAIN_HEAP_BASE ROGUE_FW_HEAP_BASE
36
37/* Offsets from start of configuration area of FW heap. */
38#define PVR_ROGUE_FWIF_CONNECTION_CTL_OFFSET 0
39#define PVR_ROGUE_FWIF_OSINIT_OFFSET \
40 (PVR_ROGUE_FWIF_CONNECTION_CTL_OFFSET + PVR_ROGUE_FW_CONFIG_HEAP_GRANULARITY)
41#define PVR_ROGUE_FWIF_SYSINIT_OFFSET \
42 (PVR_ROGUE_FWIF_OSINIT_OFFSET + PVR_ROGUE_FW_CONFIG_HEAP_GRANULARITY)
43
44#define PVR_ROGUE_FAULT_PAGE_SIZE SZ_4K
45
46#define PVR_SYNC_OBJ_SIZE sizeof(u32)
47
48const struct pvr_fw_layout_entry *
49pvr_fw_find_layout_entry(struct pvr_device *pvr_dev, enum pvr_fw_section_id id)
50{
51 const struct pvr_fw_layout_entry *layout_entries = pvr_dev->fw_dev.layout_entries;
52 u32 num_layout_entries = pvr_dev->fw_dev.header->layout_entry_num;
53 u32 entry;
54
55 for (entry = 0; entry < num_layout_entries; entry++) {
56 if (layout_entries[entry].id == id)
57 return &layout_entries[entry];
58 }
59
60 return NULL;
61}
62
63static const struct pvr_fw_layout_entry *
64pvr_fw_find_private_data(struct pvr_device *pvr_dev)
65{
66 const struct pvr_fw_layout_entry *layout_entries = pvr_dev->fw_dev.layout_entries;
67 u32 num_layout_entries = pvr_dev->fw_dev.header->layout_entry_num;
68 u32 entry;
69
70 for (entry = 0; entry < num_layout_entries; entry++) {
71 if (layout_entries[entry].id == META_PRIVATE_DATA ||
72 layout_entries[entry].id == MIPS_PRIVATE_DATA ||
73 layout_entries[entry].id == RISCV_PRIVATE_DATA)
74 return &layout_entries[entry];
75 }
76
77 return NULL;
78}
79
80#define DEV_INFO_MASK_SIZE(x) DIV_ROUND_UP(x, 64)
81
82/**
83 * pvr_fw_validate() - Parse firmware header and check compatibility
84 * @pvr_dev: Device pointer.
85 *
86 * Returns:
87 * * 0 on success, or
88 * * -EINVAL if firmware is incompatible.
89 */
90static int
91pvr_fw_validate(struct pvr_device *pvr_dev)
92{
93 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
94 const struct firmware *firmware = pvr_dev->fw_dev.firmware;
95 const struct pvr_fw_layout_entry *layout_entries;
96 const struct pvr_fw_info_header *header;
97 const u8 *fw = firmware->data;
98 u32 fw_offset = firmware->size - SZ_4K;
99 u32 layout_table_size;
100 u32 entry;
101
102 if (firmware->size < SZ_4K || (firmware->size % FW_BLOCK_SIZE))
103 return -EINVAL;
104
105 header = (const struct pvr_fw_info_header *)&fw[fw_offset];
106
107 if (header->info_version != PVR_FW_INFO_VERSION) {
108 drm_err(drm_dev, "Unsupported fw info version %u\n",
109 header->info_version);
110 return -EINVAL;
111 }
112
113 if (header->header_len != sizeof(struct pvr_fw_info_header) ||
114 header->layout_entry_size != sizeof(struct pvr_fw_layout_entry) ||
115 header->layout_entry_num > PVR_FW_INFO_MAX_NUM_ENTRIES) {
116 drm_err(drm_dev, "FW info format mismatch\n");
117 return -EINVAL;
118 }
119
120 if (!(header->flags & PVR_FW_FLAGS_OPEN_SOURCE) ||
121 header->fw_version_major > FW_MAX_SUPPORTED_MAJOR_VERSION ||
122 header->fw_version_major == 0) {
123 drm_err(drm_dev, "Unsupported FW version %u.%u (build: %u%s)\n",
124 header->fw_version_major, header->fw_version_minor,
125 header->fw_version_build,
126 (header->flags & PVR_FW_FLAGS_OPEN_SOURCE) ? " OS" : "");
127 return -EINVAL;
128 }
129
130 if (pvr_gpu_id_to_packed_bvnc(&pvr_dev->gpu_id) != header->bvnc) {
131 struct pvr_gpu_id fw_gpu_id;
132
133 packed_bvnc_to_pvr_gpu_id(header->bvnc, &fw_gpu_id);
134 drm_err(drm_dev, "FW built for incorrect GPU ID %i.%i.%i.%i (expected %i.%i.%i.%i)\n",
135 fw_gpu_id.b, fw_gpu_id.v, fw_gpu_id.n, fw_gpu_id.c,
136 pvr_dev->gpu_id.b, pvr_dev->gpu_id.v, pvr_dev->gpu_id.n, pvr_dev->gpu_id.c);
137 return -EINVAL;
138 }
139
140 fw_offset += header->header_len;
141 layout_table_size =
142 header->layout_entry_size * header->layout_entry_num;
143 if ((fw_offset + layout_table_size) > firmware->size)
144 return -EINVAL;
145
146 layout_entries = (const struct pvr_fw_layout_entry *)&fw[fw_offset];
147 for (entry = 0; entry < header->layout_entry_num; entry++) {
148 u32 start_addr = layout_entries[entry].base_addr;
149 u32 end_addr = start_addr + layout_entries[entry].alloc_size;
150
151 if (start_addr >= end_addr)
152 return -EINVAL;
153 }
154
155 fw_offset = (firmware->size - SZ_4K) - header->device_info_size;
156
157 drm_info(drm_dev, "FW version v%u.%u (build %u OS)\n", header->fw_version_major,
158 header->fw_version_minor, header->fw_version_build);
159
160 pvr_dev->fw_version.major = header->fw_version_major;
161 pvr_dev->fw_version.minor = header->fw_version_minor;
162
163 pvr_dev->fw_dev.header = header;
164 pvr_dev->fw_dev.layout_entries = layout_entries;
165
166 return 0;
167}
168
169static int
170pvr_fw_get_device_info(struct pvr_device *pvr_dev)
171{
172 const struct firmware *firmware = pvr_dev->fw_dev.firmware;
173 struct pvr_fw_device_info_header *header;
174 const u8 *fw = firmware->data;
175 const u64 *dev_info;
176 u32 fw_offset;
177
178 fw_offset = (firmware->size - SZ_4K) - pvr_dev->fw_dev.header->device_info_size;
179
180 header = (struct pvr_fw_device_info_header *)&fw[fw_offset];
181 dev_info = (u64 *)(header + 1);
182
183 pvr_device_info_set_quirks(pvr_dev, dev_info, header->brn_mask_size);
184 dev_info += header->brn_mask_size;
185
186 pvr_device_info_set_enhancements(pvr_dev, dev_info, header->ern_mask_size);
187 dev_info += header->ern_mask_size;
188
189 return pvr_device_info_set_features(pvr_dev, dev_info, header->feature_mask_size,
190 header->feature_param_size);
191}
192
193static void
194layout_get_sizes(struct pvr_device *pvr_dev)
195{
196 const struct pvr_fw_layout_entry *layout_entries = pvr_dev->fw_dev.layout_entries;
197 u32 num_layout_entries = pvr_dev->fw_dev.header->layout_entry_num;
198 struct pvr_fw_mem *fw_mem = &pvr_dev->fw_dev.mem;
199
200 fw_mem->code_alloc_size = 0;
201 fw_mem->data_alloc_size = 0;
202 fw_mem->core_code_alloc_size = 0;
203 fw_mem->core_data_alloc_size = 0;
204
205 /* Extract section sizes from FW layout table. */
206 for (u32 entry = 0; entry < num_layout_entries; entry++) {
207 switch (layout_entries[entry].type) {
208 case FW_CODE:
209 fw_mem->code_alloc_size += layout_entries[entry].alloc_size;
210 break;
211 case FW_DATA:
212 fw_mem->data_alloc_size += layout_entries[entry].alloc_size;
213 break;
214 case FW_COREMEM_CODE:
215 fw_mem->core_code_alloc_size +=
216 layout_entries[entry].alloc_size;
217 break;
218 case FW_COREMEM_DATA:
219 fw_mem->core_data_alloc_size +=
220 layout_entries[entry].alloc_size;
221 break;
222 case NONE:
223 break;
224 }
225 }
226}
227
228int
229pvr_fw_find_mmu_segment(struct pvr_device *pvr_dev, u32 addr, u32 size, void *fw_code_ptr,
230 void *fw_data_ptr, void *fw_core_code_ptr, void *fw_core_data_ptr,
231 void **host_addr_out)
232{
233 const struct pvr_fw_layout_entry *layout_entries = pvr_dev->fw_dev.layout_entries;
234 u32 num_layout_entries = pvr_dev->fw_dev.header->layout_entry_num;
235 u32 end_addr = addr + size;
236 int entry = 0;
237
238 /* Ensure requested range is not zero, and size is not causing addr to overflow. */
239 if (end_addr <= addr)
240 return -EINVAL;
241
242 for (entry = 0; entry < num_layout_entries; entry++) {
243 u32 entry_start_addr = layout_entries[entry].base_addr;
244 u32 entry_end_addr = entry_start_addr + layout_entries[entry].alloc_size;
245
246 if (addr >= entry_start_addr && addr < entry_end_addr &&
247 end_addr > entry_start_addr && end_addr <= entry_end_addr) {
248 switch (layout_entries[entry].type) {
249 case FW_CODE:
250 *host_addr_out = fw_code_ptr;
251 break;
252
253 case FW_DATA:
254 *host_addr_out = fw_data_ptr;
255 break;
256
257 case FW_COREMEM_CODE:
258 *host_addr_out = fw_core_code_ptr;
259 break;
260
261 case FW_COREMEM_DATA:
262 *host_addr_out = fw_core_data_ptr;
263 break;
264
265 default:
266 return -EINVAL;
267 }
268 /* Direct Mem write to mapped memory */
269 addr -= layout_entries[entry].base_addr;
270 addr += layout_entries[entry].alloc_offset;
271
272 /*
273 * Add offset to pointer to FW allocation only if that
274 * allocation is available
275 */
276 *(u8 **)host_addr_out += addr;
277 return 0;
278 }
279 }
280
281 return -EINVAL;
282}
283
284static int
285pvr_fw_create_fwif_connection_ctl(struct pvr_device *pvr_dev)
286{
287 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
288 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
289
290 fw_dev->fwif_connection_ctl =
291 pvr_fw_object_create_and_map_offset(pvr_dev,
292 fw_dev->fw_heap_info.config_offset +
293 PVR_ROGUE_FWIF_CONNECTION_CTL_OFFSET,
294 sizeof(*fw_dev->fwif_connection_ctl),
295 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
296 NULL, NULL,
297 &fw_dev->mem.fwif_connection_ctl_obj);
298 if (IS_ERR(fw_dev->fwif_connection_ctl)) {
299 drm_err(drm_dev,
300 "Unable to allocate FWIF connection control memory\n");
301 return PTR_ERR(fw_dev->fwif_connection_ctl);
302 }
303
304 return 0;
305}
306
307static void
308pvr_fw_fini_fwif_connection_ctl(struct pvr_device *pvr_dev)
309{
310 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
311
312 pvr_fw_object_unmap_and_destroy(fw_dev->mem.fwif_connection_ctl_obj);
313}
314
315static void
316fw_osinit_init(void *cpu_ptr, void *priv)
317{
318 struct rogue_fwif_osinit *fwif_osinit = cpu_ptr;
319 struct pvr_device *pvr_dev = priv;
320 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
321 struct pvr_fw_mem *fw_mem = &fw_dev->mem;
322
323 fwif_osinit->kernel_ccbctl_fw_addr = pvr_dev->kccb.ccb.ctrl_fw_addr;
324 fwif_osinit->kernel_ccb_fw_addr = pvr_dev->kccb.ccb.ccb_fw_addr;
325 pvr_fw_object_get_fw_addr(pvr_dev->kccb.rtn_obj,
326 &fwif_osinit->kernel_ccb_rtn_slots_fw_addr);
327
328 fwif_osinit->firmware_ccbctl_fw_addr = pvr_dev->fwccb.ctrl_fw_addr;
329 fwif_osinit->firmware_ccb_fw_addr = pvr_dev->fwccb.ccb_fw_addr;
330
331 fwif_osinit->work_est_firmware_ccbctl_fw_addr = 0;
332 fwif_osinit->work_est_firmware_ccb_fw_addr = 0;
333
334 pvr_fw_object_get_fw_addr(fw_mem->hwrinfobuf_obj,
335 &fwif_osinit->rogue_fwif_hwr_info_buf_ctl_fw_addr);
336 pvr_fw_object_get_fw_addr(fw_mem->osdata_obj, &fwif_osinit->fw_os_data_fw_addr);
337
338 fwif_osinit->hwr_debug_dump_limit = 0;
339
340 rogue_fwif_compchecks_bvnc_init(&fwif_osinit->rogue_comp_checks.hw_bvnc);
341 rogue_fwif_compchecks_bvnc_init(&fwif_osinit->rogue_comp_checks.fw_bvnc);
342}
343
344static void
345fw_osdata_init(void *cpu_ptr, void *priv)
346{
347 struct rogue_fwif_osdata *fwif_osdata = cpu_ptr;
348 struct pvr_device *pvr_dev = priv;
349 struct pvr_fw_mem *fw_mem = &pvr_dev->fw_dev.mem;
350
351 pvr_fw_object_get_fw_addr(fw_mem->power_sync_obj, &fwif_osdata->power_sync_fw_addr);
352}
353
354static void
355fw_fault_page_init(void *cpu_ptr, void *priv)
356{
357 u32 *fault_page = cpu_ptr;
358
359 for (int i = 0; i < PVR_ROGUE_FAULT_PAGE_SIZE / sizeof(*fault_page); i++)
360 fault_page[i] = 0xdeadbee0;
361}
362
363static void
364fw_sysinit_init(void *cpu_ptr, void *priv)
365{
366 struct rogue_fwif_sysinit *fwif_sysinit = cpu_ptr;
367 struct pvr_device *pvr_dev = priv;
368 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
369 struct pvr_fw_mem *fw_mem = &fw_dev->mem;
370 dma_addr_t fault_dma_addr = 0;
371 u32 clock_speed_hz = clk_get_rate(pvr_dev->core_clk);
372
373 WARN_ON(!clock_speed_hz);
374
375 WARN_ON(pvr_fw_object_get_dma_addr(fw_mem->fault_page_obj, 0, &fault_dma_addr));
376 fwif_sysinit->fault_phys_addr = (u64)fault_dma_addr;
377
378 fwif_sysinit->pds_exec_base = ROGUE_PDSCODEDATA_HEAP_BASE;
379 fwif_sysinit->usc_exec_base = ROGUE_USCCODE_HEAP_BASE;
380
381 pvr_fw_object_get_fw_addr(fw_mem->runtime_cfg_obj, &fwif_sysinit->runtime_cfg_fw_addr);
382 pvr_fw_object_get_fw_addr(fw_dev->fw_trace.tracebuf_ctrl_obj,
383 &fwif_sysinit->trace_buf_ctl_fw_addr);
384 pvr_fw_object_get_fw_addr(fw_mem->sysdata_obj, &fwif_sysinit->fw_sys_data_fw_addr);
385 pvr_fw_object_get_fw_addr(fw_mem->gpu_util_fwcb_obj,
386 &fwif_sysinit->gpu_util_fw_cb_ctl_fw_addr);
387 if (fw_mem->core_data_obj) {
388 pvr_fw_object_get_fw_addr(fw_mem->core_data_obj,
389 &fwif_sysinit->coremem_data_store.fw_addr);
390 }
391
392 /* Currently unsupported. */
393 fwif_sysinit->counter_dump_ctl.buffer_fw_addr = 0;
394 fwif_sysinit->counter_dump_ctl.size_in_dwords = 0;
395
396 /* Skip alignment checks. */
397 fwif_sysinit->align_checks = 0;
398
399 fwif_sysinit->filter_flags = 0;
400 fwif_sysinit->hw_perf_filter = 0;
401 fwif_sysinit->firmware_perf = FW_PERF_CONF_NONE;
402 fwif_sysinit->initial_core_clock_speed = clock_speed_hz;
403 fwif_sysinit->active_pm_latency_ms = 0;
404 fwif_sysinit->gpio_validation_mode = ROGUE_FWIF_GPIO_VAL_OFF;
405 fwif_sysinit->firmware_started = false;
406 fwif_sysinit->marker_val = 1;
407
408 memset(&fwif_sysinit->bvnc_km_feature_flags, 0,
409 sizeof(fwif_sysinit->bvnc_km_feature_flags));
410}
411
412#define ROGUE_FWIF_SLC_MIN_SIZE_FOR_DM_OVERLAP_KB 4
413
414static void
415fw_sysdata_init(void *cpu_ptr, void *priv)
416{
417 struct rogue_fwif_sysdata *fwif_sysdata = cpu_ptr;
418 struct pvr_device *pvr_dev = priv;
419 u32 slc_size_in_kilobytes = 0;
420 u32 config_flags = 0;
421
422 WARN_ON(PVR_FEATURE_VALUE(pvr_dev, slc_size_in_kilobytes, &slc_size_in_kilobytes));
423
424 if (slc_size_in_kilobytes < ROGUE_FWIF_SLC_MIN_SIZE_FOR_DM_OVERLAP_KB)
425 config_flags |= ROGUE_FWIF_INICFG_DISABLE_DM_OVERLAP;
426
427 fwif_sysdata->config_flags = config_flags;
428}
429
430static void
431fw_runtime_cfg_init(void *cpu_ptr, void *priv)
432{
433 struct rogue_fwif_runtime_cfg *runtime_cfg = cpu_ptr;
434 struct pvr_device *pvr_dev = priv;
435 u32 clock_speed_hz = clk_get_rate(pvr_dev->core_clk);
436
437 WARN_ON(!clock_speed_hz);
438
439 runtime_cfg->core_clock_speed = clock_speed_hz;
440 runtime_cfg->active_pm_latency_ms = 0;
441 runtime_cfg->active_pm_latency_persistant = true;
442 WARN_ON(PVR_FEATURE_VALUE(pvr_dev, num_clusters,
443 &runtime_cfg->default_dusts_num_init) != 0);
444}
445
446static void
447fw_gpu_util_fwcb_init(void *cpu_ptr, void *priv)
448{
449 struct rogue_fwif_gpu_util_fwcb *gpu_util_fwcb = cpu_ptr;
450
451 gpu_util_fwcb->last_word = PVR_FWIF_GPU_UTIL_STATE_IDLE;
452}
453
454static int
455pvr_fw_create_structures(struct pvr_device *pvr_dev)
456{
457 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
458 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
459 struct pvr_fw_mem *fw_mem = &fw_dev->mem;
460 int err;
461
462 fw_dev->power_sync = pvr_fw_object_create_and_map(pvr_dev, sizeof(*fw_dev->power_sync),
463 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
464 NULL, NULL, &fw_mem->power_sync_obj);
465 if (IS_ERR(fw_dev->power_sync)) {
466 drm_err(drm_dev, "Unable to allocate FW power_sync structure\n");
467 return PTR_ERR(fw_dev->power_sync);
468 }
469
470 fw_dev->hwrinfobuf = pvr_fw_object_create_and_map(pvr_dev, sizeof(*fw_dev->hwrinfobuf),
471 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
472 NULL, NULL, &fw_mem->hwrinfobuf_obj);
473 if (IS_ERR(fw_dev->hwrinfobuf)) {
474 drm_err(drm_dev,
475 "Unable to allocate FW hwrinfobuf structure\n");
476 err = PTR_ERR(fw_dev->hwrinfobuf);
477 goto err_release_power_sync;
478 }
479
480 err = pvr_fw_object_create(pvr_dev, PVR_SYNC_OBJ_SIZE,
481 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
482 NULL, NULL, &fw_mem->mmucache_sync_obj);
483 if (err) {
484 drm_err(drm_dev,
485 "Unable to allocate MMU cache sync object\n");
486 goto err_release_hwrinfobuf;
487 }
488
489 fw_dev->fwif_sysdata = pvr_fw_object_create_and_map(pvr_dev,
490 sizeof(*fw_dev->fwif_sysdata),
491 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
492 fw_sysdata_init, pvr_dev,
493 &fw_mem->sysdata_obj);
494 if (IS_ERR(fw_dev->fwif_sysdata)) {
495 drm_err(drm_dev, "Unable to allocate FW SYSDATA structure\n");
496 err = PTR_ERR(fw_dev->fwif_sysdata);
497 goto err_release_mmucache_sync_obj;
498 }
499
500 err = pvr_fw_object_create(pvr_dev, PVR_ROGUE_FAULT_PAGE_SIZE,
501 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
502 fw_fault_page_init, NULL, &fw_mem->fault_page_obj);
503 if (err) {
504 drm_err(drm_dev, "Unable to allocate FW fault page\n");
505 goto err_release_sysdata;
506 }
507
508 err = pvr_fw_object_create(pvr_dev, sizeof(struct rogue_fwif_gpu_util_fwcb),
509 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
510 fw_gpu_util_fwcb_init, pvr_dev, &fw_mem->gpu_util_fwcb_obj);
511 if (err) {
512 drm_err(drm_dev, "Unable to allocate GPU util FWCB\n");
513 goto err_release_fault_page;
514 }
515
516 err = pvr_fw_object_create(pvr_dev, sizeof(struct rogue_fwif_runtime_cfg),
517 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
518 fw_runtime_cfg_init, pvr_dev, &fw_mem->runtime_cfg_obj);
519 if (err) {
520 drm_err(drm_dev, "Unable to allocate FW runtime config\n");
521 goto err_release_gpu_util_fwcb;
522 }
523
524 err = pvr_fw_trace_init(pvr_dev);
525 if (err)
526 goto err_release_runtime_cfg;
527
528 fw_dev->fwif_osdata = pvr_fw_object_create_and_map(pvr_dev,
529 sizeof(*fw_dev->fwif_osdata),
530 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
531 fw_osdata_init, pvr_dev,
532 &fw_mem->osdata_obj);
533 if (IS_ERR(fw_dev->fwif_osdata)) {
534 drm_err(drm_dev, "Unable to allocate FW OSDATA structure\n");
535 err = PTR_ERR(fw_dev->fwif_osdata);
536 goto err_fw_trace_fini;
537 }
538
539 fw_dev->fwif_osinit =
540 pvr_fw_object_create_and_map_offset(pvr_dev,
541 fw_dev->fw_heap_info.config_offset +
542 PVR_ROGUE_FWIF_OSINIT_OFFSET,
543 sizeof(*fw_dev->fwif_osinit),
544 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
545 fw_osinit_init, pvr_dev, &fw_mem->osinit_obj);
546 if (IS_ERR(fw_dev->fwif_osinit)) {
547 drm_err(drm_dev, "Unable to allocate FW OSINIT structure\n");
548 err = PTR_ERR(fw_dev->fwif_osinit);
549 goto err_release_osdata;
550 }
551
552 fw_dev->fwif_sysinit =
553 pvr_fw_object_create_and_map_offset(pvr_dev,
554 fw_dev->fw_heap_info.config_offset +
555 PVR_ROGUE_FWIF_SYSINIT_OFFSET,
556 sizeof(*fw_dev->fwif_sysinit),
557 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
558 fw_sysinit_init, pvr_dev, &fw_mem->sysinit_obj);
559 if (IS_ERR(fw_dev->fwif_sysinit)) {
560 drm_err(drm_dev, "Unable to allocate FW SYSINIT structure\n");
561 err = PTR_ERR(fw_dev->fwif_sysinit);
562 goto err_release_osinit;
563 }
564
565 return 0;
566
567err_release_osinit:
568 pvr_fw_object_unmap_and_destroy(fw_mem->osinit_obj);
569
570err_release_osdata:
571 pvr_fw_object_unmap_and_destroy(fw_mem->osdata_obj);
572
573err_fw_trace_fini:
574 pvr_fw_trace_fini(pvr_dev);
575
576err_release_runtime_cfg:
577 pvr_fw_object_destroy(fw_mem->runtime_cfg_obj);
578
579err_release_gpu_util_fwcb:
580 pvr_fw_object_destroy(fw_mem->gpu_util_fwcb_obj);
581
582err_release_fault_page:
583 pvr_fw_object_destroy(fw_mem->fault_page_obj);
584
585err_release_sysdata:
586 pvr_fw_object_unmap_and_destroy(fw_mem->sysdata_obj);
587
588err_release_mmucache_sync_obj:
589 pvr_fw_object_destroy(fw_mem->mmucache_sync_obj);
590
591err_release_hwrinfobuf:
592 pvr_fw_object_unmap_and_destroy(fw_mem->hwrinfobuf_obj);
593
594err_release_power_sync:
595 pvr_fw_object_unmap_and_destroy(fw_mem->power_sync_obj);
596
597 return err;
598}
599
600static void
601pvr_fw_destroy_structures(struct pvr_device *pvr_dev)
602{
603 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
604 struct pvr_fw_mem *fw_mem = &fw_dev->mem;
605
606 pvr_fw_trace_fini(pvr_dev);
607 pvr_fw_object_destroy(fw_mem->runtime_cfg_obj);
608 pvr_fw_object_destroy(fw_mem->gpu_util_fwcb_obj);
609 pvr_fw_object_destroy(fw_mem->fault_page_obj);
610 pvr_fw_object_unmap_and_destroy(fw_mem->sysdata_obj);
611 pvr_fw_object_unmap_and_destroy(fw_mem->sysinit_obj);
612
613 pvr_fw_object_destroy(fw_mem->mmucache_sync_obj);
614 pvr_fw_object_unmap_and_destroy(fw_mem->hwrinfobuf_obj);
615 pvr_fw_object_unmap_and_destroy(fw_mem->power_sync_obj);
616 pvr_fw_object_unmap_and_destroy(fw_mem->osdata_obj);
617 pvr_fw_object_unmap_and_destroy(fw_mem->osinit_obj);
618}
619
620/**
621 * pvr_fw_process() - Process firmware image, allocate FW memory and create boot
622 * arguments
623 * @pvr_dev: Device pointer.
624 *
625 * Returns:
626 * * 0 on success, or
627 * * Any error returned by pvr_fw_object_create_and_map_offset(), or
628 * * Any error returned by pvr_fw_object_create_and_map().
629 */
630static int
631pvr_fw_process(struct pvr_device *pvr_dev)
632{
633 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
634 struct pvr_fw_mem *fw_mem = &pvr_dev->fw_dev.mem;
635 const u8 *fw = pvr_dev->fw_dev.firmware->data;
636 const struct pvr_fw_layout_entry *private_data;
637 u8 *fw_code_ptr;
638 u8 *fw_data_ptr;
639 u8 *fw_core_code_ptr;
640 u8 *fw_core_data_ptr;
641 int err;
642
643 layout_get_sizes(pvr_dev);
644
645 private_data = pvr_fw_find_private_data(pvr_dev);
646 if (!private_data)
647 return -EINVAL;
648
649 /* Allocate and map memory for firmware sections. */
650
651 /*
652 * Code allocation must be at the start of the firmware heap, otherwise
653 * firmware processor will be unable to boot.
654 *
655 * This has the useful side-effect that for every other object in the
656 * driver, a firmware address of 0 is invalid.
657 */
658 fw_code_ptr = pvr_fw_object_create_and_map_offset(pvr_dev, 0, fw_mem->code_alloc_size,
659 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
660 NULL, NULL, &fw_mem->code_obj);
661 if (IS_ERR(fw_code_ptr)) {
662 drm_err(drm_dev, "Unable to allocate FW code memory\n");
663 return PTR_ERR(fw_code_ptr);
664 }
665
666 if (pvr_dev->fw_dev.defs->has_fixed_data_addr()) {
667 u32 base_addr = private_data->base_addr & pvr_dev->fw_dev.fw_heap_info.offset_mask;
668
669 fw_data_ptr =
670 pvr_fw_object_create_and_map_offset(pvr_dev, base_addr,
671 fw_mem->data_alloc_size,
672 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
673 NULL, NULL, &fw_mem->data_obj);
674 } else {
675 fw_data_ptr = pvr_fw_object_create_and_map(pvr_dev, fw_mem->data_alloc_size,
676 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
677 NULL, NULL, &fw_mem->data_obj);
678 }
679 if (IS_ERR(fw_data_ptr)) {
680 drm_err(drm_dev, "Unable to allocate FW data memory\n");
681 err = PTR_ERR(fw_data_ptr);
682 goto err_free_fw_code_obj;
683 }
684
685 /* Core code and data sections are optional. */
686 if (fw_mem->core_code_alloc_size) {
687 fw_core_code_ptr =
688 pvr_fw_object_create_and_map(pvr_dev, fw_mem->core_code_alloc_size,
689 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
690 NULL, NULL, &fw_mem->core_code_obj);
691 if (IS_ERR(fw_core_code_ptr)) {
692 drm_err(drm_dev,
693 "Unable to allocate FW core code memory\n");
694 err = PTR_ERR(fw_core_code_ptr);
695 goto err_free_fw_data_obj;
696 }
697 } else {
698 fw_core_code_ptr = NULL;
699 }
700
701 if (fw_mem->core_data_alloc_size) {
702 fw_core_data_ptr =
703 pvr_fw_object_create_and_map(pvr_dev, fw_mem->core_data_alloc_size,
704 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
705 NULL, NULL, &fw_mem->core_data_obj);
706 if (IS_ERR(fw_core_data_ptr)) {
707 drm_err(drm_dev,
708 "Unable to allocate FW core data memory\n");
709 err = PTR_ERR(fw_core_data_ptr);
710 goto err_free_fw_core_code_obj;
711 }
712 } else {
713 fw_core_data_ptr = NULL;
714 }
715
716 fw_mem->code = kzalloc(fw_mem->code_alloc_size, GFP_KERNEL);
717 fw_mem->data = kzalloc(fw_mem->data_alloc_size, GFP_KERNEL);
718 if (fw_mem->core_code_alloc_size)
719 fw_mem->core_code = kzalloc(fw_mem->core_code_alloc_size, GFP_KERNEL);
720 if (fw_mem->core_data_alloc_size)
721 fw_mem->core_data = kzalloc(fw_mem->core_data_alloc_size, GFP_KERNEL);
722
723 if (!fw_mem->code || !fw_mem->data ||
724 (!fw_mem->core_code && fw_mem->core_code_alloc_size) ||
725 (!fw_mem->core_data && fw_mem->core_data_alloc_size)) {
726 err = -ENOMEM;
727 goto err_free_kdata;
728 }
729
730 err = pvr_dev->fw_dev.defs->fw_process(pvr_dev, fw,
731 fw_mem->code, fw_mem->data, fw_mem->core_code,
732 fw_mem->core_data, fw_mem->core_code_alloc_size);
733
734 if (err)
735 goto err_free_fw_core_data_obj;
736
737 memcpy(fw_code_ptr, fw_mem->code, fw_mem->code_alloc_size);
738 memcpy(fw_data_ptr, fw_mem->data, fw_mem->data_alloc_size);
739 if (fw_mem->core_code)
740 memcpy(fw_core_code_ptr, fw_mem->core_code, fw_mem->core_code_alloc_size);
741 if (fw_mem->core_data)
742 memcpy(fw_core_data_ptr, fw_mem->core_data, fw_mem->core_data_alloc_size);
743
744 /* We're finished with the firmware section memory on the CPU, unmap. */
745 if (fw_core_data_ptr)
746 pvr_fw_object_vunmap(fw_mem->core_data_obj);
747 if (fw_core_code_ptr)
748 pvr_fw_object_vunmap(fw_mem->core_code_obj);
749 pvr_fw_object_vunmap(fw_mem->data_obj);
750 fw_data_ptr = NULL;
751 pvr_fw_object_vunmap(fw_mem->code_obj);
752 fw_code_ptr = NULL;
753
754 err = pvr_fw_create_fwif_connection_ctl(pvr_dev);
755 if (err)
756 goto err_free_fw_core_data_obj;
757
758 return 0;
759
760err_free_kdata:
761 kfree(fw_mem->core_data);
762 kfree(fw_mem->core_code);
763 kfree(fw_mem->data);
764 kfree(fw_mem->code);
765
766err_free_fw_core_data_obj:
767 if (fw_core_data_ptr)
768 pvr_fw_object_unmap_and_destroy(fw_mem->core_data_obj);
769
770err_free_fw_core_code_obj:
771 if (fw_core_code_ptr)
772 pvr_fw_object_unmap_and_destroy(fw_mem->core_code_obj);
773
774err_free_fw_data_obj:
775 if (fw_data_ptr)
776 pvr_fw_object_vunmap(fw_mem->data_obj);
777 pvr_fw_object_destroy(fw_mem->data_obj);
778
779err_free_fw_code_obj:
780 if (fw_code_ptr)
781 pvr_fw_object_vunmap(fw_mem->code_obj);
782 pvr_fw_object_destroy(fw_mem->code_obj);
783
784 return err;
785}
786
787static int
788pvr_copy_to_fw(struct pvr_fw_object *dest_obj, u8 *src_ptr, u32 size)
789{
790 u8 *dest_ptr = pvr_fw_object_vmap(dest_obj);
791
792 if (IS_ERR(dest_ptr))
793 return PTR_ERR(dest_ptr);
794
795 memcpy(dest_ptr, src_ptr, size);
796
797 pvr_fw_object_vunmap(dest_obj);
798
799 return 0;
800}
801
802static int
803pvr_fw_reinit_code_data(struct pvr_device *pvr_dev)
804{
805 struct pvr_fw_mem *fw_mem = &pvr_dev->fw_dev.mem;
806 int err;
807
808 err = pvr_copy_to_fw(fw_mem->code_obj, fw_mem->code, fw_mem->code_alloc_size);
809 if (err)
810 return err;
811
812 err = pvr_copy_to_fw(fw_mem->data_obj, fw_mem->data, fw_mem->data_alloc_size);
813 if (err)
814 return err;
815
816 if (fw_mem->core_code) {
817 err = pvr_copy_to_fw(fw_mem->core_code_obj, fw_mem->core_code,
818 fw_mem->core_code_alloc_size);
819 if (err)
820 return err;
821 }
822
823 if (fw_mem->core_data) {
824 err = pvr_copy_to_fw(fw_mem->core_data_obj, fw_mem->core_data,
825 fw_mem->core_data_alloc_size);
826 if (err)
827 return err;
828 }
829
830 return 0;
831}
832
833static void
834pvr_fw_cleanup(struct pvr_device *pvr_dev)
835{
836 struct pvr_fw_mem *fw_mem = &pvr_dev->fw_dev.mem;
837
838 pvr_fw_fini_fwif_connection_ctl(pvr_dev);
839 if (fw_mem->core_code_obj)
840 pvr_fw_object_destroy(fw_mem->core_code_obj);
841 if (fw_mem->core_data_obj)
842 pvr_fw_object_destroy(fw_mem->core_data_obj);
843 pvr_fw_object_destroy(fw_mem->code_obj);
844 pvr_fw_object_destroy(fw_mem->data_obj);
845}
846
847/**
848 * pvr_wait_for_fw_boot() - Wait for firmware to finish booting
849 * @pvr_dev: Target PowerVR device.
850 *
851 * Returns:
852 * * 0 on success, or
853 * * -%ETIMEDOUT if firmware fails to boot within timeout.
854 */
855int
856pvr_wait_for_fw_boot(struct pvr_device *pvr_dev)
857{
858 ktime_t deadline = ktime_add_us(ktime_get(), FW_BOOT_TIMEOUT_USEC);
859 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
860
861 while (ktime_to_ns(ktime_sub(deadline, ktime_get())) > 0) {
862 if (READ_ONCE(fw_dev->fwif_sysinit->firmware_started))
863 return 0;
864 }
865
866 return -ETIMEDOUT;
867}
868
869/*
870 * pvr_fw_heap_info_init() - Calculate size and masks for FW heap
871 * @pvr_dev: Target PowerVR device.
872 * @log2_size: Log2 of raw heap size.
873 * @reserved_size: Size of reserved area of heap, in bytes. May be zero.
874 */
875void
876pvr_fw_heap_info_init(struct pvr_device *pvr_dev, u32 log2_size, u32 reserved_size)
877{
878 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
879
880 fw_dev->fw_heap_info.gpu_addr = PVR_ROGUE_FW_MAIN_HEAP_BASE;
881 fw_dev->fw_heap_info.log2_size = log2_size;
882 fw_dev->fw_heap_info.reserved_size = reserved_size;
883 fw_dev->fw_heap_info.raw_size = 1 << fw_dev->fw_heap_info.log2_size;
884 fw_dev->fw_heap_info.offset_mask = fw_dev->fw_heap_info.raw_size - 1;
885 fw_dev->fw_heap_info.config_offset = fw_dev->fw_heap_info.raw_size -
886 PVR_ROGUE_FW_CONFIG_HEAP_SIZE;
887 fw_dev->fw_heap_info.size = fw_dev->fw_heap_info.raw_size -
888 (PVR_ROGUE_FW_CONFIG_HEAP_SIZE + reserved_size);
889}
890
891/**
892 * pvr_fw_validate_init_device_info() - Validate firmware and initialise device information
893 * @pvr_dev: Target PowerVR device.
894 *
895 * This function must be called before querying device information.
896 *
897 * Returns:
898 * * 0 on success, or
899 * * -%EINVAL if firmware validation fails.
900 */
901int
902pvr_fw_validate_init_device_info(struct pvr_device *pvr_dev)
903{
904 int err;
905
906 err = pvr_fw_validate(pvr_dev);
907 if (err)
908 return err;
909
910 return pvr_fw_get_device_info(pvr_dev);
911}
912
913/**
914 * pvr_fw_init() - Initialise and boot firmware
915 * @pvr_dev: Target PowerVR device
916 *
917 * On successful completion of the function the PowerVR device will be
918 * initialised and ready to use.
919 *
920 * Returns:
921 * * 0 on success,
922 * * -%EINVAL on invalid firmware image,
923 * * -%ENOMEM on out of memory, or
924 * * -%ETIMEDOUT if firmware processor fails to boot or on register poll timeout.
925 */
926int
927pvr_fw_init(struct pvr_device *pvr_dev)
928{
929 u32 kccb_size_log2 = ROGUE_FWIF_KCCB_NUMCMDS_LOG2_DEFAULT;
930 u32 kccb_rtn_size = (1 << kccb_size_log2) * sizeof(*pvr_dev->kccb.rtn);
931 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
932 int err;
933
934 if (fw_dev->processor_type == PVR_FW_PROCESSOR_TYPE_META)
935 fw_dev->defs = &pvr_fw_defs_meta;
936 else if (fw_dev->processor_type == PVR_FW_PROCESSOR_TYPE_MIPS)
937 fw_dev->defs = &pvr_fw_defs_mips;
938 else
939 return -EINVAL;
940
941 err = fw_dev->defs->init(pvr_dev);
942 if (err)
943 return err;
944
945 drm_mm_init(&fw_dev->fw_mm, ROGUE_FW_HEAP_BASE, fw_dev->fw_heap_info.raw_size);
946 fw_dev->fw_mm_base = ROGUE_FW_HEAP_BASE;
947 spin_lock_init(&fw_dev->fw_mm_lock);
948
949 INIT_LIST_HEAD(&fw_dev->fw_objs.list);
950 err = drmm_mutex_init(from_pvr_device(pvr_dev), &fw_dev->fw_objs.lock);
951 if (err)
952 goto err_mm_takedown;
953
954 err = pvr_fw_process(pvr_dev);
955 if (err)
956 goto err_mm_takedown;
957
958 /* Initialise KCCB and FWCCB. */
959 err = pvr_kccb_init(pvr_dev);
960 if (err)
961 goto err_fw_cleanup;
962
963 err = pvr_fwccb_init(pvr_dev);
964 if (err)
965 goto err_kccb_fini;
966
967 /* Allocate memory for KCCB return slots. */
968 pvr_dev->kccb.rtn = pvr_fw_object_create_and_map(pvr_dev, kccb_rtn_size,
969 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
970 NULL, NULL, &pvr_dev->kccb.rtn_obj);
971 if (IS_ERR(pvr_dev->kccb.rtn)) {
972 err = PTR_ERR(pvr_dev->kccb.rtn);
973 goto err_fwccb_fini;
974 }
975
976 err = pvr_fw_create_structures(pvr_dev);
977 if (err)
978 goto err_kccb_rtn_release;
979
980 err = pvr_fw_start(pvr_dev);
981 if (err)
982 goto err_destroy_structures;
983
984 err = pvr_wait_for_fw_boot(pvr_dev);
985 if (err) {
986 drm_err(from_pvr_device(pvr_dev), "Firmware failed to boot\n");
987 goto err_fw_stop;
988 }
989
990 fw_dev->booted = true;
991
992 return 0;
993
994err_fw_stop:
995 pvr_fw_stop(pvr_dev);
996
997err_destroy_structures:
998 pvr_fw_destroy_structures(pvr_dev);
999
1000err_kccb_rtn_release:
1001 pvr_fw_object_unmap_and_destroy(pvr_dev->kccb.rtn_obj);
1002
1003err_fwccb_fini:
1004 pvr_ccb_fini(&pvr_dev->fwccb);
1005
1006err_kccb_fini:
1007 pvr_kccb_fini(pvr_dev);
1008
1009err_fw_cleanup:
1010 pvr_fw_cleanup(pvr_dev);
1011
1012err_mm_takedown:
1013 drm_mm_takedown(&fw_dev->fw_mm);
1014
1015 if (fw_dev->defs->fini)
1016 fw_dev->defs->fini(pvr_dev);
1017
1018 return err;
1019}
1020
1021/**
1022 * pvr_fw_fini() - Shutdown firmware processor and free associated memory
1023 * @pvr_dev: Target PowerVR device
1024 */
1025void
1026pvr_fw_fini(struct pvr_device *pvr_dev)
1027{
1028 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
1029
1030 fw_dev->booted = false;
1031
1032 pvr_fw_destroy_structures(pvr_dev);
1033 pvr_fw_object_unmap_and_destroy(pvr_dev->kccb.rtn_obj);
1034
1035 /*
1036 * Ensure FWCCB worker has finished executing before destroying FWCCB. The IRQ handler has
1037 * been unregistered at this point so no new work should be being submitted.
1038 */
1039 pvr_ccb_fini(&pvr_dev->fwccb);
1040 pvr_kccb_fini(pvr_dev);
1041 pvr_fw_cleanup(pvr_dev);
1042
1043 mutex_lock(&pvr_dev->fw_dev.fw_objs.lock);
1044 WARN_ON(!list_empty(&pvr_dev->fw_dev.fw_objs.list));
1045 mutex_unlock(&pvr_dev->fw_dev.fw_objs.lock);
1046
1047 drm_mm_takedown(&fw_dev->fw_mm);
1048
1049 if (fw_dev->defs->fini)
1050 fw_dev->defs->fini(pvr_dev);
1051}
1052
1053/**
1054 * pvr_fw_mts_schedule() - Schedule work via an MTS kick
1055 * @pvr_dev: Target PowerVR device
1056 * @val: Kick mask. Should be a combination of %ROGUE_CR_MTS_SCHEDULE_*
1057 */
1058void
1059pvr_fw_mts_schedule(struct pvr_device *pvr_dev, u32 val)
1060{
1061 /* Ensure memory is flushed before kicking MTS. */
1062 wmb();
1063
1064 pvr_cr_write32(pvr_dev, ROGUE_CR_MTS_SCHEDULE, val);
1065
1066 /* Ensure the MTS kick goes through before continuing. */
1067 mb();
1068}
1069
1070/**
1071 * pvr_fw_structure_cleanup() - Send FW cleanup request for an object
1072 * @pvr_dev: Target PowerVR device.
1073 * @type: Type of object to cleanup. Must be one of &enum rogue_fwif_cleanup_type.
1074 * @fw_obj: Pointer to FW object containing object to cleanup.
1075 * @offset: Offset within FW object of object to cleanup.
1076 *
1077 * Returns:
1078 * * 0 on success,
1079 * * -EBUSY if object is busy,
1080 * * -ETIMEDOUT on timeout, or
1081 * * -EIO if device is lost.
1082 */
1083int
1084pvr_fw_structure_cleanup(struct pvr_device *pvr_dev, u32 type, struct pvr_fw_object *fw_obj,
1085 u32 offset)
1086{
1087 struct rogue_fwif_kccb_cmd cmd;
1088 int slot_nr;
1089 int idx;
1090 int err;
1091 u32 rtn;
1092
1093 struct rogue_fwif_cleanup_request *cleanup_req = &cmd.cmd_data.cleanup_data;
1094
1095 down_read(&pvr_dev->reset_sem);
1096
1097 if (!drm_dev_enter(from_pvr_device(pvr_dev), &idx)) {
1098 err = -EIO;
1099 goto err_up_read;
1100 }
1101
1102 cmd.cmd_type = ROGUE_FWIF_KCCB_CMD_CLEANUP;
1103 cmd.kccb_flags = 0;
1104 cleanup_req->cleanup_type = type;
1105
1106 switch (type) {
1107 case ROGUE_FWIF_CLEANUP_FWCOMMONCONTEXT:
1108 pvr_fw_object_get_fw_addr_offset(fw_obj, offset,
1109 &cleanup_req->cleanup_data.context_fw_addr);
1110 break;
1111 case ROGUE_FWIF_CLEANUP_HWRTDATA:
1112 pvr_fw_object_get_fw_addr_offset(fw_obj, offset,
1113 &cleanup_req->cleanup_data.hwrt_data_fw_addr);
1114 break;
1115 case ROGUE_FWIF_CLEANUP_FREELIST:
1116 pvr_fw_object_get_fw_addr_offset(fw_obj, offset,
1117 &cleanup_req->cleanup_data.freelist_fw_addr);
1118 break;
1119 default:
1120 err = -EINVAL;
1121 goto err_drm_dev_exit;
1122 }
1123
1124 err = pvr_kccb_send_cmd(pvr_dev, &cmd, &slot_nr);
1125 if (err)
1126 goto err_drm_dev_exit;
1127
1128 err = pvr_kccb_wait_for_completion(pvr_dev, slot_nr, HZ, &rtn);
1129 if (err)
1130 goto err_drm_dev_exit;
1131
1132 if (rtn & ROGUE_FWIF_KCCB_RTN_SLOT_CLEANUP_BUSY)
1133 err = -EBUSY;
1134
1135err_drm_dev_exit:
1136 drm_dev_exit(idx);
1137
1138err_up_read:
1139 up_read(&pvr_dev->reset_sem);
1140
1141 return err;
1142}
1143
1144/**
1145 * pvr_fw_object_fw_map() - Map a FW object in firmware address space
1146 * @pvr_dev: Device pointer.
1147 * @fw_obj: FW object to map.
1148 * @dev_addr: Desired address in device space, if a specific address is
1149 * required. 0 otherwise.
1150 *
1151 * Returns:
1152 * * 0 on success, or
1153 * * -%EINVAL if @fw_obj is already mapped but has no references, or
1154 * * Any error returned by DRM.
1155 */
1156static int
1157pvr_fw_object_fw_map(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj, u64 dev_addr)
1158{
1159 struct pvr_gem_object *pvr_obj = fw_obj->gem;
1160 struct drm_gem_object *gem_obj = gem_from_pvr_gem(pvr_obj);
1161 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
1162
1163 int err;
1164
1165 spin_lock(&fw_dev->fw_mm_lock);
1166
1167 if (drm_mm_node_allocated(&fw_obj->fw_mm_node)) {
1168 err = -EINVAL;
1169 goto err_unlock;
1170 }
1171
1172 if (!dev_addr) {
1173 /*
1174 * Allocate from the main heap only (firmware heap minus
1175 * config space).
1176 */
1177 err = drm_mm_insert_node_in_range(&fw_dev->fw_mm, &fw_obj->fw_mm_node,
1178 gem_obj->size, 0, 0,
1179 fw_dev->fw_heap_info.gpu_addr,
1180 fw_dev->fw_heap_info.gpu_addr +
1181 fw_dev->fw_heap_info.size, 0);
1182 if (err)
1183 goto err_unlock;
1184 } else {
1185 fw_obj->fw_mm_node.start = dev_addr;
1186 fw_obj->fw_mm_node.size = gem_obj->size;
1187 err = drm_mm_reserve_node(&fw_dev->fw_mm, &fw_obj->fw_mm_node);
1188 if (err)
1189 goto err_unlock;
1190 }
1191
1192 spin_unlock(&fw_dev->fw_mm_lock);
1193
1194 /* Map object on GPU. */
1195 err = fw_dev->defs->vm_map(pvr_dev, fw_obj);
1196 if (err)
1197 goto err_remove_node;
1198
1199 fw_obj->fw_addr_offset = (u32)(fw_obj->fw_mm_node.start - fw_dev->fw_mm_base);
1200
1201 return 0;
1202
1203err_remove_node:
1204 spin_lock(&fw_dev->fw_mm_lock);
1205 drm_mm_remove_node(&fw_obj->fw_mm_node);
1206
1207err_unlock:
1208 spin_unlock(&fw_dev->fw_mm_lock);
1209
1210 return err;
1211}
1212
1213/**
1214 * pvr_fw_object_fw_unmap() - Unmap a previously mapped FW object
1215 * @fw_obj: FW object to unmap.
1216 *
1217 * Returns:
1218 * * 0 on success, or
1219 * * -%EINVAL if object is not currently mapped.
1220 */
1221static int
1222pvr_fw_object_fw_unmap(struct pvr_fw_object *fw_obj)
1223{
1224 struct pvr_gem_object *pvr_obj = fw_obj->gem;
1225 struct drm_gem_object *gem_obj = gem_from_pvr_gem(pvr_obj);
1226 struct pvr_device *pvr_dev = to_pvr_device(gem_obj->dev);
1227 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
1228
1229 fw_dev->defs->vm_unmap(pvr_dev, fw_obj);
1230
1231 spin_lock(&fw_dev->fw_mm_lock);
1232
1233 if (!drm_mm_node_allocated(&fw_obj->fw_mm_node)) {
1234 spin_unlock(&fw_dev->fw_mm_lock);
1235 return -EINVAL;
1236 }
1237
1238 drm_mm_remove_node(&fw_obj->fw_mm_node);
1239
1240 spin_unlock(&fw_dev->fw_mm_lock);
1241
1242 return 0;
1243}
1244
1245static void *
1246pvr_fw_object_create_and_map_common(struct pvr_device *pvr_dev, size_t size,
1247 u64 flags, u64 dev_addr,
1248 void (*init)(void *cpu_ptr, void *priv),
1249 void *init_priv, struct pvr_fw_object **fw_obj_out)
1250{
1251 struct pvr_fw_object *fw_obj;
1252 void *cpu_ptr;
1253 int err;
1254
1255 /* %DRM_PVR_BO_PM_FW_PROTECT is implicit for FW objects. */
1256 flags |= DRM_PVR_BO_PM_FW_PROTECT;
1257
1258 fw_obj = kzalloc(sizeof(*fw_obj), GFP_KERNEL);
1259 if (!fw_obj)
1260 return ERR_PTR(-ENOMEM);
1261
1262 INIT_LIST_HEAD(&fw_obj->node);
1263 fw_obj->init = init;
1264 fw_obj->init_priv = init_priv;
1265
1266 fw_obj->gem = pvr_gem_object_create(pvr_dev, size, flags);
1267 if (IS_ERR(fw_obj->gem)) {
1268 err = PTR_ERR(fw_obj->gem);
1269 fw_obj->gem = NULL;
1270 goto err_put_object;
1271 }
1272
1273 err = pvr_fw_object_fw_map(pvr_dev, fw_obj, dev_addr);
1274 if (err)
1275 goto err_put_object;
1276
1277 cpu_ptr = pvr_fw_object_vmap(fw_obj);
1278 if (IS_ERR(cpu_ptr)) {
1279 err = PTR_ERR(cpu_ptr);
1280 goto err_put_object;
1281 }
1282
1283 *fw_obj_out = fw_obj;
1284
1285 if (fw_obj->init)
1286 fw_obj->init(cpu_ptr, fw_obj->init_priv);
1287
1288 mutex_lock(&pvr_dev->fw_dev.fw_objs.lock);
1289 list_add_tail(&fw_obj->node, &pvr_dev->fw_dev.fw_objs.list);
1290 mutex_unlock(&pvr_dev->fw_dev.fw_objs.lock);
1291
1292 return cpu_ptr;
1293
1294err_put_object:
1295 pvr_fw_object_destroy(fw_obj);
1296
1297 return ERR_PTR(err);
1298}
1299
1300/**
1301 * pvr_fw_object_create() - Create a FW object and map to firmware
1302 * @pvr_dev: PowerVR device pointer.
1303 * @size: Size of object, in bytes.
1304 * @flags: Options which affect both this operation and future mapping
1305 * operations performed on the returned object. Must be a combination of
1306 * DRM_PVR_BO_* and/or PVR_BO_* flags.
1307 * @init: Initialisation callback.
1308 * @init_priv: Private pointer to pass to initialisation callback.
1309 * @fw_obj_out: Pointer to location to store created object pointer.
1310 *
1311 * %DRM_PVR_BO_DEVICE_PM_FW_PROTECT is implied for all FW objects. Consequently,
1312 * this function will fail if @flags has %DRM_PVR_BO_CPU_ALLOW_USERSPACE_ACCESS
1313 * set.
1314 *
1315 * Returns:
1316 * * 0 on success, or
1317 * * Any error returned by pvr_fw_object_create_common().
1318 */
1319int
1320pvr_fw_object_create(struct pvr_device *pvr_dev, size_t size, u64 flags,
1321 void (*init)(void *cpu_ptr, void *priv), void *init_priv,
1322 struct pvr_fw_object **fw_obj_out)
1323{
1324 void *cpu_ptr;
1325
1326 cpu_ptr = pvr_fw_object_create_and_map_common(pvr_dev, size, flags, 0, init, init_priv,
1327 fw_obj_out);
1328 if (IS_ERR(cpu_ptr))
1329 return PTR_ERR(cpu_ptr);
1330
1331 pvr_fw_object_vunmap(*fw_obj_out);
1332
1333 return 0;
1334}
1335
1336/**
1337 * pvr_fw_object_create_and_map() - Create a FW object and map to firmware and CPU
1338 * @pvr_dev: PowerVR device pointer.
1339 * @size: Size of object, in bytes.
1340 * @flags: Options which affect both this operation and future mapping
1341 * operations performed on the returned object. Must be a combination of
1342 * DRM_PVR_BO_* and/or PVR_BO_* flags.
1343 * @init: Initialisation callback.
1344 * @init_priv: Private pointer to pass to initialisation callback.
1345 * @fw_obj_out: Pointer to location to store created object pointer.
1346 *
1347 * %DRM_PVR_BO_DEVICE_PM_FW_PROTECT is implied for all FW objects. Consequently,
1348 * this function will fail if @flags has %DRM_PVR_BO_CPU_ALLOW_USERSPACE_ACCESS
1349 * set.
1350 *
1351 * Caller is responsible for calling pvr_fw_object_vunmap() to release the CPU
1352 * mapping.
1353 *
1354 * Returns:
1355 * * Pointer to CPU mapping of newly created object, or
1356 * * Any error returned by pvr_fw_object_create(), or
1357 * * Any error returned by pvr_fw_object_vmap().
1358 */
1359void *
1360pvr_fw_object_create_and_map(struct pvr_device *pvr_dev, size_t size, u64 flags,
1361 void (*init)(void *cpu_ptr, void *priv),
1362 void *init_priv, struct pvr_fw_object **fw_obj_out)
1363{
1364 return pvr_fw_object_create_and_map_common(pvr_dev, size, flags, 0, init, init_priv,
1365 fw_obj_out);
1366}
1367
1368/**
1369 * pvr_fw_object_create_and_map_offset() - Create a FW object and map to
1370 * firmware at the provided offset and to the CPU.
1371 * @pvr_dev: PowerVR device pointer.
1372 * @dev_offset: Base address of desired FW mapping, offset from start of FW heap.
1373 * @size: Size of object, in bytes.
1374 * @flags: Options which affect both this operation and future mapping
1375 * operations performed on the returned object. Must be a combination of
1376 * DRM_PVR_BO_* and/or PVR_BO_* flags.
1377 * @init: Initialisation callback.
1378 * @init_priv: Private pointer to pass to initialisation callback.
1379 * @fw_obj_out: Pointer to location to store created object pointer.
1380 *
1381 * %DRM_PVR_BO_DEVICE_PM_FW_PROTECT is implied for all FW objects. Consequently,
1382 * this function will fail if @flags has %DRM_PVR_BO_CPU_ALLOW_USERSPACE_ACCESS
1383 * set.
1384 *
1385 * Caller is responsible for calling pvr_fw_object_vunmap() to release the CPU
1386 * mapping.
1387 *
1388 * Returns:
1389 * * Pointer to CPU mapping of newly created object, or
1390 * * Any error returned by pvr_fw_object_create(), or
1391 * * Any error returned by pvr_fw_object_vmap().
1392 */
1393void *
1394pvr_fw_object_create_and_map_offset(struct pvr_device *pvr_dev,
1395 u32 dev_offset, size_t size, u64 flags,
1396 void (*init)(void *cpu_ptr, void *priv),
1397 void *init_priv, struct pvr_fw_object **fw_obj_out)
1398{
1399 u64 dev_addr = pvr_dev->fw_dev.fw_mm_base + dev_offset;
1400
1401 return pvr_fw_object_create_and_map_common(pvr_dev, size, flags, dev_addr, init, init_priv,
1402 fw_obj_out);
1403}
1404
1405/**
1406 * pvr_fw_object_destroy() - Destroy a pvr_fw_object
1407 * @fw_obj: Pointer to object to destroy.
1408 */
1409void pvr_fw_object_destroy(struct pvr_fw_object *fw_obj)
1410{
1411 struct pvr_gem_object *pvr_obj = fw_obj->gem;
1412 struct drm_gem_object *gem_obj = gem_from_pvr_gem(pvr_obj);
1413 struct pvr_device *pvr_dev = to_pvr_device(gem_obj->dev);
1414
1415 mutex_lock(&pvr_dev->fw_dev.fw_objs.lock);
1416 list_del(&fw_obj->node);
1417 mutex_unlock(&pvr_dev->fw_dev.fw_objs.lock);
1418
1419 if (drm_mm_node_allocated(&fw_obj->fw_mm_node)) {
1420 /* If we can't unmap, leak the memory. */
1421 if (WARN_ON(pvr_fw_object_fw_unmap(fw_obj)))
1422 return;
1423 }
1424
1425 if (fw_obj->gem)
1426 pvr_gem_object_put(fw_obj->gem);
1427
1428 kfree(fw_obj);
1429}
1430
1431/**
1432 * pvr_fw_object_get_fw_addr_offset() - Return address of object in firmware address space, with
1433 * given offset.
1434 * @fw_obj: Pointer to object.
1435 * @offset: Desired offset from start of object.
1436 * @fw_addr_out: Location to store address to.
1437 */
1438void pvr_fw_object_get_fw_addr_offset(struct pvr_fw_object *fw_obj, u32 offset, u32 *fw_addr_out)
1439{
1440 struct pvr_gem_object *pvr_obj = fw_obj->gem;
1441 struct pvr_device *pvr_dev = to_pvr_device(gem_from_pvr_gem(pvr_obj)->dev);
1442
1443 *fw_addr_out = pvr_dev->fw_dev.defs->get_fw_addr_with_offset(fw_obj, offset);
1444}
1445
1446/*
1447 * pvr_fw_hard_reset() - Re-initialise the FW code and data segments, and reset all global FW
1448 * structures
1449 * @pvr_dev: Device pointer
1450 *
1451 * If this function returns an error then the caller must regard the device as lost.
1452 *
1453 * Returns:
1454 * * 0 on success, or
1455 * * Any error returned by pvr_fw_init_dev_structures() or pvr_fw_reset_all().
1456 */
1457int
1458pvr_fw_hard_reset(struct pvr_device *pvr_dev)
1459{
1460 struct list_head *pos;
1461 int err;
1462
1463 /* Reset all FW objects */
1464 mutex_lock(&pvr_dev->fw_dev.fw_objs.lock);
1465
1466 list_for_each(pos, &pvr_dev->fw_dev.fw_objs.list) {
1467 struct pvr_fw_object *fw_obj = container_of(pos, struct pvr_fw_object, node);
1468 void *cpu_ptr = pvr_fw_object_vmap(fw_obj);
1469
1470 WARN_ON(IS_ERR(cpu_ptr));
1471
1472 if (!(fw_obj->gem->flags & PVR_BO_FW_NO_CLEAR_ON_RESET)) {
1473 memset(cpu_ptr, 0, pvr_gem_object_size(fw_obj->gem));
1474
1475 if (fw_obj->init)
1476 fw_obj->init(cpu_ptr, fw_obj->init_priv);
1477 }
1478
1479 pvr_fw_object_vunmap(fw_obj);
1480 }
1481
1482 mutex_unlock(&pvr_dev->fw_dev.fw_objs.lock);
1483
1484 err = pvr_fw_reinit_code_data(pvr_dev);
1485 if (err)
1486 return err;
1487
1488 return 0;
1489}
drivers/gpu/drm/imagination/pvr_fw.h+508−0
@@ -0,0 +1,508 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_FW_H
5#define PVR_FW_H
6
7#include "pvr_fw_info.h"
8#include "pvr_fw_trace.h"
9#include "pvr_gem.h"
10
11#include <drm/drm_mm.h>
12
13#include <linux/types.h>
14
15/* Forward declarations from "pvr_device.h". */
16struct pvr_device;
17struct pvr_file;
18
19/* Forward declaration from "pvr_vm.h". */
20struct pvr_vm_context;
21
22#define ROGUE_FWIF_FWCCB_NUMCMDS_LOG2 5
23
24#define ROGUE_FWIF_KCCB_NUMCMDS_LOG2_DEFAULT 7
25
26/**
27 * struct pvr_fw_object - container for firmware memory allocations
28 */
29struct pvr_fw_object {
30 /** @ref_count: FW object reference counter. */
31 struct kref ref_count;
32
33 /** @gem: GEM object backing the FW object. */
34 struct pvr_gem_object *gem;
35
36 /**
37 * @fw_mm_node: Node representing mapping in FW address space. @pvr_obj->lock must
38 * be held when writing.
39 */
40 struct drm_mm_node fw_mm_node;
41
42 /**
43 * @fw_addr_offset: Virtual address offset of firmware mapping. Only
44 * valid if @flags has %PVR_GEM_OBJECT_FLAGS_FW_MAPPED
45 * set.
46 */
47 u32 fw_addr_offset;
48
49 /**
50 * @init: Initialisation callback. Will be called on object creation and FW hard reset.
51 * Object will have been zeroed before this is called.
52 */
53 void (*init)(void *cpu_ptr, void *priv);
54
55 /** @init_priv: Private data for initialisation callback. */
56 void *init_priv;
57
58 /** @node: Node for firmware object list. */
59 struct list_head node;
60};
61
62/**
63 * struct pvr_fw_defs - FW processor function table and static definitions
64 */
65struct pvr_fw_defs {
66 /**
67 * @init:
68 *
69 * FW processor specific initialisation.
70 * @pvr_dev: Target PowerVR device.
71 *
72 * This function must call pvr_fw_heap_calculate() to initialise the firmware heap for this
73 * FW processor.
74 *
75 * This function is mandatory.
76 *
77 * Returns:
78 * * 0 on success, or
79 * * Any appropriate error on failure.
80 */
81 int (*init)(struct pvr_device *pvr_dev);
82
83 /**
84 * @fini:
85 *
86 * FW processor specific finalisation.
87 * @pvr_dev: Target PowerVR device.
88 *
89 * This function is optional.
90 */
91 void (*fini)(struct pvr_device *pvr_dev);
92
93 /**
94 * @fw_process:
95 *
96 * Load and process firmware image.
97 * @pvr_dev: Target PowerVR device.
98 * @fw: Pointer to firmware image.
99 * @fw_code_ptr: Pointer to firmware code section.
100 * @fw_data_ptr: Pointer to firmware data section.
101 * @fw_core_code_ptr: Pointer to firmware core code section. May be %NULL.
102 * @fw_core_data_ptr: Pointer to firmware core data section. May be %NULL.
103 * @core_code_alloc_size: Total allocation size of core code section.
104 *
105 * This function is mandatory.
106 *
107 * Returns:
108 * * 0 on success, or
109 * * Any appropriate error on failure.
110 */
111 int (*fw_process)(struct pvr_device *pvr_dev, const u8 *fw,
112 u8 *fw_code_ptr, u8 *fw_data_ptr, u8 *fw_core_code_ptr,
113 u8 *fw_core_data_ptr, u32 core_code_alloc_size);
114
115 /**
116 * @vm_map:
117 *
118 * Map FW object into FW processor address space.
119 * @pvr_dev: Target PowerVR device.
120 * @fw_obj: FW object to map.
121 *
122 * This function is mandatory.
123 *
124 * Returns:
125 * * 0 on success, or
126 * * Any appropriate error on failure.
127 */
128 int (*vm_map)(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj);
129
130 /**
131 * @vm_unmap:
132 *
133 * Unmap FW object from FW processor address space.
134 * @pvr_dev: Target PowerVR device.
135 * @fw_obj: FW object to map.
136 *
137 * This function is mandatory.
138 */
139 void (*vm_unmap)(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj);
140
141 /**
142 * @get_fw_addr_with_offset:
143 *
144 * Called to get address of object in firmware address space, with offset.
145 * @fw_obj: Pointer to object.
146 * @offset: Desired offset from start of object.
147 *
148 * This function is mandatory.
149 *
150 * Returns:
151 * * Address in firmware address space.
152 */
153 u32 (*get_fw_addr_with_offset)(struct pvr_fw_object *fw_obj, u32 offset);
154
155 /**
156 * @wrapper_init:
157 *
158 * Called to initialise FW wrapper.
159 * @pvr_dev: Target PowerVR device.
160 *
161 * This function is mandatory.
162 *
163 * Returns:
164 * * 0 on success.
165 * * Any appropriate error on failure.
166 */
167 int (*wrapper_init)(struct pvr_device *pvr_dev);
168
169 /**
170 * @has_fixed_data_addr:
171 *
172 * Called to check if firmware fixed data must be loaded at the address given by the
173 * firmware layout table.
174 *
175 * This function is mandatory.
176 *
177 * Returns:
178 * * %true if firmware fixed data must be loaded at the address given by the firmware
179 * layout table.
180 * * %false otherwise.
181 */
182 bool (*has_fixed_data_addr)(void);
183
184 /**
185 * @irq: FW Interrupt information.
186 *
187 * Those are processor dependent, and should be initialized by the
188 * processor backend in pvr_fw_funcs::init().
189 */
190 struct {
191 /** @enable_reg: FW interrupt enable register. */
192 u32 enable_reg;
193
194 /** @status_reg: FW interrupt status register. */
195 u32 status_reg;
196
197 /**
198 * @clear_reg: FW interrupt clear register.
199 *
200 * If @status_reg == @clear_reg, we clear by write a bit to zero,
201 * otherwise we clear by writing a bit to one.
202 */
203 u32 clear_reg;
204
205 /** @event_mask: Bitmask of events to listen for. */
206 u32 event_mask;
207
208 /** @clear_mask: Value to write to the clear_reg in order to clear FW IRQs. */
209 u32 clear_mask;
210 } irq;
211};
212
213/**
214 * struct pvr_fw_mem - FW memory allocations
215 */
216struct pvr_fw_mem {
217 /** @code_obj: Object representing firmware code. */
218 struct pvr_fw_object *code_obj;
219
220 /** @data_obj: Object representing firmware data. */
221 struct pvr_fw_object *data_obj;
222
223 /**
224 * @core_code_obj: Object representing firmware core code. May be
225 * %NULL if firmware does not contain this section.
226 */
227 struct pvr_fw_object *core_code_obj;
228
229 /**
230 * @core_data_obj: Object representing firmware core data. May be
231 * %NULL if firmware does not contain this section.
232 */
233 struct pvr_fw_object *core_data_obj;
234
235 /** @code: Driver-side copy of firmware code. */
236 u8 *code;
237
238 /** @data: Driver-side copy of firmware data. */
239 u8 *data;
240
241 /**
242 * @core_code: Driver-side copy of firmware core code. May be %NULL if firmware does not
243 * contain this section.
244 */
245 u8 *core_code;
246
247 /**
248 * @core_data: Driver-side copy of firmware core data. May be %NULL if firmware does not
249 * contain this section.
250 */
251 u8 *core_data;
252
253 /** @code_alloc_size: Allocation size of firmware code section. */
254 u32 code_alloc_size;
255
256 /** @data_alloc_size: Allocation size of firmware data section. */
257 u32 data_alloc_size;
258
259 /** @core_code_alloc_size: Allocation size of firmware core code section. */
260 u32 core_code_alloc_size;
261
262 /** @core_data_alloc_size: Allocation size of firmware core data section. */
263 u32 core_data_alloc_size;
264
265 /**
266 * @fwif_connection_ctl_obj: Object representing FWIF connection control
267 * structure.
268 */
269 struct pvr_fw_object *fwif_connection_ctl_obj;
270
271 /** @osinit_obj: Object representing FW OSINIT structure. */
272 struct pvr_fw_object *osinit_obj;
273
274 /** @sysinit_obj: Object representing FW SYSINIT structure. */
275 struct pvr_fw_object *sysinit_obj;
276
277 /** @osdata_obj: Object representing FW OSDATA structure. */
278 struct pvr_fw_object *osdata_obj;
279
280 /** @hwrinfobuf_obj: Object representing FW hwrinfobuf structure. */
281 struct pvr_fw_object *hwrinfobuf_obj;
282
283 /** @sysdata_obj: Object representing FW SYSDATA structure. */
284 struct pvr_fw_object *sysdata_obj;
285
286 /** @power_sync_obj: Object representing power sync state. */
287 struct pvr_fw_object *power_sync_obj;
288
289 /** @fault_page_obj: Object representing FW fault page. */
290 struct pvr_fw_object *fault_page_obj;
291
292 /** @gpu_util_fwcb_obj: Object representing FW GPU utilisation control structure. */
293 struct pvr_fw_object *gpu_util_fwcb_obj;
294
295 /** @runtime_cfg_obj: Object representing FW runtime config structure. */
296 struct pvr_fw_object *runtime_cfg_obj;
297
298 /** @mmucache_sync_obj: Object used as the sync parameter in an MMU cache operation. */
299 struct pvr_fw_object *mmucache_sync_obj;
300};
301
302struct pvr_fw_device {
303 /** @firmware: Handle to the firmware loaded into the device. */
304 const struct firmware *firmware;
305
306 /** @header: Pointer to firmware header. */
307 const struct pvr_fw_info_header *header;
308
309 /** @layout_entries: Pointer to firmware layout. */
310 const struct pvr_fw_layout_entry *layout_entries;
311
312 /** @mem: Structure containing objects representing firmware memory allocations. */
313 struct pvr_fw_mem mem;
314
315 /** @booted: %true if the firmware has been booted, %false otherwise. */
316 bool booted;
317
318 /**
319 * @processor_type: FW processor type for this device. Must be one of
320 * %PVR_FW_PROCESSOR_TYPE_*.
321 */
322 u16 processor_type;
323
324 /** @funcs: Function table for the FW processor used by this device. */
325 const struct pvr_fw_defs *defs;
326
327 /** @processor_data: Pointer to data specific to FW processor. */
328 union {
329 /** @mips_data: Pointer to MIPS-specific data. */
330 struct pvr_fw_mips_data *mips_data;
331 } processor_data;
332
333 /** @fw_heap_info: Firmware heap information. */
334 struct {
335 /** @gpu_addr: Base address of firmware heap in GPU address space. */
336 u64 gpu_addr;
337
338 /** @size: Size of main area of heap. */
339 u32 size;
340
341 /** @offset_mask: Mask for offsets within FW heap. */
342 u32 offset_mask;
343
344 /** @raw_size: Raw size of heap, including reserved areas. */
345 u32 raw_size;
346
347 /** @log2_size: Log2 of raw size of heap. */
348 u32 log2_size;
349
350 /** @config_offset: Offset of config area within heap. */
351 u32 config_offset;
352
353 /** @reserved_size: Size of reserved area in heap. */
354 u32 reserved_size;
355 } fw_heap_info;
356
357 /** @fw_mm: Firmware address space allocator. */
358 struct drm_mm fw_mm;
359
360 /** @fw_mm_lock: Lock protecting access to &fw_mm. */
361 spinlock_t fw_mm_lock;
362
363 /** @fw_mm_base: Base address of address space managed by @fw_mm. */
364 u64 fw_mm_base;
365
366 /**
367 * @fwif_connection_ctl: Pointer to CPU mapping of FWIF connection
368 * control structure.
369 */
370 struct rogue_fwif_connection_ctl *fwif_connection_ctl;
371
372 /** @fwif_sysinit: Pointer to CPU mapping of FW SYSINIT structure. */
373 struct rogue_fwif_sysinit *fwif_sysinit;
374
375 /** @fwif_sysdata: Pointer to CPU mapping of FW SYSDATA structure. */
376 struct rogue_fwif_sysdata *fwif_sysdata;
377
378 /** @fwif_osinit: Pointer to CPU mapping of FW OSINIT structure. */
379 struct rogue_fwif_osinit *fwif_osinit;
380
381 /** @fwif_osdata: Pointer to CPU mapping of FW OSDATA structure. */
382 struct rogue_fwif_osdata *fwif_osdata;
383
384 /** @power_sync: Pointer to CPU mapping of power sync state. */
385 u32 *power_sync;
386
387 /** @hwrinfobuf: Pointer to CPU mapping of FW HWR info buffer. */
388 struct rogue_fwif_hwrinfobuf *hwrinfobuf;
389
390 /** @fw_trace: Device firmware trace buffer state. */
391 struct pvr_fw_trace fw_trace;
392
393 /** @fw_objs: Structure tracking FW objects. */
394 struct {
395 /** @list: Head of FW object list. */
396 struct list_head list;
397
398 /** @lock: Lock protecting access to FW object list. */
399 struct mutex lock;
400 } fw_objs;
401};
402
403#define pvr_fw_irq_read_reg(pvr_dev, name) \
404 pvr_cr_read32((pvr_dev), (pvr_dev)->fw_dev.defs->irq.name ## _reg)
405
406#define pvr_fw_irq_write_reg(pvr_dev, name, value) \
407 pvr_cr_write32((pvr_dev), (pvr_dev)->fw_dev.defs->irq.name ## _reg, value)
408
409#define pvr_fw_irq_pending(pvr_dev) \
410 (pvr_fw_irq_read_reg(pvr_dev, status) & (pvr_dev)->fw_dev.defs->irq.event_mask)
411
412#define pvr_fw_irq_clear(pvr_dev) \
413 pvr_fw_irq_write_reg(pvr_dev, clear, (pvr_dev)->fw_dev.defs->irq.clear_mask)
414
415#define pvr_fw_irq_enable(pvr_dev) \
416 pvr_fw_irq_write_reg(pvr_dev, enable, (pvr_dev)->fw_dev.defs->irq.event_mask)
417
418#define pvr_fw_irq_disable(pvr_dev) \
419 pvr_fw_irq_write_reg(pvr_dev, enable, 0)
420
421extern const struct pvr_fw_defs pvr_fw_defs_meta;
422extern const struct pvr_fw_defs pvr_fw_defs_mips;
423
424int pvr_fw_validate_init_device_info(struct pvr_device *pvr_dev);
425int pvr_fw_init(struct pvr_device *pvr_dev);
426void pvr_fw_fini(struct pvr_device *pvr_dev);
427
428int pvr_wait_for_fw_boot(struct pvr_device *pvr_dev);
429
430int
431pvr_fw_hard_reset(struct pvr_device *pvr_dev);
432
433void pvr_fw_mts_schedule(struct pvr_device *pvr_dev, u32 val);
434
435void
436pvr_fw_heap_info_init(struct pvr_device *pvr_dev, u32 log2_size, u32 reserved_size);
437
438const struct pvr_fw_layout_entry *
439pvr_fw_find_layout_entry(struct pvr_device *pvr_dev, enum pvr_fw_section_id id);
440int
441pvr_fw_find_mmu_segment(struct pvr_device *pvr_dev, u32 addr, u32 size, void *fw_code_ptr,
442 void *fw_data_ptr, void *fw_core_code_ptr, void *fw_core_data_ptr,
443 void **host_addr_out);
444
445int
446pvr_fw_structure_cleanup(struct pvr_device *pvr_dev, u32 type, struct pvr_fw_object *fw_obj,
447 u32 offset);
448
449int pvr_fw_object_create(struct pvr_device *pvr_dev, size_t size, u64 flags,
450 void (*init)(void *cpu_ptr, void *priv), void *init_priv,
451 struct pvr_fw_object **pvr_obj_out);
452
453void *pvr_fw_object_create_and_map(struct pvr_device *pvr_dev, size_t size, u64 flags,
454 void (*init)(void *cpu_ptr, void *priv),
455 void *init_priv, struct pvr_fw_object **pvr_obj_out);
456
457void *
458pvr_fw_object_create_and_map_offset(struct pvr_device *pvr_dev, u32 dev_offset, size_t size,
459 u64 flags, void (*init)(void *cpu_ptr, void *priv),
460 void *init_priv, struct pvr_fw_object **pvr_obj_out);
461
462static __always_inline void *
463pvr_fw_object_vmap(struct pvr_fw_object *fw_obj)
464{
465 return pvr_gem_object_vmap(fw_obj->gem);
466}
467
468static __always_inline void
469pvr_fw_object_vunmap(struct pvr_fw_object *fw_obj)
470{
471 pvr_gem_object_vunmap(fw_obj->gem);
472}
473
474void pvr_fw_object_destroy(struct pvr_fw_object *fw_obj);
475
476static __always_inline void
477pvr_fw_object_unmap_and_destroy(struct pvr_fw_object *fw_obj)
478{
479 pvr_fw_object_vunmap(fw_obj);
480 pvr_fw_object_destroy(fw_obj);
481}
482
483/**
484 * pvr_fw_get_dma_addr() - Get DMA address for given offset in firmware object
485 * @fw_obj: Pointer to object to lookup address in.
486 * @offset: Offset within object to lookup address at.
487 * @dma_addr_out: Pointer to location to store DMA address.
488 *
489 * Returns:
490 * * 0 on success, or
491 * * -%EINVAL if object is not currently backed, or if @offset is out of valid
492 * range for this object.
493 */
494static __always_inline int
495pvr_fw_object_get_dma_addr(struct pvr_fw_object *fw_obj, u32 offset, dma_addr_t *dma_addr_out)
496{
497 return pvr_gem_get_dma_addr(fw_obj->gem, offset, dma_addr_out);
498}
499
500void pvr_fw_object_get_fw_addr_offset(struct pvr_fw_object *fw_obj, u32 offset, u32 *fw_addr_out);
501
502static __always_inline void
503pvr_fw_object_get_fw_addr(struct pvr_fw_object *fw_obj, u32 *fw_addr_out)
504{
505 pvr_fw_object_get_fw_addr_offset(fw_obj, 0, fw_addr_out);
506}
507
508#endif /* PVR_FW_H */
drivers/gpu/drm/imagination/pvr_fw_info.h+135−0
@@ -0,0 +1,135 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_FW_INFO_H
5#define PVR_FW_INFO_H
6
7#include <linux/bits.h>
8#include <linux/sizes.h>
9#include <linux/types.h>
10
11/*
12 * Firmware binary block unit in bytes.
13 * Raw data stored in FW binary will be aligned to this size.
14 */
15#define FW_BLOCK_SIZE SZ_4K
16
17/* Maximum number of entries in firmware layout table. */
18#define PVR_FW_INFO_MAX_NUM_ENTRIES 8
19
20enum pvr_fw_section_id {
21 META_CODE = 0,
22 META_PRIVATE_DATA,
23 META_COREMEM_CODE,
24 META_COREMEM_DATA,
25 MIPS_CODE,
26 MIPS_EXCEPTIONS_CODE,
27 MIPS_BOOT_CODE,
28 MIPS_PRIVATE_DATA,
29 MIPS_BOOT_DATA,
30 MIPS_STACK,
31 RISCV_UNCACHED_CODE,
32 RISCV_CACHED_CODE,
33 RISCV_PRIVATE_DATA,
34 RISCV_COREMEM_CODE,
35 RISCV_COREMEM_DATA,
36};
37
38enum pvr_fw_section_type {
39 NONE = 0,
40 FW_CODE,
41 FW_DATA,
42 FW_COREMEM_CODE,
43 FW_COREMEM_DATA,
44};
45
46/*
47 * FW binary format with FW info attached:
48 *
49 * Contents Offset
50 * +-----------------+
51 * | | 0
52 * | |
53 * | Original binary |
54 * | file |
55 * | (.ldr/.elf) |
56 * | |
57 * | |
58 * +-----------------+
59 * | Device info | FILE_SIZE - 4K - device_info_size
60 * +-----------------+
61 * | FW info header | FILE_SIZE - 4K
62 * +-----------------+
63 * | |
64 * | FW layout table |
65 * | |
66 * +-----------------+
67 * FILE_SIZE
68 */
69
70#define PVR_FW_INFO_VERSION 3
71
72#define PVR_FW_FLAGS_OPEN_SOURCE BIT(0)
73
74/** struct pvr_fw_info_header - Firmware header */
75struct pvr_fw_info_header {
76 /** @info_version: FW info header version. */
77 u32 info_version;
78 /** @header_len: Header length. */
79 u32 header_len;
80 /** @layout_entry_num: Number of entries in the layout table. */
81 u32 layout_entry_num;
82 /** @layout_entry_size: Size of an entry in the layout table. */
83 u32 layout_entry_size;
84 /** @bvnc: GPU ID supported by firmware. */
85 aligned_u64 bvnc;
86 /** @fw_page_size: Page size of processor on which firmware executes. */
87 u32 fw_page_size;
88 /** @flags: Compatibility flags. */
89 u32 flags;
90 /** @fw_version_major: Firmware major version number. */
91 u16 fw_version_major;
92 /** @fw_version_minor: Firmware minor version number. */
93 u16 fw_version_minor;
94 /** @fw_version_build: Firmware build number. */
95 u32 fw_version_build;
96 /** @device_info_size: Size of device info structure. */
97 u32 device_info_size;
98 /** @padding: Padding. */
99 u32 padding;
100};
101
102/**
103 * struct pvr_fw_layout_entry - Entry in firmware layout table, describing a
104 * section of the firmware image
105 */
106struct pvr_fw_layout_entry {
107 /** @id: Section ID. */
108 enum pvr_fw_section_id id;
109 /** @type: Section type. */
110 enum pvr_fw_section_type type;
111 /** @base_addr: Base address of section in FW address space. */
112 u32 base_addr;
113 /** @max_size: Maximum size of section, in bytes. */
114 u32 max_size;
115 /** @alloc_size: Allocation size of section, in bytes. */
116 u32 alloc_size;
117 /** @alloc_offset: Allocation offset of section. */
118 u32 alloc_offset;
119};
120
121/**
122 * struct pvr_fw_device_info_header - Device information header.
123 */
124struct pvr_fw_device_info_header {
125 /* BRN Mask size (in u64s). */
126 u64 brn_mask_size;
127 /* ERN Mask size (in u64s). */
128 u64 ern_mask_size;
129 /* Feature Mask size (in u64s). */
130 u64 feature_mask_size;
131 /* Feature Parameter size (in u64s). */
132 u64 feature_param_size;
133};
134
135#endif /* PVR_FW_INFO_H */
drivers/gpu/drm/imagination/pvr_fw_meta.c+554−0
@@ -0,0 +1,554 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_device.h"
5#include "pvr_fw.h"
6#include "pvr_fw_info.h"
7#include "pvr_gem.h"
8#include "pvr_rogue_cr_defs.h"
9#include "pvr_rogue_meta.h"
10#include "pvr_vm.h"
11
12#include <linux/compiler.h>
13#include <linux/delay.h>
14#include <linux/firmware.h>
15#include <linux/ktime.h>
16#include <linux/types.h>
17
18#define ROGUE_FW_HEAP_META_SHIFT 25 /* 32 MB */
19
20#define POLL_TIMEOUT_USEC 1000000
21
22/**
23 * pvr_meta_cr_read32() - Read a META register via the Slave Port
24 * @pvr_dev: Device pointer.
25 * @reg_addr: Address of register to read.
26 * @reg_value_out: Pointer to location to store register value.
27 *
28 * Returns:
29 * * 0 on success, or
30 * * Any error returned by pvr_cr_poll_reg32().
31 */
32int
33pvr_meta_cr_read32(struct pvr_device *pvr_dev, u32 reg_addr, u32 *reg_value_out)
34{
35 int err;
36
37 /* Wait for Slave Port to be Ready. */
38 err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_META_SP_MSLVCTRL1,
39 ROGUE_CR_META_SP_MSLVCTRL1_READY_EN |
40 ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_EN,
41 ROGUE_CR_META_SP_MSLVCTRL1_READY_EN |
42 ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_EN,
43 POLL_TIMEOUT_USEC);
44 if (err)
45 return err;
46
47 /* Issue a Read. */
48 pvr_cr_write32(pvr_dev, ROGUE_CR_META_SP_MSLVCTRL0,
49 reg_addr | ROGUE_CR_META_SP_MSLVCTRL0_RD_EN);
50 (void)pvr_cr_read32(pvr_dev, ROGUE_CR_META_SP_MSLVCTRL0); /* Fence write. */
51
52 /* Wait for Slave Port to be Ready. */
53 err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_META_SP_MSLVCTRL1,
54 ROGUE_CR_META_SP_MSLVCTRL1_READY_EN |
55 ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_EN,
56 ROGUE_CR_META_SP_MSLVCTRL1_READY_EN |
57 ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_EN,
58 POLL_TIMEOUT_USEC);
59 if (err)
60 return err;
61
62 *reg_value_out = pvr_cr_read32(pvr_dev, ROGUE_CR_META_SP_MSLVDATAX);
63
64 return 0;
65}
66
67static int
68pvr_meta_wrapper_init(struct pvr_device *pvr_dev)
69{
70 u64 garten_config;
71
72 /* Configure META to Master boot. */
73 pvr_cr_write64(pvr_dev, ROGUE_CR_META_BOOT, ROGUE_CR_META_BOOT_MODE_EN);
74
75 /* Set Garten IDLE to META idle and Set the Garten Wrapper BIF Fence address. */
76
77 /* Garten IDLE bit controlled by META. */
78 garten_config = ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_IDLE_CTRL_META;
79
80 /* The fence addr is set during the fw init sequence. */
81
82 /* Set PC = 0 for fences. */
83 garten_config &=
84 ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_PC_BASE_CLRMSK;
85 garten_config |=
86 (u64)MMU_CONTEXT_MAPPING_FWPRIV
87 << ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_PC_BASE_SHIFT;
88
89 /* Set SLC DM=META. */
90 garten_config |= ((u64)ROGUE_FW_SEGMMU_META_BIFDM_ID)
91 << ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_DM_SHIFT;
92
93 pvr_cr_write64(pvr_dev, ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG, garten_config);
94
95 return 0;
96}
97
98static __always_inline void
99add_boot_arg(u32 **boot_conf, u32 param, u32 data)
100{
101 *(*boot_conf)++ = param;
102 *(*boot_conf)++ = data;
103}
104
105static int
106meta_ldr_cmd_loadmem(struct drm_device *drm_dev, const u8 *fw,
107 struct rogue_meta_ldr_l1_data_blk *l1_data, u32 coremem_size, u8 *fw_code_ptr,
108 u8 *fw_data_ptr, u8 *fw_core_code_ptr, u8 *fw_core_data_ptr, const u32 fw_size)
109{
110 struct rogue_meta_ldr_l2_data_blk *l2_block =
111 (struct rogue_meta_ldr_l2_data_blk *)(fw +
112 l1_data->cmd_data[1]);
113 struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
114 u32 offset = l1_data->cmd_data[0];
115 u32 data_size;
116 void *write_addr;
117 int err;
118
119 /* Verify header is within bounds. */
120 if (((u8 *)l2_block - fw) >= fw_size || ((u8 *)(l2_block + 1) - fw) >= fw_size)
121 return -EINVAL;
122
123 data_size = l2_block->length - 6 /* L2 Tag length and checksum */;
124
125 /* Verify data is within bounds. */
126 if (((u8 *)l2_block->block_data - fw) >= fw_size ||
127 ((((u8 *)l2_block->block_data) + data_size) - fw) >= fw_size)
128 return -EINVAL;
129
130 if (!ROGUE_META_IS_COREMEM_CODE(offset, coremem_size) &&
131 !ROGUE_META_IS_COREMEM_DATA(offset, coremem_size)) {
132 /* Global range is aliased to local range */
133 offset &= ~META_MEM_GLOBAL_RANGE_BIT;
134 }
135
136 err = pvr_fw_find_mmu_segment(pvr_dev, offset, data_size, fw_code_ptr, fw_data_ptr,
137 fw_core_code_ptr, fw_core_data_ptr, &write_addr);
138 if (err) {
139 drm_err(drm_dev,
140 "Addr 0x%x (size: %d) not found in any firmware segment",
141 offset, data_size);
142 return err;
143 }
144
145 memcpy(write_addr, l2_block->block_data, data_size);
146
147 return 0;
148}
149
150static int
151meta_ldr_cmd_zeromem(struct drm_device *drm_dev,
152 struct rogue_meta_ldr_l1_data_blk *l1_data, u32 coremem_size,
153 u8 *fw_code_ptr, u8 *fw_data_ptr, u8 *fw_core_code_ptr, u8 *fw_core_data_ptr)
154{
155 struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
156 u32 offset = l1_data->cmd_data[0];
157 u32 byte_count = l1_data->cmd_data[1];
158 void *write_addr;
159 int err;
160
161 if (ROGUE_META_IS_COREMEM_DATA(offset, coremem_size)) {
162 /* cannot zero coremem directly */
163 return 0;
164 }
165
166 /* Global range is aliased to local range */
167 offset &= ~META_MEM_GLOBAL_RANGE_BIT;
168
169 err = pvr_fw_find_mmu_segment(pvr_dev, offset, byte_count, fw_code_ptr, fw_data_ptr,
170 fw_core_code_ptr, fw_core_data_ptr, &write_addr);
171 if (err) {
172 drm_err(drm_dev,
173 "Addr 0x%x (size: %d) not found in any firmware segment",
174 offset, byte_count);
175 return err;
176 }
177
178 memset(write_addr, 0, byte_count);
179
180 return 0;
181}
182
183static int
184meta_ldr_cmd_config(struct drm_device *drm_dev, const u8 *fw,
185 struct rogue_meta_ldr_l1_data_blk *l1_data,
186 const u32 fw_size, u32 **boot_conf_ptr)
187{
188 struct rogue_meta_ldr_l2_data_blk *l2_block =
189 (struct rogue_meta_ldr_l2_data_blk *)(fw +
190 l1_data->cmd_data[0]);
191 struct rogue_meta_ldr_cfg_blk *config_command;
192 u32 l2_block_size;
193 u32 curr_block_size = 0;
194 u32 *boot_conf = boot_conf_ptr ? *boot_conf_ptr : NULL;
195
196 /* Verify block header is within bounds. */
197 if (((u8 *)l2_block - fw) >= fw_size || ((u8 *)(l2_block + 1) - fw) >= fw_size)
198 return -EINVAL;
199
200 l2_block_size = l2_block->length - 6 /* L2 Tag length and checksum */;
201 config_command = (struct rogue_meta_ldr_cfg_blk *)l2_block->block_data;
202
203 if (((u8 *)config_command - fw) >= fw_size ||
204 ((((u8 *)config_command) + l2_block_size) - fw) >= fw_size)
205 return -EINVAL;
206
207 while (l2_block_size >= 12) {
208 if (config_command->type != ROGUE_META_LDR_CFG_WRITE)
209 return -EINVAL;
210
211 /*
212 * Only write to bootloader if we got a valid pointer to the FW
213 * code allocation.
214 */
215 if (boot_conf) {
216 u32 register_offset = config_command->block_data[0];
217 u32 register_value = config_command->block_data[1];
218
219 /* Do register write */
220 add_boot_arg(&boot_conf, register_offset,
221 register_value);
222 }
223
224 curr_block_size = 12;
225 l2_block_size -= curr_block_size;
226 config_command = (struct rogue_meta_ldr_cfg_blk
227 *)((uintptr_t)config_command +
228 curr_block_size);
229 }
230
231 if (boot_conf_ptr)
232 *boot_conf_ptr = boot_conf;
233
234 return 0;
235}
236
237/**
238 * process_ldr_command_stream() - Process LDR firmware image and populate
239 * firmware sections
240 * @pvr_dev: Device pointer.
241 * @fw: Pointer to firmware image.
242 * @fw_code_ptr: Pointer to FW code section.
243 * @fw_data_ptr: Pointer to FW data section.
244 * @fw_core_code_ptr: Pointer to FW coremem code section.
245 * @fw_core_data_ptr: Pointer to FW coremem data section.
246 * @boot_conf_ptr: Pointer to boot config argument pointer.
247 *
248 * Returns :
249 * * 0 on success, or
250 * * -EINVAL on any error in LDR command stream.
251 */
252static int
253process_ldr_command_stream(struct pvr_device *pvr_dev, const u8 *fw, u8 *fw_code_ptr,
254 u8 *fw_data_ptr, u8 *fw_core_code_ptr,
255 u8 *fw_core_data_ptr, u32 **boot_conf_ptr)
256{
257 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
258 struct rogue_meta_ldr_block_hdr *ldr_header =
259 (struct rogue_meta_ldr_block_hdr *)fw;
260 struct rogue_meta_ldr_l1_data_blk *l1_data =
261 (struct rogue_meta_ldr_l1_data_blk *)(fw + ldr_header->sl_data);
262 const u32 fw_size = pvr_dev->fw_dev.firmware->size;
263 int err;
264
265 u32 *boot_conf = boot_conf_ptr ? *boot_conf_ptr : NULL;
266 u32 coremem_size;
267
268 err = PVR_FEATURE_VALUE(pvr_dev, meta_coremem_size, &coremem_size);
269 if (err)
270 return err;
271
272 coremem_size *= SZ_1K;
273
274 while (l1_data) {
275 /* Verify block header is within bounds. */
276 if (((u8 *)l1_data - fw) >= fw_size || ((u8 *)(l1_data + 1) - fw) >= fw_size)
277 return -EINVAL;
278
279 if (ROGUE_META_LDR_BLK_IS_COMMENT(l1_data->cmd)) {
280 /* Don't process comment blocks */
281 goto next_block;
282 }
283
284 switch (l1_data->cmd & ROGUE_META_LDR_CMD_MASK)
285 case ROGUE_META_LDR_CMD_LOADMEM: {
286 err = meta_ldr_cmd_loadmem(drm_dev, fw, l1_data,
287 coremem_size,
288 fw_code_ptr, fw_data_ptr,
289 fw_core_code_ptr,
290 fw_core_data_ptr, fw_size);
291 if (err)
292 return err;
293 break;
294
295 case ROGUE_META_LDR_CMD_START_THREADS:
296 /* Don't process this block */
297 break;
298
299 case ROGUE_META_LDR_CMD_ZEROMEM:
300 err = meta_ldr_cmd_zeromem(drm_dev, l1_data,
301 coremem_size,
302 fw_code_ptr, fw_data_ptr,
303 fw_core_code_ptr,
304 fw_core_data_ptr);
305 if (err)
306 return err;
307 break;
308
309 case ROGUE_META_LDR_CMD_CONFIG:
310 err = meta_ldr_cmd_config(drm_dev, fw, l1_data, fw_size,
311 &boot_conf);
312 if (err)
313 return err;
314 break;
315
316 default:
317 return -EINVAL;
318 }
319
320next_block:
321 if (l1_data->next == 0xFFFFFFFF)
322 break;
323
324 l1_data = (struct rogue_meta_ldr_l1_data_blk *)(fw +
325 l1_data->next);
326 }
327
328 if (boot_conf_ptr)
329 *boot_conf_ptr = boot_conf;
330
331 return 0;
332}
333
334static void
335configure_seg_id(u64 seg_out_addr, u32 seg_base, u32 seg_limit, u32 seg_id,
336 u32 **boot_conf_ptr)
337{
338 u32 seg_out_addr0 = seg_out_addr & 0x00000000FFFFFFFFUL;
339 u32 seg_out_addr1 = (seg_out_addr >> 32) & 0x00000000FFFFFFFFUL;
340 u32 *boot_conf = *boot_conf_ptr;
341
342 /* META segments have a minimum size. */
343 u32 limit_off = max(seg_limit, ROGUE_FW_SEGMMU_ALIGN);
344
345 /* The limit is an offset, therefore off = size - 1. */
346 limit_off -= 1;
347
348 seg_base |= ROGUE_FW_SEGMMU_ALLTHRS_WRITEABLE;
349
350 add_boot_arg(&boot_conf, META_CR_MMCU_SEGMENT_N_BASE(seg_id), seg_base);
351 add_boot_arg(&boot_conf, META_CR_MMCU_SEGMENT_N_LIMIT(seg_id), limit_off);
352 add_boot_arg(&boot_conf, META_CR_MMCU_SEGMENT_N_OUTA0(seg_id), seg_out_addr0);
353 add_boot_arg(&boot_conf, META_CR_MMCU_SEGMENT_N_OUTA1(seg_id), seg_out_addr1);
354
355 *boot_conf_ptr = boot_conf;
356}
357
358static u64 get_fw_obj_gpu_addr(struct pvr_fw_object *fw_obj)
359{
360 struct pvr_device *pvr_dev = to_pvr_device(gem_from_pvr_gem(fw_obj->gem)->dev);
361 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
362
363 return fw_obj->fw_addr_offset + fw_dev->fw_heap_info.gpu_addr;
364}
365
366static void
367configure_seg_mmu(struct pvr_device *pvr_dev, u32 **boot_conf_ptr)
368{
369 const struct pvr_fw_layout_entry *layout_entries = pvr_dev->fw_dev.layout_entries;
370 u32 num_layout_entries = pvr_dev->fw_dev.header->layout_entry_num;
371 u64 seg_out_addr_top;
372 u32 i;
373
374 seg_out_addr_top =
375 ROGUE_FW_SEGMMU_OUTADDR_TOP_SLC(MMU_CONTEXT_MAPPING_FWPRIV,
376 ROGUE_FW_SEGMMU_META_BIFDM_ID);
377
378 for (i = 0; i < num_layout_entries; i++) {
379 /*
380 * FW code is using the bootloader segment which is already
381 * configured on boot. FW coremem code and data don't use the
382 * segment MMU. Only the FW data segment needs to be configured.
383 */
384 if (layout_entries[i].type == FW_DATA) {
385 u32 seg_id = ROGUE_FW_SEGMMU_DATA_ID;
386 u64 seg_out_addr = get_fw_obj_gpu_addr(pvr_dev->fw_dev.mem.data_obj);
387
388 seg_out_addr += layout_entries[i].alloc_offset;
389 seg_out_addr |= seg_out_addr_top;
390
391 /* Write the sequence to the bootldr. */
392 configure_seg_id(seg_out_addr,
393 layout_entries[i].base_addr,
394 layout_entries[i].alloc_size, seg_id,
395 boot_conf_ptr);
396
397 break;
398 }
399 }
400}
401
402static void
403configure_meta_caches(u32 **boot_conf_ptr)
404{
405 u32 *boot_conf = *boot_conf_ptr;
406 u32 d_cache_t0, i_cache_t0;
407 u32 d_cache_t1, i_cache_t1;
408 u32 d_cache_t2, i_cache_t2;
409 u32 d_cache_t3, i_cache_t3;
410
411 /* Initialise I/Dcache settings */
412 d_cache_t0 = META_CR_SYSC_DCPARTX_CACHED_WRITE_ENABLE;
413 d_cache_t1 = META_CR_SYSC_DCPARTX_CACHED_WRITE_ENABLE;
414 d_cache_t2 = META_CR_SYSC_DCPARTX_CACHED_WRITE_ENABLE;
415 d_cache_t3 = META_CR_SYSC_DCPARTX_CACHED_WRITE_ENABLE;
416 i_cache_t0 = 0;
417 i_cache_t1 = 0;
418 i_cache_t2 = 0;
419 i_cache_t3 = 0;
420
421 d_cache_t0 |= META_CR_SYSC_XCPARTX_LOCAL_ADDR_FULL_CACHE;
422 i_cache_t0 |= META_CR_SYSC_XCPARTX_LOCAL_ADDR_FULL_CACHE;
423
424 /* Local region MMU enhanced bypass: WIN-3 mode for code and data caches */
425 add_boot_arg(&boot_conf, META_CR_MMCU_LOCAL_EBCTRL,
426 META_CR_MMCU_LOCAL_EBCTRL_ICWIN |
427 META_CR_MMCU_LOCAL_EBCTRL_DCWIN);
428
429 /* Data cache partitioning thread 0 to 3 */
430 add_boot_arg(&boot_conf, META_CR_SYSC_DCPART(0), d_cache_t0);
431 add_boot_arg(&boot_conf, META_CR_SYSC_DCPART(1), d_cache_t1);
432 add_boot_arg(&boot_conf, META_CR_SYSC_DCPART(2), d_cache_t2);
433 add_boot_arg(&boot_conf, META_CR_SYSC_DCPART(3), d_cache_t3);
434
435 /* Enable data cache hits */
436 add_boot_arg(&boot_conf, META_CR_MMCU_DCACHE_CTRL,
437 META_CR_MMCU_XCACHE_CTRL_CACHE_HITS_EN);
438
439 /* Instruction cache partitioning thread 0 to 3 */
440 add_boot_arg(&boot_conf, META_CR_SYSC_ICPART(0), i_cache_t0);
441 add_boot_arg(&boot_conf, META_CR_SYSC_ICPART(1), i_cache_t1);
442 add_boot_arg(&boot_conf, META_CR_SYSC_ICPART(2), i_cache_t2);
443 add_boot_arg(&boot_conf, META_CR_SYSC_ICPART(3), i_cache_t3);
444
445 /* Enable instruction cache hits */
446 add_boot_arg(&boot_conf, META_CR_MMCU_ICACHE_CTRL,
447 META_CR_MMCU_XCACHE_CTRL_CACHE_HITS_EN);
448
449 add_boot_arg(&boot_conf, 0x040000C0, 0);
450
451 *boot_conf_ptr = boot_conf;
452}
453
454static int
455pvr_meta_fw_process(struct pvr_device *pvr_dev, const u8 *fw,
456 u8 *fw_code_ptr, u8 *fw_data_ptr, u8 *fw_core_code_ptr, u8 *fw_core_data_ptr,
457 u32 core_code_alloc_size)
458{
459 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
460 u32 *boot_conf;
461 int err;
462
463 boot_conf = ((u32 *)fw_code_ptr) + ROGUE_FW_BOOTLDR_CONF_OFFSET;
464
465 /* Slave port and JTAG accesses are privileged. */
466 add_boot_arg(&boot_conf, META_CR_SYSC_JTAG_THREAD,
467 META_CR_SYSC_JTAG_THREAD_PRIV_EN);
468
469 configure_seg_mmu(pvr_dev, &boot_conf);
470
471 /* Populate FW sections from LDR image. */
472 err = process_ldr_command_stream(pvr_dev, fw, fw_code_ptr, fw_data_ptr, fw_core_code_ptr,
473 fw_core_data_ptr, &boot_conf);
474 if (err)
475 return err;
476
477 configure_meta_caches(&boot_conf);
478
479 /* End argument list. */
480 add_boot_arg(&boot_conf, 0, 0);
481
482 if (fw_dev->mem.core_code_obj) {
483 u32 core_code_fw_addr;
484
485 pvr_fw_object_get_fw_addr(fw_dev->mem.core_code_obj, &core_code_fw_addr);
486 add_boot_arg(&boot_conf, core_code_fw_addr, core_code_alloc_size);
487 } else {
488 add_boot_arg(&boot_conf, 0, 0);
489 }
490 /* None of the cores supported by this driver have META DMA. */
491 add_boot_arg(&boot_conf, 0, 0);
492
493 return 0;
494}
495
496static int
497pvr_meta_init(struct pvr_device *pvr_dev)
498{
499 pvr_fw_heap_info_init(pvr_dev, ROGUE_FW_HEAP_META_SHIFT, 0);
500
501 return 0;
502}
503
504static u32
505pvr_meta_get_fw_addr_with_offset(struct pvr_fw_object *fw_obj, u32 offset)
506{
507 u32 fw_addr = fw_obj->fw_addr_offset + offset + ROGUE_FW_SEGMMU_DATA_BASE_ADDRESS;
508
509 /* META cacheability is determined by address. */
510 if (fw_obj->gem->flags & PVR_BO_FW_FLAGS_DEVICE_UNCACHED)
511 fw_addr |= ROGUE_FW_SEGMMU_DATA_META_UNCACHED |
512 ROGUE_FW_SEGMMU_DATA_VIVT_SLC_UNCACHED;
513
514 return fw_addr;
515}
516
517static int
518pvr_meta_vm_map(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj)
519{
520 struct pvr_gem_object *pvr_obj = fw_obj->gem;
521
522 return pvr_vm_map(pvr_dev->kernel_vm_ctx, pvr_obj, 0, fw_obj->fw_mm_node.start,
523 pvr_gem_object_size(pvr_obj));
524}
525
526static void
527pvr_meta_vm_unmap(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj)
528{
529 pvr_vm_unmap(pvr_dev->kernel_vm_ctx, fw_obj->fw_mm_node.start,
530 fw_obj->fw_mm_node.size);
531}
532
533static bool
534pvr_meta_has_fixed_data_addr(void)
535{
536 return false;
537}
538
539const struct pvr_fw_defs pvr_fw_defs_meta = {
540 .init = pvr_meta_init,
541 .fw_process = pvr_meta_fw_process,
542 .vm_map = pvr_meta_vm_map,
543 .vm_unmap = pvr_meta_vm_unmap,
544 .get_fw_addr_with_offset = pvr_meta_get_fw_addr_with_offset,
545 .wrapper_init = pvr_meta_wrapper_init,
546 .has_fixed_data_addr = pvr_meta_has_fixed_data_addr,
547 .irq = {
548 .enable_reg = ROGUE_CR_META_SP_MSLVIRQENABLE,
549 .status_reg = ROGUE_CR_META_SP_MSLVIRQSTATUS,
550 .clear_reg = ROGUE_CR_META_SP_MSLVIRQSTATUS,
551 .event_mask = ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT2_EN,
552 .clear_mask = ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT2_CLRMSK,
553 },
554};
drivers/gpu/drm/imagination/pvr_fw_meta.h+14−0
@@ -0,0 +1,14 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_FW_META_H
5#define PVR_FW_META_H
6
7#include <linux/types.h>
8
9/* Forward declaration from pvr_device.h */
10struct pvr_device;
11
12int pvr_meta_cr_read32(struct pvr_device *pvr_dev, u32 reg_addr, u32 *reg_value_out);
13
14#endif /* PVR_FW_META_H */
drivers/gpu/drm/imagination/pvr_fw_mips.c+252−0
@@ -0,0 +1,252 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_device.h"
5#include "pvr_fw.h"
6#include "pvr_fw_mips.h"
7#include "pvr_gem.h"
8#include "pvr_rogue_mips.h"
9#include "pvr_vm_mips.h"
10
11#include <linux/elf.h>
12#include <linux/err.h>
13#include <linux/types.h>
14
15#define ROGUE_FW_HEAP_MIPS_BASE 0xC0000000
16#define ROGUE_FW_HEAP_MIPS_SHIFT 24 /* 16 MB */
17#define ROGUE_FW_HEAP_MIPS_RESERVED_SIZE SZ_1M
18
19/**
20 * process_elf_command_stream() - Process ELF firmware image and populate
21 * firmware sections
22 * @pvr_dev: Device pointer.
23 * @fw: Pointer to firmware image.
24 * @fw_code_ptr: Pointer to FW code section.
25 * @fw_data_ptr: Pointer to FW data section.
26 * @fw_core_code_ptr: Pointer to FW coremem code section.
27 * @fw_core_data_ptr: Pointer to FW coremem data section.
28 *
29 * Returns :
30 * * 0 on success, or
31 * * -EINVAL on any error in ELF command stream.
32 */
33static int
34process_elf_command_stream(struct pvr_device *pvr_dev, const u8 *fw, u8 *fw_code_ptr,
35 u8 *fw_data_ptr, u8 *fw_core_code_ptr, u8 *fw_core_data_ptr)
36{
37 struct elf32_hdr *header = (struct elf32_hdr *)fw;
38 struct elf32_phdr *program_header = (struct elf32_phdr *)(fw + header->e_phoff);
39 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
40 u32 entry;
41 int err;
42
43 for (entry = 0; entry < header->e_phnum; entry++, program_header++) {
44 void *write_addr;
45
46 /* Only consider loadable entries in the ELF segment table */
47 if (program_header->p_type != PT_LOAD)
48 continue;
49
50 err = pvr_fw_find_mmu_segment(pvr_dev, program_header->p_vaddr,
51 program_header->p_memsz, fw_code_ptr, fw_data_ptr,
52 fw_core_code_ptr, fw_core_data_ptr, &write_addr);
53 if (err) {
54 drm_err(drm_dev,
55 "Addr 0x%x (size: %d) not found in any firmware segment",
56 program_header->p_vaddr, program_header->p_memsz);
57 return err;
58 }
59
60 /* Write to FW allocation only if available */
61 if (write_addr) {
62 memcpy(write_addr, fw + program_header->p_offset,
63 program_header->p_filesz);
64
65 memset((u8 *)write_addr + program_header->p_filesz, 0,
66 program_header->p_memsz - program_header->p_filesz);
67 }
68 }
69
70 return 0;
71}
72
73static int
74pvr_mips_init(struct pvr_device *pvr_dev)
75{
76 pvr_fw_heap_info_init(pvr_dev, ROGUE_FW_HEAP_MIPS_SHIFT, ROGUE_FW_HEAP_MIPS_RESERVED_SIZE);
77
78 return pvr_vm_mips_init(pvr_dev);
79}
80
81static void
82pvr_mips_fini(struct pvr_device *pvr_dev)
83{
84 pvr_vm_mips_fini(pvr_dev);
85}
86
87static int
88pvr_mips_fw_process(struct pvr_device *pvr_dev, const u8 *fw,
89 u8 *fw_code_ptr, u8 *fw_data_ptr, u8 *fw_core_code_ptr, u8 *fw_core_data_ptr,
90 u32 core_code_alloc_size)
91{
92 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
93 struct pvr_fw_mips_data *mips_data = fw_dev->processor_data.mips_data;
94 const struct pvr_fw_layout_entry *boot_code_entry;
95 const struct pvr_fw_layout_entry *boot_data_entry;
96 const struct pvr_fw_layout_entry *exception_code_entry;
97 const struct pvr_fw_layout_entry *stack_entry;
98 struct rogue_mipsfw_boot_data *boot_data;
99 dma_addr_t dma_addr;
100 u32 page_nr;
101 int err;
102
103 err = process_elf_command_stream(pvr_dev, fw, fw_code_ptr, fw_data_ptr, fw_core_code_ptr,
104 fw_core_data_ptr);
105 if (err)
106 return err;
107
108 boot_code_entry = pvr_fw_find_layout_entry(pvr_dev, MIPS_BOOT_CODE);
109 boot_data_entry = pvr_fw_find_layout_entry(pvr_dev, MIPS_BOOT_DATA);
110 exception_code_entry = pvr_fw_find_layout_entry(pvr_dev, MIPS_EXCEPTIONS_CODE);
111 if (!boot_code_entry || !boot_data_entry || !exception_code_entry)
112 return -EINVAL;
113
114 WARN_ON(pvr_gem_get_dma_addr(fw_dev->mem.code_obj->gem, boot_code_entry->alloc_offset,
115 &mips_data->boot_code_dma_addr));
116 WARN_ON(pvr_gem_get_dma_addr(fw_dev->mem.data_obj->gem, boot_data_entry->alloc_offset,
117 &mips_data->boot_data_dma_addr));
118 WARN_ON(pvr_gem_get_dma_addr(fw_dev->mem.code_obj->gem,
119 exception_code_entry->alloc_offset,
120 &mips_data->exception_code_dma_addr));
121
122 stack_entry = pvr_fw_find_layout_entry(pvr_dev, MIPS_STACK);
123 if (!stack_entry)
124 return -EINVAL;
125
126 boot_data = (struct rogue_mipsfw_boot_data *)(fw_data_ptr + boot_data_entry->alloc_offset +
127 ROGUE_MIPSFW_BOOTLDR_CONF_OFFSET);
128
129 WARN_ON(pvr_fw_object_get_dma_addr(fw_dev->mem.data_obj, stack_entry->alloc_offset,
130 &dma_addr));
131 boot_data->stack_phys_addr = dma_addr;
132
133 boot_data->reg_base = pvr_dev->regs_resource->start;
134
135 for (page_nr = 0; page_nr < ARRAY_SIZE(boot_data->pt_phys_addr); page_nr++) {
136 /* Firmware expects 4k pages, but host page size might be different. */
137 u32 src_page_nr = (page_nr * ROGUE_MIPSFW_PAGE_SIZE_4K) >> PAGE_SHIFT;
138 u32 page_offset = (page_nr * ROGUE_MIPSFW_PAGE_SIZE_4K) & ~PAGE_MASK;
139
140 boot_data->pt_phys_addr[page_nr] = mips_data->pt_dma_addr[src_page_nr] +
141 page_offset;
142 }
143
144 boot_data->pt_log2_page_size = ROGUE_MIPSFW_LOG2_PAGE_SIZE_4K;
145 boot_data->pt_num_pages = ROGUE_MIPSFW_MAX_NUM_PAGETABLE_PAGES;
146 boot_data->reserved1 = 0;
147 boot_data->reserved2 = 0;
148
149 return 0;
150}
151
152static int
153pvr_mips_wrapper_init(struct pvr_device *pvr_dev)
154{
155 struct pvr_fw_mips_data *mips_data = pvr_dev->fw_dev.processor_data.mips_data;
156 const u64 remap_settings = ROGUE_MIPSFW_BOOT_REMAP_LOG2_SEGMENT_SIZE;
157 u32 phys_bus_width;
158
159 int err = PVR_FEATURE_VALUE(pvr_dev, phys_bus_width, &phys_bus_width);
160
161 if (WARN_ON(err))
162 return err;
163
164 /* Currently MIPS FW only supported with physical bus width > 32 bits. */
165 if (WARN_ON(phys_bus_width <= 32))
166 return -EINVAL;
167
168 pvr_cr_write32(pvr_dev, ROGUE_CR_MIPS_WRAPPER_CONFIG,
169 (ROGUE_MIPSFW_REGISTERS_VIRTUAL_BASE >>
170 ROGUE_MIPSFW_WRAPPER_CONFIG_REGBANK_ADDR_ALIGN) |
171 ROGUE_CR_MIPS_WRAPPER_CONFIG_BOOT_ISA_MODE_MICROMIPS);
172
173 /* Configure remap for boot code, boot data and exceptions code areas. */
174 pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1,
175 ROGUE_MIPSFW_BOOT_REMAP_PHYS_ADDR_IN |
176 ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_MODE_ENABLE_EN);
177 pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2,
178 (mips_data->boot_code_dma_addr &
179 ~ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_ADDR_OUT_CLRMSK) | remap_settings);
180
181 if (PVR_HAS_QUIRK(pvr_dev, 63553)) {
182 /*
183 * WA always required on 36 bit cores, to avoid continuous unmapped memory accesses
184 * to address 0x0.
185 */
186 WARN_ON(phys_bus_width != 36);
187
188 pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1,
189 ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_MODE_ENABLE_EN);
190 pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2,
191 (mips_data->boot_code_dma_addr &
192 ~ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_ADDR_OUT_CLRMSK) |
193 remap_settings);
194 }
195
196 pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1,
197 ROGUE_MIPSFW_DATA_REMAP_PHYS_ADDR_IN |
198 ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_MODE_ENABLE_EN);
199 pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2,
200 (mips_data->boot_data_dma_addr &
201 ~ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_ADDR_OUT_CLRMSK) | remap_settings);
202
203 pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1,
204 ROGUE_MIPSFW_CODE_REMAP_PHYS_ADDR_IN |
205 ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_MODE_ENABLE_EN);
206 pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2,
207 (mips_data->exception_code_dma_addr &
208 ~ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_ADDR_OUT_CLRMSK) | remap_settings);
209
210 /* Garten IDLE bit controlled by MIPS. */
211 pvr_cr_write64(pvr_dev, ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG,
212 ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_IDLE_CTRL_META);
213
214 /* Turn on the EJTAG probe. */
215 pvr_cr_write32(pvr_dev, ROGUE_CR_MIPS_DEBUG_CONFIG, 0);
216
217 return 0;
218}
219
220static u32
221pvr_mips_get_fw_addr_with_offset(struct pvr_fw_object *fw_obj, u32 offset)
222{
223 struct pvr_device *pvr_dev = to_pvr_device(gem_from_pvr_gem(fw_obj->gem)->dev);
224
225 /* MIPS cacheability is determined by page table. */
226 return ((fw_obj->fw_addr_offset + offset) & pvr_dev->fw_dev.fw_heap_info.offset_mask) |
227 ROGUE_FW_HEAP_MIPS_BASE;
228}
229
230static bool
231pvr_mips_has_fixed_data_addr(void)
232{
233 return true;
234}
235
236const struct pvr_fw_defs pvr_fw_defs_mips = {
237 .init = pvr_mips_init,
238 .fini = pvr_mips_fini,
239 .fw_process = pvr_mips_fw_process,
240 .vm_map = pvr_vm_mips_map,
241 .vm_unmap = pvr_vm_mips_unmap,
242 .get_fw_addr_with_offset = pvr_mips_get_fw_addr_with_offset,
243 .wrapper_init = pvr_mips_wrapper_init,
244 .has_fixed_data_addr = pvr_mips_has_fixed_data_addr,
245 .irq = {
246 .enable_reg = ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE,
247 .status_reg = ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS,
248 .clear_reg = ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR,
249 .event_mask = ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS_EVENT_EN,
250 .clear_mask = ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR_EVENT_EN,
251 },
252};
drivers/gpu/drm/imagination/pvr_fw_mips.h+48−0
@@ -0,0 +1,48 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_FW_MIPS_H
5#define PVR_FW_MIPS_H
6
7#include "pvr_rogue_mips.h"
8
9#include <asm/page.h>
10#include <linux/types.h>
11
12/* Forward declaration from pvr_gem.h. */
13struct pvr_gem_object;
14
15#define PVR_MIPS_PT_PAGE_COUNT ((ROGUE_MIPSFW_MAX_NUM_PAGETABLE_PAGES * ROGUE_MIPSFW_PAGE_SIZE_4K) \
16 >> PAGE_SHIFT)
17/**
18 * struct pvr_fw_mips_data - MIPS-specific data
19 */
20struct pvr_fw_mips_data {
21 /**
22 * @pt_pages: Pages containing MIPS pagetable.
23 */
24 struct page *pt_pages[PVR_MIPS_PT_PAGE_COUNT];
25
26 /** @pt: Pointer to CPU mapping of MIPS pagetable. */
27 u32 *pt;
28
29 /** @pt_dma_addr: DMA mappings of MIPS pagetable. */
30 dma_addr_t pt_dma_addr[PVR_MIPS_PT_PAGE_COUNT];
31
32 /** @boot_code_dma_addr: DMA address of MIPS boot code. */
33 dma_addr_t boot_code_dma_addr;
34
35 /** @boot_data_dma_addr: DMA address of MIPS boot data. */
36 dma_addr_t boot_data_dma_addr;
37
38 /** @exception_code_dma_addr: DMA address of MIPS exception code. */
39 dma_addr_t exception_code_dma_addr;
40
41 /** @cache_policy: Cache policy for this processor. */
42 u32 cache_policy;
43
44 /** @pfn_mask: PFN mask for MIPS pagetable. */
45 u32 pfn_mask;
46};
47
48#endif /* PVR_FW_MIPS_H */
drivers/gpu/drm/imagination/pvr_fw_startstop.c+306−0
@@ -0,0 +1,306 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_device.h"
5#include "pvr_fw.h"
6#include "pvr_fw_meta.h"
7#include "pvr_fw_startstop.h"
8#include "pvr_rogue_cr_defs.h"
9#include "pvr_rogue_meta.h"
10#include "pvr_vm.h"
11
12#include <linux/compiler.h>
13#include <linux/delay.h>
14#include <linux/ktime.h>
15#include <linux/types.h>
16
17#define POLL_TIMEOUT_USEC 1000000
18
19static void
20rogue_axi_ace_list_init(struct pvr_device *pvr_dev)
21{
22 /* Setup AXI-ACE config. Set everything to outer cache. */
23 u64 reg_val =
24 (3U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_AWDOMAIN_NON_SNOOPING_SHIFT) |
25 (3U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_ARDOMAIN_NON_SNOOPING_SHIFT) |
26 (2U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_ARDOMAIN_CACHE_MAINTENANCE_SHIFT) |
27 (2U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_AWDOMAIN_COHERENT_SHIFT) |
28 (2U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_ARDOMAIN_COHERENT_SHIFT) |
29 (2U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_AWCACHE_COHERENT_SHIFT) |
30 (2U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_ARCACHE_COHERENT_SHIFT) |
31 (2U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_ARCACHE_CACHE_MAINTENANCE_SHIFT);
32
33 pvr_cr_write64(pvr_dev, ROGUE_CR_AXI_ACE_LITE_CONFIGURATION, reg_val);
34}
35
36static void
37rogue_bif_init(struct pvr_device *pvr_dev)
38{
39 dma_addr_t pc_dma_addr;
40 u64 pc_addr;
41
42 /* Acquire the address of the Kernel Page Catalogue. */
43 pc_dma_addr = pvr_vm_get_page_table_root_addr(pvr_dev->kernel_vm_ctx);
44
45 /* Write the kernel catalogue base. */
46 pc_addr = ((((u64)pc_dma_addr >> ROGUE_CR_BIF_CAT_BASE0_ADDR_ALIGNSHIFT)
47 << ROGUE_CR_BIF_CAT_BASE0_ADDR_SHIFT) &
48 ~ROGUE_CR_BIF_CAT_BASE0_ADDR_CLRMSK);
49
50 pvr_cr_write64(pvr_dev, BIF_CAT_BASEX(MMU_CONTEXT_MAPPING_FWPRIV),
51 pc_addr);
52}
53
54static int
55rogue_slc_init(struct pvr_device *pvr_dev)
56{
57 u16 slc_cache_line_size_bits;
58 u32 reg_val;
59 int err;
60
61 /*
62 * SLC Misc control.
63 *
64 * Note: This is a 64bit register and we set only the lower 32bits
65 * leaving the top 32bits (ROGUE_CR_SLC_CTRL_MISC_SCRAMBLE_BITS)
66 * unchanged from the HW default.
67 */
68 reg_val = (pvr_cr_read32(pvr_dev, ROGUE_CR_SLC_CTRL_MISC) &
69 ROGUE_CR_SLC_CTRL_MISC_ENABLE_PSG_HAZARD_CHECK_EN) |
70 ROGUE_CR_SLC_CTRL_MISC_ADDR_DECODE_MODE_PVR_HASH1;
71
72 err = PVR_FEATURE_VALUE(pvr_dev, slc_cache_line_size_bits, &slc_cache_line_size_bits);
73 if (err)
74 return err;
75
76 /* Bypass burst combiner if SLC line size is smaller than 1024 bits. */
77 if (slc_cache_line_size_bits < 1024)
78 reg_val |= ROGUE_CR_SLC_CTRL_MISC_BYPASS_BURST_COMBINER_EN;
79
80 if (PVR_HAS_QUIRK(pvr_dev, 71242) && !PVR_HAS_FEATURE(pvr_dev, gpu_multicore_support))
81 reg_val |= ROGUE_CR_SLC_CTRL_MISC_LAZYWB_OVERRIDE_EN;
82
83 pvr_cr_write32(pvr_dev, ROGUE_CR_SLC_CTRL_MISC, reg_val);
84
85 return 0;
86}
87
88/**
89 * pvr_fw_start() - Start FW processor and boot firmware
90 * @pvr_dev: Target PowerVR device.
91 *
92 * Returns:
93 * * 0 on success, or
94 * * Any error returned by rogue_slc_init().
95 */
96int
97pvr_fw_start(struct pvr_device *pvr_dev)
98{
99 bool has_reset2 = PVR_HAS_FEATURE(pvr_dev, xe_tpu2);
100 u64 soft_reset_mask;
101 int err;
102
103 if (PVR_HAS_FEATURE(pvr_dev, pbe2_in_xe))
104 soft_reset_mask = ROGUE_CR_SOFT_RESET__PBE2_XE__MASKFULL;
105 else
106 soft_reset_mask = ROGUE_CR_SOFT_RESET_MASKFULL;
107
108 if (PVR_HAS_FEATURE(pvr_dev, sys_bus_secure_reset)) {
109 /*
110 * Disable the default sys_bus_secure protection to perform
111 * minimal setup.
112 */
113 pvr_cr_write32(pvr_dev, ROGUE_CR_SYS_BUS_SECURE, 0);
114 (void)pvr_cr_read32(pvr_dev, ROGUE_CR_SYS_BUS_SECURE); /* Fence write */
115 }
116
117 /* Set Rogue in soft-reset. */
118 pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET, soft_reset_mask);
119 if (has_reset2)
120 pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET2, ROGUE_CR_SOFT_RESET2_MASKFULL);
121
122 /* Read soft-reset to fence previous write in order to clear the SOCIF pipeline. */
123 (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET);
124 if (has_reset2)
125 (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET2);
126
127 /* Take Rascal and Dust out of reset. */
128 pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET,
129 soft_reset_mask ^ ROGUE_CR_SOFT_RESET_RASCALDUSTS_EN);
130 if (has_reset2)
131 pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET2, 0);
132
133 (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET);
134 if (has_reset2)
135 (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET2);
136
137 /* Take everything out of reset but the FW processor. */
138 pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET, ROGUE_CR_SOFT_RESET_GARTEN_EN);
139 if (has_reset2)
140 pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET2, 0);
141
142 (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET);
143 if (has_reset2)
144 (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET2);
145
146 err = rogue_slc_init(pvr_dev);
147 if (err)
148 goto err_reset;
149
150 /* Initialise Firmware wrapper. */
151 pvr_dev->fw_dev.defs->wrapper_init(pvr_dev);
152
153 /* We must init the AXI-ACE interface before first BIF transaction. */
154 rogue_axi_ace_list_init(pvr_dev);
155
156 if (pvr_dev->fw_dev.processor_type != PVR_FW_PROCESSOR_TYPE_MIPS) {
157 /* Initialise BIF. */
158 rogue_bif_init(pvr_dev);
159 }
160
161 /* Need to wait for at least 16 cycles before taking the FW processor out of reset ... */
162 udelay(3);
163
164 pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET, 0x0);
165 (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET);
166
167 /* ... and afterwards. */
168 udelay(3);
169
170 return 0;
171
172err_reset:
173 /* Put everything back into soft-reset. */
174 pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET, soft_reset_mask);
175
176 return err;
177}
178
179/**
180 * pvr_fw_stop() - Stop FW processor
181 * @pvr_dev: Target PowerVR device.
182 *
183 * Returns:
184 * * 0 on success, or
185 * * Any error returned by pvr_cr_poll_reg32().
186 */
187int
188pvr_fw_stop(struct pvr_device *pvr_dev)
189{
190 const u32 sidekick_idle_mask = ROGUE_CR_SIDEKICK_IDLE_MASKFULL &
191 ~(ROGUE_CR_SIDEKICK_IDLE_GARTEN_EN |
192 ROGUE_CR_SIDEKICK_IDLE_SOCIF_EN |
193 ROGUE_CR_SIDEKICK_IDLE_HOSTIF_EN);
194 bool skip_garten_idle = false;
195 u32 reg_value;
196 int err;
197
198 /*
199 * Wait for Sidekick/Jones to signal IDLE except for the Garten Wrapper.
200 * For cores with the LAYOUT_MARS feature, SIDEKICK would have been
201 * powered down by the FW.
202 */
203 err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_SIDEKICK_IDLE, sidekick_idle_mask,
204 sidekick_idle_mask, POLL_TIMEOUT_USEC);
205 if (err)
206 return err;
207
208 /* Unset MTS DM association with threads. */
209 pvr_cr_write32(pvr_dev, ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC,
210 ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC_MASKFULL &
211 ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC_DM_ASSOC_CLRMSK);
212 pvr_cr_write32(pvr_dev, ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC,
213 ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC_MASKFULL &
214 ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC_DM_ASSOC_CLRMSK);
215 pvr_cr_write32(pvr_dev, ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC,
216 ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC_MASKFULL &
217 ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC_DM_ASSOC_CLRMSK);
218 pvr_cr_write32(pvr_dev, ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC,
219 ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC_MASKFULL &
220 ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC_DM_ASSOC_CLRMSK);
221
222 /* Extra Idle checks. */
223 err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_BIF_STATUS_MMU, 0,
224 ROGUE_CR_BIF_STATUS_MMU_MASKFULL,
225 POLL_TIMEOUT_USEC);
226 if (err)
227 return err;
228
229 err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_BIFPM_STATUS_MMU, 0,
230 ROGUE_CR_BIFPM_STATUS_MMU_MASKFULL,
231 POLL_TIMEOUT_USEC);
232 if (err)
233 return err;
234
235 if (!PVR_HAS_FEATURE(pvr_dev, xt_top_infrastructure)) {
236 err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_BIF_READS_EXT_STATUS, 0,
237 ROGUE_CR_BIF_READS_EXT_STATUS_MASKFULL,
238 POLL_TIMEOUT_USEC);
239 if (err)
240 return err;
241 }
242
243 err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_BIFPM_READS_EXT_STATUS, 0,
244 ROGUE_CR_BIFPM_READS_EXT_STATUS_MASKFULL,
245 POLL_TIMEOUT_USEC);
246 if (err)
247 return err;
248
249 err = pvr_cr_poll_reg64(pvr_dev, ROGUE_CR_SLC_STATUS1, 0,
250 ROGUE_CR_SLC_STATUS1_MASKFULL,
251 POLL_TIMEOUT_USEC);
252 if (err)
253 return err;
254
255 /*
256 * Wait for SLC to signal IDLE.
257 * For cores with the LAYOUT_MARS feature, SLC would have been powered
258 * down by the FW.
259 */
260 err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_SLC_IDLE,
261 ROGUE_CR_SLC_IDLE_MASKFULL,
262 ROGUE_CR_SLC_IDLE_MASKFULL, POLL_TIMEOUT_USEC);
263 if (err)
264 return err;
265
266 /*
267 * Wait for Sidekick/Jones to signal IDLE except for the Garten Wrapper.
268 * For cores with the LAYOUT_MARS feature, SIDEKICK would have been powered
269 * down by the FW.
270 */
271 err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_SIDEKICK_IDLE, sidekick_idle_mask,
272 sidekick_idle_mask, POLL_TIMEOUT_USEC);
273 if (err)
274 return err;
275
276 if (pvr_dev->fw_dev.processor_type == PVR_FW_PROCESSOR_TYPE_META) {
277 err = pvr_meta_cr_read32(pvr_dev, META_CR_TxVECINT_BHALT, &reg_value);
278 if (err)
279 return err;
280
281 /*
282 * Wait for Sidekick/Jones to signal IDLE including the Garten
283 * Wrapper if there is no debugger attached (TxVECINT_BHALT =
284 * 0x0).
285 */
286 if (reg_value)
287 skip_garten_idle = true;
288 }
289
290 if (!skip_garten_idle) {
291 err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_SIDEKICK_IDLE,
292 ROGUE_CR_SIDEKICK_IDLE_GARTEN_EN,
293 ROGUE_CR_SIDEKICK_IDLE_GARTEN_EN,
294 POLL_TIMEOUT_USEC);
295 if (err)
296 return err;
297 }
298
299 if (PVR_HAS_FEATURE(pvr_dev, pbe2_in_xe))
300 pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET,
301 ROGUE_CR_SOFT_RESET__PBE2_XE__MASKFULL);
302 else
303 pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET, ROGUE_CR_SOFT_RESET_MASKFULL);
304
305 return 0;
306}
drivers/gpu/drm/imagination/pvr_fw_startstop.h+13−0
@@ -0,0 +1,13 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_FW_STARTSTOP_H
5#define PVR_FW_STARTSTOP_H
6
7/* Forward declaration from pvr_device.h. */
8struct pvr_device;
9
10int pvr_fw_start(struct pvr_device *pvr_dev);
11int pvr_fw_stop(struct pvr_device *pvr_dev);
12
13#endif /* PVR_FW_STARTSTOP_H */
drivers/gpu/drm/imagination/pvr_fw_trace.c+515−0
@@ -0,0 +1,515 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_device.h"
5#include "pvr_gem.h"
6#include "pvr_rogue_fwif.h"
7#include "pvr_rogue_fwif_sf.h"
8#include "pvr_fw_trace.h"
9
10#include <drm/drm_drv.h>
11#include <drm/drm_file.h>
12
13#include <linux/build_bug.h>
14#include <linux/dcache.h>
15#include <linux/sysfs.h>
16#include <linux/types.h>
17
18static void
19tracebuf_ctrl_init(void *cpu_ptr, void *priv)
20{
21 struct rogue_fwif_tracebuf *tracebuf_ctrl = cpu_ptr;
22 struct pvr_fw_trace *fw_trace = priv;
23 u32 thread_nr;
24
25 tracebuf_ctrl->tracebuf_size_in_dwords = ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS;
26 tracebuf_ctrl->tracebuf_flags = 0;
27
28 if (fw_trace->group_mask)
29 tracebuf_ctrl->log_type = fw_trace->group_mask | ROGUE_FWIF_LOG_TYPE_TRACE;
30 else
31 tracebuf_ctrl->log_type = ROGUE_FWIF_LOG_TYPE_NONE;
32
33 for (thread_nr = 0; thread_nr < ARRAY_SIZE(fw_trace->buffers); thread_nr++) {
34 struct rogue_fwif_tracebuf_space *tracebuf_space =
35 &tracebuf_ctrl->tracebuf[thread_nr];
36 struct pvr_fw_trace_buffer *trace_buffer = &fw_trace->buffers[thread_nr];
37
38 pvr_fw_object_get_fw_addr(trace_buffer->buf_obj,
39 &tracebuf_space->trace_buffer_fw_addr);
40
41 tracebuf_space->trace_buffer = trace_buffer->buf;
42 tracebuf_space->trace_pointer = 0;
43 }
44}
45
46int pvr_fw_trace_init(struct pvr_device *pvr_dev)
47{
48 struct pvr_fw_trace *fw_trace = &pvr_dev->fw_dev.fw_trace;
49 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
50 u32 thread_nr;
51 int err;
52
53 for (thread_nr = 0; thread_nr < ARRAY_SIZE(fw_trace->buffers); thread_nr++) {
54 struct pvr_fw_trace_buffer *trace_buffer = &fw_trace->buffers[thread_nr];
55
56 trace_buffer->buf =
57 pvr_fw_object_create_and_map(pvr_dev,
58 ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS *
59 sizeof(*trace_buffer->buf),
60 PVR_BO_FW_FLAGS_DEVICE_UNCACHED |
61 PVR_BO_FW_NO_CLEAR_ON_RESET,
62 NULL, NULL, &trace_buffer->buf_obj);
63 if (IS_ERR(trace_buffer->buf)) {
64 drm_err(drm_dev, "Unable to allocate trace buffer\n");
65 err = PTR_ERR(trace_buffer->buf);
66 trace_buffer->buf = NULL;
67 goto err_free_buf;
68 }
69 }
70
71 /* TODO: Provide control of group mask. */
72 fw_trace->group_mask = 0;
73
74 fw_trace->tracebuf_ctrl =
75 pvr_fw_object_create_and_map(pvr_dev,
76 sizeof(*fw_trace->tracebuf_ctrl),
77 PVR_BO_FW_FLAGS_DEVICE_UNCACHED |
78 PVR_BO_FW_NO_CLEAR_ON_RESET,
79 tracebuf_ctrl_init, fw_trace,
80 &fw_trace->tracebuf_ctrl_obj);
81 if (IS_ERR(fw_trace->tracebuf_ctrl)) {
82 drm_err(drm_dev, "Unable to allocate trace buffer control structure\n");
83 err = PTR_ERR(fw_trace->tracebuf_ctrl);
84 goto err_free_buf;
85 }
86
87 BUILD_BUG_ON(ARRAY_SIZE(fw_trace->tracebuf_ctrl->tracebuf) !=
88 ARRAY_SIZE(fw_trace->buffers));
89
90 for (thread_nr = 0; thread_nr < ARRAY_SIZE(fw_trace->buffers); thread_nr++) {
91 struct rogue_fwif_tracebuf_space *tracebuf_space =
92 &fw_trace->tracebuf_ctrl->tracebuf[thread_nr];
93 struct pvr_fw_trace_buffer *trace_buffer = &fw_trace->buffers[thread_nr];
94
95 trace_buffer->tracebuf_space = tracebuf_space;
96 }
97
98 return 0;
99
100err_free_buf:
101 for (thread_nr = 0; thread_nr < ARRAY_SIZE(fw_trace->buffers); thread_nr++) {
102 struct pvr_fw_trace_buffer *trace_buffer = &fw_trace->buffers[thread_nr];
103
104 if (trace_buffer->buf)
105 pvr_fw_object_unmap_and_destroy(trace_buffer->buf_obj);
106 }
107
108 return err;
109}
110
111void pvr_fw_trace_fini(struct pvr_device *pvr_dev)
112{
113 struct pvr_fw_trace *fw_trace = &pvr_dev->fw_dev.fw_trace;
114 u32 thread_nr;
115
116 for (thread_nr = 0; thread_nr < ARRAY_SIZE(fw_trace->buffers); thread_nr++) {
117 struct pvr_fw_trace_buffer *trace_buffer = &fw_trace->buffers[thread_nr];
118
119 pvr_fw_object_unmap_and_destroy(trace_buffer->buf_obj);
120 }
121 pvr_fw_object_unmap_and_destroy(fw_trace->tracebuf_ctrl_obj);
122}
123
124/**
125 * update_logtype() - Send KCCB command to trigger FW to update logtype
126 * @pvr_dev: Target PowerVR device
127 * @group_mask: New log group mask.
128 *
129 * Returns:
130 * * 0 on success,
131 * * Any error returned by pvr_kccb_send_cmd(), or
132 * * -%EIO if the device is lost.
133 */
134static int
135update_logtype(struct pvr_device *pvr_dev, u32 group_mask)
136{
137 struct pvr_fw_trace *fw_trace = &pvr_dev->fw_dev.fw_trace;
138 struct rogue_fwif_kccb_cmd cmd;
139 int idx;
140 int err;
141
142 if (group_mask)
143 fw_trace->tracebuf_ctrl->log_type = ROGUE_FWIF_LOG_TYPE_TRACE | group_mask;
144 else
145 fw_trace->tracebuf_ctrl->log_type = ROGUE_FWIF_LOG_TYPE_NONE;
146
147 fw_trace->group_mask = group_mask;
148
149 down_read(&pvr_dev->reset_sem);
150 if (!drm_dev_enter(from_pvr_device(pvr_dev), &idx)) {
151 err = -EIO;
152 goto err_up_read;
153 }
154
155 cmd.cmd_type = ROGUE_FWIF_KCCB_CMD_LOGTYPE_UPDATE;
156 cmd.kccb_flags = 0;
157
158 err = pvr_kccb_send_cmd(pvr_dev, &cmd, NULL);
159
160 drm_dev_exit(idx);
161
162err_up_read:
163 up_read(&pvr_dev->reset_sem);
164
165 return err;
166}
167
168#if defined(CONFIG_DEBUG_FS)
169
170static int fw_trace_group_mask_show(struct seq_file *m, void *data)
171{
172 struct pvr_device *pvr_dev = m->private;
173
174 seq_printf(m, "%08x\n", pvr_dev->fw_dev.fw_trace.group_mask);
175
176 return 0;
177}
178
179static int fw_trace_group_mask_open(struct inode *inode, struct file *file)
180{
181 return single_open(file, fw_trace_group_mask_show, inode->i_private);
182}
183
184static ssize_t fw_trace_group_mask_write(struct file *file, const char __user *ubuf, size_t len,
185 loff_t *offp)
186{
187 struct seq_file *m = file->private_data;
188 struct pvr_device *pvr_dev = m->private;
189 u32 new_group_mask;
190 int err;
191
192 err = kstrtouint_from_user(ubuf, len, 0, &new_group_mask);
193 if (err)
194 return err;
195
196 err = update_logtype(pvr_dev, new_group_mask);
197 if (err)
198 return err;
199
200 pvr_dev->fw_dev.fw_trace.group_mask = new_group_mask;
201
202 return (ssize_t)len;
203}
204
205static const struct file_operations pvr_fw_trace_group_mask_fops = {
206 .owner = THIS_MODULE,
207 .open = fw_trace_group_mask_open,
208 .read = seq_read,
209 .write = fw_trace_group_mask_write,
210 .llseek = default_llseek,
211 .release = single_release,
212};
213
214struct pvr_fw_trace_seq_data {
215 /** @buffer: Pointer to copy of trace data. */
216 u32 *buffer;
217
218 /** @start_offset: Starting offset in trace data, as reported by FW. */
219 u32 start_offset;
220
221 /** @idx: Current index into trace data. */
222 u32 idx;
223
224 /** @assert_buf: Trace assert buffer, as reported by FW. */
225 struct rogue_fwif_file_info_buf assert_buf;
226};
227
228static u32 find_sfid(u32 id)
229{
230 u32 i;
231
232 for (i = 0; i < ARRAY_SIZE(stid_fmts); i++) {
233 if (stid_fmts[i].id == id)
234 return i;
235 }
236
237 return ROGUE_FW_SF_LAST;
238}
239
240static u32 read_fw_trace(struct pvr_fw_trace_seq_data *trace_seq_data, u32 offset)
241{
242 u32 idx;
243
244 idx = trace_seq_data->idx + offset;
245 if (idx >= ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS)
246 return 0;
247
248 idx = (idx + trace_seq_data->start_offset) % ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS;
249 return trace_seq_data->buffer[idx];
250}
251
252/**
253 * fw_trace_get_next() - Advance trace index to next entry
254 * @trace_seq_data: Trace sequence data.
255 *
256 * Returns:
257 * * %true if trace index is now pointing to a valid entry, or
258 * * %false if trace index is pointing to an invalid entry, or has hit the end
259 * of the trace.
260 */
261static bool fw_trace_get_next(struct pvr_fw_trace_seq_data *trace_seq_data)
262{
263 u32 id, sf_id;
264
265 while (trace_seq_data->idx < ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS) {
266 id = read_fw_trace(trace_seq_data, 0);
267 trace_seq_data->idx++;
268 if (!ROGUE_FW_LOG_VALIDID(id))
269 continue;
270 if (id == ROGUE_FW_SF_MAIN_ASSERT_FAILED) {
271 /* Assertion failure marks the end of the trace. */
272 return false;
273 }
274
275 sf_id = find_sfid(id);
276 if (sf_id == ROGUE_FW_SF_FIRST)
277 continue;
278 if (sf_id == ROGUE_FW_SF_LAST) {
279 /*
280 * Could not match with an ID in the SF table, trace is
281 * most likely corrupt from this point.
282 */
283 return false;
284 }
285
286 /* Skip over the timestamp, and any parameters. */
287 trace_seq_data->idx += 2 + ROGUE_FW_SF_PARAMNUM(id);
288
289 /* Ensure index is now pointing to a valid trace entry. */
290 id = read_fw_trace(trace_seq_data, 0);
291 if (!ROGUE_FW_LOG_VALIDID(id))
292 continue;
293
294 return true;
295 };
296
297 /* Hit end of trace data. */
298 return false;
299}
300
301/**
302 * fw_trace_get_first() - Find first valid entry in trace
303 * @trace_seq_data: Trace sequence data.
304 *
305 * Skips over invalid (usually zero) and ROGUE_FW_SF_FIRST entries.
306 *
307 * If the trace has no valid entries, this function will exit with the trace
308 * index pointing to the end of the trace. trace_seq_show() will return an error
309 * in this state.
310 */
311static void fw_trace_get_first(struct pvr_fw_trace_seq_data *trace_seq_data)
312{
313 trace_seq_data->idx = 0;
314
315 while (trace_seq_data->idx < ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS) {
316 u32 id = read_fw_trace(trace_seq_data, 0);
317
318 if (ROGUE_FW_LOG_VALIDID(id)) {
319 u32 sf_id = find_sfid(id);
320
321 if (sf_id != ROGUE_FW_SF_FIRST)
322 break;
323 }
324 trace_seq_data->idx++;
325 }
326}
327
328static void *fw_trace_seq_start(struct seq_file *s, loff_t *pos)
329{
330 struct pvr_fw_trace_seq_data *trace_seq_data = s->private;
331 u32 i;
332
333 /* Reset trace index, then advance to *pos. */
334 fw_trace_get_first(trace_seq_data);
335
336 for (i = 0; i < *pos; i++) {
337 if (!fw_trace_get_next(trace_seq_data))
338 return NULL;
339 }
340
341 return (trace_seq_data->idx < ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS) ? pos : NULL;
342}
343
344static void *fw_trace_seq_next(struct seq_file *s, void *v, loff_t *pos)
345{
346 struct pvr_fw_trace_seq_data *trace_seq_data = s->private;
347
348 (*pos)++;
349 if (!fw_trace_get_next(trace_seq_data))
350 return NULL;
351
352 return (trace_seq_data->idx < ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS) ? pos : NULL;
353}
354
355static void fw_trace_seq_stop(struct seq_file *s, void *v)
356{
357}
358
359static int fw_trace_seq_show(struct seq_file *s, void *v)
360{
361 struct pvr_fw_trace_seq_data *trace_seq_data = s->private;
362 u64 timestamp;
363 u32 id;
364 u32 sf_id;
365
366 if (trace_seq_data->idx >= ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS)
367 return -EINVAL;
368
369 id = read_fw_trace(trace_seq_data, 0);
370 /* Index is not pointing at a valid entry. */
371 if (!ROGUE_FW_LOG_VALIDID(id))
372 return -EINVAL;
373
374 sf_id = find_sfid(id);
375 /* Index is not pointing at a valid entry. */
376 if (sf_id == ROGUE_FW_SF_LAST)
377 return -EINVAL;
378
379 timestamp = read_fw_trace(trace_seq_data, 1) |
380 ((u64)read_fw_trace(trace_seq_data, 2) << 32);
381 timestamp = (timestamp & ~ROGUE_FWT_TIMESTAMP_TIME_CLRMSK) >>
382 ROGUE_FWT_TIMESTAMP_TIME_SHIFT;
383
384 seq_printf(s, "[%llu] : ", timestamp);
385 if (id == ROGUE_FW_SF_MAIN_ASSERT_FAILED) {
386 seq_printf(s, "ASSERTION %s failed at %s:%u",
387 trace_seq_data->assert_buf.info,
388 trace_seq_data->assert_buf.path,
389 trace_seq_data->assert_buf.line_num);
390 } else {
391 seq_printf(s, stid_fmts[sf_id].name,
392 read_fw_trace(trace_seq_data, 3),
393 read_fw_trace(trace_seq_data, 4),
394 read_fw_trace(trace_seq_data, 5),
395 read_fw_trace(trace_seq_data, 6),
396 read_fw_trace(trace_seq_data, 7),
397 read_fw_trace(trace_seq_data, 8),
398 read_fw_trace(trace_seq_data, 9),
399 read_fw_trace(trace_seq_data, 10),
400 read_fw_trace(trace_seq_data, 11),
401 read_fw_trace(trace_seq_data, 12),
402 read_fw_trace(trace_seq_data, 13),
403 read_fw_trace(trace_seq_data, 14),
404 read_fw_trace(trace_seq_data, 15),
405 read_fw_trace(trace_seq_data, 16),
406 read_fw_trace(trace_seq_data, 17),
407 read_fw_trace(trace_seq_data, 18),
408 read_fw_trace(trace_seq_data, 19),
409 read_fw_trace(trace_seq_data, 20),
410 read_fw_trace(trace_seq_data, 21),
411 read_fw_trace(trace_seq_data, 22));
412 }
413 seq_puts(s, "\n");
414 return 0;
415}
416
417static const struct seq_operations pvr_fw_trace_seq_ops = {
418 .start = fw_trace_seq_start,
419 .next = fw_trace_seq_next,
420 .stop = fw_trace_seq_stop,
421 .show = fw_trace_seq_show
422};
423
424static int fw_trace_open(struct inode *inode, struct file *file)
425{
426 struct pvr_fw_trace_buffer *trace_buffer = inode->i_private;
427 struct rogue_fwif_tracebuf_space *tracebuf_space =
428 trace_buffer->tracebuf_space;
429 struct pvr_fw_trace_seq_data *trace_seq_data;
430 int err;
431
432 trace_seq_data = kzalloc(sizeof(*trace_seq_data), GFP_KERNEL);
433 if (!trace_seq_data)
434 return -ENOMEM;
435
436 trace_seq_data->buffer = kcalloc(ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS,
437 sizeof(*trace_seq_data->buffer), GFP_KERNEL);
438 if (!trace_seq_data->buffer) {
439 err = -ENOMEM;
440 goto err_free_data;
441 }
442
443 /*
444 * Take a local copy of the trace buffer, as firmware may still be
445 * writing to it. This will exist as long as this file is open.
446 */
447 memcpy(trace_seq_data->buffer, trace_buffer->buf,
448 ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS * sizeof(u32));
449 trace_seq_data->start_offset = READ_ONCE(tracebuf_space->trace_pointer);
450 trace_seq_data->assert_buf = tracebuf_space->assert_buf;
451 fw_trace_get_first(trace_seq_data);
452
453 err = seq_open(file, &pvr_fw_trace_seq_ops);
454 if (err)
455 goto err_free_buffer;
456
457 ((struct seq_file *)file->private_data)->private = trace_seq_data;
458
459 return 0;
460
461err_free_buffer:
462 kfree(trace_seq_data->buffer);
463
464err_free_data:
465 kfree(trace_seq_data);
466
467 return err;
468}
469
470static int fw_trace_release(struct inode *inode, struct file *file)
471{
472 struct pvr_fw_trace_seq_data *trace_seq_data =
473 ((struct seq_file *)file->private_data)->private;
474
475 seq_release(inode, file);
476 kfree(trace_seq_data->buffer);
477 kfree(trace_seq_data);
478
479 return 0;
480}
481
482static const struct file_operations pvr_fw_trace_fops = {
483 .owner = THIS_MODULE,
484 .open = fw_trace_open,
485 .read = seq_read,
486 .llseek = seq_lseek,
487 .release = fw_trace_release,
488};
489
490void
491pvr_fw_trace_mask_update(struct pvr_device *pvr_dev, u32 old_mask, u32 new_mask)
492{
493 if (old_mask != new_mask)
494 update_logtype(pvr_dev, new_mask);
495}
496
497void
498pvr_fw_trace_debugfs_init(struct pvr_device *pvr_dev, struct dentry *dir)
499{
500 struct pvr_fw_trace *fw_trace = &pvr_dev->fw_dev.fw_trace;
501 u32 thread_nr;
502
503 static_assert(ARRAY_SIZE(fw_trace->buffers) <= 10,
504 "The filename buffer is only large enough for a single-digit thread count");
505
506 for (thread_nr = 0; thread_nr < ARRAY_SIZE(fw_trace->buffers); ++thread_nr) {
507 char filename[8];
508
509 snprintf(filename, ARRAY_SIZE(filename), "trace_%u", thread_nr);
510 debugfs_create_file(filename, 0400, dir,
511 &fw_trace->buffers[thread_nr],
512 &pvr_fw_trace_fops);
513 }
514}
515#endif
drivers/gpu/drm/imagination/pvr_fw_trace.h+78−0
@@ -0,0 +1,78 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_FW_TRACE_H
5#define PVR_FW_TRACE_H
6
7#include <drm/drm_file.h>
8#include <linux/types.h>
9
10#include "pvr_rogue_fwif.h"
11
12/* Forward declaration from pvr_device.h. */
13struct pvr_device;
14
15/* Forward declaration from pvr_gem.h. */
16struct pvr_fw_object;
17
18/* Forward declarations from pvr_rogue_fwif.h */
19struct rogue_fwif_tracebuf;
20struct rogue_fwif_tracebuf_space;
21
22/**
23 * struct pvr_fw_trace_buffer - Structure representing a trace buffer
24 */
25struct pvr_fw_trace_buffer {
26 /** @buf_obj: FW buffer object representing trace buffer. */
27 struct pvr_fw_object *buf_obj;
28
29 /** @buf: Pointer to CPU mapping of trace buffer. */
30 u32 *buf;
31
32 /**
33 * @tracebuf_space: Pointer to FW tracebuf_space structure for this
34 * trace buffer.
35 */
36 struct rogue_fwif_tracebuf_space *tracebuf_space;
37};
38
39/**
40 * struct pvr_fw_trace - Device firmware trace data
41 */
42struct pvr_fw_trace {
43 /**
44 * @tracebuf_ctrl_obj: Object representing FW trace buffer control
45 * structure.
46 */
47 struct pvr_fw_object *tracebuf_ctrl_obj;
48
49 /**
50 * @tracebuf_ctrl: Pointer to CPU mapping of FW trace buffer control
51 * structure.
52 */
53 struct rogue_fwif_tracebuf *tracebuf_ctrl;
54
55 /**
56 * @buffers: Array representing the actual trace buffers owned by this
57 * device.
58 */
59 struct pvr_fw_trace_buffer buffers[ROGUE_FW_THREAD_MAX];
60
61 /** @group_mask: Mask of enabled trace groups. */
62 u32 group_mask;
63};
64
65int pvr_fw_trace_init(struct pvr_device *pvr_dev);
66void pvr_fw_trace_fini(struct pvr_device *pvr_dev);
67
68#if defined(CONFIG_DEBUG_FS)
69/* Forward declaration from <linux/dcache.h>. */
70struct dentry;
71
72void pvr_fw_trace_mask_update(struct pvr_device *pvr_dev, u32 old_mask,
73 u32 new_mask);
74
75void pvr_fw_trace_debugfs_init(struct pvr_device *pvr_dev, struct dentry *dir);
76#endif /* defined(CONFIG_DEBUG_FS) */
77
78#endif /* PVR_FW_TRACE_H */
drivers/gpu/drm/imagination/pvr_gem.c+414−0
@@ -0,0 +1,414 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_device.h"
5#include "pvr_gem.h"
6#include "pvr_vm.h"
7
8#include <drm/drm_gem.h>
9#include <drm/drm_prime.h>
10
11#include <linux/compiler.h>
12#include <linux/compiler_attributes.h>
13#include <linux/dma-buf.h>
14#include <linux/dma-direction.h>
15#include <linux/dma-mapping.h>
16#include <linux/err.h>
17#include <linux/gfp.h>
18#include <linux/iosys-map.h>
19#include <linux/log2.h>
20#include <linux/mutex.h>
21#include <linux/pagemap.h>
22#include <linux/refcount.h>
23#include <linux/scatterlist.h>
24
25static void pvr_gem_object_free(struct drm_gem_object *obj)
26{
27 drm_gem_shmem_object_free(obj);
28}
29
30static int pvr_gem_mmap(struct drm_gem_object *gem_obj, struct vm_area_struct *vma)
31{
32 struct pvr_gem_object *pvr_obj = gem_to_pvr_gem(gem_obj);
33 struct drm_gem_shmem_object *shmem_obj = shmem_gem_from_pvr_gem(pvr_obj);
34
35 if (!(pvr_obj->flags & DRM_PVR_BO_ALLOW_CPU_USERSPACE_ACCESS))
36 return -EINVAL;
37
38 return drm_gem_shmem_mmap(shmem_obj, vma);
39}
40
41static const struct drm_gem_object_funcs pvr_gem_object_funcs = {
42 .free = pvr_gem_object_free,
43 .print_info = drm_gem_shmem_object_print_info,
44 .pin = drm_gem_shmem_object_pin,
45 .unpin = drm_gem_shmem_object_unpin,
46 .get_sg_table = drm_gem_shmem_object_get_sg_table,
47 .vmap = drm_gem_shmem_object_vmap,
48 .vunmap = drm_gem_shmem_object_vunmap,
49 .mmap = pvr_gem_mmap,
50 .vm_ops = &drm_gem_shmem_vm_ops,
51};
52
53/**
54 * pvr_gem_object_flags_validate() - Verify that a collection of PowerVR GEM
55 * mapping and/or creation flags form a valid combination.
56 * @flags: PowerVR GEM mapping/creation flags to validate.
57 *
58 * This function explicitly allows kernel-only flags. All ioctl entrypoints
59 * should do their own validation as well as relying on this function.
60 *
61 * Return:
62 * * %true if @flags contains valid mapping and/or creation flags, or
63 * * %false otherwise.
64 */
65static bool
66pvr_gem_object_flags_validate(u64 flags)
67{
68 static const u64 invalid_combinations[] = {
69 /*
70 * Memory flagged as PM/FW-protected cannot be mapped to
71 * userspace. To make this explicit, we require that the two
72 * flags allowing each of these respective features are never
73 * specified together.
74 */
75 (DRM_PVR_BO_PM_FW_PROTECT |
76 DRM_PVR_BO_ALLOW_CPU_USERSPACE_ACCESS),
77 };
78
79 int i;
80
81 /*
82 * Check for bits set in undefined regions. Reserved regions refer to
83 * options that can only be set by the kernel. These are explicitly
84 * allowed in most cases, and must be checked specifically in IOCTL
85 * callback code.
86 */
87 if ((flags & PVR_BO_UNDEFINED_MASK) != 0)
88 return false;
89
90 /*
91 * Check for all combinations of flags marked as invalid in the array
92 * above.
93 */
94 for (i = 0; i < ARRAY_SIZE(invalid_combinations); ++i) {
95 u64 combo = invalid_combinations[i];
96
97 if ((flags & combo) == combo)
98 return false;
99 }
100
101 return true;
102}
103
104/**
105 * pvr_gem_object_into_handle() - Convert a reference to an object into a
106 * userspace-accessible handle.
107 * @pvr_obj: [IN] Target PowerVR-specific object.
108 * @pvr_file: [IN] File to associate the handle with.
109 * @handle: [OUT] Pointer to store the created handle in. Remains unmodified if
110 * an error is encountered.
111 *
112 * If an error is encountered, ownership of @pvr_obj will not have been
113 * transferred. If this function succeeds, however, further use of @pvr_obj is
114 * considered undefined behaviour unless another reference to it is explicitly
115 * held.
116 *
117 * Return:
118 * * 0 on success, or
119 * * Any error encountered while attempting to allocate a handle on @pvr_file.
120 */
121int
122pvr_gem_object_into_handle(struct pvr_gem_object *pvr_obj,
123 struct pvr_file *pvr_file, u32 *handle)
124{
125 struct drm_gem_object *gem_obj = gem_from_pvr_gem(pvr_obj);
126 struct drm_file *file = from_pvr_file(pvr_file);
127
128 u32 new_handle;
129 int err;
130
131 err = drm_gem_handle_create(file, gem_obj, &new_handle);
132 if (err)
133 return err;
134
135 /*
136 * Release our reference to @pvr_obj, effectively transferring
137 * ownership to the handle.
138 */
139 pvr_gem_object_put(pvr_obj);
140
141 /*
142 * Do not store the new handle in @handle until no more errors can
143 * occur.
144 */
145 *handle = new_handle;
146
147 return 0;
148}
149
150/**
151 * pvr_gem_object_from_handle() - Obtain a reference to an object from a
152 * userspace handle.
153 * @pvr_file: PowerVR-specific file to which @handle is associated.
154 * @handle: Userspace handle referencing the target object.
155 *
156 * On return, @handle always maintains its reference to the requested object
157 * (if it had one in the first place). If this function succeeds, the returned
158 * object will hold an additional reference. When the caller is finished with
159 * the returned object, they should call pvr_gem_object_put() on it to release
160 * this reference.
161 *
162 * Return:
163 * * A pointer to the requested PowerVR-specific object on success, or
164 * * %NULL otherwise.
165 */
166struct pvr_gem_object *
167pvr_gem_object_from_handle(struct pvr_file *pvr_file, u32 handle)
168{
169 struct drm_file *file = from_pvr_file(pvr_file);
170 struct drm_gem_object *gem_obj;
171
172 gem_obj = drm_gem_object_lookup(file, handle);
173 if (!gem_obj)
174 return NULL;
175
176 return gem_to_pvr_gem(gem_obj);
177}
178
179/**
180 * pvr_gem_object_vmap() - Map a PowerVR GEM object into CPU virtual address
181 * space.
182 * @pvr_obj: Target PowerVR GEM object.
183 *
184 * Once the caller is finished with the CPU mapping, they must call
185 * pvr_gem_object_vunmap() on @pvr_obj.
186 *
187 * If @pvr_obj is CPU-cached, dma_sync_sgtable_for_cpu() is called to make
188 * sure the CPU mapping is consistent.
189 *
190 * Return:
191 * * A pointer to the CPU mapping on success,
192 * * -%ENOMEM if the mapping fails, or
193 * * Any error encountered while attempting to acquire a reference to the
194 * backing pages for @pvr_obj.
195 */
196void *
197pvr_gem_object_vmap(struct pvr_gem_object *pvr_obj)
198{
199 struct drm_gem_shmem_object *shmem_obj = shmem_gem_from_pvr_gem(pvr_obj);
200 struct drm_gem_object *obj = gem_from_pvr_gem(pvr_obj);
201 struct iosys_map map;
202 int err;
203
204 dma_resv_lock(obj->resv, NULL);
205
206 err = drm_gem_shmem_vmap(shmem_obj, &map);
207 if (err)
208 goto err_unlock;
209
210 if (pvr_obj->flags & PVR_BO_CPU_CACHED) {
211 struct device *dev = shmem_obj->base.dev->dev;
212
213 /* If shmem_obj->sgt is NULL, that means the buffer hasn't been mapped
214 * in GPU space yet.
215 */
216 if (shmem_obj->sgt)
217 dma_sync_sgtable_for_cpu(dev, shmem_obj->sgt, DMA_BIDIRECTIONAL);
218 }
219
220 dma_resv_unlock(obj->resv);
221
222 return map.vaddr;
223
224err_unlock:
225 dma_resv_unlock(obj->resv);
226
227 return ERR_PTR(err);
228}
229
230/**
231 * pvr_gem_object_vunmap() - Unmap a PowerVR memory object from CPU virtual
232 * address space.
233 * @pvr_obj: Target PowerVR GEM object.
234 *
235 * If @pvr_obj is CPU-cached, dma_sync_sgtable_for_device() is called to make
236 * sure the GPU mapping is consistent.
237 */
238void
239pvr_gem_object_vunmap(struct pvr_gem_object *pvr_obj)
240{
241 struct drm_gem_shmem_object *shmem_obj = shmem_gem_from_pvr_gem(pvr_obj);
242 struct iosys_map map = IOSYS_MAP_INIT_VADDR(shmem_obj->vaddr);
243 struct drm_gem_object *obj = gem_from_pvr_gem(pvr_obj);
244
245 if (WARN_ON(!map.vaddr))
246 return;
247
248 dma_resv_lock(obj->resv, NULL);
249
250 if (pvr_obj->flags & PVR_BO_CPU_CACHED) {
251 struct device *dev = shmem_obj->base.dev->dev;
252
253 /* If shmem_obj->sgt is NULL, that means the buffer hasn't been mapped
254 * in GPU space yet.
255 */
256 if (shmem_obj->sgt)
257 dma_sync_sgtable_for_device(dev, shmem_obj->sgt, DMA_BIDIRECTIONAL);
258 }
259
260 drm_gem_shmem_vunmap(shmem_obj, &map);
261
262 dma_resv_unlock(obj->resv);
263}
264
265/**
266 * pvr_gem_object_zero() - Zeroes the physical memory behind an object.
267 * @pvr_obj: Target PowerVR GEM object.
268 *
269 * Return:
270 * * 0 on success, or
271 * * Any error encountered while attempting to map @pvr_obj to the CPU (see
272 * pvr_gem_object_vmap()).
273 */
274static int
275pvr_gem_object_zero(struct pvr_gem_object *pvr_obj)
276{
277 void *cpu_ptr;
278
279 cpu_ptr = pvr_gem_object_vmap(pvr_obj);
280 if (IS_ERR(cpu_ptr))
281 return PTR_ERR(cpu_ptr);
282
283 memset(cpu_ptr, 0, pvr_gem_object_size(pvr_obj));
284
285 /* Make sure the zero-ing is done before vumap-ing the object. */
286 wmb();
287
288 pvr_gem_object_vunmap(pvr_obj);
289
290 return 0;
291}
292
293/**
294 * pvr_gem_create_object() - Allocate and pre-initializes a pvr_gem_object
295 * @drm_dev: DRM device creating this object.
296 * @size: Size of the object to allocate in bytes.
297 *
298 * Return:
299 * * The new pre-initialized GEM object on success,
300 * * -ENOMEM if the allocation failed.
301 */
302struct drm_gem_object *pvr_gem_create_object(struct drm_device *drm_dev, size_t size)
303{
304 struct drm_gem_object *gem_obj;
305 struct pvr_gem_object *pvr_obj;
306
307 pvr_obj = kzalloc(sizeof(*pvr_obj), GFP_KERNEL);
308 if (!pvr_obj)
309 return ERR_PTR(-ENOMEM);
310
311 gem_obj = gem_from_pvr_gem(pvr_obj);
312 gem_obj->funcs = &pvr_gem_object_funcs;
313
314 return gem_obj;
315}
316
317/**
318 * pvr_gem_object_create() - Creates a PowerVR-specific buffer object.
319 * @pvr_dev: Target PowerVR device.
320 * @size: Size of the object to allocate in bytes. Must be greater than zero.
321 * Any value which is not an exact multiple of the system page size will be
322 * rounded up to satisfy this condition.
323 * @flags: Options which affect both this operation and future mapping
324 * operations performed on the returned object. Must be a combination of
325 * DRM_PVR_BO_* and/or PVR_BO_* flags.
326 *
327 * The created object may be larger than @size, but can never be smaller. To
328 * get the exact size, call pvr_gem_object_size() on the returned pointer.
329 *
330 * Return:
331 * * The newly-minted PowerVR-specific buffer object on success,
332 * * -%EINVAL if @size is zero or @flags is not valid,
333 * * -%ENOMEM if sufficient physical memory cannot be allocated, or
334 * * Any other error returned by drm_gem_create_mmap_offset().
335 */
336struct pvr_gem_object *
337pvr_gem_object_create(struct pvr_device *pvr_dev, size_t size, u64 flags)
338{
339 struct drm_gem_shmem_object *shmem_obj;
340 struct pvr_gem_object *pvr_obj;
341 struct sg_table *sgt;
342 int err;
343
344 /* Verify @size and @flags before continuing. */
345 if (size == 0 || !pvr_gem_object_flags_validate(flags))
346 return ERR_PTR(-EINVAL);
347
348 shmem_obj = drm_gem_shmem_create(from_pvr_device(pvr_dev), size);
349 if (IS_ERR(shmem_obj))
350 return ERR_CAST(shmem_obj);
351
352 shmem_obj->pages_mark_dirty_on_put = true;
353 shmem_obj->map_wc = !(flags & PVR_BO_CPU_CACHED);
354 pvr_obj = shmem_gem_to_pvr_gem(shmem_obj);
355 pvr_obj->flags = flags;
356
357 sgt = drm_gem_shmem_get_pages_sgt(shmem_obj);
358 if (IS_ERR(sgt)) {
359 err = PTR_ERR(sgt);
360 goto err_shmem_object_free;
361 }
362
363 dma_sync_sgtable_for_device(shmem_obj->base.dev->dev, sgt,
364 DMA_BIDIRECTIONAL);
365
366 /*
367 * Do this last because pvr_gem_object_zero() requires a fully
368 * configured instance of struct pvr_gem_object.
369 */
370 pvr_gem_object_zero(pvr_obj);
371
372 return pvr_obj;
373
374err_shmem_object_free:
375 drm_gem_shmem_free(shmem_obj);
376
377 return ERR_PTR(err);
378}
379
380/**
381 * pvr_gem_get_dma_addr() - Get DMA address for given offset in object
382 * @pvr_obj: Pointer to object to lookup address in.
383 * @offset: Offset within object to lookup address at.
384 * @dma_addr_out: Pointer to location to store DMA address.
385 *
386 * Returns:
387 * * 0 on success, or
388 * * -%EINVAL if object is not currently backed, or if @offset is out of valid
389 * range for this object.
390 */
391int
392pvr_gem_get_dma_addr(struct pvr_gem_object *pvr_obj, u32 offset,
393 dma_addr_t *dma_addr_out)
394{
395 struct drm_gem_shmem_object *shmem_obj = shmem_gem_from_pvr_gem(pvr_obj);
396 u32 accumulated_offset = 0;
397 struct scatterlist *sgl;
398 unsigned int sgt_idx;
399
400 WARN_ON(!shmem_obj->sgt);
401 for_each_sgtable_dma_sg(shmem_obj->sgt, sgl, sgt_idx) {
402 u32 new_offset = accumulated_offset + sg_dma_len(sgl);
403
404 if (offset >= accumulated_offset && offset < new_offset) {
405 *dma_addr_out = sg_dma_address(sgl) +
406 (offset - accumulated_offset);
407 return 0;
408 }
409
410 accumulated_offset = new_offset;
411 }
412
413 return -EINVAL;
414}
drivers/gpu/drm/imagination/pvr_gem.h+170−0
@@ -0,0 +1,170 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_GEM_H
5#define PVR_GEM_H
6
7#include "pvr_rogue_heap_config.h"
8#include "pvr_rogue_meta.h"
9
10#include <uapi/drm/pvr_drm.h>
11
12#include <drm/drm_gem.h>
13#include <drm/drm_gem_shmem_helper.h>
14#include <drm/drm_mm.h>
15
16#include <linux/bitfield.h>
17#include <linux/bits.h>
18#include <linux/const.h>
19#include <linux/compiler_attributes.h>
20#include <linux/kernel.h>
21#include <linux/mutex.h>
22#include <linux/refcount.h>
23#include <linux/scatterlist.h>
24#include <linux/sizes.h>
25#include <linux/types.h>
26
27/* Forward declaration from "pvr_device.h". */
28struct pvr_device;
29struct pvr_file;
30
31/**
32 * DOC: Flags for DRM_IOCTL_PVR_CREATE_BO (kernel-only)
33 *
34 * Kernel-only values allowed in &pvr_gem_object->flags. The majority of options
35 * for this field are specified in the UAPI header "pvr_drm.h" with a
36 * DRM_PVR_BO_ prefix. To distinguish these internal options (which must exist
37 * in ranges marked as "reserved" in the UAPI header), we drop the DRM prefix.
38 * The public options should be used directly, DRM prefix and all.
39 *
40 * To avoid potentially confusing gaps in the UAPI options, these kernel-only
41 * options are specified "in reverse", starting at bit 63.
42 *
43 * We use "reserved" to refer to bits defined here and not exposed in the UAPI.
44 * Bits not defined anywhere are "undefined".
45 *
46 * CPU mapping options
47 * :PVR_BO_CPU_CACHED: By default, all GEM objects are mapped write-combined on the CPU. Set this
48 * flag to override this behaviour and map the object cached.
49 *
50 * Firmware options
51 * :PVR_BO_FW_NO_CLEAR_ON_RESET: By default, all FW objects are cleared and reinitialised on hard
52 * reset. Set this flag to override this behaviour and preserve buffer contents on reset.
53 */
54#define PVR_BO_CPU_CACHED BIT_ULL(63)
55
56#define PVR_BO_FW_NO_CLEAR_ON_RESET BIT_ULL(62)
57
58#define PVR_BO_KERNEL_FLAGS_MASK (PVR_BO_CPU_CACHED | PVR_BO_FW_NO_CLEAR_ON_RESET)
59
60/* Bits 61..3 are undefined. */
61/* Bits 2..0 are defined in the UAPI. */
62
63/* Other utilities. */
64#define PVR_BO_UNDEFINED_MASK ~(PVR_BO_KERNEL_FLAGS_MASK | DRM_PVR_BO_FLAGS_MASK)
65
66/*
67 * All firmware-mapped memory uses (mostly) the same flags. Specifically,
68 * firmware-mapped memory should be:
69 * * Read/write on the device,
70 * * Read/write on the CPU, and
71 * * Write-combined on the CPU.
72 *
73 * The only variation is in caching on the device.
74 */
75#define PVR_BO_FW_FLAGS_DEVICE_CACHED (ULL(0))
76#define PVR_BO_FW_FLAGS_DEVICE_UNCACHED DRM_PVR_BO_BYPASS_DEVICE_CACHE
77
78/**
79 * struct pvr_gem_object - powervr-specific wrapper for &struct drm_gem_object
80 */
81struct pvr_gem_object {
82 /**
83 * @base: The underlying &struct drm_gem_shmem_object.
84 *
85 * Do not access this member directly, instead call
86 * shem_gem_from_pvr_gem().
87 */
88 struct drm_gem_shmem_object base;
89
90 /**
91 * @flags: Options set at creation-time. Some of these options apply to
92 * the creation operation itself (which are stored here for reference)
93 * with the remainder used for mapping options to both the device and
94 * CPU. These are used every time this object is mapped, but may be
95 * changed after creation.
96 *
97 * Must be a combination of DRM_PVR_BO_* and/or PVR_BO_* flags.
98 *
99 * .. note::
100 *
101 * This member is declared const to indicate that none of these
102 * options may change or be changed throughout the object's
103 * lifetime.
104 */
105 u64 flags;
106
107};
108
109static_assert(offsetof(struct pvr_gem_object, base) == 0,
110 "offsetof(struct pvr_gem_object, base) not zero");
111
112#define shmem_gem_from_pvr_gem(pvr_obj) (&(pvr_obj)->base)
113
114#define shmem_gem_to_pvr_gem(shmem_obj) container_of_const(shmem_obj, struct pvr_gem_object, base)
115
116#define gem_from_pvr_gem(pvr_obj) (&(pvr_obj)->base.base)
117
118#define gem_to_pvr_gem(gem_obj) container_of_const(gem_obj, struct pvr_gem_object, base.base)
119
120/* Functions defined in pvr_gem.c */
121
122struct drm_gem_object *pvr_gem_create_object(struct drm_device *drm_dev, size_t size);
123
124struct pvr_gem_object *pvr_gem_object_create(struct pvr_device *pvr_dev,
125 size_t size, u64 flags);
126
127int pvr_gem_object_into_handle(struct pvr_gem_object *pvr_obj,
128 struct pvr_file *pvr_file, u32 *handle);
129struct pvr_gem_object *pvr_gem_object_from_handle(struct pvr_file *pvr_file,
130 u32 handle);
131
132static __always_inline struct sg_table *
133pvr_gem_object_get_pages_sgt(struct pvr_gem_object *pvr_obj)
134{
135 return drm_gem_shmem_get_pages_sgt(shmem_gem_from_pvr_gem(pvr_obj));
136}
137
138void *pvr_gem_object_vmap(struct pvr_gem_object *pvr_obj);
139void pvr_gem_object_vunmap(struct pvr_gem_object *pvr_obj);
140
141int pvr_gem_get_dma_addr(struct pvr_gem_object *pvr_obj, u32 offset,
142 dma_addr_t *dma_addr_out);
143
144/**
145 * pvr_gem_object_get() - Acquire reference on pvr_gem_object
146 * @pvr_obj: Pointer to object to acquire reference on.
147 */
148static __always_inline void
149pvr_gem_object_get(struct pvr_gem_object *pvr_obj)
150{
151 drm_gem_object_get(gem_from_pvr_gem(pvr_obj));
152}
153
154/**
155 * pvr_gem_object_put() - Release reference on pvr_gem_object
156 * @pvr_obj: Pointer to object to release reference on.
157 */
158static __always_inline void
159pvr_gem_object_put(struct pvr_gem_object *pvr_obj)
160{
161 drm_gem_object_put(gem_from_pvr_gem(pvr_obj));
162}
163
164static __always_inline size_t
165pvr_gem_object_size(struct pvr_gem_object *pvr_obj)
166{
167 return gem_from_pvr_gem(pvr_obj)->size;
168}
169
170#endif /* PVR_GEM_H */
drivers/gpu/drm/imagination/pvr_hwrt.c+549−0
@@ -0,0 +1,549 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_free_list.h"
5#include "pvr_hwrt.h"
6#include "pvr_gem.h"
7#include "pvr_rogue_cr_defs_client.h"
8#include "pvr_rogue_fwif.h"
9
10#include <drm/drm_gem.h>
11#include <linux/bitops.h>
12#include <linux/math.h>
13#include <linux/slab.h>
14#include <linux/xarray.h>
15#include <uapi/drm/pvr_drm.h>
16
17static_assert(ROGUE_FWIF_NUM_RTDATAS == 2);
18static_assert(ROGUE_FWIF_NUM_GEOMDATAS == 1);
19static_assert(ROGUE_FWIF_NUM_RTDATA_FREELISTS == 2);
20
21/*
22 * struct pvr_rt_mtile_info - Render target macrotile information
23 */
24struct pvr_rt_mtile_info {
25 u32 mtile_x[3];
26 u32 mtile_y[3];
27 u32 tile_max_x;
28 u32 tile_max_y;
29 u32 tile_size_x;
30 u32 tile_size_y;
31 u32 num_tiles_x;
32 u32 num_tiles_y;
33};
34
35/* Size of Shadow Render Target Cache entry */
36#define SRTC_ENTRY_SIZE sizeof(u32)
37/* Size of Renders Accumulation Array entry */
38#define RAA_ENTRY_SIZE sizeof(u32)
39
40static int
41hwrt_init_kernel_structure(struct pvr_file *pvr_file,
42 struct drm_pvr_ioctl_create_hwrt_dataset_args *args,
43 struct pvr_hwrt_dataset *hwrt)
44{
45 struct pvr_device *pvr_dev = pvr_file->pvr_dev;
46 int err;
47 int i;
48
49 hwrt->pvr_dev = pvr_dev;
50 hwrt->max_rts = args->layers;
51
52 /* Get pointers to the free lists */
53 for (i = 0; i < ARRAY_SIZE(hwrt->free_lists); i++) {
54 hwrt->free_lists[i] = pvr_free_list_lookup(pvr_file, args->free_list_handles[i]);
55 if (!hwrt->free_lists[i]) {
56 err = -EINVAL;
57 goto err_put_free_lists;
58 }
59 }
60
61 if (hwrt->free_lists[ROGUE_FW_LOCAL_FREELIST]->current_pages <
62 pvr_get_free_list_min_pages(pvr_dev)) {
63 err = -EINVAL;
64 goto err_put_free_lists;
65 }
66
67 return 0;
68
69err_put_free_lists:
70 for (i = 0; i < ARRAY_SIZE(hwrt->free_lists); i++) {
71 pvr_free_list_put(hwrt->free_lists[i]);
72 hwrt->free_lists[i] = NULL;
73 }
74
75 return err;
76}
77
78static void
79hwrt_fini_kernel_structure(struct pvr_hwrt_dataset *hwrt)
80{
81 int i;
82
83 for (i = 0; i < ARRAY_SIZE(hwrt->free_lists); i++) {
84 pvr_free_list_put(hwrt->free_lists[i]);
85 hwrt->free_lists[i] = NULL;
86 }
87}
88
89static void
90hwrt_fini_common_fw_structure(struct pvr_hwrt_dataset *hwrt)
91{
92 pvr_fw_object_destroy(hwrt->common_fw_obj);
93}
94
95static int
96get_cr_isp_mtile_size_val(struct pvr_device *pvr_dev, u32 samples,
97 struct pvr_rt_mtile_info *info, u32 *value_out)
98{
99 u32 x = info->mtile_x[0];
100 u32 y = info->mtile_y[0];
101 u32 samples_per_pixel;
102 int err;
103
104 err = PVR_FEATURE_VALUE(pvr_dev, isp_samples_per_pixel, &samples_per_pixel);
105 if (err)
106 return err;
107
108 if (samples_per_pixel == 1) {
109 if (samples >= 4)
110 x <<= 1;
111 if (samples >= 2)
112 y <<= 1;
113 } else if (samples_per_pixel == 2) {
114 if (samples >= 8)
115 x <<= 1;
116 if (samples >= 4)
117 y <<= 1;
118 } else if (samples_per_pixel == 4) {
119 if (samples >= 8)
120 y <<= 1;
121 } else {
122 WARN(true, "Unsupported ISP samples per pixel value");
123 return -EINVAL;
124 }
125
126 *value_out = ((x << ROGUE_CR_ISP_MTILE_SIZE_X_SHIFT) & ~ROGUE_CR_ISP_MTILE_SIZE_X_CLRMSK) |
127 ((y << ROGUE_CR_ISP_MTILE_SIZE_Y_SHIFT) & ~ROGUE_CR_ISP_MTILE_SIZE_Y_CLRMSK);
128
129 return 0;
130}
131
132static int
133get_cr_multisamplectl_val(u32 samples, bool y_flip, u64 *value_out)
134{
135 static const struct {
136 u8 x[8];
137 u8 y[8];
138 } sample_positions[4] = {
139 /* 1 sample */
140 {
141 .x = { 8 },
142 .y = { 8 },
143 },
144 /* 2 samples */
145 {
146 .x = { 12, 4 },
147 .y = { 12, 4 },
148 },
149 /* 4 samples */
150 {
151 .x = { 6, 14, 2, 10 },
152 .y = { 2, 6, 10, 14 },
153 },
154 /* 8 samples */
155 {
156 .x = { 9, 7, 13, 5, 3, 1, 11, 15 },
157 .y = { 5, 11, 9, 3, 13, 7, 15, 1 },
158 },
159 };
160 const int idx = fls(samples) - 1;
161 u64 value = 0;
162
163 if (idx < 0 || idx > 3)
164 return -EINVAL;
165
166 for (u32 i = 0; i < 8; i++) {
167 value |= ((u64)sample_positions[idx].x[i]) << (i * 8);
168 if (y_flip)
169 value |= (((u64)(16 - sample_positions[idx].y[i]) & 0xf)) << (i * 8 + 4);
170 else
171 value |= ((u64)sample_positions[idx].y[i]) << (i * 8 + 4);
172 }
173
174 *value_out = value;
175
176 return 0;
177}
178
179static int
180get_cr_te_aa_val(struct pvr_device *pvr_dev, u32 samples, u32 *value_out)
181{
182 u32 samples_per_pixel;
183 u32 value = 0;
184 int err = 0;
185
186 err = PVR_FEATURE_VALUE(pvr_dev, isp_samples_per_pixel, &samples_per_pixel);
187 if (err)
188 return err;
189
190 switch (samples_per_pixel) {
191 case 1:
192 if (samples >= 2)
193 value |= ROGUE_CR_TE_AA_Y_EN;
194 if (samples >= 4)
195 value |= ROGUE_CR_TE_AA_X_EN;
196 break;
197 case 2:
198 if (samples >= 2)
199 value |= ROGUE_CR_TE_AA_X2_EN;
200 if (samples >= 4)
201 value |= ROGUE_CR_TE_AA_Y_EN;
202 if (samples >= 8)
203 value |= ROGUE_CR_TE_AA_X_EN;
204 break;
205 case 4:
206 if (samples >= 2)
207 value |= ROGUE_CR_TE_AA_X2_EN;
208 if (samples >= 4)
209 value |= ROGUE_CR_TE_AA_Y2_EN;
210 if (samples >= 8)
211 value |= ROGUE_CR_TE_AA_Y_EN;
212 break;
213 default:
214 WARN(true, "Unsupported ISP samples per pixel value");
215 return -EINVAL;
216 }
217
218 *value_out = value;
219
220 return 0;
221}
222
223static void
224hwrtdata_common_init(void *cpu_ptr, void *priv)
225{
226 struct pvr_hwrt_dataset *hwrt = priv;
227
228 memcpy(cpu_ptr, &hwrt->common, sizeof(hwrt->common));
229}
230
231static int
232hwrt_init_common_fw_structure(struct pvr_file *pvr_file,
233 struct drm_pvr_ioctl_create_hwrt_dataset_args *args,
234 struct pvr_hwrt_dataset *hwrt)
235{
236 struct drm_pvr_create_hwrt_geom_data_args *geom_data_args = &args->geom_data_args;
237 struct pvr_device *pvr_dev = pvr_file->pvr_dev;
238 struct pvr_rt_mtile_info info;
239 int err;
240
241 err = PVR_FEATURE_VALUE(pvr_dev, tile_size_x, &info.tile_size_x);
242 if (WARN_ON(err))
243 return err;
244
245 err = PVR_FEATURE_VALUE(pvr_dev, tile_size_y, &info.tile_size_y);
246 if (WARN_ON(err))
247 return err;
248
249 info.num_tiles_x = DIV_ROUND_UP(args->width, info.tile_size_x);
250 info.num_tiles_y = DIV_ROUND_UP(args->height, info.tile_size_y);
251
252 if (PVR_HAS_FEATURE(pvr_dev, simple_parameter_format_version)) {
253 u32 parameter_format;
254
255 err = PVR_FEATURE_VALUE(pvr_dev, simple_parameter_format_version,
256 &parameter_format);
257 if (WARN_ON(err))
258 return err;
259
260 WARN_ON(parameter_format != 2);
261
262 /*
263 * Set up 16 macrotiles with a multiple of 2x2 tiles per macrotile, which is
264 * aligned to a tile group.
265 */
266 info.mtile_x[0] = DIV_ROUND_UP(info.num_tiles_x, 8) * 2;
267 info.mtile_y[0] = DIV_ROUND_UP(info.num_tiles_y, 8) * 2;
268 info.mtile_x[1] = 0;
269 info.mtile_y[1] = 0;
270 info.mtile_x[2] = 0;
271 info.mtile_y[2] = 0;
272 info.tile_max_x = round_up(info.num_tiles_x, 2) - 1;
273 info.tile_max_y = round_up(info.num_tiles_y, 2) - 1;
274 } else {
275 /* Set up 16 macrotiles with a multiple of 4x4 tiles per macrotile. */
276 info.mtile_x[0] = round_up(DIV_ROUND_UP(info.num_tiles_x, 4), 4);
277 info.mtile_y[0] = round_up(DIV_ROUND_UP(info.num_tiles_y, 4), 4);
278 info.mtile_x[1] = info.mtile_x[0] * 2;
279 info.mtile_y[1] = info.mtile_y[0] * 2;
280 info.mtile_x[2] = info.mtile_x[0] * 3;
281 info.mtile_y[2] = info.mtile_y[0] * 3;
282 info.tile_max_x = info.num_tiles_x - 1;
283 info.tile_max_y = info.num_tiles_y - 1;
284 }
285
286 hwrt->common.geom_caches_need_zeroing = false;
287
288 hwrt->common.isp_merge_lower_x = args->isp_merge_lower_x;
289 hwrt->common.isp_merge_lower_y = args->isp_merge_lower_y;
290 hwrt->common.isp_merge_upper_x = args->isp_merge_upper_x;
291 hwrt->common.isp_merge_upper_y = args->isp_merge_upper_y;
292 hwrt->common.isp_merge_scale_x = args->isp_merge_scale_x;
293 hwrt->common.isp_merge_scale_y = args->isp_merge_scale_y;
294
295 err = get_cr_multisamplectl_val(args->samples, false,
296 &hwrt->common.multi_sample_ctl);
297 if (err)
298 return err;
299
300 err = get_cr_multisamplectl_val(args->samples, true,
301 &hwrt->common.flipped_multi_sample_ctl);
302 if (err)
303 return err;
304
305 hwrt->common.mtile_stride = info.mtile_x[0] * info.mtile_y[0];
306
307 err = get_cr_te_aa_val(pvr_dev, args->samples, &hwrt->common.teaa);
308 if (err)
309 return err;
310
311 hwrt->common.screen_pixel_max =
312 (((args->width - 1) << ROGUE_CR_PPP_SCREEN_PIXXMAX_SHIFT) &
313 ~ROGUE_CR_PPP_SCREEN_PIXXMAX_CLRMSK) |
314 (((args->height - 1) << ROGUE_CR_PPP_SCREEN_PIXYMAX_SHIFT) &
315 ~ROGUE_CR_PPP_SCREEN_PIXYMAX_CLRMSK);
316
317 hwrt->common.te_screen =
318 ((info.tile_max_x << ROGUE_CR_TE_SCREEN_XMAX_SHIFT) &
319 ~ROGUE_CR_TE_SCREEN_XMAX_CLRMSK) |
320 ((info.tile_max_y << ROGUE_CR_TE_SCREEN_YMAX_SHIFT) &
321 ~ROGUE_CR_TE_SCREEN_YMAX_CLRMSK);
322 hwrt->common.te_mtile1 =
323 ((info.mtile_x[0] << ROGUE_CR_TE_MTILE1_X1_SHIFT) & ~ROGUE_CR_TE_MTILE1_X1_CLRMSK) |
324 ((info.mtile_x[1] << ROGUE_CR_TE_MTILE1_X2_SHIFT) & ~ROGUE_CR_TE_MTILE1_X2_CLRMSK) |
325 ((info.mtile_x[2] << ROGUE_CR_TE_MTILE1_X3_SHIFT) & ~ROGUE_CR_TE_MTILE1_X3_CLRMSK);
326 hwrt->common.te_mtile2 =
327 ((info.mtile_y[0] << ROGUE_CR_TE_MTILE2_Y1_SHIFT) & ~ROGUE_CR_TE_MTILE2_Y1_CLRMSK) |
328 ((info.mtile_y[1] << ROGUE_CR_TE_MTILE2_Y2_SHIFT) & ~ROGUE_CR_TE_MTILE2_Y2_CLRMSK) |
329 ((info.mtile_y[2] << ROGUE_CR_TE_MTILE2_Y3_SHIFT) & ~ROGUE_CR_TE_MTILE2_Y3_CLRMSK);
330
331 err = get_cr_isp_mtile_size_val(pvr_dev, args->samples, &info,
332 &hwrt->common.isp_mtile_size);
333 if (err)
334 return err;
335
336 hwrt->common.tpc_stride = geom_data_args->tpc_stride;
337 hwrt->common.tpc_size = geom_data_args->tpc_size;
338
339 hwrt->common.rgn_header_size = args->region_header_size;
340
341 err = pvr_fw_object_create(pvr_dev, sizeof(struct rogue_fwif_hwrtdata_common),
342 PVR_BO_FW_FLAGS_DEVICE_UNCACHED, hwrtdata_common_init, hwrt,
343 &hwrt->common_fw_obj);
344
345 return err;
346}
347
348static void
349hwrt_fw_data_init(void *cpu_ptr, void *priv)
350{
351 struct pvr_hwrt_data *hwrt_data = priv;
352
353 memcpy(cpu_ptr, &hwrt_data->data, sizeof(hwrt_data->data));
354}
355
356static int
357hwrt_data_init_fw_structure(struct pvr_file *pvr_file,
358 struct pvr_hwrt_dataset *hwrt,
359 struct drm_pvr_ioctl_create_hwrt_dataset_args *args,
360 struct drm_pvr_create_hwrt_rt_data_args *rt_data_args,
361 struct pvr_hwrt_data *hwrt_data)
362{
363 struct drm_pvr_create_hwrt_geom_data_args *geom_data_args = &args->geom_data_args;
364 struct pvr_device *pvr_dev = pvr_file->pvr_dev;
365 struct rogue_fwif_rta_ctl *rta_ctl;
366 int free_list_i;
367 int err;
368
369 pvr_fw_object_get_fw_addr(hwrt->common_fw_obj,
370 &hwrt_data->data.hwrt_data_common_fw_addr);
371
372 for (free_list_i = 0; free_list_i < ARRAY_SIZE(hwrt->free_lists); free_list_i++) {
373 pvr_fw_object_get_fw_addr(hwrt->free_lists[free_list_i]->fw_obj,
374 &hwrt_data->data.freelists_fw_addr[free_list_i]);
375 }
376
377 hwrt_data->data.tail_ptrs_dev_addr = geom_data_args->tpc_dev_addr;
378 hwrt_data->data.vheap_table_dev_addr = geom_data_args->vheap_table_dev_addr;
379 hwrt_data->data.rtc_dev_addr = geom_data_args->rtc_dev_addr;
380
381 hwrt_data->data.pm_mlist_dev_addr = rt_data_args->pm_mlist_dev_addr;
382 hwrt_data->data.macrotile_array_dev_addr = rt_data_args->macrotile_array_dev_addr;
383 hwrt_data->data.rgn_header_dev_addr = rt_data_args->region_header_dev_addr;
384
385 rta_ctl = &hwrt_data->data.rta_ctl;
386
387 rta_ctl->render_target_index = 0;
388 rta_ctl->active_render_targets = 0;
389 rta_ctl->valid_render_targets_fw_addr = 0;
390 rta_ctl->rta_num_partial_renders_fw_addr = 0;
391 rta_ctl->max_rts = args->layers;
392
393 if (args->layers > 1) {
394 err = pvr_fw_object_create(pvr_dev, args->layers * SRTC_ENTRY_SIZE,
395 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
396 NULL, NULL, &hwrt_data->srtc_obj);
397 if (err)
398 return err;
399 pvr_fw_object_get_fw_addr(hwrt_data->srtc_obj,
400 &rta_ctl->valid_render_targets_fw_addr);
401
402 err = pvr_fw_object_create(pvr_dev, args->layers * RAA_ENTRY_SIZE,
403 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
404 NULL, NULL, &hwrt_data->raa_obj);
405 if (err)
406 goto err_put_shadow_rt_cache;
407 pvr_fw_object_get_fw_addr(hwrt_data->raa_obj,
408 &rta_ctl->rta_num_partial_renders_fw_addr);
409 }
410
411 err = pvr_fw_object_create(pvr_dev, sizeof(struct rogue_fwif_hwrtdata),
412 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
413 hwrt_fw_data_init, hwrt_data, &hwrt_data->fw_obj);
414 if (err)
415 goto err_put_raa_obj;
416
417 pvr_free_list_add_hwrt(hwrt->free_lists[0], hwrt_data);
418
419 return 0;
420
421err_put_raa_obj:
422 if (args->layers > 1)
423 pvr_fw_object_destroy(hwrt_data->raa_obj);
424
425err_put_shadow_rt_cache:
426 if (args->layers > 1)
427 pvr_fw_object_destroy(hwrt_data->srtc_obj);
428
429 return err;
430}
431
432static void
433hwrt_data_fini_fw_structure(struct pvr_hwrt_dataset *hwrt, int hwrt_nr)
434{
435 struct pvr_hwrt_data *hwrt_data = &hwrt->data[hwrt_nr];
436
437 pvr_free_list_remove_hwrt(hwrt->free_lists[0], hwrt_data);
438
439 if (hwrt->max_rts > 1) {
440 pvr_fw_object_destroy(hwrt_data->raa_obj);
441 pvr_fw_object_destroy(hwrt_data->srtc_obj);
442 }
443
444 pvr_fw_object_destroy(hwrt_data->fw_obj);
445}
446
447/**
448 * pvr_hwrt_dataset_create() - Create a new HWRT dataset
449 * @pvr_file: Pointer to pvr_file structure.
450 * @args: Creation arguments from userspace.
451 *
452 * Return:
453 * * Pointer to new HWRT, or
454 * * ERR_PTR(-%ENOMEM) on out of memory.
455 */
456struct pvr_hwrt_dataset *
457pvr_hwrt_dataset_create(struct pvr_file *pvr_file,
458 struct drm_pvr_ioctl_create_hwrt_dataset_args *args)
459{
460 struct pvr_hwrt_dataset *hwrt;
461 int err;
462
463 /* Create and fill out the kernel structure */
464 hwrt = kzalloc(sizeof(*hwrt), GFP_KERNEL);
465
466 if (!hwrt)
467 return ERR_PTR(-ENOMEM);
468
469 kref_init(&hwrt->ref_count);
470
471 err = hwrt_init_kernel_structure(pvr_file, args, hwrt);
472 if (err < 0)
473 goto err_free;
474
475 err = hwrt_init_common_fw_structure(pvr_file, args, hwrt);
476 if (err < 0)
477 goto err_free;
478
479 for (int i = 0; i < ARRAY_SIZE(hwrt->data); i++) {
480 err = hwrt_data_init_fw_structure(pvr_file, hwrt, args,
481 &args->rt_data_args[i],
482 &hwrt->data[i]);
483 if (err < 0) {
484 i--;
485 /* Destroy already created structures. */
486 for (; i >= 0; i--)
487 hwrt_data_fini_fw_structure(hwrt, i);
488 goto err_free;
489 }
490
491 hwrt->data[i].hwrt_dataset = hwrt;
492 }
493
494 return hwrt;
495
496err_free:
497 pvr_hwrt_dataset_put(hwrt);
498
499 return ERR_PTR(err);
500}
501
502static void
503pvr_hwrt_dataset_release(struct kref *ref_count)
504{
505 struct pvr_hwrt_dataset *hwrt =
506 container_of(ref_count, struct pvr_hwrt_dataset, ref_count);
507
508 for (int i = ARRAY_SIZE(hwrt->data) - 1; i >= 0; i--) {
509 WARN_ON(pvr_fw_structure_cleanup(hwrt->pvr_dev, ROGUE_FWIF_CLEANUP_HWRTDATA,
510 hwrt->data[i].fw_obj, 0));
511 hwrt_data_fini_fw_structure(hwrt, i);
512 }
513
514 hwrt_fini_common_fw_structure(hwrt);
515 hwrt_fini_kernel_structure(hwrt);
516
517 kfree(hwrt);
518}
519
520/**
521 * pvr_destroy_hwrt_datasets_for_file: Destroy any HWRT datasets associated
522 * with the given file.
523 * @pvr_file: Pointer to pvr_file structure.
524 *
525 * Removes all HWRT datasets associated with @pvr_file from the device
526 * hwrt_dataset list and drops initial references. HWRT datasets will then be
527 * destroyed once all outstanding references are dropped.
528 */
529void pvr_destroy_hwrt_datasets_for_file(struct pvr_file *pvr_file)
530{
531 struct pvr_hwrt_dataset *hwrt;
532 unsigned long handle;
533
534 xa_for_each(&pvr_file->hwrt_handles, handle, hwrt) {
535 (void)hwrt;
536 pvr_hwrt_dataset_put(xa_erase(&pvr_file->hwrt_handles, handle));
537 }
538}
539
540/**
541 * pvr_hwrt_dataset_put() - Release reference on HWRT dataset
542 * @hwrt: Pointer to HWRT dataset to release reference on
543 */
544void
545pvr_hwrt_dataset_put(struct pvr_hwrt_dataset *hwrt)
546{
547 if (hwrt)
548 kref_put(&hwrt->ref_count, pvr_hwrt_dataset_release);
549}
drivers/gpu/drm/imagination/pvr_hwrt.h+165−0
@@ -0,0 +1,165 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_HWRT_H
5#define PVR_HWRT_H
6
7#include <linux/compiler_attributes.h>
8#include <linux/kref.h>
9#include <linux/list.h>
10#include <linux/types.h>
11#include <linux/xarray.h>
12#include <uapi/drm/pvr_drm.h>
13
14#include "pvr_device.h"
15#include "pvr_rogue_fwif_shared.h"
16
17/* Forward declaration from pvr_free_list.h. */
18struct pvr_free_list;
19
20/* Forward declaration from pvr_gem.h. */
21struct pvr_fw_object;
22
23/**
24 * struct pvr_hwrt_data - structure representing HWRT data
25 */
26struct pvr_hwrt_data {
27 /** @fw_obj: FW object representing the FW-side structure. */
28 struct pvr_fw_object *fw_obj;
29
30 /** @data: Local copy of FW-side structure. */
31 struct rogue_fwif_hwrtdata data;
32
33 /** @freelist_node: List node connecting this HWRT to the local freelist. */
34 struct list_head freelist_node;
35
36 /**
37 * @srtc_obj: FW object representing shadow render target cache.
38 *
39 * Only valid if @max_rts > 1.
40 */
41 struct pvr_fw_object *srtc_obj;
42
43 /**
44 * @raa_obj: FW object representing renders accumulation array.
45 *
46 * Only valid if @max_rts > 1.
47 */
48 struct pvr_fw_object *raa_obj;
49
50 /** @hwrt_dataset: Back pointer to owning HWRT dataset. */
51 struct pvr_hwrt_dataset *hwrt_dataset;
52};
53
54/**
55 * struct pvr_hwrt_dataset - structure representing a HWRT data set.
56 */
57struct pvr_hwrt_dataset {
58 /** @ref_count: Reference count of object. */
59 struct kref ref_count;
60
61 /** @pvr_dev: Pointer to device that owns this object. */
62 struct pvr_device *pvr_dev;
63
64 /** @common_fw_obj: FW object representing common FW-side structure. */
65 struct pvr_fw_object *common_fw_obj;
66
67 struct rogue_fwif_hwrtdata_common common;
68
69 /** @data: HWRT data structures belonging to this set. */
70 struct pvr_hwrt_data data[ROGUE_FWIF_NUM_RTDATAS];
71
72 /** @free_lists: Free lists used by HWRT data set. */
73 struct pvr_free_list *free_lists[ROGUE_FWIF_NUM_RTDATA_FREELISTS];
74
75 /** @max_rts: Maximum render targets for this HWRT data set. */
76 u16 max_rts;
77};
78
79struct pvr_hwrt_dataset *
80pvr_hwrt_dataset_create(struct pvr_file *pvr_file,
81 struct drm_pvr_ioctl_create_hwrt_dataset_args *args);
82
83void
84pvr_destroy_hwrt_datasets_for_file(struct pvr_file *pvr_file);
85
86/**
87 * pvr_hwrt_dataset_lookup() - Lookup HWRT dataset pointer from handle
88 * @pvr_file: Pointer to pvr_file structure.
89 * @handle: Object handle.
90 *
91 * Takes reference on dataset object. Call pvr_hwrt_dataset_put() to release.
92 *
93 * Returns:
94 * * The requested object on success, or
95 * * %NULL on failure (object does not exist in list, or is not a HWRT
96 * dataset)
97 */
98static __always_inline struct pvr_hwrt_dataset *
99pvr_hwrt_dataset_lookup(struct pvr_file *pvr_file, u32 handle)
100{
101 struct pvr_hwrt_dataset *hwrt;
102
103 xa_lock(&pvr_file->hwrt_handles);
104 hwrt = xa_load(&pvr_file->hwrt_handles, handle);
105
106 if (hwrt)
107 kref_get(&hwrt->ref_count);
108
109 xa_unlock(&pvr_file->hwrt_handles);
110
111 return hwrt;
112}
113
114void
115pvr_hwrt_dataset_put(struct pvr_hwrt_dataset *hwrt);
116
117/**
118 * pvr_hwrt_data_lookup() - Lookup HWRT data pointer from handle and index
119 * @pvr_file: Pointer to pvr_file structure.
120 * @handle: Object handle.
121 * @index: Index of RT data within dataset.
122 *
123 * Takes reference on dataset object. Call pvr_hwrt_data_put() to release.
124 *
125 * Returns:
126 * * The requested object on success, or
127 * * %NULL on failure (object does not exist in list, or is not a HWRT
128 * dataset, or index is out of range)
129 */
130static __always_inline struct pvr_hwrt_data *
131pvr_hwrt_data_lookup(struct pvr_file *pvr_file, u32 handle, u32 index)
132{
133 struct pvr_hwrt_dataset *hwrt_dataset = pvr_hwrt_dataset_lookup(pvr_file, handle);
134
135 if (hwrt_dataset) {
136 if (index < ARRAY_SIZE(hwrt_dataset->data))
137 return &hwrt_dataset->data[index];
138
139 pvr_hwrt_dataset_put(hwrt_dataset);
140 }
141
142 return NULL;
143}
144
145/**
146 * pvr_hwrt_data_put() - Release reference on HWRT data
147 * @hwrt: Pointer to HWRT data to release reference on
148 */
149static __always_inline void
150pvr_hwrt_data_put(struct pvr_hwrt_data *hwrt)
151{
152 if (hwrt)
153 pvr_hwrt_dataset_put(hwrt->hwrt_dataset);
154}
155
156static __always_inline struct pvr_hwrt_data *
157pvr_hwrt_data_get(struct pvr_hwrt_data *hwrt)
158{
159 if (hwrt)
160 kref_get(&hwrt->hwrt_dataset->ref_count);
161
162 return hwrt;
163}
164
165#endif /* PVR_HWRT_H */
drivers/gpu/drm/imagination/pvr_job.c+788−0
@@ -0,0 +1,788 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_context.h"
5#include "pvr_device.h"
6#include "pvr_drv.h"
7#include "pvr_gem.h"
8#include "pvr_hwrt.h"
9#include "pvr_job.h"
10#include "pvr_mmu.h"
11#include "pvr_power.h"
12#include "pvr_rogue_fwif.h"
13#include "pvr_rogue_fwif_client.h"
14#include "pvr_stream.h"
15#include "pvr_stream_defs.h"
16#include "pvr_sync.h"
17
18#include <drm/drm_exec.h>
19#include <drm/drm_gem.h>
20#include <linux/types.h>
21#include <uapi/drm/pvr_drm.h>
22
23static void pvr_job_release(struct kref *kref)
24{
25 struct pvr_job *job = container_of(kref, struct pvr_job, ref_count);
26
27 xa_erase(&job->pvr_dev->job_ids, job->id);
28
29 pvr_hwrt_data_put(job->hwrt);
30 pvr_context_put(job->ctx);
31
32 WARN_ON(job->paired_job);
33
34 pvr_queue_job_cleanup(job);
35 pvr_job_release_pm_ref(job);
36
37 kfree(job->cmd);
38 kfree(job);
39}
40
41/**
42 * pvr_job_put() - Release reference on job
43 * @job: Target job.
44 */
45void
46pvr_job_put(struct pvr_job *job)
47{
48 if (job)
49 kref_put(&job->ref_count, pvr_job_release);
50}
51
52/**
53 * pvr_job_process_stream() - Build job FW structure from stream
54 * @pvr_dev: Device pointer.
55 * @cmd_defs: Stream definition.
56 * @stream: Pointer to command stream.
57 * @stream_size: Size of command stream, in bytes.
58 * @job: Pointer to job.
59 *
60 * Caller is responsible for freeing the output structure.
61 *
62 * Returns:
63 * * 0 on success,
64 * * -%ENOMEM on out of memory, or
65 * * -%EINVAL on malformed stream.
66 */
67static int
68pvr_job_process_stream(struct pvr_device *pvr_dev, const struct pvr_stream_cmd_defs *cmd_defs,
69 void *stream, u32 stream_size, struct pvr_job *job)
70{
71 int err;
72
73 job->cmd = kzalloc(cmd_defs->dest_size, GFP_KERNEL);
74 if (!job->cmd)
75 return -ENOMEM;
76
77 job->cmd_len = cmd_defs->dest_size;
78
79 err = pvr_stream_process(pvr_dev, cmd_defs, stream, stream_size, job->cmd);
80 if (err)
81 kfree(job->cmd);
82
83 return err;
84}
85
86static int pvr_fw_cmd_init(struct pvr_device *pvr_dev, struct pvr_job *job,
87 const struct pvr_stream_cmd_defs *stream_def,
88 u64 stream_userptr, u32 stream_len)
89{
90 void *stream;
91 int err;
92
93 stream = kzalloc(stream_len, GFP_KERNEL);
94 if (!stream)
95 return -ENOMEM;
96
97 if (copy_from_user(stream, u64_to_user_ptr(stream_userptr), stream_len)) {
98 err = -EFAULT;
99 goto err_free_stream;
100 }
101
102 err = pvr_job_process_stream(pvr_dev, stream_def, stream, stream_len, job);
103
104err_free_stream:
105 kfree(stream);
106
107 return err;
108}
109
110static u32
111convert_geom_flags(u32 in_flags)
112{
113 u32 out_flags = 0;
114
115 if (in_flags & DRM_PVR_SUBMIT_JOB_GEOM_CMD_FIRST)
116 out_flags |= ROGUE_GEOM_FLAGS_FIRSTKICK;
117 if (in_flags & DRM_PVR_SUBMIT_JOB_GEOM_CMD_LAST)
118 out_flags |= ROGUE_GEOM_FLAGS_LASTKICK;
119 if (in_flags & DRM_PVR_SUBMIT_JOB_GEOM_CMD_SINGLE_CORE)
120 out_flags |= ROGUE_GEOM_FLAGS_SINGLE_CORE;
121
122 return out_flags;
123}
124
125static u32
126convert_frag_flags(u32 in_flags)
127{
128 u32 out_flags = 0;
129
130 if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_SINGLE_CORE)
131 out_flags |= ROGUE_FRAG_FLAGS_SINGLE_CORE;
132 if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_DEPTHBUFFER)
133 out_flags |= ROGUE_FRAG_FLAGS_DEPTHBUFFER;
134 if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_STENCILBUFFER)
135 out_flags |= ROGUE_FRAG_FLAGS_STENCILBUFFER;
136 if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_PREVENT_CDM_OVERLAP)
137 out_flags |= ROGUE_FRAG_FLAGS_PREVENT_CDM_OVERLAP;
138 if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_SCRATCHBUFFER)
139 out_flags |= ROGUE_FRAG_FLAGS_SCRATCHBUFFER;
140 if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_GET_VIS_RESULTS)
141 out_flags |= ROGUE_FRAG_FLAGS_GET_VIS_RESULTS;
142 if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_DISABLE_PIXELMERGE)
143 out_flags |= ROGUE_FRAG_FLAGS_DISABLE_PIXELMERGE;
144
145 return out_flags;
146}
147
148static int
149pvr_geom_job_fw_cmd_init(struct pvr_job *job,
150 struct drm_pvr_job *args)
151{
152 struct rogue_fwif_cmd_geom *cmd;
153 int err;
154
155 if (args->flags & ~DRM_PVR_SUBMIT_JOB_GEOM_CMD_FLAGS_MASK)
156 return -EINVAL;
157
158 if (job->ctx->type != DRM_PVR_CTX_TYPE_RENDER)
159 return -EINVAL;
160
161 if (!job->hwrt)
162 return -EINVAL;
163
164 job->fw_ccb_cmd_type = ROGUE_FWIF_CCB_CMD_TYPE_GEOM;
165 err = pvr_fw_cmd_init(job->pvr_dev, job, &pvr_cmd_geom_stream,
166 args->cmd_stream, args->cmd_stream_len);
167 if (err)
168 return err;
169
170 cmd = job->cmd;
171 cmd->cmd_shared.cmn.frame_num = 0;
172 cmd->flags = convert_geom_flags(args->flags);
173 pvr_fw_object_get_fw_addr(job->hwrt->fw_obj, &cmd->cmd_shared.hwrt_data_fw_addr);
174 return 0;
175}
176
177static int
178pvr_frag_job_fw_cmd_init(struct pvr_job *job,
179 struct drm_pvr_job *args)
180{
181 struct rogue_fwif_cmd_frag *cmd;
182 int err;
183
184 if (args->flags & ~DRM_PVR_SUBMIT_JOB_FRAG_CMD_FLAGS_MASK)
185 return -EINVAL;
186
187 if (job->ctx->type != DRM_PVR_CTX_TYPE_RENDER)
188 return -EINVAL;
189
190 if (!job->hwrt)
191 return -EINVAL;
192
193 job->fw_ccb_cmd_type = (args->flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_PARTIAL_RENDER) ?
194 ROGUE_FWIF_CCB_CMD_TYPE_FRAG_PR :
195 ROGUE_FWIF_CCB_CMD_TYPE_FRAG;
196 err = pvr_fw_cmd_init(job->pvr_dev, job, &pvr_cmd_frag_stream,
197 args->cmd_stream, args->cmd_stream_len);
198 if (err)
199 return err;
200
201 cmd = job->cmd;
202 cmd->cmd_shared.cmn.frame_num = 0;
203 cmd->flags = convert_frag_flags(args->flags);
204 pvr_fw_object_get_fw_addr(job->hwrt->fw_obj, &cmd->cmd_shared.hwrt_data_fw_addr);
205 return 0;
206}
207
208static u32
209convert_compute_flags(u32 in_flags)
210{
211 u32 out_flags = 0;
212
213 if (in_flags & DRM_PVR_SUBMIT_JOB_COMPUTE_CMD_PREVENT_ALL_OVERLAP)
214 out_flags |= ROGUE_COMPUTE_FLAG_PREVENT_ALL_OVERLAP;
215 if (in_flags & DRM_PVR_SUBMIT_JOB_COMPUTE_CMD_SINGLE_CORE)
216 out_flags |= ROGUE_COMPUTE_FLAG_SINGLE_CORE;
217
218 return out_flags;
219}
220
221static int
222pvr_compute_job_fw_cmd_init(struct pvr_job *job,
223 struct drm_pvr_job *args)
224{
225 struct rogue_fwif_cmd_compute *cmd;
226 int err;
227
228 if (args->flags & ~DRM_PVR_SUBMIT_JOB_COMPUTE_CMD_FLAGS_MASK)
229 return -EINVAL;
230
231 if (job->ctx->type != DRM_PVR_CTX_TYPE_COMPUTE)
232 return -EINVAL;
233
234 job->fw_ccb_cmd_type = ROGUE_FWIF_CCB_CMD_TYPE_CDM;
235 err = pvr_fw_cmd_init(job->pvr_dev, job, &pvr_cmd_compute_stream,
236 args->cmd_stream, args->cmd_stream_len);
237 if (err)
238 return err;
239
240 cmd = job->cmd;
241 cmd->common.frame_num = 0;
242 cmd->flags = convert_compute_flags(args->flags);
243 return 0;
244}
245
246static u32
247convert_transfer_flags(u32 in_flags)
248{
249 u32 out_flags = 0;
250
251 if (in_flags & DRM_PVR_SUBMIT_JOB_TRANSFER_CMD_SINGLE_CORE)
252 out_flags |= ROGUE_TRANSFER_FLAGS_SINGLE_CORE;
253
254 return out_flags;
255}
256
257static int
258pvr_transfer_job_fw_cmd_init(struct pvr_job *job,
259 struct drm_pvr_job *args)
260{
261 struct rogue_fwif_cmd_transfer *cmd;
262 int err;
263
264 if (args->flags & ~DRM_PVR_SUBMIT_JOB_TRANSFER_CMD_FLAGS_MASK)
265 return -EINVAL;
266
267 if (job->ctx->type != DRM_PVR_CTX_TYPE_TRANSFER_FRAG)
268 return -EINVAL;
269
270 job->fw_ccb_cmd_type = ROGUE_FWIF_CCB_CMD_TYPE_TQ_3D;
271 err = pvr_fw_cmd_init(job->pvr_dev, job, &pvr_cmd_transfer_stream,
272 args->cmd_stream, args->cmd_stream_len);
273 if (err)
274 return err;
275
276 cmd = job->cmd;
277 cmd->common.frame_num = 0;
278 cmd->flags = convert_transfer_flags(args->flags);
279 return 0;
280}
281
282static int
283pvr_job_fw_cmd_init(struct pvr_job *job,
284 struct drm_pvr_job *args)
285{
286 switch (args->type) {
287 case DRM_PVR_JOB_TYPE_GEOMETRY:
288 return pvr_geom_job_fw_cmd_init(job, args);
289
290 case DRM_PVR_JOB_TYPE_FRAGMENT:
291 return pvr_frag_job_fw_cmd_init(job, args);
292
293 case DRM_PVR_JOB_TYPE_COMPUTE:
294 return pvr_compute_job_fw_cmd_init(job, args);
295
296 case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
297 return pvr_transfer_job_fw_cmd_init(job, args);
298
299 default:
300 return -EINVAL;
301 }
302}
303
304/**
305 * struct pvr_job_data - Helper container for pairing jobs with the
306 * sync_ops supplied for them by the user.
307 */
308struct pvr_job_data {
309 /** @job: Pointer to the job. */
310 struct pvr_job *job;
311
312 /** @sync_ops: Pointer to the sync_ops associated with @job. */
313 struct drm_pvr_sync_op *sync_ops;
314
315 /** @sync_op_count: Number of members of @sync_ops. */
316 u32 sync_op_count;
317};
318
319/**
320 * prepare_job_syncs() - Prepare all sync objects for a single job.
321 * @pvr_file: PowerVR file.
322 * @job_data: Precreated job and sync_ops array.
323 * @signal_array: xarray to receive signal sync objects.
324 *
325 * Returns:
326 * * 0 on success, or
327 * * Any error code returned by pvr_sync_signal_array_collect_ops(),
328 * pvr_sync_add_deps_to_job(), drm_sched_job_add_resv_dependencies() or
329 * pvr_sync_signal_array_update_fences().
330 */
331static int
332prepare_job_syncs(struct pvr_file *pvr_file,
333 struct pvr_job_data *job_data,
334 struct xarray *signal_array)
335{
336 struct dma_fence *done_fence;
337 int err = pvr_sync_signal_array_collect_ops(signal_array,
338 from_pvr_file(pvr_file),
339 job_data->sync_op_count,
340 job_data->sync_ops);
341
342 if (err)
343 return err;
344
345 err = pvr_sync_add_deps_to_job(pvr_file, &job_data->job->base,
346 job_data->sync_op_count,
347 job_data->sync_ops, signal_array);
348 if (err)
349 return err;
350
351 if (job_data->job->hwrt) {
352 /* The geometry job writes the HWRT region headers, which are
353 * then read by the fragment job.
354 */
355 struct drm_gem_object *obj =
356 gem_from_pvr_gem(job_data->job->hwrt->fw_obj->gem);
357 enum dma_resv_usage usage =
358 dma_resv_usage_rw(job_data->job->type ==
359 DRM_PVR_JOB_TYPE_GEOMETRY);
360
361 dma_resv_lock(obj->resv, NULL);
362 err = drm_sched_job_add_resv_dependencies(&job_data->job->base,
363 obj->resv, usage);
364 dma_resv_unlock(obj->resv);
365 if (err)
366 return err;
367 }
368
369 /* We need to arm the job to get the job done fence. */
370 done_fence = pvr_queue_job_arm(job_data->job);
371
372 err = pvr_sync_signal_array_update_fences(signal_array,
373 job_data->sync_op_count,
374 job_data->sync_ops,
375 done_fence);
376 return err;
377}
378
379/**
380 * prepare_job_syncs_for_each() - Prepare all sync objects for an array of jobs.
381 * @file: PowerVR file.
382 * @job_data: Array of precreated jobs and their sync_ops.
383 * @job_count: Number of jobs.
384 * @signal_array: xarray to receive signal sync objects.
385 *
386 * Returns:
387 * * 0 on success, or
388 * * Any error code returned by pvr_vm_bind_job_prepare_syncs().
389 */
390static int
391prepare_job_syncs_for_each(struct pvr_file *pvr_file,
392 struct pvr_job_data *job_data,
393 u32 *job_count,
394 struct xarray *signal_array)
395{
396 for (u32 i = 0; i < *job_count; i++) {
397 int err = prepare_job_syncs(pvr_file, &job_data[i],
398 signal_array);
399
400 if (err) {
401 *job_count = i;
402 return err;
403 }
404 }
405
406 return 0;
407}
408
409static struct pvr_job *
410create_job(struct pvr_device *pvr_dev,
411 struct pvr_file *pvr_file,
412 struct drm_pvr_job *args)
413{
414 struct pvr_job *job = NULL;
415 int err;
416
417 if (!args->cmd_stream || !args->cmd_stream_len)
418 return ERR_PTR(-EINVAL);
419
420 if (args->type != DRM_PVR_JOB_TYPE_GEOMETRY &&
421 args->type != DRM_PVR_JOB_TYPE_FRAGMENT &&
422 (args->hwrt.set_handle || args->hwrt.data_index))
423 return ERR_PTR(-EINVAL);
424
425 job = kzalloc(sizeof(*job), GFP_KERNEL);
426 if (!job)
427 return ERR_PTR(-ENOMEM);
428
429 kref_init(&job->ref_count);
430 job->type = args->type;
431 job->pvr_dev = pvr_dev;
432
433 err = xa_alloc(&pvr_dev->job_ids, &job->id, job, xa_limit_32b, GFP_KERNEL);
434 if (err)
435 goto err_put_job;
436
437 job->ctx = pvr_context_lookup(pvr_file, args->context_handle);
438 if (!job->ctx) {
439 err = -EINVAL;
440 goto err_put_job;
441 }
442
443 if (args->hwrt.set_handle) {
444 job->hwrt = pvr_hwrt_data_lookup(pvr_file, args->hwrt.set_handle,
445 args->hwrt.data_index);
446 if (!job->hwrt) {
447 err = -EINVAL;
448 goto err_put_job;
449 }
450 }
451
452 err = pvr_job_fw_cmd_init(job, args);
453 if (err)
454 goto err_put_job;
455
456 err = pvr_queue_job_init(job);
457 if (err)
458 goto err_put_job;
459
460 return job;
461
462err_put_job:
463 pvr_job_put(job);
464 return ERR_PTR(err);
465}
466
467/**
468 * pvr_job_data_fini() - Cleanup all allocs used to set up job submission.
469 * @job_data: Job data array.
470 * @job_count: Number of members of @job_data.
471 */
472static void
473pvr_job_data_fini(struct pvr_job_data *job_data, u32 job_count)
474{
475 for (u32 i = 0; i < job_count; i++) {
476 pvr_job_put(job_data[i].job);
477 kvfree(job_data[i].sync_ops);
478 }
479}
480
481/**
482 * pvr_job_data_init() - Init an array of created jobs, associating them with
483 * the appropriate sync_ops args, which will be copied in.
484 * @pvr_dev: Target PowerVR device.
485 * @pvr_file: Pointer to PowerVR file structure.
486 * @job_args: Job args array copied from user.
487 * @job_count: Number of members of @job_args.
488 * @job_data_out: Job data array.
489 */
490static int pvr_job_data_init(struct pvr_device *pvr_dev,
491 struct pvr_file *pvr_file,
492 struct drm_pvr_job *job_args,
493 u32 *job_count,
494 struct pvr_job_data *job_data_out)
495{
496 int err = 0, i = 0;
497
498 for (; i < *job_count; i++) {
499 job_data_out[i].job =
500 create_job(pvr_dev, pvr_file, &job_args[i]);
501 err = PTR_ERR_OR_ZERO(job_data_out[i].job);
502
503 if (err) {
504 *job_count = i;
505 job_data_out[i].job = NULL;
506 goto err_cleanup;
507 }
508
509 err = PVR_UOBJ_GET_ARRAY(job_data_out[i].sync_ops,
510 &job_args[i].sync_ops);
511 if (err) {
512 *job_count = i;
513
514 /* Ensure the job created above is also cleaned up. */
515 i++;
516 goto err_cleanup;
517 }
518
519 job_data_out[i].sync_op_count = job_args[i].sync_ops.count;
520 }
521
522 return 0;
523
524err_cleanup:
525 pvr_job_data_fini(job_data_out, i);
526
527 return err;
528}
529
530static void
531push_jobs(struct pvr_job_data *job_data, u32 job_count)
532{
533 for (u32 i = 0; i < job_count; i++)
534 pvr_queue_job_push(job_data[i].job);
535}
536
537static int
538prepare_fw_obj_resv(struct drm_exec *exec, struct pvr_fw_object *fw_obj)
539{
540 return drm_exec_prepare_obj(exec, gem_from_pvr_gem(fw_obj->gem), 1);
541}
542
543static int
544jobs_lock_all_objs(struct drm_exec *exec, struct pvr_job_data *job_data,
545 u32 job_count)
546{
547 for (u32 i = 0; i < job_count; i++) {
548 struct pvr_job *job = job_data[i].job;
549
550 /* Grab a lock on a the context, to guard against
551 * concurrent submission to the same queue.
552 */
553 int err = drm_exec_lock_obj(exec,
554 gem_from_pvr_gem(job->ctx->fw_obj->gem));
555
556 if (err)
557 return err;
558
559 if (job->hwrt) {
560 err = prepare_fw_obj_resv(exec,
561 job->hwrt->fw_obj);
562 if (err)
563 return err;
564 }
565 }
566
567 return 0;
568}
569
570static int
571prepare_job_resvs_for_each(struct drm_exec *exec, struct pvr_job_data *job_data,
572 u32 job_count)
573{
574 drm_exec_until_all_locked(exec) {
575 int err = jobs_lock_all_objs(exec, job_data, job_count);
576
577 drm_exec_retry_on_contention(exec);
578 if (err)
579 return err;
580 }
581
582 return 0;
583}
584
585static void
586update_job_resvs(struct pvr_job *job)
587{
588 if (job->hwrt) {
589 enum dma_resv_usage usage = job->type == DRM_PVR_JOB_TYPE_GEOMETRY ?
590 DMA_RESV_USAGE_WRITE : DMA_RESV_USAGE_READ;
591 struct drm_gem_object *obj = gem_from_pvr_gem(job->hwrt->fw_obj->gem);
592
593 dma_resv_add_fence(obj->resv, &job->base.s_fence->finished, usage);
594 }
595}
596
597static void
598update_job_resvs_for_each(struct pvr_job_data *job_data, u32 job_count)
599{
600 for (u32 i = 0; i < job_count; i++)
601 update_job_resvs(job_data[i].job);
602}
603
604static bool can_combine_jobs(struct pvr_job *a, struct pvr_job *b)
605{
606 struct pvr_job *geom_job = a, *frag_job = b;
607 struct dma_fence *fence;
608 unsigned long index;
609
610 /* Geometry and fragment jobs can be combined if they are queued to the
611 * same context and targeting the same HWRT.
612 */
613 if (a->type != DRM_PVR_JOB_TYPE_GEOMETRY ||
614 b->type != DRM_PVR_JOB_TYPE_FRAGMENT ||
615 a->ctx != b->ctx ||
616 a->hwrt != b->hwrt)
617 return false;
618
619 xa_for_each(&frag_job->base.dependencies, index, fence) {
620 /* We combine when we see an explicit geom -> frag dep. */
621 if (&geom_job->base.s_fence->scheduled == fence)
622 return true;
623 }
624
625 return false;
626}
627
628static struct dma_fence *
629get_last_queued_job_scheduled_fence(struct pvr_queue *queue,
630 struct pvr_job_data *job_data,
631 u32 cur_job_pos)
632{
633 /* We iterate over the current job array in reverse order to grab the
634 * last to-be-queued job targeting the same queue.
635 */
636 for (u32 i = cur_job_pos; i > 0; i--) {
637 struct pvr_job *job = job_data[i - 1].job;
638
639 if (job->ctx == queue->ctx && job->type == queue->type)
640 return dma_fence_get(&job->base.s_fence->scheduled);
641 }
642
643 /* If we didn't find any, we just return the last queued job scheduled
644 * fence attached to the queue.
645 */
646 return dma_fence_get(queue->last_queued_job_scheduled_fence);
647}
648
649static int
650pvr_jobs_link_geom_frag(struct pvr_job_data *job_data, u32 *job_count)
651{
652 for (u32 i = 0; i < *job_count - 1; i++) {
653 struct pvr_job *geom_job = job_data[i].job;
654 struct pvr_job *frag_job = job_data[i + 1].job;
655 struct pvr_queue *frag_queue;
656 struct dma_fence *f;
657
658 if (!can_combine_jobs(job_data[i].job, job_data[i + 1].job))
659 continue;
660
661 /* The fragment job will be submitted by the geometry queue. We
662 * need to make sure it comes after all the other fragment jobs
663 * queued before it.
664 */
665 frag_queue = pvr_context_get_queue_for_job(frag_job->ctx,
666 frag_job->type);
667 f = get_last_queued_job_scheduled_fence(frag_queue, job_data,
668 i);
669 if (f) {
670 int err = drm_sched_job_add_dependency(&geom_job->base,
671 f);
672 if (err) {
673 *job_count = i;
674 return err;
675 }
676 }
677
678 /* The KCCB slot will be reserved by the geometry job, so we can
679 * drop the KCCB fence on the fragment job.
680 */
681 pvr_kccb_fence_put(frag_job->kccb_fence);
682 frag_job->kccb_fence = NULL;
683
684 geom_job->paired_job = frag_job;
685 frag_job->paired_job = geom_job;
686
687 /* Skip the fragment job we just paired to the geometry job. */
688 i++;
689 }
690
691 return 0;
692}
693
694/**
695 * pvr_submit_jobs() - Submit jobs to the GPU
696 * @pvr_dev: Target PowerVR device.
697 * @pvr_file: Pointer to PowerVR file structure.
698 * @args: Ioctl args.
699 * @job_count: Number of jobs in @jobs_args. On error this will be updated
700 * with the index into @jobs_args where the error occurred.
701 *
702 * This initial implementation is entirely synchronous; on return the GPU will
703 * be idle. This will not be the case for future implementations.
704 *
705 * Returns:
706 * * 0 on success,
707 * * -%EFAULT if arguments can not be copied from user space, or
708 * * -%EINVAL on invalid arguments, or
709 * * Any other error.
710 */
711int
712pvr_submit_jobs(struct pvr_device *pvr_dev, struct pvr_file *pvr_file,
713 struct drm_pvr_ioctl_submit_jobs_args *args)
714{
715 struct pvr_job_data *job_data = NULL;
716 struct drm_pvr_job *job_args;
717 struct xarray signal_array;
718 u32 jobs_alloced = 0;
719 struct drm_exec exec;
720 int err;
721
722 if (!args->jobs.count)
723 return -EINVAL;
724
725 err = PVR_UOBJ_GET_ARRAY(job_args, &args->jobs);
726 if (err)
727 return err;
728
729 job_data = kvmalloc_array(args->jobs.count, sizeof(*job_data),
730 GFP_KERNEL | __GFP_ZERO);
731 if (!job_data) {
732 err = -ENOMEM;
733 goto out_free;
734 }
735
736 err = pvr_job_data_init(pvr_dev, pvr_file, job_args, &args->jobs.count,
737 job_data);
738 if (err)
739 goto out_free;
740
741 jobs_alloced = args->jobs.count;
742
743 /*
744 * Flush MMU if needed - this has been deferred until now to avoid
745 * overuse of this expensive operation.
746 */
747 err = pvr_mmu_flush_exec(pvr_dev, false);
748 if (err)
749 goto out_job_data_cleanup;
750
751 drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT | DRM_EXEC_IGNORE_DUPLICATES);
752
753 xa_init_flags(&signal_array, XA_FLAGS_ALLOC);
754
755 err = prepare_job_syncs_for_each(pvr_file, job_data, &args->jobs.count,
756 &signal_array);
757 if (err)
758 goto out_exec_fini;
759
760 err = prepare_job_resvs_for_each(&exec, job_data, args->jobs.count);
761 if (err)
762 goto out_exec_fini;
763
764 err = pvr_jobs_link_geom_frag(job_data, &args->jobs.count);
765 if (err)
766 goto out_exec_fini;
767
768 /* Anything after that point must succeed because we start exposing job
769 * finished fences to the outside world.
770 */
771 update_job_resvs_for_each(job_data, args->jobs.count);
772 push_jobs(job_data, args->jobs.count);
773 pvr_sync_signal_array_push_fences(&signal_array);
774 err = 0;
775
776out_exec_fini:
777 drm_exec_fini(&exec);
778 pvr_sync_signal_array_cleanup(&signal_array);
779
780out_job_data_cleanup:
781 pvr_job_data_fini(job_data, jobs_alloced);
782
783out_free:
784 kvfree(job_data);
785 kvfree(job_args);
786
787 return err;
788}
drivers/gpu/drm/imagination/pvr_job.h+161−0
@@ -0,0 +1,161 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_JOB_H
5#define PVR_JOB_H
6
7#include <uapi/drm/pvr_drm.h>
8
9#include <linux/kref.h>
10#include <linux/types.h>
11
12#include <drm/drm_gem.h>
13#include <drm/gpu_scheduler.h>
14
15#include "pvr_power.h"
16
17/* Forward declaration from "pvr_context.h". */
18struct pvr_context;
19
20/* Forward declarations from "pvr_device.h". */
21struct pvr_device;
22struct pvr_file;
23
24/* Forward declarations from "pvr_hwrt.h". */
25struct pvr_hwrt_data;
26
27/* Forward declaration from "pvr_queue.h". */
28struct pvr_queue;
29
30struct pvr_job {
31 /** @base: drm_sched_job object. */
32 struct drm_sched_job base;
33
34 /** @ref_count: Refcount for job. */
35 struct kref ref_count;
36
37 /** @type: Type of job. */
38 enum drm_pvr_job_type type;
39
40 /** @id: Job ID number. */
41 u32 id;
42
43 /**
44 * @paired_job: Job paired to this job.
45 *
46 * This field is only meaningful for geometry and fragment jobs.
47 *
48 * Paired jobs are executed on the same context, and need to be submitted
49 * atomically to the FW, to make sure the partial render logic has a
50 * fragment job to execute when the Parameter Manager runs out of memory.
51 *
52 * The geometry job should point to the fragment job it's paired with,
53 * and the fragment job should point to the geometry job it's paired with.
54 */
55 struct pvr_job *paired_job;
56
57 /** @cccb_fence: Fence used to wait for CCCB space. */
58 struct dma_fence *cccb_fence;
59
60 /** @kccb_fence: Fence used to wait for KCCB space. */
61 struct dma_fence *kccb_fence;
62
63 /** @done_fence: Fence to signal when the job is done. */
64 struct dma_fence *done_fence;
65
66 /** @pvr_dev: Device pointer. */
67 struct pvr_device *pvr_dev;
68
69 /** @ctx: Pointer to owning context. */
70 struct pvr_context *ctx;
71
72 /** @cmd: Command data. Format depends on @type. */
73 void *cmd;
74
75 /** @cmd_len: Length of command data, in bytes. */
76 u32 cmd_len;
77
78 /**
79 * @fw_ccb_cmd_type: Firmware CCB command type. Must be one of %ROGUE_FWIF_CCB_CMD_TYPE_*.
80 */
81 u32 fw_ccb_cmd_type;
82
83 /** @hwrt: HWRT object. Will be NULL for compute and transfer jobs. */
84 struct pvr_hwrt_data *hwrt;
85
86 /**
87 * @has_pm_ref: True if the job has a power ref, thus forcing the GPU to stay on until
88 * the job is done.
89 */
90 bool has_pm_ref;
91};
92
93/**
94 * pvr_job_get() - Take additional reference on job.
95 * @job: Job pointer.
96 *
97 * Call pvr_job_put() to release.
98 *
99 * Returns:
100 * * The requested job on success, or
101 * * %NULL if no job pointer passed.
102 */
103static __always_inline struct pvr_job *
104pvr_job_get(struct pvr_job *job)
105{
106 if (job)
107 kref_get(&job->ref_count);
108
109 return job;
110}
111
112void pvr_job_put(struct pvr_job *job);
113
114/**
115 * pvr_job_release_pm_ref() - Release the PM ref if the job acquired it.
116 * @job: The job to release the PM ref on.
117 */
118static __always_inline void
119pvr_job_release_pm_ref(struct pvr_job *job)
120{
121 if (job->has_pm_ref) {
122 pvr_power_put(job->pvr_dev);
123 job->has_pm_ref = false;
124 }
125}
126
127/**
128 * pvr_job_get_pm_ref() - Get a PM ref and attach it to the job.
129 * @job: The job to attach the PM ref to.
130 *
131 * Return:
132 * * 0 on success, or
133 * * Any error returned by pvr_power_get() otherwise.
134 */
135static __always_inline int
136pvr_job_get_pm_ref(struct pvr_job *job)
137{
138 int err;
139
140 if (job->has_pm_ref)
141 return 0;
142
143 err = pvr_power_get(job->pvr_dev);
144 if (!err)
145 job->has_pm_ref = true;
146
147 return err;
148}
149
150int pvr_job_wait_first_non_signaled_native_dep(struct pvr_job *job);
151
152bool pvr_job_non_native_deps_done(struct pvr_job *job);
153
154int pvr_job_fits_in_cccb(struct pvr_job *job, unsigned long native_dep_count);
155
156void pvr_job_submit(struct pvr_job *job);
157
158int pvr_submit_jobs(struct pvr_device *pvr_dev, struct pvr_file *pvr_file,
159 struct drm_pvr_ioctl_submit_jobs_args *args);
160
161#endif /* PVR_JOB_H */
drivers/gpu/drm/imagination/pvr_mmu.c+2637−0
@@ -0,0 +1,2637 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_mmu.h"
5
6#include "pvr_ccb.h"
7#include "pvr_device.h"
8#include "pvr_fw.h"
9#include "pvr_gem.h"
10#include "pvr_power.h"
11#include "pvr_rogue_fwif.h"
12#include "pvr_rogue_mmu_defs.h"
13
14#include <drm/drm_drv.h>
15#include <linux/atomic.h>
16#include <linux/bitops.h>
17#include <linux/dma-mapping.h>
18#include <linux/kmemleak.h>
19#include <linux/minmax.h>
20#include <linux/sizes.h>
21
22#define PVR_SHIFT_FROM_SIZE(size_) (__builtin_ctzll(size_))
23#define PVR_MASK_FROM_SIZE(size_) (~((size_) - U64_C(1)))
24
25/*
26 * The value of the device page size (%PVR_DEVICE_PAGE_SIZE) is currently
27 * pegged to the host page size (%PAGE_SIZE). This chunk of macro goodness both
28 * ensures that the selected host page size corresponds to a valid device page
29 * size and sets up values needed by the MMU code below.
30 */
31#if (PVR_DEVICE_PAGE_SIZE == SZ_4K)
32# define ROGUE_MMUCTRL_PAGE_SIZE_X ROGUE_MMUCTRL_PAGE_SIZE_4KB
33# define ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT ROGUE_MMUCTRL_PAGE_4KB_RANGE_SHIFT
34# define ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK ROGUE_MMUCTRL_PAGE_4KB_RANGE_CLRMSK
35#elif (PVR_DEVICE_PAGE_SIZE == SZ_16K)
36# define ROGUE_MMUCTRL_PAGE_SIZE_X ROGUE_MMUCTRL_PAGE_SIZE_16KB
37# define ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT ROGUE_MMUCTRL_PAGE_16KB_RANGE_SHIFT
38# define ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK ROGUE_MMUCTRL_PAGE_16KB_RANGE_CLRMSK
39#elif (PVR_DEVICE_PAGE_SIZE == SZ_64K)
40# define ROGUE_MMUCTRL_PAGE_SIZE_X ROGUE_MMUCTRL_PAGE_SIZE_64KB
41# define ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT ROGUE_MMUCTRL_PAGE_64KB_RANGE_SHIFT
42# define ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK ROGUE_MMUCTRL_PAGE_64KB_RANGE_CLRMSK
43#elif (PVR_DEVICE_PAGE_SIZE == SZ_256K)
44# define ROGUE_MMUCTRL_PAGE_SIZE_X ROGUE_MMUCTRL_PAGE_SIZE_256KB
45# define ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT ROGUE_MMUCTRL_PAGE_256KB_RANGE_SHIFT
46# define ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK ROGUE_MMUCTRL_PAGE_256KB_RANGE_CLRMSK
47#elif (PVR_DEVICE_PAGE_SIZE == SZ_1M)
48# define ROGUE_MMUCTRL_PAGE_SIZE_X ROGUE_MMUCTRL_PAGE_SIZE_1MB
49# define ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT ROGUE_MMUCTRL_PAGE_1MB_RANGE_SHIFT
50# define ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK ROGUE_MMUCTRL_PAGE_1MB_RANGE_CLRMSK
51#elif (PVR_DEVICE_PAGE_SIZE == SZ_2M)
52# define ROGUE_MMUCTRL_PAGE_SIZE_X ROGUE_MMUCTRL_PAGE_SIZE_2MB
53# define ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT ROGUE_MMUCTRL_PAGE_2MB_RANGE_SHIFT
54# define ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK ROGUE_MMUCTRL_PAGE_2MB_RANGE_CLRMSK
55#else
56# error Unsupported device page size PVR_DEVICE_PAGE_SIZE
57#endif
58
59#define ROGUE_MMUCTRL_ENTRIES_PT_VALUE_X \
60 (ROGUE_MMUCTRL_ENTRIES_PT_VALUE >> \
61 (PVR_DEVICE_PAGE_SHIFT - PVR_SHIFT_FROM_SIZE(SZ_4K)))
62
63enum pvr_mmu_sync_level {
64 PVR_MMU_SYNC_LEVEL_NONE = -1,
65 PVR_MMU_SYNC_LEVEL_0 = 0,
66 PVR_MMU_SYNC_LEVEL_1 = 1,
67 PVR_MMU_SYNC_LEVEL_2 = 2,
68};
69
70#define PVR_MMU_SYNC_LEVEL_0_FLAGS (ROGUE_FWIF_MMUCACHEDATA_FLAGS_PT | \
71 ROGUE_FWIF_MMUCACHEDATA_FLAGS_INTERRUPT | \
72 ROGUE_FWIF_MMUCACHEDATA_FLAGS_TLB)
73#define PVR_MMU_SYNC_LEVEL_1_FLAGS (PVR_MMU_SYNC_LEVEL_0_FLAGS | ROGUE_FWIF_MMUCACHEDATA_FLAGS_PD)
74#define PVR_MMU_SYNC_LEVEL_2_FLAGS (PVR_MMU_SYNC_LEVEL_1_FLAGS | ROGUE_FWIF_MMUCACHEDATA_FLAGS_PC)
75
76/**
77 * pvr_mmu_set_flush_flags() - Set MMU cache flush flags for next call to
78 * pvr_mmu_flush_exec().
79 * @pvr_dev: Target PowerVR device.
80 * @flags: MMU flush flags. Must be one of %PVR_MMU_SYNC_LEVEL_*_FLAGS.
81 *
82 * This function must be called following any possible change to the MMU page
83 * tables.
84 */
85static void pvr_mmu_set_flush_flags(struct pvr_device *pvr_dev, u32 flags)
86{
87 atomic_fetch_or(flags, &pvr_dev->mmu_flush_cache_flags);
88}
89
90/**
91 * pvr_mmu_flush_request_all() - Request flush of all MMU caches when
92 * subsequently calling pvr_mmu_flush_exec().
93 * @pvr_dev: Target PowerVR device.
94 *
95 * This function must be called following any possible change to the MMU page
96 * tables.
97 */
98void pvr_mmu_flush_request_all(struct pvr_device *pvr_dev)
99{
100 pvr_mmu_set_flush_flags(pvr_dev, PVR_MMU_SYNC_LEVEL_2_FLAGS);
101}
102
103/**
104 * pvr_mmu_flush_exec() - Execute a flush of all MMU caches previously
105 * requested.
106 * @pvr_dev: Target PowerVR device.
107 * @wait: Do not return until the flush is completed.
108 *
109 * This function must be called prior to submitting any new GPU job. The flush
110 * will complete before the jobs are scheduled, so this can be called once after
111 * a series of maps. However, a single unmap should always be immediately
112 * followed by a flush and it should be explicitly waited by setting @wait.
113 *
114 * As a failure to flush the MMU caches could risk memory corruption, if the
115 * flush fails (implying the firmware is not responding) then the GPU device is
116 * marked as lost.
117 *
118 * Returns:
119 * * 0 on success when @wait is true, or
120 * * -%EIO if the device is unavailable, or
121 * * Any error encountered while submitting the flush command via the KCCB.
122 */
123int pvr_mmu_flush_exec(struct pvr_device *pvr_dev, bool wait)
124{
125 struct rogue_fwif_kccb_cmd cmd_mmu_cache = {};
126 struct rogue_fwif_mmucachedata *cmd_mmu_cache_data =
127 &cmd_mmu_cache.cmd_data.mmu_cache_data;
128 int err = 0;
129 u32 slot;
130 int idx;
131
132 if (!drm_dev_enter(from_pvr_device(pvr_dev), &idx))
133 return -EIO;
134
135 /* Can't flush MMU if the firmware hasn't booted yet. */
136 if (!pvr_dev->fw_dev.booted)
137 goto err_drm_dev_exit;
138
139 cmd_mmu_cache_data->cache_flags =
140 atomic_xchg(&pvr_dev->mmu_flush_cache_flags, 0);
141
142 if (!cmd_mmu_cache_data->cache_flags)
143 goto err_drm_dev_exit;
144
145 cmd_mmu_cache.cmd_type = ROGUE_FWIF_KCCB_CMD_MMUCACHE;
146
147 pvr_fw_object_get_fw_addr(pvr_dev->fw_dev.mem.mmucache_sync_obj,
148 &cmd_mmu_cache_data->mmu_cache_sync_fw_addr);
149 cmd_mmu_cache_data->mmu_cache_sync_update_value = 0;
150
151 err = pvr_kccb_send_cmd(pvr_dev, &cmd_mmu_cache, &slot);
152 if (err)
153 goto err_reset_and_retry;
154
155 err = pvr_kccb_wait_for_completion(pvr_dev, slot, HZ, NULL);
156 if (err)
157 goto err_reset_and_retry;
158
159 drm_dev_exit(idx);
160
161 return 0;
162
163err_reset_and_retry:
164 /*
165 * Flush command failure is most likely the result of a firmware lockup. Hard
166 * reset the GPU and retry.
167 */
168 err = pvr_power_reset(pvr_dev, true);
169 if (err)
170 goto err_drm_dev_exit; /* Device is lost. */
171
172 /* Retry sending flush request. */
173 err = pvr_kccb_send_cmd(pvr_dev, &cmd_mmu_cache, &slot);
174 if (err) {
175 pvr_device_lost(pvr_dev);
176 goto err_drm_dev_exit;
177 }
178
179 if (wait) {
180 err = pvr_kccb_wait_for_completion(pvr_dev, slot, HZ, NULL);
181 if (err)
182 pvr_device_lost(pvr_dev);
183 }
184
185err_drm_dev_exit:
186 drm_dev_exit(idx);
187
188 return err;
189}
190
191/**
192 * DOC: PowerVR Virtual Memory Handling
193 */
194/**
195 * DOC: PowerVR Virtual Memory Handling (constants)
196 *
197 * .. c:macro:: PVR_IDX_INVALID
198 *
199 * Default value for a u16-based index.
200 *
201 * This value cannot be zero, since zero is a valid index value.
202 */
203#define PVR_IDX_INVALID ((u16)(-1))
204
205/**
206 * DOC: MMU backing pages
207 */
208/**
209 * DOC: MMU backing pages (constants)
210 *
211 * .. c:macro:: PVR_MMU_BACKING_PAGE_SIZE
212 *
213 * Page size of a PowerVR device's integrated MMU. The CPU page size must be
214 * at least as large as this value for the current implementation; this is
215 * checked at compile-time.
216 */
217#define PVR_MMU_BACKING_PAGE_SIZE SZ_4K
218static_assert(PAGE_SIZE >= PVR_MMU_BACKING_PAGE_SIZE);
219
220/**
221 * struct pvr_mmu_backing_page - Represents a single page used to back a page
222 * table of any level.
223 * @dma_addr: DMA address of this page.
224 * @host_ptr: CPU address of this page.
225 * @pvr_dev: The PowerVR device to which this page is associated. **For
226 * internal use only.**
227 */
228struct pvr_mmu_backing_page {
229 dma_addr_t dma_addr;
230 void *host_ptr;
231/* private: internal use only */
232 struct page *raw_page;
233 struct pvr_device *pvr_dev;
234};
235
236/**
237 * pvr_mmu_backing_page_init() - Initialize a MMU backing page.
238 * @page: Target backing page.
239 * @pvr_dev: Target PowerVR device.
240 *
241 * This function performs three distinct operations:
242 *
243 * 1. Allocate a single page,
244 * 2. Map the page to the CPU, and
245 * 3. Map the page to DMA-space.
246 *
247 * It is expected that @page be zeroed (e.g. from kzalloc()) before calling
248 * this function.
249 *
250 * Return:
251 * * 0 on success, or
252 * * -%ENOMEM if allocation of the backing page or mapping of the backing
253 * page to DMA fails.
254 */
255static int
256pvr_mmu_backing_page_init(struct pvr_mmu_backing_page *page,
257 struct pvr_device *pvr_dev)
258{
259 struct device *dev = from_pvr_device(pvr_dev)->dev;
260
261 struct page *raw_page;
262 int err;
263
264 dma_addr_t dma_addr;
265 void *host_ptr;
266
267 raw_page = alloc_page(__GFP_ZERO | GFP_KERNEL);
268 if (!raw_page)
269 return -ENOMEM;
270
271 host_ptr = vmap(&raw_page, 1, VM_MAP, pgprot_writecombine(PAGE_KERNEL));
272 if (!host_ptr) {
273 err = -ENOMEM;
274 goto err_free_page;
275 }
276
277 dma_addr = dma_map_page(dev, raw_page, 0, PVR_MMU_BACKING_PAGE_SIZE,
278 DMA_TO_DEVICE);
279 if (dma_mapping_error(dev, dma_addr)) {
280 err = -ENOMEM;
281 goto err_unmap_page;
282 }
283
284 page->dma_addr = dma_addr;
285 page->host_ptr = host_ptr;
286 page->pvr_dev = pvr_dev;
287 page->raw_page = raw_page;
288 kmemleak_alloc(page->host_ptr, PAGE_SIZE, 1, GFP_KERNEL);
289
290 return 0;
291
292err_unmap_page:
293 vunmap(host_ptr);
294
295err_free_page:
296 __free_page(raw_page);
297
298 return err;
299}
300
301/**
302 * pvr_mmu_backing_page_fini() - Teardown a MMU backing page.
303 * @page: Target backing page.
304 *
305 * This function performs the mirror operations to pvr_mmu_backing_page_init(),
306 * in reverse order:
307 *
308 * 1. Unmap the page from DMA-space,
309 * 2. Unmap the page from the CPU, and
310 * 3. Free the page.
311 *
312 * It also zeros @page.
313 *
314 * It is a no-op to call this function a second (or further) time on any @page.
315 */
316static void
317pvr_mmu_backing_page_fini(struct pvr_mmu_backing_page *page)
318{
319 struct device *dev = from_pvr_device(page->pvr_dev)->dev;
320
321 /* Do nothing if no allocation is present. */
322 if (!page->pvr_dev)
323 return;
324
325 dma_unmap_page(dev, page->dma_addr, PVR_MMU_BACKING_PAGE_SIZE,
326 DMA_TO_DEVICE);
327
328 kmemleak_free(page->host_ptr);
329 vunmap(page->host_ptr);
330
331 __free_page(page->raw_page);
332
333 memset(page, 0, sizeof(*page));
334}
335
336/**
337 * pvr_mmu_backing_page_sync() - Flush a MMU backing page from the CPU to the
338 * device.
339 * @page: Target backing page.
340 *
341 * .. caution::
342 *
343 * **This is potentially an expensive function call.** Only call
344 * pvr_mmu_backing_page_sync() once you're sure you have no more changes to
345 * make to the backing page in the immediate future.
346 */
347static void
348pvr_mmu_backing_page_sync(struct pvr_mmu_backing_page *page, u32 flags)
349{
350 struct pvr_device *pvr_dev = page->pvr_dev;
351 struct device *dev;
352
353 /*
354 * Do nothing if no allocation is present. This may be the case if
355 * we are unmapping pages.
356 */
357 if (!pvr_dev)
358 return;
359
360 dev = from_pvr_device(pvr_dev)->dev;
361
362 dma_sync_single_for_device(dev, page->dma_addr,
363 PVR_MMU_BACKING_PAGE_SIZE, DMA_TO_DEVICE);
364
365 pvr_mmu_set_flush_flags(pvr_dev, flags);
366}
367
368/**
369 * DOC: Raw page tables
370 */
371
372#define PVR_PAGE_TABLE_TYPEOF_ENTRY(level_) \
373 typeof_member(struct pvr_page_table_l##level_##_entry_raw, val)
374
375#define PVR_PAGE_TABLE_FIELD_GET(level_, name_, field_, entry_) \
376 (((entry_).val & \
377 ~ROGUE_MMUCTRL_##name_##_DATA_##field_##_CLRMSK) >> \
378 ROGUE_MMUCTRL_##name_##_DATA_##field_##_SHIFT)
379
380#define PVR_PAGE_TABLE_FIELD_PREP(level_, name_, field_, val_) \
381 ((((PVR_PAGE_TABLE_TYPEOF_ENTRY(level_))(val_)) \
382 << ROGUE_MMUCTRL_##name_##_DATA_##field_##_SHIFT) & \
383 ~ROGUE_MMUCTRL_##name_##_DATA_##field_##_CLRMSK)
384
385/**
386 * struct pvr_page_table_l2_entry_raw - A single entry in a level 2 page table.
387 * @val: The raw value of this entry.
388 *
389 * This type is a structure for type-checking purposes. At compile-time, its
390 * size is checked against %ROGUE_MMUCTRL_ENTRY_SIZE_PC_VALUE.
391 *
392 * The value stored in this structure can be decoded using the following bitmap:
393 *
394 * .. flat-table::
395 * :widths: 1 5
396 * :stub-columns: 1
397 *
398 * * - 31..4
399 * - **Level 1 Page Table Base Address:** Bits 39..12 of the L1
400 * page table base address, which is 4KiB aligned.
401 *
402 * * - 3..2
403 * - *(reserved)*
404 *
405 * * - 1
406 * - **Pending:** When valid bit is not set, indicates that a valid
407 * entry is pending and the MMU should wait for the driver to map
408 * the entry. This is used to support page demand mapping of
409 * memory.
410 *
411 * * - 0
412 * - **Valid:** Indicates that the entry contains a valid L1 page
413 * table. If the valid bit is not set, then an attempted use of
414 * the page would result in a page fault.
415 */
416struct pvr_page_table_l2_entry_raw {
417 u32 val;
418} __packed;
419static_assert(sizeof(struct pvr_page_table_l2_entry_raw) * 8 ==
420 ROGUE_MMUCTRL_ENTRY_SIZE_PC_VALUE);
421
422static bool
423pvr_page_table_l2_entry_raw_is_valid(struct pvr_page_table_l2_entry_raw entry)
424{
425 return PVR_PAGE_TABLE_FIELD_GET(2, PC, VALID, entry);
426}
427
428/**
429 * pvr_page_table_l2_entry_raw_set() - Write a valid entry into a raw level 2
430 * page table.
431 * @entry: Target raw level 2 page table entry.
432 * @child_table_dma_addr: DMA address of the level 1 page table to be
433 * associated with @entry.
434 *
435 * When calling this function, @child_table_dma_addr must be a valid DMA
436 * address and a multiple of %ROGUE_MMUCTRL_PC_DATA_PD_BASE_ALIGNSIZE.
437 */
438static void
439pvr_page_table_l2_entry_raw_set(struct pvr_page_table_l2_entry_raw *entry,
440 dma_addr_t child_table_dma_addr)
441{
442 child_table_dma_addr >>= ROGUE_MMUCTRL_PC_DATA_PD_BASE_ALIGNSHIFT;
443
444 WRITE_ONCE(entry->val,
445 PVR_PAGE_TABLE_FIELD_PREP(2, PC, VALID, true) |
446 PVR_PAGE_TABLE_FIELD_PREP(2, PC, ENTRY_PENDING, false) |
447 PVR_PAGE_TABLE_FIELD_PREP(2, PC, PD_BASE, child_table_dma_addr));
448}
449
450static void
451pvr_page_table_l2_entry_raw_clear(struct pvr_page_table_l2_entry_raw *entry)
452{
453 WRITE_ONCE(entry->val, 0);
454}
455
456/**
457 * struct pvr_page_table_l1_entry_raw - A single entry in a level 1 page table.
458 * @val: The raw value of this entry.
459 *
460 * This type is a structure for type-checking purposes. At compile-time, its
461 * size is checked against %ROGUE_MMUCTRL_ENTRY_SIZE_PD_VALUE.
462 *
463 * The value stored in this structure can be decoded using the following bitmap:
464 *
465 * .. flat-table::
466 * :widths: 1 5
467 * :stub-columns: 1
468 *
469 * * - 63..41
470 * - *(reserved)*
471 *
472 * * - 40
473 * - **Pending:** When valid bit is not set, indicates that a valid entry
474 * is pending and the MMU should wait for the driver to map the entry.
475 * This is used to support page demand mapping of memory.
476 *
477 * * - 39..5
478 * - **Level 0 Page Table Base Address:** The way this value is
479 * interpreted depends on the page size. Bits not specified in the
480 * table below (e.g. bits 11..5 for page size 4KiB) should be
481 * considered reserved.
482 *
483 * This table shows the bits used in an L1 page table entry to
484 * represent the Physical Table Base Address for a given Page Size.
485 * Since each L1 page table entry covers 2MiB of address space, the
486 * maximum page size is 2MiB.
487 *
488 * .. flat-table::
489 * :widths: 1 1 1 1
490 * :header-rows: 1
491 * :stub-columns: 1
492 *
493 * * - Page size
494 * - L0 page table base address bits
495 * - Number of L0 page table entries
496 * - Size of L0 page table
497 *
498 * * - 4KiB
499 * - 39..12
500 * - 512
501 * - 4KiB
502 *
503 * * - 16KiB
504 * - 39..10
505 * - 128
506 * - 1KiB
507 *
508 * * - 64KiB
509 * - 39..8
510 * - 32
511 * - 256B
512 *
513 * * - 256KiB
514 * - 39..6
515 * - 8
516 * - 64B
517 *
518 * * - 1MiB
519 * - 39..5 (4 = '0')
520 * - 2
521 * - 16B
522 *
523 * * - 2MiB
524 * - 39..5 (4..3 = '00')
525 * - 1
526 * - 8B
527 *
528 * * - 4
529 * - *(reserved)*
530 *
531 * * - 3..1
532 * - **Page Size:** Sets the page size, from 4KiB to 2MiB.
533 *
534 * * - 0
535 * - **Valid:** Indicates that the entry contains a valid L0 page table.
536 * If the valid bit is not set, then an attempted use of the page would
537 * result in a page fault.
538 */
539struct pvr_page_table_l1_entry_raw {
540 u64 val;
541} __packed;
542static_assert(sizeof(struct pvr_page_table_l1_entry_raw) * 8 ==
543 ROGUE_MMUCTRL_ENTRY_SIZE_PD_VALUE);
544
545static bool
546pvr_page_table_l1_entry_raw_is_valid(struct pvr_page_table_l1_entry_raw entry)
547{
548 return PVR_PAGE_TABLE_FIELD_GET(1, PD, VALID, entry);
549}
550
551/**
552 * pvr_page_table_l1_entry_raw_set() - Write a valid entry into a raw level 1
553 * page table.
554 * @entry: Target raw level 1 page table entry.
555 * @child_table_dma_addr: DMA address of the level 0 page table to be
556 * associated with @entry.
557 *
558 * When calling this function, @child_table_dma_addr must be a valid DMA
559 * address and a multiple of 4 KiB.
560 */
561static void
562pvr_page_table_l1_entry_raw_set(struct pvr_page_table_l1_entry_raw *entry,
563 dma_addr_t child_table_dma_addr)
564{
565 WRITE_ONCE(entry->val,
566 PVR_PAGE_TABLE_FIELD_PREP(1, PD, VALID, true) |
567 PVR_PAGE_TABLE_FIELD_PREP(1, PD, ENTRY_PENDING, false) |
568 PVR_PAGE_TABLE_FIELD_PREP(1, PD, PAGE_SIZE, ROGUE_MMUCTRL_PAGE_SIZE_X) |
569 /*
570 * The use of a 4K-specific macro here is correct. It is
571 * a future optimization to allocate sub-host-page-sized
572 * blocks for individual tables, so the condition that any
573 * page table address is aligned to the size of the
574 * largest (a 4KB) table currently holds.
575 */
576 (child_table_dma_addr & ~ROGUE_MMUCTRL_PT_BASE_4KB_RANGE_CLRMSK));
577}
578
579static void
580pvr_page_table_l1_entry_raw_clear(struct pvr_page_table_l1_entry_raw *entry)
581{
582 WRITE_ONCE(entry->val, 0);
583}
584
585/**
586 * struct pvr_page_table_l0_entry_raw - A single entry in a level 0 page table.
587 * @val: The raw value of this entry.
588 *
589 * This type is a structure for type-checking purposes. At compile-time, its
590 * size is checked against %ROGUE_MMUCTRL_ENTRY_SIZE_PT_VALUE.
591 *
592 * The value stored in this structure can be decoded using the following bitmap:
593 *
594 * .. flat-table::
595 * :widths: 1 5
596 * :stub-columns: 1
597 *
598 * * - 63
599 * - *(reserved)*
600 *
601 * * - 62
602 * - **PM/FW Protect:** Indicates a protected region which only the
603 * Parameter Manager (PM) or firmware processor can write to.
604 *
605 * * - 61..40
606 * - **VP Page (High):** Virtual-physical page used for Parameter Manager
607 * (PM) memory. This field is only used if the additional level of PB
608 * virtualization is enabled. The VP Page field is needed by the PM in
609 * order to correctly reconstitute the free lists after render
610 * completion. This (High) field holds bits 39..18 of the value; the
611 * Low field holds bits 17..12. Bits 11..0 are always zero because the
612 * value is always aligned to the 4KiB page size.
613 *
614 * * - 39..12
615 * - **Physical Page Address:** The way this value is interpreted depends
616 * on the page size. Bits not specified in the table below (e.g. bits
617 * 20..12 for page size 2MiB) should be considered reserved.
618 *
619 * This table shows the bits used in an L0 page table entry to represent
620 * the Physical Page Address for a given page size (as defined in the
621 * associated L1 page table entry).
622 *
623 * .. flat-table::
624 * :widths: 1 1
625 * :header-rows: 1
626 * :stub-columns: 1
627 *
628 * * - Page size
629 * - Physical address bits
630 *
631 * * - 4KiB
632 * - 39..12
633 *
634 * * - 16KiB
635 * - 39..14
636 *
637 * * - 64KiB
638 * - 39..16
639 *
640 * * - 256KiB
641 * - 39..18
642 *
643 * * - 1MiB
644 * - 39..20
645 *
646 * * - 2MiB
647 * - 39..21
648 *
649 * * - 11..6
650 * - **VP Page (Low):** Continuation of VP Page (High).
651 *
652 * * - 5
653 * - **Pending:** When valid bit is not set, indicates that a valid entry
654 * is pending and the MMU should wait for the driver to map the entry.
655 * This is used to support page demand mapping of memory.
656 *
657 * * - 4
658 * - **PM Src:** Set on Parameter Manager (PM) allocated page table
659 * entries when indicated by the PM. Note that this bit will only be set
660 * by the PM, not by the device driver.
661 *
662 * * - 3
663 * - **SLC Bypass Control:** Specifies requests to this page should bypass
664 * the System Level Cache (SLC), if enabled in SLC configuration.
665 *
666 * * - 2
667 * - **Cache Coherency:** Indicates that the page is coherent (i.e. it
668 * does not require a cache flush between operations on the CPU and the
669 * device).
670 *
671 * * - 1
672 * - **Read Only:** If set, this bit indicates that the page is read only.
673 * An attempted write to this page would result in a write-protection
674 * fault.
675 *
676 * * - 0
677 * - **Valid:** Indicates that the entry contains a valid page. If the
678 * valid bit is not set, then an attempted use of the page would result
679 * in a page fault.
680 */
681struct pvr_page_table_l0_entry_raw {
682 u64 val;
683} __packed;
684static_assert(sizeof(struct pvr_page_table_l0_entry_raw) * 8 ==
685 ROGUE_MMUCTRL_ENTRY_SIZE_PT_VALUE);
686
687/**
688 * struct pvr_page_flags_raw - The configurable flags from a single entry in a
689 * level 0 page table.
690 * @val: The raw value of these flags. Since these are a strict subset of
691 * &struct pvr_page_table_l0_entry_raw; use that type for our member here.
692 *
693 * The flags stored in this type are: PM/FW Protect; SLC Bypass Control; Cache
694 * Coherency, and Read Only (bits 62, 3, 2 and 1 respectively).
695 *
696 * This type should never be instantiated directly; instead use
697 * pvr_page_flags_raw_create() to ensure only valid bits of @val are set.
698 */
699struct pvr_page_flags_raw {
700 struct pvr_page_table_l0_entry_raw val;
701} __packed;
702static_assert(sizeof(struct pvr_page_flags_raw) ==
703 sizeof(struct pvr_page_table_l0_entry_raw));
704
705static bool
706pvr_page_table_l0_entry_raw_is_valid(struct pvr_page_table_l0_entry_raw entry)
707{
708 return PVR_PAGE_TABLE_FIELD_GET(0, PT, VALID, entry);
709}
710
711/**
712 * pvr_page_table_l0_entry_raw_set() - Write a valid entry into a raw level 0
713 * page table.
714 * @entry: Target raw level 0 page table entry.
715 * @dma_addr: DMA address of the physical page to be associated with @entry.
716 * @flags: Options to be set on @entry.
717 *
718 * When calling this function, @child_table_dma_addr must be a valid DMA
719 * address and a multiple of %PVR_DEVICE_PAGE_SIZE.
720 *
721 * The @flags parameter is directly assigned into @entry. It is the callers
722 * responsibility to ensure that only bits specified in
723 * &struct pvr_page_flags_raw are set in @flags.
724 */
725static void
726pvr_page_table_l0_entry_raw_set(struct pvr_page_table_l0_entry_raw *entry,
727 dma_addr_t dma_addr,
728 struct pvr_page_flags_raw flags)
729{
730 WRITE_ONCE(entry->val, PVR_PAGE_TABLE_FIELD_PREP(0, PT, VALID, true) |
731 PVR_PAGE_TABLE_FIELD_PREP(0, PT, ENTRY_PENDING, false) |
732 (dma_addr & ~ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK) |
733 flags.val.val);
734}
735
736static void
737pvr_page_table_l0_entry_raw_clear(struct pvr_page_table_l0_entry_raw *entry)
738{
739 WRITE_ONCE(entry->val, 0);
740}
741
742/**
743 * pvr_page_flags_raw_create() - Initialize the flag bits of a raw level 0 page
744 * table entry.
745 * @read_only: This page is read-only (see: Read Only).
746 * @cache_coherent: This page does not require cache flushes (see: Cache
747 * Coherency).
748 * @slc_bypass: This page bypasses the device cache (see: SLC Bypass Control).
749 * @pm_fw_protect: This page is only for use by the firmware or Parameter
750 * Manager (see PM/FW Protect).
751 *
752 * For more details on the use of these four options, see their respective
753 * entries in the table under &struct pvr_page_table_l0_entry_raw.
754 *
755 * Return:
756 * A new &struct pvr_page_flags_raw instance which can be passed directly to
757 * pvr_page_table_l0_entry_raw_set() or pvr_page_table_l0_insert().
758 */
759static struct pvr_page_flags_raw
760pvr_page_flags_raw_create(bool read_only, bool cache_coherent, bool slc_bypass,
761 bool pm_fw_protect)
762{
763 struct pvr_page_flags_raw flags;
764
765 flags.val.val =
766 PVR_PAGE_TABLE_FIELD_PREP(0, PT, READ_ONLY, read_only) |
767 PVR_PAGE_TABLE_FIELD_PREP(0, PT, CC, cache_coherent) |
768 PVR_PAGE_TABLE_FIELD_PREP(0, PT, SLC_BYPASS_CTRL, slc_bypass) |
769 PVR_PAGE_TABLE_FIELD_PREP(0, PT, PM_META_PROTECT, pm_fw_protect);
770
771 return flags;
772}
773
774/**
775 * struct pvr_page_table_l2_raw - The raw data of a level 2 page table.
776 *
777 * This type is a structure for type-checking purposes. At compile-time, its
778 * size is checked against %PVR_MMU_BACKING_PAGE_SIZE.
779 */
780struct pvr_page_table_l2_raw {
781 /** @entries: The raw values of this table. */
782 struct pvr_page_table_l2_entry_raw
783 entries[ROGUE_MMUCTRL_ENTRIES_PC_VALUE];
784} __packed;
785static_assert(sizeof(struct pvr_page_table_l2_raw) == PVR_MMU_BACKING_PAGE_SIZE);
786
787/**
788 * struct pvr_page_table_l1_raw - The raw data of a level 1 page table.
789 *
790 * This type is a structure for type-checking purposes. At compile-time, its
791 * size is checked against %PVR_MMU_BACKING_PAGE_SIZE.
792 */
793struct pvr_page_table_l1_raw {
794 /** @entries: The raw values of this table. */
795 struct pvr_page_table_l1_entry_raw
796 entries[ROGUE_MMUCTRL_ENTRIES_PD_VALUE];
797} __packed;
798static_assert(sizeof(struct pvr_page_table_l1_raw) == PVR_MMU_BACKING_PAGE_SIZE);
799
800/**
801 * struct pvr_page_table_l0_raw - The raw data of a level 0 page table.
802 *
803 * This type is a structure for type-checking purposes. At compile-time, its
804 * size is checked against %PVR_MMU_BACKING_PAGE_SIZE.
805 *
806 * .. caution::
807 *
808 * The size of level 0 page tables is variable depending on the page size
809 * specified in the associated level 1 page table entry. Since the device
810 * page size in use is pegged to the host page size, it cannot vary at
811 * runtime. This structure is therefore only defined to contain the required
812 * number of entries for the current device page size. **You should never
813 * read or write beyond the last supported entry.**
814 */
815struct pvr_page_table_l0_raw {
816 /** @entries: The raw values of this table. */
817 struct pvr_page_table_l0_entry_raw
818 entries[ROGUE_MMUCTRL_ENTRIES_PT_VALUE_X];
819} __packed;
820static_assert(sizeof(struct pvr_page_table_l0_raw) <= PVR_MMU_BACKING_PAGE_SIZE);
821
822/**
823 * DOC: Mirror page tables
824 */
825
826/*
827 * We pre-declare these types because they cross-depend on pointers to each
828 * other.
829 */
830struct pvr_page_table_l1;
831struct pvr_page_table_l0;
832
833/**
834 * struct pvr_page_table_l2 - A wrapped level 2 page table.
835 *
836 * To access the raw part of this table, use pvr_page_table_l2_get_raw().
837 * Alternatively to access a raw entry directly, use
838 * pvr_page_table_l2_get_entry_raw().
839 *
840 * A level 2 page table forms the root of the page table tree structure, so
841 * this type has no &parent or &parent_idx members.
842 */
843struct pvr_page_table_l2 {
844 /**
845 * @entries: The children of this node in the page table tree
846 * structure. These are also mirror tables. The indexing of this array
847 * is identical to that of the raw equivalent
848 * (&pvr_page_table_l1_raw.entries).
849 */
850 struct pvr_page_table_l1 *entries[ROGUE_MMUCTRL_ENTRIES_PC_VALUE];
851
852 /**
853 * @backing_page: A handle to the memory which holds the raw
854 * equivalent of this table. **For internal use only.**
855 */
856 struct pvr_mmu_backing_page backing_page;
857
858 /**
859 * @entry_count: The current number of valid entries (that we know of)
860 * in this table. This value is essentially a refcount - the table is
861 * destroyed when this value is decremented to zero by
862 * pvr_page_table_l2_remove().
863 */
864 u16 entry_count;
865};
866
867/**
868 * pvr_page_table_l2_init() - Initialize a level 2 page table.
869 * @table: Target level 2 page table.
870 * @pvr_dev: Target PowerVR device
871 *
872 * It is expected that @table be zeroed (e.g. from kzalloc()) before calling
873 * this function.
874 *
875 * Return:
876 * * 0 on success, or
877 * * Any error encountered while intializing &table->backing_page using
878 * pvr_mmu_backing_page_init().
879 */
880static int
881pvr_page_table_l2_init(struct pvr_page_table_l2 *table,
882 struct pvr_device *pvr_dev)
883{
884 return pvr_mmu_backing_page_init(&table->backing_page, pvr_dev);
885}
886
887/**
888 * pvr_page_table_l2_fini() - Teardown a level 2 page table.
889 * @table: Target level 2 page table.
890 *
891 * It is an error to attempt to use @table after calling this function.
892 */
893static void
894pvr_page_table_l2_fini(struct pvr_page_table_l2 *table)
895{
896 pvr_mmu_backing_page_fini(&table->backing_page);
897}
898
899/**
900 * pvr_page_table_l2_sync() - Flush a level 2 page table from the CPU to the
901 * device.
902 * @table: Target level 2 page table.
903 *
904 * This is just a thin wrapper around pvr_mmu_backing_page_sync(), so the
905 * warning there applies here too: **Only call pvr_page_table_l2_sync() once
906 * you're sure you have no more changes to make to** @table **in the immediate
907 * future.**
908 *
909 * If child level 1 page tables of @table also need to be flushed, this should
910 * be done first using pvr_page_table_l1_sync() *before* calling this function.
911 */
912static void
913pvr_page_table_l2_sync(struct pvr_page_table_l2 *table)
914{
915 pvr_mmu_backing_page_sync(&table->backing_page, PVR_MMU_SYNC_LEVEL_2_FLAGS);
916}
917
918/**
919 * pvr_page_table_l2_get_raw() - Access the raw equivalent of a mirror level 2
920 * page table.
921 * @table: Target level 2 page table.
922 *
923 * Essentially returns the CPU address of the raw equivalent of @table, cast to
924 * a &struct pvr_page_table_l2_raw pointer.
925 *
926 * You probably want to call pvr_page_table_l2_get_entry_raw() instead.
927 *
928 * Return:
929 * The raw equivalent of @table.
930 */
931static struct pvr_page_table_l2_raw *
932pvr_page_table_l2_get_raw(struct pvr_page_table_l2 *table)
933{
934 return table->backing_page.host_ptr;
935}
936
937/**
938 * pvr_page_table_l2_get_entry_raw() - Access an entry from the raw equivalent
939 * of a mirror level 2 page table.
940 * @table: Target level 2 page table.
941 * @idx: Index of the entry to access.
942 *
943 * Technically this function returns a pointer to a slot in a raw level 2 page
944 * table, since the returned "entry" is not guaranteed to be valid. The caller
945 * must verify the validity of the entry at the returned address (perhaps using
946 * pvr_page_table_l2_entry_raw_is_valid()) before reading or overwriting it.
947 *
948 * The value of @idx is not checked here; it is the callers responsibility to
949 * ensure @idx refers to a valid index within @table before dereferencing the
950 * returned pointer.
951 *
952 * Return:
953 * A pointer to the requested raw level 2 page table entry.
954 */
955static struct pvr_page_table_l2_entry_raw *
956pvr_page_table_l2_get_entry_raw(struct pvr_page_table_l2 *table, u16 idx)
957{
958 return &pvr_page_table_l2_get_raw(table)->entries[idx];
959}
960
961/**
962 * pvr_page_table_l2_entry_is_valid() - Check if a level 2 page table entry is
963 * marked as valid.
964 * @table: Target level 2 page table.
965 * @idx: Index of the entry to check.
966 *
967 * The value of @idx is not checked here; it is the callers responsibility to
968 * ensure @idx refers to a valid index within @table before calling this
969 * function.
970 */
971static bool
972pvr_page_table_l2_entry_is_valid(struct pvr_page_table_l2 *table, u16 idx)
973{
974 struct pvr_page_table_l2_entry_raw entry_raw =
975 *pvr_page_table_l2_get_entry_raw(table, idx);
976
977 return pvr_page_table_l2_entry_raw_is_valid(entry_raw);
978}
979
980/**
981 * struct pvr_page_table_l1 - A wrapped level 1 page table.
982 *
983 * To access the raw part of this table, use pvr_page_table_l1_get_raw().
984 * Alternatively to access a raw entry directly, use
985 * pvr_page_table_l1_get_entry_raw().
986 */
987struct pvr_page_table_l1 {
988 /**
989 * @entries: The children of this node in the page table tree
990 * structure. These are also mirror tables. The indexing of this array
991 * is identical to that of the raw equivalent
992 * (&pvr_page_table_l0_raw.entries).
993 */
994 struct pvr_page_table_l0 *entries[ROGUE_MMUCTRL_ENTRIES_PD_VALUE];
995
996 /**
997 * @backing_page: A handle to the memory which holds the raw
998 * equivalent of this table. **For internal use only.**
999 */
1000 struct pvr_mmu_backing_page backing_page;
1001
1002 union {
1003 /**
1004 * @parent: The parent of this node in the page table tree structure.
1005 *
1006 * This is also a mirror table.
1007 *
1008 * Only valid when the L1 page table is active. When the L1 page table
1009 * has been removed and queued for destruction, the next_free field
1010 * should be used instead.
1011 */
1012 struct pvr_page_table_l2 *parent;
1013
1014 /**
1015 * @next_free: Pointer to the next L1 page table to take/free.
1016 *
1017 * Used to form a linked list of L1 page tables. This is used
1018 * when preallocating tables and when the page table has been
1019 * removed and queued for destruction.
1020 */
1021 struct pvr_page_table_l1 *next_free;
1022 };
1023
1024 /**
1025 * @parent_idx: The index of the entry in the parent table (see
1026 * @parent) which corresponds to this table.
1027 */
1028 u16 parent_idx;
1029
1030 /**
1031 * @entry_count: The current number of valid entries (that we know of)
1032 * in this table. This value is essentially a refcount - the table is
1033 * destroyed when this value is decremented to zero by
1034 * pvr_page_table_l1_remove().
1035 */
1036 u16 entry_count;
1037};
1038
1039/**
1040 * pvr_page_table_l1_init() - Initialize a level 1 page table.
1041 * @table: Target level 1 page table.
1042 * @pvr_dev: Target PowerVR device
1043 *
1044 * When this function returns successfully, @table is still not considered
1045 * valid. It must be inserted into the page table tree structure with
1046 * pvr_page_table_l2_insert() before it is ready for use.
1047 *
1048 * It is expected that @table be zeroed (e.g. from kzalloc()) before calling
1049 * this function.
1050 *
1051 * Return:
1052 * * 0 on success, or
1053 * * Any error encountered while intializing &table->backing_page using
1054 * pvr_mmu_backing_page_init().
1055 */
1056static int
1057pvr_page_table_l1_init(struct pvr_page_table_l1 *table,
1058 struct pvr_device *pvr_dev)
1059{
1060 table->parent_idx = PVR_IDX_INVALID;
1061
1062 return pvr_mmu_backing_page_init(&table->backing_page, pvr_dev);
1063}
1064
1065/**
1066 * pvr_page_table_l1_free() - Teardown a level 1 page table.
1067 * @table: Target level 1 page table.
1068 *
1069 * It is an error to attempt to use @table after calling this function, even
1070 * indirectly. This includes calling pvr_page_table_l2_remove(), which must
1071 * be called *before* pvr_page_table_l1_free().
1072 */
1073static void
1074pvr_page_table_l1_free(struct pvr_page_table_l1 *table)
1075{
1076 pvr_mmu_backing_page_fini(&table->backing_page);
1077 kfree(table);
1078}
1079
1080/**
1081 * pvr_page_table_l1_sync() - Flush a level 1 page table from the CPU to the
1082 * device.
1083 * @table: Target level 1 page table.
1084 *
1085 * This is just a thin wrapper around pvr_mmu_backing_page_sync(), so the
1086 * warning there applies here too: **Only call pvr_page_table_l1_sync() once
1087 * you're sure you have no more changes to make to** @table **in the immediate
1088 * future.**
1089 *
1090 * If child level 0 page tables of @table also need to be flushed, this should
1091 * be done first using pvr_page_table_l0_sync() *before* calling this function.
1092 */
1093static void
1094pvr_page_table_l1_sync(struct pvr_page_table_l1 *table)
1095{
1096 pvr_mmu_backing_page_sync(&table->backing_page, PVR_MMU_SYNC_LEVEL_1_FLAGS);
1097}
1098
1099/**
1100 * pvr_page_table_l1_get_raw() - Access the raw equivalent of a mirror level 1
1101 * page table.
1102 * @table: Target level 1 page table.
1103 *
1104 * Essentially returns the CPU address of the raw equivalent of @table, cast to
1105 * a &struct pvr_page_table_l1_raw pointer.
1106 *
1107 * You probably want to call pvr_page_table_l1_get_entry_raw() instead.
1108 *
1109 * Return:
1110 * The raw equivalent of @table.
1111 */
1112static struct pvr_page_table_l1_raw *
1113pvr_page_table_l1_get_raw(struct pvr_page_table_l1 *table)
1114{
1115 return table->backing_page.host_ptr;
1116}
1117
1118/**
1119 * pvr_page_table_l1_get_entry_raw() - Access an entry from the raw equivalent
1120 * of a mirror level 1 page table.
1121 * @table: Target level 1 page table.
1122 * @idx: Index of the entry to access.
1123 *
1124 * Technically this function returns a pointer to a slot in a raw level 1 page
1125 * table, since the returned "entry" is not guaranteed to be valid. The caller
1126 * must verify the validity of the entry at the returned address (perhaps using
1127 * pvr_page_table_l1_entry_raw_is_valid()) before reading or overwriting it.
1128 *
1129 * The value of @idx is not checked here; it is the callers responsibility to
1130 * ensure @idx refers to a valid index within @table before dereferencing the
1131 * returned pointer.
1132 *
1133 * Return:
1134 * A pointer to the requested raw level 1 page table entry.
1135 */
1136static struct pvr_page_table_l1_entry_raw *
1137pvr_page_table_l1_get_entry_raw(struct pvr_page_table_l1 *table, u16 idx)
1138{
1139 return &pvr_page_table_l1_get_raw(table)->entries[idx];
1140}
1141
1142/**
1143 * pvr_page_table_l1_entry_is_valid() - Check if a level 1 page table entry is
1144 * marked as valid.
1145 * @table: Target level 1 page table.
1146 * @idx: Index of the entry to check.
1147 *
1148 * The value of @idx is not checked here; it is the callers responsibility to
1149 * ensure @idx refers to a valid index within @table before calling this
1150 * function.
1151 */
1152static bool
1153pvr_page_table_l1_entry_is_valid(struct pvr_page_table_l1 *table, u16 idx)
1154{
1155 struct pvr_page_table_l1_entry_raw entry_raw =
1156 *pvr_page_table_l1_get_entry_raw(table, idx);
1157
1158 return pvr_page_table_l1_entry_raw_is_valid(entry_raw);
1159}
1160
1161/**
1162 * struct pvr_page_table_l0 - A wrapped level 0 page table.
1163 *
1164 * To access the raw part of this table, use pvr_page_table_l0_get_raw().
1165 * Alternatively to access a raw entry directly, use
1166 * pvr_page_table_l0_get_entry_raw().
1167 *
1168 * There is no mirror representation of an individual page, so this type has no
1169 * &entries member.
1170 */
1171struct pvr_page_table_l0 {
1172 /**
1173 * @backing_page: A handle to the memory which holds the raw
1174 * equivalent of this table. **For internal use only.**
1175 */
1176 struct pvr_mmu_backing_page backing_page;
1177
1178 union {
1179 /**
1180 * @parent: The parent of this node in the page table tree structure.
1181 *
1182 * This is also a mirror table.
1183 *
1184 * Only valid when the L0 page table is active. When the L0 page table
1185 * has been removed and queued for destruction, the next_free field
1186 * should be used instead.
1187 */
1188 struct pvr_page_table_l1 *parent;
1189
1190 /**
1191 * @next_free: Pointer to the next L0 page table to take/free.
1192 *
1193 * Used to form a linked list of L0 page tables. This is used
1194 * when preallocating tables and when the page table has been
1195 * removed and queued for destruction.
1196 */
1197 struct pvr_page_table_l0 *next_free;
1198 };
1199
1200 /**
1201 * @parent_idx: The index of the entry in the parent table (see
1202 * @parent) which corresponds to this table.
1203 */
1204 u16 parent_idx;
1205
1206 /**
1207 * @entry_count: The current number of valid entries (that we know of)
1208 * in this table. This value is essentially a refcount - the table is
1209 * destroyed when this value is decremented to zero by
1210 * pvr_page_table_l0_remove().
1211 */
1212 u16 entry_count;
1213};
1214
1215/**
1216 * pvr_page_table_l0_init() - Initialize a level 0 page table.
1217 * @table: Target level 0 page table.
1218 * @pvr_dev: Target PowerVR device
1219 *
1220 * When this function returns successfully, @table is still not considered
1221 * valid. It must be inserted into the page table tree structure with
1222 * pvr_page_table_l1_insert() before it is ready for use.
1223 *
1224 * It is expected that @table be zeroed (e.g. from kzalloc()) before calling
1225 * this function.
1226 *
1227 * Return:
1228 * * 0 on success, or
1229 * * Any error encountered while intializing &table->backing_page using
1230 * pvr_mmu_backing_page_init().
1231 */
1232static int
1233pvr_page_table_l0_init(struct pvr_page_table_l0 *table,
1234 struct pvr_device *pvr_dev)
1235{
1236 table->parent_idx = PVR_IDX_INVALID;
1237
1238 return pvr_mmu_backing_page_init(&table->backing_page, pvr_dev);
1239}
1240
1241/**
1242 * pvr_page_table_l0_free() - Teardown a level 0 page table.
1243 * @table: Target level 0 page table.
1244 *
1245 * It is an error to attempt to use @table after calling this function, even
1246 * indirectly. This includes calling pvr_page_table_l1_remove(), which must
1247 * be called *before* pvr_page_table_l0_free().
1248 */
1249static void
1250pvr_page_table_l0_free(struct pvr_page_table_l0 *table)
1251{
1252 pvr_mmu_backing_page_fini(&table->backing_page);
1253 kfree(table);
1254}
1255
1256/**
1257 * pvr_page_table_l0_sync() - Flush a level 0 page table from the CPU to the
1258 * device.
1259 * @table: Target level 0 page table.
1260 *
1261 * This is just a thin wrapper around pvr_mmu_backing_page_sync(), so the
1262 * warning there applies here too: **Only call pvr_page_table_l0_sync() once
1263 * you're sure you have no more changes to make to** @table **in the immediate
1264 * future.**
1265 *
1266 * If child pages of @table also need to be flushed, this should be done first
1267 * using a DMA sync function (e.g. dma_sync_sg_for_device()) *before* calling
1268 * this function.
1269 */
1270static void
1271pvr_page_table_l0_sync(struct pvr_page_table_l0 *table)
1272{
1273 pvr_mmu_backing_page_sync(&table->backing_page, PVR_MMU_SYNC_LEVEL_0_FLAGS);
1274}
1275
1276/**
1277 * pvr_page_table_l0_get_raw() - Access the raw equivalent of a mirror level 0
1278 * page table.
1279 * @table: Target level 0 page table.
1280 *
1281 * Essentially returns the CPU address of the raw equivalent of @table, cast to
1282 * a &struct pvr_page_table_l0_raw pointer.
1283 *
1284 * You probably want to call pvr_page_table_l0_get_entry_raw() instead.
1285 *
1286 * Return:
1287 * The raw equivalent of @table.
1288 */
1289static struct pvr_page_table_l0_raw *
1290pvr_page_table_l0_get_raw(struct pvr_page_table_l0 *table)
1291{
1292 return table->backing_page.host_ptr;
1293}
1294
1295/**
1296 * pvr_page_table_l0_get_entry_raw() - Access an entry from the raw equivalent
1297 * of a mirror level 0 page table.
1298 * @table: Target level 0 page table.
1299 * @idx: Index of the entry to access.
1300 *
1301 * Technically this function returns a pointer to a slot in a raw level 0 page
1302 * table, since the returned "entry" is not guaranteed to be valid. The caller
1303 * must verify the validity of the entry at the returned address (perhaps using
1304 * pvr_page_table_l0_entry_raw_is_valid()) before reading or overwriting it.
1305 *
1306 * The value of @idx is not checked here; it is the callers responsibility to
1307 * ensure @idx refers to a valid index within @table before dereferencing the
1308 * returned pointer. This is espcially important for level 0 page tables, which
1309 * can have a variable number of entries.
1310 *
1311 * Return:
1312 * A pointer to the requested raw level 0 page table entry.
1313 */
1314static struct pvr_page_table_l0_entry_raw *
1315pvr_page_table_l0_get_entry_raw(struct pvr_page_table_l0 *table, u16 idx)
1316{
1317 return &pvr_page_table_l0_get_raw(table)->entries[idx];
1318}
1319
1320/**
1321 * pvr_page_table_l0_entry_is_valid() - Check if a level 0 page table entry is
1322 * marked as valid.
1323 * @table: Target level 0 page table.
1324 * @idx: Index of the entry to check.
1325 *
1326 * The value of @idx is not checked here; it is the callers responsibility to
1327 * ensure @idx refers to a valid index within @table before calling this
1328 * function.
1329 */
1330static bool
1331pvr_page_table_l0_entry_is_valid(struct pvr_page_table_l0 *table, u16 idx)
1332{
1333 struct pvr_page_table_l0_entry_raw entry_raw =
1334 *pvr_page_table_l0_get_entry_raw(table, idx);
1335
1336 return pvr_page_table_l0_entry_raw_is_valid(entry_raw);
1337}
1338
1339/**
1340 * struct pvr_mmu_context - context holding data for operations at page
1341 * catalogue level, intended for use with a VM context.
1342 */
1343struct pvr_mmu_context {
1344 /** @pvr_dev: The PVR device associated with the owning VM context. */
1345 struct pvr_device *pvr_dev;
1346
1347 /** @page_table_l2: The MMU table root. */
1348 struct pvr_page_table_l2 page_table_l2;
1349};
1350
1351/**
1352 * struct pvr_page_table_ptr - A reference to a single physical page as indexed
1353 * by the page table structure.
1354 *
1355 * Intended for embedding in a &struct pvr_mmu_op_context.
1356 */
1357struct pvr_page_table_ptr {
1358 /**
1359 * @l1_table: A cached handle to the level 1 page table the
1360 * context is currently traversing.
1361 */
1362 struct pvr_page_table_l1 *l1_table;
1363
1364 /**
1365 * @l0_table: A cached handle to the level 0 page table the
1366 * context is currently traversing.
1367 */
1368 struct pvr_page_table_l0 *l0_table;
1369
1370 /**
1371 * @l2_idx: Index into the level 2 page table the context is
1372 * currently referencing.
1373 */
1374 u16 l2_idx;
1375
1376 /**
1377 * @l1_idx: Index into the level 1 page table the context is
1378 * currently referencing.
1379 */
1380 u16 l1_idx;
1381
1382 /**
1383 * @l0_idx: Index into the level 0 page table the context is
1384 * currently referencing.
1385 */
1386 u16 l0_idx;
1387};
1388
1389/**
1390 * struct pvr_mmu_op_context - context holding data for individual
1391 * device-virtual mapping operations. Intended for use with a VM bind operation.
1392 */
1393struct pvr_mmu_op_context {
1394 /** @mmu_ctx: The MMU context associated with the owning VM context. */
1395 struct pvr_mmu_context *mmu_ctx;
1396
1397 /** @map: Data specifically for map operations. */
1398 struct {
1399 /**
1400 * @sgt: Scatter gather table containing pages pinned for use by
1401 * this context - these are currently pinned when initialising
1402 * the VM bind operation.
1403 */
1404 struct sg_table *sgt;
1405
1406 /** @sgt_offset: Start address of the device-virtual mapping. */
1407 u64 sgt_offset;
1408
1409 /**
1410 * @l1_prealloc_tables: Preallocated l1 page table objects
1411 * use by this context when creating a page mapping. Linked list
1412 * fully created during initialisation.
1413 */
1414 struct pvr_page_table_l1 *l1_prealloc_tables;
1415
1416 /**
1417 * @l0_prealloc_tables: Preallocated l0 page table objects
1418 * use by this context when creating a page mapping. Linked list
1419 * fully created during initialisation.
1420 */
1421 struct pvr_page_table_l0 *l0_prealloc_tables;
1422 } map;
1423
1424 /** @unmap: Data specifically for unmap operations. */
1425 struct {
1426 /**
1427 * @l1_free_tables: Collects page table objects freed by unmap
1428 * ops. Linked list empty at creation.
1429 */
1430 struct pvr_page_table_l1 *l1_free_tables;
1431
1432 /**
1433 * @l0_free_tables: Collects page table objects freed by unmap
1434 * ops. Linked list empty at creation.
1435 */
1436 struct pvr_page_table_l0 *l0_free_tables;
1437 } unmap;
1438
1439 /**
1440 * @curr_page: A reference to a single physical page as indexed by the
1441 * page table structure.
1442 */
1443 struct pvr_page_table_ptr curr_page;
1444
1445 /**
1446 * @sync_level_required: The maximum level of the page table tree
1447 * structure which has (possibly) been modified since it was last
1448 * flushed to the device.
1449 *
1450 * This field should only be set with pvr_mmu_op_context_require_sync()
1451 * or indirectly by pvr_mmu_op_context_sync_partial().
1452 */
1453 enum pvr_mmu_sync_level sync_level_required;
1454};
1455
1456/**
1457 * pvr_page_table_l2_insert() - Insert an entry referring to a level 1 page
1458 * table into a level 2 page table.
1459 * @op_ctx: Target MMU op context pointing at the entry to insert the L1 page
1460 * table into.
1461 * @child_table: Target level 1 page table to be referenced by the new entry.
1462 *
1463 * It is the caller's responsibility to ensure @op_ctx.curr_page points to a
1464 * valid L2 entry.
1465 *
1466 * It is the caller's responsibility to execute any memory barries to ensure
1467 * that the creation of @child_table is ordered before the L2 entry is inserted.
1468 */
1469static void
1470pvr_page_table_l2_insert(struct pvr_mmu_op_context *op_ctx,
1471 struct pvr_page_table_l1 *child_table)
1472{
1473 struct pvr_page_table_l2 *l2_table =
1474 &op_ctx->mmu_ctx->page_table_l2;
1475 struct pvr_page_table_l2_entry_raw *entry_raw =
1476 pvr_page_table_l2_get_entry_raw(l2_table,
1477 op_ctx->curr_page.l2_idx);
1478
1479 pvr_page_table_l2_entry_raw_set(entry_raw,
1480 child_table->backing_page.dma_addr);
1481
1482 child_table->parent = l2_table;
1483 child_table->parent_idx = op_ctx->curr_page.l2_idx;
1484 l2_table->entries[op_ctx->curr_page.l2_idx] = child_table;
1485 ++l2_table->entry_count;
1486 op_ctx->curr_page.l1_table = child_table;
1487}
1488
1489/**
1490 * pvr_page_table_l2_remove() - Remove a level 1 page table from a level 2 page
1491 * table.
1492 * @op_ctx: Target MMU op context pointing at the L2 entry to remove.
1493 *
1494 * It is the caller's responsibility to ensure @op_ctx.curr_page points to a
1495 * valid L2 entry.
1496 */
1497static void
1498pvr_page_table_l2_remove(struct pvr_mmu_op_context *op_ctx)
1499{
1500 struct pvr_page_table_l2 *l2_table =
1501 &op_ctx->mmu_ctx->page_table_l2;
1502 struct pvr_page_table_l2_entry_raw *entry_raw =
1503 pvr_page_table_l2_get_entry_raw(l2_table,
1504 op_ctx->curr_page.l1_table->parent_idx);
1505
1506 WARN_ON(op_ctx->curr_page.l1_table->parent != l2_table);
1507
1508 pvr_page_table_l2_entry_raw_clear(entry_raw);
1509
1510 l2_table->entries[op_ctx->curr_page.l1_table->parent_idx] = NULL;
1511 op_ctx->curr_page.l1_table->parent_idx = PVR_IDX_INVALID;
1512 op_ctx->curr_page.l1_table->next_free = op_ctx->unmap.l1_free_tables;
1513 op_ctx->unmap.l1_free_tables = op_ctx->curr_page.l1_table;
1514 op_ctx->curr_page.l1_table = NULL;
1515
1516 --l2_table->entry_count;
1517}
1518
1519/**
1520 * pvr_page_table_l1_insert() - Insert an entry referring to a level 0 page
1521 * table into a level 1 page table.
1522 * @op_ctx: Target MMU op context pointing at the entry to insert the L0 page
1523 * table into.
1524 * @child_table: L0 page table to insert.
1525 *
1526 * It is the caller's responsibility to ensure @op_ctx.curr_page points to a
1527 * valid L1 entry.
1528 *
1529 * It is the caller's responsibility to execute any memory barries to ensure
1530 * that the creation of @child_table is ordered before the L1 entry is inserted.
1531 */
1532static void
1533pvr_page_table_l1_insert(struct pvr_mmu_op_context *op_ctx,
1534 struct pvr_page_table_l0 *child_table)
1535{
1536 struct pvr_page_table_l1_entry_raw *entry_raw =
1537 pvr_page_table_l1_get_entry_raw(op_ctx->curr_page.l1_table,
1538 op_ctx->curr_page.l1_idx);
1539
1540 pvr_page_table_l1_entry_raw_set(entry_raw,
1541 child_table->backing_page.dma_addr);
1542
1543 child_table->parent = op_ctx->curr_page.l1_table;
1544 child_table->parent_idx = op_ctx->curr_page.l1_idx;
1545 op_ctx->curr_page.l1_table->entries[op_ctx->curr_page.l1_idx] = child_table;
1546 ++op_ctx->curr_page.l1_table->entry_count;
1547 op_ctx->curr_page.l0_table = child_table;
1548}
1549
1550/**
1551 * pvr_page_table_l1_remove() - Remove a level 0 page table from a level 1 page
1552 * table.
1553 * @op_ctx: Target MMU op context pointing at the L1 entry to remove.
1554 *
1555 * If this function results in the L1 table becoming empty, it will be removed
1556 * from its parent level 2 page table and destroyed.
1557 *
1558 * It is the caller's responsibility to ensure @op_ctx.curr_page points to a
1559 * valid L1 entry.
1560 */
1561static void
1562pvr_page_table_l1_remove(struct pvr_mmu_op_context *op_ctx)
1563{
1564 struct pvr_page_table_l1_entry_raw *entry_raw =
1565 pvr_page_table_l1_get_entry_raw(op_ctx->curr_page.l0_table->parent,
1566 op_ctx->curr_page.l0_table->parent_idx);
1567
1568 WARN_ON(op_ctx->curr_page.l0_table->parent !=
1569 op_ctx->curr_page.l1_table);
1570
1571 pvr_page_table_l1_entry_raw_clear(entry_raw);
1572
1573 op_ctx->curr_page.l1_table->entries[op_ctx->curr_page.l0_table->parent_idx] = NULL;
1574 op_ctx->curr_page.l0_table->parent_idx = PVR_IDX_INVALID;
1575 op_ctx->curr_page.l0_table->next_free = op_ctx->unmap.l0_free_tables;
1576 op_ctx->unmap.l0_free_tables = op_ctx->curr_page.l0_table;
1577 op_ctx->curr_page.l0_table = NULL;
1578
1579 if (--op_ctx->curr_page.l1_table->entry_count == 0) {
1580 /* Clear the parent L2 page table entry. */
1581 if (op_ctx->curr_page.l1_table->parent_idx != PVR_IDX_INVALID)
1582 pvr_page_table_l2_remove(op_ctx);
1583 }
1584}
1585
1586/**
1587 * pvr_page_table_l0_insert() - Insert an entry referring to a physical page
1588 * into a level 0 page table.
1589 * @op_ctx: Target MMU op context pointing at the L0 entry to insert.
1590 * @dma_addr: Target DMA address to be referenced by the new entry.
1591 * @flags: Page options to be stored in the new entry.
1592 *
1593 * It is the caller's responsibility to ensure @op_ctx.curr_page points to a
1594 * valid L0 entry.
1595 */
1596static void
1597pvr_page_table_l0_insert(struct pvr_mmu_op_context *op_ctx,
1598 dma_addr_t dma_addr, struct pvr_page_flags_raw flags)
1599{
1600 struct pvr_page_table_l0_entry_raw *entry_raw =
1601 pvr_page_table_l0_get_entry_raw(op_ctx->curr_page.l0_table,
1602 op_ctx->curr_page.l0_idx);
1603
1604 pvr_page_table_l0_entry_raw_set(entry_raw, dma_addr, flags);
1605
1606 /*
1607 * There is no entry to set here - we don't keep a mirror of
1608 * individual pages.
1609 */
1610
1611 ++op_ctx->curr_page.l0_table->entry_count;
1612}
1613
1614/**
1615 * pvr_page_table_l0_remove() - Remove a physical page from a level 0 page
1616 * table.
1617 * @op_ctx: Target MMU op context pointing at the L0 entry to remove.
1618 *
1619 * If this function results in the L0 table becoming empty, it will be removed
1620 * from its parent L1 page table and destroyed.
1621 *
1622 * It is the caller's responsibility to ensure @op_ctx.curr_page points to a
1623 * valid L0 entry.
1624 */
1625static void
1626pvr_page_table_l0_remove(struct pvr_mmu_op_context *op_ctx)
1627{
1628 struct pvr_page_table_l0_entry_raw *entry_raw =
1629 pvr_page_table_l0_get_entry_raw(op_ctx->curr_page.l0_table,
1630 op_ctx->curr_page.l0_idx);
1631
1632 pvr_page_table_l0_entry_raw_clear(entry_raw);
1633
1634 /*
1635 * There is no entry to clear here - we don't keep a mirror of
1636 * individual pages.
1637 */
1638
1639 if (--op_ctx->curr_page.l0_table->entry_count == 0) {
1640 /* Clear the parent L1 page table entry. */
1641 if (op_ctx->curr_page.l0_table->parent_idx != PVR_IDX_INVALID)
1642 pvr_page_table_l1_remove(op_ctx);
1643 }
1644}
1645
1646/**
1647 * DOC: Page table index utilities
1648 */
1649
1650/**
1651 * pvr_page_table_l2_idx() - Calculate the level 2 page table index for a
1652 * device-virtual address.
1653 * @device_addr: Target device-virtual address.
1654 *
1655 * This function does not perform any bounds checking - it is the caller's
1656 * responsibility to ensure that @device_addr is valid before interpreting
1657 * the result.
1658 *
1659 * Return:
1660 * The index into a level 2 page table corresponding to @device_addr.
1661 */
1662static u16
1663pvr_page_table_l2_idx(u64 device_addr)
1664{
1665 return (device_addr & ~ROGUE_MMUCTRL_VADDR_PC_INDEX_CLRMSK) >>
1666 ROGUE_MMUCTRL_VADDR_PC_INDEX_SHIFT;
1667}
1668
1669/**
1670 * pvr_page_table_l1_idx() - Calculate the level 1 page table index for a
1671 * device-virtual address.
1672 * @device_addr: Target device-virtual address.
1673 *
1674 * This function does not perform any bounds checking - it is the caller's
1675 * responsibility to ensure that @device_addr is valid before interpreting
1676 * the result.
1677 *
1678 * Return:
1679 * The index into a level 1 page table corresponding to @device_addr.
1680 */
1681static u16
1682pvr_page_table_l1_idx(u64 device_addr)
1683{
1684 return (device_addr & ~ROGUE_MMUCTRL_VADDR_PD_INDEX_CLRMSK) >>
1685 ROGUE_MMUCTRL_VADDR_PD_INDEX_SHIFT;
1686}
1687
1688/**
1689 * pvr_page_table_l0_idx() - Calculate the level 0 page table index for a
1690 * device-virtual address.
1691 * @device_addr: Target device-virtual address.
1692 *
1693 * This function does not perform any bounds checking - it is the caller's
1694 * responsibility to ensure that @device_addr is valid before interpreting
1695 * the result.
1696 *
1697 * Return:
1698 * The index into a level 0 page table corresponding to @device_addr.
1699 */
1700static u16
1701pvr_page_table_l0_idx(u64 device_addr)
1702{
1703 return (device_addr & ~ROGUE_MMUCTRL_VADDR_PT_INDEX_CLRMSK) >>
1704 ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT;
1705}
1706
1707/**
1708 * DOC: High-level page table operations
1709 */
1710
1711/**
1712 * pvr_page_table_l1_get_or_insert() - Retrieves (optionally inserting if
1713 * necessary) a level 1 page table from the specified level 2 page table entry.
1714 * @op_ctx: Target MMU op context.
1715 * @should_insert: [IN] Specifies whether new page tables should be inserted
1716 * when empty page table entries are encountered during traversal.
1717 *
1718 * Return:
1719 * * 0 on success, or
1720 *
1721 * If @should_insert is %false:
1722 * * -%ENXIO if a level 1 page table would have been inserted.
1723 *
1724 * If @should_insert is %true:
1725 * * Any error encountered while inserting the level 1 page table.
1726 */
1727static int
1728pvr_page_table_l1_get_or_insert(struct pvr_mmu_op_context *op_ctx,
1729 bool should_insert)
1730{
1731 struct pvr_page_table_l2 *l2_table =
1732 &op_ctx->mmu_ctx->page_table_l2;
1733 struct pvr_page_table_l1 *table;
1734
1735 if (pvr_page_table_l2_entry_is_valid(l2_table,
1736 op_ctx->curr_page.l2_idx)) {
1737 op_ctx->curr_page.l1_table =
1738 l2_table->entries[op_ctx->curr_page.l2_idx];
1739 return 0;
1740 }
1741
1742 if (!should_insert)
1743 return -ENXIO;
1744
1745 /* Take a prealloced table. */
1746 table = op_ctx->map.l1_prealloc_tables;
1747 if (!table)
1748 return -ENOMEM;
1749
1750 /* Pop */
1751 op_ctx->map.l1_prealloc_tables = table->next_free;
1752 table->next_free = NULL;
1753
1754 /* Ensure new table is fully written out before adding to L2 page table. */
1755 wmb();
1756
1757 pvr_page_table_l2_insert(op_ctx, table);
1758
1759 return 0;
1760}
1761
1762/**
1763 * pvr_page_table_l0_get_or_insert() - Retrieves (optionally inserting if
1764 * necessary) a level 0 page table from the specified level 1 page table entry.
1765 * @op_ctx: Target MMU op context.
1766 * @should_insert: [IN] Specifies whether new page tables should be inserted
1767 * when empty page table entries are encountered during traversal.
1768 *
1769 * Return:
1770 * * 0 on success,
1771 *
1772 * If @should_insert is %false:
1773 * * -%ENXIO if a level 0 page table would have been inserted.
1774 *
1775 * If @should_insert is %true:
1776 * * Any error encountered while inserting the level 0 page table.
1777 */
1778static int
1779pvr_page_table_l0_get_or_insert(struct pvr_mmu_op_context *op_ctx,
1780 bool should_insert)
1781{
1782 struct pvr_page_table_l0 *table;
1783
1784 if (pvr_page_table_l1_entry_is_valid(op_ctx->curr_page.l1_table,
1785 op_ctx->curr_page.l1_idx)) {
1786 op_ctx->curr_page.l0_table =
1787 op_ctx->curr_page.l1_table->entries[op_ctx->curr_page.l1_idx];
1788 return 0;
1789 }
1790
1791 if (!should_insert)
1792 return -ENXIO;
1793
1794 /* Take a prealloced table. */
1795 table = op_ctx->map.l0_prealloc_tables;
1796 if (!table)
1797 return -ENOMEM;
1798
1799 /* Pop */
1800 op_ctx->map.l0_prealloc_tables = table->next_free;
1801 table->next_free = NULL;
1802
1803 /* Ensure new table is fully written out before adding to L1 page table. */
1804 wmb();
1805
1806 pvr_page_table_l1_insert(op_ctx, table);
1807
1808 return 0;
1809}
1810
1811/**
1812 * pvr_mmu_context_create() - Create an MMU context.
1813 * @pvr_dev: PVR device associated with owning VM context.
1814 *
1815 * Returns:
1816 * * Newly created MMU context object on success, or
1817 * * -%ENOMEM if no memory is available,
1818 * * Any error code returned by pvr_page_table_l2_init().
1819 */
1820struct pvr_mmu_context *pvr_mmu_context_create(struct pvr_device *pvr_dev)
1821{
1822 struct pvr_mmu_context *ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
1823 int err;
1824
1825 if (!ctx)
1826 return ERR_PTR(-ENOMEM);
1827
1828 err = pvr_page_table_l2_init(&ctx->page_table_l2, pvr_dev);
1829 if (err)
1830 return ERR_PTR(err);
1831
1832 ctx->pvr_dev = pvr_dev;
1833
1834 return ctx;
1835}
1836
1837/**
1838 * pvr_mmu_context_destroy() - Destroy an MMU context.
1839 * @ctx: Target MMU context.
1840 */
1841void pvr_mmu_context_destroy(struct pvr_mmu_context *ctx)
1842{
1843 pvr_page_table_l2_fini(&ctx->page_table_l2);
1844 kfree(ctx);
1845}
1846
1847/**
1848 * pvr_mmu_get_root_table_dma_addr() - Get the DMA address of the root of the
1849 * page table structure behind a VM context.
1850 * @ctx: Target MMU context.
1851 */
1852dma_addr_t pvr_mmu_get_root_table_dma_addr(struct pvr_mmu_context *ctx)
1853{
1854 return ctx->page_table_l2.backing_page.dma_addr;
1855}
1856
1857/**
1858 * pvr_page_table_l1_alloc() - Allocate a l1 page_table object.
1859 * @ctx: MMU context of owning VM context.
1860 *
1861 * Returns:
1862 * * Newly created page table object on success, or
1863 * * -%ENOMEM if no memory is available,
1864 * * Any error code returned by pvr_page_table_l1_init().
1865 */
1866static struct pvr_page_table_l1 *
1867pvr_page_table_l1_alloc(struct pvr_mmu_context *ctx)
1868{
1869 int err;
1870
1871 struct pvr_page_table_l1 *table =
1872 kzalloc(sizeof(*table), GFP_KERNEL);
1873
1874 if (!table)
1875 return ERR_PTR(-ENOMEM);
1876
1877 err = pvr_page_table_l1_init(table, ctx->pvr_dev);
1878 if (err) {
1879 kfree(table);
1880 return ERR_PTR(err);
1881 }
1882
1883 return table;
1884}
1885
1886/**
1887 * pvr_page_table_l0_alloc() - Allocate a l0 page_table object.
1888 * @ctx: MMU context of owning VM context.
1889 *
1890 * Returns:
1891 * * Newly created page table object on success, or
1892 * * -%ENOMEM if no memory is available,
1893 * * Any error code returned by pvr_page_table_l0_init().
1894 */
1895static struct pvr_page_table_l0 *
1896pvr_page_table_l0_alloc(struct pvr_mmu_context *ctx)
1897{
1898 int err;
1899
1900 struct pvr_page_table_l0 *table =
1901 kzalloc(sizeof(*table), GFP_KERNEL);
1902
1903 if (!table)
1904 return ERR_PTR(-ENOMEM);
1905
1906 err = pvr_page_table_l0_init(table, ctx->pvr_dev);
1907 if (err) {
1908 kfree(table);
1909 return ERR_PTR(err);
1910 }
1911
1912 return table;
1913}
1914
1915/**
1916 * pvr_mmu_op_context_require_sync() - Mark an MMU op context as requiring a
1917 * sync operation for the referenced page tables up to a specified level.
1918 * @op_ctx: Target MMU op context.
1919 * @level: Maximum page table level for which a sync is required.
1920 */
1921static void
1922pvr_mmu_op_context_require_sync(struct pvr_mmu_op_context *op_ctx,
1923 enum pvr_mmu_sync_level level)
1924{
1925 if (op_ctx->sync_level_required < level)
1926 op_ctx->sync_level_required = level;
1927}
1928
1929/**
1930 * pvr_mmu_op_context_sync_manual() - Trigger a sync of some or all of the
1931 * page tables referenced by a MMU op context.
1932 * @op_ctx: Target MMU op context.
1933 * @level: Maximum page table level to sync.
1934 *
1935 * Do not call this function directly. Instead use
1936 * pvr_mmu_op_context_sync_partial() which is checked against the current
1937 * value of &op_ctx->sync_level_required as set by
1938 * pvr_mmu_op_context_require_sync().
1939 */
1940static void
1941pvr_mmu_op_context_sync_manual(struct pvr_mmu_op_context *op_ctx,
1942 enum pvr_mmu_sync_level level)
1943{
1944 /*
1945 * We sync the page table levels in ascending order (starting from the
1946 * leaf node) to ensure consistency.
1947 */
1948
1949 WARN_ON(level < PVR_MMU_SYNC_LEVEL_NONE);
1950
1951 if (level <= PVR_MMU_SYNC_LEVEL_NONE)
1952 return;
1953
1954 if (op_ctx->curr_page.l0_table)
1955 pvr_page_table_l0_sync(op_ctx->curr_page.l0_table);
1956
1957 if (level < PVR_MMU_SYNC_LEVEL_1)
1958 return;
1959
1960 if (op_ctx->curr_page.l1_table)
1961 pvr_page_table_l1_sync(op_ctx->curr_page.l1_table);
1962
1963 if (level < PVR_MMU_SYNC_LEVEL_2)
1964 return;
1965
1966 pvr_page_table_l2_sync(&op_ctx->mmu_ctx->page_table_l2);
1967}
1968
1969/**
1970 * pvr_mmu_op_context_sync_partial() - Trigger a sync of some or all of the
1971 * page tables referenced by a MMU op context.
1972 * @op_ctx: Target MMU op context.
1973 * @level: Requested page table level to sync up to (inclusive).
1974 *
1975 * If @level is greater than the maximum level recorded by @op_ctx as requiring
1976 * a sync operation, only the previously recorded maximum will be used.
1977 *
1978 * Additionally, if @level is greater than or equal to the maximum level
1979 * recorded by @op_ctx as requiring a sync operation, that maximum level will be
1980 * reset as a full sync will be performed. This is equivalent to calling
1981 * pvr_mmu_op_context_sync().
1982 */
1983static void
1984pvr_mmu_op_context_sync_partial(struct pvr_mmu_op_context *op_ctx,
1985 enum pvr_mmu_sync_level level)
1986{
1987 /*
1988 * If the requested sync level is greater than or equal to the
1989 * currently required sync level, we do two things:
1990 * * Don't waste time syncing levels we haven't previously marked as
1991 * requiring a sync, and
1992 * * Reset the required sync level since we are about to sync
1993 * everything that was previously marked as requiring a sync.
1994 */
1995 if (level >= op_ctx->sync_level_required) {
1996 level = op_ctx->sync_level_required;
1997 op_ctx->sync_level_required = PVR_MMU_SYNC_LEVEL_NONE;
1998 }
1999
2000 pvr_mmu_op_context_sync_manual(op_ctx, level);
2001}
2002
2003/**
2004 * pvr_mmu_op_context_sync() - Trigger a sync of every page table referenced by
2005 * a MMU op context.
2006 * @op_ctx: Target MMU op context.
2007 *
2008 * The maximum level marked internally as requiring a sync will be reset so
2009 * that subsequent calls to this function will be no-ops unless @op_ctx is
2010 * otherwise updated.
2011 */
2012static void
2013pvr_mmu_op_context_sync(struct pvr_mmu_op_context *op_ctx)
2014{
2015 pvr_mmu_op_context_sync_manual(op_ctx, op_ctx->sync_level_required);
2016
2017 op_ctx->sync_level_required = PVR_MMU_SYNC_LEVEL_NONE;
2018}
2019
2020/**
2021 * pvr_mmu_op_context_load_tables() - Load pointers to tables in each level of
2022 * the page table tree structure needed to reference the physical page
2023 * referenced by a MMU op context.
2024 * @op_ctx: Target MMU op context.
2025 * @should_create: Specifies whether new page tables should be created when
2026 * empty page table entries are encountered during traversal.
2027 * @load_level_required: Maximum page table level to load.
2028 *
2029 * If @should_create is %true, this function may modify the stored required
2030 * sync level of @op_ctx as new page tables are created and inserted into their
2031 * respective parents.
2032 *
2033 * Since there is only one root page table, it is technically incorrect to call
2034 * this function with a value of @load_level_required greater than or equal to
2035 * the root level number. However, this is not explicitly disallowed here.
2036 *
2037 * Return:
2038 * * 0 on success,
2039 * * Any error returned by pvr_page_table_l1_get_or_create() if
2040 * @load_level_required >= 1 except -%ENXIO, or
2041 * * Any error returned by pvr_page_table_l0_get_or_create() if
2042 * @load_level_required >= 0 except -%ENXIO.
2043 */
2044static int
2045pvr_mmu_op_context_load_tables(struct pvr_mmu_op_context *op_ctx,
2046 bool should_create,
2047 enum pvr_mmu_sync_level load_level_required)
2048{
2049 const struct pvr_page_table_l1 *l1_head_before =
2050 op_ctx->map.l1_prealloc_tables;
2051 const struct pvr_page_table_l0 *l0_head_before =
2052 op_ctx->map.l0_prealloc_tables;
2053 int err;
2054
2055 /* Clear tables we're about to fetch in case of error states. */
2056 if (load_level_required >= PVR_MMU_SYNC_LEVEL_1)
2057 op_ctx->curr_page.l1_table = NULL;
2058
2059 if (load_level_required >= PVR_MMU_SYNC_LEVEL_0)
2060 op_ctx->curr_page.l0_table = NULL;
2061
2062 /* Get or create L1 page table. */
2063 if (load_level_required >= PVR_MMU_SYNC_LEVEL_1) {
2064 err = pvr_page_table_l1_get_or_insert(op_ctx, should_create);
2065 if (err) {
2066 /*
2067 * If @should_create is %false and no L1 page table was
2068 * found, return early but without an error. Since
2069 * pvr_page_table_l1_get_or_create() can only return
2070 * -%ENXIO if @should_create is %false, there is no
2071 * need to check it here.
2072 */
2073 if (err == -ENXIO)
2074 err = 0;
2075
2076 return err;
2077 }
2078 }
2079
2080 /* Get or create L0 page table. */
2081 if (load_level_required >= PVR_MMU_SYNC_LEVEL_0) {
2082 err = pvr_page_table_l0_get_or_insert(op_ctx, should_create);
2083 if (err) {
2084 /*
2085 * If @should_create is %false and no L0 page table was
2086 * found, return early but without an error. Since
2087 * pvr_page_table_l0_get_or_insert() can only return
2088 * -%ENXIO if @should_create is %false, there is no
2089 * need to check it here.
2090 */
2091 if (err == -ENXIO)
2092 err = 0;
2093
2094 /*
2095 * At this point, an L1 page table could have been
2096 * inserted but is now empty due to the failed attempt
2097 * at inserting an L0 page table. In this instance, we
2098 * must remove the empty L1 page table ourselves as
2099 * pvr_page_table_l1_remove() is never called as part
2100 * of the error path in
2101 * pvr_page_table_l0_get_or_insert().
2102 */
2103 if (l1_head_before != op_ctx->map.l1_prealloc_tables) {
2104 pvr_page_table_l2_remove(op_ctx);
2105 pvr_mmu_op_context_require_sync(op_ctx, PVR_MMU_SYNC_LEVEL_2);
2106 }
2107
2108 return err;
2109 }
2110 }
2111
2112 /*
2113 * A sync is only needed if table objects were inserted. This can be
2114 * inferred by checking if the pointer at the head of the linked list
2115 * has changed.
2116 */
2117 if (l1_head_before != op_ctx->map.l1_prealloc_tables)
2118 pvr_mmu_op_context_require_sync(op_ctx, PVR_MMU_SYNC_LEVEL_2);
2119 else if (l0_head_before != op_ctx->map.l0_prealloc_tables)
2120 pvr_mmu_op_context_require_sync(op_ctx, PVR_MMU_SYNC_LEVEL_1);
2121
2122 return 0;
2123}
2124
2125/**
2126 * pvr_mmu_op_context_set_curr_page() - Reassign the current page of an MMU op
2127 * context, syncing any page tables previously assigned to it which are no
2128 * longer relevant.
2129 * @op_ctx: Target MMU op context.
2130 * @device_addr: New pointer target.
2131 * @should_create: Specify whether new page tables should be created when
2132 * empty page table entries are encountered during traversal.
2133 *
2134 * This function performs a full sync on the pointer, regardless of which
2135 * levels are modified.
2136 *
2137 * Return:
2138 * * 0 on success, or
2139 * * Any error returned by pvr_mmu_op_context_load_tables().
2140 */
2141static int
2142pvr_mmu_op_context_set_curr_page(struct pvr_mmu_op_context *op_ctx,
2143 u64 device_addr, bool should_create)
2144{
2145 pvr_mmu_op_context_sync(op_ctx);
2146
2147 op_ctx->curr_page.l2_idx = pvr_page_table_l2_idx(device_addr);
2148 op_ctx->curr_page.l1_idx = pvr_page_table_l1_idx(device_addr);
2149 op_ctx->curr_page.l0_idx = pvr_page_table_l0_idx(device_addr);
2150 op_ctx->curr_page.l1_table = NULL;
2151 op_ctx->curr_page.l0_table = NULL;
2152
2153 return pvr_mmu_op_context_load_tables(op_ctx, should_create,
2154 PVR_MMU_SYNC_LEVEL_1);
2155}
2156
2157/**
2158 * pvr_mmu_op_context_next_page() - Advance the current page of an MMU op
2159 * context.
2160 * @op_ctx: Target MMU op context.
2161 * @should_create: Specify whether new page tables should be created when
2162 * empty page table entries are encountered during traversal.
2163 *
2164 * If @should_create is %false, it is the caller's responsibility to verify that
2165 * the state of the table references in @op_ctx is valid on return. If -%ENXIO
2166 * is returned, at least one of the table references is invalid. It should be
2167 * noted that @op_ctx as a whole will be left in a valid state if -%ENXIO is
2168 * returned, unlike other error codes. The caller should check which references
2169 * are invalid by comparing them to %NULL. Only &@ptr->l2_table is guaranteed
2170 * to be valid, since it represents the root of the page table tree structure.
2171 *
2172 * Return:
2173 * * 0 on success,
2174 * * -%EPERM if the operation would wrap at the top of the page table
2175 * hierarchy,
2176 * * -%ENXIO if @should_create is %false and a page table of any level would
2177 * have otherwise been created, or
2178 * * Any error returned while attempting to create missing page tables if
2179 * @should_create is %true.
2180 */
2181static int
2182pvr_mmu_op_context_next_page(struct pvr_mmu_op_context *op_ctx,
2183 bool should_create)
2184{
2185 s8 load_level_required = PVR_MMU_SYNC_LEVEL_NONE;
2186
2187 if (++op_ctx->curr_page.l0_idx != ROGUE_MMUCTRL_ENTRIES_PT_VALUE_X)
2188 goto load_tables;
2189
2190 op_ctx->curr_page.l0_idx = 0;
2191 load_level_required = PVR_MMU_SYNC_LEVEL_0;
2192
2193 if (++op_ctx->curr_page.l1_idx != ROGUE_MMUCTRL_ENTRIES_PD_VALUE)
2194 goto load_tables;
2195
2196 op_ctx->curr_page.l1_idx = 0;
2197 load_level_required = PVR_MMU_SYNC_LEVEL_1;
2198
2199 if (++op_ctx->curr_page.l2_idx != ROGUE_MMUCTRL_ENTRIES_PC_VALUE)
2200 goto load_tables;
2201
2202 /*
2203 * If the pattern continued, we would set &op_ctx->curr_page.l2_idx to
2204 * zero here. However, that would wrap the top layer of the page table
2205 * hierarchy which is not a valid operation. Instead, we warn and return
2206 * an error.
2207 */
2208 WARN(true,
2209 "%s(%p) attempted to loop the top of the page table hierarchy",
2210 __func__, op_ctx);
2211 return -EPERM;
2212
2213 /* If indices have wrapped, we need to load new tables. */
2214load_tables:
2215 /* First, flush tables which will be unloaded. */
2216 pvr_mmu_op_context_sync_partial(op_ctx, load_level_required);
2217
2218 /* Then load tables from the required level down. */
2219 return pvr_mmu_op_context_load_tables(op_ctx, should_create,
2220 load_level_required);
2221}
2222
2223/**
2224 * DOC: Single page operations
2225 */
2226
2227/**
2228 * pvr_page_create() - Create a device-virtual memory page and insert it into
2229 * a level 0 page table.
2230 * @op_ctx: Target MMU op context pointing at the device-virtual address of the
2231 * target page.
2232 * @dma_addr: DMA address of the physical page backing the created page.
2233 * @flags: Page options saved on the level 0 page table entry for reading by
2234 * the device.
2235 *
2236 * Return:
2237 * * 0 on success, or
2238 * * -%EEXIST if the requested page already exists.
2239 */
2240static int
2241pvr_page_create(struct pvr_mmu_op_context *op_ctx, dma_addr_t dma_addr,
2242 struct pvr_page_flags_raw flags)
2243{
2244 /* Do not create a new page if one already exists. */
2245 if (pvr_page_table_l0_entry_is_valid(op_ctx->curr_page.l0_table,
2246 op_ctx->curr_page.l0_idx)) {
2247 return -EEXIST;
2248 }
2249
2250 pvr_page_table_l0_insert(op_ctx, dma_addr, flags);
2251
2252 pvr_mmu_op_context_require_sync(op_ctx, PVR_MMU_SYNC_LEVEL_0);
2253
2254 return 0;
2255}
2256
2257/**
2258 * pvr_page_destroy() - Destroy a device page after removing it from its
2259 * parent level 0 page table.
2260 * @op_ctx: Target MMU op context.
2261 */
2262static void
2263pvr_page_destroy(struct pvr_mmu_op_context *op_ctx)
2264{
2265 /* Do nothing if the page does not exist. */
2266 if (!pvr_page_table_l0_entry_is_valid(op_ctx->curr_page.l0_table,
2267 op_ctx->curr_page.l0_idx)) {
2268 return;
2269 }
2270
2271 /* Clear the parent L0 page table entry. */
2272 pvr_page_table_l0_remove(op_ctx);
2273
2274 pvr_mmu_op_context_require_sync(op_ctx, PVR_MMU_SYNC_LEVEL_0);
2275}
2276
2277/**
2278 * pvr_mmu_op_context_destroy() - Destroy an MMU op context.
2279 * @op_ctx: Target MMU op context.
2280 */
2281void pvr_mmu_op_context_destroy(struct pvr_mmu_op_context *op_ctx)
2282{
2283 const bool flush_caches =
2284 op_ctx->sync_level_required != PVR_MMU_SYNC_LEVEL_NONE;
2285
2286 pvr_mmu_op_context_sync(op_ctx);
2287
2288 /* Unmaps should be flushed immediately. Map flushes can be deferred. */
2289 if (flush_caches && !op_ctx->map.sgt)
2290 pvr_mmu_flush_exec(op_ctx->mmu_ctx->pvr_dev, true);
2291
2292 while (op_ctx->map.l0_prealloc_tables) {
2293 struct pvr_page_table_l0 *tmp = op_ctx->map.l0_prealloc_tables;
2294
2295 op_ctx->map.l0_prealloc_tables =
2296 op_ctx->map.l0_prealloc_tables->next_free;
2297 pvr_page_table_l0_free(tmp);
2298 }
2299
2300 while (op_ctx->map.l1_prealloc_tables) {
2301 struct pvr_page_table_l1 *tmp = op_ctx->map.l1_prealloc_tables;
2302
2303 op_ctx->map.l1_prealloc_tables =
2304 op_ctx->map.l1_prealloc_tables->next_free;
2305 pvr_page_table_l1_free(tmp);
2306 }
2307
2308 while (op_ctx->unmap.l0_free_tables) {
2309 struct pvr_page_table_l0 *tmp = op_ctx->unmap.l0_free_tables;
2310
2311 op_ctx->unmap.l0_free_tables =
2312 op_ctx->unmap.l0_free_tables->next_free;
2313 pvr_page_table_l0_free(tmp);
2314 }
2315
2316 while (op_ctx->unmap.l1_free_tables) {
2317 struct pvr_page_table_l1 *tmp = op_ctx->unmap.l1_free_tables;
2318
2319 op_ctx->unmap.l1_free_tables =
2320 op_ctx->unmap.l1_free_tables->next_free;
2321 pvr_page_table_l1_free(tmp);
2322 }
2323
2324 kfree(op_ctx);
2325}
2326
2327/**
2328 * pvr_mmu_op_context_create() - Create an MMU op context.
2329 * @ctx: MMU context associated with owning VM context.
2330 * @sgt: Scatter gather table containing pages pinned for use by this context.
2331 * @sgt_offset: Start offset of the requested device-virtual memory mapping.
2332 * @size: Size in bytes of the requested device-virtual memory mapping. For an
2333 * unmapping, this should be zero so that no page tables are allocated.
2334 *
2335 * Returns:
2336 * * Newly created MMU op context object on success, or
2337 * * -%ENOMEM if no memory is available,
2338 * * Any error code returned by pvr_page_table_l2_init().
2339 */
2340struct pvr_mmu_op_context *
2341pvr_mmu_op_context_create(struct pvr_mmu_context *ctx, struct sg_table *sgt,
2342 u64 sgt_offset, u64 size)
2343{
2344 int err;
2345
2346 struct pvr_mmu_op_context *op_ctx =
2347 kzalloc(sizeof(*op_ctx), GFP_KERNEL);
2348
2349 if (!op_ctx)
2350 return ERR_PTR(-ENOMEM);
2351
2352 op_ctx->mmu_ctx = ctx;
2353 op_ctx->map.sgt = sgt;
2354 op_ctx->map.sgt_offset = sgt_offset;
2355 op_ctx->sync_level_required = PVR_MMU_SYNC_LEVEL_NONE;
2356
2357 if (size) {
2358 /*
2359 * The number of page table objects we need to prealloc is
2360 * indicated by the mapping size, start offset and the sizes
2361 * of the areas mapped per PT or PD. The range calculation is
2362 * identical to that for the index into a table for a device
2363 * address, so we reuse those functions here.
2364 */
2365 const u32 l1_start_idx = pvr_page_table_l2_idx(sgt_offset);
2366 const u32 l1_end_idx = pvr_page_table_l2_idx(sgt_offset + size);
2367 const u32 l1_count = l1_end_idx - l1_start_idx + 1;
2368 const u32 l0_start_idx = pvr_page_table_l1_idx(sgt_offset);
2369 const u32 l0_end_idx = pvr_page_table_l1_idx(sgt_offset + size);
2370 const u32 l0_count = l0_end_idx - l0_start_idx + 1;
2371
2372 /*
2373 * Alloc and push page table entries until we have enough of
2374 * each type, ending with linked lists of l0 and l1 entries in
2375 * reverse order.
2376 */
2377 for (int i = 0; i < l1_count; i++) {
2378 struct pvr_page_table_l1 *l1_tmp =
2379 pvr_page_table_l1_alloc(ctx);
2380
2381 err = PTR_ERR_OR_ZERO(l1_tmp);
2382 if (err)
2383 goto err_cleanup;
2384
2385 l1_tmp->next_free = op_ctx->map.l1_prealloc_tables;
2386 op_ctx->map.l1_prealloc_tables = l1_tmp;
2387 }
2388
2389 for (int i = 0; i < l0_count; i++) {
2390 struct pvr_page_table_l0 *l0_tmp =
2391 pvr_page_table_l0_alloc(ctx);
2392
2393 err = PTR_ERR_OR_ZERO(l0_tmp);
2394 if (err)
2395 goto err_cleanup;
2396
2397 l0_tmp->next_free = op_ctx->map.l0_prealloc_tables;
2398 op_ctx->map.l0_prealloc_tables = l0_tmp;
2399 }
2400 }
2401
2402 return op_ctx;
2403
2404err_cleanup:
2405 pvr_mmu_op_context_destroy(op_ctx);
2406
2407 return ERR_PTR(err);
2408}
2409
2410/**
2411 * pvr_mmu_op_context_unmap_curr_page() - Unmap pages from a memory context
2412 * starting from the current page of an MMU op context.
2413 * @op_ctx: Target MMU op context pointing at the first page to unmap.
2414 * @nr_pages: Number of pages to unmap.
2415 *
2416 * Return:
2417 * * 0 on success, or
2418 * * Any error encountered while advancing @op_ctx.curr_page with
2419 * pvr_mmu_op_context_next_page() (except -%ENXIO).
2420 */
2421static int
2422pvr_mmu_op_context_unmap_curr_page(struct pvr_mmu_op_context *op_ctx,
2423 u64 nr_pages)
2424{
2425 int err;
2426
2427 if (nr_pages == 0)
2428 return 0;
2429
2430 /*
2431 * Destroy first page outside loop, as it doesn't require a page
2432 * advance beforehand. If the L0 page table reference in
2433 * @op_ctx.curr_page is %NULL, there cannot be a mapped page at
2434 * @op_ctx.curr_page (so skip ahead).
2435 */
2436 if (op_ctx->curr_page.l0_table)
2437 pvr_page_destroy(op_ctx);
2438
2439 for (u64 page = 1; page < nr_pages; ++page) {
2440 err = pvr_mmu_op_context_next_page(op_ctx, false);
2441 /*
2442 * If the page table tree structure at @op_ctx.curr_page is
2443 * incomplete, skip ahead. We don't care about unmapping pages
2444 * that cannot exist.
2445 *
2446 * FIXME: This could be made more efficient by jumping ahead
2447 * using pvr_mmu_op_context_set_curr_page().
2448 */
2449 if (err == -ENXIO)
2450 continue;
2451 else if (err)
2452 return err;
2453
2454 pvr_page_destroy(op_ctx);
2455 }
2456
2457 return 0;
2458}
2459
2460/**
2461 * pvr_mmu_unmap() - Unmap pages from a memory context.
2462 * @op_ctx: Target MMU op context.
2463 * @device_addr: First device-virtual address to unmap.
2464 * @size: Size in bytes to unmap.
2465 *
2466 * The total amount of device-virtual memory unmapped is
2467 * @nr_pages * %PVR_DEVICE_PAGE_SIZE.
2468 *
2469 * Returns:
2470 * * 0 on success, or
2471 * * Any error code returned by pvr_page_table_ptr_init(), or
2472 * * Any error code returned by pvr_page_table_ptr_unmap().
2473 */
2474int pvr_mmu_unmap(struct pvr_mmu_op_context *op_ctx, u64 device_addr, u64 size)
2475{
2476 int err = pvr_mmu_op_context_set_curr_page(op_ctx, device_addr, false);
2477
2478 if (err)
2479 return err;
2480
2481 return pvr_mmu_op_context_unmap_curr_page(op_ctx,
2482 size >> PVR_DEVICE_PAGE_SHIFT);
2483}
2484
2485/**
2486 * pvr_mmu_map_sgl() - Map part of a scatter-gather table entry to
2487 * device-virtual memory.
2488 * @op_ctx: Target MMU op context pointing to the first page that should be
2489 * mapped.
2490 * @sgl: Target scatter-gather table entry.
2491 * @offset: Offset into @sgl to map from. Must result in a starting address
2492 * from @sgl which is CPU page-aligned.
2493 * @size: Size of the memory to be mapped in bytes. Must be a non-zero multiple
2494 * of the device page size.
2495 * @page_flags: Page options to be applied to every device-virtual memory page
2496 * in the created mapping.
2497 *
2498 * Return:
2499 * * 0 on success,
2500 * * -%EINVAL if the range specified by @offset and @size is not completely
2501 * within @sgl, or
2502 * * Any error encountered while creating a page with pvr_page_create(), or
2503 * * Any error encountered while advancing @op_ctx.curr_page with
2504 * pvr_mmu_op_context_next_page().
2505 */
2506static int
2507pvr_mmu_map_sgl(struct pvr_mmu_op_context *op_ctx, struct scatterlist *sgl,
2508 u64 offset, u64 size, struct pvr_page_flags_raw page_flags)
2509{
2510 const unsigned int pages = size >> PVR_DEVICE_PAGE_SHIFT;
2511 dma_addr_t dma_addr = sg_dma_address(sgl) + offset;
2512 const unsigned int dma_len = sg_dma_len(sgl);
2513 struct pvr_page_table_ptr ptr_copy;
2514 unsigned int page;
2515 int err;
2516
2517 if (size > dma_len || offset > dma_len - size)
2518 return -EINVAL;
2519
2520 /*
2521 * Before progressing, save a copy of the start pointer so we can use
2522 * it again if we enter an error state and have to destroy pages.
2523 */
2524 memcpy(&ptr_copy, &op_ctx->curr_page, sizeof(ptr_copy));
2525
2526 /*
2527 * Create first page outside loop, as it doesn't require a page advance
2528 * beforehand.
2529 */
2530 err = pvr_page_create(op_ctx, dma_addr, page_flags);
2531 if (err)
2532 return err;
2533
2534 for (page = 1; page < pages; ++page) {
2535 err = pvr_mmu_op_context_next_page(op_ctx, true);
2536 if (err)
2537 goto err_destroy_pages;
2538
2539 dma_addr += PVR_DEVICE_PAGE_SIZE;
2540
2541 err = pvr_page_create(op_ctx, dma_addr, page_flags);
2542 if (err)
2543 goto err_destroy_pages;
2544 }
2545
2546 return 0;
2547
2548err_destroy_pages:
2549 memcpy(&op_ctx->curr_page, &ptr_copy, sizeof(op_ctx->curr_page));
2550 err = pvr_mmu_op_context_unmap_curr_page(op_ctx, page);
2551
2552 return err;
2553}
2554
2555/**
2556 * pvr_mmu_map() - Map an object's virtual memory to physical memory.
2557 * @op_ctx: Target MMU op context.
2558 * @size: Size of memory to be mapped in bytes. Must be a non-zero multiple
2559 * of the device page size.
2560 * @flags: Flags from pvr_gem_object associated with the mapping.
2561 * @device_addr: Virtual device address to map to. Must be device page-aligned.
2562 *
2563 * Returns:
2564 * * 0 on success, or
2565 * * Any error code returned by pvr_page_table_ptr_init(), or
2566 * * Any error code returned by pvr_mmu_map_sgl(), or
2567 * * Any error code returned by pvr_page_table_ptr_next_page().
2568 */
2569int pvr_mmu_map(struct pvr_mmu_op_context *op_ctx, u64 size, u64 flags,
2570 u64 device_addr)
2571{
2572 struct pvr_page_table_ptr ptr_copy;
2573 struct pvr_page_flags_raw flags_raw;
2574 struct scatterlist *sgl;
2575 u64 mapped_size = 0;
2576 unsigned int count;
2577 int err;
2578
2579 if (!size)
2580 return 0;
2581
2582 if ((op_ctx->map.sgt_offset | size) & ~PVR_DEVICE_PAGE_MASK)
2583 return -EINVAL;
2584
2585 err = pvr_mmu_op_context_set_curr_page(op_ctx, device_addr, true);
2586 if (err)
2587 return -EINVAL;
2588
2589 memcpy(&ptr_copy, &op_ctx->curr_page, sizeof(ptr_copy));
2590
2591 flags_raw = pvr_page_flags_raw_create(false, false,
2592 flags & DRM_PVR_BO_BYPASS_DEVICE_CACHE,
2593 flags & DRM_PVR_BO_PM_FW_PROTECT);
2594
2595 /* Map scatter gather table */
2596 for_each_sgtable_dma_sg(op_ctx->map.sgt, sgl, count) {
2597 const size_t sgl_len = sg_dma_len(sgl);
2598 u64 sgl_offset, map_sgl_len;
2599
2600 if (sgl_len <= op_ctx->map.sgt_offset) {
2601 op_ctx->map.sgt_offset -= sgl_len;
2602 continue;
2603 }
2604
2605 sgl_offset = op_ctx->map.sgt_offset;
2606 map_sgl_len = min_t(u64, sgl_len - sgl_offset, size - mapped_size);
2607
2608 err = pvr_mmu_map_sgl(op_ctx, sgl, sgl_offset, map_sgl_len,
2609 flags_raw);
2610 if (err)
2611 break;
2612
2613 /*
2614 * Flag the L0 page table as requiring a flush when the MMU op
2615 * context is destroyed.
2616 */
2617 pvr_mmu_op_context_require_sync(op_ctx, PVR_MMU_SYNC_LEVEL_0);
2618
2619 op_ctx->map.sgt_offset = 0;
2620 mapped_size += map_sgl_len;
2621
2622 if (mapped_size >= size)
2623 break;
2624
2625 err = pvr_mmu_op_context_next_page(op_ctx, true);
2626 if (err)
2627 break;
2628 }
2629
2630 if (err && mapped_size) {
2631 memcpy(&op_ctx->curr_page, &ptr_copy, sizeof(op_ctx->curr_page));
2632 pvr_mmu_op_context_unmap_curr_page(op_ctx,
2633 mapped_size >> PVR_DEVICE_PAGE_SHIFT);
2634 }
2635
2636 return err;
2637}
drivers/gpu/drm/imagination/pvr_mmu.h+108−0
@@ -0,0 +1,108 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_MMU_H
5#define PVR_MMU_H
6
7#include <linux/memory.h>
8#include <linux/types.h>
9
10/* Forward declaration from "pvr_device.h" */
11struct pvr_device;
12
13/* Forward declaration from "pvr_mmu.c" */
14struct pvr_mmu_context;
15struct pvr_mmu_op_context;
16
17/* Forward declaration from "pvr_vm.c" */
18struct pvr_vm_context;
19
20/* Forward declaration from <linux/scatterlist.h> */
21struct sg_table;
22
23/**
24 * DOC: Public API (constants)
25 *
26 * .. c:macro:: PVR_DEVICE_PAGE_SIZE
27 *
28 * Fixed page size referenced by leaf nodes in the page table tree
29 * structure. In the current implementation, this value is pegged to the
30 * CPU page size (%PAGE_SIZE). It is therefore an error to specify a CPU
31 * page size which is not also a supported device page size. The supported
32 * device page sizes are: 4KiB, 16KiB, 64KiB, 256KiB, 1MiB and 2MiB.
33 *
34 * .. c:macro:: PVR_DEVICE_PAGE_SHIFT
35 *
36 * Shift value used to efficiently multiply or divide by
37 * %PVR_DEVICE_PAGE_SIZE.
38 *
39 * This value is derived from %PVR_DEVICE_PAGE_SIZE.
40 *
41 * .. c:macro:: PVR_DEVICE_PAGE_MASK
42 *
43 * Mask used to round a value down to the nearest multiple of
44 * %PVR_DEVICE_PAGE_SIZE. When bitwise negated, it will indicate whether a
45 * value is already a multiple of %PVR_DEVICE_PAGE_SIZE.
46 *
47 * This value is derived from %PVR_DEVICE_PAGE_SIZE.
48 */
49
50/* PVR_DEVICE_PAGE_SIZE determines the page size */
51#define PVR_DEVICE_PAGE_SIZE (PAGE_SIZE)
52#define PVR_DEVICE_PAGE_SHIFT (PAGE_SHIFT)
53#define PVR_DEVICE_PAGE_MASK (PAGE_MASK)
54
55/**
56 * DOC: Page table index utilities (constants)
57 *
58 * .. c:macro:: PVR_PAGE_TABLE_ADDR_SPACE_SIZE
59 *
60 * Size of device-virtual address space which can be represented in the page
61 * table structure.
62 *
63 * This value is checked at runtime against
64 * &pvr_device_features.virtual_address_space_bits by
65 * pvr_vm_create_context(), which will return an error if the feature value
66 * does not match this constant.
67 *
68 * .. admonition:: Future work
69 *
70 * It should be possible to support other values of
71 * &pvr_device_features.virtual_address_space_bits, but so far no
72 * hardware has been created which advertises an unsupported value.
73 *
74 * .. c:macro:: PVR_PAGE_TABLE_ADDR_BITS
75 *
76 * Number of bits needed to represent any value less than
77 * %PVR_PAGE_TABLE_ADDR_SPACE_SIZE exactly.
78 *
79 * .. c:macro:: PVR_PAGE_TABLE_ADDR_MASK
80 *
81 * Bitmask of device-virtual addresses which are valid in the page table
82 * structure.
83 *
84 * This value is derived from %PVR_PAGE_TABLE_ADDR_SPACE_SIZE, so the same
85 * notes on that constant apply here.
86 */
87#define PVR_PAGE_TABLE_ADDR_SPACE_SIZE SZ_1T
88#define PVR_PAGE_TABLE_ADDR_BITS __ffs(PVR_PAGE_TABLE_ADDR_SPACE_SIZE)
89#define PVR_PAGE_TABLE_ADDR_MASK (PVR_PAGE_TABLE_ADDR_SPACE_SIZE - 1)
90
91void pvr_mmu_flush_request_all(struct pvr_device *pvr_dev);
92int pvr_mmu_flush_exec(struct pvr_device *pvr_dev, bool wait);
93
94struct pvr_mmu_context *pvr_mmu_context_create(struct pvr_device *pvr_dev);
95void pvr_mmu_context_destroy(struct pvr_mmu_context *ctx);
96
97dma_addr_t pvr_mmu_get_root_table_dma_addr(struct pvr_mmu_context *ctx);
98
99void pvr_mmu_op_context_destroy(struct pvr_mmu_op_context *op_ctx);
100struct pvr_mmu_op_context *
101pvr_mmu_op_context_create(struct pvr_mmu_context *ctx,
102 struct sg_table *sgt, u64 sgt_offset, u64 size);
103
104int pvr_mmu_map(struct pvr_mmu_op_context *op_ctx, u64 size, u64 flags,
105 u64 device_addr);
106int pvr_mmu_unmap(struct pvr_mmu_op_context *op_ctx, u64 device_addr, u64 size);
107
108#endif /* PVR_MMU_H */
drivers/gpu/drm/imagination/pvr_params.c+147−0
@@ -0,0 +1,147 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_params.h"
5
6#include <linux/cache.h>
7#include <linux/moduleparam.h>
8
9static struct pvr_device_params pvr_device_param_defaults __read_mostly = {
10#define X(type_, name_, value_, desc_, ...) .name_ = (value_),
11 PVR_DEVICE_PARAMS
12#undef X
13};
14
15#define PVR_DEVICE_PARAM_NAMED(name_, type_, desc_) \
16 module_param_named(name_, pvr_device_param_defaults.name_, type_, \
17 0400); \
18 MODULE_PARM_DESC(name_, desc_);
19
20/*
21 * This list of defines must contain every type specified in "pvr_params.h" as
22 * ``PVR_PARAM_TYPE_*_C``.
23 */
24#define PVR_PARAM_TYPE_X32_MODPARAM uint
25
26#define X(type_, name_, value_, desc_, ...) \
27 PVR_DEVICE_PARAM_NAMED(name_, PVR_PARAM_TYPE_##type_##_MODPARAM, desc_);
28PVR_DEVICE_PARAMS
29#undef X
30
31int
32pvr_device_params_init(struct pvr_device_params *params)
33{
34 /*
35 * If heap-allocated parameters are added in the future (e.g.
36 * modparam's charp type), they must be handled specially here (via
37 * kstrdup() in the case of charp). Since that's not necessary yet,
38 * a straight copy will do for now. This change will also require a
39 * pvr_device_params_fini() function to free any heap-allocated copies.
40 */
41
42 *params = pvr_device_param_defaults;
43
44 return 0;
45}
46
47#if defined(CONFIG_DEBUG_FS)
48#include "pvr_device.h"
49
50#include <linux/dcache.h>
51#include <linux/debugfs.h>
52#include <linux/export.h>
53#include <linux/fs.h>
54#include <linux/stddef.h>
55
56/*
57 * This list of defines must contain every type specified in "pvr_params.h" as
58 * ``PVR_PARAM_TYPE_*_C``.
59 */
60#define PVR_PARAM_TYPE_X32_FMT "0x%08llx"
61
62#define X_SET(name_, mode_) X_SET_##mode_(name_)
63#define X_SET_DEF(name_, update_, mode_) X_SET_DEF_##mode_(name_, update_)
64
65#define X_SET_RO(name_) NULL
66#define X_SET_RW(name_) __pvr_device_param_##name_##set
67
68#define X_SET_DEF_RO(name_, update_)
69#define X_SET_DEF_RW(name_, update_) \
70 static int \
71 X_SET_RW(name_)(void *data, u64 val) \
72 { \
73 struct pvr_device *pvr_dev = data; \
74 /* This is not just (update_) to suppress -Waddress. */ \
75 if ((void *)(update_) != NULL) \
76 (update_)(pvr_dev, pvr_dev->params.name_, val); \
77 pvr_dev->params.name_ = val; \
78 return 0; \
79 }
80
81#define X(type_, name_, value_, desc_, mode_, update_) \
82 static int \
83 __pvr_device_param_##name_##_get(void *data, u64 *val) \
84 { \
85 struct pvr_device *pvr_dev = data; \
86 *val = pvr_dev->params.name_; \
87 return 0; \
88 } \
89 X_SET_DEF(name_, update_, mode_) \
90 static int \
91 __pvr_device_param_##name_##_open(struct inode *inode, \
92 struct file *file) \
93 { \
94 __simple_attr_check_format(PVR_PARAM_TYPE_##type_##_FMT, \
95 0ull); \
96 return simple_attr_open(inode, file, \
97 __pvr_device_param_##name_##_get, \
98 X_SET(name_, mode_), \
99 PVR_PARAM_TYPE_##type_##_FMT); \
100 }
101PVR_DEVICE_PARAMS
102#undef X
103
104#undef X_SET
105#undef X_SET_RO
106#undef X_SET_RW
107#undef X_SET_DEF
108#undef X_SET_DEF_RO
109#undef X_SET_DEF_RW
110
111static struct {
112#define X(type_, name_, value_, desc_, mode_, update_) \
113 const struct file_operations name_;
114 PVR_DEVICE_PARAMS
115#undef X
116} pvr_device_param_debugfs_fops = {
117#define X(type_, name_, value_, desc_, mode_, update_) \
118 .name_ = { \
119 .owner = THIS_MODULE, \
120 .open = __pvr_device_param_##name_##_open, \
121 .release = simple_attr_release, \
122 .read = simple_attr_read, \
123 .write = simple_attr_write, \
124 .llseek = generic_file_llseek, \
125 },
126 PVR_DEVICE_PARAMS
127#undef X
128};
129
130void
131pvr_params_debugfs_init(struct pvr_device *pvr_dev, struct dentry *dir)
132{
133#define X_MODE(mode_) X_MODE_##mode_
134#define X_MODE_RO 0400
135#define X_MODE_RW 0600
136
137#define X(type_, name_, value_, desc_, mode_, update_) \
138 debugfs_create_file(#name_, X_MODE(mode_), dir, pvr_dev, \
139 &pvr_device_param_debugfs_fops.name_);
140 PVR_DEVICE_PARAMS
141#undef X
142
143#undef X_MODE
144#undef X_MODE_RO
145#undef X_MODE_RW
146}
147#endif
drivers/gpu/drm/imagination/pvr_params.h+72−0
@@ -0,0 +1,72 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_PARAMS_H
5#define PVR_PARAMS_H
6
7#include "pvr_rogue_fwif.h"
8
9#include <linux/cache.h>
10#include <linux/compiler_attributes.h>
11
12/*
13 * This is the definitive list of types allowed in the definition of
14 * %PVR_DEVICE_PARAMS.
15 */
16#define PVR_PARAM_TYPE_X32_C u32
17
18/*
19 * This macro defines all device-specific parameters; that is parameters which
20 * are set independently per device.
21 *
22 * The X-macro accepts the following arguments. Arguments marked with [debugfs]
23 * are ignored when debugfs is disabled; values used for these arguments may
24 * safely be gated behind CONFIG_DEBUG_FS.
25 *
26 * @type_: The definitive list of allowed values is PVR_PARAM_TYPE_*_C.
27 * @name_: Name of the parameter. This is used both as the field name in C and
28 * stringified as the parameter name.
29 * @value_: Initial/default value.
30 * @desc_: String literal used as help text to describe the usage of this
31 * parameter.
32 * @mode_: [debugfs] One of {RO,RW}. The access mode of the debugfs entry for
33 * this parameter.
34 * @update_: [debugfs] When debugfs support is enabled, parameters may be
35 * updated at runtime. When this happens, this function will be
36 * called to allow changes to propagate. The signature of this
37 * function is:
38 *
39 * void (*)(struct pvr_device *pvr_dev, T old_val, T new_val)
40 *
41 * Where T is the C type associated with @type_.
42 *
43 * If @mode_ does not allow write access, this function will never be
44 * called. In this case, or if no update callback is required, you
45 * should specify NULL for this argument.
46 */
47#define PVR_DEVICE_PARAMS \
48 X(X32, fw_trace_mask, ROGUE_FWIF_LOG_TYPE_NONE, \
49 "Enable FW trace for the specified groups. Specifying 0 disables " \
50 "all FW tracing.", \
51 RW, pvr_fw_trace_mask_update)
52
53struct pvr_device_params {
54#define X(type_, name_, value_, desc_, ...) \
55 PVR_PARAM_TYPE_##type_##_C name_;
56 PVR_DEVICE_PARAMS
57#undef X
58};
59
60int pvr_device_params_init(struct pvr_device_params *params);
61
62#if defined(CONFIG_DEBUG_FS)
63/* Forward declaration from "pvr_device.h". */
64struct pvr_device;
65
66/* Forward declaration from <linux/dcache.h>. */
67struct dentry;
68
69void pvr_params_debugfs_init(struct pvr_device *pvr_dev, struct dentry *dir);
70#endif /* defined(CONFIG_DEBUG_FS) */
71
72#endif /* PVR_PARAMS_H */
drivers/gpu/drm/imagination/pvr_power.c+433−0
@@ -0,0 +1,433 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include "pvr_device.h"
5#include "pvr_fw.h"
6#include "pvr_fw_startstop.h"
7#include "pvr_power.h"
8#include "pvr_queue.h"
9#include "pvr_rogue_fwif.h"
10
11#include <drm/drm_drv.h>
12#include <drm/drm_managed.h>
13#include <linux/clk.h>
14#include <linux/interrupt.h>
15#include <linux/mutex.h>
16#include <linux/platform_device.h>
17#include <linux/pm_runtime.h>
18#include <linux/timer.h>
19#include <linux/types.h>
20#include <linux/workqueue.h>
21
22#define POWER_SYNC_TIMEOUT_US (1000000) /* 1s */
23
24#define WATCHDOG_TIME_MS (500)
25
26/**
27 * pvr_device_lost() - Mark GPU device as lost
28 * @pvr_dev: Target PowerVR device.
29 *
30 * This will cause the DRM device to be unplugged.
31 */
32void
33pvr_device_lost(struct pvr_device *pvr_dev)
34{
35 if (!pvr_dev->lost) {
36 pvr_dev->lost = true;
37 drm_dev_unplug(from_pvr_device(pvr_dev));
38 }
39}
40
41static int
42pvr_power_send_command(struct pvr_device *pvr_dev, struct rogue_fwif_kccb_cmd *pow_cmd)
43{
44 struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
45 u32 slot_nr;
46 u32 value;
47 int err;
48
49 WRITE_ONCE(*fw_dev->power_sync, 0);
50
51 err = pvr_kccb_send_cmd_powered(pvr_dev, pow_cmd, &slot_nr);
52 if (err)
53 return err;
54
55 /* Wait for FW to acknowledge. */
56 return readl_poll_timeout(pvr_dev->fw_dev.power_sync, value, value != 0, 100,
57 POWER_SYNC_TIMEOUT_US);
58}
59
60static int
61pvr_power_request_idle(struct pvr_device *pvr_dev)
62{
63 struct rogue_fwif_kccb_cmd pow_cmd;
64
65 /* Send FORCED_IDLE request to FW. */
66 pow_cmd.cmd_type = ROGUE_FWIF_KCCB_CMD_POW;
67 pow_cmd.cmd_data.pow_data.pow_type = ROGUE_FWIF_POW_FORCED_IDLE_REQ;
68 pow_cmd.cmd_data.pow_data.power_req_data.pow_request_type = ROGUE_FWIF_POWER_FORCE_IDLE;
69
70 return pvr_power_send_command(pvr_dev, &pow_cmd);
71}
72
73static int
74pvr_power_request_pwr_off(struct pvr_device *pvr_dev)
75{
76 struct rogue_fwif_kccb_cmd pow_cmd;
77
78 /* Send POW_OFF request to firmware. */
79 pow_cmd.cmd_type = ROGUE_FWIF_KCCB_CMD_POW;
80 pow_cmd.cmd_data.pow_data.pow_type = ROGUE_FWIF_POW_OFF_REQ;
81 pow_cmd.cmd_data.pow_data.power_req_data.forced = true;
82
83 return pvr_power_send_command(pvr_dev, &pow_cmd);
84}
85
86static int
87pvr_power_fw_disable(struct pvr_device *pvr_dev, bool hard_reset)
88{
89 if (!hard_reset) {
90 int err;
91
92 cancel_delayed_work_sync(&pvr_dev->watchdog.work);
93
94 err = pvr_power_request_idle(pvr_dev);
95 if (err)
96 return err;
97
98 err = pvr_power_request_pwr_off(pvr_dev);
99 if (err)
100 return err;
101 }
102
103 return pvr_fw_stop(pvr_dev);
104}
105
106static int
107pvr_power_fw_enable(struct pvr_device *pvr_dev)
108{
109 int err;
110
111 err = pvr_fw_start(pvr_dev);
112 if (err)
113 return err;
114
115 err = pvr_wait_for_fw_boot(pvr_dev);
116 if (err) {
117 drm_err(from_pvr_device(pvr_dev), "Firmware failed to boot\n");
118 pvr_fw_stop(pvr_dev);
119 return err;
120 }
121
122 queue_delayed_work(pvr_dev->sched_wq, &pvr_dev->watchdog.work,
123 msecs_to_jiffies(WATCHDOG_TIME_MS));
124
125 return 0;
126}
127
128bool
129pvr_power_is_idle(struct pvr_device *pvr_dev)
130{
131 /*
132 * FW power state can be out of date if a KCCB command has been submitted but the FW hasn't
133 * started processing it yet. So also check the KCCB status.
134 */
135 enum rogue_fwif_pow_state pow_state = READ_ONCE(pvr_dev->fw_dev.fwif_sysdata->pow_state);
136 bool kccb_idle = pvr_kccb_is_idle(pvr_dev);
137
138 return (pow_state == ROGUE_FWIF_POW_IDLE) && kccb_idle;
139}
140
141static bool
142pvr_watchdog_kccb_stalled(struct pvr_device *pvr_dev)
143{
144 /* Check KCCB commands are progressing. */
145 u32 kccb_cmds_executed = pvr_dev->fw_dev.fwif_osdata->kccb_cmds_executed;
146 bool kccb_is_idle = pvr_kccb_is_idle(pvr_dev);
147
148 if (pvr_dev->watchdog.old_kccb_cmds_executed == kccb_cmds_executed && !kccb_is_idle) {
149 pvr_dev->watchdog.kccb_stall_count++;
150
151 /*
152 * If we have commands pending with no progress for 2 consecutive polls then
153 * consider KCCB command processing stalled.
154 */
155 if (pvr_dev->watchdog.kccb_stall_count == 2) {
156 pvr_dev->watchdog.kccb_stall_count = 0;
157 return true;
158 }
159 } else if (pvr_dev->watchdog.old_kccb_cmds_executed == kccb_cmds_executed) {
160 bool has_active_contexts;
161
162 mutex_lock(&pvr_dev->queues.lock);
163 has_active_contexts = list_empty(&pvr_dev->queues.active);
164 mutex_unlock(&pvr_dev->queues.lock);
165
166 if (has_active_contexts) {
167 /* Send a HEALTH_CHECK command so we can verify FW is still alive. */
168 struct rogue_fwif_kccb_cmd health_check_cmd;
169
170 health_check_cmd.cmd_type = ROGUE_FWIF_KCCB_CMD_HEALTH_CHECK;
171
172 pvr_kccb_send_cmd_powered(pvr_dev, &health_check_cmd, NULL);
173 }
174 } else {
175 pvr_dev->watchdog.old_kccb_cmds_executed = kccb_cmds_executed;
176 pvr_dev->watchdog.kccb_stall_count = 0;
177 }
178
179 return false;
180}
181
182static void
183pvr_watchdog_worker(struct work_struct *work)
184{
185 struct pvr_device *pvr_dev = container_of(work, struct pvr_device,
186 watchdog.work.work);
187 bool stalled;
188
189 if (pvr_dev->lost)
190 return;
191
192 if (pm_runtime_get_if_in_use(from_pvr_device(pvr_dev)->dev) <= 0)
193 goto out_requeue;
194
195 if (!pvr_dev->fw_dev.booted)
196 goto out_pm_runtime_put;
197
198 stalled = pvr_watchdog_kccb_stalled(pvr_dev);
199
200 if (stalled) {
201 drm_err(from_pvr_device(pvr_dev), "FW stalled, trying hard reset");
202
203 pvr_power_reset(pvr_dev, true);
204 /* Device may be lost at this point. */
205 }
206
207out_pm_runtime_put:
208 pm_runtime_put(from_pvr_device(pvr_dev)->dev);
209
210out_requeue:
211 if (!pvr_dev->lost) {
212 queue_delayed_work(pvr_dev->sched_wq, &pvr_dev->watchdog.work,
213 msecs_to_jiffies(WATCHDOG_TIME_MS));
214 }
215}
216
217/**
218 * pvr_watchdog_init() - Initialise watchdog for device
219 * @pvr_dev: Target PowerVR device.
220 *
221 * Returns:
222 * * 0 on success, or
223 * * -%ENOMEM on out of memory.
224 */
225int
226pvr_watchdog_init(struct pvr_device *pvr_dev)
227{
228 INIT_DELAYED_WORK(&pvr_dev->watchdog.work, pvr_watchdog_worker);
229
230 return 0;
231}
232
233int
234pvr_power_device_suspend(struct device *dev)
235{
236 struct platform_device *plat_dev = to_platform_device(dev);
237 struct drm_device *drm_dev = platform_get_drvdata(plat_dev);
238 struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
239 int err = 0;
240 int idx;
241
242 if (!drm_dev_enter(drm_dev, &idx))
243 return -EIO;
244
245 if (pvr_dev->fw_dev.booted) {
246 err = pvr_power_fw_disable(pvr_dev, false);
247 if (err)
248 goto err_drm_dev_exit;
249 }
250
251 clk_disable_unprepare(pvr_dev->mem_clk);
252 clk_disable_unprepare(pvr_dev->sys_clk);
253 clk_disable_unprepare(pvr_dev->core_clk);
254
255err_drm_dev_exit:
256 drm_dev_exit(idx);
257
258 return err;
259}
260
261int
262pvr_power_device_resume(struct device *dev)
263{
264 struct platform_device *plat_dev = to_platform_device(dev);
265 struct drm_device *drm_dev = platform_get_drvdata(plat_dev);
266 struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
267 int idx;
268 int err;
269
270 if (!drm_dev_enter(drm_dev, &idx))
271 return -EIO;
272
273 err = clk_prepare_enable(pvr_dev->core_clk);
274 if (err)
275 goto err_drm_dev_exit;
276
277 err = clk_prepare_enable(pvr_dev->sys_clk);
278 if (err)
279 goto err_core_clk_disable;
280
281 err = clk_prepare_enable(pvr_dev->mem_clk);
282 if (err)
283 goto err_sys_clk_disable;
284
285 if (pvr_dev->fw_dev.booted) {
286 err = pvr_power_fw_enable(pvr_dev);
287 if (err)
288 goto err_mem_clk_disable;
289 }
290
291 drm_dev_exit(idx);
292
293 return 0;
294
295err_mem_clk_disable:
296 clk_disable_unprepare(pvr_dev->mem_clk);
297
298err_sys_clk_disable:
299 clk_disable_unprepare(pvr_dev->sys_clk);
300
301err_core_clk_disable:
302 clk_disable_unprepare(pvr_dev->core_clk);
303
304err_drm_dev_exit:
305 drm_dev_exit(idx);
306
307 return err;
308}
309
310int
311pvr_power_device_idle(struct device *dev)
312{
313 struct platform_device *plat_dev = to_platform_device(dev);
314 struct drm_device *drm_dev = platform_get_drvdata(plat_dev);
315 struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
316
317 return pvr_power_is_idle(pvr_dev) ? 0 : -EBUSY;
318}
319
320/**
321 * pvr_power_reset() - Reset the GPU
322 * @pvr_dev: Device pointer
323 * @hard_reset: %true for hard reset, %false for soft reset
324 *
325 * If @hard_reset is %false and the FW processor fails to respond during the reset process, this
326 * function will attempt a hard reset.
327 *
328 * If a hard reset fails then the GPU device is reported as lost.
329 *
330 * Returns:
331 * * 0 on success, or
332 * * Any error code returned by pvr_power_get, pvr_power_fw_disable or pvr_power_fw_enable().
333 */
334int
335pvr_power_reset(struct pvr_device *pvr_dev, bool hard_reset)
336{
337 bool queues_disabled = false;
338 int err;
339
340 /*
341 * Take a power reference during the reset. This should prevent any interference with the
342 * power state during reset.
343 */
344 WARN_ON(pvr_power_get(pvr_dev));
345
346 down_write(&pvr_dev->reset_sem);
347
348 if (pvr_dev->lost) {
349 err = -EIO;
350 goto err_up_write;
351 }
352
353 /* Disable IRQs for the duration of the reset. */
354 disable_irq(pvr_dev->irq);
355
356 do {
357 if (hard_reset) {
358 pvr_queue_device_pre_reset(pvr_dev);
359 queues_disabled = true;
360 }
361
362 err = pvr_power_fw_disable(pvr_dev, hard_reset);
363 if (!err) {
364 if (hard_reset) {
365 pvr_dev->fw_dev.booted = false;
366 WARN_ON(pm_runtime_force_suspend(from_pvr_device(pvr_dev)->dev));
367
368 err = pvr_fw_hard_reset(pvr_dev);
369 if (err)
370 goto err_device_lost;
371
372 err = pm_runtime_force_resume(from_pvr_device(pvr_dev)->dev);
373 pvr_dev->fw_dev.booted = true;
374 if (err)
375 goto err_device_lost;
376 } else {
377 /* Clear the FW faulted flags. */
378 pvr_dev->fw_dev.fwif_sysdata->hwr_state_flags &=
379 ~(ROGUE_FWIF_HWR_FW_FAULT |
380 ROGUE_FWIF_HWR_RESTART_REQUESTED);
381 }
382
383 pvr_fw_irq_clear(pvr_dev);
384
385 err = pvr_power_fw_enable(pvr_dev);
386 }
387
388 if (err && hard_reset)
389 goto err_device_lost;
390
391 if (err && !hard_reset) {
392 drm_err(from_pvr_device(pvr_dev), "FW stalled, trying hard reset");
393 hard_reset = true;
394 }
395 } while (err);
396
397 if (queues_disabled)
398 pvr_queue_device_post_reset(pvr_dev);
399
400 enable_irq(pvr_dev->irq);
401
402 up_write(&pvr_dev->reset_sem);
403
404 pvr_power_put(pvr_dev);
405
406 return 0;
407
408err_device_lost:
409 drm_err(from_pvr_device(pvr_dev), "GPU device lost");
410 pvr_device_lost(pvr_dev);
411
412 /* Leave IRQs disabled if the device is lost. */
413
414 if (queues_disabled)
415 pvr_queue_device_post_reset(pvr_dev);
416
417err_up_write:
418 up_write(&pvr_dev->reset_sem);
419
420 pvr_power_put(pvr_dev);
421
422 return err;
423}
424
425/**
426 * pvr_watchdog_fini() - Shutdown watchdog for device
427 * @pvr_dev: Target PowerVR device.
428 */
429void
430pvr_watchdog_fini(struct pvr_device *pvr_dev)
431{
432 cancel_delayed_work_sync(&pvr_dev->watchdog.work);
433}
drivers/gpu/drm/imagination/pvr_power.h+41−0
@@ -0,0 +1,41 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_POWER_H
5#define PVR_POWER_H
6
7#include "pvr_device.h"
8
9#include <linux/mutex.h>
10#include <linux/pm_runtime.h>
11
12int pvr_watchdog_init(struct pvr_device *pvr_dev);
13void pvr_watchdog_fini(struct pvr_device *pvr_dev);
14
15void pvr_device_lost(struct pvr_device *pvr_dev);
16
17bool pvr_power_is_idle(struct pvr_device *pvr_dev);
18
19int pvr_power_device_suspend(struct device *dev);
20int pvr_power_device_resume(struct device *dev);
21int pvr_power_device_idle(struct device *dev);
22
23int pvr_power_reset(struct pvr_device *pvr_dev, bool hard_reset);
24
25static __always_inline int
26pvr_power_get(struct pvr_device *pvr_dev)
27{
28 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
29
30 return pm_runtime_resume_and_get(drm_dev->dev);
31}
32
33static __always_inline int
34pvr_power_put(struct pvr_device *pvr_dev)
35{
36 struct drm_device *drm_dev = from_pvr_device(pvr_dev);
37
38 return pm_runtime_put(drm_dev->dev);
39}
40
41#endif /* PVR_POWER_H */
drivers/gpu/drm/imagination/pvr_queue.c+1432−0
@@ -0,0 +1,1432 @@
1// SPDX-License-Identifier: GPL-2.0-only OR MIT
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#include <drm/drm_managed.h>
5#include <drm/gpu_scheduler.h>
6
7#include "pvr_cccb.h"
8#include "pvr_context.h"
9#include "pvr_device.h"
10#include "pvr_drv.h"
11#include "pvr_job.h"
12#include "pvr_queue.h"
13#include "pvr_vm.h"
14
15#include "pvr_rogue_fwif_client.h"
16
17#define MAX_DEADLINE_MS 30000
18
19#define CTX_COMPUTE_CCCB_SIZE_LOG2 15
20#define CTX_FRAG_CCCB_SIZE_LOG2 15
21#define CTX_GEOM_CCCB_SIZE_LOG2 15
22#define CTX_TRANSFER_CCCB_SIZE_LOG2 15
23
24static int get_xfer_ctx_state_size(struct pvr_device *pvr_dev)
25{
26 u32 num_isp_store_registers;
27
28 if (PVR_HAS_FEATURE(pvr_dev, xe_memory_hierarchy)) {
29 num_isp_store_registers = 1;
30 } else {
31 int err;
32
33 err = PVR_FEATURE_VALUE(pvr_dev, num_isp_ipp_pipes, &num_isp_store_registers);
34 if (WARN_ON(err))
35 return err;
36 }
37
38 return sizeof(struct rogue_fwif_frag_ctx_state) +
39 (num_isp_store_registers *
40 sizeof(((struct rogue_fwif_frag_ctx_state *)0)->frag_reg_isp_store[0]));
41}
42
43static int get_frag_ctx_state_size(struct pvr_device *pvr_dev)
44{
45 u32 num_isp_store_registers;
46 int err;
47
48 if (PVR_HAS_FEATURE(pvr_dev, xe_memory_hierarchy)) {
49 err = PVR_FEATURE_VALUE(pvr_dev, num_raster_pipes, &num_isp_store_registers);
50 if (WARN_ON(err))
51 return err;
52
53 if (PVR_HAS_FEATURE(pvr_dev, gpu_multicore_support)) {
54 u32 xpu_max_slaves;
55
56 err = PVR_FEATURE_VALUE(pvr_dev, xpu_max_slaves, &xpu_max_slaves);
57 if (WARN_ON(err))
58 return err;
59
60 num_isp_store_registers *= (1 + xpu_max_slaves);
61 }
62 } else {
63 err = PVR_FEATURE_VALUE(pvr_dev, num_isp_ipp_pipes, &num_isp_store_registers);
64 if (WARN_ON(err))
65 return err;
66 }
67
68 return sizeof(struct rogue_fwif_frag_ctx_state) +
69 (num_isp_store_registers *
70 sizeof(((struct rogue_fwif_frag_ctx_state *)0)->frag_reg_isp_store[0]));
71}
72
73static int get_ctx_state_size(struct pvr_device *pvr_dev, enum drm_pvr_job_type type)
74{
75 switch (type) {
76 case DRM_PVR_JOB_TYPE_GEOMETRY:
77 return sizeof(struct rogue_fwif_geom_ctx_state);
78 case DRM_PVR_JOB_TYPE_FRAGMENT:
79 return get_frag_ctx_state_size(pvr_dev);
80 case DRM_PVR_JOB_TYPE_COMPUTE:
81 return sizeof(struct rogue_fwif_compute_ctx_state);
82 case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
83 return get_xfer_ctx_state_size(pvr_dev);
84 }
85
86 WARN(1, "Invalid queue type");
87 return -EINVAL;
88}
89
90static u32 get_ctx_offset(enum drm_pvr_job_type type)
91{
92 switch (type) {
93 case DRM_PVR_JOB_TYPE_GEOMETRY:
94 return offsetof(struct rogue_fwif_fwrendercontext, geom_context);
95 case DRM_PVR_JOB_TYPE_FRAGMENT:
96 return offsetof(struct rogue_fwif_fwrendercontext, frag_context);
97 case DRM_PVR_JOB_TYPE_COMPUTE:
98 return offsetof(struct rogue_fwif_fwcomputecontext, cdm_context);
99 case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
100 return offsetof(struct rogue_fwif_fwtransfercontext, tq_context);
101 }
102
103 return 0;
104}
105
106static const char *
107pvr_queue_fence_get_driver_name(struct dma_fence *f)
108{
109 return PVR_DRIVER_NAME;
110}
111
112static void pvr_queue_fence_release(struct dma_fence *f)
113{
114 struct pvr_queue_fence *fence = container_of(f, struct pvr_queue_fence, base);
115
116 pvr_context_put(fence->queue->ctx);
117 dma_fence_free(f);
118}
119
120static const char *
121pvr_queue_job_fence_get_timeline_name(struct dma_fence *f)
122{
123 struct pvr_queue_fence *fence = container_of(f, struct pvr_queue_fence, base);
124
125 switch (fence->queue->type) {
126 case DRM_PVR_JOB_TYPE_GEOMETRY:
127 return "geometry";
128
129 case DRM_PVR_JOB_TYPE_FRAGMENT:
130 return "fragment";
131
132 case DRM_PVR_JOB_TYPE_COMPUTE:
133 return "compute";
134
135 case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
136 return "transfer";
137 }
138
139 WARN(1, "Invalid queue type");
140 return "invalid";
141}
142
143static const char *
144pvr_queue_cccb_fence_get_timeline_name(struct dma_fence *f)
145{
146 struct pvr_queue_fence *fence = container_of(f, struct pvr_queue_fence, base);
147
148 switch (fence->queue->type) {
149 case DRM_PVR_JOB_TYPE_GEOMETRY:
150 return "geometry-cccb";
151
152 case DRM_PVR_JOB_TYPE_FRAGMENT:
153 return "fragment-cccb";
154
155 case DRM_PVR_JOB_TYPE_COMPUTE:
156 return "compute-cccb";
157
158 case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
159 return "transfer-cccb";
160 }
161
162 WARN(1, "Invalid queue type");
163 return "invalid";
164}
165
166static const struct dma_fence_ops pvr_queue_job_fence_ops = {
167 .get_driver_name = pvr_queue_fence_get_driver_name,
168 .get_timeline_name = pvr_queue_job_fence_get_timeline_name,
169 .release = pvr_queue_fence_release,
170};
171
172/**
173 * to_pvr_queue_job_fence() - Return a pvr_queue_fence object if the fence is
174 * backed by a UFO.
175 * @f: The dma_fence to turn into a pvr_queue_fence.
176 *
177 * Return:
178 * * A non-NULL pvr_queue_fence object if the dma_fence is backed by a UFO, or
179 * * NULL otherwise.
180 */
181static struct pvr_queue_fence *
182to_pvr_queue_job_fence(struct dma_fence *f)
183{
184 struct drm_sched_fence *sched_fence = to_drm_sched_fence(f);
185
186 if (sched_fence)
187 f = sched_fence->parent;
188
189 if (f && f->ops == &pvr_queue_job_fence_ops)
190 return container_of(f, struct pvr_queue_fence, base);
191
192 return NULL;
193}
194
195static const struct dma_fence_ops pvr_queue_cccb_fence_ops = {
196 .get_driver_name = pvr_queue_fence_get_driver_name,
197 .get_timeline_name = pvr_queue_cccb_fence_get_timeline_name,
198 .release = pvr_queue_fence_release,
199};
200
201/**
202 * pvr_queue_fence_put() - Put wrapper for pvr_queue_fence objects.
203 * @f: The dma_fence object to put.
204 *
205 * If the pvr_queue_fence has been initialized, we call dma_fence_put(),
206 * otherwise we free the object with dma_fence_free(). This allows us
207 * to do the right thing before and after pvr_queue_fence_init() had been
208 * called.
209 */
210static void pvr_queue_fence_put(struct dma_fence *f)
211{
212 if (!f)
213 return;
214
215 if (WARN_ON(f->ops &&
216 f->ops != &pvr_queue_cccb_fence_ops &&
217 f->ops != &pvr_queue_job_fence_ops))
218 return;
219
220 /* If the fence hasn't been initialized yet, free the object directly. */
221 if (f->ops)
222 dma_fence_put(f);
223 else
224 dma_fence_free(f);
225}
226
227/**
228 * pvr_queue_fence_alloc() - Allocate a pvr_queue_fence fence object
229 *
230 * Call this function to allocate job CCCB and done fences. This only
231 * allocates the objects. Initialization happens when the underlying
232 * dma_fence object is to be returned to drm_sched (in prepare_job() or
233 * run_job()).
234 *
235 * Return:
236 * * A valid pointer if the allocation succeeds, or
237 * * NULL if the allocation fails.
238 */
239static struct dma_fence *
240pvr_queue_fence_alloc(void)
241{
242 struct pvr_queue_fence *fence;
243
244 fence = kzalloc(sizeof(*fence), GFP_KERNEL);
245 if (!fence)
246 return NULL;
247
248 return &fence->base;
249}
250
251/**
252 * pvr_queue_fence_init() - Initializes a pvr_queue_fence object.
253 * @f: The fence to initialize
254 * @queue: The queue this fence belongs to.
255 * @fence_ops: The fence operations.
256 * @fence_ctx: The fence context.
257 *
258 * Wrapper around dma_fence_init() that takes care of initializing the
259 * pvr_queue_fence::queue field too.
260 */
261static void
262pvr_queue_fence_init(struct dma_fence *f,
263 struct pvr_queue *queue,
264 const struct dma_fence_ops *fence_ops,
265 struct pvr_queue_fence_ctx *fence_ctx)
266{
267 struct pvr_queue_fence *fence = container_of(f, struct pvr_queue_fence, base);
268
269 pvr_context_get(queue->ctx);
270 fence->queue = queue;
271 dma_fence_init(&fence->base, fence_ops,
272 &fence_ctx->lock, fence_ctx->id,
273 atomic_inc_return(&fence_ctx->seqno));
274}
275
276/**
277 * pvr_queue_cccb_fence_init() - Initializes a CCCB fence object.
278 * @fence: The fence to initialize.
279 * @queue: The queue this fence belongs to.
280 *
281 * Initializes a fence that can be used to wait for CCCB space.
282 *
283 * Should be called in the ::prepare_job() path, so the fence returned to
284 * drm_sched is valid.
285 */
286static void
287pvr_queue_cccb_fence_init(struct dma_fence *fence, struct pvr_queue *queue)
288{
289 pvr_queue_fence_init(fence, queue, &pvr_queue_cccb_fence_ops,
290 &queue->cccb_fence_ctx.base);
291}
292
293/**
294 * pvr_queue_job_fence_init() - Initializes a job done fence object.
295 * @fence: The fence to initialize.
296 * @queue: The queue this fence belongs to.
297 *
298 * Initializes a fence that will be signaled when the GPU is done executing
299 * a job.
300 *
301 * Should be called *before* the ::run_job() path, so the fence is initialised
302 * before being placed in the pending_list.
303 */
304static void
305pvr_queue_job_fence_init(struct dma_fence *fence, struct pvr_queue *queue)
306{
307 pvr_queue_fence_init(fence, queue, &pvr_queue_job_fence_ops,
308 &queue->job_fence_ctx);
309}
310
311/**
312 * pvr_queue_fence_ctx_init() - Queue fence context initialization.
313 * @fence_ctx: The context to initialize
314 */
315static void
316pvr_queue_fence_ctx_init(struct pvr_queue_fence_ctx *fence_ctx)
317{
318 spin_lock_init(&fence_ctx->lock);
319 fence_ctx->id = dma_fence_context_alloc(1);
320 atomic_set(&fence_ctx->seqno, 0);
321}
322
323static u32 ufo_cmds_size(u32 elem_count)
324{
325 /* We can pass at most ROGUE_FWIF_CCB_CMD_MAX_UFOS per UFO-related command. */
326 u32 full_cmd_count = elem_count / ROGUE_FWIF_CCB_CMD_MAX_UFOS;
327 u32 remaining_elems = elem_count % ROGUE_FWIF_CCB_CMD_MAX_UFOS;
328 u32 size = full_cmd_count *
329 pvr_cccb_get_size_of_cmd_with_hdr(ROGUE_FWIF_CCB_CMD_MAX_UFOS *
330 sizeof(struct rogue_fwif_ufo));
331
332 if (remaining_elems) {
333 size += pvr_cccb_get_size_of_cmd_with_hdr(remaining_elems *
334 sizeof(struct rogue_fwif_ufo));
335 }
336
337 return size;
338}
339
340static u32 job_cmds_size(struct pvr_job *job, u32 ufo_wait_count)
341{
342 /* One UFO cmd for the fence signaling, one UFO cmd per native fence native,
343 * and a command for the job itself.
344 */
345 return ufo_cmds_size(1) + ufo_cmds_size(ufo_wait_count) +
346 pvr_cccb_get_size_of_cmd_with_hdr(job->cmd_len);
347}
348
349/**
350 * job_count_remaining_native_deps() - Count the number of non-signaled native dependencies.
351 * @job: Job to operate on.
352 *
353 * Returns: Number of non-signaled native deps remaining.
354 */
355static unsigned long job_count_remaining_native_deps(struct pvr_job *job)
356{
357 unsigned long remaining_count = 0;
358 struct dma_fence *fence = NULL;
359 unsigned long index;
360
361 xa_for_each(&job->base.dependencies, index, fence) {
362 struct pvr_queue_fence *jfence;
363
364 jfence = to_pvr_queue_job_fence(fence);
365 if (!jfence)
366 continue;
367
368 if (!dma_fence_is_signaled(&jfence->base))
369 remaining_count++;
370 }
371
372 return remaining_count;
373}
374
375/**
376 * pvr_queue_get_job_cccb_fence() - Get the CCCB fence attached to a job.
377 * @queue: The queue this job will be submitted to.
378 * @job: The job to get the CCCB fence on.
379 *
380 * The CCCB fence is a synchronization primitive allowing us to delay job
381 * submission until there's enough space in the CCCB to submit the job.
382 *
383 * Return:
384 * * NULL if there's enough space in the CCCB to submit this job, or
385 * * A valid dma_fence object otherwise.
386 */
387static struct dma_fence *
388pvr_queue_get_job_cccb_fence(struct pvr_queue *queue, struct pvr_job *job)
389{
390 struct pvr_queue_fence *cccb_fence;
391 unsigned int native_deps_remaining;
392
393 /* If the fence is NULL, that means we already checked that we had
394 * enough space in the cccb for our job.
395 */
396 if (!job->cccb_fence)
397 return NULL;
398
399 mutex_lock(&queue->cccb_fence_ctx.job_lock);
400
401 /* Count remaining native dependencies and check if the job fits in the CCCB. */
402 native_deps_remaining = job_count_remaining_native_deps(job);
403 if (pvr_cccb_cmdseq_fits(&queue->cccb, job_cmds_size(job, native_deps_remaining))) {
404 pvr_queue_fence_put(job->cccb_fence);
405 job->cccb_fence = NULL;
406 goto out_unlock;
407 }
408
409 /* There should be no job attached to the CCCB fence context:
410 * drm_sched_entity guarantees that jobs are submitted one at a time.
411 */
412 if (WARN_ON(queue->cccb_fence_ctx.job))
413 pvr_job_put(queue->cccb_fence_ctx.job);
414
415 queue->cccb_fence_ctx.job = pvr_job_get(job);
416
417 /* Initialize the fence before returning it. */
418 cccb_fence = container_of(job->cccb_fence, struct pvr_queue_fence, base);
419 if (!WARN_ON(cccb_fence->queue))
420 pvr_queue_cccb_fence_init(job->cccb_fence, queue);
421
422out_unlock:
423 mutex_unlock(&queue->cccb_fence_ctx.job_lock);
424
425 return dma_fence_get(job->cccb_fence);
426}
427
428/**
429 * pvr_queue_get_job_kccb_fence() - Get the KCCB fence attached to a job.
430 * @queue: The queue this job will be submitted to.
431 * @job: The job to get the KCCB fence on.
432 *
433 * The KCCB fence is a synchronization primitive allowing us to delay job
434 * submission until there's enough space in the KCCB to submit the job.
435 *
436 * Return:
437 * * NULL if there's enough space in the KCCB to submit this job, or
438 * * A valid dma_fence object otherwise.
439 */
440static struct dma_fence *
441pvr_queue_get_job_kccb_fence(struct pvr_queue *queue, struct pvr_job *job)
442{
443 struct pvr_device *pvr_dev = queue->ctx->pvr_dev;
444 struct dma_fence *kccb_fence = NULL;
445
446 /* If the fence is NULL, that means we already checked that we had
447 * enough space in the KCCB for our job.
448 */
449 if (!job->kccb_fence)
450 return NULL;
451
452 if (!WARN_ON(job->kccb_fence->ops)) {
453 kccb_fence = pvr_kccb_reserve_slot(pvr_dev, job->kccb_fence);
454 job->kccb_fence = NULL;
455 }
456
457 return kccb_fence;
458}
459
460static struct dma_fence *
461pvr_queue_get_paired_frag_job_dep(struct pvr_queue *queue, struct pvr_job *job)
462{
463 struct pvr_job *frag_job = job->type == DRM_PVR_JOB_TYPE_GEOMETRY ?
464 job->paired_job : NULL;
465 struct dma_fence *f;
466 unsigned long index;
467
468 if (!frag_job)
469 return NULL;
470
471 xa_for_each(&frag_job->base.dependencies, index, f) {
472 /* Skip already signaled fences. */
473 if (dma_fence_is_signaled(f))
474 continue;
475
476 /* Skip our own fence. */
477 if (f == &job->base.s_fence->scheduled)
478 continue;
479
480 return dma_fence_get(f);
481 }
482
483 return frag_job->base.sched->ops->prepare_job(&frag_job->base, &queue->entity);
484}
485
486/**
487 * pvr_queue_prepare_job() - Return the next internal dependencies expressed as a dma_fence.
488 * @sched_job: The job to query the next internal dependency on
489 * @s_entity: The entity this job is queue on.
490 *
491 * After iterating over drm_sched_job::dependencies, drm_sched let the driver return
492 * its own internal dependencies. We use this function to return our internal dependencies.
493 */
494static struct dma_fence *
495pvr_queue_prepare_job(struct drm_sched_job *sched_job,
496 struct drm_sched_entity *s_entity)
497{
498 struct pvr_job *job = container_of(sched_job, struct pvr_job, base);
499 struct pvr_queue *queue = container_of(s_entity, struct pvr_queue, entity);
500 struct dma_fence *internal_dep = NULL;
501
502 /*
503 * Initialize the done_fence, so we can signal it. This must be done
504 * here because otherwise by the time of run_job() the job will end up
505 * in the pending list without a valid fence.
506 */
507 if (job->type == DRM_PVR_JOB_TYPE_FRAGMENT && job->paired_job) {
508 /*
509 * This will be called on a paired fragment job after being
510 * submitted to firmware. We can tell if this is the case and
511 * bail early from whether run_job() has been called on the
512 * geometry job, which would issue a pm ref.
513 */
514 if (job->paired_job->has_pm_ref)
515 return NULL;
516
517 /*
518 * In this case we need to use the job's own ctx to initialise
519 * the done_fence. The other steps are done in the ctx of the
520 * paired geometry job.
521 */
522 pvr_queue_job_fence_init(job->done_fence,
523 job->ctx->queues.fragment);
524 } else {
525 pvr_queue_job_fence_init(job->done_fence, queue);
526 }
527
528 /* CCCB fence is used to make sure we have enough space in the CCCB to
529 * submit our commands.
530 */
531 internal_dep = pvr_queue_get_job_cccb_fence(queue, job);
532
533 /* KCCB fence is used to make sure we have a KCCB slot to queue our
534 * CMD_KICK.
535 */
536 if (!internal_dep)
537 internal_dep = pvr_queue_get_job_kccb_fence(queue, job);
538
539 /* Any extra internal dependency should be added here, using the following
540 * pattern:
541 *
542 * if (!internal_dep)
543 * internal_dep = pvr_queue_get_job_xxxx_fence(queue, job);
544 */
545
546 /* The paired job fence should come last, when everything else is ready. */
547 if (!internal_dep)
548 internal_dep = pvr_queue_get_paired_frag_job_dep(queue, job);
549
550 return internal_dep;
551}
552
553/**
554 * pvr_queue_update_active_state_locked() - Update the queue active state.
555 * @queue: Queue to update the state on.
556 *
557 * Locked version of pvr_queue_update_active_state(). Must be called with
558 * pvr_device::queue::lock held.
559 */
560static void pvr_queue_update_active_state_locked(struct pvr_queue *queue)
561{
562 struct pvr_device *pvr_dev = queue->ctx->pvr_dev;
563
564 lockdep_assert_held(&pvr_dev->queues.lock);
565
566 /* The queue is temporary out of any list when it's being reset,
567 * we don't want a call to pvr_queue_update_active_state_locked()
568 * to re-insert it behind our back.
569 */
570 if (list_empty(&queue->node))
571 return;
572
573 if (!atomic_read(&queue->in_flight_job_count))
574 list_move_tail(&queue->node, &pvr_dev->queues.idle);
575 else
576 list_move_tail(&queue->node, &pvr_dev->queues.active);
577}
578
579/**
580 * pvr_queue_update_active_state() - Update the queue active state.
581 * @queue: Queue to update the state on.
582 *
583 * Active state is based on the in_flight_job_count value.
584 *
585 * Updating the active state implies moving the queue in or out of the
586 * active queue list, which also defines whether the queue is checked
587 * or not when a FW event is received.
588 *
589 * This function should be called any time a job is submitted or it done
590 * fence is signaled.
591 */
592static void pvr_queue_update_active_state(struct pvr_queue *queue)
593{
594 struct pvr_device *pvr_dev = queue->ctx->pvr_dev;
595
596 mutex_lock(&pvr_dev->queues.lock);
597 pvr_queue_update_active_state_locked(queue);
598 mutex_unlock(&pvr_dev->queues.lock);
599}
600
601static void pvr_queue_submit_job_to_cccb(struct pvr_job *job)
602{
603 struct pvr_queue *queue = container_of(job->base.sched, struct pvr_queue, scheduler);
604 struct rogue_fwif_ufo ufos[ROGUE_FWIF_CCB_CMD_MAX_UFOS];
605 struct pvr_cccb *cccb = &queue->cccb;
606 struct pvr_queue_fence *jfence;
607 struct dma_fence *fence;
608 unsigned long index;
609 u32 ufo_count = 0;
610
611 /* We need to add the queue to the active list before updating the CCCB,
612 * otherwise we might miss the FW event informing us that something
613 * happened on this queue.
614 */
615 atomic_inc(&queue->in_flight_job_count);
616 pvr_queue_update_active_state(queue);
617
618 xa_for_each(&job->base.dependencies, index, fence) {
619 jfence = to_pvr_queue_job_fence(fence);
620 if (!jfence)
621 continue;
622
623 /* Skip the partial render fence, we will place it at the end. */
624 if (job->type == DRM_PVR_JOB_TYPE_FRAGMENT && job->paired_job &&
625 &job->paired_job->base.s_fence->scheduled == fence)
626 continue;
627
628 if (dma_fence_is_signaled(&jfence->base))
629 continue;
630
631 pvr_fw_object_get_fw_addr(jfence->queue->timeline_ufo.fw_obj,
632 &ufos[ufo_count].addr);
633 ufos[ufo_count++].value = jfence->base.seqno;
634
635 if (ufo_count == ARRAY_SIZE(ufos)) {
636 pvr_cccb_write_command_with_header(cccb, ROGUE_FWIF_CCB_CMD_TYPE_FENCE_PR,
637 sizeof(ufos), ufos, 0, 0);
638 ufo_count = 0;
639 }
640 }
641
642 /* Partial render fence goes last. */
643 if (job->type == DRM_PVR_JOB_TYPE_FRAGMENT && job->paired_job) {
644 jfence = to_pvr_queue_job_fence(job->paired_job->done_fence);
645 if (!WARN_ON(!jfence)) {
646 pvr_fw_object_get_fw_addr(jfence->queue->timeline_ufo.fw_obj,
647 &ufos[ufo_count].addr);
648 ufos[ufo_count++].value = job->paired_job->done_fence->seqno;
649 }
650 }
651
652 if (ufo_count) {
653 pvr_cccb_write_command_with_header(cccb, ROGUE_FWIF_CCB_CMD_TYPE_FENCE_PR,
654 sizeof(ufos[0]) * ufo_count, ufos, 0, 0);
655 }
656
657 if (job->type == DRM_PVR_JOB_TYPE_GEOMETRY && job->paired_job) {
658 struct rogue_fwif_cmd_geom *cmd = job->cmd;
659
660 /* Reference value for the partial render test is the current queue fence
661 * seqno minus one.
662 */
663 pvr_fw_object_get_fw_addr(queue->timeline_ufo.fw_obj,
664 &cmd->partial_render_geom_frag_fence.addr);
665 cmd->partial_render_geom_frag_fence.value = job->done_fence->seqno - 1;
666 }
667
668 /* Submit job to FW */
669 pvr_cccb_write_command_with_header(cccb, job->fw_ccb_cmd_type, job->cmd_len, job->cmd,
670 job->id, job->id);
671
672 /* Signal the job fence. */
673 pvr_fw_object_get_fw_addr(queue->timeline_ufo.fw_obj, &ufos[0].addr);
674 ufos[0].value = job->done_fence->seqno;
675 pvr_cccb_write_command_with_header(cccb, ROGUE_FWIF_CCB_CMD_TYPE_UPDATE,
676 sizeof(ufos[0]), ufos, 0, 0);
677}
678
679/**
680 * pvr_queue_run_job() - Submit a job to the FW.
681 * @sched_job: The job to submit.
682 *
683 * This function is called when all non-native dependencies have been met and
684 * when the commands resulting from this job are guaranteed to fit in the CCCB.
685 */
686static struct dma_fence *pvr_queue_run_job(struct drm_sched_job *sched_job)
687{
688 struct pvr_job *job = container_of(sched_job, struct pvr_job, base);
689 struct pvr_device *pvr_dev = job->pvr_dev;
690 int err;
691
692 /* The fragment job is issued along the geometry job when we use combined
693 * geom+frag kicks. When we get there, we should simply return the
694 * done_fence that's been initialized earlier.
695 */
696 if (job->paired_job && job->type == DRM_PVR_JOB_TYPE_FRAGMENT &&
697 job->done_fence->ops) {
698 return dma_fence_get(job->done_fence);
699 }
700
701 /* The only kind of jobs that can be paired are geometry and fragment, and
702 * we bail out early if we see a fragment job that's paired with a geomtry
703 * job.
704 * Paired jobs must also target the same context and point to the same
705 * HWRT.
706 */
707 if (WARN_ON(job->paired_job &&
708 (job->type != DRM_PVR_JOB_TYPE_GEOMETRY ||
709 job->paired_job->type != DRM_PVR_JOB_TYPE_FRAGMENT ||
710 job->hwrt != job->paired_job->hwrt ||
711 job->ctx != job->paired_job->ctx)))
712 return ERR_PTR(-EINVAL);
713
714 err = pvr_job_get_pm_ref(job);
715 if (WARN_ON(err))
716 return ERR_PTR(err);
717
718 if (job->paired_job) {
719 err = pvr_job_get_pm_ref(job->paired_job);
720 if (WARN_ON(err))
721 return ERR_PTR(err);
722 }
723
724 /* Submit our job to the CCCB */
725 pvr_queue_submit_job_to_cccb(job);
726
727 if (job->paired_job) {
728 struct pvr_job *geom_job = job;
729 struct pvr_job *frag_job = job->paired_job;
730 struct pvr_queue *geom_queue = job->ctx->queues.geometry;
731 struct pvr_queue *frag_queue = job->ctx->queues.fragment;
732
733 /* Submit the fragment job along the geometry job and send a combined kick. */
734 pvr_queue_submit_job_to_cccb(frag_job);
735 pvr_cccb_send_kccb_combined_kick(pvr_dev,
736 &geom_queue->cccb, &frag_queue->cccb,
737 pvr_context_get_fw_addr(geom_job->ctx) +
738 geom_queue->ctx_offset,
739 pvr_context_get_fw_addr(frag_job->ctx) +
740 frag_queue->ctx_offset,
741 job->hwrt,
742 frag_job->fw_ccb_cmd_type ==
743 ROGUE_FWIF_CCB_CMD_TYPE_FRAG_PR);
744 } else {
745 struct pvr_queue *queue = container_of(job->base.sched,
746 struct pvr_queue, scheduler);
747
748 pvr_cccb_send_kccb_kick(pvr_dev, &queue->cccb,
749 pvr_context_get_fw_addr(job->ctx) + queue->ctx_offset,
750 job->hwrt);
751 }
752
753 return dma_fence_get(job->done_fence);
754}
755
756static void pvr_queue_stop(struct pvr_queue *queue, struct pvr_job *bad_job)
757{
758 drm_sched_stop(&queue->scheduler, bad_job ? &bad_job->base : NULL);
759}
760
761static void pvr_queue_start(struct pvr_queue *queue)
762{
763 struct pvr_job *job;
764
765 /* Make sure we CPU-signal the UFO object, so other queues don't get
766 * blocked waiting on it.
767 */
768 *queue->timeline_ufo.value = atomic_read(&queue->job_fence_ctx.seqno);
769
770 list_for_each_entry(job, &queue->scheduler.pending_list, base.list) {
771 if (dma_fence_is_signaled(job->done_fence)) {
772 /* Jobs might have completed after drm_sched_stop() was called.
773 * In that case, re-assign the parent field to the done_fence.
774 */
775 WARN_ON(job->base.s_fence->parent);
776 job->base.s_fence->parent = dma_fence_get(job->done_fence);
777 } else {
778 /* If we had unfinished jobs, flag the entity as guilty so no
779 * new job can be submitted.
780 */
781 atomic_set(&queue->ctx->faulty, 1);
782 }
783 }
784
785 drm_sched_start(&queue->scheduler, true);
786}
787
788/**
789 * pvr_queue_timedout_job() - Handle a job timeout event.
790 * @s_job: The job this timeout occurred on.
791 *
792 * FIXME: We don't do anything here to unblock the situation, we just stop+start
793 * the scheduler, and re-assign parent fences in the middle.
794 *
795 * Return:
796 * * DRM_GPU_SCHED_STAT_NOMINAL.
797 */
798static enum drm_gpu_sched_stat
799pvr_queue_timedout_job(struct drm_sched_job *s_job)
800{
801 struct drm_gpu_scheduler *sched = s_job->sched;
802 struct pvr_queue *queue = container_of(sched, struct pvr_queue, scheduler);
803 struct pvr_device *pvr_dev = queue->ctx->pvr_dev;
804 struct pvr_job *job;
805 u32 job_count = 0;
806
807 dev_err(sched->dev, "Job timeout\n");
808
809 /* Before we stop the scheduler, make sure the queue is out of any list, so
810 * any call to pvr_queue_update_active_state_locked() that might happen
811 * until the scheduler is really stopped doesn't end up re-inserting the
812 * queue in the active list. This would cause
813 * pvr_queue_signal_done_fences() and drm_sched_stop() to race with each
814 * other when accessing the pending_list, since drm_sched_stop() doesn't
815 * grab the job_list_lock when modifying the list (it's assuming the
816 * only other accessor is the scheduler, and it's safe to not grab the
817 * lock since it's stopped).
818 */
819 mutex_lock(&pvr_dev->queues.lock);
820 list_del_init(&queue->node);
821 mutex_unlock(&pvr_dev->queues.lock);
822
823 drm_sched_stop(sched, s_job);
824
825 /* Re-assign job parent fences. */
826 list_for_each_entry(job, &sched->pending_list, base.list) {
827 job->base.s_fence->parent = dma_fence_get(job->done_fence);
828 job_count++;
829 }
830 WARN_ON(atomic_read(&queue->in_flight_job_count) != job_count);
831
832 /* Re-insert the queue in the proper list, and kick a queue processing
833 * operation if there were jobs pending.
834 */
835 mutex_lock(&pvr_dev->queues.lock);
836 if (!job_count) {
837 list_move_tail(&queue->node, &pvr_dev->queues.idle);
838 } else {
839 atomic_set(&queue->in_flight_job_count, job_count);
840 list_move_tail(&queue->node, &pvr_dev->queues.active);
841 pvr_queue_process(queue);
842 }
843 mutex_unlock(&pvr_dev->queues.lock);
844
845 drm_sched_start(sched, true);
846
847 return DRM_GPU_SCHED_STAT_NOMINAL;
848}
849
850/**
851 * pvr_queue_free_job() - Release the reference the scheduler had on a job object.
852 * @sched_job: Job object to free.
853 */
854static void pvr_queue_free_job(struct drm_sched_job *sched_job)
855{
856 struct pvr_job *job = container_of(sched_job, struct pvr_job, base);
857
858 drm_sched_job_cleanup(sched_job);
859 job->paired_job = NULL;
860 pvr_job_put(job);
861}
862
863static const struct drm_sched_backend_ops pvr_queue_sched_ops = {
864 .prepare_job = pvr_queue_prepare_job,
865 .run_job = pvr_queue_run_job,
866 .timedout_job = pvr_queue_timedout_job,
867 .free_job = pvr_queue_free_job,
868};
869
870/**
871 * pvr_queue_fence_is_ufo_backed() - Check if a dma_fence is backed by a UFO object
872 * @f: Fence to test.
873 *
874 * A UFO-backed fence is a fence that can be signaled or waited upon FW-side.
875 * pvr_job::done_fence objects are backed by the timeline UFO attached to the queue
876 * they are pushed to, but those fences are not directly exposed to the outside
877 * world, so we also need to check if the fence we're being passed is a
878 * drm_sched_fence that was coming from our driver.
879 */
880bool pvr_queue_fence_is_ufo_backed(struct dma_fence *f)
881{
882 struct drm_sched_fence *sched_fence = f ? to_drm_sched_fence(f) : NULL;
883
884 if (sched_fence &&
885 sched_fence->sched->ops == &pvr_queue_sched_ops)
886 return true;
887
888 if (f && f->ops == &pvr_queue_job_fence_ops)
889 return true;
890
891 return false;
892}
893
894/**
895 * pvr_queue_signal_done_fences() - Signal done fences.
896 * @queue: Queue to check.
897 *
898 * Signal done fences of jobs whose seqno is less than the current value of
899 * the UFO object attached to the queue.
900 */
901static void
902pvr_queue_signal_done_fences(struct pvr_queue *queue)
903{
904 struct pvr_job *job, *tmp_job;
905 u32 cur_seqno;
906
907 spin_lock(&queue->scheduler.job_list_lock);
908 cur_seqno = *queue->timeline_ufo.value;
909 list_for_each_entry_safe(job, tmp_job, &queue->scheduler.pending_list, base.list) {
910 if ((int)(cur_seqno - lower_32_bits(job->done_fence->seqno)) < 0)
911 break;
912
913 if (!dma_fence_is_signaled(job->done_fence)) {
914 dma_fence_signal(job->done_fence);
915 pvr_job_release_pm_ref(job);
916 atomic_dec(&queue->in_flight_job_count);
917 }
918 }
919 spin_unlock(&queue->scheduler.job_list_lock);
920}
921
922/**
923 * pvr_queue_check_job_waiting_for_cccb_space() - Check if the job waiting for CCCB space
924 * can be unblocked
925 * pushed to the CCCB
926 * @queue: Queue to check
927 *
928 * If we have a job waiting for CCCB, and this job now fits in the CCCB, we signal
929 * its CCCB fence, which should kick drm_sched.
930 */
931static void
932pvr_queue_check_job_waiting_for_cccb_space(struct pvr_queue *queue)
933{
934 struct pvr_queue_fence *cccb_fence;
935 u32 native_deps_remaining;
936 struct pvr_job *job;
937
938 mutex_lock(&queue->cccb_fence_ctx.job_lock);
939 job = queue->cccb_fence_ctx.job;
940 if (!job)
941 goto out_unlock;
942
943 /* If we have a job attached to the CCCB fence context, its CCCB fence
944 * shouldn't be NULL.
945 */
946 if (WARN_ON(!job->cccb_fence)) {
947 job = NULL;
948 goto out_unlock;
949 }
950
951 /* If we get there, CCCB fence has to be initialized. */
952 cccb_fence = container_of(job->cccb_fence, struct pvr_queue_fence, base);
953 if (WARN_ON(!cccb_fence->queue)) {
954 job = NULL;
955 goto out_unlock;
956 }
957
958 /* Evict signaled dependencies before checking for CCCB space.
959 * If the job fits, signal the CCCB fence, this should unblock
960 * the drm_sched_entity.
961 */
962 native_deps_remaining = job_count_remaining_native_deps(job);
963 if (!pvr_cccb_cmdseq_fits(&queue->cccb, job_cmds_size(job, native_deps_remaining))) {
964 job = NULL;
965 goto out_unlock;
966 }
967
968 dma_fence_signal(job->cccb_fence);
969 pvr_queue_fence_put(job->cccb_fence);
970 job->cccb_fence = NULL;
971 queue->cccb_fence_ctx.job = NULL;
972
973out_unlock:
974 mutex_unlock(&queue->cccb_fence_ctx.job_lock);
975
976 pvr_job_put(job);
977}
978
979/**
980 * pvr_queue_process() - Process events that happened on a queue.
981 * @queue: Queue to check
982 *
983 * Signal job fences and check if jobs waiting for CCCB space can be unblocked.
984 */
985void pvr_queue_process(struct pvr_queue *queue)
986{
987 lockdep_assert_held(&queue->ctx->pvr_dev->queues.lock);
988
989 pvr_queue_check_job_waiting_for_cccb_space(queue);
990 pvr_queue_signal_done_fences(queue);
991 pvr_queue_update_active_state_locked(queue);
992}
993
994static u32 get_dm_type(struct pvr_queue *queue)
995{
996 switch (queue->type) {
997 case DRM_PVR_JOB_TYPE_GEOMETRY:
998 return PVR_FWIF_DM_GEOM;
999 case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
1000 case DRM_PVR_JOB_TYPE_FRAGMENT:
1001 return PVR_FWIF_DM_FRAG;
1002 case DRM_PVR_JOB_TYPE_COMPUTE:
1003 return PVR_FWIF_DM_CDM;
1004 }
1005
1006 return ~0;
1007}
1008
1009/**
1010 * init_fw_context() - Initializes the queue part of a FW context.
1011 * @queue: Queue object to initialize the FW context for.
1012 * @fw_ctx_map: The FW context CPU mapping.
1013 *
1014 * FW contexts are containing various states, one of them being a per-queue state
1015 * that needs to be initialized for each queue being exposed by a context. This
1016 * function takes care of that.
1017 */
1018static void init_fw_context(struct pvr_queue *queue, void *fw_ctx_map)
1019{
1020 struct pvr_context *ctx = queue->ctx;
1021 struct pvr_fw_object *fw_mem_ctx_obj = pvr_vm_get_fw_mem_context(ctx->vm_ctx);
1022 struct rogue_fwif_fwcommoncontext *cctx_fw;
1023 struct pvr_cccb *cccb = &queue->cccb;
1024
1025 cctx_fw = fw_ctx_map + queue->ctx_offset;
1026 cctx_fw->ccbctl_fw_addr = cccb->ctrl_fw_addr;
1027 cctx_fw->ccb_fw_addr = cccb->cccb_fw_addr;
1028
1029 cctx_fw->dm = get_dm_type(queue);
1030 cctx_fw->priority = ctx->priority;
1031 cctx_fw->priority_seq_num = 0;
1032 cctx_fw->max_deadline_ms = MAX_DEADLINE_MS;
1033 cctx_fw->pid = task_tgid_nr(current);
1034 cctx_fw->server_common_context_id = ctx->ctx_id;
1035
1036 pvr_fw_object_get_fw_addr(fw_mem_ctx_obj, &cctx_fw->fw_mem_context_fw_addr);
1037
1038 pvr_fw_object_get_fw_addr(queue->reg_state_obj, &cctx_fw->context_state_addr);
1039}
1040
1041/**
1042 * pvr_queue_cleanup_fw_context() - Wait for the FW context to be idle and clean it up.
1043 * @queue: Queue on FW context to clean up.
1044 *
1045 * Return:
1046 * * 0 on success,
1047 * * Any error returned by pvr_fw_structure_cleanup() otherwise.
1048 */
1049static int pvr_queue_cleanup_fw_context(struct pvr_queue *queue)
1050{
1051 if (!queue->ctx->fw_obj)
1052 return 0;
1053
1054 return pvr_fw_structure_cleanup(queue->ctx->pvr_dev,
1055 ROGUE_FWIF_CLEANUP_FWCOMMONCONTEXT,
1056 queue->ctx->fw_obj, queue->ctx_offset);
1057}
1058
1059/**
1060 * pvr_queue_job_init() - Initialize queue related fields in a pvr_job object.
1061 * @job: The job to initialize.
1062 *
1063 * Bind the job to a queue and allocate memory to guarantee pvr_queue_job_arm()
1064 * and pvr_queue_job_push() can't fail. We also make sure the context type is
1065 * valid and the job can fit in the CCCB.
1066 *
1067 * Return:
1068 * * 0 on success, or
1069 * * An error code if something failed.
1070 */
1071int pvr_queue_job_init(struct pvr_job *job)
1072{
1073 /* Fragment jobs need at least one native fence wait on the geometry job fence. */
1074 u32 min_native_dep_count = job->type == DRM_PVR_JOB_TYPE_FRAGMENT ? 1 : 0;
1075 struct pvr_queue *queue;
1076 int err;
1077
1078 if (atomic_read(&job->ctx->faulty))
1079 return -EIO;
1080
1081 queue = pvr_context_get_queue_for_job(job->ctx, job->type);
1082 if (!queue)
1083 return -EINVAL;
1084
1085 if (!pvr_cccb_cmdseq_can_fit(&queue->cccb, job_cmds_size(job, min_native_dep_count)))
1086 return -E2BIG;
1087
1088 err = drm_sched_job_init(&job->base, &queue->entity, 1, THIS_MODULE);
1089 if (err)
1090 return err;
1091
1092 job->cccb_fence = pvr_queue_fence_alloc();
1093 job->kccb_fence = pvr_kccb_fence_alloc();
1094 job->done_fence = pvr_queue_fence_alloc();
1095 if (!job->cccb_fence || !job->kccb_fence || !job->done_fence)
1096 return -ENOMEM;
1097
1098 return 0;
1099}
1100
1101/**
1102 * pvr_queue_job_arm() - Arm a job object.
1103 * @job: The job to arm.
1104 *
1105 * Initializes fences and return the drm_sched finished fence so it can
1106 * be exposed to the outside world. Once this function is called, you should
1107 * make sure the job is pushed using pvr_queue_job_push(), or guarantee that
1108 * no one grabbed a reference to the returned fence. The latter can happen if
1109 * we do multi-job submission, and something failed when creating/initializing
1110 * a job. In that case, we know the fence didn't leave the driver, and we
1111 * can thus guarantee nobody will wait on an dead fence object.
1112 *
1113 * Return:
1114 * * A dma_fence object.
1115 */
1116struct dma_fence *pvr_queue_job_arm(struct pvr_job *job)
1117{
1118 drm_sched_job_arm(&job->base);
1119
1120 return &job->base.s_fence->finished;
1121}
1122
1123/**
1124 * pvr_queue_job_cleanup() - Cleanup fence/scheduler related fields in the job object.
1125 * @job: The job to cleanup.
1126 *
1127 * Should be called in the job release path.
1128 */
1129void pvr_queue_job_cleanup(struct pvr_job *job)
1130{
1131 pvr_queue_fence_put(job->done_fence);
1132 pvr_queue_fence_put(job->cccb_fence);
1133 pvr_kccb_fence_put(job->kccb_fence);
1134
1135 if (job->base.s_fence)
1136 drm_sched_job_cleanup(&job->base);
1137}
1138
1139/**
1140 * pvr_queue_job_push() - Push a job to its queue.
1141 * @job: The job to push.
1142 *
1143 * Must be called after pvr_queue_job_init() and after all dependencies
1144 * have been added to the job. This will effectively queue the job to
1145 * the drm_sched_entity attached to the queue. We grab a reference on
1146 * the job object, so the caller is free to drop its reference when it's
1147 * done accessing the job object.
1148 */
1149void pvr_queue_job_push(struct pvr_job *job)
1150{
1151 struct pvr_queue *queue = container_of(job->base.sched, struct pvr_queue, scheduler);
1152
1153 /* Keep track of the last queued job scheduled fence for combined submit. */
1154 dma_fence_put(queue->last_queued_job_scheduled_fence);
1155 queue->last_queued_job_scheduled_fence = dma_fence_get(&job->base.s_fence->scheduled);
1156
1157 pvr_job_get(job);
1158 drm_sched_entity_push_job(&job->base);
1159}
1160
1161static void reg_state_init(void *cpu_ptr, void *priv)
1162{
1163 struct pvr_queue *queue = priv;
1164
1165 if (queue->type == DRM_PVR_JOB_TYPE_GEOMETRY) {
1166 struct rogue_fwif_geom_ctx_state *geom_ctx_state_fw = cpu_ptr;
1167
1168 geom_ctx_state_fw->geom_core[0].geom_reg_vdm_call_stack_pointer_init =
1169 queue->callstack_addr;
1170 }
1171}
1172
1173/**
1174 * pvr_queue_create() - Create a queue object.
1175 * @ctx: The context this queue will be attached to.
1176 * @type: The type of jobs being pushed to this queue.
1177 * @args: The arguments passed to the context creation function.
1178 * @fw_ctx_map: CPU mapping of the FW context object.
1179 *
1180 * Create a queue object that will be used to queue and track jobs.
1181 *
1182 * Return:
1183 * * A valid pointer to a pvr_queue object, or
1184 * * An error pointer if the creation/initialization failed.
1185 */
1186struct pvr_queue *pvr_queue_create(struct pvr_context *ctx,
1187 enum drm_pvr_job_type type,
1188 struct drm_pvr_ioctl_create_context_args *args,
1189 void *fw_ctx_map)
1190{
1191 static const struct {
1192 u32 cccb_size;
1193 const char *name;
1194 } props[] = {
1195 [DRM_PVR_JOB_TYPE_GEOMETRY] = {
1196 .cccb_size = CTX_GEOM_CCCB_SIZE_LOG2,
1197 .name = "geometry",
1198 },
1199 [DRM_PVR_JOB_TYPE_FRAGMENT] = {
1200 .cccb_size = CTX_FRAG_CCCB_SIZE_LOG2,
1201 .name = "fragment"
1202 },
1203 [DRM_PVR_JOB_TYPE_COMPUTE] = {
1204 .cccb_size = CTX_COMPUTE_CCCB_SIZE_LOG2,
1205 .name = "compute"
1206 },
1207 [DRM_PVR_JOB_TYPE_TRANSFER_FRAG] = {
1208 .cccb_size = CTX_TRANSFER_CCCB_SIZE_LOG2,
1209 .name = "transfer_frag"
1210 },
1211 };
1212 struct pvr_device *pvr_dev = ctx->pvr_dev;
1213 struct drm_gpu_scheduler *sched;
1214 struct pvr_queue *queue;
1215 int ctx_state_size, err;
1216 void *cpu_map;
1217
1218 if (WARN_ON(type >= sizeof(props)))
1219 return ERR_PTR(-EINVAL);
1220
1221 switch (ctx->type) {
1222 case DRM_PVR_CTX_TYPE_RENDER:
1223 if (type != DRM_PVR_JOB_TYPE_GEOMETRY &&
1224 type != DRM_PVR_JOB_TYPE_FRAGMENT)
1225 return ERR_PTR(-EINVAL);
1226 break;
1227 case DRM_PVR_CTX_TYPE_COMPUTE:
1228 if (type != DRM_PVR_JOB_TYPE_COMPUTE)
1229 return ERR_PTR(-EINVAL);
1230 break;
1231 case DRM_PVR_CTX_TYPE_TRANSFER_FRAG:
1232 if (type != DRM_PVR_JOB_TYPE_TRANSFER_FRAG)
1233 return ERR_PTR(-EINVAL);
1234 break;
1235 default:
1236 return ERR_PTR(-EINVAL);
1237 }
1238
1239 ctx_state_size = get_ctx_state_size(pvr_dev, type);
1240 if (ctx_state_size < 0)
1241 return ERR_PTR(ctx_state_size);
1242
1243 queue = kzalloc(sizeof(*queue), GFP_KERNEL);
1244 if (!queue)
1245 return ERR_PTR(-ENOMEM);
1246
1247 queue->type = type;
1248 queue->ctx_offset = get_ctx_offset(type);
1249 queue->ctx = ctx;
1250 queue->callstack_addr = args->callstack_addr;
1251 sched = &queue->scheduler;
1252 INIT_LIST_HEAD(&queue->node);
1253 mutex_init(&queue->cccb_fence_ctx.job_lock);
1254 pvr_queue_fence_ctx_init(&queue->cccb_fence_ctx.base);
1255 pvr_queue_fence_ctx_init(&queue->job_fence_ctx);
1256
1257 err = pvr_cccb_init(pvr_dev, &queue->cccb, props[type].cccb_size, props[type].name);
1258 if (err)
1259 goto err_free_queue;
1260
1261 err = pvr_fw_object_create(pvr_dev, ctx_state_size,
1262 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
1263 reg_state_init, queue, &queue->reg_state_obj);
1264 if (err)
1265 goto err_cccb_fini;
1266
1267 init_fw_context(queue, fw_ctx_map);
1268
1269 if (type != DRM_PVR_JOB_TYPE_GEOMETRY && type != DRM_PVR_JOB_TYPE_FRAGMENT &&
1270 args->callstack_addr) {
1271 err = -EINVAL;
1272 goto err_release_reg_state;
1273 }
1274
1275 cpu_map = pvr_fw_object_create_and_map(pvr_dev, sizeof(*queue->timeline_ufo.value),
1276 PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
1277 NULL, NULL, &queue->timeline_ufo.fw_obj);
1278 if (IS_ERR(cpu_map)) {
1279 err = PTR_ERR(cpu_map);
1280 goto err_release_reg_state;
1281 }
1282
1283 queue->timeline_ufo.value = cpu_map;
1284
1285 err = drm_sched_init(&queue->scheduler,
1286 &pvr_queue_sched_ops,
1287 pvr_dev->sched_wq, 1, 64 * 1024, 1,
1288 msecs_to_jiffies(500),
1289 pvr_dev->sched_wq, NULL, "pvr-queue",
1290 pvr_dev->base.dev);
1291 if (err)
1292 goto err_release_ufo;
1293
1294 err = drm_sched_entity_init(&queue->entity,
1295 DRM_SCHED_PRIORITY_MIN,
1296 &sched, 1, &ctx->faulty);
1297 if (err)
1298 goto err_sched_fini;
1299
1300 mutex_lock(&pvr_dev->queues.lock);
1301 list_add_tail(&queue->node, &pvr_dev->queues.idle);
1302 mutex_unlock(&pvr_dev->queues.lock);
1303
1304 return queue;
1305
1306err_sched_fini:
1307 drm_sched_fini(&queue->scheduler);
1308
1309err_release_ufo:
1310 pvr_fw_object_unmap_and_destroy(queue->timeline_ufo.fw_obj);
1311
1312err_release_reg_state:
1313 pvr_fw_object_destroy(queue->reg_state_obj);
1314
1315err_cccb_fini:
1316 pvr_cccb_fini(&queue->cccb);
1317
1318err_free_queue:
1319 mutex_destroy(&queue->cccb_fence_ctx.job_lock);
1320 kfree(queue);
1321
1322 return ERR_PTR(err);
1323}
1324
1325void pvr_queue_device_pre_reset(struct pvr_device *pvr_dev)
1326{
1327 struct pvr_queue *queue;
1328
1329 mutex_lock(&pvr_dev->queues.lock);
1330 list_for_each_entry(queue, &pvr_dev->queues.idle, node)
1331 pvr_queue_stop(queue, NULL);
1332 list_for_each_entry(queue, &pvr_dev->queues.active, node)
1333 pvr_queue_stop(queue, NULL);
1334 mutex_unlock(&pvr_dev->queues.lock);
1335}
1336
1337void pvr_queue_device_post_reset(struct pvr_device *pvr_dev)
1338{
1339 struct pvr_queue *queue;
1340
1341 mutex_lock(&pvr_dev->queues.lock);
1342 list_for_each_entry(queue, &pvr_dev->queues.active, node)
1343 pvr_queue_start(queue);
1344 list_for_each_entry(queue, &pvr_dev->queues.idle, node)
1345 pvr_queue_start(queue);
1346 mutex_unlock(&pvr_dev->queues.lock);
1347}
1348
1349/**
1350 * pvr_queue_kill() - Kill a queue.
1351 * @queue: The queue to kill.
1352 *
1353 * Kill the queue so no new jobs can be pushed. Should be called when the
1354 * context handle is destroyed. The queue object might last longer if jobs
1355 * are still in flight and holding a reference to the context this queue
1356 * belongs to.
1357 */
1358void pvr_queue_kill(struct pvr_queue *queue)
1359{
1360 drm_sched_entity_destroy(&queue->entity);
1361 dma_fence_put(queue->last_queued_job_scheduled_fence);
1362 queue->last_queued_job_scheduled_fence = NULL;
1363}
1364
1365/**
1366 * pvr_queue_destroy() - Destroy a queue.
1367 * @queue: The queue to destroy.
1368 *
1369 * Cleanup the queue and free the resources attached to it. Should be
1370 * called from the context release function.
1371 */
1372void pvr_queue_destroy(struct pvr_queue *queue)
1373{
1374 if (!queue)
1375 return;
1376
1377 mutex_lock(&queue->ctx->pvr_dev->queues.lock);
1378 list_del_init(&queue->node);
1379 mutex_unlock(&queue->ctx->pvr_dev->queues.lock);
1380
1381 drm_sched_fini(&queue->scheduler);
1382 drm_sched_entity_fini(&queue->entity);
1383
1384 if (WARN_ON(queue->last_queued_job_scheduled_fence))
1385 dma_fence_put(queue->last_queued_job_scheduled_fence);
1386
1387 pvr_queue_cleanup_fw_context(queue);
1388
1389 pvr_fw_object_unmap_and_destroy(queue->timeline_ufo.fw_obj);
1390 pvr_fw_object_destroy(queue->reg_state_obj);
1391 pvr_cccb_fini(&queue->cccb);
1392 mutex_destroy(&queue->cccb_fence_ctx.job_lock);
1393 kfree(queue);
1394}
1395
1396/**
1397 * pvr_queue_device_init() - Device-level initialization of queue related fields.
1398 * @pvr_dev: The device to initialize.
1399 *
1400 * Initializes all fields related to queue management in pvr_device.
1401 *
1402 * Return:
1403 * * 0 on success, or
1404 * * An error code on failure.
1405 */
1406int pvr_queue_device_init(struct pvr_device *pvr_dev)
1407{
1408 int err;
1409
1410 INIT_LIST_HEAD(&pvr_dev->queues.active);
1411 INIT_LIST_HEAD(&pvr_dev->queues.idle);
1412 err = drmm_mutex_init(from_pvr_device(pvr_dev), &pvr_dev->queues.lock);
1413 if (err)
1414 return err;
1415
1416 pvr_dev->sched_wq = alloc_workqueue("powervr-sched", WQ_UNBOUND, 0);
1417 if (!pvr_dev->sched_wq)
1418 return -ENOMEM;
1419
1420 return 0;
1421}
1422
1423/**
1424 * pvr_queue_device_fini() - Device-level cleanup of queue related fields.
1425 * @pvr_dev: The device to cleanup.
1426 *
1427 * Cleanup/free all queue-related resources attached to a pvr_device object.
1428 */
1429void pvr_queue_device_fini(struct pvr_device *pvr_dev)
1430{
1431 destroy_workqueue(pvr_dev->sched_wq);
1432}
drivers/gpu/drm/imagination/pvr_queue.h+169−0
@@ -0,0 +1,169 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4#ifndef PVR_QUEUE_H
5#define PVR_QUEUE_H
6
7#include <drm/gpu_scheduler.h>
8
9#include "pvr_cccb.h"
10#include "pvr_device.h"
11
12struct pvr_context;
13struct pvr_queue;
14
15/**
16 * struct pvr_queue_fence_ctx - Queue fence context
17 *
18 * Used to implement dma_fence_ops for pvr_job::{done,cccb}_fence.
19 */
20struct pvr_queue_fence_ctx {
21 /** @id: Fence context ID allocated with dma_fence_context_alloc(). */
22 u64 id;
23
24 /** @seqno: Sequence number incremented each time a fence is created. */
25 atomic_t seqno;
26
27 /** @lock: Lock used to synchronize access to fences allocated by this context. */
28 spinlock_t lock;
29};
30
31/**
32 * struct pvr_queue_cccb_fence_ctx - CCCB fence context
33 *
34 * Context used to manage fences controlling access to the CCCB. No fences are
35 * issued if there's enough space in the CCCB to push job commands.
36 */
37struct pvr_queue_cccb_fence_ctx {
38 /** @base: Base queue fence context. */
39 struct pvr_queue_fence_ctx base;
40
41 /**
42 * @job: Job waiting for CCCB space.
43 *
44 * Thanks to the serializationg done at the drm_sched_entity level,
45 * there's no more than one job waiting for CCCB at a given time.
46 *
47 * This field is NULL if no jobs are currently waiting for CCCB space.
48 *
49 * Must be accessed with @job_lock held.
50 */
51 struct pvr_job *job;
52
53 /** @lock: Lock protecting access to the job object. */
54 struct mutex job_lock;
55};
56
57/**
58 * struct pvr_queue_fence - Queue fence object
59 */
60struct pvr_queue_fence {
61 /** @base: Base dma_fence. */
62 struct dma_fence base;
63
64 /** @queue: Queue that created this fence. */
65 struct pvr_queue *queue;
66};
67
68/**
69 * struct pvr_queue - Job queue
70 *
71 * Used to queue and track execution of pvr_job objects.
72 */
73struct pvr_queue {
74 /** @scheduler: Single entity scheduler use to push jobs to this queue. */
75 struct drm_gpu_scheduler scheduler;
76
77 /** @entity: Scheduling entity backing this queue. */
78 struct drm_sched_entity entity;
79
80 /** @type: Type of jobs queued to this queue. */
81 enum drm_pvr_job_type type;
82
83 /** @ctx: Context object this queue is bound to. */
84 struct pvr_context *ctx;
85
86 /** @node: Used to add the queue to the active/idle queue list. */
87 struct list_head node;
88
89 /**
90 * @in_flight_job_count: Number of jobs submitted to the CCCB that
91 * have not been processed yet.
92 */
93 atomic_t in_flight_job_count;
94
95 /**
96 * @cccb_fence_ctx: CCCB fence context.
97 *
98 * Used to control access to the CCCB is full, such that we don't
99 * end up trying to push commands to the CCCB if there's not enough
100 * space to receive all commands needed for a job to complete.
101 */
102 struct pvr_queue_cccb_fence_ctx cccb_fence_ctx;
103
104 /** @job_fence_ctx: Job fence context object. */
105 struct pvr_queue_fence_ctx job_fence_ctx;
106
107 /** @timeline_ufo: Timeline UFO for the context queue. */
108 struct {
109 /** @fw_obj: FW object representing the UFO value. */
110 struct pvr_fw_object *fw_obj;
111
112 /** @value: CPU mapping of the UFO value. */
113 u32 *value;
114 } timeline_ufo;
115
116 /**
117 * last_queued_job_scheduled_fence: The scheduled fence of the last
118 * job queued to this queue.
119 *
120 * We use it to insert frag -> geom dependencies when issuing combined
121 * geom+frag jobs, to guarantee that the fragment job that's part of
122 * the combined operation comes after all fragment jobs that were queued
123 * before it.
124 */
125 struct dma_fence *last_queued_job_scheduled_fence;
126
127 /** @cccb: Client Circular Command Buffer. */
128 struct pvr_cccb cccb;
129
130 /** @reg_state_obj: FW object representing the register state of this queue. */
131 struct pvr_fw_object *reg_state_obj;
132
133 /** @ctx_offset: Offset of the queue context in the FW context object. */
134 u32 ctx_offset;
135
136 /** @callstack_addr: Initial call stack address for register state object. */
137 u64 callstack_addr;
138};
139
140bool pvr_queue_fence_is_ufo_backed(struct dma_fence *f);
141
142int pvr_queue_job_init(struct pvr_job *job);
143
144void pvr_queue_job_cleanup(struct pvr_job *job);
145
146void pvr_queue_job_push(struct pvr_job *job);
147
148struct dma_fence *pvr_queue_job_arm(struct pvr_job *job);
149
150struct pvr_queue *pvr_queue_create(struct pvr_context *ctx,
151 enum drm_pvr_job_type type,
152 struct drm_pvr_ioctl_create_context_args *args,
153 void *fw_ctx_map);
154
155void pvr_queue_kill(struct pvr_queue *queue);
156
157void pvr_queue_destroy(struct pvr_queue *queue);
158
159void pvr_queue_process(struct pvr_queue *queue);
160
161void pvr_queue_device_pre_reset(struct pvr_device *pvr_dev);
162
163void pvr_queue_device_post_reset(struct pvr_device *pvr_dev);
164
165int pvr_queue_device_init(struct pvr_device *pvr_dev);
166
167void pvr_queue_device_fini(struct pvr_device *pvr_dev);
168
169#endif /* PVR_QUEUE_H */
drivers/gpu/drm/imagination/pvr_rogue_cr_defs.h+6193−0
@@ -0,0 +1,6193 @@
1/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2/* Copyright (c) 2023 Imagination Technologies Ltd. */
3
4/* *** Autogenerated C -- do not edit *** */
5
6#ifndef PVR_ROGUE_CR_DEFS_H
7#define PVR_ROGUE_CR_DEFS_H
8
9/* clang-format off */
10
11#define ROGUE_CR_DEFS_REVISION 1
12
13/* Register ROGUE_CR_RASTERISATION_INDIRECT */
14#define ROGUE_CR_RASTERISATION_INDIRECT 0x8238U
15#define ROGUE_CR_RASTERISATION_INDIRECT_MASKFULL 0x000000000000000FULL
16#define ROGUE_CR_RASTERISATION_INDIRECT_ADDRESS_SHIFT 0U
17#define ROGUE_CR_RASTERISATION_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF0U
18
19/* Register ROGUE_CR_PBE_INDIRECT */
20#define ROGUE_CR_PBE_INDIRECT 0x83E0U
21#define ROGUE_CR_PBE_INDIRECT_MASKFULL 0x000000000000000FULL
22#define ROGUE_CR_PBE_INDIRECT_ADDRESS_SHIFT 0U
23#define ROGUE_CR_PBE_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF0U
24
25/* Register ROGUE_CR_PBE_PERF_INDIRECT */
26#define ROGUE_CR_PBE_PERF_INDIRECT 0x83D8U
27#define ROGUE_CR_PBE_PERF_INDIRECT_MASKFULL 0x000000000000000FULL
28#define ROGUE_CR_PBE_PERF_INDIRECT_ADDRESS_SHIFT 0U
29#define ROGUE_CR_PBE_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF0U
30
31/* Register ROGUE_CR_TPU_PERF_INDIRECT */
32#define ROGUE_CR_TPU_PERF_INDIRECT 0x83F0U
33#define ROGUE_CR_TPU_PERF_INDIRECT_MASKFULL 0x0000000000000007ULL
34#define ROGUE_CR_TPU_PERF_INDIRECT_ADDRESS_SHIFT 0U
35#define ROGUE_CR_TPU_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF8U
36
37/* Register ROGUE_CR_RASTERISATION_PERF_INDIRECT */
38#define ROGUE_CR_RASTERISATION_PERF_INDIRECT 0x8318U
39#define ROGUE_CR_RASTERISATION_PERF_INDIRECT_MASKFULL 0x000000000000000FULL
40#define ROGUE_CR_RASTERISATION_PERF_INDIRECT_ADDRESS_SHIFT 0U
41#define ROGUE_CR_RASTERISATION_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF0U
42
43/* Register ROGUE_CR_TPU_MCU_L0_PERF_INDIRECT */
44#define ROGUE_CR_TPU_MCU_L0_PERF_INDIRECT 0x8028U
45#define ROGUE_CR_TPU_MCU_L0_PERF_INDIRECT_MASKFULL 0x0000000000000007ULL
46#define ROGUE_CR_TPU_MCU_L0_PERF_INDIRECT_ADDRESS_SHIFT 0U
47#define ROGUE_CR_TPU_MCU_L0_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF8U
48
49/* Register ROGUE_CR_USC_PERF_INDIRECT */
50#define ROGUE_CR_USC_PERF_INDIRECT 0x8030U
51#define ROGUE_CR_USC_PERF_INDIRECT_MASKFULL 0x000000000000000FULL
52#define ROGUE_CR_USC_PERF_INDIRECT_ADDRESS_SHIFT 0U
53#define ROGUE_CR_USC_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF0U
54
55/* Register ROGUE_CR_BLACKPEARL_INDIRECT */
56#define ROGUE_CR_BLACKPEARL_INDIRECT 0x8388U
57#define ROGUE_CR_BLACKPEARL_INDIRECT_MASKFULL 0x0000000000000003ULL
58#define ROGUE_CR_BLACKPEARL_INDIRECT_ADDRESS_SHIFT 0U
59#define ROGUE_CR_BLACKPEARL_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFFCU
60
61/* Register ROGUE_CR_BLACKPEARL_PERF_INDIRECT */
62#define ROGUE_CR_BLACKPEARL_PERF_INDIRECT 0x83F8U
63#define ROGUE_CR_BLACKPEARL_PERF_INDIRECT_MASKFULL 0x0000000000000003ULL
64#define ROGUE_CR_BLACKPEARL_PERF_INDIRECT_ADDRESS_SHIFT 0U
65#define ROGUE_CR_BLACKPEARL_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFFCU
66
67/* Register ROGUE_CR_TEXAS3_PERF_INDIRECT */
68#define ROGUE_CR_TEXAS3_PERF_INDIRECT 0x83D0U
69#define ROGUE_CR_TEXAS3_PERF_INDIRECT_MASKFULL 0x0000000000000007ULL
70#define ROGUE_CR_TEXAS3_PERF_INDIRECT_ADDRESS_SHIFT 0U
71#define ROGUE_CR_TEXAS3_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF8U
72
73/* Register ROGUE_CR_TEXAS_PERF_INDIRECT */
74#define ROGUE_CR_TEXAS_PERF_INDIRECT 0x8288U
75#define ROGUE_CR_TEXAS_PERF_INDIRECT_MASKFULL 0x0000000000000003ULL
76#define ROGUE_CR_TEXAS_PERF_INDIRECT_ADDRESS_SHIFT 0U
77#define ROGUE_CR_TEXAS_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFFCU
78
79/* Register ROGUE_CR_BX_TU_PERF_INDIRECT */
80#define ROGUE_CR_BX_TU_PERF_INDIRECT 0xC900U
81#define ROGUE_CR_BX_TU_PERF_INDIRECT_MASKFULL 0x0000000000000003ULL
82#define ROGUE_CR_BX_TU_PERF_INDIRECT_ADDRESS_SHIFT 0U
83#define ROGUE_CR_BX_TU_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFFCU
84
85/* Register ROGUE_CR_CLK_CTRL */
86#define ROGUE_CR_CLK_CTRL 0x0000U
87#define ROGUE_CR_CLK_CTRL__PBE2_XE__MASKFULL 0xFFFFFF003F3FFFFFULL
88#define ROGUE_CR_CLK_CTRL__S7_TOP__MASKFULL 0xCFCF03000F3F3F0FULL
89#define ROGUE_CR_CLK_CTRL_MASKFULL 0xFFFFFF003F3FFFFFULL
90#define ROGUE_CR_CLK_CTRL_BIF_TEXAS_SHIFT 62U
91#define ROGUE_CR_CLK_CTRL_BIF_TEXAS_CLRMSK 0x3FFFFFFFFFFFFFFFULL
92#define ROGUE_CR_CLK_CTRL_BIF_TEXAS_OFF 0x0000000000000000ULL
93#define ROGUE_CR_CLK_CTRL_BIF_TEXAS_ON 0x4000000000000000ULL
94#define ROGUE_CR_CLK_CTRL_BIF_TEXAS_AUTO 0x8000000000000000ULL
95#define ROGUE_CR_CLK_CTRL_IPP_SHIFT 60U
96#define ROGUE_CR_CLK_CTRL_IPP_CLRMSK 0xCFFFFFFFFFFFFFFFULL
97#define ROGUE_CR_CLK_CTRL_IPP_OFF 0x0000000000000000ULL
98#define ROGUE_CR_CLK_CTRL_IPP_ON 0x1000000000000000ULL
99#define ROGUE_CR_CLK_CTRL_IPP_AUTO 0x2000000000000000ULL
100#define ROGUE_CR_CLK_CTRL_FBC_SHIFT 58U
101#define ROGUE_CR_CLK_CTRL_FBC_CLRMSK 0xF3FFFFFFFFFFFFFFULL
102#define ROGUE_CR_CLK_CTRL_FBC_OFF 0x0000000000000000ULL
103#define ROGUE_CR_CLK_CTRL_FBC_ON 0x0400000000000000ULL
104#define ROGUE_CR_CLK_CTRL_FBC_AUTO 0x0800000000000000ULL
105#define ROGUE_CR_CLK_CTRL_FBDC_SHIFT 56U
106#define ROGUE_CR_CLK_CTRL_FBDC_CLRMSK 0xFCFFFFFFFFFFFFFFULL
107#define ROGUE_CR_CLK_CTRL_FBDC_OFF 0x0000000000000000ULL
108#define ROGUE_CR_CLK_CTRL_FBDC_ON 0x0100000000000000ULL
109#define ROGUE_CR_CLK_CTRL_FBDC_AUTO 0x0200000000000000ULL
110#define ROGUE_CR_CLK_CTRL_FB_TLCACHE_SHIFT 54U
111#define ROGUE_CR_CLK_CTRL_FB_TLCACHE_CLRMSK 0xFF3FFFFFFFFFFFFFULL
112#define ROGUE_CR_CLK_CTRL_FB_TLCACHE_OFF 0x0000000000000000ULL
113#define ROGUE_CR_CLK_CTRL_FB_TLCACHE_ON 0x0040000000000000ULL
114#define ROGUE_CR_CLK_CTRL_FB_TLCACHE_AUTO 0x0080000000000000ULL
115#define ROGUE_CR_CLK_CTRL_USCS_SHIFT 52U
116#define ROGUE_CR_CLK_CTRL_USCS_CLRMSK 0xFFCFFFFFFFFFFFFFULL
117#define ROGUE_CR_CLK_CTRL_USCS_OFF 0x0000000000000000ULL
118#define ROGUE_CR_CLK_CTRL_USCS_ON 0x0010000000000000ULL
119#define ROGUE_CR_CLK_CTRL_USCS_AUTO 0x0020000000000000ULL
120#define ROGUE_CR_CLK_CTRL_PBE_SHIFT 50U
121#define ROGUE_CR_CLK_CTRL_PBE_CLRMSK 0xFFF3FFFFFFFFFFFFULL
122#define ROGUE_CR_CLK_CTRL_PBE_OFF 0x0000000000000000ULL
123#define ROGUE_CR_CLK_CTRL_PBE_ON 0x0004000000000000ULL
124#define ROGUE_CR_CLK_CTRL_PBE_AUTO 0x0008000000000000ULL
125#define ROGUE_CR_CLK_CTRL_MCU_L1_SHIFT 48U
126#define ROGUE_CR_CLK_CTRL_MCU_L1_CLRMSK 0xFFFCFFFFFFFFFFFFULL
127#define ROGUE_CR_CLK_CTRL_MCU_L1_OFF 0x0000000000000000ULL
128#define ROGUE_CR_CLK_CTRL_MCU_L1_ON 0x0001000000000000ULL
129#define ROGUE_CR_CLK_CTRL_MCU_L1_AUTO 0x0002000000000000ULL
130#define ROGUE_CR_CLK_CTRL_CDM_SHIFT 46U
131#define ROGUE_CR_CLK_CTRL_CDM_CLRMSK 0xFFFF3FFFFFFFFFFFULL
132#define ROGUE_CR_CLK_CTRL_CDM_OFF 0x0000000000000000ULL
133#define ROGUE_CR_CLK_CTRL_CDM_ON 0x0000400000000000ULL
134#define ROGUE_CR_CLK_CTRL_CDM_AUTO 0x0000800000000000ULL
135#define ROGUE_CR_CLK_CTRL_SIDEKICK_SHIFT 44U
136#define ROGUE_CR_CLK_CTRL_SIDEKICK_CLRMSK 0xFFFFCFFFFFFFFFFFULL
137#define ROGUE_CR_CLK_CTRL_SIDEKICK_OFF 0x0000000000000000ULL
138#define ROGUE_CR_CLK_CTRL_SIDEKICK_ON 0x0000100000000000ULL
139#define ROGUE_CR_CLK_CTRL_SIDEKICK_AUTO 0x0000200000000000ULL
140#define ROGUE_CR_CLK_CTRL_BIF_SIDEKICK_SHIFT 42U
141#define ROGUE_CR_CLK_CTRL_BIF_SIDEKICK_CLRMSK 0xFFFFF3FFFFFFFFFFULL
142#define ROGUE_CR_CLK_CTRL_BIF_SIDEKICK_OFF 0x0000000000000000ULL
143#define ROGUE_CR_CLK_CTRL_BIF_SIDEKICK_ON 0x0000040000000000ULL
144#define ROGUE_CR_CLK_CTRL_BIF_SIDEKICK_AUTO 0x0000080000000000ULL
145#define ROGUE_CR_CLK_CTRL_BIF_SHIFT 40U
146#define ROGUE_CR_CLK_CTRL_BIF_CLRMSK 0xFFFFFCFFFFFFFFFFULL
147#define ROGUE_CR_CLK_CTRL_BIF_OFF 0x0000000000000000ULL
148#define ROGUE_CR_CLK_CTRL_BIF_ON 0x0000010000000000ULL
149#define ROGUE_CR_CLK_CTRL_BIF_AUTO 0x0000020000000000ULL
150#define ROGUE_CR_CLK_CTRL_TPU_MCU_DEMUX_SHIFT 28U
151#define ROGUE_CR_CLK_CTRL_TPU_MCU_DEMUX_CLRMSK 0xFFFFFFFFCFFFFFFFULL
152#define ROGUE_CR_CLK_CTRL_TPU_MCU_DEMUX_OFF 0x0000000000000000ULL
153#define ROGUE_CR_CLK_CTRL_TPU_MCU_DEMUX_ON 0x0000000010000000ULL
154#define ROGUE_CR_CLK_CTRL_TPU_MCU_DEMUX_AUTO 0x0000000020000000ULL
155#define ROGUE_CR_CLK_CTRL_MCU_L0_SHIFT 26U
156#define ROGUE_CR_CLK_CTRL_MCU_L0_CLRMSK 0xFFFFFFFFF3FFFFFFULL
157#define ROGUE_CR_CLK_CTRL_MCU_L0_OFF 0x0000000000000000ULL
158#define ROGUE_CR_CLK_CTRL_MCU_L0_ON 0x0000000004000000ULL
159#define ROGUE_CR_CLK_CTRL_MCU_L0_AUTO 0x0000000008000000ULL
160#define ROGUE_CR_CLK_CTRL_TPU_SHIFT 24U
161#define ROGUE_CR_CLK_CTRL_TPU_CLRMSK 0xFFFFFFFFFCFFFFFFULL
162#define ROGUE_CR_CLK_CTRL_TPU_OFF 0x0000000000000000ULL
163#define ROGUE_CR_CLK_CTRL_TPU_ON 0x0000000001000000ULL
164#define ROGUE_CR_CLK_CTRL_TPU_AUTO 0x0000000002000000ULL
165#define ROGUE_CR_CLK_CTRL_USC_SHIFT 20U
166#define ROGUE_CR_CLK_CTRL_USC_CLRMSK 0xFFFFFFFFFFCFFFFFULL
167#define ROGUE_CR_CLK_CTRL_USC_OFF 0x0000000000000000ULL
168#define ROGUE_CR_CLK_CTRL_USC_ON 0x0000000000100000ULL
169#define ROGUE_CR_CLK_CTRL_USC_AUTO 0x0000000000200000ULL
170#define ROGUE_CR_CLK_CTRL_TLA_SHIFT 18U
171#define ROGUE_CR_CLK_CTRL_TLA_CLRMSK 0xFFFFFFFFFFF3FFFFULL
172#define ROGUE_CR_CLK_CTRL_TLA_OFF 0x0000000000000000ULL
173#define ROGUE_CR_CLK_CTRL_TLA_ON 0x0000000000040000ULL
174#define ROGUE_CR_CLK_CTRL_TLA_AUTO 0x0000000000080000ULL
175#define ROGUE_CR_CLK_CTRL_SLC_SHIFT 16U
176#define ROGUE_CR_CLK_CTRL_SLC_CLRMSK 0xFFFFFFFFFFFCFFFFULL
177#define ROGUE_CR_CLK_CTRL_SLC_OFF 0x0000000000000000ULL
178#define ROGUE_CR_CLK_CTRL_SLC_ON 0x0000000000010000ULL
179#define ROGUE_CR_CLK_CTRL_SLC_AUTO 0x0000000000020000ULL
180#define ROGUE_CR_CLK_CTRL_UVS_SHIFT 14U
181#define ROGUE_CR_CLK_CTRL_UVS_CLRMSK 0xFFFFFFFFFFFF3FFFULL
182#define ROGUE_CR_CLK_CTRL_UVS_OFF 0x0000000000000000ULL
183#define ROGUE_CR_CLK_CTRL_UVS_ON 0x0000000000004000ULL
184#define ROGUE_CR_CLK_CTRL_UVS_AUTO 0x0000000000008000ULL
185#define ROGUE_CR_CLK_CTRL_PDS_SHIFT 12U
186#define ROGUE_CR_CLK_CTRL_PDS_CLRMSK 0xFFFFFFFFFFFFCFFFULL
187#define ROGUE_CR_CLK_CTRL_PDS_OFF 0x0000000000000000ULL
188#define ROGUE_CR_CLK_CTRL_PDS_ON 0x0000000000001000ULL
189#define ROGUE_CR_CLK_CTRL_PDS_AUTO 0x0000000000002000ULL
190#define ROGUE_CR_CLK_CTRL_VDM_SHIFT 10U
191#define ROGUE_CR_CLK_CTRL_VDM_CLRMSK 0xFFFFFFFFFFFFF3FFULL
192#define ROGUE_CR_CLK_CTRL_VDM_OFF 0x0000000000000000ULL
193#define ROGUE_CR_CLK_CTRL_VDM_ON 0x0000000000000400ULL
194#define ROGUE_CR_CLK_CTRL_VDM_AUTO 0x0000000000000800ULL
195#define ROGUE_CR_CLK_CTRL_PM_SHIFT 8U
196#define ROGUE_CR_CLK_CTRL_PM_CLRMSK 0xFFFFFFFFFFFFFCFFULL
197#define ROGUE_CR_CLK_CTRL_PM_OFF 0x0000000000000000ULL
198#define ROGUE_CR_CLK_CTRL_PM_ON 0x0000000000000100ULL
199#define ROGUE_CR_CLK_CTRL_PM_AUTO 0x0000000000000200ULL
200#define ROGUE_CR_CLK_CTRL_GPP_SHIFT 6U
201#define ROGUE_CR_CLK_CTRL_GPP_CLRMSK 0xFFFFFFFFFFFFFF3FULL
202#define ROGUE_CR_CLK_CTRL_GPP_OFF 0x0000000000000000ULL
203#define ROGUE_CR_CLK_CTRL_GPP_ON 0x0000000000000040ULL
204#define ROGUE_CR_CLK_CTRL_GPP_AUTO 0x0000000000000080ULL
205#define ROGUE_CR_CLK_CTRL_TE_SHIFT 4U
206#define ROGUE_CR_CLK_CTRL_TE_CLRMSK 0xFFFFFFFFFFFFFFCFULL
207#define ROGUE_CR_CLK_CTRL_TE_OFF 0x0000000000000000ULL
208#define ROGUE_CR_CLK_CTRL_TE_ON 0x0000000000000010ULL
209#define ROGUE_CR_CLK_CTRL_TE_AUTO 0x0000000000000020ULL
210#define ROGUE_CR_CLK_CTRL_TSP_SHIFT 2U
211#define ROGUE_CR_CLK_CTRL_TSP_CLRMSK 0xFFFFFFFFFFFFFFF3ULL
212#define ROGUE_CR_CLK_CTRL_TSP_OFF 0x0000000000000000ULL
213#define ROGUE_CR_CLK_CTRL_TSP_ON 0x0000000000000004ULL
214#define ROGUE_CR_CLK_CTRL_TSP_AUTO 0x0000000000000008ULL
215#define ROGUE_CR_CLK_CTRL_ISP_SHIFT 0U
216#define ROGUE_CR_CLK_CTRL_ISP_CLRMSK 0xFFFFFFFFFFFFFFFCULL
217#define ROGUE_CR_CLK_CTRL_ISP_OFF 0x0000000000000000ULL
218#define ROGUE_CR_CLK_CTRL_ISP_ON 0x0000000000000001ULL
219#define ROGUE_CR_CLK_CTRL_ISP_AUTO 0x0000000000000002ULL
220
221/* Register ROGUE_CR_CLK_STATUS */
222#define ROGUE_CR_CLK_STATUS 0x0008U
223#define ROGUE_CR_CLK_STATUS__PBE2_XE__MASKFULL 0x00000001FFF077FFULL
224#define ROGUE_CR_CLK_STATUS__S7_TOP__MASKFULL 0x00000001B3101773ULL
225#define ROGUE_CR_CLK_STATUS_MASKFULL 0x00000001FFF077FFULL
226#define ROGUE_CR_CLK_STATUS_MCU_FBTC_SHIFT 32U
227#define ROGUE_CR_CLK_STATUS_MCU_FBTC_CLRMSK 0xFFFFFFFEFFFFFFFFULL
228#define ROGUE_CR_CLK_STATUS_MCU_FBTC_GATED 0x0000000000000000ULL
229#define ROGUE_CR_CLK_STATUS_MCU_FBTC_RUNNING 0x0000000100000000ULL
230#define ROGUE_CR_CLK_STATUS_BIF_TEXAS_SHIFT 31U
231#define ROGUE_CR_CLK_STATUS_BIF_TEXAS_CLRMSK 0xFFFFFFFF7FFFFFFFULL
232#define ROGUE_CR_CLK_STATUS_BIF_TEXAS_GATED 0x0000000000000000ULL
233#define ROGUE_CR_CLK_STATUS_BIF_TEXAS_RUNNING 0x0000000080000000ULL
234#define ROGUE_CR_CLK_STATUS_IPP_SHIFT 30U
235#define ROGUE_CR_CLK_STATUS_IPP_CLRMSK 0xFFFFFFFFBFFFFFFFULL
236#define ROGUE_CR_CLK_STATUS_IPP_GATED 0x0000000000000000ULL
237#define ROGUE_CR_CLK_STATUS_IPP_RUNNING 0x0000000040000000ULL
238#define ROGUE_CR_CLK_STATUS_FBC_SHIFT 29U
239#define ROGUE_CR_CLK_STATUS_FBC_CLRMSK 0xFFFFFFFFDFFFFFFFULL
240#define ROGUE_CR_CLK_STATUS_FBC_GATED 0x0000000000000000ULL
241#define ROGUE_CR_CLK_STATUS_FBC_RUNNING 0x0000000020000000ULL
242#define ROGUE_CR_CLK_STATUS_FBDC_SHIFT 28U
243#define ROGUE_CR_CLK_STATUS_FBDC_CLRMSK 0xFFFFFFFFEFFFFFFFULL
244#define ROGUE_CR_CLK_STATUS_FBDC_GATED 0x0000000000000000ULL
245#define ROGUE_CR_CLK_STATUS_FBDC_RUNNING 0x0000000010000000ULL
246#define ROGUE_CR_CLK_STATUS_FB_TLCACHE_SHIFT 27U
247#define ROGUE_CR_CLK_STATUS_FB_TLCACHE_CLRMSK 0xFFFFFFFFF7FFFFFFULL
248#define ROGUE_CR_CLK_STATUS_FB_TLCACHE_GATED 0x0000000000000000ULL
249#define ROGUE_CR_CLK_STATUS_FB_TLCACHE_RUNNING 0x0000000008000000ULL
250#define ROGUE_CR_CLK_STATUS_USCS_SHIFT 26U
251#define ROGUE_CR_CLK_STATUS_USCS_CLRMSK 0xFFFFFFFFFBFFFFFFULL
252#define ROGUE_CR_CLK_STATUS_USCS_GATED 0x0000000000000000ULL
253#define ROGUE_CR_CLK_STATUS_USCS_RUNNING 0x0000000004000000ULL
254#define ROGUE_CR_CLK_STATUS_PBE_SHIFT 25U
255#define ROGUE_CR_CLK_STATUS_PBE_CLRMSK 0xFFFFFFFFFDFFFFFFULL
256#define ROGUE_CR_CLK_STATUS_PBE_GATED 0x0000000000000000ULL
257#define ROGUE_CR_CLK_STATUS_PBE_RUNNING 0x0000000002000000ULL
258#define ROGUE_CR_CLK_STATUS_MCU_L1_SHIFT 24U
259#define ROGUE_CR_CLK_STATUS_MCU_L1_CLRMSK 0xFFFFFFFFFEFFFFFFULL
260#define ROGUE_CR_CLK_STATUS_MCU_L1_GATED 0x0000000000000000ULL
261#define ROGUE_CR_CLK_STATUS_MCU_L1_RUNNING 0x0000000001000000ULL
262#define ROGUE_CR_CLK_STATUS_CDM_SHIFT 23U
263#define ROGUE_CR_CLK_STATUS_CDM_CLRMSK 0xFFFFFFFFFF7FFFFFULL
264#define ROGUE_CR_CLK_STATUS_CDM_GATED 0x0000000000000000ULL
265#define ROGUE_CR_CLK_STATUS_CDM_RUNNING 0x0000000000800000ULL
266#define ROGUE_CR_CLK_STATUS_SIDEKICK_SHIFT 22U
267#define ROGUE_CR_CLK_STATUS_SIDEKICK_CLRMSK 0xFFFFFFFFFFBFFFFFULL
268#define ROGUE_CR_CLK_STATUS_SIDEKICK_GATED 0x0000000000000000ULL
269#define ROGUE_CR_CLK_STATUS_SIDEKICK_RUNNING 0x0000000000400000ULL
270#define ROGUE_CR_CLK_STATUS_BIF_SIDEKICK_SHIFT 21U
271#define ROGUE_CR_CLK_STATUS_BIF_SIDEKICK_CLRMSK 0xFFFFFFFFFFDFFFFFULL
272#define ROGUE_CR_CLK_STATUS_BIF_SIDEKICK_GATED 0x0000000000000000ULL
273#define ROGUE_CR_CLK_STATUS_BIF_SIDEKICK_RUNNING 0x0000000000200000ULL
274#define ROGUE_CR_CLK_STATUS_BIF_SHIFT 20U
275#define ROGUE_CR_CLK_STATUS_BIF_CLRMSK 0xFFFFFFFFFFEFFFFFULL
276#define ROGUE_CR_CLK_STATUS_BIF_GATED 0x0000000000000000ULL
277#define ROGUE_CR_CLK_STATUS_BIF_RUNNING 0x0000000000100000ULL
278#define ROGUE_CR_CLK_STATUS_TPU_MCU_DEMUX_SHIFT 14U
279#define ROGUE_CR_CLK_STATUS_TPU_MCU_DEMUX_CLRMSK 0xFFFFFFFFFFFFBFFFULL
280#define ROGUE_CR_CLK_STATUS_TPU_MCU_DEMUX_GATED 0x0000000000000000ULL
281#define ROGUE_CR_CLK_STATUS_TPU_MCU_DEMUX_RUNNING 0x0000000000004000ULL
282#define ROGUE_CR_CLK_STATUS_MCU_L0_SHIFT 13U
283#define ROGUE_CR_CLK_STATUS_MCU_L0_CLRMSK 0xFFFFFFFFFFFFDFFFULL
284#define ROGUE_CR_CLK_STATUS_MCU_L0_GATED 0x0000000000000000ULL
285#define ROGUE_CR_CLK_STATUS_MCU_L0_RUNNING 0x0000000000002000ULL
286#define ROGUE_CR_CLK_STATUS_TPU_SHIFT 12U
287#define ROGUE_CR_CLK_STATUS_TPU_CLRMSK 0xFFFFFFFFFFFFEFFFULL
288#define ROGUE_CR_CLK_STATUS_TPU_GATED 0x0000000000000000ULL
289#define ROGUE_CR_CLK_STATUS_TPU_RUNNING 0x0000000000001000ULL
290#define ROGUE_CR_CLK_STATUS_USC_SHIFT 10U
291#define ROGUE_CR_CLK_STATUS_USC_CLRMSK 0xFFFFFFFFFFFFFBFFULL
292#define ROGUE_CR_CLK_STATUS_USC_GATED 0x0000000000000000ULL
293#define ROGUE_CR_CLK_STATUS_USC_RUNNING 0x0000000000000400ULL
294#define ROGUE_CR_CLK_STATUS_TLA_SHIFT 9U
295#define ROGUE_CR_CLK_STATUS_TLA_CLRMSK 0xFFFFFFFFFFFFFDFFULL
296#define ROGUE_CR_CLK_STATUS_TLA_GATED 0x0000000000000000ULL
297#define ROGUE_CR_CLK_STATUS_TLA_RUNNING 0x0000000000000200ULL
298#define ROGUE_CR_CLK_STATUS_SLC_SHIFT 8U
299#define ROGUE_CR_CLK_STATUS_SLC_CLRMSK 0xFFFFFFFFFFFFFEFFULL
300#define ROGUE_CR_CLK_STATUS_SLC_GATED 0x0000000000000000ULL
301#define ROGUE_CR_CLK_STATUS_SLC_RUNNING 0x0000000000000100ULL
302#define ROGUE_CR_CLK_STATUS_UVS_SHIFT 7U
303#define ROGUE_CR_CLK_STATUS_UVS_CLRMSK 0xFFFFFFFFFFFFFF7FULL
304#define ROGUE_CR_CLK_STATUS_UVS_GATED 0x0000000000000000ULL
305#define ROGUE_CR_CLK_STATUS_UVS_RUNNING 0x0000000000000080ULL
306#define ROGUE_CR_CLK_STATUS_PDS_SHIFT 6U
307#define ROGUE_CR_CLK_STATUS_PDS_CLRMSK 0xFFFFFFFFFFFFFFBFULL
308#define ROGUE_CR_CLK_STATUS_PDS_GATED 0x0000000000000000ULL
309#define ROGUE_CR_CLK_STATUS_PDS_RUNNING 0x0000000000000040ULL
310#define ROGUE_CR_CLK_STATUS_VDM_SHIFT 5U
311#define ROGUE_CR_CLK_STATUS_VDM_CLRMSK 0xFFFFFFFFFFFFFFDFULL
312#define ROGUE_CR_CLK_STATUS_VDM_GATED 0x0000000000000000ULL
313#define ROGUE_CR_CLK_STATUS_VDM_RUNNING 0x0000000000000020ULL
314#define ROGUE_CR_CLK_STATUS_PM_SHIFT 4U
315#define ROGUE_CR_CLK_STATUS_PM_CLRMSK 0xFFFFFFFFFFFFFFEFULL
316#define ROGUE_CR_CLK_STATUS_PM_GATED 0x0000000000000000ULL
317#define ROGUE_CR_CLK_STATUS_PM_RUNNING 0x0000000000000010ULL
318#define ROGUE_CR_CLK_STATUS_GPP_SHIFT 3U
319#define ROGUE_CR_CLK_STATUS_GPP_CLRMSK 0xFFFFFFFFFFFFFFF7ULL
320#define ROGUE_CR_CLK_STATUS_GPP_GATED 0x0000000000000000ULL
321#define ROGUE_CR_CLK_STATUS_GPP_RUNNING 0x0000000000000008ULL
322#define ROGUE_CR_CLK_STATUS_TE_SHIFT 2U
323#define ROGUE_CR_CLK_STATUS_TE_CLRMSK 0xFFFFFFFFFFFFFFFBULL
324#define ROGUE_CR_CLK_STATUS_TE_GATED 0x0000000000000000ULL
325#define ROGUE_CR_CLK_STATUS_TE_RUNNING 0x0000000000000004ULL
326#define ROGUE_CR_CLK_STATUS_TSP_SHIFT 1U
327#define ROGUE_CR_CLK_STATUS_TSP_CLRMSK 0xFFFFFFFFFFFFFFFDULL
328#define ROGUE_CR_CLK_STATUS_TSP_GATED 0x0000000000000000ULL
329#define ROGUE_CR_CLK_STATUS_TSP_RUNNING 0x0000000000000002ULL
330#define ROGUE_CR_CLK_STATUS_ISP_SHIFT 0U
331#define ROGUE_CR_CLK_STATUS_ISP_CLRMSK 0xFFFFFFFFFFFFFFFEULL
332#define ROGUE_CR_CLK_STATUS_ISP_GATED 0x0000000000000000ULL
333#define ROGUE_CR_CLK_STATUS_ISP_RUNNING 0x0000000000000001ULL
334
335/* Register ROGUE_CR_CORE_ID */
336#define ROGUE_CR_CORE_ID__PBVNC 0x0020U
337#define ROGUE_CR_CORE_ID__PBVNC__MASKFULL 0xFFFFFFFFFFFFFFFFULL
338#define ROGUE_CR_CORE_ID__PBVNC__BRANCH_ID_SHIFT 48U
339#define ROGUE_CR_CORE_ID__PBVNC__BRANCH_ID_CLRMSK 0x0000FFFFFFFFFFFFULL
340#define ROGUE_CR_CORE_ID__PBVNC__VERSION_ID_SHIFT 32U
341#define ROGUE_CR_CORE_ID__PBVNC__VERSION_ID_CLRMSK 0xFFFF0000FFFFFFFFULL
342#define ROGUE_CR_CORE_ID__PBVNC__NUMBER_OF_SCALABLE_UNITS_SHIFT 16U
343#define ROGUE_CR_CORE_ID__PBVNC__NUMBER_OF_SCALABLE_UNITS_CLRMSK 0xFFFFFFFF0000FFFFULL
344#define ROGUE_CR_CORE_ID__PBVNC__CONFIG_ID_SHIFT 0U
345#define ROGUE_CR_CORE_ID__PBVNC__CONFIG_ID_CLRMSK 0xFFFFFFFFFFFF0000ULL
346
347/* Register ROGUE_CR_CORE_ID */
348#define ROGUE_CR_CORE_ID 0x0018U
349#define ROGUE_CR_CORE_ID_MASKFULL 0x00000000FFFFFFFFULL
350#define ROGUE_CR_CORE_ID_ID_SHIFT 16U
351#define ROGUE_CR_CORE_ID_ID_CLRMSK 0x0000FFFFU
352#define ROGUE_CR_CORE_ID_CONFIG_SHIFT 0U
353#define ROGUE_CR_CORE_ID_CONFIG_CLRMSK 0xFFFF0000U
354
355/* Register ROGUE_CR_CORE_REVISION */
356#define ROGUE_CR_CORE_REVISION 0x0020U
357#define ROGUE_CR_CORE_REVISION_MASKFULL 0x00000000FFFFFFFFULL
358#define ROGUE_CR_CORE_REVISION_DESIGNER_SHIFT 24U
359#define ROGUE_CR_CORE_REVISION_DESIGNER_CLRMSK 0x00FFFFFFU
360#define ROGUE_CR_CORE_REVISION_MAJOR_SHIFT 16U
361#define ROGUE_CR_CORE_REVISION_MAJOR_CLRMSK 0xFF00FFFFU
362#define ROGUE_CR_CORE_REVISION_MINOR_SHIFT 8U
363#define ROGUE_CR_CORE_REVISION_MINOR_CLRMSK 0xFFFF00FFU
364#define ROGUE_CR_CORE_REVISION_MAINTENANCE_SHIFT 0U
365#define ROGUE_CR_CORE_REVISION_MAINTENANCE_CLRMSK 0xFFFFFF00U
366
367/* Register ROGUE_CR_DESIGNER_REV_FIELD1 */
368#define ROGUE_CR_DESIGNER_REV_FIELD1 0x0028U
369#define ROGUE_CR_DESIGNER_REV_FIELD1_MASKFULL 0x00000000FFFFFFFFULL
370#define ROGUE_CR_DESIGNER_REV_FIELD1_DESIGNER_REV_FIELD1_SHIFT 0U
371#define ROGUE_CR_DESIGNER_REV_FIELD1_DESIGNER_REV_FIELD1_CLRMSK 0x00000000U
372
373/* Register ROGUE_CR_DESIGNER_REV_FIELD2 */
374#define ROGUE_CR_DESIGNER_REV_FIELD2 0x0030U
375#define ROGUE_CR_DESIGNER_REV_FIELD2_MASKFULL 0x00000000FFFFFFFFULL
376#define ROGUE_CR_DESIGNER_REV_FIELD2_DESIGNER_REV_FIELD2_SHIFT 0U
377#define ROGUE_CR_DESIGNER_REV_FIELD2_DESIGNER_REV_FIELD2_CLRMSK 0x00000000U
378
379/* Register ROGUE_CR_CHANGESET_NUMBER */
380#define ROGUE_CR_CHANGESET_NUMBER 0x0040U
381#define ROGUE_CR_CHANGESET_NUMBER_MASKFULL 0xFFFFFFFFFFFFFFFFULL
382#define ROGUE_CR_CHANGESET_NUMBER_CHANGESET_NUMBER_SHIFT 0U
383#define ROGUE_CR_CHANGESET_NUMBER_CHANGESET_NUMBER_CLRMSK 0x0000000000000000ULL
384
385/* Register ROGUE_CR_CLK_XTPLUS_CTRL */
386#define ROGUE_CR_CLK_XTPLUS_CTRL 0x0080U
387#define ROGUE_CR_CLK_XTPLUS_CTRL_MASKFULL 0x0000003FFFFF0000ULL
388#define ROGUE_CR_CLK_XTPLUS_CTRL_TDM_SHIFT 36U
389#define ROGUE_CR_CLK_XTPLUS_CTRL_TDM_CLRMSK 0xFFFFFFCFFFFFFFFFULL
390#define ROGUE_CR_CLK_XTPLUS_CTRL_TDM_OFF 0x0000000000000000ULL
391#define ROGUE_CR_CLK_XTPLUS_CTRL_TDM_ON 0x0000001000000000ULL
392#define ROGUE_CR_CLK_XTPLUS_CTRL_TDM_AUTO 0x0000002000000000ULL
393#define ROGUE_CR_CLK_XTPLUS_CTRL_ASTC_SHIFT 34U
394#define ROGUE_CR_CLK_XTPLUS_CTRL_ASTC_CLRMSK 0xFFFFFFF3FFFFFFFFULL
395#define ROGUE_CR_CLK_XTPLUS_CTRL_ASTC_OFF 0x0000000000000000ULL
396#define ROGUE_CR_CLK_XTPLUS_CTRL_ASTC_ON 0x0000000400000000ULL
397#define ROGUE_CR_CLK_XTPLUS_CTRL_ASTC_AUTO 0x0000000800000000ULL
398#define ROGUE_CR_CLK_XTPLUS_CTRL_IPF_SHIFT 32U
399#define ROGUE_CR_CLK_XTPLUS_CTRL_IPF_CLRMSK 0xFFFFFFFCFFFFFFFFULL
400#define ROGUE_CR_CLK_XTPLUS_CTRL_IPF_OFF 0x0000000000000000ULL
401#define ROGUE_CR_CLK_XTPLUS_CTRL_IPF_ON 0x0000000100000000ULL
402#define ROGUE_CR_CLK_XTPLUS_CTRL_IPF_AUTO 0x0000000200000000ULL
403#define ROGUE_CR_CLK_XTPLUS_CTRL_COMPUTE_SHIFT 30U
404#define ROGUE_CR_CLK_XTPLUS_CTRL_COMPUTE_CLRMSK 0xFFFFFFFF3FFFFFFFULL
405#define ROGUE_CR_CLK_XTPLUS_CTRL_COMPUTE_OFF 0x0000000000000000ULL
406#define ROGUE_CR_CLK_XTPLUS_CTRL_COMPUTE_ON 0x0000000040000000ULL
407#define ROGUE_CR_CLK_XTPLUS_CTRL_COMPUTE_AUTO 0x0000000080000000ULL
408#define ROGUE_CR_CLK_XTPLUS_CTRL_PIXEL_SHIFT 28U
409#define ROGUE_CR_CLK_XTPLUS_CTRL_PIXEL_CLRMSK 0xFFFFFFFFCFFFFFFFULL
410#define ROGUE_CR_CLK_XTPLUS_CTRL_PIXEL_OFF 0x0000000000000000ULL
411#define ROGUE_CR_CLK_XTPLUS_CTRL_PIXEL_ON 0x0000000010000000ULL
412#define ROGUE_CR_CLK_XTPLUS_CTRL_PIXEL_AUTO 0x0000000020000000ULL
413#define ROGUE_CR_CLK_XTPLUS_CTRL_VERTEX_SHIFT 26U
414#define ROGUE_CR_CLK_XTPLUS_CTRL_VERTEX_CLRMSK 0xFFFFFFFFF3FFFFFFULL
415#define ROGUE_CR_CLK_XTPLUS_CTRL_VERTEX_OFF 0x0000000000000000ULL
416#define ROGUE_CR_CLK_XTPLUS_CTRL_VERTEX_ON 0x0000000004000000ULL
417#define ROGUE_CR_CLK_XTPLUS_CTRL_VERTEX_AUTO 0x0000000008000000ULL
418#define ROGUE_CR_CLK_XTPLUS_CTRL_USCPS_SHIFT 24U
419#define ROGUE_CR_CLK_XTPLUS_CTRL_USCPS_CLRMSK 0xFFFFFFFFFCFFFFFFULL
420#define ROGUE_CR_CLK_XTPLUS_CTRL_USCPS_OFF 0x0000000000000000ULL
421#define ROGUE_CR_CLK_XTPLUS_CTRL_USCPS_ON 0x0000000001000000ULL
422#define ROGUE_CR_CLK_XTPLUS_CTRL_USCPS_AUTO 0x0000000002000000ULL
423#define ROGUE_CR_CLK_XTPLUS_CTRL_PDS_SHARED_SHIFT 22U
424#define ROGUE_CR_CLK_XTPLUS_CTRL_PDS_SHARED_CLRMSK 0xFFFFFFFFFF3FFFFFULL
425#define ROGUE_CR_CLK_XTPLUS_CTRL_PDS_SHARED_OFF 0x0000000000000000ULL
426#define ROGUE_CR_CLK_XTPLUS_CTRL_PDS_SHARED_ON 0x0000000000400000ULL
427#define ROGUE_CR_CLK_XTPLUS_CTRL_PDS_SHARED_AUTO 0x0000000000800000ULL
428#define ROGUE_CR_CLK_XTPLUS_CTRL_BIF_BLACKPEARL_SHIFT 20U
429#define ROGUE_CR_CLK_XTPLUS_CTRL_BIF_BLACKPEARL_CLRMSK 0xFFFFFFFFFFCFFFFFULL
430#define ROGUE_CR_CLK_XTPLUS_CTRL_BIF_BLACKPEARL_OFF 0x0000000000000000ULL
431#define ROGUE_CR_CLK_XTPLUS_CTRL_BIF_BLACKPEARL_ON 0x0000000000100000ULL
432#define ROGUE_CR_CLK_XTPLUS_CTRL_BIF_BLACKPEARL_AUTO 0x0000000000200000ULL
433#define ROGUE_CR_CLK_XTPLUS_CTRL_USC_SHARED_SHIFT 18U
434#define ROGUE_CR_CLK_XTPLUS_CTRL_USC_SHARED_CLRMSK 0xFFFFFFFFFFF3FFFFULL
435#define ROGUE_CR_CLK_XTPLUS_CTRL_USC_SHARED_OFF 0x0000000000000000ULL
436#define ROGUE_CR_CLK_XTPLUS_CTRL_USC_SHARED_ON 0x0000000000040000ULL
437#define ROGUE_CR_CLK_XTPLUS_CTRL_USC_SHARED_AUTO 0x0000000000080000ULL
438#define ROGUE_CR_CLK_XTPLUS_CTRL_GEOMETRY_SHIFT 16U
439#define ROGUE_CR_CLK_XTPLUS_CTRL_GEOMETRY_CLRMSK 0xFFFFFFFFFFFCFFFFULL
440#define ROGUE_CR_CLK_XTPLUS_CTRL_GEOMETRY_OFF 0x0000000000000000ULL
441#define ROGUE_CR_CLK_XTPLUS_CTRL_GEOMETRY_ON 0x0000000000010000ULL
442#define ROGUE_CR_CLK_XTPLUS_CTRL_GEOMETRY_AUTO 0x0000000000020000ULL
443
444/* Register ROGUE_CR_CLK_XTPLUS_STATUS */
445#define ROGUE_CR_CLK_XTPLUS_STATUS 0x0088U
446#define ROGUE_CR_CLK_XTPLUS_STATUS_MASKFULL 0x00000000000007FFULL
447#define ROGUE_CR_CLK_XTPLUS_STATUS_TDM_SHIFT 10U
448#define ROGUE_CR_CLK_XTPLUS_STATUS_TDM_CLRMSK 0xFFFFFFFFFFFFFBFFULL
449#define ROGUE_CR_CLK_XTPLUS_STATUS_TDM_GATED 0x0000000000000000ULL
450#define ROGUE_CR_CLK_XTPLUS_STATUS_TDM_RUNNING 0x0000000000000400ULL
451#define ROGUE_CR_CLK_XTPLUS_STATUS_IPF_SHIFT 9U
452#define ROGUE_CR_CLK_XTPLUS_STATUS_IPF_CLRMSK 0xFFFFFFFFFFFFFDFFULL
453#define ROGUE_CR_CLK_XTPLUS_STATUS_IPF_GATED 0x0000000000000000ULL
454#define ROGUE_CR_CLK_XTPLUS_STATUS_IPF_RUNNING 0x0000000000000200ULL
455#define ROGUE_CR_CLK_XTPLUS_STATUS_COMPUTE_SHIFT 8U
456#define ROGUE_CR_CLK_XTPLUS_STATUS_COMPUTE_CLRMSK 0xFFFFFFFFFFFFFEFFULL
457#define ROGUE_CR_CLK_XTPLUS_STATUS_COMPUTE_GATED 0x0000000000000000ULL
458#define ROGUE_CR_CLK_XTPLUS_STATUS_COMPUTE_RUNNING 0x0000000000000100ULL
459#define ROGUE_CR_CLK_XTPLUS_STATUS_ASTC_SHIFT 7U
460#define ROGUE_CR_CLK_XTPLUS_STATUS_ASTC_CLRMSK 0xFFFFFFFFFFFFFF7FULL
461#define ROGUE_CR_CLK_XTPLUS_STATUS_ASTC_GATED 0x0000000000000000ULL
462#define ROGUE_CR_CLK_XTPLUS_STATUS_ASTC_RUNNING 0x0000000000000080ULL
463#define ROGUE_CR_CLK_XTPLUS_STATUS_PIXEL_SHIFT 6U
464#define ROGUE_CR_CLK_XTPLUS_STATUS_PIXEL_CLRMSK 0xFFFFFFFFFFFFFFBFULL
465#define ROGUE_CR_CLK_XTPLUS_STATUS_PIXEL_GATED 0x0000000000000000ULL
466#define ROGUE_CR_CLK_XTPLUS_STATUS_PIXEL_RUNNING 0x0000000000000040ULL
467#define ROGUE_CR_CLK_XTPLUS_STATUS_VERTEX_SHIFT 5U
468#define ROGUE_CR_CLK_XTPLUS_STATUS_VERTEX_CLRMSK 0xFFFFFFFFFFFFFFDFULL
469#define ROGUE_CR_CLK_XTPLUS_STATUS_VERTEX_GATED 0x0000000000000000ULL
470#define ROGUE_CR_CLK_XTPLUS_STATUS_VERTEX_RUNNING 0x0000000000000020ULL
471#define ROGUE_CR_CLK_XTPLUS_STATUS_USCPS_SHIFT 4U
472#define ROGUE_CR_CLK_XTPLUS_STATUS_USCPS_CLRMSK 0xFFFFFFFFFFFFFFEFULL
473#define ROGUE_CR_CLK_XTPLUS_STATUS_USCPS_GATED 0x0000000000000000ULL
474#define ROGUE_CR_CLK_XTPLUS_STATUS_USCPS_RUNNING 0x0000000000000010ULL
475#define ROGUE_CR_CLK_XTPLUS_STATUS_PDS_SHARED_SHIFT 3U
476#define ROGUE_CR_CLK_XTPLUS_STATUS_PDS_SHARED_CLRMSK 0xFFFFFFFFFFFFFFF7ULL
477#define ROGUE_CR_CLK_XTPLUS_STATUS_PDS_SHARED_GATED 0x0000000000000000ULL
478#define ROGUE_CR_CLK_XTPLUS_STATUS_PDS_SHARED_RUNNING 0x0000000000000008ULL
479#define ROGUE_CR_CLK_XTPLUS_STATUS_BIF_BLACKPEARL_SHIFT 2U
480#define ROGUE_CR_CLK_XTPLUS_STATUS_BIF_BLACKPEARL_CLRMSK 0xFFFFFFFFFFFFFFFBULL
481#define ROGUE_CR_CLK_XTPLUS_STATUS_BIF_BLACKPEARL_GATED 0x0000000000000000ULL
482#define ROGUE_CR_CLK_XTPLUS_STATUS_BIF_BLACKPEARL_RUNNING 0x0000000000000004ULL
483#define ROGUE_CR_CLK_XTPLUS_STATUS_USC_SHARED_SHIFT 1U
484#define ROGUE_CR_CLK_XTPLUS_STATUS_USC_SHARED_CLRMSK 0xFFFFFFFFFFFFFFFDULL
485#define ROGUE_CR_CLK_XTPLUS_STATUS_USC_SHARED_GATED 0x0000000000000000ULL
486#define ROGUE_CR_CLK_XTPLUS_STATUS_USC_SHARED_RUNNING 0x0000000000000002ULL
487#define ROGUE_CR_CLK_XTPLUS_STATUS_GEOMETRY_SHIFT 0U
488#define ROGUE_CR_CLK_XTPLUS_STATUS_GEOMETRY_CLRMSK 0xFFFFFFFFFFFFFFFEULL
489#define ROGUE_CR_CLK_XTPLUS_STATUS_GEOMETRY_GATED 0x0000000000000000ULL
490#define ROGUE_CR_CLK_XTPLUS_STATUS_GEOMETRY_RUNNING 0x0000000000000001ULL
491
492/* Register ROGUE_CR_SOFT_RESET */
493#define ROGUE_CR_SOFT_RESET 0x0100U
494#define ROGUE_CR_SOFT_RESET__PBE2_XE__MASKFULL 0xFFEFFFFFFFFFFC3DULL
495#define ROGUE_CR_SOFT_RESET_MASKFULL 0x00E7FFFFFFFFFC3DULL
496#define ROGUE_CR_SOFT_RESET_PHANTOM3_CORE_SHIFT 63U
497#define ROGUE_CR_SOFT_RESET_PHANTOM3_CORE_CLRMSK 0x7FFFFFFFFFFFFFFFULL
498#define ROGUE_CR_SOFT_RESET_PHANTOM3_CORE_EN 0x8000000000000000ULL
499#define ROGUE_CR_SOFT_RESET_PHANTOM2_CORE_SHIFT 62U
500#define ROGUE_CR_SOFT_RESET_PHANTOM2_CORE_CLRMSK 0xBFFFFFFFFFFFFFFFULL
501#define ROGUE_CR_SOFT_RESET_PHANTOM2_CORE_EN 0x4000000000000000ULL
502#define ROGUE_CR_SOFT_RESET_BERNADO2_CORE_SHIFT 61U
503#define ROGUE_CR_SOFT_RESET_BERNADO2_CORE_CLRMSK 0xDFFFFFFFFFFFFFFFULL
504#define ROGUE_CR_SOFT_RESET_BERNADO2_CORE_EN 0x2000000000000000ULL
505#define ROGUE_CR_SOFT_RESET_JONES_CORE_SHIFT 60U
506#define ROGUE_CR_SOFT_RESET_JONES_CORE_CLRMSK 0xEFFFFFFFFFFFFFFFULL
507#define ROGUE_CR_SOFT_RESET_JONES_CORE_EN 0x1000000000000000ULL
508#define ROGUE_CR_SOFT_RESET_TILING_CORE_SHIFT 59U
509#define ROGUE_CR_SOFT_RESET_TILING_CORE_CLRMSK 0xF7FFFFFFFFFFFFFFULL
510#define ROGUE_CR_SOFT_RESET_TILING_CORE_EN 0x0800000000000000ULL
511#define ROGUE_CR_SOFT_RESET_TE3_SHIFT 58U
512#define ROGUE_CR_SOFT_RESET_TE3_CLRMSK 0xFBFFFFFFFFFFFFFFULL
513#define ROGUE_CR_SOFT_RESET_TE3_EN 0x0400000000000000ULL
514#define ROGUE_CR_SOFT_RESET_VCE_SHIFT 57U
515#define ROGUE_CR_SOFT_RESET_VCE_CLRMSK 0xFDFFFFFFFFFFFFFFULL
516#define ROGUE_CR_SOFT_RESET_VCE_EN 0x0200000000000000ULL
517#define ROGUE_CR_SOFT_RESET_VBS_SHIFT 56U
518#define ROGUE_CR_SOFT_RESET_VBS_CLRMSK 0xFEFFFFFFFFFFFFFFULL
519#define ROGUE_CR_SOFT_RESET_VBS_EN 0x0100000000000000ULL
520#define ROGUE_CR_SOFT_RESET_DPX1_CORE_SHIFT 55U
521#define ROGUE_CR_SOFT_RESET_DPX1_CORE_CLRMSK 0xFF7FFFFFFFFFFFFFULL
522#define ROGUE_CR_SOFT_RESET_DPX1_CORE_EN 0x0080000000000000ULL
523#define ROGUE_CR_SOFT_RESET_DPX0_CORE_SHIFT 54U
524#define ROGUE_CR_SOFT_RESET_DPX0_CORE_CLRMSK 0xFFBFFFFFFFFFFFFFULL
525#define ROGUE_CR_SOFT_RESET_DPX0_CORE_EN 0x0040000000000000ULL
526#define ROGUE_CR_SOFT_RESET_FBA_SHIFT 53U
527#define ROGUE_CR_SOFT_RESET_FBA_CLRMSK 0xFFDFFFFFFFFFFFFFULL
528#define ROGUE_CR_SOFT_RESET_FBA_EN 0x0020000000000000ULL
529#define ROGUE_CR_SOFT_RESET_FB_CDC_SHIFT 51U
530#define ROGUE_CR_SOFT_RESET_FB_CDC_CLRMSK 0xFFF7FFFFFFFFFFFFULL
531#define ROGUE_CR_SOFT_RESET_FB_CDC_EN 0x0008000000000000ULL
532#define ROGUE_CR_SOFT_RESET_SH_SHIFT 50U
533#define ROGUE_CR_SOFT_RESET_SH_CLRMSK 0xFFFBFFFFFFFFFFFFULL
534#define ROGUE_CR_SOFT_RESET_SH_EN 0x0004000000000000ULL
535#define ROGUE_CR_SOFT_RESET_VRDM_SHIFT 49U
536#define ROGUE_CR_SOFT_RESET_VRDM_CLRMSK 0xFFFDFFFFFFFFFFFFULL
537#define ROGUE_CR_SOFT_RESET_VRDM_EN 0x0002000000000000ULL
538#define ROGUE_CR_SOFT_RESET_MCU_FBTC_SHIFT 48U
539#define ROGUE_CR_SOFT_RESET_MCU_FBTC_CLRMSK 0xFFFEFFFFFFFFFFFFULL
540#define ROGUE_CR_SOFT_RESET_MCU_FBTC_EN 0x0001000000000000ULL
541#define ROGUE_CR_SOFT_RESET_PHANTOM1_CORE_SHIFT 47U
542#define ROGUE_CR_SOFT_RESET_PHANTOM1_CORE_CLRMSK 0xFFFF7FFFFFFFFFFFULL
543#define ROGUE_CR_SOFT_RESET_PHANTOM1_CORE_EN 0x0000800000000000ULL
544#define ROGUE_CR_SOFT_RESET_PHANTOM0_CORE_SHIFT 46U
545#define ROGUE_CR_SOFT_RESET_PHANTOM0_CORE_CLRMSK 0xFFFFBFFFFFFFFFFFULL
546#define ROGUE_CR_SOFT_RESET_PHANTOM0_CORE_EN 0x0000400000000000ULL
547#define ROGUE_CR_SOFT_RESET_BERNADO1_CORE_SHIFT 45U
548#define ROGUE_CR_SOFT_RESET_BERNADO1_CORE_CLRMSK 0xFFFFDFFFFFFFFFFFULL
549#define ROGUE_CR_SOFT_RESET_BERNADO1_CORE_EN 0x0000200000000000ULL
550#define ROGUE_CR_SOFT_RESET_BERNADO0_CORE_SHIFT 44U
551#define ROGUE_CR_SOFT_RESET_BERNADO0_CORE_CLRMSK 0xFFFFEFFFFFFFFFFFULL
552#define ROGUE_CR_SOFT_RESET_BERNADO0_CORE_EN 0x0000100000000000ULL
553#define ROGUE_CR_SOFT_RESET_IPP_SHIFT 43U
554#define ROGUE_CR_SOFT_RESET_IPP_CLRMSK 0xFFFFF7FFFFFFFFFFULL
555#define ROGUE_CR_SOFT_RESET_IPP_EN 0x0000080000000000ULL
556#define ROGUE_CR_SOFT_RESET_BIF_TEXAS_SHIFT 42U
557#define ROGUE_CR_SOFT_RESET_BIF_TEXAS_CLRMSK 0xFFFFFBFFFFFFFFFFULL
558#define ROGUE_CR_SOFT_RESET_BIF_TEXAS_EN 0x0000040000000000ULL
559#define ROGUE_CR_SOFT_RESET_TORNADO_CORE_SHIFT 41U
560#define ROGUE_CR_SOFT_RESET_TORNADO_CORE_CLRMSK 0xFFFFFDFFFFFFFFFFULL
561#define ROGUE_CR_SOFT_RESET_TORNADO_CORE_EN 0x0000020000000000ULL
562#define ROGUE_CR_SOFT_RESET_DUST_H_CORE_SHIFT 40U
563#define ROGUE_CR_SOFT_RESET_DUST_H_CORE_CLRMSK 0xFFFFFEFFFFFFFFFFULL
564#define ROGUE_CR_SOFT_RESET_DUST_H_CORE_EN 0x0000010000000000ULL
565#define ROGUE_CR_SOFT_RESET_DUST_G_CORE_SHIFT 39U
566#define ROGUE_CR_SOFT_RESET_DUST_G_CORE_CLRMSK 0xFFFFFF7FFFFFFFFFULL
567#define ROGUE_CR_SOFT_RESET_DUST_G_CORE_EN 0x0000008000000000ULL
568#define ROGUE_CR_SOFT_RESET_DUST_F_CORE_SHIFT 38U
569#define ROGUE_CR_SOFT_RESET_DUST_F_CORE_CLRMSK 0xFFFFFFBFFFFFFFFFULL
570#define ROGUE_CR_SOFT_RESET_DUST_F_CORE_EN 0x0000004000000000ULL
571#define ROGUE_CR_SOFT_RESET_DUST_E_CORE_SHIFT 37U
572#define ROGUE_CR_SOFT_RESET_DUST_E_CORE_CLRMSK 0xFFFFFFDFFFFFFFFFULL
573#define ROGUE_CR_SOFT_RESET_DUST_E_CORE_EN 0x0000002000000000ULL
574#define ROGUE_CR_SOFT_RESET_DUST_D_CORE_SHIFT 36U
575#define ROGUE_CR_SOFT_RESET_DUST_D_CORE_CLRMSK 0xFFFFFFEFFFFFFFFFULL
576#define ROGUE_CR_SOFT_RESET_DUST_D_CORE_EN 0x0000001000000000ULL
577#define ROGUE_CR_SOFT_RESET_DUST_C_CORE_SHIFT 35U
578#define ROGUE_CR_SOFT_RESET_DUST_C_CORE_CLRMSK 0xFFFFFFF7FFFFFFFFULL
579#define ROGUE_CR_SOFT_RESET_DUST_C_CORE_EN 0x0000000800000000ULL
580#define ROGUE_CR_SOFT_RESET_MMU_SHIFT 34U
581#define ROGUE_CR_SOFT_RESET_MMU_CLRMSK 0xFFFFFFFBFFFFFFFFULL
582#define ROGUE_CR_SOFT_RESET_MMU_EN 0x0000000400000000ULL
583#define ROGUE_CR_SOFT_RESET_BIF1_SHIFT 33U
584#define ROGUE_CR_SOFT_RESET_BIF1_CLRMSK 0xFFFFFFFDFFFFFFFFULL
585#define ROGUE_CR_SOFT_RESET_BIF1_EN 0x0000000200000000ULL
586#define ROGUE_CR_SOFT_RESET_GARTEN_SHIFT 32U
587#define ROGUE_CR_SOFT_RESET_GARTEN_CLRMSK 0xFFFFFFFEFFFFFFFFULL
588#define ROGUE_CR_SOFT_RESET_GARTEN_EN 0x0000000100000000ULL
589#define ROGUE_CR_SOFT_RESET_CPU_SHIFT 32U
590#define ROGUE_CR_SOFT_RESET_CPU_CLRMSK 0xFFFFFFFEFFFFFFFFULL
591#define ROGUE_CR_SOFT_RESET_CPU_EN 0x0000000100000000ULL
592#define ROGUE_CR_SOFT_RESET_RASCAL_CORE_SHIFT 31U
593#define ROGUE_CR_SOFT_RESET_RASCAL_CORE_CLRMSK 0xFFFFFFFF7FFFFFFFULL
594#define ROGUE_CR_SOFT_RESET_RASCAL_CORE_EN 0x0000000080000000ULL
595#define ROGUE_CR_SOFT_RESET_DUST_B_CORE_SHIFT 30U
596#define ROGUE_CR_SOFT_RESET_DUST_B_CORE_CLRMSK 0xFFFFFFFFBFFFFFFFULL
597#define ROGUE_CR_SOFT_RESET_DUST_B_CORE_EN 0x0000000040000000ULL
598#define ROGUE_CR_SOFT_RESET_DUST_A_CORE_SHIFT 29U
599#define ROGUE_CR_SOFT_RESET_DUST_A_CORE_CLRMSK 0xFFFFFFFFDFFFFFFFULL
600#define ROGUE_CR_SOFT_RESET_DUST_A_CORE_EN 0x0000000020000000ULL
601#define ROGUE_CR_SOFT_RESET_FB_TLCACHE_SHIFT 28U
602#define ROGUE_CR_SOFT_RESET_FB_TLCACHE_CLRMSK 0xFFFFFFFFEFFFFFFFULL
603#define ROGUE_CR_SOFT_RESET_FB_TLCACHE_EN 0x0000000010000000ULL
604#define ROGUE_CR_SOFT_RESET_SLC_SHIFT 27U
605#define ROGUE_CR_SOFT_RESET_SLC_CLRMSK 0xFFFFFFFFF7FFFFFFULL
606#define ROGUE_CR_SOFT_RESET_SLC_EN 0x0000000008000000ULL
607#define ROGUE_CR_SOFT_RESET_TLA_SHIFT 26U
608#define ROGUE_CR_SOFT_RESET_TLA_CLRMSK 0xFFFFFFFFFBFFFFFFULL
609#define ROGUE_CR_SOFT_RESET_TLA_EN 0x0000000004000000ULL
610#define ROGUE_CR_SOFT_RESET_UVS_SHIFT 25U
611#define ROGUE_CR_SOFT_RESET_UVS_CLRMSK 0xFFFFFFFFFDFFFFFFULL
612#define ROGUE_CR_SOFT_RESET_UVS_EN 0x0000000002000000ULL
613#define ROGUE_CR_SOFT_RESET_TE_SHIFT 24U
614#define ROGUE_CR_SOFT_RESET_TE_CLRMSK 0xFFFFFFFFFEFFFFFFULL
615#define ROGUE_CR_SOFT_RESET_TE_EN 0x0000000001000000ULL
616#define ROGUE_CR_SOFT_RESET_GPP_SHIFT 23U
617#define ROGUE_CR_SOFT_RESET_GPP_CLRMSK 0xFFFFFFFFFF7FFFFFULL
618#define ROGUE_CR_SOFT_RESET_GPP_EN 0x0000000000800000ULL
619#define ROGUE_CR_SOFT_RESET_FBDC_SHIFT 22U
620#define ROGUE_CR_SOFT_RESET_FBDC_CLRMSK 0xFFFFFFFFFFBFFFFFULL
621#define ROGUE_CR_SOFT_RESET_FBDC_EN 0x0000000000400000ULL
622#define ROGUE_CR_SOFT_RESET_FBC_SHIFT 21U
623#define ROGUE_CR_SOFT_RESET_FBC_CLRMSK 0xFFFFFFFFFFDFFFFFULL
624#define ROGUE_CR_SOFT_RESET_FBC_EN 0x0000000000200000ULL
625#define ROGUE_CR_SOFT_RESET_PM_SHIFT 20U
626#define ROGUE_CR_SOFT_RESET_PM_CLRMSK 0xFFFFFFFFFFEFFFFFULL
627#define ROGUE_CR_SOFT_RESET_PM_EN 0x0000000000100000ULL
628#define ROGUE_CR_SOFT_RESET_PBE_SHIFT 19U
629#define ROGUE_CR_SOFT_RESET_PBE_CLRMSK 0xFFFFFFFFFFF7FFFFULL
630#define ROGUE_CR_SOFT_RESET_PBE_EN 0x0000000000080000ULL
631#define ROGUE_CR_SOFT_RESET_USC_SHARED_SHIFT 18U
632#define ROGUE_CR_SOFT_RESET_USC_SHARED_CLRMSK 0xFFFFFFFFFFFBFFFFULL
633#define ROGUE_CR_SOFT_RESET_USC_SHARED_EN 0x0000000000040000ULL
634#define ROGUE_CR_SOFT_RESET_MCU_L1_SHIFT 17U
635#define ROGUE_CR_SOFT_RESET_MCU_L1_CLRMSK 0xFFFFFFFFFFFDFFFFULL
636#define ROGUE_CR_SOFT_RESET_MCU_L1_EN 0x0000000000020000ULL
637#define ROGUE_CR_SOFT_RESET_BIF_SHIFT 16U
638#define ROGUE_CR_SOFT_RESET_BIF_CLRMSK 0xFFFFFFFFFFFEFFFFULL
639#define ROGUE_CR_SOFT_RESET_BIF_EN 0x0000000000010000ULL
640#define ROGUE_CR_SOFT_RESET_CDM_SHIFT 15U
641#define ROGUE_CR_SOFT_RESET_CDM_CLRMSK 0xFFFFFFFFFFFF7FFFULL
642#define ROGUE_CR_SOFT_RESET_CDM_EN 0x0000000000008000ULL
643#define ROGUE_CR_SOFT_RESET_VDM_SHIFT 14U
644#define ROGUE_CR_SOFT_RESET_VDM_CLRMSK 0xFFFFFFFFFFFFBFFFULL
645#define ROGUE_CR_SOFT_RESET_VDM_EN 0x0000000000004000ULL
646#define ROGUE_CR_SOFT_RESET_TESS_SHIFT 13U
647#define ROGUE_CR_SOFT_RESET_TESS_CLRMSK 0xFFFFFFFFFFFFDFFFULL
648#define ROGUE_CR_SOFT_RESET_TESS_EN 0x0000000000002000ULL
649#define ROGUE_CR_SOFT_RESET_PDS_SHIFT 12U
650#define ROGUE_CR_SOFT_RESET_PDS_CLRMSK 0xFFFFFFFFFFFFEFFFULL
651#define ROGUE_CR_SOFT_RESET_PDS_EN 0x0000000000001000ULL
652#define ROGUE_CR_SOFT_RESET_ISP_SHIFT 11U
653#define ROGUE_CR_SOFT_RESET_ISP_CLRMSK 0xFFFFFFFFFFFFF7FFULL
654#define ROGUE_CR_SOFT_RESET_ISP_EN 0x0000000000000800ULL
655#define ROGUE_CR_SOFT_RESET_TSP_SHIFT 10U
656#define ROGUE_CR_SOFT_RESET_TSP_CLRMSK 0xFFFFFFFFFFFFFBFFULL
657#define ROGUE_CR_SOFT_RESET_TSP_EN 0x0000000000000400ULL
658#define ROGUE_CR_SOFT_RESET_SYSARB_SHIFT 5U
659#define ROGUE_CR_SOFT_RESET_SYSARB_CLRMSK 0xFFFFFFFFFFFFFFDFULL
660#define ROGUE_CR_SOFT_RESET_SYSARB_EN 0x0000000000000020ULL
661#define ROGUE_CR_SOFT_RESET_TPU_MCU_DEMUX_SHIFT 4U
662#define ROGUE_CR_SOFT_RESET_TPU_MCU_DEMUX_CLRMSK 0xFFFFFFFFFFFFFFEFULL
663#define ROGUE_CR_SOFT_RESET_TPU_MCU_DEMUX_EN 0x0000000000000010ULL
664#define ROGUE_CR_SOFT_RESET_MCU_L0_SHIFT 3U
665#define ROGUE_CR_SOFT_RESET_MCU_L0_CLRMSK 0xFFFFFFFFFFFFFFF7ULL
666#define ROGUE_CR_SOFT_RESET_MCU_L0_EN 0x0000000000000008ULL
667#define ROGUE_CR_SOFT_RESET_TPU_SHIFT 2U
668#define ROGUE_CR_SOFT_RESET_TPU_CLRMSK 0xFFFFFFFFFFFFFFFBULL
669#define ROGUE_CR_SOFT_RESET_TPU_EN 0x0000000000000004ULL
670#define ROGUE_CR_SOFT_RESET_USC_SHIFT 0U
671#define ROGUE_CR_SOFT_RESET_USC_CLRMSK 0xFFFFFFFFFFFFFFFEULL
672#define ROGUE_CR_SOFT_RESET_USC_EN 0x0000000000000001ULL
673
674/* Register ROGUE_CR_SOFT_RESET2 */
675#define ROGUE_CR_SOFT_RESET2 0x0108U
676#define ROGUE_CR_SOFT_RESET2_MASKFULL 0x00000000001FFFFFULL
677#define ROGUE_CR_SOFT_RESET2_SPFILTER_SHIFT 12U
678#define ROGUE_CR_SOFT_RESET2_SPFILTER_CLRMSK 0xFFE00FFFU
679#define ROGUE_CR_SOFT_RESET2_TDM_SHIFT 11U
680#define ROGUE_CR_SOFT_RESET2_TDM_CLRMSK 0xFFFFF7FFU
681#define ROGUE_CR_SOFT_RESET2_TDM_EN 0x00000800U
682#define ROGUE_CR_SOFT_RESET2_ASTC_SHIFT 10U
683#define ROGUE_CR_SOFT_RESET2_ASTC_CLRMSK 0xFFFFFBFFU
684#define ROGUE_CR_SOFT_RESET2_ASTC_EN 0x00000400U
685#define ROGUE_CR_SOFT_RESET2_BLACKPEARL_SHIFT 9U
686#define ROGUE_CR_SOFT_RESET2_BLACKPEARL_CLRMSK 0xFFFFFDFFU
687#define ROGUE_CR_SOFT_RESET2_BLACKPEARL_EN 0x00000200U
688#define ROGUE_CR_SOFT_RESET2_USCPS_SHIFT 8U
689#define ROGUE_CR_SOFT_RESET2_USCPS_CLRMSK 0xFFFFFEFFU
690#define ROGUE_CR_SOFT_RESET2_USCPS_EN 0x00000100U
691#define ROGUE_CR_SOFT_RESET2_IPF_SHIFT 7U
692#define ROGUE_CR_SOFT_RESET2_IPF_CLRMSK 0xFFFFFF7FU
693#define ROGUE_CR_SOFT_RESET2_IPF_EN 0x00000080U
694#define ROGUE_CR_SOFT_RESET2_GEOMETRY_SHIFT 6U
695#define ROGUE_CR_SOFT_RESET2_GEOMETRY_CLRMSK 0xFFFFFFBFU
696#define ROGUE_CR_SOFT_RESET2_GEOMETRY_EN 0x00000040U
697#define ROGUE_CR_SOFT_RESET2_USC_SHARED_SHIFT 5U
698#define ROGUE_CR_SOFT_RESET2_USC_SHARED_CLRMSK 0xFFFFFFDFU
699#define ROGUE_CR_SOFT_RESET2_USC_SHARED_EN 0x00000020U
700#define ROGUE_CR_SOFT_RESET2_PDS_SHARED_SHIFT 4U
701#define ROGUE_CR_SOFT_RESET2_PDS_SHARED_CLRMSK 0xFFFFFFEFU
702#define ROGUE_CR_SOFT_RESET2_PDS_SHARED_EN 0x00000010U
703#define ROGUE_CR_SOFT_RESET2_BIF_BLACKPEARL_SHIFT 3U
704#define ROGUE_CR_SOFT_RESET2_BIF_BLACKPEARL_CLRMSK 0xFFFFFFF7U
705#define ROGUE_CR_SOFT_RESET2_BIF_BLACKPEARL_EN 0x00000008U
706#define ROGUE_CR_SOFT_RESET2_PIXEL_SHIFT 2U
707#define ROGUE_CR_SOFT_RESET2_PIXEL_CLRMSK 0xFFFFFFFBU
708#define ROGUE_CR_SOFT_RESET2_PIXEL_EN 0x00000004U
709#define ROGUE_CR_SOFT_RESET2_CDM_SHIFT 1U
710#define ROGUE_CR_SOFT_RESET2_CDM_CLRMSK 0xFFFFFFFDU
711#define ROGUE_CR_SOFT_RESET2_CDM_EN 0x00000002U
712#define ROGUE_CR_SOFT_RESET2_VERTEX_SHIFT 0U
713#define ROGUE_CR_SOFT_RESET2_VERTEX_CLRMSK 0xFFFFFFFEU
714#define ROGUE_CR_SOFT_RESET2_VERTEX_EN 0x00000001U
715
716/* Register ROGUE_CR_EVENT_STATUS */
717#define ROGUE_CR_EVENT_STATUS 0x0130U
718#define ROGUE_CR_EVENT_STATUS__ROGUEXE__MASKFULL 0x00000000E01DFFFFULL
719#define ROGUE_CR_EVENT_STATUS__SIGNALS__MASKFULL 0x00000000E007FFFFULL
720#define ROGUE_CR_EVENT_STATUS_MASKFULL 0x00000000FFFFFFFFULL
721#define ROGUE_CR_EVENT_STATUS_TDM_FENCE_FINISHED_SHIFT 31U
722#define ROGUE_CR_EVENT_STATUS_TDM_FENCE_FINISHED_CLRMSK 0x7FFFFFFFU
723#define ROGUE_CR_EVENT_STATUS_TDM_FENCE_FINISHED_EN 0x80000000U
724#define ROGUE_CR_EVENT_STATUS_TDM_BUFFER_STALL_SHIFT 30U
725#define ROGUE_CR_EVENT_STATUS_TDM_BUFFER_STALL_CLRMSK 0xBFFFFFFFU
726#define ROGUE_CR_EVENT_STATUS_TDM_BUFFER_STALL_EN 0x40000000U
727#define ROGUE_CR_EVENT_STATUS_COMPUTE_SIGNAL_FAILURE_SHIFT 29U
728#define ROGUE_CR_EVENT_STATUS_COMPUTE_SIGNAL_FAILURE_CLRMSK 0xDFFFFFFFU
729#define ROGUE_CR_EVENT_STATUS_COMPUTE_SIGNAL_FAILURE_EN 0x20000000U
730#define ROGUE_CR_EVENT_STATUS_DPX_OUT_OF_MEMORY_SHIFT 28U
731#define ROGUE_CR_EVENT_STATUS_DPX_OUT_OF_MEMORY_CLRMSK 0xEFFFFFFFU
732#define ROGUE_CR_EVENT_STATUS_DPX_OUT_OF_MEMORY_EN 0x10000000U
733#define ROGUE_CR_EVENT_STATUS_DPX_MMU_PAGE_FAULT_SHIFT 27U
734#define ROGUE_CR_EVENT_STATUS_DPX_MMU_PAGE_FAULT_CLRMSK 0xF7FFFFFFU
735#define ROGUE_CR_EVENT_STATUS_DPX_MMU_PAGE_FAULT_EN 0x08000000U
736#define ROGUE_CR_EVENT_STATUS_RPM_OUT_OF_MEMORY_SHIFT 26U
737#define ROGUE_CR_EVENT_STATUS_RPM_OUT_OF_MEMORY_CLRMSK 0xFBFFFFFFU
738#define ROGUE_CR_EVENT_STATUS_RPM_OUT_OF_MEMORY_EN 0x04000000U
739#define ROGUE_CR_EVENT_STATUS_FBA_FC3_FINISHED_SHIFT 25U
740#define ROGUE_CR_EVENT_STATUS_FBA_FC3_FINISHED_CLRMSK 0xFDFFFFFFU
741#define ROGUE_CR_EVENT_STATUS_FBA_FC3_FINISHED_EN 0x02000000U
742#define ROGUE_CR_EVENT_STATUS_FBA_FC2_FINISHED_SHIFT 24U
743#define ROGUE_CR_EVENT_STATUS_FBA_FC2_FINISHED_CLRMSK 0xFEFFFFFFU
744#define ROGUE_CR_EVENT_STATUS_FBA_FC2_FINISHED_EN 0x01000000U
745#define ROGUE_CR_EVENT_STATUS_FBA_FC1_FINISHED_SHIFT 23U
746#define ROGUE_CR_EVENT_STATUS_FBA_FC1_FINISHED_CLRMSK 0xFF7FFFFFU
747#define ROGUE_CR_EVENT_STATUS_FBA_FC1_FINISHED_EN 0x00800000U
748#define ROGUE_CR_EVENT_STATUS_FBA_FC0_FINISHED_SHIFT 22U
749#define ROGUE_CR_EVENT_STATUS_FBA_FC0_FINISHED_CLRMSK 0xFFBFFFFFU
750#define ROGUE_CR_EVENT_STATUS_FBA_FC0_FINISHED_EN 0x00400000U
751#define ROGUE_CR_EVENT_STATUS_RDM_FC3_FINISHED_SHIFT 21U
752#define ROGUE_CR_EVENT_STATUS_RDM_FC3_FINISHED_CLRMSK 0xFFDFFFFFU
753#define ROGUE_CR_EVENT_STATUS_RDM_FC3_FINISHED_EN 0x00200000U
754#define ROGUE_CR_EVENT_STATUS_RDM_FC2_FINISHED_SHIFT 20U
755#define ROGUE_CR_EVENT_STATUS_RDM_FC2_FINISHED_CLRMSK 0xFFEFFFFFU
756#define ROGUE_CR_EVENT_STATUS_RDM_FC2_FINISHED_EN 0x00100000U
757#define ROGUE_CR_EVENT_STATUS_SAFETY_SHIFT 20U
758#define ROGUE_CR_EVENT_STATUS_SAFETY_CLRMSK 0xFFEFFFFFU
759#define ROGUE_CR_EVENT_STATUS_SAFETY_EN 0x00100000U
760#define ROGUE_CR_EVENT_STATUS_RDM_FC1_FINISHED_SHIFT 19U
761#define ROGUE_CR_EVENT_STATUS_RDM_FC1_FINISHED_CLRMSK 0xFFF7FFFFU
762#define ROGUE_CR_EVENT_STATUS_RDM_FC1_FINISHED_EN 0x00080000U
763#define ROGUE_CR_EVENT_STATUS_SLAVE_REQ_SHIFT 19U
764#define ROGUE_CR_EVENT_STATUS_SLAVE_REQ_CLRMSK 0xFFF7FFFFU
765#define ROGUE_CR_EVENT_STATUS_SLAVE_REQ_EN 0x00080000U
766#define ROGUE_CR_EVENT_STATUS_RDM_FC0_FINISHED_SHIFT 18U
767#define ROGUE_CR_EVENT_STATUS_RDM_FC0_FINISHED_CLRMSK 0xFFFBFFFFU
768#define ROGUE_CR_EVENT_STATUS_RDM_FC0_FINISHED_EN 0x00040000U
769#define ROGUE_CR_EVENT_STATUS_TDM_CONTEXT_STORE_FINISHED_SHIFT 18U
770#define ROGUE_CR_EVENT_STATUS_TDM_CONTEXT_STORE_FINISHED_CLRMSK 0xFFFBFFFFU
771#define ROGUE_CR_EVENT_STATUS_TDM_CONTEXT_STORE_FINISHED_EN 0x00040000U
772#define ROGUE_CR_EVENT_STATUS_SHG_FINISHED_SHIFT 17U
773#define ROGUE_CR_EVENT_STATUS_SHG_FINISHED_CLRMSK 0xFFFDFFFFU
774#define ROGUE_CR_EVENT_STATUS_SHG_FINISHED_EN 0x00020000U
775#define ROGUE_CR_EVENT_STATUS_SPFILTER_SIGNAL_UPDATE_SHIFT 17U
776#define ROGUE_CR_EVENT_STATUS_SPFILTER_SIGNAL_UPDATE_CLRMSK 0xFFFDFFFFU
777#define ROGUE_CR_EVENT_STATUS_SPFILTER_SIGNAL_UPDATE_EN 0x00020000U
778#define ROGUE_CR_EVENT_STATUS_COMPUTE_BUFFER_STALL_SHIFT 16U
779#define ROGUE_CR_EVENT_STATUS_COMPUTE_BUFFER_STALL_CLRMSK 0xFFFEFFFFU
780#define ROGUE_CR_EVENT_STATUS_COMPUTE_BUFFER_STALL_EN 0x00010000U
781#define ROGUE_CR_EVENT_STATUS_USC_TRIGGER_SHIFT 15U
782#define ROGUE_CR_EVENT_STATUS_USC_TRIGGER_CLRMSK 0xFFFF7FFFU
783#define ROGUE_CR_EVENT_STATUS_USC_TRIGGER_EN 0x00008000U
784#define ROGUE_CR_EVENT_STATUS_ZLS_FINISHED_SHIFT 14U
785#define ROGUE_CR_EVENT_STATUS_ZLS_FINISHED_CLRMSK 0xFFFFBFFFU
786#define ROGUE_CR_EVENT_STATUS_ZLS_FINISHED_EN 0x00004000U
787#define ROGUE_CR_EVENT_STATUS_GPIO_ACK_SHIFT 13U
788#define ROGUE_CR_EVENT_STATUS_GPIO_ACK_CLRMSK 0xFFFFDFFFU
789#define ROGUE_CR_EVENT_STATUS_GPIO_ACK_EN 0x00002000U
790#define ROGUE_CR_EVENT_STATUS_GPIO_REQ_SHIFT 12U
791#define ROGUE_CR_EVENT_STATUS_GPIO_REQ_CLRMSK 0xFFFFEFFFU
792#define ROGUE_CR_EVENT_STATUS_GPIO_REQ_EN 0x00001000U
793#define ROGUE_CR_EVENT_STATUS_POWER_ABORT_SHIFT 11U
794#define ROGUE_CR_EVENT_STATUS_POWER_ABORT_CLRMSK 0xFFFFF7FFU
795#define ROGUE_CR_EVENT_STATUS_POWER_ABORT_EN 0x00000800U
796#define ROGUE_CR_EVENT_STATUS_POWER_COMPLETE_SHIFT 10U
797#define ROGUE_CR_EVENT_STATUS_POWER_COMPLETE_CLRMSK 0xFFFFFBFFU
798#define ROGUE_CR_EVENT_STATUS_POWER_COMPLETE_EN 0x00000400U
799#define ROGUE_CR_EVENT_STATUS_MMU_PAGE_FAULT_SHIFT 9U
800#define ROGUE_CR_EVENT_STATUS_MMU_PAGE_FAULT_CLRMSK 0xFFFFFDFFU
801#define ROGUE_CR_EVENT_STATUS_MMU_PAGE_FAULT_EN 0x00000200U
802#define ROGUE_CR_EVENT_STATUS_PM_3D_MEM_FREE_SHIFT 8U
803#define ROGUE_CR_EVENT_STATUS_PM_3D_MEM_FREE_CLRMSK 0xFFFFFEFFU
804#define ROGUE_CR_EVENT_STATUS_PM_3D_MEM_FREE_EN 0x00000100U
805#define ROGUE_CR_EVENT_STATUS_PM_OUT_OF_MEMORY_SHIFT 7U
806#define ROGUE_CR_EVENT_STATUS_PM_OUT_OF_MEMORY_CLRMSK 0xFFFFFF7FU
807#define ROGUE_CR_EVENT_STATUS_PM_OUT_OF_MEMORY_EN 0x00000080U
808#define ROGUE_CR_EVENT_STATUS_TA_TERMINATE_SHIFT 6U
809#define ROGUE_CR_EVENT_STATUS_TA_TERMINATE_CLRMSK 0xFFFFFFBFU
810#define ROGUE_CR_EVENT_STATUS_TA_TERMINATE_EN 0x00000040U
811#define ROGUE_CR_EVENT_STATUS_TA_FINISHED_SHIFT 5U
812#define ROGUE_CR_EVENT_STATUS_TA_FINISHED_CLRMSK 0xFFFFFFDFU
813#define ROGUE_CR_EVENT_STATUS_TA_FINISHED_EN 0x00000020U
814#define ROGUE_CR_EVENT_STATUS_ISP_END_MACROTILE_SHIFT 4U
815#define ROGUE_CR_EVENT_STATUS_ISP_END_MACROTILE_CLRMSK 0xFFFFFFEFU
816#define ROGUE_CR_EVENT_STATUS_ISP_END_MACROTILE_EN 0x00000010U
817#define ROGUE_CR_EVENT_STATUS_PIXELBE_END_RENDER_SHIFT 3U
818#define ROGUE_CR_EVENT_STATUS_PIXELBE_END_RENDER_CLRMSK 0xFFFFFFF7U
819#define ROGUE_CR_EVENT_STATUS_PIXELBE_END_RENDER_EN 0x00000008U
820#define ROGUE_CR_EVENT_STATUS_COMPUTE_FINISHED_SHIFT 2U
821#define ROGUE_CR_EVENT_STATUS_COMPUTE_FINISHED_CLRMSK 0xFFFFFFFBU
822#define ROGUE_CR_EVENT_STATUS_COMPUTE_FINISHED_EN 0x00000004U
823#define ROGUE_CR_EVENT_STATUS_KERNEL_FINISHED_SHIFT 1U
824#define ROGUE_CR_EVENT_STATUS_KERNEL_FINISHED_CLRMSK 0xFFFFFFFDU
825#define ROGUE_CR_EVENT_STATUS_KERNEL_FINISHED_EN 0x00000002U
826#define ROGUE_CR_EVENT_STATUS_TLA_COMPLETE_SHIFT 0U
827#define ROGUE_CR_EVENT_STATUS_TLA_COMPLETE_CLRMSK 0xFFFFFFFEU
828#define ROGUE_CR_EVENT_STATUS_TLA_COMPLETE_EN 0x00000001U
829
830/* Register ROGUE_CR_TIMER */
831#define ROGUE_CR_TIMER 0x0160U
832#define ROGUE_CR_TIMER_MASKFULL 0x8000FFFFFFFFFFFFULL
833#define ROGUE_CR_TIMER_BIT31_SHIFT 63U
834#define ROGUE_CR_TIMER_BIT31_CLRMSK 0x7FFFFFFFFFFFFFFFULL
835#define ROGUE_CR_TIMER_BIT31_EN 0x8000000000000000ULL
836#define ROGUE_CR_TIMER_VALUE_SHIFT 0U
837#define ROGUE_CR_TIMER_VALUE_CLRMSK 0xFFFF000000000000ULL
838
839/* Register ROGUE_CR_TLA_STATUS */
840#define ROGUE_CR_TLA_STATUS 0x0178U
841#define ROGUE_CR_TLA_STATUS_MASKFULL 0xFFFFFFFFFFFFFFFFULL
842#define ROGUE_CR_TLA_STATUS_BLIT_COUNT_SHIFT 39U
843#define ROGUE_CR_TLA_STATUS_BLIT_COUNT_CLRMSK 0x0000007FFFFFFFFFULL
844#define ROGUE_CR_TLA_STATUS_REQUEST_SHIFT 7U
845#define ROGUE_CR_TLA_STATUS_REQUEST_CLRMSK 0xFFFFFF800000007FULL
846#define ROGUE_CR_TLA_STATUS_FIFO_FULLNESS_SHIFT 1U
847#define ROGUE_CR_TLA_STATUS_FIFO_FULLNESS_CLRMSK 0xFFFFFFFFFFFFFF81ULL
848#define ROGUE_CR_TLA_STATUS_BUSY_SHIFT 0U
849#define ROGUE_CR_TLA_STATUS_BUSY_CLRMSK 0xFFFFFFFFFFFFFFFEULL
850#define ROGUE_CR_TLA_STATUS_BUSY_EN 0x0000000000000001ULL
851
852/* Register ROGUE_CR_PM_PARTIAL_RENDER_ENABLE */
853#define ROGUE_CR_PM_PARTIAL_RENDER_ENABLE 0x0338U
854#define ROGUE_CR_PM_PARTIAL_RENDER_ENABLE_MASKFULL 0x0000000000000001ULL
855#define ROGUE_CR_PM_PARTIAL_RENDER_ENABLE_OP_SHIFT 0U
856#define ROGUE_CR_PM_PARTIAL_RENDER_ENABLE_OP_CLRMSK 0xFFFFFFFEU
857#define ROGUE_CR_PM_PARTIAL_RENDER_ENABLE_OP_EN 0x00000001U
858
859/* Register ROGUE_CR_SIDEKICK_IDLE */
860#define ROGUE_CR_SIDEKICK_IDLE 0x03C8U
861#define ROGUE_CR_SIDEKICK_IDLE_MASKFULL 0x000000000000007FULL
862#define ROGUE_CR_SIDEKICK_IDLE_FB_CDC_SHIFT 6U
863#define ROGUE_CR_SIDEKICK_IDLE_FB_CDC_CLRMSK 0xFFFFFFBFU
864#define ROGUE_CR_SIDEKICK_IDLE_FB_CDC_EN 0x00000040U
865#define ROGUE_CR_SIDEKICK_IDLE_MMU_SHIFT 5U
866#define ROGUE_CR_SIDEKICK_IDLE_MMU_CLRMSK 0xFFFFFFDFU
867#define ROGUE_CR_SIDEKICK_IDLE_MMU_EN 0x00000020U
868#define ROGUE_CR_SIDEKICK_IDLE_BIF128_SHIFT 4U
869#define ROGUE_CR_SIDEKICK_IDLE_BIF128_CLRMSK 0xFFFFFFEFU
870#define ROGUE_CR_SIDEKICK_IDLE_BIF128_EN 0x00000010U
871#define ROGUE_CR_SIDEKICK_IDLE_TLA_SHIFT 3U
872#define ROGUE_CR_SIDEKICK_IDLE_TLA_CLRMSK 0xFFFFFFF7U
873#define ROGUE_CR_SIDEKICK_IDLE_TLA_EN 0x00000008U
874#define ROGUE_CR_SIDEKICK_IDLE_GARTEN_SHIFT 2U
875#define ROGUE_CR_SIDEKICK_IDLE_GARTEN_CLRMSK 0xFFFFFFFBU
876#define ROGUE_CR_SIDEKICK_IDLE_GARTEN_EN 0x00000004U
877#define ROGUE_CR_SIDEKICK_IDLE_HOSTIF_SHIFT 1U
878#define ROGUE_CR_SIDEKICK_IDLE_HOSTIF_CLRMSK 0xFFFFFFFDU
879#define ROGUE_CR_SIDEKICK_IDLE_HOSTIF_EN 0x00000002U
880#define ROGUE_CR_SIDEKICK_IDLE_SOCIF_SHIFT 0U
881#define ROGUE_CR_SIDEKICK_IDLE_SOCIF_CLRMSK 0xFFFFFFFEU
882#define ROGUE_CR_SIDEKICK_IDLE_SOCIF_EN 0x00000001U
883
884/* Register ROGUE_CR_MARS_IDLE */
885#define ROGUE_CR_MARS_IDLE 0x08F8U
886#define ROGUE_CR_MARS_IDLE_MASKFULL 0x0000000000000007ULL
887#define ROGUE_CR_MARS_IDLE_MH_SYSARB0_SHIFT 2U
888#define ROGUE_CR_MARS_IDLE_MH_SYSARB0_CLRMSK 0xFFFFFFFBU
889#define ROGUE_CR_MARS_IDLE_MH_SYSARB0_EN 0x00000004U
890#define ROGUE_CR_MARS_IDLE_CPU_SHIFT 1U
891#define ROGUE_CR_MARS_IDLE_CPU_CLRMSK 0xFFFFFFFDU
892#define ROGUE_CR_MARS_IDLE_CPU_EN 0x00000002U
893#define ROGUE_CR_MARS_IDLE_SOCIF_SHIFT 0U
894#define ROGUE_CR_MARS_IDLE_SOCIF_CLRMSK 0xFFFFFFFEU
895#define ROGUE_CR_MARS_IDLE_SOCIF_EN 0x00000001U
896
897/* Register ROGUE_CR_VDM_CONTEXT_STORE_STATUS */
898#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS 0x0430U
899#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_MASKFULL 0x00000000000000F3ULL
900#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_LAST_PIPE_SHIFT 4U
901#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_LAST_PIPE_CLRMSK 0xFFFFFF0FU
902#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_NEED_RESUME_SHIFT 1U
903#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_NEED_RESUME_CLRMSK 0xFFFFFFFDU
904#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_NEED_RESUME_EN 0x00000002U
905#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_COMPLETE_SHIFT 0U
906#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_COMPLETE_CLRMSK 0xFFFFFFFEU
907#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_COMPLETE_EN 0x00000001U
908
909/* Register ROGUE_CR_VDM_CONTEXT_STORE_TASK0 */
910#define ROGUE_CR_VDM_CONTEXT_STORE_TASK0 0x0438U
911#define ROGUE_CR_VDM_CONTEXT_STORE_TASK0_MASKFULL 0xFFFFFFFFFFFFFFFFULL
912#define ROGUE_CR_VDM_CONTEXT_STORE_TASK0_PDS_STATE1_SHIFT 32U
913#define ROGUE_CR_VDM_CONTEXT_STORE_TASK0_PDS_STATE1_CLRMSK 0x00000000FFFFFFFFULL
914#define ROGUE_CR_VDM_CONTEXT_STORE_TASK0_PDS_STATE0_SHIFT 0U
915#define ROGUE_CR_VDM_CONTEXT_STORE_TASK0_PDS_STATE0_CLRMSK 0xFFFFFFFF00000000ULL
916
917/* Register ROGUE_CR_VDM_CONTEXT_STORE_TASK1 */
918#define ROGUE_CR_VDM_CONTEXT_STORE_TASK1 0x0440U
919#define ROGUE_CR_VDM_CONTEXT_STORE_TASK1_MASKFULL 0x00000000FFFFFFFFULL
920#define ROGUE_CR_VDM_CONTEXT_STORE_TASK1_PDS_STATE2_SHIFT 0U
921#define ROGUE_CR_VDM_CONTEXT_STORE_TASK1_PDS_STATE2_CLRMSK 0x00000000U
922
923/* Register ROGUE_CR_VDM_CONTEXT_STORE_TASK2 */
924#define ROGUE_CR_VDM_CONTEXT_STORE_TASK2 0x0448U
925#define ROGUE_CR_VDM_CONTEXT_STORE_TASK2_MASKFULL 0xFFFFFFFFFFFFFFFFULL
926#define ROGUE_CR_VDM_CONTEXT_STORE_TASK2_STREAM_OUT2_SHIFT 32U
927#define ROGUE_CR_VDM_CONTEXT_STORE_TASK2_STREAM_OUT2_CLRMSK 0x00000000FFFFFFFFULL
928#define ROGUE_CR_VDM_CONTEXT_STORE_TASK2_STREAM_OUT1_SHIFT 0U
929#define ROGUE_CR_VDM_CONTEXT_STORE_TASK2_STREAM_OUT1_CLRMSK 0xFFFFFFFF00000000ULL
930
931/* Register ROGUE_CR_VDM_CONTEXT_RESUME_TASK0 */
932#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK0 0x0450U
933#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK0_MASKFULL 0xFFFFFFFFFFFFFFFFULL
934#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK0_PDS_STATE1_SHIFT 32U
935#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK0_PDS_STATE1_CLRMSK 0x00000000FFFFFFFFULL
936#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK0_PDS_STATE0_SHIFT 0U
937#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK0_PDS_STATE0_CLRMSK 0xFFFFFFFF00000000ULL
938
939/* Register ROGUE_CR_VDM_CONTEXT_RESUME_TASK1 */
940#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK1 0x0458U
941#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK1_MASKFULL 0x00000000FFFFFFFFULL
942#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK1_PDS_STATE2_SHIFT 0U
943#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK1_PDS_STATE2_CLRMSK 0x00000000U
944
945/* Register ROGUE_CR_VDM_CONTEXT_RESUME_TASK2 */
946#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK2 0x0460U
947#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK2_MASKFULL 0xFFFFFFFFFFFFFFFFULL
948#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK2_STREAM_OUT2_SHIFT 32U
949#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK2_STREAM_OUT2_CLRMSK 0x00000000FFFFFFFFULL
950#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK2_STREAM_OUT1_SHIFT 0U
951#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK2_STREAM_OUT1_CLRMSK 0xFFFFFFFF00000000ULL
952
953/* Register ROGUE_CR_CDM_CONTEXT_STORE_STATUS */
954#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS 0x04A0U
955#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_MASKFULL 0x0000000000000003ULL
956#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_NEED_RESUME_SHIFT 1U
957#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_NEED_RESUME_CLRMSK 0xFFFFFFFDU
958#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_NEED_RESUME_EN 0x00000002U
959#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_COMPLETE_SHIFT 0U
960#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_COMPLETE_CLRMSK 0xFFFFFFFEU
961#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_COMPLETE_EN 0x00000001U
962
963/* Register ROGUE_CR_CDM_CONTEXT_PDS0 */
964#define ROGUE_CR_CDM_CONTEXT_PDS0 0x04A8U
965#define ROGUE_CR_CDM_CONTEXT_PDS0_MASKFULL 0xFFFFFFF0FFFFFFF0ULL
966#define ROGUE_CR_CDM_CONTEXT_PDS0_DATA_ADDR_SHIFT 36U
967#define ROGUE_CR_CDM_CONTEXT_PDS0_DATA_ADDR_CLRMSK 0x0000000FFFFFFFFFULL
968#define ROGUE_CR_CDM_CONTEXT_PDS0_DATA_ADDR_ALIGNSHIFT 4U
969#define ROGUE_CR_CDM_CONTEXT_PDS0_DATA_ADDR_ALIGNSIZE 16U
970#define ROGUE_CR_CDM_CONTEXT_PDS0_CODE_ADDR_SHIFT 4U
971#define ROGUE_CR_CDM_CONTEXT_PDS0_CODE_ADDR_CLRMSK 0xFFFFFFFF0000000FULL
972#define ROGUE_CR_CDM_CONTEXT_PDS0_CODE_ADDR_ALIGNSHIFT 4U
973#define ROGUE_CR_CDM_CONTEXT_PDS0_CODE_ADDR_ALIGNSIZE 16U
974
975/* Register ROGUE_CR_CDM_CONTEXT_PDS1 */
976#define ROGUE_CR_CDM_CONTEXT_PDS1 0x04B0U
977#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__MASKFULL 0x000000007FFFFFFFULL
978#define ROGUE_CR_CDM_CONTEXT_PDS1_MASKFULL 0x000000003FFFFFFFULL
979#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__PDS_SEQ_DEP_SHIFT 30U
980#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__PDS_SEQ_DEP_CLRMSK 0xBFFFFFFFU
981#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__PDS_SEQ_DEP_EN 0x40000000U
982#define ROGUE_CR_CDM_CONTEXT_PDS1_PDS_SEQ_DEP_SHIFT 29U
983#define ROGUE_CR_CDM_CONTEXT_PDS1_PDS_SEQ_DEP_CLRMSK 0xDFFFFFFFU
984#define ROGUE_CR_CDM_CONTEXT_PDS1_PDS_SEQ_DEP_EN 0x20000000U
985#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__USC_SEQ_DEP_SHIFT 29U
986#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__USC_SEQ_DEP_CLRMSK 0xDFFFFFFFU
987#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__USC_SEQ_DEP_EN 0x20000000U
988#define ROGUE_CR_CDM_CONTEXT_PDS1_USC_SEQ_DEP_SHIFT 28U
989#define ROGUE_CR_CDM_CONTEXT_PDS1_USC_SEQ_DEP_CLRMSK 0xEFFFFFFFU
990#define ROGUE_CR_CDM_CONTEXT_PDS1_USC_SEQ_DEP_EN 0x10000000U
991#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__TARGET_SHIFT 28U
992#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__TARGET_CLRMSK 0xEFFFFFFFU
993#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__TARGET_EN 0x10000000U
994#define ROGUE_CR_CDM_CONTEXT_PDS1_TARGET_SHIFT 27U
995#define ROGUE_CR_CDM_CONTEXT_PDS1_TARGET_CLRMSK 0xF7FFFFFFU
996#define ROGUE_CR_CDM_CONTEXT_PDS1_TARGET_EN 0x08000000U
997#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__UNIFIED_SIZE_SHIFT 22U
998#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__UNIFIED_SIZE_CLRMSK 0xF03FFFFFU
999#define ROGUE_CR_CDM_CONTEXT_PDS1_UNIFIED_SIZE_SHIFT 21U
1000#define ROGUE_CR_CDM_CONTEXT_PDS1_UNIFIED_SIZE_CLRMSK 0xF81FFFFFU
1001#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__COMMON_SHARED_SHIFT 21U
1002#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__COMMON_SHARED_CLRMSK 0xFFDFFFFFU
1003#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__COMMON_SHARED_EN 0x00200000U
1004#define ROGUE_CR_CDM_CONTEXT_PDS1_COMMON_SHARED_SHIFT 20U
1005#define ROGUE_CR_CDM_CONTEXT_PDS1_COMMON_SHARED_CLRMSK 0xFFEFFFFFU
1006#define ROGUE_CR_CDM_CONTEXT_PDS1_COMMON_SHARED_EN 0x00100000U
1007#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__COMMON_SIZE_SHIFT 12U
1008#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__COMMON_SIZE_CLRMSK 0xFFE00FFFU
1009#define ROGUE_CR_CDM_CONTEXT_PDS1_COMMON_SIZE_SHIFT 11U
1010#define ROGUE_CR_CDM_CONTEXT_PDS1_COMMON_SIZE_CLRMSK 0xFFF007FFU
1011#define ROGUE_CR_CDM_CONTEXT_PDS1_TEMP_SIZE_SHIFT 7U
1012#define ROGUE_CR_CDM_CONTEXT_PDS1_TEMP_SIZE_CLRMSK 0xFFFFF87FU
1013#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__TEMP_SIZE_SHIFT 7U
1014#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__TEMP_SIZE_CLRMSK 0xFFFFF07FU
1015#define ROGUE_CR_CDM_CONTEXT_PDS1_DATA_SIZE_SHIFT 1U
1016#define ROGUE_CR_CDM_CONTEXT_PDS1_DATA_SIZE_CLRMSK 0xFFFFFF81U
1017#define ROGUE_CR_CDM_CONTEXT_PDS1_FENCE_SHIFT 0U
1018#define ROGUE_CR_CDM_CONTEXT_PDS1_FENCE_CLRMSK 0xFFFFFFFEU
1019#define ROGUE_CR_CDM_CONTEXT_PDS1_FENCE_EN 0x00000001U
1020
1021/* Register ROGUE_CR_CDM_TERMINATE_PDS */
1022#define ROGUE_CR_CDM_TERMINATE_PDS 0x04B8U
1023#define ROGUE_CR_CDM_TERMINATE_PDS_MASKFULL 0xFFFFFFF0FFFFFFF0ULL
1024#define ROGUE_CR_CDM_TERMINATE_PDS_DATA_ADDR_SHIFT 36U
1025#define ROGUE_CR_CDM_TERMINATE_PDS_DATA_ADDR_CLRMSK 0x0000000FFFFFFFFFULL
1026#define ROGUE_CR_CDM_TERMINATE_PDS_DATA_ADDR_ALIGNSHIFT 4U
1027#define ROGUE_CR_CDM_TERMINATE_PDS_DATA_ADDR_ALIGNSIZE 16U
1028#define ROGUE_CR_CDM_TERMINATE_PDS_CODE_ADDR_SHIFT 4U
1029#define ROGUE_CR_CDM_TERMINATE_PDS_CODE_ADDR_CLRMSK 0xFFFFFFFF0000000FULL
1030#define ROGUE_CR_CDM_TERMINATE_PDS_CODE_ADDR_ALIGNSHIFT 4U
1031#define ROGUE_CR_CDM_TERMINATE_PDS_CODE_ADDR_ALIGNSIZE 16U
1032
1033/* Register ROGUE_CR_CDM_TERMINATE_PDS1 */
1034#define ROGUE_CR_CDM_TERMINATE_PDS1 0x04C0U
1035#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__MASKFULL 0x000000007FFFFFFFULL
1036#define ROGUE_CR_CDM_TERMINATE_PDS1_MASKFULL 0x000000003FFFFFFFULL
1037#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__PDS_SEQ_DEP_SHIFT 30U
1038#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__PDS_SEQ_DEP_CLRMSK 0xBFFFFFFFU
1039#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__PDS_SEQ_DEP_EN 0x40000000U
1040#define ROGUE_CR_CDM_TERMINATE_PDS1_PDS_SEQ_DEP_SHIFT 29U
1041#define ROGUE_CR_CDM_TERMINATE_PDS1_PDS_SEQ_DEP_CLRMSK 0xDFFFFFFFU
1042#define ROGUE_CR_CDM_TERMINATE_PDS1_PDS_SEQ_DEP_EN 0x20000000U
1043#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__USC_SEQ_DEP_SHIFT 29U
1044#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__USC_SEQ_DEP_CLRMSK 0xDFFFFFFFU
1045#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__USC_SEQ_DEP_EN 0x20000000U
1046#define ROGUE_CR_CDM_TERMINATE_PDS1_USC_SEQ_DEP_SHIFT 28U
1047#define ROGUE_CR_CDM_TERMINATE_PDS1_USC_SEQ_DEP_CLRMSK 0xEFFFFFFFU
1048#define ROGUE_CR_CDM_TERMINATE_PDS1_USC_SEQ_DEP_EN 0x10000000U
1049#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__TARGET_SHIFT 28U
1050#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__TARGET_CLRMSK 0xEFFFFFFFU
1051#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__TARGET_EN 0x10000000U
1052#define ROGUE_CR_CDM_TERMINATE_PDS1_TARGET_SHIFT 27U
1053#define ROGUE_CR_CDM_TERMINATE_PDS1_TARGET_CLRMSK 0xF7FFFFFFU
1054#define ROGUE_CR_CDM_TERMINATE_PDS1_TARGET_EN 0x08000000U
1055#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__UNIFIED_SIZE_SHIFT 22U
1056#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__UNIFIED_SIZE_CLRMSK 0xF03FFFFFU
1057#define ROGUE_CR_CDM_TERMINATE_PDS1_UNIFIED_SIZE_SHIFT 21U
1058#define ROGUE_CR_CDM_TERMINATE_PDS1_UNIFIED_SIZE_CLRMSK 0xF81FFFFFU
1059#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__COMMON_SHARED_SHIFT 21U
1060#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__COMMON_SHARED_CLRMSK 0xFFDFFFFFU
1061#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__COMMON_SHARED_EN 0x00200000U
1062#define ROGUE_CR_CDM_TERMINATE_PDS1_COMMON_SHARED_SHIFT 20U
1063#define ROGUE_CR_CDM_TERMINATE_PDS1_COMMON_SHARED_CLRMSK 0xFFEFFFFFU
1064#define ROGUE_CR_CDM_TERMINATE_PDS1_COMMON_SHARED_EN 0x00100000U
1065#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__COMMON_SIZE_SHIFT 12U
1066#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__COMMON_SIZE_CLRMSK 0xFFE00FFFU
1067#define ROGUE_CR_CDM_TERMINATE_PDS1_COMMON_SIZE_SHIFT 11U
1068#define ROGUE_CR_CDM_TERMINATE_PDS1_COMMON_SIZE_CLRMSK 0xFFF007FFU
1069#define ROGUE_CR_CDM_TERMINATE_PDS1_TEMP_SIZE_SHIFT 7U
1070#define ROGUE_CR_CDM_TERMINATE_PDS1_TEMP_SIZE_CLRMSK 0xFFFFF87FU
1071#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__TEMP_SIZE_SHIFT 7U
1072#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__TEMP_SIZE_CLRMSK 0xFFFFF07FU
1073#define ROGUE_CR_CDM_TERMINATE_PDS1_DATA_SIZE_SHIFT 1U
1074#define ROGUE_CR_CDM_TERMINATE_PDS1_DATA_SIZE_CLRMSK 0xFFFFFF81U
1075#define ROGUE_CR_CDM_TERMINATE_PDS1_FENCE_SHIFT 0U
1076#define ROGUE_CR_CDM_TERMINATE_PDS1_FENCE_CLRMSK 0xFFFFFFFEU
1077#define ROGUE_CR_CDM_TERMINATE_PDS1_FENCE_EN 0x00000001U
1078
1079/* Register ROGUE_CR_CDM_CONTEXT_LOAD_PDS0 */
1080#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0 0x04D8U
1081#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_MASKFULL 0xFFFFFFF0FFFFFFF0ULL
1082#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_DATA_ADDR_SHIFT 36U
1083#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_DATA_ADDR_CLRMSK 0x0000000FFFFFFFFFULL
1084#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_DATA_ADDR_ALIGNSHIFT 4U
1085#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_DATA_ADDR_ALIGNSIZE 16U
1086#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_CODE_ADDR_SHIFT 4U
1087#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_CODE_ADDR_CLRMSK 0xFFFFFFFF0000000FULL
1088#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_CODE_ADDR_ALIGNSHIFT 4U
1089#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_CODE_ADDR_ALIGNSIZE 16U
1090
1091/* Register ROGUE_CR_CDM_CONTEXT_LOAD_PDS1 */
1092#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1 0x04E0U
1093#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__MASKFULL 0x000000007FFFFFFFULL
1094#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_MASKFULL 0x000000003FFFFFFFULL
1095#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__PDS_SEQ_DEP_SHIFT 30U
1096#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__PDS_SEQ_DEP_CLRMSK 0xBFFFFFFFU
1097#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__PDS_SEQ_DEP_EN 0x40000000U
1098#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_PDS_SEQ_DEP_SHIFT 29U
1099#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_PDS_SEQ_DEP_CLRMSK 0xDFFFFFFFU
1100#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_PDS_SEQ_DEP_EN 0x20000000U
1101#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__USC_SEQ_DEP_SHIFT 29U
1102#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__USC_SEQ_DEP_CLRMSK 0xDFFFFFFFU
1103#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__USC_SEQ_DEP_EN 0x20000000U
1104#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_USC_SEQ_DEP_SHIFT 28U
1105#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_USC_SEQ_DEP_CLRMSK 0xEFFFFFFFU
1106#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_USC_SEQ_DEP_EN 0x10000000U
1107#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__TARGET_SHIFT 28U
1108#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__TARGET_CLRMSK 0xEFFFFFFFU
1109#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__TARGET_EN 0x10000000U
1110#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_TARGET_SHIFT 27U
1111#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_TARGET_CLRMSK 0xF7FFFFFFU
1112#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_TARGET_EN 0x08000000U
1113#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__UNIFIED_SIZE_SHIFT 22U
1114#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__UNIFIED_SIZE_CLRMSK 0xF03FFFFFU
1115#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_UNIFIED_SIZE_SHIFT 21U
1116#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_UNIFIED_SIZE_CLRMSK 0xF81FFFFFU
1117#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__COMMON_SHARED_SHIFT 21U
1118#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__COMMON_SHARED_CLRMSK 0xFFDFFFFFU
1119#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__COMMON_SHARED_EN 0x00200000U
1120#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_COMMON_SHARED_SHIFT 20U
1121#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_COMMON_SHARED_CLRMSK 0xFFEFFFFFU
1122#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_COMMON_SHARED_EN 0x00100000U
1123#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__COMMON_SIZE_SHIFT 12U
1124#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__COMMON_SIZE_CLRMSK 0xFFE00FFFU
1125#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_COMMON_SIZE_SHIFT 11U
1126#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_COMMON_SIZE_CLRMSK 0xFFF007FFU
1127#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_TEMP_SIZE_SHIFT 7U
1128#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_TEMP_SIZE_CLRMSK 0xFFFFF87FU
1129#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__TEMP_SIZE_SHIFT 7U
1130#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__TEMP_SIZE_CLRMSK 0xFFFFF07FU
1131#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_DATA_SIZE_SHIFT 1U
1132#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_DATA_SIZE_CLRMSK 0xFFFFFF81U
1133#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_FENCE_SHIFT 0U
1134#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_FENCE_CLRMSK 0xFFFFFFFEU
1135#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_FENCE_EN 0x00000001U
1136
1137/* Register ROGUE_CR_MIPS_WRAPPER_CONFIG */
1138#define ROGUE_CR_MIPS_WRAPPER_CONFIG 0x0810U
1139#define ROGUE_CR_MIPS_WRAPPER_CONFIG_MASKFULL 0x000001030F01FFFFULL
1140#define ROGUE_CR_MIPS_WRAPPER_CONFIG_FW_IDLE_ENABLE_SHIFT 40U
1141#define ROGUE_CR_MIPS_WRAPPER_CONFIG_FW_IDLE_ENABLE_CLRMSK 0xFFFFFEFFFFFFFFFFULL
1142#define ROGUE_CR_MIPS_WRAPPER_CONFIG_FW_IDLE_ENABLE_EN 0x0000010000000000ULL
1143#define ROGUE_CR_MIPS_WRAPPER_CONFIG_DISABLE_BOOT_SHIFT 33U
1144#define ROGUE_CR_MIPS_WRAPPER_CONFIG_DISABLE_BOOT_CLRMSK 0xFFFFFFFDFFFFFFFFULL
1145#define ROGUE_CR_MIPS_WRAPPER_CONFIG_DISABLE_BOOT_EN 0x0000000200000000ULL
1146#define ROGUE_CR_MIPS_WRAPPER_CONFIG_L2_CACHE_OFF_SHIFT 32U
1147#define ROGUE_CR_MIPS_WRAPPER_CONFIG_L2_CACHE_OFF_CLRMSK 0xFFFFFFFEFFFFFFFFULL
1148#define ROGUE_CR_MIPS_WRAPPER_CONFIG_L2_CACHE_OFF_EN 0x0000000100000000ULL
1149#define ROGUE_CR_MIPS_WRAPPER_CONFIG_OS_ID_SHIFT 25U
1150#define ROGUE_CR_MIPS_WRAPPER_CONFIG_OS_ID_CLRMSK 0xFFFFFFFFF1FFFFFFULL
1151#define ROGUE_CR_MIPS_WRAPPER_CONFIG_TRUSTED_SHIFT 24U
1152#define ROGUE_CR_MIPS_WRAPPER_CONFIG_TRUSTED_CLRMSK 0xFFFFFFFFFEFFFFFFULL
1153#define ROGUE_CR_MIPS_WRAPPER_CONFIG_TRUSTED_EN 0x0000000001000000ULL
1154#define ROGUE_CR_MIPS_WRAPPER_CONFIG_BOOT_ISA_MODE_SHIFT 16U
1155#define ROGUE_CR_MIPS_WRAPPER_CONFIG_BOOT_ISA_MODE_CLRMSK 0xFFFFFFFFFFFEFFFFULL
1156#define ROGUE_CR_MIPS_WRAPPER_CONFIG_BOOT_ISA_MODE_MIPS32 0x0000000000000000ULL
1157#define ROGUE_CR_MIPS_WRAPPER_CONFIG_BOOT_ISA_MODE_MICROMIPS 0x0000000000010000ULL
1158#define ROGUE_CR_MIPS_WRAPPER_CONFIG_REGBANK_BASE_ADDR_SHIFT 0U
1159#define ROGUE_CR_MIPS_WRAPPER_CONFIG_REGBANK_BASE_ADDR_CLRMSK 0xFFFFFFFFFFFF0000ULL
1160
1161/* Register ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1 */
1162#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1 0x0818U
1163#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_MASKFULL 0x00000000FFFFF001ULL
1164#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_BASE_ADDR_IN_SHIFT 12U
1165#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1166#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_MODE_ENABLE_SHIFT 0U
1167#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1168#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_MODE_ENABLE_EN 0x0000000000000001ULL
1169
1170/* Register ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2 */
1171#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2 0x0820U
1172#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_MASKFULL 0x000000FFFFFFF1FFULL
1173#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_ADDR_OUT_SHIFT 12U
1174#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_ADDR_OUT_CLRMSK 0xFFFFFF0000000FFFULL
1175#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_OS_ID_SHIFT 6U
1176#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_OS_ID_CLRMSK 0xFFFFFFFFFFFFFE3FULL
1177#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_TRUSTED_SHIFT 5U
1178#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_TRUSTED_CLRMSK 0xFFFFFFFFFFFFFFDFULL
1179#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_TRUSTED_EN 0x0000000000000020ULL
1180#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_REGION_SIZE_POW2_SHIFT 0U
1181#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_REGION_SIZE_POW2_CLRMSK 0xFFFFFFFFFFFFFFE0ULL
1182
1183/* Register ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1 */
1184#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1 0x0828U
1185#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_MASKFULL 0x00000000FFFFF001ULL
1186#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_BASE_ADDR_IN_SHIFT 12U
1187#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1188#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_MODE_ENABLE_SHIFT 0U
1189#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1190#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_MODE_ENABLE_EN 0x0000000000000001ULL
1191
1192/* Register ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2 */
1193#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2 0x0830U
1194#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_MASKFULL 0x000000FFFFFFF1FFULL
1195#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_ADDR_OUT_SHIFT 12U
1196#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_ADDR_OUT_CLRMSK 0xFFFFFF0000000FFFULL
1197#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_OS_ID_SHIFT 6U
1198#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_OS_ID_CLRMSK 0xFFFFFFFFFFFFFE3FULL
1199#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_TRUSTED_SHIFT 5U
1200#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_TRUSTED_CLRMSK 0xFFFFFFFFFFFFFFDFULL
1201#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_TRUSTED_EN 0x0000000000000020ULL
1202#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_REGION_SIZE_POW2_SHIFT 0U
1203#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_REGION_SIZE_POW2_CLRMSK 0xFFFFFFFFFFFFFFE0ULL
1204
1205/* Register ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1 */
1206#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1 0x0838U
1207#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_MASKFULL 0x00000000FFFFF001ULL
1208#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_BASE_ADDR_IN_SHIFT 12U
1209#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1210#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_MODE_ENABLE_SHIFT 0U
1211#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1212#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_MODE_ENABLE_EN 0x0000000000000001ULL
1213
1214/* Register ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2 */
1215#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2 0x0840U
1216#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_MASKFULL 0x000000FFFFFFF1FFULL
1217#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_ADDR_OUT_SHIFT 12U
1218#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_ADDR_OUT_CLRMSK 0xFFFFFF0000000FFFULL
1219#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_OS_ID_SHIFT 6U
1220#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_OS_ID_CLRMSK 0xFFFFFFFFFFFFFE3FULL
1221#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_TRUSTED_SHIFT 5U
1222#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_TRUSTED_CLRMSK 0xFFFFFFFFFFFFFFDFULL
1223#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_TRUSTED_EN 0x0000000000000020ULL
1224#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_REGION_SIZE_POW2_SHIFT 0U
1225#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_REGION_SIZE_POW2_CLRMSK 0xFFFFFFFFFFFFFFE0ULL
1226
1227/* Register ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1 */
1228#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1 0x0848U
1229#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1_MASKFULL 0x00000000FFFFF001ULL
1230#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1_BASE_ADDR_IN_SHIFT 12U
1231#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1232#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1_MODE_ENABLE_SHIFT 0U
1233#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1234#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1_MODE_ENABLE_EN 0x0000000000000001ULL
1235
1236/* Register ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2 */
1237#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2 0x0850U
1238#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_MASKFULL 0x000000FFFFFFF1FFULL
1239#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_ADDR_OUT_SHIFT 12U
1240#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_ADDR_OUT_CLRMSK 0xFFFFFF0000000FFFULL
1241#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_OS_ID_SHIFT 6U
1242#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_OS_ID_CLRMSK 0xFFFFFFFFFFFFFE3FULL
1243#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_TRUSTED_SHIFT 5U
1244#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_TRUSTED_CLRMSK 0xFFFFFFFFFFFFFFDFULL
1245#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_TRUSTED_EN 0x0000000000000020ULL
1246#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_REGION_SIZE_POW2_SHIFT 0U
1247#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_REGION_SIZE_POW2_CLRMSK 0xFFFFFFFFFFFFFFE0ULL
1248
1249/* Register ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1 */
1250#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1 0x0858U
1251#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_MASKFULL 0x00000000FFFFF001ULL
1252#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_BASE_ADDR_IN_SHIFT 12U
1253#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1254#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_MODE_ENABLE_SHIFT 0U
1255#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1256#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_MODE_ENABLE_EN 0x0000000000000001ULL
1257
1258/* Register ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2 */
1259#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2 0x0860U
1260#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_MASKFULL 0x000000FFFFFFF1FFULL
1261#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_ADDR_OUT_SHIFT 12U
1262#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_ADDR_OUT_CLRMSK 0xFFFFFF0000000FFFULL
1263#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_OS_ID_SHIFT 6U
1264#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_OS_ID_CLRMSK 0xFFFFFFFFFFFFFE3FULL
1265#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_TRUSTED_SHIFT 5U
1266#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_TRUSTED_CLRMSK 0xFFFFFFFFFFFFFFDFULL
1267#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_TRUSTED_EN 0x0000000000000020ULL
1268#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_REGION_SIZE_POW2_SHIFT 0U
1269#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_REGION_SIZE_POW2_CLRMSK 0xFFFFFFFFFFFFFFE0ULL
1270
1271/* Register ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS */
1272#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS 0x0868U
1273#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS_MASKFULL 0x00000001FFFFFFFFULL
1274#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS_EVENT_SHIFT 32U
1275#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS_EVENT_CLRMSK 0xFFFFFFFEFFFFFFFFULL
1276#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS_EVENT_EN 0x0000000100000000ULL
1277#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS_ADDRESS_SHIFT 0U
1278#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS_ADDRESS_CLRMSK 0xFFFFFFFF00000000ULL
1279
1280/* Register ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_CLEAR */
1281#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_CLEAR 0x0870U
1282#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_CLEAR_MASKFULL 0x0000000000000001ULL
1283#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_CLEAR_EVENT_SHIFT 0U
1284#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_CLEAR_EVENT_CLRMSK 0xFFFFFFFEU
1285#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_CLEAR_EVENT_EN 0x00000001U
1286
1287/* Register ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG */
1288#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG 0x0878U
1289#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_MASKFULL 0xFFFFFFF7FFFFFFBFULL
1290#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_ADDR_OUT_SHIFT 36U
1291#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_ADDR_OUT_CLRMSK 0x0000000FFFFFFFFFULL
1292#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_OS_ID_SHIFT 32U
1293#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_OS_ID_CLRMSK 0xFFFFFFF8FFFFFFFFULL
1294#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_BASE_ADDR_IN_SHIFT 12U
1295#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1296#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_TRUSTED_SHIFT 11U
1297#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_TRUSTED_CLRMSK 0xFFFFFFFFFFFFF7FFULL
1298#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_TRUSTED_EN 0x0000000000000800ULL
1299#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_SHIFT 7U
1300#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_CLRMSK 0xFFFFFFFFFFFFF87FULL
1301#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_4KB 0x0000000000000000ULL
1302#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_16KB 0x0000000000000080ULL
1303#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_64KB 0x0000000000000100ULL
1304#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_256KB 0x0000000000000180ULL
1305#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_1MB 0x0000000000000200ULL
1306#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_4MB 0x0000000000000280ULL
1307#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_16MB 0x0000000000000300ULL
1308#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_64MB 0x0000000000000380ULL
1309#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_256MB 0x0000000000000400ULL
1310#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_ENTRY_SHIFT 1U
1311#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_ENTRY_CLRMSK 0xFFFFFFFFFFFFFFC1ULL
1312#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_MODE_ENABLE_SHIFT 0U
1313#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1314#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_MODE_ENABLE_EN 0x0000000000000001ULL
1315
1316/* Register ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ */
1317#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ 0x0880U
1318#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ_MASKFULL 0x000000000000003FULL
1319#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ_ENTRY_SHIFT 1U
1320#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ_ENTRY_CLRMSK 0xFFFFFFC1U
1321#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ_REQUEST_SHIFT 0U
1322#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ_REQUEST_CLRMSK 0xFFFFFFFEU
1323#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ_REQUEST_EN 0x00000001U
1324
1325/* Register ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA */
1326#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA 0x0888U
1327#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_MASKFULL 0xFFFFFFF7FFFFFF81ULL
1328#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_ADDR_OUT_SHIFT 36U
1329#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_ADDR_OUT_CLRMSK 0x0000000FFFFFFFFFULL
1330#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_OS_ID_SHIFT 32U
1331#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_OS_ID_CLRMSK 0xFFFFFFF8FFFFFFFFULL
1332#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_BASE_ADDR_IN_SHIFT 12U
1333#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1334#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_TRUSTED_SHIFT 11U
1335#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_TRUSTED_CLRMSK 0xFFFFFFFFFFFFF7FFULL
1336#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_TRUSTED_EN 0x0000000000000800ULL
1337#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_REGION_SIZE_SHIFT 7U
1338#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_REGION_SIZE_CLRMSK 0xFFFFFFFFFFFFF87FULL
1339#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_MODE_ENABLE_SHIFT 0U
1340#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1341#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_MODE_ENABLE_EN 0x0000000000000001ULL
1342
1343/* Register ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE */
1344#define ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE 0x08A0U
1345#define ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE_MASKFULL 0x0000000000000001ULL
1346#define ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE_EVENT_SHIFT 0U
1347#define ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE_EVENT_CLRMSK 0xFFFFFFFEU
1348#define ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE_EVENT_EN 0x00000001U
1349
1350/* Register ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS */
1351#define ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS 0x08A8U
1352#define ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS_MASKFULL 0x0000000000000001ULL
1353#define ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS_EVENT_SHIFT 0U
1354#define ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS_EVENT_CLRMSK 0xFFFFFFFEU
1355#define ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS_EVENT_EN 0x00000001U
1356
1357/* Register ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR */
1358#define ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR 0x08B0U
1359#define ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR_MASKFULL 0x0000000000000001ULL
1360#define ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR_EVENT_SHIFT 0U
1361#define ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR_EVENT_CLRMSK 0xFFFFFFFEU
1362#define ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR_EVENT_EN 0x00000001U
1363
1364/* Register ROGUE_CR_MIPS_WRAPPER_NMI_ENABLE */
1365#define ROGUE_CR_MIPS_WRAPPER_NMI_ENABLE 0x08B8U
1366#define ROGUE_CR_MIPS_WRAPPER_NMI_ENABLE_MASKFULL 0x0000000000000001ULL
1367#define ROGUE_CR_MIPS_WRAPPER_NMI_ENABLE_EVENT_SHIFT 0U
1368#define ROGUE_CR_MIPS_WRAPPER_NMI_ENABLE_EVENT_CLRMSK 0xFFFFFFFEU
1369#define ROGUE_CR_MIPS_WRAPPER_NMI_ENABLE_EVENT_EN 0x00000001U
1370
1371/* Register ROGUE_CR_MIPS_WRAPPER_NMI_EVENT */
1372#define ROGUE_CR_MIPS_WRAPPER_NMI_EVENT 0x08C0U
1373#define ROGUE_CR_MIPS_WRAPPER_NMI_EVENT_MASKFULL 0x0000000000000001ULL
1374#define ROGUE_CR_MIPS_WRAPPER_NMI_EVENT_TRIGGER_SHIFT 0U
1375#define ROGUE_CR_MIPS_WRAPPER_NMI_EVENT_TRIGGER_CLRMSK 0xFFFFFFFEU
1376#define ROGUE_CR_MIPS_WRAPPER_NMI_EVENT_TRIGGER_EN 0x00000001U
1377
1378/* Register ROGUE_CR_MIPS_DEBUG_CONFIG */
1379#define ROGUE_CR_MIPS_DEBUG_CONFIG 0x08C8U
1380#define ROGUE_CR_MIPS_DEBUG_CONFIG_MASKFULL 0x0000000000000001ULL
1381#define ROGUE_CR_MIPS_DEBUG_CONFIG_DISABLE_PROBE_DEBUG_SHIFT 0U
1382#define ROGUE_CR_MIPS_DEBUG_CONFIG_DISABLE_PROBE_DEBUG_CLRMSK 0xFFFFFFFEU
1383#define ROGUE_CR_MIPS_DEBUG_CONFIG_DISABLE_PROBE_DEBUG_EN 0x00000001U
1384
1385/* Register ROGUE_CR_MIPS_EXCEPTION_STATUS */
1386#define ROGUE_CR_MIPS_EXCEPTION_STATUS 0x08D0U
1387#define ROGUE_CR_MIPS_EXCEPTION_STATUS_MASKFULL 0x000000000000003FULL
1388#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_SLEEP_SHIFT 5U
1389#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_SLEEP_CLRMSK 0xFFFFFFDFU
1390#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_SLEEP_EN 0x00000020U
1391#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NMI_TAKEN_SHIFT 4U
1392#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NMI_TAKEN_CLRMSK 0xFFFFFFEFU
1393#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NMI_TAKEN_EN 0x00000010U
1394#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NEST_EXL_SHIFT 3U
1395#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NEST_EXL_CLRMSK 0xFFFFFFF7U
1396#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NEST_EXL_EN 0x00000008U
1397#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NEST_ERL_SHIFT 2U
1398#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NEST_ERL_CLRMSK 0xFFFFFFFBU
1399#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NEST_ERL_EN 0x00000004U
1400#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_EXL_SHIFT 1U
1401#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_EXL_CLRMSK 0xFFFFFFFDU
1402#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_EXL_EN 0x00000002U
1403#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_ERL_SHIFT 0U
1404#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_ERL_CLRMSK 0xFFFFFFFEU
1405#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_ERL_EN 0x00000001U
1406
1407/* Register ROGUE_CR_MIPS_WRAPPER_STATUS */
1408#define ROGUE_CR_MIPS_WRAPPER_STATUS 0x08E8U
1409#define ROGUE_CR_MIPS_WRAPPER_STATUS_MASKFULL 0x00000000000000FFULL
1410#define ROGUE_CR_MIPS_WRAPPER_STATUS_OUTSTANDING_REQUESTS_SHIFT 0U
1411#define ROGUE_CR_MIPS_WRAPPER_STATUS_OUTSTANDING_REQUESTS_CLRMSK 0xFFFFFF00U
1412
1413/* Register ROGUE_CR_XPU_BROADCAST */
1414#define ROGUE_CR_XPU_BROADCAST 0x0890U
1415#define ROGUE_CR_XPU_BROADCAST_MASKFULL 0x00000000000001FFULL
1416#define ROGUE_CR_XPU_BROADCAST_MASK_SHIFT 0U
1417#define ROGUE_CR_XPU_BROADCAST_MASK_CLRMSK 0xFFFFFE00U
1418
1419/* Register ROGUE_CR_META_SP_MSLVDATAX */
1420#define ROGUE_CR_META_SP_MSLVDATAX 0x0A00U
1421#define ROGUE_CR_META_SP_MSLVDATAX_MASKFULL 0x00000000FFFFFFFFULL
1422#define ROGUE_CR_META_SP_MSLVDATAX_MSLVDATAX_SHIFT 0U
1423#define ROGUE_CR_META_SP_MSLVDATAX_MSLVDATAX_CLRMSK 0x00000000U
1424
1425/* Register ROGUE_CR_META_SP_MSLVDATAT */
1426#define ROGUE_CR_META_SP_MSLVDATAT 0x0A08U
1427#define ROGUE_CR_META_SP_MSLVDATAT_MASKFULL 0x00000000FFFFFFFFULL
1428#define ROGUE_CR_META_SP_MSLVDATAT_MSLVDATAT_SHIFT 0U
1429#define ROGUE_CR_META_SP_MSLVDATAT_MSLVDATAT_CLRMSK 0x00000000U
1430
1431/* Register ROGUE_CR_META_SP_MSLVCTRL0 */
1432#define ROGUE_CR_META_SP_MSLVCTRL0 0x0A10U
1433#define ROGUE_CR_META_SP_MSLVCTRL0_MASKFULL 0x00000000FFFFFFFFULL
1434#define ROGUE_CR_META_SP_MSLVCTRL0_ADDR_SHIFT 2U
1435#define ROGUE_CR_META_SP_MSLVCTRL0_ADDR_CLRMSK 0x00000003U
1436#define ROGUE_CR_META_SP_MSLVCTRL0_AUTOINCR_SHIFT 1U
1437#define ROGUE_CR_META_SP_MSLVCTRL0_AUTOINCR_CLRMSK 0xFFFFFFFDU
1438#define ROGUE_CR_META_SP_MSLVCTRL0_AUTOINCR_EN 0x00000002U
1439#define ROGUE_CR_META_SP_MSLVCTRL0_RD_SHIFT 0U
1440#define ROGUE_CR_META_SP_MSLVCTRL0_RD_CLRMSK 0xFFFFFFFEU
1441#define ROGUE_CR_META_SP_MSLVCTRL0_RD_EN 0x00000001U
1442
1443/* Register ROGUE_CR_META_SP_MSLVCTRL1 */
1444#define ROGUE_CR_META_SP_MSLVCTRL1 0x0A18U
1445#define ROGUE_CR_META_SP_MSLVCTRL1_MASKFULL 0x00000000F7F4003FULL
1446#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERRTHREAD_SHIFT 30U
1447#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERRTHREAD_CLRMSK 0x3FFFFFFFU
1448#define ROGUE_CR_META_SP_MSLVCTRL1_LOCK2_INTERLOCK_SHIFT 29U
1449#define ROGUE_CR_META_SP_MSLVCTRL1_LOCK2_INTERLOCK_CLRMSK 0xDFFFFFFFU
1450#define ROGUE_CR_META_SP_MSLVCTRL1_LOCK2_INTERLOCK_EN 0x20000000U
1451#define ROGUE_CR_META_SP_MSLVCTRL1_ATOMIC_INTERLOCK_SHIFT 28U
1452#define ROGUE_CR_META_SP_MSLVCTRL1_ATOMIC_INTERLOCK_CLRMSK 0xEFFFFFFFU
1453#define ROGUE_CR_META_SP_MSLVCTRL1_ATOMIC_INTERLOCK_EN 0x10000000U
1454#define ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_SHIFT 26U
1455#define ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_CLRMSK 0xFBFFFFFFU
1456#define ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_EN 0x04000000U
1457#define ROGUE_CR_META_SP_MSLVCTRL1_COREMEM_IDLE_SHIFT 25U
1458#define ROGUE_CR_META_SP_MSLVCTRL1_COREMEM_IDLE_CLRMSK 0xFDFFFFFFU
1459#define ROGUE_CR_META_SP_MSLVCTRL1_COREMEM_IDLE_EN 0x02000000U
1460#define ROGUE_CR_META_SP_MSLVCTRL1_READY_SHIFT 24U
1461#define ROGUE_CR_META_SP_MSLVCTRL1_READY_CLRMSK 0xFEFFFFFFU
1462#define ROGUE_CR_META_SP_MSLVCTRL1_READY_EN 0x01000000U
1463#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERRID_SHIFT 21U
1464#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERRID_CLRMSK 0xFF1FFFFFU
1465#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERR_SHIFT 20U
1466#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERR_CLRMSK 0xFFEFFFFFU
1467#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERR_EN 0x00100000U
1468#define ROGUE_CR_META_SP_MSLVCTRL1_WR_ACTIVE_SHIFT 18U
1469#define ROGUE_CR_META_SP_MSLVCTRL1_WR_ACTIVE_CLRMSK 0xFFFBFFFFU
1470#define ROGUE_CR_META_SP_MSLVCTRL1_WR_ACTIVE_EN 0x00040000U
1471#define ROGUE_CR_META_SP_MSLVCTRL1_THREAD_SHIFT 4U
1472#define ROGUE_CR_META_SP_MSLVCTRL1_THREAD_CLRMSK 0xFFFFFFCFU
1473#define ROGUE_CR_META_SP_MSLVCTRL1_TRANS_SIZE_SHIFT 2U
1474#define ROGUE_CR_META_SP_MSLVCTRL1_TRANS_SIZE_CLRMSK 0xFFFFFFF3U
1475#define ROGUE_CR_META_SP_MSLVCTRL1_BYTE_ROUND_SHIFT 0U
1476#define ROGUE_CR_META_SP_MSLVCTRL1_BYTE_ROUND_CLRMSK 0xFFFFFFFCU
1477
1478/* Register ROGUE_CR_META_SP_MSLVHANDSHKE */
1479#define ROGUE_CR_META_SP_MSLVHANDSHKE 0x0A50U
1480#define ROGUE_CR_META_SP_MSLVHANDSHKE_MASKFULL 0x000000000000000FULL
1481#define ROGUE_CR_META_SP_MSLVHANDSHKE_INPUT_SHIFT 2U
1482#define ROGUE_CR_META_SP_MSLVHANDSHKE_INPUT_CLRMSK 0xFFFFFFF3U
1483#define ROGUE_CR_META_SP_MSLVHANDSHKE_OUTPUT_SHIFT 0U
1484#define ROGUE_CR_META_SP_MSLVHANDSHKE_OUTPUT_CLRMSK 0xFFFFFFFCU
1485
1486/* Register ROGUE_CR_META_SP_MSLVT0KICK */
1487#define ROGUE_CR_META_SP_MSLVT0KICK 0x0A80U
1488#define ROGUE_CR_META_SP_MSLVT0KICK_MASKFULL 0x000000000000FFFFULL
1489#define ROGUE_CR_META_SP_MSLVT0KICK_MSLVT0KICK_SHIFT 0U
1490#define ROGUE_CR_META_SP_MSLVT0KICK_MSLVT0KICK_CLRMSK 0xFFFF0000U
1491
1492/* Register ROGUE_CR_META_SP_MSLVT0KICKI */
1493#define ROGUE_CR_META_SP_MSLVT0KICKI 0x0A88U
1494#define ROGUE_CR_META_SP_MSLVT0KICKI_MASKFULL 0x000000000000FFFFULL
1495#define ROGUE_CR_META_SP_MSLVT0KICKI_MSLVT0KICKI_SHIFT 0U
1496#define ROGUE_CR_META_SP_MSLVT0KICKI_MSLVT0KICKI_CLRMSK 0xFFFF0000U
1497
1498/* Register ROGUE_CR_META_SP_MSLVT1KICK */
1499#define ROGUE_CR_META_SP_MSLVT1KICK 0x0A90U
1500#define ROGUE_CR_META_SP_MSLVT1KICK_MASKFULL 0x000000000000FFFFULL
1501#define ROGUE_CR_META_SP_MSLVT1KICK_MSLVT1KICK_SHIFT 0U
1502#define ROGUE_CR_META_SP_MSLVT1KICK_MSLVT1KICK_CLRMSK 0xFFFF0000U
1503
1504/* Register ROGUE_CR_META_SP_MSLVT1KICKI */
1505#define ROGUE_CR_META_SP_MSLVT1KICKI 0x0A98U
1506#define ROGUE_CR_META_SP_MSLVT1KICKI_MASKFULL 0x000000000000FFFFULL
1507#define ROGUE_CR_META_SP_MSLVT1KICKI_MSLVT1KICKI_SHIFT 0U
1508#define ROGUE_CR_META_SP_MSLVT1KICKI_MSLVT1KICKI_CLRMSK 0xFFFF0000U
1509
1510/* Register ROGUE_CR_META_SP_MSLVT2KICK */
1511#define ROGUE_CR_META_SP_MSLVT2KICK 0x0AA0U
1512#define ROGUE_CR_META_SP_MSLVT2KICK_MASKFULL 0x000000000000FFFFULL
1513#define ROGUE_CR_META_SP_MSLVT2KICK_MSLVT2KICK_SHIFT 0U
1514#define ROGUE_CR_META_SP_MSLVT2KICK_MSLVT2KICK_CLRMSK 0xFFFF0000U
1515
1516/* Register ROGUE_CR_META_SP_MSLVT2KICKI */
1517#define ROGUE_CR_META_SP_MSLVT2KICKI 0x0AA8U
1518#define ROGUE_CR_META_SP_MSLVT2KICKI_MASKFULL 0x000000000000FFFFULL
1519#define ROGUE_CR_META_SP_MSLVT2KICKI_MSLVT2KICKI_SHIFT 0U
1520#define ROGUE_CR_META_SP_MSLVT2KICKI_MSLVT2KICKI_CLRMSK 0xFFFF0000U
1521
1522/* Register ROGUE_CR_META_SP_MSLVT3KICK */
1523#define ROGUE_CR_META_SP_MSLVT3KICK 0x0AB0U
1524#define ROGUE_CR_META_SP_MSLVT3KICK_MASKFULL 0x000000000000FFFFULL
1525#define ROGUE_CR_META_SP_MSLVT3KICK_MSLVT3KICK_SHIFT 0U
1526#define ROGUE_CR_META_SP_MSLVT3KICK_MSLVT3KICK_CLRMSK 0xFFFF0000U
1527
1528/* Register ROGUE_CR_META_SP_MSLVT3KICKI */
1529#define ROGUE_CR_META_SP_MSLVT3KICKI 0x0AB8U
1530#define ROGUE_CR_META_SP_MSLVT3KICKI_MASKFULL 0x000000000000FFFFULL
1531#define ROGUE_CR_META_SP_MSLVT3KICKI_MSLVT3KICKI_SHIFT 0U
1532#define ROGUE_CR_META_SP_MSLVT3KICKI_MSLVT3KICKI_CLRMSK 0xFFFF0000U
1533
1534/* Register ROGUE_CR_META_SP_MSLVRST */
1535#define ROGUE_CR_META_SP_MSLVRST 0x0AC0U
1536#define ROGUE_CR_META_SP_MSLVRST_MASKFULL 0x0000000000000001ULL
1537#define ROGUE_CR_META_SP_MSLVRST_SOFTRESET_SHIFT 0U
1538#define ROGUE_CR_META_SP_MSLVRST_SOFTRESET_CLRMSK 0xFFFFFFFEU
1539#define ROGUE_CR_META_SP_MSLVRST_SOFTRESET_EN 0x00000001U
1540
1541/* Register ROGUE_CR_META_SP_MSLVIRQSTATUS */
1542#define ROGUE_CR_META_SP_MSLVIRQSTATUS 0x0AC8U
1543#define ROGUE_CR_META_SP_MSLVIRQSTATUS_MASKFULL 0x000000000000000CULL
1544#define ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT3_SHIFT 3U
1545#define ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT3_CLRMSK 0xFFFFFFF7U
1546#define ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT3_EN 0x00000008U
1547#define ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT2_SHIFT 2U
1548#define ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT2_CLRMSK 0xFFFFFFFBU
1549#define ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT2_EN 0x00000004U
1550
1551/* Register ROGUE_CR_META_SP_MSLVIRQENABLE */
1552#define ROGUE_CR_META_SP_MSLVIRQENABLE 0x0AD0U
1553#define ROGUE_CR_META_SP_MSLVIRQENABLE_MASKFULL 0x000000000000000CULL
1554#define ROGUE_CR_META_SP_MSLVIRQENABLE_EVENT1_SHIFT 3U
1555#define ROGUE_CR_META_SP_MSLVIRQENABLE_EVENT1_CLRMSK 0xFFFFFFF7U
1556#define ROGUE_CR_META_SP_MSLVIRQENABLE_EVENT1_EN 0x00000008U
1557#define ROGUE_CR_META_SP_MSLVIRQENABLE_EVENT0_SHIFT 2U
1558#define ROGUE_CR_META_SP_MSLVIRQENABLE_EVENT0_CLRMSK 0xFFFFFFFBU
1559#define ROGUE_CR_META_SP_MSLVIRQENABLE_EVENT0_EN 0x00000004U
1560
1561/* Register ROGUE_CR_META_SP_MSLVIRQLEVEL */
1562#define ROGUE_CR_META_SP_MSLVIRQLEVEL 0x0AD8U
1563#define ROGUE_CR_META_SP_MSLVIRQLEVEL_MASKFULL 0x0000000000000001ULL
1564#define ROGUE_CR_META_SP_MSLVIRQLEVEL_MODE_SHIFT 0U
1565#define ROGUE_CR_META_SP_MSLVIRQLEVEL_MODE_CLRMSK 0xFFFFFFFEU
1566#define ROGUE_CR_META_SP_MSLVIRQLEVEL_MODE_EN 0x00000001U
1567
1568/* Register ROGUE_CR_MTS_SCHEDULE */
1569#define ROGUE_CR_MTS_SCHEDULE 0x0B00U
1570#define ROGUE_CR_MTS_SCHEDULE_MASKFULL 0x00000000000001FFULL
1571#define ROGUE_CR_MTS_SCHEDULE_HOST_SHIFT 8U
1572#define ROGUE_CR_MTS_SCHEDULE_HOST_CLRMSK 0xFFFFFEFFU
1573#define ROGUE_CR_MTS_SCHEDULE_HOST_BG_TIMER 0x00000000U
1574#define ROGUE_CR_MTS_SCHEDULE_HOST_HOST 0x00000100U
1575#define ROGUE_CR_MTS_SCHEDULE_PRIORITY_SHIFT 6U
1576#define ROGUE_CR_MTS_SCHEDULE_PRIORITY_CLRMSK 0xFFFFFF3FU
1577#define ROGUE_CR_MTS_SCHEDULE_PRIORITY_PRT0 0x00000000U
1578#define ROGUE_CR_MTS_SCHEDULE_PRIORITY_PRT1 0x00000040U
1579#define ROGUE_CR_MTS_SCHEDULE_PRIORITY_PRT2 0x00000080U
1580#define ROGUE_CR_MTS_SCHEDULE_PRIORITY_PRT3 0x000000C0U
1581#define ROGUE_CR_MTS_SCHEDULE_CONTEXT_SHIFT 5U
1582#define ROGUE_CR_MTS_SCHEDULE_CONTEXT_CLRMSK 0xFFFFFFDFU
1583#define ROGUE_CR_MTS_SCHEDULE_CONTEXT_BGCTX 0x00000000U
1584#define ROGUE_CR_MTS_SCHEDULE_CONTEXT_INTCTX 0x00000020U
1585#define ROGUE_CR_MTS_SCHEDULE_TASK_SHIFT 4U
1586#define ROGUE_CR_MTS_SCHEDULE_TASK_CLRMSK 0xFFFFFFEFU
1587#define ROGUE_CR_MTS_SCHEDULE_TASK_NON_COUNTED 0x00000000U
1588#define ROGUE_CR_MTS_SCHEDULE_TASK_COUNTED 0x00000010U
1589#define ROGUE_CR_MTS_SCHEDULE_DM_SHIFT 0U
1590#define ROGUE_CR_MTS_SCHEDULE_DM_CLRMSK 0xFFFFFFF0U
1591#define ROGUE_CR_MTS_SCHEDULE_DM_DM0 0x00000000U
1592#define ROGUE_CR_MTS_SCHEDULE_DM_DM1 0x00000001U
1593#define ROGUE_CR_MTS_SCHEDULE_DM_DM2 0x00000002U
1594#define ROGUE_CR_MTS_SCHEDULE_DM_DM3 0x00000003U
1595#define ROGUE_CR_MTS_SCHEDULE_DM_DM4 0x00000004U
1596#define ROGUE_CR_MTS_SCHEDULE_DM_DM5 0x00000005U
1597#define ROGUE_CR_MTS_SCHEDULE_DM_DM6 0x00000006U
1598#define ROGUE_CR_MTS_SCHEDULE_DM_DM7 0x00000007U
1599#define ROGUE_CR_MTS_SCHEDULE_DM_DM_ALL 0x0000000FU
1600
1601/* Register ROGUE_CR_MTS_SCHEDULE1 */
1602#define ROGUE_CR_MTS_SCHEDULE1 0x10B00U
1603#define ROGUE_CR_MTS_SCHEDULE1_MASKFULL 0x00000000000001FFULL
1604#define ROGUE_CR_MTS_SCHEDULE1_HOST_SHIFT 8U
1605#define ROGUE_CR_MTS_SCHEDULE1_HOST_CLRMSK 0xFFFFFEFFU
1606#define ROGUE_CR_MTS_SCHEDULE1_HOST_BG_TIMER 0x00000000U
1607#define ROGUE_CR_MTS_SCHEDULE1_HOST_HOST 0x00000100U
1608#define ROGUE_CR_MTS_SCHEDULE1_PRIORITY_SHIFT 6U
1609#define ROGUE_CR_MTS_SCHEDULE1_PRIORITY_CLRMSK 0xFFFFFF3FU
1610#define ROGUE_CR_MTS_SCHEDULE1_PRIORITY_PRT0 0x00000000U
1611#define ROGUE_CR_MTS_SCHEDULE1_PRIORITY_PRT1 0x00000040U
1612#define ROGUE_CR_MTS_SCHEDULE1_PRIORITY_PRT2 0x00000080U
1613#define ROGUE_CR_MTS_SCHEDULE1_PRIORITY_PRT3 0x000000C0U
1614#define ROGUE_CR_MTS_SCHEDULE1_CONTEXT_SHIFT 5U
1615#define ROGUE_CR_MTS_SCHEDULE1_CONTEXT_CLRMSK 0xFFFFFFDFU
1616#define ROGUE_CR_MTS_SCHEDULE1_CONTEXT_BGCTX 0x00000000U
1617#define ROGUE_CR_MTS_SCHEDULE1_CONTEXT_INTCTX 0x00000020U
1618#define ROGUE_CR_MTS_SCHEDULE1_TASK_SHIFT 4U
1619#define ROGUE_CR_MTS_SCHEDULE1_TASK_CLRMSK 0xFFFFFFEFU
1620#define ROGUE_CR_MTS_SCHEDULE1_TASK_NON_COUNTED 0x00000000U
1621#define ROGUE_CR_MTS_SCHEDULE1_TASK_COUNTED 0x00000010U
1622#define ROGUE_CR_MTS_SCHEDULE1_DM_SHIFT 0U
1623#define ROGUE_CR_MTS_SCHEDULE1_DM_CLRMSK 0xFFFFFFF0U
1624#define ROGUE_CR_MTS_SCHEDULE1_DM_DM0 0x00000000U
1625#define ROGUE_CR_MTS_SCHEDULE1_DM_DM1 0x00000001U
1626#define ROGUE_CR_MTS_SCHEDULE1_DM_DM2 0x00000002U
1627#define ROGUE_CR_MTS_SCHEDULE1_DM_DM3 0x00000003U
1628#define ROGUE_CR_MTS_SCHEDULE1_DM_DM4 0x00000004U
1629#define ROGUE_CR_MTS_SCHEDULE1_DM_DM5 0x00000005U
1630#define ROGUE_CR_MTS_SCHEDULE1_DM_DM6 0x00000006U
1631#define ROGUE_CR_MTS_SCHEDULE1_DM_DM7 0x00000007U
1632#define ROGUE_CR_MTS_SCHEDULE1_DM_DM_ALL 0x0000000FU
1633
1634/* Register ROGUE_CR_MTS_SCHEDULE2 */
1635#define ROGUE_CR_MTS_SCHEDULE2 0x20B00U
1636#define ROGUE_CR_MTS_SCHEDULE2_MASKFULL 0x00000000000001FFULL
1637#define ROGUE_CR_MTS_SCHEDULE2_HOST_SHIFT 8U
1638#define ROGUE_CR_MTS_SCHEDULE2_HOST_CLRMSK 0xFFFFFEFFU
1639#define ROGUE_CR_MTS_SCHEDULE2_HOST_BG_TIMER 0x00000000U
1640#define ROGUE_CR_MTS_SCHEDULE2_HOST_HOST 0x00000100U
1641#define ROGUE_CR_MTS_SCHEDULE2_PRIORITY_SHIFT 6U
1642#define ROGUE_CR_MTS_SCHEDULE2_PRIORITY_CLRMSK 0xFFFFFF3FU
1643#define ROGUE_CR_MTS_SCHEDULE2_PRIORITY_PRT0 0x00000000U
1644#define ROGUE_CR_MTS_SCHEDULE2_PRIORITY_PRT1 0x00000040U
1645#define ROGUE_CR_MTS_SCHEDULE2_PRIORITY_PRT2 0x00000080U
1646#define ROGUE_CR_MTS_SCHEDULE2_PRIORITY_PRT3 0x000000C0U
1647#define ROGUE_CR_MTS_SCHEDULE2_CONTEXT_SHIFT 5U
1648#define ROGUE_CR_MTS_SCHEDULE2_CONTEXT_CLRMSK 0xFFFFFFDFU
1649#define ROGUE_CR_MTS_SCHEDULE2_CONTEXT_BGCTX 0x00000000U
1650#define ROGUE_CR_MTS_SCHEDULE2_CONTEXT_INTCTX 0x00000020U
1651#define ROGUE_CR_MTS_SCHEDULE2_TASK_SHIFT 4U
1652#define ROGUE_CR_MTS_SCHEDULE2_TASK_CLRMSK 0xFFFFFFEFU
1653#define ROGUE_CR_MTS_SCHEDULE2_TASK_NON_COUNTED 0x00000000U
1654#define ROGUE_CR_MTS_SCHEDULE2_TASK_COUNTED 0x00000010U
1655#define ROGUE_CR_MTS_SCHEDULE2_DM_SHIFT 0U
1656#define ROGUE_CR_MTS_SCHEDULE2_DM_CLRMSK 0xFFFFFFF0U
1657#define ROGUE_CR_MTS_SCHEDULE2_DM_DM0 0x00000000U
1658#define ROGUE_CR_MTS_SCHEDULE2_DM_DM1 0x00000001U
1659#define ROGUE_CR_MTS_SCHEDULE2_DM_DM2 0x00000002U
1660#define ROGUE_CR_MTS_SCHEDULE2_DM_DM3 0x00000003U
1661#define ROGUE_CR_MTS_SCHEDULE2_DM_DM4 0x00000004U
1662#define ROGUE_CR_MTS_SCHEDULE2_DM_DM5 0x00000005U
1663#define ROGUE_CR_MTS_SCHEDULE2_DM_DM6 0x00000006U
1664#define ROGUE_CR_MTS_SCHEDULE2_DM_DM7 0x00000007U
1665#define ROGUE_CR_MTS_SCHEDULE2_DM_DM_ALL 0x0000000FU
1666
1667/* Register ROGUE_CR_MTS_SCHEDULE3 */
1668#define ROGUE_CR_MTS_SCHEDULE3 0x30B00U
1669#define ROGUE_CR_MTS_SCHEDULE3_MASKFULL 0x00000000000001FFULL
1670#define ROGUE_CR_MTS_SCHEDULE3_HOST_SHIFT 8U
1671#define ROGUE_CR_MTS_SCHEDULE3_HOST_CLRMSK 0xFFFFFEFFU
1672#define ROGUE_CR_MTS_SCHEDULE3_HOST_BG_TIMER 0x00000000U
1673#define ROGUE_CR_MTS_SCHEDULE3_HOST_HOST 0x00000100U
1674#define ROGUE_CR_MTS_SCHEDULE3_PRIORITY_SHIFT 6U
1675#define ROGUE_CR_MTS_SCHEDULE3_PRIORITY_CLRMSK 0xFFFFFF3FU
1676#define ROGUE_CR_MTS_SCHEDULE3_PRIORITY_PRT0 0x00000000U
1677#define ROGUE_CR_MTS_SCHEDULE3_PRIORITY_PRT1 0x00000040U
1678#define ROGUE_CR_MTS_SCHEDULE3_PRIORITY_PRT2 0x00000080U
1679#define ROGUE_CR_MTS_SCHEDULE3_PRIORITY_PRT3 0x000000C0U
1680#define ROGUE_CR_MTS_SCHEDULE3_CONTEXT_SHIFT 5U
1681#define ROGUE_CR_MTS_SCHEDULE3_CONTEXT_CLRMSK 0xFFFFFFDFU
1682#define ROGUE_CR_MTS_SCHEDULE3_CONTEXT_BGCTX 0x00000000U
1683#define ROGUE_CR_MTS_SCHEDULE3_CONTEXT_INTCTX 0x00000020U
1684#define ROGUE_CR_MTS_SCHEDULE3_TASK_SHIFT 4U
1685#define ROGUE_CR_MTS_SCHEDULE3_TASK_CLRMSK 0xFFFFFFEFU
1686#define ROGUE_CR_MTS_SCHEDULE3_TASK_NON_COUNTED 0x00000000U
1687#define ROGUE_CR_MTS_SCHEDULE3_TASK_COUNTED 0x00000010U
1688#define ROGUE_CR_MTS_SCHEDULE3_DM_SHIFT 0U
1689#define ROGUE_CR_MTS_SCHEDULE3_DM_CLRMSK 0xFFFFFFF0U
1690#define ROGUE_CR_MTS_SCHEDULE3_DM_DM0 0x00000000U
1691#define ROGUE_CR_MTS_SCHEDULE3_DM_DM1 0x00000001U
1692#define ROGUE_CR_MTS_SCHEDULE3_DM_DM2 0x00000002U
1693#define ROGUE_CR_MTS_SCHEDULE3_DM_DM3 0x00000003U
1694#define ROGUE_CR_MTS_SCHEDULE3_DM_DM4 0x00000004U
1695#define ROGUE_CR_MTS_SCHEDULE3_DM_DM5 0x00000005U
1696#define ROGUE_CR_MTS_SCHEDULE3_DM_DM6 0x00000006U
1697#define ROGUE_CR_MTS_SCHEDULE3_DM_DM7 0x00000007U
1698#define ROGUE_CR_MTS_SCHEDULE3_DM_DM_ALL 0x0000000FU
1699
1700/* Register ROGUE_CR_MTS_SCHEDULE4 */
1701#define ROGUE_CR_MTS_SCHEDULE4 0x40B00U
1702#define ROGUE_CR_MTS_SCHEDULE4_MASKFULL 0x00000000000001FFULL
1703#define ROGUE_CR_MTS_SCHEDULE4_HOST_SHIFT 8U
1704#define ROGUE_CR_MTS_SCHEDULE4_HOST_CLRMSK 0xFFFFFEFFU
1705#define ROGUE_CR_MTS_SCHEDULE4_HOST_BG_TIMER 0x00000000U
1706#define ROGUE_CR_MTS_SCHEDULE4_HOST_HOST 0x00000100U
1707#define ROGUE_CR_MTS_SCHEDULE4_PRIORITY_SHIFT 6U
1708#define ROGUE_CR_MTS_SCHEDULE4_PRIORITY_CLRMSK 0xFFFFFF3FU
1709#define ROGUE_CR_MTS_SCHEDULE4_PRIORITY_PRT0 0x00000000U
1710#define ROGUE_CR_MTS_SCHEDULE4_PRIORITY_PRT1 0x00000040U
1711#define ROGUE_CR_MTS_SCHEDULE4_PRIORITY_PRT2 0x00000080U
1712#define ROGUE_CR_MTS_SCHEDULE4_PRIORITY_PRT3 0x000000C0U
1713#define ROGUE_CR_MTS_SCHEDULE4_CONTEXT_SHIFT 5U
1714#define ROGUE_CR_MTS_SCHEDULE4_CONTEXT_CLRMSK 0xFFFFFFDFU
1715#define ROGUE_CR_MTS_SCHEDULE4_CONTEXT_BGCTX 0x00000000U
1716#define ROGUE_CR_MTS_SCHEDULE4_CONTEXT_INTCTX 0x00000020U
1717#define ROGUE_CR_MTS_SCHEDULE4_TASK_SHIFT 4U
1718#define ROGUE_CR_MTS_SCHEDULE4_TASK_CLRMSK 0xFFFFFFEFU
1719#define ROGUE_CR_MTS_SCHEDULE4_TASK_NON_COUNTED 0x00000000U
1720#define ROGUE_CR_MTS_SCHEDULE4_TASK_COUNTED 0x00000010U
1721#define ROGUE_CR_MTS_SCHEDULE4_DM_SHIFT 0U
1722#define ROGUE_CR_MTS_SCHEDULE4_DM_CLRMSK 0xFFFFFFF0U
1723#define ROGUE_CR_MTS_SCHEDULE4_DM_DM0 0x00000000U
1724#define ROGUE_CR_MTS_SCHEDULE4_DM_DM1 0x00000001U
1725#define ROGUE_CR_MTS_SCHEDULE4_DM_DM2 0x00000002U
1726#define ROGUE_CR_MTS_SCHEDULE4_DM_DM3 0x00000003U
1727#define ROGUE_CR_MTS_SCHEDULE4_DM_DM4 0x00000004U
1728#define ROGUE_CR_MTS_SCHEDULE4_DM_DM5 0x00000005U
1729#define ROGUE_CR_MTS_SCHEDULE4_DM_DM6 0x00000006U
1730#define ROGUE_CR_MTS_SCHEDULE4_DM_DM7 0x00000007U
1731#define ROGUE_CR_MTS_SCHEDULE4_DM_DM_ALL 0x0000000FU
1732
1733/* Register ROGUE_CR_MTS_SCHEDULE5 */
1734#define ROGUE_CR_MTS_SCHEDULE5 0x50B00U
1735#define ROGUE_CR_MTS_SCHEDULE5_MASKFULL 0x00000000000001FFULL
1736#define ROGUE_CR_MTS_SCHEDULE5_HOST_SHIFT 8U
1737#define ROGUE_CR_MTS_SCHEDULE5_HOST_CLRMSK 0xFFFFFEFFU
1738#define ROGUE_CR_MTS_SCHEDULE5_HOST_BG_TIMER 0x00000000U
1739#define ROGUE_CR_MTS_SCHEDULE5_HOST_HOST 0x00000100U
1740#define ROGUE_CR_MTS_SCHEDULE5_PRIORITY_SHIFT 6U
1741#define ROGUE_CR_MTS_SCHEDULE5_PRIORITY_CLRMSK 0xFFFFFF3FU
1742#define ROGUE_CR_MTS_SCHEDULE5_PRIORITY_PRT0 0x00000000U
1743#define ROGUE_CR_MTS_SCHEDULE5_PRIORITY_PRT1 0x00000040U
1744#define ROGUE_CR_MTS_SCHEDULE5_PRIORITY_PRT2 0x00000080U
1745#define ROGUE_CR_MTS_SCHEDULE5_PRIORITY_PRT3 0x000000C0U
1746#define ROGUE_CR_MTS_SCHEDULE5_CONTEXT_SHIFT 5U
1747#define ROGUE_CR_MTS_SCHEDULE5_CONTEXT_CLRMSK 0xFFFFFFDFU
1748#define ROGUE_CR_MTS_SCHEDULE5_CONTEXT_BGCTX 0x00000000U
1749#define ROGUE_CR_MTS_SCHEDULE5_CONTEXT_INTCTX 0x00000020U
1750#define ROGUE_CR_MTS_SCHEDULE5_TASK_SHIFT 4U
1751#define ROGUE_CR_MTS_SCHEDULE5_TASK_CLRMSK 0xFFFFFFEFU
1752#define ROGUE_CR_MTS_SCHEDULE5_TASK_NON_COUNTED 0x00000000U
1753#define ROGUE_CR_MTS_SCHEDULE5_TASK_COUNTED 0x00000010U
1754#define ROGUE_CR_MTS_SCHEDULE5_DM_SHIFT 0U
1755#define ROGUE_CR_MTS_SCHEDULE5_DM_CLRMSK 0xFFFFFFF0U
1756#define ROGUE_CR_MTS_SCHEDULE5_DM_DM0 0x00000000U
1757#define ROGUE_CR_MTS_SCHEDULE5_DM_DM1 0x00000001U
1758#define ROGUE_CR_MTS_SCHEDULE5_DM_DM2 0x00000002U
1759#define ROGUE_CR_MTS_SCHEDULE5_DM_DM3 0x00000003U
1760#define ROGUE_CR_MTS_SCHEDULE5_DM_DM4 0x00000004U
1761#define ROGUE_CR_MTS_SCHEDULE5_DM_DM5 0x00000005U
1762#define ROGUE_CR_MTS_SCHEDULE5_DM_DM6 0x00000006U
1763#define ROGUE_CR_MTS_SCHEDULE5_DM_DM7 0x00000007U
1764#define ROGUE_CR_MTS_SCHEDULE5_DM_DM_ALL 0x0000000FU
1765
1766/* Register ROGUE_CR_MTS_SCHEDULE6 */
1767#define ROGUE_CR_MTS_SCHEDULE6 0x60B00U
1768#define ROGUE_CR_MTS_SCHEDULE6_MASKFULL 0x00000000000001FFULL
1769#define ROGUE_CR_MTS_SCHEDULE6_HOST_SHIFT 8U
1770#define ROGUE_CR_MTS_SCHEDULE6_HOST_CLRMSK 0xFFFFFEFFU
1771#define ROGUE_CR_MTS_SCHEDULE6_HOST_BG_TIMER 0x00000000U
1772#define ROGUE_CR_MTS_SCHEDULE6_HOST_HOST 0x00000100U
1773#define ROGUE_CR_MTS_SCHEDULE6_PRIORITY_SHIFT 6U
1774#define ROGUE_CR_MTS_SCHEDULE6_PRIORITY_CLRMSK 0xFFFFFF3FU
1775#define ROGUE_CR_MTS_SCHEDULE6_PRIORITY_PRT0 0x00000000U
1776#define ROGUE_CR_MTS_SCHEDULE6_PRIORITY_PRT1 0x00000040U
1777#define ROGUE_CR_MTS_SCHEDULE6_PRIORITY_PRT2 0x00000080U
1778#define ROGUE_CR_MTS_SCHEDULE6_PRIORITY_PRT3 0x000000C0U
1779#define ROGUE_CR_MTS_SCHEDULE6_CONTEXT_SHIFT 5U
1780#define ROGUE_CR_MTS_SCHEDULE6_CONTEXT_CLRMSK 0xFFFFFFDFU
1781#define ROGUE_CR_MTS_SCHEDULE6_CONTEXT_BGCTX 0x00000000U
1782#define ROGUE_CR_MTS_SCHEDULE6_CONTEXT_INTCTX 0x00000020U
1783#define ROGUE_CR_MTS_SCHEDULE6_TASK_SHIFT 4U
1784#define ROGUE_CR_MTS_SCHEDULE6_TASK_CLRMSK 0xFFFFFFEFU
1785#define ROGUE_CR_MTS_SCHEDULE6_TASK_NON_COUNTED 0x00000000U
1786#define ROGUE_CR_MTS_SCHEDULE6_TASK_COUNTED 0x00000010U
1787#define ROGUE_CR_MTS_SCHEDULE6_DM_SHIFT 0U
1788#define ROGUE_CR_MTS_SCHEDULE6_DM_CLRMSK 0xFFFFFFF0U
1789#define ROGUE_CR_MTS_SCHEDULE6_DM_DM0 0x00000000U
1790#define ROGUE_CR_MTS_SCHEDULE6_DM_DM1 0x00000001U
1791#define ROGUE_CR_MTS_SCHEDULE6_DM_DM2 0x00000002U
1792#define ROGUE_CR_MTS_SCHEDULE6_DM_DM3 0x00000003U
1793#define ROGUE_CR_MTS_SCHEDULE6_DM_DM4 0x00000004U
1794#define ROGUE_CR_MTS_SCHEDULE6_DM_DM5 0x00000005U
1795#define ROGUE_CR_MTS_SCHEDULE6_DM_DM6 0x00000006U
1796#define ROGUE_CR_MTS_SCHEDULE6_DM_DM7 0x00000007U
1797#define ROGUE_CR_MTS_SCHEDULE6_DM_DM_ALL 0x0000000FU
1798
1799/* Register ROGUE_CR_MTS_SCHEDULE7 */
1800#define ROGUE_CR_MTS_SCHEDULE7 0x70B00U
1801#define ROGUE_CR_MTS_SCHEDULE7_MASKFULL 0x00000000000001FFULL
1802#define ROGUE_CR_MTS_SCHEDULE7_HOST_SHIFT 8U
1803#define ROGUE_CR_MTS_SCHEDULE7_HOST_CLRMSK 0xFFFFFEFFU
1804#define ROGUE_CR_MTS_SCHEDULE7_HOST_BG_TIMER 0x00000000U
1805#define ROGUE_CR_MTS_SCHEDULE7_HOST_HOST 0x00000100U
1806#define ROGUE_CR_MTS_SCHEDULE7_PRIORITY_SHIFT 6U
1807#define ROGUE_CR_MTS_SCHEDULE7_PRIORITY_CLRMSK 0xFFFFFF3FU
1808#define ROGUE_CR_MTS_SCHEDULE7_PRIORITY_PRT0 0x00000000U
1809#define ROGUE_CR_MTS_SCHEDULE7_PRIORITY_PRT1 0x00000040U
1810#define ROGUE_CR_MTS_SCHEDULE7_PRIORITY_PRT2 0x00000080U
1811#define ROGUE_CR_MTS_SCHEDULE7_PRIORITY_PRT3 0x000000C0U
1812#define ROGUE_CR_MTS_SCHEDULE7_CONTEXT_SHIFT 5U
1813#define ROGUE_CR_MTS_SCHEDULE7_CONTEXT_CLRMSK 0xFFFFFFDFU
1814#define ROGUE_CR_MTS_SCHEDULE7_CONTEXT_BGCTX 0x00000000U
1815#define ROGUE_CR_MTS_SCHEDULE7_CONTEXT_INTCTX 0x00000020U
1816#define ROGUE_CR_MTS_SCHEDULE7_TASK_SHIFT 4U
1817#define ROGUE_CR_MTS_SCHEDULE7_TASK_CLRMSK 0xFFFFFFEFU
1818#define ROGUE_CR_MTS_SCHEDULE7_TASK_NON_COUNTED 0x00000000U
1819#define ROGUE_CR_MTS_SCHEDULE7_TASK_COUNTED 0x00000010U
1820#define ROGUE_CR_MTS_SCHEDULE7_DM_SHIFT 0U
1821#define ROGUE_CR_MTS_SCHEDULE7_DM_CLRMSK 0xFFFFFFF0U
1822#define ROGUE_CR_MTS_SCHEDULE7_DM_DM0 0x00000000U
1823#define ROGUE_CR_MTS_SCHEDULE7_DM_DM1 0x00000001U
1824#define ROGUE_CR_MTS_SCHEDULE7_DM_DM2 0x00000002U
1825#define ROGUE_CR_MTS_SCHEDULE7_DM_DM3 0x00000003U
1826#define ROGUE_CR_MTS_SCHEDULE7_DM_DM4 0x00000004U
1827#define ROGUE_CR_MTS_SCHEDULE7_DM_DM5 0x00000005U
1828#define ROGUE_CR_MTS_SCHEDULE7_DM_DM6 0x00000006U
1829#define ROGUE_CR_MTS_SCHEDULE7_DM_DM7 0x00000007U
1830#define ROGUE_CR_MTS_SCHEDULE7_DM_DM_ALL 0x0000000FU
1831
1832/* Register ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC */
1833#define ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC 0x0B30U
1834#define ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC_MASKFULL 0x000000000000FFFFULL
1835#define ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC_DM_ASSOC_SHIFT 0U
1836#define ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC_DM_ASSOC_CLRMSK 0xFFFF0000U
1837
1838/* Register ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC */
1839#define ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC 0x0B38U
1840#define ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC_MASKFULL 0x000000000000FFFFULL
1841#define ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC_DM_ASSOC_SHIFT 0U
1842#define ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC_DM_ASSOC_CLRMSK 0xFFFF0000U
1843
1844/* Register ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC */
1845#define ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC 0x0B40U
1846#define ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC_MASKFULL 0x000000000000FFFFULL
1847#define ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC_DM_ASSOC_SHIFT 0U
1848#define ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC_DM_ASSOC_CLRMSK 0xFFFF0000U
1849
1850/* Register ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC */
1851#define ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC 0x0B48U
1852#define ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC_MASKFULL 0x000000000000FFFFULL
1853#define ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC_DM_ASSOC_SHIFT 0U
1854#define ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC_DM_ASSOC_CLRMSK 0xFFFF0000U
1855
1856/* Register ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG */
1857#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG 0x0B50U
1858#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG__S7_TOP__MASKFULL 0x000FF0FFFFFFF701ULL
1859#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_MASKFULL 0x0000FFFFFFFFF001ULL
1860#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_PC_BASE_SHIFT 44U
1861#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_PC_BASE_CLRMSK 0xFFFF0FFFFFFFFFFFULL
1862#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG__S7_TOP__FENCE_PC_BASE_SHIFT 44U
1863#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG__S7_TOP__FENCE_PC_BASE_CLRMSK 0xFFF00FFFFFFFFFFFULL
1864#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_DM_SHIFT 40U
1865#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_DM_CLRMSK 0xFFFFF0FFFFFFFFFFULL
1866#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_ADDR_SHIFT 12U
1867#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_ADDR_CLRMSK 0xFFFFFF0000000FFFULL
1868#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_PERSISTENCE_SHIFT 9U
1869#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_PERSISTENCE_CLRMSK 0xFFFFFFFFFFFFF9FFULL
1870#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_SLC_COHERENT_SHIFT 8U
1871#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_SLC_COHERENT_CLRMSK 0xFFFFFFFFFFFFFEFFULL
1872#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_SLC_COHERENT_EN 0x0000000000000100ULL
1873#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_IDLE_CTRL_SHIFT 0U
1874#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_IDLE_CTRL_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1875#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_IDLE_CTRL_META 0x0000000000000000ULL
1876#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_IDLE_CTRL_MTS 0x0000000000000001ULL
1877
1878/* Register ROGUE_CR_MTS_DM0_INTERRUPT_ENABLE */
1879#define ROGUE_CR_MTS_DM0_INTERRUPT_ENABLE 0x0B58U
1880#define ROGUE_CR_MTS_DM0_INTERRUPT_ENABLE_MASKFULL 0x00000000FFFFFFFFULL
1881#define ROGUE_CR_MTS_DM0_INTERRUPT_ENABLE_INT_ENABLE_SHIFT 0U
1882#define ROGUE_CR_MTS_DM0_INTERRUPT_ENABLE_INT_ENABLE_CLRMSK 0x00000000U
1883
1884/* Register ROGUE_CR_MTS_DM1_INTERRUPT_ENABLE */
1885#define ROGUE_CR_MTS_DM1_INTERRUPT_ENABLE 0x0B60U
1886#define ROGUE_CR_MTS_DM1_INTERRUPT_ENABLE_MASKFULL 0x00000000FFFFFFFFULL
1887#define ROGUE_CR_MTS_DM1_INTERRUPT_ENABLE_INT_ENABLE_SHIFT 0U
1888#define ROGUE_CR_MTS_DM1_INTERRUPT_ENABLE_INT_ENABLE_CLRMSK 0x00000000U
1889
1890/* Register ROGUE_CR_MTS_DM2_INTERRUPT_ENABLE */
1891#define ROGUE_CR_MTS_DM2_INTERRUPT_ENABLE 0x0B68U
1892#define ROGUE_CR_MTS_DM2_INTERRUPT_ENABLE_MASKFULL 0x00000000FFFFFFFFULL
1893#define ROGUE_CR_MTS_DM2_INTERRUPT_ENABLE_INT_ENABLE_SHIFT 0U
1894#define ROGUE_CR_MTS_DM2_INTERRUPT_ENABLE_INT_ENABLE_CLRMSK 0x00000000U
1895
1896/* Register ROGUE_CR_MTS_DM3_INTERRUPT_ENABLE */
1897#define ROGUE_CR_MTS_DM3_INTERRUPT_ENABLE 0x0B70U
1898#define ROGUE_CR_MTS_DM3_INTERRUPT_ENABLE_MASKFULL 0x00000000FFFFFFFFULL
1899#define ROGUE_CR_MTS_DM3_INTERRUPT_ENABLE_INT_ENABLE_SHIFT 0U
1900#define ROGUE_CR_MTS_DM3_INTERRUPT_ENABLE_INT_ENABLE_CLRMSK 0x00000000U
1901
1902/* Register ROGUE_CR_MTS_DM4_INTERRUPT_ENABLE */
1903#define ROGUE_CR_MTS_DM4_INTERRUPT_ENABLE 0x0B78U
1904#define ROGUE_CR_MTS_DM4_INTERRUPT_ENABLE_MASKFULL 0x00000000FFFFFFFFULL
1905#define ROGUE_CR_MTS_DM4_INTERRUPT_ENABLE_INT_ENABLE_SHIFT 0U
1906#define ROGUE_CR_MTS_DM4_INTERRUPT_ENABLE_INT_ENABLE_CLRMSK 0x00000000U
1907
1908/* Register ROGUE_CR_MTS_DM5_INTERRUPT_ENABLE */
1909#define ROGUE_CR_MTS_DM5_INTERRUPT_ENABLE 0x0B80U
1910#define ROGUE_CR_MTS_DM5_INTERRUPT_ENABLE_MASKFULL 0x00000000FFFFFFFFULL
1911#define ROGUE_CR_MTS_DM5_INTERRUPT_ENABLE_INT_ENABLE_SHIFT 0U
1912#define ROGUE_CR_MTS_DM5_INTERRUPT_ENABLE_INT_ENABLE_CLRMSK 0x00000000U
1913
1914/* Register ROGUE_CR_MTS_INTCTX */
1915#define ROGUE_CR_MTS_INTCTX 0x0B98U
1916#define ROGUE_CR_MTS_INTCTX_MASKFULL 0x000000003FFFFFFFULL
1917#define ROGUE_CR_MTS_INTCTX_DM_HOST_SCHEDULE_SHIFT 22U
1918#define ROGUE_CR_MTS_INTCTX_DM_HOST_SCHEDULE_CLRMSK 0xC03FFFFFU
1919#define ROGUE_CR_MTS_INTCTX_DM_PTR_SHIFT 18U
1920#define ROGUE_CR_MTS_INTCTX_DM_PTR_CLRMSK 0xFFC3FFFFU
1921#define ROGUE_CR_MTS_INTCTX_THREAD_ACTIVE_SHIFT 16U
1922#define ROGUE_CR_MTS_INTCTX_THREAD_ACTIVE_CLRMSK 0xFFFCFFFFU
1923#define ROGUE_CR_MTS_INTCTX_DM_TIMER_SCHEDULE_SHIFT 8U
1924#define ROGUE_CR_MTS_INTCTX_DM_TIMER_SCHEDULE_CLRMSK 0xFFFF00FFU
1925#define ROGUE_CR_MTS_INTCTX_DM_INTERRUPT_SCHEDULE_SHIFT 0U
1926#define ROGUE_CR_MTS_INTCTX_DM_INTERRUPT_SCHEDULE_CLRMSK 0xFFFFFF00U
1927
1928/* Register ROGUE_CR_MTS_BGCTX */
1929#define ROGUE_CR_MTS_BGCTX 0x0BA0U
1930#define ROGUE_CR_MTS_BGCTX_MASKFULL 0x0000000000003FFFULL
1931#define ROGUE_CR_MTS_BGCTX_DM_PTR_SHIFT 10U
1932#define ROGUE_CR_MTS_BGCTX_DM_PTR_CLRMSK 0xFFFFC3FFU
1933#define ROGUE_CR_MTS_BGCTX_THREAD_ACTIVE_SHIFT 8U
1934#define ROGUE_CR_MTS_BGCTX_THREAD_ACTIVE_CLRMSK 0xFFFFFCFFU
1935#define ROGUE_CR_MTS_BGCTX_DM_NONCOUNTED_SCHEDULE_SHIFT 0U
1936#define ROGUE_CR_MTS_BGCTX_DM_NONCOUNTED_SCHEDULE_CLRMSK 0xFFFFFF00U
1937
1938/* Register ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE */
1939#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE 0x0BA8U
1940#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_MASKFULL 0xFFFFFFFFFFFFFFFFULL
1941#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM7_SHIFT 56U
1942#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM7_CLRMSK 0x00FFFFFFFFFFFFFFULL
1943#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM6_SHIFT 48U
1944#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM6_CLRMSK 0xFF00FFFFFFFFFFFFULL
1945#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM5_SHIFT 40U
1946#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM5_CLRMSK 0xFFFF00FFFFFFFFFFULL
1947#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM4_SHIFT 32U
1948#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM4_CLRMSK 0xFFFFFF00FFFFFFFFULL
1949#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM3_SHIFT 24U
1950#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM3_CLRMSK 0xFFFFFFFF00FFFFFFULL
1951#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM2_SHIFT 16U
1952#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM2_CLRMSK 0xFFFFFFFFFF00FFFFULL
1953#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM1_SHIFT 8U
1954#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM1_CLRMSK 0xFFFFFFFFFFFF00FFULL
1955#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM0_SHIFT 0U
1956#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM0_CLRMSK 0xFFFFFFFFFFFFFF00ULL
1957
1958/* Register ROGUE_CR_MTS_GPU_INT_STATUS */
1959#define ROGUE_CR_MTS_GPU_INT_STATUS 0x0BB0U
1960#define ROGUE_CR_MTS_GPU_INT_STATUS_MASKFULL 0x00000000FFFFFFFFULL
1961#define ROGUE_CR_MTS_GPU_INT_STATUS_STATUS_SHIFT 0U
1962#define ROGUE_CR_MTS_GPU_INT_STATUS_STATUS_CLRMSK 0x00000000U
1963
1964/* Register ROGUE_CR_MTS_SCHEDULE_ENABLE */
1965#define ROGUE_CR_MTS_SCHEDULE_ENABLE 0x0BC8U
1966#define ROGUE_CR_MTS_SCHEDULE_ENABLE_MASKFULL 0x00000000000000FFULL
1967#define ROGUE_CR_MTS_SCHEDULE_ENABLE_MASK_SHIFT 0U
1968#define ROGUE_CR_MTS_SCHEDULE_ENABLE_MASK_CLRMSK 0xFFFFFF00U
1969
1970/* Register ROGUE_CR_IRQ_OS0_EVENT_STATUS */
1971#define ROGUE_CR_IRQ_OS0_EVENT_STATUS 0x0BD8U
1972#define ROGUE_CR_IRQ_OS0_EVENT_STATUS_MASKFULL 0x0000000000000001ULL
1973#define ROGUE_CR_IRQ_OS0_EVENT_STATUS_SOURCE_SHIFT 0U
1974#define ROGUE_CR_IRQ_OS0_EVENT_STATUS_SOURCE_CLRMSK 0xFFFFFFFEU
1975#define ROGUE_CR_IRQ_OS0_EVENT_STATUS_SOURCE_EN 0x00000001U
1976
1977/* Register ROGUE_CR_IRQ_OS0_EVENT_CLEAR */
1978#define ROGUE_CR_IRQ_OS0_EVENT_CLEAR 0x0BE8U
1979#define ROGUE_CR_IRQ_OS0_EVENT_CLEAR_MASKFULL 0x0000000000000001ULL
1980#define ROGUE_CR_IRQ_OS0_EVENT_CLEAR_SOURCE_SHIFT 0U
1981#define ROGUE_CR_IRQ_OS0_EVENT_CLEAR_SOURCE_CLRMSK 0xFFFFFFFEU
1982#define ROGUE_CR_IRQ_OS0_EVENT_CLEAR_SOURCE_EN 0x00000001U
1983
1984/* Register ROGUE_CR_IRQ_OS1_EVENT_STATUS */
1985#define ROGUE_CR_IRQ_OS1_EVENT_STATUS 0x10BD8U
1986#define ROGUE_CR_IRQ_OS1_EVENT_STATUS_MASKFULL 0x0000000000000001ULL
1987#define ROGUE_CR_IRQ_OS1_EVENT_STATUS_SOURCE_SHIFT 0U
1988#define ROGUE_CR_IRQ_OS1_EVENT_STATUS_SOURCE_CLRMSK 0xFFFFFFFEU
1989#define ROGUE_CR_IRQ_OS1_EVENT_STATUS_SOURCE_EN 0x00000001U
1990
1991/* Register ROGUE_CR_IRQ_OS1_EVENT_CLEAR */
1992#define ROGUE_CR_IRQ_OS1_EVENT_CLEAR 0x10BE8U
1993#define ROGUE_CR_IRQ_OS1_EVENT_CLEAR_MASKFULL 0x0000000000000001ULL
1994#define ROGUE_CR_IRQ_OS1_EVENT_CLEAR_SOURCE_SHIFT 0U
1995#define ROGUE_CR_IRQ_OS1_EVENT_CLEAR_SOURCE_CLRMSK 0xFFFFFFFEU
1996#define ROGUE_CR_IRQ_OS1_EVENT_CLEAR_SOURCE_EN 0x00000001U
1997
1998/* Register ROGUE_CR_IRQ_OS2_EVENT_STATUS */
1999#define ROGUE_CR_IRQ_OS2_EVENT_STATUS 0x20BD8U
2000#define ROGUE_CR_IRQ_OS2_EVENT_STATUS_MASKFULL 0x0000000000000001ULL
2001#define ROGUE_CR_IRQ_OS2_EVENT_STATUS_SOURCE_SHIFT 0U
2002#define ROGUE_CR_IRQ_OS2_EVENT_STATUS_SOURCE_CLRMSK 0xFFFFFFFEU
2003#define ROGUE_CR_IRQ_OS2_EVENT_STATUS_SOURCE_EN 0x00000001U
2004
2005/* Register ROGUE_CR_IRQ_OS2_EVENT_CLEAR */
2006#define ROGUE_CR_IRQ_OS2_EVENT_CLEAR 0x20BE8U
2007#define ROGUE_CR_IRQ_OS2_EVENT_CLEAR_MASKFULL 0x0000000000000001ULL
2008#define ROGUE_CR_IRQ_OS2_EVENT_CLEAR_SOURCE_SHIFT 0U
2009#define ROGUE_CR_IRQ_OS2_EVENT_CLEAR_SOURCE_CLRMSK 0xFFFFFFFEU
2010#define ROGUE_CR_IRQ_OS2_EVENT_CLEAR_SOURCE_EN 0x00000001U
2011
2012/* Register ROGUE_CR_IRQ_OS3_EVENT_STATUS */
2013#define ROGUE_CR_IRQ_OS3_EVENT_STATUS 0x30BD8U
2014#define ROGUE_CR_IRQ_OS3_EVENT_STATUS_MASKFULL 0x0000000000000001ULL
2015#define ROGUE_CR_IRQ_OS3_EVENT_STATUS_SOURCE_SHIFT 0U
2016#define ROGUE_CR_IRQ_OS3_EVENT_STATUS_SOURCE_CLRMSK 0xFFFFFFFEU
2017#define ROGUE_CR_IRQ_OS3_EVENT_STATUS_SOURCE_EN 0x00000001U
2018
2019/* Register ROGUE_CR_IRQ_OS3_EVENT_CLEAR */
2020#define ROGUE_CR_IRQ_OS3_EVENT_CLEAR 0x30BE8U
2021#define ROGUE_CR_IRQ_OS3_EVENT_CLEAR_MASKFULL 0x0000000000000001ULL
2022#define ROGUE_CR_IRQ_OS3_EVENT_CLEAR_SOURCE_SHIFT 0U
2023#define ROGUE_CR_IRQ_OS3_EVENT_CLEAR_SOURCE_CLRMSK 0xFFFFFFFEU
2024#define ROGUE_CR_IRQ_OS3_EVENT_CLEAR_SOURCE_EN 0x00000001U
2025
2026/* Register ROGUE_CR_IRQ_OS4_EVENT_STATUS */
2027#define ROGUE_CR_IRQ_OS4_EVENT_STATUS 0x40BD8U
2028#define ROGUE_CR_IRQ_OS4_EVENT_STATUS_MASKFULL 0x0000000000000001ULL
2029#define ROGUE_CR_IRQ_OS4_EVENT_STATUS_SOURCE_SHIFT 0U
2030#define ROGUE_CR_IRQ_OS4_EVENT_STATUS_SOURCE_CLRMSK 0xFFFFFFFEU
2031#define ROGUE_CR_IRQ_OS4_EVENT_STATUS_SOURCE_EN 0x00000001U
2032
2033/* Register ROGUE_CR_IRQ_OS4_EVENT_CLEAR */
2034#define ROGUE_CR_IRQ_OS4_EVENT_CLEAR 0x40BE8U
2035#define ROGUE_CR_IRQ_OS4_EVENT_CLEAR_MASKFULL 0x0000000000000001ULL
2036#define ROGUE_CR_IRQ_OS4_EVENT_CLEAR_SOURCE_SHIFT 0U
2037#define ROGUE_CR_IRQ_OS4_EVENT_CLEAR_SOURCE_CLRMSK 0xFFFFFFFEU
2038#define ROGUE_CR_IRQ_OS4_EVENT_CLEAR_SOURCE_EN 0x00000001U
2039

4154 more lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_cr_defs_client.h+159−0

This diff is large: its 160 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_defs.h+179−0

This diff is large: its 180 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif.h+2188−0

This diff is large: its 2189 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_check.h+493−0

This diff is large: its 494 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_client.h+373−0

This diff is large: its 374 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_client_check.h+133−0

This diff is large: its 134 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_common.h+60−0

This diff is large: its 61 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_dev_info.h+113−0

This diff is large: its 114 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_resetframework.h+28−0

This diff is large: its 29 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_sf.h+1648−0

This diff is large: its 1649 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_shared.h+258−0

This diff is large: its 259 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_shared_check.h+108−0

This diff is large: its 109 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_stream.h+78−0

This diff is large: its 79 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_heap_config.h+113−0

This diff is large: its 114 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_meta.h+356−0

This diff is large: its 357 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_mips.h+335−0

This diff is large: its 336 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_mips_check.h+58−0

This diff is large: its 59 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_mmu_defs.h+136−0

This diff is large: its 137 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_stream.c+285−0

This diff is large: its 286 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_stream.h+75−0

This diff is large: its 76 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_stream_defs.c+351−0

This diff is large: its 352 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_stream_defs.h+16−0

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_sync.c+289−0

This diff is large: its 290 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_sync.h+84−0

This diff is large: its 85 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_vm.c+1107−0

This diff is large: its 1108 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_vm.h+65−0

This diff is large: its 66 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_vm_mips.c+238−0

This diff is large: its 239 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_vm_mips.h+22−0

This diff is large: its 23 lines aren’t shown here.

drivers/gpu/drm/imx/dcss/dcss-drv.c+2−4

This diff is large: its 27 lines aren’t shown here.

drivers/gpu/drm/imx/lcdc/imx-lcdc.c+2−4

This diff is large: its 25 lines aren’t shown here.

drivers/gpu/drm/kmb/kmb_drv.c+2−3

This diff is large: its 26 lines aren’t shown here.

drivers/gpu/drm/mediatek/mtk_disp_ovl_adaptor.c+2−3

This diff is large: its 19 lines aren’t shown here.

drivers/gpu/drm/mediatek/mtk_ethdr.c+2−3

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/meson/meson_dw_mipi_dsi.c+2−4

This diff is large: its 24 lines aren’t shown here.

drivers/gpu/drm/nouveau/nouveau_platform.c+2−3

This diff is large: its 20 lines aren’t shown here.

drivers/gpu/drm/nouveau/nv04_fence.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/nouveau/nvkm/engine/fifo/chan.c+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/panel/panel-elida-kd35t133.c+3−34

This diff is large: its 107 lines aren’t shown here.

drivers/gpu/drm/panel/panel-newvision-nv3051d.c+46−11

This diff is large: its 93 lines aren’t shown here.

drivers/gpu/drm/rockchip/rk3066_hdmi.c+18−27

This diff is large: its 93 lines aren’t shown here.

drivers/gpu/drm/rockchip/rockchip_drm_vop2.c+5−0

This diff is large: its 24 lines aren’t shown here.

drivers/gpu/drm/rockchip/rockchip_vop2_reg.c+2−0

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/sprd/sprd_dpu.c+2−4

This diff is large: its 19 lines aren’t shown here.

drivers/gpu/drm/sprd/sprd_drm.c+2−3

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/sprd/sprd_dsi.c+2−4

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/tilcdc/tilcdc_drv.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/tiny/arcpgu.c+2−4

This diff is large: its 25 lines aren’t shown here.

drivers/video/fbdev/simplefb.c+4−0

This diff is large: its 11 lines aren’t shown here.

include/drm/drm_edid.h+0−5

This diff is large: its 12 lines aren’t shown here.

include/drm/drm_gpuvm.h+28−0

This diff is large: its 33 lines aren’t shown here.

include/drm/drm_print.h+1−1

This diff is large: its 9 lines aren’t shown here.

include/linux/sizes.h+9−0

This diff is large: its 18 lines aren’t shown here.

include/uapi/drm/pvr_drm.h+1297−0

This diff is large: its 1298 lines aren’t shown here.

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t