History

Merge tag 'drm-misc-next-2023-11-23' of git://anongit.freedesktop.org/drm/drm-misc into drm-next

drm-misc-next for 6.8: UAPI Changes: Cross-subsystem Changes: Core Changes: - Drop deprecated drm_kms_helper.edid_firmware module parameter Driver Changes: - Convert platform drivers remove callback to return void - imagination: Introduction of the Imagination GPU Support - rockchip: - rk3066_hdmi: Convert to atomic - vop2: Support NV20 and NV30 - panel: - elida-kd35t133: PM reworks - New panels: Powkiddy RK2023

Daniel Vetter committed 2 years ago

Commit b26ca732 parents c79b972 815d8b0115 files · +34629 −201Browse files

Unified · Split

115 files
  1. Documentation/devicetree/bindings/display/panel/newvision,nv3051d.yaml +1 −1
  2. Documentation/devicetree/bindings/gpu/img,powervr.yaml +73 −0
  3. Documentation/gpu/drivers.rst +2 −0
  4. Documentation/gpu/imagination/index.rst +13 −0
  5. Documentation/gpu/imagination/uapi.rst +174 −0
  6. MAINTAINERS +10 −0
  7. drivers/accel/qaic/qaic_data.c +1 −1
  8. drivers/dma-buf/dma-buf.c +2 −2
  9. drivers/gpu/drm/Kconfig +2 −0
  10. drivers/gpu/drm/Makefile +1 −0
  11. drivers/gpu/drm/armada/armada_crtc.c +2 −3
  12. drivers/gpu/drm/armada/armada_drv.c +2 −3
  13. drivers/gpu/drm/bridge/cadence/cdns-mhdp8546-core.c +10 −12
  14. drivers/gpu/drm/bridge/ti-tpd12s015.c +2 −4
  15. drivers/gpu/drm/drm_edid_load.c +0 −16
  16. drivers/gpu/drm/drm_kms_helper_common.c +0 −32
  17. drivers/gpu/drm/etnaviv/etnaviv_drv.c +2 −4
  18. drivers/gpu/drm/etnaviv/etnaviv_gpu.c +2 −3
  19. drivers/gpu/drm/imagination/Kconfig +18 −0
  20. drivers/gpu/drm/imagination/Makefile +35 −0
  21. drivers/gpu/drm/imagination/pvr_ccb.c +645 −0
  22. drivers/gpu/drm/imagination/pvr_ccb.h +71 −0
  23. drivers/gpu/drm/imagination/pvr_cccb.c +267 −0
  24. drivers/gpu/drm/imagination/pvr_cccb.h +109 −0
  25. drivers/gpu/drm/imagination/pvr_context.c +464 −0
  26. drivers/gpu/drm/imagination/pvr_context.h +205 −0
  27. drivers/gpu/drm/imagination/pvr_debugfs.c +53 −0
  28. drivers/gpu/drm/imagination/pvr_debugfs.h +29 −0
  29. drivers/gpu/drm/imagination/pvr_device.c +658 −0
  30. drivers/gpu/drm/imagination/pvr_device.h +710 −0
  31. drivers/gpu/drm/imagination/pvr_device_info.c +254 −0
  32. drivers/gpu/drm/imagination/pvr_device_info.h +186 −0
  33. drivers/gpu/drm/imagination/pvr_drv.c +1501 −0
  34. drivers/gpu/drm/imagination/pvr_drv.h +129 −0
  35. drivers/gpu/drm/imagination/pvr_free_list.c +625 −0
  36. drivers/gpu/drm/imagination/pvr_free_list.h +195 −0
  37. drivers/gpu/drm/imagination/pvr_fw.c +1489 −0
  38. drivers/gpu/drm/imagination/pvr_fw.h +508 −0
  39. drivers/gpu/drm/imagination/pvr_fw_info.h +135 −0
  40. drivers/gpu/drm/imagination/pvr_fw_meta.c +554 −0
  41. drivers/gpu/drm/imagination/pvr_fw_meta.h +14 −0
  42. drivers/gpu/drm/imagination/pvr_fw_mips.c +252 −0
  43. drivers/gpu/drm/imagination/pvr_fw_mips.h +48 −0
  44. drivers/gpu/drm/imagination/pvr_fw_startstop.c +306 −0
  45. drivers/gpu/drm/imagination/pvr_fw_startstop.h +13 −0
  46. drivers/gpu/drm/imagination/pvr_fw_trace.c +515 −0
  47. drivers/gpu/drm/imagination/pvr_fw_trace.h +78 −0
  48. drivers/gpu/drm/imagination/pvr_gem.c +414 −0
  49. drivers/gpu/drm/imagination/pvr_gem.h +170 −0
  50. drivers/gpu/drm/imagination/pvr_hwrt.c +549 −0
  51. drivers/gpu/drm/imagination/pvr_hwrt.h +165 −0
  52. drivers/gpu/drm/imagination/pvr_job.c +788 −0
  53. drivers/gpu/drm/imagination/pvr_job.h +161 −0
  54. drivers/gpu/drm/imagination/pvr_mmu.c +2637 −0
  55. drivers/gpu/drm/imagination/pvr_mmu.h +108 −0
  56. drivers/gpu/drm/imagination/pvr_params.c +147 −0
  57. drivers/gpu/drm/imagination/pvr_params.h +72 −0
  58. drivers/gpu/drm/imagination/pvr_power.c +433 −0
  59. drivers/gpu/drm/imagination/pvr_power.h +41 −0
  60. drivers/gpu/drm/imagination/pvr_queue.c +1432 −0
  61. drivers/gpu/drm/imagination/pvr_queue.h +169 −0
  62. drivers/gpu/drm/imagination/pvr_rogue_cr_defs.h +6193 −0
  63. drivers/gpu/drm/imagination/pvr_rogue_cr_defs_client.h +159 −0
  64. drivers/gpu/drm/imagination/pvr_rogue_defs.h +179 −0
  65. drivers/gpu/drm/imagination/pvr_rogue_fwif.h +2188 −0
  66. drivers/gpu/drm/imagination/pvr_rogue_fwif_check.h +493 −0
  67. drivers/gpu/drm/imagination/pvr_rogue_fwif_client.h +373 −0
  68. drivers/gpu/drm/imagination/pvr_rogue_fwif_client_check.h +133 −0
  69. drivers/gpu/drm/imagination/pvr_rogue_fwif_common.h +60 −0
  70. drivers/gpu/drm/imagination/pvr_rogue_fwif_dev_info.h +113 −0
  71. drivers/gpu/drm/imagination/pvr_rogue_fwif_resetframework.h +28 −0
  72. drivers/gpu/drm/imagination/pvr_rogue_fwif_sf.h +1648 −0
  73. drivers/gpu/drm/imagination/pvr_rogue_fwif_shared.h +258 −0
  74. drivers/gpu/drm/imagination/pvr_rogue_fwif_shared_check.h +108 −0
  75. drivers/gpu/drm/imagination/pvr_rogue_fwif_stream.h +78 −0
  76. drivers/gpu/drm/imagination/pvr_rogue_heap_config.h +113 −0
  77. drivers/gpu/drm/imagination/pvr_rogue_meta.h +356 −0
  78. drivers/gpu/drm/imagination/pvr_rogue_mips.h +335 −0
  79. drivers/gpu/drm/imagination/pvr_rogue_mips_check.h +58 −0
  80. drivers/gpu/drm/imagination/pvr_rogue_mmu_defs.h +136 −0
  81. drivers/gpu/drm/imagination/pvr_stream.c +285 −0
  82. drivers/gpu/drm/imagination/pvr_stream.h +75 −0
  83. drivers/gpu/drm/imagination/pvr_stream_defs.c +351 −0
  84. drivers/gpu/drm/imagination/pvr_stream_defs.h +16 −0
  85. drivers/gpu/drm/imagination/pvr_sync.c +289 −0
  86. drivers/gpu/drm/imagination/pvr_sync.h +84 −0
  87. drivers/gpu/drm/imagination/pvr_vm.c +1107 −0
  88. drivers/gpu/drm/imagination/pvr_vm.h +65 −0
  89. drivers/gpu/drm/imagination/pvr_vm_mips.c +238 −0
  90. drivers/gpu/drm/imagination/pvr_vm_mips.h +22 −0
  91. drivers/gpu/drm/imx/dcss/dcss-drv.c +2 −4
  92. drivers/gpu/drm/imx/lcdc/imx-lcdc.c +2 −4
  93. drivers/gpu/drm/kmb/kmb_drv.c +2 −3
  94. drivers/gpu/drm/mediatek/mtk_disp_ovl_adaptor.c +2 −3
  95. drivers/gpu/drm/mediatek/mtk_ethdr.c +2 −3
  96. drivers/gpu/drm/meson/meson_dw_mipi_dsi.c +2 −4
  97. drivers/gpu/drm/nouveau/nouveau_platform.c +2 −3
  98. drivers/gpu/drm/nouveau/nv04_fence.c +1 −1
  99. drivers/gpu/drm/nouveau/nvkm/engine/fifo/chan.c +0 −1
  100. drivers/gpu/drm/panel/panel-elida-kd35t133.c +3 −34
  101. drivers/gpu/drm/panel/panel-newvision-nv3051d.c +46 −11
  102. drivers/gpu/drm/rockchip/rk3066_hdmi.c +18 −27
  103. drivers/gpu/drm/rockchip/rockchip_drm_vop2.c +5 −0
  104. drivers/gpu/drm/rockchip/rockchip_vop2_reg.c +2 −0
  105. drivers/gpu/drm/sprd/sprd_dpu.c +2 −4
  106. drivers/gpu/drm/sprd/sprd_drm.c +2 −3
  107. drivers/gpu/drm/sprd/sprd_dsi.c +2 −4
  108. drivers/gpu/drm/tilcdc/tilcdc_drv.c +1 −1
  109. drivers/gpu/drm/tiny/arcpgu.c +2 −4
  110. drivers/video/fbdev/simplefb.c +4 −0
  111. include/drm/drm_edid.h +0 −5
  112. include/drm/drm_gpuvm.h +28 −0
  113. include/drm/drm_print.h +1 −1
  114. include/linux/sizes.h +9 −0
  115. include/uapi/drm/pvr_drm.h +1297 −0
Documentation/devicetree/bindings/display/panel/newvision,nv3051d.yaml+1−1
@@ -21,7 +21,7 @@ properties: · 20 unmodified lines
2121 - enum:
2222 - anbernic,rg351v-panel
2323 - anbernic,rg353p-panel
24− - anbernic,rg353v-panel
24+ - powkiddy,rk2023-panel
2525 - const: newvision,nv3051d
2626
2727 reg: true
Documentation/devicetree/bindings/gpu/img,powervr.yaml+73−0
@@ -0,0 +1,73 @@
1+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
2+# Copyright (c) 2023 Imagination Technologies Ltd.
3+%YAML 1.2
4+---
5+$id: http://devicetree.org/schemas/gpu/img,powervr.yaml#
6+$schema: http://devicetree.org/meta-schemas/core.yaml#
7+
8+title: Imagination Technologies PowerVR and IMG GPU
9+
10+maintainers:
11+ - Frank Binns <frank.binns@imgtec.com>
12+
13+properties:
14+ compatible:
15+ items:
16+ - enum:
17+ - ti,am62-gpu
18+ - const: img,img-axe # IMG AXE GPU model/revision is fully discoverable
19+
20+ reg:
21+ maxItems: 1
22+
23+ clocks:
24+ minItems: 1
25+ maxItems: 3
26+
27+ clock-names:
28+ items:
29+ - const: core
30+ - const: mem
31+ - const: sys
32+ minItems: 1
33+
34+ interrupts:
35+ maxItems: 1
36+
37+ power-domains:
38+ maxItems: 1
39+
40+required:
41+ - compatible
42+ - reg
43+ - clocks
44+ - clock-names
45+ - interrupts
46+
47+additionalProperties: false
48+
49+allOf:
50+ - if:
51+ properties:
52+ compatible:
53+ contains:
54+ const: ti,am62-gpu
55+ then:
56+ properties:
57+ clocks:
58+ maxItems: 1
59+
60+examples:
61+ - |
62+ #include <dt-bindings/interrupt-controller/irq.h>
63+ #include <dt-bindings/interrupt-controller/arm-gic.h>
64+ #include <dt-bindings/soc/ti,sci_pm_domain.h>
65+
66+ gpu@fd00000 {
67+ compatible = "ti,am62-gpu", "img,img-axe";
68+ reg = <0x0fd00000 0x20000>;
69+ clocks = <&k3_clks 187 0>;
70+ clock-names = "core";
71+ interrupts = <GIC_SPI 86 IRQ_TYPE_LEVEL_HIGH>;
72+ power-domains = <&k3_pds 187 TI_SCI_PD_EXCLUSIVE>;
73+ };
Documentation/gpu/drivers.rst+2−0
@@ -3,9 +3,11 @@ GPU Driver Documentation · 2 unmodified lines
33 ========================
44
55 .. toctree::
6+ :maxdepth: 3
67
78 amdgpu/index
89 i915
10+ imagination/index
911 mcde
1012 meson
1113 pl111
Documentation/gpu/imagination/index.rst+13−0
@@ -0,0 +1,13 @@
1+=======================================
2+drm/imagination PowerVR Graphics Driver
3+=======================================
4+
5+.. kernel-doc:: drivers/gpu/drm/imagination/pvr_drv.c
6+ :doc: PowerVR Graphics Driver
7+
8+Contents
9+========
10+.. toctree::
11+ :maxdepth: 2
12+
13+ uapi
Documentation/gpu/imagination/uapi.rst+174−0
@@ -0,0 +1,174 @@
1+====
2+UAPI
3+====
4+The sources associated with this section can be found in ``pvr_drm.h``.
5+
6+.. kernel-doc:: include/uapi/drm/pvr_drm.h
7+ :doc: PowerVR UAPI
8+
9+OBJECT ARRAYS
10+=============
11+.. kernel-doc:: include/uapi/drm/pvr_drm.h
12+ :identifiers: drm_pvr_obj_array
13+
14+.. kernel-doc:: include/uapi/drm/pvr_drm.h
15+ :identifiers: DRM_PVR_OBJ_ARRAY
16+
17+IOCTLS
18+======
19+.. kernel-doc:: include/uapi/drm/pvr_drm.h
20+ :doc: PowerVR IOCTL interface
21+
22+.. kernel-doc:: include/uapi/drm/pvr_drm.h
23+ :identifiers: PVR_IOCTL
24+
25+DEV_QUERY
26+---------
27+.. kernel-doc:: include/uapi/drm/pvr_drm.h
28+ :doc: PowerVR IOCTL DEV_QUERY interface
29+
30+.. kernel-doc:: include/uapi/drm/pvr_drm.h
31+ :identifiers: drm_pvr_dev_query
32+
33+.. kernel-doc:: include/uapi/drm/pvr_drm.h
34+ :identifiers: drm_pvr_ioctl_dev_query_args
35+
36+.. kernel-doc:: include/uapi/drm/pvr_drm.h
37+ :identifiers: drm_pvr_dev_query_gpu_info
38+ drm_pvr_dev_query_runtime_info
39+ drm_pvr_dev_query_hwrt_info
40+ drm_pvr_dev_query_quirks
41+ drm_pvr_dev_query_enhancements
42+
43+.. kernel-doc:: include/uapi/drm/pvr_drm.h
44+ :identifiers: drm_pvr_heap_id
45+ drm_pvr_heap
46+ drm_pvr_dev_query_heap_info
47+
48+.. kernel-doc:: include/uapi/drm/pvr_drm.h
49+ :doc: Flags for DRM_PVR_DEV_QUERY_HEAP_INFO_GET.
50+
51+.. kernel-doc:: include/uapi/drm/pvr_drm.h
52+ :identifiers: drm_pvr_static_data_area_usage
53+ drm_pvr_static_data_area
54+ drm_pvr_dev_query_static_data_areas
55+
56+CREATE_BO
57+---------
58+.. kernel-doc:: include/uapi/drm/pvr_drm.h
59+ :doc: PowerVR IOCTL CREATE_BO interface
60+
61+.. kernel-doc:: include/uapi/drm/pvr_drm.h
62+ :identifiers: drm_pvr_ioctl_create_bo_args
63+
64+.. kernel-doc:: include/uapi/drm/pvr_drm.h
65+ :doc: Flags for CREATE_BO
66+
67+GET_BO_MMAP_OFFSET
68+------------------
69+.. kernel-doc:: include/uapi/drm/pvr_drm.h
70+ :doc: PowerVR IOCTL GET_BO_MMAP_OFFSET interface
71+
72+.. kernel-doc:: include/uapi/drm/pvr_drm.h
73+ :identifiers: drm_pvr_ioctl_get_bo_mmap_offset_args
74+
75+CREATE_VM_CONTEXT and DESTROY_VM_CONTEXT
76+----------------------------------------
77+.. kernel-doc:: include/uapi/drm/pvr_drm.h
78+ :doc: PowerVR IOCTL CREATE_VM_CONTEXT and DESTROY_VM_CONTEXT interfaces
79+
80+.. kernel-doc:: include/uapi/drm/pvr_drm.h
81+ :identifiers: drm_pvr_ioctl_create_vm_context_args
82+ drm_pvr_ioctl_destroy_vm_context_args
83+
84+VM_MAP and VM_UNMAP
85+-------------------
86+.. kernel-doc:: include/uapi/drm/pvr_drm.h
87+ :doc: PowerVR IOCTL VM_MAP and VM_UNMAP interfaces
88+
89+.. kernel-doc:: include/uapi/drm/pvr_drm.h
90+ :identifiers: drm_pvr_ioctl_vm_map_args
91+ drm_pvr_ioctl_vm_unmap_args
92+
93+CREATE_CONTEXT and DESTROY_CONTEXT
94+----------------------------------
95+.. kernel-doc:: include/uapi/drm/pvr_drm.h
96+ :doc: PowerVR IOCTL CREATE_CONTEXT and DESTROY_CONTEXT interfaces
97+
98+.. kernel-doc:: include/uapi/drm/pvr_drm.h
99+ :identifiers: drm_pvr_ioctl_create_context_args
100+
101+.. kernel-doc:: include/uapi/drm/pvr_drm.h
102+ :identifiers: drm_pvr_ctx_priority
103+ drm_pvr_ctx_type
104+ drm_pvr_static_render_context_state
105+ drm_pvr_static_render_context_state_format
106+ drm_pvr_reset_framework
107+ drm_pvr_reset_framework_format
108+
109+.. kernel-doc:: include/uapi/drm/pvr_drm.h
110+ :identifiers: drm_pvr_ioctl_destroy_context_args
111+
112+CREATE_FREE_LIST and DESTROY_FREE_LIST
113+--------------------------------------
114+.. kernel-doc:: include/uapi/drm/pvr_drm.h
115+ :doc: PowerVR IOCTL CREATE_FREE_LIST and DESTROY_FREE_LIST interfaces
116+
117+.. kernel-doc:: include/uapi/drm/pvr_drm.h
118+ :identifiers: drm_pvr_ioctl_create_free_list_args
119+
120+.. kernel-doc:: include/uapi/drm/pvr_drm.h
121+ :identifiers: drm_pvr_ioctl_destroy_free_list_args
122+
123+CREATE_HWRT_DATASET and DESTROY_HWRT_DATASET
124+--------------------------------------
125+.. kernel-doc:: include/uapi/drm/pvr_drm.h
126+ :doc: PowerVR IOCTL CREATE_HWRT_DATASET and DESTROY_HWRT_DATASET interfaces
127+
128+.. kernel-doc:: include/uapi/drm/pvr_drm.h
129+ :identifiers: drm_pvr_ioctl_create_hwrt_dataset_args
130+
131+.. kernel-doc:: include/uapi/drm/pvr_drm.h
132+ :identifiers: drm_pvr_create_hwrt_geom_data_args
133+ drm_pvr_create_hwrt_rt_data_args
134+
135+.. kernel-doc:: include/uapi/drm/pvr_drm.h
136+ :identifiers: drm_pvr_ioctl_destroy_hwrt_dataset_args
137+
138+SUBMIT_JOBS
139+-----------
140+.. kernel-doc:: include/uapi/drm/pvr_drm.h
141+ :doc: PowerVR IOCTL SUBMIT_JOBS interface
142+
143+.. kernel-doc:: include/uapi/drm/pvr_drm.h
144+ :doc: Flags for the drm_pvr_sync_op object.
145+
146+.. kernel-doc:: include/uapi/drm/pvr_drm.h
147+ :identifiers: drm_pvr_ioctl_submit_jobs_args
148+
149+.. kernel-doc:: include/uapi/drm/pvr_drm.h
150+ :doc: Flags for SUBMIT_JOB ioctl geometry command.
151+
152+.. kernel-doc:: include/uapi/drm/pvr_drm.h
153+ :doc: Flags for SUBMIT_JOB ioctl fragment command.
154+
155+.. kernel-doc:: include/uapi/drm/pvr_drm.h
156+ :doc: Flags for SUBMIT_JOB ioctl compute command.
157+
158+.. kernel-doc:: include/uapi/drm/pvr_drm.h
159+ :doc: Flags for SUBMIT_JOB ioctl transfer command.
160+
161+.. kernel-doc:: include/uapi/drm/pvr_drm.h
162+ :identifiers: drm_pvr_sync_op
163+ drm_pvr_job_type
164+ drm_pvr_hwrt_data_ref
165+ drm_pvr_job
166+
167+Internal notes
168+==============
169+.. kernel-doc:: drivers/gpu/drm/imagination/pvr_device.h
170+ :doc: IOCTL validation helpers
171+
172+.. kernel-doc:: drivers/gpu/drm/imagination/pvr_device.h
173+ :identifiers: PVR_STATIC_ASSERT_64BIT_ALIGNED PVR_IOCTL_UNION_PADDING_CHECK
174+ pvr_ioctl_union_padding_check
MAINTAINERS+10−0
@@ -10391,6 +10391,16 @@ IMGTEC IR DECODER DRIVER · 10390 unmodified lines
1039110391 S: Orphan
1039210392 F: drivers/media/rc/img-ir/
1039310393
10394+IMGTEC POWERVR DRM DRIVER
10395+M: Frank Binns <frank.binns@imgtec.com>
10396+M: Donald Robson <donald.robson@imgtec.com>
10397+M: Matt Coster <matt.coster@imgtec.com>
10398+S: Supported
10399+F: Documentation/devicetree/bindings/gpu/img,powervr.yaml
10400+F: Documentation/gpu/imagination/
10401+F: drivers/gpu/drm/imagination/
10402+F: include/uapi/drm/pvr_drm.h
10403+
1039410404 IMON SOUNDGRAPH USB IR RECEIVER
1039510405 M: Sean Young <sean@mess.org>
1039610406 L: linux-media@vger.kernel.org
drivers/accel/qaic/qaic_data.c+1−1
@@ -452,7 +452,7 @@ static int create_sgt(struct qaic_device *qdev, struct sg_table **sgt_out, u64 s · 451 unmodified lines
452452 * later
453453 */
454454 buf_extra = (PAGE_SIZE - size % PAGE_SIZE) % PAGE_SIZE;
455− max_order = min(MAX_ORDER - 1, get_order(size));
455+ max_order = min(MAX_ORDER, get_order(size));
456456 } else {
457457 /* allocate a single page for book keeping */
458458 nr_pages = 1;
drivers/dma-buf/dma-buf.c+2−2
@@ -46,12 +46,12 @@ static char *dmabuffs_dname(struct dentry *dentry, char *buffer, int buflen) · 45 unmodified lines
4646 {
4747 struct dma_buf *dmabuf;
4848 char name[DMA_BUF_NAME_LEN];
49− size_t ret = 0;
49+ ssize_t ret = 0;
5050
5151 dmabuf = dentry->d_fsdata;
5252 spin_lock(&dmabuf->name_lock);
5353 if (dmabuf->name)
54− ret = strlcpy(name, dmabuf->name, DMA_BUF_NAME_LEN);
54+ ret = strscpy(name, dmabuf->name, sizeof(name));
5555 spin_unlock(&dmabuf->name_lock);
5656
5757 return dynamic_dname(buffer, buflen, "/%s:%s",
drivers/gpu/drm/Kconfig+2−0
@@ -394,6 +394,8 @@ source "drivers/gpu/drm/solomon/Kconfig" · 393 unmodified lines
394394
395395 source "drivers/gpu/drm/sprd/Kconfig"
396396
397+source "drivers/gpu/drm/imagination/Kconfig"
398+
397399 config DRM_HYPERV
398400 tristate "DRM Support for Hyper-V synthetic video device"
399401 depends on DRM && PCI && MMU && HYPERV
drivers/gpu/drm/Makefile+1−0
@@ -199,3 +199,4 @@ obj-$(CONFIG_DRM_HYPERV) += hyperv/ · 198 unmodified lines
199199 obj-y += solomon/
200200 obj-$(CONFIG_DRM_SPRD) += sprd/
201201 obj-$(CONFIG_DRM_LOONGSON) += loongson/
202+obj-$(CONFIG_DRM_POWERVR) += imagination/
drivers/gpu/drm/armada/armada_crtc.c+2−3
@@ -1066,10 +1066,9 @@ static int armada_lcd_probe(struct platform_device *pdev) · 1065 unmodified lines
10661066 return component_add(&pdev->dev, &armada_lcd_ops);
10671067 }
10681068
1069−static int armada_lcd_remove(struct platform_device *pdev)
1069+static void armada_lcd_remove(struct platform_device *pdev)
10701070 {
10711071 component_del(&pdev->dev, &armada_lcd_ops);
1072− return 0;
10731072 }
10741073
10751074 static const struct of_device_id armada_lcd_of_match[] = {
@@ -1095,7 +1094,7 @@ MODULE_DEVICE_TABLE(platform, armada_lcd_platform_ids); · 19 unmodified lines
10951094
10961095 struct platform_driver armada_lcd_platform_driver = {
10971096 .probe = armada_lcd_probe,
1098− .remove = armada_lcd_remove,
1097+ .remove_new = armada_lcd_remove,
10991098 .driver = {
11001099 .name = "armada-lcd",
11011100 .owner = THIS_MODULE,
drivers/gpu/drm/armada/armada_drv.c+2−3
@@ -226,10 +226,9 @@ static int armada_drm_probe(struct platform_device *pdev) · 225 unmodified lines
226226 match);
227227 }
228228
229−static int armada_drm_remove(struct platform_device *pdev)
229+static void armada_drm_remove(struct platform_device *pdev)
230230 {
231231 component_master_del(&pdev->dev, &armada_master_ops);
232− return 0;
233232 }
234233
235234 static void armada_drm_shutdown(struct platform_device *pdev)
@@ -249,7 +248,7 @@ MODULE_DEVICE_TABLE(platform, armada_drm_platform_ids); · 13 unmodified lines
249248
250249 static struct platform_driver armada_drm_platform_driver = {
251250 .probe = armada_drm_probe,
252− .remove = armada_drm_remove,
251+ .remove_new = armada_drm_remove,
253252 .shutdown = armada_drm_shutdown,
254253 .driver = {
255254 .name = "armada-drm",
drivers/gpu/drm/bridge/cadence/cdns-mhdp8546-core.c+10−12
@@ -2596,11 +2596,10 @@ clk_disable: · 2595 unmodified lines
25962596 return ret;
25972597 }
25982598
2599−static int cdns_mhdp_remove(struct platform_device *pdev)
2599+static void cdns_mhdp_remove(struct platform_device *pdev)
26002600 {
26012601 struct cdns_mhdp_device *mhdp = platform_get_drvdata(pdev);
26022602 unsigned long timeout = msecs_to_jiffies(100);
2603− bool stop_fw = false;
26042603 int ret;
26052604
26062605 drm_bridge_remove(&mhdp->bridge);
@@ -2608,18 +2607,19 @@ static int cdns_mhdp_remove(struct platform_device *pdev) · 1 unmodified line
26082607 ret = wait_event_timeout(mhdp->fw_load_wq,
26092608 mhdp->hw_state == MHDP_HW_READY,
26102609 timeout);
2611− if (ret == 0)
2612− dev_err(mhdp->dev, "%s: Timeout waiting for fw loading\n",
2613− __func__);
2614− else
2615− stop_fw = true;
2616−
26172610 spin_lock(&mhdp->start_lock);
26182611 mhdp->hw_state = MHDP_HW_STOPPED;
26192612 spin_unlock(&mhdp->start_lock);
26202613
2621− if (stop_fw)
2614+ if (ret == 0) {
2615+ dev_err(mhdp->dev, "%s: Timeout waiting for fw loading\n",
2616+ __func__);
2617+ } else {
26222618 ret = cdns_mhdp_set_firmware_active(mhdp, false);
2619+ if (ret)
2620+ dev_err(mhdp->dev, "Failed to stop firmware (%pe)\n",
2621+ ERR_PTR(ret));
2622+ }
26232623
26242624 phy_exit(mhdp->phy);
26252625
@@ -2634,8 +2634,6 @@ static int cdns_mhdp_remove(struct platform_device *pdev) · 8 unmodified lines
26342634 /* Ignoring mhdp->hdcp.check_work and mhdp->hdcp.prop_work here. */
26352635
26362636 clk_disable_unprepare(mhdp->clk);
2637−
2638− return ret;
26392637 }
26402638
26412639 static const struct of_device_id mhdp_ids[] = {
@@ -2658,7 +2656,7 @@ static struct platform_driver mhdp_driver = { · 16 unmodified lines
26582656 .of_match_table = mhdp_ids,
26592657 },
26602658 .probe = cdns_mhdp_probe,
2661− .remove = cdns_mhdp_remove,
2659+ .remove_new = cdns_mhdp_remove,
26622660 };
26632661 module_platform_driver(mhdp_driver);
26642662
drivers/gpu/drm/bridge/ti-tpd12s015.c+2−4
@@ -179,13 +179,11 @@ static int tpd12s015_probe(struct platform_device *pdev) · 178 unmodified lines
179179 return 0;
180180 }
181181
182−static int __exit tpd12s015_remove(struct platform_device *pdev)
182+static void tpd12s015_remove(struct platform_device *pdev)
183183 {
184184 struct tpd12s015_device *tpd = platform_get_drvdata(pdev);
185185
186186 drm_bridge_remove(&tpd->bridge);
187−
188− return 0;
189187 }
190188
191189 static const struct of_device_id tpd12s015_of_match[] = {
@@ -197,7 +195,7 @@ MODULE_DEVICE_TABLE(of, tpd12s015_of_match); · 5 unmodified lines
197195
198196 static struct platform_driver tpd12s015_driver = {
199197 .probe = tpd12s015_probe,
200− .remove = __exit_p(tpd12s015_remove),
198+ .remove_new = tpd12s015_remove,
201199 .driver = {
202200 .name = "tpd12s015",
203201 .of_match_table = tpd12s015_of_match,
drivers/gpu/drm/drm_edid_load.c+0−16
@@ -23,22 +23,6 @@ module_param_string(edid_firmware, edid_firmware, sizeof(edid_firmware), 0644); · 22 unmodified lines
2323 MODULE_PARM_DESC(edid_firmware, "Do not probe monitor, use specified EDID blob "
2424 "from built-in data or /lib/firmware instead. ");
2525
26−/* Use only for backward compatibility with drm_kms_helper.edid_firmware */
27−int __drm_set_edid_firmware_path(const char *path)
28−{
29− scnprintf(edid_firmware, sizeof(edid_firmware), "%s", path);
30−
31− return 0;
32−}
33−EXPORT_SYMBOL(__drm_set_edid_firmware_path);
34−
35−/* Use only for backward compatibility with drm_kms_helper.edid_firmware */
36−int __drm_get_edid_firmware_path(char *buf, size_t bufsize)
37−{
38− return scnprintf(buf, bufsize, "%s", edid_firmware);
39−}
40−EXPORT_SYMBOL(__drm_get_edid_firmware_path);
41−
4226 #define GENERIC_EDIDS 6
4327 static const char * const generic_edid_name[GENERIC_EDIDS] = {
4428 "edid/800x600.bin",
drivers/gpu/drm/drm_kms_helper_common.c+0−32
@@ -27,38 +27,6 @@ · 26 unmodified lines
2727
2828 #include <linux/module.h>
2929
30−#include <drm/drm_edid.h>
31−#include <drm/drm_print.h>
32−
33−#include "drm_crtc_helper_internal.h"
34−
3530 MODULE_AUTHOR("David Airlie, Jesse Barnes");
3631 MODULE_DESCRIPTION("DRM KMS helper");
3732 MODULE_LICENSE("GPL and additional rights");
38−
39−#if IS_ENABLED(CONFIG_DRM_LOAD_EDID_FIRMWARE)
40−
41−/* Backward compatibility for drm_kms_helper.edid_firmware */
42−static int edid_firmware_set(const char *val, const struct kernel_param *kp)
43−{
44− DRM_NOTE("drm_kms_helper.edid_firmware is deprecated, please use drm.edid_firmware instead.\n");
45−
46− return __drm_set_edid_firmware_path(val);
47−}
48−
49−static int edid_firmware_get(char *buffer, const struct kernel_param *kp)
50−{
51− return __drm_get_edid_firmware_path(buffer, PAGE_SIZE);
52−}
53−
54−static const struct kernel_param_ops edid_firmware_ops = {
55− .set = edid_firmware_set,
56− .get = edid_firmware_get,
57−};
58−
59−module_param_cb(edid_firmware, &edid_firmware_ops, NULL, 0644);
60−__MODULE_PARM_TYPE(edid_firmware, "charp");
61−MODULE_PARM_DESC(edid_firmware,
62− "DEPRECATED. Use drm.edid_firmware module parameter instead.");
63−
64−#endif
drivers/gpu/drm/etnaviv/etnaviv_drv.c+2−4
@@ -640,16 +640,14 @@ static int etnaviv_pdev_probe(struct platform_device *pdev) · 639 unmodified lines
640640 return component_master_add_with_match(dev, &etnaviv_master_ops, match);
641641 }
642642
643−static int etnaviv_pdev_remove(struct platform_device *pdev)
643+static void etnaviv_pdev_remove(struct platform_device *pdev)
644644 {
645645 component_master_del(&pdev->dev, &etnaviv_master_ops);
646−
647− return 0;
648646 }
649647
650648 static struct platform_driver etnaviv_platform_driver = {
651649 .probe = etnaviv_pdev_probe,
652− .remove = etnaviv_pdev_remove,
650+ .remove_new = etnaviv_pdev_remove,
653651 .driver = {
654652 .name = "etnaviv",
655653 },
drivers/gpu/drm/etnaviv/etnaviv_gpu.c+2−3
@@ -1904,11 +1904,10 @@ static int etnaviv_gpu_platform_probe(struct platform_device *pdev) · 1903 unmodified lines
19041904 return 0;
19051905 }
19061906
1907−static int etnaviv_gpu_platform_remove(struct platform_device *pdev)
1907+static void etnaviv_gpu_platform_remove(struct platform_device *pdev)
19081908 {
19091909 component_del(&pdev->dev, &gpu_ops);
19101910 pm_runtime_disable(&pdev->dev);
1911− return 0;
19121911 }
19131912
19141913 static int etnaviv_gpu_rpm_suspend(struct device *dev)
@@ -1970,6 +1969,6 @@ struct platform_driver etnaviv_gpu_driver = { · 55 unmodified lines
19701969 .of_match_table = etnaviv_gpu_match,
19711970 },
19721971 .probe = etnaviv_gpu_platform_probe,
1973− .remove = etnaviv_gpu_platform_remove,
1972+ .remove_new = etnaviv_gpu_platform_remove,
19741973 .id_table = gpu_ids,
19751974 };
drivers/gpu/drm/imagination/Kconfig+18−0
@@ -0,0 +1,18 @@
1+# SPDX-License-Identifier: GPL-2.0-only OR MIT
2+# Copyright (c) 2023 Imagination Technologies Ltd.
3+
4+config DRM_POWERVR
5+ tristate "Imagination Technologies PowerVR (Series 6 and later) & IMG Graphics"
6+ depends on ARM64
7+ depends on DRM
8+ depends on PM
9+ select DRM_EXEC
10+ select DRM_GEM_SHMEM_HELPER
11+ select DRM_SCHED
12+ select DRM_GPUVM
13+ select FW_LOADER
14+ help
15+ Choose this option if you have a system that has an Imagination
16+ Technologies PowerVR (Series 6 or later) or IMG GPU.
17+
18+ If "M" is selected, the module will be called powervr.
drivers/gpu/drm/imagination/Makefile+35−0
@@ -0,0 +1,35 @@
1+# SPDX-License-Identifier: GPL-2.0-only OR MIT
2+# Copyright (c) 2023 Imagination Technologies Ltd.
3+
4+subdir-ccflags-y := -I$(srctree)/$(src)
5+
6+powervr-y := \
7+ pvr_ccb.o \
8+ pvr_cccb.o \
9+ pvr_context.o \
10+ pvr_device.o \
11+ pvr_device_info.o \
12+ pvr_drv.o \
13+ pvr_free_list.o \
14+ pvr_fw.o \
15+ pvr_fw_meta.o \
16+ pvr_fw_mips.o \
17+ pvr_fw_startstop.o \
18+ pvr_fw_trace.o \
19+ pvr_gem.o \
20+ pvr_hwrt.o \
21+ pvr_job.o \
22+ pvr_mmu.o \
23+ pvr_params.o \
24+ pvr_power.o \
25+ pvr_queue.o \
26+ pvr_stream.o \
27+ pvr_stream_defs.o \
28+ pvr_sync.o \
29+ pvr_vm.o \
30+ pvr_vm_mips.o
31+
32+powervr-$(CONFIG_DEBUG_FS) += \
33+ pvr_debugfs.o
34+
35+obj-$(CONFIG_DRM_POWERVR) += powervr.o
drivers/gpu/drm/imagination/pvr_ccb.c+645−0
@@ -0,0 +1,645 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_ccb.h"
5+#include "pvr_device.h"
6+#include "pvr_drv.h"
7+#include "pvr_free_list.h"
8+#include "pvr_fw.h"
9+#include "pvr_gem.h"
10+#include "pvr_power.h"
11+
12+#include <drm/drm_managed.h>
13+#include <linux/compiler.h>
14+#include <linux/delay.h>
15+#include <linux/jiffies.h>
16+#include <linux/kernel.h>
17+#include <linux/mutex.h>
18+#include <linux/types.h>
19+#include <linux/workqueue.h>
20+
21+#define RESERVE_SLOT_TIMEOUT (1 * HZ) /* 1s */
22+#define RESERVE_SLOT_MIN_RETRIES 10
23+
24+static void
25+ccb_ctrl_init(void *cpu_ptr, void *priv)
26+{
27+ struct rogue_fwif_ccb_ctl *ctrl = cpu_ptr;
28+ struct pvr_ccb *pvr_ccb = priv;
29+
30+ ctrl->write_offset = 0;
31+ ctrl->read_offset = 0;
32+ ctrl->wrap_mask = pvr_ccb->num_cmds - 1;
33+ ctrl->cmd_size = pvr_ccb->cmd_size;
34+}
35+
36+/**
37+ * pvr_ccb_init() - Initialise a CCB
38+ * @pvr_dev: Device pointer.
39+ * @pvr_ccb: Pointer to CCB structure to initialise.
40+ * @num_cmds_log2: Log2 of number of commands in this CCB.
41+ * @cmd_size: Command size for this CCB.
42+ *
43+ * Return:
44+ * * Zero on success, or
45+ * * Any error code returned by pvr_fw_object_create_and_map().
46+ */
47+static int
48+pvr_ccb_init(struct pvr_device *pvr_dev, struct pvr_ccb *pvr_ccb,
49+ u32 num_cmds_log2, size_t cmd_size)
50+{
51+ u32 num_cmds = 1 << num_cmds_log2;
52+ u32 ccb_size = num_cmds * cmd_size;
53+ int err;
54+
55+ pvr_ccb->num_cmds = num_cmds;
56+ pvr_ccb->cmd_size = cmd_size;
57+
58+ err = drmm_mutex_init(from_pvr_device(pvr_dev), &pvr_ccb->lock);
59+ if (err)
60+ return err;
61+
62+ /*
63+ * Map CCB and control structure as uncached, so we don't have to flush
64+ * CPU cache repeatedly when polling for space.
65+ */
66+ pvr_ccb->ctrl = pvr_fw_object_create_and_map(pvr_dev, sizeof(*pvr_ccb->ctrl),
67+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
68+ ccb_ctrl_init, pvr_ccb, &pvr_ccb->ctrl_obj);
69+ if (IS_ERR(pvr_ccb->ctrl))
70+ return PTR_ERR(pvr_ccb->ctrl);
71+
72+ pvr_ccb->ccb = pvr_fw_object_create_and_map(pvr_dev, ccb_size,
73+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
74+ NULL, NULL, &pvr_ccb->ccb_obj);
75+ if (IS_ERR(pvr_ccb->ccb)) {
76+ err = PTR_ERR(pvr_ccb->ccb);
77+ goto err_free_ctrl;
78+ }
79+
80+ pvr_fw_object_get_fw_addr(pvr_ccb->ctrl_obj, &pvr_ccb->ctrl_fw_addr);
81+ pvr_fw_object_get_fw_addr(pvr_ccb->ccb_obj, &pvr_ccb->ccb_fw_addr);
82+
83+ WRITE_ONCE(pvr_ccb->ctrl->write_offset, 0);
84+ WRITE_ONCE(pvr_ccb->ctrl->read_offset, 0);
85+ WRITE_ONCE(pvr_ccb->ctrl->wrap_mask, num_cmds - 1);
86+ WRITE_ONCE(pvr_ccb->ctrl->cmd_size, cmd_size);
87+
88+ return 0;
89+
90+err_free_ctrl:
91+ pvr_fw_object_unmap_and_destroy(pvr_ccb->ctrl_obj);
92+
93+ return err;
94+}
95+
96+/**
97+ * pvr_ccb_fini() - Release CCB structure
98+ * @pvr_ccb: CCB to release.
99+ */
100+void
101+pvr_ccb_fini(struct pvr_ccb *pvr_ccb)
102+{
103+ pvr_fw_object_unmap_and_destroy(pvr_ccb->ccb_obj);
104+ pvr_fw_object_unmap_and_destroy(pvr_ccb->ctrl_obj);
105+}
106+
107+/**
108+ * pvr_ccb_slot_available_locked() - Test whether any slots are available in CCB
109+ * @pvr_ccb: CCB to test.
110+ * @write_offset: Address to store number of next available slot. May be %NULL.
111+ *
112+ * Caller must hold @pvr_ccb->lock.
113+ *
114+ * Return:
115+ * * %true if a slot is available, or
116+ * * %false if no slot is available.
117+ */
118+static __always_inline bool
119+pvr_ccb_slot_available_locked(struct pvr_ccb *pvr_ccb, u32 *write_offset)
120+{
121+ struct rogue_fwif_ccb_ctl *ctrl = pvr_ccb->ctrl;
122+ u32 next_write_offset = (READ_ONCE(ctrl->write_offset) + 1) & READ_ONCE(ctrl->wrap_mask);
123+
124+ lockdep_assert_held(&pvr_ccb->lock);
125+
126+ if (READ_ONCE(ctrl->read_offset) != next_write_offset) {
127+ if (write_offset)
128+ *write_offset = next_write_offset;
129+ return true;
130+ }
131+
132+ return false;
133+}
134+
135+static void
136+process_fwccb_command(struct pvr_device *pvr_dev, struct rogue_fwif_fwccb_cmd *cmd)
137+{
138+ switch (cmd->cmd_type) {
139+ case ROGUE_FWIF_FWCCB_CMD_REQUEST_GPU_RESTART:
140+ pvr_power_reset(pvr_dev, false);
141+ break;
142+
143+ case ROGUE_FWIF_FWCCB_CMD_FREELISTS_RECONSTRUCTION:
144+ pvr_free_list_process_reconstruct_req(pvr_dev,
145+ &cmd->cmd_data.cmd_freelists_reconstruction);
146+ break;
147+
148+ case ROGUE_FWIF_FWCCB_CMD_FREELIST_GROW:
149+ pvr_free_list_process_grow_req(pvr_dev, &cmd->cmd_data.cmd_free_list_gs);
150+ break;
151+
152+ default:
153+ drm_info(from_pvr_device(pvr_dev), "Received unknown FWCCB command %x\n",
154+ cmd->cmd_type);
155+ break;
156+ }
157+}
158+
159+/**
160+ * pvr_fwccb_process() - Process any pending FWCCB commands
161+ * @pvr_dev: Target PowerVR device
162+ */
163+void pvr_fwccb_process(struct pvr_device *pvr_dev)
164+{
165+ struct rogue_fwif_fwccb_cmd *fwccb = pvr_dev->fwccb.ccb;
166+ struct rogue_fwif_ccb_ctl *ctrl = pvr_dev->fwccb.ctrl;
167+ u32 read_offset;
168+
169+ mutex_lock(&pvr_dev->fwccb.lock);
170+
171+ while ((read_offset = READ_ONCE(ctrl->read_offset)) != READ_ONCE(ctrl->write_offset)) {
172+ struct rogue_fwif_fwccb_cmd cmd = fwccb[read_offset];
173+
174+ WRITE_ONCE(ctrl->read_offset, (read_offset + 1) & READ_ONCE(ctrl->wrap_mask));
175+
176+ /* Drop FWCCB lock while we process command. */
177+ mutex_unlock(&pvr_dev->fwccb.lock);
178+
179+ process_fwccb_command(pvr_dev, &cmd);
180+
181+ mutex_lock(&pvr_dev->fwccb.lock);
182+ }
183+
184+ mutex_unlock(&pvr_dev->fwccb.lock);
185+}
186+
187+/**
188+ * pvr_kccb_capacity() - Returns the maximum number of usable KCCB slots.
189+ * @pvr_dev: Target PowerVR device
190+ *
191+ * Return:
192+ * * The maximum number of active slots.
193+ */
194+static u32 pvr_kccb_capacity(struct pvr_device *pvr_dev)
195+{
196+ /* Capacity is the number of slot minus one to cope with the wrapping
197+ * mechanisms. If we were to use all slots, we might end up with
198+ * read_offset == write_offset, which the FW considers as a KCCB-is-empty
199+ * condition.
200+ */
201+ return pvr_dev->kccb.slot_count - 1;
202+}
203+
204+/**
205+ * pvr_kccb_used_slot_count_locked() - Get the number of used slots
206+ * @pvr_dev: Device pointer.
207+ *
208+ * KCCB lock must be held.
209+ *
210+ * Return:
211+ * * The number of slots currently used.
212+ */
213+static u32
214+pvr_kccb_used_slot_count_locked(struct pvr_device *pvr_dev)
215+{
216+ struct pvr_ccb *pvr_ccb = &pvr_dev->kccb.ccb;
217+ struct rogue_fwif_ccb_ctl *ctrl = pvr_ccb->ctrl;
218+ u32 wr_offset = READ_ONCE(ctrl->write_offset);
219+ u32 rd_offset = READ_ONCE(ctrl->read_offset);
220+ u32 used_count;
221+
222+ lockdep_assert_held(&pvr_ccb->lock);
223+
224+ if (wr_offset >= rd_offset)
225+ used_count = wr_offset - rd_offset;
226+ else
227+ used_count = wr_offset + pvr_dev->kccb.slot_count - rd_offset;
228+
229+ return used_count;
230+}
231+
232+/**
233+ * pvr_kccb_send_cmd_reserved_powered() - Send command to the KCCB, with the PM ref
234+ * held and a slot pre-reserved
235+ * @pvr_dev: Device pointer.
236+ * @cmd: Command to sent.
237+ * @kccb_slot: Address to store the KCCB slot for this command. May be %NULL.
238+ */
239+void
240+pvr_kccb_send_cmd_reserved_powered(struct pvr_device *pvr_dev,
241+ struct rogue_fwif_kccb_cmd *cmd,
242+ u32 *kccb_slot)
243+{
244+ struct pvr_ccb *pvr_ccb = &pvr_dev->kccb.ccb;
245+ struct rogue_fwif_kccb_cmd *kccb = pvr_ccb->ccb;
246+ struct rogue_fwif_ccb_ctl *ctrl = pvr_ccb->ctrl;
247+ u32 old_write_offset;
248+ u32 new_write_offset;
249+
250+ WARN_ON(pvr_dev->lost);
251+
252+ mutex_lock(&pvr_ccb->lock);
253+
254+ if (WARN_ON(!pvr_dev->kccb.reserved_count))
255+ goto out_unlock;
256+
257+ old_write_offset = READ_ONCE(ctrl->write_offset);
258+
259+ /* We reserved the slot, we should have one available. */
260+ if (WARN_ON(!pvr_ccb_slot_available_locked(pvr_ccb, &new_write_offset)))
261+ goto out_unlock;
262+
263+ memcpy(&kccb[old_write_offset], cmd,
264+ sizeof(struct rogue_fwif_kccb_cmd));
265+ if (kccb_slot) {
266+ *kccb_slot = old_write_offset;
267+ /* Clear return status for this slot. */
268+ WRITE_ONCE(pvr_dev->kccb.rtn[old_write_offset],
269+ ROGUE_FWIF_KCCB_RTN_SLOT_NO_RESPONSE);
270+ }
271+ mb(); /* memory barrier */
272+ WRITE_ONCE(ctrl->write_offset, new_write_offset);
273+ pvr_dev->kccb.reserved_count--;
274+
275+ /* Kick MTS */
276+ pvr_fw_mts_schedule(pvr_dev,
277+ PVR_FWIF_DM_GP & ~ROGUE_CR_MTS_SCHEDULE_DM_CLRMSK);
278+
279+out_unlock:
280+ mutex_unlock(&pvr_ccb->lock);
281+}
282+
283+/**
284+ * pvr_kccb_try_reserve_slot() - Try to reserve a KCCB slot
285+ * @pvr_dev: Device pointer.
286+ *
287+ * Return:
288+ * * true if a KCCB slot was reserved, or
289+ * * false otherwise.
290+ */
291+static bool pvr_kccb_try_reserve_slot(struct pvr_device *pvr_dev)
292+{
293+ bool reserved = false;
294+ u32 used_count;
295+
296+ mutex_lock(&pvr_dev->kccb.ccb.lock);
297+
298+ used_count = pvr_kccb_used_slot_count_locked(pvr_dev);
299+ if (pvr_dev->kccb.reserved_count < pvr_kccb_capacity(pvr_dev) - used_count) {
300+ pvr_dev->kccb.reserved_count++;
301+ reserved = true;
302+ }
303+
304+ mutex_unlock(&pvr_dev->kccb.ccb.lock);
305+
306+ return reserved;
307+}
308+
309+/**
310+ * pvr_kccb_reserve_slot_sync() - Try to reserve a slot synchronously
311+ * @pvr_dev: Device pointer.
312+ *
313+ * Return:
314+ * * 0 on success, or
315+ * * -EBUSY if no slots were reserved after %RESERVE_SLOT_TIMEOUT, with a minimum of
316+ * %RESERVE_SLOT_MIN_RETRIES retries.
317+ */
318+static int pvr_kccb_reserve_slot_sync(struct pvr_device *pvr_dev)
319+{
320+ unsigned long start_timestamp = jiffies;
321+ bool reserved = false;
322+ u32 retries = 0;
323+
324+ while ((jiffies - start_timestamp) < (u32)RESERVE_SLOT_TIMEOUT ||
325+ retries < RESERVE_SLOT_MIN_RETRIES) {
326+ reserved = pvr_kccb_try_reserve_slot(pvr_dev);
327+ if (reserved)
328+ break;
329+
330+ usleep_range(1, 50);
331+
332+ if (retries < U32_MAX)
333+ retries++;
334+ }
335+
336+ return reserved ? 0 : -EBUSY;
337+}
338+
339+/**
340+ * pvr_kccb_send_cmd_powered() - Send command to the KCCB, with a PM ref held
341+ * @pvr_dev: Device pointer.
342+ * @cmd: Command to sent.
343+ * @kccb_slot: Address to store the KCCB slot for this command. May be %NULL.
344+ *
345+ * Returns:
346+ * * Zero on success, or
347+ * * -EBUSY if timeout while waiting for a free KCCB slot.
348+ */
349+int
350+pvr_kccb_send_cmd_powered(struct pvr_device *pvr_dev, struct rogue_fwif_kccb_cmd *cmd,
351+ u32 *kccb_slot)
352+{
353+ int err;
354+
355+ err = pvr_kccb_reserve_slot_sync(pvr_dev);
356+ if (err)
357+ return err;
358+
359+ pvr_kccb_send_cmd_reserved_powered(pvr_dev, cmd, kccb_slot);
360+ return 0;
361+}
362+
363+/**
364+ * pvr_kccb_send_cmd() - Send command to the KCCB
365+ * @pvr_dev: Device pointer.
366+ * @cmd: Command to sent.
367+ * @kccb_slot: Address to store the KCCB slot for this command. May be %NULL.
368+ *
369+ * Returns:
370+ * * Zero on success, or
371+ * * -EBUSY if timeout while waiting for a free KCCB slot.
372+ */
373+int
374+pvr_kccb_send_cmd(struct pvr_device *pvr_dev, struct rogue_fwif_kccb_cmd *cmd,
375+ u32 *kccb_slot)
376+{
377+ int err;
378+
379+ err = pvr_power_get(pvr_dev);
380+ if (err)
381+ return err;
382+
383+ err = pvr_kccb_send_cmd_powered(pvr_dev, cmd, kccb_slot);
384+
385+ pvr_power_put(pvr_dev);
386+
387+ return err;
388+}
389+
390+/**
391+ * pvr_kccb_wait_for_completion() - Wait for a KCCB command to complete
392+ * @pvr_dev: Device pointer.
393+ * @slot_nr: KCCB slot to wait on.
394+ * @timeout: Timeout length (in jiffies).
395+ * @rtn_out: Location to store KCCB command result. May be %NULL.
396+ *
397+ * Returns:
398+ * * Zero on success, or
399+ * * -ETIMEDOUT on timeout.
400+ */
401+int
402+pvr_kccb_wait_for_completion(struct pvr_device *pvr_dev, u32 slot_nr,
403+ u32 timeout, u32 *rtn_out)
404+{
405+ int ret = wait_event_timeout(pvr_dev->kccb.rtn_q, READ_ONCE(pvr_dev->kccb.rtn[slot_nr]) &
406+ ROGUE_FWIF_KCCB_RTN_SLOT_CMD_EXECUTED, timeout);
407+
408+ if (ret && rtn_out)
409+ *rtn_out = READ_ONCE(pvr_dev->kccb.rtn[slot_nr]);
410+
411+ return ret ? 0 : -ETIMEDOUT;
412+}
413+
414+/**
415+ * pvr_kccb_is_idle() - Returns whether the device's KCCB is idle
416+ * @pvr_dev: Device pointer
417+ *
418+ * Returns:
419+ * * %true if the KCCB is idle (contains no commands), or
420+ * * %false if the KCCB contains pending commands.
421+ */
422+bool
423+pvr_kccb_is_idle(struct pvr_device *pvr_dev)
424+{
425+ struct rogue_fwif_ccb_ctl *ctrl = pvr_dev->kccb.ccb.ctrl;
426+ bool idle;
427+
428+ mutex_lock(&pvr_dev->kccb.ccb.lock);
429+
430+ idle = (READ_ONCE(ctrl->write_offset) == READ_ONCE(ctrl->read_offset));
431+
432+ mutex_unlock(&pvr_dev->kccb.ccb.lock);
433+
434+ return idle;
435+}
436+
437+static const char *
438+pvr_kccb_fence_get_driver_name(struct dma_fence *f)
439+{
440+ return PVR_DRIVER_NAME;
441+}
442+
443+static const char *
444+pvr_kccb_fence_get_timeline_name(struct dma_fence *f)
445+{
446+ return "kccb";
447+}
448+
449+static const struct dma_fence_ops pvr_kccb_fence_ops = {
450+ .get_driver_name = pvr_kccb_fence_get_driver_name,
451+ .get_timeline_name = pvr_kccb_fence_get_timeline_name,
452+};
453+
454+/**
455+ * struct pvr_kccb_fence - Fence object used to wait for a KCCB slot
456+ */
457+struct pvr_kccb_fence {
458+ /** @base: Base dma_fence object. */
459+ struct dma_fence base;
460+
461+ /** @node: Node used to insert the fence in the pvr_device::kccb::waiters list. */
462+ struct list_head node;
463+};
464+
465+/**
466+ * pvr_kccb_wake_up_waiters() - Check the KCCB waiters
467+ * @pvr_dev: Target PowerVR device
468+ *
469+ * Signal as many KCCB fences as we have slots available.
470+ */
471+void pvr_kccb_wake_up_waiters(struct pvr_device *pvr_dev)
472+{
473+ struct pvr_kccb_fence *fence, *tmp_fence;
474+ u32 used_count, available_count;
475+
476+ /* Wake up those waiting for KCCB slot execution. */
477+ wake_up_all(&pvr_dev->kccb.rtn_q);
478+
479+ /* Then iterate over all KCCB fences and signal as many as we can. */
480+ mutex_lock(&pvr_dev->kccb.ccb.lock);
481+ used_count = pvr_kccb_used_slot_count_locked(pvr_dev);
482+
483+ if (WARN_ON(used_count + pvr_dev->kccb.reserved_count > pvr_kccb_capacity(pvr_dev)))
484+ goto out_unlock;
485+
486+ available_count = pvr_kccb_capacity(pvr_dev) - used_count - pvr_dev->kccb.reserved_count;
487+ list_for_each_entry_safe(fence, tmp_fence, &pvr_dev->kccb.waiters, node) {
488+ if (!available_count)
489+ break;
490+
491+ list_del(&fence->node);
492+ pvr_dev->kccb.reserved_count++;
493+ available_count--;
494+ dma_fence_signal(&fence->base);
495+ dma_fence_put(&fence->base);
496+ }
497+
498+out_unlock:
499+ mutex_unlock(&pvr_dev->kccb.ccb.lock);
500+}
501+
502+/**
503+ * pvr_kccb_fini() - Cleanup device KCCB
504+ * @pvr_dev: Target PowerVR device
505+ */
506+void pvr_kccb_fini(struct pvr_device *pvr_dev)
507+{
508+ pvr_ccb_fini(&pvr_dev->kccb.ccb);
509+ WARN_ON(!list_empty(&pvr_dev->kccb.waiters));
510+ WARN_ON(pvr_dev->kccb.reserved_count);
511+}
512+
513+/**
514+ * pvr_kccb_init() - Initialise device KCCB
515+ * @pvr_dev: Target PowerVR device
516+ *
517+ * Returns:
518+ * * 0 on success, or
519+ * * Any error returned by pvr_ccb_init().
520+ */
521+int
522+pvr_kccb_init(struct pvr_device *pvr_dev)
523+{
524+ pvr_dev->kccb.slot_count = 1 << ROGUE_FWIF_KCCB_NUMCMDS_LOG2_DEFAULT;
525+ INIT_LIST_HEAD(&pvr_dev->kccb.waiters);
526+ pvr_dev->kccb.fence_ctx.id = dma_fence_context_alloc(1);
527+ spin_lock_init(&pvr_dev->kccb.fence_ctx.lock);
528+
529+ return pvr_ccb_init(pvr_dev, &pvr_dev->kccb.ccb,
530+ ROGUE_FWIF_KCCB_NUMCMDS_LOG2_DEFAULT,
531+ sizeof(struct rogue_fwif_kccb_cmd));
532+}
533+
534+/**
535+ * pvr_kccb_fence_alloc() - Allocate a pvr_kccb_fence object
536+ *
537+ * Return:
538+ * * NULL if the allocation fails, or
539+ * * A valid dma_fence pointer otherwise.
540+ */
541+struct dma_fence *pvr_kccb_fence_alloc(void)
542+{
543+ struct pvr_kccb_fence *kccb_fence;
544+
545+ kccb_fence = kzalloc(sizeof(*kccb_fence), GFP_KERNEL);
546+ if (!kccb_fence)
547+ return NULL;
548+
549+ return &kccb_fence->base;
550+}
551+
552+/**
553+ * pvr_kccb_fence_put() - Drop a KCCB fence reference
554+ * @fence: The fence to drop the reference on.
555+ *
556+ * If the fence hasn't been initialized yet, dma_fence_free() is called. This
557+ * way we have a single function taking care of both cases.
558+ */
559+void pvr_kccb_fence_put(struct dma_fence *fence)
560+{
561+ if (!fence)
562+ return;
563+
564+ if (!fence->ops) {
565+ dma_fence_free(fence);
566+ } else {
567+ WARN_ON(fence->ops != &pvr_kccb_fence_ops);
568+ dma_fence_put(fence);
569+ }
570+}
571+
572+/**
573+ * pvr_kccb_reserve_slot() - Reserve a KCCB slot for later use
574+ * @pvr_dev: Target PowerVR device
575+ * @f: KCCB fence object previously allocated with pvr_kccb_fence_alloc()
576+ *
577+ * Try to reserve a KCCB slot, and if there's no slot available,
578+ * initializes the fence object and queue it to the waiters list.
579+ *
580+ * If NULL is returned, that means the slot is reserved. In that case,
581+ * the @f is freed and shouldn't be accessed after that point.
582+ *
583+ * Return:
584+ * * NULL if a slot was available directly, or
585+ * * A valid dma_fence object to wait on if no slot was available.
586+ */
587+struct dma_fence *
588+pvr_kccb_reserve_slot(struct pvr_device *pvr_dev, struct dma_fence *f)
589+{
590+ struct pvr_kccb_fence *fence = container_of(f, struct pvr_kccb_fence, base);
591+ struct dma_fence *out_fence = NULL;
592+ u32 used_count;
593+
594+ mutex_lock(&pvr_dev->kccb.ccb.lock);
595+
596+ used_count = pvr_kccb_used_slot_count_locked(pvr_dev);
597+ if (pvr_dev->kccb.reserved_count >= pvr_kccb_capacity(pvr_dev) - used_count) {
598+ dma_fence_init(&fence->base, &pvr_kccb_fence_ops,
599+ &pvr_dev->kccb.fence_ctx.lock,
600+ pvr_dev->kccb.fence_ctx.id,
601+ atomic_inc_return(&pvr_dev->kccb.fence_ctx.seqno));
602+ out_fence = dma_fence_get(&fence->base);
603+ list_add_tail(&fence->node, &pvr_dev->kccb.waiters);
604+ } else {
605+ pvr_kccb_fence_put(f);
606+ pvr_dev->kccb.reserved_count++;
607+ }
608+
609+ mutex_unlock(&pvr_dev->kccb.ccb.lock);
610+
611+ return out_fence;
612+}
613+
614+/**
615+ * pvr_kccb_release_slot() - Release a KCCB slot reserved with
616+ * pvr_kccb_reserve_slot()
617+ * @pvr_dev: Target PowerVR device
618+ *
619+ * Should only be called if something failed after the
620+ * pvr_kccb_reserve_slot() call and you know you won't call
621+ * pvr_kccb_send_cmd_reserved().
622+ */
623+void pvr_kccb_release_slot(struct pvr_device *pvr_dev)
624+{
625+ mutex_lock(&pvr_dev->kccb.ccb.lock);
626+ if (!WARN_ON(!pvr_dev->kccb.reserved_count))
627+ pvr_dev->kccb.reserved_count--;
628+ mutex_unlock(&pvr_dev->kccb.ccb.lock);
629+}
630+
631+/**
632+ * pvr_fwccb_init() - Initialise device FWCCB
633+ * @pvr_dev: Target PowerVR device
634+ *
635+ * Returns:
636+ * * 0 on success, or
637+ * * Any error returned by pvr_ccb_init().
638+ */
639+int
640+pvr_fwccb_init(struct pvr_device *pvr_dev)
641+{
642+ return pvr_ccb_init(pvr_dev, &pvr_dev->fwccb,
643+ ROGUE_FWIF_FWCCB_NUMCMDS_LOG2,
644+ sizeof(struct rogue_fwif_fwccb_cmd));
645+}
drivers/gpu/drm/imagination/pvr_ccb.h+71−0
@@ -0,0 +1,71 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_CCB_H
5+#define PVR_CCB_H
6+
7+#include "pvr_rogue_fwif.h"
8+
9+#include <linux/mutex.h>
10+#include <linux/types.h>
11+
12+/* Forward declaration from pvr_device.h. */
13+struct pvr_device;
14+
15+/* Forward declaration from pvr_gem.h. */
16+struct pvr_fw_object;
17+
18+struct pvr_ccb {
19+ /** @ctrl_obj: FW object representing CCB control structure. */
20+ struct pvr_fw_object *ctrl_obj;
21+ /** @ccb_obj: FW object representing CCB. */
22+ struct pvr_fw_object *ccb_obj;
23+
24+ /** @ctrl_fw_addr: FW virtual address of CCB control structure. */
25+ u32 ctrl_fw_addr;
26+ /** @ccb_fw_addr: FW virtual address of CCB. */
27+ u32 ccb_fw_addr;
28+
29+ /** @num_cmds: Number of commands in this CCB. */
30+ u32 num_cmds;
31+
32+ /** @cmd_size: Size of each command in this CCB, in bytes. */
33+ u32 cmd_size;
34+
35+ /** @lock: Mutex protecting @ctrl and @ccb. */
36+ struct mutex lock;
37+ /**
38+ * @ctrl: Kernel mapping of CCB control structure. @lock must be held
39+ * when accessing.
40+ */
41+ struct rogue_fwif_ccb_ctl *ctrl;
42+ /** @ccb: Kernel mapping of CCB. @lock must be held when accessing. */
43+ void *ccb;
44+};
45+
46+int pvr_kccb_init(struct pvr_device *pvr_dev);
47+void pvr_kccb_fini(struct pvr_device *pvr_dev);
48+int pvr_fwccb_init(struct pvr_device *pvr_dev);
49+void pvr_ccb_fini(struct pvr_ccb *ccb);
50+
51+void pvr_fwccb_process(struct pvr_device *pvr_dev);
52+
53+struct dma_fence *pvr_kccb_fence_alloc(void);
54+void pvr_kccb_fence_put(struct dma_fence *fence);
55+struct dma_fence *
56+pvr_kccb_reserve_slot(struct pvr_device *pvr_dev, struct dma_fence *f);
57+void pvr_kccb_release_slot(struct pvr_device *pvr_dev);
58+int pvr_kccb_send_cmd(struct pvr_device *pvr_dev,
59+ struct rogue_fwif_kccb_cmd *cmd, u32 *kccb_slot);
60+int pvr_kccb_send_cmd_powered(struct pvr_device *pvr_dev,
61+ struct rogue_fwif_kccb_cmd *cmd,
62+ u32 *kccb_slot);
63+void pvr_kccb_send_cmd_reserved_powered(struct pvr_device *pvr_dev,
64+ struct rogue_fwif_kccb_cmd *cmd,
65+ u32 *kccb_slot);
66+int pvr_kccb_wait_for_completion(struct pvr_device *pvr_dev, u32 slot_nr, u32 timeout,
67+ u32 *rtn_out);
68+bool pvr_kccb_is_idle(struct pvr_device *pvr_dev);
69+void pvr_kccb_wake_up_waiters(struct pvr_device *pvr_dev);
70+
71+#endif /* PVR_CCB_H */
drivers/gpu/drm/imagination/pvr_cccb.c+267−0
@@ -0,0 +1,267 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_ccb.h"
5+#include "pvr_cccb.h"
6+#include "pvr_device.h"
7+#include "pvr_gem.h"
8+#include "pvr_hwrt.h"
9+
10+#include <linux/compiler.h>
11+#include <linux/delay.h>
12+#include <linux/jiffies.h>
13+#include <linux/mutex.h>
14+#include <linux/types.h>
15+
16+static __always_inline u32
17+get_ccb_space(u32 w_off, u32 r_off, u32 ccb_size)
18+{
19+ return (((r_off) - (w_off)) + ((ccb_size) - 1)) & ((ccb_size) - 1);
20+}
21+
22+static void
23+cccb_ctrl_init(void *cpu_ptr, void *priv)
24+{
25+ struct rogue_fwif_cccb_ctl *ctrl = cpu_ptr;
26+ struct pvr_cccb *pvr_cccb = priv;
27+
28+ WRITE_ONCE(ctrl->write_offset, 0);
29+ WRITE_ONCE(ctrl->read_offset, 0);
30+ WRITE_ONCE(ctrl->dep_offset, 0);
31+ WRITE_ONCE(ctrl->wrap_mask, pvr_cccb->wrap_mask);
32+}
33+
34+/**
35+ * pvr_cccb_init() - Initialise a Client CCB
36+ * @pvr_dev: Device pointer.
37+ * @pvr_cccb: Pointer to Client CCB structure to initialise.
38+ * @size_log2: Log2 size of Client CCB in bytes.
39+ * @name: Name of owner of Client CCB. Used for fence context.
40+ *
41+ * Return:
42+ * * Zero on success, or
43+ * * Any error code returned by pvr_fw_object_create_and_map().
44+ */
45+int
46+pvr_cccb_init(struct pvr_device *pvr_dev, struct pvr_cccb *pvr_cccb,
47+ u32 size_log2, const char *name)
48+{
49+ size_t size = 1 << size_log2;
50+ int err;
51+
52+ pvr_cccb->size = size;
53+ pvr_cccb->write_offset = 0;
54+ pvr_cccb->wrap_mask = size - 1;
55+
56+ /*
57+ * Map CCCB and control structure as uncached, so we don't have to flush
58+ * CPU cache repeatedly when polling for space.
59+ */
60+ pvr_cccb->ctrl = pvr_fw_object_create_and_map(pvr_dev, sizeof(*pvr_cccb->ctrl),
61+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
62+ cccb_ctrl_init, pvr_cccb,
63+ &pvr_cccb->ctrl_obj);
64+ if (IS_ERR(pvr_cccb->ctrl))
65+ return PTR_ERR(pvr_cccb->ctrl);
66+
67+ pvr_cccb->cccb = pvr_fw_object_create_and_map(pvr_dev, size,
68+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
69+ NULL, NULL, &pvr_cccb->cccb_obj);
70+ if (IS_ERR(pvr_cccb->cccb)) {
71+ err = PTR_ERR(pvr_cccb->cccb);
72+ goto err_free_ctrl;
73+ }
74+
75+ pvr_fw_object_get_fw_addr(pvr_cccb->ctrl_obj, &pvr_cccb->ctrl_fw_addr);
76+ pvr_fw_object_get_fw_addr(pvr_cccb->cccb_obj, &pvr_cccb->cccb_fw_addr);
77+
78+ return 0;
79+
80+err_free_ctrl:
81+ pvr_fw_object_unmap_and_destroy(pvr_cccb->ctrl_obj);
82+
83+ return err;
84+}
85+
86+/**
87+ * pvr_cccb_fini() - Release Client CCB structure
88+ * @pvr_cccb: Client CCB to release.
89+ */
90+void
91+pvr_cccb_fini(struct pvr_cccb *pvr_cccb)
92+{
93+ pvr_fw_object_unmap_and_destroy(pvr_cccb->cccb_obj);
94+ pvr_fw_object_unmap_and_destroy(pvr_cccb->ctrl_obj);
95+}
96+
97+/**
98+ * pvr_cccb_cmdseq_fits() - Check if a command sequence fits in the CCCB
99+ * @pvr_cccb: Target Client CCB.
100+ * @size: Size of the command sequence.
101+ *
102+ * Check if a command sequence fits in the CCCB we have at hand.
103+ *
104+ * Return:
105+ * * true if the command sequence fits in the CCCB, or
106+ * * false otherwise.
107+ */
108+bool pvr_cccb_cmdseq_fits(struct pvr_cccb *pvr_cccb, size_t size)
109+{
110+ struct rogue_fwif_cccb_ctl *ctrl = pvr_cccb->ctrl;
111+ u32 read_offset, remaining;
112+ bool fits = false;
113+
114+ read_offset = READ_ONCE(ctrl->read_offset);
115+ remaining = pvr_cccb->size - pvr_cccb->write_offset;
116+
117+ /* Always ensure we have enough room for a padding command at the end of the CCCB.
118+ * If our command sequence does not fit, reserve the remaining space for a padding
119+ * command.
120+ */
121+ if (size + PADDING_COMMAND_SIZE > remaining)
122+ size += remaining;
123+
124+ if (get_ccb_space(pvr_cccb->write_offset, read_offset, pvr_cccb->size) >= size)
125+ fits = true;
126+
127+ return fits;
128+}
129+
130+/**
131+ * pvr_cccb_write_command_with_header() - Write a command + command header to a
132+ * Client CCB
133+ * @pvr_cccb: Target Client CCB.
134+ * @cmd_type: Client CCB command type. Must be one of %ROGUE_FWIF_CCB_CMD_TYPE_*.
135+ * @cmd_size: Size of command in bytes.
136+ * @cmd_data: Pointer to command to write.
137+ * @ext_job_ref: External job reference.
138+ * @int_job_ref: Internal job reference.
139+ *
140+ * Caller must make sure there's enough space in CCCB to queue this command. This
141+ * can be done by calling pvr_cccb_cmdseq_fits().
142+ *
143+ * This function is not protected by any lock. The caller must ensure there's
144+ * no concurrent caller, which should be guaranteed by the drm_sched model (job
145+ * submission is serialized in drm_sched_main()).
146+ */
147+void
148+pvr_cccb_write_command_with_header(struct pvr_cccb *pvr_cccb, u32 cmd_type, u32 cmd_size,
149+ void *cmd_data, u32 ext_job_ref, u32 int_job_ref)
150+{
151+ u32 sz_with_hdr = pvr_cccb_get_size_of_cmd_with_hdr(cmd_size);
152+ struct rogue_fwif_ccb_cmd_header cmd_header = {
153+ .cmd_type = cmd_type,
154+ .cmd_size = ALIGN(cmd_size, 8),
155+ .ext_job_ref = ext_job_ref,
156+ .int_job_ref = int_job_ref,
157+ };
158+ struct rogue_fwif_cccb_ctl *ctrl = pvr_cccb->ctrl;
159+ u32 remaining = pvr_cccb->size - pvr_cccb->write_offset;
160+ u32 required_size, cccb_space, read_offset;
161+
162+ /*
163+ * Always ensure we have enough room for a padding command at the end of
164+ * the CCCB.
165+ */
166+ if (remaining < sz_with_hdr + PADDING_COMMAND_SIZE) {
167+ /*
168+ * Command would need to wrap, so we need to pad the remainder
169+ * of the CCCB.
170+ */
171+ required_size = sz_with_hdr + remaining;
172+ } else {
173+ required_size = sz_with_hdr;
174+ }
175+
176+ read_offset = READ_ONCE(ctrl->read_offset);
177+ cccb_space = get_ccb_space(pvr_cccb->write_offset, read_offset, pvr_cccb->size);
178+ if (WARN_ON(cccb_space < required_size))
179+ return;
180+
181+ if (required_size != sz_with_hdr) {
182+ /* Add padding command */
183+ struct rogue_fwif_ccb_cmd_header pad_cmd = {
184+ .cmd_type = ROGUE_FWIF_CCB_CMD_TYPE_PADDING,
185+ .cmd_size = remaining - sizeof(pad_cmd),
186+ };
187+
188+ memcpy(&pvr_cccb->cccb[pvr_cccb->write_offset], &pad_cmd, sizeof(pad_cmd));
189+ pvr_cccb->write_offset = 0;
190+ }
191+
192+ memcpy(&pvr_cccb->cccb[pvr_cccb->write_offset], &cmd_header, sizeof(cmd_header));
193+ memcpy(&pvr_cccb->cccb[pvr_cccb->write_offset + sizeof(cmd_header)], cmd_data, cmd_size);
194+ pvr_cccb->write_offset += sz_with_hdr;
195+}
196+
197+static void fill_cmd_kick_data(struct pvr_cccb *cccb, u32 ctx_fw_addr,
198+ struct pvr_hwrt_data *hwrt,
199+ struct rogue_fwif_kccb_cmd_kick_data *k)
200+{
201+ k->context_fw_addr = ctx_fw_addr;
202+ k->client_woff_update = cccb->write_offset;
203+ k->client_wrap_mask_update = cccb->wrap_mask;
204+
205+ if (hwrt) {
206+ u32 cleanup_state_offset = offsetof(struct rogue_fwif_hwrtdata, cleanup_state);
207+
208+ pvr_fw_object_get_fw_addr_offset(hwrt->fw_obj, cleanup_state_offset,
209+ &k->cleanup_ctl_fw_addr[k->num_cleanup_ctl++]);
210+ }
211+}
212+
213+/**
214+ * pvr_cccb_send_kccb_kick: Send KCCB kick to trigger command processing
215+ * @pvr_dev: Device pointer.
216+ * @pvr_cccb: Pointer to CCCB to process.
217+ * @cctx_fw_addr: FW virtual address for context owning this Client CCB.
218+ * @hwrt: HWRT data set associated with this kick. May be %NULL.
219+ *
220+ * You must call pvr_kccb_reserve_slot() and wait for the returned fence to
221+ * signal (if this function didn't return NULL) before calling
222+ * pvr_cccb_send_kccb_kick().
223+ */
224+void
225+pvr_cccb_send_kccb_kick(struct pvr_device *pvr_dev,
226+ struct pvr_cccb *pvr_cccb, u32 cctx_fw_addr,
227+ struct pvr_hwrt_data *hwrt)
228+{
229+ struct rogue_fwif_kccb_cmd cmd_kick = {
230+ .cmd_type = ROGUE_FWIF_KCCB_CMD_KICK,
231+ };
232+
233+ fill_cmd_kick_data(pvr_cccb, cctx_fw_addr, hwrt, &cmd_kick.cmd_data.cmd_kick_data);
234+
235+ /* Make sure the writes to the CCCB are flushed before sending the KICK. */
236+ wmb();
237+
238+ pvr_kccb_send_cmd_reserved_powered(pvr_dev, &cmd_kick, NULL);
239+}
240+
241+void
242+pvr_cccb_send_kccb_combined_kick(struct pvr_device *pvr_dev,
243+ struct pvr_cccb *geom_cccb,
244+ struct pvr_cccb *frag_cccb,
245+ u32 geom_ctx_fw_addr,
246+ u32 frag_ctx_fw_addr,
247+ struct pvr_hwrt_data *hwrt,
248+ bool frag_is_pr)
249+{
250+ struct rogue_fwif_kccb_cmd cmd_kick = {
251+ .cmd_type = ROGUE_FWIF_KCCB_CMD_COMBINED_GEOM_FRAG_KICK,
252+ };
253+
254+ fill_cmd_kick_data(geom_cccb, geom_ctx_fw_addr, hwrt,
255+ &cmd_kick.cmd_data.combined_geom_frag_cmd_kick_data.geom_cmd_kick_data);
256+
257+ /* If this is a partial-render job, we don't attach resources to cleanup-ctl array,
258+ * because the resources are already retained by the geometry job.
259+ */
260+ fill_cmd_kick_data(frag_cccb, frag_ctx_fw_addr, frag_is_pr ? NULL : hwrt,
261+ &cmd_kick.cmd_data.combined_geom_frag_cmd_kick_data.frag_cmd_kick_data);
262+
263+ /* Make sure the writes to the CCCB are flushed before sending the KICK. */
264+ wmb();
265+
266+ pvr_kccb_send_cmd_reserved_powered(pvr_dev, &cmd_kick, NULL);
267+}
drivers/gpu/drm/imagination/pvr_cccb.h+109−0
@@ -0,0 +1,109 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_CCCB_H
5+#define PVR_CCCB_H
6+
7+#include "pvr_rogue_fwif.h"
8+#include "pvr_rogue_fwif_shared.h"
9+
10+#include <linux/mutex.h>
11+#include <linux/types.h>
12+
13+#define PADDING_COMMAND_SIZE sizeof(struct rogue_fwif_ccb_cmd_header)
14+
15+/* Forward declaration from pvr_device.h. */
16+struct pvr_device;
17+
18+/* Forward declaration from pvr_gem.h. */
19+struct pvr_fw_object;
20+
21+/* Forward declaration from pvr_hwrt.h. */
22+struct pvr_hwrt_data;
23+
24+struct pvr_cccb {
25+ /** @ctrl_obj: FW object representing CCCB control structure. */
26+ struct pvr_fw_object *ctrl_obj;
27+
28+ /** @ccb_obj: FW object representing CCCB. */
29+ struct pvr_fw_object *cccb_obj;
30+
31+ /**
32+ * @ctrl: Kernel mapping of CCCB control structure. @lock must be held
33+ * when accessing.
34+ */
35+ struct rogue_fwif_cccb_ctl *ctrl;
36+
37+ /** @cccb: Kernel mapping of CCCB. @lock must be held when accessing.*/
38+ u8 *cccb;
39+
40+ /** @ctrl_fw_addr: FW virtual address of CCCB control structure. */
41+ u32 ctrl_fw_addr;
42+ /** @ccb_fw_addr: FW virtual address of CCCB. */
43+ u32 cccb_fw_addr;
44+
45+ /** @size: Size of CCCB in bytes. */
46+ size_t size;
47+
48+ /** @write_offset: CCCB write offset. */
49+ u32 write_offset;
50+
51+ /** @wrap_mask: CCCB wrap mask. */
52+ u32 wrap_mask;
53+};
54+
55+int pvr_cccb_init(struct pvr_device *pvr_dev, struct pvr_cccb *cccb,
56+ u32 size_log2, const char *name);
57+void pvr_cccb_fini(struct pvr_cccb *cccb);
58+
59+void pvr_cccb_write_command_with_header(struct pvr_cccb *pvr_cccb,
60+ u32 cmd_type, u32 cmd_size, void *cmd_data,
61+ u32 ext_job_ref, u32 int_job_ref);
62+void pvr_cccb_send_kccb_kick(struct pvr_device *pvr_dev,
63+ struct pvr_cccb *pvr_cccb, u32 cctx_fw_addr,
64+ struct pvr_hwrt_data *hwrt);
65+void pvr_cccb_send_kccb_combined_kick(struct pvr_device *pvr_dev,
66+ struct pvr_cccb *geom_cccb,
67+ struct pvr_cccb *frag_cccb,
68+ u32 geom_ctx_fw_addr,
69+ u32 frag_ctx_fw_addr,
70+ struct pvr_hwrt_data *hwrt,
71+ bool frag_is_pr);
72+bool pvr_cccb_cmdseq_fits(struct pvr_cccb *pvr_cccb, size_t size);
73+
74+/**
75+ * pvr_cccb_get_size_of_cmd_with_hdr() - Get the size of a command and its header.
76+ * @cmd_size: Command size.
77+ *
78+ * Returns the size of the command and its header.
79+ */
80+static __always_inline u32
81+pvr_cccb_get_size_of_cmd_with_hdr(u32 cmd_size)
82+{
83+ WARN_ON(!IS_ALIGNED(cmd_size, 8));
84+ return sizeof(struct rogue_fwif_ccb_cmd_header) + ALIGN(cmd_size, 8);
85+}
86+
87+/**
88+ * pvr_cccb_cmdseq_can_fit() - Check if a command sequence can fit in the CCCB.
89+ * @size: Command sequence size.
90+ *
91+ * Returns:
92+ * * true it the CCCB is big enough to contain a command sequence, or
93+ * * false otherwise.
94+ */
95+static __always_inline bool
96+pvr_cccb_cmdseq_can_fit(struct pvr_cccb *pvr_cccb, size_t size)
97+{
98+ /* We divide the capacity by two to simplify our CCCB fencing logic:
99+ * we want to be sure that, no matter what we had queued before, we
100+ * are able to either queue our command sequence at the end or add a
101+ * padding command and queue the command sequence at the beginning
102+ * of the CCCB. If the command sequence size is bigger than half the
103+ * CCCB capacity, we'd have to queue the padding command and make sure
104+ * the FW is done processing it before queueing our command sequence.
105+ */
106+ return size + PADDING_COMMAND_SIZE <= pvr_cccb->size / 2;
107+}
108+
109+#endif /* PVR_CCCB_H */
drivers/gpu/drm/imagination/pvr_context.c+464−0
@@ -0,0 +1,464 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_cccb.h"
5+#include "pvr_context.h"
6+#include "pvr_device.h"
7+#include "pvr_drv.h"
8+#include "pvr_gem.h"
9+#include "pvr_job.h"
10+#include "pvr_power.h"
11+#include "pvr_rogue_fwif.h"
12+#include "pvr_rogue_fwif_common.h"
13+#include "pvr_rogue_fwif_resetframework.h"
14+#include "pvr_stream.h"
15+#include "pvr_stream_defs.h"
16+#include "pvr_vm.h"
17+
18+#include <drm/drm_auth.h>
19+#include <drm/drm_managed.h>
20+#include <linux/errno.h>
21+#include <linux/kernel.h>
22+#include <linux/sched.h>
23+#include <linux/slab.h>
24+#include <linux/string.h>
25+#include <linux/types.h>
26+#include <linux/xarray.h>
27+
28+static int
29+remap_priority(struct pvr_file *pvr_file, s32 uapi_priority,
30+ enum pvr_context_priority *priority_out)
31+{
32+ switch (uapi_priority) {
33+ case DRM_PVR_CTX_PRIORITY_LOW:
34+ *priority_out = PVR_CTX_PRIORITY_LOW;
35+ break;
36+ case DRM_PVR_CTX_PRIORITY_NORMAL:
37+ *priority_out = PVR_CTX_PRIORITY_MEDIUM;
38+ break;
39+ case DRM_PVR_CTX_PRIORITY_HIGH:
40+ if (!capable(CAP_SYS_NICE) && !drm_is_current_master(from_pvr_file(pvr_file)))
41+ return -EACCES;
42+ *priority_out = PVR_CTX_PRIORITY_HIGH;
43+ break;
44+ default:
45+ return -EINVAL;
46+ }
47+
48+ return 0;
49+}
50+
51+static int get_fw_obj_size(enum drm_pvr_ctx_type type)
52+{
53+ switch (type) {
54+ case DRM_PVR_CTX_TYPE_RENDER:
55+ return sizeof(struct rogue_fwif_fwrendercontext);
56+ case DRM_PVR_CTX_TYPE_COMPUTE:
57+ return sizeof(struct rogue_fwif_fwcomputecontext);
58+ case DRM_PVR_CTX_TYPE_TRANSFER_FRAG:
59+ return sizeof(struct rogue_fwif_fwtransfercontext);
60+ }
61+
62+ return -EINVAL;
63+}
64+
65+static int
66+process_static_context_state(struct pvr_device *pvr_dev, const struct pvr_stream_cmd_defs *cmd_defs,
67+ u64 stream_user_ptr, u32 stream_size, void *dest)
68+{
69+ void *stream;
70+ int err;
71+
72+ stream = kzalloc(stream_size, GFP_KERNEL);
73+ if (!stream)
74+ return -ENOMEM;
75+
76+ if (copy_from_user(stream, u64_to_user_ptr(stream_user_ptr), stream_size)) {
77+ err = -EFAULT;
78+ goto err_free;
79+ }
80+
81+ err = pvr_stream_process(pvr_dev, cmd_defs, stream, stream_size, dest);
82+ if (err)
83+ goto err_free;
84+
85+ kfree(stream);
86+
87+ return 0;
88+
89+err_free:
90+ kfree(stream);
91+
92+ return err;
93+}
94+
95+static int init_render_fw_objs(struct pvr_context *ctx,
96+ struct drm_pvr_ioctl_create_context_args *args,
97+ void *fw_ctx_map)
98+{
99+ struct rogue_fwif_static_rendercontext_state *static_rendercontext_state;
100+ struct rogue_fwif_fwrendercontext *fw_render_context = fw_ctx_map;
101+
102+ if (!args->static_context_state_len)
103+ return -EINVAL;
104+
105+ static_rendercontext_state = &fw_render_context->static_render_context_state;
106+
107+ /* Copy static render context state from userspace. */
108+ return process_static_context_state(ctx->pvr_dev,
109+ &pvr_static_render_context_state_stream,
110+ args->static_context_state,
111+ args->static_context_state_len,
112+ &static_rendercontext_state->ctxswitch_regs[0]);
113+}
114+
115+static int init_compute_fw_objs(struct pvr_context *ctx,
116+ struct drm_pvr_ioctl_create_context_args *args,
117+ void *fw_ctx_map)
118+{
119+ struct rogue_fwif_fwcomputecontext *fw_compute_context = fw_ctx_map;
120+ struct rogue_fwif_cdm_registers_cswitch *ctxswitch_regs;
121+
122+ if (!args->static_context_state_len)
123+ return -EINVAL;
124+
125+ ctxswitch_regs = &fw_compute_context->static_compute_context_state.ctxswitch_regs;
126+
127+ /* Copy static render context state from userspace. */
128+ return process_static_context_state(ctx->pvr_dev,
129+ &pvr_static_compute_context_state_stream,
130+ args->static_context_state,
131+ args->static_context_state_len,
132+ ctxswitch_regs);
133+}
134+
135+static int init_transfer_fw_objs(struct pvr_context *ctx,
136+ struct drm_pvr_ioctl_create_context_args *args,
137+ void *fw_ctx_map)
138+{
139+ if (args->static_context_state_len)
140+ return -EINVAL;
141+
142+ return 0;
143+}
144+
145+static int init_fw_objs(struct pvr_context *ctx,
146+ struct drm_pvr_ioctl_create_context_args *args,
147+ void *fw_ctx_map)
148+{
149+ switch (ctx->type) {
150+ case DRM_PVR_CTX_TYPE_RENDER:
151+ return init_render_fw_objs(ctx, args, fw_ctx_map);
152+ case DRM_PVR_CTX_TYPE_COMPUTE:
153+ return init_compute_fw_objs(ctx, args, fw_ctx_map);
154+ case DRM_PVR_CTX_TYPE_TRANSFER_FRAG:
155+ return init_transfer_fw_objs(ctx, args, fw_ctx_map);
156+ }
157+
158+ return -EINVAL;
159+}
160+
161+static void
162+ctx_fw_data_init(void *cpu_ptr, void *priv)
163+{
164+ struct pvr_context *ctx = priv;
165+
166+ memcpy(cpu_ptr, ctx->data, ctx->data_size);
167+}
168+
169+/**
170+ * pvr_context_destroy_queues() - Destroy all queues attached to a context.
171+ * @ctx: Context to destroy queues on.
172+ *
173+ * Should be called when the last reference to a context object is dropped.
174+ * It releases all resources attached to the queues bound to this context.
175+ */
176+static void pvr_context_destroy_queues(struct pvr_context *ctx)
177+{
178+ switch (ctx->type) {
179+ case DRM_PVR_CTX_TYPE_RENDER:
180+ pvr_queue_destroy(ctx->queues.fragment);
181+ pvr_queue_destroy(ctx->queues.geometry);
182+ break;
183+ case DRM_PVR_CTX_TYPE_COMPUTE:
184+ pvr_queue_destroy(ctx->queues.compute);
185+ break;
186+ case DRM_PVR_CTX_TYPE_TRANSFER_FRAG:
187+ pvr_queue_destroy(ctx->queues.transfer);
188+ break;
189+ }
190+}
191+
192+/**
193+ * pvr_context_create_queues() - Create all queues attached to a context.
194+ * @ctx: Context to create queues on.
195+ * @args: Context creation arguments passed by userspace.
196+ * @fw_ctx_map: CPU mapping of the FW context object.
197+ *
198+ * Return:
199+ * * 0 on success, or
200+ * * A negative error code otherwise.
201+ */
202+static int pvr_context_create_queues(struct pvr_context *ctx,
203+ struct drm_pvr_ioctl_create_context_args *args,
204+ void *fw_ctx_map)
205+{
206+ int err;
207+
208+ switch (ctx->type) {
209+ case DRM_PVR_CTX_TYPE_RENDER:
210+ ctx->queues.geometry = pvr_queue_create(ctx, DRM_PVR_JOB_TYPE_GEOMETRY,
211+ args, fw_ctx_map);
212+ if (IS_ERR(ctx->queues.geometry)) {
213+ err = PTR_ERR(ctx->queues.geometry);
214+ ctx->queues.geometry = NULL;
215+ goto err_destroy_queues;
216+ }
217+
218+ ctx->queues.fragment = pvr_queue_create(ctx, DRM_PVR_JOB_TYPE_FRAGMENT,
219+ args, fw_ctx_map);
220+ if (IS_ERR(ctx->queues.fragment)) {
221+ err = PTR_ERR(ctx->queues.fragment);
222+ ctx->queues.fragment = NULL;
223+ goto err_destroy_queues;
224+ }
225+ return 0;
226+
227+ case DRM_PVR_CTX_TYPE_COMPUTE:
228+ ctx->queues.compute = pvr_queue_create(ctx, DRM_PVR_JOB_TYPE_COMPUTE,
229+ args, fw_ctx_map);
230+ if (IS_ERR(ctx->queues.compute)) {
231+ err = PTR_ERR(ctx->queues.compute);
232+ ctx->queues.compute = NULL;
233+ goto err_destroy_queues;
234+ }
235+ return 0;
236+
237+ case DRM_PVR_CTX_TYPE_TRANSFER_FRAG:
238+ ctx->queues.transfer = pvr_queue_create(ctx, DRM_PVR_JOB_TYPE_TRANSFER_FRAG,
239+ args, fw_ctx_map);
240+ if (IS_ERR(ctx->queues.transfer)) {
241+ err = PTR_ERR(ctx->queues.transfer);
242+ ctx->queues.transfer = NULL;
243+ goto err_destroy_queues;
244+ }
245+ return 0;
246+ }
247+
248+ return -EINVAL;
249+
250+err_destroy_queues:
251+ pvr_context_destroy_queues(ctx);
252+ return err;
253+}
254+
255+/**
256+ * pvr_context_kill_queues() - Kill queues attached to context.
257+ * @ctx: Context to kill queues on.
258+ *
259+ * Killing the queues implies making them unusable for future jobs, while still
260+ * letting the currently submitted jobs a chance to finish. Queue resources will
261+ * stay around until pvr_context_destroy_queues() is called.
262+ */
263+static void pvr_context_kill_queues(struct pvr_context *ctx)
264+{
265+ switch (ctx->type) {
266+ case DRM_PVR_CTX_TYPE_RENDER:
267+ pvr_queue_kill(ctx->queues.fragment);
268+ pvr_queue_kill(ctx->queues.geometry);
269+ break;
270+ case DRM_PVR_CTX_TYPE_COMPUTE:
271+ pvr_queue_kill(ctx->queues.compute);
272+ break;
273+ case DRM_PVR_CTX_TYPE_TRANSFER_FRAG:
274+ pvr_queue_kill(ctx->queues.transfer);
275+ break;
276+ }
277+}
278+
279+/**
280+ * pvr_context_create() - Create a context.
281+ * @pvr_file: File to attach the created context to.
282+ * @args: Context creation arguments.
283+ *
284+ * Return:
285+ * * 0 on success, or
286+ * * A negative error code on failure.
287+ */
288+int pvr_context_create(struct pvr_file *pvr_file, struct drm_pvr_ioctl_create_context_args *args)
289+{
290+ struct pvr_device *pvr_dev = pvr_file->pvr_dev;
291+ struct pvr_context *ctx;
292+ int ctx_size;
293+ int err;
294+
295+ /* Context creation flags are currently unused and must be zero. */
296+ if (args->flags)
297+ return -EINVAL;
298+
299+ ctx_size = get_fw_obj_size(args->type);
300+ if (ctx_size < 0)
301+ return ctx_size;
302+
303+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
304+ if (!ctx)
305+ return -ENOMEM;
306+
307+ ctx->data_size = ctx_size;
308+ ctx->type = args->type;
309+ ctx->flags = args->flags;
310+ ctx->pvr_dev = pvr_dev;
311+ kref_init(&ctx->ref_count);
312+
313+ err = remap_priority(pvr_file, args->priority, &ctx->priority);
314+ if (err)
315+ goto err_free_ctx;
316+
317+ ctx->vm_ctx = pvr_vm_context_lookup(pvr_file, args->vm_context_handle);
318+ if (IS_ERR(ctx->vm_ctx)) {
319+ err = PTR_ERR(ctx->vm_ctx);
320+ goto err_free_ctx;
321+ }
322+
323+ ctx->data = kzalloc(ctx_size, GFP_KERNEL);
324+ if (!ctx->data) {
325+ err = -ENOMEM;
326+ goto err_put_vm;
327+ }
328+
329+ err = pvr_context_create_queues(ctx, args, ctx->data);
330+ if (err)
331+ goto err_free_ctx_data;
332+
333+ err = init_fw_objs(ctx, args, ctx->data);
334+ if (err)
335+ goto err_destroy_queues;
336+
337+ err = pvr_fw_object_create(pvr_dev, ctx_size, PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
338+ ctx_fw_data_init, ctx, &ctx->fw_obj);
339+ if (err)
340+ goto err_free_ctx_data;
341+
342+ err = xa_alloc(&pvr_dev->ctx_ids, &ctx->ctx_id, ctx, xa_limit_32b, GFP_KERNEL);
343+ if (err)
344+ goto err_destroy_fw_obj;
345+
346+ err = xa_alloc(&pvr_file->ctx_handles, &args->handle, ctx, xa_limit_32b, GFP_KERNEL);
347+ if (err) {
348+ /*
349+ * It's possible that another thread could have taken a reference on the context at
350+ * this point as it is in the ctx_ids xarray. Therefore instead of directly
351+ * destroying the context, drop a reference instead.
352+ */
353+ pvr_context_put(ctx);
354+ return err;
355+ }
356+
357+ return 0;
358+
359+err_destroy_fw_obj:
360+ pvr_fw_object_destroy(ctx->fw_obj);
361+
362+err_destroy_queues:
363+ pvr_context_destroy_queues(ctx);
364+
365+err_free_ctx_data:
366+ kfree(ctx->data);
367+
368+err_put_vm:
369+ pvr_vm_context_put(ctx->vm_ctx);
370+
371+err_free_ctx:
372+ kfree(ctx);
373+ return err;
374+}
375+
376+static void
377+pvr_context_release(struct kref *ref_count)
378+{
379+ struct pvr_context *ctx =
380+ container_of(ref_count, struct pvr_context, ref_count);
381+ struct pvr_device *pvr_dev = ctx->pvr_dev;
382+
383+ xa_erase(&pvr_dev->ctx_ids, ctx->ctx_id);
384+ pvr_context_destroy_queues(ctx);
385+ pvr_fw_object_destroy(ctx->fw_obj);
386+ kfree(ctx->data);
387+ pvr_vm_context_put(ctx->vm_ctx);
388+ kfree(ctx);
389+}
390+
391+/**
392+ * pvr_context_put() - Release reference on context
393+ * @ctx: Target context.
394+ */
395+void
396+pvr_context_put(struct pvr_context *ctx)
397+{
398+ if (ctx)
399+ kref_put(&ctx->ref_count, pvr_context_release);
400+}
401+
402+/**
403+ * pvr_context_destroy() - Destroy context
404+ * @pvr_file: Pointer to pvr_file structure.
405+ * @handle: Userspace context handle.
406+ *
407+ * Removes context from context list and drops initial reference. Context will
408+ * then be destroyed once all outstanding references are dropped.
409+ *
410+ * Return:
411+ * * 0 on success, or
412+ * * -%EINVAL if context not in context list.
413+ */
414+int
415+pvr_context_destroy(struct pvr_file *pvr_file, u32 handle)
416+{
417+ struct pvr_context *ctx = xa_erase(&pvr_file->ctx_handles, handle);
418+
419+ if (!ctx)
420+ return -EINVAL;
421+
422+ /* Make sure nothing can be queued to the queues after that point. */
423+ pvr_context_kill_queues(ctx);
424+
425+ /* Release the reference held by the handle set. */
426+ pvr_context_put(ctx);
427+
428+ return 0;
429+}
430+
431+/**
432+ * pvr_destroy_contexts_for_file: Destroy any contexts associated with the given file
433+ * @pvr_file: Pointer to pvr_file structure.
434+ *
435+ * Removes all contexts associated with @pvr_file from the device context list and drops initial
436+ * references. Contexts will then be destroyed once all outstanding references are dropped.
437+ */
438+void pvr_destroy_contexts_for_file(struct pvr_file *pvr_file)
439+{
440+ struct pvr_context *ctx;
441+ unsigned long handle;
442+
443+ xa_for_each(&pvr_file->ctx_handles, handle, ctx)
444+ pvr_context_destroy(pvr_file, handle);
445+}
446+
447+/**
448+ * pvr_context_device_init() - Device level initialization for queue related resources.
449+ * @pvr_dev: The device to initialize.
450+ */
451+void pvr_context_device_init(struct pvr_device *pvr_dev)
452+{
453+ xa_init_flags(&pvr_dev->ctx_ids, XA_FLAGS_ALLOC1);
454+}
455+
456+/**
457+ * pvr_context_device_fini() - Device level cleanup for queue related resources.
458+ * @pvr_dev: The device to cleanup.
459+ */
460+void pvr_context_device_fini(struct pvr_device *pvr_dev)
461+{
462+ WARN_ON(!xa_empty(&pvr_dev->ctx_ids));
463+ xa_destroy(&pvr_dev->ctx_ids);
464+}
drivers/gpu/drm/imagination/pvr_context.h+205−0
@@ -0,0 +1,205 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_CONTEXT_H
5+#define PVR_CONTEXT_H
6+
7+#include <drm/gpu_scheduler.h>
8+
9+#include <linux/compiler_attributes.h>
10+#include <linux/dma-fence.h>
11+#include <linux/kref.h>
12+#include <linux/types.h>
13+#include <linux/xarray.h>
14+#include <uapi/drm/pvr_drm.h>
15+
16+#include "pvr_cccb.h"
17+#include "pvr_device.h"
18+#include "pvr_queue.h"
19+
20+/* Forward declaration from pvr_gem.h. */
21+struct pvr_fw_object;
22+
23+enum pvr_context_priority {
24+ PVR_CTX_PRIORITY_LOW = 0,
25+ PVR_CTX_PRIORITY_MEDIUM,
26+ PVR_CTX_PRIORITY_HIGH,
27+};
28+
29+/**
30+ * struct pvr_context - Context data
31+ */
32+struct pvr_context {
33+ /** @ref_count: Refcount for context. */
34+ struct kref ref_count;
35+
36+ /** @pvr_dev: Pointer to owning device. */
37+ struct pvr_device *pvr_dev;
38+
39+ /** @vm_ctx: Pointer to associated VM context. */
40+ struct pvr_vm_context *vm_ctx;
41+
42+ /** @type: Type of context. */
43+ enum drm_pvr_ctx_type type;
44+
45+ /** @flags: Context flags. */
46+ u32 flags;
47+
48+ /** @priority: Context priority*/
49+ enum pvr_context_priority priority;
50+
51+ /** @fw_obj: FW object representing FW-side context data. */
52+ struct pvr_fw_object *fw_obj;
53+
54+ /** @data: Pointer to local copy of FW context data. */
55+ void *data;
56+
57+ /** @data_size: Size of FW context data, in bytes. */
58+ u32 data_size;
59+
60+ /** @ctx_id: FW context ID. */
61+ u32 ctx_id;
62+
63+ /**
64+ * @faulty: Set to 1 when the context queues had unfinished job when
65+ * a GPU reset happened.
66+ *
67+ * In that case, the context is in an inconsistent state and can't be
68+ * used anymore.
69+ */
70+ atomic_t faulty;
71+
72+ /** @queues: Union containing all kind of queues. */
73+ union {
74+ struct {
75+ /** @geometry: Geometry queue. */
76+ struct pvr_queue *geometry;
77+
78+ /** @fragment: Fragment queue. */
79+ struct pvr_queue *fragment;
80+ };
81+
82+ /** @compute: Compute queue. */
83+ struct pvr_queue *compute;
84+
85+ /** @compute: Transfer queue. */
86+ struct pvr_queue *transfer;
87+ } queues;
88+};
89+
90+static __always_inline struct pvr_queue *
91+pvr_context_get_queue_for_job(struct pvr_context *ctx, enum drm_pvr_job_type type)
92+{
93+ switch (type) {
94+ case DRM_PVR_JOB_TYPE_GEOMETRY:
95+ return ctx->type == DRM_PVR_CTX_TYPE_RENDER ? ctx->queues.geometry : NULL;
96+ case DRM_PVR_JOB_TYPE_FRAGMENT:
97+ return ctx->type == DRM_PVR_CTX_TYPE_RENDER ? ctx->queues.fragment : NULL;
98+ case DRM_PVR_JOB_TYPE_COMPUTE:
99+ return ctx->type == DRM_PVR_CTX_TYPE_COMPUTE ? ctx->queues.compute : NULL;
100+ case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
101+ return ctx->type == DRM_PVR_CTX_TYPE_TRANSFER_FRAG ? ctx->queues.transfer : NULL;
102+ }
103+
104+ return NULL;
105+}
106+
107+/**
108+ * pvr_context_get() - Take additional reference on context.
109+ * @ctx: Context pointer.
110+ *
111+ * Call pvr_context_put() to release.
112+ *
113+ * Returns:
114+ * * The requested context on success, or
115+ * * %NULL if no context pointer passed.
116+ */
117+static __always_inline struct pvr_context *
118+pvr_context_get(struct pvr_context *ctx)
119+{
120+ if (ctx)
121+ kref_get(&ctx->ref_count);
122+
123+ return ctx;
124+}
125+
126+/**
127+ * pvr_context_lookup() - Lookup context pointer from handle and file.
128+ * @pvr_file: Pointer to pvr_file structure.
129+ * @handle: Context handle.
130+ *
131+ * Takes reference on context. Call pvr_context_put() to release.
132+ *
133+ * Return:
134+ * * The requested context on success, or
135+ * * %NULL on failure (context does not exist, or does not belong to @pvr_file).
136+ */
137+static __always_inline struct pvr_context *
138+pvr_context_lookup(struct pvr_file *pvr_file, u32 handle)
139+{
140+ struct pvr_context *ctx;
141+
142+ /* Take the array lock to protect against context removal. */
143+ xa_lock(&pvr_file->ctx_handles);
144+ ctx = pvr_context_get(xa_load(&pvr_file->ctx_handles, handle));
145+ xa_unlock(&pvr_file->ctx_handles);
146+
147+ return ctx;
148+}
149+
150+/**
151+ * pvr_context_lookup_id() - Lookup context pointer from ID.
152+ * @pvr_dev: Device pointer.
153+ * @id: FW context ID.
154+ *
155+ * Takes reference on context. Call pvr_context_put() to release.
156+ *
157+ * Return:
158+ * * The requested context on success, or
159+ * * %NULL on failure (context does not exist).
160+ */
161+static __always_inline struct pvr_context *
162+pvr_context_lookup_id(struct pvr_device *pvr_dev, u32 id)
163+{
164+ struct pvr_context *ctx;
165+
166+ /* Take the array lock to protect against context removal. */
167+ xa_lock(&pvr_dev->ctx_ids);
168+
169+ /* Contexts are removed from the ctx_ids set in the context release path,
170+ * meaning the ref_count reached zero before they get removed. We need
171+ * to make sure we're not trying to acquire a context that's being
172+ * destroyed.
173+ */
174+ ctx = xa_load(&pvr_dev->ctx_ids, id);
175+ if (!kref_get_unless_zero(&ctx->ref_count))
176+ ctx = NULL;
177+
178+ xa_unlock(&pvr_dev->ctx_ids);
179+
180+ return ctx;
181+}
182+
183+static __always_inline u32
184+pvr_context_get_fw_addr(struct pvr_context *ctx)
185+{
186+ u32 ctx_fw_addr = 0;
187+
188+ pvr_fw_object_get_fw_addr(ctx->fw_obj, &ctx_fw_addr);
189+
190+ return ctx_fw_addr;
191+}
192+
193+void pvr_context_put(struct pvr_context *ctx);
194+
195+int pvr_context_create(struct pvr_file *pvr_file, struct drm_pvr_ioctl_create_context_args *args);
196+
197+int pvr_context_destroy(struct pvr_file *pvr_file, u32 handle);
198+
199+void pvr_destroy_contexts_for_file(struct pvr_file *pvr_file);
200+
201+void pvr_context_device_init(struct pvr_device *pvr_dev);
202+
203+void pvr_context_device_fini(struct pvr_device *pvr_dev);
204+
205+#endif /* PVR_CONTEXT_H */
drivers/gpu/drm/imagination/pvr_debugfs.c+53−0
@@ -0,0 +1,53 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_debugfs.h"
5+
6+#include "pvr_device.h"
7+#include "pvr_fw_trace.h"
8+#include "pvr_params.h"
9+
10+#include <linux/dcache.h>
11+#include <linux/debugfs.h>
12+#include <linux/err.h>
13+#include <linux/kernel.h>
14+#include <linux/types.h>
15+
16+#include <drm/drm_device.h>
17+#include <drm/drm_file.h>
18+#include <drm/drm_print.h>
19+
20+static const struct pvr_debugfs_entry pvr_debugfs_entries[] = {
21+ {"pvr_params", pvr_params_debugfs_init},
22+ {"pvr_fw", pvr_fw_trace_debugfs_init},
23+};
24+
25+void
26+pvr_debugfs_init(struct drm_minor *minor)
27+{
28+ struct drm_device *drm_dev = minor->dev;
29+ struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
30+ struct dentry *root = minor->debugfs_root;
31+ size_t i;
32+
33+ for (i = 0; i < ARRAY_SIZE(pvr_debugfs_entries); ++i) {
34+ const struct pvr_debugfs_entry *entry = &pvr_debugfs_entries[i];
35+ struct dentry *dir;
36+
37+ dir = debugfs_create_dir(entry->name, root);
38+ if (IS_ERR(dir)) {
39+ drm_warn(drm_dev,
40+ "failed to create debugfs dir '%s' (err=%d)",
41+ entry->name, (int)PTR_ERR(dir));
42+ continue;
43+ }
44+
45+ entry->init(pvr_dev, dir);
46+ }
47+}
48+
49+/*
50+ * Since all entries are created under &drm_minor->debugfs_root, there's no
51+ * need for a pvr_debugfs_fini() as DRM will clean up everything under its root
52+ * automatically.
53+ */
drivers/gpu/drm/imagination/pvr_debugfs.h+29−0
@@ -0,0 +1,29 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_DEBUGFS_H
5+#define PVR_DEBUGFS_H
6+
7+/* Forward declaration from <drm/drm_drv.h>. */
8+struct drm_minor;
9+
10+#if defined(CONFIG_DEBUG_FS)
11+/* Forward declaration from "pvr_device.h". */
12+struct pvr_device;
13+
14+/* Forward declaration from <linux/dcache.h>. */
15+struct dentry;
16+
17+struct pvr_debugfs_entry {
18+ const char *name;
19+ void (*init)(struct pvr_device *pvr_dev, struct dentry *dir);
20+};
21+
22+void pvr_debugfs_init(struct drm_minor *minor);
23+#else /* defined(CONFIG_DEBUG_FS) */
24+#include <linux/compiler_attributes.h>
25+
26+static __always_inline void pvr_debugfs_init(struct drm_minor *minor) {}
27+#endif /* defined(CONFIG_DEBUG_FS) */
28+
29+#endif /* PVR_DEBUGFS_H */
drivers/gpu/drm/imagination/pvr_device.c+658−0
@@ -0,0 +1,658 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_device.h"
5+#include "pvr_device_info.h"
6+
7+#include "pvr_fw.h"
8+#include "pvr_params.h"
9+#include "pvr_power.h"
10+#include "pvr_queue.h"
11+#include "pvr_rogue_cr_defs.h"
12+#include "pvr_stream.h"
13+#include "pvr_vm.h"
14+
15+#include <drm/drm_print.h>
16+
17+#include <linux/bitfield.h>
18+#include <linux/clk.h>
19+#include <linux/compiler_attributes.h>
20+#include <linux/compiler_types.h>
21+#include <linux/dma-mapping.h>
22+#include <linux/err.h>
23+#include <linux/firmware.h>
24+#include <linux/gfp.h>
25+#include <linux/interrupt.h>
26+#include <linux/platform_device.h>
27+#include <linux/pm_runtime.h>
28+#include <linux/slab.h>
29+#include <linux/stddef.h>
30+#include <linux/types.h>
31+#include <linux/workqueue.h>
32+
33+/* Major number for the supported version of the firmware. */
34+#define PVR_FW_VERSION_MAJOR 1
35+
36+/**
37+ * pvr_device_reg_init() - Initialize kernel access to a PowerVR device's
38+ * control registers.
39+ * @pvr_dev: Target PowerVR device.
40+ *
41+ * Sets struct pvr_device->regs.
42+ *
43+ * This method of mapping the device control registers into memory ensures that
44+ * they are unmapped when the driver is detached (i.e. no explicit cleanup is
45+ * required).
46+ *
47+ * Return:
48+ * * 0 on success, or
49+ * * Any error returned by devm_platform_ioremap_resource().
50+ */
51+static int
52+pvr_device_reg_init(struct pvr_device *pvr_dev)
53+{
54+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
55+ struct platform_device *plat_dev = to_platform_device(drm_dev->dev);
56+ struct resource *regs_resource;
57+ void __iomem *regs;
58+
59+ pvr_dev->regs_resource = NULL;
60+ pvr_dev->regs = NULL;
61+
62+ regs = devm_platform_get_and_ioremap_resource(plat_dev, 0, &regs_resource);
63+ if (IS_ERR(regs))
64+ return dev_err_probe(drm_dev->dev, PTR_ERR(regs),
65+ "failed to ioremap gpu registers\n");
66+
67+ pvr_dev->regs = regs;
68+ pvr_dev->regs_resource = regs_resource;
69+
70+ return 0;
71+}
72+
73+/**
74+ * pvr_device_clk_init() - Initialize clocks required by a PowerVR device
75+ * @pvr_dev: Target PowerVR device.
76+ *
77+ * Sets struct pvr_device->core_clk, struct pvr_device->sys_clk and
78+ * struct pvr_device->mem_clk.
79+ *
80+ * Three clocks are required by the PowerVR device: core, sys and mem. On
81+ * return, this function guarantees that the clocks are in one of the following
82+ * states:
83+ *
84+ * * All successfully initialized,
85+ * * Core errored, sys and mem uninitialized,
86+ * * Core deinitialized, sys errored, mem uninitialized, or
87+ * * Core and sys deinitialized, mem errored.
88+ *
89+ * Return:
90+ * * 0 on success,
91+ * * Any error returned by devm_clk_get(), or
92+ * * Any error returned by devm_clk_get_optional().
93+ */
94+static int pvr_device_clk_init(struct pvr_device *pvr_dev)
95+{
96+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
97+ struct clk *core_clk;
98+ struct clk *sys_clk;
99+ struct clk *mem_clk;
100+
101+ core_clk = devm_clk_get(drm_dev->dev, "core");
102+ if (IS_ERR(core_clk))
103+ return dev_err_probe(drm_dev->dev, PTR_ERR(core_clk),
104+ "failed to get core clock\n");
105+
106+ sys_clk = devm_clk_get_optional(drm_dev->dev, "sys");
107+ if (IS_ERR(sys_clk))
108+ return dev_err_probe(drm_dev->dev, PTR_ERR(core_clk),
109+ "failed to get sys clock\n");
110+
111+ mem_clk = devm_clk_get_optional(drm_dev->dev, "mem");
112+ if (IS_ERR(mem_clk))
113+ return dev_err_probe(drm_dev->dev, PTR_ERR(core_clk),
114+ "failed to get mem clock\n");
115+
116+ pvr_dev->core_clk = core_clk;
117+ pvr_dev->sys_clk = sys_clk;
118+ pvr_dev->mem_clk = mem_clk;
119+
120+ return 0;
121+}
122+
123+/**
124+ * pvr_device_process_active_queues() - Process all queue related events.
125+ * @pvr_dev: PowerVR device to check
126+ *
127+ * This is called any time we receive a FW event. It iterates over all
128+ * active queues and calls pvr_queue_process() on them.
129+ */
130+void pvr_device_process_active_queues(struct pvr_device *pvr_dev)
131+{
132+ struct pvr_queue *queue, *tmp_queue;
133+ LIST_HEAD(active_queues);
134+
135+ mutex_lock(&pvr_dev->queues.lock);
136+
137+ /* Move all active queues to a temporary list. Queues that remain
138+ * active after we're done processing them are re-inserted to
139+ * the queues.active list by pvr_queue_process().
140+ */
141+ list_splice_init(&pvr_dev->queues.active, &active_queues);
142+
143+ list_for_each_entry_safe(queue, tmp_queue, &active_queues, node)
144+ pvr_queue_process(queue);
145+
146+ mutex_unlock(&pvr_dev->queues.lock);
147+}
148+
149+static irqreturn_t pvr_device_irq_thread_handler(int irq, void *data)
150+{
151+ struct pvr_device *pvr_dev = data;
152+ irqreturn_t ret = IRQ_NONE;
153+
154+ /* We are in the threaded handler, we can keep dequeuing events until we
155+ * don't see any. This should allow us to reduce the number of interrupts
156+ * when the GPU is receiving a massive amount of short jobs.
157+ */
158+ while (pvr_fw_irq_pending(pvr_dev)) {
159+ pvr_fw_irq_clear(pvr_dev);
160+
161+ if (pvr_dev->fw_dev.booted) {
162+ pvr_fwccb_process(pvr_dev);
163+ pvr_kccb_wake_up_waiters(pvr_dev);
164+ pvr_device_process_active_queues(pvr_dev);
165+ }
166+
167+ pm_runtime_mark_last_busy(from_pvr_device(pvr_dev)->dev);
168+
169+ ret = IRQ_HANDLED;
170+ }
171+
172+ /* Unmask FW irqs before returning, so new interrupts can be received. */
173+ pvr_fw_irq_enable(pvr_dev);
174+ return ret;
175+}
176+
177+static irqreturn_t pvr_device_irq_handler(int irq, void *data)
178+{
179+ struct pvr_device *pvr_dev = data;
180+
181+ if (!pvr_fw_irq_pending(pvr_dev))
182+ return IRQ_NONE; /* Spurious IRQ - ignore. */
183+
184+ /* Mask the FW interrupts before waking up the thread. Will be unmasked
185+ * when the thread handler is done processing events.
186+ */
187+ pvr_fw_irq_disable(pvr_dev);
188+ return IRQ_WAKE_THREAD;
189+}
190+
191+/**
192+ * pvr_device_irq_init() - Initialise IRQ required by a PowerVR device
193+ * @pvr_dev: Target PowerVR device.
194+ *
195+ * Returns:
196+ * * 0 on success,
197+ * * Any error returned by platform_get_irq_byname(), or
198+ * * Any error returned by request_irq().
199+ */
200+static int
201+pvr_device_irq_init(struct pvr_device *pvr_dev)
202+{
203+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
204+ struct platform_device *plat_dev = to_platform_device(drm_dev->dev);
205+
206+ init_waitqueue_head(&pvr_dev->kccb.rtn_q);
207+
208+ pvr_dev->irq = platform_get_irq(plat_dev, 0);
209+ if (pvr_dev->irq < 0)
210+ return pvr_dev->irq;
211+
212+ /* Clear any pending events before requesting the IRQ line. */
213+ pvr_fw_irq_clear(pvr_dev);
214+ pvr_fw_irq_enable(pvr_dev);
215+
216+ return request_threaded_irq(pvr_dev->irq, pvr_device_irq_handler,
217+ pvr_device_irq_thread_handler,
218+ IRQF_SHARED, "gpu", pvr_dev);
219+}
220+
221+/**
222+ * pvr_device_irq_fini() - Deinitialise IRQ required by a PowerVR device
223+ * @pvr_dev: Target PowerVR device.
224+ */
225+static void
226+pvr_device_irq_fini(struct pvr_device *pvr_dev)
227+{
228+ free_irq(pvr_dev->irq, pvr_dev);
229+}
230+
231+/**
232+ * pvr_build_firmware_filename() - Construct a PowerVR firmware filename
233+ * @pvr_dev: Target PowerVR device.
234+ * @base: First part of the filename.
235+ * @major: Major version number.
236+ *
237+ * A PowerVR firmware filename consists of three parts separated by underscores
238+ * (``'_'``) along with a '.fw' file suffix. The first part is the exact value
239+ * of @base, the second part is the hardware version string derived from @pvr_fw
240+ * and the final part is the firmware version number constructed from @major with
241+ * a 'v' prefix, e.g. powervr/rogue_4.40.2.51_v1.fw.
242+ *
243+ * The returned string will have been slab allocated and must be freed with
244+ * kfree().
245+ *
246+ * Return:
247+ * * The constructed filename on success, or
248+ * * Any error returned by kasprintf().
249+ */
250+static char *
251+pvr_build_firmware_filename(struct pvr_device *pvr_dev, const char *base,
252+ u8 major)
253+{
254+ struct pvr_gpu_id *gpu_id = &pvr_dev->gpu_id;
255+
256+ return kasprintf(GFP_KERNEL, "%s_%d.%d.%d.%d_v%d.fw", base, gpu_id->b,
257+ gpu_id->v, gpu_id->n, gpu_id->c, major);
258+}
259+
260+static void
261+pvr_release_firmware(void *data)
262+{
263+ struct pvr_device *pvr_dev = data;
264+
265+ release_firmware(pvr_dev->fw_dev.firmware);
266+}
267+
268+/**
269+ * pvr_request_firmware() - Load firmware for a PowerVR device
270+ * @pvr_dev: Target PowerVR device.
271+ *
272+ * See pvr_build_firmware_filename() for details on firmware file naming.
273+ *
274+ * Return:
275+ * * 0 on success,
276+ * * Any error returned by pvr_build_firmware_filename(), or
277+ * * Any error returned by request_firmware().
278+ */
279+static int
280+pvr_request_firmware(struct pvr_device *pvr_dev)
281+{
282+ struct drm_device *drm_dev = &pvr_dev->base;
283+ char *filename;
284+ const struct firmware *fw;
285+ int err;
286+
287+ filename = pvr_build_firmware_filename(pvr_dev, "powervr/rogue",
288+ PVR_FW_VERSION_MAJOR);
289+ if (IS_ERR(filename))
290+ return PTR_ERR(filename);
291+
292+ /*
293+ * This function takes a copy of &filename, meaning we can free our
294+ * instance before returning.
295+ */
296+ err = request_firmware(&fw, filename, pvr_dev->base.dev);
297+ if (err) {
298+ drm_err(drm_dev, "failed to load firmware %s (err=%d)\n",
299+ filename, err);
300+ goto err_free_filename;
301+ }
302+
303+ drm_info(drm_dev, "loaded firmware %s\n", filename);
304+ kfree(filename);
305+
306+ pvr_dev->fw_dev.firmware = fw;
307+
308+ return devm_add_action_or_reset(drm_dev->dev, pvr_release_firmware, pvr_dev);
309+
310+err_free_filename:
311+ kfree(filename);
312+
313+ return err;
314+}
315+
316+/**
317+ * pvr_load_gpu_id() - Load a PowerVR device's GPU ID (BVNC) from control registers.
318+ *
319+ * Sets struct pvr_dev.gpu_id.
320+ *
321+ * @pvr_dev: Target PowerVR device.
322+ */
323+static void
324+pvr_load_gpu_id(struct pvr_device *pvr_dev)
325+{
326+ struct pvr_gpu_id *gpu_id = &pvr_dev->gpu_id;
327+ u64 bvnc;
328+
329+ /*
330+ * Try reading the BVNC using the newer (cleaner) method first. If the
331+ * B value is zero, fall back to the older method.
332+ */
333+ bvnc = pvr_cr_read64(pvr_dev, ROGUE_CR_CORE_ID__PBVNC);
334+
335+ gpu_id->b = PVR_CR_FIELD_GET(bvnc, CORE_ID__PBVNC__BRANCH_ID);
336+ if (gpu_id->b != 0) {
337+ gpu_id->v = PVR_CR_FIELD_GET(bvnc, CORE_ID__PBVNC__VERSION_ID);
338+ gpu_id->n = PVR_CR_FIELD_GET(bvnc, CORE_ID__PBVNC__NUMBER_OF_SCALABLE_UNITS);
339+ gpu_id->c = PVR_CR_FIELD_GET(bvnc, CORE_ID__PBVNC__CONFIG_ID);
340+ } else {
341+ u32 core_rev = pvr_cr_read32(pvr_dev, ROGUE_CR_CORE_REVISION);
342+ u32 core_id = pvr_cr_read32(pvr_dev, ROGUE_CR_CORE_ID);
343+ u16 core_id_config = PVR_CR_FIELD_GET(core_id, CORE_ID_CONFIG);
344+
345+ gpu_id->b = PVR_CR_FIELD_GET(core_rev, CORE_REVISION_MAJOR);
346+ gpu_id->v = PVR_CR_FIELD_GET(core_rev, CORE_REVISION_MINOR);
347+ gpu_id->n = FIELD_GET(0xFF00, core_id_config);
348+ gpu_id->c = FIELD_GET(0x00FF, core_id_config);
349+ }
350+}
351+
352+/**
353+ * pvr_set_dma_info() - Set PowerVR device DMA information
354+ * @pvr_dev: Target PowerVR device.
355+ *
356+ * Sets the DMA mask and max segment size for the PowerVR device.
357+ *
358+ * Return:
359+ * * 0 on success,
360+ * * Any error returned by PVR_FEATURE_VALUE(), or
361+ * * Any error returned by dma_set_mask().
362+ */
363+
364+static int
365+pvr_set_dma_info(struct pvr_device *pvr_dev)
366+{
367+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
368+ u16 phys_bus_width;
369+ int err;
370+
371+ err = PVR_FEATURE_VALUE(pvr_dev, phys_bus_width, &phys_bus_width);
372+ if (err) {
373+ drm_err(drm_dev, "Failed to get device physical bus width\n");
374+ return err;
375+ }
376+
377+ err = dma_set_mask(drm_dev->dev, DMA_BIT_MASK(phys_bus_width));
378+ if (err) {
379+ drm_err(drm_dev, "Failed to set DMA mask (err=%d)\n", err);
380+ return err;
381+ }
382+
383+ dma_set_max_seg_size(drm_dev->dev, UINT_MAX);
384+
385+ return 0;
386+}
387+
388+/**
389+ * pvr_device_gpu_init() - GPU-specific initialization for a PowerVR device
390+ * @pvr_dev: Target PowerVR device.
391+ *
392+ * The following steps are taken to ensure the device is ready:
393+ *
394+ * 1. Read the hardware version information from control registers,
395+ * 2. Initialise the hardware feature information,
396+ * 3. Setup the device DMA information,
397+ * 4. Setup the device-scoped memory context, and
398+ * 5. Load firmware into the device.
399+ *
400+ * Return:
401+ * * 0 on success,
402+ * * -%ENODEV if the GPU is not supported,
403+ * * Any error returned by pvr_set_dma_info(),
404+ * * Any error returned by pvr_memory_context_init(), or
405+ * * Any error returned by pvr_request_firmware().
406+ */
407+static int
408+pvr_device_gpu_init(struct pvr_device *pvr_dev)
409+{
410+ int err;
411+
412+ pvr_load_gpu_id(pvr_dev);
413+
414+ err = pvr_request_firmware(pvr_dev);
415+ if (err)
416+ return err;
417+
418+ err = pvr_fw_validate_init_device_info(pvr_dev);
419+ if (err)
420+ return err;
421+
422+ if (PVR_HAS_FEATURE(pvr_dev, meta))
423+ pvr_dev->fw_dev.processor_type = PVR_FW_PROCESSOR_TYPE_META;
424+ else if (PVR_HAS_FEATURE(pvr_dev, mips))
425+ pvr_dev->fw_dev.processor_type = PVR_FW_PROCESSOR_TYPE_MIPS;
426+ else if (PVR_HAS_FEATURE(pvr_dev, riscv_fw_processor))
427+ pvr_dev->fw_dev.processor_type = PVR_FW_PROCESSOR_TYPE_RISCV;
428+ else
429+ return -EINVAL;
430+
431+ pvr_stream_create_musthave_masks(pvr_dev);
432+
433+ err = pvr_set_dma_info(pvr_dev);
434+ if (err)
435+ return err;
436+
437+ if (pvr_dev->fw_dev.processor_type != PVR_FW_PROCESSOR_TYPE_MIPS) {
438+ pvr_dev->kernel_vm_ctx = pvr_vm_create_context(pvr_dev, false);
439+ if (IS_ERR(pvr_dev->kernel_vm_ctx))
440+ return PTR_ERR(pvr_dev->kernel_vm_ctx);
441+ }
442+
443+ err = pvr_fw_init(pvr_dev);
444+ if (err)
445+ goto err_vm_ctx_put;
446+
447+ return 0;
448+
449+err_vm_ctx_put:
450+ if (pvr_dev->fw_dev.processor_type != PVR_FW_PROCESSOR_TYPE_MIPS) {
451+ pvr_vm_context_put(pvr_dev->kernel_vm_ctx);
452+ pvr_dev->kernel_vm_ctx = NULL;
453+ }
454+
455+ return err;
456+}
457+
458+/**
459+ * pvr_device_gpu_fini() - GPU-specific deinitialization for a PowerVR device
460+ * @pvr_dev: Target PowerVR device.
461+ */
462+static void
463+pvr_device_gpu_fini(struct pvr_device *pvr_dev)
464+{
465+ pvr_fw_fini(pvr_dev);
466+
467+ if (pvr_dev->fw_dev.processor_type != PVR_FW_PROCESSOR_TYPE_MIPS) {
468+ WARN_ON(!pvr_vm_context_put(pvr_dev->kernel_vm_ctx));
469+ pvr_dev->kernel_vm_ctx = NULL;
470+ }
471+}
472+
473+/**
474+ * pvr_device_init() - Initialize a PowerVR device
475+ * @pvr_dev: Target PowerVR device.
476+ *
477+ * If this function returns successfully, the device will have been fully
478+ * initialized. Otherwise, any parts of the device initialized before an error
479+ * occurs will be de-initialized before returning.
480+ *
481+ * NOTE: The initialization steps currently taken are the bare minimum required
482+ * to read from the control registers. The device is unlikely to function
483+ * until further initialization steps are added. [This note should be
484+ * removed when that happens.]
485+ *
486+ * Return:
487+ * * 0 on success,
488+ * * Any error returned by pvr_device_reg_init(),
489+ * * Any error returned by pvr_device_clk_init(), or
490+ * * Any error returned by pvr_device_gpu_init().
491+ */
492+int
493+pvr_device_init(struct pvr_device *pvr_dev)
494+{
495+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
496+ struct device *dev = drm_dev->dev;
497+ int err;
498+
499+ /*
500+ * Setup device parameters. We do this first in case other steps
501+ * depend on them.
502+ */
503+ err = pvr_device_params_init(&pvr_dev->params);
504+ if (err)
505+ return err;
506+
507+ /* Enable and initialize clocks required for the device to operate. */
508+ err = pvr_device_clk_init(pvr_dev);
509+ if (err)
510+ return err;
511+
512+ /* Explicitly power the GPU so we can access control registers before the FW is booted. */
513+ err = pm_runtime_resume_and_get(dev);
514+ if (err)
515+ return err;
516+
517+ /* Map the control registers into memory. */
518+ err = pvr_device_reg_init(pvr_dev);
519+ if (err)
520+ goto err_pm_runtime_put;
521+
522+ /* Perform GPU-specific initialization steps. */
523+ err = pvr_device_gpu_init(pvr_dev);
524+ if (err)
525+ goto err_pm_runtime_put;
526+
527+ err = pvr_device_irq_init(pvr_dev);
528+ if (err)
529+ goto err_device_gpu_fini;
530+
531+ pm_runtime_put(dev);
532+
533+ return 0;
534+
535+err_device_gpu_fini:
536+ pvr_device_gpu_fini(pvr_dev);
537+
538+err_pm_runtime_put:
539+ pm_runtime_put_sync_suspend(dev);
540+
541+ return err;
542+}
543+
544+/**
545+ * pvr_device_fini() - Deinitialize a PowerVR device
546+ * @pvr_dev: Target PowerVR device.
547+ */
548+void
549+pvr_device_fini(struct pvr_device *pvr_dev)
550+{
551+ /*
552+ * Deinitialization stages are performed in reverse order compared to
553+ * the initialization stages in pvr_device_init().
554+ */
555+ pvr_device_irq_fini(pvr_dev);
556+ pvr_device_gpu_fini(pvr_dev);
557+}
558+
559+bool
560+pvr_device_has_uapi_quirk(struct pvr_device *pvr_dev, u32 quirk)
561+{
562+ switch (quirk) {
563+ case 47217:
564+ return PVR_HAS_QUIRK(pvr_dev, 47217);
565+ case 48545:
566+ return PVR_HAS_QUIRK(pvr_dev, 48545);
567+ case 49927:
568+ return PVR_HAS_QUIRK(pvr_dev, 49927);
569+ case 51764:
570+ return PVR_HAS_QUIRK(pvr_dev, 51764);
571+ case 62269:
572+ return PVR_HAS_QUIRK(pvr_dev, 62269);
573+ default:
574+ return false;
575+ };
576+}
577+
578+bool
579+pvr_device_has_uapi_enhancement(struct pvr_device *pvr_dev, u32 enhancement)
580+{
581+ switch (enhancement) {
582+ case 35421:
583+ return PVR_HAS_ENHANCEMENT(pvr_dev, 35421);
584+ case 42064:
585+ return PVR_HAS_ENHANCEMENT(pvr_dev, 42064);
586+ default:
587+ return false;
588+ };
589+}
590+
591+/**
592+ * pvr_device_has_feature() - Look up device feature based on feature definition
593+ * @pvr_dev: Device pointer.
594+ * @feature: Feature to look up. Should be one of %PVR_FEATURE_*.
595+ *
596+ * Returns:
597+ * * %true if feature is present on device, or
598+ * * %false if feature is not present on device.
599+ */
600+bool
601+pvr_device_has_feature(struct pvr_device *pvr_dev, u32 feature)
602+{
603+ switch (feature) {
604+ case PVR_FEATURE_CLUSTER_GROUPING:
605+ return PVR_HAS_FEATURE(pvr_dev, cluster_grouping);
606+
607+ case PVR_FEATURE_COMPUTE_MORTON_CAPABLE:
608+ return PVR_HAS_FEATURE(pvr_dev, compute_morton_capable);
609+
610+ case PVR_FEATURE_FB_CDC_V4:
611+ return PVR_HAS_FEATURE(pvr_dev, fb_cdc_v4);
612+
613+ case PVR_FEATURE_GPU_MULTICORE_SUPPORT:
614+ return PVR_HAS_FEATURE(pvr_dev, gpu_multicore_support);
615+
616+ case PVR_FEATURE_ISP_ZLS_D24_S8_PACKING_OGL_MODE:
617+ return PVR_HAS_FEATURE(pvr_dev, isp_zls_d24_s8_packing_ogl_mode);
618+
619+ case PVR_FEATURE_S7_TOP_INFRASTRUCTURE:
620+ return PVR_HAS_FEATURE(pvr_dev, s7_top_infrastructure);
621+
622+ case PVR_FEATURE_TESSELLATION:
623+ return PVR_HAS_FEATURE(pvr_dev, tessellation);
624+
625+ case PVR_FEATURE_TPU_DM_GLOBAL_REGISTERS:
626+ return PVR_HAS_FEATURE(pvr_dev, tpu_dm_global_registers);
627+
628+ case PVR_FEATURE_VDM_DRAWINDIRECT:
629+ return PVR_HAS_FEATURE(pvr_dev, vdm_drawindirect);
630+
631+ case PVR_FEATURE_VDM_OBJECT_LEVEL_LLS:
632+ return PVR_HAS_FEATURE(pvr_dev, vdm_object_level_lls);
633+
634+ case PVR_FEATURE_ZLS_SUBTILE:
635+ return PVR_HAS_FEATURE(pvr_dev, zls_subtile);
636+
637+ /* Derived features. */
638+ case PVR_FEATURE_CDM_USER_MODE_QUEUE: {
639+ u8 cdm_control_stream_format = 0;
640+
641+ PVR_FEATURE_VALUE(pvr_dev, cdm_control_stream_format, &cdm_control_stream_format);
642+ return (cdm_control_stream_format >= 2 && cdm_control_stream_format <= 4);
643+ }
644+
645+ case PVR_FEATURE_REQUIRES_FB_CDC_ZLS_SETUP:
646+ if (PVR_HAS_FEATURE(pvr_dev, fbcdc_algorithm)) {
647+ u8 fbcdc_algorithm = 0;
648+
649+ PVR_FEATURE_VALUE(pvr_dev, fbcdc_algorithm, &fbcdc_algorithm);
650+ return (fbcdc_algorithm < 3 || PVR_HAS_FEATURE(pvr_dev, fb_cdc_v4));
651+ }
652+ return false;
653+
654+ default:
655+ WARN(true, "Looking up undefined feature %u\n", feature);
656+ return false;
657+ }
658+}
drivers/gpu/drm/imagination/pvr_device.h+710−0
@@ -0,0 +1,710 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_DEVICE_H
5+#define PVR_DEVICE_H
6+
7+#include "pvr_ccb.h"
8+#include "pvr_device_info.h"
9+#include "pvr_fw.h"
10+#include "pvr_params.h"
11+#include "pvr_rogue_fwif_stream.h"
12+#include "pvr_stream.h"
13+
14+#include <drm/drm_device.h>
15+#include <drm/drm_file.h>
16+#include <drm/drm_mm.h>
17+
18+#include <linux/bits.h>
19+#include <linux/compiler_attributes.h>
20+#include <linux/compiler_types.h>
21+#include <linux/io.h>
22+#include <linux/iopoll.h>
23+#include <linux/kernel.h>
24+#include <linux/math.h>
25+#include <linux/mutex.h>
26+#include <linux/timer.h>
27+#include <linux/types.h>
28+#include <linux/wait.h>
29+#include <linux/workqueue.h>
30+#include <linux/xarray.h>
31+
32+/* Forward declaration from <linux/clk.h>. */
33+struct clk;
34+
35+/* Forward declaration from <linux/firmware.h>. */
36+struct firmware;
37+
38+/**
39+ * struct pvr_gpu_id - Hardware GPU ID information for a PowerVR device
40+ * @b: Branch ID.
41+ * @v: Version ID.
42+ * @n: Number of scalable units.
43+ * @c: Config ID.
44+ */
45+struct pvr_gpu_id {
46+ u16 b, v, n, c;
47+};
48+
49+/**
50+ * struct pvr_fw_version - Firmware version information
51+ * @major: Major version number.
52+ * @minor: Minor version number.
53+ */
54+struct pvr_fw_version {
55+ u16 major, minor;
56+};
57+
58+/**
59+ * struct pvr_device - powervr-specific wrapper for &struct drm_device
60+ */
61+struct pvr_device {
62+ /**
63+ * @base: The underlying &struct drm_device.
64+ *
65+ * Do not access this member directly, instead call
66+ * from_pvr_device().
67+ */
68+ struct drm_device base;
69+
70+ /** @gpu_id: GPU ID detected at runtime. */
71+ struct pvr_gpu_id gpu_id;
72+
73+ /**
74+ * @features: Hardware feature information.
75+ *
76+ * Do not access this member directly, instead use PVR_HAS_FEATURE()
77+ * or PVR_FEATURE_VALUE() macros.
78+ */
79+ struct pvr_device_features features;
80+
81+ /**
82+ * @quirks: Hardware quirk information.
83+ *
84+ * Do not access this member directly, instead use PVR_HAS_QUIRK().
85+ */
86+ struct pvr_device_quirks quirks;
87+
88+ /**
89+ * @enhancements: Hardware enhancement information.
90+ *
91+ * Do not access this member directly, instead use
92+ * PVR_HAS_ENHANCEMENT().
93+ */
94+ struct pvr_device_enhancements enhancements;
95+
96+ /** @fw_version: Firmware version detected at runtime. */
97+ struct pvr_fw_version fw_version;
98+
99+ /** @regs_resource: Resource representing device control registers. */
100+ struct resource *regs_resource;
101+
102+ /**
103+ * @regs: Device control registers.
104+ *
105+ * These are mapped into memory when the device is initialized; that
106+ * location is where this pointer points.
107+ */
108+ void __iomem *regs;
109+
110+ /**
111+ * @core_clk: General core clock.
112+ *
113+ * This is the primary clock used by the entire GPU core.
114+ */
115+ struct clk *core_clk;
116+
117+ /**
118+ * @sys_clk: Optional system bus clock.
119+ *
120+ * This may be used on some platforms to provide an independent clock to the SoC Interface
121+ * (SOCIF). If present, this needs to be enabled/disabled together with @core_clk.
122+ */
123+ struct clk *sys_clk;
124+
125+ /**
126+ * @mem_clk: Optional memory clock.
127+ *
128+ * This may be used on some platforms to provide an independent clock to the Memory
129+ * Interface (MEMIF). If present, this needs to be enabled/disabled together with @core_clk.
130+ */
131+ struct clk *mem_clk;
132+
133+ /** @irq: IRQ number. */
134+ int irq;
135+
136+ /** @fwccb: Firmware CCB. */
137+ struct pvr_ccb fwccb;
138+
139+ /**
140+ * @kernel_vm_ctx: Virtual memory context used for kernel mappings.
141+ *
142+ * This is used for mappings in the firmware address region when a META firmware processor
143+ * is in use.
144+ *
145+ * When a MIPS firmware processor is in use, this will be %NULL.
146+ */
147+ struct pvr_vm_context *kernel_vm_ctx;
148+
149+ /** @fw_dev: Firmware related data. */
150+ struct pvr_fw_device fw_dev;
151+
152+ /**
153+ * @params: Device-specific parameters.
154+ *
155+ * The values of these parameters are initialized from the
156+ * defaults specified as module parameters. They may be
157+ * modified at runtime via debugfs (if enabled).
158+ */
159+ struct pvr_device_params params;
160+
161+ /** @stream_musthave_quirks: Bit array of "must-have" quirks for stream commands. */
162+ u32 stream_musthave_quirks[PVR_STREAM_TYPE_MAX][PVR_STREAM_EXTHDR_TYPE_MAX];
163+
164+ /**
165+ * @mmu_flush_cache_flags: Records which MMU caches require flushing
166+ * before submitting the next job.
167+ */
168+ atomic_t mmu_flush_cache_flags;
169+
170+ /**
171+ * @ctx_ids: Array of contexts belonging to this device. Array members
172+ * are of type "struct pvr_context *".
173+ *
174+ * This array is used to allocate IDs used by the firmware.
175+ */
176+ struct xarray ctx_ids;
177+
178+ /**
179+ * @free_list_ids: Array of free lists belonging to this device. Array members
180+ * are of type "struct pvr_free_list *".
181+ *
182+ * This array is used to allocate IDs used by the firmware.
183+ */
184+ struct xarray free_list_ids;
185+
186+ /**
187+ * @job_ids: Array of jobs belonging to this device. Array members
188+ * are of type "struct pvr_job *".
189+ */
190+ struct xarray job_ids;
191+
192+ /**
193+ * @queues: Queue-related fields.
194+ */
195+ struct {
196+ /** @active: Active queue list. */
197+ struct list_head active;
198+
199+ /** @idle: Idle queue list. */
200+ struct list_head idle;
201+
202+ /** @lock: Lock protecting access to the active/idle lists. */
203+ struct mutex lock;
204+ } queues;
205+
206+ struct {
207+ /** @work: Work item for watchdog callback. */
208+ struct delayed_work work;
209+
210+ /** @old_kccb_cmds_executed: KCCB command execution count at last watchdog poll. */
211+ u32 old_kccb_cmds_executed;
212+
213+ /** @kccb_stall_count: Number of watchdog polls KCCB has been stalled for. */
214+ u32 kccb_stall_count;
215+ } watchdog;
216+
217+ struct {
218+ /** @ccb: Kernel CCB. */
219+ struct pvr_ccb ccb;
220+
221+ /** @rtn_q: Waitqueue for KCCB command return waiters. */
222+ wait_queue_head_t rtn_q;
223+
224+ /** @rtn_obj: Object representing KCCB return slots. */
225+ struct pvr_fw_object *rtn_obj;
226+
227+ /**
228+ * @rtn: Pointer to CPU mapping of KCCB return slots. Must be accessed by
229+ * READ_ONCE()/WRITE_ONCE().
230+ */
231+ u32 *rtn;
232+
233+ /** @slot_count: Total number of KCCB slots available. */
234+ u32 slot_count;
235+
236+ /** @reserved_count: Number of KCCB slots reserved for future use. */
237+ u32 reserved_count;
238+
239+ /**
240+ * @waiters: List of KCCB slot waiters.
241+ */
242+ struct list_head waiters;
243+
244+ /** @fence_ctx: KCCB fence context. */
245+ struct {
246+ /** @id: KCCB fence context ID allocated with dma_fence_context_alloc(). */
247+ u64 id;
248+
249+ /** @seqno: Sequence number incremented each time a fence is created. */
250+ atomic_t seqno;
251+
252+ /**
253+ * @lock: Lock used to synchronize access to fences allocated by this
254+ * context.
255+ */
256+ spinlock_t lock;
257+ } fence_ctx;
258+ } kccb;
259+
260+ /**
261+ * @lost: %true if the device has been lost.
262+ *
263+ * This variable is set if the device has become irretrievably unavailable, e.g. if the
264+ * firmware processor has stopped responding and can not be revived via a hard reset.
265+ */
266+ bool lost;
267+
268+ /**
269+ * @reset_sem: Reset semaphore.
270+ *
271+ * GPU reset code will lock this for writing. Any code that submits commands to the firmware
272+ * that isn't in an IRQ handler or on the scheduler workqueue must lock this for reading.
273+ * Once this has been successfully locked, &pvr_dev->lost _must_ be checked, and -%EIO must
274+ * be returned if it is set.
275+ */
276+ struct rw_semaphore reset_sem;
277+
278+ /** @sched_wq: Workqueue for schedulers. */
279+ struct workqueue_struct *sched_wq;
280+};
281+
282+/**
283+ * struct pvr_file - powervr-specific data to be assigned to &struct
284+ * drm_file.driver_priv
285+ */
286+struct pvr_file {
287+ /**
288+ * @file: A reference to the parent &struct drm_file.
289+ *
290+ * Do not access this member directly, instead call from_pvr_file().
291+ */
292+ struct drm_file *file;
293+
294+ /**
295+ * @pvr_dev: A reference to the powervr-specific wrapper for the
296+ * associated device. Saves on repeated calls to
297+ * to_pvr_device().
298+ */
299+ struct pvr_device *pvr_dev;
300+
301+ /**
302+ * @ctx_handles: Array of contexts belonging to this file. Array members
303+ * are of type "struct pvr_context *".
304+ *
305+ * This array is used to allocate handles returned to userspace.
306+ */
307+ struct xarray ctx_handles;
308+
309+ /**
310+ * @free_list_handles: Array of free lists belonging to this file. Array
311+ * members are of type "struct pvr_free_list *".
312+ *
313+ * This array is used to allocate handles returned to userspace.
314+ */
315+ struct xarray free_list_handles;
316+
317+ /**
318+ * @hwrt_handles: Array of HWRT datasets belonging to this file. Array
319+ * members are of type "struct pvr_hwrt_dataset *".
320+ *
321+ * This array is used to allocate handles returned to userspace.
322+ */
323+ struct xarray hwrt_handles;
324+
325+ /**
326+ * @vm_ctx_handles: Array of VM contexts belonging to this file. Array
327+ * members are of type "struct pvr_vm_context *".
328+ *
329+ * This array is used to allocate handles returned to userspace.
330+ */
331+ struct xarray vm_ctx_handles;
332+};
333+
334+/**
335+ * PVR_HAS_FEATURE() - Tests whether a PowerVR device has a given feature
336+ * @pvr_dev: [IN] Target PowerVR device.
337+ * @feature: [IN] Hardware feature name.
338+ *
339+ * Feature names are derived from those found in &struct pvr_device_features by
340+ * dropping the 'has_' prefix, which is applied by this macro.
341+ *
342+ * Return:
343+ * * true if the named feature is present in the hardware
344+ * * false if the named feature is not present in the hardware
345+ */
346+#define PVR_HAS_FEATURE(pvr_dev, feature) ((pvr_dev)->features.has_##feature)
347+
348+/**
349+ * PVR_FEATURE_VALUE() - Gets a PowerVR device feature value
350+ * @pvr_dev: [IN] Target PowerVR device.
351+ * @feature: [IN] Feature name.
352+ * @value_out: [OUT] Feature value.
353+ *
354+ * This macro will get a feature value for those features that have values.
355+ * If the feature is not present, nothing will be stored to @value_out.
356+ *
357+ * Feature names are derived from those found in &struct pvr_device_features by
358+ * dropping the 'has_' prefix.
359+ *
360+ * Return:
361+ * * 0 on success, or
362+ * * -%EINVAL if the named feature is not present in the hardware
363+ */
364+#define PVR_FEATURE_VALUE(pvr_dev, feature, value_out) \
365+ ({ \
366+ struct pvr_device *_pvr_dev = pvr_dev; \
367+ int _ret = -EINVAL; \
368+ if (_pvr_dev->features.has_##feature) { \
369+ *(value_out) = _pvr_dev->features.feature; \
370+ _ret = 0; \
371+ } \
372+ _ret; \
373+ })
374+
375+/**
376+ * PVR_HAS_QUIRK() - Tests whether a physical device has a given quirk
377+ * @pvr_dev: [IN] Target PowerVR device.
378+ * @quirk: [IN] Hardware quirk name.
379+ *
380+ * Quirk numbers are derived from those found in #pvr_device_quirks by
381+ * dropping the 'has_brn' prefix, which is applied by this macro.
382+ *
383+ * Returns
384+ * * true if the quirk is present in the hardware, or
385+ * * false if the quirk is not present in the hardware.
386+ */
387+#define PVR_HAS_QUIRK(pvr_dev, quirk) ((pvr_dev)->quirks.has_brn##quirk)
388+
389+/**
390+ * PVR_HAS_ENHANCEMENT() - Tests whether a physical device has a given
391+ * enhancement
392+ * @pvr_dev: [IN] Target PowerVR device.
393+ * @enhancement: [IN] Hardware enhancement name.
394+ *
395+ * Enhancement numbers are derived from those found in #pvr_device_enhancements
396+ * by dropping the 'has_ern' prefix, which is applied by this macro.
397+ *
398+ * Returns
399+ * * true if the enhancement is present in the hardware, or
400+ * * false if the enhancement is not present in the hardware.
401+ */
402+#define PVR_HAS_ENHANCEMENT(pvr_dev, enhancement) ((pvr_dev)->enhancements.has_ern##enhancement)
403+
404+#define from_pvr_device(pvr_dev) (&(pvr_dev)->base)
405+
406+#define to_pvr_device(drm_dev) container_of_const(drm_dev, struct pvr_device, base)
407+
408+#define from_pvr_file(pvr_file) ((pvr_file)->file)
409+
410+#define to_pvr_file(file) ((file)->driver_priv)
411+
412+/**
413+ * PVR_PACKED_BVNC() - Packs B, V, N and C values into a 64-bit unsigned integer
414+ * @b: Branch ID.
415+ * @v: Version ID.
416+ * @n: Number of scalable units.
417+ * @c: Config ID.
418+ *
419+ * The packed layout is as follows:
420+ *
421+ * +--------+--------+--------+-------+
422+ * | 63..48 | 47..32 | 31..16 | 15..0 |
423+ * +========+========+========+=======+
424+ * | B | V | N | C |
425+ * +--------+--------+--------+-------+
426+ *
427+ * pvr_gpu_id_to_packed_bvnc() should be used instead of this macro when a
428+ * &struct pvr_gpu_id is available in order to ensure proper type checking.
429+ *
430+ * Return: Packed BVNC.
431+ */
432+/* clang-format off */
433+#define PVR_PACKED_BVNC(b, v, n, c) \
434+ ((((u64)(b) & GENMASK_ULL(15, 0)) << 48) | \
435+ (((u64)(v) & GENMASK_ULL(15, 0)) << 32) | \
436+ (((u64)(n) & GENMASK_ULL(15, 0)) << 16) | \
437+ (((u64)(c) & GENMASK_ULL(15, 0)) << 0))
438+/* clang-format on */
439+
440+/**
441+ * pvr_gpu_id_to_packed_bvnc() - Packs B, V, N and C values into a 64-bit
442+ * unsigned integer
443+ * @gpu_id: GPU ID.
444+ *
445+ * The packed layout is as follows:
446+ *
447+ * +--------+--------+--------+-------+
448+ * | 63..48 | 47..32 | 31..16 | 15..0 |
449+ * +========+========+========+=======+
450+ * | B | V | N | C |
451+ * +--------+--------+--------+-------+
452+ *
453+ * This should be used in preference to PVR_PACKED_BVNC() when a &struct
454+ * pvr_gpu_id is available in order to ensure proper type checking.
455+ *
456+ * Return: Packed BVNC.
457+ */
458+static __always_inline u64
459+pvr_gpu_id_to_packed_bvnc(struct pvr_gpu_id *gpu_id)
460+{
461+ return PVR_PACKED_BVNC(gpu_id->b, gpu_id->v, gpu_id->n, gpu_id->c);
462+}
463+
464+static __always_inline void
465+packed_bvnc_to_pvr_gpu_id(u64 bvnc, struct pvr_gpu_id *gpu_id)
466+{
467+ gpu_id->b = (bvnc & GENMASK_ULL(63, 48)) >> 48;
468+ gpu_id->v = (bvnc & GENMASK_ULL(47, 32)) >> 32;
469+ gpu_id->n = (bvnc & GENMASK_ULL(31, 16)) >> 16;
470+ gpu_id->c = bvnc & GENMASK_ULL(15, 0);
471+}
472+
473+int pvr_device_init(struct pvr_device *pvr_dev);
474+void pvr_device_fini(struct pvr_device *pvr_dev);
475+void pvr_device_reset(struct pvr_device *pvr_dev);
476+
477+bool
478+pvr_device_has_uapi_quirk(struct pvr_device *pvr_dev, u32 quirk);
479+bool
480+pvr_device_has_uapi_enhancement(struct pvr_device *pvr_dev, u32 enhancement);
481+bool
482+pvr_device_has_feature(struct pvr_device *pvr_dev, u32 feature);
483+
484+/**
485+ * PVR_CR_FIELD_GET() - Extract a single field from a PowerVR control register
486+ * @val: Value of the target register.
487+ * @field: Field specifier, as defined in "pvr_rogue_cr_defs.h".
488+ *
489+ * Return: The extracted field.
490+ */
491+#define PVR_CR_FIELD_GET(val, field) FIELD_GET(~ROGUE_CR_##field##_CLRMSK, val)
492+
493+/**
494+ * pvr_cr_read32() - Read a 32-bit register from a PowerVR device
495+ * @pvr_dev: Target PowerVR device.
496+ * @reg: Target register.
497+ *
498+ * Return: The value of the requested register.
499+ */
500+static __always_inline u32
501+pvr_cr_read32(struct pvr_device *pvr_dev, u32 reg)
502+{
503+ return ioread32(pvr_dev->regs + reg);
504+}
505+
506+/**
507+ * pvr_cr_read64() - Read a 64-bit register from a PowerVR device
508+ * @pvr_dev: Target PowerVR device.
509+ * @reg: Target register.
510+ *
511+ * Return: The value of the requested register.
512+ */
513+static __always_inline u64
514+pvr_cr_read64(struct pvr_device *pvr_dev, u32 reg)
515+{
516+ return ioread64(pvr_dev->regs + reg);
517+}
518+
519+/**
520+ * pvr_cr_write32() - Write to a 32-bit register in a PowerVR device
521+ * @pvr_dev: Target PowerVR device.
522+ * @reg: Target register.
523+ * @val: Value to write.
524+ */
525+static __always_inline void
526+pvr_cr_write32(struct pvr_device *pvr_dev, u32 reg, u32 val)
527+{
528+ iowrite32(val, pvr_dev->regs + reg);
529+}
530+
531+/**
532+ * pvr_cr_write64() - Write to a 64-bit register in a PowerVR device
533+ * @pvr_dev: Target PowerVR device.
534+ * @reg: Target register.
535+ * @val: Value to write.
536+ */
537+static __always_inline void
538+pvr_cr_write64(struct pvr_device *pvr_dev, u32 reg, u64 val)
539+{
540+ iowrite64(val, pvr_dev->regs + reg);
541+}
542+
543+/**
544+ * pvr_cr_poll_reg32() - Wait for a 32-bit register to match a given value by
545+ * polling
546+ * @pvr_dev: Target PowerVR device.
547+ * @reg_addr: Address of register.
548+ * @reg_value: Expected register value (after masking).
549+ * @reg_mask: Mask of bits valid for comparison with @reg_value.
550+ * @timeout_usec: Timeout length, in us.
551+ *
552+ * Returns:
553+ * * 0 on success, or
554+ * * -%ETIMEDOUT on timeout.
555+ */
556+static __always_inline int
557+pvr_cr_poll_reg32(struct pvr_device *pvr_dev, u32 reg_addr, u32 reg_value,
558+ u32 reg_mask, u64 timeout_usec)
559+{
560+ u32 value;
561+
562+ return readl_poll_timeout(pvr_dev->regs + reg_addr, value,
563+ (value & reg_mask) == reg_value, 0, timeout_usec);
564+}
565+
566+/**
567+ * pvr_cr_poll_reg64() - Wait for a 64-bit register to match a given value by
568+ * polling
569+ * @pvr_dev: Target PowerVR device.
570+ * @reg_addr: Address of register.
571+ * @reg_value: Expected register value (after masking).
572+ * @reg_mask: Mask of bits valid for comparison with @reg_value.
573+ * @timeout_usec: Timeout length, in us.
574+ *
575+ * Returns:
576+ * * 0 on success, or
577+ * * -%ETIMEDOUT on timeout.
578+ */
579+static __always_inline int
580+pvr_cr_poll_reg64(struct pvr_device *pvr_dev, u32 reg_addr, u64 reg_value,
581+ u64 reg_mask, u64 timeout_usec)
582+{
583+ u64 value;
584+
585+ return readq_poll_timeout(pvr_dev->regs + reg_addr, value,
586+ (value & reg_mask) == reg_value, 0, timeout_usec);
587+}
588+
589+/**
590+ * pvr_round_up_to_cacheline_size() - Round up a provided size to be cacheline
591+ * aligned
592+ * @pvr_dev: Target PowerVR device.
593+ * @size: Initial size, in bytes.
594+ *
595+ * Returns:
596+ * * Size aligned to cacheline size.
597+ */
598+static __always_inline size_t
599+pvr_round_up_to_cacheline_size(struct pvr_device *pvr_dev, size_t size)
600+{
601+ u16 slc_cacheline_size_bits = 0;
602+ u16 slc_cacheline_size_bytes;
603+
604+ WARN_ON(!PVR_HAS_FEATURE(pvr_dev, slc_cache_line_size_bits));
605+ PVR_FEATURE_VALUE(pvr_dev, slc_cache_line_size_bits,
606+ &slc_cacheline_size_bits);
607+ slc_cacheline_size_bytes = slc_cacheline_size_bits / 8;
608+
609+ return round_up(size, slc_cacheline_size_bytes);
610+}
611+
612+/**
613+ * DOC: IOCTL validation helpers
614+ *
615+ * To validate the constraints imposed on IOCTL argument structs, a collection
616+ * of macros and helper functions exist in ``pvr_device.h``.
617+ *
618+ * Of the current helpers, it should only be necessary to call
619+ * PVR_IOCTL_UNION_PADDING_CHECK() directly. This macro should be used once in
620+ * every code path which extracts a union member from a struct passed from
621+ * userspace.
622+ */
623+
624+/**
625+ * pvr_ioctl_union_padding_check() - Validate that the implicit padding between
626+ * the end of a union member and the end of the union itself is zeroed.
627+ * @instance: Pointer to the instance of the struct to validate.
628+ * @union_offset: Offset into the type of @instance of the target union. Must
629+ * be 64-bit aligned.
630+ * @union_size: Size of the target union in the type of @instance. Must be
631+ * 64-bit aligned.
632+ * @member_size: Size of the target member in the target union specified by
633+ * @union_offset and @union_size. It is assumed that the offset of the target
634+ * member is zero relative to @union_offset. Must be 64-bit aligned.
635+ *
636+ * You probably want to use PVR_IOCTL_UNION_PADDING_CHECK() instead of calling
637+ * this function directly, since that macro abstracts away much of the setup,
638+ * and also provides some static validation. See its docs for details.
639+ *
640+ * Return:
641+ * * %true if every byte between the end of the used member of the union and
642+ * the end of that union is zeroed, or
643+ * * %false otherwise.
644+ */
645+static __always_inline bool
646+pvr_ioctl_union_padding_check(void *instance, size_t union_offset,
647+ size_t union_size, size_t member_size)
648+{
649+ /*
650+ * void pointer arithmetic is technically illegal - cast to a byte
651+ * pointer so this addition works safely.
652+ */
653+ void *padding_start = ((u8 *)instance) + union_offset + member_size;
654+ size_t padding_size = union_size - member_size;
655+
656+ return !memchr_inv(padding_start, 0, padding_size);
657+}
658+
659+/**
660+ * PVR_STATIC_ASSERT_64BIT_ALIGNED() - Inline assertion for 64-bit alignment.
661+ * @static_expr_: Target expression to evaluate.
662+ *
663+ * If @static_expr_ does not evaluate to a constant integer which would be a
664+ * 64-bit aligned address (i.e. a multiple of 8), compilation will fail.
665+ *
666+ * Return:
667+ * The value of @static_expr_.
668+ */
669+#define PVR_STATIC_ASSERT_64BIT_ALIGNED(static_expr_) \
670+ ({ \
671+ static_assert(((static_expr_) & (sizeof(u64) - 1)) == 0); \
672+ (static_expr_); \
673+ })
674+
675+/**
676+ * PVR_IOCTL_UNION_PADDING_CHECK() - Validate that the implicit padding between
677+ * the end of a union member and the end of the union itself is zeroed.
678+ * @struct_instance_: An expression which evaluates to a pointer to a UAPI data
679+ * struct.
680+ * @union_: The name of the union member of @struct_instance_ to check. If the
681+ * union member is nested within the type of @struct_instance_, this may
682+ * contain the member access operator (".").
683+ * @member_: The name of the member of @union_ to assess.
684+ *
685+ * This is a wrapper around pvr_ioctl_union_padding_check() which performs
686+ * alignment checks and simplifies things for the caller.
687+ *
688+ * Return:
689+ * * %true if every byte in @struct_instance_ between the end of @member_ and
690+ * the end of @union_ is zeroed, or
691+ * * %false otherwise.
692+ */
693+#define PVR_IOCTL_UNION_PADDING_CHECK(struct_instance_, union_, member_) \
694+ ({ \
695+ typeof(struct_instance_) __instance = (struct_instance_); \
696+ size_t __union_offset = PVR_STATIC_ASSERT_64BIT_ALIGNED( \
697+ offsetof(typeof(*__instance), union_)); \
698+ size_t __union_size = PVR_STATIC_ASSERT_64BIT_ALIGNED( \
699+ sizeof(__instance->union_)); \
700+ size_t __member_size = PVR_STATIC_ASSERT_64BIT_ALIGNED( \
701+ sizeof(__instance->union_.member_)); \
702+ pvr_ioctl_union_padding_check(__instance, __union_offset, \
703+ __union_size, __member_size); \
704+ })
705+
706+#define PVR_FW_PROCESSOR_TYPE_META 0
707+#define PVR_FW_PROCESSOR_TYPE_MIPS 1
708+#define PVR_FW_PROCESSOR_TYPE_RISCV 2
709+
710+#endif /* PVR_DEVICE_H */
drivers/gpu/drm/imagination/pvr_device_info.c+254−0
@@ -0,0 +1,254 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_device.h"
5+#include "pvr_device_info.h"
6+#include "pvr_rogue_fwif_dev_info.h"
7+
8+#include <drm/drm_print.h>
9+
10+#include <linux/bits.h>
11+#include <linux/minmax.h>
12+#include <linux/stddef.h>
13+#include <linux/types.h>
14+
15+#define QUIRK_MAPPING(quirk) \
16+ [PVR_FW_HAS_BRN_##quirk] = offsetof(struct pvr_device, quirks.has_brn##quirk)
17+
18+static const uintptr_t quirks_mapping[] = {
19+ QUIRK_MAPPING(44079),
20+ QUIRK_MAPPING(47217),
21+ QUIRK_MAPPING(48492),
22+ QUIRK_MAPPING(48545),
23+ QUIRK_MAPPING(49927),
24+ QUIRK_MAPPING(50767),
25+ QUIRK_MAPPING(51764),
26+ QUIRK_MAPPING(62269),
27+ QUIRK_MAPPING(63142),
28+ QUIRK_MAPPING(63553),
29+ QUIRK_MAPPING(66011),
30+ QUIRK_MAPPING(71242),
31+};
32+
33+#undef QUIRK_MAPPING
34+
35+#define ENHANCEMENT_MAPPING(enhancement) \
36+ [PVR_FW_HAS_ERN_##enhancement] = offsetof(struct pvr_device, \
37+ enhancements.has_ern##enhancement)
38+
39+static const uintptr_t enhancements_mapping[] = {
40+ ENHANCEMENT_MAPPING(35421),
41+ ENHANCEMENT_MAPPING(38020),
42+ ENHANCEMENT_MAPPING(38748),
43+ ENHANCEMENT_MAPPING(42064),
44+ ENHANCEMENT_MAPPING(42290),
45+ ENHANCEMENT_MAPPING(42606),
46+ ENHANCEMENT_MAPPING(47025),
47+ ENHANCEMENT_MAPPING(57596),
48+};
49+
50+#undef ENHANCEMENT_MAPPING
51+
52+static void pvr_device_info_set_common(struct pvr_device *pvr_dev, const u64 *bitmask,
53+ u32 bitmask_size, const uintptr_t *mapping, u32 mapping_max)
54+{
55+ const u32 mapping_max_size = (mapping_max + 63) >> 6;
56+ const u32 nr_bits = min(bitmask_size * 64, mapping_max);
57+
58+ /* Warn if any unsupported values in the bitmask. */
59+ if (bitmask_size > mapping_max_size) {
60+ if (mapping == quirks_mapping)
61+ drm_warn(from_pvr_device(pvr_dev), "Unsupported quirks in firmware image");
62+ else
63+ drm_warn(from_pvr_device(pvr_dev),
64+ "Unsupported enhancements in firmware image");
65+ } else if (bitmask_size == mapping_max_size && (mapping_max & 63)) {
66+ u64 invalid_mask = ~0ull << (mapping_max & 63);
67+
68+ if (bitmask[bitmask_size - 1] & invalid_mask) {
69+ if (mapping == quirks_mapping)
70+ drm_warn(from_pvr_device(pvr_dev),
71+ "Unsupported quirks in firmware image");
72+ else
73+ drm_warn(from_pvr_device(pvr_dev),
74+ "Unsupported enhancements in firmware image");
75+ }
76+ }
77+
78+ for (u32 i = 0; i < nr_bits; i++) {
79+ if (bitmask[i >> 6] & BIT_ULL(i & 63))
80+ *(bool *)((u8 *)pvr_dev + mapping[i]) = true;
81+ }
82+}
83+
84+/**
85+ * pvr_device_info_set_quirks() - Set device quirks from device information in firmware
86+ * @pvr_dev: Device pointer.
87+ * @quirks: Pointer to quirks mask in device information.
88+ * @quirks_size: Size of quirks mask, in u64s.
89+ */
90+void pvr_device_info_set_quirks(struct pvr_device *pvr_dev, const u64 *quirks, u32 quirks_size)
91+{
92+ BUILD_BUG_ON(ARRAY_SIZE(quirks_mapping) != PVR_FW_HAS_BRN_MAX);
93+
94+ pvr_device_info_set_common(pvr_dev, quirks, quirks_size, quirks_mapping,
95+ ARRAY_SIZE(quirks_mapping));
96+}
97+
98+/**
99+ * pvr_device_info_set_enhancements() - Set device enhancements from device information in firmware
100+ * @pvr_dev: Device pointer.
101+ * @enhancements: Pointer to enhancements mask in device information.
102+ * @enhancements_size: Size of enhancements mask, in u64s.
103+ */
104+void pvr_device_info_set_enhancements(struct pvr_device *pvr_dev, const u64 *enhancements,
105+ u32 enhancements_size)
106+{
107+ BUILD_BUG_ON(ARRAY_SIZE(enhancements_mapping) != PVR_FW_HAS_ERN_MAX);
108+
109+ pvr_device_info_set_common(pvr_dev, enhancements, enhancements_size,
110+ enhancements_mapping, ARRAY_SIZE(enhancements_mapping));
111+}
112+
113+#define FEATURE_MAPPING(fw_feature, feature) \
114+ [PVR_FW_HAS_FEATURE_##fw_feature] = { \
115+ .flag_offset = offsetof(struct pvr_device, features.has_##feature), \
116+ .value_offset = 0 \
117+ }
118+
119+#define FEATURE_MAPPING_VALUE(fw_feature, feature) \
120+ [PVR_FW_HAS_FEATURE_##fw_feature] = { \
121+ .flag_offset = offsetof(struct pvr_device, features.has_##feature), \
122+ .value_offset = offsetof(struct pvr_device, features.feature) \
123+ }
124+
125+static const struct {
126+ uintptr_t flag_offset;
127+ uintptr_t value_offset;
128+} features_mapping[] = {
129+ FEATURE_MAPPING(AXI_ACELITE, axi_acelite),
130+ FEATURE_MAPPING_VALUE(CDM_CONTROL_STREAM_FORMAT, cdm_control_stream_format),
131+ FEATURE_MAPPING(CLUSTER_GROUPING, cluster_grouping),
132+ FEATURE_MAPPING_VALUE(COMMON_STORE_SIZE_IN_DWORDS, common_store_size_in_dwords),
133+ FEATURE_MAPPING(COMPUTE, compute),
134+ FEATURE_MAPPING(COMPUTE_MORTON_CAPABLE, compute_morton_capable),
135+ FEATURE_MAPPING(COMPUTE_OVERLAP, compute_overlap),
136+ FEATURE_MAPPING(COREID_PER_OS, coreid_per_os),
137+ FEATURE_MAPPING(DYNAMIC_DUST_POWER, dynamic_dust_power),
138+ FEATURE_MAPPING_VALUE(ECC_RAMS, ecc_rams),
139+ FEATURE_MAPPING_VALUE(FBCDC, fbcdc),
140+ FEATURE_MAPPING_VALUE(FBCDC_ALGORITHM, fbcdc_algorithm),
141+ FEATURE_MAPPING_VALUE(FBCDC_ARCHITECTURE, fbcdc_architecture),
142+ FEATURE_MAPPING_VALUE(FBC_MAX_DEFAULT_DESCRIPTORS, fbc_max_default_descriptors),
143+ FEATURE_MAPPING_VALUE(FBC_MAX_LARGE_DESCRIPTORS, fbc_max_large_descriptors),
144+ FEATURE_MAPPING(FB_CDC_V4, fb_cdc_v4),
145+ FEATURE_MAPPING(GPU_MULTICORE_SUPPORT, gpu_multicore_support),
146+ FEATURE_MAPPING(GPU_VIRTUALISATION, gpu_virtualisation),
147+ FEATURE_MAPPING(GS_RTA_SUPPORT, gs_rta_support),
148+ FEATURE_MAPPING(IRQ_PER_OS, irq_per_os),
149+ FEATURE_MAPPING_VALUE(ISP_MAX_TILES_IN_FLIGHT, isp_max_tiles_in_flight),
150+ FEATURE_MAPPING_VALUE(ISP_SAMPLES_PER_PIXEL, isp_samples_per_pixel),
151+ FEATURE_MAPPING(ISP_ZLS_D24_S8_PACKING_OGL_MODE, isp_zls_d24_s8_packing_ogl_mode),
152+ FEATURE_MAPPING_VALUE(LAYOUT_MARS, layout_mars),
153+ FEATURE_MAPPING_VALUE(MAX_PARTITIONS, max_partitions),
154+ FEATURE_MAPPING_VALUE(META, meta),
155+ FEATURE_MAPPING_VALUE(META_COREMEM_SIZE, meta_coremem_size),
156+ FEATURE_MAPPING(MIPS, mips),
157+ FEATURE_MAPPING_VALUE(NUM_CLUSTERS, num_clusters),
158+ FEATURE_MAPPING_VALUE(NUM_ISP_IPP_PIPES, num_isp_ipp_pipes),
159+ FEATURE_MAPPING_VALUE(NUM_OSIDS, num_osids),
160+ FEATURE_MAPPING_VALUE(NUM_RASTER_PIPES, num_raster_pipes),
161+ FEATURE_MAPPING(PBE2_IN_XE, pbe2_in_xe),
162+ FEATURE_MAPPING(PBVNC_COREID_REG, pbvnc_coreid_reg),
163+ FEATURE_MAPPING(PERFBUS, perfbus),
164+ FEATURE_MAPPING(PERF_COUNTER_BATCH, perf_counter_batch),
165+ FEATURE_MAPPING_VALUE(PHYS_BUS_WIDTH, phys_bus_width),
166+ FEATURE_MAPPING(RISCV_FW_PROCESSOR, riscv_fw_processor),
167+ FEATURE_MAPPING(ROGUEXE, roguexe),
168+ FEATURE_MAPPING(S7_TOP_INFRASTRUCTURE, s7_top_infrastructure),
169+ FEATURE_MAPPING(SIMPLE_INTERNAL_PARAMETER_FORMAT, simple_internal_parameter_format),
170+ FEATURE_MAPPING(SIMPLE_INTERNAL_PARAMETER_FORMAT_V2, simple_internal_parameter_format_v2),
171+ FEATURE_MAPPING_VALUE(SIMPLE_PARAMETER_FORMAT_VERSION, simple_parameter_format_version),
172+ FEATURE_MAPPING_VALUE(SLC_BANKS, slc_banks),
173+ FEATURE_MAPPING_VALUE(SLC_CACHE_LINE_SIZE_BITS, slc_cache_line_size_bits),
174+ FEATURE_MAPPING(SLC_SIZE_CONFIGURABLE, slc_size_configurable),
175+ FEATURE_MAPPING_VALUE(SLC_SIZE_IN_KILOBYTES, slc_size_in_kilobytes),
176+ FEATURE_MAPPING(SOC_TIMER, soc_timer),
177+ FEATURE_MAPPING(SYS_BUS_SECURE_RESET, sys_bus_secure_reset),
178+ FEATURE_MAPPING(TESSELLATION, tessellation),
179+ FEATURE_MAPPING(TILE_REGION_PROTECTION, tile_region_protection),
180+ FEATURE_MAPPING_VALUE(TILE_SIZE_X, tile_size_x),
181+ FEATURE_MAPPING_VALUE(TILE_SIZE_Y, tile_size_y),
182+ FEATURE_MAPPING(TLA, tla),
183+ FEATURE_MAPPING(TPU_CEM_DATAMASTER_GLOBAL_REGISTERS, tpu_cem_datamaster_global_registers),
184+ FEATURE_MAPPING(TPU_DM_GLOBAL_REGISTERS, tpu_dm_global_registers),
185+ FEATURE_MAPPING(TPU_FILTERING_MODE_CONTROL, tpu_filtering_mode_control),
186+ FEATURE_MAPPING_VALUE(USC_MIN_OUTPUT_REGISTERS_PER_PIX, usc_min_output_registers_per_pix),
187+ FEATURE_MAPPING(VDM_DRAWINDIRECT, vdm_drawindirect),
188+ FEATURE_MAPPING(VDM_OBJECT_LEVEL_LLS, vdm_object_level_lls),
189+ FEATURE_MAPPING_VALUE(VIRTUAL_ADDRESS_SPACE_BITS, virtual_address_space_bits),
190+ FEATURE_MAPPING(WATCHDOG_TIMER, watchdog_timer),
191+ FEATURE_MAPPING(WORKGROUP_PROTECTION, workgroup_protection),
192+ FEATURE_MAPPING_VALUE(XE_ARCHITECTURE, xe_architecture),
193+ FEATURE_MAPPING(XE_MEMORY_HIERARCHY, xe_memory_hierarchy),
194+ FEATURE_MAPPING(XE_TPU2, xe_tpu2),
195+ FEATURE_MAPPING_VALUE(XPU_MAX_REGBANKS_ADDR_WIDTH, xpu_max_regbanks_addr_width),
196+ FEATURE_MAPPING_VALUE(XPU_MAX_SLAVES, xpu_max_slaves),
197+ FEATURE_MAPPING_VALUE(XPU_REGISTER_BROADCAST, xpu_register_broadcast),
198+ FEATURE_MAPPING(XT_TOP_INFRASTRUCTURE, xt_top_infrastructure),
199+ FEATURE_MAPPING(ZLS_SUBTILE, zls_subtile),
200+};
201+
202+#undef FEATURE_MAPPING_VALUE
203+#undef FEATURE_MAPPING
204+
205+/**
206+ * pvr_device_info_set_features() - Set device features from device information in firmware
207+ * @pvr_dev: Device pointer.
208+ * @features: Pointer to features mask in device information.
209+ * @features_size: Size of features mask, in u64s.
210+ * @feature_param_size: Size of feature parameters, in u64s.
211+ *
212+ * Returns:
213+ * * 0 on success, or
214+ * * -%EINVAL on malformed stream.
215+ */
216+int pvr_device_info_set_features(struct pvr_device *pvr_dev, const u64 *features, u32 features_size,
217+ u32 feature_param_size)
218+{
219+ const u32 mapping_max = ARRAY_SIZE(features_mapping);
220+ const u32 mapping_max_size = (mapping_max + 63) >> 6;
221+ const u32 nr_bits = min(features_size * 64, mapping_max);
222+ const u64 *feature_params = features + features_size;
223+ u32 param_idx = 0;
224+
225+ BUILD_BUG_ON(ARRAY_SIZE(features_mapping) != PVR_FW_HAS_FEATURE_MAX);
226+
227+ /* Verify no unsupported values in the bitmask. */
228+ if (features_size > mapping_max_size) {
229+ drm_warn(from_pvr_device(pvr_dev), "Unsupported features in firmware image");
230+ } else if (features_size == mapping_max_size && (mapping_max & 63)) {
231+ u64 invalid_mask = ~0ull << (mapping_max & 63);
232+
233+ if (features[features_size - 1] & invalid_mask)
234+ drm_warn(from_pvr_device(pvr_dev),
235+ "Unsupported features in firmware image");
236+ }
237+
238+ for (u32 i = 0; i < nr_bits; i++) {
239+ if (features[i >> 6] & BIT_ULL(i & 63)) {
240+ *(bool *)((u8 *)pvr_dev + features_mapping[i].flag_offset) = true;
241+
242+ if (features_mapping[i].value_offset) {
243+ if (param_idx >= feature_param_size)
244+ return -EINVAL;
245+
246+ *(u64 *)((u8 *)pvr_dev + features_mapping[i].value_offset) =
247+ feature_params[param_idx];
248+ param_idx++;
249+ }
250+ }
251+ }
252+
253+ return 0;
254+}
drivers/gpu/drm/imagination/pvr_device_info.h+186−0
@@ -0,0 +1,186 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_DEVICE_INFO_H
5+#define PVR_DEVICE_INFO_H
6+
7+#include <linux/types.h>
8+
9+struct pvr_device;
10+
11+/*
12+ * struct pvr_device_features - Hardware feature information
13+ */
14+struct pvr_device_features {
15+ bool has_axi_acelite;
16+ bool has_cdm_control_stream_format;
17+ bool has_cluster_grouping;
18+ bool has_common_store_size_in_dwords;
19+ bool has_compute;
20+ bool has_compute_morton_capable;
21+ bool has_compute_overlap;
22+ bool has_coreid_per_os;
23+ bool has_dynamic_dust_power;
24+ bool has_ecc_rams;
25+ bool has_fb_cdc_v4;
26+ bool has_fbc_max_default_descriptors;
27+ bool has_fbc_max_large_descriptors;
28+ bool has_fbcdc;
29+ bool has_fbcdc_algorithm;
30+ bool has_fbcdc_architecture;
31+ bool has_gpu_multicore_support;
32+ bool has_gpu_virtualisation;
33+ bool has_gs_rta_support;
34+ bool has_irq_per_os;
35+ bool has_isp_max_tiles_in_flight;
36+ bool has_isp_samples_per_pixel;
37+ bool has_isp_zls_d24_s8_packing_ogl_mode;
38+ bool has_layout_mars;
39+ bool has_max_partitions;
40+ bool has_meta;
41+ bool has_meta_coremem_size;
42+ bool has_mips;
43+ bool has_num_clusters;
44+ bool has_num_isp_ipp_pipes;
45+ bool has_num_osids;
46+ bool has_num_raster_pipes;
47+ bool has_pbe2_in_xe;
48+ bool has_pbvnc_coreid_reg;
49+ bool has_perfbus;
50+ bool has_perf_counter_batch;
51+ bool has_phys_bus_width;
52+ bool has_riscv_fw_processor;
53+ bool has_roguexe;
54+ bool has_s7_top_infrastructure;
55+ bool has_simple_internal_parameter_format;
56+ bool has_simple_internal_parameter_format_v2;
57+ bool has_simple_parameter_format_version;
58+ bool has_slc_banks;
59+ bool has_slc_cache_line_size_bits;
60+ bool has_slc_size_configurable;
61+ bool has_slc_size_in_kilobytes;
62+ bool has_soc_timer;
63+ bool has_sys_bus_secure_reset;
64+ bool has_tessellation;
65+ bool has_tile_region_protection;
66+ bool has_tile_size_x;
67+ bool has_tile_size_y;
68+ bool has_tla;
69+ bool has_tpu_cem_datamaster_global_registers;
70+ bool has_tpu_dm_global_registers;
71+ bool has_tpu_filtering_mode_control;
72+ bool has_usc_min_output_registers_per_pix;
73+ bool has_vdm_drawindirect;
74+ bool has_vdm_object_level_lls;
75+ bool has_virtual_address_space_bits;
76+ bool has_watchdog_timer;
77+ bool has_workgroup_protection;
78+ bool has_xe_architecture;
79+ bool has_xe_memory_hierarchy;
80+ bool has_xe_tpu2;
81+ bool has_xpu_max_regbanks_addr_width;
82+ bool has_xpu_max_slaves;
83+ bool has_xpu_register_broadcast;
84+ bool has_xt_top_infrastructure;
85+ bool has_zls_subtile;
86+
87+ u64 cdm_control_stream_format;
88+ u64 common_store_size_in_dwords;
89+ u64 ecc_rams;
90+ u64 fbc_max_default_descriptors;
91+ u64 fbc_max_large_descriptors;
92+ u64 fbcdc;
93+ u64 fbcdc_algorithm;
94+ u64 fbcdc_architecture;
95+ u64 isp_max_tiles_in_flight;
96+ u64 isp_samples_per_pixel;
97+ u64 layout_mars;
98+ u64 max_partitions;
99+ u64 meta;
100+ u64 meta_coremem_size;
101+ u64 num_clusters;
102+ u64 num_isp_ipp_pipes;
103+ u64 num_osids;
104+ u64 num_raster_pipes;
105+ u64 phys_bus_width;
106+ u64 simple_parameter_format_version;
107+ u64 slc_banks;
108+ u64 slc_cache_line_size_bits;
109+ u64 slc_size_in_kilobytes;
110+ u64 tile_size_x;
111+ u64 tile_size_y;
112+ u64 usc_min_output_registers_per_pix;
113+ u64 virtual_address_space_bits;
114+ u64 xe_architecture;
115+ u64 xpu_max_regbanks_addr_width;
116+ u64 xpu_max_slaves;
117+ u64 xpu_register_broadcast;
118+};
119+
120+/*
121+ * struct pvr_device_quirks - Hardware quirk information
122+ */
123+struct pvr_device_quirks {
124+ bool has_brn44079;
125+ bool has_brn47217;
126+ bool has_brn48492;
127+ bool has_brn48545;
128+ bool has_brn49927;
129+ bool has_brn50767;
130+ bool has_brn51764;
131+ bool has_brn62269;
132+ bool has_brn63142;
133+ bool has_brn63553;
134+ bool has_brn66011;
135+ bool has_brn71242;
136+};
137+
138+/*
139+ * struct pvr_device_enhancements - Hardware enhancement information
140+ */
141+struct pvr_device_enhancements {
142+ bool has_ern35421;
143+ bool has_ern38020;
144+ bool has_ern38748;
145+ bool has_ern42064;
146+ bool has_ern42290;
147+ bool has_ern42606;
148+ bool has_ern47025;
149+ bool has_ern57596;
150+};
151+
152+void pvr_device_info_set_quirks(struct pvr_device *pvr_dev, const u64 *bitmask,
153+ u32 bitmask_len);
154+void pvr_device_info_set_enhancements(struct pvr_device *pvr_dev, const u64 *bitmask,
155+ u32 bitmask_len);
156+int pvr_device_info_set_features(struct pvr_device *pvr_dev, const u64 *features, u32 features_size,
157+ u32 feature_param_size);
158+
159+/*
160+ * Meta cores
161+ *
162+ * These are the values for the 'meta' feature when the feature is present
163+ * (as per &struct pvr_device_features)/
164+ */
165+#define PVR_META_MTP218 (1)
166+#define PVR_META_MTP219 (2)
167+#define PVR_META_LTP218 (3)
168+#define PVR_META_LTP217 (4)
169+
170+enum {
171+ PVR_FEATURE_CDM_USER_MODE_QUEUE,
172+ PVR_FEATURE_CLUSTER_GROUPING,
173+ PVR_FEATURE_COMPUTE_MORTON_CAPABLE,
174+ PVR_FEATURE_FB_CDC_V4,
175+ PVR_FEATURE_GPU_MULTICORE_SUPPORT,
176+ PVR_FEATURE_ISP_ZLS_D24_S8_PACKING_OGL_MODE,
177+ PVR_FEATURE_REQUIRES_FB_CDC_ZLS_SETUP,
178+ PVR_FEATURE_S7_TOP_INFRASTRUCTURE,
179+ PVR_FEATURE_TESSELLATION,
180+ PVR_FEATURE_TPU_DM_GLOBAL_REGISTERS,
181+ PVR_FEATURE_VDM_DRAWINDIRECT,
182+ PVR_FEATURE_VDM_OBJECT_LEVEL_LLS,
183+ PVR_FEATURE_ZLS_SUBTILE,
184+};
185+
186+#endif /* PVR_DEVICE_INFO_H */
drivers/gpu/drm/imagination/pvr_drv.c+1501−0
@@ -0,0 +1,1501 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_context.h"
5+#include "pvr_debugfs.h"
6+#include "pvr_device.h"
7+#include "pvr_drv.h"
8+#include "pvr_free_list.h"
9+#include "pvr_gem.h"
10+#include "pvr_hwrt.h"
11+#include "pvr_job.h"
12+#include "pvr_mmu.h"
13+#include "pvr_power.h"
14+#include "pvr_rogue_defs.h"
15+#include "pvr_rogue_fwif_client.h"
16+#include "pvr_rogue_fwif_shared.h"
17+#include "pvr_vm.h"
18+
19+#include <uapi/drm/pvr_drm.h>
20+
21+#include <drm/drm_device.h>
22+#include <drm/drm_drv.h>
23+#include <drm/drm_file.h>
24+#include <drm/drm_gem.h>
25+#include <drm/drm_ioctl.h>
26+
27+#include <linux/err.h>
28+#include <linux/export.h>
29+#include <linux/fs.h>
30+#include <linux/kernel.h>
31+#include <linux/mod_devicetable.h>
32+#include <linux/module.h>
33+#include <linux/moduleparam.h>
34+#include <linux/of_device.h>
35+#include <linux/of_platform.h>
36+#include <linux/platform_device.h>
37+#include <linux/pm_runtime.h>
38+#include <linux/xarray.h>
39+
40+/**
41+ * DOC: PowerVR (Series 6 and later) and IMG Graphics Driver
42+ *
43+ * This driver supports the following PowerVR/IMG graphics cores from Imagination Technologies:
44+ *
45+ * * AXE-1-16M (found in Texas Instruments AM62)
46+ */
47+
48+/**
49+ * pvr_ioctl_create_bo() - IOCTL to create a GEM buffer object.
50+ * @drm_dev: [IN] Target DRM device.
51+ * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
52+ * &struct drm_pvr_ioctl_create_bo_args.
53+ * @file: [IN] DRM file-private data.
54+ *
55+ * Called from userspace with %DRM_IOCTL_PVR_CREATE_BO.
56+ *
57+ * Return:
58+ * * 0 on success,
59+ * * -%EINVAL if the value of &drm_pvr_ioctl_create_bo_args.size is zero
60+ * or wider than &typedef size_t,
61+ * * -%EINVAL if any bits in &drm_pvr_ioctl_create_bo_args.flags that are
62+ * reserved or undefined are set,
63+ * * -%EINVAL if any padding fields in &drm_pvr_ioctl_create_bo_args are not
64+ * zero,
65+ * * Any error encountered while creating the object (see
66+ * pvr_gem_object_create()), or
67+ * * Any error encountered while transferring ownership of the object into a
68+ * userspace-accessible handle (see pvr_gem_object_into_handle()).
69+ */
70+static int
71+pvr_ioctl_create_bo(struct drm_device *drm_dev, void *raw_args,
72+ struct drm_file *file)
73+{
74+ struct drm_pvr_ioctl_create_bo_args *args = raw_args;
75+ struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
76+ struct pvr_file *pvr_file = to_pvr_file(file);
77+
78+ struct pvr_gem_object *pvr_obj;
79+ size_t sanitized_size;
80+
81+ int idx;
82+ int err;
83+
84+ if (!drm_dev_enter(drm_dev, &idx))
85+ return -EIO;
86+
87+ /* All padding fields must be zeroed. */
88+ if (args->_padding_c != 0) {
89+ err = -EINVAL;
90+ goto err_drm_dev_exit;
91+ }
92+
93+ /*
94+ * On 64-bit platforms (our primary target), size_t is a u64. However,
95+ * on other architectures we have to check for overflow when casting
96+ * down to size_t from u64.
97+ *
98+ * We also disallow zero-sized allocations, and reserved (kernel-only)
99+ * flags.
100+ */
101+ if (args->size > SIZE_MAX || args->size == 0 || args->flags &
102+ ~DRM_PVR_BO_FLAGS_MASK || args->size & (PVR_DEVICE_PAGE_SIZE - 1)) {
103+ err = -EINVAL;
104+ goto err_drm_dev_exit;
105+ }
106+
107+ sanitized_size = (size_t)args->size;
108+
109+ /*
110+ * Create a buffer object and transfer ownership to a userspace-
111+ * accessible handle.
112+ */
113+ pvr_obj = pvr_gem_object_create(pvr_dev, sanitized_size, args->flags);
114+ if (IS_ERR(pvr_obj)) {
115+ err = PTR_ERR(pvr_obj);
116+ goto err_drm_dev_exit;
117+ }
118+
119+ /* This function will not modify &args->handle unless it succeeds. */
120+ err = pvr_gem_object_into_handle(pvr_obj, pvr_file, &args->handle);
121+ if (err)
122+ goto err_destroy_obj;
123+
124+ drm_dev_exit(idx);
125+
126+ return 0;
127+
128+err_destroy_obj:
129+ /*
130+ * GEM objects are refcounted, so there is no explicit destructor
131+ * function. Instead, we release the singular reference we currently
132+ * hold on the object and let GEM take care of the rest.
133+ */
134+ pvr_gem_object_put(pvr_obj);
135+
136+err_drm_dev_exit:
137+ drm_dev_exit(idx);
138+
139+ return err;
140+}
141+
142+/**
143+ * pvr_ioctl_get_bo_mmap_offset() - IOCTL to generate a "fake" offset to be
144+ * used when calling mmap() from userspace to map the given GEM buffer object
145+ * @drm_dev: [IN] DRM device (unused).
146+ * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
147+ * &struct drm_pvr_ioctl_get_bo_mmap_offset_args.
148+ * @file: [IN] DRM file private data.
149+ *
150+ * Called from userspace with %DRM_IOCTL_PVR_GET_BO_MMAP_OFFSET.
151+ *
152+ * This IOCTL does *not* perform an mmap. See the docs on
153+ * &struct drm_pvr_ioctl_get_bo_mmap_offset_args for details.
154+ *
155+ * Return:
156+ * * 0 on success,
157+ * * -%ENOENT if the handle does not reference a valid GEM buffer object,
158+ * * -%EINVAL if any padding fields in &struct
159+ * drm_pvr_ioctl_get_bo_mmap_offset_args are not zero, or
160+ * * Any error returned by drm_gem_create_mmap_offset().
161+ */
162+static int
163+pvr_ioctl_get_bo_mmap_offset(struct drm_device *drm_dev, void *raw_args,
164+ struct drm_file *file)
165+{
166+ struct drm_pvr_ioctl_get_bo_mmap_offset_args *args = raw_args;
167+ struct pvr_file *pvr_file = to_pvr_file(file);
168+ struct pvr_gem_object *pvr_obj;
169+ struct drm_gem_object *gem_obj;
170+ int idx;
171+ int ret;
172+
173+ if (!drm_dev_enter(drm_dev, &idx))
174+ return -EIO;
175+
176+ /* All padding fields must be zeroed. */
177+ if (args->_padding_4 != 0) {
178+ ret = -EINVAL;
179+ goto err_drm_dev_exit;
180+ }
181+
182+ /*
183+ * Obtain a kernel reference to the buffer object. This reference is
184+ * counted and must be manually dropped before returning. If a buffer
185+ * object cannot be found for the specified handle, return -%ENOENT (No
186+ * such file or directory).
187+ */
188+ pvr_obj = pvr_gem_object_from_handle(pvr_file, args->handle);
189+ if (!pvr_obj) {
190+ ret = -ENOENT;
191+ goto err_drm_dev_exit;
192+ }
193+
194+ gem_obj = gem_from_pvr_gem(pvr_obj);
195+
196+ /*
197+ * Allocate a fake offset which can be used in userspace calls to mmap
198+ * on the DRM device file. If this fails, return the error code. This
199+ * operation is idempotent.
200+ */
201+ ret = drm_gem_create_mmap_offset(gem_obj);
202+ if (ret != 0) {
203+ /* Drop our reference to the buffer object. */
204+ drm_gem_object_put(gem_obj);
205+ goto err_drm_dev_exit;
206+ }
207+
208+ /*
209+ * Read out the fake offset allocated by the earlier call to
210+ * drm_gem_create_mmap_offset.
211+ */
212+ args->offset = drm_vma_node_offset_addr(&gem_obj->vma_node);
213+
214+ /* Drop our reference to the buffer object. */
215+ pvr_gem_object_put(pvr_obj);
216+
217+err_drm_dev_exit:
218+ drm_dev_exit(idx);
219+
220+ return ret;
221+}
222+
223+static __always_inline u64
224+pvr_fw_version_packed(u32 major, u32 minor)
225+{
226+ return ((u64)major << 32) | minor;
227+}
228+
229+static u32
230+rogue_get_common_store_partition_space_size(struct pvr_device *pvr_dev)
231+{
232+ u32 max_partitions = 0;
233+ u32 tile_size_x = 0;
234+ u32 tile_size_y = 0;
235+
236+ PVR_FEATURE_VALUE(pvr_dev, tile_size_x, &tile_size_x);
237+ PVR_FEATURE_VALUE(pvr_dev, tile_size_y, &tile_size_y);
238+ PVR_FEATURE_VALUE(pvr_dev, max_partitions, &max_partitions);
239+
240+ if (tile_size_x == 16 && tile_size_y == 16) {
241+ u32 usc_min_output_registers_per_pix = 0;
242+
243+ PVR_FEATURE_VALUE(pvr_dev, usc_min_output_registers_per_pix,
244+ &usc_min_output_registers_per_pix);
245+
246+ return tile_size_x * tile_size_y * max_partitions *
247+ usc_min_output_registers_per_pix;
248+ }
249+
250+ return max_partitions * 1024;
251+}
252+
253+static u32
254+rogue_get_common_store_alloc_region_size(struct pvr_device *pvr_dev)
255+{
256+ u32 common_store_size_in_dwords = 512 * 4 * 4;
257+ u32 alloc_region_size;
258+
259+ PVR_FEATURE_VALUE(pvr_dev, common_store_size_in_dwords, &common_store_size_in_dwords);
260+
261+ alloc_region_size = common_store_size_in_dwords - (256U * 4U) -
262+ rogue_get_common_store_partition_space_size(pvr_dev);
263+
264+ if (PVR_HAS_QUIRK(pvr_dev, 44079)) {
265+ u32 common_store_split_point = (768U * 4U * 4U);
266+
267+ return min(common_store_split_point - (256U * 4U), alloc_region_size);
268+ }
269+
270+ return alloc_region_size;
271+}
272+
273+static inline u32
274+rogue_get_num_phantoms(struct pvr_device *pvr_dev)
275+{
276+ u32 num_clusters = 1;
277+
278+ PVR_FEATURE_VALUE(pvr_dev, num_clusters, &num_clusters);
279+
280+ return ROGUE_REQ_NUM_PHANTOMS(num_clusters);
281+}
282+
283+static inline u32
284+rogue_get_max_coeffs(struct pvr_device *pvr_dev)
285+{
286+ u32 max_coeff_additional_portion = ROGUE_MAX_VERTEX_SHARED_REGISTERS;
287+ u32 pending_allocation_shared_regs = 2U * 1024U;
288+ u32 pending_allocation_coeff_regs = 0U;
289+ u32 num_phantoms = rogue_get_num_phantoms(pvr_dev);
290+ u32 tiles_in_flight = 0;
291+ u32 max_coeff_pixel_portion;
292+
293+ PVR_FEATURE_VALUE(pvr_dev, isp_max_tiles_in_flight, &tiles_in_flight);
294+ max_coeff_pixel_portion = DIV_ROUND_UP(tiles_in_flight, num_phantoms);
295+ max_coeff_pixel_portion *= ROGUE_MAX_PIXEL_SHARED_REGISTERS;
296+
297+ /*
298+ * Compute tasks on cores with BRN48492 and without compute overlap may lock
299+ * up without two additional lines of coeffs.
300+ */
301+ if (PVR_HAS_QUIRK(pvr_dev, 48492) && !PVR_HAS_FEATURE(pvr_dev, compute_overlap))
302+ pending_allocation_coeff_regs = 2U * 1024U;
303+
304+ if (PVR_HAS_ENHANCEMENT(pvr_dev, 38748))
305+ pending_allocation_shared_regs = 0;
306+
307+ if (PVR_HAS_ENHANCEMENT(pvr_dev, 38020))
308+ max_coeff_additional_portion += ROGUE_MAX_COMPUTE_SHARED_REGISTERS;
309+
310+ return rogue_get_common_store_alloc_region_size(pvr_dev) + pending_allocation_coeff_regs -
311+ (max_coeff_pixel_portion + max_coeff_additional_portion +
312+ pending_allocation_shared_regs);
313+}
314+
315+static inline u32
316+rogue_get_cdm_max_local_mem_size_regs(struct pvr_device *pvr_dev)
317+{
318+ u32 available_coeffs_in_dwords = rogue_get_max_coeffs(pvr_dev);
319+
320+ if (PVR_HAS_QUIRK(pvr_dev, 48492) && PVR_HAS_FEATURE(pvr_dev, roguexe) &&
321+ !PVR_HAS_FEATURE(pvr_dev, compute_overlap)) {
322+ /* Driver must not use the 2 reserved lines. */
323+ available_coeffs_in_dwords -= ROGUE_CSRM_LINE_SIZE_IN_DWORDS * 2;
324+ }
325+
326+ /*
327+ * The maximum amount of local memory available to a kernel is the minimum
328+ * of the total number of coefficient registers available and the max common
329+ * store allocation size which can be made by the CDM.
330+ *
331+ * If any coeff lines are reserved for tessellation or pixel then we need to
332+ * subtract those too.
333+ */
334+ return min(available_coeffs_in_dwords, (u32)ROGUE_MAX_PER_KERNEL_LOCAL_MEM_SIZE_REGS);
335+}
336+
337+/**
338+ * pvr_dev_query_gpu_info_get()
339+ * @pvr_dev: Device pointer.
340+ * @args: [IN] Device query arguments containing a pointer to a userspace
341+ * struct drm_pvr_dev_query_gpu_info.
342+ *
343+ * If the query object pointer is NULL, the size field is updated with the
344+ * expected size of the query object.
345+ *
346+ * Returns:
347+ * * 0 on success, or if size is requested using a NULL pointer, or
348+ * * -%E2BIG if the indicated length of the allocation is less than is
349+ * required to contain the copied data, or
350+ * * -%EFAULT if local memory could not be copied to userspace.
351+ */
352+static int
353+pvr_dev_query_gpu_info_get(struct pvr_device *pvr_dev,
354+ struct drm_pvr_ioctl_dev_query_args *args)
355+{
356+ struct drm_pvr_dev_query_gpu_info gpu_info = {0};
357+ int err;
358+
359+ if (!args->pointer) {
360+ args->size = sizeof(struct drm_pvr_dev_query_gpu_info);
361+ return 0;
362+ }
363+
364+ gpu_info.gpu_id =
365+ pvr_gpu_id_to_packed_bvnc(&pvr_dev->gpu_id);
366+ gpu_info.num_phantoms = rogue_get_num_phantoms(pvr_dev);
367+
368+ err = PVR_UOBJ_SET(args->pointer, args->size, gpu_info);
369+ if (err < 0)
370+ return err;
371+
372+ if (args->size > sizeof(gpu_info))
373+ args->size = sizeof(gpu_info);
374+ return 0;
375+}
376+
377+/**
378+ * pvr_dev_query_runtime_info_get()
379+ * @pvr_dev: Device pointer.
380+ * @args: [IN] Device query arguments containing a pointer to a userspace
381+ * struct drm_pvr_dev_query_runtime_info.
382+ *
383+ * If the query object pointer is NULL, the size field is updated with the
384+ * expected size of the query object.
385+ *
386+ * Returns:
387+ * * 0 on success, or if size is requested using a NULL pointer, or
388+ * * -%E2BIG if the indicated length of the allocation is less than is
389+ * required to contain the copied data, or
390+ * * -%EFAULT if local memory could not be copied to userspace.
391+ */
392+static int
393+pvr_dev_query_runtime_info_get(struct pvr_device *pvr_dev,
394+ struct drm_pvr_ioctl_dev_query_args *args)
395+{
396+ struct drm_pvr_dev_query_runtime_info runtime_info = {0};
397+ int err;
398+
399+ if (!args->pointer) {
400+ args->size = sizeof(struct drm_pvr_dev_query_runtime_info);
401+ return 0;
402+ }
403+
404+ runtime_info.free_list_min_pages =
405+ pvr_get_free_list_min_pages(pvr_dev);
406+ runtime_info.free_list_max_pages =
407+ ROGUE_PM_MAX_FREELIST_SIZE / ROGUE_PM_PAGE_SIZE;
408+ runtime_info.common_store_alloc_region_size =
409+ rogue_get_common_store_alloc_region_size(pvr_dev);
410+ runtime_info.common_store_partition_space_size =
411+ rogue_get_common_store_partition_space_size(pvr_dev);
412+ runtime_info.max_coeffs = rogue_get_max_coeffs(pvr_dev);
413+ runtime_info.cdm_max_local_mem_size_regs =
414+ rogue_get_cdm_max_local_mem_size_regs(pvr_dev);
415+
416+ err = PVR_UOBJ_SET(args->pointer, args->size, runtime_info);
417+ if (err < 0)
418+ return err;
419+
420+ if (args->size > sizeof(runtime_info))
421+ args->size = sizeof(runtime_info);
422+ return 0;
423+}
424+
425+/**
426+ * pvr_dev_query_quirks_get() - Unpack array of quirks at the address given
427+ * in a struct drm_pvr_dev_query_quirks, or gets the amount of space required
428+ * for it.
429+ * @pvr_dev: Device pointer.
430+ * @args: [IN] Device query arguments containing a pointer to a userspace
431+ * struct drm_pvr_dev_query_query_quirks.
432+ *
433+ * If the query object pointer is NULL, the size field is updated with the
434+ * expected size of the query object.
435+ * If the userspace pointer in the query object is NULL, or the count is
436+ * short, no data is copied.
437+ * The count field will be updated to that copied, or if either pointer is
438+ * NULL, that which would have been copied.
439+ * The size field in the query object will be updated to the size copied.
440+ *
441+ * Returns:
442+ * * 0 on success, or if size/count is requested using a NULL pointer, or
443+ * * -%EINVAL if args contained non-zero reserved fields, or
444+ * * -%E2BIG if the indicated length of the allocation is less than is
445+ * required to contain the copied data, or
446+ * * -%EFAULT if local memory could not be copied to userspace.
447+ */
448+static int
449+pvr_dev_query_quirks_get(struct pvr_device *pvr_dev,
450+ struct drm_pvr_ioctl_dev_query_args *args)
451+{
452+ /*
453+ * @FIXME - hardcoding of numbers here is intended as an
454+ * intermediate step so the UAPI can be fixed, but requires a
455+ * a refactor in the future to store them in a more appropriate
456+ * location
457+ */
458+ static const u32 umd_quirks_musthave[] = {
459+ 47217,
460+ 49927,
461+ 62269,
462+ };
463+ static const u32 umd_quirks[] = {
464+ 48545,
465+ 51764,
466+ };
467+ struct drm_pvr_dev_query_quirks query;
468+ u32 out[ARRAY_SIZE(umd_quirks_musthave) + ARRAY_SIZE(umd_quirks)];
469+ size_t out_musthave_count = 0;
470+ size_t out_count = 0;
471+ int err;
472+
473+ if (!args->pointer) {
474+ args->size = sizeof(struct drm_pvr_dev_query_quirks);
475+ return 0;
476+ }
477+
478+ err = PVR_UOBJ_GET(query, args->size, args->pointer);
479+
480+ if (err < 0)
481+ return err;
482+ if (query._padding_c)
483+ return -EINVAL;
484+
485+ for (int i = 0; i < ARRAY_SIZE(umd_quirks_musthave); i++) {
486+ if (pvr_device_has_uapi_quirk(pvr_dev, umd_quirks_musthave[i])) {
487+ out[out_count++] = umd_quirks_musthave[i];
488+ out_musthave_count++;
489+ }
490+ }
491+
492+ for (int i = 0; i < ARRAY_SIZE(umd_quirks); i++) {
493+ if (pvr_device_has_uapi_quirk(pvr_dev, umd_quirks[i]))
494+ out[out_count++] = umd_quirks[i];
495+ }
496+
497+ if (!query.quirks)
498+ goto copy_out;
499+ if (query.count < out_count)
500+ return -E2BIG;
501+
502+ if (copy_to_user(u64_to_user_ptr(query.quirks), out,
503+ out_count * sizeof(u32))) {
504+ return -EFAULT;
505+ }
506+
507+ query.musthave_count = out_musthave_count;
508+
509+copy_out:
510+ query.count = out_count;
511+ err = PVR_UOBJ_SET(args->pointer, args->size, query);
512+ if (err < 0)
513+ return err;
514+
515+ args->size = sizeof(query);
516+ return 0;
517+}
518+
519+/**
520+ * pvr_dev_query_enhancements_get() - Unpack array of enhancements at the
521+ * address given in a struct drm_pvr_dev_query_enhancements, or gets the amount
522+ * of space required for it.
523+ * @pvr_dev: Device pointer.
524+ * @args: [IN] Device query arguments containing a pointer to a userspace
525+ * struct drm_pvr_dev_query_enhancements.
526+ *
527+ * If the query object pointer is NULL, the size field is updated with the
528+ * expected size of the query object.
529+ * If the userspace pointer in the query object is NULL, or the count is
530+ * short, no data is copied.
531+ * The count field will be updated to that copied, or if either pointer is
532+ * NULL, that which would have been copied.
533+ * The size field in the query object will be updated to the size copied.
534+ *
535+ * Returns:
536+ * * 0 on success, or if size/count is requested using a NULL pointer, or
537+ * * -%EINVAL if args contained non-zero reserved fields, or
538+ * * -%E2BIG if the indicated length of the allocation is less than is
539+ * required to contain the copied data, or
540+ * * -%EFAULT if local memory could not be copied to userspace.
541+ */
542+static int
543+pvr_dev_query_enhancements_get(struct pvr_device *pvr_dev,
544+ struct drm_pvr_ioctl_dev_query_args *args)
545+{
546+ /*
547+ * @FIXME - hardcoding of numbers here is intended as an
548+ * intermediate step so the UAPI can be fixed, but requires a
549+ * a refactor in the future to store them in a more appropriate
550+ * location
551+ */
552+ const u32 umd_enhancements[] = {
553+ 35421,
554+ 42064,
555+ };
556+ struct drm_pvr_dev_query_enhancements query;
557+ u32 out[ARRAY_SIZE(umd_enhancements)];
558+ size_t out_idx = 0;
559+ int err;
560+
561+ if (!args->pointer) {
562+ args->size = sizeof(struct drm_pvr_dev_query_enhancements);
563+ return 0;
564+ }
565+
566+ err = PVR_UOBJ_GET(query, args->size, args->pointer);
567+
568+ if (err < 0)
569+ return err;
570+ if (query._padding_a)
571+ return -EINVAL;
572+ if (query._padding_c)
573+ return -EINVAL;
574+
575+ for (int i = 0; i < ARRAY_SIZE(umd_enhancements); i++) {
576+ if (pvr_device_has_uapi_enhancement(pvr_dev, umd_enhancements[i]))
577+ out[out_idx++] = umd_enhancements[i];
578+ }
579+
580+ if (!query.enhancements)
581+ goto copy_out;
582+ if (query.count < out_idx)
583+ return -E2BIG;
584+
585+ if (copy_to_user(u64_to_user_ptr(query.enhancements), out,
586+ out_idx * sizeof(u32))) {
587+ return -EFAULT;
588+ }
589+
590+copy_out:
591+ query.count = out_idx;
592+ err = PVR_UOBJ_SET(args->pointer, args->size, query);
593+ if (err < 0)
594+ return err;
595+
596+ args->size = sizeof(query);
597+ return 0;
598+}
599+
600+/**
601+ * pvr_ioctl_dev_query() - IOCTL to copy information about a device
602+ * @drm_dev: [IN] DRM device.
603+ * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
604+ * &struct drm_pvr_ioctl_dev_query_args.
605+ * @file: [IN] DRM file private data.
606+ *
607+ * Called from userspace with %DRM_IOCTL_PVR_DEV_QUERY.
608+ * If the given receiving struct pointer is NULL, or the indicated size is too
609+ * small, the expected size of the struct type will be returned in the size
610+ * argument field.
611+ *
612+ * Return:
613+ * * 0 on success or when fetching the size with args->pointer == NULL, or
614+ * * -%E2BIG if the indicated size of the receiving struct is less than is
615+ * required to contain the copied data, or
616+ * * -%EINVAL if the indicated struct type is unknown, or
617+ * * -%ENOMEM if local memory could not be allocated, or
618+ * * -%EFAULT if local memory could not be copied to userspace.
619+ */
620+static int
621+pvr_ioctl_dev_query(struct drm_device *drm_dev, void *raw_args,
622+ struct drm_file *file)
623+{
624+ struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
625+ struct drm_pvr_ioctl_dev_query_args *args = raw_args;
626+ int idx;
627+ int ret = -EINVAL;
628+
629+ if (!drm_dev_enter(drm_dev, &idx))
630+ return -EIO;
631+
632+ switch ((enum drm_pvr_dev_query)args->type) {
633+ case DRM_PVR_DEV_QUERY_GPU_INFO_GET:
634+ ret = pvr_dev_query_gpu_info_get(pvr_dev, args);
635+ break;
636+
637+ case DRM_PVR_DEV_QUERY_RUNTIME_INFO_GET:
638+ ret = pvr_dev_query_runtime_info_get(pvr_dev, args);
639+ break;
640+
641+ case DRM_PVR_DEV_QUERY_QUIRKS_GET:
642+ ret = pvr_dev_query_quirks_get(pvr_dev, args);
643+ break;
644+
645+ case DRM_PVR_DEV_QUERY_ENHANCEMENTS_GET:
646+ ret = pvr_dev_query_enhancements_get(pvr_dev, args);
647+ break;
648+
649+ case DRM_PVR_DEV_QUERY_HEAP_INFO_GET:
650+ ret = pvr_heap_info_get(pvr_dev, args);
651+ break;
652+
653+ case DRM_PVR_DEV_QUERY_STATIC_DATA_AREAS_GET:
654+ ret = pvr_static_data_areas_get(pvr_dev, args);
655+ break;
656+ }
657+
658+ drm_dev_exit(idx);
659+
660+ return ret;
661+}
662+
663+/**
664+ * pvr_ioctl_create_context() - IOCTL to create a context
665+ * @drm_dev: [IN] DRM device.
666+ * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
667+ * &struct drm_pvr_ioctl_create_context_args.
668+ * @file: [IN] DRM file private data.
669+ *
670+ * Called from userspace with %DRM_IOCTL_PVR_CREATE_CONTEXT.
671+ *
672+ * Return:
673+ * * 0 on success, or
674+ * * -%EINVAL if provided arguments are invalid, or
675+ * * -%EFAULT if arguments can't be copied from userspace, or
676+ * * Any error returned by pvr_create_render_context().
677+ */
678+static int
679+pvr_ioctl_create_context(struct drm_device *drm_dev, void *raw_args,
680+ struct drm_file *file)
681+{
682+ struct drm_pvr_ioctl_create_context_args *args = raw_args;
683+ struct pvr_file *pvr_file = file->driver_priv;
684+ int idx;
685+ int ret;
686+
687+ if (!drm_dev_enter(drm_dev, &idx))
688+ return -EIO;
689+
690+ ret = pvr_context_create(pvr_file, args);
691+
692+ drm_dev_exit(idx);
693+
694+ return ret;
695+}
696+
697+/**
698+ * pvr_ioctl_destroy_context() - IOCTL to destroy a context
699+ * @drm_dev: [IN] DRM device.
700+ * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
701+ * &struct drm_pvr_ioctl_destroy_context_args.
702+ * @file: [IN] DRM file private data.
703+ *
704+ * Called from userspace with %DRM_IOCTL_PVR_DESTROY_CONTEXT.
705+ *
706+ * Return:
707+ * * 0 on success, or
708+ * * -%EINVAL if context not in context list.
709+ */
710+static int
711+pvr_ioctl_destroy_context(struct drm_device *drm_dev, void *raw_args,
712+ struct drm_file *file)
713+{
714+ struct drm_pvr_ioctl_destroy_context_args *args = raw_args;
715+ struct pvr_file *pvr_file = file->driver_priv;
716+
717+ if (args->_padding_4)
718+ return -EINVAL;
719+
720+ return pvr_context_destroy(pvr_file, args->handle);
721+}
722+
723+/**
724+ * pvr_ioctl_create_free_list() - IOCTL to create a free list
725+ * @drm_dev: [IN] DRM device.
726+ * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
727+ * &struct drm_pvr_ioctl_create_free_list_args.
728+ * @file: [IN] DRM file private data.
729+ *
730+ * Called from userspace with %DRM_IOCTL_PVR_CREATE_FREE_LIST.
731+ *
732+ * Return:
733+ * * 0 on success, or
734+ * * Any error returned by pvr_free_list_create().
735+ */
736+static int
737+pvr_ioctl_create_free_list(struct drm_device *drm_dev, void *raw_args,
738+ struct drm_file *file)
739+{
740+ struct drm_pvr_ioctl_create_free_list_args *args = raw_args;
741+ struct pvr_file *pvr_file = to_pvr_file(file);
742+ struct pvr_free_list *free_list;
743+ int idx;
744+ int err;
745+
746+ if (!drm_dev_enter(drm_dev, &idx))
747+ return -EIO;
748+
749+ free_list = pvr_free_list_create(pvr_file, args);
750+ if (IS_ERR(free_list)) {
751+ err = PTR_ERR(free_list);
752+ goto err_drm_dev_exit;
753+ }
754+
755+ /* Allocate object handle for userspace. */
756+ err = xa_alloc(&pvr_file->free_list_handles,
757+ &args->handle,
758+ free_list,
759+ xa_limit_32b,
760+ GFP_KERNEL);
761+ if (err < 0)
762+ goto err_cleanup;
763+
764+ drm_dev_exit(idx);
765+
766+ return 0;
767+
768+err_cleanup:
769+ pvr_free_list_put(free_list);
770+
771+err_drm_dev_exit:
772+ drm_dev_exit(idx);
773+
774+ return err;
775+}
776+
777+/**
778+ * pvr_ioctl_destroy_free_list() - IOCTL to destroy a free list
779+ * @drm_dev: [IN] DRM device.
780+ * @raw_args: [IN] Arguments passed to this IOCTL. This must be of type
781+ * &struct drm_pvr_ioctl_destroy_free_list_args.
782+ * @file: [IN] DRM file private data.
783+ *
784+ * Called from userspace with %DRM_IOCTL_PVR_DESTROY_FREE_LIST.
785+ *
786+ * Return:
787+ * * 0 on success, or
788+ * * -%EINVAL if free list not in object list.
789+ */
790+static int
791+pvr_ioctl_destroy_free_list(struct drm_device *drm_dev, void *raw_args,
792+ struct drm_file *file)
793+{
794+ struct drm_pvr_ioctl_destroy_free_list_args *args = raw_args;
795+ struct pvr_file *pvr_file = to_pvr_file(file);
796+ struct pvr_free_list *free_list;
797+
798+ if (args->_padding_4)
799+ return -EINVAL;
800+
801+ free_list = xa_erase(&pvr_file->free_list_handles, args->handle);
802+ if (!free_list)
803+ return -EINVAL;
804+
805+ pvr_free_list_put(free_list);
806+ return 0;
807+}
808+
809+/**
810+ * pvr_ioctl_create_hwrt_dataset() - IOCTL to create a HWRT dataset
811+ * @drm_dev: [IN] DRM device.
812+ * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
813+ * &struct drm_pvr_ioctl_create_hwrt_dataset_args.
814+ * @file: [IN] DRM file private data.
815+ *
816+ * Called from userspace with %DRM_IOCTL_PVR_CREATE_HWRT_DATASET.
817+ *
818+ * Return:
819+ * * 0 on success, or
820+ * * Any error returned by pvr_hwrt_dataset_create().
821+ */
822+static int
823+pvr_ioctl_create_hwrt_dataset(struct drm_device *drm_dev, void *raw_args,
824+ struct drm_file *file)
825+{
826+ struct drm_pvr_ioctl_create_hwrt_dataset_args *args = raw_args;
827+ struct pvr_file *pvr_file = to_pvr_file(file);
828+ struct pvr_hwrt_dataset *hwrt;
829+ int idx;
830+ int err;
831+
832+ if (!drm_dev_enter(drm_dev, &idx))
833+ return -EIO;
834+
835+ hwrt = pvr_hwrt_dataset_create(pvr_file, args);
836+ if (IS_ERR(hwrt)) {
837+ err = PTR_ERR(hwrt);
838+ goto err_drm_dev_exit;
839+ }
840+
841+ /* Allocate object handle for userspace. */
842+ err = xa_alloc(&pvr_file->hwrt_handles,
843+ &args->handle,
844+ hwrt,
845+ xa_limit_32b,
846+ GFP_KERNEL);
847+ if (err < 0)
848+ goto err_cleanup;
849+
850+ drm_dev_exit(idx);
851+
852+ return 0;
853+
854+err_cleanup:
855+ pvr_hwrt_dataset_put(hwrt);
856+
857+err_drm_dev_exit:
858+ drm_dev_exit(idx);
859+
860+ return err;
861+}
862+
863+/**
864+ * pvr_ioctl_destroy_hwrt_dataset() - IOCTL to destroy a HWRT dataset
865+ * @drm_dev: [IN] DRM device.
866+ * @raw_args: [IN] Arguments passed to this IOCTL. This must be of type
867+ * &struct drm_pvr_ioctl_destroy_hwrt_dataset_args.
868+ * @file: [IN] DRM file private data.
869+ *
870+ * Called from userspace with %DRM_IOCTL_PVR_DESTROY_HWRT_DATASET.
871+ *
872+ * Return:
873+ * * 0 on success, or
874+ * * -%EINVAL if HWRT dataset not in object list.
875+ */
876+static int
877+pvr_ioctl_destroy_hwrt_dataset(struct drm_device *drm_dev, void *raw_args,
878+ struct drm_file *file)
879+{
880+ struct drm_pvr_ioctl_destroy_hwrt_dataset_args *args = raw_args;
881+ struct pvr_file *pvr_file = to_pvr_file(file);
882+ struct pvr_hwrt_dataset *hwrt;
883+
884+ if (args->_padding_4)
885+ return -EINVAL;
886+
887+ hwrt = xa_erase(&pvr_file->hwrt_handles, args->handle);
888+ if (!hwrt)
889+ return -EINVAL;
890+
891+ pvr_hwrt_dataset_put(hwrt);
892+ return 0;
893+}
894+
895+/**
896+ * pvr_ioctl_create_vm_context() - IOCTL to create a VM context
897+ * @drm_dev: [IN] DRM device.
898+ * @raw_args: [IN/OUT] Arguments passed to this IOCTL. This must be of type
899+ * &struct drm_pvr_ioctl_create_vm_context_args.
900+ * @file: [IN] DRM file private data.
901+ *
902+ * Called from userspace with %DRM_IOCTL_PVR_CREATE_VM_CONTEXT.
903+ *
904+ * Return:
905+ * * 0 on success, or
906+ * * Any error returned by pvr_vm_create_context().
907+ */
908+static int
909+pvr_ioctl_create_vm_context(struct drm_device *drm_dev, void *raw_args,
910+ struct drm_file *file)
911+{
912+ struct drm_pvr_ioctl_create_vm_context_args *args = raw_args;
913+ struct pvr_file *pvr_file = to_pvr_file(file);
914+ struct pvr_vm_context *vm_ctx;
915+ int idx;
916+ int err;
917+
918+ if (!drm_dev_enter(drm_dev, &idx))
919+ return -EIO;
920+
921+ if (args->_padding_4) {
922+ err = -EINVAL;
923+ goto err_drm_dev_exit;
924+ }
925+
926+ vm_ctx = pvr_vm_create_context(pvr_file->pvr_dev, true);
927+ if (IS_ERR(vm_ctx)) {
928+ err = PTR_ERR(vm_ctx);
929+ goto err_drm_dev_exit;
930+ }
931+
932+ /* Allocate object handle for userspace. */
933+ err = xa_alloc(&pvr_file->vm_ctx_handles,
934+ &args->handle,
935+ vm_ctx,
936+ xa_limit_32b,
937+ GFP_KERNEL);
938+ if (err < 0)
939+ goto err_cleanup;
940+
941+ drm_dev_exit(idx);
942+
943+ return 0;
944+
945+err_cleanup:
946+ pvr_vm_context_put(vm_ctx);
947+
948+err_drm_dev_exit:
949+ drm_dev_exit(idx);
950+
951+ return err;
952+}
953+
954+/**
955+ * pvr_ioctl_destroy_vm_context() - IOCTL to destroy a VM context
956+* @drm_dev: [IN] DRM device.
957+* @raw_args: [IN] Arguments passed to this IOCTL. This must be of type
958+* &struct drm_pvr_ioctl_destroy_vm_context_args.
959+* @file: [IN] DRM file private data.
960+*
961+* Called from userspace with %DRM_IOCTL_PVR_DESTROY_VM_CONTEXT.
962+*
963+* Return:
964+* * 0 on success, or
965+* * -%EINVAL if object not in object list.
966+ */
967+static int
968+pvr_ioctl_destroy_vm_context(struct drm_device *drm_dev, void *raw_args,
969+ struct drm_file *file)
970+{
971+ struct drm_pvr_ioctl_destroy_vm_context_args *args = raw_args;
972+ struct pvr_file *pvr_file = to_pvr_file(file);
973+ struct pvr_vm_context *vm_ctx;
974+
975+ if (args->_padding_4)
976+ return -EINVAL;
977+
978+ vm_ctx = xa_erase(&pvr_file->vm_ctx_handles, args->handle);
979+ if (!vm_ctx)
980+ return -EINVAL;
981+
982+ pvr_vm_context_put(vm_ctx);
983+ return 0;
984+}
985+
986+/**
987+ * pvr_ioctl_vm_map() - IOCTL to map buffer to GPU address space.
988+ * @drm_dev: [IN] DRM device.
989+ * @raw_args: [IN] Arguments passed to this IOCTL. This must be of type
990+ * &struct drm_pvr_ioctl_vm_map_args.
991+ * @file: [IN] DRM file private data.
992+ *
993+ * Called from userspace with %DRM_IOCTL_PVR_VM_MAP.
994+ *
995+ * Return:
996+ * * 0 on success,
997+ * * -%EINVAL if &drm_pvr_ioctl_vm_op_map_args.flags is not zero,
998+ * * -%EINVAL if the bounds specified by &drm_pvr_ioctl_vm_op_map_args.offset
999+ * and &drm_pvr_ioctl_vm_op_map_args.size are not valid or do not fall
1000+ * within the buffer object specified by
1001+ * &drm_pvr_ioctl_vm_op_map_args.handle,
1002+ * * -%EINVAL if the bounds specified by
1003+ * &drm_pvr_ioctl_vm_op_map_args.device_addr and
1004+ * &drm_pvr_ioctl_vm_op_map_args.size do not form a valid device-virtual
1005+ * address range which falls entirely within a single heap, or
1006+ * * -%ENOENT if &drm_pvr_ioctl_vm_op_map_args.handle does not refer to a
1007+ * valid PowerVR buffer object.
1008+ */
1009+static int
1010+pvr_ioctl_vm_map(struct drm_device *drm_dev, void *raw_args,
1011+ struct drm_file *file)
1012+{
1013+ struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
1014+ struct drm_pvr_ioctl_vm_map_args *args = raw_args;
1015+ struct pvr_file *pvr_file = to_pvr_file(file);
1016+ struct pvr_vm_context *vm_ctx;
1017+
1018+ struct pvr_gem_object *pvr_obj;
1019+ size_t pvr_obj_size;
1020+
1021+ u64 offset_plus_size;
1022+ int idx;
1023+ int err;
1024+
1025+ if (!drm_dev_enter(drm_dev, &idx))
1026+ return -EIO;
1027+
1028+ /* Initial validation of args. */
1029+ if (args->_padding_14) {
1030+ err = -EINVAL;
1031+ goto err_drm_dev_exit;
1032+ }
1033+
1034+ if (args->flags != 0 ||
1035+ check_add_overflow(args->offset, args->size, &offset_plus_size) ||
1036+ !pvr_find_heap_containing(pvr_dev, args->device_addr, args->size)) {
1037+ err = -EINVAL;
1038+ goto err_drm_dev_exit;
1039+ }
1040+
1041+ vm_ctx = pvr_vm_context_lookup(pvr_file, args->vm_context_handle);
1042+ if (!vm_ctx) {
1043+ err = -EINVAL;
1044+ goto err_drm_dev_exit;
1045+ }
1046+
1047+ pvr_obj = pvr_gem_object_from_handle(pvr_file, args->handle);
1048+ if (!pvr_obj) {
1049+ err = -ENOENT;
1050+ goto err_put_vm_context;
1051+ }
1052+
1053+ pvr_obj_size = pvr_gem_object_size(pvr_obj);
1054+
1055+ /*
1056+ * Validate offset and size args. The alignment of these will be
1057+ * checked when mapping; for now just check that they're within valid
1058+ * bounds
1059+ */
1060+ if (args->offset >= pvr_obj_size || offset_plus_size > pvr_obj_size) {
1061+ err = -EINVAL;
1062+ goto err_put_pvr_object;
1063+ }
1064+
1065+ err = pvr_vm_map(vm_ctx, pvr_obj, args->offset,
1066+ args->device_addr, args->size);
1067+ if (err)
1068+ goto err_put_pvr_object;
1069+
1070+ /*
1071+ * In order to set up the mapping, we needed a reference to &pvr_obj.
1072+ * However, pvr_vm_map() obtains and stores its own reference, so we
1073+ * must release ours before returning.
1074+ */
1075+
1076+err_put_pvr_object:
1077+ pvr_gem_object_put(pvr_obj);
1078+
1079+err_put_vm_context:
1080+ pvr_vm_context_put(vm_ctx);
1081+
1082+err_drm_dev_exit:
1083+ drm_dev_exit(idx);
1084+
1085+ return err;
1086+}
1087+
1088+/**
1089+ * pvr_ioctl_vm_unmap() - IOCTL to unmap buffer from GPU address space.
1090+ * @drm_dev: [IN] DRM device.
1091+ * @raw_args: [IN] Arguments passed to this IOCTL. This must be of type
1092+ * &struct drm_pvr_ioctl_vm_unmap_args.
1093+ * @file: [IN] DRM file private data.
1094+ *
1095+ * Called from userspace with %DRM_IOCTL_PVR_VM_UNMAP.
1096+ *
1097+ * Return:
1098+ * * 0 on success,
1099+ * * -%EINVAL if &drm_pvr_ioctl_vm_op_unmap_args.device_addr is not a valid
1100+ * device page-aligned device-virtual address, or
1101+ * * -%ENOENT if there is currently no PowerVR buffer object mapped at
1102+ * &drm_pvr_ioctl_vm_op_unmap_args.device_addr.
1103+ */
1104+static int
1105+pvr_ioctl_vm_unmap(struct drm_device *drm_dev, void *raw_args,
1106+ struct drm_file *file)
1107+{
1108+ struct drm_pvr_ioctl_vm_unmap_args *args = raw_args;
1109+ struct pvr_file *pvr_file = to_pvr_file(file);
1110+ struct pvr_vm_context *vm_ctx;
1111+ int err;
1112+
1113+ /* Initial validation of args. */
1114+ if (args->_padding_4)
1115+ return -EINVAL;
1116+
1117+ vm_ctx = pvr_vm_context_lookup(pvr_file, args->vm_context_handle);
1118+ if (!vm_ctx)
1119+ return -EINVAL;
1120+
1121+ err = pvr_vm_unmap(vm_ctx, args->device_addr, args->size);
1122+
1123+ pvr_vm_context_put(vm_ctx);
1124+
1125+ return err;
1126+}
1127+
1128+/*
1129+ * pvr_ioctl_submit_job() - IOCTL to submit a job to the GPU
1130+ * @drm_dev: [IN] DRM device.
1131+ * @raw_args: [IN] Arguments passed to this IOCTL. This must be of type
1132+ * &struct drm_pvr_ioctl_submit_job_args.
1133+ * @file: [IN] DRM file private data.
1134+ *
1135+ * Called from userspace with %DRM_IOCTL_PVR_SUBMIT_JOB.
1136+ *
1137+ * Return:
1138+ * * 0 on success, or
1139+ * * -%EINVAL if arguments are invalid.
1140+ */
1141+static int
1142+pvr_ioctl_submit_jobs(struct drm_device *drm_dev, void *raw_args,
1143+ struct drm_file *file)
1144+{
1145+ struct drm_pvr_ioctl_submit_jobs_args *args = raw_args;
1146+ struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
1147+ struct pvr_file *pvr_file = to_pvr_file(file);
1148+ int idx;
1149+ int err;
1150+
1151+ if (!drm_dev_enter(drm_dev, &idx))
1152+ return -EIO;
1153+
1154+ err = pvr_submit_jobs(pvr_dev, pvr_file, args);
1155+
1156+ drm_dev_exit(idx);
1157+
1158+ return err;
1159+}
1160+
1161+int
1162+pvr_get_uobj(u64 usr_ptr, u32 usr_stride, u32 min_stride, u32 obj_size, void *out)
1163+{
1164+ if (usr_stride < min_stride)
1165+ return -EINVAL;
1166+
1167+ return copy_struct_from_user(out, obj_size, u64_to_user_ptr(usr_ptr), usr_stride);
1168+}
1169+
1170+int
1171+pvr_set_uobj(u64 usr_ptr, u32 usr_stride, u32 min_stride, u32 obj_size, const void *in)
1172+{
1173+ if (usr_stride < min_stride)
1174+ return -EINVAL;
1175+
1176+ if (copy_to_user(u64_to_user_ptr(usr_ptr), in, min_t(u32, usr_stride, obj_size)))
1177+ return -EFAULT;
1178+
1179+ if (usr_stride > obj_size &&
1180+ clear_user(u64_to_user_ptr(usr_ptr + obj_size), usr_stride - obj_size)) {
1181+ return -EFAULT;
1182+ }
1183+
1184+ return 0;
1185+}
1186+
1187+int
1188+pvr_get_uobj_array(const struct drm_pvr_obj_array *in, u32 min_stride, u32 obj_size, void **out)
1189+{
1190+ int ret = 0;
1191+ void *out_alloc;
1192+
1193+ if (in->stride < min_stride)
1194+ return -EINVAL;
1195+
1196+ if (!in->count)
1197+ return 0;
1198+
1199+ out_alloc = kvmalloc_array(in->count, obj_size, GFP_KERNEL);
1200+ if (!out_alloc)
1201+ return -ENOMEM;
1202+
1203+ if (obj_size == in->stride) {
1204+ if (copy_from_user(out_alloc, u64_to_user_ptr(in->array),
1205+ (unsigned long)obj_size * in->count))
1206+ ret = -EFAULT;
1207+ } else {
1208+ void __user *in_ptr = u64_to_user_ptr(in->array);
1209+ void *out_ptr = out_alloc;
1210+
1211+ for (u32 i = 0; i < in->count; i++) {
1212+ ret = copy_struct_from_user(out_ptr, obj_size, in_ptr, in->stride);
1213+ if (ret)
1214+ break;
1215+
1216+ out_ptr += obj_size;
1217+ in_ptr += in->stride;
1218+ }
1219+ }
1220+
1221+ if (ret) {
1222+ kvfree(out_alloc);
1223+ return ret;
1224+ }
1225+
1226+ *out = out_alloc;
1227+ return 0;
1228+}
1229+
1230+int
1231+pvr_set_uobj_array(const struct drm_pvr_obj_array *out, u32 min_stride, u32 obj_size,
1232+ const void *in)
1233+{
1234+ if (out->stride < min_stride)
1235+ return -EINVAL;
1236+
1237+ if (!out->count)
1238+ return 0;
1239+
1240+ if (obj_size == out->stride) {
1241+ if (copy_to_user(u64_to_user_ptr(out->array), in,
1242+ (unsigned long)obj_size * out->count))
1243+ return -EFAULT;
1244+ } else {
1245+ u32 cpy_elem_size = min_t(u32, out->stride, obj_size);
1246+ void __user *out_ptr = u64_to_user_ptr(out->array);
1247+ const void *in_ptr = in;
1248+
1249+ for (u32 i = 0; i < out->count; i++) {
1250+ if (copy_to_user(out_ptr, in_ptr, cpy_elem_size))
1251+ return -EFAULT;
1252+
1253+ out_ptr += obj_size;
1254+ in_ptr += out->stride;
1255+ }
1256+
1257+ if (out->stride > obj_size &&
1258+ clear_user(u64_to_user_ptr(out->array + obj_size),
1259+ out->stride - obj_size)) {
1260+ return -EFAULT;
1261+ }
1262+ }
1263+
1264+ return 0;
1265+}
1266+
1267+#define DRM_PVR_IOCTL(_name, _func, _flags) \
1268+ DRM_IOCTL_DEF_DRV(PVR_##_name, pvr_ioctl_##_func, _flags)
1269+
1270+/* clang-format off */
1271+
1272+static const struct drm_ioctl_desc pvr_drm_driver_ioctls[] = {
1273+ DRM_PVR_IOCTL(DEV_QUERY, dev_query, DRM_RENDER_ALLOW),
1274+ DRM_PVR_IOCTL(CREATE_BO, create_bo, DRM_RENDER_ALLOW),
1275+ DRM_PVR_IOCTL(GET_BO_MMAP_OFFSET, get_bo_mmap_offset, DRM_RENDER_ALLOW),
1276+ DRM_PVR_IOCTL(CREATE_VM_CONTEXT, create_vm_context, DRM_RENDER_ALLOW),
1277+ DRM_PVR_IOCTL(DESTROY_VM_CONTEXT, destroy_vm_context, DRM_RENDER_ALLOW),
1278+ DRM_PVR_IOCTL(VM_MAP, vm_map, DRM_RENDER_ALLOW),
1279+ DRM_PVR_IOCTL(VM_UNMAP, vm_unmap, DRM_RENDER_ALLOW),
1280+ DRM_PVR_IOCTL(CREATE_CONTEXT, create_context, DRM_RENDER_ALLOW),
1281+ DRM_PVR_IOCTL(DESTROY_CONTEXT, destroy_context, DRM_RENDER_ALLOW),
1282+ DRM_PVR_IOCTL(CREATE_FREE_LIST, create_free_list, DRM_RENDER_ALLOW),
1283+ DRM_PVR_IOCTL(DESTROY_FREE_LIST, destroy_free_list, DRM_RENDER_ALLOW),
1284+ DRM_PVR_IOCTL(CREATE_HWRT_DATASET, create_hwrt_dataset, DRM_RENDER_ALLOW),
1285+ DRM_PVR_IOCTL(DESTROY_HWRT_DATASET, destroy_hwrt_dataset, DRM_RENDER_ALLOW),
1286+ DRM_PVR_IOCTL(SUBMIT_JOBS, submit_jobs, DRM_RENDER_ALLOW),
1287+};
1288+
1289+/* clang-format on */
1290+
1291+#undef DRM_PVR_IOCTL
1292+
1293+/**
1294+ * pvr_drm_driver_open() - Driver callback when a new &struct drm_file is opened
1295+ * @drm_dev: [IN] DRM device.
1296+ * @file: [IN] DRM file private data.
1297+ *
1298+ * Allocates powervr-specific file private data (&struct pvr_file).
1299+ *
1300+ * Registered in &pvr_drm_driver.
1301+ *
1302+ * Return:
1303+ * * 0 on success,
1304+ * * -%ENOMEM if the allocation of a &struct ipvr_file fails, or
1305+ * * Any error returned by pvr_memory_context_init().
1306+ */
1307+static int
1308+pvr_drm_driver_open(struct drm_device *drm_dev, struct drm_file *file)
1309+{
1310+ struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
1311+ struct pvr_file *pvr_file;
1312+
1313+ pvr_file = kzalloc(sizeof(*pvr_file), GFP_KERNEL);
1314+ if (!pvr_file)
1315+ return -ENOMEM;
1316+
1317+ /*
1318+ * Store reference to base DRM file private data for use by
1319+ * from_pvr_file.
1320+ */
1321+ pvr_file->file = file;
1322+
1323+ /*
1324+ * Store reference to powervr-specific outer device struct in file
1325+ * private data for convenient access.
1326+ */
1327+ pvr_file->pvr_dev = pvr_dev;
1328+
1329+ xa_init_flags(&pvr_file->ctx_handles, XA_FLAGS_ALLOC1);
1330+ xa_init_flags(&pvr_file->free_list_handles, XA_FLAGS_ALLOC1);
1331+ xa_init_flags(&pvr_file->hwrt_handles, XA_FLAGS_ALLOC1);
1332+ xa_init_flags(&pvr_file->vm_ctx_handles, XA_FLAGS_ALLOC1);
1333+
1334+ /*
1335+ * Store reference to powervr-specific file private data in DRM file
1336+ * private data.
1337+ */
1338+ file->driver_priv = pvr_file;
1339+
1340+ return 0;
1341+}
1342+
1343+/**
1344+ * pvr_drm_driver_postclose() - One of the driver callbacks when a &struct
1345+ * drm_file is closed.
1346+ * @drm_dev: [IN] DRM device (unused).
1347+ * @file: [IN] DRM file private data.
1348+ *
1349+ * Frees powervr-specific file private data (&struct pvr_file).
1350+ *
1351+ * Registered in &pvr_drm_driver.
1352+ */
1353+static void
1354+pvr_drm_driver_postclose(__always_unused struct drm_device *drm_dev,
1355+ struct drm_file *file)
1356+{
1357+ struct pvr_file *pvr_file = to_pvr_file(file);
1358+
1359+ /* Kill remaining contexts. */
1360+ pvr_destroy_contexts_for_file(pvr_file);
1361+
1362+ /* Drop references on any remaining objects. */
1363+ pvr_destroy_free_lists_for_file(pvr_file);
1364+ pvr_destroy_hwrt_datasets_for_file(pvr_file);
1365+ pvr_destroy_vm_contexts_for_file(pvr_file);
1366+
1367+ kfree(pvr_file);
1368+ file->driver_priv = NULL;
1369+}
1370+
1371+DEFINE_DRM_GEM_FOPS(pvr_drm_driver_fops);
1372+
1373+static struct drm_driver pvr_drm_driver = {
1374+ .driver_features = DRIVER_GEM | DRIVER_GEM_GPUVA | DRIVER_RENDER |
1375+ DRIVER_SYNCOBJ | DRIVER_SYNCOBJ_TIMELINE,
1376+ .open = pvr_drm_driver_open,
1377+ .postclose = pvr_drm_driver_postclose,
1378+ .ioctls = pvr_drm_driver_ioctls,
1379+ .num_ioctls = ARRAY_SIZE(pvr_drm_driver_ioctls),
1380+ .fops = &pvr_drm_driver_fops,
1381+#if defined(CONFIG_DEBUG_FS)
1382+ .debugfs_init = pvr_debugfs_init,
1383+#endif
1384+
1385+ .name = PVR_DRIVER_NAME,
1386+ .desc = PVR_DRIVER_DESC,
1387+ .date = PVR_DRIVER_DATE,
1388+ .major = PVR_DRIVER_MAJOR,
1389+ .minor = PVR_DRIVER_MINOR,
1390+ .patchlevel = PVR_DRIVER_PATCHLEVEL,
1391+
1392+ .gem_prime_import_sg_table = drm_gem_shmem_prime_import_sg_table,
1393+ .gem_create_object = pvr_gem_create_object,
1394+};
1395+
1396+static int
1397+pvr_probe(struct platform_device *plat_dev)
1398+{
1399+ struct pvr_device *pvr_dev;
1400+ struct drm_device *drm_dev;
1401+ int err;
1402+
1403+ pvr_dev = devm_drm_dev_alloc(&plat_dev->dev, &pvr_drm_driver,
1404+ struct pvr_device, base);
1405+ if (IS_ERR(pvr_dev))
1406+ return PTR_ERR(pvr_dev);
1407+
1408+ drm_dev = &pvr_dev->base;
1409+
1410+ platform_set_drvdata(plat_dev, drm_dev);
1411+
1412+ init_rwsem(&pvr_dev->reset_sem);
1413+
1414+ pvr_context_device_init(pvr_dev);
1415+
1416+ err = pvr_queue_device_init(pvr_dev);
1417+ if (err)
1418+ goto err_context_fini;
1419+
1420+ devm_pm_runtime_enable(&plat_dev->dev);
1421+ pm_runtime_mark_last_busy(&plat_dev->dev);
1422+
1423+ pm_runtime_set_autosuspend_delay(&plat_dev->dev, 50);
1424+ pm_runtime_use_autosuspend(&plat_dev->dev);
1425+ pvr_watchdog_init(pvr_dev);
1426+
1427+ err = pvr_device_init(pvr_dev);
1428+ if (err)
1429+ goto err_watchdog_fini;
1430+
1431+ err = drm_dev_register(drm_dev, 0);
1432+ if (err)
1433+ goto err_device_fini;
1434+
1435+ xa_init_flags(&pvr_dev->free_list_ids, XA_FLAGS_ALLOC1);
1436+ xa_init_flags(&pvr_dev->job_ids, XA_FLAGS_ALLOC1);
1437+
1438+ return 0;
1439+
1440+err_device_fini:
1441+ pvr_device_fini(pvr_dev);
1442+
1443+err_watchdog_fini:
1444+ pvr_watchdog_fini(pvr_dev);
1445+
1446+ pvr_queue_device_fini(pvr_dev);
1447+
1448+err_context_fini:
1449+ pvr_context_device_fini(pvr_dev);
1450+
1451+ return err;
1452+}
1453+
1454+static int
1455+pvr_remove(struct platform_device *plat_dev)
1456+{
1457+ struct drm_device *drm_dev = platform_get_drvdata(plat_dev);
1458+ struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
1459+
1460+ WARN_ON(!xa_empty(&pvr_dev->job_ids));
1461+ WARN_ON(!xa_empty(&pvr_dev->free_list_ids));
1462+
1463+ xa_destroy(&pvr_dev->job_ids);
1464+ xa_destroy(&pvr_dev->free_list_ids);
1465+
1466+ pm_runtime_suspend(drm_dev->dev);
1467+ pvr_device_fini(pvr_dev);
1468+ drm_dev_unplug(drm_dev);
1469+ pvr_watchdog_fini(pvr_dev);
1470+ pvr_queue_device_fini(pvr_dev);
1471+ pvr_context_device_fini(pvr_dev);
1472+
1473+ return 0;
1474+}
1475+
1476+static const struct of_device_id dt_match[] = {
1477+ { .compatible = "img,img-axe", .data = NULL },
1478+ {}
1479+};
1480+MODULE_DEVICE_TABLE(of, dt_match);
1481+
1482+static const struct dev_pm_ops pvr_pm_ops = {
1483+ RUNTIME_PM_OPS(pvr_power_device_suspend, pvr_power_device_resume, pvr_power_device_idle)
1484+};
1485+
1486+static struct platform_driver pvr_driver = {
1487+ .probe = pvr_probe,
1488+ .remove = pvr_remove,
1489+ .driver = {
1490+ .name = PVR_DRIVER_NAME,
1491+ .pm = &pvr_pm_ops,
1492+ .of_match_table = dt_match,
1493+ },
1494+};
1495+module_platform_driver(pvr_driver);
1496+
1497+MODULE_AUTHOR("Imagination Technologies Ltd.");
1498+MODULE_DESCRIPTION(PVR_DRIVER_DESC);
1499+MODULE_LICENSE("Dual MIT/GPL");
1500+MODULE_IMPORT_NS(DMA_BUF);
1501+MODULE_FIRMWARE("powervr/rogue_33.15.11.3_v1.fw");
drivers/gpu/drm/imagination/pvr_drv.h+129−0
@@ -0,0 +1,129 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_DRV_H
5+#define PVR_DRV_H
6+
7+#include "linux/compiler_attributes.h"
8+#include <uapi/drm/pvr_drm.h>
9+
10+#define PVR_DRIVER_NAME "powervr"
11+#define PVR_DRIVER_DESC "Imagination PowerVR (Series 6 and later) & IMG Graphics"
12+#define PVR_DRIVER_DATE "20230904"
13+
14+/*
15+ * Driver interface version:
16+ * - 1.0: Initial interface
17+ */
18+#define PVR_DRIVER_MAJOR 1
19+#define PVR_DRIVER_MINOR 0
20+#define PVR_DRIVER_PATCHLEVEL 0
21+
22+int pvr_get_uobj(u64 usr_ptr, u32 usr_size, u32 min_size, u32 obj_size, void *out);
23+int pvr_set_uobj(u64 usr_ptr, u32 usr_size, u32 min_size, u32 obj_size, const void *in);
24+int pvr_get_uobj_array(const struct drm_pvr_obj_array *in, u32 min_stride, u32 obj_size,
25+ void **out);
26+int pvr_set_uobj_array(const struct drm_pvr_obj_array *out, u32 min_stride, u32 obj_size,
27+ const void *in);
28+
29+#define PVR_UOBJ_MIN_SIZE_INTERNAL(_typename, _last_mandatory_field) \
30+ (offsetof(_typename, _last_mandatory_field) + \
31+ sizeof(((_typename *)NULL)->_last_mandatory_field))
32+
33+/* NOLINTBEGIN(bugprone-macro-parentheses) */
34+#define PVR_UOBJ_DECL(_typename, _last_mandatory_field) \
35+ , _typename : PVR_UOBJ_MIN_SIZE_INTERNAL(_typename, _last_mandatory_field)
36+/* NOLINTEND(bugprone-macro-parentheses) */
37+
38+/**
39+ * DOC: PVR user objects.
40+ *
41+ * Macros used to aid copying structured and array data to and from
42+ * userspace. Objects can differ in size, provided the minimum size
43+ * allowed is specified (using the last mandatory field in the struct).
44+ * All types used with PVR_UOBJ_GET/SET macros must be listed here under
45+ * PVR_UOBJ_MIN_SIZE, with the last mandatory struct field specified.
46+ */
47+
48+/**
49+ * PVR_UOBJ_MIN_SIZE() - Fetch the minimum copy size of a compatible type object.
50+ * @_obj_name: The name of the object. Cannot be a typename - this is deduced.
51+ *
52+ * This cannot fail. Using the macro with an incompatible type will result in a
53+ * compiler error.
54+ *
55+ * To add compatibility for a type, list it within the macro in an orderly
56+ * fashion. The second argument is the name of the last mandatory field of the
57+ * struct type, which is used to calculate the size. See also PVR_UOBJ_DECL().
58+ *
59+ * Return: The minimum copy size.
60+ */
61+#define PVR_UOBJ_MIN_SIZE(_obj_name) _Generic(_obj_name \
62+ PVR_UOBJ_DECL(struct drm_pvr_job, hwrt) \
63+ PVR_UOBJ_DECL(struct drm_pvr_sync_op, value) \
64+ PVR_UOBJ_DECL(struct drm_pvr_dev_query_gpu_info, num_phantoms) \
65+ PVR_UOBJ_DECL(struct drm_pvr_dev_query_runtime_info, cdm_max_local_mem_size_regs) \
66+ PVR_UOBJ_DECL(struct drm_pvr_dev_query_quirks, _padding_c) \
67+ PVR_UOBJ_DECL(struct drm_pvr_dev_query_enhancements, _padding_c) \
68+ PVR_UOBJ_DECL(struct drm_pvr_heap, page_size_log2) \
69+ PVR_UOBJ_DECL(struct drm_pvr_dev_query_heap_info, heaps) \
70+ PVR_UOBJ_DECL(struct drm_pvr_static_data_area, offset) \
71+ PVR_UOBJ_DECL(struct drm_pvr_dev_query_static_data_areas, static_data_areas) \
72+ )
73+
74+/**
75+ * PVR_UOBJ_GET() - Copies from _src_usr_ptr to &_dest_obj.
76+ * @_dest_obj: The destination container object in kernel space.
77+ * @_usr_size: The size of the source container in user space.
78+ * @_src_usr_ptr: __u64 raw pointer to the source container in user space.
79+ *
80+ * Return: Error code. See pvr_get_uobj().
81+ */
82+#define PVR_UOBJ_GET(_dest_obj, _usr_size, _src_usr_ptr) \
83+ pvr_get_uobj(_src_usr_ptr, _usr_size, \
84+ PVR_UOBJ_MIN_SIZE(_dest_obj), \
85+ sizeof(_dest_obj), &(_dest_obj))
86+
87+/**
88+ * PVR_UOBJ_SET() - Copies from &_src_obj to _dest_usr_ptr.
89+ * @_dest_usr_ptr: __u64 raw pointer to the destination container in user space.
90+ * @_usr_size: The size of the destination container in user space.
91+ * @_src_obj: The source container object in kernel space.
92+ *
93+ * Return: Error code. See pvr_set_uobj().
94+ */
95+#define PVR_UOBJ_SET(_dest_usr_ptr, _usr_size, _src_obj) \
96+ pvr_set_uobj(_dest_usr_ptr, _usr_size, \
97+ PVR_UOBJ_MIN_SIZE(_src_obj), \
98+ sizeof(_src_obj), &(_src_obj))
99+
100+/**
101+ * PVR_UOBJ_GET_ARRAY() - Copies from @_src_drm_pvr_obj_array.array to
102+ * alloced memory and returns a pointer in _dest_array.
103+ * @_dest_array: The destination C array object in kernel space.
104+ * @_src_drm_pvr_obj_array: The &struct drm_pvr_obj_array containing a __u64 raw
105+ * pointer to the source C array in user space and the size of each array
106+ * element in user space (the 'stride').
107+ *
108+ * Return: Error code. See pvr_get_uobj_array().
109+ */
110+#define PVR_UOBJ_GET_ARRAY(_dest_array, _src_drm_pvr_obj_array) \
111+ pvr_get_uobj_array(_src_drm_pvr_obj_array, \
112+ PVR_UOBJ_MIN_SIZE((_dest_array)[0]), \
113+ sizeof((_dest_array)[0]), (void **)&(_dest_array))
114+
115+/**
116+ * PVR_UOBJ_SET_ARRAY() - Copies from _src_array to @_dest_drm_pvr_obj_array.array.
117+ * @_dest_drm_pvr_obj_array: The &struct drm_pvr_obj_array containing a __u64 raw
118+ * pointer to the destination C array in user space and the size of each array
119+ * element in user space (the 'stride').
120+ * @_src_array: The source C array object in kernel space.
121+ *
122+ * Return: Error code. See pvr_set_uobj_array().
123+ */
124+#define PVR_UOBJ_SET_ARRAY(_dest_drm_pvr_obj_array, _src_array) \
125+ pvr_set_uobj_array(_dest_drm_pvr_obj_array, \
126+ PVR_UOBJ_MIN_SIZE((_src_array)[0]), \
127+ sizeof((_src_array)[0]), _src_array)
128+
129+#endif /* PVR_DRV_H */
drivers/gpu/drm/imagination/pvr_free_list.c+625−0
@@ -0,0 +1,625 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_free_list.h"
5+#include "pvr_gem.h"
6+#include "pvr_hwrt.h"
7+#include "pvr_rogue_fwif.h"
8+#include "pvr_vm.h"
9+
10+#include <drm/drm_gem.h>
11+#include <linux/slab.h>
12+#include <linux/xarray.h>
13+#include <uapi/drm/pvr_drm.h>
14+
15+#define FREE_LIST_ENTRY_SIZE sizeof(u32)
16+
17+#define FREE_LIST_ALIGNMENT \
18+ ((ROGUE_BIF_PM_FREELIST_BASE_ADDR_ALIGNSIZE / FREE_LIST_ENTRY_SIZE) - 1)
19+
20+#define FREE_LIST_MIN_PAGES 50
21+#define FREE_LIST_MIN_PAGES_BRN66011 40
22+#define FREE_LIST_MIN_PAGES_ROGUEXE 25
23+
24+/**
25+ * pvr_get_free_list_min_pages() - Get minimum free list size for this device
26+ * @pvr_dev: Device pointer.
27+ *
28+ * Returns:
29+ * * Minimum free list size, in PM physical pages.
30+ */
31+u32
32+pvr_get_free_list_min_pages(struct pvr_device *pvr_dev)
33+{
34+ u32 value;
35+
36+ if (PVR_HAS_FEATURE(pvr_dev, roguexe)) {
37+ if (PVR_HAS_QUIRK(pvr_dev, 66011))
38+ value = FREE_LIST_MIN_PAGES_BRN66011;
39+ else
40+ value = FREE_LIST_MIN_PAGES_ROGUEXE;
41+ } else {
42+ value = FREE_LIST_MIN_PAGES;
43+ }
44+
45+ return value;
46+}
47+
48+static int
49+free_list_create_kernel_structure(struct pvr_file *pvr_file,
50+ struct drm_pvr_ioctl_create_free_list_args *args,
51+ struct pvr_free_list *free_list)
52+{
53+ struct pvr_gem_object *free_list_obj;
54+ struct pvr_vm_context *vm_ctx;
55+ u64 free_list_size;
56+ int err;
57+
58+ if (args->grow_threshold > 100 ||
59+ args->initial_num_pages > args->max_num_pages ||
60+ args->grow_num_pages > args->max_num_pages ||
61+ args->max_num_pages == 0 ||
62+ (args->initial_num_pages < args->max_num_pages && !args->grow_num_pages) ||
63+ (args->initial_num_pages == args->max_num_pages && args->grow_num_pages))
64+ return -EINVAL;
65+
66+ if ((args->initial_num_pages & FREE_LIST_ALIGNMENT) ||
67+ (args->max_num_pages & FREE_LIST_ALIGNMENT) ||
68+ (args->grow_num_pages & FREE_LIST_ALIGNMENT))
69+ return -EINVAL;
70+
71+ vm_ctx = pvr_vm_context_lookup(pvr_file, args->vm_context_handle);
72+ if (!vm_ctx)
73+ return -EINVAL;
74+
75+ free_list_obj = pvr_vm_find_gem_object(vm_ctx, args->free_list_gpu_addr,
76+ NULL, &free_list_size);
77+ if (!free_list_obj) {
78+ err = -EINVAL;
79+ goto err_put_vm_context;
80+ }
81+
82+ if ((free_list_obj->flags & DRM_PVR_BO_ALLOW_CPU_USERSPACE_ACCESS) ||
83+ !(free_list_obj->flags & DRM_PVR_BO_PM_FW_PROTECT) ||
84+ free_list_size < (args->max_num_pages * FREE_LIST_ENTRY_SIZE)) {
85+ err = -EINVAL;
86+ goto err_put_free_list_obj;
87+ }
88+
89+ free_list->pvr_dev = pvr_file->pvr_dev;
90+ free_list->current_pages = 0;
91+ free_list->max_pages = args->max_num_pages;
92+ free_list->grow_pages = args->grow_num_pages;
93+ free_list->grow_threshold = args->grow_threshold;
94+ free_list->obj = free_list_obj;
95+ free_list->free_list_gpu_addr = args->free_list_gpu_addr;
96+ free_list->initial_num_pages = args->initial_num_pages;
97+
98+ pvr_vm_context_put(vm_ctx);
99+
100+ return 0;
101+
102+err_put_free_list_obj:
103+ pvr_gem_object_put(free_list_obj);
104+
105+err_put_vm_context:
106+ pvr_vm_context_put(vm_ctx);
107+
108+ return err;
109+}
110+
111+static void
112+free_list_destroy_kernel_structure(struct pvr_free_list *free_list)
113+{
114+ WARN_ON(!list_empty(&free_list->hwrt_list));
115+
116+ pvr_gem_object_put(free_list->obj);
117+}
118+
119+/**
120+ * calculate_free_list_ready_pages_locked() - Function to work out the number of free
121+ * list pages to reserve for growing within
122+ * the FW without having to wait for the
123+ * host to progress a grow request
124+ * @free_list: Pointer to free list.
125+ * @pages: Total pages currently in free list.
126+ *
127+ * If the threshold or grow size means less than the alignment size (4 pages on
128+ * Rogue), then the feature is not used.
129+ *
130+ * Caller must hold &free_list->lock.
131+ *
132+ * Return: number of pages to reserve.
133+ */
134+static u32
135+calculate_free_list_ready_pages_locked(struct pvr_free_list *free_list, u32 pages)
136+{
137+ u32 ready_pages;
138+
139+ lockdep_assert_held(&free_list->lock);
140+
141+ ready_pages = ((pages * free_list->grow_threshold) / 100);
142+
143+ /* The number of pages must be less than the grow size. */
144+ ready_pages = min(ready_pages, free_list->grow_pages);
145+
146+ /*
147+ * The number of pages must be a multiple of the free list align size.
148+ */
149+ ready_pages &= ~FREE_LIST_ALIGNMENT;
150+
151+ return ready_pages;
152+}
153+
154+static u32
155+calculate_free_list_ready_pages(struct pvr_free_list *free_list, u32 pages)
156+{
157+ u32 ret;
158+
159+ mutex_lock(&free_list->lock);
160+
161+ ret = calculate_free_list_ready_pages_locked(free_list, pages);
162+
163+ mutex_unlock(&free_list->lock);
164+
165+ return ret;
166+}
167+
168+static void
169+free_list_fw_init(void *cpu_ptr, void *priv)
170+{
171+ struct rogue_fwif_freelist *fw_data = cpu_ptr;
172+ struct pvr_free_list *free_list = priv;
173+ u32 ready_pages;
174+
175+ /* Fill out FW structure */
176+ ready_pages = calculate_free_list_ready_pages(free_list,
177+ free_list->initial_num_pages);
178+
179+ fw_data->max_pages = free_list->max_pages;
180+ fw_data->current_pages = free_list->initial_num_pages - ready_pages;
181+ fw_data->grow_pages = free_list->grow_pages;
182+ fw_data->ready_pages = ready_pages;
183+ fw_data->freelist_id = free_list->fw_id;
184+ fw_data->grow_pending = false;
185+ fw_data->current_stack_top = fw_data->current_pages - 1;
186+ fw_data->freelist_dev_addr = free_list->free_list_gpu_addr;
187+ fw_data->current_dev_addr = (fw_data->freelist_dev_addr +
188+ ((fw_data->max_pages - fw_data->current_pages) *
189+ FREE_LIST_ENTRY_SIZE)) &
190+ ~((u64)ROGUE_BIF_PM_FREELIST_BASE_ADDR_ALIGNSIZE - 1);
191+}
192+
193+static int
194+free_list_create_fw_structure(struct pvr_file *pvr_file,
195+ struct drm_pvr_ioctl_create_free_list_args *args,
196+ struct pvr_free_list *free_list)
197+{
198+ struct pvr_device *pvr_dev = pvr_file->pvr_dev;
199+
200+ /*
201+ * Create and map the FW structure so we can initialise it. This is not
202+ * accessed on the CPU side post-initialisation so the mapping lifetime
203+ * is only for this function.
204+ */
205+ free_list->fw_data = pvr_fw_object_create_and_map(pvr_dev, sizeof(*free_list->fw_data),
206+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
207+ free_list_fw_init, free_list,
208+ &free_list->fw_obj);
209+ if (IS_ERR(free_list->fw_data))
210+ return PTR_ERR(free_list->fw_data);
211+
212+ return 0;
213+}
214+
215+static void
216+free_list_destroy_fw_structure(struct pvr_free_list *free_list)
217+{
218+ pvr_fw_object_unmap_and_destroy(free_list->fw_obj);
219+}
220+
221+static int
222+pvr_free_list_insert_pages_locked(struct pvr_free_list *free_list,
223+ struct sg_table *sgt, u32 offset, u32 num_pages)
224+{
225+ struct sg_dma_page_iter dma_iter;
226+ u32 *page_list;
227+
228+ lockdep_assert_held(&free_list->lock);
229+
230+ page_list = pvr_gem_object_vmap(free_list->obj);
231+ if (IS_ERR(page_list))
232+ return PTR_ERR(page_list);
233+
234+ offset /= FREE_LIST_ENTRY_SIZE;
235+ /* clang-format off */
236+ for_each_sgtable_dma_page(sgt, &dma_iter, 0) {
237+ dma_addr_t dma_addr = sg_page_iter_dma_address(&dma_iter);
238+ u64 dma_pfn = dma_addr >>
239+ ROGUE_BIF_PM_PHYSICAL_PAGE_ALIGNSHIFT;
240+ u32 dma_addr_offset;
241+
242+ BUILD_BUG_ON(ROGUE_BIF_PM_PHYSICAL_PAGE_SIZE > PAGE_SIZE);
243+
244+ for (dma_addr_offset = 0; dma_addr_offset < PAGE_SIZE;
245+ dma_addr_offset += ROGUE_BIF_PM_PHYSICAL_PAGE_SIZE) {
246+ WARN_ON_ONCE(dma_pfn >> 32);
247+
248+ page_list[offset++] = (u32)dma_pfn;
249+ dma_pfn++;
250+
251+ num_pages--;
252+ if (!num_pages)
253+ break;
254+ }
255+
256+ if (!num_pages)
257+ break;
258+ };
259+ /* clang-format on */
260+
261+ /* Make sure our free_list update is flushed. */
262+ wmb();
263+
264+ pvr_gem_object_vunmap(free_list->obj);
265+
266+ return 0;
267+}
268+
269+static int
270+pvr_free_list_insert_node_locked(struct pvr_free_list_node *free_list_node)
271+{
272+ struct pvr_free_list *free_list = free_list_node->free_list;
273+ struct sg_table *sgt;
274+ u32 start_page;
275+ u32 offset;
276+ int err;
277+
278+ lockdep_assert_held(&free_list->lock);
279+
280+ start_page = free_list->max_pages - free_list->current_pages -
281+ free_list_node->num_pages;
282+ offset = (start_page * FREE_LIST_ENTRY_SIZE) &
283+ ~((u64)ROGUE_BIF_PM_FREELIST_BASE_ADDR_ALIGNSIZE - 1);
284+
285+ sgt = drm_gem_shmem_get_pages_sgt(&free_list_node->mem_obj->base);
286+ if (WARN_ON(IS_ERR(sgt)))
287+ return PTR_ERR(sgt);
288+
289+ err = pvr_free_list_insert_pages_locked(free_list, sgt,
290+ offset, free_list_node->num_pages);
291+ if (!err)
292+ free_list->current_pages += free_list_node->num_pages;
293+
294+ return err;
295+}
296+
297+static int
298+pvr_free_list_grow(struct pvr_free_list *free_list, u32 num_pages)
299+{
300+ struct pvr_device *pvr_dev = free_list->pvr_dev;
301+ struct pvr_free_list_node *free_list_node;
302+ int err;
303+
304+ mutex_lock(&free_list->lock);
305+
306+ if (num_pages & FREE_LIST_ALIGNMENT) {
307+ err = -EINVAL;
308+ goto err_unlock;
309+ }
310+
311+ free_list_node = kzalloc(sizeof(*free_list_node), GFP_KERNEL);
312+ if (!free_list_node) {
313+ err = -ENOMEM;
314+ goto err_unlock;
315+ }
316+
317+ free_list_node->num_pages = num_pages;
318+ free_list_node->free_list = free_list;
319+
320+ free_list_node->mem_obj = pvr_gem_object_create(pvr_dev,
321+ num_pages <<
322+ ROGUE_BIF_PM_PHYSICAL_PAGE_ALIGNSHIFT,
323+ PVR_BO_FW_FLAGS_DEVICE_CACHED);
324+ if (IS_ERR(free_list_node->mem_obj)) {
325+ err = PTR_ERR(free_list_node->mem_obj);
326+ goto err_free;
327+ }
328+
329+ err = pvr_free_list_insert_node_locked(free_list_node);
330+ if (err)
331+ goto err_destroy_gem_object;
332+
333+ list_add_tail(&free_list_node->node, &free_list->mem_block_list);
334+
335+ /*
336+ * Reserve a number ready pages to allow the FW to process OOM quickly
337+ * and asynchronously request a grow.
338+ */
339+ free_list->ready_pages =
340+ calculate_free_list_ready_pages_locked(free_list,
341+ free_list->current_pages);
342+ free_list->current_pages -= free_list->ready_pages;
343+
344+ mutex_unlock(&free_list->lock);
345+
346+ return 0;
347+
348+err_destroy_gem_object:
349+ pvr_gem_object_put(free_list_node->mem_obj);
350+
351+err_free:
352+ kfree(free_list_node);
353+
354+err_unlock:
355+ mutex_unlock(&free_list->lock);
356+
357+ return err;
358+}
359+
360+void pvr_free_list_process_grow_req(struct pvr_device *pvr_dev,
361+ struct rogue_fwif_fwccb_cmd_freelist_gs_data *req)
362+{
363+ struct pvr_free_list *free_list = pvr_free_list_lookup_id(pvr_dev, req->freelist_id);
364+ struct rogue_fwif_kccb_cmd resp_cmd = {
365+ .cmd_type = ROGUE_FWIF_KCCB_CMD_FREELIST_GROW_UPDATE,
366+ };
367+ struct rogue_fwif_freelist_gs_data *resp = &resp_cmd.cmd_data.free_list_gs_data;
368+ u32 grow_pages = 0;
369+
370+ /* If we don't have a freelist registered for this ID, we can't do much. */
371+ if (WARN_ON(!free_list))
372+ return;
373+
374+ /* Since the FW made the request, it has already consumed the ready pages,
375+ * update the host struct.
376+ */
377+ free_list->current_pages += free_list->ready_pages;
378+ free_list->ready_pages = 0;
379+
380+ /* If the grow succeeds, update the grow_pages argument. */
381+ if (!pvr_free_list_grow(free_list, free_list->grow_pages))
382+ grow_pages = free_list->grow_pages;
383+
384+ /* Now prepare the response and send it back to the FW. */
385+ pvr_fw_object_get_fw_addr(free_list->fw_obj, &resp->freelist_fw_addr);
386+ resp->delta_pages = grow_pages;
387+ resp->new_pages = free_list->current_pages + free_list->ready_pages;
388+ resp->ready_pages = free_list->ready_pages;
389+ pvr_free_list_put(free_list);
390+
391+ WARN_ON(pvr_kccb_send_cmd(pvr_dev, &resp_cmd, NULL));
392+}
393+
394+static void
395+pvr_free_list_free_node(struct pvr_free_list_node *free_list_node)
396+{
397+ pvr_gem_object_put(free_list_node->mem_obj);
398+
399+ kfree(free_list_node);
400+}
401+
402+/**
403+ * pvr_free_list_create() - Create a new free list and return an object pointer
404+ * @pvr_file: Pointer to pvr_file structure.
405+ * @args: Creation arguments from userspace.
406+ *
407+ * Return:
408+ * * Pointer to new free_list, or
409+ * * ERR_PTR(-%ENOMEM) on out of memory.
410+ */
411+struct pvr_free_list *
412+pvr_free_list_create(struct pvr_file *pvr_file,
413+ struct drm_pvr_ioctl_create_free_list_args *args)
414+{
415+ struct pvr_free_list *free_list;
416+ int err;
417+
418+ /* Create and fill out the kernel structure */
419+ free_list = kzalloc(sizeof(*free_list), GFP_KERNEL);
420+
421+ if (!free_list)
422+ return ERR_PTR(-ENOMEM);
423+
424+ kref_init(&free_list->ref_count);
425+ INIT_LIST_HEAD(&free_list->mem_block_list);
426+ INIT_LIST_HEAD(&free_list->hwrt_list);
427+ mutex_init(&free_list->lock);
428+
429+ err = free_list_create_kernel_structure(pvr_file, args, free_list);
430+ if (err < 0)
431+ goto err_free;
432+
433+ /* Allocate global object ID for firmware. */
434+ err = xa_alloc(&pvr_file->pvr_dev->free_list_ids,
435+ &free_list->fw_id,
436+ free_list,
437+ xa_limit_32b,
438+ GFP_KERNEL);
439+ if (err)
440+ goto err_destroy_kernel_structure;
441+
442+ err = free_list_create_fw_structure(pvr_file, args, free_list);
443+ if (err < 0)
444+ goto err_free_fw_id;
445+
446+ err = pvr_free_list_grow(free_list, args->initial_num_pages);
447+ if (err < 0)
448+ goto err_fw_struct_cleanup;
449+
450+ return free_list;
451+
452+err_fw_struct_cleanup:
453+ WARN_ON(pvr_fw_structure_cleanup(free_list->pvr_dev,
454+ ROGUE_FWIF_CLEANUP_FREELIST,
455+ free_list->fw_obj, 0));
456+
457+err_free_fw_id:
458+ xa_erase(&free_list->pvr_dev->free_list_ids, free_list->fw_id);
459+
460+err_destroy_kernel_structure:
461+ free_list_destroy_kernel_structure(free_list);
462+
463+err_free:
464+ mutex_destroy(&free_list->lock);
465+ kfree(free_list);
466+
467+ return ERR_PTR(err);
468+}
469+
470+static void
471+pvr_free_list_release(struct kref *ref_count)
472+{
473+ struct pvr_free_list *free_list =
474+ container_of(ref_count, struct pvr_free_list, ref_count);
475+ struct list_head *pos, *n;
476+ int err;
477+
478+ xa_erase(&free_list->pvr_dev->free_list_ids, free_list->fw_id);
479+
480+ err = pvr_fw_structure_cleanup(free_list->pvr_dev,
481+ ROGUE_FWIF_CLEANUP_FREELIST,
482+ free_list->fw_obj, 0);
483+ if (err == -EBUSY) {
484+ /* Flush the FWCCB to process any HWR or freelist reconstruction
485+ * request that might keep the freelist busy, and try again.
486+ */
487+ pvr_fwccb_process(free_list->pvr_dev);
488+ err = pvr_fw_structure_cleanup(free_list->pvr_dev,
489+ ROGUE_FWIF_CLEANUP_FREELIST,
490+ free_list->fw_obj, 0);
491+ }
492+
493+ WARN_ON(err);
494+
495+ /* clang-format off */
496+ list_for_each_safe(pos, n, &free_list->mem_block_list) {
497+ struct pvr_free_list_node *free_list_node =
498+ container_of(pos, struct pvr_free_list_node, node);
499+
500+ list_del(pos);
501+ pvr_free_list_free_node(free_list_node);
502+ }
503+ /* clang-format on */
504+
505+ free_list_destroy_kernel_structure(free_list);
506+ free_list_destroy_fw_structure(free_list);
507+ mutex_destroy(&free_list->lock);
508+ kfree(free_list);
509+}
510+
511+/**
512+ * pvr_destroy_free_lists_for_file: Destroy any free lists associated with the
513+ * given file.
514+ * @pvr_file: Pointer to pvr_file structure.
515+ *
516+ * Removes all free lists associated with @pvr_file from the device free_list
517+ * list and drops initial references. Free lists will then be destroyed once
518+ * all outstanding references are dropped.
519+ */
520+void pvr_destroy_free_lists_for_file(struct pvr_file *pvr_file)
521+{
522+ struct pvr_free_list *free_list;
523+ unsigned long handle;
524+
525+ xa_for_each(&pvr_file->free_list_handles, handle, free_list) {
526+ (void)free_list;
527+ pvr_free_list_put(xa_erase(&pvr_file->free_list_handles, handle));
528+ }
529+}
530+
531+/**
532+ * pvr_free_list_put() - Release reference on free list
533+ * @free_list: Pointer to list to release reference on
534+ */
535+void
536+pvr_free_list_put(struct pvr_free_list *free_list)
537+{
538+ if (free_list)
539+ kref_put(&free_list->ref_count, pvr_free_list_release);
540+}
541+
542+void pvr_free_list_add_hwrt(struct pvr_free_list *free_list, struct pvr_hwrt_data *hwrt_data)
543+{
544+ mutex_lock(&free_list->lock);
545+
546+ list_add_tail(&hwrt_data->freelist_node, &free_list->hwrt_list);
547+
548+ mutex_unlock(&free_list->lock);
549+}
550+
551+void pvr_free_list_remove_hwrt(struct pvr_free_list *free_list, struct pvr_hwrt_data *hwrt_data)
552+{
553+ mutex_lock(&free_list->lock);
554+
555+ list_del(&hwrt_data->freelist_node);
556+
557+ mutex_unlock(&free_list->lock);
558+}
559+
560+static void
561+pvr_free_list_reconstruct(struct pvr_device *pvr_dev, u32 freelist_id)
562+{
563+ struct pvr_free_list *free_list = pvr_free_list_lookup_id(pvr_dev, freelist_id);
564+ struct pvr_free_list_node *free_list_node;
565+ struct rogue_fwif_freelist *fw_data;
566+ struct pvr_hwrt_data *hwrt_data;
567+
568+ if (!free_list)
569+ return;
570+
571+ mutex_lock(&free_list->lock);
572+
573+ /* Rebuild the free list based on the memory block list. */
574+ free_list->current_pages = 0;
575+
576+ list_for_each_entry(free_list_node, &free_list->mem_block_list, node)
577+ WARN_ON(pvr_free_list_insert_node_locked(free_list_node));
578+
579+ /*
580+ * Remove the ready pages, which are reserved to allow the FW to process OOM quickly and
581+ * asynchronously request a grow.
582+ */
583+ free_list->current_pages -= free_list->ready_pages;
584+
585+ fw_data = free_list->fw_data;
586+ fw_data->current_stack_top = fw_data->current_pages - 1;
587+ fw_data->allocated_page_count = 0;
588+ fw_data->allocated_mmu_page_count = 0;
589+
590+ /* Reset the state of any associated HWRTs. */
591+ list_for_each_entry(hwrt_data, &free_list->hwrt_list, freelist_node) {
592+ struct rogue_fwif_hwrtdata *hwrt_fw_data = pvr_fw_object_vmap(hwrt_data->fw_obj);
593+
594+ if (!WARN_ON(IS_ERR(hwrt_fw_data))) {
595+ hwrt_fw_data->state = ROGUE_FWIF_RTDATA_STATE_HWR;
596+ hwrt_fw_data->hwrt_data_flags &= ~HWRTDATA_HAS_LAST_GEOM;
597+ }
598+
599+ pvr_fw_object_vunmap(hwrt_data->fw_obj);
600+ }
601+
602+ mutex_unlock(&free_list->lock);
603+
604+ pvr_free_list_put(free_list);
605+}
606+
607+void
608+pvr_free_list_process_reconstruct_req(struct pvr_device *pvr_dev,
609+ struct rogue_fwif_fwccb_cmd_freelists_reconstruction_data *req)
610+{
611+ struct rogue_fwif_kccb_cmd resp_cmd = {
612+ .cmd_type = ROGUE_FWIF_KCCB_CMD_FREELISTS_RECONSTRUCTION_UPDATE,
613+ };
614+ struct rogue_fwif_freelists_reconstruction_data *resp =
615+ &resp_cmd.cmd_data.free_lists_reconstruction_data;
616+
617+ for (u32 i = 0; i < req->freelist_count; i++)
618+ pvr_free_list_reconstruct(pvr_dev, req->freelist_ids[i]);
619+
620+ resp->freelist_count = req->freelist_count;
621+ memcpy(resp->freelist_ids, req->freelist_ids,
622+ req->freelist_count * sizeof(resp->freelist_ids[0]));
623+
624+ WARN_ON(pvr_kccb_send_cmd(pvr_dev, &resp_cmd, NULL));
625+}
drivers/gpu/drm/imagination/pvr_free_list.h+195−0
@@ -0,0 +1,195 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_FREE_LIST_H
5+#define PVR_FREE_LIST_H
6+
7+#include <linux/compiler_attributes.h>
8+#include <linux/kref.h>
9+#include <linux/list.h>
10+#include <linux/mutex.h>
11+#include <linux/types.h>
12+#include <linux/xarray.h>
13+#include <uapi/drm/pvr_drm.h>
14+
15+#include "pvr_device.h"
16+
17+/* Forward declaration from pvr_gem.h. */
18+struct pvr_fw_object;
19+
20+/* Forward declaration from pvr_gem.h. */
21+struct pvr_gem_object;
22+
23+/* Forward declaration from pvr_hwrt.h. */
24+struct pvr_hwrt_data;
25+
26+/**
27+ * struct pvr_free_list_node - structure representing an allocation in the free
28+ * list
29+ */
30+struct pvr_free_list_node {
31+ /** @node: List node for &pvr_free_list.mem_block_list. */
32+ struct list_head node;
33+
34+ /** @free_list: Pointer to owning free list. */
35+ struct pvr_free_list *free_list;
36+
37+ /** @num_pages: Number of pages in this node. */
38+ u32 num_pages;
39+
40+ /** @mem_obj: GEM object representing the pages in this node. */
41+ struct pvr_gem_object *mem_obj;
42+};
43+
44+/**
45+ * struct pvr_free_list - structure representing a free list
46+ */
47+struct pvr_free_list {
48+ /** @ref_count: Reference count of object. */
49+ struct kref ref_count;
50+
51+ /** @pvr_dev: Pointer to device that owns this object. */
52+ struct pvr_device *pvr_dev;
53+
54+ /** @obj: GEM object representing the free list. */
55+ struct pvr_gem_object *obj;
56+
57+ /** @fw_obj: FW object representing the FW-side structure. */
58+ struct pvr_fw_object *fw_obj;
59+
60+ /** @fw_data: Pointer to CPU mapping of the FW-side structure. */
61+ struct rogue_fwif_freelist *fw_data;
62+
63+ /**
64+ * @lock: Mutex protecting modification of the free list. Must be held when accessing any
65+ * of the members below.
66+ */
67+ struct mutex lock;
68+
69+ /** @fw_id: Firmware ID for this object. */
70+ u32 fw_id;
71+
72+ /** @current_pages: Current number of pages in free list. */
73+ u32 current_pages;
74+
75+ /** @max_pages: Maximum number of pages in free list. */
76+ u32 max_pages;
77+
78+ /** @grow_pages: Pages to grow free list by per request. */
79+ u32 grow_pages;
80+
81+ /**
82+ * @grow_threshold: Percentage of FL memory used that should trigger a
83+ * new grow request.
84+ */
85+ u32 grow_threshold;
86+
87+ /**
88+ * @ready_pages: Number of pages reserved for FW to use while a grow
89+ * request is being processed.
90+ */
91+ u32 ready_pages;
92+
93+ /** @mem_block_list: List of memory blocks in this free list. */
94+ struct list_head mem_block_list;
95+
96+ /** @hwrt_list: List of HWRTs using this free list. */
97+ struct list_head hwrt_list;
98+
99+ /** @initial_num_pages: Initial number of pages in free list. */
100+ u32 initial_num_pages;
101+
102+ /** @free_list_gpu_addr: Address of free list in GPU address space. */
103+ u64 free_list_gpu_addr;
104+};
105+
106+struct pvr_free_list *
107+pvr_free_list_create(struct pvr_file *pvr_file,
108+ struct drm_pvr_ioctl_create_free_list_args *args);
109+
110+void
111+pvr_destroy_free_lists_for_file(struct pvr_file *pvr_file);
112+
113+u32
114+pvr_get_free_list_min_pages(struct pvr_device *pvr_dev);
115+
116+static __always_inline struct pvr_free_list *
117+pvr_free_list_get(struct pvr_free_list *free_list)
118+{
119+ if (free_list)
120+ kref_get(&free_list->ref_count);
121+
122+ return free_list;
123+}
124+
125+/**
126+ * pvr_free_list_lookup() - Lookup free list pointer from handle and file
127+ * @pvr_file: Pointer to pvr_file structure.
128+ * @handle: Object handle.
129+ *
130+ * Takes reference on free list object. Call pvr_free_list_put() to release.
131+ *
132+ * Returns:
133+ * * The requested object on success, or
134+ * * %NULL on failure (object does not exist in list, is not a free list, or
135+ * does not belong to @pvr_file)
136+ */
137+static __always_inline struct pvr_free_list *
138+pvr_free_list_lookup(struct pvr_file *pvr_file, u32 handle)
139+{
140+ struct pvr_free_list *free_list;
141+
142+ xa_lock(&pvr_file->free_list_handles);
143+ free_list = pvr_free_list_get(xa_load(&pvr_file->free_list_handles, handle));
144+ xa_unlock(&pvr_file->free_list_handles);
145+
146+ return free_list;
147+}
148+
149+/**
150+ * pvr_free_list_lookup_id() - Lookup free list pointer from FW ID
151+ * @pvr_dev: Device pointer.
152+ * @id: FW object ID.
153+ *
154+ * Takes reference on free list object. Call pvr_free_list_put() to release.
155+ *
156+ * Returns:
157+ * * The requested object on success, or
158+ * * %NULL on failure (object does not exist in list, or is not a free list)
159+ */
160+static __always_inline struct pvr_free_list *
161+pvr_free_list_lookup_id(struct pvr_device *pvr_dev, u32 id)
162+{
163+ struct pvr_free_list *free_list;
164+
165+ xa_lock(&pvr_dev->free_list_ids);
166+
167+ /* Contexts are removed from the ctx_ids set in the context release path,
168+ * meaning the ref_count reached zero before they get removed. We need
169+ * to make sure we're not trying to acquire a context that's being
170+ * destroyed.
171+ */
172+ free_list = xa_load(&pvr_dev->free_list_ids, id);
173+ if (free_list && !kref_get_unless_zero(&free_list->ref_count))
174+ free_list = NULL;
175+ xa_unlock(&pvr_dev->free_list_ids);
176+
177+ return free_list;
178+}
179+
180+void
181+pvr_free_list_put(struct pvr_free_list *free_list);
182+
183+void
184+pvr_free_list_add_hwrt(struct pvr_free_list *free_list, struct pvr_hwrt_data *hwrt_data);
185+void
186+pvr_free_list_remove_hwrt(struct pvr_free_list *free_list, struct pvr_hwrt_data *hwrt_data);
187+
188+void pvr_free_list_process_grow_req(struct pvr_device *pvr_dev,
189+ struct rogue_fwif_fwccb_cmd_freelist_gs_data *req);
190+
191+void
192+pvr_free_list_process_reconstruct_req(struct pvr_device *pvr_dev,
193+ struct rogue_fwif_fwccb_cmd_freelists_reconstruction_data *req);
194+
195+#endif /* PVR_FREE_LIST_H */
drivers/gpu/drm/imagination/pvr_fw.c+1489−0
@@ -0,0 +1,1489 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_ccb.h"
5+#include "pvr_device.h"
6+#include "pvr_device_info.h"
7+#include "pvr_fw.h"
8+#include "pvr_fw_info.h"
9+#include "pvr_fw_startstop.h"
10+#include "pvr_fw_trace.h"
11+#include "pvr_gem.h"
12+#include "pvr_power.h"
13+#include "pvr_rogue_fwif_dev_info.h"
14+#include "pvr_rogue_heap_config.h"
15+#include "pvr_vm.h"
16+
17+#include <drm/drm_drv.h>
18+#include <drm/drm_managed.h>
19+#include <drm/drm_mm.h>
20+#include <linux/clk.h>
21+#include <linux/firmware.h>
22+#include <linux/math.h>
23+#include <linux/minmax.h>
24+#include <linux/sizes.h>
25+
26+#define FW_MAX_SUPPORTED_MAJOR_VERSION 1
27+
28+#define FW_BOOT_TIMEOUT_USEC 5000000
29+
30+/* Config heap occupies top 192k of the firmware heap. */
31+#define PVR_ROGUE_FW_CONFIG_HEAP_GRANULARITY SZ_64K
32+#define PVR_ROGUE_FW_CONFIG_HEAP_SIZE (3 * PVR_ROGUE_FW_CONFIG_HEAP_GRANULARITY)
33+
34+/* Main firmware allocations should come from the remainder of the heap. */
35+#define PVR_ROGUE_FW_MAIN_HEAP_BASE ROGUE_FW_HEAP_BASE
36+
37+/* Offsets from start of configuration area of FW heap. */
38+#define PVR_ROGUE_FWIF_CONNECTION_CTL_OFFSET 0
39+#define PVR_ROGUE_FWIF_OSINIT_OFFSET \
40+ (PVR_ROGUE_FWIF_CONNECTION_CTL_OFFSET + PVR_ROGUE_FW_CONFIG_HEAP_GRANULARITY)
41+#define PVR_ROGUE_FWIF_SYSINIT_OFFSET \
42+ (PVR_ROGUE_FWIF_OSINIT_OFFSET + PVR_ROGUE_FW_CONFIG_HEAP_GRANULARITY)
43+
44+#define PVR_ROGUE_FAULT_PAGE_SIZE SZ_4K
45+
46+#define PVR_SYNC_OBJ_SIZE sizeof(u32)
47+
48+const struct pvr_fw_layout_entry *
49+pvr_fw_find_layout_entry(struct pvr_device *pvr_dev, enum pvr_fw_section_id id)
50+{
51+ const struct pvr_fw_layout_entry *layout_entries = pvr_dev->fw_dev.layout_entries;
52+ u32 num_layout_entries = pvr_dev->fw_dev.header->layout_entry_num;
53+ u32 entry;
54+
55+ for (entry = 0; entry < num_layout_entries; entry++) {
56+ if (layout_entries[entry].id == id)
57+ return &layout_entries[entry];
58+ }
59+
60+ return NULL;
61+}
62+
63+static const struct pvr_fw_layout_entry *
64+pvr_fw_find_private_data(struct pvr_device *pvr_dev)
65+{
66+ const struct pvr_fw_layout_entry *layout_entries = pvr_dev->fw_dev.layout_entries;
67+ u32 num_layout_entries = pvr_dev->fw_dev.header->layout_entry_num;
68+ u32 entry;
69+
70+ for (entry = 0; entry < num_layout_entries; entry++) {
71+ if (layout_entries[entry].id == META_PRIVATE_DATA ||
72+ layout_entries[entry].id == MIPS_PRIVATE_DATA ||
73+ layout_entries[entry].id == RISCV_PRIVATE_DATA)
74+ return &layout_entries[entry];
75+ }
76+
77+ return NULL;
78+}
79+
80+#define DEV_INFO_MASK_SIZE(x) DIV_ROUND_UP(x, 64)
81+
82+/**
83+ * pvr_fw_validate() - Parse firmware header and check compatibility
84+ * @pvr_dev: Device pointer.
85+ *
86+ * Returns:
87+ * * 0 on success, or
88+ * * -EINVAL if firmware is incompatible.
89+ */
90+static int
91+pvr_fw_validate(struct pvr_device *pvr_dev)
92+{
93+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
94+ const struct firmware *firmware = pvr_dev->fw_dev.firmware;
95+ const struct pvr_fw_layout_entry *layout_entries;
96+ const struct pvr_fw_info_header *header;
97+ const u8 *fw = firmware->data;
98+ u32 fw_offset = firmware->size - SZ_4K;
99+ u32 layout_table_size;
100+ u32 entry;
101+
102+ if (firmware->size < SZ_4K || (firmware->size % FW_BLOCK_SIZE))
103+ return -EINVAL;
104+
105+ header = (const struct pvr_fw_info_header *)&fw[fw_offset];
106+
107+ if (header->info_version != PVR_FW_INFO_VERSION) {
108+ drm_err(drm_dev, "Unsupported fw info version %u\n",
109+ header->info_version);
110+ return -EINVAL;
111+ }
112+
113+ if (header->header_len != sizeof(struct pvr_fw_info_header) ||
114+ header->layout_entry_size != sizeof(struct pvr_fw_layout_entry) ||
115+ header->layout_entry_num > PVR_FW_INFO_MAX_NUM_ENTRIES) {
116+ drm_err(drm_dev, "FW info format mismatch\n");
117+ return -EINVAL;
118+ }
119+
120+ if (!(header->flags & PVR_FW_FLAGS_OPEN_SOURCE) ||
121+ header->fw_version_major > FW_MAX_SUPPORTED_MAJOR_VERSION ||
122+ header->fw_version_major == 0) {
123+ drm_err(drm_dev, "Unsupported FW version %u.%u (build: %u%s)\n",
124+ header->fw_version_major, header->fw_version_minor,
125+ header->fw_version_build,
126+ (header->flags & PVR_FW_FLAGS_OPEN_SOURCE) ? " OS" : "");
127+ return -EINVAL;
128+ }
129+
130+ if (pvr_gpu_id_to_packed_bvnc(&pvr_dev->gpu_id) != header->bvnc) {
131+ struct pvr_gpu_id fw_gpu_id;
132+
133+ packed_bvnc_to_pvr_gpu_id(header->bvnc, &fw_gpu_id);
134+ drm_err(drm_dev, "FW built for incorrect GPU ID %i.%i.%i.%i (expected %i.%i.%i.%i)\n",
135+ fw_gpu_id.b, fw_gpu_id.v, fw_gpu_id.n, fw_gpu_id.c,
136+ pvr_dev->gpu_id.b, pvr_dev->gpu_id.v, pvr_dev->gpu_id.n, pvr_dev->gpu_id.c);
137+ return -EINVAL;
138+ }
139+
140+ fw_offset += header->header_len;
141+ layout_table_size =
142+ header->layout_entry_size * header->layout_entry_num;
143+ if ((fw_offset + layout_table_size) > firmware->size)
144+ return -EINVAL;
145+
146+ layout_entries = (const struct pvr_fw_layout_entry *)&fw[fw_offset];
147+ for (entry = 0; entry < header->layout_entry_num; entry++) {
148+ u32 start_addr = layout_entries[entry].base_addr;
149+ u32 end_addr = start_addr + layout_entries[entry].alloc_size;
150+
151+ if (start_addr >= end_addr)
152+ return -EINVAL;
153+ }
154+
155+ fw_offset = (firmware->size - SZ_4K) - header->device_info_size;
156+
157+ drm_info(drm_dev, "FW version v%u.%u (build %u OS)\n", header->fw_version_major,
158+ header->fw_version_minor, header->fw_version_build);
159+
160+ pvr_dev->fw_version.major = header->fw_version_major;
161+ pvr_dev->fw_version.minor = header->fw_version_minor;
162+
163+ pvr_dev->fw_dev.header = header;
164+ pvr_dev->fw_dev.layout_entries = layout_entries;
165+
166+ return 0;
167+}
168+
169+static int
170+pvr_fw_get_device_info(struct pvr_device *pvr_dev)
171+{
172+ const struct firmware *firmware = pvr_dev->fw_dev.firmware;
173+ struct pvr_fw_device_info_header *header;
174+ const u8 *fw = firmware->data;
175+ const u64 *dev_info;
176+ u32 fw_offset;
177+
178+ fw_offset = (firmware->size - SZ_4K) - pvr_dev->fw_dev.header->device_info_size;
179+
180+ header = (struct pvr_fw_device_info_header *)&fw[fw_offset];
181+ dev_info = (u64 *)(header + 1);
182+
183+ pvr_device_info_set_quirks(pvr_dev, dev_info, header->brn_mask_size);
184+ dev_info += header->brn_mask_size;
185+
186+ pvr_device_info_set_enhancements(pvr_dev, dev_info, header->ern_mask_size);
187+ dev_info += header->ern_mask_size;
188+
189+ return pvr_device_info_set_features(pvr_dev, dev_info, header->feature_mask_size,
190+ header->feature_param_size);
191+}
192+
193+static void
194+layout_get_sizes(struct pvr_device *pvr_dev)
195+{
196+ const struct pvr_fw_layout_entry *layout_entries = pvr_dev->fw_dev.layout_entries;
197+ u32 num_layout_entries = pvr_dev->fw_dev.header->layout_entry_num;
198+ struct pvr_fw_mem *fw_mem = &pvr_dev->fw_dev.mem;
199+
200+ fw_mem->code_alloc_size = 0;
201+ fw_mem->data_alloc_size = 0;
202+ fw_mem->core_code_alloc_size = 0;
203+ fw_mem->core_data_alloc_size = 0;
204+
205+ /* Extract section sizes from FW layout table. */
206+ for (u32 entry = 0; entry < num_layout_entries; entry++) {
207+ switch (layout_entries[entry].type) {
208+ case FW_CODE:
209+ fw_mem->code_alloc_size += layout_entries[entry].alloc_size;
210+ break;
211+ case FW_DATA:
212+ fw_mem->data_alloc_size += layout_entries[entry].alloc_size;
213+ break;
214+ case FW_COREMEM_CODE:
215+ fw_mem->core_code_alloc_size +=
216+ layout_entries[entry].alloc_size;
217+ break;
218+ case FW_COREMEM_DATA:
219+ fw_mem->core_data_alloc_size +=
220+ layout_entries[entry].alloc_size;
221+ break;
222+ case NONE:
223+ break;
224+ }
225+ }
226+}
227+
228+int
229+pvr_fw_find_mmu_segment(struct pvr_device *pvr_dev, u32 addr, u32 size, void *fw_code_ptr,
230+ void *fw_data_ptr, void *fw_core_code_ptr, void *fw_core_data_ptr,
231+ void **host_addr_out)
232+{
233+ const struct pvr_fw_layout_entry *layout_entries = pvr_dev->fw_dev.layout_entries;
234+ u32 num_layout_entries = pvr_dev->fw_dev.header->layout_entry_num;
235+ u32 end_addr = addr + size;
236+ int entry = 0;
237+
238+ /* Ensure requested range is not zero, and size is not causing addr to overflow. */
239+ if (end_addr <= addr)
240+ return -EINVAL;
241+
242+ for (entry = 0; entry < num_layout_entries; entry++) {
243+ u32 entry_start_addr = layout_entries[entry].base_addr;
244+ u32 entry_end_addr = entry_start_addr + layout_entries[entry].alloc_size;
245+
246+ if (addr >= entry_start_addr && addr < entry_end_addr &&
247+ end_addr > entry_start_addr && end_addr <= entry_end_addr) {
248+ switch (layout_entries[entry].type) {
249+ case FW_CODE:
250+ *host_addr_out = fw_code_ptr;
251+ break;
252+
253+ case FW_DATA:
254+ *host_addr_out = fw_data_ptr;
255+ break;
256+
257+ case FW_COREMEM_CODE:
258+ *host_addr_out = fw_core_code_ptr;
259+ break;
260+
261+ case FW_COREMEM_DATA:
262+ *host_addr_out = fw_core_data_ptr;
263+ break;
264+
265+ default:
266+ return -EINVAL;
267+ }
268+ /* Direct Mem write to mapped memory */
269+ addr -= layout_entries[entry].base_addr;
270+ addr += layout_entries[entry].alloc_offset;
271+
272+ /*
273+ * Add offset to pointer to FW allocation only if that
274+ * allocation is available
275+ */
276+ *(u8 **)host_addr_out += addr;
277+ return 0;
278+ }
279+ }
280+
281+ return -EINVAL;
282+}
283+
284+static int
285+pvr_fw_create_fwif_connection_ctl(struct pvr_device *pvr_dev)
286+{
287+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
288+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
289+
290+ fw_dev->fwif_connection_ctl =
291+ pvr_fw_object_create_and_map_offset(pvr_dev,
292+ fw_dev->fw_heap_info.config_offset +
293+ PVR_ROGUE_FWIF_CONNECTION_CTL_OFFSET,
294+ sizeof(*fw_dev->fwif_connection_ctl),
295+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
296+ NULL, NULL,
297+ &fw_dev->mem.fwif_connection_ctl_obj);
298+ if (IS_ERR(fw_dev->fwif_connection_ctl)) {
299+ drm_err(drm_dev,
300+ "Unable to allocate FWIF connection control memory\n");
301+ return PTR_ERR(fw_dev->fwif_connection_ctl);
302+ }
303+
304+ return 0;
305+}
306+
307+static void
308+pvr_fw_fini_fwif_connection_ctl(struct pvr_device *pvr_dev)
309+{
310+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
311+
312+ pvr_fw_object_unmap_and_destroy(fw_dev->mem.fwif_connection_ctl_obj);
313+}
314+
315+static void
316+fw_osinit_init(void *cpu_ptr, void *priv)
317+{
318+ struct rogue_fwif_osinit *fwif_osinit = cpu_ptr;
319+ struct pvr_device *pvr_dev = priv;
320+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
321+ struct pvr_fw_mem *fw_mem = &fw_dev->mem;
322+
323+ fwif_osinit->kernel_ccbctl_fw_addr = pvr_dev->kccb.ccb.ctrl_fw_addr;
324+ fwif_osinit->kernel_ccb_fw_addr = pvr_dev->kccb.ccb.ccb_fw_addr;
325+ pvr_fw_object_get_fw_addr(pvr_dev->kccb.rtn_obj,
326+ &fwif_osinit->kernel_ccb_rtn_slots_fw_addr);
327+
328+ fwif_osinit->firmware_ccbctl_fw_addr = pvr_dev->fwccb.ctrl_fw_addr;
329+ fwif_osinit->firmware_ccb_fw_addr = pvr_dev->fwccb.ccb_fw_addr;
330+
331+ fwif_osinit->work_est_firmware_ccbctl_fw_addr = 0;
332+ fwif_osinit->work_est_firmware_ccb_fw_addr = 0;
333+
334+ pvr_fw_object_get_fw_addr(fw_mem->hwrinfobuf_obj,
335+ &fwif_osinit->rogue_fwif_hwr_info_buf_ctl_fw_addr);
336+ pvr_fw_object_get_fw_addr(fw_mem->osdata_obj, &fwif_osinit->fw_os_data_fw_addr);
337+
338+ fwif_osinit->hwr_debug_dump_limit = 0;
339+
340+ rogue_fwif_compchecks_bvnc_init(&fwif_osinit->rogue_comp_checks.hw_bvnc);
341+ rogue_fwif_compchecks_bvnc_init(&fwif_osinit->rogue_comp_checks.fw_bvnc);
342+}
343+
344+static void
345+fw_osdata_init(void *cpu_ptr, void *priv)
346+{
347+ struct rogue_fwif_osdata *fwif_osdata = cpu_ptr;
348+ struct pvr_device *pvr_dev = priv;
349+ struct pvr_fw_mem *fw_mem = &pvr_dev->fw_dev.mem;
350+
351+ pvr_fw_object_get_fw_addr(fw_mem->power_sync_obj, &fwif_osdata->power_sync_fw_addr);
352+}
353+
354+static void
355+fw_fault_page_init(void *cpu_ptr, void *priv)
356+{
357+ u32 *fault_page = cpu_ptr;
358+
359+ for (int i = 0; i < PVR_ROGUE_FAULT_PAGE_SIZE / sizeof(*fault_page); i++)
360+ fault_page[i] = 0xdeadbee0;
361+}
362+
363+static void
364+fw_sysinit_init(void *cpu_ptr, void *priv)
365+{
366+ struct rogue_fwif_sysinit *fwif_sysinit = cpu_ptr;
367+ struct pvr_device *pvr_dev = priv;
368+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
369+ struct pvr_fw_mem *fw_mem = &fw_dev->mem;
370+ dma_addr_t fault_dma_addr = 0;
371+ u32 clock_speed_hz = clk_get_rate(pvr_dev->core_clk);
372+
373+ WARN_ON(!clock_speed_hz);
374+
375+ WARN_ON(pvr_fw_object_get_dma_addr(fw_mem->fault_page_obj, 0, &fault_dma_addr));
376+ fwif_sysinit->fault_phys_addr = (u64)fault_dma_addr;
377+
378+ fwif_sysinit->pds_exec_base = ROGUE_PDSCODEDATA_HEAP_BASE;
379+ fwif_sysinit->usc_exec_base = ROGUE_USCCODE_HEAP_BASE;
380+
381+ pvr_fw_object_get_fw_addr(fw_mem->runtime_cfg_obj, &fwif_sysinit->runtime_cfg_fw_addr);
382+ pvr_fw_object_get_fw_addr(fw_dev->fw_trace.tracebuf_ctrl_obj,
383+ &fwif_sysinit->trace_buf_ctl_fw_addr);
384+ pvr_fw_object_get_fw_addr(fw_mem->sysdata_obj, &fwif_sysinit->fw_sys_data_fw_addr);
385+ pvr_fw_object_get_fw_addr(fw_mem->gpu_util_fwcb_obj,
386+ &fwif_sysinit->gpu_util_fw_cb_ctl_fw_addr);
387+ if (fw_mem->core_data_obj) {
388+ pvr_fw_object_get_fw_addr(fw_mem->core_data_obj,
389+ &fwif_sysinit->coremem_data_store.fw_addr);
390+ }
391+
392+ /* Currently unsupported. */
393+ fwif_sysinit->counter_dump_ctl.buffer_fw_addr = 0;
394+ fwif_sysinit->counter_dump_ctl.size_in_dwords = 0;
395+
396+ /* Skip alignment checks. */
397+ fwif_sysinit->align_checks = 0;
398+
399+ fwif_sysinit->filter_flags = 0;
400+ fwif_sysinit->hw_perf_filter = 0;
401+ fwif_sysinit->firmware_perf = FW_PERF_CONF_NONE;
402+ fwif_sysinit->initial_core_clock_speed = clock_speed_hz;
403+ fwif_sysinit->active_pm_latency_ms = 0;
404+ fwif_sysinit->gpio_validation_mode = ROGUE_FWIF_GPIO_VAL_OFF;
405+ fwif_sysinit->firmware_started = false;
406+ fwif_sysinit->marker_val = 1;
407+
408+ memset(&fwif_sysinit->bvnc_km_feature_flags, 0,
409+ sizeof(fwif_sysinit->bvnc_km_feature_flags));
410+}
411+
412+#define ROGUE_FWIF_SLC_MIN_SIZE_FOR_DM_OVERLAP_KB 4
413+
414+static void
415+fw_sysdata_init(void *cpu_ptr, void *priv)
416+{
417+ struct rogue_fwif_sysdata *fwif_sysdata = cpu_ptr;
418+ struct pvr_device *pvr_dev = priv;
419+ u32 slc_size_in_kilobytes = 0;
420+ u32 config_flags = 0;
421+
422+ WARN_ON(PVR_FEATURE_VALUE(pvr_dev, slc_size_in_kilobytes, &slc_size_in_kilobytes));
423+
424+ if (slc_size_in_kilobytes < ROGUE_FWIF_SLC_MIN_SIZE_FOR_DM_OVERLAP_KB)
425+ config_flags |= ROGUE_FWIF_INICFG_DISABLE_DM_OVERLAP;
426+
427+ fwif_sysdata->config_flags = config_flags;
428+}
429+
430+static void
431+fw_runtime_cfg_init(void *cpu_ptr, void *priv)
432+{
433+ struct rogue_fwif_runtime_cfg *runtime_cfg = cpu_ptr;
434+ struct pvr_device *pvr_dev = priv;
435+ u32 clock_speed_hz = clk_get_rate(pvr_dev->core_clk);
436+
437+ WARN_ON(!clock_speed_hz);
438+
439+ runtime_cfg->core_clock_speed = clock_speed_hz;
440+ runtime_cfg->active_pm_latency_ms = 0;
441+ runtime_cfg->active_pm_latency_persistant = true;
442+ WARN_ON(PVR_FEATURE_VALUE(pvr_dev, num_clusters,
443+ &runtime_cfg->default_dusts_num_init) != 0);
444+}
445+
446+static void
447+fw_gpu_util_fwcb_init(void *cpu_ptr, void *priv)
448+{
449+ struct rogue_fwif_gpu_util_fwcb *gpu_util_fwcb = cpu_ptr;
450+
451+ gpu_util_fwcb->last_word = PVR_FWIF_GPU_UTIL_STATE_IDLE;
452+}
453+
454+static int
455+pvr_fw_create_structures(struct pvr_device *pvr_dev)
456+{
457+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
458+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
459+ struct pvr_fw_mem *fw_mem = &fw_dev->mem;
460+ int err;
461+
462+ fw_dev->power_sync = pvr_fw_object_create_and_map(pvr_dev, sizeof(*fw_dev->power_sync),
463+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
464+ NULL, NULL, &fw_mem->power_sync_obj);
465+ if (IS_ERR(fw_dev->power_sync)) {
466+ drm_err(drm_dev, "Unable to allocate FW power_sync structure\n");
467+ return PTR_ERR(fw_dev->power_sync);
468+ }
469+
470+ fw_dev->hwrinfobuf = pvr_fw_object_create_and_map(pvr_dev, sizeof(*fw_dev->hwrinfobuf),
471+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
472+ NULL, NULL, &fw_mem->hwrinfobuf_obj);
473+ if (IS_ERR(fw_dev->hwrinfobuf)) {
474+ drm_err(drm_dev,
475+ "Unable to allocate FW hwrinfobuf structure\n");
476+ err = PTR_ERR(fw_dev->hwrinfobuf);
477+ goto err_release_power_sync;
478+ }
479+
480+ err = pvr_fw_object_create(pvr_dev, PVR_SYNC_OBJ_SIZE,
481+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
482+ NULL, NULL, &fw_mem->mmucache_sync_obj);
483+ if (err) {
484+ drm_err(drm_dev,
485+ "Unable to allocate MMU cache sync object\n");
486+ goto err_release_hwrinfobuf;
487+ }
488+
489+ fw_dev->fwif_sysdata = pvr_fw_object_create_and_map(pvr_dev,
490+ sizeof(*fw_dev->fwif_sysdata),
491+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
492+ fw_sysdata_init, pvr_dev,
493+ &fw_mem->sysdata_obj);
494+ if (IS_ERR(fw_dev->fwif_sysdata)) {
495+ drm_err(drm_dev, "Unable to allocate FW SYSDATA structure\n");
496+ err = PTR_ERR(fw_dev->fwif_sysdata);
497+ goto err_release_mmucache_sync_obj;
498+ }
499+
500+ err = pvr_fw_object_create(pvr_dev, PVR_ROGUE_FAULT_PAGE_SIZE,
501+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
502+ fw_fault_page_init, NULL, &fw_mem->fault_page_obj);
503+ if (err) {
504+ drm_err(drm_dev, "Unable to allocate FW fault page\n");
505+ goto err_release_sysdata;
506+ }
507+
508+ err = pvr_fw_object_create(pvr_dev, sizeof(struct rogue_fwif_gpu_util_fwcb),
509+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
510+ fw_gpu_util_fwcb_init, pvr_dev, &fw_mem->gpu_util_fwcb_obj);
511+ if (err) {
512+ drm_err(drm_dev, "Unable to allocate GPU util FWCB\n");
513+ goto err_release_fault_page;
514+ }
515+
516+ err = pvr_fw_object_create(pvr_dev, sizeof(struct rogue_fwif_runtime_cfg),
517+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
518+ fw_runtime_cfg_init, pvr_dev, &fw_mem->runtime_cfg_obj);
519+ if (err) {
520+ drm_err(drm_dev, "Unable to allocate FW runtime config\n");
521+ goto err_release_gpu_util_fwcb;
522+ }
523+
524+ err = pvr_fw_trace_init(pvr_dev);
525+ if (err)
526+ goto err_release_runtime_cfg;
527+
528+ fw_dev->fwif_osdata = pvr_fw_object_create_and_map(pvr_dev,
529+ sizeof(*fw_dev->fwif_osdata),
530+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
531+ fw_osdata_init, pvr_dev,
532+ &fw_mem->osdata_obj);
533+ if (IS_ERR(fw_dev->fwif_osdata)) {
534+ drm_err(drm_dev, "Unable to allocate FW OSDATA structure\n");
535+ err = PTR_ERR(fw_dev->fwif_osdata);
536+ goto err_fw_trace_fini;
537+ }
538+
539+ fw_dev->fwif_osinit =
540+ pvr_fw_object_create_and_map_offset(pvr_dev,
541+ fw_dev->fw_heap_info.config_offset +
542+ PVR_ROGUE_FWIF_OSINIT_OFFSET,
543+ sizeof(*fw_dev->fwif_osinit),
544+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
545+ fw_osinit_init, pvr_dev, &fw_mem->osinit_obj);
546+ if (IS_ERR(fw_dev->fwif_osinit)) {
547+ drm_err(drm_dev, "Unable to allocate FW OSINIT structure\n");
548+ err = PTR_ERR(fw_dev->fwif_osinit);
549+ goto err_release_osdata;
550+ }
551+
552+ fw_dev->fwif_sysinit =
553+ pvr_fw_object_create_and_map_offset(pvr_dev,
554+ fw_dev->fw_heap_info.config_offset +
555+ PVR_ROGUE_FWIF_SYSINIT_OFFSET,
556+ sizeof(*fw_dev->fwif_sysinit),
557+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
558+ fw_sysinit_init, pvr_dev, &fw_mem->sysinit_obj);
559+ if (IS_ERR(fw_dev->fwif_sysinit)) {
560+ drm_err(drm_dev, "Unable to allocate FW SYSINIT structure\n");
561+ err = PTR_ERR(fw_dev->fwif_sysinit);
562+ goto err_release_osinit;
563+ }
564+
565+ return 0;
566+
567+err_release_osinit:
568+ pvr_fw_object_unmap_and_destroy(fw_mem->osinit_obj);
569+
570+err_release_osdata:
571+ pvr_fw_object_unmap_and_destroy(fw_mem->osdata_obj);
572+
573+err_fw_trace_fini:
574+ pvr_fw_trace_fini(pvr_dev);
575+
576+err_release_runtime_cfg:
577+ pvr_fw_object_destroy(fw_mem->runtime_cfg_obj);
578+
579+err_release_gpu_util_fwcb:
580+ pvr_fw_object_destroy(fw_mem->gpu_util_fwcb_obj);
581+
582+err_release_fault_page:
583+ pvr_fw_object_destroy(fw_mem->fault_page_obj);
584+
585+err_release_sysdata:
586+ pvr_fw_object_unmap_and_destroy(fw_mem->sysdata_obj);
587+
588+err_release_mmucache_sync_obj:
589+ pvr_fw_object_destroy(fw_mem->mmucache_sync_obj);
590+
591+err_release_hwrinfobuf:
592+ pvr_fw_object_unmap_and_destroy(fw_mem->hwrinfobuf_obj);
593+
594+err_release_power_sync:
595+ pvr_fw_object_unmap_and_destroy(fw_mem->power_sync_obj);
596+
597+ return err;
598+}
599+
600+static void
601+pvr_fw_destroy_structures(struct pvr_device *pvr_dev)
602+{
603+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
604+ struct pvr_fw_mem *fw_mem = &fw_dev->mem;
605+
606+ pvr_fw_trace_fini(pvr_dev);
607+ pvr_fw_object_destroy(fw_mem->runtime_cfg_obj);
608+ pvr_fw_object_destroy(fw_mem->gpu_util_fwcb_obj);
609+ pvr_fw_object_destroy(fw_mem->fault_page_obj);
610+ pvr_fw_object_unmap_and_destroy(fw_mem->sysdata_obj);
611+ pvr_fw_object_unmap_and_destroy(fw_mem->sysinit_obj);
612+
613+ pvr_fw_object_destroy(fw_mem->mmucache_sync_obj);
614+ pvr_fw_object_unmap_and_destroy(fw_mem->hwrinfobuf_obj);
615+ pvr_fw_object_unmap_and_destroy(fw_mem->power_sync_obj);
616+ pvr_fw_object_unmap_and_destroy(fw_mem->osdata_obj);
617+ pvr_fw_object_unmap_and_destroy(fw_mem->osinit_obj);
618+}
619+
620+/**
621+ * pvr_fw_process() - Process firmware image, allocate FW memory and create boot
622+ * arguments
623+ * @pvr_dev: Device pointer.
624+ *
625+ * Returns:
626+ * * 0 on success, or
627+ * * Any error returned by pvr_fw_object_create_and_map_offset(), or
628+ * * Any error returned by pvr_fw_object_create_and_map().
629+ */
630+static int
631+pvr_fw_process(struct pvr_device *pvr_dev)
632+{
633+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
634+ struct pvr_fw_mem *fw_mem = &pvr_dev->fw_dev.mem;
635+ const u8 *fw = pvr_dev->fw_dev.firmware->data;
636+ const struct pvr_fw_layout_entry *private_data;
637+ u8 *fw_code_ptr;
638+ u8 *fw_data_ptr;
639+ u8 *fw_core_code_ptr;
640+ u8 *fw_core_data_ptr;
641+ int err;
642+
643+ layout_get_sizes(pvr_dev);
644+
645+ private_data = pvr_fw_find_private_data(pvr_dev);
646+ if (!private_data)
647+ return -EINVAL;
648+
649+ /* Allocate and map memory for firmware sections. */
650+
651+ /*
652+ * Code allocation must be at the start of the firmware heap, otherwise
653+ * firmware processor will be unable to boot.
654+ *
655+ * This has the useful side-effect that for every other object in the
656+ * driver, a firmware address of 0 is invalid.
657+ */
658+ fw_code_ptr = pvr_fw_object_create_and_map_offset(pvr_dev, 0, fw_mem->code_alloc_size,
659+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
660+ NULL, NULL, &fw_mem->code_obj);
661+ if (IS_ERR(fw_code_ptr)) {
662+ drm_err(drm_dev, "Unable to allocate FW code memory\n");
663+ return PTR_ERR(fw_code_ptr);
664+ }
665+
666+ if (pvr_dev->fw_dev.defs->has_fixed_data_addr()) {
667+ u32 base_addr = private_data->base_addr & pvr_dev->fw_dev.fw_heap_info.offset_mask;
668+
669+ fw_data_ptr =
670+ pvr_fw_object_create_and_map_offset(pvr_dev, base_addr,
671+ fw_mem->data_alloc_size,
672+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
673+ NULL, NULL, &fw_mem->data_obj);
674+ } else {
675+ fw_data_ptr = pvr_fw_object_create_and_map(pvr_dev, fw_mem->data_alloc_size,
676+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
677+ NULL, NULL, &fw_mem->data_obj);
678+ }
679+ if (IS_ERR(fw_data_ptr)) {
680+ drm_err(drm_dev, "Unable to allocate FW data memory\n");
681+ err = PTR_ERR(fw_data_ptr);
682+ goto err_free_fw_code_obj;
683+ }
684+
685+ /* Core code and data sections are optional. */
686+ if (fw_mem->core_code_alloc_size) {
687+ fw_core_code_ptr =
688+ pvr_fw_object_create_and_map(pvr_dev, fw_mem->core_code_alloc_size,
689+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
690+ NULL, NULL, &fw_mem->core_code_obj);
691+ if (IS_ERR(fw_core_code_ptr)) {
692+ drm_err(drm_dev,
693+ "Unable to allocate FW core code memory\n");
694+ err = PTR_ERR(fw_core_code_ptr);
695+ goto err_free_fw_data_obj;
696+ }
697+ } else {
698+ fw_core_code_ptr = NULL;
699+ }
700+
701+ if (fw_mem->core_data_alloc_size) {
702+ fw_core_data_ptr =
703+ pvr_fw_object_create_and_map(pvr_dev, fw_mem->core_data_alloc_size,
704+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
705+ NULL, NULL, &fw_mem->core_data_obj);
706+ if (IS_ERR(fw_core_data_ptr)) {
707+ drm_err(drm_dev,
708+ "Unable to allocate FW core data memory\n");
709+ err = PTR_ERR(fw_core_data_ptr);
710+ goto err_free_fw_core_code_obj;
711+ }
712+ } else {
713+ fw_core_data_ptr = NULL;
714+ }
715+
716+ fw_mem->code = kzalloc(fw_mem->code_alloc_size, GFP_KERNEL);
717+ fw_mem->data = kzalloc(fw_mem->data_alloc_size, GFP_KERNEL);
718+ if (fw_mem->core_code_alloc_size)
719+ fw_mem->core_code = kzalloc(fw_mem->core_code_alloc_size, GFP_KERNEL);
720+ if (fw_mem->core_data_alloc_size)
721+ fw_mem->core_data = kzalloc(fw_mem->core_data_alloc_size, GFP_KERNEL);
722+
723+ if (!fw_mem->code || !fw_mem->data ||
724+ (!fw_mem->core_code && fw_mem->core_code_alloc_size) ||
725+ (!fw_mem->core_data && fw_mem->core_data_alloc_size)) {
726+ err = -ENOMEM;
727+ goto err_free_kdata;
728+ }
729+
730+ err = pvr_dev->fw_dev.defs->fw_process(pvr_dev, fw,
731+ fw_mem->code, fw_mem->data, fw_mem->core_code,
732+ fw_mem->core_data, fw_mem->core_code_alloc_size);
733+
734+ if (err)
735+ goto err_free_fw_core_data_obj;
736+
737+ memcpy(fw_code_ptr, fw_mem->code, fw_mem->code_alloc_size);
738+ memcpy(fw_data_ptr, fw_mem->data, fw_mem->data_alloc_size);
739+ if (fw_mem->core_code)
740+ memcpy(fw_core_code_ptr, fw_mem->core_code, fw_mem->core_code_alloc_size);
741+ if (fw_mem->core_data)
742+ memcpy(fw_core_data_ptr, fw_mem->core_data, fw_mem->core_data_alloc_size);
743+
744+ /* We're finished with the firmware section memory on the CPU, unmap. */
745+ if (fw_core_data_ptr)
746+ pvr_fw_object_vunmap(fw_mem->core_data_obj);
747+ if (fw_core_code_ptr)
748+ pvr_fw_object_vunmap(fw_mem->core_code_obj);
749+ pvr_fw_object_vunmap(fw_mem->data_obj);
750+ fw_data_ptr = NULL;
751+ pvr_fw_object_vunmap(fw_mem->code_obj);
752+ fw_code_ptr = NULL;
753+
754+ err = pvr_fw_create_fwif_connection_ctl(pvr_dev);
755+ if (err)
756+ goto err_free_fw_core_data_obj;
757+
758+ return 0;
759+
760+err_free_kdata:
761+ kfree(fw_mem->core_data);
762+ kfree(fw_mem->core_code);
763+ kfree(fw_mem->data);
764+ kfree(fw_mem->code);
765+
766+err_free_fw_core_data_obj:
767+ if (fw_core_data_ptr)
768+ pvr_fw_object_unmap_and_destroy(fw_mem->core_data_obj);
769+
770+err_free_fw_core_code_obj:
771+ if (fw_core_code_ptr)
772+ pvr_fw_object_unmap_and_destroy(fw_mem->core_code_obj);
773+
774+err_free_fw_data_obj:
775+ if (fw_data_ptr)
776+ pvr_fw_object_vunmap(fw_mem->data_obj);
777+ pvr_fw_object_destroy(fw_mem->data_obj);
778+
779+err_free_fw_code_obj:
780+ if (fw_code_ptr)
781+ pvr_fw_object_vunmap(fw_mem->code_obj);
782+ pvr_fw_object_destroy(fw_mem->code_obj);
783+
784+ return err;
785+}
786+
787+static int
788+pvr_copy_to_fw(struct pvr_fw_object *dest_obj, u8 *src_ptr, u32 size)
789+{
790+ u8 *dest_ptr = pvr_fw_object_vmap(dest_obj);
791+
792+ if (IS_ERR(dest_ptr))
793+ return PTR_ERR(dest_ptr);
794+
795+ memcpy(dest_ptr, src_ptr, size);
796+
797+ pvr_fw_object_vunmap(dest_obj);
798+
799+ return 0;
800+}
801+
802+static int
803+pvr_fw_reinit_code_data(struct pvr_device *pvr_dev)
804+{
805+ struct pvr_fw_mem *fw_mem = &pvr_dev->fw_dev.mem;
806+ int err;
807+
808+ err = pvr_copy_to_fw(fw_mem->code_obj, fw_mem->code, fw_mem->code_alloc_size);
809+ if (err)
810+ return err;
811+
812+ err = pvr_copy_to_fw(fw_mem->data_obj, fw_mem->data, fw_mem->data_alloc_size);
813+ if (err)
814+ return err;
815+
816+ if (fw_mem->core_code) {
817+ err = pvr_copy_to_fw(fw_mem->core_code_obj, fw_mem->core_code,
818+ fw_mem->core_code_alloc_size);
819+ if (err)
820+ return err;
821+ }
822+
823+ if (fw_mem->core_data) {
824+ err = pvr_copy_to_fw(fw_mem->core_data_obj, fw_mem->core_data,
825+ fw_mem->core_data_alloc_size);
826+ if (err)
827+ return err;
828+ }
829+
830+ return 0;
831+}
832+
833+static void
834+pvr_fw_cleanup(struct pvr_device *pvr_dev)
835+{
836+ struct pvr_fw_mem *fw_mem = &pvr_dev->fw_dev.mem;
837+
838+ pvr_fw_fini_fwif_connection_ctl(pvr_dev);
839+ if (fw_mem->core_code_obj)
840+ pvr_fw_object_destroy(fw_mem->core_code_obj);
841+ if (fw_mem->core_data_obj)
842+ pvr_fw_object_destroy(fw_mem->core_data_obj);
843+ pvr_fw_object_destroy(fw_mem->code_obj);
844+ pvr_fw_object_destroy(fw_mem->data_obj);
845+}
846+
847+/**
848+ * pvr_wait_for_fw_boot() - Wait for firmware to finish booting
849+ * @pvr_dev: Target PowerVR device.
850+ *
851+ * Returns:
852+ * * 0 on success, or
853+ * * -%ETIMEDOUT if firmware fails to boot within timeout.
854+ */
855+int
856+pvr_wait_for_fw_boot(struct pvr_device *pvr_dev)
857+{
858+ ktime_t deadline = ktime_add_us(ktime_get(), FW_BOOT_TIMEOUT_USEC);
859+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
860+
861+ while (ktime_to_ns(ktime_sub(deadline, ktime_get())) > 0) {
862+ if (READ_ONCE(fw_dev->fwif_sysinit->firmware_started))
863+ return 0;
864+ }
865+
866+ return -ETIMEDOUT;
867+}
868+
869+/*
870+ * pvr_fw_heap_info_init() - Calculate size and masks for FW heap
871+ * @pvr_dev: Target PowerVR device.
872+ * @log2_size: Log2 of raw heap size.
873+ * @reserved_size: Size of reserved area of heap, in bytes. May be zero.
874+ */
875+void
876+pvr_fw_heap_info_init(struct pvr_device *pvr_dev, u32 log2_size, u32 reserved_size)
877+{
878+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
879+
880+ fw_dev->fw_heap_info.gpu_addr = PVR_ROGUE_FW_MAIN_HEAP_BASE;
881+ fw_dev->fw_heap_info.log2_size = log2_size;
882+ fw_dev->fw_heap_info.reserved_size = reserved_size;
883+ fw_dev->fw_heap_info.raw_size = 1 << fw_dev->fw_heap_info.log2_size;
884+ fw_dev->fw_heap_info.offset_mask = fw_dev->fw_heap_info.raw_size - 1;
885+ fw_dev->fw_heap_info.config_offset = fw_dev->fw_heap_info.raw_size -
886+ PVR_ROGUE_FW_CONFIG_HEAP_SIZE;
887+ fw_dev->fw_heap_info.size = fw_dev->fw_heap_info.raw_size -
888+ (PVR_ROGUE_FW_CONFIG_HEAP_SIZE + reserved_size);
889+}
890+
891+/**
892+ * pvr_fw_validate_init_device_info() - Validate firmware and initialise device information
893+ * @pvr_dev: Target PowerVR device.
894+ *
895+ * This function must be called before querying device information.
896+ *
897+ * Returns:
898+ * * 0 on success, or
899+ * * -%EINVAL if firmware validation fails.
900+ */
901+int
902+pvr_fw_validate_init_device_info(struct pvr_device *pvr_dev)
903+{
904+ int err;
905+
906+ err = pvr_fw_validate(pvr_dev);
907+ if (err)
908+ return err;
909+
910+ return pvr_fw_get_device_info(pvr_dev);
911+}
912+
913+/**
914+ * pvr_fw_init() - Initialise and boot firmware
915+ * @pvr_dev: Target PowerVR device
916+ *
917+ * On successful completion of the function the PowerVR device will be
918+ * initialised and ready to use.
919+ *
920+ * Returns:
921+ * * 0 on success,
922+ * * -%EINVAL on invalid firmware image,
923+ * * -%ENOMEM on out of memory, or
924+ * * -%ETIMEDOUT if firmware processor fails to boot or on register poll timeout.
925+ */
926+int
927+pvr_fw_init(struct pvr_device *pvr_dev)
928+{
929+ u32 kccb_size_log2 = ROGUE_FWIF_KCCB_NUMCMDS_LOG2_DEFAULT;
930+ u32 kccb_rtn_size = (1 << kccb_size_log2) * sizeof(*pvr_dev->kccb.rtn);
931+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
932+ int err;
933+
934+ if (fw_dev->processor_type == PVR_FW_PROCESSOR_TYPE_META)
935+ fw_dev->defs = &pvr_fw_defs_meta;
936+ else if (fw_dev->processor_type == PVR_FW_PROCESSOR_TYPE_MIPS)
937+ fw_dev->defs = &pvr_fw_defs_mips;
938+ else
939+ return -EINVAL;
940+
941+ err = fw_dev->defs->init(pvr_dev);
942+ if (err)
943+ return err;
944+
945+ drm_mm_init(&fw_dev->fw_mm, ROGUE_FW_HEAP_BASE, fw_dev->fw_heap_info.raw_size);
946+ fw_dev->fw_mm_base = ROGUE_FW_HEAP_BASE;
947+ spin_lock_init(&fw_dev->fw_mm_lock);
948+
949+ INIT_LIST_HEAD(&fw_dev->fw_objs.list);
950+ err = drmm_mutex_init(from_pvr_device(pvr_dev), &fw_dev->fw_objs.lock);
951+ if (err)
952+ goto err_mm_takedown;
953+
954+ err = pvr_fw_process(pvr_dev);
955+ if (err)
956+ goto err_mm_takedown;
957+
958+ /* Initialise KCCB and FWCCB. */
959+ err = pvr_kccb_init(pvr_dev);
960+ if (err)
961+ goto err_fw_cleanup;
962+
963+ err = pvr_fwccb_init(pvr_dev);
964+ if (err)
965+ goto err_kccb_fini;
966+
967+ /* Allocate memory for KCCB return slots. */
968+ pvr_dev->kccb.rtn = pvr_fw_object_create_and_map(pvr_dev, kccb_rtn_size,
969+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
970+ NULL, NULL, &pvr_dev->kccb.rtn_obj);
971+ if (IS_ERR(pvr_dev->kccb.rtn)) {
972+ err = PTR_ERR(pvr_dev->kccb.rtn);
973+ goto err_fwccb_fini;
974+ }
975+
976+ err = pvr_fw_create_structures(pvr_dev);
977+ if (err)
978+ goto err_kccb_rtn_release;
979+
980+ err = pvr_fw_start(pvr_dev);
981+ if (err)
982+ goto err_destroy_structures;
983+
984+ err = pvr_wait_for_fw_boot(pvr_dev);
985+ if (err) {
986+ drm_err(from_pvr_device(pvr_dev), "Firmware failed to boot\n");
987+ goto err_fw_stop;
988+ }
989+
990+ fw_dev->booted = true;
991+
992+ return 0;
993+
994+err_fw_stop:
995+ pvr_fw_stop(pvr_dev);
996+
997+err_destroy_structures:
998+ pvr_fw_destroy_structures(pvr_dev);
999+
1000+err_kccb_rtn_release:
1001+ pvr_fw_object_unmap_and_destroy(pvr_dev->kccb.rtn_obj);
1002+
1003+err_fwccb_fini:
1004+ pvr_ccb_fini(&pvr_dev->fwccb);
1005+
1006+err_kccb_fini:
1007+ pvr_kccb_fini(pvr_dev);
1008+
1009+err_fw_cleanup:
1010+ pvr_fw_cleanup(pvr_dev);
1011+
1012+err_mm_takedown:
1013+ drm_mm_takedown(&fw_dev->fw_mm);
1014+
1015+ if (fw_dev->defs->fini)
1016+ fw_dev->defs->fini(pvr_dev);
1017+
1018+ return err;
1019+}
1020+
1021+/**
1022+ * pvr_fw_fini() - Shutdown firmware processor and free associated memory
1023+ * @pvr_dev: Target PowerVR device
1024+ */
1025+void
1026+pvr_fw_fini(struct pvr_device *pvr_dev)
1027+{
1028+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
1029+
1030+ fw_dev->booted = false;
1031+
1032+ pvr_fw_destroy_structures(pvr_dev);
1033+ pvr_fw_object_unmap_and_destroy(pvr_dev->kccb.rtn_obj);
1034+
1035+ /*
1036+ * Ensure FWCCB worker has finished executing before destroying FWCCB. The IRQ handler has
1037+ * been unregistered at this point so no new work should be being submitted.
1038+ */
1039+ pvr_ccb_fini(&pvr_dev->fwccb);
1040+ pvr_kccb_fini(pvr_dev);
1041+ pvr_fw_cleanup(pvr_dev);
1042+
1043+ mutex_lock(&pvr_dev->fw_dev.fw_objs.lock);
1044+ WARN_ON(!list_empty(&pvr_dev->fw_dev.fw_objs.list));
1045+ mutex_unlock(&pvr_dev->fw_dev.fw_objs.lock);
1046+
1047+ drm_mm_takedown(&fw_dev->fw_mm);
1048+
1049+ if (fw_dev->defs->fini)
1050+ fw_dev->defs->fini(pvr_dev);
1051+}
1052+
1053+/**
1054+ * pvr_fw_mts_schedule() - Schedule work via an MTS kick
1055+ * @pvr_dev: Target PowerVR device
1056+ * @val: Kick mask. Should be a combination of %ROGUE_CR_MTS_SCHEDULE_*
1057+ */
1058+void
1059+pvr_fw_mts_schedule(struct pvr_device *pvr_dev, u32 val)
1060+{
1061+ /* Ensure memory is flushed before kicking MTS. */
1062+ wmb();
1063+
1064+ pvr_cr_write32(pvr_dev, ROGUE_CR_MTS_SCHEDULE, val);
1065+
1066+ /* Ensure the MTS kick goes through before continuing. */
1067+ mb();
1068+}
1069+
1070+/**
1071+ * pvr_fw_structure_cleanup() - Send FW cleanup request for an object
1072+ * @pvr_dev: Target PowerVR device.
1073+ * @type: Type of object to cleanup. Must be one of &enum rogue_fwif_cleanup_type.
1074+ * @fw_obj: Pointer to FW object containing object to cleanup.
1075+ * @offset: Offset within FW object of object to cleanup.
1076+ *
1077+ * Returns:
1078+ * * 0 on success,
1079+ * * -EBUSY if object is busy,
1080+ * * -ETIMEDOUT on timeout, or
1081+ * * -EIO if device is lost.
1082+ */
1083+int
1084+pvr_fw_structure_cleanup(struct pvr_device *pvr_dev, u32 type, struct pvr_fw_object *fw_obj,
1085+ u32 offset)
1086+{
1087+ struct rogue_fwif_kccb_cmd cmd;
1088+ int slot_nr;
1089+ int idx;
1090+ int err;
1091+ u32 rtn;
1092+
1093+ struct rogue_fwif_cleanup_request *cleanup_req = &cmd.cmd_data.cleanup_data;
1094+
1095+ down_read(&pvr_dev->reset_sem);
1096+
1097+ if (!drm_dev_enter(from_pvr_device(pvr_dev), &idx)) {
1098+ err = -EIO;
1099+ goto err_up_read;
1100+ }
1101+
1102+ cmd.cmd_type = ROGUE_FWIF_KCCB_CMD_CLEANUP;
1103+ cmd.kccb_flags = 0;
1104+ cleanup_req->cleanup_type = type;
1105+
1106+ switch (type) {
1107+ case ROGUE_FWIF_CLEANUP_FWCOMMONCONTEXT:
1108+ pvr_fw_object_get_fw_addr_offset(fw_obj, offset,
1109+ &cleanup_req->cleanup_data.context_fw_addr);
1110+ break;
1111+ case ROGUE_FWIF_CLEANUP_HWRTDATA:
1112+ pvr_fw_object_get_fw_addr_offset(fw_obj, offset,
1113+ &cleanup_req->cleanup_data.hwrt_data_fw_addr);
1114+ break;
1115+ case ROGUE_FWIF_CLEANUP_FREELIST:
1116+ pvr_fw_object_get_fw_addr_offset(fw_obj, offset,
1117+ &cleanup_req->cleanup_data.freelist_fw_addr);
1118+ break;
1119+ default:
1120+ err = -EINVAL;
1121+ goto err_drm_dev_exit;
1122+ }
1123+
1124+ err = pvr_kccb_send_cmd(pvr_dev, &cmd, &slot_nr);
1125+ if (err)
1126+ goto err_drm_dev_exit;
1127+
1128+ err = pvr_kccb_wait_for_completion(pvr_dev, slot_nr, HZ, &rtn);
1129+ if (err)
1130+ goto err_drm_dev_exit;
1131+
1132+ if (rtn & ROGUE_FWIF_KCCB_RTN_SLOT_CLEANUP_BUSY)
1133+ err = -EBUSY;
1134+
1135+err_drm_dev_exit:
1136+ drm_dev_exit(idx);
1137+
1138+err_up_read:
1139+ up_read(&pvr_dev->reset_sem);
1140+
1141+ return err;
1142+}
1143+
1144+/**
1145+ * pvr_fw_object_fw_map() - Map a FW object in firmware address space
1146+ * @pvr_dev: Device pointer.
1147+ * @fw_obj: FW object to map.
1148+ * @dev_addr: Desired address in device space, if a specific address is
1149+ * required. 0 otherwise.
1150+ *
1151+ * Returns:
1152+ * * 0 on success, or
1153+ * * -%EINVAL if @fw_obj is already mapped but has no references, or
1154+ * * Any error returned by DRM.
1155+ */
1156+static int
1157+pvr_fw_object_fw_map(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj, u64 dev_addr)
1158+{
1159+ struct pvr_gem_object *pvr_obj = fw_obj->gem;
1160+ struct drm_gem_object *gem_obj = gem_from_pvr_gem(pvr_obj);
1161+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
1162+
1163+ int err;
1164+
1165+ spin_lock(&fw_dev->fw_mm_lock);
1166+
1167+ if (drm_mm_node_allocated(&fw_obj->fw_mm_node)) {
1168+ err = -EINVAL;
1169+ goto err_unlock;
1170+ }
1171+
1172+ if (!dev_addr) {
1173+ /*
1174+ * Allocate from the main heap only (firmware heap minus
1175+ * config space).
1176+ */
1177+ err = drm_mm_insert_node_in_range(&fw_dev->fw_mm, &fw_obj->fw_mm_node,
1178+ gem_obj->size, 0, 0,
1179+ fw_dev->fw_heap_info.gpu_addr,
1180+ fw_dev->fw_heap_info.gpu_addr +
1181+ fw_dev->fw_heap_info.size, 0);
1182+ if (err)
1183+ goto err_unlock;
1184+ } else {
1185+ fw_obj->fw_mm_node.start = dev_addr;
1186+ fw_obj->fw_mm_node.size = gem_obj->size;
1187+ err = drm_mm_reserve_node(&fw_dev->fw_mm, &fw_obj->fw_mm_node);
1188+ if (err)
1189+ goto err_unlock;
1190+ }
1191+
1192+ spin_unlock(&fw_dev->fw_mm_lock);
1193+
1194+ /* Map object on GPU. */
1195+ err = fw_dev->defs->vm_map(pvr_dev, fw_obj);
1196+ if (err)
1197+ goto err_remove_node;
1198+
1199+ fw_obj->fw_addr_offset = (u32)(fw_obj->fw_mm_node.start - fw_dev->fw_mm_base);
1200+
1201+ return 0;
1202+
1203+err_remove_node:
1204+ spin_lock(&fw_dev->fw_mm_lock);
1205+ drm_mm_remove_node(&fw_obj->fw_mm_node);
1206+
1207+err_unlock:
1208+ spin_unlock(&fw_dev->fw_mm_lock);
1209+
1210+ return err;
1211+}
1212+
1213+/**
1214+ * pvr_fw_object_fw_unmap() - Unmap a previously mapped FW object
1215+ * @fw_obj: FW object to unmap.
1216+ *
1217+ * Returns:
1218+ * * 0 on success, or
1219+ * * -%EINVAL if object is not currently mapped.
1220+ */
1221+static int
1222+pvr_fw_object_fw_unmap(struct pvr_fw_object *fw_obj)
1223+{
1224+ struct pvr_gem_object *pvr_obj = fw_obj->gem;
1225+ struct drm_gem_object *gem_obj = gem_from_pvr_gem(pvr_obj);
1226+ struct pvr_device *pvr_dev = to_pvr_device(gem_obj->dev);
1227+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
1228+
1229+ fw_dev->defs->vm_unmap(pvr_dev, fw_obj);
1230+
1231+ spin_lock(&fw_dev->fw_mm_lock);
1232+
1233+ if (!drm_mm_node_allocated(&fw_obj->fw_mm_node)) {
1234+ spin_unlock(&fw_dev->fw_mm_lock);
1235+ return -EINVAL;
1236+ }
1237+
1238+ drm_mm_remove_node(&fw_obj->fw_mm_node);
1239+
1240+ spin_unlock(&fw_dev->fw_mm_lock);
1241+
1242+ return 0;
1243+}
1244+
1245+static void *
1246+pvr_fw_object_create_and_map_common(struct pvr_device *pvr_dev, size_t size,
1247+ u64 flags, u64 dev_addr,
1248+ void (*init)(void *cpu_ptr, void *priv),
1249+ void *init_priv, struct pvr_fw_object **fw_obj_out)
1250+{
1251+ struct pvr_fw_object *fw_obj;
1252+ void *cpu_ptr;
1253+ int err;
1254+
1255+ /* %DRM_PVR_BO_PM_FW_PROTECT is implicit for FW objects. */
1256+ flags |= DRM_PVR_BO_PM_FW_PROTECT;
1257+
1258+ fw_obj = kzalloc(sizeof(*fw_obj), GFP_KERNEL);
1259+ if (!fw_obj)
1260+ return ERR_PTR(-ENOMEM);
1261+
1262+ INIT_LIST_HEAD(&fw_obj->node);
1263+ fw_obj->init = init;
1264+ fw_obj->init_priv = init_priv;
1265+
1266+ fw_obj->gem = pvr_gem_object_create(pvr_dev, size, flags);
1267+ if (IS_ERR(fw_obj->gem)) {
1268+ err = PTR_ERR(fw_obj->gem);
1269+ fw_obj->gem = NULL;
1270+ goto err_put_object;
1271+ }
1272+
1273+ err = pvr_fw_object_fw_map(pvr_dev, fw_obj, dev_addr);
1274+ if (err)
1275+ goto err_put_object;
1276+
1277+ cpu_ptr = pvr_fw_object_vmap(fw_obj);
1278+ if (IS_ERR(cpu_ptr)) {
1279+ err = PTR_ERR(cpu_ptr);
1280+ goto err_put_object;
1281+ }
1282+
1283+ *fw_obj_out = fw_obj;
1284+
1285+ if (fw_obj->init)
1286+ fw_obj->init(cpu_ptr, fw_obj->init_priv);
1287+
1288+ mutex_lock(&pvr_dev->fw_dev.fw_objs.lock);
1289+ list_add_tail(&fw_obj->node, &pvr_dev->fw_dev.fw_objs.list);
1290+ mutex_unlock(&pvr_dev->fw_dev.fw_objs.lock);
1291+
1292+ return cpu_ptr;
1293+
1294+err_put_object:
1295+ pvr_fw_object_destroy(fw_obj);
1296+
1297+ return ERR_PTR(err);
1298+}
1299+
1300+/**
1301+ * pvr_fw_object_create() - Create a FW object and map to firmware
1302+ * @pvr_dev: PowerVR device pointer.
1303+ * @size: Size of object, in bytes.
1304+ * @flags: Options which affect both this operation and future mapping
1305+ * operations performed on the returned object. Must be a combination of
1306+ * DRM_PVR_BO_* and/or PVR_BO_* flags.
1307+ * @init: Initialisation callback.
1308+ * @init_priv: Private pointer to pass to initialisation callback.
1309+ * @fw_obj_out: Pointer to location to store created object pointer.
1310+ *
1311+ * %DRM_PVR_BO_DEVICE_PM_FW_PROTECT is implied for all FW objects. Consequently,
1312+ * this function will fail if @flags has %DRM_PVR_BO_CPU_ALLOW_USERSPACE_ACCESS
1313+ * set.
1314+ *
1315+ * Returns:
1316+ * * 0 on success, or
1317+ * * Any error returned by pvr_fw_object_create_common().
1318+ */
1319+int
1320+pvr_fw_object_create(struct pvr_device *pvr_dev, size_t size, u64 flags,
1321+ void (*init)(void *cpu_ptr, void *priv), void *init_priv,
1322+ struct pvr_fw_object **fw_obj_out)
1323+{
1324+ void *cpu_ptr;
1325+
1326+ cpu_ptr = pvr_fw_object_create_and_map_common(pvr_dev, size, flags, 0, init, init_priv,
1327+ fw_obj_out);
1328+ if (IS_ERR(cpu_ptr))
1329+ return PTR_ERR(cpu_ptr);
1330+
1331+ pvr_fw_object_vunmap(*fw_obj_out);
1332+
1333+ return 0;
1334+}
1335+
1336+/**
1337+ * pvr_fw_object_create_and_map() - Create a FW object and map to firmware and CPU
1338+ * @pvr_dev: PowerVR device pointer.
1339+ * @size: Size of object, in bytes.
1340+ * @flags: Options which affect both this operation and future mapping
1341+ * operations performed on the returned object. Must be a combination of
1342+ * DRM_PVR_BO_* and/or PVR_BO_* flags.
1343+ * @init: Initialisation callback.
1344+ * @init_priv: Private pointer to pass to initialisation callback.
1345+ * @fw_obj_out: Pointer to location to store created object pointer.
1346+ *
1347+ * %DRM_PVR_BO_DEVICE_PM_FW_PROTECT is implied for all FW objects. Consequently,
1348+ * this function will fail if @flags has %DRM_PVR_BO_CPU_ALLOW_USERSPACE_ACCESS
1349+ * set.
1350+ *
1351+ * Caller is responsible for calling pvr_fw_object_vunmap() to release the CPU
1352+ * mapping.
1353+ *
1354+ * Returns:
1355+ * * Pointer to CPU mapping of newly created object, or
1356+ * * Any error returned by pvr_fw_object_create(), or
1357+ * * Any error returned by pvr_fw_object_vmap().
1358+ */
1359+void *
1360+pvr_fw_object_create_and_map(struct pvr_device *pvr_dev, size_t size, u64 flags,
1361+ void (*init)(void *cpu_ptr, void *priv),
1362+ void *init_priv, struct pvr_fw_object **fw_obj_out)
1363+{
1364+ return pvr_fw_object_create_and_map_common(pvr_dev, size, flags, 0, init, init_priv,
1365+ fw_obj_out);
1366+}
1367+
1368+/**
1369+ * pvr_fw_object_create_and_map_offset() - Create a FW object and map to
1370+ * firmware at the provided offset and to the CPU.
1371+ * @pvr_dev: PowerVR device pointer.
1372+ * @dev_offset: Base address of desired FW mapping, offset from start of FW heap.
1373+ * @size: Size of object, in bytes.
1374+ * @flags: Options which affect both this operation and future mapping
1375+ * operations performed on the returned object. Must be a combination of
1376+ * DRM_PVR_BO_* and/or PVR_BO_* flags.
1377+ * @init: Initialisation callback.
1378+ * @init_priv: Private pointer to pass to initialisation callback.
1379+ * @fw_obj_out: Pointer to location to store created object pointer.
1380+ *
1381+ * %DRM_PVR_BO_DEVICE_PM_FW_PROTECT is implied for all FW objects. Consequently,
1382+ * this function will fail if @flags has %DRM_PVR_BO_CPU_ALLOW_USERSPACE_ACCESS
1383+ * set.
1384+ *
1385+ * Caller is responsible for calling pvr_fw_object_vunmap() to release the CPU
1386+ * mapping.
1387+ *
1388+ * Returns:
1389+ * * Pointer to CPU mapping of newly created object, or
1390+ * * Any error returned by pvr_fw_object_create(), or
1391+ * * Any error returned by pvr_fw_object_vmap().
1392+ */
1393+void *
1394+pvr_fw_object_create_and_map_offset(struct pvr_device *pvr_dev,
1395+ u32 dev_offset, size_t size, u64 flags,
1396+ void (*init)(void *cpu_ptr, void *priv),
1397+ void *init_priv, struct pvr_fw_object **fw_obj_out)
1398+{
1399+ u64 dev_addr = pvr_dev->fw_dev.fw_mm_base + dev_offset;
1400+
1401+ return pvr_fw_object_create_and_map_common(pvr_dev, size, flags, dev_addr, init, init_priv,
1402+ fw_obj_out);
1403+}
1404+
1405+/**
1406+ * pvr_fw_object_destroy() - Destroy a pvr_fw_object
1407+ * @fw_obj: Pointer to object to destroy.
1408+ */
1409+void pvr_fw_object_destroy(struct pvr_fw_object *fw_obj)
1410+{
1411+ struct pvr_gem_object *pvr_obj = fw_obj->gem;
1412+ struct drm_gem_object *gem_obj = gem_from_pvr_gem(pvr_obj);
1413+ struct pvr_device *pvr_dev = to_pvr_device(gem_obj->dev);
1414+
1415+ mutex_lock(&pvr_dev->fw_dev.fw_objs.lock);
1416+ list_del(&fw_obj->node);
1417+ mutex_unlock(&pvr_dev->fw_dev.fw_objs.lock);
1418+
1419+ if (drm_mm_node_allocated(&fw_obj->fw_mm_node)) {
1420+ /* If we can't unmap, leak the memory. */
1421+ if (WARN_ON(pvr_fw_object_fw_unmap(fw_obj)))
1422+ return;
1423+ }
1424+
1425+ if (fw_obj->gem)
1426+ pvr_gem_object_put(fw_obj->gem);
1427+
1428+ kfree(fw_obj);
1429+}
1430+
1431+/**
1432+ * pvr_fw_object_get_fw_addr_offset() - Return address of object in firmware address space, with
1433+ * given offset.
1434+ * @fw_obj: Pointer to object.
1435+ * @offset: Desired offset from start of object.
1436+ * @fw_addr_out: Location to store address to.
1437+ */
1438+void pvr_fw_object_get_fw_addr_offset(struct pvr_fw_object *fw_obj, u32 offset, u32 *fw_addr_out)
1439+{
1440+ struct pvr_gem_object *pvr_obj = fw_obj->gem;
1441+ struct pvr_device *pvr_dev = to_pvr_device(gem_from_pvr_gem(pvr_obj)->dev);
1442+
1443+ *fw_addr_out = pvr_dev->fw_dev.defs->get_fw_addr_with_offset(fw_obj, offset);
1444+}
1445+
1446+/*
1447+ * pvr_fw_hard_reset() - Re-initialise the FW code and data segments, and reset all global FW
1448+ * structures
1449+ * @pvr_dev: Device pointer
1450+ *
1451+ * If this function returns an error then the caller must regard the device as lost.
1452+ *
1453+ * Returns:
1454+ * * 0 on success, or
1455+ * * Any error returned by pvr_fw_init_dev_structures() or pvr_fw_reset_all().
1456+ */
1457+int
1458+pvr_fw_hard_reset(struct pvr_device *pvr_dev)
1459+{
1460+ struct list_head *pos;
1461+ int err;
1462+
1463+ /* Reset all FW objects */
1464+ mutex_lock(&pvr_dev->fw_dev.fw_objs.lock);
1465+
1466+ list_for_each(pos, &pvr_dev->fw_dev.fw_objs.list) {
1467+ struct pvr_fw_object *fw_obj = container_of(pos, struct pvr_fw_object, node);
1468+ void *cpu_ptr = pvr_fw_object_vmap(fw_obj);
1469+
1470+ WARN_ON(IS_ERR(cpu_ptr));
1471+
1472+ if (!(fw_obj->gem->flags & PVR_BO_FW_NO_CLEAR_ON_RESET)) {
1473+ memset(cpu_ptr, 0, pvr_gem_object_size(fw_obj->gem));
1474+
1475+ if (fw_obj->init)
1476+ fw_obj->init(cpu_ptr, fw_obj->init_priv);
1477+ }
1478+
1479+ pvr_fw_object_vunmap(fw_obj);
1480+ }
1481+
1482+ mutex_unlock(&pvr_dev->fw_dev.fw_objs.lock);
1483+
1484+ err = pvr_fw_reinit_code_data(pvr_dev);
1485+ if (err)
1486+ return err;
1487+
1488+ return 0;
1489+}
drivers/gpu/drm/imagination/pvr_fw.h+508−0
@@ -0,0 +1,508 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_FW_H
5+#define PVR_FW_H
6+
7+#include "pvr_fw_info.h"
8+#include "pvr_fw_trace.h"
9+#include "pvr_gem.h"
10+
11+#include <drm/drm_mm.h>
12+
13+#include <linux/types.h>
14+
15+/* Forward declarations from "pvr_device.h". */
16+struct pvr_device;
17+struct pvr_file;
18+
19+/* Forward declaration from "pvr_vm.h". */
20+struct pvr_vm_context;
21+
22+#define ROGUE_FWIF_FWCCB_NUMCMDS_LOG2 5
23+
24+#define ROGUE_FWIF_KCCB_NUMCMDS_LOG2_DEFAULT 7
25+
26+/**
27+ * struct pvr_fw_object - container for firmware memory allocations
28+ */
29+struct pvr_fw_object {
30+ /** @ref_count: FW object reference counter. */
31+ struct kref ref_count;
32+
33+ /** @gem: GEM object backing the FW object. */
34+ struct pvr_gem_object *gem;
35+
36+ /**
37+ * @fw_mm_node: Node representing mapping in FW address space. @pvr_obj->lock must
38+ * be held when writing.
39+ */
40+ struct drm_mm_node fw_mm_node;
41+
42+ /**
43+ * @fw_addr_offset: Virtual address offset of firmware mapping. Only
44+ * valid if @flags has %PVR_GEM_OBJECT_FLAGS_FW_MAPPED
45+ * set.
46+ */
47+ u32 fw_addr_offset;
48+
49+ /**
50+ * @init: Initialisation callback. Will be called on object creation and FW hard reset.
51+ * Object will have been zeroed before this is called.
52+ */
53+ void (*init)(void *cpu_ptr, void *priv);
54+
55+ /** @init_priv: Private data for initialisation callback. */
56+ void *init_priv;
57+
58+ /** @node: Node for firmware object list. */
59+ struct list_head node;
60+};
61+
62+/**
63+ * struct pvr_fw_defs - FW processor function table and static definitions
64+ */
65+struct pvr_fw_defs {
66+ /**
67+ * @init:
68+ *
69+ * FW processor specific initialisation.
70+ * @pvr_dev: Target PowerVR device.
71+ *
72+ * This function must call pvr_fw_heap_calculate() to initialise the firmware heap for this
73+ * FW processor.
74+ *
75+ * This function is mandatory.
76+ *
77+ * Returns:
78+ * * 0 on success, or
79+ * * Any appropriate error on failure.
80+ */
81+ int (*init)(struct pvr_device *pvr_dev);
82+
83+ /**
84+ * @fini:
85+ *
86+ * FW processor specific finalisation.
87+ * @pvr_dev: Target PowerVR device.
88+ *
89+ * This function is optional.
90+ */
91+ void (*fini)(struct pvr_device *pvr_dev);
92+
93+ /**
94+ * @fw_process:
95+ *
96+ * Load and process firmware image.
97+ * @pvr_dev: Target PowerVR device.
98+ * @fw: Pointer to firmware image.
99+ * @fw_code_ptr: Pointer to firmware code section.
100+ * @fw_data_ptr: Pointer to firmware data section.
101+ * @fw_core_code_ptr: Pointer to firmware core code section. May be %NULL.
102+ * @fw_core_data_ptr: Pointer to firmware core data section. May be %NULL.
103+ * @core_code_alloc_size: Total allocation size of core code section.
104+ *
105+ * This function is mandatory.
106+ *
107+ * Returns:
108+ * * 0 on success, or
109+ * * Any appropriate error on failure.
110+ */
111+ int (*fw_process)(struct pvr_device *pvr_dev, const u8 *fw,
112+ u8 *fw_code_ptr, u8 *fw_data_ptr, u8 *fw_core_code_ptr,
113+ u8 *fw_core_data_ptr, u32 core_code_alloc_size);
114+
115+ /**
116+ * @vm_map:
117+ *
118+ * Map FW object into FW processor address space.
119+ * @pvr_dev: Target PowerVR device.
120+ * @fw_obj: FW object to map.
121+ *
122+ * This function is mandatory.
123+ *
124+ * Returns:
125+ * * 0 on success, or
126+ * * Any appropriate error on failure.
127+ */
128+ int (*vm_map)(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj);
129+
130+ /**
131+ * @vm_unmap:
132+ *
133+ * Unmap FW object from FW processor address space.
134+ * @pvr_dev: Target PowerVR device.
135+ * @fw_obj: FW object to map.
136+ *
137+ * This function is mandatory.
138+ */
139+ void (*vm_unmap)(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj);
140+
141+ /**
142+ * @get_fw_addr_with_offset:
143+ *
144+ * Called to get address of object in firmware address space, with offset.
145+ * @fw_obj: Pointer to object.
146+ * @offset: Desired offset from start of object.
147+ *
148+ * This function is mandatory.
149+ *
150+ * Returns:
151+ * * Address in firmware address space.
152+ */
153+ u32 (*get_fw_addr_with_offset)(struct pvr_fw_object *fw_obj, u32 offset);
154+
155+ /**
156+ * @wrapper_init:
157+ *
158+ * Called to initialise FW wrapper.
159+ * @pvr_dev: Target PowerVR device.
160+ *
161+ * This function is mandatory.
162+ *
163+ * Returns:
164+ * * 0 on success.
165+ * * Any appropriate error on failure.
166+ */
167+ int (*wrapper_init)(struct pvr_device *pvr_dev);
168+
169+ /**
170+ * @has_fixed_data_addr:
171+ *
172+ * Called to check if firmware fixed data must be loaded at the address given by the
173+ * firmware layout table.
174+ *
175+ * This function is mandatory.
176+ *
177+ * Returns:
178+ * * %true if firmware fixed data must be loaded at the address given by the firmware
179+ * layout table.
180+ * * %false otherwise.
181+ */
182+ bool (*has_fixed_data_addr)(void);
183+
184+ /**
185+ * @irq: FW Interrupt information.
186+ *
187+ * Those are processor dependent, and should be initialized by the
188+ * processor backend in pvr_fw_funcs::init().
189+ */
190+ struct {
191+ /** @enable_reg: FW interrupt enable register. */
192+ u32 enable_reg;
193+
194+ /** @status_reg: FW interrupt status register. */
195+ u32 status_reg;
196+
197+ /**
198+ * @clear_reg: FW interrupt clear register.
199+ *
200+ * If @status_reg == @clear_reg, we clear by write a bit to zero,
201+ * otherwise we clear by writing a bit to one.
202+ */
203+ u32 clear_reg;
204+
205+ /** @event_mask: Bitmask of events to listen for. */
206+ u32 event_mask;
207+
208+ /** @clear_mask: Value to write to the clear_reg in order to clear FW IRQs. */
209+ u32 clear_mask;
210+ } irq;
211+};
212+
213+/**
214+ * struct pvr_fw_mem - FW memory allocations
215+ */
216+struct pvr_fw_mem {
217+ /** @code_obj: Object representing firmware code. */
218+ struct pvr_fw_object *code_obj;
219+
220+ /** @data_obj: Object representing firmware data. */
221+ struct pvr_fw_object *data_obj;
222+
223+ /**
224+ * @core_code_obj: Object representing firmware core code. May be
225+ * %NULL if firmware does not contain this section.
226+ */
227+ struct pvr_fw_object *core_code_obj;
228+
229+ /**
230+ * @core_data_obj: Object representing firmware core data. May be
231+ * %NULL if firmware does not contain this section.
232+ */
233+ struct pvr_fw_object *core_data_obj;
234+
235+ /** @code: Driver-side copy of firmware code. */
236+ u8 *code;
237+
238+ /** @data: Driver-side copy of firmware data. */
239+ u8 *data;
240+
241+ /**
242+ * @core_code: Driver-side copy of firmware core code. May be %NULL if firmware does not
243+ * contain this section.
244+ */
245+ u8 *core_code;
246+
247+ /**
248+ * @core_data: Driver-side copy of firmware core data. May be %NULL if firmware does not
249+ * contain this section.
250+ */
251+ u8 *core_data;
252+
253+ /** @code_alloc_size: Allocation size of firmware code section. */
254+ u32 code_alloc_size;
255+
256+ /** @data_alloc_size: Allocation size of firmware data section. */
257+ u32 data_alloc_size;
258+
259+ /** @core_code_alloc_size: Allocation size of firmware core code section. */
260+ u32 core_code_alloc_size;
261+
262+ /** @core_data_alloc_size: Allocation size of firmware core data section. */
263+ u32 core_data_alloc_size;
264+
265+ /**
266+ * @fwif_connection_ctl_obj: Object representing FWIF connection control
267+ * structure.
268+ */
269+ struct pvr_fw_object *fwif_connection_ctl_obj;
270+
271+ /** @osinit_obj: Object representing FW OSINIT structure. */
272+ struct pvr_fw_object *osinit_obj;
273+
274+ /** @sysinit_obj: Object representing FW SYSINIT structure. */
275+ struct pvr_fw_object *sysinit_obj;
276+
277+ /** @osdata_obj: Object representing FW OSDATA structure. */
278+ struct pvr_fw_object *osdata_obj;
279+
280+ /** @hwrinfobuf_obj: Object representing FW hwrinfobuf structure. */
281+ struct pvr_fw_object *hwrinfobuf_obj;
282+
283+ /** @sysdata_obj: Object representing FW SYSDATA structure. */
284+ struct pvr_fw_object *sysdata_obj;
285+
286+ /** @power_sync_obj: Object representing power sync state. */
287+ struct pvr_fw_object *power_sync_obj;
288+
289+ /** @fault_page_obj: Object representing FW fault page. */
290+ struct pvr_fw_object *fault_page_obj;
291+
292+ /** @gpu_util_fwcb_obj: Object representing FW GPU utilisation control structure. */
293+ struct pvr_fw_object *gpu_util_fwcb_obj;
294+
295+ /** @runtime_cfg_obj: Object representing FW runtime config structure. */
296+ struct pvr_fw_object *runtime_cfg_obj;
297+
298+ /** @mmucache_sync_obj: Object used as the sync parameter in an MMU cache operation. */
299+ struct pvr_fw_object *mmucache_sync_obj;
300+};
301+
302+struct pvr_fw_device {
303+ /** @firmware: Handle to the firmware loaded into the device. */
304+ const struct firmware *firmware;
305+
306+ /** @header: Pointer to firmware header. */
307+ const struct pvr_fw_info_header *header;
308+
309+ /** @layout_entries: Pointer to firmware layout. */
310+ const struct pvr_fw_layout_entry *layout_entries;
311+
312+ /** @mem: Structure containing objects representing firmware memory allocations. */
313+ struct pvr_fw_mem mem;
314+
315+ /** @booted: %true if the firmware has been booted, %false otherwise. */
316+ bool booted;
317+
318+ /**
319+ * @processor_type: FW processor type for this device. Must be one of
320+ * %PVR_FW_PROCESSOR_TYPE_*.
321+ */
322+ u16 processor_type;
323+
324+ /** @funcs: Function table for the FW processor used by this device. */
325+ const struct pvr_fw_defs *defs;
326+
327+ /** @processor_data: Pointer to data specific to FW processor. */
328+ union {
329+ /** @mips_data: Pointer to MIPS-specific data. */
330+ struct pvr_fw_mips_data *mips_data;
331+ } processor_data;
332+
333+ /** @fw_heap_info: Firmware heap information. */
334+ struct {
335+ /** @gpu_addr: Base address of firmware heap in GPU address space. */
336+ u64 gpu_addr;
337+
338+ /** @size: Size of main area of heap. */
339+ u32 size;
340+
341+ /** @offset_mask: Mask for offsets within FW heap. */
342+ u32 offset_mask;
343+
344+ /** @raw_size: Raw size of heap, including reserved areas. */
345+ u32 raw_size;
346+
347+ /** @log2_size: Log2 of raw size of heap. */
348+ u32 log2_size;
349+
350+ /** @config_offset: Offset of config area within heap. */
351+ u32 config_offset;
352+
353+ /** @reserved_size: Size of reserved area in heap. */
354+ u32 reserved_size;
355+ } fw_heap_info;
356+
357+ /** @fw_mm: Firmware address space allocator. */
358+ struct drm_mm fw_mm;
359+
360+ /** @fw_mm_lock: Lock protecting access to &fw_mm. */
361+ spinlock_t fw_mm_lock;
362+
363+ /** @fw_mm_base: Base address of address space managed by @fw_mm. */
364+ u64 fw_mm_base;
365+
366+ /**
367+ * @fwif_connection_ctl: Pointer to CPU mapping of FWIF connection
368+ * control structure.
369+ */
370+ struct rogue_fwif_connection_ctl *fwif_connection_ctl;
371+
372+ /** @fwif_sysinit: Pointer to CPU mapping of FW SYSINIT structure. */
373+ struct rogue_fwif_sysinit *fwif_sysinit;
374+
375+ /** @fwif_sysdata: Pointer to CPU mapping of FW SYSDATA structure. */
376+ struct rogue_fwif_sysdata *fwif_sysdata;
377+
378+ /** @fwif_osinit: Pointer to CPU mapping of FW OSINIT structure. */
379+ struct rogue_fwif_osinit *fwif_osinit;
380+
381+ /** @fwif_osdata: Pointer to CPU mapping of FW OSDATA structure. */
382+ struct rogue_fwif_osdata *fwif_osdata;
383+
384+ /** @power_sync: Pointer to CPU mapping of power sync state. */
385+ u32 *power_sync;
386+
387+ /** @hwrinfobuf: Pointer to CPU mapping of FW HWR info buffer. */
388+ struct rogue_fwif_hwrinfobuf *hwrinfobuf;
389+
390+ /** @fw_trace: Device firmware trace buffer state. */
391+ struct pvr_fw_trace fw_trace;
392+
393+ /** @fw_objs: Structure tracking FW objects. */
394+ struct {
395+ /** @list: Head of FW object list. */
396+ struct list_head list;
397+
398+ /** @lock: Lock protecting access to FW object list. */
399+ struct mutex lock;
400+ } fw_objs;
401+};
402+
403+#define pvr_fw_irq_read_reg(pvr_dev, name) \
404+ pvr_cr_read32((pvr_dev), (pvr_dev)->fw_dev.defs->irq.name ## _reg)
405+
406+#define pvr_fw_irq_write_reg(pvr_dev, name, value) \
407+ pvr_cr_write32((pvr_dev), (pvr_dev)->fw_dev.defs->irq.name ## _reg, value)
408+
409+#define pvr_fw_irq_pending(pvr_dev) \
410+ (pvr_fw_irq_read_reg(pvr_dev, status) & (pvr_dev)->fw_dev.defs->irq.event_mask)
411+
412+#define pvr_fw_irq_clear(pvr_dev) \
413+ pvr_fw_irq_write_reg(pvr_dev, clear, (pvr_dev)->fw_dev.defs->irq.clear_mask)
414+
415+#define pvr_fw_irq_enable(pvr_dev) \
416+ pvr_fw_irq_write_reg(pvr_dev, enable, (pvr_dev)->fw_dev.defs->irq.event_mask)
417+
418+#define pvr_fw_irq_disable(pvr_dev) \
419+ pvr_fw_irq_write_reg(pvr_dev, enable, 0)
420+
421+extern const struct pvr_fw_defs pvr_fw_defs_meta;
422+extern const struct pvr_fw_defs pvr_fw_defs_mips;
423+
424+int pvr_fw_validate_init_device_info(struct pvr_device *pvr_dev);
425+int pvr_fw_init(struct pvr_device *pvr_dev);
426+void pvr_fw_fini(struct pvr_device *pvr_dev);
427+
428+int pvr_wait_for_fw_boot(struct pvr_device *pvr_dev);
429+
430+int
431+pvr_fw_hard_reset(struct pvr_device *pvr_dev);
432+
433+void pvr_fw_mts_schedule(struct pvr_device *pvr_dev, u32 val);
434+
435+void
436+pvr_fw_heap_info_init(struct pvr_device *pvr_dev, u32 log2_size, u32 reserved_size);
437+
438+const struct pvr_fw_layout_entry *
439+pvr_fw_find_layout_entry(struct pvr_device *pvr_dev, enum pvr_fw_section_id id);
440+int
441+pvr_fw_find_mmu_segment(struct pvr_device *pvr_dev, u32 addr, u32 size, void *fw_code_ptr,
442+ void *fw_data_ptr, void *fw_core_code_ptr, void *fw_core_data_ptr,
443+ void **host_addr_out);
444+
445+int
446+pvr_fw_structure_cleanup(struct pvr_device *pvr_dev, u32 type, struct pvr_fw_object *fw_obj,
447+ u32 offset);
448+
449+int pvr_fw_object_create(struct pvr_device *pvr_dev, size_t size, u64 flags,
450+ void (*init)(void *cpu_ptr, void *priv), void *init_priv,
451+ struct pvr_fw_object **pvr_obj_out);
452+
453+void *pvr_fw_object_create_and_map(struct pvr_device *pvr_dev, size_t size, u64 flags,
454+ void (*init)(void *cpu_ptr, void *priv),
455+ void *init_priv, struct pvr_fw_object **pvr_obj_out);
456+
457+void *
458+pvr_fw_object_create_and_map_offset(struct pvr_device *pvr_dev, u32 dev_offset, size_t size,
459+ u64 flags, void (*init)(void *cpu_ptr, void *priv),
460+ void *init_priv, struct pvr_fw_object **pvr_obj_out);
461+
462+static __always_inline void *
463+pvr_fw_object_vmap(struct pvr_fw_object *fw_obj)
464+{
465+ return pvr_gem_object_vmap(fw_obj->gem);
466+}
467+
468+static __always_inline void
469+pvr_fw_object_vunmap(struct pvr_fw_object *fw_obj)
470+{
471+ pvr_gem_object_vunmap(fw_obj->gem);
472+}
473+
474+void pvr_fw_object_destroy(struct pvr_fw_object *fw_obj);
475+
476+static __always_inline void
477+pvr_fw_object_unmap_and_destroy(struct pvr_fw_object *fw_obj)
478+{
479+ pvr_fw_object_vunmap(fw_obj);
480+ pvr_fw_object_destroy(fw_obj);
481+}
482+
483+/**
484+ * pvr_fw_get_dma_addr() - Get DMA address for given offset in firmware object
485+ * @fw_obj: Pointer to object to lookup address in.
486+ * @offset: Offset within object to lookup address at.
487+ * @dma_addr_out: Pointer to location to store DMA address.
488+ *
489+ * Returns:
490+ * * 0 on success, or
491+ * * -%EINVAL if object is not currently backed, or if @offset is out of valid
492+ * range for this object.
493+ */
494+static __always_inline int
495+pvr_fw_object_get_dma_addr(struct pvr_fw_object *fw_obj, u32 offset, dma_addr_t *dma_addr_out)
496+{
497+ return pvr_gem_get_dma_addr(fw_obj->gem, offset, dma_addr_out);
498+}
499+
500+void pvr_fw_object_get_fw_addr_offset(struct pvr_fw_object *fw_obj, u32 offset, u32 *fw_addr_out);
501+
502+static __always_inline void
503+pvr_fw_object_get_fw_addr(struct pvr_fw_object *fw_obj, u32 *fw_addr_out)
504+{
505+ pvr_fw_object_get_fw_addr_offset(fw_obj, 0, fw_addr_out);
506+}
507+
508+#endif /* PVR_FW_H */
drivers/gpu/drm/imagination/pvr_fw_info.h+135−0
@@ -0,0 +1,135 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_FW_INFO_H
5+#define PVR_FW_INFO_H
6+
7+#include <linux/bits.h>
8+#include <linux/sizes.h>
9+#include <linux/types.h>
10+
11+/*
12+ * Firmware binary block unit in bytes.
13+ * Raw data stored in FW binary will be aligned to this size.
14+ */
15+#define FW_BLOCK_SIZE SZ_4K
16+
17+/* Maximum number of entries in firmware layout table. */
18+#define PVR_FW_INFO_MAX_NUM_ENTRIES 8
19+
20+enum pvr_fw_section_id {
21+ META_CODE = 0,
22+ META_PRIVATE_DATA,
23+ META_COREMEM_CODE,
24+ META_COREMEM_DATA,
25+ MIPS_CODE,
26+ MIPS_EXCEPTIONS_CODE,
27+ MIPS_BOOT_CODE,
28+ MIPS_PRIVATE_DATA,
29+ MIPS_BOOT_DATA,
30+ MIPS_STACK,
31+ RISCV_UNCACHED_CODE,
32+ RISCV_CACHED_CODE,
33+ RISCV_PRIVATE_DATA,
34+ RISCV_COREMEM_CODE,
35+ RISCV_COREMEM_DATA,
36+};
37+
38+enum pvr_fw_section_type {
39+ NONE = 0,
40+ FW_CODE,
41+ FW_DATA,
42+ FW_COREMEM_CODE,
43+ FW_COREMEM_DATA,
44+};
45+
46+/*
47+ * FW binary format with FW info attached:
48+ *
49+ * Contents Offset
50+ * +-----------------+
51+ * | | 0
52+ * | |
53+ * | Original binary |
54+ * | file |
55+ * | (.ldr/.elf) |
56+ * | |
57+ * | |
58+ * +-----------------+
59+ * | Device info | FILE_SIZE - 4K - device_info_size
60+ * +-----------------+
61+ * | FW info header | FILE_SIZE - 4K
62+ * +-----------------+
63+ * | |
64+ * | FW layout table |
65+ * | |
66+ * +-----------------+
67+ * FILE_SIZE
68+ */
69+
70+#define PVR_FW_INFO_VERSION 3
71+
72+#define PVR_FW_FLAGS_OPEN_SOURCE BIT(0)
73+
74+/** struct pvr_fw_info_header - Firmware header */
75+struct pvr_fw_info_header {
76+ /** @info_version: FW info header version. */
77+ u32 info_version;
78+ /** @header_len: Header length. */
79+ u32 header_len;
80+ /** @layout_entry_num: Number of entries in the layout table. */
81+ u32 layout_entry_num;
82+ /** @layout_entry_size: Size of an entry in the layout table. */
83+ u32 layout_entry_size;
84+ /** @bvnc: GPU ID supported by firmware. */
85+ aligned_u64 bvnc;
86+ /** @fw_page_size: Page size of processor on which firmware executes. */
87+ u32 fw_page_size;
88+ /** @flags: Compatibility flags. */
89+ u32 flags;
90+ /** @fw_version_major: Firmware major version number. */
91+ u16 fw_version_major;
92+ /** @fw_version_minor: Firmware minor version number. */
93+ u16 fw_version_minor;
94+ /** @fw_version_build: Firmware build number. */
95+ u32 fw_version_build;
96+ /** @device_info_size: Size of device info structure. */
97+ u32 device_info_size;
98+ /** @padding: Padding. */
99+ u32 padding;
100+};
101+
102+/**
103+ * struct pvr_fw_layout_entry - Entry in firmware layout table, describing a
104+ * section of the firmware image
105+ */
106+struct pvr_fw_layout_entry {
107+ /** @id: Section ID. */
108+ enum pvr_fw_section_id id;
109+ /** @type: Section type. */
110+ enum pvr_fw_section_type type;
111+ /** @base_addr: Base address of section in FW address space. */
112+ u32 base_addr;
113+ /** @max_size: Maximum size of section, in bytes. */
114+ u32 max_size;
115+ /** @alloc_size: Allocation size of section, in bytes. */
116+ u32 alloc_size;
117+ /** @alloc_offset: Allocation offset of section. */
118+ u32 alloc_offset;
119+};
120+
121+/**
122+ * struct pvr_fw_device_info_header - Device information header.
123+ */
124+struct pvr_fw_device_info_header {
125+ /* BRN Mask size (in u64s). */
126+ u64 brn_mask_size;
127+ /* ERN Mask size (in u64s). */
128+ u64 ern_mask_size;
129+ /* Feature Mask size (in u64s). */
130+ u64 feature_mask_size;
131+ /* Feature Parameter size (in u64s). */
132+ u64 feature_param_size;
133+};
134+
135+#endif /* PVR_FW_INFO_H */
drivers/gpu/drm/imagination/pvr_fw_meta.c+554−0
@@ -0,0 +1,554 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_device.h"
5+#include "pvr_fw.h"
6+#include "pvr_fw_info.h"
7+#include "pvr_gem.h"
8+#include "pvr_rogue_cr_defs.h"
9+#include "pvr_rogue_meta.h"
10+#include "pvr_vm.h"
11+
12+#include <linux/compiler.h>
13+#include <linux/delay.h>
14+#include <linux/firmware.h>
15+#include <linux/ktime.h>
16+#include <linux/types.h>
17+
18+#define ROGUE_FW_HEAP_META_SHIFT 25 /* 32 MB */
19+
20+#define POLL_TIMEOUT_USEC 1000000
21+
22+/**
23+ * pvr_meta_cr_read32() - Read a META register via the Slave Port
24+ * @pvr_dev: Device pointer.
25+ * @reg_addr: Address of register to read.
26+ * @reg_value_out: Pointer to location to store register value.
27+ *
28+ * Returns:
29+ * * 0 on success, or
30+ * * Any error returned by pvr_cr_poll_reg32().
31+ */
32+int
33+pvr_meta_cr_read32(struct pvr_device *pvr_dev, u32 reg_addr, u32 *reg_value_out)
34+{
35+ int err;
36+
37+ /* Wait for Slave Port to be Ready. */
38+ err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_META_SP_MSLVCTRL1,
39+ ROGUE_CR_META_SP_MSLVCTRL1_READY_EN |
40+ ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_EN,
41+ ROGUE_CR_META_SP_MSLVCTRL1_READY_EN |
42+ ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_EN,
43+ POLL_TIMEOUT_USEC);
44+ if (err)
45+ return err;
46+
47+ /* Issue a Read. */
48+ pvr_cr_write32(pvr_dev, ROGUE_CR_META_SP_MSLVCTRL0,
49+ reg_addr | ROGUE_CR_META_SP_MSLVCTRL0_RD_EN);
50+ (void)pvr_cr_read32(pvr_dev, ROGUE_CR_META_SP_MSLVCTRL0); /* Fence write. */
51+
52+ /* Wait for Slave Port to be Ready. */
53+ err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_META_SP_MSLVCTRL1,
54+ ROGUE_CR_META_SP_MSLVCTRL1_READY_EN |
55+ ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_EN,
56+ ROGUE_CR_META_SP_MSLVCTRL1_READY_EN |
57+ ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_EN,
58+ POLL_TIMEOUT_USEC);
59+ if (err)
60+ return err;
61+
62+ *reg_value_out = pvr_cr_read32(pvr_dev, ROGUE_CR_META_SP_MSLVDATAX);
63+
64+ return 0;
65+}
66+
67+static int
68+pvr_meta_wrapper_init(struct pvr_device *pvr_dev)
69+{
70+ u64 garten_config;
71+
72+ /* Configure META to Master boot. */
73+ pvr_cr_write64(pvr_dev, ROGUE_CR_META_BOOT, ROGUE_CR_META_BOOT_MODE_EN);
74+
75+ /* Set Garten IDLE to META idle and Set the Garten Wrapper BIF Fence address. */
76+
77+ /* Garten IDLE bit controlled by META. */
78+ garten_config = ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_IDLE_CTRL_META;
79+
80+ /* The fence addr is set during the fw init sequence. */
81+
82+ /* Set PC = 0 for fences. */
83+ garten_config &=
84+ ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_PC_BASE_CLRMSK;
85+ garten_config |=
86+ (u64)MMU_CONTEXT_MAPPING_FWPRIV
87+ << ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_PC_BASE_SHIFT;
88+
89+ /* Set SLC DM=META. */
90+ garten_config |= ((u64)ROGUE_FW_SEGMMU_META_BIFDM_ID)
91+ << ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_DM_SHIFT;
92+
93+ pvr_cr_write64(pvr_dev, ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG, garten_config);
94+
95+ return 0;
96+}
97+
98+static __always_inline void
99+add_boot_arg(u32 **boot_conf, u32 param, u32 data)
100+{
101+ *(*boot_conf)++ = param;
102+ *(*boot_conf)++ = data;
103+}
104+
105+static int
106+meta_ldr_cmd_loadmem(struct drm_device *drm_dev, const u8 *fw,
107+ struct rogue_meta_ldr_l1_data_blk *l1_data, u32 coremem_size, u8 *fw_code_ptr,
108+ u8 *fw_data_ptr, u8 *fw_core_code_ptr, u8 *fw_core_data_ptr, const u32 fw_size)
109+{
110+ struct rogue_meta_ldr_l2_data_blk *l2_block =
111+ (struct rogue_meta_ldr_l2_data_blk *)(fw +
112+ l1_data->cmd_data[1]);
113+ struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
114+ u32 offset = l1_data->cmd_data[0];
115+ u32 data_size;
116+ void *write_addr;
117+ int err;
118+
119+ /* Verify header is within bounds. */
120+ if (((u8 *)l2_block - fw) >= fw_size || ((u8 *)(l2_block + 1) - fw) >= fw_size)
121+ return -EINVAL;
122+
123+ data_size = l2_block->length - 6 /* L2 Tag length and checksum */;
124+
125+ /* Verify data is within bounds. */
126+ if (((u8 *)l2_block->block_data - fw) >= fw_size ||
127+ ((((u8 *)l2_block->block_data) + data_size) - fw) >= fw_size)
128+ return -EINVAL;
129+
130+ if (!ROGUE_META_IS_COREMEM_CODE(offset, coremem_size) &&
131+ !ROGUE_META_IS_COREMEM_DATA(offset, coremem_size)) {
132+ /* Global range is aliased to local range */
133+ offset &= ~META_MEM_GLOBAL_RANGE_BIT;
134+ }
135+
136+ err = pvr_fw_find_mmu_segment(pvr_dev, offset, data_size, fw_code_ptr, fw_data_ptr,
137+ fw_core_code_ptr, fw_core_data_ptr, &write_addr);
138+ if (err) {
139+ drm_err(drm_dev,
140+ "Addr 0x%x (size: %d) not found in any firmware segment",
141+ offset, data_size);
142+ return err;
143+ }
144+
145+ memcpy(write_addr, l2_block->block_data, data_size);
146+
147+ return 0;
148+}
149+
150+static int
151+meta_ldr_cmd_zeromem(struct drm_device *drm_dev,
152+ struct rogue_meta_ldr_l1_data_blk *l1_data, u32 coremem_size,
153+ u8 *fw_code_ptr, u8 *fw_data_ptr, u8 *fw_core_code_ptr, u8 *fw_core_data_ptr)
154+{
155+ struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
156+ u32 offset = l1_data->cmd_data[0];
157+ u32 byte_count = l1_data->cmd_data[1];
158+ void *write_addr;
159+ int err;
160+
161+ if (ROGUE_META_IS_COREMEM_DATA(offset, coremem_size)) {
162+ /* cannot zero coremem directly */
163+ return 0;
164+ }
165+
166+ /* Global range is aliased to local range */
167+ offset &= ~META_MEM_GLOBAL_RANGE_BIT;
168+
169+ err = pvr_fw_find_mmu_segment(pvr_dev, offset, byte_count, fw_code_ptr, fw_data_ptr,
170+ fw_core_code_ptr, fw_core_data_ptr, &write_addr);
171+ if (err) {
172+ drm_err(drm_dev,
173+ "Addr 0x%x (size: %d) not found in any firmware segment",
174+ offset, byte_count);
175+ return err;
176+ }
177+
178+ memset(write_addr, 0, byte_count);
179+
180+ return 0;
181+}
182+
183+static int
184+meta_ldr_cmd_config(struct drm_device *drm_dev, const u8 *fw,
185+ struct rogue_meta_ldr_l1_data_blk *l1_data,
186+ const u32 fw_size, u32 **boot_conf_ptr)
187+{
188+ struct rogue_meta_ldr_l2_data_blk *l2_block =
189+ (struct rogue_meta_ldr_l2_data_blk *)(fw +
190+ l1_data->cmd_data[0]);
191+ struct rogue_meta_ldr_cfg_blk *config_command;
192+ u32 l2_block_size;
193+ u32 curr_block_size = 0;
194+ u32 *boot_conf = boot_conf_ptr ? *boot_conf_ptr : NULL;
195+
196+ /* Verify block header is within bounds. */
197+ if (((u8 *)l2_block - fw) >= fw_size || ((u8 *)(l2_block + 1) - fw) >= fw_size)
198+ return -EINVAL;
199+
200+ l2_block_size = l2_block->length - 6 /* L2 Tag length and checksum */;
201+ config_command = (struct rogue_meta_ldr_cfg_blk *)l2_block->block_data;
202+
203+ if (((u8 *)config_command - fw) >= fw_size ||
204+ ((((u8 *)config_command) + l2_block_size) - fw) >= fw_size)
205+ return -EINVAL;
206+
207+ while (l2_block_size >= 12) {
208+ if (config_command->type != ROGUE_META_LDR_CFG_WRITE)
209+ return -EINVAL;
210+
211+ /*
212+ * Only write to bootloader if we got a valid pointer to the FW
213+ * code allocation.
214+ */
215+ if (boot_conf) {
216+ u32 register_offset = config_command->block_data[0];
217+ u32 register_value = config_command->block_data[1];
218+
219+ /* Do register write */
220+ add_boot_arg(&boot_conf, register_offset,
221+ register_value);
222+ }
223+
224+ curr_block_size = 12;
225+ l2_block_size -= curr_block_size;
226+ config_command = (struct rogue_meta_ldr_cfg_blk
227+ *)((uintptr_t)config_command +
228+ curr_block_size);
229+ }
230+
231+ if (boot_conf_ptr)
232+ *boot_conf_ptr = boot_conf;
233+
234+ return 0;
235+}
236+
237+/**
238+ * process_ldr_command_stream() - Process LDR firmware image and populate
239+ * firmware sections
240+ * @pvr_dev: Device pointer.
241+ * @fw: Pointer to firmware image.
242+ * @fw_code_ptr: Pointer to FW code section.
243+ * @fw_data_ptr: Pointer to FW data section.
244+ * @fw_core_code_ptr: Pointer to FW coremem code section.
245+ * @fw_core_data_ptr: Pointer to FW coremem data section.
246+ * @boot_conf_ptr: Pointer to boot config argument pointer.
247+ *
248+ * Returns :
249+ * * 0 on success, or
250+ * * -EINVAL on any error in LDR command stream.
251+ */
252+static int
253+process_ldr_command_stream(struct pvr_device *pvr_dev, const u8 *fw, u8 *fw_code_ptr,
254+ u8 *fw_data_ptr, u8 *fw_core_code_ptr,
255+ u8 *fw_core_data_ptr, u32 **boot_conf_ptr)
256+{
257+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
258+ struct rogue_meta_ldr_block_hdr *ldr_header =
259+ (struct rogue_meta_ldr_block_hdr *)fw;
260+ struct rogue_meta_ldr_l1_data_blk *l1_data =
261+ (struct rogue_meta_ldr_l1_data_blk *)(fw + ldr_header->sl_data);
262+ const u32 fw_size = pvr_dev->fw_dev.firmware->size;
263+ int err;
264+
265+ u32 *boot_conf = boot_conf_ptr ? *boot_conf_ptr : NULL;
266+ u32 coremem_size;
267+
268+ err = PVR_FEATURE_VALUE(pvr_dev, meta_coremem_size, &coremem_size);
269+ if (err)
270+ return err;
271+
272+ coremem_size *= SZ_1K;
273+
274+ while (l1_data) {
275+ /* Verify block header is within bounds. */
276+ if (((u8 *)l1_data - fw) >= fw_size || ((u8 *)(l1_data + 1) - fw) >= fw_size)
277+ return -EINVAL;
278+
279+ if (ROGUE_META_LDR_BLK_IS_COMMENT(l1_data->cmd)) {
280+ /* Don't process comment blocks */
281+ goto next_block;
282+ }
283+
284+ switch (l1_data->cmd & ROGUE_META_LDR_CMD_MASK)
285+ case ROGUE_META_LDR_CMD_LOADMEM: {
286+ err = meta_ldr_cmd_loadmem(drm_dev, fw, l1_data,
287+ coremem_size,
288+ fw_code_ptr, fw_data_ptr,
289+ fw_core_code_ptr,
290+ fw_core_data_ptr, fw_size);
291+ if (err)
292+ return err;
293+ break;
294+
295+ case ROGUE_META_LDR_CMD_START_THREADS:
296+ /* Don't process this block */
297+ break;
298+
299+ case ROGUE_META_LDR_CMD_ZEROMEM:
300+ err = meta_ldr_cmd_zeromem(drm_dev, l1_data,
301+ coremem_size,
302+ fw_code_ptr, fw_data_ptr,
303+ fw_core_code_ptr,
304+ fw_core_data_ptr);
305+ if (err)
306+ return err;
307+ break;
308+
309+ case ROGUE_META_LDR_CMD_CONFIG:
310+ err = meta_ldr_cmd_config(drm_dev, fw, l1_data, fw_size,
311+ &boot_conf);
312+ if (err)
313+ return err;
314+ break;
315+
316+ default:
317+ return -EINVAL;
318+ }
319+
320+next_block:
321+ if (l1_data->next == 0xFFFFFFFF)
322+ break;
323+
324+ l1_data = (struct rogue_meta_ldr_l1_data_blk *)(fw +
325+ l1_data->next);
326+ }
327+
328+ if (boot_conf_ptr)
329+ *boot_conf_ptr = boot_conf;
330+
331+ return 0;
332+}
333+
334+static void
335+configure_seg_id(u64 seg_out_addr, u32 seg_base, u32 seg_limit, u32 seg_id,
336+ u32 **boot_conf_ptr)
337+{
338+ u32 seg_out_addr0 = seg_out_addr & 0x00000000FFFFFFFFUL;
339+ u32 seg_out_addr1 = (seg_out_addr >> 32) & 0x00000000FFFFFFFFUL;
340+ u32 *boot_conf = *boot_conf_ptr;
341+
342+ /* META segments have a minimum size. */
343+ u32 limit_off = max(seg_limit, ROGUE_FW_SEGMMU_ALIGN);
344+
345+ /* The limit is an offset, therefore off = size - 1. */
346+ limit_off -= 1;
347+
348+ seg_base |= ROGUE_FW_SEGMMU_ALLTHRS_WRITEABLE;
349+
350+ add_boot_arg(&boot_conf, META_CR_MMCU_SEGMENT_N_BASE(seg_id), seg_base);
351+ add_boot_arg(&boot_conf, META_CR_MMCU_SEGMENT_N_LIMIT(seg_id), limit_off);
352+ add_boot_arg(&boot_conf, META_CR_MMCU_SEGMENT_N_OUTA0(seg_id), seg_out_addr0);
353+ add_boot_arg(&boot_conf, META_CR_MMCU_SEGMENT_N_OUTA1(seg_id), seg_out_addr1);
354+
355+ *boot_conf_ptr = boot_conf;
356+}
357+
358+static u64 get_fw_obj_gpu_addr(struct pvr_fw_object *fw_obj)
359+{
360+ struct pvr_device *pvr_dev = to_pvr_device(gem_from_pvr_gem(fw_obj->gem)->dev);
361+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
362+
363+ return fw_obj->fw_addr_offset + fw_dev->fw_heap_info.gpu_addr;
364+}
365+
366+static void
367+configure_seg_mmu(struct pvr_device *pvr_dev, u32 **boot_conf_ptr)
368+{
369+ const struct pvr_fw_layout_entry *layout_entries = pvr_dev->fw_dev.layout_entries;
370+ u32 num_layout_entries = pvr_dev->fw_dev.header->layout_entry_num;
371+ u64 seg_out_addr_top;
372+ u32 i;
373+
374+ seg_out_addr_top =
375+ ROGUE_FW_SEGMMU_OUTADDR_TOP_SLC(MMU_CONTEXT_MAPPING_FWPRIV,
376+ ROGUE_FW_SEGMMU_META_BIFDM_ID);
377+
378+ for (i = 0; i < num_layout_entries; i++) {
379+ /*
380+ * FW code is using the bootloader segment which is already
381+ * configured on boot. FW coremem code and data don't use the
382+ * segment MMU. Only the FW data segment needs to be configured.
383+ */
384+ if (layout_entries[i].type == FW_DATA) {
385+ u32 seg_id = ROGUE_FW_SEGMMU_DATA_ID;
386+ u64 seg_out_addr = get_fw_obj_gpu_addr(pvr_dev->fw_dev.mem.data_obj);
387+
388+ seg_out_addr += layout_entries[i].alloc_offset;
389+ seg_out_addr |= seg_out_addr_top;
390+
391+ /* Write the sequence to the bootldr. */
392+ configure_seg_id(seg_out_addr,
393+ layout_entries[i].base_addr,
394+ layout_entries[i].alloc_size, seg_id,
395+ boot_conf_ptr);
396+
397+ break;
398+ }
399+ }
400+}
401+
402+static void
403+configure_meta_caches(u32 **boot_conf_ptr)
404+{
405+ u32 *boot_conf = *boot_conf_ptr;
406+ u32 d_cache_t0, i_cache_t0;
407+ u32 d_cache_t1, i_cache_t1;
408+ u32 d_cache_t2, i_cache_t2;
409+ u32 d_cache_t3, i_cache_t3;
410+
411+ /* Initialise I/Dcache settings */
412+ d_cache_t0 = META_CR_SYSC_DCPARTX_CACHED_WRITE_ENABLE;
413+ d_cache_t1 = META_CR_SYSC_DCPARTX_CACHED_WRITE_ENABLE;
414+ d_cache_t2 = META_CR_SYSC_DCPARTX_CACHED_WRITE_ENABLE;
415+ d_cache_t3 = META_CR_SYSC_DCPARTX_CACHED_WRITE_ENABLE;
416+ i_cache_t0 = 0;
417+ i_cache_t1 = 0;
418+ i_cache_t2 = 0;
419+ i_cache_t3 = 0;
420+
421+ d_cache_t0 |= META_CR_SYSC_XCPARTX_LOCAL_ADDR_FULL_CACHE;
422+ i_cache_t0 |= META_CR_SYSC_XCPARTX_LOCAL_ADDR_FULL_CACHE;
423+
424+ /* Local region MMU enhanced bypass: WIN-3 mode for code and data caches */
425+ add_boot_arg(&boot_conf, META_CR_MMCU_LOCAL_EBCTRL,
426+ META_CR_MMCU_LOCAL_EBCTRL_ICWIN |
427+ META_CR_MMCU_LOCAL_EBCTRL_DCWIN);
428+
429+ /* Data cache partitioning thread 0 to 3 */
430+ add_boot_arg(&boot_conf, META_CR_SYSC_DCPART(0), d_cache_t0);
431+ add_boot_arg(&boot_conf, META_CR_SYSC_DCPART(1), d_cache_t1);
432+ add_boot_arg(&boot_conf, META_CR_SYSC_DCPART(2), d_cache_t2);
433+ add_boot_arg(&boot_conf, META_CR_SYSC_DCPART(3), d_cache_t3);
434+
435+ /* Enable data cache hits */
436+ add_boot_arg(&boot_conf, META_CR_MMCU_DCACHE_CTRL,
437+ META_CR_MMCU_XCACHE_CTRL_CACHE_HITS_EN);
438+
439+ /* Instruction cache partitioning thread 0 to 3 */
440+ add_boot_arg(&boot_conf, META_CR_SYSC_ICPART(0), i_cache_t0);
441+ add_boot_arg(&boot_conf, META_CR_SYSC_ICPART(1), i_cache_t1);
442+ add_boot_arg(&boot_conf, META_CR_SYSC_ICPART(2), i_cache_t2);
443+ add_boot_arg(&boot_conf, META_CR_SYSC_ICPART(3), i_cache_t3);
444+
445+ /* Enable instruction cache hits */
446+ add_boot_arg(&boot_conf, META_CR_MMCU_ICACHE_CTRL,
447+ META_CR_MMCU_XCACHE_CTRL_CACHE_HITS_EN);
448+
449+ add_boot_arg(&boot_conf, 0x040000C0, 0);
450+
451+ *boot_conf_ptr = boot_conf;
452+}
453+
454+static int
455+pvr_meta_fw_process(struct pvr_device *pvr_dev, const u8 *fw,
456+ u8 *fw_code_ptr, u8 *fw_data_ptr, u8 *fw_core_code_ptr, u8 *fw_core_data_ptr,
457+ u32 core_code_alloc_size)
458+{
459+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
460+ u32 *boot_conf;
461+ int err;
462+
463+ boot_conf = ((u32 *)fw_code_ptr) + ROGUE_FW_BOOTLDR_CONF_OFFSET;
464+
465+ /* Slave port and JTAG accesses are privileged. */
466+ add_boot_arg(&boot_conf, META_CR_SYSC_JTAG_THREAD,
467+ META_CR_SYSC_JTAG_THREAD_PRIV_EN);
468+
469+ configure_seg_mmu(pvr_dev, &boot_conf);
470+
471+ /* Populate FW sections from LDR image. */
472+ err = process_ldr_command_stream(pvr_dev, fw, fw_code_ptr, fw_data_ptr, fw_core_code_ptr,
473+ fw_core_data_ptr, &boot_conf);
474+ if (err)
475+ return err;
476+
477+ configure_meta_caches(&boot_conf);
478+
479+ /* End argument list. */
480+ add_boot_arg(&boot_conf, 0, 0);
481+
482+ if (fw_dev->mem.core_code_obj) {
483+ u32 core_code_fw_addr;
484+
485+ pvr_fw_object_get_fw_addr(fw_dev->mem.core_code_obj, &core_code_fw_addr);
486+ add_boot_arg(&boot_conf, core_code_fw_addr, core_code_alloc_size);
487+ } else {
488+ add_boot_arg(&boot_conf, 0, 0);
489+ }
490+ /* None of the cores supported by this driver have META DMA. */
491+ add_boot_arg(&boot_conf, 0, 0);
492+
493+ return 0;
494+}
495+
496+static int
497+pvr_meta_init(struct pvr_device *pvr_dev)
498+{
499+ pvr_fw_heap_info_init(pvr_dev, ROGUE_FW_HEAP_META_SHIFT, 0);
500+
501+ return 0;
502+}
503+
504+static u32
505+pvr_meta_get_fw_addr_with_offset(struct pvr_fw_object *fw_obj, u32 offset)
506+{
507+ u32 fw_addr = fw_obj->fw_addr_offset + offset + ROGUE_FW_SEGMMU_DATA_BASE_ADDRESS;
508+
509+ /* META cacheability is determined by address. */
510+ if (fw_obj->gem->flags & PVR_BO_FW_FLAGS_DEVICE_UNCACHED)
511+ fw_addr |= ROGUE_FW_SEGMMU_DATA_META_UNCACHED |
512+ ROGUE_FW_SEGMMU_DATA_VIVT_SLC_UNCACHED;
513+
514+ return fw_addr;
515+}
516+
517+static int
518+pvr_meta_vm_map(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj)
519+{
520+ struct pvr_gem_object *pvr_obj = fw_obj->gem;
521+
522+ return pvr_vm_map(pvr_dev->kernel_vm_ctx, pvr_obj, 0, fw_obj->fw_mm_node.start,
523+ pvr_gem_object_size(pvr_obj));
524+}
525+
526+static void
527+pvr_meta_vm_unmap(struct pvr_device *pvr_dev, struct pvr_fw_object *fw_obj)
528+{
529+ pvr_vm_unmap(pvr_dev->kernel_vm_ctx, fw_obj->fw_mm_node.start,
530+ fw_obj->fw_mm_node.size);
531+}
532+
533+static bool
534+pvr_meta_has_fixed_data_addr(void)
535+{
536+ return false;
537+}
538+
539+const struct pvr_fw_defs pvr_fw_defs_meta = {
540+ .init = pvr_meta_init,
541+ .fw_process = pvr_meta_fw_process,
542+ .vm_map = pvr_meta_vm_map,
543+ .vm_unmap = pvr_meta_vm_unmap,
544+ .get_fw_addr_with_offset = pvr_meta_get_fw_addr_with_offset,
545+ .wrapper_init = pvr_meta_wrapper_init,
546+ .has_fixed_data_addr = pvr_meta_has_fixed_data_addr,
547+ .irq = {
548+ .enable_reg = ROGUE_CR_META_SP_MSLVIRQENABLE,
549+ .status_reg = ROGUE_CR_META_SP_MSLVIRQSTATUS,
550+ .clear_reg = ROGUE_CR_META_SP_MSLVIRQSTATUS,
551+ .event_mask = ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT2_EN,
552+ .clear_mask = ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT2_CLRMSK,
553+ },
554+};
drivers/gpu/drm/imagination/pvr_fw_meta.h+14−0
@@ -0,0 +1,14 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_FW_META_H
5+#define PVR_FW_META_H
6+
7+#include <linux/types.h>
8+
9+/* Forward declaration from pvr_device.h */
10+struct pvr_device;
11+
12+int pvr_meta_cr_read32(struct pvr_device *pvr_dev, u32 reg_addr, u32 *reg_value_out);
13+
14+#endif /* PVR_FW_META_H */
drivers/gpu/drm/imagination/pvr_fw_mips.c+252−0
@@ -0,0 +1,252 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_device.h"
5+#include "pvr_fw.h"
6+#include "pvr_fw_mips.h"
7+#include "pvr_gem.h"
8+#include "pvr_rogue_mips.h"
9+#include "pvr_vm_mips.h"
10+
11+#include <linux/elf.h>
12+#include <linux/err.h>
13+#include <linux/types.h>
14+
15+#define ROGUE_FW_HEAP_MIPS_BASE 0xC0000000
16+#define ROGUE_FW_HEAP_MIPS_SHIFT 24 /* 16 MB */
17+#define ROGUE_FW_HEAP_MIPS_RESERVED_SIZE SZ_1M
18+
19+/**
20+ * process_elf_command_stream() - Process ELF firmware image and populate
21+ * firmware sections
22+ * @pvr_dev: Device pointer.
23+ * @fw: Pointer to firmware image.
24+ * @fw_code_ptr: Pointer to FW code section.
25+ * @fw_data_ptr: Pointer to FW data section.
26+ * @fw_core_code_ptr: Pointer to FW coremem code section.
27+ * @fw_core_data_ptr: Pointer to FW coremem data section.
28+ *
29+ * Returns :
30+ * * 0 on success, or
31+ * * -EINVAL on any error in ELF command stream.
32+ */
33+static int
34+process_elf_command_stream(struct pvr_device *pvr_dev, const u8 *fw, u8 *fw_code_ptr,
35+ u8 *fw_data_ptr, u8 *fw_core_code_ptr, u8 *fw_core_data_ptr)
36+{
37+ struct elf32_hdr *header = (struct elf32_hdr *)fw;
38+ struct elf32_phdr *program_header = (struct elf32_phdr *)(fw + header->e_phoff);
39+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
40+ u32 entry;
41+ int err;
42+
43+ for (entry = 0; entry < header->e_phnum; entry++, program_header++) {
44+ void *write_addr;
45+
46+ /* Only consider loadable entries in the ELF segment table */
47+ if (program_header->p_type != PT_LOAD)
48+ continue;
49+
50+ err = pvr_fw_find_mmu_segment(pvr_dev, program_header->p_vaddr,
51+ program_header->p_memsz, fw_code_ptr, fw_data_ptr,
52+ fw_core_code_ptr, fw_core_data_ptr, &write_addr);
53+ if (err) {
54+ drm_err(drm_dev,
55+ "Addr 0x%x (size: %d) not found in any firmware segment",
56+ program_header->p_vaddr, program_header->p_memsz);
57+ return err;
58+ }
59+
60+ /* Write to FW allocation only if available */
61+ if (write_addr) {
62+ memcpy(write_addr, fw + program_header->p_offset,
63+ program_header->p_filesz);
64+
65+ memset((u8 *)write_addr + program_header->p_filesz, 0,
66+ program_header->p_memsz - program_header->p_filesz);
67+ }
68+ }
69+
70+ return 0;
71+}
72+
73+static int
74+pvr_mips_init(struct pvr_device *pvr_dev)
75+{
76+ pvr_fw_heap_info_init(pvr_dev, ROGUE_FW_HEAP_MIPS_SHIFT, ROGUE_FW_HEAP_MIPS_RESERVED_SIZE);
77+
78+ return pvr_vm_mips_init(pvr_dev);
79+}
80+
81+static void
82+pvr_mips_fini(struct pvr_device *pvr_dev)
83+{
84+ pvr_vm_mips_fini(pvr_dev);
85+}
86+
87+static int
88+pvr_mips_fw_process(struct pvr_device *pvr_dev, const u8 *fw,
89+ u8 *fw_code_ptr, u8 *fw_data_ptr, u8 *fw_core_code_ptr, u8 *fw_core_data_ptr,
90+ u32 core_code_alloc_size)
91+{
92+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
93+ struct pvr_fw_mips_data *mips_data = fw_dev->processor_data.mips_data;
94+ const struct pvr_fw_layout_entry *boot_code_entry;
95+ const struct pvr_fw_layout_entry *boot_data_entry;
96+ const struct pvr_fw_layout_entry *exception_code_entry;
97+ const struct pvr_fw_layout_entry *stack_entry;
98+ struct rogue_mipsfw_boot_data *boot_data;
99+ dma_addr_t dma_addr;
100+ u32 page_nr;
101+ int err;
102+
103+ err = process_elf_command_stream(pvr_dev, fw, fw_code_ptr, fw_data_ptr, fw_core_code_ptr,
104+ fw_core_data_ptr);
105+ if (err)
106+ return err;
107+
108+ boot_code_entry = pvr_fw_find_layout_entry(pvr_dev, MIPS_BOOT_CODE);
109+ boot_data_entry = pvr_fw_find_layout_entry(pvr_dev, MIPS_BOOT_DATA);
110+ exception_code_entry = pvr_fw_find_layout_entry(pvr_dev, MIPS_EXCEPTIONS_CODE);
111+ if (!boot_code_entry || !boot_data_entry || !exception_code_entry)
112+ return -EINVAL;
113+
114+ WARN_ON(pvr_gem_get_dma_addr(fw_dev->mem.code_obj->gem, boot_code_entry->alloc_offset,
115+ &mips_data->boot_code_dma_addr));
116+ WARN_ON(pvr_gem_get_dma_addr(fw_dev->mem.data_obj->gem, boot_data_entry->alloc_offset,
117+ &mips_data->boot_data_dma_addr));
118+ WARN_ON(pvr_gem_get_dma_addr(fw_dev->mem.code_obj->gem,
119+ exception_code_entry->alloc_offset,
120+ &mips_data->exception_code_dma_addr));
121+
122+ stack_entry = pvr_fw_find_layout_entry(pvr_dev, MIPS_STACK);
123+ if (!stack_entry)
124+ return -EINVAL;
125+
126+ boot_data = (struct rogue_mipsfw_boot_data *)(fw_data_ptr + boot_data_entry->alloc_offset +
127+ ROGUE_MIPSFW_BOOTLDR_CONF_OFFSET);
128+
129+ WARN_ON(pvr_fw_object_get_dma_addr(fw_dev->mem.data_obj, stack_entry->alloc_offset,
130+ &dma_addr));
131+ boot_data->stack_phys_addr = dma_addr;
132+
133+ boot_data->reg_base = pvr_dev->regs_resource->start;
134+
135+ for (page_nr = 0; page_nr < ARRAY_SIZE(boot_data->pt_phys_addr); page_nr++) {
136+ /* Firmware expects 4k pages, but host page size might be different. */
137+ u32 src_page_nr = (page_nr * ROGUE_MIPSFW_PAGE_SIZE_4K) >> PAGE_SHIFT;
138+ u32 page_offset = (page_nr * ROGUE_MIPSFW_PAGE_SIZE_4K) & ~PAGE_MASK;
139+
140+ boot_data->pt_phys_addr[page_nr] = mips_data->pt_dma_addr[src_page_nr] +
141+ page_offset;
142+ }
143+
144+ boot_data->pt_log2_page_size = ROGUE_MIPSFW_LOG2_PAGE_SIZE_4K;
145+ boot_data->pt_num_pages = ROGUE_MIPSFW_MAX_NUM_PAGETABLE_PAGES;
146+ boot_data->reserved1 = 0;
147+ boot_data->reserved2 = 0;
148+
149+ return 0;
150+}
151+
152+static int
153+pvr_mips_wrapper_init(struct pvr_device *pvr_dev)
154+{
155+ struct pvr_fw_mips_data *mips_data = pvr_dev->fw_dev.processor_data.mips_data;
156+ const u64 remap_settings = ROGUE_MIPSFW_BOOT_REMAP_LOG2_SEGMENT_SIZE;
157+ u32 phys_bus_width;
158+
159+ int err = PVR_FEATURE_VALUE(pvr_dev, phys_bus_width, &phys_bus_width);
160+
161+ if (WARN_ON(err))
162+ return err;
163+
164+ /* Currently MIPS FW only supported with physical bus width > 32 bits. */
165+ if (WARN_ON(phys_bus_width <= 32))
166+ return -EINVAL;
167+
168+ pvr_cr_write32(pvr_dev, ROGUE_CR_MIPS_WRAPPER_CONFIG,
169+ (ROGUE_MIPSFW_REGISTERS_VIRTUAL_BASE >>
170+ ROGUE_MIPSFW_WRAPPER_CONFIG_REGBANK_ADDR_ALIGN) |
171+ ROGUE_CR_MIPS_WRAPPER_CONFIG_BOOT_ISA_MODE_MICROMIPS);
172+
173+ /* Configure remap for boot code, boot data and exceptions code areas. */
174+ pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1,
175+ ROGUE_MIPSFW_BOOT_REMAP_PHYS_ADDR_IN |
176+ ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_MODE_ENABLE_EN);
177+ pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2,
178+ (mips_data->boot_code_dma_addr &
179+ ~ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_ADDR_OUT_CLRMSK) | remap_settings);
180+
181+ if (PVR_HAS_QUIRK(pvr_dev, 63553)) {
182+ /*
183+ * WA always required on 36 bit cores, to avoid continuous unmapped memory accesses
184+ * to address 0x0.
185+ */
186+ WARN_ON(phys_bus_width != 36);
187+
188+ pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1,
189+ ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_MODE_ENABLE_EN);
190+ pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2,
191+ (mips_data->boot_code_dma_addr &
192+ ~ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_ADDR_OUT_CLRMSK) |
193+ remap_settings);
194+ }
195+
196+ pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1,
197+ ROGUE_MIPSFW_DATA_REMAP_PHYS_ADDR_IN |
198+ ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_MODE_ENABLE_EN);
199+ pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2,
200+ (mips_data->boot_data_dma_addr &
201+ ~ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_ADDR_OUT_CLRMSK) | remap_settings);
202+
203+ pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1,
204+ ROGUE_MIPSFW_CODE_REMAP_PHYS_ADDR_IN |
205+ ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_MODE_ENABLE_EN);
206+ pvr_cr_write64(pvr_dev, ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2,
207+ (mips_data->exception_code_dma_addr &
208+ ~ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_ADDR_OUT_CLRMSK) | remap_settings);
209+
210+ /* Garten IDLE bit controlled by MIPS. */
211+ pvr_cr_write64(pvr_dev, ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG,
212+ ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_IDLE_CTRL_META);
213+
214+ /* Turn on the EJTAG probe. */
215+ pvr_cr_write32(pvr_dev, ROGUE_CR_MIPS_DEBUG_CONFIG, 0);
216+
217+ return 0;
218+}
219+
220+static u32
221+pvr_mips_get_fw_addr_with_offset(struct pvr_fw_object *fw_obj, u32 offset)
222+{
223+ struct pvr_device *pvr_dev = to_pvr_device(gem_from_pvr_gem(fw_obj->gem)->dev);
224+
225+ /* MIPS cacheability is determined by page table. */
226+ return ((fw_obj->fw_addr_offset + offset) & pvr_dev->fw_dev.fw_heap_info.offset_mask) |
227+ ROGUE_FW_HEAP_MIPS_BASE;
228+}
229+
230+static bool
231+pvr_mips_has_fixed_data_addr(void)
232+{
233+ return true;
234+}
235+
236+const struct pvr_fw_defs pvr_fw_defs_mips = {
237+ .init = pvr_mips_init,
238+ .fini = pvr_mips_fini,
239+ .fw_process = pvr_mips_fw_process,
240+ .vm_map = pvr_vm_mips_map,
241+ .vm_unmap = pvr_vm_mips_unmap,
242+ .get_fw_addr_with_offset = pvr_mips_get_fw_addr_with_offset,
243+ .wrapper_init = pvr_mips_wrapper_init,
244+ .has_fixed_data_addr = pvr_mips_has_fixed_data_addr,
245+ .irq = {
246+ .enable_reg = ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE,
247+ .status_reg = ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS,
248+ .clear_reg = ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR,
249+ .event_mask = ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS_EVENT_EN,
250+ .clear_mask = ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR_EVENT_EN,
251+ },
252+};
drivers/gpu/drm/imagination/pvr_fw_mips.h+48−0
@@ -0,0 +1,48 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_FW_MIPS_H
5+#define PVR_FW_MIPS_H
6+
7+#include "pvr_rogue_mips.h"
8+
9+#include <asm/page.h>
10+#include <linux/types.h>
11+
12+/* Forward declaration from pvr_gem.h. */
13+struct pvr_gem_object;
14+
15+#define PVR_MIPS_PT_PAGE_COUNT ((ROGUE_MIPSFW_MAX_NUM_PAGETABLE_PAGES * ROGUE_MIPSFW_PAGE_SIZE_4K) \
16+ >> PAGE_SHIFT)
17+/**
18+ * struct pvr_fw_mips_data - MIPS-specific data
19+ */
20+struct pvr_fw_mips_data {
21+ /**
22+ * @pt_pages: Pages containing MIPS pagetable.
23+ */
24+ struct page *pt_pages[PVR_MIPS_PT_PAGE_COUNT];
25+
26+ /** @pt: Pointer to CPU mapping of MIPS pagetable. */
27+ u32 *pt;
28+
29+ /** @pt_dma_addr: DMA mappings of MIPS pagetable. */
30+ dma_addr_t pt_dma_addr[PVR_MIPS_PT_PAGE_COUNT];
31+
32+ /** @boot_code_dma_addr: DMA address of MIPS boot code. */
33+ dma_addr_t boot_code_dma_addr;
34+
35+ /** @boot_data_dma_addr: DMA address of MIPS boot data. */
36+ dma_addr_t boot_data_dma_addr;
37+
38+ /** @exception_code_dma_addr: DMA address of MIPS exception code. */
39+ dma_addr_t exception_code_dma_addr;
40+
41+ /** @cache_policy: Cache policy for this processor. */
42+ u32 cache_policy;
43+
44+ /** @pfn_mask: PFN mask for MIPS pagetable. */
45+ u32 pfn_mask;
46+};
47+
48+#endif /* PVR_FW_MIPS_H */
drivers/gpu/drm/imagination/pvr_fw_startstop.c+306−0
@@ -0,0 +1,306 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_device.h"
5+#include "pvr_fw.h"
6+#include "pvr_fw_meta.h"
7+#include "pvr_fw_startstop.h"
8+#include "pvr_rogue_cr_defs.h"
9+#include "pvr_rogue_meta.h"
10+#include "pvr_vm.h"
11+
12+#include <linux/compiler.h>
13+#include <linux/delay.h>
14+#include <linux/ktime.h>
15+#include <linux/types.h>
16+
17+#define POLL_TIMEOUT_USEC 1000000
18+
19+static void
20+rogue_axi_ace_list_init(struct pvr_device *pvr_dev)
21+{
22+ /* Setup AXI-ACE config. Set everything to outer cache. */
23+ u64 reg_val =
24+ (3U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_AWDOMAIN_NON_SNOOPING_SHIFT) |
25+ (3U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_ARDOMAIN_NON_SNOOPING_SHIFT) |
26+ (2U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_ARDOMAIN_CACHE_MAINTENANCE_SHIFT) |
27+ (2U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_AWDOMAIN_COHERENT_SHIFT) |
28+ (2U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_ARDOMAIN_COHERENT_SHIFT) |
29+ (2U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_AWCACHE_COHERENT_SHIFT) |
30+ (2U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_ARCACHE_COHERENT_SHIFT) |
31+ (2U << ROGUE_CR_AXI_ACE_LITE_CONFIGURATION_ARCACHE_CACHE_MAINTENANCE_SHIFT);
32+
33+ pvr_cr_write64(pvr_dev, ROGUE_CR_AXI_ACE_LITE_CONFIGURATION, reg_val);
34+}
35+
36+static void
37+rogue_bif_init(struct pvr_device *pvr_dev)
38+{
39+ dma_addr_t pc_dma_addr;
40+ u64 pc_addr;
41+
42+ /* Acquire the address of the Kernel Page Catalogue. */
43+ pc_dma_addr = pvr_vm_get_page_table_root_addr(pvr_dev->kernel_vm_ctx);
44+
45+ /* Write the kernel catalogue base. */
46+ pc_addr = ((((u64)pc_dma_addr >> ROGUE_CR_BIF_CAT_BASE0_ADDR_ALIGNSHIFT)
47+ << ROGUE_CR_BIF_CAT_BASE0_ADDR_SHIFT) &
48+ ~ROGUE_CR_BIF_CAT_BASE0_ADDR_CLRMSK);
49+
50+ pvr_cr_write64(pvr_dev, BIF_CAT_BASEX(MMU_CONTEXT_MAPPING_FWPRIV),
51+ pc_addr);
52+}
53+
54+static int
55+rogue_slc_init(struct pvr_device *pvr_dev)
56+{
57+ u16 slc_cache_line_size_bits;
58+ u32 reg_val;
59+ int err;
60+
61+ /*
62+ * SLC Misc control.
63+ *
64+ * Note: This is a 64bit register and we set only the lower 32bits
65+ * leaving the top 32bits (ROGUE_CR_SLC_CTRL_MISC_SCRAMBLE_BITS)
66+ * unchanged from the HW default.
67+ */
68+ reg_val = (pvr_cr_read32(pvr_dev, ROGUE_CR_SLC_CTRL_MISC) &
69+ ROGUE_CR_SLC_CTRL_MISC_ENABLE_PSG_HAZARD_CHECK_EN) |
70+ ROGUE_CR_SLC_CTRL_MISC_ADDR_DECODE_MODE_PVR_HASH1;
71+
72+ err = PVR_FEATURE_VALUE(pvr_dev, slc_cache_line_size_bits, &slc_cache_line_size_bits);
73+ if (err)
74+ return err;
75+
76+ /* Bypass burst combiner if SLC line size is smaller than 1024 bits. */
77+ if (slc_cache_line_size_bits < 1024)
78+ reg_val |= ROGUE_CR_SLC_CTRL_MISC_BYPASS_BURST_COMBINER_EN;
79+
80+ if (PVR_HAS_QUIRK(pvr_dev, 71242) && !PVR_HAS_FEATURE(pvr_dev, gpu_multicore_support))
81+ reg_val |= ROGUE_CR_SLC_CTRL_MISC_LAZYWB_OVERRIDE_EN;
82+
83+ pvr_cr_write32(pvr_dev, ROGUE_CR_SLC_CTRL_MISC, reg_val);
84+
85+ return 0;
86+}
87+
88+/**
89+ * pvr_fw_start() - Start FW processor and boot firmware
90+ * @pvr_dev: Target PowerVR device.
91+ *
92+ * Returns:
93+ * * 0 on success, or
94+ * * Any error returned by rogue_slc_init().
95+ */
96+int
97+pvr_fw_start(struct pvr_device *pvr_dev)
98+{
99+ bool has_reset2 = PVR_HAS_FEATURE(pvr_dev, xe_tpu2);
100+ u64 soft_reset_mask;
101+ int err;
102+
103+ if (PVR_HAS_FEATURE(pvr_dev, pbe2_in_xe))
104+ soft_reset_mask = ROGUE_CR_SOFT_RESET__PBE2_XE__MASKFULL;
105+ else
106+ soft_reset_mask = ROGUE_CR_SOFT_RESET_MASKFULL;
107+
108+ if (PVR_HAS_FEATURE(pvr_dev, sys_bus_secure_reset)) {
109+ /*
110+ * Disable the default sys_bus_secure protection to perform
111+ * minimal setup.
112+ */
113+ pvr_cr_write32(pvr_dev, ROGUE_CR_SYS_BUS_SECURE, 0);
114+ (void)pvr_cr_read32(pvr_dev, ROGUE_CR_SYS_BUS_SECURE); /* Fence write */
115+ }
116+
117+ /* Set Rogue in soft-reset. */
118+ pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET, soft_reset_mask);
119+ if (has_reset2)
120+ pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET2, ROGUE_CR_SOFT_RESET2_MASKFULL);
121+
122+ /* Read soft-reset to fence previous write in order to clear the SOCIF pipeline. */
123+ (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET);
124+ if (has_reset2)
125+ (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET2);
126+
127+ /* Take Rascal and Dust out of reset. */
128+ pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET,
129+ soft_reset_mask ^ ROGUE_CR_SOFT_RESET_RASCALDUSTS_EN);
130+ if (has_reset2)
131+ pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET2, 0);
132+
133+ (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET);
134+ if (has_reset2)
135+ (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET2);
136+
137+ /* Take everything out of reset but the FW processor. */
138+ pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET, ROGUE_CR_SOFT_RESET_GARTEN_EN);
139+ if (has_reset2)
140+ pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET2, 0);
141+
142+ (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET);
143+ if (has_reset2)
144+ (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET2);
145+
146+ err = rogue_slc_init(pvr_dev);
147+ if (err)
148+ goto err_reset;
149+
150+ /* Initialise Firmware wrapper. */
151+ pvr_dev->fw_dev.defs->wrapper_init(pvr_dev);
152+
153+ /* We must init the AXI-ACE interface before first BIF transaction. */
154+ rogue_axi_ace_list_init(pvr_dev);
155+
156+ if (pvr_dev->fw_dev.processor_type != PVR_FW_PROCESSOR_TYPE_MIPS) {
157+ /* Initialise BIF. */
158+ rogue_bif_init(pvr_dev);
159+ }
160+
161+ /* Need to wait for at least 16 cycles before taking the FW processor out of reset ... */
162+ udelay(3);
163+
164+ pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET, 0x0);
165+ (void)pvr_cr_read64(pvr_dev, ROGUE_CR_SOFT_RESET);
166+
167+ /* ... and afterwards. */
168+ udelay(3);
169+
170+ return 0;
171+
172+err_reset:
173+ /* Put everything back into soft-reset. */
174+ pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET, soft_reset_mask);
175+
176+ return err;
177+}
178+
179+/**
180+ * pvr_fw_stop() - Stop FW processor
181+ * @pvr_dev: Target PowerVR device.
182+ *
183+ * Returns:
184+ * * 0 on success, or
185+ * * Any error returned by pvr_cr_poll_reg32().
186+ */
187+int
188+pvr_fw_stop(struct pvr_device *pvr_dev)
189+{
190+ const u32 sidekick_idle_mask = ROGUE_CR_SIDEKICK_IDLE_MASKFULL &
191+ ~(ROGUE_CR_SIDEKICK_IDLE_GARTEN_EN |
192+ ROGUE_CR_SIDEKICK_IDLE_SOCIF_EN |
193+ ROGUE_CR_SIDEKICK_IDLE_HOSTIF_EN);
194+ bool skip_garten_idle = false;
195+ u32 reg_value;
196+ int err;
197+
198+ /*
199+ * Wait for Sidekick/Jones to signal IDLE except for the Garten Wrapper.
200+ * For cores with the LAYOUT_MARS feature, SIDEKICK would have been
201+ * powered down by the FW.
202+ */
203+ err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_SIDEKICK_IDLE, sidekick_idle_mask,
204+ sidekick_idle_mask, POLL_TIMEOUT_USEC);
205+ if (err)
206+ return err;
207+
208+ /* Unset MTS DM association with threads. */
209+ pvr_cr_write32(pvr_dev, ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC,
210+ ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC_MASKFULL &
211+ ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC_DM_ASSOC_CLRMSK);
212+ pvr_cr_write32(pvr_dev, ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC,
213+ ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC_MASKFULL &
214+ ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC_DM_ASSOC_CLRMSK);
215+ pvr_cr_write32(pvr_dev, ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC,
216+ ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC_MASKFULL &
217+ ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC_DM_ASSOC_CLRMSK);
218+ pvr_cr_write32(pvr_dev, ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC,
219+ ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC_MASKFULL &
220+ ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC_DM_ASSOC_CLRMSK);
221+
222+ /* Extra Idle checks. */
223+ err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_BIF_STATUS_MMU, 0,
224+ ROGUE_CR_BIF_STATUS_MMU_MASKFULL,
225+ POLL_TIMEOUT_USEC);
226+ if (err)
227+ return err;
228+
229+ err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_BIFPM_STATUS_MMU, 0,
230+ ROGUE_CR_BIFPM_STATUS_MMU_MASKFULL,
231+ POLL_TIMEOUT_USEC);
232+ if (err)
233+ return err;
234+
235+ if (!PVR_HAS_FEATURE(pvr_dev, xt_top_infrastructure)) {
236+ err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_BIF_READS_EXT_STATUS, 0,
237+ ROGUE_CR_BIF_READS_EXT_STATUS_MASKFULL,
238+ POLL_TIMEOUT_USEC);
239+ if (err)
240+ return err;
241+ }
242+
243+ err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_BIFPM_READS_EXT_STATUS, 0,
244+ ROGUE_CR_BIFPM_READS_EXT_STATUS_MASKFULL,
245+ POLL_TIMEOUT_USEC);
246+ if (err)
247+ return err;
248+
249+ err = pvr_cr_poll_reg64(pvr_dev, ROGUE_CR_SLC_STATUS1, 0,
250+ ROGUE_CR_SLC_STATUS1_MASKFULL,
251+ POLL_TIMEOUT_USEC);
252+ if (err)
253+ return err;
254+
255+ /*
256+ * Wait for SLC to signal IDLE.
257+ * For cores with the LAYOUT_MARS feature, SLC would have been powered
258+ * down by the FW.
259+ */
260+ err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_SLC_IDLE,
261+ ROGUE_CR_SLC_IDLE_MASKFULL,
262+ ROGUE_CR_SLC_IDLE_MASKFULL, POLL_TIMEOUT_USEC);
263+ if (err)
264+ return err;
265+
266+ /*
267+ * Wait for Sidekick/Jones to signal IDLE except for the Garten Wrapper.
268+ * For cores with the LAYOUT_MARS feature, SIDEKICK would have been powered
269+ * down by the FW.
270+ */
271+ err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_SIDEKICK_IDLE, sidekick_idle_mask,
272+ sidekick_idle_mask, POLL_TIMEOUT_USEC);
273+ if (err)
274+ return err;
275+
276+ if (pvr_dev->fw_dev.processor_type == PVR_FW_PROCESSOR_TYPE_META) {
277+ err = pvr_meta_cr_read32(pvr_dev, META_CR_TxVECINT_BHALT, &reg_value);
278+ if (err)
279+ return err;
280+
281+ /*
282+ * Wait for Sidekick/Jones to signal IDLE including the Garten
283+ * Wrapper if there is no debugger attached (TxVECINT_BHALT =
284+ * 0x0).
285+ */
286+ if (reg_value)
287+ skip_garten_idle = true;
288+ }
289+
290+ if (!skip_garten_idle) {
291+ err = pvr_cr_poll_reg32(pvr_dev, ROGUE_CR_SIDEKICK_IDLE,
292+ ROGUE_CR_SIDEKICK_IDLE_GARTEN_EN,
293+ ROGUE_CR_SIDEKICK_IDLE_GARTEN_EN,
294+ POLL_TIMEOUT_USEC);
295+ if (err)
296+ return err;
297+ }
298+
299+ if (PVR_HAS_FEATURE(pvr_dev, pbe2_in_xe))
300+ pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET,
301+ ROGUE_CR_SOFT_RESET__PBE2_XE__MASKFULL);
302+ else
303+ pvr_cr_write64(pvr_dev, ROGUE_CR_SOFT_RESET, ROGUE_CR_SOFT_RESET_MASKFULL);
304+
305+ return 0;
306+}
drivers/gpu/drm/imagination/pvr_fw_startstop.h+13−0
@@ -0,0 +1,13 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_FW_STARTSTOP_H
5+#define PVR_FW_STARTSTOP_H
6+
7+/* Forward declaration from pvr_device.h. */
8+struct pvr_device;
9+
10+int pvr_fw_start(struct pvr_device *pvr_dev);
11+int pvr_fw_stop(struct pvr_device *pvr_dev);
12+
13+#endif /* PVR_FW_STARTSTOP_H */
drivers/gpu/drm/imagination/pvr_fw_trace.c+515−0
@@ -0,0 +1,515 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_device.h"
5+#include "pvr_gem.h"
6+#include "pvr_rogue_fwif.h"
7+#include "pvr_rogue_fwif_sf.h"
8+#include "pvr_fw_trace.h"
9+
10+#include <drm/drm_drv.h>
11+#include <drm/drm_file.h>
12+
13+#include <linux/build_bug.h>
14+#include <linux/dcache.h>
15+#include <linux/sysfs.h>
16+#include <linux/types.h>
17+
18+static void
19+tracebuf_ctrl_init(void *cpu_ptr, void *priv)
20+{
21+ struct rogue_fwif_tracebuf *tracebuf_ctrl = cpu_ptr;
22+ struct pvr_fw_trace *fw_trace = priv;
23+ u32 thread_nr;
24+
25+ tracebuf_ctrl->tracebuf_size_in_dwords = ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS;
26+ tracebuf_ctrl->tracebuf_flags = 0;
27+
28+ if (fw_trace->group_mask)
29+ tracebuf_ctrl->log_type = fw_trace->group_mask | ROGUE_FWIF_LOG_TYPE_TRACE;
30+ else
31+ tracebuf_ctrl->log_type = ROGUE_FWIF_LOG_TYPE_NONE;
32+
33+ for (thread_nr = 0; thread_nr < ARRAY_SIZE(fw_trace->buffers); thread_nr++) {
34+ struct rogue_fwif_tracebuf_space *tracebuf_space =
35+ &tracebuf_ctrl->tracebuf[thread_nr];
36+ struct pvr_fw_trace_buffer *trace_buffer = &fw_trace->buffers[thread_nr];
37+
38+ pvr_fw_object_get_fw_addr(trace_buffer->buf_obj,
39+ &tracebuf_space->trace_buffer_fw_addr);
40+
41+ tracebuf_space->trace_buffer = trace_buffer->buf;
42+ tracebuf_space->trace_pointer = 0;
43+ }
44+}
45+
46+int pvr_fw_trace_init(struct pvr_device *pvr_dev)
47+{
48+ struct pvr_fw_trace *fw_trace = &pvr_dev->fw_dev.fw_trace;
49+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
50+ u32 thread_nr;
51+ int err;
52+
53+ for (thread_nr = 0; thread_nr < ARRAY_SIZE(fw_trace->buffers); thread_nr++) {
54+ struct pvr_fw_trace_buffer *trace_buffer = &fw_trace->buffers[thread_nr];
55+
56+ trace_buffer->buf =
57+ pvr_fw_object_create_and_map(pvr_dev,
58+ ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS *
59+ sizeof(*trace_buffer->buf),
60+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED |
61+ PVR_BO_FW_NO_CLEAR_ON_RESET,
62+ NULL, NULL, &trace_buffer->buf_obj);
63+ if (IS_ERR(trace_buffer->buf)) {
64+ drm_err(drm_dev, "Unable to allocate trace buffer\n");
65+ err = PTR_ERR(trace_buffer->buf);
66+ trace_buffer->buf = NULL;
67+ goto err_free_buf;
68+ }
69+ }
70+
71+ /* TODO: Provide control of group mask. */
72+ fw_trace->group_mask = 0;
73+
74+ fw_trace->tracebuf_ctrl =
75+ pvr_fw_object_create_and_map(pvr_dev,
76+ sizeof(*fw_trace->tracebuf_ctrl),
77+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED |
78+ PVR_BO_FW_NO_CLEAR_ON_RESET,
79+ tracebuf_ctrl_init, fw_trace,
80+ &fw_trace->tracebuf_ctrl_obj);
81+ if (IS_ERR(fw_trace->tracebuf_ctrl)) {
82+ drm_err(drm_dev, "Unable to allocate trace buffer control structure\n");
83+ err = PTR_ERR(fw_trace->tracebuf_ctrl);
84+ goto err_free_buf;
85+ }
86+
87+ BUILD_BUG_ON(ARRAY_SIZE(fw_trace->tracebuf_ctrl->tracebuf) !=
88+ ARRAY_SIZE(fw_trace->buffers));
89+
90+ for (thread_nr = 0; thread_nr < ARRAY_SIZE(fw_trace->buffers); thread_nr++) {
91+ struct rogue_fwif_tracebuf_space *tracebuf_space =
92+ &fw_trace->tracebuf_ctrl->tracebuf[thread_nr];
93+ struct pvr_fw_trace_buffer *trace_buffer = &fw_trace->buffers[thread_nr];
94+
95+ trace_buffer->tracebuf_space = tracebuf_space;
96+ }
97+
98+ return 0;
99+
100+err_free_buf:
101+ for (thread_nr = 0; thread_nr < ARRAY_SIZE(fw_trace->buffers); thread_nr++) {
102+ struct pvr_fw_trace_buffer *trace_buffer = &fw_trace->buffers[thread_nr];
103+
104+ if (trace_buffer->buf)
105+ pvr_fw_object_unmap_and_destroy(trace_buffer->buf_obj);
106+ }
107+
108+ return err;
109+}
110+
111+void pvr_fw_trace_fini(struct pvr_device *pvr_dev)
112+{
113+ struct pvr_fw_trace *fw_trace = &pvr_dev->fw_dev.fw_trace;
114+ u32 thread_nr;
115+
116+ for (thread_nr = 0; thread_nr < ARRAY_SIZE(fw_trace->buffers); thread_nr++) {
117+ struct pvr_fw_trace_buffer *trace_buffer = &fw_trace->buffers[thread_nr];
118+
119+ pvr_fw_object_unmap_and_destroy(trace_buffer->buf_obj);
120+ }
121+ pvr_fw_object_unmap_and_destroy(fw_trace->tracebuf_ctrl_obj);
122+}
123+
124+/**
125+ * update_logtype() - Send KCCB command to trigger FW to update logtype
126+ * @pvr_dev: Target PowerVR device
127+ * @group_mask: New log group mask.
128+ *
129+ * Returns:
130+ * * 0 on success,
131+ * * Any error returned by pvr_kccb_send_cmd(), or
132+ * * -%EIO if the device is lost.
133+ */
134+static int
135+update_logtype(struct pvr_device *pvr_dev, u32 group_mask)
136+{
137+ struct pvr_fw_trace *fw_trace = &pvr_dev->fw_dev.fw_trace;
138+ struct rogue_fwif_kccb_cmd cmd;
139+ int idx;
140+ int err;
141+
142+ if (group_mask)
143+ fw_trace->tracebuf_ctrl->log_type = ROGUE_FWIF_LOG_TYPE_TRACE | group_mask;
144+ else
145+ fw_trace->tracebuf_ctrl->log_type = ROGUE_FWIF_LOG_TYPE_NONE;
146+
147+ fw_trace->group_mask = group_mask;
148+
149+ down_read(&pvr_dev->reset_sem);
150+ if (!drm_dev_enter(from_pvr_device(pvr_dev), &idx)) {
151+ err = -EIO;
152+ goto err_up_read;
153+ }
154+
155+ cmd.cmd_type = ROGUE_FWIF_KCCB_CMD_LOGTYPE_UPDATE;
156+ cmd.kccb_flags = 0;
157+
158+ err = pvr_kccb_send_cmd(pvr_dev, &cmd, NULL);
159+
160+ drm_dev_exit(idx);
161+
162+err_up_read:
163+ up_read(&pvr_dev->reset_sem);
164+
165+ return err;
166+}
167+
168+#if defined(CONFIG_DEBUG_FS)
169+
170+static int fw_trace_group_mask_show(struct seq_file *m, void *data)
171+{
172+ struct pvr_device *pvr_dev = m->private;
173+
174+ seq_printf(m, "%08x\n", pvr_dev->fw_dev.fw_trace.group_mask);
175+
176+ return 0;
177+}
178+
179+static int fw_trace_group_mask_open(struct inode *inode, struct file *file)
180+{
181+ return single_open(file, fw_trace_group_mask_show, inode->i_private);
182+}
183+
184+static ssize_t fw_trace_group_mask_write(struct file *file, const char __user *ubuf, size_t len,
185+ loff_t *offp)
186+{
187+ struct seq_file *m = file->private_data;
188+ struct pvr_device *pvr_dev = m->private;
189+ u32 new_group_mask;
190+ int err;
191+
192+ err = kstrtouint_from_user(ubuf, len, 0, &new_group_mask);
193+ if (err)
194+ return err;
195+
196+ err = update_logtype(pvr_dev, new_group_mask);
197+ if (err)
198+ return err;
199+
200+ pvr_dev->fw_dev.fw_trace.group_mask = new_group_mask;
201+
202+ return (ssize_t)len;
203+}
204+
205+static const struct file_operations pvr_fw_trace_group_mask_fops = {
206+ .owner = THIS_MODULE,
207+ .open = fw_trace_group_mask_open,
208+ .read = seq_read,
209+ .write = fw_trace_group_mask_write,
210+ .llseek = default_llseek,
211+ .release = single_release,
212+};
213+
214+struct pvr_fw_trace_seq_data {
215+ /** @buffer: Pointer to copy of trace data. */
216+ u32 *buffer;
217+
218+ /** @start_offset: Starting offset in trace data, as reported by FW. */
219+ u32 start_offset;
220+
221+ /** @idx: Current index into trace data. */
222+ u32 idx;
223+
224+ /** @assert_buf: Trace assert buffer, as reported by FW. */
225+ struct rogue_fwif_file_info_buf assert_buf;
226+};
227+
228+static u32 find_sfid(u32 id)
229+{
230+ u32 i;
231+
232+ for (i = 0; i < ARRAY_SIZE(stid_fmts); i++) {
233+ if (stid_fmts[i].id == id)
234+ return i;
235+ }
236+
237+ return ROGUE_FW_SF_LAST;
238+}
239+
240+static u32 read_fw_trace(struct pvr_fw_trace_seq_data *trace_seq_data, u32 offset)
241+{
242+ u32 idx;
243+
244+ idx = trace_seq_data->idx + offset;
245+ if (idx >= ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS)
246+ return 0;
247+
248+ idx = (idx + trace_seq_data->start_offset) % ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS;
249+ return trace_seq_data->buffer[idx];
250+}
251+
252+/**
253+ * fw_trace_get_next() - Advance trace index to next entry
254+ * @trace_seq_data: Trace sequence data.
255+ *
256+ * Returns:
257+ * * %true if trace index is now pointing to a valid entry, or
258+ * * %false if trace index is pointing to an invalid entry, or has hit the end
259+ * of the trace.
260+ */
261+static bool fw_trace_get_next(struct pvr_fw_trace_seq_data *trace_seq_data)
262+{
263+ u32 id, sf_id;
264+
265+ while (trace_seq_data->idx < ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS) {
266+ id = read_fw_trace(trace_seq_data, 0);
267+ trace_seq_data->idx++;
268+ if (!ROGUE_FW_LOG_VALIDID(id))
269+ continue;
270+ if (id == ROGUE_FW_SF_MAIN_ASSERT_FAILED) {
271+ /* Assertion failure marks the end of the trace. */
272+ return false;
273+ }
274+
275+ sf_id = find_sfid(id);
276+ if (sf_id == ROGUE_FW_SF_FIRST)
277+ continue;
278+ if (sf_id == ROGUE_FW_SF_LAST) {
279+ /*
280+ * Could not match with an ID in the SF table, trace is
281+ * most likely corrupt from this point.
282+ */
283+ return false;
284+ }
285+
286+ /* Skip over the timestamp, and any parameters. */
287+ trace_seq_data->idx += 2 + ROGUE_FW_SF_PARAMNUM(id);
288+
289+ /* Ensure index is now pointing to a valid trace entry. */
290+ id = read_fw_trace(trace_seq_data, 0);
291+ if (!ROGUE_FW_LOG_VALIDID(id))
292+ continue;
293+
294+ return true;
295+ };
296+
297+ /* Hit end of trace data. */
298+ return false;
299+}
300+
301+/**
302+ * fw_trace_get_first() - Find first valid entry in trace
303+ * @trace_seq_data: Trace sequence data.
304+ *
305+ * Skips over invalid (usually zero) and ROGUE_FW_SF_FIRST entries.
306+ *
307+ * If the trace has no valid entries, this function will exit with the trace
308+ * index pointing to the end of the trace. trace_seq_show() will return an error
309+ * in this state.
310+ */
311+static void fw_trace_get_first(struct pvr_fw_trace_seq_data *trace_seq_data)
312+{
313+ trace_seq_data->idx = 0;
314+
315+ while (trace_seq_data->idx < ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS) {
316+ u32 id = read_fw_trace(trace_seq_data, 0);
317+
318+ if (ROGUE_FW_LOG_VALIDID(id)) {
319+ u32 sf_id = find_sfid(id);
320+
321+ if (sf_id != ROGUE_FW_SF_FIRST)
322+ break;
323+ }
324+ trace_seq_data->idx++;
325+ }
326+}
327+
328+static void *fw_trace_seq_start(struct seq_file *s, loff_t *pos)
329+{
330+ struct pvr_fw_trace_seq_data *trace_seq_data = s->private;
331+ u32 i;
332+
333+ /* Reset trace index, then advance to *pos. */
334+ fw_trace_get_first(trace_seq_data);
335+
336+ for (i = 0; i < *pos; i++) {
337+ if (!fw_trace_get_next(trace_seq_data))
338+ return NULL;
339+ }
340+
341+ return (trace_seq_data->idx < ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS) ? pos : NULL;
342+}
343+
344+static void *fw_trace_seq_next(struct seq_file *s, void *v, loff_t *pos)
345+{
346+ struct pvr_fw_trace_seq_data *trace_seq_data = s->private;
347+
348+ (*pos)++;
349+ if (!fw_trace_get_next(trace_seq_data))
350+ return NULL;
351+
352+ return (trace_seq_data->idx < ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS) ? pos : NULL;
353+}
354+
355+static void fw_trace_seq_stop(struct seq_file *s, void *v)
356+{
357+}
358+
359+static int fw_trace_seq_show(struct seq_file *s, void *v)
360+{
361+ struct pvr_fw_trace_seq_data *trace_seq_data = s->private;
362+ u64 timestamp;
363+ u32 id;
364+ u32 sf_id;
365+
366+ if (trace_seq_data->idx >= ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS)
367+ return -EINVAL;
368+
369+ id = read_fw_trace(trace_seq_data, 0);
370+ /* Index is not pointing at a valid entry. */
371+ if (!ROGUE_FW_LOG_VALIDID(id))
372+ return -EINVAL;
373+
374+ sf_id = find_sfid(id);
375+ /* Index is not pointing at a valid entry. */
376+ if (sf_id == ROGUE_FW_SF_LAST)
377+ return -EINVAL;
378+
379+ timestamp = read_fw_trace(trace_seq_data, 1) |
380+ ((u64)read_fw_trace(trace_seq_data, 2) << 32);
381+ timestamp = (timestamp & ~ROGUE_FWT_TIMESTAMP_TIME_CLRMSK) >>
382+ ROGUE_FWT_TIMESTAMP_TIME_SHIFT;
383+
384+ seq_printf(s, "[%llu] : ", timestamp);
385+ if (id == ROGUE_FW_SF_MAIN_ASSERT_FAILED) {
386+ seq_printf(s, "ASSERTION %s failed at %s:%u",
387+ trace_seq_data->assert_buf.info,
388+ trace_seq_data->assert_buf.path,
389+ trace_seq_data->assert_buf.line_num);
390+ } else {
391+ seq_printf(s, stid_fmts[sf_id].name,
392+ read_fw_trace(trace_seq_data, 3),
393+ read_fw_trace(trace_seq_data, 4),
394+ read_fw_trace(trace_seq_data, 5),
395+ read_fw_trace(trace_seq_data, 6),
396+ read_fw_trace(trace_seq_data, 7),
397+ read_fw_trace(trace_seq_data, 8),
398+ read_fw_trace(trace_seq_data, 9),
399+ read_fw_trace(trace_seq_data, 10),
400+ read_fw_trace(trace_seq_data, 11),
401+ read_fw_trace(trace_seq_data, 12),
402+ read_fw_trace(trace_seq_data, 13),
403+ read_fw_trace(trace_seq_data, 14),
404+ read_fw_trace(trace_seq_data, 15),
405+ read_fw_trace(trace_seq_data, 16),
406+ read_fw_trace(trace_seq_data, 17),
407+ read_fw_trace(trace_seq_data, 18),
408+ read_fw_trace(trace_seq_data, 19),
409+ read_fw_trace(trace_seq_data, 20),
410+ read_fw_trace(trace_seq_data, 21),
411+ read_fw_trace(trace_seq_data, 22));
412+ }
413+ seq_puts(s, "\n");
414+ return 0;
415+}
416+
417+static const struct seq_operations pvr_fw_trace_seq_ops = {
418+ .start = fw_trace_seq_start,
419+ .next = fw_trace_seq_next,
420+ .stop = fw_trace_seq_stop,
421+ .show = fw_trace_seq_show
422+};
423+
424+static int fw_trace_open(struct inode *inode, struct file *file)
425+{
426+ struct pvr_fw_trace_buffer *trace_buffer = inode->i_private;
427+ struct rogue_fwif_tracebuf_space *tracebuf_space =
428+ trace_buffer->tracebuf_space;
429+ struct pvr_fw_trace_seq_data *trace_seq_data;
430+ int err;
431+
432+ trace_seq_data = kzalloc(sizeof(*trace_seq_data), GFP_KERNEL);
433+ if (!trace_seq_data)
434+ return -ENOMEM;
435+
436+ trace_seq_data->buffer = kcalloc(ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS,
437+ sizeof(*trace_seq_data->buffer), GFP_KERNEL);
438+ if (!trace_seq_data->buffer) {
439+ err = -ENOMEM;
440+ goto err_free_data;
441+ }
442+
443+ /*
444+ * Take a local copy of the trace buffer, as firmware may still be
445+ * writing to it. This will exist as long as this file is open.
446+ */
447+ memcpy(trace_seq_data->buffer, trace_buffer->buf,
448+ ROGUE_FW_TRACE_BUF_DEFAULT_SIZE_IN_DWORDS * sizeof(u32));
449+ trace_seq_data->start_offset = READ_ONCE(tracebuf_space->trace_pointer);
450+ trace_seq_data->assert_buf = tracebuf_space->assert_buf;
451+ fw_trace_get_first(trace_seq_data);
452+
453+ err = seq_open(file, &pvr_fw_trace_seq_ops);
454+ if (err)
455+ goto err_free_buffer;
456+
457+ ((struct seq_file *)file->private_data)->private = trace_seq_data;
458+
459+ return 0;
460+
461+err_free_buffer:
462+ kfree(trace_seq_data->buffer);
463+
464+err_free_data:
465+ kfree(trace_seq_data);
466+
467+ return err;
468+}
469+
470+static int fw_trace_release(struct inode *inode, struct file *file)
471+{
472+ struct pvr_fw_trace_seq_data *trace_seq_data =
473+ ((struct seq_file *)file->private_data)->private;
474+
475+ seq_release(inode, file);
476+ kfree(trace_seq_data->buffer);
477+ kfree(trace_seq_data);
478+
479+ return 0;
480+}
481+
482+static const struct file_operations pvr_fw_trace_fops = {
483+ .owner = THIS_MODULE,
484+ .open = fw_trace_open,
485+ .read = seq_read,
486+ .llseek = seq_lseek,
487+ .release = fw_trace_release,
488+};
489+
490+void
491+pvr_fw_trace_mask_update(struct pvr_device *pvr_dev, u32 old_mask, u32 new_mask)
492+{
493+ if (old_mask != new_mask)
494+ update_logtype(pvr_dev, new_mask);
495+}
496+
497+void
498+pvr_fw_trace_debugfs_init(struct pvr_device *pvr_dev, struct dentry *dir)
499+{
500+ struct pvr_fw_trace *fw_trace = &pvr_dev->fw_dev.fw_trace;
501+ u32 thread_nr;
502+
503+ static_assert(ARRAY_SIZE(fw_trace->buffers) <= 10,
504+ "The filename buffer is only large enough for a single-digit thread count");
505+
506+ for (thread_nr = 0; thread_nr < ARRAY_SIZE(fw_trace->buffers); ++thread_nr) {
507+ char filename[8];
508+
509+ snprintf(filename, ARRAY_SIZE(filename), "trace_%u", thread_nr);
510+ debugfs_create_file(filename, 0400, dir,
511+ &fw_trace->buffers[thread_nr],
512+ &pvr_fw_trace_fops);
513+ }
514+}
515+#endif
drivers/gpu/drm/imagination/pvr_fw_trace.h+78−0
@@ -0,0 +1,78 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_FW_TRACE_H
5+#define PVR_FW_TRACE_H
6+
7+#include <drm/drm_file.h>
8+#include <linux/types.h>
9+
10+#include "pvr_rogue_fwif.h"
11+
12+/* Forward declaration from pvr_device.h. */
13+struct pvr_device;
14+
15+/* Forward declaration from pvr_gem.h. */
16+struct pvr_fw_object;
17+
18+/* Forward declarations from pvr_rogue_fwif.h */
19+struct rogue_fwif_tracebuf;
20+struct rogue_fwif_tracebuf_space;
21+
22+/**
23+ * struct pvr_fw_trace_buffer - Structure representing a trace buffer
24+ */
25+struct pvr_fw_trace_buffer {
26+ /** @buf_obj: FW buffer object representing trace buffer. */
27+ struct pvr_fw_object *buf_obj;
28+
29+ /** @buf: Pointer to CPU mapping of trace buffer. */
30+ u32 *buf;
31+
32+ /**
33+ * @tracebuf_space: Pointer to FW tracebuf_space structure for this
34+ * trace buffer.
35+ */
36+ struct rogue_fwif_tracebuf_space *tracebuf_space;
37+};
38+
39+/**
40+ * struct pvr_fw_trace - Device firmware trace data
41+ */
42+struct pvr_fw_trace {
43+ /**
44+ * @tracebuf_ctrl_obj: Object representing FW trace buffer control
45+ * structure.
46+ */
47+ struct pvr_fw_object *tracebuf_ctrl_obj;
48+
49+ /**
50+ * @tracebuf_ctrl: Pointer to CPU mapping of FW trace buffer control
51+ * structure.
52+ */
53+ struct rogue_fwif_tracebuf *tracebuf_ctrl;
54+
55+ /**
56+ * @buffers: Array representing the actual trace buffers owned by this
57+ * device.
58+ */
59+ struct pvr_fw_trace_buffer buffers[ROGUE_FW_THREAD_MAX];
60+
61+ /** @group_mask: Mask of enabled trace groups. */
62+ u32 group_mask;
63+};
64+
65+int pvr_fw_trace_init(struct pvr_device *pvr_dev);
66+void pvr_fw_trace_fini(struct pvr_device *pvr_dev);
67+
68+#if defined(CONFIG_DEBUG_FS)
69+/* Forward declaration from <linux/dcache.h>. */
70+struct dentry;
71+
72+void pvr_fw_trace_mask_update(struct pvr_device *pvr_dev, u32 old_mask,
73+ u32 new_mask);
74+
75+void pvr_fw_trace_debugfs_init(struct pvr_device *pvr_dev, struct dentry *dir);
76+#endif /* defined(CONFIG_DEBUG_FS) */
77+
78+#endif /* PVR_FW_TRACE_H */
drivers/gpu/drm/imagination/pvr_gem.c+414−0
@@ -0,0 +1,414 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_device.h"
5+#include "pvr_gem.h"
6+#include "pvr_vm.h"
7+
8+#include <drm/drm_gem.h>
9+#include <drm/drm_prime.h>
10+
11+#include <linux/compiler.h>
12+#include <linux/compiler_attributes.h>
13+#include <linux/dma-buf.h>
14+#include <linux/dma-direction.h>
15+#include <linux/dma-mapping.h>
16+#include <linux/err.h>
17+#include <linux/gfp.h>
18+#include <linux/iosys-map.h>
19+#include <linux/log2.h>
20+#include <linux/mutex.h>
21+#include <linux/pagemap.h>
22+#include <linux/refcount.h>
23+#include <linux/scatterlist.h>
24+
25+static void pvr_gem_object_free(struct drm_gem_object *obj)
26+{
27+ drm_gem_shmem_object_free(obj);
28+}
29+
30+static int pvr_gem_mmap(struct drm_gem_object *gem_obj, struct vm_area_struct *vma)
31+{
32+ struct pvr_gem_object *pvr_obj = gem_to_pvr_gem(gem_obj);
33+ struct drm_gem_shmem_object *shmem_obj = shmem_gem_from_pvr_gem(pvr_obj);
34+
35+ if (!(pvr_obj->flags & DRM_PVR_BO_ALLOW_CPU_USERSPACE_ACCESS))
36+ return -EINVAL;
37+
38+ return drm_gem_shmem_mmap(shmem_obj, vma);
39+}
40+
41+static const struct drm_gem_object_funcs pvr_gem_object_funcs = {
42+ .free = pvr_gem_object_free,
43+ .print_info = drm_gem_shmem_object_print_info,
44+ .pin = drm_gem_shmem_object_pin,
45+ .unpin = drm_gem_shmem_object_unpin,
46+ .get_sg_table = drm_gem_shmem_object_get_sg_table,
47+ .vmap = drm_gem_shmem_object_vmap,
48+ .vunmap = drm_gem_shmem_object_vunmap,
49+ .mmap = pvr_gem_mmap,
50+ .vm_ops = &drm_gem_shmem_vm_ops,
51+};
52+
53+/**
54+ * pvr_gem_object_flags_validate() - Verify that a collection of PowerVR GEM
55+ * mapping and/or creation flags form a valid combination.
56+ * @flags: PowerVR GEM mapping/creation flags to validate.
57+ *
58+ * This function explicitly allows kernel-only flags. All ioctl entrypoints
59+ * should do their own validation as well as relying on this function.
60+ *
61+ * Return:
62+ * * %true if @flags contains valid mapping and/or creation flags, or
63+ * * %false otherwise.
64+ */
65+static bool
66+pvr_gem_object_flags_validate(u64 flags)
67+{
68+ static const u64 invalid_combinations[] = {
69+ /*
70+ * Memory flagged as PM/FW-protected cannot be mapped to
71+ * userspace. To make this explicit, we require that the two
72+ * flags allowing each of these respective features are never
73+ * specified together.
74+ */
75+ (DRM_PVR_BO_PM_FW_PROTECT |
76+ DRM_PVR_BO_ALLOW_CPU_USERSPACE_ACCESS),
77+ };
78+
79+ int i;
80+
81+ /*
82+ * Check for bits set in undefined regions. Reserved regions refer to
83+ * options that can only be set by the kernel. These are explicitly
84+ * allowed in most cases, and must be checked specifically in IOCTL
85+ * callback code.
86+ */
87+ if ((flags & PVR_BO_UNDEFINED_MASK) != 0)
88+ return false;
89+
90+ /*
91+ * Check for all combinations of flags marked as invalid in the array
92+ * above.
93+ */
94+ for (i = 0; i < ARRAY_SIZE(invalid_combinations); ++i) {
95+ u64 combo = invalid_combinations[i];
96+
97+ if ((flags & combo) == combo)
98+ return false;
99+ }
100+
101+ return true;
102+}
103+
104+/**
105+ * pvr_gem_object_into_handle() - Convert a reference to an object into a
106+ * userspace-accessible handle.
107+ * @pvr_obj: [IN] Target PowerVR-specific object.
108+ * @pvr_file: [IN] File to associate the handle with.
109+ * @handle: [OUT] Pointer to store the created handle in. Remains unmodified if
110+ * an error is encountered.
111+ *
112+ * If an error is encountered, ownership of @pvr_obj will not have been
113+ * transferred. If this function succeeds, however, further use of @pvr_obj is
114+ * considered undefined behaviour unless another reference to it is explicitly
115+ * held.
116+ *
117+ * Return:
118+ * * 0 on success, or
119+ * * Any error encountered while attempting to allocate a handle on @pvr_file.
120+ */
121+int
122+pvr_gem_object_into_handle(struct pvr_gem_object *pvr_obj,
123+ struct pvr_file *pvr_file, u32 *handle)
124+{
125+ struct drm_gem_object *gem_obj = gem_from_pvr_gem(pvr_obj);
126+ struct drm_file *file = from_pvr_file(pvr_file);
127+
128+ u32 new_handle;
129+ int err;
130+
131+ err = drm_gem_handle_create(file, gem_obj, &new_handle);
132+ if (err)
133+ return err;
134+
135+ /*
136+ * Release our reference to @pvr_obj, effectively transferring
137+ * ownership to the handle.
138+ */
139+ pvr_gem_object_put(pvr_obj);
140+
141+ /*
142+ * Do not store the new handle in @handle until no more errors can
143+ * occur.
144+ */
145+ *handle = new_handle;
146+
147+ return 0;
148+}
149+
150+/**
151+ * pvr_gem_object_from_handle() - Obtain a reference to an object from a
152+ * userspace handle.
153+ * @pvr_file: PowerVR-specific file to which @handle is associated.
154+ * @handle: Userspace handle referencing the target object.
155+ *
156+ * On return, @handle always maintains its reference to the requested object
157+ * (if it had one in the first place). If this function succeeds, the returned
158+ * object will hold an additional reference. When the caller is finished with
159+ * the returned object, they should call pvr_gem_object_put() on it to release
160+ * this reference.
161+ *
162+ * Return:
163+ * * A pointer to the requested PowerVR-specific object on success, or
164+ * * %NULL otherwise.
165+ */
166+struct pvr_gem_object *
167+pvr_gem_object_from_handle(struct pvr_file *pvr_file, u32 handle)
168+{
169+ struct drm_file *file = from_pvr_file(pvr_file);
170+ struct drm_gem_object *gem_obj;
171+
172+ gem_obj = drm_gem_object_lookup(file, handle);
173+ if (!gem_obj)
174+ return NULL;
175+
176+ return gem_to_pvr_gem(gem_obj);
177+}
178+
179+/**
180+ * pvr_gem_object_vmap() - Map a PowerVR GEM object into CPU virtual address
181+ * space.
182+ * @pvr_obj: Target PowerVR GEM object.
183+ *
184+ * Once the caller is finished with the CPU mapping, they must call
185+ * pvr_gem_object_vunmap() on @pvr_obj.
186+ *
187+ * If @pvr_obj is CPU-cached, dma_sync_sgtable_for_cpu() is called to make
188+ * sure the CPU mapping is consistent.
189+ *
190+ * Return:
191+ * * A pointer to the CPU mapping on success,
192+ * * -%ENOMEM if the mapping fails, or
193+ * * Any error encountered while attempting to acquire a reference to the
194+ * backing pages for @pvr_obj.
195+ */
196+void *
197+pvr_gem_object_vmap(struct pvr_gem_object *pvr_obj)
198+{
199+ struct drm_gem_shmem_object *shmem_obj = shmem_gem_from_pvr_gem(pvr_obj);
200+ struct drm_gem_object *obj = gem_from_pvr_gem(pvr_obj);
201+ struct iosys_map map;
202+ int err;
203+
204+ dma_resv_lock(obj->resv, NULL);
205+
206+ err = drm_gem_shmem_vmap(shmem_obj, &map);
207+ if (err)
208+ goto err_unlock;
209+
210+ if (pvr_obj->flags & PVR_BO_CPU_CACHED) {
211+ struct device *dev = shmem_obj->base.dev->dev;
212+
213+ /* If shmem_obj->sgt is NULL, that means the buffer hasn't been mapped
214+ * in GPU space yet.
215+ */
216+ if (shmem_obj->sgt)
217+ dma_sync_sgtable_for_cpu(dev, shmem_obj->sgt, DMA_BIDIRECTIONAL);
218+ }
219+
220+ dma_resv_unlock(obj->resv);
221+
222+ return map.vaddr;
223+
224+err_unlock:
225+ dma_resv_unlock(obj->resv);
226+
227+ return ERR_PTR(err);
228+}
229+
230+/**
231+ * pvr_gem_object_vunmap() - Unmap a PowerVR memory object from CPU virtual
232+ * address space.
233+ * @pvr_obj: Target PowerVR GEM object.
234+ *
235+ * If @pvr_obj is CPU-cached, dma_sync_sgtable_for_device() is called to make
236+ * sure the GPU mapping is consistent.
237+ */
238+void
239+pvr_gem_object_vunmap(struct pvr_gem_object *pvr_obj)
240+{
241+ struct drm_gem_shmem_object *shmem_obj = shmem_gem_from_pvr_gem(pvr_obj);
242+ struct iosys_map map = IOSYS_MAP_INIT_VADDR(shmem_obj->vaddr);
243+ struct drm_gem_object *obj = gem_from_pvr_gem(pvr_obj);
244+
245+ if (WARN_ON(!map.vaddr))
246+ return;
247+
248+ dma_resv_lock(obj->resv, NULL);
249+
250+ if (pvr_obj->flags & PVR_BO_CPU_CACHED) {
251+ struct device *dev = shmem_obj->base.dev->dev;
252+
253+ /* If shmem_obj->sgt is NULL, that means the buffer hasn't been mapped
254+ * in GPU space yet.
255+ */
256+ if (shmem_obj->sgt)
257+ dma_sync_sgtable_for_device(dev, shmem_obj->sgt, DMA_BIDIRECTIONAL);
258+ }
259+
260+ drm_gem_shmem_vunmap(shmem_obj, &map);
261+
262+ dma_resv_unlock(obj->resv);
263+}
264+
265+/**
266+ * pvr_gem_object_zero() - Zeroes the physical memory behind an object.
267+ * @pvr_obj: Target PowerVR GEM object.
268+ *
269+ * Return:
270+ * * 0 on success, or
271+ * * Any error encountered while attempting to map @pvr_obj to the CPU (see
272+ * pvr_gem_object_vmap()).
273+ */
274+static int
275+pvr_gem_object_zero(struct pvr_gem_object *pvr_obj)
276+{
277+ void *cpu_ptr;
278+
279+ cpu_ptr = pvr_gem_object_vmap(pvr_obj);
280+ if (IS_ERR(cpu_ptr))
281+ return PTR_ERR(cpu_ptr);
282+
283+ memset(cpu_ptr, 0, pvr_gem_object_size(pvr_obj));
284+
285+ /* Make sure the zero-ing is done before vumap-ing the object. */
286+ wmb();
287+
288+ pvr_gem_object_vunmap(pvr_obj);
289+
290+ return 0;
291+}
292+
293+/**
294+ * pvr_gem_create_object() - Allocate and pre-initializes a pvr_gem_object
295+ * @drm_dev: DRM device creating this object.
296+ * @size: Size of the object to allocate in bytes.
297+ *
298+ * Return:
299+ * * The new pre-initialized GEM object on success,
300+ * * -ENOMEM if the allocation failed.
301+ */
302+struct drm_gem_object *pvr_gem_create_object(struct drm_device *drm_dev, size_t size)
303+{
304+ struct drm_gem_object *gem_obj;
305+ struct pvr_gem_object *pvr_obj;
306+
307+ pvr_obj = kzalloc(sizeof(*pvr_obj), GFP_KERNEL);
308+ if (!pvr_obj)
309+ return ERR_PTR(-ENOMEM);
310+
311+ gem_obj = gem_from_pvr_gem(pvr_obj);
312+ gem_obj->funcs = &pvr_gem_object_funcs;
313+
314+ return gem_obj;
315+}
316+
317+/**
318+ * pvr_gem_object_create() - Creates a PowerVR-specific buffer object.
319+ * @pvr_dev: Target PowerVR device.
320+ * @size: Size of the object to allocate in bytes. Must be greater than zero.
321+ * Any value which is not an exact multiple of the system page size will be
322+ * rounded up to satisfy this condition.
323+ * @flags: Options which affect both this operation and future mapping
324+ * operations performed on the returned object. Must be a combination of
325+ * DRM_PVR_BO_* and/or PVR_BO_* flags.
326+ *
327+ * The created object may be larger than @size, but can never be smaller. To
328+ * get the exact size, call pvr_gem_object_size() on the returned pointer.
329+ *
330+ * Return:
331+ * * The newly-minted PowerVR-specific buffer object on success,
332+ * * -%EINVAL if @size is zero or @flags is not valid,
333+ * * -%ENOMEM if sufficient physical memory cannot be allocated, or
334+ * * Any other error returned by drm_gem_create_mmap_offset().
335+ */
336+struct pvr_gem_object *
337+pvr_gem_object_create(struct pvr_device *pvr_dev, size_t size, u64 flags)
338+{
339+ struct drm_gem_shmem_object *shmem_obj;
340+ struct pvr_gem_object *pvr_obj;
341+ struct sg_table *sgt;
342+ int err;
343+
344+ /* Verify @size and @flags before continuing. */
345+ if (size == 0 || !pvr_gem_object_flags_validate(flags))
346+ return ERR_PTR(-EINVAL);
347+
348+ shmem_obj = drm_gem_shmem_create(from_pvr_device(pvr_dev), size);
349+ if (IS_ERR(shmem_obj))
350+ return ERR_CAST(shmem_obj);
351+
352+ shmem_obj->pages_mark_dirty_on_put = true;
353+ shmem_obj->map_wc = !(flags & PVR_BO_CPU_CACHED);
354+ pvr_obj = shmem_gem_to_pvr_gem(shmem_obj);
355+ pvr_obj->flags = flags;
356+
357+ sgt = drm_gem_shmem_get_pages_sgt(shmem_obj);
358+ if (IS_ERR(sgt)) {
359+ err = PTR_ERR(sgt);
360+ goto err_shmem_object_free;
361+ }
362+
363+ dma_sync_sgtable_for_device(shmem_obj->base.dev->dev, sgt,
364+ DMA_BIDIRECTIONAL);
365+
366+ /*
367+ * Do this last because pvr_gem_object_zero() requires a fully
368+ * configured instance of struct pvr_gem_object.
369+ */
370+ pvr_gem_object_zero(pvr_obj);
371+
372+ return pvr_obj;
373+
374+err_shmem_object_free:
375+ drm_gem_shmem_free(shmem_obj);
376+
377+ return ERR_PTR(err);
378+}
379+
380+/**
381+ * pvr_gem_get_dma_addr() - Get DMA address for given offset in object
382+ * @pvr_obj: Pointer to object to lookup address in.
383+ * @offset: Offset within object to lookup address at.
384+ * @dma_addr_out: Pointer to location to store DMA address.
385+ *
386+ * Returns:
387+ * * 0 on success, or
388+ * * -%EINVAL if object is not currently backed, or if @offset is out of valid
389+ * range for this object.
390+ */
391+int
392+pvr_gem_get_dma_addr(struct pvr_gem_object *pvr_obj, u32 offset,
393+ dma_addr_t *dma_addr_out)
394+{
395+ struct drm_gem_shmem_object *shmem_obj = shmem_gem_from_pvr_gem(pvr_obj);
396+ u32 accumulated_offset = 0;
397+ struct scatterlist *sgl;
398+ unsigned int sgt_idx;
399+
400+ WARN_ON(!shmem_obj->sgt);
401+ for_each_sgtable_dma_sg(shmem_obj->sgt, sgl, sgt_idx) {
402+ u32 new_offset = accumulated_offset + sg_dma_len(sgl);
403+
404+ if (offset >= accumulated_offset && offset < new_offset) {
405+ *dma_addr_out = sg_dma_address(sgl) +
406+ (offset - accumulated_offset);
407+ return 0;
408+ }
409+
410+ accumulated_offset = new_offset;
411+ }
412+
413+ return -EINVAL;
414+}
drivers/gpu/drm/imagination/pvr_gem.h+170−0
@@ -0,0 +1,170 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_GEM_H
5+#define PVR_GEM_H
6+
7+#include "pvr_rogue_heap_config.h"
8+#include "pvr_rogue_meta.h"
9+
10+#include <uapi/drm/pvr_drm.h>
11+
12+#include <drm/drm_gem.h>
13+#include <drm/drm_gem_shmem_helper.h>
14+#include <drm/drm_mm.h>
15+
16+#include <linux/bitfield.h>
17+#include <linux/bits.h>
18+#include <linux/const.h>
19+#include <linux/compiler_attributes.h>
20+#include <linux/kernel.h>
21+#include <linux/mutex.h>
22+#include <linux/refcount.h>
23+#include <linux/scatterlist.h>
24+#include <linux/sizes.h>
25+#include <linux/types.h>
26+
27+/* Forward declaration from "pvr_device.h". */
28+struct pvr_device;
29+struct pvr_file;
30+
31+/**
32+ * DOC: Flags for DRM_IOCTL_PVR_CREATE_BO (kernel-only)
33+ *
34+ * Kernel-only values allowed in &pvr_gem_object->flags. The majority of options
35+ * for this field are specified in the UAPI header "pvr_drm.h" with a
36+ * DRM_PVR_BO_ prefix. To distinguish these internal options (which must exist
37+ * in ranges marked as "reserved" in the UAPI header), we drop the DRM prefix.
38+ * The public options should be used directly, DRM prefix and all.
39+ *
40+ * To avoid potentially confusing gaps in the UAPI options, these kernel-only
41+ * options are specified "in reverse", starting at bit 63.
42+ *
43+ * We use "reserved" to refer to bits defined here and not exposed in the UAPI.
44+ * Bits not defined anywhere are "undefined".
45+ *
46+ * CPU mapping options
47+ * :PVR_BO_CPU_CACHED: By default, all GEM objects are mapped write-combined on the CPU. Set this
48+ * flag to override this behaviour and map the object cached.
49+ *
50+ * Firmware options
51+ * :PVR_BO_FW_NO_CLEAR_ON_RESET: By default, all FW objects are cleared and reinitialised on hard
52+ * reset. Set this flag to override this behaviour and preserve buffer contents on reset.
53+ */
54+#define PVR_BO_CPU_CACHED BIT_ULL(63)
55+
56+#define PVR_BO_FW_NO_CLEAR_ON_RESET BIT_ULL(62)
57+
58+#define PVR_BO_KERNEL_FLAGS_MASK (PVR_BO_CPU_CACHED | PVR_BO_FW_NO_CLEAR_ON_RESET)
59+
60+/* Bits 61..3 are undefined. */
61+/* Bits 2..0 are defined in the UAPI. */
62+
63+/* Other utilities. */
64+#define PVR_BO_UNDEFINED_MASK ~(PVR_BO_KERNEL_FLAGS_MASK | DRM_PVR_BO_FLAGS_MASK)
65+
66+/*
67+ * All firmware-mapped memory uses (mostly) the same flags. Specifically,
68+ * firmware-mapped memory should be:
69+ * * Read/write on the device,
70+ * * Read/write on the CPU, and
71+ * * Write-combined on the CPU.
72+ *
73+ * The only variation is in caching on the device.
74+ */
75+#define PVR_BO_FW_FLAGS_DEVICE_CACHED (ULL(0))
76+#define PVR_BO_FW_FLAGS_DEVICE_UNCACHED DRM_PVR_BO_BYPASS_DEVICE_CACHE
77+
78+/**
79+ * struct pvr_gem_object - powervr-specific wrapper for &struct drm_gem_object
80+ */
81+struct pvr_gem_object {
82+ /**
83+ * @base: The underlying &struct drm_gem_shmem_object.
84+ *
85+ * Do not access this member directly, instead call
86+ * shem_gem_from_pvr_gem().
87+ */
88+ struct drm_gem_shmem_object base;
89+
90+ /**
91+ * @flags: Options set at creation-time. Some of these options apply to
92+ * the creation operation itself (which are stored here for reference)
93+ * with the remainder used for mapping options to both the device and
94+ * CPU. These are used every time this object is mapped, but may be
95+ * changed after creation.
96+ *
97+ * Must be a combination of DRM_PVR_BO_* and/or PVR_BO_* flags.
98+ *
99+ * .. note::
100+ *
101+ * This member is declared const to indicate that none of these
102+ * options may change or be changed throughout the object's
103+ * lifetime.
104+ */
105+ u64 flags;
106+
107+};
108+
109+static_assert(offsetof(struct pvr_gem_object, base) == 0,
110+ "offsetof(struct pvr_gem_object, base) not zero");
111+
112+#define shmem_gem_from_pvr_gem(pvr_obj) (&(pvr_obj)->base)
113+
114+#define shmem_gem_to_pvr_gem(shmem_obj) container_of_const(shmem_obj, struct pvr_gem_object, base)
115+
116+#define gem_from_pvr_gem(pvr_obj) (&(pvr_obj)->base.base)
117+
118+#define gem_to_pvr_gem(gem_obj) container_of_const(gem_obj, struct pvr_gem_object, base.base)
119+
120+/* Functions defined in pvr_gem.c */
121+
122+struct drm_gem_object *pvr_gem_create_object(struct drm_device *drm_dev, size_t size);
123+
124+struct pvr_gem_object *pvr_gem_object_create(struct pvr_device *pvr_dev,
125+ size_t size, u64 flags);
126+
127+int pvr_gem_object_into_handle(struct pvr_gem_object *pvr_obj,
128+ struct pvr_file *pvr_file, u32 *handle);
129+struct pvr_gem_object *pvr_gem_object_from_handle(struct pvr_file *pvr_file,
130+ u32 handle);
131+
132+static __always_inline struct sg_table *
133+pvr_gem_object_get_pages_sgt(struct pvr_gem_object *pvr_obj)
134+{
135+ return drm_gem_shmem_get_pages_sgt(shmem_gem_from_pvr_gem(pvr_obj));
136+}
137+
138+void *pvr_gem_object_vmap(struct pvr_gem_object *pvr_obj);
139+void pvr_gem_object_vunmap(struct pvr_gem_object *pvr_obj);
140+
141+int pvr_gem_get_dma_addr(struct pvr_gem_object *pvr_obj, u32 offset,
142+ dma_addr_t *dma_addr_out);
143+
144+/**
145+ * pvr_gem_object_get() - Acquire reference on pvr_gem_object
146+ * @pvr_obj: Pointer to object to acquire reference on.
147+ */
148+static __always_inline void
149+pvr_gem_object_get(struct pvr_gem_object *pvr_obj)
150+{
151+ drm_gem_object_get(gem_from_pvr_gem(pvr_obj));
152+}
153+
154+/**
155+ * pvr_gem_object_put() - Release reference on pvr_gem_object
156+ * @pvr_obj: Pointer to object to release reference on.
157+ */
158+static __always_inline void
159+pvr_gem_object_put(struct pvr_gem_object *pvr_obj)
160+{
161+ drm_gem_object_put(gem_from_pvr_gem(pvr_obj));
162+}
163+
164+static __always_inline size_t
165+pvr_gem_object_size(struct pvr_gem_object *pvr_obj)
166+{
167+ return gem_from_pvr_gem(pvr_obj)->size;
168+}
169+
170+#endif /* PVR_GEM_H */
drivers/gpu/drm/imagination/pvr_hwrt.c+549−0
@@ -0,0 +1,549 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_free_list.h"
5+#include "pvr_hwrt.h"
6+#include "pvr_gem.h"
7+#include "pvr_rogue_cr_defs_client.h"
8+#include "pvr_rogue_fwif.h"
9+
10+#include <drm/drm_gem.h>
11+#include <linux/bitops.h>
12+#include <linux/math.h>
13+#include <linux/slab.h>
14+#include <linux/xarray.h>
15+#include <uapi/drm/pvr_drm.h>
16+
17+static_assert(ROGUE_FWIF_NUM_RTDATAS == 2);
18+static_assert(ROGUE_FWIF_NUM_GEOMDATAS == 1);
19+static_assert(ROGUE_FWIF_NUM_RTDATA_FREELISTS == 2);
20+
21+/*
22+ * struct pvr_rt_mtile_info - Render target macrotile information
23+ */
24+struct pvr_rt_mtile_info {
25+ u32 mtile_x[3];
26+ u32 mtile_y[3];
27+ u32 tile_max_x;
28+ u32 tile_max_y;
29+ u32 tile_size_x;
30+ u32 tile_size_y;
31+ u32 num_tiles_x;
32+ u32 num_tiles_y;
33+};
34+
35+/* Size of Shadow Render Target Cache entry */
36+#define SRTC_ENTRY_SIZE sizeof(u32)
37+/* Size of Renders Accumulation Array entry */
38+#define RAA_ENTRY_SIZE sizeof(u32)
39+
40+static int
41+hwrt_init_kernel_structure(struct pvr_file *pvr_file,
42+ struct drm_pvr_ioctl_create_hwrt_dataset_args *args,
43+ struct pvr_hwrt_dataset *hwrt)
44+{
45+ struct pvr_device *pvr_dev = pvr_file->pvr_dev;
46+ int err;
47+ int i;
48+
49+ hwrt->pvr_dev = pvr_dev;
50+ hwrt->max_rts = args->layers;
51+
52+ /* Get pointers to the free lists */
53+ for (i = 0; i < ARRAY_SIZE(hwrt->free_lists); i++) {
54+ hwrt->free_lists[i] = pvr_free_list_lookup(pvr_file, args->free_list_handles[i]);
55+ if (!hwrt->free_lists[i]) {
56+ err = -EINVAL;
57+ goto err_put_free_lists;
58+ }
59+ }
60+
61+ if (hwrt->free_lists[ROGUE_FW_LOCAL_FREELIST]->current_pages <
62+ pvr_get_free_list_min_pages(pvr_dev)) {
63+ err = -EINVAL;
64+ goto err_put_free_lists;
65+ }
66+
67+ return 0;
68+
69+err_put_free_lists:
70+ for (i = 0; i < ARRAY_SIZE(hwrt->free_lists); i++) {
71+ pvr_free_list_put(hwrt->free_lists[i]);
72+ hwrt->free_lists[i] = NULL;
73+ }
74+
75+ return err;
76+}
77+
78+static void
79+hwrt_fini_kernel_structure(struct pvr_hwrt_dataset *hwrt)
80+{
81+ int i;
82+
83+ for (i = 0; i < ARRAY_SIZE(hwrt->free_lists); i++) {
84+ pvr_free_list_put(hwrt->free_lists[i]);
85+ hwrt->free_lists[i] = NULL;
86+ }
87+}
88+
89+static void
90+hwrt_fini_common_fw_structure(struct pvr_hwrt_dataset *hwrt)
91+{
92+ pvr_fw_object_destroy(hwrt->common_fw_obj);
93+}
94+
95+static int
96+get_cr_isp_mtile_size_val(struct pvr_device *pvr_dev, u32 samples,
97+ struct pvr_rt_mtile_info *info, u32 *value_out)
98+{
99+ u32 x = info->mtile_x[0];
100+ u32 y = info->mtile_y[0];
101+ u32 samples_per_pixel;
102+ int err;
103+
104+ err = PVR_FEATURE_VALUE(pvr_dev, isp_samples_per_pixel, &samples_per_pixel);
105+ if (err)
106+ return err;
107+
108+ if (samples_per_pixel == 1) {
109+ if (samples >= 4)
110+ x <<= 1;
111+ if (samples >= 2)
112+ y <<= 1;
113+ } else if (samples_per_pixel == 2) {
114+ if (samples >= 8)
115+ x <<= 1;
116+ if (samples >= 4)
117+ y <<= 1;
118+ } else if (samples_per_pixel == 4) {
119+ if (samples >= 8)
120+ y <<= 1;
121+ } else {
122+ WARN(true, "Unsupported ISP samples per pixel value");
123+ return -EINVAL;
124+ }
125+
126+ *value_out = ((x << ROGUE_CR_ISP_MTILE_SIZE_X_SHIFT) & ~ROGUE_CR_ISP_MTILE_SIZE_X_CLRMSK) |
127+ ((y << ROGUE_CR_ISP_MTILE_SIZE_Y_SHIFT) & ~ROGUE_CR_ISP_MTILE_SIZE_Y_CLRMSK);
128+
129+ return 0;
130+}
131+
132+static int
133+get_cr_multisamplectl_val(u32 samples, bool y_flip, u64 *value_out)
134+{
135+ static const struct {
136+ u8 x[8];
137+ u8 y[8];
138+ } sample_positions[4] = {
139+ /* 1 sample */
140+ {
141+ .x = { 8 },
142+ .y = { 8 },
143+ },
144+ /* 2 samples */
145+ {
146+ .x = { 12, 4 },
147+ .y = { 12, 4 },
148+ },
149+ /* 4 samples */
150+ {
151+ .x = { 6, 14, 2, 10 },
152+ .y = { 2, 6, 10, 14 },
153+ },
154+ /* 8 samples */
155+ {
156+ .x = { 9, 7, 13, 5, 3, 1, 11, 15 },
157+ .y = { 5, 11, 9, 3, 13, 7, 15, 1 },
158+ },
159+ };
160+ const int idx = fls(samples) - 1;
161+ u64 value = 0;
162+
163+ if (idx < 0 || idx > 3)
164+ return -EINVAL;
165+
166+ for (u32 i = 0; i < 8; i++) {
167+ value |= ((u64)sample_positions[idx].x[i]) << (i * 8);
168+ if (y_flip)
169+ value |= (((u64)(16 - sample_positions[idx].y[i]) & 0xf)) << (i * 8 + 4);
170+ else
171+ value |= ((u64)sample_positions[idx].y[i]) << (i * 8 + 4);
172+ }
173+
174+ *value_out = value;
175+
176+ return 0;
177+}
178+
179+static int
180+get_cr_te_aa_val(struct pvr_device *pvr_dev, u32 samples, u32 *value_out)
181+{
182+ u32 samples_per_pixel;
183+ u32 value = 0;
184+ int err = 0;
185+
186+ err = PVR_FEATURE_VALUE(pvr_dev, isp_samples_per_pixel, &samples_per_pixel);
187+ if (err)
188+ return err;
189+
190+ switch (samples_per_pixel) {
191+ case 1:
192+ if (samples >= 2)
193+ value |= ROGUE_CR_TE_AA_Y_EN;
194+ if (samples >= 4)
195+ value |= ROGUE_CR_TE_AA_X_EN;
196+ break;
197+ case 2:
198+ if (samples >= 2)
199+ value |= ROGUE_CR_TE_AA_X2_EN;
200+ if (samples >= 4)
201+ value |= ROGUE_CR_TE_AA_Y_EN;
202+ if (samples >= 8)
203+ value |= ROGUE_CR_TE_AA_X_EN;
204+ break;
205+ case 4:
206+ if (samples >= 2)
207+ value |= ROGUE_CR_TE_AA_X2_EN;
208+ if (samples >= 4)
209+ value |= ROGUE_CR_TE_AA_Y2_EN;
210+ if (samples >= 8)
211+ value |= ROGUE_CR_TE_AA_Y_EN;
212+ break;
213+ default:
214+ WARN(true, "Unsupported ISP samples per pixel value");
215+ return -EINVAL;
216+ }
217+
218+ *value_out = value;
219+
220+ return 0;
221+}
222+
223+static void
224+hwrtdata_common_init(void *cpu_ptr, void *priv)
225+{
226+ struct pvr_hwrt_dataset *hwrt = priv;
227+
228+ memcpy(cpu_ptr, &hwrt->common, sizeof(hwrt->common));
229+}
230+
231+static int
232+hwrt_init_common_fw_structure(struct pvr_file *pvr_file,
233+ struct drm_pvr_ioctl_create_hwrt_dataset_args *args,
234+ struct pvr_hwrt_dataset *hwrt)
235+{
236+ struct drm_pvr_create_hwrt_geom_data_args *geom_data_args = &args->geom_data_args;
237+ struct pvr_device *pvr_dev = pvr_file->pvr_dev;
238+ struct pvr_rt_mtile_info info;
239+ int err;
240+
241+ err = PVR_FEATURE_VALUE(pvr_dev, tile_size_x, &info.tile_size_x);
242+ if (WARN_ON(err))
243+ return err;
244+
245+ err = PVR_FEATURE_VALUE(pvr_dev, tile_size_y, &info.tile_size_y);
246+ if (WARN_ON(err))
247+ return err;
248+
249+ info.num_tiles_x = DIV_ROUND_UP(args->width, info.tile_size_x);
250+ info.num_tiles_y = DIV_ROUND_UP(args->height, info.tile_size_y);
251+
252+ if (PVR_HAS_FEATURE(pvr_dev, simple_parameter_format_version)) {
253+ u32 parameter_format;
254+
255+ err = PVR_FEATURE_VALUE(pvr_dev, simple_parameter_format_version,
256+ &parameter_format);
257+ if (WARN_ON(err))
258+ return err;
259+
260+ WARN_ON(parameter_format != 2);
261+
262+ /*
263+ * Set up 16 macrotiles with a multiple of 2x2 tiles per macrotile, which is
264+ * aligned to a tile group.
265+ */
266+ info.mtile_x[0] = DIV_ROUND_UP(info.num_tiles_x, 8) * 2;
267+ info.mtile_y[0] = DIV_ROUND_UP(info.num_tiles_y, 8) * 2;
268+ info.mtile_x[1] = 0;
269+ info.mtile_y[1] = 0;
270+ info.mtile_x[2] = 0;
271+ info.mtile_y[2] = 0;
272+ info.tile_max_x = round_up(info.num_tiles_x, 2) - 1;
273+ info.tile_max_y = round_up(info.num_tiles_y, 2) - 1;
274+ } else {
275+ /* Set up 16 macrotiles with a multiple of 4x4 tiles per macrotile. */
276+ info.mtile_x[0] = round_up(DIV_ROUND_UP(info.num_tiles_x, 4), 4);
277+ info.mtile_y[0] = round_up(DIV_ROUND_UP(info.num_tiles_y, 4), 4);
278+ info.mtile_x[1] = info.mtile_x[0] * 2;
279+ info.mtile_y[1] = info.mtile_y[0] * 2;
280+ info.mtile_x[2] = info.mtile_x[0] * 3;
281+ info.mtile_y[2] = info.mtile_y[0] * 3;
282+ info.tile_max_x = info.num_tiles_x - 1;
283+ info.tile_max_y = info.num_tiles_y - 1;
284+ }
285+
286+ hwrt->common.geom_caches_need_zeroing = false;
287+
288+ hwrt->common.isp_merge_lower_x = args->isp_merge_lower_x;
289+ hwrt->common.isp_merge_lower_y = args->isp_merge_lower_y;
290+ hwrt->common.isp_merge_upper_x = args->isp_merge_upper_x;
291+ hwrt->common.isp_merge_upper_y = args->isp_merge_upper_y;
292+ hwrt->common.isp_merge_scale_x = args->isp_merge_scale_x;
293+ hwrt->common.isp_merge_scale_y = args->isp_merge_scale_y;
294+
295+ err = get_cr_multisamplectl_val(args->samples, false,
296+ &hwrt->common.multi_sample_ctl);
297+ if (err)
298+ return err;
299+
300+ err = get_cr_multisamplectl_val(args->samples, true,
301+ &hwrt->common.flipped_multi_sample_ctl);
302+ if (err)
303+ return err;
304+
305+ hwrt->common.mtile_stride = info.mtile_x[0] * info.mtile_y[0];
306+
307+ err = get_cr_te_aa_val(pvr_dev, args->samples, &hwrt->common.teaa);
308+ if (err)
309+ return err;
310+
311+ hwrt->common.screen_pixel_max =
312+ (((args->width - 1) << ROGUE_CR_PPP_SCREEN_PIXXMAX_SHIFT) &
313+ ~ROGUE_CR_PPP_SCREEN_PIXXMAX_CLRMSK) |
314+ (((args->height - 1) << ROGUE_CR_PPP_SCREEN_PIXYMAX_SHIFT) &
315+ ~ROGUE_CR_PPP_SCREEN_PIXYMAX_CLRMSK);
316+
317+ hwrt->common.te_screen =
318+ ((info.tile_max_x << ROGUE_CR_TE_SCREEN_XMAX_SHIFT) &
319+ ~ROGUE_CR_TE_SCREEN_XMAX_CLRMSK) |
320+ ((info.tile_max_y << ROGUE_CR_TE_SCREEN_YMAX_SHIFT) &
321+ ~ROGUE_CR_TE_SCREEN_YMAX_CLRMSK);
322+ hwrt->common.te_mtile1 =
323+ ((info.mtile_x[0] << ROGUE_CR_TE_MTILE1_X1_SHIFT) & ~ROGUE_CR_TE_MTILE1_X1_CLRMSK) |
324+ ((info.mtile_x[1] << ROGUE_CR_TE_MTILE1_X2_SHIFT) & ~ROGUE_CR_TE_MTILE1_X2_CLRMSK) |
325+ ((info.mtile_x[2] << ROGUE_CR_TE_MTILE1_X3_SHIFT) & ~ROGUE_CR_TE_MTILE1_X3_CLRMSK);
326+ hwrt->common.te_mtile2 =
327+ ((info.mtile_y[0] << ROGUE_CR_TE_MTILE2_Y1_SHIFT) & ~ROGUE_CR_TE_MTILE2_Y1_CLRMSK) |
328+ ((info.mtile_y[1] << ROGUE_CR_TE_MTILE2_Y2_SHIFT) & ~ROGUE_CR_TE_MTILE2_Y2_CLRMSK) |
329+ ((info.mtile_y[2] << ROGUE_CR_TE_MTILE2_Y3_SHIFT) & ~ROGUE_CR_TE_MTILE2_Y3_CLRMSK);
330+
331+ err = get_cr_isp_mtile_size_val(pvr_dev, args->samples, &info,
332+ &hwrt->common.isp_mtile_size);
333+ if (err)
334+ return err;
335+
336+ hwrt->common.tpc_stride = geom_data_args->tpc_stride;
337+ hwrt->common.tpc_size = geom_data_args->tpc_size;
338+
339+ hwrt->common.rgn_header_size = args->region_header_size;
340+
341+ err = pvr_fw_object_create(pvr_dev, sizeof(struct rogue_fwif_hwrtdata_common),
342+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED, hwrtdata_common_init, hwrt,
343+ &hwrt->common_fw_obj);
344+
345+ return err;
346+}
347+
348+static void
349+hwrt_fw_data_init(void *cpu_ptr, void *priv)
350+{
351+ struct pvr_hwrt_data *hwrt_data = priv;
352+
353+ memcpy(cpu_ptr, &hwrt_data->data, sizeof(hwrt_data->data));
354+}
355+
356+static int
357+hwrt_data_init_fw_structure(struct pvr_file *pvr_file,
358+ struct pvr_hwrt_dataset *hwrt,
359+ struct drm_pvr_ioctl_create_hwrt_dataset_args *args,
360+ struct drm_pvr_create_hwrt_rt_data_args *rt_data_args,
361+ struct pvr_hwrt_data *hwrt_data)
362+{
363+ struct drm_pvr_create_hwrt_geom_data_args *geom_data_args = &args->geom_data_args;
364+ struct pvr_device *pvr_dev = pvr_file->pvr_dev;
365+ struct rogue_fwif_rta_ctl *rta_ctl;
366+ int free_list_i;
367+ int err;
368+
369+ pvr_fw_object_get_fw_addr(hwrt->common_fw_obj,
370+ &hwrt_data->data.hwrt_data_common_fw_addr);
371+
372+ for (free_list_i = 0; free_list_i < ARRAY_SIZE(hwrt->free_lists); free_list_i++) {
373+ pvr_fw_object_get_fw_addr(hwrt->free_lists[free_list_i]->fw_obj,
374+ &hwrt_data->data.freelists_fw_addr[free_list_i]);
375+ }
376+
377+ hwrt_data->data.tail_ptrs_dev_addr = geom_data_args->tpc_dev_addr;
378+ hwrt_data->data.vheap_table_dev_addr = geom_data_args->vheap_table_dev_addr;
379+ hwrt_data->data.rtc_dev_addr = geom_data_args->rtc_dev_addr;
380+
381+ hwrt_data->data.pm_mlist_dev_addr = rt_data_args->pm_mlist_dev_addr;
382+ hwrt_data->data.macrotile_array_dev_addr = rt_data_args->macrotile_array_dev_addr;
383+ hwrt_data->data.rgn_header_dev_addr = rt_data_args->region_header_dev_addr;
384+
385+ rta_ctl = &hwrt_data->data.rta_ctl;
386+
387+ rta_ctl->render_target_index = 0;
388+ rta_ctl->active_render_targets = 0;
389+ rta_ctl->valid_render_targets_fw_addr = 0;
390+ rta_ctl->rta_num_partial_renders_fw_addr = 0;
391+ rta_ctl->max_rts = args->layers;
392+
393+ if (args->layers > 1) {
394+ err = pvr_fw_object_create(pvr_dev, args->layers * SRTC_ENTRY_SIZE,
395+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
396+ NULL, NULL, &hwrt_data->srtc_obj);
397+ if (err)
398+ return err;
399+ pvr_fw_object_get_fw_addr(hwrt_data->srtc_obj,
400+ &rta_ctl->valid_render_targets_fw_addr);
401+
402+ err = pvr_fw_object_create(pvr_dev, args->layers * RAA_ENTRY_SIZE,
403+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
404+ NULL, NULL, &hwrt_data->raa_obj);
405+ if (err)
406+ goto err_put_shadow_rt_cache;
407+ pvr_fw_object_get_fw_addr(hwrt_data->raa_obj,
408+ &rta_ctl->rta_num_partial_renders_fw_addr);
409+ }
410+
411+ err = pvr_fw_object_create(pvr_dev, sizeof(struct rogue_fwif_hwrtdata),
412+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
413+ hwrt_fw_data_init, hwrt_data, &hwrt_data->fw_obj);
414+ if (err)
415+ goto err_put_raa_obj;
416+
417+ pvr_free_list_add_hwrt(hwrt->free_lists[0], hwrt_data);
418+
419+ return 0;
420+
421+err_put_raa_obj:
422+ if (args->layers > 1)
423+ pvr_fw_object_destroy(hwrt_data->raa_obj);
424+
425+err_put_shadow_rt_cache:
426+ if (args->layers > 1)
427+ pvr_fw_object_destroy(hwrt_data->srtc_obj);
428+
429+ return err;
430+}
431+
432+static void
433+hwrt_data_fini_fw_structure(struct pvr_hwrt_dataset *hwrt, int hwrt_nr)
434+{
435+ struct pvr_hwrt_data *hwrt_data = &hwrt->data[hwrt_nr];
436+
437+ pvr_free_list_remove_hwrt(hwrt->free_lists[0], hwrt_data);
438+
439+ if (hwrt->max_rts > 1) {
440+ pvr_fw_object_destroy(hwrt_data->raa_obj);
441+ pvr_fw_object_destroy(hwrt_data->srtc_obj);
442+ }
443+
444+ pvr_fw_object_destroy(hwrt_data->fw_obj);
445+}
446+
447+/**
448+ * pvr_hwrt_dataset_create() - Create a new HWRT dataset
449+ * @pvr_file: Pointer to pvr_file structure.
450+ * @args: Creation arguments from userspace.
451+ *
452+ * Return:
453+ * * Pointer to new HWRT, or
454+ * * ERR_PTR(-%ENOMEM) on out of memory.
455+ */
456+struct pvr_hwrt_dataset *
457+pvr_hwrt_dataset_create(struct pvr_file *pvr_file,
458+ struct drm_pvr_ioctl_create_hwrt_dataset_args *args)
459+{
460+ struct pvr_hwrt_dataset *hwrt;
461+ int err;
462+
463+ /* Create and fill out the kernel structure */
464+ hwrt = kzalloc(sizeof(*hwrt), GFP_KERNEL);
465+
466+ if (!hwrt)
467+ return ERR_PTR(-ENOMEM);
468+
469+ kref_init(&hwrt->ref_count);
470+
471+ err = hwrt_init_kernel_structure(pvr_file, args, hwrt);
472+ if (err < 0)
473+ goto err_free;
474+
475+ err = hwrt_init_common_fw_structure(pvr_file, args, hwrt);
476+ if (err < 0)
477+ goto err_free;
478+
479+ for (int i = 0; i < ARRAY_SIZE(hwrt->data); i++) {
480+ err = hwrt_data_init_fw_structure(pvr_file, hwrt, args,
481+ &args->rt_data_args[i],
482+ &hwrt->data[i]);
483+ if (err < 0) {
484+ i--;
485+ /* Destroy already created structures. */
486+ for (; i >= 0; i--)
487+ hwrt_data_fini_fw_structure(hwrt, i);
488+ goto err_free;
489+ }
490+
491+ hwrt->data[i].hwrt_dataset = hwrt;
492+ }
493+
494+ return hwrt;
495+
496+err_free:
497+ pvr_hwrt_dataset_put(hwrt);
498+
499+ return ERR_PTR(err);
500+}
501+
502+static void
503+pvr_hwrt_dataset_release(struct kref *ref_count)
504+{
505+ struct pvr_hwrt_dataset *hwrt =
506+ container_of(ref_count, struct pvr_hwrt_dataset, ref_count);
507+
508+ for (int i = ARRAY_SIZE(hwrt->data) - 1; i >= 0; i--) {
509+ WARN_ON(pvr_fw_structure_cleanup(hwrt->pvr_dev, ROGUE_FWIF_CLEANUP_HWRTDATA,
510+ hwrt->data[i].fw_obj, 0));
511+ hwrt_data_fini_fw_structure(hwrt, i);
512+ }
513+
514+ hwrt_fini_common_fw_structure(hwrt);
515+ hwrt_fini_kernel_structure(hwrt);
516+
517+ kfree(hwrt);
518+}
519+
520+/**
521+ * pvr_destroy_hwrt_datasets_for_file: Destroy any HWRT datasets associated
522+ * with the given file.
523+ * @pvr_file: Pointer to pvr_file structure.
524+ *
525+ * Removes all HWRT datasets associated with @pvr_file from the device
526+ * hwrt_dataset list and drops initial references. HWRT datasets will then be
527+ * destroyed once all outstanding references are dropped.
528+ */
529+void pvr_destroy_hwrt_datasets_for_file(struct pvr_file *pvr_file)
530+{
531+ struct pvr_hwrt_dataset *hwrt;
532+ unsigned long handle;
533+
534+ xa_for_each(&pvr_file->hwrt_handles, handle, hwrt) {
535+ (void)hwrt;
536+ pvr_hwrt_dataset_put(xa_erase(&pvr_file->hwrt_handles, handle));
537+ }
538+}
539+
540+/**
541+ * pvr_hwrt_dataset_put() - Release reference on HWRT dataset
542+ * @hwrt: Pointer to HWRT dataset to release reference on
543+ */
544+void
545+pvr_hwrt_dataset_put(struct pvr_hwrt_dataset *hwrt)
546+{
547+ if (hwrt)
548+ kref_put(&hwrt->ref_count, pvr_hwrt_dataset_release);
549+}
drivers/gpu/drm/imagination/pvr_hwrt.h+165−0
@@ -0,0 +1,165 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_HWRT_H
5+#define PVR_HWRT_H
6+
7+#include <linux/compiler_attributes.h>
8+#include <linux/kref.h>
9+#include <linux/list.h>
10+#include <linux/types.h>
11+#include <linux/xarray.h>
12+#include <uapi/drm/pvr_drm.h>
13+
14+#include "pvr_device.h"
15+#include "pvr_rogue_fwif_shared.h"
16+
17+/* Forward declaration from pvr_free_list.h. */
18+struct pvr_free_list;
19+
20+/* Forward declaration from pvr_gem.h. */
21+struct pvr_fw_object;
22+
23+/**
24+ * struct pvr_hwrt_data - structure representing HWRT data
25+ */
26+struct pvr_hwrt_data {
27+ /** @fw_obj: FW object representing the FW-side structure. */
28+ struct pvr_fw_object *fw_obj;
29+
30+ /** @data: Local copy of FW-side structure. */
31+ struct rogue_fwif_hwrtdata data;
32+
33+ /** @freelist_node: List node connecting this HWRT to the local freelist. */
34+ struct list_head freelist_node;
35+
36+ /**
37+ * @srtc_obj: FW object representing shadow render target cache.
38+ *
39+ * Only valid if @max_rts > 1.
40+ */
41+ struct pvr_fw_object *srtc_obj;
42+
43+ /**
44+ * @raa_obj: FW object representing renders accumulation array.
45+ *
46+ * Only valid if @max_rts > 1.
47+ */
48+ struct pvr_fw_object *raa_obj;
49+
50+ /** @hwrt_dataset: Back pointer to owning HWRT dataset. */
51+ struct pvr_hwrt_dataset *hwrt_dataset;
52+};
53+
54+/**
55+ * struct pvr_hwrt_dataset - structure representing a HWRT data set.
56+ */
57+struct pvr_hwrt_dataset {
58+ /** @ref_count: Reference count of object. */
59+ struct kref ref_count;
60+
61+ /** @pvr_dev: Pointer to device that owns this object. */
62+ struct pvr_device *pvr_dev;
63+
64+ /** @common_fw_obj: FW object representing common FW-side structure. */
65+ struct pvr_fw_object *common_fw_obj;
66+
67+ struct rogue_fwif_hwrtdata_common common;
68+
69+ /** @data: HWRT data structures belonging to this set. */
70+ struct pvr_hwrt_data data[ROGUE_FWIF_NUM_RTDATAS];
71+
72+ /** @free_lists: Free lists used by HWRT data set. */
73+ struct pvr_free_list *free_lists[ROGUE_FWIF_NUM_RTDATA_FREELISTS];
74+
75+ /** @max_rts: Maximum render targets for this HWRT data set. */
76+ u16 max_rts;
77+};
78+
79+struct pvr_hwrt_dataset *
80+pvr_hwrt_dataset_create(struct pvr_file *pvr_file,
81+ struct drm_pvr_ioctl_create_hwrt_dataset_args *args);
82+
83+void
84+pvr_destroy_hwrt_datasets_for_file(struct pvr_file *pvr_file);
85+
86+/**
87+ * pvr_hwrt_dataset_lookup() - Lookup HWRT dataset pointer from handle
88+ * @pvr_file: Pointer to pvr_file structure.
89+ * @handle: Object handle.
90+ *
91+ * Takes reference on dataset object. Call pvr_hwrt_dataset_put() to release.
92+ *
93+ * Returns:
94+ * * The requested object on success, or
95+ * * %NULL on failure (object does not exist in list, or is not a HWRT
96+ * dataset)
97+ */
98+static __always_inline struct pvr_hwrt_dataset *
99+pvr_hwrt_dataset_lookup(struct pvr_file *pvr_file, u32 handle)
100+{
101+ struct pvr_hwrt_dataset *hwrt;
102+
103+ xa_lock(&pvr_file->hwrt_handles);
104+ hwrt = xa_load(&pvr_file->hwrt_handles, handle);
105+
106+ if (hwrt)
107+ kref_get(&hwrt->ref_count);
108+
109+ xa_unlock(&pvr_file->hwrt_handles);
110+
111+ return hwrt;
112+}
113+
114+void
115+pvr_hwrt_dataset_put(struct pvr_hwrt_dataset *hwrt);
116+
117+/**
118+ * pvr_hwrt_data_lookup() - Lookup HWRT data pointer from handle and index
119+ * @pvr_file: Pointer to pvr_file structure.
120+ * @handle: Object handle.
121+ * @index: Index of RT data within dataset.
122+ *
123+ * Takes reference on dataset object. Call pvr_hwrt_data_put() to release.
124+ *
125+ * Returns:
126+ * * The requested object on success, or
127+ * * %NULL on failure (object does not exist in list, or is not a HWRT
128+ * dataset, or index is out of range)
129+ */
130+static __always_inline struct pvr_hwrt_data *
131+pvr_hwrt_data_lookup(struct pvr_file *pvr_file, u32 handle, u32 index)
132+{
133+ struct pvr_hwrt_dataset *hwrt_dataset = pvr_hwrt_dataset_lookup(pvr_file, handle);
134+
135+ if (hwrt_dataset) {
136+ if (index < ARRAY_SIZE(hwrt_dataset->data))
137+ return &hwrt_dataset->data[index];
138+
139+ pvr_hwrt_dataset_put(hwrt_dataset);
140+ }
141+
142+ return NULL;
143+}
144+
145+/**
146+ * pvr_hwrt_data_put() - Release reference on HWRT data
147+ * @hwrt: Pointer to HWRT data to release reference on
148+ */
149+static __always_inline void
150+pvr_hwrt_data_put(struct pvr_hwrt_data *hwrt)
151+{
152+ if (hwrt)
153+ pvr_hwrt_dataset_put(hwrt->hwrt_dataset);
154+}
155+
156+static __always_inline struct pvr_hwrt_data *
157+pvr_hwrt_data_get(struct pvr_hwrt_data *hwrt)
158+{
159+ if (hwrt)
160+ kref_get(&hwrt->hwrt_dataset->ref_count);
161+
162+ return hwrt;
163+}
164+
165+#endif /* PVR_HWRT_H */
drivers/gpu/drm/imagination/pvr_job.c+788−0
@@ -0,0 +1,788 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_context.h"
5+#include "pvr_device.h"
6+#include "pvr_drv.h"
7+#include "pvr_gem.h"
8+#include "pvr_hwrt.h"
9+#include "pvr_job.h"
10+#include "pvr_mmu.h"
11+#include "pvr_power.h"
12+#include "pvr_rogue_fwif.h"
13+#include "pvr_rogue_fwif_client.h"
14+#include "pvr_stream.h"
15+#include "pvr_stream_defs.h"
16+#include "pvr_sync.h"
17+
18+#include <drm/drm_exec.h>
19+#include <drm/drm_gem.h>
20+#include <linux/types.h>
21+#include <uapi/drm/pvr_drm.h>
22+
23+static void pvr_job_release(struct kref *kref)
24+{
25+ struct pvr_job *job = container_of(kref, struct pvr_job, ref_count);
26+
27+ xa_erase(&job->pvr_dev->job_ids, job->id);
28+
29+ pvr_hwrt_data_put(job->hwrt);
30+ pvr_context_put(job->ctx);
31+
32+ WARN_ON(job->paired_job);
33+
34+ pvr_queue_job_cleanup(job);
35+ pvr_job_release_pm_ref(job);
36+
37+ kfree(job->cmd);
38+ kfree(job);
39+}
40+
41+/**
42+ * pvr_job_put() - Release reference on job
43+ * @job: Target job.
44+ */
45+void
46+pvr_job_put(struct pvr_job *job)
47+{
48+ if (job)
49+ kref_put(&job->ref_count, pvr_job_release);
50+}
51+
52+/**
53+ * pvr_job_process_stream() - Build job FW structure from stream
54+ * @pvr_dev: Device pointer.
55+ * @cmd_defs: Stream definition.
56+ * @stream: Pointer to command stream.
57+ * @stream_size: Size of command stream, in bytes.
58+ * @job: Pointer to job.
59+ *
60+ * Caller is responsible for freeing the output structure.
61+ *
62+ * Returns:
63+ * * 0 on success,
64+ * * -%ENOMEM on out of memory, or
65+ * * -%EINVAL on malformed stream.
66+ */
67+static int
68+pvr_job_process_stream(struct pvr_device *pvr_dev, const struct pvr_stream_cmd_defs *cmd_defs,
69+ void *stream, u32 stream_size, struct pvr_job *job)
70+{
71+ int err;
72+
73+ job->cmd = kzalloc(cmd_defs->dest_size, GFP_KERNEL);
74+ if (!job->cmd)
75+ return -ENOMEM;
76+
77+ job->cmd_len = cmd_defs->dest_size;
78+
79+ err = pvr_stream_process(pvr_dev, cmd_defs, stream, stream_size, job->cmd);
80+ if (err)
81+ kfree(job->cmd);
82+
83+ return err;
84+}
85+
86+static int pvr_fw_cmd_init(struct pvr_device *pvr_dev, struct pvr_job *job,
87+ const struct pvr_stream_cmd_defs *stream_def,
88+ u64 stream_userptr, u32 stream_len)
89+{
90+ void *stream;
91+ int err;
92+
93+ stream = kzalloc(stream_len, GFP_KERNEL);
94+ if (!stream)
95+ return -ENOMEM;
96+
97+ if (copy_from_user(stream, u64_to_user_ptr(stream_userptr), stream_len)) {
98+ err = -EFAULT;
99+ goto err_free_stream;
100+ }
101+
102+ err = pvr_job_process_stream(pvr_dev, stream_def, stream, stream_len, job);
103+
104+err_free_stream:
105+ kfree(stream);
106+
107+ return err;
108+}
109+
110+static u32
111+convert_geom_flags(u32 in_flags)
112+{
113+ u32 out_flags = 0;
114+
115+ if (in_flags & DRM_PVR_SUBMIT_JOB_GEOM_CMD_FIRST)
116+ out_flags |= ROGUE_GEOM_FLAGS_FIRSTKICK;
117+ if (in_flags & DRM_PVR_SUBMIT_JOB_GEOM_CMD_LAST)
118+ out_flags |= ROGUE_GEOM_FLAGS_LASTKICK;
119+ if (in_flags & DRM_PVR_SUBMIT_JOB_GEOM_CMD_SINGLE_CORE)
120+ out_flags |= ROGUE_GEOM_FLAGS_SINGLE_CORE;
121+
122+ return out_flags;
123+}
124+
125+static u32
126+convert_frag_flags(u32 in_flags)
127+{
128+ u32 out_flags = 0;
129+
130+ if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_SINGLE_CORE)
131+ out_flags |= ROGUE_FRAG_FLAGS_SINGLE_CORE;
132+ if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_DEPTHBUFFER)
133+ out_flags |= ROGUE_FRAG_FLAGS_DEPTHBUFFER;
134+ if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_STENCILBUFFER)
135+ out_flags |= ROGUE_FRAG_FLAGS_STENCILBUFFER;
136+ if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_PREVENT_CDM_OVERLAP)
137+ out_flags |= ROGUE_FRAG_FLAGS_PREVENT_CDM_OVERLAP;
138+ if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_SCRATCHBUFFER)
139+ out_flags |= ROGUE_FRAG_FLAGS_SCRATCHBUFFER;
140+ if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_GET_VIS_RESULTS)
141+ out_flags |= ROGUE_FRAG_FLAGS_GET_VIS_RESULTS;
142+ if (in_flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_DISABLE_PIXELMERGE)
143+ out_flags |= ROGUE_FRAG_FLAGS_DISABLE_PIXELMERGE;
144+
145+ return out_flags;
146+}
147+
148+static int
149+pvr_geom_job_fw_cmd_init(struct pvr_job *job,
150+ struct drm_pvr_job *args)
151+{
152+ struct rogue_fwif_cmd_geom *cmd;
153+ int err;
154+
155+ if (args->flags & ~DRM_PVR_SUBMIT_JOB_GEOM_CMD_FLAGS_MASK)
156+ return -EINVAL;
157+
158+ if (job->ctx->type != DRM_PVR_CTX_TYPE_RENDER)
159+ return -EINVAL;
160+
161+ if (!job->hwrt)
162+ return -EINVAL;
163+
164+ job->fw_ccb_cmd_type = ROGUE_FWIF_CCB_CMD_TYPE_GEOM;
165+ err = pvr_fw_cmd_init(job->pvr_dev, job, &pvr_cmd_geom_stream,
166+ args->cmd_stream, args->cmd_stream_len);
167+ if (err)
168+ return err;
169+
170+ cmd = job->cmd;
171+ cmd->cmd_shared.cmn.frame_num = 0;
172+ cmd->flags = convert_geom_flags(args->flags);
173+ pvr_fw_object_get_fw_addr(job->hwrt->fw_obj, &cmd->cmd_shared.hwrt_data_fw_addr);
174+ return 0;
175+}
176+
177+static int
178+pvr_frag_job_fw_cmd_init(struct pvr_job *job,
179+ struct drm_pvr_job *args)
180+{
181+ struct rogue_fwif_cmd_frag *cmd;
182+ int err;
183+
184+ if (args->flags & ~DRM_PVR_SUBMIT_JOB_FRAG_CMD_FLAGS_MASK)
185+ return -EINVAL;
186+
187+ if (job->ctx->type != DRM_PVR_CTX_TYPE_RENDER)
188+ return -EINVAL;
189+
190+ if (!job->hwrt)
191+ return -EINVAL;
192+
193+ job->fw_ccb_cmd_type = (args->flags & DRM_PVR_SUBMIT_JOB_FRAG_CMD_PARTIAL_RENDER) ?
194+ ROGUE_FWIF_CCB_CMD_TYPE_FRAG_PR :
195+ ROGUE_FWIF_CCB_CMD_TYPE_FRAG;
196+ err = pvr_fw_cmd_init(job->pvr_dev, job, &pvr_cmd_frag_stream,
197+ args->cmd_stream, args->cmd_stream_len);
198+ if (err)
199+ return err;
200+
201+ cmd = job->cmd;
202+ cmd->cmd_shared.cmn.frame_num = 0;
203+ cmd->flags = convert_frag_flags(args->flags);
204+ pvr_fw_object_get_fw_addr(job->hwrt->fw_obj, &cmd->cmd_shared.hwrt_data_fw_addr);
205+ return 0;
206+}
207+
208+static u32
209+convert_compute_flags(u32 in_flags)
210+{
211+ u32 out_flags = 0;
212+
213+ if (in_flags & DRM_PVR_SUBMIT_JOB_COMPUTE_CMD_PREVENT_ALL_OVERLAP)
214+ out_flags |= ROGUE_COMPUTE_FLAG_PREVENT_ALL_OVERLAP;
215+ if (in_flags & DRM_PVR_SUBMIT_JOB_COMPUTE_CMD_SINGLE_CORE)
216+ out_flags |= ROGUE_COMPUTE_FLAG_SINGLE_CORE;
217+
218+ return out_flags;
219+}
220+
221+static int
222+pvr_compute_job_fw_cmd_init(struct pvr_job *job,
223+ struct drm_pvr_job *args)
224+{
225+ struct rogue_fwif_cmd_compute *cmd;
226+ int err;
227+
228+ if (args->flags & ~DRM_PVR_SUBMIT_JOB_COMPUTE_CMD_FLAGS_MASK)
229+ return -EINVAL;
230+
231+ if (job->ctx->type != DRM_PVR_CTX_TYPE_COMPUTE)
232+ return -EINVAL;
233+
234+ job->fw_ccb_cmd_type = ROGUE_FWIF_CCB_CMD_TYPE_CDM;
235+ err = pvr_fw_cmd_init(job->pvr_dev, job, &pvr_cmd_compute_stream,
236+ args->cmd_stream, args->cmd_stream_len);
237+ if (err)
238+ return err;
239+
240+ cmd = job->cmd;
241+ cmd->common.frame_num = 0;
242+ cmd->flags = convert_compute_flags(args->flags);
243+ return 0;
244+}
245+
246+static u32
247+convert_transfer_flags(u32 in_flags)
248+{
249+ u32 out_flags = 0;
250+
251+ if (in_flags & DRM_PVR_SUBMIT_JOB_TRANSFER_CMD_SINGLE_CORE)
252+ out_flags |= ROGUE_TRANSFER_FLAGS_SINGLE_CORE;
253+
254+ return out_flags;
255+}
256+
257+static int
258+pvr_transfer_job_fw_cmd_init(struct pvr_job *job,
259+ struct drm_pvr_job *args)
260+{
261+ struct rogue_fwif_cmd_transfer *cmd;
262+ int err;
263+
264+ if (args->flags & ~DRM_PVR_SUBMIT_JOB_TRANSFER_CMD_FLAGS_MASK)
265+ return -EINVAL;
266+
267+ if (job->ctx->type != DRM_PVR_CTX_TYPE_TRANSFER_FRAG)
268+ return -EINVAL;
269+
270+ job->fw_ccb_cmd_type = ROGUE_FWIF_CCB_CMD_TYPE_TQ_3D;
271+ err = pvr_fw_cmd_init(job->pvr_dev, job, &pvr_cmd_transfer_stream,
272+ args->cmd_stream, args->cmd_stream_len);
273+ if (err)
274+ return err;
275+
276+ cmd = job->cmd;
277+ cmd->common.frame_num = 0;
278+ cmd->flags = convert_transfer_flags(args->flags);
279+ return 0;
280+}
281+
282+static int
283+pvr_job_fw_cmd_init(struct pvr_job *job,
284+ struct drm_pvr_job *args)
285+{
286+ switch (args->type) {
287+ case DRM_PVR_JOB_TYPE_GEOMETRY:
288+ return pvr_geom_job_fw_cmd_init(job, args);
289+
290+ case DRM_PVR_JOB_TYPE_FRAGMENT:
291+ return pvr_frag_job_fw_cmd_init(job, args);
292+
293+ case DRM_PVR_JOB_TYPE_COMPUTE:
294+ return pvr_compute_job_fw_cmd_init(job, args);
295+
296+ case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
297+ return pvr_transfer_job_fw_cmd_init(job, args);
298+
299+ default:
300+ return -EINVAL;
301+ }
302+}
303+
304+/**
305+ * struct pvr_job_data - Helper container for pairing jobs with the
306+ * sync_ops supplied for them by the user.
307+ */
308+struct pvr_job_data {
309+ /** @job: Pointer to the job. */
310+ struct pvr_job *job;
311+
312+ /** @sync_ops: Pointer to the sync_ops associated with @job. */
313+ struct drm_pvr_sync_op *sync_ops;
314+
315+ /** @sync_op_count: Number of members of @sync_ops. */
316+ u32 sync_op_count;
317+};
318+
319+/**
320+ * prepare_job_syncs() - Prepare all sync objects for a single job.
321+ * @pvr_file: PowerVR file.
322+ * @job_data: Precreated job and sync_ops array.
323+ * @signal_array: xarray to receive signal sync objects.
324+ *
325+ * Returns:
326+ * * 0 on success, or
327+ * * Any error code returned by pvr_sync_signal_array_collect_ops(),
328+ * pvr_sync_add_deps_to_job(), drm_sched_job_add_resv_dependencies() or
329+ * pvr_sync_signal_array_update_fences().
330+ */
331+static int
332+prepare_job_syncs(struct pvr_file *pvr_file,
333+ struct pvr_job_data *job_data,
334+ struct xarray *signal_array)
335+{
336+ struct dma_fence *done_fence;
337+ int err = pvr_sync_signal_array_collect_ops(signal_array,
338+ from_pvr_file(pvr_file),
339+ job_data->sync_op_count,
340+ job_data->sync_ops);
341+
342+ if (err)
343+ return err;
344+
345+ err = pvr_sync_add_deps_to_job(pvr_file, &job_data->job->base,
346+ job_data->sync_op_count,
347+ job_data->sync_ops, signal_array);
348+ if (err)
349+ return err;
350+
351+ if (job_data->job->hwrt) {
352+ /* The geometry job writes the HWRT region headers, which are
353+ * then read by the fragment job.
354+ */
355+ struct drm_gem_object *obj =
356+ gem_from_pvr_gem(job_data->job->hwrt->fw_obj->gem);
357+ enum dma_resv_usage usage =
358+ dma_resv_usage_rw(job_data->job->type ==
359+ DRM_PVR_JOB_TYPE_GEOMETRY);
360+
361+ dma_resv_lock(obj->resv, NULL);
362+ err = drm_sched_job_add_resv_dependencies(&job_data->job->base,
363+ obj->resv, usage);
364+ dma_resv_unlock(obj->resv);
365+ if (err)
366+ return err;
367+ }
368+
369+ /* We need to arm the job to get the job done fence. */
370+ done_fence = pvr_queue_job_arm(job_data->job);
371+
372+ err = pvr_sync_signal_array_update_fences(signal_array,
373+ job_data->sync_op_count,
374+ job_data->sync_ops,
375+ done_fence);
376+ return err;
377+}
378+
379+/**
380+ * prepare_job_syncs_for_each() - Prepare all sync objects for an array of jobs.
381+ * @file: PowerVR file.
382+ * @job_data: Array of precreated jobs and their sync_ops.
383+ * @job_count: Number of jobs.
384+ * @signal_array: xarray to receive signal sync objects.
385+ *
386+ * Returns:
387+ * * 0 on success, or
388+ * * Any error code returned by pvr_vm_bind_job_prepare_syncs().
389+ */
390+static int
391+prepare_job_syncs_for_each(struct pvr_file *pvr_file,
392+ struct pvr_job_data *job_data,
393+ u32 *job_count,
394+ struct xarray *signal_array)
395+{
396+ for (u32 i = 0; i < *job_count; i++) {
397+ int err = prepare_job_syncs(pvr_file, &job_data[i],
398+ signal_array);
399+
400+ if (err) {
401+ *job_count = i;
402+ return err;
403+ }
404+ }
405+
406+ return 0;
407+}
408+
409+static struct pvr_job *
410+create_job(struct pvr_device *pvr_dev,
411+ struct pvr_file *pvr_file,
412+ struct drm_pvr_job *args)
413+{
414+ struct pvr_job *job = NULL;
415+ int err;
416+
417+ if (!args->cmd_stream || !args->cmd_stream_len)
418+ return ERR_PTR(-EINVAL);
419+
420+ if (args->type != DRM_PVR_JOB_TYPE_GEOMETRY &&
421+ args->type != DRM_PVR_JOB_TYPE_FRAGMENT &&
422+ (args->hwrt.set_handle || args->hwrt.data_index))
423+ return ERR_PTR(-EINVAL);
424+
425+ job = kzalloc(sizeof(*job), GFP_KERNEL);
426+ if (!job)
427+ return ERR_PTR(-ENOMEM);
428+
429+ kref_init(&job->ref_count);
430+ job->type = args->type;
431+ job->pvr_dev = pvr_dev;
432+
433+ err = xa_alloc(&pvr_dev->job_ids, &job->id, job, xa_limit_32b, GFP_KERNEL);
434+ if (err)
435+ goto err_put_job;
436+
437+ job->ctx = pvr_context_lookup(pvr_file, args->context_handle);
438+ if (!job->ctx) {
439+ err = -EINVAL;
440+ goto err_put_job;
441+ }
442+
443+ if (args->hwrt.set_handle) {
444+ job->hwrt = pvr_hwrt_data_lookup(pvr_file, args->hwrt.set_handle,
445+ args->hwrt.data_index);
446+ if (!job->hwrt) {
447+ err = -EINVAL;
448+ goto err_put_job;
449+ }
450+ }
451+
452+ err = pvr_job_fw_cmd_init(job, args);
453+ if (err)
454+ goto err_put_job;
455+
456+ err = pvr_queue_job_init(job);
457+ if (err)
458+ goto err_put_job;
459+
460+ return job;
461+
462+err_put_job:
463+ pvr_job_put(job);
464+ return ERR_PTR(err);
465+}
466+
467+/**
468+ * pvr_job_data_fini() - Cleanup all allocs used to set up job submission.
469+ * @job_data: Job data array.
470+ * @job_count: Number of members of @job_data.
471+ */
472+static void
473+pvr_job_data_fini(struct pvr_job_data *job_data, u32 job_count)
474+{
475+ for (u32 i = 0; i < job_count; i++) {
476+ pvr_job_put(job_data[i].job);
477+ kvfree(job_data[i].sync_ops);
478+ }
479+}
480+
481+/**
482+ * pvr_job_data_init() - Init an array of created jobs, associating them with
483+ * the appropriate sync_ops args, which will be copied in.
484+ * @pvr_dev: Target PowerVR device.
485+ * @pvr_file: Pointer to PowerVR file structure.
486+ * @job_args: Job args array copied from user.
487+ * @job_count: Number of members of @job_args.
488+ * @job_data_out: Job data array.
489+ */
490+static int pvr_job_data_init(struct pvr_device *pvr_dev,
491+ struct pvr_file *pvr_file,
492+ struct drm_pvr_job *job_args,
493+ u32 *job_count,
494+ struct pvr_job_data *job_data_out)
495+{
496+ int err = 0, i = 0;
497+
498+ for (; i < *job_count; i++) {
499+ job_data_out[i].job =
500+ create_job(pvr_dev, pvr_file, &job_args[i]);
501+ err = PTR_ERR_OR_ZERO(job_data_out[i].job);
502+
503+ if (err) {
504+ *job_count = i;
505+ job_data_out[i].job = NULL;
506+ goto err_cleanup;
507+ }
508+
509+ err = PVR_UOBJ_GET_ARRAY(job_data_out[i].sync_ops,
510+ &job_args[i].sync_ops);
511+ if (err) {
512+ *job_count = i;
513+
514+ /* Ensure the job created above is also cleaned up. */
515+ i++;
516+ goto err_cleanup;
517+ }
518+
519+ job_data_out[i].sync_op_count = job_args[i].sync_ops.count;
520+ }
521+
522+ return 0;
523+
524+err_cleanup:
525+ pvr_job_data_fini(job_data_out, i);
526+
527+ return err;
528+}
529+
530+static void
531+push_jobs(struct pvr_job_data *job_data, u32 job_count)
532+{
533+ for (u32 i = 0; i < job_count; i++)
534+ pvr_queue_job_push(job_data[i].job);
535+}
536+
537+static int
538+prepare_fw_obj_resv(struct drm_exec *exec, struct pvr_fw_object *fw_obj)
539+{
540+ return drm_exec_prepare_obj(exec, gem_from_pvr_gem(fw_obj->gem), 1);
541+}
542+
543+static int
544+jobs_lock_all_objs(struct drm_exec *exec, struct pvr_job_data *job_data,
545+ u32 job_count)
546+{
547+ for (u32 i = 0; i < job_count; i++) {
548+ struct pvr_job *job = job_data[i].job;
549+
550+ /* Grab a lock on a the context, to guard against
551+ * concurrent submission to the same queue.
552+ */
553+ int err = drm_exec_lock_obj(exec,
554+ gem_from_pvr_gem(job->ctx->fw_obj->gem));
555+
556+ if (err)
557+ return err;
558+
559+ if (job->hwrt) {
560+ err = prepare_fw_obj_resv(exec,
561+ job->hwrt->fw_obj);
562+ if (err)
563+ return err;
564+ }
565+ }
566+
567+ return 0;
568+}
569+
570+static int
571+prepare_job_resvs_for_each(struct drm_exec *exec, struct pvr_job_data *job_data,
572+ u32 job_count)
573+{
574+ drm_exec_until_all_locked(exec) {
575+ int err = jobs_lock_all_objs(exec, job_data, job_count);
576+
577+ drm_exec_retry_on_contention(exec);
578+ if (err)
579+ return err;
580+ }
581+
582+ return 0;
583+}
584+
585+static void
586+update_job_resvs(struct pvr_job *job)
587+{
588+ if (job->hwrt) {
589+ enum dma_resv_usage usage = job->type == DRM_PVR_JOB_TYPE_GEOMETRY ?
590+ DMA_RESV_USAGE_WRITE : DMA_RESV_USAGE_READ;
591+ struct drm_gem_object *obj = gem_from_pvr_gem(job->hwrt->fw_obj->gem);
592+
593+ dma_resv_add_fence(obj->resv, &job->base.s_fence->finished, usage);
594+ }
595+}
596+
597+static void
598+update_job_resvs_for_each(struct pvr_job_data *job_data, u32 job_count)
599+{
600+ for (u32 i = 0; i < job_count; i++)
601+ update_job_resvs(job_data[i].job);
602+}
603+
604+static bool can_combine_jobs(struct pvr_job *a, struct pvr_job *b)
605+{
606+ struct pvr_job *geom_job = a, *frag_job = b;
607+ struct dma_fence *fence;
608+ unsigned long index;
609+
610+ /* Geometry and fragment jobs can be combined if they are queued to the
611+ * same context and targeting the same HWRT.
612+ */
613+ if (a->type != DRM_PVR_JOB_TYPE_GEOMETRY ||
614+ b->type != DRM_PVR_JOB_TYPE_FRAGMENT ||
615+ a->ctx != b->ctx ||
616+ a->hwrt != b->hwrt)
617+ return false;
618+
619+ xa_for_each(&frag_job->base.dependencies, index, fence) {
620+ /* We combine when we see an explicit geom -> frag dep. */
621+ if (&geom_job->base.s_fence->scheduled == fence)
622+ return true;
623+ }
624+
625+ return false;
626+}
627+
628+static struct dma_fence *
629+get_last_queued_job_scheduled_fence(struct pvr_queue *queue,
630+ struct pvr_job_data *job_data,
631+ u32 cur_job_pos)
632+{
633+ /* We iterate over the current job array in reverse order to grab the
634+ * last to-be-queued job targeting the same queue.
635+ */
636+ for (u32 i = cur_job_pos; i > 0; i--) {
637+ struct pvr_job *job = job_data[i - 1].job;
638+
639+ if (job->ctx == queue->ctx && job->type == queue->type)
640+ return dma_fence_get(&job->base.s_fence->scheduled);
641+ }
642+
643+ /* If we didn't find any, we just return the last queued job scheduled
644+ * fence attached to the queue.
645+ */
646+ return dma_fence_get(queue->last_queued_job_scheduled_fence);
647+}
648+
649+static int
650+pvr_jobs_link_geom_frag(struct pvr_job_data *job_data, u32 *job_count)
651+{
652+ for (u32 i = 0; i < *job_count - 1; i++) {
653+ struct pvr_job *geom_job = job_data[i].job;
654+ struct pvr_job *frag_job = job_data[i + 1].job;
655+ struct pvr_queue *frag_queue;
656+ struct dma_fence *f;
657+
658+ if (!can_combine_jobs(job_data[i].job, job_data[i + 1].job))
659+ continue;
660+
661+ /* The fragment job will be submitted by the geometry queue. We
662+ * need to make sure it comes after all the other fragment jobs
663+ * queued before it.
664+ */
665+ frag_queue = pvr_context_get_queue_for_job(frag_job->ctx,
666+ frag_job->type);
667+ f = get_last_queued_job_scheduled_fence(frag_queue, job_data,
668+ i);
669+ if (f) {
670+ int err = drm_sched_job_add_dependency(&geom_job->base,
671+ f);
672+ if (err) {
673+ *job_count = i;
674+ return err;
675+ }
676+ }
677+
678+ /* The KCCB slot will be reserved by the geometry job, so we can
679+ * drop the KCCB fence on the fragment job.
680+ */
681+ pvr_kccb_fence_put(frag_job->kccb_fence);
682+ frag_job->kccb_fence = NULL;
683+
684+ geom_job->paired_job = frag_job;
685+ frag_job->paired_job = geom_job;
686+
687+ /* Skip the fragment job we just paired to the geometry job. */
688+ i++;
689+ }
690+
691+ return 0;
692+}
693+
694+/**
695+ * pvr_submit_jobs() - Submit jobs to the GPU
696+ * @pvr_dev: Target PowerVR device.
697+ * @pvr_file: Pointer to PowerVR file structure.
698+ * @args: Ioctl args.
699+ * @job_count: Number of jobs in @jobs_args. On error this will be updated
700+ * with the index into @jobs_args where the error occurred.
701+ *
702+ * This initial implementation is entirely synchronous; on return the GPU will
703+ * be idle. This will not be the case for future implementations.
704+ *
705+ * Returns:
706+ * * 0 on success,
707+ * * -%EFAULT if arguments can not be copied from user space, or
708+ * * -%EINVAL on invalid arguments, or
709+ * * Any other error.
710+ */
711+int
712+pvr_submit_jobs(struct pvr_device *pvr_dev, struct pvr_file *pvr_file,
713+ struct drm_pvr_ioctl_submit_jobs_args *args)
714+{
715+ struct pvr_job_data *job_data = NULL;
716+ struct drm_pvr_job *job_args;
717+ struct xarray signal_array;
718+ u32 jobs_alloced = 0;
719+ struct drm_exec exec;
720+ int err;
721+
722+ if (!args->jobs.count)
723+ return -EINVAL;
724+
725+ err = PVR_UOBJ_GET_ARRAY(job_args, &args->jobs);
726+ if (err)
727+ return err;
728+
729+ job_data = kvmalloc_array(args->jobs.count, sizeof(*job_data),
730+ GFP_KERNEL | __GFP_ZERO);
731+ if (!job_data) {
732+ err = -ENOMEM;
733+ goto out_free;
734+ }
735+
736+ err = pvr_job_data_init(pvr_dev, pvr_file, job_args, &args->jobs.count,
737+ job_data);
738+ if (err)
739+ goto out_free;
740+
741+ jobs_alloced = args->jobs.count;
742+
743+ /*
744+ * Flush MMU if needed - this has been deferred until now to avoid
745+ * overuse of this expensive operation.
746+ */
747+ err = pvr_mmu_flush_exec(pvr_dev, false);
748+ if (err)
749+ goto out_job_data_cleanup;
750+
751+ drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT | DRM_EXEC_IGNORE_DUPLICATES);
752+
753+ xa_init_flags(&signal_array, XA_FLAGS_ALLOC);
754+
755+ err = prepare_job_syncs_for_each(pvr_file, job_data, &args->jobs.count,
756+ &signal_array);
757+ if (err)
758+ goto out_exec_fini;
759+
760+ err = prepare_job_resvs_for_each(&exec, job_data, args->jobs.count);
761+ if (err)
762+ goto out_exec_fini;
763+
764+ err = pvr_jobs_link_geom_frag(job_data, &args->jobs.count);
765+ if (err)
766+ goto out_exec_fini;
767+
768+ /* Anything after that point must succeed because we start exposing job
769+ * finished fences to the outside world.
770+ */
771+ update_job_resvs_for_each(job_data, args->jobs.count);
772+ push_jobs(job_data, args->jobs.count);
773+ pvr_sync_signal_array_push_fences(&signal_array);
774+ err = 0;
775+
776+out_exec_fini:
777+ drm_exec_fini(&exec);
778+ pvr_sync_signal_array_cleanup(&signal_array);
779+
780+out_job_data_cleanup:
781+ pvr_job_data_fini(job_data, jobs_alloced);
782+
783+out_free:
784+ kvfree(job_data);
785+ kvfree(job_args);
786+
787+ return err;
788+}
drivers/gpu/drm/imagination/pvr_job.h+161−0
@@ -0,0 +1,161 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_JOB_H
5+#define PVR_JOB_H
6+
7+#include <uapi/drm/pvr_drm.h>
8+
9+#include <linux/kref.h>
10+#include <linux/types.h>
11+
12+#include <drm/drm_gem.h>
13+#include <drm/gpu_scheduler.h>
14+
15+#include "pvr_power.h"
16+
17+/* Forward declaration from "pvr_context.h". */
18+struct pvr_context;
19+
20+/* Forward declarations from "pvr_device.h". */
21+struct pvr_device;
22+struct pvr_file;
23+
24+/* Forward declarations from "pvr_hwrt.h". */
25+struct pvr_hwrt_data;
26+
27+/* Forward declaration from "pvr_queue.h". */
28+struct pvr_queue;
29+
30+struct pvr_job {
31+ /** @base: drm_sched_job object. */
32+ struct drm_sched_job base;
33+
34+ /** @ref_count: Refcount for job. */
35+ struct kref ref_count;
36+
37+ /** @type: Type of job. */
38+ enum drm_pvr_job_type type;
39+
40+ /** @id: Job ID number. */
41+ u32 id;
42+
43+ /**
44+ * @paired_job: Job paired to this job.
45+ *
46+ * This field is only meaningful for geometry and fragment jobs.
47+ *
48+ * Paired jobs are executed on the same context, and need to be submitted
49+ * atomically to the FW, to make sure the partial render logic has a
50+ * fragment job to execute when the Parameter Manager runs out of memory.
51+ *
52+ * The geometry job should point to the fragment job it's paired with,
53+ * and the fragment job should point to the geometry job it's paired with.
54+ */
55+ struct pvr_job *paired_job;
56+
57+ /** @cccb_fence: Fence used to wait for CCCB space. */
58+ struct dma_fence *cccb_fence;
59+
60+ /** @kccb_fence: Fence used to wait for KCCB space. */
61+ struct dma_fence *kccb_fence;
62+
63+ /** @done_fence: Fence to signal when the job is done. */
64+ struct dma_fence *done_fence;
65+
66+ /** @pvr_dev: Device pointer. */
67+ struct pvr_device *pvr_dev;
68+
69+ /** @ctx: Pointer to owning context. */
70+ struct pvr_context *ctx;
71+
72+ /** @cmd: Command data. Format depends on @type. */
73+ void *cmd;
74+
75+ /** @cmd_len: Length of command data, in bytes. */
76+ u32 cmd_len;
77+
78+ /**
79+ * @fw_ccb_cmd_type: Firmware CCB command type. Must be one of %ROGUE_FWIF_CCB_CMD_TYPE_*.
80+ */
81+ u32 fw_ccb_cmd_type;
82+
83+ /** @hwrt: HWRT object. Will be NULL for compute and transfer jobs. */
84+ struct pvr_hwrt_data *hwrt;
85+
86+ /**
87+ * @has_pm_ref: True if the job has a power ref, thus forcing the GPU to stay on until
88+ * the job is done.
89+ */
90+ bool has_pm_ref;
91+};
92+
93+/**
94+ * pvr_job_get() - Take additional reference on job.
95+ * @job: Job pointer.
96+ *
97+ * Call pvr_job_put() to release.
98+ *
99+ * Returns:
100+ * * The requested job on success, or
101+ * * %NULL if no job pointer passed.
102+ */
103+static __always_inline struct pvr_job *
104+pvr_job_get(struct pvr_job *job)
105+{
106+ if (job)
107+ kref_get(&job->ref_count);
108+
109+ return job;
110+}
111+
112+void pvr_job_put(struct pvr_job *job);
113+
114+/**
115+ * pvr_job_release_pm_ref() - Release the PM ref if the job acquired it.
116+ * @job: The job to release the PM ref on.
117+ */
118+static __always_inline void
119+pvr_job_release_pm_ref(struct pvr_job *job)
120+{
121+ if (job->has_pm_ref) {
122+ pvr_power_put(job->pvr_dev);
123+ job->has_pm_ref = false;
124+ }
125+}
126+
127+/**
128+ * pvr_job_get_pm_ref() - Get a PM ref and attach it to the job.
129+ * @job: The job to attach the PM ref to.
130+ *
131+ * Return:
132+ * * 0 on success, or
133+ * * Any error returned by pvr_power_get() otherwise.
134+ */
135+static __always_inline int
136+pvr_job_get_pm_ref(struct pvr_job *job)
137+{
138+ int err;
139+
140+ if (job->has_pm_ref)
141+ return 0;
142+
143+ err = pvr_power_get(job->pvr_dev);
144+ if (!err)
145+ job->has_pm_ref = true;
146+
147+ return err;
148+}
149+
150+int pvr_job_wait_first_non_signaled_native_dep(struct pvr_job *job);
151+
152+bool pvr_job_non_native_deps_done(struct pvr_job *job);
153+
154+int pvr_job_fits_in_cccb(struct pvr_job *job, unsigned long native_dep_count);
155+
156+void pvr_job_submit(struct pvr_job *job);
157+
158+int pvr_submit_jobs(struct pvr_device *pvr_dev, struct pvr_file *pvr_file,
159+ struct drm_pvr_ioctl_submit_jobs_args *args);
160+
161+#endif /* PVR_JOB_H */
drivers/gpu/drm/imagination/pvr_mmu.c+2637−0
@@ -0,0 +1,2637 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_mmu.h"
5+
6+#include "pvr_ccb.h"
7+#include "pvr_device.h"
8+#include "pvr_fw.h"
9+#include "pvr_gem.h"
10+#include "pvr_power.h"
11+#include "pvr_rogue_fwif.h"
12+#include "pvr_rogue_mmu_defs.h"
13+
14+#include <drm/drm_drv.h>
15+#include <linux/atomic.h>
16+#include <linux/bitops.h>
17+#include <linux/dma-mapping.h>
18+#include <linux/kmemleak.h>
19+#include <linux/minmax.h>
20+#include <linux/sizes.h>
21+
22+#define PVR_SHIFT_FROM_SIZE(size_) (__builtin_ctzll(size_))
23+#define PVR_MASK_FROM_SIZE(size_) (~((size_) - U64_C(1)))
24+
25+/*
26+ * The value of the device page size (%PVR_DEVICE_PAGE_SIZE) is currently
27+ * pegged to the host page size (%PAGE_SIZE). This chunk of macro goodness both
28+ * ensures that the selected host page size corresponds to a valid device page
29+ * size and sets up values needed by the MMU code below.
30+ */
31+#if (PVR_DEVICE_PAGE_SIZE == SZ_4K)
32+# define ROGUE_MMUCTRL_PAGE_SIZE_X ROGUE_MMUCTRL_PAGE_SIZE_4KB
33+# define ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT ROGUE_MMUCTRL_PAGE_4KB_RANGE_SHIFT
34+# define ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK ROGUE_MMUCTRL_PAGE_4KB_RANGE_CLRMSK
35+#elif (PVR_DEVICE_PAGE_SIZE == SZ_16K)
36+# define ROGUE_MMUCTRL_PAGE_SIZE_X ROGUE_MMUCTRL_PAGE_SIZE_16KB
37+# define ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT ROGUE_MMUCTRL_PAGE_16KB_RANGE_SHIFT
38+# define ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK ROGUE_MMUCTRL_PAGE_16KB_RANGE_CLRMSK
39+#elif (PVR_DEVICE_PAGE_SIZE == SZ_64K)
40+# define ROGUE_MMUCTRL_PAGE_SIZE_X ROGUE_MMUCTRL_PAGE_SIZE_64KB
41+# define ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT ROGUE_MMUCTRL_PAGE_64KB_RANGE_SHIFT
42+# define ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK ROGUE_MMUCTRL_PAGE_64KB_RANGE_CLRMSK
43+#elif (PVR_DEVICE_PAGE_SIZE == SZ_256K)
44+# define ROGUE_MMUCTRL_PAGE_SIZE_X ROGUE_MMUCTRL_PAGE_SIZE_256KB
45+# define ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT ROGUE_MMUCTRL_PAGE_256KB_RANGE_SHIFT
46+# define ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK ROGUE_MMUCTRL_PAGE_256KB_RANGE_CLRMSK
47+#elif (PVR_DEVICE_PAGE_SIZE == SZ_1M)
48+# define ROGUE_MMUCTRL_PAGE_SIZE_X ROGUE_MMUCTRL_PAGE_SIZE_1MB
49+# define ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT ROGUE_MMUCTRL_PAGE_1MB_RANGE_SHIFT
50+# define ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK ROGUE_MMUCTRL_PAGE_1MB_RANGE_CLRMSK
51+#elif (PVR_DEVICE_PAGE_SIZE == SZ_2M)
52+# define ROGUE_MMUCTRL_PAGE_SIZE_X ROGUE_MMUCTRL_PAGE_SIZE_2MB
53+# define ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT ROGUE_MMUCTRL_PAGE_2MB_RANGE_SHIFT
54+# define ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK ROGUE_MMUCTRL_PAGE_2MB_RANGE_CLRMSK
55+#else
56+# error Unsupported device page size PVR_DEVICE_PAGE_SIZE
57+#endif
58+
59+#define ROGUE_MMUCTRL_ENTRIES_PT_VALUE_X \
60+ (ROGUE_MMUCTRL_ENTRIES_PT_VALUE >> \
61+ (PVR_DEVICE_PAGE_SHIFT - PVR_SHIFT_FROM_SIZE(SZ_4K)))
62+
63+enum pvr_mmu_sync_level {
64+ PVR_MMU_SYNC_LEVEL_NONE = -1,
65+ PVR_MMU_SYNC_LEVEL_0 = 0,
66+ PVR_MMU_SYNC_LEVEL_1 = 1,
67+ PVR_MMU_SYNC_LEVEL_2 = 2,
68+};
69+
70+#define PVR_MMU_SYNC_LEVEL_0_FLAGS (ROGUE_FWIF_MMUCACHEDATA_FLAGS_PT | \
71+ ROGUE_FWIF_MMUCACHEDATA_FLAGS_INTERRUPT | \
72+ ROGUE_FWIF_MMUCACHEDATA_FLAGS_TLB)
73+#define PVR_MMU_SYNC_LEVEL_1_FLAGS (PVR_MMU_SYNC_LEVEL_0_FLAGS | ROGUE_FWIF_MMUCACHEDATA_FLAGS_PD)
74+#define PVR_MMU_SYNC_LEVEL_2_FLAGS (PVR_MMU_SYNC_LEVEL_1_FLAGS | ROGUE_FWIF_MMUCACHEDATA_FLAGS_PC)
75+
76+/**
77+ * pvr_mmu_set_flush_flags() - Set MMU cache flush flags for next call to
78+ * pvr_mmu_flush_exec().
79+ * @pvr_dev: Target PowerVR device.
80+ * @flags: MMU flush flags. Must be one of %PVR_MMU_SYNC_LEVEL_*_FLAGS.
81+ *
82+ * This function must be called following any possible change to the MMU page
83+ * tables.
84+ */
85+static void pvr_mmu_set_flush_flags(struct pvr_device *pvr_dev, u32 flags)
86+{
87+ atomic_fetch_or(flags, &pvr_dev->mmu_flush_cache_flags);
88+}
89+
90+/**
91+ * pvr_mmu_flush_request_all() - Request flush of all MMU caches when
92+ * subsequently calling pvr_mmu_flush_exec().
93+ * @pvr_dev: Target PowerVR device.
94+ *
95+ * This function must be called following any possible change to the MMU page
96+ * tables.
97+ */
98+void pvr_mmu_flush_request_all(struct pvr_device *pvr_dev)
99+{
100+ pvr_mmu_set_flush_flags(pvr_dev, PVR_MMU_SYNC_LEVEL_2_FLAGS);
101+}
102+
103+/**
104+ * pvr_mmu_flush_exec() - Execute a flush of all MMU caches previously
105+ * requested.
106+ * @pvr_dev: Target PowerVR device.
107+ * @wait: Do not return until the flush is completed.
108+ *
109+ * This function must be called prior to submitting any new GPU job. The flush
110+ * will complete before the jobs are scheduled, so this can be called once after
111+ * a series of maps. However, a single unmap should always be immediately
112+ * followed by a flush and it should be explicitly waited by setting @wait.
113+ *
114+ * As a failure to flush the MMU caches could risk memory corruption, if the
115+ * flush fails (implying the firmware is not responding) then the GPU device is
116+ * marked as lost.
117+ *
118+ * Returns:
119+ * * 0 on success when @wait is true, or
120+ * * -%EIO if the device is unavailable, or
121+ * * Any error encountered while submitting the flush command via the KCCB.
122+ */
123+int pvr_mmu_flush_exec(struct pvr_device *pvr_dev, bool wait)
124+{
125+ struct rogue_fwif_kccb_cmd cmd_mmu_cache = {};
126+ struct rogue_fwif_mmucachedata *cmd_mmu_cache_data =
127+ &cmd_mmu_cache.cmd_data.mmu_cache_data;
128+ int err = 0;
129+ u32 slot;
130+ int idx;
131+
132+ if (!drm_dev_enter(from_pvr_device(pvr_dev), &idx))
133+ return -EIO;
134+
135+ /* Can't flush MMU if the firmware hasn't booted yet. */
136+ if (!pvr_dev->fw_dev.booted)
137+ goto err_drm_dev_exit;
138+
139+ cmd_mmu_cache_data->cache_flags =
140+ atomic_xchg(&pvr_dev->mmu_flush_cache_flags, 0);
141+
142+ if (!cmd_mmu_cache_data->cache_flags)
143+ goto err_drm_dev_exit;
144+
145+ cmd_mmu_cache.cmd_type = ROGUE_FWIF_KCCB_CMD_MMUCACHE;
146+
147+ pvr_fw_object_get_fw_addr(pvr_dev->fw_dev.mem.mmucache_sync_obj,
148+ &cmd_mmu_cache_data->mmu_cache_sync_fw_addr);
149+ cmd_mmu_cache_data->mmu_cache_sync_update_value = 0;
150+
151+ err = pvr_kccb_send_cmd(pvr_dev, &cmd_mmu_cache, &slot);
152+ if (err)
153+ goto err_reset_and_retry;
154+
155+ err = pvr_kccb_wait_for_completion(pvr_dev, slot, HZ, NULL);
156+ if (err)
157+ goto err_reset_and_retry;
158+
159+ drm_dev_exit(idx);
160+
161+ return 0;
162+
163+err_reset_and_retry:
164+ /*
165+ * Flush command failure is most likely the result of a firmware lockup. Hard
166+ * reset the GPU and retry.
167+ */
168+ err = pvr_power_reset(pvr_dev, true);
169+ if (err)
170+ goto err_drm_dev_exit; /* Device is lost. */
171+
172+ /* Retry sending flush request. */
173+ err = pvr_kccb_send_cmd(pvr_dev, &cmd_mmu_cache, &slot);
174+ if (err) {
175+ pvr_device_lost(pvr_dev);
176+ goto err_drm_dev_exit;
177+ }
178+
179+ if (wait) {
180+ err = pvr_kccb_wait_for_completion(pvr_dev, slot, HZ, NULL);
181+ if (err)
182+ pvr_device_lost(pvr_dev);
183+ }
184+
185+err_drm_dev_exit:
186+ drm_dev_exit(idx);
187+
188+ return err;
189+}
190+
191+/**
192+ * DOC: PowerVR Virtual Memory Handling
193+ */
194+/**
195+ * DOC: PowerVR Virtual Memory Handling (constants)
196+ *
197+ * .. c:macro:: PVR_IDX_INVALID
198+ *
199+ * Default value for a u16-based index.
200+ *
201+ * This value cannot be zero, since zero is a valid index value.
202+ */
203+#define PVR_IDX_INVALID ((u16)(-1))
204+
205+/**
206+ * DOC: MMU backing pages
207+ */
208+/**
209+ * DOC: MMU backing pages (constants)
210+ *
211+ * .. c:macro:: PVR_MMU_BACKING_PAGE_SIZE
212+ *
213+ * Page size of a PowerVR device's integrated MMU. The CPU page size must be
214+ * at least as large as this value for the current implementation; this is
215+ * checked at compile-time.
216+ */
217+#define PVR_MMU_BACKING_PAGE_SIZE SZ_4K
218+static_assert(PAGE_SIZE >= PVR_MMU_BACKING_PAGE_SIZE);
219+
220+/**
221+ * struct pvr_mmu_backing_page - Represents a single page used to back a page
222+ * table of any level.
223+ * @dma_addr: DMA address of this page.
224+ * @host_ptr: CPU address of this page.
225+ * @pvr_dev: The PowerVR device to which this page is associated. **For
226+ * internal use only.**
227+ */
228+struct pvr_mmu_backing_page {
229+ dma_addr_t dma_addr;
230+ void *host_ptr;
231+/* private: internal use only */
232+ struct page *raw_page;
233+ struct pvr_device *pvr_dev;
234+};
235+
236+/**
237+ * pvr_mmu_backing_page_init() - Initialize a MMU backing page.
238+ * @page: Target backing page.
239+ * @pvr_dev: Target PowerVR device.
240+ *
241+ * This function performs three distinct operations:
242+ *
243+ * 1. Allocate a single page,
244+ * 2. Map the page to the CPU, and
245+ * 3. Map the page to DMA-space.
246+ *
247+ * It is expected that @page be zeroed (e.g. from kzalloc()) before calling
248+ * this function.
249+ *
250+ * Return:
251+ * * 0 on success, or
252+ * * -%ENOMEM if allocation of the backing page or mapping of the backing
253+ * page to DMA fails.
254+ */
255+static int
256+pvr_mmu_backing_page_init(struct pvr_mmu_backing_page *page,
257+ struct pvr_device *pvr_dev)
258+{
259+ struct device *dev = from_pvr_device(pvr_dev)->dev;
260+
261+ struct page *raw_page;
262+ int err;
263+
264+ dma_addr_t dma_addr;
265+ void *host_ptr;
266+
267+ raw_page = alloc_page(__GFP_ZERO | GFP_KERNEL);
268+ if (!raw_page)
269+ return -ENOMEM;
270+
271+ host_ptr = vmap(&raw_page, 1, VM_MAP, pgprot_writecombine(PAGE_KERNEL));
272+ if (!host_ptr) {
273+ err = -ENOMEM;
274+ goto err_free_page;
275+ }
276+
277+ dma_addr = dma_map_page(dev, raw_page, 0, PVR_MMU_BACKING_PAGE_SIZE,
278+ DMA_TO_DEVICE);
279+ if (dma_mapping_error(dev, dma_addr)) {
280+ err = -ENOMEM;
281+ goto err_unmap_page;
282+ }
283+
284+ page->dma_addr = dma_addr;
285+ page->host_ptr = host_ptr;
286+ page->pvr_dev = pvr_dev;
287+ page->raw_page = raw_page;
288+ kmemleak_alloc(page->host_ptr, PAGE_SIZE, 1, GFP_KERNEL);
289+
290+ return 0;
291+
292+err_unmap_page:
293+ vunmap(host_ptr);
294+
295+err_free_page:
296+ __free_page(raw_page);
297+
298+ return err;
299+}
300+
301+/**
302+ * pvr_mmu_backing_page_fini() - Teardown a MMU backing page.
303+ * @page: Target backing page.
304+ *
305+ * This function performs the mirror operations to pvr_mmu_backing_page_init(),
306+ * in reverse order:
307+ *
308+ * 1. Unmap the page from DMA-space,
309+ * 2. Unmap the page from the CPU, and
310+ * 3. Free the page.
311+ *
312+ * It also zeros @page.
313+ *
314+ * It is a no-op to call this function a second (or further) time on any @page.
315+ */
316+static void
317+pvr_mmu_backing_page_fini(struct pvr_mmu_backing_page *page)
318+{
319+ struct device *dev = from_pvr_device(page->pvr_dev)->dev;
320+
321+ /* Do nothing if no allocation is present. */
322+ if (!page->pvr_dev)
323+ return;
324+
325+ dma_unmap_page(dev, page->dma_addr, PVR_MMU_BACKING_PAGE_SIZE,
326+ DMA_TO_DEVICE);
327+
328+ kmemleak_free(page->host_ptr);
329+ vunmap(page->host_ptr);
330+
331+ __free_page(page->raw_page);
332+
333+ memset(page, 0, sizeof(*page));
334+}
335+
336+/**
337+ * pvr_mmu_backing_page_sync() - Flush a MMU backing page from the CPU to the
338+ * device.
339+ * @page: Target backing page.
340+ *
341+ * .. caution::
342+ *
343+ * **This is potentially an expensive function call.** Only call
344+ * pvr_mmu_backing_page_sync() once you're sure you have no more changes to
345+ * make to the backing page in the immediate future.
346+ */
347+static void
348+pvr_mmu_backing_page_sync(struct pvr_mmu_backing_page *page, u32 flags)
349+{
350+ struct pvr_device *pvr_dev = page->pvr_dev;
351+ struct device *dev;
352+
353+ /*
354+ * Do nothing if no allocation is present. This may be the case if
355+ * we are unmapping pages.
356+ */
357+ if (!pvr_dev)
358+ return;
359+
360+ dev = from_pvr_device(pvr_dev)->dev;
361+
362+ dma_sync_single_for_device(dev, page->dma_addr,
363+ PVR_MMU_BACKING_PAGE_SIZE, DMA_TO_DEVICE);
364+
365+ pvr_mmu_set_flush_flags(pvr_dev, flags);
366+}
367+
368+/**
369+ * DOC: Raw page tables
370+ */
371+
372+#define PVR_PAGE_TABLE_TYPEOF_ENTRY(level_) \
373+ typeof_member(struct pvr_page_table_l##level_##_entry_raw, val)
374+
375+#define PVR_PAGE_TABLE_FIELD_GET(level_, name_, field_, entry_) \
376+ (((entry_).val & \
377+ ~ROGUE_MMUCTRL_##name_##_DATA_##field_##_CLRMSK) >> \
378+ ROGUE_MMUCTRL_##name_##_DATA_##field_##_SHIFT)
379+
380+#define PVR_PAGE_TABLE_FIELD_PREP(level_, name_, field_, val_) \
381+ ((((PVR_PAGE_TABLE_TYPEOF_ENTRY(level_))(val_)) \
382+ << ROGUE_MMUCTRL_##name_##_DATA_##field_##_SHIFT) & \
383+ ~ROGUE_MMUCTRL_##name_##_DATA_##field_##_CLRMSK)
384+
385+/**
386+ * struct pvr_page_table_l2_entry_raw - A single entry in a level 2 page table.
387+ * @val: The raw value of this entry.
388+ *
389+ * This type is a structure for type-checking purposes. At compile-time, its
390+ * size is checked against %ROGUE_MMUCTRL_ENTRY_SIZE_PC_VALUE.
391+ *
392+ * The value stored in this structure can be decoded using the following bitmap:
393+ *
394+ * .. flat-table::
395+ * :widths: 1 5
396+ * :stub-columns: 1
397+ *
398+ * * - 31..4
399+ * - **Level 1 Page Table Base Address:** Bits 39..12 of the L1
400+ * page table base address, which is 4KiB aligned.
401+ *
402+ * * - 3..2
403+ * - *(reserved)*
404+ *
405+ * * - 1
406+ * - **Pending:** When valid bit is not set, indicates that a valid
407+ * entry is pending and the MMU should wait for the driver to map
408+ * the entry. This is used to support page demand mapping of
409+ * memory.
410+ *
411+ * * - 0
412+ * - **Valid:** Indicates that the entry contains a valid L1 page
413+ * table. If the valid bit is not set, then an attempted use of
414+ * the page would result in a page fault.
415+ */
416+struct pvr_page_table_l2_entry_raw {
417+ u32 val;
418+} __packed;
419+static_assert(sizeof(struct pvr_page_table_l2_entry_raw) * 8 ==
420+ ROGUE_MMUCTRL_ENTRY_SIZE_PC_VALUE);
421+
422+static bool
423+pvr_page_table_l2_entry_raw_is_valid(struct pvr_page_table_l2_entry_raw entry)
424+{
425+ return PVR_PAGE_TABLE_FIELD_GET(2, PC, VALID, entry);
426+}
427+
428+/**
429+ * pvr_page_table_l2_entry_raw_set() - Write a valid entry into a raw level 2
430+ * page table.
431+ * @entry: Target raw level 2 page table entry.
432+ * @child_table_dma_addr: DMA address of the level 1 page table to be
433+ * associated with @entry.
434+ *
435+ * When calling this function, @child_table_dma_addr must be a valid DMA
436+ * address and a multiple of %ROGUE_MMUCTRL_PC_DATA_PD_BASE_ALIGNSIZE.
437+ */
438+static void
439+pvr_page_table_l2_entry_raw_set(struct pvr_page_table_l2_entry_raw *entry,
440+ dma_addr_t child_table_dma_addr)
441+{
442+ child_table_dma_addr >>= ROGUE_MMUCTRL_PC_DATA_PD_BASE_ALIGNSHIFT;
443+
444+ WRITE_ONCE(entry->val,
445+ PVR_PAGE_TABLE_FIELD_PREP(2, PC, VALID, true) |
446+ PVR_PAGE_TABLE_FIELD_PREP(2, PC, ENTRY_PENDING, false) |
447+ PVR_PAGE_TABLE_FIELD_PREP(2, PC, PD_BASE, child_table_dma_addr));
448+}
449+
450+static void
451+pvr_page_table_l2_entry_raw_clear(struct pvr_page_table_l2_entry_raw *entry)
452+{
453+ WRITE_ONCE(entry->val, 0);
454+}
455+
456+/**
457+ * struct pvr_page_table_l1_entry_raw - A single entry in a level 1 page table.
458+ * @val: The raw value of this entry.
459+ *
460+ * This type is a structure for type-checking purposes. At compile-time, its
461+ * size is checked against %ROGUE_MMUCTRL_ENTRY_SIZE_PD_VALUE.
462+ *
463+ * The value stored in this structure can be decoded using the following bitmap:
464+ *
465+ * .. flat-table::
466+ * :widths: 1 5
467+ * :stub-columns: 1
468+ *
469+ * * - 63..41
470+ * - *(reserved)*
471+ *
472+ * * - 40
473+ * - **Pending:** When valid bit is not set, indicates that a valid entry
474+ * is pending and the MMU should wait for the driver to map the entry.
475+ * This is used to support page demand mapping of memory.
476+ *
477+ * * - 39..5
478+ * - **Level 0 Page Table Base Address:** The way this value is
479+ * interpreted depends on the page size. Bits not specified in the
480+ * table below (e.g. bits 11..5 for page size 4KiB) should be
481+ * considered reserved.
482+ *
483+ * This table shows the bits used in an L1 page table entry to
484+ * represent the Physical Table Base Address for a given Page Size.
485+ * Since each L1 page table entry covers 2MiB of address space, the
486+ * maximum page size is 2MiB.
487+ *
488+ * .. flat-table::
489+ * :widths: 1 1 1 1
490+ * :header-rows: 1
491+ * :stub-columns: 1
492+ *
493+ * * - Page size
494+ * - L0 page table base address bits
495+ * - Number of L0 page table entries
496+ * - Size of L0 page table
497+ *
498+ * * - 4KiB
499+ * - 39..12
500+ * - 512
501+ * - 4KiB
502+ *
503+ * * - 16KiB
504+ * - 39..10
505+ * - 128
506+ * - 1KiB
507+ *
508+ * * - 64KiB
509+ * - 39..8
510+ * - 32
511+ * - 256B
512+ *
513+ * * - 256KiB
514+ * - 39..6
515+ * - 8
516+ * - 64B
517+ *
518+ * * - 1MiB
519+ * - 39..5 (4 = '0')
520+ * - 2
521+ * - 16B
522+ *
523+ * * - 2MiB
524+ * - 39..5 (4..3 = '00')
525+ * - 1
526+ * - 8B
527+ *
528+ * * - 4
529+ * - *(reserved)*
530+ *
531+ * * - 3..1
532+ * - **Page Size:** Sets the page size, from 4KiB to 2MiB.
533+ *
534+ * * - 0
535+ * - **Valid:** Indicates that the entry contains a valid L0 page table.
536+ * If the valid bit is not set, then an attempted use of the page would
537+ * result in a page fault.
538+ */
539+struct pvr_page_table_l1_entry_raw {
540+ u64 val;
541+} __packed;
542+static_assert(sizeof(struct pvr_page_table_l1_entry_raw) * 8 ==
543+ ROGUE_MMUCTRL_ENTRY_SIZE_PD_VALUE);
544+
545+static bool
546+pvr_page_table_l1_entry_raw_is_valid(struct pvr_page_table_l1_entry_raw entry)
547+{
548+ return PVR_PAGE_TABLE_FIELD_GET(1, PD, VALID, entry);
549+}
550+
551+/**
552+ * pvr_page_table_l1_entry_raw_set() - Write a valid entry into a raw level 1
553+ * page table.
554+ * @entry: Target raw level 1 page table entry.
555+ * @child_table_dma_addr: DMA address of the level 0 page table to be
556+ * associated with @entry.
557+ *
558+ * When calling this function, @child_table_dma_addr must be a valid DMA
559+ * address and a multiple of 4 KiB.
560+ */
561+static void
562+pvr_page_table_l1_entry_raw_set(struct pvr_page_table_l1_entry_raw *entry,
563+ dma_addr_t child_table_dma_addr)
564+{
565+ WRITE_ONCE(entry->val,
566+ PVR_PAGE_TABLE_FIELD_PREP(1, PD, VALID, true) |
567+ PVR_PAGE_TABLE_FIELD_PREP(1, PD, ENTRY_PENDING, false) |
568+ PVR_PAGE_TABLE_FIELD_PREP(1, PD, PAGE_SIZE, ROGUE_MMUCTRL_PAGE_SIZE_X) |
569+ /*
570+ * The use of a 4K-specific macro here is correct. It is
571+ * a future optimization to allocate sub-host-page-sized
572+ * blocks for individual tables, so the condition that any
573+ * page table address is aligned to the size of the
574+ * largest (a 4KB) table currently holds.
575+ */
576+ (child_table_dma_addr & ~ROGUE_MMUCTRL_PT_BASE_4KB_RANGE_CLRMSK));
577+}
578+
579+static void
580+pvr_page_table_l1_entry_raw_clear(struct pvr_page_table_l1_entry_raw *entry)
581+{
582+ WRITE_ONCE(entry->val, 0);
583+}
584+
585+/**
586+ * struct pvr_page_table_l0_entry_raw - A single entry in a level 0 page table.
587+ * @val: The raw value of this entry.
588+ *
589+ * This type is a structure for type-checking purposes. At compile-time, its
590+ * size is checked against %ROGUE_MMUCTRL_ENTRY_SIZE_PT_VALUE.
591+ *
592+ * The value stored in this structure can be decoded using the following bitmap:
593+ *
594+ * .. flat-table::
595+ * :widths: 1 5
596+ * :stub-columns: 1
597+ *
598+ * * - 63
599+ * - *(reserved)*
600+ *
601+ * * - 62
602+ * - **PM/FW Protect:** Indicates a protected region which only the
603+ * Parameter Manager (PM) or firmware processor can write to.
604+ *
605+ * * - 61..40
606+ * - **VP Page (High):** Virtual-physical page used for Parameter Manager
607+ * (PM) memory. This field is only used if the additional level of PB
608+ * virtualization is enabled. The VP Page field is needed by the PM in
609+ * order to correctly reconstitute the free lists after render
610+ * completion. This (High) field holds bits 39..18 of the value; the
611+ * Low field holds bits 17..12. Bits 11..0 are always zero because the
612+ * value is always aligned to the 4KiB page size.
613+ *
614+ * * - 39..12
615+ * - **Physical Page Address:** The way this value is interpreted depends
616+ * on the page size. Bits not specified in the table below (e.g. bits
617+ * 20..12 for page size 2MiB) should be considered reserved.
618+ *
619+ * This table shows the bits used in an L0 page table entry to represent
620+ * the Physical Page Address for a given page size (as defined in the
621+ * associated L1 page table entry).
622+ *
623+ * .. flat-table::
624+ * :widths: 1 1
625+ * :header-rows: 1
626+ * :stub-columns: 1
627+ *
628+ * * - Page size
629+ * - Physical address bits
630+ *
631+ * * - 4KiB
632+ * - 39..12
633+ *
634+ * * - 16KiB
635+ * - 39..14
636+ *
637+ * * - 64KiB
638+ * - 39..16
639+ *
640+ * * - 256KiB
641+ * - 39..18
642+ *
643+ * * - 1MiB
644+ * - 39..20
645+ *
646+ * * - 2MiB
647+ * - 39..21
648+ *
649+ * * - 11..6
650+ * - **VP Page (Low):** Continuation of VP Page (High).
651+ *
652+ * * - 5
653+ * - **Pending:** When valid bit is not set, indicates that a valid entry
654+ * is pending and the MMU should wait for the driver to map the entry.
655+ * This is used to support page demand mapping of memory.
656+ *
657+ * * - 4
658+ * - **PM Src:** Set on Parameter Manager (PM) allocated page table
659+ * entries when indicated by the PM. Note that this bit will only be set
660+ * by the PM, not by the device driver.
661+ *
662+ * * - 3
663+ * - **SLC Bypass Control:** Specifies requests to this page should bypass
664+ * the System Level Cache (SLC), if enabled in SLC configuration.
665+ *
666+ * * - 2
667+ * - **Cache Coherency:** Indicates that the page is coherent (i.e. it
668+ * does not require a cache flush between operations on the CPU and the
669+ * device).
670+ *
671+ * * - 1
672+ * - **Read Only:** If set, this bit indicates that the page is read only.
673+ * An attempted write to this page would result in a write-protection
674+ * fault.
675+ *
676+ * * - 0
677+ * - **Valid:** Indicates that the entry contains a valid page. If the
678+ * valid bit is not set, then an attempted use of the page would result
679+ * in a page fault.
680+ */
681+struct pvr_page_table_l0_entry_raw {
682+ u64 val;
683+} __packed;
684+static_assert(sizeof(struct pvr_page_table_l0_entry_raw) * 8 ==
685+ ROGUE_MMUCTRL_ENTRY_SIZE_PT_VALUE);
686+
687+/**
688+ * struct pvr_page_flags_raw - The configurable flags from a single entry in a
689+ * level 0 page table.
690+ * @val: The raw value of these flags. Since these are a strict subset of
691+ * &struct pvr_page_table_l0_entry_raw; use that type for our member here.
692+ *
693+ * The flags stored in this type are: PM/FW Protect; SLC Bypass Control; Cache
694+ * Coherency, and Read Only (bits 62, 3, 2 and 1 respectively).
695+ *
696+ * This type should never be instantiated directly; instead use
697+ * pvr_page_flags_raw_create() to ensure only valid bits of @val are set.
698+ */
699+struct pvr_page_flags_raw {
700+ struct pvr_page_table_l0_entry_raw val;
701+} __packed;
702+static_assert(sizeof(struct pvr_page_flags_raw) ==
703+ sizeof(struct pvr_page_table_l0_entry_raw));
704+
705+static bool
706+pvr_page_table_l0_entry_raw_is_valid(struct pvr_page_table_l0_entry_raw entry)
707+{
708+ return PVR_PAGE_TABLE_FIELD_GET(0, PT, VALID, entry);
709+}
710+
711+/**
712+ * pvr_page_table_l0_entry_raw_set() - Write a valid entry into a raw level 0
713+ * page table.
714+ * @entry: Target raw level 0 page table entry.
715+ * @dma_addr: DMA address of the physical page to be associated with @entry.
716+ * @flags: Options to be set on @entry.
717+ *
718+ * When calling this function, @child_table_dma_addr must be a valid DMA
719+ * address and a multiple of %PVR_DEVICE_PAGE_SIZE.
720+ *
721+ * The @flags parameter is directly assigned into @entry. It is the callers
722+ * responsibility to ensure that only bits specified in
723+ * &struct pvr_page_flags_raw are set in @flags.
724+ */
725+static void
726+pvr_page_table_l0_entry_raw_set(struct pvr_page_table_l0_entry_raw *entry,
727+ dma_addr_t dma_addr,
728+ struct pvr_page_flags_raw flags)
729+{
730+ WRITE_ONCE(entry->val, PVR_PAGE_TABLE_FIELD_PREP(0, PT, VALID, true) |
731+ PVR_PAGE_TABLE_FIELD_PREP(0, PT, ENTRY_PENDING, false) |
732+ (dma_addr & ~ROGUE_MMUCTRL_PAGE_X_RANGE_CLRMSK) |
733+ flags.val.val);
734+}
735+
736+static void
737+pvr_page_table_l0_entry_raw_clear(struct pvr_page_table_l0_entry_raw *entry)
738+{
739+ WRITE_ONCE(entry->val, 0);
740+}
741+
742+/**
743+ * pvr_page_flags_raw_create() - Initialize the flag bits of a raw level 0 page
744+ * table entry.
745+ * @read_only: This page is read-only (see: Read Only).
746+ * @cache_coherent: This page does not require cache flushes (see: Cache
747+ * Coherency).
748+ * @slc_bypass: This page bypasses the device cache (see: SLC Bypass Control).
749+ * @pm_fw_protect: This page is only for use by the firmware or Parameter
750+ * Manager (see PM/FW Protect).
751+ *
752+ * For more details on the use of these four options, see their respective
753+ * entries in the table under &struct pvr_page_table_l0_entry_raw.
754+ *
755+ * Return:
756+ * A new &struct pvr_page_flags_raw instance which can be passed directly to
757+ * pvr_page_table_l0_entry_raw_set() or pvr_page_table_l0_insert().
758+ */
759+static struct pvr_page_flags_raw
760+pvr_page_flags_raw_create(bool read_only, bool cache_coherent, bool slc_bypass,
761+ bool pm_fw_protect)
762+{
763+ struct pvr_page_flags_raw flags;
764+
765+ flags.val.val =
766+ PVR_PAGE_TABLE_FIELD_PREP(0, PT, READ_ONLY, read_only) |
767+ PVR_PAGE_TABLE_FIELD_PREP(0, PT, CC, cache_coherent) |
768+ PVR_PAGE_TABLE_FIELD_PREP(0, PT, SLC_BYPASS_CTRL, slc_bypass) |
769+ PVR_PAGE_TABLE_FIELD_PREP(0, PT, PM_META_PROTECT, pm_fw_protect);
770+
771+ return flags;
772+}
773+
774+/**
775+ * struct pvr_page_table_l2_raw - The raw data of a level 2 page table.
776+ *
777+ * This type is a structure for type-checking purposes. At compile-time, its
778+ * size is checked against %PVR_MMU_BACKING_PAGE_SIZE.
779+ */
780+struct pvr_page_table_l2_raw {
781+ /** @entries: The raw values of this table. */
782+ struct pvr_page_table_l2_entry_raw
783+ entries[ROGUE_MMUCTRL_ENTRIES_PC_VALUE];
784+} __packed;
785+static_assert(sizeof(struct pvr_page_table_l2_raw) == PVR_MMU_BACKING_PAGE_SIZE);
786+
787+/**
788+ * struct pvr_page_table_l1_raw - The raw data of a level 1 page table.
789+ *
790+ * This type is a structure for type-checking purposes. At compile-time, its
791+ * size is checked against %PVR_MMU_BACKING_PAGE_SIZE.
792+ */
793+struct pvr_page_table_l1_raw {
794+ /** @entries: The raw values of this table. */
795+ struct pvr_page_table_l1_entry_raw
796+ entries[ROGUE_MMUCTRL_ENTRIES_PD_VALUE];
797+} __packed;
798+static_assert(sizeof(struct pvr_page_table_l1_raw) == PVR_MMU_BACKING_PAGE_SIZE);
799+
800+/**
801+ * struct pvr_page_table_l0_raw - The raw data of a level 0 page table.
802+ *
803+ * This type is a structure for type-checking purposes. At compile-time, its
804+ * size is checked against %PVR_MMU_BACKING_PAGE_SIZE.
805+ *
806+ * .. caution::
807+ *
808+ * The size of level 0 page tables is variable depending on the page size
809+ * specified in the associated level 1 page table entry. Since the device
810+ * page size in use is pegged to the host page size, it cannot vary at
811+ * runtime. This structure is therefore only defined to contain the required
812+ * number of entries for the current device page size. **You should never
813+ * read or write beyond the last supported entry.**
814+ */
815+struct pvr_page_table_l0_raw {
816+ /** @entries: The raw values of this table. */
817+ struct pvr_page_table_l0_entry_raw
818+ entries[ROGUE_MMUCTRL_ENTRIES_PT_VALUE_X];
819+} __packed;
820+static_assert(sizeof(struct pvr_page_table_l0_raw) <= PVR_MMU_BACKING_PAGE_SIZE);
821+
822+/**
823+ * DOC: Mirror page tables
824+ */
825+
826+/*
827+ * We pre-declare these types because they cross-depend on pointers to each
828+ * other.
829+ */
830+struct pvr_page_table_l1;
831+struct pvr_page_table_l0;
832+
833+/**
834+ * struct pvr_page_table_l2 - A wrapped level 2 page table.
835+ *
836+ * To access the raw part of this table, use pvr_page_table_l2_get_raw().
837+ * Alternatively to access a raw entry directly, use
838+ * pvr_page_table_l2_get_entry_raw().
839+ *
840+ * A level 2 page table forms the root of the page table tree structure, so
841+ * this type has no &parent or &parent_idx members.
842+ */
843+struct pvr_page_table_l2 {
844+ /**
845+ * @entries: The children of this node in the page table tree
846+ * structure. These are also mirror tables. The indexing of this array
847+ * is identical to that of the raw equivalent
848+ * (&pvr_page_table_l1_raw.entries).
849+ */
850+ struct pvr_page_table_l1 *entries[ROGUE_MMUCTRL_ENTRIES_PC_VALUE];
851+
852+ /**
853+ * @backing_page: A handle to the memory which holds the raw
854+ * equivalent of this table. **For internal use only.**
855+ */
856+ struct pvr_mmu_backing_page backing_page;
857+
858+ /**
859+ * @entry_count: The current number of valid entries (that we know of)
860+ * in this table. This value is essentially a refcount - the table is
861+ * destroyed when this value is decremented to zero by
862+ * pvr_page_table_l2_remove().
863+ */
864+ u16 entry_count;
865+};
866+
867+/**
868+ * pvr_page_table_l2_init() - Initialize a level 2 page table.
869+ * @table: Target level 2 page table.
870+ * @pvr_dev: Target PowerVR device
871+ *
872+ * It is expected that @table be zeroed (e.g. from kzalloc()) before calling
873+ * this function.
874+ *
875+ * Return:
876+ * * 0 on success, or
877+ * * Any error encountered while intializing &table->backing_page using
878+ * pvr_mmu_backing_page_init().
879+ */
880+static int
881+pvr_page_table_l2_init(struct pvr_page_table_l2 *table,
882+ struct pvr_device *pvr_dev)
883+{
884+ return pvr_mmu_backing_page_init(&table->backing_page, pvr_dev);
885+}
886+
887+/**
888+ * pvr_page_table_l2_fini() - Teardown a level 2 page table.
889+ * @table: Target level 2 page table.
890+ *
891+ * It is an error to attempt to use @table after calling this function.
892+ */
893+static void
894+pvr_page_table_l2_fini(struct pvr_page_table_l2 *table)
895+{
896+ pvr_mmu_backing_page_fini(&table->backing_page);
897+}
898+
899+/**
900+ * pvr_page_table_l2_sync() - Flush a level 2 page table from the CPU to the
901+ * device.
902+ * @table: Target level 2 page table.
903+ *
904+ * This is just a thin wrapper around pvr_mmu_backing_page_sync(), so the
905+ * warning there applies here too: **Only call pvr_page_table_l2_sync() once
906+ * you're sure you have no more changes to make to** @table **in the immediate
907+ * future.**
908+ *
909+ * If child level 1 page tables of @table also need to be flushed, this should
910+ * be done first using pvr_page_table_l1_sync() *before* calling this function.
911+ */
912+static void
913+pvr_page_table_l2_sync(struct pvr_page_table_l2 *table)
914+{
915+ pvr_mmu_backing_page_sync(&table->backing_page, PVR_MMU_SYNC_LEVEL_2_FLAGS);
916+}
917+
918+/**
919+ * pvr_page_table_l2_get_raw() - Access the raw equivalent of a mirror level 2
920+ * page table.
921+ * @table: Target level 2 page table.
922+ *
923+ * Essentially returns the CPU address of the raw equivalent of @table, cast to
924+ * a &struct pvr_page_table_l2_raw pointer.
925+ *
926+ * You probably want to call pvr_page_table_l2_get_entry_raw() instead.
927+ *
928+ * Return:
929+ * The raw equivalent of @table.
930+ */
931+static struct pvr_page_table_l2_raw *
932+pvr_page_table_l2_get_raw(struct pvr_page_table_l2 *table)
933+{
934+ return table->backing_page.host_ptr;
935+}
936+
937+/**
938+ * pvr_page_table_l2_get_entry_raw() - Access an entry from the raw equivalent
939+ * of a mirror level 2 page table.
940+ * @table: Target level 2 page table.
941+ * @idx: Index of the entry to access.
942+ *
943+ * Technically this function returns a pointer to a slot in a raw level 2 page
944+ * table, since the returned "entry" is not guaranteed to be valid. The caller
945+ * must verify the validity of the entry at the returned address (perhaps using
946+ * pvr_page_table_l2_entry_raw_is_valid()) before reading or overwriting it.
947+ *
948+ * The value of @idx is not checked here; it is the callers responsibility to
949+ * ensure @idx refers to a valid index within @table before dereferencing the
950+ * returned pointer.
951+ *
952+ * Return:
953+ * A pointer to the requested raw level 2 page table entry.
954+ */
955+static struct pvr_page_table_l2_entry_raw *
956+pvr_page_table_l2_get_entry_raw(struct pvr_page_table_l2 *table, u16 idx)
957+{
958+ return &pvr_page_table_l2_get_raw(table)->entries[idx];
959+}
960+
961+/**
962+ * pvr_page_table_l2_entry_is_valid() - Check if a level 2 page table entry is
963+ * marked as valid.
964+ * @table: Target level 2 page table.
965+ * @idx: Index of the entry to check.
966+ *
967+ * The value of @idx is not checked here; it is the callers responsibility to
968+ * ensure @idx refers to a valid index within @table before calling this
969+ * function.
970+ */
971+static bool
972+pvr_page_table_l2_entry_is_valid(struct pvr_page_table_l2 *table, u16 idx)
973+{
974+ struct pvr_page_table_l2_entry_raw entry_raw =
975+ *pvr_page_table_l2_get_entry_raw(table, idx);
976+
977+ return pvr_page_table_l2_entry_raw_is_valid(entry_raw);
978+}
979+
980+/**
981+ * struct pvr_page_table_l1 - A wrapped level 1 page table.
982+ *
983+ * To access the raw part of this table, use pvr_page_table_l1_get_raw().
984+ * Alternatively to access a raw entry directly, use
985+ * pvr_page_table_l1_get_entry_raw().
986+ */
987+struct pvr_page_table_l1 {
988+ /**
989+ * @entries: The children of this node in the page table tree
990+ * structure. These are also mirror tables. The indexing of this array
991+ * is identical to that of the raw equivalent
992+ * (&pvr_page_table_l0_raw.entries).
993+ */
994+ struct pvr_page_table_l0 *entries[ROGUE_MMUCTRL_ENTRIES_PD_VALUE];
995+
996+ /**
997+ * @backing_page: A handle to the memory which holds the raw
998+ * equivalent of this table. **For internal use only.**
999+ */
1000+ struct pvr_mmu_backing_page backing_page;
1001+
1002+ union {
1003+ /**
1004+ * @parent: The parent of this node in the page table tree structure.
1005+ *
1006+ * This is also a mirror table.
1007+ *
1008+ * Only valid when the L1 page table is active. When the L1 page table
1009+ * has been removed and queued for destruction, the next_free field
1010+ * should be used instead.
1011+ */
1012+ struct pvr_page_table_l2 *parent;
1013+
1014+ /**
1015+ * @next_free: Pointer to the next L1 page table to take/free.
1016+ *
1017+ * Used to form a linked list of L1 page tables. This is used
1018+ * when preallocating tables and when the page table has been
1019+ * removed and queued for destruction.
1020+ */
1021+ struct pvr_page_table_l1 *next_free;
1022+ };
1023+
1024+ /**
1025+ * @parent_idx: The index of the entry in the parent table (see
1026+ * @parent) which corresponds to this table.
1027+ */
1028+ u16 parent_idx;
1029+
1030+ /**
1031+ * @entry_count: The current number of valid entries (that we know of)
1032+ * in this table. This value is essentially a refcount - the table is
1033+ * destroyed when this value is decremented to zero by
1034+ * pvr_page_table_l1_remove().
1035+ */
1036+ u16 entry_count;
1037+};
1038+
1039+/**
1040+ * pvr_page_table_l1_init() - Initialize a level 1 page table.
1041+ * @table: Target level 1 page table.
1042+ * @pvr_dev: Target PowerVR device
1043+ *
1044+ * When this function returns successfully, @table is still not considered
1045+ * valid. It must be inserted into the page table tree structure with
1046+ * pvr_page_table_l2_insert() before it is ready for use.
1047+ *
1048+ * It is expected that @table be zeroed (e.g. from kzalloc()) before calling
1049+ * this function.
1050+ *
1051+ * Return:
1052+ * * 0 on success, or
1053+ * * Any error encountered while intializing &table->backing_page using
1054+ * pvr_mmu_backing_page_init().
1055+ */
1056+static int
1057+pvr_page_table_l1_init(struct pvr_page_table_l1 *table,
1058+ struct pvr_device *pvr_dev)
1059+{
1060+ table->parent_idx = PVR_IDX_INVALID;
1061+
1062+ return pvr_mmu_backing_page_init(&table->backing_page, pvr_dev);
1063+}
1064+
1065+/**
1066+ * pvr_page_table_l1_free() - Teardown a level 1 page table.
1067+ * @table: Target level 1 page table.
1068+ *
1069+ * It is an error to attempt to use @table after calling this function, even
1070+ * indirectly. This includes calling pvr_page_table_l2_remove(), which must
1071+ * be called *before* pvr_page_table_l1_free().
1072+ */
1073+static void
1074+pvr_page_table_l1_free(struct pvr_page_table_l1 *table)
1075+{
1076+ pvr_mmu_backing_page_fini(&table->backing_page);
1077+ kfree(table);
1078+}
1079+
1080+/**
1081+ * pvr_page_table_l1_sync() - Flush a level 1 page table from the CPU to the
1082+ * device.
1083+ * @table: Target level 1 page table.
1084+ *
1085+ * This is just a thin wrapper around pvr_mmu_backing_page_sync(), so the
1086+ * warning there applies here too: **Only call pvr_page_table_l1_sync() once
1087+ * you're sure you have no more changes to make to** @table **in the immediate
1088+ * future.**
1089+ *
1090+ * If child level 0 page tables of @table also need to be flushed, this should
1091+ * be done first using pvr_page_table_l0_sync() *before* calling this function.
1092+ */
1093+static void
1094+pvr_page_table_l1_sync(struct pvr_page_table_l1 *table)
1095+{
1096+ pvr_mmu_backing_page_sync(&table->backing_page, PVR_MMU_SYNC_LEVEL_1_FLAGS);
1097+}
1098+
1099+/**
1100+ * pvr_page_table_l1_get_raw() - Access the raw equivalent of a mirror level 1
1101+ * page table.
1102+ * @table: Target level 1 page table.
1103+ *
1104+ * Essentially returns the CPU address of the raw equivalent of @table, cast to
1105+ * a &struct pvr_page_table_l1_raw pointer.
1106+ *
1107+ * You probably want to call pvr_page_table_l1_get_entry_raw() instead.
1108+ *
1109+ * Return:
1110+ * The raw equivalent of @table.
1111+ */
1112+static struct pvr_page_table_l1_raw *
1113+pvr_page_table_l1_get_raw(struct pvr_page_table_l1 *table)
1114+{
1115+ return table->backing_page.host_ptr;
1116+}
1117+
1118+/**
1119+ * pvr_page_table_l1_get_entry_raw() - Access an entry from the raw equivalent
1120+ * of a mirror level 1 page table.
1121+ * @table: Target level 1 page table.
1122+ * @idx: Index of the entry to access.
1123+ *
1124+ * Technically this function returns a pointer to a slot in a raw level 1 page
1125+ * table, since the returned "entry" is not guaranteed to be valid. The caller
1126+ * must verify the validity of the entry at the returned address (perhaps using
1127+ * pvr_page_table_l1_entry_raw_is_valid()) before reading or overwriting it.
1128+ *
1129+ * The value of @idx is not checked here; it is the callers responsibility to
1130+ * ensure @idx refers to a valid index within @table before dereferencing the
1131+ * returned pointer.
1132+ *
1133+ * Return:
1134+ * A pointer to the requested raw level 1 page table entry.
1135+ */
1136+static struct pvr_page_table_l1_entry_raw *
1137+pvr_page_table_l1_get_entry_raw(struct pvr_page_table_l1 *table, u16 idx)
1138+{
1139+ return &pvr_page_table_l1_get_raw(table)->entries[idx];
1140+}
1141+
1142+/**
1143+ * pvr_page_table_l1_entry_is_valid() - Check if a level 1 page table entry is
1144+ * marked as valid.
1145+ * @table: Target level 1 page table.
1146+ * @idx: Index of the entry to check.
1147+ *
1148+ * The value of @idx is not checked here; it is the callers responsibility to
1149+ * ensure @idx refers to a valid index within @table before calling this
1150+ * function.
1151+ */
1152+static bool
1153+pvr_page_table_l1_entry_is_valid(struct pvr_page_table_l1 *table, u16 idx)
1154+{
1155+ struct pvr_page_table_l1_entry_raw entry_raw =
1156+ *pvr_page_table_l1_get_entry_raw(table, idx);
1157+
1158+ return pvr_page_table_l1_entry_raw_is_valid(entry_raw);
1159+}
1160+
1161+/**
1162+ * struct pvr_page_table_l0 - A wrapped level 0 page table.
1163+ *
1164+ * To access the raw part of this table, use pvr_page_table_l0_get_raw().
1165+ * Alternatively to access a raw entry directly, use
1166+ * pvr_page_table_l0_get_entry_raw().
1167+ *
1168+ * There is no mirror representation of an individual page, so this type has no
1169+ * &entries member.
1170+ */
1171+struct pvr_page_table_l0 {
1172+ /**
1173+ * @backing_page: A handle to the memory which holds the raw
1174+ * equivalent of this table. **For internal use only.**
1175+ */
1176+ struct pvr_mmu_backing_page backing_page;
1177+
1178+ union {
1179+ /**
1180+ * @parent: The parent of this node in the page table tree structure.
1181+ *
1182+ * This is also a mirror table.
1183+ *
1184+ * Only valid when the L0 page table is active. When the L0 page table
1185+ * has been removed and queued for destruction, the next_free field
1186+ * should be used instead.
1187+ */
1188+ struct pvr_page_table_l1 *parent;
1189+
1190+ /**
1191+ * @next_free: Pointer to the next L0 page table to take/free.
1192+ *
1193+ * Used to form a linked list of L0 page tables. This is used
1194+ * when preallocating tables and when the page table has been
1195+ * removed and queued for destruction.
1196+ */
1197+ struct pvr_page_table_l0 *next_free;
1198+ };
1199+
1200+ /**
1201+ * @parent_idx: The index of the entry in the parent table (see
1202+ * @parent) which corresponds to this table.
1203+ */
1204+ u16 parent_idx;
1205+
1206+ /**
1207+ * @entry_count: The current number of valid entries (that we know of)
1208+ * in this table. This value is essentially a refcount - the table is
1209+ * destroyed when this value is decremented to zero by
1210+ * pvr_page_table_l0_remove().
1211+ */
1212+ u16 entry_count;
1213+};
1214+
1215+/**
1216+ * pvr_page_table_l0_init() - Initialize a level 0 page table.
1217+ * @table: Target level 0 page table.
1218+ * @pvr_dev: Target PowerVR device
1219+ *
1220+ * When this function returns successfully, @table is still not considered
1221+ * valid. It must be inserted into the page table tree structure with
1222+ * pvr_page_table_l1_insert() before it is ready for use.
1223+ *
1224+ * It is expected that @table be zeroed (e.g. from kzalloc()) before calling
1225+ * this function.
1226+ *
1227+ * Return:
1228+ * * 0 on success, or
1229+ * * Any error encountered while intializing &table->backing_page using
1230+ * pvr_mmu_backing_page_init().
1231+ */
1232+static int
1233+pvr_page_table_l0_init(struct pvr_page_table_l0 *table,
1234+ struct pvr_device *pvr_dev)
1235+{
1236+ table->parent_idx = PVR_IDX_INVALID;
1237+
1238+ return pvr_mmu_backing_page_init(&table->backing_page, pvr_dev);
1239+}
1240+
1241+/**
1242+ * pvr_page_table_l0_free() - Teardown a level 0 page table.
1243+ * @table: Target level 0 page table.
1244+ *
1245+ * It is an error to attempt to use @table after calling this function, even
1246+ * indirectly. This includes calling pvr_page_table_l1_remove(), which must
1247+ * be called *before* pvr_page_table_l0_free().
1248+ */
1249+static void
1250+pvr_page_table_l0_free(struct pvr_page_table_l0 *table)
1251+{
1252+ pvr_mmu_backing_page_fini(&table->backing_page);
1253+ kfree(table);
1254+}
1255+
1256+/**
1257+ * pvr_page_table_l0_sync() - Flush a level 0 page table from the CPU to the
1258+ * device.
1259+ * @table: Target level 0 page table.
1260+ *
1261+ * This is just a thin wrapper around pvr_mmu_backing_page_sync(), so the
1262+ * warning there applies here too: **Only call pvr_page_table_l0_sync() once
1263+ * you're sure you have no more changes to make to** @table **in the immediate
1264+ * future.**
1265+ *
1266+ * If child pages of @table also need to be flushed, this should be done first
1267+ * using a DMA sync function (e.g. dma_sync_sg_for_device()) *before* calling
1268+ * this function.
1269+ */
1270+static void
1271+pvr_page_table_l0_sync(struct pvr_page_table_l0 *table)
1272+{
1273+ pvr_mmu_backing_page_sync(&table->backing_page, PVR_MMU_SYNC_LEVEL_0_FLAGS);
1274+}
1275+
1276+/**
1277+ * pvr_page_table_l0_get_raw() - Access the raw equivalent of a mirror level 0
1278+ * page table.
1279+ * @table: Target level 0 page table.
1280+ *
1281+ * Essentially returns the CPU address of the raw equivalent of @table, cast to
1282+ * a &struct pvr_page_table_l0_raw pointer.
1283+ *
1284+ * You probably want to call pvr_page_table_l0_get_entry_raw() instead.
1285+ *
1286+ * Return:
1287+ * The raw equivalent of @table.
1288+ */
1289+static struct pvr_page_table_l0_raw *
1290+pvr_page_table_l0_get_raw(struct pvr_page_table_l0 *table)
1291+{
1292+ return table->backing_page.host_ptr;
1293+}
1294+
1295+/**
1296+ * pvr_page_table_l0_get_entry_raw() - Access an entry from the raw equivalent
1297+ * of a mirror level 0 page table.
1298+ * @table: Target level 0 page table.
1299+ * @idx: Index of the entry to access.
1300+ *
1301+ * Technically this function returns a pointer to a slot in a raw level 0 page
1302+ * table, since the returned "entry" is not guaranteed to be valid. The caller
1303+ * must verify the validity of the entry at the returned address (perhaps using
1304+ * pvr_page_table_l0_entry_raw_is_valid()) before reading or overwriting it.
1305+ *
1306+ * The value of @idx is not checked here; it is the callers responsibility to
1307+ * ensure @idx refers to a valid index within @table before dereferencing the
1308+ * returned pointer. This is espcially important for level 0 page tables, which
1309+ * can have a variable number of entries.
1310+ *
1311+ * Return:
1312+ * A pointer to the requested raw level 0 page table entry.
1313+ */
1314+static struct pvr_page_table_l0_entry_raw *
1315+pvr_page_table_l0_get_entry_raw(struct pvr_page_table_l0 *table, u16 idx)
1316+{
1317+ return &pvr_page_table_l0_get_raw(table)->entries[idx];
1318+}
1319+
1320+/**
1321+ * pvr_page_table_l0_entry_is_valid() - Check if a level 0 page table entry is
1322+ * marked as valid.
1323+ * @table: Target level 0 page table.
1324+ * @idx: Index of the entry to check.
1325+ *
1326+ * The value of @idx is not checked here; it is the callers responsibility to
1327+ * ensure @idx refers to a valid index within @table before calling this
1328+ * function.
1329+ */
1330+static bool
1331+pvr_page_table_l0_entry_is_valid(struct pvr_page_table_l0 *table, u16 idx)
1332+{
1333+ struct pvr_page_table_l0_entry_raw entry_raw =
1334+ *pvr_page_table_l0_get_entry_raw(table, idx);
1335+
1336+ return pvr_page_table_l0_entry_raw_is_valid(entry_raw);
1337+}
1338+
1339+/**
1340+ * struct pvr_mmu_context - context holding data for operations at page
1341+ * catalogue level, intended for use with a VM context.
1342+ */
1343+struct pvr_mmu_context {
1344+ /** @pvr_dev: The PVR device associated with the owning VM context. */
1345+ struct pvr_device *pvr_dev;
1346+
1347+ /** @page_table_l2: The MMU table root. */
1348+ struct pvr_page_table_l2 page_table_l2;
1349+};
1350+
1351+/**
1352+ * struct pvr_page_table_ptr - A reference to a single physical page as indexed
1353+ * by the page table structure.
1354+ *
1355+ * Intended for embedding in a &struct pvr_mmu_op_context.
1356+ */
1357+struct pvr_page_table_ptr {
1358+ /**
1359+ * @l1_table: A cached handle to the level 1 page table the
1360+ * context is currently traversing.
1361+ */
1362+ struct pvr_page_table_l1 *l1_table;
1363+
1364+ /**
1365+ * @l0_table: A cached handle to the level 0 page table the
1366+ * context is currently traversing.
1367+ */
1368+ struct pvr_page_table_l0 *l0_table;
1369+
1370+ /**
1371+ * @l2_idx: Index into the level 2 page table the context is
1372+ * currently referencing.
1373+ */
1374+ u16 l2_idx;
1375+
1376+ /**
1377+ * @l1_idx: Index into the level 1 page table the context is
1378+ * currently referencing.
1379+ */
1380+ u16 l1_idx;
1381+
1382+ /**
1383+ * @l0_idx: Index into the level 0 page table the context is
1384+ * currently referencing.
1385+ */
1386+ u16 l0_idx;
1387+};
1388+
1389+/**
1390+ * struct pvr_mmu_op_context - context holding data for individual
1391+ * device-virtual mapping operations. Intended for use with a VM bind operation.
1392+ */
1393+struct pvr_mmu_op_context {
1394+ /** @mmu_ctx: The MMU context associated with the owning VM context. */
1395+ struct pvr_mmu_context *mmu_ctx;
1396+
1397+ /** @map: Data specifically for map operations. */
1398+ struct {
1399+ /**
1400+ * @sgt: Scatter gather table containing pages pinned for use by
1401+ * this context - these are currently pinned when initialising
1402+ * the VM bind operation.
1403+ */
1404+ struct sg_table *sgt;
1405+
1406+ /** @sgt_offset: Start address of the device-virtual mapping. */
1407+ u64 sgt_offset;
1408+
1409+ /**
1410+ * @l1_prealloc_tables: Preallocated l1 page table objects
1411+ * use by this context when creating a page mapping. Linked list
1412+ * fully created during initialisation.
1413+ */
1414+ struct pvr_page_table_l1 *l1_prealloc_tables;
1415+
1416+ /**
1417+ * @l0_prealloc_tables: Preallocated l0 page table objects
1418+ * use by this context when creating a page mapping. Linked list
1419+ * fully created during initialisation.
1420+ */
1421+ struct pvr_page_table_l0 *l0_prealloc_tables;
1422+ } map;
1423+
1424+ /** @unmap: Data specifically for unmap operations. */
1425+ struct {
1426+ /**
1427+ * @l1_free_tables: Collects page table objects freed by unmap
1428+ * ops. Linked list empty at creation.
1429+ */
1430+ struct pvr_page_table_l1 *l1_free_tables;
1431+
1432+ /**
1433+ * @l0_free_tables: Collects page table objects freed by unmap
1434+ * ops. Linked list empty at creation.
1435+ */
1436+ struct pvr_page_table_l0 *l0_free_tables;
1437+ } unmap;
1438+
1439+ /**
1440+ * @curr_page: A reference to a single physical page as indexed by the
1441+ * page table structure.
1442+ */
1443+ struct pvr_page_table_ptr curr_page;
1444+
1445+ /**
1446+ * @sync_level_required: The maximum level of the page table tree
1447+ * structure which has (possibly) been modified since it was last
1448+ * flushed to the device.
1449+ *
1450+ * This field should only be set with pvr_mmu_op_context_require_sync()
1451+ * or indirectly by pvr_mmu_op_context_sync_partial().
1452+ */
1453+ enum pvr_mmu_sync_level sync_level_required;
1454+};
1455+
1456+/**
1457+ * pvr_page_table_l2_insert() - Insert an entry referring to a level 1 page
1458+ * table into a level 2 page table.
1459+ * @op_ctx: Target MMU op context pointing at the entry to insert the L1 page
1460+ * table into.
1461+ * @child_table: Target level 1 page table to be referenced by the new entry.
1462+ *
1463+ * It is the caller's responsibility to ensure @op_ctx.curr_page points to a
1464+ * valid L2 entry.
1465+ *
1466+ * It is the caller's responsibility to execute any memory barries to ensure
1467+ * that the creation of @child_table is ordered before the L2 entry is inserted.
1468+ */
1469+static void
1470+pvr_page_table_l2_insert(struct pvr_mmu_op_context *op_ctx,
1471+ struct pvr_page_table_l1 *child_table)
1472+{
1473+ struct pvr_page_table_l2 *l2_table =
1474+ &op_ctx->mmu_ctx->page_table_l2;
1475+ struct pvr_page_table_l2_entry_raw *entry_raw =
1476+ pvr_page_table_l2_get_entry_raw(l2_table,
1477+ op_ctx->curr_page.l2_idx);
1478+
1479+ pvr_page_table_l2_entry_raw_set(entry_raw,
1480+ child_table->backing_page.dma_addr);
1481+
1482+ child_table->parent = l2_table;
1483+ child_table->parent_idx = op_ctx->curr_page.l2_idx;
1484+ l2_table->entries[op_ctx->curr_page.l2_idx] = child_table;
1485+ ++l2_table->entry_count;
1486+ op_ctx->curr_page.l1_table = child_table;
1487+}
1488+
1489+/**
1490+ * pvr_page_table_l2_remove() - Remove a level 1 page table from a level 2 page
1491+ * table.
1492+ * @op_ctx: Target MMU op context pointing at the L2 entry to remove.
1493+ *
1494+ * It is the caller's responsibility to ensure @op_ctx.curr_page points to a
1495+ * valid L2 entry.
1496+ */
1497+static void
1498+pvr_page_table_l2_remove(struct pvr_mmu_op_context *op_ctx)
1499+{
1500+ struct pvr_page_table_l2 *l2_table =
1501+ &op_ctx->mmu_ctx->page_table_l2;
1502+ struct pvr_page_table_l2_entry_raw *entry_raw =
1503+ pvr_page_table_l2_get_entry_raw(l2_table,
1504+ op_ctx->curr_page.l1_table->parent_idx);
1505+
1506+ WARN_ON(op_ctx->curr_page.l1_table->parent != l2_table);
1507+
1508+ pvr_page_table_l2_entry_raw_clear(entry_raw);
1509+
1510+ l2_table->entries[op_ctx->curr_page.l1_table->parent_idx] = NULL;
1511+ op_ctx->curr_page.l1_table->parent_idx = PVR_IDX_INVALID;
1512+ op_ctx->curr_page.l1_table->next_free = op_ctx->unmap.l1_free_tables;
1513+ op_ctx->unmap.l1_free_tables = op_ctx->curr_page.l1_table;
1514+ op_ctx->curr_page.l1_table = NULL;
1515+
1516+ --l2_table->entry_count;
1517+}
1518+
1519+/**
1520+ * pvr_page_table_l1_insert() - Insert an entry referring to a level 0 page
1521+ * table into a level 1 page table.
1522+ * @op_ctx: Target MMU op context pointing at the entry to insert the L0 page
1523+ * table into.
1524+ * @child_table: L0 page table to insert.
1525+ *
1526+ * It is the caller's responsibility to ensure @op_ctx.curr_page points to a
1527+ * valid L1 entry.
1528+ *
1529+ * It is the caller's responsibility to execute any memory barries to ensure
1530+ * that the creation of @child_table is ordered before the L1 entry is inserted.
1531+ */
1532+static void
1533+pvr_page_table_l1_insert(struct pvr_mmu_op_context *op_ctx,
1534+ struct pvr_page_table_l0 *child_table)
1535+{
1536+ struct pvr_page_table_l1_entry_raw *entry_raw =
1537+ pvr_page_table_l1_get_entry_raw(op_ctx->curr_page.l1_table,
1538+ op_ctx->curr_page.l1_idx);
1539+
1540+ pvr_page_table_l1_entry_raw_set(entry_raw,
1541+ child_table->backing_page.dma_addr);
1542+
1543+ child_table->parent = op_ctx->curr_page.l1_table;
1544+ child_table->parent_idx = op_ctx->curr_page.l1_idx;
1545+ op_ctx->curr_page.l1_table->entries[op_ctx->curr_page.l1_idx] = child_table;
1546+ ++op_ctx->curr_page.l1_table->entry_count;
1547+ op_ctx->curr_page.l0_table = child_table;
1548+}
1549+
1550+/**
1551+ * pvr_page_table_l1_remove() - Remove a level 0 page table from a level 1 page
1552+ * table.
1553+ * @op_ctx: Target MMU op context pointing at the L1 entry to remove.
1554+ *
1555+ * If this function results in the L1 table becoming empty, it will be removed
1556+ * from its parent level 2 page table and destroyed.
1557+ *
1558+ * It is the caller's responsibility to ensure @op_ctx.curr_page points to a
1559+ * valid L1 entry.
1560+ */
1561+static void
1562+pvr_page_table_l1_remove(struct pvr_mmu_op_context *op_ctx)
1563+{
1564+ struct pvr_page_table_l1_entry_raw *entry_raw =
1565+ pvr_page_table_l1_get_entry_raw(op_ctx->curr_page.l0_table->parent,
1566+ op_ctx->curr_page.l0_table->parent_idx);
1567+
1568+ WARN_ON(op_ctx->curr_page.l0_table->parent !=
1569+ op_ctx->curr_page.l1_table);
1570+
1571+ pvr_page_table_l1_entry_raw_clear(entry_raw);
1572+
1573+ op_ctx->curr_page.l1_table->entries[op_ctx->curr_page.l0_table->parent_idx] = NULL;
1574+ op_ctx->curr_page.l0_table->parent_idx = PVR_IDX_INVALID;
1575+ op_ctx->curr_page.l0_table->next_free = op_ctx->unmap.l0_free_tables;
1576+ op_ctx->unmap.l0_free_tables = op_ctx->curr_page.l0_table;
1577+ op_ctx->curr_page.l0_table = NULL;
1578+
1579+ if (--op_ctx->curr_page.l1_table->entry_count == 0) {
1580+ /* Clear the parent L2 page table entry. */
1581+ if (op_ctx->curr_page.l1_table->parent_idx != PVR_IDX_INVALID)
1582+ pvr_page_table_l2_remove(op_ctx);
1583+ }
1584+}
1585+
1586+/**
1587+ * pvr_page_table_l0_insert() - Insert an entry referring to a physical page
1588+ * into a level 0 page table.
1589+ * @op_ctx: Target MMU op context pointing at the L0 entry to insert.
1590+ * @dma_addr: Target DMA address to be referenced by the new entry.
1591+ * @flags: Page options to be stored in the new entry.
1592+ *
1593+ * It is the caller's responsibility to ensure @op_ctx.curr_page points to a
1594+ * valid L0 entry.
1595+ */
1596+static void
1597+pvr_page_table_l0_insert(struct pvr_mmu_op_context *op_ctx,
1598+ dma_addr_t dma_addr, struct pvr_page_flags_raw flags)
1599+{
1600+ struct pvr_page_table_l0_entry_raw *entry_raw =
1601+ pvr_page_table_l0_get_entry_raw(op_ctx->curr_page.l0_table,
1602+ op_ctx->curr_page.l0_idx);
1603+
1604+ pvr_page_table_l0_entry_raw_set(entry_raw, dma_addr, flags);
1605+
1606+ /*
1607+ * There is no entry to set here - we don't keep a mirror of
1608+ * individual pages.
1609+ */
1610+
1611+ ++op_ctx->curr_page.l0_table->entry_count;
1612+}
1613+
1614+/**
1615+ * pvr_page_table_l0_remove() - Remove a physical page from a level 0 page
1616+ * table.
1617+ * @op_ctx: Target MMU op context pointing at the L0 entry to remove.
1618+ *
1619+ * If this function results in the L0 table becoming empty, it will be removed
1620+ * from its parent L1 page table and destroyed.
1621+ *
1622+ * It is the caller's responsibility to ensure @op_ctx.curr_page points to a
1623+ * valid L0 entry.
1624+ */
1625+static void
1626+pvr_page_table_l0_remove(struct pvr_mmu_op_context *op_ctx)
1627+{
1628+ struct pvr_page_table_l0_entry_raw *entry_raw =
1629+ pvr_page_table_l0_get_entry_raw(op_ctx->curr_page.l0_table,
1630+ op_ctx->curr_page.l0_idx);
1631+
1632+ pvr_page_table_l0_entry_raw_clear(entry_raw);
1633+
1634+ /*
1635+ * There is no entry to clear here - we don't keep a mirror of
1636+ * individual pages.
1637+ */
1638+
1639+ if (--op_ctx->curr_page.l0_table->entry_count == 0) {
1640+ /* Clear the parent L1 page table entry. */
1641+ if (op_ctx->curr_page.l0_table->parent_idx != PVR_IDX_INVALID)
1642+ pvr_page_table_l1_remove(op_ctx);
1643+ }
1644+}
1645+
1646+/**
1647+ * DOC: Page table index utilities
1648+ */
1649+
1650+/**
1651+ * pvr_page_table_l2_idx() - Calculate the level 2 page table index for a
1652+ * device-virtual address.
1653+ * @device_addr: Target device-virtual address.
1654+ *
1655+ * This function does not perform any bounds checking - it is the caller's
1656+ * responsibility to ensure that @device_addr is valid before interpreting
1657+ * the result.
1658+ *
1659+ * Return:
1660+ * The index into a level 2 page table corresponding to @device_addr.
1661+ */
1662+static u16
1663+pvr_page_table_l2_idx(u64 device_addr)
1664+{
1665+ return (device_addr & ~ROGUE_MMUCTRL_VADDR_PC_INDEX_CLRMSK) >>
1666+ ROGUE_MMUCTRL_VADDR_PC_INDEX_SHIFT;
1667+}
1668+
1669+/**
1670+ * pvr_page_table_l1_idx() - Calculate the level 1 page table index for a
1671+ * device-virtual address.
1672+ * @device_addr: Target device-virtual address.
1673+ *
1674+ * This function does not perform any bounds checking - it is the caller's
1675+ * responsibility to ensure that @device_addr is valid before interpreting
1676+ * the result.
1677+ *
1678+ * Return:
1679+ * The index into a level 1 page table corresponding to @device_addr.
1680+ */
1681+static u16
1682+pvr_page_table_l1_idx(u64 device_addr)
1683+{
1684+ return (device_addr & ~ROGUE_MMUCTRL_VADDR_PD_INDEX_CLRMSK) >>
1685+ ROGUE_MMUCTRL_VADDR_PD_INDEX_SHIFT;
1686+}
1687+
1688+/**
1689+ * pvr_page_table_l0_idx() - Calculate the level 0 page table index for a
1690+ * device-virtual address.
1691+ * @device_addr: Target device-virtual address.
1692+ *
1693+ * This function does not perform any bounds checking - it is the caller's
1694+ * responsibility to ensure that @device_addr is valid before interpreting
1695+ * the result.
1696+ *
1697+ * Return:
1698+ * The index into a level 0 page table corresponding to @device_addr.
1699+ */
1700+static u16
1701+pvr_page_table_l0_idx(u64 device_addr)
1702+{
1703+ return (device_addr & ~ROGUE_MMUCTRL_VADDR_PT_INDEX_CLRMSK) >>
1704+ ROGUE_MMUCTRL_PAGE_X_RANGE_SHIFT;
1705+}
1706+
1707+/**
1708+ * DOC: High-level page table operations
1709+ */
1710+
1711+/**
1712+ * pvr_page_table_l1_get_or_insert() - Retrieves (optionally inserting if
1713+ * necessary) a level 1 page table from the specified level 2 page table entry.
1714+ * @op_ctx: Target MMU op context.
1715+ * @should_insert: [IN] Specifies whether new page tables should be inserted
1716+ * when empty page table entries are encountered during traversal.
1717+ *
1718+ * Return:
1719+ * * 0 on success, or
1720+ *
1721+ * If @should_insert is %false:
1722+ * * -%ENXIO if a level 1 page table would have been inserted.
1723+ *
1724+ * If @should_insert is %true:
1725+ * * Any error encountered while inserting the level 1 page table.
1726+ */
1727+static int
1728+pvr_page_table_l1_get_or_insert(struct pvr_mmu_op_context *op_ctx,
1729+ bool should_insert)
1730+{
1731+ struct pvr_page_table_l2 *l2_table =
1732+ &op_ctx->mmu_ctx->page_table_l2;
1733+ struct pvr_page_table_l1 *table;
1734+
1735+ if (pvr_page_table_l2_entry_is_valid(l2_table,
1736+ op_ctx->curr_page.l2_idx)) {
1737+ op_ctx->curr_page.l1_table =
1738+ l2_table->entries[op_ctx->curr_page.l2_idx];
1739+ return 0;
1740+ }
1741+
1742+ if (!should_insert)
1743+ return -ENXIO;
1744+
1745+ /* Take a prealloced table. */
1746+ table = op_ctx->map.l1_prealloc_tables;
1747+ if (!table)
1748+ return -ENOMEM;
1749+
1750+ /* Pop */
1751+ op_ctx->map.l1_prealloc_tables = table->next_free;
1752+ table->next_free = NULL;
1753+
1754+ /* Ensure new table is fully written out before adding to L2 page table. */
1755+ wmb();
1756+
1757+ pvr_page_table_l2_insert(op_ctx, table);
1758+
1759+ return 0;
1760+}
1761+
1762+/**
1763+ * pvr_page_table_l0_get_or_insert() - Retrieves (optionally inserting if
1764+ * necessary) a level 0 page table from the specified level 1 page table entry.
1765+ * @op_ctx: Target MMU op context.
1766+ * @should_insert: [IN] Specifies whether new page tables should be inserted
1767+ * when empty page table entries are encountered during traversal.
1768+ *
1769+ * Return:
1770+ * * 0 on success,
1771+ *
1772+ * If @should_insert is %false:
1773+ * * -%ENXIO if a level 0 page table would have been inserted.
1774+ *
1775+ * If @should_insert is %true:
1776+ * * Any error encountered while inserting the level 0 page table.
1777+ */
1778+static int
1779+pvr_page_table_l0_get_or_insert(struct pvr_mmu_op_context *op_ctx,
1780+ bool should_insert)
1781+{
1782+ struct pvr_page_table_l0 *table;
1783+
1784+ if (pvr_page_table_l1_entry_is_valid(op_ctx->curr_page.l1_table,
1785+ op_ctx->curr_page.l1_idx)) {
1786+ op_ctx->curr_page.l0_table =
1787+ op_ctx->curr_page.l1_table->entries[op_ctx->curr_page.l1_idx];
1788+ return 0;
1789+ }
1790+
1791+ if (!should_insert)
1792+ return -ENXIO;
1793+
1794+ /* Take a prealloced table. */
1795+ table = op_ctx->map.l0_prealloc_tables;
1796+ if (!table)
1797+ return -ENOMEM;
1798+
1799+ /* Pop */
1800+ op_ctx->map.l0_prealloc_tables = table->next_free;
1801+ table->next_free = NULL;
1802+
1803+ /* Ensure new table is fully written out before adding to L1 page table. */
1804+ wmb();
1805+
1806+ pvr_page_table_l1_insert(op_ctx, table);
1807+
1808+ return 0;
1809+}
1810+
1811+/**
1812+ * pvr_mmu_context_create() - Create an MMU context.
1813+ * @pvr_dev: PVR device associated with owning VM context.
1814+ *
1815+ * Returns:
1816+ * * Newly created MMU context object on success, or
1817+ * * -%ENOMEM if no memory is available,
1818+ * * Any error code returned by pvr_page_table_l2_init().
1819+ */
1820+struct pvr_mmu_context *pvr_mmu_context_create(struct pvr_device *pvr_dev)
1821+{
1822+ struct pvr_mmu_context *ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
1823+ int err;
1824+
1825+ if (!ctx)
1826+ return ERR_PTR(-ENOMEM);
1827+
1828+ err = pvr_page_table_l2_init(&ctx->page_table_l2, pvr_dev);
1829+ if (err)
1830+ return ERR_PTR(err);
1831+
1832+ ctx->pvr_dev = pvr_dev;
1833+
1834+ return ctx;
1835+}
1836+
1837+/**
1838+ * pvr_mmu_context_destroy() - Destroy an MMU context.
1839+ * @ctx: Target MMU context.
1840+ */
1841+void pvr_mmu_context_destroy(struct pvr_mmu_context *ctx)
1842+{
1843+ pvr_page_table_l2_fini(&ctx->page_table_l2);
1844+ kfree(ctx);
1845+}
1846+
1847+/**
1848+ * pvr_mmu_get_root_table_dma_addr() - Get the DMA address of the root of the
1849+ * page table structure behind a VM context.
1850+ * @ctx: Target MMU context.
1851+ */
1852+dma_addr_t pvr_mmu_get_root_table_dma_addr(struct pvr_mmu_context *ctx)
1853+{
1854+ return ctx->page_table_l2.backing_page.dma_addr;
1855+}
1856+
1857+/**
1858+ * pvr_page_table_l1_alloc() - Allocate a l1 page_table object.
1859+ * @ctx: MMU context of owning VM context.
1860+ *
1861+ * Returns:
1862+ * * Newly created page table object on success, or
1863+ * * -%ENOMEM if no memory is available,
1864+ * * Any error code returned by pvr_page_table_l1_init().
1865+ */
1866+static struct pvr_page_table_l1 *
1867+pvr_page_table_l1_alloc(struct pvr_mmu_context *ctx)
1868+{
1869+ int err;
1870+
1871+ struct pvr_page_table_l1 *table =
1872+ kzalloc(sizeof(*table), GFP_KERNEL);
1873+
1874+ if (!table)
1875+ return ERR_PTR(-ENOMEM);
1876+
1877+ err = pvr_page_table_l1_init(table, ctx->pvr_dev);
1878+ if (err) {
1879+ kfree(table);
1880+ return ERR_PTR(err);
1881+ }
1882+
1883+ return table;
1884+}
1885+
1886+/**
1887+ * pvr_page_table_l0_alloc() - Allocate a l0 page_table object.
1888+ * @ctx: MMU context of owning VM context.
1889+ *
1890+ * Returns:
1891+ * * Newly created page table object on success, or
1892+ * * -%ENOMEM if no memory is available,
1893+ * * Any error code returned by pvr_page_table_l0_init().
1894+ */
1895+static struct pvr_page_table_l0 *
1896+pvr_page_table_l0_alloc(struct pvr_mmu_context *ctx)
1897+{
1898+ int err;
1899+
1900+ struct pvr_page_table_l0 *table =
1901+ kzalloc(sizeof(*table), GFP_KERNEL);
1902+
1903+ if (!table)
1904+ return ERR_PTR(-ENOMEM);
1905+
1906+ err = pvr_page_table_l0_init(table, ctx->pvr_dev);
1907+ if (err) {
1908+ kfree(table);
1909+ return ERR_PTR(err);
1910+ }
1911+
1912+ return table;
1913+}
1914+
1915+/**
1916+ * pvr_mmu_op_context_require_sync() - Mark an MMU op context as requiring a
1917+ * sync operation for the referenced page tables up to a specified level.
1918+ * @op_ctx: Target MMU op context.
1919+ * @level: Maximum page table level for which a sync is required.
1920+ */
1921+static void
1922+pvr_mmu_op_context_require_sync(struct pvr_mmu_op_context *op_ctx,
1923+ enum pvr_mmu_sync_level level)
1924+{
1925+ if (op_ctx->sync_level_required < level)
1926+ op_ctx->sync_level_required = level;
1927+}
1928+
1929+/**
1930+ * pvr_mmu_op_context_sync_manual() - Trigger a sync of some or all of the
1931+ * page tables referenced by a MMU op context.
1932+ * @op_ctx: Target MMU op context.
1933+ * @level: Maximum page table level to sync.
1934+ *
1935+ * Do not call this function directly. Instead use
1936+ * pvr_mmu_op_context_sync_partial() which is checked against the current
1937+ * value of &op_ctx->sync_level_required as set by
1938+ * pvr_mmu_op_context_require_sync().
1939+ */
1940+static void
1941+pvr_mmu_op_context_sync_manual(struct pvr_mmu_op_context *op_ctx,
1942+ enum pvr_mmu_sync_level level)
1943+{
1944+ /*
1945+ * We sync the page table levels in ascending order (starting from the
1946+ * leaf node) to ensure consistency.
1947+ */
1948+
1949+ WARN_ON(level < PVR_MMU_SYNC_LEVEL_NONE);
1950+
1951+ if (level <= PVR_MMU_SYNC_LEVEL_NONE)
1952+ return;
1953+
1954+ if (op_ctx->curr_page.l0_table)
1955+ pvr_page_table_l0_sync(op_ctx->curr_page.l0_table);
1956+
1957+ if (level < PVR_MMU_SYNC_LEVEL_1)
1958+ return;
1959+
1960+ if (op_ctx->curr_page.l1_table)
1961+ pvr_page_table_l1_sync(op_ctx->curr_page.l1_table);
1962+
1963+ if (level < PVR_MMU_SYNC_LEVEL_2)
1964+ return;
1965+
1966+ pvr_page_table_l2_sync(&op_ctx->mmu_ctx->page_table_l2);
1967+}
1968+
1969+/**
1970+ * pvr_mmu_op_context_sync_partial() - Trigger a sync of some or all of the
1971+ * page tables referenced by a MMU op context.
1972+ * @op_ctx: Target MMU op context.
1973+ * @level: Requested page table level to sync up to (inclusive).
1974+ *
1975+ * If @level is greater than the maximum level recorded by @op_ctx as requiring
1976+ * a sync operation, only the previously recorded maximum will be used.
1977+ *
1978+ * Additionally, if @level is greater than or equal to the maximum level
1979+ * recorded by @op_ctx as requiring a sync operation, that maximum level will be
1980+ * reset as a full sync will be performed. This is equivalent to calling
1981+ * pvr_mmu_op_context_sync().
1982+ */
1983+static void
1984+pvr_mmu_op_context_sync_partial(struct pvr_mmu_op_context *op_ctx,
1985+ enum pvr_mmu_sync_level level)
1986+{
1987+ /*
1988+ * If the requested sync level is greater than or equal to the
1989+ * currently required sync level, we do two things:
1990+ * * Don't waste time syncing levels we haven't previously marked as
1991+ * requiring a sync, and
1992+ * * Reset the required sync level since we are about to sync
1993+ * everything that was previously marked as requiring a sync.
1994+ */
1995+ if (level >= op_ctx->sync_level_required) {
1996+ level = op_ctx->sync_level_required;
1997+ op_ctx->sync_level_required = PVR_MMU_SYNC_LEVEL_NONE;
1998+ }
1999+
2000+ pvr_mmu_op_context_sync_manual(op_ctx, level);
2001+}
2002+
2003+/**
2004+ * pvr_mmu_op_context_sync() - Trigger a sync of every page table referenced by
2005+ * a MMU op context.
2006+ * @op_ctx: Target MMU op context.
2007+ *
2008+ * The maximum level marked internally as requiring a sync will be reset so
2009+ * that subsequent calls to this function will be no-ops unless @op_ctx is
2010+ * otherwise updated.
2011+ */
2012+static void
2013+pvr_mmu_op_context_sync(struct pvr_mmu_op_context *op_ctx)
2014+{
2015+ pvr_mmu_op_context_sync_manual(op_ctx, op_ctx->sync_level_required);
2016+
2017+ op_ctx->sync_level_required = PVR_MMU_SYNC_LEVEL_NONE;
2018+}
2019+
2020+/**
2021+ * pvr_mmu_op_context_load_tables() - Load pointers to tables in each level of
2022+ * the page table tree structure needed to reference the physical page
2023+ * referenced by a MMU op context.
2024+ * @op_ctx: Target MMU op context.
2025+ * @should_create: Specifies whether new page tables should be created when
2026+ * empty page table entries are encountered during traversal.
2027+ * @load_level_required: Maximum page table level to load.
2028+ *
2029+ * If @should_create is %true, this function may modify the stored required
2030+ * sync level of @op_ctx as new page tables are created and inserted into their
2031+ * respective parents.
2032+ *
2033+ * Since there is only one root page table, it is technically incorrect to call
2034+ * this function with a value of @load_level_required greater than or equal to
2035+ * the root level number. However, this is not explicitly disallowed here.
2036+ *
2037+ * Return:
2038+ * * 0 on success,
2039+ * * Any error returned by pvr_page_table_l1_get_or_create() if
2040+ * @load_level_required >= 1 except -%ENXIO, or
2041+ * * Any error returned by pvr_page_table_l0_get_or_create() if
2042+ * @load_level_required >= 0 except -%ENXIO.
2043+ */
2044+static int
2045+pvr_mmu_op_context_load_tables(struct pvr_mmu_op_context *op_ctx,
2046+ bool should_create,
2047+ enum pvr_mmu_sync_level load_level_required)
2048+{
2049+ const struct pvr_page_table_l1 *l1_head_before =
2050+ op_ctx->map.l1_prealloc_tables;
2051+ const struct pvr_page_table_l0 *l0_head_before =
2052+ op_ctx->map.l0_prealloc_tables;
2053+ int err;
2054+
2055+ /* Clear tables we're about to fetch in case of error states. */
2056+ if (load_level_required >= PVR_MMU_SYNC_LEVEL_1)
2057+ op_ctx->curr_page.l1_table = NULL;
2058+
2059+ if (load_level_required >= PVR_MMU_SYNC_LEVEL_0)
2060+ op_ctx->curr_page.l0_table = NULL;
2061+
2062+ /* Get or create L1 page table. */
2063+ if (load_level_required >= PVR_MMU_SYNC_LEVEL_1) {
2064+ err = pvr_page_table_l1_get_or_insert(op_ctx, should_create);
2065+ if (err) {
2066+ /*
2067+ * If @should_create is %false and no L1 page table was
2068+ * found, return early but without an error. Since
2069+ * pvr_page_table_l1_get_or_create() can only return
2070+ * -%ENXIO if @should_create is %false, there is no
2071+ * need to check it here.
2072+ */
2073+ if (err == -ENXIO)
2074+ err = 0;
2075+
2076+ return err;
2077+ }
2078+ }
2079+
2080+ /* Get or create L0 page table. */
2081+ if (load_level_required >= PVR_MMU_SYNC_LEVEL_0) {
2082+ err = pvr_page_table_l0_get_or_insert(op_ctx, should_create);
2083+ if (err) {
2084+ /*
2085+ * If @should_create is %false and no L0 page table was
2086+ * found, return early but without an error. Since
2087+ * pvr_page_table_l0_get_or_insert() can only return
2088+ * -%ENXIO if @should_create is %false, there is no
2089+ * need to check it here.
2090+ */
2091+ if (err == -ENXIO)
2092+ err = 0;
2093+
2094+ /*
2095+ * At this point, an L1 page table could have been
2096+ * inserted but is now empty due to the failed attempt
2097+ * at inserting an L0 page table. In this instance, we
2098+ * must remove the empty L1 page table ourselves as
2099+ * pvr_page_table_l1_remove() is never called as part
2100+ * of the error path in
2101+ * pvr_page_table_l0_get_or_insert().
2102+ */
2103+ if (l1_head_before != op_ctx->map.l1_prealloc_tables) {
2104+ pvr_page_table_l2_remove(op_ctx);
2105+ pvr_mmu_op_context_require_sync(op_ctx, PVR_MMU_SYNC_LEVEL_2);
2106+ }
2107+
2108+ return err;
2109+ }
2110+ }
2111+
2112+ /*
2113+ * A sync is only needed if table objects were inserted. This can be
2114+ * inferred by checking if the pointer at the head of the linked list
2115+ * has changed.
2116+ */
2117+ if (l1_head_before != op_ctx->map.l1_prealloc_tables)
2118+ pvr_mmu_op_context_require_sync(op_ctx, PVR_MMU_SYNC_LEVEL_2);
2119+ else if (l0_head_before != op_ctx->map.l0_prealloc_tables)
2120+ pvr_mmu_op_context_require_sync(op_ctx, PVR_MMU_SYNC_LEVEL_1);
2121+
2122+ return 0;
2123+}
2124+
2125+/**
2126+ * pvr_mmu_op_context_set_curr_page() - Reassign the current page of an MMU op
2127+ * context, syncing any page tables previously assigned to it which are no
2128+ * longer relevant.
2129+ * @op_ctx: Target MMU op context.
2130+ * @device_addr: New pointer target.
2131+ * @should_create: Specify whether new page tables should be created when
2132+ * empty page table entries are encountered during traversal.
2133+ *
2134+ * This function performs a full sync on the pointer, regardless of which
2135+ * levels are modified.
2136+ *
2137+ * Return:
2138+ * * 0 on success, or
2139+ * * Any error returned by pvr_mmu_op_context_load_tables().
2140+ */
2141+static int
2142+pvr_mmu_op_context_set_curr_page(struct pvr_mmu_op_context *op_ctx,
2143+ u64 device_addr, bool should_create)
2144+{
2145+ pvr_mmu_op_context_sync(op_ctx);
2146+
2147+ op_ctx->curr_page.l2_idx = pvr_page_table_l2_idx(device_addr);
2148+ op_ctx->curr_page.l1_idx = pvr_page_table_l1_idx(device_addr);
2149+ op_ctx->curr_page.l0_idx = pvr_page_table_l0_idx(device_addr);
2150+ op_ctx->curr_page.l1_table = NULL;
2151+ op_ctx->curr_page.l0_table = NULL;
2152+
2153+ return pvr_mmu_op_context_load_tables(op_ctx, should_create,
2154+ PVR_MMU_SYNC_LEVEL_1);
2155+}
2156+
2157+/**
2158+ * pvr_mmu_op_context_next_page() - Advance the current page of an MMU op
2159+ * context.
2160+ * @op_ctx: Target MMU op context.
2161+ * @should_create: Specify whether new page tables should be created when
2162+ * empty page table entries are encountered during traversal.
2163+ *
2164+ * If @should_create is %false, it is the caller's responsibility to verify that
2165+ * the state of the table references in @op_ctx is valid on return. If -%ENXIO
2166+ * is returned, at least one of the table references is invalid. It should be
2167+ * noted that @op_ctx as a whole will be left in a valid state if -%ENXIO is
2168+ * returned, unlike other error codes. The caller should check which references
2169+ * are invalid by comparing them to %NULL. Only &@ptr->l2_table is guaranteed
2170+ * to be valid, since it represents the root of the page table tree structure.
2171+ *
2172+ * Return:
2173+ * * 0 on success,
2174+ * * -%EPERM if the operation would wrap at the top of the page table
2175+ * hierarchy,
2176+ * * -%ENXIO if @should_create is %false and a page table of any level would
2177+ * have otherwise been created, or
2178+ * * Any error returned while attempting to create missing page tables if
2179+ * @should_create is %true.
2180+ */
2181+static int
2182+pvr_mmu_op_context_next_page(struct pvr_mmu_op_context *op_ctx,
2183+ bool should_create)
2184+{
2185+ s8 load_level_required = PVR_MMU_SYNC_LEVEL_NONE;
2186+
2187+ if (++op_ctx->curr_page.l0_idx != ROGUE_MMUCTRL_ENTRIES_PT_VALUE_X)
2188+ goto load_tables;
2189+
2190+ op_ctx->curr_page.l0_idx = 0;
2191+ load_level_required = PVR_MMU_SYNC_LEVEL_0;
2192+
2193+ if (++op_ctx->curr_page.l1_idx != ROGUE_MMUCTRL_ENTRIES_PD_VALUE)
2194+ goto load_tables;
2195+
2196+ op_ctx->curr_page.l1_idx = 0;
2197+ load_level_required = PVR_MMU_SYNC_LEVEL_1;
2198+
2199+ if (++op_ctx->curr_page.l2_idx != ROGUE_MMUCTRL_ENTRIES_PC_VALUE)
2200+ goto load_tables;
2201+
2202+ /*
2203+ * If the pattern continued, we would set &op_ctx->curr_page.l2_idx to
2204+ * zero here. However, that would wrap the top layer of the page table
2205+ * hierarchy which is not a valid operation. Instead, we warn and return
2206+ * an error.
2207+ */
2208+ WARN(true,
2209+ "%s(%p) attempted to loop the top of the page table hierarchy",
2210+ __func__, op_ctx);
2211+ return -EPERM;
2212+
2213+ /* If indices have wrapped, we need to load new tables. */
2214+load_tables:
2215+ /* First, flush tables which will be unloaded. */
2216+ pvr_mmu_op_context_sync_partial(op_ctx, load_level_required);
2217+
2218+ /* Then load tables from the required level down. */
2219+ return pvr_mmu_op_context_load_tables(op_ctx, should_create,
2220+ load_level_required);
2221+}
2222+
2223+/**
2224+ * DOC: Single page operations
2225+ */
2226+
2227+/**
2228+ * pvr_page_create() - Create a device-virtual memory page and insert it into
2229+ * a level 0 page table.
2230+ * @op_ctx: Target MMU op context pointing at the device-virtual address of the
2231+ * target page.
2232+ * @dma_addr: DMA address of the physical page backing the created page.
2233+ * @flags: Page options saved on the level 0 page table entry for reading by
2234+ * the device.
2235+ *
2236+ * Return:
2237+ * * 0 on success, or
2238+ * * -%EEXIST if the requested page already exists.
2239+ */
2240+static int
2241+pvr_page_create(struct pvr_mmu_op_context *op_ctx, dma_addr_t dma_addr,
2242+ struct pvr_page_flags_raw flags)
2243+{
2244+ /* Do not create a new page if one already exists. */
2245+ if (pvr_page_table_l0_entry_is_valid(op_ctx->curr_page.l0_table,
2246+ op_ctx->curr_page.l0_idx)) {
2247+ return -EEXIST;
2248+ }
2249+
2250+ pvr_page_table_l0_insert(op_ctx, dma_addr, flags);
2251+
2252+ pvr_mmu_op_context_require_sync(op_ctx, PVR_MMU_SYNC_LEVEL_0);
2253+
2254+ return 0;
2255+}
2256+
2257+/**
2258+ * pvr_page_destroy() - Destroy a device page after removing it from its
2259+ * parent level 0 page table.
2260+ * @op_ctx: Target MMU op context.
2261+ */
2262+static void
2263+pvr_page_destroy(struct pvr_mmu_op_context *op_ctx)
2264+{
2265+ /* Do nothing if the page does not exist. */
2266+ if (!pvr_page_table_l0_entry_is_valid(op_ctx->curr_page.l0_table,
2267+ op_ctx->curr_page.l0_idx)) {
2268+ return;
2269+ }
2270+
2271+ /* Clear the parent L0 page table entry. */
2272+ pvr_page_table_l0_remove(op_ctx);
2273+
2274+ pvr_mmu_op_context_require_sync(op_ctx, PVR_MMU_SYNC_LEVEL_0);
2275+}
2276+
2277+/**
2278+ * pvr_mmu_op_context_destroy() - Destroy an MMU op context.
2279+ * @op_ctx: Target MMU op context.
2280+ */
2281+void pvr_mmu_op_context_destroy(struct pvr_mmu_op_context *op_ctx)
2282+{
2283+ const bool flush_caches =
2284+ op_ctx->sync_level_required != PVR_MMU_SYNC_LEVEL_NONE;
2285+
2286+ pvr_mmu_op_context_sync(op_ctx);
2287+
2288+ /* Unmaps should be flushed immediately. Map flushes can be deferred. */
2289+ if (flush_caches && !op_ctx->map.sgt)
2290+ pvr_mmu_flush_exec(op_ctx->mmu_ctx->pvr_dev, true);
2291+
2292+ while (op_ctx->map.l0_prealloc_tables) {
2293+ struct pvr_page_table_l0 *tmp = op_ctx->map.l0_prealloc_tables;
2294+
2295+ op_ctx->map.l0_prealloc_tables =
2296+ op_ctx->map.l0_prealloc_tables->next_free;
2297+ pvr_page_table_l0_free(tmp);
2298+ }
2299+
2300+ while (op_ctx->map.l1_prealloc_tables) {
2301+ struct pvr_page_table_l1 *tmp = op_ctx->map.l1_prealloc_tables;
2302+
2303+ op_ctx->map.l1_prealloc_tables =
2304+ op_ctx->map.l1_prealloc_tables->next_free;
2305+ pvr_page_table_l1_free(tmp);
2306+ }
2307+
2308+ while (op_ctx->unmap.l0_free_tables) {
2309+ struct pvr_page_table_l0 *tmp = op_ctx->unmap.l0_free_tables;
2310+
2311+ op_ctx->unmap.l0_free_tables =
2312+ op_ctx->unmap.l0_free_tables->next_free;
2313+ pvr_page_table_l0_free(tmp);
2314+ }
2315+
2316+ while (op_ctx->unmap.l1_free_tables) {
2317+ struct pvr_page_table_l1 *tmp = op_ctx->unmap.l1_free_tables;
2318+
2319+ op_ctx->unmap.l1_free_tables =
2320+ op_ctx->unmap.l1_free_tables->next_free;
2321+ pvr_page_table_l1_free(tmp);
2322+ }
2323+
2324+ kfree(op_ctx);
2325+}
2326+
2327+/**
2328+ * pvr_mmu_op_context_create() - Create an MMU op context.
2329+ * @ctx: MMU context associated with owning VM context.
2330+ * @sgt: Scatter gather table containing pages pinned for use by this context.
2331+ * @sgt_offset: Start offset of the requested device-virtual memory mapping.
2332+ * @size: Size in bytes of the requested device-virtual memory mapping. For an
2333+ * unmapping, this should be zero so that no page tables are allocated.
2334+ *
2335+ * Returns:
2336+ * * Newly created MMU op context object on success, or
2337+ * * -%ENOMEM if no memory is available,
2338+ * * Any error code returned by pvr_page_table_l2_init().
2339+ */
2340+struct pvr_mmu_op_context *
2341+pvr_mmu_op_context_create(struct pvr_mmu_context *ctx, struct sg_table *sgt,
2342+ u64 sgt_offset, u64 size)
2343+{
2344+ int err;
2345+
2346+ struct pvr_mmu_op_context *op_ctx =
2347+ kzalloc(sizeof(*op_ctx), GFP_KERNEL);
2348+
2349+ if (!op_ctx)
2350+ return ERR_PTR(-ENOMEM);
2351+
2352+ op_ctx->mmu_ctx = ctx;
2353+ op_ctx->map.sgt = sgt;
2354+ op_ctx->map.sgt_offset = sgt_offset;
2355+ op_ctx->sync_level_required = PVR_MMU_SYNC_LEVEL_NONE;
2356+
2357+ if (size) {
2358+ /*
2359+ * The number of page table objects we need to prealloc is
2360+ * indicated by the mapping size, start offset and the sizes
2361+ * of the areas mapped per PT or PD. The range calculation is
2362+ * identical to that for the index into a table for a device
2363+ * address, so we reuse those functions here.
2364+ */
2365+ const u32 l1_start_idx = pvr_page_table_l2_idx(sgt_offset);
2366+ const u32 l1_end_idx = pvr_page_table_l2_idx(sgt_offset + size);
2367+ const u32 l1_count = l1_end_idx - l1_start_idx + 1;
2368+ const u32 l0_start_idx = pvr_page_table_l1_idx(sgt_offset);
2369+ const u32 l0_end_idx = pvr_page_table_l1_idx(sgt_offset + size);
2370+ const u32 l0_count = l0_end_idx - l0_start_idx + 1;
2371+
2372+ /*
2373+ * Alloc and push page table entries until we have enough of
2374+ * each type, ending with linked lists of l0 and l1 entries in
2375+ * reverse order.
2376+ */
2377+ for (int i = 0; i < l1_count; i++) {
2378+ struct pvr_page_table_l1 *l1_tmp =
2379+ pvr_page_table_l1_alloc(ctx);
2380+
2381+ err = PTR_ERR_OR_ZERO(l1_tmp);
2382+ if (err)
2383+ goto err_cleanup;
2384+
2385+ l1_tmp->next_free = op_ctx->map.l1_prealloc_tables;
2386+ op_ctx->map.l1_prealloc_tables = l1_tmp;
2387+ }
2388+
2389+ for (int i = 0; i < l0_count; i++) {
2390+ struct pvr_page_table_l0 *l0_tmp =
2391+ pvr_page_table_l0_alloc(ctx);
2392+
2393+ err = PTR_ERR_OR_ZERO(l0_tmp);
2394+ if (err)
2395+ goto err_cleanup;
2396+
2397+ l0_tmp->next_free = op_ctx->map.l0_prealloc_tables;
2398+ op_ctx->map.l0_prealloc_tables = l0_tmp;
2399+ }
2400+ }
2401+
2402+ return op_ctx;
2403+
2404+err_cleanup:
2405+ pvr_mmu_op_context_destroy(op_ctx);
2406+
2407+ return ERR_PTR(err);
2408+}
2409+
2410+/**
2411+ * pvr_mmu_op_context_unmap_curr_page() - Unmap pages from a memory context
2412+ * starting from the current page of an MMU op context.
2413+ * @op_ctx: Target MMU op context pointing at the first page to unmap.
2414+ * @nr_pages: Number of pages to unmap.
2415+ *
2416+ * Return:
2417+ * * 0 on success, or
2418+ * * Any error encountered while advancing @op_ctx.curr_page with
2419+ * pvr_mmu_op_context_next_page() (except -%ENXIO).
2420+ */
2421+static int
2422+pvr_mmu_op_context_unmap_curr_page(struct pvr_mmu_op_context *op_ctx,
2423+ u64 nr_pages)
2424+{
2425+ int err;
2426+
2427+ if (nr_pages == 0)
2428+ return 0;
2429+
2430+ /*
2431+ * Destroy first page outside loop, as it doesn't require a page
2432+ * advance beforehand. If the L0 page table reference in
2433+ * @op_ctx.curr_page is %NULL, there cannot be a mapped page at
2434+ * @op_ctx.curr_page (so skip ahead).
2435+ */
2436+ if (op_ctx->curr_page.l0_table)
2437+ pvr_page_destroy(op_ctx);
2438+
2439+ for (u64 page = 1; page < nr_pages; ++page) {
2440+ err = pvr_mmu_op_context_next_page(op_ctx, false);
2441+ /*
2442+ * If the page table tree structure at @op_ctx.curr_page is
2443+ * incomplete, skip ahead. We don't care about unmapping pages
2444+ * that cannot exist.
2445+ *
2446+ * FIXME: This could be made more efficient by jumping ahead
2447+ * using pvr_mmu_op_context_set_curr_page().
2448+ */
2449+ if (err == -ENXIO)
2450+ continue;
2451+ else if (err)
2452+ return err;
2453+
2454+ pvr_page_destroy(op_ctx);
2455+ }
2456+
2457+ return 0;
2458+}
2459+
2460+/**
2461+ * pvr_mmu_unmap() - Unmap pages from a memory context.
2462+ * @op_ctx: Target MMU op context.
2463+ * @device_addr: First device-virtual address to unmap.
2464+ * @size: Size in bytes to unmap.
2465+ *
2466+ * The total amount of device-virtual memory unmapped is
2467+ * @nr_pages * %PVR_DEVICE_PAGE_SIZE.
2468+ *
2469+ * Returns:
2470+ * * 0 on success, or
2471+ * * Any error code returned by pvr_page_table_ptr_init(), or
2472+ * * Any error code returned by pvr_page_table_ptr_unmap().
2473+ */
2474+int pvr_mmu_unmap(struct pvr_mmu_op_context *op_ctx, u64 device_addr, u64 size)
2475+{
2476+ int err = pvr_mmu_op_context_set_curr_page(op_ctx, device_addr, false);
2477+
2478+ if (err)
2479+ return err;
2480+
2481+ return pvr_mmu_op_context_unmap_curr_page(op_ctx,
2482+ size >> PVR_DEVICE_PAGE_SHIFT);
2483+}
2484+
2485+/**
2486+ * pvr_mmu_map_sgl() - Map part of a scatter-gather table entry to
2487+ * device-virtual memory.
2488+ * @op_ctx: Target MMU op context pointing to the first page that should be
2489+ * mapped.
2490+ * @sgl: Target scatter-gather table entry.
2491+ * @offset: Offset into @sgl to map from. Must result in a starting address
2492+ * from @sgl which is CPU page-aligned.
2493+ * @size: Size of the memory to be mapped in bytes. Must be a non-zero multiple
2494+ * of the device page size.
2495+ * @page_flags: Page options to be applied to every device-virtual memory page
2496+ * in the created mapping.
2497+ *
2498+ * Return:
2499+ * * 0 on success,
2500+ * * -%EINVAL if the range specified by @offset and @size is not completely
2501+ * within @sgl, or
2502+ * * Any error encountered while creating a page with pvr_page_create(), or
2503+ * * Any error encountered while advancing @op_ctx.curr_page with
2504+ * pvr_mmu_op_context_next_page().
2505+ */
2506+static int
2507+pvr_mmu_map_sgl(struct pvr_mmu_op_context *op_ctx, struct scatterlist *sgl,
2508+ u64 offset, u64 size, struct pvr_page_flags_raw page_flags)
2509+{
2510+ const unsigned int pages = size >> PVR_DEVICE_PAGE_SHIFT;
2511+ dma_addr_t dma_addr = sg_dma_address(sgl) + offset;
2512+ const unsigned int dma_len = sg_dma_len(sgl);
2513+ struct pvr_page_table_ptr ptr_copy;
2514+ unsigned int page;
2515+ int err;
2516+
2517+ if (size > dma_len || offset > dma_len - size)
2518+ return -EINVAL;
2519+
2520+ /*
2521+ * Before progressing, save a copy of the start pointer so we can use
2522+ * it again if we enter an error state and have to destroy pages.
2523+ */
2524+ memcpy(&ptr_copy, &op_ctx->curr_page, sizeof(ptr_copy));
2525+
2526+ /*
2527+ * Create first page outside loop, as it doesn't require a page advance
2528+ * beforehand.
2529+ */
2530+ err = pvr_page_create(op_ctx, dma_addr, page_flags);
2531+ if (err)
2532+ return err;
2533+
2534+ for (page = 1; page < pages; ++page) {
2535+ err = pvr_mmu_op_context_next_page(op_ctx, true);
2536+ if (err)
2537+ goto err_destroy_pages;
2538+
2539+ dma_addr += PVR_DEVICE_PAGE_SIZE;
2540+
2541+ err = pvr_page_create(op_ctx, dma_addr, page_flags);
2542+ if (err)
2543+ goto err_destroy_pages;
2544+ }
2545+
2546+ return 0;
2547+
2548+err_destroy_pages:
2549+ memcpy(&op_ctx->curr_page, &ptr_copy, sizeof(op_ctx->curr_page));
2550+ err = pvr_mmu_op_context_unmap_curr_page(op_ctx, page);
2551+
2552+ return err;
2553+}
2554+
2555+/**
2556+ * pvr_mmu_map() - Map an object's virtual memory to physical memory.
2557+ * @op_ctx: Target MMU op context.
2558+ * @size: Size of memory to be mapped in bytes. Must be a non-zero multiple
2559+ * of the device page size.
2560+ * @flags: Flags from pvr_gem_object associated with the mapping.
2561+ * @device_addr: Virtual device address to map to. Must be device page-aligned.
2562+ *
2563+ * Returns:
2564+ * * 0 on success, or
2565+ * * Any error code returned by pvr_page_table_ptr_init(), or
2566+ * * Any error code returned by pvr_mmu_map_sgl(), or
2567+ * * Any error code returned by pvr_page_table_ptr_next_page().
2568+ */
2569+int pvr_mmu_map(struct pvr_mmu_op_context *op_ctx, u64 size, u64 flags,
2570+ u64 device_addr)
2571+{
2572+ struct pvr_page_table_ptr ptr_copy;
2573+ struct pvr_page_flags_raw flags_raw;
2574+ struct scatterlist *sgl;
2575+ u64 mapped_size = 0;
2576+ unsigned int count;
2577+ int err;
2578+
2579+ if (!size)
2580+ return 0;
2581+
2582+ if ((op_ctx->map.sgt_offset | size) & ~PVR_DEVICE_PAGE_MASK)
2583+ return -EINVAL;
2584+
2585+ err = pvr_mmu_op_context_set_curr_page(op_ctx, device_addr, true);
2586+ if (err)
2587+ return -EINVAL;
2588+
2589+ memcpy(&ptr_copy, &op_ctx->curr_page, sizeof(ptr_copy));
2590+
2591+ flags_raw = pvr_page_flags_raw_create(false, false,
2592+ flags & DRM_PVR_BO_BYPASS_DEVICE_CACHE,
2593+ flags & DRM_PVR_BO_PM_FW_PROTECT);
2594+
2595+ /* Map scatter gather table */
2596+ for_each_sgtable_dma_sg(op_ctx->map.sgt, sgl, count) {
2597+ const size_t sgl_len = sg_dma_len(sgl);
2598+ u64 sgl_offset, map_sgl_len;
2599+
2600+ if (sgl_len <= op_ctx->map.sgt_offset) {
2601+ op_ctx->map.sgt_offset -= sgl_len;
2602+ continue;
2603+ }
2604+
2605+ sgl_offset = op_ctx->map.sgt_offset;
2606+ map_sgl_len = min_t(u64, sgl_len - sgl_offset, size - mapped_size);
2607+
2608+ err = pvr_mmu_map_sgl(op_ctx, sgl, sgl_offset, map_sgl_len,
2609+ flags_raw);
2610+ if (err)
2611+ break;
2612+
2613+ /*
2614+ * Flag the L0 page table as requiring a flush when the MMU op
2615+ * context is destroyed.
2616+ */
2617+ pvr_mmu_op_context_require_sync(op_ctx, PVR_MMU_SYNC_LEVEL_0);
2618+
2619+ op_ctx->map.sgt_offset = 0;
2620+ mapped_size += map_sgl_len;
2621+
2622+ if (mapped_size >= size)
2623+ break;
2624+
2625+ err = pvr_mmu_op_context_next_page(op_ctx, true);
2626+ if (err)
2627+ break;
2628+ }
2629+
2630+ if (err && mapped_size) {
2631+ memcpy(&op_ctx->curr_page, &ptr_copy, sizeof(op_ctx->curr_page));
2632+ pvr_mmu_op_context_unmap_curr_page(op_ctx,
2633+ mapped_size >> PVR_DEVICE_PAGE_SHIFT);
2634+ }
2635+
2636+ return err;
2637+}
drivers/gpu/drm/imagination/pvr_mmu.h+108−0
@@ -0,0 +1,108 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_MMU_H
5+#define PVR_MMU_H
6+
7+#include <linux/memory.h>
8+#include <linux/types.h>
9+
10+/* Forward declaration from "pvr_device.h" */
11+struct pvr_device;
12+
13+/* Forward declaration from "pvr_mmu.c" */
14+struct pvr_mmu_context;
15+struct pvr_mmu_op_context;
16+
17+/* Forward declaration from "pvr_vm.c" */
18+struct pvr_vm_context;
19+
20+/* Forward declaration from <linux/scatterlist.h> */
21+struct sg_table;
22+
23+/**
24+ * DOC: Public API (constants)
25+ *
26+ * .. c:macro:: PVR_DEVICE_PAGE_SIZE
27+ *
28+ * Fixed page size referenced by leaf nodes in the page table tree
29+ * structure. In the current implementation, this value is pegged to the
30+ * CPU page size (%PAGE_SIZE). It is therefore an error to specify a CPU
31+ * page size which is not also a supported device page size. The supported
32+ * device page sizes are: 4KiB, 16KiB, 64KiB, 256KiB, 1MiB and 2MiB.
33+ *
34+ * .. c:macro:: PVR_DEVICE_PAGE_SHIFT
35+ *
36+ * Shift value used to efficiently multiply or divide by
37+ * %PVR_DEVICE_PAGE_SIZE.
38+ *
39+ * This value is derived from %PVR_DEVICE_PAGE_SIZE.
40+ *
41+ * .. c:macro:: PVR_DEVICE_PAGE_MASK
42+ *
43+ * Mask used to round a value down to the nearest multiple of
44+ * %PVR_DEVICE_PAGE_SIZE. When bitwise negated, it will indicate whether a
45+ * value is already a multiple of %PVR_DEVICE_PAGE_SIZE.
46+ *
47+ * This value is derived from %PVR_DEVICE_PAGE_SIZE.
48+ */
49+
50+/* PVR_DEVICE_PAGE_SIZE determines the page size */
51+#define PVR_DEVICE_PAGE_SIZE (PAGE_SIZE)
52+#define PVR_DEVICE_PAGE_SHIFT (PAGE_SHIFT)
53+#define PVR_DEVICE_PAGE_MASK (PAGE_MASK)
54+
55+/**
56+ * DOC: Page table index utilities (constants)
57+ *
58+ * .. c:macro:: PVR_PAGE_TABLE_ADDR_SPACE_SIZE
59+ *
60+ * Size of device-virtual address space which can be represented in the page
61+ * table structure.
62+ *
63+ * This value is checked at runtime against
64+ * &pvr_device_features.virtual_address_space_bits by
65+ * pvr_vm_create_context(), which will return an error if the feature value
66+ * does not match this constant.
67+ *
68+ * .. admonition:: Future work
69+ *
70+ * It should be possible to support other values of
71+ * &pvr_device_features.virtual_address_space_bits, but so far no
72+ * hardware has been created which advertises an unsupported value.
73+ *
74+ * .. c:macro:: PVR_PAGE_TABLE_ADDR_BITS
75+ *
76+ * Number of bits needed to represent any value less than
77+ * %PVR_PAGE_TABLE_ADDR_SPACE_SIZE exactly.
78+ *
79+ * .. c:macro:: PVR_PAGE_TABLE_ADDR_MASK
80+ *
81+ * Bitmask of device-virtual addresses which are valid in the page table
82+ * structure.
83+ *
84+ * This value is derived from %PVR_PAGE_TABLE_ADDR_SPACE_SIZE, so the same
85+ * notes on that constant apply here.
86+ */
87+#define PVR_PAGE_TABLE_ADDR_SPACE_SIZE SZ_1T
88+#define PVR_PAGE_TABLE_ADDR_BITS __ffs(PVR_PAGE_TABLE_ADDR_SPACE_SIZE)
89+#define PVR_PAGE_TABLE_ADDR_MASK (PVR_PAGE_TABLE_ADDR_SPACE_SIZE - 1)
90+
91+void pvr_mmu_flush_request_all(struct pvr_device *pvr_dev);
92+int pvr_mmu_flush_exec(struct pvr_device *pvr_dev, bool wait);
93+
94+struct pvr_mmu_context *pvr_mmu_context_create(struct pvr_device *pvr_dev);
95+void pvr_mmu_context_destroy(struct pvr_mmu_context *ctx);
96+
97+dma_addr_t pvr_mmu_get_root_table_dma_addr(struct pvr_mmu_context *ctx);
98+
99+void pvr_mmu_op_context_destroy(struct pvr_mmu_op_context *op_ctx);
100+struct pvr_mmu_op_context *
101+pvr_mmu_op_context_create(struct pvr_mmu_context *ctx,
102+ struct sg_table *sgt, u64 sgt_offset, u64 size);
103+
104+int pvr_mmu_map(struct pvr_mmu_op_context *op_ctx, u64 size, u64 flags,
105+ u64 device_addr);
106+int pvr_mmu_unmap(struct pvr_mmu_op_context *op_ctx, u64 device_addr, u64 size);
107+
108+#endif /* PVR_MMU_H */
drivers/gpu/drm/imagination/pvr_params.c+147−0
@@ -0,0 +1,147 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_params.h"
5+
6+#include <linux/cache.h>
7+#include <linux/moduleparam.h>
8+
9+static struct pvr_device_params pvr_device_param_defaults __read_mostly = {
10+#define X(type_, name_, value_, desc_, ...) .name_ = (value_),
11+ PVR_DEVICE_PARAMS
12+#undef X
13+};
14+
15+#define PVR_DEVICE_PARAM_NAMED(name_, type_, desc_) \
16+ module_param_named(name_, pvr_device_param_defaults.name_, type_, \
17+ 0400); \
18+ MODULE_PARM_DESC(name_, desc_);
19+
20+/*
21+ * This list of defines must contain every type specified in "pvr_params.h" as
22+ * ``PVR_PARAM_TYPE_*_C``.
23+ */
24+#define PVR_PARAM_TYPE_X32_MODPARAM uint
25+
26+#define X(type_, name_, value_, desc_, ...) \
27+ PVR_DEVICE_PARAM_NAMED(name_, PVR_PARAM_TYPE_##type_##_MODPARAM, desc_);
28+PVR_DEVICE_PARAMS
29+#undef X
30+
31+int
32+pvr_device_params_init(struct pvr_device_params *params)
33+{
34+ /*
35+ * If heap-allocated parameters are added in the future (e.g.
36+ * modparam's charp type), they must be handled specially here (via
37+ * kstrdup() in the case of charp). Since that's not necessary yet,
38+ * a straight copy will do for now. This change will also require a
39+ * pvr_device_params_fini() function to free any heap-allocated copies.
40+ */
41+
42+ *params = pvr_device_param_defaults;
43+
44+ return 0;
45+}
46+
47+#if defined(CONFIG_DEBUG_FS)
48+#include "pvr_device.h"
49+
50+#include <linux/dcache.h>
51+#include <linux/debugfs.h>
52+#include <linux/export.h>
53+#include <linux/fs.h>
54+#include <linux/stddef.h>
55+
56+/*
57+ * This list of defines must contain every type specified in "pvr_params.h" as
58+ * ``PVR_PARAM_TYPE_*_C``.
59+ */
60+#define PVR_PARAM_TYPE_X32_FMT "0x%08llx"
61+
62+#define X_SET(name_, mode_) X_SET_##mode_(name_)
63+#define X_SET_DEF(name_, update_, mode_) X_SET_DEF_##mode_(name_, update_)
64+
65+#define X_SET_RO(name_) NULL
66+#define X_SET_RW(name_) __pvr_device_param_##name_##set
67+
68+#define X_SET_DEF_RO(name_, update_)
69+#define X_SET_DEF_RW(name_, update_) \
70+ static int \
71+ X_SET_RW(name_)(void *data, u64 val) \
72+ { \
73+ struct pvr_device *pvr_dev = data; \
74+ /* This is not just (update_) to suppress -Waddress. */ \
75+ if ((void *)(update_) != NULL) \
76+ (update_)(pvr_dev, pvr_dev->params.name_, val); \
77+ pvr_dev->params.name_ = val; \
78+ return 0; \
79+ }
80+
81+#define X(type_, name_, value_, desc_, mode_, update_) \
82+ static int \
83+ __pvr_device_param_##name_##_get(void *data, u64 *val) \
84+ { \
85+ struct pvr_device *pvr_dev = data; \
86+ *val = pvr_dev->params.name_; \
87+ return 0; \
88+ } \
89+ X_SET_DEF(name_, update_, mode_) \
90+ static int \
91+ __pvr_device_param_##name_##_open(struct inode *inode, \
92+ struct file *file) \
93+ { \
94+ __simple_attr_check_format(PVR_PARAM_TYPE_##type_##_FMT, \
95+ 0ull); \
96+ return simple_attr_open(inode, file, \
97+ __pvr_device_param_##name_##_get, \
98+ X_SET(name_, mode_), \
99+ PVR_PARAM_TYPE_##type_##_FMT); \
100+ }
101+PVR_DEVICE_PARAMS
102+#undef X
103+
104+#undef X_SET
105+#undef X_SET_RO
106+#undef X_SET_RW
107+#undef X_SET_DEF
108+#undef X_SET_DEF_RO
109+#undef X_SET_DEF_RW
110+
111+static struct {
112+#define X(type_, name_, value_, desc_, mode_, update_) \
113+ const struct file_operations name_;
114+ PVR_DEVICE_PARAMS
115+#undef X
116+} pvr_device_param_debugfs_fops = {
117+#define X(type_, name_, value_, desc_, mode_, update_) \
118+ .name_ = { \
119+ .owner = THIS_MODULE, \
120+ .open = __pvr_device_param_##name_##_open, \
121+ .release = simple_attr_release, \
122+ .read = simple_attr_read, \
123+ .write = simple_attr_write, \
124+ .llseek = generic_file_llseek, \
125+ },
126+ PVR_DEVICE_PARAMS
127+#undef X
128+};
129+
130+void
131+pvr_params_debugfs_init(struct pvr_device *pvr_dev, struct dentry *dir)
132+{
133+#define X_MODE(mode_) X_MODE_##mode_
134+#define X_MODE_RO 0400
135+#define X_MODE_RW 0600
136+
137+#define X(type_, name_, value_, desc_, mode_, update_) \
138+ debugfs_create_file(#name_, X_MODE(mode_), dir, pvr_dev, \
139+ &pvr_device_param_debugfs_fops.name_);
140+ PVR_DEVICE_PARAMS
141+#undef X
142+
143+#undef X_MODE
144+#undef X_MODE_RO
145+#undef X_MODE_RW
146+}
147+#endif
drivers/gpu/drm/imagination/pvr_params.h+72−0
@@ -0,0 +1,72 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_PARAMS_H
5+#define PVR_PARAMS_H
6+
7+#include "pvr_rogue_fwif.h"
8+
9+#include <linux/cache.h>
10+#include <linux/compiler_attributes.h>
11+
12+/*
13+ * This is the definitive list of types allowed in the definition of
14+ * %PVR_DEVICE_PARAMS.
15+ */
16+#define PVR_PARAM_TYPE_X32_C u32
17+
18+/*
19+ * This macro defines all device-specific parameters; that is parameters which
20+ * are set independently per device.
21+ *
22+ * The X-macro accepts the following arguments. Arguments marked with [debugfs]
23+ * are ignored when debugfs is disabled; values used for these arguments may
24+ * safely be gated behind CONFIG_DEBUG_FS.
25+ *
26+ * @type_: The definitive list of allowed values is PVR_PARAM_TYPE_*_C.
27+ * @name_: Name of the parameter. This is used both as the field name in C and
28+ * stringified as the parameter name.
29+ * @value_: Initial/default value.
30+ * @desc_: String literal used as help text to describe the usage of this
31+ * parameter.
32+ * @mode_: [debugfs] One of {RO,RW}. The access mode of the debugfs entry for
33+ * this parameter.
34+ * @update_: [debugfs] When debugfs support is enabled, parameters may be
35+ * updated at runtime. When this happens, this function will be
36+ * called to allow changes to propagate. The signature of this
37+ * function is:
38+ *
39+ * void (*)(struct pvr_device *pvr_dev, T old_val, T new_val)
40+ *
41+ * Where T is the C type associated with @type_.
42+ *
43+ * If @mode_ does not allow write access, this function will never be
44+ * called. In this case, or if no update callback is required, you
45+ * should specify NULL for this argument.
46+ */
47+#define PVR_DEVICE_PARAMS \
48+ X(X32, fw_trace_mask, ROGUE_FWIF_LOG_TYPE_NONE, \
49+ "Enable FW trace for the specified groups. Specifying 0 disables " \
50+ "all FW tracing.", \
51+ RW, pvr_fw_trace_mask_update)
52+
53+struct pvr_device_params {
54+#define X(type_, name_, value_, desc_, ...) \
55+ PVR_PARAM_TYPE_##type_##_C name_;
56+ PVR_DEVICE_PARAMS
57+#undef X
58+};
59+
60+int pvr_device_params_init(struct pvr_device_params *params);
61+
62+#if defined(CONFIG_DEBUG_FS)
63+/* Forward declaration from "pvr_device.h". */
64+struct pvr_device;
65+
66+/* Forward declaration from <linux/dcache.h>. */
67+struct dentry;
68+
69+void pvr_params_debugfs_init(struct pvr_device *pvr_dev, struct dentry *dir);
70+#endif /* defined(CONFIG_DEBUG_FS) */
71+
72+#endif /* PVR_PARAMS_H */
drivers/gpu/drm/imagination/pvr_power.c+433−0
@@ -0,0 +1,433 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include "pvr_device.h"
5+#include "pvr_fw.h"
6+#include "pvr_fw_startstop.h"
7+#include "pvr_power.h"
8+#include "pvr_queue.h"
9+#include "pvr_rogue_fwif.h"
10+
11+#include <drm/drm_drv.h>
12+#include <drm/drm_managed.h>
13+#include <linux/clk.h>
14+#include <linux/interrupt.h>
15+#include <linux/mutex.h>
16+#include <linux/platform_device.h>
17+#include <linux/pm_runtime.h>
18+#include <linux/timer.h>
19+#include <linux/types.h>
20+#include <linux/workqueue.h>
21+
22+#define POWER_SYNC_TIMEOUT_US (1000000) /* 1s */
23+
24+#define WATCHDOG_TIME_MS (500)
25+
26+/**
27+ * pvr_device_lost() - Mark GPU device as lost
28+ * @pvr_dev: Target PowerVR device.
29+ *
30+ * This will cause the DRM device to be unplugged.
31+ */
32+void
33+pvr_device_lost(struct pvr_device *pvr_dev)
34+{
35+ if (!pvr_dev->lost) {
36+ pvr_dev->lost = true;
37+ drm_dev_unplug(from_pvr_device(pvr_dev));
38+ }
39+}
40+
41+static int
42+pvr_power_send_command(struct pvr_device *pvr_dev, struct rogue_fwif_kccb_cmd *pow_cmd)
43+{
44+ struct pvr_fw_device *fw_dev = &pvr_dev->fw_dev;
45+ u32 slot_nr;
46+ u32 value;
47+ int err;
48+
49+ WRITE_ONCE(*fw_dev->power_sync, 0);
50+
51+ err = pvr_kccb_send_cmd_powered(pvr_dev, pow_cmd, &slot_nr);
52+ if (err)
53+ return err;
54+
55+ /* Wait for FW to acknowledge. */
56+ return readl_poll_timeout(pvr_dev->fw_dev.power_sync, value, value != 0, 100,
57+ POWER_SYNC_TIMEOUT_US);
58+}
59+
60+static int
61+pvr_power_request_idle(struct pvr_device *pvr_dev)
62+{
63+ struct rogue_fwif_kccb_cmd pow_cmd;
64+
65+ /* Send FORCED_IDLE request to FW. */
66+ pow_cmd.cmd_type = ROGUE_FWIF_KCCB_CMD_POW;
67+ pow_cmd.cmd_data.pow_data.pow_type = ROGUE_FWIF_POW_FORCED_IDLE_REQ;
68+ pow_cmd.cmd_data.pow_data.power_req_data.pow_request_type = ROGUE_FWIF_POWER_FORCE_IDLE;
69+
70+ return pvr_power_send_command(pvr_dev, &pow_cmd);
71+}
72+
73+static int
74+pvr_power_request_pwr_off(struct pvr_device *pvr_dev)
75+{
76+ struct rogue_fwif_kccb_cmd pow_cmd;
77+
78+ /* Send POW_OFF request to firmware. */
79+ pow_cmd.cmd_type = ROGUE_FWIF_KCCB_CMD_POW;
80+ pow_cmd.cmd_data.pow_data.pow_type = ROGUE_FWIF_POW_OFF_REQ;
81+ pow_cmd.cmd_data.pow_data.power_req_data.forced = true;
82+
83+ return pvr_power_send_command(pvr_dev, &pow_cmd);
84+}
85+
86+static int
87+pvr_power_fw_disable(struct pvr_device *pvr_dev, bool hard_reset)
88+{
89+ if (!hard_reset) {
90+ int err;
91+
92+ cancel_delayed_work_sync(&pvr_dev->watchdog.work);
93+
94+ err = pvr_power_request_idle(pvr_dev);
95+ if (err)
96+ return err;
97+
98+ err = pvr_power_request_pwr_off(pvr_dev);
99+ if (err)
100+ return err;
101+ }
102+
103+ return pvr_fw_stop(pvr_dev);
104+}
105+
106+static int
107+pvr_power_fw_enable(struct pvr_device *pvr_dev)
108+{
109+ int err;
110+
111+ err = pvr_fw_start(pvr_dev);
112+ if (err)
113+ return err;
114+
115+ err = pvr_wait_for_fw_boot(pvr_dev);
116+ if (err) {
117+ drm_err(from_pvr_device(pvr_dev), "Firmware failed to boot\n");
118+ pvr_fw_stop(pvr_dev);
119+ return err;
120+ }
121+
122+ queue_delayed_work(pvr_dev->sched_wq, &pvr_dev->watchdog.work,
123+ msecs_to_jiffies(WATCHDOG_TIME_MS));
124+
125+ return 0;
126+}
127+
128+bool
129+pvr_power_is_idle(struct pvr_device *pvr_dev)
130+{
131+ /*
132+ * FW power state can be out of date if a KCCB command has been submitted but the FW hasn't
133+ * started processing it yet. So also check the KCCB status.
134+ */
135+ enum rogue_fwif_pow_state pow_state = READ_ONCE(pvr_dev->fw_dev.fwif_sysdata->pow_state);
136+ bool kccb_idle = pvr_kccb_is_idle(pvr_dev);
137+
138+ return (pow_state == ROGUE_FWIF_POW_IDLE) && kccb_idle;
139+}
140+
141+static bool
142+pvr_watchdog_kccb_stalled(struct pvr_device *pvr_dev)
143+{
144+ /* Check KCCB commands are progressing. */
145+ u32 kccb_cmds_executed = pvr_dev->fw_dev.fwif_osdata->kccb_cmds_executed;
146+ bool kccb_is_idle = pvr_kccb_is_idle(pvr_dev);
147+
148+ if (pvr_dev->watchdog.old_kccb_cmds_executed == kccb_cmds_executed && !kccb_is_idle) {
149+ pvr_dev->watchdog.kccb_stall_count++;
150+
151+ /*
152+ * If we have commands pending with no progress for 2 consecutive polls then
153+ * consider KCCB command processing stalled.
154+ */
155+ if (pvr_dev->watchdog.kccb_stall_count == 2) {
156+ pvr_dev->watchdog.kccb_stall_count = 0;
157+ return true;
158+ }
159+ } else if (pvr_dev->watchdog.old_kccb_cmds_executed == kccb_cmds_executed) {
160+ bool has_active_contexts;
161+
162+ mutex_lock(&pvr_dev->queues.lock);
163+ has_active_contexts = list_empty(&pvr_dev->queues.active);
164+ mutex_unlock(&pvr_dev->queues.lock);
165+
166+ if (has_active_contexts) {
167+ /* Send a HEALTH_CHECK command so we can verify FW is still alive. */
168+ struct rogue_fwif_kccb_cmd health_check_cmd;
169+
170+ health_check_cmd.cmd_type = ROGUE_FWIF_KCCB_CMD_HEALTH_CHECK;
171+
172+ pvr_kccb_send_cmd_powered(pvr_dev, &health_check_cmd, NULL);
173+ }
174+ } else {
175+ pvr_dev->watchdog.old_kccb_cmds_executed = kccb_cmds_executed;
176+ pvr_dev->watchdog.kccb_stall_count = 0;
177+ }
178+
179+ return false;
180+}
181+
182+static void
183+pvr_watchdog_worker(struct work_struct *work)
184+{
185+ struct pvr_device *pvr_dev = container_of(work, struct pvr_device,
186+ watchdog.work.work);
187+ bool stalled;
188+
189+ if (pvr_dev->lost)
190+ return;
191+
192+ if (pm_runtime_get_if_in_use(from_pvr_device(pvr_dev)->dev) <= 0)
193+ goto out_requeue;
194+
195+ if (!pvr_dev->fw_dev.booted)
196+ goto out_pm_runtime_put;
197+
198+ stalled = pvr_watchdog_kccb_stalled(pvr_dev);
199+
200+ if (stalled) {
201+ drm_err(from_pvr_device(pvr_dev), "FW stalled, trying hard reset");
202+
203+ pvr_power_reset(pvr_dev, true);
204+ /* Device may be lost at this point. */
205+ }
206+
207+out_pm_runtime_put:
208+ pm_runtime_put(from_pvr_device(pvr_dev)->dev);
209+
210+out_requeue:
211+ if (!pvr_dev->lost) {
212+ queue_delayed_work(pvr_dev->sched_wq, &pvr_dev->watchdog.work,
213+ msecs_to_jiffies(WATCHDOG_TIME_MS));
214+ }
215+}
216+
217+/**
218+ * pvr_watchdog_init() - Initialise watchdog for device
219+ * @pvr_dev: Target PowerVR device.
220+ *
221+ * Returns:
222+ * * 0 on success, or
223+ * * -%ENOMEM on out of memory.
224+ */
225+int
226+pvr_watchdog_init(struct pvr_device *pvr_dev)
227+{
228+ INIT_DELAYED_WORK(&pvr_dev->watchdog.work, pvr_watchdog_worker);
229+
230+ return 0;
231+}
232+
233+int
234+pvr_power_device_suspend(struct device *dev)
235+{
236+ struct platform_device *plat_dev = to_platform_device(dev);
237+ struct drm_device *drm_dev = platform_get_drvdata(plat_dev);
238+ struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
239+ int err = 0;
240+ int idx;
241+
242+ if (!drm_dev_enter(drm_dev, &idx))
243+ return -EIO;
244+
245+ if (pvr_dev->fw_dev.booted) {
246+ err = pvr_power_fw_disable(pvr_dev, false);
247+ if (err)
248+ goto err_drm_dev_exit;
249+ }
250+
251+ clk_disable_unprepare(pvr_dev->mem_clk);
252+ clk_disable_unprepare(pvr_dev->sys_clk);
253+ clk_disable_unprepare(pvr_dev->core_clk);
254+
255+err_drm_dev_exit:
256+ drm_dev_exit(idx);
257+
258+ return err;
259+}
260+
261+int
262+pvr_power_device_resume(struct device *dev)
263+{
264+ struct platform_device *plat_dev = to_platform_device(dev);
265+ struct drm_device *drm_dev = platform_get_drvdata(plat_dev);
266+ struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
267+ int idx;
268+ int err;
269+
270+ if (!drm_dev_enter(drm_dev, &idx))
271+ return -EIO;
272+
273+ err = clk_prepare_enable(pvr_dev->core_clk);
274+ if (err)
275+ goto err_drm_dev_exit;
276+
277+ err = clk_prepare_enable(pvr_dev->sys_clk);
278+ if (err)
279+ goto err_core_clk_disable;
280+
281+ err = clk_prepare_enable(pvr_dev->mem_clk);
282+ if (err)
283+ goto err_sys_clk_disable;
284+
285+ if (pvr_dev->fw_dev.booted) {
286+ err = pvr_power_fw_enable(pvr_dev);
287+ if (err)
288+ goto err_mem_clk_disable;
289+ }
290+
291+ drm_dev_exit(idx);
292+
293+ return 0;
294+
295+err_mem_clk_disable:
296+ clk_disable_unprepare(pvr_dev->mem_clk);
297+
298+err_sys_clk_disable:
299+ clk_disable_unprepare(pvr_dev->sys_clk);
300+
301+err_core_clk_disable:
302+ clk_disable_unprepare(pvr_dev->core_clk);
303+
304+err_drm_dev_exit:
305+ drm_dev_exit(idx);
306+
307+ return err;
308+}
309+
310+int
311+pvr_power_device_idle(struct device *dev)
312+{
313+ struct platform_device *plat_dev = to_platform_device(dev);
314+ struct drm_device *drm_dev = platform_get_drvdata(plat_dev);
315+ struct pvr_device *pvr_dev = to_pvr_device(drm_dev);
316+
317+ return pvr_power_is_idle(pvr_dev) ? 0 : -EBUSY;
318+}
319+
320+/**
321+ * pvr_power_reset() - Reset the GPU
322+ * @pvr_dev: Device pointer
323+ * @hard_reset: %true for hard reset, %false for soft reset
324+ *
325+ * If @hard_reset is %false and the FW processor fails to respond during the reset process, this
326+ * function will attempt a hard reset.
327+ *
328+ * If a hard reset fails then the GPU device is reported as lost.
329+ *
330+ * Returns:
331+ * * 0 on success, or
332+ * * Any error code returned by pvr_power_get, pvr_power_fw_disable or pvr_power_fw_enable().
333+ */
334+int
335+pvr_power_reset(struct pvr_device *pvr_dev, bool hard_reset)
336+{
337+ bool queues_disabled = false;
338+ int err;
339+
340+ /*
341+ * Take a power reference during the reset. This should prevent any interference with the
342+ * power state during reset.
343+ */
344+ WARN_ON(pvr_power_get(pvr_dev));
345+
346+ down_write(&pvr_dev->reset_sem);
347+
348+ if (pvr_dev->lost) {
349+ err = -EIO;
350+ goto err_up_write;
351+ }
352+
353+ /* Disable IRQs for the duration of the reset. */
354+ disable_irq(pvr_dev->irq);
355+
356+ do {
357+ if (hard_reset) {
358+ pvr_queue_device_pre_reset(pvr_dev);
359+ queues_disabled = true;
360+ }
361+
362+ err = pvr_power_fw_disable(pvr_dev, hard_reset);
363+ if (!err) {
364+ if (hard_reset) {
365+ pvr_dev->fw_dev.booted = false;
366+ WARN_ON(pm_runtime_force_suspend(from_pvr_device(pvr_dev)->dev));
367+
368+ err = pvr_fw_hard_reset(pvr_dev);
369+ if (err)
370+ goto err_device_lost;
371+
372+ err = pm_runtime_force_resume(from_pvr_device(pvr_dev)->dev);
373+ pvr_dev->fw_dev.booted = true;
374+ if (err)
375+ goto err_device_lost;
376+ } else {
377+ /* Clear the FW faulted flags. */
378+ pvr_dev->fw_dev.fwif_sysdata->hwr_state_flags &=
379+ ~(ROGUE_FWIF_HWR_FW_FAULT |
380+ ROGUE_FWIF_HWR_RESTART_REQUESTED);
381+ }
382+
383+ pvr_fw_irq_clear(pvr_dev);
384+
385+ err = pvr_power_fw_enable(pvr_dev);
386+ }
387+
388+ if (err && hard_reset)
389+ goto err_device_lost;
390+
391+ if (err && !hard_reset) {
392+ drm_err(from_pvr_device(pvr_dev), "FW stalled, trying hard reset");
393+ hard_reset = true;
394+ }
395+ } while (err);
396+
397+ if (queues_disabled)
398+ pvr_queue_device_post_reset(pvr_dev);
399+
400+ enable_irq(pvr_dev->irq);
401+
402+ up_write(&pvr_dev->reset_sem);
403+
404+ pvr_power_put(pvr_dev);
405+
406+ return 0;
407+
408+err_device_lost:
409+ drm_err(from_pvr_device(pvr_dev), "GPU device lost");
410+ pvr_device_lost(pvr_dev);
411+
412+ /* Leave IRQs disabled if the device is lost. */
413+
414+ if (queues_disabled)
415+ pvr_queue_device_post_reset(pvr_dev);
416+
417+err_up_write:
418+ up_write(&pvr_dev->reset_sem);
419+
420+ pvr_power_put(pvr_dev);
421+
422+ return err;
423+}
424+
425+/**
426+ * pvr_watchdog_fini() - Shutdown watchdog for device
427+ * @pvr_dev: Target PowerVR device.
428+ */
429+void
430+pvr_watchdog_fini(struct pvr_device *pvr_dev)
431+{
432+ cancel_delayed_work_sync(&pvr_dev->watchdog.work);
433+}
drivers/gpu/drm/imagination/pvr_power.h+41−0
@@ -0,0 +1,41 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_POWER_H
5+#define PVR_POWER_H
6+
7+#include "pvr_device.h"
8+
9+#include <linux/mutex.h>
10+#include <linux/pm_runtime.h>
11+
12+int pvr_watchdog_init(struct pvr_device *pvr_dev);
13+void pvr_watchdog_fini(struct pvr_device *pvr_dev);
14+
15+void pvr_device_lost(struct pvr_device *pvr_dev);
16+
17+bool pvr_power_is_idle(struct pvr_device *pvr_dev);
18+
19+int pvr_power_device_suspend(struct device *dev);
20+int pvr_power_device_resume(struct device *dev);
21+int pvr_power_device_idle(struct device *dev);
22+
23+int pvr_power_reset(struct pvr_device *pvr_dev, bool hard_reset);
24+
25+static __always_inline int
26+pvr_power_get(struct pvr_device *pvr_dev)
27+{
28+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
29+
30+ return pm_runtime_resume_and_get(drm_dev->dev);
31+}
32+
33+static __always_inline int
34+pvr_power_put(struct pvr_device *pvr_dev)
35+{
36+ struct drm_device *drm_dev = from_pvr_device(pvr_dev);
37+
38+ return pm_runtime_put(drm_dev->dev);
39+}
40+
41+#endif /* PVR_POWER_H */
drivers/gpu/drm/imagination/pvr_queue.c+1432−0
@@ -0,0 +1,1432 @@
1+// SPDX-License-Identifier: GPL-2.0-only OR MIT
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#include <drm/drm_managed.h>
5+#include <drm/gpu_scheduler.h>
6+
7+#include "pvr_cccb.h"
8+#include "pvr_context.h"
9+#include "pvr_device.h"
10+#include "pvr_drv.h"
11+#include "pvr_job.h"
12+#include "pvr_queue.h"
13+#include "pvr_vm.h"
14+
15+#include "pvr_rogue_fwif_client.h"
16+
17+#define MAX_DEADLINE_MS 30000
18+
19+#define CTX_COMPUTE_CCCB_SIZE_LOG2 15
20+#define CTX_FRAG_CCCB_SIZE_LOG2 15
21+#define CTX_GEOM_CCCB_SIZE_LOG2 15
22+#define CTX_TRANSFER_CCCB_SIZE_LOG2 15
23+
24+static int get_xfer_ctx_state_size(struct pvr_device *pvr_dev)
25+{
26+ u32 num_isp_store_registers;
27+
28+ if (PVR_HAS_FEATURE(pvr_dev, xe_memory_hierarchy)) {
29+ num_isp_store_registers = 1;
30+ } else {
31+ int err;
32+
33+ err = PVR_FEATURE_VALUE(pvr_dev, num_isp_ipp_pipes, &num_isp_store_registers);
34+ if (WARN_ON(err))
35+ return err;
36+ }
37+
38+ return sizeof(struct rogue_fwif_frag_ctx_state) +
39+ (num_isp_store_registers *
40+ sizeof(((struct rogue_fwif_frag_ctx_state *)0)->frag_reg_isp_store[0]));
41+}
42+
43+static int get_frag_ctx_state_size(struct pvr_device *pvr_dev)
44+{
45+ u32 num_isp_store_registers;
46+ int err;
47+
48+ if (PVR_HAS_FEATURE(pvr_dev, xe_memory_hierarchy)) {
49+ err = PVR_FEATURE_VALUE(pvr_dev, num_raster_pipes, &num_isp_store_registers);
50+ if (WARN_ON(err))
51+ return err;
52+
53+ if (PVR_HAS_FEATURE(pvr_dev, gpu_multicore_support)) {
54+ u32 xpu_max_slaves;
55+
56+ err = PVR_FEATURE_VALUE(pvr_dev, xpu_max_slaves, &xpu_max_slaves);
57+ if (WARN_ON(err))
58+ return err;
59+
60+ num_isp_store_registers *= (1 + xpu_max_slaves);
61+ }
62+ } else {
63+ err = PVR_FEATURE_VALUE(pvr_dev, num_isp_ipp_pipes, &num_isp_store_registers);
64+ if (WARN_ON(err))
65+ return err;
66+ }
67+
68+ return sizeof(struct rogue_fwif_frag_ctx_state) +
69+ (num_isp_store_registers *
70+ sizeof(((struct rogue_fwif_frag_ctx_state *)0)->frag_reg_isp_store[0]));
71+}
72+
73+static int get_ctx_state_size(struct pvr_device *pvr_dev, enum drm_pvr_job_type type)
74+{
75+ switch (type) {
76+ case DRM_PVR_JOB_TYPE_GEOMETRY:
77+ return sizeof(struct rogue_fwif_geom_ctx_state);
78+ case DRM_PVR_JOB_TYPE_FRAGMENT:
79+ return get_frag_ctx_state_size(pvr_dev);
80+ case DRM_PVR_JOB_TYPE_COMPUTE:
81+ return sizeof(struct rogue_fwif_compute_ctx_state);
82+ case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
83+ return get_xfer_ctx_state_size(pvr_dev);
84+ }
85+
86+ WARN(1, "Invalid queue type");
87+ return -EINVAL;
88+}
89+
90+static u32 get_ctx_offset(enum drm_pvr_job_type type)
91+{
92+ switch (type) {
93+ case DRM_PVR_JOB_TYPE_GEOMETRY:
94+ return offsetof(struct rogue_fwif_fwrendercontext, geom_context);
95+ case DRM_PVR_JOB_TYPE_FRAGMENT:
96+ return offsetof(struct rogue_fwif_fwrendercontext, frag_context);
97+ case DRM_PVR_JOB_TYPE_COMPUTE:
98+ return offsetof(struct rogue_fwif_fwcomputecontext, cdm_context);
99+ case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
100+ return offsetof(struct rogue_fwif_fwtransfercontext, tq_context);
101+ }
102+
103+ return 0;
104+}
105+
106+static const char *
107+pvr_queue_fence_get_driver_name(struct dma_fence *f)
108+{
109+ return PVR_DRIVER_NAME;
110+}
111+
112+static void pvr_queue_fence_release(struct dma_fence *f)
113+{
114+ struct pvr_queue_fence *fence = container_of(f, struct pvr_queue_fence, base);
115+
116+ pvr_context_put(fence->queue->ctx);
117+ dma_fence_free(f);
118+}
119+
120+static const char *
121+pvr_queue_job_fence_get_timeline_name(struct dma_fence *f)
122+{
123+ struct pvr_queue_fence *fence = container_of(f, struct pvr_queue_fence, base);
124+
125+ switch (fence->queue->type) {
126+ case DRM_PVR_JOB_TYPE_GEOMETRY:
127+ return "geometry";
128+
129+ case DRM_PVR_JOB_TYPE_FRAGMENT:
130+ return "fragment";
131+
132+ case DRM_PVR_JOB_TYPE_COMPUTE:
133+ return "compute";
134+
135+ case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
136+ return "transfer";
137+ }
138+
139+ WARN(1, "Invalid queue type");
140+ return "invalid";
141+}
142+
143+static const char *
144+pvr_queue_cccb_fence_get_timeline_name(struct dma_fence *f)
145+{
146+ struct pvr_queue_fence *fence = container_of(f, struct pvr_queue_fence, base);
147+
148+ switch (fence->queue->type) {
149+ case DRM_PVR_JOB_TYPE_GEOMETRY:
150+ return "geometry-cccb";
151+
152+ case DRM_PVR_JOB_TYPE_FRAGMENT:
153+ return "fragment-cccb";
154+
155+ case DRM_PVR_JOB_TYPE_COMPUTE:
156+ return "compute-cccb";
157+
158+ case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
159+ return "transfer-cccb";
160+ }
161+
162+ WARN(1, "Invalid queue type");
163+ return "invalid";
164+}
165+
166+static const struct dma_fence_ops pvr_queue_job_fence_ops = {
167+ .get_driver_name = pvr_queue_fence_get_driver_name,
168+ .get_timeline_name = pvr_queue_job_fence_get_timeline_name,
169+ .release = pvr_queue_fence_release,
170+};
171+
172+/**
173+ * to_pvr_queue_job_fence() - Return a pvr_queue_fence object if the fence is
174+ * backed by a UFO.
175+ * @f: The dma_fence to turn into a pvr_queue_fence.
176+ *
177+ * Return:
178+ * * A non-NULL pvr_queue_fence object if the dma_fence is backed by a UFO, or
179+ * * NULL otherwise.
180+ */
181+static struct pvr_queue_fence *
182+to_pvr_queue_job_fence(struct dma_fence *f)
183+{
184+ struct drm_sched_fence *sched_fence = to_drm_sched_fence(f);
185+
186+ if (sched_fence)
187+ f = sched_fence->parent;
188+
189+ if (f && f->ops == &pvr_queue_job_fence_ops)
190+ return container_of(f, struct pvr_queue_fence, base);
191+
192+ return NULL;
193+}
194+
195+static const struct dma_fence_ops pvr_queue_cccb_fence_ops = {
196+ .get_driver_name = pvr_queue_fence_get_driver_name,
197+ .get_timeline_name = pvr_queue_cccb_fence_get_timeline_name,
198+ .release = pvr_queue_fence_release,
199+};
200+
201+/**
202+ * pvr_queue_fence_put() - Put wrapper for pvr_queue_fence objects.
203+ * @f: The dma_fence object to put.
204+ *
205+ * If the pvr_queue_fence has been initialized, we call dma_fence_put(),
206+ * otherwise we free the object with dma_fence_free(). This allows us
207+ * to do the right thing before and after pvr_queue_fence_init() had been
208+ * called.
209+ */
210+static void pvr_queue_fence_put(struct dma_fence *f)
211+{
212+ if (!f)
213+ return;
214+
215+ if (WARN_ON(f->ops &&
216+ f->ops != &pvr_queue_cccb_fence_ops &&
217+ f->ops != &pvr_queue_job_fence_ops))
218+ return;
219+
220+ /* If the fence hasn't been initialized yet, free the object directly. */
221+ if (f->ops)
222+ dma_fence_put(f);
223+ else
224+ dma_fence_free(f);
225+}
226+
227+/**
228+ * pvr_queue_fence_alloc() - Allocate a pvr_queue_fence fence object
229+ *
230+ * Call this function to allocate job CCCB and done fences. This only
231+ * allocates the objects. Initialization happens when the underlying
232+ * dma_fence object is to be returned to drm_sched (in prepare_job() or
233+ * run_job()).
234+ *
235+ * Return:
236+ * * A valid pointer if the allocation succeeds, or
237+ * * NULL if the allocation fails.
238+ */
239+static struct dma_fence *
240+pvr_queue_fence_alloc(void)
241+{
242+ struct pvr_queue_fence *fence;
243+
244+ fence = kzalloc(sizeof(*fence), GFP_KERNEL);
245+ if (!fence)
246+ return NULL;
247+
248+ return &fence->base;
249+}
250+
251+/**
252+ * pvr_queue_fence_init() - Initializes a pvr_queue_fence object.
253+ * @f: The fence to initialize
254+ * @queue: The queue this fence belongs to.
255+ * @fence_ops: The fence operations.
256+ * @fence_ctx: The fence context.
257+ *
258+ * Wrapper around dma_fence_init() that takes care of initializing the
259+ * pvr_queue_fence::queue field too.
260+ */
261+static void
262+pvr_queue_fence_init(struct dma_fence *f,
263+ struct pvr_queue *queue,
264+ const struct dma_fence_ops *fence_ops,
265+ struct pvr_queue_fence_ctx *fence_ctx)
266+{
267+ struct pvr_queue_fence *fence = container_of(f, struct pvr_queue_fence, base);
268+
269+ pvr_context_get(queue->ctx);
270+ fence->queue = queue;
271+ dma_fence_init(&fence->base, fence_ops,
272+ &fence_ctx->lock, fence_ctx->id,
273+ atomic_inc_return(&fence_ctx->seqno));
274+}
275+
276+/**
277+ * pvr_queue_cccb_fence_init() - Initializes a CCCB fence object.
278+ * @fence: The fence to initialize.
279+ * @queue: The queue this fence belongs to.
280+ *
281+ * Initializes a fence that can be used to wait for CCCB space.
282+ *
283+ * Should be called in the ::prepare_job() path, so the fence returned to
284+ * drm_sched is valid.
285+ */
286+static void
287+pvr_queue_cccb_fence_init(struct dma_fence *fence, struct pvr_queue *queue)
288+{
289+ pvr_queue_fence_init(fence, queue, &pvr_queue_cccb_fence_ops,
290+ &queue->cccb_fence_ctx.base);
291+}
292+
293+/**
294+ * pvr_queue_job_fence_init() - Initializes a job done fence object.
295+ * @fence: The fence to initialize.
296+ * @queue: The queue this fence belongs to.
297+ *
298+ * Initializes a fence that will be signaled when the GPU is done executing
299+ * a job.
300+ *
301+ * Should be called *before* the ::run_job() path, so the fence is initialised
302+ * before being placed in the pending_list.
303+ */
304+static void
305+pvr_queue_job_fence_init(struct dma_fence *fence, struct pvr_queue *queue)
306+{
307+ pvr_queue_fence_init(fence, queue, &pvr_queue_job_fence_ops,
308+ &queue->job_fence_ctx);
309+}
310+
311+/**
312+ * pvr_queue_fence_ctx_init() - Queue fence context initialization.
313+ * @fence_ctx: The context to initialize
314+ */
315+static void
316+pvr_queue_fence_ctx_init(struct pvr_queue_fence_ctx *fence_ctx)
317+{
318+ spin_lock_init(&fence_ctx->lock);
319+ fence_ctx->id = dma_fence_context_alloc(1);
320+ atomic_set(&fence_ctx->seqno, 0);
321+}
322+
323+static u32 ufo_cmds_size(u32 elem_count)
324+{
325+ /* We can pass at most ROGUE_FWIF_CCB_CMD_MAX_UFOS per UFO-related command. */
326+ u32 full_cmd_count = elem_count / ROGUE_FWIF_CCB_CMD_MAX_UFOS;
327+ u32 remaining_elems = elem_count % ROGUE_FWIF_CCB_CMD_MAX_UFOS;
328+ u32 size = full_cmd_count *
329+ pvr_cccb_get_size_of_cmd_with_hdr(ROGUE_FWIF_CCB_CMD_MAX_UFOS *
330+ sizeof(struct rogue_fwif_ufo));
331+
332+ if (remaining_elems) {
333+ size += pvr_cccb_get_size_of_cmd_with_hdr(remaining_elems *
334+ sizeof(struct rogue_fwif_ufo));
335+ }
336+
337+ return size;
338+}
339+
340+static u32 job_cmds_size(struct pvr_job *job, u32 ufo_wait_count)
341+{
342+ /* One UFO cmd for the fence signaling, one UFO cmd per native fence native,
343+ * and a command for the job itself.
344+ */
345+ return ufo_cmds_size(1) + ufo_cmds_size(ufo_wait_count) +
346+ pvr_cccb_get_size_of_cmd_with_hdr(job->cmd_len);
347+}
348+
349+/**
350+ * job_count_remaining_native_deps() - Count the number of non-signaled native dependencies.
351+ * @job: Job to operate on.
352+ *
353+ * Returns: Number of non-signaled native deps remaining.
354+ */
355+static unsigned long job_count_remaining_native_deps(struct pvr_job *job)
356+{
357+ unsigned long remaining_count = 0;
358+ struct dma_fence *fence = NULL;
359+ unsigned long index;
360+
361+ xa_for_each(&job->base.dependencies, index, fence) {
362+ struct pvr_queue_fence *jfence;
363+
364+ jfence = to_pvr_queue_job_fence(fence);
365+ if (!jfence)
366+ continue;
367+
368+ if (!dma_fence_is_signaled(&jfence->base))
369+ remaining_count++;
370+ }
371+
372+ return remaining_count;
373+}
374+
375+/**
376+ * pvr_queue_get_job_cccb_fence() - Get the CCCB fence attached to a job.
377+ * @queue: The queue this job will be submitted to.
378+ * @job: The job to get the CCCB fence on.
379+ *
380+ * The CCCB fence is a synchronization primitive allowing us to delay job
381+ * submission until there's enough space in the CCCB to submit the job.
382+ *
383+ * Return:
384+ * * NULL if there's enough space in the CCCB to submit this job, or
385+ * * A valid dma_fence object otherwise.
386+ */
387+static struct dma_fence *
388+pvr_queue_get_job_cccb_fence(struct pvr_queue *queue, struct pvr_job *job)
389+{
390+ struct pvr_queue_fence *cccb_fence;
391+ unsigned int native_deps_remaining;
392+
393+ /* If the fence is NULL, that means we already checked that we had
394+ * enough space in the cccb for our job.
395+ */
396+ if (!job->cccb_fence)
397+ return NULL;
398+
399+ mutex_lock(&queue->cccb_fence_ctx.job_lock);
400+
401+ /* Count remaining native dependencies and check if the job fits in the CCCB. */
402+ native_deps_remaining = job_count_remaining_native_deps(job);
403+ if (pvr_cccb_cmdseq_fits(&queue->cccb, job_cmds_size(job, native_deps_remaining))) {
404+ pvr_queue_fence_put(job->cccb_fence);
405+ job->cccb_fence = NULL;
406+ goto out_unlock;
407+ }
408+
409+ /* There should be no job attached to the CCCB fence context:
410+ * drm_sched_entity guarantees that jobs are submitted one at a time.
411+ */
412+ if (WARN_ON(queue->cccb_fence_ctx.job))
413+ pvr_job_put(queue->cccb_fence_ctx.job);
414+
415+ queue->cccb_fence_ctx.job = pvr_job_get(job);
416+
417+ /* Initialize the fence before returning it. */
418+ cccb_fence = container_of(job->cccb_fence, struct pvr_queue_fence, base);
419+ if (!WARN_ON(cccb_fence->queue))
420+ pvr_queue_cccb_fence_init(job->cccb_fence, queue);
421+
422+out_unlock:
423+ mutex_unlock(&queue->cccb_fence_ctx.job_lock);
424+
425+ return dma_fence_get(job->cccb_fence);
426+}
427+
428+/**
429+ * pvr_queue_get_job_kccb_fence() - Get the KCCB fence attached to a job.
430+ * @queue: The queue this job will be submitted to.
431+ * @job: The job to get the KCCB fence on.
432+ *
433+ * The KCCB fence is a synchronization primitive allowing us to delay job
434+ * submission until there's enough space in the KCCB to submit the job.
435+ *
436+ * Return:
437+ * * NULL if there's enough space in the KCCB to submit this job, or
438+ * * A valid dma_fence object otherwise.
439+ */
440+static struct dma_fence *
441+pvr_queue_get_job_kccb_fence(struct pvr_queue *queue, struct pvr_job *job)
442+{
443+ struct pvr_device *pvr_dev = queue->ctx->pvr_dev;
444+ struct dma_fence *kccb_fence = NULL;
445+
446+ /* If the fence is NULL, that means we already checked that we had
447+ * enough space in the KCCB for our job.
448+ */
449+ if (!job->kccb_fence)
450+ return NULL;
451+
452+ if (!WARN_ON(job->kccb_fence->ops)) {
453+ kccb_fence = pvr_kccb_reserve_slot(pvr_dev, job->kccb_fence);
454+ job->kccb_fence = NULL;
455+ }
456+
457+ return kccb_fence;
458+}
459+
460+static struct dma_fence *
461+pvr_queue_get_paired_frag_job_dep(struct pvr_queue *queue, struct pvr_job *job)
462+{
463+ struct pvr_job *frag_job = job->type == DRM_PVR_JOB_TYPE_GEOMETRY ?
464+ job->paired_job : NULL;
465+ struct dma_fence *f;
466+ unsigned long index;
467+
468+ if (!frag_job)
469+ return NULL;
470+
471+ xa_for_each(&frag_job->base.dependencies, index, f) {
472+ /* Skip already signaled fences. */
473+ if (dma_fence_is_signaled(f))
474+ continue;
475+
476+ /* Skip our own fence. */
477+ if (f == &job->base.s_fence->scheduled)
478+ continue;
479+
480+ return dma_fence_get(f);
481+ }
482+
483+ return frag_job->base.sched->ops->prepare_job(&frag_job->base, &queue->entity);
484+}
485+
486+/**
487+ * pvr_queue_prepare_job() - Return the next internal dependencies expressed as a dma_fence.
488+ * @sched_job: The job to query the next internal dependency on
489+ * @s_entity: The entity this job is queue on.
490+ *
491+ * After iterating over drm_sched_job::dependencies, drm_sched let the driver return
492+ * its own internal dependencies. We use this function to return our internal dependencies.
493+ */
494+static struct dma_fence *
495+pvr_queue_prepare_job(struct drm_sched_job *sched_job,
496+ struct drm_sched_entity *s_entity)
497+{
498+ struct pvr_job *job = container_of(sched_job, struct pvr_job, base);
499+ struct pvr_queue *queue = container_of(s_entity, struct pvr_queue, entity);
500+ struct dma_fence *internal_dep = NULL;
501+
502+ /*
503+ * Initialize the done_fence, so we can signal it. This must be done
504+ * here because otherwise by the time of run_job() the job will end up
505+ * in the pending list without a valid fence.
506+ */
507+ if (job->type == DRM_PVR_JOB_TYPE_FRAGMENT && job->paired_job) {
508+ /*
509+ * This will be called on a paired fragment job after being
510+ * submitted to firmware. We can tell if this is the case and
511+ * bail early from whether run_job() has been called on the
512+ * geometry job, which would issue a pm ref.
513+ */
514+ if (job->paired_job->has_pm_ref)
515+ return NULL;
516+
517+ /*
518+ * In this case we need to use the job's own ctx to initialise
519+ * the done_fence. The other steps are done in the ctx of the
520+ * paired geometry job.
521+ */
522+ pvr_queue_job_fence_init(job->done_fence,
523+ job->ctx->queues.fragment);
524+ } else {
525+ pvr_queue_job_fence_init(job->done_fence, queue);
526+ }
527+
528+ /* CCCB fence is used to make sure we have enough space in the CCCB to
529+ * submit our commands.
530+ */
531+ internal_dep = pvr_queue_get_job_cccb_fence(queue, job);
532+
533+ /* KCCB fence is used to make sure we have a KCCB slot to queue our
534+ * CMD_KICK.
535+ */
536+ if (!internal_dep)
537+ internal_dep = pvr_queue_get_job_kccb_fence(queue, job);
538+
539+ /* Any extra internal dependency should be added here, using the following
540+ * pattern:
541+ *
542+ * if (!internal_dep)
543+ * internal_dep = pvr_queue_get_job_xxxx_fence(queue, job);
544+ */
545+
546+ /* The paired job fence should come last, when everything else is ready. */
547+ if (!internal_dep)
548+ internal_dep = pvr_queue_get_paired_frag_job_dep(queue, job);
549+
550+ return internal_dep;
551+}
552+
553+/**
554+ * pvr_queue_update_active_state_locked() - Update the queue active state.
555+ * @queue: Queue to update the state on.
556+ *
557+ * Locked version of pvr_queue_update_active_state(). Must be called with
558+ * pvr_device::queue::lock held.
559+ */
560+static void pvr_queue_update_active_state_locked(struct pvr_queue *queue)
561+{
562+ struct pvr_device *pvr_dev = queue->ctx->pvr_dev;
563+
564+ lockdep_assert_held(&pvr_dev->queues.lock);
565+
566+ /* The queue is temporary out of any list when it's being reset,
567+ * we don't want a call to pvr_queue_update_active_state_locked()
568+ * to re-insert it behind our back.
569+ */
570+ if (list_empty(&queue->node))
571+ return;
572+
573+ if (!atomic_read(&queue->in_flight_job_count))
574+ list_move_tail(&queue->node, &pvr_dev->queues.idle);
575+ else
576+ list_move_tail(&queue->node, &pvr_dev->queues.active);
577+}
578+
579+/**
580+ * pvr_queue_update_active_state() - Update the queue active state.
581+ * @queue: Queue to update the state on.
582+ *
583+ * Active state is based on the in_flight_job_count value.
584+ *
585+ * Updating the active state implies moving the queue in or out of the
586+ * active queue list, which also defines whether the queue is checked
587+ * or not when a FW event is received.
588+ *
589+ * This function should be called any time a job is submitted or it done
590+ * fence is signaled.
591+ */
592+static void pvr_queue_update_active_state(struct pvr_queue *queue)
593+{
594+ struct pvr_device *pvr_dev = queue->ctx->pvr_dev;
595+
596+ mutex_lock(&pvr_dev->queues.lock);
597+ pvr_queue_update_active_state_locked(queue);
598+ mutex_unlock(&pvr_dev->queues.lock);
599+}
600+
601+static void pvr_queue_submit_job_to_cccb(struct pvr_job *job)
602+{
603+ struct pvr_queue *queue = container_of(job->base.sched, struct pvr_queue, scheduler);
604+ struct rogue_fwif_ufo ufos[ROGUE_FWIF_CCB_CMD_MAX_UFOS];
605+ struct pvr_cccb *cccb = &queue->cccb;
606+ struct pvr_queue_fence *jfence;
607+ struct dma_fence *fence;
608+ unsigned long index;
609+ u32 ufo_count = 0;
610+
611+ /* We need to add the queue to the active list before updating the CCCB,
612+ * otherwise we might miss the FW event informing us that something
613+ * happened on this queue.
614+ */
615+ atomic_inc(&queue->in_flight_job_count);
616+ pvr_queue_update_active_state(queue);
617+
618+ xa_for_each(&job->base.dependencies, index, fence) {
619+ jfence = to_pvr_queue_job_fence(fence);
620+ if (!jfence)
621+ continue;
622+
623+ /* Skip the partial render fence, we will place it at the end. */
624+ if (job->type == DRM_PVR_JOB_TYPE_FRAGMENT && job->paired_job &&
625+ &job->paired_job->base.s_fence->scheduled == fence)
626+ continue;
627+
628+ if (dma_fence_is_signaled(&jfence->base))
629+ continue;
630+
631+ pvr_fw_object_get_fw_addr(jfence->queue->timeline_ufo.fw_obj,
632+ &ufos[ufo_count].addr);
633+ ufos[ufo_count++].value = jfence->base.seqno;
634+
635+ if (ufo_count == ARRAY_SIZE(ufos)) {
636+ pvr_cccb_write_command_with_header(cccb, ROGUE_FWIF_CCB_CMD_TYPE_FENCE_PR,
637+ sizeof(ufos), ufos, 0, 0);
638+ ufo_count = 0;
639+ }
640+ }
641+
642+ /* Partial render fence goes last. */
643+ if (job->type == DRM_PVR_JOB_TYPE_FRAGMENT && job->paired_job) {
644+ jfence = to_pvr_queue_job_fence(job->paired_job->done_fence);
645+ if (!WARN_ON(!jfence)) {
646+ pvr_fw_object_get_fw_addr(jfence->queue->timeline_ufo.fw_obj,
647+ &ufos[ufo_count].addr);
648+ ufos[ufo_count++].value = job->paired_job->done_fence->seqno;
649+ }
650+ }
651+
652+ if (ufo_count) {
653+ pvr_cccb_write_command_with_header(cccb, ROGUE_FWIF_CCB_CMD_TYPE_FENCE_PR,
654+ sizeof(ufos[0]) * ufo_count, ufos, 0, 0);
655+ }
656+
657+ if (job->type == DRM_PVR_JOB_TYPE_GEOMETRY && job->paired_job) {
658+ struct rogue_fwif_cmd_geom *cmd = job->cmd;
659+
660+ /* Reference value for the partial render test is the current queue fence
661+ * seqno minus one.
662+ */
663+ pvr_fw_object_get_fw_addr(queue->timeline_ufo.fw_obj,
664+ &cmd->partial_render_geom_frag_fence.addr);
665+ cmd->partial_render_geom_frag_fence.value = job->done_fence->seqno - 1;
666+ }
667+
668+ /* Submit job to FW */
669+ pvr_cccb_write_command_with_header(cccb, job->fw_ccb_cmd_type, job->cmd_len, job->cmd,
670+ job->id, job->id);
671+
672+ /* Signal the job fence. */
673+ pvr_fw_object_get_fw_addr(queue->timeline_ufo.fw_obj, &ufos[0].addr);
674+ ufos[0].value = job->done_fence->seqno;
675+ pvr_cccb_write_command_with_header(cccb, ROGUE_FWIF_CCB_CMD_TYPE_UPDATE,
676+ sizeof(ufos[0]), ufos, 0, 0);
677+}
678+
679+/**
680+ * pvr_queue_run_job() - Submit a job to the FW.
681+ * @sched_job: The job to submit.
682+ *
683+ * This function is called when all non-native dependencies have been met and
684+ * when the commands resulting from this job are guaranteed to fit in the CCCB.
685+ */
686+static struct dma_fence *pvr_queue_run_job(struct drm_sched_job *sched_job)
687+{
688+ struct pvr_job *job = container_of(sched_job, struct pvr_job, base);
689+ struct pvr_device *pvr_dev = job->pvr_dev;
690+ int err;
691+
692+ /* The fragment job is issued along the geometry job when we use combined
693+ * geom+frag kicks. When we get there, we should simply return the
694+ * done_fence that's been initialized earlier.
695+ */
696+ if (job->paired_job && job->type == DRM_PVR_JOB_TYPE_FRAGMENT &&
697+ job->done_fence->ops) {
698+ return dma_fence_get(job->done_fence);
699+ }
700+
701+ /* The only kind of jobs that can be paired are geometry and fragment, and
702+ * we bail out early if we see a fragment job that's paired with a geomtry
703+ * job.
704+ * Paired jobs must also target the same context and point to the same
705+ * HWRT.
706+ */
707+ if (WARN_ON(job->paired_job &&
708+ (job->type != DRM_PVR_JOB_TYPE_GEOMETRY ||
709+ job->paired_job->type != DRM_PVR_JOB_TYPE_FRAGMENT ||
710+ job->hwrt != job->paired_job->hwrt ||
711+ job->ctx != job->paired_job->ctx)))
712+ return ERR_PTR(-EINVAL);
713+
714+ err = pvr_job_get_pm_ref(job);
715+ if (WARN_ON(err))
716+ return ERR_PTR(err);
717+
718+ if (job->paired_job) {
719+ err = pvr_job_get_pm_ref(job->paired_job);
720+ if (WARN_ON(err))
721+ return ERR_PTR(err);
722+ }
723+
724+ /* Submit our job to the CCCB */
725+ pvr_queue_submit_job_to_cccb(job);
726+
727+ if (job->paired_job) {
728+ struct pvr_job *geom_job = job;
729+ struct pvr_job *frag_job = job->paired_job;
730+ struct pvr_queue *geom_queue = job->ctx->queues.geometry;
731+ struct pvr_queue *frag_queue = job->ctx->queues.fragment;
732+
733+ /* Submit the fragment job along the geometry job and send a combined kick. */
734+ pvr_queue_submit_job_to_cccb(frag_job);
735+ pvr_cccb_send_kccb_combined_kick(pvr_dev,
736+ &geom_queue->cccb, &frag_queue->cccb,
737+ pvr_context_get_fw_addr(geom_job->ctx) +
738+ geom_queue->ctx_offset,
739+ pvr_context_get_fw_addr(frag_job->ctx) +
740+ frag_queue->ctx_offset,
741+ job->hwrt,
742+ frag_job->fw_ccb_cmd_type ==
743+ ROGUE_FWIF_CCB_CMD_TYPE_FRAG_PR);
744+ } else {
745+ struct pvr_queue *queue = container_of(job->base.sched,
746+ struct pvr_queue, scheduler);
747+
748+ pvr_cccb_send_kccb_kick(pvr_dev, &queue->cccb,
749+ pvr_context_get_fw_addr(job->ctx) + queue->ctx_offset,
750+ job->hwrt);
751+ }
752+
753+ return dma_fence_get(job->done_fence);
754+}
755+
756+static void pvr_queue_stop(struct pvr_queue *queue, struct pvr_job *bad_job)
757+{
758+ drm_sched_stop(&queue->scheduler, bad_job ? &bad_job->base : NULL);
759+}
760+
761+static void pvr_queue_start(struct pvr_queue *queue)
762+{
763+ struct pvr_job *job;
764+
765+ /* Make sure we CPU-signal the UFO object, so other queues don't get
766+ * blocked waiting on it.
767+ */
768+ *queue->timeline_ufo.value = atomic_read(&queue->job_fence_ctx.seqno);
769+
770+ list_for_each_entry(job, &queue->scheduler.pending_list, base.list) {
771+ if (dma_fence_is_signaled(job->done_fence)) {
772+ /* Jobs might have completed after drm_sched_stop() was called.
773+ * In that case, re-assign the parent field to the done_fence.
774+ */
775+ WARN_ON(job->base.s_fence->parent);
776+ job->base.s_fence->parent = dma_fence_get(job->done_fence);
777+ } else {
778+ /* If we had unfinished jobs, flag the entity as guilty so no
779+ * new job can be submitted.
780+ */
781+ atomic_set(&queue->ctx->faulty, 1);
782+ }
783+ }
784+
785+ drm_sched_start(&queue->scheduler, true);
786+}
787+
788+/**
789+ * pvr_queue_timedout_job() - Handle a job timeout event.
790+ * @s_job: The job this timeout occurred on.
791+ *
792+ * FIXME: We don't do anything here to unblock the situation, we just stop+start
793+ * the scheduler, and re-assign parent fences in the middle.
794+ *
795+ * Return:
796+ * * DRM_GPU_SCHED_STAT_NOMINAL.
797+ */
798+static enum drm_gpu_sched_stat
799+pvr_queue_timedout_job(struct drm_sched_job *s_job)
800+{
801+ struct drm_gpu_scheduler *sched = s_job->sched;
802+ struct pvr_queue *queue = container_of(sched, struct pvr_queue, scheduler);
803+ struct pvr_device *pvr_dev = queue->ctx->pvr_dev;
804+ struct pvr_job *job;
805+ u32 job_count = 0;
806+
807+ dev_err(sched->dev, "Job timeout\n");
808+
809+ /* Before we stop the scheduler, make sure the queue is out of any list, so
810+ * any call to pvr_queue_update_active_state_locked() that might happen
811+ * until the scheduler is really stopped doesn't end up re-inserting the
812+ * queue in the active list. This would cause
813+ * pvr_queue_signal_done_fences() and drm_sched_stop() to race with each
814+ * other when accessing the pending_list, since drm_sched_stop() doesn't
815+ * grab the job_list_lock when modifying the list (it's assuming the
816+ * only other accessor is the scheduler, and it's safe to not grab the
817+ * lock since it's stopped).
818+ */
819+ mutex_lock(&pvr_dev->queues.lock);
820+ list_del_init(&queue->node);
821+ mutex_unlock(&pvr_dev->queues.lock);
822+
823+ drm_sched_stop(sched, s_job);
824+
825+ /* Re-assign job parent fences. */
826+ list_for_each_entry(job, &sched->pending_list, base.list) {
827+ job->base.s_fence->parent = dma_fence_get(job->done_fence);
828+ job_count++;
829+ }
830+ WARN_ON(atomic_read(&queue->in_flight_job_count) != job_count);
831+
832+ /* Re-insert the queue in the proper list, and kick a queue processing
833+ * operation if there were jobs pending.
834+ */
835+ mutex_lock(&pvr_dev->queues.lock);
836+ if (!job_count) {
837+ list_move_tail(&queue->node, &pvr_dev->queues.idle);
838+ } else {
839+ atomic_set(&queue->in_flight_job_count, job_count);
840+ list_move_tail(&queue->node, &pvr_dev->queues.active);
841+ pvr_queue_process(queue);
842+ }
843+ mutex_unlock(&pvr_dev->queues.lock);
844+
845+ drm_sched_start(sched, true);
846+
847+ return DRM_GPU_SCHED_STAT_NOMINAL;
848+}
849+
850+/**
851+ * pvr_queue_free_job() - Release the reference the scheduler had on a job object.
852+ * @sched_job: Job object to free.
853+ */
854+static void pvr_queue_free_job(struct drm_sched_job *sched_job)
855+{
856+ struct pvr_job *job = container_of(sched_job, struct pvr_job, base);
857+
858+ drm_sched_job_cleanup(sched_job);
859+ job->paired_job = NULL;
860+ pvr_job_put(job);
861+}
862+
863+static const struct drm_sched_backend_ops pvr_queue_sched_ops = {
864+ .prepare_job = pvr_queue_prepare_job,
865+ .run_job = pvr_queue_run_job,
866+ .timedout_job = pvr_queue_timedout_job,
867+ .free_job = pvr_queue_free_job,
868+};
869+
870+/**
871+ * pvr_queue_fence_is_ufo_backed() - Check if a dma_fence is backed by a UFO object
872+ * @f: Fence to test.
873+ *
874+ * A UFO-backed fence is a fence that can be signaled or waited upon FW-side.
875+ * pvr_job::done_fence objects are backed by the timeline UFO attached to the queue
876+ * they are pushed to, but those fences are not directly exposed to the outside
877+ * world, so we also need to check if the fence we're being passed is a
878+ * drm_sched_fence that was coming from our driver.
879+ */
880+bool pvr_queue_fence_is_ufo_backed(struct dma_fence *f)
881+{
882+ struct drm_sched_fence *sched_fence = f ? to_drm_sched_fence(f) : NULL;
883+
884+ if (sched_fence &&
885+ sched_fence->sched->ops == &pvr_queue_sched_ops)
886+ return true;
887+
888+ if (f && f->ops == &pvr_queue_job_fence_ops)
889+ return true;
890+
891+ return false;
892+}
893+
894+/**
895+ * pvr_queue_signal_done_fences() - Signal done fences.
896+ * @queue: Queue to check.
897+ *
898+ * Signal done fences of jobs whose seqno is less than the current value of
899+ * the UFO object attached to the queue.
900+ */
901+static void
902+pvr_queue_signal_done_fences(struct pvr_queue *queue)
903+{
904+ struct pvr_job *job, *tmp_job;
905+ u32 cur_seqno;
906+
907+ spin_lock(&queue->scheduler.job_list_lock);
908+ cur_seqno = *queue->timeline_ufo.value;
909+ list_for_each_entry_safe(job, tmp_job, &queue->scheduler.pending_list, base.list) {
910+ if ((int)(cur_seqno - lower_32_bits(job->done_fence->seqno)) < 0)
911+ break;
912+
913+ if (!dma_fence_is_signaled(job->done_fence)) {
914+ dma_fence_signal(job->done_fence);
915+ pvr_job_release_pm_ref(job);
916+ atomic_dec(&queue->in_flight_job_count);
917+ }
918+ }
919+ spin_unlock(&queue->scheduler.job_list_lock);
920+}
921+
922+/**
923+ * pvr_queue_check_job_waiting_for_cccb_space() - Check if the job waiting for CCCB space
924+ * can be unblocked
925+ * pushed to the CCCB
926+ * @queue: Queue to check
927+ *
928+ * If we have a job waiting for CCCB, and this job now fits in the CCCB, we signal
929+ * its CCCB fence, which should kick drm_sched.
930+ */
931+static void
932+pvr_queue_check_job_waiting_for_cccb_space(struct pvr_queue *queue)
933+{
934+ struct pvr_queue_fence *cccb_fence;
935+ u32 native_deps_remaining;
936+ struct pvr_job *job;
937+
938+ mutex_lock(&queue->cccb_fence_ctx.job_lock);
939+ job = queue->cccb_fence_ctx.job;
940+ if (!job)
941+ goto out_unlock;
942+
943+ /* If we have a job attached to the CCCB fence context, its CCCB fence
944+ * shouldn't be NULL.
945+ */
946+ if (WARN_ON(!job->cccb_fence)) {
947+ job = NULL;
948+ goto out_unlock;
949+ }
950+
951+ /* If we get there, CCCB fence has to be initialized. */
952+ cccb_fence = container_of(job->cccb_fence, struct pvr_queue_fence, base);
953+ if (WARN_ON(!cccb_fence->queue)) {
954+ job = NULL;
955+ goto out_unlock;
956+ }
957+
958+ /* Evict signaled dependencies before checking for CCCB space.
959+ * If the job fits, signal the CCCB fence, this should unblock
960+ * the drm_sched_entity.
961+ */
962+ native_deps_remaining = job_count_remaining_native_deps(job);
963+ if (!pvr_cccb_cmdseq_fits(&queue->cccb, job_cmds_size(job, native_deps_remaining))) {
964+ job = NULL;
965+ goto out_unlock;
966+ }
967+
968+ dma_fence_signal(job->cccb_fence);
969+ pvr_queue_fence_put(job->cccb_fence);
970+ job->cccb_fence = NULL;
971+ queue->cccb_fence_ctx.job = NULL;
972+
973+out_unlock:
974+ mutex_unlock(&queue->cccb_fence_ctx.job_lock);
975+
976+ pvr_job_put(job);
977+}
978+
979+/**
980+ * pvr_queue_process() - Process events that happened on a queue.
981+ * @queue: Queue to check
982+ *
983+ * Signal job fences and check if jobs waiting for CCCB space can be unblocked.
984+ */
985+void pvr_queue_process(struct pvr_queue *queue)
986+{
987+ lockdep_assert_held(&queue->ctx->pvr_dev->queues.lock);
988+
989+ pvr_queue_check_job_waiting_for_cccb_space(queue);
990+ pvr_queue_signal_done_fences(queue);
991+ pvr_queue_update_active_state_locked(queue);
992+}
993+
994+static u32 get_dm_type(struct pvr_queue *queue)
995+{
996+ switch (queue->type) {
997+ case DRM_PVR_JOB_TYPE_GEOMETRY:
998+ return PVR_FWIF_DM_GEOM;
999+ case DRM_PVR_JOB_TYPE_TRANSFER_FRAG:
1000+ case DRM_PVR_JOB_TYPE_FRAGMENT:
1001+ return PVR_FWIF_DM_FRAG;
1002+ case DRM_PVR_JOB_TYPE_COMPUTE:
1003+ return PVR_FWIF_DM_CDM;
1004+ }
1005+
1006+ return ~0;
1007+}
1008+
1009+/**
1010+ * init_fw_context() - Initializes the queue part of a FW context.
1011+ * @queue: Queue object to initialize the FW context for.
1012+ * @fw_ctx_map: The FW context CPU mapping.
1013+ *
1014+ * FW contexts are containing various states, one of them being a per-queue state
1015+ * that needs to be initialized for each queue being exposed by a context. This
1016+ * function takes care of that.
1017+ */
1018+static void init_fw_context(struct pvr_queue *queue, void *fw_ctx_map)
1019+{
1020+ struct pvr_context *ctx = queue->ctx;
1021+ struct pvr_fw_object *fw_mem_ctx_obj = pvr_vm_get_fw_mem_context(ctx->vm_ctx);
1022+ struct rogue_fwif_fwcommoncontext *cctx_fw;
1023+ struct pvr_cccb *cccb = &queue->cccb;
1024+
1025+ cctx_fw = fw_ctx_map + queue->ctx_offset;
1026+ cctx_fw->ccbctl_fw_addr = cccb->ctrl_fw_addr;
1027+ cctx_fw->ccb_fw_addr = cccb->cccb_fw_addr;
1028+
1029+ cctx_fw->dm = get_dm_type(queue);
1030+ cctx_fw->priority = ctx->priority;
1031+ cctx_fw->priority_seq_num = 0;
1032+ cctx_fw->max_deadline_ms = MAX_DEADLINE_MS;
1033+ cctx_fw->pid = task_tgid_nr(current);
1034+ cctx_fw->server_common_context_id = ctx->ctx_id;
1035+
1036+ pvr_fw_object_get_fw_addr(fw_mem_ctx_obj, &cctx_fw->fw_mem_context_fw_addr);
1037+
1038+ pvr_fw_object_get_fw_addr(queue->reg_state_obj, &cctx_fw->context_state_addr);
1039+}
1040+
1041+/**
1042+ * pvr_queue_cleanup_fw_context() - Wait for the FW context to be idle and clean it up.
1043+ * @queue: Queue on FW context to clean up.
1044+ *
1045+ * Return:
1046+ * * 0 on success,
1047+ * * Any error returned by pvr_fw_structure_cleanup() otherwise.
1048+ */
1049+static int pvr_queue_cleanup_fw_context(struct pvr_queue *queue)
1050+{
1051+ if (!queue->ctx->fw_obj)
1052+ return 0;
1053+
1054+ return pvr_fw_structure_cleanup(queue->ctx->pvr_dev,
1055+ ROGUE_FWIF_CLEANUP_FWCOMMONCONTEXT,
1056+ queue->ctx->fw_obj, queue->ctx_offset);
1057+}
1058+
1059+/**
1060+ * pvr_queue_job_init() - Initialize queue related fields in a pvr_job object.
1061+ * @job: The job to initialize.
1062+ *
1063+ * Bind the job to a queue and allocate memory to guarantee pvr_queue_job_arm()
1064+ * and pvr_queue_job_push() can't fail. We also make sure the context type is
1065+ * valid and the job can fit in the CCCB.
1066+ *
1067+ * Return:
1068+ * * 0 on success, or
1069+ * * An error code if something failed.
1070+ */
1071+int pvr_queue_job_init(struct pvr_job *job)
1072+{
1073+ /* Fragment jobs need at least one native fence wait on the geometry job fence. */
1074+ u32 min_native_dep_count = job->type == DRM_PVR_JOB_TYPE_FRAGMENT ? 1 : 0;
1075+ struct pvr_queue *queue;
1076+ int err;
1077+
1078+ if (atomic_read(&job->ctx->faulty))
1079+ return -EIO;
1080+
1081+ queue = pvr_context_get_queue_for_job(job->ctx, job->type);
1082+ if (!queue)
1083+ return -EINVAL;
1084+
1085+ if (!pvr_cccb_cmdseq_can_fit(&queue->cccb, job_cmds_size(job, min_native_dep_count)))
1086+ return -E2BIG;
1087+
1088+ err = drm_sched_job_init(&job->base, &queue->entity, 1, THIS_MODULE);
1089+ if (err)
1090+ return err;
1091+
1092+ job->cccb_fence = pvr_queue_fence_alloc();
1093+ job->kccb_fence = pvr_kccb_fence_alloc();
1094+ job->done_fence = pvr_queue_fence_alloc();
1095+ if (!job->cccb_fence || !job->kccb_fence || !job->done_fence)
1096+ return -ENOMEM;
1097+
1098+ return 0;
1099+}
1100+
1101+/**
1102+ * pvr_queue_job_arm() - Arm a job object.
1103+ * @job: The job to arm.
1104+ *
1105+ * Initializes fences and return the drm_sched finished fence so it can
1106+ * be exposed to the outside world. Once this function is called, you should
1107+ * make sure the job is pushed using pvr_queue_job_push(), or guarantee that
1108+ * no one grabbed a reference to the returned fence. The latter can happen if
1109+ * we do multi-job submission, and something failed when creating/initializing
1110+ * a job. In that case, we know the fence didn't leave the driver, and we
1111+ * can thus guarantee nobody will wait on an dead fence object.
1112+ *
1113+ * Return:
1114+ * * A dma_fence object.
1115+ */
1116+struct dma_fence *pvr_queue_job_arm(struct pvr_job *job)
1117+{
1118+ drm_sched_job_arm(&job->base);
1119+
1120+ return &job->base.s_fence->finished;
1121+}
1122+
1123+/**
1124+ * pvr_queue_job_cleanup() - Cleanup fence/scheduler related fields in the job object.
1125+ * @job: The job to cleanup.
1126+ *
1127+ * Should be called in the job release path.
1128+ */
1129+void pvr_queue_job_cleanup(struct pvr_job *job)
1130+{
1131+ pvr_queue_fence_put(job->done_fence);
1132+ pvr_queue_fence_put(job->cccb_fence);
1133+ pvr_kccb_fence_put(job->kccb_fence);
1134+
1135+ if (job->base.s_fence)
1136+ drm_sched_job_cleanup(&job->base);
1137+}
1138+
1139+/**
1140+ * pvr_queue_job_push() - Push a job to its queue.
1141+ * @job: The job to push.
1142+ *
1143+ * Must be called after pvr_queue_job_init() and after all dependencies
1144+ * have been added to the job. This will effectively queue the job to
1145+ * the drm_sched_entity attached to the queue. We grab a reference on
1146+ * the job object, so the caller is free to drop its reference when it's
1147+ * done accessing the job object.
1148+ */
1149+void pvr_queue_job_push(struct pvr_job *job)
1150+{
1151+ struct pvr_queue *queue = container_of(job->base.sched, struct pvr_queue, scheduler);
1152+
1153+ /* Keep track of the last queued job scheduled fence for combined submit. */
1154+ dma_fence_put(queue->last_queued_job_scheduled_fence);
1155+ queue->last_queued_job_scheduled_fence = dma_fence_get(&job->base.s_fence->scheduled);
1156+
1157+ pvr_job_get(job);
1158+ drm_sched_entity_push_job(&job->base);
1159+}
1160+
1161+static void reg_state_init(void *cpu_ptr, void *priv)
1162+{
1163+ struct pvr_queue *queue = priv;
1164+
1165+ if (queue->type == DRM_PVR_JOB_TYPE_GEOMETRY) {
1166+ struct rogue_fwif_geom_ctx_state *geom_ctx_state_fw = cpu_ptr;
1167+
1168+ geom_ctx_state_fw->geom_core[0].geom_reg_vdm_call_stack_pointer_init =
1169+ queue->callstack_addr;
1170+ }
1171+}
1172+
1173+/**
1174+ * pvr_queue_create() - Create a queue object.
1175+ * @ctx: The context this queue will be attached to.
1176+ * @type: The type of jobs being pushed to this queue.
1177+ * @args: The arguments passed to the context creation function.
1178+ * @fw_ctx_map: CPU mapping of the FW context object.
1179+ *
1180+ * Create a queue object that will be used to queue and track jobs.
1181+ *
1182+ * Return:
1183+ * * A valid pointer to a pvr_queue object, or
1184+ * * An error pointer if the creation/initialization failed.
1185+ */
1186+struct pvr_queue *pvr_queue_create(struct pvr_context *ctx,
1187+ enum drm_pvr_job_type type,
1188+ struct drm_pvr_ioctl_create_context_args *args,
1189+ void *fw_ctx_map)
1190+{
1191+ static const struct {
1192+ u32 cccb_size;
1193+ const char *name;
1194+ } props[] = {
1195+ [DRM_PVR_JOB_TYPE_GEOMETRY] = {
1196+ .cccb_size = CTX_GEOM_CCCB_SIZE_LOG2,
1197+ .name = "geometry",
1198+ },
1199+ [DRM_PVR_JOB_TYPE_FRAGMENT] = {
1200+ .cccb_size = CTX_FRAG_CCCB_SIZE_LOG2,
1201+ .name = "fragment"
1202+ },
1203+ [DRM_PVR_JOB_TYPE_COMPUTE] = {
1204+ .cccb_size = CTX_COMPUTE_CCCB_SIZE_LOG2,
1205+ .name = "compute"
1206+ },
1207+ [DRM_PVR_JOB_TYPE_TRANSFER_FRAG] = {
1208+ .cccb_size = CTX_TRANSFER_CCCB_SIZE_LOG2,
1209+ .name = "transfer_frag"
1210+ },
1211+ };
1212+ struct pvr_device *pvr_dev = ctx->pvr_dev;
1213+ struct drm_gpu_scheduler *sched;
1214+ struct pvr_queue *queue;
1215+ int ctx_state_size, err;
1216+ void *cpu_map;
1217+
1218+ if (WARN_ON(type >= sizeof(props)))
1219+ return ERR_PTR(-EINVAL);
1220+
1221+ switch (ctx->type) {
1222+ case DRM_PVR_CTX_TYPE_RENDER:
1223+ if (type != DRM_PVR_JOB_TYPE_GEOMETRY &&
1224+ type != DRM_PVR_JOB_TYPE_FRAGMENT)
1225+ return ERR_PTR(-EINVAL);
1226+ break;
1227+ case DRM_PVR_CTX_TYPE_COMPUTE:
1228+ if (type != DRM_PVR_JOB_TYPE_COMPUTE)
1229+ return ERR_PTR(-EINVAL);
1230+ break;
1231+ case DRM_PVR_CTX_TYPE_TRANSFER_FRAG:
1232+ if (type != DRM_PVR_JOB_TYPE_TRANSFER_FRAG)
1233+ return ERR_PTR(-EINVAL);
1234+ break;
1235+ default:
1236+ return ERR_PTR(-EINVAL);
1237+ }
1238+
1239+ ctx_state_size = get_ctx_state_size(pvr_dev, type);
1240+ if (ctx_state_size < 0)
1241+ return ERR_PTR(ctx_state_size);
1242+
1243+ queue = kzalloc(sizeof(*queue), GFP_KERNEL);
1244+ if (!queue)
1245+ return ERR_PTR(-ENOMEM);
1246+
1247+ queue->type = type;
1248+ queue->ctx_offset = get_ctx_offset(type);
1249+ queue->ctx = ctx;
1250+ queue->callstack_addr = args->callstack_addr;
1251+ sched = &queue->scheduler;
1252+ INIT_LIST_HEAD(&queue->node);
1253+ mutex_init(&queue->cccb_fence_ctx.job_lock);
1254+ pvr_queue_fence_ctx_init(&queue->cccb_fence_ctx.base);
1255+ pvr_queue_fence_ctx_init(&queue->job_fence_ctx);
1256+
1257+ err = pvr_cccb_init(pvr_dev, &queue->cccb, props[type].cccb_size, props[type].name);
1258+ if (err)
1259+ goto err_free_queue;
1260+
1261+ err = pvr_fw_object_create(pvr_dev, ctx_state_size,
1262+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
1263+ reg_state_init, queue, &queue->reg_state_obj);
1264+ if (err)
1265+ goto err_cccb_fini;
1266+
1267+ init_fw_context(queue, fw_ctx_map);
1268+
1269+ if (type != DRM_PVR_JOB_TYPE_GEOMETRY && type != DRM_PVR_JOB_TYPE_FRAGMENT &&
1270+ args->callstack_addr) {
1271+ err = -EINVAL;
1272+ goto err_release_reg_state;
1273+ }
1274+
1275+ cpu_map = pvr_fw_object_create_and_map(pvr_dev, sizeof(*queue->timeline_ufo.value),
1276+ PVR_BO_FW_FLAGS_DEVICE_UNCACHED,
1277+ NULL, NULL, &queue->timeline_ufo.fw_obj);
1278+ if (IS_ERR(cpu_map)) {
1279+ err = PTR_ERR(cpu_map);
1280+ goto err_release_reg_state;
1281+ }
1282+
1283+ queue->timeline_ufo.value = cpu_map;
1284+
1285+ err = drm_sched_init(&queue->scheduler,
1286+ &pvr_queue_sched_ops,
1287+ pvr_dev->sched_wq, 1, 64 * 1024, 1,
1288+ msecs_to_jiffies(500),
1289+ pvr_dev->sched_wq, NULL, "pvr-queue",
1290+ pvr_dev->base.dev);
1291+ if (err)
1292+ goto err_release_ufo;
1293+
1294+ err = drm_sched_entity_init(&queue->entity,
1295+ DRM_SCHED_PRIORITY_MIN,
1296+ &sched, 1, &ctx->faulty);
1297+ if (err)
1298+ goto err_sched_fini;
1299+
1300+ mutex_lock(&pvr_dev->queues.lock);
1301+ list_add_tail(&queue->node, &pvr_dev->queues.idle);
1302+ mutex_unlock(&pvr_dev->queues.lock);
1303+
1304+ return queue;
1305+
1306+err_sched_fini:
1307+ drm_sched_fini(&queue->scheduler);
1308+
1309+err_release_ufo:
1310+ pvr_fw_object_unmap_and_destroy(queue->timeline_ufo.fw_obj);
1311+
1312+err_release_reg_state:
1313+ pvr_fw_object_destroy(queue->reg_state_obj);
1314+
1315+err_cccb_fini:
1316+ pvr_cccb_fini(&queue->cccb);
1317+
1318+err_free_queue:
1319+ mutex_destroy(&queue->cccb_fence_ctx.job_lock);
1320+ kfree(queue);
1321+
1322+ return ERR_PTR(err);
1323+}
1324+
1325+void pvr_queue_device_pre_reset(struct pvr_device *pvr_dev)
1326+{
1327+ struct pvr_queue *queue;
1328+
1329+ mutex_lock(&pvr_dev->queues.lock);
1330+ list_for_each_entry(queue, &pvr_dev->queues.idle, node)
1331+ pvr_queue_stop(queue, NULL);
1332+ list_for_each_entry(queue, &pvr_dev->queues.active, node)
1333+ pvr_queue_stop(queue, NULL);
1334+ mutex_unlock(&pvr_dev->queues.lock);
1335+}
1336+
1337+void pvr_queue_device_post_reset(struct pvr_device *pvr_dev)
1338+{
1339+ struct pvr_queue *queue;
1340+
1341+ mutex_lock(&pvr_dev->queues.lock);
1342+ list_for_each_entry(queue, &pvr_dev->queues.active, node)
1343+ pvr_queue_start(queue);
1344+ list_for_each_entry(queue, &pvr_dev->queues.idle, node)
1345+ pvr_queue_start(queue);
1346+ mutex_unlock(&pvr_dev->queues.lock);
1347+}
1348+
1349+/**
1350+ * pvr_queue_kill() - Kill a queue.
1351+ * @queue: The queue to kill.
1352+ *
1353+ * Kill the queue so no new jobs can be pushed. Should be called when the
1354+ * context handle is destroyed. The queue object might last longer if jobs
1355+ * are still in flight and holding a reference to the context this queue
1356+ * belongs to.
1357+ */
1358+void pvr_queue_kill(struct pvr_queue *queue)
1359+{
1360+ drm_sched_entity_destroy(&queue->entity);
1361+ dma_fence_put(queue->last_queued_job_scheduled_fence);
1362+ queue->last_queued_job_scheduled_fence = NULL;
1363+}
1364+
1365+/**
1366+ * pvr_queue_destroy() - Destroy a queue.
1367+ * @queue: The queue to destroy.
1368+ *
1369+ * Cleanup the queue and free the resources attached to it. Should be
1370+ * called from the context release function.
1371+ */
1372+void pvr_queue_destroy(struct pvr_queue *queue)
1373+{
1374+ if (!queue)
1375+ return;
1376+
1377+ mutex_lock(&queue->ctx->pvr_dev->queues.lock);
1378+ list_del_init(&queue->node);
1379+ mutex_unlock(&queue->ctx->pvr_dev->queues.lock);
1380+
1381+ drm_sched_fini(&queue->scheduler);
1382+ drm_sched_entity_fini(&queue->entity);
1383+
1384+ if (WARN_ON(queue->last_queued_job_scheduled_fence))
1385+ dma_fence_put(queue->last_queued_job_scheduled_fence);
1386+
1387+ pvr_queue_cleanup_fw_context(queue);
1388+
1389+ pvr_fw_object_unmap_and_destroy(queue->timeline_ufo.fw_obj);
1390+ pvr_fw_object_destroy(queue->reg_state_obj);
1391+ pvr_cccb_fini(&queue->cccb);
1392+ mutex_destroy(&queue->cccb_fence_ctx.job_lock);
1393+ kfree(queue);
1394+}
1395+
1396+/**
1397+ * pvr_queue_device_init() - Device-level initialization of queue related fields.
1398+ * @pvr_dev: The device to initialize.
1399+ *
1400+ * Initializes all fields related to queue management in pvr_device.
1401+ *
1402+ * Return:
1403+ * * 0 on success, or
1404+ * * An error code on failure.
1405+ */
1406+int pvr_queue_device_init(struct pvr_device *pvr_dev)
1407+{
1408+ int err;
1409+
1410+ INIT_LIST_HEAD(&pvr_dev->queues.active);
1411+ INIT_LIST_HEAD(&pvr_dev->queues.idle);
1412+ err = drmm_mutex_init(from_pvr_device(pvr_dev), &pvr_dev->queues.lock);
1413+ if (err)
1414+ return err;
1415+
1416+ pvr_dev->sched_wq = alloc_workqueue("powervr-sched", WQ_UNBOUND, 0);
1417+ if (!pvr_dev->sched_wq)
1418+ return -ENOMEM;
1419+
1420+ return 0;
1421+}
1422+
1423+/**
1424+ * pvr_queue_device_fini() - Device-level cleanup of queue related fields.
1425+ * @pvr_dev: The device to cleanup.
1426+ *
1427+ * Cleanup/free all queue-related resources attached to a pvr_device object.
1428+ */
1429+void pvr_queue_device_fini(struct pvr_device *pvr_dev)
1430+{
1431+ destroy_workqueue(pvr_dev->sched_wq);
1432+}
drivers/gpu/drm/imagination/pvr_queue.h+169−0
@@ -0,0 +1,169 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+#ifndef PVR_QUEUE_H
5+#define PVR_QUEUE_H
6+
7+#include <drm/gpu_scheduler.h>
8+
9+#include "pvr_cccb.h"
10+#include "pvr_device.h"
11+
12+struct pvr_context;
13+struct pvr_queue;
14+
15+/**
16+ * struct pvr_queue_fence_ctx - Queue fence context
17+ *
18+ * Used to implement dma_fence_ops for pvr_job::{done,cccb}_fence.
19+ */
20+struct pvr_queue_fence_ctx {
21+ /** @id: Fence context ID allocated with dma_fence_context_alloc(). */
22+ u64 id;
23+
24+ /** @seqno: Sequence number incremented each time a fence is created. */
25+ atomic_t seqno;
26+
27+ /** @lock: Lock used to synchronize access to fences allocated by this context. */
28+ spinlock_t lock;
29+};
30+
31+/**
32+ * struct pvr_queue_cccb_fence_ctx - CCCB fence context
33+ *
34+ * Context used to manage fences controlling access to the CCCB. No fences are
35+ * issued if there's enough space in the CCCB to push job commands.
36+ */
37+struct pvr_queue_cccb_fence_ctx {
38+ /** @base: Base queue fence context. */
39+ struct pvr_queue_fence_ctx base;
40+
41+ /**
42+ * @job: Job waiting for CCCB space.
43+ *
44+ * Thanks to the serializationg done at the drm_sched_entity level,
45+ * there's no more than one job waiting for CCCB at a given time.
46+ *
47+ * This field is NULL if no jobs are currently waiting for CCCB space.
48+ *
49+ * Must be accessed with @job_lock held.
50+ */
51+ struct pvr_job *job;
52+
53+ /** @lock: Lock protecting access to the job object. */
54+ struct mutex job_lock;
55+};
56+
57+/**
58+ * struct pvr_queue_fence - Queue fence object
59+ */
60+struct pvr_queue_fence {
61+ /** @base: Base dma_fence. */
62+ struct dma_fence base;
63+
64+ /** @queue: Queue that created this fence. */
65+ struct pvr_queue *queue;
66+};
67+
68+/**
69+ * struct pvr_queue - Job queue
70+ *
71+ * Used to queue and track execution of pvr_job objects.
72+ */
73+struct pvr_queue {
74+ /** @scheduler: Single entity scheduler use to push jobs to this queue. */
75+ struct drm_gpu_scheduler scheduler;
76+
77+ /** @entity: Scheduling entity backing this queue. */
78+ struct drm_sched_entity entity;
79+
80+ /** @type: Type of jobs queued to this queue. */
81+ enum drm_pvr_job_type type;
82+
83+ /** @ctx: Context object this queue is bound to. */
84+ struct pvr_context *ctx;
85+
86+ /** @node: Used to add the queue to the active/idle queue list. */
87+ struct list_head node;
88+
89+ /**
90+ * @in_flight_job_count: Number of jobs submitted to the CCCB that
91+ * have not been processed yet.
92+ */
93+ atomic_t in_flight_job_count;
94+
95+ /**
96+ * @cccb_fence_ctx: CCCB fence context.
97+ *
98+ * Used to control access to the CCCB is full, such that we don't
99+ * end up trying to push commands to the CCCB if there's not enough
100+ * space to receive all commands needed for a job to complete.
101+ */
102+ struct pvr_queue_cccb_fence_ctx cccb_fence_ctx;
103+
104+ /** @job_fence_ctx: Job fence context object. */
105+ struct pvr_queue_fence_ctx job_fence_ctx;
106+
107+ /** @timeline_ufo: Timeline UFO for the context queue. */
108+ struct {
109+ /** @fw_obj: FW object representing the UFO value. */
110+ struct pvr_fw_object *fw_obj;
111+
112+ /** @value: CPU mapping of the UFO value. */
113+ u32 *value;
114+ } timeline_ufo;
115+
116+ /**
117+ * last_queued_job_scheduled_fence: The scheduled fence of the last
118+ * job queued to this queue.
119+ *
120+ * We use it to insert frag -> geom dependencies when issuing combined
121+ * geom+frag jobs, to guarantee that the fragment job that's part of
122+ * the combined operation comes after all fragment jobs that were queued
123+ * before it.
124+ */
125+ struct dma_fence *last_queued_job_scheduled_fence;
126+
127+ /** @cccb: Client Circular Command Buffer. */
128+ struct pvr_cccb cccb;
129+
130+ /** @reg_state_obj: FW object representing the register state of this queue. */
131+ struct pvr_fw_object *reg_state_obj;
132+
133+ /** @ctx_offset: Offset of the queue context in the FW context object. */
134+ u32 ctx_offset;
135+
136+ /** @callstack_addr: Initial call stack address for register state object. */
137+ u64 callstack_addr;
138+};
139+
140+bool pvr_queue_fence_is_ufo_backed(struct dma_fence *f);
141+
142+int pvr_queue_job_init(struct pvr_job *job);
143+
144+void pvr_queue_job_cleanup(struct pvr_job *job);
145+
146+void pvr_queue_job_push(struct pvr_job *job);
147+
148+struct dma_fence *pvr_queue_job_arm(struct pvr_job *job);
149+
150+struct pvr_queue *pvr_queue_create(struct pvr_context *ctx,
151+ enum drm_pvr_job_type type,
152+ struct drm_pvr_ioctl_create_context_args *args,
153+ void *fw_ctx_map);
154+
155+void pvr_queue_kill(struct pvr_queue *queue);
156+
157+void pvr_queue_destroy(struct pvr_queue *queue);
158+
159+void pvr_queue_process(struct pvr_queue *queue);
160+
161+void pvr_queue_device_pre_reset(struct pvr_device *pvr_dev);
162+
163+void pvr_queue_device_post_reset(struct pvr_device *pvr_dev);
164+
165+int pvr_queue_device_init(struct pvr_device *pvr_dev);
166+
167+void pvr_queue_device_fini(struct pvr_device *pvr_dev);
168+
169+#endif /* PVR_QUEUE_H */
drivers/gpu/drm/imagination/pvr_rogue_cr_defs.h+6193−0
@@ -0,0 +1,6193 @@
1+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
2+/* Copyright (c) 2023 Imagination Technologies Ltd. */
3+
4+/* *** Autogenerated C -- do not edit *** */
5+
6+#ifndef PVR_ROGUE_CR_DEFS_H
7+#define PVR_ROGUE_CR_DEFS_H
8+
9+/* clang-format off */
10+
11+#define ROGUE_CR_DEFS_REVISION 1
12+
13+/* Register ROGUE_CR_RASTERISATION_INDIRECT */
14+#define ROGUE_CR_RASTERISATION_INDIRECT 0x8238U
15+#define ROGUE_CR_RASTERISATION_INDIRECT_MASKFULL 0x000000000000000FULL
16+#define ROGUE_CR_RASTERISATION_INDIRECT_ADDRESS_SHIFT 0U
17+#define ROGUE_CR_RASTERISATION_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF0U
18+
19+/* Register ROGUE_CR_PBE_INDIRECT */
20+#define ROGUE_CR_PBE_INDIRECT 0x83E0U
21+#define ROGUE_CR_PBE_INDIRECT_MASKFULL 0x000000000000000FULL
22+#define ROGUE_CR_PBE_INDIRECT_ADDRESS_SHIFT 0U
23+#define ROGUE_CR_PBE_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF0U
24+
25+/* Register ROGUE_CR_PBE_PERF_INDIRECT */
26+#define ROGUE_CR_PBE_PERF_INDIRECT 0x83D8U
27+#define ROGUE_CR_PBE_PERF_INDIRECT_MASKFULL 0x000000000000000FULL
28+#define ROGUE_CR_PBE_PERF_INDIRECT_ADDRESS_SHIFT 0U
29+#define ROGUE_CR_PBE_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF0U
30+
31+/* Register ROGUE_CR_TPU_PERF_INDIRECT */
32+#define ROGUE_CR_TPU_PERF_INDIRECT 0x83F0U
33+#define ROGUE_CR_TPU_PERF_INDIRECT_MASKFULL 0x0000000000000007ULL
34+#define ROGUE_CR_TPU_PERF_INDIRECT_ADDRESS_SHIFT 0U
35+#define ROGUE_CR_TPU_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF8U
36+
37+/* Register ROGUE_CR_RASTERISATION_PERF_INDIRECT */
38+#define ROGUE_CR_RASTERISATION_PERF_INDIRECT 0x8318U
39+#define ROGUE_CR_RASTERISATION_PERF_INDIRECT_MASKFULL 0x000000000000000FULL
40+#define ROGUE_CR_RASTERISATION_PERF_INDIRECT_ADDRESS_SHIFT 0U
41+#define ROGUE_CR_RASTERISATION_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF0U
42+
43+/* Register ROGUE_CR_TPU_MCU_L0_PERF_INDIRECT */
44+#define ROGUE_CR_TPU_MCU_L0_PERF_INDIRECT 0x8028U
45+#define ROGUE_CR_TPU_MCU_L0_PERF_INDIRECT_MASKFULL 0x0000000000000007ULL
46+#define ROGUE_CR_TPU_MCU_L0_PERF_INDIRECT_ADDRESS_SHIFT 0U
47+#define ROGUE_CR_TPU_MCU_L0_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF8U
48+
49+/* Register ROGUE_CR_USC_PERF_INDIRECT */
50+#define ROGUE_CR_USC_PERF_INDIRECT 0x8030U
51+#define ROGUE_CR_USC_PERF_INDIRECT_MASKFULL 0x000000000000000FULL
52+#define ROGUE_CR_USC_PERF_INDIRECT_ADDRESS_SHIFT 0U
53+#define ROGUE_CR_USC_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF0U
54+
55+/* Register ROGUE_CR_BLACKPEARL_INDIRECT */
56+#define ROGUE_CR_BLACKPEARL_INDIRECT 0x8388U
57+#define ROGUE_CR_BLACKPEARL_INDIRECT_MASKFULL 0x0000000000000003ULL
58+#define ROGUE_CR_BLACKPEARL_INDIRECT_ADDRESS_SHIFT 0U
59+#define ROGUE_CR_BLACKPEARL_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFFCU
60+
61+/* Register ROGUE_CR_BLACKPEARL_PERF_INDIRECT */
62+#define ROGUE_CR_BLACKPEARL_PERF_INDIRECT 0x83F8U
63+#define ROGUE_CR_BLACKPEARL_PERF_INDIRECT_MASKFULL 0x0000000000000003ULL
64+#define ROGUE_CR_BLACKPEARL_PERF_INDIRECT_ADDRESS_SHIFT 0U
65+#define ROGUE_CR_BLACKPEARL_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFFCU
66+
67+/* Register ROGUE_CR_TEXAS3_PERF_INDIRECT */
68+#define ROGUE_CR_TEXAS3_PERF_INDIRECT 0x83D0U
69+#define ROGUE_CR_TEXAS3_PERF_INDIRECT_MASKFULL 0x0000000000000007ULL
70+#define ROGUE_CR_TEXAS3_PERF_INDIRECT_ADDRESS_SHIFT 0U
71+#define ROGUE_CR_TEXAS3_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFF8U
72+
73+/* Register ROGUE_CR_TEXAS_PERF_INDIRECT */
74+#define ROGUE_CR_TEXAS_PERF_INDIRECT 0x8288U
75+#define ROGUE_CR_TEXAS_PERF_INDIRECT_MASKFULL 0x0000000000000003ULL
76+#define ROGUE_CR_TEXAS_PERF_INDIRECT_ADDRESS_SHIFT 0U
77+#define ROGUE_CR_TEXAS_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFFCU
78+
79+/* Register ROGUE_CR_BX_TU_PERF_INDIRECT */
80+#define ROGUE_CR_BX_TU_PERF_INDIRECT 0xC900U
81+#define ROGUE_CR_BX_TU_PERF_INDIRECT_MASKFULL 0x0000000000000003ULL
82+#define ROGUE_CR_BX_TU_PERF_INDIRECT_ADDRESS_SHIFT 0U
83+#define ROGUE_CR_BX_TU_PERF_INDIRECT_ADDRESS_CLRMSK 0xFFFFFFFCU
84+
85+/* Register ROGUE_CR_CLK_CTRL */
86+#define ROGUE_CR_CLK_CTRL 0x0000U
87+#define ROGUE_CR_CLK_CTRL__PBE2_XE__MASKFULL 0xFFFFFF003F3FFFFFULL
88+#define ROGUE_CR_CLK_CTRL__S7_TOP__MASKFULL 0xCFCF03000F3F3F0FULL
89+#define ROGUE_CR_CLK_CTRL_MASKFULL 0xFFFFFF003F3FFFFFULL
90+#define ROGUE_CR_CLK_CTRL_BIF_TEXAS_SHIFT 62U
91+#define ROGUE_CR_CLK_CTRL_BIF_TEXAS_CLRMSK 0x3FFFFFFFFFFFFFFFULL
92+#define ROGUE_CR_CLK_CTRL_BIF_TEXAS_OFF 0x0000000000000000ULL
93+#define ROGUE_CR_CLK_CTRL_BIF_TEXAS_ON 0x4000000000000000ULL
94+#define ROGUE_CR_CLK_CTRL_BIF_TEXAS_AUTO 0x8000000000000000ULL
95+#define ROGUE_CR_CLK_CTRL_IPP_SHIFT 60U
96+#define ROGUE_CR_CLK_CTRL_IPP_CLRMSK 0xCFFFFFFFFFFFFFFFULL
97+#define ROGUE_CR_CLK_CTRL_IPP_OFF 0x0000000000000000ULL
98+#define ROGUE_CR_CLK_CTRL_IPP_ON 0x1000000000000000ULL
99+#define ROGUE_CR_CLK_CTRL_IPP_AUTO 0x2000000000000000ULL
100+#define ROGUE_CR_CLK_CTRL_FBC_SHIFT 58U
101+#define ROGUE_CR_CLK_CTRL_FBC_CLRMSK 0xF3FFFFFFFFFFFFFFULL
102+#define ROGUE_CR_CLK_CTRL_FBC_OFF 0x0000000000000000ULL
103+#define ROGUE_CR_CLK_CTRL_FBC_ON 0x0400000000000000ULL
104+#define ROGUE_CR_CLK_CTRL_FBC_AUTO 0x0800000000000000ULL
105+#define ROGUE_CR_CLK_CTRL_FBDC_SHIFT 56U
106+#define ROGUE_CR_CLK_CTRL_FBDC_CLRMSK 0xFCFFFFFFFFFFFFFFULL
107+#define ROGUE_CR_CLK_CTRL_FBDC_OFF 0x0000000000000000ULL
108+#define ROGUE_CR_CLK_CTRL_FBDC_ON 0x0100000000000000ULL
109+#define ROGUE_CR_CLK_CTRL_FBDC_AUTO 0x0200000000000000ULL
110+#define ROGUE_CR_CLK_CTRL_FB_TLCACHE_SHIFT 54U
111+#define ROGUE_CR_CLK_CTRL_FB_TLCACHE_CLRMSK 0xFF3FFFFFFFFFFFFFULL
112+#define ROGUE_CR_CLK_CTRL_FB_TLCACHE_OFF 0x0000000000000000ULL
113+#define ROGUE_CR_CLK_CTRL_FB_TLCACHE_ON 0x0040000000000000ULL
114+#define ROGUE_CR_CLK_CTRL_FB_TLCACHE_AUTO 0x0080000000000000ULL
115+#define ROGUE_CR_CLK_CTRL_USCS_SHIFT 52U
116+#define ROGUE_CR_CLK_CTRL_USCS_CLRMSK 0xFFCFFFFFFFFFFFFFULL
117+#define ROGUE_CR_CLK_CTRL_USCS_OFF 0x0000000000000000ULL
118+#define ROGUE_CR_CLK_CTRL_USCS_ON 0x0010000000000000ULL
119+#define ROGUE_CR_CLK_CTRL_USCS_AUTO 0x0020000000000000ULL
120+#define ROGUE_CR_CLK_CTRL_PBE_SHIFT 50U
121+#define ROGUE_CR_CLK_CTRL_PBE_CLRMSK 0xFFF3FFFFFFFFFFFFULL
122+#define ROGUE_CR_CLK_CTRL_PBE_OFF 0x0000000000000000ULL
123+#define ROGUE_CR_CLK_CTRL_PBE_ON 0x0004000000000000ULL
124+#define ROGUE_CR_CLK_CTRL_PBE_AUTO 0x0008000000000000ULL
125+#define ROGUE_CR_CLK_CTRL_MCU_L1_SHIFT 48U
126+#define ROGUE_CR_CLK_CTRL_MCU_L1_CLRMSK 0xFFFCFFFFFFFFFFFFULL
127+#define ROGUE_CR_CLK_CTRL_MCU_L1_OFF 0x0000000000000000ULL
128+#define ROGUE_CR_CLK_CTRL_MCU_L1_ON 0x0001000000000000ULL
129+#define ROGUE_CR_CLK_CTRL_MCU_L1_AUTO 0x0002000000000000ULL
130+#define ROGUE_CR_CLK_CTRL_CDM_SHIFT 46U
131+#define ROGUE_CR_CLK_CTRL_CDM_CLRMSK 0xFFFF3FFFFFFFFFFFULL
132+#define ROGUE_CR_CLK_CTRL_CDM_OFF 0x0000000000000000ULL
133+#define ROGUE_CR_CLK_CTRL_CDM_ON 0x0000400000000000ULL
134+#define ROGUE_CR_CLK_CTRL_CDM_AUTO 0x0000800000000000ULL
135+#define ROGUE_CR_CLK_CTRL_SIDEKICK_SHIFT 44U
136+#define ROGUE_CR_CLK_CTRL_SIDEKICK_CLRMSK 0xFFFFCFFFFFFFFFFFULL
137+#define ROGUE_CR_CLK_CTRL_SIDEKICK_OFF 0x0000000000000000ULL
138+#define ROGUE_CR_CLK_CTRL_SIDEKICK_ON 0x0000100000000000ULL
139+#define ROGUE_CR_CLK_CTRL_SIDEKICK_AUTO 0x0000200000000000ULL
140+#define ROGUE_CR_CLK_CTRL_BIF_SIDEKICK_SHIFT 42U
141+#define ROGUE_CR_CLK_CTRL_BIF_SIDEKICK_CLRMSK 0xFFFFF3FFFFFFFFFFULL
142+#define ROGUE_CR_CLK_CTRL_BIF_SIDEKICK_OFF 0x0000000000000000ULL
143+#define ROGUE_CR_CLK_CTRL_BIF_SIDEKICK_ON 0x0000040000000000ULL
144+#define ROGUE_CR_CLK_CTRL_BIF_SIDEKICK_AUTO 0x0000080000000000ULL
145+#define ROGUE_CR_CLK_CTRL_BIF_SHIFT 40U
146+#define ROGUE_CR_CLK_CTRL_BIF_CLRMSK 0xFFFFFCFFFFFFFFFFULL
147+#define ROGUE_CR_CLK_CTRL_BIF_OFF 0x0000000000000000ULL
148+#define ROGUE_CR_CLK_CTRL_BIF_ON 0x0000010000000000ULL
149+#define ROGUE_CR_CLK_CTRL_BIF_AUTO 0x0000020000000000ULL
150+#define ROGUE_CR_CLK_CTRL_TPU_MCU_DEMUX_SHIFT 28U
151+#define ROGUE_CR_CLK_CTRL_TPU_MCU_DEMUX_CLRMSK 0xFFFFFFFFCFFFFFFFULL
152+#define ROGUE_CR_CLK_CTRL_TPU_MCU_DEMUX_OFF 0x0000000000000000ULL
153+#define ROGUE_CR_CLK_CTRL_TPU_MCU_DEMUX_ON 0x0000000010000000ULL
154+#define ROGUE_CR_CLK_CTRL_TPU_MCU_DEMUX_AUTO 0x0000000020000000ULL
155+#define ROGUE_CR_CLK_CTRL_MCU_L0_SHIFT 26U
156+#define ROGUE_CR_CLK_CTRL_MCU_L0_CLRMSK 0xFFFFFFFFF3FFFFFFULL
157+#define ROGUE_CR_CLK_CTRL_MCU_L0_OFF 0x0000000000000000ULL
158+#define ROGUE_CR_CLK_CTRL_MCU_L0_ON 0x0000000004000000ULL
159+#define ROGUE_CR_CLK_CTRL_MCU_L0_AUTO 0x0000000008000000ULL
160+#define ROGUE_CR_CLK_CTRL_TPU_SHIFT 24U
161+#define ROGUE_CR_CLK_CTRL_TPU_CLRMSK 0xFFFFFFFFFCFFFFFFULL
162+#define ROGUE_CR_CLK_CTRL_TPU_OFF 0x0000000000000000ULL
163+#define ROGUE_CR_CLK_CTRL_TPU_ON 0x0000000001000000ULL
164+#define ROGUE_CR_CLK_CTRL_TPU_AUTO 0x0000000002000000ULL
165+#define ROGUE_CR_CLK_CTRL_USC_SHIFT 20U
166+#define ROGUE_CR_CLK_CTRL_USC_CLRMSK 0xFFFFFFFFFFCFFFFFULL
167+#define ROGUE_CR_CLK_CTRL_USC_OFF 0x0000000000000000ULL
168+#define ROGUE_CR_CLK_CTRL_USC_ON 0x0000000000100000ULL
169+#define ROGUE_CR_CLK_CTRL_USC_AUTO 0x0000000000200000ULL
170+#define ROGUE_CR_CLK_CTRL_TLA_SHIFT 18U
171+#define ROGUE_CR_CLK_CTRL_TLA_CLRMSK 0xFFFFFFFFFFF3FFFFULL
172+#define ROGUE_CR_CLK_CTRL_TLA_OFF 0x0000000000000000ULL
173+#define ROGUE_CR_CLK_CTRL_TLA_ON 0x0000000000040000ULL
174+#define ROGUE_CR_CLK_CTRL_TLA_AUTO 0x0000000000080000ULL
175+#define ROGUE_CR_CLK_CTRL_SLC_SHIFT 16U
176+#define ROGUE_CR_CLK_CTRL_SLC_CLRMSK 0xFFFFFFFFFFFCFFFFULL
177+#define ROGUE_CR_CLK_CTRL_SLC_OFF 0x0000000000000000ULL
178+#define ROGUE_CR_CLK_CTRL_SLC_ON 0x0000000000010000ULL
179+#define ROGUE_CR_CLK_CTRL_SLC_AUTO 0x0000000000020000ULL
180+#define ROGUE_CR_CLK_CTRL_UVS_SHIFT 14U
181+#define ROGUE_CR_CLK_CTRL_UVS_CLRMSK 0xFFFFFFFFFFFF3FFFULL
182+#define ROGUE_CR_CLK_CTRL_UVS_OFF 0x0000000000000000ULL
183+#define ROGUE_CR_CLK_CTRL_UVS_ON 0x0000000000004000ULL
184+#define ROGUE_CR_CLK_CTRL_UVS_AUTO 0x0000000000008000ULL
185+#define ROGUE_CR_CLK_CTRL_PDS_SHIFT 12U
186+#define ROGUE_CR_CLK_CTRL_PDS_CLRMSK 0xFFFFFFFFFFFFCFFFULL
187+#define ROGUE_CR_CLK_CTRL_PDS_OFF 0x0000000000000000ULL
188+#define ROGUE_CR_CLK_CTRL_PDS_ON 0x0000000000001000ULL
189+#define ROGUE_CR_CLK_CTRL_PDS_AUTO 0x0000000000002000ULL
190+#define ROGUE_CR_CLK_CTRL_VDM_SHIFT 10U
191+#define ROGUE_CR_CLK_CTRL_VDM_CLRMSK 0xFFFFFFFFFFFFF3FFULL
192+#define ROGUE_CR_CLK_CTRL_VDM_OFF 0x0000000000000000ULL
193+#define ROGUE_CR_CLK_CTRL_VDM_ON 0x0000000000000400ULL
194+#define ROGUE_CR_CLK_CTRL_VDM_AUTO 0x0000000000000800ULL
195+#define ROGUE_CR_CLK_CTRL_PM_SHIFT 8U
196+#define ROGUE_CR_CLK_CTRL_PM_CLRMSK 0xFFFFFFFFFFFFFCFFULL
197+#define ROGUE_CR_CLK_CTRL_PM_OFF 0x0000000000000000ULL
198+#define ROGUE_CR_CLK_CTRL_PM_ON 0x0000000000000100ULL
199+#define ROGUE_CR_CLK_CTRL_PM_AUTO 0x0000000000000200ULL
200+#define ROGUE_CR_CLK_CTRL_GPP_SHIFT 6U
201+#define ROGUE_CR_CLK_CTRL_GPP_CLRMSK 0xFFFFFFFFFFFFFF3FULL
202+#define ROGUE_CR_CLK_CTRL_GPP_OFF 0x0000000000000000ULL
203+#define ROGUE_CR_CLK_CTRL_GPP_ON 0x0000000000000040ULL
204+#define ROGUE_CR_CLK_CTRL_GPP_AUTO 0x0000000000000080ULL
205+#define ROGUE_CR_CLK_CTRL_TE_SHIFT 4U
206+#define ROGUE_CR_CLK_CTRL_TE_CLRMSK 0xFFFFFFFFFFFFFFCFULL
207+#define ROGUE_CR_CLK_CTRL_TE_OFF 0x0000000000000000ULL
208+#define ROGUE_CR_CLK_CTRL_TE_ON 0x0000000000000010ULL
209+#define ROGUE_CR_CLK_CTRL_TE_AUTO 0x0000000000000020ULL
210+#define ROGUE_CR_CLK_CTRL_TSP_SHIFT 2U
211+#define ROGUE_CR_CLK_CTRL_TSP_CLRMSK 0xFFFFFFFFFFFFFFF3ULL
212+#define ROGUE_CR_CLK_CTRL_TSP_OFF 0x0000000000000000ULL
213+#define ROGUE_CR_CLK_CTRL_TSP_ON 0x0000000000000004ULL
214+#define ROGUE_CR_CLK_CTRL_TSP_AUTO 0x0000000000000008ULL
215+#define ROGUE_CR_CLK_CTRL_ISP_SHIFT 0U
216+#define ROGUE_CR_CLK_CTRL_ISP_CLRMSK 0xFFFFFFFFFFFFFFFCULL
217+#define ROGUE_CR_CLK_CTRL_ISP_OFF 0x0000000000000000ULL
218+#define ROGUE_CR_CLK_CTRL_ISP_ON 0x0000000000000001ULL
219+#define ROGUE_CR_CLK_CTRL_ISP_AUTO 0x0000000000000002ULL
220+
221+/* Register ROGUE_CR_CLK_STATUS */
222+#define ROGUE_CR_CLK_STATUS 0x0008U
223+#define ROGUE_CR_CLK_STATUS__PBE2_XE__MASKFULL 0x00000001FFF077FFULL
224+#define ROGUE_CR_CLK_STATUS__S7_TOP__MASKFULL 0x00000001B3101773ULL
225+#define ROGUE_CR_CLK_STATUS_MASKFULL 0x00000001FFF077FFULL
226+#define ROGUE_CR_CLK_STATUS_MCU_FBTC_SHIFT 32U
227+#define ROGUE_CR_CLK_STATUS_MCU_FBTC_CLRMSK 0xFFFFFFFEFFFFFFFFULL
228+#define ROGUE_CR_CLK_STATUS_MCU_FBTC_GATED 0x0000000000000000ULL
229+#define ROGUE_CR_CLK_STATUS_MCU_FBTC_RUNNING 0x0000000100000000ULL
230+#define ROGUE_CR_CLK_STATUS_BIF_TEXAS_SHIFT 31U
231+#define ROGUE_CR_CLK_STATUS_BIF_TEXAS_CLRMSK 0xFFFFFFFF7FFFFFFFULL
232+#define ROGUE_CR_CLK_STATUS_BIF_TEXAS_GATED 0x0000000000000000ULL
233+#define ROGUE_CR_CLK_STATUS_BIF_TEXAS_RUNNING 0x0000000080000000ULL
234+#define ROGUE_CR_CLK_STATUS_IPP_SHIFT 30U
235+#define ROGUE_CR_CLK_STATUS_IPP_CLRMSK 0xFFFFFFFFBFFFFFFFULL
236+#define ROGUE_CR_CLK_STATUS_IPP_GATED 0x0000000000000000ULL
237+#define ROGUE_CR_CLK_STATUS_IPP_RUNNING 0x0000000040000000ULL
238+#define ROGUE_CR_CLK_STATUS_FBC_SHIFT 29U
239+#define ROGUE_CR_CLK_STATUS_FBC_CLRMSK 0xFFFFFFFFDFFFFFFFULL
240+#define ROGUE_CR_CLK_STATUS_FBC_GATED 0x0000000000000000ULL
241+#define ROGUE_CR_CLK_STATUS_FBC_RUNNING 0x0000000020000000ULL
242+#define ROGUE_CR_CLK_STATUS_FBDC_SHIFT 28U
243+#define ROGUE_CR_CLK_STATUS_FBDC_CLRMSK 0xFFFFFFFFEFFFFFFFULL
244+#define ROGUE_CR_CLK_STATUS_FBDC_GATED 0x0000000000000000ULL
245+#define ROGUE_CR_CLK_STATUS_FBDC_RUNNING 0x0000000010000000ULL
246+#define ROGUE_CR_CLK_STATUS_FB_TLCACHE_SHIFT 27U
247+#define ROGUE_CR_CLK_STATUS_FB_TLCACHE_CLRMSK 0xFFFFFFFFF7FFFFFFULL
248+#define ROGUE_CR_CLK_STATUS_FB_TLCACHE_GATED 0x0000000000000000ULL
249+#define ROGUE_CR_CLK_STATUS_FB_TLCACHE_RUNNING 0x0000000008000000ULL
250+#define ROGUE_CR_CLK_STATUS_USCS_SHIFT 26U
251+#define ROGUE_CR_CLK_STATUS_USCS_CLRMSK 0xFFFFFFFFFBFFFFFFULL
252+#define ROGUE_CR_CLK_STATUS_USCS_GATED 0x0000000000000000ULL
253+#define ROGUE_CR_CLK_STATUS_USCS_RUNNING 0x0000000004000000ULL
254+#define ROGUE_CR_CLK_STATUS_PBE_SHIFT 25U
255+#define ROGUE_CR_CLK_STATUS_PBE_CLRMSK 0xFFFFFFFFFDFFFFFFULL
256+#define ROGUE_CR_CLK_STATUS_PBE_GATED 0x0000000000000000ULL
257+#define ROGUE_CR_CLK_STATUS_PBE_RUNNING 0x0000000002000000ULL
258+#define ROGUE_CR_CLK_STATUS_MCU_L1_SHIFT 24U
259+#define ROGUE_CR_CLK_STATUS_MCU_L1_CLRMSK 0xFFFFFFFFFEFFFFFFULL
260+#define ROGUE_CR_CLK_STATUS_MCU_L1_GATED 0x0000000000000000ULL
261+#define ROGUE_CR_CLK_STATUS_MCU_L1_RUNNING 0x0000000001000000ULL
262+#define ROGUE_CR_CLK_STATUS_CDM_SHIFT 23U
263+#define ROGUE_CR_CLK_STATUS_CDM_CLRMSK 0xFFFFFFFFFF7FFFFFULL
264+#define ROGUE_CR_CLK_STATUS_CDM_GATED 0x0000000000000000ULL
265+#define ROGUE_CR_CLK_STATUS_CDM_RUNNING 0x0000000000800000ULL
266+#define ROGUE_CR_CLK_STATUS_SIDEKICK_SHIFT 22U
267+#define ROGUE_CR_CLK_STATUS_SIDEKICK_CLRMSK 0xFFFFFFFFFFBFFFFFULL
268+#define ROGUE_CR_CLK_STATUS_SIDEKICK_GATED 0x0000000000000000ULL
269+#define ROGUE_CR_CLK_STATUS_SIDEKICK_RUNNING 0x0000000000400000ULL
270+#define ROGUE_CR_CLK_STATUS_BIF_SIDEKICK_SHIFT 21U
271+#define ROGUE_CR_CLK_STATUS_BIF_SIDEKICK_CLRMSK 0xFFFFFFFFFFDFFFFFULL
272+#define ROGUE_CR_CLK_STATUS_BIF_SIDEKICK_GATED 0x0000000000000000ULL
273+#define ROGUE_CR_CLK_STATUS_BIF_SIDEKICK_RUNNING 0x0000000000200000ULL
274+#define ROGUE_CR_CLK_STATUS_BIF_SHIFT 20U
275+#define ROGUE_CR_CLK_STATUS_BIF_CLRMSK 0xFFFFFFFFFFEFFFFFULL
276+#define ROGUE_CR_CLK_STATUS_BIF_GATED 0x0000000000000000ULL
277+#define ROGUE_CR_CLK_STATUS_BIF_RUNNING 0x0000000000100000ULL
278+#define ROGUE_CR_CLK_STATUS_TPU_MCU_DEMUX_SHIFT 14U
279+#define ROGUE_CR_CLK_STATUS_TPU_MCU_DEMUX_CLRMSK 0xFFFFFFFFFFFFBFFFULL
280+#define ROGUE_CR_CLK_STATUS_TPU_MCU_DEMUX_GATED 0x0000000000000000ULL
281+#define ROGUE_CR_CLK_STATUS_TPU_MCU_DEMUX_RUNNING 0x0000000000004000ULL
282+#define ROGUE_CR_CLK_STATUS_MCU_L0_SHIFT 13U
283+#define ROGUE_CR_CLK_STATUS_MCU_L0_CLRMSK 0xFFFFFFFFFFFFDFFFULL
284+#define ROGUE_CR_CLK_STATUS_MCU_L0_GATED 0x0000000000000000ULL
285+#define ROGUE_CR_CLK_STATUS_MCU_L0_RUNNING 0x0000000000002000ULL
286+#define ROGUE_CR_CLK_STATUS_TPU_SHIFT 12U
287+#define ROGUE_CR_CLK_STATUS_TPU_CLRMSK 0xFFFFFFFFFFFFEFFFULL
288+#define ROGUE_CR_CLK_STATUS_TPU_GATED 0x0000000000000000ULL
289+#define ROGUE_CR_CLK_STATUS_TPU_RUNNING 0x0000000000001000ULL
290+#define ROGUE_CR_CLK_STATUS_USC_SHIFT 10U
291+#define ROGUE_CR_CLK_STATUS_USC_CLRMSK 0xFFFFFFFFFFFFFBFFULL
292+#define ROGUE_CR_CLK_STATUS_USC_GATED 0x0000000000000000ULL
293+#define ROGUE_CR_CLK_STATUS_USC_RUNNING 0x0000000000000400ULL
294+#define ROGUE_CR_CLK_STATUS_TLA_SHIFT 9U
295+#define ROGUE_CR_CLK_STATUS_TLA_CLRMSK 0xFFFFFFFFFFFFFDFFULL
296+#define ROGUE_CR_CLK_STATUS_TLA_GATED 0x0000000000000000ULL
297+#define ROGUE_CR_CLK_STATUS_TLA_RUNNING 0x0000000000000200ULL
298+#define ROGUE_CR_CLK_STATUS_SLC_SHIFT 8U
299+#define ROGUE_CR_CLK_STATUS_SLC_CLRMSK 0xFFFFFFFFFFFFFEFFULL
300+#define ROGUE_CR_CLK_STATUS_SLC_GATED 0x0000000000000000ULL
301+#define ROGUE_CR_CLK_STATUS_SLC_RUNNING 0x0000000000000100ULL
302+#define ROGUE_CR_CLK_STATUS_UVS_SHIFT 7U
303+#define ROGUE_CR_CLK_STATUS_UVS_CLRMSK 0xFFFFFFFFFFFFFF7FULL
304+#define ROGUE_CR_CLK_STATUS_UVS_GATED 0x0000000000000000ULL
305+#define ROGUE_CR_CLK_STATUS_UVS_RUNNING 0x0000000000000080ULL
306+#define ROGUE_CR_CLK_STATUS_PDS_SHIFT 6U
307+#define ROGUE_CR_CLK_STATUS_PDS_CLRMSK 0xFFFFFFFFFFFFFFBFULL
308+#define ROGUE_CR_CLK_STATUS_PDS_GATED 0x0000000000000000ULL
309+#define ROGUE_CR_CLK_STATUS_PDS_RUNNING 0x0000000000000040ULL
310+#define ROGUE_CR_CLK_STATUS_VDM_SHIFT 5U
311+#define ROGUE_CR_CLK_STATUS_VDM_CLRMSK 0xFFFFFFFFFFFFFFDFULL
312+#define ROGUE_CR_CLK_STATUS_VDM_GATED 0x0000000000000000ULL
313+#define ROGUE_CR_CLK_STATUS_VDM_RUNNING 0x0000000000000020ULL
314+#define ROGUE_CR_CLK_STATUS_PM_SHIFT 4U
315+#define ROGUE_CR_CLK_STATUS_PM_CLRMSK 0xFFFFFFFFFFFFFFEFULL
316+#define ROGUE_CR_CLK_STATUS_PM_GATED 0x0000000000000000ULL
317+#define ROGUE_CR_CLK_STATUS_PM_RUNNING 0x0000000000000010ULL
318+#define ROGUE_CR_CLK_STATUS_GPP_SHIFT 3U
319+#define ROGUE_CR_CLK_STATUS_GPP_CLRMSK 0xFFFFFFFFFFFFFFF7ULL
320+#define ROGUE_CR_CLK_STATUS_GPP_GATED 0x0000000000000000ULL
321+#define ROGUE_CR_CLK_STATUS_GPP_RUNNING 0x0000000000000008ULL
322+#define ROGUE_CR_CLK_STATUS_TE_SHIFT 2U
323+#define ROGUE_CR_CLK_STATUS_TE_CLRMSK 0xFFFFFFFFFFFFFFFBULL
324+#define ROGUE_CR_CLK_STATUS_TE_GATED 0x0000000000000000ULL
325+#define ROGUE_CR_CLK_STATUS_TE_RUNNING 0x0000000000000004ULL
326+#define ROGUE_CR_CLK_STATUS_TSP_SHIFT 1U
327+#define ROGUE_CR_CLK_STATUS_TSP_CLRMSK 0xFFFFFFFFFFFFFFFDULL
328+#define ROGUE_CR_CLK_STATUS_TSP_GATED 0x0000000000000000ULL
329+#define ROGUE_CR_CLK_STATUS_TSP_RUNNING 0x0000000000000002ULL
330+#define ROGUE_CR_CLK_STATUS_ISP_SHIFT 0U
331+#define ROGUE_CR_CLK_STATUS_ISP_CLRMSK 0xFFFFFFFFFFFFFFFEULL
332+#define ROGUE_CR_CLK_STATUS_ISP_GATED 0x0000000000000000ULL
333+#define ROGUE_CR_CLK_STATUS_ISP_RUNNING 0x0000000000000001ULL
334+
335+/* Register ROGUE_CR_CORE_ID */
336+#define ROGUE_CR_CORE_ID__PBVNC 0x0020U
337+#define ROGUE_CR_CORE_ID__PBVNC__MASKFULL 0xFFFFFFFFFFFFFFFFULL
338+#define ROGUE_CR_CORE_ID__PBVNC__BRANCH_ID_SHIFT 48U
339+#define ROGUE_CR_CORE_ID__PBVNC__BRANCH_ID_CLRMSK 0x0000FFFFFFFFFFFFULL
340+#define ROGUE_CR_CORE_ID__PBVNC__VERSION_ID_SHIFT 32U
341+#define ROGUE_CR_CORE_ID__PBVNC__VERSION_ID_CLRMSK 0xFFFF0000FFFFFFFFULL
342+#define ROGUE_CR_CORE_ID__PBVNC__NUMBER_OF_SCALABLE_UNITS_SHIFT 16U
343+#define ROGUE_CR_CORE_ID__PBVNC__NUMBER_OF_SCALABLE_UNITS_CLRMSK 0xFFFFFFFF0000FFFFULL
344+#define ROGUE_CR_CORE_ID__PBVNC__CONFIG_ID_SHIFT 0U
345+#define ROGUE_CR_CORE_ID__PBVNC__CONFIG_ID_CLRMSK 0xFFFFFFFFFFFF0000ULL
346+
347+/* Register ROGUE_CR_CORE_ID */
348+#define ROGUE_CR_CORE_ID 0x0018U
349+#define ROGUE_CR_CORE_ID_MASKFULL 0x00000000FFFFFFFFULL
350+#define ROGUE_CR_CORE_ID_ID_SHIFT 16U
351+#define ROGUE_CR_CORE_ID_ID_CLRMSK 0x0000FFFFU
352+#define ROGUE_CR_CORE_ID_CONFIG_SHIFT 0U
353+#define ROGUE_CR_CORE_ID_CONFIG_CLRMSK 0xFFFF0000U
354+
355+/* Register ROGUE_CR_CORE_REVISION */
356+#define ROGUE_CR_CORE_REVISION 0x0020U
357+#define ROGUE_CR_CORE_REVISION_MASKFULL 0x00000000FFFFFFFFULL
358+#define ROGUE_CR_CORE_REVISION_DESIGNER_SHIFT 24U
359+#define ROGUE_CR_CORE_REVISION_DESIGNER_CLRMSK 0x00FFFFFFU
360+#define ROGUE_CR_CORE_REVISION_MAJOR_SHIFT 16U
361+#define ROGUE_CR_CORE_REVISION_MAJOR_CLRMSK 0xFF00FFFFU
362+#define ROGUE_CR_CORE_REVISION_MINOR_SHIFT 8U
363+#define ROGUE_CR_CORE_REVISION_MINOR_CLRMSK 0xFFFF00FFU
364+#define ROGUE_CR_CORE_REVISION_MAINTENANCE_SHIFT 0U
365+#define ROGUE_CR_CORE_REVISION_MAINTENANCE_CLRMSK 0xFFFFFF00U
366+
367+/* Register ROGUE_CR_DESIGNER_REV_FIELD1 */
368+#define ROGUE_CR_DESIGNER_REV_FIELD1 0x0028U
369+#define ROGUE_CR_DESIGNER_REV_FIELD1_MASKFULL 0x00000000FFFFFFFFULL
370+#define ROGUE_CR_DESIGNER_REV_FIELD1_DESIGNER_REV_FIELD1_SHIFT 0U
371+#define ROGUE_CR_DESIGNER_REV_FIELD1_DESIGNER_REV_FIELD1_CLRMSK 0x00000000U
372+
373+/* Register ROGUE_CR_DESIGNER_REV_FIELD2 */
374+#define ROGUE_CR_DESIGNER_REV_FIELD2 0x0030U
375+#define ROGUE_CR_DESIGNER_REV_FIELD2_MASKFULL 0x00000000FFFFFFFFULL
376+#define ROGUE_CR_DESIGNER_REV_FIELD2_DESIGNER_REV_FIELD2_SHIFT 0U
377+#define ROGUE_CR_DESIGNER_REV_FIELD2_DESIGNER_REV_FIELD2_CLRMSK 0x00000000U
378+
379+/* Register ROGUE_CR_CHANGESET_NUMBER */
380+#define ROGUE_CR_CHANGESET_NUMBER 0x0040U
381+#define ROGUE_CR_CHANGESET_NUMBER_MASKFULL 0xFFFFFFFFFFFFFFFFULL
382+#define ROGUE_CR_CHANGESET_NUMBER_CHANGESET_NUMBER_SHIFT 0U
383+#define ROGUE_CR_CHANGESET_NUMBER_CHANGESET_NUMBER_CLRMSK 0x0000000000000000ULL
384+
385+/* Register ROGUE_CR_CLK_XTPLUS_CTRL */
386+#define ROGUE_CR_CLK_XTPLUS_CTRL 0x0080U
387+#define ROGUE_CR_CLK_XTPLUS_CTRL_MASKFULL 0x0000003FFFFF0000ULL
388+#define ROGUE_CR_CLK_XTPLUS_CTRL_TDM_SHIFT 36U
389+#define ROGUE_CR_CLK_XTPLUS_CTRL_TDM_CLRMSK 0xFFFFFFCFFFFFFFFFULL
390+#define ROGUE_CR_CLK_XTPLUS_CTRL_TDM_OFF 0x0000000000000000ULL
391+#define ROGUE_CR_CLK_XTPLUS_CTRL_TDM_ON 0x0000001000000000ULL
392+#define ROGUE_CR_CLK_XTPLUS_CTRL_TDM_AUTO 0x0000002000000000ULL
393+#define ROGUE_CR_CLK_XTPLUS_CTRL_ASTC_SHIFT 34U
394+#define ROGUE_CR_CLK_XTPLUS_CTRL_ASTC_CLRMSK 0xFFFFFFF3FFFFFFFFULL
395+#define ROGUE_CR_CLK_XTPLUS_CTRL_ASTC_OFF 0x0000000000000000ULL
396+#define ROGUE_CR_CLK_XTPLUS_CTRL_ASTC_ON 0x0000000400000000ULL
397+#define ROGUE_CR_CLK_XTPLUS_CTRL_ASTC_AUTO 0x0000000800000000ULL
398+#define ROGUE_CR_CLK_XTPLUS_CTRL_IPF_SHIFT 32U
399+#define ROGUE_CR_CLK_XTPLUS_CTRL_IPF_CLRMSK 0xFFFFFFFCFFFFFFFFULL
400+#define ROGUE_CR_CLK_XTPLUS_CTRL_IPF_OFF 0x0000000000000000ULL
401+#define ROGUE_CR_CLK_XTPLUS_CTRL_IPF_ON 0x0000000100000000ULL
402+#define ROGUE_CR_CLK_XTPLUS_CTRL_IPF_AUTO 0x0000000200000000ULL
403+#define ROGUE_CR_CLK_XTPLUS_CTRL_COMPUTE_SHIFT 30U
404+#define ROGUE_CR_CLK_XTPLUS_CTRL_COMPUTE_CLRMSK 0xFFFFFFFF3FFFFFFFULL
405+#define ROGUE_CR_CLK_XTPLUS_CTRL_COMPUTE_OFF 0x0000000000000000ULL
406+#define ROGUE_CR_CLK_XTPLUS_CTRL_COMPUTE_ON 0x0000000040000000ULL
407+#define ROGUE_CR_CLK_XTPLUS_CTRL_COMPUTE_AUTO 0x0000000080000000ULL
408+#define ROGUE_CR_CLK_XTPLUS_CTRL_PIXEL_SHIFT 28U
409+#define ROGUE_CR_CLK_XTPLUS_CTRL_PIXEL_CLRMSK 0xFFFFFFFFCFFFFFFFULL
410+#define ROGUE_CR_CLK_XTPLUS_CTRL_PIXEL_OFF 0x0000000000000000ULL
411+#define ROGUE_CR_CLK_XTPLUS_CTRL_PIXEL_ON 0x0000000010000000ULL
412+#define ROGUE_CR_CLK_XTPLUS_CTRL_PIXEL_AUTO 0x0000000020000000ULL
413+#define ROGUE_CR_CLK_XTPLUS_CTRL_VERTEX_SHIFT 26U
414+#define ROGUE_CR_CLK_XTPLUS_CTRL_VERTEX_CLRMSK 0xFFFFFFFFF3FFFFFFULL
415+#define ROGUE_CR_CLK_XTPLUS_CTRL_VERTEX_OFF 0x0000000000000000ULL
416+#define ROGUE_CR_CLK_XTPLUS_CTRL_VERTEX_ON 0x0000000004000000ULL
417+#define ROGUE_CR_CLK_XTPLUS_CTRL_VERTEX_AUTO 0x0000000008000000ULL
418+#define ROGUE_CR_CLK_XTPLUS_CTRL_USCPS_SHIFT 24U
419+#define ROGUE_CR_CLK_XTPLUS_CTRL_USCPS_CLRMSK 0xFFFFFFFFFCFFFFFFULL
420+#define ROGUE_CR_CLK_XTPLUS_CTRL_USCPS_OFF 0x0000000000000000ULL
421+#define ROGUE_CR_CLK_XTPLUS_CTRL_USCPS_ON 0x0000000001000000ULL
422+#define ROGUE_CR_CLK_XTPLUS_CTRL_USCPS_AUTO 0x0000000002000000ULL
423+#define ROGUE_CR_CLK_XTPLUS_CTRL_PDS_SHARED_SHIFT 22U
424+#define ROGUE_CR_CLK_XTPLUS_CTRL_PDS_SHARED_CLRMSK 0xFFFFFFFFFF3FFFFFULL
425+#define ROGUE_CR_CLK_XTPLUS_CTRL_PDS_SHARED_OFF 0x0000000000000000ULL
426+#define ROGUE_CR_CLK_XTPLUS_CTRL_PDS_SHARED_ON 0x0000000000400000ULL
427+#define ROGUE_CR_CLK_XTPLUS_CTRL_PDS_SHARED_AUTO 0x0000000000800000ULL
428+#define ROGUE_CR_CLK_XTPLUS_CTRL_BIF_BLACKPEARL_SHIFT 20U
429+#define ROGUE_CR_CLK_XTPLUS_CTRL_BIF_BLACKPEARL_CLRMSK 0xFFFFFFFFFFCFFFFFULL
430+#define ROGUE_CR_CLK_XTPLUS_CTRL_BIF_BLACKPEARL_OFF 0x0000000000000000ULL
431+#define ROGUE_CR_CLK_XTPLUS_CTRL_BIF_BLACKPEARL_ON 0x0000000000100000ULL
432+#define ROGUE_CR_CLK_XTPLUS_CTRL_BIF_BLACKPEARL_AUTO 0x0000000000200000ULL
433+#define ROGUE_CR_CLK_XTPLUS_CTRL_USC_SHARED_SHIFT 18U
434+#define ROGUE_CR_CLK_XTPLUS_CTRL_USC_SHARED_CLRMSK 0xFFFFFFFFFFF3FFFFULL
435+#define ROGUE_CR_CLK_XTPLUS_CTRL_USC_SHARED_OFF 0x0000000000000000ULL
436+#define ROGUE_CR_CLK_XTPLUS_CTRL_USC_SHARED_ON 0x0000000000040000ULL
437+#define ROGUE_CR_CLK_XTPLUS_CTRL_USC_SHARED_AUTO 0x0000000000080000ULL
438+#define ROGUE_CR_CLK_XTPLUS_CTRL_GEOMETRY_SHIFT 16U
439+#define ROGUE_CR_CLK_XTPLUS_CTRL_GEOMETRY_CLRMSK 0xFFFFFFFFFFFCFFFFULL
440+#define ROGUE_CR_CLK_XTPLUS_CTRL_GEOMETRY_OFF 0x0000000000000000ULL
441+#define ROGUE_CR_CLK_XTPLUS_CTRL_GEOMETRY_ON 0x0000000000010000ULL
442+#define ROGUE_CR_CLK_XTPLUS_CTRL_GEOMETRY_AUTO 0x0000000000020000ULL
443+
444+/* Register ROGUE_CR_CLK_XTPLUS_STATUS */
445+#define ROGUE_CR_CLK_XTPLUS_STATUS 0x0088U
446+#define ROGUE_CR_CLK_XTPLUS_STATUS_MASKFULL 0x00000000000007FFULL
447+#define ROGUE_CR_CLK_XTPLUS_STATUS_TDM_SHIFT 10U
448+#define ROGUE_CR_CLK_XTPLUS_STATUS_TDM_CLRMSK 0xFFFFFFFFFFFFFBFFULL
449+#define ROGUE_CR_CLK_XTPLUS_STATUS_TDM_GATED 0x0000000000000000ULL
450+#define ROGUE_CR_CLK_XTPLUS_STATUS_TDM_RUNNING 0x0000000000000400ULL
451+#define ROGUE_CR_CLK_XTPLUS_STATUS_IPF_SHIFT 9U
452+#define ROGUE_CR_CLK_XTPLUS_STATUS_IPF_CLRMSK 0xFFFFFFFFFFFFFDFFULL
453+#define ROGUE_CR_CLK_XTPLUS_STATUS_IPF_GATED 0x0000000000000000ULL
454+#define ROGUE_CR_CLK_XTPLUS_STATUS_IPF_RUNNING 0x0000000000000200ULL
455+#define ROGUE_CR_CLK_XTPLUS_STATUS_COMPUTE_SHIFT 8U
456+#define ROGUE_CR_CLK_XTPLUS_STATUS_COMPUTE_CLRMSK 0xFFFFFFFFFFFFFEFFULL
457+#define ROGUE_CR_CLK_XTPLUS_STATUS_COMPUTE_GATED 0x0000000000000000ULL
458+#define ROGUE_CR_CLK_XTPLUS_STATUS_COMPUTE_RUNNING 0x0000000000000100ULL
459+#define ROGUE_CR_CLK_XTPLUS_STATUS_ASTC_SHIFT 7U
460+#define ROGUE_CR_CLK_XTPLUS_STATUS_ASTC_CLRMSK 0xFFFFFFFFFFFFFF7FULL
461+#define ROGUE_CR_CLK_XTPLUS_STATUS_ASTC_GATED 0x0000000000000000ULL
462+#define ROGUE_CR_CLK_XTPLUS_STATUS_ASTC_RUNNING 0x0000000000000080ULL
463+#define ROGUE_CR_CLK_XTPLUS_STATUS_PIXEL_SHIFT 6U
464+#define ROGUE_CR_CLK_XTPLUS_STATUS_PIXEL_CLRMSK 0xFFFFFFFFFFFFFFBFULL
465+#define ROGUE_CR_CLK_XTPLUS_STATUS_PIXEL_GATED 0x0000000000000000ULL
466+#define ROGUE_CR_CLK_XTPLUS_STATUS_PIXEL_RUNNING 0x0000000000000040ULL
467+#define ROGUE_CR_CLK_XTPLUS_STATUS_VERTEX_SHIFT 5U
468+#define ROGUE_CR_CLK_XTPLUS_STATUS_VERTEX_CLRMSK 0xFFFFFFFFFFFFFFDFULL
469+#define ROGUE_CR_CLK_XTPLUS_STATUS_VERTEX_GATED 0x0000000000000000ULL
470+#define ROGUE_CR_CLK_XTPLUS_STATUS_VERTEX_RUNNING 0x0000000000000020ULL
471+#define ROGUE_CR_CLK_XTPLUS_STATUS_USCPS_SHIFT 4U
472+#define ROGUE_CR_CLK_XTPLUS_STATUS_USCPS_CLRMSK 0xFFFFFFFFFFFFFFEFULL
473+#define ROGUE_CR_CLK_XTPLUS_STATUS_USCPS_GATED 0x0000000000000000ULL
474+#define ROGUE_CR_CLK_XTPLUS_STATUS_USCPS_RUNNING 0x0000000000000010ULL
475+#define ROGUE_CR_CLK_XTPLUS_STATUS_PDS_SHARED_SHIFT 3U
476+#define ROGUE_CR_CLK_XTPLUS_STATUS_PDS_SHARED_CLRMSK 0xFFFFFFFFFFFFFFF7ULL
477+#define ROGUE_CR_CLK_XTPLUS_STATUS_PDS_SHARED_GATED 0x0000000000000000ULL
478+#define ROGUE_CR_CLK_XTPLUS_STATUS_PDS_SHARED_RUNNING 0x0000000000000008ULL
479+#define ROGUE_CR_CLK_XTPLUS_STATUS_BIF_BLACKPEARL_SHIFT 2U
480+#define ROGUE_CR_CLK_XTPLUS_STATUS_BIF_BLACKPEARL_CLRMSK 0xFFFFFFFFFFFFFFFBULL
481+#define ROGUE_CR_CLK_XTPLUS_STATUS_BIF_BLACKPEARL_GATED 0x0000000000000000ULL
482+#define ROGUE_CR_CLK_XTPLUS_STATUS_BIF_BLACKPEARL_RUNNING 0x0000000000000004ULL
483+#define ROGUE_CR_CLK_XTPLUS_STATUS_USC_SHARED_SHIFT 1U
484+#define ROGUE_CR_CLK_XTPLUS_STATUS_USC_SHARED_CLRMSK 0xFFFFFFFFFFFFFFFDULL
485+#define ROGUE_CR_CLK_XTPLUS_STATUS_USC_SHARED_GATED 0x0000000000000000ULL
486+#define ROGUE_CR_CLK_XTPLUS_STATUS_USC_SHARED_RUNNING 0x0000000000000002ULL
487+#define ROGUE_CR_CLK_XTPLUS_STATUS_GEOMETRY_SHIFT 0U
488+#define ROGUE_CR_CLK_XTPLUS_STATUS_GEOMETRY_CLRMSK 0xFFFFFFFFFFFFFFFEULL
489+#define ROGUE_CR_CLK_XTPLUS_STATUS_GEOMETRY_GATED 0x0000000000000000ULL
490+#define ROGUE_CR_CLK_XTPLUS_STATUS_GEOMETRY_RUNNING 0x0000000000000001ULL
491+
492+/* Register ROGUE_CR_SOFT_RESET */
493+#define ROGUE_CR_SOFT_RESET 0x0100U
494+#define ROGUE_CR_SOFT_RESET__PBE2_XE__MASKFULL 0xFFEFFFFFFFFFFC3DULL
495+#define ROGUE_CR_SOFT_RESET_MASKFULL 0x00E7FFFFFFFFFC3DULL
496+#define ROGUE_CR_SOFT_RESET_PHANTOM3_CORE_SHIFT 63U
497+#define ROGUE_CR_SOFT_RESET_PHANTOM3_CORE_CLRMSK 0x7FFFFFFFFFFFFFFFULL
498+#define ROGUE_CR_SOFT_RESET_PHANTOM3_CORE_EN 0x8000000000000000ULL
499+#define ROGUE_CR_SOFT_RESET_PHANTOM2_CORE_SHIFT 62U
500+#define ROGUE_CR_SOFT_RESET_PHANTOM2_CORE_CLRMSK 0xBFFFFFFFFFFFFFFFULL
501+#define ROGUE_CR_SOFT_RESET_PHANTOM2_CORE_EN 0x4000000000000000ULL
502+#define ROGUE_CR_SOFT_RESET_BERNADO2_CORE_SHIFT 61U
503+#define ROGUE_CR_SOFT_RESET_BERNADO2_CORE_CLRMSK 0xDFFFFFFFFFFFFFFFULL
504+#define ROGUE_CR_SOFT_RESET_BERNADO2_CORE_EN 0x2000000000000000ULL
505+#define ROGUE_CR_SOFT_RESET_JONES_CORE_SHIFT 60U
506+#define ROGUE_CR_SOFT_RESET_JONES_CORE_CLRMSK 0xEFFFFFFFFFFFFFFFULL
507+#define ROGUE_CR_SOFT_RESET_JONES_CORE_EN 0x1000000000000000ULL
508+#define ROGUE_CR_SOFT_RESET_TILING_CORE_SHIFT 59U
509+#define ROGUE_CR_SOFT_RESET_TILING_CORE_CLRMSK 0xF7FFFFFFFFFFFFFFULL
510+#define ROGUE_CR_SOFT_RESET_TILING_CORE_EN 0x0800000000000000ULL
511+#define ROGUE_CR_SOFT_RESET_TE3_SHIFT 58U
512+#define ROGUE_CR_SOFT_RESET_TE3_CLRMSK 0xFBFFFFFFFFFFFFFFULL
513+#define ROGUE_CR_SOFT_RESET_TE3_EN 0x0400000000000000ULL
514+#define ROGUE_CR_SOFT_RESET_VCE_SHIFT 57U
515+#define ROGUE_CR_SOFT_RESET_VCE_CLRMSK 0xFDFFFFFFFFFFFFFFULL
516+#define ROGUE_CR_SOFT_RESET_VCE_EN 0x0200000000000000ULL
517+#define ROGUE_CR_SOFT_RESET_VBS_SHIFT 56U
518+#define ROGUE_CR_SOFT_RESET_VBS_CLRMSK 0xFEFFFFFFFFFFFFFFULL
519+#define ROGUE_CR_SOFT_RESET_VBS_EN 0x0100000000000000ULL
520+#define ROGUE_CR_SOFT_RESET_DPX1_CORE_SHIFT 55U
521+#define ROGUE_CR_SOFT_RESET_DPX1_CORE_CLRMSK 0xFF7FFFFFFFFFFFFFULL
522+#define ROGUE_CR_SOFT_RESET_DPX1_CORE_EN 0x0080000000000000ULL
523+#define ROGUE_CR_SOFT_RESET_DPX0_CORE_SHIFT 54U
524+#define ROGUE_CR_SOFT_RESET_DPX0_CORE_CLRMSK 0xFFBFFFFFFFFFFFFFULL
525+#define ROGUE_CR_SOFT_RESET_DPX0_CORE_EN 0x0040000000000000ULL
526+#define ROGUE_CR_SOFT_RESET_FBA_SHIFT 53U
527+#define ROGUE_CR_SOFT_RESET_FBA_CLRMSK 0xFFDFFFFFFFFFFFFFULL
528+#define ROGUE_CR_SOFT_RESET_FBA_EN 0x0020000000000000ULL
529+#define ROGUE_CR_SOFT_RESET_FB_CDC_SHIFT 51U
530+#define ROGUE_CR_SOFT_RESET_FB_CDC_CLRMSK 0xFFF7FFFFFFFFFFFFULL
531+#define ROGUE_CR_SOFT_RESET_FB_CDC_EN 0x0008000000000000ULL
532+#define ROGUE_CR_SOFT_RESET_SH_SHIFT 50U
533+#define ROGUE_CR_SOFT_RESET_SH_CLRMSK 0xFFFBFFFFFFFFFFFFULL
534+#define ROGUE_CR_SOFT_RESET_SH_EN 0x0004000000000000ULL
535+#define ROGUE_CR_SOFT_RESET_VRDM_SHIFT 49U
536+#define ROGUE_CR_SOFT_RESET_VRDM_CLRMSK 0xFFFDFFFFFFFFFFFFULL
537+#define ROGUE_CR_SOFT_RESET_VRDM_EN 0x0002000000000000ULL
538+#define ROGUE_CR_SOFT_RESET_MCU_FBTC_SHIFT 48U
539+#define ROGUE_CR_SOFT_RESET_MCU_FBTC_CLRMSK 0xFFFEFFFFFFFFFFFFULL
540+#define ROGUE_CR_SOFT_RESET_MCU_FBTC_EN 0x0001000000000000ULL
541+#define ROGUE_CR_SOFT_RESET_PHANTOM1_CORE_SHIFT 47U
542+#define ROGUE_CR_SOFT_RESET_PHANTOM1_CORE_CLRMSK 0xFFFF7FFFFFFFFFFFULL
543+#define ROGUE_CR_SOFT_RESET_PHANTOM1_CORE_EN 0x0000800000000000ULL
544+#define ROGUE_CR_SOFT_RESET_PHANTOM0_CORE_SHIFT 46U
545+#define ROGUE_CR_SOFT_RESET_PHANTOM0_CORE_CLRMSK 0xFFFFBFFFFFFFFFFFULL
546+#define ROGUE_CR_SOFT_RESET_PHANTOM0_CORE_EN 0x0000400000000000ULL
547+#define ROGUE_CR_SOFT_RESET_BERNADO1_CORE_SHIFT 45U
548+#define ROGUE_CR_SOFT_RESET_BERNADO1_CORE_CLRMSK 0xFFFFDFFFFFFFFFFFULL
549+#define ROGUE_CR_SOFT_RESET_BERNADO1_CORE_EN 0x0000200000000000ULL
550+#define ROGUE_CR_SOFT_RESET_BERNADO0_CORE_SHIFT 44U
551+#define ROGUE_CR_SOFT_RESET_BERNADO0_CORE_CLRMSK 0xFFFFEFFFFFFFFFFFULL
552+#define ROGUE_CR_SOFT_RESET_BERNADO0_CORE_EN 0x0000100000000000ULL
553+#define ROGUE_CR_SOFT_RESET_IPP_SHIFT 43U
554+#define ROGUE_CR_SOFT_RESET_IPP_CLRMSK 0xFFFFF7FFFFFFFFFFULL
555+#define ROGUE_CR_SOFT_RESET_IPP_EN 0x0000080000000000ULL
556+#define ROGUE_CR_SOFT_RESET_BIF_TEXAS_SHIFT 42U
557+#define ROGUE_CR_SOFT_RESET_BIF_TEXAS_CLRMSK 0xFFFFFBFFFFFFFFFFULL
558+#define ROGUE_CR_SOFT_RESET_BIF_TEXAS_EN 0x0000040000000000ULL
559+#define ROGUE_CR_SOFT_RESET_TORNADO_CORE_SHIFT 41U
560+#define ROGUE_CR_SOFT_RESET_TORNADO_CORE_CLRMSK 0xFFFFFDFFFFFFFFFFULL
561+#define ROGUE_CR_SOFT_RESET_TORNADO_CORE_EN 0x0000020000000000ULL
562+#define ROGUE_CR_SOFT_RESET_DUST_H_CORE_SHIFT 40U
563+#define ROGUE_CR_SOFT_RESET_DUST_H_CORE_CLRMSK 0xFFFFFEFFFFFFFFFFULL
564+#define ROGUE_CR_SOFT_RESET_DUST_H_CORE_EN 0x0000010000000000ULL
565+#define ROGUE_CR_SOFT_RESET_DUST_G_CORE_SHIFT 39U
566+#define ROGUE_CR_SOFT_RESET_DUST_G_CORE_CLRMSK 0xFFFFFF7FFFFFFFFFULL
567+#define ROGUE_CR_SOFT_RESET_DUST_G_CORE_EN 0x0000008000000000ULL
568+#define ROGUE_CR_SOFT_RESET_DUST_F_CORE_SHIFT 38U
569+#define ROGUE_CR_SOFT_RESET_DUST_F_CORE_CLRMSK 0xFFFFFFBFFFFFFFFFULL
570+#define ROGUE_CR_SOFT_RESET_DUST_F_CORE_EN 0x0000004000000000ULL
571+#define ROGUE_CR_SOFT_RESET_DUST_E_CORE_SHIFT 37U
572+#define ROGUE_CR_SOFT_RESET_DUST_E_CORE_CLRMSK 0xFFFFFFDFFFFFFFFFULL
573+#define ROGUE_CR_SOFT_RESET_DUST_E_CORE_EN 0x0000002000000000ULL
574+#define ROGUE_CR_SOFT_RESET_DUST_D_CORE_SHIFT 36U
575+#define ROGUE_CR_SOFT_RESET_DUST_D_CORE_CLRMSK 0xFFFFFFEFFFFFFFFFULL
576+#define ROGUE_CR_SOFT_RESET_DUST_D_CORE_EN 0x0000001000000000ULL
577+#define ROGUE_CR_SOFT_RESET_DUST_C_CORE_SHIFT 35U
578+#define ROGUE_CR_SOFT_RESET_DUST_C_CORE_CLRMSK 0xFFFFFFF7FFFFFFFFULL
579+#define ROGUE_CR_SOFT_RESET_DUST_C_CORE_EN 0x0000000800000000ULL
580+#define ROGUE_CR_SOFT_RESET_MMU_SHIFT 34U
581+#define ROGUE_CR_SOFT_RESET_MMU_CLRMSK 0xFFFFFFFBFFFFFFFFULL
582+#define ROGUE_CR_SOFT_RESET_MMU_EN 0x0000000400000000ULL
583+#define ROGUE_CR_SOFT_RESET_BIF1_SHIFT 33U
584+#define ROGUE_CR_SOFT_RESET_BIF1_CLRMSK 0xFFFFFFFDFFFFFFFFULL
585+#define ROGUE_CR_SOFT_RESET_BIF1_EN 0x0000000200000000ULL
586+#define ROGUE_CR_SOFT_RESET_GARTEN_SHIFT 32U
587+#define ROGUE_CR_SOFT_RESET_GARTEN_CLRMSK 0xFFFFFFFEFFFFFFFFULL
588+#define ROGUE_CR_SOFT_RESET_GARTEN_EN 0x0000000100000000ULL
589+#define ROGUE_CR_SOFT_RESET_CPU_SHIFT 32U
590+#define ROGUE_CR_SOFT_RESET_CPU_CLRMSK 0xFFFFFFFEFFFFFFFFULL
591+#define ROGUE_CR_SOFT_RESET_CPU_EN 0x0000000100000000ULL
592+#define ROGUE_CR_SOFT_RESET_RASCAL_CORE_SHIFT 31U
593+#define ROGUE_CR_SOFT_RESET_RASCAL_CORE_CLRMSK 0xFFFFFFFF7FFFFFFFULL
594+#define ROGUE_CR_SOFT_RESET_RASCAL_CORE_EN 0x0000000080000000ULL
595+#define ROGUE_CR_SOFT_RESET_DUST_B_CORE_SHIFT 30U
596+#define ROGUE_CR_SOFT_RESET_DUST_B_CORE_CLRMSK 0xFFFFFFFFBFFFFFFFULL
597+#define ROGUE_CR_SOFT_RESET_DUST_B_CORE_EN 0x0000000040000000ULL
598+#define ROGUE_CR_SOFT_RESET_DUST_A_CORE_SHIFT 29U
599+#define ROGUE_CR_SOFT_RESET_DUST_A_CORE_CLRMSK 0xFFFFFFFFDFFFFFFFULL
600+#define ROGUE_CR_SOFT_RESET_DUST_A_CORE_EN 0x0000000020000000ULL
601+#define ROGUE_CR_SOFT_RESET_FB_TLCACHE_SHIFT 28U
602+#define ROGUE_CR_SOFT_RESET_FB_TLCACHE_CLRMSK 0xFFFFFFFFEFFFFFFFULL
603+#define ROGUE_CR_SOFT_RESET_FB_TLCACHE_EN 0x0000000010000000ULL
604+#define ROGUE_CR_SOFT_RESET_SLC_SHIFT 27U
605+#define ROGUE_CR_SOFT_RESET_SLC_CLRMSK 0xFFFFFFFFF7FFFFFFULL
606+#define ROGUE_CR_SOFT_RESET_SLC_EN 0x0000000008000000ULL
607+#define ROGUE_CR_SOFT_RESET_TLA_SHIFT 26U
608+#define ROGUE_CR_SOFT_RESET_TLA_CLRMSK 0xFFFFFFFFFBFFFFFFULL
609+#define ROGUE_CR_SOFT_RESET_TLA_EN 0x0000000004000000ULL
610+#define ROGUE_CR_SOFT_RESET_UVS_SHIFT 25U
611+#define ROGUE_CR_SOFT_RESET_UVS_CLRMSK 0xFFFFFFFFFDFFFFFFULL
612+#define ROGUE_CR_SOFT_RESET_UVS_EN 0x0000000002000000ULL
613+#define ROGUE_CR_SOFT_RESET_TE_SHIFT 24U
614+#define ROGUE_CR_SOFT_RESET_TE_CLRMSK 0xFFFFFFFFFEFFFFFFULL
615+#define ROGUE_CR_SOFT_RESET_TE_EN 0x0000000001000000ULL
616+#define ROGUE_CR_SOFT_RESET_GPP_SHIFT 23U
617+#define ROGUE_CR_SOFT_RESET_GPP_CLRMSK 0xFFFFFFFFFF7FFFFFULL
618+#define ROGUE_CR_SOFT_RESET_GPP_EN 0x0000000000800000ULL
619+#define ROGUE_CR_SOFT_RESET_FBDC_SHIFT 22U
620+#define ROGUE_CR_SOFT_RESET_FBDC_CLRMSK 0xFFFFFFFFFFBFFFFFULL
621+#define ROGUE_CR_SOFT_RESET_FBDC_EN 0x0000000000400000ULL
622+#define ROGUE_CR_SOFT_RESET_FBC_SHIFT 21U
623+#define ROGUE_CR_SOFT_RESET_FBC_CLRMSK 0xFFFFFFFFFFDFFFFFULL
624+#define ROGUE_CR_SOFT_RESET_FBC_EN 0x0000000000200000ULL
625+#define ROGUE_CR_SOFT_RESET_PM_SHIFT 20U
626+#define ROGUE_CR_SOFT_RESET_PM_CLRMSK 0xFFFFFFFFFFEFFFFFULL
627+#define ROGUE_CR_SOFT_RESET_PM_EN 0x0000000000100000ULL
628+#define ROGUE_CR_SOFT_RESET_PBE_SHIFT 19U
629+#define ROGUE_CR_SOFT_RESET_PBE_CLRMSK 0xFFFFFFFFFFF7FFFFULL
630+#define ROGUE_CR_SOFT_RESET_PBE_EN 0x0000000000080000ULL
631+#define ROGUE_CR_SOFT_RESET_USC_SHARED_SHIFT 18U
632+#define ROGUE_CR_SOFT_RESET_USC_SHARED_CLRMSK 0xFFFFFFFFFFFBFFFFULL
633+#define ROGUE_CR_SOFT_RESET_USC_SHARED_EN 0x0000000000040000ULL
634+#define ROGUE_CR_SOFT_RESET_MCU_L1_SHIFT 17U
635+#define ROGUE_CR_SOFT_RESET_MCU_L1_CLRMSK 0xFFFFFFFFFFFDFFFFULL
636+#define ROGUE_CR_SOFT_RESET_MCU_L1_EN 0x0000000000020000ULL
637+#define ROGUE_CR_SOFT_RESET_BIF_SHIFT 16U
638+#define ROGUE_CR_SOFT_RESET_BIF_CLRMSK 0xFFFFFFFFFFFEFFFFULL
639+#define ROGUE_CR_SOFT_RESET_BIF_EN 0x0000000000010000ULL
640+#define ROGUE_CR_SOFT_RESET_CDM_SHIFT 15U
641+#define ROGUE_CR_SOFT_RESET_CDM_CLRMSK 0xFFFFFFFFFFFF7FFFULL
642+#define ROGUE_CR_SOFT_RESET_CDM_EN 0x0000000000008000ULL
643+#define ROGUE_CR_SOFT_RESET_VDM_SHIFT 14U
644+#define ROGUE_CR_SOFT_RESET_VDM_CLRMSK 0xFFFFFFFFFFFFBFFFULL
645+#define ROGUE_CR_SOFT_RESET_VDM_EN 0x0000000000004000ULL
646+#define ROGUE_CR_SOFT_RESET_TESS_SHIFT 13U
647+#define ROGUE_CR_SOFT_RESET_TESS_CLRMSK 0xFFFFFFFFFFFFDFFFULL
648+#define ROGUE_CR_SOFT_RESET_TESS_EN 0x0000000000002000ULL
649+#define ROGUE_CR_SOFT_RESET_PDS_SHIFT 12U
650+#define ROGUE_CR_SOFT_RESET_PDS_CLRMSK 0xFFFFFFFFFFFFEFFFULL
651+#define ROGUE_CR_SOFT_RESET_PDS_EN 0x0000000000001000ULL
652+#define ROGUE_CR_SOFT_RESET_ISP_SHIFT 11U
653+#define ROGUE_CR_SOFT_RESET_ISP_CLRMSK 0xFFFFFFFFFFFFF7FFULL
654+#define ROGUE_CR_SOFT_RESET_ISP_EN 0x0000000000000800ULL
655+#define ROGUE_CR_SOFT_RESET_TSP_SHIFT 10U
656+#define ROGUE_CR_SOFT_RESET_TSP_CLRMSK 0xFFFFFFFFFFFFFBFFULL
657+#define ROGUE_CR_SOFT_RESET_TSP_EN 0x0000000000000400ULL
658+#define ROGUE_CR_SOFT_RESET_SYSARB_SHIFT 5U
659+#define ROGUE_CR_SOFT_RESET_SYSARB_CLRMSK 0xFFFFFFFFFFFFFFDFULL
660+#define ROGUE_CR_SOFT_RESET_SYSARB_EN 0x0000000000000020ULL
661+#define ROGUE_CR_SOFT_RESET_TPU_MCU_DEMUX_SHIFT 4U
662+#define ROGUE_CR_SOFT_RESET_TPU_MCU_DEMUX_CLRMSK 0xFFFFFFFFFFFFFFEFULL
663+#define ROGUE_CR_SOFT_RESET_TPU_MCU_DEMUX_EN 0x0000000000000010ULL
664+#define ROGUE_CR_SOFT_RESET_MCU_L0_SHIFT 3U
665+#define ROGUE_CR_SOFT_RESET_MCU_L0_CLRMSK 0xFFFFFFFFFFFFFFF7ULL
666+#define ROGUE_CR_SOFT_RESET_MCU_L0_EN 0x0000000000000008ULL
667+#define ROGUE_CR_SOFT_RESET_TPU_SHIFT 2U
668+#define ROGUE_CR_SOFT_RESET_TPU_CLRMSK 0xFFFFFFFFFFFFFFFBULL
669+#define ROGUE_CR_SOFT_RESET_TPU_EN 0x0000000000000004ULL
670+#define ROGUE_CR_SOFT_RESET_USC_SHIFT 0U
671+#define ROGUE_CR_SOFT_RESET_USC_CLRMSK 0xFFFFFFFFFFFFFFFEULL
672+#define ROGUE_CR_SOFT_RESET_USC_EN 0x0000000000000001ULL
673+
674+/* Register ROGUE_CR_SOFT_RESET2 */
675+#define ROGUE_CR_SOFT_RESET2 0x0108U
676+#define ROGUE_CR_SOFT_RESET2_MASKFULL 0x00000000001FFFFFULL
677+#define ROGUE_CR_SOFT_RESET2_SPFILTER_SHIFT 12U
678+#define ROGUE_CR_SOFT_RESET2_SPFILTER_CLRMSK 0xFFE00FFFU
679+#define ROGUE_CR_SOFT_RESET2_TDM_SHIFT 11U
680+#define ROGUE_CR_SOFT_RESET2_TDM_CLRMSK 0xFFFFF7FFU
681+#define ROGUE_CR_SOFT_RESET2_TDM_EN 0x00000800U
682+#define ROGUE_CR_SOFT_RESET2_ASTC_SHIFT 10U
683+#define ROGUE_CR_SOFT_RESET2_ASTC_CLRMSK 0xFFFFFBFFU
684+#define ROGUE_CR_SOFT_RESET2_ASTC_EN 0x00000400U
685+#define ROGUE_CR_SOFT_RESET2_BLACKPEARL_SHIFT 9U
686+#define ROGUE_CR_SOFT_RESET2_BLACKPEARL_CLRMSK 0xFFFFFDFFU
687+#define ROGUE_CR_SOFT_RESET2_BLACKPEARL_EN 0x00000200U
688+#define ROGUE_CR_SOFT_RESET2_USCPS_SHIFT 8U
689+#define ROGUE_CR_SOFT_RESET2_USCPS_CLRMSK 0xFFFFFEFFU
690+#define ROGUE_CR_SOFT_RESET2_USCPS_EN 0x00000100U
691+#define ROGUE_CR_SOFT_RESET2_IPF_SHIFT 7U
692+#define ROGUE_CR_SOFT_RESET2_IPF_CLRMSK 0xFFFFFF7FU
693+#define ROGUE_CR_SOFT_RESET2_IPF_EN 0x00000080U
694+#define ROGUE_CR_SOFT_RESET2_GEOMETRY_SHIFT 6U
695+#define ROGUE_CR_SOFT_RESET2_GEOMETRY_CLRMSK 0xFFFFFFBFU
696+#define ROGUE_CR_SOFT_RESET2_GEOMETRY_EN 0x00000040U
697+#define ROGUE_CR_SOFT_RESET2_USC_SHARED_SHIFT 5U
698+#define ROGUE_CR_SOFT_RESET2_USC_SHARED_CLRMSK 0xFFFFFFDFU
699+#define ROGUE_CR_SOFT_RESET2_USC_SHARED_EN 0x00000020U
700+#define ROGUE_CR_SOFT_RESET2_PDS_SHARED_SHIFT 4U
701+#define ROGUE_CR_SOFT_RESET2_PDS_SHARED_CLRMSK 0xFFFFFFEFU
702+#define ROGUE_CR_SOFT_RESET2_PDS_SHARED_EN 0x00000010U
703+#define ROGUE_CR_SOFT_RESET2_BIF_BLACKPEARL_SHIFT 3U
704+#define ROGUE_CR_SOFT_RESET2_BIF_BLACKPEARL_CLRMSK 0xFFFFFFF7U
705+#define ROGUE_CR_SOFT_RESET2_BIF_BLACKPEARL_EN 0x00000008U
706+#define ROGUE_CR_SOFT_RESET2_PIXEL_SHIFT 2U
707+#define ROGUE_CR_SOFT_RESET2_PIXEL_CLRMSK 0xFFFFFFFBU
708+#define ROGUE_CR_SOFT_RESET2_PIXEL_EN 0x00000004U
709+#define ROGUE_CR_SOFT_RESET2_CDM_SHIFT 1U
710+#define ROGUE_CR_SOFT_RESET2_CDM_CLRMSK 0xFFFFFFFDU
711+#define ROGUE_CR_SOFT_RESET2_CDM_EN 0x00000002U
712+#define ROGUE_CR_SOFT_RESET2_VERTEX_SHIFT 0U
713+#define ROGUE_CR_SOFT_RESET2_VERTEX_CLRMSK 0xFFFFFFFEU
714+#define ROGUE_CR_SOFT_RESET2_VERTEX_EN 0x00000001U
715+
716+/* Register ROGUE_CR_EVENT_STATUS */
717+#define ROGUE_CR_EVENT_STATUS 0x0130U
718+#define ROGUE_CR_EVENT_STATUS__ROGUEXE__MASKFULL 0x00000000E01DFFFFULL
719+#define ROGUE_CR_EVENT_STATUS__SIGNALS__MASKFULL 0x00000000E007FFFFULL
720+#define ROGUE_CR_EVENT_STATUS_MASKFULL 0x00000000FFFFFFFFULL
721+#define ROGUE_CR_EVENT_STATUS_TDM_FENCE_FINISHED_SHIFT 31U
722+#define ROGUE_CR_EVENT_STATUS_TDM_FENCE_FINISHED_CLRMSK 0x7FFFFFFFU
723+#define ROGUE_CR_EVENT_STATUS_TDM_FENCE_FINISHED_EN 0x80000000U
724+#define ROGUE_CR_EVENT_STATUS_TDM_BUFFER_STALL_SHIFT 30U
725+#define ROGUE_CR_EVENT_STATUS_TDM_BUFFER_STALL_CLRMSK 0xBFFFFFFFU
726+#define ROGUE_CR_EVENT_STATUS_TDM_BUFFER_STALL_EN 0x40000000U
727+#define ROGUE_CR_EVENT_STATUS_COMPUTE_SIGNAL_FAILURE_SHIFT 29U
728+#define ROGUE_CR_EVENT_STATUS_COMPUTE_SIGNAL_FAILURE_CLRMSK 0xDFFFFFFFU
729+#define ROGUE_CR_EVENT_STATUS_COMPUTE_SIGNAL_FAILURE_EN 0x20000000U
730+#define ROGUE_CR_EVENT_STATUS_DPX_OUT_OF_MEMORY_SHIFT 28U
731+#define ROGUE_CR_EVENT_STATUS_DPX_OUT_OF_MEMORY_CLRMSK 0xEFFFFFFFU
732+#define ROGUE_CR_EVENT_STATUS_DPX_OUT_OF_MEMORY_EN 0x10000000U
733+#define ROGUE_CR_EVENT_STATUS_DPX_MMU_PAGE_FAULT_SHIFT 27U
734+#define ROGUE_CR_EVENT_STATUS_DPX_MMU_PAGE_FAULT_CLRMSK 0xF7FFFFFFU
735+#define ROGUE_CR_EVENT_STATUS_DPX_MMU_PAGE_FAULT_EN 0x08000000U
736+#define ROGUE_CR_EVENT_STATUS_RPM_OUT_OF_MEMORY_SHIFT 26U
737+#define ROGUE_CR_EVENT_STATUS_RPM_OUT_OF_MEMORY_CLRMSK 0xFBFFFFFFU
738+#define ROGUE_CR_EVENT_STATUS_RPM_OUT_OF_MEMORY_EN 0x04000000U
739+#define ROGUE_CR_EVENT_STATUS_FBA_FC3_FINISHED_SHIFT 25U
740+#define ROGUE_CR_EVENT_STATUS_FBA_FC3_FINISHED_CLRMSK 0xFDFFFFFFU
741+#define ROGUE_CR_EVENT_STATUS_FBA_FC3_FINISHED_EN 0x02000000U
742+#define ROGUE_CR_EVENT_STATUS_FBA_FC2_FINISHED_SHIFT 24U
743+#define ROGUE_CR_EVENT_STATUS_FBA_FC2_FINISHED_CLRMSK 0xFEFFFFFFU
744+#define ROGUE_CR_EVENT_STATUS_FBA_FC2_FINISHED_EN 0x01000000U
745+#define ROGUE_CR_EVENT_STATUS_FBA_FC1_FINISHED_SHIFT 23U
746+#define ROGUE_CR_EVENT_STATUS_FBA_FC1_FINISHED_CLRMSK 0xFF7FFFFFU
747+#define ROGUE_CR_EVENT_STATUS_FBA_FC1_FINISHED_EN 0x00800000U
748+#define ROGUE_CR_EVENT_STATUS_FBA_FC0_FINISHED_SHIFT 22U
749+#define ROGUE_CR_EVENT_STATUS_FBA_FC0_FINISHED_CLRMSK 0xFFBFFFFFU
750+#define ROGUE_CR_EVENT_STATUS_FBA_FC0_FINISHED_EN 0x00400000U
751+#define ROGUE_CR_EVENT_STATUS_RDM_FC3_FINISHED_SHIFT 21U
752+#define ROGUE_CR_EVENT_STATUS_RDM_FC3_FINISHED_CLRMSK 0xFFDFFFFFU
753+#define ROGUE_CR_EVENT_STATUS_RDM_FC3_FINISHED_EN 0x00200000U
754+#define ROGUE_CR_EVENT_STATUS_RDM_FC2_FINISHED_SHIFT 20U
755+#define ROGUE_CR_EVENT_STATUS_RDM_FC2_FINISHED_CLRMSK 0xFFEFFFFFU
756+#define ROGUE_CR_EVENT_STATUS_RDM_FC2_FINISHED_EN 0x00100000U
757+#define ROGUE_CR_EVENT_STATUS_SAFETY_SHIFT 20U
758+#define ROGUE_CR_EVENT_STATUS_SAFETY_CLRMSK 0xFFEFFFFFU
759+#define ROGUE_CR_EVENT_STATUS_SAFETY_EN 0x00100000U
760+#define ROGUE_CR_EVENT_STATUS_RDM_FC1_FINISHED_SHIFT 19U
761+#define ROGUE_CR_EVENT_STATUS_RDM_FC1_FINISHED_CLRMSK 0xFFF7FFFFU
762+#define ROGUE_CR_EVENT_STATUS_RDM_FC1_FINISHED_EN 0x00080000U
763+#define ROGUE_CR_EVENT_STATUS_SLAVE_REQ_SHIFT 19U
764+#define ROGUE_CR_EVENT_STATUS_SLAVE_REQ_CLRMSK 0xFFF7FFFFU
765+#define ROGUE_CR_EVENT_STATUS_SLAVE_REQ_EN 0x00080000U
766+#define ROGUE_CR_EVENT_STATUS_RDM_FC0_FINISHED_SHIFT 18U
767+#define ROGUE_CR_EVENT_STATUS_RDM_FC0_FINISHED_CLRMSK 0xFFFBFFFFU
768+#define ROGUE_CR_EVENT_STATUS_RDM_FC0_FINISHED_EN 0x00040000U
769+#define ROGUE_CR_EVENT_STATUS_TDM_CONTEXT_STORE_FINISHED_SHIFT 18U
770+#define ROGUE_CR_EVENT_STATUS_TDM_CONTEXT_STORE_FINISHED_CLRMSK 0xFFFBFFFFU
771+#define ROGUE_CR_EVENT_STATUS_TDM_CONTEXT_STORE_FINISHED_EN 0x00040000U
772+#define ROGUE_CR_EVENT_STATUS_SHG_FINISHED_SHIFT 17U
773+#define ROGUE_CR_EVENT_STATUS_SHG_FINISHED_CLRMSK 0xFFFDFFFFU
774+#define ROGUE_CR_EVENT_STATUS_SHG_FINISHED_EN 0x00020000U
775+#define ROGUE_CR_EVENT_STATUS_SPFILTER_SIGNAL_UPDATE_SHIFT 17U
776+#define ROGUE_CR_EVENT_STATUS_SPFILTER_SIGNAL_UPDATE_CLRMSK 0xFFFDFFFFU
777+#define ROGUE_CR_EVENT_STATUS_SPFILTER_SIGNAL_UPDATE_EN 0x00020000U
778+#define ROGUE_CR_EVENT_STATUS_COMPUTE_BUFFER_STALL_SHIFT 16U
779+#define ROGUE_CR_EVENT_STATUS_COMPUTE_BUFFER_STALL_CLRMSK 0xFFFEFFFFU
780+#define ROGUE_CR_EVENT_STATUS_COMPUTE_BUFFER_STALL_EN 0x00010000U
781+#define ROGUE_CR_EVENT_STATUS_USC_TRIGGER_SHIFT 15U
782+#define ROGUE_CR_EVENT_STATUS_USC_TRIGGER_CLRMSK 0xFFFF7FFFU
783+#define ROGUE_CR_EVENT_STATUS_USC_TRIGGER_EN 0x00008000U
784+#define ROGUE_CR_EVENT_STATUS_ZLS_FINISHED_SHIFT 14U
785+#define ROGUE_CR_EVENT_STATUS_ZLS_FINISHED_CLRMSK 0xFFFFBFFFU
786+#define ROGUE_CR_EVENT_STATUS_ZLS_FINISHED_EN 0x00004000U
787+#define ROGUE_CR_EVENT_STATUS_GPIO_ACK_SHIFT 13U
788+#define ROGUE_CR_EVENT_STATUS_GPIO_ACK_CLRMSK 0xFFFFDFFFU
789+#define ROGUE_CR_EVENT_STATUS_GPIO_ACK_EN 0x00002000U
790+#define ROGUE_CR_EVENT_STATUS_GPIO_REQ_SHIFT 12U
791+#define ROGUE_CR_EVENT_STATUS_GPIO_REQ_CLRMSK 0xFFFFEFFFU
792+#define ROGUE_CR_EVENT_STATUS_GPIO_REQ_EN 0x00001000U
793+#define ROGUE_CR_EVENT_STATUS_POWER_ABORT_SHIFT 11U
794+#define ROGUE_CR_EVENT_STATUS_POWER_ABORT_CLRMSK 0xFFFFF7FFU
795+#define ROGUE_CR_EVENT_STATUS_POWER_ABORT_EN 0x00000800U
796+#define ROGUE_CR_EVENT_STATUS_POWER_COMPLETE_SHIFT 10U
797+#define ROGUE_CR_EVENT_STATUS_POWER_COMPLETE_CLRMSK 0xFFFFFBFFU
798+#define ROGUE_CR_EVENT_STATUS_POWER_COMPLETE_EN 0x00000400U
799+#define ROGUE_CR_EVENT_STATUS_MMU_PAGE_FAULT_SHIFT 9U
800+#define ROGUE_CR_EVENT_STATUS_MMU_PAGE_FAULT_CLRMSK 0xFFFFFDFFU
801+#define ROGUE_CR_EVENT_STATUS_MMU_PAGE_FAULT_EN 0x00000200U
802+#define ROGUE_CR_EVENT_STATUS_PM_3D_MEM_FREE_SHIFT 8U
803+#define ROGUE_CR_EVENT_STATUS_PM_3D_MEM_FREE_CLRMSK 0xFFFFFEFFU
804+#define ROGUE_CR_EVENT_STATUS_PM_3D_MEM_FREE_EN 0x00000100U
805+#define ROGUE_CR_EVENT_STATUS_PM_OUT_OF_MEMORY_SHIFT 7U
806+#define ROGUE_CR_EVENT_STATUS_PM_OUT_OF_MEMORY_CLRMSK 0xFFFFFF7FU
807+#define ROGUE_CR_EVENT_STATUS_PM_OUT_OF_MEMORY_EN 0x00000080U
808+#define ROGUE_CR_EVENT_STATUS_TA_TERMINATE_SHIFT 6U
809+#define ROGUE_CR_EVENT_STATUS_TA_TERMINATE_CLRMSK 0xFFFFFFBFU
810+#define ROGUE_CR_EVENT_STATUS_TA_TERMINATE_EN 0x00000040U
811+#define ROGUE_CR_EVENT_STATUS_TA_FINISHED_SHIFT 5U
812+#define ROGUE_CR_EVENT_STATUS_TA_FINISHED_CLRMSK 0xFFFFFFDFU
813+#define ROGUE_CR_EVENT_STATUS_TA_FINISHED_EN 0x00000020U
814+#define ROGUE_CR_EVENT_STATUS_ISP_END_MACROTILE_SHIFT 4U
815+#define ROGUE_CR_EVENT_STATUS_ISP_END_MACROTILE_CLRMSK 0xFFFFFFEFU
816+#define ROGUE_CR_EVENT_STATUS_ISP_END_MACROTILE_EN 0x00000010U
817+#define ROGUE_CR_EVENT_STATUS_PIXELBE_END_RENDER_SHIFT 3U
818+#define ROGUE_CR_EVENT_STATUS_PIXELBE_END_RENDER_CLRMSK 0xFFFFFFF7U
819+#define ROGUE_CR_EVENT_STATUS_PIXELBE_END_RENDER_EN 0x00000008U
820+#define ROGUE_CR_EVENT_STATUS_COMPUTE_FINISHED_SHIFT 2U
821+#define ROGUE_CR_EVENT_STATUS_COMPUTE_FINISHED_CLRMSK 0xFFFFFFFBU
822+#define ROGUE_CR_EVENT_STATUS_COMPUTE_FINISHED_EN 0x00000004U
823+#define ROGUE_CR_EVENT_STATUS_KERNEL_FINISHED_SHIFT 1U
824+#define ROGUE_CR_EVENT_STATUS_KERNEL_FINISHED_CLRMSK 0xFFFFFFFDU
825+#define ROGUE_CR_EVENT_STATUS_KERNEL_FINISHED_EN 0x00000002U
826+#define ROGUE_CR_EVENT_STATUS_TLA_COMPLETE_SHIFT 0U
827+#define ROGUE_CR_EVENT_STATUS_TLA_COMPLETE_CLRMSK 0xFFFFFFFEU
828+#define ROGUE_CR_EVENT_STATUS_TLA_COMPLETE_EN 0x00000001U
829+
830+/* Register ROGUE_CR_TIMER */
831+#define ROGUE_CR_TIMER 0x0160U
832+#define ROGUE_CR_TIMER_MASKFULL 0x8000FFFFFFFFFFFFULL
833+#define ROGUE_CR_TIMER_BIT31_SHIFT 63U
834+#define ROGUE_CR_TIMER_BIT31_CLRMSK 0x7FFFFFFFFFFFFFFFULL
835+#define ROGUE_CR_TIMER_BIT31_EN 0x8000000000000000ULL
836+#define ROGUE_CR_TIMER_VALUE_SHIFT 0U
837+#define ROGUE_CR_TIMER_VALUE_CLRMSK 0xFFFF000000000000ULL
838+
839+/* Register ROGUE_CR_TLA_STATUS */
840+#define ROGUE_CR_TLA_STATUS 0x0178U
841+#define ROGUE_CR_TLA_STATUS_MASKFULL 0xFFFFFFFFFFFFFFFFULL
842+#define ROGUE_CR_TLA_STATUS_BLIT_COUNT_SHIFT 39U
843+#define ROGUE_CR_TLA_STATUS_BLIT_COUNT_CLRMSK 0x0000007FFFFFFFFFULL
844+#define ROGUE_CR_TLA_STATUS_REQUEST_SHIFT 7U
845+#define ROGUE_CR_TLA_STATUS_REQUEST_CLRMSK 0xFFFFFF800000007FULL
846+#define ROGUE_CR_TLA_STATUS_FIFO_FULLNESS_SHIFT 1U
847+#define ROGUE_CR_TLA_STATUS_FIFO_FULLNESS_CLRMSK 0xFFFFFFFFFFFFFF81ULL
848+#define ROGUE_CR_TLA_STATUS_BUSY_SHIFT 0U
849+#define ROGUE_CR_TLA_STATUS_BUSY_CLRMSK 0xFFFFFFFFFFFFFFFEULL
850+#define ROGUE_CR_TLA_STATUS_BUSY_EN 0x0000000000000001ULL
851+
852+/* Register ROGUE_CR_PM_PARTIAL_RENDER_ENABLE */
853+#define ROGUE_CR_PM_PARTIAL_RENDER_ENABLE 0x0338U
854+#define ROGUE_CR_PM_PARTIAL_RENDER_ENABLE_MASKFULL 0x0000000000000001ULL
855+#define ROGUE_CR_PM_PARTIAL_RENDER_ENABLE_OP_SHIFT 0U
856+#define ROGUE_CR_PM_PARTIAL_RENDER_ENABLE_OP_CLRMSK 0xFFFFFFFEU
857+#define ROGUE_CR_PM_PARTIAL_RENDER_ENABLE_OP_EN 0x00000001U
858+
859+/* Register ROGUE_CR_SIDEKICK_IDLE */
860+#define ROGUE_CR_SIDEKICK_IDLE 0x03C8U
861+#define ROGUE_CR_SIDEKICK_IDLE_MASKFULL 0x000000000000007FULL
862+#define ROGUE_CR_SIDEKICK_IDLE_FB_CDC_SHIFT 6U
863+#define ROGUE_CR_SIDEKICK_IDLE_FB_CDC_CLRMSK 0xFFFFFFBFU
864+#define ROGUE_CR_SIDEKICK_IDLE_FB_CDC_EN 0x00000040U
865+#define ROGUE_CR_SIDEKICK_IDLE_MMU_SHIFT 5U
866+#define ROGUE_CR_SIDEKICK_IDLE_MMU_CLRMSK 0xFFFFFFDFU
867+#define ROGUE_CR_SIDEKICK_IDLE_MMU_EN 0x00000020U
868+#define ROGUE_CR_SIDEKICK_IDLE_BIF128_SHIFT 4U
869+#define ROGUE_CR_SIDEKICK_IDLE_BIF128_CLRMSK 0xFFFFFFEFU
870+#define ROGUE_CR_SIDEKICK_IDLE_BIF128_EN 0x00000010U
871+#define ROGUE_CR_SIDEKICK_IDLE_TLA_SHIFT 3U
872+#define ROGUE_CR_SIDEKICK_IDLE_TLA_CLRMSK 0xFFFFFFF7U
873+#define ROGUE_CR_SIDEKICK_IDLE_TLA_EN 0x00000008U
874+#define ROGUE_CR_SIDEKICK_IDLE_GARTEN_SHIFT 2U
875+#define ROGUE_CR_SIDEKICK_IDLE_GARTEN_CLRMSK 0xFFFFFFFBU
876+#define ROGUE_CR_SIDEKICK_IDLE_GARTEN_EN 0x00000004U
877+#define ROGUE_CR_SIDEKICK_IDLE_HOSTIF_SHIFT 1U
878+#define ROGUE_CR_SIDEKICK_IDLE_HOSTIF_CLRMSK 0xFFFFFFFDU
879+#define ROGUE_CR_SIDEKICK_IDLE_HOSTIF_EN 0x00000002U
880+#define ROGUE_CR_SIDEKICK_IDLE_SOCIF_SHIFT 0U
881+#define ROGUE_CR_SIDEKICK_IDLE_SOCIF_CLRMSK 0xFFFFFFFEU
882+#define ROGUE_CR_SIDEKICK_IDLE_SOCIF_EN 0x00000001U
883+
884+/* Register ROGUE_CR_MARS_IDLE */
885+#define ROGUE_CR_MARS_IDLE 0x08F8U
886+#define ROGUE_CR_MARS_IDLE_MASKFULL 0x0000000000000007ULL
887+#define ROGUE_CR_MARS_IDLE_MH_SYSARB0_SHIFT 2U
888+#define ROGUE_CR_MARS_IDLE_MH_SYSARB0_CLRMSK 0xFFFFFFFBU
889+#define ROGUE_CR_MARS_IDLE_MH_SYSARB0_EN 0x00000004U
890+#define ROGUE_CR_MARS_IDLE_CPU_SHIFT 1U
891+#define ROGUE_CR_MARS_IDLE_CPU_CLRMSK 0xFFFFFFFDU
892+#define ROGUE_CR_MARS_IDLE_CPU_EN 0x00000002U
893+#define ROGUE_CR_MARS_IDLE_SOCIF_SHIFT 0U
894+#define ROGUE_CR_MARS_IDLE_SOCIF_CLRMSK 0xFFFFFFFEU
895+#define ROGUE_CR_MARS_IDLE_SOCIF_EN 0x00000001U
896+
897+/* Register ROGUE_CR_VDM_CONTEXT_STORE_STATUS */
898+#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS 0x0430U
899+#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_MASKFULL 0x00000000000000F3ULL
900+#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_LAST_PIPE_SHIFT 4U
901+#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_LAST_PIPE_CLRMSK 0xFFFFFF0FU
902+#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_NEED_RESUME_SHIFT 1U
903+#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_NEED_RESUME_CLRMSK 0xFFFFFFFDU
904+#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_NEED_RESUME_EN 0x00000002U
905+#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_COMPLETE_SHIFT 0U
906+#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_COMPLETE_CLRMSK 0xFFFFFFFEU
907+#define ROGUE_CR_VDM_CONTEXT_STORE_STATUS_COMPLETE_EN 0x00000001U
908+
909+/* Register ROGUE_CR_VDM_CONTEXT_STORE_TASK0 */
910+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK0 0x0438U
911+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK0_MASKFULL 0xFFFFFFFFFFFFFFFFULL
912+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK0_PDS_STATE1_SHIFT 32U
913+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK0_PDS_STATE1_CLRMSK 0x00000000FFFFFFFFULL
914+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK0_PDS_STATE0_SHIFT 0U
915+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK0_PDS_STATE0_CLRMSK 0xFFFFFFFF00000000ULL
916+
917+/* Register ROGUE_CR_VDM_CONTEXT_STORE_TASK1 */
918+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK1 0x0440U
919+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK1_MASKFULL 0x00000000FFFFFFFFULL
920+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK1_PDS_STATE2_SHIFT 0U
921+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK1_PDS_STATE2_CLRMSK 0x00000000U
922+
923+/* Register ROGUE_CR_VDM_CONTEXT_STORE_TASK2 */
924+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK2 0x0448U
925+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK2_MASKFULL 0xFFFFFFFFFFFFFFFFULL
926+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK2_STREAM_OUT2_SHIFT 32U
927+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK2_STREAM_OUT2_CLRMSK 0x00000000FFFFFFFFULL
928+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK2_STREAM_OUT1_SHIFT 0U
929+#define ROGUE_CR_VDM_CONTEXT_STORE_TASK2_STREAM_OUT1_CLRMSK 0xFFFFFFFF00000000ULL
930+
931+/* Register ROGUE_CR_VDM_CONTEXT_RESUME_TASK0 */
932+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK0 0x0450U
933+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK0_MASKFULL 0xFFFFFFFFFFFFFFFFULL
934+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK0_PDS_STATE1_SHIFT 32U
935+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK0_PDS_STATE1_CLRMSK 0x00000000FFFFFFFFULL
936+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK0_PDS_STATE0_SHIFT 0U
937+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK0_PDS_STATE0_CLRMSK 0xFFFFFFFF00000000ULL
938+
939+/* Register ROGUE_CR_VDM_CONTEXT_RESUME_TASK1 */
940+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK1 0x0458U
941+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK1_MASKFULL 0x00000000FFFFFFFFULL
942+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK1_PDS_STATE2_SHIFT 0U
943+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK1_PDS_STATE2_CLRMSK 0x00000000U
944+
945+/* Register ROGUE_CR_VDM_CONTEXT_RESUME_TASK2 */
946+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK2 0x0460U
947+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK2_MASKFULL 0xFFFFFFFFFFFFFFFFULL
948+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK2_STREAM_OUT2_SHIFT 32U
949+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK2_STREAM_OUT2_CLRMSK 0x00000000FFFFFFFFULL
950+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK2_STREAM_OUT1_SHIFT 0U
951+#define ROGUE_CR_VDM_CONTEXT_RESUME_TASK2_STREAM_OUT1_CLRMSK 0xFFFFFFFF00000000ULL
952+
953+/* Register ROGUE_CR_CDM_CONTEXT_STORE_STATUS */
954+#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS 0x04A0U
955+#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_MASKFULL 0x0000000000000003ULL
956+#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_NEED_RESUME_SHIFT 1U
957+#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_NEED_RESUME_CLRMSK 0xFFFFFFFDU
958+#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_NEED_RESUME_EN 0x00000002U
959+#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_COMPLETE_SHIFT 0U
960+#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_COMPLETE_CLRMSK 0xFFFFFFFEU
961+#define ROGUE_CR_CDM_CONTEXT_STORE_STATUS_COMPLETE_EN 0x00000001U
962+
963+/* Register ROGUE_CR_CDM_CONTEXT_PDS0 */
964+#define ROGUE_CR_CDM_CONTEXT_PDS0 0x04A8U
965+#define ROGUE_CR_CDM_CONTEXT_PDS0_MASKFULL 0xFFFFFFF0FFFFFFF0ULL
966+#define ROGUE_CR_CDM_CONTEXT_PDS0_DATA_ADDR_SHIFT 36U
967+#define ROGUE_CR_CDM_CONTEXT_PDS0_DATA_ADDR_CLRMSK 0x0000000FFFFFFFFFULL
968+#define ROGUE_CR_CDM_CONTEXT_PDS0_DATA_ADDR_ALIGNSHIFT 4U
969+#define ROGUE_CR_CDM_CONTEXT_PDS0_DATA_ADDR_ALIGNSIZE 16U
970+#define ROGUE_CR_CDM_CONTEXT_PDS0_CODE_ADDR_SHIFT 4U
971+#define ROGUE_CR_CDM_CONTEXT_PDS0_CODE_ADDR_CLRMSK 0xFFFFFFFF0000000FULL
972+#define ROGUE_CR_CDM_CONTEXT_PDS0_CODE_ADDR_ALIGNSHIFT 4U
973+#define ROGUE_CR_CDM_CONTEXT_PDS0_CODE_ADDR_ALIGNSIZE 16U
974+
975+/* Register ROGUE_CR_CDM_CONTEXT_PDS1 */
976+#define ROGUE_CR_CDM_CONTEXT_PDS1 0x04B0U
977+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__MASKFULL 0x000000007FFFFFFFULL
978+#define ROGUE_CR_CDM_CONTEXT_PDS1_MASKFULL 0x000000003FFFFFFFULL
979+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__PDS_SEQ_DEP_SHIFT 30U
980+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__PDS_SEQ_DEP_CLRMSK 0xBFFFFFFFU
981+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__PDS_SEQ_DEP_EN 0x40000000U
982+#define ROGUE_CR_CDM_CONTEXT_PDS1_PDS_SEQ_DEP_SHIFT 29U
983+#define ROGUE_CR_CDM_CONTEXT_PDS1_PDS_SEQ_DEP_CLRMSK 0xDFFFFFFFU
984+#define ROGUE_CR_CDM_CONTEXT_PDS1_PDS_SEQ_DEP_EN 0x20000000U
985+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__USC_SEQ_DEP_SHIFT 29U
986+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__USC_SEQ_DEP_CLRMSK 0xDFFFFFFFU
987+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__USC_SEQ_DEP_EN 0x20000000U
988+#define ROGUE_CR_CDM_CONTEXT_PDS1_USC_SEQ_DEP_SHIFT 28U
989+#define ROGUE_CR_CDM_CONTEXT_PDS1_USC_SEQ_DEP_CLRMSK 0xEFFFFFFFU
990+#define ROGUE_CR_CDM_CONTEXT_PDS1_USC_SEQ_DEP_EN 0x10000000U
991+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__TARGET_SHIFT 28U
992+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__TARGET_CLRMSK 0xEFFFFFFFU
993+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__TARGET_EN 0x10000000U
994+#define ROGUE_CR_CDM_CONTEXT_PDS1_TARGET_SHIFT 27U
995+#define ROGUE_CR_CDM_CONTEXT_PDS1_TARGET_CLRMSK 0xF7FFFFFFU
996+#define ROGUE_CR_CDM_CONTEXT_PDS1_TARGET_EN 0x08000000U
997+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__UNIFIED_SIZE_SHIFT 22U
998+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__UNIFIED_SIZE_CLRMSK 0xF03FFFFFU
999+#define ROGUE_CR_CDM_CONTEXT_PDS1_UNIFIED_SIZE_SHIFT 21U
1000+#define ROGUE_CR_CDM_CONTEXT_PDS1_UNIFIED_SIZE_CLRMSK 0xF81FFFFFU
1001+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__COMMON_SHARED_SHIFT 21U
1002+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__COMMON_SHARED_CLRMSK 0xFFDFFFFFU
1003+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__COMMON_SHARED_EN 0x00200000U
1004+#define ROGUE_CR_CDM_CONTEXT_PDS1_COMMON_SHARED_SHIFT 20U
1005+#define ROGUE_CR_CDM_CONTEXT_PDS1_COMMON_SHARED_CLRMSK 0xFFEFFFFFU
1006+#define ROGUE_CR_CDM_CONTEXT_PDS1_COMMON_SHARED_EN 0x00100000U
1007+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__COMMON_SIZE_SHIFT 12U
1008+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__COMMON_SIZE_CLRMSK 0xFFE00FFFU
1009+#define ROGUE_CR_CDM_CONTEXT_PDS1_COMMON_SIZE_SHIFT 11U
1010+#define ROGUE_CR_CDM_CONTEXT_PDS1_COMMON_SIZE_CLRMSK 0xFFF007FFU
1011+#define ROGUE_CR_CDM_CONTEXT_PDS1_TEMP_SIZE_SHIFT 7U
1012+#define ROGUE_CR_CDM_CONTEXT_PDS1_TEMP_SIZE_CLRMSK 0xFFFFF87FU
1013+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__TEMP_SIZE_SHIFT 7U
1014+#define ROGUE_CR_CDM_CONTEXT_PDS1__TEMPSIZE8__TEMP_SIZE_CLRMSK 0xFFFFF07FU
1015+#define ROGUE_CR_CDM_CONTEXT_PDS1_DATA_SIZE_SHIFT 1U
1016+#define ROGUE_CR_CDM_CONTEXT_PDS1_DATA_SIZE_CLRMSK 0xFFFFFF81U
1017+#define ROGUE_CR_CDM_CONTEXT_PDS1_FENCE_SHIFT 0U
1018+#define ROGUE_CR_CDM_CONTEXT_PDS1_FENCE_CLRMSK 0xFFFFFFFEU
1019+#define ROGUE_CR_CDM_CONTEXT_PDS1_FENCE_EN 0x00000001U
1020+
1021+/* Register ROGUE_CR_CDM_TERMINATE_PDS */
1022+#define ROGUE_CR_CDM_TERMINATE_PDS 0x04B8U
1023+#define ROGUE_CR_CDM_TERMINATE_PDS_MASKFULL 0xFFFFFFF0FFFFFFF0ULL
1024+#define ROGUE_CR_CDM_TERMINATE_PDS_DATA_ADDR_SHIFT 36U
1025+#define ROGUE_CR_CDM_TERMINATE_PDS_DATA_ADDR_CLRMSK 0x0000000FFFFFFFFFULL
1026+#define ROGUE_CR_CDM_TERMINATE_PDS_DATA_ADDR_ALIGNSHIFT 4U
1027+#define ROGUE_CR_CDM_TERMINATE_PDS_DATA_ADDR_ALIGNSIZE 16U
1028+#define ROGUE_CR_CDM_TERMINATE_PDS_CODE_ADDR_SHIFT 4U
1029+#define ROGUE_CR_CDM_TERMINATE_PDS_CODE_ADDR_CLRMSK 0xFFFFFFFF0000000FULL
1030+#define ROGUE_CR_CDM_TERMINATE_PDS_CODE_ADDR_ALIGNSHIFT 4U
1031+#define ROGUE_CR_CDM_TERMINATE_PDS_CODE_ADDR_ALIGNSIZE 16U
1032+
1033+/* Register ROGUE_CR_CDM_TERMINATE_PDS1 */
1034+#define ROGUE_CR_CDM_TERMINATE_PDS1 0x04C0U
1035+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__MASKFULL 0x000000007FFFFFFFULL
1036+#define ROGUE_CR_CDM_TERMINATE_PDS1_MASKFULL 0x000000003FFFFFFFULL
1037+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__PDS_SEQ_DEP_SHIFT 30U
1038+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__PDS_SEQ_DEP_CLRMSK 0xBFFFFFFFU
1039+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__PDS_SEQ_DEP_EN 0x40000000U
1040+#define ROGUE_CR_CDM_TERMINATE_PDS1_PDS_SEQ_DEP_SHIFT 29U
1041+#define ROGUE_CR_CDM_TERMINATE_PDS1_PDS_SEQ_DEP_CLRMSK 0xDFFFFFFFU
1042+#define ROGUE_CR_CDM_TERMINATE_PDS1_PDS_SEQ_DEP_EN 0x20000000U
1043+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__USC_SEQ_DEP_SHIFT 29U
1044+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__USC_SEQ_DEP_CLRMSK 0xDFFFFFFFU
1045+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__USC_SEQ_DEP_EN 0x20000000U
1046+#define ROGUE_CR_CDM_TERMINATE_PDS1_USC_SEQ_DEP_SHIFT 28U
1047+#define ROGUE_CR_CDM_TERMINATE_PDS1_USC_SEQ_DEP_CLRMSK 0xEFFFFFFFU
1048+#define ROGUE_CR_CDM_TERMINATE_PDS1_USC_SEQ_DEP_EN 0x10000000U
1049+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__TARGET_SHIFT 28U
1050+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__TARGET_CLRMSK 0xEFFFFFFFU
1051+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__TARGET_EN 0x10000000U
1052+#define ROGUE_CR_CDM_TERMINATE_PDS1_TARGET_SHIFT 27U
1053+#define ROGUE_CR_CDM_TERMINATE_PDS1_TARGET_CLRMSK 0xF7FFFFFFU
1054+#define ROGUE_CR_CDM_TERMINATE_PDS1_TARGET_EN 0x08000000U
1055+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__UNIFIED_SIZE_SHIFT 22U
1056+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__UNIFIED_SIZE_CLRMSK 0xF03FFFFFU
1057+#define ROGUE_CR_CDM_TERMINATE_PDS1_UNIFIED_SIZE_SHIFT 21U
1058+#define ROGUE_CR_CDM_TERMINATE_PDS1_UNIFIED_SIZE_CLRMSK 0xF81FFFFFU
1059+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__COMMON_SHARED_SHIFT 21U
1060+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__COMMON_SHARED_CLRMSK 0xFFDFFFFFU
1061+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__COMMON_SHARED_EN 0x00200000U
1062+#define ROGUE_CR_CDM_TERMINATE_PDS1_COMMON_SHARED_SHIFT 20U
1063+#define ROGUE_CR_CDM_TERMINATE_PDS1_COMMON_SHARED_CLRMSK 0xFFEFFFFFU
1064+#define ROGUE_CR_CDM_TERMINATE_PDS1_COMMON_SHARED_EN 0x00100000U
1065+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__COMMON_SIZE_SHIFT 12U
1066+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__COMMON_SIZE_CLRMSK 0xFFE00FFFU
1067+#define ROGUE_CR_CDM_TERMINATE_PDS1_COMMON_SIZE_SHIFT 11U
1068+#define ROGUE_CR_CDM_TERMINATE_PDS1_COMMON_SIZE_CLRMSK 0xFFF007FFU
1069+#define ROGUE_CR_CDM_TERMINATE_PDS1_TEMP_SIZE_SHIFT 7U
1070+#define ROGUE_CR_CDM_TERMINATE_PDS1_TEMP_SIZE_CLRMSK 0xFFFFF87FU
1071+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__TEMP_SIZE_SHIFT 7U
1072+#define ROGUE_CR_CDM_TERMINATE_PDS1__TEMPSIZE8__TEMP_SIZE_CLRMSK 0xFFFFF07FU
1073+#define ROGUE_CR_CDM_TERMINATE_PDS1_DATA_SIZE_SHIFT 1U
1074+#define ROGUE_CR_CDM_TERMINATE_PDS1_DATA_SIZE_CLRMSK 0xFFFFFF81U
1075+#define ROGUE_CR_CDM_TERMINATE_PDS1_FENCE_SHIFT 0U
1076+#define ROGUE_CR_CDM_TERMINATE_PDS1_FENCE_CLRMSK 0xFFFFFFFEU
1077+#define ROGUE_CR_CDM_TERMINATE_PDS1_FENCE_EN 0x00000001U
1078+
1079+/* Register ROGUE_CR_CDM_CONTEXT_LOAD_PDS0 */
1080+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0 0x04D8U
1081+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_MASKFULL 0xFFFFFFF0FFFFFFF0ULL
1082+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_DATA_ADDR_SHIFT 36U
1083+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_DATA_ADDR_CLRMSK 0x0000000FFFFFFFFFULL
1084+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_DATA_ADDR_ALIGNSHIFT 4U
1085+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_DATA_ADDR_ALIGNSIZE 16U
1086+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_CODE_ADDR_SHIFT 4U
1087+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_CODE_ADDR_CLRMSK 0xFFFFFFFF0000000FULL
1088+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_CODE_ADDR_ALIGNSHIFT 4U
1089+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS0_CODE_ADDR_ALIGNSIZE 16U
1090+
1091+/* Register ROGUE_CR_CDM_CONTEXT_LOAD_PDS1 */
1092+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1 0x04E0U
1093+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__MASKFULL 0x000000007FFFFFFFULL
1094+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_MASKFULL 0x000000003FFFFFFFULL
1095+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__PDS_SEQ_DEP_SHIFT 30U
1096+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__PDS_SEQ_DEP_CLRMSK 0xBFFFFFFFU
1097+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__PDS_SEQ_DEP_EN 0x40000000U
1098+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_PDS_SEQ_DEP_SHIFT 29U
1099+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_PDS_SEQ_DEP_CLRMSK 0xDFFFFFFFU
1100+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_PDS_SEQ_DEP_EN 0x20000000U
1101+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__USC_SEQ_DEP_SHIFT 29U
1102+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__USC_SEQ_DEP_CLRMSK 0xDFFFFFFFU
1103+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__USC_SEQ_DEP_EN 0x20000000U
1104+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_USC_SEQ_DEP_SHIFT 28U
1105+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_USC_SEQ_DEP_CLRMSK 0xEFFFFFFFU
1106+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_USC_SEQ_DEP_EN 0x10000000U
1107+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__TARGET_SHIFT 28U
1108+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__TARGET_CLRMSK 0xEFFFFFFFU
1109+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__TARGET_EN 0x10000000U
1110+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_TARGET_SHIFT 27U
1111+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_TARGET_CLRMSK 0xF7FFFFFFU
1112+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_TARGET_EN 0x08000000U
1113+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__UNIFIED_SIZE_SHIFT 22U
1114+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__UNIFIED_SIZE_CLRMSK 0xF03FFFFFU
1115+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_UNIFIED_SIZE_SHIFT 21U
1116+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_UNIFIED_SIZE_CLRMSK 0xF81FFFFFU
1117+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__COMMON_SHARED_SHIFT 21U
1118+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__COMMON_SHARED_CLRMSK 0xFFDFFFFFU
1119+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__COMMON_SHARED_EN 0x00200000U
1120+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_COMMON_SHARED_SHIFT 20U
1121+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_COMMON_SHARED_CLRMSK 0xFFEFFFFFU
1122+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_COMMON_SHARED_EN 0x00100000U
1123+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__COMMON_SIZE_SHIFT 12U
1124+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__COMMON_SIZE_CLRMSK 0xFFE00FFFU
1125+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_COMMON_SIZE_SHIFT 11U
1126+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_COMMON_SIZE_CLRMSK 0xFFF007FFU
1127+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_TEMP_SIZE_SHIFT 7U
1128+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_TEMP_SIZE_CLRMSK 0xFFFFF87FU
1129+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__TEMP_SIZE_SHIFT 7U
1130+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1__TEMPSIZE8__TEMP_SIZE_CLRMSK 0xFFFFF07FU
1131+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_DATA_SIZE_SHIFT 1U
1132+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_DATA_SIZE_CLRMSK 0xFFFFFF81U
1133+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_FENCE_SHIFT 0U
1134+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_FENCE_CLRMSK 0xFFFFFFFEU
1135+#define ROGUE_CR_CDM_CONTEXT_LOAD_PDS1_FENCE_EN 0x00000001U
1136+
1137+/* Register ROGUE_CR_MIPS_WRAPPER_CONFIG */
1138+#define ROGUE_CR_MIPS_WRAPPER_CONFIG 0x0810U
1139+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_MASKFULL 0x000001030F01FFFFULL
1140+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_FW_IDLE_ENABLE_SHIFT 40U
1141+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_FW_IDLE_ENABLE_CLRMSK 0xFFFFFEFFFFFFFFFFULL
1142+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_FW_IDLE_ENABLE_EN 0x0000010000000000ULL
1143+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_DISABLE_BOOT_SHIFT 33U
1144+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_DISABLE_BOOT_CLRMSK 0xFFFFFFFDFFFFFFFFULL
1145+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_DISABLE_BOOT_EN 0x0000000200000000ULL
1146+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_L2_CACHE_OFF_SHIFT 32U
1147+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_L2_CACHE_OFF_CLRMSK 0xFFFFFFFEFFFFFFFFULL
1148+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_L2_CACHE_OFF_EN 0x0000000100000000ULL
1149+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_OS_ID_SHIFT 25U
1150+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_OS_ID_CLRMSK 0xFFFFFFFFF1FFFFFFULL
1151+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_TRUSTED_SHIFT 24U
1152+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_TRUSTED_CLRMSK 0xFFFFFFFFFEFFFFFFULL
1153+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_TRUSTED_EN 0x0000000001000000ULL
1154+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_BOOT_ISA_MODE_SHIFT 16U
1155+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_BOOT_ISA_MODE_CLRMSK 0xFFFFFFFFFFFEFFFFULL
1156+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_BOOT_ISA_MODE_MIPS32 0x0000000000000000ULL
1157+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_BOOT_ISA_MODE_MICROMIPS 0x0000000000010000ULL
1158+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_REGBANK_BASE_ADDR_SHIFT 0U
1159+#define ROGUE_CR_MIPS_WRAPPER_CONFIG_REGBANK_BASE_ADDR_CLRMSK 0xFFFFFFFFFFFF0000ULL
1160+
1161+/* Register ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1 */
1162+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1 0x0818U
1163+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_MASKFULL 0x00000000FFFFF001ULL
1164+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_BASE_ADDR_IN_SHIFT 12U
1165+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1166+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_MODE_ENABLE_SHIFT 0U
1167+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1168+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG1_MODE_ENABLE_EN 0x0000000000000001ULL
1169+
1170+/* Register ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2 */
1171+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2 0x0820U
1172+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_MASKFULL 0x000000FFFFFFF1FFULL
1173+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_ADDR_OUT_SHIFT 12U
1174+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_ADDR_OUT_CLRMSK 0xFFFFFF0000000FFFULL
1175+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_OS_ID_SHIFT 6U
1176+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_OS_ID_CLRMSK 0xFFFFFFFFFFFFFE3FULL
1177+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_TRUSTED_SHIFT 5U
1178+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_TRUSTED_CLRMSK 0xFFFFFFFFFFFFFFDFULL
1179+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_TRUSTED_EN 0x0000000000000020ULL
1180+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_REGION_SIZE_POW2_SHIFT 0U
1181+#define ROGUE_CR_MIPS_ADDR_REMAP1_CONFIG2_REGION_SIZE_POW2_CLRMSK 0xFFFFFFFFFFFFFFE0ULL
1182+
1183+/* Register ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1 */
1184+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1 0x0828U
1185+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_MASKFULL 0x00000000FFFFF001ULL
1186+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_BASE_ADDR_IN_SHIFT 12U
1187+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1188+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_MODE_ENABLE_SHIFT 0U
1189+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1190+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG1_MODE_ENABLE_EN 0x0000000000000001ULL
1191+
1192+/* Register ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2 */
1193+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2 0x0830U
1194+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_MASKFULL 0x000000FFFFFFF1FFULL
1195+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_ADDR_OUT_SHIFT 12U
1196+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_ADDR_OUT_CLRMSK 0xFFFFFF0000000FFFULL
1197+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_OS_ID_SHIFT 6U
1198+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_OS_ID_CLRMSK 0xFFFFFFFFFFFFFE3FULL
1199+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_TRUSTED_SHIFT 5U
1200+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_TRUSTED_CLRMSK 0xFFFFFFFFFFFFFFDFULL
1201+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_TRUSTED_EN 0x0000000000000020ULL
1202+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_REGION_SIZE_POW2_SHIFT 0U
1203+#define ROGUE_CR_MIPS_ADDR_REMAP2_CONFIG2_REGION_SIZE_POW2_CLRMSK 0xFFFFFFFFFFFFFFE0ULL
1204+
1205+/* Register ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1 */
1206+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1 0x0838U
1207+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_MASKFULL 0x00000000FFFFF001ULL
1208+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_BASE_ADDR_IN_SHIFT 12U
1209+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1210+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_MODE_ENABLE_SHIFT 0U
1211+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1212+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG1_MODE_ENABLE_EN 0x0000000000000001ULL
1213+
1214+/* Register ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2 */
1215+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2 0x0840U
1216+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_MASKFULL 0x000000FFFFFFF1FFULL
1217+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_ADDR_OUT_SHIFT 12U
1218+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_ADDR_OUT_CLRMSK 0xFFFFFF0000000FFFULL
1219+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_OS_ID_SHIFT 6U
1220+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_OS_ID_CLRMSK 0xFFFFFFFFFFFFFE3FULL
1221+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_TRUSTED_SHIFT 5U
1222+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_TRUSTED_CLRMSK 0xFFFFFFFFFFFFFFDFULL
1223+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_TRUSTED_EN 0x0000000000000020ULL
1224+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_REGION_SIZE_POW2_SHIFT 0U
1225+#define ROGUE_CR_MIPS_ADDR_REMAP3_CONFIG2_REGION_SIZE_POW2_CLRMSK 0xFFFFFFFFFFFFFFE0ULL
1226+
1227+/* Register ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1 */
1228+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1 0x0848U
1229+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1_MASKFULL 0x00000000FFFFF001ULL
1230+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1_BASE_ADDR_IN_SHIFT 12U
1231+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1232+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1_MODE_ENABLE_SHIFT 0U
1233+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1234+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG1_MODE_ENABLE_EN 0x0000000000000001ULL
1235+
1236+/* Register ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2 */
1237+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2 0x0850U
1238+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_MASKFULL 0x000000FFFFFFF1FFULL
1239+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_ADDR_OUT_SHIFT 12U
1240+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_ADDR_OUT_CLRMSK 0xFFFFFF0000000FFFULL
1241+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_OS_ID_SHIFT 6U
1242+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_OS_ID_CLRMSK 0xFFFFFFFFFFFFFE3FULL
1243+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_TRUSTED_SHIFT 5U
1244+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_TRUSTED_CLRMSK 0xFFFFFFFFFFFFFFDFULL
1245+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_TRUSTED_EN 0x0000000000000020ULL
1246+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_REGION_SIZE_POW2_SHIFT 0U
1247+#define ROGUE_CR_MIPS_ADDR_REMAP4_CONFIG2_REGION_SIZE_POW2_CLRMSK 0xFFFFFFFFFFFFFFE0ULL
1248+
1249+/* Register ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1 */
1250+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1 0x0858U
1251+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_MASKFULL 0x00000000FFFFF001ULL
1252+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_BASE_ADDR_IN_SHIFT 12U
1253+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1254+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_MODE_ENABLE_SHIFT 0U
1255+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1256+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG1_MODE_ENABLE_EN 0x0000000000000001ULL
1257+
1258+/* Register ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2 */
1259+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2 0x0860U
1260+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_MASKFULL 0x000000FFFFFFF1FFULL
1261+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_ADDR_OUT_SHIFT 12U
1262+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_ADDR_OUT_CLRMSK 0xFFFFFF0000000FFFULL
1263+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_OS_ID_SHIFT 6U
1264+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_OS_ID_CLRMSK 0xFFFFFFFFFFFFFE3FULL
1265+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_TRUSTED_SHIFT 5U
1266+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_TRUSTED_CLRMSK 0xFFFFFFFFFFFFFFDFULL
1267+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_TRUSTED_EN 0x0000000000000020ULL
1268+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_REGION_SIZE_POW2_SHIFT 0U
1269+#define ROGUE_CR_MIPS_ADDR_REMAP5_CONFIG2_REGION_SIZE_POW2_CLRMSK 0xFFFFFFFFFFFFFFE0ULL
1270+
1271+/* Register ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS */
1272+#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS 0x0868U
1273+#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS_MASKFULL 0x00000001FFFFFFFFULL
1274+#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS_EVENT_SHIFT 32U
1275+#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS_EVENT_CLRMSK 0xFFFFFFFEFFFFFFFFULL
1276+#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS_EVENT_EN 0x0000000100000000ULL
1277+#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS_ADDRESS_SHIFT 0U
1278+#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_STATUS_ADDRESS_CLRMSK 0xFFFFFFFF00000000ULL
1279+
1280+/* Register ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_CLEAR */
1281+#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_CLEAR 0x0870U
1282+#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_CLEAR_MASKFULL 0x0000000000000001ULL
1283+#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_CLEAR_EVENT_SHIFT 0U
1284+#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_CLEAR_EVENT_CLRMSK 0xFFFFFFFEU
1285+#define ROGUE_CR_MIPS_ADDR_REMAP_UNMAPPED_CLEAR_EVENT_EN 0x00000001U
1286+
1287+/* Register ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG */
1288+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG 0x0878U
1289+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_MASKFULL 0xFFFFFFF7FFFFFFBFULL
1290+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_ADDR_OUT_SHIFT 36U
1291+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_ADDR_OUT_CLRMSK 0x0000000FFFFFFFFFULL
1292+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_OS_ID_SHIFT 32U
1293+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_OS_ID_CLRMSK 0xFFFFFFF8FFFFFFFFULL
1294+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_BASE_ADDR_IN_SHIFT 12U
1295+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1296+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_TRUSTED_SHIFT 11U
1297+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_TRUSTED_CLRMSK 0xFFFFFFFFFFFFF7FFULL
1298+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_TRUSTED_EN 0x0000000000000800ULL
1299+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_SHIFT 7U
1300+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_CLRMSK 0xFFFFFFFFFFFFF87FULL
1301+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_4KB 0x0000000000000000ULL
1302+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_16KB 0x0000000000000080ULL
1303+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_64KB 0x0000000000000100ULL
1304+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_256KB 0x0000000000000180ULL
1305+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_1MB 0x0000000000000200ULL
1306+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_4MB 0x0000000000000280ULL
1307+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_16MB 0x0000000000000300ULL
1308+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_64MB 0x0000000000000380ULL
1309+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_REGION_SIZE_256MB 0x0000000000000400ULL
1310+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_ENTRY_SHIFT 1U
1311+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_ENTRY_CLRMSK 0xFFFFFFFFFFFFFFC1ULL
1312+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_MODE_ENABLE_SHIFT 0U
1313+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1314+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_CONFIG_MODE_ENABLE_EN 0x0000000000000001ULL
1315+
1316+/* Register ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ */
1317+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ 0x0880U
1318+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ_MASKFULL 0x000000000000003FULL
1319+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ_ENTRY_SHIFT 1U
1320+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ_ENTRY_CLRMSK 0xFFFFFFC1U
1321+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ_REQUEST_SHIFT 0U
1322+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ_REQUEST_CLRMSK 0xFFFFFFFEU
1323+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_READ_REQUEST_EN 0x00000001U
1324+
1325+/* Register ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA */
1326+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA 0x0888U
1327+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_MASKFULL 0xFFFFFFF7FFFFFF81ULL
1328+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_ADDR_OUT_SHIFT 36U
1329+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_ADDR_OUT_CLRMSK 0x0000000FFFFFFFFFULL
1330+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_OS_ID_SHIFT 32U
1331+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_OS_ID_CLRMSK 0xFFFFFFF8FFFFFFFFULL
1332+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_BASE_ADDR_IN_SHIFT 12U
1333+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_BASE_ADDR_IN_CLRMSK 0xFFFFFFFF00000FFFULL
1334+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_TRUSTED_SHIFT 11U
1335+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_TRUSTED_CLRMSK 0xFFFFFFFFFFFFF7FFULL
1336+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_TRUSTED_EN 0x0000000000000800ULL
1337+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_REGION_SIZE_SHIFT 7U
1338+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_REGION_SIZE_CLRMSK 0xFFFFFFFFFFFFF87FULL
1339+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_MODE_ENABLE_SHIFT 0U
1340+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_MODE_ENABLE_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1341+#define ROGUE_CR_MIPS_ADDR_REMAP_RANGE_DATA_MODE_ENABLE_EN 0x0000000000000001ULL
1342+
1343+/* Register ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE */
1344+#define ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE 0x08A0U
1345+#define ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE_MASKFULL 0x0000000000000001ULL
1346+#define ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE_EVENT_SHIFT 0U
1347+#define ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE_EVENT_CLRMSK 0xFFFFFFFEU
1348+#define ROGUE_CR_MIPS_WRAPPER_IRQ_ENABLE_EVENT_EN 0x00000001U
1349+
1350+/* Register ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS */
1351+#define ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS 0x08A8U
1352+#define ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS_MASKFULL 0x0000000000000001ULL
1353+#define ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS_EVENT_SHIFT 0U
1354+#define ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS_EVENT_CLRMSK 0xFFFFFFFEU
1355+#define ROGUE_CR_MIPS_WRAPPER_IRQ_STATUS_EVENT_EN 0x00000001U
1356+
1357+/* Register ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR */
1358+#define ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR 0x08B0U
1359+#define ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR_MASKFULL 0x0000000000000001ULL
1360+#define ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR_EVENT_SHIFT 0U
1361+#define ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR_EVENT_CLRMSK 0xFFFFFFFEU
1362+#define ROGUE_CR_MIPS_WRAPPER_IRQ_CLEAR_EVENT_EN 0x00000001U
1363+
1364+/* Register ROGUE_CR_MIPS_WRAPPER_NMI_ENABLE */
1365+#define ROGUE_CR_MIPS_WRAPPER_NMI_ENABLE 0x08B8U
1366+#define ROGUE_CR_MIPS_WRAPPER_NMI_ENABLE_MASKFULL 0x0000000000000001ULL
1367+#define ROGUE_CR_MIPS_WRAPPER_NMI_ENABLE_EVENT_SHIFT 0U
1368+#define ROGUE_CR_MIPS_WRAPPER_NMI_ENABLE_EVENT_CLRMSK 0xFFFFFFFEU
1369+#define ROGUE_CR_MIPS_WRAPPER_NMI_ENABLE_EVENT_EN 0x00000001U
1370+
1371+/* Register ROGUE_CR_MIPS_WRAPPER_NMI_EVENT */
1372+#define ROGUE_CR_MIPS_WRAPPER_NMI_EVENT 0x08C0U
1373+#define ROGUE_CR_MIPS_WRAPPER_NMI_EVENT_MASKFULL 0x0000000000000001ULL
1374+#define ROGUE_CR_MIPS_WRAPPER_NMI_EVENT_TRIGGER_SHIFT 0U
1375+#define ROGUE_CR_MIPS_WRAPPER_NMI_EVENT_TRIGGER_CLRMSK 0xFFFFFFFEU
1376+#define ROGUE_CR_MIPS_WRAPPER_NMI_EVENT_TRIGGER_EN 0x00000001U
1377+
1378+/* Register ROGUE_CR_MIPS_DEBUG_CONFIG */
1379+#define ROGUE_CR_MIPS_DEBUG_CONFIG 0x08C8U
1380+#define ROGUE_CR_MIPS_DEBUG_CONFIG_MASKFULL 0x0000000000000001ULL
1381+#define ROGUE_CR_MIPS_DEBUG_CONFIG_DISABLE_PROBE_DEBUG_SHIFT 0U
1382+#define ROGUE_CR_MIPS_DEBUG_CONFIG_DISABLE_PROBE_DEBUG_CLRMSK 0xFFFFFFFEU
1383+#define ROGUE_CR_MIPS_DEBUG_CONFIG_DISABLE_PROBE_DEBUG_EN 0x00000001U
1384+
1385+/* Register ROGUE_CR_MIPS_EXCEPTION_STATUS */
1386+#define ROGUE_CR_MIPS_EXCEPTION_STATUS 0x08D0U
1387+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_MASKFULL 0x000000000000003FULL
1388+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_SLEEP_SHIFT 5U
1389+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_SLEEP_CLRMSK 0xFFFFFFDFU
1390+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_SLEEP_EN 0x00000020U
1391+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NMI_TAKEN_SHIFT 4U
1392+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NMI_TAKEN_CLRMSK 0xFFFFFFEFU
1393+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NMI_TAKEN_EN 0x00000010U
1394+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NEST_EXL_SHIFT 3U
1395+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NEST_EXL_CLRMSK 0xFFFFFFF7U
1396+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NEST_EXL_EN 0x00000008U
1397+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NEST_ERL_SHIFT 2U
1398+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NEST_ERL_CLRMSK 0xFFFFFFFBU
1399+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_NEST_ERL_EN 0x00000004U
1400+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_EXL_SHIFT 1U
1401+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_EXL_CLRMSK 0xFFFFFFFDU
1402+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_EXL_EN 0x00000002U
1403+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_ERL_SHIFT 0U
1404+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_ERL_CLRMSK 0xFFFFFFFEU
1405+#define ROGUE_CR_MIPS_EXCEPTION_STATUS_SI_ERL_EN 0x00000001U
1406+
1407+/* Register ROGUE_CR_MIPS_WRAPPER_STATUS */
1408+#define ROGUE_CR_MIPS_WRAPPER_STATUS 0x08E8U
1409+#define ROGUE_CR_MIPS_WRAPPER_STATUS_MASKFULL 0x00000000000000FFULL
1410+#define ROGUE_CR_MIPS_WRAPPER_STATUS_OUTSTANDING_REQUESTS_SHIFT 0U
1411+#define ROGUE_CR_MIPS_WRAPPER_STATUS_OUTSTANDING_REQUESTS_CLRMSK 0xFFFFFF00U
1412+
1413+/* Register ROGUE_CR_XPU_BROADCAST */
1414+#define ROGUE_CR_XPU_BROADCAST 0x0890U
1415+#define ROGUE_CR_XPU_BROADCAST_MASKFULL 0x00000000000001FFULL
1416+#define ROGUE_CR_XPU_BROADCAST_MASK_SHIFT 0U
1417+#define ROGUE_CR_XPU_BROADCAST_MASK_CLRMSK 0xFFFFFE00U
1418+
1419+/* Register ROGUE_CR_META_SP_MSLVDATAX */
1420+#define ROGUE_CR_META_SP_MSLVDATAX 0x0A00U
1421+#define ROGUE_CR_META_SP_MSLVDATAX_MASKFULL 0x00000000FFFFFFFFULL
1422+#define ROGUE_CR_META_SP_MSLVDATAX_MSLVDATAX_SHIFT 0U
1423+#define ROGUE_CR_META_SP_MSLVDATAX_MSLVDATAX_CLRMSK 0x00000000U
1424+
1425+/* Register ROGUE_CR_META_SP_MSLVDATAT */
1426+#define ROGUE_CR_META_SP_MSLVDATAT 0x0A08U
1427+#define ROGUE_CR_META_SP_MSLVDATAT_MASKFULL 0x00000000FFFFFFFFULL
1428+#define ROGUE_CR_META_SP_MSLVDATAT_MSLVDATAT_SHIFT 0U
1429+#define ROGUE_CR_META_SP_MSLVDATAT_MSLVDATAT_CLRMSK 0x00000000U
1430+
1431+/* Register ROGUE_CR_META_SP_MSLVCTRL0 */
1432+#define ROGUE_CR_META_SP_MSLVCTRL0 0x0A10U
1433+#define ROGUE_CR_META_SP_MSLVCTRL0_MASKFULL 0x00000000FFFFFFFFULL
1434+#define ROGUE_CR_META_SP_MSLVCTRL0_ADDR_SHIFT 2U
1435+#define ROGUE_CR_META_SP_MSLVCTRL0_ADDR_CLRMSK 0x00000003U
1436+#define ROGUE_CR_META_SP_MSLVCTRL0_AUTOINCR_SHIFT 1U
1437+#define ROGUE_CR_META_SP_MSLVCTRL0_AUTOINCR_CLRMSK 0xFFFFFFFDU
1438+#define ROGUE_CR_META_SP_MSLVCTRL0_AUTOINCR_EN 0x00000002U
1439+#define ROGUE_CR_META_SP_MSLVCTRL0_RD_SHIFT 0U
1440+#define ROGUE_CR_META_SP_MSLVCTRL0_RD_CLRMSK 0xFFFFFFFEU
1441+#define ROGUE_CR_META_SP_MSLVCTRL0_RD_EN 0x00000001U
1442+
1443+/* Register ROGUE_CR_META_SP_MSLVCTRL1 */
1444+#define ROGUE_CR_META_SP_MSLVCTRL1 0x0A18U
1445+#define ROGUE_CR_META_SP_MSLVCTRL1_MASKFULL 0x00000000F7F4003FULL
1446+#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERRTHREAD_SHIFT 30U
1447+#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERRTHREAD_CLRMSK 0x3FFFFFFFU
1448+#define ROGUE_CR_META_SP_MSLVCTRL1_LOCK2_INTERLOCK_SHIFT 29U
1449+#define ROGUE_CR_META_SP_MSLVCTRL1_LOCK2_INTERLOCK_CLRMSK 0xDFFFFFFFU
1450+#define ROGUE_CR_META_SP_MSLVCTRL1_LOCK2_INTERLOCK_EN 0x20000000U
1451+#define ROGUE_CR_META_SP_MSLVCTRL1_ATOMIC_INTERLOCK_SHIFT 28U
1452+#define ROGUE_CR_META_SP_MSLVCTRL1_ATOMIC_INTERLOCK_CLRMSK 0xEFFFFFFFU
1453+#define ROGUE_CR_META_SP_MSLVCTRL1_ATOMIC_INTERLOCK_EN 0x10000000U
1454+#define ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_SHIFT 26U
1455+#define ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_CLRMSK 0xFBFFFFFFU
1456+#define ROGUE_CR_META_SP_MSLVCTRL1_GBLPORT_IDLE_EN 0x04000000U
1457+#define ROGUE_CR_META_SP_MSLVCTRL1_COREMEM_IDLE_SHIFT 25U
1458+#define ROGUE_CR_META_SP_MSLVCTRL1_COREMEM_IDLE_CLRMSK 0xFDFFFFFFU
1459+#define ROGUE_CR_META_SP_MSLVCTRL1_COREMEM_IDLE_EN 0x02000000U
1460+#define ROGUE_CR_META_SP_MSLVCTRL1_READY_SHIFT 24U
1461+#define ROGUE_CR_META_SP_MSLVCTRL1_READY_CLRMSK 0xFEFFFFFFU
1462+#define ROGUE_CR_META_SP_MSLVCTRL1_READY_EN 0x01000000U
1463+#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERRID_SHIFT 21U
1464+#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERRID_CLRMSK 0xFF1FFFFFU
1465+#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERR_SHIFT 20U
1466+#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERR_CLRMSK 0xFFEFFFFFU
1467+#define ROGUE_CR_META_SP_MSLVCTRL1_DEFERR_EN 0x00100000U
1468+#define ROGUE_CR_META_SP_MSLVCTRL1_WR_ACTIVE_SHIFT 18U
1469+#define ROGUE_CR_META_SP_MSLVCTRL1_WR_ACTIVE_CLRMSK 0xFFFBFFFFU
1470+#define ROGUE_CR_META_SP_MSLVCTRL1_WR_ACTIVE_EN 0x00040000U
1471+#define ROGUE_CR_META_SP_MSLVCTRL1_THREAD_SHIFT 4U
1472+#define ROGUE_CR_META_SP_MSLVCTRL1_THREAD_CLRMSK 0xFFFFFFCFU
1473+#define ROGUE_CR_META_SP_MSLVCTRL1_TRANS_SIZE_SHIFT 2U
1474+#define ROGUE_CR_META_SP_MSLVCTRL1_TRANS_SIZE_CLRMSK 0xFFFFFFF3U
1475+#define ROGUE_CR_META_SP_MSLVCTRL1_BYTE_ROUND_SHIFT 0U
1476+#define ROGUE_CR_META_SP_MSLVCTRL1_BYTE_ROUND_CLRMSK 0xFFFFFFFCU
1477+
1478+/* Register ROGUE_CR_META_SP_MSLVHANDSHKE */
1479+#define ROGUE_CR_META_SP_MSLVHANDSHKE 0x0A50U
1480+#define ROGUE_CR_META_SP_MSLVHANDSHKE_MASKFULL 0x000000000000000FULL
1481+#define ROGUE_CR_META_SP_MSLVHANDSHKE_INPUT_SHIFT 2U
1482+#define ROGUE_CR_META_SP_MSLVHANDSHKE_INPUT_CLRMSK 0xFFFFFFF3U
1483+#define ROGUE_CR_META_SP_MSLVHANDSHKE_OUTPUT_SHIFT 0U
1484+#define ROGUE_CR_META_SP_MSLVHANDSHKE_OUTPUT_CLRMSK 0xFFFFFFFCU
1485+
1486+/* Register ROGUE_CR_META_SP_MSLVT0KICK */
1487+#define ROGUE_CR_META_SP_MSLVT0KICK 0x0A80U
1488+#define ROGUE_CR_META_SP_MSLVT0KICK_MASKFULL 0x000000000000FFFFULL
1489+#define ROGUE_CR_META_SP_MSLVT0KICK_MSLVT0KICK_SHIFT 0U
1490+#define ROGUE_CR_META_SP_MSLVT0KICK_MSLVT0KICK_CLRMSK 0xFFFF0000U
1491+
1492+/* Register ROGUE_CR_META_SP_MSLVT0KICKI */
1493+#define ROGUE_CR_META_SP_MSLVT0KICKI 0x0A88U
1494+#define ROGUE_CR_META_SP_MSLVT0KICKI_MASKFULL 0x000000000000FFFFULL
1495+#define ROGUE_CR_META_SP_MSLVT0KICKI_MSLVT0KICKI_SHIFT 0U
1496+#define ROGUE_CR_META_SP_MSLVT0KICKI_MSLVT0KICKI_CLRMSK 0xFFFF0000U
1497+
1498+/* Register ROGUE_CR_META_SP_MSLVT1KICK */
1499+#define ROGUE_CR_META_SP_MSLVT1KICK 0x0A90U
1500+#define ROGUE_CR_META_SP_MSLVT1KICK_MASKFULL 0x000000000000FFFFULL
1501+#define ROGUE_CR_META_SP_MSLVT1KICK_MSLVT1KICK_SHIFT 0U
1502+#define ROGUE_CR_META_SP_MSLVT1KICK_MSLVT1KICK_CLRMSK 0xFFFF0000U
1503+
1504+/* Register ROGUE_CR_META_SP_MSLVT1KICKI */
1505+#define ROGUE_CR_META_SP_MSLVT1KICKI 0x0A98U
1506+#define ROGUE_CR_META_SP_MSLVT1KICKI_MASKFULL 0x000000000000FFFFULL
1507+#define ROGUE_CR_META_SP_MSLVT1KICKI_MSLVT1KICKI_SHIFT 0U
1508+#define ROGUE_CR_META_SP_MSLVT1KICKI_MSLVT1KICKI_CLRMSK 0xFFFF0000U
1509+
1510+/* Register ROGUE_CR_META_SP_MSLVT2KICK */
1511+#define ROGUE_CR_META_SP_MSLVT2KICK 0x0AA0U
1512+#define ROGUE_CR_META_SP_MSLVT2KICK_MASKFULL 0x000000000000FFFFULL
1513+#define ROGUE_CR_META_SP_MSLVT2KICK_MSLVT2KICK_SHIFT 0U
1514+#define ROGUE_CR_META_SP_MSLVT2KICK_MSLVT2KICK_CLRMSK 0xFFFF0000U
1515+
1516+/* Register ROGUE_CR_META_SP_MSLVT2KICKI */
1517+#define ROGUE_CR_META_SP_MSLVT2KICKI 0x0AA8U
1518+#define ROGUE_CR_META_SP_MSLVT2KICKI_MASKFULL 0x000000000000FFFFULL
1519+#define ROGUE_CR_META_SP_MSLVT2KICKI_MSLVT2KICKI_SHIFT 0U
1520+#define ROGUE_CR_META_SP_MSLVT2KICKI_MSLVT2KICKI_CLRMSK 0xFFFF0000U
1521+
1522+/* Register ROGUE_CR_META_SP_MSLVT3KICK */
1523+#define ROGUE_CR_META_SP_MSLVT3KICK 0x0AB0U
1524+#define ROGUE_CR_META_SP_MSLVT3KICK_MASKFULL 0x000000000000FFFFULL
1525+#define ROGUE_CR_META_SP_MSLVT3KICK_MSLVT3KICK_SHIFT 0U
1526+#define ROGUE_CR_META_SP_MSLVT3KICK_MSLVT3KICK_CLRMSK 0xFFFF0000U
1527+
1528+/* Register ROGUE_CR_META_SP_MSLVT3KICKI */
1529+#define ROGUE_CR_META_SP_MSLVT3KICKI 0x0AB8U
1530+#define ROGUE_CR_META_SP_MSLVT3KICKI_MASKFULL 0x000000000000FFFFULL
1531+#define ROGUE_CR_META_SP_MSLVT3KICKI_MSLVT3KICKI_SHIFT 0U
1532+#define ROGUE_CR_META_SP_MSLVT3KICKI_MSLVT3KICKI_CLRMSK 0xFFFF0000U
1533+
1534+/* Register ROGUE_CR_META_SP_MSLVRST */
1535+#define ROGUE_CR_META_SP_MSLVRST 0x0AC0U
1536+#define ROGUE_CR_META_SP_MSLVRST_MASKFULL 0x0000000000000001ULL
1537+#define ROGUE_CR_META_SP_MSLVRST_SOFTRESET_SHIFT 0U
1538+#define ROGUE_CR_META_SP_MSLVRST_SOFTRESET_CLRMSK 0xFFFFFFFEU
1539+#define ROGUE_CR_META_SP_MSLVRST_SOFTRESET_EN 0x00000001U
1540+
1541+/* Register ROGUE_CR_META_SP_MSLVIRQSTATUS */
1542+#define ROGUE_CR_META_SP_MSLVIRQSTATUS 0x0AC8U
1543+#define ROGUE_CR_META_SP_MSLVIRQSTATUS_MASKFULL 0x000000000000000CULL
1544+#define ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT3_SHIFT 3U
1545+#define ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT3_CLRMSK 0xFFFFFFF7U
1546+#define ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT3_EN 0x00000008U
1547+#define ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT2_SHIFT 2U
1548+#define ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT2_CLRMSK 0xFFFFFFFBU
1549+#define ROGUE_CR_META_SP_MSLVIRQSTATUS_TRIGVECT2_EN 0x00000004U
1550+
1551+/* Register ROGUE_CR_META_SP_MSLVIRQENABLE */
1552+#define ROGUE_CR_META_SP_MSLVIRQENABLE 0x0AD0U
1553+#define ROGUE_CR_META_SP_MSLVIRQENABLE_MASKFULL 0x000000000000000CULL
1554+#define ROGUE_CR_META_SP_MSLVIRQENABLE_EVENT1_SHIFT 3U
1555+#define ROGUE_CR_META_SP_MSLVIRQENABLE_EVENT1_CLRMSK 0xFFFFFFF7U
1556+#define ROGUE_CR_META_SP_MSLVIRQENABLE_EVENT1_EN 0x00000008U
1557+#define ROGUE_CR_META_SP_MSLVIRQENABLE_EVENT0_SHIFT 2U
1558+#define ROGUE_CR_META_SP_MSLVIRQENABLE_EVENT0_CLRMSK 0xFFFFFFFBU
1559+#define ROGUE_CR_META_SP_MSLVIRQENABLE_EVENT0_EN 0x00000004U
1560+
1561+/* Register ROGUE_CR_META_SP_MSLVIRQLEVEL */
1562+#define ROGUE_CR_META_SP_MSLVIRQLEVEL 0x0AD8U
1563+#define ROGUE_CR_META_SP_MSLVIRQLEVEL_MASKFULL 0x0000000000000001ULL
1564+#define ROGUE_CR_META_SP_MSLVIRQLEVEL_MODE_SHIFT 0U
1565+#define ROGUE_CR_META_SP_MSLVIRQLEVEL_MODE_CLRMSK 0xFFFFFFFEU
1566+#define ROGUE_CR_META_SP_MSLVIRQLEVEL_MODE_EN 0x00000001U
1567+
1568+/* Register ROGUE_CR_MTS_SCHEDULE */
1569+#define ROGUE_CR_MTS_SCHEDULE 0x0B00U
1570+#define ROGUE_CR_MTS_SCHEDULE_MASKFULL 0x00000000000001FFULL
1571+#define ROGUE_CR_MTS_SCHEDULE_HOST_SHIFT 8U
1572+#define ROGUE_CR_MTS_SCHEDULE_HOST_CLRMSK 0xFFFFFEFFU
1573+#define ROGUE_CR_MTS_SCHEDULE_HOST_BG_TIMER 0x00000000U
1574+#define ROGUE_CR_MTS_SCHEDULE_HOST_HOST 0x00000100U
1575+#define ROGUE_CR_MTS_SCHEDULE_PRIORITY_SHIFT 6U
1576+#define ROGUE_CR_MTS_SCHEDULE_PRIORITY_CLRMSK 0xFFFFFF3FU
1577+#define ROGUE_CR_MTS_SCHEDULE_PRIORITY_PRT0 0x00000000U
1578+#define ROGUE_CR_MTS_SCHEDULE_PRIORITY_PRT1 0x00000040U
1579+#define ROGUE_CR_MTS_SCHEDULE_PRIORITY_PRT2 0x00000080U
1580+#define ROGUE_CR_MTS_SCHEDULE_PRIORITY_PRT3 0x000000C0U
1581+#define ROGUE_CR_MTS_SCHEDULE_CONTEXT_SHIFT 5U
1582+#define ROGUE_CR_MTS_SCHEDULE_CONTEXT_CLRMSK 0xFFFFFFDFU
1583+#define ROGUE_CR_MTS_SCHEDULE_CONTEXT_BGCTX 0x00000000U
1584+#define ROGUE_CR_MTS_SCHEDULE_CONTEXT_INTCTX 0x00000020U
1585+#define ROGUE_CR_MTS_SCHEDULE_TASK_SHIFT 4U
1586+#define ROGUE_CR_MTS_SCHEDULE_TASK_CLRMSK 0xFFFFFFEFU
1587+#define ROGUE_CR_MTS_SCHEDULE_TASK_NON_COUNTED 0x00000000U
1588+#define ROGUE_CR_MTS_SCHEDULE_TASK_COUNTED 0x00000010U
1589+#define ROGUE_CR_MTS_SCHEDULE_DM_SHIFT 0U
1590+#define ROGUE_CR_MTS_SCHEDULE_DM_CLRMSK 0xFFFFFFF0U
1591+#define ROGUE_CR_MTS_SCHEDULE_DM_DM0 0x00000000U
1592+#define ROGUE_CR_MTS_SCHEDULE_DM_DM1 0x00000001U
1593+#define ROGUE_CR_MTS_SCHEDULE_DM_DM2 0x00000002U
1594+#define ROGUE_CR_MTS_SCHEDULE_DM_DM3 0x00000003U
1595+#define ROGUE_CR_MTS_SCHEDULE_DM_DM4 0x00000004U
1596+#define ROGUE_CR_MTS_SCHEDULE_DM_DM5 0x00000005U
1597+#define ROGUE_CR_MTS_SCHEDULE_DM_DM6 0x00000006U
1598+#define ROGUE_CR_MTS_SCHEDULE_DM_DM7 0x00000007U
1599+#define ROGUE_CR_MTS_SCHEDULE_DM_DM_ALL 0x0000000FU
1600+
1601+/* Register ROGUE_CR_MTS_SCHEDULE1 */
1602+#define ROGUE_CR_MTS_SCHEDULE1 0x10B00U
1603+#define ROGUE_CR_MTS_SCHEDULE1_MASKFULL 0x00000000000001FFULL
1604+#define ROGUE_CR_MTS_SCHEDULE1_HOST_SHIFT 8U
1605+#define ROGUE_CR_MTS_SCHEDULE1_HOST_CLRMSK 0xFFFFFEFFU
1606+#define ROGUE_CR_MTS_SCHEDULE1_HOST_BG_TIMER 0x00000000U
1607+#define ROGUE_CR_MTS_SCHEDULE1_HOST_HOST 0x00000100U
1608+#define ROGUE_CR_MTS_SCHEDULE1_PRIORITY_SHIFT 6U
1609+#define ROGUE_CR_MTS_SCHEDULE1_PRIORITY_CLRMSK 0xFFFFFF3FU
1610+#define ROGUE_CR_MTS_SCHEDULE1_PRIORITY_PRT0 0x00000000U
1611+#define ROGUE_CR_MTS_SCHEDULE1_PRIORITY_PRT1 0x00000040U
1612+#define ROGUE_CR_MTS_SCHEDULE1_PRIORITY_PRT2 0x00000080U
1613+#define ROGUE_CR_MTS_SCHEDULE1_PRIORITY_PRT3 0x000000C0U
1614+#define ROGUE_CR_MTS_SCHEDULE1_CONTEXT_SHIFT 5U
1615+#define ROGUE_CR_MTS_SCHEDULE1_CONTEXT_CLRMSK 0xFFFFFFDFU
1616+#define ROGUE_CR_MTS_SCHEDULE1_CONTEXT_BGCTX 0x00000000U
1617+#define ROGUE_CR_MTS_SCHEDULE1_CONTEXT_INTCTX 0x00000020U
1618+#define ROGUE_CR_MTS_SCHEDULE1_TASK_SHIFT 4U
1619+#define ROGUE_CR_MTS_SCHEDULE1_TASK_CLRMSK 0xFFFFFFEFU
1620+#define ROGUE_CR_MTS_SCHEDULE1_TASK_NON_COUNTED 0x00000000U
1621+#define ROGUE_CR_MTS_SCHEDULE1_TASK_COUNTED 0x00000010U
1622+#define ROGUE_CR_MTS_SCHEDULE1_DM_SHIFT 0U
1623+#define ROGUE_CR_MTS_SCHEDULE1_DM_CLRMSK 0xFFFFFFF0U
1624+#define ROGUE_CR_MTS_SCHEDULE1_DM_DM0 0x00000000U
1625+#define ROGUE_CR_MTS_SCHEDULE1_DM_DM1 0x00000001U
1626+#define ROGUE_CR_MTS_SCHEDULE1_DM_DM2 0x00000002U
1627+#define ROGUE_CR_MTS_SCHEDULE1_DM_DM3 0x00000003U
1628+#define ROGUE_CR_MTS_SCHEDULE1_DM_DM4 0x00000004U
1629+#define ROGUE_CR_MTS_SCHEDULE1_DM_DM5 0x00000005U
1630+#define ROGUE_CR_MTS_SCHEDULE1_DM_DM6 0x00000006U
1631+#define ROGUE_CR_MTS_SCHEDULE1_DM_DM7 0x00000007U
1632+#define ROGUE_CR_MTS_SCHEDULE1_DM_DM_ALL 0x0000000FU
1633+
1634+/* Register ROGUE_CR_MTS_SCHEDULE2 */
1635+#define ROGUE_CR_MTS_SCHEDULE2 0x20B00U
1636+#define ROGUE_CR_MTS_SCHEDULE2_MASKFULL 0x00000000000001FFULL
1637+#define ROGUE_CR_MTS_SCHEDULE2_HOST_SHIFT 8U
1638+#define ROGUE_CR_MTS_SCHEDULE2_HOST_CLRMSK 0xFFFFFEFFU
1639+#define ROGUE_CR_MTS_SCHEDULE2_HOST_BG_TIMER 0x00000000U
1640+#define ROGUE_CR_MTS_SCHEDULE2_HOST_HOST 0x00000100U
1641+#define ROGUE_CR_MTS_SCHEDULE2_PRIORITY_SHIFT 6U
1642+#define ROGUE_CR_MTS_SCHEDULE2_PRIORITY_CLRMSK 0xFFFFFF3FU
1643+#define ROGUE_CR_MTS_SCHEDULE2_PRIORITY_PRT0 0x00000000U
1644+#define ROGUE_CR_MTS_SCHEDULE2_PRIORITY_PRT1 0x00000040U
1645+#define ROGUE_CR_MTS_SCHEDULE2_PRIORITY_PRT2 0x00000080U
1646+#define ROGUE_CR_MTS_SCHEDULE2_PRIORITY_PRT3 0x000000C0U
1647+#define ROGUE_CR_MTS_SCHEDULE2_CONTEXT_SHIFT 5U
1648+#define ROGUE_CR_MTS_SCHEDULE2_CONTEXT_CLRMSK 0xFFFFFFDFU
1649+#define ROGUE_CR_MTS_SCHEDULE2_CONTEXT_BGCTX 0x00000000U
1650+#define ROGUE_CR_MTS_SCHEDULE2_CONTEXT_INTCTX 0x00000020U
1651+#define ROGUE_CR_MTS_SCHEDULE2_TASK_SHIFT 4U
1652+#define ROGUE_CR_MTS_SCHEDULE2_TASK_CLRMSK 0xFFFFFFEFU
1653+#define ROGUE_CR_MTS_SCHEDULE2_TASK_NON_COUNTED 0x00000000U
1654+#define ROGUE_CR_MTS_SCHEDULE2_TASK_COUNTED 0x00000010U
1655+#define ROGUE_CR_MTS_SCHEDULE2_DM_SHIFT 0U
1656+#define ROGUE_CR_MTS_SCHEDULE2_DM_CLRMSK 0xFFFFFFF0U
1657+#define ROGUE_CR_MTS_SCHEDULE2_DM_DM0 0x00000000U
1658+#define ROGUE_CR_MTS_SCHEDULE2_DM_DM1 0x00000001U
1659+#define ROGUE_CR_MTS_SCHEDULE2_DM_DM2 0x00000002U
1660+#define ROGUE_CR_MTS_SCHEDULE2_DM_DM3 0x00000003U
1661+#define ROGUE_CR_MTS_SCHEDULE2_DM_DM4 0x00000004U
1662+#define ROGUE_CR_MTS_SCHEDULE2_DM_DM5 0x00000005U
1663+#define ROGUE_CR_MTS_SCHEDULE2_DM_DM6 0x00000006U
1664+#define ROGUE_CR_MTS_SCHEDULE2_DM_DM7 0x00000007U
1665+#define ROGUE_CR_MTS_SCHEDULE2_DM_DM_ALL 0x0000000FU
1666+
1667+/* Register ROGUE_CR_MTS_SCHEDULE3 */
1668+#define ROGUE_CR_MTS_SCHEDULE3 0x30B00U
1669+#define ROGUE_CR_MTS_SCHEDULE3_MASKFULL 0x00000000000001FFULL
1670+#define ROGUE_CR_MTS_SCHEDULE3_HOST_SHIFT 8U
1671+#define ROGUE_CR_MTS_SCHEDULE3_HOST_CLRMSK 0xFFFFFEFFU
1672+#define ROGUE_CR_MTS_SCHEDULE3_HOST_BG_TIMER 0x00000000U
1673+#define ROGUE_CR_MTS_SCHEDULE3_HOST_HOST 0x00000100U
1674+#define ROGUE_CR_MTS_SCHEDULE3_PRIORITY_SHIFT 6U
1675+#define ROGUE_CR_MTS_SCHEDULE3_PRIORITY_CLRMSK 0xFFFFFF3FU
1676+#define ROGUE_CR_MTS_SCHEDULE3_PRIORITY_PRT0 0x00000000U
1677+#define ROGUE_CR_MTS_SCHEDULE3_PRIORITY_PRT1 0x00000040U
1678+#define ROGUE_CR_MTS_SCHEDULE3_PRIORITY_PRT2 0x00000080U
1679+#define ROGUE_CR_MTS_SCHEDULE3_PRIORITY_PRT3 0x000000C0U
1680+#define ROGUE_CR_MTS_SCHEDULE3_CONTEXT_SHIFT 5U
1681+#define ROGUE_CR_MTS_SCHEDULE3_CONTEXT_CLRMSK 0xFFFFFFDFU
1682+#define ROGUE_CR_MTS_SCHEDULE3_CONTEXT_BGCTX 0x00000000U
1683+#define ROGUE_CR_MTS_SCHEDULE3_CONTEXT_INTCTX 0x00000020U
1684+#define ROGUE_CR_MTS_SCHEDULE3_TASK_SHIFT 4U
1685+#define ROGUE_CR_MTS_SCHEDULE3_TASK_CLRMSK 0xFFFFFFEFU
1686+#define ROGUE_CR_MTS_SCHEDULE3_TASK_NON_COUNTED 0x00000000U
1687+#define ROGUE_CR_MTS_SCHEDULE3_TASK_COUNTED 0x00000010U
1688+#define ROGUE_CR_MTS_SCHEDULE3_DM_SHIFT 0U
1689+#define ROGUE_CR_MTS_SCHEDULE3_DM_CLRMSK 0xFFFFFFF0U
1690+#define ROGUE_CR_MTS_SCHEDULE3_DM_DM0 0x00000000U
1691+#define ROGUE_CR_MTS_SCHEDULE3_DM_DM1 0x00000001U
1692+#define ROGUE_CR_MTS_SCHEDULE3_DM_DM2 0x00000002U
1693+#define ROGUE_CR_MTS_SCHEDULE3_DM_DM3 0x00000003U
1694+#define ROGUE_CR_MTS_SCHEDULE3_DM_DM4 0x00000004U
1695+#define ROGUE_CR_MTS_SCHEDULE3_DM_DM5 0x00000005U
1696+#define ROGUE_CR_MTS_SCHEDULE3_DM_DM6 0x00000006U
1697+#define ROGUE_CR_MTS_SCHEDULE3_DM_DM7 0x00000007U
1698+#define ROGUE_CR_MTS_SCHEDULE3_DM_DM_ALL 0x0000000FU
1699+
1700+/* Register ROGUE_CR_MTS_SCHEDULE4 */
1701+#define ROGUE_CR_MTS_SCHEDULE4 0x40B00U
1702+#define ROGUE_CR_MTS_SCHEDULE4_MASKFULL 0x00000000000001FFULL
1703+#define ROGUE_CR_MTS_SCHEDULE4_HOST_SHIFT 8U
1704+#define ROGUE_CR_MTS_SCHEDULE4_HOST_CLRMSK 0xFFFFFEFFU
1705+#define ROGUE_CR_MTS_SCHEDULE4_HOST_BG_TIMER 0x00000000U
1706+#define ROGUE_CR_MTS_SCHEDULE4_HOST_HOST 0x00000100U
1707+#define ROGUE_CR_MTS_SCHEDULE4_PRIORITY_SHIFT 6U
1708+#define ROGUE_CR_MTS_SCHEDULE4_PRIORITY_CLRMSK 0xFFFFFF3FU
1709+#define ROGUE_CR_MTS_SCHEDULE4_PRIORITY_PRT0 0x00000000U
1710+#define ROGUE_CR_MTS_SCHEDULE4_PRIORITY_PRT1 0x00000040U
1711+#define ROGUE_CR_MTS_SCHEDULE4_PRIORITY_PRT2 0x00000080U
1712+#define ROGUE_CR_MTS_SCHEDULE4_PRIORITY_PRT3 0x000000C0U
1713+#define ROGUE_CR_MTS_SCHEDULE4_CONTEXT_SHIFT 5U
1714+#define ROGUE_CR_MTS_SCHEDULE4_CONTEXT_CLRMSK 0xFFFFFFDFU
1715+#define ROGUE_CR_MTS_SCHEDULE4_CONTEXT_BGCTX 0x00000000U
1716+#define ROGUE_CR_MTS_SCHEDULE4_CONTEXT_INTCTX 0x00000020U
1717+#define ROGUE_CR_MTS_SCHEDULE4_TASK_SHIFT 4U
1718+#define ROGUE_CR_MTS_SCHEDULE4_TASK_CLRMSK 0xFFFFFFEFU
1719+#define ROGUE_CR_MTS_SCHEDULE4_TASK_NON_COUNTED 0x00000000U
1720+#define ROGUE_CR_MTS_SCHEDULE4_TASK_COUNTED 0x00000010U
1721+#define ROGUE_CR_MTS_SCHEDULE4_DM_SHIFT 0U
1722+#define ROGUE_CR_MTS_SCHEDULE4_DM_CLRMSK 0xFFFFFFF0U
1723+#define ROGUE_CR_MTS_SCHEDULE4_DM_DM0 0x00000000U
1724+#define ROGUE_CR_MTS_SCHEDULE4_DM_DM1 0x00000001U
1725+#define ROGUE_CR_MTS_SCHEDULE4_DM_DM2 0x00000002U
1726+#define ROGUE_CR_MTS_SCHEDULE4_DM_DM3 0x00000003U
1727+#define ROGUE_CR_MTS_SCHEDULE4_DM_DM4 0x00000004U
1728+#define ROGUE_CR_MTS_SCHEDULE4_DM_DM5 0x00000005U
1729+#define ROGUE_CR_MTS_SCHEDULE4_DM_DM6 0x00000006U
1730+#define ROGUE_CR_MTS_SCHEDULE4_DM_DM7 0x00000007U
1731+#define ROGUE_CR_MTS_SCHEDULE4_DM_DM_ALL 0x0000000FU
1732+
1733+/* Register ROGUE_CR_MTS_SCHEDULE5 */
1734+#define ROGUE_CR_MTS_SCHEDULE5 0x50B00U
1735+#define ROGUE_CR_MTS_SCHEDULE5_MASKFULL 0x00000000000001FFULL
1736+#define ROGUE_CR_MTS_SCHEDULE5_HOST_SHIFT 8U
1737+#define ROGUE_CR_MTS_SCHEDULE5_HOST_CLRMSK 0xFFFFFEFFU
1738+#define ROGUE_CR_MTS_SCHEDULE5_HOST_BG_TIMER 0x00000000U
1739+#define ROGUE_CR_MTS_SCHEDULE5_HOST_HOST 0x00000100U
1740+#define ROGUE_CR_MTS_SCHEDULE5_PRIORITY_SHIFT 6U
1741+#define ROGUE_CR_MTS_SCHEDULE5_PRIORITY_CLRMSK 0xFFFFFF3FU
1742+#define ROGUE_CR_MTS_SCHEDULE5_PRIORITY_PRT0 0x00000000U
1743+#define ROGUE_CR_MTS_SCHEDULE5_PRIORITY_PRT1 0x00000040U
1744+#define ROGUE_CR_MTS_SCHEDULE5_PRIORITY_PRT2 0x00000080U
1745+#define ROGUE_CR_MTS_SCHEDULE5_PRIORITY_PRT3 0x000000C0U
1746+#define ROGUE_CR_MTS_SCHEDULE5_CONTEXT_SHIFT 5U
1747+#define ROGUE_CR_MTS_SCHEDULE5_CONTEXT_CLRMSK 0xFFFFFFDFU
1748+#define ROGUE_CR_MTS_SCHEDULE5_CONTEXT_BGCTX 0x00000000U
1749+#define ROGUE_CR_MTS_SCHEDULE5_CONTEXT_INTCTX 0x00000020U
1750+#define ROGUE_CR_MTS_SCHEDULE5_TASK_SHIFT 4U
1751+#define ROGUE_CR_MTS_SCHEDULE5_TASK_CLRMSK 0xFFFFFFEFU
1752+#define ROGUE_CR_MTS_SCHEDULE5_TASK_NON_COUNTED 0x00000000U
1753+#define ROGUE_CR_MTS_SCHEDULE5_TASK_COUNTED 0x00000010U
1754+#define ROGUE_CR_MTS_SCHEDULE5_DM_SHIFT 0U
1755+#define ROGUE_CR_MTS_SCHEDULE5_DM_CLRMSK 0xFFFFFFF0U
1756+#define ROGUE_CR_MTS_SCHEDULE5_DM_DM0 0x00000000U
1757+#define ROGUE_CR_MTS_SCHEDULE5_DM_DM1 0x00000001U
1758+#define ROGUE_CR_MTS_SCHEDULE5_DM_DM2 0x00000002U
1759+#define ROGUE_CR_MTS_SCHEDULE5_DM_DM3 0x00000003U
1760+#define ROGUE_CR_MTS_SCHEDULE5_DM_DM4 0x00000004U
1761+#define ROGUE_CR_MTS_SCHEDULE5_DM_DM5 0x00000005U
1762+#define ROGUE_CR_MTS_SCHEDULE5_DM_DM6 0x00000006U
1763+#define ROGUE_CR_MTS_SCHEDULE5_DM_DM7 0x00000007U
1764+#define ROGUE_CR_MTS_SCHEDULE5_DM_DM_ALL 0x0000000FU
1765+
1766+/* Register ROGUE_CR_MTS_SCHEDULE6 */
1767+#define ROGUE_CR_MTS_SCHEDULE6 0x60B00U
1768+#define ROGUE_CR_MTS_SCHEDULE6_MASKFULL 0x00000000000001FFULL
1769+#define ROGUE_CR_MTS_SCHEDULE6_HOST_SHIFT 8U
1770+#define ROGUE_CR_MTS_SCHEDULE6_HOST_CLRMSK 0xFFFFFEFFU
1771+#define ROGUE_CR_MTS_SCHEDULE6_HOST_BG_TIMER 0x00000000U
1772+#define ROGUE_CR_MTS_SCHEDULE6_HOST_HOST 0x00000100U
1773+#define ROGUE_CR_MTS_SCHEDULE6_PRIORITY_SHIFT 6U
1774+#define ROGUE_CR_MTS_SCHEDULE6_PRIORITY_CLRMSK 0xFFFFFF3FU
1775+#define ROGUE_CR_MTS_SCHEDULE6_PRIORITY_PRT0 0x00000000U
1776+#define ROGUE_CR_MTS_SCHEDULE6_PRIORITY_PRT1 0x00000040U
1777+#define ROGUE_CR_MTS_SCHEDULE6_PRIORITY_PRT2 0x00000080U
1778+#define ROGUE_CR_MTS_SCHEDULE6_PRIORITY_PRT3 0x000000C0U
1779+#define ROGUE_CR_MTS_SCHEDULE6_CONTEXT_SHIFT 5U
1780+#define ROGUE_CR_MTS_SCHEDULE6_CONTEXT_CLRMSK 0xFFFFFFDFU
1781+#define ROGUE_CR_MTS_SCHEDULE6_CONTEXT_BGCTX 0x00000000U
1782+#define ROGUE_CR_MTS_SCHEDULE6_CONTEXT_INTCTX 0x00000020U
1783+#define ROGUE_CR_MTS_SCHEDULE6_TASK_SHIFT 4U
1784+#define ROGUE_CR_MTS_SCHEDULE6_TASK_CLRMSK 0xFFFFFFEFU
1785+#define ROGUE_CR_MTS_SCHEDULE6_TASK_NON_COUNTED 0x00000000U
1786+#define ROGUE_CR_MTS_SCHEDULE6_TASK_COUNTED 0x00000010U
1787+#define ROGUE_CR_MTS_SCHEDULE6_DM_SHIFT 0U
1788+#define ROGUE_CR_MTS_SCHEDULE6_DM_CLRMSK 0xFFFFFFF0U
1789+#define ROGUE_CR_MTS_SCHEDULE6_DM_DM0 0x00000000U
1790+#define ROGUE_CR_MTS_SCHEDULE6_DM_DM1 0x00000001U
1791+#define ROGUE_CR_MTS_SCHEDULE6_DM_DM2 0x00000002U
1792+#define ROGUE_CR_MTS_SCHEDULE6_DM_DM3 0x00000003U
1793+#define ROGUE_CR_MTS_SCHEDULE6_DM_DM4 0x00000004U
1794+#define ROGUE_CR_MTS_SCHEDULE6_DM_DM5 0x00000005U
1795+#define ROGUE_CR_MTS_SCHEDULE6_DM_DM6 0x00000006U
1796+#define ROGUE_CR_MTS_SCHEDULE6_DM_DM7 0x00000007U
1797+#define ROGUE_CR_MTS_SCHEDULE6_DM_DM_ALL 0x0000000FU
1798+
1799+/* Register ROGUE_CR_MTS_SCHEDULE7 */
1800+#define ROGUE_CR_MTS_SCHEDULE7 0x70B00U
1801+#define ROGUE_CR_MTS_SCHEDULE7_MASKFULL 0x00000000000001FFULL
1802+#define ROGUE_CR_MTS_SCHEDULE7_HOST_SHIFT 8U
1803+#define ROGUE_CR_MTS_SCHEDULE7_HOST_CLRMSK 0xFFFFFEFFU
1804+#define ROGUE_CR_MTS_SCHEDULE7_HOST_BG_TIMER 0x00000000U
1805+#define ROGUE_CR_MTS_SCHEDULE7_HOST_HOST 0x00000100U
1806+#define ROGUE_CR_MTS_SCHEDULE7_PRIORITY_SHIFT 6U
1807+#define ROGUE_CR_MTS_SCHEDULE7_PRIORITY_CLRMSK 0xFFFFFF3FU
1808+#define ROGUE_CR_MTS_SCHEDULE7_PRIORITY_PRT0 0x00000000U
1809+#define ROGUE_CR_MTS_SCHEDULE7_PRIORITY_PRT1 0x00000040U
1810+#define ROGUE_CR_MTS_SCHEDULE7_PRIORITY_PRT2 0x00000080U
1811+#define ROGUE_CR_MTS_SCHEDULE7_PRIORITY_PRT3 0x000000C0U
1812+#define ROGUE_CR_MTS_SCHEDULE7_CONTEXT_SHIFT 5U
1813+#define ROGUE_CR_MTS_SCHEDULE7_CONTEXT_CLRMSK 0xFFFFFFDFU
1814+#define ROGUE_CR_MTS_SCHEDULE7_CONTEXT_BGCTX 0x00000000U
1815+#define ROGUE_CR_MTS_SCHEDULE7_CONTEXT_INTCTX 0x00000020U
1816+#define ROGUE_CR_MTS_SCHEDULE7_TASK_SHIFT 4U
1817+#define ROGUE_CR_MTS_SCHEDULE7_TASK_CLRMSK 0xFFFFFFEFU
1818+#define ROGUE_CR_MTS_SCHEDULE7_TASK_NON_COUNTED 0x00000000U
1819+#define ROGUE_CR_MTS_SCHEDULE7_TASK_COUNTED 0x00000010U
1820+#define ROGUE_CR_MTS_SCHEDULE7_DM_SHIFT 0U
1821+#define ROGUE_CR_MTS_SCHEDULE7_DM_CLRMSK 0xFFFFFFF0U
1822+#define ROGUE_CR_MTS_SCHEDULE7_DM_DM0 0x00000000U
1823+#define ROGUE_CR_MTS_SCHEDULE7_DM_DM1 0x00000001U
1824+#define ROGUE_CR_MTS_SCHEDULE7_DM_DM2 0x00000002U
1825+#define ROGUE_CR_MTS_SCHEDULE7_DM_DM3 0x00000003U
1826+#define ROGUE_CR_MTS_SCHEDULE7_DM_DM4 0x00000004U
1827+#define ROGUE_CR_MTS_SCHEDULE7_DM_DM5 0x00000005U
1828+#define ROGUE_CR_MTS_SCHEDULE7_DM_DM6 0x00000006U
1829+#define ROGUE_CR_MTS_SCHEDULE7_DM_DM7 0x00000007U
1830+#define ROGUE_CR_MTS_SCHEDULE7_DM_DM_ALL 0x0000000FU
1831+
1832+/* Register ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC */
1833+#define ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC 0x0B30U
1834+#define ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC_MASKFULL 0x000000000000FFFFULL
1835+#define ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC_DM_ASSOC_SHIFT 0U
1836+#define ROGUE_CR_MTS_BGCTX_THREAD0_DM_ASSOC_DM_ASSOC_CLRMSK 0xFFFF0000U
1837+
1838+/* Register ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC */
1839+#define ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC 0x0B38U
1840+#define ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC_MASKFULL 0x000000000000FFFFULL
1841+#define ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC_DM_ASSOC_SHIFT 0U
1842+#define ROGUE_CR_MTS_BGCTX_THREAD1_DM_ASSOC_DM_ASSOC_CLRMSK 0xFFFF0000U
1843+
1844+/* Register ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC */
1845+#define ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC 0x0B40U
1846+#define ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC_MASKFULL 0x000000000000FFFFULL
1847+#define ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC_DM_ASSOC_SHIFT 0U
1848+#define ROGUE_CR_MTS_INTCTX_THREAD0_DM_ASSOC_DM_ASSOC_CLRMSK 0xFFFF0000U
1849+
1850+/* Register ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC */
1851+#define ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC 0x0B48U
1852+#define ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC_MASKFULL 0x000000000000FFFFULL
1853+#define ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC_DM_ASSOC_SHIFT 0U
1854+#define ROGUE_CR_MTS_INTCTX_THREAD1_DM_ASSOC_DM_ASSOC_CLRMSK 0xFFFF0000U
1855+
1856+/* Register ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG */
1857+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG 0x0B50U
1858+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG__S7_TOP__MASKFULL 0x000FF0FFFFFFF701ULL
1859+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_MASKFULL 0x0000FFFFFFFFF001ULL
1860+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_PC_BASE_SHIFT 44U
1861+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_PC_BASE_CLRMSK 0xFFFF0FFFFFFFFFFFULL
1862+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG__S7_TOP__FENCE_PC_BASE_SHIFT 44U
1863+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG__S7_TOP__FENCE_PC_BASE_CLRMSK 0xFFF00FFFFFFFFFFFULL
1864+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_DM_SHIFT 40U
1865+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_DM_CLRMSK 0xFFFFF0FFFFFFFFFFULL
1866+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_ADDR_SHIFT 12U
1867+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_ADDR_CLRMSK 0xFFFFFF0000000FFFULL
1868+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_PERSISTENCE_SHIFT 9U
1869+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_PERSISTENCE_CLRMSK 0xFFFFFFFFFFFFF9FFULL
1870+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_SLC_COHERENT_SHIFT 8U
1871+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_SLC_COHERENT_CLRMSK 0xFFFFFFFFFFFFFEFFULL
1872+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_FENCE_SLC_COHERENT_EN 0x0000000000000100ULL
1873+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_IDLE_CTRL_SHIFT 0U
1874+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_IDLE_CTRL_CLRMSK 0xFFFFFFFFFFFFFFFEULL
1875+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_IDLE_CTRL_META 0x0000000000000000ULL
1876+#define ROGUE_CR_MTS_GARTEN_WRAPPER_CONFIG_IDLE_CTRL_MTS 0x0000000000000001ULL
1877+
1878+/* Register ROGUE_CR_MTS_DM0_INTERRUPT_ENABLE */
1879+#define ROGUE_CR_MTS_DM0_INTERRUPT_ENABLE 0x0B58U
1880+#define ROGUE_CR_MTS_DM0_INTERRUPT_ENABLE_MASKFULL 0x00000000FFFFFFFFULL
1881+#define ROGUE_CR_MTS_DM0_INTERRUPT_ENABLE_INT_ENABLE_SHIFT 0U
1882+#define ROGUE_CR_MTS_DM0_INTERRUPT_ENABLE_INT_ENABLE_CLRMSK 0x00000000U
1883+
1884+/* Register ROGUE_CR_MTS_DM1_INTERRUPT_ENABLE */
1885+#define ROGUE_CR_MTS_DM1_INTERRUPT_ENABLE 0x0B60U
1886+#define ROGUE_CR_MTS_DM1_INTERRUPT_ENABLE_MASKFULL 0x00000000FFFFFFFFULL
1887+#define ROGUE_CR_MTS_DM1_INTERRUPT_ENABLE_INT_ENABLE_SHIFT 0U
1888+#define ROGUE_CR_MTS_DM1_INTERRUPT_ENABLE_INT_ENABLE_CLRMSK 0x00000000U
1889+
1890+/* Register ROGUE_CR_MTS_DM2_INTERRUPT_ENABLE */
1891+#define ROGUE_CR_MTS_DM2_INTERRUPT_ENABLE 0x0B68U
1892+#define ROGUE_CR_MTS_DM2_INTERRUPT_ENABLE_MASKFULL 0x00000000FFFFFFFFULL
1893+#define ROGUE_CR_MTS_DM2_INTERRUPT_ENABLE_INT_ENABLE_SHIFT 0U
1894+#define ROGUE_CR_MTS_DM2_INTERRUPT_ENABLE_INT_ENABLE_CLRMSK 0x00000000U
1895+
1896+/* Register ROGUE_CR_MTS_DM3_INTERRUPT_ENABLE */
1897+#define ROGUE_CR_MTS_DM3_INTERRUPT_ENABLE 0x0B70U
1898+#define ROGUE_CR_MTS_DM3_INTERRUPT_ENABLE_MASKFULL 0x00000000FFFFFFFFULL
1899+#define ROGUE_CR_MTS_DM3_INTERRUPT_ENABLE_INT_ENABLE_SHIFT 0U
1900+#define ROGUE_CR_MTS_DM3_INTERRUPT_ENABLE_INT_ENABLE_CLRMSK 0x00000000U
1901+
1902+/* Register ROGUE_CR_MTS_DM4_INTERRUPT_ENABLE */
1903+#define ROGUE_CR_MTS_DM4_INTERRUPT_ENABLE 0x0B78U
1904+#define ROGUE_CR_MTS_DM4_INTERRUPT_ENABLE_MASKFULL 0x00000000FFFFFFFFULL
1905+#define ROGUE_CR_MTS_DM4_INTERRUPT_ENABLE_INT_ENABLE_SHIFT 0U
1906+#define ROGUE_CR_MTS_DM4_INTERRUPT_ENABLE_INT_ENABLE_CLRMSK 0x00000000U
1907+
1908+/* Register ROGUE_CR_MTS_DM5_INTERRUPT_ENABLE */
1909+#define ROGUE_CR_MTS_DM5_INTERRUPT_ENABLE 0x0B80U
1910+#define ROGUE_CR_MTS_DM5_INTERRUPT_ENABLE_MASKFULL 0x00000000FFFFFFFFULL
1911+#define ROGUE_CR_MTS_DM5_INTERRUPT_ENABLE_INT_ENABLE_SHIFT 0U
1912+#define ROGUE_CR_MTS_DM5_INTERRUPT_ENABLE_INT_ENABLE_CLRMSK 0x00000000U
1913+
1914+/* Register ROGUE_CR_MTS_INTCTX */
1915+#define ROGUE_CR_MTS_INTCTX 0x0B98U
1916+#define ROGUE_CR_MTS_INTCTX_MASKFULL 0x000000003FFFFFFFULL
1917+#define ROGUE_CR_MTS_INTCTX_DM_HOST_SCHEDULE_SHIFT 22U
1918+#define ROGUE_CR_MTS_INTCTX_DM_HOST_SCHEDULE_CLRMSK 0xC03FFFFFU
1919+#define ROGUE_CR_MTS_INTCTX_DM_PTR_SHIFT 18U
1920+#define ROGUE_CR_MTS_INTCTX_DM_PTR_CLRMSK 0xFFC3FFFFU
1921+#define ROGUE_CR_MTS_INTCTX_THREAD_ACTIVE_SHIFT 16U
1922+#define ROGUE_CR_MTS_INTCTX_THREAD_ACTIVE_CLRMSK 0xFFFCFFFFU
1923+#define ROGUE_CR_MTS_INTCTX_DM_TIMER_SCHEDULE_SHIFT 8U
1924+#define ROGUE_CR_MTS_INTCTX_DM_TIMER_SCHEDULE_CLRMSK 0xFFFF00FFU
1925+#define ROGUE_CR_MTS_INTCTX_DM_INTERRUPT_SCHEDULE_SHIFT 0U
1926+#define ROGUE_CR_MTS_INTCTX_DM_INTERRUPT_SCHEDULE_CLRMSK 0xFFFFFF00U
1927+
1928+/* Register ROGUE_CR_MTS_BGCTX */
1929+#define ROGUE_CR_MTS_BGCTX 0x0BA0U
1930+#define ROGUE_CR_MTS_BGCTX_MASKFULL 0x0000000000003FFFULL
1931+#define ROGUE_CR_MTS_BGCTX_DM_PTR_SHIFT 10U
1932+#define ROGUE_CR_MTS_BGCTX_DM_PTR_CLRMSK 0xFFFFC3FFU
1933+#define ROGUE_CR_MTS_BGCTX_THREAD_ACTIVE_SHIFT 8U
1934+#define ROGUE_CR_MTS_BGCTX_THREAD_ACTIVE_CLRMSK 0xFFFFFCFFU
1935+#define ROGUE_CR_MTS_BGCTX_DM_NONCOUNTED_SCHEDULE_SHIFT 0U
1936+#define ROGUE_CR_MTS_BGCTX_DM_NONCOUNTED_SCHEDULE_CLRMSK 0xFFFFFF00U
1937+
1938+/* Register ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE */
1939+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE 0x0BA8U
1940+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_MASKFULL 0xFFFFFFFFFFFFFFFFULL
1941+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM7_SHIFT 56U
1942+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM7_CLRMSK 0x00FFFFFFFFFFFFFFULL
1943+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM6_SHIFT 48U
1944+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM6_CLRMSK 0xFF00FFFFFFFFFFFFULL
1945+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM5_SHIFT 40U
1946+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM5_CLRMSK 0xFFFF00FFFFFFFFFFULL
1947+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM4_SHIFT 32U
1948+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM4_CLRMSK 0xFFFFFF00FFFFFFFFULL
1949+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM3_SHIFT 24U
1950+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM3_CLRMSK 0xFFFFFFFF00FFFFFFULL
1951+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM2_SHIFT 16U
1952+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM2_CLRMSK 0xFFFFFFFFFF00FFFFULL
1953+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM1_SHIFT 8U
1954+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM1_CLRMSK 0xFFFFFFFFFFFF00FFULL
1955+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM0_SHIFT 0U
1956+#define ROGUE_CR_MTS_BGCTX_COUNTED_SCHEDULE_DM0_CLRMSK 0xFFFFFFFFFFFFFF00ULL
1957+
1958+/* Register ROGUE_CR_MTS_GPU_INT_STATUS */
1959+#define ROGUE_CR_MTS_GPU_INT_STATUS 0x0BB0U
1960+#define ROGUE_CR_MTS_GPU_INT_STATUS_MASKFULL 0x00000000FFFFFFFFULL
1961+#define ROGUE_CR_MTS_GPU_INT_STATUS_STATUS_SHIFT 0U
1962+#define ROGUE_CR_MTS_GPU_INT_STATUS_STATUS_CLRMSK 0x00000000U
1963+
1964+/* Register ROGUE_CR_MTS_SCHEDULE_ENABLE */
1965+#define ROGUE_CR_MTS_SCHEDULE_ENABLE 0x0BC8U
1966+#define ROGUE_CR_MTS_SCHEDULE_ENABLE_MASKFULL 0x00000000000000FFULL
1967+#define ROGUE_CR_MTS_SCHEDULE_ENABLE_MASK_SHIFT 0U
1968+#define ROGUE_CR_MTS_SCHEDULE_ENABLE_MASK_CLRMSK 0xFFFFFF00U
1969+
1970+/* Register ROGUE_CR_IRQ_OS0_EVENT_STATUS */
1971+#define ROGUE_CR_IRQ_OS0_EVENT_STATUS 0x0BD8U
1972+#define ROGUE_CR_IRQ_OS0_EVENT_STATUS_MASKFULL 0x0000000000000001ULL
1973+#define ROGUE_CR_IRQ_OS0_EVENT_STATUS_SOURCE_SHIFT 0U
1974+#define ROGUE_CR_IRQ_OS0_EVENT_STATUS_SOURCE_CLRMSK 0xFFFFFFFEU
1975+#define ROGUE_CR_IRQ_OS0_EVENT_STATUS_SOURCE_EN 0x00000001U
1976+
1977+/* Register ROGUE_CR_IRQ_OS0_EVENT_CLEAR */
1978+#define ROGUE_CR_IRQ_OS0_EVENT_CLEAR 0x0BE8U
1979+#define ROGUE_CR_IRQ_OS0_EVENT_CLEAR_MASKFULL 0x0000000000000001ULL
1980+#define ROGUE_CR_IRQ_OS0_EVENT_CLEAR_SOURCE_SHIFT 0U
1981+#define ROGUE_CR_IRQ_OS0_EVENT_CLEAR_SOURCE_CLRMSK 0xFFFFFFFEU
1982+#define ROGUE_CR_IRQ_OS0_EVENT_CLEAR_SOURCE_EN 0x00000001U
1983+
1984+/* Register ROGUE_CR_IRQ_OS1_EVENT_STATUS */
1985+#define ROGUE_CR_IRQ_OS1_EVENT_STATUS 0x10BD8U
1986+#define ROGUE_CR_IRQ_OS1_EVENT_STATUS_MASKFULL 0x0000000000000001ULL
1987+#define ROGUE_CR_IRQ_OS1_EVENT_STATUS_SOURCE_SHIFT 0U
1988+#define ROGUE_CR_IRQ_OS1_EVENT_STATUS_SOURCE_CLRMSK 0xFFFFFFFEU
1989+#define ROGUE_CR_IRQ_OS1_EVENT_STATUS_SOURCE_EN 0x00000001U
1990+
1991+/* Register ROGUE_CR_IRQ_OS1_EVENT_CLEAR */
1992+#define ROGUE_CR_IRQ_OS1_EVENT_CLEAR 0x10BE8U
1993+#define ROGUE_CR_IRQ_OS1_EVENT_CLEAR_MASKFULL 0x0000000000000001ULL
1994+#define ROGUE_CR_IRQ_OS1_EVENT_CLEAR_SOURCE_SHIFT 0U
1995+#define ROGUE_CR_IRQ_OS1_EVENT_CLEAR_SOURCE_CLRMSK 0xFFFFFFFEU
1996+#define ROGUE_CR_IRQ_OS1_EVENT_CLEAR_SOURCE_EN 0x00000001U
1997+
1998+/* Register ROGUE_CR_IRQ_OS2_EVENT_STATUS */
1999+#define ROGUE_CR_IRQ_OS2_EVENT_STATUS 0x20BD8U
2000+#define ROGUE_CR_IRQ_OS2_EVENT_STATUS_MASKFULL 0x0000000000000001ULL
2001+#define ROGUE_CR_IRQ_OS2_EVENT_STATUS_SOURCE_SHIFT 0U
2002+#define ROGUE_CR_IRQ_OS2_EVENT_STATUS_SOURCE_CLRMSK 0xFFFFFFFEU
2003+#define ROGUE_CR_IRQ_OS2_EVENT_STATUS_SOURCE_EN 0x00000001U
2004+
2005+/* Register ROGUE_CR_IRQ_OS2_EVENT_CLEAR */
2006+#define ROGUE_CR_IRQ_OS2_EVENT_CLEAR 0x20BE8U
2007+#define ROGUE_CR_IRQ_OS2_EVENT_CLEAR_MASKFULL 0x0000000000000001ULL
2008+#define ROGUE_CR_IRQ_OS2_EVENT_CLEAR_SOURCE_SHIFT 0U
2009+#define ROGUE_CR_IRQ_OS2_EVENT_CLEAR_SOURCE_CLRMSK 0xFFFFFFFEU
2010+#define ROGUE_CR_IRQ_OS2_EVENT_CLEAR_SOURCE_EN 0x00000001U
2011+
2012+/* Register ROGUE_CR_IRQ_OS3_EVENT_STATUS */
2013+#define ROGUE_CR_IRQ_OS3_EVENT_STATUS 0x30BD8U
2014+#define ROGUE_CR_IRQ_OS3_EVENT_STATUS_MASKFULL 0x0000000000000001ULL
2015+#define ROGUE_CR_IRQ_OS3_EVENT_STATUS_SOURCE_SHIFT 0U
2016+#define ROGUE_CR_IRQ_OS3_EVENT_STATUS_SOURCE_CLRMSK 0xFFFFFFFEU
2017+#define ROGUE_CR_IRQ_OS3_EVENT_STATUS_SOURCE_EN 0x00000001U
2018+
2019+/* Register ROGUE_CR_IRQ_OS3_EVENT_CLEAR */
2020+#define ROGUE_CR_IRQ_OS3_EVENT_CLEAR 0x30BE8U
2021+#define ROGUE_CR_IRQ_OS3_EVENT_CLEAR_MASKFULL 0x0000000000000001ULL
2022+#define ROGUE_CR_IRQ_OS3_EVENT_CLEAR_SOURCE_SHIFT 0U
2023+#define ROGUE_CR_IRQ_OS3_EVENT_CLEAR_SOURCE_CLRMSK 0xFFFFFFFEU
2024+#define ROGUE_CR_IRQ_OS3_EVENT_CLEAR_SOURCE_EN 0x00000001U
2025+
2026+/* Register ROGUE_CR_IRQ_OS4_EVENT_STATUS */
2027+#define ROGUE_CR_IRQ_OS4_EVENT_STATUS 0x40BD8U
2028+#define ROGUE_CR_IRQ_OS4_EVENT_STATUS_MASKFULL 0x0000000000000001ULL
2029+#define ROGUE_CR_IRQ_OS4_EVENT_STATUS_SOURCE_SHIFT 0U
2030+#define ROGUE_CR_IRQ_OS4_EVENT_STATUS_SOURCE_CLRMSK 0xFFFFFFFEU
2031+#define ROGUE_CR_IRQ_OS4_EVENT_STATUS_SOURCE_EN 0x00000001U
2032+
2033+/* Register ROGUE_CR_IRQ_OS4_EVENT_CLEAR */
2034+#define ROGUE_CR_IRQ_OS4_EVENT_CLEAR 0x40BE8U
2035+#define ROGUE_CR_IRQ_OS4_EVENT_CLEAR_MASKFULL 0x0000000000000001ULL
2036+#define ROGUE_CR_IRQ_OS4_EVENT_CLEAR_SOURCE_SHIFT 0U
2037+#define ROGUE_CR_IRQ_OS4_EVENT_CLEAR_SOURCE_CLRMSK 0xFFFFFFFEU
2038+#define ROGUE_CR_IRQ_OS4_EVENT_CLEAR_SOURCE_EN 0x00000001U
2039+

4154 more lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_cr_defs_client.h+159−0

This diff is large: its 160 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_defs.h+179−0

This diff is large: its 180 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif.h+2188−0

This diff is large: its 2189 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_check.h+493−0

This diff is large: its 494 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_client.h+373−0

This diff is large: its 374 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_client_check.h+133−0

This diff is large: its 134 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_common.h+60−0

This diff is large: its 61 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_dev_info.h+113−0

This diff is large: its 114 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_resetframework.h+28−0

This diff is large: its 29 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_sf.h+1648−0

This diff is large: its 1649 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_shared.h+258−0

This diff is large: its 259 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_shared_check.h+108−0

This diff is large: its 109 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_fwif_stream.h+78−0

This diff is large: its 79 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_heap_config.h+113−0

This diff is large: its 114 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_meta.h+356−0

This diff is large: its 357 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_mips.h+335−0

This diff is large: its 336 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_mips_check.h+58−0

This diff is large: its 59 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_rogue_mmu_defs.h+136−0

This diff is large: its 137 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_stream.c+285−0

This diff is large: its 286 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_stream.h+75−0

This diff is large: its 76 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_stream_defs.c+351−0

This diff is large: its 352 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_stream_defs.h+16−0

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_sync.c+289−0

This diff is large: its 290 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_sync.h+84−0

This diff is large: its 85 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_vm.c+1107−0

This diff is large: its 1108 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_vm.h+65−0

This diff is large: its 66 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_vm_mips.c+238−0

This diff is large: its 239 lines aren’t shown here.

drivers/gpu/drm/imagination/pvr_vm_mips.h+22−0

This diff is large: its 23 lines aren’t shown here.

drivers/gpu/drm/imx/dcss/dcss-drv.c+2−4

This diff is large: its 27 lines aren’t shown here.

drivers/gpu/drm/imx/lcdc/imx-lcdc.c+2−4

This diff is large: its 25 lines aren’t shown here.

drivers/gpu/drm/kmb/kmb_drv.c+2−3

This diff is large: its 26 lines aren’t shown here.

drivers/gpu/drm/mediatek/mtk_disp_ovl_adaptor.c+2−3

This diff is large: its 19 lines aren’t shown here.

drivers/gpu/drm/mediatek/mtk_ethdr.c+2−3

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/meson/meson_dw_mipi_dsi.c+2−4

This diff is large: its 24 lines aren’t shown here.

drivers/gpu/drm/nouveau/nouveau_platform.c+2−3

This diff is large: its 20 lines aren’t shown here.

drivers/gpu/drm/nouveau/nv04_fence.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/nouveau/nvkm/engine/fifo/chan.c+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/panel/panel-elida-kd35t133.c+3−34

This diff is large: its 107 lines aren’t shown here.

drivers/gpu/drm/panel/panel-newvision-nv3051d.c+46−11

This diff is large: its 93 lines aren’t shown here.

drivers/gpu/drm/rockchip/rk3066_hdmi.c+18−27

This diff is large: its 93 lines aren’t shown here.

drivers/gpu/drm/rockchip/rockchip_drm_vop2.c+5−0

This diff is large: its 24 lines aren’t shown here.

drivers/gpu/drm/rockchip/rockchip_vop2_reg.c+2−0

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/sprd/sprd_dpu.c+2−4

This diff is large: its 19 lines aren’t shown here.

drivers/gpu/drm/sprd/sprd_drm.c+2−3

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/sprd/sprd_dsi.c+2−4

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/tilcdc/tilcdc_drv.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/tiny/arcpgu.c+2−4

This diff is large: its 25 lines aren’t shown here.

drivers/video/fbdev/simplefb.c+4−0

This diff is large: its 11 lines aren’t shown here.

include/drm/drm_edid.h+0−5

This diff is large: its 12 lines aren’t shown here.

include/drm/drm_gpuvm.h+28−0

This diff is large: its 33 lines aren’t shown here.

include/drm/drm_print.h+1−1

This diff is large: its 9 lines aren’t shown here.

include/linux/sizes.h+9−0

This diff is large: its 18 lines aren’t shown here.

include/uapi/drm/pvr_drm.h+1297−0

This diff is large: its 1298 lines aren’t shown here.

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t